Creating New Blocks

Overview

There are two ways you can create and use blocks for Smart Filters:

1. With annotated GLSL code
2. With TypeScript + GLSL code

Designing Your Block

Terminology

Uniforms vs. Consts

In a Smart Filter block, input connection points become uniforms, and properties of the block become consts.

Important GLSL Coding Requirements

These requirements ensure the optimizer can collapse sections of the Smart Filter into a single draw call while minimizing code size and ensuring correctness:

1. Helper functions (any function in the fragment shader other than the main function) must not directly access uniforms

   • Instead, pass uniform values into the helper function
   • Note: Helper functions may access const and define values

2. Only read from textures in the main function, do not pass a sampler2D into a helper function for it to read from

3. Reduce the texture2D() calls when possible

   • For example, instead of:
   if (mode == 1.) {
     color = texture2D(input, vUV);
   } else {
     color = texture2D(input, 1 - vUV);
   }

   • do this:
   if (mode == 1.) {
     uvToUse = vUV;
   } else {
     uvToUse = 1 - vUV;
   }
   color = texture2D(input, uvToUse);

4. Be sure your fragment shaders return values in the 0-1 range

   • Why: You can't rely on intermediate textures for clamping your values, because when your block is included in a Smart Filter and run through the optimizer, it will return its value directly to the next block in the Smart Filter instead of writing to an intermediate texture.
   • Debugging tip: If you see visual differences in the Smart Filter Editor when testing a Smart Filter and toggling the optimizer on and off, you can enable the "Clamp Return Values" debug mode (directly below the Optimize Smart Filter toggle) to have it inject clamps for you when it connects blocks. If that removes the inconsistency when toggling the optimizer on and off, that indicates you have a block that needs to clamp its output to be compatible with the optimizer.

Testing Optimizer Compatibility

If you use the Annotated GLSL Code method, you can easily test that your block is optimizer-compatible by adding it to a Smart Filter in the Smart Filter Editor and then turning on the "Optimize Smart Filter" toggle in the Options pane on the right. Any errors will be displayed in the console at the bottom. See the following sections for more details.

Using Annotated GLSL Code

This approach doesn't support custom vertex shaders or custom binding logic, but is the easiest to get started with and you can load these blocks directly into the Smart Filters Editor.

Naming Convention

The Smart Filters core comes with a build tool to convert these annotated GLSL files into TypeScript classes. If you plan to use your block in a package, such as @babylonjs/smart-filters-blocks, or in your own project that will use these tools, you should follow this naming convention:

blackAndWhiteBlock.block.glsl

Getting Started

The easiest way to create a custom block using annotated GLSL code is to:

1. Author the file in the editor of your choice
2. Open the Smart Filters Editor
3. Click "Add" under "Custom Blocks" then select your GLSL file
4. Watch the console UI at the bottom of the editor for any errors
5. Drag your block into the Smart Filter and wire it up
6. To see changes, just modify your GLSL file, click "Add" under "Custom Blocks" again, and select it
   • Your new block definition will be loaded and the current Smart Filter rebuilt, so you'll see your changes immediately

Sample

// { "smartFilterBlockType": "SerializedTintBlock" }

uniform sampler2D input;
// { "default": { "r": 1, "g": 0, "b": 0 } }
uniform vec3 tint;
// { "default": 0.25 }
uniform float amount;

vec4 mainImage(vec2 vUV) { // main
    vec4 color = texture2D(input, vUV);
    vec3 tinted = mix(color.rgb, tint, amount);
    return vec4(tinted, color.a);
}

Creating Block Properties

Block properties are consts in the GLSL. If you want to introduce a block property (which unlike a uniform can only be set before the runtime is created), you can add an annotation directly above the const declaration like this:

// { "property": true }
const float mode = 1.;

When the block is selected in the Smart Filters Editor, you'll see the mode appear in the properties panel on the right: Expand

You can optionally supply a list of values to appear in a dropdown like this:

// { "property": { "options": { "Normal": 0, "Crazy": 1, "Odd": 2 } } }
const float mode = 1.;

And it will appear like this in the Smart Filters Editor:

Expand

Annotation Requirements

1. The file must start with a // comment containing a serialized JSON object of type GlslHeader
   • The smartFilterBlockType property is required
   • See shaderConverter.ts for the full object definition
2. There must be a single main function which takes in a vec2 named vUV and returns a vec4, and it must have a comment on its line like this: // main
3. Functions must be declared with the open curly brace on the same line as the function name
4. Any uniforms that should have the same value across all instances of the same block should have a comment on their line like this: // single
5. Uniforms can have metadata, such as a default value if no connection is made to their corresponding connection point
   • In the line immediately above the uniform declaration, start with a // comment followed by a JSON object
   • The supported metadata is described in the UniformMetadataProperties type in shaderConverter.ts

Using Your Block

In the Smart Filters Editor

1. You can use it right away in the Smart Filters Editor by clicking "Add" in "Custom Blocks"

Contributing it to the Smart Filters block library

1. Clone BabylonJS/Babylon.js
2. Add your .glsl file to the appropriate path in packages/dev/smartFilterBlocks/src/blocks
3. Ensure it has a namespace defined in the GlslHeader which matches the path
4. The build will create a blackAndWhiteBlock.block.ts file from your blackAndWhiteBlock.block.glsl file
5. Ensure that auto generated blackAndWhiteBlock.block.ts file is exported from the appropriate index.ts files
6. To test your block in a local build of the Smart Filter Editor, either:
   • Select Smart Filter Editor development (Edge) or Smart Filter Editor development (Chrome) in Visual Studio Code's Run and Debug menu then hit F5
   • Run npm run watch:assets:smart-filters in a terminal, then npm run watch:source:smart-filters in another terminal, and finally npm run serve -w @tools/smart-filters-editor in a third terminal
7. Create a Pull Request

In Your Own Project (Build Time Deserialization)

When you have the blocks available at build time, it's advantageous to deserialize them at that point so the work doesn't have to be done at runtime. You also get the benefit of property accessors for input connection points.

1. See Configuring Your Build below
2. Your annotated GLSL (e.g. blackAndWhiteBlock.block.glsl) is converted into a TypeScript class by the build (e.g. blackAndWhiteBlock.block.ts)
3. You can then import that class and use it in your Smart Filters:

import { BlackAndWhiteBlock } from "./blackAndWhiteBlock.block.js";

const block = new BlackAndWhiteBlock(smartFilter, "B&W Block");

In Your Own Project (Runtime Deserialization)

Runtime deserialization is easier, and necessary if you need to consume blocks that are not available at build time, but it is slower when loading the block for the first time.

Here's how it is done:

const blockDefinition = importCustomBlockDefinition(serializedAnnotatedGlsl);
const block = CustomShaderBlock.Create(smartFilter, "Name Of This Instance", blockDefinition);
// Access the block inputs connection points via block.inputs and the output connection point via .output

Using TypeScript + GLSL Code

Use this approach if you need a custom vertex shader, custom binding logic, or do not need to load the block implementation dynamically at runtime in your application.

Naming Convention

The Smart Filters core comes with a build tool to convert your GLSL files into TypeScript files that export a ShaderProgram your TypeScript block class can use. If you plan to use your block in a package, such as @babylonjs/smart-filters-blocks, or in your own project that will use these tools, you should follow this naming convention:

compositionBlock.fragment.glsl

Adding to the Babylon Blocks Library

1. Clone BabylonJS/Babylon.js
2. Run npm install
3. Now create a TypeScript and GLSL file for your block
   • See compositionBlock.ts and compositionBlock.fragment.glsl as examples
4. Note that the TypeScript file imports a compositionBlock.fragment.js file which the build auto-generates from the .glsl file
5. Add an entry to builtInBlockRegistrations.ts
6. To test your block in a local build of the Smart Filter Editor, either:
   • Select Smart Filter Editor development (Edge) or Smart Filter Editor development (Chrome) in Visual Studio Code's Run and Debug menu then hit F5
   • Run npm run watch:assets:smart-filters in a terminal, then npm run watch:source:smart-filters in another terminal, and finally npm run serve -w @tools/smart-filters-editor in a third terminal

Adding to Your Own Project

1. See Configuring Your Build below
2. You'll create a TypeScript file for your block (e.g. compositionBlock.ts) as well as a fragment shader file (e.g. compositionBlock.fragment.glsl)
3. You'll import the ShaderProgram and uniforms list from the file that buildShader.js creates at build time (e.g. compositionBlock.fragment.js) like this:

import { uniforms, shaderProgram } from "./compositionBlock.fragment.js";

Block Design Requirements

1. The main fragment shader function must take in vec2 vUV (the name must match)
2. You can choose if you want your shader block to be disableable
   • Disableable means that it will have a disabled input connection point, and will automatically pass the main input texture through to the output if the block is disabled
   • This is helpful in applications that may allow the user to turn many effects on and off, since this allows effects to be disabled instantly without the need to build a new Smart Filter runtime
   • There are different disable strategies you can choose to get the best shader performance, based on the details of your block (see BlockDisableStrategy in disableableShaderBlock.ts)
   • Do not use the AutoSample strategy if your shader modifies vUV before sampling the main input texture. AutoSample samples the texture with the original vUV; use the Manual strategy instead
   • You must specify a main input when creating a disableable shader block
     • In your fragment shader, you must put // main on the line that declares the uniform to be treated as the main input texture
   • If you choose the manual strategy, your fragment shader must check a uniform named disabled, which the system will add for you, and if it is true, return texture2D(your_main_input_texture, vUV)
3. Uniform names must match connection point names
4. The ShaderBinding.bind() function must not assume texture inputs will be defined
   • If the Smart Filter has been optimized, texture inputs may be replaced with return values from other blocks instead of texture uniforms
   • For example, don't have logic that branches based on if the texture input was defined

Creating Block Properties

To create a property mapped to a const in a TypeScript + GLSL Code block, follow these steps:

1. Annotate the const in the GLSL code like this:

// { "property": true }
const float mode = 1.;

2. Add a property to your new block's class (with optional editableInPropertyPage metadata to make it visible in the Smart Filters Editor):

@editableInPropertyPage("Mode", PropertyTypeForEdition.List, "PROPERTIES", {
    notifiers: { rebuild: true },
    options: [
        { label: "Normal", value: 0 },
        { label: "Crazy", value: 1 },
        { label: "Odd", value: 2 },
    ],
})
public mode: number = 1;

3. Override getShaderProgram() in the new block's class

/**
 * Gets the shader program to use to render the block.
 * This adds the per-instance const values to the shader program.
 * @returns The shader program to use to render the block
 */
public override getShaderProgram() {
    const staticShaderProgram = super.getShaderProgram();

    // Since we are making changes only for this instance of the block, and
    // the disableableShaderProgram is static, we make a copy and modify that.
    const shaderProgramForThisInstance = CloneShaderProgram(staticShaderProgram);
    shaderProgramForThisInstance.fragment.constPerInstance = `const float _mode_ = ${this.mode.toFixed(1)};`;

    return shaderProgramForThisInstance;
}

It will then appear in the Smart Filters Editor like this:

Expand

Important Notes

Configuring Your Build

If you plan to include your new block in your own project and want to deserialize it at build time instead of runtime for the best performance, you can use build tools supplied by the Smart Filters core.

1. As part of your build, you'll need to run a tool called buildShaders.js and pass it a path to your blocks and the name of the Smart Filters package as parameters. Use scripts similar to these in your package.json (update ./src/blocks to the path where your block files can be found):

"build": "npm run build:runTools && npm run build:blocks",
"build:runTools": "node ./node_modules/@babylonjs/smart-filters/dist/utils/buildTools/buildShaders.js ./src/blocks @babylonjs/smart-filters",
"build:blocks": "tsc -p ./tsconfig.build.json",

2. You can also use the provided watchShaders.js file for watch mode
   • Note you'll need to reference the js version of the utility in your node_modules folder
3. Add these lines to your .gitignore file so that these auto-generated files aren't committed to your repo:

**/*.fragment.ts
**/*.block.ts

1. If you use VSCode, we suggest that you add this to your .vscode/settings.json file so that these auto-generated files don't appear in your editor:

"files.exclude": {
    "**/*.fragment.ts": true,
    "**/*.block.ts": true
},