## Overview

There are two ways you can create and use blocks for Smart Filters:

1. With annotated GLSL code
1. With TypeScript + GLSL code

| | Annotated GLSL | TS + GLSL |
| ----------------------------------------------------------------------------------------------------------- | -------------- | --------- |
| Can use block in your own project | Yes | Yes |
| Can contribute block to @babylonjs/smart-filters-blocks | Yes | Yes |
| Can load directly in the Smart Filters Editor (SFE) | Yes | No |
| Can do custom logic in your bind function (e.g. math or packing multiple inputs into a single uniform) | No | Yes |

## Designing Your Block

### Terminology

| Term | Definition |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Design Time | When you are instantiating blocks, connecting them, and setting initial values of properties and input blocks |
| Runtime | Begins after you have called createRuntimeAsync() and are working with the SmartFilterRuntime returned - at this point input block values can be changed, but changes to block properties will have no effect on that instance of the SmartFilterRuntime |

### Uniforms vs. Consts

In a Smart Filter block, input connection points become uniforms, and properties of the block become consts.

| Pick | When |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Uniform | If the value needs to be settable at runtime (for example, to achieve an animation) |
| Const | If the value will never need to be changed at runtime. The benefit of a const is that the shader compiler can better optimize the code since it knows the value will never change. |

## Important GLSL Coding Requirements

<Alert severity="info">
Regardless of which approach you use for building your blocks, you need to make sure that the GLSL code for the block follows these requirements so that it is compatible with the Smart Filter optimizer.
</Alert>

These requirements ensure the optimizer can collapse sections of the Smart Filter into a single draw call while minimizing code size and ensuring correctness:

1. Helper functions (any function in the fragment shader other than the main function) must not directly access uniforms
 - Instead, pass uniform values into the helper function
 - Note: Helper functions may access const and define values
1. Only read from textures in the main function, do not pass a sampler2D into a helper function for it to read from
1. Reduce the texture2D() calls when possible
 - For example, instead of:
 ```GLSL
 if (mode == 1.) {
 color = texture2D(input, vUV);
 } else {
 color = texture2D(input, 1 - vUV);
 }
 ```
 - do this:
 ```GLSL
 if (mode == 1.) {
 uvToUse = vUV;
 } else {
 uvToUse = 1 - vUV;
 }
 color = texture2D(input, uvToUse);
 ```
1. Be sure your fragment shaders return values in the 0-1 range

 - Why: You can't rely on intermediate textures for clamping your values, because when your block is included in a Smart Filter and run through the optimizer, it
 will return its value directly to the next block in the Smart Filter instead of writing to an intermediate texture.
 - Debugging tip: If you see visual differences in the Smart Filter Editor when testing a Smart Filter and toggling the optimizer on and off, you can enable the "Clamp Return Values" debug mode (directly below the Optimize Smart Filter toggle) to have it inject clamps for you when it connects blocks. If that removes the inconsistency when toggling the optimizer on and off, that indicates you have a block that needs to clamp its output to be compatible with the optimizer.

### Testing Optimizer Compatibility

If you use the Annotated GLSL Code method, you can easily test that your block is optimizer-compatible by adding it to a Smart Filter in the Smart Filter Editor and then turning on the "Optimize Smart Filter" toggle in the Options pane on the right. Any errors will be displayed in the console at the bottom. See the following sections for more details.

## Using Annotated GLSL Code

This approach doesn't support custom vertex shaders or custom binding logic, but is the easiest to get started with and you can load these blocks directly into the [Smart Filters Editor](https://sfe.babylonjs.com).

### Naming Convention

The Smart Filters core comes with a build tool to convert these annotated GLSL files into TypeScript classes. If you plan to use your block in a package, such as @babylonjs/smart-filters-blocks, or in your own project that will use these tools, you should follow this naming convention:

`blackAndWhiteBlock.block.glsl`

### Getting Started

The easiest way to create a custom block using annotated GLSL code is to:

1. Author the file in the editor of your choice
1. Open the [Smart Filters Editor](https://sfe.babylonjs.com)
1. Click "Add" under "Custom Blocks" then select your GLSL file
1. Watch the console UI at the bottom of the editor for any errors
1. Drag your block into the Smart Filter and wire it up
1. To see changes, just modify your GLSL file, click "Add" under "Custom Blocks" again, and select it
 - Your new block definition will be loaded and the current Smart Filter rebuilt, so you'll see your changes immediately

### Sample

```annotated-GLSL
// { "smartFilterBlockType": "SerializedTintBlock" }

uniform sampler2D input;
// { "default": { "r": 1, "g": 0, "b": 0 } }
uniform vec3 tint;
// { "default": 0.25 }
uniform float amount;

vec4 mainImage(vec2 vUV) { // main
 vec4 color = texture2D(input, vUV);
 vec3 tinted = mix(color.rgb, tint, amount);
 return vec4(tinted, color.a);
}
```

### Creating Block Properties

Block properties are consts in the GLSL. If you want to introduce a block property (which unlike a uniform can only be set before the runtime is created), you can add an annotation directly above the const declaration like this:

```annotated-GLSL
// { "property": true }
const float mode = 1.;
```

When the block is selected in the Smart Filters Editor, you'll see the mode appear in the properties panel on the right:
![SFE](/img/how_to/smart-filters/const-prop-sfe-textbox.webp)

You can optionally supply a list of values to appear in a dropdown like this:

```annotated-GLSL
// { "property": { "options": { "Normal": 0, "Crazy": 1, "Odd": 2 } } }
const float mode = 1.;
```

And it will appear like this in the Smart Filters Editor:

![SFE](/img/how_to/smart-filters/const-prop-sfe-options.webp)

### Annotation Requirements

1. The file must start with a // comment containing a serialized JSON object of type `GlslHeader`
 - The smartFilterBlockType property is required
 - See [shaderConverter.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/smartFilters/src/utils/buildTools/shaderConverter.ts) for the full object definition
1. There must be a single main function which takes in a vec2 named vUV and returns a vec4, and it must have a comment on its line like this:
 `// main`
1. Functions must be declared with the open curly brace on the same line as the function name
1. Any uniforms that should have the same value across all instances of the same block should have a comment on their line like this:
 `// single`
1. Uniforms can have metadata, such as a default value if no connection is made to their corresponding connection point
 - In the line immediately above the uniform declaration, start with a // comment followed by a JSON object
 - The supported metadata is described in the `UniformMetadataProperties` type in [shaderConverter.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/smartFilters/src/utils/buildTools/shaderConverter.ts)

### Using Your Block

#### In the Smart Filters Editor

1. You can use it right away in the [Smart Filters Editor](https://sfe.babylonjs.com) by clicking "Add" in "Custom Blocks"

#### Contributing it to the Smart Filters block library

1. Clone [BabylonJS/Babylon.js](https://github.com/BabylonJS/Babylon.js)
1. Add your .glsl file to the appropriate path in packages/dev/smartFilterBlocks/src/blocks
1. Ensure it has a namespace defined in the `GlslHeader` which matches the path
1. The build will create a blackAndWhiteBlock.block.ts file from your blackAndWhiteBlock.block.glsl file
1. Ensure that auto generated blackAndWhiteBlock.block.ts file is exported from the appropriate index.ts files
1. To test your block in a local build of the Smart Filter Editor, either:
 - Select `Smart Filter Editor development (Edge)` or `Smart Filter Editor development (Chrome)` in Visual Studio Code's `Run and Debug` menu then hit F5
 - Run `npm run watch:assets:smart-filters` in a terminal, then `npm run watch:source:smart-filters` in another terminal, and finally `npm run serve -w @tools/smart-filters-editor` in a third terminal
1. Create a Pull Request

#### In Your Own Project (Build Time Deserialization)

When you have the blocks available at build time, it's advantageous to deserialize them at that point so the work doesn't have to be done at runtime. You also get the benefit of property accessors for input connection points.

1. See _Configuring Your Build_ below
1. Your annotated GLSL (e.g. blackAndWhiteBlock.block.glsl) is converted into a TypeScript class by the build (e.g. blackAndWhiteBlock.block.ts)
1. You can then import that class and use it in your Smart Filters:

```typescript
import { BlackAndWhiteBlock } from "./blackAndWhiteBlock.block.js";

const block = new BlackAndWhiteBlock(smartFilter, "B&W Block");
```

#### In Your Own Project (Runtime Deserialization)

Runtime deserialization is easier, and necessary if you need to consume blocks that are not available at build time, but it is slower when loading the block for the first time.

Here's how it is done:

```typescript
const blockDefinition = importCustomBlockDefinition(serializedAnnotatedGlsl);
const block = CustomShaderBlock.Create(smartFilter, "Name Of This Instance", blockDefinition);
// Access the block inputs connection points via block.inputs and the output connection point via .output
```

## Using TypeScript + GLSL Code

Use this approach if you need a custom vertex shader, custom binding logic, or do not need to load the block implementation dynamically at runtime in your application.

### Naming Convention

The Smart Filters core comes with a build tool to convert your GLSL files into TypeScript files that export a `ShaderProgram` your TypeScript block class can use. If you plan to use your block in a package, such as @babylonjs/smart-filters-blocks, or in your own project that will use these tools, you should follow this naming convention:

`compositionBlock.fragment.glsl`

### Adding to the Babylon Blocks Library

1. Clone [BabylonJS/Babylon.js](https://github.com/BabylonJS/Babylon.js)
1. Run `npm install`
1. Now create a TypeScript and GLSL file for your block
 - See compositionBlock.ts and compositionBlock.fragment.glsl as examples
1. Note that the TypeScript file imports a compositionBlock.fragment.js file which the build auto-generates from the .glsl file
1. Add an entry to [builtInBlockRegistrations.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/smartFilterBlocks/src/registration/builtInBlockRegistrations.ts)
1. To test your block in a local build of the Smart Filter Editor, either:
 - Select `Smart Filter Editor development (Edge)` or `Smart Filter Editor development (Chrome)` in Visual Studio Code's `Run and Debug` menu then hit F5
 - Run `npm run watch:assets:smart-filters` in a terminal, then `npm run watch:source:smart-filters` in another terminal, and finally `npm run serve -w @tools/smart-filters-editor` in a third terminal

### Adding to Your Own Project

1. See _Configuring Your Build_ below
1. You'll create a TypeScript file for your block (e.g. compositionBlock.ts) as well as a fragment shader file (e.g. compositionBlock.fragment.glsl)
1. You'll import the ShaderProgram and uniforms list from the file that buildShader.js creates at build time (e.g. compositionBlock.fragment.js) like this:

```typescript
import { uniforms, shaderProgram } from "./compositionBlock.fragment.js";
```

### Block Design Requirements

1. The main fragment shader function must take in `vec2 vUV` (the name must match)
1. You can choose if you want your shader block to be disableable
 - Disableable means that it will have a disabled input connection point, and will automatically pass the main input texture through to the output if the block is disabled
 - This is helpful in applications that may allow the user to turn many effects on and off, since this allows effects to be disabled instantly without the need to build a new Smart Filter runtime
 - There are different disable strategies you can choose to get the best shader performance, based on the details of your block (see `BlockDisableStrategy` in [disableableShaderBlock.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/smartFilters/src/blockFoundation/disableableShaderBlock.ts))
 - Do not use the `AutoSample` strategy if your shader modifies `vUV` before sampling the main input texture. `AutoSample` samples the texture with the original `vUV`; use the `Manual` strategy instead
 - You must specify a main input when creating a disableable shader block
 - In your fragment shader, you must put `// main` on the line that declares the uniform to be treated as the main input texture
 - If you choose the manual strategy, your fragment shader must check a uniform named `disabled`, which the system will add for you, and if it is true, return texture2D(your_main_input_texture, vUV)
1. Uniform names must match connection point names
1. The ShaderBinding.bind() function must not assume texture inputs will be defined
 - If the Smart Filter has been optimized, texture inputs may be replaced with return values from other blocks instead of texture uniforms
 - For example, don't have logic that branches based on if the texture input was defined

### Creating Block Properties

To create a property mapped to a const in a TypeScript + GLSL Code block, follow these steps:

1. Annotate the const in the GLSL code like this:

```annotated-GLSL
// { "property": true }
const float mode = 1.;
```

2. Add a property to your new block's class (with optional editableInPropertyPage metadata to make it visible in the Smart Filters Editor):

```typescript
@editableInPropertyPage("Mode", PropertyTypeForEdition.List, "PROPERTIES", {
 notifiers: { rebuild: true },
 options: [
 { label: "Normal", value: 0 },
 { label: "Crazy", value: 1 },
 { label: "Odd", value: 2 },
 ],
})
public mode: number = 1;

```

3. Override getShaderProgram() in the new block's class

```typescript
/**
 * Gets the shader program to use to render the block.
 * This adds the per-instance const values to the shader program.
 * @returns The shader program to use to render the block
 */
public override getShaderProgram() {
 const staticShaderProgram = super.getShaderProgram();

 // Since we are making changes only for this instance of the block, and
 // the disableableShaderProgram is static, we make a copy and modify that.
 const shaderProgramForThisInstance = CloneShaderProgram(staticShaderProgram);
 shaderProgramForThisInstance.fragment.constPerInstance = `const float _mode_ = ${this.mode.toFixed(1)};`;

 return shaderProgramForThisInstance;
}
```

It will then appear in the Smart Filters Editor like this:

![SFE](/img/how_to/smart-filters/const-prop-sfe-options.webp)

#### Important Notes

<Alert severity="info">

1. In your getShaderProgram() override, you must call CloneShaderProgram to ensure that the value you set is not applied to other instances of your new block type.
1. In the string you set constPerInstance to, you must add underscores before and after the symbol name you used in your GLSL

</Alert>

### Configuring Your Build

If you plan to include your new block in your own project and want to deserialize it at build time instead of runtime for the best performance, you can use build tools supplied by the Smart Filters core.

1. As part of your build, you'll need to run a tool called `buildShaders.js` and pass it a path to your blocks and the name of the Smart Filters package as parameters. Use scripts similar to these in your package.json (update ./src/blocks to the path where your block files can be found):

```package.json
"build": "npm run build:runTools && npm run build:blocks",
"build:runTools": "node ./node_modules/@babylonjs/smart-filters/dist/utils/buildTools/buildShaders.js ./src/blocks @babylonjs/smart-filters",
"build:blocks": "tsc -p ./tsconfig.build.json",
```

2. You can also use the provided [watchShaders.js](https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/smartFilters/src/utils/buildTools/watchShaders.ts) file for watch mode
 - Note you'll need to reference the js version of the utility in your node_modules folder
1. Add these lines to your `.gitignore` file so that these auto-generated files aren't committed to your repo:

```text
**/*.fragment.ts
**/*.block.ts
```

1. If you use VSCode, we suggest that you add this to your `.vscode/settings.json` file so that these auto-generated files don't appear in your editor:

```json
"files.exclude": {
 "**/*.fragment.ts": true,
 "**/*.block.ts": true
},
```

## Overview

The entire system has been split into two parts: the Smart Filter and the Smart Filter Runtime.

## Design Principles

1. Keep the Smart Filter core simple and ensure the runtime is fast
2. Provide an easy-to-use API surface
3. Support only loading the code you need, and support delay loading that code (e.g. deserialization is async so deserialization code doesn't have to be loaded until it is used)

### Smart Filter

The notion of a `SmartFilter` is a graph of blocks (all inheriting from `BaseBlock`) linked to each other through `ConnectionPoint`s.

During the initialization phase, the `SmartFilter` examines the graph and builds a list of commands to execute during each frame, storing them in a `CommandBuffer`:

- This keeps each frame render very fast by minimizing the amount of code running and branches in that code.
- Basically, every `BaseBlock` during its `generateCommandsAndGatherInitPromises` step will register in the `CommandBuffer` only the work it has to execute according to its options.
- In the end the render loop only needs to execute each command in the list.
- This also allows for easy injection of debug tools or logging in between each command without growing the minimum required code to run a filter.
- Each block is responsible for "injecting" its command into the Smart Filter's `CommandBuffer`.
- For example: the `ShaderBlock` highlights how convenient it can be to only switch once at init time between rendering to the main render target (if linked to the output) or to an intermediate texture if in the middle of the chain.

Each block is responsible for exposing its inputs and outputs through `registerInput` and `registerOutput`:

- This creates and stores the related `ConnectionPoint`.
- Any property values of a block must be updated only before `initialize` is called as the runtime won't reference it at render time.
- All values that need to be dynamically updated after `initialize` must be defined as connection points or 'StrongRef'.
- By convention, blocks defined in TypeScript should expose their input connection points as properties for convenience.

The `ConnectionPoints` are:

- Strongly typed, allowing more type safety while creating a `SmartFilter` in code, yet keeping enough flexibility to be understood efficiently at runtime.
- They are also responsible for their compatibility when linked to each other.

The Smart Filter is fully abstracted away from the runtime and does not even require a Babylon Engine to work with. It is only responsible for holding the "graph" of the Smart Filter or the "map" of all its blocks. A Smart Filter only needs a name to be created:

```typescript
const smartFilter = new SmartFilter("Simplest");
```

Once a Smart Filter exists, one can add as many blocks as necessary to it:

```typescript
const blur = new BlurBlock(smartFilter, "blur");
```

Then, the various blocks can be linked together:

```typescript
videoInput.output.connectTo(blur.input);
blur.output.connectTo(blackAndWhite.input);
```

Finally the last block should be linked into the Smart Filter output:

```typescript
titleInput.output.connectTo(smartFilter.output);
```

### Optimizations

You can activate two optimizations once you've created your graph and have an instance of `SmartFilter` ready.

The first is a graph optimizer which attempts to "merge" the "compatible" shader blocks to create an optimized version of the graph, thus reducing the final number of draw calls (as there are as many draw calls as there are shader blocks in the graph). This optimization pass will create a new instance of `SmartFilter` which you can use in place of the initial instance. Here's how to do it:

```typescript
const vfo = new SmartFilterOptimizer(smartFilter, {
 // filters is an (unoptimized) instance of SmartFilter
 maxSamplersInFragmentShader: engine.getCaps().maxTexturesImageUnits,
});
const optimizedSmartFilter = vfo.optimize();
```

One caveat is that a fragment shader has a limited number of samplers it can use, so we shouldn't merge blocks (even if they're compatible) once we've reached this limit. You should normally use the current GPU limit, so it is recommended to pass engine.getCaps().maxTexturesImageUnits as the maxSamplersInFragmentShader value.

The second optimization is a texture analyzer that traverses the graph and recycles textures between blocks (where possible), in order to limit the total number of textures used by the graph. You can activate it as follows:

```typescript
const rtg = new RenderTargetGenerator(true); // true to minimize the number of textures created
const runtime = await filter.createRuntimeAsync(this.engine, rtg);
```

### Alpha

Smart Filter blocks operate on straight alpha. If you need premultiplied alpha for your output, you can use a PreMultiplyAlphaBlock at the end of your Smart Filter. Note that the Smart Filter Editor sets the Engine option [premultipliedAlpha](/typedoc/interfaces/BABYLON.EngineOptions#premultipliedalpha) to false when creating the ThinEngine so it doesn't expect premultiplied alpha, so Smart Filters in the Smart Filter Editor do not need to use a PreMultiplyAlphaBlock.

## Runtime

To keep a clear separation between edit time, resource management, and render time, the filter itself cannot be rendered directly. To do so, a runtime needs to be created from the filter:

```typescript
const runtime = await smartFilter.createRuntimeAsync(engine);
```

<Alert severity="info">
Note that the same filter can be used across different runtimes.
</Alert>

The runtime contains the list of resources required to render the filter, such as intermediate textures, shaders, and buffers. It also owns a CommandBuffer containing the list of all the actions required to render the filter.

For instance the content of the command buffer for the simplest filter would be:

```console
----- Command buffer commands -----
 Owner: OutputBlock (output) - Command: OutputBlock.render
-----------------------------------
```

Whereas a more complex one could look like:

```console
----- Command buffer commands -----
 Owner: DirectionalBlurBlock (blurIV) - Command: DirectionalBlurBlock.render
 Owner: DirectionalBlurBlock (blurIH) - Command: DirectionalBlurBlock.render
 Owner: DirectionalBlurBlock (blurV) - Command: DirectionalBlurBlock.render
 Owner: DirectionalBlurBlock (blurH) - Command: DirectionalBlurBlock.render
 Owner: BlackAndWhiteBlock (blackAndWhite) - Command: BlackAndWhiteBlock.render
 Owner: FrameBlock (frame) - Command: FrameBlock.renderToCanvas
-----------------------------------
```

The command buffer is accessible through the runtime for debugging and logging purposes. The list of commands can be extended with custom ones if necessary, such as for introspection as we do in ./src/command/commandBufferDebugger.ts.

Rendering the current runtime is as simple as `runtime.render();`

<Alert severity="info">
Note that the runtime should be disposed once it is no longer needed to free GPU memory and prevent leaks.
</Alert>

## A Few Core Principles

- Be CPU efficient: for instance, keeping the number of commands and the amount of branching in commands as low as possible.
- Be memory efficient: no commands should allocate memory as it could trigger some garbage collection which could result in frame loss.
- Be GPU efficient: the graph and texture optimizers minimize the number of "passes" required to render an image and the GPU resources used by the graph.

## How to Install

Install the Smart Filters core package: `npm install @babylonjs/smart-filters`.

The package has been marked as `sideEffects: false` so you can freely import from the index and know that tree shaking will remove any code you don't end up using.

It requires the following peer dependencies:

- @babylonjs/core

## Important Notes

<Alert severity="info">
Do not rely on the path of files within the `@babylonjs/smart-filters` package.
 - Instead, import from the root index and allow tree-shaking to remove anything you don't use.
 - Otherwise, future updates could move or rename files and require you to update your references.

</Alert>

## Simplest Usage

This sample creates a trivial hardcoded SmartFilter, creates the runtime, and then calls render() to render one frame to the canvas.

<Alert severity="info">
Note: Smart Filters are currently compatible with the Babylon Engine and ThinEngine.
</Alert>

```typescript
// Create a SmartFilter
const smartFilter = new SmartFilter("Simplest");
const titleInput = new InputBlock(smartFilter, "logo", ConnectionPointType.Texture, logoTexture);
titleInput.output.connectTo(smartFilter.output);

// Create a SmartFilterRuntime
const engine = new ThinEngine(canvas, true);
const runtime = await smartFilter.createRuntimeAsync(engine);

// Render one frame
runtime.render();
```

## Using the Optimizer

This sample creates a more interesting SmartFilter, uses the SmartFilterOptimizer to optimize the SmartFilter, and creates a RenderTargetGenerator with optimizations enabled.

```typescript
// Create a SmartFilter
const smartFilter = new SmartFilter("BlackAndWhiteAndPixelate");
const logoInput = new InputBlock(smartFilter, "logo", ConnectionPointType.Texture, createStrongRef(logoTexture));
const blackAndWhite = new BlackAndWhiteBlock(smartFilter, "blackAndWhite");
const pixelate = new PixelateBlock(smartFilter, "pixelate");
const pixelateIntensity = new InputBlock(smartFilter, "intensity", ConnectionPointType.Float, 0.4);

logoInput.output.connectTo(blackAndWhite.input);
blackAndWhite.output.connectTo(pixelate.input);
pixelateIntensity.output.connectTo(pixelate.intensity);
pixelate.output.connectTo(smartFilter.output);

// Create and call the optimizer
const optimizer = new SmartFilterOptimizer(smartFilter);
const optimizedSmartFilter = optimizer.optimize();

// Create a RenderTargetGenerator with optimization enabled
const rtg = new RenderTargetGenerator(true);

// Create the ThinEngine and the SmartFilterRuntime
const engine = new ThinEngine(canvas, true);
const runtime = await optimizedSmartFilter.createRuntimeAsync(engine, rtg);

// Render one frame
runtime.render();
```

## Rendering to Another Render Target

In this example, we build a trivial hardcoded Smart Filter, then have it render to a RenderTargetWrapper that we supply instead of rendering to the canvas:

```typescript
// Create a SmartFilter
const smartFilter = new SmartFilter("Simplest");
const titleInput = new InputBlock(smartFilter, "logo", ConnectionPointType.Texture, logoTexture);
titleInput.output.connectTo(smartFilter.output);

// Tell the SmartFilter to render to RenderTargetWrapper we supply
smartFilter.outputBlock.renderTargetWrapper = myRenderTargetWrapper;

// Create a SmartFilterRuntime
const engine = new ThinEngine(canvas, true);
const runtime = await smartFilter.createRuntimeAsync(engine);

// Render one frame
runtime.render();
```

Notes:

1. The choice between rendering to the canvas and to a renderTargetWrapper is locked in for a runtime when you create the `SmartFilterRuntime`.
 - To change from rendering to the canvas to rendering to a RenderTargetWrapper, or the reverse, create a new runtime.
1. If you started with a RenderTargetWrapper, you can change outputBlock.renderTargetWrapper to a different RenderTargetWrapper at runtime, and the next render() call will target it instead.
 - If the new renderTargetWrapper is a different size from the previous one, call smartFilter.resize() before the next render() call is made.

## Handling Resize

If the canvas changes size (or you are using outputBlock.renderTargetWrapper and change to a new RenderTargetWrapper), be sure to call smartFilter.resize() to ensure all intermediate textures are resized as well.

## Full Example

See [Introduction To Smart Filters](../introductionToSmartFilters) for more information about the demo application included in the Smart Filters repo.

## Overview

Smart Filters are "full screen" effects built by composing blocks together into a `SmartFilter` object, from which a `SmartFilterRuntime` is created.

You can use the built-in blocks or use blocks you create yourself to build your Smart Filters.

The Smart Filter core comes with an optimizer which can take a `SmartFilter` and return an optimized `SmartFilter` which consolidates the shader blocks into as few as possible. The `RenderTargetGenerator` used when creating a `SmartFilterRuntime` also contains an optimizer which will reuse intermediate textures whenever possible within the Smart Filter.

## How To Explore

### With the Smart Filter Editor

To familiarize yourself with Smart Filters, you can start by trying out the [Smart Filter Editor](https://sfe.babylonjs.com).
![SFE](/img/how_to/smart-filters/sfe-default.webp)

This tool lets you visually construct and run Smart Filters using the provided built-in blocks, as well as custom blocks you create and load into the tool, all before you even clone the repo.

Read more about the Smart Filters Editor [here](../../../../toolsAndResources/sfe/)

### In the Playground

You can also experiment with Smart Filters in the Playground!

<Playground id="#N7VSE6#4" title="Smart Filter Camera" description="Demonstrates how a Smart Filter can be used to apply an effect to the camera in a scene." />
<Playground id="#1VIWKL#5" title="Smart Filter Camera from SFE" description="Demonstrates how a Smart Filter saved from the Smart Filter Editor can be used to apply an effect to the camera in a scene." />

### With Code

Clone [BabylonJS/Babylon.js](https://github.com/BabylonJS/Babylon.js), and you can see how the Smart Filters Editor application uses Smart Filters.

Once you've cloned, run `npm install` then choose one of these approaches to build the Smart Filter Editor:

- Select `Smart Filter Editor development (Edge)` or `Smart Filter Editor development (Chrome)` in Visual Studio Code's `Run and Debug` menu then hit F5
- Run `npm run watch:assets:smart-filters` in a terminal, then `npm run watch:source:smart-filters` in another terminal, and finally `npm run serve -w @tools/smart-filters-editor` in a third terminal

#### Key Files

- [app.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/tools/smartFiltersEditor/src/app.ts)
 - The main file for the Smart Filter Editor - it handles loading and saving Smart Filters and launching the editor control
- [smartFilterLoader.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/tools/smartFiltersEditor/src/smartFilterLoader.ts)
 - The helper which takes a deserializer and uses it to load Smart Filters
- [smartFilterRenderer.ts](https://github.com/BabylonJS/Babylon.js/blob/master/packages/tools/smartFiltersEditor/src/smartFilterRenderer.ts)
 - The helper that takes a loaded Smart Filter and begins rendering it


Babylon Smart Filters is a node-based graph system for building GPU-accelerated 2D effects for scenarios like real-time video filters such as background blur. While the Node Material Editor is a graphical tool for creating shaders, Smart Filters is for generating a pipeline of shaders. We call them smart because this feature includes an optimizer that automatically minimizes the number of draw calls and intermediate textures for you.

You can build Smart Filters programmatically:

```javascript
const smartFilter = new SmartFilter("Default");

const pixelateBlock = new PixelateBlock(smartFilter, "Pixelate");

const textureInputBlock = new InputBlock(smartFilter, "Texture", ConnectionPointType.Texture, null);
textureInputBlock.editorData = {
  url: "/assets/logo.png",
  urlTypeHint: "image",
  flipY: true,
  anisotropicFilteringLevel: null,
  forcedExtension: null,
};

const intensityInputBlock = new InputBlock(smartFilter, "Intensity", ConnectionPointType.Float, 0.5);
intensityInputBlock.editorData = {
  animationType: null,
  valueDeltaPerMs: null,
  min: 0.0,
  max: 1.0,
};

textureInputBlock.output.connectTo(pixelateBlock.input);
intensityInputBlock.output.connectTo(pixelateBlock.intensity);
pixelateBlock.output.connectTo(smartFilter.output);
```

Or visually using a new tool, the [Smart Filter Editor](https://sfe.babylonjs.com):
![SFE](/img/how_to/smart-filters/sfe-default.webp)

See the following subtopics to learn how to get started with Smart Filters, how to integrate them into your projects, how to create custom blocks, and more.



# In-Depth layerMask

`layerMask` is a property you can find on many objects and may be a bit difficult to grasp at first. This page should give you a good understanding of how it works across Babylon.js objects.

## layerMask meaning

`layerMask` is a value that can be assigned to different objects (like meshes, cameras, layers, particle systems, etc.) and is used to determine the visibility of one object from another (for example, whether a mesh is visible from a camera or whether a layer should be displayed after a given camera is processed).

The only two things to understand with `layerMask` are that:
* it can be any number from 0 to 0xFFFFFFFF (or 4294967295, but it’s easier to write 0xFFFFFFFF)
* when computing a visibility flag, two values of `layerMask` are **AND**ed and if the result is different from 0 the flag is true

So, for example, if `camera.layerMask = 35` and `mesh.layerMask = 2`, `35 & 2 == 2 != 0`, so the mesh is visible from this camera. We often show these numbers in hexadecimal because doing the **AND** as a mental calculation is easier that way: `35=0x23` and `2=0x02` and `35 & 2 == 0x23 & 0x02`, and from this latter representation it’s easier to see that the result is 2 (well, at least when you are a bit comfortable with hexadecimal!).

## Usage in Babylon.js

In Babylon, by default all `layerMask` properties are `0x0FFFFFFF` (for Camera, Mesh, Layer, …). So, when it comes to determining visibility, `0x0FFFFFFF & 0x0FFFFFFF != 0` and everything is always visible (at least from the standpoint of the `layerMask` test).

As you can see, there are 4 bits not set by default in the `layerMask` (the leading `0` before the first `F`). This makes `layerMask` easier to use, as a common use case is to hide some meshes from a camera. To achieve that, you need to set a `layerMask` on the meshes so that the **AND** operation with the `layerMask` of the camera (`0x0FFFFFFF` by default) is 0: you can use `0x10000000`, `0x20000000`, `0x40000000` or `0x80000000`. Now you can see why showing these numbers in hexadecimal helps: it’s a lot easier to see that `0x10000000 & 0x0FFFFFFF = 0` than `268435456 & 268435455`!

If the default value for `layerMask` was `0xFFFFFFFF`, you would also need to update the `layerMask` of your camera to have your meshes invisible from it because `0xFFFFFFFF` is an all-one 32 bit value. So the `0x0FFFFFFF` value is simply a default value that can help make your life a little easier, but in all generality you can put any value in the `layerMask` property of cameras, meshes, layers, etc (as in the `35 & 2` example above) and simply understand how it is used to compute visibility.

### Layers and Skeleton Animations
If the active camera in a scene is using a layerMask to hide meshes that are animated with skeletons, then other cameras that do show the meshes will not show the animation because the skeletons are not being updated. A solution is to prepare the skeleton before each frame is rendered.

```typescript
let scene: BABYLON.Scene
let avatar: BABYLON.TransformNode;

scene.onBeforeRenderObservable.add(onBeforeRender);

function onBeforeRender(scene:BABYLON.Scene, eventState: BABYLON.EventState){
    const meshes = avatar.getChildMeshes(false);

    meshes.forEach(prepareSkeleton);
}

function prepareSkeleton(mesh:BABYLON.AbstractMesh){
    const skeleton = mesh.skeleton;

    if (!skeleton) return;

    skeleton.prepare();
}
```


Smart Filters

Coming next

How To Use Smart Filters

Learn how to get started with Smart Filters

Introduction To Smart Filters

Learn how to get started with Smart Filters

How Smart Filters Work

Learn how the Smart Filters core works

Creating New Blocks

Learn how to get started with Smart Filters