Image Filter Control

The Babylon.js Filter Control is a web control built on top of Babylon.js to apply filters to pictures on web pages at blazing-fast speed.

Introduction

Filtering images to apply effects can be tedious and slow on the CPU. The best place to do this, for a wide variety of effects, is on the GPU. But setting up an entire WebGL pipeline just to process images can be tricky, especially if you want to benefit from WebGL 2-to-1 fallback and work around well-known platform issues.

To greatly simplify this task we introduced the ImageFilter Control.

ImageFilter

How to use

Installation

To get started with the image filter control, you first need to install the controls npm package.

npm install @babylonjs/controls

To reduce the size of your web page, the controls library is based on the ES6 version of @babylonjs/core used as a peer dependency. Therefore, if you are not relying on it so far in your project, you also need to install core:

npm install @babylonjs/core

Instantiation

Once done, you can now import the control in your code:

import { ImageFilter } from "@babylonjs/controls/imageFilter";

And simply instantiate it on your page:

const imageFilter = new ImageFilter(filterCanvas);

You simply need to provide a canvas on which we can use a WebGL context. You can also provide another Babylon.js control to share the WebGL context.

Also, if you need more specific engine configurations or want to use post processes, you should manually use your own engine:

const engine = new Engine(filterCanvas);
const imageFilter = new ImageFilter(engine);

By default, the control relies on ThinEngine to optimize your bundle, but it might have limitations you do not want in your experience.

Using Post Process

In order to apply an existing post process as the image filter, you can use the following code:

const blackAndWhitePostProcess = new BlackAndWhitePostProcess("bw", 1, null, undefined, engine);
backAndWhiteFilter.filter(imageToProcess, blackAndWhitePostProcess);

Where imageToProcess could be either:

the url of a picture.
a video element (the current visible frame of the video will be used)
another canvas element (the current visible state will be used)

This will apply the post process to the provided input and display it in the canvas.

Using Custom Shader

In order to apply a custom shader as the image filter, you can use the following code:

const customEffectWrapper = new EffectWrapper({
  name: "Custom",
  engine: customFilter.engine,
  fragmentShader: `
        // Samplers
        varying vec2 vUV;
        uniform sampler2D textureSampler;
        
        void main(void) 
        {
            gl_FragColor = texture2D(textureSampler, vUV);

            // Swizzle channels
            float r = gl_FragColor.r;
            gl_FragColor.r = gl_FragColor.b;
            gl_FragColor.b = r;
        }
    `,
  samplerNames: ["textureSampler"],
});
customFilter.filter(imageToProcess, customEffectWrapper);

Where imageToProcess could be either:

the url of a picture.
a video element (the current visible frame of the video will be used)
another canvas element (the current visible state will be used)

This will apply the custom shader passed as a parameter to the picture. By default, vUV is available as a varying that defines the full output as texture coordinates. textureSampler needs to be present and defines the texture corresponding to the input parameter.

Please note that if you need to add custom uniforms or samplers, they should be defined in the effect wrapper lists:

const customEffectWrapper = new EffectWrapper({
  name: "Custom",
  engine: customFilter.engine,
  fragmentShader: `
        varying vec2 vUV;
        
        // Default Sampler
        uniform sampler2D textureSampler;

        // Custom uniforms
        uniform sampler2D otherTexture;
        uniform vec3 colorOffset;

        const vec2 scale = vec2(0.25, 1.);

        void main(void) 
        {
            gl_FragColor = texture2D(textureSampler, vUV);

            // Swizzle channels
            float r = gl_FragColor.r;
            gl_FragColor.r = gl_FragColor.b;
            gl_FragColor.b = r;
            gl_FragColor.rgb += clamp(colorOffset, 0., 1.);

            gl_FragColor.rgb *= texture2D(otherTexture, vUV * scale).rgb;
        }
    `,
  // Defines the list of existing samplers (default + customs).
  samplerNames: ["textureSampler", "otherTexture"],
  // Defines the list of existing uniform to be bound.
  uniformNames: ["colorOffset"],
});

Then you can simply bind them either in onApply:

customEffectWrapper.onApplyObservable.add(() => {
  // Sets the custom values.
  customEffectWrapper.effect.setTexture("otherTexture", otherTexture);
  customEffectWrapper.effect.setFloat3("colorOffset", 0.2, 0, 0.2);
});

Or during the render loop:

// Rely on the underlying engine render loop to update the filter result every frame.
engine.runRenderLoop(() => {
  // Only render if the custom texture is ready (the default one is
  // checked for you by the render function)
  if (!otherTexture.isReady()) {
    return;
  }

  // Sets the custom values.
  time += engine.getDeltaTime() / 1000;
  customEffectWrapper.effect.setTexture("otherTexture", otherTexture);
  customEffectWrapper.effect.setFloat3("colorOffset", Math.cos(time) * 0.5 + 0.5, 0, Math.sin(time) * 0.5 + 0.5);

  // Render. Please note we are using render instead of filter to improve
  // performances of real time filter. filter is creating a promise and will therefore
  // generate some lags and garbage.
  customFilter.render(mainTexture, customEffectWrapper);
});

Finally, if you are relying on new textures, you need to wait for them to be ready before rendering.

Process to the Canvas

This is by far the simplest option if you have a canvas on your page. You simply need to use the following code to fit the provided element to the canvas size:

imageFilter.filter(imageToProcess, filter);

On the previous line, imageToProcess could be either:

the url of a picture.
a video element (the current visible frame of the video will be used)
another canvas element (the current visible state will be used)

This is the default behavior.

Process to a Babylon Texture

Instead of filtering directly to a canvas, you may prefer to create only a Babylon.js texture on the GPU. For this, you can use the following function:

const texture = imageFilter.getFilteredTexture(imageToResize, { width: 128, height: 100 }, filter);

Like before, imageToResize could be either:

the url of a picture.
a video element (the current visible frame of the video will be used).
another canvas element (the current visible state will be used).

You also need to provide the size you want your texture to have on the GPU.

Now you are free to use this texture with any other controls.

Real-Time Filtering

Instead of filtering only once, you might want to create dynamic real-time effects. For this, you can simply render the effect during the render loop:

// Rely on the underlying engine render loop to update the filter result every frame.
engine.runRenderLoop(() => {
  // Only render if the custom texture is ready (the default one is
  // checked for you by the render function)
  if (!otherTexture.isReady()) {
    return;
  }

  // Sets the custom values.
  time += engine.getDeltaTime() / 1000;
  customEffectWrapper.effect.setTexture("otherTexture", otherTexture);
  customEffectWrapper.effect.setFloat3("colorOffset", Math.cos(time) * 0.5 + 0.5, 0, Math.sin(time) * 0.5 + 0.5);

  // Render. Please note we are using render instead of filter to improve
  // performances of real time filter. filter is creating a promise and will therefore
  // generate some lags and garbage.
  customFilter.render(mainTexture, customEffectWrapper);
});

This requires using the render function instead of the filter function to improve performance.

Full Code Sample

You can find the entirety of the code sample above on Github if you want to see it in action and better understand how some of the features can be used.

Live Demo

Please have a look at the Live Image Filter Demo to better appreciate how it works.