
# Image Filter Control

The Babylon.js Filter Control is a web control built on top of Babylon.js to apply filters to pictures on web pages at blazing-fast speed.

## Introduction

Filtering images to apply effects can be tedious and slow on the CPU. The best place to do this, for a wide variety of effects, is on the GPU. But setting up an entire WebGL pipeline just to process images can be tricky, especially if you want to benefit from WebGL 2-to-1 fallback and work around well-known platform issues.

To greatly simplify this task we introduced the `ImageFilter` Control.

![ImageFilter](/img/features/controls/imageFilter.webp)

## How to use

### Installation

To get started with the image filter control, you first need to install the controls npm package.

```javascript
npm install @babylonjs/controls
```

To reduce the size of your web page, the controls library is based on the ES6 version of `@babylonjs/core` used as a peer dependency. Therefore, if you are not relying on it so far in your project, you also need to install core:

```javascript
npm install @babylonjs/core
```

### Instantiation

Once done, you can now import the control in your code:

```javascript
import { ImageFilter } from "@babylonjs/controls/imageFilter";
```

And simply instantiate it on your page:

```javascript
const imageFilter = new ImageFilter(filterCanvas);
```

You simply need to provide a canvas on which we can use a WebGL context. You can also provide another Babylon.js control to share the WebGL context.

Also, if you need more specific engine configurations or want to use post processes, you should manually use your own engine:

```javascript
const engine = new Engine(filterCanvas);
const imageFilter = new ImageFilter(engine);
```

By default, the control relies on ThinEngine to optimize your bundle, but it might have limitations you do not want in your experience.

## Using Post Process

In order to apply an existing post process as the image filter, you can use the following code:

```javascript
const blackAndWhitePostProcess = new BlackAndWhitePostProcess("bw", 1, null, undefined, engine);
backAndWhiteFilter.filter(imageToProcess, blackAndWhitePostProcess);
```

Where imageToProcess could be either:

- the url of a picture.
- a video element (the current visible frame of the video will be used)
- another canvas element (the current visible state will be used)

This will apply the post process to the provided input and display it in the canvas.

## Using Custom Shader

In order to apply a custom shader as the image filter, you can use the following code:

```javascript
const customEffectWrapper = new EffectWrapper({
  name: "Custom",
  engine: customFilter.engine,
  fragmentShader: `
        // Samplers
        varying vec2 vUV;
        uniform sampler2D textureSampler;
        
        void main(void) 
        {
            gl_FragColor = texture2D(textureSampler, vUV);

            // Swizzle channels
            float r = gl_FragColor.r;
            gl_FragColor.r = gl_FragColor.b;
            gl_FragColor.b = r;
        }
    `,
  samplerNames: ["textureSampler"],
});
customFilter.filter(imageToProcess, customEffectWrapper);
```

Where imageToProcess could be either:

- the url of a picture.
- a video element (the current visible frame of the video will be used)
- another canvas element (the current visible state will be used)

This will apply the custom shader passed as a parameter to the picture. By default, `vUV` is available as a varying that defines the full output as texture coordinates. `textureSampler` needs to be present and defines the texture corresponding to the input parameter.

Please note that if you need to add custom uniforms or samplers, they should be defined in the effect wrapper lists:

```javascript
const customEffectWrapper = new EffectWrapper({
  name: "Custom",
  engine: customFilter.engine,
  fragmentShader: `
        varying vec2 vUV;
        
        // Default Sampler
        uniform sampler2D textureSampler;

        // Custom uniforms
        uniform sampler2D otherTexture;
        uniform vec3 colorOffset;

        const vec2 scale = vec2(0.25, 1.);

        void main(void) 
        {
            gl_FragColor = texture2D(textureSampler, vUV);

            // Swizzle channels
            float r = gl_FragColor.r;
            gl_FragColor.r = gl_FragColor.b;
            gl_FragColor.b = r;
            gl_FragColor.rgb += clamp(colorOffset, 0., 1.);

            gl_FragColor.rgb *= texture2D(otherTexture, vUV * scale).rgb;
        }
    `,
  // Defines the list of existing samplers (default + customs).
  samplerNames: ["textureSampler", "otherTexture"],
  // Defines the list of existing uniform to be bound.
  uniformNames: ["colorOffset"],
});
```

Then you can simply bind them either in `onApply`:

```javascript
customEffectWrapper.onApplyObservable.add(() => {
  // Sets the custom values.
  customEffectWrapper.effect.setTexture("otherTexture", otherTexture);
  customEffectWrapper.effect.setFloat3("colorOffset", 0.2, 0, 0.2);
});
```

Or during the render loop:

```javascript
// Rely on the underlying engine render loop to update the filter result every frame.
engine.runRenderLoop(() => {
  // Only render if the custom texture is ready (the default one is
  // checked for you by the render function)
  if (!otherTexture.isReady()) {
    return;
  }

  // Sets the custom values.
  time += engine.getDeltaTime() / 1000;
  customEffectWrapper.effect.setTexture("otherTexture", otherTexture);
  customEffectWrapper.effect.setFloat3("colorOffset", Math.cos(time) * 0.5 + 0.5, 0, Math.sin(time) * 0.5 + 0.5);

  // Render. Please note we are using render instead of filter to improve
  // performances of real time filter. filter is creating a promise and will therefore
  // generate some lags and garbage.
  customFilter.render(mainTexture, customEffectWrapper);
});
```

Finally, if you are relying on new textures, you need to wait for them to be ready before rendering.

## Process to the Canvas

This is by far the simplest option if you have a canvas on your page. You simply need to use the following code to fit the provided element to the canvas size:

```javascript
imageFilter.filter(imageToProcess, filter);
```

On the previous line, imageToProcess could be either:

- the url of a picture.
- a video element (the current visible frame of the video will be used)
- another canvas element (the current visible state will be used)

This is the default behavior.

## Process to a Babylon Texture

Instead of filtering directly to a canvas, you may prefer to create only a Babylon.js texture on the GPU. For this, you can use the following function:

```javascript
const texture = imageFilter.getFilteredTexture(imageToResize, { width: 128, height: 100 }, filter);
```

Like before, imageToResize could be either:

- the url of a picture.
- a video element (the current visible frame of the video will be used).
- another canvas element (the current visible state will be used).

You also need to provide the size you want your texture to have on the GPU.

Now you are free to use this texture with any other controls.

## Real-Time Filtering

Instead of filtering only once, you might want to create dynamic real-time effects. For this, you can simply render the effect during the render loop:

```javascript
// Rely on the underlying engine render loop to update the filter result every frame.
engine.runRenderLoop(() => {
  // Only render if the custom texture is ready (the default one is
  // checked for you by the render function)
  if (!otherTexture.isReady()) {
    return;
  }

  // Sets the custom values.
  time += engine.getDeltaTime() / 1000;
  customEffectWrapper.effect.setTexture("otherTexture", otherTexture);
  customEffectWrapper.effect.setFloat3("colorOffset", Math.cos(time) * 0.5 + 0.5, 0, Math.sin(time) * 0.5 + 0.5);

  // Render. Please note we are using render instead of filter to improve
  // performances of real time filter. filter is creating a promise and will therefore
  // generate some lags and garbage.
  customFilter.render(mainTexture, customEffectWrapper);
});
```

This requires using the render function instead of the filter function to improve performance.

## Full Code Sample

You can find the entirety of the code sample above on [Github](https://github.com/BabylonJS/Controls/blob/master/www/imageFilter/index.ts) if you want to see it in action and better understand how some of the features can be used.

## Live Demo

Please have a look at the [Live Image Filter Demo](https://controls.babylonjs.com/imageFilter) to better appreciate how it works.



# Resizer Control

The Babylon.js Resizer Control is a web control built on top of Babylon.js to accelerate the resizing of pictures on web pages at blazing-fast speed.

## Introduction

It is common on the web to need to resize a picture on demand. It is easy enough to do with static pictures and CSS, but what about capturing a frame from a video or a snapshot from a canvas?

This component can also be paired with the [Timeline](./timeline) and helps resize thumbnails before uploading them to the GPU, without relying on a separate canvas element and thus avoiding the extra copies required.

![Resizer](/img/features/controls/resizer.webp)

## How to use

### Installation

To get started with the resizer, you first need to install the controls npm package.

```javascript
npm install @babylonjs/controls
```

To reduce the size of your web page, the controls library is based on the ES6 version of `@babylonjs/core` used as a peer dependency. Therefore, if you are not relying on it so far in your project, you also need to install core:

```javascript
npm install @babylonjs/core
```

Please note that, while the controls are still in preview, it might be preferable to use their latest update with:

```javascript
npm install @babylonjs/controls@preview
npm install @babylonjs/core@preview
```

### Instantiation

Once done, you can now import the resizer control in your code:

```javascript
import { Resizer } from "@babylonjs/controls/resizer";
```

And simply instantiate it on your page:

```javascript
const resizer = new Resizer(resizerCanvas);
```

You simply need to provide a canvas on which we can use a WebGL context. You can also provide another Babylon.js control to share the WebGL context.

## Resizing to the canvas

This is by far the simplest option if you have a canvas on your page. You simply need to use the following code to fit the provided element to the canvas size:

```javascript
resizer.resize(imageToResize);
```

On the previous line, imageToResize could be either:

- the url of a picture.
- a video element (the current visible frame of the video will be used)
- another canvas element (the current visible state will be used)

The source image will be stretched in both directions to fit the target canvas.

## Resizing to a Babylon Texture

Instead of resizing directly to a canvas, you could prefer to only create a Babylon.js texture on the GPU. For this, you can use the following function:

```javascript
const texture = resizer.getResizedTexture(imageToResize, { width: 128, height: 100 });
```

Like before, imageToResize could be either:

- the url of a picture.
- a video element (the current visible frame of the video will be used).
- another canvas element (the current visible state will be used).

You also need to provide the size you want your texture to have on the GPU.

Now you are free to use this texture with any other controls.

## How to Use with the Timeline

Creating the texture is one thing, but being able to use it is even better.

For instance, you can simply pair the resizer with the timeline:

```javascript
const resizer = new Resizer(canvas);
const timeline = new Timeline(resizer, {
    [...]
    getThumbnailCallback: (time: number) => {
        hiddenVideo.currentTime = time;
        return resizer.getResizedTexture(hiddenVideo, { width: 128, height: 100 });
    }
});
```

The two important points are:

- Use a shared control context by passing one control to the constructor of the other one.
- Returning a resized texture from the thumbnail callback.

That's it: both controls are linked together, making efficient use of the GPU without wasting resources on thumbnails that are larger than needed.

This is actually the technique we are relying on in the demo. The source can be seen on [Github](https://github.com/BabylonJS/Controls/blob/master/www/timeline/index.ts)

## Full Code Sample

You can find the entirety of the code sample above on [Github](https://github.com/BabylonJS/Controls/blob/master/www/resizer/index.ts) if you want to see it in action and better understand how some of the features can be used.

## Live Demo

Please have a look at the [Live Resizer Demo](https://controls.babylonjs.com/resizer) to better appreciate how it works.



# Video Timeline Control

The Babylon.js Timeline Control is a web control built on top of Babylon.js that accelerates some of the typical features associated with video timelines.

## Introduction

One of the inherent issues encountered while scrolling through a video timeline built from IMG tags is performance. The more pictures there are to scroll through, the slower and less smooth the interactions with the web page become.

To make the web page more responsive, we rely on WebGL through Babylon.js to create an easy-to-reuse control that displays a video timeline.

![Timeline](/img/features/controls/timeline.webp)

## How to use

### Installation

To begin with the timeline, you first need to install the controls npm package.

```
npm install @babylonjs/controls
```

To reduce the size of your web page, the controls library is based on the ES6 version of `@babylonjs/core` used as a peer dependency. Therefore, if you are not relying on it so far in your project, you also need to install core:

```
npm install @babylonjs/core
```

### Instantiation

Once done, you can now import the timeline control into your code:

```
import { Timeline } from "@babylonjs/controls/timeline";
```

And simply instantiate it in your page:

```
const timeline = new Timeline(timelineCanvas, {
        totalDuration: 60,
        thumbnailWidth: 128,
        thumbnailHeight: 120,
        loadingTextureURI: "./assets/loading.png",
        getThumbnailCallback: (time: number, done: (input: any) => void) => {
            // This is strictly for demo purpose and should not be used in prod as it creates as many videos
            // as there are thumbnails all over the timeline.
            const hiddenVideo = document.createElement("video");
            document.body.append(hiddenVideo);
            hiddenVideo.style.display = "none";

            hiddenVideo.setAttribute("playsinline", "");
            hiddenVideo.muted = true;
            hiddenVideo.autoplay = navigator.userAgent.indexOf("Edge") > 0 ? false : true;
            hiddenVideo.loop = false;

            hiddenVideo.onloadeddata = () => {
                if (time === 0) {
                    done(hiddenVideo);
                }
                else {
                    hiddenVideo.onseeked = () => {
                        done(hiddenVideo);
                    }
                    hiddenVideo.currentTime = time;
                }
            }

            hiddenVideo.src = "./assets/test.mp4?" + time;
            hiddenVideo.load();
        }
    });
```

The main element to provide is a canvas on which we can use a WebGL context to render the timeline. You can also provide another Babylon.js control to share the WebGL context. For instance, you could share the context with a Resizer to use a resized image directly as a thumbnail without an extra copy.

As we do not want to be opinionated about the UX you prefer or about thumbnail generation itself, it is all left to your implementation. The only information you need to provide is:

- _totalDuration_: the full duration of the video to cover (used to compute when all the thumbnails should be generated).
- _thumbnailWidth_: defines the width of your generated thumbnails (used during rendering to know how much space they should take in the timeline)
- _thumbnailHeight_: defines the height of your generated thumbnails (used to keep the ratio intact with the width previously defined)
- _loadingTextureURI_: defines the URL of an image used as a temporary replacement for a thumbnail that is not fully loaded yet (this helps handle network latency gracefully)
- _getThumbnailCallback_: a callback through which you provide the thumbnail that matches the requested time parameter. Through the done function, you can return either a video element set to the right frame (as in the previous code sample), a canvas element containing, for instance, some preprocessed data, or an image URL. More choices will probably be added here, for instance to support texture atlases, as they might be a useful transport format.

You can also provide some non-mandatory options:

- _useClosestThumbnailAsLoadingTexture_: it is set to true by default, and if enabled, it will try to use the closest loaded thumbnail (in time) instead of the loading one as soon as some are ready.

### Render Loop

Finally, you can start rendering the thumbnails:

```
    timeline.runRenderLoop();
```

It is also possible to pass a custom callback to this method to ensure that some custom code would run every frame before rendering the timeline:

```
    timeline.runRenderLoop(() => {
        // Sync the timeline with a playing video
        if (!mainVideo.paused) {
            timeline.setCurrentTime(mainVideo.currentTime);
        }
    });
```

### Changing the current zoom level

By default, the timeline is not zoomed at all, it shows as many thumbnails as fit in the canvas distributed evenly along the entire video.

You can zoom until you see one thumbnail per chosen time unit of the video (so if the total duration is 60 and you are zoomed at 100%, you will be able to see 60 thumbnails in total when scrolling from left to right).

To zoom in/out, you can call the following code:

```
    // value is the percentage of zoom desired 0 for the entire video 100 for one thumbnail per unit
    timeline.setVisibleDurationZoom(value);
```

### Changing the current time

As changing the zoom level does not change the number of visible thumbnails, you need a way to control where you currently are in the list of thumbnails. The following code can be used to do so:

```
    // Sets the left side of the canvas to the current chosen time
    // The rest of the visible thumbnails are evenly distributed according to the chosen
    // zoom level.
    timeline.setCurrentTime(value);
```

The current time will be automatically bound between 0 and the max available position.

### Cache Warm Up

In case you have some of the thumbnails ready before waiting for the callback, you can submit them into the cache like this:

```
    timeline.addThumbnail(textureData, time);
```

The `textureData` types are the same as those allowed to be returned in the callback: a texture, a video, a canvas, or an image URL.

This can help if you generate the thumbnails client-side to warm up the cache and thus reduce thumbnail load time.

## Full Code Sample

You can find the entirety of the code sample above on [Github](https://github.com/BabylonJS/Controls/blob/master/www/timeline/index.ts) if you want to see it in action and better understand how some of the features can be used.

## Live Demo

Please have a look at the [Live Timeline Demo](https://controls.babylonjs.com/timeline) to better appreciate how it works.



# Controls

Babylon.js Controls are a set of regular web controls supercharged with hardware-accelerated rendering through Babylon.js to provide blazing-fast performance.

Babylon.js provides a unified API on top of **WebGL**, **WebGL2**, and **WebGPU** that controls can leverage to unleash the raw power of your GPU.

![Resizer](/img/features/controls/resizer.webp)


Sample https://github.com/sebavan/BabylonjsVideoProcessingSample



# How to use the WebVR experience helper

## WebVR vs WebXR

The WebVR Experience helper is deprecated and very limited in functionality. **It is strongly recommended that new projects use the [WebXR experience helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers)**. For more information, check out our [introduction to WebXR](/features/featuresDeepDive/webXR/introToWebXR).


Controls

Video Overview

Further reading

Coming next

Resizer Control

The Babylon.js Resizer Control is a web control built on top of Babylon.js to accelerate the resizing of pictures on web pages at blazing-fast speed.

Timeline Control

The Babylon.js Timeline Control is a web control built on top of Babylon.js that accelerates some of the typical features associated with video timelines.

Image Filter Control

The Babylon.js Filter Control is a web control built on top of Babylon.js to apply filters to pictures on web pages at blazing-fast speed.