Timeline Control

The Babylon.js Timeline Control is a web control built on top of Babylon.js that accelerates some of the typical features associated with video timelines.

Introduction

One of the inherent issues encountered while scrolling through a video timeline built from IMG tags is performance. The more pictures there are to scroll through, the slower and less smooth the interactions with the web page become.

To make the web page more responsive, we rely on WebGL through Babylon.js to create an easy-to-reuse control that displays a video timeline.

Timeline

How to use

Installation

To begin with the timeline, you first need to install the controls npm package.

npm install @babylonjs/controls

To reduce the size of your web page, the controls library is based on the ES6 version of @babylonjs/core used as a peer dependency. Therefore, if you are not relying on it so far in your project, you also need to install core:

npm install @babylonjs/core

Instantiation

Once done, you can now import the timeline control into your code:

import { Timeline } from "@babylonjs/controls/timeline";

And simply instantiate it in your page:

const timeline = new Timeline(timelineCanvas, {
        totalDuration: 60,
        thumbnailWidth: 128,
        thumbnailHeight: 120,
        loadingTextureURI: "./assets/loading.png",
        getThumbnailCallback: (time: number, done: (input: any) => void) => {
            // This is strictly for demo purpose and should not be used in prod as it creates as many videos
            // as there are thumbnails all over the timeline.
            const hiddenVideo = document.createElement("video");
            document.body.append(hiddenVideo);
            hiddenVideo.style.display = "none";

            hiddenVideo.setAttribute("playsinline", "");
            hiddenVideo.muted = true;
            hiddenVideo.autoplay = navigator.userAgent.indexOf("Edge") > 0 ? false : true;
            hiddenVideo.loop = false;

            hiddenVideo.onloadeddata = () => {
                if (time === 0) {
                    done(hiddenVideo);
                }
                else {
                    hiddenVideo.onseeked = () => {
                        done(hiddenVideo);
                    }
                    hiddenVideo.currentTime = time;
                }
            }

            hiddenVideo.src = "./assets/test.mp4?" + time;
            hiddenVideo.load();
        }
    });

The main element to provide is a canvas on which we can use a WebGL context to render the timeline. You can also provide another Babylon.js control to share the WebGL context. For instance, you could share the context with a Resizer to use a resized image directly as a thumbnail without an extra copy.

As we do not want to be opinionated about the UX you prefer or about thumbnail generation itself, it is all left to your implementation. The only information you need to provide is:

totalDuration: the full duration of the video to cover (used to compute when all the thumbnails should be generated).
thumbnailWidth: defines the width of your generated thumbnails (used during rendering to know how much space they should take in the timeline)
thumbnailHeight: defines the height of your generated thumbnails (used to keep the ratio intact with the width previously defined)
loadingTextureURI: defines the URL of an image used as a temporary replacement for a thumbnail that is not fully loaded yet (this helps handle network latency gracefully)
getThumbnailCallback: a callback through which you provide the thumbnail that matches the requested time parameter. Through the done function, you can return either a video element set to the right frame (as in the previous code sample), a canvas element containing, for instance, some preprocessed data, or an image URL. More choices will probably be added here, for instance to support texture atlases, as they might be a useful transport format.

You can also provide some non-mandatory options:

useClosestThumbnailAsLoadingTexture: it is set to true by default, and if enabled, it will try to use the closest loaded thumbnail (in time) instead of the loading one as soon as some are ready.

Render Loop

Finally, you can start rendering the thumbnails:

timeline.runRenderLoop();

It is also possible to pass a custom callback to this method to ensure that some custom code would run every frame before rendering the timeline:

timeline.runRenderLoop(() => {
        // Sync the timeline with a playing video
        if (!mainVideo.paused) {
            timeline.setCurrentTime(mainVideo.currentTime);
        }
    });

Changing the current zoom level

By default, the timeline is not zoomed at all, it shows as many thumbnails as fit in the canvas distributed evenly along the entire video.

You can zoom until you see one thumbnail per chosen time unit of the video (so if the total duration is 60 and you are zoomed at 100%, you will be able to see 60 thumbnails in total when scrolling from left to right).

To zoom in/out, you can call the following code:

// value is the percentage of zoom desired 0 for the entire video 100 for one thumbnail per unit
    timeline.setVisibleDurationZoom(value);

Changing the current time

As changing the zoom level does not change the number of visible thumbnails, you need a way to control where you currently are in the list of thumbnails. The following code can be used to do so:

// Sets the left side of the canvas to the current chosen time
    // The rest of the visible thumbnails are evenly distributed according to the chosen
    // zoom level.
    timeline.setCurrentTime(value);

The current time will be automatically bound between 0 and the max available position.

Cache Warm Up

In case you have some of the thumbnails ready before waiting for the callback, you can submit them into the cache like this:

timeline.addThumbnail(textureData, time);

The textureData types are the same as those allowed to be returned in the callback: a texture, a video, a canvas, or an image URL.

This can help if you generate the thumbnails client-side to warm up the cache and thus reduce thumbnail load time.

Full Code Sample

You can find the entirety of the code sample above on Github if you want to see it in action and better understand how some of the features can be used.

Live Demo

Please have a look at the Live Timeline Demo to better appreciate how it works.