## Frame Graph Introduction

### Naming

First, a word about the names.

We used “Frame Graph” as the base name for our main classes (in the `src/FrameGraph/` directory), because it is the name that their inventor used in the first place (see [FrameGraph: Extensible Rendering Architecture in Frostbite](https://www.slideshare.net/DICEStudio/framegraph-extensible-rendering-architecture-in-frostbite)), and also because we don't only have rendering tasks in a frame graph (for example, we also have a culling task).

However, we have used the name “Render Graph” for the node framework (under the directory `src/FrameGraph/Node/`), built on top of the frame graph framework. This makes it easy to differentiate the two frameworks (the frame graph framework is autonomous; you don't have to use the node render graph framework if you don't want to), and it also gives us a nice acronym (NRG - Node Render Graph) :).

### Description

A frame graph is a DAG (Directed Acyclic Graph - a bit like NME / NGE) in which each node (task) represents a render pass, or more generally a task performed when rendering a frame (we have tasks that are not render passes, such as the culling task).

Each task declares its input and output resources.

The resources can be textures, lists of renderable meshes, cameras, or lights. As for textures, they are allocated and managed by the frame graph subsystem, not directly by the tasks. This allows us to optimize texture allocation and reuse textures during graph execution, thus saving GPU memory.

By default, resources do not persist between executions of a render graph, unless a resource is specifically labeled as “persistent” (think of a texture that must be reused from one frame to the next). In our implementation, persistent textures are used to implement “ping-pong” textures, where we change the read and write textures on each frame (used to implement the temporal antialiasing task, for example).

To clarify the ideas, here is a simple graph:

![Basic graph](/img/frameGraph/basic_graph.webp)

The “Color Texture”, “Depth Texture”, “Camera” and “Object List” nodes are input resources (respectively, of the texture, depth texture, camera and object list type). “Clear” and “Main Rendering” are two tasks, the first clears a texture/depth texture and the second renders objects in a texture. “Output” is the output buffer (think of it as the screen output).

As a user, the process of creating and using a frame graph is as follows:
* Create a frame graph, either by using the `FrameGraphXXX` classes (see [Frame Graph Framework Description](/features/featuresDeepDive/frameGraph/frameGraphClassFramework)), or by loading a node render graph (see [Node Render Graph Blocks](/features/featuresDeepDive/frameGraph/frameGraphBlocks)).
* Build the frame graph (`await FrameGraph.buildAsync()` or `await NodeRenderGraph.buildAsync()`). Calling these functions will also wait until the graph is ready to be displayed (it waits until all tasks + all internal states are ready).
 
At this point, the frame graph can be safely executed: call `FrameGraph.execute()`, or simply set `scene.frameGraph = myFrameGraph`, in which case the scene's rendering loop will call `execute`.

Note that in this scenario, you never have to manage passes: you will only need to create passes when you create your own tasks. As a user, you simply use the existing tasks (see [Frame Graph Task List](/features/featuresDeepDive/frameGraph/frameGraphClassFramework/frameGraphTaskList) for the list of existing tasks in the framework), creating an instance of the task, setting its input parameters to reasonable values, and adding the task to the frame graph. See [Introduction to Frame Graph classes](/features/featuresDeepDive/frameGraph/frameGraphClassFramework/frameGraphClassOverview) for more information.

### Benefits

A frame graph provides high-level knowledge of the whole frame, which:
* simplifies and optimizes resource management
* simplifies the configuration of the rendering pipeline => Visual Editor!
* simplifies asynchronous computation and resource barriers. This is an advantage for native implementations, but for the web, we don't (yet?) have asynchronous computation and we don't have to manage resource barriers ourselves.
* allows for autonomous and efficient rendering modules
* overcomes some of the limitations of our hard-coded pipeline.
 
The last advantage is particularly important, as it enables things that are not possible in our current fixed pipeline, which can be described as follows (this is what the existing `Scene.render` method does):
1. Animate
1. Update cameras
1. Render RTTs declared at the scene level
1. For each camera
 1. Evaluate and dispatch active meshes for rendering / Animate and dispatch particle systems for rendering
 1. Software skinning
 1. Rendering of RTT declared at camera level
 1. Rendering of active meshes
 1. Apply post-processing to the camera
 
We have a number of observables that allow you to inject code between these stages, as well as internal components that add functionality at key points (such as shadows, layers, effect layers, etc.), but the order of tasks is always fixed and strongly centered on the camera, as you can see.

With a frame graph, nothing is defined in advance; you simply create tasks and their interconnections. The camera has no particular status; it is a resource that you can use to construct the graph, in the same way that you can use textures or lists of objects.

Automatic texture management optimizes texture allocation and can lead to substantial GPU memory savings, depending on your graph:

![Example of GPU memory saving](/img/frameGraph/gpu_memory_saving.webp) 

In this simple graph, reusing textures saves 25% of the GPU's memory, but we have already seen savings of 40% or more in other graphs.

## Frame Graph Usage

There are two different ways to use a frame graph in a scene.

### Use a frame graph instead of the scene render loop

This is the best way to use frame graphs, but because the implementation is not yet fully complete (in particular, a number of rendering tasks are not yet available for frame graphs), it is also possible to use one (or more) frame graph(s) at the same time as the existing scene render loop, which is the subject of the next section.

In this mode, the execution of a frame graph largely replaces the flow of operations normally performed by `Scene.render`, as described above. To activate this mode, simply define a frame graph instance in `Scene.frameGraph`. In this mode:
* There is no longer an active camera! `Scene.activeCamera` can be defined by a frame graph task if it needs it to run, but it is reset to `null` once the task is finished. This is logical, because the camera is no longer the central resource that it was when frame graphs were not used. Therefore, you should not rely on `Scene.activeCamera` always having a non-null value in your code.
* You must define `Scene.cameraToUseForPointers` for the camera to use for pointer operations. By default, if nothing is defined for this property, `Scene.activeCamera` is used. But since this property is now `null` most of the time, pointer operations will not work as expected if you do not define `Scene.cameraToUseForPointers`.
* You will not be able to define the parameters of a certain number of components via the inspector (such as rendering pipelines, effect layers, post-processing) because these are now simple tasks within a frame graph. As a workaround, if you use a node render graph to generate the frame graph, you will be able to set parameters in the node render graph editor. We will also look at how to update the inspector (when possible), so that we can adjust the settings from this tool.
* Most of the existing observables notified by `Scene.render` are no longer notified. As explained above, the execution of the frame graph replaces much of `Scene.render` (see below), so a lot of code is no longer executed.
 
Regarding the last point, this is what `Scene.render` does when `Scene.frameGraph` is defined:
1. Notifies `Scene.onBeforeAnimationsObservable`
1. Calls `Scene.animate`
1. Notifies `Scene.onBeforeRenderObservable`
1. Updates the cameras
1. Notifies `Scene.onBeforeRenderTargetsRenderObservable`
1. Render RTTs declared at the scene level
1. Notifies `Scene.onAfterRenderTargetsRenderObservable`
1. Updates the world's mesh matrices
1. Animates the particle systems
1. Executes the frame graph
1. Notifies `Scene.onAfterRenderObservable`
 
As you can see, only five observables are notified in this case.

Note that the "Render RTTs declared at the scene level" step has been integrated into frame graphs to simplify porting legacy code to frame graphs.

### Use a frame graph in addition to the existing scene render loop

In some cases, you may want to keep the existing scene rendering loop and use frame graphs as a complement. This may be because some features are still missing in frame graphs (so using a frame graph instead of the render loop is not possible), or because you want to keep your existing code and only need the additional features provided by frame graphs.

In this mode, you do not assign a value to `Scene.frameGraph` and you simply execute a frame graph yourself by calling its `execute()` method. The texture(s) updated by the execution of the frame graph can be retrieved and reused in your own code, or the output of the execution can be directly generated on the screen.

This is the simplest mode to use: using one (or more!) frame graphs is just another feature you can exploit in your own code.

## Examples of frame graphs

By way of illustration, below are two examples of the two cases outlined above.


# Node Render Graph Blocks

The Node Render Graph system is based on a set of blocks representing different tasks and resources. Each block has its own set of inputs and outputs, allowing users to link them together to create complex interactions.

The preferred way to use the node framework is to use the [Node Render Graph Editor](https://nrge.babylonjs.com/) (NRGE), but you can also use it programmatically by creating instances of the different `NodeRenderGraphXXX` classes and connecting them with connection points.


See the [Basic Concepts and Getting Started](/features/featuresDeepDive/frameGraph/frameGraphBasicConcepts) section for general information on nomenclature and an overview of the frame graph concept.

This section describes the frame graph class framework:
* [Overview of the frame graph framework](/features/featuresDeepDive/frameGraph/frameGraphClassFramework/frameGraphClassOverview) describes the main classes that are part of the framework.
* [Writing custom frame graph tasks](/features/featuresDeepDive/frameGraph/frameGraphClassFramework/frameGraphWritingTasks) explains how to implement your own tasks, including composite tasks that use other tasks internally.
* [List of task classes in the frame graph framework](/features/featuresDeepDive/frameGraph/frameGraphClassFramework/frameGraphTaskList) describes all the tasks that can be used with a frame graph.
 
Note that the implementation of the framework is inspired by Unity's [Render Graph System](https://docs.unity3d.com/6000.2/Documentation/Manual/urp/render-graph-introduction.html).


Here you will find several examples of how frame graphs can be used in your own projects or as inspiration when creating your own frame graphs.


## Best Practices

### Set scene.frameGraph before calling FrameGraph.buildAsync

If your frame graph is to be used at the scene level, you must set it before calling `FrameGraph.buildAsync()`, because `FrameGraph.whenReadyAsync()` (which, by default, is called as part of the execution of `buildAsync`) may require that **scene.frameGraph** be set for correct processing.

### Set isMainObjectRenderer on the appropriate object renderer

You should always have one and only one object renderer with its **isMainObjectRenderer** property set to *true*.

It will help to locate the main object renderer in the frame graph when multiple object renderers are used.

This is useful for the inspector to know which object renderer to use for additional rendering features like wireframe rendering or frustum light debugging.

It is also used to determine the main camera used by the frame graph: this is the camera used by the main object renderer.

### Set the texture sampling mode in the correct location

When writing your own post-processing task that extends `FrameGraphPostProcessTask`, you may need to set the sampling mode of a texture: don't do it via the `additionalBindings` parameter of the `FrameGraphPostProcessTask.record(skipCreationOfDisabledPasses, additionalExecute, additionalBindings)` method, but via the `additionalExecute` parameter instead!

It is tempting to do something like:

```typescript
const pass = super.record(
 skipCreationOfDisabledPasses,
 undefined,
 (context) => {
 context.setTextureSamplingMode(this.circleOfConfusionTexture, this.circleOfConfusionSamplingMode);
 context.bindTextureHandle(this._postProcessDrawWrapper.effect!, "circleOfConfusionSampler", this.circleOfConfusionTexture);
 }
);
```

but the correct way to do it is as follows:

```typescript
const pass = super.record(
 skipCreationOfDisabledPasses,
 (context) => {
 context.setTextureSamplingMode(this.circleOfConfusionTexture, this.circleOfConfusionSamplingMode);
 },
 (context) => {
 context.bindTextureHandle(this._postProcessDrawWrapper.effect!, "circleOfConfusionSampler", this.circleOfConfusionTexture);
 }
);
```

This is because `additionalBindings` is executed too late in the process, and setting the texture sampling mode at that point messes with the internal state of the engine (at least in WebGL).

### Ensure that your FrameGraphTask.record() implementation handles multiple calls

The `FrameGraphTask.record()` method can be called multiple times during the lifetime of your frame graph (if the graph is being rebuilt, for example due to a window resize). You must ensure that the objects created by your implementation of `record()` are properly deleted before being recreated by a new call (such as observers, for example).

### Avoid a texture copy if you are using at least one post-process in your graph.

If you use at least one post-process in your graph, you can avoid copying a texture by connecting the back buffer color texture to the **target** input of your last post-process.

Do:
![No texture copy](/img/frameGraph/copytexture_optim.webp)

Don't:
![Texture copy](/img/frameGraph/copytexture_nooptim.webp)

In fact, the `Output` block (or `FrameGraphCopyToBackbufferColorTask` if you create a graph manually) is optimized so that it does not copy the texture connected to its **texture** input if that texture is already the back buffer color texture.

Furthermore, the final post-processing does not need to allocate a new texture, as it will render directly into the color texture of the back buffer.

### Use NodeRenderGraph.onBeforeBuildObservable to update the graph before building it

Often, you will need to update one or more blocks of a node render graph before building it. The best place to do this is inside an `onBeforeBuildObservable` observer, as this observable is notified just before the graph is built.

This can help you better structure your code, instead of having all the code at the global level.

For example (taken from [this PG](https://playground.babylonjs.com/#1QCA2M#35)):

```typescript
nrg.onBeforeBuildObservable.add(() => {
 const customRenderingTask = nrg.getBlockByName("CustomRendering").task;

 customRenderingTask.objectRenderer.setMaterialForRendering(ground, groundMatForCustomRendering);

 const cameraBlock = nrg.getBlockByName("Camera");
 cameraBlock.value = camera;

 const camera2Block = nrg.getBlockByName("Camera2");
 camera2Block.value = camera2;

 const objectListBlock = nrg.getBlockByName("ObjectList");
 objectListBlock.value = { meshes: scene.meshes, particleSystems: [] };

 const objectList2Block = nrg.getBlockByName("ObjectList2");
 objectList2Block.value = { meshes: scene.meshes.slice(), particleSystems: scene.particleSystems };
});
```

## FAQ

### Why am I getting the error message "Trying to bind a null texture/sampler"?

You will get this error (in the browser console log) with WebGPU if you render to the default back-buffer color texture and use at least one post-process.

This is because post-processing reads from the texture connected to its **source** input, which is impossible if the texture is the back buffer color texture (at least in WebGL, so also prohibited in WebGPU).

Reproduction: <NRGE id="#HVWCTE#1" title="Using the backbuffer color texture with a post-process" description="Error generated when using the backbuffer color texture with a post-process" isMain={false} category="NodeRenderGraph"/>

Note that in WebGL, you will not get any error messages, but the preview window will remain black.

### Why am I getting the error message "Uncaught (in promise) Error: FrameGraph._computeTextureLifespan: Texture handle "XX" not found in the texture manager."?

You probably forgot to add a task to the frame graph!

When you create a task programmatically (i.e., without using a node render graph), don't forget to add it to the frame graph:

```typescript
const renderTask = new BABYLON.FrameGraphObjectRendererTask("renderObjects", frameGraph, scene);

renderTask.targetTexture = clearTask.outputTexture;
renderTask.depthTexture = clearTask.outputDepthTexture;
renderTask.objectList = rlist;
renderTask.camera = camera;

frameGraph.addTask(renderTask); // <-- Don't forget!
```

### Why isn't my skybox showing up?

You must set **camera.ignoreCameraMaxZ** to *true* for skyboxes to work as expected with frame graphs. See the comment in [FrameGraph: adding a new task + several fixes](https://github.com/BabylonJS/Babylon.js/pull/17381) for more details.

Reproduction: <Playground id="#HBLXYO" title="Skybox not showing up" description="Frame graph with a skybox mesh not showing up" isMain={false}/>

Replace *false* with *true* on line 15 to fix the issue.

Note that the problem does not stem from the skybox itself, but from the **Mesh.ignoreCameraMaxZ** property when set to *true* (which is the case for the skybox).

### How to visualize textures generated by my frame graph?

For debugging purposes, it may be useful to be able to see all the textures generated by your frame graph in the inspector.

To enable this feature, pass *true* as the second parameter of the `FrameGraph` constructor:

```typescript
const frameGraph = new BABYLON.FrameGraph(scene, true);
```

With a node render graph, set the **debugTextures** option to *true* when loading the snippet:

```typescript
const nrg = await BABYLON.NodeRenderGraph.ParseFromSnippetAsync("PSA9PS#150", scene, { debugTextures: true });

or

const nrg = BABYLON.NodeRenderGraph.Parse(frameGraphJSONData, scene, { debugTextures: true });
await nrg.whenReadyAsync();
```

Note that due to texture reuse, it is possible that a texture may be used by multiple tasks!

For this reason, when debugging a frame graph, you can disable texture reuse by setting `frameGraph.optimizeTextureAllocation = false`.

### Why isn't the environment helper working?

In order for the environment helper to work correctly with frame graphs (such as the mirror effect, for example), you must define the frame graph in the scene **before** creating the environment helper:
```typescript
...
scene.frameGraph = <your_frame_graph>;
...
const helper = scene.createDefaultEnvironment(options);
```

Example: <Playground id="#MJNICE#1759" title="Environment helper with frame graphs" description="Making the environment helper work with frame graphs" isMain={false}/>

### Why don't MirrorTexture, RefractionTexture or ReflectionProbe work?

In order for `MirrorTexture`, `RefractionTexture` and `ProbeTexture` to work as expected with frame graphs, you must add them to the **scene.customRenderTargets** array after creating them. If the texture is to be rendered with a specific camera, set **activeCamera** correctly.

`MirrorTexture` example: <Playground id="#LVTTQX#359" title="Mirror texture with frame graphs" description="Making the MirrorTexture work with frame graphs" isMain={false}/>

`RefractionTexture` example: <Playground id="#22KZUW#752" title="Refraction texture with frame graphs" description="Making the RefractionTexture work with frame graphs" isMain={false}/>

`ProbeTexture` example: <Playground id="#KA93U#1722" title="Probe texture with frame graphs" description="Making the ProbeTexture work with frame graphs" isMain={false}/>


## The Babylon.js Frame Graph System

The [Frame Graph](/features/featuresDeepDive/frameGraph/frameGraphBasicConcepts) system is a powerful tool for creating and configuring the different rendering passes used over the course of a frame.

It is a modular system that lets you create autonomous and efficient rendering modules, improving extensibility through decoupled and composable modules.

Frame graphs simplify and optimize resource management: by reusing textures, you can potentially save tons of GPU memory, depending on the complexity of your graph!

Frame graphs also allow you to do things that were not previously possible with our fixed rendering loop.

Thanks to the new visual editor, [Node Render Graph Editor](https://nrge.babylonjs.com/) (NRGE), anyone can easily create the layout of a frame visually, without any programming!


Frame Graph

The Babylon.js Frame Graph System

Coming next

Frame Graph Basic Concepts and Getting Started

Learn all about the Babylon.js Frame Graph system.

Frame Graph Framework Description

Learn all about the Babylon.js Frame Graph system.

Node Render Graph Blocks

A detailed overview of the Node Render Graph blocks.

Frame Graph Examples

Real-world use cases for frame graphs.

Frame Graph Frequently Asked Questions and Best Practices

Frequently asked questions and best practices about frame graphs