> Learn about advanced concepts of the Flow Graph system. Source: https://doc.babylonjs.com/features/featuresDeepDive/flowGraph/flowGraphAdvanced # Flow Graph Advanced Usage ## Logging and debugging You can enable a very verbose logger on your context. Each context can log every action it performs. Note that this is very verbose, as each connection value is also logged. To enable logging, set the `enableLogging` variable to true: ```javascript const ctx = graph.createContext(); ctx.enableLogging = true; ``` The logger will then be available on the context: ```javascript ctx.logger !== undefined; ``` The logger adds its logs to an array of actions in the logger object. It also logs to the console by default, but this can be disabled by setting the `logToConsole` variable to false. ```javascript ctx.logger.logToConsole = false; // the array of actions console.log(ctx.logger.log); ``` ## Coordinator sound assets A [FlowGraphCoordinator](https://doc.babylonjs.com/typedoc/classes/babylon.flowgraphcoordinator) can declare named Audio V2 sound assets. Audio blocks can then use the asset name instead of holding a direct sound object. The declaration is serialized with the coordinator, so the same sound reference continues to work after the graph is saved and parsed again. ```javascript const coordinator = new FlowGraphCoordinator({ scene }); await coordinator.addSoundAssetAsync({ name: "coin", url: "/sounds/coin.mp3", }); const graph = coordinator.createGraph(); const context = graph.createContext(); const sceneReady = new FlowGraphSceneReadyEventBlock(); const playSound = new FlowGraphPlaySoundBlock(); graph.addEventBlock(sceneReady); playSound.sound.setValue("coin", context); sceneReady.done.connectTo(playSound.in); coordinator.start(); ``` The `PlaySound`, `StopSound`, `PauseSound`, `SetSoundVolume`, `GetSoundVolume`, `IsSoundPlaying`, and `SoundEnded` blocks accept any of these values for their `sound` input: - An Audio V2 sound object. - The name of an asset declared on the coordinator. - A serialized reference such as `{ name: "coin" }`. Use `getSoundAsset(name)` to get the loaded sound, `soundAssets` to list the declarations, and `removeSoundAsset(name)` to remove a declaration. Removing an asset disposes its loaded sound. Adding another asset with the same name replaces the declaration and disposes the previous sound. Blocks that refer to the name use the replacement, and `SoundEnded` updates its listener even if the asset is loaded or replaced after the graph starts. The asset URL can also be a `data:` URL. This embeds the audio in the serialized coordinator. A normal URL keeps the serialized data smaller, but that URL must still be available when the graph is parsed. By default, `addSoundAssetAsync` uses the most recently created Audio V2 engine. It creates a shared fallback engine when no engine exists. The fallback engine is disposed after the last coordinator using it is disposed. You can pass an audio engine as the second argument when the graph must use a specific engine: ```javascript await coordinator.addSoundAssetAsync({ name: "coin", url: "/sounds/coin.mp3" }, audioEngine); ``` If a sound fails to load, `addSoundAssetAsync` rejects, but the declaration remains available for serialization or a later replacement. When a serialized coordinator is parsed, sound-load failures produce warnings and do not stop the graphs from being parsed. You can also pass `audioEngine` in the options for `ParseCoordinatorAsync`. ## Create your own block When creating your first block, you first need to decide whether it is a data block or an execution block. Most blocks are probably data blocks (i.e., they provide passive data), unless the block needs to react to an event or a signal input. > [!NOTE] > If you want your block to work as part of serialization and deserialization, you will also need to add the block to the block creation factory. This section comes later in this document. ### Data block To create a data block, you just need to extend the `FlowGraphBlock` class and implement the `_updateOutputs` method. The `_updateOutputs` method is called when one of the block's outputs is requested by another block. Let's take a look at a very simple block as an example - the ArrayIndexBlock (removing the imports for readability): ```javascript /** * This simple Util block takes an array as input and selects a single element from it. */ export class FlowGraphArrayIndexBlock extends FlowGraphBlock { /** * Input connection: The array to select from. */ public readonly array: FlowGraphDataConnection; /** * Input connection: The index to select. */ public readonly index: FlowGraphDataConnection; /** * Output connection: The selected element. */ public readonly value: FlowGraphDataConnection; /** * Construct a FlowGraphArrayIndexBlock. * @param config construction parameters */ constructor(public override config: IFlowGraphBlockConfiguration) { super(config); this.array = this.registerDataInput("array", RichTypeAny); this.index = this.registerDataInput("index", RichTypeFlowGraphInteger, new FlowGraphInteger(-1)); this.value = this.registerDataOutput("value", RichTypeAny); } /** * @internal */ public override _updateOutputs(context: FlowGraphContext): void { const array = this.array.getValue(context); const index = getNumericValue(this.index.getValue(context)); if (array && index >= 0 && index < array.length) { this.value.setValue(array[index], context); } } /** * Serializes this block (override if needed) * @param serializationObject the object to serialize to */ public override serialize(serializationObject?: any): void { super.serialize(serializationObject); } public override getClassName(): string { return FlowGraphBlockNames.ArrayIndex; } } // register the class in the type registry of the framework RegisterClass(FlowGraphBlockNames.ArrayIndex, FlowGraphArrayIndexBlock); ``` Let's break down the code: First, we define the inputs and outputs of this block: ```javascript public readonly array: FlowGraphDataConnection; public readonly index: FlowGraphDataConnection; public readonly value: FlowGraphDataConnection; ``` They are being created in the constructor: ```javascript this.array = this.registerDataInput("array", RichTypeAny); // define the index as type integer, and also assign a default value to it. The 3rd variable is optional, and each type has its own default value this.index = this.registerDataInput("index", RichTypeFlowGraphInteger, new FlowGraphInteger(-1)); this.value = this.registerDataOutput("value", RichTypeAny); ``` Notice how each one is defined as either an input or an output, and has a type. The type is important if we want proper default values. For example, the default value of a number is 0, a boolean is true, and so on. This information can be found in `flowGraphRichTypes.ts`. Now we have the data inputs and the data outputs, and we need to write the logic to update them, when requested by another block. This is done in the `_updateOutputs` method: ```javascript public override _updateOutputs(context: FlowGraphContext): void { // get the array from the data input. If this is connected to another block, the other block's logic will be executed to get the array const array = this.array.getValue(context); // get the index from the data input. const index = this.index.getValue(context).value; // this is a flow graph integer, this is why we have `.value` if (array && index >= 0 && index < array.length) { // now populate the output with the right value this.value.setValue(array[index], context); } else { // optionally, we could also set it to null, because either the array is null or the index is out of bounds // this is important because we don't have exception handling here. The execution node taking this value will be able to trigger an error if it is null this.value.setValue(null, context); } } ``` This call is cached -- the function will be executed only once per execution ID, which is updated on each frame. There is no need to worry about reduced performance from consecutive calls. ### Execution block An execution block is a bit more complex than a data block. There are a few decisions that need to be made when implementing one: 1. Is the block asynchronous or synchronous? 2. What are my input signals? is `in` enough? 3. What are my output signals? is `out` enough? 4. Do I need to handle errors? 5. Are there signal-events I need to trigger? The basic class that needs to be extended is `FlowGraphExecutionBlock`. This class has a few methods that need to be overridden, an `in` input signal and `error` output signal. However, there are several other classes you can extend, depending on your needs: - FlowGraphExecutionBlockWithOutSignal - adds an `out` output signal - FlowGraphAsyncExecutionBlock - adds `done` to the `FlowGraphExecutionBlockWithOutSignal` class, and allows preparing and canceling an async task - FlowGraphEventBlock - a special extension of `FlowGraphAsyncExecutionBlock` that can react to scene-based events (which we will discuss in the next section) ### FlowGraphExecutionBlock Let's look at a simple example of `FlowGraphExecutionBlock`, the `FlowGraphBranchBlock`: ```javascript /** * A block that evaluates a condition and activates one of two branches. */ export class FlowGraphBranchBlock extends FlowGraphExecutionBlock { /** * Input connection: The condition to evaluate. */ public readonly condition: FlowGraphDataConnection; /** * Output connection: The branch to execute if the condition is true. */ public readonly onTrue: FlowGraphSignalConnection; /** * Output connection: The branch to execute if the condition is false. */ public readonly onFalse: FlowGraphSignalConnection; constructor(config?: IFlowGraphBlockConfiguration) { super(config); this.condition = this.registerDataInput("condition", RichTypeBoolean); this.onTrue = this._registerSignalOutput("onTrue"); this.onFalse = this._registerSignalOutput("onFalse"); } public _execute(context: FlowGraphContext): void { if (this.condition.getValue(context)) { this.onTrue._activateSignal(context); } else { this.onFalse._activateSignal(context); } } /** * @returns class name of the block. */ public override getClassName(): string { return FlowGraphBlockNames.Branch; } } ``` Let's break down the code: First, we define the inputs and outputs of this block: ```javascript public readonly condition: FlowGraphDataConnection; public readonly onTrue: FlowGraphSignalConnection; public readonly onFalse: FlowGraphSignalConnection; ``` The first, `condition`, is a data input. The other two are signal outputs, which are used to trigger the next block in the graph. The constructor is similar to the data block: ```javascript this.condition = this.registerDataInput("condition", RichTypeBoolean); // signal output don't need any type, as they hold no value this.onTrue = this._registerSignalOutput("onTrue"); this.onFalse = this._registerSignalOutput("onFalse"); ``` Now, the important part. When any input signal is triggered on any execution block, the `_execute` method is called. This is where the logic of the block is defined: ```javascript public _execute(context: FlowGraphContext), _callingSignal?: FlowGraphSignalConnection: void { // get the value of condition if (this.condition.getValue(context)) { // if it is true - execute the onTrue signal this.onTrue._activateSignal(context); } else { // otherwise execute the onFalse trigger this.onFalse._activateSignal(context); } } ``` The second variable of this function is an optional trigger -- it is the input signal that triggered this execution. We can use it this way (but this is not needed in this case): ```javascript public _execute(context: FlowGraphContext, callingSignal?: FlowGraphSignalConnection): void { if (callingSignal === this.in) { // do something } } ``` This is helpful when we have more than one input signal (for example, a reset signal). #### FlowGraphAsyncExecutionBlock An async block needs a few more functions to work correctly. Let's look at the SetDelay block (removing any unrelated code for readability): ```javascript /** * Block that sets a delay in seconds before activating the output signal. */ export class FlowGraphSetDelayBlock extends FlowGraphAsyncExecutionBlock { /** * The maximum number of parallel delays that can be set per node. */ public static MaxParallelDelayCount = 100; /** * Input signal: If activated the delayed activations set by this block will be canceled. */ public readonly cancel: FlowGraphSignalConnection; /** * Input connection: The duration of the delay in seconds. */ public readonly duration: FlowGraphDataConnection; /** * Output connection: The last delay index that was set. */ public readonly lastDelayIndex: FlowGraphDataConnection; constructor(config?: IFlowGraphBlockConfiguration) { super(config); this.cancel = this._registerSignalInput("cancel"); this.duration = this.registerDataInput("duration", RichTypeNumber); this.lastDelayIndex = this.registerDataOutput("lastDelayIndex", RichTypeNumber, -1); } public _preparePendingTasks(context: FlowGraphContext): void { const duration = this.duration.getValue(context); if (duration < 0 || isNaN(duration) || !isFinite(duration)) { this.error.payload = { message: "Invalid duration" }; return this.error._activateSignal(context); } // active delays are global to the context const activeDelays: number = context._getGlobalContextVariable("activeDelays", 0); if (activeDelays >= FlowGraphSetDelayBlock.MaxParallelDelayCount) { this.error.payload = { message: "Max parallel delays reached" }; return this.error._activateSignal(context); } // get the last global delay index const lastDelayIndex: number = context._getGlobalContextVariable("lastDelayIndex", -1); // these are block-specific and not global const timers = context._getExecutionVariable(this, "pendingDelays", [] as AdvancedTimer[]); const scene = context.configuration.scene; const timer: AdvancedTimer = new AdvancedTimer({ timeout: duration * 1000, // duration is in seconds contextObservable: scene.onBeforeRenderObservable, onEnded: () => this._onEnded(timer, context), }); timer.start(); const newIndex = lastDelayIndex + 1; this.lastDelayIndex.setValue(newIndex, context); context._setGlobalContextVariable("lastDelayIndex", newIndex); timers[newIndex] = timer; context._setExecutionVariable(this, "pendingDelays", timers); } public _cancelPendingTasks(context: FlowGraphContext): void { const timers = context._getExecutionVariable(this, "pendingDelays", [] as AdvancedTimer[]); for (const timer of timers) { timer?.dispose(); } context._deleteExecutionVariable(this, "pendingDelays"); this.lastDelayIndex.setValue(-1, context); } public _execute(context: FlowGraphContext, callingSignal: FlowGraphSignalConnection): void { if (callingSignal === this.cancel) { this._cancelPendingTasks(context); return; } else { this._preparePendingTasks(context); this.out._activateSignal(context); } } private _onEnded(timer: AdvancedTimer, context: FlowGraphContext) { const timers = context._getExecutionVariable(this, "pendingDelays", [] as AdvancedTimer[]); const index = timers.indexOf(timer); if (index !== -1) { timers.splice(index, 1); } else { Logger.Warn("FlowGraphTimerBlock: Timer ended but was not found in the running timers list"); } context._removePendingBlock(this); this.done._activateSignal(context); } } ``` Let's dive into the code. I will skip the first stage (defining the inputs and outputs) and go straight to the `_execute` method: ```javascript public _execute(context: FlowGraphContext, callingSignal: FlowGraphSignalConnection): void { if (callingSignal === this.cancel) { this._cancelPendingTasks(context); return; } else { this._preparePendingTasks(context); // trigger out to act as proxy. This is important for chaining! Further nodes can be executed on the same frame right after this one. this.out._activateSignal(context); } } ``` We can see that the execute function is still important for async execution as well. Here, we call the `_preparePendingTasks` method, which is where we define the logic of the block. Any async execution should allow cancellation, and this is done by checking whether the `cancel` signal was triggered. If it was, we call the `_cancelPendingTasks` method. Let's look at the `_preparePendingTasks` method and code-doc it thoroughly: ```javascript public _preparePendingTasks(context: FlowGraphContext): void { // get the value of duration const duration = this.duration.getValue(context); // validate the value and make sure we can create a delay based on this value if (duration < 0 || isNaN(duration) || !isFinite(duration)) { // set the error payload, in case the handling block wants to know what happened this.error.payload = { message: "Invalid duration" }; // trigger the error signal and don't continue return this.error._activateSignal(context); } // active delays are global to the context // remember, the global context is where we can set variables that are accessed by all blocks const activeDelays: number = context._getGlobalContextVariable("activeDelays", 0); // too many active delays? if (activeDelays >= FlowGraphSetDelayBlock.MaxParallelDelayCount) { // set the error payload, in case the handling block wants to know what happened this.error.payload = { message: "Max parallel delays reached" }; // trigger the error signal and don't continue return this.error._activateSignal(context); } // get the last global delay index const lastDelayIndex: number = context._getGlobalContextVariable("lastDelayIndex", -1); // these are block-specific and not global const timers = context._getExecutionVariable(this, "pendingDelays", [] as AdvancedTimer[]); const scene = context.configuration.scene; // create a new timer const timer: AdvancedTimer = new AdvancedTimer({ timeout: duration * 1000, // duration is in seconds contextObservable: scene.onBeforeRenderObservable, onEnded: () => this._onEnded(timer, context), }); timer.start(); const newIndex = lastDelayIndex + 1; this.lastDelayIndex.setValue(newIndex, context); // update the global context variable context._setGlobalContextVariable("lastDelayIndex", newIndex); timers[newIndex] = timer; // update the block-specific variable context._setExecutionVariable(this, "pendingDelays", timers); } ``` This function will start the timer and store it in the context for a future time when it is needed. Now let's look at the `_cancelPendingTasks` method: ```javascript public _cancelPendingTasks(context: FlowGraphContext): void { // get the timers from the context const timers = context._getExecutionVariable(this, "pendingDelays", [] as AdvancedTimer[]); // cancel each timer for (const timer of timers) { timer?.dispose(); } // delete the timers from the context context._deleteExecutionVariable(this, "pendingDelays"); // set the last delay index to -1 this.lastDelayIndex.setValue(-1, context); } ``` This method cancels all timers that were created by this block. This is important because we don't want to have any pending timers when the block is disposed. And finally, the function that will be triggered when the timer ends: ```javascript private _onEnded(timer: AdvancedTimer, context: FlowGraphContext) { // get the timers from the context const timers = context._getExecutionVariable(this, "pendingDelays", [] as AdvancedTimer[]); // get the index of the timer that ended const index = timers.indexOf(timer); // remove it from the list if (index !== -1) { timers.splice(index, 1); } else { Logger.Warn("FlowGraphTimerBlock: Timer ended but was not found in the running timers list"); } // remove the block from the pending blocks context._removePendingBlock(this); // trigger done signal this.done._activateSignal(context); } ``` This method removes the timer from the list of timers and triggers the `done` signal. This is the most important step -- we need to trigger the `done` signal so that the next block in the graph can execute. Overriding the `_executeOnTick` method is optional. This method is called on each frame, and can be used to update the block's logic. For example, if you want to update the block's logic based on the current frame, you can do it here. ```javascript public override _executeOnTick(_context: FlowGraphContext): void { // this function is called on each frame, and can be used to update the block's logic } ``` ### Block factory The block factory is used when a graph is being deserialized. It holds the list of available blocks including custom added blocks, and returns the class needed to create the block. If you are implementing a block in the framework, you can simply add it to the blockFactory function in `flowGraphBlockFactory.ts`: ```javascript export function blockFactory(blockName: FlowGraphBlockNames | string): () => Promise { switch (blockName) { case FlowGraphBlockNames.PlayAnimation: return async () => (await import("./Execution/Animation/flowGraphPlayAnimationBlock")).FlowGraphPlayAnimationBlock; // ... // add this before the last default case case FlowGraphBlockNames.MyNewBlock: // async-return the right file and the right class return async () => (await import("./MyNewBlock")).FlowGraphMyNewBlock; //... default: //... } } ``` The block factory is async-oriented to allow for lazy loading of blocks. This is important when an implementation only wants to load the blocks it needs, and not all of them. If you are implementing an external custom block - wonderful! You can add a custom block to the block factory using the exposed `addToBlockFactory` function: ```javascript /** * If you want to add a new block to the block factory, you should use this function. * Please be sure to choose a unique name and define the responsible module. * @param module the name of the module that is responsible for the block * @param blockName the name of the block. This should be unique. * @param factory an async factory function to generate the block */ export function addToBlockFactory(module: string, blockName: string, factory: () => Promise): void { customBlocks[`${module}/${blockName}`] = factory; } ``` This function requires you to define the module the block exists in. This is your decision and is only there for the sake of structure: ```javascript const module = "MyModuleBlocks"; // add the block to the factory addToBlockFactory(module, "ColorChangerBlock", async () => (await import("./MyNewBlock")).ColorChangerBlock); ``` Now the block will be available when deserialization requests the block named `MyModuleBlocks/ColorChangerBlock`.