@babylonjs/smart-filters 0.1.0-alpha

package/readme.md

@@ -0,0 +1,165 @@
+# Babylon.js Smart Filters - PREVIEW
+
+# PREVIEW WARNING
+
+This package is currently in preview form, and updates will likely include breaking changes. It is not yet intended to be used by other projects.
+
+## Core
+
+A Smart Filter is a node based description of effects to apply on various inputs in order to create a visual output on a canvas element.
+
+The main usage is for video processing and composition but it could be easily reused for picture edition or procedural creation.
+
+## How to install
+
+This can be installed with `npm install @babylonjs/smart-filters`.
+
+It requires the following peer dependencies:
+
+-   @babylonjs/core
+
+## How to use
+
+The overall usage would look like:
+
+```typescript
+// Create a filter
+const smartFilter = new SmartFilter("Simplest");
+const titleInput = new InputBlock(smartFilter, "logo", ConnectionPointType.Texture, logoTexture);
+titleInput.output.connectTo(smartFilter.output);
+
+// Create a runtime to display the filter
+const engine = new ThinEngine(canvas, true);
+const runtime = await filter.createRuntimeAsync(engine);
+
+// Render one frame
+runtime.render();
+```
+
+The package has been marked as `sideEffects: false` so you can freely import from the index and know that tree shaking will remove any code you don't end up using.
+
+## How it works
+
+The entire system has been split in two parts the filter and the runtime.
+
+### SmartFilter
+
+The notion of a `SmartFilter` is a graph of blocks (all inheriting from `BaseBlock`) linked to each other through `ConnectionPoint`s.
+
+During the initialization phase, the `SmartFilter` examines the graph and builds a list of commands to execute during each frame and stores them in a `CommandBuffer`:
+
+-   This keeps each frame render very fast by ensuring the work is branchless.
+-   Basically, every `BaseBlock` during its `initialize` step will register in the `CommandBuffer` only the work it has to execute according to its options.
+-   In the end the render loop it only needs to run and execute each command in the list.
+-   This also allows for easy injection of debug tools or logging in between each command without growing the minimum required code to run a filter.
+-   As discussed, the `Blocks` are responsible for "injecting" their command in the filter `CommandBuffer`.
+-   For example: the `ShaderBlock` highlights how convenient it can be to only switch once at init time between rendering to the main frame buffer (if linked to the output) or to an intermediate texture if in the middle of the chain.
+
+Each block is responsible for exposing their inputs and outputs through `registerInput` and `registerOutput`:
+
+-   This creates and stores the related `ConnectionPoint`.
+-   All property values of a block must be updated only before `initialize` is called as the runtime won't reference it at render time.
+-   All values that need to be dynamically updated after `initialize` must be defined as connection points or 'StrongRef'.
+
+The `ConnectionPoints` are:
+
+-   Strongly typed, allowing more type safety while creating `SmartFilter` by code, yet keeping enough flexibility to be understood efficiently at runtime.
+-   They are also responsible for their compatibility when linked to each other.
+
+The Smart Filter is fully abstracted away from the runtime notion and does not even require a Babylon Engine to work with. It is only responsible to hold the "graph" of the filter or the "map" of all its blocks. A Filter only needs a name to be created:
+
+```typescript
+const smartFilter = new SmartFilter("Simplest");
+```
+
+Once a filter exist, one can add to it as many Blocks as necessary
+
+```typescript
+const blur = new BlurBlock(smartFilter, "blur");
+```
+
+Then, the various blocks can be linked together:
+
+```typescript
+videoInput.output.connectTo(blur.input);
+blur.output.connectTo(blackAndWhite.input);
+```
+
+Finally the last block should be linked into the Smart Filter output:
+
+```typescript
+titleInput.output.connectTo(smartFilter.output);
+```
+
+### Optimizations
+
+You can activate two optimizations once you've created your graph and have an instance of `SmartFilter` ready.
+
+The first is a graph optimizer which attempts to "merge" the "compatible" shader blocks to create an optimized version of the graph, thus reducing the final number of draw calls (as there are as many draw calls as there are shader blocks in the graph). This optimization pass will create a new instance of `SmartFilter`, which you can use in place of the initial instance. Here's how to do it:
+
+```typescript
+const forceMaxSamplersInFragmentShader = 0;
+const vfo = new SmartFilterOptimizer(smartFilter, {
+    // filters is an (unoptimized) instance of SmartFilter
+    maxSamplersInFragmentShader: forceMaxSamplersInFragmentShader || engine.getCaps().maxTexturesImageUnits,
+});
+const smartFilterOptimized = vfo.optimize()!; // filters is now an optimized instance of SmartFilter
+```
+
+One caveat is that a fragment shader has a limited number of samplers it can use, so we shouldn't merge blocks (even if they're compatible) once we've reached this limit. You should normally use the current GPU limit, so keep `forceMaxSamplersInFragmentShader` at 0 in the code above, to use the correct value for the GPU.
+
+The second optimization is a texture analyzer that traverses the graph and recycles textures between blocks (where possible), in order to limit the total number of textures used by the graph. You can activate it as follows:
+
+```typescript
+const rtg = new RenderTargetGenerator(true); // true to minimize the number of textures created
+const runtime = await filter.createRuntimeAsync(this.engine, rtg);
+```
+
+See the next section for details of how to create a runtime.
+
+### Runtime
+
+To keep a nice separation between edition time, resource management and render time, the filter itself cannot be rendered directly. In order to do so, a runtime needs to be created from the filter:
+
+```typescript
+const runtime = await smartFilter.createRuntimeAsync(engine);
+```
+
+_Note_ the same filter can be used across different runtimes.
+
+The runtime contains the list of resources required to render the filter like (intermediate textures, shaders and buffers). It is also owning a Command buffer containing the list of all the actions required to display the filter.
+
+For instance the content of the command buffer for the simplest filter would be:
+
+```
+----- Command buffer commands -----
+    Owner: OutputBlock (output) - Command: OutputBlock.render
+-----------------------------------
+```
+
+Whereas the one of a complex one could look like:
+
+```
+----- Command buffer commands -----
+    Owner: DirectionalBlurBlock (blurIV) - Command: DirectionalBlurBlock.render
+    Owner: DirectionalBlurBlock (blurIH) - Command: DirectionalBlurBlock.render
+    Owner: DirectionalBlurBlock (blurV) - Command: DirectionalBlurBlock.render
+    Owner: DirectionalBlurBlock (blurH) - Command: DirectionalBlurBlock.render
+    Owner: BlackAndWhiteBlock (blackAndWhite) - Command: BlackAndWhiteBlock.render
+    Owner: FrameBlock (frame) - Command: FrameBlock.renderToCanvas
+-----------------------------------
+```
+
+The command buffer is accessible through the runtime for debugging and logging purpose. The list of command could therefore be extended with custom ones if necessary of be parsed for introspection purpose as we do in our [logger](./src/command/commandBufferDebugger.ts).
+
+Rendering the current runtime is as simple as `runtime.render();`.
+
+_Note_ The runtime should be disposed once it is not used anymore to free the GPU memory and prevent leaks.
+
+## A few core Principles
+
+The overall system is trying at best to follow 3 simple rules:
+
+-   Be CPU efficient: for instance, we are trying to be branchless in most of our commands and we try to keep the number of commands as low as possible.
+-   Be memory efficient: no commands should allocate memory as it could trigger some garbage collection at the expense of frame loss.
+-   Be GPU efficient: the graph and texture optimizers minimize the number of "passes" required to render an image and the GPU resources used by the graph.

package/src/IDisposable.ts

@@ -0,0 +1,9 @@
+/**
+ * Define an interface for all classes that will hold resources
+ */
+export interface IDisposable {
+    /**
+     * Releases all held resources
+     */
+    dispose(): void;
+}

package/src/blocks/aggregateBlock.ts

@@ -0,0 +1,121 @@
+import type { ConnectionPoint } from "../connection/connectionPoint";
+import type { ConnectionPointType } from "../connection/connectionPointType";
+
+import { BaseBlock } from "../blocks/baseBlock.js";
+
+/**
+ * The aggregate block class is the base class for all blocks that be created from other blocks.
+ *
+ * It is responsible for managing a hidden chain of smart filter blocks in order and expose them through
+ * its own connection points.
+ *
+ * The internal state is basically a filter itself.
+ */
+export abstract class AggregateBlock extends BaseBlock {
+    /**
+     * The class name of the block.
+     */
+    public static override ClassName = "AggregateBlock";
+
+    /**
+     * The list of relationships between the internal graph output and the outside ones.
+     */
+    private readonly _aggregatedOutputs: [ConnectionPoint, ConnectionPoint][] = [];
+
+    /**
+     * The list of relationships between the internal graph inputs and the outside ones.
+     */
+    private readonly _aggregatedInputs: [ConnectionPoint, ConnectionPoint][] = [];
+
+    /**
+     * @internal
+     * Merges the internal graph into the SmartFilter
+     */
+    public _mergeIntoSmartFilter(mergedAggregateBlocks: AggregateBlock[]): void {
+        // Rewire output connections
+        for (const [internalConnectionPoint, externalConnectionPoint] of this._aggregatedOutputs) {
+            const endpointsToConnectTo = externalConnectionPoint.endpoints.slice();
+            externalConnectionPoint.disconnectAllEndpoints();
+            for (const endpoint of endpointsToConnectTo) {
+                internalConnectionPoint.connectTo(endpoint);
+            }
+        }
+
+        // Rewire input connections
+        for (const [internalConnectionPoint, externalConnectionPoint] of this._aggregatedInputs) {
+            const connectedToExternalConnectionPoint = externalConnectionPoint.connectedTo;
+            if (connectedToExternalConnectionPoint) {
+                connectedToExternalConnectionPoint.disconnectFrom(externalConnectionPoint);
+                connectedToExternalConnectionPoint.connectTo(internalConnectionPoint);
+            }
+        }
+
+        // Tell any internal aggregate blocks to merge
+        // Must be done after the inputs and outputs were merged at our level, or the internal aggregate block may not be wired up to anything
+        for (const aggregateOutput of this._aggregatedOutputs) {
+            const internalConnectionPoint = aggregateOutput[0];
+            internalConnectionPoint.ownerBlock.visit({}, (block: BaseBlock, _extraData: Object) => {
+                if (block instanceof AggregateBlock) {
+                    block._mergeIntoSmartFilter(mergedAggregateBlocks);
+                }
+            });
+        }
+
+        // Add ourselves to the list of merged aggregate blocks
+        mergedAggregateBlocks.push(this);
+    }
+
+    /**
+     * @internal
+     * Undoes a previous mergeIntoSmartFilter call.
+     */
+    public _unmergeFromSmartFilter(): void {
+        for (const [internalConnectionPoint, externalConnectionPoint] of this._aggregatedOutputs) {
+            const endpointsToConnectTo = internalConnectionPoint.endpoints.slice();
+            internalConnectionPoint.disconnectAllEndpoints();
+            for (const endpoint of endpointsToConnectTo) {
+                externalConnectionPoint.connectTo(endpoint);
+            }
+        }
+
+        for (const [internalConnectionPoint, externalConnectionPoint] of this._aggregatedInputs) {
+            const connectedToInternalConnectionPoint = internalConnectionPoint.connectedTo;
+            if (connectedToInternalConnectionPoint) {
+                connectedToInternalConnectionPoint.disconnectFrom(internalConnectionPoint);
+                connectedToInternalConnectionPoint.connectTo(externalConnectionPoint);
+            }
+        }
+    }
+
+    /**
+     * Registers an input connection from the internal graph as an input of the aggregated graph.
+     * @param name - The name of the exposed input connection point
+     * @param internalConnectionPoint - The input connection point in the inner graph to expose as an input on the aggregate block
+     * @returns the connection point referencing the input block
+     */
+    protected _registerSubfilterInput<U extends ConnectionPointType>(
+        name: string,
+        internalConnectionPoint: ConnectionPoint<U>
+    ): ConnectionPoint<U> {
+        const externalInputConnectionPoint = this._registerInput(name, internalConnectionPoint.type);
+
+        this._aggregatedInputs.push([internalConnectionPoint, externalInputConnectionPoint]);
+        return externalInputConnectionPoint;
+    }
+
+    /**
+     * Registers an output connection point from the internal graph as an output of the aggregated graph.
+     * @param name - The name of the exposed output connection point
+     * @param internalConnectionPoint - The output connection point in the inner graph to expose as an output on the aggregate block
+     * @returns the connection point referencing the output connection point
+     */
+    protected _registerSubfilterOutput<U extends ConnectionPointType>(
+        name: string,
+        internalConnectionPoint: ConnectionPoint<U>
+    ): ConnectionPoint<U> {
+        const externalOutputConnectionPoint = this._registerOutput(name, internalConnectionPoint.type);
+
+        this._aggregatedOutputs.push([internalConnectionPoint, externalOutputConnectionPoint]);
+        return externalOutputConnectionPoint;
+    }
+}

package/src/blocks/baseBlock.ts

@@ -0,0 +1,341 @@
+import type { Nullable } from "@babylonjs/core/types";
+import { UniqueIdGenerator } from "@babylonjs/core/Misc/uniqueIdGenerator.js";
+
+import { ConnectionPointType, type ConnectionPointValue } from "../connection/connectionPointType.js";
+import type { InitializationData, SmartFilter } from "../smartFilter";
+import type { ICommandOwner } from "../command/command";
+import { ConnectionPoint, type RuntimeData } from "../connection/connectionPoint.js";
+import { ConnectionPointWithDefault } from "../connection/connectionPointWithDefault.js";
+import { ConnectionPointDirection } from "../connection/connectionPointDirection.js";
+
+/**
+ * Defines a callback function that is triggered when visiting a block,
+ * It also carries over extra data preventing the need to use global variables or closures.
+ */
+export type BlockVisitor<T extends object> = (block: BaseBlock, extraData: T) => void;
+
+/**
+ * This class represents the base class for all smart filter blocks.
+ *
+ * It defines the basic structure of a smart filter block and provides the base implementation for
+ * managing the connection points.
+ *
+ * It enforces common behavior for all smart filter blocks.
+ */
+export abstract class BaseBlock implements ICommandOwner {
+    protected static _alreadyVisitedBlocks = new Set<BaseBlock>();
+
+    /**
+     * The class name of the block.
+     */
+    public static ClassName = "BaseBlock";
+
+    /**
+     * The smart filter the block belongs to.
+     */
+    public readonly smartFilter: SmartFilter;
+
+    /**
+     * Global unique id of the block (This is unique for the current session).
+     */
+    public readonly uniqueId: number;
+
+    /**
+     * The name of the block. This is used to identify the block in the smart filter or in debug.
+     */
+    public readonly name: string;
+
+    /**
+     * User provided comments about the block. It can be used to document the block.
+     */
+    public comments: Nullable<string> = null;
+
+    private readonly _inputs: ConnectionPoint[] = [];
+    private readonly _outputs: ConnectionPoint[] = [];
+
+    /**
+     * Instantiates a new block.
+     * @param smartFilter - Defines the smart filter the block belongs to
+     * @param name - Defines the name of the block
+     * @param disableOptimization - Defines if the block is optimizable or not
+     */
+    constructor(
+        smartFilter: SmartFilter,
+        name: string,
+        public readonly disableOptimization = false
+    ) {
+        this.uniqueId = UniqueIdGenerator.UniqueId;
+        this.name = name;
+        this.smartFilter = smartFilter;
+
+        // Register the block in the smart filter
+        smartFilter.registerBlock(this);
+    }
+
+    /**
+     * Returns the inputs connection points of the current block.
+     */
+    public get inputs(): ReadonlyArray<ConnectionPoint> {
+        return this._inputs;
+    }
+
+    /**
+     * Returns the outputs connection points of the current block.
+     */
+    public get outputs(): ReadonlyArray<ConnectionPoint> {
+        return this._outputs;
+    }
+
+    /**
+     * Returns if the block is an input block.
+     */
+    public get isInput(): boolean {
+        return this._inputs.length === 0;
+    }
+
+    /**
+     * Returns if the block is an output block.
+     */
+    public get isOutput(): boolean {
+        return this._outputs.length === 0;
+    }
+
+    /**
+     * @returns the class name of the block
+     */
+    public getClassName(): string {
+        // Note that we use a static property instead of doing this.constructor.name to avoid problems with minifiers that would change the name of the class
+        return (this.constructor as typeof BaseBlock).ClassName;
+    }
+
+    /**
+     * Checks if the block is an "ancestor" of another giving block.
+     * @param block - Defines the block to check against
+     * @returns True if the block is an ancestor of the given block, otherwise false
+     */
+    public isAnAncestorOf(block: BaseBlock): boolean {
+        for (const output of this._outputs) {
+            if (!output.endpoints.length) {
+                continue;
+            }
+
+            for (const endpoint of output.endpoints) {
+                if (endpoint.ownerBlock === block) {
+                    return true;
+                }
+                if (endpoint.ownerBlock.isAnAncestorOf(block)) {
+                    return true;
+                }
+            }
+        }
+
+        return false;
+    }
+
+    protected _visitInputs<T extends object>(
+        extraData: T,
+        callback: BlockVisitor<T>,
+        alreadyVisited: Set<BaseBlock>
+    ): void {
+        for (const input of this.inputs) {
+            if (!input.connectedTo) {
+                continue;
+            }
+
+            const block = input.connectedTo.ownerBlock;
+
+            block.visit(extraData, callback, alreadyVisited);
+        }
+    }
+
+    /**
+     * Visits the block and its inputs recursively.
+     * When starting from the smart filter output block, this will visit all the blocks in the smart filter.
+     * Note that it's a depth first visit: the callback is called on the block AFTER visiting its inputs.
+     * @param extraData - The extra data to pass to the callback
+     * @param callback - The callback to call on each block
+     * @param alreadyVisitedBlocks  - Defines the set of blocks already visited (if not provided, a new set will be created)
+     */
+    public visit<T extends object>(
+        extraData: T,
+        callback: BlockVisitor<T>,
+        alreadyVisitedBlocks?: Set<BaseBlock>
+    ): void {
+        if (!alreadyVisitedBlocks) {
+            alreadyVisitedBlocks = BaseBlock._alreadyVisitedBlocks;
+            alreadyVisitedBlocks.clear();
+        }
+
+        if (!alreadyVisitedBlocks.has(this)) {
+            alreadyVisitedBlocks.add(this);
+
+            this._visitInputs(extraData, callback, alreadyVisitedBlocks);
+
+            callback(this, extraData);
+        }
+    }
+
+    /**
+     * Finds the input connection point with the given name.
+     * @param name - Name of the input to find
+     * @returns The connection point with the given name or null if not found
+     */
+    public findInput<U extends ConnectionPointType>(name: string): Nullable<ConnectionPoint<U>> {
+        for (const input of this._inputs) {
+            if (input.name === name) {
+                return input as ConnectionPoint<U>;
+            }
+        }
+
+        return null;
+    }
+
+    /**
+     * Disconnects the block from the graph.
+     * @param _disconnectedConnections - Stores the connections that have been broken in the process. You can reconnect them later if needed.
+     */
+    public disconnectFromGraph(_disconnectedConnections?: [ConnectionPoint, ConnectionPoint][]): void {}
+
+    /**
+     * Prepares the block for runtime.
+     * This is called by the smart filter just before creating the smart filter runtime, and by the optimizer.
+     */
+    public prepareForRuntime(): void {}
+
+    /**
+     * Propagates the runtime data - telling all outputs to propagate their runtime data forward through the graph
+     */
+    public propagateRuntimeData(): void {
+        for (const output of this._outputs) {
+            output.propagateRuntimeData();
+        }
+    }
+
+    /**
+     * Generates the commands needed to execute the block at runtime and gathers promises for initialization work
+     * @param initializationData - The initialization data to use
+     * @param _finalOutput - Defines if the block is the final output of the smart filter
+     */
+    public generateCommandsAndGatherInitPromises(initializationData: InitializationData, _finalOutput: boolean): void {
+        // Check if any inputs are Textures which aren't yet ready, and if so, ensure init waits for them to be ready
+        for (const input of this._inputs) {
+            if (input.type === ConnectionPointType.Texture) {
+                const texture = input.runtimeData?.value as Nullable<ConnectionPointValue<typeof input.type>>;
+                if (texture && !texture.isReady()) {
+                    const internalTexture = texture.getInternalTexture();
+                    if (internalTexture) {
+                        const textureReadyPromise = new Promise<void>((resolve, reject) => {
+                            internalTexture.onLoadedObservable.add(() => {
+                                resolve();
+                            });
+                            internalTexture.onErrorObservable.add((error) => {
+                                reject(error);
+                            });
+                        });
+                        initializationData.initializationPromises.push(textureReadyPromise);
+                    }
+                }
+            }
+        }
+    }
+
+    /**
+     * Disconnects all the inputs and outputs from the Block.
+     */
+    public disconnect(): void {
+        // Detach inputs
+        for (const input of this._inputs) {
+            input.connectedTo?.disconnectFrom(input);
+        }
+
+        // Detach outputs
+        for (const output of this._outputs) {
+            output.disconnectAllEndpoints();
+        }
+    }
+
+    /**
+     * Registers a new input connection point in the block which must have a connection before the graph can be used.
+     * @param name - Defines the name of the input connection point
+     * @param type - Defines the type of the input connection point
+     * @param defaultValue - Defines the optional default value of the input connection point to use if not connection is made
+     * @returns The new ConnectionPoint
+     * @internal
+     */
+    public _registerInput<U extends ConnectionPointType>(
+        name: string,
+        type: U,
+        defaultValue: Nullable<RuntimeData<U>> = null
+    ): ConnectionPoint<U> {
+        const input = new ConnectionPoint(name, this, type, ConnectionPointDirection.Input, defaultValue);
+        this._inputs.push(input);
+        return input;
+    }
+
+    /**
+     * Registers a new input connection point in the block which doesn't require a connection because it has a default value.
+     * @param name - Defines the name of the input connection point
+     * @param type - Defines the type of the input connection point
+     * @param defaultValue - Defines the default value to use if nothing is connected to this connection point
+     * @returns The new ConnectionPointWithDefault
+     * @internal
+     */
+    public _registerOptionalInput<U extends ConnectionPointType>(
+        name: string,
+        type: U,
+        defaultValue: RuntimeData<U>
+    ): ConnectionPointWithDefault<U> {
+        const input = new ConnectionPointWithDefault(name, this, type, ConnectionPointDirection.Input, defaultValue);
+        this._inputs.push(input);
+        return input;
+    }
+
+    /**
+     * Registers a new output connection point in the block.
+     * @param name - Defines the name of the output connection point
+     * @param type - Defines the type of the output connection point
+     * @returns The new output connection point
+     * @internal
+     */
+    public _registerOutput<U extends ConnectionPointType>(name: string, type: U): ConnectionPoint<U> {
+        const output = new ConnectionPoint(name, this, type, ConnectionPointDirection.Output);
+        this._outputs.push(output);
+        return output;
+    }
+
+    /**
+     * Registers a new output connection point in the block that always has runtimeData because it has a default value and doesn't allow it to be overwritten with null.
+     * @param name - Defines the name of the output connection point
+     * @param type - Defines the type of the output connection point
+     * @param initialValue - Defines the initial value of the output connection point
+     * @returns The new output connection point with a default value
+     * @internal
+     */
+    public _registerOutputWithDefault<U extends ConnectionPointType>(
+        name: string,
+        type: U,
+        initialValue: RuntimeData<U>
+    ): ConnectionPointWithDefault<U> {
+        const output = new ConnectionPointWithDefault(name, this, type, ConnectionPointDirection.Output, initialValue);
+        this._outputs.push(output);
+        return output;
+    }
+
+    /**
+     * Gets the required RuntimeData for the given input, throwing with a clear message if it is null
+     * @param input - The input to get the runtime data for
+     * @returns The runtimeData or throws if it was undefined
+     */
+    protected _confirmRuntimeDataSupplied<U extends ConnectionPointType = ConnectionPointType>(
+        input: ConnectionPoint<U>
+    ): RuntimeData<U> {
+        if (!input.runtimeData) {
+            throw new Error(
+                `The ${ConnectionPointType[input.type]} input named "${
+                    input.name
+                }" is missing for the ${this.getClassName()} named "${this.name}"`
+            );
+        }
+        return input.runtimeData;
+    }
+}