oidn-web 0.3.5 → 0.4.0

package/CHANGELOG.md

@@ -0,0 +1,49 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## [0.4.0] - 2026-08-20
+
+oidn-web 0.4.0 replaces the TensorFlow.js inference stack with a purpose-built, model-driven WebGPU runtime. Existing `initUNetFromURL` and `initUNetFromBuffer` integrations remain supported while gaining native FP16, adaptive scheduling, runtime diagnostics, and stronger model validation.
+
+### Highlights
+
+- Removed TensorFlow.js and its WebGPU backend from the runtime dependencies.
+- Added a custom WGSL U-Net executor with automatic FP16 selection and a deterministic FP32 fallback.
+- Added built-in topology descriptors for the current OIDN small and large RT models, including clean auxiliary variants.
+- Reduced the reference Vite ESM bundle from approximately 168.8 KB to 31.1 KB gzip compared with 0.3.5, a reduction of about 82%.
+- Reached a 27.6 ms median for a 512 x 512 Clean Aux Large inference on the tested Apple GPU, versus 35.0 ms for the TensorFlow.js baseline (1.27x faster). Performance depends on the browser, GPU, model, and tile size.
+
+### Added
+
+- Model-driven graph planning and lifetime-aware activation-buffer reuse.
+- Versioned `UNetModelSpec` descriptors, topology detection, and validation of tensor names, layouts, data types, byte lengths, kernel shapes, bias shapes, and graph channel flow.
+- `modelSpec` initialization option for future or custom OIDN model topologies.
+- `npm run model:inspect` for model hashes, tensor signatures, and descriptor compatibility checks.
+- Dynamic tile sizing with configurable minimum, initial, and maximum tile sizes and a target GPU time.
+- GPU queue backpressure between tiles to keep cancellation and rendering interaction responsive.
+- Runtime diagnostics through `getRuntimeInfo()`, including the selected engine, precision, model, kernel capabilities, tile state, and resource statistics.
+- Per-layer GPU profiling through `profileNextExecution()` and `getLastExecutionProfile()` when `timestamp-query` is enabled.
+- Cross-version browser benchmarks with queue-complete timings, sampled output validation, TFJS baseline comparison, and per-layer GPU profiles.
+- Explicit, idempotent resource disposal and resource lifecycle accounting.
+- Automated model, graph, scheduler, resource leak, allocation rollback, and late-async-cleanup tests.
+
+### Changed
+
+- U-Net inference now runs directly in WGSL. The stable `auto` engine selects the native WGSL backend and no longer initializes TensorFlow.js.
+- FP16-capable devices retain TZA half-float weights and use short FP16 FMA accumulation groups folded into FP32 accumulators. The final output remains FP32.
+- Devices without `shader-f16` automatically use native FP32 inference.
+- Convolution tensors and activations use a blocked four-channel layout with channel-specialized shaders.
+- Decoder `upsample + concat + conv` patterns are fused, and all network passes for a tile are encoded into one command buffer.
+- Shape-independent pipelines are compiled asynchronously before initialization resolves to avoid first-denoise shader compilation stalls.
+- `maxTileSize` is now a hard upper bound for the adaptive tile controller.
+- A shared `GPUDevice` uses FP16 only when `shader-f16` was requested during device creation; WebGPU features cannot be enabled afterward.
+
+### Migration notes
+
+- No changes are required for basic `initUNetFromURL` or `initUNetFromBuffer` usage.
+- Call `dispose()` when a U-Net instance is no longer needed.
+- When supplying an existing `GPUDevice`, request `shader-f16` before creating the device if FP16 inference is desired.
+- Set `dynamicTile: false` to restore fixed-size tiling.
+
+[0.4.0]: https://github.com/pissang/oidn-web/compare/v0.3.5...v0.4.0

package/README.md

@@ -9,9 +9,22 @@ It's used in the [Vector to 3D](https://www.figma.com/community/plugin/126460021
 | :----------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------: |
 | ![](https://github.com/pissang/oidn-web/blob/main/examples/test/ground-truth.png 'Ground Truth') | ![](https://github.com/pissang/oidn-web/blob/main/examples/test/noisy.png 'Noisy') | ![](https://github.com/pissang/oidn-web/blob/main/examples/test/denoised.png 'Denoised') |
 
-## How it Works.
+## How it works
 
-It uses [tfjs](https://github.com/tensorflow/tfjs) to build the UNet model used by the OIDN. Then use the model to do prediction with a WebGPU backend from the image data.
+The OIDN U-Net runs directly on WebGPU with model-driven WGSL compute
+pipelines. Convolution activations use a blocked
+four-channel layout, decoder `upsample + concat + conv` patterns are fused,
+and all network dispatches for a tile are submitted in one command buffer.
+FP32 convolutions use channel-specialized implicit-GEMM tiles; FP16 uses
+channel-specialized vector FMA and separate max-pool passes, selected from the
+same model descriptor.
+
+TZA half-float weights stay half-float when the device enables `shader-f16`.
+FP16 products are accumulated in short half-precision groups and periodically
+folded into FP32 accumulators; the final output is FP32. Devices without
+`shader-f16` automatically use the native FP32 path. Shape-independent compute
+pipelines compile asynchronously before initialization resolves, so first-use
+shader compilation does not interrupt an interactive denoise.
 
 ## How to Use
 
@@ -42,7 +55,7 @@ initUNetFromURL('./weights/rt_ldr.tza').then((unet) => {
   const abortDenoising = unet.tileExecute({
     // The color input for LDR image is 4 channels.
     // In the format of Uint8ClampedArray or Uint8Array.
-    color: { data: noisyImageData, width, height },
+    color: noisyImageData,
     done(denoised) {
       console.log('Finished');
     },
@@ -111,7 +124,7 @@ If you already have a WebGPU path tracer. You can integrate the oidn-web into yo
 initUNetFromURL(
   './weights/rt_hdr_alb_nrm.tza',
   {
-    // Share GPUDevice and GPUAdapterInfo to the TFJS WebGPU backend
+    // Share GPUDevice and GPUAdapterInfo with the native WGSL runtime.
     device,
     adapterInfo
   },
@@ -153,6 +166,129 @@ initUNetFromURL('./weights/rt_hdr_alb_nrm_small.tza', ...);
 
 Other combinations can be found in the [oidn-weights](https://github.com/RenderKit/oidn-weights)
 
+### FP16 and runtime information
+
+Standalone initialization requests `shader-f16` when the adapter supports it.
+When sharing a device, optional features must be requested when that device is
+created; WebGPU features cannot be enabled afterward.
+
+```ts
+const requiredFeatures = adapter.features.has('shader-f16')
+  ? ['shader-f16']
+  : [];
+const device = await adapter.requestDevice({ requiredFeatures });
+
+const unet = await initUNetFromURL(
+  modelUrl,
+  { device, adapterInfo },
+  {
+    aux: true,
+    hdr: true,
+    precision: 'auto' // 'fp16' enforces support; 'fp32' is deterministic fallback
+  }
+);
+
+console.log(unet.getRuntimeInfo());
+// { gpuEngine: 'wgsl', precision: 'fp16', model: 'oidn-unet-large-v1', ... }
+```
+
+For one-shot native GPU timings, request a profile immediately before an
+execution. This is available when the shared device enabled `timestamp-query`:
+
+```ts
+if (unet.profileNextExecution()) {
+  unet.tileExecute({
+    color,
+    albedo,
+    normal,
+    done: async () => {
+      console.table((await unet.getLastExecutionProfile()).layers);
+    }
+  });
+}
+```
+
+### Updating to a new OIDN model
+
+TZA stores tensors but not the executable graph. The runtime therefore keeps
+the graph in a versioned `UNetModelSpec`, separate from shader and precision
+code. Built-in descriptors cover the current OIDN small and large RT U-Nets.
+At load time the descriptor is detected from the complete tensor-name set, and
+tensor layout, dtype, byte length, kernel shape, bias shape, and graph channel
+flow are validated before GPU resources are created.
+
+If an OIDN update keeps one of these topologies and tensor names, changed
+channel widths are handled automatically. If it adds or renames nodes, add a
+new descriptor (or pass `modelSpec`) and its validation fixture. Existing graph
+fusion rules apply to the new descriptor without changes to WGSL kernels.
+
+Use the inspection command to get a stable SHA-256, full tensor signature, and
+descriptor compatibility result for an upstream weight file:
+
+```shell
+npm run model:inspect -- weights/rt_hdr_alb_nrm.tza
+```
+
+```ts
+const unet = await initUNetFromURL(newModelUrl, backend, {
+  aux: true,
+  hdr: true,
+  modelSpec: newOidnModelSpec
+});
+```
+
+### GPU backpressure and dynamic tiles
+
+`tileExecute` waits for the submitted GPU work of a tile before scheduling the
+next tile. This keeps at most one OIDN tile in flight, which makes cancellation
+responsive instead of leaving queued denoising work ahead of interactive
+rendering.
+
+Tile sizing is adaptive by default. `maxTileSize` is a hard upper bound; the
+completed GPU time of a tiled execution adjusts the tile size used by the next
+execution. Single-tile images do not affect the estimate. The default range
+starts at 384 pixels, does not go below 256, and targets about 16 ms of GPU work
+per tile.
+
+```ts
+initUNetFromURL('./weights/rt_hdr_alb_nrm.tza', backend, {
+  aux: true,
+  hdr: true,
+  maxTileSize: 512,
+  dynamicTile: {
+    minTileSize: 256,
+    initialTileSize: 384,
+    targetTileTimeMs: 16
+  }
+});
+
+// Restore fixed-size behavior when deterministic tiling is preferred.
+initUNetFromURL('./weights/rt_hdr_alb_nrm.tza', backend, {
+  aux: true,
+  hdr: true,
+  maxTileSize: 512,
+  dynamicTile: false
+});
+```
+
+### Benchmark the native runtime against TFJS
+
+The browser benchmark automatically finds the nearest ancestor whose package
+still depends on TensorFlow.js, builds that commit in a temporary worktree, and
+compares it with the current WGSL FP32 and FP16 runtimes. Each measured run
+waits for the WebGPU queue to finish, so the result includes execution rather
+than only JavaScript command submission. It also samples output against the
+TFJS FP32 result and, when timestamp queries are supported, reports the five
+most expensive native network nodes.
+
+```shell
+npm run benchmark -- --width 512 --height 512 --tile-size 512 --runs 5
+```
+
+Results are printed as a table and written to
+`benchmarks/results/latest.{json,md}`. Use `--baseline <commit>` to pin an
+explicit historical version or `--chrome <path>` to select a browser.
+
 ## Credits
 
 Huge thanks to Max Liani for his series: https://maxliani.wordpress.com/2023/03/17/dnnd-1-a-deep-neural-network-dive/. My work is mostly inspired by it.