oidn-web 0.3.5 → 0.5.0

package/CHANGELOG.md

@@ -0,0 +1,94 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## [0.5.0] - 2026-10-09
+
+oidn-web 0.5.0 simplifies device setup, makes tiled execution always settle, and switches FP16 convolutions to implicit GEMM. It contains breaking changes for TypeScript callers that pass `adapterInfo`, for code that constructs `UNet` directly, and for code that relies on square tiles or per-frame tile pacing.
+
+### Breaking changes
+
+- `initUNetFromURL` and `initUNetFromBuffer` take `{ device }` as the second argument. `adapterInfo` was only needed by the removed TensorFlow.js backend and is no longer accepted by the type. Extra properties are ignored at runtime, but TypeScript rejects an object literal that still contains `adapterInfo`.
+- `new UNet(tensors, device, options)` takes a `GPUDevice` instead of a `{ device, adapterInfo }` object.
+- `tileExecute` now continues on the event loop between tiles by default instead of waiting for `requestAnimationFrame`. Pass `scheduling: 'animation-frame'` to keep pacing tiles to display frames.
+- Tiles are balanced rectangles with overlap only on edges shared with another tile, instead of fixed-size squares. Tile count, position, and size reported to `progress` differ from 0.4.0. The default `dynamicTile.initialTileSize` is now 432.
+- `UNetExecutionStats` no longer has `tileWidth` and `tileHeight`. Use `tileColumns`, `tileRows`, `tileOverlap`, `inputPixelCount`, and `inputShapeCount` instead.
+- FP16 convolutions use implicit GEMM by default. Output differs slightly from 0.4.0 because accumulation order changed. Set `kernel: 'direct'` to reproduce the 0.4.0 FP16 path.
+
+### Added
+
+- `hdrTransfer: 'log'` for RTLightmap HDR models, alongside the default PU transfer.
+- `error` callback on `tileExecute` for asynchronous failures, WebGPU device loss, and exceptions thrown by `progress` or `done`. `progress` and `done` may return promises.
+- `scheduling: 'event-loop' | 'animation-frame'`, `tileOverlap`, and `wholeImage` options on `tileExecute`.
+- `prepareForImage(width, height)` to create per-shape GPU resources before the first denoise.
+- `planTileGrid` and its `TilePlan`, `PlannedTile`, and `TileRect` types.
+- Experimental `gemm` tuning options. These are intended for benchmarking and are not covered by semver.
+- `hdrTransfer` and `activeExecutionCount` in `getRuntimeInfo()`, and the active GEMM configuration under `kernel.gemm`.
+
+### Changed
+
+- Implicit GEMM tiles are selected by output alignment and GPU limits, with optimized addressing, weight layout, register tiles, shared-memory layout, and pooling access.
+- The final RGB convolution uses an adaptive shared-memory cache.
+- Adaptive tile sizing uses the smoothed P75 tile GPU time, excludes the cold first tile, ignores cancelled and single-tile work, and buckets input shapes to at most two sizes.
+- `animation-frame` scheduling falls back to a 100 ms timer so execution still completes in hidden tabs.
+
+### Fixed
+
+- Tiled execution always settles with `done` or `error`, or stops silently after abort, including when `requestAnimationFrame` never fires or the device is lost.
+- Completed executions release their device-loss listeners.
+- Edge tiles whose size is not a multiple of 16 replicate edge pixels into the padded model input.
+- The npm package no longer includes benchmark results, tests, and scripts.
+
+### Migration notes
+
+- Replace `{ device, adapterInfo }` with `{ device }`.
+- Replace `new UNet(tensors, { device, adapterInfo }, options)` with `new UNet(tensors, device, options)`.
+- Interactive renderers that share the GPU with OIDN should pass `scheduling: 'animation-frame'`.
+- Pass an `error` callback to handle failures. Without one, failures are logged to the console.
+
+## [0.4.0] - 2026-08-20
+
+oidn-web 0.4.0 replaces the TensorFlow.js inference stack with a purpose-built, model-driven WebGPU runtime. Existing `initUNetFromURL` and `initUNetFromBuffer` integrations remain supported while gaining native FP16, adaptive scheduling, runtime diagnostics, and stronger model validation.
+
+### Highlights
+
+- Removed TensorFlow.js and its WebGPU backend from the runtime dependencies.
+- Added a custom WGSL U-Net executor with automatic FP16 selection and a deterministic FP32 fallback.
+- Added built-in topology descriptors for the current OIDN small and large RT models, including clean auxiliary variants.
+- Reduced the reference Vite ESM bundle from approximately 168.8 KB to 31.1 KB gzip compared with 0.3.5, a reduction of about 82%.
+- Reached a 27.6 ms median for a 512 x 512 Clean Aux Large inference on the tested Apple GPU, versus 35.0 ms for the TensorFlow.js baseline (1.27x faster). Performance depends on the browser, GPU, model, and tile size.
+
+### Added
+
+- Model-driven graph planning and lifetime-aware activation-buffer reuse.
+- Versioned `UNetModelSpec` descriptors, topology detection, and validation of tensor names, layouts, data types, byte lengths, kernel shapes, bias shapes, and graph channel flow.
+- `modelSpec` initialization option for future or custom OIDN model topologies.
+- `npm run model:inspect` for model hashes, tensor signatures, and descriptor compatibility checks.
+- Dynamic tile sizing with configurable minimum, initial, and maximum tile sizes and a target GPU time.
+- GPU queue backpressure between tiles to keep cancellation and rendering interaction responsive.
+- Runtime diagnostics through `getRuntimeInfo()`, including the selected engine, precision, model, kernel capabilities, tile state, and resource statistics.
+- Per-layer GPU profiling through `profileNextExecution()` and `getLastExecutionProfile()` when `timestamp-query` is enabled.
+- Cross-version browser benchmarks with queue-complete timings, sampled output validation, TFJS baseline comparison, and per-layer GPU profiles.
+- Explicit, idempotent resource disposal and resource lifecycle accounting.
+- Automated model, graph, scheduler, resource leak, allocation rollback, and late-async-cleanup tests.
+
+### Changed
+
+- U-Net inference now runs directly in WGSL. The stable `auto` engine selects the native WGSL backend and no longer initializes TensorFlow.js.
+- FP16-capable devices retain TZA half-float weights and use short FP16 FMA accumulation groups folded into FP32 accumulators. The final output remains FP32.
+- Devices without `shader-f16` automatically use native FP32 inference.
+- Convolution tensors and activations use a blocked four-channel layout with channel-specialized shaders.
+- Decoder `upsample + concat + conv` patterns are fused, and all network passes for a tile are encoded into one command buffer.
+- Shape-independent pipelines are compiled asynchronously before initialization resolves to avoid first-denoise shader compilation stalls.
+- `maxTileSize` is now a hard upper bound for the adaptive tile controller.
+- A shared `GPUDevice` uses FP16 only when `shader-f16` was requested during device creation; WebGPU features cannot be enabled afterward.
+
+### Migration notes
+
+- No changes are required for basic `initUNetFromURL` or `initUNetFromBuffer` usage.
+- Call `dispose()` when a U-Net instance is no longer needed.
+- When supplying an existing `GPUDevice`, request `shader-f16` before creating the device if FP16 inference is desired.
+- Set `dynamicTile: false` to restore fixed-size tiling.
+
+[0.5.0]: https://github.com/pissang/oidn-web/compare/v0.4.0...v0.5.0
+[0.4.0]: https://github.com/pissang/oidn-web/compare/v0.3.5...v0.4.0

package/README.md

@@ -9,9 +9,22 @@ It's used in the [Vector to 3D](https://www.figma.com/community/plugin/126460021
 | :----------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------: |
 | ![](https://github.com/pissang/oidn-web/blob/main/examples/test/ground-truth.png 'Ground Truth') | ![](https://github.com/pissang/oidn-web/blob/main/examples/test/noisy.png 'Noisy') | ![](https://github.com/pissang/oidn-web/blob/main/examples/test/denoised.png 'Denoised') |
 
-## How it Works.
+## How it works
 
-It uses [tfjs](https://github.com/tensorflow/tfjs) to build the UNet model used by the OIDN. Then use the model to do prediction with a WebGPU backend from the image data.
+The OIDN U-Net runs directly on WebGPU with model-driven WGSL compute
+pipelines. Convolution activations use a blocked
+four-channel layout, decoder `upsample + concat + conv` patterns are fused,
+and all network dispatches for a tile are submitted in one command buffer.
+FP16 and FP32 convolutions use channel-specialized implicit-GEMM tiles by
+default, with a direct convolution for the final output layer and separate
+max-pool passes.
+
+TZA half-float weights stay half-float when the device enables `shader-f16`.
+FP16 products are accumulated in short half-precision groups and periodically
+folded into FP32 accumulators; the final output is FP32. Devices without
+`shader-f16` automatically use the native FP32 path. Shape-independent compute
+pipelines compile asynchronously before initialization resolves, so first-use
+shader compilation does not interrupt an interactive denoise.
 
 ## How to Use
 
@@ -37,12 +50,12 @@ initUNetFromURL('./weights/rt_ldr.tza').then((unet) => {
     .getContext('2d')
     .getImageData(0, 0, width, height);
 
-  // Tile execute the denoising.
-  // If the resolution is high. It will split the input into tiles and execute one tile per frame.
+  // Tile execute the denoising. High resolutions use balanced rectangular
+  // tiles with overlap only at boundaries shared by another tile.
   const abortDenoising = unet.tileExecute({
     // The color input for LDR image is 4 channels.
     // In the format of Uint8ClampedArray or Uint8Array.
-    color: { data: noisyImageData, width, height },
+    color: noisyImageData,
     done(denoised) {
       console.log('Finished');
     },
@@ -76,6 +89,22 @@ initUNetFromURL('./weights/rt_hdr.tza', undefined, {
 });
 ```
 
+HDR transfer defaults to the PU curve used by the regular RT models. Models
+trained for the RTLightmap filter use the logarithmic curve from upstream OIDN;
+select it explicitly when loading such weights:
+
+```ts
+const lightmap = await initUNetFromURL('./weights/rtlightmap_hdr.tza', undefined, {
+  hdr: true,
+  hdrTransfer: 'log'
+});
+```
+
+The `log` transfer maps `y` to `log(1 + y) / log(65505)` and reverses this
+before writing HDR output. For GPU-buffer inputs, callers may continue to
+pre-scale the complete image once and leave the runtime's `inputScale` at its
+existing default of `1`.
+
 ### Use auxiliary images
 
 ```ts
@@ -111,9 +140,8 @@ If you already have a WebGPU path tracer. You can integrate the oidn-web into yo
 initUNetFromURL(
   './weights/rt_hdr_alb_nrm.tza',
   {
-    // Share GPUDevice and GPUAdapterInfo to the TFJS WebGPU backend
-    device,
-    adapterInfo
+    // Share the GPUDevice with the native WGSL runtime.
+    device
   },
   {
     aux: true,
@@ -153,6 +181,178 @@ initUNetFromURL('./weights/rt_hdr_alb_nrm_small.tza', ...);
 
 Other combinations can be found in the [oidn-weights](https://github.com/RenderKit/oidn-weights)
 
+### FP16 and runtime information
+
+Standalone initialization requests `shader-f16` when the adapter supports it.
+When sharing a device, optional features must be requested when that device is
+created; WebGPU features cannot be enabled afterward.
+
+```ts
+const requiredFeatures = adapter.features.has('shader-f16')
+  ? ['shader-f16']
+  : [];
+const device = await adapter.requestDevice({ requiredFeatures });
+
+const unet = await initUNetFromURL(
+  modelUrl,
+  { device },
+  {
+    aux: true,
+    hdr: true,
+    precision: 'auto' // 'fp16' enforces support; 'fp32' is deterministic fallback
+  }
+);
+
+console.log(unet.getRuntimeInfo());
+// { gpuEngine: 'wgsl', precision: 'fp16', model: 'oidn-unet-large-v1', ... }
+```
+
+For one-shot native GPU timings, request a profile immediately before an
+execution. This is available when the shared device enabled `timestamp-query`:
+
+```ts
+if (unet.profileNextExecution()) {
+  unet.tileExecute({
+    color,
+    albedo,
+    normal,
+    done: async () => {
+      console.table((await unet.getLastExecutionProfile()).layers);
+    }
+  });
+}
+```
+
+### Updating to a new OIDN model
+
+TZA stores tensors but not the executable graph. The runtime therefore keeps
+the graph in a versioned `UNetModelSpec`, separate from shader and precision
+code. Built-in descriptors cover the current OIDN small and large RT U-Nets.
+At load time the descriptor is detected from the complete tensor-name set, and
+tensor layout, dtype, byte length, kernel shape, bias shape, and graph channel
+flow are validated before GPU resources are created.
+
+If an OIDN update keeps one of these topologies and tensor names, changed
+channel widths are handled automatically. If it adds or renames nodes, add a
+new descriptor (or pass `modelSpec`) and its validation fixture. Existing graph
+fusion rules apply to the new descriptor without changes to WGSL kernels.
+
+Use the inspection command to get a stable SHA-256, full tensor signature, and
+descriptor compatibility result for an upstream weight file:
+
+```shell
+npm run model:inspect -- weights/rt_hdr_alb_nrm.tza
+```
+
+```ts
+const unet = await initUNetFromURL(newModelUrl, undefined, {
+  aux: true,
+  hdr: true,
+  modelSpec: newOidnModelSpec
+});
+```
+
+### GPU backpressure, scheduling, and dynamic tiles
+
+`tileExecute` waits for the submitted GPU work of a tile before scheduling the
+next tile. This keeps at most one OIDN tile in flight, which makes cancellation
+responsive instead of leaving queued denoising work ahead of interactive
+rendering.
+
+Between tiles, JavaScript yields according to `scheduling`:
+
+- `'event-loop'` (default) continues on the next macrotask. Execution finishes
+  as soon as the GPU allows and also completes in background tabs.
+- `'animation-frame'` waits for the next display frame, so at most one tile
+  runs per frame. Use it when OIDN shares the GPU with an interactive renderer
+  and frame rate matters more than denoise latency. If no frame arrives within
+  100 ms (for example in a hidden tab), the next tile runs anyway.
+
+Asynchronous failures, including WebGPU device loss and exceptions thrown by
+`progress` or `done`, are reported to the optional `error` callback. Unless the
+returned abort function is called first, every execution ends by calling `done`
+or `error`; without an `error` callback, failures are logged to the console.
+
+Tile sizing is adaptive by default. `maxTileSize` is a hard upper bound. Each
+execution partitions the image into balanced rectangular output regions and
+adds model context only on edges shared with another tile. Input shapes are
+bucketed to at most two sizes so the native execution cache remains stable.
+The smoothed P75 GPU time of completed tiled executions adjusts the maximum tile
+size used by the next execution. The potentially cold first tile is excluded,
+cancelled and single-tile work is ignored, and a layout is held for at least two
+complete executions. The default range starts at 432 pixels, does not go below
+256, changes in 16-pixel steps, and targets about 16 ms of GPU work per tile.
+
+```ts
+const unet = await initUNetFromURL('./weights/rt_hdr_alb_nrm.tza', undefined, {
+  aux: true,
+  hdr: true,
+  maxTileSize: 512,
+  dynamicTile: {
+    minTileSize: 256,
+    initialTileSize: 432,
+    targetTileTimeMs: 16
+  }
+});
+
+// An interactive renderer can pace tiles to display frames. A smaller halo
+// lowers per-tile cost; the default overlap is half of the model receptive
+// field rounded up to 16 pixels.
+const abortDenoising = unet.tileExecute({
+  color,
+  albedo,
+  normal,
+  tileOverlap: 80,
+  scheduling: 'animation-frame',
+  done(denoised) {
+    // ...
+  },
+  error(reason) {
+    console.error(reason);
+  }
+});
+
+// Restore fixed-size behavior when deterministic tiling is preferred.
+initUNetFromURL('./weights/rt_hdr_alb_nrm.tza', undefined, {
+  aux: true,
+  hdr: true,
+  maxTileSize: 512,
+  dynamicTile: false
+});
+```
+
+### Warm up and whole-image execution
+
+The native runtime creates per-shape GPU buffers and pipelines the first time
+it sees a tile input shape. Call `prepareForImage` with the expected image size while a loading
+state is still visible, so the first `tileExecute` does not stall:
+
+```ts
+await unet.prepareForImage(width, height);
+```
+
+Pass `wholeImage: true` to `tileExecute` (and to `prepareForImage`) to run the
+complete image as one tile and skip tiling overhead. It ignores `maxTileSize`,
+so the image must fit the device's buffer and dispatch limits.
+
+### Benchmark the native runtime against TFJS
+
+The browser benchmark automatically finds the nearest ancestor whose package
+still depends on TensorFlow.js, builds that commit in a temporary worktree, and
+compares it with the current WGSL FP32 and FP16 runtimes. Each measured run
+waits for the WebGPU queue to finish, so the result includes execution rather
+than only JavaScript command submission. It also samples output against the
+TFJS FP32 result and, when timestamp queries are supported, reports the five
+most expensive native network nodes.
+
+```shell
+npm run benchmark -- --width 512 --height 512 --tile-size 512 --runs 5
+```
+
+Results are printed as a table and written to
+`benchmarks/results/latest.{json,md}`. Use `--baseline <commit>` to pin an
+explicit historical version or `--chrome <path>` to select a browser.
+
 ## Credits
 
 Huge thanks to Max Liani for his series: https://maxliani.wordpress.com/2023/03/17/dnnd-1-a-deep-neural-network-dive/. My work is mostly inspired by it.