gpucanvas 0.3.0 → 0.4.0

package/README.md

@@ -1,14 +1,48 @@
 # gpucanvas
 
-Put thousands of live DOM nodes on an infinite canvas and let the GPU pan and zoom them.
+Put thousands of live DOM nodes on an infinite canvas: the GPU draws them, an editor selects,
+moves and resizes them, shaders and transitions run on them, and an AI agent can read them and
+point at them.
 
 ![The same 5,000 live React screens as plain DOM (left) and in gpucanvas (right), with each side's frame rate measured live](docs/benchmark.gif)
 
 The same session on both sides: 5,000 live React screens, the same camera path, a message typed and sent. Left, every screen is plain DOM under a CSS transform. Right, gpucanvas. Recorded in Chrome on a MacBook Pro; the frame rates are measured live in each run. Run it yourself in the [live demo](https://berkantay.github.io/gpucanvas/).
 
+- **[Canvas](#how-it-works).** Live React or plain DOM nodes, drawn by WebGPU in one instanced
+  pass. Pan and zoom never touch the DOM, and clicks, typing and state still reach the real
+  elements.
+- **[Editor](#editing).** Click to select, double-click to go a layer deeper, down to any element
+  or React component. Move and resize with snapping guides.
+- **[Shader effects](#shader-effects).** Run WGSL on live components: a glass lens, frost,
+  shockwaves, glitch, VHS, or your own.
+- **[Snapshots](#snapshots).** Any node as an image, pixel diffs between versions, and a
+  filmstrip of its history.
+- **[Agent view](#agent-view).** What is on the canvas as a short JSON tree with ids, and an
+  image of any region, for an AI agent to read and point at.
+- **[Transitions](#transitions).** Dissolve, wipe or fade a node out and another in, on the
+  GPU, while both stay live.
+- **[Marks](#marks).** Arrows between elements, comment pins, notes and live cursors that
+  follow the camera.
+
+![An agent cursor flags a heading, links two buttons and drops a pin; the screen wipes out and dissolves back; a glass lens follows the pointer; shockwaves ripple from clicks](docs/features.gif)
+
+Marks from an agent, a wipe and a dissolve, the glass lens and shockwaves, all on live
+components. Recorded in Chrome from the demo.
+
 ## Try it
 
-Open **[berkantay.github.io/gpucanvas](https://berkantay.github.io/gpucanvas/)** in Chrome 157 or later. Thirty agent cursors build the first view, then the camera pulls out to all 5,000 screens. Press `b` to run the benchmark tour.
+Open **[berkantay.github.io/gpucanvas](https://berkantay.github.io/gpucanvas/)** in Chrome 157 or
+later. Thirty agent cursors build the first view, then the camera pulls out to all 5,000 screens.
+
+- Click a screen to select it, double-click to go deeper, drag to move it.
+- Pick a shader effect in the toolbar, or open one directly:
+  [glass](https://berkantay.github.io/gpucanvas/?effect=glass),
+  [shockwave](https://berkantay.github.io/gpucanvas/?effect=shockwave) (click anywhere),
+  [frost](https://berkantay.github.io/gpucanvas/?effect=frost),
+  [glitch](https://berkantay.github.io/gpucanvas/?effect=glitch).
+- Select a screen and press `D` to dissolve it, `W` to wipe it, or `S` to save it as a PNG.
+- In the console, `agentView.describe()` lists what is in view, and `marks` draws on it.
+- Press `b` to run the benchmark tour.
 
 The GPU path needs HTML-in-Canvas, which is behind a flag:
 
@@ -18,7 +52,7 @@ The GPU path needs HTML-in-Canvas, which is behind a flag:
 
 Without the flag the demo runs the DOM fallback with 1,000 screens and says so.
 
-Status: early (0.3). The API can still change.
+Status: early (0.4). The API can still change.
 
 ```sh
 npm install gpucanvas
@@ -131,8 +165,9 @@ Built in:
   `dither` (a lens of big dithered pixels).
 - On each press: `shockwave`, a ring that bends the node as it spreads.
 - Running on their own: `glitch`, `vhs`, `glow`, `hue`, `ripple`.
-- Still: `blur`, `pixelate`, `grayscale`, and `dissolve` for a node leaving or arriving (its
-  parameter is the progress, 0 to 1).
+- Still: `blur`, `pixelate`, `grayscale`.
+- [Transitions](#transitions): `dissolve`, `wipe` and `fade`, for a node leaving or arriving
+  (the first parameter is the progress, 0 to 1).
 
 Animated effects draw every frame while their node is in view; still ones cost nothing once
 drawn.
@@ -158,12 +193,104 @@ const scanlines = defineEffect({
 `EffectInput` has `uv` (0 to 1 across the node), `size` in CSS pixels, `time` in seconds,
 `zoom`, the four `params`, the `pointer` and the last `press` in the node's uv, and
 `sincePress` in seconds. `source(uv)` reads the node's pixels, premultiplied, and is
-transparent outside it; `hash` and `noise` give random values. An effect that draws past its node, like a glow, declares how far with
-`pad`. Effects read the node's whole snapshot, so very close in they are softer than plain nodes.
+transparent outside it; `hash` and `noise` give random values. An effect that draws past its
+node, like a glow, declares how far with `pad`. Effects read the node's whole snapshot, so very
+close in they are softer than plain nodes. In React, `<CanvasNode effect={effects.glass}>` does
+the same.
 
 The DOM backend shows effects that have a `css` look as a CSS `filter` (animated ones as filter
 animations) and the rest as plain nodes. The demo's toolbar puts one effect on every component.
 
+## Snapshots
+
+Every node is already drawn by the GPU, so its pixels are one call away, with no headless
+browser:
+
+```ts
+import { diffImages, recordFilmstrip, toBlob } from 'gpucanvas';
+
+const before = await canvas.snapshot(node, { scale: 2 }); // ImageData, drawn fresh
+const png = await toBlob(before); // a PNG Blob
+
+// ...the screen changes...
+const after = await canvas.snapshot(node, { scale: 2 });
+const { changed, bounds, image } = diffImages(before, after); // what moved, and where
+
+// A snapshot after every change, once it settles.
+const strip = recordFilmstrip(canvas, node, { scale: 0.25, max: 30 });
+strip.frames; // [{ at, image }, ...]
+strip.stop();
+```
+
+A snapshot is the node as it looks now, at any scale, whether or not it is in view. It takes a
+paint or two, about 40 ms for a 1,500 by 1,000 pixel screen. In the demo, select a screen and
+press `S` to save it. Snapshots need the GPU backend; the DOM backend rejects.
+
+## Agent view
+
+An agent reads the canvas the way a person sees it: what is there, what it says, and where:
+
+```ts
+import { createAgentView } from 'gpucanvas';
+
+const view = createAgentView(canvas);
+view.describe(); // what is in view, as JSON
+// [{ id: 'e35', tag: 'div', rect: { x: -3552, y: -2561, width: 1496, height: 958 }, children: [
+//   { id: 'e6', tag: 'h1', role: 'heading', text: 'Welcome back', rect: { ... } },
+//   { id: 'e18', tag: 'input', role: 'textbox', name: 'Email', rect: { ... } },
+//   { id: 'e23', tag: 'input', role: 'checkbox', name: 'Keep me signed in', checked: true, ... },
+//   { id: 'e25', tag: 'button', role: 'button', text: 'Sign in', rect: { ... } }, ...] }]
+
+view.element('e25'); // the element behind an id
+await view.capture('e6', { scale: 2 }); // ImageData of that element
+await view.capture({ x: 0, y: 0, width: 4000, height: 3000 }, { scale: 0.25 }); // any region
+```
+
+Wrappers that add nothing are left out, so a sign-in screen is about 35 entries and 3.5 KB of
+JSON, read in 3 ms. Pass `rect` to read another region and `name` to label elements with your
+own names, such as their React component's. `capture` needs the GPU backend.
+
+## Transitions
+
+Any effect whose first parameter is progress (0 whole, 1 gone) animates as a transition:
+`dissolve`, `wipe` and `fade` are built in.
+
+```ts
+import { animateEffect, effects, swapNodes } from 'gpucanvas';
+
+await animateEffect(canvas, node, effects.wipe, { duration: 600, params: [90] }); // wipe downward
+await animateEffect(canvas, node, effects.wipe, { from: 1, to: 0 }); // and back
+
+// One screen burns away while the next appears in its place.
+await swapNodes(canvas, oldScreen, newScreen, { effect: effects.dissolve, duration: 800 });
+canvas.removeNode(oldScreen);
+```
+
+A node that ends whole loses the effect; one that ends gone stays hidden until it is removed.
+Starting another animation on a node replaces the running one, which resolves `false`. In the
+demo, select a screen and press `D` to dissolve it or `W` to wipe it. The DOM backend fades.
+
+## Marks
+
+Draw on top of the nodes. Marks follow the camera and the elements they are attached to, and
+stay the same size on screen at any zoom:
+
+```ts
+import { createMarks } from 'gpucanvas';
+
+const marks = createMarks(canvas);
+marks.connect(button, nextScreen, { label: 'goes to' }); // a curved arrow
+marks.pin(card, { label: '3', text: 'Rename to Continue?', onClick: openThread });
+marks.note(view.element('e6'), 'This heading is 2px off the grid'); // what an agent means
+const cursor = marks.cursor('Ada', { color: '#e11d48' });
+cursor.move({ x: 120, y: 80 }); // glides there, in world units
+marks.clear();
+```
+
+Arrows and pins also take a world point in place of an element. Marks are SVG and HTML in a
+layer above the canvas, so they work with both backends. In the demo, both are on `window`:
+`agentView` and `marks`.
+
 ## Run the demo
 
 ```sh
@@ -171,6 +298,12 @@ bun install
 bun run dev        # http://localhost:5199/?mode=gpu&n=10000  (mode: auto | gpu | dom)
 ```
 
+## Releases
+
+Publishing a GitHub release for a `vX.Y.Z` tag that matches `package.json` publishes the package
+to npm from GitHub Actions, with provenance, through npm trusted publishing. See
+[release.yml](.github/workflows/release.yml).
+
 ## How it works
 
 1. **Pages.** Nodes are live DOM inside `<canvas content="drawable">`, packed side by side into
@@ -186,7 +319,8 @@ bun run dev        # http://localhost:5199/?mode=gpu&n=10000  (mode: auto | gpu
 4. **Tiles up close.** Past 100% zoom on a 2x display, only the visible part of a node is
    rasterized, as sharp tiles, center first. While its content animates, one half-resolution preview stands in.
 5. **One draw call.** Each frame culls nodes against the view and draws every visible one with
-   one instanced quad draw. Panning and zooming never touch the DOM. Raster work has a
+   one instanced quad draw, or one per run of nodes sharing a shader effect. Panning and zooming
+   never touch the DOM. Raster work has a
    per-frame budget, so the camera stays smooth while pixels fill in.
 6. **Live updates.** A MutationObserver marks a node dirty when its content changes, and
    loading images, web fonts and CSS transitions trigger a redraw. Animated content on nodes
@@ -200,7 +334,7 @@ bun run dev        # http://localhost:5199/?mode=gpu&n=10000  (mode: auto | gpu
 The demo's benchmark (press `b`) runs a 30-second tour: a message typed and sent, a chart
 animating, a push to 500% zoom, shift-click picks and a selection sweep, while 30 agent cursors
 build screens around the view. Every frame is timed. At 5,000 screens it measures 113 to 115 fps
-for gpucanvas and 18 to 20 for DOM.
+for gpucanvas and 18 to 20 for DOM, and 138 for gpucanvas in the effects run below.
 
 The table below was measured with the 0.1 benchmark, a 94-second session from the overview into
 clusters of app screens, close-ups up to 500% zoom, 20 clicks and typed messages, then back out.
@@ -212,6 +346,20 @@ clusters of app screens, close-ups up to 500% zoom, 20 clicks and typed messages
 
 DOM is the same screens in one CSS-transformed container, which is also the fallback backend.
 
+Shader effects on every one of the 5,000 screens, the same 30-second tour, vsync off:
+
+| Effect         | gpucanvas avg fps | p95     |
+| -------------- | ----------------- | ------- |
+| none           | 138               | 15.2 ms |
+| glow           | 135               | 15.6 ms |
+| blur (32 taps) | 138               | 16.7 ms |
+| glass          | 122               | 18.2 ms |
+| frost          | 146               | 13.8 ms |
+
+Effects draw from each node's snapshot in the same pass, so most cost nothing measurable; glass,
+which samples around the pointer with a rim and shadow, costs about 12%. Each figure is one run,
+so a gap of a few percent, like frost above none, is noise rather than a speedup.
+
 ## Chrome 157 findings (differ from the WICG explainer)
 
 - **Hit-test registration.** A drawable is hit-tested only after