gpucanvas 0.3.1 → 0.4.1

package/README.md

@@ -1,7 +1,8 @@
 # gpucanvas
 
 Put thousands of live DOM nodes on an infinite canvas: the GPU draws them, an editor selects,
-moves and resizes them, and shaders run on them.
+moves and resizes them, shaders and transitions run on them, and an AI agent can read them and
+point at them.
 
 ![The same 5,000 live React screens as plain DOM (left) and in gpucanvas (right), with each side's frame rate measured live](docs/benchmark.gif)
 
@@ -14,18 +15,34 @@ The same session on both sides: 5,000 live React screens, the same camera path,
   or React component. Move and resize with snapping guides.
 - **[Shader effects](#shader-effects).** Run WGSL on live components: a glass lens, frost,
   shockwaves, glitch, VHS, or your own.
+- **[Snapshots](#snapshots).** Any node as an image, pixel diffs between versions, and a
+  filmstrip of its history.
+- **[Agent view](#agent-view).** What is on the canvas as a short JSON tree with ids, and an
+  image of any region, for an AI agent to read and point at.
+- **[Transitions](#transitions).** Dissolve, wipe or fade a node out and another in, on the
+  GPU, while both stay live.
+- **[Marks](#marks).** Arrows between elements, comment pins, notes and live cursors that
+  follow the camera.
+
+![An agent cursor flags a heading, links two buttons and drops a pin; the screen wipes out and dissolves back; a glass lens follows the pointer; shockwaves ripple from clicks](docs/features.gif)
+
+Marks from an agent, a wipe and a dissolve, the glass lens and shockwaves, all on live
+components. Recorded in Chrome from the demo.
 
 ## Try it
 
 Open **[berkantay.github.io/gpucanvas](https://berkantay.github.io/gpucanvas/)** in Chrome 157 or
 later. Thirty agent cursors build the first view, then the camera pulls out to all 5,000 screens.
 
-- Click a screen to select it, double-click to go deeper, drag to move it.
+- Click a screen to select it, double-click to go deeper, drag to move it. Double-click a text
+  field to type in it.
 - Pick a shader effect in the toolbar, or open one directly:
   [glass](https://berkantay.github.io/gpucanvas/?effect=glass),
   [shockwave](https://berkantay.github.io/gpucanvas/?effect=shockwave) (click anywhere),
   [frost](https://berkantay.github.io/gpucanvas/?effect=frost),
   [glitch](https://berkantay.github.io/gpucanvas/?effect=glitch).
+- Select a screen and press `D` to dissolve it, `W` to wipe it, or `S` to save it as a PNG.
+- In the console, `agentView.describe()` lists what is in view, and `marks` draws on it.
 - Press `b` to run the benchmark tour.
 
 The GPU path needs HTML-in-Canvas, which is behind a flag:
@@ -36,7 +53,7 @@ The GPU path needs HTML-in-Canvas, which is behind a flag:
 
 Without the flag the demo runs the DOM fallback with 1,000 screens and says so.
 
-Status: early (0.3). The API can still change.
+Status: early (0.4). The API can still change.
 
 ```sh
 npm install gpucanvas
@@ -84,10 +101,10 @@ so every double-click visibly changes something.
 
 | Input              | Does                                                                         |
 | ------------------ | ---------------------------------------------------------------------------- |
-| Double-click       | One layer deeper                                                             |
+| Double-click       | One layer deeper; on a text field, starts typing in it                       |
 | Enter, Shift+Enter | First child, parent                                                          |
 | Tab, Shift+Tab     | Next and previous sibling                                                    |
-| Escape             | Cancels a drag, then goes up a layer, then clears                            |
+| Escape             | Ends typing, cancels a drag, then goes up a layer, then clears               |
 | Drag               | Moves, snapping to the edges and centers of neighbors (Shift locks the axis) |
 | Handles            | Resize (Shift on a corner keeps the ratio)                                   |
 | Drag on open space | A box that selects the layers it touches, inside the layer you are in        |
@@ -149,8 +166,9 @@ Built in:
   `dither` (a lens of big dithered pixels).
 - On each press: `shockwave`, a ring that bends the node as it spreads.
 - Running on their own: `glitch`, `vhs`, `glow`, `hue`, `ripple`.
-- Still: `blur`, `pixelate`, `grayscale`, and `dissolve` for a node leaving or arriving (its
-  parameter is the progress, 0 to 1).
+- Still: `blur`, `pixelate`, `grayscale`.
+- [Transitions](#transitions): `dissolve`, `wipe` and `fade`, for a node leaving or arriving
+  (the first parameter is the progress, 0 to 1).
 
 Animated effects draw every frame while their node is in view; still ones cost nothing once
 drawn.
@@ -184,6 +202,96 @@ the same.
 The DOM backend shows effects that have a `css` look as a CSS `filter` (animated ones as filter
 animations) and the rest as plain nodes. The demo's toolbar puts one effect on every component.
 
+## Snapshots
+
+Every node is already drawn by the GPU, so its pixels are one call away, with no headless
+browser:
+
+```ts
+import { diffImages, recordFilmstrip, toBlob } from 'gpucanvas';
+
+const before = await canvas.snapshot(node, { scale: 2 }); // ImageData, drawn fresh
+const png = await toBlob(before); // a PNG Blob
+
+// ...the screen changes...
+const after = await canvas.snapshot(node, { scale: 2 });
+const { changed, bounds, image } = diffImages(before, after); // what moved, and where
+
+// A snapshot after every change, once it settles.
+const strip = recordFilmstrip(canvas, node, { scale: 0.25, max: 30 });
+strip.frames; // [{ at, image }, ...]
+strip.stop();
+```
+
+A snapshot is the node as it looks now, at any scale, whether or not it is in view. It takes a
+paint or two, about 40 ms for a 1,500 by 1,000 pixel screen. In the demo, select a screen and
+press `S` to save it. Snapshots need the GPU backend; the DOM backend rejects.
+
+## Agent view
+
+An agent reads the canvas the way a person sees it: what is there, what it says, and where:
+
+```ts
+import { createAgentView } from 'gpucanvas';
+
+const view = createAgentView(canvas);
+view.describe(); // what is in view, as JSON
+// [{ id: 'e35', tag: 'div', rect: { x: -3552, y: -2561, width: 1496, height: 958 }, children: [
+//   { id: 'e6', tag: 'h1', role: 'heading', text: 'Welcome back', rect: { ... } },
+//   { id: 'e18', tag: 'input', role: 'textbox', name: 'Email', rect: { ... } },
+//   { id: 'e23', tag: 'input', role: 'checkbox', name: 'Keep me signed in', checked: true, ... },
+//   { id: 'e25', tag: 'button', role: 'button', text: 'Sign in', rect: { ... } }, ...] }]
+
+view.element('e25'); // the element behind an id
+await view.capture('e6', { scale: 2 }); // ImageData of that element
+await view.capture({ x: 0, y: 0, width: 4000, height: 3000 }, { scale: 0.25 }); // any region
+```
+
+Wrappers that add nothing are left out, so a sign-in screen is about 35 entries and 3.5 KB of
+JSON, read in 3 ms. Pass `rect` to read another region and `name` to label elements with your
+own names, such as their React component's. `capture` needs the GPU backend.
+
+## Transitions
+
+Any effect whose first parameter is progress (0 whole, 1 gone) animates as a transition:
+`dissolve`, `wipe` and `fade` are built in.
+
+```ts
+import { animateEffect, effects, swapNodes } from 'gpucanvas';
+
+await animateEffect(canvas, node, effects.wipe, { duration: 600, params: [90] }); // wipe downward
+await animateEffect(canvas, node, effects.wipe, { from: 1, to: 0 }); // and back
+
+// One screen burns away while the next appears in its place.
+await swapNodes(canvas, oldScreen, newScreen, { effect: effects.dissolve, duration: 800 });
+canvas.removeNode(oldScreen);
+```
+
+A node that ends whole loses the effect; one that ends gone stays hidden until it is removed.
+Starting another animation on a node replaces the running one, which resolves `false`. In the
+demo, select a screen and press `D` to dissolve it or `W` to wipe it. The DOM backend fades.
+
+## Marks
+
+Draw on top of the nodes. Marks follow the camera and the elements they are attached to, and
+stay the same size on screen at any zoom:
+
+```ts
+import { createMarks } from 'gpucanvas';
+
+const marks = createMarks(canvas);
+marks.connect(button, nextScreen, { label: 'goes to' }); // a curved arrow
+marks.pin(card, { label: '3', text: 'Rename to Continue?', onClick: openThread });
+marks.note(view.element('e6'), 'This heading is 2px off the grid'); // what an agent means
+const cursor = marks.cursor('Ada', { color: '#e11d48' });
+cursor.move({ x: 120, y: 80 }); // glides there, in world units
+marks.clear();
+```
+
+Arrows and pins also take a world point in place of an element. Marks are SVG and HTML in a
+layer above the canvas, so they work with both backends. In the demo, both are on `window`:
+`agentView` and `marks`.
+
 ## Run the demo
 
 ```sh
@@ -227,7 +335,7 @@ to npm from GitHub Actions, with provenance, through npm trusted publishing. See
 The demo's benchmark (press `b`) runs a 30-second tour: a message typed and sent, a chart
 animating, a push to 500% zoom, shift-click picks and a selection sweep, while 30 agent cursors
 build screens around the view. Every frame is timed. At 5,000 screens it measures 113 to 115 fps
-for gpucanvas and 18 to 20 for DOM.
+for gpucanvas and 18 to 20 for DOM, and 138 for gpucanvas in the effects run below.
 
 The table below was measured with the 0.1 benchmark, a 94-second session from the overview into
 clusters of app screens, close-ups up to 500% zoom, 20 clicks and typed messages, then back out.
@@ -239,6 +347,20 @@ clusters of app screens, close-ups up to 500% zoom, 20 clicks and typed messages
 
 DOM is the same screens in one CSS-transformed container, which is also the fallback backend.
 
+Shader effects on every one of the 5,000 screens, the same 30-second tour, vsync off:
+
+| Effect         | gpucanvas avg fps | p95     |
+| -------------- | ----------------- | ------- |
+| none           | 138               | 15.2 ms |
+| glow           | 135               | 15.6 ms |
+| blur (32 taps) | 138               | 16.7 ms |
+| glass          | 122               | 18.2 ms |
+| frost          | 146               | 13.8 ms |
+
+Effects draw from each node's snapshot in the same pass, so most cost nothing measurable; glass,
+which samples around the pointer with a rim and shadow, costs about 12%. Each figure is one run,
+so a gap of a few percent, like frost above none, is noise rather than a speedup.
+
 ## Chrome 157 findings (differ from the WICG explainer)
 
 - **Hit-test registration.** A drawable is hit-tested only after
@@ -262,8 +384,6 @@ DOM is the same screens in one CSS-transformed container, which is also the fall
 
 ## Next
 
-- **Snapshots as data.** Every node is already a texture: export it as an image without a
-  headless browser, diff two versions of a screen on the GPU, keep a filmstrip of its history.
 - **Keyboard and accessibility.** Parked pages use `content-visibility: hidden`, so Tab skips
   their nodes. A focused node already stays live; moving focus across pages needs a focus model.
 - **Workers.** Composite in a worker with `captureElementImage`, and drive culling from the GPU.