gpucanvas 0.1.0 → 0.3.0

package/README.md

@@ -4,9 +4,21 @@ Put thousands of live DOM nodes on an infinite canvas and let the GPU pan and zo
 
 ![The same 5,000 live React screens as plain DOM (left) and in gpucanvas (right), with each side's frame rate measured live](docs/benchmark.gif)
 
-The same session on both sides: 5,000 live React screens, the same camera path, a message typed and sent. Left, every screen is plain DOM under a CSS transform. Right, gpucanvas. Recorded in Chrome on a MacBook Pro; the frame rates are measured live in each run. Run it yourself with the demo below.
+The same session on both sides: 5,000 live React screens, the same camera path, a message typed and sent. Left, every screen is plain DOM under a CSS transform. Right, gpucanvas. Recorded in Chrome on a MacBook Pro; the frame rates are measured live in each run. Run it yourself in the [live demo](https://berkantay.github.io/gpucanvas/).
 
-Status: early (0.1). The API can still change.
+## Try it
+
+Open **[berkantay.github.io/gpucanvas](https://berkantay.github.io/gpucanvas/)** in Chrome 157 or later. Thirty agent cursors build the first view, then the camera pulls out to all 5,000 screens. Press `b` to run the benchmark tour.
+
+The GPU path needs HTML-in-Canvas, which is behind a flag:
+
+1. Open `chrome://flags/#canvas-draw-element`
+2. Set it to Enabled and relaunch Chrome
+3. Reload the demo
+
+Without the flag the demo runs the DOM fallback with 1,000 screens and says so.
+
+Status: early (0.3). The API can still change.
 
 ```sh
 npm install gpucanvas
@@ -29,6 +41,129 @@ HTML-in-Canvas are available (Chrome 157+), nodes are snapshotted into a GPU atl
 one instanced draw call. Everywhere else it falls back to a CSS-transformed container with the
 same API.
 
+## Editing
+
+`bindEditor` turns the canvas into an editor: select, drill in, move and resize. The GPU draws
+the outlines, 2 CSS pixels wide at any zoom, and they follow their element's layout. Names,
+sizes, handles and snapping guides sit in a thin overlay above it.
+
+```ts
+import { bindEditor } from 'gpucanvas';
+
+const editor = bindEditor(canvas, {
+  pick: 'element', // 'node' (default), 'element', or a function
+  trigger: 'click', // an edit mode; the default 'shift-click' leaves clicks to your components
+  onChange: (layers) => console.log(layers),
+  onTransform: ({ phase, layers, to }) => {
+    // The editor never moves anything itself. Apply `to` with updateNode, state or styles.
+  },
+});
+```
+
+A click selects the outermost layer under the pointer, a whole node. Each double-click goes one
+layer deeper, and later clicks stay at that depth. Wrappers the size of their parent are skipped,
+so every double-click visibly changes something.
+
+| Input              | Does                                                                         |
+| ------------------ | ---------------------------------------------------------------------------- |
+| Double-click       | One layer deeper                                                             |
+| Enter, Shift+Enter | First child, parent                                                          |
+| Tab, Shift+Tab     | Next and previous sibling                                                    |
+| Escape             | Cancels a drag, then goes up a layer, then clears                            |
+| Drag               | Moves, snapping to the edges and centers of neighbors (Shift locks the axis) |
+| Handles            | Resize (Shift on a corner keeps the ratio)                                   |
+| Drag on open space | A box that selects the layers it touches, inside the layer you are in        |
+
+With the default `trigger: 'shift-click'`, the same works with Shift held, and clicks still reach
+your components. A layer is an array of elements, so one layer can be a component that renders
+several. `pick` decides the layers:
+
+- `'node'`: whole nodes only.
+- `'element'`: every element from the node down to the one under the pointer.
+- a function of that element, returning the layers that hold it, outermost first.
+
+In React, `useEditor` gives the selection as state, and `pickComponents` makes the layers React
+components:
+
+```tsx
+import { labelComponent, pickComponents, useEditor } from 'gpucanvas/react';
+
+// Inside <GpuCanvas>:
+const { selected, clear } = useEditor({ pick: pickComponents, label: labelComponent });
+// Click a screen, double-click: SignIn, SignInCard, Button.
+```
+
+`pickComponents` and `componentOf` read React's internal fibers, as click-to-component tools do.
+Production builds usually minify component names, so mark components with your own attributes
+when you need names there.
+
+The building blocks, for your own interactions:
+
+```ts
+canvas.getElementAt(point); // deepest element at a world point
+canvas.getElementRect(el); // any element's rect in world units
+canvas.getNodeOf(el); // the node that holds it
+canvas.getNodesIn(rect); // nodes overlapping a world rect
+canvas.setSelection(elements); // outline these; [] clears
+canvas.setHover(el); // a lighter outline; null clears
+canvas.screenToWorld(point); // surface pixels to world units, and worldToScreen back
+```
+
+Colors are the `selectionColor` and `hoverColor` options. The demo picks React components in the edit mode;
+`?pick=element` or `?pick=node` picks elements or whole screens, and `?edit=0` leaves clicks to
+the components.
+
+## Shader effects
+
+Components stay live DOM, and the GPU draws them through a shader. One call per node:
+
+```ts
+import { effects } from 'gpucanvas';
+
+canvas.setEffect(node, effects.ripple); // the effect's default parameters
+canvas.setEffect(node, effects.glow, [1, 0.4, 0.2, 32]); // orange, 32 px reach
+canvas.setEffect(node, null); // back to plain
+```
+
+Built in:
+
+- Following the pointer: `glass` (a magnifying lens), `frost` (ice, clear around the pointer),
+  `dither` (a lens of big dithered pixels).
+- On each press: `shockwave`, a ring that bends the node as it spreads.
+- Running on their own: `glitch`, `vhs`, `glow`, `hue`, `ripple`.
+- Still: `blur`, `pixelate`, `grayscale`, and `dissolve` for a node leaving or arriving (its
+  parameter is the progress, 0 to 1).
+
+Animated effects draw every frame while their node is in view; still ones cost nothing once
+drawn.
+
+Your own is a WGSL function over the node's pixels:
+
+```ts
+import { defineEffect } from 'gpucanvas';
+
+const scanlines = defineEffect({
+  name: 'scanlines',
+  animated: true,
+  params: [3], // line spacing in CSS pixels
+  wgsl: /* wgsl */ `
+    fn effect(input: EffectInput) -> vec4f {
+      let color = source(input.uv);
+      let y = input.uv.y * input.size.y + input.time * 20.0;
+      return color * (0.85 + 0.15 * step(0.5, fract(y / input.params.x)));
+    }`,
+});
+```
+
+`EffectInput` has `uv` (0 to 1 across the node), `size` in CSS pixels, `time` in seconds,
+`zoom`, the four `params`, the `pointer` and the last `press` in the node's uv, and
+`sincePress` in seconds. `source(uv)` reads the node's pixels, premultiplied, and is
+transparent outside it; `hash` and `noise` give random values. An effect that draws past its node, like a glow, declares how far with
+`pad`. Effects read the node's whole snapshot, so very close in they are softer than plain nodes.
+
+The DOM backend shows effects that have a `css` look as a CSS `filter` (animated ones as filter
+animations) and the rest as plain nodes. The demo's toolbar puts one effect on every component.
+
 ## Run the demo
 
 ```sh
@@ -38,32 +173,44 @@ bun run dev        # http://localhost:5199/?mode=gpu&n=10000  (mode: auto | gpu
 
 ## How it works
 
-1. Nodes live inside `<canvas content="drawable">`.
-2. A node that needs pixels is promoted (`drawable` attribute). In the next paint event its
-   snapshot is rasterized with `queue.drawElementImageToTexture` into a texture-array atlas, at a
-   power-of-two scale of the zoom.
-3. After that the node is parked with `display: none`. Every drawable costs Chrome layout and a
-   recording on each rendering update, so only nodes waiting for a raster stay drawable.
-4. A MutationObserver marks a node dirty when its content changes, which re-promotes it.
-5. Each frame culls nodes against the view and draws every visible one with one instanced quad
-   draw. Nodes without a snapshot yet draw as placeholders.
-6. While the camera moves, only missing snapshots are rasterized. Blurry ones are upgraded once
-   it rests, center first. Zooming out keeps the sharper snapshots.
-7. Only the node under the pointer is live (hit-testable). Pans and zooms cost no DOM work.
-
-## Measured (M-series Mac, Chrome 157, DPR 2, 120 Hz)
-
-The 6-second scripted path runs overview, then 100% zoom, then a pan, then overview again.
-
-| Nodes  | gpu avg fps | gpu p95 | dom avg fps | dom p95  | mount gpu / dom |
-| ------ | ----------- | ------- | ----------- | -------- | --------------- |
-| 1,000  | 120         | 10.1 ms | 120         | 10.1 ms  | 1 s / 1 s       |
-| 10,000 | 98          | 16.7 ms | 85          | 17.0 ms  | 1 s / 1.3 s     |
-| 50,000 | 64          | 33.4 ms | 7           | 189.3 ms | 7 s / 13 s      |
-
-- At 50,000 nodes the overview shows placeholders: nodes under 6 screen pixels wide are not
-  rasterized.
-- At 10,000 nodes the overview takes about 6 s to fill.
+1. **Pages.** Nodes are live DOM inside `<canvas content="drawable">`, packed side by side into
+   pages of 2048 CSS pixels. Chrome records a page as one snapshot, so one
+   `drawElementImageToTexture` call rasterizes dozens of nodes at once.
+2. **Rasterize, then park.** A page with nodes waiting for pixels becomes `drawable`. In the next
+   paint event it is drawn into a texture-array atlas, whole or one node at a time when only a
+   few changed. Then it is parked with `content-visibility: hidden`, which keeps layout but skips
+   paint, so the browser spends nothing on it per frame.
+3. **Thumbnails first.** Every node gets a 1/16 scale thumbnail in an atlas layer that is never
+   evicted, so a zoomed-out view of thousands of screens is always filled in. Sharper snapshots
+   go on top at power-of-two scales, so zooming reuses them.
+4. **Tiles up close.** Past 100% zoom on a 2x display, only the visible part of a node is
+   rasterized, as sharp tiles, center first. While its content animates, one half-resolution preview stands in.
+5. **One draw call.** Each frame culls nodes against the view and draws every visible one with
+   one instanced quad draw. Panning and zooming never touch the DOM. Raster work has a
+   per-frame budget, so the camera stays smooth while pixels fill in.
+6. **Live updates.** A MutationObserver marks a node dirty when its content changes, and
+   loading images, web fonts and CSS transitions trigger a redraw. Animated content on nodes
+   under 200 screen pixels refreshes at most four times a second.
+7. **Input.** Only the page under the pointer is registered for hit testing, and a CSS transform
+   puts the node under the pointer exactly where the GPU drew it. Clicks, typing, selection and
+   hover all go to the real element. A focused node keeps its page live.
+
+## Measured (M-series Mac, Chrome 157, 1600 x 900 window, DPR 1)
+
+The demo's benchmark (press `b`) runs a 30-second tour: a message typed and sent, a chart
+animating, a push to 500% zoom, shift-click picks and a selection sweep, while 30 agent cursors
+build screens around the view. Every frame is timed. At 5,000 screens it measures 113 to 115 fps
+for gpucanvas and 18 to 20 for DOM.
+
+The table below was measured with the 0.1 benchmark, a 94-second session from the overview into
+clusters of app screens, close-ups up to 500% zoom, 20 clicks and typed messages, then back out.
+
+| Screens | gpucanvas avg fps | gpucanvas p95 | DOM avg fps | DOM p95  | Mount gpucanvas / DOM |
+| ------- | ----------------- | ------------- | ----------- | -------- | --------------------- |
+| 5,000   | 117               | 9.2 ms        | 23          | 50.2 ms  | 1.2 s / 1.3 s         |
+| 10,000  | 108               | 16.7 ms       | 9           | 133.8 ms | 1.8 s / 3.4 s         |
+
+DOM is the same screens in one CSS-transformed container, which is also the fallback backend.
 
 ## Chrome 157 findings (differ from the WICG explainer)
 
@@ -78,13 +225,16 @@ The 6-second scripted path runs overview, then 100% zoom, then a pan, then overv
 - **Newly drawable elements.** An element that just became drawable is not reported in
   `changedElements`. Its snapshot is available from the next paint event.
 - **Per-call cost.** Each `drawElementImageToTexture` call allocates and fully rasterizes, about
-  250 to 550 calls per second in total. This, not compositing, limits the fill rate.
+  250 to 550 calls per second in total. This, not compositing, limits the fill rate, which is
+  why nodes are batched into pages.
+- **Cropping.** A source rectangle still rasterizes the whole source element at the requested
+  scale before cropping. Zoomed in, a node therefore moves to a page of its own.
 - **Drawable count.** Recording cost scales with the number of drawables (and non-`display:none`
   canvas children) on every `requestPaint`. 10,000 always-drawable nodes run at 22 fps even when
   idle.
 
 ## Next
 
-- **Keyboard and accessibility.** Parked nodes are `display: none`, so Tab skips them. Needs a
-  focus model, such as a mirror or pinning the focused node.
+- **Keyboard and accessibility.** Parked pages use `content-visibility: hidden`, so Tab skips
+  their nodes. A focused node already stays live; moving focus across pages needs a focus model.
 - **Workers.** Composite in a worker with `captureElementImage`, and drive culling from the GPU.