webview-preview 1.1.2 → 1.2.0

package/AGENTS.md

@@ -66,6 +66,30 @@ element.getClientRects().length > 0
 If the project's config has macros (`macros` in `webview-preview.config.mjs` — signing in, opening a dialog), use them
 (`macro:signIn`): they are already correct. A flow you repeat more than twice belongs there too.
 
+## When the user wants to show you what to change
+
+If the user says "I'll show you" or points at UI in words that do not pin an element down, let them point instead:
+
+```
+npx webview-preview /screen <steps to reach it> annotate:review
+```
+
+- It opens a **real window** and **waits for the person** — there is no run limit. Start it in the background (or
+  tell the user to run it) and wait for the process to end; do not poll the window.
+- Read the stdout summary first: one line per note with the component, `file:line` and the note's text. The full
+  data is in `<prefix>annotate-review.json` next to the snapshots; pictures are `annotate-review.png` (with markers)
+  and `annotate-review-e1.png` … (each element). Open a picture only after the JSON.
+- `component.source` with `confidence: "exact"` is where that element's JSX is written; `"nearest"` is where the
+  component around it is used (text inside a widget) — edit the call site, not the widget, unless the note says the
+  widget itself is wrong; `"file"` (Vue) has no line. Treat all of it as a strong hint and check the code.
+- `selector` works in `measure:` / `snap:@` / `click:` for checking your change in the same state. If
+  `selectorQuality` is `"positional"`, it breaks once the DOM changes — measure before you edit.
+- Notes can come from several pages (`note.url`), and from a frozen state (`note.isFrozen`): that state may not be
+  reachable with steps alone — say so if you cannot reproduce it.
+- `status: "cancelled"` means the person closed it without notes; ask them, do not guess.
+- Private elements come as `•••` with `isPrivate: true`; do not ask the user to remove the masking.
+- Never put `annotate` into CI or a script that nobody watches; it never ends there.
+
 ## Errors in the page
 
 After the steps the tool prints the page's `console.error` calls, uncaught exceptions and unhandled rejections
@@ -244,4 +268,4 @@ host and DPI handling of a native window can differ.
   line. `assert:` shows only the final value; for `a === b` print `a` first if it fails.
 - Keyboard and hover behaviour checked with `key:` / `hover:` where the change touches them.
 - You looked at one snapshot of the final state.
-- You listed what could not be checked: fractional scroll, the other platform's engine, real backend data.
+- You listed what could not be checked: fractional scroll, the other platform's engine, real backend data.

package/README.md

@@ -28,6 +28,7 @@ plain init scripts and fit there too.
 - [Quick start](#quick-start)
 - [Config file](#config-file)
 - [Steps](#steps)
+- [Annotate — show what to change](#annotate--show-what-to-change)
 - [Macros](#macros)
 - [Errors in the page](#errors-in-the-page)
 - [Command line](#command-line)
@@ -239,6 +240,7 @@ Steps run one after another, after the page has loaded and settled for 1.5 s.
 | `film:<name>:<ms>[/<n>][@<selector>]` | Takes n frames (10 by default, up to 60) evenly over ms, starting now, and puts them into one `<prefix><name>.png`, 5 per row, left to right. With a selector every frame is cropped to the area the element covered during the whole film (plus `--snap-margin`). Prints the real time of every frame: `film open: 10 frames at 0 55 111 … 500 ms`. Put it right after the step that starts the animation. |
 | `resize:<W>x<H>` | Resizes the window. |
 | `macro:<name>` | Runs the steps of a macro from the config. See [Macros](#macros). |
+| `annotate` / `annotate:<name>` | Opens a real window with a panel where a person clicks elements and writes what to change, and waits for **Done**. Writes a JSON and a Markdown report with pictures. See [Annotate](#annotate--show-what-to-change). |
 
 ### Selectors
 
@@ -274,6 +276,107 @@ npx webview-preview / \
 
 On Windows PowerShell use the same double quotes; inside them, keep JavaScript strings in single quotes.
 
+## Annotate — show what to change
+
+Describing a change in words ("the button on the right is a bit off") leaves whoever makes it guessing which element
+you mean. `annotate` lets you point at it instead: the run opens a real window, you click elements and write a note
+next to each, press **Done**, and get the exact element, where it comes from in the code and how it looks now.
+
+```sh
+npx webview-preview /settings annotate
+npx webview-preview /auth macro:signIn "click:text=Stats" annotate:stats snap:after
+npx webview-preview https://example.com annotate          # any site works
+```
+
+![Picking an element: the outline shows the element, its parent and its size; the tools are on the left](assets/readme/annotate-pick.png)
+
+The steps before `annotate` reach the screen; the steps after it run once you are done. The step waits for you as
+long as it takes (no run limit unless you pass `--timeout`) and always opens a window — WebKitGTK as with `--headed`,
+the other engines without `headless`.
+
+**The panel** sits in the top-left corner, as a strip of icons; drag it by the dots on top.
+
+| Tool | Key | What it does |
+|---|---|---|
+| Show / hide tools | `T` | Folds the strip down to one icon. |
+| Pick | `P` | Hover outlines an element with its size and its parent (`div.card › span "23" · 253 × 40`); a click picks it. Clicks do not reach the page while Pick is on — turn it off to use the app (open a menu, go to another screen), then on again. |
+| Notes | `N` | The list of notes. It also opens when you pick an element, with the cursor in the note's text. |
+| Freeze | `F` | Counts down 3 s (on the button) — hover what has to stay open meanwhile. Then animations stop and the page no longer hears the mouse leave or focus go (`mouseleave`, `pointerout`, `blur`, `focusout`, mouse moves), so tooltips and menus stay open to be picked. `F` again unfreezes. A menu that closes on a timer or by its own logic can still close. |
+| Key bindings | — | Click an action, press the new key. Saved for the next runs in your user settings (`~/.config/webview-preview/annotate.json` on Linux, `%APPDATA%\webview-preview\` on Windows, `~/Library/Application Support/webview-preview/` on macOS). |
+| Done / Cancel | `Ctrl+Enter` | Done writes the report; Cancel or closing the window ends with nothing. |
+
+- **Click** — a new note; **Shift+click** — adds the element to the open note ("align these two"); clicking an
+  element that already has a note opens that note.
+- **Alt + mouse wheel** or `↑` / `↓` — the parent / back to the child, when the click would land on an inner `<span>`.
+- **Escape** closes the notes card; the notes stay. The **general note** at the bottom is about the whole screen.
+- **Several pages:** go to another screen (turn Pick off, navigate, turn it on) — the notes stay, also when the page
+  reloads. Elements are cut out the moment you pick them; coming back to a page shows their markers again.
+- Keys follow the physical key, so they work in any keyboard layout.
+
+![Two notes: the second one is about two buttons at once and is open in the notes card](assets/readme/annotate-notes.png)
+
+![Freeze: the tooltip that appears on hover stays open, so it can be picked](assets/readme/annotate-freeze.png)
+
+**What you get.** A summary on stdout, one line per note:
+
+```
+annotate stats: done · 2 note(s) · 2 element(s)
+  1. StyledTextWidget span.text "1 284" (src/routes/stats/stats.parts.tsx:22) — make the number smaller
+     e1: [class*="stats__tiles"] > div:nth-child(1) > span:nth-child(2)
+  → .webview-preview/webkitgtk-annotate-stats.json
+  → .webview-preview/webkitgtk-annotate-stats.md
+```
+
+and files in the snapshot folder:
+
+| File | What |
+|---|---|
+| `<prefix>annotate-<name>.png` | The screen where you pressed Done, with the numbered markers. |
+| `<prefix>annotate-<name>-<id>.png` | Every element cut out, without markers. |
+| `<prefix>annotate-<name>.json` | Everything: notes, the page of each note, and for every element its selector, text, size, box, font and colours, and its component. |
+| `<prefix>annotate-<name>.md` | A report for people: paste it into an issue as it is, pictures included. |
+
+![The marked screen from the report: numbered markers on the picked elements, the e-mail painted over](assets/readme/annotate-report.png)
+
+These pictures are not snapshots of the run: `--compare` and `--update-baseline` leave them alone.
+
+**Try it** on the demo app in this repository (a small React page made for these pictures):
+
+```sh
+cd examples/demo && npm install
+npx webview-preview / annotate
+```
+
+**Where an element comes from in the code** — in development builds only:
+
+- **React** (16–19): the component, the chain of components above it, and the file and line where its JSX is written,
+  mapped back through the dev server's source maps. `confidence: "exact"` is the element's own JSX;
+  `"nearest"` is where the component around it is used (for text inside a widget).
+- **Vue** 2 and 3: the component, its chain and its `.vue` file (`confidence: "file"`, no line).
+- Anything else, and production builds: no component, the rest of the report as usual.
+
+Component names and lines come from React's and Vue's internals and are a strong hint, not a promise. The
+`selector` is for this session: `selectorQuality: "positional"` (it has `:nth-child`) breaks when the page changes —
+add a `data-testid` where you need a lasting one.
+
+**Strict sites** (Content Security Policy, Trusted Types — YouTube, for example) work: the panel builds its DOM
+without `innerHTML` and adopts its styles instead of inserting a `<style>`.
+
+**Private data.** The report holds pictures and text from the page, and it may go to someone else or to an AI agent.
+Some elements are hidden by default: password fields, card and one-time-code fields (`autocomplete`), and anything
+inside `[data-private]` or `[data-webview-preview-private]`. Add your own selectors with `--private <selector>`
+(repeatable) or in the config:
+
+```js
+export default { annotate: { private: ["[class*=email]", ".token"] } }
+```
+
+Hidden elements are painted over on screen and in every picture (line by line, so the text around them stays). Their
+text becomes `•••`, also inside a picked parent, and their `aria-label` is left out of names and selectors. They get
+`isPrivate: true`; the report counts them (`maskedCount`), and the panel says how many are hidden. It is a safety
+net, not a guarantee — text drawn on a canvas or an e-mail in a plain `<span>` is not recognised — so prefer test
+accounts and mock data when the report leaves your machine.
+
 ## Macros
 
 Steps you repeat — signing in, opening a dialog — go into the config under a name:
@@ -384,6 +487,7 @@ const result = await preview({
 result.engine; // "webkitgtk"
 result.saved;  // ["/abs/path/.webview-preview/webkitgtk-dark.png"]
 result.failed; // false
+result.annotations; // one report per annotate step (the same object as the .json file), or { status: "cancelled" }
 ```
 
 `preview()` reads the config file the same way the CLI does; pass `config: false` to skip it or `config: "path"` to
@@ -629,13 +733,13 @@ invisible rounding noise; raise it a little if antialiasing differs between mach
 Snapshots pile up in the snapshot folder. Clear them when you no longer need them:
 
 ```sh
-npx webview-preview clean                  # delete every *.png and report.html in the snapshot folder
+npx webview-preview clean                  # delete every *.png, report.html and annotate report in the snapshot folder
 npx webview-preview clean --dry-run        # only list what would be deleted
 npx webview-preview clean --older-than 2h  # only files older than 2 hours (30m, 7d …)
 ```
 
 `clean` reads the same config (`out`, `baseline`) as a normal run. It deletes only `*.png` files directly in the
-snapshot folder (including `.diff.png`) and `report.html`, never the baselines, and refuses to run when the snapshot folder is the
+snapshot folder (including `.diff.png`), `report.html` and the `annotate-*.json` / `.md` reports, never the baselines, and refuses to run when the snapshot folder is the
 project folder, your home folder, the disk root or the baseline folder.
 
 ## Recipes
@@ -755,6 +859,9 @@ The tests open `test/fixture/index.html` and check snapshots, input, focus order
 `film:`, compare and the report. GitHub Actions runs them on every push: WebKitGTK and Chromium on Linux, Edge and
 Chrome on Windows, WebKit on macOS; snapshots and reports of every run are kept as artifacts.
 
+The pictures in this README come from `examples/demo`: `node examples/demo/readme-images.mjs` takes them again
+(after `npm install` in `examples/demo`).
+
 ## License
 
-MIT © n0t4u404
+MIT © n0t4u404. The icons of the annotate panel are from [Lucide](https://lucide.dev) (ISC).

package/bin/webview-preview.mjs

@@ -24,7 +24,7 @@ const FLAGS = {
 	"--color-scheme": "colorScheme",
 	"--latency": "latency"
 }
-const LIST_FLAGS = { "--init": "init", "--init-file": "initFile", "--mock-file": "mockFile" }
+const LIST_FLAGS = { "--init": "init", "--init-file": "initFile", "--mock-file": "mockFile", "--private": "private" }
 
 const HELP = `Usage: webview-preview [route-or-url] [steps...] [options]
        webview-preview clean [--dry-run] [--older-than 2h] [--out <dir>] [--config <file>]
@@ -79,6 +79,7 @@ Options:
   --base-url <url> prepended to routes that are not full URLs
   --init <code>    run code before the page's own scripts, on every load (repeatable)
   --init-file <f>  same, from a file (repeatable)
+  --private <css>  annotate: hide these elements' text and pixels in the report too (repeatable)
   --mock-file <f>  answer fetch/XHR requests from a .json/.mjs file of mocks (repeatable);
                    every request is listed in window.__webviewPreviewRequests
   --log-requests   only list fetch/XHR requests in window.__webviewPreviewRequests

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "webview-preview",
-	"version": "1.1.2",
+	"version": "1.2.0",
 	"description": "Open your UI in the engine your desktop app really runs on (WebKitGTK, Edge/WebView2, WebKit), drive it with steps and save PNG snapshots.",
 	"license": "MIT",
 	"author": "n0t4u404",

package/src/core/annotate-report.mjs

@@ -0,0 +1,66 @@
+import { writeFileSync } from "node:fs";
+import path from "node:path";
+
+const routeOf = (url, baseUrl) => (baseUrl && url.startsWith(baseUrl) ? url.slice(baseUrl.length) || "/" : url);
+
+const localTime = date => {
+	const pad = value => String(value).padStart(2, "0");
+	return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())} ${pad(date.getHours())}:${pad(date.getMinutes())}`;
+}
+
+const sourceText = source => (source ? `${source.file}${source.line ? `:${source.line}` : ""}` : null);
+
+const markdown = report => {
+	const lines = [`# annotate: ${report.name}`, ""];
+	const { page } = report;
+	lines.push(`${page.route} · ${page.engine} · ${page.size.width}×${page.size.height} · ${localTime(new Date(page.takenAt))}`, "");
+	if (report.image) lines.push(`![${report.name}](${report.image})`, "");
+	for (const note of report.notes) {
+		lines.push(`## ${note.id}. ${note.text || "(no text)"}`, "");
+		if (note.url && note.url !== page.url) lines.push(`On another page: \`${routeOf(note.url, report.baseUrl)}\``, "");
+		for (const id of note.elements) {
+			const element = report.elements[id];
+			if (!element) continue;
+			const where = sourceText(element.component?.source);
+			const component = element.component?.name ? `\`${element.component.name}\` ` : "";
+			lines.push(`- ${component}${element.label}${where ? ` — \`${where}\`` : ""}`);
+			lines.push(`  - selector: \`${element.selector}\``);
+			if (element.image) lines.push("", `  ![${id}](${element.image})`, "");
+		}
+		lines.push("");
+	}
+	if (report.generalNote) lines.push("## General note", "", report.generalNote, "");
+	return lines.join("\n").replace(/\n{3,}/g, "\n\n").trimEnd();
+}
+
+export const writeAnnotationReport = ({ annotation, out, prefix, engine, url, baseUrl, size }) => {
+	const { result } = annotation;
+	const report = {
+		version: 1,
+		baseUrl,
+		name: annotation.name,
+		status: annotation.status,
+		page: {
+			url: result?.url ?? url,
+			route: routeOf(result?.url ?? url, baseUrl),
+			engine,
+			size: result?.viewport ?? size,
+			scale: annotation.scale ?? 1,
+			takenAt: new Date().toISOString()
+		},
+		image: annotation.image ?? null,
+		notes: result?.notes ?? [],
+		generalNote: result?.generalNote ?? null,
+		maskedCount: result?.maskedCount ?? 0,
+		elements: result?.elements ?? {}
+	}
+	if (result) {
+		delete result.url;
+		delete result.viewport;
+	}
+	const base = path.join(out, `${prefix}annotate-${annotation.name}`);
+	writeFileSync(`${base}.md`, markdown(report));
+	delete report.baseUrl;
+	writeFileSync(`${base}.json`, JSON.stringify(report, null, "\t"));
+	return { json: `${base}.json`, markdown: `${base}.md`, report }
+}

package/src/core/annotate-settings.mjs

@@ -0,0 +1,30 @@
+import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+
+const FOLDER = "webview-preview";
+const FILE = "annotate.json";
+
+export const settingsFile = () => {
+	if (process.env.WEBVIEW_PREVIEW_SETTINGS) return process.env.WEBVIEW_PREVIEW_SETTINGS;
+	if (process.platform === "linux") return path.join(process.env.XDG_CONFIG_HOME || path.join(os.homedir(), ".config"), FOLDER, FILE);
+	if (process.platform === "win32") return path.join(process.env.APPDATA || path.join(os.homedir(), "AppData", "Roaming"), FOLDER, FILE);
+	if (process.platform === "darwin") return path.join(os.homedir(), "Library", "Application Support", FOLDER, FILE);
+	return path.join(os.homedir(), `.${FOLDER}`, FILE);
+}
+
+export const loadAnnotateKeys = () => {
+	try {
+		const { keys } = JSON.parse(readFileSync(settingsFile(), "utf8"));
+		return keys && typeof keys === "object" ? keys : null;
+	} catch {
+		return null;
+	}
+}
+
+export const saveAnnotateKeys = keys => {
+	const file = settingsFile();
+	mkdirSync(path.dirname(file), { recursive: true });
+	writeFileSync(file, JSON.stringify({ keys }, null, "\t"));
+	return file;
+}

package/src/core/source-map.mjs

@@ -0,0 +1,140 @@
+import { existsSync, readdirSync } from "node:fs";
+import { readFile } from "node:fs/promises";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const BASE64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+const BASE64_VALUES = new Map([...BASE64].map((character, index) => [character, index]));
+const SOURCE_MAP_COMMENT = /\/\/[#@]\s*sourceMappingURL=(\S+)\s*$/m;
+
+export const decodeVlq = segment => {
+	const values = [];
+	let value = 0;
+	let shift = 0;
+	for (const character of segment) {
+		const digit = BASE64_VALUES.get(character);
+		if (digit === undefined) throw new Error(`bad source map character "${character}"`);
+		value += (digit & 31) << shift;
+		if (digit & 32) {
+			shift += 5;
+			continue;
+		}
+		values.push(value & 1 ? -(value >>> 1) : value >>> 1);
+		value = 0;
+		shift = 0;
+	}
+	return values;
+}
+
+export const originalPosition = (map, line, column) => {
+	const lines = map.mappings.split(";");
+	if (line < 1 || line > lines.length) return null;
+	let sourceIndex = 0;
+	let sourceLine = 0;
+	let sourceColumn = 0;
+	let best = null;
+	for (let index = 0; index < line; index += 1) {
+		let generatedColumn = 0;
+		for (const segment of lines[index].split(",")) {
+			if (!segment) continue;
+			const [columnDelta, sourceDelta, lineDelta, sourceColumnDelta] = decodeVlq(segment);
+			generatedColumn += columnDelta;
+			if (sourceDelta === undefined) continue;
+			sourceIndex += sourceDelta;
+			sourceLine += lineDelta;
+			sourceColumn += sourceColumnDelta;
+			if (index === line - 1 && generatedColumn <= column - 1) {
+				best = { source: map.sources[sourceIndex], line: sourceLine + 1, column: sourceColumn + 1 }
+			}
+		}
+	}
+	return best;
+}
+
+const readText = async url => {
+	if (url.startsWith("file:")) return readFile(fileURLToPath(url), "utf8");
+	const response = await fetch(url);
+	if (!response.ok) throw new Error(`${response.status}`);
+	return response.text();
+}
+
+const loadMap = async moduleUrl => {
+	const code = await readText(moduleUrl);
+	const reference = SOURCE_MAP_COMMENT.exec(code)?.[1];
+	if (!reference) return null;
+	if (reference.startsWith("data:")) {
+		const comma = reference.indexOf(",");
+		const isBase64 = reference.slice(0, comma).endsWith(";base64");
+		const body = reference.slice(comma + 1);
+		return JSON.parse(isBase64 ? Buffer.from(body, "base64").toString("utf8") : decodeURIComponent(body));
+	}
+	return JSON.parse(await readText(new URL(reference, moduleUrl).href));
+}
+
+const projectPath = (file, baseDir) => {
+	if (!path.isAbsolute(file)) return file.replaceAll("\\", "/");
+	const relative = path.relative(baseDir, path.resolve(file));
+	const isInside = relative !== "" && !relative.startsWith("..") && !path.isAbsolute(relative);
+	return (isInside ? relative : file).replaceAll("\\", "/");
+}
+
+const SKIPPED_FOLDERS = new Set(["node_modules", ".git", "dist", "build", "target", "out"]);
+
+const projectRoots = baseDir => {
+	const roots = [baseDir];
+	const walk = (folder, depth) => {
+		if (depth === 0) return;
+		let entries = [];
+		try {
+			entries = readdirSync(folder, { withFileTypes: true });
+		} catch {
+			return;
+		}
+		for (const entry of entries) {
+			if (!entry.isDirectory() || entry.name.startsWith(".") || SKIPPED_FOLDERS.has(entry.name)) continue;
+			const child = path.join(folder, entry.name);
+			roots.push(child);
+			walk(child, depth - 1);
+		}
+	}
+	walk(baseDir, 2);
+	return roots;
+}
+
+const onDisk = (file, roots) => {
+	if (existsSync(file)) return file;
+	return roots.map(root => path.join(root, file)).find(candidate => existsSync(candidate)) ?? file;
+}
+
+const sourcePath = (map, source, moduleUrl, baseDir, roots) => {
+	const root = map.sourceRoot ? new URL(map.sourceRoot, moduleUrl).href : moduleUrl;
+	let file = source;
+	if (/^[a-z][\w+.-]*:/i.test(source)) file = source.startsWith("file:") ? fileURLToPath(source) : source;
+	else if (!path.isAbsolute(source)) {
+		const resolved = new URL(source, root);
+		file = resolved.protocol === "file:" ? fileURLToPath(resolved) : decodeURIComponent(resolved.pathname);
+	}
+	if (path.isAbsolute(file)) file = onDisk(file, roots());
+	return projectPath(file, baseDir);
+}
+
+export const createSourceResolver = baseDir => {
+	const maps = new Map();
+	let roots = null;
+	const getRoots = () => roots ??= projectRoots(baseDir);
+	const mapFor = url => {
+		if (!maps.has(url)) maps.set(url, loadMap(url).catch(() => null));
+		return maps.get(url);
+	}
+	return async frame => {
+		if (!frame) return null;
+		if (frame.file) {
+			const file = path.isAbsolute(frame.file) ? onDisk(frame.file, getRoots()) : frame.file;
+			return { file: projectPath(file, baseDir), line: frame.line, column: frame.column, how: frame.how ?? "debug-source" }
+		}
+		const map = await mapFor(frame.url);
+		const position = map && originalPosition(map, frame.line, frame.column);
+		if (!position) return { file: frame.url, line: frame.line, column: frame.column, how: "debug-stack", isMapped: false }
+		return { file: sourcePath(map, position.source, frame.url, baseDir, getRoots), line: position.line, column: position.column, how: "debug-stack" }
+	}
+}

package/src/core/steps.mjs

@@ -16,7 +16,7 @@ export const runTimeoutMs = (steps, waitForTimeoutMs) => TIMEOUT_MS + steps.redu
 					: STEP_BUDGET_MS
 ), 0);
 
-const STEP_KINDS = new Set(["js", "jsnow", "wait", "waitfor", "snap", "print", "measure", "assert", "focusorder", "resize", "key", "type", "hover", "click", "offline", "online", "latency", "film"]);
+const STEP_KINDS = new Set(["js", "jsnow", "wait", "waitfor", "snap", "print", "measure", "assert", "focusorder", "resize", "key", "type", "hover", "click", "offline", "online", "latency", "film", "annotate"]);
 export const ASSERT_MARKER = "__webviewPreviewAssert:";
 export const FOCUS_MARKER = "__webviewPreviewFocus:";
 const FOCUS_ORDER_DEFAULT = 20;
@@ -235,6 +235,11 @@ export const parseSteps = steps => steps.flatMap(step => {
 		return { kind, value, check: waitForExpression(selector, isGone) }
 	}
 	if (kind === "focusorder") return focusOrderSteps(value, step);
+	if (kind === "annotate") {
+		const name = value || "annotate";
+		if (!/^[\w.-]+$/.test(name)) throw new Error(`annotate needs a name of letters, digits, "-", "_" or ".", got "${step}"`);
+		return { kind, value: name }
+	}
 	if (kind === "film") return filmStep(value, step);
 	if (NETWORK_STEPS.has(kind)) return networkStep(kind, value, step);
 	if (kind === "assert") {

package/src/engines/playwright.mjs

@@ -19,14 +19,15 @@ const loadPlaywright = async () => {
 export const runPlaywright = async ({
 	engine, url, steps, init = [], size, locale,
 	colorScheme, isMotionReduced = false, out, prefix, scale,
-	snapMargin = 8, waitForTimeoutMs = WAIT_FOR_TIMEOUT_MS, timeoutMs = TIMEOUT_MS, log
+	snapMargin = 8, isHeaded = false, waitForTimeoutMs = WAIT_FOR_TIMEOUT_MS, timeoutMs = TIMEOUT_MS, log
 }) => {
 	const deadline = timeoutMs > 0 ? Date.now() + timeoutMs : Infinity;
 	const playwright = await loadPlaywright();
 	const browserType = engine === "webkit" ? playwright.webkit : playwright.chromium;
-	const browser = await browserType.launch({ channel: CHANNELS[engine], headless: true });
+	const browser = await browserType.launch({ channel: CHANNELS[engine], headless: !isHeaded });
 	const saved = [];
 	const films = [];
+	const annotations = [];
 	let failed = false;
 
 	try {
@@ -71,8 +72,68 @@ export const runPlaywright = async ({
 			films.push({ name: step.value, file: path.join(out, `${prefix}${step.value}.png`), scale: scale ?? 1, viewport: page.viewportSize(), frames });
 		}
 
+		const cutOut = async (rect, file) => {
+			try {
+				await page.screenshot({ path: file, clip: clipRect(rect, snapMargin, page.viewportSize()) });
+				return path.basename(file);
+			} catch {
+				return null;
+			}
+		}
+
+		const annotate = async step => {
+			const base = `${prefix}annotate-${step.value}`;
+			const images = {}
+			let draft = null;
+			await page.evaluate(step.start);
+			for (;;) {
+				if (page.isClosed()) {
+					annotations.push({ name: step.value, status: "cancelled", result: null });
+					return;
+				}
+				if (Date.now() > deadline) {
+					annotations.push({ name: step.value, status: "cancelled", result: null });
+					throw new Error(`timed out after ${timeoutMs} ms`);
+				}
+				const text = await page.evaluate(step.check).catch(() => null);
+				const answer = text && text !== "missing" ? JSON.parse(text) : null;
+				if (text === "missing") {
+					await page.evaluate(`${draft ? `window.${step.restoreKey} = ${JSON.stringify(draft)};` : ""}${step.start}`).catch(() => undefined);
+				} else if (answer?.status === "open") {
+					if (answer.draft) draft = answer.draft;
+					if (answer.captures?.length) {
+						await page.evaluate(step.hide).catch(() => undefined);
+						await page.waitForTimeout(80);
+						for (const capture of answer.captures) {
+							const image = await cutOut(capture.rect, path.join(out, `${base}-${capture.id}.png`));
+							if (image) images[capture.id] = image;
+						}
+						await page.evaluate(step.show).catch(() => undefined);
+					}
+				} else if (answer) {
+					const annotation = { ...answer, name: step.value, scale: scale ?? 1 }
+					if (annotation.status === "done" && annotation.result) {
+						await page.screenshot({ path: path.join(out, `${base}.png`) });
+						annotation.image = `${base}.png`;
+						await page.evaluate(step.cleanup).catch(() => undefined);
+						await page.waitForTimeout(160);
+						for (const [id, element] of Object.entries(annotation.result.elements)) {
+							element.image = element.isLive
+								? await cutOut(element.rect, path.join(out, `${base}-${id}.png`)) ?? images[id] ?? null
+								: images[id] ?? null;
+						}
+						await page.evaluate(step.dispose).catch(() => undefined);
+					}
+					annotations.push(annotation);
+					return;
+				}
+				await new Promise(resolve => setTimeout(resolve, step.pollMs));
+			}
+		}
+
 		const runStep = async step => {
 			const { kind, value } = step;
+			if (kind === "annotate") return annotate(step);
 			if (kind === "snap") {
 				const file = path.join(out, `${prefix}${value}.png`);
 				const clip = step.selector ? clipRect(await elementRect(step), snapMargin, page.viewportSize()) : undefined;
@@ -133,5 +194,5 @@ export const runPlaywright = async ({
 		await browser.close();
 	}
 
-	return { saved, failed, films }
+	return { saved, failed, films, annotations }
 }

package/src/engines/webkitgtk.mjs

@@ -48,6 +48,7 @@ export const runWebkitGtk = ({
 
 	const saved = [];
 	const films = [];
+	const annotations = [];
 	let buffer = "";
 	child.stdout.setEncoding("utf8");
 	child.stdout.on("data", chunk => {
@@ -61,6 +62,7 @@ export const runWebkitGtk = ({
 			const value = JSON.parse(line.slice(tab + 1));
 			if (kind === "saved") saved.push(value);
 			else if (kind === "film") films.push(value);
+			else if (kind === "annotate") annotations.push(value);
 			else if (kind === "print") log.info(value);
 			else if (kind === "error") log.error(value);
 			else if (kind === "init-errors") reportInitErrors(value, log);
@@ -73,5 +75,5 @@ export const runWebkitGtk = ({
 		}
 	});
 	child.once("error", reject);
-	child.once("exit", code => resolve({ saved, failed: code !== 0, films }));
+	child.once("exit", code => resolve({ saved, failed: code !== 0, films, annotations }));
 });