@tandryukha/aidemo 0.8.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/docs/AUTHORING.md CHANGED
@@ -5,8 +5,9 @@ producing a demo video with aidemo. It is served by the engine itself, so it
5
5
  always matches the engine that will render:
6
6
 
7
7
  - **MCP** (preferred for agents): call the `get_authoring_guide` tool on the
8
- `aidemo` MCP server — it returns this document.
9
- - **CLI**: `aidemo guide` prints it.
8
+ `aidemo` MCP server — it returns this document (pass `topic` — `core`,
9
+ `schema`, `attention`, `polish`, `chatgpt-apps`, `debug`… — for one slice).
10
+ - **CLI**: `aidemo guide` prints it (`--topic core` / `--list` for a slice).
10
11
 
11
12
  You are the **demo director**: turn a feature + loose directions into a
12
13
  polished 30–60s narrated, captioned browser-demo MP4. You author the
@@ -24,14 +25,20 @@ Every operation exists on both surfaces. Agents should prefer the MCP server
24
25
  |---|---|---|
25
26
  | Read this guide | `get_authoring_guide` | `aidemo guide` |
26
27
  | Storyboard JSON Schema | `get_storyboard_schema` | — (see below) |
27
- | Validate a storyboard | `validate_storyboard` | (validated on every run) |
28
- | Scaffold a demo | `init_demo` | `aidemo init <name>` |
28
+ | Validate a storyboard | `validate_storyboard` | `aidemo validate <dir>` (or `--file <path>`; `--json`) |
29
+ | Lint / pacing forecast (no browser) | `lint_storyboard` | `aidemo lint <dir>` (`--file`, `--lang`, `--json`, `--strict`) |
30
+ | Scaffold a demo | `init_demo` (`fromUrl` drafts from a live page) | `aidemo init <name> [--from-url <url>]` |
31
+ | Draft from a Playwright trace / test | `import_trace` | `aidemo import-trace <trace.zip \| spec.ts> --name <name>` |
29
32
  | Environment check | `doctor` | `aidemo doctor` |
33
+ | Discover selectors on a page | `inspect` (job) | `aidemo inspect <url> [--dir <dir>] [--frame name=sel] [--storage-state <f>] [--cookie k=v] [--no-setup]` |
30
34
  | Dry-run the flow | `probe` (job) | `aidemo probe <dir>` |
31
- | Full pipeline | `render` (job) | `aidemo render <dir>` |
35
+ | Full pipeline | `render` (job) | `aidemo render <dir>` (`--strict` fails the run on compose warnings) |
32
36
  | One stage | `voice` / `record` / `captions` / `compose` (jobs) | `aidemo voice\|record\|captions\|compose <dir>` |
33
37
  | README GIF | `gif` (job) | `aidemo gif <dir>` |
34
38
  | Named stills (screenshot mode) | `stills` (job) | `aidemo stills <dir>` |
39
+ | Frames for review | `frames` (job) | `aidemo frames <dir> [--every 3] [--source raw\|take] [--sheet] [--columns 5]` |
40
+ | Post-render QA report | `qa` (job) | `aidemo qa <dir> [--lang] [--json] [--strict]` |
41
+ | Walkthrough export (HTML + Markdown + frames + captions) | `walkthrough` (job) | `aidemo walkthrough <dir> [--lang] [--width]` |
35
42
  | Job progress / result | `job_status`, `job_list`, `job_cancel` | (CLI runs block in the foreground) |
36
43
 
37
44
  **The job model (MCP).** Pipeline operations touch TTS/STT, a real Chrome, or
@@ -47,9 +54,44 @@ instead.
47
54
  **Always pass absolute demo directories** to MCP tools — the server's working
48
55
  directory is not necessarily your repo (Codex registers servers globally).
49
56
 
50
- **Validate early.** Run `validate_storyboard` after every storyboard edit
51
- (cheap, structured errors) instead of discovering schema issues inside a
52
- render job.
57
+ **Validate early.** Run `validate_storyboard` (CLI: `aidemo validate <dir>`,
58
+ non-zero exit on issues) after every storyboard edit — cheap, structured
59
+ errors — instead of discovering schema issues inside a render job.
60
+
61
+ **Inspect before you write a target.** `inspect {dir, url}` (CLI: `aidemo
62
+ inspect <url> --dir <dir>`) opens the page in the recording profile (so a
63
+ logged-in app shows its real UI), waits for it to settle, and returns every
64
+ visible interactive element — role, name, whether it's in the viewport — with
65
+ a ranked list of selectors that are **unique on that page right now**
66
+ (`[data-testid=…]` → `#id` → `[aria-label=…]` → `button:has-text("…")` →
67
+ `input[name=…]`/`[placeholder=…]` → `tag.class` → a short structural path),
68
+ plus the headings and any iframes (for the `frames` block; pass `frames` to
69
+ scan inside them). Use it instead of reading source or guessing: copy the
70
+ first selector of the element you mean. It writes `logs/inspect-<ts>.json`
71
+ and a screenshot next to it. Elements below the fold are listed too (marked
72
+ `↓` on the CLI); a `scrollTo` on them works as usual.
73
+
74
+ `inspect` runs the storyboard's **`setup`** block by default — preflight hook,
75
+ `storageState`, cookies — so a gated app scans as the logged-in user instead of
76
+ returning the login screen's selectors; it also picks up the storyboard's
77
+ `video` size (so "in viewport" means what the take will see) and its `frames`
78
+ map. Pass `--storage-state <file>` / `--cookie "k=v;domain=host"` for a one-off
79
+ seed without touching the storyboard, or `--no-setup` to scan signed-out.
80
+
81
+ **Lint before you spend a take.** `lint_storyboard` (CLI: `aidemo lint <dir>`)
82
+ is a browser-free preflight that predicts what compose will do to each scene:
83
+ it estimates the action time from the ops (a `type` at human speed, a scroll's
84
+ easing, a `pause`, a `goto`) and compares it with the narration's word count at
85
+ the language's speaking rate. A scene whose actions finish long before the
86
+ voice does will be **held on a frozen frame** (`[scene-freeze]`, with the
87
+ predicted hold %); one whose actions run far past the voice will be **sped up
88
+ to the x1.6 ceiling and cut** (`[overrun-cut]`). It also flags the classic
89
+ selector/wait pitfalls (`type`+`Enter` with no wait behind it, `:text-is` on a
90
+ nested label, `focus` with no `zoom` block, an anchored `waitForChange` regex,
91
+ a scene that opens with a navigation click, `music.cue` no-ops). `validate`
92
+ prints the same findings; probe/record/render run it automatically (non-fatal)
93
+ so the forecast is in the log next to the take. The same numbers come back
94
+ **measured** in `output/report.json` after compose (see "Verify").
53
95
 
54
96
  ## Pipeline (what the engine does)
55
97
 
@@ -70,7 +112,16 @@ network.
70
112
  one thing to prove, audience, tone, length, CTA. Otherwise infer sensible
71
113
  defaults.
72
114
  2. **Scaffold**: `init_demo` / `aidemo init <name>` → creates `demos/<name>/`
73
- with a starter `generated/storyboard.json`.
115
+ with a starter `generated/storyboard.json`. When you already know the URL,
116
+ pass `fromUrl` (`--from-url`): the engine inspects the page first and the
117
+ draft uses its **real** headings as scenes and its **real** unique selectors
118
+ as beats (search box → `type`, first CTA → a `hover` you turn into the
119
+ click), with everything else it saw in `_candidates` and `input/brief.md`.
120
+ Repeated heading text gets an explicit `nth=`, and a page with no headings
121
+ at all (a single-page app that paints its own layout) drafts scroll beats
122
+ instead, so the skeleton still shows how long the page is. No LLM runs — you
123
+ still write the narration and choose the flow; you just never start from
124
+ `#search` placeholders.
74
125
  3. **Confirm selectors for a new/changed flow.** Don't guess. The cheapest way
75
126
  is a **probe** — a record-only dry run (narration optional) that drives the
76
127
  real flow in ~90 s without spending TTS or a full take, so you can verify
@@ -78,7 +129,10 @@ network.
78
129
  `storyboard.frames`. A probe that hits a bad selector leaves a screenshot +
79
130
  frame dump in `logs/` (surfaced as `failureArtifacts` in `job_status`).
80
131
  4. **Write the storyboard** (schema + principles below), then
81
- `validate_storyboard`.
132
+ `validate_storyboard` and `lint_storyboard` — fix every `[scene-freeze]` /
133
+ `[overrun-cut]` the lint predicts *before* paying for voice: add on-screen
134
+ beats (hover, scroll, focus, a `pause`) to a scene that is all narration, or
135
+ trim the narration of a scene that is all action.
82
136
  5. **Render**: run the `render` job (headed for real ChatGPT: keep
83
137
  `headless: false` / drop `--headless`). Then inspect `output/final-demo.mp4`
84
138
  — extract a few frames with ffmpeg and look at them.
@@ -89,7 +143,8 @@ network.
89
143
  `compose` — voice **skips unchanged scenes** (hashes narration + voice
90
144
  plan), so only the edited scene is re-voiced and an approved take is
91
145
  preserved (`force` re-voices all; `scene <id>` targets one). Re-record only
92
- if the browser flow changed.
146
+ if the browser flow changed — and then only from the first changed scene
147
+ (`record --from-scene <id>` / `fromScene`): earlier scenes keep their take.
93
148
 
94
149
  ## Demo-director principles (make it feel human-made)
95
150
 
@@ -108,23 +163,61 @@ network.
108
163
  video to its narration and freeze-holds a static page for any remainder — so
109
164
  a scene that's mostly a pause should have shorter narration, and a busy scene
110
165
  more.
166
+ - **Pre-navigate: put nav clicks at the END of the prior scene.** In a
167
+ screen-tour where each scene starts with a navigation click, the destination
168
+ appears 2–3 s late (the cursor glides across the *old* screen before the
169
+ click lands), desyncing narration from visuals — the narration for screen N
170
+ plays over screen N−1. Instead, end each scene with the click that opens the
171
+ *next* scene's screen: every scene then opens already on the screen its
172
+ narration describes.
111
173
  - **End on a clear CTA.**
112
174
  - **Voice plan per scene**: pick `voiceId` (marin/alloy/verse/… on the default
113
175
  OpenAI TTS; an ElevenLabs voice id if the engine runs with
114
176
  `AIDEMO_TTS_PROVIDER=elevenlabs`; a Kokoro voice id — af_heart, am_adam, … —
115
177
  if it runs with `AIDEMO_TTS_PROVIDER=local`; both non-default providers
116
178
  ignore `instructions`) and `instructions` to steer tone/emotion/pace. Keep it
117
- consistent across scenes.
179
+ consistent across scenes. **`voice.pronounce`** (`{"aidemo":"A.I. demo",
180
+ "SQL":"sequel"}`, whole-word, case-sensitive, storyboard-level or per
181
+ scene) rewrites only what the TTS *hears* — the narration text, and so the
182
+ captions, keep the written form. Use it for product names, acronyms, and
183
+ version strings the voice mangles.
118
184
 
119
185
  ## Storyboard schema (quick reference)
120
186
 
121
187
  The precise contract is the JSON Schema from `get_storyboard_schema`
122
188
  (generated from the engine's own zod schema, `src/types.ts`).
123
189
 
124
- Top level: `title`, `language?`, `targetLengthSeconds?`, `video{width,height}`
125
- (default 1280x720), `frames{ name: iframeSelector }`,
126
- `voice{voiceId,instructions,speed}` (default, scenes may override), `music?`,
127
- `zoom?`, `intro?`, `outro?`, `transition?`, `output?`, `scenes[]`.
190
+ Top level: `title`, `language?`, `knownTerms?`, `targetLengthSeconds?`,
191
+ `video{width,height,deviceScaleFactor?}` (default 1280x720; see *Capture
192
+ resolution*), `frames{ name: iframeSelector }`,
193
+ `voice{voiceId,instructions,speed,pronounce?}` (default, scenes may override), `music?`,
194
+ `zoom?`, `intro?`, `outro?`, `transition?`, `hold?`, `output?`, `setup?`,
195
+ `attention?`, `keystrokes?`, `captions?`, `redact?`, `hide?`, `frame?`,
196
+ `brand?`, `scenes[]`.
197
+
198
+ `setup?` prepares the take before the first action — for cookie-gated or
199
+ fixture-rotating sites, so you never hand-write a Playwright seed script:
200
+ - `storageState`: path (relative to the demo dir) to a Playwright storageState
201
+ JSON (`{cookies:[…], origins:[{origin, localStorage:[{name,value}]}]}`, e.g.
202
+ saved by `context.storageState({path})`). Its cookies are added to the
203
+ profile and each origin's localStorage entries set before the storyboard
204
+ runs. CLI/MCP: `--storage-state <file>` / `storageState`.
205
+ - `cookies: [{name, value, domain, path?, secure?, httpOnly?, sameSite?,
206
+ expires?}]` — add cookies directly. CLI: `--cookie "name=value;domain=host"`
207
+ (repeatable, DevTools-style attributes; MCP: `cookies`).
208
+ - `preflight`: a shell command run from the demo dir before **every** take
209
+ (record/probe/render), after the profile is resolved and before Chrome
210
+ launches — "resolved" means the path is computed (and already wiped when
211
+ `--fresh`), with no browser holding it, so the hook may safely delete,
212
+ re-create and seed that directory — re-seed a fixture, or patch this storyboard for whichever variant
213
+ the app serves today (the storyboard is **re-read after the hook**). Env:
214
+ `AIDEMO_DEMO_DIR`, `AIDEMO_STORYBOARD`, `AIDEMO_PROFILE`. Non-zero exit
215
+ aborts the take with the hook's output.
216
+ - `expectState: true` — the profile is seeded on purpose; silence the
217
+ carried-over-state warning (implied by `storageState`/`cookies`; CLI/MCP:
218
+ `--profile-seeded` / `profileSeeded`).
219
+ Seeding composes with `--fresh`: the wipe happens first, then the seeds go in
220
+ — a clean identity that still carries exactly the cookie the gate needs.
128
221
 
129
222
  `language?` is a BCP-47/ISO-639-1 code (e.g. `"et"`) describing what language
130
223
  the base `narration` is already written in — set it for a monolingual
@@ -133,11 +226,20 @@ render switch: it only feeds the `captions` STT language hint (see below).
133
226
  Distinct from `--lang`, which *selects* a scene's `narrations[code]`
134
227
  translation (Multi-language renders, below).
135
228
 
229
+ `knownTerms?: string[]` is a glossary of product names, brands and jargon
230
+ Whisper reliably mis-hears (`["fitness.ee", "aidemo", "MaxFit"]`). They are
231
+ sent at the *front* of the STT prompt — Whisper only biases on roughly its
232
+ first 224 tokens, so a term buried in a long script never reaches it. Cheaper
233
+ than `captions --offline`, which fixes spelling but gives up real word timing.
234
+
136
235
  Cinematic keys (all opt-in; omit for the plain look):
137
236
  - `zoom: {scale?=1.55, easeMs?=600, holdMs?=1700}` — **auto-zoom on focus**:
138
237
  every click/typed prompt zooms in on the interaction point at compose time,
139
238
  holds, eases out; near-consecutive focus points pan instead of bouncing.
140
- Set `"zoom": false` on a scene to opt just that scene out.
239
+ Set `"zoom": false` on a scene to opt just that scene out. On narrow
240
+ (mobile) viewports `scale` is capped so a zoom always leaves ≥360 logical px
241
+ of the page in frame — a desktop-tuned 1.35x otherwise crops a 430px take's
242
+ headline off the edge. Compose logs the cap when it applies.
141
243
  - `intro` / `outro: {title, subtitle?, durationMs?=2600, background?, accent?,
142
244
  fadeMs?=350}` — typographic title cards; narration/captions shift
143
245
  automatically and music runs under the cards.
@@ -153,16 +255,67 @@ Cinematic keys (all opt-in; omit for the plain look):
153
255
  `output.loudness` (see below).
154
256
  - `transition: {type:"crossfade", durationMs?=400}` — **cross-dissolve every
155
257
  scene boundary** instead of hard-cutting (see below).
156
- - `output: {width?, height?, fit?="contain", background?, loudness?}` — **render
258
+ - `hold: {mode?="freeze"|"drift", driftScale?=1.06, backoffMs?=0,
259
+ minDriftMs?=2000, driftInFrame?=false}` — what a
260
+ scene shows while the narration outlasts its actions: a frozen frame
261
+ (default) or a **slow Ken-Burns drift** so it never reads as a stall (see
262
+ below).
263
+ - `output: {width?, height?, fit?, background?, loudness?, trimLeadingBlank?,
264
+ blankOpenCapMs?}` — **render
157
265
  at a different size/aspect** (`width`+`height`, set together) and/or **set the
158
266
  master loudness** (`loudness`), e.g. a vertical social clip (see below).
159
267
  - `motionBlur: {frames?=3}` — **subtle motion blur** on fast motion (cursor,
160
268
  scroll, zoom pan); static UI stays sharp (see below).
161
- - `cursor: {hidden?, hideScenes?, scale?}` — **compose-time cursor control**:
162
- hide or resize the cursor post-hoc instead of baking it (see below).
163
-
164
- Each scene: `id`, `narration`, `voice?`, `music?`, `zoom?` (false to disable),
269
+ - `cursor: {hidden?, hideScenes?, scale?, style?, color?, outline?,
270
+ outlineWidth?, halo?}` — **compose-time cursor control**: hide, resize,
271
+ restyle (`"arrow"` | `"dot"`) or re-contrast the cursor post-hoc instead of
272
+ baking it (see below).
273
+ - `attention: {color?, clicks?, cursorClear?}` — accent color for
274
+ highlight/spotlight/callout overlays; `clicks:true` draws a click ring
275
+ wherever a click lands (needs the `cursor` block); `cursorClear:true` parks
276
+ the cursor off the element an attention beat is about, so the pointer never
277
+ sits on top of the thing you just highlighted. See *Attention*.
278
+ - `keystrokes: true` — show a keystroke chip ("⌘ K", "Enter") on every `press`
279
+ (per-action `keystrokes` overrides). See *Attention*.
280
+ - `captions: {position?="bottom"|"top", style?="pill"|"bar"|"none", font?,
281
+ size?, color?, background?, maxWords?, maxCueMs?, minCueMs?, maxLines?,
282
+ maxCharsPerLine?}` — caption strip placement and look; scenes may
283
+ override `position` with their own `captions`. Compose also auto-flips a cue
284
+ to the top while an overlay occupies the bottom band. See *Attention* and
285
+ *Produced look*.
286
+ - `redact: [{selector, frame?, blur?}]` — **blur regions at compose time**
287
+ (prices, emails, tokens); scenes may add their own `redact`. See *Attention*.
288
+ - `hide: [selector, …]` — hide elements at **record** time (teasers, cookie
289
+ bars, ad slots); scenes may add their own `hide`. See *Attention*.
290
+ - `frame: {padding?, background?, radius?, shadow?, chrome?="none"|"browser"|"mac",
291
+ safeTop?, title?, url?}` (`url: "auto"` tracks the page per scene) — pad the video onto a styled canvas with a rounded,
292
+ shadowed window and optional browser chrome. See *Produced look*.
293
+ - `brand: {logo?, accent?, font?, watermark?}` — brand kit: accent for cards /
294
+ frame tint / attention overlays, font for cards + captions + callouts +
295
+ chrome, logo on cards and as a watermark. See *Produced look*.
296
+ - `autoIdle: true | {enabled?, minMs?=1500, noise?=0.003}` — **trim dead air
297
+ compose was never told about**: it scans the take for spans where nothing on
298
+ screen moves and caps them like an annotated wait, so a slow XHR or a long
299
+ `pause` stops inflating the scene. Opt-in; per-scene `autoIdle` overrides it
300
+ (`false` protects a scene that is deliberately still). Detected spans show up
301
+ as `autoIdleMs` per scene in `output/report.json`. Annotating the wait
302
+ (`waitForChange {idle:true}`, `waitForWidget`) is still better — it says WHY
303
+ the pause exists and lint can forecast it; `autoIdle` is the safety net for
304
+ waits you can't annotate. It only ever caps still footage, so a scene that is
305
+ deliberately motionless (a held product shot) wants `autoIdle: false`.
306
+ - `output: {preset?, chapters?, poster?, walkthrough?, width?, height?, fit?,
307
+ loudness?}` — presets `youtube|short|readme-gif|x`, MP4 chapter markers
308
+ from scene `title`s, `output/poster.png`, the walkthrough bundle. See
309
+ *Produced look* and *Transitions*.
310
+
311
+ Each scene: `id`, `title?` (chapter name), `narration`, `voice?`, `music?`,
312
+ `zoom?` (false to disable), `autoIdle?` (overrides the top-level setting),
313
+ `transition?` (`false` hard-cuts *into* this scene, or an object overriding the
314
+ top-level crossfade for that one boundary), `captions?`, `redact?`, `hide?`,
165
315
  `actions[]`.
316
+ Narration may carry `{{@name}}` anchor markers (see *Narration-anchored
317
+ beats*); the engine strips them at load and keeps the word index on the
318
+ scene as `anchors` (don't author that field).
166
319
 
167
320
  ## Transitions, output sizing & loudness
168
321
 
@@ -181,9 +334,36 @@ scene join is re-encoded (the default hard-cut path is a lossless stream-copy
181
334
  concat). Needs ≥2 scenes; a shorter `durationMs` (250–400) reads as a snappy
182
335
  dissolve, longer (600+) as a slow cinematic fade.
183
336
 
337
+ A dissolve across a **navigation** ghosts — two unrelated pages bleed through
338
+ each other — so compose hard-cuts any boundary where the next scene opens on a
339
+ load or either side is a `goto`/`back`. Override per boundary with a scene's
340
+ own `transition`: `"transition": false` on a scene forces a hard cut into it,
341
+ and `"transition": {"durationMs": 700}` overrides the top-level timing for that
342
+ one join. The compose log names every auto-cut boundary.
343
+
344
+ **`hold: {mode?, driftScale?, backoffMs?}`** — controls the **hold** compose
345
+ adds when a scene's narration runs longer than its (already x1.6-slowed)
346
+ recording. By default the last frame is frozen for the remainder
347
+ (`mode:"freeze"`, byte-identical to earlier versions). `mode:"drift"` replaces
348
+ the freeze with a slow, eased push-in on that frame (`driftScale`, 1–1.5,
349
+ default 1.06 ≈ 6% over the hold) — motion that reads as intentional rather
350
+ than a stalled screen. `backoffMs` (0–2000) takes the held frame from that
351
+ many ms *before* the scene's end instead of the very last frame, so a hold
352
+ never lands on a half-drawn transition, a spinner, or a white flash at the
353
+ tail of a navigation. A drift that is too short reads as a jitter rather than a
354
+ move, so drifts below `minDriftMs` (default 2000) fall back to a freeze; so
355
+ does a drift inside a `frame` block, where the static window chrome makes the
356
+ push-in obviously fake — set `driftInFrame: true` if you want it anyway. Compose
357
+ logs each fallback (`drift hold → freeze`). Only scenes that actually hold are affected; a scene
358
+ whose actions fill the narration is untouched, and a scene with no hold is
359
+ byte-identical. The compose log and `output/report.json` list every scene's
360
+ hold length and percentage; the lint predicts them before the take.
361
+
184
362
  **`output: {width, height, fit?, background?}`** — reframes the finished video
185
363
  (cards and captions already baked in) to a target size, applied as the last
186
- step. `fit`:
364
+ step. `fit` defaults to `"contain"`, and a `preset`'s `fit` applies only when
365
+ you don't set one — so `{"preset": "youtube", "fit": "cover"}` exports
366
+ edge-to-edge with no letterbox bars. `fit`:
187
367
  - `"contain"` (default) — scale to fit inside `width×height` and pad the
188
368
  remainder with `background` (ffmpeg color syntax, e.g. `black`, `0x1a1a1a`;
189
369
  default black). Letterbox/pillarbox bars, nothing cropped.
@@ -215,6 +395,244 @@ Override or disable via `loudness`:
215
395
  Targets are the ffmpeg `loudnorm` `I` (LUFS), `TP` (dBTP), and `LRA` (LU). The
216
396
  pass runs last over the muxed audio and pins the rate back to 44.1 kHz.
217
397
 
398
+ ## Narration-anchored beats (`{{@name}}` + `anchor`)
399
+
400
+ By default compose stretches a whole scene by one factor so its take fills
401
+ the narration. That keeps things in sync on average, but the click you care
402
+ about still lands wherever the recording put it — often a second before or
403
+ after the word that describes it. Anchors pin a beat to a word:
404
+
405
+ ```json
406
+ {
407
+ "id": "s3",
408
+ "narration": "Add the one you want, and it {{@drop}}drops straight into your basket.",
409
+ "actions": [
410
+ { "op": "pause", "ms": 1200 },
411
+ { "op": "click", "target": { "selector": "[data-testid=add-to-cart]" }, "anchor": "drop" }
412
+ ]
413
+ }
414
+ ```
415
+
416
+ - Write `{{@name}}` immediately before the word the beat should land on
417
+ (`[\w.-]+` names; the marker is stripped before TTS, captions and lint, so
418
+ the narration reads and sounds unchanged). Put `anchor: "name"` on the
419
+ action — a click, type, press, scrollTo, hover, or an attention beat. The
420
+ moment used is the click/press itself (the focus event), not the cursor
421
+ glide before it.
422
+ - Compose then retimes the scene **piecewise**: each stretch between anchors
423
+ gets its own factor so the action plays exactly as the word is spoken, and
424
+ the remainder of the scene fills the narration as before. Every piece stays
425
+ within the usual x0.5–x1.6 limits — if the beat still can't be reached, the
426
+ render succeeds and reports **`anchor-unreachable`** with the miss in ms
427
+ (`report.json` → scene `anchors[]`: `targetMs`, `landedMs`, `offMs`). The
428
+ fix is authoring: give the action more lead-in (a `pause` before it) when
429
+ it lands early, or trim what happens before it (mark waits idle, move the
430
+ marker later) when it lands late.
431
+ - Several anchors per scene are fine (`{{@add}} … {{@checkout}}`); they must
432
+ be in the same order as their actions. One marker per action; lint flags a
433
+ marker without an action (`anchor-unused`), an action without a marker
434
+ (`anchor-missing`) and a name reused on two actions (`anchor-duplicate`).
435
+ - Scenes without anchors are untouched (the single-piece path is the exact
436
+ pre-anchor arithmetic), so adding one anchor never shifts another scene.
437
+ - Multi-language: put markers in each `narrations[lang]` string too; the
438
+ engine keeps a per-language word index, so the beat lands on the translated
439
+ word.
440
+ - With STT captions the word time comes from the transcript (matched to the
441
+ script by normalized token, falling back to the proportional position); with
442
+ script-timed captions it comes from the same length-weighted model the
443
+ captions use. Re-run `captions` after changing a narration so the anchor
444
+ reads fresh timings.
445
+
446
+ ## Attention: highlight, spotlight, callout, keystrokes, redact, hide
447
+
448
+ The attention layer is how a demo says *look here* without a voice-over
449
+ "as you can see". Everything except `hide` is **compose-time**: the player
450
+ measures the element's box during the take and logs a timeline marker; compose
451
+ rasterizes the mark (headless-Chrome PNG, baseline `overlay`) and draws it
452
+ **under the cursor and before the zoom**, so it rides the camera like page
453
+ content. Restyle, retime or remove a mark → recompose, never re-record. Every
454
+ key is opt-in; a storyboard without them renders exactly as before.
455
+
456
+ **Beats (actions):**
457
+ - `{op:"highlight", target, holdMs?=1600, style?}` — an outline around the
458
+ element. `style: {color?, thickness?=3, padding?=6, shape?="box"|"ring"}`.
459
+ - `{op:"spotlight", target, holdMs?=1600, dimTo?=0.55, padding?=10, style?}` —
460
+ dims the whole frame except the element (with a thin accent edge).
461
+ - `{op:"callout", target, text, placement?="auto"|"top"|"bottom"|"left"|"right",
462
+ holdMs?=2000, style?}` — a short label (≤80 chars) pinned to the element
463
+ with an arrow. `auto` picks below, then above, then right.
464
+
465
+ Each beat **dwells** at record time for `min(holdMs, 800)` ms so the eye
466
+ registers it before the next action; the overlay itself stays for `holdMs` of
467
+ *content* time (clipped to the scene). Pair a beat with the narration that
468
+ names the thing: `highlight` the field right before you `type` into it,
469
+ `spotlight` the result the narration explains, `callout` the one number that
470
+ matters. Don't stack more than one beat per sentence.
471
+
472
+ **Keystroke chips.** `keystrokes: true` (top level) or `keystrokes: true` on a
473
+ `press` shows a chip bottom-right ("⌘ K", "Enter", "Esc", "⇧ Tab") for ~0.9 s
474
+ after the key goes down. Shortcuts-driven products (command palettes, editors)
475
+ read far better with it; typing (`type`) never shows chips — the text on
476
+ screen is the feedback.
477
+
478
+ **Click rings.** `attention: {clicks: true}` with a `cursor` block draws a
479
+ three-step ring where every click lands (the compose-time twin of the baked
480
+ cursor's ripple). `attention.color` sets the accent for rings and beats
481
+ (default `#ff5a5f`).
482
+
483
+ **Cursor style.** `cursor: {style:"dot", color?, scale?}` draws a soft dot
484
+ instead of the arrow — the screen-recorder look; the dot is centered on the
485
+ recorded point, the arrow's tip is.
486
+
487
+ **Captions placement.** `captions: {position:"top"}` (top level or per scene)
488
+ moves the strip to the top edge. Without a per-scene override, compose
489
+ **auto-flips** any cue whose window overlaps a highlight/spotlight/callout
490
+ box or a keystroke chip sitting in the bottom band, so a mark and a caption
491
+ never collide (`report.json` → `attention.captionsFlipped`).
492
+
493
+ **Cursor gets out of the way.** `attention: {cursorClear: true}` moves the
494
+ pointer off the element a highlight/spotlight/callout is about (to the nearest
495
+ free side, at record time) before the mark is drawn — otherwise the cursor
496
+ often sits exactly on the thing you just asked the viewer to look at. Pair it
497
+ with `hover {anchor:"edge"}` when you want a specific side.
498
+
499
+ **Late layout.** Attention rects are measured after the element's box has been
500
+ *stable* for a beat (bounded wait), and re-measured after the dwell: if the
501
+ target moved more than ~24px in the meantime (a font swap, a lazy image, an
502
+ async row) the take uses the settled position and records a warning on that
503
+ action in `timeline.json` (also printed by record). A target that scrolled out
504
+ of view warns too, instead of silently marking empty screen.
505
+
506
+ **Redact (compose-time blur).** `redact: [{selector, frame?, blur?=14}]` at
507
+ the top level and/or per scene. After every action the player re-measures
508
+ each selector (all matches, up to 8) and logs a span per box; compose crops,
509
+ blurs and overlays each span back — prices, emails, API keys, customer names
510
+ never ship, and the blur strength is a recompose. Boxes that move (scrolls,
511
+ re-layouts) start a new span at the next action; a region that moves *during*
512
+ a scroll is blurred at its post-scroll position, so put the scroll before the
513
+ data appears or blur a container rather than a cell.
514
+
515
+ **Hide (record-time).** `hide: ["footer", ".promo-banner"]` at the top level
516
+ (injected before any page script, survives navigations) and/or per scene
517
+ (applied at scene start, removed at scene end) sets `visibility:hidden` on
518
+ the matches — for teasers, chat widgets, cookie bars, and ad slots that would
519
+ photobomb the take. This is the **one** record-time exception to
520
+ compose-time polish: hiding is cheap to redo, and a hidden element cannot be
521
+ un-hidden by a recompose, so keep the list to things you'd never want in any
522
+ cut. Lint-check with a probe: hidden elements still take their layout space.
523
+
524
+ ## Produced look: frame, brand, caption styles, presets, chapters, poster
525
+
526
+ Everything here is compose-time and opt-in — a storyboard without these keys
527
+ renders exactly as before, and every tweak is a recompose. Frame and cards
528
+ are headless-Chrome-rasterized PNGs over core `pad`/`overlay` filters, so the
529
+ look is portable across ffmpeg builds. Read the report (`output/report.json`)
530
+ and `aidemo frames` after the first render with a frame: the canvas is bigger
531
+ than the recording, so check that captions and chips sit where you expect.
532
+
533
+ **`frame`** — the "Screen Studio" look: the recording becomes a rounded,
534
+ shadowed window on a styled canvas, with captions in the padding band below.
535
+ ```json
536
+ "frame": { "chrome": "mac", "url": "app.example.com", "padding": 56, "radius": 14 }
537
+ ```
538
+ - `padding` (default 48, logical px) grows the canvas on every side; the
539
+ canvas is `(width + 2·padding) × (height + 2·padding [+ 40 chrome])`. With
540
+ `output.preset`/`width`/`height` set, the padding grows on one axis so the
541
+ canvas already has the target aspect — the resize is a pure scale, never
542
+ letterbox bars around the frame.
543
+ - `background`: any CSS color/gradient. Default: a dark gradient tinted with
544
+ `brand.accent` when set.
545
+ - `radius` (default 14), `shadow` (default true).
546
+ - `chrome`: `"browser"` (neutral dots + address pill) or `"mac"` (traffic
547
+ lights). `url` (wins) or `title` fills the pill; omit both for an empty bar.
548
+ `"url": "auto"` **tracks the page**: each scene's pill shows the host (plus
549
+ path) the scene is actually on, taken from the take's `goto`s — use it for a
550
+ demo that moves between hosts, where one fixed address would be a lie. The
551
+ pill only changes at scene boundaries (it's a time-gated overlay, not a live
552
+ browser).
553
+ For a phone-sized take (`video: {width: 390, height: 844}` — record at the
554
+ mobile viewport) use `"iphone"` (rounded bezel + dynamic island) or
555
+ `"android"` (bezel + punch-hole): no bar, `radius` defaults to 40, and the
556
+ camera cut-out sits over the top of the video, so keep the app's own status
557
+ bar area clear or accept it being covered. Pair with `output.preset:
558
+ "short"` for a 9:16 export — the padding grows to fill the aspect.
559
+ - Zoom, cursor, attention overlays and redaction all happen **before** the
560
+ frame, so they ride inside the window; captions, keystroke chips, cards and
561
+ the watermark render at the canvas size.
562
+
563
+ **`brand`** — one place for identity, applied everywhere it should show:
564
+ ```json
565
+ "brand": { "logo": "brand/logo.png", "accent": "#3b82f6", "font": "Inter, sans-serif",
566
+ "watermark": { "position": "bottom-right", "opacity": 0.85, "scale": 0.11 } }
567
+ ```
568
+ - `accent` → intro/outro rule color (unless the card sets its own), the frame
569
+ gradient tint, and the default attention color (`attention.color` wins).
570
+ - `font` → cards, captions (`captions.font` wins), callouts, frame chrome. Use
571
+ a family that exists on the render machine; the fallback is the system sans.
572
+ - `logo` (png/svg/jpg, relative to the demo dir): drawn above the rule on
573
+ intro/outro cards and, unless `watermark.enabled:false`, as a watermark over
574
+ the content at `position` with `opacity` (0–1) and `scale` (fraction of the
575
+ frame width, default 0.11). A missing file is a compose warning
576
+ (`brand-logo-missing`), not a failure.
577
+
578
+ **Caption styles** (`captions`, top level): `style` is `"pill"` (default —
579
+ centered rounded box), `"bar"` (full-width band flush with the frame edge; the
580
+ usual choice under a `frame`), or `"none"` (no captions burned — the
581
+ SRT/VTT files are still written, for players and platforms that render their
582
+ own). `font`, `size` (default 30, logical px), `color`, `background` (any
583
+ CSS) restyle the strip; `position` still flips per scene / automatically.
584
+
585
+ **Output presets** (`output.preset`) fill `width`/`height`/`fit` when you
586
+ don't set them explicitly: `youtube` 1920×1080, `short` 1080×1920,
587
+ `readme-gif` 960×540, `x` 1280×720 — all `fit:"contain"` (letterbox, never
588
+ crop). Explicit `width`/`height`/`fit` win over the preset.
589
+
590
+ **Chapters** (`output.chapters: true`): the MP4 gets one chapter marker per
591
+ scene at its content start (after the intro card), named from the scene's
592
+ `title` (or its `id`). YouTube, QuickTime, VLC and mpv show them as a
593
+ navigable table of contents. Give every scene a short `title` when you turn
594
+ this on.
595
+
596
+ **Walkthrough** (`output.walkthrough: true` — honoured by `render` *and* by a
597
+ standalone `compose` — or `aidemo walkthrough <dir>` / the `walkthrough` job any
598
+ time after a render): one take → several
599
+ artifacts in `output/walkthrough/`: `index.html` (one card per scene with
600
+ its payoff frame, title, narration and a jump-to-time button over the video;
601
+ ← / → keys step through, Enter plays from a step), `guide.md` (the same as a
602
+ README/SOP section: heading + frame + narration per scene), `scene-NN-<id>.png`
603
+ frames, the SRT/VTT captions, and `walkthrough.json`. Frames come from the
604
+ **final** video, so they carry zoom, cursor, overlays and the frame. Scene
605
+ `title`s name the steps (falling back to ids).
606
+
607
+ **Poster** (`output.poster: true`): also writes `output/poster.png`
608
+ (`poster.<lang>.png` for language variants) — the first content frame after
609
+ the intro card, for READMEs, social cards and `<video poster>`. The path is
610
+ in `report.json` as `poster`. It skips past a blank opening (below), so the
611
+ poster isn't a white rectangle; `"poster": 4200` takes the still at that
612
+ millisecond instead, when you want a specific moment.
613
+
614
+ **Blank opening** (`output.trimLeadingBlank: true`): a take that starts on a
615
+ page still painting opens on white/blank frames. With this on, compose trims
616
+ the flat lead of the first scene (capped by `output.blankOpenCapMs`, default
617
+ 1200) and re-aligns the scene's cursor, zoom and anchors to the trim. Compose
618
+ always *measures* the opening either way and reports it as `blankOpenMs` plus
619
+ a `blank-open` warning past ~400 ms; `aidemo lint` forecasts the same pitfall
620
+ from a first scene that opens with a `goto` and no wait.
621
+
622
+ ## Capture resolution (`video.deviceScaleFactor`)
623
+
624
+ `video: {width, height}` is the **logical** viewport — the CSS px the page
625
+ lays out in. `deviceScaleFactor` (1–3, default 1) is how many real pixels each
626
+ of those gets, exactly like a Retina display: `{"width": 1280, "height": 720,
627
+ "deviceScaleFactor": 2}` records a 2560×1440 file of a 1280×720 page. Text and
628
+ UI keep their on-screen *size* but get twice the pixels, which is the fix for
629
+ a demo that looks soft after a zoom or a 1920 export, and for mobile-viewport
630
+ takes whose small type turns to mush. Cost: bigger raw footage and a slower
631
+ compose — 2 is enough for any delivery size; 3 is for stills. Bumping it
632
+ changes the recording, so it's a re-record, not a recompose. `aidemo lint`
633
+ warns when a viewport that small is about to be blown up on export
634
+ (`viewport-legibility`).
635
+
218
636
  ## Motion blur & cursor
219
637
 
220
638
  Two more compose-time polish keys, both opt-in and portable (baseline `tmix` /
@@ -235,6 +653,10 @@ without re-recording**:
235
653
  - `hideScenes: ["s5"]` — hide the cursor only on those scenes (e.g. a
236
654
  full-screen result or confirmation).
237
655
  - `scale: 1.3` — resize the cursor (1 = the 24px baseline arrow).
656
+ - `color`, `outline` (stroke color), `outlineWidth` (0–6) and `halo: true`
657
+ (a soft dark glow behind it) — make the cursor readable over a brand-colored
658
+ or busy surface, where a plain white arrow disappears. `halo` is the
659
+ one-key fix; `outline` is for matching a brand palette exactly.
238
660
 
239
661
  How it works: with a `cursor` block present, `record` leaves the take
240
662
  **cursor-free** and logs the cursor path into `timeline.json`; `compose` draws
@@ -246,22 +668,71 @@ baked at record time as before (the default).
246
668
 
247
669
  ## Action vocabulary
248
670
 
249
- A `target` is `{selector}` or `{frame,selector}` or `{named:"composer"}`:
250
- - `{op:"goto", url}`
671
+ A `target` is `{selector}` or `{frame,selector}` or `{named:"composer"}`.
672
+ Every action also accepts `comment?`, `optional?` (best-effort, see below),
673
+ `retry?` (0–5 extra attempts, interactions and `assert` only, a beat apart —
674
+ for UI that re-renders under the cursor) and `anchor?` (land this action on a
675
+ `{{@name}}` word in the narration — see *Narration-anchored beats*):
676
+ - `{op:"goto", url}` — waits for `domcontentloaded`, then (capped at ~3.4 s)
677
+ for the network to go quiet and fonts to load; that extra wait is recorded
678
+ as trimmable idle (`"load"`), so a first click never lands on a skeleton
679
+ screen and the take doesn't get longer for it.
251
680
  - `{op:"type", target, text, humanize?}` — human-cadence typing
252
681
  - `{op:"press", key}` — e.g. "Enter"
253
- - `{op:"click", target}` · `{op:"hover", target}`
254
- - `{op:"scrollTo", target, easing?, durationMs?}` · `{op:"scrollBy", dy,
255
- easing?, durationMs?}` — easing presets: `"smooth"` (default) | `"snappy"` |
256
- `"glide"` | `"linear"`
682
+ - `{op:"click", target, followPopup?}` · `{op:"hover", target, anchor?,
683
+ offset?}` — `anchor` (`"center"` default, or `"edge"|"top"|"bottom"|"left"|
684
+ "right"`) and `offset:{x,y}` aim the pointer at a *part* of the element:
685
+ `"anchor": "top"` hovers the top edge of a row so the tooltip it opens isn't
686
+ covered by the cursor, `"edge"` parks just outside it.
687
+
688
+ `followPopup:true` handles a link/button that opens a **new tab**
689
+ (`target=_blank`, `window.open`): the tab is closed and the recorded tab
690
+ navigates to its URL. The take is one window — a second tab is never in the
691
+ video, so there is no `newTab`/`switchTab`; write the story in one tab.
692
+ - `{op:"back"}` — browser history back, same readiness wait as `goto`.
693
+ - `{op:"dialog", action?="accept"|"dismiss", promptText?, holdMs?, required?}`
694
+ — **arm a native dialog handler** for the *next* action in the scene.
695
+ `alert`/`confirm`/`window.print` and friends block the page and are invisible
696
+ to a recording otherwise (Playwright auto-dismisses an unhandled dialog, so
697
+ the take silently loses the step). Put the `dialog` action immediately
698
+ **before** the click that triggers it; `holdMs` waits before answering (the page
699
+ stays paused, so the moment reads as deliberate), `promptText`
700
+ answers a `window.prompt`, and `required: true` fails the scene if no dialog
701
+ ever appeared (the default is best-effort).
702
+ - `{op:"select", target, value? | label?}` — native `<select>`: the cursor
703
+ clicks it, the option is committed programmatically (the OS dropdown never
704
+ paints into a recording), `change` fires.
705
+ - `{op:"drag", target, to:{target} | to:{x,y}}` — press on `target`, glide
706
+ with the cursor, release on the destination (pointer-event and HTML5 DnD
707
+ libraries both see a real drag; the cursor path is recorded).
708
+ - `{op:"upload", target, files:[…]}` — attach files: `target` is the file
709
+ input (set directly) or the button that opens the file chooser (clicked; the
710
+ chooser is answered headlessly, no OS dialog in the take). Paths resolve
711
+ against the demo dir (`input/sample.csv`); a missing file fails the take.
712
+ - `{op:"scrollTo", target, easing?, durationMs?, state?, settleMs?}` ·
713
+ `{op:"scrollBy", dy, target?, easing?, durationMs?, settleMs?}` — easing
714
+ presets: `"smooth"` (default) | `"snappy"` | `"glide"` | `"linear"`.
715
+ `scrollBy` with a `target` is **scoped to that element's own scroller**: the
716
+ wheel is dispatched over it and `dy` is clamped to the room its nearest
717
+ scrollable ancestor has left, so a bottomed-out inner panel never chains the
718
+ wheel to the page and slides the whole app off-screen (the log shows the
719
+ clamp). `settleMs` waits until the scroll position has been still for that
720
+ long (≤3 s) instead of a fixed pause — use it on smooth-scrolling pages
721
+ before a click. `scrollTo … state:"attached"` accepts a zero-size mount
722
+ point (see `waitFor`).
257
723
  - `{op:"focus", target, scale?, holdMs?}` — deliberate zoom beat on an element
258
724
  without clicking it (needs top-level `zoom` enabled)
259
725
  - `{op:"still", name}` — **screenshot mode**: mark a named still at this beat.
260
726
  A pure timeline marker (no screenshot at record time); `aidemo stills` /
261
727
  `render` extract `output/stills/<name>.png` from the clean take. See *Stills /
262
728
  screenshot mode* below.
263
- - `{op:"waitFor", target, timeoutMs?}` — normal wait (fires instantly if the
264
- selector already matches — no good for in-place changes)
729
+ - `{op:"waitFor", target, timeoutMs?, state?}` — normal wait (fires instantly
730
+ if the selector already matches — no good for in-place changes). Default
731
+ waits for **visible**; `state:"attached"` only requires the element to be in
732
+ the DOM — for a lazily-filled empty `<div>` (0 px tall until something
733
+ mounts into it on scroll), which is never "visible" and would time out. A
734
+ timeout names the total budget waited (`not visible after 30000ms (budget
735
+ 30000ms)`), not Playwright's last poll chunk.
265
736
  - `{op:"waitForWidget", target, textMatches?, label?, timeoutMs?}` — **records
266
737
  the wait as idle** so compose trims/speeds it. Use for every ChatGPT
267
738
  "thinking" wait for a brand-**new** widget. When a prompt could render
@@ -287,12 +758,47 @@ A `target` is `{selector}` or `{frame,selector}` or `{named:"composer"}`:
287
758
  `"2"` for a 1→2 re-render. `idle:true` records it as trimmable idle like
288
759
  `waitForWidget`.
289
760
  - `{op:"pause", ms}` — a deliberate on-screen beat (not trimmed)
761
+ - `{op:"moveTo", target}` · `{op:"moveTo", x, y}` — glide the cursor to an
762
+ element's center or an absolute viewport point **without clicking**: park it
763
+ out of the way before a reveal, or point at what the narration names.
764
+ - `{op:"assert", target?, textMatches?, url?, timeoutMs?=5000}` — **prove the
765
+ payoff happened**: polls until `target` is visible (and its text matches
766
+ `textMatches`, JS regex, no inline flags — use `[Cc]onfirmed`) and/or the
767
+ page URL matches `url`; otherwise **fails the take with a named error**
768
+ (`assert failed after 5000ms: text "…" does not match /…/`). Put one after
769
+ the moment the demo exists to show (order confirmed, item added) so a
770
+ spinner never ships as a video. `optional:true` turns a miss into a logged
771
+ skip. Pair it with `select`/`drag`/`upload`: assert the visible result
772
+ (`"3 files attached"`, the dropped card) — the actions themselves only prove
773
+ the input was delivered.
290
774
 
291
775
  Target `last`/`nth` pick among matching **frames** on framed targets and among
292
776
  matching **elements** on plain (frameless) targets — e.g.
293
777
  `{selector:"[data-message-author-role=\"assistant\"]", last:true}` is the
294
778
  newest reply.
295
779
 
780
+ **Selector gotcha — nested label text:** for buttons whose label sits in a
781
+ nested text node (common in react-native-web, MUI, and icon+label buttons like
782
+ `<button role="button"><svg/><div>Library</div></button>`), prefer
783
+ `:has-text("Library")` over `:text-is("Library")`. Playwright's `:text-is`
784
+ matches the *smallest* element whose text equals the string — the inner
785
+ `<div>`, not the clickable wrapper — so
786
+ `[role="button"]:text-is("Library")` matches **0 elements** and fails the
787
+ take, while `[role="button"]:has-text("Library")` matches the button (the
788
+ player takes `.first()`).
789
+
790
+ Any action also accepts **`optional: true`** — best-effort: when its target
791
+ never appears or the interaction fails, the player logs and continues instead
792
+ of failing the take. Use it for state-dependent one-time UI, e.g. collapsing
793
+ the ChatGPT sidebar persists in the Chrome profile, so a re-run finds the
794
+ close button absent and a required click would kill the run. Optional
795
+ interactions (click/type/hover/scrollTo/focus) probe ~3 s for the target
796
+ instead of stalling out the full action timeout; optional waits honor their
797
+ own `timeoutMs` and then continue. Don't mark actions the demo's visuals
798
+ depend on — a skipped optional action is only a log line. (In `--golden`
799
+ probes, optional actions always record `ok: true` and no `found`, keeping the
800
+ baseline deterministic.)
801
+
296
802
  Typing into the composer while the prior reply is still streaming is safe: the
297
803
  player waits the composer out and **records that wait as trimmable idle**, so a
298
804
  long streaming turn can't blow the next scene past its narration.
@@ -311,7 +817,23 @@ storyboard viewport, and **aborts** when the computed crop doesn't match the
311
817
  viewport instead of shipping a mis-cropped take
312
818
  (`AIDEMO_NATIVE_CROP_UNSAFE=1` downgrades the abort to a loud warning — only
313
819
  for debugging, and then review every frame). No automatic check can see
314
- everything, so eyeball the frames anyway. `record` also preserves the
820
+ everything, so eyeball the frames anyway.
821
+
822
+ **⚠ The wrong-window failure mode (and the guard against it).** Perfect
823
+ geometry still doesn't prove the right *pixels* were recorded: a screen grab
824
+ captures whatever is painted at the crop rectangle, so if the driven browser
825
+ window is occluded by another window, sitting on a different display than the
826
+ captured one, or covered by a notification, the take shows *someone else's
827
+ window* — the automation itself succeeds (selectors match, timeline is
828
+ written), so nothing else fails. That's both silent garbage and a privacy
829
+ hazard (whatever was on screen — a password manager, DMs, an OTP page — ships
830
+ in the "publish-ready" video). The recorder raises the browser window before
831
+ capture and, after cropping, **verifies the captured content against a CDP
832
+ screenshot of the page** taken at stop time, aborting on a mismatch (same
833
+ `AIDEMO_NATIVE_CROP_UNSAFE=1` escape hatch). Keep the browser window
834
+ front-most and unoccluded on the captured display for the whole take — or use
835
+ the default `--capture playwright`, which records the page buffer directly
836
+ and is immune to occlusion and to anything else on your screen. `record` also preserves the
315
837
  previous take as one `.prev` generation (`recordings/raw.prev.*` +
316
838
  `generated/timeline.prev.json`) instead of deleting it, so a bad re-record
317
839
  never destroys the last good take — copy the `.prev` files back over the
@@ -370,17 +892,39 @@ Confirmed by recording a real production shopping app (2026-07-06). The engine
370
892
  already handles the frame/timing/stealth cases below; you mostly need to author
371
893
  correctly.
372
894
 
895
+ **Profile state is the silent failure.** The recording profile persists cookies
896
+ and localStorage between takes — including anything a `probe` just did. A demo
897
+ whose story starts at a first-run gate, an onboarding step, an empty state or a
898
+ one-shot flow will *skip that beat* on the next take and record the opposite of
899
+ what the narration says, with nothing failing and nothing to see in review.
900
+ - Record such demos with **`--fresh`** (`record`/`probe`/`render`): the take
901
+ runs against a wiped throwaway profile in the demo dir, so every run starts
902
+ from a clean browser identity. Not for logged-in demos — a fresh profile has
903
+ no login.
904
+ - Otherwise `record` warns when the profile already holds state for the
905
+ storyboard's first `goto` origin. When that state is deliberate (a login, a
906
+ cookie gate you seeded), acknowledge it with `--profile-seeded` /
907
+ `setup.expectState: true` — seeding via `setup.storageState`/`cookies`
908
+ implies it — so the log stays readable.
909
+ - **`aidemo profile path`** prints the shared profile's location (handy when
910
+ the engine came from Homebrew or an npx cache); **`aidemo profile reset`**
911
+ wipes it.
912
+
373
913
  **Login & profile (the crux).**
374
914
  - Use a dedicated Chrome profile logged into ChatGPT with the app's dev
375
915
  connector enabled (`AIDEMO_CHROME_PROFILE` or the `profile` option). Run
376
- **headed**; quit any Chrome on that profile first (Playwright needs the lock).
916
+ **headed**; quit any Chrome on that profile first (Playwright needs the
917
+ lock — `record` detects a still-running Chrome on the profile and aborts
918
+ with a clear message, and cleans up a stale lock from a crashed Chrome).
377
919
  - You **cannot copy** an existing ChatGPT login in: real Chrome encrypts
378
920
  cookies with the macOS Keychain, but Playwright launches with
379
921
  `--use-mock-keychain --password-store=basic`, so copied cookies won't
380
922
  decrypt. Log in **once** in a Chrome that uses the *same* mock store, then
381
- quit and record:
923
+ quit and record. **`aidemo login [profile]`** does this for you: it opens a
924
+ normal Chrome on the recording profile with those flags, waits for you to
925
+ log in and quit, then reports the profile ready. (Manual equivalent:
382
926
  `open -na "Google Chrome" --args --user-data-dir=<profile> --password-store=basic --use-mock-keychain https://chatgpt.com/`
383
- (A normal window like this also dodges Google's SSO automation block.)
927
+ — a normal window like this also dodges Google's SSO automation block.)
384
928
  - Cloudflare: the engine launches with
385
929
  `--disable-blink-features=AutomationControlled` so `navigator.webdriver` is
386
930
  false and the profile's real `cf_clearance` is honored (otherwise you get a
@@ -422,10 +966,17 @@ Always set `"last": true` on widget targets (newest widget for this turn).
422
966
  End the scene on the visible hand-off CTA and let narration cover the jump.
423
967
  (Recorder keeps the largest `.webm` so the stray new-tab recording is
424
968
  discarded.)
425
- - Collapse the sidebar (`[data-testid="close-sidebar-button"]`, persist it in
426
- the profile) to hide chat-history PII and clean the frame. Note the home
427
- screen greets the account by first name for ~2 s — blur or accept per the
428
- owner.
969
+ - Collapse the sidebar (`[data-testid="close-sidebar-button"]`) to hide
970
+ chat-history PII and clean the frame — and mark that click **`optional:
971
+ true`**: the collapsed state persists in the Chrome profile, so on a re-run
972
+ the close button is absent and a required click would fail the take. Note
973
+ the home screen greets the account by first name for ~2 s — blur or accept
974
+ per the owner.
975
+ - **Takes mutate real app state.** Every probe and take drives the live app —
976
+ a `log_meal` demo actually logs a meal, an `add_to_cart` demo actually fills
977
+ a cart. Probe + a few takes will drift your app's data; reset/re-seed your
978
+ fixture or test account between the probe and the take (and between takes)
979
+ if the demo depends on its starting state.
429
980
  - **Widget-side mutations render in the SAME widget, not a new one** (v2
430
981
  recording, 2026-07-06): click-Add renders a basket bar into the widget's
431
982
  `[data-cart-bar]` slot; the cart's `±` re-renders the row in place. Wait with
@@ -470,8 +1021,33 @@ Always set `"last": true` on widget targets (newest widget for this turn).
470
1021
  `logs/fail-<scene>-<n>.{png,json}` (screenshot + which frames matched).
471
1022
  - **A failed take is salvaged, not lost**: `record` writes a partial
472
1023
  `timeline.json` and keeps the main recording even when a late scene fails, so
473
- a scene-7-of-7 failure doesn't discard the good footage. Re-run to get a
474
- clean take.
1024
+ a scene-7-of-7 failure doesn't discard the good footage. Then **resume
1025
+ instead of re-recording**: `record {fromScene: "s7"}` (CLI `record <dir>
1026
+ --from-scene s7`, also on `render`) keeps the earlier scenes' footage and
1027
+ timeline, replays their actions at speed only to rebuild the app state, and
1028
+ records from `s7` on. Reuse is guarded by a per-scene hash of the actions
1029
+ (+ hide/redact/viewport/cursor mode/`setup`/`params`): an edited earlier
1030
+ scene refuses with "resume from `<id>` or earlier". A resume must also keep
1031
+ the same capture mode and viewport as the take it continues — compose refuses
1032
+ a timeline whose raw files differ in size rather than misplacing every
1033
+ overlay on the reused scenes.
1034
+ - **Flows you cannot replay**: when the demo *earns* one-shot state (a balance
1035
+ goes 10 → 20, an invite is consumed, a coupon burns), `--from-scene` is not a
1036
+ speed optimisation — it is the only correct retry. A full re-record replays
1037
+ actions against an app that has already moved on, so it either fails or
1038
+ records the wrong numbers; resuming keeps the earned footage and picks up at
1039
+ the scene that broke. Pair it with `setup.expectState: true` so the
1040
+ carried-over-state warning stays quiet on the seeded profile.
1041
+ **But a plain resume still *replays* the earlier scenes' actions** to rebuild
1042
+ state — on a state-earning flow that replay itself fails (the first scene
1043
+ waits for a "stranger" element the profile no longer shows). Add
1044
+ `--no-replay` (`record`/`render`, `{noReplay: true}`) to skip the replayed
1045
+ scenes' actions entirely and trust the profile as it stands; the scene you
1046
+ resume from must then open with its own `goto`. This is also how you record
1047
+ one storyboard across two real days (day 0 today, day 1 after the app's
1048
+ date rolls over) on one profile.
1049
+ The same flag is the cheap way to re-shoot
1050
+ a tail you changed after an approved take.
475
1051
 
476
1052
  `init_demo` / `aidemo init` scaffolds a storyboard already using all of the
477
1053
  above.
@@ -553,8 +1129,46 @@ renders relative to its own dir, keep any local asset paths (e.g. a music
553
1129
 
554
1130
  ## Verify before declaring done
555
1131
 
556
- Play (or frame-extract) `output/final-demo.mp4`: cursor glides and clicks
557
- pulse, narration matches on-screen actions, captions are readable and in sync,
1132
+ **A selector that matched nothing** fails the take with drift suggestions:
1133
+ the failure log (and `logs/fail-<scene>-<n>.json`, `driftCandidates`) lists
1134
+ the interactive elements that most resemble what the selector asked for —
1135
+ by testid / id / text similarity — each with a unique selector to paste, and
1136
+ `logs/drift-<scene>-<n>.json` holds the full ranked list. If nothing similar
1137
+ is on the page, you are on the wrong screen or state (a gate, a stale
1138
+ profile, a navigation that didn't land): check the screenshot first.
1139
+
1140
+ Start with **`output/report.json`** (written by every compose; the `render` /
1141
+ `compose` job results carry `report` and `warnings`): per scene it records
1142
+ the recorded length, the narration target, the retime `factor` applied, and
1143
+ `holdMs`/`holdPct` (how much of the scene is a held frame), plus the tail and
1144
+ blank-frame trims, `autoIdleMs` (motionless footage `autoIdle` capped, 0 when
1145
+ off), focus events kept vs dropped, and `warnings[]` —
1146
+ `scene-freeze` (a scene is >40% held frame), `overrun-trim` (actions ran past
1147
+ the narration even at the x1.6 ceiling and were cut), `blank-tail`,
1148
+ `stale-captions`, `focus-dropped`, `cursor-missing`, `anchor-unreachable`
1149
+ (an anchored action could not reach its word within the retime limits; the
1150
+ scene's `anchors[]` shows `targetMs` / `landedMs` / `offMs`). A held scene
1151
+ is a storyboard problem, not a compose problem: give it on-screen beats or
1152
+ shorten its narration, then re-run `voice` + `compose`.
1153
+
1154
+ Then run **`aidemo qa <dir>`** (MCP: the `qa` job) on the finished file — a
1155
+ browser-free, API-free audit of what actually shipped: length vs
1156
+ `targetLengthSeconds`, a blank opening or blank poster, master loudness (LUFS
1157
+ / true peak) and whether the music bed is audible under the narration,
1158
+ per-scene hold percentage and compose warnings, caption cue stats, and
1159
+ hygiene checks (a fixed `frame.url` on a demo that visits several hosts, a
1160
+ non-English demo with ASCII-only chapter titles). `--json` for the structured
1161
+ report, `--strict` to exit non-zero on any finding.
1162
+
1163
+ `render`/`compose` also take **`--strict`** (exit non-zero when compose emitted
1164
+ any warning — the CI gate), and every warning that has a moment on screen
1165
+ leaves a PNG in `output/warnings/<name>.png` so you can look at the frame
1166
+ instead of guessing. For a fast visual scan, `aidemo frames <dir> --sheet`
1167
+ writes a single contact-sheet PNG (`--columns`, default 5) next to the frames.
1168
+
1169
+ Then play (or frame-extract — `aidemo frames <dir>` / the `frames` job)
1170
+ `output/final-demo.mp4`: cursor glides and clicks pulse, narration matches
1171
+ on-screen actions, captions are readable and in sync,
558
1172
  no dead air, and the key moment (e.g. checkout confirmed) is actually visible
559
1173
  on screen. If the demo is headed for a README, run the `gif` job — GIFs
560
1174
  autoplay on GitHub; MP4s don't.
@@ -566,15 +1180,72 @@ CLI. If nothing came up, skip this.
566
1180
 
567
1181
  ## Debugging
568
1182
 
1183
+ - **`generated/timeline.json` → `scenes[].actions[]`** is the per-action
1184
+ record of the take: `{index, op, target, startMs, endMs, ok, skipped?,
1185
+ retries?, warnings?[]}`. Read it to see which action ate the time, which
1186
+ optional action was skipped and why, and which action window saw failed
1187
+ requests (`"2 failed request(s): 500 POST /api/cart …"`) — the record-side
1188
+ twin of `output/report.json`.
569
1189
  - Every run tees its output to `<demo>/logs/<command>.log`; a failed take also
570
1190
  leaves `logs/fail-<scene>-<n>.{png,json}`. `job_status` surfaces all of these
571
1191
  paths on failure.
1192
+ - The fail JSON (and the error text) carries **`failedRequests`**: every
1193
+ HTTP ≥400 response and network failure since the previous action, with
1194
+ method, URL, status and the first bytes of an xhr/fetch error body. A click
1195
+ that "did nothing" because the backend answered 500 looks like a click miss
1196
+ in the screenshot; this is where the reason shows up.
1197
+ - **`output/report.json`** (every compose) says *why* a scene looks slow or
1198
+ cut: per-scene retime factor, hold %, trims, and the `warnings[]` list; the
1199
+ compose log prints the same warnings. `aidemo lint <dir>` predicts them
1200
+ without a browser.
1201
+ - **`aidemo frames <dir> --every 3`** (MCP `frames`) dumps evenly spaced PNGs
1202
+ from `output/final-demo.mp4` into `output/frames/` — look at them instead of
1203
+ hand-running `ffmpeg -ss`. `--source raw` samples the latest raw file;
1204
+ `--source take` samples the whole recorded take by walking `timeline.json`,
1205
+ so a resumed take's earlier scenes are included and each frame is named with
1206
+ the scene it came from.
1207
+ - **Resumed takes** (`--from-scene`): `timeline.json` scenes reused from an
1208
+ earlier take carry `source` (their raw file, `recordings/raw.keep-*`) and
1209
+ `leadInMs`; `recordings/raw.*` holds only the new tail. `frames --source
1210
+ raw` therefore shows the new part only — use `--source take` (or review the
1211
+ composed video) to see the whole thing.
572
1212
  - `AIDEMO_KEEP_TMP=1` preserves `.compose-tmp/` intermediates when debugging
573
1213
  compose.
574
1214
  - `doctor` checks Node, ffmpeg, Chrome, the TTS/STT endpoint (and flags
575
1215
  LLM-only servers like Ollama, which have no audio endpoints — point
576
1216
  `OPENAI_BASE_URL` at a speech server such as speaches instead).
577
1217
 
1218
+ ## Import from Playwright (trace or test file)
1219
+
1220
+ When the product already has an e2e test for the flow, don't re-derive the
1221
+ selectors: `import_trace {file, name}` / `aidemo import-trace <file> --name
1222
+ <name>` takes a **`trace.zip`** (`npx playwright test --trace on`, or
1223
+ `context.tracing.start/stop`) or a **`*.spec.ts`** file and writes
1224
+ `demos/<name>/` with a draft storyboard:
1225
+
1226
+ - the run's own actions become beats — `goto`, `fill`→`type`, `click`,
1227
+ `hover`, `press`, `selectOption`→`select`, `setInputFiles`→`upload`,
1228
+ `waitForSelector`→`waitForWidget`, `dragAndDrop`→`drag`, `expect(...)
1229
+ .toBeVisible/toHaveText/toHaveURL/toBeChecked/toHaveAttribute`→`assert`
1230
+ (the last two fold the condition into the selector as `>> :scope:checked` /
1231
+ `>> :scope[href="…"]`); `toHaveScreenshot` is dropped with a note — it
1232
+ asserts pixels, not a story beat; calls that failed in the trace are dropped;
1233
+ - `getByRole/getByTestId/getByText/getByPlaceholder` and `internal:` engines
1234
+ become plain storyboard selectors (`role=button[name="Add to basket"i]`,
1235
+ `[data-testid="x"]`, `text=…`); `getByLabel` stays an `internal:label="…"i`
1236
+ selector, which `page.locator` resolves to the labelled control;
1237
+ `frameLocator` chains become a `frames` entry + `target.frame`;
1238
+ `.first()/.nth()/.last()` become `nth`/`last`; in a test file a locator
1239
+ parked in a `const` is followed to its use;
1240
+ - scenes are cut at each `goto` and before a click whose payoff is an explicit
1241
+ wait or assertion; narration is `<narrate: …>` placeholders; `_notes` lists
1242
+ every approximation (a statement the line-based test parser could not read —
1243
+ a page-object helper, say — or an upload whose file you must copy into
1244
+ `input/`).
1245
+
1246
+ Then write the narration, merge or split scenes to one beat each, and
1247
+ `probe`. Nothing is inferred by a model — the storyboard is the test, retold.
1248
+
578
1249
  ## Demo as regression test
579
1250
 
580
1251
  A probe run is a deterministic replay of the storyboard, so it doubles as a
@@ -595,8 +1266,13 @@ prints a readable field-level diff (`$.scenes[2].actions[0].found: expected
595
1266
  true, got false`) and exits non-zero. A missing baseline is a clear error
596
1267
  telling you to run `--update-golden` first. In golden mode a failing action is
597
1268
  recorded (`ok:false`) and the run continues, so you get the full diff instead of
598
- an abort at the first break. (The MCP `probe` tool takes the same `updateGolden`
599
- / `golden` params and returns `golden.match` + `golden.diffs` in its result.)
1269
+ an abort at the first break — and each broken selector gets the same
1270
+ diagnostics as a hard failure: `logs/fail-<scene>-<n>.png` plus
1271
+ `logs/drift-<scene>-<n>.json` with the nearest interactive elements and their
1272
+ unique selectors, listed under the diff as `selector drift → …`. Read the drift
1273
+ file, swap the selector, re-probe. (The MCP `probe` tool takes the same
1274
+ `updateGolden` / `golden` params and returns `golden.match`, `golden.diffs` and
1275
+ `golden.drift[]` — `{scene, action, file}` — in its result.)
600
1276
 
601
1277
  Wire it into CI so a breaking UI change is a failed check:
602
1278
 
@@ -617,8 +1293,8 @@ Estonian "uus kuub" came back "Scoop", "AI-treener" came back "EI treener").
617
1293
  To keep the transcript honest, the request is biased with what the engine
618
1294
  already knows about the script:
619
1295
 
620
- - **Prompt bias (always on).** The storyboard's narration text (in scene
621
- order) is sent as Whisper's `prompt`, nudging the transcript to converge on
1296
+ - **Prompt bias (always on).** Any top-level `knownTerms` glossary, then the
1297
+ storyboard's narration text (in scene order), is sent as Whisper's `prompt`, nudging the transcript to converge on
622
1298
  the actual scripted words/spelling instead of guessing from audio phonetics
623
1299
  alone — while still keeping real word-level timing from the audio. This
624
1300
  changes STT output for every non-trivial demo (strictly for the better:
@@ -633,7 +1309,8 @@ already knows about the script:
633
1309
  monolingual non-English demo) without editing the storyboard.
634
1310
 
635
1311
  **Non-English narration:** prompt bias + a language hint fix the common case,
636
- but Whisper can still misspell unusual or compound words. If burned-in
1312
+ but Whisper can still misspell unusual or compound words. Add the offenders to
1313
+ top-level `knownTerms` first — that keeps real word timings. If burned-in
637
1314
  captions still look wrong, **`aidemo captions <dir> --offline` is the
638
1315
  guaranteed-correct fallback** — cues are derived directly from the storyboard
639
1316
  script (correct spelling by construction, since there's no transcription at
@@ -642,6 +1319,15 @@ all) using each scene's measured duration from `voice.json`; only the timing
642
1319
  measured speech rhythm). `captions` logs a one-line reminder of this whenever
643
1320
  the resolved language isn't English.
644
1321
 
1322
+ **Cue segmentation.** Cues are broken at sentence ends first, then at clause
1323
+ punctuation, then before a conjunction, and only then on length — so a cue
1324
+ stops reading like it was cut mid-thought ("we open the" / "settings panel").
1325
+ Very short cues are merged forward. Tune per storyboard under `captions`:
1326
+ `maxWords` (12), `maxCueMs` (6000), `minCueMs` (1000), `maxLines` (2),
1327
+ `maxCharsPerLine` (38). `captions` logs the resulting cue stats (count, median
1328
+ length, how many fall under `minCueMs`), and `aidemo qa` flags a run with lots
1329
+ of flash cues or mid-clause splits.
1330
+
645
1331
  ## Multi-language renders (one take, N languages)
646
1332
 
647
1333
  Record the browser flow **once**, then ship the same demo in several languages —