@tandryukha/aidemo 0.8.0 → 0.14.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/record-demo/SKILL.md +5 -2
- package/README.md +16 -4
- package/docs/AUTHORING.md +609 -45
- package/package.json +3 -2
- package/src/anchors.ts +128 -0
- package/src/attention.ts +365 -0
- package/src/caption-render.ts +17 -8
- package/src/captions.ts +70 -6
- package/src/capture.ts +113 -0
- package/src/cards.ts +15 -6
- package/src/cli.ts +580 -22
- package/src/compose.ts +1006 -93
- package/src/cursor-overlay.ts +7 -113
- package/src/cursor.ts +9 -3
- package/src/distribute.ts +37 -4
- package/src/expr.ts +130 -0
- package/src/ffmpeg.ts +156 -0
- package/src/frame.ts +205 -0
- package/src/frames.ts +159 -0
- package/src/golden.ts +27 -0
- package/src/guide.ts +102 -0
- package/src/i18n.ts +7 -4
- package/src/import-trace.ts +629 -0
- package/src/inspect.ts +307 -0
- package/src/lint.ts +601 -0
- package/src/login.ts +143 -0
- package/src/mcp/jobs.ts +4 -1
- package/src/mcp/server.ts +403 -26
- package/src/player.ts +975 -32
- package/src/profile.ts +128 -0
- package/src/project.ts +12 -2
- package/src/recorder.ts +252 -15
- package/src/setup.ts +248 -0
- package/src/starter.ts +244 -0
- package/src/stills.ts +15 -4
- package/src/types.ts +643 -5
- package/src/variants.ts +1 -1
- package/src/voice-local.ts +7 -1
- package/src/voice.ts +60 -1
- package/src/walkthrough.ts +250 -0
- package/src/zoom.ts +85 -39
package/docs/AUTHORING.md
CHANGED
|
@@ -5,8 +5,9 @@ producing a demo video with aidemo. It is served by the engine itself, so it
|
|
|
5
5
|
always matches the engine that will render:
|
|
6
6
|
|
|
7
7
|
- **MCP** (preferred for agents): call the `get_authoring_guide` tool on the
|
|
8
|
-
`aidemo` MCP server — it returns this document
|
|
9
|
-
|
|
8
|
+
`aidemo` MCP server — it returns this document (pass `topic` — `core`,
|
|
9
|
+
`schema`, `attention`, `polish`, `chatgpt-apps`, `debug`… — for one slice).
|
|
10
|
+
- **CLI**: `aidemo guide` prints it (`--topic core` / `--list` for a slice).
|
|
10
11
|
|
|
11
12
|
You are the **demo director**: turn a feature + loose directions into a
|
|
12
13
|
polished 30–60s narrated, captioned browser-demo MP4. You author the
|
|
@@ -24,14 +25,19 @@ Every operation exists on both surfaces. Agents should prefer the MCP server
|
|
|
24
25
|
|---|---|---|
|
|
25
26
|
| Read this guide | `get_authoring_guide` | `aidemo guide` |
|
|
26
27
|
| Storyboard JSON Schema | `get_storyboard_schema` | — (see below) |
|
|
27
|
-
| Validate a storyboard | `validate_storyboard` | (
|
|
28
|
-
|
|
|
28
|
+
| Validate a storyboard | `validate_storyboard` | `aidemo validate <dir>` (or `--file <path>`; `--json`) |
|
|
29
|
+
| Lint / pacing forecast (no browser) | `lint_storyboard` | `aidemo lint <dir>` (`--file`, `--lang`, `--json`, `--strict`) |
|
|
30
|
+
| Scaffold a demo | `init_demo` (`fromUrl` drafts from a live page) | `aidemo init <name> [--from-url <url>]` |
|
|
31
|
+
| Draft from a Playwright trace / test | `import_trace` | `aidemo import-trace <trace.zip \| spec.ts> --name <name>` |
|
|
29
32
|
| Environment check | `doctor` | `aidemo doctor` |
|
|
33
|
+
| Discover selectors on a page | `inspect` (job) | `aidemo inspect <url> [--dir <dir>] [--frame name=sel]` |
|
|
30
34
|
| Dry-run the flow | `probe` (job) | `aidemo probe <dir>` |
|
|
31
35
|
| Full pipeline | `render` (job) | `aidemo render <dir>` |
|
|
32
36
|
| One stage | `voice` / `record` / `captions` / `compose` (jobs) | `aidemo voice\|record\|captions\|compose <dir>` |
|
|
33
37
|
| README GIF | `gif` (job) | `aidemo gif <dir>` |
|
|
34
38
|
| Named stills (screenshot mode) | `stills` (job) | `aidemo stills <dir>` |
|
|
39
|
+
| Frames for review | `frames` (job) | `aidemo frames <dir> [--every 3] [--source raw\|take]` |
|
|
40
|
+
| Walkthrough export (HTML + Markdown + frames + captions) | `walkthrough` (job) | `aidemo walkthrough <dir> [--lang] [--width]` |
|
|
35
41
|
| Job progress / result | `job_status`, `job_list`, `job_cancel` | (CLI runs block in the foreground) |
|
|
36
42
|
|
|
37
43
|
**The job model (MCP).** Pipeline operations touch TTS/STT, a real Chrome, or
|
|
@@ -47,9 +53,37 @@ instead.
|
|
|
47
53
|
**Always pass absolute demo directories** to MCP tools — the server's working
|
|
48
54
|
directory is not necessarily your repo (Codex registers servers globally).
|
|
49
55
|
|
|
50
|
-
**Validate early.** Run `validate_storyboard`
|
|
51
|
-
|
|
52
|
-
render job.
|
|
56
|
+
**Validate early.** Run `validate_storyboard` (CLI: `aidemo validate <dir>`,
|
|
57
|
+
non-zero exit on issues) after every storyboard edit — cheap, structured
|
|
58
|
+
errors — instead of discovering schema issues inside a render job.
|
|
59
|
+
|
|
60
|
+
**Inspect before you write a target.** `inspect {dir, url}` (CLI: `aidemo
|
|
61
|
+
inspect <url> --dir <dir>`) opens the page in the recording profile (so a
|
|
62
|
+
logged-in app shows its real UI), waits for it to settle, and returns every
|
|
63
|
+
visible interactive element — role, name, whether it's in the viewport — with
|
|
64
|
+
a ranked list of selectors that are **unique on that page right now**
|
|
65
|
+
(`[data-testid=…]` → `#id` → `[aria-label=…]` → `button:has-text("…")` →
|
|
66
|
+
`input[name=…]`/`[placeholder=…]` → `tag.class` → a short structural path),
|
|
67
|
+
plus the headings and any iframes (for the `frames` block; pass `frames` to
|
|
68
|
+
scan inside them). Use it instead of reading source or guessing: copy the
|
|
69
|
+
first selector of the element you mean. It writes `logs/inspect-<ts>.json`
|
|
70
|
+
and a screenshot next to it. Elements below the fold are listed too (marked
|
|
71
|
+
`↓` on the CLI); a `scrollTo` on them works as usual.
|
|
72
|
+
|
|
73
|
+
**Lint before you spend a take.** `lint_storyboard` (CLI: `aidemo lint <dir>`)
|
|
74
|
+
is a browser-free preflight that predicts what compose will do to each scene:
|
|
75
|
+
it estimates the action time from the ops (a `type` at human speed, a scroll's
|
|
76
|
+
easing, a `pause`, a `goto`) and compares it with the narration's word count at
|
|
77
|
+
the language's speaking rate. A scene whose actions finish long before the
|
|
78
|
+
voice does will be **held on a frozen frame** (`[scene-freeze]`, with the
|
|
79
|
+
predicted hold %); one whose actions run far past the voice will be **sped up
|
|
80
|
+
to the x1.6 ceiling and cut** (`[overrun-cut]`). It also flags the classic
|
|
81
|
+
selector/wait pitfalls (`type`+`Enter` with no wait behind it, `:text-is` on a
|
|
82
|
+
nested label, `focus` with no `zoom` block, an anchored `waitForChange` regex,
|
|
83
|
+
a scene that opens with a navigation click, `music.cue` no-ops). `validate`
|
|
84
|
+
prints the same findings; probe/record/render run it automatically (non-fatal)
|
|
85
|
+
so the forecast is in the log next to the take. The same numbers come back
|
|
86
|
+
**measured** in `output/report.json` after compose (see "Verify").
|
|
53
87
|
|
|
54
88
|
## Pipeline (what the engine does)
|
|
55
89
|
|
|
@@ -70,7 +104,16 @@ network.
|
|
|
70
104
|
one thing to prove, audience, tone, length, CTA. Otherwise infer sensible
|
|
71
105
|
defaults.
|
|
72
106
|
2. **Scaffold**: `init_demo` / `aidemo init <name>` → creates `demos/<name>/`
|
|
73
|
-
with a starter `generated/storyboard.json`.
|
|
107
|
+
with a starter `generated/storyboard.json`. When you already know the URL,
|
|
108
|
+
pass `fromUrl` (`--from-url`): the engine inspects the page first and the
|
|
109
|
+
draft uses its **real** headings as scenes and its **real** unique selectors
|
|
110
|
+
as beats (search box → `type`, first CTA → a `hover` you turn into the
|
|
111
|
+
click), with everything else it saw in `_candidates` and `input/brief.md`.
|
|
112
|
+
Repeated heading text gets an explicit `nth=`, and a page with no headings
|
|
113
|
+
at all (a single-page app that paints its own layout) drafts scroll beats
|
|
114
|
+
instead, so the skeleton still shows how long the page is. No LLM runs — you
|
|
115
|
+
still write the narration and choose the flow; you just never start from
|
|
116
|
+
`#search` placeholders.
|
|
74
117
|
3. **Confirm selectors for a new/changed flow.** Don't guess. The cheapest way
|
|
75
118
|
is a **probe** — a record-only dry run (narration optional) that drives the
|
|
76
119
|
real flow in ~90 s without spending TTS or a full take, so you can verify
|
|
@@ -78,7 +121,10 @@ network.
|
|
|
78
121
|
`storyboard.frames`. A probe that hits a bad selector leaves a screenshot +
|
|
79
122
|
frame dump in `logs/` (surfaced as `failureArtifacts` in `job_status`).
|
|
80
123
|
4. **Write the storyboard** (schema + principles below), then
|
|
81
|
-
`validate_storyboard
|
|
124
|
+
`validate_storyboard` and `lint_storyboard` — fix every `[scene-freeze]` /
|
|
125
|
+
`[overrun-cut]` the lint predicts *before* paying for voice: add on-screen
|
|
126
|
+
beats (hover, scroll, focus, a `pause`) to a scene that is all narration, or
|
|
127
|
+
trim the narration of a scene that is all action.
|
|
82
128
|
5. **Render**: run the `render` job (headed for real ChatGPT: keep
|
|
83
129
|
`headless: false` / drop `--headless`). Then inspect `output/final-demo.mp4`
|
|
84
130
|
— extract a few frames with ffmpeg and look at them.
|
|
@@ -89,7 +135,8 @@ network.
|
|
|
89
135
|
`compose` — voice **skips unchanged scenes** (hashes narration + voice
|
|
90
136
|
plan), so only the edited scene is re-voiced and an approved take is
|
|
91
137
|
preserved (`force` re-voices all; `scene <id>` targets one). Re-record only
|
|
92
|
-
if the browser flow changed
|
|
138
|
+
if the browser flow changed — and then only from the first changed scene
|
|
139
|
+
(`record --from-scene <id>` / `fromScene`): earlier scenes keep their take.
|
|
93
140
|
|
|
94
141
|
## Demo-director principles (make it feel human-made)
|
|
95
142
|
|
|
@@ -108,23 +155,60 @@ network.
|
|
|
108
155
|
video to its narration and freeze-holds a static page for any remainder — so
|
|
109
156
|
a scene that's mostly a pause should have shorter narration, and a busy scene
|
|
110
157
|
more.
|
|
158
|
+
- **Pre-navigate: put nav clicks at the END of the prior scene.** In a
|
|
159
|
+
screen-tour where each scene starts with a navigation click, the destination
|
|
160
|
+
appears 2–3 s late (the cursor glides across the *old* screen before the
|
|
161
|
+
click lands), desyncing narration from visuals — the narration for screen N
|
|
162
|
+
plays over screen N−1. Instead, end each scene with the click that opens the
|
|
163
|
+
*next* scene's screen: every scene then opens already on the screen its
|
|
164
|
+
narration describes.
|
|
111
165
|
- **End on a clear CTA.**
|
|
112
166
|
- **Voice plan per scene**: pick `voiceId` (marin/alloy/verse/… on the default
|
|
113
167
|
OpenAI TTS; an ElevenLabs voice id if the engine runs with
|
|
114
168
|
`AIDEMO_TTS_PROVIDER=elevenlabs`; a Kokoro voice id — af_heart, am_adam, … —
|
|
115
169
|
if it runs with `AIDEMO_TTS_PROVIDER=local`; both non-default providers
|
|
116
170
|
ignore `instructions`) and `instructions` to steer tone/emotion/pace. Keep it
|
|
117
|
-
consistent across scenes.
|
|
171
|
+
consistent across scenes. **`voice.pronounce`** (`{"aidemo":"A.I. demo",
|
|
172
|
+
"SQL":"sequel"}`, whole-word, case-sensitive, storyboard-level or per
|
|
173
|
+
scene) rewrites only what the TTS *hears* — the narration text, and so the
|
|
174
|
+
captions, keep the written form. Use it for product names, acronyms, and
|
|
175
|
+
version strings the voice mangles.
|
|
118
176
|
|
|
119
177
|
## Storyboard schema (quick reference)
|
|
120
178
|
|
|
121
179
|
The precise contract is the JSON Schema from `get_storyboard_schema`
|
|
122
180
|
(generated from the engine's own zod schema, `src/types.ts`).
|
|
123
181
|
|
|
124
|
-
Top level: `title`, `language?`, `targetLengthSeconds?`, `video{width,height}`
|
|
182
|
+
Top level: `title`, `language?`, `knownTerms?`, `targetLengthSeconds?`, `video{width,height}`
|
|
125
183
|
(default 1280x720), `frames{ name: iframeSelector }`,
|
|
126
|
-
`voice{voiceId,instructions,speed}` (default, scenes may override), `music?`,
|
|
127
|
-
`zoom?`, `intro?`, `outro?`, `transition?`, `output?`, `
|
|
184
|
+
`voice{voiceId,instructions,speed,pronounce?}` (default, scenes may override), `music?`,
|
|
185
|
+
`zoom?`, `intro?`, `outro?`, `transition?`, `hold?`, `output?`, `setup?`,
|
|
186
|
+
`attention?`, `keystrokes?`, `captions?`, `redact?`, `hide?`, `frame?`,
|
|
187
|
+
`brand?`, `scenes[]`.
|
|
188
|
+
|
|
189
|
+
`setup?` prepares the take before the first action — for cookie-gated or
|
|
190
|
+
fixture-rotating sites, so you never hand-write a Playwright seed script:
|
|
191
|
+
- `storageState`: path (relative to the demo dir) to a Playwright storageState
|
|
192
|
+
JSON (`{cookies:[…], origins:[{origin, localStorage:[{name,value}]}]}`, e.g.
|
|
193
|
+
saved by `context.storageState({path})`). Its cookies are added to the
|
|
194
|
+
profile and each origin's localStorage entries set before the storyboard
|
|
195
|
+
runs. CLI/MCP: `--storage-state <file>` / `storageState`.
|
|
196
|
+
- `cookies: [{name, value, domain, path?, secure?, httpOnly?, sameSite?,
|
|
197
|
+
expires?}]` — add cookies directly. CLI: `--cookie "name=value;domain=host"`
|
|
198
|
+
(repeatable, DevTools-style attributes; MCP: `cookies`).
|
|
199
|
+
- `preflight`: a shell command run from the demo dir before **every** take
|
|
200
|
+
(record/probe/render), after the profile is resolved and before Chrome
|
|
201
|
+
launches — "resolved" means the path is computed (and already wiped when
|
|
202
|
+
`--fresh`), with no browser holding it, so the hook may safely delete,
|
|
203
|
+
re-create and seed that directory — re-seed a fixture, or patch this storyboard for whichever variant
|
|
204
|
+
the app serves today (the storyboard is **re-read after the hook**). Env:
|
|
205
|
+
`AIDEMO_DEMO_DIR`, `AIDEMO_STORYBOARD`, `AIDEMO_PROFILE`. Non-zero exit
|
|
206
|
+
aborts the take with the hook's output.
|
|
207
|
+
- `expectState: true` — the profile is seeded on purpose; silence the
|
|
208
|
+
carried-over-state warning (implied by `storageState`/`cookies`; CLI/MCP:
|
|
209
|
+
`--profile-seeded` / `profileSeeded`).
|
|
210
|
+
Seeding composes with `--fresh`: the wipe happens first, then the seeds go in
|
|
211
|
+
— a clean identity that still carries exactly the cookie the gate needs.
|
|
128
212
|
|
|
129
213
|
`language?` is a BCP-47/ISO-639-1 code (e.g. `"et"`) describing what language
|
|
130
214
|
the base `narration` is already written in — set it for a monolingual
|
|
@@ -133,11 +217,20 @@ render switch: it only feeds the `captions` STT language hint (see below).
|
|
|
133
217
|
Distinct from `--lang`, which *selects* a scene's `narrations[code]`
|
|
134
218
|
translation (Multi-language renders, below).
|
|
135
219
|
|
|
220
|
+
`knownTerms?: string[]` is a glossary of product names, brands and jargon
|
|
221
|
+
Whisper reliably mis-hears (`["fitness.ee", "aidemo", "MaxFit"]`). They are
|
|
222
|
+
sent at the *front* of the STT prompt — Whisper only biases on roughly its
|
|
223
|
+
first 224 tokens, so a term buried in a long script never reaches it. Cheaper
|
|
224
|
+
than `captions --offline`, which fixes spelling but gives up real word timing.
|
|
225
|
+
|
|
136
226
|
Cinematic keys (all opt-in; omit for the plain look):
|
|
137
227
|
- `zoom: {scale?=1.55, easeMs?=600, holdMs?=1700}` — **auto-zoom on focus**:
|
|
138
228
|
every click/typed prompt zooms in on the interaction point at compose time,
|
|
139
229
|
holds, eases out; near-consecutive focus points pan instead of bouncing.
|
|
140
|
-
Set `"zoom": false` on a scene to opt just that scene out.
|
|
230
|
+
Set `"zoom": false` on a scene to opt just that scene out. On narrow
|
|
231
|
+
(mobile) viewports `scale` is capped so a zoom always leaves ≥360 logical px
|
|
232
|
+
of the page in frame — a desktop-tuned 1.35x otherwise crops a 430px take's
|
|
233
|
+
headline off the edge. Compose logs the cap when it applies.
|
|
141
234
|
- `intro` / `outro: {title, subtitle?, durationMs?=2600, background?, accent?,
|
|
142
235
|
fadeMs?=350}` — typographic title cards; narration/captions shift
|
|
143
236
|
automatically and music runs under the cards.
|
|
@@ -153,16 +246,59 @@ Cinematic keys (all opt-in; omit for the plain look):
|
|
|
153
246
|
`output.loudness` (see below).
|
|
154
247
|
- `transition: {type:"crossfade", durationMs?=400}` — **cross-dissolve every
|
|
155
248
|
scene boundary** instead of hard-cutting (see below).
|
|
249
|
+
- `hold: {mode?="freeze"|"drift", driftScale?=1.06, backoffMs?=0}` — what a
|
|
250
|
+
scene shows while the narration outlasts its actions: a frozen frame
|
|
251
|
+
(default) or a **slow Ken-Burns drift** so it never reads as a stall (see
|
|
252
|
+
below).
|
|
156
253
|
- `output: {width?, height?, fit?="contain", background?, loudness?}` — **render
|
|
157
254
|
at a different size/aspect** (`width`+`height`, set together) and/or **set the
|
|
158
255
|
master loudness** (`loudness`), e.g. a vertical social clip (see below).
|
|
159
256
|
- `motionBlur: {frames?=3}` — **subtle motion blur** on fast motion (cursor,
|
|
160
257
|
scroll, zoom pan); static UI stays sharp (see below).
|
|
161
|
-
- `cursor: {hidden?, hideScenes?, scale?}` — **compose-time
|
|
162
|
-
hide
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
`
|
|
258
|
+
- `cursor: {hidden?, hideScenes?, scale?, style?, color?}` — **compose-time
|
|
259
|
+
cursor control**: hide, resize or restyle (`"arrow"` | `"dot"`) the cursor
|
|
260
|
+
post-hoc instead of baking it (see below).
|
|
261
|
+
- `attention: {color?, clicks?}` — accent color for highlight/spotlight/callout
|
|
262
|
+
overlays; `clicks:true` draws a click ring wherever a click lands (needs the
|
|
263
|
+
`cursor` block). See *Attention*.
|
|
264
|
+
- `keystrokes: true` — show a keystroke chip ("⌘ K", "Enter") on every `press`
|
|
265
|
+
(per-action `keystrokes` overrides). See *Attention*.
|
|
266
|
+
- `captions: {position?="bottom"|"top", style?="pill"|"bar"|"none", font?,
|
|
267
|
+
size?, color?, background?}` — caption strip placement and look; scenes may
|
|
268
|
+
override `position` with their own `captions`. Compose also auto-flips a cue
|
|
269
|
+
to the top while an overlay occupies the bottom band. See *Attention* and
|
|
270
|
+
*Produced look*.
|
|
271
|
+
- `redact: [{selector, frame?, blur?}]` — **blur regions at compose time**
|
|
272
|
+
(prices, emails, tokens); scenes may add their own `redact`. See *Attention*.
|
|
273
|
+
- `hide: [selector, …]` — hide elements at **record** time (teasers, cookie
|
|
274
|
+
bars, ad slots); scenes may add their own `hide`. See *Attention*.
|
|
275
|
+
- `frame: {padding?, background?, radius?, shadow?, chrome?="none"|"browser"|"mac",
|
|
276
|
+
safeTop?, title?, url?}` — pad the video onto a styled canvas with a rounded,
|
|
277
|
+
shadowed window and optional browser chrome. See *Produced look*.
|
|
278
|
+
- `brand: {logo?, accent?, font?, watermark?}` — brand kit: accent for cards /
|
|
279
|
+
frame tint / attention overlays, font for cards + captions + callouts +
|
|
280
|
+
chrome, logo on cards and as a watermark. See *Produced look*.
|
|
281
|
+
- `autoIdle: true | {enabled?, minMs?=1500, noise?=0.003}` — **trim dead air
|
|
282
|
+
compose was never told about**: it scans the take for spans where nothing on
|
|
283
|
+
screen moves and caps them like an annotated wait, so a slow XHR or a long
|
|
284
|
+
`pause` stops inflating the scene. Opt-in; per-scene `autoIdle` overrides it
|
|
285
|
+
(`false` protects a scene that is deliberately still). Detected spans show up
|
|
286
|
+
as `autoIdleMs` per scene in `output/report.json`. Annotating the wait
|
|
287
|
+
(`waitForChange {idle:true}`, `waitForWidget`) is still better — it says WHY
|
|
288
|
+
the pause exists and lint can forecast it; `autoIdle` is the safety net for
|
|
289
|
+
waits you can't annotate. It only ever caps still footage, so a scene that is
|
|
290
|
+
deliberately motionless (a held product shot) wants `autoIdle: false`.
|
|
291
|
+
- `output: {preset?, chapters?, poster?, walkthrough?, width?, height?, fit?,
|
|
292
|
+
loudness?}` — presets `youtube|short|readme-gif|x`, MP4 chapter markers
|
|
293
|
+
from scene `title`s, `output/poster.png`, the walkthrough bundle. See
|
|
294
|
+
*Produced look* and *Transitions*.
|
|
295
|
+
|
|
296
|
+
Each scene: `id`, `title?` (chapter name), `narration`, `voice?`, `music?`,
|
|
297
|
+
`zoom?` (false to disable), `autoIdle?` (overrides the top-level setting),
|
|
298
|
+
`captions?`, `redact?`, `hide?`, `actions[]`.
|
|
299
|
+
Narration may carry `{{@name}}` anchor markers (see *Narration-anchored
|
|
300
|
+
beats*); the engine strips them at load and keeps the word index on the
|
|
301
|
+
scene as `anchors` (don't author that field).
|
|
166
302
|
|
|
167
303
|
## Transitions, output sizing & loudness
|
|
168
304
|
|
|
@@ -181,6 +317,20 @@ scene join is re-encoded (the default hard-cut path is a lossless stream-copy
|
|
|
181
317
|
concat). Needs ≥2 scenes; a shorter `durationMs` (250–400) reads as a snappy
|
|
182
318
|
dissolve, longer (600+) as a slow cinematic fade.
|
|
183
319
|
|
|
320
|
+
**`hold: {mode?, driftScale?, backoffMs?}`** — controls the **hold** compose
|
|
321
|
+
adds when a scene's narration runs longer than its (already x1.6-slowed)
|
|
322
|
+
recording. By default the last frame is frozen for the remainder
|
|
323
|
+
(`mode:"freeze"`, byte-identical to earlier versions). `mode:"drift"` replaces
|
|
324
|
+
the freeze with a slow, eased push-in on that frame (`driftScale`, 1–1.5,
|
|
325
|
+
default 1.06 ≈ 6% over the hold) — motion that reads as intentional rather
|
|
326
|
+
than a stalled screen. `backoffMs` (0–2000) takes the held frame from that
|
|
327
|
+
many ms *before* the scene's end instead of the very last frame, so a hold
|
|
328
|
+
never lands on a half-drawn transition, a spinner, or a white flash at the
|
|
329
|
+
tail of a navigation. Only scenes that actually hold are affected; a scene
|
|
330
|
+
whose actions fill the narration is untouched, and a scene with no hold is
|
|
331
|
+
byte-identical. The compose log and `output/report.json` list every scene's
|
|
332
|
+
hold length and percentage; the lint predicts them before the take.
|
|
333
|
+
|
|
184
334
|
**`output: {width, height, fit?, background?}`** — reframes the finished video
|
|
185
335
|
(cards and captions already baked in) to a target size, applied as the last
|
|
186
336
|
step. `fit`:
|
|
@@ -215,6 +365,202 @@ Override or disable via `loudness`:
|
|
|
215
365
|
Targets are the ffmpeg `loudnorm` `I` (LUFS), `TP` (dBTP), and `LRA` (LU). The
|
|
216
366
|
pass runs last over the muxed audio and pins the rate back to 44.1 kHz.
|
|
217
367
|
|
|
368
|
+
## Narration-anchored beats (`{{@name}}` + `anchor`)
|
|
369
|
+
|
|
370
|
+
By default compose stretches a whole scene by one factor so its take fills
|
|
371
|
+
the narration. That keeps things in sync on average, but the click you care
|
|
372
|
+
about still lands wherever the recording put it — often a second before or
|
|
373
|
+
after the word that describes it. Anchors pin a beat to a word:
|
|
374
|
+
|
|
375
|
+
```json
|
|
376
|
+
{
|
|
377
|
+
"id": "s3",
|
|
378
|
+
"narration": "Add the one you want, and it {{@drop}}drops straight into your basket.",
|
|
379
|
+
"actions": [
|
|
380
|
+
{ "op": "pause", "ms": 1200 },
|
|
381
|
+
{ "op": "click", "target": { "selector": "[data-testid=add-to-cart]" }, "anchor": "drop" }
|
|
382
|
+
]
|
|
383
|
+
}
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
- Write `{{@name}}` immediately before the word the beat should land on
|
|
387
|
+
(`[\w.-]+` names; the marker is stripped before TTS, captions and lint, so
|
|
388
|
+
the narration reads and sounds unchanged). Put `anchor: "name"` on the
|
|
389
|
+
action — a click, type, press, scrollTo, hover, or an attention beat. The
|
|
390
|
+
moment used is the click/press itself (the focus event), not the cursor
|
|
391
|
+
glide before it.
|
|
392
|
+
- Compose then retimes the scene **piecewise**: each stretch between anchors
|
|
393
|
+
gets its own factor so the action plays exactly as the word is spoken, and
|
|
394
|
+
the remainder of the scene fills the narration as before. Every piece stays
|
|
395
|
+
within the usual x0.5–x1.6 limits — if the beat still can't be reached, the
|
|
396
|
+
render succeeds and reports **`anchor-unreachable`** with the miss in ms
|
|
397
|
+
(`report.json` → scene `anchors[]`: `targetMs`, `landedMs`, `offMs`). The
|
|
398
|
+
fix is authoring: give the action more lead-in (a `pause` before it) when
|
|
399
|
+
it lands early, or trim what happens before it (mark waits idle, move the
|
|
400
|
+
marker later) when it lands late.
|
|
401
|
+
- Several anchors per scene are fine (`{{@add}} … {{@checkout}}`); they must
|
|
402
|
+
be in the same order as their actions. One marker per action; lint flags a
|
|
403
|
+
marker without an action (`anchor-unused`), an action without a marker
|
|
404
|
+
(`anchor-missing`) and a name reused on two actions (`anchor-duplicate`).
|
|
405
|
+
- Scenes without anchors are untouched (the single-piece path is the exact
|
|
406
|
+
pre-anchor arithmetic), so adding one anchor never shifts another scene.
|
|
407
|
+
- Multi-language: put markers in each `narrations[lang]` string too; the
|
|
408
|
+
engine keeps a per-language word index, so the beat lands on the translated
|
|
409
|
+
word.
|
|
410
|
+
- With STT captions the word time comes from the transcript (matched to the
|
|
411
|
+
script by normalized token, falling back to the proportional position); with
|
|
412
|
+
script-timed captions it comes from the same length-weighted model the
|
|
413
|
+
captions use. Re-run `captions` after changing a narration so the anchor
|
|
414
|
+
reads fresh timings.
|
|
415
|
+
|
|
416
|
+
## Attention: highlight, spotlight, callout, keystrokes, redact, hide
|
|
417
|
+
|
|
418
|
+
The attention layer is how a demo says *look here* without a voice-over
|
|
419
|
+
"as you can see". Everything except `hide` is **compose-time**: the player
|
|
420
|
+
measures the element's box during the take and logs a timeline marker; compose
|
|
421
|
+
rasterizes the mark (headless-Chrome PNG, baseline `overlay`) and draws it
|
|
422
|
+
**under the cursor and before the zoom**, so it rides the camera like page
|
|
423
|
+
content. Restyle, retime or remove a mark → recompose, never re-record. Every
|
|
424
|
+
key is opt-in; a storyboard without them renders exactly as before.
|
|
425
|
+
|
|
426
|
+
**Beats (actions):**
|
|
427
|
+
- `{op:"highlight", target, holdMs?=1600, style?}` — an outline around the
|
|
428
|
+
element. `style: {color?, thickness?=3, padding?=6, shape?="box"|"ring"}`.
|
|
429
|
+
- `{op:"spotlight", target, holdMs?=1600, dimTo?=0.55, padding?=10, style?}` —
|
|
430
|
+
dims the whole frame except the element (with a thin accent edge).
|
|
431
|
+
- `{op:"callout", target, text, placement?="auto"|"top"|"bottom"|"left"|"right",
|
|
432
|
+
holdMs?=2000, style?}` — a short label (≤80 chars) pinned to the element
|
|
433
|
+
with an arrow. `auto` picks below, then above, then right.
|
|
434
|
+
|
|
435
|
+
Each beat **dwells** at record time for `min(holdMs, 800)` ms so the eye
|
|
436
|
+
registers it before the next action; the overlay itself stays for `holdMs` of
|
|
437
|
+
*content* time (clipped to the scene). Pair a beat with the narration that
|
|
438
|
+
names the thing: `highlight` the field right before you `type` into it,
|
|
439
|
+
`spotlight` the result the narration explains, `callout` the one number that
|
|
440
|
+
matters. Don't stack more than one beat per sentence.
|
|
441
|
+
|
|
442
|
+
**Keystroke chips.** `keystrokes: true` (top level) or `keystrokes: true` on a
|
|
443
|
+
`press` shows a chip bottom-right ("⌘ K", "Enter", "Esc", "⇧ Tab") for ~0.9 s
|
|
444
|
+
after the key goes down. Shortcuts-driven products (command palettes, editors)
|
|
445
|
+
read far better with it; typing (`type`) never shows chips — the text on
|
|
446
|
+
screen is the feedback.
|
|
447
|
+
|
|
448
|
+
**Click rings.** `attention: {clicks: true}` with a `cursor` block draws a
|
|
449
|
+
three-step ring where every click lands (the compose-time twin of the baked
|
|
450
|
+
cursor's ripple). `attention.color` sets the accent for rings and beats
|
|
451
|
+
(default `#ff5a5f`).
|
|
452
|
+
|
|
453
|
+
**Cursor style.** `cursor: {style:"dot", color?, scale?}` draws a soft dot
|
|
454
|
+
instead of the arrow — the screen-recorder look; the dot is centered on the
|
|
455
|
+
recorded point, the arrow's tip is.
|
|
456
|
+
|
|
457
|
+
**Captions placement.** `captions: {position:"top"}` (top level or per scene)
|
|
458
|
+
moves the strip to the top edge. Without a per-scene override, compose
|
|
459
|
+
**auto-flips** any cue whose window overlaps a highlight/spotlight/callout
|
|
460
|
+
box or a keystroke chip sitting in the bottom band, so a mark and a caption
|
|
461
|
+
never collide (`report.json` → `attention.captionsFlipped`).
|
|
462
|
+
|
|
463
|
+
**Redact (compose-time blur).** `redact: [{selector, frame?, blur?=14}]` at
|
|
464
|
+
the top level and/or per scene. After every action the player re-measures
|
|
465
|
+
each selector (all matches, up to 8) and logs a span per box; compose crops,
|
|
466
|
+
blurs and overlays each span back — prices, emails, API keys, customer names
|
|
467
|
+
never ship, and the blur strength is a recompose. Boxes that move (scrolls,
|
|
468
|
+
re-layouts) start a new span at the next action; a region that moves *during*
|
|
469
|
+
a scroll is blurred at its post-scroll position, so put the scroll before the
|
|
470
|
+
data appears or blur a container rather than a cell.
|
|
471
|
+
|
|
472
|
+
**Hide (record-time).** `hide: ["footer", ".promo-banner"]` at the top level
|
|
473
|
+
(injected before any page script, survives navigations) and/or per scene
|
|
474
|
+
(applied at scene start, removed at scene end) sets `visibility:hidden` on
|
|
475
|
+
the matches — for teasers, chat widgets, cookie bars, and ad slots that would
|
|
476
|
+
photobomb the take. This is the **one** record-time exception to
|
|
477
|
+
compose-time polish: hiding is cheap to redo, and a hidden element cannot be
|
|
478
|
+
un-hidden by a recompose, so keep the list to things you'd never want in any
|
|
479
|
+
cut. Lint-check with a probe: hidden elements still take their layout space.
|
|
480
|
+
|
|
481
|
+
## Produced look: frame, brand, caption styles, presets, chapters, poster
|
|
482
|
+
|
|
483
|
+
Everything here is compose-time and opt-in — a storyboard without these keys
|
|
484
|
+
renders exactly as before, and every tweak is a recompose. Frame and cards
|
|
485
|
+
are headless-Chrome-rasterized PNGs over core `pad`/`overlay` filters, so the
|
|
486
|
+
look is portable across ffmpeg builds. Read the report (`output/report.json`)
|
|
487
|
+
and `aidemo frames` after the first render with a frame: the canvas is bigger
|
|
488
|
+
than the recording, so check that captions and chips sit where you expect.
|
|
489
|
+
|
|
490
|
+
**`frame`** — the "Screen Studio" look: the recording becomes a rounded,
|
|
491
|
+
shadowed window on a styled canvas, with captions in the padding band below.
|
|
492
|
+
```json
|
|
493
|
+
"frame": { "chrome": "mac", "url": "app.example.com", "padding": 56, "radius": 14 }
|
|
494
|
+
```
|
|
495
|
+
- `padding` (default 48, logical px) grows the canvas on every side; the
|
|
496
|
+
canvas is `(width + 2·padding) × (height + 2·padding [+ 40 chrome])`. With
|
|
497
|
+
`output.preset`/`width`/`height` set, the padding grows on one axis so the
|
|
498
|
+
canvas already has the target aspect — the resize is a pure scale, never
|
|
499
|
+
letterbox bars around the frame.
|
|
500
|
+
- `background`: any CSS color/gradient. Default: a dark gradient tinted with
|
|
501
|
+
`brand.accent` when set.
|
|
502
|
+
- `radius` (default 14), `shadow` (default true).
|
|
503
|
+
- `chrome`: `"browser"` (neutral dots + address pill) or `"mac"` (traffic
|
|
504
|
+
lights). `url` (wins) or `title` fills the pill; omit both for an empty bar.
|
|
505
|
+
For a phone-sized take (`video: {width: 390, height: 844}` — record at the
|
|
506
|
+
mobile viewport) use `"iphone"` (rounded bezel + dynamic island) or
|
|
507
|
+
`"android"` (bezel + punch-hole): no bar, `radius` defaults to 40, and the
|
|
508
|
+
camera cut-out sits over the top of the video, so keep the app's own status
|
|
509
|
+
bar area clear or accept it being covered. Pair with `output.preset:
|
|
510
|
+
"short"` for a 9:16 export — the padding grows to fill the aspect.
|
|
511
|
+
- Zoom, cursor, attention overlays and redaction all happen **before** the
|
|
512
|
+
frame, so they ride inside the window; captions, keystroke chips, cards and
|
|
513
|
+
the watermark render at the canvas size.
|
|
514
|
+
|
|
515
|
+
**`brand`** — one place for identity, applied everywhere it should show:
|
|
516
|
+
```json
|
|
517
|
+
"brand": { "logo": "brand/logo.png", "accent": "#3b82f6", "font": "Inter, sans-serif",
|
|
518
|
+
"watermark": { "position": "bottom-right", "opacity": 0.85, "scale": 0.11 } }
|
|
519
|
+
```
|
|
520
|
+
- `accent` → intro/outro rule color (unless the card sets its own), the frame
|
|
521
|
+
gradient tint, and the default attention color (`attention.color` wins).
|
|
522
|
+
- `font` → cards, captions (`captions.font` wins), callouts, frame chrome. Use
|
|
523
|
+
a family that exists on the render machine; the fallback is the system sans.
|
|
524
|
+
- `logo` (png/svg/jpg, relative to the demo dir): drawn above the rule on
|
|
525
|
+
intro/outro cards and, unless `watermark.enabled:false`, as a watermark over
|
|
526
|
+
the content at `position` with `opacity` (0–1) and `scale` (fraction of the
|
|
527
|
+
frame width, default 0.11). A missing file is a compose warning
|
|
528
|
+
(`brand-logo-missing`), not a failure.
|
|
529
|
+
|
|
530
|
+
**Caption styles** (`captions`, top level): `style` is `"pill"` (default —
|
|
531
|
+
centered rounded box), `"bar"` (full-width band flush with the frame edge; the
|
|
532
|
+
usual choice under a `frame`), or `"none"` (no captions burned — the
|
|
533
|
+
SRT/VTT files are still written, for players and platforms that render their
|
|
534
|
+
own). `font`, `size` (default 30, logical px), `color`, `background` (any
|
|
535
|
+
CSS) restyle the strip; `position` still flips per scene / automatically.
|
|
536
|
+
|
|
537
|
+
**Output presets** (`output.preset`) fill `width`/`height`/`fit` when you
|
|
538
|
+
don't set them explicitly: `youtube` 1920×1080, `short` 1080×1920,
|
|
539
|
+
`readme-gif` 960×540, `x` 1280×720 — all `fit:"contain"` (letterbox, never
|
|
540
|
+
crop). Explicit `width`/`height`/`fit` win over the preset.
|
|
541
|
+
|
|
542
|
+
**Chapters** (`output.chapters: true`): the MP4 gets one chapter marker per
|
|
543
|
+
scene at its content start (after the intro card), named from the scene's
|
|
544
|
+
`title` (or its `id`). YouTube, QuickTime, VLC and mpv show them as a
|
|
545
|
+
navigable table of contents. Give every scene a short `title` when you turn
|
|
546
|
+
this on.
|
|
547
|
+
|
|
548
|
+
**Walkthrough** (`output.walkthrough: true` — honoured by `render` *and* by a
|
|
549
|
+
standalone `compose` — or `aidemo walkthrough <dir>` / the `walkthrough` job any
|
|
550
|
+
time after a render): one take → several
|
|
551
|
+
artifacts in `output/walkthrough/`: `index.html` (one card per scene with
|
|
552
|
+
its payoff frame, title, narration and a jump-to-time button over the video;
|
|
553
|
+
← / → keys step through, Enter plays from a step), `guide.md` (the same as a
|
|
554
|
+
README/SOP section: heading + frame + narration per scene), `scene-NN-<id>.png`
|
|
555
|
+
frames, the SRT/VTT captions, and `walkthrough.json`. Frames come from the
|
|
556
|
+
**final** video, so they carry zoom, cursor, overlays and the frame. Scene
|
|
557
|
+
`title`s name the steps (falling back to ids).
|
|
558
|
+
|
|
559
|
+
**Poster** (`output.poster: true`): also writes `output/poster.png`
|
|
560
|
+
(`poster.<lang>.png` for language variants) — the first content frame after
|
|
561
|
+
the intro card, for READMEs, social cards and `<video poster>`. The path is
|
|
562
|
+
in `report.json` as `poster`.
|
|
563
|
+
|
|
218
564
|
## Motion blur & cursor
|
|
219
565
|
|
|
220
566
|
Two more compose-time polish keys, both opt-in and portable (baseline `tmix` /
|
|
@@ -246,22 +592,57 @@ baked at record time as before (the default).
|
|
|
246
592
|
|
|
247
593
|
## Action vocabulary
|
|
248
594
|
|
|
249
|
-
A `target` is `{selector}` or `{frame,selector}` or `{named:"composer"}
|
|
250
|
-
|
|
595
|
+
A `target` is `{selector}` or `{frame,selector}` or `{named:"composer"}`.
|
|
596
|
+
Every action also accepts `comment?`, `optional?` (best-effort, see below),
|
|
597
|
+
`retry?` (0–5 extra attempts, interactions and `assert` only, a beat apart —
|
|
598
|
+
for UI that re-renders under the cursor) and `anchor?` (land this action on a
|
|
599
|
+
`{{@name}}` word in the narration — see *Narration-anchored beats*):
|
|
600
|
+
- `{op:"goto", url}` — waits for `domcontentloaded`, then (capped at ~3.4 s)
|
|
601
|
+
for the network to go quiet and fonts to load; that extra wait is recorded
|
|
602
|
+
as trimmable idle (`"load"`), so a first click never lands on a skeleton
|
|
603
|
+
screen and the take doesn't get longer for it.
|
|
251
604
|
- `{op:"type", target, text, humanize?}` — human-cadence typing
|
|
252
605
|
- `{op:"press", key}` — e.g. "Enter"
|
|
253
|
-
- `{op:"click", target}` · `{op:"hover", target}`
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
606
|
+
- `{op:"click", target, followPopup?}` · `{op:"hover", target}` —
|
|
607
|
+
`followPopup:true` handles a link/button that opens a **new tab**
|
|
608
|
+
(`target=_blank`, `window.open`): the tab is closed and the recorded tab
|
|
609
|
+
navigates to its URL. The take is one window — a second tab is never in the
|
|
610
|
+
video, so there is no `newTab`/`switchTab`; write the story in one tab.
|
|
611
|
+
- `{op:"back"}` — browser history back, same readiness wait as `goto`.
|
|
612
|
+
- `{op:"select", target, value? | label?}` — native `<select>`: the cursor
|
|
613
|
+
clicks it, the option is committed programmatically (the OS dropdown never
|
|
614
|
+
paints into a recording), `change` fires.
|
|
615
|
+
- `{op:"drag", target, to:{target} | to:{x,y}}` — press on `target`, glide
|
|
616
|
+
with the cursor, release on the destination (pointer-event and HTML5 DnD
|
|
617
|
+
libraries both see a real drag; the cursor path is recorded).
|
|
618
|
+
- `{op:"upload", target, files:[…]}` — attach files: `target` is the file
|
|
619
|
+
input (set directly) or the button that opens the file chooser (clicked; the
|
|
620
|
+
chooser is answered headlessly, no OS dialog in the take). Paths resolve
|
|
621
|
+
against the demo dir (`input/sample.csv`); a missing file fails the take.
|
|
622
|
+
- `{op:"scrollTo", target, easing?, durationMs?, state?, settleMs?}` ·
|
|
623
|
+
`{op:"scrollBy", dy, target?, easing?, durationMs?, settleMs?}` — easing
|
|
624
|
+
presets: `"smooth"` (default) | `"snappy"` | `"glide"` | `"linear"`.
|
|
625
|
+
`scrollBy` with a `target` is **scoped to that element's own scroller**: the
|
|
626
|
+
wheel is dispatched over it and `dy` is clamped to the room its nearest
|
|
627
|
+
scrollable ancestor has left, so a bottomed-out inner panel never chains the
|
|
628
|
+
wheel to the page and slides the whole app off-screen (the log shows the
|
|
629
|
+
clamp). `settleMs` waits until the scroll position has been still for that
|
|
630
|
+
long (≤3 s) instead of a fixed pause — use it on smooth-scrolling pages
|
|
631
|
+
before a click. `scrollTo … state:"attached"` accepts a zero-size mount
|
|
632
|
+
point (see `waitFor`).
|
|
257
633
|
- `{op:"focus", target, scale?, holdMs?}` — deliberate zoom beat on an element
|
|
258
634
|
without clicking it (needs top-level `zoom` enabled)
|
|
259
635
|
- `{op:"still", name}` — **screenshot mode**: mark a named still at this beat.
|
|
260
636
|
A pure timeline marker (no screenshot at record time); `aidemo stills` /
|
|
261
637
|
`render` extract `output/stills/<name>.png` from the clean take. See *Stills /
|
|
262
638
|
screenshot mode* below.
|
|
263
|
-
- `{op:"waitFor", target, timeoutMs?}` — normal wait (fires instantly
|
|
264
|
-
selector already matches — no good for in-place changes)
|
|
639
|
+
- `{op:"waitFor", target, timeoutMs?, state?}` — normal wait (fires instantly
|
|
640
|
+
if the selector already matches — no good for in-place changes). Default
|
|
641
|
+
waits for **visible**; `state:"attached"` only requires the element to be in
|
|
642
|
+
the DOM — for a lazily-filled empty `<div>` (0 px tall until something
|
|
643
|
+
mounts into it on scroll), which is never "visible" and would time out. A
|
|
644
|
+
timeout names the total budget waited (`not visible after 30000ms (budget
|
|
645
|
+
30000ms)`), not Playwright's last poll chunk.
|
|
265
646
|
- `{op:"waitForWidget", target, textMatches?, label?, timeoutMs?}` — **records
|
|
266
647
|
the wait as idle** so compose trims/speeds it. Use for every ChatGPT
|
|
267
648
|
"thinking" wait for a brand-**new** widget. When a prompt could render
|
|
@@ -287,12 +668,47 @@ A `target` is `{selector}` or `{frame,selector}` or `{named:"composer"}`:
|
|
|
287
668
|
`"2"` for a 1→2 re-render. `idle:true` records it as trimmable idle like
|
|
288
669
|
`waitForWidget`.
|
|
289
670
|
- `{op:"pause", ms}` — a deliberate on-screen beat (not trimmed)
|
|
671
|
+
- `{op:"moveTo", target}` · `{op:"moveTo", x, y}` — glide the cursor to an
|
|
672
|
+
element's center or an absolute viewport point **without clicking**: park it
|
|
673
|
+
out of the way before a reveal, or point at what the narration names.
|
|
674
|
+
- `{op:"assert", target?, textMatches?, url?, timeoutMs?=5000}` — **prove the
|
|
675
|
+
payoff happened**: polls until `target` is visible (and its text matches
|
|
676
|
+
`textMatches`, JS regex, no inline flags — use `[Cc]onfirmed`) and/or the
|
|
677
|
+
page URL matches `url`; otherwise **fails the take with a named error**
|
|
678
|
+
(`assert failed after 5000ms: text "…" does not match /…/`). Put one after
|
|
679
|
+
the moment the demo exists to show (order confirmed, item added) so a
|
|
680
|
+
spinner never ships as a video. `optional:true` turns a miss into a logged
|
|
681
|
+
skip. Pair it with `select`/`drag`/`upload`: assert the visible result
|
|
682
|
+
(`"3 files attached"`, the dropped card) — the actions themselves only prove
|
|
683
|
+
the input was delivered.
|
|
290
684
|
|
|
291
685
|
Target `last`/`nth` pick among matching **frames** on framed targets and among
|
|
292
686
|
matching **elements** on plain (frameless) targets — e.g.
|
|
293
687
|
`{selector:"[data-message-author-role=\"assistant\"]", last:true}` is the
|
|
294
688
|
newest reply.
|
|
295
689
|
|
|
690
|
+
**Selector gotcha — nested label text:** for buttons whose label sits in a
|
|
691
|
+
nested text node (common in react-native-web, MUI, and icon+label buttons like
|
|
692
|
+
`<button role="button"><svg/><div>Library</div></button>`), prefer
|
|
693
|
+
`:has-text("Library")` over `:text-is("Library")`. Playwright's `:text-is`
|
|
694
|
+
matches the *smallest* element whose text equals the string — the inner
|
|
695
|
+
`<div>`, not the clickable wrapper — so
|
|
696
|
+
`[role="button"]:text-is("Library")` matches **0 elements** and fails the
|
|
697
|
+
take, while `[role="button"]:has-text("Library")` matches the button (the
|
|
698
|
+
player takes `.first()`).
|
|
699
|
+
|
|
700
|
+
Any action also accepts **`optional: true`** — best-effort: when its target
|
|
701
|
+
never appears or the interaction fails, the player logs and continues instead
|
|
702
|
+
of failing the take. Use it for state-dependent one-time UI, e.g. collapsing
|
|
703
|
+
the ChatGPT sidebar persists in the Chrome profile, so a re-run finds the
|
|
704
|
+
close button absent and a required click would kill the run. Optional
|
|
705
|
+
interactions (click/type/hover/scrollTo/focus) probe ~3 s for the target
|
|
706
|
+
instead of stalling out the full action timeout; optional waits honor their
|
|
707
|
+
own `timeoutMs` and then continue. Don't mark actions the demo's visuals
|
|
708
|
+
depend on — a skipped optional action is only a log line. (In `--golden`
|
|
709
|
+
probes, optional actions always record `ok: true` and no `found`, keeping the
|
|
710
|
+
baseline deterministic.)
|
|
711
|
+
|
|
296
712
|
Typing into the composer while the prior reply is still streaming is safe: the
|
|
297
713
|
player waits the composer out and **records that wait as trimmable idle**, so a
|
|
298
714
|
long streaming turn can't blow the next scene past its narration.
|
|
@@ -311,7 +727,23 @@ storyboard viewport, and **aborts** when the computed crop doesn't match the
|
|
|
311
727
|
viewport instead of shipping a mis-cropped take
|
|
312
728
|
(`AIDEMO_NATIVE_CROP_UNSAFE=1` downgrades the abort to a loud warning — only
|
|
313
729
|
for debugging, and then review every frame). No automatic check can see
|
|
314
|
-
everything, so eyeball the frames anyway.
|
|
730
|
+
everything, so eyeball the frames anyway.
|
|
731
|
+
|
|
732
|
+
**⚠ The wrong-window failure mode (and the guard against it).** Perfect
|
|
733
|
+
geometry still doesn't prove the right *pixels* were recorded: a screen grab
|
|
734
|
+
captures whatever is painted at the crop rectangle, so if the driven browser
|
|
735
|
+
window is occluded by another window, sitting on a different display than the
|
|
736
|
+
captured one, or covered by a notification, the take shows *someone else's
|
|
737
|
+
window* — the automation itself succeeds (selectors match, timeline is
|
|
738
|
+
written), so nothing else fails. That's both silent garbage and a privacy
|
|
739
|
+
hazard (whatever was on screen — a password manager, DMs, an OTP page — ships
|
|
740
|
+
in the "publish-ready" video). The recorder raises the browser window before
|
|
741
|
+
capture and, after cropping, **verifies the captured content against a CDP
|
|
742
|
+
screenshot of the page** taken at stop time, aborting on a mismatch (same
|
|
743
|
+
`AIDEMO_NATIVE_CROP_UNSAFE=1` escape hatch). Keep the browser window
|
|
744
|
+
front-most and unoccluded on the captured display for the whole take — or use
|
|
745
|
+
the default `--capture playwright`, which records the page buffer directly
|
|
746
|
+
and is immune to occlusion and to anything else on your screen. `record` also preserves the
|
|
315
747
|
previous take as one `.prev` generation (`recordings/raw.prev.*` +
|
|
316
748
|
`generated/timeline.prev.json`) instead of deleting it, so a bad re-record
|
|
317
749
|
never destroys the last good take — copy the `.prev` files back over the
|
|
@@ -370,17 +802,39 @@ Confirmed by recording a real production shopping app (2026-07-06). The engine
|
|
|
370
802
|
already handles the frame/timing/stealth cases below; you mostly need to author
|
|
371
803
|
correctly.
|
|
372
804
|
|
|
805
|
+
**Profile state is the silent failure.** The recording profile persists cookies
|
|
806
|
+
and localStorage between takes — including anything a `probe` just did. A demo
|
|
807
|
+
whose story starts at a first-run gate, an onboarding step, an empty state or a
|
|
808
|
+
one-shot flow will *skip that beat* on the next take and record the opposite of
|
|
809
|
+
what the narration says, with nothing failing and nothing to see in review.
|
|
810
|
+
- Record such demos with **`--fresh`** (`record`/`probe`/`render`): the take
|
|
811
|
+
runs against a wiped throwaway profile in the demo dir, so every run starts
|
|
812
|
+
from a clean browser identity. Not for logged-in demos — a fresh profile has
|
|
813
|
+
no login.
|
|
814
|
+
- Otherwise `record` warns when the profile already holds state for the
|
|
815
|
+
storyboard's first `goto` origin. When that state is deliberate (a login, a
|
|
816
|
+
cookie gate you seeded), acknowledge it with `--profile-seeded` /
|
|
817
|
+
`setup.expectState: true` — seeding via `setup.storageState`/`cookies`
|
|
818
|
+
implies it — so the log stays readable.
|
|
819
|
+
- **`aidemo profile path`** prints the shared profile's location (handy when
|
|
820
|
+
the engine came from Homebrew or an npx cache); **`aidemo profile reset`**
|
|
821
|
+
wipes it.
|
|
822
|
+
|
|
373
823
|
**Login & profile (the crux).**
|
|
374
824
|
- Use a dedicated Chrome profile logged into ChatGPT with the app's dev
|
|
375
825
|
connector enabled (`AIDEMO_CHROME_PROFILE` or the `profile` option). Run
|
|
376
|
-
**headed**; quit any Chrome on that profile first (Playwright needs the
|
|
826
|
+
**headed**; quit any Chrome on that profile first (Playwright needs the
|
|
827
|
+
lock — `record` detects a still-running Chrome on the profile and aborts
|
|
828
|
+
with a clear message, and cleans up a stale lock from a crashed Chrome).
|
|
377
829
|
- You **cannot copy** an existing ChatGPT login in: real Chrome encrypts
|
|
378
830
|
cookies with the macOS Keychain, but Playwright launches with
|
|
379
831
|
`--use-mock-keychain --password-store=basic`, so copied cookies won't
|
|
380
832
|
decrypt. Log in **once** in a Chrome that uses the *same* mock store, then
|
|
381
|
-
quit and record:
|
|
833
|
+
quit and record. **`aidemo login [profile]`** does this for you: it opens a
|
|
834
|
+
normal Chrome on the recording profile with those flags, waits for you to
|
|
835
|
+
log in and quit, then reports the profile ready. (Manual equivalent:
|
|
382
836
|
`open -na "Google Chrome" --args --user-data-dir=<profile> --password-store=basic --use-mock-keychain https://chatgpt.com/`
|
|
383
|
-
|
|
837
|
+
— a normal window like this also dodges Google's SSO automation block.)
|
|
384
838
|
- Cloudflare: the engine launches with
|
|
385
839
|
`--disable-blink-features=AutomationControlled` so `navigator.webdriver` is
|
|
386
840
|
false and the profile's real `cf_clearance` is honored (otherwise you get a
|
|
@@ -422,10 +876,17 @@ Always set `"last": true` on widget targets (newest widget for this turn).
|
|
|
422
876
|
End the scene on the visible hand-off CTA and let narration cover the jump.
|
|
423
877
|
(Recorder keeps the largest `.webm` so the stray new-tab recording is
|
|
424
878
|
discarded.)
|
|
425
|
-
- Collapse the sidebar (`[data-testid="close-sidebar-button"]
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
879
|
+
- Collapse the sidebar (`[data-testid="close-sidebar-button"]`) to hide
|
|
880
|
+
chat-history PII and clean the frame — and mark that click **`optional:
|
|
881
|
+
true`**: the collapsed state persists in the Chrome profile, so on a re-run
|
|
882
|
+
the close button is absent and a required click would fail the take. Note
|
|
883
|
+
the home screen greets the account by first name for ~2 s — blur or accept
|
|
884
|
+
per the owner.
|
|
885
|
+
- **Takes mutate real app state.** Every probe and take drives the live app —
|
|
886
|
+
a `log_meal` demo actually logs a meal, an `add_to_cart` demo actually fills
|
|
887
|
+
a cart. Probe + a few takes will drift your app's data; reset/re-seed your
|
|
888
|
+
fixture or test account between the probe and the take (and between takes)
|
|
889
|
+
if the demo depends on its starting state.
|
|
429
890
|
- **Widget-side mutations render in the SAME widget, not a new one** (v2
|
|
430
891
|
recording, 2026-07-06): click-Add renders a basket bar into the widget's
|
|
431
892
|
`[data-cart-bar]` slot; the cart's `±` re-renders the row in place. Wait with
|
|
@@ -470,8 +931,25 @@ Always set `"last": true` on widget targets (newest widget for this turn).
|
|
|
470
931
|
`logs/fail-<scene>-<n>.{png,json}` (screenshot + which frames matched).
|
|
471
932
|
- **A failed take is salvaged, not lost**: `record` writes a partial
|
|
472
933
|
`timeline.json` and keeps the main recording even when a late scene fails, so
|
|
473
|
-
a scene-7-of-7 failure doesn't discard the good footage.
|
|
474
|
-
|
|
934
|
+
a scene-7-of-7 failure doesn't discard the good footage. Then **resume
|
|
935
|
+
instead of re-recording**: `record {fromScene: "s7"}` (CLI `record <dir>
|
|
936
|
+
--from-scene s7`, also on `render`) keeps the earlier scenes' footage and
|
|
937
|
+
timeline, replays their actions at speed only to rebuild the app state, and
|
|
938
|
+
records from `s7` on. Reuse is guarded by a per-scene hash of the actions
|
|
939
|
+
(+ hide/redact/viewport/cursor mode/`setup`/`params`): an edited earlier
|
|
940
|
+
scene refuses with "resume from `<id>` or earlier". A resume must also keep
|
|
941
|
+
the same capture mode and viewport as the take it continues — compose refuses
|
|
942
|
+
a timeline whose raw files differ in size rather than misplacing every
|
|
943
|
+
overlay on the reused scenes.
|
|
944
|
+
- **Flows you cannot replay**: when the demo *earns* one-shot state (a balance
|
|
945
|
+
goes 10 → 20, an invite is consumed, a coupon burns), `--from-scene` is not a
|
|
946
|
+
speed optimisation — it is the only correct retry. A full re-record replays
|
|
947
|
+
actions against an app that has already moved on, so it either fails or
|
|
948
|
+
records the wrong numbers; resuming keeps the earned footage and picks up at
|
|
949
|
+
the scene that broke. Pair it with `setup.expectState: true` so the
|
|
950
|
+
carried-over-state warning stays quiet on the seeded profile.
|
|
951
|
+
The same flag is the cheap way to re-shoot
|
|
952
|
+
a tail you changed after an approved take.
|
|
475
953
|
|
|
476
954
|
`init_demo` / `aidemo init` scaffolds a storyboard already using all of the
|
|
477
955
|
above.
|
|
@@ -553,8 +1031,31 @@ renders relative to its own dir, keep any local asset paths (e.g. a music
|
|
|
553
1031
|
|
|
554
1032
|
## Verify before declaring done
|
|
555
1033
|
|
|
556
|
-
|
|
557
|
-
|
|
1034
|
+
**A selector that matched nothing** fails the take with drift suggestions:
|
|
1035
|
+
the failure log (and `logs/fail-<scene>-<n>.json`, `driftCandidates`) lists
|
|
1036
|
+
the interactive elements that most resemble what the selector asked for —
|
|
1037
|
+
by testid / id / text similarity — each with a unique selector to paste, and
|
|
1038
|
+
`logs/drift-<scene>-<n>.json` holds the full ranked list. If nothing similar
|
|
1039
|
+
is on the page, you are on the wrong screen or state (a gate, a stale
|
|
1040
|
+
profile, a navigation that didn't land): check the screenshot first.
|
|
1041
|
+
|
|
1042
|
+
Start with **`output/report.json`** (written by every compose; the `render` /
|
|
1043
|
+
`compose` job results carry `report` and `warnings`): per scene it records
|
|
1044
|
+
the recorded length, the narration target, the retime `factor` applied, and
|
|
1045
|
+
`holdMs`/`holdPct` (how much of the scene is a held frame), plus the tail and
|
|
1046
|
+
blank-frame trims, `autoIdleMs` (motionless footage `autoIdle` capped, 0 when
|
|
1047
|
+
off), focus events kept vs dropped, and `warnings[]` —
|
|
1048
|
+
`scene-freeze` (a scene is >40% held frame), `overrun-trim` (actions ran past
|
|
1049
|
+
the narration even at the x1.6 ceiling and were cut), `blank-tail`,
|
|
1050
|
+
`stale-captions`, `focus-dropped`, `cursor-missing`, `anchor-unreachable`
|
|
1051
|
+
(an anchored action could not reach its word within the retime limits; the
|
|
1052
|
+
scene's `anchors[]` shows `targetMs` / `landedMs` / `offMs`). A held scene
|
|
1053
|
+
is a storyboard problem, not a compose problem: give it on-screen beats or
|
|
1054
|
+
shorten its narration, then re-run `voice` + `compose`.
|
|
1055
|
+
|
|
1056
|
+
Then play (or frame-extract — `aidemo frames <dir>` / the `frames` job)
|
|
1057
|
+
`output/final-demo.mp4`: cursor glides and clicks pulse, narration matches
|
|
1058
|
+
on-screen actions, captions are readable and in sync,
|
|
558
1059
|
no dead air, and the key moment (e.g. checkout confirmed) is actually visible
|
|
559
1060
|
on screen. If the demo is headed for a README, run the `gif` job — GIFs
|
|
560
1061
|
autoplay on GitHub; MP4s don't.
|
|
@@ -566,15 +1067,72 @@ CLI. If nothing came up, skip this.
|
|
|
566
1067
|
|
|
567
1068
|
## Debugging
|
|
568
1069
|
|
|
1070
|
+
- **`generated/timeline.json` → `scenes[].actions[]`** is the per-action
|
|
1071
|
+
record of the take: `{index, op, target, startMs, endMs, ok, skipped?,
|
|
1072
|
+
retries?, warnings?[]}`. Read it to see which action ate the time, which
|
|
1073
|
+
optional action was skipped and why, and which action window saw failed
|
|
1074
|
+
requests (`"2 failed request(s): 500 POST /api/cart …"`) — the record-side
|
|
1075
|
+
twin of `output/report.json`.
|
|
569
1076
|
- Every run tees its output to `<demo>/logs/<command>.log`; a failed take also
|
|
570
1077
|
leaves `logs/fail-<scene>-<n>.{png,json}`. `job_status` surfaces all of these
|
|
571
1078
|
paths on failure.
|
|
1079
|
+
- The fail JSON (and the error text) carries **`failedRequests`**: every
|
|
1080
|
+
HTTP ≥400 response and network failure since the previous action, with
|
|
1081
|
+
method, URL, status and the first bytes of an xhr/fetch error body. A click
|
|
1082
|
+
that "did nothing" because the backend answered 500 looks like a click miss
|
|
1083
|
+
in the screenshot; this is where the reason shows up.
|
|
1084
|
+
- **`output/report.json`** (every compose) says *why* a scene looks slow or
|
|
1085
|
+
cut: per-scene retime factor, hold %, trims, and the `warnings[]` list; the
|
|
1086
|
+
compose log prints the same warnings. `aidemo lint <dir>` predicts them
|
|
1087
|
+
without a browser.
|
|
1088
|
+
- **`aidemo frames <dir> --every 3`** (MCP `frames`) dumps evenly spaced PNGs
|
|
1089
|
+
from `output/final-demo.mp4` into `output/frames/` — look at them instead of
|
|
1090
|
+
hand-running `ffmpeg -ss`. `--source raw` samples the latest raw file;
|
|
1091
|
+
`--source take` samples the whole recorded take by walking `timeline.json`,
|
|
1092
|
+
so a resumed take's earlier scenes are included and each frame is named with
|
|
1093
|
+
the scene it came from.
|
|
1094
|
+
- **Resumed takes** (`--from-scene`): `timeline.json` scenes reused from an
|
|
1095
|
+
earlier take carry `source` (their raw file, `recordings/raw.keep-*`) and
|
|
1096
|
+
`leadInMs`; `recordings/raw.*` holds only the new tail. `frames --source
|
|
1097
|
+
raw` therefore shows the new part only — use `--source take` (or review the
|
|
1098
|
+
composed video) to see the whole thing.
|
|
572
1099
|
- `AIDEMO_KEEP_TMP=1` preserves `.compose-tmp/` intermediates when debugging
|
|
573
1100
|
compose.
|
|
574
1101
|
- `doctor` checks Node, ffmpeg, Chrome, the TTS/STT endpoint (and flags
|
|
575
1102
|
LLM-only servers like Ollama, which have no audio endpoints — point
|
|
576
1103
|
`OPENAI_BASE_URL` at a speech server such as speaches instead).
|
|
577
1104
|
|
|
1105
|
+
## Import from Playwright (trace or test file)
|
|
1106
|
+
|
|
1107
|
+
When the product already has an e2e test for the flow, don't re-derive the
|
|
1108
|
+
selectors: `import_trace {file, name}` / `aidemo import-trace <file> --name
|
|
1109
|
+
<name>` takes a **`trace.zip`** (`npx playwright test --trace on`, or
|
|
1110
|
+
`context.tracing.start/stop`) or a **`*.spec.ts`** file and writes
|
|
1111
|
+
`demos/<name>/` with a draft storyboard:
|
|
1112
|
+
|
|
1113
|
+
- the run's own actions become beats — `goto`, `fill`→`type`, `click`,
|
|
1114
|
+
`hover`, `press`, `selectOption`→`select`, `setInputFiles`→`upload`,
|
|
1115
|
+
`waitForSelector`→`waitForWidget`, `dragAndDrop`→`drag`, `expect(...)
|
|
1116
|
+
.toBeVisible/toHaveText/toHaveURL/toBeChecked/toHaveAttribute`→`assert`
|
|
1117
|
+
(the last two fold the condition into the selector as `>> :scope:checked` /
|
|
1118
|
+
`>> :scope[href="…"]`); `toHaveScreenshot` is dropped with a note — it
|
|
1119
|
+
asserts pixels, not a story beat; calls that failed in the trace are dropped;
|
|
1120
|
+
- `getByRole/getByTestId/getByText/getByPlaceholder` and `internal:` engines
|
|
1121
|
+
become plain storyboard selectors (`role=button[name="Add to basket"i]`,
|
|
1122
|
+
`[data-testid="x"]`, `text=…`); `getByLabel` stays an `internal:label="…"i`
|
|
1123
|
+
selector, which `page.locator` resolves to the labelled control;
|
|
1124
|
+
`frameLocator` chains become a `frames` entry + `target.frame`;
|
|
1125
|
+
`.first()/.nth()/.last()` become `nth`/`last`; in a test file a locator
|
|
1126
|
+
parked in a `const` is followed to its use;
|
|
1127
|
+
- scenes are cut at each `goto` and before a click whose payoff is an explicit
|
|
1128
|
+
wait or assertion; narration is `<narrate: …>` placeholders; `_notes` lists
|
|
1129
|
+
every approximation (a statement the line-based test parser could not read —
|
|
1130
|
+
a page-object helper, say — or an upload whose file you must copy into
|
|
1131
|
+
`input/`).
|
|
1132
|
+
|
|
1133
|
+
Then write the narration, merge or split scenes to one beat each, and
|
|
1134
|
+
`probe`. Nothing is inferred by a model — the storyboard is the test, retold.
|
|
1135
|
+
|
|
578
1136
|
## Demo as regression test
|
|
579
1137
|
|
|
580
1138
|
A probe run is a deterministic replay of the storyboard, so it doubles as a
|
|
@@ -595,8 +1153,13 @@ prints a readable field-level diff (`$.scenes[2].actions[0].found: expected
|
|
|
595
1153
|
true, got false`) and exits non-zero. A missing baseline is a clear error
|
|
596
1154
|
telling you to run `--update-golden` first. In golden mode a failing action is
|
|
597
1155
|
recorded (`ok:false`) and the run continues, so you get the full diff instead of
|
|
598
|
-
an abort at the first break
|
|
599
|
-
|
|
1156
|
+
an abort at the first break — and each broken selector gets the same
|
|
1157
|
+
diagnostics as a hard failure: `logs/fail-<scene>-<n>.png` plus
|
|
1158
|
+
`logs/drift-<scene>-<n>.json` with the nearest interactive elements and their
|
|
1159
|
+
unique selectors, listed under the diff as `selector drift → …`. Read the drift
|
|
1160
|
+
file, swap the selector, re-probe. (The MCP `probe` tool takes the same
|
|
1161
|
+
`updateGolden` / `golden` params and returns `golden.match`, `golden.diffs` and
|
|
1162
|
+
`golden.drift[]` — `{scene, action, file}` — in its result.)
|
|
600
1163
|
|
|
601
1164
|
Wire it into CI so a breaking UI change is a failed check:
|
|
602
1165
|
|
|
@@ -617,8 +1180,8 @@ Estonian "uus kuub" came back "Scoop", "AI-treener" came back "EI treener").
|
|
|
617
1180
|
To keep the transcript honest, the request is biased with what the engine
|
|
618
1181
|
already knows about the script:
|
|
619
1182
|
|
|
620
|
-
- **Prompt bias (always on).**
|
|
621
|
-
order) is sent as Whisper's `prompt`, nudging the transcript to converge on
|
|
1183
|
+
- **Prompt bias (always on).** Any top-level `knownTerms` glossary, then the
|
|
1184
|
+
storyboard's narration text (in scene order), is sent as Whisper's `prompt`, nudging the transcript to converge on
|
|
622
1185
|
the actual scripted words/spelling instead of guessing from audio phonetics
|
|
623
1186
|
alone — while still keeping real word-level timing from the audio. This
|
|
624
1187
|
changes STT output for every non-trivial demo (strictly for the better:
|
|
@@ -633,7 +1196,8 @@ already knows about the script:
|
|
|
633
1196
|
monolingual non-English demo) without editing the storyboard.
|
|
634
1197
|
|
|
635
1198
|
**Non-English narration:** prompt bias + a language hint fix the common case,
|
|
636
|
-
but Whisper can still misspell unusual or compound words.
|
|
1199
|
+
but Whisper can still misspell unusual or compound words. Add the offenders to
|
|
1200
|
+
top-level `knownTerms` first — that keeps real word timings. If burned-in
|
|
637
1201
|
captions still look wrong, **`aidemo captions <dir> --offline` is the
|
|
638
1202
|
guaranteed-correct fallback** — cues are derived directly from the storyboard
|
|
639
1203
|
script (correct spelling by construction, since there's no transcription at
|