@tandryukha/aidemo 0.15.1 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -1
- package/docs/AUTHORING.md +37 -7
- package/package.json +5 -2
- package/src/attention.ts +5 -4
- package/src/browser.ts +108 -0
- package/src/caption-render.ts +2 -2
- package/src/captions.ts +80 -2
- package/src/cards.ts +2 -2
- package/src/cli.ts +6 -1
- package/src/compose.ts +13 -6
- package/src/config.ts +120 -1
- package/src/cursor.ts +2 -2
- package/src/distribute.ts +79 -4
- package/src/ffmpeg.ts +167 -114
- package/src/frame.ts +2 -2
- package/src/inspect.ts +25 -4
- package/src/login.ts +6 -2
- package/src/mcp/server.ts +32 -6
- package/src/qa.ts +30 -1
- package/src/recorder.ts +11 -3
- package/src/safe-regex.ts +7 -2
- package/src/types.ts +4 -2
package/README.md
CHANGED
|
@@ -161,7 +161,8 @@ open examples/local-demo/output/final-demo.mp4 # xdg-open on Linux, start on W
|
|
|
161
161
|
Voice/captions need `OPENAI_API_KEY` in `.env` — **or** `AIDEMO_TTS_PROVIDER=local`
|
|
162
162
|
(no key, offline), **or** `OPENAI_BASE_URL` at a local server. See
|
|
163
163
|
[docs/LOCAL_MODELS.md](docs/LOCAL_MODELS.md). No Playwright browser download is
|
|
164
|
-
needed — the engine uses your system Chrome (`channel: "chrome"
|
|
164
|
+
needed — the engine uses your system Chrome (`channel: "chrome"`; see
|
|
165
|
+
[Setup](#setup) to use a different browser).
|
|
165
166
|
|
|
166
167
|
[](https://github.com/tandryukha/aidemo/releases/download/v0.3.0/quickstart-demo.mp4)
|
|
167
168
|
|
|
@@ -214,6 +215,10 @@ Codex).
|
|
|
214
215
|
- **Pipeline tools run as jobs** — `probe`/`record`/`render`/`voice`/`captions`/
|
|
215
216
|
`compose`/`gif` return a `jobId` immediately; `job_status` reports stage,
|
|
216
217
|
per-scene progress, and (on failure) the screenshot/frame-dump paths.
|
|
218
|
+
- **`.env` works like the CLI** — the server loads its cwd's and the engine's
|
|
219
|
+
`.env` at startup, and since MCP clients often start it outside your repo, each
|
|
220
|
+
tool given a demo dir also applies that repo's `.env` (nearest one walking up
|
|
221
|
+
from the demo dir to the repo root). `doctor {dir}` shows the result.
|
|
217
222
|
|
|
218
223
|
## Why it's built this way
|
|
219
224
|
|
|
@@ -253,6 +258,22 @@ npm install
|
|
|
253
258
|
cp .env.example .env # add OPENAI_API_KEY, or use AIDEMO_TTS_PROVIDER=local (no key)
|
|
254
259
|
```
|
|
255
260
|
|
|
261
|
+
**Choosing the browser.** Every launch (recording, `inspect`, caption/card
|
|
262
|
+
rasterizing) uses system Google Chrome by default. Two env vars (shell or
|
|
263
|
+
`.env`) override that, highest precedence first:
|
|
264
|
+
|
|
265
|
+
- `AIDEMO_CHROME_EXECUTABLE=/path/to/chrome` — an explicit Chromium-based binary.
|
|
266
|
+
- `AIDEMO_CHROME_CHANNEL=<channel>` — a Playwright channel: `chrome` (default),
|
|
267
|
+
`chrome-beta`, `chrome-dev`, `chrome-canary`, `msedge`, `msedge-beta`,
|
|
268
|
+
`msedge-dev`, `msedge-canary`, or `chromium`. `chromium` (Chrome for Testing)
|
|
269
|
+
is the fix when an enterprise-managed Chrome closes automated sessions after
|
|
270
|
+
a few seconds ("Target page, context or browser has been closed"); it needs a
|
|
271
|
+
one-time `npx playwright install chromium`.
|
|
272
|
+
|
|
273
|
+
`aidemo doctor` reports which browser will be used and whether it was found.
|
|
274
|
+
`aidemo login` opens the same browser, so the profile you log into matches the
|
|
275
|
+
one you record with.
|
|
276
|
+
|
|
256
277
|
## Project layout (per demo)
|
|
257
278
|
|
|
258
279
|
```
|
package/docs/AUTHORING.md
CHANGED
|
@@ -70,6 +70,12 @@ scan inside them). Use it instead of reading source or guessing: copy the
|
|
|
70
70
|
first selector of the element you mean. It writes `logs/inspect-<ts>.json`
|
|
71
71
|
and a screenshot next to it. Elements below the fold are listed too (marked
|
|
72
72
|
`↓` on the CLI); a `scrollTo` on them works as usual.
|
|
73
|
+
The result includes `target.status` (`available` or `blocked`). Common
|
|
74
|
+
security-verification and login pages return `blocked` with a reason: their
|
|
75
|
+
selectors describe the interstitial, not the requested product. Inspection
|
|
76
|
+
still writes its JSON and screenshot for diagnosis; `init --from-url` stops
|
|
77
|
+
instead of drafting scenes from a blocked page. Open the page with an
|
|
78
|
+
authenticated profile or use supported setup seeding, then inspect again.
|
|
73
79
|
|
|
74
80
|
`inspect` runs the storyboard's **`setup`** block by default — preflight hook,
|
|
75
81
|
`storageState`, cookies — so a gated app scans as the logged-in user instead of
|
|
@@ -613,12 +619,21 @@ millisecond instead, when you want a specific moment.
|
|
|
613
619
|
|
|
614
620
|
**Blank opening** (`output.trimLeadingBlank: true`): a take that starts on a
|
|
615
621
|
page still painting opens on white/blank frames. With this on, compose trims
|
|
616
|
-
the
|
|
617
|
-
|
|
618
|
-
always *measures* the opening either way
|
|
619
|
-
|
|
622
|
+
the blank lead of the first scene down to `output.blankOpenCapMs` (default
|
|
623
|
+
100 ms kept) and re-aligns the scene's cursor, zoom and anchors to the trim.
|
|
624
|
+
Compose always *measures* the opening either way — on the page footage,
|
|
625
|
+
before captions and overlays — and reports it as `blankOpenMs` plus a
|
|
626
|
+
`blank-open` warning past ~250 ms; `aidemo lint` forecasts the same pitfall
|
|
620
627
|
from a first scene that opens with a `goto` and no wait.
|
|
621
628
|
|
|
629
|
+
"Blank" means *empty*, not *static*: one colour covering the frame **and** no
|
|
630
|
+
text or outlines beyond a cursor-sized mark. A white pre-paint or a solid
|
|
631
|
+
splash (any colour, black included) is blank; a motionless page that shows
|
|
632
|
+
something — a dark terminal with a prompt line, a login box on a big solid
|
|
633
|
+
background — is content, so it is never trimmed or warned about. The same
|
|
634
|
+
test guards the freeze-hold tail trim (`blank-tail`). A frame showing only a
|
|
635
|
+
cursor, a lone `$` or a small spinner still reads as blank.
|
|
636
|
+
|
|
622
637
|
## Capture resolution (`video.deviceScaleFactor`)
|
|
623
638
|
|
|
624
639
|
`video: {width, height}` is the **logical** viewport — the CSS px the page
|
|
@@ -763,7 +778,7 @@ for UI that re-renders under the cursor) and `anchor?` (land this action on a
|
|
|
763
778
|
out of the way before a reveal, or point at what the narration names.
|
|
764
779
|
- `{op:"assert", target?, textMatches?, url?, timeoutMs?=5000}` — **prove the
|
|
765
780
|
payoff happened**: polls until `target` is visible (and its text matches
|
|
766
|
-
`textMatches`,
|
|
781
|
+
`textMatches`, RE2-compatible regex, no inline flags — use `[Cc]onfirmed`) and/or the
|
|
767
782
|
page URL matches `url`; otherwise **fails the take with a named error**
|
|
768
783
|
(`assert failed after 5000ms: text "…" does not match /…/`). Put one after
|
|
769
784
|
the moment the demo exists to show (order confirmed, item added) so a
|
|
@@ -1211,8 +1226,14 @@ CLI. If nothing came up, skip this.
|
|
|
1211
1226
|
composed video) to see the whole thing.
|
|
1212
1227
|
- `AIDEMO_KEEP_TMP=1` preserves `.compose-tmp/` intermediates when debugging
|
|
1213
1228
|
compose.
|
|
1214
|
-
-
|
|
1215
|
-
|
|
1229
|
+
- Browser: every launch uses system Chrome unless `AIDEMO_CHROME_EXECUTABLE`
|
|
1230
|
+
(explicit binary) or `AIDEMO_CHROME_CHANNEL` (`chromium` — Chrome for
|
|
1231
|
+
Testing, after `npx playwright install chromium` — `chrome-beta`, `msedge`,
|
|
1232
|
+
…) is set; the executable wins. Use `chromium` when a managed Chrome closes
|
|
1233
|
+
the automated session after a few seconds ("Target page, context or browser
|
|
1234
|
+
has been closed").
|
|
1235
|
+
- `doctor` checks Node, ffmpeg, the browser in use, the TTS/STT endpoint
|
|
1236
|
+
(and flags LLM-only servers like Ollama, which have no audio endpoints — point
|
|
1216
1237
|
`OPENAI_BASE_URL` at a speech server such as speaches instead).
|
|
1217
1238
|
|
|
1218
1239
|
## Import from Playwright (trace or test file)
|
|
@@ -1319,6 +1340,15 @@ all) using each scene's measured duration from `voice.json`; only the timing
|
|
|
1319
1340
|
measured speech rhythm). `captions` logs a one-line reminder of this whenever
|
|
1320
1341
|
the resolved language isn't English.
|
|
1321
1342
|
|
|
1343
|
+
The STT caption step compares each scene's recognized words with its known
|
|
1344
|
+
script and warns when at least 20% differ. Review those captions before
|
|
1345
|
+
publishing. `aidemo captions <dir> --align-script` uses the script's spelling
|
|
1346
|
+
and punctuation while retaining the measured STT word times; words missing
|
|
1347
|
+
from the transcript are placed between their nearest timed neighbours. This
|
|
1348
|
+
is useful for misheard names and inflected words. If the spoken narration
|
|
1349
|
+
actually differs from the script, correct the script or use the original STT
|
|
1350
|
+
captions instead.
|
|
1351
|
+
|
|
1322
1352
|
**Cue segmentation.** Cues are broken at sentence ends first, then at clause
|
|
1323
1353
|
punctuation, then before a conjunction, and only then on length — so a cue
|
|
1324
1354
|
stops reading like it was cut mid-thought ("we open the" / "settings panel").
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tandryukha/aidemo",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.16.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"description": "AI Demo Engine — turn a storyboard into a narrated, captioned product demo video",
|
|
6
6
|
"type": "module",
|
|
@@ -28,15 +28,18 @@
|
|
|
28
28
|
"build": "tsc --noEmit",
|
|
29
29
|
"serve-example": "node examples/local-demo/serve.mjs",
|
|
30
30
|
"mcp-smoke": "node test/mcp-smoke.mjs",
|
|
31
|
+
"test:blank-open": "tsx --test test/blank-open.mjs",
|
|
32
|
+
"mcp-env-test": "node test/mcp-env.mjs",
|
|
31
33
|
"guard-openai-key": "tsx scripts/guard-openai-key.ts",
|
|
32
34
|
"sync:plugin-skill": "cp .claude/skills/record-demo/SKILL.md plugins/record-demo/skills/record-demo/SKILL.md",
|
|
33
35
|
"check:plugin-skill": "diff -q .claude/skills/record-demo/SKILL.md plugins/record-demo/skills/record-demo/SKILL.md"
|
|
34
36
|
},
|
|
35
37
|
"dependencies": {
|
|
36
|
-
"@modelcontextprotocol/sdk": "1.
|
|
38
|
+
"@modelcontextprotocol/sdk": "1.32.1",
|
|
37
39
|
"commander": "^12.1.0",
|
|
38
40
|
"openai": "^4.67.0",
|
|
39
41
|
"playwright": "^1.55.1",
|
|
42
|
+
"re2js": "2.8.6",
|
|
40
43
|
"tsx": "^4.23.1",
|
|
41
44
|
"zod": "^3.23.8",
|
|
42
45
|
"zod-to-json-schema": "3.25.2"
|
package/src/attention.ts
CHANGED
|
@@ -12,7 +12,8 @@
|
|
|
12
12
|
* every PNG is rendered at deviceScaleFactor = pxScale so it lands 1:1.
|
|
13
13
|
*/
|
|
14
14
|
|
|
15
|
-
import {
|
|
15
|
+
import type { Page } from "playwright";
|
|
16
|
+
import { launchChrome } from "./browser.js";
|
|
16
17
|
import { resolve } from "node:path";
|
|
17
18
|
import type { AttentionEvent, AttentionStyle, RedactSpan } from "./types.js";
|
|
18
19
|
import { ensureDir } from "./util.js";
|
|
@@ -107,7 +108,7 @@ export async function renderAttentionPngs(
|
|
|
107
108
|
): Promise<OverlayItem[]> {
|
|
108
109
|
if (events.length === 0) return [];
|
|
109
110
|
await ensureDir(outDir);
|
|
110
|
-
const browser = await
|
|
111
|
+
const browser = await launchChrome({ headless: true });
|
|
111
112
|
const context = await browser.newContext({
|
|
112
113
|
viewport: { width: videoW, height: videoH },
|
|
113
114
|
deviceScaleFactor: pxScale,
|
|
@@ -231,7 +232,7 @@ export async function renderKeyChipPngs(
|
|
|
231
232
|
): Promise<OverlayItem[]> {
|
|
232
233
|
if (keys.length === 0) return [];
|
|
233
234
|
await ensureDir(outDir);
|
|
234
|
-
const browser = await
|
|
235
|
+
const browser = await launchChrome({ headless: true });
|
|
235
236
|
const context = await browser.newContext({
|
|
236
237
|
viewport: { width: 600, height: 120 },
|
|
237
238
|
deviceScaleFactor: pxScale,
|
|
@@ -287,7 +288,7 @@ export async function renderClickRingPngs(
|
|
|
287
288
|
): Promise<OverlayItem[]> {
|
|
288
289
|
if (clicks.length === 0) return [];
|
|
289
290
|
await ensureDir(outDir);
|
|
290
|
-
const browser = await
|
|
291
|
+
const browser = await launchChrome({ headless: true });
|
|
291
292
|
const context = await browser.newContext({
|
|
292
293
|
viewport: { width: 64, height: 64 },
|
|
293
294
|
deviceScaleFactor: pxScale,
|
package/src/browser.ts
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Single choke point for every Playwright browser launch (issue #80).
|
|
3
|
+
*
|
|
4
|
+
* Every launch site — recorder, inspect, caption/card/frame/attention/cursor
|
|
5
|
+
* rasterizers — goes through launchChrome()/launchChromePersistent(), which
|
|
6
|
+
* merge chromeLaunchOptions() (AIDEMO_CHROME_EXECUTABLE > AIDEMO_CHROME_CHANNEL
|
|
7
|
+
* > `channel: "chrome"`) into the caller's options. With neither env var set
|
|
8
|
+
* the options are exactly the engine's historical `{ channel: "chrome", ... }`.
|
|
9
|
+
*/
|
|
10
|
+
import { chromium, type Browser, type BrowserContext, type LaunchOptions } from "playwright";
|
|
11
|
+
import { BrowserSetupError, chromeLaunchOptions, type ChromeLaunchTarget } from "./config.js";
|
|
12
|
+
|
|
13
|
+
type PersistentOptions = NonNullable<Parameters<typeof chromium.launchPersistentContext>[1]>;
|
|
14
|
+
|
|
15
|
+
/** Human-readable description of the configured browser, for logs/doctor. */
|
|
16
|
+
export function describeChromeTarget(target: ChromeLaunchTarget = chromeLaunchOptions()): string {
|
|
17
|
+
if ("executablePath" in target) return `executable ${target.executablePath} (AIDEMO_CHROME_EXECUTABLE)`;
|
|
18
|
+
return target.channel === "chrome" && !process.env.AIDEMO_CHROME_CHANNEL?.trim()
|
|
19
|
+
? `channel chrome (default — system Google Chrome)`
|
|
20
|
+
: `channel ${target.channel} (AIDEMO_CHROME_CHANNEL)`;
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
const MISSING_RE =
|
|
24
|
+
/Executable doesn't exist|is not found at|distribution '[^']*' is not found|npx playwright install|ENOENT/i;
|
|
25
|
+
|
|
26
|
+
/**
|
|
27
|
+
* Turn a "browser binary not found" launch failure into an actionable error.
|
|
28
|
+
* Any other failure is returned untouched so callers' own wrapping (e.g. the
|
|
29
|
+
* recorder's profile-lock hint) still sees the original message.
|
|
30
|
+
*/
|
|
31
|
+
export function explainLaunchError(err: unknown, target: ChromeLaunchTarget): Error {
|
|
32
|
+
const e = err instanceof Error ? err : new Error(String(err));
|
|
33
|
+
if (!MISSING_RE.test(e.message)) return e;
|
|
34
|
+
let hint: string;
|
|
35
|
+
if ("executablePath" in target) {
|
|
36
|
+
hint =
|
|
37
|
+
`AIDEMO_CHROME_EXECUTABLE=${target.executablePath} could not be launched — ` +
|
|
38
|
+
`point it at an existing Chromium-based browser binary, or unset it.`;
|
|
39
|
+
} else if (target.channel === "chromium") {
|
|
40
|
+
hint =
|
|
41
|
+
`Playwright's Chromium (AIDEMO_CHROME_CHANNEL=chromium) is not installed — ` +
|
|
42
|
+
`run: npx playwright install chromium (or unset AIDEMO_CHROME_CHANNEL to use system Chrome).`;
|
|
43
|
+
} else if (target.channel === "chrome") {
|
|
44
|
+
hint =
|
|
45
|
+
`Google Chrome was not found. Install Chrome, or use Playwright's Chromium: ` +
|
|
46
|
+
`npx playwright install chromium && set AIDEMO_CHROME_CHANNEL=chromium ` +
|
|
47
|
+
`(or AIDEMO_CHROME_EXECUTABLE=/path/to/chrome).`;
|
|
48
|
+
} else {
|
|
49
|
+
hint =
|
|
50
|
+
`Browser channel "${target.channel}" (AIDEMO_CHROME_CHANNEL) is not installed — ` +
|
|
51
|
+
`install it (npx playwright install ${target.channel}) or unset AIDEMO_CHROME_CHANNEL.`;
|
|
52
|
+
}
|
|
53
|
+
const out = new BrowserSetupError(`${hint}\nOriginal error: ${e.message}`);
|
|
54
|
+
(out as Error & { cause?: unknown }).cause = e;
|
|
55
|
+
return out;
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
/** chromium.launch() with the configured browser. */
|
|
59
|
+
export async function launchChrome(options: Omit<LaunchOptions, "channel" | "executablePath"> = {}): Promise<Browser> {
|
|
60
|
+
const target = chromeLaunchOptions();
|
|
61
|
+
try {
|
|
62
|
+
return await chromium.launch({ ...target, ...options });
|
|
63
|
+
} catch (err) {
|
|
64
|
+
throw explainLaunchError(err, target);
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
/** chromium.launchPersistentContext() with the configured browser. */
|
|
69
|
+
export async function launchChromePersistent(
|
|
70
|
+
userDataDir: string,
|
|
71
|
+
options: Omit<PersistentOptions, "channel" | "executablePath"> = {}
|
|
72
|
+
): Promise<BrowserContext> {
|
|
73
|
+
const target = chromeLaunchOptions();
|
|
74
|
+
try {
|
|
75
|
+
return await chromium.launchPersistentContext(userDataDir, { ...target, ...options });
|
|
76
|
+
} catch (err) {
|
|
77
|
+
throw explainLaunchError(err, target);
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
/**
|
|
82
|
+
* Doctor probe for a NON-default browser target: launch it headless once and
|
|
83
|
+
* read its version. Returns `{ version }` or `{ error }` (actionable message).
|
|
84
|
+
*/
|
|
85
|
+
export async function probeConfiguredBrowser(): Promise<{ version: string } | { error: string }> {
|
|
86
|
+
let browser: Browser | undefined;
|
|
87
|
+
try {
|
|
88
|
+
browser = await launchChrome({ headless: true, timeout: 20000 });
|
|
89
|
+
return { version: browser.version() };
|
|
90
|
+
} catch (err) {
|
|
91
|
+
return { error: (err as Error).message.split("\n")[0] };
|
|
92
|
+
} finally {
|
|
93
|
+
await browser?.close().catch(() => {});
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
/**
|
|
98
|
+
* Binary for a NON-Playwright launch (`aidemo login` opens a normal window on
|
|
99
|
+
* the profile) when the configured browser isn't system Chrome: the
|
|
100
|
+
* AIDEMO_CHROME_EXECUTABLE path, or Playwright's Chromium for
|
|
101
|
+
* AIDEMO_CHROME_CHANNEL=chromium. null = keep login's system-Chrome launch.
|
|
102
|
+
*/
|
|
103
|
+
export function manualLaunchBinary(): string | null {
|
|
104
|
+
const target = chromeLaunchOptions();
|
|
105
|
+
if ("executablePath" in target) return target.executablePath;
|
|
106
|
+
if (target.channel === "chromium") return chromium.executablePath();
|
|
107
|
+
return null;
|
|
108
|
+
}
|
package/src/caption-render.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import { launchChrome } from "./browser.js";
|
|
2
2
|
import { resolve } from "node:path";
|
|
3
3
|
import type { Cue } from "./captions.js";
|
|
4
4
|
import type { CaptionsConfig } from "./types.js";
|
|
@@ -28,7 +28,7 @@ export async function renderCaptionPngs(
|
|
|
28
28
|
look: CaptionsConfig & { font?: string } = {}
|
|
29
29
|
): Promise<RenderedCue[]> {
|
|
30
30
|
await ensureDir(outDir);
|
|
31
|
-
const browser = await
|
|
31
|
+
const browser = await launchChrome({ headless: true });
|
|
32
32
|
// Layout at the storyboard's logical width; deviceScaleFactor matches the
|
|
33
33
|
// raw video's pixel density (1 for Playwright capture, 2 for native/OBS
|
|
34
34
|
// retina capture) so the PNG overlays the frame 1:1.
|
package/src/captions.ts
CHANGED
|
@@ -96,6 +96,8 @@ const MAX_PROMPT_CHARS = 900;
|
|
|
96
96
|
export interface CaptionsSttOptions extends SceneProgress {
|
|
97
97
|
/** ISO-639-1/BCP-47 language hint for Whisper, overriding --lang / storyboard.language. */
|
|
98
98
|
language?: string;
|
|
99
|
+
/** Use storyboard spelling/punctuation while retaining STT word timing. */
|
|
100
|
+
alignScript?: boolean;
|
|
99
101
|
}
|
|
100
102
|
|
|
101
103
|
/**
|
|
@@ -168,6 +170,7 @@ export async function generateCaptions(
|
|
|
168
170
|
config,
|
|
169
171
|
prompt,
|
|
170
172
|
language,
|
|
173
|
+
...(opts.alignScript ? { alignScript: true } : {}),
|
|
171
174
|
});
|
|
172
175
|
const prior = await readCaptionsManifest(project);
|
|
173
176
|
if (prior && prior.mode === "stt" && prior.inputHash === inputHash) {
|
|
@@ -197,13 +200,32 @@ export async function generateCaptions(
|
|
|
197
200
|
if (words.length === 0) {
|
|
198
201
|
log("no word timestamps returned; captions may be empty");
|
|
199
202
|
}
|
|
200
|
-
const
|
|
203
|
+
const sceneWords = splitWordsByScene(words, ids, sceneEnds, config.gapMs);
|
|
204
|
+
let captionWords = words;
|
|
205
|
+
if (storyboard && sceneEnds.length === storyboard.scenes.length) {
|
|
206
|
+
const aligned: Word[] = [];
|
|
207
|
+
let sceneStartMs = 0;
|
|
208
|
+
for (let i = 0; i < storyboard.scenes.length; i++) {
|
|
209
|
+
const scene = storyboard.scenes[i];
|
|
210
|
+
const timed = sceneWords[i]?.words ?? [];
|
|
211
|
+
const result = alignCaptionWords(scene.narration, timed, sceneEnds[i] - sceneStartMs);
|
|
212
|
+
if (result.driftPct >= 20) {
|
|
213
|
+
log(`⚠ caption drift in scene ${scene.id}: ${result.driftPct}% of script words differ from STT; ${opts.alignScript ? "script text aligned to STT timing — review against the audio" : "review captions or use --align-script"}`);
|
|
214
|
+
}
|
|
215
|
+
for (const w of result.words) aligned.push({ ...w, start: w.start + sceneStartMs / 1000, end: w.end + sceneStartMs / 1000 });
|
|
216
|
+
sceneStartMs = sceneEnds[i] + config.gapMs;
|
|
217
|
+
}
|
|
218
|
+
if (opts.alignScript) captionWords = aligned;
|
|
219
|
+
} else if (opts.alignScript) {
|
|
220
|
+
throw new Error("--align-script requires a storyboard and matching voice scene timings");
|
|
221
|
+
}
|
|
222
|
+
const cues = groupWords(captionWords, sceneEnds, seg);
|
|
201
223
|
await writeCaptionFiles(project, cues);
|
|
202
224
|
await writeCaptionsManifest(project, {
|
|
203
225
|
mode: "stt",
|
|
204
226
|
inputHash,
|
|
205
227
|
config,
|
|
206
|
-
scenes: splitWordsByScene(
|
|
228
|
+
scenes: opts.alignScript ? splitWordsByScene(captionWords, ids, sceneEnds, config.gapMs) : sceneWords,
|
|
207
229
|
cues,
|
|
208
230
|
});
|
|
209
231
|
reportDone();
|
|
@@ -245,6 +267,62 @@ function splitWordsByScene(
|
|
|
245
267
|
return out;
|
|
246
268
|
}
|
|
247
269
|
|
|
270
|
+
/** Align written words to measured STT timestamps. Inserted script words share
|
|
271
|
+
* the space between their nearest timed neighbours; STT-only words are dropped. */
|
|
272
|
+
export function alignCaptionWords(
|
|
273
|
+
narration: string,
|
|
274
|
+
spoken: Word[],
|
|
275
|
+
durationMs: number
|
|
276
|
+
): { words: Word[]; driftPct: number } {
|
|
277
|
+
const script = narration.split(/\s+/).filter(Boolean);
|
|
278
|
+
if (!script.length) return { words: [], driftPct: spoken.length ? 100 : 0 };
|
|
279
|
+
if (!spoken.length) return { words: deriveSceneWords(narration, durationMs), driftPct: 100 };
|
|
280
|
+
const norm = (s: string) => s.normalize("NFKC").toLocaleLowerCase().replace(/[^\p{L}\p{N}]/gu, "");
|
|
281
|
+
const n = script.length, m = spoken.length;
|
|
282
|
+
const dp = Array.from({ length: n + 1 }, () => new Uint16Array(m + 1));
|
|
283
|
+
for (let i = 0; i <= n; i++) dp[i][0] = i;
|
|
284
|
+
for (let j = 0; j <= m; j++) dp[0][j] = j;
|
|
285
|
+
for (let i = 1; i <= n; i++) for (let j = 1; j <= m; j++) {
|
|
286
|
+
dp[i][j] = Math.min(
|
|
287
|
+
dp[i - 1][j - 1] + (norm(script[i - 1]) === norm(spoken[j - 1].word) ? 0 : 1),
|
|
288
|
+
dp[i - 1][j] + 1,
|
|
289
|
+
dp[i][j - 1] + 1
|
|
290
|
+
);
|
|
291
|
+
}
|
|
292
|
+
const matched: Array<number | undefined> = Array(n).fill(undefined);
|
|
293
|
+
let i = n, j = m;
|
|
294
|
+
while (i && j) {
|
|
295
|
+
const cost = norm(script[i - 1]) === norm(spoken[j - 1].word) ? 0 : 1;
|
|
296
|
+
if (dp[i][j] === dp[i - 1][j - 1] + cost) {
|
|
297
|
+
matched[--i] = --j;
|
|
298
|
+
} else if (dp[i][j] === dp[i - 1][j] + 1) {
|
|
299
|
+
i--;
|
|
300
|
+
} else {
|
|
301
|
+
j--;
|
|
302
|
+
}
|
|
303
|
+
}
|
|
304
|
+
const duration = Math.max(0, durationMs / 1000);
|
|
305
|
+
const words: Word[] = script.map((word, k) => {
|
|
306
|
+
const at = matched[k];
|
|
307
|
+
return at === undefined ? { word, start: 0, end: 0 } : {
|
|
308
|
+
word, start: Math.max(0, spoken[at].start), end: Math.min(duration, spoken[at].end),
|
|
309
|
+
};
|
|
310
|
+
});
|
|
311
|
+
for (let k = 0; k < n;) {
|
|
312
|
+
if (matched[k] !== undefined) { k++; continue; }
|
|
313
|
+
const first = k;
|
|
314
|
+
while (k < n && matched[k] === undefined) k++;
|
|
315
|
+
const start = first ? words[first - 1].end : 0;
|
|
316
|
+
const end = k < n ? words[k].start : duration;
|
|
317
|
+
const span = Math.max(0, end - start);
|
|
318
|
+
for (let t = first; t < k; t++) {
|
|
319
|
+
words[t].start = start + span * (t - first) / (k - first);
|
|
320
|
+
words[t].end = start + span * (t - first + 1) / (k - first);
|
|
321
|
+
}
|
|
322
|
+
}
|
|
323
|
+
return { words, driftPct: Math.round(100 * dp[n][m] / Math.max(n, m)) };
|
|
324
|
+
}
|
|
325
|
+
|
|
248
326
|
/** Scene-relative words for a scene; script-timed fallback when the manifest has none. */
|
|
249
327
|
export async function sceneWordsFor(
|
|
250
328
|
project: Project,
|
package/src/cards.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import { launchChrome } from "./browser.js";
|
|
2
2
|
import type { Brand, Card } from "./types.js";
|
|
3
3
|
|
|
4
4
|
/**
|
|
@@ -17,7 +17,7 @@ export async function renderCardPng(
|
|
|
17
17
|
brand?: Brand,
|
|
18
18
|
logoDataUri?: string
|
|
19
19
|
): Promise<void> {
|
|
20
|
-
const browser = await
|
|
20
|
+
const browser = await launchChrome({ headless: true });
|
|
21
21
|
const context = await browser.newContext({
|
|
22
22
|
viewport: { width: logicalW, height: logicalH },
|
|
23
23
|
deviceScaleFactor: scale,
|
package/src/cli.ts
CHANGED
|
@@ -664,6 +664,9 @@ program
|
|
|
664
664
|
process.stdout.write(JSON.stringify(res, null, 2) + "\n");
|
|
665
665
|
return;
|
|
666
666
|
}
|
|
667
|
+
if (res.target.status === "blocked") {
|
|
668
|
+
process.stdout.write(`⚠ inspect blocked: ${res.target.reason}\n`);
|
|
669
|
+
}
|
|
667
670
|
process.stdout.write(`${res.title || "(untitled)"} — ${res.finalUrl}\n`);
|
|
668
671
|
if (res.headings.length) {
|
|
669
672
|
process.stdout.write(
|
|
@@ -967,6 +970,7 @@ program
|
|
|
967
970
|
.option("--param <kv>", PARAM_OPT_DESC, collectKv, [])
|
|
968
971
|
.option("--lang <code>", LANG_OPT_DESC)
|
|
969
972
|
.option("--langs <codes>", LANGS_OPT_DESC)
|
|
973
|
+
.option("--align-script", "restore storyboard spelling and punctuation using STT word timing", false)
|
|
970
974
|
.option(
|
|
971
975
|
"--stt-lang <code>",
|
|
972
976
|
"override the Whisper STT language hint (ISO-639-1, e.g. et) — defaults to " +
|
|
@@ -982,6 +986,7 @@ program
|
|
|
982
986
|
lang?: string;
|
|
983
987
|
langs?: string;
|
|
984
988
|
sttLang?: string;
|
|
989
|
+
alignScript?: boolean;
|
|
985
990
|
}
|
|
986
991
|
) => {
|
|
987
992
|
const langs = langsFrom(opts);
|
|
@@ -998,7 +1003,7 @@ program
|
|
|
998
1003
|
if (opts.offline) {
|
|
999
1004
|
await generateCaptionsOffline(project, sb);
|
|
1000
1005
|
} else {
|
|
1001
|
-
await generateCaptions(project, sb, { language: opts.sttLang });
|
|
1006
|
+
await generateCaptions(project, sb, { language: opts.sttLang, alignScript: opts.alignScript });
|
|
1002
1007
|
}
|
|
1003
1008
|
}
|
|
1004
1009
|
}
|
package/src/compose.ts
CHANGED
|
@@ -236,7 +236,7 @@ export async function compose(
|
|
|
236
236
|
let leadTrimMs = 0;
|
|
237
237
|
if (i === 0 && storyboard.output?.trimLeadingBlank && srcMs > 1000) {
|
|
238
238
|
const cap = storyboard.output.blankOpenCapMs ?? 100;
|
|
239
|
-
const flatLead = await probeFlatLeadMs(rawSegPath);
|
|
239
|
+
const flatLead = await probeFlatLeadMs(rawSegPath, 4000, 0, pxScale);
|
|
240
240
|
if (flatLead > cap + 80) {
|
|
241
241
|
leadTrimMs = Math.min(flatLead - cap, srcMs - 500);
|
|
242
242
|
srcMs -= leadTrimMs;
|
|
@@ -251,7 +251,7 @@ export async function compose(
|
|
|
251
251
|
// (issue #36), so drop a flat tail before deciding how long to hold —
|
|
252
252
|
// but only when we would actually hold, and never the whole segment.
|
|
253
253
|
if (srcMs > 0 && targetMs > srcMs + 400) {
|
|
254
|
-
const flatMs = await probeFlatTailMs(rawSegPath, srcMs);
|
|
254
|
+
const flatMs = await probeFlatTailMs(rawSegPath, srcMs, 4000, pxScale);
|
|
255
255
|
const trimTo = srcMs - flatMs;
|
|
256
256
|
if (flatMs > 200 && trimTo > 500) {
|
|
257
257
|
blankTrimMs = flatMs;
|
|
@@ -668,6 +668,13 @@ export async function compose(
|
|
|
668
668
|
);
|
|
669
669
|
}
|
|
670
670
|
|
|
671
|
+
// Blank-open measurement (issues #47, #78), taken HERE — on the assembled
|
|
672
|
+
// scenes, before redaction, attention, compose-time cursor, zoom, frame and
|
|
673
|
+
// captions — because the final MP4 can't tell a blank page from content: a
|
|
674
|
+
// caption pill over a white pre-paint has as much text as a terminal prompt.
|
|
675
|
+
// Same timeline as the final video minus the intro card.
|
|
676
|
+
const blankOpenMs = await probeFlatLeadMs(content, 4000, 0, pxScale).catch(() => 0);
|
|
677
|
+
|
|
671
678
|
// Redaction first: blur boxes ride every later pass (cursor, zoom, blur).
|
|
672
679
|
if (redactPlaced.length) {
|
|
673
680
|
let src = content;
|
|
@@ -1094,10 +1101,10 @@ export async function compose(
|
|
|
1094
1101
|
: null;
|
|
1095
1102
|
await muxAudio(project, storyboard, finalVideo, introMs, chapters);
|
|
1096
1103
|
ok(`final video → ${project.outputPath}`);
|
|
1097
|
-
// Blank-open
|
|
1098
|
-
// at a fixed offset, which on a demo
|
|
1099
|
-
// white load frame; pick the first
|
|
1100
|
-
|
|
1104
|
+
// Blank-open warning + poster (issue #47; measured after scene assembly,
|
|
1105
|
+
// above). The poster used to be taken at a fixed offset, which on a demo
|
|
1106
|
+
// that opens with a `goto` is exactly the white load frame; pick the first
|
|
1107
|
+
// frame that actually shows content instead.
|
|
1101
1108
|
if (blankOpenMs > 250) {
|
|
1102
1109
|
warn(
|
|
1103
1110
|
"blank-open",
|
package/src/config.ts
CHANGED
|
@@ -66,7 +66,9 @@ async function readEnvFile(envPath: string): Promise<Record<string, string>> {
|
|
|
66
66
|
* instead. See scripts/guard-openai-key.ts, which fails the build if this
|
|
67
67
|
* skip-if-already-set shape ever comes back. Ambient only wins here when
|
|
68
68
|
* NEITHER .env file defines the key at all (e.g. a deliberate one-off invocation
|
|
69
|
-
* like `OPENAI_API_KEY=sk-... aidemo record ...`). Called once at CLI startup
|
|
69
|
+
* like `OPENAI_API_KEY=sk-... aidemo record ...`). Called once at CLI startup —
|
|
70
|
+
* which includes `aidemo mcp`; MCP tools then layer the demo's project .env on
|
|
71
|
+
* top via applyProjectEnv(), since the server's cwd is rarely the project.
|
|
70
72
|
*/
|
|
71
73
|
export async function loadEnv(): Promise<void> {
|
|
72
74
|
const cwdEnv = resolve(process.cwd(), ".env");
|
|
@@ -79,6 +81,71 @@ export async function loadEnv(): Promise<void> {
|
|
|
79
81
|
}
|
|
80
82
|
}
|
|
81
83
|
|
|
84
|
+
/**
|
|
85
|
+
* The `.env` a CLI user of this demo dir would have loaded: the nearest
|
|
86
|
+
* `.env` walking up from `dir`, stopping at the enclosing repo root (the
|
|
87
|
+
* first ancestor holding `.git`) — i.e. the cwd/.env of someone running
|
|
88
|
+
* `aidemo <cmd> demos/<name>` from their repo. null when there is none.
|
|
89
|
+
*/
|
|
90
|
+
async function findProjectEnv(dir: string): Promise<string | null> {
|
|
91
|
+
let cur = resolve(dir);
|
|
92
|
+
for (;;) {
|
|
93
|
+
const candidate = resolve(cur, ".env");
|
|
94
|
+
try {
|
|
95
|
+
await fs.access(candidate);
|
|
96
|
+
return candidate;
|
|
97
|
+
} catch {
|
|
98
|
+
// keep walking
|
|
99
|
+
}
|
|
100
|
+
try {
|
|
101
|
+
await fs.access(resolve(cur, ".git"));
|
|
102
|
+
return null; // repo root reached without a .env
|
|
103
|
+
} catch {
|
|
104
|
+
// not a repo root
|
|
105
|
+
}
|
|
106
|
+
const parent = resolve(cur, "..");
|
|
107
|
+
if (parent === cur) return null;
|
|
108
|
+
cur = parent;
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
/** Keys a project .env overrode → their value before the first override (undefined = unset). */
|
|
113
|
+
const projectEnvSaved = new Map<string, string | undefined>();
|
|
114
|
+
let projectEnvFile: string | null = null;
|
|
115
|
+
|
|
116
|
+
/**
|
|
117
|
+
* MCP counterpart of loadEnv()'s cwd/.env layer (issue #77). loadEnv() runs
|
|
118
|
+
* once at startup for `aidemo mcp` too, but an MCP client usually spawns the
|
|
119
|
+
* server with a cwd that is NOT the consumer's repo (and an npx engine root
|
|
120
|
+
* has no .env), so the project's .env was never seen. Tools that take a demo
|
|
121
|
+
* dir call this to apply that project's .env (see findProjectEnv) with the
|
|
122
|
+
* same precedence as the CLI's cwd/.env: it overrides the engine .env and the
|
|
123
|
+
* ambient environment. Switching to another project first restores every key
|
|
124
|
+
* the previous project's .env set, so values never leak between repos.
|
|
125
|
+
* Logs only to stderr (stdout is the JSON-RPC channel). Returns the file applied.
|
|
126
|
+
*/
|
|
127
|
+
export async function applyProjectEnv(dir: string): Promise<string | null> {
|
|
128
|
+
const envFile = await findProjectEnv(dir);
|
|
129
|
+
const values = envFile ? await readEnvFile(envFile) : {};
|
|
130
|
+
for (const [key, prev] of projectEnvSaved) {
|
|
131
|
+
if (key in values) continue;
|
|
132
|
+
if (prev === undefined) delete process.env[key];
|
|
133
|
+
else process.env[key] = prev;
|
|
134
|
+
projectEnvSaved.delete(key);
|
|
135
|
+
}
|
|
136
|
+
for (const [key, val] of Object.entries(values)) {
|
|
137
|
+
if (!projectEnvSaved.has(key)) projectEnvSaved.set(key, process.env[key]);
|
|
138
|
+
process.env[key] = val;
|
|
139
|
+
}
|
|
140
|
+
if (envFile !== projectEnvFile) {
|
|
141
|
+
process.stderr.write(
|
|
142
|
+
envFile ? ` env: loaded ${envFile}\n` : ` env: no project .env for ${resolve(dir)}\n`
|
|
143
|
+
);
|
|
144
|
+
projectEnvFile = envFile;
|
|
145
|
+
}
|
|
146
|
+
return envFile;
|
|
147
|
+
}
|
|
148
|
+
|
|
82
149
|
/**
|
|
83
150
|
* Custom endpoint for any OpenAI-compatible server (LocalAI, speaches,
|
|
84
151
|
* Kokoro-FastAPI, …). AIDEMO_OPENAI_BASE_URL wins over the SDK-standard
|
|
@@ -220,3 +287,55 @@ export const obsPassword = () => process.env.AIDEMO_OBS_PASSWORD;
|
|
|
220
287
|
*/
|
|
221
288
|
export const nativeCropUnsafe = () =>
|
|
222
289
|
process.env.AIDEMO_NATIVE_CROP_UNSAFE === "1";
|
|
290
|
+
|
|
291
|
+
/** Playwright browser channels the engine accepts for AIDEMO_CHROME_CHANNEL. */
|
|
292
|
+
export const CHROME_CHANNELS = [
|
|
293
|
+
"chrome",
|
|
294
|
+
"chrome-beta",
|
|
295
|
+
"chrome-dev",
|
|
296
|
+
"chrome-canary",
|
|
297
|
+
"msedge",
|
|
298
|
+
"msedge-beta",
|
|
299
|
+
"msedge-dev",
|
|
300
|
+
"msedge-canary",
|
|
301
|
+
"chromium",
|
|
302
|
+
] as const;
|
|
303
|
+
export type ChromeChannel = (typeof CHROME_CHANNELS)[number];
|
|
304
|
+
|
|
305
|
+
/**
|
|
306
|
+
* A browser-selection problem (bad AIDEMO_CHROME_CHANNEL, browser not
|
|
307
|
+
* installed) — already actionable, so callers rethrow it without adding
|
|
308
|
+
* unrelated hints (e.g. the recorder's profile-lock advice).
|
|
309
|
+
*/
|
|
310
|
+
export class BrowserSetupError extends Error {
|
|
311
|
+
override name = "BrowserSetupError";
|
|
312
|
+
}
|
|
313
|
+
|
|
314
|
+
/** Which browser binary every Playwright launch uses — spread into launch options. */
|
|
315
|
+
export type ChromeLaunchTarget = { channel: ChromeChannel } | { executablePath: string };
|
|
316
|
+
|
|
317
|
+
/**
|
|
318
|
+
* The browser every launch site uses (issue #80). Precedence:
|
|
319
|
+
* 1. AIDEMO_CHROME_EXECUTABLE — explicit binary path (`{ executablePath }`)
|
|
320
|
+
* 2. AIDEMO_CHROME_CHANNEL — a Playwright channel (`{ channel }`), e.g.
|
|
321
|
+
* `chromium` (Chrome for Testing — for managed Chrome that kills automated
|
|
322
|
+
* sessions; needs `npx playwright install chromium`)
|
|
323
|
+
* 3. default `{ channel: "chrome" }` — system Google Chrome, exactly the
|
|
324
|
+
* engine's historical launch options.
|
|
325
|
+
* Throws on an unknown channel name. Re-read on every call (env is live).
|
|
326
|
+
*/
|
|
327
|
+
export function chromeLaunchOptions(): ChromeLaunchTarget {
|
|
328
|
+
const exe = process.env.AIDEMO_CHROME_EXECUTABLE?.trim();
|
|
329
|
+
if (exe) return { executablePath: exe };
|
|
330
|
+
const raw = process.env.AIDEMO_CHROME_CHANNEL?.trim();
|
|
331
|
+
if (!raw) return { channel: "chrome" };
|
|
332
|
+
const channel = raw.toLowerCase();
|
|
333
|
+
if (!(CHROME_CHANNELS as readonly string[]).includes(channel)) {
|
|
334
|
+
throw new BrowserSetupError(
|
|
335
|
+
`AIDEMO_CHROME_CHANNEL="${raw}" is not a supported browser channel. ` +
|
|
336
|
+
`Use one of: ${CHROME_CHANNELS.join(", ")} — or set AIDEMO_CHROME_EXECUTABLE ` +
|
|
337
|
+
`to a Chromium-based browser binary instead.`
|
|
338
|
+
);
|
|
339
|
+
}
|
|
340
|
+
return { channel: channel as ChromeChannel };
|
|
341
|
+
}
|