@tandryukha/aidemo 0.16.1 → 0.17.0

package/README.md

@@ -103,8 +103,10 @@ storyboard.json
    → compose   trim idle · sync · auto-zoom · cards · caption · mux → output/final-demo.mp4
 ```
 
-**4 · What you get** — `output/final-demo.mp4`, plus a README-ready GIF
-(`aidemo gif`) and named stills (`aidemo stills`) from the same take. UI changed?
+**4 · What you get** — `output/final-demo.mp4` with `final-demo.srt`/`.vtt`
+captions beside it (timed to the video; `captions.track` also embeds them as a
+subtitle track viewers can toggle), plus a README-ready GIF (`aidemo gif`) and
+named stills (`aidemo stills`) from the same take. UI changed?
 Re-run against the same storyboard — no re-recording by hand.
 
 The design goal: demos that look **human-made and snappy**, not like an AI
@@ -282,7 +284,7 @@ demos/<name>/          ← your working area (untracked; scaffold with `aidemo i
   generated/  storyboard.json  timeline.json  captions.{srt,vtt,cues.json}
   recordings/ raw.webm (or raw.mp4 for native/OBS capture)
   audio/      scene-*.mp3  narration.mp3  voice.json
-  output/     final-demo.mp4
+  output/     final-demo.mp4  final-demo.{srt,vtt} (captions timed to the video)
   logs/       <command>.log  fail-<scene>-<n>.{png,json} (on a failed action)
 ```
 

package/docs/AUTHORING.md

@@ -104,7 +104,8 @@ so the forecast is in the log next to the take. The same numbers come back
 `storyboard.json` → **voice** (TTS per scene) → **record** (drives Chrome,
 injected animated cursor, video + timeline) → **captions** (Whisper word
 timing) → **compose** (trim idle "thinking" spans, sync each scene's video to
-its narration length, overlay captions, mux audio) → `output/final-demo.mp4`.
+its narration length, overlay captions, mux audio) → `output/final-demo.mp4`,
+with `final-demo.srt`/`.vtt` captions beside it, timed to the video.
 
 No LLM runs during capture — the recording is a deterministic replay of your
 fixed action-spec. Voice and captions call an OpenAI-compatible endpoint
@@ -285,10 +286,11 @@ Cinematic keys (all opt-in; omit for the plain look):
   (per-action `keystrokes` overrides). See *Attention*.
 - `captions: {position?="bottom"|"top", style?="pill"|"bar"|"none", font?,
   size?, color?, background?, maxWords?, maxCueMs?, minCueMs?, maxLines?,
-  maxCharsPerLine?}` — caption strip placement and look; scenes may
-  override `position` with their own `captions`. Compose also auto-flips a cue
-  to the top while an overlay occupies the bottom band. See *Attention* and
-  *Produced look*.
+  maxCharsPerLine?, track?=false}` — caption strip placement and look; scenes
+  may override `position` with their own `captions`. Compose also auto-flips a
+  cue to the top while an overlay occupies the bottom band. `track: true`
+  (top level only) also embeds the captions as a soft subtitle track. See
+  *Attention* and *Produced look*.
 - `redact: [{selector, frame?, blur?}]` — **blur regions at compose time**
   (prices, emails, tokens); scenes may add their own `redact`. See *Attention*.
 - `hide: [selector, …]` — hide elements at **record** time (teasers, cookie
@@ -583,10 +585,27 @@ shadowed window on a styled canvas, with captions in the padding band below.
 
 **Caption styles** (`captions`, top level): `style` is `"pill"` (default —
 centered rounded box), `"bar"` (full-width band flush with the frame edge; the
-usual choice under a `frame`), or `"none"` (no captions burned — the
-SRT/VTT files are still written, for players and platforms that render their
-own). `font`, `size` (default 30, logical px), `color`, `background` (any
-CSS) restyle the strip; `position` still flips per scene / automatically.
+usual choice under a `frame`), or `"none"` (no captions burned — use the
+sidecar files or the subtitle track below instead). `font`, `size` (default
+30, logical px), `color`, `background` (any CSS) restyle the strip;
+`position` still flips per scene / automatically.
+
+**Caption files and subtitle track.** Every compose writes
+`output/final-demo.srt` and `output/final-demo.vtt` (`final-demo.<lang>.*`
+for language renders) beside the video, **timed to the final video**: every
+cue is shifted by the intro card's length, exactly like the burned strip and
+the narration. That's what a player or host (YouTube, Loom, Confluence, an
+LMS) wants when it loads captions by file name. `generated/captions.*` stay
+narration-relative (t=0 is the first narrated word) for the engine's own
+use, so don't ship those next to a video that has an intro card. With
+`captions.track: true` compose also muxes the same cues into the MP4 as a
+`mov_text` subtitle track, tagged with the narration language (`--lang`, else
+the storyboard's `language`, else English) and **off until the viewer turns
+it on**. Re-composing replaces the track rather than adding another. For
+captions a viewer can toggle instead of burned ones, combine
+`style: "none"` with `track: true`; with burned captions as well, a viewer
+who turns the track on sees them twice (lint note `caption-track-burned`).
+`report.json` → `captions.srt` / `.vtt` / `.track` record what was written.
 
 **Output presets** (`output.preset`) fill `width`/`height`/`fit` when you
 don't set them explicitly: `youtube` 1920×1080, `short` 1080×1920,
@@ -606,7 +625,8 @@ artifacts in `output/walkthrough/`: `index.html` (one card per scene with
 its payoff frame, title, narration and a jump-to-time button over the video;
 ← / → keys step through, Enter plays from a step), `guide.md` (the same as a
 README/SOP section: heading + frame + narration per scene), `scene-NN-<id>.png`
-frames, the SRT/VTT captions, and `walkthrough.json`. Frames come from the
+frames, the SRT/VTT captions (timed to the video, like the sidecars), and
+`walkthrough.json`. Frames come from the
 **final** video, so they carry zoom, cursor, overlays and the frame. Scene
 `title`s name the steps (falling back to ids).
 
@@ -1162,7 +1182,11 @@ off), focus events kept vs dropped, and `warnings[]` —
 the narration even at the x1.6 ceiling and were cut), `blank-tail`,
 `stale-captions`, `focus-dropped`, `cursor-missing`, `anchor-unreachable`
 (an anchored action could not reach its word within the retime limits; the
-scene's `anchors[]` shows `targetMs` / `landedMs` / `offMs`). A held scene
+scene's `anchors[]` shows `targetMs` / `landedMs` / `offMs`),
+`caption-track-empty` (`captions.track` with no caption cues: run `captions`
+first) and `caption-track-failed` (the ffmpeg build could not mux the
+subtitle track; the video ships without it and the sidecars still carry the
+captions). A held scene
 is a storyboard problem, not a compose problem: give it on-screen beats or
 shorten its narration, then re-run `voice` + `compose`.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tandryukha/aidemo",
-  "version": "0.16.1",
+  "version": "0.17.0",
   "license": "MIT",
   "description": "AI Demo Engine — turn a storyboard into a narrated, captioned product demo video",
   "type": "module",
@@ -37,12 +37,11 @@
   "dependencies": {
     "@modelcontextprotocol/sdk": "1.32.2",
     "commander": "^12.1.0",
-    "openai": "^4.67.0",
+    "openai": "^6.49.0",
     "playwright": "^1.55.1",
-    "re2js": "2.8.6",
+    "re2js": "2.9.1",
     "tsx": "^4.23.1",
-    "zod": "^3.23.8",
-    "zod-to-json-schema": "3.25.2"
+    "zod": "^4.6.7"
   },
   "devDependencies": {
     "@types/node": "^22.20.1",

package/src/caption-track.ts

@@ -0,0 +1,176 @@
+import { promises as fs } from "node:fs";
+import type { Project } from "./project.js";
+import { toSrt, toVtt, type Cue } from "./captions.js";
+import { exists, readJson } from "./util.js";
+
+/**
+ * Captions timed to the FINAL video (issue #89).
+ *
+ * generated/captions.* are narration-relative: t=0 is the start of the
+ * narration track. The final MP4 opens on the intro card, and compose delays
+ * the narration (adelay) and the burned cues by the card's length — so those
+ * files run early by exactly that much against final-demo.mp4. Compose writes
+ * re-timed copies beside the video (what players and hosts pick up by file
+ * name) and, with `captions.track`, muxes them as an off-by-default subtitle
+ * track. The walkthrough bundle ships the same re-timed cues.
+ */
+
+/** Cues re-timed onto the final video: the intro card delays every cue by `introMs`. */
+export function shiftCues(cues: Cue[], introMs: number): Cue[] {
+  if (!introMs) return cues;
+  return cues.map((c) => ({ ...c, startMs: c.startMs + introMs, endMs: c.endMs + introMs }));
+}
+
+/**
+ * This project's caption cues (language-scoped), re-timed onto the final
+ * video. null when the captions stage has produced none.
+ */
+export async function finalVideoCues(project: Project, introMs: number): Promise<Cue[] | null> {
+  if (!(await exists(project.captionsCuesPath))) return null;
+  const cues = (await readJson<Cue[]>(project.captionsCuesPath)) ?? [];
+  return cues.length ? shiftCues(cues, introMs) : null;
+}
+
+/**
+ * Write output/final-demo[.<lang>].srt/.vtt for these cues, or — when there
+ * are none — remove sidecars an earlier compose left, so a stale file never
+ * sits beside a video it doesn't match.
+ */
+export async function writeCaptionSidecars(
+  project: Project,
+  cues: Cue[] | null
+): Promise<{ srt: string; vtt: string } | null> {
+  if (!cues?.length) {
+    await Promise.all(
+      [project.outputSrtPath, project.outputVttPath].map((p) => fs.rm(p, { force: true }))
+    );
+    return null;
+  }
+  await fs.writeFile(project.outputSrtPath, toSrt(cues));
+  await fs.writeFile(project.outputVttPath, toVtt(cues));
+  return { srt: project.outputSrtPath, vtt: project.outputVttPath };
+}
+
+/** ISO 639-1 → ISO 639-2/T, the three-letter code an MP4 track's language carries. */
+const ISO639_2T: Record<string, string> = {
+  af: "afr", ar: "ara", bg: "bul", bn: "ben", ca: "cat", cs: "ces", cy: "cym",
+  da: "dan", de: "deu", el: "ell", en: "eng", es: "spa", et: "est", eu: "eus",
+  fa: "fas", fi: "fin", fr: "fra", ga: "gle", gl: "glg", he: "heb", hi: "hin",
+  hr: "hrv", hu: "hun", hy: "hye", id: "ind", is: "isl", it: "ita", ja: "jpn",
+  ka: "kat", kk: "kaz", ko: "kor", lt: "lit", lv: "lav", mk: "mkd", ms: "msa",
+  mt: "mlt", nb: "nob", nl: "nld", nn: "nno", no: "nor", pl: "pol", pt: "por",
+  ro: "ron", ru: "rus", sk: "slk", sl: "slv", sq: "sqi", sr: "srp", sv: "swe",
+  sw: "swa", ta: "tam", te: "tel", th: "tha", tl: "tgl", tr: "tur", uk: "ukr",
+  ur: "urd", vi: "vie", zh: "zho",
+};
+
+/**
+ * MP4 language code for a BCP-47 tag ("en", "en-GB", "pt-BR"): its ISO
+ * 639-2/T code. A tag that already leads with a three-letter code ("fil",
+ * "yue") passes through; anything unknown is "und" (undetermined).
+ */
+export function mp4Language(tag: string | undefined): string {
+  const primary = (tag ?? "").trim().toLowerCase().split(/[-_]/)[0];
+  if (/^[a-z]{3}$/.test(primary)) return primary;
+  return ISO639_2T[primary] ?? "und";
+}
+
+/** Track handler types that carry subtitles/captions in ISO-BMFF / QuickTime. */
+const SUBTITLE_HANDLERS = new Set(["sbtl", "text", "subt"]);
+
+interface Box {
+  type: string;
+  start: number;
+  size: number;
+  header: number;
+}
+
+/** The boxes laid end to end in buf[from, to); stops at the first malformed one. */
+function* boxes(buf: Buffer, from: number, to: number): Generator<Box> {
+  let p = from;
+  while (p + 8 <= to) {
+    let size = buf.readUInt32BE(p);
+    const type = buf.toString("latin1", p + 4, p + 8);
+    let header = 8;
+    if (size === 1) {
+      if (p + 16 > to) return;
+      size = Number(buf.readBigUInt64BE(p + 8));
+      header = 16;
+    } else if (size === 0) {
+      size = to - p;
+    }
+    if (size < header || p + size > to) return;
+    yield { type, start: p, size, header };
+    p += size;
+  }
+}
+
+const child = (buf: Buffer, parent: Box, type: string): Box | undefined => {
+  for (const b of boxes(buf, parent.start + parent.header, parent.start + parent.size)) {
+    if (b.type === type) return b;
+  }
+  return undefined;
+};
+
+/**
+ * Clear `track_enabled` in every subtitle track's `tkhd`, in place — one byte
+ * per track, nothing moves, so the faststart layout survives. ffmpeg's mov
+ * muxer enables the first track of each media type whatever `-disposition`
+ * says, and players show an enabled subtitle track from the start; with the
+ * flag cleared the track is listed but off until the viewer picks it.
+ * `tracks` = subtitle tracks found, `changed` = flags actually cleared.
+ */
+export async function disableSubtitleTracks(
+  file: string
+): Promise<{ tracks: number; changed: number }> {
+  const fh = await fs.open(file, "r+");
+  try {
+    const { size: fileSize } = await fh.stat();
+    // Top-level boxes by header only (mdat is most of the file): find moov.
+    const head = Buffer.alloc(16);
+    let pos = 0;
+    let moov: { start: number; size: number } | null = null;
+    while (pos + 8 <= fileSize) {
+      const { bytesRead } = await fh.read(head, 0, 16, pos);
+      if (bytesRead < 8) break;
+      let size = head.readUInt32BE(0);
+      const type = head.toString("latin1", 4, 8);
+      if (size === 1) size = bytesRead >= 16 ? Number(head.readBigUInt64BE(8)) : 0;
+      else if (size === 0) size = fileSize - pos;
+      if (size < 8) break;
+      if (type === "moov") {
+        moov = { start: pos, size };
+        break;
+      }
+      pos += size;
+    }
+    if (!moov) return { tracks: 0, changed: 0 };
+    const buf = Buffer.alloc(moov.size);
+    await fh.read(buf, 0, moov.size, moov.start);
+    const moovBox = boxes(buf, 0, buf.length).next().value;
+    if (!moovBox) return { tracks: 0, changed: 0 };
+    let tracks = 0;
+    let changed = 0;
+    for (const trak of boxes(buf, moovBox.header, moovBox.size)) {
+      if (trak.type !== "trak") continue;
+      const tkhd = child(buf, trak, "tkhd");
+      const mdia = child(buf, trak, "mdia");
+      const hdlr = mdia && child(buf, mdia, "hdlr");
+      if (!tkhd || !hdlr) continue;
+      // hdlr payload: version(1) flags(3) pre_defined(4) handler_type(4)
+      const at = hdlr.start + hdlr.header + 8;
+      if (!SUBTITLE_HANDLERS.has(buf.toString("latin1", at, at + 4))) continue;
+      tracks++;
+      // tkhd payload: version(1) flags(3); track_enabled = bit 0 of the low byte.
+      const flags = tkhd.start + tkhd.header + 3;
+      if (buf[flags] & 0x01) {
+        buf[flags] &= ~0x01;
+        await fh.write(buf, flags, 1, moov.start + flags);
+        changed++;
+      }
+    }
+    return { tracks, changed };
+  } finally {
+    await fh.close();
+  }
+}

package/src/captions.ts

@@ -707,7 +707,7 @@ export function cueStats(cues: Cue[], minCueMs = DEFAULT_SEG.minCueMs): {
   return { short, midClause, midClausePct: Math.round((midClause / breaks) * 100) };
 }
 
-function toSrt(cues: Cue[]): string {
+export function toSrt(cues: Cue[]): string {
   return (
     cues
       .map(
@@ -718,7 +718,7 @@ function toSrt(cues: Cue[]): string {
   );
 }
 
-function toVtt(cues: Cue[]): string {
+export function toVtt(cues: Cue[]): string {
   const vt = (ms: number) => srtTime(ms).replace(",", ".");
   return (
     "WEBVTT\n\n" +

package/src/compose.ts

@@ -19,7 +19,14 @@ import {
   HoldSchema,
   AutoIdleSchema,
 } from "./types.js";
-import { sceneWordsFor, type Cue } from "./captions.js";
+import { sceneWordsFor, toSrt, type Cue } from "./captions.js";
+import {
+  disableSubtitleTracks,
+  finalVideoCues,
+  mp4Language,
+  writeCaptionSidecars,
+} from "./caption-track.js";
+import { resolveNarrationLanguage } from "./i18n.js";
 import { wordStartMs } from "./anchors.js";
 import { renderCaptionPngs } from "./caption-render.js";
 import { renderCardPng } from "./cards.js";
@@ -1099,8 +1106,60 @@ export async function compose(
           : []),
       ])
     : null;
-  await muxAudio(project, storyboard, finalVideo, introMs, chapters);
+  // Captions timed to the FINAL video (issue #89): the cues the strip burns,
+  // shifted by the intro card exactly like the overlay and the narration's
+  // adelay. Always written beside the MP4; `captions.track` also embeds them.
+  const videoCues = await finalVideoCues(project, introMs);
+  let subtitle: SubtitleTrack | null = null;
+  if (storyboard.captions?.track) {
+    if (videoCues) {
+      subtitle = {
+        path: resolve(tmp, "track.srt"),
+        language: mp4Language(resolveNarrationLanguage(storyboard, project.lang) ?? "en"),
+      };
+      await fs.writeFile(subtitle.path, toSrt(videoCues));
+    } else {
+      warn(
+        "caption-track-empty",
+        `captions.track is set but there are no caption cues — run: aidemo captions ${project.dir}`
+      );
+    }
+  }
+  try {
+    await muxAudio(project, storyboard, finalVideo, introMs, chapters, subtitle);
+  } catch (err) {
+    if (!subtitle) throw err;
+    // An ffmpeg build without the srt demuxer / mov_text encoder: keep the
+    // video, drop the track — the sidecars below still carry the captions.
+    // runFfmpeg's message is "ffmpeg exited N" + the stderr tail; ffmpeg
+    // names the root cause on the first stderr line.
+    const [exit, ...stderr] = String((err as Error).message ?? err).split("\n");
+    const why = stderr.map((l) => l.trim()).find(Boolean);
+    warn(
+      "caption-track-failed",
+      `could not embed the subtitle track, so the video has none (the .srt/.vtt sidecars ` +
+        `still carry the captions): ${exit}${why ? ` — ${why.slice(0, 200)}` : ""}`
+    );
+    subtitle = null;
+    await muxAudio(project, storyboard, finalVideo, introMs, chapters);
+  }
+  if (subtitle) {
+    const { tracks } = await disableSubtitleTracks(project.outputPath);
+    if (tracks === 0) {
+      warn("caption-track-failed", "the muxed MP4 carries no subtitle track");
+      subtitle = null;
+    } else {
+      log(`subtitle track: ${subtitle.language} (mov_text, off until the viewer turns it on)`);
+    }
+  }
   ok(`final video → ${project.outputPath}`);
+  const sidecars = await writeCaptionSidecars(project, videoCues);
+  if (sidecars) {
+    ok(
+      `captions → ${relative(project.dir, sidecars.srt)} + .vtt (timed to the video` +
+        `${introMs ? `: cues +${introMs}ms for the intro card` : ""})`
+    );
+  }
   // Blank-open warning + poster (issue #47; measured after scene assembly,
   // above). The poster used to be taken at a fixed offset, which on a demo
   // that opens with a `goto` is exactly the white load frame; pick the first
@@ -1166,7 +1225,12 @@ export async function compose(
     scenes: sceneReports,
     zoom: { focusTotal: zoomEvents.length, focusDropped },
     cursorPoints: cursorPts.length,
-    captions: { cues: captionsResult.cues, passes: captionsResult.passes },
+    captions: {
+      cues: captionsResult.cues,
+      passes: captionsResult.passes,
+      ...(sidecars ? { srt: sidecars.srt, vtt: sidecars.vtt } : {}),
+      ...(subtitle ? { track: subtitle.language } : {}),
+    },
     hold: hold.mode,
     ...(poster ? { poster } : {}),
     blankOpenMs: Math.round(blankOpenMs),
@@ -1902,7 +1966,7 @@ async function burnCaptions(
 ): Promise<{ video: string; cues: number; passes: number; flipped: number }> {
   const none = { video: silentVideo, cues: 0, passes: 0, flipped: 0 };
   if (placement.look?.style === "none") {
-    log("captions: style none — not burned (SRT/VTT files still written)");
+    log("captions: style none — not burned (the .srt/.vtt beside the video carry them)");
     return none;
   }
   if (!(await exists(project.captionsCuesPath))) return none;
@@ -2007,12 +2071,20 @@ async function fileToDataUri(path: string): Promise<string> {
   return `data:${mime};base64,${buf.toString("base64")}`;
 }
 
+/** An SRT (already on the final video's clock) to embed as a soft subtitle track. */
+interface SubtitleTrack {
+  path: string;
+  /** ISO 639-2 code for the track's language tag. */
+  language: string;
+}
+
 async function muxAudio(
   project: Project,
   storyboard: Storyboard,
   video: string,
   introMs: number,
-  chaptersPath: string | null = null
+  chaptersPath: string | null = null,
+  subtitle: SubtitleTrack | null = null
 ): Promise<void> {
   let musicPath = await resolveMusic(project, storyboard);
   const videoMs = await probeDurationMs(video);
@@ -2041,6 +2113,12 @@ async function muxAudio(
     chaptersIdx = musicPath ? 3 : 2;
     args.push("-i", chaptersPath);
   }
+  // Subtitle SRT last, so every index above is unchanged without one.
+  let subtitleIdx = -1;
+  if (subtitle) {
+    subtitleIdx = 2 + (musicPath ? 1 : 0) + (chaptersPath ? 1 : 0);
+    args.push("-i", subtitle.path);
+  }
 
   const delay = introMs > 0 ? `,adelay=${Math.round(introMs)}:all=1` : "";
   const filters: string[] = [];
@@ -2111,6 +2189,7 @@ async function muxAudio(
 
   args.push("-filter_complex", filters.join(";"));
   args.push("-map", "0:v", "-map", "[aout]");
+  if (subtitle) args.push("-map", `${subtitleIdx}:s`);
   if (chaptersIdx >= 0) args.push("-map_metadata", String(chaptersIdx));
   args.push(
     "-c:v",
@@ -2118,7 +2197,21 @@ async function muxAudio(
     "-c:a",
     "aac",
     "-b:a",
-    "192k",
+    "192k"
+  );
+  if (subtitle) {
+    // mov_text is the MP4 text-subtitle codec; not default — and compose then
+    // clears the tkhd enabled flag ffmpeg sets anyway (disableSubtitleTracks).
+    args.push(
+      "-c:s",
+      "mov_text",
+      "-metadata:s:s:0",
+      `language=${subtitle.language}`,
+      "-disposition:s:0",
+      "0"
+    );
+  }
+  args.push(
     // moov atom up front so the MP4 streams/scrubs in a browser tab or a PR
     // preview before it has fully downloaded (frames identical; container only).
     "-movflags",

package/src/lint.ts

@@ -532,6 +532,27 @@ export function lintStoryboard(
       fix: "add a `cursor: {}` block (the take is then recorded cursor-free and the cursor is overlaid at compose)",
     });
   }
+  if (sb.captions?.track && sb.captions.style !== "none") {
+    push({
+      severity: "info",
+      code: "caption-track-burned",
+      message:
+        "`captions.track` embeds a soft subtitle track while the captions are also burned in — " +
+        "a viewer who turns the track on sees every caption twice",
+      fix: 'set `captions.style: "none"` for toggleable captions only, or drop `track` to keep just the burned strip',
+    });
+  }
+  for (const scene of sb.scenes) {
+    if (scene.captions?.track != null) {
+      push({
+        severity: "info",
+        code: "caption-track-scene",
+        scene: scene.id,
+        message: "`captions.track` is storyboard-level only — a scene's `captions` just moves the strip",
+        fix: "move `track` to the top-level `captions` block",
+      });
+    }
+  }
   if (sb.zoom && sb.zoom.enabled !== false) {
     const capped = clampScaleForWidth(sb.zoom.scale, sb.video.width);
     if (capped < sb.zoom.scale) {

package/src/mcp/server.ts

@@ -1,7 +1,6 @@
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
 import { z } from "zod";
-import { zodToJsonSchema } from "zod-to-json-schema";
 import { readFile } from "node:fs/promises";
 import { resolve } from "node:path";
 import {
@@ -58,6 +57,30 @@ tool (title + body — environment/log context is auto-attached) before you
 finish; skip if nothing came up. A failed job_status result also carries a
 feedbackHint pointing back here.`;
 
+/**
+ * JSON Schema for storyboard.json, generated from the engine's zod schema.
+ * Input semantics (a field with a default is optional — what an author
+ * writes), draft-07, every sub-schema inlined, and unknown keys rejected
+ * (`additionalProperties: false`): zod drops unknown keys silently, so the
+ * schema says they are not part of the contract. Same contract the
+ * zod-to-json-schema export had before the zod 4 migration.
+ */
+function storyboardJsonSchema(): Record<string, unknown> {
+  return z.toJSONSchema(StoryboardSchema, {
+    target: "draft-7",
+    io: "input",
+    unrepresentable: "any",
+    override: (ctx) => {
+      if (
+        ctx.zodSchema._zod.def.type === "object" &&
+        ctx.jsonSchema.additionalProperties === undefined
+      ) {
+        ctx.jsonSchema.additionalProperties = false;
+      }
+    },
+  }) as Record<string, unknown>;
+}
+
 type ToolResult = {
   content: Array<{ type: "text"; text: string }>;
   structuredContent?: Record<string, unknown>;
@@ -100,7 +123,7 @@ const JOB_STATUS_SHAPE = {
   endedAt: z.string().nullable(),
   logFile: z.string(),
   logTail: z.array(z.string()),
-  result: z.record(z.unknown()).optional(),
+  result: z.record(z.string(), z.unknown()).optional(),
   error: z
     .object({
       message: z.string(),
@@ -307,12 +330,12 @@ export function buildMcpServer(): { server: McpServer; jobs: JobManager } {
         "JSON Schema for storyboard.json, generated from the engine's own " +
         "zod schema — the exact contract the engine validates against.",
       inputSchema: {},
-      outputSchema: { schema: z.record(z.unknown()) },
+      outputSchema: { schema: z.record(z.string(), z.unknown()) },
       annotations: { readOnlyHint: true },
     },
     async () =>
       jsonResult({
-        schema: zodToJsonSchema(StoryboardSchema, { $refStrategy: "none" }),
+        schema: storyboardJsonSchema(),
       })
   );
 
@@ -1103,6 +1126,8 @@ export function buildMcpServer(): { server: McpServer; jobs: JobManager } {
           ...(walkthrough ? { walkthrough } : {}),
           timeline: lp.timelinePath,
           captionsSrt: lp.captionsSrtPath,
+          // Sidecars timed to the final video (+ the embedded track's language).
+          captions: report.captions,
           report: lp.reportPath,
           durationMs: report.durationMs,
           // warningCount lets an orchestrating agent gate on the render
@@ -1179,7 +1204,7 @@ export function buildMcpServer(): { server: McpServer; jobs: JobManager } {
 
   registerJob(
     "compose",
-    "Trim, sync, mux and caption into output/final-demo.mp4.",
+    "Trim, sync, mux and caption into output/final-demo.mp4 (+ final-demo.srt/.vtt timed to it).",
     {
       dir: DIR_INPUT,
       gif: z.boolean().optional().describe("also export output/final-demo.gif"),
@@ -1197,6 +1222,8 @@ export function buildMcpServer(): { server: McpServer; jobs: JobManager } {
         return {
           output: lp.outputPath,
           ...(gifPath ? { gif: gifPath } : {}),
+          // Sidecars timed to the final video (+ the embedded track's language).
+          captions: report.captions,
           report: lp.reportPath,
           durationMs: report.durationMs,
           // warningCount lets an orchestrating agent gate on the render
@@ -1438,11 +1465,7 @@ export function buildMcpServer(): { server: McpServer; jobs: JobManager } {
         {
           uri: uri.href,
           mimeType: "application/json",
-          text: JSON.stringify(
-            zodToJsonSchema(StoryboardSchema, { $refStrategy: "none" }),
-            null,
-            2
-          ),
+          text: JSON.stringify(storyboardJsonSchema(), null, 2),
         },
       ],
     })

package/src/project.ts

@@ -12,7 +12,7 @@ import { ensureDir, exists, readJson, writeJson, log } from "./util.js";
  *     generated/storyboard.json  timeline.json  captions.srt  captions.vtt
  *     recordings/raw.webm
  *     audio/scene-*.mp3  narration.mp3  voice.json
- *     output/final-demo.mp4
+ *     output/final-demo.mp4  final-demo.srt  final-demo.vtt
  *     logs/
  */
 export class Project {
@@ -117,6 +117,17 @@ export class Project {
   get outputPath() {
     return this.p("output", this.lang ? `final-demo.${this.lang}.mp4` : "final-demo.mp4");
   }
+  /**
+   * Caption sidecars named after the video (final-demo[.<lang>].srt/.vtt), so
+   * a player that loads captions by file name finds them. Timed to the FINAL
+   * video — unlike generated/captions.*, which start at the first narrated word.
+   */
+  get outputSrtPath() {
+    return this.outputPath.replace(/\.mp4$/, ".srt");
+  }
+  get outputVttPath() {
+    return this.outputPath.replace(/\.mp4$/, ".vtt");
+  }
   get gifPath() {
     return this.p("output", this.lang ? `final-demo.${this.lang}.gif` : "final-demo.gif");
   }

package/src/types.ts

@@ -503,6 +503,15 @@ export const CaptionsConfigSchema = z.object({
   maxLines: z.number().int().min(1).max(3).optional(),
   /** Characters per line used to size a cue. Default 38. */
   maxCharsPerLine: z.number().int().min(16).max(80).optional(),
+  /**
+   * Also embed the captions in the MP4 as a soft subtitle track (mov_text,
+   * tagged with the narration language) that players leave OFF until the
+   * viewer turns it on (issue #89). Storyboard level only. Pair it with
+   * `style: "none"` for toggleable captions instead of burned ones. Compose
+   * always writes the same cues beside the video as final-demo.srt/.vtt;
+   * this adds the in-file track. Default false.
+   */
+  track: z.boolean().optional(),
 });
 export type CaptionsConfig = z.infer<typeof CaptionsConfigSchema>;
 
@@ -1236,7 +1245,8 @@ export type Timeline = z.infer<typeof TimelineSchema>;
 
 export const ComposeWarningSchema = z.object({
   /** Stable code: scene-freeze | overrun-trim | blank-tail | stale-captions |
-   *  focus-dropped | cursor-missing | scene-no-narration */
+   *  focus-dropped | cursor-missing | scene-no-narration | caption-track-empty |
+   *  caption-track-failed */
   code: z.string(),
   scene: z.string().optional(),
   message: z.string(),
@@ -1287,7 +1297,16 @@ export const ComposeReportSchema = z.object({
   scenes: z.array(ComposeSceneReportSchema),
   zoom: z.object({ focusTotal: z.number(), focusDropped: z.number() }),
   cursorPoints: z.number(),
-  captions: z.object({ cues: z.number(), passes: z.number() }),
+  captions: z.object({
+    /** Cues burned into the picture (0 with `style: "none"`). */
+    cues: z.number(),
+    passes: z.number(),
+    /** Sidecars timed to the final video (intro card included), when there are cues. */
+    srt: z.string().optional(),
+    vtt: z.string().optional(),
+    /** ISO 639-2 language of the embedded subtitle track (`captions.track`). */
+    track: z.string().optional(),
+  }),
   /** Attention overlays drawn (highlight/spotlight/callout), key chips, click rings, redact spans. */
   attention: z
     .object({

package/src/walkthrough.ts

@@ -12,6 +12,8 @@ import { resolve, isAbsolute, basename } from "node:path";
 import { Project } from "./project.js";
 import { ComposeReportSchema, type Storyboard } from "./types.js";
 import { runFfmpeg } from "./ffmpeg.js";
+import { toSrt, toVtt } from "./captions.js";
+import { finalVideoCues } from "./caption-track.js";
 import { ensureDir, exists, readJson, writeJson, log, step } from "./util.js";
 
 export interface WalkthroughOptions {
@@ -112,17 +114,16 @@ export async function exportWalkthrough(
     log(`scene ${r.id}: ${mmss(startMs)} → ${image}`);
   }
 
-  // Captions travel with the walkthrough (players and LMSs want the files).
+  // Captions travel with the walkthrough (players and LMSs want the files),
+  // timed to the video they accompany — generated/captions.* start at the
+  // first narrated word, i.e. early by the intro card (issue #89).
   const captions: { srt?: string; vtt?: string } = {};
-  for (const [key, src] of [
-    ["srt", project.captionsSrtPath],
-    ["vtt", project.captionsVttPath],
-  ] as const) {
-    if (await exists(src)) {
-      const dst = resolve(dir, basename(src));
-      await fs.copyFile(src, dst);
-      captions[key] = dst;
-    }
+  const cues = await finalVideoCues(project, introMs);
+  if (cues) {
+    captions.srt = resolve(dir, basename(project.captionsSrtPath));
+    captions.vtt = resolve(dir, basename(project.captionsVttPath));
+    await fs.writeFile(captions.srt, toSrt(cues));
+    await fs.writeFile(captions.vtt, toVtt(cues));
   }
 
   const videoRel = `../${basename(video)}`;