reelkit-cli 0.3.1 → 0.4.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reelkit-cli",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "description": "CLI and Claude skill for making short-form video with a shared asset library.",
   "license": "MIT",
   "author": "Daniel Livshin",

package/skill/SKILL.md

@@ -25,6 +25,8 @@ For each file the user gives, look at it, then register it with a description of
 `reelkit assets upload ./logo.png --describe "Acme logo, white wordmark on blue"`
 Add `--footage` for a video the motion design should be laid over. User files stay on this machine. A video of someone or something on a plain background can be made transparent: `--green` keys a green background locally for free, and `--cutout` removes any background on the server (the video is sent to Reelkit, so ask first; see `reference/clips.md`).
 
+If the user points at an existing video ("make one like this", a link or a file), it is a reference: you learn how it is built and make something new in that spirit. Read `reference/references.md`, then `reelkit ref download <url-or-file>` and `reelkit ref analyze <id>`, and look at the frames it saves. Take its structure, pace and motion; never its footage, music or words. A link is fetched on the user's machine and they are responsible for the right to download it; ask before running `reelkit ref analyze` without `--no-transcript`, because the audio (never the video) is sent to Reelkit to be transcribed.
+
 ### 2. Look
 Read `reference/styles.md` and agree the video's look with the user: one of its looks, or their own. If they already described what they want, match it and confirm in one sentence. Ask anything still open in one message, each question with a default. If any on-screen text will be Hebrew, read `reference/hebrew-rtl.md` now: it changes how words may enter and how lines are written.
 
@@ -63,6 +65,7 @@ Write `plan.json`:
 - `clipPrompt` is set only for `clip` scenes (a generated or reused video clip is the scene's picture; the rest of the scene uses `imageTags` and `shareable` as an illustration does). Clips are scarce: most videos have none or one or two.
 - `userAssetIds` lists the ids of the user's files shown in that scene.
 - `pace` is `slow`, `normal` or `fast`.
+- When the video follows a reference, add `"reference": { "id": "<id>", "take": ["fast cuts every ~1.2s"] }` (1 to 6 notes on what you took).
 
 Run `reelkit plan check`; it prints the estimated length to tell the user. Fix everything under "Fix these". Act on "Worth improving" unless you have a good reason not to.
 

package/skill/reference/clips.md

@@ -35,6 +35,7 @@ A clip is a few seconds of generated video. It is the most expensive thing Reelk
 A green-screen clip is a subject filmed on a flat pure green background. It is for a person or an object that you place over something of your own: a presenter pointing at the user's screenshot, a character beside your title, a product over a gradient.
 - Generate it with `reelkit assets gen clip --scene <sceneId> --green`. The original is saved as `assets/clip-<sceneId>.mp4` and the green is keyed out locally into `assets/clip-<sceneId>.webm`, a video with a transparent background. The manifest has both: `s.clipKey` and `s.clipKeyedKey`. Use the `.webm`.
 - Prompt it as a subject, not a scene: "A woman in a blue jacket, waist up, waves and then points to her left. Flat, evenly lit, pure green background (#00FF00)." Keep the whole subject inside the frame with room around it; no green anywhere on the subject (clothes, props, eyes); no shadows, floor or reflections on the background; no camera move, because keying a moving frame edge flickers.
+- A video model rarely produces a true chroma green; it usually gives a soft sage. When the green is too dull to key cleanly, the command does not key it: it cuts the subject out on the server instead (see "No green screen" below), says so, and uses cutout seconds from the quota. The result is used the same way.
 - If the keyed edges look green or ragged in the preview, the prompt was at fault: regenerate once with a flatter green and an evenly lit subject, using `--redo`. A clip left unkeyed (the command says so) is keyed again for free by running the same command again.
 - Layer it above the scene's background and below the captions:
 

package/skill/reference/references.md

@@ -0,0 +1,32 @@
+---
+name: references
+description: Use when the user says "make one like this" or gives a link or file of an existing video to match - how to bring it in, measure it, look at it, and turn how it is built into a plan, without copying anything from it.
+---
+
+# References: "make one like this"
+
+A reference is an existing video the user wants the new one to feel like. You learn how it is built (its structure, pace and motion) and make something new in that spirit. Nothing of it is ever reused.
+
+## Bring it in
+1. Ask for the video as a file if the user has it. A file on this machine is the clean path: `reelkit ref download ./example.mp4` copies it. A link works too, but it is fetched on the user's machine with yt-dlp (which they must have installed; the command says how if not), and the user is responsible for having the right to download that video. Say so in one sentence when you use a link. A reference is at most 10 minutes.
+2. `reelkit ref analyze <id>` measures it on this machine: scenes, cut times, pace, colours, loudness and tempo. It writes `refs/<id>/breakdown.json` and two sample frames per scene in `refs/<id>/frames/`. `reelkit ref list` shows what is in the project.
+3. Ask before it sends anything. The transcript is made on the Reelkit server, so the reference's audio (never the video) leaves the user's machine and is deleted there when the job ends. Ask "I can also transcribe its audio on the Reelkit server to understand the structure: OK?" and run `reelkit ref analyze <id> --no-transcript` if they say no. `reelkit whoami` shows the monthly transcription seconds left. If the transcript is refused (quota, not logged in), the breakdown is still saved; `reelkit ref analyze <id> --redo` adds the transcript later.
+
+## Look, then read
+Open the frames in `refs/<id>/frames/` and look at them one scene after another: how big the type is, where it sits, how much of the screen is picture and how much is colour, how busy each scene is. Then read `breakdown.json`: `pacing.averageShotSec`, `pacing.firstCutSec`, `pacing.cutsPerSecond`, the `palette`, and the `transcript` if there is one.
+
+## What to take, and what never to
+Take:
+- the structure: how it opens, how many beats it has, how it ends
+- the pace and the cut rhythm
+- how type and layout are used (big words on colour, small captions over footage, one idea per screen)
+- the kind of motion (snappy, floaty, hard cuts, slow pushes)
+- the mood of its music, as a description to search the library with
+
+Never take its footage, its music, its exact words, its logos or its faces. The transcript is for understanding the structure (where the hook ends, where the point lands), never for copying the script: write your own words about the user's subject.
+
+## Turn the pace into the plan
+- Scene count and length come from `pacing.averageShotSec`: a video of 30 seconds with shots of about 1.5 seconds is many quick scenes; since a plan has 3 to 8 scenes, group several reference shots into one scene that changes its picture inside (stacked text, quick swaps) rather than stretching it.
+- The hook's length comes from `pacing.firstCutSec`: if the reference holds its first shot for 1.2 seconds, your first scene is one short line.
+- Write the plan's `reference`: `{ "id": "<id>", "take": ["fast cuts every ~1.2s", "big type on colour fields"] }`, one to six short notes of what you took. `reelkit plan check` needs the analysis to exist and says when your scenes are much slower or faster than the reference.
+- When the reference cuts on the beat (`pacing.cutsOnBeat` high, say 0.7 or more, with `audio.tempoBpm`), plan the new video's scene changes on a regular grid at a similar tempo, and search the library for music of a similar tempo and mood. The reference's own music is never used.

package/src/cli.ts

@@ -7,6 +7,7 @@ import { init } from "./commands/init";
 import { install } from "./commands/install";
 import { openInBrowser } from "./open";
 import { planCheck } from "./commands/plan";
+import { refAnalyze, refAudio, refDownload, refList } from "./commands/ref";
 import type { Ctx, Result } from "./context";
 import { readFileSync } from "node:fs";
 
@@ -90,6 +91,17 @@ gen.command("clip [prompt]").description("Generate a scene's video clip (takes m
   .option("--share", "also add it to the shared library (only a generic clip)").option("--redo", "generate a new clip even if the scene has one").option("--resume <id>", "keep waiting for a clip already started, without paying again")
   .action(run((ctx, prompt, opts) => assetsGenClip(ctx, prompt, opts)));
 
+const ref = program.command("ref").description("Use an existing video as a reference for how a new one should feel (it is measured here; nothing of it goes into the output)");
+ref.command("download <url-or-file>").description("Bring a video into the project as a reference: a link is fetched on this machine with yt-dlp (you need the right to download it), a file is copied")
+  .option("--redo", "download a link again even if it is already in the project")
+  .action(run((ctx, input: string, opts: { redo?: boolean }) => refDownload(ctx, input, opts)));
+ref.command("audio <id>").description("Save the reference's audio as refs/<id>/audio.mp3 (on this machine)").action(run((ctx, id: string) => refAudio(ctx, id)));
+ref.command("analyze <id>").description("Measure how the reference is built: scenes, cuts, pacing, colours, sound and tempo, with sample frames. Its audio (never the video) is sent to the Reelkit server to be transcribed unless --no-transcript")
+  .option("--no-transcript", "do not send the audio to the Reelkit server; everything else is measured here")
+  .option("--redo", "measure again even if a breakdown exists")
+  .action(run((ctx, id: string, opts: { transcript?: boolean; redo?: boolean }) => refAnalyze(ctx, id, opts)));
+ref.command("list").description("List the references in this project").action(run((ctx) => refList(ctx)));
+
 program.command("plan").description("Work with plan.json").command("check").description("Validate plan.json and list what to fix or improve").action(run(planCheck));
 
 program.command("check").description("Check the composition in src/ without rendering").action(run(check));

package/src/commands/assets.ts

@@ -5,7 +5,7 @@ import { ApiFailure, download, uploadTo } from "../api/client";
 import { client, type Ctx, type Result } from "../context";
 import { LibraryKindSchema, MAX_CUTOUT_SECONDS, MAX_UPLOAD_BYTES, UploadKindSchema, type CutoutType } from "../contract";
 import { PACE_SPEED, type AssetManifest, type AssetRecord, type ScenePlan } from "../pipeline/schema";
-import { keyGreen } from "../project/chromakey";
+import { GreenTooDullError, keyGreen } from "../project/chromakey";
 import { buildManifest, voiceoverStale, type ClipRecord, type Voiceover } from "../project/manifest";
 import { loadPlan } from "./plan";
 import { FILE_NAME } from "../render/validate";
@@ -388,6 +388,10 @@ export async function assetsGenClip(
     if (opts.green && have.greenScreen && !have.keyedKey) {
       // The clip was paid for but not keyed (ffmpeg failed or --green came later): keying is free, so do it now.
       const keyed = await keyScene(project, scene.id, have, key);
+      if (!keyed.ok && (keyed.data as { dull?: boolean } | undefined)?.dull) {
+        const cut = await cutoutScene(ctx, project, scene.id, have, now);
+        return { ...cut, data: { path: have.key, ...(cut.ok ? { keyedPath: keyedKey } : {}) }, summary: cut.ok ? `The green in scene ${scene.id}'s clip was too dull to key on this machine, so the subject was cut out on the server instead. ${cut.summary}` : cut.summary };
+      }
       if (!keyed.ok) return keyed;
       return { ok: true, data: { path: have.key, keyedPath: keyedKey }, summary: `Keyed the green out of scene ${scene.id}'s clip into ${keyedKey}.` };
     }
@@ -421,20 +425,23 @@ export async function assetsGenClip(
         const record: ClipRecord = { key: path, greenScreen: green, durationSec: st.durationSec ?? seconds };
         setSceneClip(project, scene.id, record);
         let keyedPath: string | undefined;
+        let dullGreen = false;
         if (green || opts.green) {
           // The paid clip is already saved above; if keying fails, running the command again keys it without a new charge.
           const keyed = await keyScene(project, scene.id, { ...record, greenScreen: true }, key);
-          if (!keyed.ok) return keyed;
-          keyedPath = keyedKey;
+          // A model's green is often too dull to key. The subject is then cut out on the server, which needs no green at all.
+          if (!keyed.ok && (keyed.data as { dull?: boolean } | undefined)?.dull) dullGreen = true;
+          else if (!keyed.ok) return keyed;
+          else keyedPath = keyedKey;
         }
         let cutoutNote: string | undefined;
-        if (opts.cutout) {
+        if (opts.cutout || dullGreen) {
           // The paid clip is already saved above; if the cutout fails, running the command again does only the cutout.
           const cut = await cutoutScene(ctx, project, scene.id, record, now);
           const shared = st.libraryId ? `, and shared it to the library as ${st.libraryId} (awaiting review)` : "";
           if (!cut.ok) return { ...cut, data: { id, path, durationSec: record.durationSec, libraryId: st.libraryId ?? null }, summary: `Generated the clip for scene ${scene.id}${shared}. ${cut.summary}` };
           keyedPath = keyedKey;
-          cutoutNote = cut.summary;
+          cutoutNote = dullGreen ? `The generated green was too dull to key on this machine, so the subject was cut out on the server instead. ${cut.summary}` : cut.summary;
         }
         const { note } = tryManifest(project, plan);
         return {
@@ -508,7 +515,7 @@ async function keyScene(project: Project, sceneId: string, record: ClipRecord, k
   try {
     await key(project.path(record.key), project.path(keyedKey));
   } catch (e) {
-    return { ok: false, data: { path: record.key }, summary: `The clip is saved at ${record.key}, but the green could not be keyed out: ${e instanceof Error ? e.message : String(e)}` };
+    return { ok: false, data: { path: record.key, ...(e instanceof GreenTooDullError ? { dull: true } : {}) }, summary: `The clip is saved at ${record.key}, but the green could not be keyed out: ${e instanceof Error ? e.message : String(e)}` };
   }
   setSceneClip(project, sceneId, { ...record, greenScreen: true, keyedKey });
   return { ok: true, summary: "" };

package/src/commands/auth.ts

@@ -113,6 +113,6 @@ export async function whoami(ctx: Ctx): Promise<Result> {
   const q = me.quota;
   return {
     ok: true, data: me,
-    summary: `${me.handle}\nVoiceover: ${q.voiceoverChars.used}/${q.voiceoverChars.limit} characters\nImages: ${q.images.used}/${q.images.limit}\nClips: ${q.clips.used}/${q.clips.limit}\nCutouts: ${q.cutoutSeconds.used}/${q.cutoutSeconds.limit} seconds\nResets: ${q.resetsAt.slice(0, 10)}\nContributions: ${me.contributions}`,
+    summary: `${me.handle}\nVoiceover: ${q.voiceoverChars.used}/${q.voiceoverChars.limit} characters\nImages: ${q.images.used}/${q.images.limit}\nClips: ${q.clips.used}/${q.clips.limit}\nCutouts: ${q.cutoutSeconds.used}/${q.cutoutSeconds.limit} seconds\nTranscription: ${q.transcribeSeconds.used}/${q.transcribeSeconds.limit} seconds\nResets: ${q.resetsAt.slice(0, 10)}\nContributions: ${me.contributions}`,
   };
 }

package/src/commands/plan.ts

@@ -15,6 +15,25 @@ export function loadPlan(project: Project): ScenePlan {
   return parsed.data;
 }
 
+// What a plan that follows a reference must satisfy: the reference has to be measured, and the pace should be of the same order.
+function referenceChecks(project: Project, plan: ScenePlan): { mustFix: string[]; improve: string[] } {
+  if (!plan.reference) return { mustFix: [], improve: [] };
+  const id = plan.reference.id;
+  if (!/^r-[0-9a-f]{8}$/.test(id) || !project.exists(`refs/${id}/ref.json`))
+    return { mustFix: [`reference ${id} is not in this project. Run \`reelkit ref download <url-or-file>\` and use the id it prints, or remove "reference" from plan.json.`], improve: [] };
+  const file = `refs/${id}/breakdown.json`;
+  if (!project.exists(file)) return { mustFix: [`reference ${id} has not been analysed yet. Run \`reelkit ref analyze ${id}\`.`], improve: [] };
+  let average: unknown;
+  try { average = project.readJson<{ pacing?: { averageShotSec?: unknown } }>(file).pacing?.averageShotSec; } catch { average = undefined; }
+  if (typeof average !== "number" || !(average > 0)) return { mustFix: [`${file} has no average shot length. Run \`reelkit ref analyze ${id} --redo\`.`], improve: [] };
+  const ours = estimateLength(plan).seconds / plan.scenes.length;
+  const ratio = ours / average;
+  const improve = ratio > 2 || ratio < 0.5
+    ? [`The plan's scenes average ${ours.toFixed(1)}s but the reference's shots average ${average.toFixed(1)}s: ${ratio > 2 ? "the new video will feel much slower" : "the new video will feel much faster"}. Change the number of scenes or the narration length, or say in reference.take why the pace differs.`]
+    : [];
+  return { mustFix: [], improve };
+}
+
 // Hard rules (mustFix) stop the video; soft notes (shouldImprove) are advice for a better script.
 export async function planCheck(ctx: Ctx): Promise<Result> {
   const project = openProject(ctx.cwd);
@@ -34,8 +53,9 @@ export async function planCheck(ctx: Ctx): Promise<Result> {
   const footage = project.footage();
   const assets = project.assets().filter((a) => a.id !== footage?.id);
   const voiceIds = loadCredentials(ctx.env) ? (await client(ctx)("voices", {})).voices.map((v) => v.id) : undefined;
-  const mustFix = validatePlan(parsed.data, { aspect: project.config().aspect, footage, assets, voiceIds });
-  const shouldImprove = mustFix.length ? [] : reviewPlan(parsed.data, { footage });
+  const ref = referenceChecks(project, parsed.data);
+  const mustFix = [...validatePlan(parsed.data, { aspect: project.config().aspect, footage, assets, voiceIds }), ...ref.mustFix];
+  const shouldImprove = mustFix.length ? [] : [...reviewPlan(parsed.data, { footage }), ...ref.improve];
   const length = estimateLength(parsed.data);
   const estimatedSeconds = Math.round(length.seconds);
   const lines = [

package/src/commands/ref.ts

@@ -0,0 +1,307 @@
+import { execFile } from "node:child_process";
+import { createHash, randomUUID } from "node:crypto";
+import { copyFileSync, existsSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync } from "node:fs";
+import { basename, extname, resolve } from "node:path";
+import { z } from "zod";
+import { ApiFailure, uploadTo } from "../api/client";
+import { client, type Ctx, type Result } from "../context";
+import { MAX_TRANSCRIBE_SECONDS } from "../contract";
+import { loadCredentials } from "../credentials";
+import { analyzeBeats, cutsOnBeat } from "../project/beats";
+import { openProject, type Project } from "../project/project";
+import {
+  audioDuration, decodeMono, detectCuts, extractFrame, extractMp3, loudnessLufs, nearestAspect, paletteOfFrames, paletteOfVideo, probeVideo, REF_ASPECTS, toMp4,
+} from "../project/refmeasure";
+
+// A reference is for learning how a video is built, so ten minutes is more than it needs and keeps every step quick.
+export const MAX_REF_SECONDS = 600;
+const MAX_SCENES = 40;
+const REF_ID = /^r-[0-9a-f]{8}$/;
+
+export const RefRecordSchema = z.object({
+  id: z.string(), source: z.string(), urlHash: z.string().optional(),
+  durationSec: z.number(), width: z.number(), height: z.number(), aspect: z.enum(REF_ASPECTS), fps: z.number(), hasAudio: z.boolean(), createdAt: z.string(),
+});
+export type RefRecord = z.infer<typeof RefRecordSchema>;
+
+export type Breakdown = {
+  source: string; durationSec: number; width: number; height: number; aspect: RefRecord["aspect"]; fps: number;
+  scenes: { index: number; startSec: number; endSec: number; durationSec: number; frames: string[]; palette: string[] }[];
+  // True when the video has more cuts than the 40 scenes listed: the last scene then holds everything after the 39th cut.
+  scenesCapped?: boolean;
+  pacing: { cuts: number; averageShotSec: number; shortestShotSec: number; longestShotSec: number; cutsPerSecond: number; firstCutSec?: number; cutsOnBeat?: number };
+  palette: string[];
+  audio: { hasAudio: boolean; loudnessLufs?: number; tempoBpm?: number; beatConfidence: number };
+  beats?: number[];
+  transcript?: { language?: string; text: string; segments: { text: string; startSec: number; endSec: number }[] };
+};
+
+export const REF_NOTICE = "You are responsible for having the right to download this video; it is used only as a reference: none of its footage or music goes into the output.";
+
+// The external program that fetches a link. Tests give their own so that nothing touches the network.
+export type YtdlpRunner = { available(): Promise<boolean>; run(args: string[]): Promise<void> };
+export type RefDeps = { ytdlp?: YtdlpRunner };
+
+const realYtdlp: YtdlpRunner = {
+  available: () => new Promise((done) => { execFile("yt-dlp", ["--version"], (err) => done(!err)); }),
+  run: (args) => new Promise((done, fail) => {
+    // No shell: the link is one element of the argument list and can never become a command.
+    execFile("yt-dlp", args, { maxBuffer: 64 * 1024 * 1024 }, (err, _out, stderr) => {
+      if (!err) return done();
+      const last = String(stderr).trim().split("\n").filter(Boolean).at(-1) ?? err.message;
+      fail(new Error(last.replace(/^ERROR:\s*/, "")));
+    });
+  }),
+};
+
+const YTDLP_MISSING = "yt-dlp is not installed, and links need it. Install it (macOS: `brew install yt-dlp`, or `pipx install yt-dlp`) and run the command again; a video file on this machine works without it.";
+
+// One video, at most 1080p, as mp4, and nothing longer than the limit: the flags that keep a link from fetching a playlist, a channel or a huge file.
+export function ytdlpArgs(url: string, dest: string): string[] {
+  return [
+    "--ignore-config", "--no-playlist", "--max-downloads", "1", "--no-progress", "--no-warnings",
+    "--match-filter", `duration<=${MAX_REF_SECONDS}`,
+    "-f", "bv*[height<=1080][ext=mp4]+ba[ext=m4a]/b[height<=1080][ext=mp4]/bv*[height<=1080]+ba/b[height<=1080]",
+    "--merge-output-format", "mp4", "-o", dest, "--", url,
+  ];
+}
+
+const dirOf = (id: string) => `refs/${id}`;
+const fileOf = (id: string, name: string) => `refs/${id}/${name}`;
+
+function loadRef(project: Project, id: string): RefRecord {
+  if (!REF_ID.test(id)) throw new Error(`"${id}" is not a reference id. Run \`reelkit ref list\` to see them.`);
+  if (!project.exists(fileOf(id, "ref.json"))) throw new Error(`There is no reference ${id} in this project. Run \`reelkit ref list\`, or \`reelkit ref download <url-or-file>\`.`);
+  const parsed = RefRecordSchema.safeParse(project.readJson(fileOf(id, "ref.json")));
+  if (!parsed.success) throw new Error(`refs/${id}/ref.json is not valid. Download the reference again with \`reelkit ref download\`.`);
+  return parsed.data;
+}
+
+function allRefs(project: Project): RefRecord[] {
+  if (!project.exists("refs")) return [];
+  const out: RefRecord[] = [];
+  for (const name of readdirSync(project.path("refs")).sort()) {
+    if (!REF_ID.test(name) || !project.exists(fileOf(name, "ref.json"))) continue;
+    const parsed = RefRecordSchema.safeParse(project.readJsonOr(fileOf(name, "ref.json"), null));
+    if (parsed.success) out.push(parsed.data);
+  }
+  return out;
+}
+
+const fmt = (n: number) => String(Math.round(n * 10) / 10);
+const minutes = (sec: number) => (sec >= 90 ? `${Math.round(sec / 60)} minutes` : `${Math.round(sec)} seconds`);
+const tooLong = (what: string, sec: number) => `${what} is ${minutes(sec)} long and a reference is limited to ${MAX_REF_SECONDS / 60} minutes. Pick a shorter video, or trim it and give the file.`;
+
+// A link has a scheme of two or more letters; "C:\\clip.mp4" is a path.
+const looksLikeLink = (s: string) => /^[a-z][a-z0-9+.-]+:/i.test(s);
+
+export async function refDownload(ctx: Ctx, input: string, opts: { redo?: boolean } = {}, deps: RefDeps = {}): Promise<Result> {
+  const project = openProject(ctx.cwd);
+  input = input.trim();
+  const isLink = looksLikeLink(input);
+  let url: URL | undefined;
+  if (isLink) {
+    try { url = new URL(input); } catch { return { ok: false, summary: `"${input}" is not a link that can be downloaded. Give an http or https link, or a video file.` }; }
+    if (url.protocol !== "http:" && url.protocol !== "https:") return { ok: false, summary: `Only http and https links can be downloaded, not ${url.protocol}. Give an http or https link, or a video file.` };
+  }
+
+  const urlHash = url ? createHash("sha256").update(url.href).digest("hex") : undefined;
+  let id = `r-${randomUUID().replace(/-/g, "").slice(0, 8)}`;
+  if (urlHash) {
+    const have = allRefs(project).find((r) => r.urlHash === urlHash && project.exists(fileOf(r.id, "video.mp4")));
+    if (have && !opts.redo) return { ok: true, data: have, summary: `This link is already downloaded as ${have.id} (refs/${have.id}/video.mp4). Use --redo to download it again. Next: \`reelkit ref analyze ${have.id}\`.\n${REF_NOTICE}` };
+    if (have) { id = have.id; rmSync(project.path(dirOf(id)), { recursive: true, force: true }); }
+  }
+
+  const video = project.path(fileOf(id, "video.mp4"));
+  const cleanup = () => rmSync(project.path(dirOf(id)), { recursive: true, force: true });
+  let source: string;
+  try {
+    if (url) {
+      const runner = deps.ytdlp ?? realYtdlp;
+      if (!(await runner.available())) return { ok: false, summary: YTDLP_MISSING };
+      source = url.href;
+      mkdirSync(project.path(dirOf(id)), { recursive: true });
+      ctx.log(`Downloading ${url.href} with yt-dlp.`);
+      try { await runner.run(ytdlpArgs(url.href, video)); }
+      catch (e) { cleanup(); return { ok: false, summary: `yt-dlp could not download this link: ${e instanceof Error ? e.message : String(e)} Check the link, or download the video yourself and give its file.` }; }
+      if (!existsSync(video)) { cleanup(); return { ok: false, summary: "yt-dlp finished but saved no video. The link may be a page without a video, or longer than 10 minutes. Check the link, or give a video file." }; }
+    } else {
+      const from = resolve(ctx.cwd, input);
+      if (!existsSync(from) || !statSync(from).isFile()) return { ok: false, summary: `${basename(from)} does not exist. Check the path.` };
+      source = basename(from);
+      const pre = await probeVideo(from, basename(from));
+      if (pre.durationSec > MAX_REF_SECONDS) return { ok: false, summary: tooLong(basename(from), pre.durationSec) };
+      mkdirSync(project.path(dirOf(id)), { recursive: true });
+      // mp4 is copied as it is; any other container is converted, so that refs/<id>/video.mp4 is an mp4 whatever was given.
+      if ([".mp4", ".m4v"].includes(extname(from).toLowerCase())) copyFileSync(from, video);
+      else await toMp4(from, video);
+    }
+    const info = await probeVideo(video, source);
+    if (info.durationSec > MAX_REF_SECONDS) { cleanup(); return { ok: false, summary: tooLong("This video", info.durationSec) }; }
+    const record: RefRecord = {
+      id, source, ...(urlHash ? { urlHash } : {}), ...info, aspect: nearestAspect(info.width, info.height), createdAt: new Date().toISOString(),
+    };
+    project.writeJson(fileOf(id, "ref.json"), record);
+    return {
+      ok: true, data: record,
+      summary: `Saved reference ${id}: ${fmt(info.durationSec)}s, ${info.width}x${info.height} (${record.aspect}), ${fmt(info.fps)} fps, ${info.hasAudio ? "with audio" : "no audio"}, at refs/${id}/video.mp4. Next: \`reelkit ref analyze ${id}\`.\n${REF_NOTICE}`,
+    };
+  } catch (e) {
+    cleanup();
+    // Every failure here is already a plain line written for the person who gave the video.
+    return { ok: false, summary: e instanceof Error ? e.message : String(e) };
+  }
+}
+
+// The audio of a reference as a small mp3, made once.
+async function ensureAudio(project: Project, ref: RefRecord): Promise<string> {
+  const rel = fileOf(ref.id, "audio.mp3");
+  if (!project.exists(rel)) await extractMp3(project.path(fileOf(ref.id, "video.mp4")), project.path(rel));
+  return rel;
+}
+
+export async function refAudio(ctx: Ctx, id: string): Promise<Result> {
+  const project = openProject(ctx.cwd);
+  const ref = loadRef(project, id);
+  if (!project.exists(fileOf(id, "video.mp4"))) return { ok: false, summary: `refs/${id}/video.mp4 is missing. Download the reference again with \`reelkit ref download --redo\`.` };
+  if (!ref.hasAudio) return { ok: false, summary: `Reference ${id} has no audio track, so there is no audio to extract.` };
+  const had = project.exists(fileOf(id, "audio.mp3"));
+  const rel = await ensureAudio(project, ref);
+  const bytes = statSync(project.path(rel)).size;
+  return { ok: true, data: { path: rel, bytes, skipped: had }, summary: had ? `The audio of ${id} is already at ${rel}.` : `Saved the audio of ${id} at ${rel} (mono, 16 kHz, ${Math.round(bytes / 1024)} KB).` };
+}
+
+export async function refList(ctx: Ctx): Promise<Result> {
+  const project = openProject(ctx.cwd);
+  const refs = allRefs(project).map((r) => ({ id: r.id, source: r.source, durationSec: r.durationSec, analyzed: project.exists(fileOf(r.id, "breakdown.json")) }));
+  return {
+    ok: true, data: { references: refs },
+    summary: refs.length
+      ? refs.map((r) => `${r.id}  ${fmt(r.durationSec)}s  ${r.analyzed ? "analysed" : "not analysed"}  ${r.source}`).join("\n")
+      : "No references in this project yet. Run `reelkit ref download <url-or-file>`.",
+  };
+}
+
+type TranscriptOutcome = { transcript?: NonNullable<Breakdown["transcript"]>; note: string };
+
+// The audio (never the video) goes to the server and comes back as text. Whatever goes wrong here costs the transcript only: the measurements
+// of the video do not depend on it.
+async function transcribe(ctx: Ctx, project: Project, ref: RefRecord): Promise<TranscriptOutcome> {
+  const retry = `\`reelkit ref analyze ${ref.id} --redo\``;
+  if (!loadCredentials(ctx.env)) return { note: `No transcript: you are not logged in. Run \`reelkit auth login\`, then ${retry}.` };
+  try {
+    const rel = await ensureAudio(project, ref);
+    const path = project.path(rel);
+    const bytes = readFileSync(path);
+    const api = client(ctx);
+    ctx.log("Sending the audio (not the video) to the Reelkit server to be transcribed.");
+    const durationSec = Math.min(MAX_TRANSCRIBE_SECONDS, Math.max(0.1, await audioDuration(path)));
+    const up = await api("transcribeStart", { filename: "audio.mp3", contentType: "audio/mpeg", bytes: bytes.length, durationSec });
+    await uploadTo(up.uploadUrl, new Uint8Array(bytes), "audio/mpeg");
+    const t = await api("transcribeRun", { id: up.id });
+    return {
+      transcript: { ...(t.language ? { language: t.language } : {}), text: t.text, segments: t.segments },
+      note: `The audio (not the video) was sent to the Reelkit server to be transcribed and was deleted there. It used ${Math.ceil(t.durationSec)} seconds of your monthly transcription quota (\`reelkit whoami\`).`,
+    };
+  } catch (e) {
+    const why = e instanceof ApiFailure ? e.message : `the Reelkit server could not be reached (${e instanceof Error ? e.message : String(e)}).`;
+    return { note: `No transcript: ${why.endsWith(".") ? why : `${why}.`} Retry with ${retry}.` };
+  }
+}
+
+// Runs `fn` over the items, `size` at a time, and keeps the order of the results.
+async function inBatches<T, R>(items: T[], size: number, fn: (item: T, index: number) => Promise<R>): Promise<R[]> {
+  const out: R[] = [];
+  for (let i = 0; i < items.length; i += size) out.push(...(await Promise.all(items.slice(i, i + size).map((x, j) => fn(x, i + j)))));
+  return out;
+}
+
+const round2 = (n: number) => Math.round(n * 100) / 100;
+
+async function measure(ctx: Ctx, project: Project, ref: RefRecord): Promise<Breakdown> {
+  const video = project.path(fileOf(ref.id, "video.mp4"));
+  ctx.log("Finding the cuts.");
+  const cuts = await detectCuts(video, ref.durationSec);
+  // The scenes listed are capped; the pacing below is measured on every cut.
+  const sceneCuts = cuts.slice(0, MAX_SCENES - 1);
+  const bounds = [0, ...sceneCuts, ref.durationSec];
+  rmSync(project.path(fileOf(ref.id, "frames")), { recursive: true, force: true });
+  ctx.log(`Saving sample frames of ${bounds.length - 1} scene(s).`);
+  // Each scene is a few short ffmpeg runs; running several scenes at once keeps a video with many cuts quick.
+  const scenes = await inBatches(bounds.slice(0, -1), 6, async (start, i) => {
+    const end = bounds[i + 1]!, len = end - start;
+    const n = String(i + 1).padStart(2, "0");
+    const frames = [`${dirOf(ref.id)}/frames/scene-${n}-1.jpg`, `${dirOf(ref.id)}/frames/scene-${n}-2.jpg`];
+    // A little inside the end, so that the last frame of the video can be read.
+    await extractFrame(video, start + len * 0.25, project.path(frames[0]!));
+    await extractFrame(video, Math.min(start + len * 0.75, ref.durationSec - 0.05), project.path(frames[1]!));
+    return { index: i + 1, startSec: round2(start), endSec: round2(end), durationSec: round2(len), frames, palette: await paletteOfFrames(frames.map((f) => project.path(f))) };
+  });
+  ctx.log("Measuring colours and sound.");
+  const palette = await paletteOfVideo(video, ref.durationSec);
+
+  const shots = [0, ...cuts, ref.durationSec].map((t, i, all) => (i ? t - all[i - 1]! : 0)).slice(1);
+  const pacing: Breakdown["pacing"] = {
+    cuts: cuts.length, averageShotSec: round2(ref.durationSec / shots.length), shortestShotSec: round2(Math.min(...shots)), longestShotSec: round2(Math.max(...shots)),
+    cutsPerSecond: round2(cuts.length / ref.durationSec), ...(cuts.length ? { firstCutSec: round2(cuts[0]!) } : {}),
+  };
+
+  const audio: Breakdown["audio"] = { hasAudio: ref.hasAudio, beatConfidence: 0 };
+  let beats: number[] | undefined;
+  if (ref.hasAudio) {
+    // A track that cannot be measured is a reference without a tempo, not a failed analysis.
+    const loud = await loudnessLufs(video).catch(() => undefined);
+    if (loud !== undefined) audio.loudnessLufs = loud;
+    try {
+      const rate = 11025;
+      const r = analyzeBeats(await decodeMono(video, rate), rate);
+      audio.beatConfidence = r.confidence;
+      if (r.tempoBpm !== undefined && r.beats) {
+        audio.tempoBpm = r.tempoBpm;
+        beats = r.beats.filter((t) => t <= ref.durationSec);
+        if (cuts.length) pacing.cutsOnBeat = round2(cutsOnBeat(cuts, beats));
+      }
+    } catch { /* no tempo */ }
+  }
+  return { source: ref.source, durationSec: ref.durationSec, width: ref.width, height: ref.height, aspect: ref.aspect, fps: ref.fps, scenes, ...(cuts.length > MAX_SCENES - 1 ? { scenesCapped: true } : {}), pacing, palette, audio, ...(beats ? { beats } : {}) };
+}
+
+// A short text a person or an agent can read at once.
+function digest(id: string, b: Breakdown, notes: string[]): string {
+  const lines = [
+    `Reference ${id}: ${fmt(b.durationSec)}s, ${b.width}x${b.height} (${b.aspect}), ${fmt(b.fps)} fps.`,
+    b.pacing.cuts
+      ? `Shape: ${b.scenes.length}${b.scenesCapped ? "+" : ""} scenes, ${b.pacing.cuts} cuts; average shot ${fmt(b.pacing.averageShotSec)}s (shortest ${fmt(b.pacing.shortestShotSec)}s, longest ${fmt(b.pacing.longestShotSec)}s); first cut at ${fmt(b.pacing.firstCutSec!)}s.`
+      : "Shape: one continuous shot, no cuts found.",
+    `Palette: ${b.palette.join(", ")}.`,
+    !b.audio.hasAudio ? "Audio: none."
+      : `Audio:${b.audio.loudnessLufs !== undefined ? ` ${b.audio.loudnessLufs} LUFS;` : ""} ${b.audio.tempoBpm !== undefined ? `tempo about ${Math.round(b.audio.tempoBpm)} BPM (confidence ${b.audio.beatConfidence})` : "no clear beat"}${b.pacing.cutsOnBeat !== undefined ? `; ${Math.round(b.pacing.cutsOnBeat * b.pacing.cuts)} of ${b.pacing.cuts} cuts land on a beat` : ""}.`,
+  ];
+  if (b.transcript) lines.push(`Transcript${b.transcript.language ? ` (${b.transcript.language})` : ""}: "${b.transcript.text.slice(0, 200)}${b.transcript.text.length > 200 ? "..." : ""}"`);
+  lines.push(...notes);
+  lines.push(`Look at the frames now: refs/${id}/frames/ has two per scene (scene-01-1.jpg, scene-01-2.jpg, ...). The full numbers are in refs/${id}/breakdown.json.`);
+  return lines.join("\n");
+}
+
+export async function refAnalyze(ctx: Ctx, id: string, opts: { transcript?: boolean; redo?: boolean } = {}): Promise<Result> {
+  const project = openProject(ctx.cwd);
+  const ref = loadRef(project, id);
+  if (!project.exists(fileOf(id, "video.mp4"))) return { ok: false, summary: `refs/${id}/video.mp4 is missing. Download the reference again with \`reelkit ref download --redo\`.` };
+  const file = fileOf(id, "breakdown.json");
+  if (project.exists(file) && !opts.redo) {
+    const have = project.readJson<Breakdown>(file);
+    return { ok: true, data: have, summary: digest(id, have, [`Already analysed: this is the saved breakdown. Use --redo to measure again.`]) };
+  }
+  const breakdown = await measure(ctx, project, ref);
+  const notes: string[] = [];
+  if (opts.transcript === false) notes.push(`No transcript (--no-transcript). Nothing was sent anywhere. To add it later: \`reelkit ref analyze ${id} --redo\`.`);
+  else if (!ref.hasAudio) notes.push("No transcript: the reference has no audio. Nothing was sent anywhere.");
+  else {
+    const t = await transcribe(ctx, project, ref);
+    if (t.transcript) breakdown.transcript = t.transcript;
+    notes.push(t.note);
+  }
+  project.writeJson(file, breakdown);
+  return { ok: true, data: breakdown, summary: digest(id, breakdown, notes) };
+}

package/src/contract/index.ts

@@ -67,6 +67,15 @@
 //   20 seconds is refused by the request schema.
 // - When the cutout quota is used up, `cutoutRun` is 429 `quota_exceeded` with the reset date. When the server has no GPU service configured,
 //   `cutoutStart` and `cutoutRun` are 400 `invalid_request` with the message "Background removal is not available on this server yet."
+// - A transcript of a reference video's audio is a job with two calls. `transcribeStart` checks the request and answers with an id and an
+//   `uploadUrl` (charging nothing; the client sends the audio there exactly as for a library upload); `transcribeRun` checks that the audio
+//   arrived (else 400 `invalid_request`), reserves `ceil(durationSec)` seconds from the caller's monthly transcription quota, transcribes,
+//   deletes the audio and answers with the transcript. Only the audio is sent, never the video. The audio is deleted from the server when the
+//   call ends, whatever the result. The transcript is kept with the job, so calling `transcribeRun` again for the same id returns the same
+//   transcript and does not charge again. A failed run is not charged: the seconds are given back. It is charged in whole seconds of audio
+//   (`ceil(durationSec)`), and audio longer than 600 seconds is refused by the request schema. Segment times are in seconds from the start
+//   of the audio. A transcribe id belongs to its caller: another user's id, or an unknown one, is `not_found`.
+// - When the transcription quota is used up, `transcribeRun` is 429 `quota_exceeded` with the reset date.
 // - `durationSec` is the decoded audio length; `words` are in seconds from the start of that audio.
 // - Search returns published items only, and leaves out items that are not matches at all; the default `limit` is 8.
 import { z } from "zod";
@@ -101,6 +110,12 @@ export const CutoutTypeSchema = z.enum(["video/mp4", "video/quicktime", "video/w
 export type CutoutType = z.infer<typeof CutoutTypeSchema>;
 // The longest video a cutout takes, in seconds.
 export const MAX_CUTOUT_SECONDS = 20;
+// The audio types a transcription takes, the most bytes it takes (25 MB) and the longest audio in seconds.
+export const TranscribeTypeSchema = z.enum(["audio/mpeg", "audio/mp4", "audio/wav"]);
+export type TranscribeType = z.infer<typeof TranscribeTypeSchema>;
+export const MAX_TRANSCRIBE_BYTES = 26_214_400;
+export const MAX_TRANSCRIBE_SECONDS = 600;
+export const TranscriptSegmentSchema = z.object({ text: z.string(), startSec: z.number(), endSec: z.number() });
 
 export const ErrorCodeSchema = z.enum(["unauthenticated", "quota_exceeded", "not_found", "invalid_request", "server_error"]);
 export type ErrorCode = z.infer<typeof ErrorCodeSchema>;
@@ -136,7 +151,7 @@ export type Voice = z.infer<typeof VoiceSchema>;
 const Meter = z.object({ used: z.number(), limit: z.number() });
 export const MeSchema = z.object({
   userId: z.string(), handle: z.string(),
-  quota: z.object({ voiceoverChars: Meter, images: Meter, clips: Meter, cutoutSeconds: Meter, resetsAt: z.string() }),
+  quota: z.object({ voiceoverChars: Meter, images: Meter, clips: Meter, cutoutSeconds: Meter, transcribeSeconds: Meter, resetsAt: z.string() }),
   // How many of the user's own items are published in the shared library. An item still in review, or sent back, is not counted.
   contributions: z.number(),
 });
@@ -207,6 +222,15 @@ export const routes = {
       url: z.string().optional(), ext: z.enum(["webm"]).optional(), contentType: z.literal("video/webm").optional(),
     }).refine((r) => (r.status === "done") === (r.url !== undefined && r.ext !== undefined && r.contentType !== undefined) && (r.status === "failed") === (r.message !== undefined),
       "url, ext and contentType belong to done and message to failed")),
+  transcribeStart: route("POST", "/transcripts", true,
+    z.object({
+      filename: text(200), contentType: TranscribeTypeSchema,
+      bytes: z.number().int().min(1).max(MAX_TRANSCRIBE_BYTES), durationSec: z.number().positive().max(MAX_TRANSCRIBE_SECONDS),
+    }),
+    z.object({ id: z.string(), uploadUrl: z.string() })),
+  transcribeRun: route("POST", "/transcripts/run", true,
+    z.object({ id: ItemId }),
+    z.object({ id: z.string(), text: z.string(), language: z.string().optional(), segments: z.array(TranscriptSegmentSchema), durationSec: z.number().positive() })),
   publicLibrary: route("GET", "/public/library", false,
     z.object({ q: text(200).optional(), kind: LibraryKindSchema.optional(), page: z.coerce.number().int().min(1).max(10000).optional() }),
     // With `q` the items are the best matches by meaning, best first, each with `match` (0..1), in one page (`hasMore` false). Under heavy

package/src/pipeline/schema.ts

@@ -28,6 +28,12 @@ export const ScenePlanSchema = z.object({
   // Optional only so plans stored before voices existed still load; a new plan must choose one.
   voiceId: z.string().optional().describe("id of the narration voice, one of the ids from `reelkit assets voices`, chosen to suit the idea, audience and language"),
   pace: z.enum(["slow", "normal", "fast"]).optional().describe("speaking pace: slow for calm or emotional, normal by default, fast for high-energy"),
+  // A video the new one is made "like": what is taken from it is how it feels (structure, pacing, motion), never its footage, music or words.
+  // Optional, so plans stored before references existed still load.
+  reference: z.object({
+    id: z.string().describe("the id of a reference in this project, from `reelkit ref list`"),
+    take: z.array(z.string()).min(1).max(6).describe("short notes on what is taken from it, e.g. \"fast cuts every ~1.2s\", \"big type on colour fields\""),
+  }).optional(),
   scenes: z.array(SceneSchema).min(3).max(8),
 });
 

package/src/project/beats.ts

@@ -0,0 +1,130 @@
+// Finds the tempo and the beat times of a piece of audio, on this machine, with no libraries. The audio is given as mono samples.
+//
+// The idea: loud moments (a drum hit, a click, a word) make the energy of the sound jump. The list of those jumps over time is the onset curve.
+// If the music has a pulse, the curve repeats itself after one beat, so the tempo is the delay at which the curve best matches a copy of itself.
+
+export type BeatAnalysis = {
+  // How sure the tempo is, from 0 (nothing repeats) to 1 (a perfect pulse). Real music is usually 0.3 to 0.7.
+  confidence: number;
+  // Present only when the pulse is clear enough to trust; a made-up tempo is worse than none.
+  tempoBpm?: number;
+  // Beat times in seconds from the start of the samples, on the grid of `tempoBpm`. Present with `tempoBpm`.
+  beats?: number[];
+};
+
+// Below this the autocorrelation peak is what noise produces by chance, so no tempo is reported.
+export const MIN_BEAT_CONFIDENCE = 0.2;
+
+const WINDOW = 512;
+const HOP = 256;
+const MIN_BPM = 60;
+const MAX_BPM = 200;
+// The range people actually move to; when a tempo outside it scores nearly as well, it is the half or double of one inside it.
+const PREFERRED_LOW = 80;
+const PREFERRED_HIGH = 160;
+const BPM_STEP = 0.5;
+// How many multiples of the beat length are compared: a real pulse matches at two, three and four beats too, noise does not.
+const HARMONICS = 4;
+
+// Energy per window, as a curve that jumps up where the sound gets louder. Quiet onsets count as much as loud ones (log scale), and the
+// curve is smoothed a little so a hit that falls between two windows still shows up in both.
+function onsetCurve(samples: Float32Array): Float64Array {
+  const frames = Math.max(0, Math.floor((samples.length - WINDOW) / HOP) + 1);
+  const energy = new Float64Array(frames);
+  for (let f = 0; f < frames; f++) {
+    let sum = 0;
+    const start = f * HOP;
+    for (let i = 0; i < WINDOW; i++) { const s = samples[start + i]!; sum += s * s; }
+    energy[f] = Math.log1p(100 * Math.sqrt(sum / WINDOW));
+  }
+  const rise = new Float64Array(frames);
+  for (let f = 1; f < frames; f++) rise[f] = Math.max(0, energy[f]! - energy[f - 1]!);
+  const smooth = new Float64Array(frames);
+  for (let f = 0; f < frames; f++) smooth[f] = 0.25 * (rise[f - 1] ?? 0) + 0.5 * rise[f]! + 0.25 * (rise[f + 1] ?? 0);
+  return smooth;
+}
+
+// The value of the curve at a position between two windows.
+function at(curve: Float64Array, pos: number): number {
+  if (pos < 0 || pos > curve.length - 1) return 0;
+  const i = Math.floor(pos), frac = pos - i;
+  return curve[i]! * (1 - frac) + (curve[i + 1] ?? curve[i]!) * frac;
+}
+
+// How much the curve looks like itself `lag` windows later, from 0 to about 1.
+function selfSimilarity(curve: Float64Array, lag: number): number {
+  const n = Math.floor(curve.length - lag);
+  if (n < 8) return 0;
+  let cross = 0, own = 0, shifted = 0;
+  for (let i = 0; i < n; i++) {
+    const a = curve[i]!, b = at(curve, i + lag);
+    cross += a * b; own += a * a; shifted += b * b;
+  }
+  return own > 0 && shifted > 0 ? cross / Math.sqrt(own * shifted) : 0;
+}
+
+export function analyzeBeats(samples: Float32Array, sampleRate: number): BeatAnalysis {
+  const curve = onsetCurve(samples);
+  if (curve.length < 16) return { confidence: 0 };
+  const mean = curve.reduce((s, v) => s + v, 0) / curve.length;
+  // The mean is taken out so that a steady loudness (which has no beat) does not look like a match.
+  for (let i = 0; i < curve.length; i++) curve[i] = curve[i]! - mean;
+  let energy = 0;
+  for (const v of curve) energy += v * v;
+  if (energy < 1e-9) return { confidence: 0 };
+
+  const framesPerSec = sampleRate / HOP;
+  const candidates: { bpm: number; score: number }[] = [];
+  for (let bpm = MIN_BPM; bpm <= MAX_BPM + 1e-9; bpm += BPM_STEP) {
+    const period = (60 / bpm) * framesPerSec;
+    let total = 0, used = 0;
+    for (let m = 1; m <= HARMONICS; m++) {
+      if (m * period > curve.length / 2) break;
+      total += selfSimilarity(curve, m * period);
+      used++;
+    }
+    candidates.push({ bpm, score: used ? total / used : 0 });
+  }
+  const isPeak = (i: number) => candidates[i]!.score >= (candidates[i - 1]?.score ?? -Infinity) && candidates[i]!.score >= (candidates[i + 1]?.score ?? -Infinity);
+  let best = 0;
+  for (let i = 1; i < candidates.length; i++) if (candidates[i]!.score > candidates[best]!.score) best = i;
+  const inPreferred = (i: number) => candidates[i]!.bpm >= PREFERRED_LOW && candidates[i]!.bpm <= PREFERRED_HIGH;
+  if (!inPreferred(best)) {
+    let alt = -1;
+    for (let i = 0; i < candidates.length; i++) if (inPreferred(i) && isPeak(i) && (alt < 0 || candidates[i]!.score > candidates[alt]!.score)) alt = i;
+    if (alt >= 0 && candidates[alt]!.score >= 0.8 * candidates[best]!.score) best = alt;
+  }
+
+  const confidence = Math.max(0, Math.min(1, candidates[best]!.score));
+  if (confidence < MIN_BEAT_CONFIDENCE) return { confidence: round(confidence, 2) };
+
+  // The peak lies between two of the tested tempos: a parabola through the three scores finds where it really is.
+  let bpm = candidates[best]!.bpm;
+  const l = candidates[best - 1]?.score, c = candidates[best]!.score, r = candidates[best + 1]?.score;
+  if (l !== undefined && r !== undefined && l - 2 * c + r < 0) bpm += (BPM_STEP * 0.5 * (l - r)) / (l - 2 * c + r);
+
+  // Where the beats fall: the shift of the grid that lands on the most onset.
+  const periodSec = 60 / bpm;
+  // Frame f describes the window that starts at f * HOP; a rise there means the sound arrived in the later part of that window.
+  const posOf = (t: number) => (t * sampleRate - 0.75 * WINDOW) / HOP;
+  const duration = samples.length / sampleRate;
+  let bestPhase = 0, bestHit = -Infinity;
+  for (let phase = 0; phase < periodSec; phase += HOP / 4 / sampleRate) {
+    let hit = 0;
+    for (let t = phase; t < duration; t += periodSec) hit += at(curve, posOf(t));
+    if (hit > bestHit) { bestHit = hit; bestPhase = phase; }
+  }
+  const beats: number[] = [];
+  // A beat that lands just before the first sample (a track that starts on a hit) is the first beat, at 0.
+  for (let t = bestPhase - periodSec; t < duration; t += periodSec) if (t >= -0.03) beats.push(round(Math.max(0, t), 3));
+  return { confidence: round(confidence, 2), tempoBpm: round(bpm, 1), beats };
+}
+
+// The share of cuts (0..1) that fall within `toleranceSec` of a beat.
+export function cutsOnBeat(cuts: number[], beats: number[], toleranceSec = 0.08): number {
+  if (!cuts.length || !beats.length) return 0;
+  const near = cuts.filter((c) => beats.some((b) => Math.abs(b - c) <= toleranceSec)).length;
+  return near / cuts.length;
+}
+
+const round = (n: number, digits: number) => Math.round(n * 10 ** digits) / 10 ** digits;

package/src/project/chromakey.ts

@@ -12,12 +12,19 @@ export type KeyOptions = {
   requireGreen?: boolean;
 };
 
+// Thrown when the background is green but not a saturated chroma green, or (for a clip that was asked to be on green) not green at
+// all. Video models rarely produce a true chroma green: they give a soft sage or olive that sits too close to skin and grey to be
+// keyed without taking the subject with it. The caller should cut the subject out on the server instead.
+export class GreenTooDullError extends Error {
+  constructor() { super("The green background is too dull to key cleanly: it is not a saturated chroma green. Use --cutout instead, which removes any background."); this.name = "GreenTooDullError"; }
+}
+
 const NOT_GREEN = "This video does not have a green background in its corners, so there is nothing to key out. Film the subject in front of an evenly lit green background that fills the frame.";
 
 // The colour to key on, read from the first frame's four corners. A real green screen, filmed or generated, is never exactly
 // #00FF00: it is whatever green the light and the encoder made of it, and keying on the wrong green leaves a haze. Undefined when
 // the corners are not green.
-async function cornerGreen(input: string): Promise<string | undefined> {
+async function cornerGreen(input: string): Promise<{ colour: string; saturated: boolean } | undefined> {
   const corner = (x: string, y: string) => `crop=24:24:${x}:${y},scale=1:1:flags=area`;
   const picks = ["0:0", "iw-24:0", "0:ih-24", "iw-24:ih-24"].map((c) => { const [x, y] = c.split(":") as [string, string]; return corner(x, y); });
   const colours: [number, number, number][] = [];
@@ -29,7 +36,9 @@ async function cornerGreen(input: string): Promise<string | undefined> {
   // Most corners must agree: a subject may reach one corner, but not three.
   if (green.length < 3) return undefined;
   const avg = (i: 0 | 1 | 2) => Math.round(green.reduce((sum, c) => sum + c[i], 0) / green.length);
-  return `0x${[avg(0), avg(1), avg(2)].map((n) => n.toString(16).padStart(2, "0")).join("")}`;
+  const [r, g, b] = [avg(0), avg(1), avg(2)];
+  // How far the green stands above the other two channels. Studio and pure greens are far above 60; a sage green is around 30.
+  return { colour: `0x${[r, g, b].map((n) => n.toString(16).padStart(2, "0")).join("")}`, saturated: g - Math.max(r, b) >= 60 };
 }
 
 // Turns the green of a green-screen clip into transparency: VP9 with an alpha channel, which Remotion plays with `transparent`.
@@ -39,12 +48,13 @@ export async function keyGreen(input: string, output: string, opts: KeyOptions =
   try {
     const found = await cornerGreen(input);
     if (!found && opts.requireGreen) throw new Error(NOT_GREEN);
-    const filter = `chromakey=color=${found ?? PURE_GREEN}:similarity=0.16:blend=0.08,despill=type=green:mix=0.5:expand=0,format=yuva420p`;
+    if (!found || !found.saturated) throw new GreenTooDullError();
+    const filter = `chromakey=color=${found.colour ?? PURE_GREEN}:similarity=0.16:blend=0.08,despill=type=green:mix=0.5:expand=0,format=yuva420p`;
     const audio = opts.keepAudio ? ["-c:a", "libopus", "-b:a", "96k"] : ["-an"];
     await run("ffmpeg", ["-v", "error", "-y", "-i", input, "-vf", filter, "-c:v", "libvpx-vp9", "-pix_fmt", "yuva420p", "-b:v", "0", "-crf", "30", ...audio, output]);
   } catch (e) {
     rmSync(output, { force: true });
-    if (e instanceof Error && e.message === NOT_GREEN) throw e;
+    if (e instanceof GreenTooDullError || (e instanceof Error && e.message === NOT_GREEN)) throw e;
     if ((e as NodeJS.ErrnoException)?.code === "ENOENT") throw new Error("ffmpeg was not found. Install ffmpeg (macOS: `brew install ffmpeg`) and run the command again.");
     throw new Error("ffmpeg could not key the green out of the clip. The original clip is saved; run the command again to retry.");
   }

package/src/project/refmeasure.ts

@@ -0,0 +1,148 @@
+// Measuring a reference video with ffmpeg and ffprobe, all on this machine. Every failure is a one-line Error for the person who gave the video.
+import { execFile } from "node:child_process";
+import { mkdirSync } from "node:fs";
+import { dirname } from "node:path";
+import { promisify } from "node:util";
+
+const exec = promisify(execFile);
+
+export const REF_ASPECTS = ["9:16", "16:9", "1:1", "4:5"] as const;
+export type RefAspect = (typeof REF_ASPECTS)[number];
+
+async function tool(cmd: "ffmpeg" | "ffprobe", args: string[]): Promise<{ stdout: Buffer; stderr: string }> {
+  try {
+    const r = await exec(cmd, args, { maxBuffer: 256 * 1024 * 1024, encoding: "buffer" });
+    return { stdout: r.stdout, stderr: r.stderr.toString("utf8") };
+  } catch (e) {
+    if ((e as NodeJS.ErrnoException)?.code === "ENOENT") throw new Error(`${cmd} was not found. Install ffmpeg (macOS: \`brew install ffmpeg\`) and try again.`);
+    throw e;
+  }
+}
+
+export type VideoInfo = { durationSec: number; width: number; height: number; fps: number; hasAudio: boolean };
+
+const rate = (r: string | undefined): number => {
+  const [a, b] = (r ?? "").split("/").map(Number);
+  return a && b ? a / b : a || 0;
+};
+
+export async function probeVideo(path: string, name: string): Promise<VideoInfo> {
+  const unreadable = new Error(`${name} could not be read as a video. It may be damaged or not a video: try another file or link.`);
+  let info: { streams?: { codec_type?: string; width?: number; height?: number; avg_frame_rate?: string; r_frame_rate?: string; duration?: string }[]; format?: { duration?: string } };
+  try {
+    info = JSON.parse((await tool("ffprobe", ["-v", "error", "-print_format", "json", "-show_format", "-show_streams", path])).stdout.toString("utf8"));
+  } catch (e) {
+    if (e instanceof Error && e.message.startsWith("ffprobe was not found")) throw e;
+    throw unreadable;
+  }
+  const v = info.streams?.find((s) => s.codec_type === "video");
+  const duration = Number(info.format?.duration ?? v?.duration);
+  if (!v || !(v.width! > 0) || !(v.height! > 0) || !(duration > 0)) throw unreadable;
+  const fps = rate(v.avg_frame_rate) || rate(v.r_frame_rate);
+  return { durationSec: Math.round(duration * 100) / 100, width: v.width!, height: v.height!, fps: Math.round(fps * 100) / 100, hasAudio: Boolean(info.streams?.some((s) => s.codec_type === "audio")) };
+}
+
+// The nearest of the four shapes, by how far the ratios are apart on a log scale (so 2:1 is as far from 1:1 as 1:2).
+export function nearestAspect(width: number, height: number): RefAspect {
+  const ratio = width / height;
+  const value: Record<RefAspect, number> = { "9:16": 9 / 16, "16:9": 16 / 9, "1:1": 1, "4:5": 4 / 5 };
+  return REF_ASPECTS.reduce((best, a) => (Math.abs(Math.log(ratio / value[a])) < Math.abs(Math.log(ratio / value[best])) ? a : best));
+}
+
+const MIN_SHOT_SEC = 0.25;
+
+// The times where the picture changes completely. Cuts closer than a quarter of a second to each other, to the start or to the end are one
+// flash or a fade, not a shot of their own.
+export async function detectCuts(video: string, durationSec: number, threshold = 0.3): Promise<number[]> {
+  const { stderr } = await tool("ffmpeg", ["-hide_banner", "-nostats", "-i", video, "-an", "-vf", `select='gt(scene,${threshold})',showinfo`, "-f", "null", "-"]);
+  const times = [...stderr.matchAll(/pts_time:\s*([0-9.]+)/g)].map((m) => Number(m[1])).sort((a, b) => a - b);
+  const cuts: number[] = [];
+  for (const t of times) {
+    if (t < MIN_SHOT_SEC || durationSec - t < MIN_SHOT_SEC) continue;
+    if (cuts.length && t - cuts[cuts.length - 1]! < MIN_SHOT_SEC) continue;
+    cuts.push(Math.round(t * 1000) / 1000);
+  }
+  return cuts;
+}
+
+// One JPEG of the video at `atSec`, at most 540 pixels on the long side.
+export async function extractFrame(video: string, atSec: number, dest: string): Promise<void> {
+  mkdirSync(dirname(dest), { recursive: true });
+  await tool("ffmpeg", ["-v", "error", "-y", "-ss", atSec.toFixed(3), "-i", video, "-frames:v", "1",
+    "-vf", "scale=w='if(gt(iw,ih),min(540,iw),-2)':h='if(gt(iw,ih),-2,min(540,ih))'", "-q:v", "4", dest]);
+}
+
+type Pixel = [number, number, number];
+
+// Averages of the main colours: each pixel goes in one of 64 buckets (four levels per channel), and the fullest buckets, skipping one whose
+// colour is close to a colour already taken, are named by the average of their pixels.
+export function dominantColors(pixels: Pixel[], max: number): string[] {
+  const buckets = new Map<number, { n: number; sum: Pixel }>();
+  for (const [r, g, b] of pixels) {
+    const key = (r >> 6) * 16 + (g >> 6) * 4 + (b >> 6);
+    const e = buckets.get(key) ?? { n: 0, sum: [0, 0, 0] as Pixel };
+    e.n++; e.sum[0] += r; e.sum[1] += g; e.sum[2] += b;
+    buckets.set(key, e);
+  }
+  const chosen: Pixel[] = [];
+  for (const e of [...buckets.values()].sort((a, b) => b.n - a.n)) {
+    const c = e.sum.map((v) => Math.round(v / e.n)) as Pixel;
+    if (chosen.some((o) => Math.hypot(o[0] - c[0], o[1] - c[1], o[2] - c[2]) < 48)) continue;
+    chosen.push(c);
+    if (chosen.length === max) break;
+  }
+  return chosen.map(([r, g, b]) => `#${[r, g, b].map((v) => v.toString(16).padStart(2, "0")).join("")}`);
+}
+
+// Every picture is shrunk to 4 by 4 pixels, which averages away detail and leaves the colour of each part of the frame.
+async function tinyPixels(inputArgs: string[], filter: string): Promise<Pixel[]> {
+  const { stdout } = await tool("ffmpeg", ["-v", "error", ...inputArgs, "-an", "-vf", `${filter}scale=4:4:flags=area`, "-f", "rawvideo", "-pix_fmt", "rgb24", "-"]);
+  const out: Pixel[] = [];
+  for (let i = 0; i + 2 < stdout.length; i += 3) out.push([stdout[i]!, stdout[i + 1]!, stdout[i + 2]!]);
+  return out;
+}
+
+export async function paletteOfVideo(video: string, durationSec: number, max = 5): Promise<string[]> {
+  // At most about 120 pictures however long the video is.
+  const fps = Math.max(0.15, Math.min(2, 120 / durationSec));
+  return dominantColors(await tinyPixels(["-i", video], `fps=${fps.toFixed(3)},`), max);
+}
+
+export async function paletteOfFrames(frames: string[], max = 3): Promise<string[]> {
+  const pixels: Pixel[] = [];
+  for (const f of frames) pixels.push(...(await tinyPixels(["-i", f], "")));
+  return dominantColors(pixels, max);
+}
+
+// The integrated loudness in LUFS, or undefined when there is none (silence is minus infinity).
+export async function loudnessLufs(video: string): Promise<number | undefined> {
+  const { stderr } = await tool("ffmpeg", ["-hide_banner", "-nostats", "-i", video, "-vn", "-af", "ebur128=framelog=quiet", "-f", "null", "-"]);
+  const all = [...stderr.matchAll(/\bI:\s+(-?[0-9.]+)\s+LUFS/g)];
+  const value = Number(all.at(-1)?.[1]);
+  return Number.isFinite(value) ? Math.round(value * 10) / 10 : undefined;
+}
+
+// The audio as mono samples between -1 and 1, for the beat analysis.
+export async function decodeMono(video: string, sampleRate: number): Promise<Float32Array> {
+  const { stdout } = await tool("ffmpeg", ["-v", "error", "-i", video, "-vn", "-ac", "1", "-ar", String(sampleRate), "-f", "s16le", "-"]);
+  const n = Math.floor(stdout.length / 2);
+  const out = new Float32Array(n);
+  for (let i = 0; i < n; i++) out[i] = stdout.readInt16LE(i * 2) / 32768;
+  return out;
+}
+
+// Speech needs little: mono, 16 kHz and 64 kbit/s keep a ten-minute reference under 5 MB.
+export async function extractMp3(video: string, dest: string): Promise<void> {
+  mkdirSync(dirname(dest), { recursive: true });
+  await tool("ffmpeg", ["-v", "error", "-y", "-i", video, "-vn", "-ac", "1", "-ar", "16000", "-b:a", "64k", "-c:a", "libmp3lame", dest]);
+}
+
+export async function toMp4(source: string, dest: string): Promise<void> {
+  mkdirSync(dirname(dest), { recursive: true });
+  await tool("ffmpeg", ["-v", "error", "-y", "-i", source, "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-movflags", "+faststart", dest]);
+}
+
+export async function audioDuration(path: string): Promise<number> {
+  const { stdout } = await tool("ffprobe", ["-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path]);
+  return Number(stdout.toString("utf8").trim());
+}

package/src/testing/conformance.ts

@@ -23,8 +23,11 @@ export type Harness = {
   // Optional, for the cutout tests. A file name that makes a cutout job fail (the fake fails any name containing "cutout-fail"); without
   // it the failing-job case is skipped. The cutout tests wait between status asks for `clipPollMs` too.
   cutoutFailFilename?: string;
+  // Optional, for the transcript tests. A file name that makes a transcription fail (the fake fails any name containing "transcribe-fail");
+  // without it the failing-job case is skipped.
+  transcribeFailFilename?: string;
 };
-export type StartOptions = { voiceoverChars?: number; images?: number; clips?: number; cutoutSeconds?: number };
+export type StartOptions = { voiceoverChars?: number; images?: number; clips?: number; cutoutSeconds?: number; transcribeSeconds?: number };
 
 const PNG = new Uint8Array(Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64"));
 const DATE = /\d{4}-\d{2}-\d{2}/;
@@ -87,6 +90,8 @@ export function runConformance(name: string, start: (opts: StartOptions) => Prom
         expect(me.quota.clips.limit).toBeGreaterThan(0);
         expect(me.quota.cutoutSeconds.used).toBe(0);
         expect(me.quota.cutoutSeconds.limit).toBeGreaterThan(0);
+        expect(me.quota.transcribeSeconds.used).toBe(0);
+        expect(me.quota.transcribeSeconds.limit).toBeGreaterThan(0);
         expect(me.quota.voiceoverChars.limit).toBeGreaterThan(0);
         expect(me.contributions).toBe(0);
         const resets = new Date(me.quota.resetsAt);
@@ -522,6 +527,107 @@ export function runConformance(name: string, start: (opts: StartOptions) => Prom
       });
     });
 
+    describe("transcripts", () => {
+      const AUDIO = new Uint8Array([73, 68, 51, 4, ...new Array(2000).fill(9)]);
+      const meta = (over: Record<string, unknown> = {}) => ({ filename: "talk.mp3", contentType: "audio/mpeg" as const, bytes: AUDIO.length, durationSec: 12.4, ...over });
+      const sent = async (api: Api, over: Record<string, unknown> = {}) => {
+        const started = await api("transcribeStart", meta(over) as never);
+        await put(started.uploadUrl, AUDIO, (over.contentType as string | undefined) ?? "audio/mpeg");
+        return started;
+      };
+
+      it("start, upload, run: text and segments with increasing times inside the audio; charged in whole seconds to this user only", async () => {
+        const api = await as("user-trn0000001"), other = await as("user-trn0000002");
+        const started = await api("transcribeStart", meta());
+        expect(started.id).toBeTruthy();
+        expect(started.uploadUrl).toBeTruthy();
+        expect((await api("me", {})).quota.transcribeSeconds.used).toBe(0);
+        expect((await put(started.uploadUrl, AUDIO, "audio/mpeg")).ok).toBe(true);
+        const t = await api("transcribeRun", { id: started.id });
+        expect(t.id).toBe(started.id);
+        expect(t.text.length).toBeGreaterThan(0);
+        expect(t.durationSec).toBeGreaterThan(0);
+        expect(t.segments.length).toBeGreaterThan(0);
+        let last = 0;
+        for (const s of t.segments) {
+          expect(s.text.length).toBeGreaterThan(0);
+          expect(s.startSec).toBeGreaterThanOrEqual(last);
+          expect(s.endSec).toBeGreaterThanOrEqual(s.startSec);
+          expect(s.endSec).toBeLessThanOrEqual(t.durationSec + 0.5);
+          last = s.endSec;
+        }
+        expect((await api("me", {})).quota.transcribeSeconds.used).toBe(13);
+        expect((await other("me", {})).quota.transcribeSeconds.used).toBe(0);
+      });
+
+      it("a run before the audio arrived is invalid_request and charges nothing", async () => {
+        const api = await as("user-trn0000003");
+        const started = await api("transcribeStart", meta());
+        expect((await failure(api("transcribeRun", { id: started.id }))).code).toBe("invalid_request");
+        expect((await api("me", {})).quota.transcribeSeconds.used).toBe(0);
+        await put(started.uploadUrl, AUDIO, "audio/mpeg");
+        expect((await api("transcribeRun", { id: started.id })).text.length).toBeGreaterThan(0);
+      });
+
+      it("running twice returns the same transcript and charges once", async () => {
+        const api = await as("user-trn0000004");
+        const started = await sent(api);
+        const first = await api("transcribeRun", { id: started.id });
+        const second = await api("transcribeRun", { id: started.id });
+        expect(second).toEqual(first);
+        expect((await api("me", {})).quota.transcribeSeconds.used).toBe(13);
+      });
+
+      it("a transcript id belongs to its caller: another user, or an unknown id, is not_found", async () => {
+        const api = await as("user-trn0000005"), other = await as("user-trn0000006");
+        const started = await sent(api);
+        await api("transcribeRun", { id: started.id });
+        expect((await failure(other("transcribeRun", { id: started.id }))).code).toBe("not_found");
+        expect((await failure(api("transcribeRun", { id: "tr-nosuchjob" }))).code).toBe("not_found");
+        expect((await other("me", {})).quota.transcribeSeconds.used).toBe(0);
+      });
+
+      it("a failing job is a server error and is not charged", async () => {
+        if (!h.transcribeFailFilename) return;
+        const api = await as("user-trn0000007");
+        const started = await sent(api, { filename: h.transcribeFailFilename });
+        expect((await failure(api("transcribeRun", { id: started.id }))).code).toBe("server_error");
+        expect((await api("me", {})).quota.transcribeSeconds.used).toBe(0);
+      });
+
+      it("at the limit a run is quota_exceeded with the reset date, charged nothing; other users are unaffected", async () => {
+        const t = await start({ transcribeSeconds: 5 });
+        try {
+          const api = createClient({ baseUrl: t.baseUrl, token: await t.login("user-trnq000001") });
+          const other = createClient({ baseUrl: t.baseUrl, token: await t.login("user-trnq000002") });
+          const first = await sent(api, { durationSec: 3 });
+          await api("transcribeRun", { id: first.id });
+          const second = await sent(api, { durationSec: 3 });
+          const e = await failure(api("transcribeRun", { id: second.id }));
+          expect(e.code).toBe("quota_exceeded");
+          expect(e.message).toMatch(DATE);
+          const me = await api("me", {});
+          expect(me.quota.transcribeSeconds).toEqual({ used: 3, limit: 5 });
+          expect(e.message).toContain(me.quota.resetsAt.slice(0, 10));
+          const third = await sent(other, { durationSec: 3 });
+          expect((await other("transcribeRun", { id: third.id })).text.length).toBeGreaterThan(0);
+        } finally { await t.close(); }
+      });
+
+      it("refuses audio over 600 seconds, a type that is not audio, a size out of range and a missing login", async () => {
+        const api = await as("user-trn0000008");
+        for (const bad of [{ durationSec: 600.5 }, { durationSec: 0 }, { durationSec: -1 }, { contentType: "video/mp4" }, { bytes: 0 }, { bytes: 26_214_401 }, { filename: "a\u0000b.mp3" }]) {
+          expect((await failure(api("transcribeStart", meta(bad) as never))).code, JSON.stringify(bad)).toBe("invalid_request");
+        }
+        for (const ok of [{ durationSec: 600 }, { contentType: "audio/mp4" as const }, { contentType: "audio/wav" as const }, { bytes: 26_214_400 }]) {
+          expect((await api("transcribeStart", meta(ok))).id).toBeTruthy();
+        }
+        expect((await failure(api("transcribeRun", { id: "../x" }))).code).toBe("invalid_request");
+        expect((await failure(anon("transcribeStart", meta()))).code).toBe("unauthenticated");
+        expect((await failure(anon("transcribeRun", { id: "tr-x" }))).code).toBe("unauthenticated");
+      });
+    });
+
     describe("quota", () => {
       it("passes exactly at the limit; one over is quota_exceeded with the reset date; other users are unaffected", async () => {
         const t = await start({ voiceoverChars: 10, images: 1 });

package/src/testing/fake-api.ts

@@ -81,10 +81,14 @@ function makeCutout(): Blob {
   } finally { rmSync(dir, { recursive: true, force: true }); }
 }
 
+// A transcript job: the audio arrives at the upload URL; `run` reserves the seconds, deletes the audio and keeps the answer, so a second run is free.
+type TranscriptJob = { owner: string; filename: string; durationSec: number; file?: Uint8Array; result?: { id: string; text: string; language: string; segments: { text: string; startSec: number; endSec: number }[]; durationSec: number } };
+const TRANSCRIBE_FAIL = "transcribe-fail";
+
 export type FakeApi = Awaited<ReturnType<typeof startFakeApi>>;
 
 // An in-memory stand-in for the Reelkit API. It implements every route in the contract.
-export async function startFakeApi(opts: { voiceoverCharLimit?: number; imageLimit?: number; clipLimit?: number; noClipProvider?: boolean; cutoutSecondsLimit?: number; noCutoutService?: boolean; autoApprove?: boolean } = {}) {
+export async function startFakeApi(opts: { voiceoverCharLimit?: number; imageLimit?: number; clipLimit?: number; noClipProvider?: boolean; cutoutSecondsLimit?: number; noCutoutService?: boolean; transcribeSecondsLimit?: number; autoApprove?: boolean } = {}) {
   // token -> the user it belongs to
   const tokens = new Map<string, string>();
   const devices = new Map<string, { userCode: string; approved: boolean; userId: string }>();
@@ -96,15 +100,16 @@ export async function startFakeApi(opts: { voiceoverCharLimit?: number; imageLim
   // `sink` is where the accepted bytes go: a library item, or a cutout job.
   const uploads = new Map<string, { contentType: string; bytes: number; used: boolean; sink: { has(): boolean; put(bytes: Uint8Array): void } | undefined }>();
   // Totals over every user (for tests of the CLI), and the same counts per user (what a quota is measured against).
-  const usage = { chars: 0, images: 0, clips: 0, cutoutSeconds: 0, pulls: 0, uploads: 0 };
-  const meters = new Map<string, { chars: number; images: number; clips: number; cutoutSeconds: number; uploads: number }>();
+  const usage = { chars: 0, images: 0, clips: 0, cutoutSeconds: 0, transcribeSeconds: 0, pulls: 0, uploads: 0 };
+  const meters = new Map<string, { chars: number; images: number; clips: number; cutoutSeconds: number; transcribeSeconds: number; uploads: number }>();
   const meter = (userId: string | undefined) => {
     const id = userId ?? "u-test";
-    if (!meters.has(id)) meters.set(id, { chars: 0, images: 0, clips: 0, cutoutSeconds: 0, uploads: 0 });
+    if (!meters.has(id)) meters.set(id, { chars: 0, images: 0, clips: 0, cutoutSeconds: 0, transcribeSeconds: 0, uploads: 0 });
     return meters.get(id)!;
   };
   const resetDate = () => nextMonthStart().toISOString();
-  const limits = { chars: opts.voiceoverCharLimit ?? 10_000, images: opts.imageLimit ?? 30, clips: opts.clipLimit ?? 5, cutoutSeconds: opts.cutoutSecondsLimit ?? 120 };
+  const limits = { chars: opts.voiceoverCharLimit ?? 10_000, images: opts.imageLimit ?? 30, clips: opts.clipLimit ?? 5, cutoutSeconds: opts.cutoutSecondsLimit ?? 120, transcribeSeconds: opts.transcribeSecondsLimit ?? 1800 };
+  const transcripts = new Map<string, TranscriptJob>();
   const jobs = new Map<string, ClipJob>();
   const cutouts = new Map<string, CutoutJob>();
   let cutoutFile: Blob | undefined;
@@ -141,7 +146,7 @@ export async function startFakeApi(opts: { voiceoverCharLimit?: number; imageLim
       const m = meter(userId);
       return {
         userId: userId ?? "u-test", handle: handleOf(userId ?? "u-test"),
-        quota: { voiceoverChars: { used: m.chars, limit: limits.chars }, images: { used: m.images, limit: limits.images }, clips: { used: m.clips, limit: limits.clips }, cutoutSeconds: { used: m.cutoutSeconds, limit: limits.cutoutSeconds }, resetsAt: resetDate() },
+        quota: { voiceoverChars: { used: m.chars, limit: limits.chars }, images: { used: m.images, limit: limits.images }, clips: { used: m.clips, limit: limits.clips }, cutoutSeconds: { used: m.cutoutSeconds, limit: limits.cutoutSeconds }, transcribeSeconds: { used: m.transcribeSeconds, limit: limits.transcribeSeconds }, resetsAt: resetDate() },
         // What the user gave the shared library that was accepted: their own items that are published. One waiting for review does not count.
         contributions: [...store.values()].filter((x) => x.owner === userId && x.committed && x.item.visibility === "published").length,
       };
@@ -281,6 +286,37 @@ export async function startFakeApi(opts: { voiceoverCharLimit?: number; imageLim
       }
       return { id, status: "done", url: fileUrl((cutoutFile ??= makeCutout())), ext: "webm", contentType: "video/webm" };
     },
+    transcribeStart: (input, { userId }) => {
+      const id = nextId("tr");
+      const job: TranscriptJob = { owner: userId ?? "u-test", filename: input.filename, durationSec: input.durationSec };
+      transcripts.set(id, job);
+      const token = secret();
+      uploads.set(token, { contentType: input.contentType, bytes: input.bytes, used: false, sink: { has: () => Boolean(job.file), put: (bytes) => { job.file = bytes; } } });
+      return { id, uploadUrl: `${origin}/upload/${token}` };
+    },
+    transcribeRun: ({ id }, { userId }) => {
+      const job = transcripts.get(id);
+      if (!job || job.owner !== (userId ?? "u-test")) throw new Fail(404, "not_found", `No transcript with id ${id}.`);
+      // Asked again, it answers the same transcript without a second charge.
+      if (job.result) return job.result;
+      if (!job.file) throw new Fail(400, "invalid_request", `Nothing was uploaded for ${id}. Send the audio to the upload URL first.`);
+      const seconds = Math.ceil(job.durationSec);
+      const m = meter(userId);
+      if (m.transcribeSeconds + seconds > limits.transcribeSeconds) throw new Fail(429, "quota_exceeded", `Transcription quota used up. It resets on ${resetDate().slice(0, 10)}.`);
+      // The audio is deleted when the call ends, whatever the result; a failure never keeps the seconds.
+      job.file = undefined;
+      if (job.filename.includes(TRANSCRIBE_FAIL)) throw new Fail(500, "server_error", "The audio could not be transcribed.");
+      m.transcribeSeconds += seconds;
+      usage.transcribeSeconds += seconds;
+      const base = job.filename.replace(/\.[^.]*$/, "").replace(/[^A-Za-z0-9]+/g, " ").trim() || "audio";
+      const half = Math.round((job.durationSec / 2) * 100) / 100;
+      job.result = {
+        id, language: "en", durationSec: job.durationSec,
+        text: `This is ${base}. It ends here.`,
+        segments: [{ text: `This is ${base}.`, startSec: 0, endSec: half }, { text: "It ends here.", startSec: half, endSec: job.durationSec }],
+      };
+      return job.result;
+    },
     publicLibrary: ({ q, kind, page }) => {
       // Published items only. Newest first; with a query, best match first and equal matches newest first.
       const want = q ? words(q) : undefined;
@@ -348,6 +384,8 @@ export async function startFakeApi(opts: { voiceoverCharLimit?: number; imageLim
     setClipLimit(n: number) { limits.clips = n; },
     // The same for the seconds of video a user may have cut out.
     setCutoutLimit(n: number) { limits.cutoutSeconds = n; },
+    // The same for the seconds of audio a user may have transcribed.
+    setTranscribeLimit(n: number) { limits.transcribeSeconds = n; },
     // Stops cutout jobs from finishing (true) or lets them finish again (false).
     holdCutouts(hold: boolean) { cutoutsHeld = hold; },
     approve(userCode: string, userId = "u-test") { for (const d of devices.values()) if (d.userCode === userCode) { d.approved = true; d.userId = userId; } },