@mevdragon/vidfarm-devcli 0.10.0 → 0.12.0

package/README.md

@@ -50,7 +50,7 @@ Everything runs locally (`RECORDS_DRIVER=local`, `STORAGE_DRIVER=local`), includ
 
 Flags: `--port` (default 3000), `--dir` (default `./.vidfarm-local`), `--key` (bootstrap/browser key, or `VIDFARM_API_KEY`), `--fork <id>`, `--no-open`. `vidfarm <template_id>` is an alias for `serve <template_id>`.
 
-Beyond the local editor, the devcli also wraps the **entire director REST flow** as 1:1 commands — `discover`, `inspiration-add`, `fork`, `decompose`, `snapshot`, `render`, `approve`, `schedule`, `posts`, `upload`/`download`, `clips` (see below), plus a raw `vidfarm api <METHOD> <path>` passthrough. Each maps to exactly one REST route and prints the prod frontend URL to open. Run `vidfarm --help` for the full surface, or see `SKILL.director.md` § "`vidfarm-devcli` — full command surface".
+Beyond the local editor, the devcli also wraps the **entire director REST flow** as 1:1 commands — `discover`, `inspiration-add`, `fork`, `decompose`, `snapshot`, `render`, `approve`, `schedule`, `posts`, `upload`/`download`, `clips` (see below), `tts`/`stt` (see below), plus a raw `vidfarm api <METHOD> <path>` passthrough. Each maps to exactly one REST route and prints the prod frontend URL to open. Run `vidfarm --help` for the full surface, or see `SKILL.director.md` § "`vidfarm-devcli` — full command surface".
 
 ## Clip hunting: `vidfarm clips`
 
@@ -70,6 +70,24 @@ vidfarm clips scan --cloud --url "https://youtube.com/watch?v=…" --duration 30
 
 Duration is a **soft range** (10→5–20s, 30→20–40s), `--range` means only those windows are decoded (the cost saver on long videos), and `--no-text` selects text-free scenes — caption *removal* (GhostCut) stays a per-finished-clip primitive, never a long-form pass. Run `vidfarm clips --help` for everything (`list`, `match`, `preset`, `sources`, `export`).
 
+## Speech: `vidfarm tts` / `vidfarm stt`
+
+Text-to-speech and speech-to-text, **local-first** like `clips`: both default to your own `OPENAI_API_KEY` / `GEMINI_API_KEY` / `OPENROUTER_API_KEY` (no cloud job, no wallet), with `--cloud` as the explicit backup that runs the platform primitive on your saved BYOK keys instead. (Speech can't ride the local `claude`/`codex` agent — no audio I/O — so the local path always needs a provider key.)
+
+```bash
+# text → narration audio; the VOICE STYLE is promptable
+vidfarm tts "Welcome back. Today we test the three most viral hooks." \
+  --style "energetic UGC creator, conversational" --voice coral --out narration.mp3
+
+# video or audio → transcript, BOTH formats in one pass:
+#   simple subtitles (transcript.txt + timed transcript.srt)
+#   + multi-speaker segments (transcript.json: {speaker, start_sec, end_sec, text})
+vidfarm stt ./podcast.mp4 --out transcript          # local ffmpeg demux + your key
+vidfarm stt "https://cdn.example.com/clip.mp4" --cloud   # backup: POST /api/v1/primitives/audio/transcribe
+```
+
+Speaker labels (diarization, on by default) need a Gemini key — OpenAI/OpenRouter keys degrade to a plain single-speaker transcript. The matching REST primitives are `POST /api/v1/primitives/audio/speech` and `/audio/transcribe` (aliases `/tts`, `/stt`).
+
 ### Crash telemetry & opt-out
 
 The devcli can report **unexpected crashes** (bugs — not bad args, HTTP 4xx, or network blips) to Sentry so they can be fixed. It is **off unless a dedicated CLI Sentry DSN is configured** (`VIDFARM_DEVCLI_SENTRY_DSN`), reports crash-only, and **scrubs home-dir paths, API keys, and secrets** before anything leaves your machine (PII off). To opt out entirely:

package/SKILL.director.md

@@ -125,9 +125,9 @@ The editor loads `composition.html` and `composition.json` from the fork's worki
 The editor is a full timeline surface:
 
 - **Timeline** — multi-track, drag/trim/split clips, group/ungroup layers
-- **Inspector panel** — per-layer property editor (text, color, font, position, size, media source, playback start, volume). Image layers also expose an **Animate (Ken Burns)** dropdown — a slow pan/zoom over the clip's full duration (see below).
+- **Inspector panel** — per-layer property editor (text, color, font, position, size, media source, playback start, volume). Image layers also expose an **Animate (Ken Burns)** dropdown — a slow pan/zoom over the clip's full duration (see below). Text/caption layers expose an **Animated captions** preset picker — word-by-word TikTok/CapCut caption styles (see below) — plus active-word color, highlight color, and uppercase controls.
 - **Canvas frame overlay** — 8-handle resize + WYSIWYG positioning
-- **Chat panel** — AI assistant that can programmatically edit the composition (`add_layer`, `remove_layer`, `set_layer_timing/visual/style/media/text`, `group_layers`, `generate_layer`, `replace_composition_html`, ...). The assistant uses the caller's saved provider keys. **`generate_layer`** is the one-shot "generate AI media and place it": it submits the `/videos/generate` or `/images/generate` primitive job, drops a placeholder clip into the target slot (fill a blank gap, replace a scene, or overlay), and auto-swaps in the finished media when the job settles — no manual polling. In-flight generations surface in `editor_context.pending_generations`.
+- **Chat panel** — AI assistant that can programmatically edit the composition (`add_layer`, `remove_layer`, `set_layer_timing/visual/style/media/text`, `set_captions`, `group_layers`, `generate_layer`, `replace_composition_html`, ...). The assistant uses the caller's saved provider keys. **`generate_layer`** is the one-shot "generate AI media and place it": it submits the `/videos/generate` or `/images/generate` primitive job, drops a placeholder clip into the target slot (fill a blank gap, replace a scene, or overlay), and auto-swaps in the finished media when the job settles — no manual polling. In-flight generations surface in `editor_context.pending_generations`.
 - **Auto-save** — every commit writes composition.html + composition.json to the working state. No explicit save button.
 
 The editor persists to:
@@ -303,6 +303,19 @@ Three surfaces:
 - **Editor UI (human)** — the Inspector's **Animate (Ken Burns)** dropdown on any selected image.
 - **Desktop agents (devcli)** — `vidfarm place ./work --src <img> --kind image --ken-burns zoom-in [--ken-burns-intensity 0.3]`, or fused with generation: `vidfarm generate image --prompt "…" --place ./work --ken-burns pan-left`.
 
+### Animated captions — word-by-word caption styles (TikTok/CapCut)
+
+Compositions support **first-class animated captions**: caption layers whose words animate one at a time, synced to the spoken audio, identical in the editor preview and both render paths. A caption *run* is one layer per cue (a page of ~3-5 words) on a shared track; each layer carries `data-caption-animation="<preset>"` and per-word `<span data-cap-word>` timing — no keyframe authoring, and the text stays editable like any text layer.
+
+**Preset styles** (each = an animation + a look; every field individually overridable): `spotlight` (active word gets a rounded highlight pill on bold outlined text — the classic Hormozi/CapCut look), `karaoke` (words fill with color as spoken and stay lit), `word-pop` (one word at a time, punchy scale-in), `stack-up` (words fade/rise in and accumulate), `neon` (active word glow pulse), `bounce` (active word bounces). Overridables: `caption_active_color` (spoken-word color), `caption_highlight_color` (spotlight pill), `caption_uppercase`, plus the normal text fields (`color`, `background`, `background_style`, `font_family`, `font_size`).
+
+**Word timings**: real word-level timestamps come from OpenAI STT (whisper-1); Gemini/OpenRouter transcripts get char-weighted estimated word windows, which still read well. Hand-typed text is auto-paged and estimated.
+
+Three surfaces:
+- **Editor web copilot** — `editor_action` with `action_type=set_captions`: pass `captions=[{text,start,duration,words?}]` cues derived from the transcript (`video_context` gives timestamped segments), or just `text` (+ `start`/`duration` window) to auto-page. Pick `caption_style`; the call **replaces** all existing animated caption layers, so it's also the "restyle everything" verb. Per-layer: `set_layer_style` with `caption_*` fields restyles one cue, `set_layer_text` rewrites its words, `caption_animation="none"` flattens back to static text. Never hand-build word-by-word captions from many `add_layer` calls.
+- **Editor UI (human)** — right-click → **Insert Captions** drops a starter cue; the Inspector's **Animated captions** picker switches styles on any selected text/caption layer, with active-word/highlight color and uppercase controls next to it.
+- **Desktop agents (devcli)** — `vidfarm captions generate ./work [--style spotlight]` transcribes the composition's own narration LOCAL-FIRST on your key (prefers OpenAI for real word timestamps) and writes the cue layers into `composition.html` on disk (a running `serve` live-morphs it). Alternates: `--audio <file|url>`, `--srt <file>` (e.g. from `vidfarm stt`), or `--text "<script>"`. Then `vidfarm captions style ./work --style karaoke` (restyle, same text/timing), `captions list`, `captions clear`.
+
 ### Local file paths as media (skip the S3 upload)
 
 `vidfarm place --src` (and `approve --video/--media`) accept a **local file path**, not just a URL — so a power user bulk-building compositions from a folder of clips on their own machine never has to upload every asset to Vidfarm storage:
@@ -502,7 +515,11 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
 | `vidfarm fork <template_id>` | `POST /api/v1/compositions` | fork a template |
 | `vidfarm pull <forkId> [--dir <p>]` | `GET .../compositions/:forkId/{composition.html,json,video-context.json,cast.json}` | sync a fork to disk + print gaps/scene keys |
 | `vidfarm generate <image\|video> --prompt "…"` | `POST /api/v1/primitives/{images,videos}/generate` (polls job) | generate AI media → finished URL |
+| `vidfarm tts "…" [--style "…"] [--voice <v>] [--out <file>]` | (LOCAL-FIRST: your own OPENAI/GEMINI/OPENROUTER_API_KEY → audio file on disk; `--cloud` = `POST /api/v1/primitives/audio/speech` + poll) | text → narration audio with a promptable voice style |
+| `vidfarm stt <file\|url> [--out <base>] [--no-diarize]` (alias: `transcribe`) | (LOCAL-FIRST: local ffmpeg demux + your own key; `--cloud` = `POST /api/v1/primitives/audio/transcribe` + poll) | video/audio → transcript in BOTH formats: simple subtitles (txt + SRT) and multi-speaker segments (json) |
 | `vidfarm place <dir> --src <url\|file> [--at\|--replace]` | (edits local composition.html; local files → serve disk store or temp upload) | drop media (URL **or local file**) into a gap / over a scene |
+| `vidfarm captions generate <dir> [--style <preset>] [--audio <f>\|--srt <f>\|--text "…"]` | (LOCAL-FIRST: STT on your own key — OpenAI = real word timestamps — then edits local composition.html) | transcribe narration → animated word-by-word caption cues |
+| `vidfarm captions style <dir> --style <preset>` / `captions list` / `captions clear` | (edits local composition.html) | restyle / inspect / remove animated captions |
 | `vidfarm decompose <forkId>` | `POST .../compositions/:forkId/auto-decompose` | split source into scenes |
 | `vidfarm remove-video-captions <forkId>` (alias: `ghostcut`) | `GET .../compositions/:forkId/remove-video-captions` | subtitle-removal status |
 | `vidfarm snapshot <forkId>` | `POST .../compositions/:forkId/versions` | save composition fork version |
@@ -730,6 +747,70 @@ curl -X POST "$VIDFARM_BASE/api/v1/primitives/media/dedupe" \
   }'
 ```
 
+## Primitive: tts (text → speech)
+
+Generate spoken narration audio from text on the caller's saved AI provider keys (OpenAI, Gemini, or OpenRouter — BYOK, never wallet-billed). The **voice style is promptable**: `instructions` is a free-form direction for tone, pacing, accent, emotion, and persona, on top of the provider's `voice` preset. Result is a durable Vidfarm audio URL (mp3/wav) ready to drop into a composition as an audio layer.
+
+- `POST /api/v1/primitives/audio/speech` (alias: `POST /api/v1/primitives/tts`)
+- Body: `{ "tracer": "...", "payload": { "text": "...", "voice"?: "...", "instructions"?: "...", "provider"?: "openai" | "gemini" | "openrouter", "model"?: "...", "output_format"?: "mp3" | "wav" }, "webhook_url"?: "..." }`
+  - `text` (required, ≤8000 chars) — the words to speak, verbatim. Aliases: `input`, `script`.
+  - `voice` (optional) — provider voice preset (OpenAI: `alloy`, `ash`, `coral`, …; Gemini: `Kore`, `Puck`, `Charon`, …).
+  - `instructions` (optional) — the voice-STYLE prompt, e.g. `"calm, warm bedtime narrator with a slight British accent"` or `"excited sports announcer, fast paced"`. Aliases: `style`, `style_instructions`, `voice_style`.
+  - `output_format` defaults to `mp3` (note: Gemini answers in wav regardless — read `audio.content_type`).
+- Response: standard primitive job. Poll to completion, then read `primary_file_url` / `audio.file_url`.
+- Billing: BYOK — the speech call runs on the customer's own provider key and is never wallet-billed.
+- devcli: `vidfarm tts "…" --style "…"` runs LOCAL-FIRST on your own env key (no job, no cloud); add `--cloud` to run this route instead.
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/audio/speech" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "demo-tts",
+    "payload": {
+      "text": "Welcome back. Today we are testing the three most viral hooks.",
+      "voice": "coral",
+      "instructions": "energetic UGC creator, conversational, slightly breathless"
+    }
+  }'
+```
+
+## Primitive: stt (video/audio → transcript)
+
+Transcribe speech from **any video or audio URL** on the caller's saved AI provider keys (video sources are demuxed to audio automatically via the platform ffmpeg seam). One job returns **BOTH formats**:
+
+1. **Simple subtitle version** — the plain transcript (`output.text`, `transcript.txt`) plus timed SRT cues (`output.subtitles.srt`, `transcript.srt`) with no speaker labels — drop-in captions.
+2. **Advanced multi-speaker version** — speaker-attributed, timestamped segments (`output.segments`, each `{ speaker, start_sec, end_sec, text }`, with `output.speakers` listing the cast) for multi-narration sources like podcasts, interviews, and skits — stored durably as `transcript.json`.
+
+- `POST /api/v1/primitives/audio/transcribe` (alias: `POST /api/v1/primitives/stt`)
+- Body: `{ "tracer": "...", "payload": { "source_url": "https://...", "diarize"?: true, "language"?: "en", "prompt"?: "...", "provider"?: "gemini" | "openai" | "openrouter", "model"?: "...", "media_type"?: "auto" | "video" | "audio" }, "webhook_url"?: "..." }`
+  - `source_url` (required) — video OR audio URL. Aliases: `source_video_url`, `source_audio_url`, `video_url`, `audio_url`, `url`. For a fork's composition, pick the right source via `GET /api/v1/compositions/:forkId/remove-video-captions` first.
+  - `diarize` (default `true`) — attribute segments to speakers. **Speaker labels need a Gemini key** (audio-native model); OpenAI/OpenRouter keys degrade to a plain single-speaker transcript (`diarization: "none"`).
+  - `prompt` (optional) — domain/vocabulary hint; `language` (optional) — expected language code.
+- Response: standard primitive job. Poll to completion; `output` carries both formats inline (very long transcripts set `text_truncated: true` and keep the full copy in the `transcript.json` artifact), and the artifacts `transcript.json` / `transcript.srt` / `transcript.txt` are durable URLs.
+- Limits: ~40 minutes of speech per job (20MB extracted audio) — trim longer sources with `/videos/trim` or `/audio/trim` and transcribe in parts.
+- Billing: the AI transcription is BYOK (never wallet-billed); only the tiny audio-demux compute step bills (`primitive_media_lambda`), like every media primitive.
+- For the CURRENT fork, the decompose-time transcript may already exist via `video-context.json` — prefer it before paying for a new transcription; use this primitive for other URLs, multi-speaker attribution, or SRT output.
+- devcli: `vidfarm stt <file|url>` runs LOCAL-FIRST (local ffmpeg + your own key); add `--cloud` to run this route instead.
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/audio/transcribe" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "demo-stt",
+    "payload": {
+      "source_url": "https://cdn.example.com/podcast-clip.mp4",
+      "diarize": true,
+      "language": "en"
+    }
+  }'
+```
+
 ## Brainstorm primitives
 
 The `brainstorm/*` primitives are the strategy toolkit. They are reusable, billable AI reasoning steps — the same family the AI Copilot exposes as chip suggestions. Treat **product placement** as a first-class member of this family, right alongside angles and hooks:

package/SKILL.platform.md

@@ -134,6 +134,20 @@ A composition is a single HTML document plus a JSON sidecar.
        style="position:absolute;left:5%;top:60%;...">
     HOOK LINE
   </div>
+  <!-- ANIMATED caption cue (word-by-word): the layer carries the preset +
+       per-cue vars; each word is a span whose inline animation-delay/duration
+       IS its spoken window relative to the layer's data-start. -->
+  <div data-hf-id="element_cap_x_0" data-layer-kind="caption"
+       data-caption-animation="highlight-pop" data-caption-uppercase="1"
+       data-start="3.0" data-duration="1.8" data-track-index="2"
+       style="...;--vf-cap-hl:#7C3AED;text-transform:uppercase;overflow:visible">
+    <span data-vf-text-lines="true"><span data-vf-text-inline="true">
+      <span data-cap-word="true" data-word-start="0" data-word-end="0.4"
+            style="animation-delay:0s;animation-duration:0.4s">Stop</span>
+      <span data-cap-word="true" data-word-start="0.4" data-word-end="1.1"
+            style="animation-delay:0.4s;animation-duration:0.7s">scrolling</span>
+    </span></span>
+  </div>
   <!-- more layers... -->
   <script type="module"><!-- composition-runtime.ts injected inline --></script>
 </div>
@@ -177,8 +191,8 @@ The same runtime is used inside the Trackpad Editor iframe AND when a published
 - Renders the composition into a preview iframe using a `blob:` URL
 - Bidirectionally syncs timeline drag → HTML `data-start`/`data-duration` mutations, and inspector edits → text/style changes
 - Persists every commit back to the API as PUT `composition.html` and PATCH `composition.json`
-- Exposes an **Editor Action API** so the Chat panel can invoke 13 typed actions (`add_layer`, `remove_layer`, `set_layer_start`, `set_layer_duration`, `set_layer_style`, `group_layers`, `ungroup_layers`, `replace_composition_html`, etc.)
-- Runs the composition-runtime script inside the preview iframe to keep timeline scrub in sync with the current time. That runtime also scrubs **Ken Burns** image animations (CSS pan/zoom on `[data-kenburns]` `<img>` layers) to the playhead, so the preview matches the render frame-for-frame. Presets and keyframes live in `src/hyperframes/composition.ts` (`KEN_BURNS_CSS`), mirrored into the editor bundle and the serve-time HTML rewrite.
+- Exposes an **Editor Action API** so the Chat panel can invoke typed actions (`add_layer`, `remove_layer`, `set_layer_start`, `set_layer_duration`, `set_layer_style`, `set_captions` — animated word-by-word caption runs — `group_layers`, `ungroup_layers`, `replace_composition_html`, etc.)
+- Runs the composition-runtime script inside the preview iframe to keep timeline scrub in sync with the current time. That runtime also scrubs **Ken Burns** image animations (CSS pan/zoom on `[data-kenburns]` `<img>` layers) and **animated captions** (word-by-word `vf-cap-*` animations on `[data-caption-animation]` caption layers — Ken Burns normalizes to clip progress, captions seek to the absolute offset `time − data-start` because each word's inline delay/duration is its spoken window) to the playhead, so the preview matches the render frame-for-frame. Presets and keyframes live in `src/hyperframes/composition.ts` (`KEN_BURNS_CSS`, `CAPTION_ANIM_CSS` + `CAPTION_STYLE_PRESETS`), mirrored into the editor bundle and the serve-time HTML rewrite; `normalizePublishHtml` re-injects both stylesheets (marker-guarded) into legacy compositions at render time. Caption gotcha: the layer's inline `text-transform`/`overflow` must be set on the element (the stylesheet's `[data-caption-uppercase]` rule loses to inline styles), and cue text edits must rebuild the word spans (`refreshCaptionWordSpans`) rather than setting `textContent`.
 
 The editor is a single monolithic React component (~9k lines). It builds cleanly to a static bundle deployed to CloudFront under `/editor/`.
 
@@ -264,8 +278,9 @@ Primitives are hosted operations that are NOT templates — they're reusable bui
 - `download_media_video` — social/media URL → durable MP4
 - `hyperframes_render` — composition HTML → MP4 (the render primitive publishing uses)
 - `brainstorm` — LLM brainstorm route
+- `tts` / `stt` — speech, BYOK on the caller's saved provider keys. `POST /api/v1/primitives/audio/speech` (`primitive:tts`) turns text into narration audio with a promptable voice style (`instructions`); `POST /api/v1/primitives/audio/transcribe` (`primitive:stt`) transcribes any video or audio URL (video demuxes through the primitive-media ffmpeg seam first, 64kbps mono, ~40min/20MB cap) and returns **two formats**: a simple subtitle transcript (plain text + SRT cues) and an advanced multi-speaker version (speaker-attributed timed segments; diarization is Gemini-only — JSON-mode audio transcription in `src/services/speech.ts`). The raw provider speech calls live in `src/services/speech.ts`, shared by `ProviderService` (leased BYOK keys, usage metering) and the devcli `vidfarm tts`/`vidfarm stt` commands, which run LOCAL-FIRST on the user's own env key (local ffmpeg demux, no job, no wallet) with `--cloud` as the explicit backup route.
 
-Directors call primitives when they want a one-off asset before feeding it into a template. Templates call primitives internally as workflow steps. All primitives are metered against the caller's wallet.
+Directors call primitives when they want a one-off asset before feeding it into a template. Templates call primitives internally as workflow steps. All primitives are metered against the caller's wallet (BYOK AI calls excepted — e.g. tts/stt model spend rides the caller's own provider key; only compute like the stt demux step wallet-bills).
 
 ## Billing