@mevdragon/vidfarm-devcli 0.11.0 → 0.13.0

package/SKILL.director.md

@@ -125,9 +125,9 @@ The editor loads `composition.html` and `composition.json` from the fork's worki
 The editor is a full timeline surface:
 
 - **Timeline** — multi-track, drag/trim/split clips, group/ungroup layers
-- **Inspector panel** — per-layer property editor (text, color, font, position, size, media source, playback start, volume). Image layers also expose an **Animate (Ken Burns)** dropdown — a slow pan/zoom over the clip's full duration (see below).
+- **Inspector panel** — per-layer property editor (text, color, font, position, size, media source, playback start, volume). Image layers also expose an **Animate (Ken Burns)** dropdown — a slow pan/zoom over the clip's full duration (see below). Text/caption layers expose an **Animated captions** preset picker — word-by-word TikTok/CapCut caption styles (see below) — plus active-word color, highlight color, and uppercase controls.
 - **Canvas frame overlay** — 8-handle resize + WYSIWYG positioning
-- **Chat panel** — AI assistant that can programmatically edit the composition (`add_layer`, `remove_layer`, `set_layer_timing/visual/style/media/text`, `group_layers`, `generate_layer`, `replace_composition_html`, ...). The assistant uses the caller's saved provider keys. **`generate_layer`** is the one-shot "generate AI media and place it": it submits the `/videos/generate` or `/images/generate` primitive job, drops a placeholder clip into the target slot (fill a blank gap, replace a scene, or overlay), and auto-swaps in the finished media when the job settles — no manual polling. In-flight generations surface in `editor_context.pending_generations`.
+- **Chat panel** — AI assistant that can programmatically edit the composition (`add_layer`, `remove_layer`, `set_layer_timing/visual/style/media/text`, `set_captions`, `group_layers`, `generate_layer`, `replace_composition_html`, ...). The assistant uses the caller's saved provider keys. **`generate_layer`** is the one-shot "generate AI media and place it": it submits the `/videos/generate` or `/images/generate` primitive job, drops a placeholder clip into the target slot (fill a blank gap, replace a scene, or overlay), and auto-swaps in the finished media when the job settles — no manual polling. In-flight generations surface in `editor_context.pending_generations`.
 - **Auto-save** — every commit writes composition.html + composition.json to the working state. No explicit save button.
 
 The editor persists to:
@@ -236,7 +236,7 @@ Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/
 ```
 
 - **Poll** `GET /clips/scan/:scanId` until the source `status` is `complete` (or `failed` — crashed pipelines are reconciled to `failed`, pollers never spin).
-- **Results**: `GET /clips/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /clips/search` `{ query: "someone looks confused" }`, per-clip download at `GET /clips/:clipId/download`. Clips carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, and (with a gemini/openai key) a semantic embedding.
+- **Results**: `GET /clips/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /clips/search` `{ query: "someone looks confused" }`, per-clip download at `GET /clips/:clipId/download`. Clips carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, the hunt's `tracer`, and (with a gemini/openai key) a semantic embedding.
 - **Billing** — **AWS compute only** (`clip_scan_lambda` GB-seconds + Step Functions transitions, a fraction of a cent for typical hunts; the 202 response includes a `compute_estimate`). The AI tagging/refine runs on **your saved provider key (BYOK)** and is never wallet-billed. Submission is wallet-gated (402 when empty).
 - **The original stays temporary** — URL-ingested and uploaded sources live ONLY in your temp folder (`users/…/temporary/clip-sources/…`) with a hard **30-day TTL** (auto-deleted from S3 + the temp-file list). The hunted clips themselves are durable.
 - **Captions rule** — "no captions / no on-screen text" hunts are handled by scene SELECTION (the refine pass drops text-heavy scenes). **Never run GhostCut caption removal on a long-form source** (it is hard-capped at ~15 minutes); to actually erase burned-in text, apply `POST /api/v1/primitives/videos/remove-captions` to individual FINISHED clips afterwards.
@@ -303,6 +303,22 @@ Three surfaces:
 - **Editor UI (human)** — the Inspector's **Animate (Ken Burns)** dropdown on any selected image.
 - **Desktop agents (devcli)** — `vidfarm place ./work --src <img> --kind image --ken-burns zoom-in [--ken-burns-intensity 0.3]`, or fused with generation: `vidfarm generate image --prompt "…" --place ./work --ken-burns pan-left`.
 
+### Animated captions — word-by-word caption styles (TikTok/CapCut)
+
+Compositions support **first-class animated captions**: caption layers whose words animate one at a time, synced to the spoken audio, identical in the editor preview and both render paths. A caption *run* is one layer per cue (a page of ~3-5 words) on a shared track; each layer carries `data-caption-animation="<preset>"` and per-word `<span data-cap-word>` timing — no keyframe authoring, and the text stays editable like any text layer.
+
+**Preset styles** (each = an animation + a look; every field individually overridable): `spotlight` (active word gets a rounded highlight pill on bold outlined text — the classic Hormozi/CapCut look), `karaoke` (words fill with color as spoken and stay lit), `word-pop` (one word at a time, punchy scale-in), `stack-up` (words fade/rise in and accumulate), `neon` (active word glow pulse), `bounce` (active word bounces). Overridables: `caption_active_color` (spoken-word color), `caption_highlight_color` (spotlight pill), `caption_uppercase`, plus the normal text fields (`color`, `background`, `background_style`, `font_family`, `font_size`).
+
+**Word timings**: real word-level timestamps come from OpenAI STT (whisper-1); Gemini/OpenRouter transcripts get char-weighted estimated word windows, which still read well. Hand-typed text is auto-paged and estimated.
+
+Four surfaces:
+- **Editor web copilot** — for narration that needs TRANSCRIBING (a TTS voiceover layer, replaced audio, word-accurate karaoke), call the **captions primitive**: `POST /api/v1/primitives/audio/captions` `{ tracer, payload: { source_url: <narration URL>, style } }` — one job transcribes with word-level timings and returns `captions` (the cue array) plus `set_captions_action` (fully-formed `editor_action` arguments); apply verbatim with `editor_action action_type=set_captions`. For speech that is the decomposed source video's own audio, skip the job: derive cues from `video_context`'s timestamped segments and call `set_captions` directly with `captions=[{text,start,duration,words?}]`, or just `text` (+ `start`/`duration` window) to auto-page. Pick `caption_style`; the call **replaces** all existing animated caption layers, so it's also the "restyle everything" verb. Per-layer: `set_layer_style` with `caption_*` fields restyles one cue, `set_layer_text` rewrites its words, `caption_animation="none"` flattens back to static text. Never hand-build word-by-word captions from many `add_layer` calls.
+- **REST one-step (headless/automation)** — `POST /api/v1/compositions/:forkId/captions` `{ audio_url?|text?, style?, animation?, active_color?, highlight_color?, uppercase?, max_words_per_cue?, offset_sec?, provider?, language? }` transcribes the fork's own narration (auto-resolves: explicit `audio_url` → audio layer → backing video → source video) and writes the animated caption layers into the working composition **server-side** in one synchronous call — BYOK STT, non-billing. On a local `vidfarm serve` box, open editor tabs live-morph the result. Do **not** use it while a cloud editor session is open (the editor's next auto-save would overwrite it) — in-editor, mutate through `set_captions` instead.
+- **Editor UI (human)** — right-click → **Insert Captions** drops a starter cue; the Inspector's **Animated captions** picker switches styles on any selected text/caption layer, with active-word/highlight color and uppercase controls next to it.
+- **Desktop agents (devcli)** — `vidfarm captions generate ./work [--style spotlight]` transcribes the composition's own narration LOCAL-FIRST on your key (prefers OpenAI for real word timestamps) and writes the cue layers into `composition.html` on disk (a running `serve` live-morphs it). Alternates: `--audio <file|url>`, `--srt <file>` (e.g. from `vidfarm stt`), or `--text "<script>"`. Then `vidfarm captions style ./work --style karaoke` (restyle, same text/timing), `captions list`, `captions clear`.
+
+The plain STT primitive grew the same hook: `POST /api/v1/primitives/audio/transcribe` accepts `word_timestamps: true` to attach word-level timings to each transcript segment (real on an OpenAI key via whisper-1; other providers stay segment-level).
+
 ### Local file paths as media (skip the S3 upload)
 
 `vidfarm place --src` (and `approve --video/--media`) accept a **local file path**, not just a URL — so a power user bulk-building compositions from a folder of clips on their own machine never has to upload every asset to Vidfarm storage:
@@ -505,6 +521,8 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
 | `vidfarm tts "…" [--style "…"] [--voice <v>] [--out <file>]` | (LOCAL-FIRST: your own OPENAI/GEMINI/OPENROUTER_API_KEY → audio file on disk; `--cloud` = `POST /api/v1/primitives/audio/speech` + poll) | text → narration audio with a promptable voice style |
 | `vidfarm stt <file\|url> [--out <base>] [--no-diarize]` (alias: `transcribe`) | (LOCAL-FIRST: local ffmpeg demux + your own key; `--cloud` = `POST /api/v1/primitives/audio/transcribe` + poll) | video/audio → transcript in BOTH formats: simple subtitles (txt + SRT) and multi-speaker segments (json) |
 | `vidfarm place <dir> --src <url\|file> [--at\|--replace]` | (edits local composition.html; local files → serve disk store or temp upload) | drop media (URL **or local file**) into a gap / over a scene |
+| `vidfarm captions generate <dir> [--style <preset>] [--audio <f>\|--srt <f>\|--text "…"]` | (LOCAL-FIRST: STT on your own key — OpenAI = real word timestamps — then edits local composition.html) | transcribe narration → animated word-by-word caption cues |
+| `vidfarm captions style <dir> --style <preset>` / `captions list` / `captions clear` | (edits local composition.html) | restyle / inspect / remove animated captions |
 | `vidfarm decompose <forkId>` | `POST .../compositions/:forkId/auto-decompose` | split source into scenes |
 | `vidfarm remove-video-captions <forkId>` (alias: `ghostcut`) | `GET .../compositions/:forkId/remove-video-captions` | subtitle-removal status |
 | `vidfarm snapshot <forkId>` | `POST .../compositions/:forkId/versions` | save composition fork version |

package/SKILL.platform.md

@@ -134,6 +134,20 @@ A composition is a single HTML document plus a JSON sidecar.
        style="position:absolute;left:5%;top:60%;...">
     HOOK LINE
   </div>
+  <!-- ANIMATED caption cue (word-by-word): the layer carries the preset +
+       per-cue vars; each word is a span whose inline animation-delay/duration
+       IS its spoken window relative to the layer's data-start. -->
+  <div data-hf-id="element_cap_x_0" data-layer-kind="caption"
+       data-caption-animation="highlight-pop" data-caption-uppercase="1"
+       data-start="3.0" data-duration="1.8" data-track-index="2"
+       style="...;--vf-cap-hl:#7C3AED;text-transform:uppercase;overflow:visible">
+    <span data-vf-text-lines="true"><span data-vf-text-inline="true">
+      <span data-cap-word="true" data-word-start="0" data-word-end="0.4"
+            style="animation-delay:0s;animation-duration:0.4s">Stop</span>
+      <span data-cap-word="true" data-word-start="0.4" data-word-end="1.1"
+            style="animation-delay:0.4s;animation-duration:0.7s">scrolling</span>
+    </span></span>
+  </div>
   <!-- more layers... -->
   <script type="module"><!-- composition-runtime.ts injected inline --></script>
 </div>
@@ -177,8 +191,8 @@ The same runtime is used inside the Trackpad Editor iframe AND when a published
 - Renders the composition into a preview iframe using a `blob:` URL
 - Bidirectionally syncs timeline drag → HTML `data-start`/`data-duration` mutations, and inspector edits → text/style changes
 - Persists every commit back to the API as PUT `composition.html` and PATCH `composition.json`
-- Exposes an **Editor Action API** so the Chat panel can invoke 13 typed actions (`add_layer`, `remove_layer`, `set_layer_start`, `set_layer_duration`, `set_layer_style`, `group_layers`, `ungroup_layers`, `replace_composition_html`, etc.)
-- Runs the composition-runtime script inside the preview iframe to keep timeline scrub in sync with the current time. That runtime also scrubs **Ken Burns** image animations (CSS pan/zoom on `[data-kenburns]` `<img>` layers) to the playhead, so the preview matches the render frame-for-frame. Presets and keyframes live in `src/hyperframes/composition.ts` (`KEN_BURNS_CSS`), mirrored into the editor bundle and the serve-time HTML rewrite.
+- Exposes an **Editor Action API** so the Chat panel can invoke typed actions (`add_layer`, `remove_layer`, `set_layer_start`, `set_layer_duration`, `set_layer_style`, `set_captions` — animated word-by-word caption runs — `group_layers`, `ungroup_layers`, `replace_composition_html`, etc.)
+- Runs the composition-runtime script inside the preview iframe to keep timeline scrub in sync with the current time. That runtime also scrubs **Ken Burns** image animations (CSS pan/zoom on `[data-kenburns]` `<img>` layers) and **animated captions** (word-by-word `vf-cap-*` animations on `[data-caption-animation]` caption layers — Ken Burns normalizes to clip progress, captions seek to the absolute offset `time − data-start` because each word's inline delay/duration is its spoken window) to the playhead, so the preview matches the render frame-for-frame. Presets and keyframes live in `src/hyperframes/composition.ts` (`KEN_BURNS_CSS`, `CAPTION_ANIM_CSS` + `CAPTION_STYLE_PRESETS`), mirrored into the editor bundle and the serve-time HTML rewrite; `normalizePublishHtml` re-injects both stylesheets (marker-guarded) into legacy compositions at render time. Caption gotcha: the layer's inline `text-transform`/`overflow` must be set on the element (the stylesheet's `[data-caption-uppercase]` rule loses to inline styles), and cue text edits must rebuild the word spans (`refreshCaptionWordSpans`) rather than setting `textContent`. Speech→captions REST surfaces: `primitive:captions` (`POST /api/v1/primitives/audio/captions`, src/primitive-registry.ts) is the composition-agnostic job — STT with `wordTimestamps` (openai/whisper-1 real, others estimated via `estimateCaptionWords`) paged by `src/services/captions.ts` into cues + a ready `set_captions_action`; `POST /api/v1/compositions/:forkId/captions` (src/app.ts, next to the composition.html PUT) is the one-step variant that resolves the fork's narration, transcribes on raw customer keys (`loadCustomerVisionKeys` + `transcribeSpeechWithKey`), applies via `insertCaptionLayers` (shared with the devcli), and persists through the same sanitize path — intentionally NOT watch-suppressed so serve tabs live-morph; not for open cloud editor sessions (auto-save clobber). Scrub-stability guard (both `syncCaptionClip` copies — `src/composition-runtime.ts` + the editor's `runtimeScript()`): Chrome permanently drops a CSS-paused `fill:none` animation once its `currentTime` is pushed past the effect end, so the scrub parks `fill:none` word animations at 0 when the playhead is past their window (visually identical resting state, keeps them alive for scrub-back) while `fill:forwards` presets overshoot to hold their final keyframe; missing word animations are restarted by cycling `animation-name` through `none`. Live element patches that rebuild word spans must re-scrub via the `__player.syncCaptions()` runtime API (the editor calls it after `patchLiveElement`/`previewLiveElement`) or fresh spans sit paused-at-0 (invisible for word-pop/stack-up) until the next seek.
 
 The editor is a single monolithic React component (~9k lines). It builds cleanly to a static bundle deployed to CloudFront under `/editor/`.
 
@@ -301,7 +315,7 @@ Per-stage config is loaded via `.env.<stage>`. Critical variables:
 - `AWS_REGION`, AWS credentials
 - `VIDFARM_<STAGE>_PUBLIC_BASE_URL` — the API base
 - `VIDFARM_<STAGE>_ENCRYPTION_SECRET` — DynamoDB field encryption
-- `VIDFARM_<STAGE>_API_KEY_SALT` — API key hashing
+- `VIDFARM_<STAGE>_API_KEY_SALT` — API key hashing. Rotating it invalidates previously-issued keys on that stage only; the shared staging table can hold keys hashed under several salts (deployed stage vs local dev boxes). `getVisibleApiKey` (src/app.ts) is salt-aware: it returns the newest active key whose `key_hash` validates under THIS server's salt (minting one if none do), so boot pages / Settings never hand the browser a key the serving box would reject — the editor chat and all its server-executed tools authenticate with that embedded key.
 - `VIDFARM_<STAGE>_BOOTSTRAP_*` — initial admin user
 - `VIDFARM_<STAGE>_RESEND_*` — email
 - `VIDFARM_<STAGE>_SENTRY_*` — Sentry DSN