@mevdragon/vidfarm-devcli 0.20.6 → 0.20.9

package/.agents/skills/vidfarm-director/references/editor-workflows.md

@@ -0,0 +1,365 @@
+## Agentic editing: the three axes (SWAP ↔ REPLACE)
+
+Almost every editor session is a director taking a template / fork / project and **re-working** it, and a re-work only ever touches three independent axes — **SCENES** (the video/image clips carrying the visuals), **AUDIO** (narration/voiceover, music, SFX), and **TEXT** (captions, titles, overlays). On each axis the intent sits on a **SWAP ↔ REPLACE** spectrum, and the three axes in one request can sit at different points — decide each before you act:
+
+- **SWAP (light, cheap, common)** — keep the structure, change content **in place**: rewrite a caption/text layer (`set_layer_text` / `set_captions`), swap one clip's media for another of the same kind keeping its timing + geometry (`set_layer_media`), or re-voice narration (`vidfarm speech regenerate` / `/audio/regenerate-speech` → mute the old span, `place`/`add_layer` the new audio). Many jobs are **only** a text swap — a two-minute change; just do it.
+- **REPLACE (heavy)** — throw out that axis's content and rebuild it: restructure the scenes outright (new clip count / new beats via `remove_layer` + `add_layer`/`generate_layer`, or one `replace_composition_html`), lay a brand-new audio bed, or rewrite every caption. The **hardest** end is a full **re-theme** where the ONLY thing preserved is the **viral DNA** (hook shape, pacing, scene-count rhythm, transition/caption style) while every scene, every word, and the audio are all replaced for the director's new subject.
+
+Name the plan back in these terms ("I'll SWAP the captions and REPLACE the scenes"), then execute axis by axis. Be **proactive at the heavy end** — carry the whole transformation, don't wait to be micro-managed one layer at a time.
+
+**Fuel a scene REPLACE with raw clips, not expensive AI video.** A heavy scenes-axis replace needs footage; sources in cost order: (1) the director's own library — search `/raws` and `/files` (`vidfarm raws search …`, `vidfarm files --search …` / `browse_files`); (2) **HUNT new raws** out of a long-form source (podcast/VOD/webinar or any YouTube/TikTok/IG/X URL) — `vidfarm raws scan <src> --prompt "<what the new scenes need>" --aspect <canvas> [--duration N --no-text --range …]` (local-first, free compute) or the async `POST /clips/scan` / `/raws/scan`; then reuse the picks (`set_layer_media` / `vidfarm set-media` to swap in place, `add_layer`/`vidfarm place` for net-new scenes); (3) `generate_layer` / `vidfarm generate` AI generation — the **expensive last resort**, only for scenes no real clip can cover. When a big scene re-work is asked for but no footage is given, **ask for a source to hunt (or point at the raws library) before AI-generating** — see [Raws](#raws-long-form--short-form-raws) and [Generate AI media …](#generate-ai-media-and-drop-it-on-the-timeline).
+
+**Audio is natively multi-track — overlay narration + music + SFX, each at its own volume.** A composition mixes UNLIMITED simultaneous `<audio>` layers; each sits on its own `data-track-index` and carries its own `data-volume` (0–2, default 1), and the runtime mixes them with per-track volume honored identically in the preview and the exported MP4 (a real ffmpeg `amix` of every audio layer at render). So you never need a pre-mixed file — lay **narration/voiceover at ~1.0 on one track, a music bed at ~0.1–0.2 on a separate track, and SFX on their own tracks**, each via `add_layer kind=audio` (web) / `vidfarm place --kind audio --volume …` (devcli), tuning levels later with the Inspector's Volume slider or `set_layer_media` (`volume`, `muted`). **The key move when recreating a template whose original baked music + narration into ONE audio track: rebuild it as TWO independent tracks** — a fresh narration track (`/audio/speech`, or same-voice reword via `/audio/regenerate-speech` / `vidfarm speech regenerate`) at ~1.0 and a separate real music track at ~0.1–0.2 — then mute/remove the original combined source-audio layer. This gives the director independent voice and music volume, and works around AI TTS being unable to emit narration+music in one file: you compose the mix on the timeline. Honesty: you can't un-mix / stem-separate the original's baked audio — the two tracks are a fresh narration track **plus** a real music file (owned / user-provided / `browse_files` / generated with `/api/v1/primitives/music/generate` (`vidfarm music`)), never a faked or duplicated voice layer.
+
+## The three paintbrushes & two replication harnesses
+
+Vidfarm is deliberately founder-friendly and thrift-first: **we do not spend expensive AI credits on every scene.** The three axes above tell you *what* is being re-worked; the **three paintbrushes** tell you *how* to paint each scene once you've decided to REPLACE it. A **technical replication decomposition** is exactly this — for the target template, name **when / where / what** each beat should be painted with.
+
+> **This decomposition is now a materialized artifact.** Every decomposed fork carries a **replication harness** (`replication-harness.json` / `editor_context.replication_harness` / `GET .../replication-harness.json`) that already contains BOTH plans as concrete per-beat brush assignments plus reusable-asset guidance and viral-DNA guardrails. When it's present, **execute its beats** instead of re-deriving them; the methodology below is how it was built and the fallback when it's absent (older forks / free tier). See the route section further down.
+
+1. **Raw clips** — cut + remix existing long-form / short-form footage. The workhorse. Source in cost order: the director's own `/raws` + `/files` library → HUNT new raws out of a URL/VOD (`vidfarm raws scan`) → only then generate. Combining a **background video + a foreground video** (greenscreen / picture-in-picture) covers a huge share of "video meme" formats with zero generation.
+2. **HTML/JS hyperframes** — video-from-HTML. Animate text, image elements, logos, stickers, charts and lower-thirds with CSS/declarative motion (web editor) or JS adapters (anime.js/GSAP/Lottie/Three, devcli only). Deterministic, cheap, re-themeable — the right brush for titles, kinetic captions, data beats, brand cards, and any "graphic" scene.
+3. **Pure AI generation** — AI image / video / voice / music. The most expensive brush (AI **video** especially, $1–$10+); the last resort for a beat no clip or hyperframe can cover.
+
+**Reusable asset library.** Directors accumulate logos, stickers, reactions, b-roll, a-roll, and a brand media kit. The decomposition should have an **opinion on when and where to reuse these** — reactions and stickers for beats, the media kit for brand identity, b-roll to cover cuts. But usage can quietly break the format: reskinning past a load-bearing beat kills what made it go viral. **Anchor every asset/paintbrush call to the harness** (`viral_dna`, `viral_dna.emotional_punch`, `editor-harness.json`, `static_vs_pivot`) — reuse and re-skin the *dressing*, preserve the *DNA*.
+
+For any replication, offer the director **two harnesses** and recommend (A) unless they've asked for premium or the budget covers it:
+
+**(A) Cheap & efficient** — the default. Reach for, in roughly this order:
+- **Recaption text** — often the whole job is rewriting the caption/title layers (`set_captions` / `set_layer_text`). A two-minute change.
+- **Background + foreground video** memes — composite two clips instead of generating one.
+- **Animate HTML + image elements** with hyperframes (Ken Burns on a still, kinetic type, animated logo/sticker) instead of AI motion.
+- **Reuse media elements** from the library; **AI-generate a reusable element once** (a sticker, a logo treatment, a character sprite) then reuse it across scenes rather than regenerating per-scene.
+- **Greenscreen** — chroma-key a subject onto new backgrounds (`image_remove_background_greenscreen` / `media_overlay`).
+- **Raw-clip long-form and remix** — hunt + rearrange existing footage.
+- Lean on the **library of memes / reactions / b-roll / a-roll** and the **brand media kit**.
+- **Only if genuinely needed**, use AI image → and, last, AI video / voice / music.
+
+**(B) Best quality** — when the director wants premium and budget allows:
+- **AI video generation by default** for hero scenes.
+- **Storyboard with AI image first** (cheap stills to lock composition/subject), then generate motion from those references for consistency.
+- **Adversarially grade** the output with a coding agent (Claude Code / Codex / any capable AI agent) — render, critique against the harness, iterate — rather than shipping the first generation.
+
+**Be generous to free-tier / no-account directors.** Vidfarm's replication harness is open source, and an everyday capable AI agent should be able to follow this decomposition and rebuild a template **without a Vidfarm wallet** — clipping, hyperframes, and local render (`vidfarm serve`) all run on free compute. Keep the recommendation self-contained and pragmatic; don't gate the cheap path behind paid primitives.
+
+**Free tier does its OWN decomposition.** The catch on the free tier (local devcli, no account): nobody has pre-decomposed the reference video for them. There is no `video-context.json`, `editor-harness.json`, `replication-harness.json`, or `scene-annotations.json` to `vidfarm pull` — **the user and their agent must watch the video, break it into scenes/audio/text, read the viral DNA, and choose the three paintbrushes by hand.** Coach them through that; the harness above is the method to do it. **Paid Vidfarm accounts skip the manual pass**: they get a large library of **pre-decomposed viral templates** (the decompose passes already run) plus **prompt-harness best practices distilled from platform-scale learnings** — so a fork arrives already carrying its DNA, editor harness, **replication harness (both cheap + best-quality plans, per-beat brush assignments)**, and per-scene recreation annotations. It's fair to surface that upgrade when a free-tier user is decomposing by hand.
+
+## Edit in the Trackpad Editor
+
+The editor is a full timeline surface:
+
+- **Timeline** — multi-track, drag/trim/split clips, group/ungroup layers
+- **Inspector panel** — per-layer property editor (text, color, font, position, size, media source, playback start, volume). Image layers also expose an **Animate (Ken Burns)** dropdown — a slow pan/zoom over the clip's full duration (see below). Text/caption layers expose an **Animated captions** preset picker — word-by-word TikTok/CapCut caption styles (see below) — plus active-word color, highlight color, and uppercase controls.
+- **Canvas frame overlay** — 8-handle resize + WYSIWYG positioning
+- **Chat panel** — AI assistant that can programmatically edit the composition (`add_layer`, `remove_layer`, `set_layer_timing/visual/style/media/text`, `set_layer_keyframes` — script-free CSS keyframe motion on one layer — `set_captions`, `group_layers`, `nudge_layers`, `ripple_edit`, `trim_layer`, `set_layer_zindex`, `generate_layer`, `replace_composition_html`, ...). The assistant uses the caller's saved provider keys, and reads an enriched `editor_context` (per-layer transitions / Ken Burns / custom-animation state; `recent_action_results` now reports successes as well as failures). See "Script-free keyframe motion & timeline verbs" below for the newer motion/timeline actions. **`generate_layer`** is the one-shot "generate AI media and place it": it submits the `/videos/generate` or `/images/generate` primitive job, drops a placeholder clip into the target slot (fill a blank gap, replace a scene, or overlay), and auto-swaps in the finished media when the job settles — no manual polling. In-flight generations surface in `editor_context.pending_generations`.
+- **Auto-save** — every commit writes composition.html + composition.json to the working state. No explicit save button.
+
+The editor persists to:
+
+- `PUT /api/v1/compositions/:forkId/composition.html` — replace HTML
+- `PATCH /api/v1/compositions/:forkId/composition.json` — patch metadata
+
+Automation that needs to bypass the UI can call these endpoints directly.
+
+Reads also work anonymously since the `compositions/forks/*` S3 prefix is public-read — clients that only need to fetch (not mutate) can bypass the API and hit S3 directly:
+
+- `GET https://<bucket>.s3.<region>.amazonaws.com/compositions/forks/<forkId>/working/composition.html`
+- `GET .../working/composition.json`
+- `GET .../working/manifest.json` — fork identity + genealogy (`parent_fork_id`, `parent_version`, `latest_version`, direct URLs for html/json/parent manifest). Also served via `GET /api/v1/compositions/:forkId/manifest.json` and `.../versions/:v/manifest.json`.
+
+Treat the `forkId` as an unguessable bearer token — anyone with it can view the composition.
+
+## Auto-decompose
+
+If the fork's source is a raw video (not a pre-decomposed template), call auto-decompose to split it into scenes:
+
+```
+POST /api/v1/compositions/:forkId/auto-decompose
+Content-Type: application/json
+
+{ "mode": "smart" | "time-slice", "html": "<optional current html>", "requested_scenes": null, "target_scene_seconds": null, "user_prompt": "optional guidance" }
+```
+
+Two modes:
+
+- **`smart`** — uses the caller's Gemini/OpenAI/OpenRouter key. Samples frames, extracts scenes with labels, literal visual **descriptions**, and viral notes, extracts on-screen captions with positions, **transcribes the audio track verbatim** (timestamped segments; Gemini first, OpenAI Whisper fallback; skipped when the video has no audio), and detects viral DNA (hook, retention, payoff, plus **`emotional_punch`** — the vibe/joke/intonation that makes it FEEL, reasoned over BOTH the frames and the transcript so the trending sound and vocal delivery are captured, not just the visuals). Takes 30-60 seconds. Auto-fires GhostCut for subtitle removal in the background.
+- **`time-slice`** — deterministic equal-duration split. Instant. No AI provider needed.
+
+**Source length cap.** Vidfarm caps auto-decompose (and all inspiration ingest) at **120 seconds** of source video. If the source is longer, the API responds with `400 { ok: false, code: "source_too_long", duration_seconds, max_duration_seconds }` and no scenes are written. Trim the source before forking / ingesting.
+
+Response:
+
+```
+{ "ok": true, "mode": "smart", "scene_count": 6, "caption_count": 4, "duration_seconds": 12.4, "ghostcut_pending": true, "ghostcut_task_id": "..." }
+```
+
+If `ghostcut_pending: true`, poll `POST /api/v1/compositions/:forkId/remove-video-captions-poll` (legacy alias: `/ghostcut-poll`) until status transitions to `done` or `failed`.
+
+Read-only state (does **not** advance the job or bill): `GET /api/v1/compositions/:forkId/remove-video-captions` (legacy alias: `/ghostcut`). Returns `{ status, task_id, original_source_url, mirrored_url, submitted_at_ms, completed_at_ms, failure_reason }`. This is how you read the composition's **two video sources**: `original_source_url` is the **original video** (raw upload, captions intact) and `mirrored_url` is the **decomposed video** (the processed, caption-free copy the timeline renders from). Use this when you want both URLs without triggering another poll — the AI copilot uses it to pick between the original video and the decomposed video before feeding a video into a primitive route (image extract, dedupe, trim, ai video edit, etc.). Prefer the **decomposed video** (`mirrored_url`) when `status === "done"` and the downstream call should be caption-free; prefer the **original video** (`original_source_url`) when the user explicitly wants the raw upload or when `status !== "done"`.
+
+## Video context (transcript + scene descriptions)
+
+`GET /api/v1/compositions/:forkId/video-context.json` — read-only, non-billing. Returns everything the smart decompose learned about the source video, so an agent can ground its work in what the video actually **says** and **shows**:
+
+```
+{
+  "ok": true,
+  "status": "ready" | "none",
+  "summary": "...",
+  "duration_seconds": 12.4,
+  "transcript": {
+    "text": "full verbatim transcript",
+    "language": "en",
+    "provider": "gemini",
+    "segments": [{ "start": 0.0, "end": 2.4, "text": "..." }]
+  },
+  "scenes": [{
+    "slug": "hook_reveal",
+    "start": 0, "duration": 2.5,
+    "label": "Hook",
+    "description": "Literal visual description of what's on screen in this scene",
+    "viral_note": "...",
+    "transcript_excerpt": "what is spoken during this scene"
+  }],
+  "viral_dna": {
+    "trend_tagline": "...", "hook": "...", "retention": "...", "payoff": "...",
+    "emotional_punch": {
+      "core_emotion": "smug relief-laugh", "tone": "deadpan comedic",
+      "arc": "flat setup → mounting absurdity → sudden turn", "peak_moment": "the reveal at ~0:07",
+      "mechanism": "subversion + trending-sound beat drop synced to the cut",
+      "humor": "the joke is X — it works because Y (rebuild the SAME joke around a new subject)",
+      "delivery": "flat VO cadence, hold the beat before the punch, beat drop lands on the reveal",
+      "preserve": ["keep the beat drop on the reveal cut", "keep the flat unbothered delivery"]
+    },
+    "static_vs_pivot": {
+      "summary": "Keep the reveal structure and audio swagger; swap the subject-specific visuals and copy.",
+      "scene_replacement": {
+        "overall": "replace_some",
+        "reason": "the proof/reveal beats are load-bearing, but the subject matter can pivot",
+        "must_keep": ["final reveal beat", "setup-to-payoff contrast"],
+        "should_replace": ["subject-specific graphics", "brand-specific screenshots"],
+        "load_bearing_scenes": ["hook setup", "payoff reveal"]
+      },
+      "narration": {
+        "overall": "customize_script",
+        "reason": "the spoken words are source-specific and stop making sense after a subject swap",
+        "same_timing_required": true,
+        "use_premium_tts_when_customizing": true
+      },
+      "music": {
+        "overall": "recreate_under_narration",
+        "reason": "the music energy matters, but the original combined track should not stay under a rewritten script",
+        "volume_guidance": "keep low under VO unless the beat-drop is the joke engine"
+      },
+      "captions": {
+        "overall": "retime_to_new_narration",
+        "reason": "caption animation is tied to the spoken words",
+        "animation_dependency": "word-by-word timing follows narration cadence"
+      },
+      "visual_rebuild": {
+        "default_strategy": "ai_images_with_ken_burns",
+        "reason": "most replacement scenes are graphic/still-like and do not need expensive motion video"
+      },
+      "viral_dna": {
+        "must_keep": ["hook structure", "payoff timing", "audio attitude"],
+        "flexible": ["subject matter", "product imagery", "copy details"],
+        "do_not_change": ["core reveal mechanic", "caption cadence"]
+      }
+    },
+    "preserve": ["..."], "avoid": ["..."], "promotions": ["..."], "keywords": ["..."]
+  }
+}
+```
+
+`status: "none"` means the fork was never smart-decomposed — run `POST /auto-decompose` with `mode: "smart"` first. `transcript` is `null` when the source has no audio track or transcription failed.
+
+**`viral_dna.emotional_punch`** is the FEELING of the format and what makes it land — the part a remix most often flattens. Where hook/retention/payoff are the mechanical structure, this captures the vibe, the joke, and the intonation. It is now **audio-aware**: the decompose feeds the transcript into the reasoning, so `mechanism`/`delivery` account for the trending sound, the music-bed drop, and vocal cadence (on short-form the sound is frequently the biggest driver of the punch). When you re-theme or remix, rebuild `humor`'s joke around the new subject rather than dropping it, match `tone`/`delivery` so intonation and comedic timing survive, and never trade the `peak_moment` payoff for a flat product plug.
+
+**`viral_dna.static_vs_pivot`** is the first-class lazy-prompt playbook. It tells you what must stay versus what can pivot when the user gives a thin prompt like "recreate this for my business" and little else. Use it to decide:
+
+- whether most scenes should be replaced or only some
+- whether narration must be customized or the original audio should stay
+- whether background music must be recreated under a new voice
+- whether animated captions need fresh timing against new narration
+- whether cheap AI images + Ken Burns are sufficient, or whether real/AI motion footage is required
+- which beats are load-bearing viral DNA versus merely subject-specific dressing
+
+Use it whenever you need to know what the video says or shows: writing/translating captions, matching hooks or dubs to spoken audio, or planning scene-level edits.
+
+- **Editor chat (frontend AI)** exposes this as the optional `video_context` tool — the copilot calls it on demand.
+- **Desktop agents (Claude Code / Codex)**: fetch the `video-context.json` route directly (e.g. `vidfarm api GET /api/v1/compositions/<forkId>/video-context.json`) to ground edits in what the video says and shows.
+
+## Editor harness (how to edit like this)
+
+`GET /api/v1/compositions/:forkId/editor-harness.json` — read-only, non-billing. The decompose pass's **technical editing direction**: *how* to edit so the result recreates the source's STYLE. It's the "how to edit like this" companion to the viral DNA's "why it works". Read it before any re-theme, multi-scene rebuild, or restyle.
+
+```
+{
+  "ok": true,
+  "status": "ready" | "empty" | "none",
+  "harness": {
+    "one_liner": "Fast first-person VO over gameplay b-roll, karaoke captions, hard cuts.",
+    "format": "reddit_story_gameplay",
+    "aspect_ratio": "9:16",
+    "pacing":      { "cut_rhythm": "fast", "avg_scene_seconds": 2.4, "cuts_per_10s": 4, "energy_curve": "front-loaded hook then steady" },
+    "typography":  { "caption_style": "karaoke", "placement": "center-lower", "font_character": "bold condensed all-caps", "emphasis": "active-word yellow", "text_density": "heavy" },
+    "broll":       { "reliance": "heavy", "sourcing": "screen-recording", "shot_kinds": ["b_roll"], "cadence": "new visual ~every 2s" },
+    "transitions": { "default": "cut", "intro": "fade", "outro": "none", "usage": "hard cuts only" },
+    "audio":       { "voiceover": "first-person narration", "music": "upbeat lofi bed, low", "sfx": "whoosh on cuts", "captions_from": "voiceover" },
+    "emotional":   { "target_feeling": "smug relief-laugh", "tone": "deadpan comedic", "comedic_timing": "hold a beat, then hard cut on the punch word", "intonation": "flat unbothered VO, no upspeak", "vibe_anchors": ["keep the trending sound, land the beat drop on the reveal cut", "hold silence before the payoff"] },
+    "scenes":      [{ "role": "hook", "importance": "critical", "must_keep": true, "note": "...", "edit_bias": "swap subject, keep timing+caption cadence" }],
+    "important_scenes": ["..."], "editing_bias": ["..."], "do": ["..."], "dont": ["..."]
+  }
+}
+```
+
+**Use it to pick concrete moves:** `typography.caption_style` maps to a `set_captions` preset; `transitions.*` to a `set_transitions` call; keep `pacing` (cut rhythm / avg_scene_seconds) when adding or splitting scenes; follow `broll.reliance`/`sourcing` to decide HUNT raws (`/raws/scan`) vs generate; lay `audio.*`. **`emotional`** is HOW to keep the FEELING while you swap the subject — the vibe/joke/intonation/sound are the first things a remix flattens, so honor `emotional.comedic_timing` (the held beat / hard cut that sells the joke), preserve `emotional.intonation` when you re-voice narration, and treat every `emotional.vibe_anchors` entry as a must-do (on short-form the trending sound / beat drop is often the biggest carrier of the punch — keep it and land it on the same cut). Pair that with `viral_dna.static_vs_pivot`: if narration is `customize_script`, revoice and then retime captions/scenes that depend on the old VO; if visual_rebuild is `ai_images_with_ken_burns`, do the cheap still+motion path before reaching for AI video; if a scene is listed under `load_bearing_scenes`, preserve its beat even when you swap the subject. **Protect** any beat in `important_scenes` or with `must_keep:true` / `importance:"critical"` — swap the subject but preserve its timing, role, and caption cadence. Treat `editing_bias`/`do`/`dont` as hard constraints; an explicit user instruction still wins. `status:"none"` → run `POST /auto-decompose` first.
+
+- **Editor chat (frontend AI)** already receives the harness inline in `editor_context.editor_harness` (no tool call needed); it can also fetch this route via `http_request`.
+- **Desktop agents (Claude Code / Codex)**: `vidfarm pull` writes `editor-harness.json` to the fork dir alongside `video-context.json`, and also materializes a merged agent bundle at `.harness/context.json` plus `.harness/agent-guide.md`. This deliberately does **not** write a top-level `AGENTS.md`, because many user repos already own that file. For local scripting or agentic edits, read `.harness/agent-guide.md` and `.harness/context.json` FIRST; do not freestyle from `composition.html` alone. The `pull` grounding line reports the generated brief paths.
+
+### `GET /api/v1/compositions/:forkId/replication-harness.json` — the technical replication analysis
+
+Read-only, non-billing. The **BUILD** companion to the editor harness (which is STYLE): the same "three paintbrushes & two replication harnesses" methodology above, but materialized per-beat for THIS template. Where `editor-harness.json` says *how to edit to feel right*, this says *which tool to reach for* — so an agent (or a free-tier user) rebuilds thrift-first instead of AI-generating every scene.
+
+```
+{
+  "ok": true,
+  "status": "ready" | "empty" | "none",
+  "harness": {
+    "summary": "Recaption + reuse: 90% of this is raw clips + hyperframes captions; AI only for the impossible reveal shot.",
+    "recommended_strategy": "cheap_efficient",
+    "paintbrush_rationale": "Talking-head + text overlays — footage is huntable and every graphic is HTML; AI video buys almost nothing here.",
+    "motion_style": { "animation_feel": "punchy kinetic type, hard cuts, sticker pop-ins", "graphic_elements": ["count-up counter", "arrow pointer", "sticker reactions"], "hyperframes_candidates": ["all captions", "the stat counter", "the end-card logo lockup"], "typography_motion": "per-word pop, active word highlighted" },
+    "asset_opportunities": ["reuse the brand-kit logo lockup on the end card", "a 'mind-blown' sticker for the reveal beat"],
+    "viral_dna_guardrails": ["never cover the reveal frame with a sticker", "keep the trending sound; land the beat drop on the reveal cut"],
+    "free_tier_note": "ffmpeg to clip the source, an HTML→video renderer (hyperframes) for every caption/graphic, chroma-key for greenscreen, a local TTS for narration — no paid primitive needed.",
+    "cheap":   { "strategy": "cheap_efficient", "one_liner": "...", "method_mix": "80% raw_clip + hyperframes; ai_gen only the reveal", "brush_breakdown": { "raw_clip": "...", "hyperframes": "...", "reusable_asset": "...", "ai_gen": "..." },
+                 "scenes": [{ "role": "hook", "timestamp": "0:00-0:03", "method": "raw_clip", "fallback_method": "hyperframes", "technique": "recaption over a hunted talking-head clip", "instructions": "...", "assets_needed": ["hook clip from /raws"], "viral_dna_guard": "keep the on-cam energy + first 3 words", "est_credits": "low" }],
+                 "reusable_assets": [{ "asset_type": "sticker", "where": "reveal beat", "source": "user_library", "purpose": "punch up the reveal", "reuse_note": "reuse across the series", "viral_dna_risk": "do NOT let it cover the reveal frame" }],
+                 "pipeline": ["hunt hook+payoff raws", "recaption in hyperframes", "reuse logo end-card", "AI-gen only the reveal"], "ai_spend": "one image→video for the reveal only", "tradeoffs": "...", "do": ["..."], "dont": ["..."] },
+    "quality": { "strategy": "best_quality", "one_liner": "AI-gen hero scenes, storyboard-first", ... }
+  }
+}
+```
+
+**Use it:** when the user wants this cheap / "without burning credits" / in bulk, execute `harness.cheap.scenes[]` beat by beat — `raw_clip` → hunt via `/raws/scan` + `set_layer_media`; `hyperframes` → build the text/graphic as a CSS/anime.js layer (NEVER AI video for on-screen text); `reusable_asset` → pull from `/files` or generate ONCE and reuse; `ai_gen` → only where the beat says so. When they want premium, follow `harness.quality`. **Always** honor each scene's `viral_dna_guard` and the top-level `viral_dna_guardrails` — that is the discipline that stops an eager sticker/logo/footage-swap from flattening the template. `replication_harness` is delivered inline to the editor chat and pulled to disk as `replication-harness.json` (rendered into `.harness/agent-guide.md`) for desktop agents. `status:"none"` → run `POST /auto-decompose` first.
+
+## Generate AI media and drop it on the timeline
+
+Both surfaces can generate a NEW AI video/image (optionally conditioned on reference images) and place it into the timeline — filling a blank gap or replacing a scene — using the async generation primitives (`POST /api/v1/primitives/videos/generate`, `/images/generate`). Generation is a **queued job**: the POST returns `202 { job_id }`, and the finished media URL only appears after the job settles (`result.primary_file_url` / `video.file_url` / an artifact `public_url`). Poll `GET /api/v1/user/me/jobs/:jobId` until `status: "succeeded"`.
+
+**Cost gate (see the Cost spectrum section):** AI **video** generation is the expensive end of the spectrum ($1–$10+/video) — **ask the director's permission before generating video**, and prefer reusing existing footage (My Files / their computer / a web search) or filming when it would do. **Image** generation is cheap — use it freely without asking. When a director wants specificity that only AI video can deliver and their budget allows, this is the right tool; just confirm first.
+
+- **Editor web copilot**: use the `generate_layer` editor_action — it does the whole flow (submit job → placeholder clip → auto-swap on completion) and reads `pending_generations` from `editor_context`. Set `media_type`, `prompt`, `aspect_ratio` (match the canvas), and `intent` (`fill_gap` with `start`/`duration`, `replace_layer` with `replace_layer_key`, or `add`). For character consistency pass a cast `reference_url` — or the character's `character_sprite_card.png` viewUrl from `/files/characters/<slug>/` (see "Recurring characters are first-class") — in `input_references` (video) / `prompt_attachments` (image).
+
+- **Desktop agents (Claude Code / Codex)** — two devcli commands:
+  ```bash
+  # 1) Sync the fork to disk and see where the blank space and scenes are
+  vidfarm pull <forkId> --dir ./work
+  #   → prints timeline gaps (e.g. 4-7s) and scene/layer keys, canvas aspect ratio,
+  #     and whether video-context.json / cast.json / editor-harness.json are present for grounding.
+
+  # 2) Generate + place in one call (fill a gap)
+  vidfarm generate video --prompt "her walking through the door" \
+      --ref ./cast/girl.png --aspect-ratio 9:16 --duration 4 \
+      --place ./work --at 4
+  #   → polls the job to the finished MP4, then inserts it at 4s in ./work/composition.html
+
+  # …or replace a scene with a generated still
+  vidfarm generate image --prompt "product hero on marble" --aspect-ratio 9:16 \
+      --place ./work --replace <layer_key>
+
+  # 3) Push it back (or it live-morphs automatically under `vidfarm serve`)
+  vidfarm publish --dir ./work --fork <forkId>
+  ```
+  `--ref @localfile` (or a bare path) auto-uploads the image and uses its durable URL as a reference. `generate` prints the media URL (drop `--place` to just get the URL); `place` can be run separately on any media URL (`vidfarm place ./work --src <url> --at <time>|--replace <layer_key>`). Placement produces the exact same clip markup the browser editor makes, so a human can finish it in the Trackpad Editor.
+
+### Fitting imperfect-dimension media (aspect mismatch)
+
+A clip/image whose **native aspect differs from the canvas** — a 16:9 landscape source on a 9:16 vertical frame is the classic case, also a tall screenshot on a wide frame, a square logo — will NOT look good on the default `object-fit:cover`, which fills the frame and **silently center-crops** the overflow (landscape → loses left/right, often cutting the subject in half; portrait → loses top/bottom). Decide the fit deliberately:
+
+- **`object_fit:"cover"`** (default) fills + crops — right for most social footage **when paired with a subject-aware `object_position`**.
+- **`object_position`** aims WHERE the cover-crop lands: a keyword (`left`, `right`, `top`, `bottom`, `"top left"`, …) or a **percentage pair** (`"30% 50%"` = 30% from the left, 50% down). Landscape clip with the subject on the left → `"left"`/`"25% 50%"`; tall screenshot with key text up top → `"top"`.
+- **`object_fit:"contain"`** shows the WHOLE media with black letterbox/pillarbox bars — use ONLY when nothing may be cropped (an infographic, a whole screenshot, a chart, a logo); a 16:9 source full-canvas on a 9:16 frame reads unfinished in `contain`.
+- **Blurred-letterbox band** for a whole landscape clip without ugly bars: place it as a centered band (`width:100, height:~34, y:~33` on 9:16) with `contain`, plus a DUPLICATE full-canvas copy on a lower track behind it at `cover` + heavy blur.
+- Never `object_fit:"fill"` (stretch/distort) unless intentional. You usually know a clip's aspect from how you sourced it (a hunted raw's `--aspect`, a generated clip's `--aspect-ratio`); if you truly need exact dims, `vidfarm probe` / `GET /videos/probe`. When you REPLACE a full-canvas scene, keep it full canvas + `cover` + a focal `object_position` rather than shrinking; when you GENERATE, request the canvas aspect so no crop is needed.
+
+Two surfaces:
+- **Editor web/serve copilot** — `editor_action` `action_type=set_layer_media` with `object_fit` and `object_position` (both also seedable on `add_layer`/`generate_layer`). `editor_context.layers[]` reports each clip's current `object_fit`/`object_position` so you can read before re-cropping.
+- **Desktop agents (devcli)** — `vidfarm place ./work --src <url> --object-fit cover --object-position "25% 50%"` (also on `--replace <layer_key>`).
+
+### Ken Burns — animate still images (slow pan/zoom)
+
+Still images can carry a **Ken Burns effect**: a slow pan or zoom that runs across the clip's full duration, in the editor preview and both render paths identically. This is what makes a slideshow of stills feel like motion footage. It is a first-class layer property — the image layer's `<img>` gets `data-kenburns="<preset>"`; there is no separate keyframe authoring.
+
+**Presets:** `zoom-in`, `zoom-out`, `pan-left`, `pan-right`, `pan-up`, `pan-down`, `zoom-in-left`, `zoom-in-right`. Optional **intensity** `0.04`–`0.5` (default `0.18`; ~`0.1` subtle, ~`0.3` dramatic) controls how far it travels/zooms.
+
+When the director says **"animate the images"**, **"make the photos move"**, **"add Ken Burns"**, or similar, apply a preset to **every** still-image layer — vary presets across adjacent clips (zoom-in, then pan-left, then zoom-out, …) so consecutive stills don't repeat the same motion. Zoom presets suit centered subjects (products, faces); pan presets suit wide scenery or tall screenshots. Only images support it — never set it on video, text, or audio.
+
+Three surfaces:
+- **Editor web copilot** — `editor_action` with `action_type=set_layer_media`, `ken_burns=<preset>` (`none` removes it), optional `ken_burns_intensity`. One call per image layer. You can also seed it on `add_layer` (kind=image) or `generate_layer` (media_type=image) so a freshly placed still arrives already animated.
+- **Editor UI (human)** — the Inspector's **Animate (Ken Burns)** dropdown on any selected image.
+- **Desktop agents (devcli)** — `vidfarm place ./work --src <img> --kind image --ken-burns zoom-in [--ken-burns-intensity 0.3]`, or fused with generation: `vidfarm generate image --prompt "…" --place ./work --ken-burns pan-left`.
+
+### Animated captions — word-by-word caption styles (TikTok/CapCut)
+
+Compositions support **first-class animated captions**: caption layers whose words animate one at a time, synced to the spoken audio, identical in the editor preview and both render paths. A caption *run* is one layer per cue (a page of ~3-5 words) on a shared track; each layer carries `data-caption-animation="<preset>"` and per-word `<span data-cap-word>` timing — no keyframe authoring, and the text stays editable like any text layer.
+
+**Preset styles** (each = an animation + a look; every field individually overridable): `spotlight` (active word gets a rounded highlight pill on bold outlined text — the classic Hormozi/CapCut look), `karaoke` (words fill with color as spoken and stay lit), `word-pop` (one word at a time, punchy scale-in), `stack-up` (words fade/rise in and accumulate), `neon` (active word glow pulse), `bounce` (active word bounces). Overridables: `caption_active_color` (spoken-word color), `caption_highlight_color` (spotlight pill), `caption_uppercase`, plus the normal text fields (`color`, `background`, `background_style`, `font_family`, `font_size`).
+
+**Word timings**: real word-level timestamps come from OpenAI STT (whisper-1); Gemini/OpenRouter transcripts get char-weighted estimated word windows, which still read well. Hand-typed text is auto-paged and estimated.
+
+Four surfaces:
+- **Editor web copilot** — for narration that needs TRANSCRIBING (a TTS voiceover layer, replaced audio, word-accurate karaoke), call the **captions primitive**: `POST /api/v1/primitives/audio/captions` `{ tracer, payload: { source_url: <narration URL>, style } }` — one job transcribes with word-level timings and returns `captions` (the cue array) plus `set_captions_action` (fully-formed `editor_action` arguments); apply verbatim with `editor_action action_type=set_captions`. For speech that is the decomposed source video's own audio, skip the job: derive cues from `video_context`'s timestamped segments and call `set_captions` directly with `captions=[{text,start,duration,words?}]`, or just `text` (+ `start`/`duration` window) to auto-page. Pick `caption_style`; the call **replaces** all existing animated caption layers, so it's also the "restyle everything" verb. Per-layer: `set_layer_style` with `caption_*` fields restyles one cue, `set_layer_text` rewrites its words, `caption_animation="none"` flattens back to static text. Never hand-build word-by-word captions from many `add_layer` calls.
+- **REST one-step (headless/automation)** — `POST /api/v1/compositions/:forkId/captions` `{ audio_url?|text?, style?, animation?, active_color?, highlight_color?, uppercase?, max_words_per_cue?, offset_sec?, provider?, language? }` transcribes the fork's own narration (auto-resolves: explicit `audio_url` → audio layer → backing video → source video) and writes the animated caption layers into the working composition **server-side** in one synchronous call — BYOK STT, non-billing. On a local `vidfarm serve` box, open editor tabs live-morph the result. Do **not** use it while a cloud editor session is open (the editor's next auto-save would overwrite it) — in-editor, mutate through `set_captions` instead.
+- **Editor UI (human)** — right-click → **Insert Captions** drops a starter cue; the Inspector's **Animated captions** picker switches styles on any selected text/caption layer, with active-word/highlight color and uppercase controls next to it.
+- **Desktop agents (devcli)** — `vidfarm captions generate ./work [--style spotlight]` transcribes the composition's own narration LOCAL-FIRST on your key (prefers OpenAI for real word timestamps) and writes the cue layers into `composition.html` on disk (a running `serve` live-morphs it). Alternates: `--audio <file|url>`, `--srt <file>` (e.g. from `vidfarm stt`), or `--text "<script>"`. Then `vidfarm captions style ./work --style karaoke` (restyle, same text/timing), `captions list`, `captions clear`.
+
+The plain STT primitive grew the same hook: `POST /api/v1/primitives/audio/transcribe` accepts `word_timestamps: true` to attach word-level timings to each transcript segment (real on an OpenAI key via whisper-1; other providers stay segment-level).
+
+### Script-free keyframe motion & timeline verbs
+
+Beyond the Ken Burns / transition / animated-caption presets, the copilot can hand-author motion and re-time layers directly (all `editor_action` verbs; all preview AND render identically):
+
+- **`set_layer_keyframes`** — author a script-free CSS `@keyframes` animation on ONE layer: `opacity`, `translate_x` / `translate_y` (% of the layer's own box), `scale`, and `rotate` (deg), keyed across the clip. Params: `layer_key`, `keyframes: [{ offset 0..1, opacity?, translate_x?, translate_y?, scale?, rotate? }]`, optional `keyframe_easing`, optional `keyframe_duration` (defaults to the clip's duration). This is the durable way to hand-craft motion beyond the Ken Burns / transition / caption preset vocabulary.
+- **`nudge_layers`** — relative timeline move: `layer_key` (or `layer_keys`) + `delta_start` (seconds) and/or `delta_track` (lanes). Group-aware.
+- **`ripple_edit`** — insert (`delta_start > 0`) or close (`delta_start < 0`) time at `at_time`, shifting every downstream clip.
+- **`trim_layer`** — move one edge: `edge=start|end` + `to_time` (seconds). A left trim also advances the media in-point for video/audio.
+- **`set_layer_zindex`** — restack: `z_order=front|back|forward|backward` (stacking == track index) or an explicit `track`.
+
+**Named layer-edit verbs (web `editor_action` ↔ devcli command parity).** Every fine edit has a devcli twin that mutates a pulled `composition.html` on disk (same `data-*` contract, so a human can finish it in the editor):
+
+| Intent | Web `editor_action` | devcli command |
+|---|---|---|
+| Rewrite a text/caption layer's words | `set_layer_text` | `vidfarm set-text <dir> --layer <k> --text "…"` |
+| Restyle text/color/typography (+ `line_height`, `letter_spacing`, `opacity`) | `set_layer_style` | `vidfarm set-style <dir> --layer <k> [--color --background --background-style --font-family --font-weight --italic --underline --text-align --font-size --border-radius --line-height --letter-spacing --opacity]` |
+| Geometry + constant opacity | `set_layer_visual` | `vidfarm set-visual <dir> --layer <k> [--x --y --width --height --font-size --border-radius --opacity]` |
+| Replace/edit a media layer **IN PLACE** (src swap + subrange + audio + fit + ken-burns + transitions) | `set_layer_media` | `vidfarm set-media <dir> --layer <k> [--src <url\|/raws/…\|file> --playback-start --source-out --duration --volume --muted --object-fit --object-position --ken-burns --transition --transition-out]` |
+| Set the stable slug/note handle | `set_layer_identity` | `vidfarm set-identity <dir> --layer <k> [--slug --note]` |
+| Clone a layer | `duplicate_layer` | `vidfarm duplicate <dir> --layer <k> [--start --track --new-key]` |
+| Cut a clip in two | `split_layer` | `vidfarm split <dir> --layer <k> --at <sec>` |
+| Set timing directly (absolute) | `set_layer_timing` | `vidfarm retime <dir> --layer <k> [--start --duration --track --playback-start]` |
+| **Canvas / theme** (whole composition) | `set_composition` | `vidfarm set-composition <dir> [--width --height --duration --background]` |
+
+`set-media` edits the media layer **in place** — the src swap keeps the same node/key/geometry/timing (the `set_layer_media` twin), so downstream references and the layer's DNA survive. Reach for `place --replace <layer_key>` only when you *want* a fresh clip that inherits the slot's timing+geometry (it deletes the node and mints a new key). Subrange is the web in/out picker in one gesture: `--playback-start 3 --source-out 8` = "use seconds 3–8 of the source" (sets the in-point + a 5s timeline duration). `--opacity` is a **constant** level (a ghosted underlay); a fade over time is `set_layer_keyframes` / `vidfarm keyframes`. `line_height`/`letter_spacing` reach typography the presets can't (condensed all-caps, tight/loose leading). `set-composition` resizes the frame (`--width`/`--height`), retargets total render length (`--duration`), or recolors the canvas (`--background`) — written on the composition root per the hyperframes contract. Every command takes `--json` and writes the file in place.
+
+**Web editor = declarative/CSS motion only.** In the web editor, motion is CSS-only. The JS runtime adapters (anime.js, GSAP, Lottie, Three.js, TypeGPU) are a **local-devcli-only** capability — they are stripped from the composition on save, and `replace_composition_html` now **rejects `<script>`-bearing HTML** in the web editor. On the web, author motion with CSS `@keyframes` (via `set_layer_keyframes` or a `<style>` block) plus the preset vocabulary (Ken Burns, transitions, animated captions). Scripted / adapter-driven compositions belong to the desktop `vidfarm serve` (or pulled-fork) flow, which renders them correctly through `vidfarm render`.
+
+**Enriched editor context.** The per-layer snapshot the copilot reads now includes each layer's `transition` / `transition_out` / `transition_duration`, its `ken_burns` preset, and `animation` (the custom CSS keyframe name it authored), and `recent_action_results` now reports successes (`ok: true` + a summary) as well as failures — so the model can see the effect of what it just did instead of only what broke. For exact markup it can also `GET /api/v1/compositions/:forkId/composition.html`.
+
+### Local file paths as media (skip the S3 upload)
+
+`vidfarm place --src` (and `approve --video/--media`) accept a **local file path**, not just a URL — so a power user bulk-building compositions from a folder of clips on their own machine never has to upload every asset to Vidfarm storage:
+
+- **On a `vidfarm serve` box (the free bulk workflow)** — `place`'s target composition lives under `<data-dir>/storage/compositions/forks/<forkId>/working/`, so a local `--src` file is **copied straight onto that box's own disk store** (`.vidfarm-local/storage/users/local-media/…`) and referenced by the box's own `http://localhost:3000/storage/…` URL. **Nothing touches S3.** The in-process local renderer fetches it over localhost for free. This is the intended path for bulk-generating many videos from local assets without bloating your durable library.
+  ```bash
+  # serve box running on :3000 — reference clips straight off disk, render locally, free
+  vidfarm place ./.vidfarm-local/storage/compositions/forks/<forkId>/working \
+      --src ~/footage/hook-042.mp4 --at 0
+  vidfarm render <forkId>            # local, $0.00
+  ```
+  If you ran `serve --port <N>`, pass `--base-url http://localhost:<N>` so the reference URL points at the right origin. A localhost `/storage` URL only resolves against the serve box that holds the file, so these compositions render/preview **locally**; to render in the cloud or share them, upload the media instead (below).
+- **Anywhere else (e.g. a `vidfarm pull` dir headed for cloud render/publish)** — the cloud renderer can't reach your disk, so a local `--src` file is **uploaded to the ephemeral temp store** first, namescoped under a throwaway `temp/` folder by default (override with `--folder`). Use this when the output must be a durable, shareable URL.
+
+**`/temp` folder convention.** When you *do* need to upload throwaway media to Vidfarm (cloud render, `approve`, one-off URLs), keep it under a dedicated `temp/` folder — `vidfarm upload clip.mp4 --folder temp`, `vidfarm place … --folder temp`, `vidfarm approve … --folder temp` — so scratch assets stay quarantined in one place you can periodically purge, instead of cluttering your persistent My Files library or the temp-store root.

package/.agents/skills/vidfarm-director/references/onboarding.md

@@ -0,0 +1,28 @@
+## Getting started / onboarding a new director
+
+Only run this flow when the director signals they **don't know where to start** ("getting started", "help me set up", "I don't know where to begin"). If they already know what they want, skip straight to it — never force onboarding. Directors can also jump to any single step (e.g. "just find me a good template for X" → go straight to template discovery).
+
+The point of onboarding is to build **durable, reusable context** in My Files, namescoped under the director's product/offer folder, so future chats and agents can read it back. Save each artifact as Markdown with `browse_files` write (web copilot) or `vidfarm put-file` (devcli):
+
+1. **Product context** → `About.md` (basic offer/product) or a deeper `Interview.md` if they want depth. Drive the interview questions with `brainstorm/coldstart`.
+2. **Awareness level** (Eugene Schwartz — problem-aware, solution-unaware, …) → `awareness-levels.md`. If it's genuinely unknown after thinking it through, note that ads for **every** level should be made and tested. Use `brainstorm/awareness_stages`.
+3. **Persuasive angles** → `persuasive-angles.md`, via `brainstorm/angles`.
+4. **Hooks** → `ad-hooks.md`, via `brainstorm/hooks`.
+5. **Brand assets & demos** — ask if they have logos/mascots/themes (suggest a `/brand-assets/` folder, e.g. `/brand-assets/logo.png`) or product demos / screen recordings (suggest a `/product-demos/` folder). `browse_files list` / `vidfarm files` first to see what they already uploaded; filenames should be descriptive and every asset worth finding later should get **notes** (`annotate-file` / `browse_files annotate`) so `files --search` works months from now. If they have a recurring character/mascot, set up its `/files/characters/<slug>/` trio now — `<character_id>.json` (e.g. `character_zara.json`) + `character_sprite_card.png` + `character_about.md` (see "Recurring characters are first-class").
+6. **Budget** — ask roughly what they want to spend per video, and map it to the Cost spectrum (free reuse+local render → pennies for cloud render → ~$1 for some AI scenes → $10+ for heavy AI gen). This sets which approach you default to and whether AI **video** generation is on the table (ask permission before using it; image gen is cheap and fine). Budget can also be revisited per editor project.
+7. **Recommend & adapt a template** — pair what you now know about the offer against the decomposed template catalog (`GET /discover/feed?q=<offer>`, read each result's `promotions`/`keywords`/`summary`), recommend the best 3-6, then fork and **modify** the winner to fit their offer. Prefer already-decomposed templates so the director skips the ~$0.10 decompose cost.
+
+**Assume multiple offers.** My Files is multi-offer (see the My Files section) — namescope every onboarding artifact under the right product/offer/region folder (`acme-skincare/About.md`, not a bare `About.md`) so one brand's context never bleeds into another's.
+
+## Default assistance pattern
+
+When a director asks "make me a video", the default sequence is:
+
+1. Ask what source material and what template style they want
+2. Discover a matching template with `GET /discover/feed`
+3. Fork it
+4. If they want an entirely new source video, guide them to `auto-decompose` first
+5. Open the Trackpad Editor and let them make edits
+6. Publish and share
+
+Prefer specific templates over primitives when a template exists that already captures the desired production pattern.