@aicopycoders/exodus 2026.7.200-beta.7 → 2026.7.200-beta.9

package/assets/references/lipsync-alternatives.md

@@ -1,217 +0,0 @@
-# Lip-Sync Alternatives — getting a CHOSEN voice out of the on-screen actor's mouth
-
-**Research date:** 2026-07-14
-**For:** Unblocking voice-paired talking-head scene clips after Kie's `gemini-omni-video` `audio_ids` path started returning "flagged by Website as violating content policies" on every render (GitHub issue #561).
-**Goal:** A chosen voice — ideally an ElevenLabs voice the user already uses — spoken by the on-screen actor, lip-synced, in each per-scene clip (usually 9:16, 720p, 4–10 s), then stitched as today.
-**Method:** Primary sources = first-party API docs (ai.google.dev, sync.so/docs, fal.ai model pages, developers.heygen.com, d-id.com, hedra.com, replicate.com) plus Kie's own pages. Kie's marketing pages (`kie.ai/*`) and `kie.ai/market` return **HTTP 403 to automated fetch**; several `docs.kie.ai/market/...` model pages 404 at the guessed path, so some Kie-specific slugs/prices are corroborated from mirror hosts (fal, WaveSpeed, Replicate, Atlas Cloud) that run the same upstream model and are flagged accordingly. Anything not pinned to a first-party doc in this session is marked **[unverified]**.
-**Confidence labels:** `[docs]` = first-party API doc · `[mirror]` = same upstream model on another host's doc (fal/Replicate/WaveSpeed) · `[vendor]` = vendor marketing/help page · `[unverified]` = could not confirm from a primary source this session.
-
----
-
-## TL;DR — recommendation table
-
-The two integration shapes matter more than the model list:
-
-- **RETARGET** = feed an *already-rendered* scene clip (video) + an audio track → model reanimates only the mouth, preserves background/body/camera. **Drops into the existing per-scene render → stitch flow as a new post-render step. No change to the video model.** This is the fit for our pipeline.
-- **GENERATE** = feed an *image* (a portrait / first frame) + audio → model animates a talking head from scratch. This **replaces** the Kie video model for talking scenes (loses the motion/scene the video model produced).
-
-| Rank | Path | Shape | Existing BYOK? | ~Cost, 9:16 720p, ~8 s | Voice source | Moderation risk | Verdict |
-|---|---|---|---|---|---|---|---|
-| **1** | **Kie `volcengine/video-to-video-lip-sync`** (retarget rendered clip) | RETARGET | ✅ **Kie BYOK exists — same jobs API** | not documented — surfaces via `creditsConsumed` `[docs]` | any audio file (our Voiceover node's ElevenLabs MP3s, ≤10 MB) | Low (edits our own footage) — **live-probe before building** | **Zero new provider; verified post-research (see addendum). Try first.** |
-| **2** | **sync.so `lipsync-2`** (retarget rendered clip) | RETARGET | ❌ new provider (but built-in ElevenLabs TTS) | ~**$0.32** ($0.04/s) `[docs]` | video's own audio **or** ElevenLabs voiceId+script native in-API | Low (edits our own footage) | **Best quality reputation; fallback if Volcengine quality/pricing disappoints** |
-| **2b** | **Kling Lip-Sync A2V** (`kwaivgi/kling-lip-sync`, retarget) via fal/Replicate | RETARGET | ❌ new provider (NOT in Kie's catalog — verified via docs index) | ~**$0.11–0.15**/run ($0.014/input-s, 5 s increments) `[mirror]` | any audio file (drop in ElevenLabs MP3) | Low | Cheapest retarget, but needs fal/Replicate BYOK |
-| 3 | **fal `veed/lipsync`** / **`fal-ai/latentsync`** (retarget) | RETARGET | ❌ new provider | veed ~$0.24 ($0.15/5 s); latentsync **$0.20 flat ≤40 s** `[docs]` | any audio file | Low | Cheap retarget fallbacks; latentsync is the price floor |
-| 4 | **Kie InfiniteTalk** (`infinitalk/image-to-talking-video`) | GENERATE | ✅ **Kie BYOK exists** | ~**$0.30**/clip (10 credits/5 s @720p) `[vendor]` | uploaded audio (ElevenLabs MP3), ≤15 s | Low–med | **Only option that reuses existing BYOK — but replaces the video model, image-driven** |
-| 5 | **Kie OmniHuman 1.5** (`bytedance/omni-human`) | GENERATE | ✅ Kie BYOK exists | ~$0.12/s upstream `[mirror]` | uploaded audio | Low–med | Higher-fidelity generate-from-photo on existing BYOK |
-| 6 | Hedra Character-3 | GENERATE | ❌ new provider | subscription credits, ~$15–75/mo | uploaded audio / text+TTS | Low | Best-in-class generate quality; sub model, not per-clip PAYG |
-| 7 | HeyGen / D-ID photo-avatar | GENERATE | ❌ new provider | ~$0.05/s ($3/min) `[vendor]` | uploaded audio or TTS | Low | Presenter/UGC avatars; least "our footage" |
-| — | **Google Gemini Omni direct** (bypass Kie) | GENERATE, **no audio input** | ❌ new provider | ~$0.10–0.75/s (Veo tiers) `[docs]` | **none — cannot supply a voice** | n/a | **Ruled out: API explicitly forbids audio upload & voice editing** |
-| — | Seedance `reference_audio_urls` (Kie) | (not lip-sync) | ✅ Kie | — | voice-style *reference*, not spoken content | — | **Not a lip-sync/voice-content lever — see note** |
-
-**Bottom line:** the goal ("chosen ElevenLabs voice, lip-synced, on the actor already in our rendered clip") is a **RETARGET** problem, and the video-generation APIs (including Gemini Omni direct) don't solve it. Rank 1–3 keep the Kie video model exactly as-is and add one post-render lip-sync call per scene, then mux/stitch. **sync.so is the standout** because it retargets our rendered clip *and* has ElevenLabs TTS built into the same request. If avoiding a brand-new provider is paramount, Kie InfiniteTalk (rank 4) is the only path on existing BYOK — but it's generate-from-image, so it changes what the talking scenes look like.
-
----
-
-## Branch 1 — Google Gemini / Omni video API directly (bypass Kie)
-
-**Source (first-party):** https://ai.google.dev/gemini-api/docs/omni · pricing https://ai.google.dev/gemini-api/docs/pricing
-
-### Does it accept audio input for the generated speech / lip-sync? **No.**
-The Gemini Omni video docs state plainly `[docs]`:
-- *"Uploading audio references is unsupported in the current version of the API."*
-- *"Voice editing is not supported."*
-
-The documented request accepts only: **text** (`{"type":"text","text":"…"}`), **images** (`{"type":"image","data":<base64>,"mime_type":"image/jpeg"}`), **videos for editing** (`{"type":"document","uri":<video_file_uri>}`), and **prior generated-video references** via a `previous_interaction_id` parameter. **There is no audio input parameter anywhere in the schema.** `system instructions`, `temperature`, `top_p`, and `negative prompts` are also unsupported (negatives must be embedded in the prompt). `[docs]`
-
-**Consequence:** the first-party API **cannot** take an ElevenLabs voice, an uploaded audio file, or a voice character to drive speech/lip-sync. It generates whatever voice it wants natively. This is the same capability gap that Kie's `audio_ids` param was papering over — and going direct removes the param entirely. **Branch 1 does not meet the goal.**
-
-### Moderation — and why Kie's flag is Google-side
-The Omni docs document these safety controls `[docs]`:
-- *"Content safety filters are applied to both input prompts and generated video (and depend on your region)."*
-- *"Prompts that violate usage policies will be blocked."*
-- *"Uploading and editing images containing certain recognizable people is not supported."*
-- *"Uploading and editing images containing minors is not supported in [the] EEA, Switzerland, and the United Kingdom."*
-- English is *"fully supported"*; other languages *"have not been evaluated."*
-
-The "flagged by **Website**" wording our probes saw, combined with "prompts that violate usage policies will be blocked" applying to the *underlying Gemini/Omni model*, is consistent with a **Google-side content-policy block on the audio/voice pathway**, not a Kie quirk — Kie's `audio_ids` (preset voice characters) likely routes into a voice feature Google's safety layer rejects for this prompt class, or Kie is bolting on a voice step the first-party model doesn't sanction. Since the first-party API forbids audio input outright, there is **no direct-API config that re-enables voice-paired renders**. `[docs]` + reasoned inference.
-
-### Access, pricing, availability
-- **Video models on the Gemini API are the Veo family**, billed per second of output. Google's list rate for **Veo 3.1** is **$0.75/s** (~$6.00 / 8 s clip) on the Gemini API and Vertex; **Veo 3.0** was $0.50/s video-only / $0.75/s with audio (deprecated, shutdown 2026-06-30). A **standard-tier 720p** path bills ~**5,792 tokens/s of 720p video ≈ $0.10/s** [unverified exact tier mapping]. `[docs]` (pricing page) + `[vendor]` calculators.
-- **Vertex AI vs Gemini API:** same Veo models, same "no audio upload" capability for generation; Vertex adds enterprise auth/quota but does **not** add an audio-input lip-sync mode [unverified beyond Veo parity].
-- **vs Kie:** Kie's ~63 credits (~$0.32) per 4 s 720p clip is competitive with — often cheaper than — Google-direct Veo, and Kie's unified async job API is already integrated. Going direct means a **new "Google AI" BYOK provider**, a Files API upload step for image inputs, and Google's own async long-running-operation polling — **material new surface for zero gain on the voice goal.**
-
-**Branch 1 verdict: ruled out.** The capability we need (supply-your-own-voice / lip-sync to given audio) is explicitly not in the first-party API. Do not build a Google-direct integration to solve #561.
-
----
-
-## Branch 2 — lip-sync-to-audio models (render clip → ElevenLabs MP3 → retarget the mouth)
-
-Ordered by fit. **RETARGET** models take our rendered clip; **GENERATE** models take a still image.
-
-### 2a. sync.so (Sync Labs) — `lipsync-2` / `lipsync-2-pro` / `sync-3` — **RETARGET** ✅ best fit
-**Source (first-party):** https://sync.so/docs/api-reference/api/generate-api/create · https://sync.so/docs/models/lipsync · https://sync.so/docs/introduction
-
-- **Shape:** retargets an **existing video** with new audio — *"retargets existing video with new audio, applying lipsync synchronization to match speaker mouth movements."* `[docs]`
-- **Inputs (`CreateGenerationDto`):** an array requiring **exactly one visual input** (video `url` **or** `assetId`; image also accepted) **and one audio-or-text input** `[docs]`:
-  - Audio as `{ "url": … }` or `assetId`, **or**
-  - **TTS**: `{ "provider": "elevenlabs", "voiceId": <ElevenLabs voice id>, "script": <text>, "stability", "similarityBoost" }` — **native ElevenLabs integration in the same request.** `[docs]`
-- **Models:** `sync-3`, `lipsync-2`, `lipsync-2-pro`, `lipsync-1.9.0-beta`, `react-1`. `[docs]`
-- **Options:** `sync_mode` (bounce/loop/cut_off/silence/remap for duration mismatch), `temperature` (0–1, default 0.5, expressiveness), `active_speaker_detection`, `occlusion_detection_enabled`, `webhookUrl`. `[docs]`
-- **API pattern:** **async** — POST returns `status: PENDING` → `PROCESSING` → `COMPLETED|FAILED|REJECTED`; poll or webhook. Long videos auto-chunked into 30–40 s segments (irrelevant for our 4–10 s clips). `[docs]`
-- **Pricing:** **`lipsync-2` = $0.04/s**, `sync-3` = $0.133/s (at 25 fps). ~**$0.32 for an 8 s clip** on lipsync-2. `[docs]` (pricing page / cost-estimation endpoint available).
-- **Moderation:** `REJECTED` state exists for policy violations [unverified what triggers it]; editing our *own* generated footage is far lower risk than generating a likeness.
-- **Fit:** ⭐ Perfect. Keep the Kie video model; after each scene renders, call sync.so with `video_url = <rendered clip>` and either the ElevenLabs MP3 we already generate **or** `provider:elevenlabs, voiceId, script` and skip our own TTS step. Output replaces the scene clip pre-stitch.
-
-### 2b. Kling Lip-Sync — Audio-to-Video (`kwaivgi/kling-lip-sync`) — **RETARGET** ✅ cheap
-**Sources:** fal API doc https://fal.ai/models/fal-ai/kling-video/lipsync/audio-to-video/api `[docs]` · Replicate https://replicate.com/kwaivgi/kling-lip-sync `[mirror]` · WaveSpeed https://wavespeed.ai/models/kwaivgi/kling-lipsync/audio-to-video `[mirror]`
-
-- **Shape:** *"give it a clean voice track plus a video and it reanimates the mouth region… preserves the original background, body motion, and camera work."* Works on Kling-generated **or other-source** video. `[mirror]`
-- **Inputs:** `video_url` + `audio_url` (also a text→TTS variant). `[docs]`
-  - **Video:** .mp4/.mov, ≤100 MB, **2–10 s** (some hosts list 2–60 s), **720p/1080p only**, width/height 720–1920 px. `[docs]`
-  - **Audio:** 2–60 s, ≤5 MB, MP3/WAV/AAC/OGG. `[docs]`
-- **API pattern:** async queue + webhooks. `[docs]`
-- **Pricing:** ~**$0.15/run minimum**, or **$0.014 per input-video-second rounded to the nearest 5 s** → ~$0.11 for a 4–8 s clip. `[mirror]` (WaveSpeed/fal)
-- **Moderation:** marked "Commercial use / Partner"; no documented person-restriction policy. `[docs]`
-- **Kie hosting:** **[unverified]** — Kie hosts multiple Kling models (`kling-3.0/video` etc.) but I could **not** confirm this session that Kie exposes the *video-to-video* `kling-lip-sync` retarget slug (its `docs.kie.ai/market/kling/lip-sync` path 404'd). **If BYOK-Kie already exposes it, this becomes the top pick** (existing provider + cheap + retarget). Worth a live probe against the Kie account.
-- **Fit:** ⭐ Excellent and cheapest solid retarget. 10 s cap suits our clip length.
-
-### 2c. fal.ai catalog (retarget options)
-**Source:** https://fal.ai/models/veed/lipsync · https://fal.ai/models/fal-ai/latentsync · https://fal.ai/models/fal-ai/sync-lipsync
-- **`veed/lipsync`** — RETARGET (video→video), *"realistic lipsync from any audio."* **$0.15 / 5 s** (~$0.24/8 s). `[docs]`
-- **`fal-ai/latentsync`** (ByteDance LatentSync, audio-conditioned latent diffusion) — RETARGET. **$0.20 flat for ≤40 s**, then $0.005/s → cheapest for our clip lengths. Real-life + anime. `[docs]`
-- **`fal-ai/sync-lipsync` (1.9)** — RETARGET, **$0.70/min ≈ $0.012/s** (sync.so model, older). `[docs]`
-- **`fal-ai/kling-video/lipsync/audio-to-video`** — the Kling retarget above, on fal. `[docs]`
-- **Fit:** all RETARGET, all take our rendered clip + any audio (drop the ElevenLabs MP3). fal is a single BYOK that unlocks several — good hedge if we want to A/B lip-sync engines. latentsync is the price floor; veed is a quality step up.
-
-### 2d. Kie InfiniteTalk (`infinitalk/image-to-talking-video`) — **GENERATE** ✅ only path on existing BYOK
-**Sources:** https://kie.ai/infinitalk `[vendor]` (403 to fetch; corroborated) · integration guide https://www.fromdev.com/2025/11/how-to-integrate-infinite-talk-api-using-kie-ai-a-step-by-step-guide.html · FairStack model index (price)
-- **Shape:** MeiGen-AI InfiniteTalk — turns an **image + audio** into a talking avatar (sparse-frame video-dubbing framework, identity-preserving, "infinite" length). **Generate-from-image, not retarget-a-clip.** `[vendor]`
-- **Inputs (via `POST https://api.kie.ai/api/v1/jobs/createTask`):** `image_url` (JPEG/PNG/WebP), `audio_url` (MP3/WAV/AAC/OGG, **≤15 s**), `prompt`, `resolution` **480p or 720p only**, `seed` (10000–1000000). `[mirror]` (integration guide reflecting Kie's schema)
-- **Pricing:** first 5 s = **5 credits @480p / 10 @720p / 15 @1080p**, then per-second at the same rate → ~**$0.30/clip @720p** (FairStack lists "InfiniteTalk (Kie.ai) — $0.300/clip"). `[vendor]`
-- **Fit:** ✅ **Only option that reuses the existing Kie BYOK key.** BUT it's image-driven: we'd feed a scene *frame* (or actor portrait) + the ElevenLabs MP3, and it **regenerates** the talking head — we lose whatever motion/scene the Kie video model produced. Good if talking scenes can be "portrait talks to camera"; wrong if the scene needs the video model's action. 15 s audio cap fits our 4–10 s clips.
-
-### 2e. Kie OmniHuman 1.5 (`bytedance/omni-human`) — **GENERATE**
-**Sources:** https://kie.ai/omnihuman-1-5 · https://kie.ai/omni-human-api `[vendor]` (403; corroborated) · fal/upstream pricing `[mirror]`
-- **Shape:** ByteDance OmniHuman 1.5 — single portrait **image + audio** → realistic talking video with natural lip-sync, expressions, lifelike/full-body motion. Generate-from-image. `[vendor]`
-- **Pricing:** upstream ~**$0.12/s** (fal listing for OmniHuman). Kie bills by generated duration; exact Kie credit/s [unverified]. `[mirror]`
-- **Fit:** Higher-fidelity generate-from-photo than InfiniteTalk, on existing Kie BYOK. Same caveat: replaces the video model for talking scenes. Kling AI Avatar 2.0 is a third Kie generate-from-image option (audio-driven, multilingual, up to 5 min; Kie benchmarks it above OmniHuman-1.5 and HeyGen). `[vendor]`
-
-### 2f. Runway (Act-Two / Lip Sync / Characters API)
-**Sources:** https://help.runwayml.com/hc/en-us/articles/42311337895827 · https://runwayml.com/apps/add-dialogue · changelog `[vendor]`
-- **Act-Two:** performance-capture (driving video → target character), auto audio-visual sync; you can **change the voice** within the interface. **Lip Sync** app extended to **45 s** (from 20 s). Characters API is **live for developers at dev.runwayml.com**. `[vendor]`
-- **Shape:** closest Runway analog to retarget is Lip Sync / "Add Dialogue" (animate a video/image with a voice). API is available (Gen-4/Characters on dev.runwayml.com) but **per-second API pricing for Act-Two/Lip Sync was not published** this session. `[vendor]` / [unverified pricing]
-- **Fit:** Plausible RETARGET/talking-photo path, but pricing opacity and a heavier API make it a second-tier choice vs sync.so/fal/Kling.
-
-### 2g. Hedra Character-3 — **GENERATE**
-**Sources:** https://www.hedra.com/docs/pages/getting_started/quickstart · https://www.hedra.com/video-models/hedra-character-3 · https://www.hedra.com/plans `[vendor]`
-- **Shape:** omnimodal — processes **image + text + audio simultaneously** → talking avatar with strong lip-sync, micro-expressions (blinks, eye shifts, head tilts), full-body motion. Generate-from-image. `[vendor]`
-- **Access/pricing:** requires a **paid subscription** (API key + purchased API credits). Free (100 cr, watermarked); Basic $15/mo (1,500 cr); Creator $30/mo (5,400 cr); Professional $75/mo (14,400 cr). **Credit/subscription model, not clean per-clip PAYG.** `[vendor]`
-- **Fit:** Best-in-class *generate* quality for talking heads, but subscription billing fits a per-user BYOK-PAYG pipeline poorly, and it's still generate-from-image (replaces the video model).
-
-### 2h. HeyGen API / D-ID — **GENERATE** (presenter avatars)
-**Sources:** https://developers.heygen.com/ · https://help.heygen.com/en/articles/10060327 · https://www.d-id.com/pricing/api/ `[vendor]`
-- **HeyGen:** Photo Avatar from a still → animates face, **syncs lips to your script or an uploaded audio file** ("dub or replace audio on a video with a provided audio file"). Pay-as-you-go since Feb 2026: **~$1 = 1 min @720/1080p**; engines from **$0.0167/s** (Avatar III Digital Twin) to **$0.05/s** (Avatar IV Photo Avatar) to $0.0667/s. Min top-up $5. `[vendor]`
-- **D-ID:** talking-head from a photo + audio/TTS, API **~$0.05/s ($3/min)**. `[vendor]`
-- **Fit:** These are *presenter/spokesperson* generators from a photo — they don't retarget our rendered scene, and outputs read as "avatar tool," which is off for naturalistic UGC scene action. Use only if a talking-portrait scene is acceptable and we want a polished presenter look. HeyGen's "replace audio on a video" mode is worth a second look as a possible retarget-ish path [unverified whether it accepts arbitrary external video].
-
-### 2i. Replicate-hosted options
-**Source:** https://replicate.com/collections/lipsync `[docs]`
-Replicate's lipsync collection hosts the same upstream models as first-party, each with a documented JSON schema and PAYG billing: **`sync/lipsync-2-pro`**, **`kwaivgi/kling-lip-sync`** (retarget), **`bytedance/omni-human`** (generate), **`pixverse-ai/lipsync`**, **`zsxkib/multitalk`** (multi-person conversational lipsync — multiple audio clips → back-and-forth). Good single-BYOK hedge (one Replicate key → several retarget + generate engines) but generally pricier/less predictable than fal for the same models.
-
----
-
-## Seedance `reference_audio_urls` note (Kie `bytedance/seedance-2`)
-
-Per Kie's Seedance 2.0 doc (already captured in `native-audio-dialogue.md`, source https://docs.kie.ai/market/... Seedance page): `reference_audio_urls` accepts **≤3 clips, 2–15 s, wav/mp3** and is a **voice/style *reference* lever — NOT a documented dialogue-content or lip-sync-to-given-audio field.** `[docs]` It biases the *character/timbre* of the model's natively-generated speech; it does **not** make the model speak a supplied MP3 verbatim, and it does not lip-sync to an external audio track. **It does not solve the goal** (a chosen ElevenLabs voice saying our exact lines, lip-synced). Treat Seedance native audio as "model invents a voice, loosely style-guided," same class of problem as gemini-omni — not a voice-control solution.
-
----
-
-## Recommended path + integration sketch
-
-### Primary recommendation: add a post-render **RETARGET lip-sync step**, keep the Kie video model
-The pipeline already produces per-scene clips and per-scene ElevenLabs MP3s (George default) — they're just not muxed/synced. The clean fix is a new stage between "scene clip rendered" and "stitch":
-
-```
-storyboard → [Kie video model renders scene clip (silent or discard native audio)]
-           → [ElevenLabs TTS: scene.dialogue → MP3]  (already exists)
-           → [LIP-SYNC RETARGET: clip + MP3 → mouth-synced clip]   ← NEW
-           → [ffmpeg stitch]  (already exists, now with real synced VO)
-```
-
-This is **additive** — no change to the storyboard, the video-model picker, or the stitch stage — and it finally makes the existing ElevenLabs Voiceover node pay off.
-
-**Engine choice, in order:**
-1. **sync.so `lipsync-2`** — retargets the rendered clip and can take **either** our ElevenLabs MP3 **or** an inline `provider:elevenlabs, voiceId, script` TTS block (letting us drop our own TTS call). Async job pattern mirrors Kie's (create → poll/webhook), so it slots into the existing async render machinery. ~$0.32/8 s clip. New BYOK: a sync.so API key. **Start here.**
-2. **Before committing to a new provider, live-probe whether BYOK-Kie already exposes `kwaivgi/kling-lip-sync` (video-to-video).** If yes, it's the cheapest retarget (~$0.11–0.15/clip) on the *existing* Kie key — zero new provider surface. `[unverified]` this session; a 1-call probe settles it.
-3. **fal (`veed/lipsync` or `fal-ai/latentsync`)** as a fallback/second engine — one fal BYOK unlocks several retarget models for A/B'ing quality; latentsync at $0.20/≤40 s is the price floor.
-
-### If a new provider is a hard no (must stay on existing Kie/ElevenLabs BYOK only)
-Use **Kie InfiniteTalk** (`infinitalk/image-to-talking-video`) or **OmniHuman** (`bytedance/omni-human`): feed a **scene frame/portrait + the ElevenLabs MP3** → talking clip, on the existing Kie key (~$0.30/clip @720p, 720p cap, ≤15 s audio for InfiniteTalk). **Accept the tradeoff:** these *generate* the talking head from a still, so talking scenes become "portrait speaks to camera" and lose the video model's motion/action. Fine for direct-address UGC beats; wrong for scenes that need staged action.
-
-### Integration considerations (both paths)
-- **Async + polling/webhook**, same as the current Kie render loop — reuse it. sync.so and Kie share the create→poll shape; sync.so adds a cost-estimation endpoint.
-- **Resolution/duration:** retarget models cap at 720p/1080p and ~10 s (Kling) or auto-chunk (sync.so) — fine for our 4–10 s 9:16 720p clips. InfiniteTalk caps at 720p + 15 s audio.
-- **Cost delta:** retarget adds ~$0.04/s (sync) or ~$0.014/s (Kling) *on top of* the ~$0.32/4 s Kie render — call it +$0.15–0.35/scene. A 6-scene ad adds roughly $1–2.
-- **Moderation:** retargeting **our own rendered footage** is materially lower policy-risk than the gemini-omni `audio_ids` path that #561 hit, and far lower than uploading a real person's likeness — the whole reason this route dodges the block.
-- **Voice fidelity:** all retarget engines take an arbitrary MP3, so the **user's exact ElevenLabs voice** is preserved end-to-end (we generate it, they lip-sync to it) — directly satisfying "a chosen ElevenLabs voice out of the actor's mouth."
-
----
-
-## Source list (primary first)
-- Gemini Omni API (no audio input, safety): https://ai.google.dev/gemini-api/docs/omni
-- Gemini/Veo pricing: https://ai.google.dev/gemini-api/docs/pricing
-- sync.so create-generation schema: https://sync.so/docs/api-reference/api/generate-api/create · models https://sync.so/docs/models/lipsync · intro https://sync.so/docs/introduction
-- fal Kling lipsync A2V: https://fal.ai/models/fal-ai/kling-video/lipsync/audio-to-video/api
-- fal veed/lipsync: https://fal.ai/models/veed/lipsync · latentsync: https://fal.ai/models/fal-ai/latentsync · sync-lipsync: https://fal.ai/models/fal-ai/sync-lipsync
-- Replicate lipsync collection: https://replicate.com/collections/lipsync · kling-lip-sync: https://replicate.com/kwaivgi/kling-lip-sync
-- WaveSpeed Kling lipsync (specs/price): https://wavespeed.ai/models/kwaivgi/kling-lipsync/audio-to-video
-- Kie InfiniteTalk: https://kie.ai/infinitalk · integration guide: https://www.fromdev.com/2025/11/how-to-integrate-infinite-talk-api-using-kie-ai-a-step-by-step-guide.html
-- Kie OmniHuman: https://kie.ai/omnihuman-1-5 · https://kie.ai/omni-human-api
-- Hedra Character-3: https://www.hedra.com/video-models/hedra-character-3 · quickstart https://www.hedra.com/docs/pages/getting_started/quickstart · plans https://www.hedra.com/plans
-- HeyGen API: https://developers.heygen.com/ · pricing https://help.heygen.com/en/articles/10060327-heygen-api-pricing-explained
-- D-ID API pricing: https://www.d-id.com/pricing/api/
-- Runway Act-Two: https://help.runwayml.com/hc/en-us/articles/42311337895827-Performance-Capture-with-Act-Two · add-dialogue https://runwayml.com/apps/add-dialogue
-- Seedance `reference_audio_urls`: see workspace/references/native-audio-dialogue.md (Kie Seedance 2.0 doc)
-
----
-
-## Post-research verification addendum (main session, 2026-07-14)
-
-Two load-bearing claims were re-verified against primary sources after the research pass:
-
-### Kie DOES host a video-to-video lip-sync retarget model — `volcengine/video-to-video-lip-sync` `[docs]`
-Found via Kie's docs index (https://docs.kie.ai/llms.txt); full doc: https://docs.kie.ai/market/volcengine/video-to-video-lip-sync.md
-
-- **Shape:** RETARGET — required inputs are `mode` (`lite` = single-person frontal, faster; `basic` = complex scenes + scene segmentation), `video_url`, `audio_url`. Runs on the SAME `POST /api/v1/jobs/createTask` transport the pipeline already speaks, on the existing Kie BYOK key.
-- **Fit:** our scene clips are single-person frontal talking heads (→ `lite`); audio accepts MP3 ≤10 MB — exactly what the existing Voiceover node's ElevenLabs synthesis produces per scene. `align_audio` loops video if audio outruns it; output is mp4 25 fps with duration matched to the audio.
-- **Video constraints:** 360p–1080p (ours: 720p ✓), H.264, ≤500 MB, 24–60 fps.
-- **Unknowns:** pricing is NOT documented (Kie convention: real cost surfaces via `creditsConsumed` on the record — live-probe it) and output quality is unassessed. Given the #561 surprise, run a live probe (rendered clip + ElevenLabs MP3) before building the pipeline leg.
-- This supersedes the earlier "[unverified] Kie hosting of Kling lip-sync" row: Kie's docs index shows NO Kling A2V lip-sync slug — the Kling entries are `kling/ai-avatar-standard|pro` (GENERATE-from-image), plus `omnihuman-1-5` and `infinitalk/from-audio` (both GENERATE).
-
-### sync.so ElevenLabs TTS input — confirmed `[docs]`
-https://sync.so/docs/api-reference/api/generate-api/create — generation request accepts `{type: "text", provider: {name: "elevenlabs"}, voiceId: <ElevenLabs voice id>, script}` alongside a `{type: "video", url}` retarget input; models `sync-3`, `lipsync-2`, `lipsync-2-pro`, `lipsync-1.9.0-beta`, `react-1`. (Note: our pipeline already synthesizes ElevenLabs MP3s in the Voiceover node, so the built-in TTS is a convenience, not a requirement — which strengthens the case for trying Kie/Volcengine first.)
-
-### Revised recommendation
-1. **Live-probe Kie `volcengine/video-to-video-lip-sync`** (existing BYOK, existing transport, existing ElevenLabs VO): render one talking-head clip (native audio OFF, mouth policy suppressed), synthesize one ElevenLabs line, submit lite-mode retarget, judge quality + read `creditsConsumed`.
-2. If quality or price disappoints → **sync.so `lipsync-2`** (new BYOK provider, ~$0.04/s, strongest reputation).
-3. Integration shape either way: Video node gains a "voiceover lip-sync" mode = per-scene silent-ish render → Voiceover MP3 → retarget call → retargeted clip replaces the raw clip in the stitch. The storyboard's speech-fit budget already paces voText to scene durations.

package/assets/references/meta-ads-mcp-probe.md

@@ -1,167 +0,0 @@
-# Meta Ads MCP — live probe findings (wayfinder #563)
-
-Probed 2026-07-14 against Brad's real accounts via the **official Meta Ads MCP**
-(`https://mcp.facebook.com/ads`, remote HTTP MCP, OAuth-in-browser). All calls
-read-only. Primary probe account: **Matt Beard FPI** (act `…820395`, ~$62.9k
-spend / 1.07M impressions in the trailing 90 days). Feeds the enrichment-path
-ticket (#565) and the bridge contract (#566) on map #562.
-
-## TL;DR for the specs
-
-1. **The winner query works end-to-end**: per-ad `amount_spent`, `results`
-   (labeled with the result type), `cost_per_result`, `ctr`, `cpm`,
-   `impressions`, `lead`, `purchase_roas` over any date preset or custom
-   range, sortable by any metric. A "top ads by spend, then threshold on
-   cost-per-result" winner definition is executable **agent-side over sorted
-   results** — server-side *metric filtering* consistently 500s (see gotchas).
-2. **The Ad Library cross-reference exists but is fuzzy**: `ads_library_search`
-   by `page_ids` returns the public **`adArchiveId` directly** (the result `id`;
-   `ad_snapshot_url` = `facebook.com/ads/library/?id=<archiveId>`). But library
-   results carry only `ad_creative_link_title` + creation/delivery timestamps —
-   **no body text, no ad-account ad id** — so account-ad → library-ad mapping is
-   a **fingerprint match** (creative `title` == library `ad_creative_link_title`,
-   ± creation-time proximity), not a direct key. No tool/field exposes
-   `adArchiveId` from the account side.
-3. **Images are fully retrievable; videos are not.** Image detail gives a
-   signed full-res CDN `url` (+ `url_128`, `permalink_url`, dimensions) —
-   downloadable now, but the URL **expires** (`oe=` param). Video detail gives
-   `length`, poster `picture`, and a *relative* `permalink_url` — **no source /
-   download URL**. This confirms the map's scrape-first asset strategy, with
-   MCP image-pull as gap-filler and **no MCP gap-filler for video files**.
-4. **Naming comes through intact** at every level (`name` on ad/adset/campaign,
-   creative `name`, video `title` — titles even encode aspect ratios like
-   `-9x16`). Naming-conventions guidance can rely on free-text ad names plus
-   `campaign_id`/`adset_id` for hierarchy grouping.
-5. **Setup surface is one URL**: add `https://mcp.facebook.com/ads` as a remote
-   MCP server; OAuth happens in the browser with the user's own Facebook login.
-   Per-account gates: `is_ads_mcp_enabled` and `is_queryable` (disabled/flagged
-   accounts come back with a human-readable `not_queryable_reason`).
-
-## Tool-by-tool findings
-
-### `ads_get_ad_accounts`
-Returns **every** account the user can access — Brad sees 13 across 5
-businesses (own + client accounts), each with `ad_account_id`,
-`ad_account_name`, owning `business_id`/`business_name`, `account_status`,
-`currency`, `is_ads_mcp_enabled`, `is_queryable`, `not_queryable_reason`.
-Two disabled accounts surfaced with the flag reason string. **Multi-account
-selection is a real UX step the skill must handle** (fog item on the map,
-now sharpened). Account names can be empty strings.
-
-### `ads_get_ad_entities` (the insights workhorse)
-- Levels: `account` / `campaign` / `adset` / `ad`. Metrics require
-  `date_preset` (e.g. `last_90d`, `maximum`) or `time_range` JSON;
-  `time_increment` gives time series.
-- Real sample (ad level, last_90d, sorted `amount_spent_descending`):
-
-  ```json
-  {
-    "id": "120241083993850557",
-    "name": "PT-Intro-me-and-ai-Image-Promoted",
-    "effective_status": "ADSET_PAUSED",
-    "creative_id": "952613677477713",
-    "campaign_id": "120239743848310557",
-    "adset_id": "120241083975980557",
-    "amount_spent": "$8,980.38 USD",
-    "results": {"value": "98 (Website appointments scheduled)"},
-    "cost_per_result": {"value": "$91.64 USD (Website appointments scheduled)"},
-    "impressions": "212,816", "ctr": "4.20%", "cpm": "$42.20 USD",
-    "lead": "7", "created_time": "April 17, 2026"
-  }
-  ```
-- **Values are formatted display strings**, not numbers (`"$8,980.38 USD"`,
-  `"4.20%"`, `"Not available"` for zero-delivery ads). Exception: account
-  level also returns numeric `amount_spent_cents`. The bridge/skill must
-  parse display strings (or treat them as verdict-snapshot text, which is
-  what the map's verdict-snapshot design wants anyway).
-- `results`/`cost_per_result` self-label the optimization outcome ("Website
-  appointments scheduled", "Website purchases") — ideal raw material for the
-  verdict snapshot, and it makes result-type explicit across mixed-objective
-  accounts.
-- Field catalog via `ads_get_field_context`: `spend` aliases `amount_spent`;
-  `lead`, `purchase_roas`, `results`, `cost_per_result`, `clicks`, video
-  metrics (`video_thruplay_watched_actions`, `video_p50_watched_actions` —
-  metric-only, not filterable/sortable) exist. **Not present**: `purchase`,
-  `cost_per_purchase`, `cost_per_lead`, `roas`, `campaign_name`/`adset_name`
-  (use ids + a campaign-level call), `link_click`, `hook_rate`,
-  `ad_archive_id`, `preview_shareable_link`, `effective_object_story_id`
-  (creative-side only).
-- Hierarchy: ads carry `campaign_id` + `adset_id`; `objective` lives on
-  campaign/ad, `optimization_goal` on adset.
-
-### `ads_get_creatives`
-The content payload. By id, returns `body` (primary text, full), `title`
-(headline), `call_to_action_type`, `object_type`, `image_hash`, `image_url`
-(signed CDN), `thumbnail_url`, `object_story_id` +
-`effective_object_story_id` (`<pageId>_<postId>`) and
-`effective_instagram_media_id`. Listing mode returns only
-id/name/account_id/status — **must re-fetch by `creative_ids` for content**.
-`link_url` was absent on the probed `SHARE`-type creative (link lives in the
-story post). The `<pageId>_<postId>` story id is the strongest identity hook
-for enrichment (post permalink = `facebook.com/<pageId>/posts/<postId>`).
-
-### `ads_get_ad_images` / `ads_get_ad_videos`
-- Image by hash: `url` full-res signed CDN JPEG + `url_128` + `permalink_url`
-  + `width/height` + timestamps. Downloadable **now**; URL expires later.
-- Video by id: `title`, `length` (seconds float), poster `picture`, relative
-  `permalink_url` (`/<pageId>/videos/<videoId>/…`). **No source URL.**
-- **Gotcha (both tools + creatives): the documented "omit `fields` → all
-  fields" default is broken on by-id lookups** — a `video_ids` lookup without
-  `fields` returned only `{id}`. Always pass explicit `fields`.
-
-### `ads_library_search`
-By `page_ids` (page id from the creative's `effective_object_story_id` prefix
-or `ads_get_ad_account_pages`): returns per ad `id` (**= adArchiveId**),
-`page_id`, `page_name`, `ad_creative_link_title`, `ad_creation_time`,
-`ad_delivery_start_time`, `ad_snapshot_url`, `currency`, plus
-`estimated_total_count` (48 for the probed page). Requires the caller to have
-an active ad account. No body text, no media URLs → cross-reference by
-title + time fingerprint, then hand the `adArchiveId` to the existing
-ScrapeCreators pipeline for full public payload (media, transcription).
-
-### `ads_get_ad_preview`
-Returns an actual **rendered creative image** as in-context image content +
-an authenticated, expiring `preview_url` iframe (business.facebook.com). Good
-for HITL confirmation inside the skill ("is THIS the winner you mean?");
-not a durable asset source.
-
-### `ads_get_ad_account_pages`
-Returns `page_id`s but `page_name` came back `"(unknown)"` on the probed
-account — get the human page name from the library search result instead.
-
-## Gotchas / reliability
-
-- **Metric filtering 500s.** Every `filtering` on `ad.amount_spent`
-  (GREATER_THAN, dollar or cent values) → `Internal Server Error`, repeatably.
-  Attribute filters (`ad.effective_status IN [ACTIVE]`) work. Metric *sorting*
-  works. → Winner definitions execute as: sort by metric server-side, pull top
-  N, threshold agent-side.
-- **Transient 500s** happen on entity queries too (one wide 12-field call
-  failed once, then an identical narrower call succeeded; a later identical
-  call with the same fields also succeeded). Retry-once is a sane default.
-- Only ONE `breakdowns` value per call; only documented params exist.
-- The `ads_insights_*` tools (`performance_trend`, `advertiser_context`, etc.)
-  are opinionated analysis wrappers (trend narratives, funnel context), not
-  raw data — the skill's mining procedure should stay on `ads_get_ad_entities`.
-- Every tool takes an `advertiser_request` verbatim-user-words param — a
-  Meta-side audit surface; the skill should pass the user's actual ask through.
-
-## What this settles for the open tickets
-
-- **#565 (enrichment)**: key = fingerprint match into Ad Library
-  (`title` + creation time → `adArchiveId`) → ScrapeCreators; MCP `image_url`
-  is the image gap-filler (download immediately — signed URLs expire); there is
-  **no video-file gap-filler** via MCP — video enrichment depends on the ad
-  still being scrapeable (or the post being public via
-  `effective_object_story_id`).
-- **#566 (bridge contract)**: payload fields available agent-side per winner:
-  ad id, name, campaign/adset ids, creative id, body, title, CTA type,
-  story/IG ids, image hash + bytes, video id + length + poster, metrics as
-  display strings (+ result-type labels), account id/currency, page id,
-  matched `adArchiveId` (best-effort). Numbers arrive as formatted strings —
-  the contract should either accept snapshot strings verbatim or require the
-  agent to parse.
-- **#564 (interview)**: winner definitions must be expressible as
-  sort-metric + date-window + agent-side thresholds; `results` self-labeling
-  means the definition can be per-result-type; multi-account users (13 visible
-  here) need an explicit account-selection step.

package/assets/references/native-audio-dialogue.md

@@ -1,178 +0,0 @@
-# Native-Audio Dialogue — per-model way to make `dialogue` actually get SPOKEN (in English)
-
-**Research date:** 2026-07-13
-**For:** Build ticket #547 (send storyboard `dialogue` to the native-audio video models). Feeds map #545.
-**Scope:** The four v1 picker models, all via Kie.ai — `veo3_fast` (Veo 3.1 Fast), `veo3` (Veo 3.1 Quality), `bytedance/seedance-2` (Seedance 2.0), `kling-3.0/video` (Kling 3.0).
-**Method:** Primary = docs.kie.ai model pages (fetchable), Google's official Veo prompting guide (cloud.google.com + ai.google.dev), fal.ai model guides, Kling official blog (blocked to automated fetch — corroborated via fal + community). Community tricks flagged inline. Builds on the #529 catalog (`workspace/references/kie-video-models.md`) and the #533 bake-off (`workspace/output/bakeoff-533/`).
-**Confidence labels:** `[docs]` verified on a first-party API/prompting doc · `[official-guide]` first-party prompting guide (Google/Kling/ByteDance) · `[vendor-guide]` fal.ai / infra vendor guide · `[community]` community-reported, use with caution.
-
----
-
-## The core problem (confirmed in code)
-
-`buildClipPrompt` in `scout/src/workflow/video/clips.ts` never reads `scene.dialogue`. It only threads `videoPrompt` + `voText` (and `voText` is dropped entirely when native audio is ON). So on a native-audio run the model is told `generate_audio: true` / `sound: true` with **no words to say**, and it invents speech-shaped babble. Every fix below is about getting the `scene.dialogue[]` lines (`{ speaker, line }`, see `convex/lib/workflow/storyboardContract.ts`) into the prompt in the format each family actually speaks — in English, without burning in captions.
-
-**None of the four models accepts spoken dialogue as a structured API field.** On all four, through Kie, dialogue is **in-prompt text only**. (Details in §"Structured audio params".)
-
----
-
-## TL;DR table
-
-| Family | Dialogue embedding format (recommended) | Force English | Subtitle guard | Structured audio params (Kie) | Pacing guidance |
-|---|---|---|---|---|---|
-| **Veo 3.1 Fast / Quality** (`veo3_fast`,`veo3`) | Inline attribution **with quotes**, per Google's own guide: `She says, in English: "…"`. Multi-speaker = separate attributed sentences. | Write English inside the quotes **and** add an explicit clause `spoken in English (American accent)`. Keep `enableTranslation:false` (it translates the *prompt* to English and can paraphrase quoted lines). No language param. `[docs]`/`[official-guide]` | **No `negativePrompt` param on Kie's `/veo/generate`** — in-prompt only. Keep `ANTI_SUBTITLE_CORE` first; community: quotes can trigger captions, so the anti-caption clause is load-bearing for Veo. `[docs]`/`[community]` | **None.** Audio always on, not togglable; no voice/reference-audio field. `[docs]` | Keep each line short; dialogue lives inside a 4/6/8s clip. Google: short lines sync best. `[official-guide]` |
-| **Seedance 2.0** (`bytedance/seedance-2`) | **Speaker label + colon + double quotes**: `The senior woman, arms crossed, says: "…"`. Short lines; insert written beats (`She pauses, then continues:`) as resync anchors between lines. `[vendor-guide]` | Add per-scene `American accent, conversational tone` + write English text. Mandarin is its most-consistent lip-sync language, so English **must** be stated explicitly or it drifts. `[vendor-guide]`/`[community]` | No first-party subtitle guidance found. Keep `ANTI_SUBTITLE_CORE`. Quoted lines are the documented format, so the core's on-screen-text ban complements (not conflicts). `[community]` | `generate_audio` bool (default true). `reference_audio_urls` (≤3, 2–15s, wav/mp3) exists — a voice/style reference lever, **not** a documented dialogue-content field. `[docs]` | ~**12 words / 10s**, ~**20 words / 15s** (~1.2 wps); sync drops past 10s; record "mental" pace ~80% of natural. **Tighter than the repo's 17 chars/s VO budget.** `[vendor-guide]` |
-| **Kling 3.0** (`kling-3.0/video`) | **Bracketed speaker label + tone, colon, quotes**: `[Character A: Exhausted Partner, trembling voice]: "…"`. Consistent labels (no pronouns); for multi-speaker insert a switch beat. `[official-guide]`/`[vendor-guide]` | Specify language in the prompt (`spoken in English`) — helps phoneme mapping. Supports EN natively but also CN/JA/KO/ES, so state it. No language param. `[official-guide]`/`[community]` | No first-party subtitle guidance found. Keep `ANTI_SUBTITLE_CORE`. `[community]` | `sound` bool. `kling_elements[].audio` (5–30s) is an element-level audio lever, not a dialogue-voice field. `[docs]` | Short, single-speaker segments sync best; use Master/`pro` mode when lip-sync is scrutinized; medium shots > extreme close-ups. `[official-guide]` |
-
----
-
-## Recommended rider / format per API family (implementation-ready for #547)
-
-### Where it slots into the existing rider stack
-
-`buildClipPrompt` today (native audio ON) emits, in order:
-
-1. `ANTI_SUBTITLE_CORE`
-2. `MOTION_POLICY`
-3. `FIRST_FRAME_RIDER` (if a Scene Frame is wired)
-4. `body` (the `videoPrompt`)
-
-**Add a DIALOGUE block as a new final section, ONLY when `nativeAudio === true` AND `scene.dialogue.length > 0`.** Rules:
-
-- Keep `ANTI_SUBTITLE_CORE` **first and unchanged** — it is the only subtitle guard Veo has (no `negativePrompt` param), and its wording bans *on-screen* text ("no dialogue text, no transcript text"), which does not suppress *spoken* audio.
-- The dialogue block is **family-specific** (Veo/Seedance/Kling format differently), so it must be built where the family is known. Two clean options:
-  - **(A)** Thread `scene.dialogue` + `family` into `buildClipPrompt` and branch there.
-  - **(B)** Keep `buildClipPrompt` family-agnostic; in `generateSceneClip` (which already has `spec.family`) append a `buildDialogueBlock(family, scene.dialogue)` string to the returned prompt.
-  Recommend **(B)** — smallest change, keeps `buildClipPrompt` pure, and the per-family request builders already live next to it.
-- `buildClipPrompt` currently accepts `voText` but **not** `dialogue`. #547 must thread `scene.dialogue` through `generateSceneClip` (it already has the full `scene`).
-- When `nativeAudio === false`, do **not** emit dialogue (current behavior — mouths shut, VO track). Unchanged.
-
-### Concrete template strings
-
-Let `LANG = "spoken in English (natural American accent)"`.
-
-**Veo (`veo3_fast`, `veo3`)** — inline, quoted, per Google's guide:
-```
-DIALOGUE (on-camera, lip-synced, {LANG}; no on-screen captions):
-<Speaker A description> says: "<line 1>"
-<Speaker B description> replies: "<line 2>"
-```
-- Single speaker → one attributed sentence.
-- Use the storyboard `speaker` string as the attribution ("The woman", "The barista"). Quotes around the exact words per Google's official example: `A woman says, "We have to leave now."`
-- Do **not** set `enableTranslation:true` (it translates/paraphrases the prompt to English and can rewrite the quoted line). Current code already sends `enableTranslation:false` — keep it.
-
-**Seedance (`bytedance/seedance-2`)** — label + colon + quotes, short lines, beats:
-```
-DIALOGUE ({LANG}, conversational tone; lip-synced to the speaker):
-<Speaker A description> says: "<short line>"
-[She pauses, then continues:] "<next short line>"
-```
-- Colon-then-quotes is the fal-documented Seedance format: `The senior woman, arms crossed, says: "You ran that client call better than I would have."`
-- Split anything over ~10–12 words into two lines with a written beat between them (`She pauses, then continues:`) — Seedance uses written beats as resync anchors.
-- The `American accent, conversational tone` clause is the sourced mitigation for its Mandarin-lip-sync bias — include it every scene.
-
-**Kling (`kling-3.0/video`)** — bracketed labelled speakers:
-```
-DIALOGUE ({LANG}):
-[<Speaker A label>, <tone>]: "<line>"
-[<Speaker B label>, <tone>]: "<line>"
-```
-- Consistent bracket labels, no pronouns (`[Character A: Exhausted Partner, trembling voice]: "You never listen to me."`).
-- Map the storyboard `speaker` to the label and (optionally) infer a tone from `scene.narrative`.
-- Prefer `mode: "pro"` when dialogue is central (official: use Master/high mode for scrutinized lip-sync). Current `klingModeFromResolution` already honors the operator's std/pro pick.
-
-### Request-builder param changes
-
-- **`buildVeoBody`** — no new params. `enableTranslation:false` stays. Dialogue rides in `prompt`. (Kie Veo exposes no audio/voice/negativePrompt field — verified on the docs.)
-- **`buildSeedanceInput`** — no *required* new params; `generate_audio` already tracks the toggle. Optional future lever: `reference_audio_urls` for voice consistency (§Cross-scene). Dialogue rides in `prompt`.
-- **`buildKlingInput`** — no new params; `sound` already tracks the toggle. Dialogue rides in `prompt`.
-
-**Net:** #547 is a **prompt-assembly change, not a request-schema change.** The only transport-level opportunity (Seedance `reference_audio_urls`) is optional and unverified for dialogue voice.
-
----
-
-## Per-model deep dives
-
-### Google Veo 3.1 (`veo3_fast`, `veo3`) — Kie `/api/v1/veo/generate`
-
-- **Dialogue format — quotes, inline attribution.** Google's official *Ultimate prompting guide for Veo 3.1* and the Gemini API docs both use quotation marks with a speaker + verb: `A woman says, "We have to leave now."` and `"This must be the key," he murmured.` The guide lists three audio cue types — Dialogue (quotes), Sound Effects (describe explicitly), Ambient — and says the model "captures the nuance of these cues to generate a synchronized soundtrack." `[official-guide]`
-- **Colon-vs-quotes tension (must reconcile).** A widespread community technique claims you should use a **colon and NO quotation marks** (`the man says: hello there`) because Veo "may interpret quotation marks as text to display" and burn in captions. `[community]` This directly contradicts Google's own quoted examples. Resolution for #547: **follow Google (quotes) for reliable lip-sync**, and neutralize the caption risk with the explicit anti-subtitle clause (which we already have) rather than dropping quotes. If Veo captions still appear in testing, the fallback is the colon-no-quotes variant — treat as a tuning knob, not the default.
-- **Subtitle suppression.** Kie's `/veo/generate` exposes **no `negativePrompt` field** (verified — the param list is prompt/model/aspect_ratio/duration/resolution/generationType/imageUrls/enableTranslation/watermark/callBackUrl only; ai.google.dev's Gemini-API Veo page likewise shows no negativePrompt for the dialogue flow). So the only caption lever is **in-prompt text** — exactly what `ANTI_SUBTITLE_CORE` provides. Community adds "no subtitles"/"no captions" phrasing and keeping dialogue under the 8s clip length so the model "focuses on visuals and audio rather than captions." `[docs]`/`[community]`
-- **English forcing.** No language param. `enableTranslation` "enable[s] prompt translation to English … before video generation" — i.e. it translates *your prompt*, not the spoken output; for an already-English prompt it's a no-op at best and a paraphrase risk at worst, so **keep it false**. Force English by (a) writing the words in English inside the quotes and (b) an explicit `spoken in English (American accent)` clause; Veo "will handle the accent and lip-sync automatically" from the written language. `[docs]`/`[community]`
-- **Audio toggle.** "All videos ship with background audio by default"; rarely suppressed for sensitive scenes. Not togglable on Kie — matches `audioTogglable:false` in `models.ts` (code already forces `nativeAudio=true` for the veo family). `[docs]`
-- **Bake-off note (#533):** Veo 3.1 Fast native-audio clip generated in ~10s wall (anchored) — the fastest of the four; the audio dialogue shoot-out cell is `dlg-veo3-quality.mp4` / `clipref-s1.mp4`. Sora 2 was excluded (paused on Kie). See `workspace/output/bakeoff-533/index.html`.
-
-Sources: [docs.kie.ai/veo3-api/generate-veo-3-video](https://docs.kie.ai/veo3-api/generate-veo-3-video) · [Ultimate prompting guide for Veo 3.1 (Google Cloud)](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1) · [Gemini API Veo docs](https://ai.google.dev/gemini-api/docs/veo) · [GlobalGPT Veo 3.1 dialogue guide](https://www.glbgpt.com/hub/how-to-make-characters-speak-in-veo-3-1-the-ultimate-guide-to-dialogue-audio-lip-sync/) `[community]` · [VidAU "no subtitles"](https://www.vidau.ai/veo-3-prompt-no-subtitles/) `[community]`
-
-### ByteDance Seedance 2.0 (`bytedance/seedance-2`) — Kie jobs API
-
-- **Dialogue format — label, colon, double quotes.** fal.ai's Seedance 2.0 prompting guide: "Put the spoken words in double quotes, and the model voices them, matching the lip movement to each line," e.g. `The senior woman, arms crossed, says: "You ran that client call better than I would have."` For multi-speaker, hold a two-shot then cut to the speaker, "lips matched to each line." Keep lines short — "Long monologues drift out of sync, so I split a speech into a couple of shorter lines and let the cuts carry it." `[vendor-guide]`
-- **Written beats as resync anchors.** "Insert a written beat ('She pauses, then continues:') between sentences — Seedance uses written beats as resync anchors." Directly useful for splitting a scene's multiple `dialogue` lines. `[vendor-guide]`
-- **English forcing / drift.** Seedance supports 8+ languages incl. English/Mandarin, but "Mandarin produces the most consistent lip sync," and separate generations of the same character drift in accent unless pinned. Sourced mitigation: **"Specify accent explicitly in every scene prompt: 'American accent, conversational tone.'"** No API language param. The specific "drifts to Chinese/Mandarin" failure the map calls out is **community-reported and consistent with these vendor guides, but I could not pin a primary Reddit/X thread this session** — treat the drift as real-risk, the accent clause as the sourced fix. `[vendor-guide]`/`[community]`
-- **Subtitle risk.** No first-party statement that quoted Seedance dialogue burns in captions; no first-party suppression guidance found. Keep `ANTI_SUBTITLE_CORE`. `[community]`
-- **Pacing (important — conflicts with the repo budget).** fal / UGC-Copilot guidance: **~12 words per 10s, ~20 words per 15s (~1.2 words/sec)**; "short sentences … five to ten words per line is the sweet spot"; sync quality "drops past 10" seconds. The storyboard gate (`VO_CHARS_PER_SEC = 17`, `sceneSpeechSec`) budgets ~17 chars/s ≈ ~3 words/s — **~2.5× looser than Seedance actually speaks cleanly.** A scene that "fits" at 17 cps can still overflow / rush / desync on Seedance native audio. **Recommendation for #547 (or a fast-follow):** use a tighter chars-per-second budget for native-audio *dialogue* (≈9–10 cps, ~1.2–1.5 wps) than for VO narration. Seedance does not hard-truncate; it rushes and desyncs. `[vendor-guide]`
-- **Bake-off note (#533):** Seedance native audio was slowest by far — measured ≈24 min/clip, which is why `POLL_TIMEOUT_VIDEO_MS` was raised to 35 min in `kie-video.ts`. It won "best native audio" per the map, but at a steep latency/cost. `dlg-seedance.mp4` vs the silent control `dlg-seedance-silent.mp4` vs pixr mux `dlg-elevenlabs-mux.mp4`.
-
-Sources: [docs.kie.ai/market/bytedance/seedance-2](https://docs.kie.ai/market/bytedance/seedance-2) · [fal.ai Seedance 2.0 prompting guide](https://fal.ai/learn/tools/seedance-2-0-prompting-guide) · [UGC-Copilot Seedance native audio guide](https://ugccopilot.ai/blog/seedance-2-native-audio-generation-guide/) `[community]` · [Cutout.pro Seedance audio guide](https://www.cutout.pro/learn/blog-seedance-2-0-audio-guide/) `[community]`
-
-### Kling 3.0 (`kling-3.0/video`) — Kie jobs API
-
-- **Dialogue format — bracketed labelled speakers.** fal.ai's Kling 3.0 guide: `[Character Label, tone/emotion]: "dialogue text"`, e.g. `[Character A: Exhausted Partner, trembling frustrated voice]: "You never listen to me."` Structured naming (P1): "Character labels must be consistent. Avoid pronouns or synonyms." For sequencing/multi-speaker, insert a switch beat ("this is when the speaker switches"). `[vendor-guide]`
-- **Official audio behavior.** Kling's own blog (Kling VIDEO 3.0 / 3.0 Omni "Native Audio") — dialogue, sound and lip movement are generated together. Best-practice guidance (corroborated across Kling + fal + community): short simple sentences beat rapid complex speech; **specify the language in the prompt to help phoneme mapping**; medium shots sync better than extreme close-ups; avoid overlapping dialogue — single-speaker segments sync much better; use Master/high mode when lip-sync will be scrutinized. `[official-guide]`/`[community]` (Note: the official Kling blog returns HTTP 446 to automated fetch; content is corroborated via the search excerpt + fal guide, not a direct page read.)
-- **Language support / forcing.** Supports Chinese, English, Japanese, Korean, Spanish + dialects/accents (American/British/Indian English). No language param — state `spoken in English` in the prompt. `[official-guide]`/`[community]`
-- **Kie param nuance.** On Kie, the native-audio toggle is `sound` (bool) and `prompt` "takes effect when multi_shots is false" — our builder sends `multi_shots:false`, so the single `prompt` (with the dialogue block) is the right channel. Multi-shot mode would need the `multi_prompt[]` array instead (each shot's `prompt` ≤500 chars); v1 does not use it. `[docs]`
-- **Bake-off note (#533):** `dlg-kling.mp4`, ~103s gen — much faster than Seedance, comparable to Veo Quality.
-
-Sources: [docs.kie.ai/market/kling/kling-3-0](https://docs.kie.ai/market/kling/kling-3-0) · [Kling official: Kling Video 3.0 Omni Native Lip Sync & Audio](https://kling.ai/blog/kling-video-3-omni-native-lip-sync-audio-guide) (446 to automated fetch) · [fal.ai Kling 3.0 prompting guide](https://blog.fal.ai/kling-3-0-prompting-guide/) · [Imagine.art Kling 3.0 prompt guide](https://www.imagine.art/blogs/kling-3-0-prompt-guide) `[community]`
-
----
-
-## Cross-scene voice consistency (map fog — levers that exist)
-
-The problem: each scene is an independently-generated clip, so the same actor gets a **different synthesized voice** each time. No picker model exposes a first-class "voice ID" through Kie's jobs/veo API. Levers found, none verified to solve it:
-
-- **Seedance `reference_audio_urls`** `[docs]` — up to 3 audio clips (2–15s, wav/mp3), also accepts `asset://{assetId}`. Documented as a *reference* input (style/voice conditioning), **not** as the dialogue-content source. **Fog:** it *might* anchor timbre if the same reference clip is passed to every scene — but the docs don't say it transfers voice identity, and whether the model speaks the *prompt* text in the *reference* voice is untested. This is the single most promising Kie-native lever; worth an empirical probe before relying on it. Seedance is also the only one of the four with any audio-input field.
-- **Kling `kling_elements[].audio`** `[docs]` — per-element audio (5–30s) tied to an `@element`. Element references are aimed at *visual* character consistency (2–4 images/element); the audio slot is under-documented for voice transfer. **Fog.**
-- **Veo** — **no audio input at all.** No lever for cross-scene voice. `[docs]`
-- **Higher-level "voice fingerprint" features** (e.g. fal's "AI Twins" persona voice-locking) exist at the infra layer but are **not exposed through Kie's jobs API**, so not usable in this pipeline as-is. `[community]`
-
-**Practical read for v1:** there is no reliable cross-scene voice-consistency lever through Kie today. If it matters, the more controllable path remains the old pixr architecture — render silent + one ElevenLabs voice muxed across all scenes (the `dlg-elevenlabs-mux.mp4` control in #533). Keep cross-scene voice in the fog for #547; if pursued, the first experiment is Seedance `reference_audio_urls` with a fixed reference clip.
-
----
-
-## Duration / pacing behavior (per family)
-
-- **Seedance:** ~12 words/10s, ~20 words/15s; sync degrades past ~10s and with long lines; does **not** hard-truncate — it rushes and drifts out of sync. Written beats resync it. `[vendor-guide]`
-- **Veo:** fixed 4/6/8s clips; keep dialogue short enough to land inside the clip; Google/community both say short lines sync best and staying under 8s reduces caption bleed. `[official-guide]`/`[community]`
-- **Kling:** 3–15s; short single-speaker segments sync best; no published words/sec number found. `[official-guide]`
-- **Repo gate today:** `sceneSpeechSec = max(voText, dialogue) ÷ 17 chars/s` fits *both* tracks to `durationSec` and fails loud rather than truncating. For **VO narration** 17 cps is reasonable; for **native lip-synced dialogue** it is too loose (Seedance ≈ 7–8 cps of clean speech). **Actionable:** consider a separate, tighter cps for native-audio dialogue-fit. (Flagged as a recommendation; whether #547 changes the budget or just formats the block is a scoping call.)
-
----
-
-## Open questions / unverifiable this session
-
-1. **Veo colon-vs-quotes for captions** — Google officially uses quotes; the "quotes burn captions, use a colon" claim is community-only and contradicts the official examples. Not resolvable without a live Veo-on-Kie A/B. Default to quotes + anti-caption clause; keep colon-no-quotes as a fallback knob. `[community]`
-2. **Seedance→Chinese drift** — consistent with vendor guides ("Mandarin most consistent," "specify accent every scene") but no primary Reddit/X thread pinned this session. The accent clause is the sourced mitigation; the drift itself is community-level. `[community]`
-3. **Whether `ANTI_SUBTITLE_CORE`'s "no dialogue text" wording ever suppresses *spoken* audio** — it targets on-screen typography, so it *shouldn't*, but a paranoid model could misread it. Low risk; worth watching in the first native-audio+dialogue test. Not observed in #533 (which ran without a dialogue block).
-4. **Seedance `reference_audio_urls` as a voice-consistency lever** — exists in the API but its effect on spoken-dialogue voice is undocumented. Needs an empirical probe.
-5. **Kie Veo `negativePrompt`** — absent from Kie's `/veo/generate` param list (Vertex AI Veo has one; Kie's endpoint does not expose it). Confident but worth a 1-line confirm if #547 wants a param-based caption guard.
-6. **Exact per-family words/sec** — only Seedance publishes numbers; Veo/Kling give qualitative "keep it short." Any tighter dialogue-fit budget will be model-family-specific and partly empirical.
-
----
-
-## Sources
-
-Primary / first-party:
-- [docs.kie.ai — Veo 3.1 generate](https://docs.kie.ai/veo3-api/generate-veo-3-video), [Seedance 2.0](https://docs.kie.ai/market/bytedance/seedance-2), [Kling 3.0](https://docs.kie.ai/market/kling/kling-3-0)
-- [Google Cloud — Ultimate prompting guide for Veo 3.1](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1)
-- [Google AI for Developers — Generate videos with Veo 3.1 (Gemini API)](https://ai.google.dev/gemini-api/docs/veo)
-- [Kling.ai — Kling Video 3.0 Omni Native Lip Sync & Audio](https://kling.ai/blog/kling-video-3-omni-native-lip-sync-audio-guide) (446 to automated fetch; corroborated indirectly)
-
-Vendor guides (fal.ai):
-- [Seedance 2.0 prompting guide](https://fal.ai/learn/tools/seedance-2-0-prompting-guide), [Kling 3.0 prompting guide](https://blog.fal.ai/kling-3-0-prompting-guide/)
-
-Community (labeled inline):
-- [GlobalGPT Veo 3.1 dialogue/lip-sync](https://www.glbgpt.com/hub/how-to-make-characters-speak-in-veo-3-1-the-ultimate-guide-to-dialogue-audio-lip-sync/) · [VidAU Veo no-subtitles](https://www.vidau.ai/veo-3-prompt-no-subtitles/) · [UGC-Copilot Seedance native audio](https://ugccopilot.ai/blog/seedance-2-native-audio-generation-guide/) · [Cutout.pro Seedance audio](https://www.cutout.pro/learn/blog-seedance-2-0-audio-guide/) · [Imagine.art Kling 3.0](https://www.imagine.art/blogs/kling-3-0-prompt-guide)
-
-Local prior art:
-- `workspace/references/kie-video-models.md` (#529 catalog) · `workspace/output/bakeoff-533/index.html` + `manifest.json` (#533 native-audio bake-off) · `scout/src/workflow/video/clips.ts`, `models.ts`, `kie-video.ts`, `storyboard.ts` · `convex/lib/workflow/storyboardContract.ts`