@aicopycoders/exodus 2026.7.200-beta.7 → 2026.7.200-beta.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -31,7 +31,7 @@ This folder usually holds **two related products that share the same Genesis eng
31
31
 
32
32
  ## Multi-Brand Operating Notes
33
33
 
34
- Any user can create and own unlimited brands — `npx @aicopycoders/exodus brand create "<name>"` (or the dashboard); it is NOT admin-only, and brand names are scoped per user (two users can each have a "Matt Beard"). One account key then switches across every brand the user owns (admins: every brand). The only exception is a brand someone ELSE invited the user into — they don't own it, so their key stays pinned to just that one. Never tell a user they can't make brands or are "locked to one brand" just because their key shows `role: member`. The active brand resolves **folder > pointer > key default**: being inside a brand's subfolder wins, then the `brand use` pointer, then the key's default brand.
34
+ Any user can create and own unlimited brands — `npx @aicopycoders/exodus brand create "<name>"` (or the dashboard); it is NOT admin-only, and brand names are scoped per user (two users can each have an "Evergreen Fitness"). One account key then switches across every brand the user owns (admins: every brand). The only exception is a brand someone ELSE invited the user into — they don't own it, so their key stays pinned to just that one. Never tell a user they can't make brands or are "locked to one brand" just because their key shows `role: member`. The active brand resolves **folder > pointer > key default**: being inside a brand's subfolder wins, then the `brand use` pointer, then the key's default brand.
35
35
 
36
36
  - **Identify the active brand:** `npx @aicopycoders/exodus brand current` (also says whether it came from the folder or the pointer)
37
37
  - **List accessible brands:** `npx @aicopycoders/exodus brand list`
@@ -18,7 +18,7 @@ Returns:
18
18
 
19
19
  ## Strategic Context
20
20
 
21
- One API key, many brands. **Any user can create and own unlimited brands** — via `npx @aicopycoders/exodus brand create "<name>"` or the dashboard. This is NOT admin-only, and brand names are scoped per user (two different users can each have a "Matt Beard"). The key works for every brand the user OWNS (admin keys see everything); the CLI attaches `X-Active-Brand: <slug>` on every request. The ONLY "pinned to one brand" case is a brand someone ELSE invited the user into: they don't own it, so the header is ignored for it and it's the single brand their key sees. Never tell a user they can't create brands or are "locked to one brand" because their key shows `role: member` — a member who owns brands sees and creates as many as they want.
21
+ One API key, many brands. **Any user can create and own unlimited brands** — via `npx @aicopycoders/exodus brand create "<name>"` or the dashboard. This is NOT admin-only, and brand names are scoped per user (two different users can each have an "Evergreen Fitness"). The key works for every brand the user OWNS (admin keys see everything); the CLI attaches `X-Active-Brand: <slug>` on every request. The ONLY "pinned to one brand" case is a brand someone ELSE invited the user into: they don't own it, so the header is ignored for it and it's the single brand their key sees. Never tell a user they can't create brands or are "locked to one brand" because their key shows `role: member` — a member who owns brands sees and creates as many as they want.
22
22
 
23
23
  The active brand resolves **folder > pointer > key default**: on a multi-brand install each brand has a subfolder (marked by `.exodus-brand.json`), and running from inside one targets that brand automatically — it wins over the `{ activeBrand: "<slug>" }` pointer in `.exodus/state.json` that `brand use` writes. `brand current` tells you which one applied.
24
24
 
@@ -50,7 +50,7 @@ Shows role, the active brand (marked `★`), and the full accessible list. Flags
50
50
  ### Create a new brand
51
51
 
52
52
  ```bash
53
- npx @aicopycoders/exodus brand create "Matt Beard"
53
+ npx @aicopycoders/exodus brand create "Evergreen Fitness"
54
54
  ```
55
55
 
56
56
  Creates a brand the user owns (any user can — no admin role required), then switches the CLI into it and, on multi-brand installs, sets up its subfolder. Brand names are scoped per user, so this succeeds even if another account already has a brand by that name (the slug is auto-disambiguated under the hood). Tell the user the next step is the primer (`exodus, set up my brand primer`). If the dashboard ever reports a name conflict for a brand the user is *sure* they've never made, that's a separate bug — not a sign the name is taken for them.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@aicopycoders/exodus",
3
- "version": "2026.7.200-beta.7",
3
+ "version": "2026.7.200-beta.8",
4
4
  "description": "CLI + Claude Code skills for the Copycoders Exodus ad pipeline",
5
5
  "repository": {
6
6
  "type": "git",
@@ -1,207 +0,0 @@
1
- # Kie.ai Video Model Catalog — Research Findings
2
-
3
- **Research date:** 2026-07-13
4
- **For:** Wayfinder ticket #529 (map #528 — video in the Workflow Builder)
5
- **Author:** research agent
6
- **Method:** Primary sources = https://kie.ai + https://docs.kie.ai (model + API-reference pages). Marketing pages under `kie.ai/*` (pricing, market) return **HTTP 403 to automated fetch**, so capability facts come from the `docs.kie.ai` model pages (fetchable) and pricing comes from Kie-published/secondary sources anchored to Kie's stated **$0.005/credit** rate. Anything I could not pin to a primary Kie page is flagged **[verify]**. Local code claims come from reading this repo + the `archive/pixar-v2` git tag.
7
-
8
- > **Caveat on pricing precision:** Kie updates model prices frequently and the canonical `kie.ai/pricing` / `kie.ai/market` pages block automated fetch. The per-clip USD figures below are directionally correct (and internally consistent with the $0.005/credit anchor), but **confirm the exact credit cost on the live pricing page / each model's doc page before hard-coding any number in the app.**
9
-
10
- ---
11
-
12
- ## 1. Executive summary + recommended v1 list
13
-
14
- Kie.ai is a **unified multi-model gateway**: one API key, one async job pattern (`/api/v1/jobs/createTask` → poll `recordInfo` or receive a `callBackUrl` webhook), 30+ video models switchable by a `model` slug. It exposes today (video): **Google Veo 3 / 3.1**, **Kling (2.1, 3.0)**, **ByteDance Seedance (2.0, 2.0 Fast, 2.5, 1.5 Pro)**, **OpenAI Sora 2 / Sora 2 Pro**, **Hailuo/MiniMax (2.3 Pro/Standard)**, **Wan (2.6 family)**, **Runway**, plus Grok Imagine. All four "big" families now support **image-to-video** and most now support **native audio** — a material change since the pixr era.
15
-
16
- **Recommended v1 model list for the Video-node picker** (criteria: image-to-video quality, native audio, price, reliability):
17
-
18
- | Rank | Model | `model` slug | Why | Native audio | I2V |
19
- |------|-------|--------------|-----|:---:|:---:|
20
- | **Default** | **Veo 3.1 Fast** | `veo3_fast` | Best price/quality with native dialogue+SFX audio; first-frame + first/last-frame + reference I2V; 9:16 & 16:9; battle-tested endpoint. ~$0.30–0.40 / 8s. | ✅ | ✅ |
21
- | 2 | **Seedance 2.0** | `bytedance/seedance-2` | Strongest multimodal keyframe control (first/last frame + up to 9 reference images + reference video/audio), native audio default-on, cheap, and it's the exact model family the old pixr clip stage already drove. | ✅ | ✅ |
22
- | 3 | **Kling 3.0** | `kling-3.0/video` | Multi-shot storytelling + `@element` character-consistency (2–4 ref images/element, max 3), up to 15s, native sound, up to 4K. Best for character-driven multi-shot ads. | ✅ | ✅ |
23
- | 4 (premium) | **Veo 3.1 Quality** | `veo3` | Highest fidelity when budget allows (~$1.25–2.00 / 8s). Same param surface as Fast. | ✅ | ✅ |
24
- | 5 (premium) | **Sora 2 / Pro** | `sora-2` / `sora-2-pro` | Best physics/realism + synced native audio; pricier, no free tier. Good "hero clip" option. | ✅ | ✅ |
25
-
26
- **Default = Veo 3.1 Fast (`veo3_fast`)**: it is the sweet spot of native-audio quality, image-to-video support (single keyframe *and* first/last frame *and* reference), vertical-native output for ad formats, and low cost, on Kie's most mature video endpoint.
27
-
28
- **Native-audio landscape:** Veo 3.1, Kling 3.0, Seedance 2.0, Sora 2, and Hailuo 2.3 all generate audio natively now. This is the single biggest shift vs. the pixr pipeline, which ran Seedance **silent** and overlaid ElevenLabs VO. For v1 the native-audio toggle should default **on** for models that support it, with a clear "silent (BYO audio)" option that maps to each model's `generate_audio:false` / `sound:false` flag.
29
-
30
- **Pricing ballpark:** credits bill at **$0.005/credit (200 credits = $1)**. An 8s Veo 3 Fast clip ≈ $0.30–0.40; Veo 3 Quality ≈ $1.25–2.00; Kling ≈ $0.07/s; Seedance 2.0 Fast ≈ $0.02/s; Hailuo 2.3 1080p ≈ $0.40/clip. Sora 2 is the priciest household name.
31
-
32
- **API pattern:** async job — `POST /api/v1/jobs/createTask` returns a `taskId`; then either poll `GET /api/v1/jobs/recordInfo?taskId=...` (states: `waiting → queuing → generating → success|fail`) or supply `callBackUrl` for a webhook. Rate limit ~**20 create requests / 10s / account**. Errors: 401 unauthorized, 402 insufficient credits, 429 rate-limited, 500 server error (Veo also has a dedicated `/api/v1/veo/generate` endpoint).
33
-
34
- **Biggest change vs the pixr Kie client:** (1) Seedance now defaults to **native audio on** and adds first/last-frame + up to 9 reference images + reference video/audio (pixr hard-coded `generate_audio:false` and one `reference_image_urls`); (2) the whole modern video roster (Veo 3.1, Kling 3.0, Sora 2, Hailuo, Wan) is available through the *same* unified `createTask` the pixr client already spoke — so adding models is mostly a slug + param-map change, not a new transport.
35
-
36
- ---
37
-
38
- ## 2. Per-model capability table
39
-
40
- All models are invoked through `POST https://api.kie.ai/api/v1/jobs/createTask` with `{ model, input, callBackUrl }` **unless noted** (Veo also has a dedicated endpoint). "Upstream" = capability from the provider's own docs, not confirmed on Kie's page (Kie's exposed params may be a subset).
41
-
42
- ### Google Veo 3.1 — `veo3` / `veo3_fast` / `veo3_lite`
43
- Source: https://docs.kie.ai/veo3-api/generate-veo-3-video , https://docs.kie.ai/veo3-api/quickstart , https://docs.kie.ai/veo3-api/extend-video
44
-
45
- - **Endpoint:** dedicated `POST https://api.kie.ai/api/v1/veo/generate` (also surfaced in the Market). Extend endpoint exists for lengthening clips.
46
- - **Image-to-video:** ✅ `imageUrls` array (**1–3 images**). `generationType`: `TEXT_2_VIDEO`, `FIRST_AND_LAST_FRAMES_2_VIDEO`, `REFERENCE_2_VIDEO`. So it supports single keyframe, first+last frame, and reference-image (character/style) modes.
47
- - **Native audio:** ✅ "All videos ship with background audio by default." Veo 3 is known for synced dialogue + SFX (upstream).
48
- - **Duration:** `4`, `6`, or `8` s (default 8).
49
- - **Aspect ratio:** `16:9`, `9:16`, `Auto` (default 16:9).
50
- - **Resolution:** `720p`, `1080p`, `4k` (default 720p).
51
- - **Other params:** `watermark`, `enableTranslation` (auto-translate prompt to EN), `callBackUrl`, `seeds` [verify].
52
- - **Tiers:** `veo3` (quality), `veo3_fast` (default, cheaper), `veo3_lite` (cheapest/high-volume).
53
-
54
- ### Kling 3.0 — `kling-3.0/video`
55
- Source: https://docs.kie.ai/market/kling/kling-3-0 , https://docs.kie.ai/market/kling/motion-control-v3
56
-
57
- - **Image-to-video:** ✅ `image_urls` = first/last frame images (optional when using element references).
58
- - **Reference / character consistency:** ✅ `@element_name` syntax in the prompt; **max 3 elements** per task; image elements accept **2–4 image URLs** (JPG/PNG, ≤10MB each) for character consistency across shots. Also a dedicated **motion-control** variant.
59
- - **Native audio:** ✅ `sound` boolean (defaults **true** in multi-shot mode); native audio up to 15s.
60
- - **Multi-shot:** ✅ `multi_shots` boolean → pass an **array of prompts** for multi-shot storytelling.
61
- - **Duration:** integer **3–15** s.
62
- - **Aspect ratio:** `16:9`, `9:16`, `1:1` (auto-adapts when `image_urls` given).
63
- - **Resolution (`mode`):** `std` = 720p, `pro` = 1080p, `4K` = 2160p (per AR: 16:9 / 9:16 / 1:1).
64
-
65
- ### ByteDance Seedance 2.0 — `bytedance/seedance-2` (+ `-fast`, `2.5`, `1.5-pro`)
66
- Source: https://docs.kie.ai/market/bytedance/seedance-2 , https://docs.kie.ai/market/bytedance/seedance-2-fast
67
-
68
- - **Image-to-video:** ✅ `first_frame_url` and/or `last_frame_url` (URL or `asset://{assetId}`). Three mutually-exclusive modes: first-frame I2V, first+last-frame I2V, or multimodal reference generation.
69
- - **Reference / character consistency:** ✅ `reference_image_urls` (**max 9**; jpeg/png/webp/bmp/tiff/gif), `reference_video_urls` (**max 3**, 2–15s, mp4/mov), `reference_audio_urls` (**max 3**, 2–15s, wav/mp3). This is the richest keyframe/reference surface of any model here.
70
- - **Native audio:** ✅ `generate_audio` boolean, **default `true`** (was run silent by pixr).
71
- - **Duration:** **4–15** s (default 5).
72
- - **Aspect ratio:** `1:1`, `4:3`, `3:4`, `16:9`, `9:16`, `21:9`, `adaptive` (default 16:9).
73
- - **Resolution:** `480p`, `720p`, `1080p`, `4k` (default 720p).
74
- - **Other:** `nsfw_checker`, `web_search`, `prompt` 3–20,000 chars, `callBackUrl`.
75
- - **`bytedance/seedance-2-fast`:** cheaper/faster variant, same param family.
76
-
77
- ### OpenAI Sora 2 — `sora-2` / `sora-2-pro` (+ Characters)
78
- Source: https://docs.kie.ai/market/sora2/... (index confirmed via search; individual pages render as SPA and 404'd to automated fetch — **capability confirmed, param names [verify] on live docs**)
79
-
80
- - **Variants:** `sora-2` (standard), `sora-2-pro` (higher quality), plus **Characters** and **Image-to-Video** / **Text-to-Video** modes.
81
- - **Image-to-video:** ✅ (dedicated Image-to-Video page).
82
- - **Native audio:** ✅ synced dialogue + ambient (Sora 2's headline feature, upstream).
83
- - **Strength:** best physics/realism. No free tier; priciest of the household names.
84
- - **Duration / AR / resolution:** [verify on the specific `sora2` doc pages].
85
-
86
- ### Hailuo / MiniMax 2.3 — `hailuo/...` (e.g. `2-3-image-to-video-pro`)
87
- Source: https://docs.kie.ai/market/hailuo/2-3-image-to-video-pro (via search)
88
-
89
- - **Image-to-video:** ✅ dedicated I2V Pro + Standard tiers.
90
- - **Native audio:** ✅ (2.3 generation) [verify exact flag].
91
- - **Strength:** character animation + facial expression / talking-head, avatar content.
92
- - **Pricing:** ~80 credits (~$0.40) for a 1080p clip [verify].
93
-
94
- ### Wan 2.6 — `wan/2.6-*`
95
- Source: https://kie.ai/market/image-to-video (via search)
96
-
97
- - Variants: `Wan 2.6 Image-to-Video`, `Text-to-Video`, `Video-to-Video`, `2.6-flash-image-to-video`, `2-6-flash-video-to-video`.
98
- - **Image-to-video:** ✅. Native audio / duration / AR: [verify on live docs]. Budget option.
99
-
100
- ### Runway — `runway/...`
101
- Source: https://docs.kie.ai/runway-api/quickstart
102
-
103
- - Dedicated Runway API surface for high-quality I2V/T2V. Params (Gen-3/Gen-4 family) [verify on live docs]. Traditionally **no native audio** (upstream). Lower priority for v1.
104
-
105
- ---
106
-
107
- ## 3. Pricing (credits → USD)
108
-
109
- **Credit rate (primary anchor):** Kie bills at **$0.005 per credit** → **200 credits = $1.00**. Different models bill per-second, per-clip, or per-image; the unit is shown per model on the pricing page.
110
- Source: https://kie.ai/pricing (403 to fetch), corroborated by https://skywork.ai/... and https://kie.ai/v3-api-pricing.
111
-
112
- | Model | Kie price (per clip / rate) | Credits | Notes |
113
- |-------|------------------------------|---------|-------|
114
- | **Veo 3 / 3.1 Fast** | ~**$0.30–0.40** / 8s w/ audio | 60–80 | Recent update lowered Fast to ~60 cr ($0.30) [verify] |
115
- | **Veo 3 / 3.1 Quality** | ~**$1.25–2.00** / 8s w/ audio | 250–400 | Recent update lowered to ~250 cr ($1.25) [verify] |
116
- | **Kling 2.1** | Std **$0.125**/5s · Pro **$0.25**/5s · Master **$0.80**/5s | 25 / 50 / 160 | Kling 3.0 similar tiered ($0.07/s cited) [verify] |
117
- | **Seedance 2.0 Fast** | ~**$0.022**/s (~$0.11 for 5s) | ~4.4/s | Cheapest production-grade [verify] |
118
- | **Seedance 2.0** | between Fast and premium [verify] | — | Native audio default-on |
119
- | **Hailuo 2.3** | ~**$0.40** / 1080p clip | ~80 | [verify] |
120
- | **Sora 2 / Pro** | priciest household name [verify] | — | No free tier |
121
- | **Runway / Wan** | budget–mid [verify] | — | — |
122
-
123
- > Kie claims ~30% below official API pricing on most models, up to 60–70% on select high-demand models. **Treat every USD figure above as [verify] against the live pricing page before hard-coding.**
124
-
125
- ---
126
-
127
- ## 4. API shape
128
-
129
- **Unified async job pattern** (all Market models):
130
-
131
- 1. **Create:** `POST https://api.kie.ai/api/v1/jobs/createTask`
132
- Headers: `Authorization: Bearer <KIE_API_KEY>`, `Content-Type: application/json`
133
- Body: `{ "model": "<slug>", "input": { ...model params... }, "callBackUrl": "<optional webhook>" }`
134
- Returns: `{ code, msg, data: { taskId } }`.
135
- 2. **Get result — two options:**
136
- - **Poll:** `GET https://api.kie.ai/api/v1/jobs/recordInfo?taskId=<id>` → `data.state` ∈ `waiting | queuing | generating | success | fail`; on `success`, `data.resultJson` is a JSON string containing `resultUrls[]` (video URL). On `fail`, `data.failCode` / `data.failMsg`.
137
- - **Webhook:** supply `callBackUrl` in createTask; Kie POSTs completion, "eliminating the need for polling" (recommended for production).
138
- 3. **Credit check:** `GET https://api.kie.ai/api/v1/chat/credit`.
139
-
140
- **Veo exception:** Veo 3.x also has a **dedicated endpoint** `POST https://api.kie.ai/api/v1/veo/generate` (+ get-details + extend), separate from the generic jobs endpoint.
141
-
142
- **Rate limits:** ~**20 generation requests per 10 seconds per account** (from the Getting-Started guide).
143
-
144
- **Error / status semantics:** `401` unauthorized (bad key), `402` insufficient credits, `429` rate-limited, `500` server error. Job-level failures surface as `state:"fail"` + `failCode`/`failMsg` (Kie relays upstream provider errors, incl. safety blocks, in `failMsg`). HTTP 200 on createTask means **accepted, not complete**.
145
-
146
- Sources: https://docs.kie.ai/ , https://docs.kie.ai/market/quickstart , https://docs.kie.ai/market/common/get-task-detail
147
-
148
- ---
149
-
150
- ## 5. Diff vs the prior integration (pixr client, `archive/pixar-v2`)
151
-
152
- The torn-out pixr pipeline's Kie client lives at **`archive/pixar-v2:scout/src/pixar/kie-client.ts`** (also `archive/pixar-v2:scout/src/image-ads/image-gen.ts` and the PowerShell reference `archive/pixar-v2:contrib/mcastelli99/2026-05-13/code/generate_videos_daily_glide.ps1`).
153
-
154
- **What the pixr client used:**
155
- - **Same transport** we'd reuse today: unified `POST /api/v1/jobs/createTask` + poll `GET /api/v1/jobs/recordInfo`, Bearer auth, `resultJson.resultUrls[0]`. Confirmed identical to the current image stack.
156
- - **Video model:** `bytedance/seedance-2` (`MODEL_VIDEO_SEEDANCE`), called via `buildClipRequest` with:
157
- `{ prompt, aspect_ratio: "9:16", resolution: "720p", duration, generate_audio: false, reference_image_urls: [<one image>] }`.
158
- **Audio was hard-coded OFF** — with defense-in-depth throwing if a caller passed `generateAudio !== false` — per Mario's "AUDIO_ARCHITECTURE.md" (Seedance audio drifted pacing). Audio came from a **separate ElevenLabs direct call** (`kieGenerateVoiceover` hits `api.elevenlabs.io/v1/text-to-speech/{voice_id}` directly, **not** Kie — an earlier attempt at ElevenLabs-via-Kie 422'd on the slug `elevenlabs/text-to-speech-multilingual-v2`).
159
- - **Image models:** `gpt-image-2-text-to-image` (T2I) and `gpt-image-2-image-to-image` (I2I, via `input_urls: [refUrl]`), with `aspect_ratio`, `output_format:"png"`, `resolution:"1K"`.
160
- - **Retry/robustness:** exponential backoff on transient 5xx / "try again later" / rate-limit, each retry minting a fresh task; video poll timeout 20 min, image 7 min.
161
-
162
- **What has changed in Kie's current API vs that client:**
163
- 1. **Seedance native audio is now default-ON.** Current `bytedance/seedance-2` defaults `generate_audio:true`; the pixr "silent + overlay ElevenLabs" architecture is now **optional**, not required. v1 can drop the separate VO stage for models with native audio (or keep silent mode as an explicit toggle).
164
- 2. **Seedance keyframe/reference surface expanded dramatically.** Then: one `reference_image_urls[]`. Now: `first_frame_url` + `last_frame_url` (true first/last-frame I2V) **plus** `reference_image_urls` (max 9), `reference_video_urls` (max 3), `reference_audio_urls` (max 3), `web_search`, `nsfw_checker`. Duration widened to 4–15s, resolution adds 480p/1080p/4k, AR adds 1:1/4:3/3:4/21:9/adaptive.
165
- 3. **The whole modern roster is now on the same endpoint.** pixr only ever drove Seedance for clips. Today Veo 3.1 (also its own `/veo/generate`), Kling 3.0, Sora 2, Hailuo 2.3, Wan 2.6 are all reachable via the same `createTask` + slug the pixr client already spoke — so the Video-node picker is largely a **slug + input-schema map**, not new transport code.
166
- 4. **Webhook callbacks** (`callBackUrl`) are now first-class and recommended over polling for production; pixr was poll-only.
167
- 5. **ElevenLabs-via-Kie** remains awkward (pixr went direct); if the storyboard ever needs standalone VO, prefer the model's native audio or a direct ElevenLabs call rather than routing TTS through Kie.
168
-
169
- ---
170
-
171
- ## 6. Current image-node render stack (Scene Frames will likely reuse this)
172
-
173
- The **working tree** (not the archive) drives Kie for images in two places, both via the identical unified `createTask` + `recordInfo` poll pattern:
174
-
175
- - **`src/lib/modules/providers/image-generator/kie-nano-banana.ts`** — `KieNanoBananaProvider`, slug `"nano-banana-pro"` but **actually targets `gpt-image-2-text-to-image`** (`DEFAULT_MODEL`). Input: `{ prompt, nsfw_checker:false }` only — it does **not** send aspect_ratio/size (comment: model ignores them). createTask → poll `recordInfo` (2s interval, 120s timeout) → `resultJson.resultUrls[0]`. Maps safety failures to `SafetyFilterError`.
176
- ⚠️ **Naming mismatch worth flagging:** the provider is called "nano-banana-pro" but sends the `gpt-image-2-text-to-image` slug. Kie *also* offers real `nano-banana` and `seedream-4` image models — if Scene Frames wants Nano Banana quality, it must send the actual nano-banana slug, not this provider as-is.
177
- - **`src/lib/meme/kie.ts`** — meme generator's image client. Same `gpt-image-2-text-to-image` slug, but sends `{ prompt, aspect_ratio:"1:1", resolution:"1K", output_format:"PNG" }`. Strict BYOK (member's own Kie key, no env fallback), 5-min poll timeout, one-shot 429 backoff.
178
-
179
- **Endpoint shape confirmed in-repo:** `POST https://api.kie.ai/api/v1/jobs/createTask` + `GET https://api.kie.ai/api/v1/jobs/recordInfo?taskId=...`, Bearer auth, `resultJson.resultUrls[0]`. **This is the same transport a Video-node would use** — the Scene Frames stage can reuse this createTask+poll helper wholesale and only swap the model slug + input schema (and should pass `aspect_ratio`/`image_size` to models that honor them, unlike gpt-image-2).
180
-
181
- Other Kie touch-points in the working tree (for reference): `convex/pipelineKeys.ts`, `convex/userKeys.ts`, `convex/lib/byok.ts` (BYOK key storage/resolution), `convex/doctor.ts` (key health check), `src/app/(dashboard)/settings/_components/pipeline-keys-tab.tsx` (Settings → Keys UI).
182
-
183
- ---
184
-
185
- ## Sources
186
-
187
- Primary (Kie):
188
- - https://docs.kie.ai/ (Getting Started — API pattern, rate limit 20/10s)
189
- - https://docs.kie.ai/market/quickstart (unified createTask / recordInfo / callBackUrl)
190
- - https://docs.kie.ai/market/common/get-task-detail
191
- - https://docs.kie.ai/veo3-api/generate-veo-3-video , /veo3-api/quickstart , /veo3-api/extend-video (Veo 3.1)
192
- - https://docs.kie.ai/market/kling/kling-3-0 , /market/kling/motion-control-v3 (Kling 3.0)
193
- - https://docs.kie.ai/market/bytedance/seedance-2 , /market/bytedance/seedance-2-fast (Seedance 2.0)
194
- - https://docs.kie.ai/market/sora2/... (Sora 2 — index confirmed; per-page params [verify])
195
- - https://docs.kie.ai/market/hailuo/2-3-image-to-video-pro (Hailuo 2.3)
196
- - https://docs.kie.ai/runway-api/quickstart (Runway)
197
- - https://kie.ai/ , https://kie.ai/market , https://kie.ai/pricing , https://kie.ai/v3-api-pricing (403 to automated fetch — pricing via secondary corroboration)
198
-
199
- Pricing corroboration (secondary, anchored to Kie's $0.005/credit):
200
- - https://skywork.ai/skypage/en/Kie.ai-API...1976113187525816320 (credit rate $0.005; error codes 401/402/429/500)
201
- - https://devtk.ai/en/blog/ai-video-generation-pricing-2026/
202
- - https://fluxnote.io/guides/ai-video-model-pricing-comparison-2026
203
- - https://evolink.ai/blog/best-ai-video-generation-models-2026-pricing-guide
204
-
205
- Local code (this repo):
206
- - Working tree: `src/lib/modules/providers/image-generator/kie-nano-banana.ts`, `src/lib/meme/kie.ts`
207
- - Archive tag: `archive/pixar-v2:scout/src/pixar/kie-client.ts`, `archive/pixar-v2:scout/src/image-ads/image-gen.ts`
@@ -1,217 +0,0 @@
1
- # Lip-Sync Alternatives — getting a CHOSEN voice out of the on-screen actor's mouth
2
-
3
- **Research date:** 2026-07-14
4
- **For:** Unblocking voice-paired talking-head scene clips after Kie's `gemini-omni-video` `audio_ids` path started returning "flagged by Website as violating content policies" on every render (GitHub issue #561).
5
- **Goal:** A chosen voice — ideally an ElevenLabs voice the user already uses — spoken by the on-screen actor, lip-synced, in each per-scene clip (usually 9:16, 720p, 4–10 s), then stitched as today.
6
- **Method:** Primary sources = first-party API docs (ai.google.dev, sync.so/docs, fal.ai model pages, developers.heygen.com, d-id.com, hedra.com, replicate.com) plus Kie's own pages. Kie's marketing pages (`kie.ai/*`) and `kie.ai/market` return **HTTP 403 to automated fetch**; several `docs.kie.ai/market/...` model pages 404 at the guessed path, so some Kie-specific slugs/prices are corroborated from mirror hosts (fal, WaveSpeed, Replicate, Atlas Cloud) that run the same upstream model and are flagged accordingly. Anything not pinned to a first-party doc in this session is marked **[unverified]**.
7
- **Confidence labels:** `[docs]` = first-party API doc · `[mirror]` = same upstream model on another host's doc (fal/Replicate/WaveSpeed) · `[vendor]` = vendor marketing/help page · `[unverified]` = could not confirm from a primary source this session.
8
-
9
- ---
10
-
11
- ## TL;DR — recommendation table
12
-
13
- The two integration shapes matter more than the model list:
14
-
15
- - **RETARGET** = feed an *already-rendered* scene clip (video) + an audio track → model reanimates only the mouth, preserves background/body/camera. **Drops into the existing per-scene render → stitch flow as a new post-render step. No change to the video model.** This is the fit for our pipeline.
16
- - **GENERATE** = feed an *image* (a portrait / first frame) + audio → model animates a talking head from scratch. This **replaces** the Kie video model for talking scenes (loses the motion/scene the video model produced).
17
-
18
- | Rank | Path | Shape | Existing BYOK? | ~Cost, 9:16 720p, ~8 s | Voice source | Moderation risk | Verdict |
19
- |---|---|---|---|---|---|---|---|
20
- | **1** | **Kie `volcengine/video-to-video-lip-sync`** (retarget rendered clip) | RETARGET | ✅ **Kie BYOK exists — same jobs API** | not documented — surfaces via `creditsConsumed` `[docs]` | any audio file (our Voiceover node's ElevenLabs MP3s, ≤10 MB) | Low (edits our own footage) — **live-probe before building** | **Zero new provider; verified post-research (see addendum). Try first.** |
21
- | **2** | **sync.so `lipsync-2`** (retarget rendered clip) | RETARGET | ❌ new provider (but built-in ElevenLabs TTS) | ~**$0.32** ($0.04/s) `[docs]` | video's own audio **or** ElevenLabs voiceId+script native in-API | Low (edits our own footage) | **Best quality reputation; fallback if Volcengine quality/pricing disappoints** |
22
- | **2b** | **Kling Lip-Sync A2V** (`kwaivgi/kling-lip-sync`, retarget) via fal/Replicate | RETARGET | ❌ new provider (NOT in Kie's catalog — verified via docs index) | ~**$0.11–0.15**/run ($0.014/input-s, 5 s increments) `[mirror]` | any audio file (drop in ElevenLabs MP3) | Low | Cheapest retarget, but needs fal/Replicate BYOK |
23
- | 3 | **fal `veed/lipsync`** / **`fal-ai/latentsync`** (retarget) | RETARGET | ❌ new provider | veed ~$0.24 ($0.15/5 s); latentsync **$0.20 flat ≤40 s** `[docs]` | any audio file | Low | Cheap retarget fallbacks; latentsync is the price floor |
24
- | 4 | **Kie InfiniteTalk** (`infinitalk/image-to-talking-video`) | GENERATE | ✅ **Kie BYOK exists** | ~**$0.30**/clip (10 credits/5 s @720p) `[vendor]` | uploaded audio (ElevenLabs MP3), ≤15 s | Low–med | **Only option that reuses existing BYOK — but replaces the video model, image-driven** |
25
- | 5 | **Kie OmniHuman 1.5** (`bytedance/omni-human`) | GENERATE | ✅ Kie BYOK exists | ~$0.12/s upstream `[mirror]` | uploaded audio | Low–med | Higher-fidelity generate-from-photo on existing BYOK |
26
- | 6 | Hedra Character-3 | GENERATE | ❌ new provider | subscription credits, ~$15–75/mo | uploaded audio / text+TTS | Low | Best-in-class generate quality; sub model, not per-clip PAYG |
27
- | 7 | HeyGen / D-ID photo-avatar | GENERATE | ❌ new provider | ~$0.05/s ($3/min) `[vendor]` | uploaded audio or TTS | Low | Presenter/UGC avatars; least "our footage" |
28
- | — | **Google Gemini Omni direct** (bypass Kie) | GENERATE, **no audio input** | ❌ new provider | ~$0.10–0.75/s (Veo tiers) `[docs]` | **none — cannot supply a voice** | n/a | **Ruled out: API explicitly forbids audio upload & voice editing** |
29
- | — | Seedance `reference_audio_urls` (Kie) | (not lip-sync) | ✅ Kie | — | voice-style *reference*, not spoken content | — | **Not a lip-sync/voice-content lever — see note** |
30
-
31
- **Bottom line:** the goal ("chosen ElevenLabs voice, lip-synced, on the actor already in our rendered clip") is a **RETARGET** problem, and the video-generation APIs (including Gemini Omni direct) don't solve it. Rank 1–3 keep the Kie video model exactly as-is and add one post-render lip-sync call per scene, then mux/stitch. **sync.so is the standout** because it retargets our rendered clip *and* has ElevenLabs TTS built into the same request. If avoiding a brand-new provider is paramount, Kie InfiniteTalk (rank 4) is the only path on existing BYOK — but it's generate-from-image, so it changes what the talking scenes look like.
32
-
33
- ---
34
-
35
- ## Branch 1 — Google Gemini / Omni video API directly (bypass Kie)
36
-
37
- **Source (first-party):** https://ai.google.dev/gemini-api/docs/omni · pricing https://ai.google.dev/gemini-api/docs/pricing
38
-
39
- ### Does it accept audio input for the generated speech / lip-sync? **No.**
40
- The Gemini Omni video docs state plainly `[docs]`:
41
- - *"Uploading audio references is unsupported in the current version of the API."*
42
- - *"Voice editing is not supported."*
43
-
44
- The documented request accepts only: **text** (`{"type":"text","text":"…"}`), **images** (`{"type":"image","data":<base64>,"mime_type":"image/jpeg"}`), **videos for editing** (`{"type":"document","uri":<video_file_uri>}`), and **prior generated-video references** via a `previous_interaction_id` parameter. **There is no audio input parameter anywhere in the schema.** `system instructions`, `temperature`, `top_p`, and `negative prompts` are also unsupported (negatives must be embedded in the prompt). `[docs]`
45
-
46
- **Consequence:** the first-party API **cannot** take an ElevenLabs voice, an uploaded audio file, or a voice character to drive speech/lip-sync. It generates whatever voice it wants natively. This is the same capability gap that Kie's `audio_ids` param was papering over — and going direct removes the param entirely. **Branch 1 does not meet the goal.**
47
-
48
- ### Moderation — and why Kie's flag is Google-side
49
- The Omni docs document these safety controls `[docs]`:
50
- - *"Content safety filters are applied to both input prompts and generated video (and depend on your region)."*
51
- - *"Prompts that violate usage policies will be blocked."*
52
- - *"Uploading and editing images containing certain recognizable people is not supported."*
53
- - *"Uploading and editing images containing minors is not supported in [the] EEA, Switzerland, and the United Kingdom."*
54
- - English is *"fully supported"*; other languages *"have not been evaluated."*
55
-
56
- The "flagged by **Website**" wording our probes saw, combined with "prompts that violate usage policies will be blocked" applying to the *underlying Gemini/Omni model*, is consistent with a **Google-side content-policy block on the audio/voice pathway**, not a Kie quirk — Kie's `audio_ids` (preset voice characters) likely routes into a voice feature Google's safety layer rejects for this prompt class, or Kie is bolting on a voice step the first-party model doesn't sanction. Since the first-party API forbids audio input outright, there is **no direct-API config that re-enables voice-paired renders**. `[docs]` + reasoned inference.
57
-
58
- ### Access, pricing, availability
59
- - **Video models on the Gemini API are the Veo family**, billed per second of output. Google's list rate for **Veo 3.1** is **$0.75/s** (~$6.00 / 8 s clip) on the Gemini API and Vertex; **Veo 3.0** was $0.50/s video-only / $0.75/s with audio (deprecated, shutdown 2026-06-30). A **standard-tier 720p** path bills ~**5,792 tokens/s of 720p video ≈ $0.10/s** [unverified exact tier mapping]. `[docs]` (pricing page) + `[vendor]` calculators.
60
- - **Vertex AI vs Gemini API:** same Veo models, same "no audio upload" capability for generation; Vertex adds enterprise auth/quota but does **not** add an audio-input lip-sync mode [unverified beyond Veo parity].
61
- - **vs Kie:** Kie's ~63 credits (~$0.32) per 4 s 720p clip is competitive with — often cheaper than — Google-direct Veo, and Kie's unified async job API is already integrated. Going direct means a **new "Google AI" BYOK provider**, a Files API upload step for image inputs, and Google's own async long-running-operation polling — **material new surface for zero gain on the voice goal.**
62
-
63
- **Branch 1 verdict: ruled out.** The capability we need (supply-your-own-voice / lip-sync to given audio) is explicitly not in the first-party API. Do not build a Google-direct integration to solve #561.
64
-
65
- ---
66
-
67
- ## Branch 2 — lip-sync-to-audio models (render clip → ElevenLabs MP3 → retarget the mouth)
68
-
69
- Ordered by fit. **RETARGET** models take our rendered clip; **GENERATE** models take a still image.
70
-
71
- ### 2a. sync.so (Sync Labs) — `lipsync-2` / `lipsync-2-pro` / `sync-3` — **RETARGET** ✅ best fit
72
- **Source (first-party):** https://sync.so/docs/api-reference/api/generate-api/create · https://sync.so/docs/models/lipsync · https://sync.so/docs/introduction
73
-
74
- - **Shape:** retargets an **existing video** with new audio — *"retargets existing video with new audio, applying lipsync synchronization to match speaker mouth movements."* `[docs]`
75
- - **Inputs (`CreateGenerationDto`):** an array requiring **exactly one visual input** (video `url` **or** `assetId`; image also accepted) **and one audio-or-text input** `[docs]`:
76
- - Audio as `{ "url": … }` or `assetId`, **or**
77
- - **TTS**: `{ "provider": "elevenlabs", "voiceId": <ElevenLabs voice id>, "script": <text>, "stability", "similarityBoost" }` — **native ElevenLabs integration in the same request.** `[docs]`
78
- - **Models:** `sync-3`, `lipsync-2`, `lipsync-2-pro`, `lipsync-1.9.0-beta`, `react-1`. `[docs]`
79
- - **Options:** `sync_mode` (bounce/loop/cut_off/silence/remap for duration mismatch), `temperature` (0–1, default 0.5, expressiveness), `active_speaker_detection`, `occlusion_detection_enabled`, `webhookUrl`. `[docs]`
80
- - **API pattern:** **async** — POST returns `status: PENDING` → `PROCESSING` → `COMPLETED|FAILED|REJECTED`; poll or webhook. Long videos auto-chunked into 30–40 s segments (irrelevant for our 4–10 s clips). `[docs]`
81
- - **Pricing:** **`lipsync-2` = $0.04/s**, `sync-3` = $0.133/s (at 25 fps). ~**$0.32 for an 8 s clip** on lipsync-2. `[docs]` (pricing page / cost-estimation endpoint available).
82
- - **Moderation:** `REJECTED` state exists for policy violations [unverified what triggers it]; editing our *own* generated footage is far lower risk than generating a likeness.
83
- - **Fit:** ⭐ Perfect. Keep the Kie video model; after each scene renders, call sync.so with `video_url = <rendered clip>` and either the ElevenLabs MP3 we already generate **or** `provider:elevenlabs, voiceId, script` and skip our own TTS step. Output replaces the scene clip pre-stitch.
84
-
85
- ### 2b. Kling Lip-Sync — Audio-to-Video (`kwaivgi/kling-lip-sync`) — **RETARGET** ✅ cheap
86
- **Sources:** fal API doc https://fal.ai/models/fal-ai/kling-video/lipsync/audio-to-video/api `[docs]` · Replicate https://replicate.com/kwaivgi/kling-lip-sync `[mirror]` · WaveSpeed https://wavespeed.ai/models/kwaivgi/kling-lipsync/audio-to-video `[mirror]`
87
-
88
- - **Shape:** *"give it a clean voice track plus a video and it reanimates the mouth region… preserves the original background, body motion, and camera work."* Works on Kling-generated **or other-source** video. `[mirror]`
89
- - **Inputs:** `video_url` + `audio_url` (also a text→TTS variant). `[docs]`
90
- - **Video:** .mp4/.mov, ≤100 MB, **2–10 s** (some hosts list 2–60 s), **720p/1080p only**, width/height 720–1920 px. `[docs]`
91
- - **Audio:** 2–60 s, ≤5 MB, MP3/WAV/AAC/OGG. `[docs]`
92
- - **API pattern:** async queue + webhooks. `[docs]`
93
- - **Pricing:** ~**$0.15/run minimum**, or **$0.014 per input-video-second rounded to the nearest 5 s** → ~$0.11 for a 4–8 s clip. `[mirror]` (WaveSpeed/fal)
94
- - **Moderation:** marked "Commercial use / Partner"; no documented person-restriction policy. `[docs]`
95
- - **Kie hosting:** **[unverified]** — Kie hosts multiple Kling models (`kling-3.0/video` etc.) but I could **not** confirm this session that Kie exposes the *video-to-video* `kling-lip-sync` retarget slug (its `docs.kie.ai/market/kling/lip-sync` path 404'd). **If BYOK-Kie already exposes it, this becomes the top pick** (existing provider + cheap + retarget). Worth a live probe against the Kie account.
96
- - **Fit:** ⭐ Excellent and cheapest solid retarget. 10 s cap suits our clip length.
97
-
98
- ### 2c. fal.ai catalog (retarget options)
99
- **Source:** https://fal.ai/models/veed/lipsync · https://fal.ai/models/fal-ai/latentsync · https://fal.ai/models/fal-ai/sync-lipsync
100
- - **`veed/lipsync`** — RETARGET (video→video), *"realistic lipsync from any audio."* **$0.15 / 5 s** (~$0.24/8 s). `[docs]`
101
- - **`fal-ai/latentsync`** (ByteDance LatentSync, audio-conditioned latent diffusion) — RETARGET. **$0.20 flat for ≤40 s**, then $0.005/s → cheapest for our clip lengths. Real-life + anime. `[docs]`
102
- - **`fal-ai/sync-lipsync` (1.9)** — RETARGET, **$0.70/min ≈ $0.012/s** (sync.so model, older). `[docs]`
103
- - **`fal-ai/kling-video/lipsync/audio-to-video`** — the Kling retarget above, on fal. `[docs]`
104
- - **Fit:** all RETARGET, all take our rendered clip + any audio (drop the ElevenLabs MP3). fal is a single BYOK that unlocks several — good hedge if we want to A/B lip-sync engines. latentsync is the price floor; veed is a quality step up.
105
-
106
- ### 2d. Kie InfiniteTalk (`infinitalk/image-to-talking-video`) — **GENERATE** ✅ only path on existing BYOK
107
- **Sources:** https://kie.ai/infinitalk `[vendor]` (403 to fetch; corroborated) · integration guide https://www.fromdev.com/2025/11/how-to-integrate-infinite-talk-api-using-kie-ai-a-step-by-step-guide.html · FairStack model index (price)
108
- - **Shape:** MeiGen-AI InfiniteTalk — turns an **image + audio** into a talking avatar (sparse-frame video-dubbing framework, identity-preserving, "infinite" length). **Generate-from-image, not retarget-a-clip.** `[vendor]`
109
- - **Inputs (via `POST https://api.kie.ai/api/v1/jobs/createTask`):** `image_url` (JPEG/PNG/WebP), `audio_url` (MP3/WAV/AAC/OGG, **≤15 s**), `prompt`, `resolution` **480p or 720p only**, `seed` (10000–1000000). `[mirror]` (integration guide reflecting Kie's schema)
110
- - **Pricing:** first 5 s = **5 credits @480p / 10 @720p / 15 @1080p**, then per-second at the same rate → ~**$0.30/clip @720p** (FairStack lists "InfiniteTalk (Kie.ai) — $0.300/clip"). `[vendor]`
111
- - **Fit:** ✅ **Only option that reuses the existing Kie BYOK key.** BUT it's image-driven: we'd feed a scene *frame* (or actor portrait) + the ElevenLabs MP3, and it **regenerates** the talking head — we lose whatever motion/scene the Kie video model produced. Good if talking scenes can be "portrait talks to camera"; wrong if the scene needs the video model's action. 15 s audio cap fits our 4–10 s clips.
112
-
113
- ### 2e. Kie OmniHuman 1.5 (`bytedance/omni-human`) — **GENERATE**
114
- **Sources:** https://kie.ai/omnihuman-1-5 · https://kie.ai/omni-human-api `[vendor]` (403; corroborated) · fal/upstream pricing `[mirror]`
115
- - **Shape:** ByteDance OmniHuman 1.5 — single portrait **image + audio** → realistic talking video with natural lip-sync, expressions, lifelike/full-body motion. Generate-from-image. `[vendor]`
116
- - **Pricing:** upstream ~**$0.12/s** (fal listing for OmniHuman). Kie bills by generated duration; exact Kie credit/s [unverified]. `[mirror]`
117
- - **Fit:** Higher-fidelity generate-from-photo than InfiniteTalk, on existing Kie BYOK. Same caveat: replaces the video model for talking scenes. Kling AI Avatar 2.0 is a third Kie generate-from-image option (audio-driven, multilingual, up to 5 min; Kie benchmarks it above OmniHuman-1.5 and HeyGen). `[vendor]`
118
-
119
- ### 2f. Runway (Act-Two / Lip Sync / Characters API)
120
- **Sources:** https://help.runwayml.com/hc/en-us/articles/42311337895827 · https://runwayml.com/apps/add-dialogue · changelog `[vendor]`
121
- - **Act-Two:** performance-capture (driving video → target character), auto audio-visual sync; you can **change the voice** within the interface. **Lip Sync** app extended to **45 s** (from 20 s). Characters API is **live for developers at dev.runwayml.com**. `[vendor]`
122
- - **Shape:** closest Runway analog to retarget is Lip Sync / "Add Dialogue" (animate a video/image with a voice). API is available (Gen-4/Characters on dev.runwayml.com) but **per-second API pricing for Act-Two/Lip Sync was not published** this session. `[vendor]` / [unverified pricing]
123
- - **Fit:** Plausible RETARGET/talking-photo path, but pricing opacity and a heavier API make it a second-tier choice vs sync.so/fal/Kling.
124
-
125
- ### 2g. Hedra Character-3 — **GENERATE**
126
- **Sources:** https://www.hedra.com/docs/pages/getting_started/quickstart · https://www.hedra.com/video-models/hedra-character-3 · https://www.hedra.com/plans `[vendor]`
127
- - **Shape:** omnimodal — processes **image + text + audio simultaneously** → talking avatar with strong lip-sync, micro-expressions (blinks, eye shifts, head tilts), full-body motion. Generate-from-image. `[vendor]`
128
- - **Access/pricing:** requires a **paid subscription** (API key + purchased API credits). Free (100 cr, watermarked); Basic $15/mo (1,500 cr); Creator $30/mo (5,400 cr); Professional $75/mo (14,400 cr). **Credit/subscription model, not clean per-clip PAYG.** `[vendor]`
129
- - **Fit:** Best-in-class *generate* quality for talking heads, but subscription billing fits a per-user BYOK-PAYG pipeline poorly, and it's still generate-from-image (replaces the video model).
130
-
131
- ### 2h. HeyGen API / D-ID — **GENERATE** (presenter avatars)
132
- **Sources:** https://developers.heygen.com/ · https://help.heygen.com/en/articles/10060327 · https://www.d-id.com/pricing/api/ `[vendor]`
133
- - **HeyGen:** Photo Avatar from a still → animates face, **syncs lips to your script or an uploaded audio file** ("dub or replace audio on a video with a provided audio file"). Pay-as-you-go since Feb 2026: **~$1 = 1 min @720/1080p**; engines from **$0.0167/s** (Avatar III Digital Twin) to **$0.05/s** (Avatar IV Photo Avatar) to $0.0667/s. Min top-up $5. `[vendor]`
134
- - **D-ID:** talking-head from a photo + audio/TTS, API **~$0.05/s ($3/min)**. `[vendor]`
135
- - **Fit:** These are *presenter/spokesperson* generators from a photo — they don't retarget our rendered scene, and outputs read as "avatar tool," which is off for naturalistic UGC scene action. Use only if a talking-portrait scene is acceptable and we want a polished presenter look. HeyGen's "replace audio on a video" mode is worth a second look as a possible retarget-ish path [unverified whether it accepts arbitrary external video].
136
-
137
- ### 2i. Replicate-hosted options
138
- **Source:** https://replicate.com/collections/lipsync `[docs]`
139
- Replicate's lipsync collection hosts the same upstream models as first-party, each with a documented JSON schema and PAYG billing: **`sync/lipsync-2-pro`**, **`kwaivgi/kling-lip-sync`** (retarget), **`bytedance/omni-human`** (generate), **`pixverse-ai/lipsync`**, **`zsxkib/multitalk`** (multi-person conversational lipsync — multiple audio clips → back-and-forth). Good single-BYOK hedge (one Replicate key → several retarget + generate engines) but generally pricier/less predictable than fal for the same models.
140
-
141
- ---
142
-
143
- ## Seedance `reference_audio_urls` note (Kie `bytedance/seedance-2`)
144
-
145
- Per Kie's Seedance 2.0 doc (already captured in `native-audio-dialogue.md`, source https://docs.kie.ai/market/... Seedance page): `reference_audio_urls` accepts **≤3 clips, 2–15 s, wav/mp3** and is a **voice/style *reference* lever — NOT a documented dialogue-content or lip-sync-to-given-audio field.** `[docs]` It biases the *character/timbre* of the model's natively-generated speech; it does **not** make the model speak a supplied MP3 verbatim, and it does not lip-sync to an external audio track. **It does not solve the goal** (a chosen ElevenLabs voice saying our exact lines, lip-synced). Treat Seedance native audio as "model invents a voice, loosely style-guided," same class of problem as gemini-omni — not a voice-control solution.
146
-
147
- ---
148
-
149
- ## Recommended path + integration sketch
150
-
151
- ### Primary recommendation: add a post-render **RETARGET lip-sync step**, keep the Kie video model
152
- The pipeline already produces per-scene clips and per-scene ElevenLabs MP3s (George default) — they're just not muxed/synced. The clean fix is a new stage between "scene clip rendered" and "stitch":
153
-
154
- ```
155
- storyboard → [Kie video model renders scene clip (silent or discard native audio)]
156
- → [ElevenLabs TTS: scene.dialogue → MP3] (already exists)
157
- → [LIP-SYNC RETARGET: clip + MP3 → mouth-synced clip] ← NEW
158
- → [ffmpeg stitch] (already exists, now with real synced VO)
159
- ```
160
-
161
- This is **additive** — no change to the storyboard, the video-model picker, or the stitch stage — and it finally makes the existing ElevenLabs Voiceover node pay off.
162
-
163
- **Engine choice, in order:**
164
- 1. **sync.so `lipsync-2`** — retargets the rendered clip and can take **either** our ElevenLabs MP3 **or** an inline `provider:elevenlabs, voiceId, script` TTS block (letting us drop our own TTS call). Async job pattern mirrors Kie's (create → poll/webhook), so it slots into the existing async render machinery. ~$0.32/8 s clip. New BYOK: a sync.so API key. **Start here.**
165
- 2. **Before committing to a new provider, live-probe whether BYOK-Kie already exposes `kwaivgi/kling-lip-sync` (video-to-video).** If yes, it's the cheapest retarget (~$0.11–0.15/clip) on the *existing* Kie key — zero new provider surface. `[unverified]` this session; a 1-call probe settles it.
166
- 3. **fal (`veed/lipsync` or `fal-ai/latentsync`)** as a fallback/second engine — one fal BYOK unlocks several retarget models for A/B'ing quality; latentsync at $0.20/≤40 s is the price floor.
167
-
168
- ### If a new provider is a hard no (must stay on existing Kie/ElevenLabs BYOK only)
169
- Use **Kie InfiniteTalk** (`infinitalk/image-to-talking-video`) or **OmniHuman** (`bytedance/omni-human`): feed a **scene frame/portrait + the ElevenLabs MP3** → talking clip, on the existing Kie key (~$0.30/clip @720p, 720p cap, ≤15 s audio for InfiniteTalk). **Accept the tradeoff:** these *generate* the talking head from a still, so talking scenes become "portrait speaks to camera" and lose the video model's motion/action. Fine for direct-address UGC beats; wrong for scenes that need staged action.
170
-
171
- ### Integration considerations (both paths)
172
- - **Async + polling/webhook**, same as the current Kie render loop — reuse it. sync.so and Kie share the create→poll shape; sync.so adds a cost-estimation endpoint.
173
- - **Resolution/duration:** retarget models cap at 720p/1080p and ~10 s (Kling) or auto-chunk (sync.so) — fine for our 4–10 s 9:16 720p clips. InfiniteTalk caps at 720p + 15 s audio.
174
- - **Cost delta:** retarget adds ~$0.04/s (sync) or ~$0.014/s (Kling) *on top of* the ~$0.32/4 s Kie render — call it +$0.15–0.35/scene. A 6-scene ad adds roughly $1–2.
175
- - **Moderation:** retargeting **our own rendered footage** is materially lower policy-risk than the gemini-omni `audio_ids` path that #561 hit, and far lower than uploading a real person's likeness — the whole reason this route dodges the block.
176
- - **Voice fidelity:** all retarget engines take an arbitrary MP3, so the **user's exact ElevenLabs voice** is preserved end-to-end (we generate it, they lip-sync to it) — directly satisfying "a chosen ElevenLabs voice out of the actor's mouth."
177
-
178
- ---
179
-
180
- ## Source list (primary first)
181
- - Gemini Omni API (no audio input, safety): https://ai.google.dev/gemini-api/docs/omni
182
- - Gemini/Veo pricing: https://ai.google.dev/gemini-api/docs/pricing
183
- - sync.so create-generation schema: https://sync.so/docs/api-reference/api/generate-api/create · models https://sync.so/docs/models/lipsync · intro https://sync.so/docs/introduction
184
- - fal Kling lipsync A2V: https://fal.ai/models/fal-ai/kling-video/lipsync/audio-to-video/api
185
- - fal veed/lipsync: https://fal.ai/models/veed/lipsync · latentsync: https://fal.ai/models/fal-ai/latentsync · sync-lipsync: https://fal.ai/models/fal-ai/sync-lipsync
186
- - Replicate lipsync collection: https://replicate.com/collections/lipsync · kling-lip-sync: https://replicate.com/kwaivgi/kling-lip-sync
187
- - WaveSpeed Kling lipsync (specs/price): https://wavespeed.ai/models/kwaivgi/kling-lipsync/audio-to-video
188
- - Kie InfiniteTalk: https://kie.ai/infinitalk · integration guide: https://www.fromdev.com/2025/11/how-to-integrate-infinite-talk-api-using-kie-ai-a-step-by-step-guide.html
189
- - Kie OmniHuman: https://kie.ai/omnihuman-1-5 · https://kie.ai/omni-human-api
190
- - Hedra Character-3: https://www.hedra.com/video-models/hedra-character-3 · quickstart https://www.hedra.com/docs/pages/getting_started/quickstart · plans https://www.hedra.com/plans
191
- - HeyGen API: https://developers.heygen.com/ · pricing https://help.heygen.com/en/articles/10060327-heygen-api-pricing-explained
192
- - D-ID API pricing: https://www.d-id.com/pricing/api/
193
- - Runway Act-Two: https://help.runwayml.com/hc/en-us/articles/42311337895827-Performance-Capture-with-Act-Two · add-dialogue https://runwayml.com/apps/add-dialogue
194
- - Seedance `reference_audio_urls`: see workspace/references/native-audio-dialogue.md (Kie Seedance 2.0 doc)
195
-
196
- ---
197
-
198
- ## Post-research verification addendum (main session, 2026-07-14)
199
-
200
- Two load-bearing claims were re-verified against primary sources after the research pass:
201
-
202
- ### Kie DOES host a video-to-video lip-sync retarget model — `volcengine/video-to-video-lip-sync` `[docs]`
203
- Found via Kie's docs index (https://docs.kie.ai/llms.txt); full doc: https://docs.kie.ai/market/volcengine/video-to-video-lip-sync.md
204
-
205
- - **Shape:** RETARGET — required inputs are `mode` (`lite` = single-person frontal, faster; `basic` = complex scenes + scene segmentation), `video_url`, `audio_url`. Runs on the SAME `POST /api/v1/jobs/createTask` transport the pipeline already speaks, on the existing Kie BYOK key.
206
- - **Fit:** our scene clips are single-person frontal talking heads (→ `lite`); audio accepts MP3 ≤10 MB — exactly what the existing Voiceover node's ElevenLabs synthesis produces per scene. `align_audio` loops video if audio outruns it; output is mp4 25 fps with duration matched to the audio.
207
- - **Video constraints:** 360p–1080p (ours: 720p ✓), H.264, ≤500 MB, 24–60 fps.
208
- - **Unknowns:** pricing is NOT documented (Kie convention: real cost surfaces via `creditsConsumed` on the record — live-probe it) and output quality is unassessed. Given the #561 surprise, run a live probe (rendered clip + ElevenLabs MP3) before building the pipeline leg.
209
- - This supersedes the earlier "[unverified] Kie hosting of Kling lip-sync" row: Kie's docs index shows NO Kling A2V lip-sync slug — the Kling entries are `kling/ai-avatar-standard|pro` (GENERATE-from-image), plus `omnihuman-1-5` and `infinitalk/from-audio` (both GENERATE).
210
-
211
- ### sync.so ElevenLabs TTS input — confirmed `[docs]`
212
- https://sync.so/docs/api-reference/api/generate-api/create — generation request accepts `{type: "text", provider: {name: "elevenlabs"}, voiceId: <ElevenLabs voice id>, script}` alongside a `{type: "video", url}` retarget input; models `sync-3`, `lipsync-2`, `lipsync-2-pro`, `lipsync-1.9.0-beta`, `react-1`. (Note: our pipeline already synthesizes ElevenLabs MP3s in the Voiceover node, so the built-in TTS is a convenience, not a requirement — which strengthens the case for trying Kie/Volcengine first.)
213
-
214
- ### Revised recommendation
215
- 1. **Live-probe Kie `volcengine/video-to-video-lip-sync`** (existing BYOK, existing transport, existing ElevenLabs VO): render one talking-head clip (native audio OFF, mouth policy suppressed), synthesize one ElevenLabs line, submit lite-mode retarget, judge quality + read `creditsConsumed`.
216
- 2. If quality or price disappoints → **sync.so `lipsync-2`** (new BYOK provider, ~$0.04/s, strongest reputation).
217
- 3. Integration shape either way: Video node gains a "voiceover lip-sync" mode = per-scene silent-ish render → Voiceover MP3 → retarget call → retargeted clip replaces the raw clip in the stitch. The storyboard's speech-fit budget already paces voText to scene durations.
@@ -1,167 +0,0 @@
1
- # Meta Ads MCP — live probe findings (wayfinder #563)
2
-
3
- Probed 2026-07-14 against Brad's real accounts via the **official Meta Ads MCP**
4
- (`https://mcp.facebook.com/ads`, remote HTTP MCP, OAuth-in-browser). All calls
5
- read-only. Primary probe account: **Matt Beard FPI** (act `…820395`, ~$62.9k
6
- spend / 1.07M impressions in the trailing 90 days). Feeds the enrichment-path
7
- ticket (#565) and the bridge contract (#566) on map #562.
8
-
9
- ## TL;DR for the specs
10
-
11
- 1. **The winner query works end-to-end**: per-ad `amount_spent`, `results`
12
- (labeled with the result type), `cost_per_result`, `ctr`, `cpm`,
13
- `impressions`, `lead`, `purchase_roas` over any date preset or custom
14
- range, sortable by any metric. A "top ads by spend, then threshold on
15
- cost-per-result" winner definition is executable **agent-side over sorted
16
- results** — server-side *metric filtering* consistently 500s (see gotchas).
17
- 2. **The Ad Library cross-reference exists but is fuzzy**: `ads_library_search`
18
- by `page_ids` returns the public **`adArchiveId` directly** (the result `id`;
19
- `ad_snapshot_url` = `facebook.com/ads/library/?id=<archiveId>`). But library
20
- results carry only `ad_creative_link_title` + creation/delivery timestamps —
21
- **no body text, no ad-account ad id** — so account-ad → library-ad mapping is
22
- a **fingerprint match** (creative `title` == library `ad_creative_link_title`,
23
- ± creation-time proximity), not a direct key. No tool/field exposes
24
- `adArchiveId` from the account side.
25
- 3. **Images are fully retrievable; videos are not.** Image detail gives a
26
- signed full-res CDN `url` (+ `url_128`, `permalink_url`, dimensions) —
27
- downloadable now, but the URL **expires** (`oe=` param). Video detail gives
28
- `length`, poster `picture`, and a *relative* `permalink_url` — **no source /
29
- download URL**. This confirms the map's scrape-first asset strategy, with
30
- MCP image-pull as gap-filler and **no MCP gap-filler for video files**.
31
- 4. **Naming comes through intact** at every level (`name` on ad/adset/campaign,
32
- creative `name`, video `title` — titles even encode aspect ratios like
33
- `-9x16`). Naming-conventions guidance can rely on free-text ad names plus
34
- `campaign_id`/`adset_id` for hierarchy grouping.
35
- 5. **Setup surface is one URL**: add `https://mcp.facebook.com/ads` as a remote
36
- MCP server; OAuth happens in the browser with the user's own Facebook login.
37
- Per-account gates: `is_ads_mcp_enabled` and `is_queryable` (disabled/flagged
38
- accounts come back with a human-readable `not_queryable_reason`).
39
-
40
- ## Tool-by-tool findings
41
-
42
- ### `ads_get_ad_accounts`
43
- Returns **every** account the user can access — Brad sees 13 across 5
44
- businesses (own + client accounts), each with `ad_account_id`,
45
- `ad_account_name`, owning `business_id`/`business_name`, `account_status`,
46
- `currency`, `is_ads_mcp_enabled`, `is_queryable`, `not_queryable_reason`.
47
- Two disabled accounts surfaced with the flag reason string. **Multi-account
48
- selection is a real UX step the skill must handle** (fog item on the map,
49
- now sharpened). Account names can be empty strings.
50
-
51
- ### `ads_get_ad_entities` (the insights workhorse)
52
- - Levels: `account` / `campaign` / `adset` / `ad`. Metrics require
53
- `date_preset` (e.g. `last_90d`, `maximum`) or `time_range` JSON;
54
- `time_increment` gives time series.
55
- - Real sample (ad level, last_90d, sorted `amount_spent_descending`):
56
-
57
- ```json
58
- {
59
- "id": "120241083993850557",
60
- "name": "PT-Intro-me-and-ai-Image-Promoted",
61
- "effective_status": "ADSET_PAUSED",
62
- "creative_id": "952613677477713",
63
- "campaign_id": "120239743848310557",
64
- "adset_id": "120241083975980557",
65
- "amount_spent": "$8,980.38 USD",
66
- "results": {"value": "98 (Website appointments scheduled)"},
67
- "cost_per_result": {"value": "$91.64 USD (Website appointments scheduled)"},
68
- "impressions": "212,816", "ctr": "4.20%", "cpm": "$42.20 USD",
69
- "lead": "7", "created_time": "April 17, 2026"
70
- }
71
- ```
72
- - **Values are formatted display strings**, not numbers (`"$8,980.38 USD"`,
73
- `"4.20%"`, `"Not available"` for zero-delivery ads). Exception: account
74
- level also returns numeric `amount_spent_cents`. The bridge/skill must
75
- parse display strings (or treat them as verdict-snapshot text, which is
76
- what the map's verdict-snapshot design wants anyway).
77
- - `results`/`cost_per_result` self-label the optimization outcome ("Website
78
- appointments scheduled", "Website purchases") — ideal raw material for the
79
- verdict snapshot, and it makes result-type explicit across mixed-objective
80
- accounts.
81
- - Field catalog via `ads_get_field_context`: `spend` aliases `amount_spent`;
82
- `lead`, `purchase_roas`, `results`, `cost_per_result`, `clicks`, video
83
- metrics (`video_thruplay_watched_actions`, `video_p50_watched_actions` —
84
- metric-only, not filterable/sortable) exist. **Not present**: `purchase`,
85
- `cost_per_purchase`, `cost_per_lead`, `roas`, `campaign_name`/`adset_name`
86
- (use ids + a campaign-level call), `link_click`, `hook_rate`,
87
- `ad_archive_id`, `preview_shareable_link`, `effective_object_story_id`
88
- (creative-side only).
89
- - Hierarchy: ads carry `campaign_id` + `adset_id`; `objective` lives on
90
- campaign/ad, `optimization_goal` on adset.
91
-
92
- ### `ads_get_creatives`
93
- The content payload. By id, returns `body` (primary text, full), `title`
94
- (headline), `call_to_action_type`, `object_type`, `image_hash`, `image_url`
95
- (signed CDN), `thumbnail_url`, `object_story_id` +
96
- `effective_object_story_id` (`<pageId>_<postId>`) and
97
- `effective_instagram_media_id`. Listing mode returns only
98
- id/name/account_id/status — **must re-fetch by `creative_ids` for content**.
99
- `link_url` was absent on the probed `SHARE`-type creative (link lives in the
100
- story post). The `<pageId>_<postId>` story id is the strongest identity hook
101
- for enrichment (post permalink = `facebook.com/<pageId>/posts/<postId>`).
102
-
103
- ### `ads_get_ad_images` / `ads_get_ad_videos`
104
- - Image by hash: `url` full-res signed CDN JPEG + `url_128` + `permalink_url`
105
- + `width/height` + timestamps. Downloadable **now**; URL expires later.
106
- - Video by id: `title`, `length` (seconds float), poster `picture`, relative
107
- `permalink_url` (`/<pageId>/videos/<videoId>/…`). **No source URL.**
108
- - **Gotcha (both tools + creatives): the documented "omit `fields` → all
109
- fields" default is broken on by-id lookups** — a `video_ids` lookup without
110
- `fields` returned only `{id}`. Always pass explicit `fields`.
111
-
112
- ### `ads_library_search`
113
- By `page_ids` (page id from the creative's `effective_object_story_id` prefix
114
- or `ads_get_ad_account_pages`): returns per ad `id` (**= adArchiveId**),
115
- `page_id`, `page_name`, `ad_creative_link_title`, `ad_creation_time`,
116
- `ad_delivery_start_time`, `ad_snapshot_url`, `currency`, plus
117
- `estimated_total_count` (48 for the probed page). Requires the caller to have
118
- an active ad account. No body text, no media URLs → cross-reference by
119
- title + time fingerprint, then hand the `adArchiveId` to the existing
120
- ScrapeCreators pipeline for full public payload (media, transcription).
121
-
122
- ### `ads_get_ad_preview`
123
- Returns an actual **rendered creative image** as in-context image content +
124
- an authenticated, expiring `preview_url` iframe (business.facebook.com). Good
125
- for HITL confirmation inside the skill ("is THIS the winner you mean?");
126
- not a durable asset source.
127
-
128
- ### `ads_get_ad_account_pages`
129
- Returns `page_id`s but `page_name` came back `"(unknown)"` on the probed
130
- account — get the human page name from the library search result instead.
131
-
132
- ## Gotchas / reliability
133
-
134
- - **Metric filtering 500s.** Every `filtering` on `ad.amount_spent`
135
- (GREATER_THAN, dollar or cent values) → `Internal Server Error`, repeatably.
136
- Attribute filters (`ad.effective_status IN [ACTIVE]`) work. Metric *sorting*
137
- works. → Winner definitions execute as: sort by metric server-side, pull top
138
- N, threshold agent-side.
139
- - **Transient 500s** happen on entity queries too (one wide 12-field call
140
- failed once, then an identical narrower call succeeded; a later identical
141
- call with the same fields also succeeded). Retry-once is a sane default.
142
- - Only ONE `breakdowns` value per call; only documented params exist.
143
- - The `ads_insights_*` tools (`performance_trend`, `advertiser_context`, etc.)
144
- are opinionated analysis wrappers (trend narratives, funnel context), not
145
- raw data — the skill's mining procedure should stay on `ads_get_ad_entities`.
146
- - Every tool takes an `advertiser_request` verbatim-user-words param — a
147
- Meta-side audit surface; the skill should pass the user's actual ask through.
148
-
149
- ## What this settles for the open tickets
150
-
151
- - **#565 (enrichment)**: key = fingerprint match into Ad Library
152
- (`title` + creation time → `adArchiveId`) → ScrapeCreators; MCP `image_url`
153
- is the image gap-filler (download immediately — signed URLs expire); there is
154
- **no video-file gap-filler** via MCP — video enrichment depends on the ad
155
- still being scrapeable (or the post being public via
156
- `effective_object_story_id`).
157
- - **#566 (bridge contract)**: payload fields available agent-side per winner:
158
- ad id, name, campaign/adset ids, creative id, body, title, CTA type,
159
- story/IG ids, image hash + bytes, video id + length + poster, metrics as
160
- display strings (+ result-type labels), account id/currency, page id,
161
- matched `adArchiveId` (best-effort). Numbers arrive as formatted strings —
162
- the contract should either accept snapshot strings verbatim or require the
163
- agent to parse.
164
- - **#564 (interview)**: winner definitions must be expressible as
165
- sort-metric + date-window + agent-side thresholds; `results` self-labeling
166
- means the definition can be per-result-type; multi-account users (13 visible
167
- here) need an explicit account-selection step.
@@ -1,178 +0,0 @@
1
- # Native-Audio Dialogue — per-model way to make `dialogue` actually get SPOKEN (in English)
2
-
3
- **Research date:** 2026-07-13
4
- **For:** Build ticket #547 (send storyboard `dialogue` to the native-audio video models). Feeds map #545.
5
- **Scope:** The four v1 picker models, all via Kie.ai — `veo3_fast` (Veo 3.1 Fast), `veo3` (Veo 3.1 Quality), `bytedance/seedance-2` (Seedance 2.0), `kling-3.0/video` (Kling 3.0).
6
- **Method:** Primary = docs.kie.ai model pages (fetchable), Google's official Veo prompting guide (cloud.google.com + ai.google.dev), fal.ai model guides, Kling official blog (blocked to automated fetch — corroborated via fal + community). Community tricks flagged inline. Builds on the #529 catalog (`workspace/references/kie-video-models.md`) and the #533 bake-off (`workspace/output/bakeoff-533/`).
7
- **Confidence labels:** `[docs]` verified on a first-party API/prompting doc · `[official-guide]` first-party prompting guide (Google/Kling/ByteDance) · `[vendor-guide]` fal.ai / infra vendor guide · `[community]` community-reported, use with caution.
8
-
9
- ---
10
-
11
- ## The core problem (confirmed in code)
12
-
13
- `buildClipPrompt` in `scout/src/workflow/video/clips.ts` never reads `scene.dialogue`. It only threads `videoPrompt` + `voText` (and `voText` is dropped entirely when native audio is ON). So on a native-audio run the model is told `generate_audio: true` / `sound: true` with **no words to say**, and it invents speech-shaped babble. Every fix below is about getting the `scene.dialogue[]` lines (`{ speaker, line }`, see `convex/lib/workflow/storyboardContract.ts`) into the prompt in the format each family actually speaks — in English, without burning in captions.
14
-
15
- **None of the four models accepts spoken dialogue as a structured API field.** On all four, through Kie, dialogue is **in-prompt text only**. (Details in §"Structured audio params".)
16
-
17
- ---
18
-
19
- ## TL;DR table
20
-
21
- | Family | Dialogue embedding format (recommended) | Force English | Subtitle guard | Structured audio params (Kie) | Pacing guidance |
22
- |---|---|---|---|---|---|
23
- | **Veo 3.1 Fast / Quality** (`veo3_fast`,`veo3`) | Inline attribution **with quotes**, per Google's own guide: `She says, in English: "…"`. Multi-speaker = separate attributed sentences. | Write English inside the quotes **and** add an explicit clause `spoken in English (American accent)`. Keep `enableTranslation:false` (it translates the *prompt* to English and can paraphrase quoted lines). No language param. `[docs]`/`[official-guide]` | **No `negativePrompt` param on Kie's `/veo/generate`** — in-prompt only. Keep `ANTI_SUBTITLE_CORE` first; community: quotes can trigger captions, so the anti-caption clause is load-bearing for Veo. `[docs]`/`[community]` | **None.** Audio always on, not togglable; no voice/reference-audio field. `[docs]` | Keep each line short; dialogue lives inside a 4/6/8s clip. Google: short lines sync best. `[official-guide]` |
24
- | **Seedance 2.0** (`bytedance/seedance-2`) | **Speaker label + colon + double quotes**: `The senior woman, arms crossed, says: "…"`. Short lines; insert written beats (`She pauses, then continues:`) as resync anchors between lines. `[vendor-guide]` | Add per-scene `American accent, conversational tone` + write English text. Mandarin is its most-consistent lip-sync language, so English **must** be stated explicitly or it drifts. `[vendor-guide]`/`[community]` | No first-party subtitle guidance found. Keep `ANTI_SUBTITLE_CORE`. Quoted lines are the documented format, so the core's on-screen-text ban complements (not conflicts). `[community]` | `generate_audio` bool (default true). `reference_audio_urls` (≤3, 2–15s, wav/mp3) exists — a voice/style reference lever, **not** a documented dialogue-content field. `[docs]` | ~**12 words / 10s**, ~**20 words / 15s** (~1.2 wps); sync drops past 10s; record "mental" pace ~80% of natural. **Tighter than the repo's 17 chars/s VO budget.** `[vendor-guide]` |
25
- | **Kling 3.0** (`kling-3.0/video`) | **Bracketed speaker label + tone, colon, quotes**: `[Character A: Exhausted Partner, trembling voice]: "…"`. Consistent labels (no pronouns); for multi-speaker insert a switch beat. `[official-guide]`/`[vendor-guide]` | Specify language in the prompt (`spoken in English`) — helps phoneme mapping. Supports EN natively but also CN/JA/KO/ES, so state it. No language param. `[official-guide]`/`[community]` | No first-party subtitle guidance found. Keep `ANTI_SUBTITLE_CORE`. `[community]` | `sound` bool. `kling_elements[].audio` (5–30s) is an element-level audio lever, not a dialogue-voice field. `[docs]` | Short, single-speaker segments sync best; use Master/`pro` mode when lip-sync is scrutinized; medium shots > extreme close-ups. `[official-guide]` |
26
-
27
- ---
28
-
29
- ## Recommended rider / format per API family (implementation-ready for #547)
30
-
31
- ### Where it slots into the existing rider stack
32
-
33
- `buildClipPrompt` today (native audio ON) emits, in order:
34
-
35
- 1. `ANTI_SUBTITLE_CORE`
36
- 2. `MOTION_POLICY`
37
- 3. `FIRST_FRAME_RIDER` (if a Scene Frame is wired)
38
- 4. `body` (the `videoPrompt`)
39
-
40
- **Add a DIALOGUE block as a new final section, ONLY when `nativeAudio === true` AND `scene.dialogue.length > 0`.** Rules:
41
-
42
- - Keep `ANTI_SUBTITLE_CORE` **first and unchanged** — it is the only subtitle guard Veo has (no `negativePrompt` param), and its wording bans *on-screen* text ("no dialogue text, no transcript text"), which does not suppress *spoken* audio.
43
- - The dialogue block is **family-specific** (Veo/Seedance/Kling format differently), so it must be built where the family is known. Two clean options:
44
- - **(A)** Thread `scene.dialogue` + `family` into `buildClipPrompt` and branch there.
45
- - **(B)** Keep `buildClipPrompt` family-agnostic; in `generateSceneClip` (which already has `spec.family`) append a `buildDialogueBlock(family, scene.dialogue)` string to the returned prompt.
46
- Recommend **(B)** — smallest change, keeps `buildClipPrompt` pure, and the per-family request builders already live next to it.
47
- - `buildClipPrompt` currently accepts `voText` but **not** `dialogue`. #547 must thread `scene.dialogue` through `generateSceneClip` (it already has the full `scene`).
48
- - When `nativeAudio === false`, do **not** emit dialogue (current behavior — mouths shut, VO track). Unchanged.
49
-
50
- ### Concrete template strings
51
-
52
- Let `LANG = "spoken in English (natural American accent)"`.
53
-
54
- **Veo (`veo3_fast`, `veo3`)** — inline, quoted, per Google's guide:
55
- ```
56
- DIALOGUE (on-camera, lip-synced, {LANG}; no on-screen captions):
57
- <Speaker A description> says: "<line 1>"
58
- <Speaker B description> replies: "<line 2>"
59
- ```
60
- - Single speaker → one attributed sentence.
61
- - Use the storyboard `speaker` string as the attribution ("The woman", "The barista"). Quotes around the exact words per Google's official example: `A woman says, "We have to leave now."`
62
- - Do **not** set `enableTranslation:true` (it translates/paraphrases the prompt to English and can rewrite the quoted line). Current code already sends `enableTranslation:false` — keep it.
63
-
64
- **Seedance (`bytedance/seedance-2`)** — label + colon + quotes, short lines, beats:
65
- ```
66
- DIALOGUE ({LANG}, conversational tone; lip-synced to the speaker):
67
- <Speaker A description> says: "<short line>"
68
- [She pauses, then continues:] "<next short line>"
69
- ```
70
- - Colon-then-quotes is the fal-documented Seedance format: `The senior woman, arms crossed, says: "You ran that client call better than I would have."`
71
- - Split anything over ~10–12 words into two lines with a written beat between them (`She pauses, then continues:`) — Seedance uses written beats as resync anchors.
72
- - The `American accent, conversational tone` clause is the sourced mitigation for its Mandarin-lip-sync bias — include it every scene.
73
-
74
- **Kling (`kling-3.0/video`)** — bracketed labelled speakers:
75
- ```
76
- DIALOGUE ({LANG}):
77
- [<Speaker A label>, <tone>]: "<line>"
78
- [<Speaker B label>, <tone>]: "<line>"
79
- ```
80
- - Consistent bracket labels, no pronouns (`[Character A: Exhausted Partner, trembling voice]: "You never listen to me."`).
81
- - Map the storyboard `speaker` to the label and (optionally) infer a tone from `scene.narrative`.
82
- - Prefer `mode: "pro"` when dialogue is central (official: use Master/high mode for scrutinized lip-sync). Current `klingModeFromResolution` already honors the operator's std/pro pick.
83
-
84
- ### Request-builder param changes
85
-
86
- - **`buildVeoBody`** — no new params. `enableTranslation:false` stays. Dialogue rides in `prompt`. (Kie Veo exposes no audio/voice/negativePrompt field — verified on the docs.)
87
- - **`buildSeedanceInput`** — no *required* new params; `generate_audio` already tracks the toggle. Optional future lever: `reference_audio_urls` for voice consistency (§Cross-scene). Dialogue rides in `prompt`.
88
- - **`buildKlingInput`** — no new params; `sound` already tracks the toggle. Dialogue rides in `prompt`.
89
-
90
- **Net:** #547 is a **prompt-assembly change, not a request-schema change.** The only transport-level opportunity (Seedance `reference_audio_urls`) is optional and unverified for dialogue voice.
91
-
92
- ---
93
-
94
- ## Per-model deep dives
95
-
96
- ### Google Veo 3.1 (`veo3_fast`, `veo3`) — Kie `/api/v1/veo/generate`
97
-
98
- - **Dialogue format — quotes, inline attribution.** Google's official *Ultimate prompting guide for Veo 3.1* and the Gemini API docs both use quotation marks with a speaker + verb: `A woman says, "We have to leave now."` and `"This must be the key," he murmured.` The guide lists three audio cue types — Dialogue (quotes), Sound Effects (describe explicitly), Ambient — and says the model "captures the nuance of these cues to generate a synchronized soundtrack." `[official-guide]`
99
- - **Colon-vs-quotes tension (must reconcile).** A widespread community technique claims you should use a **colon and NO quotation marks** (`the man says: hello there`) because Veo "may interpret quotation marks as text to display" and burn in captions. `[community]` This directly contradicts Google's own quoted examples. Resolution for #547: **follow Google (quotes) for reliable lip-sync**, and neutralize the caption risk with the explicit anti-subtitle clause (which we already have) rather than dropping quotes. If Veo captions still appear in testing, the fallback is the colon-no-quotes variant — treat as a tuning knob, not the default.
100
- - **Subtitle suppression.** Kie's `/veo/generate` exposes **no `negativePrompt` field** (verified — the param list is prompt/model/aspect_ratio/duration/resolution/generationType/imageUrls/enableTranslation/watermark/callBackUrl only; ai.google.dev's Gemini-API Veo page likewise shows no negativePrompt for the dialogue flow). So the only caption lever is **in-prompt text** — exactly what `ANTI_SUBTITLE_CORE` provides. Community adds "no subtitles"/"no captions" phrasing and keeping dialogue under the 8s clip length so the model "focuses on visuals and audio rather than captions." `[docs]`/`[community]`
101
- - **English forcing.** No language param. `enableTranslation` "enable[s] prompt translation to English … before video generation" — i.e. it translates *your prompt*, not the spoken output; for an already-English prompt it's a no-op at best and a paraphrase risk at worst, so **keep it false**. Force English by (a) writing the words in English inside the quotes and (b) an explicit `spoken in English (American accent)` clause; Veo "will handle the accent and lip-sync automatically" from the written language. `[docs]`/`[community]`
102
- - **Audio toggle.** "All videos ship with background audio by default"; rarely suppressed for sensitive scenes. Not togglable on Kie — matches `audioTogglable:false` in `models.ts` (code already forces `nativeAudio=true` for the veo family). `[docs]`
103
- - **Bake-off note (#533):** Veo 3.1 Fast native-audio clip generated in ~10s wall (anchored) — the fastest of the four; the audio dialogue shoot-out cell is `dlg-veo3-quality.mp4` / `clipref-s1.mp4`. Sora 2 was excluded (paused on Kie). See `workspace/output/bakeoff-533/index.html`.
104
-
105
- Sources: [docs.kie.ai/veo3-api/generate-veo-3-video](https://docs.kie.ai/veo3-api/generate-veo-3-video) · [Ultimate prompting guide for Veo 3.1 (Google Cloud)](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1) · [Gemini API Veo docs](https://ai.google.dev/gemini-api/docs/veo) · [GlobalGPT Veo 3.1 dialogue guide](https://www.glbgpt.com/hub/how-to-make-characters-speak-in-veo-3-1-the-ultimate-guide-to-dialogue-audio-lip-sync/) `[community]` · [VidAU "no subtitles"](https://www.vidau.ai/veo-3-prompt-no-subtitles/) `[community]`
106
-
107
- ### ByteDance Seedance 2.0 (`bytedance/seedance-2`) — Kie jobs API
108
-
109
- - **Dialogue format — label, colon, double quotes.** fal.ai's Seedance 2.0 prompting guide: "Put the spoken words in double quotes, and the model voices them, matching the lip movement to each line," e.g. `The senior woman, arms crossed, says: "You ran that client call better than I would have."` For multi-speaker, hold a two-shot then cut to the speaker, "lips matched to each line." Keep lines short — "Long monologues drift out of sync, so I split a speech into a couple of shorter lines and let the cuts carry it." `[vendor-guide]`
110
- - **Written beats as resync anchors.** "Insert a written beat ('She pauses, then continues:') between sentences — Seedance uses written beats as resync anchors." Directly useful for splitting a scene's multiple `dialogue` lines. `[vendor-guide]`
111
- - **English forcing / drift.** Seedance supports 8+ languages incl. English/Mandarin, but "Mandarin produces the most consistent lip sync," and separate generations of the same character drift in accent unless pinned. Sourced mitigation: **"Specify accent explicitly in every scene prompt: 'American accent, conversational tone.'"** No API language param. The specific "drifts to Chinese/Mandarin" failure the map calls out is **community-reported and consistent with these vendor guides, but I could not pin a primary Reddit/X thread this session** — treat the drift as real-risk, the accent clause as the sourced fix. `[vendor-guide]`/`[community]`
112
- - **Subtitle risk.** No first-party statement that quoted Seedance dialogue burns in captions; no first-party suppression guidance found. Keep `ANTI_SUBTITLE_CORE`. `[community]`
113
- - **Pacing (important — conflicts with the repo budget).** fal / UGC-Copilot guidance: **~12 words per 10s, ~20 words per 15s (~1.2 words/sec)**; "short sentences … five to ten words per line is the sweet spot"; sync quality "drops past 10" seconds. The storyboard gate (`VO_CHARS_PER_SEC = 17`, `sceneSpeechSec`) budgets ~17 chars/s ≈ ~3 words/s — **~2.5× looser than Seedance actually speaks cleanly.** A scene that "fits" at 17 cps can still overflow / rush / desync on Seedance native audio. **Recommendation for #547 (or a fast-follow):** use a tighter chars-per-second budget for native-audio *dialogue* (≈9–10 cps, ~1.2–1.5 wps) than for VO narration. Seedance does not hard-truncate; it rushes and desyncs. `[vendor-guide]`
114
- - **Bake-off note (#533):** Seedance native audio was slowest by far — measured ≈24 min/clip, which is why `POLL_TIMEOUT_VIDEO_MS` was raised to 35 min in `kie-video.ts`. It won "best native audio" per the map, but at a steep latency/cost. `dlg-seedance.mp4` vs the silent control `dlg-seedance-silent.mp4` vs pixr mux `dlg-elevenlabs-mux.mp4`.
115
-
116
- Sources: [docs.kie.ai/market/bytedance/seedance-2](https://docs.kie.ai/market/bytedance/seedance-2) · [fal.ai Seedance 2.0 prompting guide](https://fal.ai/learn/tools/seedance-2-0-prompting-guide) · [UGC-Copilot Seedance native audio guide](https://ugccopilot.ai/blog/seedance-2-native-audio-generation-guide/) `[community]` · [Cutout.pro Seedance audio guide](https://www.cutout.pro/learn/blog-seedance-2-0-audio-guide/) `[community]`
117
-
118
- ### Kling 3.0 (`kling-3.0/video`) — Kie jobs API
119
-
120
- - **Dialogue format — bracketed labelled speakers.** fal.ai's Kling 3.0 guide: `[Character Label, tone/emotion]: "dialogue text"`, e.g. `[Character A: Exhausted Partner, trembling frustrated voice]: "You never listen to me."` Structured naming (P1): "Character labels must be consistent. Avoid pronouns or synonyms." For sequencing/multi-speaker, insert a switch beat ("this is when the speaker switches"). `[vendor-guide]`
121
- - **Official audio behavior.** Kling's own blog (Kling VIDEO 3.0 / 3.0 Omni "Native Audio") — dialogue, sound and lip movement are generated together. Best-practice guidance (corroborated across Kling + fal + community): short simple sentences beat rapid complex speech; **specify the language in the prompt to help phoneme mapping**; medium shots sync better than extreme close-ups; avoid overlapping dialogue — single-speaker segments sync much better; use Master/high mode when lip-sync will be scrutinized. `[official-guide]`/`[community]` (Note: the official Kling blog returns HTTP 446 to automated fetch; content is corroborated via the search excerpt + fal guide, not a direct page read.)
122
- - **Language support / forcing.** Supports Chinese, English, Japanese, Korean, Spanish + dialects/accents (American/British/Indian English). No language param — state `spoken in English` in the prompt. `[official-guide]`/`[community]`
123
- - **Kie param nuance.** On Kie, the native-audio toggle is `sound` (bool) and `prompt` "takes effect when multi_shots is false" — our builder sends `multi_shots:false`, so the single `prompt` (with the dialogue block) is the right channel. Multi-shot mode would need the `multi_prompt[]` array instead (each shot's `prompt` ≤500 chars); v1 does not use it. `[docs]`
124
- - **Bake-off note (#533):** `dlg-kling.mp4`, ~103s gen — much faster than Seedance, comparable to Veo Quality.
125
-
126
- Sources: [docs.kie.ai/market/kling/kling-3-0](https://docs.kie.ai/market/kling/kling-3-0) · [Kling official: Kling Video 3.0 Omni Native Lip Sync & Audio](https://kling.ai/blog/kling-video-3-omni-native-lip-sync-audio-guide) (446 to automated fetch) · [fal.ai Kling 3.0 prompting guide](https://blog.fal.ai/kling-3-0-prompting-guide/) · [Imagine.art Kling 3.0 prompt guide](https://www.imagine.art/blogs/kling-3-0-prompt-guide) `[community]`
127
-
128
- ---
129
-
130
- ## Cross-scene voice consistency (map fog — levers that exist)
131
-
132
- The problem: each scene is an independently-generated clip, so the same actor gets a **different synthesized voice** each time. No picker model exposes a first-class "voice ID" through Kie's jobs/veo API. Levers found, none verified to solve it:
133
-
134
- - **Seedance `reference_audio_urls`** `[docs]` — up to 3 audio clips (2–15s, wav/mp3), also accepts `asset://{assetId}`. Documented as a *reference* input (style/voice conditioning), **not** as the dialogue-content source. **Fog:** it *might* anchor timbre if the same reference clip is passed to every scene — but the docs don't say it transfers voice identity, and whether the model speaks the *prompt* text in the *reference* voice is untested. This is the single most promising Kie-native lever; worth an empirical probe before relying on it. Seedance is also the only one of the four with any audio-input field.
135
- - **Kling `kling_elements[].audio`** `[docs]` — per-element audio (5–30s) tied to an `@element`. Element references are aimed at *visual* character consistency (2–4 images/element); the audio slot is under-documented for voice transfer. **Fog.**
136
- - **Veo** — **no audio input at all.** No lever for cross-scene voice. `[docs]`
137
- - **Higher-level "voice fingerprint" features** (e.g. fal's "AI Twins" persona voice-locking) exist at the infra layer but are **not exposed through Kie's jobs API**, so not usable in this pipeline as-is. `[community]`
138
-
139
- **Practical read for v1:** there is no reliable cross-scene voice-consistency lever through Kie today. If it matters, the more controllable path remains the old pixr architecture — render silent + one ElevenLabs voice muxed across all scenes (the `dlg-elevenlabs-mux.mp4` control in #533). Keep cross-scene voice in the fog for #547; if pursued, the first experiment is Seedance `reference_audio_urls` with a fixed reference clip.
140
-
141
- ---
142
-
143
- ## Duration / pacing behavior (per family)
144
-
145
- - **Seedance:** ~12 words/10s, ~20 words/15s; sync degrades past ~10s and with long lines; does **not** hard-truncate — it rushes and drifts out of sync. Written beats resync it. `[vendor-guide]`
146
- - **Veo:** fixed 4/6/8s clips; keep dialogue short enough to land inside the clip; Google/community both say short lines sync best and staying under 8s reduces caption bleed. `[official-guide]`/`[community]`
147
- - **Kling:** 3–15s; short single-speaker segments sync best; no published words/sec number found. `[official-guide]`
148
- - **Repo gate today:** `sceneSpeechSec = max(voText, dialogue) ÷ 17 chars/s` fits *both* tracks to `durationSec` and fails loud rather than truncating. For **VO narration** 17 cps is reasonable; for **native lip-synced dialogue** it is too loose (Seedance ≈ 7–8 cps of clean speech). **Actionable:** consider a separate, tighter cps for native-audio dialogue-fit. (Flagged as a recommendation; whether #547 changes the budget or just formats the block is a scoping call.)
149
-
150
- ---
151
-
152
- ## Open questions / unverifiable this session
153
-
154
- 1. **Veo colon-vs-quotes for captions** — Google officially uses quotes; the "quotes burn captions, use a colon" claim is community-only and contradicts the official examples. Not resolvable without a live Veo-on-Kie A/B. Default to quotes + anti-caption clause; keep colon-no-quotes as a fallback knob. `[community]`
155
- 2. **Seedance→Chinese drift** — consistent with vendor guides ("Mandarin most consistent," "specify accent every scene") but no primary Reddit/X thread pinned this session. The accent clause is the sourced mitigation; the drift itself is community-level. `[community]`
156
- 3. **Whether `ANTI_SUBTITLE_CORE`'s "no dialogue text" wording ever suppresses *spoken* audio** — it targets on-screen typography, so it *shouldn't*, but a paranoid model could misread it. Low risk; worth watching in the first native-audio+dialogue test. Not observed in #533 (which ran without a dialogue block).
157
- 4. **Seedance `reference_audio_urls` as a voice-consistency lever** — exists in the API but its effect on spoken-dialogue voice is undocumented. Needs an empirical probe.
158
- 5. **Kie Veo `negativePrompt`** — absent from Kie's `/veo/generate` param list (Vertex AI Veo has one; Kie's endpoint does not expose it). Confident but worth a 1-line confirm if #547 wants a param-based caption guard.
159
- 6. **Exact per-family words/sec** — only Seedance publishes numbers; Veo/Kling give qualitative "keep it short." Any tighter dialogue-fit budget will be model-family-specific and partly empirical.
160
-
161
- ---
162
-
163
- ## Sources
164
-
165
- Primary / first-party:
166
- - [docs.kie.ai — Veo 3.1 generate](https://docs.kie.ai/veo3-api/generate-veo-3-video), [Seedance 2.0](https://docs.kie.ai/market/bytedance/seedance-2), [Kling 3.0](https://docs.kie.ai/market/kling/kling-3-0)
167
- - [Google Cloud — Ultimate prompting guide for Veo 3.1](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1)
168
- - [Google AI for Developers — Generate videos with Veo 3.1 (Gemini API)](https://ai.google.dev/gemini-api/docs/veo)
169
- - [Kling.ai — Kling Video 3.0 Omni Native Lip Sync & Audio](https://kling.ai/blog/kling-video-3-omni-native-lip-sync-audio-guide) (446 to automated fetch; corroborated indirectly)
170
-
171
- Vendor guides (fal.ai):
172
- - [Seedance 2.0 prompting guide](https://fal.ai/learn/tools/seedance-2-0-prompting-guide), [Kling 3.0 prompting guide](https://blog.fal.ai/kling-3-0-prompting-guide/)
173
-
174
- Community (labeled inline):
175
- - [GlobalGPT Veo 3.1 dialogue/lip-sync](https://www.glbgpt.com/hub/how-to-make-characters-speak-in-veo-3-1-the-ultimate-guide-to-dialogue-audio-lip-sync/) · [VidAU Veo no-subtitles](https://www.vidau.ai/veo-3-prompt-no-subtitles/) · [UGC-Copilot Seedance native audio](https://ugccopilot.ai/blog/seedance-2-native-audio-generation-guide/) · [Cutout.pro Seedance audio](https://www.cutout.pro/learn/blog-seedance-2-0-audio-guide/) · [Imagine.art Kling 3.0](https://www.imagine.art/blogs/kling-3-0-prompt-guide)
176
-
177
- Local prior art:
178
- - `workspace/references/kie-video-models.md` (#529 catalog) · `workspace/output/bakeoff-533/index.html` + `manifest.json` (#533 native-audio bake-off) · `scout/src/workflow/video/clips.ts`, `models.ts`, `kie-video.ts`, `storyboard.ts` · `convex/lib/workflow/storyboardContract.ts`
@@ -1,160 +0,0 @@
1
- # Own-Brand Winners — enrichment path design (wayfinder #565)
2
-
3
- Designed 2026-07-14 from the [MCP probe findings](meta-ads-mcp-probe.md) (#563)
4
- plus a code audit of the live swipe rail (`scout/src/scrapecreators.ts`,
5
- `scout/src/trigger/swipe-mine.ts`, `swipe-transcribe.ts`, `image-classify.ts`,
6
- `convex/swipeAds.ts`, `convex/imageClassification.ts`). Answers: what is the
7
- exact path from a designated winner (ad-account identity) to a complete stored
8
- package? Feeds the bridge contract (#566) and the build specs (#567).
9
-
10
- ## TL;DR decisions
11
-
12
- 1. **Match server-side on ad copy, not agent-side on the MCP title fingerprint.**
13
- The bridge payload carries the account-side ground truth (body, title,
14
- created_time, format, page id); a new import task scrapes the brand's own
15
- page once via the existing `fetchCompanyAds` and matches winners to library
16
- ads by **normalized body+title equality** (tiebreak: earliest `start_date`).
17
- Full ScrapeCreators snapshots include body text — a far stronger key than
18
- the library-search `title + time` fingerprint the probe had to settle for,
19
- and it drops `ads_library_search` from the skill entirely.
20
- 2. **One new Trigger task (`own-brand-import`), everything downstream as-is.**
21
- The task reuses the scrapecreators.ts helpers + the `/api/v2/swipes/batch`
22
- upsert, stores **only matched winners**, then fires the existing
23
- `swipe-transcribe` and `image-classify` sweeps, which pick own-brand rows up
24
- with zero changes (both sweep global "missing transcript" / "unclassified"
25
- states).
26
- 3. **Gap-filler = agent-pushed assets for unmatched winners** (ad stopped
27
- running): text is always complete from MCP; images are downloaded by the
28
- agent immediately (signed CDN URLs expire) and uploaded as bytes via the
29
- existing Convex `generateUploadUrl` pattern; **video files have no MCP
30
- source — the skill asks the user for their own creative file** (it's their
31
- ad), else the package is stored partial.
32
- 4. **Mirror winner media into Convex storage at import.** fbcdn URLs expire;
33
- competitor rows survive because the daily re-scrape refreshes them, but
34
- winners are long-lived fuel imported once. Store the storage **serve URL in
35
- the existing `imageUrl`/`videoUrl` fields** so every downstream consumer
36
- (pickers, classify, transcribe, gallery) works unchanged.
37
- 5. **Own-brand `brands` rows must be flagged out of the daily swipe-mine cron**
38
- (`isOwnBrand: true`, mine filters it): otherwise the nightly sweep scrapes
39
- the user's whole page and stores every active ad — violating winners-only.
40
- 6. **Completeness is an explicit stored field**, not derived: `complete` |
41
- `partial` (+ what's missing), surfaced amber in the UI tab (the #478
42
- partial-delivery pattern).
43
-
44
- ## Stage 0 — what the agent holds per winner (input to the bridge)
45
-
46
- Per the probe, after the winner interview designates ads, the agent can gather
47
- via MCP (`ads_get_ad_entities` + `ads_get_creatives` + `ads_get_ad_images` /
48
- `ads_get_ad_videos`, all with explicit `fields`):
49
-
50
- | Field | Source | Notes |
51
- |---|---|---|
52
- | account ad id, name, campaign/adset ids | entities | hierarchy + provenance |
53
- | `created_time` | entities | matching tiebreak + daysRunning fallback |
54
- | body (primary text), title (headline), CTA type | creative | ground truth for matching |
55
- | `effective_object_story_id` (`<pageId>_<postId>`) | creative | **page id prefix** = scrape target |
56
- | image hash → full-res signed URL | ad_images | **download immediately**, expires |
57
- | video id, length, poster URL | ad_videos | no source URL — cannot download |
58
- | metrics as display strings + result-type labels | entities | verdict snapshot material |
59
-
60
- The bridge payload (#566) should carry all of the above per winner, plus the
61
- page id once per import.
62
-
63
- ## Stage 1 — matching: winner → `adArchiveId`
64
-
65
- Runs inside `own-brand-import` (server-side, ScrapeCreators BYOK key snapshot
66
- exactly like `swipe-mine`'s manual-run pattern):
67
-
68
- 1. `fetchCompanyAds(pageId)` — one paginated scrape of the user's own page.
69
- Returns full snapshots: `body.text`, `title`, `start_date`, media URLs,
70
- `collation_count`.
71
- 2. For each winner: **normalize** (trim, collapse whitespace, strip zero-width
72
- chars, normalize curly quotes) and match on **body equality**; require
73
- title equality too when both sides have a title.
74
- 3. Ambiguity: multiple library ads with identical copy are creative variants /
75
- republishes — any is acceptable (same creative); pick the **earliest
76
- `start_date`** (longest `daysRunning`, best longevity signal for pickers).
77
- 4. Fallback when the account creative has no body (some `SHARE`-type creatives
78
- keep copy on the story post): match on unique title + `created_time`
79
- proximity (±72h). Non-unique → treat as unmatched (gap-filler), never guess.
80
- 5. Matched → optionally `fetchAdLibraryAdById(archiveId, get_transcript=true)`
81
- for the inline transcript (the idea-ad-library / workflow-executor pattern);
82
- store it at insert so most video winners never wait for the transcribe cron.
83
-
84
- **Failure modes:** page scrape empty (wrong page id, zero active ads) → all
85
- winners unmatched → gap-filler; the Ad Library only lists **running** ads for
86
- non-political advertisers, so any stopped winner is *expected* to be unmatched
87
- — that's the designed-for case, not an error. One spec-time item:
88
- `fetchCompanyAds` hardcodes `country: "US"` — the import path must make this
89
- configurable (or "ALL" if the endpoint accepts it) or non-US advertisers will
90
- false-unmatch.
91
-
92
- ## Stage 2 — gap-filler protocol (unmatched winners)
93
-
94
- What MCP can and cannot supply fills the row directly:
95
-
96
- - **Copy**: body/title/CTA come from the creative — always complete. No
97
- library `linkUrl`/`linkDescription`; acceptable loss (fields stay empty).
98
- - **Image**: the agent downloads the signed CDN URL **at interview time** (not
99
- at push time — it expires) and the CLI uploads bytes: request an upload URL
100
- (the `creativeSuite.generateUploadUrl` service pattern), PUT the file, record
101
- the storage id. Import stores the serve URL in `imageUrl`.
102
- - **Video**: no MCP file. Ladder: (a) the skill asks the user to point at
103
- their own creative file — they made the ad, they usually have it — CLI
104
- uploads it; (b) no file → store partial: copy + poster image (poster URL
105
- *is* downloadable now) and no transcript. Once a video lands in storage, its
106
- serve URL in `videoUrl` makes the existing `pendingTranscripts` sweep
107
- transcribe it automatically (note: `transcribeVideo` rejects >25MB —
108
- acceptable v1 cap, flag oversize uploads as partial-transcript).
109
- - **Identity**: gap-filled rows get a synthetic archive id `own:<accountAdId>`
110
- — stable, unique, satisfies the `by_adArchiveId` dedup index, can't collide
111
- with numeric library ids. `isActive: false`, `startDate` from
112
- `created_time`.
113
-
114
- ## Stage 3 — reuse audit
115
-
116
- | Piece | Verdict |
117
- |---|---|
118
- | `scrapecreators.ts` helpers (fetch, extract*, classifyFormat, transcribe) | **as-is** — the import task composes them |
119
- | `/api/v2/swipes/batch` → `swipeAds.upsertBatch` | **as-is** — winners-only array in, dedup by archive id |
120
- | `swipe-transcribe` (`pendingTranscripts` sweep) | **as-is** — global format=video-missing-transcript scan includes own-brand rows; also fire it at import end |
121
- | `image-classify` (`getUnclassifiedSwipeAds`) | **as-is** — sweeps `classificationStatus === undefined`; own-brand rows enter automatically |
122
- | Consumption (`swipeAdToText`, pickers, `listForWorkspacePublic`) | **as-is** — own-brand rows are swipeAds under a workspace brand; they flow into every picker. Spec check (per map): run-dialog top-N ranks by `daysRunning` — matched winners inherit real longevity, gap-filled ones get `created_time`-derived values, and the winner fields below let pickers boost designated winners explicitly |
123
- | `swipe-mine` task | **not reused** (fetch-everything shape violates winners-only); its building blocks are |
124
- | Daily `swipe-mine` cron | **needs a filter**: skip `isOwnBrand` brands (decision 5) |
125
- | New code | one Trigger task `own-brand-import` + one HTTP route to start it (BYOK key snapshot like `swipe.startMineRun`) + the CLI ingestion command (#566's contract) |
126
-
127
- ## Stage 4 — completeness contract
128
-
129
- A package is **usable generative fuel** when:
130
-
131
- - **every winner**: verdict snapshot (composed sentence + metric display
132
- strings) — non-negotiable, it's the point of the feature — plus body or
133
- title, and provenance (account ad id, campaign/adset ids).
134
- - **image winner**: + full-res image (storage-mirrored). Missing image →
135
- `partial` (copy bots still work; ref-match image engines can't use it).
136
- - **video winner**: + transcript (from library inline, sweep, or uploaded
137
- file). Video file itself is desired (gallery, future variations) but
138
- transcript is the fuel; no transcript AND no video → `partial`.
139
-
140
- Store it explicitly: `enrichmentStatus: "complete" | "partial"` +
141
- `enrichmentMissing: string[]` (e.g. `["video-file", "transcript"]`), set by the
142
- import task, re-evaluated when a sweep fills a gap. UI shows amber on partial
143
- rows with the missing list (the #478 pattern).
144
-
145
- New optional `swipeAds` columns this implies for #567 (all additive):
146
- winner verdict text + metrics snapshot, `designatedAt`, source account
147
- ad/campaign/adset ids, `enrichmentSource: "library-match" | "agent-push"`,
148
- `enrichmentStatus` / `enrichmentMissing`, `imageStorageId` / `videoStorageId`
149
- (lifecycle), plus `brands.isOwnBrand`.
150
-
151
- ## Open items handed to #566 / #567
152
-
153
- - Bridge contract (#566): exact payload shape per winner (Stage 0 table), how
154
- the CLI transports image/video bytes (upload-URL handshake vs multipart),
155
- idempotency on re-push (upsert by account ad id → synthetic/matched archive
156
- id), and where the "ask the user for the video file" step lives in the skill.
157
- - Spec (#567): country parameter on the import scrape; winner-boost rule in
158
- top-selection heuristics; amber partial UI in the Swipe Mining tab; whether
159
- matched winners' media also gets storage-mirrored at import (recommended:
160
- yes, both paths, decision 4).