@slatesvideo/shared 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -0
- package/dist/clients/cloud.js +32 -1
- package/dist/clients/desktop.d.ts +10 -4
- package/dist/clients/desktop.js +95 -7
- package/dist/index.d.ts +2 -1
- package/dist/index.js +1 -0
- package/dist/operations/index.d.ts +207 -1
- package/dist/operations/index.js +1513 -32
- package/dist/skills/content.d.ts +2 -0
- package/dist/skills/content.js +20 -0
- package/package.json +31 -4
- package/skills/slates-character-turnaround.md +53 -0
- package/skills/slates-cost-discipline.md +96 -0
- package/skills/slates-direct-response-ad.md +66 -0
- package/skills/slates-edit-and-iterate.md +48 -0
- package/skills/slates-one-prompt-film.md +67 -0
- package/skills/slates-prompting-flux-2-max.md +106 -0
- package/skills/slates-prompting-kling-v3.md +160 -0
- package/skills/slates-prompting-lip-sync.md +169 -0
- package/skills/slates-prompting-motion-transfer.md +148 -0
- package/skills/slates-prompting-nano-banana-2.md +147 -0
- package/skills/slates-prompting-seedance.md +144 -0
- package/skills/slates-prompting-seedream-5-lite.md +74 -0
- package/skills/slates-prompting-veo-3.md +149 -0
- package/skills/slates-storyboard-from-script.md +49 -0
- package/skills/slates-vision-feedback-loop.md +62 -0
- package/dist/auth.d.ts.map +0 -1
- package/dist/auth.js.map +0 -1
- package/dist/clients/cloud.d.ts.map +0 -1
- package/dist/clients/cloud.js.map +0 -1
- package/dist/clients/desktop.d.ts.map +0 -1
- package/dist/clients/desktop.js.map +0 -1
- package/dist/index.d.ts.map +0 -1
- package/dist/index.js.map +0 -1
- package/dist/operations/index.d.ts.map +0 -1
- package/dist/operations/index.js.map +0 -1
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: slates-prompting-flux-2-max
|
|
3
|
+
description: How to prompt FLUX.2 Max (Black Forest Labs image model). Read before calling slates_generate_image with model flux-2-max, or slates_edit_image with editModel flux-2-max. FLUX.2 wants front-loaded structure, real camera vocabulary, and positive-only phrasing — no negative prompts, no tag soup.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# FLUX.2 Max — prompting
|
|
7
|
+
|
|
8
|
+
Black Forest Labs' top image model, routed via fal.ai. In Slates: `slates_generate_image` with `model: flux-2-max` (REQUIRES projectId — no headless path), priced per resolution (1k/2k/4k — call `slates_estimate_generation_cost` for current numbers, never quote from memory). Strengths vs Nano Banana 2: photoreal texture, less censored, precise hex-color control, strong typography. Reference images route through FLUX's edit endpoint and carry a lower per-model cap than NB2's 14.
|
|
9
|
+
|
|
10
|
+
## Core structure — front-load what matters
|
|
11
|
+
|
|
12
|
+
```
|
|
13
|
+
Subject + Action + Style + Context
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Word order is weight. FLUX.2 attends hardest to the start of the prompt: main subject → key action → critical style → essential context → secondary details.
|
|
17
|
+
|
|
18
|
+
**Length:** 10-30 words for concept tests, 30-80 words for most work, 80+ only for genuinely complex scenes.
|
|
19
|
+
|
|
20
|
+
## Photorealism: name real gear, not "professional photo"
|
|
21
|
+
|
|
22
|
+
The single biggest realism lever is concrete camera vocabulary:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
Shot on Hasselblad X2D, 80mm lens, f/2.8, natural lighting
|
|
26
|
+
Shot on Sony A7IV, 35mm, golden hour, shallow depth of field
|
|
27
|
+
Kodak Portra 400, natural grain, organic colors
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Era cues work the same way: "early digital camera, slight noise, flash photography, candid" reads 2000s digicam; "film grain, warm color cast, soft focus" reads 80s.
|
|
31
|
+
|
|
32
|
+
For portraits add: natural skin texture, realistic pores, subtle imperfections, soft diffused lighting.
|
|
33
|
+
|
|
34
|
+
## No negative prompts — reframe positively
|
|
35
|
+
|
|
36
|
+
FLUX.2 has no negative prompt support. Describe the presence you want, not the absence:
|
|
37
|
+
|
|
38
|
+
- ❌ "no blur" → ✅ "sharp focus throughout"
|
|
39
|
+
- ❌ "no people" → ✅ "empty scene"
|
|
40
|
+
- ❌ "no harsh shadows" → ✅ "soft, diffused lighting"
|
|
41
|
+
|
|
42
|
+
## Hex colors — bind them to objects
|
|
43
|
+
|
|
44
|
+
FLUX.2 matches hex codes, but only when each code is attached to a specific object:
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
walls in hex #C4725A, sofa in #1B6B6F, accent pillows #E8A847
|
|
48
|
+
gradient starting with color #02eb3c and finishing with color #edfa3c
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
❌ "use #FF0000 somewhere" — unbound colors land inconsistently.
|
|
52
|
+
|
|
53
|
+
## Text rendering
|
|
54
|
+
|
|
55
|
+
Quote the exact text, then place and style it:
|
|
56
|
+
|
|
57
|
+
```
|
|
58
|
+
The text 'OPEN' appears in red neon letters above the door
|
|
59
|
+
Logo text 'ACME' in color #FF5733, ultra-bold decorative serif, centered
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Specify placement relative to other elements, font family feel (serif / sans / script), and relative size ("large headline," "small body copy").
|
|
63
|
+
|
|
64
|
+
## JSON prompting for production work
|
|
65
|
+
|
|
66
|
+
For multi-element scenes that must come out exactly right (product shots, infographics, brand work), FLUX.2 parses structured JSON prompts:
|
|
67
|
+
|
|
68
|
+
```json
|
|
69
|
+
{
|
|
70
|
+
"scene": "Professional studio product photography on polished concrete",
|
|
71
|
+
"subjects": [{ "description": "matte black ceramic mug with steam", "position": "center foreground" }],
|
|
72
|
+
"style": "commercial product photography",
|
|
73
|
+
"color_palette": ["#1B1B1B", "#E8A847"],
|
|
74
|
+
"lighting": "three-point softbox, soft diffused highlights",
|
|
75
|
+
"camera": { "lens-mm": 85, "f-number": "f/5.6" }
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Use natural language for exploration, JSON when the layout is locked and you're matching a spec.
|
|
80
|
+
|
|
81
|
+
## Reference images (edit path)
|
|
82
|
+
|
|
83
|
+
In Slates, pass `referenceAssetIds` on `slates_generate_image` — FLUX routes them through its edit endpoint. Label every reference's role in the prompt text ("subject from image 1, style from image 2, background from image 3"); unlabeled references get blended unpredictably. For surgical changes to one existing image use `slates_edit_image` with `editModel: flux-2-max` (note: FLUX edits ignore extra referenceAssetIds — that's NB2-only).
|
|
84
|
+
|
|
85
|
+
## Character consistency across a series
|
|
86
|
+
|
|
87
|
+
Define the character exhaustively once, then repeat those exact descriptors verbatim in every subsequent prompt. FLUX has no memory between generations — the repeated description IS the consistency mechanism.
|
|
88
|
+
|
|
89
|
+
## Common failure modes + fixes
|
|
90
|
+
|
|
91
|
+
| Failure | Fix |
|
|
92
|
+
|---|---|
|
|
93
|
+
| Generic "AI look" on photoreal | Name a camera body + lens + f-stop instead of "professional photo" |
|
|
94
|
+
| Colors drift from brand spec | Bind each hex code to a named object |
|
|
95
|
+
| Text garbled | Quote the exact string, specify font feel + placement + size |
|
|
96
|
+
| Multi-reference blend chaos | Label each reference's role explicitly in the prompt |
|
|
97
|
+
| Wanted element missing | Move it earlier in the prompt — order is weight |
|
|
98
|
+
|
|
99
|
+
## Pre-flight: references arrive inline, refer by code
|
|
100
|
+
|
|
101
|
+
When you pass `referenceAssetIds`, the first call returns the references **inline as image content blocks** with a cost estimate and `requires_confirm: true`. Look at them — revise the prompt if they suggest a different composition or style — then re-call with `confirm=true`. Refer to each asset by its short code (`IMG-A12 — Beach Sunset`) when talking to the user; it matches the badge on their gallery thumbnail.
|
|
102
|
+
|
|
103
|
+
## Sources
|
|
104
|
+
|
|
105
|
+
- [Black Forest Labs — FLUX.2 Prompting Guide](https://docs.bfl.ml/guides/prompting_guide_flux2)
|
|
106
|
+
- [fal.ai — FLUX.2 [max] Prompt Guide](https://fal.ai/learn/devs/flux-2-max-prompt-guide)
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: slates-prompting-kling-v3
|
|
3
|
+
description: How to prompt Kling V3.0 (Kuaishou). Read before calling slates_generate_video with kling-v3.0-std, kling-v3.0-pro, or kling-v3.0-omni. Kling has dialogue + SFX + ambient native syntax (Omni adds multi-character dialogue and language codes). Multi-shot rules differ from Seedance/Veo — don't cross syntaxes.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Kling V3.0 — prompting
|
|
7
|
+
|
|
8
|
+
Kuaishou's video model. Three tiers: `kling-v3.0-std` (general use, no audio), `kling-v3.0-pro` (higher visual quality, no audio), `kling-v3.0-omni` (multi-character dialogue + audio-visual co-generation).
|
|
9
|
+
|
|
10
|
+
Up to 15s. Multi-shot supported (up to 6 cuts in 15s total). Strong on image-to-video — preserves identity, layout, and text from the input image well.
|
|
11
|
+
|
|
12
|
+
## Subject definition rule (verbatim, fal blog)
|
|
13
|
+
|
|
14
|
+
> "Define your core subjects clearly at the beginning of the prompt and keep descriptions consistent across shots."
|
|
15
|
+
|
|
16
|
+
## Dialogue syntax
|
|
17
|
+
|
|
18
|
+
```
|
|
19
|
+
Character says, "exact words here"
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Use quotation marks for precise speech. Languages (Omni only): EN, ZH, JA, KO, ES.
|
|
23
|
+
|
|
24
|
+
## Voice direction formula (Omni)
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
Gender + Age Range + Voice Quality + Speech Rate + Emotional Tone + Language
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Example:
|
|
31
|
+
```
|
|
32
|
+
[Character A: Detective, mid-40s, raspy voice, slow cadence, weary]: "I've seen this before."
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Tone phrases that fire:
|
|
36
|
+
- `speaking in a hushed, trembling whisper`
|
|
37
|
+
- `shouting with commanding authority`
|
|
38
|
+
- `clear, fearful voice`
|
|
39
|
+
- `with a trembling voice, "I'm scared"`
|
|
40
|
+
|
|
41
|
+
## The `Immediately` keyword (Omni only)
|
|
42
|
+
|
|
43
|
+
Without `Immediately`, Kling adds a natural conversational beat between speakers. With it, dialogue is back-to-back. Use when timing matters.
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
[Alice]: "Get down!" Immediately, [Bob]: "Where?"
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## Speaker label discipline
|
|
50
|
+
|
|
51
|
+
Unique labels per character. **No pronouns or synonyms after first introduction** — they cause voice drift.
|
|
52
|
+
|
|
53
|
+
✅ `[Character A: Black-suited Agent]` ... `[Character A: Black-suited Agent]: "Stop."`
|
|
54
|
+
❌ `[Agent]... then he says...`
|
|
55
|
+
|
|
56
|
+
## Multi-character dialogue (Omni)
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
Alice says in English, "Hello!" Then Bob replies in Spanish, "¡Hola!"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Sound effects, ambient noise, music
|
|
63
|
+
|
|
64
|
+
```
|
|
65
|
+
SFX: thunder cracks, footsteps approaching
|
|
66
|
+
Ambient noise: city traffic, birds chirping, ocean waves
|
|
67
|
+
Background music: tense orchestral strings, low cello
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
SFX accepts physical-cause specificity:
|
|
71
|
+
- ✅ `SFX: heavy boots on wet pavement, distant siren wailing`
|
|
72
|
+
- ❌ `SFX: footsteps`
|
|
73
|
+
|
|
74
|
+
## Image-to-video guidance
|
|
75
|
+
|
|
76
|
+
**Verbatim (fal blog):**
|
|
77
|
+
> "Treat the input image as an anchor. Kling 3.0 excels at preserving the identity, layout, and text details. Focus prompts on how the scene evolves *from* the image: subtle movements, camera motion, or environmental changes."
|
|
78
|
+
|
|
79
|
+
**Don't re-describe what's already in the image.** Focus on motion, changes, evolution.
|
|
80
|
+
|
|
81
|
+
## Multi-shot — what makes them hit
|
|
82
|
+
|
|
83
|
+
**Hard cap: total duration ≤ 15s across all shots. Max 6 cuts.**
|
|
84
|
+
|
|
85
|
+
Hit conditions:
|
|
86
|
+
- Shot labels are explicit: `Shot 1:`, `Shot 2:`
|
|
87
|
+
- One primary action per shot
|
|
88
|
+
- Subject described identically in each shot block
|
|
89
|
+
- Camera move per shot is **one verb**, not a chain
|
|
90
|
+
- Per-shot blocks: 30-60 words
|
|
91
|
+
|
|
92
|
+
Miss conditions:
|
|
93
|
+
- Compressing narrative into one paragraph
|
|
94
|
+
- Pronoun-only references after the first shot
|
|
95
|
+
- Mixing camera moves within a shot ("pan then orbit then push in")
|
|
96
|
+
- Extreme wide → extreme close in adjacent shots without reference images
|
|
97
|
+
|
|
98
|
+
## Element references (Omni)
|
|
99
|
+
|
|
100
|
+
Upload 2-4 multi-angle reference photos per character/object. Tag inline:
|
|
101
|
+
|
|
102
|
+
```
|
|
103
|
+
@element1 is the protagonist (refs: front, side, back angles).
|
|
104
|
+
@element2 is the antagonist.
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
## Negative prompting — has a real field
|
|
108
|
+
|
|
109
|
+
Kling exposes `negative_prompt` on the fal endpoint (different from Seedance which has none). Default block to start from:
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
blurry, low quality, watermark, text overlay, distorted hands, extra fingers,
|
|
113
|
+
duplicate limbs, unnatural skin texture, overly saturated colors, lens flare,
|
|
114
|
+
floating objects, inconsistent shadows, jittery, flickering, morphing face
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
Layer scene-specific suppressions on top.
|
|
118
|
+
|
|
119
|
+
## Cinematic tactics
|
|
120
|
+
|
|
121
|
+
- **Motion adverb precision** modulates motion energy directly: `slowly`, `rapidly`, `gently`, `explosively`
|
|
122
|
+
- **Camera vocabulary that registers as instructions:** profile shot, tracking, following, freezing, panning, "moving in sync with the subject"
|
|
123
|
+
- **One primary camera move per shot** — never stack
|
|
124
|
+
|
|
125
|
+
## Tier choice
|
|
126
|
+
|
|
127
|
+
- **Standard**: general use, no audio
|
|
128
|
+
- **Pro**: higher visual quality, no audio
|
|
129
|
+
- **Omni**: multi-character dialogue, audio-visual co-gen, language codes, `@elementN` references
|
|
130
|
+
|
|
131
|
+
Pick by capability: need dialogue/audio → Omni; need maximum visual quality silent → Pro; everything else → Standard. Prices change — call `slates_estimate_generation_cost` or `slates_list_available_models` for current numbers before choosing a tier.
|
|
132
|
+
|
|
133
|
+
## Benchmark prompt structure
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
[Character A: <role>, <voice quality>]: "<line>." Immediately, [Character B: <role>, <voice quality>]: "<reply>."
|
|
137
|
+
Ambient noise: <soundscape>.
|
|
138
|
+
Camera <single move>.
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Cinematic example (paraphrasing fal blog patterns):
|
|
142
|
+
> "Shot 1: Wide establishing shot of a neon-lit alleyway in heavy rain, steam rising from grates. Camera slowly tracks forward.
|
|
143
|
+
> Shot 2: Medium shot of a detective in a trench coat ducking under an awning, water dripping from his hat brim. [Detective: weary, raspy]: 'I knew she'd come back.' Ambient noise: distant traffic, rain on metal.
|
|
144
|
+
> Shot 3: Close-up on his eyes, narrowing as headlights flash across his face."
|
|
145
|
+
|
|
146
|
+
## Pre-flight: references arrive inline, refer by code
|
|
147
|
+
|
|
148
|
+
When you call `slates_generate_video` with `firstFrameAssetId` or `ingredientAssetIds`, the first call returns those references **inline as image content blocks** alongside cost + `requires_confirm: true`. Look at them, revise prompt if needed, then re-call with `confirm=true`. Kling Omni multi-character with several ingredient images especially benefits — confirm each character image lands cleanly before spending.
|
|
149
|
+
|
|
150
|
+
When talking to the user about the gen, refer to each reference by its short code: `IMG-A12 — Detective Closeup`. The user sees that code as a gallery badge.
|
|
151
|
+
|
|
152
|
+
- ✅ "I'm anchoring on **IMG-A12** as the detective and **IMG-A18** as the alleyway environment — Omni will handle the line delivery in EN."
|
|
153
|
+
- ❌ "I'm using the detective image and the alley one..." (which alley? Three exist.)
|
|
154
|
+
|
|
155
|
+
## Sources
|
|
156
|
+
|
|
157
|
+
- [fal.ai — Kling 3.0 Prompting Guide](https://blog.fal.ai/kling-3-0-prompting-guide/)
|
|
158
|
+
- [Vidguru — Kling 3.0 Omni Guide](https://www.vidguru.ai/blog/kling-3.0-omni-guide.html)
|
|
159
|
+
- [AcceptPrompt — Kling 3 Prompt Guide](https://www.acceptprompt.com/blog/kling-3-prompt-guide)
|
|
160
|
+
- [DataCamp — Kling 3.0 Tutorial](https://www.datacamp.com/tutorial/kling-3-0)
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: slates-prompting-lip-sync
|
|
3
|
+
description: How to set up Kling lip-sync. Read before calling slates_generate_lip_sync. Two flows — video→video re-dub and image→video avatar — with different inputs, pricing, and gotchas. Voice catalog, framing rules, audio file constraints, and which tier to pick.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Kling lip-sync — setup guide
|
|
7
|
+
|
|
8
|
+
Two distinct flows, both 5-second outputs:
|
|
9
|
+
|
|
10
|
+
| Flow | Source | Model | Cost (5s) | Use case |
|
|
11
|
+
|------|--------|-------|-----------|----------|
|
|
12
|
+
| Re-dub | video clip | kling-lip-sync-video | $0.11 | Replace dialogue on an existing talking head |
|
|
13
|
+
| Avatar standard | still image | ai-avatar/v2/standard | $0.42 | Animate a portrait into a talking avatar |
|
|
14
|
+
| Avatar pro | still image | ai-avatar/v2/pro | $0.86 | Higher facial fidelity for hero shots |
|
|
15
|
+
|
|
16
|
+
Pick `sourceType` deliberately — it decides the pricing tier and the underlying endpoint.
|
|
17
|
+
|
|
18
|
+
## Choosing video vs avatar
|
|
19
|
+
|
|
20
|
+
Use **video** (re-dub) when:
|
|
21
|
+
- A talking-head clip already exists (Slates-generated, recorded, or imported)
|
|
22
|
+
- The mouth/face is already moving and only the audio needs to change
|
|
23
|
+
- $0.11 is hard to beat for short dialogue replacement
|
|
24
|
+
|
|
25
|
+
Use **avatar** when:
|
|
26
|
+
- Only a still portrait exists
|
|
27
|
+
- The character needs to come alive from a single image
|
|
28
|
+
- Identity + face fidelity matter (avatar-pro for hero shots, standard for everything else)
|
|
29
|
+
|
|
30
|
+
## Source asset constraints
|
|
31
|
+
|
|
32
|
+
### Video flow (`sourceType: 'video'`)
|
|
33
|
+
- Format: mp4 or mov
|
|
34
|
+
- Duration: 2–10s (lip-sync output is always 5s — long videos get trimmed)
|
|
35
|
+
- Resolution: 720p or 1080p (480p will be rejected)
|
|
36
|
+
- Max file size: 100MB
|
|
37
|
+
- Face must be visible and roughly facing camera. Profile shots fail.
|
|
38
|
+
- Existing audio is replaced.
|
|
39
|
+
|
|
40
|
+
### Avatar flow (`sourceType: 'image'`)
|
|
41
|
+
- Min 512×512, PNG/JPG/WebP
|
|
42
|
+
- **Face occupies 60–70% of frame.** This is the single biggest avatar quality lever.
|
|
43
|
+
- Eyes open, mouth neutral, looking near-camera. Side profile = bad output.
|
|
44
|
+
- Single subject, clean background. Group photos confuse the face anchor.
|
|
45
|
+
|
|
46
|
+
## Audio source
|
|
47
|
+
|
|
48
|
+
Two ways to drive the lips:
|
|
49
|
+
|
|
50
|
+
### TTS (`audioMethod: 'tts'`)
|
|
51
|
+
- Pass `ttsText` (the words spoken)
|
|
52
|
+
- Optional: `ttsVoice` (default `oversea_male1`), `ttsLanguage` (default EN), `ttsSpeed` (default 1.0)
|
|
53
|
+
- **Hard cap: 120 characters of text.** Longer = silently truncated.
|
|
54
|
+
- Languages: EN, ZH, JA, KO, ES
|
|
55
|
+
|
|
56
|
+
### Upload (`audioMethod: 'upload'`)
|
|
57
|
+
- Pass `audioFilePath` — absolute path to an audio file on the user's machine
|
|
58
|
+
- Format: mp3, wav, m4a, ogg, aac
|
|
59
|
+
- Max 5MB
|
|
60
|
+
- Duration: 2–60s (output is 5s — longer audio gets trimmed)
|
|
61
|
+
- Single clean voice. Music underneath, multiple speakers, or noisy mics produce garbage lips.
|
|
62
|
+
|
|
63
|
+
Prefer upload for production-quality voice. TTS for fast iteration / placeholder dialogue.
|
|
64
|
+
|
|
65
|
+
## Voice catalog (TTS)
|
|
66
|
+
|
|
67
|
+
Reliable English voices (verified working on the fal endpoint as of 2026):
|
|
68
|
+
|
|
69
|
+
| Voice ID | Description |
|
|
70
|
+
|----------|-------------|
|
|
71
|
+
| `oversea_male1` | Male, English — default, stable |
|
|
72
|
+
| `commercial_lady_en_f-v1` | Female commercial English |
|
|
73
|
+
| `uk_boy1` | Young man, UK accent |
|
|
74
|
+
| `uk_man2` | Man, UK accent |
|
|
75
|
+
| `uk_oldman3` | Older man, UK accent |
|
|
76
|
+
| `calm_story1` | Storyteller / narrator |
|
|
77
|
+
|
|
78
|
+
Avoid `reader_en_m-v1` — listed in fal.ai docs but returns "Voice id not found" in production.
|
|
79
|
+
|
|
80
|
+
Full 48-voice list (ZH, JA, KO included): https://fal.ai/models/fal-ai/kling-video/lipsync/text-to-video/api
|
|
81
|
+
|
|
82
|
+
## Speech-rate notes
|
|
83
|
+
|
|
84
|
+
`ttsSpeed` range 0.5–2.0:
|
|
85
|
+
- 0.8–1.0: natural conversational
|
|
86
|
+
- 1.1–1.3: punchy ad delivery
|
|
87
|
+
- 1.4+: rushed, clips consonants
|
|
88
|
+
- 0.6–0.7: slow, weighty (good for dramatic lines)
|
|
89
|
+
|
|
90
|
+
Default 1.0 unless the line specifically calls for slower or faster cadence.
|
|
91
|
+
|
|
92
|
+
## Avatar prompt usage
|
|
93
|
+
|
|
94
|
+
The `prompt` parameter on avatar-v2 (standard + pro) is **scene context**, not motion direction. The mouth animation comes from the audio — the prompt sets ambiance, lighting, micro-expression.
|
|
95
|
+
|
|
96
|
+
Good:
|
|
97
|
+
- `Soft rim light, warm office, gentle confident smile between sentences.`
|
|
98
|
+
- `Cool blue evening light through a window, focused intent expression.`
|
|
99
|
+
|
|
100
|
+
Bad (the model ignores motion verbs):
|
|
101
|
+
- ❌ `She turns her head, raises an eyebrow, then speaks.`
|
|
102
|
+
- ❌ `Hand gestures while talking.`
|
|
103
|
+
|
|
104
|
+
Default `"."` is fine if you have nothing useful to add.
|
|
105
|
+
|
|
106
|
+
## Tier selection — avatar standard vs pro
|
|
107
|
+
|
|
108
|
+
**Use standard** when:
|
|
109
|
+
- Drafts, A/B testing voices, internal review reels
|
|
110
|
+
- Wide / medium shots where face isn't the focal point
|
|
111
|
+
- Cost matters more than micro-expression fidelity
|
|
112
|
+
|
|
113
|
+
**Use pro** when:
|
|
114
|
+
- Final ads where the avatar's face fills the screen
|
|
115
|
+
- The character is named / branded — identity drift kills the take
|
|
116
|
+
- You're already paying $1+ for the surrounding video pipeline
|
|
117
|
+
|
|
118
|
+
Don't default to pro. The $0.44 delta per take adds up across iteration.
|
|
119
|
+
|
|
120
|
+
## Common failure modes
|
|
121
|
+
|
|
122
|
+
| Symptom | Likely cause | Fix |
|
|
123
|
+
|---------|--------------|-----|
|
|
124
|
+
| Lip movement looks "rubber" / disconnected | Source face <60% of frame | Re-crop the still tighter |
|
|
125
|
+
| Voice doesn't match character age/gender | Default voice id used | Pick from voice catalog |
|
|
126
|
+
| Output truncated mid-word | TTS text >120 chars | Shorten or chain two takes |
|
|
127
|
+
| Garbled mouth on uploaded audio | Background music / multi-voice | Use clean dialogue-only audio |
|
|
128
|
+
| "Voice id not found" 422 | Hit `reader_en_m-v1` | Switch to `oversea_male1` |
|
|
129
|
+
| Avatar eyes drift / cross | Source had closed/angled eyes | Pick a frame with neutral open eyes |
|
|
130
|
+
| Generation completes but lips don't move | Profile shot / face >70° off-axis | Use a near-frontal portrait |
|
|
131
|
+
|
|
132
|
+
## Cost discipline
|
|
133
|
+
|
|
134
|
+
- Video re-dub at $0.11 is the cheapest dialogue iteration in the entire Slates stack — use it for voice A/B testing
|
|
135
|
+
- Avatar standard at $0.42 is fine for medium use
|
|
136
|
+
- Avatar pro at $0.86 trips the >$0.50 confirm gate — explicit user OK required every time
|
|
137
|
+
- All 5s. There is no shorter option.
|
|
138
|
+
|
|
139
|
+
## Workflow patterns
|
|
140
|
+
|
|
141
|
+
**Voice A/B test (cheap):**
|
|
142
|
+
1. Generate one base talking-head video clip with Veo or Seedance (~$1.20)
|
|
143
|
+
2. Run `slates_generate_lip_sync` with `sourceType: 'video'` against 3–5 different `ttsVoice` values
|
|
144
|
+
3. Total cost: $1.20 + (5 × $0.11) = $1.75 to compare voices
|
|
145
|
+
|
|
146
|
+
**Brand avatar from a single portrait:**
|
|
147
|
+
1. Generate or upload the hero portrait (face fills frame, eyes open, neutral mouth)
|
|
148
|
+
2. Avatar standard for first-pass dialogue takes
|
|
149
|
+
3. Avatar pro only on the final selected take
|
|
150
|
+
|
|
151
|
+
**Avoid:**
|
|
152
|
+
- Avatar pro on first iteration (waste — facial fidelity isn't visible until you've locked the line)
|
|
153
|
+
- TTS for final ads (production should use real voice or cloned voice — the upload flow)
|
|
154
|
+
- Uploading raw recordings — clean noise + level the file first, lip detection is sensitive
|
|
155
|
+
|
|
156
|
+
## Confirm gate: cost + codes, no inline preview
|
|
157
|
+
|
|
158
|
+
Lip-sync is mechanical — the model re-syncs the chosen source to the chosen audio. The confirm response carries the source asset's code so you can announce it in chat.
|
|
159
|
+
|
|
160
|
+
- ✅ "Lip-syncing **IMG-A12 — Founder Portrait** to the new line. $0.86 on avatar-pro. Confirm?"
|
|
161
|
+
- ❌ "Using the founder image..." (which? Three exist.)
|
|
162
|
+
|
|
163
|
+
Don't second-guess the source. If the output is wrong, iterate on source choice or audio, not on a refinement prompt (there isn't one).
|
|
164
|
+
|
|
165
|
+
## Sources
|
|
166
|
+
|
|
167
|
+
- [fal.ai — Kling LipSync API](https://fal.ai/models/fal-ai/kling-video/lipsync/text-to-video/api)
|
|
168
|
+
- [fal.ai — AI Avatar v2 Standard](https://fal.ai/models/fal-ai/kling-video/ai-avatar/v2/standard/api)
|
|
169
|
+
- [fal.ai — AI Avatar v2 Pro](https://fal.ai/models/fal-ai/kling-video/ai-avatar/v2/pro/api)
|
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: slates-prompting-motion-transfer
|
|
3
|
+
description: How to set up Kling Motion Control (motion transfer). Read before calling slates_generate_motion_transfer. Reference image (character) + driving video (motion source) → new video of the character performing the motion. Asset selection rules, character_orientation choice, std vs pro tier, and prompt usage.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Kling Motion Control — setup guide
|
|
7
|
+
|
|
8
|
+
Take a still **target image** (your character) and a **source video** (the motion you want), produce a new 5s video of your character performing the source video's motion.
|
|
9
|
+
|
|
10
|
+
| Tier | Cost (5s) | Use case |
|
|
11
|
+
|------|-----------|----------|
|
|
12
|
+
| std (`kling-mc-std-5s`) | $0.95 | General motion transfer |
|
|
13
|
+
| pro (`kling-mc-pro-5s`) | $1.26 | Cleaner anatomy, better identity preservation |
|
|
14
|
+
|
|
15
|
+
Both tiers trip the >$0.50 confirm gate. User OK required every time.
|
|
16
|
+
|
|
17
|
+
## Inputs
|
|
18
|
+
|
|
19
|
+
- `sourceVideoAssetId` — driving video. **Must be a realistic human** with clear proportions. Anime/cartoon/CG driving videos fail.
|
|
20
|
+
- `targetImageAssetId` — character to be animated. Can be any style (cartoon, anime, realistic, painted).
|
|
21
|
+
- Both must already exist as assets in the project. Use `slates_list_assets` to find them or upload first.
|
|
22
|
+
|
|
23
|
+
## Source video constraints
|
|
24
|
+
|
|
25
|
+
- Realistic human (not animated, not CG)
|
|
26
|
+
- Entire body OR upper body visible — head must not be obstructed
|
|
27
|
+
- Subject occupies a clear share of the frame
|
|
28
|
+
- Single primary subject. Multi-person driving videos confuse the motion anchor.
|
|
29
|
+
- Clean motion — choppy / cut-edited driving videos produce jittery output
|
|
30
|
+
|
|
31
|
+
Good driving video sources:
|
|
32
|
+
- Reference dance footage with one subject
|
|
33
|
+
- Walking / gesture / posing clips
|
|
34
|
+
- Talking-head footage when paired with character_orientation: 'video'
|
|
35
|
+
|
|
36
|
+
Bad driving video sources:
|
|
37
|
+
- Music videos with multi-shot edits
|
|
38
|
+
- Anime / animation clips
|
|
39
|
+
- Heavily stylized footage with smoke / particles obscuring the body
|
|
40
|
+
- Footage where the subject's head leaves frame mid-clip
|
|
41
|
+
|
|
42
|
+
## Target image constraints
|
|
43
|
+
|
|
44
|
+
- Character body proportions clearly visible
|
|
45
|
+
- Character occupies >5% of image area (not a tiny figure in a wide shot)
|
|
46
|
+
- Single character. Group images break the identity anchor.
|
|
47
|
+
- Any artistic style works — cartoon, anime, painted, realistic, 3D render
|
|
48
|
+
|
|
49
|
+
Avoid:
|
|
50
|
+
- Extreme close-up of just the face (no body to drive)
|
|
51
|
+
- Character partially cropped at the waist when the driving video is full-body
|
|
52
|
+
- Multiple characters
|
|
53
|
+
|
|
54
|
+
## character_orientation — the most-missed choice
|
|
55
|
+
|
|
56
|
+
This single parameter changes the output dramatically. Pick deliberately.
|
|
57
|
+
|
|
58
|
+
| Value | Output framing | Max source duration | Best for |
|
|
59
|
+
|-------|----------------|---------------------|----------|
|
|
60
|
+
| `video` | Matches driving video framing | Up to 30s source | Complex full-body motion (dance, action, athletics) |
|
|
61
|
+
| `image` | Matches target image framing | Up to 10s source | Camera moves, simpler motion, preserving original composition |
|
|
62
|
+
|
|
63
|
+
**Default `video`** when the driving video has the look you want (most cases).
|
|
64
|
+
|
|
65
|
+
Switch to `image` when the target image's composition is the brand asset and the motion is secondary (e.g., a hero shot of a character that needs subtle gesture, not a full performance).
|
|
66
|
+
|
|
67
|
+
## Tier choice — std vs pro
|
|
68
|
+
|
|
69
|
+
**std ($0.95)** for:
|
|
70
|
+
- Drafts, motion exploration, blocking
|
|
71
|
+
- Group scenes where the character isn't a hero shot
|
|
72
|
+
- When the budget is tight and the motion is the focus
|
|
73
|
+
|
|
74
|
+
**pro ($1.26)** for:
|
|
75
|
+
- Final hero takes
|
|
76
|
+
- Branded characters where identity drift = unacceptable
|
|
77
|
+
- Anatomically complex motion (limbs crossing, fast direction changes)
|
|
78
|
+
- Anime / cartoon target images — pro handles non-realistic styles better
|
|
79
|
+
|
|
80
|
+
Don't default to pro. The $0.31 delta compounds fast across iteration.
|
|
81
|
+
|
|
82
|
+
## Prompt usage (optional)
|
|
83
|
+
|
|
84
|
+
The `prompt` field is **scene/style refinement**, not motion direction. The motion comes from the driving video — the prompt sets ambiance, lighting, additional detail.
|
|
85
|
+
|
|
86
|
+
Good:
|
|
87
|
+
- `Soft afternoon sunlight, dust motes in the air, vintage warm color grade.`
|
|
88
|
+
- `Clean studio backdrop, sharp focus on the character.`
|
|
89
|
+
|
|
90
|
+
Bad (model ignores motion verbs — they're already in the driving video):
|
|
91
|
+
- ❌ `She spins faster and jumps higher.`
|
|
92
|
+
- ❌ `Add more energy to the dance.`
|
|
93
|
+
|
|
94
|
+
Leave it empty if you don't have a specific atmospheric note.
|
|
95
|
+
|
|
96
|
+
## Common failure modes
|
|
97
|
+
|
|
98
|
+
| Symptom | Likely cause | Fix |
|
|
99
|
+
|---------|--------------|-----|
|
|
100
|
+
| Limbs distort / extra fingers | std tier, complex motion | Switch to pro |
|
|
101
|
+
| Character identity drifts | Target image cropped too tight | Use a fuller-body target |
|
|
102
|
+
| Output looks "stuck" / minimal motion | Driving video subject too small in frame | Pick a driving video where the subject fills more of the frame |
|
|
103
|
+
| Cartoon target turns realistic | std tier on stylized art | Switch to pro — handles non-realistic styles better |
|
|
104
|
+
| Garbled output entirely | Anime / CG driving video | Use realistic human driving footage |
|
|
105
|
+
| Wrong framing on output | character_orientation set wrong | Try the other value |
|
|
106
|
+
| Background bleeds through character | Target image had complex background | Use a target with cleaner background separation |
|
|
107
|
+
|
|
108
|
+
## Workflow patterns
|
|
109
|
+
|
|
110
|
+
**Reference dance to brand character:**
|
|
111
|
+
1. Generate or upload the brand character as a still image (clean background, full body, single subject)
|
|
112
|
+
2. Find driving footage — a clean reference video of the dance you want
|
|
113
|
+
3. Upload both as project assets
|
|
114
|
+
4. Run motion transfer with `motionModel: 'kling-mc-pro'`, `characterOrientation: 'video'`
|
|
115
|
+
5. Total cost: $1.26 per 5s take
|
|
116
|
+
|
|
117
|
+
**Subtle motion on a hero portrait:**
|
|
118
|
+
1. Use the locked hero portrait as the target image
|
|
119
|
+
2. Pick a driving video with subtle gesture (head turn, slight posture shift)
|
|
120
|
+
3. `characterOrientation: 'image'` to preserve the portrait's framing
|
|
121
|
+
4. std tier is fine for this case — motion isn't dramatic
|
|
122
|
+
|
|
123
|
+
**Avoid:**
|
|
124
|
+
- Pro tier on first iteration — waste, switch to it once the motion + framing combo is locked
|
|
125
|
+
- Cartoon driving videos — guaranteed failure
|
|
126
|
+
- Cropped or partial target characters — identity will drift
|
|
127
|
+
- Long driving videos when output is 5s — pick the best 5s of the source upfront
|
|
128
|
+
|
|
129
|
+
## Cost discipline
|
|
130
|
+
|
|
131
|
+
- 5 seconds, no shorter option
|
|
132
|
+
- Both tiers trip the >$0.50 confirm gate — every call needs explicit user OK
|
|
133
|
+
- Iteration is expensive: 4 takes at pro = $5.04. Lock framing + driving video before tier-up to pro.
|
|
134
|
+
- Always run a single std take first to validate the motion + framing combo before committing to pro
|
|
135
|
+
|
|
136
|
+
## Confirm gate: cost + codes, no inline preview
|
|
137
|
+
|
|
138
|
+
Motion transfer is mechanical — the model deterministically applies source motion to target image. Both tiers trip the >$0.50 confirm gate; the response includes the asset codes for source and target so you can announce them in chat.
|
|
139
|
+
|
|
140
|
+
- ✅ "Transferring motion from **VID-V3** onto **IMG-A12 — Detective Closeup**. $1.26, confirm?"
|
|
141
|
+
- ❌ "Using the walk video and the detective image..." (multiple of each in the project.)
|
|
142
|
+
|
|
143
|
+
Don't second-guess the assets the user picked — the model executes the transfer. If the output is wrong, iterate on motion source or target choice, not on a refinement prompt.
|
|
144
|
+
|
|
145
|
+
## Sources
|
|
146
|
+
|
|
147
|
+
- [fal.ai — Kling Motion Control V3 Standard](https://fal.ai/models/fal-ai/kling-video/v3/standard/motion-control)
|
|
148
|
+
- [fal.ai — Kling Motion Control V3 Pro](https://fal.ai/models/fal-ai/kling-video/v3/pro/motion-control)
|