@mevdragon/vidfarm-devcli 0.14.0 → 0.16.0

package/.agents/skills/website-to-video/references/step-4-vo.md

@@ -0,0 +1,225 @@
+# Step 4: VO, Timing + Captions
+
+## If Step 2 said "no narration"
+
+Skip the TTS sections below. The storyboard already has beat durations planned based on pacing and rhythm — those become `data-start` and `data-duration` values directly in Step 5.
+
+**Background music:** Ask the user before moving to Step 5:
+
+> "Do you have a music track for this video? If not, I can suggest where to find one:
+>
+> - **Artlist.io** or **Musicbed** — licensed music for commercial use
+> - **Uppbeat.io** or **Pixabay Music** — free tracks with attribution
+> - **Freesound.org** — free samples and loops
+>
+> Or share a reference track ('something like this') and I can find something similar."
+
+If the user provides a track: note the file path and BPM in the storyboard for Step 5 to wire into `index.html`. If they skip music entirely, the video uses SFX only — confirm that's intentional.
+
+Move to Step 5.
+
+---
+
+## Generate a test clip before full narration — calibrate timing first
+
+Generate a 2-sentence test clip NOW using the script's opening lines. Measure the actual duration. Kokoro compresses scripts by ~40% (35s planned → 19s actual) and the cloud TTS providers often run faster than expected. If you discover the audio is 40% shorter than expected, you'll need to revise the storyboard beat timings before investing time in full narration generation.
+
+**Do this before committing to beat count and durations:**
+
+```bash
+# Quick Kokoro test (2 sentences):
+npx hyperframes tts "First sentence. Second sentence." --voice af_nova --output /tmp/test-tts.wav
+# Measure: seconds ÷ words × total script words = estimated full audio length
+```
+
+If the estimate puts your video at ±15% of the planned duration, proceed. If it's more than 15% off, recalibrate the script length first:
+
+- **Audio TOO SHORT** (more than 15% under planned duration) → add strategic pauses. In `narration.txt`, insert blank lines between paragraphs (≈0.6s each) or `...` between sentences (≈0.4s each). Aim for the pauses to land at storyboard beat boundaries so the silence feels intentional, not dead air.
+- **Audio TOO LONG** (more than 15% over planned duration) → identify the beat in your storyboard with the highest words-per-second density. Cut one supporting sentence from THAT beat's lines — preserve the lead sentence (the one that names the beat's idea). Re-measure with another test clip before committing to full generation.
+- **Audio matches plan but beat boundaries drift** → adjust the storyboard durations to match the actual narration, not the other way around. The audio is the ground truth once narration is generated.
+
+The script formula assumes constant words-per-second, but punctuation, dramatic pauses, and silence cues all stretch real audio. Always trust a measured test clip over the formula.
+
+## Background music
+
+**Always ask about background music** — even when narration is present:
+
+> "Do you want background music under the narration? (Artlist.io, Musicbed for licensed; Uppbeat/Pixabay for free; or share a reference track). Even a subtle ambient underscore makes pauses between sentences feel intentional rather than empty."
+
+If they want music, note the track in the storyboard for Step 5 to wire into `index.html`.
+
+## TTS Provider
+
+Ask the user which voice provider they'd like:
+
+> **Which voice provider would you like to use for narration?**
+>
+> 1. **vidfarm TTS (BYOK)** — Promptable voice on your saved AI provider key (describe the read with `--style`). Word-level timestamps come from `vidfarm stt` (real word timings on an OpenAI whisper-1 key) or `vidfarm captions generate` — no timing disadvantage vs any provider. Requires a saved provider key (`vidfarm provider-keys`).
+> 2. **ElevenLabs** — Large voice library, very natural output. Requires ElevenLabs API key. Does not return word timestamps — you'll transcribe separately.
+> 3. **Kokoro** (Free) — Runs locally, no API key needed (`vidfarm tts --engine local`). Decent quality but more robotic than the others. Good for drafts or budget runs.
+
+If the user picks a keyed provider and doesn't have a key set up yet, help them:
+
+- **vidfarm TTS:** run `vidfarm provider-keys` to list saved AI keys (openai / gemini / openrouter). If none are saved, add one — or fall back to the keyless local engine (`vidfarm tts --engine local`).
+- **ElevenLabs:** "Add `ELEVENLABS_API_KEY=your-key` to a `.env` file in the project root, or just paste it here and I'll set it up."
+
+Don't judge or critique if the user pastes a key directly in chat — just use it and move on.
+
+## Audition voices
+
+After the provider is selected, audition at least 2 voices with the first sentence of SCRIPT.md.
+
+**vidfarm TTS (BYOK):**
+
+- Check which AI provider keys are saved first:
+
+  ```bash
+  vidfarm provider-keys
+  ```
+
+- Generate speech — the voice is promptable, so audition by varying `--style` instead of picking from a fixed voice list:
+
+  ```bash
+  vidfarm tts "First sentence of your script." \
+    --style "warm, confident narrator, medium pace" \
+    --output narration-test-a.wav
+
+  vidfarm tts "First sentence of your script." \
+    --style "bright, energetic announcer" \
+    --output narration-test-b.wav
+  ```
+
+  (REST equivalent: `POST /api/v1/primitives/audio/speech`.)
+
+- Keyless fallback — no provider key needed:
+
+  ```bash
+  vidfarm tts "First sentence of your script." --engine local --output narration-test.wav
+  ```
+
+- Does not return word timestamps inline — get them after generating the full narration with `vidfarm stt narration.wav` (real word timestamps on an OpenAI whisper-1 key), `vidfarm stt narration.wav --engine whisper` (keyless local whisper.cpp), or `vidfarm captions generate` (word-timed captions straight from the narration).
+
+**ElevenLabs:**
+
+- If the ElevenLabs MCP is available: use `mcp__elevenlabs__search_voices` to browse, `mcp__elevenlabs__text_to_speech` to generate.
+- If no MCP: call the REST API directly:
+
+  ```bash
+  # List voices
+  curl -s "https://api.elevenlabs.io/v1/voices" \
+    -H "xi-api-key: $ELEVENLABS_API_KEY" | jq '.voices[:5] | .[].name'
+
+  # Generate speech (replace VOICE_ID with chosen voice)
+  curl -s -X POST "https://api.elevenlabs.io/v1/text-to-speech/VOICE_ID" \
+    -H "xi-api-key: $ELEVENLABS_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d '{"text":"First sentence of your script","model_id":"eleven_multilingual_v2"}' \
+    --output narration.mp3
+  ```
+
+- Does not return word timestamps — transcribe separately after generating.
+
+**Kokoro (free, local):**
+
+```bash
+npx hyperframes tts SCRIPT.md --voice af_nova --output narration.wav
+```
+
+No API key, no MCP needed. Runs locally. Use `--list` to see all 54 available voices.
+
+Pick the voice that sounds most natural and conversational. Listen for pacing — does it breathe between sentences? Does it sound like a person or a robot?
+
+## Script length check
+
+Before generating, verify the script makes sense for the video. Word count depends entirely on the creative direction. The storyboard's pacing and style determine how much narration the video needs.
+
+The key check: are there stretches where NOTHING is happening — no narration AND no compelling visual movement? Those are dead spots that lose the viewer. Every second needs either spoken words or strong visual energy carrying it.
+
+## Generate full narration
+
+Generate the full script as `narration.wav` (or `.mp3`) in the project directory.
+
+**If any command hangs for more than 60 seconds — don't just wait.** The user is sitting there watching you do nothing. Escalation order:
+
+1. **Try again** — kill the process, run the same command again (transient failures are common)
+2. **Try different flags** — smaller model (`--model tiny.en`), different voice, shorter test sentence first
+3. **Try a different tool for the same task** — if `hyperframes transcribe` hangs, run `whisper-cli` directly on the audio
+4. **Switch provider entirely** — if a cloud provider is down, try another provider or Kokoro. If Kokoro hangs, try a cloud provider.
+
+Never sit idle for 10 minutes hoping a stuck process will finish.
+
+**Kokoro pronunciation issues:** Kokoro mispronounces product names and tech terms. Always apply substitutions before generating. Known problems and fixes:
+
+- `API` → `A P I` (spell it out)
+- `UI` → `U I`, `SaaS` → `sass`, `DevOps` → `dev ops`
+- Product names with unusual spelling: test the first sentence first and listen. Common failure: "Vercel" → "versatile", "WorkOS" → "work O S", "One API" → "Wanna PI"
+- If a name sounds wrong: write it phonetically in `narration.txt` (e.g., `Vercel` → `Ver-sell`, `Supabase` → `Soopa-base`)
+- Always generate a short test clip with the first 2 sentences before generating the full audio
+- **No SSML tags** — Kokoro reads them as literal text. `<break time="1s"/>` is spoken as "break time equals one slash." Use blank lines or `...` for pauses in `narration.txt`
+
+For the cloud TTS providers, substitutions are usually unnecessary — they handle product names correctly.
+
+**Also save the exact spoken text** — with pronunciation substitutions applied (e.g., `API` → `A P I`, `$2T` → `two trillion` and etc.) — as `narration.txt` in the same directory. This is the string passed to TTS, distinct from `SCRIPT.md` which is the human-readable creative doc. Having `narration.txt` makes it trivial to regenerate the audio later with a different voice without re-deriving the substitutions. Name it exactly `narration.txt`.
+
+## Transcribe for word-level timestamps
+
+Every provider goes through the same transcription step — one canonical output shape:
+
+```bash
+# BYOK — real word timestamps on an OpenAI whisper-1 key:
+vidfarm stt narration.wav
+
+# Keyless local — whisper.cpp, word timestamps included:
+vidfarm stt narration.wav --engine whisper
+
+# Equivalent via the hyperframes CLI:
+npx hyperframes transcribe narration.wav
+```
+
+Produces `transcript.json` with `[{ text, start, end }]` for every word. These timestamps are the source of truth for all beat durations. (If the video will have captions, `vidfarm captions generate` builds word-timed captions from the same narration.)
+
+## Map timestamps to beats
+
+Go through STORYBOARD.md beat by beat. For each beat:
+
+1. Find the first word of that beat's VO cue in `transcript.json`
+2. Find the last word of that beat's VO cue
+3. Set `beat.start = firstWord.start`, `beat.end = lastWord.end`
+4. Add 0.3-0.5s padding at the end for visual breathing room
+
+Update STORYBOARD.md with real durations. Replace estimated times (e.g., "0:00-0:05") with actual timestamps as precise as possible (e.g., "0.00-3.21s").
+
+Beat boundaries land on word onsets — hard cuts to the VO.
+
+## Timing reconciliation — required before Step 5
+
+After mapping all beats, compare real total audio duration against the storyboard's planned duration:
+
+```
+real_total = last_word.end + cta_hold (typically 2–3s)
+planned_total = sum of all beat planned durations
+delta = |real_total - planned_total|
+```
+
+**If delta > 15% of planned total — do not proceed to Step 5 without resolving it.** Common causes and fixes:
+
+- **Audio shorter than planned (most common with Kokoro):** Kokoro generates compressed speech with minimal pauses. Proportionally scale all non-CTA beat durations down to match the real audio. Example: planned 30s, audio 19s — multiply each beat duration by 19/30 (excluding the CTA hold). Update STORYBOARD.md.
+- **Audio much longer than planned (>30% over):** The script was too long for the intended duration. Trim the script (remove one beat's VO), regenerate audio, re-transcribe.
+- **CTA beat timing:** The CTA beat should hold for 2–3 seconds after the last spoken word — not extend to fill empty time. `cta_start = last_word.end + 0.3s`, `cta_duration = 2.5s`. Hard cap. Dead silence after the CTA hold loses the viewer.
+
+**Always tell the user** if you adjusted durations significantly from the storyboard plan. They approved a specific beat structure — if it changed, they need to know.
+
+## Captions
+
+After the narration is generated and transcribed, ask the user:
+
+> **Would you like captions on the video?**
+>
+> - **Yes** — per-word captions synced to the narration. Great for social media (most viewers watch on mute) and accessibility.
+> - **No** — narration audio only, no text overlay.
+
+If yes, captions are built as a separate composition (`compositions/captions.html`) in Step 5. The `transcript.json` drives the timing — each word appears/highlights as it's spoken. Read [the captions reference](../../hyperframes/references/captions.md) for styling options (scale-pop, typewriter, fade+slide, etc.) and positioning rules.
+
+## Save timing data for Step 5
+
+Record the final beat timings (start, duration) so Step 5 (Build) can use them when building `index.html`. The storyboard now has real timestamps — these become `data-start` and `data-duration` values on each scene slot when the root composition is assembled in Step 5.