@officexapp/vidfarm-devcli 0.21.10

package/.agents/skills/vidfarm-media/references/tts.md

@@ -0,0 +1,111 @@
+# Music / TTS / STT — voices, timings, formats
+
+Three tiers. **ElevenLabs (default)** powers music, high-quality narration, and Scribe
+transcription — on vidfarm's platform key billed to the wallet (`use_wallet_credits: true`, the
+default) or on the customer's OWN saved ElevenLabs key (`use_wallet_credits: false`). **BYOK
+cloud** rides the user's own openai/gemini/openrouter key through vidfarm (TTS/STT fallback when
+`use_wallet_credits` is false and no ElevenLabs key is present). **Keyless local** runs Kokoro-82M
+and whisper.cpp on the user's machine with no account. There is never a sign-in step for the user.
+
+## use_wallet_credits (default true — keep it on)
+
+`music`, `tts` (`--cloud`), and `stt` (`--cloud`) default to ElevenLabs on the platform key,
+billed to the customer's wallet. Recommend keeping it on for the best voices/music with zero
+setup. Turn it OFF only to (a) save wallet credits or (b) use the customer's own ElevenLabs key.
+With it off and no ElevenLabs key, TTS/STT fall back to a BYOK openai/gemini/openrouter key; music
+always needs ElevenLabs. ElevenLabs in Settings → AI keys is **audio-only** — it does NOT count as
+a qualified text/image/video key, so the user still needs an openai/gemini/openrouter key for the
+core generation features.
+
+## Music — `vidfarm music`
+
+```bash
+vidfarm music "chill lo-fi hip hop beat with jazzy piano" --length 30 --json
+```
+
+- ElevenLabs music generation (`POST /api/v1/primitives/music/generate`). `--length <sec>` or
+  `--length-ms`, default 30s, max 5 min. `--own-key` uses the customer's ElevenLabs key instead of
+  the wallet. Returns a durable mp3; mount it as its own `<audio>` layer ~0.1–0.2 under narration.
+- For fine control, the primitive also accepts an ElevenLabs `composition_plan` (section-by-section
+  lyrics/style) — see the loadable `music` skill pack for the plan schema.
+
+## Voices — `vidfarm voices`
+
+```bash
+vidfarm voices                 # platform-account voices (voice_id, name, labels, preview_url)
+vidfarm voices --own-key       # the customer's own ElevenLabs account voices
+```
+
+- `GET /api/v1/primitives/audio/voices` (`?use_wallet_credits=false` for the user's own key).
+- **Default a sensible voice and TELL the user they can pick from many** — surface a few names +
+  the returned `voice_library_url`, then pass the chosen `voice_id` to `tts --voice <voice_id>`.
+
+## Preflight
+
+- `vidfarm provider-keys` lists saved platform keys; env `OPENAI_API_KEY` / `GEMINI_API_KEY` / `OPENROUTER_API_KEY` enable the local-first BYOK path directly.
+- Any key present → prefer `vidfarm tts` / `vidfarm stt` (better voices, promptable style, real word timestamps on OpenAI).
+- No key → the local engines below need nothing. State the choice in one line and continue; never stop the workflow to ask for credentials.
+
+## BYOK cloud TTS — `vidfarm tts`
+
+```bash
+vidfarm tts "Welcome back to the channel" --style "calm, warm bedtime narrator" --voice alloy --out narration.mp3 --json
+```
+
+- **Local-first**: runs directly on your env key (no cloud job, no wallet). `--cloud` is the explicit backup — an async platform job on your saved provider keys via `POST /api/v1/primitives/audio/speech` (poll the returned `job_id`).
+- **Promptable voice**: `--style` (alias `--instructions`) directs tone/pacing/accent/emotion/persona — "excited sports announcer", "whispered documentary aside".
+- **Voices** (provider inferred from the voice when `--provider` is omitted; wrong-family voices are rejected with a clear error):
+  - OpenAI (`gpt-4o-mini-tts`): `alloy, ash, ballad, cedar, coral, echo, fable, marin, nova, onyx, sage, shimmer, verse`
+  - Gemini (`gemini-*-tts`, TitleCase, also the default family for OpenRouter TTS): `Kore, Puck, Charon, Aoede, Fenrir, Leda, Orus, Zephyr, Achernar, Achird, Algenib, Algieba, Alnilam, Autonoe, Callirrhoe, Despina, Enceladus, Erinome, Gacrux, Iapetus, Laomedeia, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zubenelgenubi`
+- **Formats**: `--format mp3|wav` (default mp3). `--json` prints `{ ok, provider, model, out, … }`.
+
+## Voice-matched regeneration — "same speaker, new words"
+
+```bash
+# Platform job on saved keys (no devcli command yet); poll the returned job_id.
+POST /api/v1/primitives/audio/regenerate-speech   # alias: /api/v1/primitives/regenerate-speech
+{ "tracer": "…", "payload": {
+    "source_url": "…",                 # video or audio; video audio is demuxed automatically
+    "rewrite_instruction": "mention the summer sale instead of the discount code",
+    "text": "…",                       # OR: the exact new script (one of the two is required)
+    "voice": "…", "instructions": "…", # optional preset override / extra style guidance
+    "provider": "…", "model": "…", "language": "…", "output_format": "mp3|wav"
+} }
+```
+
+One job: listens to the source, profiles the dominant speaker with an audio-native Gemini call (verbatim transcript + gender/age/accent/pace/pitch/energy profile + a dense style direction + the closest preset in both voice families), rewords the transcript per `rewrite_instruction` (keeping length and rhythm) or takes the exact `text`, then regenerates via TTS with the matched preset + profiled style instructions. Result: durable `audioUrl`, `script`, `original_transcript`, `voice_profile`, `voice_match`, plus a `regenerate-speech.json` artifact.
+
+- **Honesty rule (non-negotiable): regenerate-speech is an approximation, never a clone.** It runs on OpenAI/Gemini, which expose preset voices with style steering only (no self-serve cloning of an arbitrary speaker). `voice_match` says `"approximate"` or `"none"`; always present it that way. (ElevenLabs IS wired into vidfarm for music, preset-voice TTS, and Scribe STT — but regenerate-speech does not use it for speaker cloning; treat any voice-match as an approximation.)
+- **Voice profiling needs a saved Gemini key** (it is the audio-native LLM). With only openai/openrouter keys the job degrades gracefully: STT transcript + reword still work, but the voice is a default or explicitly passed preset and `voice_match` is `"none"`.
+- Sources over ~40 minutes of speech must be trimmed first (same cap as `/audio/transcribe`).
+- To swap the result into a composition: mute the original span (`/videos/mute` or the layer's volume), then place `audioUrl` as a new audio layer.
+
+```bash
+npx hyperframes tts "Welcome back" -v af_heart -o line.wav --json   # {"ok":true,"durationSeconds":…,"outputPath":…}
+```
+
+- Workflow prose sometimes calls this "`--engine local`"; the actual keyless engine is the `hyperframes` CLI (the engine script `scripts/audio.mjs` invokes it for you and resolves a repo-local install before falling back to `npx -y hyperframes`).
+- Voices: `af_heart` (default), `af_nova`, `af_sky`, `am_adam`, `am_michael`, `bf_emma`, `bf_isabella`, `bm_george`, `ef_dora`, `ff_siwis`, `jf_alpha`, `zf_xiaobei` — prefix = language (`a` en-US, `b` en-GB, `e` es, `f` fr, `j` ja, `z` zh). `--list` prints them.
+- `-s <speed>` multiplier; `--lang` overrides phonemizer. Output is WAV. First run downloads model weights (~340 MB) into `~/.cache/hyperframes/tts` — still no account.
+- Text can be a `.txt` file path — use that for long or dash-leading lines.
+
+## Word timestamps (for captions and beat-sync)
+
+| Path | Command | What you get |
+| --- | --- | --- |
+| Real timings, BYOK | `vidfarm stt <file\|url> --out transcript` | OpenAI key routes to **whisper-1** (the only cloud model with true word-level timestamps): plain text + timed SRT cues + speaker-labeled segments (`.json`/`.srt`/`.txt`). Gemini key adds diarization (speaker labels) but estimates word windows. |
+| Real timings, straight into a composition | `vidfarm captions generate <dir>` | Transcribes the mounted narration and lays down animated word-by-word caption layers; whisper-1 timings when an OpenAI key exists, char-weight estimates inside each cue otherwise. |
+| Real timings, zero keys | `npx hyperframes transcribe <audio> --json --dir <tmp>` | Local whisper.cpp (`small.en` default, auto language switch): writes word-level `transcript.json` (`[{id,text,start,end}]`, seconds) and reports `durationSeconds`. `--optional` exits 0 when whisper is unavailable. Always point `--dir` at a scratch dir — it patches caption HTML under that dir. |
+| Estimation fallback | (automatic in `scripts/audio.mjs`) | Words spread across the audio duration by proportional character weighting — same approach vidfarm's caption system uses. Good enough for gentle caption pacing; not for beat-slams. |
+
+## STT extras
+
+- `vidfarm stt` is local-first on your env key (Gemini preferred — it can label speakers); `--no-diarize` for a plain transcript; `--cloud` for the platform job (`POST /api/v1/primitives/audio/transcribe`).
+- `--out <base>` writes all three formats (`.json` + `.srt` + `.txt`); an explicit extension picks one.
+
+## Rules of thumb
+
+- Narration for a video: engine script first (`scripts/audio.mjs`) — it picks cloud vs local per the request and never blocks on a missing key.
+- One-off line or a voice audition: `vidfarm tts` (key) or `npx hyperframes tts` (no key).
+- Captions that must land on the beat: get an OpenAI key path (whisper-1) or local whisper.cpp; don't beat-sync estimated timings.
+- Speed for Kokoro is a synthesis parameter (`-s`); for cloud voices direct pacing through `--style` instead.