@mevdragon/vidfarm-devcli 0.14.0 → 0.15.0

package/.agents/skills/vidfarm-media/references/tts.md

@@ -0,0 +1,54 @@
+# TTS / STT providers — voices, timings, formats
+
+Two tiers, one preflight. **BYOK cloud** rides the user's own provider key through vidfarm; **keyless local** runs Kokoro-82M and whisper.cpp on the user's machine with no account and no key. There is never a sign-in step.
+
+## Preflight
+
+- `vidfarm provider-keys` lists saved platform keys; env `OPENAI_API_KEY` / `GEMINI_API_KEY` / `OPENROUTER_API_KEY` enable the local-first BYOK path directly.
+- Any key present → prefer `vidfarm tts` / `vidfarm stt` (better voices, promptable style, real word timestamps on OpenAI).
+- No key → the local engines below need nothing. State the choice in one line and continue; never stop the workflow to ask for credentials.
+
+## BYOK cloud TTS — `vidfarm tts`
+
+```bash
+vidfarm tts "Welcome back to the channel" --style "calm, warm bedtime narrator" --voice alloy --out narration.mp3 --json
+```
+
+- **Local-first**: runs directly on your env key (no cloud job, no wallet). `--cloud` is the explicit backup — an async platform job on your saved provider keys via `POST /api/v1/primitives/audio/speech` (poll the returned `job_id`).
+- **Promptable voice**: `--style` (alias `--instructions`) directs tone/pacing/accent/emotion/persona — "excited sports announcer", "whispered documentary aside".
+- **Voices** (provider inferred from the voice when `--provider` is omitted; wrong-family voices are rejected with a clear error):
+  - OpenAI (`gpt-4o-mini-tts`): `alloy, ash, ballad, cedar, coral, echo, fable, marin, nova, onyx, sage, shimmer, verse`
+  - Gemini (`gemini-*-tts`, TitleCase, also the default family for OpenRouter TTS): `Kore, Puck, Charon, Aoede, Fenrir, Leda, Orus, Zephyr, Achernar, Achird, Algenib, Algieba, Alnilam, Autonoe, Callirrhoe, Despina, Enceladus, Erinome, Gacrux, Iapetus, Laomedeia, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zubenelgenubi`
+- **Formats**: `--format mp3|wav` (default mp3). `--json` prints `{ ok, provider, model, out, … }`.
+
+## Keyless local TTS — Kokoro-82M
+
+```bash
+npx hyperframes tts "Welcome back" -v af_heart -o line.wav --json   # {"ok":true,"durationSeconds":…,"outputPath":…}
+```
+
+- Workflow prose sometimes calls this "`--engine local`"; the actual keyless engine is the `hyperframes` CLI (the engine script `scripts/audio.mjs` invokes it for you and resolves a repo-local install before falling back to `npx -y hyperframes`).
+- Voices: `af_heart` (default), `af_nova`, `af_sky`, `am_adam`, `am_michael`, `bf_emma`, `bf_isabella`, `bm_george`, `ef_dora`, `ff_siwis`, `jf_alpha`, `zf_xiaobei` — prefix = language (`a` en-US, `b` en-GB, `e` es, `f` fr, `j` ja, `z` zh). `--list` prints them.
+- `-s <speed>` multiplier; `--lang` overrides phonemizer. Output is WAV. First run downloads model weights (~340 MB) into `~/.cache/hyperframes/tts` — still no account.
+- Text can be a `.txt` file path — use that for long or dash-leading lines.
+
+## Word timestamps (for captions and beat-sync)
+
+| Path | Command | What you get |
+| --- | --- | --- |
+| Real timings, BYOK | `vidfarm stt <file\|url> --out transcript` | OpenAI key routes to **whisper-1** (the only cloud model with true word-level timestamps): plain text + timed SRT cues + speaker-labeled segments (`.json`/`.srt`/`.txt`). Gemini key adds diarization (speaker labels) but estimates word windows. |
+| Real timings, straight into a composition | `vidfarm captions generate <dir>` | Transcribes the mounted narration and lays down animated word-by-word caption layers; whisper-1 timings when an OpenAI key exists, char-weight estimates inside each cue otherwise. |
+| Real timings, zero keys | `npx hyperframes transcribe <audio> --json --dir <tmp>` | Local whisper.cpp (`small.en` default, auto language switch): writes word-level `transcript.json` (`[{id,text,start,end}]`, seconds) and reports `durationSeconds`. `--optional` exits 0 when whisper is unavailable. Always point `--dir` at a scratch dir — it patches caption HTML under that dir. |
+| Estimation fallback | (automatic in `scripts/audio.mjs`) | Words spread across the audio duration by proportional character weighting — same approach vidfarm's caption system uses. Good enough for gentle caption pacing; not for beat-slams. |
+
+## STT extras
+
+- `vidfarm stt` is local-first on your env key (Gemini preferred — it can label speakers); `--no-diarize` for a plain transcript; `--cloud` for the platform job (`POST /api/v1/primitives/audio/transcribe`).
+- `--out <base>` writes all three formats (`.json` + `.srt` + `.txt`); an explicit extension picks one.
+
+## Rules of thumb
+
+- Narration for a video: engine script first (`scripts/audio.mjs`) — it picks cloud vs local per the request and never blocks on a missing key.
+- One-off line or a voice audition: `vidfarm tts` (key) or `npx hyperframes tts` (no key).
+- Captions that must land on the beat: get an OpenAI key path (whisper-1) or local whisper.cpp; don't beat-sync estimated timings.
+- Speed for Kokoro is a synthesis parameter (`-s`); for cloud voices direct pacing through `--style` instead.