@ignix-social/editor 0.1.0

package/vendor/hyperframes/dist/skills/media-use/audio/references/sfx.md

@@ -0,0 +1,42 @@
+# Sound effects (SFX)
+
+Named sound effects, produced by the shared audio engine (`scripts/audio.mjs` → `scripts/lib/sfx.mjs`). **Provider-gated** by the engine's one switch — whether a HeyGen credential is present, decided once (not per cue):
+
+- **HeyGen credential present → retrieve every cue** from HeyGen's audio library (`/v3/audio/sounds`, `type=sound_effects`, `min_score=0.4`). Search-and-download, **not** generation. The bundled library is NOT consulted.
+- **No credential → the bundled 21-file library** (`assets/sfx/` + `manifest.json`): match each cue name, copy the matched file into the project. Offline, deterministic, free.
+
+There is no `npx hyperframes sfx` command. SFX is never generated — it is retrieved (online) or taken from the bundled library (offline).
+
+## Cues — request → meta
+
+Each line names the effects it wants: `lines[].sfx: ["whoosh", "ui click"]`. The engine flattens these into cues, resolves them per the switch, dedupes identical `(id, name)` pairs (the same effect named twice downloads/copies once), and writes `audio_meta.sfx[]`:
+
+```jsonc
+{
+  "id": "3",                       // joins the cue to the caller's model (frame / scene / segment)
+  "name": "whoosh",
+  "file": "assets/sfx/whoosh.mp3", // downloaded or copied, relative to project root
+  "source": "heygen" | "local",    // which route resolved it
+  "offset_s": 0,                   // delay from the line's start
+  "duration_s": 0.57,
+  "volume": 0.35                   // SFX sit UNDER voice + BGM
+}
+```
+
+A cue that matches nothing is **skipped** (recorded as an anomaly); SFX never blocks a render. Neither route replaces a file of yours already at the output name: the cue gets the next free name (`whoosh-2.mp3`), reported as an anomaly, and `file` carries the real path.
+
+## HeyGen retrieval (credentialed)
+
+`searchSounds(name, "sound_effects", { limit: 3, minScore: 0.4 })` → top hit → `assets/sfx/<slug>.mp3`. Results are ranked by `score` (each carries a presigned `audio_url`, `duration`, `description`). The floor is **0.4** because good SFX hits score ~0.5–0.67 — below the API's default `0.7`, which would silently drop most named cues (only whoosh/swoosh-family clears 0.7). `duration_s` comes from the result (else 1.0). Name effects concretely (`glass shatter`, not `dramatic sound`); a vague query returns a poor match.
+
+## Bundled library (no credential)
+
+21 curated files in `assets/sfx/`, indexed by `manifest.json` — `{ file, duration, description }` per key (e.g. `whoosh`, `pop`, `click`, `chime`, `riser`, `impact-bass-1`, `glitch-1`, `typing`, …). A cue name resolves by **manifest key, file basename, or slug**, so `whoosh`, `whoosh.mp3`, or `"ui click"` (→ slug) all match. Matched files are copied into the project's `assets/sfx/`; `duration_s` comes from the manifest, so timing is known **offline** — e.g. `riser` is 10.03s, so trigger it at `climax − 10.03s`. The manifest's `description` field carries placement hints per effect; read `assets/sfx/manifest.json` for the full set and usage.
+
+## Rules
+
+- **Volume ~0.35.** SFX must sit under narration and BGM, not fight them.
+- **No match → skip, don't fail.** A missing effect logs an anomaly and moves on; never a render blocker.
+- **Retrieval (credentialed) or bundled library (offline) — never generation.** You search HeyGen by text, or match a name against the 21-file manifest.
+- **One asset per distinct name.** Reuse across lines is deduped to a single download/copy, many cues.
+- **The switch is global, not per cue.** With a credential, retrieval handles even the long tail (effects not in the 21); without one, only the 21 bundled names resolve.

package/vendor/hyperframes/dist/skills/media-use/audio/references/transcribe.md

@@ -0,0 +1,52 @@
+# Transcription
+
+Create normalized word-level timestamps. **Always specify `--model` explicitly** — the CLI default is `small.en`, which silently translates non-English audio into English.
+
+```bash
+npx hyperframes transcribe audio.mp3  --model small.en             # known English
+npx hyperframes transcribe video.mp4  --model small --language es  # known Spanish
+npx hyperframes transcribe audio.mp3  --model small                # unknown language (auto-detect)
+npx hyperframes transcribe subtitles.srt                           # import existing
+npx hyperframes transcribe subtitles.vtt
+npx hyperframes transcribe openai-response.json
+```
+
+## Language Rule (Non-Negotiable)
+
+`.en` models (`tiny.en` / `base.en` / `small.en` / `medium.en`) **translate** non-English audio into English. This silently destroys the original language.
+
+1. **Known English** → `--model small.en` (or `medium.en` for music / noisy audio)
+2. **Known non-English** → `--model small --language <iso-code>` (no `.en` suffix)
+3. **Unknown language** → `--model small` (whisper auto-detects)
+
+**CLI default is `small.en`** — do not rely on it; always pass `--model` to make the choice explicit. `--language` also filters out non-target-language segments from mixed-language audio.
+
+## Model Sizes
+
+| Model      | Size   | Speed    | When                                  |
+| ---------- | ------ | -------- | ------------------------------------- |
+| `tiny`     | 75 MB  | Fastest  | Quick previews, smoke tests           |
+| `base`     | 142 MB | Fast     | Short clips, clear audio              |
+| `small`    | 466 MB | Moderate | Default for most multilingual content |
+| `medium`   | 1.5 GB | Slow     | Music with vocals, noisy audio        |
+| `large-v3` | 3.1 GB | Slowest  | Production quality                    |
+
+### Picking a model by content type
+
+1. Speech over silence / light background → `small.en`
+2. Speech over music, or music with vocals → start with `medium.en`
+3. Produced music track (vocals + full instrumentation) → start with `medium.en`; expect to need manual lyrics or an external API ([`captions/transcript-handling.md`](captions/transcript-handling.md) → "Using External Transcription APIs")
+4. Multilingual → `medium` or `large-v3` (no `.en` suffix), pair with `--language`
+
+## Output Shape
+
+Compositions consume a flat array of word objects. The `id` (`w0`, `w1`, …) is added during normalization for stable references in caption overrides; optional for backwards compatibility.
+
+```json
+[
+  { "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
+  { "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
+]
+```
+
+For mandatory caption-quality checks, retry rules, and the OpenAI/Groq Whisper API import path, see `captions/transcript-handling.md`.

package/vendor/hyperframes/dist/skills/media-use/audio/references/tts-to-captions.md

@@ -0,0 +1,35 @@
+# TTS → Captions
+
+When no recorded voiceover exists, generate one and obtain word-level caption timing. Two paths depending on which TTS provider is in use:
+
+## Path A — HeyGen (single call, no Whisper)
+
+HeyGen returns word timestamps in the same response as the audio. Use the
+bundled REST helper (the `hyperframes tts` command is Kokoro-only):
+
+```bash
+node skills/media-use/audio/scripts/heygen-tts.mjs \
+  script.txt --output narration.wav --words narration.words.json
+```
+
+`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.
+
+## Path B — Gemini / ElevenLabs / Kokoro (TTS → transcription)
+
+These adapters supply audio without word data. The shared audio engine runs
+transcription automatically when timings are absent. For Gemini, use the
+request in [Text to speech](tts.md#gemini-narration), then consume
+`audio_meta.json` → `voices[].words`.
+
+For a standalone local Kokoro generation, generate the audio, then transcribe:
+
+```bash
+npx hyperframes tts script.txt --voice af_heart --output narration.wav
+npx hyperframes transcribe narration.wav --model small.en   # voice af_heart is American English
+```
+
+Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
+
+For Gemini, verify that transcription preserved the script, especially names,
+numbers, and delivery pauses. If `words` is empty, resolve the transcription
+failure before captioning. Generate and align again after changing the read.

package/vendor/hyperframes/dist/skills/media-use/audio/references/tts.md

@@ -0,0 +1,260 @@
+# Text To Speech
+
+`npx hyperframes tts` synthesizes locally with Kokoro. It does not accept a
+`--provider` or `--words` flag. For HeyGen audio plus word timestamps, use the
+bundled `heygen-tts.mjs` script below.
+
+> **Run the Preflight first — no credential is not a green light to silently use the local voice.** Before generating a voiceover, complete the sign-in **Preflight** (see `../../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen voices, or continue offline with local Kokoro). This applies to a one-off "generate a voiceover" request just as much as inside a full workflow.
+
+## Narrating a HyperFrames docs video
+
+Anything that will sit on hyperframes.heygen.com uses one narrator, so the site
+does not sound like two products.
+
+|                |                                                |
+| -------------- | ---------------------------------------------- |
+| Voice          | **River** — "Relaxed, Neutral, Informative"    |
+| Provider       | ElevenLabs                                     |
+| `voice_id`     | `SAz9YHcvj6GT2YYXdXww`                         |
+| Model          | `eleven_multilingual_v2`                       |
+| Pace           | 145–155 words per minute, with room to breathe |
+| Music under it | about −31 LUFS, never masking the voice        |
+
+```bash
+curl -s -X POST "https://api.elevenlabs.io/v1/text-to-speech/SAz9YHcvj6GT2YYXdXww" \
+  -H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
+  -d '{"text":"...","model_id":"eleven_multilingual_v2"}' -o take.mp3
+```
+
+This is the voice every user-journey film on the docs site already uses. Falling
+back to local Kokoro because a key was not to hand produces a film that sounds
+wrong beside the others — three docs videos were built that way and had to be
+re-voiced. If you cannot reach ElevenLabs, say so and stop rather than
+substituting a different voice.
+
+Use another voice only for a documented reason, and write the reason down.
+
+## Available routes
+
+Gemini is an explicit alternative to the automatic provider order below. A
+request to use Gemini already chooses the provider; do not redirect that user
+to HeyGen sign-in. Read the Gemini section for its credential requirement.
+
+| Order | Provider          | Env trigger                                                          | Voice IDs                                                                                  | Word timestamps                           | Audio format         |
+| ----- | ----------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------- | -------------------- |
+| 1     | HeyGen (Starfish) | `$HEYGEN_ACCESS_TOKEN` / `$HEYGEN_API_KEY` / `~/.heygen/credentials` | UUIDs from `GET /v3/voices?engine=starfish`, or your cloned voice ids (`heygen-voice.mjs`) | **Yes** (`word_timestamps[]` in response) | mp3 → wav via ffmpeg |
+| 2     | ElevenLabs        | `$ELEVENLABS_API_KEY`                                                | UUIDs from elevenlabs.io dashboard                                                         | No                                        | mp3 → wav via ffmpeg |
+| 3     | Kokoro-82M        | always (local fallback)                                              | `am_michael`, `af_heart`, … (54 voices)                                                    | No                                        | wav direct           |
+
+```bash
+# Local Kokoro CLI
+npx hyperframes tts "Welcome to HyperFrames" -o narration.wav
+```
+
+## Self-contained HeyGen (no CLI) — `scripts/heygen-tts.mjs`
+
+The published `hyperframes tts` CLI synthesizes locally with Kokoro only. When you
+want HeyGen specifically — best quality **plus** word timestamps in one call — use
+the skill's bundled script, which calls the HeyGen v3 REST API directly and needs
+no CLI provider plumbing:
+
+The script resolves a HeyGen credential the same way the CLI does — first source
+wins: a host-injected OAuth `$HEYGEN_ACCESS_TOKEN` → `$HEYGEN_API_KEY` →
+`$HYPERFRAMES_API_KEY` → a project `.env` (auto-loaded,
+walks up ≤5 dirs) → `~/.heygen/credentials` (shared with heygen-cli;
+`$HEYGEN_CONFIG_DIR` overrides the dir). An OAuth login is sent as
+`Authorization: Bearer`; an API key as `X-Api-Key`; both include
+`X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance
+(10 min/month) before paid usage; API keys follow normal API billing. If the
+only credential is an expired OAuth token it stops with a hint to run
+`npx hyperframes auth refresh`.
+
+```bash
+# Only needed if you haven't run `npx hyperframes auth login`:
+export HEYGEN_API_KEY=...   # or put it in a project .env
+
+# Synthesize + capture word timestamps in one call (skips a Whisper pass)
+node skills/media-use/audio/scripts/heygen-tts.mjs \
+  "Welcome to HyperFrames." -o narration.wav --words narration.words.json
+
+node skills/media-use/audio/scripts/heygen-tts.mjs ./script.txt -o narration.wav
+node skills/media-use/audio/scripts/heygen-tts.mjs --list   # public starfish voices
+```
+
+- **Voice:** `--voice <id>` must be a **starfish** voice_id (`--list`, or `GET /v3/voices?engine=starfish`) or one of your cloned voice ids (below). v2-catalog ids are rejected with HTTP 400. Omit `--voice` (English) and it defaults to **Marcia** (`05f19352e8f74b0392a8f411eba40de1`, a fixed default so the choice is deterministic). Non-English with no `--voice` falls back to the first matching catalog voice.
+- **Output:** `.wav` → transcoded to 44.1k mono via ffmpeg; `.mp3` → raw bytes (no ffmpeg needed).
+- **Words:** `--words <path>` writes the flat `[{id,text,start,end}]` shape below, drop-in for the captions pipeline. HeyGen's `<start>`/`<end>` boundary sentinels are filtered out and ids are re-contiguous.
+- **Non-English:** `--lang <code>` (anything but `en`) is sent as the request `language`.
+
+## Cloned voices — `scripts/heygen-voice.mjs`
+
+Clone a voice from an mp3 or wav recording, then pass the printed id to
+`heygen-tts.mjs --voice`. Same credential resolution as above. Any API refusal
+(clone limit reached, plan upgrade required, voice not found) prints HeyGen's own
+message to stderr and exits 1.
+
+```bash
+node skills/media-use/audio/scripts/heygen-voice.mjs clone take.mp3 --name "My voice"  # → {"voice_id":"..."}
+node skills/media-use/audio/scripts/heygen-voice.mjs list --prefix "My"   # → [{"voice_id","name","created_at"}]
+node skills/media-use/audio/scripts/heygen-voice.mjs delete <voice_id>
+```
+
+`clone` waits up to 120 s for the clone to finish. `list` reads
+`GET /v3/voices?type=private`.
+
+## When to use which provider
+
+| Goal                                                      | Use                                                       |
+| --------------------------------------------------------- | --------------------------------------------------------- |
+| Best voice quality + word timestamps in one call          | **HeyGen**                                                |
+| Drop-in cloud TTS, big voice catalog                      | **ElevenLabs**                                            |
+| Offline, no API key, fast iteration                       | **Kokoro**                                                |
+| Directed delivery with Gemini prebuilt or custom voices   | **Gemini** (explicit selection; transcription for timing) |
+| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …)       |
+
+## Gemini narration
+
+Use the shared audio engine, not `hyperframes tts`. Authenticate with either:
+
+- `GEMINI_API_KEY` or `GOOGLE_API_KEY` (first set key wins).
+- A service-account JSON file at `GOOGLE_APPLICATION_CREDENTIALS`, or injected
+  JSON in `GCS_CREDS`. Install `google-auth requests` in the Python 3 environment
+  used by the helper. A configured file takes precedence over injected JSON.
+
+API keys take precedence over service accounts: unset both key variables to use
+OAuth. Never put credentials in a request file or composition. The helper
+obtains a fresh OAuth token for each generation with the
+`generative-language.retriever` scope. The quota project resolves from
+`GOOGLE_CLOUD_PROJECT`, then `GCLOUD_PROJECT_ID`, then the service-account JSON.
+User ADC files and metadata-server authentication are not supported.
+
+Both routes call the Gemini Developer Interactions API, not Cloud TTS or Vertex
+AI. Those APIs have separate model catalogs and access requirements.
+
+Save this as `audio_request.json` in the project:
+
+```json
+{
+  "provider": "gemini",
+  "tts_model": "gemini-3.8-flash-tts",
+  "voice": "Kore",
+  "lang": "en",
+  "style": "Warm, clear, conversational. Leave a short pause between sentences.",
+  "lines": [
+    { "id": "intro", "text": "Every word has a moment. Let the picture follow the voice." }
+  ],
+  "bgm": { "mode": "none" }
+}
+```
+
+```bash
+node <SKILL_DIR>/audio/scripts/audio.mjs \
+  --request ./audio_request.json --hyperframes . --out ./audio_meta.json --only tts
+```
+
+The engine saves `assets/voice/intro.wav` (or `intro-2.wav` when a file of yours already has that name; `voices[].path` says which), measures its duration, and transcribes
+it into `voices[].words` in `audio_meta.json`. Check that every requested line
+has audio and nonempty word timings before building a captioned video. Review
+the timings against the actual audio; transcription is estimated alignment,
+not native TTS timestamps. Do not distribute words evenly across a clip.
+
+- **Models:** `gemini-3.8-flash-tts` (default), `gemini-3.8-flash-lite-tts`,
+  `gemini-3.1-flash-tts-preview`, `gemini-2.5-pro-preview-tts`, and
+  `gemini-2.5-flash-preview-tts`. Use these exact Developer API IDs; Cloud TTS
+  aliases such as `gemini-2.5-flash-tts` are not accepted.
+- **Voice:** `Kore` by default; pass another prebuilt voice or an existing custom
+  voice ID for 3.8. Older models require prebuilt voices. Creating or replicating
+  voices is outside this helper.
+- **Delivery:** Put directions in `style`, not in spoken `text`. Each line can
+  override `style`. 3.8 uses structured annotations; older models receive a
+  delivery prompt before the transcript. Check that directions were not spoken.
+  Use style for pacing; numeric `speed` must be omitted or 1.
+- **Audio:** 3.8 returns a complete WAV. Older models return mono 16-bit PCM,
+  which the helper wraps as WAV at the returned sample rate without resampling.
+  No generation calls run during playback or rendering.
+- **Timing:** This adapter requests no native word timestamps. It uses the
+  existing transcription pass. `lang` selects transcription language; Gemini
+  infers speech language from the text.
+- **Workflow adapters:** Product-launch, faceless-explainer, and PR-video audio
+  scripts accept `--provider gemini --voice Kore --tts-model gemini-3.8-flash-tts
+--style "Warm and clear"`.
+
+An API error is reported as a failed line; an explicitly chosen Gemini voice
+never silently falls back to another provider. Check the engine's anomalies
+and output metadata, not only its exit code.
+
+API contract: [Google's speech generation guide](https://ai.google.dev/gemini-api/docs/speech-generation).
+
+## ffmpeg requirement
+
+HeyGen + ElevenLabs return mp3. The bundled HeyGen helper transcodes to wav
+when `--output` ends in `.wav` (the default and what downstream `ffprobe` +
+Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`.
+Without `ffmpeg` on PATH, wav output from cloud providers fails; the local
+Kokoro CLI writes wav directly.
+
+## Voice selection (Kokoro)
+
+Default `af_heart`. Curated picks:
+
+| Content type      | Voice                  |
+| ----------------- | ---------------------- |
+| Product demo      | `af_heart`, `af_nova`  |
+| Tutorial / how-to | `am_adam`, `bf_emma`   |
+| Marketing / promo | `af_sky`, `am_michael` |
+| Documentation     | `bf_emma`, `bm_george` |
+| Casual / social   | `af_heart`, `af_sky`   |
+
+Run `npx hyperframes tts --list` for the bundled set.
+
+## Multilingual (Kokoro voice prefix → language)
+
+The first letter of a Kokoro voice ID picks the phonemizer language; `--lang` overrides auto-detection.
+
+| Prefix | Language             |
+| ------ | -------------------- |
+| `a`    | American English     |
+| `b`    | British English      |
+| `e`    | Spanish              |
+| `f`    | French               |
+| `h`    | Hindi                |
+| `i`    | Italian              |
+| `j`    | Japanese             |
+| `p`    | Brazilian Portuguese |
+| `z`    | Mandarin             |
+
+```bash
+npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora
+npx hyperframes tts "Today is a nice day" --voice af_heart
+```
+
+Valid `--lang` codes (only needed to override the voice's auto-detected language): `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`.
+
+Non-English phonemization requires `espeak-ng` system-wide (`brew install espeak-ng` / `apt-get install espeak-ng`).
+
+## Speed
+
+- `0.7-0.8` — tutorial, complex content, accessibility
+- `1.0` — natural pace (default)
+- `1.1-1.2` — intros, transitions, upbeat content
+- `1.5+` — rarely appropriate, test carefully
+
+The `hyperframes tts` command honors `--speed` for Kokoro. Provider-specific
+helpers document their own pacing controls.
+
+## Long scripts
+
+Past a few paragraphs, write the text to a `.txt` file and pass the path. Inputs over ~5 minutes of speech may benefit from splitting into segments.
+
+## HeyGen word-timestamp shape
+
+When `--words <path>` is passed to a HeyGen call, the file is written in the same flat shape `transcribe` produces — drop-in compatible with the captions pipeline:
+
+```json
+[
+  { "id": "w0", "text": "Hi", "start": 0.0, "end": 0.21 },
+  { "id": "w1", "text": "there", "start": 0.22, "end": 0.55 }
+]
+```
+
+For ElevenLabs / Kokoro, run `npx hyperframes transcribe narration.wav --model small.en` to get the same shape.