videowright 0.1.0

package/skill/references/audio/sync.md

@@ -0,0 +1,237 @@
+# Sync Video to Audio Track
+
+## When this is loaded
+
+An audio track has been built and approved, and you need to compute per-segment timing that syncs the video to the audio. This reference extends the VO-level sync algorithm in [voiceover/sync_algorithm.md](voiceover/sync_algorithm.md) to handle multi-source audio tracks (VO + SFX + music). For VO-only tracks, the VO-only shortcut at the bottom of this file applies -- it delegates to the existing sync algorithm unchanged.
+
+## Overview
+
+The sync stage reads the approved track's `plan_snapshot.md` and the source VO `timing.json` files to compute a `Timing` object. This `Timing` is written into the track's `track.ts` and drives segment advance timing during playback and render.
+
+The procedure is an agent reasoning step, not a deterministic function. You read structured data, apply mapping rules, and produce timing values that align visual beats to audio events.
+
+## Inputs
+
+1. **Plan snapshot**: `audio/tracks/vN/plan_snapshot.md` from the active track. Contains all cues with their Source, Slice, Place at, and Notes fields.
+2. **Per-segment script**: from `PLAN.md` (the `## Script` section with subsections per segment id).
+3. **VO timing data**: `timing.json` from each VO source referenced by cues in the snapshot. Located at `audio/originals/voiceovers/<slug>/timing.json`.
+4. **Segment ids**: in timeline order, plus each segment's `advances` array length (tells you how many advances each segment needs).
+
+## Output
+
+An updated `timing` field in `audio/tracks/vN/track.ts`:
+
+```ts
+timing: {
+  perSegment: {
+    'intro':          [4.2],
+    'feature-cards':  [2.1, 5.8, 9.3, 12.0],
+    'outro':          [3.5],
+  },
+},
+```
+
+Each value array has the same length as the segment's `advances` array. Values are segment-relative seconds (time since the segment started playing, not since the audio started).
+
+## The sync procedure
+
+### Step 1: Parse VO cues from the snapshot
+
+Read the plan snapshot. For each cue whose `Source:` points at `audio/originals/voiceovers/...`, extract:
+
+- **Source path**: which VO folder
+- **Slice**: `full file` or `<start>-<end>` (seconds within the source)
+- **Place at**: where the cue starts in the track (seconds)
+- **Notes**: any inline text about what the VO says
+
+### Step 2: Load VO timing data
+
+For each unique VO source referenced by a cue, load `timing.json` from that source folder:
+
+```json
+{
+  "words": [
+    { "word": "Welcome", "start": 0.0, "end": 0.45 },
+    { "word": "to", "start": 0.47, "end": 0.55 },
+    ...
+  ]
+}
+```
+
+Timestamps in timing.json are seconds from the start of the VO audio file.
+
+### Step 3: Map VO word times to track times
+
+For each VO cue with `Slice: a-b` (or `full file`, where a=0 and b=file length) and `Place at: p`:
+
+> **Effective track time** of a word originally at source time `w`:
+> `track_time = p + (w - a)`, valid only when `a <= w <= b`.
+
+Words outside the slice range are excluded from this cue's mapping. If a word appears in multiple cues (overlapping slices from the same VO), it has multiple effective track times -- this is rare but valid.
+
+Example:
+- VO cue: `Slice: 1.2-5.6`, `Place at: 3.4`
+- Word at source time 2.0s: `track_time = 3.4 + (2.0 - 1.2) = 4.2s`
+- Word at source time 5.0s: `track_time = 3.4 + (5.0 - 1.2) = 7.2s`
+- Word at source time 0.5s: excluded (outside slice range)
+
+### Step 4: Map script text to timing words
+
+Walk through the per-segment script from PLAN.md. For each segment's script section, find the corresponding words in the timing data by text matching.
+
+- Match is case-insensitive and ignores punctuation.
+- Provider timing may include words from pause markers or SSML annotations that were in the provider script but not the PLAN script -- skip those.
+- If the provider timing has significantly different text (indicating the TTS changed wording), flag this to the user.
+
+### Step 5: Find segment boundaries
+
+For each segment, determine the advance times:
+
+- **Align to the next segment's VO onset, not the current segment's VO offset.** Each segment's final advance should land just *before* the next segment's first spoken word, so the voiceover starts right after the transition.
+- Find the first word of the *next* segment's script section on the track timeline. Place the segment boundary 0.1-0.3s before that word's effective track start time.
+- For the **last segment** (no next segment), find the last word in the segment's script section and add 0.3-0.5s after its end time.
+
+### Step 6: Handle multi-advance segments
+
+Segments with multiple advances have internal beats (`waitForNext()` calls). For these:
+
+1. **Count the advances.** The segment's `advances` array length tells you how many beats are needed.
+2. **Identify beat positions.** Look for natural break points in the segment's script:
+   - `[pause for animation]` markers in PLAN.md.
+   - Sentence boundaries that align with visual transitions.
+   - Content transition cues: "Next,...", "And now,...", "Finally,...".
+3. **Place internal advances** at the effective track time of the words at these break points, converted to segment-relative time.
+4. **Place the final advance** just before the next segment's first VO word (per Step 5).
+
+### Step 7: Incorporate SFX and music anchors
+
+Read `Notes:` fields from non-VO cues (SFX, music) in the plan snapshot. Look for time-specific hints:
+
+- "beat drops at 5.3s" -- a music moment that could align with a visual transition
+- "whoosh at 12.0s" -- a SFX that should coincide with a visual event
+- "typing starts at 8.2s" -- a SFX placement that the video should sync to
+
+These anchors are secondary to VO timing. Use them to fine-tune advance placement when a visual beat should land on an audio event. If an anchor conflicts with VO-derived timing, VO timing wins.
+
+### Step 8: Convert to segment-relative times
+
+All advance values in the `Timing` are segment-relative (seconds since the segment started), not track-absolute.
+
+```
+segment_start = sum of all previous segments' final advance times
+                (cumulative from the start of the track)
+advance_time = absolute_track_time - segment_start
+```
+
+**Worked example** (3 segments, 1 advance each):
+
+| Segment | Track-absolute advance | segment_start | Segment-relative advance |
+|---|---|---|---|
+| intro | 4.2s | 0.0s (first segment) | 4.2 - 0.0 = **4.2s** |
+| feature-cards | 16.2s | 4.2s (intro's final advance) | 16.2 - 4.2 = **12.0s** |
+| outro | 19.7s | 16.2s (feature-cards' final advance) | 19.7 - 16.2 = **3.5s** |
+
+For multi-advance segments, `segment_start` is still the cumulative total of all previous segments' **final** advance values. Internal advances within a segment are converted using the same `segment_start`.
+
+### Step 9: Apply the "audio always wins" rule
+
+The video duration adapts to match the audio:
+
+- If the audio for a segment is shorter than the segment's current advances suggest, compress the advances.
+- If the audio is longer, stretch the advances.
+- The last advance of the last segment should land at or near the end of the audio track.
+- Never truncate audio. Never pad with silence.
+
+## Writing the timing
+
+### Update track.ts
+
+Write the computed `timing` into the active track's `track.ts`:
+
+```ts
+timing: {
+  perSegment: {
+    'intro': [4.2],
+    'feature-cards': [2.1, 5.8, 9.3, 12.0],
+    'outro': [3.5],
+  },
+},
+```
+
+### Present to the user
+
+Show the timing with annotations:
+
+```
+Proposed timing for audio track v3:
+
+  intro (1 advance):
+    [0] 4.2s -- "...set us apart." (end of intro narration)
+
+  feature-cards (4 advances):
+    [0] 2.1s -- "...across devices." (end of collaboration section)
+    [1] 5.8s -- "...in one view." (end of analytics section)
+    [2] 9.3s -- "...and more." (end of integrations section)
+    [3] 12.0s -- transition (next VO starts at ~12.2s)
+
+  outro (1 advance):
+    [0] 3.5s -- "...Thanks for watching." (end of video)
+
+  Total track duration: 19.7s
+```
+
+For each advance, show:
+- The segment-relative time in seconds.
+- A snippet of the script text the advance lands on.
+- What the advance does (internal beat vs. segment transition).
+
+### Iteration
+
+The user may request adjustments:
+
+- "Move the second beat in feature-cards 0.5s later" -- adjust that advance value.
+- "The intro feels rushed" -- add buffer to the intro's advance.
+- "Combine the first two beats" -- this changes the number of advances, which means a `waitForNext()` needs to be removed from segment code. Flag the code change.
+
+After each adjustment, re-present the timing. When the user confirms, write it into `track.ts`.
+
+### Append sync log entry
+
+After the timing is finalized, append to `audio/audio_plan.md`:
+
+```markdown
+## YYYY-MM-DD HH:MM -- Synced timing to vN
+**Segments:** intro, feature-cards, outro
+**Total duration:** 19.7s
+```
+
+### Animation sync pass
+
+After writing timing, consider whether in-segment animations need adjustment. Load [voiceover/animation_sync.md](voiceover/animation_sync.md) if the audio track is being set as the default and the video has fully automated animations (not all `waitForNext`-driven).
+
+## VO-only shortcut
+
+For VO-only tracks (single VO cue, full file, placed at 0s), the sync procedure is identical to the existing VO sync algorithm in [voiceover/sync_algorithm.md](voiceover/sync_algorithm.md). The mapping simplifies:
+
+- `a = 0`, `p = 0`, so `track_time = w` (source time equals track time).
+- No SFX/music anchors to incorporate.
+- The full existing sync algorithm applies unchanged.
+
+## Edge cases
+
+| Situation | Behavior |
+|---|---|
+| Plan snapshot has no VO cues | No word-level timing data. Use SFX/music anchors if available. Otherwise, fall back to the segment's existing `advances` values. |
+| VO timing.json is missing | Error. Direct the user to generate timing data (via TTS with timestamps or STT transcription). |
+| VO slice out of range | Error before computing. "VO v3 is 4.0s; slice 1.2-5.6 exceeds source." |
+| Multiple VO cues from the same source with overlapping slices | Valid. A word may have multiple effective track times. Use the most relevant one for the segment being synced. |
+| Segment has no script (silent segment) | Use the segment's existing `advances` values. The segment passes through without VO-derived timing. |
+| SFX anchor contradicts VO timing | VO timing wins. Note the conflict in the timing presentation. |
+| Track has no timing.json references (SFX/music only, no VO) | Use SFX/music anchors plus segment `advances` as the base. This is unusual but valid. |
+
+## Relationship to other files
+
+- **[voiceover/sync_algorithm.md](voiceover/sync_algorithm.md)** -- the VO-level sync algorithm. Still the reference for parsing timing.json and matching words to script. This file extends it to audio-track-level sync.
+- **[voiceover/animation_sync.md](voiceover/animation_sync.md)** -- the in-segment animation adjustment pass. Run after sync when the track is set as default.
+- **[build.md](build.md)** -- the build workflow that precedes sync.
+- **[audio_plan.md](audio_plan.md)** -- the plan format specification.

package/skill/references/audio/voiceover/animation_sync.md

@@ -0,0 +1,142 @@
+# Animation Sync
+
+## When this is loaded
+
+A voiceover is being set as the default for a video, and you need to perform a one-time sync pass to adjust in-segment animations to align with audio beats.
+
+## What this is
+
+When an audio track is set as the default (`default_audio_track` in `timeline.ts`), the agent performs a one-time manual pass over segment code to align fully automated animations with the audio track's timing. This is a planning-time agent action, not a runtime mechanism.
+
+## What to sync
+
+Look for **fully automated animations** -- animations that run on a fixed clock inside a segment, not gated on user interaction (`waitForNext()`). Examples:
+
+### Animations to consider
+
+- **`ctx.hold(ms)` calls** -- if a hold duration no longer matches the voiceover timing, adjust it.
+- **CSS transitions with explicit durations** -- `transition: transform 2s ease` where the duration should match an audio beat.
+- **CSS animation durations** -- `animation: fadeIn 1.5s forwards`.
+- **GSAP timelines** -- `.to(el, { duration: 2, ... })` where the duration should align with narration.
+- **Lottie playback speed** -- if a Lottie animation's duration needs to match a segment beat.
+- **Element appearance delays** -- WAAPI `delay` values or CSS animation delays that stage element reveals.
+
+### Animations to leave alone
+
+- **Transition animations** between segments (handled by the transition system, not segment code).
+- **Infinite/looping animations** that run as backgrounds (e.g., a subtle pulsing gradient).
+- **User-gated animations** that only trigger on `waitForNext()` -- these are already controlled by the `Timing`.
+
+## The sync procedure
+
+### Step 1: Read the voiceover's Timing
+
+Get the `perSegment` timing from the voiceover object. Each segment's advance times tell you when visual beats should land.
+
+### Step 2: Walk each segment
+
+For each segment in the timeline:
+
+1. Read the segment's code (`segments/<id>/index.ts`).
+2. Identify fully automated animations (see "What to sync" above).
+3. Compare the animation's current duration/timing with the voiceover's advance schedule for that segment.
+
+### Step 3: Adjust durations
+
+Where an automated animation's duration should align with an audio beat, adjust it:
+
+**Example: hold duration adjustment**
+
+Before:
+```ts
+async play(ctx) {
+  // Title animation
+  el.classList.add('visible');
+  await ctx.hold(3000);  // original: 3 seconds
+}
+// advances: [3.0]
+```
+
+After (voiceover timing says the intro narration ends at 4.2s):
+```ts
+async play(ctx) {
+  // Title animation
+  el.classList.add('visible');
+  await ctx.hold(4200);  // adjusted to match voiceover timing
+}
+// advances are now driven by the Timing object, not this array
+```
+
+**Example: CSS animation adjustment**
+
+Before:
+```ts
+mount(el) {
+  el.innerHTML = `
+    <style>
+      .title { animation: slideIn 1.5s ease-out forwards; }
+    </style>
+    <h1 class="title">Hello</h1>
+  `;
+}
+```
+
+After (narration starts 0.5s in, so title should be visible by then):
+```ts
+mount(el) {
+  el.innerHTML = `
+    <style>
+      .title { animation: slideIn 0.4s ease-out forwards; }
+    </style>
+    <h1 class="title">Hello</h1>
+  `;
+}
+```
+
+### Step 4: Add timing comments
+
+When adjusting a duration to match voiceover timing, add a brief comment noting why:
+
+```ts
+await ctx.hold(4200);  // synced to voiceover: intro narration ends at 4.2s
+```
+
+This helps future editors understand the magic number.
+
+### Step 5: Render-safety review
+
+After adjusting each segment, run the render-safety CR checklist from [create_or_edit_video.md](../../create_or_edit_video.md) (Step 2b) against the modified code. The sync pass may introduce or reveal non-idiomatic patterns -- for example, animations that depend on external network fetches or non-deterministic input. Fix any issues before moving to the next segment.
+
+## Tradeoffs
+
+- **Sync is to the default audio track only.** If the user later switches to a different audio track via `--audio-track <other-id>`, advance timing updates automatically (driven by the new track's `Timing`), but in-segment animation durations remain tuned to the original default.
+- **No runtime re-sync.** The sync is a one-time manual edit. There is no mechanism to dynamically adjust animation durations at playback time based on the active audio track.
+- **Re-sync is available.** If the user changes the default audio track, they can ask the agent to re-run the animation sync pass. The agent reads the new `Timing` and adjusts durations again.
+
+## When to skip animation sync
+
+- The video has no fully automated animations (all timing is `waitForNext`-driven).
+- The user explicitly says they do not want animation adjustments.
+- The audio track is not being set as the default (it is an alternate take that will be used via `--audio-track <id>` only).
+
+In these cases, set `default_audio_track` in `timeline.ts` without modifying segment code.
+
+## Presenting changes to the user
+
+After the sync pass, summarize the changes:
+
+```
+Animation sync for default voiceover "v1":
+
+  intro:
+    - ctx.hold(3000) -> ctx.hold(4200) -- match intro narration end
+    - CSS .title animation: 1.5s -> 0.4s -- title visible before narration starts
+
+  feature-cards:
+    - No automated animations found (all waitForNext-driven)
+
+  outro:
+    - ctx.hold(2000) -> ctx.hold(3500) -- match outro narration end
+```
+
+Let the user review before committing the changes.

package/skill/references/audio/voiceover/provider_script.md

@@ -0,0 +1,153 @@
+# Provider Script
+
+## When this is loaded
+
+You have a confirmed script in PLAN.md and need to transform it into a provider-specific format for TTS generation.
+
+## What `provider_script.md` is
+
+The `provider_script.md` file is a transformation of the PLAN.md script with provider-specific annotations. It is the text that the user copies into the TTS portal (or the agent sends via API) to generate audio. It lives at:
+
+```
+videos/<video>/audio/originals/voiceovers/<slug>/provider_script.md
+```
+
+The file is markdown for human readability, but its content is the exact text to paste into the provider. Segment headings are stripped -- only the narration text and annotations are included.
+
+## Target: ElevenLabs v2
+
+All provider scripts target **ElevenLabs v2** (`eleven_multilingual_v2`). This model is chosen because it supports exact pause timing via SSML `<break>` tags, which is critical for syncing audio to video animations.
+
+**Do NOT use v3-style tags.** ElevenLabs v3 introduced inline emotion tags like `[excited]`, `[calm]`, `[whispering]`, etc. These tags are **silently ignored** by the v2 model. Never include them in a provider script.
+
+## Generating the provider script
+
+### Step 1: Read the confirmed script from PLAN.md
+
+Extract the script sections in timeline order.
+
+### Step 2: Apply delivery style through writing
+
+Since v2 does not have v3's emotion tag system, tone and emotion are conveyed through **how the text is written** -- punctuation, sentence structure, and word choice. The v2 model's prosody engine responds to natural language cues:
+
+#### The v2 writing toolkit
+
+| Technique | Effect on v2 delivery | Example |
+|---|---|---|
+| **Exclamation marks** | Increased energy, slight pitch rise | "This changes everything!" |
+| **Question marks** | Rising intonation | "Ready to get started?" |
+| **Ellipsis** (`...`) | Natural micro-pause (~0.3-0.5s), trailing-off feel | "And then... something unexpected." |
+| **Em-dash** (`--`) | Brief breath pause (~0.2-0.4s), interruption feel | "The result -- stunning." |
+| **ALL CAPS** | Slight emphasis on the word (use sparingly) | "This is EXACTLY what we needed." |
+| **Short punchy sentences** | Energetic, urgent pacing | "It's fast. It's reliable. It just works." |
+| **Long flowing sentences** | Calm, measured delivery | "Over the course of the next few minutes, we'll walk through each of the core features that make this possible." |
+| **Commas and semicolons** | Micro-pacing within a sentence | "First, the dashboard; then, the analytics." |
+| **Repeated punctuation** | Heightened emotion (use very sparingly) | "This is incredible!!" |
+| **Parenthetical asides** | Softer, conspiratorial tone | "The best part (and this surprised us too) is the speed." |
+
+#### What v2 cannot do
+
+- **No explicit emotion control.** You cannot tag a section as `[excited]` or `[whispering]`. The closest approximation is through writing style (see toolkit above).
+- **No voice speed control via script text.** Speaking rate is controlled by the speed slider in the portal or the `speed` parameter in the API, not by script content.
+- **No pitch control.** v2 does not support pitch-shifting tags.
+
+This is a deliberate trade-off: v2 gives us **deterministic pause timing** via `<break>` tags at the cost of explicit emotion tags. For voiceover-to-video sync, predictable timing is more valuable than fine-grained emotion control.
+
+### Step 3: Add pauses
+
+Where the PLAN.md script has `[pause for animation]` markers, insert pause mechanisms appropriate to the desired duration:
+
+#### Pause reference (v2)
+
+| Pause type | Syntax | Duration | Use for |
+|---|---|---|---|
+| Natural micro-pause | `...` (ellipsis) | ~0.3-0.5s | Trailing off, rhetorical beat |
+| Breath pause | `--` (em-dash) | ~0.2-0.4s | Parenthetical, interruption |
+| Sentence break | Period + new sentence | ~0.5-0.8s | Normal sentence transition |
+| Medium pause | `<break time="1.0s" />` | Exact (1.0s) | Animation beat between ideas |
+| Long pause | `<break time="2.5s" />` | Exact (2.5s) | Major visual transition |
+| Very long pause | `<break time="4.0s" />` | Exact (up to ~5s) | Extended animation sequence |
+
+**The `<break>` SSML tag is the primary mechanism for precise pauses.** It is supported by v2 and produces exact, deterministic timing. Use it for any pause of 1 second or more where the video needs time for a visual transition.
+
+Syntax: `<break time="X.Xs" />` where `X.X` is seconds (e.g., `"1.5s"`, `"0.8s"`, `"3.0s"`).
+
+For pauses under 0.5 seconds, ellipses and em-dashes often sound more natural than a `<break>` tag. For pauses over 0.5 seconds, use `<break>`.
+
+**Important: `<break>` cannot be the first element in the script.** ElevenLabs does not support a `<break>` tag at the very start of the text -- the TTS engine requires spoken text before the first break. If the video needs an initial silent pause before narration begins, start with a spoken word and place the first `<break>` after it, or handle the initial silence through video timing (e.g., a leading silent segment).
+
+### Step 4: Handle segment boundaries
+
+The provider script is one continuous block of text (not divided by segment). Segment boundaries from PLAN.md become natural pause points in the provider script. Insert a `<break>` tag at each segment transition to give the audio natural breathing room:
+
+```
+...set us apart.
+
+<break time="1.0s" />
+
+First up: real-time collaboration.
+```
+
+The duration of segment-boundary breaks depends on the video's pacing. A good default is 1.0-1.5 seconds.
+
+### Step 5: Pronunciation and special terms
+
+v2 handles pronunciation through these mechanisms:
+
+- **Spell out letters:** "A P I" (with spaces) for letter-by-letter pronunciation.
+- **Phonetic hint:** For unusual names, spell them phonetically nearby: "Istio (is-tee-oh)" or just "is-tee-oh" if the correct name is not needed in audio.
+- **URLs:** Write as speech: "acme dot com" instead of "acme.com".
+- **Numbers:** "one hundred twenty three" instead of "123" if the TTS reads it oddly. Test first -- ElevenLabs v2 generally handles numbers well.
+
+v2 does NOT support IPA phoneme tags or SSML `<phoneme>` elements. Use inline phonetic spelling as the workaround.
+
+## Output format
+
+Write the provider script as a single markdown file:
+
+```markdown
+# Provider Script
+
+> Provider: ElevenLabs v2 (eleven_multilingual_v2)
+> Voice: [voice name from selection, e.g. "Asher"]
+> Style notes: Conversational, warm
+
+---
+
+Welcome to Acme Product. Today we'll walk through the three features that set us apart.
+
+<break time="1.2s" />
+
+First up: real-time collaboration! Your team can edit simultaneously, with changes syncing instantly across devices.
+
+<break time="2.0s" />
+
+Next, the analytics dashboard. Track engagement, conversion, and retention -- in one view.
+
+<break time="2.0s" />
+
+Finally, integrations. Connect with the tools you already use... Slack, GitHub, Jira, and more.
+
+<break time="1.0s" />
+
+Ready to get started? Visit acme dot com for a free trial. Thanks for watching.
+```
+
+### Key conventions in the output
+
+- The header block (blockquote) is metadata for the user, not pasted into the portal or sent to the API.
+- Everything below the `---` is the text to paste or send.
+- `<break>` tags are inline where exact pauses should occur.
+- Punctuation-driven pauses (ellipses, em-dashes) are inline for natural micro-pauses.
+- URLs are written phonetically ("acme dot com") to avoid TTS mispronunciation.
+- **No v3 emotion tags** (`[excited]`, `[calm]`, `[whispering]`, etc.) -- these are silently ignored by v2.
+
+## Presenting to the user
+
+After generating the provider script:
+
+1. Show the full script with a brief explanation of the annotations used.
+2. Tell the user:
+   > Copy everything below the horizontal rule into the ElevenLabs portal, or the agent will send it via the API. See the provider reference for step-by-step instructions.
+3. Write the file to `audio/originals/voiceovers/<slug>/provider_script.md`.
+4. Proceed to the provider walkthrough: [providers/elevenlabs.md](providers/elevenlabs.md).