@amg555/frameforge-mcp 2.4.10 → 2.7.1

package/plugins/video-pipeline/skills/html-renderer/references/directing.md

@@ -0,0 +1,165 @@
+# Directing a Video Film — Professional Method
+
+Adapted from `lemomo-ai/lemo-opuscar` DIRECTOR.md.
+Applies to any FrameForge render: HTML compositor, Remotion, or FFmpeg pipeline.
+
+A film is judged in this order: **sound, rhythm, camera, directing** (performance, staging, emotional arc).
+Good-looking frames are only the starting point.
+
+---
+
+## 1. Before You Write Anything — Find a Benchmark
+
+Before writing treatment or storyboard, pick one or two reference works (films, title sequences, ads) that set the bar for this style **and this topic**. They are your creative choice, not the demo's. Write down:
+
+- **What to learn**: composition, pacing, camera grammar, colour logic, score structure
+- **What NOT to copy**: characters, designs, melodies, specific shots, logos, fonts
+
+The benchmark lifts quality more than any rule below.
+
+---
+
+## 2. Shape the Story (3-act minimum)
+
+- **One subject, one goal, one turn.** A character (or object, product, idea) wants something, something changes, it ends differently from how it began.
+- **Hook in the first 3 seconds.** No slow logo reveal before anything happens.
+- **An ending that echoes.** Bookend the opening, reveal the scale, or let the character do the thing right the second time.
+- **One native move**: a moment only this medium can do (a fold that becomes a transition, ink that blooms into the next scene, a type grid that snaps into the logo). Put it at the emotional peak.
+- **Cut the extras.** One gag the audience reads at thumbnail size beats three they can't.
+
+---
+
+## 3. TREATMENT.md Before Any Code
+
+Write `TREATMENT.md` before drawing or coding anything:
+
+```
+1. Three candidate structures (e.g.: single journey, before/after, countdown, list-that-turns)
+   → choose one + reason why
+2. Logline + arc: setup → turn → ending (one sentence)
+3. Benchmark: what to learn / what not to copy
+4. Shot list: for every shot → framing (wide/full/medium/close/insert), angle,
+   camera move (push/pull/pan/track/crane/zoom), duration, WHY
+5. Beat sheet: second by second
+6. Cue map: tempo, bar grid, instrumentation per section,
+   on which beat every cut / key action / subtitle lands
+7. Sound design table: for each section → ambience bed, main foley, music state
+8. Subtitle and title design: the type is part of the style
+```
+
+---
+
+## 4. Sound Is Half the Film
+
+- **Every visible action has a sound**: landings, page turns, cuts, drips, clicks, cloth. Match the material (paper, wood, metal, ink and glass all sound different).
+- **Three audio layers**:
+  1. **Ambience** (room, wind, rain, city)
+  2. **Foley** (tied to specific actions)
+  3. **Music** (bed — duck under dialogue and key foley)
+- **At least two real silences** (or near-silences) before the turn or the emotional peak. The first sound after silence should be one of the most important sounds in the film.
+- **J-cuts and L-cuts**: use sound as a transition at least twice:
+  - **J-cut**: bring the next scene's sound in before the picture cuts
+  - **L-cut**: let the last scene's sound run over into the next
+- **Avoid generic "piano and strings"** — score in the instruments of the style's sound palette.
+- **Earshot professional levels**: music bed sits around −18 to −14 LUFS for dialogue scenes, individual effects 3–6 dB over the bed. Target −14 LUFS final mix (social/web delivery).
+
+---
+
+## 5. Rhythm: Music Grid Comes First
+
+- Write the **cue map** before animating. The picture locks to the grid.
+- **Vary the pace**: alternate fast and slow; include one clear acceleration or deceleration; give the audience one breath (a long take or held pause).
+- A film at one even speed has failed — contrast is what creates rhythm.
+- One action, one sound, one cut — but don't cut on every beat. Leave time to see.
+- **Pace for the viewer, not the clock**:
+  - Hold every subtitle at least 1.8 s (never shorter than spoken line + 0.6 s)
+  - After text finishes animating in, hold for roughly `(letters ÷ 15 + 1.5) s` in English
+  - Hold a title card at least 4 s
+  - Give failures and gags enough time to be understood
+  - Fast is fine: make it fast with **fewer words per screen**, not by cutting before people finish reading
+
+---
+
+## 6. Camera: Every Shot Needs a Reason
+
+- Use at least **four different camera moves** and real changes of framing. 2D has a camera too: push, pull, parallax, focus pulls, frames within frames.
+- **One signature shot** people remember: a oner, a scale reveal, a transition made from the medium itself.
+- **Design transitions inside the medium** (a page turn, a carved line growing, ink spreading) as one consistent grammar. Avoid default fades and hard cuts.
+- At key moments the subject fills **at least a third of the frame height**, with clear staging and a readable silhouette.
+
+---
+
+## 7. Performance (Character / Animated Elements)
+
+- **Anticipation → action → follow-through** on every meaningful move.
+- Characters change emotion. They do not just slide across the screen.
+- **Motion continuity**:
+  - Blend every pose with keyframes and easing — never switch poses inside an `if`
+  - The only exception is a deliberate one-frame comic pop
+  - Check shoulders and arms on a large action test sheet before animating
+  - Held props sit **between the palms**, at a depth between the two arms
+  - A rolling object turns by `distance ÷ radius`
+- Step through key actions at 0.2 s intervals, enlarged, before finalizing.
+- Check every expression at final size. "Worried" and "angry" are one eyebrow flip apart.
+
+---
+
+## 8. The Review Loop (Run Before Delivery)
+
+1. **Contact sheet**: every shot's first, middle, and last frames — one sheet
+2. **Strip**: every frame of a key motion (transition, take, jump, throw) — 0.1–0.15 s step
+3. **Crop**: full-resolution detail check on faces, hands, contacts, glows
+4. **One full-speed viewing with sound** — watch it as the viewer will
+
+What to check at each step:
+- **Read**: is the event of each shot clear from its sheet alone?
+- **Timing**: does every read have enough time to land before the next?
+- **Motion**: does every move have anticipation and follow-through? Any pops or snaps?
+- **Boil**: in a strip, does stable linework stay stable between frames?
+- **Contacts**: do feet touch the ground? Do held objects touch the hands?
+- **Transitions**: does every seam open and close with one?
+- **Rules**: any text, any 3D, any dead stretch?
+- **Sound**: every row of the sound design table actually in the mix?
+
+---
+
+## 9. Failures Seen Most Often
+
+| Failure | Fix |
+|---------|-----|
+| Subject too small, too far, or pushed to frame edge | At key moments, fill ≥1/3 of frame height |
+| Too dark to read, or colour on same colour | Check contrast at delivery size |
+| Subtitles covering the subject | Position captions in clear negative space |
+| A gag too fast to understand | Give it more frames — fast ≠ short |
+| Blank frames in transitions | Verify every seam with a strip |
+| Timed effect that doesn't track subject after camera moves | One world→screen projection function throughout |
+| Everything at one even speed | Vary shot length by ≥4× across the film |
+| Beats that replace each other instead of carrying | Something on screen must survive every section boundary |
+| Full-bleed footage for the whole video | The genre is 70% white/negative space; full-bleed is the exception |
+
+---
+
+## 10. Delivery Checklist
+
+```
+□ final.mp4 (−14 LUFS, correct fps, no black frames, no NaN frames)
+□ poster.jpg (first clear frame)
+□ final.srt (when narrated — in sync, not covering key action)
+□ TREATMENT.md
+□ CREDITS (every third-party asset: music, fonts, images, voice engine + source + licence)
+□ build.sh (one-command reproducible render)
+□ Source code / project folder
+```
+
+**Tell the user** (in their language):
+- Where the project folder and the film are
+- That CREDITS lists third-party assets and their licences
+- That they are responsible for the materials they supply
+
+---
+
+## 11. Copyright Red Lines
+
+- References teach grammar only. Never copy characters, designs, melodies, shots, logos, or type designs.
+- No existing works, games, brands, or events appear in the film unless the user owns them or has the right.
+- Draw fictional stand-ins when rights are unclear.

package/plugins/video-pipeline/skills/html-renderer/references/earshot-audio.md

@@ -0,0 +1,201 @@
+# Earshot — Professional Audio Quality Workflow
+
+Adapted from `Leonxlnx/earshot` SKILL.md.
+AI-native audio analysis, verification, and synthesis. 
+**Core principle: Never say how audio sounds without measurement data behind it.**
+
+---
+
+## Why Earshot Matters for Agents
+
+Agents cannot hear. Without measurement, every audio decision is a guess. Earshot provides:
+- **Semantic matching**: describe a sound and find the closest asset
+- **Realness detection**: is this audio synthesized code or a real recording?
+- **Professional critique**: does the mix meet broadcast/launch film standards?
+- **Synthesis optimization**: make a sound in code, measure it, improve it
+
+---
+
+## The 5 Loops
+
+### Loop 1: Choose an Asset
+
+```
+1. RANK: earshot rank "<description>" downloads/sfx/ --top 5
+   → shortlist from a folder by semantic match
+
+2. FIND: earshot find winner.ogg "<description>" --top 3
+   → best moments as sample trims (start/end in seconds)
+   → Each segment kind: stroke (one onset → decay), event (strokes that belong together),
+     window (stretch for beds/textures)
+   → "head cut" / "tail cut" = trim starts/ends inside a sound — don't use as-is
+
+3. DESCRIBE: earshot describe winner.ogg --start 1.12 --end 1.48
+   → what it actually sounds like; check attack/decay shape fits the picture
+
+4. REALNESS: earshot realness winner.ogg --start 1.12 --end 1.48
+   → p_real: is this recorded audio or code-synthesized?
+   → Thresholds for clips (≤6 s): lo 0.17 / hi 0.77
+   → "likely code-synthesized" → don't use if brief asks for real recorded sound
+   → Clean pro-synthesized tones (UI sounds, chimes, synth stabs) also read as code-synthesized
+
+5. COMPARE: earshot ab a.ogg b.ogg --query "<description>"
+   → which is closer, and how B differs from A in measured words
+```
+
+### Loop 2: Review a Mix or Film
+
+```
+1. TIMELINE: earshot timeline film.mp4 --cues cues.json
+   → listening transcript: events with time, level, and tags
+   → sections, loudness curve, picture cuts, sync per cut/cue
+   → sync offset_ms = sound minus picture (negative = early)
+   → A cut listed with no hit can still carry a soft whoosh — check with sheet before adding
+
+2. SHEET: earshot sheet film.mp4 -o sheet.png --cues cues.json
+   → annotated PNG: frames, sections, labelled events, spectrogram, loudness, beats, cue markers
+   → Green = within a frame, Amber = within 100 ms, Red = no sound
+   → OPEN THE PNG — gaps, pile-ups, masking, and late hits are obvious at a glance
+
+3. CRITIQUE: earshot critique film.mp4
+   → scorecard against measured range of professional launch films
+   → verdict: launch-grade / close / needs work / not launch-grade
+   → Fix critical rows first, re-run 1-3 on the new render
+
+4. MEASURE: earshot measure mix.wav
+   → LUFS, true peak dBTP, LRA, noise floor, attack/decay, brightness, bands
+```
+
+### Loop 3: A/B a Change
+
+```
+earshot ab before.wav after.wav --query "<description>"
+```
+- Which is closer, and how B differs from A in measured words (louder, brighter, longer decay)
+- Change one thing at a time
+
+### Loop 4: Make a Sound in Code (Earshot Forge)
+
+When the brief calls for a **made** sound (not chosen from a library):
+
+1. Write a recipe `render(params, r) -> 48 kHz array` using DSP primitives:
+   - Modal resonators, contact excitations, noise colours
+   - Time-varying SVF filter, computed room, mic with noise floor
+   - Model the physics of the real thing (what strikes what, what rings, how long, in what room)
+2. Run a search round with budget and reps
+3. **Evaluate by measurement** (not by model score):
+   - Tonal balance: which bands fall outside every reference
+   - `robust`: mean over new seeds, not the lucky best render
+   - Spectrograms vs reference: `ffmpeg -i x.wav -lavfi showspectrumpic=s=900x360:scale=log x.png`
+4. Fix recipe structure for what's wrong; repeat
+
+**Goodhart warning**: The search optimises earshot's models. Made sounds routinely out-score real recordings on them, which means the critic is satisfied, not that the sound is better. Judge by measured closeness to references.
+
+### Loop 5: Make a Film's Sound Kit
+
+Use when a code-made film needs its effects rendered from earshot's shipped sound bank:
+
+```bash
+# Initialize a kit from a preset
+earshot kit init --preset calm -o audio/kit.json
+# calm = text-led, crisp = energetic product films, minimal = effects-only, launch = music-led
+
+# Check candidates
+earshot sfx list --kind hit
+
+# Render the kit
+earshot kit render audio/kit.json --out audio/kit
+
+# Verify
+earshot kit check audio/kit.json --out audio/kit
+```
+
+**0.4 families for music-led launch films**:
+- `ui_pop` — rounded mid pops for chips, bubbles, rows arriving (most-used effect)
+- `knock` — mid-band wood and plastic
+- `cut_hit` — short sub hits with a kick
+- `mallet` — gated glockenspiel/celesta/marimba notes, soft pluck, chime runs (supports `--pitch`)
+- `drip` — water drops (pitch rises)
+- `shutter` — snap zooms, whip-cuts
+- `reverse_swell` — peaks just after a cut (one per film)
+- Plus: `whoosh_big`, `thump`, `riser`, `swish`, `glitch`, `notify`
+
+**Levels against the music, not fixed**:
+```python
+from earshot.forge import kit
+lock = kit.render('audio/kit.json', 'audio/kit')  # takes carry l50_lufs; roles bed_db
+role = lock['roles']['pop']
+take = role['takes'][i]
+bed = kit.bed_level(music, t_cue)  # the music's median 50ms level
+gain = kit.gain_db(take, role, bed)  # dB on the take file (never over -1 dBFS peak)
+start = kit.place(take, role, t_cue)  # onset on the frame for hits/touches, peak for whoosh
+```
+
+Duck the music **6–12 dB for 0.3–0.5 s** before a hit, drop, big whoosh, or swell; bring it back on the hit.
+
+---
+
+## Professional Audio Numbers
+
+| Number | What it is | Trust |
+|--------|-----------|-------|
+| LUFS, dBTP, LRA, dBFS, ms, Hz | Measurements (BS.1770 meter, spectrum, envelope) | Exact — use for levels |
+| z (match/rank/find) | How far a description stands out among vocabulary | z ≥ 3 strong, 2.5–3 plausible, 2–2.5 weak, < 2 no match |
+| p_real | Real-recorded vs code-synthesized | Strong evidence, not proof |
+| cosine | Raw CLAP similarity | Never a probability; compare only same sound + same query |
+| AudioSet labels (0..1) | Independent per-class probabilities | Good for broad classes; hears only up to 8 kHz |
+| sync offset_ms | Sound minus picture per cut | Trust when well above chance |
+| critique status | Value vs range of 7 professional launch films | A range, not a target |
+
+---
+
+## Writing Audio Queries
+
+Describe the sound, not its use:
+- ✓ "a heavy wooden door slamming in a small room"
+- ✗ "a sound for the logo"
+
+Give alternatives that are realistic confusions:
+- ✓ `--alts "a click vs a typewriter key vs a thock"`
+
+Job words are not heard. Check with 2–3 phrasings and keep the best.
+
+---
+
+## Professional Levels Quick Reference
+
+| Platform | Target LUFS | True Peak |
+|----------|-------------|-----------|
+| YouTube | −14 LUFS | −1 dBTP |
+| Podcast | −16 LUFS | −1 dBTP |
+| Netflix | −27 LUFS | −2 dBTP |
+| Instagram/TikTok | −14 LUFS | −1 dBTP |
+| Broadcast | −23 LUFS | −1 dBTP |
+| Music streaming | −14 LUFS | −1 dBTP |
+
+Music under a voice: ~0.14 relative volume (≈ −17 dB).  
+Each SFX hit sits 3–6 dB over the music bed.
+
+---
+
+## FrameForge Integration
+
+| Earshot concept | FrameForge tool | Parameter |
+|----------------|-----------------|-----------|
+| `earshot measure` | `mix_audio_pro` | Returns LUFS, dBTP in result |
+| Sound-on-beat timing | `cue('sfxname', t, dur)` in edit.js | `cue()` in engine.js |
+| Platform LUFS target | `mix_audio_pro` | `platform: "youtube" \| "podcast" \| "netflix"` |
+| Room acoustics | `mix_audio_pro` | `roomProfile: "studio_dry" \| "broadcast" \| "cinematic"` |
+| SFX synthesis types | `mix_audio_pro` | `sfxType: "chime" \| "buzz" \| "tick" \| "swish" \| ...` |
+| Noise gate | `mix_audio_pro` | `noiseGate: true` |
+
+---
+
+## Earshot Limits (Cite These When They Matter)
+
+- **Taste is not measured**: earshot says what a sound is and how it measures, not whether it is beautiful or right for the story. The user decides on taste; give them the evidence and the shortlist.
+- **No speech recognition**: it can tell there is speech, not what is said.
+- **CLAP hears 50 Hz – 14 kHz**, AST up to 8 kHz: check "air" and low-pass problems with `measure (bandwidth_hz, bands)`.
+- **Very short sounds** are padded to a 10 s window; isolated hits read best with `find`/`describe` on a trimmed segment.
+- **Picture cuts** from frame-difference detector: hard cuts and whips are reliable, slow crossfades are missed or found late.
+- **Licence and provenance** are not audio questions: earshot cannot check them.