ffmpeg-skill 1.16.0 → 1.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +13 -8
- package/SKILL.md +5 -5
- package/docs/contract.md +28 -10
- package/package.json +1 -1
- package/references/gotchas.md +4 -0
- package/references/scripts.md +264 -4
- package/scripts/_common/__init__.py +25 -3
- package/scripts/_common/asr.py +369 -0
- package/scripts/_common/decision.py +346 -0
- package/scripts/_common/text.py +142 -2
- package/scripts/_contract.py +4 -2
- package/scripts/batch.py +287 -22
- package/scripts/caption.py +150 -198
- package/scripts/cut.py +136 -1
- package/scripts/render.py +289 -24
- package/scripts/scenes.py +81 -7
- package/scripts/silence.py +207 -5
package/README.md
CHANGED
|
@@ -82,7 +82,7 @@ An agent that "knows FFmpeg" still guesses: it assumes a frame rate, picks a cod
|
|
|
82
82
|
- **Structured tools, not shell strings.** Each operation is a script with typed arguments. Nothing runs through a shell; no filter graph is accepted from the caller.
|
|
83
83
|
- **A contract the agent can read.** `contract --json` states, for every tool, what it takes, what it writes, which FFmpeg components it needs and how the result is verified. The MCP surface is derived from it.
|
|
84
84
|
- **Verification after execution.** The result is probed, checked against the destination's spec and, when the picture changed, looked at as a contact sheet.
|
|
85
|
-
- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a whisper is installed, never required.
|
|
85
|
+
- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a whisper is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing.
|
|
86
86
|
|
|
87
87
|
## Quick start
|
|
88
88
|
|
|
@@ -190,16 +190,16 @@ These are the rules the skill file gives the agent and the code enforces.
|
|
|
190
190
|
| Tool | What it does |
|
|
191
191
|
|---|---|
|
|
192
192
|
| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision (`hdr` for BT.2020 or PQ/HLG, `hdr_signal` for a real PQ/HLG/DV transfer only), colour space, rotation, every audio stream; `--analyze` flags Log footage |
|
|
193
|
-
| `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments` |
|
|
193
|
+
| `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments`; `--beats` measures the music's beat grid (tempo, beat times, confidence) |
|
|
194
194
|
| `look.py` | Contact sheet, single frames, side-by-side comparison as PNG so the agent can see what it made; `--safe NAME` shades the zones a platform's own UI covers |
|
|
195
195
|
|
|
196
196
|
**Editing**
|
|
197
197
|
|
|
198
198
|
| Tool | What it does |
|
|
199
199
|
|---|---|
|
|
200
|
-
| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio; reports `precision` |
|
|
200
|
+
| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio; reports `precision`; `--snap beats` moves the in/out points onto a measured beat, or refuses when there is no measurable pulse |
|
|
201
201
|
| `join.py` | Concatenate clips with xfade transitions, normalising size, fps, sample rate and channel layout (the widest clip's, or `--channels`); audio-only inputs are joined as audio |
|
|
202
|
-
| `silence.py` | Detect and remove dead air (jump cuts) with a margin around speech; list or export the cut list |
|
|
202
|
+
| `silence.py` | Detect and remove dead air (jump cuts) with a margin around speech; list or export the cut list; `--filler` also removes filler words, but only where a speech engine timed them |
|
|
203
203
|
| `fit.py` | Fit to a duration (pitch-preserving speed change or trim, smooth slow-mo) and/or aspect ratio (pad, crop or `--fit blur`'s blurred, darkened fill, with `--crop-x`/`--crop-y` to keep an off-centre subject) and/or exact `--width`/`--height`; rotate 90/180/270, flip h/v; force constant fps |
|
|
204
204
|
| `crop.py` | Crop to an exact pixel rectangle (`--x --y --width --height`) — distinct from `fit.py --fit crop`, which crops to an aspect ratio it computes itself |
|
|
205
205
|
| `cropdetect.py` | Measure existing black letterbox/pillarbox bars and report the `crop.py`-ready rectangle that removes them — analysis only, writes no file |
|
|
@@ -234,7 +234,7 @@ These are the rules the skill file gives the agent and the code enforces.
|
|
|
234
234
|
|
|
235
235
|
| Tool | What it does |
|
|
236
236
|
|---|---|
|
|
237
|
-
| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area with a phrase-aware breaker (`--wrap phrase|measured`) and `--max-lines`/`--min-duration`/`--offset`; `--mode mux` takes a repeated `--srt file:lang` for several language-tagged, toggleable tracks in one file; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; optional local transcription |
|
|
237
|
+
| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area with a phrase-aware breaker (`--wrap phrase|measured`) and `--max-lines`/`--min-duration`/`--offset`; `--mode mux` takes a repeated `--srt file:lang` for several language-tagged, toggleable tracks in one file; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; `--fit-size` shrinks the size until a cue fits `--max-lines` instead of splitting the sentence; optional local transcription |
|
|
238
238
|
| `overlay.py` | Logos, watermarks and titles with position, time range, opacity, fades; `--platform NAME` keeps them clear of that destination's UI; `--video` for picture-in-picture, `--chromakey` for green-screen compositing |
|
|
239
239
|
| `graphics.py` | Lower-thirds, title cards, chapter chips, progress bars, countdowns, corner bugs, social stickers, opening hook cards and meme captions drawn by FFmpeg from a brand kit; `--platform NAME` keeps them inside that destination's safe zone; `--text-render` routes shaping scripts through libass and `--emoji-assets` composites colour emoji |
|
|
240
240
|
| `color.py` | HDR10 / HLG / Dolby Vision → SDR BT.709 tone mapping, DV layer stripping, 3D LUT (.cube), colour-tag rewriting, typed primary correction (exposure/contrast/saturation/gamma/white balance/lift-gain/levels/curves) |
|
|
@@ -252,8 +252,8 @@ These are the rules the skill file gives the agent and the code enforces.
|
|
|
252
252
|
|
|
253
253
|
| Tool | What it does |
|
|
254
254
|
|---|---|
|
|
255
|
-
| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after
|
|
256
|
-
| `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch` |
|
|
255
|
+
| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after`, `--cache DIR`/`--from STAGE` (reuse the stages that did not change); `--template NAME INPUT` renders a shipped delivery template (`--template all` writes the whole social pack plus its table) |
|
|
256
|
+
| `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch`; `--jobs N` processes several files at once under one shared `--timeout` |
|
|
257
257
|
| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list |
|
|
258
258
|
| `verify.py` | Run the toolchain on real device files and report PASS / FAIL per step |
|
|
259
259
|
|
|
@@ -361,7 +361,11 @@ The short list for humans. The agent-facing version, with the reasoning, is the
|
|
|
361
361
|
- **Loudness targets.** −14 LUFS / −1 dBTP for YouTube and social platforms (the `loudness.py` default), `-I -16 --tp -1.5` for podcasts, `-I -23` for broadcast. A clip measured at −40 LUFS or below is room tone, not content; raising it raises the noise. Check true peak as well as LUFS: `check.py file --platform podcast` measures both.
|
|
362
362
|
- **Frame changes first, text second.** Captions and overlays burned before a crop or resize end up off-frame. Reframe, then caption.
|
|
363
363
|
- **Cropping 16:9 to 9:16 discards 70 % of the width.** `fit.py --fit crop` centres by default; pass `--crop-x`/`--crop-y` toward the subject, or pad with `--fit pad --pad-fill blur`. Look at the contact sheet before deciding.
|
|
364
|
-
- **Phrase-aware caption breaking (1.16).** `caption.py`/`graphics.py --wrap phrase` (the default) never breaks inside a word or on the wrong side of a hyphen, never leaves a lone digit, kana or punctuation pair on a line, prefers Japanese sentence ends and particles over a mid-word break, and never ends a line on an article or preposition. All four are penalties over break positions that already fit, so no line is widened and the line count never changes; `--wrap measured` restores 1.15's width-only wrap. The text itself is never rewritten or shortened.
|
|
364
|
+
- **Phrase-aware caption breaking (1.16).** `caption.py`/`graphics.py --wrap phrase` (the default) never breaks inside a word or on the wrong side of a hyphen, never leaves a lone digit, kana or punctuation pair on a line, prefers Japanese sentence ends and particles over a mid-word break, and never ends a line on an article or preposition. All four are penalties over break positions that already fit, so no line is widened and the line count never changes; `--wrap measured` restores 1.15's width-only wrap. The text itself is never rewritten or shortened. Since 1.16.1 a Thai run and a katakana word are never broken inside (Thai writes no space inside a phrase and there is no dictionary: the break goes where you put a space or `|`), and a line wider than the safe width is reported as `overlong` with the fix named.
|
|
365
|
+
- **Caption size fitted to the cue (1.17).** At a platform caption size a line holds about six em, so an ordinary sentence needs four lines and `--max-lines 2` used to cut it into consecutive cues — half the sentence arriving late. `caption.py --fit-size` (default `auto`) now walks the size down until every cue fits, *before* laying the cues out, with a legibility floor of 4.5 % of the frame height (`--min-size`, default 13 ASS units). `--fit-size off` is 1.16 byte for byte, an explicit `--size` or a `brand.json` size is never overridden, and the caption text is still never rewritten to make it fit.
|
|
366
|
+
- **Beat-synced cuts (1.17).** `scenes.py --beats` reports the measured grid — tempo, beat times, and a confidence built from how far the winning autocorrelation lag stands above the others and how many onsets land on it. `cut.py --snap beats` (and a `"snap"` block in a `render.py` project) moves in/out points to the nearest beat within `--snap-tolerance` — and only onto the grid points a measured onset actually marks, never onto the regular grid's continuation through a passage with no music in it. Below `--min-confidence` it **refuses**: a cut point may move to a measured beat and may not appear from one, so speech and ambience get an honest "no steady pulse here" instead of an invented grid.
|
|
367
|
+
- **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Whisper stays optional: `--transcribe` with no engine installed refuses and names the three installs.
|
|
368
|
+
- **Throughput (1.17).** `batch.py --jobs N` runs several files at once, capped at `min(N, cpu_count, 8)` and sharing one `--timeout` budget rather than one per item; the per-item table keeps its order. `render.py --cache DIR` reuses stages whose inputs and arguments did not change, so swapping an export preset re-runs export only. The cache is opt-in with no default directory, and the ffmpeg, skill and contract versions are part of every key, so a cache is never reused across them.
|
|
365
369
|
- **Audiogram (1.16).** `waveform.py --image cover.png` (or `render.py --template audiogram`) puts the waveform over a still plate for an episode that has no picture, with `--platform` for the frame, `--title` and burnt-in captions. The image is a local file you give: nothing is fetched and no cover art is ever invented.
|
|
366
370
|
- **Emoji in captions and titles (1.15).** `caption.py`/`graphics.py --emoji-assets DIR` composites a PNG per emoji (Twemoji/Noto naming, `1f389.png`) on top of the text, because drawtext cannot load a colour emoji font at all and an installed one does not prove libass will draw it in colour — `doctor --json`'s `fonts.emoji` answers that from a render probe. Without assets the run still succeeds and reports `mode: mono`. Nothing is ever downloaded.
|
|
367
371
|
- **Indic and Thai text shaped correctly in titles and lower-thirds (1.15).** `graphics.py` renders Devanagari, Bengali, Tamil, Thai and Lao through libass automatically (`text_renderer: "ass"`), because drawtext never reorders matras or re-clusters marks; Arabic and Hebrew were already correct on a fribidi build. `--text-render drawtext` with such a script is refused, never rendered wrongly.
|
|
@@ -407,6 +411,7 @@ type on every OS.
|
|
|
407
411
|
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
|
|
408
412
|
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
|
|
409
413
|
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
|
|
414
|
+
| **90 / 90** | 1.16.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 30 runs, chapters and subtitle streams re-probed, check.py re-run on every delivery output) on the set grown to 90 prompts (audiogram, auto chapters, multi-language tracks, caption breaking): routing 90/90, honest refusals and failures 90/90 with 0 false successes and 0 raw ffmpeg calls, report format 89/90 (one `Done (partially):`), user's language 90/90 by regex (89/90 by Opus), audiogram 2/2 with the cover behind the waveform and nothing fetched, auto chapters 2/2 with `Chapter N` titles only, delivery 16/16 platform pass, trigger set 45/45, Opus quality mean 4.17. The honest part: the phrase breaker never gets to act at the platform caption sizes (a five-word cue does not fit two lines at TikTok size, so the split is byte-identical to 1.15.1), Thai still breaks inside words, and a katakana word was split — 1.16.1 is the patch and the finding is written up in `evals/results/iteration-17.json` |
|
|
410
415
|
| **82 / 82** | 1.15.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, stills extracted inside the emoji window, check.py re-run on every delivery output) on the set grown to 82 prompts (emoji captions and title cards, a Hindi and a Thai lower-third): routing 82/82, honest refusals and failures 82/82 with 0 false successes and 0 raw ffmpeg calls, report format 82/82 (both iteration-15 label defects closed: `dl8` and `he2` now carry one `Failed:`), user's language 82/82 by regex (81/82 by Opus: one Spanish report with three English labels), non-Latin glyphs 11/11 (Devanagari through `graphics.py` is fixed; Thai lower-third and captions correct), emoji visible in colour in 3/3 runs given PNG assets and reported monochrome in the one that was not, visual check 23/24, delivery 12/13 one encode and 13/13 platform pass, trigger set 40/40, Opus quality mean 4.68. Still open: the caption breaker splits phrases (`dl1`, `dl4` unchanged) — queued for 1.16.0. Tokens per run flat at 73.3k on the same 76. Details in `evals/results/iteration-16.json` |
|
|
411
416
|
| **76 / 76** | 1.14.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 26 runs, check.py re-run on every delivery output): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 by regex (75/76 by Opus: one Spanish report with three English labels), visual check 18/18, trigger set 38/38, Opus quality mean 4.58. The delivery templates did their job: 12 of 13 delivery requests went through `render.py --template`, finished in one encode (was 3 of 7) and all 13 pass their platform check (was 7 of 8). Tokens per run flat at 73.4k. Details in `evals/results/iteration-15.json` |
|
|
412
417
|
| **76 / 76** | 1.13.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 76 prompts: 18 in Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish and Italian, and 8 delivery requests (TikTok, Reels, Shorts, LinkedIn, Douyin, podcast): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 across seventeen languages, visual check 18/18, trigger set 38/38, Opus quality mean 4.65 over the 26 new runs. One real defect found: Hindi through `graphics.py` (drawtext) comes out wrong-shaped even though the font covers Devanagari; captions through libass are fine (queued for 1.15.0). Four delivery runs spent a second encode for loudness, which 1.14.0's templates address. Tokens per run flat at 72.3k. Details in `evals/results/iteration-14.json` |
|
package/SKILL.md
CHANGED
|
@@ -93,6 +93,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
|
|
|
93
93
|
| "add subtitles from this SRT", "burn in captions" | `caption.py input.mp4 --srt subs.srt` |
|
|
94
94
|
| "caption it with these lines" (text with times) | `caption.py input.mp4 --text cues.txt` |
|
|
95
95
|
| "keep the subtitles toggleable", "mux in an SRT" | `caption.py input.mp4 --srt subs.srt --mode mux`; repeat `--srt file:lang` for several languages, `.mkv` for more than two |
|
|
96
|
+
| "the captions are tiny / three lines on a Short" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` for 1.16 behaviour, `--min-size` sets the floor) |
|
|
96
97
|
| "our logo top-right", "a watermark" | `overlay.py input.mp4 --image logo.png --position top-right --scale 200` |
|
|
97
98
|
| "a title for the first 4 seconds" | `overlay.py input.mp4 --text "Title" --position top --start 0 --end 4 --fade 0.4` |
|
|
98
99
|
| "webcam clip in the corner", "picture-in-picture" | `overlay.py input.mp4 --video webcam.mp4 --position bottom-right --scale 480` |
|
|
@@ -180,17 +181,17 @@ When a step fails, replace `Done:` with `Failed:` and keep the rest honest:
|
|
|
180
181
|
Failed: color.py --lut grade.cube exited 1 — ffmpeg: "Unable to parse LUT file" (the .cube is not a valid LUT)
|
|
181
182
|
Steps: probe -> color (failed); nothing written
|
|
182
183
|
Check: nothing to verify
|
|
183
|
-
Look: not needed
|
|
184
|
+
Look: not needed
|
|
184
185
|
Notes: send a valid .cube, or say if you want the clip left as is
|
|
185
186
|
```
|
|
186
187
|
|
|
187
|
-
A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure still gets all five labels, never prose headings. A refusal that still delivers something is `Failed:` — the label answers the request as asked; the alternative goes in `Notes:`. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
|
|
188
|
+
A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure still gets all five labels, never prose headings. A refusal that still delivers something is `Failed:`, never a third label like `Done (partially):` — the label answers the request as asked; the alternative goes in `Notes:`. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
|
|
188
189
|
|
|
189
190
|
Every script prints `{"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool | timeout | verification | interrupted, "message": ...}}` with `--json` and exits non-zero; quote the message, never paraphrase it.
|
|
190
191
|
|
|
191
192
|
## Things that look right but are wrong
|
|
192
193
|
|
|
193
|
-
One line each
|
|
194
|
+
One line each; open the linked `references/gotchas.md` section when the job is in that area.
|
|
194
195
|
|
|
195
196
|
- HDR (iPhone, HDR10) re-encoded through an SDR path goes flat; the scripts keep HDR, and `hdr: true` is wider than `hdr_signal: true` (a real PQ/HLG/DV transfer). Details: [#hdr-and-colour](references/gotchas.md#hdr-and-colour)
|
|
196
197
|
- Log footage (S-Log/V-Log/C-Log) is tagged SDR and looks grey: `probe.py --analyze`, then `color.py --lut` before anything else. Details: [#log-footage](references/gotchas.md#log-footage)
|
|
@@ -198,7 +199,7 @@ One line each, each enough to act on; open the linked `references/gotchas.md` se
|
|
|
198
199
|
- VFR phone/screen recordings: re-encodes conform to CFR, `cut.py` switches to `--accurate`; pick the rate with `fit.py --fps` when the average is odd. Details: [#variable-frame-rate](references/gotchas.md#variable-frame-rate)
|
|
199
200
|
- Sync/multicam `confidence` under 0.3 (or a huge offset) is probably wrong — check every camera; these align audio, never lip sync. Details: [#sync-multicam-and-drift](references/gotchas.md#sync-multicam-and-drift)
|
|
200
201
|
- "Normalised" audio can still clip (check true peak), and ambience at -40 LUFS or below must never be raised to a speech target. Details: [#loudness-and-ambience](references/gotchas.md#loudness-and-ambience)
|
|
201
|
-
- Captions burned before a crop/resize land off-frame
|
|
202
|
+
- Captions burned before a crop/resize land off-frame, and burned small then upscaled by `export.py` they come out soft. Details: [#captions-fonts-and-text-order](references/gotchas.md#captions-fonts-and-text-order)
|
|
202
203
|
- Emoji need `--emoji-assets DIR` (a PNG per glyph) to render in colour; without it they come out monochrome and the run says so. Details: [#emoji](references/gotchas.md#emoji)
|
|
203
204
|
- `graphics.py` renders Devanagari, Bengali, Tamil and Thai through libass automatically — drawtext cannot shape them.
|
|
204
205
|
- Non-Latin text picks a font by script since 1.12; `doctor --json` `fonts.scripts` says which languages this machine renders; no font = failed job. Details: [#fonts-by-script](references/gotchas.md#fonts-by-script)
|
|
@@ -207,4 +208,3 @@ One line each, each enough to act on; open the linked `references/gotchas.md` se
|
|
|
207
208
|
- `yuv420p` needs even dimensions and phone rotation tags are honoured, both automatically. Details: [#dimensions-and-rotation](references/gotchas.md#dimensions-and-rotation)
|
|
208
209
|
- `scenes.py --highlights` ranks by loudness (or duration), never by meaning: check the sheet before treating picks as final. Details: [#highlights](references/gotchas.md#highlights)
|
|
209
210
|
- Three hand-chained re-encodes should be one `render.py` project; re-encodes use x264 `medium`. Details: [#chaining-and-speed](references/gotchas.md#chaining-and-speed)
|
|
210
|
-
- Windows drawtext crashes on some builds (#100): pass `--font-file` explicitly if one does. Details: `references/ci-platform-pitfalls.md`
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`1.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`1.17.0`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -88,19 +88,19 @@ spelling keeps working until 2.0.
|
|
|
88
88
|
|
|
89
89
|
| What 2.0 removes | Since | Replacement | To be ready today |
|
|
90
90
|
|---|---|---|---|
|
|
91
|
-
| The per-tool v1 success keys next to `result_v2` (`output`, `probe`, `commands`, `verified`, `verification` and each tool's own keys at the top level) | 1.
|
|
92
|
-
| `--crf` as an alias of `--quality` on every re-encoding tool that takes `--quality` (`export.py` keeps `--crf`: its preset chooses the encoder) | 1.
|
|
93
|
-
| `json` and `progress` in the MCP `inputSchema` | 1.
|
|
94
|
-
| `hdr` meaning "BT.2020 primaries *or* a PQ/HLG transfer" in `probe` | 1.
|
|
95
|
-
| Overwriting an existing output with only a warning | 1.
|
|
91
|
+
| The per-tool v1 success keys next to `result_v2` (`output`, `probe`, `commands`, `verified`, `verification` and each tool's own keys at the top level) | 1.17.0 | `result_v2`, promoted to the top level in 2.0 | Run with `FFMPEG_SKILL_RESULT_V2=1` and read `result_v2` (`metrics`, `notes`, `details`) instead of the top-level keys |
|
|
92
|
+
| `--crf` as an alias of `--quality` on every re-encoding tool that takes `--quality` (`export.py` keeps `--crf`: its preset chooses the encoder) | 1.17.0 | `--quality N` (the same CRF scale, codec-neutral) | Pass `--quality`; `--crf` warns on stderr and is marked in `--help` |
|
|
93
|
+
| `json` and `progress` in the MCP `inputSchema` | 1.17.0 | nothing: the transport sets them itself | Stop sending them from an MCP client; run the server with `FFMPEG_SKILL_MCP_LEAN=1` to see the 2.0 schema |
|
|
94
|
+
| `hdr` meaning "BT.2020 primaries *or* a PQ/HLG transfer" in `probe` | 1.17.0 | `hdr_signal` (true only for PQ / HLG / Dolby Vision); in 2.0 `hdr` takes that meaning | Key on `hdr_signal` for "is this a real HDR signal" and on `hdr_format` for the `BT.2020 SDR` case |
|
|
95
|
+
| Overwriting an existing output with only a warning | 1.17.0 | `--overwrite` as explicit consent (refused without it from 2.0) | Set `FFMPEG_SKILL_NO_OVERWRITE=1` (the recommended agent setting) and pass `--overwrite` where a replacement is intended |
|
|
96
96
|
|
|
97
97
|
## Skill
|
|
98
98
|
|
|
99
99
|
```json
|
|
100
100
|
{
|
|
101
101
|
"contract_version": "1.0",
|
|
102
|
-
"deprecated": [{"what": "...", "since": "1.
|
|
103
|
-
"skill": {"id": "ffmpeg-skill", "version": "1.
|
|
102
|
+
"deprecated": [{"what": "...", "since": "1.17.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
103
|
+
"skill": {"id": "ffmpeg-skill", "version": "1.17.0", "execution_mode": "local", "kind": "execution",
|
|
104
104
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
105
105
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
106
106
|
"requirements": {"python": ">=3.9 (standard library only)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
|
|
@@ -128,7 +128,7 @@ One entry per tool under `tools`, sorted by id. Tool ids are stable:
|
|
|
128
128
|
| `output_schema` | what `--json` prints on stdout |
|
|
129
129
|
| `supports_dry_run`, `dry_run` | whether `--dry-run` plans without running ffmpeg or writing files |
|
|
130
130
|
| `supports_json` | whether `--json` exists |
|
|
131
|
-
| `supports_json_brief` | whether `--json-brief` exists (1.
|
|
131
|
+
| `supports_json_brief` | whether `--json-brief` exists (1.17.0): the same success document with `probe` replaced by a compact `summary` (`duration_s`, `width`, `height`, `fps`, `vcodec`, `acodec`, `channels`, and `lufs` when the tool measured one), `commands` replaced by the number of commands run, and the per-step `verification` list dropped (its verdict stays in `verified`). Tool-specific keys are unchanged, `--json`'s own output is unchanged, and a failure prints the same failure document either way |
|
|
132
132
|
| `mutates_input` | always `false`: no tool overwrites its input |
|
|
133
133
|
| `produces_artifact` | writes a file (media, PNG, HTML, EDL) |
|
|
134
134
|
| `verification` | `{required, tools}`: which tools to run on the output afterwards |
|
|
@@ -212,7 +212,7 @@ gives you:
|
|
|
212
212
|
|---|---|
|
|
213
213
|
| `bit_exact` | probe, check, scenes, look |
|
|
214
214
|
| `content_equivalent` (same media, bytes may differ between encoder builds) | every encoding tool, cut, sync, report |
|
|
215
|
-
| `cached` | batch (content-hash cache, re-runs skip unchanged inputs) |
|
|
215
|
+
| `cached` | batch (content-hash cache, re-runs skip unchanged inputs); render **when `--cache DIR` is given** — the hint stays `content_equivalent` because that is what render is without the flag, and the cache is opt-in |
|
|
216
216
|
| `environment_dependent` | verify |
|
|
217
217
|
|
|
218
218
|
## `provides`
|
|
@@ -314,6 +314,12 @@ Names: `ffmpeg`, `ffprobe`, `encoder:<name>`, `filter:<name>`, `bsf:<name>`,
|
|
|
314
314
|
to omit detection. Nothing from the environment other than those lists and the
|
|
315
315
|
ffmpeg/ffprobe/python versions is printed; no environment variables, no paths.
|
|
316
316
|
|
|
317
|
+
`external:whisper` is **optional** for two tools since 1.17: `caption.py`
|
|
318
|
+
(`--transcribe`) and `silence.py` (`--filler --transcribe`). Neither requires
|
|
319
|
+
it — both take a transcript the caller already has (`--srt`/`--words`), and
|
|
320
|
+
both refuse with the same three install lines when asked to make one with no
|
|
321
|
+
engine present. Whisper is never a dependency of this skill.
|
|
322
|
+
|
|
317
323
|
`doctor` has three states per capability. `available` and `missing` come from a listing
|
|
318
324
|
that was read; `unknown` means the listing that would prove the capability could not be
|
|
319
325
|
read (`ffmpeg -filters` in a layout the parser does not recognise, or ffmpeg exiting
|
|
@@ -433,6 +439,18 @@ given a different type):
|
|
|
433
439
|
| `auto_chapters` | `metadata.py --auto-chapters` | `{source, min_chapter, max_chapters, proposed, kept, titles, chapters, description_block, files}`. `titles` is always `"placeholder"`: the machine-readable form of "the skill did not name these". Each chapter carries its `evidence` (`start`, `silence`, `scene`, or `silence+scene` with the span, its length and the cut time) |
|
|
434
440
|
| `audiogram` | `waveform.py` (every run) | `{style, background, image, position, vis_height, platform, captions, title, stages, verified}`. `background` is `"image"` or `"color"`; `verified` is true when the render probes at the asked-for frame size, frame rate and within 0.05 s of the source audio, and is `false` under `--dry-run`, where nothing was rendered to verify |
|
|
435
441
|
|
|
442
|
+
Per-tool keys added in 1.17, all additive:
|
|
443
|
+
|
|
444
|
+
| key | tool | what it holds |
|
|
445
|
+
|---|---|---|
|
|
446
|
+
| `fit_size`, `size_requested`, `size_used`, `size_floor`, `size_pct_height`, `shrunk`, `fit_scope`, `fit_exhausted` | `caption.py` | siblings inside the same `caption` block: which mode fitted the size (`auto`/`on`/`off`), the size asked for and the size used in ASS units, the floor (13 = 4.5 % of the frame height), that size as a percentage of the frame, how many cues the shrink rescued, `file` or `cue` scope, whether the floor was reached with cues still split, and `size_source` (`input`, or `platform-frame` when a plan was written before the input existed). `size_used` is `null` when there was no geometry to fit against at all |
|
|
447
|
+
| `beats`, `beat_grid` | `scenes.py --beats` | the measured beat times, and `{supported_beats, tempo_bpm, interval, confidence, phase, onsets, supported, unsupported, method, step_s, range_bpm, usable}`. `supported_beats` is the subset of the regular grid that a measured onset marks — the only list a tool that moves a cut may snap to. `usable` is `confidence >= --min-confidence`; a low confidence is reported, not refused — `scenes.py` measures, it does not act |
|
|
448
|
+
| `snap` | `cut.py --snap beats`, `render.py` | `{mode, tolerance, confidence, tempo_bpm, grid, grid_points, moved, snapped, unchanged, source}`. `grid` is `"supported"`: points are moved only onto grid points a measured onset marks, never onto the regular grid's continuation through a silent passage. `moved` has exactly one row per in/out point given (`from`, `to`, `delta`, `snapped`, `beat_index`) — a point is never added or dropped, and `to` is always either a measured beat or the caller's own value |
|
|
449
|
+
| `filler`, `removed_seconds_total` | `silence.py --filler` | `{lang, source, engine, words, removed, removed_count, removed_seconds, removed_words, word_timings, list, warnings}`. The existing `removed_seconds` is unchanged in name and meaning — the seconds of *silence* removed, which is what it has always held — and `removed_seconds_total` is the additive sibling covering silence plus filler |
|
|
450
|
+
| `jobs`, `jobs_requested`, `wall_seconds`, `item_seconds_total`, `timed_out` | `batch.py` | the parallelism actually applied and the number asked for, the batch's wall clock, the sum of the per-item times (so the speed-up can be quoted), and whether the shared timeout budget ran out. A timed-out item carries `"skipped": "timeout"` in its result row |
|
|
451
|
+
| `cache` | `render.py --cache` | `{dir, ffmpeg, hits, misses, saved_seconds, entries}`, plus `would_hit` under `--dry-run`. The ffmpeg build banner, the skill version, the contract version, the forwarded flags (`--fast`, `--codec`, …) and the output's extension are all part of every key, so a cache is never reused across any of them — a `--fast` draft is never served to a run that did not ask for one |
|
|
452
|
+
|
|
453
|
+
|
|
436
454
|
`check.py` also gains an informational `subtitles` row on **every** platform:
|
|
437
455
|
`PASS` when every soft subtitle stream carries a language tag, `WARN` when one
|
|
438
456
|
does not (or when there are none). Like `channels` and `chapters` it is never
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.17.0",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
package/references/gotchas.md
CHANGED
|
@@ -122,6 +122,10 @@ printing one line — `font: /usr/share/fonts/.../wqy-zenhei.ttc (covers ko)`.
|
|
|
122
122
|
`brew install --cask font-noto-sans-cjk font-noto-sans-arabic`, or point at a
|
|
123
123
|
file with `--font-file` (`overlay.py`, `graphics.py`) / `--fonts-dir`
|
|
124
124
|
(`caption.py`).
|
|
125
|
+
- Windows drawtext crashes on some builds (#100) when it has to resolve a font by
|
|
126
|
+
family name: pass `--font-file` explicitly if one does. `default_font_file()`
|
|
127
|
+
already resolves a file for you on Windows, so this bites only a hand-built
|
|
128
|
+
drawtext call. More: `references/ci-platform-pitfalls.md`.
|
|
125
129
|
- Han characters alone (no kana, no hangul) are read as Chinese. Japanese or
|
|
126
130
|
Korean hanja text with no kana needs `--lang ja` / `--lang ko`
|
|
127
131
|
(`caption.py --language` is the same flag), or `"lang"` in brand.json.
|
package/references/scripts.md
CHANGED
|
@@ -64,6 +64,7 @@ BT.2020 primaries on an SDR transfer, which `hdr_format` names "BT.2020 SDR" --
|
|
|
64
64
|
### cut.py — cut / join segments
|
|
65
65
|
```
|
|
66
66
|
cut.py INPUT [--start T] [--end T | --duration T] [--segments A-B,C-D,...] [--accurate] [-o OUT]
|
|
67
|
+
cut.py INPUT --start T --end T --snap beats [--snap-tolerance 0.12] [--snap-source FILE] [--min-confidence 0.5]
|
|
67
68
|
```
|
|
68
69
|
Times accept `12.5`, `1:30`, `00:01:30.250`. Default is `-c copy` (snaps to
|
|
69
70
|
keyframes, instant, lossless); if the snapped result deviates more than
|
|
@@ -74,6 +75,49 @@ the result was "lossless stream copy" or "re-encoded"; when the snap forced a
|
|
|
74
75
|
re-encode, the result's `lossless_alternative` names the nearest keyframe
|
|
75
76
|
`--start` that would stream-copy instead, so the trade can be offered.
|
|
76
77
|
|
|
78
|
+
**`--snap beats` (1.17)** moves each in/out point to the nearest *measured*
|
|
79
|
+
beat within `--snap-tolerance` seconds (default 0.12, about a quarter of a beat
|
|
80
|
+
at 120 BPM). The grid comes from the input's own audio, or from
|
|
81
|
+
`--snap-source` — either a `scenes.py --beats --json` document (no second
|
|
82
|
+
decode) or a separate music file. The snap happens *before* the
|
|
83
|
+
keyframe/tolerance decision, so lossless-vs-accurate is judged on where the cut
|
|
84
|
+
actually lands.
|
|
85
|
+
|
|
86
|
+
"Measured" is exact: the only points a cut may move onto are the
|
|
87
|
+
`supported_beats` — the grid points a measured onset actually marks — never the
|
|
88
|
+
full regular grid, which by construction runs on through a passage with no music
|
|
89
|
+
in it. A cut asked for inside that passage stays where it was asked for, and
|
|
90
|
+
`snap.grid` is `"supported"` with `snap.grid_points` saying how many there were.
|
|
91
|
+
`--min-confidence` must be greater than 0: at 0 every grid is "reliable",
|
|
92
|
+
including one measured from noise.
|
|
93
|
+
|
|
94
|
+
The number of in/out points never changes: a point with no beat inside the
|
|
95
|
+
tolerance is returned unchanged, and no point is ever invented. Three
|
|
96
|
+
`kind: input` refusals, nothing written:
|
|
97
|
+
|
|
98
|
+
- confidence below `--min-confidence` → *"no reliable beat grid in this audio
|
|
99
|
+
(confidence 0.21, needs 0.5): cutting to invented beats would move your in/out
|
|
100
|
+
points to times nothing in the audio supports. Re-run with `--snap none`, or
|
|
101
|
+
pass `--snap-source` from a music bed."*
|
|
102
|
+
- no audio stream → names `--snap none`.
|
|
103
|
+
- no in/out point at all (a whole-file copy) → there is nothing to snap, and
|
|
104
|
+
this tool never proposes cuts of its own.
|
|
105
|
+
|
|
106
|
+
Result: `snap.mode`, `snap.tolerance`, `snap.confidence`, `snap.tempo_bpm`,
|
|
107
|
+
`snap.moved` (one row per point, with `from`/`to`/`delta`/`snapped`),
|
|
108
|
+
`snap.snapped`, `snap.unchanged`, `snap.grid`, `snap.grid_points`, and
|
|
109
|
+
`snap.source` (`"measured"` or the path). In a `render.py` result, `snap.clips`
|
|
110
|
+
carries one entry per snapped clip with `clip` naming its index, and the first
|
|
111
|
+
entry's keys are repeated at the top level for a single-clip project. A clip
|
|
112
|
+
served from `--cache` was snapped when it was first rendered, and says so
|
|
113
|
+
(`source: "cache"`) rather than reporting `snap: null`. A `--snap-source`
|
|
114
|
+
document that does
|
|
115
|
+
not carry `beat_grid.supported_beats` (one written before 1.17) is refused
|
|
116
|
+
rather than treated as if every grid point were supported, as is one whose
|
|
117
|
+
`tempo_bpm` is null while it lists beats.
|
|
118
|
+
`render.py` forwards a project's `"snap": {"to": "beats", ...}` (project-wide
|
|
119
|
+
or per clip) to this flag and reports what came back.
|
|
120
|
+
|
|
77
121
|
### fit.py — target duration and/or aspect, rotate/flip
|
|
78
122
|
```
|
|
79
123
|
fit.py INPUT [--duration T --method speed|trim [--from-center] [--max-speed 4]]
|
|
@@ -422,6 +466,9 @@ each shorter clip's last frame (with silence) out to the longest.
|
|
|
422
466
|
### silence.py — remove dead air / jump cuts
|
|
423
467
|
```
|
|
424
468
|
silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [-o OUT]
|
|
469
|
+
silence.py INPUT --filler --words transcript.json [--filler-lang auto|en|ja|es|de|fr|pt|it]
|
|
470
|
+
[--filler-words FILE] [--filler-extra W,W] [--filler-keep W,W] [--filler-pad 0.02]
|
|
471
|
+
[--transcribe] [--filler-list] [--max-cuts 400]
|
|
425
472
|
```
|
|
426
473
|
Runs `silencedetect`, keeps `--margin` seconds of air around speech, drops
|
|
427
474
|
gaps shorter than `--min-silence`, and re-encodes once with `select`/`aselect`
|
|
@@ -430,6 +477,60 @@ without rendering; `--edl` saves the kept ranges in `cut.py --segments` format
|
|
|
430
477
|
so the user can edit the list by hand. Quiet rooms need `--threshold -40`
|
|
431
478
|
to `-45`; noisy ones `-30`. Always tell the user how many seconds were removed.
|
|
432
479
|
|
|
480
|
+
**`--filler` (1.17)** removes filler words as well, through the same
|
|
481
|
+
`keep_ranges()`/`aselect` graph with more, shorter ranges. It is **never**
|
|
482
|
+
applied without measured word timings: pass `--words` (a whisper JSON carrying
|
|
483
|
+
per-word `start`/`end`) or `--transcribe` (the same local engine `caption.py`
|
|
484
|
+
uses, never required). There is no heuristic fallback — finding an "um" by
|
|
485
|
+
looking for short quiet blips would cut real speech, so the tool refuses
|
|
486
|
+
instead. `--filler-list` reports what would go and writes nothing.
|
|
487
|
+
|
|
488
|
+
Built-in lists, one per language:
|
|
489
|
+
|
|
490
|
+
| lang | words |
|
|
491
|
+
|---|---|
|
|
492
|
+
| en | um, uh, erm, hmm, mm, mhm, er, ah |
|
|
493
|
+
| ja | えー, えーと, えっと, あの, あのー, その, そのー, まあ, なんか |
|
|
494
|
+
| es | eh, este, esto, mmm |
|
|
495
|
+
| de | äh, ähm, hm |
|
|
496
|
+
| fr | euh, hein |
|
|
497
|
+
| pt | é, hum |
|
|
498
|
+
| it | ehm |
|
|
499
|
+
|
|
500
|
+
**Discourse markers are not disfluencies.** `like`, `tipo` and `cioè` are
|
|
501
|
+
deliberately *not* in the default lists: they are grammatical words in most
|
|
502
|
+
sentences, and removing them cuts meaning rather than noise — a judgement about
|
|
503
|
+
content, which this skill does not make. They are reachable with
|
|
504
|
+
`--filler-extra like`, which says so in its own `--help`. Japanese `なんか` is
|
|
505
|
+
in the `ja` list because leaving it out makes the flag useless for Japanese,
|
|
506
|
+
and it is orthographically identical to the pronoun use — so every run that
|
|
507
|
+
removes one warns, and `--filler-keep なんか` takes it back out. Matching is
|
|
508
|
+
case-folded, punctuation-stripped and whole-token only: `umbrella` survives.
|
|
509
|
+
|
|
510
|
+
Three refusals, all `kind: input`, all before any encode:
|
|
511
|
+
|
|
512
|
+
- `--filler` with neither `--words` nor `--transcribe` → names both flags.
|
|
513
|
+
- `--transcribe` with no engine on PATH → the same message `caption.py` gives,
|
|
514
|
+
with the three install lines.
|
|
515
|
+
- `--transcribe` where the engine runs but its build produces no word-level
|
|
516
|
+
timings → names that engine, says some builds do not support word timestamps,
|
|
517
|
+
and points at `--words`. (`--transcribe` drives whichever engine is installed
|
|
518
|
+
with *its* word-timestamp option — whisper.cpp `--output-json-full`,
|
|
519
|
+
faster-whisper `word_timestamps=True`, openai-whisper `--word_timestamps
|
|
520
|
+
True` — because an SRT cannot answer this: a cue has a start and an end, a
|
|
521
|
+
word does not. `--words` is the tested path and the one to prefer.)
|
|
522
|
+
- a transcript with segments but no word-level timings → says that cutting on
|
|
523
|
+
segment boundaries would remove whole sentences, and how to re-run whisper.
|
|
524
|
+
|
|
525
|
+
Results: `filler.removed_count`, `filler.removed_seconds`, `filler.removed`
|
|
526
|
+
(one entry per span), `filler.lang`, `filler.list`, `filler.word_timings` and
|
|
527
|
+
`filler.warnings`. The existing `removed_seconds` keeps exactly the meaning it
|
|
528
|
+
has always had — the seconds of **silence** this run removed, the figure the
|
|
529
|
+
same run would report without `--filler` — and `removed_seconds_total` is the
|
|
530
|
+
additive sibling covering everything that went. A filler word quiet enough to
|
|
531
|
+
sit inside a detected silence is merged into it rather than counted twice, so
|
|
532
|
+
the two figures can be equal.
|
|
533
|
+
|
|
433
534
|
### join.py — concatenate with transitions
|
|
434
535
|
```
|
|
435
536
|
join.py CLIP1 CLIP2 [...] [--transition fade|dissolve|wipeleft|slideleft|fadeblack|fadewhite|circleopen|none]
|
|
@@ -447,6 +548,7 @@ output; the result says so with `dropped_non_av_streams: true`.
|
|
|
447
548
|
```
|
|
448
549
|
render.py --init project.json # starter file
|
|
449
550
|
render.py project.json [--fast] [--dry-run] [--stop-after STAGE] [--work DIR --keep]
|
|
551
|
+
render.py project.json --cache DIR [--from STAGE] # reuse identical stages from a previous run
|
|
450
552
|
render.py plan.json # execute a plan written by <tool> --plan plan.json
|
|
451
553
|
```
|
|
452
554
|
A plan is a single tool's dry run as an artifact: `cut.py in.mp4 --start 2 --end 8
|
|
@@ -479,6 +581,40 @@ entries and the file path are validated before the first stage runs, and the
|
|
|
479
581
|
stage plans its `metadata.py` command under `--dry-run`/`--plan` like every
|
|
480
582
|
other stage, so the plan lists `chapters` and the run does the same work.
|
|
481
583
|
|
|
584
|
+
**`--cache DIR` (1.17)** reuses the artifact of a stage that already ran with
|
|
585
|
+
exactly the same arguments and the same inputs. Opt-in only: **there is no
|
|
586
|
+
default cache directory** — one appearing on someone's disk unasked would
|
|
587
|
+
contradict this tool's "a plan leaves nothing behind" posture. The key is a
|
|
588
|
+
sha1 over the stage name, the tool, its arguments (every existing path replaced
|
|
589
|
+
by its content hash), the input hashes, the **forwarded** flags (`--fast`,
|
|
590
|
+
`--overwrite`, `--timeout`, `--codec`), the output's extension, and **the ffmpeg
|
|
591
|
+
build banner, the skill version and the contract version**. `--fast` matters as
|
|
592
|
+
much as any of them: it rewrites every child's preset to `veryfast`, so without
|
|
593
|
+
it in the key a `--cache --fast` draft would be served back to a later run that
|
|
594
|
+
asked for the delivery. The banner rather than `major.minor` because two 7.1.x
|
|
595
|
+
builds with different libx264 produce different bytes from the same command. Those last three are in the key deliberately: a
|
|
596
|
+
different build simply *misses* rather than being asked to trust a file it did
|
|
597
|
+
not write, and a stage whose implementation changed cannot serve back an
|
|
598
|
+
artifact the old one produced. Each entry is `DIR/<key><ext>` plus a
|
|
599
|
+
`DIR/<key>.json` sidecar (stage, versions, creation time, size, seconds). A hit
|
|
600
|
+
hardlinks the artifact into the work directory, or copies it where the
|
|
601
|
+
filesystem will not link — never moves it, since the cache has to outlive the
|
|
602
|
+
run's own cleanup. Any mismatch is a silent miss.
|
|
603
|
+
|
|
604
|
+
`stages_done` is unchanged: a cached stage is still a stage that happened.
|
|
605
|
+
Nothing is written under `--dry-run`, which instead reports `cache.would_hit`.
|
|
606
|
+
Result: `cache.dir`, `cache.ffmpeg`, `cache.hits`, `cache.misses`,
|
|
607
|
+
`cache.saved_seconds`, `cache.entries`. `--from STAGE` starts at that stage and
|
|
608
|
+
takes every earlier one from the cache; without `--cache`, or when an earlier
|
|
609
|
+
stage is not there, it refuses (`kind: input`) rather than quietly re-encoding
|
|
610
|
+
what it promised to skip. An unwritable cache directory is `kind: output`.
|
|
611
|
+
|
|
612
|
+
`"snap": {"to": "beats", "tolerance": 0.12, "min_confidence": 0.5, "source":
|
|
613
|
+
"music.mp3"}` at the project root (or inside one clip) forwards `--snap beats`
|
|
614
|
+
to the clip cut for every clip that has `in`/`out`; the measured grid comes back
|
|
615
|
+
in the render result's `snap`. A project without `"snap"` builds the command
|
|
616
|
+
line 1.16 built.
|
|
617
|
+
|
|
482
618
|
Stages: clips (cut, optional speed) → join (transition) → silence → fit →
|
|
483
619
|
captions → graphics → overlays → audio → loudness → export → chapters → check. Keys mirror the
|
|
484
620
|
CLI flags of each script (see the docstring); a key `render.py` does not read -- at
|
|
@@ -549,6 +685,7 @@ already has a picture.
|
|
|
549
685
|
### scenes.py — scene changes and highlight candidates
|
|
550
686
|
```
|
|
551
687
|
scenes.py INPUT [--threshold 10] [--min-scene 1] [--highlights N [--target SECONDS] [--max-scene 15]] [--edl picks.txt] [--sheet scenes.png] [--json]
|
|
688
|
+
scenes.py INPUT --beats [--beat-step 0.01] [--beat-range 60-200] [--min-confidence 0.5] [--json]
|
|
552
689
|
```
|
|
553
690
|
Lists scenes with audio energy, the loudest moments, and (with
|
|
554
691
|
`--highlights`) proposes N ranges that add up to `--target` seconds, biased to
|
|
@@ -560,6 +697,45 @@ Dissolves and very slow fades are not cuts and will be missed. Highlights are
|
|
|
560
697
|
a proposal engine, not a judgement of content: tell the user what it picked
|
|
561
698
|
and why (energy, scene length).
|
|
562
699
|
|
|
700
|
+
**`--beats` (1.17)** measures the music's beat grid and reports it:
|
|
701
|
+
|
|
702
|
+
```json
|
|
703
|
+
"beats": [0.0, 0.5, 1.0, ...], // 25 of them over a 12 s click track
|
|
704
|
+
"beat_grid": {"tempo_bpm": 120.0, "interval": 0.5, "confidence": 0.997,
|
|
705
|
+
"supported_beats": [0.0, 0.5, 1.0, ...], // the 24 an onset marks
|
|
706
|
+
"phase": 0.0, "onsets": 24, "supported": 24, "unsupported": 1,
|
|
707
|
+
"method": "rms-flux-autocorrelation", "step_s": 0.01,
|
|
708
|
+
"range_bpm": [60, 200], "usable": true}
|
|
709
|
+
```
|
|
710
|
+
|
|
711
|
+
Method: onset strength as the half-wave-rectified first difference of
|
|
712
|
+
`log(envelope)`, peaks above `median + 1.5·MAD` over a ±1 s window with a 60 ms
|
|
713
|
+
refractory gap, tempo from the autocorrelation of the onset signal inside
|
|
714
|
+
`--beat-range` with its half and double checked, phase chosen to catch the most
|
|
715
|
+
onset strength. `confidence` is half how far the winning lag stands above the
|
|
716
|
+
other lags (in standard deviations) and half the fraction of onsets that land
|
|
717
|
+
on the grid. With `--beats` the file is decoded once, at 22050 Hz, and the
|
|
718
|
+
`supported + unsupported == len(beats)` always, and `supported_beats` is that
|
|
719
|
+
supported subset — the list `cut.py --snap beats` moves onto.
|
|
720
|
+
|
|
721
|
+
With `--beats` the file is decoded once, at 22050 Hz, and the 0.5 s scene
|
|
722
|
+
envelope is derived from that same pass rather than from a second 8 kHz decode.
|
|
723
|
+
One consequence worth knowing: `scenes.py X --json` and `scenes.py X --beats
|
|
724
|
+
--json` report very slightly different `audio_rms`/`audio_peak` figures for the
|
|
725
|
+
same file, because the two envelopes are built from PCM at different rates. The
|
|
726
|
+
scene boundaries and their ranking are unaffected; only the fourth decimal of
|
|
727
|
+
the level moves. `--beats` also holds ~2.75x the samples in memory, which is
|
|
728
|
+
worth knowing on a feature-length input.
|
|
729
|
+
|
|
730
|
+
**A beat grid is a measurement of the music's periodicity, not of where a cut
|
|
731
|
+
belongs.** A low confidence means the audio has no steady pulse — speech,
|
|
732
|
+
ambience, rubato — and the skill will not snap to a grid it cannot measure.
|
|
733
|
+
`scenes.py` is the analysis tool, so it *reports* a weak measurement
|
|
734
|
+
(`usable: false`) rather than refusing it; refusing belongs to the tools that
|
|
735
|
+
would change a file on the strength of it (`cut.py --snap beats`). `--edl` with
|
|
736
|
+
`--beats` is unchanged: beats are never written as an EDL, because a beat is
|
|
737
|
+
not a cut. No audio stream is a `kind: input` refusal.
|
|
738
|
+
|
|
563
739
|
### check.py — pre-delivery compliance
|
|
564
740
|
```
|
|
565
741
|
check.py INPUT --platform youtube|shorts|reels|tiktok|x|linkedin|facebook|broadcast|podcast|custom [--no-loudness] [--json]
|
|
@@ -582,13 +758,42 @@ construction, not by two lists agreeing.
|
|
|
582
758
|
|
|
583
759
|
### batch.py — same recipe over a folder, cached
|
|
584
760
|
```
|
|
585
|
-
batch.py FOLDER --recipe batch.json [--force] [--watch SECONDS] [--json]
|
|
761
|
+
batch.py FOLDER --recipe batch.json [--force] [--watch SECONDS] [--jobs N|auto] [--json]
|
|
586
762
|
```
|
|
587
763
|
`batch.json` holds either `steps` (a list of script argv with `{in}`/`{out}`
|
|
588
764
|
placeholders, chained) or `project` (a render project applied per file).
|
|
589
765
|
Outputs land in `output_dir` with `suffix`; a content-hash cache skips files
|
|
590
766
|
already done with the same recipe. Use `--dry-run` to preview the plan.
|
|
591
767
|
|
|
768
|
+
**`--jobs N` (1.17)** processes N files at once (threads: the work is
|
|
769
|
+
subprocess waiting). Capped at `min(N, cpu_count, 8)` — every item is itself an
|
|
770
|
+
ffmpeg that already threads across cores, so beyond a few concurrent encodes
|
|
771
|
+
the jobs contend and wall-clock stops improving while memory does not. A number
|
|
772
|
+
above the cap is clamped with a note, not refused, and both `jobs` and
|
|
773
|
+
`jobs_requested` are in the result. `--jobs auto` is `min(cpu_count, 4)`.
|
|
774
|
+
|
|
775
|
+
- **One timeout budget for the whole batch**, not one per item: `--timeout` is
|
|
776
|
+
computed into a deadline once, no new item starts after it, and the run exits
|
|
777
|
+
124 `kind: timeout` with the items that never started marked
|
|
778
|
+
`"skipped": "timeout"`. That shared budget applies when a `--timeout` was
|
|
779
|
+
actually given, or when `--jobs > 1` asked for the batch to be treated as one
|
|
780
|
+
piece of work; the default sequential path with the default timeout is 1.16's
|
|
781
|
+
behaviour exactly, where a long folder was never cut off part-way.
|
|
782
|
+
- **Determinism**: every item is written into its own slot in one list indexed
|
|
783
|
+
by position in the sorted file list — cached hits included, which is what keeps
|
|
784
|
+
the table in file order when the cache is only partially warm — and each item's
|
|
785
|
+
log lines are buffered and flushed in file order, so the per-item table and the
|
|
786
|
+
log read exactly as a serial run's whatever order the encodes finish in.
|
|
787
|
+
- **A failed item is a row, not a dead run**: a worker that raises becomes
|
|
788
|
+
`ok: false` with its reason, and the summary and the table are still printed.
|
|
789
|
+
Ctrl-C cancels what has not started, keeps what finished, and exits 130
|
|
790
|
+
`kind: interrupted` with the partial table.
|
|
791
|
+
- With `--jobs > 1` each item gets its own work subdirectory
|
|
792
|
+
`<workdir>/<index>-<stem>/`, because step file names are stem-derived and two
|
|
793
|
+
sources sharing a stem would otherwise write over each other. `--jobs 1` (the
|
|
794
|
+
default) keeps the flat layout and is 1.16's path exactly.
|
|
795
|
+
- `--watch` composes: each pass is parallel, passes are sequential.
|
|
796
|
+
|
|
592
797
|
### caption.py --transcribe — optional local speech-to-text
|
|
593
798
|
If `whisper-cli` (whisper.cpp), `faster-whisper` or `whisper` is installed,
|
|
594
799
|
`caption.py input.mp4 --transcribe [--language ja] [--model base]` writes the
|
|
@@ -726,11 +931,62 @@ caption.py INPUT --srt FILE[:LANG] | --ass FILE | --text CUES.txt [--write-srt O
|
|
|
726
931
|
[--audio-stream N] [--fps N] [--lang XX] [--offset TIME]
|
|
727
932
|
[--max-lines N] [--min-duration S] [--wrap phrase|measured]
|
|
728
933
|
[--font NAME] [--fonts-dir DIR] [--size N] [--color RRGGBB] [--outline N] [--outline-color RRGGBB]
|
|
934
|
+
[--fit-size auto|on|off] [--min-size N] [--fit-size-scope file|cue]
|
|
729
935
|
[--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N]
|
|
730
936
|
[--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass]
|
|
731
937
|
[--emoji auto|color|png|mono|none] [--emoji-assets DIR] [--emoji-scale 1.0] [--emoji-max 60] [-o OUT]
|
|
732
938
|
caption.py --text CUES.txt --write-srt OUT.srt # generate the SRT only
|
|
733
939
|
```
|
|
940
|
+
**`--fit-size` (1.17): the size is fitted before a cue is split.** At the TikTok
|
|
941
|
+
caption size (24 ASS units against the 288-line script grid) a line has about
|
|
942
|
+
six em, so an ordinary sentence needs four lines — and `--max-lines 2` then cut
|
|
943
|
+
it into consecutive cues, so half of it arrived late. That was the size, not the
|
|
944
|
+
breaker. `--fit-size` walks the size down until every cue wraps within
|
|
945
|
+
`--max-lines`, and only then lays the cues out.
|
|
946
|
+
|
|
947
|
+
- `auto` (default) shrinks only a size the *skill* chose. An explicit `--size`,
|
|
948
|
+
or a `brand.json` `styles.caption.size`, is a statement about the look and is
|
|
949
|
+
never overridden.
|
|
950
|
+
- `on` always fits, `--size` or not. **`off` is 1.16.1 exactly, byte for byte**
|
|
951
|
+
(a pinned ASS fixture asserts it).
|
|
952
|
+
- `--min-size` is the floor, default `ass_units(0.045) = 13` — **4.5 % of the
|
|
953
|
+
frame height**, one floor for every destination (87 px of type on a 1920-tall
|
|
954
|
+
frame, above the ~3.5 % where mobile legibility and the platforms' own caption
|
|
955
|
+
UIs bottom out). Nothing per-platform is measured, so nothing is claimed.
|
|
956
|
+
- `--fit-size-scope file` (default) uses one size for the whole file; `cue`
|
|
957
|
+
writes a per-cue `{\fsN}` override. A size that changes from cue to cue reads
|
|
958
|
+
as a mistake, so it is opt-in — it exists for the one outlier cue that would
|
|
959
|
+
otherwise shrink a ten-minute file. Each cue is laid out at the size it will
|
|
960
|
+
be **drawn** at: a cue drawn larger has a narrower line in em, so wrapping
|
|
961
|
+
everything to the file-wide budget and then drawing some cues large would put
|
|
962
|
+
lines off the side of the frame.
|
|
963
|
+
|
|
964
|
+
The text is **never** touched: this skill does not rewrite, shorten or
|
|
965
|
+
paraphrase a caption to make it fit. Below the floor the cue is split exactly as
|
|
966
|
+
before and `fit_exhausted: true` says so, with `--min-size` named as the flag
|
|
967
|
+
that would go smaller and `|` as the manual break. `--min-size` above `--size`
|
|
968
|
+
is a `kind: input` refusal. `--mode mux` is unaffected: soft subtitles carry no
|
|
969
|
+
size, so the SRT is the one 1.16 wrote.
|
|
970
|
+
|
|
971
|
+
Results, alongside the existing caption stats:
|
|
972
|
+
`fit_size`, `size_requested`, `size_used`, `size_floor`, `size_pct_height`,
|
|
973
|
+
`shrunk`, `fit_scope`, `fit_exhausted`, and `size_source` (`input` or
|
|
974
|
+
`platform-frame`).
|
|
975
|
+
|
|
976
|
+
Under `--dry-run`/`--plan` on an input that does not exist yet there is no
|
|
977
|
+
geometry to measure. With `--platform` the destination's own frame is used —
|
|
978
|
+
that frame *is* what the real run will have, so the planned `FontSize` is the
|
|
979
|
+
one the run will burn. Without a platform nothing can stand in for the frame
|
|
980
|
+
and `size_used` is `null`, rather than presenting the requested size as a
|
|
981
|
+
fitted one.
|
|
982
|
+
|
|
983
|
+
| ASS size | px on 1920 | % frame h | em per line |
|
|
984
|
+
|---|---|---|---|
|
|
985
|
+
| 24 (default) | 160.0 | 8.3 % | 6.08 |
|
|
986
|
+
| 19 | 126.7 | 6.6 % | 7.67 |
|
|
987
|
+
| 16 | 106.7 | 5.6 % | 9.11 |
|
|
988
|
+
| 13 (floor) | 86.7 | 4.5 % | 11.22 |
|
|
989
|
+
|
|
734
990
|
Text cue format, one per line: `0:00-0:03 Hello`, `00:00:03.500 --> 00:00:06 Two | lines`,
|
|
735
991
|
or `00:00:03:15 --> 00:00:06:00 SMPTE non-drop-frame timecode` (`hh:mm:ss:ff`, frame count
|
|
736
992
|
converted with `--fps`, or the input video's own fps when `--input` is given and `--fps` is
|
|
@@ -758,9 +1014,13 @@ frame width) at the chosen `--size`, measured per script — CJK and Thai count
|
|
|
758
1014
|
full em per character, Latin per character from a table read off DejaVu Sans (so
|
|
759
1015
|
an all-caps line measures as wide as it draws), Cyrillic/Greek about 0.55,
|
|
760
1016
|
Arabic/Hebrew 0.6, Devanagari 0.7, and a combining mark nothing at all —
|
|
761
|
-
breaking between characters for CJK
|
|
762
|
-
between a character and the combining marks that belong
|
|
763
|
-
|
|
1017
|
+
breaking between characters for CJK (a katakana word stays whole) and at spaces
|
|
1018
|
+
otherwise, but never between a character and the combining marks that belong
|
|
1019
|
+
to it (Devanagari matras, Arabic and Hebrew points). Thai (1.16.1) is never
|
|
1020
|
+
broken inside a run: it writes no space inside a phrase and the wrapper has no
|
|
1021
|
+
dictionary, so the break goes where you put a space or a `|`, and a run with
|
|
1022
|
+
none stays long on its own line — `caption.py` counts such lines as `overlong`
|
|
1023
|
+
and says so. A cue that would need more than `--max-lines` (default 2) is split
|
|
764
1024
|
into consecutive cues sharing its time; a cue shorter than `--min-duration`
|
|
765
1025
|
(default 1.0 s) is held longer, never past the next cue's start; `--offset
|
|
766
1026
|
TIME` shifts every cue (seconds, `mm:ss`, `hh:mm:ss.ms` or `hh:mm:ss:ff`, a
|