ffmpeg-skill 1.11.1 → 1.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +11 -9
- package/SKILL.md +27 -21
- package/docs/contract.md +43 -9
- package/package.json +1 -1
- package/references/gotchas.md +57 -7
- package/references/scripts.md +119 -10
- package/scripts/_common.py +378 -0
- package/scripts/_contract.py +60 -8
- package/scripts/audio.py +99 -5
- package/scripts/caption.py +449 -16
- package/scripts/check.py +15 -0
- package/scripts/graphics.py +18 -3
- package/scripts/loudness.py +3 -0
- package/scripts/overlay.py +9 -1
- package/scripts/render.py +69 -15
package/README.md
CHANGED
|
@@ -196,15 +196,15 @@ These are the rules the skill file gives the agent and the code enforces. Togeth
|
|
|
196
196
|
|
|
197
197
|
| Tool | What it does |
|
|
198
198
|
|---|---|
|
|
199
|
-
| `audio.py` | Voice clean-up chain, FFT denoise, typed compressor / limiter / gate, music bed with sidechain ducking, fades, 5.1 → stereo, track replacement, extraction (`-o out.wav`), `--audio-stream N` |
|
|
199
|
+
| `audio.py` | Voice clean-up chain at three strengths (`--voice light\|medium\|strong`), FFT denoise, typed compressor / limiter / gate, music bed with sidechain ducking (`--duck-amount/-threshold/-attack/-release`), a never-ducked effects bed (`--effects`), `--stereo-widen`, fades, 5.1 → stereo, track replacement, extraction (`-o out.wav`), `--audio-stream N` |
|
|
200
200
|
| `sync.py` | Offset between two recordings by audio cross-correlation (1 ms, pure Python), clock-drift correction; aligned video or audio out (audio-to-audio only — no lip-sync/face detection) |
|
|
201
|
-
| `loudness.py` | Two-pass EBU R128 `loudnorm` to −14 LUFS / −1 dBTP or any target, video stream-copied; the written file is measured again and re-encoded until it meets `--tp` (lossy encoders overshoot); `--measure-only` |
|
|
201
|
+
| `loudness.py` | Two-pass EBU R128 `loudnorm` to −14 LUFS / −1 dBTP or any target (`--lra` for the range), video stream-copied; the written file is measured again and re-encoded until it meets `--tp` (lossy encoders overshoot); `--measure-only` |
|
|
202
202
|
|
|
203
203
|
**Picture**
|
|
204
204
|
|
|
205
205
|
| Tool | What it does |
|
|
206
206
|
|---|---|
|
|
207
|
-
| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; animated and word-by-word karaoke timed to the speech energy; optional local transcription |
|
|
207
|
+
| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area by measured width with `--max-lines`/`--min-duration`/`--offset`; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; optional local transcription |
|
|
208
208
|
| `overlay.py` | Logos, watermarks and titles with position, time range, opacity, fades; `--video` for picture-in-picture, `--chromakey` for green-screen compositing |
|
|
209
209
|
| `graphics.py` | Lower-thirds, title cards, chapter chips, progress bars, countdowns, corner bugs drawn by FFmpeg from a brand kit |
|
|
210
210
|
| `color.py` | HDR10 / HLG / Dolby Vision → SDR BT.709 tone mapping, DV layer stripping, 3D LUT (.cube), colour-tag rewriting, typed primary correction (exposure/contrast/saturation/gamma/white balance/lift-gain/levels/curves) |
|
|
@@ -215,14 +215,14 @@ These are the rules the skill file gives the agent and the code enforces. Togeth
|
|
|
215
215
|
|---|---|
|
|
216
216
|
| `export.py` | Presets `youtube`, `youtube4k`, `reels`, `x`, `prores`, `h265`, `gif`, all tagged BT.709; `--normalize` meets the platform's loudness in the same call (`render.py` turns it on by default for platform presets) |
|
|
217
217
|
| `proxy.py` | Small, low-bitrate proxy for downstream AI analysis/preview/editing decisions — resize by `--width`/`--scale`, proxy-grade `--crf` (deprecated alias of `--quality`), `--fps`, `--no-audio`; not a delivery preset |
|
|
218
|
-
| `check.py` | PASS / WARN / FAIL against YouTube, Shorts, Reels, TikTok, X, LinkedIn, broadcast and podcast specs, with the fix for each failure and a `format` / `judgement` kind per row |
|
|
218
|
+
| `check.py` | PASS / WARN / FAIL against YouTube, Shorts, Reels, TikTok, X, LinkedIn, broadcast and podcast specs (podcast also reports chapter markers and channel count), with the fix for each failure and a `format` / `judgement` kind per row |
|
|
219
219
|
| `report.py` | Single-file HTML delivery report: before/after sheets, media facts, loudness, compliance, the commands run |
|
|
220
220
|
|
|
221
221
|
**Orchestration**
|
|
222
222
|
|
|
223
223
|
| Tool | What it does |
|
|
224
224
|
|---|---|
|
|
225
|
-
| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays, music, loudness, export, check); `--init`, `--dry-run`, `--stop-after` |
|
|
225
|
+
| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after` |
|
|
226
226
|
| `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch` |
|
|
227
227
|
| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list |
|
|
228
228
|
| `verify.py` | Run the toolchain on real device files and report PASS / FAIL per step |
|
|
@@ -237,7 +237,7 @@ WAV, FLAC, MP3, M4A/AAC, OGG and Opus go through `probe`, `cut`, `join`, `silenc
|
|
|
237
237
|
- **Join.** `join.py intro.wav episode.m4a outro.wav -o full.flac` resamples every clip to one rate and channel layout and crossfades them (`--transition none` for a butt join). Audio and video inputs cannot be mixed in one join.
|
|
238
238
|
- **Sample-accurate trims.** `cut.py talk.wav --start 1.2345 --end 2.3456 --accurate` trims at the sample; the JSON reports `precision` (`packet` for a stream copy, `sample` for PCM / FLAC, `codec_frame` when a lossy encoder frames the audio again, `frame` for video) and the measured `duration_error_ms`. A `.wav` never receives compressed packets.
|
|
239
239
|
- **Typed dynamics.** `audio.py --compress --comp-threshold -20 --comp-ratio 4`, `--limit --limit-ceiling -1`, `--gate --gate-threshold -45`. Each flag is one documented option of FFmpeg's `acompressor`, `alimiter` or `agate`, range-checked before ffmpeg runs; no filter string is accepted from the caller.
|
|
240
|
-
- **Loudness.** `loudness.py talk.wav -I -16 --tp -1.5 -o talk.m4a` for podcast levels; `check.py talk.m4a --platform podcast` measures LUFS and true peak.
|
|
240
|
+
- **Loudness.** `loudness.py talk.wav -I -16 --tp -1.5 -o talk.m4a` for podcast levels; `check.py talk.m4a --platform podcast` measures LUFS and true peak and reports the chapter markers and channel count.
|
|
241
241
|
|
|
242
242
|
Picture tools (`fit`, `caption`, `overlay`, `graphics`, `color`, `export`, `scenes`, `look`) refuse an audio file with "input has no video stream" instead of inventing a picture.
|
|
243
243
|
|
|
@@ -319,7 +319,7 @@ npx ffmpeg-skill doctor # human-readable
|
|
|
319
319
|
npx ffmpeg-skill doctor --json # available / missing / missing_optional / unknown / detection / errors / tools / gpu_encoders
|
|
320
320
|
```
|
|
321
321
|
|
|
322
|
-
`doctor` reads `ffmpeg -encoders`, `-filters` and `-bsfs` and resolves every capability the contract declares against this machine's build. Three states per capability: `available`, `missing`, `unknown`. Exit 0 when everything required is available, 1 when something required is missing, 2 when nothing is proven missing but a required capability is unknown. With detection on (the default), `contract --json` carries the same lists under `capabilities`. `doctor --json`'s `tools` field folds that down to one answer per tool — `{"caption": {"usable": "no", "missing": ["filter:subtitles"], "fix": "..."}, ...}` — so "is `doctor` overall `ok`" and "can I run `caption.py` on this machine" are answered separately: a plain Homebrew `ffmpeg` is `ok` for tools that don't need `subtitles`/`drawtext`/`zscale`, while `caption`'s own `usable` is `"no"`.
|
|
322
|
+
`doctor` reads `ffmpeg -encoders`, `-filters` and `-bsfs` and resolves every capability the contract declares against this machine's build. Three states per capability: `available`, `missing`, `unknown`. Exit 0 when everything required is available, 1 when something required is missing, 2 when nothing is proven missing but a required capability is unknown. With detection on (the default), `contract --json` carries the same lists under `capabilities`. `doctor` also reports fonts: the default drawtext family, and `fonts.scripts` — one `available`/`missing`/`unknown` per writing system (ja, zh, ko, ar, he, hi, th, ru, el) with the file it would use — so "can this machine render Korean captions" is answered before the job, not after. `doctor --json`'s `tools` field folds that down to one answer per tool — `{"caption": {"usable": "no", "missing": ["filter:subtitles"], "fix": "..."}, ...}` — so "is `doctor` overall `ok`" and "can I run `caption.py` on this machine" are answered separately: a plain Homebrew `ffmpeg` is `ok` for tools that don't need `subtitles`/`drawtext`/`zscale`, while `caption`'s own `usable` is `"no"`.
|
|
323
323
|
|
|
324
324
|
`doctor --json`'s `gpu_encoders` reports which GPU-backed encoders (`nvenc`, `videotoolbox`, `qsv`, `vaapi`, `amf`) this ffmpeg *build* was compiled with — read from `-encoders` alone, so it proves the capability shipped, not that the GPU/driver on this machine will actually accept a job (that needs a real encode, which `doctor`'s introspection never runs). No tool here uses one yet — every tool still assumes CPU x264/x265 — so this is purely informational and never affects `ok` or any tool's `usable`. GPU-accelerated encoding stays deliberately off the roadmap until there's a real-hardware-verified design for it (build-presence alone is not proof a job will succeed) — not a promised feature, just an honest "not yet, and not without proof it actually works."
|
|
325
325
|
|
|
@@ -333,7 +333,7 @@ The short list for humans. The agent-facing version, with the reasoning, is the
|
|
|
333
333
|
- **Loudness targets.** −14 LUFS / −1 dBTP for YouTube and social platforms (the `loudness.py` default), `-I -16 --tp -1.5` for podcasts, `-I -23` for broadcast. A clip measured at −40 LUFS or below is room tone, not content; raising it raises the noise. Check true peak as well as LUFS: `check.py file --platform podcast` measures both.
|
|
334
334
|
- **Frame changes first, text second.** Captions and overlays burned before a crop or resize end up off-frame. Reframe, then caption.
|
|
335
335
|
- **Cropping 16:9 to 9:16 discards 70 % of the width.** `fit.py --fit crop` centres by default; pass `--crop-x`/`--crop-y` toward the subject, or pad with `--fit pad --pad-fill blur`. Look at the contact sheet before deciding.
|
|
336
|
-
- **Non-Latin
|
|
336
|
+
- **Non-Latin text picks a font by script (1.12).** Japanese, Chinese, Korean, Arabic, Hebrew, Devanagari, Thai, Cyrillic and Greek cues, titles and overlays resolve a font file that covers them automatically, and a machine with no such font fails the job (`kind: input`) instead of rendering boxes. `doctor --json`'s `fonts.scripts` says which languages this machine can render; `--lang ja|ko` disambiguates Han-only text; an explicit `--font`/`--font-file` is always kept.
|
|
337
337
|
- **Silence detection finds nothing?** The default threshold is −35 dBFS. The tool prints a hint with the track's measured level; raise the threshold (`silence.py --threshold -25`) or shorten `--min-silence`.
|
|
338
338
|
- **Sync results carry a confidence.** Below 0.3, or an offset near the edge of the analysis window, is probably wrong: enlarge `--analyze-seconds` or find a clap. Recordings over ten minutes from separate devices need `sync.py --fix-drift`.
|
|
339
339
|
- **Outputs are never overwritten silently.** An existing output path is warned about today and refused from 2.0; set `FFMPEG_SKILL_NO_OVERWRITE=1` (the recommended agent setting) to get the refusal now and pass `--overwrite` where a replacement is intended.
|
|
@@ -363,7 +363,9 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th
|
|
|
363
363
|
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
|
|
364
364
|
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
|
|
365
365
|
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
|
|
366
|
-
| **
|
|
366
|
+
| **50 / 50** | 1.12.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 50 prompts with two each in Chinese, Korean, Spanish, Portuguese, French, German and Arabic: routing 50/50, honest refusals and failures 50/50 with 0 false successes and 0 raw ffmpeg calls, report format 50/50, user's language 50/50 across nine languages, visual check 13/13, trigger set 29/29, Opus quality mean 4.83. Every non-Latin caption and lower-third picked a covering font by itself and rendered real glyphs (Arabic shaped and right-to-left); tokens per run unchanged at 72.3k. Details in `evals/results/iteration-13.json` |
|
|
367
|
+
| **36 / 36** | 1.11.1 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22, Opus quality mean 4.75. The 1.11.1 wording did what it said (`doctor` before a job 23 of 36 runs → 0, `--json-brief` 4 → 23) and tokens per run stayed flat at 71.8k, because about 64k of every run is the host's own context; the token-diet theme closes here. Details in `evals/results/iteration-12.json` |
|
|
368
|
+
| **36 / 36** | 1.11.0 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22. First iteration to measure tokens per run: mean 72.2k against 68.7k at 1.10.0, mostly a fixed per-run floor the skill does not control (a refusal run that only reads SKILL.md costs about 64k), plus `doctor` on 23 of 36 runs; 1.11.1 rewords step 0 and iteration 12 re-measures. Details in `evals/results/iteration-11.json` |
|
|
367
369
|
| **108 / 108** | 1.10.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + focused Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, report format 108/108 by both graders (the harness now names the five labels), user's language 105/108 (every Japanese request in Japanese; 3 English requests drifted to Spanish or Portuguese), visual check 21/24, trigger set 22/22; real-device corpus 101/101 steps PASS. Details in `evals/results/iteration-10.json` |
|
|
368
370
|
| **108 / 108** | 1.9.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + independent Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 23/26, user's language 105/108 (every Japanese request answered in Japanese; 3 English requests drifted to Spanish), report format 108/108 by regex (65/108 by the stricter grader, which now counts any missing label), trigger set 22/22; 12 of 15 platform jobs were one encode and `render.py` rendered once in 3/3 (was 1/3). The 1.9.0 time grammar was not used by any agent. Details in `evals/results/iteration-9.json` |
|
|
369
371
|
| **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` |
|
package/SKILL.md
CHANGED
|
@@ -5,36 +5,36 @@ description: 'Edit video and audio with local FFmpeg from natural-language reque
|
|
|
5
5
|
|
|
6
6
|
# ffmpeg-skill
|
|
7
7
|
|
|
8
|
-
Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`. This file is enough to do a job: the table below routes the request
|
|
8
|
+
Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`. This file is enough to do a job: the table below routes the request and `--help` on the one script you are about to run is the cheapest full flag list. The reference files cost as much to read as this file does, so open one only when it answers a question you actually have: `references/scripts.md` (every flag of all 42 scripts, for comparing tools), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end of this file).
|
|
9
9
|
|
|
10
|
-
Shared flags, on every script: `--dry-run`; `--json` (output path, a probe of the output, the commands run); `--json-brief` (that document trimmed to status/output/verified, a compact `summary` and the command count —
|
|
10
|
+
Shared flags, on every script: `--dry-run`; `--json` (output path, a probe of the output, the commands run); `--json-brief` (that document trimmed to status/output/verified, a compact `summary` and the command count — prefer it on every writing step); `--fast` (preview quality); `--progress`; `--timeout SECONDS` (`kind: timeout`, default 1800); `--overwrite` (consent to replace an existing output — without it the tool warns today, refuses from 2.0); `--plan FILE` (the dry run as a plan document `render.py FILE` executes later, refusing if an input changed: "plan → confirm → execute" in one round trip). Every re-encoding tool also takes `--codec h264|hevc|av1|prores` and `--quality N` (CRF scale, replaces the deprecated `--crf`): unset, SDR is x264 and HDR is x265 Main10; `prores` needs an explicit `-o NAME.mov`, `h264` refuses an HDR source (`color.py --to-sdr` first).
|
|
11
11
|
|
|
12
12
|
Writing tools run nothing under `--dry-run`; `probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness` and `stabilize` may still run ffmpeg/ffprobe to measure or analyse — they just don't write their final artifact (nor side files such as `--edl`, `--sheet` or a generated `.ass`); `verify` accepts the flag but ignores it. Exact per-tool semantics: `contract --json`'s `dry_run` field (or `docs/contract.md`).
|
|
13
13
|
|
|
14
14
|
## Workflow (always follow this order)
|
|
15
15
|
|
|
16
|
-
0. **Environment, only on failure.** Do not start a job with `doctor`: on a working machine it
|
|
16
|
+
0. **Environment, only on failure.** Do not start a job with `doctor`: on a working machine it tells you nothing the job needs, and on a broken one the script fails on its own with `kind: missing_tool` (no ffmpeg) or an ffmpeg error naming the filter or encoder (`No such filter: 'subtitles'`). Run `python3 <skill-dir>/scripts/_contract.py doctor` (also `npx ffmpeg-skill doctor`; there is no doctor.py) after such a failure, or when the user asks what the machine can do. Read `ok` and the tool's `usable`; if `usable` isn't `yes`, report the missing capability (usually `libass`, `zscale` or an encoder) instead of discovering it through a runtime failure. `contract --json`'s full tool schema is for a *planning* agent choosing a tool from an abstract goal, not for this workflow.
|
|
17
17
|
1. **Probe what you must plan from.** Run `probe.py` on each input you plan the edit from — duration, fps, resolution, codecs, channels, `variable_frame_rate_suspected` — and whenever the user asks a question about a file. You do not need a separate probe before every edit: every writing tool's `--json` already carries its input and a probe of the output it wrote. Plan from real numbers, never assumptions.
|
|
18
18
|
2. **Prefer lossless.** If the request can be met without re-encoding (plain cuts on keyframes, remuxing, audio-only changes), do not re-encode. `cut.py` and `loudness.py` stream-copy video by default; pass `--accurate` to `cut.py` only for frame-exact cuts.
|
|
19
19
|
3. **Plan with `--dry-run --json`, then execute.** Trust `--json`, not a dry run's human-readable summary line, for any number after the plan (dimensions there can be a placeholder, not a computed preview — `docs/contract.md`). Use it to confirm a plan before long encodes and to report exact facts. `--fast` is preview quality (x264 veryfast), `--progress` prints percent/ETA on stderr. Never point `-o` at a file you did not create in this job unless the user asked for it to be replaced; pass `--overwrite` only then.
|
|
20
20
|
4. **Chain in a sensible order.** Colour (HDR→SDR / LUT) → cut → join → silence → fit → caption/overlay → sync → audio → loudness → export. Frame changes (fit/crop) before captions and overlays, so text is sized for the final frame. Re-encode as few times as possible: intermediates at CRF 18 (the default), `export.py` only for the last step. **Three or more steps: use `render.py` with a project.json** — one call, one JSON, one place for the user to change a number — rather than hand-chaining tools.
|
|
21
|
-
5. **Check the deliverable.** Before reporting, run `check.py OUTPUT --platform X` for the destination the user named. Each row is `format` or `judgement`. Format rows (codec, pixel format, size, true peak, colour tags, VFR) are safe to fix mechanically. Judgement rows change the content: duration (cut loses material
|
|
22
|
-
6. **Verify the output.** Confirm duration, resolution, fps and audio match the request — from the writing tool's own `--json`/`--json-brief` probe, or `probe.py`
|
|
21
|
+
5. **Check the deliverable.** Before reporting, run `check.py OUTPUT --platform X` for the destination the user named. Each row is `format` or `judgement`. Format rows (codec, pixel format, size, true peak, colour tags, VFR) are safe to fix mechanically. Judgement rows change the content: duration (cut loses material), aspect (crop loses edges), fps (drops motion), loudness (ambience must not be boosted) — fix those only when the request already implies the answer, otherwise state the choice and its cost in one line. Mention WARNs; do not chase them.
|
|
22
|
+
6. **Verify the output.** Confirm duration, resolution, fps and audio match the request — from the writing tool's own `--json`/`--json-brief` probe, or `probe.py` — and report those numbers ("final.mp4: 59.98 s, 1080x1920, 30 fps, AAC stereo"). A step is done only when the script exited 0 and the output probes as expected: a non-zero exit, a missing or empty file, or a probe that contradicts the request is a failure, and the report says so with the script's error message.
|
|
23
23
|
7. **Keep the user's originals.** Never overwrite the source; write new files next to the input or where the user asked. Set `FFMPEG_SKILL_NO_OVERWRITE=1` in the environment you run these scripts in: an existing output path is then refused (`kind: input`) instead of warned about, and `--overwrite` stays the one way to say "yes, replace it". It is the recommended agent setting — an agent picking output names cannot see which files the user already cares about — and it is what 2.0 does by default.
|
|
24
|
-
8. **Look at the picture.** Whenever the picture changed (captions, overlays, graphics, crop/pad, resize, colour, transitions, a `join.py` that scaled or padded a clip to the first clip's frame, a `color.py --to-sdr` that tone-maps an HDR source) run `look.py OUTPUT --tiles 3x2` (or `--at T` for one frame) and view the PNG; the full 4x3 sheet is for a job about layout across the whole clip. The job is not finished until the report's `Look:` line names that PNG — a probe cannot see a caption sitting on someone's face. Audio-only jobs
|
|
24
|
+
8. **Look at the picture.** Whenever the picture changed (captions, overlays, graphics, crop/pad, resize, colour, transitions, a `join.py` that scaled or padded a clip to the first clip's frame, a `color.py --to-sdr` that tone-maps an HDR source) run `look.py OUTPUT --tiles 3x2` (or `--at T` for one frame) and view the PNG; the full 4x3 sheet is for a job about layout across the whole clip. The job is not finished until the report's `Look:` line names that PNG — a probe cannot see a caption sitting on someone's face. Audio-only jobs write `Look: not needed`; there is no picture. What to look for splits like `check.py`'s rows in step 5:
|
|
25
25
|
- **Mechanical (this skill's own job to verify and report):** the specified text/logo is present at the specified position, subtitles/text appear at the specified timestamps, dimensions are even. Letterboxing/pillarboxing from `fit.py --fit pad` is the *correct* result of that mode, never a defect to flag.
|
|
26
26
|
- **Judgement (report it, don't silently pass or fail):** whether a subject or face is cut off, whether text sits over a face, whether colours look washed out, whether a transition lands. These need deciding what the subject *is*, which belongs to the calling agent (see "What this skill does and does not decide") — say what you see in one line and let them judge it.
|
|
27
27
|
With no vision capability, write `Look: PATH (pixels not inspected; agent has no image view)` — never claim a picture was inspected when it wasn't, and don't stall waiting for a capability that isn't there.
|
|
28
28
|
|
|
29
29
|
## Before you run anything: what to ask, what to assume
|
|
30
30
|
|
|
31
|
-
Ask one short question only when the answer changes the output materially and the request does not imply it. When several things are open at once (a vague "make it for social media" leaves destination, aspect
|
|
31
|
+
Ask one short question only when the answer changes the output materially and the request does not imply it. When several things are open at once (a vague "make it for social media" leaves destination, aspect, length and captions open), don't ask them one per turn: propose one bundle with your defaults and let the user change any part ("Reels: 9:16 with padding, trimmed to 60 s, -14 LUFS, no captions — OK, or change something?"). One question, one answer, then the run. Never ask for what `probe.py` can tell you.
|
|
32
32
|
|
|
33
33
|
- **Destination** decides aspect, length limit, loudness and codec; "for Reels" answers all four. No destination named and a plain cut/caption: keep the source format and say so. If the user says "export", "post" or "deliver", ask where.
|
|
34
34
|
- **Duration** ("make it 60 s") without a method: speed up for ≤1.5× changes, trim otherwise, and say which you chose. Ask when the content is a talk (trimming loses words) and the change is large.
|
|
35
35
|
- **Captions** without a text source: `--transcribe` if a local whisper exists, otherwise ask for the text or a timed file; never invent dialogue.
|
|
36
36
|
- **Fonts and brand**: if the user mentions a brand, colours or "our font", ask for or create `brand.json` once and reuse it.
|
|
37
|
-
- **CJK / non-Latin text**:
|
|
37
|
+
- **CJK / non-Latin text**: let the tool pick the font by script (`--font` turns that off); `--lang ja|ko` for Han-only text. `doctor --json` `.fonts.scripts` says what renders here. Tofu is a failed job, not a style.
|
|
38
38
|
- **Crop position** for `--fit crop`: centre by default, but when the request or the source names an off-centre subject ("keep the product on the right", "don't cut off my hands", someone visibly off-centre in the sheet) use `--crop-x`/`--crop-y` (0=left/top, 1=right/bottom) instead of a silent centre guess. Ask which edge to keep when the sheet shows the subject near an edge and the request doesn't say.
|
|
39
39
|
- Anything else (transition type, caption style): pick the conventional default, say what you picked, offer the alternative in one line.
|
|
40
40
|
|
|
@@ -51,13 +51,13 @@ This skill cuts, joins, measures, syncs, exports and checks files — it execute
|
|
|
51
51
|
|
|
52
52
|
The line: same input + same explicit parameters always producing the same verifiable output belongs here; anything that depends on taste, content understanding or what looks or sounds good belongs to whoever makes that judgement. This skill executes parameters it is given, never infers them from what something looks or sounds like.
|
|
53
53
|
|
|
54
|
-
If a request needs an FFmpeg feature none of the 42 scripts expose, say so and name the closest built-in option (`--dry-run` to show what would run, or a documented limitation) — never fall back to guessing a raw `ffmpeg`/`ffprobe` invocation or a hand-built filter graph outside `scripts/*.py`. A raw command bypasses every guarantee this skill makes (no shell, typed arguments, verification afterwards)
|
|
54
|
+
If a request needs an FFmpeg feature none of the 42 scripts expose, say so and name the closest built-in option (`--dry-run` to show what would run, or a documented limitation) — never fall back to guessing a raw `ffmpeg`/`ffprobe` invocation or a hand-built filter graph outside `scripts/*.py`. A raw command bypasses every guarantee this skill makes (no shell, typed arguments, verification afterwards), so it is never the fallback when a script's flag doesn't cover something.
|
|
55
55
|
|
|
56
56
|
## Request → script
|
|
57
57
|
|
|
58
58
|
This table and `doctor --json`'s `tools` list are the source of truth for what exists: name only a script you have seen in one of them, never a plausible-sounding one (there is no `doctor.py`, no `trim.py`, no `subtitle.py`).
|
|
59
59
|
|
|
60
|
-
Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`, and the times in cue and chapter files — take seconds, `mm:ss(.fff)`, `hh:mm:ss(.fff)` or four-part SMPTE `hh:mm:ss:ff`, with `@fps` naming the rate (`00:01:02:15@29.97`); tolerance-style flags that are a length rather than a point in time (`--min-silence`, `--margin`, `--min-keep`, `--fade`) are plain seconds. Use the timecode forms when the user pastes an editor's
|
|
60
|
+
Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`, and the times in cue and chapter files — take seconds, `mm:ss(.fff)`, `hh:mm:ss(.fff)` or four-part SMPTE `hh:mm:ss:ff`, with `@fps` naming the rate (`00:01:02:15@29.97`); tolerance-style flags that are a length rather than a point in time (`--min-silence`, `--margin`, `--min-keep`, `--fade`) are plain seconds. Use the timecode forms when the user pastes an editor's or NLE cue sheet, so nothing is converted by hand.
|
|
61
61
|
|
|
62
62
|
| User says | Do |
|
|
63
63
|
|-----------|----|
|
|
@@ -87,7 +87,7 @@ Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offse
|
|
|
87
87
|
| "speed up here, slow-mo there" (known segments) | `speedramp.py action.mp4 --segment 0-3:1.0 --segment 3-4:0.25 --segment 4-8:2.0` |
|
|
88
88
|
| "loop this clip to fill 30 seconds" | `loop.py bg_loop.mp4 --duration 30` |
|
|
89
89
|
| "cut to the product shot 0:12-0:16", "B-roll over this bit" | `broll.py talk.mp4 --insert product.mp4 --at 12 --end 16` (repeat `--insert/--at`; `--audio b\|mix`) |
|
|
90
|
-
| "add chapters", "chapter markers for YouTube" | `metadata.py episode.mp4 --chapters chapters.txt` (`TIME TITLE` per line; streams copied) |
|
|
90
|
+
| "add chapters", "chapter markers for YouTube" | `metadata.py episode.mp4 --chapters chapters.txt` (`TIME TITLE` per line; streams copied; a `render.py` project spells it `"chapters"`) |
|
|
91
91
|
| "set the title / artist / comment" | `metadata.py episode.mp4 --title "Episode 12" --artist "Studio"` |
|
|
92
92
|
| "put these videos in a 4x2 grid" | `grid.py t1.mp4 ... t8.mp4 --cols 4 --rows 2` |
|
|
93
93
|
| "add subtitles from this SRT", "burn in captions" | `caption.py input.mp4 --srt subs.srt` |
|
|
@@ -98,7 +98,7 @@ Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offse
|
|
|
98
98
|
| "webcam clip in the corner", "picture-in-picture" | `overlay.py input.mp4 --video webcam.mp4 --position bottom-right --scale 480` |
|
|
99
99
|
| "remove the green screen" | `overlay.py bg.mp4 --video greenscreen.mp4 --chromakey 0x00ff00` |
|
|
100
100
|
| "sync the lav mic", "line up two cameras" | `sync.py camera.mp4 mic.wav --replace-audio` / `sync.py camA.mp4 camB.mp4 --trim-second` |
|
|
101
|
-
| "fix the audio levels", "normalise to -14 LUFS" | `loudness.py input.mp4` (`-I -16 --tp -1.5` podcast, `-I -23` broadcast) |
|
|
101
|
+
| "fix the audio levels", "normalise to -14 LUFS" | `loudness.py input.mp4` (`-I -16 --tp -1.5` podcast, `-I -23` broadcast; `--lra N` for the range) |
|
|
102
102
|
| "cut this and make it HEVC / AV1 / ProRes" (output codec named) | `cut.py input.mp4 --start 0:10 --end 0:40 --codec hevc` (`--codec`/`--quality` on any re-encoding tool; ProRes needs `-o NAME.mov`) |
|
|
103
103
|
| "export for YouTube / Reels / X", "a ProRes master" | `export.py input.mp4 --preset youtube\|reels\|x\|prores\|h265` (`--normalize` hits the loudness spec in the same call) |
|
|
104
104
|
| "make a GIF preview" | `export.py input.mp4 --preset gif` |
|
|
@@ -109,6 +109,7 @@ Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offse
|
|
|
109
109
|
| "what would you run?", "don't render yet" | any script with `--dry-run` |
|
|
110
110
|
| "a 60 s highlight from this hour" | `scenes.py long.mp4 --highlights 6 --target 60 --edl picks.txt` → `cut.py --segments` |
|
|
111
111
|
| "is this OK to upload?" | `check.py final.mp4 --platform reels` |
|
|
112
|
+
| "a podcast episode with chapters" | `loudness.py ep.wav -I -16 --tp -1.5` → `metadata.py ep.m4a --chapters chapters.txt` → `check.py ep.m4a --platform podcast` (chapters and channels rows) |
|
|
112
113
|
| "several changes to the same edit", 3+ steps | `render.py --init project.json`, edit, `render.py project.json` |
|
|
113
114
|
| "a lower third with my name", "countdown intro", "progress bar" | `graphics.py input.mp4 --template lower-third --name "..." --title "..." --start 2 --end 8` |
|
|
114
115
|
| "use our brand fonts/colours/logo" | `--brand brand.json` on caption/overlay/graphics, or `"brand"` in project.json |
|
|
@@ -124,8 +125,10 @@ Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offse
|
|
|
124
125
|
| "apply this LUT", "convert the S-Log footage" | `color.py input.mp4 --lut grade.cube [--lut-strength 0.7]` |
|
|
125
126
|
| "the colours are tagged wrong" | `color.py input.mp4 --retag bt709` (stream copy; re-encodes only if the copy can't carry it — see `reencoded`) |
|
|
126
127
|
| "brighten it / punch up contrast / fix white balance" | `color.py input.mp4 --correct --exposure 0.3 --contrast 1.1 --saturation 1.05 --temperature 5600 --tint -0.05` |
|
|
127
|
-
| "clean up the audio", "remove the hiss" | `audio.py input.mp4 --voice` (speech) or `--denoise` |
|
|
128
|
-
| "add background music under the talking" | `audio.py input.mp4 --music bed.mp3 --duck --fade-out 3` |
|
|
128
|
+
| "clean up the audio", "remove the hiss" | `audio.py input.mp4 --voice` (speech; `--voice light\|medium\|strong`) or `--denoise` |
|
|
129
|
+
| "add background music under the talking" | `audio.py input.mp4 --music bed.mp3 --duck --fade-out 3` (`--effects sfx.wav` adds a third bed, never ducked; project levels: `audio.stems`) |
|
|
130
|
+
| "make the music duck harder / come back faster" | add `--duck-amount 18 --duck-threshold -30 --duck-release 250` (`--duck-attack` too) |
|
|
131
|
+
| "the mix sounds narrow", "wider stereo" | `audio.py band.wav --stereo-widen 0.5` (needs a real stereo source; 5.1 needs `--downmix`) |
|
|
129
132
|
| "convert the 5.1 to stereo" | `audio.py input.mov --downmix` |
|
|
130
133
|
| "swap in the narration track" | `audio.py input.mp4 --replace narration.wav` |
|
|
131
134
|
| "pull the audio out", "give me the sound as WAV" | `audio.py input.mp4 -o input.wav` (an audio extension drops the picture; `--audio-stream 1` picks a track) |
|
|
@@ -137,12 +140,12 @@ Timestamp flags — `--start`, `--end`, `--at`, `--from`, `--duration`, `--offse
|
|
|
137
140
|
|
|
138
141
|
## Audio-only files
|
|
139
142
|
|
|
140
|
-
Audio is a first-class input: `probe.py`, `cut.py`, `silence.py`, `loudness.py`, `audio.py`, `sync.py` and `check.py --platform podcast` take WAV, FLAC, MP3, M4A/AAC, OGG and Opus, and the output extension picks the format. `Look: not needed` in the report; `Check:` still applies
|
|
143
|
+
Audio is a first-class input: `probe.py`, `cut.py`, `silence.py`, `loudness.py`, `audio.py`, `sync.py` and `check.py --platform podcast` take WAV, FLAC, MP3, M4A/AAC, OGG and Opus, and the output extension picks the format. `Look: not needed` in the report; `Check:` still applies. Scripts that need a picture (`fit`, `caption`, `overlay`, `graphics`, `color`, `export`, `scenes`, `look`) refuse an audio file with "input has no video stream" — say so instead of forcing a video wrapper. The same commands work with `talk.wav` in place of `talk.mp4`; audio recipes, packet vs sample precision, joining and extracting one track: `references/gotchas.md#audio-only-files`.
|
|
141
144
|
|
|
142
145
|
|
|
143
146
|
## Report format
|
|
144
147
|
|
|
145
|
-
Reply in the language the request itself is written in — the
|
|
148
|
+
Reply in the language the request itself is written in — the user's own sentences, not a language the request talks about (a request for subtitles in another language is still answered in the language it was written in) and not the language of a tool's error text or file names. Any language works the same way. Keep the field labels (`Done:`, `Steps:`, `Check:`, `Look:`, `Notes:`) in English: they read like log fields and stay recognisable across languages. Everything around them — the sentences, any question, any explanation of a judgement call — is in the user's language. Never default to English because the tool names and flags are English, and never drift because the job was short or the report is a failure: a one-line "file does not exist" is written in the request's language too. A mid-conversation switch follows the user's latest message. This holds for a one-command job: English `Done:`/`Steps:` sentences with one word of the user's language in `Notes:` is an English report — the descriptions are in the user's language even when the values are technical.
|
|
146
149
|
|
|
147
150
|
Finish every job with this shape (numbers from a tool's `--json` or `probe.py`/`check.py`, not memory):
|
|
148
151
|
|
|
@@ -154,7 +157,7 @@ Look: final_sheet.png (captions inside the safe area, logo top-right)
|
|
|
154
157
|
Notes: source was VFR, conformed to 30 fps; audio was mono, made stereo
|
|
155
158
|
```
|
|
156
159
|
|
|
157
|
-
The same five lines for a Japanese request
|
|
160
|
+
The same five lines for a Japanese request:
|
|
158
161
|
|
|
159
162
|
```
|
|
160
163
|
Done: final.mp4 — 59.98 秒、1080x1920、30 fps、H.264、AAC ステレオ、-14.1 LUFS
|
|
@@ -164,7 +167,9 @@ Look: final_sheet.png(字幕はセーフエリア内、ロゴは右上)
|
|
|
164
167
|
Notes: 元は VFR だったので 30 fps に揃えた。音声はモノラルだったのでステレオにした
|
|
165
168
|
```
|
|
166
169
|
|
|
167
|
-
|
|
170
|
+
Same shape in every other language, labels still English — zh: `Done: final.mp4 — 59.98 秒、1080x1920、30 fps、H.264` / `Steps: 0:12-1:12 剪切 -> 9:16 裁剪 -> 字幕 -> Reels 导出`; ko: `Done: final.mp4 — 59.98초, 1080x1920, 30 fps, H.264` / `Steps: 0:12-1:12 컷 -> 9:16 크롭 -> 자막 -> Reels 내보내기`.
|
|
171
|
+
|
|
172
|
+
Keep it to those five lines plus anything the user must decide. Attach the contact sheet when the edit touched the picture. Never report success without the output probe; never describe a fix you did not run.
|
|
168
173
|
|
|
169
174
|
When a step fails, replace `Done:` with `Failed:` and keep the rest honest:
|
|
170
175
|
|
|
@@ -176,9 +181,9 @@ Look: not needed (nothing written)
|
|
|
176
181
|
Notes: send a valid .cube, or say if you want the clip left as is
|
|
177
182
|
```
|
|
178
183
|
|
|
179
|
-
A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run (usually only probe), `Look: not needed`. Both keep the five labels so a failed report scans like a successful one —
|
|
184
|
+
A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run (usually only probe), `Look: not needed`. Both keep the five labels so a failed report scans like a successful one — the shortest failure still gets all five lines, never prose headings. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
|
|
180
185
|
|
|
181
|
-
Every script prints `{"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool | timeout | verification | interrupted, "message": ...}}` with `--json` and exits non-zero; quote the message,
|
|
186
|
+
Every script prints `{"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool | timeout | verification | interrupted, "message": ...}}` with `--json` and exits non-zero; quote the message, never paraphrase it into a success.
|
|
182
187
|
|
|
183
188
|
## Things that look right but are wrong
|
|
184
189
|
|
|
@@ -190,7 +195,8 @@ One line each, and each line is enough to act on; open the linked `references/go
|
|
|
190
195
|
- VFR phone/screen recordings: re-encodes conform to CFR, `cut.py` switches to `--accurate`; pick the rate with `fit.py --fps` when the average is odd. Details: [#variable-frame-rate](references/gotchas.md#variable-frame-rate)
|
|
191
196
|
- Sync/multicam `confidence` under 0.3 (or a huge offset) is probably wrong — check every camera, and remember these align audio, never lip sync. Details: [#sync-multicam-and-drift](references/gotchas.md#sync-multicam-and-drift)
|
|
192
197
|
- "Normalised" audio can still clip (check true peak), and ambience at -40 LUFS or below must never be raised to a speech target. Details: [#loudness-and-ambience](references/gotchas.md#loudness-and-ambience)
|
|
193
|
-
- Captions burned before a crop/resize land off-frame; burned small then upscaled by `export.py` they come out soft — fit to the delivery size first.
|
|
198
|
+
- Captions burned before a crop/resize land off-frame; burned small then upscaled by `export.py` they come out soft — fit to the delivery size first. Details: [#captions-fonts-and-text-order](references/gotchas.md#captions-fonts-and-text-order)
|
|
199
|
+
- Non-Latin text picks a font by script since 1.12; `doctor --json` `fonts.scripts` says which languages this machine renders; no font = failed job, not tofu. Details: [#fonts-by-script](references/gotchas.md#fonts-by-script)
|
|
194
200
|
- `--fit crop` 16:9 → 9:16 throws away 70 % of the width, 60→30 fps halves the motion, and "60 seconds" by speed or by trim are different answers — say which and why. Details: [#reframing-fps-and-duration](references/gotchas.md#reframing-fps-and-duration)
|
|
195
201
|
- `yuv420p` needs even dimensions and phone rotation tags are honoured, both automatically. Details: [#dimensions-and-rotation](references/gotchas.md#dimensions-and-rotation)
|
|
196
202
|
- `scenes.py --highlights` ranks by loudness (or duration), never by meaning: check the sheet before treating picks as final. Details: [#highlights](references/gotchas.md#highlights)
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`1.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`1.13.0`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -88,19 +88,19 @@ spelling keeps working until 2.0.
|
|
|
88
88
|
|
|
89
89
|
| What 2.0 removes | Since | Replacement | To be ready today |
|
|
90
90
|
|---|---|---|---|
|
|
91
|
-
| The per-tool v1 success keys next to `result_v2` (`output`, `probe`, `commands`, `verified`, `verification` and each tool's own keys at the top level) | 1.
|
|
92
|
-
| `--crf` as an alias of `--quality` on every re-encoding tool that takes `--quality` (`export.py` keeps `--crf`: its preset chooses the encoder) | 1.
|
|
93
|
-
| `json` and `progress` in the MCP `inputSchema` | 1.
|
|
94
|
-
| `hdr` meaning "BT.2020 primaries *or* a PQ/HLG transfer" in `probe` | 1.
|
|
95
|
-
| Overwriting an existing output with only a warning | 1.
|
|
91
|
+
| The per-tool v1 success keys next to `result_v2` (`output`, `probe`, `commands`, `verified`, `verification` and each tool's own keys at the top level) | 1.13.0 | `result_v2`, promoted to the top level in 2.0 | Run with `FFMPEG_SKILL_RESULT_V2=1` and read `result_v2` (`metrics`, `notes`, `details`) instead of the top-level keys |
|
|
92
|
+
| `--crf` as an alias of `--quality` on every re-encoding tool that takes `--quality` (`export.py` keeps `--crf`: its preset chooses the encoder) | 1.13.0 | `--quality N` (the same CRF scale, codec-neutral) | Pass `--quality`; `--crf` warns on stderr and is marked in `--help` |
|
|
93
|
+
| `json` and `progress` in the MCP `inputSchema` | 1.13.0 | nothing: the transport sets them itself | Stop sending them from an MCP client; run the server with `FFMPEG_SKILL_MCP_LEAN=1` to see the 2.0 schema |
|
|
94
|
+
| `hdr` meaning "BT.2020 primaries *or* a PQ/HLG transfer" in `probe` | 1.13.0 | `hdr_signal` (true only for PQ / HLG / Dolby Vision); in 2.0 `hdr` takes that meaning | Key on `hdr_signal` for "is this a real HDR signal" and on `hdr_format` for the `BT.2020 SDR` case |
|
|
95
|
+
| Overwriting an existing output with only a warning | 1.13.0 | `--overwrite` as explicit consent (refused without it from 2.0) | Set `FFMPEG_SKILL_NO_OVERWRITE=1` (the recommended agent setting) and pass `--overwrite` where a replacement is intended |
|
|
96
96
|
|
|
97
97
|
## Skill
|
|
98
98
|
|
|
99
99
|
```json
|
|
100
100
|
{
|
|
101
101
|
"contract_version": "1.0",
|
|
102
|
-
"deprecated": [{"what": "...", "since": "1.
|
|
103
|
-
"skill": {"id": "ffmpeg-skill", "version": "1.
|
|
102
|
+
"deprecated": [{"what": "...", "since": "1.13.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
103
|
+
"skill": {"id": "ffmpeg-skill", "version": "1.13.0", "execution_mode": "local", "kind": "execution",
|
|
104
104
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
105
105
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
106
106
|
"requirements": {"python": ">=3.9 (standard library only)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
|
|
@@ -128,7 +128,7 @@ One entry per tool under `tools`, sorted by id. Tool ids are stable:
|
|
|
128
128
|
| `output_schema` | what `--json` prints on stdout |
|
|
129
129
|
| `supports_dry_run`, `dry_run` | whether `--dry-run` plans without running ffmpeg or writing files |
|
|
130
130
|
| `supports_json` | whether `--json` exists |
|
|
131
|
-
| `supports_json_brief` | whether `--json-brief` exists (1.
|
|
131
|
+
| `supports_json_brief` | whether `--json-brief` exists (1.13.0): the same success document with `probe` replaced by a compact `summary` (`duration_s`, `width`, `height`, `fps`, `vcodec`, `acodec`, `channels`, and `lufs` when the tool measured one), `commands` replaced by the number of commands run, and the per-step `verification` list dropped (its verdict stays in `verified`). Tool-specific keys are unchanged, `--json`'s own output is unchanged, and a failure prints the same failure document either way |
|
|
132
132
|
| `mutates_input` | always `false`: no tool overwrites its input |
|
|
133
133
|
| `produces_artifact` | writes a file (media, PNG, HTML, EDL) |
|
|
134
134
|
| `verification` | `{required, tools}`: which tools to run on the output afterwards |
|
|
@@ -319,6 +319,25 @@ when `fc-match` itself is not on PATH or fails. Like `gpu_encoders`, this is pur
|
|
|
319
319
|
and never affects `ok` or any tool's `usable` — a substituted font is not a broken tool, just a
|
|
320
320
|
typeface the caller didn't ask for.
|
|
321
321
|
|
|
322
|
+
Since 1.12 the same field also carries `scripts`: one entry per writing system the tools detect,
|
|
323
|
+
`{"ja": {"status": "available"|"missing"|"unknown", "file": "/path/to/font.ttc"|null}, "zh": ...,
|
|
324
|
+
"ko": ..., "ar": ..., "he": ..., "hi": ..., "th": ..., "ru": ..., "el": ...}`. It answers "which
|
|
325
|
+
languages can this machine actually render", which no filter or encoder capability asks:
|
|
326
|
+
`available` means `fc-list :lang=<code>` (Linux/macOS) or a known system font file (Windows) covers
|
|
327
|
+
the script, `missing` means fontconfig knows none, `unknown` means there is no working fontconfig to
|
|
328
|
+
ask (no `fc-list` on PATH, or it failed). `caption.py`, `graphics.py` and `overlay.py --text` resolve
|
|
329
|
+
a font by script automatically and fail with `kind: input` rather than render boxes **only for
|
|
330
|
+
`missing`**: `unknown` is not `missing` here any more than anywhere else in this document — the job
|
|
331
|
+
runs with the font as given and one info line says the coverage could not be verified. So a
|
|
332
|
+
`missing` script here is a job that will not run until a font is installed — but, like the default
|
|
333
|
+
font, it never affects `ok` or any tool's
|
|
334
|
+
`usable` (the tool works, this machine just has no glyphs for that language). The plain-text
|
|
335
|
+
`doctor` summarises the whole map on one `fonts:` line, which also carries the default font's
|
|
336
|
+
`detail` in brackets when its status is not `available` and that detail is short enough to keep the
|
|
337
|
+
line to one screen width; a longer explanation, and the per-script files, are `--json` only.
|
|
338
|
+
`--lang`/`--language` (caption, graphics) is the hint that says whether Han-only text is Chinese,
|
|
339
|
+
Japanese or Korean.
|
|
340
|
+
|
|
322
341
|
## Invocation
|
|
323
342
|
|
|
324
343
|
Structured arguments are the canonical way to call a tool, on the CLI or through MCP.
|
|
@@ -335,6 +354,21 @@ runs a shell, evaluates strings, or executes anything other than the named scrip
|
|
|
335
354
|
|
|
336
355
|
## JSON output
|
|
337
356
|
|
|
357
|
+
Per-tool keys added in 1.13: `audio` (`audio.py`) reports the mix it built — the
|
|
358
|
+
`--voice` level, `stereo_widen`, whether an `--effects` bed was mixed, and with
|
|
359
|
+
`--music` the `music_volume` plus a `duck` object naming the threshold (dB and
|
|
360
|
+
linear), ratio, attack and release actually used, or `null` when `--duck` was not
|
|
361
|
+
given. `loudness.py` reports `measured` (the input's loudnorm measurement,
|
|
362
|
+
including `input_lra`) and `targets` (the requested lufs / tp / lra).
|
|
363
|
+
`check.py --platform podcast` adds two informational rows to `checks`,
|
|
364
|
+
`channels` and `chapters`.
|
|
365
|
+
|
|
366
|
+
The MCP `audio` tool publishes `voice` as `{"type": "string", "enum": ["light",
|
|
367
|
+
"medium", "strong"]}`. A client that still sends the 1.12 boolean `{"voice":
|
|
368
|
+
true}` keeps working: `true` emits the bare `--voice`, which is `medium` — the
|
|
369
|
+
chain the flag has always produced. Send the string when you can; the boolean
|
|
370
|
+
is accepted at runtime and means `medium`.
|
|
371
|
+
|
|
338
372
|
Success (`exit 0`): one document matching `output_schema`, always with
|
|
339
373
|
`status: "completed"`, `output`, `dry_run`, `commands`, and `probe` of the output when a
|
|
340
374
|
file was written. `probe` prints its measurement document directly.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.13.0",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
package/references/gotchas.md
CHANGED
|
@@ -74,6 +74,14 @@ clip measured at -40 LUFS or below is room tone, wind or nothing; raising it
|
|
|
74
74
|
25 dB raises the noise, not the content. Leave the level, say so, and offer music
|
|
75
75
|
or narration.
|
|
76
76
|
|
|
77
|
+
Do not add a speech gate in front of the measurement either: `loudnorm`'s EBU
|
|
78
|
+
R128 integrated loudness already applies the −70 LUFS absolute and −10 LU
|
|
79
|
+
relative gates, which drop the same quiet blocks a `silencedetect` pass would.
|
|
80
|
+
A speech-span gate was measured against the whole-file measurement on every
|
|
81
|
+
fixture in the repo, including one that is half digital silence, and moved the
|
|
82
|
+
result by at most 0.6 LU — inside `check.py`'s own ±1 LU tolerance — for the
|
|
83
|
+
cost of a second full decode. That is why `loudness.py` has no speech-gate flag.
|
|
84
|
+
|
|
77
85
|
## Text and framing
|
|
78
86
|
|
|
79
87
|
### Captions, fonts and text order
|
|
@@ -83,18 +91,60 @@ come out soft (a 1280x720 source fit to 9:16 is 406x720 until export scales it t
|
|
|
83
91
|
1080x1920) — fit to the delivery size first
|
|
84
92
|
(`fit.py --width 1080 --height 1920`), then caption, then export.
|
|
85
93
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
(
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
94
|
+
Cue length and timing are handled for you since 1.12: `caption.py` wraps every
|
|
95
|
+
cue to the safe area by measured width at the chosen `--size`, splits a cue past
|
|
96
|
+
`--max-lines` (default 2) into consecutive cues, holds a cue shorter than
|
|
97
|
+
`--min-duration` (default 1.0 s) — never past the next cue — and shifts
|
|
98
|
+
everything by `--offset SECONDS`. It reports what it changed on one `cues:` line
|
|
99
|
+
and writes the adjusted copy next to the output, never over the file you passed
|
|
100
|
+
in. Wrapping needs the input video (the line width comes from its real frame
|
|
101
|
+
size); with `--write-srt` alone only the timing flags apply.
|
|
93
102
|
|
|
94
103
|
Windows drawtext crashes on some real builds (#100): the drawtext tools resolve a
|
|
95
104
|
concrete `--font-file` by default, which avoids it; if one still crashes, pass
|
|
96
105
|
`--font-file` explicitly. Details: `references/ci-platform-pitfalls.md`.
|
|
97
106
|
|
|
107
|
+
### Fonts by script
|
|
108
|
+
libass and drawtext draw an empty box per character they have no glyph for, and
|
|
109
|
+
ffmpeg still exits 0 — a video full of tofu is the classic "it worked" failure.
|
|
110
|
+
Since 1.12 `caption.py`, `graphics.py` and `overlay.py --text` detect the script
|
|
111
|
+
of the text they are about to draw (Japanese, Chinese, Korean, Arabic, Hebrew,
|
|
112
|
+
Devanagari, Thai, Cyrillic, Greek) and resolve a font file that covers it,
|
|
113
|
+
printing one line — `font: /usr/share/fonts/.../wqy-zenhei.ttc (covers ko)`.
|
|
114
|
+
**No font for the script is a failed job** (`kind: input`), not a warning.
|
|
115
|
+
|
|
116
|
+
- What this machine can render: `python3 scripts/_contract.py doctor --json`,
|
|
117
|
+
field `fonts.scripts` (`available` / `missing` / `unknown` per language, with
|
|
118
|
+
the file it would use). The plain-text `doctor` says the same in one line.
|
|
119
|
+
- What fontconfig has: `fc-list ":lang=ja" file family` (`ja`, `zh-cn`, `ko`,
|
|
120
|
+
`ar`, `he`, `hi`, `th`, `ru`, `el`).
|
|
121
|
+
- Install: `apt install fonts-noto-cjk fonts-noto-core`, or
|
|
122
|
+
`brew install --cask font-noto-sans-cjk font-noto-sans-arabic`, or point at a
|
|
123
|
+
file with `--font-file` (`overlay.py`, `graphics.py`) / `--fonts-dir`
|
|
124
|
+
(`caption.py`).
|
|
125
|
+
- Han characters alone (no kana, no hangul) are read as Chinese. Japanese or
|
|
126
|
+
Korean hanja text with no kana needs `--lang ja` / `--lang ko`
|
|
127
|
+
(`caption.py --language` is the same flag), or `"lang"` in brand.json.
|
|
128
|
+
- An explicit `--font`, an explicit `--font-file`, or a font your brand file
|
|
129
|
+
itself names is always kept, even when fontconfig says it does not cover the script: you get one info
|
|
130
|
+
line saying so, not a silent substitution. A brand file that never names a
|
|
131
|
+
font is not a choice — the script still picks one.
|
|
132
|
+
- `--fonts-dir` (`caption.py`) adds faces to the search, it does not switch the
|
|
133
|
+
check off: if nothing in the directory covers the script, one line says so and
|
|
134
|
+
a covering font is resolved as usual.
|
|
135
|
+
- **No fontconfig is `unknown`, not `missing`.** With no `fc-list` on PATH (or
|
|
136
|
+
one that fails) coverage cannot be verified: the job runs with the font as
|
|
137
|
+
given behind one info line, because libass and drawtext have font backends of
|
|
138
|
+
their own. Only fontconfig answering "nothing covers this" fails the job.
|
|
139
|
+
- **RTL:** libass shapes and reorders Arabic and Hebrew correctly, so
|
|
140
|
+
`caption.py` — which renders every subtitle through libass, `subtitles=` and
|
|
141
|
+
`ass=` alike — is right for them by construction. `drawtext` (`overlay.py
|
|
142
|
+
--text`, `graphics.py`) depends on the build: `ffmpeg -version` showing
|
|
143
|
+
`--enable-libfribidi` (and `--enable-libharfbuzz`) shapes and reorders RTL
|
|
144
|
+
correctly too; a build without them draws logical order with unjoined
|
|
145
|
+
letterforms. Nothing in the tools checks this, so on an unknown machine a
|
|
146
|
+
caption is the safe place for Arabic/Hebrew.
|
|
147
|
+
|
|
98
148
|
### Reframing, fps and duration
|
|
99
149
|
`--fit crop` to reach 9:16 from 16:9 throws away 70 % of the width: a wide shot
|
|
100
150
|
loses people at the edges. Check the sheet; pad (bars), `--crop-x`/`--crop-y`
|