ffmpeg-skill 2.7.0 → 2.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -4
- package/SKILL.md +5 -5
- package/docs/contract.md +15 -6
- package/package.json +1 -1
- package/references/gotchas.md +27 -5
- package/references/scripts.md +79 -5
- package/scripts/_common/__init__.py +6 -6
- package/scripts/_common/asr.py +53 -9
- package/scripts/_common/decision.py +13 -0
- package/scripts/_common/drawtext.py +4 -1
- package/scripts/_common/emit.py +12 -3
- package/scripts/_common/probe.py +92 -0
- package/scripts/_common/runner.py +54 -15
- package/scripts/_contract.py +39 -14
- package/scripts/batch.py +28 -11
- package/scripts/cut.py +1311 -53
- package/scripts/graphics.py +30 -7
- package/scripts/silence.py +32 -3
package/README.md
CHANGED
|
@@ -202,7 +202,7 @@ These are the rules the skill file gives the agent and the code enforces.
|
|
|
202
202
|
|
|
203
203
|
| Tool | What it does |
|
|
204
204
|
|---|---|
|
|
205
|
-
| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio; reports `precision`; `--snap beats` moves the in/out points onto a measured beat, or refuses when there is no measurable pulse |
|
|
205
|
+
| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio, `--keep-hevc` to re-encode an HEVC source as HEVC, `--edit-list` to keep an `.mp4` copy's start exact, `--vfr-guard` to choose the VFR check; reports `precision`; `--snap beats` moves the in/out points onto a measured beat, or refuses when there is no measurable pulse |
|
|
206
206
|
| `join.py` | Concatenate clips with xfade transitions, normalising size, fps, sample rate and channel layout (the widest clip's, or `--channels`); audio-only inputs are joined as audio; `--list parts.txt` reads generated segments (TTS lines) and checks every one first — missing, empty or unreadable parts are all named in one refusal, or skipped and reported with `--on-missing skip` |
|
|
207
207
|
| `silence.py` | Detect and remove dead air (jump cuts) with a margin around speech; list or export the cut list; `--filler` also removes filler words, but only where a speech engine timed them; `--speech-aware` keeps a breath inside a sentence and cuts only at sentence-boundary pauses, composing with `--filler` through the same `keep_ranges()` |
|
|
208
208
|
| `fit.py` | Fit to a duration (pitch-preserving speed change or trim, smooth slow-mo) and/or aspect ratio (pad, crop or `--fit blur`'s blurred, darkened fill, with `--crop-x`/`--crop-y` to keep an off-centre subject) and/or exact `--width`/`--height`; rotate 90/180/270, flip h/v; force constant fps |
|
|
@@ -362,8 +362,8 @@ npx ffmpeg-skill doctor --json # available / missing / missing_optional / unkn
|
|
|
362
362
|
|
|
363
363
|
The short list for humans. The agent-facing version, with the reasoning, is the "Things that look right but are wrong" and "Gotchas" sections of [SKILL.md](SKILL.md).
|
|
364
364
|
|
|
365
|
-
- **Variable frame rate (phone and screen recordings).** `probe.py` flags it; every re-encoding tool conforms to a constant rate automatically
|
|
366
|
-
- **Lossless cuts snap to keyframes.** A stream-copy cut can start up to one GOP earlier than asked. `cut.py` re-encodes when the snap exceeds 0.5 s (`--tolerance` changes the limit). For a strictly lossless file pass `--tolerance -1
|
|
365
|
+
- **Variable frame rate (phone and screen recordings).** `probe.py` flags it; every re-encoding tool conforms to a constant rate automatically. `cut.py` switches to frame-accurate mode on it, because copy-cuts on VFR land on the wrong frame, and samples the frame timing to say whether that was needed (`--vfr-guard sampled` decides on the sample instead; `--vfr-guard off` keeps the copy). Choose the rate yourself with `fit.py input.mp4 --fps 30` when the measured average is odd.
|
|
366
|
+
- **Lossless cuts snap to keyframes.** A stream-copy cut can start up to one GOP earlier than asked. `cut.py` re-encodes when the snap exceeds 0.5 s (`--tolerance` changes the limit). For a strictly lossless file pass `--tolerance -1` (plus `--vfr-guard off` on a variable-frame-rate source), and expect the cut to land on the nearest earlier keyframe; the JSON result lists them under `nearest_keyframes`. A `--segments` join can still be re-cut: open-GOP video, a later segment of B-frame `.mp4`/`.mov` video that starts between keyframes (start it on a keyframe, or write `.mkv`), or a join that fails its check. For a single `.mp4`/`.mov` cut, `--edit-list` keeps the copy and starts the picture where asked: the keyframe's pre-roll is stored but hidden by an MP4 edit list, which a player that ignores edit lists shows.
|
|
367
367
|
- **HDR stays HDR.** When the probe reports HDR (HDR10, HLG, Dolby Vision, BT.2020), the tools keep it rather than flatten it. Convert deliberately with `color.py --to-sdr` before H.264 deliverables or LUT work. `export.py` platform presets are SDR and warn on HDR input.
|
|
368
368
|
- **Loudness targets.** −14 LUFS / −1 dBTP for YouTube and social platforms (the `loudness.py` default), `-I -16 --tp -1.5` for podcasts, `-I -23` for broadcast. A clip measured at −40 LUFS or below is room tone, not content; raising it raises the noise. Check true peak as well as LUFS: `check.py file --platform podcast` measures both.
|
|
369
369
|
- **Frame changes first, text second.** Captions and overlays burned before a crop or resize end up off-frame. Reframe, then caption.
|
|
@@ -401,7 +401,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th
|
|
|
401
401
|
**What is tested where.** The contract and the test suite (`tests/test_contract.py`,
|
|
402
402
|
`tests/test_all.py`, which aggregates one module per tool group — `test_analysis.py`,
|
|
403
403
|
`test_editing.py`, `test_audio.py`, `test_picture.py`, `test_delivery.py`,
|
|
404
|
-
`test_orchestration.py` — over the shared footage in `tests/_fixtures.py`) run on Linux, macOS
|
|
404
|
+
`test_orchestration.py`, `test_accel.py`, `test_asr.py`, `test_cut_copy.py` — over the shared footage in `tests/_fixtures.py`) run on Linux, macOS
|
|
405
405
|
and Windows on every pull request, minus the handful of POSIX-shim tests listed under
|
|
406
406
|
[Development](#development). The real-device media corpus
|
|
407
407
|
(`tests/corpus.py`) has been run on Linux and macOS; the full corpus has **not** been run on
|
package/SKILL.md
CHANGED
|
@@ -7,7 +7,7 @@ description: 'Edit video and audio with local FFmpeg from natural-language reque
|
|
|
7
7
|
|
|
8
8
|
Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`, and delivery templates in `templates/`. This file is enough to do a job: the table below routes the request, and `--help` on the script about to run is the cheapest full flag list. A reference file costs as much as this one; open one only for a question you have: `references/scripts.md` (every flag of all 42 scripts), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end). MCP: `tools/list` shows only the core 12 by default; the other 30 (per this table) stay callable by name via `tools/call`, described by `contract --json`; set `FFMPEG_SKILL_MCP_FULL=1` for all 42.
|
|
9
9
|
|
|
10
|
-
Shared flags (every script): `--dry-run`; `--json` (output path, its probe, the commands run); `--json-brief` (status/output/verified + a `summary`; prefer on writing steps); `--fast` (preview quality); `--progress` (percent/ETA); `--timeout SECONDS` (`kind: timeout`, default 1800); `--overwrite` (step 7); `--plan FILE` (a dry run `render.py FILE` runs later; refuses if an input changed). Re-encoding tools also take `--hw` (Apple Silicon GPU, big files; `FFMPEG_SKILL_HW=1` = default, not export), `--codec h264|hevc|av1|prores` and `--quality N`: unset, SDR is x264, HDR is x265 Main10; `prores` needs `-o NAME.mov`, `h264` refuses HDR (`color.py --to-sdr` first).
|
|
10
|
+
Shared flags (every script): `--dry-run`; `--json` (output path, its probe, the commands run); `--json-brief` (status/output/verified + a `summary`; prefer on writing steps); `--fast` (preview quality); `--progress` (percent/ETA); `--timeout SECONDS` (`kind: timeout`, default 1800); `--overwrite` (step 7); `--plan FILE` (a dry run `render.py FILE` runs later; refuses if an input changed). Re-encoding tools also take `--hw` (Apple Silicon GPU, big files; `FFMPEG_SKILL_HW=1` = default, not export), `--codec h264|hevc|av1|prores` and `--quality N`: unset, SDR is x264 (`cut.py --keep-hevc`: HEVC source stays HEVC), HDR is x265 Main10; `prores` needs `-o NAME.mov`, `h264` refuses HDR (`color.py --to-sdr` first).
|
|
11
11
|
|
|
12
12
|
Writing tools run nothing under `--dry-run`; the measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) may still run ffmpeg/ffprobe, skipping artifacts/side files (`--edl`, `--sheet`, a generated `.ass`); `verify` ignores the flag. Per tool: `contract --json` `dry_run`.
|
|
13
13
|
|
|
@@ -52,7 +52,7 @@ This skill cuts, joins, measures, syncs, exports and checks files — it execute
|
|
|
52
52
|
|
|
53
53
|
The line: same input + same explicit parameters always producing the same verifiable output belongs here; anything depending on taste, understanding or what looks or sounds good belongs to whoever makes that judgement.
|
|
54
54
|
|
|
55
|
-
If a request needs an FFmpeg feature none of the 42 scripts expose, say so and name the closest built-in option — never guess a raw `ffmpeg`/`ffprobe` invocation outside `scripts/*.py`. It bypasses every guarantee this skill makes, so never a fallback
|
|
55
|
+
If a request needs an FFmpeg feature none of the 42 scripts expose, say so and name the closest built-in option — never guess a raw `ffmpeg`/`ffprobe` invocation outside `scripts/*.py`. It bypasses every guarantee this skill makes, so it is never a fallback.
|
|
56
56
|
|
|
57
57
|
## Request → script
|
|
58
58
|
|
|
@@ -65,7 +65,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
|
|
|
65
65
|
| "cut from 1:20 to 2:05", "trim the first 10 s" | `cut.py input.mp4 --start 1:20 --end 2:05` |
|
|
66
66
|
| "keep only these parts", "remove the middle" | `cut.py input.mp4 --segments 0-1:00,1:30-2:00` |
|
|
67
67
|
| "make it exactly 60 seconds" | `fit.py input.mp4 --duration 60` (speed) or `--method trim` |
|
|
68
|
-
| "cut this and make it HEVC / AV1 / ProRes" (output codec named) | `cut.py input.mp4 --start 0:10 --end 0:40 --codec hevc` (
|
|
68
|
+
| "cut this and make it HEVC / AV1 / ProRes" (output codec named) | `cut.py input.mp4 --start 0:10 --end 0:40 --codec hevc` (ProRes needs `-o NAME.mov`) |
|
|
69
69
|
| "cut out the pauses", "jump cuts" | `silence.py input.mp4 [--threshold -40 --min-silence 0.8]` |
|
|
70
70
|
| "cut the ums and uhs", "remove the filler words" | `silence.py input.mp4 --filler --words words.json` (measured word timings; `--transcribe` makes them) |
|
|
71
71
|
| "don't cut inside a sentence, just the real pauses" | `silence.py input.mp4 --speech-aware` — a breath under `--min-silence` inside a sentence is kept, only sentence-boundary pauses cut; composes with `--filler` into one list |
|
|
@@ -206,8 +206,8 @@ One line each; open the linked `references/gotchas.md` section when the job is i
|
|
|
206
206
|
|
|
207
207
|
- HDR (iPhone, HDR10) re-encoded through an SDR path goes flat; the scripts keep HDR; `hdr: true` is a real PQ/HLG/DV signal, `bt2020_or_hdr: true` also BT.2020 SDR. -> [#hdr-and-colour](references/gotchas.md#hdr-and-colour)
|
|
208
208
|
- Log footage (S-Log/V-Log/C-Log) is tagged SDR and looks grey: `probe.py --analyze`, then `color.py --lut` first. -> [#log-footage](references/gotchas.md#log-footage)
|
|
209
|
-
- A `-c copy` cut can start on a wrong
|
|
210
|
-
- VFR phone/screen
|
|
209
|
+
- A `-c copy` cut can start on a wrong/frozen frame; `cut.py` re-encodes past a 0.5 s snap, respect it; `--edit-list` hides an .mp4's pre-roll. -> [#keyframe-cuts](references/gotchas.md#keyframe-cuts)
|
|
210
|
+
- VFR phone/screen clips: re-encodes go CFR; `cut.py` re-encodes (`--vfr-guard sampled`: only measured VFR); odd rate: `fit.py --fps`. -> [#variable-frame-rate](references/gotchas.md#variable-frame-rate)
|
|
211
211
|
- Sync/multicam `confidence` under 0.3 (or a huge offset) is suspect — check every camera; these align audio, never lip sync. -> [#sync-multicam-and-drift](references/gotchas.md#sync-multicam-and-drift)
|
|
212
212
|
- "Normalised" audio can still clip (check true peak); ambience at -40 LUFS or below must never be raised to a speech target. -> [#loudness-and-ambience](references/gotchas.md#loudness-and-ambience)
|
|
213
213
|
- Captions burned before a crop/resize land off-frame; burned small then upscaled by `export.py` come out soft. -> [#captions-fonts-and-text-order](references/gotchas.md#captions-fonts-and-text-order)
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`2.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`2.8.1`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -119,7 +119,7 @@ JSON would have been rewritten for that. The flat keys, typed per tool in `outpu
|
|
|
119
119
|
"contract_version": "1.0",
|
|
120
120
|
"deprecated": [],
|
|
121
121
|
"removed": [{"what": "...", "since": "1.10.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
122
|
-
"skill": {"id": "ffmpeg-skill", "version": "2.
|
|
122
|
+
"skill": {"id": "ffmpeg-skill", "version": "2.8.1", "execution_mode": "local", "kind": "execution",
|
|
123
123
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
124
124
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
125
125
|
"requirements": {"python": ">=3.9 (standard library only)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
|
|
@@ -216,7 +216,9 @@ BT.709 for SDR; av1 is SVT-AV1 with libaom as the fallback; prores is 422 HQ and
|
|
|
216
216
|
`.mov`/`.mkv` output; h264 refuses an HDR source (`kind: input`). `--quality` is the CRF scale;
|
|
217
217
|
its default is the tool's own (18, `proxy.py` 30). 1.x also accepted `--crf` as an alias; 2.0
|
|
218
218
|
removed it (`export.py` keeps its own `--crf`). Without `--codec` the encoder is what it always was (x264 for SDR, x265
|
|
219
|
-
Main10 for HDR), so the flags add no behaviour to a caller that does not pass them.
|
|
219
|
+
Main10 for HDR), so the flags add no behaviour to a caller that does not pass them.
|
|
220
|
+
`cut.py --keep-hevc` (opt-in) re-encodes an SDR HEVC source as x265 8-bit BT.709 instead of x264; an HDR or
|
|
221
|
+
BT.2020 source keeps the Main10 line with its own colour tags, with or without the flag. The
|
|
220
222
|
encoder each value needs is listed under the tool's optional capabilities (`--codec hevc` and
|
|
221
223
|
so on). `export.py` refuses `--codec`: its presets decide the codec.
|
|
222
224
|
|
|
@@ -245,7 +247,7 @@ runs the tool with the planned `argv`, then the verify steps (a `check` step may
|
|
|
245
247
|
### Repeatability
|
|
246
248
|
|
|
247
249
|
No tool keeps state or uses randomness. `deterministic_inputs` is `false` only for
|
|
248
|
-
`verify`, whose output includes timings. `idempotency_hint` says what "same inputs"
|
|
250
|
+
`verify` and `cropdetect`, whose output includes timings or machine state. `idempotency_hint` says what "same inputs"
|
|
249
251
|
gives you:
|
|
250
252
|
|
|
251
253
|
| Hint | Tools |
|
|
@@ -253,7 +255,7 @@ gives you:
|
|
|
253
255
|
| `bit_exact` | probe, check, scenes, look, metadata, background |
|
|
254
256
|
| `content_equivalent` (same media, bytes may differ between encoder builds) | every encoding tool, cut, sync, report |
|
|
255
257
|
| `cached` | batch (content-hash cache, re-runs skip unchanged inputs); render **when `--cache DIR` is given** — the hint stays `content_equivalent` because that is what render is without the flag, and the cache is opt-in |
|
|
256
|
-
| `environment_dependent` | verify |
|
|
258
|
+
| `environment_dependent` | verify, cropdetect |
|
|
257
259
|
|
|
258
260
|
The hints describe the CPU encoders. Under `--hw`, or `FFMPEG_SKILL_HW=1` on the machine, a tool
|
|
259
261
|
that encodes video runs VideoToolbox, whose bytes are not promised identical from run to run:
|
|
@@ -566,13 +568,20 @@ Per-tool keys added after 2.2.2, all additive:
|
|
|
566
568
|
|
|
567
569
|
| key | tool | what it holds |
|
|
568
570
|
|---|---|---|
|
|
571
|
+
| `boomerang`, `notes` | `loop.py` | `boomerang` is `true` for `--boomerang` (frames 0..N-1 then N-2..1, repeated, so neither turnaround frame is shown twice); `notes` says the audio was dropped when the source had any (a boomerang is silent) |
|
|
569
572
|
| `sdr_path`, `notes` | `color.py --to-sdr` | `sdr_path` is `"tonemap"` for PQ / HLG / Dolby Vision input (and `--force` on an untagged file) or `"gamut"` for BT.2020 primaries on an SDR transfer, which is converted to BT.709 without a tone map (`--tonemap`/`--peak`/`--desat` do not apply); `notes` says which path was taken and why |
|
|
570
573
|
| `silent` | `join.py` (every run) | `[{index, path, peak_db, at, end}]`: inputs whose whole-file audio peak (volumedetect) is at or below `--silence-threshold` (default -50 dBFS) and that `--on-silent warn` (default) joined anyway, with a `notes` line; `at`/`end` are where the input sits in the output, in seconds (a transition overlaps its neighbours). An input named by `--allow-silent N[,N]` (1-based) is a planned pause: always joined, listed with `intended: true`, no warning or note, whatever `--on-silent` says. `[]` otherwise. `--on-silent fail` names them in the `kind: input` refusal's `problems` (reason `silent (peak -91.0 dBFS)`), `skip` lists them under `skipped`. An input with no audio stream is never silent; `verified` is unaffected |
|
|
571
574
|
| `short_segments`, `duplicates` | `join.py` (every run) | warnings, never refusals, and the join command is unchanged: `short_segments: [{index, path, duration}]` names each measured input shorter than 2 frames at the join's fps (an audio-only join: shorter than 0.05 s); `duplicates: [{path, indices}]` names a path (compared resolved) listed more than once — repeating a clip can be intended. `[]` when none; each non-empty one adds a `notes` line |
|
|
572
575
|
| `silent` | `waveform.py` (every run) | `true` when the input audio's whole-file peak is at or below `--silence-threshold` (default -50 dBFS), which draws a flat line; `--on-silent warn` (default) renders it anyway with a `notes` line, `fail` refuses (`kind: input`) before ffmpeg runs. `false` when audible, `null` under `--dry-run` (nothing measured). `verified` is unaffected |
|
|
573
576
|
| `encoder`, `hw`, `notes` | every re-encoding tool, `export.py`, `render.py`, `batch.py` (and its rows), `waveform.py` | `encoder`: the video encoder the last command ran (after any GPU→CPU fallback; under `--dry-run`, the planned one), `copy` for a stream copy; absent when no command encoded video. A tool that runs other tools reports its stages': `render.py` its stages and an executed plan's tool, `waveform.py` the title/caption stage that writes its file (its `commands` include that stage's). `hw` is present when VideoToolbox was asked for (`--hw`, `FFMPEG_SKILL_HW=1`, or a stage or `batch.py` item that reported it) and on `export.py` when `FFMPEG_SKILL_HW=1` is set but its delivery preset does not take it: `{requested, source: flag\|env, used, fallback, notes}`. `requested` says whether this run's encodes were asked onto VideoToolbox (`false` for that export), `used` whether the encoder that ran last is VideoToolbox (`null` under `--dry-run`, which ran nothing, and when the tool encoded nothing itself: `batch.py`, whose top-level `hw` gathers its items' `requested`/`source`/`fallback`/`notes` while each `results` row carries the item's own `encoder`, and `hw`/`notes` when VideoToolbox was asked for), `fallback` whether a job VideoToolbox refused was re-encoded on the CPU, and `notes` why. Only a VideoToolbox failure (the encoder could not open, or its compression session failed) is retried on the CPU; any other failure is reported as it happened, once. Every VideoToolbox encode adds a top-level `notes` line: its quality is not CRF-equivalent (`--quality` / a preset's CRF maps to `-q:v` approximately), and `verified` stays what it is for every tool, the measured properties of the output. A `kind: ffmpeg` failure under `--hw` / `FFMPEG_SKILL_HW` carries `encoder` and `hw` for the command that ran last (a stage's, when `render.py`/`waveform.py` re-raise it); when the command itself failed inside the run (not a tool's own check afterwards, such as `cut.py`'s), its message also names that encoder |
|
|
574
|
-
| `transcription` | `caption.py --transcribe` (every `--mode`); inside `filler` for `silence.py --filler --transcribe` | `{routing, detected_language?, engine, model, language}`: the engine and model that made the transcript; `language` is the language the engine was told -- the one named (`--language`, `--filler-lang`), else the one auto's detector named -- and
|
|
577
|
+
| `transcription` | `caption.py --transcribe` (every `--mode`); inside `filler` for `silence.py --filler --transcribe` | `{routing, detected_language?, engine, model, language}`: the engine and model that made the transcript; `language` is the language the engine was told -- the one named (`--language`, `--filler-lang`), else the one auto's detector named -- and, when nothing was named, the language the engine decoded where it reports one (whisper.cpp's word-timing JSON, faster-whisper, openai-whisper's word-timing JSON), else `null`; `detected_language` is present when auto's whisper.cpp detector named one; `routing` says why `--engine auto` chose the engine (`"requested"` for a named engine), ending `, no Parakeet engine transcribed it` when auto chose Parakeet and a Whisper engine made the transcript after every Parakeet engine failed |
|
|
575
578
|
| `notes` | `silence.py --filler --transcribe` (new for `silence.py`; `caption.py` already had it) | present when `--engine auto` ran the English-only Parakeet model on English *assumed* (no language given or detectable, and no Whisper engine installed or none that succeeded): the transcript is wrong if the speech is not English. `caption.py --transcribe` puts the same line in its `notes` |
|
|
579
|
+
| `reencode_reason`, `segment_precision`, `least_exact_precision`, `start_snapped` | `cut.py` (every run) | `reencode_reason`: why anything was re-encoded, distinct values in first-seen order, `[]` for a clean copy: `requested` (`--accurate`), `codec` (`--codec`), `vfr`, `vfr_inconclusive`, `pcm_container`, `copy_failed`, `tolerance`, `concat_fallback` (a `--segments` join that could not stay a stream copy, which now also sets `reencoded: true` and `mode: hybrid`; it reported `copy` before). `segment_precision`: each `--segments` segment's precision (`null` for one segment); `least_exact_precision`: the least exact of them, on every run. `precision` and `keyframe_snapped` keep their meaning (`keyframe_snapped` is `precision == "packet"`, a stream copy end to end); `least_exact_precision` differs from `precision` only for an audio-only join of copied and re-encoded parts. `start_snapped`: measured, `true` when a copied segment's picture starts more than one frame from its requested start (or its start could not be measured: an unknown start is never claimed exact), `false` when every start is where asked or everything re-encoded, `null` under `--dry-run`. `requested_segments`/`requested_duration` keep their meaning, the request clamped to the media's duration: a segment ended with the video (less than a frame of sound trimmed) shows in `duration_delta_seconds` and a `notes` line |
|
|
580
|
+
| `edit_list`, `stored_preroll_seconds`, `av_start_skew_seconds`, `notes` | `cut.py` (every run) | `edit_list`: `true` only for a single-segment `.mp4`/`.m4v`/`.mov` stream copy run with `--edit-list`, which keeps the MP4 edit list the copy writes instead of `-avoid_negative_ts make_zero` (the default, unchanged): the keyframe's pre-roll is stored but hidden, so the picture starts at `--start`. `stored_preroll_seconds`: with `edit_list`, the seconds decoded from the keyframe the copy's seek lands on (measured with the seek ffmpeg makes) up to the first presented frame; `null` otherwise. Under `--edit-list` a copy is judged by its video's length (a `notes` line says so when the container is longer), and one that starts where asked but ends past `--tolerance` re-encodes (`tolerance`) without a `lossless_alternative`. `av_start_skew_seconds`: the output's audio start minus its video start (`null` when either is missing or under `--dry-run`); on a stream copy, when it is more than max(2 frames, 0.1 s) away from the source's own audio/video offset where the copy's picture starts (the keyframe it began from; `--start` under `--edit-list`), a `notes` line names `--accurate`, and `--edit-list` for a single `.mp4`/`.mov` copy whose start snapped. A re-encode carries no such note. The skew is reported, never repaired. `notes`: what the run did that a caller should know, in plain sentences (`[]` when nothing) |
|
|
581
|
+
| `video.start_time`, `audio.start_time` | `probe.py` (and every `probe` block a tool returns) | where each stream starts, in seconds on the file's own clock (ffprobe's stream `start_time`; `null` when unknown). A video stream that starts after its audio, or the reverse, is how a lossless cut's A/V skew shows; `cut.py` reads them for `av_start_skew_seconds` |
|
|
582
|
+
| `cut_reencode_reasons` | `batch.py` (and its `results` rows) | a row whose recipe ran `cut.py` lists each `reencode_reason` its `cut.py` calls named, once per call, beside `cut_reencoded`; the top level counts them across the folder (`{}` when no call re-encoded, `null` when no `cut.py` step ran) on a completed batch, as `cut_stream_copy` does: a failure document carries `results`, whose rows hold each row's reasons; a re-encoded call in a row served from a cache an earlier 2.x batch wrote, which recorded no reasons, counts as `unknown`. `cut_stream_copy` is unchanged |
|
|
583
|
+
| `vfr_check` | `cut.py` (a video input that could still stream-copy) | the sampled frame timing, `{heuristic: true, measured, method: "sampled", guard, windows, deltas}`; `measured` is `sampled_cfr`, `vfr` or `inconclusive`. `--vfr-guard average` (the default) keeps the 2.x guard: a source whose nominal and average frame rates differ (`variable_frame_rate_suspected`) re-encodes as `--accurate` (reason `vfr`, stderr says "variable-frame-rate"), and only then is the timing sampled and reported, with a `notes` line naming `--vfr-guard sampled` when it measured `sampled_cfr`; the key is absent otherwise. `--vfr-guard sampled` samples before every copy and re-encodes on `vfr` (reason `vfr`) or `inconclusive` (fewer than 20 intervals; `vfr_inconclusive`); `--vfr-guard off` samples and keeps the copy, with a `notes` line. Absent for audio-only input or output, `--accurate`, `--codec` and a pending `--dry-run` input |
|
|
584
|
+
| `join_check`, `segment_end_snap_seconds` | `cut.py --segments` | `join_check`: a video stream-copy join measured by demuxing, `{packets, expected_packets, max_step_seconds, audio_offset_ms, audio_parts_checked, ok}`. `expected_packets` counts the source's video packets from each part's start keyframe to its end (or the end of the video); `max_step_seconds` is the largest presentation step, `null` when `vfr_check` measured anything but `sampled_cfr` (a `--vfr-guard off` join of variable timing: count only), and each step at a join must be the step its parts predict within 1 ms. `audio_offset_ms`: per part, its sound's shift minus its picture's in ms (+ = sound late), from audio packets matched by CRC32 to the source stream shaped like the joined audio (FFmpeg's default selection may copy one other than `a:0`; an MPEG-TS source's AAC is hashed without its ADTS headers); `null` for a part with no unique match (digital silence, a periodic tone), and as a whole when the source has no audio or the video check failed. `audio_parts_checked` counts the measured parts. `ok` is the conjunction (count, steps, every measured part within 5 ms); `ok: false` means the join was re-cut (`concat_fallback`) and this is the failed measurement. `null` for a single segment, audio, `--dry-run`, any re-encode forced for every segment, a segment that re-encoded, or a join re-cut before a copy was tried (open GOPs, a later B-frame segment starting between keyframes, parts that differ, a concat error). `segment_end_snap_seconds`: per segment, the signed seconds its end moved to the keyframe the copied part stops at, the first at or after the requested end (`null` for a part that reaches the end of the video, a part of a source with no B-frames, cut to its requested end as in 2.5.1, or a Matroska/MPEG-TS part, which is not snapped); `null` as a whole unless the output is those copied parts |
|
|
576
585
|
|
|
577
586
|
`caption.py --transcribe` and `silence.py --filler --transcribe`: when a local speech engine ran
|
|
578
587
|
and produced no cue, the refusal is `"<engine> found no speech in <input>"` with `kind: input`,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.8.1",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
package/references/gotchas.md
CHANGED
|
@@ -37,17 +37,39 @@ S-Log, V-Log and C-Log look grey and low-contrast but are tagged SDR. Run
|
|
|
37
37
|
|
|
38
38
|
### Keyframe cuts
|
|
39
39
|
A lossless `cut.py` result may start up to one GOP (often 1–10 s) earlier than
|
|
40
|
-
requested; the script re-encodes automatically when the
|
|
41
|
-
|
|
42
|
-
|
|
40
|
+
requested (`start_snapped: true`); the script re-encodes automatically when the
|
|
41
|
+
deviation exceeds 0.5 s. With `--edit-list`, a single-segment `.mp4`/`.m4v`/`.mov` copy
|
|
42
|
+
keeps the MP4 edit list instead: the pre-roll back to the keyframe is stored but
|
|
43
|
+
hidden, so the picture starts at the requested time (`edit_list`,
|
|
44
|
+
`stored_preroll_seconds`). A player or tool that ignores edit lists shows that
|
|
45
|
+
pre-roll. The **end** still lands on a packet boundary, a few frames long, and
|
|
46
|
+
those extra frames can skip. `.mkv` output has no edit list, and a `--segments`
|
|
47
|
+
join shows each part from its keyframe. On B-frame video an `.mp4`/`.mov`
|
|
48
|
+
`--segments` part's end moves to the first keyframe at or after it, within the
|
|
49
|
+
same tolerance (`segment_end_snap_seconds`); no-B-frame and `.mkv`/`.ts` parts
|
|
50
|
+
are not snapped. Every copy join is checked (frames, steps, each part's sound
|
|
51
|
+
against its picture) and re-cut when wrong. Open-GOP HEVC (iPhone "High Efficiency")
|
|
52
|
+
always re-encodes a `--segments` join, even with `--tolerance -1`, and so does a B-frame `.mp4`/`.mov`
|
|
53
|
+
join whose later segment starts between keyframes (start it on a keyframe, or write `.mkv`). If the user insists on lossless output, pass
|
|
54
|
+
`--tolerance -1` and tell them the cut lands on the nearest earlier keyframe.
|
|
55
|
+
Where a copy begins is measured with the seek ffmpeg makes: on B-frame `.mp4`/`.mov`
|
|
56
|
+
a start a few frames before a keyframe can begin at that keyframe or one GOP
|
|
57
|
+
earlier (`start_snapped: true` either way).
|
|
58
|
+
`av_start_skew_seconds` warns when a copy's sound and picture start apart
|
|
59
|
+
(Core Media HEVC once gave 3.7 s of sound with no picture); it is reported, not
|
|
60
|
+
repaired: `--accurate` is the fix, or `--edit-list` for a lossless `.mp4`/`.m4v`/`.mov`. A `-c copy` cut on VFR or a
|
|
43
61
|
non-keyframe boundary produces a file that "works" but starts on a frozen or
|
|
44
62
|
wrong frame — respect the automatic re-encode rather than forcing the copy.
|
|
45
63
|
|
|
46
64
|
### Variable frame rate
|
|
47
65
|
`probe.py` sets `variable_frame_rate_suspected` when `r_frame_rate` and
|
|
48
66
|
`avg_frame_rate` disagree (phone and screen recordings). Every re-encoding
|
|
49
|
-
script then adds `-fps_mode cfr` at the source's average rate
|
|
50
|
-
|
|
67
|
+
script then adds `-fps_mode cfr` at the source's average rate. That flag is a
|
|
68
|
+
whole-file average, and a phone clip at 29.98 against a nominal 30 trips it, so
|
|
69
|
+
`cut.py` re-encodes on it by default and reports packet timestamps sampled in up
|
|
70
|
+
to five windows (`vfr_check`), with a note when they are constant;
|
|
71
|
+
`--vfr-guard sampled` switches to `--accurate` only when they are irregular or too
|
|
72
|
+
few to judge, and `--vfr-guard off` keeps the copy anyway. Pick the rate
|
|
51
73
|
explicitly with `fit.py --fps 30|60` when the average is odd (e.g. 23.4 fps from
|
|
52
74
|
dropped frames).
|
|
53
75
|
|
package/references/scripts.md
CHANGED
|
@@ -56,24 +56,91 @@ between tools.
|
|
|
56
56
|
```
|
|
57
57
|
probe.py INPUT... [--compact] [--field duration|video.fps|...]
|
|
58
58
|
```
|
|
59
|
-
JSON with `duration`, `video{codec,width,height,fps,pix_fmt,color_space,rotation,variable_frame_rate_suspected,hdr,hdr_signal,bt2020_or_hdr,hdr_format}`
|
|
59
|
+
JSON with `duration`, `video{codec,width,height,fps,pix_fmt,color_space,rotation,variable_frame_rate_suspected,hdr,hdr_signal,bt2020_or_hdr,hdr_format,start_time}`
|
|
60
60
|
(`hdr`, and `hdr_signal` equal to it, is true only for a PQ / HLG transfer or Dolby Vision -- since 2.0;
|
|
61
61
|
`bt2020_or_hdr` also counts BT.2020 primaries on an SDR transfer, which `hdr_format` names "BT.2020 SDR"),
|
|
62
|
-
`audio{codec,channels,sample_rate}
|
|
62
|
+
`audio{codec,channels,sample_rate,start_time}` (`start_time`: where the stream starts on the file's
|
|
63
|
+
own clock, `null` when unknown). `--compact` gives one line per file.
|
|
63
64
|
|
|
64
65
|
### cut.py — cut / join segments
|
|
65
66
|
```
|
|
66
|
-
cut.py INPUT [--start T] [--end T | --duration T] [--segments A-B,C-D,...] [--accurate] [-o OUT]
|
|
67
|
+
cut.py INPUT [--start T] [--end T | --duration T] [--segments A-B,C-D,...] [--accurate] [--edit-list] [--keep-hevc] [--vfr-guard average|sampled|off] [-o OUT]
|
|
67
68
|
cut.py INPUT --start T --end T --snap beats [--snap-tolerance 0.12] [--snap-source FILE] [--min-confidence 0.5]
|
|
68
69
|
```
|
|
69
70
|
Times accept `12.5`, `1:30`, `00:01:30.250`. Default is `-c copy` (snaps to
|
|
70
71
|
keyframes, instant, lossless); if the snapped result deviates more than
|
|
71
72
|
`--tolerance` (0.5 s) from the request, that segment is re-encoded automatically
|
|
72
|
-
(
|
|
73
|
+
(CRF 18: x264, or x265 Main10 with the source's own tags for an HDR or BT.2020 source;
|
|
74
|
+
`--keep-hevc` re-encodes an SDR HEVC source (not BT.2020: that keeps its Main10 line and colour tags) as x265 8-bit BT.709, on VideoToolbox under `--hw`;
|
|
75
|
+
`--codec` overrides all of them). `--accurate` always re-encodes; `--tolerance -1` never does for a keyframe snap
|
|
76
|
+
(a `--segments` join of open-GOP video, one whose later B-frame `.mp4`/`.mov` segment starts
|
|
77
|
+
between keyframes, or one that fails its check, still re-cuts).
|
|
73
78
|
Multiple segments are concatenated in the order given. stderr reports whether
|
|
74
79
|
the result was "lossless stream copy" or "re-encoded"; when the snap forced a
|
|
75
80
|
re-encode, the result's `lossless_alternative` names the nearest keyframe
|
|
76
81
|
`--start` that would stream-copy instead, so the trade can be offered.
|
|
82
|
+
A single `.mp4`/`.mov` stream copy shifts its timestamps to zero
|
|
83
|
+
(`-avoid_negative_ts make_zero`), so the picture starts at the keyframe.
|
|
84
|
+
`--edit-list` keeps its MP4 edit list instead, so the picture starts at
|
|
85
|
+
`--start`: `edit_list` and `stored_preroll_seconds` say so, `notes` warns that a
|
|
86
|
+
player which ignores edit lists shows the pre-roll, and the copy's length is
|
|
87
|
+
judged by its video (a copy that starts where asked but ends past `--tolerance`
|
|
88
|
+
re-encodes without offering another `--start`). `av_start_skew_seconds` is
|
|
89
|
+
audio start minus video start; a stream copy gets a note when it is more than
|
|
90
|
+
max(2 frames, 0.1 s) away from the source's own offset where the picture starts
|
|
91
|
+
(the keyframe the copy began from; `--start` under `--edit-list`). It is
|
|
92
|
+
reported, not repaired. `-ss`/`-t` are passed to the microsecond; `--accurate`
|
|
93
|
+
seeks a second early and trims on the output side, so frames just before a
|
|
94
|
+
keyframe are kept. `keyframe_snapped` is true for every stream copy (`precision`
|
|
95
|
+
`packet`); `start_snapped` is measured: true when a copied picture starts more
|
|
96
|
+
than a frame from `--start`, or when its start could not be measured (an unknown start is
|
|
97
|
+
never claimed exact); `null` under `--dry-run`.
|
|
98
|
+
`reencode_reason` lists every cause of a re-encode (`requested`, `codec`, `vfr`,
|
|
99
|
+
`vfr_inconclusive`, `pcm_container`, `copy_failed`, `tolerance`, `concat_fallback`);
|
|
100
|
+
`--segments` adds `segment_precision`, and `least_exact_precision` is the least exact one.
|
|
101
|
+
A `--segments` video copy join adds `join_check` (`packets`, `expected_packets`,
|
|
102
|
+
`max_step_seconds`, `audio_offset_ms` per part, `audio_parts_checked`, `ok`: the
|
|
103
|
+
written file measured against the source) and `segment_end_snap_seconds` (where
|
|
104
|
+
each part's end moved to its keyframe).
|
|
105
|
+
|
|
106
|
+
**VFR guard (`--vfr-guard`).** `average` (default): a source whose nominal and
|
|
107
|
+
average frame rates differ (`variable_frame_rate_suspected`) re-encodes as
|
|
108
|
+
`--accurate` (reason `vfr`), and `cut.py` then samples its packet timestamps (up
|
|
109
|
+
to five 6 s windows; no decoding) into `vfr_check` (`measured`: `sampled_cfr`,
|
|
110
|
+
`vfr` or `inconclusive`, plus `guard`); a `sampled_cfr` there adds a note naming
|
|
111
|
+
`--vfr-guard sampled`. `sampled` samples before every copy and re-encodes only on
|
|
112
|
+
`vfr` / `inconclusive` (reasons `vfr` / `vfr_inconclusive`), so a phone clip at
|
|
113
|
+
29.98 against a nominal 30 copies; `off` samples and keeps the copy with a note.
|
|
114
|
+
Irregular timing between the windows is not seen; `--accurate` is always available.
|
|
115
|
+
|
|
116
|
+
A source with no B-frames (iPhone "Most Compatible" H.264) cuts its parts as
|
|
117
|
+
every 2.x release did: from the keyframe at (or before) each start to the end
|
|
118
|
+
asked for, each part within `--tolerance`. On B-frame video a copied
|
|
119
|
+
`.mp4`/`.mov` part runs from the keyframe at (or before) its start to the first
|
|
120
|
+
keyframe at or after its end (never an earlier one), each end within
|
|
121
|
+
`--tolerance`, so the join is the source's own frames with no gaps; a later
|
|
122
|
+
segment that starts between keyframes there is re-cut (its hidden pre-roll
|
|
123
|
+
cannot cross a join). A source with **open GOPs** (x265's default, iPhone "High
|
|
124
|
+
Efficiency" HEVC) cannot be copied exactly across a join and is re-cut
|
|
125
|
+
(`concat_fallback`, named in `notes`). Every copy join is measured after it is
|
|
126
|
+
written (`join_check`): the frame count, each step at a join against the step
|
|
127
|
+
its parts predict, and each part's sound against its picture (audio packets
|
|
128
|
+
matched to the source; more than 5 ms off fails, a part with no unique match
|
|
129
|
+
is unmeasured). One that fails is re-cut. Only windows around the segments
|
|
130
|
+
(about 10 s each side, widened as needed) are read from the source, so a long
|
|
131
|
+
source costs what its segments do.
|
|
132
|
+
The parts are joined by stream copy only when they all copied and match: the
|
|
133
|
+
same streams with the same codec parameters, rotation, colour tags and
|
|
134
|
+
extradata. Otherwise every segment is **re-cut from
|
|
135
|
+
the source** into one re-encode (`concat_fallback`): frame-exact boundaries, the
|
|
136
|
+
audio's offset from the video kept, subtitles dropped and reported, and at most
|
|
137
|
+
32 segments per ffmpeg call. A segment shorter than one video frame is refused.
|
|
138
|
+
With `--accurate` the segments are always cut that way, in one encode (a part
|
|
139
|
+
encoded on its own ran an AAC frame past its picture, leaving a hole at every
|
|
140
|
+
join). A segment other than the last that runs past the end of the video is
|
|
141
|
+
ended with the video when it runs less than a frame past it, and otherwise keeps
|
|
142
|
+
its sound with the last frame held -- which forces the re-cut (`concat_fallback`,
|
|
143
|
+
named in `notes`). The last segment keeps the source's own sound-only tail.
|
|
77
144
|
|
|
78
145
|
**`--snap beats` (1.17)** moves each in/out point to the nearest *measured*
|
|
79
146
|
beat within `--snap-tolerance` seconds (default 0.12, about a quarter of a beat
|
|
@@ -550,7 +617,7 @@ Three refusals, all `kind: input`, all before any encode:
|
|
|
550
617
|
|
|
551
618
|
Results: `filler.removed_count`, `filler.removed_seconds`, `filler.removed`
|
|
552
619
|
(one entry per span), `filler.lang`, `filler.list`, `filler.word_timings` and
|
|
553
|
-
`filler.warnings
|
|
620
|
+
`filler.warnings`; `--filler-lang auto` refuses (`kind: input`) a transcript in a language with no built-in list, so pass `--filler-words FILE` for it. With `--transcribe`, `filler.source` is `whisper:ENGINE` or
|
|
554
621
|
`parakeet:ENGINE` and `filler.transcription` says which engine, model, language
|
|
555
622
|
and routing made the words; a top-level `notes` line says when `--engine auto`
|
|
556
623
|
ran the English-only Parakeet model on English assumed (no `--filler-lang`, and
|
|
@@ -905,6 +972,13 @@ batch.py FOLDER --recipe batch.json [--force] [--watch SECONDS] [--jobs N|auto]
|
|
|
905
972
|
placeholders, chained) or `project` (a render project applied per file).
|
|
906
973
|
Outputs land in `output_dir` with `suffix`; a content-hash cache skips files
|
|
907
974
|
already done with the same recipe. Use `--dry-run` to preview the plan.
|
|
975
|
+
A recipe that runs `cut.py` rolls its calls up: each of those `results` rows carries
|
|
976
|
+
`cut_reencoded` (each call's `reencoded`) and `cut_reencode_reasons` (each
|
|
977
|
+
`reencode_reason` its calls named, once per call),
|
|
978
|
+
and the top level `cut_stream_copy` (`{calls, stream_copy, reencoded,
|
|
979
|
+
stream_copy_rate}`) and `cut_reencode_reasons` (counts per reason; `unknown` for a
|
|
980
|
+
re-encoded call cached by an earlier 2.x batch); both are `null`
|
|
981
|
+
when no `cut.py` step ran.
|
|
908
982
|
|
|
909
983
|
**`--jobs N` (1.17)** processes N files at once (threads: the work is
|
|
910
984
|
subprocess waiting). Capped at `min(N, cpu_count, 8)` — every item is itself an
|
|
@@ -59,7 +59,7 @@ from _common.runner import (
|
|
|
59
59
|
ffmpeg_version, flush_drawtext_textfiles, INSTALL_HINTS, install_signal_handlers, _is_ffmpeg, _limit_for,
|
|
60
60
|
_odd_dimension_retry, _on_signal, _OutputLock, _pid_dead, place_output, PROBE_TIMEOUT, _progress_line,
|
|
61
61
|
read_text_or_die, refuse_output_is_input, _remember_output, require_tool, run, run_analysis, _run_captured,
|
|
62
|
-
run_keeping_subtitles, run_tool, _run_with_progress, shell_quote, _SIGNALS_INSTALLED, _stage_existing_output,
|
|
62
|
+
run_keeping_subtitles, run_tool, _run_with_progress, shell_quote, sibling_temp, _SIGNALS_INSTALLED, _stage_existing_output,
|
|
63
63
|
STATE, _timed_out, _unwatch, _watch, X264_PRESETS
|
|
64
64
|
)
|
|
65
65
|
from _common.emit import (
|
|
@@ -68,7 +68,7 @@ from _common.emit import (
|
|
|
68
68
|
)
|
|
69
69
|
from _common.probe import (
|
|
70
70
|
analyze_levels, _aspect_string, _bit_depth, decode_pcm_mono, detect_scenes, detect_silences, fingerprint,
|
|
71
|
-
_fraction, keyframes_near, SCORE_RE, SIL_RE, decode_gray_frames,
|
|
71
|
+
_fraction, keyframes_near, classify_frame_timing, measure_frame_timing, SCORE_RE, SIL_RE, decode_gray_frames,
|
|
72
72
|
measured_level_dbfs, AUDIO_MEDIA_EXT, MEDIA_EXT, _output_failed, probe, rms_envelope, _to_float, _to_int, verify_output
|
|
73
73
|
)
|
|
74
74
|
from _common.decision import (
|
|
@@ -83,7 +83,7 @@ from _common.decision import (
|
|
|
83
83
|
_onset_strength, _pick_onsets, _autocorrelation_peak, _grid_score,
|
|
84
84
|
brand_states_font, cfr_args, concat_list_line, db_to_linear, default_output, encoder_args, escape_filter_path,
|
|
85
85
|
fmt_secs, fmt_smpte_time, fmt_srt_time, frame_size, is_audio_output, load_brand, MissingFpsError, pad_filters, parse_time,
|
|
86
|
-
signed_time_arg, SVT_PRESET, time_arg, video_args, x264_args, _x264_raw
|
|
86
|
+
signed_time_arg, source_codec_video_args, SVT_PRESET, time_arg, video_args, x264_args, _x264_raw
|
|
87
87
|
)
|
|
88
88
|
from _common.asr import (
|
|
89
89
|
ASR_ENGINES, ASR_INSTALL_HINT, _asr_run, die_no_engine, parse_srt, transcribe, _transcribe_in,
|
|
@@ -200,7 +200,7 @@ __all__ = [
|
|
|
200
200
|
"font_family_for_script", "font_family_of_file", "FONT_FLAG_HINT", "font_for_script", "FONT_INSTALL_HINT",
|
|
201
201
|
"fonts_dir_covers_script", "_fraction", "has_emoji", "info", "INSTALL_HINTS", "install_signal_handlers",
|
|
202
202
|
"is_audio_output", "_is_emoji_base", "_is_emoji_char", "_is_ffmpeg", "_is_mark", "_KEYCAP", "_KEYCAP_BASES",
|
|
203
|
-
"keyframes_near", "LANGUAGE_NAMES", "LATIN_EM", "LEADING_VOWELS", "_libass_color_probe", "_limit_for",
|
|
203
|
+
"keyframes_near", "classify_frame_timing", "measure_frame_timing", "LANGUAGE_NAMES", "LATIN_EM", "LEADING_VOWELS", "_libass_color_probe", "_limit_for",
|
|
204
204
|
"load_brand", "measured_level_dbfs", "AUDIO_MEDIA_EXT", "MEDIA_EXT", "MissingFpsError", "needs_shaping", "NO_SPACE_SCRIPTS",
|
|
205
205
|
"_odd_dimension_retry", "_on_signal", "_output_failed", "_OutputLock", "pad_filters", "parse_time", "_pid_dead",
|
|
206
206
|
"place_output", "_plan_at_exit", "_plan_inputs", "_PLAN_STRIP", "PLAN_VERSION", "PREFERRED_FAMILIES",
|
|
@@ -209,10 +209,10 @@ __all__ = [
|
|
|
209
209
|
"decode_gray_frames", "frame_flow", "label_shot_flow", "MOTION_GRID", "MOTION_SEARCH", "MOTION_STATIC_PX", "MOTION_PAN_SPREAD",
|
|
210
210
|
"_run_captured", "run_keeping_subtitles", "run_tool", "_run_with_progress", "_SCRIPT_FONT_CACHE",
|
|
211
211
|
"_script_font_entry", "script_font_for_text", "script_font_status", "_script_font_uncached", "_SCRIPT_RANGES",
|
|
212
|
-
"SCRIPTS", "_sdr_bt709", "_set_current_ctx", "_SHAPING_BUILD_CACHE", "SHAPING_SCRIPTS", "shell_quote",
|
|
212
|
+
"SCRIPTS", "_sdr_bt709", "_set_current_ctx", "_SHAPING_BUILD_CACHE", "SHAPING_SCRIPTS", "shell_quote", "sibling_temp",
|
|
213
213
|
"_SIGNALS_INSTALLED", "signed_time_arg", "_stage_existing_output", "STATE", "SVT_PRESET", "text_width_em",
|
|
214
214
|
"time_arg", "_timed_out", "_to_float", "_to_int", "_unwatch", "validate_color", "verify_output",
|
|
215
|
-
"video_args", "_VS15", "_VS16", "_watch", "WINDOWS_FONTS", "write_plan", "x264_args", "X264_PRESETS",
|
|
215
|
+
"source_codec_video_args", "video_args", "_VS15", "_VS16", "_watch", "WINDOWS_FONTS", "write_plan", "x264_args", "X264_PRESETS",
|
|
216
216
|
"_x264_raw", "_ZWJ",
|
|
217
217
|
"ASR_ENGINES", "ASR_INSTALL_HINT", "_asr_run", "die_no_engine", "parse_srt", "transcribe",
|
|
218
218
|
"_transcribe_in", "transcribe_words", "_words_from_openai_whisper_json",
|
package/scripts/_common/asr.py
CHANGED
|
@@ -301,6 +301,7 @@ def _skipped_parakeet(route: "Route", language: Optional[str], language_flag: st
|
|
|
301
301
|
def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int,
|
|
302
302
|
ffmpeg: str, shutil, subprocess, wanted: str = "auto") -> Transcription:
|
|
303
303
|
from _common import run_analysis, STATE
|
|
304
|
+
language = engine_language(language)
|
|
304
305
|
wav = os.path.join(tmpdir, "audio.wav")
|
|
305
306
|
# A wav in our own temp dir: a measurement input for the engine, not a deliverable, so it
|
|
306
307
|
# is not a run() call (no --dry-run gate, not recorded), but it keeps the time limit and
|
|
@@ -360,12 +361,14 @@ def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str
|
|
|
360
361
|
import threading
|
|
361
362
|
result: list = []
|
|
362
363
|
crashed: list = []
|
|
364
|
+
heard: list = [] # the language faster-whisper decoded, when it names one
|
|
363
365
|
|
|
364
366
|
def work() -> None:
|
|
365
367
|
try:
|
|
366
368
|
m = WhisperModel(model, device="cpu", compute_type="int8")
|
|
367
|
-
segments,
|
|
369
|
+
segments, decoded = m.transcribe(wav, language=lang, word_timestamps=False)
|
|
368
370
|
result.extend((seg.start, seg.end, seg.text.strip()) for seg in segments if seg.text.strip())
|
|
371
|
+
heard.append(getattr(decoded, "language", None))
|
|
369
372
|
except Exception as exc: # noqa: BLE001 -- a model that would not load or run: a failed engine
|
|
370
373
|
crashed.append(exc)
|
|
371
374
|
|
|
@@ -388,7 +391,8 @@ def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str
|
|
|
388
391
|
info("transcribed with faster-whisper")
|
|
389
392
|
write_srt(cues, out_srt)
|
|
390
393
|
return Transcription(cues=cues, engine="faster-whisper",
|
|
391
|
-
facts=dict(facts, engine="faster-whisper", model=model,
|
|
394
|
+
facts=dict(facts, engine="faster-whisper", model=model,
|
|
395
|
+
language=lang or _heard_language(heard)))
|
|
392
396
|
except ImportError:
|
|
393
397
|
pass
|
|
394
398
|
# 3. openai-whisper CLI
|
|
@@ -508,6 +512,23 @@ def _words_from_whisper_cpp_json(path: str) -> "List[Dict[str, Any]]":
|
|
|
508
512
|
return [w for w in out if w["word"].strip()]
|
|
509
513
|
|
|
510
514
|
|
|
515
|
+
def _whisper_json_language(path: str) -> Optional[str]:
|
|
516
|
+
"""The language a Whisper engine's JSON says it decoded: whisper.cpp's `result.language`, or
|
|
517
|
+
openai-whisper's top-level `language`. None when the file has neither."""
|
|
518
|
+
try:
|
|
519
|
+
doc = json.loads(Path(path).read_text(encoding="utf-8")) or {}
|
|
520
|
+
lang = str(((doc.get("result") or {}).get("language")) or doc.get("language") or "").strip().lower()
|
|
521
|
+
except (OSError, ValueError, AttributeError):
|
|
522
|
+
return None
|
|
523
|
+
return lang or None
|
|
524
|
+
|
|
525
|
+
|
|
526
|
+
def _heard_language(heard: list) -> Optional[str]:
|
|
527
|
+
"""The language an in-process engine decoded (faster-whisper's info.language), lower-cased, or None."""
|
|
528
|
+
lang = next((str(h).strip().lower() for h in heard if h), "")
|
|
529
|
+
return lang or None
|
|
530
|
+
|
|
531
|
+
|
|
511
532
|
def _words_from_openai_whisper_json(path: str) -> "List[Dict[str, Any]]":
|
|
512
533
|
"""openai-whisper --word_timestamps True --output_format json: segments[].words[]."""
|
|
513
534
|
try:
|
|
@@ -644,6 +665,22 @@ def _is_english(language: Optional[str]) -> bool:
|
|
|
644
665
|
return (language or "").lower().split("-")[0].split("_")[0] in ("en", "eng", "english")
|
|
645
666
|
|
|
646
667
|
|
|
668
|
+
def engine_language(language: Optional[str]) -> Optional[str]:
|
|
669
|
+
"""`--language` as the engines take it. "auto" (caption.py's documented default, spelled out)
|
|
670
|
+
is no language: None, the undetected route. Every spelling of English (en-US, eng, English)
|
|
671
|
+
is "en", which Parakeet accepts as English and Whisper accepts as a language code; any other
|
|
672
|
+
value is its two-letter primary language (ja-JP -> ja)."""
|
|
673
|
+
named = (language or "").strip()
|
|
674
|
+
if not named or named.lower() == "auto":
|
|
675
|
+
return None
|
|
676
|
+
if _is_english(named):
|
|
677
|
+
return "en"
|
|
678
|
+
# Whisper takes the two-letter code: a regional or script subtag (ja-JP, pt-BR, zh-Hant) names the
|
|
679
|
+
# same language to it, and an ISO 639-2 code (jpn) is passed as given
|
|
680
|
+
primary = named.replace("_", "-").split("-")[0].lower()
|
|
681
|
+
return primary if len(primary) == 2 else named
|
|
682
|
+
|
|
683
|
+
|
|
647
684
|
def _whisper_cpp_cli(shutil) -> Optional[str]:
|
|
648
685
|
cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp")
|
|
649
686
|
if not cli:
|
|
@@ -675,7 +712,7 @@ def detect_language(wav: str, shutil, subprocess) -> Optional[str]:
|
|
|
675
712
|
if not cli:
|
|
676
713
|
return None
|
|
677
714
|
model = next((p for p in (_whisper_cpp_model(m) for m in ("tiny", "base", "small", "tiny.en"))
|
|
678
|
-
if os.path.exists(p) and not p
|
|
715
|
+
if os.path.exists(p) and not _english_only_whisper_model(p)), None)
|
|
679
716
|
if not model:
|
|
680
717
|
return None
|
|
681
718
|
proc = _asr_run([cli, "-m", model, "-f", wav, "-dl", "-l", "auto"], subprocess, "whisper.cpp language detection")
|
|
@@ -756,9 +793,10 @@ def _names_parakeet_model(name: str) -> bool:
|
|
|
756
793
|
|
|
757
794
|
|
|
758
795
|
def _english_only_whisper_model(name: str) -> bool:
|
|
759
|
-
"""An English-only Whisper model: tiny.en, base.en, ... or a ggml-*.en.bin file
|
|
796
|
+
"""An English-only Whisper model: tiny.en, base.en, ... or a ggml-*.en.bin file, in any
|
|
797
|
+
quantisation (ggml-base.en-q5_1.bin, tiny.en-q8_0) or variant (ggml-small.en-tdrz.bin)."""
|
|
760
798
|
base = os.path.basename(name).lower()
|
|
761
|
-
return (base[:-4] if base.endswith(".bin") else base)
|
|
799
|
+
return re.search(r"\.en(-[a-z0-9_]+)*$", base[:-4] if base.endswith(".bin") else base) is not None
|
|
762
800
|
|
|
763
801
|
|
|
764
802
|
def _whisper_ready(shutil, model: Optional[str]) -> bool:
|
|
@@ -913,6 +951,7 @@ def transcribe_words_result(video: str, language: "Optional[str]" = None, model:
|
|
|
913
951
|
from _common import require_tool, run_analysis, STATE
|
|
914
952
|
ffmpeg = require_tool("ffmpeg")
|
|
915
953
|
wanted = requested_engine(engine)
|
|
954
|
+
language = engine_language(language)
|
|
916
955
|
parakeet_model = None if model == "base" else model
|
|
917
956
|
tmpdir = _tempfile.mkdtemp(prefix="ffskill_asrw_")
|
|
918
957
|
try:
|
|
@@ -962,7 +1001,8 @@ def transcribe_words_result(video: str, language: "Optional[str]" = None, model:
|
|
|
962
1001
|
words = _words_from_whisper_cpp_json(base + ".json")
|
|
963
1002
|
info(f"word timings from whisper.cpp ({len(words)} words)")
|
|
964
1003
|
return Transcription(words=words, engine="whisper.cpp",
|
|
965
|
-
facts=dict(facts, engine="whisper.cpp", model=model_path,
|
|
1004
|
+
facts=dict(facts, engine="whisper.cpp", model=model_path,
|
|
1005
|
+
language=lang or _whisper_json_language(base + ".json")))
|
|
966
1006
|
line = _failure_line(proc)
|
|
967
1007
|
info("whisper.cpp found but produced no word-timing JSON: " + line)
|
|
968
1008
|
tried.append(_failed("whisper.cpp", "found but produced no word-timing JSON: " + line))
|
|
@@ -976,11 +1016,13 @@ def transcribe_words_result(video: str, language: "Optional[str]" = None, model:
|
|
|
976
1016
|
import threading
|
|
977
1017
|
collected: list = []
|
|
978
1018
|
crashed: list = []
|
|
1019
|
+
heard: list = []
|
|
979
1020
|
|
|
980
1021
|
def work() -> None:
|
|
981
1022
|
try:
|
|
982
1023
|
m = WhisperModel(model, device="cpu", compute_type="int8")
|
|
983
|
-
segments,
|
|
1024
|
+
segments, decoded = m.transcribe(wav, language=lang, word_timestamps=True)
|
|
1025
|
+
heard.append(getattr(decoded, "language", None))
|
|
984
1026
|
for seg in segments:
|
|
985
1027
|
for w in (getattr(seg, "words", None) or []):
|
|
986
1028
|
collected.append({"word": str(w.word).strip(),
|
|
@@ -1002,7 +1044,8 @@ def transcribe_words_result(video: str, language: "Optional[str]" = None, model:
|
|
|
1002
1044
|
else:
|
|
1003
1045
|
info(f"word timings from faster-whisper ({len(collected)} words)")
|
|
1004
1046
|
return Transcription(words=[w for w in collected if w["word"]], engine="faster-whisper",
|
|
1005
|
-
facts=dict(facts, engine="faster-whisper", model=model,
|
|
1047
|
+
facts=dict(facts, engine="faster-whisper", model=model,
|
|
1048
|
+
language=lang or _heard_language(heard)))
|
|
1006
1049
|
except ImportError:
|
|
1007
1050
|
pass
|
|
1008
1051
|
|
|
@@ -1018,7 +1061,8 @@ def transcribe_words_result(video: str, language: "Optional[str]" = None, model:
|
|
|
1018
1061
|
words = _words_from_openai_whisper_json(doc)
|
|
1019
1062
|
info(f"word timings from openai-whisper ({len(words)} words)")
|
|
1020
1063
|
return Transcription(words=words, engine="openai-whisper",
|
|
1021
|
-
facts=dict(facts, engine="openai-whisper", model=model,
|
|
1064
|
+
facts=dict(facts, engine="openai-whisper", model=model,
|
|
1065
|
+
language=lang or _whisper_json_language(doc)))
|
|
1022
1066
|
line = _failure_line(proc)
|
|
1023
1067
|
info("openai-whisper found but produced no word-timing JSON: " + line)
|
|
1024
1068
|
tried.append(_failed("openai-whisper", "found but produced no word-timing JSON: " + line))
|
|
@@ -478,6 +478,19 @@ def video_args(meta: Optional[Dict[str, Any]], crf: int = 18, preset: str = "med
|
|
|
478
478
|
return _maybe_hw("hevc", crf, meta, True, _x265_hdr_default(v, crf, preset))
|
|
479
479
|
|
|
480
480
|
|
|
481
|
+
def source_codec_video_args(meta: Optional[Dict[str, Any]], crf: int = 18, preset: str = "medium",
|
|
482
|
+
keep_hevc: bool = False) -> List[str]:
|
|
483
|
+
"""video_args(), except that with `keep_hevc` (cut.py --keep-hevc) an SDR HEVC source re-encodes
|
|
484
|
+
to HEVC (8-bit, BT.709-tagged, on VideoToolbox when --hw asks) instead of H.264: a cut is a trim,
|
|
485
|
+
so a re-encoded segment can come out in the codec the source (and any copied segment beside it)
|
|
486
|
+
is in. Opt-in within 2.x, where an SDR re-encode stays x264 unless asked. An HDR or BT.2020
|
|
487
|
+
source always takes video_args()'s Main10 line, and --codec always wins."""
|
|
488
|
+
v = (meta or {}).get("video") or {}
|
|
489
|
+
if keep_hevc and not STATE.codec and v.get("codec") == "hevc" and not v.get("bt2020_or_hdr"):
|
|
490
|
+
return encoder_args("hevc", crf, preset, meta)
|
|
491
|
+
return video_args(meta, crf, preset)
|
|
492
|
+
|
|
493
|
+
|
|
481
494
|
def _x265_hdr_default(v: Dict[str, Any], crf: int, preset: str) -> List[str]:
|
|
482
495
|
cs = v.get("color_space") or "bt2020nc"
|
|
483
496
|
prim = v.get("color_primaries") or "bt2020"
|
|
@@ -115,7 +115,7 @@ def drawtext_text_opts(text: str, tmpdir: "Optional[str]" = None) -> str:
|
|
|
115
115
|
`]`, `;` and `\` all reach the picture verbatim -- 1.15 fixes `overlay.py --text "it's 100%
|
|
116
116
|
done"` losing both characters. A newline is kept (`\r\n` and `\r` become `\n`): drawtext
|
|
117
117
|
draws it as a line break, and graphics.py's wrap puts them there on purpose (stripping them
|
|
118
|
-
ran a wrapped title off both edges of the frame). A tab, vertical tab or form
|
|
118
|
+
ran a wrapped title off both edges of the frame); trailing newlines are dropped. A tab, vertical tab or form
|
|
119
119
|
feed becomes a space, as the wrap measures it; other control characters are stripped (that
|
|
120
120
|
dates from the inline-escape route, where they broke the graph parser).
|
|
121
121
|
|
|
@@ -127,6 +127,9 @@ def drawtext_text_opts(text: str, tmpdir: "Optional[str]" = None) -> str:
|
|
|
127
127
|
"""
|
|
128
128
|
cleaned = (text or "").replace("\r\n", "\n").replace("\r", "\n")
|
|
129
129
|
cleaned = re.sub(r"[\x00-\x08\x0e-\x1f\x7f]", "", re.sub(r"[\t\x0b\x0c]", " ", cleaned))
|
|
130
|
+
# drawtext ignores one trailing newline and draws a second as a blank line, which would
|
|
131
|
+
# centre the block half a line too high; a label that is only newlines stays as it is
|
|
132
|
+
cleaned = cleaned.rstrip("\n") or cleaned
|
|
130
133
|
import hashlib
|
|
131
134
|
name = "t_" + hashlib.sha256(cleaned.encode("utf-8")).hexdigest()[:16] + ".txt"
|
|
132
135
|
if tmpdir is None:
|