ffmpeg-skill 1.17.3 → 1.18.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/SKILL.md +18 -15
- package/docs/contract.md +22 -10
- package/package.json +1 -1
- package/references/ci-platform-pitfalls.md +1 -1
- package/references/scripts.md +85 -10
- package/scripts/_common/__init__.py +8 -5
- package/scripts/_common/decision.py +86 -0
- package/scripts/_common/drawtext.py +125 -0
- package/scripts/_common/emoji.py +350 -0
- package/scripts/_common/fonts.py +437 -0
- package/scripts/_common/probe.py +26 -0
- package/scripts/_common/text.py +59 -1642
- package/scripts/_common/wrap.py +778 -0
- package/scripts/_contract.py +8 -3
- package/scripts/cropdetect.py +59 -1
- package/scripts/multicam.py +85 -5
- package/scripts/scenes.py +87 -2
- package/scripts/silence.py +46 -1
- package/scripts/sync.py +100 -52
package/SKILL.md
CHANGED
|
@@ -7,21 +7,21 @@ description: 'Edit video and audio with local FFmpeg from natural-language reque
|
|
|
7
7
|
|
|
8
8
|
Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`, and delivery templates in `templates/`. This file is enough to do a job: the table below routes the request, and `--help` on the script you are about to run is the cheapest full flag list. A reference file costs as much to read as this file; open one only for a question you have: `references/scripts.md` (every flag of all 42 scripts), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end).
|
|
9
9
|
|
|
10
|
-
Shared flags, on every script: `--dry-run`; `--json` (output path, a probe of the output, the commands run); `--json-brief` (
|
|
10
|
+
Shared flags, on every script: `--dry-run`; `--json` (output path, a probe of the output, the commands run); `--json-brief` (trimmed to status/output/verified plus a `summary`; prefer it on writing steps); `--fast` (preview quality); `--progress`; `--timeout SECONDS` (`kind: timeout`, default 1800); `--overwrite` (step 7); `--plan FILE` (the dry run as a plan `render.py FILE` runs later; refuses if an input changed). Re-encoding tools also take `--codec h264|hevc|av1|prores` and `--quality N`: unset, SDR is x264, HDR is x265 Main10; `prores` needs `-o NAME.mov`, `h264` refuses HDR (`color.py --to-sdr` first).
|
|
11
11
|
|
|
12
12
|
Writing tools run nothing under `--dry-run`; the measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) may still run ffmpeg/ffprobe — they just don't write their artifact, nor side files (`--edl`, `--sheet`, a generated `.ass`); `verify` ignores the flag. Per-tool: `contract --json`'s `dry_run` field.
|
|
13
13
|
|
|
14
14
|
## Workflow (always follow this order)
|
|
15
15
|
|
|
16
|
-
0. **Environment, only on failure.** Never start a job with `doctor`: a broken machine fails on its own with `kind: missing_tool` or an ffmpeg error naming the filter/encoder. Run `python3 <skill-dir>/scripts/_contract.py doctor` (or `npx ffmpeg-skill doctor`) after such a failure, or when asked what the machine can do: read `ok
|
|
17
|
-
1. **Probe what you must plan from.** Run `probe.py` on each input you plan
|
|
16
|
+
0. **Environment, only on failure.** Never start a job with `doctor`: a broken machine fails on its own with `kind: missing_tool` or an ffmpeg error naming the filter/encoder. Run `python3 <skill-dir>/scripts/_contract.py doctor` (or `npx ffmpeg-skill doctor`) after such a failure, or when asked what the machine can do: read `ok`/`usable`, report the missing capability. `contract --json`'s tool schema is for a *planning* agent, not this workflow.
|
|
17
|
+
1. **Probe what you must plan from.** Run `probe.py` on each input you plan from — duration, fps, resolution, codecs, channels, `variable_frame_rate_suspected` — and whenever the user asks about a file. No separate probe before every edit: every writing tool's `--json` already carries its input and a probe of the output. Plan from real numbers, never assumptions.
|
|
18
18
|
2. **Prefer lossless.** If the request can be met without re-encoding (plain cuts on keyframes, remuxing, audio-only changes), do not re-encode. `cut.py` and `loudness.py` stream-copy video by default; `--accurate` on `cut.py` only for frame-exact cuts.
|
|
19
|
-
3. **Plan with `--dry-run --json`, then execute.** Trust `--json`, not a dry run's summary line, for any number in the plan (a dimension there can be a placeholder). Use it before long encodes
|
|
20
|
-
4. **Chain in a sensible order.** A delivery request with no other editing is one template run (`render.py --template NAME INPUT`), not a hand-built chain. Otherwise: colour (HDR→SDR / LUT) → cut → join → silence → fit → caption/overlay → sync → audio → loudness → export. Frame changes before captions
|
|
21
|
-
5. **Check the deliverable.** Before reporting, run `check.py OUTPUT --platform X` for the destination
|
|
19
|
+
3. **Plan with `--dry-run --json`, then execute.** Trust `--json`, not a dry run's summary line, for any number in the plan (a dimension there can be a placeholder). Use it before long encodes, to report facts. `--fast` is preview quality, `--progress` prints percent/ETA on stderr. Never point `-o` at a file you did not create in this job unless the user asked for it to be replaced; pass `--overwrite` only then.
|
|
20
|
+
4. **Chain in a sensible order.** A delivery request with no other editing is one template run (`render.py --template NAME INPUT`), not a hand-built chain. Otherwise: colour (HDR→SDR / LUT) → cut → join → silence → fit → caption/overlay → sync → audio → loudness → export. Frame changes before captions, so text is sized for the final frame. Re-encode as few times as possible: intermediates at CRF 18, `export.py` last. **Three or more steps: `render.py` with a project.json.**
|
|
21
|
+
5. **Check the deliverable.** Before reporting, run `check.py OUTPUT --platform X` for the destination named (a template run already does). Format rows (codec, pixel format, size, true peak, colour tags, VFR) are safe to fix. Judgement rows change content: duration (cut loses material), aspect (crop loses edges), fps (drops motion), loudness (ambience must not be boosted) — fix only when the request implies the answer, else state the choice and its cost. Mention WARNs; do not chase them.
|
|
22
22
|
6. **Verify the output.** Confirm duration, resolution, fps and audio match the request — from the writing tool's own `--json`/`--json-brief` probe, or `probe.py` — and report those numbers ("final.mp4: 59.98 s, 1080x1920, 30 fps, AAC stereo"). A step is done only when the script exited 0 and the output probes as expected: a non-zero exit, a missing or empty file, or a probe that contradicts the request is a failure reported with the script's error message.
|
|
23
|
-
7. **Keep the user's originals.** Never overwrite the source; write new files next to
|
|
24
|
-
8. **Look at the picture.** Whenever the picture changed (captions, overlays, graphics, crop/pad, resize, colour, transitions, a `join.py`
|
|
23
|
+
7. **Keep the user's originals.** Never overwrite the source; write new files next to input or where asked. Set `FFMPEG_SKILL_NO_OVERWRITE=1` in the environment: an existing output path is then refused (`kind: input`) instead of warned about, `--overwrite` the one way to say "yes, replace it". Recommended agent setting, and 2.0's default.
|
|
24
|
+
8. **Look at the picture.** Whenever the picture changed (captions, overlays, graphics, crop/pad, resize, colour, transitions, a `join.py` scaling a clip to the first clip's frame, `color.py --to-sdr`) run `look.py OUTPUT --tiles 3x2` (or `--at T` for one frame) and view the PNG; the full 4x3 sheet is for a job about layout across the whole clip. Not finished until `Look:` names that PNG — a probe cannot see a caption on someone's face. Audio-only jobs write `Look: not needed`. What to look for splits like `check.py`'s rows in step 5:
|
|
25
25
|
- **Mechanical (this skill's own job to verify and report):** the specified text/logo is at the specified position, subtitles appear at the specified timestamps, dimensions are even. Letterboxing from `fit.py --fit pad` is the *correct* result of that mode, never a defect to flag.
|
|
26
26
|
- **Judgement (report it, don't silently pass or fail):** whether a subject or face is cut off, text sits over a face, colours look washed out, a transition lands. These need deciding what the subject *is*, which belongs to the calling agent — say what you see in one line and let them judge it.
|
|
27
27
|
With no vision capability, write `Look: PATH (pixels not inspected; agent has no image view)` — never claim a picture was inspected when it wasn't, and don't stall waiting for a capability that isn't there.
|
|
@@ -52,7 +52,7 @@ This skill cuts, joins, measures, syncs, exports and checks files — it execute
|
|
|
52
52
|
|
|
53
53
|
The line: same input + same explicit parameters always producing the same verifiable output belongs here; anything depending on taste, content understanding or what looks or sounds good belongs to whoever makes that judgement.
|
|
54
54
|
|
|
55
|
-
If a request needs an FFmpeg feature none of the 42 scripts expose, say so and name the closest built-in option — never guess a raw `ffmpeg`/`ffprobe` invocation
|
|
55
|
+
If a request needs an FFmpeg feature none of the 42 scripts expose, say so and name the closest built-in option — never guess a raw `ffmpeg`/`ffprobe` invocation outside `scripts/*.py`. A raw command bypasses every guarantee this skill makes, so it is never the fallback when a script's flag doesn't cover something.
|
|
56
56
|
|
|
57
57
|
## Request → script
|
|
58
58
|
|
|
@@ -100,21 +100,24 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
|
|
|
100
100
|
| "webcam clip in the corner", "picture-in-picture" | `overlay.py input.mp4 --video webcam.mp4 --position bottom-right --scale 480` |
|
|
101
101
|
| "remove the green screen" | `overlay.py bg.mp4 --video greenscreen.mp4 --chromakey 0x00ff00` |
|
|
102
102
|
| "turn this podcast into a video", "audiogram" | `render.py --template audiogram ep.m4a --image cover.png` — waveform over a still or colour plate; give an image or colour, nothing is fetched |
|
|
103
|
-
| "sync the lav mic", "line up two cameras" | `sync.py camera.mp4 mic.wav --replace-audio` / `sync.py camA.mp4 camB.mp4 --trim-second` |
|
|
103
|
+
| "sync the lav mic", "line up two cameras" | `sync.py camera.mp4 mic.wav --replace-audio` / `sync.py camA.mp4 camB.mp4 --trim-second`; a third+ recorder is another positional (`sync.py ref.mp4 mic.wav cam2.mp4`), one offsets JSON |
|
|
104
104
|
| "fix the audio levels", "normalise to -14 LUFS" | `loudness.py input.mp4` (`-I -16 --tp -1.5` podcast, `-I -23` broadcast; `--lra N` for the range) |
|
|
105
105
|
| "cut this and make it HEVC / AV1 / ProRes" (output codec named) | `cut.py input.mp4 --start 0:10 --end 0:40 --codec hevc` (`--codec`/`--quality` on any re-encoding tool; ProRes needs `-o NAME.mov`) |
|
|
106
|
-
| "make this a TikTok / Reel / Short / YouTube / X / LinkedIn / podcast" | `render.py --template tiktok\|reels\|shorts\|youtube-shorts\|youtube\|x\|linkedin\|facebook\|podcast input.mp4 [--cues cues.txt
|
|
106
|
+
| "make this a TikTok / Reel / Short / YouTube / X / LinkedIn / podcast" | `render.py --template tiktok\|reels\|shorts\|youtube-shorts\|youtube\|x\|linkedin\|facebook\|podcast input.mp4 [--cues cues.txt] [--title "..."]` — frame, captions, loudness, export and check in one command (`--list-templates`, `--write-project` to edit first) |
|
|
107
107
|
| "post it everywhere", "one edit for every platform" | `render.py --template all input.mp4 --cues cues.txt` (or a comma list) → one file per destination plus `<name>_pack.md` (`report.py --pack` renders the HTML) |
|
|
108
108
|
| "export for YouTube / Reels / X", "a ProRes master" | `export.py input.mp4 --preset youtube\|reels\|tiktok\|shorts\|linkedin\|facebook\|x\|prores\|h265` (`--normalize` hits the loudness spec in the same call; `youtube-hdr` keeps HDR, `youtube-av1` writes AV1) |
|
|
109
109
|
| "make a GIF preview" | `export.py input.mp4 --preset gif` |
|
|
110
110
|
| "a small proxy / cheap preview file" | `proxy.py input.mp4 [--width 640 --no-audio]` — not a delivery preset (that is `export.py`) |
|
|
111
111
|
| "cut out the pauses", "jump cuts" | `silence.py input.mp4 [--threshold -40 --min-silence 0.8]` |
|
|
112
112
|
| "cut the ums and uhs", "remove the filler words" | `silence.py input.mp4 --filler --words words.json` (measured word timings; `--transcribe` makes them) |
|
|
113
|
+
| "don't cut inside a sentence, just the real pauses" | `silence.py input.mp4 --speech-aware` — a breath under `--min-silence` inside a sentence is kept, only sentence-boundary pauses cut; composes with `--filler` into one list |
|
|
113
114
|
| "stitch these clips", "add a crossfade" | `join.py a.mp4 b.mp4 c.mp4 --transition fade --duration 0.5` |
|
|
114
115
|
| "show me what it looks like", "are the captions readable" | `look.py output.mp4 --tiles 3x2`, then view the PNG |
|
|
115
116
|
| "what would you run?", "don't render yet" | any script with `--dry-run` |
|
|
116
117
|
| "a 60 s highlight from this hour" | `scenes.py long.mp4 --highlights 6 --target 60 --edl picks.txt` → `cut.py --segments` |
|
|
117
118
|
| "cut on the beat", "edit it to the music" | `scenes.py track.mp4 --beats --json > beats.json`, then `cut.py input.mp4 --segments ... --snap beats --snap-source beats.json` (`--snap-source` carries the measured grid over) |
|
|
119
|
+
| "which shots are static vs moving", "volume peaks per second", "is this speech or music" | `scenes.py input.mp4 --shots` (static/pan/motion per shot, measured flow) / `--audio-peaks` (dBFS list) / `--speech` (a speech-vs-music ratio, not a classification) — combine, or alone |
|
|
120
|
+
| "where does the subject move, so I can crop it myself" | `cropdetect.py input.mp4 --motion-centre` — motion centroid per second, report-only; the calling agent picks the crop |
|
|
118
121
|
| "is this OK to upload?" | `check.py final.mp4 --platform reels` |
|
|
119
122
|
| "a podcast episode with chapters" | `loudness.py ep.wav -I -16 --tp -1.5` → `metadata.py ep.m4a --chapters chapters.txt` → `check.py ep.m4a --platform podcast` (chapters and channels rows) |
|
|
120
123
|
| "several changes to the same edit", 3+ steps | `render.py --init project.json`, edit, `render.py project.json` |
|
|
@@ -126,7 +129,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
|
|
|
126
129
|
| "send me a summary of what you did" | `report.py --before raw.mov --after final.mp4 --platform youtube -o report.html` |
|
|
127
130
|
| "do this to every file in the folder", "use all the cores" | `batch.py FOLDER --recipe batch.json --jobs auto` (steps or a render project; cached) |
|
|
128
131
|
| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (needs a local whisper; else `--text`) |
|
|
129
|
-
| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` |
|
|
132
|
+
| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` (manual) or `--switch energy` (auto-cuts to the loudest camera, `--min-shot`, `--edl`) |
|
|
130
133
|
| "iPhone Dolby Vision clip looks wrong" | `color.py clip.mov --to-sdr` or `--strip-dovi` (keep HDR, drop the DV layer) |
|
|
131
134
|
| "does it look like Log / S-Log?" | `probe.py clip.mp4 --analyze` (`looks_like_log`) then `color.py --lut` |
|
|
132
135
|
| "test the tool on my real files" | `verify.py ~/Footage --report verify.md` |
|
|
@@ -150,12 +153,12 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
|
|
|
150
153
|
|
|
151
154
|
## Audio-only files
|
|
152
155
|
|
|
153
|
-
Audio is a first-class input: `probe
|
|
156
|
+
Audio is a first-class input: `probe`, `cut`, `silence`, `loudness`, `audio`, `sync`, `check --platform podcast`, `render --template podcast` take WAV, FLAC, MP3, M4A/AAC, OGG, Opus; output extension picks the format. Scripts needing a picture (`fit`, `caption`, `overlay`, `graphics`, `color`, `export`, `scenes`, `look`) refuse audio with "input has no video stream" — don't force a video wrapper. Recipes: `references/gotchas.md#audio-only-files`.
|
|
154
157
|
|
|
155
158
|
|
|
156
159
|
## Report format
|
|
157
160
|
|
|
158
|
-
Reply in the language the request
|
|
161
|
+
Reply in the language the request is written in — subtitles in another language are still reported in the request's language. Keep the labels (`Done:`, `Steps:`, `Check:`, `Look:`, `Notes:`) in English; everything else is the user's language. Never drift because the job was short or failed: even a one-line "file does not exist". A mid-conversation switch follows the user's latest message.
|
|
159
162
|
|
|
160
163
|
Finish every job with this shape (numbers from `--json` or `probe.py`/`check.py`, not memory):
|
|
161
164
|
|
|
@@ -192,7 +195,7 @@ Notes: send a valid .cube, or say if you want the clip left as is
|
|
|
192
195
|
Those filler lines are sentences, not labels: the same report for a Japanese request ends
|
|
193
196
|
`Check: 検証するものなし` / `Look: 不要`.
|
|
194
197
|
|
|
195
|
-
A refusal
|
|
198
|
+
A refusal uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure gets all five labels, never headings. A partial result is `Done:` with the shortfall in `Notes:`; a refusal that still delivers something is `Failed:` — never a third label like `Done (partially):`. Quote a failure's `error.hint` in `Notes:`: it names the flag change a retry needs.
|
|
196
199
|
|
|
197
200
|
Every script prints `{"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool | timeout | verification | interrupted, "message": ...}}` with `--json` and exits non-zero; quote the message, never paraphrase it.
|
|
198
201
|
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`1.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`1.18.1`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -88,19 +88,19 @@ spelling keeps working until 2.0.
|
|
|
88
88
|
|
|
89
89
|
| What 2.0 removes | Since | Replacement | To be ready today |
|
|
90
90
|
|---|---|---|---|
|
|
91
|
-
| The per-tool v1 success keys next to `result_v2` (`output`, `probe`, `commands`, `verified`, `verification` and each tool's own keys at the top level) | 1.
|
|
92
|
-
| `--crf` as an alias of `--quality` on every re-encoding tool that takes `--quality` (`export.py` keeps `--crf`: its preset chooses the encoder) | 1.
|
|
93
|
-
| `json` and `progress` in the MCP `inputSchema` | 1.
|
|
94
|
-
| `hdr` meaning "BT.2020 primaries *or* a PQ/HLG transfer" in `probe` | 1.
|
|
95
|
-
| Overwriting an existing output with only a warning | 1.
|
|
91
|
+
| The per-tool v1 success keys next to `result_v2` (`output`, `probe`, `commands`, `verified`, `verification` and each tool's own keys at the top level) | 1.18.1 | `result_v2`, promoted to the top level in 2.0 | Run with `FFMPEG_SKILL_RESULT_V2=1` and read `result_v2` (`metrics`, `notes`, `details`) instead of the top-level keys |
|
|
92
|
+
| `--crf` as an alias of `--quality` on every re-encoding tool that takes `--quality` (`export.py` keeps `--crf`: its preset chooses the encoder) | 1.18.1 | `--quality N` (the same CRF scale, codec-neutral) | Pass `--quality`; `--crf` warns on stderr and is marked in `--help` |
|
|
93
|
+
| `json` and `progress` in the MCP `inputSchema` | 1.18.1 | nothing: the transport sets them itself | Stop sending them from an MCP client; run the server with `FFMPEG_SKILL_MCP_LEAN=1` to see the 2.0 schema |
|
|
94
|
+
| `hdr` meaning "BT.2020 primaries *or* a PQ/HLG transfer" in `probe` | 1.18.1 | `hdr_signal` (true only for PQ / HLG / Dolby Vision); in 2.0 `hdr` takes that meaning | Key on `hdr_signal` for "is this a real HDR signal" and on `hdr_format` for the `BT.2020 SDR` case |
|
|
95
|
+
| Overwriting an existing output with only a warning | 1.18.1 | `--overwrite` as explicit consent (refused without it from 2.0) | Set `FFMPEG_SKILL_NO_OVERWRITE=1` (the recommended agent setting) and pass `--overwrite` where a replacement is intended |
|
|
96
96
|
|
|
97
97
|
## Skill
|
|
98
98
|
|
|
99
99
|
```json
|
|
100
100
|
{
|
|
101
101
|
"contract_version": "1.0",
|
|
102
|
-
"deprecated": [{"what": "...", "since": "1.
|
|
103
|
-
"skill": {"id": "ffmpeg-skill", "version": "1.
|
|
102
|
+
"deprecated": [{"what": "...", "since": "1.18.1", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
103
|
+
"skill": {"id": "ffmpeg-skill", "version": "1.18.1", "execution_mode": "local", "kind": "execution",
|
|
104
104
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
105
105
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
106
106
|
"requirements": {"python": ">=3.9 (standard library only)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
|
|
@@ -128,7 +128,7 @@ One entry per tool under `tools`, sorted by id. Tool ids are stable:
|
|
|
128
128
|
| `output_schema` | what `--json` prints on stdout |
|
|
129
129
|
| `supports_dry_run`, `dry_run` | whether `--dry-run` plans without running ffmpeg or writing files |
|
|
130
130
|
| `supports_json` | whether `--json` exists |
|
|
131
|
-
| `supports_json_brief` | whether `--json-brief` exists (1.
|
|
131
|
+
| `supports_json_brief` | whether `--json-brief` exists (1.18.1): the same success document with `probe` replaced by a compact `summary` (`duration_s`, `width`, `height`, `fps`, `vcodec`, `acodec`, `channels`, and `lufs` when the tool measured one), `commands` replaced by the number of commands run, and the per-step `verification` list dropped (its verdict stays in `verified`). Tool-specific keys are unchanged, `--json`'s own output is unchanged, and a failure prints the same failure document either way |
|
|
132
132
|
| `mutates_input` | always `false`: no tool overwrites its input |
|
|
133
133
|
| `produces_artifact` | writes a file (media, PNG, HTML, EDL) |
|
|
134
134
|
| `verification` | `{required, tools}`: which tools to run on the output afterwards |
|
|
@@ -450,7 +450,7 @@ Per-tool keys added in 1.17, all additive:
|
|
|
450
450
|
| `jobs`, `jobs_requested`, `wall_seconds`, `item_seconds_total`, `timed_out` | `batch.py` | the parallelism actually applied and the number asked for, the batch's wall clock, the sum of the per-item times (so the speed-up can be quoted), and whether the shared timeout budget ran out. A timed-out item carries `"skipped": "timeout"` in its result row |
|
|
451
451
|
| `cache` | `render.py --cache` | `{dir, ffmpeg, hits, misses, saved_seconds, entries}`, plus `would_hit` under `--dry-run`. The ffmpeg build banner, the skill version, the contract version, the forwarded flags (`--fast`, `--codec`, …) and the output's extension are all part of every key, so a cache is never reused across any of them — a `--fast` draft is never served to a run that did not ask for one |
|
|
452
452
|
|
|
453
|
-
Per-tool keys added in 1.
|
|
453
|
+
Per-tool keys added in 1.18.1, all additive:
|
|
454
454
|
|
|
455
455
|
| key | tool | what it holds |
|
|
456
456
|
|---|---|---|
|
|
@@ -458,6 +458,18 @@ Per-tool keys added in 1.17.3, all additive:
|
|
|
458
458
|
| `text_unchanged` | `caption.py` | a sibling inside the `caption` block, **burn mode only** (`--mode mux` never touches the text and omits the key): `true` when the drawn text equals the cues that were handed in — nothing transcribed, no cue dropped, no cue **split** across two consecutive cues and no glyph stripped (`--emoji none`). Wrapping, line breaks and timing do not count: the words are the same. This tool never rewrites, shortens or translates a cue, so the key is a statement of what happened, not a judgement of the text |
|
|
459
459
|
|
|
460
460
|
|
|
461
|
+
Per-tool keys added in 1.18.1, all additive:
|
|
462
|
+
|
|
463
|
+
| key | tool | what it holds |
|
|
464
|
+
|---|---|---|
|
|
465
|
+
| `shots` | `scenes.py --shots` | `[{start, end, label, flow_magnitude}]` per detected scene, `label` one of `static`/`pan`/`motion` from a lightweight block-matching optical-flow proxy (frames decoded at 4 fps, 48x27, no external dependency). A shot too short to sample two frames is `static` with `flow_magnitude: 0` |
|
|
466
|
+
| `audio_peaks_db` | `scenes.py --audio-peaks` | `[{time, level}]`, measured dBFS loudness peaks. A **new** key: the pre-existing `audio_peaks` (always reported, unrelated unitless RMS figures used for `--highlights` scoring) keeps its 1.0 meaning unchanged |
|
|
467
|
+
| `speech` | `scenes.py --speech` | `[{time, speech_music_ratio}]`, a per-second zero-crossing-rate ratio against the file's own median — a measured proxy, not a speech/music classification |
|
|
468
|
+
| `motion_centre` | `cropdetect.py --motion-centre` | `[{time, x, y, x_frac, y_frac, motion}]` per second, sampled over the same windows as the crop-bar detection. `x`/`y` are source pixels, `x_frac`/`y_frac` a 0..1 fraction of `source_width`/`source_height`; a window with no measured motion reports `x`/`y`/`x_frac`/`y_frac: null`. Report only — this tool never picks a reframe |
|
|
469
|
+
| `speech_aware`, `speech_aware.breaths` | `silence.py --speech-aware` | `{min_silence, floor, breaths_kept, breaths_kept_seconds, breaths}`. `breaths` are the sub-`--min-silence` gaps kept because they sit inside a sentence; the removal list (`silences`, `keep`, `removed_seconds`) already reflects the speech-aware classification. Composes with `--filler` through the same `keep_ranges()`/`merge_spans()` pipeline, so `--speech-aware --filler` produces one removal list |
|
|
470
|
+
| `sources` | `sync.py` | `[{path, offset_s, confidence, drift_ppm}]`, one entry per SOURCE. Present for every run, including the original single-SOURCE shape (where it mirrors the top-level `second`/`offset_seconds`/`confidence` additively). With 2+ SOURCEs it is the *only* per-source shape: there is no top-level `second`/`offset_seconds` because there is no single pair to put there |
|
|
471
|
+
| `switch_mode`, `min_shot` | `multicam.py --switch energy` | `"energy"` and the `--min-shot` value used (default 1.5s), alongside the existing `cuts` (`[[start, end, camera], ...]`) which already carries the camera index for `--edl`'s companion cut list |
|
|
472
|
+
|
|
461
473
|
`check.py` also gains an informational `subtitles` row on **every** platform:
|
|
462
474
|
`PASS` when every soft subtitle stream carries a language tag, `WARN` when one
|
|
463
475
|
does not (or when there are none). Like `channels` and `chapters` it is never
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.18.1",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
|
@@ -128,7 +128,7 @@ these had ever shown up before.
|
|
|
128
128
|
are still spelled exactly that way: `_common` is a package since the refactor release after
|
|
129
129
|
1.15.0, and its `__init__.py` re-exports every name the single module defined, so
|
|
130
130
|
`_common.drawtext_boxborderw()`, `_common.ffmpeg_version()` and `_common.bt709_tag_args()`
|
|
131
|
-
below resolve through the facade unchanged. The definitions are in `_common/
|
|
131
|
+
below resolve through the facade unchanged. The definitions are in `_common/drawtext.py`,
|
|
132
132
|
`_common/runner.py` and `_common/color.py` respectively.
|
|
133
133
|
- **`showwaves` keeps emitting frames after the audio ends, `-shortest` notwithstanding.** A
|
|
134
134
|
12 s source came out 14.08 s on 5.1.1. `waveform.py` now also passes `-t <source duration>`.
|
package/references/scripts.md
CHANGED
|
@@ -161,7 +161,7 @@ rounded, if they aren't.
|
|
|
161
161
|
|
|
162
162
|
### cropdetect.py — measure black bars, report the crop rectangle
|
|
163
163
|
```
|
|
164
|
-
cropdetect.py INPUT [--seconds N] [--samples N] [--limit F] [--round N]
|
|
164
|
+
cropdetect.py INPUT [--seconds N] [--samples N] [--limit F] [--round N] [--motion-centre]
|
|
165
165
|
```
|
|
166
166
|
Measurement only -- writes no file. Samples `--samples` windows spread
|
|
167
167
|
across the file (default 5, totalling `--seconds` 10s of footage) and
|
|
@@ -174,6 +174,14 @@ removing detected bars is wanted -- genuine letterboxed content (a
|
|
|
174
174
|
scope-ratio film in a 16:9 frame) "detects" the same way as accidental
|
|
175
175
|
bars; look at the frame before cropping it away.
|
|
176
176
|
|
|
177
|
+
`--motion-centre` (1.18) reports the motion centroid once per second, sampled
|
|
178
|
+
across the same windows: `motion_centre: [{time, x, y, x_frac, y_frac, motion}]`,
|
|
179
|
+
`x`/`y` in source pixels and `x_frac`/`y_frac` as a 0..1 fraction of
|
|
180
|
+
`source_width`/`source_height`. A frame with no measured motion in a window
|
|
181
|
+
reports `x`/`y`/`x_frac`/`y_frac: null` and `motion: 0`. Report only, like the
|
|
182
|
+
crop rectangle above -- this hands the calling agent numbers to reframe a 9:16
|
|
183
|
+
crop with; it never picks the subject or the crop box itself.
|
|
184
|
+
|
|
177
185
|
### deinterlace.py — deinterlace interlaced footage
|
|
178
186
|
```
|
|
179
187
|
deinterlace.py INPUT [--mode frame|field] [--parity auto|tff|bff] [--only-interlaced] [-o OUT]
|
|
@@ -465,7 +473,7 @@ each shorter clip's last frame (with silence) out to the longest.
|
|
|
465
473
|
|
|
466
474
|
### silence.py — remove dead air / jump cuts
|
|
467
475
|
```
|
|
468
|
-
silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [-o OUT]
|
|
476
|
+
silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [--speech-aware] [-o OUT]
|
|
469
477
|
silence.py INPUT --filler --words transcript.json [--filler-lang auto|en|ja|es|de|fr|pt|it]
|
|
470
478
|
[--filler-words FILE] [--filler-extra W,W] [--filler-keep W,W] [--filler-pad 0.02]
|
|
471
479
|
[--transcribe] [--filler-list] [--max-cuts 400]
|
|
@@ -531,6 +539,17 @@ additive sibling covering everything that went. A filler word quiet enough to
|
|
|
531
539
|
sit inside a detected silence is merged into it rather than counted twice, so
|
|
532
540
|
the two figures can be equal.
|
|
533
541
|
|
|
542
|
+
**`--speech-aware` (1.18)** keeps breaths shorter than `--min-silence` when
|
|
543
|
+
they sit inside a sentence, and only cuts at sentence-boundary pauses
|
|
544
|
+
(`--min-silence` or longer). It re-runs silence detection at a much shorter
|
|
545
|
+
floor to measure the short gaps at all, then classifies each one: shorter than
|
|
546
|
+
`--min-silence` is an in-sentence breath (kept, listed in
|
|
547
|
+
`speech_aware.breaths`); `--min-silence` or longer is a sentence boundary (cut,
|
|
548
|
+
same as the plain flag). It composes with `--filler` through the same
|
|
549
|
+
`keep_ranges()`/`merge_spans()` pipeline the filler-inside-a-silence fix
|
|
550
|
+
already uses, so `--speech-aware --filler` produces one removal list. `--edl`
|
|
551
|
+
writes the resulting cut list exactly as it does today.
|
|
552
|
+
|
|
534
553
|
### join.py — concatenate with transitions
|
|
535
554
|
```
|
|
536
555
|
join.py CLIP1 CLIP2 [...] [--transition fade|dissolve|wipeleft|slideleft|fadeblack|fadewhite|circleopen|none]
|
|
@@ -701,6 +720,7 @@ already has a picture.
|
|
|
701
720
|
```
|
|
702
721
|
scenes.py INPUT [--threshold 10] [--min-scene 1] [--highlights N [--target SECONDS] [--max-scene 15]] [--edl picks.txt] [--sheet scenes.png] [--json]
|
|
703
722
|
scenes.py INPUT --beats [--beat-step 0.01] [--beat-range 60-200] [--min-confidence 0.5] [--json]
|
|
723
|
+
scenes.py INPUT [--shots] [--audio-peaks] [--speech] [--json]
|
|
704
724
|
```
|
|
705
725
|
Lists scenes with audio energy, the loudest moments, and (with
|
|
706
726
|
`--highlights`) proposes N ranges that add up to `--target` seconds, biased to
|
|
@@ -751,6 +771,29 @@ would change a file on the strength of it (`cut.py --snap beats`). `--edl` with
|
|
|
751
771
|
`--beats` is unchanged: beats are never written as an EDL, because a beat is
|
|
752
772
|
not a cut. No audio stream is a `kind: input` refusal.
|
|
753
773
|
|
|
774
|
+
**`--shots`, `--audio-peaks`, `--speech` (1.18)** are three independent
|
|
775
|
+
measurements that combine with each other and with `--beats`/`--highlights`:
|
|
776
|
+
|
|
777
|
+
- `--shots` labels each already-detected scene `static`/`pan`/`motion` by a
|
|
778
|
+
lightweight block-matching optical-flow proxy (frames decoded at 4 fps,
|
|
779
|
+
48x27, no external dependency), reporting `shots: [{start, end, label,
|
|
780
|
+
flow_magnitude}]`. A shot too short to sample two frames is `static` with
|
|
781
|
+
`flow_magnitude: 0` — there is nothing to measure motion between. A proxy,
|
|
782
|
+
the same spirit as `--rank-by`: it reports what a coarse block match saw,
|
|
783
|
+
not what is interesting about the shot.
|
|
784
|
+
- `--audio-peaks` reports loudness peaks as measured dBFS: `audio_peaks_db:
|
|
785
|
+
[{time, level}]`. This is a new key, kept separate from the pre-existing
|
|
786
|
+
(always-on) `audio_peaks` list, whose entries are an unrelated unitless RMS
|
|
787
|
+
figure used for `--highlights` scoring — a different unit needed a
|
|
788
|
+
different key so `audio_peaks`'s meaning does not change underneath a
|
|
789
|
+
caller reading it since 1.0.
|
|
790
|
+
- `--speech` reports a per-second zero-crossing-rate ratio,
|
|
791
|
+
`speech: [{time, speech_music_ratio}]` — speech's rapid consonant
|
|
792
|
+
transients raise the zero-crossing rate; sustained tones (music, room
|
|
793
|
+
tone) cross zero at a steadier rate. `1.0` means "typical for this file's
|
|
794
|
+
own median," not an absolute threshold. This is a measured number, not a
|
|
795
|
+
speech/music classification — nothing here decides which stretch is which.
|
|
796
|
+
|
|
754
797
|
### check.py — pre-delivery compliance
|
|
755
798
|
```
|
|
756
799
|
check.py INPUT --platform youtube|shorts|reels|tiktok|x|linkedin|facebook|broadcast|podcast|custom [--no-loudness] [--json]
|
|
@@ -905,7 +948,8 @@ hand the path to the user together with the numbers.
|
|
|
905
948
|
|
|
906
949
|
### multicam.py — align several cameras and switch between them
|
|
907
950
|
```
|
|
908
|
-
multicam.py REF CAM2 [CAM3 ...] [--switch "START-END:CAM,..."
|
|
951
|
+
multicam.py REF CAM2 [CAM3 ...] [--switch "START-END:CAM,..."|energy [--min-shot 1.5]] | [--auto N]
|
|
952
|
+
[--audio IDX] [--fix-drift] [--edl cuts.txt]
|
|
909
953
|
[--offsets-only] [--width W --height H --fps N] [-o OUT]
|
|
910
954
|
```
|
|
911
955
|
All inputs are aligned to the first one by audio (same engine as `sync.py`,
|
|
@@ -915,6 +959,23 @@ simply alternates every N seconds. Audio comes from the reference unless
|
|
|
915
959
|
`--audio` picks another input, e.g. an external recorder that has no video.
|
|
916
960
|
`--offsets-only` reports offsets and confidence without rendering.
|
|
917
961
|
|
|
962
|
+
**`--switch energy` (1.18)** auto-switches to whichever camera (of those with
|
|
963
|
+
a video stream) measures the loudest audio at each 0.25 s window on the
|
|
964
|
+
reference timeline, then folds any run shorter than `--min-shot` (default
|
|
965
|
+
1.5 s) into its neighbour so the cut never lingers on a shot too short to
|
|
966
|
+
read. A measured loudest-camera pick, the same spirit as `scenes.py
|
|
967
|
+
--rank-by audio`: a proxy for who is talking, not a judgement — a loud crowd
|
|
968
|
+
or a hot mic wins over a quiet subject exactly like the scene ranking does.
|
|
969
|
+
`--edl` (1.18, any switch mode) writes the resulting cut list as `cut.py
|
|
970
|
+
--segments`-format `START-END` lines; the camera index for each cut is
|
|
971
|
+
already in the JSON `cuts` field (`[[start, end, camera], ...]`), which is
|
|
972
|
+
what `--edl` leaves out on purpose — one file for an NLE's cut list, one
|
|
973
|
+
field for the camera it came from. The multicam timeline needs no dedicated
|
|
974
|
+
project format: each cut is a `render.py` clip (`{"src": ..., "in": ...,
|
|
975
|
+
"out": ...}` on that camera's own timeline, shifted by its measured offset),
|
|
976
|
+
so a switch list can be re-rendered with different `--min-shot` values by
|
|
977
|
+
editing `project.json`'s `clips`, not by inventing a second timeline schema.
|
|
978
|
+
|
|
918
979
|
### verify.py — real-footage verification kit
|
|
919
980
|
```
|
|
920
981
|
verify.py FILES_OR_FOLDERS [--quick] [--report verify.md] [--out DIR --keep] [--seconds 6] [--json]
|
|
@@ -1185,18 +1246,18 @@ draws whatever glyph the text font has; `--emoji none` strips them).
|
|
|
1185
1246
|
|
|
1186
1247
|
### sync.py — offset detection, alignment, drift correction
|
|
1187
1248
|
```
|
|
1188
|
-
sync.py REFERENCE
|
|
1189
|
-
[--replace-audio | --trim-second] [-o OUT]
|
|
1249
|
+
sync.py REFERENCE SOURCE [SOURCE ...] [--json] [--max-offset 30] [--analyze-seconds 120]
|
|
1250
|
+
[--fix-drift [--drift-window 60]] [--replace-audio | --trim-second] [-o OUT]
|
|
1190
1251
|
```
|
|
1191
1252
|
Cross-correlates loudness envelopes: coarse FFT search (20 ms), then a direct
|
|
1192
1253
|
1 ms refinement (pure Python, a 2-minute window takes ~1-3 s). Positive offset
|
|
1193
|
-
= the
|
|
1194
|
-
video with the
|
|
1195
|
-
keeps the reference's full length -- a shorter or head-trimmed
|
|
1254
|
+
= the source recording started later. `--replace-audio` writes the reference
|
|
1255
|
+
video with the source file's audio aligned (video stream copied); the output
|
|
1256
|
+
keeps the reference's full length -- a shorter or head-trimmed source file is
|
|
1196
1257
|
padded with silence, never allowed to cut the picture.
|
|
1197
|
-
`--trim-second` writes the
|
|
1258
|
+
`--trim-second` writes the source file shifted to the reference timeline.
|
|
1198
1259
|
`--fix-drift` measures the offset again near the end of the overlap, reports
|
|
1199
|
-
the clock difference in ppm, and resamples the
|
|
1260
|
+
the clock difference in ppm, and resamples the source file so a 60-minute
|
|
1200
1261
|
take stays in sync (typical consumer devices drift 20-500 ppm = up to 1.8 s/h).
|
|
1201
1262
|
Use it whenever the recording is longer than ~10 minutes. Check `confidence`
|
|
1202
1263
|
(0–1, normalised correlation with a runner-up penalty); below 0.3 the match is
|
|
@@ -1206,6 +1267,20 @@ window 95 %, misses flagged below 0.3. Keep `--analyze-seconds` at least 4×
|
|
|
1206
1267
|
`--max-offset` (default 120 s vs 30 s): lags with under 35 % overlap are
|
|
1207
1268
|
ignored, so an offset larger than ~60 % of the window cannot be found.
|
|
1208
1269
|
|
|
1270
|
+
**One reference, 1+ SOURCE (1.18).** With exactly one SOURCE the CLI keeps its
|
|
1271
|
+
original shape byte for byte -- `second`, `offset_seconds`, `confidence`,
|
|
1272
|
+
`meaning`, `drift` at the top level, `--replace-audio`/`--trim-second`
|
|
1273
|
+
available -- with the same measurement additively mirrored under a new
|
|
1274
|
+
`sources: [{path, offset_s, confidence, drift_ppm}]` list. With 2+ SOURCEs the
|
|
1275
|
+
result is one offsets JSON, `{reference, sources: [...]}`, and no flat
|
|
1276
|
+
`second`/`offset_seconds` (there is no single pair to put there);
|
|
1277
|
+
`--replace-audio`/`--trim-second` refuse with `kind: input`, since each writes
|
|
1278
|
+
one synced output and there is more than one source to choose from. This is
|
|
1279
|
+
the same offset/drift measurement `multicam.py` already loops over per camera
|
|
1280
|
+
internally (it imports `sync.measure_offset`); `sync.py`'s own N-source CLI
|
|
1281
|
+
now exposes that loop directly, for aligning cameras without cutting between
|
|
1282
|
+
them.
|
|
1283
|
+
|
|
1209
1284
|
### color.py — HDR to SDR, LUTs, colour tags, Dolby Vision
|
|
1210
1285
|
```
|
|
1211
1286
|
color.py INPUT --to-sdr [--tonemap hable|mobius|reinhard|bt2390] [--peak 1000] [--desat 0] [-o OUT]
|
|
@@ -7,8 +7,9 @@ error reporting, and provides a compact media probe used by every script.
|
|
|
7
7
|
Since the refactor release after 1.15.0 the helpers live in one module per responsibility --
|
|
8
8
|
runner (process execution and timeouts), probe (ffprobe and the measured facts), decision (the
|
|
9
9
|
pure copy-vs-re-encode and capability choices), emit (result documents, die(), info()), color
|
|
10
|
-
(colour tags and the HDR paths) and
|
|
11
|
-
|
|
10
|
+
(colour tags and the HDR paths) and, since the refactor after 1.17.3, fonts, emoji, drawtext and wrap
|
|
11
|
+
(text.py is a re-export shim over those four) -- and this file is a facade that re-exports every name
|
|
12
|
+
they define. `import _common` and `from _common import x` mean
|
|
12
13
|
exactly what they meant when this was one 3072-line module; nothing else about the package is
|
|
13
14
|
part of the contract.
|
|
14
15
|
"""
|
|
@@ -67,12 +68,13 @@ from _common.emit import (
|
|
|
67
68
|
)
|
|
68
69
|
from _common.probe import (
|
|
69
70
|
analyze_levels, _aspect_string, _bit_depth, decode_pcm_mono, detect_scenes, detect_silences, fingerprint,
|
|
70
|
-
_fraction, keyframes_near, SCORE_RE, SIL_RE,
|
|
71
|
+
_fraction, keyframes_near, SCORE_RE, SIL_RE, decode_gray_frames,
|
|
71
72
|
measured_level_dbfs, MEDIA_EXT, _output_failed, probe, rms_envelope, _to_float, _to_int, verify_output
|
|
72
73
|
)
|
|
73
74
|
from _common.decision import (
|
|
74
75
|
aac_args, add_pad_fill_args, audio_codec_for, AUDIO_CODECS, brand_caption_style, BRAND_DEFAULTS,
|
|
75
76
|
description_block, _evidence_rank, fmt_chapter_time, propose_chapters,
|
|
77
|
+
frame_flow, label_shot_flow, MOTION_GRID, MOTION_SEARCH, MOTION_STATIC_PX, MOTION_PAN_SPREAD,
|
|
76
78
|
filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOURSE_MARKERS, FILLER_MAX_WORD,
|
|
77
79
|
FILLER_MIN_GAP, FILLER_PAD, normalise_filler_token,
|
|
78
80
|
beat_grid, snap_points, BEAT_MIN_CONFIDENCE, BEAT_ONSET_K, BEAT_OCTAVE_MARGIN,
|
|
@@ -112,7 +114,7 @@ from _common.text import (
|
|
|
112
114
|
SHAPING_SCRIPTS, text_width_em, _VS15, _VS16, WINDOWS_FONTS, _ZWJ
|
|
113
115
|
)
|
|
114
116
|
|
|
115
|
-
from _common import asr, color, decision, runner, text # noqa: F401,E402
|
|
117
|
+
from _common import asr, color, decision, drawtext, emoji, fonts, runner, text, wrap # noqa: F401,E402
|
|
116
118
|
|
|
117
119
|
# `_common.emit` and `_common.probe` are the FUNCTIONS, as they have always been -- the
|
|
118
120
|
# from-imports above rebound the package attribute the submodule import had set. The two modules
|
|
@@ -123,7 +125,7 @@ from _common import asr, color, decision, runner, text # noqa: F401,E402
|
|
|
123
125
|
_emit_module = sys.modules["_common.emit"]
|
|
124
126
|
_probe_module = sys.modules["_common.probe"]
|
|
125
127
|
|
|
126
|
-
_MODULES = (runner, _emit_module, _probe_module, decision, color, text, asr)
|
|
128
|
+
_MODULES = (runner, _emit_module, _probe_module, decision, color, text, fonts, emoji, drawtext, wrap, asr)
|
|
127
129
|
|
|
128
130
|
|
|
129
131
|
class _Facade(_types.ModuleType):
|
|
@@ -204,6 +206,7 @@ __all__ = [
|
|
|
204
206
|
"place_output", "_plan_at_exit", "_plan_inputs", "_PLAN_STRIP", "PLAN_VERSION", "PREFERRED_FAMILIES",
|
|
205
207
|
"print_json", "probe", "PROBE_TIMEOUT", "_progress_line", "read_text_or_die", "refuse_output_is_input",
|
|
206
208
|
"_remember_output", "require_tool", "resolve_emoji_assets", "_result_v2", "rms_envelope", "run", "run_analysis",
|
|
209
|
+
"decode_gray_frames", "frame_flow", "label_shot_flow", "MOTION_GRID", "MOTION_SEARCH", "MOTION_STATIC_PX", "MOTION_PAN_SPREAD",
|
|
207
210
|
"_run_captured", "run_keeping_subtitles", "run_tool", "_run_with_progress", "_SCRIPT_FONT_CACHE",
|
|
208
211
|
"_script_font_entry", "script_font_for_text", "script_font_status", "_script_font_uncached", "_SCRIPT_RANGES",
|
|
209
212
|
"SCRIPTS", "_sdr_bt709", "_set_current_ctx", "_SHAPING_BUILD_CACHE", "SHAPING_SCRIPTS", "shell_quote",
|
|
@@ -864,3 +864,89 @@ def filler_spans(words, wordlist, *, pad: float = FILLER_PAD, min_gap: float = F
|
|
|
864
864
|
m["start"] = round(m["start"], 4)
|
|
865
865
|
m["end"] = round(m["end"], 4)
|
|
866
866
|
return merged
|
|
867
|
+
|
|
868
|
+
|
|
869
|
+
# --- 1.18.0: lightweight block-matching motion estimate (scenes.py --shots, cropdetect.py
|
|
870
|
+
# --motion-centre) ---------------------------------------------------------------------------
|
|
871
|
+
|
|
872
|
+
MOTION_GRID = 4 # NxN anchor blocks per frame
|
|
873
|
+
MOTION_SEARCH = 3 # +/- pixels searched per block, at the decoded (low) resolution
|
|
874
|
+
MOTION_STATIC_PX = 0.35 # average per-frame displacement below this, at decode resolution, is "static"
|
|
875
|
+
MOTION_PAN_SPREAD = 0.6 # block-to-block direction agreement above this (0..1) reads as a pan
|
|
876
|
+
|
|
877
|
+
|
|
878
|
+
def _block_match(prev: bytes, cur: bytes, w: int, h: int, cx: int, cy: int, half: int, search: int) -> "Tuple[float, float]":
|
|
879
|
+
"""(dx, dy) that best aligns a `half*2` square centred at (cx, cy) in `prev` to `cur`,
|
|
880
|
+
searched over +/- `search` px by sum-of-absolute-differences. Coordinates and the returned
|
|
881
|
+
offset are in decoded-frame pixels (a handful of pixels a side at 1.18.0's sample size)."""
|
|
882
|
+
x0, y0 = max(half, min(w - half - 1, cx)), max(half, min(h - half - 1, cy))
|
|
883
|
+
ref = [prev[(y0 + dy) * w + (x0 + dx)] for dy in range(-half, half + 1) for dx in range(-half, half + 1)]
|
|
884
|
+
|
|
885
|
+
def sad_at(xx: int, yy: int) -> "Optional[int]":
|
|
886
|
+
if xx - half < 0 or xx + half >= w or yy - half < 0 or yy + half >= h:
|
|
887
|
+
return None
|
|
888
|
+
sad = 0
|
|
889
|
+
for dy in range(-half, half + 1):
|
|
890
|
+
row = (yy + dy) * w
|
|
891
|
+
for dx in range(-half, half + 1):
|
|
892
|
+
sad += abs(ref[(dy + half) * (2 * half + 1) + (dx + half)] - cur[row + xx + dx])
|
|
893
|
+
return sad
|
|
894
|
+
|
|
895
|
+
# Zero shift is the tie-break candidate, not the search order's first cell: on a textureless
|
|
896
|
+
# block (a flat colour, sky, an out-of-focus background) every offset scores the same SAD, and
|
|
897
|
+
# without an explicit tie towards "no motion" the scan used to report the search window's
|
|
898
|
+
# first corner as the measured displacement -- a still frame with nothing to match against
|
|
899
|
+
# read as steady motion in one direction, every time.
|
|
900
|
+
best_sad, best = sad_at(x0, y0), (0.0, 0.0)
|
|
901
|
+
if best_sad is None:
|
|
902
|
+
best_sad = float("inf")
|
|
903
|
+
for sy in range(-search, search + 1):
|
|
904
|
+
for sx in range(-search, search + 1):
|
|
905
|
+
if sx == 0 and sy == 0:
|
|
906
|
+
continue
|
|
907
|
+
sad = sad_at(x0 + sx, y0 + sy)
|
|
908
|
+
if sad is not None and sad < best_sad:
|
|
909
|
+
best_sad, best = sad, (float(sx), float(sy))
|
|
910
|
+
return best
|
|
911
|
+
|
|
912
|
+
|
|
913
|
+
def frame_flow(prev: bytes, cur: bytes, w: int, h: int, *, grid: int = MOTION_GRID,
|
|
914
|
+
search: int = MOTION_SEARCH) -> "Dict[str, Any]":
|
|
915
|
+
"""One measurement between two consecutive decoded grayscale frames: the mean block
|
|
916
|
+
displacement vector, its magnitude, and how consistently the blocks agree on direction
|
|
917
|
+
(0 = every block moved a different way, 1 = every block agrees -- a pan or dolly moves the
|
|
918
|
+
whole frame one way, on-screen motion inside a mostly-static frame does not)."""
|
|
919
|
+
half = max(1, min(w, h) // (grid * 3))
|
|
920
|
+
vecs: "List[Tuple[float, float]]" = []
|
|
921
|
+
for gy in range(grid):
|
|
922
|
+
for gx in range(grid):
|
|
923
|
+
cx = int((gx + 0.5) * w / grid)
|
|
924
|
+
cy = int((gy + 0.5) * h / grid)
|
|
925
|
+
vecs.append(_block_match(prev, cur, w, h, cx, cy, half, search))
|
|
926
|
+
mdx = sum(v[0] for v in vecs) / len(vecs)
|
|
927
|
+
mdy = sum(v[1] for v in vecs) / len(vecs)
|
|
928
|
+
magnitude = math.hypot(mdx, mdy)
|
|
929
|
+
mean_len = sum(math.hypot(*v) for v in vecs) / len(vecs)
|
|
930
|
+
agreement = (magnitude / mean_len) if mean_len > 1e-6 else 1.0 # 1.0 = every block agrees
|
|
931
|
+
return {"dx": mdx, "dy": mdy, "magnitude": magnitude, "agreement": min(1.0, agreement)}
|
|
932
|
+
|
|
933
|
+
|
|
934
|
+
def label_shot_flow(flows: "Sequence[Dict[str, Any]]") -> "Dict[str, Any]":
|
|
935
|
+
"""{label, flow_magnitude} for one shot from its per-frame-pair flow measurements.
|
|
936
|
+
|
|
937
|
+
static: mean displacement below MOTION_STATIC_PX. pan: above it, and blocks agree on
|
|
938
|
+
direction (a camera move shifts the whole frame). motion: above it, blocks disagree (motion
|
|
939
|
+
inside an otherwise still frame -- handheld jitter, or a subject moving across a static
|
|
940
|
+
background). This is a measured proxy, the same spirit as scenes.py --rank-by: it reports
|
|
941
|
+
what a coarse block match saw, not what is interesting about the shot."""
|
|
942
|
+
if not flows:
|
|
943
|
+
return {"label": "static", "flow_magnitude": 0.0}
|
|
944
|
+
magnitude = sum(f["magnitude"] for f in flows) / len(flows)
|
|
945
|
+
agreement = sum(f["agreement"] for f in flows) / len(flows)
|
|
946
|
+
if magnitude < MOTION_STATIC_PX:
|
|
947
|
+
label = "static"
|
|
948
|
+
elif agreement >= MOTION_PAN_SPREAD:
|
|
949
|
+
label = "pan"
|
|
950
|
+
else:
|
|
951
|
+
label = "motion"
|
|
952
|
+
return {"label": label, "flow_magnitude": round(magnitude, 3)}
|