ffmpeg-skill 2.19.0 → 2.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +28 -0
- package/docs/contract.md +12 -2
- package/package.json +1 -1
- package/references/scripts.md +60 -0
- package/scripts/_common/probe.py +16 -0
- package/scripts/audio.py +2 -1
- package/scripts/export.py +14 -0
- package/scripts/join.py +25 -5
- package/scripts/loudness.py +41 -9
- package/scripts/silence.py +32 -1
- package/scripts/waveform.py +2 -1
package/README.md
CHANGED
|
@@ -412,6 +412,33 @@ The short list for humans. The agent-facing version, with the reasoning, is the
|
|
|
412
412
|
|
|
413
413
|
Each operation is a script whose flags are its argparse parser, and that parser is what `contract --json` and the MCP `inputSchema` are built from. CI fails the build when those drift from the scripts. A caller-supplied filter graph, `-vf` string, or other ffmpeg option is not an argument any script accepts. `doctor` reads `ffmpeg -filters` (and `-encoders`, `-bsfs`) and reports a filter a tool needs before a job runs into it. MCP does accept an `argv` array, and that array is the named script's own CLI flags, still bound to that script: the script's flags, never an ffmpeg command line and never a shell.
|
|
414
414
|
|
|
415
|
+
### Why 42 tools? Isn't that overkill?
|
|
416
|
+
|
|
417
|
+
One tool per job, because each job promises something different, and the tool measures that promise after the run. `loudness.py` re-measures integrated loudness and true peak. `cut.py` reports how far a cut snapped to a keyframe. `silence.py` returns the ranges it kept. `export.py` checks the preset's maximum length. A value a tool cannot read fails that step as `not_measured`; it does not pass. One tool with a mode flag would need as large a schema, and its checks would be vaguer.
|
|
418
|
+
|
|
419
|
+
An agent does not read 42 schemas. Over MCP, `tools/list` advertises the core 12 (bold below). The other 30 are called by name through `tools/call` when a job needs one, and `FFMPEG_SKILL_MCP_FULL=1` lists all 42. The groups are the ones in [Tools](#tools).
|
|
420
|
+
|
|
421
|
+
<!-- why42:start -->
|
|
422
|
+
| Group | MCP default / all | Tools |
|
|
423
|
+
|---|---|---|
|
|
424
|
+
| Analysis and inspection | 2 / 3 | **probe**, **look**, scenes |
|
|
425
|
+
| Editing | 3 / 24 | **cut**, **fit**, **silence**, join, crop, cropdetect, deinterlace, denoise, redact, sphere, straighten, insert, background, reverse, stabilize, sequence, waveform, freeze, pad, speedramp, loop, broll, metadata, grid |
|
|
426
|
+
| Audio | 2 / 3 | **audio**, **loudness**, sync |
|
|
427
|
+
| Picture | 2 / 4 | **caption**, **graphics**, overlay, color |
|
|
428
|
+
| Delivery | 2 / 4 | **export**, **check**, proxy, report |
|
|
429
|
+
| Orchestration | 1 / 4 | **render**, batch, multicam, verify |
|
|
430
|
+
| **Total** | **12 / 42** | |
|
|
431
|
+
<!-- why42:end -->
|
|
432
|
+
|
|
433
|
+
Without the other 30, an agent writes raw ffmpeg for those jobs, and a mistake there often still exits 0:
|
|
434
|
+
|
|
435
|
+
- `stabilize.py` runs vidstab in two passes, and the second pass needs the first pass's transforms file.
|
|
436
|
+
- `color.py` builds the HDR-to-SDR chain. With a wrong colour tag the run still finishes, and the picture comes out washed out.
|
|
437
|
+
- `join.py` puts transitions between clips that differ in size, frame rate or audio layout. Done by hand, those clips have to be normalised first or xfade fails.
|
|
438
|
+
- `sync.py` and `multicam.py` find the offset between two recordings by audio cross-correlation. No ffmpeg filter measures it.
|
|
439
|
+
|
|
440
|
+
Every command a tool ran is in its `--json` `commands`, shell-quoted so it can be pasted and run again, and `--dry-run` prints the commands without running anything. In the 100-prompt agent eval (1.17.0, [Tested on real footage](#tested-on-real-footage)) the agent made no raw ffmpeg calls and reported no false successes.
|
|
441
|
+
|
|
415
442
|
### How does ffmpeg-skill compare with Remotion?
|
|
416
443
|
|
|
417
444
|
ffmpeg-skill edits footage you already have, on this machine, with FFmpeg tools whose flags are a machine-readable contract (`contract --json`). Remotion renders a video from React components. They complement each other: this repository does not call Remotion, and a file Remotion writes is an ordinary input (`probe.py`, `cut.py`, `fit.py`, `export.py`, `check.py`). [`remotion-ffmpeg-skill`](https://github.com/kajisho5/remotion-ffmpeg-skill) joins the two. It measures the footage with this repo, draws the graphics in Remotion from those measurements, and checks and delivers the result with this repo again. Install both (`npx ffmpeg-skill`, then `npx remotion-ffmpeg-skill`). As with the other skills, the dependency runs one way: it calls these tools, and nothing here calls it. HyperFrames renders a video from HTML and is the same kind of neighbour: a file it writes is an ordinary input too.
|
|
@@ -614,6 +641,7 @@ Contributing a change: see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
|
614
641
|
| [SECURITY.md](SECURITY.md) | how to report a vulnerability privately |
|
|
615
642
|
| [SKILL.md](SKILL.md) | what the agent reads: workflow, request → tool map, audio-only rules, report format, pitfalls |
|
|
616
643
|
| [references/scripts.md](references/scripts.md) | per-flag reference for every tool |
|
|
644
|
+
| [references/scripts.md § Verdict thresholds](references/scripts.md#verdict-thresholds-what-passes) | every number behind `verified` and `check.py`'s rows (tolerances, silence levels, per-platform loudness and length), each with the constant or flag it comes from |
|
|
617
645
|
| [references/devices.md](references/devices.md) | real-device notes (iPhone HDR, GoPro, DJI, screen recordings) |
|
|
618
646
|
| [references/ci-platform-pitfalls.md](references/ci-platform-pitfalls.md) | per-OS ffmpeg/CI behaviour differences already diagnosed once — read before re-diagnosing a Windows/macOS-only test failure |
|
|
619
647
|
| [references/process-pitfalls.md](references/process-pitfalls.md) | process mistakes already made once (breaking a pinned test by narrowing a capability list, retrying a git/GitHub operation this environment can't do, re-designing a fixture instead of recognising a real platform difference) — a living record, add to it whenever one recurs |
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`2.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`2.21.0`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -131,7 +131,7 @@ JSON would have been rewritten for that. The flat keys, typed per tool in `outpu
|
|
|
131
131
|
"deprecated": [{"what": "running on Python 3.9 (the requirements.python floor)", "since": "2.10.9", "replacement": "Python 3.10 or later; ...", "removed_in": "3.0.0", "where": "behaviour"},
|
|
132
132
|
{"what": "running on FFmpeg 5.x (the requirements.ffmpeg floor)", "since": "2.14.0", "replacement": "FFmpeg 6.1 or later; ...", "removed_in": "3.0.0", "where": "behaviour"}],
|
|
133
133
|
"removed": [{"what": "...", "since": "1.10.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
134
|
-
"skill": {"id": "ffmpeg-skill", "version": "2.
|
|
134
|
+
"skill": {"id": "ffmpeg-skill", "version": "2.21.0", "execution_mode": "local", "kind": "execution",
|
|
135
135
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
136
136
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
137
137
|
"requirements": {"python": ">=3.9 (standard library only; 3.9 deprecated since 2.10.9, removed in 3.0.0; 3.10+ recommended)", "ffmpeg": ">=5.0 (5.x deprecated since 2.14.0, removed in 3.0.0; 6.1+ recommended)", "ffprobe": ">=5.0"},
|
|
@@ -699,6 +699,16 @@ from where the expected value came, and what was not measured.
|
|
|
699
699
|
| `audio_substituted` | `join.py` (video join) | the clips that had no audio stream, whose span of the join is generated silence; `notes` says the same |
|
|
700
700
|
| loudness `tolerance.tp` | `loudness.py`, `export.py`, and `check.py`'s true-peak row | the one dBTP margin every tool applies (`_platforms.TP_TOLERANCE_DB`, 0.1 dB); before 2.18 it was 0.1, 0.05 and 0.0 in the three tools |
|
|
701
701
|
|
|
702
|
+
Keys added in 2.20, all additive, from the same paper's baseline experiment (each was a length or a
|
|
703
|
+
silence a tool reported as fine while another measurement disagreed):
|
|
704
|
+
|
|
705
|
+
| key | tool | what it holds |
|
|
706
|
+
|---|---|---|
|
|
707
|
+
| `verification[]` step `duration` | `export.py` (a preset with a platform maximum, without `--allow-long`) | the written file's length against the maximum, measured as `check.py` measures it (the container's duration, `tolerance` 0). A length that cannot be read fails the step with `reason: not_measured` |
|
|
708
|
+
| `verification[]` step `duration` (`property: audio_duration`) | `loudness.py` | the output's sound against the input's, to one AAC frame (at least 0.03 s): a levels pass changes no timing. A length that cannot be read fails the step with `reason: not_measured` |
|
|
709
|
+
| `inputs_duration`, `transition_shortening`; step `duration`'s `inputs_total` | `join.py` | the clips' own total and how much the transitions took out of it. `expected_duration` is derived from the transition the join applied, so it cannot show that on its own |
|
|
710
|
+
| `silence_kinds`, `silence_breakdown` | `silence.py` | per detected silence, `digital` (at or below -90 dBFS) or `below_threshold` (quiet sound under `--threshold`), and the count and seconds of each with the level that defines it |
|
|
711
|
+
|
|
702
712
|
Per-tool keys added after 2.10.11 ("nothing happens silently"), all additive:
|
|
703
713
|
|
|
704
714
|
| key | tool | what it holds |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.21.0",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
package/references/scripts.md
CHANGED
|
@@ -14,7 +14,67 @@ yourself, which is the only way for a tool with no input file (`caption.py
|
|
|
14
14
|
anywhere is `kind: input` naming the flag. Nothing else about times differs
|
|
15
15
|
between tools.
|
|
16
16
|
|
|
17
|
+
## Verdict thresholds (what passes)
|
|
18
|
+
|
|
19
|
+
The numbers behind `verified`, `check.py`'s PASS/WARN/FAIL rows and the cuts the tools make. A
|
|
20
|
+
result also carries the tolerance it used in each `verification[]` step (`tolerance`, since 2.18).
|
|
21
|
+
Every row names the constant or flag it comes from, and a test holds the table to it
|
|
22
|
+
(`test_the_thresholds_table_matches_the_code`).
|
|
23
|
+
|
|
24
|
+
<!-- thresholds:start -->
|
|
25
|
+
| what | value | source | used by |
|
|
26
|
+
|---|---|---|---|
|
|
27
|
+
| true peak may exceed the dBTP ceiling by | 0.1 dB | `_platforms.TP_TOLERANCE_DB` | `loudness.py`, `check.py`, `export.py` |
|
|
28
|
+
| a normalised file is on its `-I` target within | 1.0 LU | `loudness.LUFS_TOLERANCE` | `loudness.py` |
|
|
29
|
+
| a levels pass keeps the sound's length to (or one AAC frame, 1024 samples, if longer) | 0.03 s | `loudness.AUDIO_LENGTH_TOLERANCE` | `loudness.py` |
|
|
30
|
+
| a joined picture is as long as the clips less the transitions, to (frames) | 1.5 | `join.DURATION_TOLERANCE_FRAMES` | `join.py` |
|
|
31
|
+
| ... and at least | 0.05 s | `join.DURATION_TOLERANCE_MIN` | `join.py` |
|
|
32
|
+
| a lossless cut may land this far from the asked time before it re-encodes | 0.5 s | `cut.py --tolerance` | `cut.py` |
|
|
33
|
+
| silence is sound at or below | -35 dBFS | `silence.py --threshold` | `silence.py` |
|
|
34
|
+
| ... lasting at least | 0.6 s | `silence.py --min-silence` | `silence.py` |
|
|
35
|
+
| a removed silence is digital silence when 99% of it sits at or below | -90 dBFS | `silence.DIGITAL_SILENCE_DB` | `silence.py` (`silence_kinds`) |
|
|
36
|
+
| a track is silent when its peak is at or below | -50 dBFS | `_common.probe.SILENT_PEAK_DBFS` | `check.py`; the `--silence-threshold` default of `join.py`, `audio.py`, `waveform.py` |
|
|
37
|
+
| a filler word longer than this is a held vowel, kept | 1.2 s | `_common.decision.FILLER_MAX_WORD` | `silence.py --filler` |
|
|
38
|
+
| a pause this long between two tokens of an unspaced script is a word boundary | 0.1 s | `_common.decision.FILLER_BOUNDARY_GAP` | `silence.py --filler` |
|
|
39
|
+
| `--content` black: WARN above this share of the duration | 0.10 | `check.BLACK_WARN` | `check.py --content` |
|
|
40
|
+
| `--content` black: FAIL at this share | 0.95 | `check.BLACK_FAIL` | `check.py --content` |
|
|
41
|
+
| `--content` black: a pixel is black below this share of the luma range | 0.10 | `_common.probe.CONTENT_BLACK_PIX_TH` | `check.py --content` |
|
|
42
|
+
| `--content` black: a black run shorter than this is not counted | 0.1 s | `_common.probe.CONTENT_MIN_SPAN` | `check.py --content` |
|
|
43
|
+
| `--content` frozen: WARN when the longest span is above this, or above 30% | 3.0 s | `check.FROZEN_WARN_S` | `check.py --content` |
|
|
44
|
+
| `--content` frozen: FAIL when one span covers this share | 0.95 | `check.FROZEN_FAIL` | `check.py --content` |
|
|
45
|
+
| `--content` frozen: a picture unchanged for at least this long is frozen | 0.5 s | `_common.probe.CONTENT_FREEZE_MIN` | `check.py --content` |
|
|
46
|
+
| `--content` frozen: frames this close count as unchanged (freezedetect noise) | -60 dB | `_common.probe.CONTENT_FREEZE_NOISE` | `check.py --content` |
|
|
47
|
+
| `--content` silence: WARN above this share below the silent level | 0.50 | `check.SILENCE_WARN` | `check.py --content` |
|
|
48
|
+
| `--content` silence: FAIL at this share | 0.95 | `check.SILENCE_FAIL` | `check.py --content` |
|
|
49
|
+
| `--content` silence: a silent run shorter than this is not counted | 0.5 s | `_common.probe.CONTENT_SILENCE_MIN` | `check.py --content` |
|
|
50
|
+
| `--content` dropout: a gap of sound at or below | -80 dBFS | `_common.probe.CONTENT_DROPOUT_DB` | `check.py --content` |
|
|
51
|
+
| ... between (s) and 1 s long, with sound on both sides | 0.02 s | `_common.probe.CONTENT_DROPOUT_MIN` | `check.py --content` |
|
|
52
|
+
| a beat grid below this confidence is not snapped to | 0.5 | `_common.decision.BEAT_MIN_CONFIDENCE` | `cut.py --snap beats`, `scenes.py --beats` |
|
|
53
|
+
<!-- thresholds:end -->
|
|
54
|
+
|
|
55
|
+
Delivery targets per platform (`scripts/_platforms.py`). A file passes loudness within the
|
|
56
|
+
platform's tolerance of its target, and true peak at or below its ceiling plus 0.1 dB. `export.py`
|
|
57
|
+
holds the written length to the maximum with no tolerance, as `check.py` does.
|
|
58
|
+
|
|
59
|
+
<!-- platforms:start -->
|
|
60
|
+
| platform | loudness | tolerance | true peak | max duration |
|
|
61
|
+
|---|---|---|---|---|
|
|
62
|
+
| `tiktok` | -14 LUFS | ±2 LU | -1 dBTP | 600 s |
|
|
63
|
+
| `reels` | -14 LUFS | ±2 LU | -1 dBTP | 90 s |
|
|
64
|
+
| `shorts` | -14 LUFS | ±2 LU | -1 dBTP | 180 s |
|
|
65
|
+
| `youtube` | -14 LUFS | ±2 LU | -1 dBTP | 43200 s |
|
|
66
|
+
| `youtube-hdr` | -14 LUFS | ±2 LU | -1 dBTP | 43200 s |
|
|
67
|
+
| `youtube-av1` | -14 LUFS | ±2 LU | -1 dBTP | 43200 s |
|
|
68
|
+
| `x` | -14 LUFS | ±3 LU | -1 dBTP | 140 s |
|
|
69
|
+
| `linkedin` | -14 LUFS | ±3 LU | -1 dBTP | 600 s |
|
|
70
|
+
| `facebook` | -14 LUFS | ±3 LU | -1 dBTP | 14400 s |
|
|
71
|
+
| `podcast` | -16 LUFS | ±1 LU | -1 dBTP | — |
|
|
72
|
+
| `broadcast` | -23 LUFS | ±1 LU | -1 dBTP | — |
|
|
73
|
+
| `custom` | — | ±2 LU | -1 dBTP | — |
|
|
74
|
+
<!-- platforms:end -->
|
|
75
|
+
|
|
17
76
|
## Contents
|
|
77
|
+
- verdict thresholds — what passes, and where each number lives
|
|
18
78
|
- probe.py — inspect
|
|
19
79
|
- cut.py — cut / join segments
|
|
20
80
|
- fit.py — target duration and/or aspect, rotate/flip
|
package/scripts/_common/probe.py
CHANGED
|
@@ -300,6 +300,9 @@ def probe(path: str, role: str = "input") -> Dict[str, Any]:
|
|
|
300
300
|
"bitrate": _to_int(audio.get("bit_rate")),
|
|
301
301
|
"duration": _to_float(audio.get("duration")),
|
|
302
302
|
"start_time": _to_float(audio.get("start_time")),
|
|
303
|
+
# Matroska/WebM give no stream duration, only a DURATION tag; the container's own length
|
|
304
|
+
# is the longest stream's, which is not the sound's when the picture runs longer (2.20)
|
|
305
|
+
"tagged_duration": _tag_seconds((audio.get("tags") or {}).get("DURATION")),
|
|
303
306
|
}
|
|
304
307
|
# every audio stream in file order: index n here is `-map 0:a:n` (audio.py --audio-stream n)
|
|
305
308
|
out["audio_streams"] = [{
|
|
@@ -497,6 +500,19 @@ def analyze_levels(path: str, seconds: float = 20.0) -> Dict[str, Any]:
|
|
|
497
500
|
}
|
|
498
501
|
|
|
499
502
|
|
|
503
|
+
def _tag_seconds(value: Any) -> "Optional[float]":
|
|
504
|
+
"""A Matroska DURATION tag ("00:00:04.008000000") in seconds, or None."""
|
|
505
|
+
try:
|
|
506
|
+
if "-" in str(value): # "-00:00:01": int("-00") is 0, so the sign would be lost
|
|
507
|
+
return None
|
|
508
|
+
h, m, sec = str(value).split(":")
|
|
509
|
+
seconds = int(h) * 3600 + int(m) * 60 + float(sec)
|
|
510
|
+
except (TypeError, ValueError):
|
|
511
|
+
return None
|
|
512
|
+
# float() reads "nan" and "inf": a length that is not a finite, non-negative number is none
|
|
513
|
+
return seconds if math.isfinite(seconds) and seconds >= 0 else None
|
|
514
|
+
|
|
515
|
+
|
|
500
516
|
def _to_float(v: Any) -> Optional[float]:
|
|
501
517
|
try:
|
|
502
518
|
return float(v) if v is not None else None
|
package/scripts/audio.py
CHANGED
|
@@ -25,6 +25,7 @@ import sys
|
|
|
25
25
|
from pathlib import Path
|
|
26
26
|
from typing import Any, Dict, List
|
|
27
27
|
|
|
28
|
+
from _common.probe import SILENT_PEAK_DBFS
|
|
28
29
|
from _common import STATE, add_common, dry_run_input_pending, require_tool, apply_common, audio_codec_for, db_to_linear, default_output, die, emit, ffmpeg_base, info, is_audio_output, measured_level_dbfs, probe, run, run_keeping_subtitles, fmt_secs
|
|
29
30
|
from _common.emit import warn
|
|
30
31
|
|
|
@@ -216,7 +217,7 @@ def main() -> int:
|
|
|
216
217
|
ap.add_argument("--on-silent", choices=["warn", "fail"], default="warn",
|
|
217
218
|
help="a --music/--replace/--effects file (or, under --duck, the voice) whose peak is at or below "
|
|
218
219
|
"--silence-threshold: warn (default: mix it, report it under `silent`) or fail (refuse before ffmpeg runs)")
|
|
219
|
-
ap.add_argument("--silence-threshold", type=float, default
|
|
220
|
+
ap.add_argument("--silence-threshold", type=float, default=SILENT_PEAK_DBFS, metavar="DBFS",
|
|
220
221
|
help="peak level in dBFS at or below which a track counts as silent (default -50)")
|
|
221
222
|
add_common(ap)
|
|
222
223
|
args = ap.parse_args()
|
package/scripts/export.py
CHANGED
|
@@ -333,6 +333,20 @@ def main() -> int:
|
|
|
333
333
|
elif platform and has_audio and args.normalize:
|
|
334
334
|
spec = PLATFORMS[platform]
|
|
335
335
|
notes.append(f"[dry-run] --normalize: loudness.py -I {spec['lufs']:g} --tp {spec['tp']:g} would run on the written file if it misses {platform}'s spec")
|
|
336
|
+
if p["max"] and not args.allow_long and not STATE.dry_run:
|
|
337
|
+
# the preset's length promise, measured the way check.py measures it (the container's
|
|
338
|
+
# duration, no tolerance): before 2.20 nothing re-read the length, and a --normalize pass
|
|
339
|
+
# that left the sound 0.1 s long shipped a 90.10 s Reels file as verified
|
|
340
|
+
written = probe(output, role="output").get("duration")
|
|
341
|
+
if written is not None:
|
|
342
|
+
step = {"step": "duration", "property": "duration", "ok": written <= p["max"],
|
|
343
|
+
"expected": {"max": p["max"]}, "measured": round(written, 3),
|
|
344
|
+
"delta": round(written - p["max"], 3), "tolerance": 0.0, "provenance": "specification"}
|
|
345
|
+
else:
|
|
346
|
+
# the preset promised a length; one that could not be read is not one that held
|
|
347
|
+
step = {"step": "duration", "property": "duration", "ok": False, "expected": {"max": p["max"]},
|
|
348
|
+
"reason": "not_measured: the written file reports no duration"}
|
|
349
|
+
extra.setdefault("verification", []).append(step)
|
|
336
350
|
if notes:
|
|
337
351
|
extra["notes"] = notes
|
|
338
352
|
# the delivery keeps the source's picture and, except as a GIF, its sound
|
package/scripts/join.py
CHANGED
|
@@ -39,6 +39,7 @@ import os
|
|
|
39
39
|
import sys
|
|
40
40
|
from typing import Any, Dict, List, Optional, Tuple
|
|
41
41
|
|
|
42
|
+
from _common.probe import SILENT_PEAK_DBFS
|
|
42
43
|
from _common import STATE, AUDIO_MEDIA_EXT, draws_chroma, drawn_chain, label_chain, matrix_chain, video_args, aac_args, add_common, apply_common, audio_codec_for, default_output, die, dry_run_input_pending, emit, ffmpeg_base, info, measured_level_dbfs, is_audio_output, probe, require_tool, run, validate_color, X264_PRESETS, fmt_secs
|
|
43
44
|
from _common.emit import warn
|
|
44
45
|
|
|
@@ -50,6 +51,9 @@ STATE_SKIPPED: List[Dict[str, Any]] = []
|
|
|
50
51
|
# --dry-run: inputs that do not exist yet, planned on as an earlier step's output
|
|
51
52
|
STATE_PENDING: List[Dict[str, Any]] = []
|
|
52
53
|
STATE_NOTES: List[str] = []
|
|
54
|
+
# the joined picture's length against the clips' sum less the transitions: 1.5 frames, at least 0.05 s
|
|
55
|
+
DURATION_TOLERANCE_FRAMES = 1.5
|
|
56
|
+
DURATION_TOLERANCE_MIN = 0.05
|
|
53
57
|
# --on-silent warn: inputs whose audio peak is at or below --silence-threshold, joined anyway
|
|
54
58
|
STATE_SILENT: List[Dict[str, Any]] = []
|
|
55
59
|
# warnings, never refusals: a clip shorter than MIN_SEGMENT_FRAMES frames (audio-only: shorter than
|
|
@@ -81,6 +85,20 @@ def unpending_length(durs: List[float], d: float) -> Optional[float]:
|
|
|
81
85
|
return round(sum(durs) - d * (len(durs) - 1), 3)
|
|
82
86
|
|
|
83
87
|
|
|
88
|
+
def overlap_report(durs: List[float], expected: Optional[float]) -> Dict[str, Any]:
|
|
89
|
+
"""The clips' own total next to the joined length (2.20). The joined length is derived from
|
|
90
|
+
the transition the join itself applied, so a check against it alone cannot show that a fade
|
|
91
|
+
took time out of the material; the sum of the inputs is the yardstick the user holds."""
|
|
92
|
+
if expected is None or STATE_PENDING:
|
|
93
|
+
return {}
|
|
94
|
+
total = round(sum(durs), 3)
|
|
95
|
+
out: Dict[str, Any] = {"inputs_duration": total, "transition_shortening": round(total - expected, 3)}
|
|
96
|
+
if out["transition_shortening"] > 0.0005:
|
|
97
|
+
STATE_NOTES.append(f"the transitions overlap the clips: {fmt_secs(total)} of clips play as {fmt_secs(expected)} "
|
|
98
|
+
f"({out['transition_shortening']:.3f} s shorter)")
|
|
99
|
+
return out
|
|
100
|
+
|
|
101
|
+
|
|
84
102
|
def clip_length(meta: Dict[str, Any], fps: float) -> float:
|
|
85
103
|
"""The one length a clip holds in a join, for its picture and its sound alike: the
|
|
86
104
|
video stream's, or the audio stream's when the sound runs more than a frame past the picture
|
|
@@ -175,7 +193,8 @@ def join_audio(args: argparse.Namespace, metas: List[dict]) -> int:
|
|
|
175
193
|
die(f"{output} is {a.get('sample_rate')} Hz {a.get('channels')} ch, expected {rate} Hz {channels} ch")
|
|
176
194
|
info(f"wrote {output} ({fmt_secs(r['duration'])}, {expected_text(expected)}, audio {a.get('codec')} {channels}ch {rate}Hz, {n} clips, "
|
|
177
195
|
+ ("crossfade" if d else "butt join") + ")")
|
|
178
|
-
|
|
196
|
+
overlap = overlap_report(durs, expected)
|
|
197
|
+
emit(output, mode="audio", **overlap, **reported(), clips=n, transition=args.transition if d else "none", expected_duration=expected,
|
|
179
198
|
sample_rate=rate, channels=channels, video=False)
|
|
180
199
|
return 0
|
|
181
200
|
|
|
@@ -301,7 +320,7 @@ def main() -> int:
|
|
|
301
320
|
ap.add_argument("--allow-silent", metavar="N[,N...]",
|
|
302
321
|
help="input numbers (1-based) whose silence is intended, e.g. a planned pause: always joined, "
|
|
303
322
|
"listed under `silent` with intended: true, no warning, whatever --on-silent says")
|
|
304
|
-
ap.add_argument("--silence-threshold", type=float, default
|
|
323
|
+
ap.add_argument("--silence-threshold", type=float, default=SILENT_PEAK_DBFS, metavar="DB",
|
|
305
324
|
help="peak level in dBFS at or below which an input counts as silent (default -50)")
|
|
306
325
|
ap.add_argument("-o", "--output", help="output file (default: <first>_joined.mp4)")
|
|
307
326
|
ap.add_argument("--transition", choices=TRANSITIONS, default="fade", help="transition between clips (default fade)")
|
|
@@ -564,11 +583,11 @@ def main() -> int:
|
|
|
564
583
|
measured = (r.get("video") or {}).get("duration") or r.get("duration")
|
|
565
584
|
verification = []
|
|
566
585
|
if expected is not None and measured is not None and not STATE.dry_run:
|
|
567
|
-
tol = max(
|
|
586
|
+
tol = max(DURATION_TOLERANCE_FRAMES / fps, DURATION_TOLERANCE_MIN)
|
|
568
587
|
verification.append({"step": "duration", "property": "video_duration", "ok": abs(measured - expected) <= tol,
|
|
569
588
|
"measured": round(measured, 3), "expected": expected,
|
|
570
589
|
"delta": round(measured - expected, 3), "tolerance": round(tol, 3),
|
|
571
|
-
"provenance": "input_premeasurement"})
|
|
590
|
+
"provenance": "input_premeasurement", "inputs_total": round(sum(lens), 3)})
|
|
572
591
|
# a clip with no audio stream is joined over generated silence: the join is right to do that, but
|
|
573
592
|
# the report must say which clips' sound is not theirs (the paper's join finding: absence was
|
|
574
593
|
# filled and nothing in the result showed it)
|
|
@@ -577,7 +596,8 @@ def main() -> int:
|
|
|
577
596
|
STATE_NOTES.append(f"{len(substituted)} clip(s) had no audio stream; their span of the join is generated "
|
|
578
597
|
"silence: " + ", ".join(os.path.basename(p) for p in substituted))
|
|
579
598
|
info(f"wrote {output} ({fmt_secs(measured)}, {expected_text(expected)}, {w}x{h} @ {fps:g}fps, {n} clips, {args.transition})")
|
|
580
|
-
|
|
599
|
+
overlap = overlap_report(lens, expected)
|
|
600
|
+
emit(output, mode="video", **overlap, **reported(), clips=n, transition=args.transition, expected_duration=expected,
|
|
581
601
|
verification=verification, expect_streams={"video": True, "audio": True},
|
|
582
602
|
audio_substituted=substituted,
|
|
583
603
|
dropped_non_av_streams=any(m.get("subtitle_streams") or m.get("data_streams") for m in metas))
|
package/scripts/loudness.py
CHANGED
|
@@ -20,6 +20,9 @@ import re
|
|
|
20
20
|
import sys
|
|
21
21
|
|
|
22
22
|
from _platforms import TP_TOLERANCE_DB
|
|
23
|
+
|
|
24
|
+
AUDIO_LENGTH_TOLERANCE = 0.03 # seconds: one AAC frame at 48 kHz is 0.021 s
|
|
25
|
+
LUFS_TOLERANCE = 1.0 # LU: a normalised file within this of the requested -I is on target
|
|
23
26
|
from _common import STATE, add_common, apply_common, emit, AUDIO_CODECS, audio_codec_for, default_output, die, ffmpeg_base, info, probe, require_tool, run, run_analysis, run_keeping_subtitles, dry_run_input_pending
|
|
24
27
|
|
|
25
28
|
|
|
@@ -93,6 +96,12 @@ def main() -> int:
|
|
|
93
96
|
bitrate = args.audio_bitrate or "192k"
|
|
94
97
|
|
|
95
98
|
dropped_streams = False
|
|
99
|
+
# loudnorm emits its last block whole, and through the AAC encoder that came out as 0.1 s more
|
|
100
|
+
# sound than the input had (a 90.000 s Reels export left at 90.100 s, which check.py fails):
|
|
101
|
+
# the sound is trimmed back to the input's own length. `duration=` counts from the first sample,
|
|
102
|
+
# so sound that starts late keeps its end (`end=` is a timestamp, and cut it). atrim follows
|
|
103
|
+
# loudnorm directly: an aresample between them fails channel-layout negotiation on FFmpeg 5.1
|
|
104
|
+
in_audio = audio_length(meta)
|
|
96
105
|
|
|
97
106
|
def encode(tp: float, bitrate: str) -> None:
|
|
98
107
|
nonlocal dropped_streams
|
|
@@ -101,6 +110,8 @@ def main() -> int:
|
|
|
101
110
|
f":measured_I={stats['input_i']}:measured_TP={stats['input_tp']}:measured_LRA={stats['input_lra']}"
|
|
102
111
|
f":measured_thresh={stats['input_thresh']}:offset={stats['target_offset']}:linear=true:print_format=summary"
|
|
103
112
|
)
|
|
113
|
+
if in_audio:
|
|
114
|
+
af += f",atrim=duration={float(in_audio):.6f}"
|
|
104
115
|
cmd = ffmpeg_base() + ["-i", args.input, "-af", af, "-ar", str(sr)]
|
|
105
116
|
if ext in AUDIO_CODECS or not meta.get("video"):
|
|
106
117
|
cmd += ["-vn"] + audio_codec_for(output, bitrate)
|
|
@@ -149,7 +160,7 @@ def main() -> int:
|
|
|
149
160
|
if ext not in AUDIO_CODECS or "aac" in AUDIO_CODECS[ext]:
|
|
150
161
|
result["audio_bitrate_used"] = bitrate
|
|
151
162
|
result["encodes"] = rounds + 1
|
|
152
|
-
if not after.get("silent") and abs(float(after["input_i"]) - args.lufs) >
|
|
163
|
+
if not after.get("silent") and abs(float(after["input_i"]) - args.lufs) > LUFS_TOLERANCE:
|
|
153
164
|
result["note"] = (f"integrated loudness is {float(after['input_i']) - args.lufs:+.1f} LU from the target because the "
|
|
154
165
|
f"true-peak ceiling had to absorb the encoder's overshoot; a lossless delivery (wav/flac) or a pinned "
|
|
155
166
|
f"higher --audio-bitrate keeps both")
|
|
@@ -158,19 +169,40 @@ def main() -> int:
|
|
|
158
169
|
f"the encoder overshoots more than the loudnorm ceiling can absorb at this bitrate",
|
|
159
170
|
kind="verification", output=output, result=result,
|
|
160
171
|
hint="raise --audio-bitrate (e.g. 256k) or deliver a lossless format (wav/flac) and let the platform encode")
|
|
172
|
+
verification = [{"step": "loudness",
|
|
173
|
+
"ok": bool(after.get("silent")) or (abs(float(after["input_i"]) - args.lufs) <= LUFS_TOLERANCE and float(after["input_tp"]) <= args.tp + TP_TOLERANCE_DB),
|
|
174
|
+
"lufs": float(after["input_i"]), "tp": float(after["input_tp"]), "target_lufs": args.lufs, "target_tp": args.tp,
|
|
175
|
+
"property": "loudness", "expected": {"lufs": args.lufs, "tp_max": args.tp},
|
|
176
|
+
"measured": {"lufs": float(after["input_i"]), "tp": float(after["input_tp"])},
|
|
177
|
+
"delta": {"lufs": round(float(after["input_i"]) - args.lufs, 2), "tp": round(float(after["input_tp"]) - args.tp, 2)},
|
|
178
|
+
"tolerance": {"lufs": LUFS_TOLERANCE, "tp": TP_TOLERANCE_DB}, "provenance": "user_specified"}]
|
|
179
|
+
written = probe(output, role="output")
|
|
180
|
+
out_audio = audio_length(written)
|
|
181
|
+
if in_audio and out_audio:
|
|
182
|
+
# a levels pass changes no timing: the sound is as long as the input's, to one AAC frame
|
|
183
|
+
tol = max(1024.0 / float(sr), AUDIO_LENGTH_TOLERANCE)
|
|
184
|
+
verification.append({"step": "duration", "property": "audio_duration", "ok": abs(out_audio - in_audio) <= tol,
|
|
185
|
+
"expected": round(float(in_audio), 3), "measured": round(float(out_audio), 3),
|
|
186
|
+
"delta": round(float(out_audio) - float(in_audio), 3), "tolerance": round(tol, 3),
|
|
187
|
+
"provenance": "input_premeasurement"})
|
|
188
|
+
else:
|
|
189
|
+
# a length that could not be read is not a length that matched
|
|
190
|
+
verification.append({"step": "duration", "property": "audio_duration", "ok": False,
|
|
191
|
+
"reason": "not_measured: the " + ("input" if not in_audio else "output") + " reports no duration"})
|
|
161
192
|
emit(output, result=result, dropped_non_av_streams=dropped_streams,
|
|
162
193
|
measured={k: stats[k] for k in ("input_i", "input_tp", "input_lra", "input_thresh", "target_offset", "silent")},
|
|
163
|
-
targets={"lufs": args.lufs, "tp": args.tp, "lra": args.lra},
|
|
164
|
-
verification=[{"step": "loudness",
|
|
165
|
-
"ok": bool(after.get("silent")) or (abs(float(after["input_i"]) - args.lufs) <= 1.0 and float(after["input_tp"]) <= args.tp + TP_TOLERANCE_DB),
|
|
166
|
-
"lufs": float(after["input_i"]), "tp": float(after["input_tp"]), "target_lufs": args.lufs, "target_tp": args.tp,
|
|
167
|
-
"property": "loudness", "expected": {"lufs": args.lufs, "tp_max": args.tp},
|
|
168
|
-
"measured": {"lufs": float(after["input_i"]), "tp": float(after["input_tp"])},
|
|
169
|
-
"delta": {"lufs": round(float(after["input_i"]) - args.lufs, 2), "tp": round(float(after["input_tp"]) - args.tp, 2)},
|
|
170
|
-
"tolerance": {"lufs": 1.0, "tp": TP_TOLERANCE_DB}, "provenance": "user_specified"}])
|
|
194
|
+
targets={"lufs": args.lufs, "tp": args.tp, "lra": args.lra}, verification=verification)
|
|
171
195
|
return 0
|
|
172
196
|
|
|
173
197
|
|
|
198
|
+
def audio_length(meta: dict) -> "Optional[float]":
|
|
199
|
+
"""The sound's own length: the stream's duration, else Matroska's DURATION tag. Never the
|
|
200
|
+
container's, which is the picture's when the picture runs longer (and would fail a correct
|
|
201
|
+
output against an expected length the sound never had)."""
|
|
202
|
+
audio = meta.get("audio") or {}
|
|
203
|
+
return audio.get("duration") or audio.get("tagged_duration")
|
|
204
|
+
|
|
205
|
+
|
|
174
206
|
def _kbps(value: str) -> int:
|
|
175
207
|
v = value.lower().rstrip("k")
|
|
176
208
|
try:
|
package/scripts/silence.py
CHANGED
|
@@ -15,7 +15,7 @@ import argparse
|
|
|
15
15
|
import json
|
|
16
16
|
import os
|
|
17
17
|
import sys
|
|
18
|
-
from typing import List, Tuple
|
|
18
|
+
from typing import Any, Dict, List, Tuple
|
|
19
19
|
|
|
20
20
|
# `detect` moved into _common/probe.py in 1.16.0 so metadata.py --auto-chapters can measure the
|
|
21
21
|
# same silences without importing this tool; the body is unchanged and the name still lives here.
|
|
@@ -44,6 +44,7 @@ def merge_spans(spans: List[Tuple[float, float]]) -> List[Tuple[float, float]]:
|
|
|
44
44
|
return out
|
|
45
45
|
|
|
46
46
|
|
|
47
|
+
DIGITAL_SILENCE_DB = -90.0 # 2.20: at or below this a span is digital silence (16-bit dither sits near -96)
|
|
47
48
|
BREATH_FLOOR = 0.12 # shortest gap silencedetect is asked for under --speech-aware
|
|
48
49
|
|
|
49
50
|
|
|
@@ -255,6 +256,33 @@ def _words_from_transcript(data):
|
|
|
255
256
|
return out, had_segments, untimed
|
|
256
257
|
|
|
257
258
|
|
|
259
|
+
def silence_kinds(path: str, silences: "List[Tuple[float, float]]", duration: float,
|
|
260
|
+
threshold: float) -> "Dict[str, Any]":
|
|
261
|
+
"""Which detected silences are digital silence and which are quiet sound under the threshold
|
|
262
|
+
(2.20). Both are removed alike, but they are not the same finding: a room tone, a breath or a
|
|
263
|
+
whisper under --threshold is sound, and a report that called all of it "silence" hid that the
|
|
264
|
+
cut took audible material. One more silencedetect pass at DIGITAL_SILENCE_DB finds the digital
|
|
265
|
+
spans; a detected silence that is at least 99% digital counts as digital."""
|
|
266
|
+
if not silences or STATE.dry_run:
|
|
267
|
+
return {}
|
|
268
|
+
digital = [(s, duration if e == float("inf") else e) for s, e in detect(path, DIGITAL_SILENCE_DB, 0.02)]
|
|
269
|
+
labels, tally = [], {"digital": [0, 0.0], "below_threshold": [0, 0.0]}
|
|
270
|
+
for s, e in silences:
|
|
271
|
+
e = duration if e == float("inf") else e
|
|
272
|
+
span = max(e - s, 1e-9)
|
|
273
|
+
covered = sum(max(0.0, min(e, de) - max(s, ds)) for ds, de in digital)
|
|
274
|
+
kind = "digital" if covered / span >= 0.99 else "below_threshold"
|
|
275
|
+
labels.append(kind)
|
|
276
|
+
tally[kind][0] += 1
|
|
277
|
+
tally[kind][1] += e - s
|
|
278
|
+
return {"silence_kinds": labels,
|
|
279
|
+
"silence_breakdown": {"digital": {"count": tally["digital"][0], "seconds": round(tally["digital"][1], 3),
|
|
280
|
+
"level_db": DIGITAL_SILENCE_DB},
|
|
281
|
+
"below_threshold": {"count": tally["below_threshold"][0],
|
|
282
|
+
"seconds": round(tally["below_threshold"][1], 3),
|
|
283
|
+
"level_db": threshold}}}
|
|
284
|
+
|
|
285
|
+
|
|
258
286
|
def main() -> int:
|
|
259
287
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
260
288
|
ap.add_argument("input")
|
|
@@ -344,6 +372,7 @@ def main() -> int:
|
|
|
344
372
|
if args.filler:
|
|
345
373
|
silence_keeps = keep_ranges(merge_spans(list(silences)), duration, args.margin, args.min_keep)
|
|
346
374
|
silence_only = max(0.0, duration - sum(e - s for s, e in silence_keeps))
|
|
375
|
+
kinds = silence_kinds(args.input, silences, duration, args.threshold)
|
|
347
376
|
summary = {
|
|
348
377
|
"silences": [[round(s, 3), None if e == float("inf") else round(e, 3)] for s, e in silences],
|
|
349
378
|
"keep": [[round(s, 3), round(e, 3)] for s, e in keeps],
|
|
@@ -351,6 +380,8 @@ def main() -> int:
|
|
|
351
380
|
"kept_duration": round(kept, 3),
|
|
352
381
|
"removed_seconds": round(silence_only, 3),
|
|
353
382
|
}
|
|
383
|
+
if kinds:
|
|
384
|
+
summary.update(kinds)
|
|
354
385
|
if args.speech_aware:
|
|
355
386
|
summary["speech_aware"] = {
|
|
356
387
|
"min_silence": args.min_silence, "floor": min(BREATH_FLOOR, args.min_silence),
|
package/scripts/waveform.py
CHANGED
|
@@ -35,6 +35,7 @@ import os
|
|
|
35
35
|
import sys
|
|
36
36
|
|
|
37
37
|
from _platforms import PLATFORMS, PLATFORM_CHOICES, resolve as resolve_platform
|
|
38
|
+
from _common.probe import SILENT_PEAK_DBFS
|
|
38
39
|
from _common import (add_common, apply_common, aac_args, default_output, die, draws_chroma, emit, ffmpeg_base,
|
|
39
40
|
info, load_brand, pad_filters, probe, rgb_chain_to_bt709, run, STATE, validate_color, video_args, X264_PRESETS,
|
|
40
41
|
fmt_secs, child_args, run_tool, measured_level_dbfs)
|
|
@@ -60,7 +61,7 @@ def main() -> int:
|
|
|
60
61
|
ap.add_argument("--on-silent", choices=["warn", "fail"], default="warn",
|
|
61
62
|
help="input audio whose peak is at or below --silence-threshold (the render would be a flat "
|
|
62
63
|
"line): warn (default: render it, report `silent: true`) or fail (refuse before ffmpeg runs)")
|
|
63
|
-
ap.add_argument("--silence-threshold", type=float, default
|
|
64
|
+
ap.add_argument("--silence-threshold", type=float, default=SILENT_PEAK_DBFS, metavar="DBFS",
|
|
64
65
|
help="peak level in dBFS at or below which the input counts as silent (default -50)")
|
|
65
66
|
ag = ap.add_argument_group("audiogram (1.16)",
|
|
66
67
|
"The visualisation over a picture, for an episode that has no video. The image is a "
|