ffmpeg-skill 0.7.0 → 0.8.0

package/README.md

@@ -124,6 +124,22 @@ More examples: [examples/README.md](examples/README.md). To see everything run e
 
 All scripts: Python 3.9+, standard library only, `--help`, non-zero exit + stderr message on failure.
 
+## Measured, not assumed
+
+`tests/corpus.py` downloads public real-device footage (GoPro, DJI, iPhone incl. Dolby Vision, Android screen recordings, HDR10, 24p, Tears of Steel) and runs the toolchain on it; `tests/bench_sync.py`, `bench_silence.py` and `bench_scenes.py` score the algorithms against known ground truth.
+
+| What | Result (0.8.0, local ffmpeg 6.1) |
+|---|---|
+| Real-device corpus, 10 files | 92 verify steps, all pass after fixes |
+| sync.py, ±30 s offsets, gain/noise/EQ, real dialogue+music | 120 s windows (the documented rule): 40/40 within 10 ms, max 1.1 ms. 60 s stress windows: 95 % within 10 ms, 4 of 5 misses flagged by confidence |
+| silence.py, 20 cases, known gaps | 0 missed gaps, ≤ 1 ms leftover silence |
+| scenes.py, 53 hard cuts between single takes (GoPro/DJI/iPhone/…) | precision 0.95, recall 1.00, F1 0.97 at the default threshold |
+
+```bash
+python3 tests/corpus.py --fetch --verify     # ~1.4 GB download, then verify (slow on 4K)
+python3 tests/bench_sync.py --cases 100
+```
+
 ## MCP
 
 ```json

package/SKILL.md

@@ -167,8 +167,12 @@ scenes.py INPUT [--threshold 10] [--min-scene 1] [--highlights N [--target SECON
 Lists scenes with audio energy, the loudest moments, and (with
 `--highlights`) proposes N ranges that add up to `--target` seconds, biased to
 the loudest window of each scene. Review the sheet + JSON, adjust the EDL, then
-`cut.py --segments`. It is a proposal engine, not a judgement of content:
-tell the user what it picked and why (energy, scene length).
+`cut.py --segments`. Cut detection is a one-frame spike test (benchmark on
+hard cuts between real single takes: precision 0.95, recall 1.00 at the default
+threshold; raise `--threshold` to 12 for 0.98 precision at 0.94 recall).
+Dissolves and very slow fades are not cuts and will be missed. Highlights are
+a proposal engine, not a judgement of content: tell the user what it picked
+and why (energy, scene length).
 
 ### check.py — pre-delivery compliance
 ```
@@ -300,7 +304,12 @@ video with the second file's audio aligned (video stream copied).
 the clock difference in ppm, and resamples the second file so a 60-minute
 take stays in sync (typical consumer devices drift 20-500 ppm = up to 1.8 s/h).
 Use it whenever the recording is longer than ~10 minutes. Check `confidence`
-(0–1); below ~0.3 the match is doubtful — use a window with a clear event.
+(0–1, normalised correlation with a runner-up penalty); below 0.3 the match is
+doubtful. Benchmark on real dialogue/music (±30 s offsets, gain, noise, EQ):
+with the default 120 s window 40/40 within 10 ms (max 1.1 ms); with a 60 s
+window 95 %, misses flagged below 0.3. Keep `--analyze-seconds` at least 4×
+`--max-offset` (default 120 s vs 30 s): lags with under 35 % overlap are
+ignored, so an offset larger than ~60 % of the window cannot be found.
 
 ### color.py — HDR to SDR, LUTs, colour tags, Dolby Vision
 ```

package/bin/install.js

@@ -88,7 +88,11 @@ for (const t of targets) {
     }
     fs.rmSync(t.dir, { recursive: true, force: true });
     fs.mkdirSync(t.dir, { recursive: true });
-    for (const item of PAYLOAD) copyRecursive(path.join(ROOT, item), path.join(t.dir, item));
+    for (const item of PAYLOAD) {
+      const src = path.join(ROOT, item);
+      if (!fs.existsSync(src)) { if (item !== 'SKILL.md' && item !== 'scripts') continue; throw new Error(`missing ${item} in package`); }
+      copyRecursive(src, path.join(t.dir, item));
+    }
     console.log(`installed ${t.label}: ${t.dir}`);
   } catch (err) {
     failed = true;

package/mcp/server.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""ffmpeg-skill as an MCP server (stdio, JSON-RPC 2.0) — standard library only.
+
+Every script in ../scripts becomes a tool; arguments are passed as an argv list
+or as a flat object of flags. Results are the script's --json output.
+
+Run:
+  python3 mcp/server.py                         # stdio transport
+Claude Desktop / Claude Code config example:
+  {"mcpServers": {"ffmpeg-skill": {"command": "python3", "args": ["/path/to/ffmpeg-skill/mcp/server.py"]}}}
+"""
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any, Dict, List
+
+HERE = Path(__file__).resolve().parent
+SCRIPTS = HERE.parent / "scripts"
+PROTOCOL_VERSION = "2024-11-05"
+
+TOOLS: Dict[str, str] = {
+    "probe": "Inspect a media file: duration, fps/VFR, resolution, codecs, HDR, rotation, audio. Args: inputs (list), analyze (bool), compact (bool).",
+    "cut": "Cut a range or several segments, lossless when possible. Args: input, start, end, duration, segments, accurate, output.",
+    "fit": "Fit to a duration (speed/trim) and/or aspect (pad/crop). Args: input, duration, method, aspect, fit, width, fps, smooth, output.",
+    "caption": "Burn SRT/ASS or timed text; animated/karaoke styles; brand. Args: input, srt, ass, text, animate, karaoke, font, size, position, brand, output.",
+    "overlay": "Composite a logo/image/text with timing and fade. Args: input, image, text, logo, position, start, end, fade, opacity, scale, brand, output.",
+    "graphics": "Motion-graphics templates: lower-third, title, chapter, progress, countdown, bug. Args: input, template, name, title, subtitle, start, end, brand, output.",
+    "sync": "Detect offset between two recordings by audio, optional drift fix. Args: reference, second, fix_drift, replace_audio, trim_second, output.",
+    "multicam": "Align N cameras/recorders and switch between them. Args: inputs (list), switch, auto, audio, fix_drift, output.",
+    "audio": "Denoise/voice chain, music bed with ducking, fades, downmix, replace. Args: input, voice, denoise, music, duck, music_volume, fade_in, fade_out, downmix, replace, output.",
+    "loudness": "Two-pass EBU R128 normalisation. Args: input, lufs, tp, measure_only, output.",
+    "silence": "Remove dead air / list silences. Args: input, threshold, min_silence, margin, list, edl, output.",
+    "join": "Concatenate clips with transitions, normalising size/fps/audio. Args: inputs (list), transition, duration, width, height, fps, output.",
+    "color": "HDR→SDR tone mapping, LUTs, retag, strip Dolby Vision. Args: input, to_sdr, lut, retag, strip_dovi, tonemap, output.",
+    "export": "Platform presets: youtube, youtube4k, reels, x, prores, h265, gif. Args: input, preset, fit, output.",
+    "check": "Pre-delivery compliance per platform. Args: input, platform, no_loudness.",
+    "scenes": "Scene changes, audio peaks, highlight proposals. Args: input, highlights, target, edl, sheet.",
+    "look": "Contact sheet / frames / before-after PNG for visual checks. Args: input, at (list), tiles, compare, output.",
+    "render": "Render a whole edit from project.json. Args: project, fast, stop_after, init.",
+    "verify": "Run the toolchain on real files and report PASS/FAIL. Args: paths (list), quick, report.",
+    "report": "HTML delivery report. Args: after, before, platform, commands, notes, title, output.",
+}
+POSITIONAL = {"probe": ["inputs"], "sync": ["reference", "second"], "multicam": ["inputs"], "join": ["inputs"], "verify": ["paths"], "render": ["project"]}
+
+
+def build_argv(name: str, args: Dict[str, Any]) -> List[str]:
+    if isinstance(args.get("argv"), list):
+        argv = [str(a) for a in args["argv"]]
+        if name not in ("look", "probe") and "--json" not in argv and "--help" not in argv:
+            argv.append("--json")
+        return argv
+    argv: List[str] = []
+    args = dict(args)
+    pos = POSITIONAL.get(name, ["input"])
+    for key in pos:
+        val = args.pop(key, None)
+        if val is None:
+            continue
+        if isinstance(val, list):
+            argv += [str(v) for v in val]
+        else:
+            argv.append(str(val))
+    for key, val in args.items():
+        if val is None or val is False:
+            continue
+        flag = "--" + key.replace("_", "-")
+        if key == "output":
+            flag = "-o"
+        if key == "lufs" and name == "loudness":
+            flag = "-I"
+        if val is True:
+            argv.append(flag)
+        elif isinstance(val, list):
+            for v in val:
+                argv += [flag, str(v)]
+        else:
+            argv += [flag, str(val)]
+    if name not in ("look", "probe") and "--json" not in argv and "--help" not in argv:
+        argv.append("--json")
+    return argv
+
+
+def call_tool(name: str, args: Dict[str, Any]) -> Dict[str, Any]:
+    script = SCRIPTS / f"{name}.py"
+    if not script.exists():
+        return {"isError": True, "content": [{"type": "text", "text": f"unknown tool {name}"}]}
+    argv = build_argv(name, args or {})
+    proc = subprocess.run([sys.executable, str(script)] + argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+    stdout = proc.stdout.strip()
+    text = stdout
+    structured = None
+    try:
+        structured = json.loads(stdout) if stdout.startswith("{") or stdout.startswith("[") else None
+    except ValueError:
+        structured = None
+    if proc.returncode != 0:
+        err = proc.stderr.strip().splitlines()
+        tail = "\n".join(err[-12:])
+        return {"isError": True, "content": [{"type": "text", "text": f"{name} failed (exit {proc.returncode})\n{tail}"}]}
+    if structured is None:
+        text = stdout or "\n".join(proc.stderr.strip().splitlines()[-5:])
+    result: Dict[str, Any] = {"content": [{"type": "text", "text": text}]}
+    if structured is not None:
+        result["structuredContent"] = structured if isinstance(structured, dict) else {"result": structured}
+    return result
+
+
+def tool_list() -> List[Dict[str, Any]]:
+    out = []
+    for name, desc in TOOLS.items():
+        out.append({
+            "name": name,
+            "description": desc + " Pass either named args (flags without dashes, underscores for hyphens) or argv (raw CLI list). Media paths must be absolute.",
+            "inputSchema": {"type": "object", "properties": {"argv": {"type": "array", "items": {"type": "string"}, "description": "raw CLI arguments"}}, "additionalProperties": True},
+        })
+    return out
+
+
+def handle(req: Dict[str, Any]) -> Dict[str, Any]:
+    method = req.get("method")
+    params = req.get("params") or {}
+    if method == "initialize":
+        return {"protocolVersion": PROTOCOL_VERSION, "capabilities": {"tools": {}}, "serverInfo": {"name": "ffmpeg-skill", "version": version()}}
+    if method == "tools/list":
+        return {"tools": tool_list()}
+    if method == "tools/call":
+        return call_tool(params.get("name", ""), params.get("arguments") or {})
+    if method == "ping":
+        return {}
+    raise KeyError(method)
+
+
+def version() -> str:
+    try:
+        return json.loads((HERE.parent / "package.json").read_text())["version"]
+    except Exception:
+        return "0"
+
+
+def main() -> int:
+    if "--list" in sys.argv:
+        for t in tool_list():
+            print(f"{t['name']:10s} {t['description'].split(' Pass either')[0]}")
+        return 0
+    if "--call" in sys.argv:  # debugging helper: --call NAME '{"input": "..."}'
+        i = sys.argv.index("--call")
+        name = sys.argv[i + 1]
+        args = json.loads(sys.argv[i + 2]) if len(sys.argv) > i + 2 else {}
+        print(json.dumps(call_tool(name, args), indent=2))
+        return 0
+    stdin = sys.stdin.buffer
+    stdout = sys.stdout.buffer
+    for raw in stdin:
+        line = raw.decode("utf-8", errors="replace").strip()
+        if not line:
+            continue
+        try:
+            req = json.loads(line)
+        except ValueError:
+            continue
+        if "id" not in req:  # notification
+            continue
+        try:
+            result = handle(req)
+            resp = {"jsonrpc": "2.0", "id": req["id"], "result": result}
+        except KeyError as exc:
+            resp = {"jsonrpc": "2.0", "id": req["id"], "error": {"code": -32601, "message": f"method not found: {exc}"}}
+        except Exception as exc:  # noqa: BLE001
+            resp = {"jsonrpc": "2.0", "id": req["id"], "error": {"code": -32000, "message": str(exc)}}
+        stdout.write((json.dumps(resp) + "\n").encode("utf-8"))
+        stdout.flush()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ffmpeg-skill",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "description": "Agent Skill that lets coding agents (Claude Code, Cursor, Codex) do professional video editing with local FFmpeg: MCP server, batch processing, declarative project rendering, brand kits, motion-graphics templates, HTML delivery reports, scene detection, delivery checks, cut, silence removal, transitions, multicam, captions, sync with drift correction, HDR to SDR, LUTs, audio clean-up and ducking, loudness, platform exports. No API keys, no cloud, no dependencies.",
   "keywords": ["ffmpeg", "video", "agent-skill", "claude-code", "cursor", "codex", "skill", "video-editing"],
   "license": "MIT",
@@ -17,6 +17,7 @@
   "files": [
     "bin/",
     "scripts/",
+    "mcp/",
     "SKILL.md",
     "README.md",
     "LICENSE"

package/scripts/loudness.py

@@ -34,7 +34,9 @@ def measure(path: str, I: float, tp: float, lra: float) -> dict:
     data = json.loads(m.group(0))
     for k in ("input_i", "input_tp", "input_lra", "input_thresh", "target_offset"):
         if data.get(k) in (None, "-inf", "inf", "nan"):
-            die(f"loudnorm returned unusable value for {k}: {data.get(k)} (silent input?)")
+            data["silent"] = True
+            return data
+    data["silent"] = False
     return data
 
 
@@ -57,6 +59,12 @@ def main() -> int:
         die("input has no audio stream")
 
     stats = measure(args.input, args.lufs, args.tp, args.lra)
+    if stats.get("silent"):
+        info("audio is silent (integrated loudness -inf); nothing to normalise")
+        if args.measure_only:
+            print(json.dumps({"silent": True, "input_i": "-inf"}, indent=2))
+            return 0
+        die("input audio is silent; loudness normalisation is meaningless (use audio.py --replace to add a track)")
     info(f"measured: {float(stats['input_i']):.1f} LUFS, TP {float(stats['input_tp']):.1f} dBTP, LRA {float(stats['input_lra']):.1f} LU")
     if args.measure_only:
         print(json.dumps({k: stats[k] for k in ("input_i", "input_tp", "input_lra", "input_thresh", "target_offset")}, indent=2))
@@ -79,7 +87,8 @@ def main() -> int:
     run(cmd)
 
     after = measure(output, args.lufs, args.tp, args.lra)
-    info(f"result:   {float(after['input_i']):.1f} LUFS, TP {float(after['input_tp']):.1f} dBTP (target {args.lufs} LUFS)")
+    if not after.get("silent"):
+        info(f"result:   {float(after['input_i']):.1f} LUFS, TP {float(after['input_tp']):.1f} dBTP (target {args.lufs} LUFS)")
     emit(output)
     return 0
 

package/scripts/scenes.py

@@ -23,17 +23,49 @@ from typing import Dict, List, Tuple
 
 from _common import add_common, apply_common, die, emit, ffmpeg_base, info, print_json, probe, require_tool, run
 
-SCENE_RE = re.compile(r"lavfi\.scd\.time=([0-9.]+)")
+SCORE_RE = re.compile(r"frame:(\d+)\s+pts:\d+\s+pts_time:([0-9.]+)")
 
 
-def detect_scenes(path: str, threshold: float, min_len: float, duration: float) -> List[float]:
+def detect_scenes(path: str, threshold: float, min_len: float, duration: float, ratio: float = 3.0) -> List[float]:
+    """Scene cuts = frames whose scdet score is above `threshold` AND stands out from its
+    neighbourhood (score > ratio x median of the surrounding +-12 frames). Sustained motion,
+    flashes and fast pans raise the score on many consecutive frames and are rejected;
+    a real cut is a one-frame spike. On real footage this roughly doubles precision at
+    equal recall compared with the raw scdet threshold."""
     ffmpeg = require_tool("ffmpeg")
     proc = subprocess.run([ffmpeg, "-hide_banner", "-nostdin", "-i", path, "-an", "-vf",
-                           f"scale=320:-2,scdet=threshold={threshold}:sc_pass=1,metadata=print:file=-", "-f", "null", "-"],
+                           "scale=320:-2,scdet=threshold=0:sc_pass=1,metadata=print:file=-", "-f", "null", "-"],
                           stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-    times = [float(t) for t in SCENE_RE.findall(proc.stdout)]
+    times: List[float] = []
+    scores: List[float] = []
+    cur_t = None
+    for line in proc.stdout.splitlines():
+        m = SCORE_RE.match(line)
+        if m:
+            cur_t = float(m.group(2))
+            continue
+        if line.startswith("lavfi.scd.score=") and cur_t is not None:
+            try:
+                times.append(cur_t)
+                scores.append(float(line.split("=", 1)[1]))
+            except ValueError:
+                pass
     cuts = [0.0]
-    for t in times:
+    if not scores:
+        return cuts
+    w = 12
+    for i, sc in enumerate(scores):
+        if sc < threshold:
+            continue
+        lo, hi = max(0, i - w), min(len(scores), i + w + 1)
+        neigh = sorted(scores[lo:i] + scores[i + 1:hi])
+        med = neigh[len(neigh) // 2] if neigh else 0.0
+        if sc < ratio * max(med, 0.5):
+            continue
+        # keep only the local maximum inside +-2 frames
+        if any(scores[j] > sc for j in range(max(0, i - 2), min(len(scores), i + 3)) if j != i):
+            continue
+        t = times[i]
         if t - cuts[-1] >= min_len:
             cuts.append(t)
     if duration - cuts[-1] < min_len and len(cuts) > 1:
@@ -60,7 +92,8 @@ def audio_envelope(path: str, step_s: float) -> List[float]:
 def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
     ap.add_argument("input")
-    ap.add_argument("--threshold", type=float, default=10.0, help="scdet threshold 0-100 (default 10; lower = more cuts)")
+    ap.add_argument("--threshold", type=float, default=8.0, help="minimum scdet score for a cut, 0-100 (default 8)")
+    ap.add_argument("--ratio", type=float, default=3.0, help="a cut must exceed this multiple of the neighbouring frames' median score (default 3; lower = more cuts)")
     ap.add_argument("--min-scene", type=float, default=1.0, help="ignore cuts closer than this in seconds (default 1)")
     ap.add_argument("--highlights", type=int, default=0, help="number of highlight ranges to propose")
     ap.add_argument("--target", type=float, help="with --highlights: total seconds the picks should add up to (trims long scenes)")
@@ -75,7 +108,7 @@ def main() -> int:
     if not meta.get("video"):
         die("input has no video stream")
     dur = meta.get("duration") or 0.0
-    cuts = detect_scenes(args.input, args.threshold, args.min_scene, dur)
+    cuts = detect_scenes(args.input, args.threshold, args.min_scene, dur, args.ratio)
     bounds = cuts + [dur]
     step_s = 0.5
     env = audio_envelope(args.input, step_s) if meta.get("audio") else []

package/scripts/sync.py

@@ -92,6 +92,14 @@ def ifft(a: List[complex]) -> List[complex]:
 
 
 def cross_correlate(ref: List[float], other: List[float], max_lag: int):
+    """Normalised cross-correlation over the overlapping region only.
+
+    The raw FFT correlation sum grows with the overlap length, so with a 60 s window a
+    correct 28 s offset (32 s overlap) loses to a wrong 2 s offset (58 s overlap) on
+    music-like material. Dividing each lag by the energy of the overlapping parts
+    (prefix sums, O(1) per lag) makes lags comparable and turns the peak value into a
+    real similarity score in 0..1 that doubles as the confidence.
+    """
     n = 1
     while n < len(ref) + len(other):
         n <<= 1
@@ -99,15 +107,50 @@ def cross_correlate(ref: List[float], other: List[float], max_lag: int):
     fb = fft([complex(x) for x in other] + [0j] * (n - len(other)))
     prod = [x * y.conjugate() for x, y in zip(fa, fb)]
     corr = ifft(prod)
-    # corr[k] = sum ref[i+k]*other[i]  -> lag k means 'other' is delayed by k relative to ref? see below
-    best_lag, best_val = 0, -float("inf")
+    # prefix sums of squares for overlap energy
+    def prefix(v: List[float]) -> List[float]:
+        out = [0.0]
+        acc = 0.0
+        for x in v:
+            acc += x * x
+            out.append(acc)
+        return out
+    pr, po = prefix(ref), prefix(other)
+    lr, lo = len(ref), len(other)
     max_lag = min(max_lag, n // 2 - 1)
+    best_lag, best_val, second = 0, -float("inf"), -float("inf")
+    # ignore lags with less than 35 % overlap: with the documented rule (analysis window >= 4x the
+    # largest expected offset) true offsets always keep >= 75 % overlap, while short-overlap lags are
+    # where coincidental matches on quasi-periodic material (music, tone beds) live
+    min_overlap = max(10, int(0.35 * min(lr, lo)))
+    scores = []
     for lag in range(-max_lag, max_lag + 1):
-        val = corr[lag % n].real
+        # corr[lag] = sum_i ref[i] * other[i - lag]  -> ref index range and other index range overlap:
+        r0, r1 = max(0, lag), min(lr, lo + lag)
+        if r1 - r0 < min_overlap:
+            continue
+        e_ref = pr[r1] - pr[r0]
+        e_oth = po[r1 - lag] - po[r0 - lag]
+        denom = math.sqrt(e_ref * e_oth)
+        if denom <= 0:
+            continue
+        val = corr[lag % n].real / denom
+        # mild preference for longer overlaps: a perfect match over 55 % of the window must not tie
+        # with a perfect match over 100 % (quasi-periodic material). Exponent 0.5: with the window rule (>= 4x offset) a true match keeps >= 75 % overlap (x0.87) while a coincidental 55 % match drops to x0.74; keeps large true
+        # offsets (28 s in 60 s = 53 % overlap -> x0.94) competitive while still breaking exact ties.
+        val *= ((r1 - r0) / min(lr, lo)) ** 0.5
+        scores.append((val, lag))
         if val > best_val:
+            second = best_val
             best_val, best_lag = val, lag
-    energy = math.sqrt(sum(x * x for x in ref) * sum(x * x for x in other)) or 1.0
-    return best_lag, best_val / energy
+        elif val > second and abs(lag - best_lag) > 5:
+            second = val
+    # confidence: peak similarity, penalised when a distant runner-up is nearly as good
+    conf = max(0.0, min(1.0, best_val))
+    if second > -float("inf") and best_val > 0:
+        margin = (best_val - second) / best_val
+        conf *= min(1.0, 0.5 + margin)
+    return best_lag, conf
 
 
 def refine(ref_s: List[float], oth_s: List[float], coarse_offset: float, fine_step: int, window_s: float) -> float:

package/scripts/verify.py

@@ -118,7 +118,7 @@ def main() -> int:
                 plan.append(("overlay text", ["overlay.py", cut, "--text", "verify", "--position", "top-left", "-o", f"{stem}_ovl.mp4"] + fast))
                 plan.append(("look sheet", ["look.py", cut, "-o", f"{stem}_sheet.png"]))
                 if (meta.get("video") or {}).get("hdr"):
-                    plan.append(("color to-sdr", ["color.py", str(f), "--to-sdr", "-o", f"{stem}_sdr.mp4"] + fast))
+                    plan.append(("color to-sdr", ["color.py", f"{stem}_acc.mp4", "--to-sdr", "-o", f"{stem}_sdr.mp4"] + fast))
                     plan.append(("hdr preserved", ["__check_hdr__", f"{stem}_acc.mp4"]))
                 plan.append(("probe analyze", ["probe.py", cut, "--analyze"]))
             plan.append(("export x", ["export.py", cut, "--preset", "x", "-o", f"{stem}_x.mp4"]))