ffmpeg-skill 2.10.11 → 2.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -3
- package/docs/contract.md +47 -7
- package/package.json +1 -1
- package/references/scripts.md +14 -7
- package/scripts/_common/__init__.py +4 -2
- package/scripts/_common/emit.py +154 -2
- package/scripts/_common/runner.py +136 -33
- package/scripts/_contract.py +50 -2
- package/scripts/render.py +91 -31
package/README.md
CHANGED
|
@@ -338,7 +338,7 @@ The contract is generated from the code that runs, not maintained beside it. For
|
|
|
338
338
|
|
|
339
339
|
Next to the tool list the document carries a top-level `deprecated` list (1.10): what 2.0.0 removes, since when, the replacement and the surface it lives on. `docs/contract.md` "What 2.0 changes" is written from it.
|
|
340
340
|
|
|
341
|
-
`contract_version` (1.0) is separate from the skill version, so a consumer can pin the shape and read the version for provenance. The document also states the invocation mapping (structured arguments → argv), the JSON shapes for success and failure (`{"status": "failed", "error": {"kind": "input | ffmpeg | output | missing_tool | timeout | verification | interrupted", "message": …}}`), and that no tool runs a shell or executes anything other than the named script, `ffmpeg` and `ffprobe`. Field-by-field reference: [docs/contract.md](docs/contract.md).
|
|
341
|
+
`contract_version` (1.0) is separate from the skill version, so a consumer can pin the shape and read the version for provenance. The document also states the invocation mapping (structured arguments → argv), the JSON shapes for success and failure (`{"status": "failed", "error": {"kind": "input | ffmpeg | output | missing_tool | timeout | verification | interrupted", "message": …}}`; a refusal adds `error.reason`, a stable code such as `output_exists` or `plan_inputs_changed`, and `error.details`, the facts behind it; warnings, fallbacks and skipped steps a run decided on its own are in `diagnostics`), and that no tool runs a shell or executes anything other than the named script, `ffmpeg` and `ffprobe`. Field-by-field reference: [docs/contract.md](docs/contract.md).
|
|
342
342
|
|
|
343
343
|
### MCP
|
|
344
344
|
|
|
@@ -388,7 +388,8 @@ The short list for humans. The agent-facing version, with the reasoning, is the
|
|
|
388
388
|
- **Non-Latin text picks a font by script (1.12).** Japanese, Chinese, Korean, Arabic, Hebrew, Devanagari, Thai, Cyrillic and Greek cues, titles and overlays resolve a font file that covers them automatically, and a machine with no such font fails the job (`kind: input`) instead of rendering boxes. `doctor --json`'s `fonts.scripts` says which languages this machine can render; `--lang ja|ko` disambiguates Han-only text; an explicit `--font`/`--font-file` is always kept.
|
|
389
389
|
- **Silence detection finds nothing?** The default threshold is −35 dBFS. The tool prints a hint with the track's measured level; raise the threshold (`silence.py --threshold -25`) or shorten `--min-silence`.
|
|
390
390
|
- **Sync results carry a confidence.** Below 0.3, or an offset near the edge of the analysis window, is probably wrong: enlarge `--analyze-seconds` or find a clap. Recordings over ten minutes from separate devices need `sync.py --fix-drift`.
|
|
391
|
-
- **Outputs are never overwritten silently.** An existing output path is refused (`kind: input`, before anything runs); pass `--overwrite` where a replacement is intended.
|
|
391
|
+
- **Outputs are never overwritten silently.** An existing output path is refused (`kind: input`, `error.reason: output_exists`, before anything runs); pass `--overwrite` where a replacement is intended.
|
|
392
|
+
- **Nothing happens silently.** A refusal names its `error.reason` and `error.details`; a stale output lock taken over, a run without a lock and a plan input nobody could fingerprint are recorded in `diagnostics` and on stderr. A refused plan and a lock takeover also append a line to an audit log (`<plan>.audit.jsonl`, or `render.py --audit-log FILE`).
|
|
392
393
|
- **Long chains belong in a plan.** Three hand-chained re-encodes lose quality and are hard to change; `render.py` runs the whole edit from one JSON file, and `--dry-run` shows every ffmpeg command before anything is written.
|
|
393
394
|
|
|
394
395
|
## FAQ
|
|
@@ -403,7 +404,7 @@ ffmpeg-skill edits footage you already have, on this machine, with FFmpeg tools
|
|
|
403
404
|
|
|
404
405
|
### Does it validate the filter graph before running?
|
|
405
406
|
|
|
406
|
-
No. `--dry-run` prints the ffmpeg command a writing tool would run and does not start ffmpeg, so ffmpeg never parses that filter graph. ffprobe still runs when the tool probes an input that exists. Unless `--overwrite` is passed, an existing output is refused (`kind: input`) before ffmpeg. An output that is the same file as an input is refused either way. No media file is written, and `verified` is false; `--plan FILE` does write the plan JSON. A path that is not on disk yet is not measured: the probe is a stub (0×0, duration 0), so a number taken from it is not a measurement. The measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) still run their measurements; only the artifact is skipped, and `silence` with `--filler --transcribe` also runs the speech engine, which may download a model. `overlay` measures the lines of a multi-line `--text` with one drawtext on a generated frame on FFmpeg 8.0, 8.0.1 and 8.1.0, where it draws them one by one. `verify` accepts `--dry-run` and ignores it: its steps run. `--plan FILE` is that dry run written down — the planned argv, the command lines, and a fingerprint of each existing input (size, plus a sha256 of the first and last 8 MiB). `render.py` and `verify.py` refuse `--plan`. `render.py FILE` runs a plan only when those fingerprints still match, by running the tool again with the planned argv, not by executing the recorded ffmpeg lines; `--dry-run` on that run only prints them. `doctor` reports whether the filters, encoders and bitstream filters the contract names are in this build (`ffmpeg -filters`, `-encoders`, `-bsfs`) and which fonts are installed. A name in that listing is not a check of the graph this command would build, and a listed GPU encoder is not a successful encode.
|
|
407
|
+
No. `--dry-run` prints the ffmpeg command a writing tool would run and does not start ffmpeg, so ffmpeg never parses that filter graph. ffprobe still runs when the tool probes an input that exists. Unless `--overwrite` is passed, an existing output is refused (`kind: input`) before ffmpeg. An output that is the same file as an input is refused either way. No media file is written, and `verified` is false; `--plan FILE` does write the plan JSON. A path that is not on disk yet is not measured: the probe is a stub (0×0, duration 0), so a number taken from it is not a measurement. The measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) still run their measurements; only the artifact is skipped, and `silence` with `--filler --transcribe` also runs the speech engine, which may download a model. `overlay` measures the lines of a multi-line `--text` with one drawtext on a generated frame on FFmpeg 8.0, 8.0.1 and 8.1.0, where it draws them one by one. `verify` accepts `--dry-run` and ignores it: its steps run. `--plan FILE` is that dry run written down — the planned argv, the command lines, and a fingerprint of each existing input (size, plus a sha256 of the first and last 8 MiB). `render.py` and `verify.py` refuse `--plan`. `render.py FILE` runs a plan only when those fingerprints still match (a mismatch is refused with each input's expected and actual fingerprint in `error.details`, and logged to `<plan>.audit.jsonl`), by running the tool again with the planned argv, not by executing the recorded ffmpeg lines; `--dry-run` on that run only prints them. `doctor` reports whether the filters, encoders and bitstream filters the contract names are in this build (`ffmpeg -filters`, `-encoders`, `-bsfs`) and which fonts are installed. A name in that listing is not a check of the graph this command would build, and a listed GPU encoder is not a successful encode.
|
|
407
408
|
|
|
408
409
|
### Can it remove burned-in subtitles or logos?
|
|
409
410
|
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`2.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`2.11.0`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -125,7 +125,7 @@ JSON would have been rewritten for that. The flat keys, typed per tool in `outpu
|
|
|
125
125
|
"contract_version": "1.0",
|
|
126
126
|
"deprecated": [{"what": "running on Python 3.9 (the requirements.python floor)", "since": "2.10.9", "replacement": "Python 3.10 or later; ...", "removed_in": "3.0.0", "where": "behaviour"}],
|
|
127
127
|
"removed": [{"what": "...", "since": "1.10.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
128
|
-
"skill": {"id": "ffmpeg-skill", "version": "2.
|
|
128
|
+
"skill": {"id": "ffmpeg-skill", "version": "2.11.0", "execution_mode": "local", "kind": "execution",
|
|
129
129
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
130
130
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
131
131
|
"requirements": {"python": ">=3.9 (standard library only; 3.9 deprecated since 2.10.9, removed in 3.0.0; 3.10+ recommended)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
|
|
@@ -259,16 +259,23 @@ asks for the CPU and gets what ffmpeg says about it. What ran is in the result's
|
|
|
259
259
|
|
|
260
260
|
`--plan FILE` (1.6) is a dry run that also writes a plan document: `{"plan_version": 1,
|
|
261
261
|
"tool", "argv", "cwd", "inputs": [{"path", "size", "sha256_head_tail"}], "commands",
|
|
262
|
-
"output", "verify": [{"tool": "probe"}, {"tool": "check", "platform"}], "notes"}`. It
|
|
262
|
+
"unbound_inputs", "output", "verify": [{"tool": "probe"}, {"tool": "check", "platform"}], "notes"}`. It
|
|
263
263
|
implies `--dry-run`, so the same execution rules apply; `inputs` covers the `-i` files of the
|
|
264
264
|
planned commands, every existing file named in argv (a recipe, a still, an SRT) and the
|
|
265
|
-
subtitle/LUT/font files a filter reads
|
|
265
|
+
subtitle/LUT/font files a filter reads; `unbound_inputs` (absolute paths, additive to version 1)
|
|
266
|
+
names the inputs that did not exist yet when the plan was made (an earlier step's output), which
|
|
267
|
+
therefore carry no fingerprint. Tools that print their document without `--json`
|
|
266
268
|
(`probe`, the analysis tools) still write the plan at exit; `verify.py` and `render.py`
|
|
267
269
|
refuse `--plan` (their steps run for real; a project file is already a plan). `render.py FILE` executes a plan:
|
|
268
|
-
it refuses (`kind: input`) when an input's size or head/tail
|
|
270
|
+
it refuses (`kind: input`, `error.reason: plan_inputs_changed`) when an input's size or head/tail
|
|
271
|
+
hash differs from the plan, with every input's `{path, status, expected, actual}` in
|
|
272
|
+
`error.details.inputs` (`actual` is `"missing"` for a file that is gone) beside `plan_sha256` and
|
|
273
|
+
`checked_at`, and appends one record to the audit log (below); a wrong `plan_version`, an unknown
|
|
274
|
+
tool, a vanished `cwd` or a malformed `argv` is refused the same way under its own reason. It then
|
|
269
275
|
runs the tool with the planned `argv`, then the verify steps (a `check` step may also carry
|
|
270
|
-
`"content": true`, added by hand, for `check.py --content`), and reports `plan`, `
|
|
271
|
-
`tool_result` and `
|
|
276
|
+
`"content": true`, added by hand, for `check.py --content`), and reports `plan`, `plan_sha256`,
|
|
277
|
+
`tool`, `tool_result`, `check` and `unbound_inputs` (each one also a `diagnostics.warnings` entry:
|
|
278
|
+
nothing checked that file). `plan_version` is bumped when the document's shape changes.
|
|
272
279
|
|
|
273
280
|
### Repeatability
|
|
274
281
|
|
|
@@ -732,6 +739,39 @@ strings. `retryable` is currently always `false`: none of the kinds are distingu
|
|
|
732
739
|
today from a deterministic failure that would fail identically on a blind retry, so nothing
|
|
733
740
|
here claims otherwise until real exit-code/stderr sniffing exists to back that up.
|
|
734
741
|
|
|
742
|
+
A refusal also carries `error.reason`, a stable snake_case code for *why*
|
|
743
|
+
(`contract --json` → `json_output.error_reasons` lists them: `output_exists`, `output_is_input`,
|
|
744
|
+
`output_is_directory`, `output_dir_missing`, `output_dir_not_writable`, `output_locked`, and
|
|
745
|
+
`render.py`'s `plan_*` reasons), and `error.details`, the facts the message is written from, so a
|
|
746
|
+
caller never parses the prose:
|
|
747
|
+
|
|
748
|
+
```
|
|
749
|
+
"error": {"kind": "input", "message": "refusing to overwrite existing output 'out.mp4': ...",
|
|
750
|
+
"code": "INPUT_INVALID", "retryable": false, "hint": "pass --overwrite, or choose another -o path",
|
|
751
|
+
"reason": "output_exists",
|
|
752
|
+
"details": {"outputs": [{"path": "out.mp4", "resolved": "/abs/out.mp4", "size": 48213, "mtime_ns": 1729660000000000000}]}}
|
|
753
|
+
```
|
|
754
|
+
|
|
755
|
+
A tool that runs another (`render.py` stages and plans) forwards the child's `reason` and
|
|
756
|
+
`details` in its own `error`. Both are absent on a failure that is not a refusal, like `hint`.
|
|
757
|
+
|
|
758
|
+
**Diagnostics.** What a run decided on its own -- a warning, a fallback it took, a step it
|
|
759
|
+
skipped -- is recorded as it happens (a stderr line too) and lands in the document as
|
|
760
|
+
`diagnostics: {warnings, fallbacks, skipped}`, each entry `{code, message, ...facts}`; it is
|
|
761
|
+
present on success and failure documents alike, only when one list is not empty, and an entry a
|
|
762
|
+
`render.py` stage reported carries `stage`. Codes today: `stale_lock_taken_over` (fallbacks),
|
|
763
|
+
`output_unlocked`, `lock_deadline_unset`, `audit_log_unwritable` and `plan_unbound_inputs`
|
|
764
|
+
(warnings).
|
|
765
|
+
|
|
766
|
+
**Audit log.** Events a later reader must be able to reconstruct are appended as JSON lines,
|
|
767
|
+
`{ts, tool, event, pid, plan_sha256, details}` (`plan_sha256` on plan events): `plan_rejected`,
|
|
768
|
+
`lock_takeover`, `output_unlocked`, and `output_is_input_refused` when a log was named. The file
|
|
769
|
+
is `render.py --audit-log FILE` when given; otherwise a refused plan's `<plan>.audit.jsonl` next
|
|
770
|
+
to the plan, and a lock event's hidden `.<output>.ffskill-audit.jsonl` beside the output. Nothing
|
|
771
|
+
is appended under `--dry-run`. A record that cannot be written is a `diagnostics.warnings` entry
|
|
772
|
+
(`audit_log_unwritable`) and never fails or blocks the run; the refusal's `details.audit_log`
|
|
773
|
+
says `{path, written}` either way.
|
|
774
|
+
|
|
735
775
|
## MCP relationship
|
|
736
776
|
|
|
737
777
|
`mcp/server.py` is a transport. It holds no tool table and no schema of its own:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.11.0",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
package/references/scripts.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Script reference
|
|
2
2
|
|
|
3
|
-
Every script prints the same information with `--help`; this file exists so the agent can read several at once. All scripts accept `--dry-run`, `--json`, `--json-brief` (since 1.11.0: the same JSON result trimmed to `status`, `output`, `dry_run`, `verified`, a compact `summary` of the output probe -- duration_s, width, height, fps, vcodec, acodec, channels, and lufs when measured -- the tool's own keys, and the count of commands instead of the command lines; it implies `--json`, leaves `--json`'s own output untouched, and failures print the usual full failure document), `--fast`, `--progress`, `--timeout SECONDS`, `--overwrite`, `--plan FILE` (the dry run written as a plan document that `render.py FILE` executes later; see render.py), `-o OUT`; every re-encoding tool and `export.py` also take `--hw`/`--no-hw` (h264/hevc/prores on Apple VideoToolbox, Apple Silicon only, CRF mapped to `-q:v`; AV1 stays on the CPU; a job VideoToolbox itself refuses -- the encoder cannot open or its compression session fails -- is re-encoded on the CPU and reported in `hw.fallback`/`hw.notes`, any other failure is not retried; a GPU encode adds a `notes` line that its quality is not CRF-equivalent; `FFMPEG_SKILL_HW=1` makes it the default for every tool except `export.py`, whose delivery presets need `--hw` itself; `render.py`/`batch.py --hw`/`--no-hw` apply to every stage, export included; results name the `encoder` that ran); every editing tool that re-encodes (not `export.py`, whose preset decides the codec) also takes `--codec h264|hevc|av1|prores` (the encoder for the re-encode; default x264 for SDR, x265 Main10 for HDR, unchanged) and `--quality N` (CRF scale, default 18 -- 30 for `proxy.py`; up to 63 for av1; ignored by prores). 2.0 removed `--crf` from these tools (it was `--quality`'s alias); `export.py` keeps its own `--crf`, which is not an alias. Every tool refuses (`kind: input`, before anything runs) to replace an existing output unless `--overwrite` is given. `--codec hevc` on SDR writes 8-bit BT.709 HEVC (`hvc1`), `av1` uses SVT-AV1 (libaom fallback), `prores` is 422 HQ and needs an explicit `-o NAME.mov` (or `.mkv`), `h264` refuses an HDR source (`kind: input`, run `color.py --to-sdr` first). `export.py` keeps choosing the codec from its preset and has neither `--codec` nor `--quality`; a `render.py` project cannot choose a codec either -- but `--dry-run` only guarantees nothing is written for writing tools: `probe` (read-only, `--dry-run` changes nothing) still runs ffprobe, `check`/`sync`/`multicam`/`scenes`/`cropdetect`/`report`/`silence`/`loudness`/`stabilize`/`join` still run their ffmpeg/ffprobe measurements (a dry-run plan rests on real numbers; they just don't write the final artifact), `overlay` measures a multi-line `--text`'s lines on FFmpeg 8.0, 8.0.1 and 8.1.0 (one drawtext on a generated frame), and `verify` accepts the flag but ignores it entirely. Exact per-tool semantics: `contract --json`'s `dry_run` field (or `docs/contract.md`).
|
|
3
|
+
Every script prints the same information with `--help`; this file exists so the agent can read several at once. All scripts accept `--dry-run`, `--json`, `--json-brief` (since 1.11.0: the same JSON result trimmed to `status`, `output`, `dry_run`, `verified`, a compact `summary` of the output probe -- duration_s, width, height, fps, vcodec, acodec, channels, and lufs when measured -- the tool's own keys, and the count of commands instead of the command lines; it implies `--json`, leaves `--json`'s own output untouched, and failures print the usual full failure document), `--fast`, `--progress`, `--timeout SECONDS`, `--overwrite`, `--plan FILE` (the dry run written as a plan document that `render.py FILE` executes later; see render.py), `-o OUT`; every re-encoding tool and `export.py` also take `--hw`/`--no-hw` (h264/hevc/prores on Apple VideoToolbox, Apple Silicon only, CRF mapped to `-q:v`; AV1 stays on the CPU; a job VideoToolbox itself refuses -- the encoder cannot open or its compression session fails -- is re-encoded on the CPU and reported in `hw.fallback`/`hw.notes`, any other failure is not retried; a GPU encode adds a `notes` line that its quality is not CRF-equivalent; `FFMPEG_SKILL_HW=1` makes it the default for every tool except `export.py`, whose delivery presets need `--hw` itself; `render.py`/`batch.py --hw`/`--no-hw` apply to every stage, export included; results name the `encoder` that ran); every editing tool that re-encodes (not `export.py`, whose preset decides the codec) also takes `--codec h264|hevc|av1|prores` (the encoder for the re-encode; default x264 for SDR, x265 Main10 for HDR, unchanged) and `--quality N` (CRF scale, default 18 -- 30 for `proxy.py`; up to 63 for av1; ignored by prores). 2.0 removed `--crf` from these tools (it was `--quality`'s alias); `export.py` keeps its own `--crf`, which is not an alias. Every tool refuses (`kind: input`, `error.reason: output_exists`, before anything runs) to replace an existing output unless `--overwrite` is given. Refusals carry `error.reason` and `error.details` (the facts behind the message; `contract --json` → `json_output.error_reasons`), and what a run decided on its own -- a stale output lock taken over, no lock possible, a plan input nobody fingerprinted -- is in `diagnostics: {warnings, fallbacks, skipped}` as well as on stderr. `--codec hevc` on SDR writes 8-bit BT.709 HEVC (`hvc1`), `av1` uses SVT-AV1 (libaom fallback), `prores` is 422 HQ and needs an explicit `-o NAME.mov` (or `.mkv`), `h264` refuses an HDR source (`kind: input`, run `color.py --to-sdr` first). `export.py` keeps choosing the codec from its preset and has neither `--codec` nor `--quality`; a `render.py` project cannot choose a codec either -- but `--dry-run` only guarantees nothing is written for writing tools: `probe` (read-only, `--dry-run` changes nothing) still runs ffprobe, `check`/`sync`/`multicam`/`scenes`/`cropdetect`/`report`/`silence`/`loudness`/`stabilize`/`join` still run their ffmpeg/ffprobe measurements (a dry-run plan rests on real numbers; they just don't write the final artifact), `overlay` measures a multi-line `--text`'s lines on FFmpeg 8.0, 8.0.1 and 8.1.0 (one drawtext on a generated frame), and `verify` accepts the flag but ignores it entirely. Exact per-tool semantics: `contract --json`'s `dry_run` field (or `docs/contract.md`).
|
|
4
4
|
|
|
5
5
|
## Time grammar (every time-taking flag, 1.9)
|
|
6
6
|
|
|
@@ -709,7 +709,7 @@ output; the result says so with `dropped_non_av_streams: true`.
|
|
|
709
709
|
render.py --init project.json # starter file
|
|
710
710
|
render.py project.json [--fast] [--dry-run] [--stop-after STAGE] [--work DIR --keep]
|
|
711
711
|
render.py project.json --cache DIR [--from STAGE] # reuse identical stages from a previous run
|
|
712
|
-
render.py plan.json
|
|
712
|
+
render.py plan.json [--audit-log FILE] # execute a plan written by <tool> --plan plan.json
|
|
713
713
|
render.py project.json --export-timeline edit.fcpxml|edit.edl|edit.otio # the cut for an editor, nothing rendered
|
|
714
714
|
```
|
|
715
715
|
`--export-timeline FILE` (2.1) writes the project's cut as an editor timeline instead of
|
|
@@ -734,12 +734,19 @@ few frames longer, because `join.py` offsets its crossfades by each part's conta
|
|
|
734
734
|
audio included.
|
|
735
735
|
A plan is a single tool's dry run as an artifact: `cut.py in.mp4 --start 2 --end 8
|
|
736
736
|
--plan cut.json` writes `{plan_version, tool, argv, inputs (path, size, sha256 of
|
|
737
|
-
head+tail), commands, output, verify}` and runs nothing
|
|
738
|
-
|
|
739
|
-
|
|
737
|
+
head+tail), unbound_inputs, commands, output, verify}` and runs nothing
|
|
738
|
+
(`unbound_inputs`: inputs that did not exist yet, so carry no fingerprint).
|
|
739
|
+
`render.py cut.json` re-fingerprints the inputs (refusing, `kind: input`,
|
|
740
|
+
`error.reason: plan_inputs_changed`, if any changed since the plan: every input's
|
|
741
|
+
`{path, status, expected, actual}` -- `actual` `"missing"` for a file that is gone --
|
|
742
|
+
plus `plan_sha256` and `checked_at` are in `error.details`, and one JSON line goes
|
|
743
|
+
to `<plan>.audit.jsonl`, or to `--audit-log FILE`; a log that cannot be written is
|
|
744
|
+
a `diagnostics.warnings` entry, never the failure, and a `--dry-run` writes none),
|
|
745
|
+
runs the tool with the planned argv, then the verify steps (probe; `check`
|
|
740
746
|
for a `--platform` or a platform export preset; add `"content": true` to a check
|
|
741
|
-
step for the black/frozen/silence rows), and reports `plan`, `tool`,
|
|
742
|
-
`tool_result` and `
|
|
747
|
+
step for the black/frozen/silence rows), and reports `plan`, `plan_sha256`, `tool`,
|
|
748
|
+
`tool_result`, `check` and `unbound_inputs` (each also a `diagnostics.warnings`
|
|
749
|
+
entry: nothing checked those files). Show the plan to the user, get the yes, execute:
|
|
743
750
|
one round trip instead of re-deriving the command.
|
|
744
751
|
`"export": {"preset": "reels", "normalize": true}` forwards `export.py --normalize`
|
|
745
752
|
so the rendered file meets the platform's loudness without a separate pass. Since
|
|
@@ -63,8 +63,9 @@ from _common.runner import (
|
|
|
63
63
|
STATE, _timed_out, _unwatch, _watch, X264_PRESETS
|
|
64
64
|
)
|
|
65
65
|
from _common.emit import (
|
|
66
|
-
_brief, _BRIEF_DROP, _brief_summary, _CURRENT_CTX, die, emit, info, _plan_at_exit, _plan_inputs,
|
|
67
|
-
PLAN_VERSION, print_json, _set_current_ctx,
|
|
66
|
+
absorb_stage, audit, _brief, _BRIEF_DROP, _brief_summary, _CURRENT_CTX, die, emit, info, note, _plan_at_exit, _plan_inputs,
|
|
67
|
+
_plan_unbound_inputs, _PLAN_STRIP, PLAN_VERSION, print_json, record_fallback, record_skip, refuse, _set_current_ctx, warn,
|
|
68
|
+
write_plan
|
|
68
69
|
)
|
|
69
70
|
from _common.probe import (
|
|
70
71
|
analyze_levels, _aspect_string, _bit_depth, decode_pcm_mono, detect_scenes, detect_silences, fingerprint,
|
|
@@ -216,6 +217,7 @@ __all__ = [
|
|
|
216
217
|
"time_arg", "_timed_out", "_to_float", "_to_int", "_unwatch", "validate_color", "verify_output",
|
|
217
218
|
"source_codec_video_args", "drawn_chain", "draws_chroma", "drawn_stream_format", "is_neutral_color", "rgb_island", "label_chain", "matrix_chain", "rgb_chain_to_bt709", "bt709_label_filter", "rgb_to_bt709_filter", "is_rgb", "video_args", "_VS15", "_VS16", "_watch", "WINDOWS_FONTS", "write_plan", "x264_args", "X264_PRESETS",
|
|
218
219
|
"_x264_raw", "_ZWJ",
|
|
220
|
+
"absorb_stage", "audit", "note", "_plan_unbound_inputs", "record_fallback", "record_skip", "refuse", "warn",
|
|
219
221
|
"ASR_ENGINES", "ASR_INSTALL_HINT", "_asr_run", "die_no_engine", "parse_srt", "transcribe",
|
|
220
222
|
"_transcribe_in", "transcribe_words", "_words_from_openai_whisper_json",
|
|
221
223
|
"_words_from_whisper_cpp_json", "whisper_word_timings", "write_srt",
|
package/scripts/_common/emit.py
CHANGED
|
@@ -13,7 +13,8 @@ from typing import Any, Dict, List, Optional, Sequence, Tuple
|
|
|
13
13
|
from _common.runner import Context, ERROR_CODE, ERROR_RETRYABLE, STATE
|
|
14
14
|
|
|
15
15
|
|
|
16
|
-
def die(msg: str, code: int = 1, kind: str = "input", *, ctx: "Optional[Context]" = None,
|
|
16
|
+
def die(msg: str, code: int = 1, kind: str = "input", *, ctx: "Optional[Context]" = None,
|
|
17
|
+
error_extra: "Optional[Dict[str, Any]]" = None, **extra: Any) -> "None":
|
|
17
18
|
"""Exit with a message. Under --json also print a machine-readable failure document
|
|
18
19
|
(status: failed) on stdout so callers get the same shape as a success; exit codes are unchanged.
|
|
19
20
|
|
|
@@ -21,7 +22,10 @@ def die(msg: str, code: int = 1, kind: str = "input", *, ctx: "Optional[Context]
|
|
|
21
22
|
platform rows, render.py's check stage, batch.py's per-item results, verify.py's steps) keeps
|
|
22
23
|
reporting that detail while the top-level status says failed. Before 1.4.3 those four printed
|
|
23
24
|
`status: "completed"` next to a non-zero exit code, so a caller keying on the status alone
|
|
24
|
-
read a failed delivery as a success.
|
|
25
|
+
read a failed delivery as a success.
|
|
26
|
+
|
|
27
|
+
`error_extra` goes inside `error` (refuse() passes `reason` and `details` there); the
|
|
28
|
+
collector's records (warn() and friends) are merged into the document as emit() merges them."""
|
|
25
29
|
hint = extra.pop("hint", None)
|
|
26
30
|
ctx = ctx or STATE # 1.10: the optional per-request Context (2.0 makes it required); STATE is the default instance
|
|
27
31
|
_set_current_ctx(ctx) # the atexit hook has no argument: it reads the ctx emit()/die() last used
|
|
@@ -48,11 +52,103 @@ def die(msg: str, code: int = 1, kind: str = "input", *, ctx: "Optional[Context]
|
|
|
48
52
|
}
|
|
49
53
|
if hint:
|
|
50
54
|
doc["error"]["hint"] = hint
|
|
55
|
+
for key, value in (error_extra or {}).items():
|
|
56
|
+
if value is not None:
|
|
57
|
+
doc["error"][key] = value
|
|
51
58
|
doc.update(extra)
|
|
59
|
+
_merge_collector(doc, ctx)
|
|
52
60
|
print_json(doc)
|
|
53
61
|
sys.exit(code)
|
|
54
62
|
|
|
55
63
|
|
|
64
|
+
def refuse(kind: str, reason: str, msg: str, *, hint: Optional[str] = None, code: int = 1,
|
|
65
|
+
ctx: "Optional[Context]" = None, extra: "Optional[Dict[str, Any]]" = None, **details: Any) -> "None":
|
|
66
|
+
"""die() for a refusal a caller has to act on: `error.reason` is a stable snake_case code for
|
|
67
|
+
*why* (output_exists, output_is_input, output_locked, plan_inputs_changed, ...), and
|
|
68
|
+
`error.details` holds the facts the message is written from (paths, expected and actual
|
|
69
|
+
values), so a caller never parses the prose. `extra` adds top-level keys as die()'s do."""
|
|
70
|
+
die(msg, code=code, kind=kind, ctx=ctx, hint=hint, error_extra={"reason": reason, "details": json_safe(details)}, **(extra or {}))
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
def _record(bucket: str, msg: str, code: str, ctx: "Optional[Context]", data: Dict[str, Any], prefix: str) -> Dict[str, Any]:
|
|
74
|
+
ctx = ctx or STATE
|
|
75
|
+
entry: Dict[str, Any] = {"code": code, "message": msg}
|
|
76
|
+
entry.update(data)
|
|
77
|
+
getattr(ctx, bucket).append(entry)
|
|
78
|
+
sys.stderr.write(f"{prefix}{msg}\n")
|
|
79
|
+
return entry
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
def warn(msg: str, code: str, *, ctx: "Optional[Context]" = None, **data: Any) -> Dict[str, Any]:
|
|
83
|
+
"""Say something went other than asked, on stderr AND in the document
|
|
84
|
+
(`diagnostics.warnings`: {code, message, ...data}). Use it where a bare info("warning: ...")
|
|
85
|
+
would leave the caller of --json nothing to read."""
|
|
86
|
+
return _record("warnings", msg, code, ctx, data, "warning: ")
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
def record_fallback(msg: str, code: str, *, ctx: "Optional[Context]" = None, **data: Any) -> Dict[str, Any]:
|
|
90
|
+
"""The run took another route than the asked one and went on (`diagnostics.fallbacks`)."""
|
|
91
|
+
return _record("fallbacks", msg, code, ctx, data, "")
|
|
92
|
+
|
|
93
|
+
|
|
94
|
+
def record_skip(msg: str, code: str, *, ctx: "Optional[Context]" = None, **data: Any) -> Dict[str, Any]:
|
|
95
|
+
"""A step that was asked for, or would normally run, did not (`diagnostics.skipped`)."""
|
|
96
|
+
return _record("skipped", msg, code, ctx, data, "")
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
def note(msg: str, ctx: "Optional[Context]" = None) -> None:
|
|
100
|
+
"""A plain `notes` line that also goes to stderr."""
|
|
101
|
+
ctx = ctx or STATE
|
|
102
|
+
if msg not in ctx.notes:
|
|
103
|
+
ctx.notes.append(msg)
|
|
104
|
+
sys.stderr.write(f"{msg}\n")
|
|
105
|
+
|
|
106
|
+
|
|
107
|
+
def _merge_collector(doc: Dict[str, Any], ctx: "Context") -> None:
|
|
108
|
+
"""The collector into a result or failure document: ctx.notes join `notes` (once each), and
|
|
109
|
+
the three record lists become `diagnostics`, present only when one of them is not empty
|
|
110
|
+
(`warnings` and `skipped` are not top-level keys: check.py's `warnings` is a count and join.py's
|
|
111
|
+
`skipped` its left-out inputs)."""
|
|
112
|
+
extra_notes = [n for n in ctx.notes if n not in (doc.get("notes") or [])]
|
|
113
|
+
if extra_notes:
|
|
114
|
+
doc["notes"] = list(doc.get("notes") or []) + extra_notes
|
|
115
|
+
diag = {k: list(getattr(ctx, k)) for k in ("warnings", "fallbacks", "skipped") if getattr(ctx, k)}
|
|
116
|
+
if diag:
|
|
117
|
+
merged = dict(doc.get("diagnostics") or {})
|
|
118
|
+
for k, v in diag.items():
|
|
119
|
+
merged[k] = list(merged.get(k) or []) + [e for e in v if e not in (merged.get(k) or [])]
|
|
120
|
+
doc["diagnostics"] = merged
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
def audit(event: str, details: Dict[str, Any], *, default_path: Optional[str] = None, plan_sha256: Optional[str] = None,
|
|
124
|
+
ctx: "Optional[Context]" = None) -> Dict[str, Any]:
|
|
125
|
+
"""Append one JSON line to the audit log: {ts, tool, event, pid, plan_sha256?, details}.
|
|
126
|
+
The log is render.py's --audit-log FILE when given, else the event's own `default_path` (a
|
|
127
|
+
plan's `<plan>.audit.jsonl`, a lock's hidden sibling); neither = nothing to write. Never under
|
|
128
|
+
--dry-run (a dry run writes nothing). A write that fails is a recorded warning, never a
|
|
129
|
+
failure of the run. Returns what happened, for the caller's `details.audit_log`."""
|
|
130
|
+
import datetime
|
|
131
|
+
ctx = ctx or STATE
|
|
132
|
+
path = ctx.audit_log or default_path
|
|
133
|
+
if not path:
|
|
134
|
+
return {"path": None, "written": False, "reason": "no_audit_log"}
|
|
135
|
+
if ctx.dry_run:
|
|
136
|
+
return {"path": path, "written": False, "reason": "dry_run"}
|
|
137
|
+
rec: Dict[str, Any] = {"ts": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
|
|
138
|
+
"tool": os.path.splitext(os.path.basename(sys.argv[0] or "?"))[0], "event": event, "pid": os.getpid()}
|
|
139
|
+
if plan_sha256:
|
|
140
|
+
rec["plan_sha256"] = plan_sha256
|
|
141
|
+
rec["details"] = details
|
|
142
|
+
try:
|
|
143
|
+
with open(path, "a", encoding="utf-8") as f:
|
|
144
|
+
f.write(json.dumps(json_safe(rec), ensure_ascii=False, sort_keys=True) + "\n")
|
|
145
|
+
except OSError as exc:
|
|
146
|
+
warn(f"could not append the {event} record to the audit log {path} ({exc}); the run is unaffected",
|
|
147
|
+
"audit_log_unwritable", ctx=ctx, path=path, event=event, error=str(exc))
|
|
148
|
+
return {"path": path, "written": False, "reason": "unwritable", "error": str(exc)}
|
|
149
|
+
return {"path": path, "written": True}
|
|
150
|
+
|
|
151
|
+
|
|
56
152
|
def info(msg: str, ctx: "Optional[Context]" = None) -> None:
|
|
57
153
|
# under --dry-run nothing is written; do not let scripts claim otherwise
|
|
58
154
|
ctx = ctx or STATE
|
|
@@ -195,6 +291,24 @@ def absorb_stage_hw(stage: str, doc: Any, ctx: "Optional[Context]" = None) -> No
|
|
|
195
291
|
"used": hw.get("used"), "fallback": bool(hw.get("fallback")), "quality_note": quality})
|
|
196
292
|
|
|
197
293
|
|
|
294
|
+
def absorb_stage(stage: str, doc: Any, ctx: "Optional[Context]" = None) -> Dict[str, Any]:
|
|
295
|
+
"""A child tool's --json document into this result: its `hw` facts (absorb_stage_hw) and its
|
|
296
|
+
`diagnostics` records, each tagged with the stage it came from. Returns the fields a re-raise
|
|
297
|
+
of the stage's failure carries in its own `error`: the child's `reason` and `details`."""
|
|
298
|
+
ctx = ctx or STATE
|
|
299
|
+
absorb_stage_hw(stage, doc, ctx)
|
|
300
|
+
if not isinstance(doc, dict):
|
|
301
|
+
return {}
|
|
302
|
+
for bucket in ("warnings", "fallbacks", "skipped"):
|
|
303
|
+
for entry in (doc.get("diagnostics") or {}).get(bucket) or []:
|
|
304
|
+
if isinstance(entry, dict):
|
|
305
|
+
tagged = dict(entry, stage=entry.get("stage") or stage)
|
|
306
|
+
if tagged not in getattr(ctx, bucket):
|
|
307
|
+
getattr(ctx, bucket).append(tagged)
|
|
308
|
+
err = doc.get("error") if isinstance(doc.get("error"), dict) else {}
|
|
309
|
+
return {k: err[k] for k in ("reason", "details") if err.get(k) is not None}
|
|
310
|
+
|
|
311
|
+
|
|
198
312
|
def steps_encoder_report(steps: Sequence["Tuple[str, Any]"]) -> Dict[str, Any]:
|
|
199
313
|
"""batch.py's per-item `encoder`, `hw` and `notes`, from the --json documents of the steps
|
|
200
314
|
(child processes) that made the item: the last encoder a step ran (a later stream copy keeps
|
|
@@ -246,6 +360,7 @@ def emit(output: Optional[str], *, ctx: "Optional[Context]" = None, **extra: Any
|
|
|
246
360
|
hw_notes = [n for n in _hw_notes_for(ctx) if n not in (doc.get("notes") or [])]
|
|
247
361
|
if hw_notes:
|
|
248
362
|
doc["notes"] = list(doc.get("notes") or []) + hw_notes
|
|
363
|
+
_merge_collector(doc, ctx)
|
|
249
364
|
if ctx.plan:
|
|
250
365
|
doc["plan"] = write_plan(ctx.plan, output, extra, ctx=ctx)
|
|
251
366
|
print_json(_brief(doc, meta) if ctx.json_brief else doc)
|
|
@@ -345,6 +460,41 @@ def _plan_inputs(commands: Sequence[str], argv: Sequence[str] = (), ctx: "Option
|
|
|
345
460
|
return seen
|
|
346
461
|
|
|
347
462
|
|
|
463
|
+
def _plan_unbound_inputs(commands: Sequence[str], ctx: "Optional[Context]" = None) -> List[str]:
|
|
464
|
+
"""The inputs _plan_inputs() could not fingerprint because they did not exist when the plan
|
|
465
|
+
was made: a planned command's `-i` file (or a side input registered through
|
|
466
|
+
escape_filter_path) that is not on disk and is not written by an earlier planned command (the
|
|
467
|
+
tool's own intermediates are bound by the command that makes them). render.py reports them on
|
|
468
|
+
execution: nothing checks that their content is what the plan was made for."""
|
|
469
|
+
import shlex
|
|
470
|
+
produced: List[str] = []
|
|
471
|
+
out: List[str] = []
|
|
472
|
+
|
|
473
|
+
def consider(p: str) -> None:
|
|
474
|
+
if (not p or p == "-" or p.startswith("pipe:") or os.path.exists(p)
|
|
475
|
+
or os.path.abspath(p) in produced or os.path.abspath(p) in out):
|
|
476
|
+
return
|
|
477
|
+
out.append(os.path.abspath(p))
|
|
478
|
+
for p in (ctx or STATE).plan_inputs:
|
|
479
|
+
consider(p)
|
|
480
|
+
for line in commands:
|
|
481
|
+
try:
|
|
482
|
+
toks = shlex.split(line.split("] ", 1)[1] if line.startswith("[dry-run] ") else line)
|
|
483
|
+
except ValueError:
|
|
484
|
+
continue
|
|
485
|
+
fmt = None # an input's -f (lavfi's "inputs" are filter graphs, not files)
|
|
486
|
+
for i, tok in enumerate(toks[:-1]):
|
|
487
|
+
if tok == "-f":
|
|
488
|
+
fmt = toks[i + 1]
|
|
489
|
+
elif tok == "-i":
|
|
490
|
+
if fmt != "lavfi":
|
|
491
|
+
consider(toks[i + 1])
|
|
492
|
+
fmt = None
|
|
493
|
+
if toks:
|
|
494
|
+
produced.append(os.path.abspath(toks[-1]))
|
|
495
|
+
return out
|
|
496
|
+
|
|
497
|
+
|
|
348
498
|
def write_plan(path: str, output: Optional[str], extra: Dict[str, Any], ctx: "Optional[Context]" = None) -> str:
|
|
349
499
|
"""The dry run as an artifact: what will run, on which exact inputs, producing what, checked
|
|
350
500
|
how. `render.py PLAN` executes it after re-fingerprinting the inputs (issue #189 C).
|
|
@@ -382,6 +532,8 @@ def write_plan(path: str, output: Optional[str], extra: Dict[str, Any], ctx: "Op
|
|
|
382
532
|
"argv": cleaned,
|
|
383
533
|
"cwd": os.getcwd(),
|
|
384
534
|
"inputs": [fingerprint(p) for p in _plan_inputs(ctx.commands, cleaned, ctx)],
|
|
535
|
+
# inputs that did not exist yet, so carry no fingerprint (render.py reports them on execution)
|
|
536
|
+
"unbound_inputs": _plan_unbound_inputs(ctx.commands, ctx),
|
|
385
537
|
"commands": list(ctx.commands),
|
|
386
538
|
"output": os.path.abspath(output) if output else None,
|
|
387
539
|
"verify": verify,
|
|
@@ -217,7 +217,8 @@ class Context:
|
|
|
217
217
|
"""
|
|
218
218
|
|
|
219
219
|
__slots__ = ("dry_run", "json", "json_brief", "progress", "fast", "duration_hint", "commands", "timeout", "overwrite", "written", "preexisting", "plan", "plan_written", "plan_inputs", "codec",
|
|
220
|
-
"hw", "hw_source", "hw_notes", "hw_swaps", "hw_fallback", "hw_env_ignored", "hw_stages", "hw_auto"
|
|
220
|
+
"hw", "hw_source", "hw_notes", "hw_swaps", "hw_fallback", "hw_env_ignored", "hw_stages", "hw_auto",
|
|
221
|
+
"notes", "warnings", "fallbacks", "skipped", "audit_log")
|
|
221
222
|
|
|
222
223
|
def __init__(self) -> None:
|
|
223
224
|
self.reset()
|
|
@@ -246,6 +247,15 @@ class Context:
|
|
|
246
247
|
self.hw_env_ignored = False # $FFMPEG_SKILL_HW=1 was set but this tool's delivery preset does not take it (export.py)
|
|
247
248
|
self.hw_stages: List[Dict[str, Any]] = [] # render.py: the `hw` facts each stage (a child process) reported
|
|
248
249
|
self.hw_auto: List[str] = [] # CPU encoders (libx264/libx265) this ffmpeg build lacks, whose encodes ran on VideoToolbox without --hw
|
|
250
|
+
# The collector (docs/design-decisions.md, "Nothing happens silently"): what this run
|
|
251
|
+
# decided on its own -- a warning, a fallback, a step it skipped -- recorded as it happens
|
|
252
|
+
# by warn()/record_fallback()/record_skip() (stderr too) and merged by emit()/die() into
|
|
253
|
+
# the document: `notes` (strings) and `diagnostics: {warnings, fallbacks, skipped}`.
|
|
254
|
+
self.notes: List[str] = []
|
|
255
|
+
self.warnings: List[Dict[str, Any]] = []
|
|
256
|
+
self.fallbacks: List[Dict[str, Any]] = []
|
|
257
|
+
self.skipped: List[Dict[str, Any]] = []
|
|
258
|
+
self.audit_log: Optional[str] = None # render.py --audit-log FILE: where audit() appends; None = the event's own default
|
|
249
259
|
|
|
250
260
|
|
|
251
261
|
STATE = Context()
|
|
@@ -531,12 +541,21 @@ def _check_no_overwrite_input(cmd: Sequence[str]) -> None:
|
|
|
531
541
|
if a == "-i" and i + 1 < len(cmd):
|
|
532
542
|
inp = cmd[i + 1]
|
|
533
543
|
try:
|
|
534
|
-
|
|
535
|
-
die(f"refusing to run: output {output!r} is the same file as input {inp!r} "
|
|
536
|
-
f"(would overwrite it while ffmpeg is still reading it) -- choose a different --output/-o path",
|
|
537
|
-
kind="input")
|
|
544
|
+
same = os.path.realpath(inp) == out_real
|
|
538
545
|
except OSError:
|
|
539
546
|
continue
|
|
547
|
+
if same:
|
|
548
|
+
_refuse_output_is_input(output, inp, out_real, "would overwrite it while ffmpeg is still reading it")
|
|
549
|
+
|
|
550
|
+
|
|
551
|
+
def _refuse_output_is_input(output: str, inp: str, real: str, why: str) -> None:
|
|
552
|
+
"""The one refusal both output-is-input guards end in: structured, and in the audit log when
|
|
553
|
+
one is configured (render.py --audit-log)."""
|
|
554
|
+
details = {"output": output, "input": inp, "resolved": real}
|
|
555
|
+
audit("output_is_input_refused", details)
|
|
556
|
+
refuse("input", "output_is_input",
|
|
557
|
+
f"refusing to run: output {output!r} is the same file as input {inp!r} ({why}) -- choose a different --output/-o path",
|
|
558
|
+
**details)
|
|
540
559
|
|
|
541
560
|
|
|
542
561
|
def refuse_output_is_input(output: str, *inputs: str) -> None:
|
|
@@ -555,8 +574,7 @@ def refuse_output_is_input(output: str, *inputs: str) -> None:
|
|
|
555
574
|
except OSError:
|
|
556
575
|
continue
|
|
557
576
|
if same:
|
|
558
|
-
|
|
559
|
-
f"(the result would replace the source) -- choose a different --output/-o path", kind="input")
|
|
577
|
+
_refuse_output_is_input(output, inp, out_real, "the result would replace the source")
|
|
560
578
|
|
|
561
579
|
|
|
562
580
|
def _check_output_path(cmd: Sequence[str]) -> None:
|
|
@@ -567,12 +585,15 @@ def _check_output_path(cmd: Sequence[str]) -> None:
|
|
|
567
585
|
if output == "-" or output.startswith("pipe:") or output.startswith("-"):
|
|
568
586
|
return
|
|
569
587
|
if os.path.isdir(output):
|
|
570
|
-
|
|
588
|
+
refuse("input", "output_is_directory",
|
|
589
|
+
f"output {output!r} is a directory; pass a file path (e.g. {os.path.join(output, 'result.mp4')!r})", output=output)
|
|
571
590
|
parent = os.path.dirname(os.path.abspath(output))
|
|
572
591
|
if not os.path.isdir(parent):
|
|
573
|
-
|
|
592
|
+
refuse("input", "output_dir_missing",
|
|
593
|
+
f"output directory {parent!r} does not exist; create it first (this tool never creates directories)",
|
|
594
|
+
output=output, directory=parent)
|
|
574
595
|
if not os.access(parent, os.W_OK):
|
|
575
|
-
|
|
596
|
+
refuse("input", "output_dir_not_writable", f"output directory {parent!r} is not writable", output=output, directory=parent)
|
|
576
597
|
|
|
577
598
|
|
|
578
599
|
EVEN_SCALE = "scale=trunc(iw/2)*2:trunc(ih/2)*2"
|
|
@@ -602,14 +623,29 @@ def _pid_dead(pid: int) -> bool:
|
|
|
602
623
|
return False
|
|
603
624
|
|
|
604
625
|
|
|
626
|
+
LOCK_HOUR = 3600.0
|
|
627
|
+
LOCK_UNLIMITED = 10 * 365 * 86400.0 # --timeout 0: the deadline is pushed past any run (and kept short of 2038)
|
|
628
|
+
|
|
629
|
+
|
|
605
630
|
class _OutputLock:
|
|
606
631
|
"""Two runs writing the same output at once used to both report `completed` while one of
|
|
607
632
|
them described the other's file (sweep F1). A lock file next to the output, created with
|
|
608
|
-
O_EXCL and holding the writer's pid, makes the second run refuse
|
|
609
|
-
|
|
633
|
+
O_EXCL and holding the writer's pid, makes the second run refuse (`kind: input`,
|
|
634
|
+
`reason: output_locked`, the holder in `details`).
|
|
635
|
+
|
|
636
|
+
Stale locks: a lock whose pid is dead is taken over. A lock whose pid is alive is taken over
|
|
637
|
+
only once its holder's own time limit has passed: the writer sets the lock's mtime to that
|
|
638
|
+
deadline minus an hour, so "older than an hour" -- the rule every version since 1.x applies,
|
|
639
|
+
which this keeps compatible in both directions -- means "past the deadline". The deadline is
|
|
640
|
+
child_limit() from now (4x --timeout + 60 s, the ceiling on everything one run() does under
|
|
641
|
+
the lock); with --timeout 0 there is none and a live holder is never taken over. Through 2.10
|
|
642
|
+
the hour ran from the lock's creation, so a legitimate run longer than an hour lost its lock
|
|
643
|
+
to a second writer. A takeover is recorded (`diagnostics.fallbacks`, stderr, and the audit
|
|
644
|
+
log); so is running without a lock where none can be created (`diagnostics.warnings`)."""
|
|
610
645
|
def __init__(self, output: str) -> None:
|
|
611
646
|
self.path: Optional[str] = None
|
|
612
647
|
self.fd: Optional[int] = None
|
|
648
|
+
self.output = output
|
|
613
649
|
if output == "-" or output.startswith("pipe:") or output.startswith("-"):
|
|
614
650
|
return
|
|
615
651
|
d, base = os.path.split(os.path.abspath(output))
|
|
@@ -618,33 +654,95 @@ class _OutputLock:
|
|
|
618
654
|
def __enter__(self) -> "_OutputLock":
|
|
619
655
|
if not self.path:
|
|
620
656
|
return self
|
|
657
|
+
taken: Optional[Dict[str, Any]] = None
|
|
621
658
|
for attempt in (0, 1):
|
|
622
659
|
try:
|
|
623
660
|
self.fd = os.open(self.path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
|
624
661
|
os.write(self.fd, str(os.getpid()).encode())
|
|
662
|
+
self._set_deadline()
|
|
663
|
+
if taken is not None:
|
|
664
|
+
self._record_takeover(taken)
|
|
625
665
|
return self
|
|
626
666
|
except FileExistsError:
|
|
627
|
-
|
|
667
|
+
holder = self._holder()
|
|
668
|
+
if attempt == 0 and holder["stale"]:
|
|
628
669
|
try:
|
|
629
670
|
os.remove(self.path)
|
|
630
|
-
except OSError:
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
|
|
671
|
+
except OSError as exc:
|
|
672
|
+
# Windows will not delete a file another process holds open, and a live
|
|
673
|
+
# holder keeps its lock open until it finishes: the takeover fails and
|
|
674
|
+
# the run is refused below, with the reason, instead of passing silently
|
|
675
|
+
holder["takeover_error"] = str(exc)
|
|
676
|
+
holder["stale"] = False
|
|
677
|
+
else:
|
|
678
|
+
taken = holder
|
|
679
|
+
continue
|
|
680
|
+
name = os.path.basename(self.path)[1:-len(".ffskill-lock")]
|
|
681
|
+
refuse("input", "output_locked",
|
|
682
|
+
f"another run is writing {name!r} right now (lock {self.path}"
|
|
683
|
+
+ (f", pid {holder['pid']}" if holder.get("pid") else "") + "); wait for it or choose a different --output/-o path",
|
|
684
|
+
hint=("wait for that run, or choose another -o path; if pid "
|
|
685
|
+
f"{holder.get('pid')} is not an ffmpeg-skill run, delete {self.path}"),
|
|
686
|
+
lock=self.path, output=os.path.abspath(self.output), **{k: v for k, v in holder.items() if k != "stale"})
|
|
687
|
+
except OSError as exc:
|
|
688
|
+
# an unlockable location (a read-only directory surfaces elsewhere): the run goes
|
|
689
|
+
# on without a lock, and says so -- two runs could now both write this output
|
|
690
|
+
details = {"lock": self.path, "output": os.path.abspath(self.output), "error": str(exc)}
|
|
691
|
+
warn(f"could not create the output lock {self.path} ({exc}); running without one, so a second run "
|
|
692
|
+
"writing the same output at the same time would not be refused", "output_unlocked", **details)
|
|
693
|
+
audit("output_unlocked", details, default_path=self._audit_default())
|
|
694
|
+
return self
|
|
637
695
|
return self
|
|
638
696
|
|
|
639
|
-
def
|
|
697
|
+
def _audit_default(self) -> Optional[str]:
|
|
698
|
+
if not self.path:
|
|
699
|
+
return None
|
|
700
|
+
d, base = os.path.split(os.path.abspath(self.output))
|
|
701
|
+
return os.path.join(d, f".{base}.ffskill-audit.jsonl")
|
|
702
|
+
|
|
703
|
+
def _set_deadline(self) -> None:
|
|
704
|
+
import time
|
|
705
|
+
limit = child_limit()
|
|
706
|
+
deadline = time.time() + (limit if limit else LOCK_UNLIMITED)
|
|
640
707
|
try:
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
|
|
708
|
+
os.utime(self.path, (deadline - LOCK_HOUR, deadline - LOCK_HOUR))
|
|
709
|
+
except (OSError, OverflowError, ValueError) as exc:
|
|
710
|
+
warn(f"could not date the output lock {self.path} ({exc}); another run may take it over an hour from now",
|
|
711
|
+
"lock_deadline_unset", lock=self.path, error=str(exc))
|
|
712
|
+
|
|
713
|
+
def _holder(self) -> Dict[str, Any]:
|
|
714
|
+
"""Who holds the lock and whether it is stale: {pid, pid_alive, deadline (UTC, the
|
|
715
|
+
mtime plus an hour), why: pid_dead | past_deadline | unreadable | held, stale}."""
|
|
716
|
+
import datetime
|
|
717
|
+
import time
|
|
718
|
+
out: Dict[str, Any] = {"pid": None, "pid_alive": None, "deadline": None, "why": "unreadable", "stale": True}
|
|
719
|
+
try:
|
|
720
|
+
with open(self.path, encoding="utf-8", errors="replace") as f:
|
|
721
|
+
text = f.read().strip()
|
|
722
|
+
deadline = os.path.getmtime(self.path) + LOCK_HOUR
|
|
723
|
+
pid = int(text or "0")
|
|
646
724
|
except (OSError, ValueError):
|
|
647
|
-
return
|
|
725
|
+
return out
|
|
726
|
+
out["deadline"] = datetime.datetime.fromtimestamp(deadline, datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
727
|
+
out["pid"] = pid if pid > 0 else None
|
|
728
|
+
if pid > 0 and _pid_dead(pid):
|
|
729
|
+
out.update(pid_alive=False, why="pid_dead")
|
|
730
|
+
return out
|
|
731
|
+
out["pid_alive"] = pid > 0 or None
|
|
732
|
+
if time.time() > deadline:
|
|
733
|
+
out["why"] = "past_deadline"
|
|
734
|
+
return out
|
|
735
|
+
out.update(why="held", stale=False)
|
|
736
|
+
return out
|
|
737
|
+
|
|
738
|
+
def _record_takeover(self, holder: Dict[str, Any]) -> None:
|
|
739
|
+
details = {"lock": self.path, "output": os.path.abspath(self.output),
|
|
740
|
+
**{k: v for k, v in holder.items() if k != "stale"}}
|
|
741
|
+
why = {"pid_dead": f"its pid {holder.get('pid')} no longer runs",
|
|
742
|
+
"past_deadline": f"its holder (pid {holder.get('pid')}, still running) is past its own time limit",
|
|
743
|
+
}.get(holder.get("why") or "", "it could not be read")
|
|
744
|
+
details["audit_log"] = audit("lock_takeover", details, default_path=self._audit_default())
|
|
745
|
+
record_fallback(f"took over the stale output lock {self.path}: {why}", "stale_lock_taken_over", **details)
|
|
648
746
|
|
|
649
747
|
def __exit__(self, *exc: Any) -> None:
|
|
650
748
|
if self.fd is not None:
|
|
@@ -698,6 +796,7 @@ def refuse_existing_outputs(paths: Sequence[str]) -> None:
|
|
|
698
796
|
.srt/.ass sidecars) calls this up front so a hand-edited sidecar is refused like the video
|
|
699
797
|
is, and a dry run predicts the same refusal."""
|
|
700
798
|
existing: List[str] = []
|
|
799
|
+
facts: List[Dict[str, Any]] = []
|
|
701
800
|
for output in paths:
|
|
702
801
|
if not output:
|
|
703
802
|
continue
|
|
@@ -708,21 +807,25 @@ def refuse_existing_outputs(paths: Sequence[str]) -> None:
|
|
|
708
807
|
continue
|
|
709
808
|
if not exists or real in STATE.written:
|
|
710
809
|
continue
|
|
810
|
+
size = mtime_ns = None
|
|
711
811
|
try:
|
|
712
812
|
st = os.stat(output)
|
|
713
813
|
STATE.preexisting[real] = (st.st_size, st.st_mtime_ns)
|
|
814
|
+
size, mtime_ns = st.st_size, st.st_mtime_ns
|
|
714
815
|
except OSError:
|
|
715
816
|
pass
|
|
716
817
|
if output not in existing:
|
|
717
818
|
existing.append(output)
|
|
819
|
+
facts.append({"path": output, "resolved": real, "size": size, "mtime_ns": mtime_ns})
|
|
718
820
|
if not existing or STATE.overwrite:
|
|
719
821
|
return
|
|
822
|
+
hint = "pass --overwrite, or choose another -o path"
|
|
720
823
|
if len(existing) == 1:
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
|
|
824
|
+
refuse("input", "output_exists",
|
|
825
|
+
f"refusing to overwrite existing output {existing[0]!r}: pass --overwrite to replace it, or choose another -o path",
|
|
826
|
+
hint=hint, outputs=facts)
|
|
827
|
+
refuse("input", "output_exists", "refusing to overwrite existing outputs " + ", ".join(repr(p) for p in existing)
|
|
828
|
+
+ ": pass --overwrite to replace them, or choose another -o path", hint=hint, outputs=facts)
|
|
726
829
|
|
|
727
830
|
|
|
728
831
|
def _remember_output(cmd: Sequence[str]) -> None:
|
|
@@ -1415,4 +1518,4 @@ def read_text_or_die(path: str, flag: str) -> str:
|
|
|
1415
1518
|
# emit needs runner's Context/STATE/ERROR_CODE, so the two form a cycle that has to be cut
|
|
1416
1519
|
# somewhere: by the time this line runs every name emit reads from runner above is defined, and
|
|
1417
1520
|
# every name runner reads from emit is only ever read inside a function body, never at import.
|
|
1418
|
-
from _common.emit import _plan_at_exit, die, hw_report, info # noqa: E402
|
|
1521
|
+
from _common.emit import _plan_at_exit, audit, die, hw_report, info, record_fallback, refuse, warn # noqa: E402
|
package/scripts/_contract.py
CHANGED
|
@@ -397,6 +397,37 @@ def input_schema(parser: argparse.ArgumentParser) -> Dict[str, Any]:
|
|
|
397
397
|
return schema
|
|
398
398
|
|
|
399
399
|
|
|
400
|
+
# The collector's records (docs/design-decisions.md, "Nothing happens silently"): on success and
|
|
401
|
+
# failure documents alike, present only when one list is not empty.
|
|
402
|
+
DIAGNOSTICS_SCHEMA = {
|
|
403
|
+
"type": "object",
|
|
404
|
+
"description": "what the run decided on its own, recorded as it happened (each also a stderr line): warnings (something went "
|
|
405
|
+
"other than asked), fallbacks (another route was taken and the run went on), skipped (a step that would "
|
|
406
|
+
"normally run did not). Each entry is {code, message, ...facts}; an entry a render.py stage reported carries "
|
|
407
|
+
"`stage`. Absent when all three are empty",
|
|
408
|
+
"properties": {k: {"type": "array", "items": {"type": "object", "properties": {"code": {"type": "string"}, "message": {"type": "string"}}}}
|
|
409
|
+
for k in ("warnings", "fallbacks", "skipped")},
|
|
410
|
+
}
|
|
411
|
+
|
|
412
|
+
# error.reason: a stable code for why a refusal happened, beside `kind` (which says what class of
|
|
413
|
+
# failure it is). Every refuse() call in scripts/ names one of these (a test reads the sources).
|
|
414
|
+
ERROR_REASONS = {
|
|
415
|
+
"output_exists": "an output already exists and --overwrite was not given; details.outputs: [{path, resolved, size, mtime_ns}]",
|
|
416
|
+
"output_is_input": "the output resolves to the same file as an input; details: {output, input, resolved}",
|
|
417
|
+
"output_is_directory": "the output path is a directory; details: {output}",
|
|
418
|
+
"output_dir_missing": "the output's directory does not exist; details: {output, directory}",
|
|
419
|
+
"output_dir_not_writable": "the output's directory is not writable; details: {output, directory}",
|
|
420
|
+
"output_locked": "another run holds the output's lock; details: {lock, output, pid, pid_alive, deadline, why}",
|
|
421
|
+
"plan_inputs_changed": "render.py PLAN: a fingerprinted input is missing or changed; details: {plan, plan_sha256, checked_at, "
|
|
422
|
+
"inputs: [{path, status: unchanged|changed|missing, expected: {size, sha256_head_tail}, actual: {size, "
|
|
423
|
+
"sha256_head_tail} | \"missing\"}], unbound_inputs, audit_log}",
|
|
424
|
+
"plan_version_unsupported": "render.py PLAN: plan_version is not one this version executes; details: {plan, plan_sha256, checked_at, plan_version, supported, audit_log}",
|
|
425
|
+
"plan_unknown_tool": "render.py PLAN: the plan names no tool render.py can run; details: {plan, plan_sha256, checked_at, tool, audit_log}",
|
|
426
|
+
"plan_cwd_missing": "render.py PLAN: the directory the plan was made in is gone; details: {plan, plan_sha256, checked_at, cwd, audit_log}",
|
|
427
|
+
"plan_argv_invalid": "render.py PLAN: argv is not a list; details: {plan, plan_sha256, checked_at, argv_type, audit_log}",
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
|
|
400
431
|
def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str, Any]:
|
|
401
432
|
"""What the tool prints on stdout with --json (keys observed in the implementation). `hw`: the
|
|
402
433
|
tool takes --hw, so its `notes` can also carry a VideoToolbox encode's quality line."""
|
|
@@ -411,7 +442,8 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
|
|
|
411
442
|
"hw": {"type": "object", "description": "present when VideoToolbox was asked for (--hw, $FFMPEG_SKILL_HW, a render.py stage, a batch.py item), on export.py when $FFMPEG_SKILL_HW=1 is set but its delivery preset does not take it, and when this ffmpeg build has no libx264/libx265 and an encode ran on VideoToolbox instead: {requested: whether this run's encodes were asked onto VideoToolbox (false for that export, and for the build without libx264/libx265), source: flag|env|auto (auto: nobody asked, the build has no libx264/libx265 and this Apple Silicon machine ran VideoToolbox in its place), used: true|false whether the encoder that ran last is VideoToolbox, null under --dry-run (nothing ran) and when this process ran no encode itself (batch.py: its stages are child processes; its top-level hw gathers the items' requested/source/fallback/notes and each results row carries its item's encoder and hw), fallback: a job VideoToolbox refused (encoder open or compression session failure; nothing else is retried) was re-encoded on the CPU, notes: why an encode stayed on or fell back to the CPU, and, when $FFMPEG_SKILL_HW chose the GPU and it ran, what that costs and that --no-hw opts out}. A VideoToolbox encode also adds a top-level notes line: its quality is not CRF-equivalent. A kind: ffmpeg failure under --hw / $FFMPEG_SKILL_HW carries encoder and hw too"},
|
|
412
443
|
"plan": {"type": "string", "description": "with --plan FILE: the plan document written (the run itself is a dry run)"},
|
|
413
444
|
"verified": {"type": "boolean", "description": "true only when the artifact was written, probed, and every self-check the tool ran (verification) met its target; false under --dry-run"},
|
|
414
|
-
"verification": {"type": "array", "items": {"type": "object", "properties": {"step": {"type": "string"}, "ok": {"type": "boolean"}}}, "description": "what the tool itself verified: probe, plus loudness (loudness.py, export platform presets) or check (render)"}
|
|
445
|
+
"verification": {"type": "array", "items": {"type": "object", "properties": {"step": {"type": "string"}, "ok": {"type": "boolean"}}}, "description": "what the tool itself verified: probe, plus loudness (loudness.py, export platform presets) or check (render)"},
|
|
446
|
+
"diagnostics": DIAGNOSTICS_SCHEMA}
|
|
415
447
|
extra: Dict[str, Any] = {}
|
|
416
448
|
if name == "check":
|
|
417
449
|
extra = {"platform": {"type": "string"}, "ok": {"type": "boolean"}, "failed": {"type": "integer"}, "warnings": {"type": "integer"},
|
|
@@ -451,6 +483,8 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
|
|
|
451
483
|
elif name == "render":
|
|
452
484
|
extra = {"stages": {"type": "array", "items": {"type": "string"}}, "check": {"type": ["object", "null"]},
|
|
453
485
|
"plan": {"type": "string", "description": "when the argument was a plan.json (written by <tool> --plan): its path"},
|
|
486
|
+
"plan_sha256": {"type": ["string", "null"], "description": "plan execution: sha256 of the plan file as executed"},
|
|
487
|
+
"unbound_inputs": {"type": "array", "items": {"type": "string"}, "description": "plan execution: inputs that did not exist when the plan was made, so carry no fingerprint and were not checked (also a diagnostics.warnings entry, code plan_unbound_inputs)"},
|
|
454
488
|
"tool": {"type": "string", "description": "plan execution: the tool the plan ran"},
|
|
455
489
|
"tool_result": {"type": "object", "description": "plan execution: the tool's own --json document"},
|
|
456
490
|
"timeline": {"type": "object", "description": "--export-timeline: format, rate, duration, frames, clips, transition, "
|
|
@@ -1692,7 +1726,21 @@ def build(detect: bool = True) -> Dict[str, Any]:
|
|
|
1692
1726
|
},
|
|
1693
1727
|
"json_output": {
|
|
1694
1728
|
"success": {"status": "completed", "exit_code": 0, "stdout": "one JSON document (output_schema)"},
|
|
1695
|
-
"failure": {"status": "failed", "exit_code": "non-zero (127 when ffmpeg/ffprobe is missing)", "stdout": "{\"status\": \"failed\", \"exit_code\": N, \"error\": {\"kind\": ..., \"message\":
|
|
1729
|
+
"failure": {"status": "failed", "exit_code": "non-zero (127 when ffmpeg/ffprobe is missing)", "stdout": "{\"status\": \"failed\", \"exit_code\": N, \"error\": {\"kind\": ..., \"message\": ..., \"code\": ..., \"retryable\": false}, \"commands\": [...]} when --json was given; a refusal adds error.reason (error_reasons) and error.details (the facts the message is written from), and a run that recorded warnings, fallbacks or skipped steps before failing carries diagnostics", "stderr": "human-readable message",
|
|
1730
|
+
"error": {"kind": "error_kinds", "message": "the reason in prose", "code": "a static relabelling of kind", "retryable": "always false",
|
|
1731
|
+
"hint": "optional: the flag change that would make a retry meaningful",
|
|
1732
|
+
"reason": "optional, on refusals: a stable code from error_reasons",
|
|
1733
|
+
"details": "optional, with reason: an object of the facts behind the refusal (paths, expected and actual values)"},
|
|
1734
|
+
"diagnostics": DIAGNOSTICS_SCHEMA["description"]},
|
|
1735
|
+
"error_reasons": ERROR_REASONS,
|
|
1736
|
+
"audit_log": {"format": "JSON lines, appended: {ts, tool, event, pid, plan_sha256 (plan events), details}",
|
|
1737
|
+
"events": {"plan_rejected": "render.py refused a plan (details: the refusal's error.details plus reason)",
|
|
1738
|
+
"lock_takeover": "a stale output lock was taken over (details: lock, output, pid, pid_alive, deadline, why)",
|
|
1739
|
+
"output_unlocked": "no output lock could be created and the run went on without one",
|
|
1740
|
+
"output_is_input_refused": "the output-is-input refusal (only when an audit log was named)"},
|
|
1741
|
+
"location": "render.py --audit-log FILE when given; otherwise a refused plan's <plan>.audit.jsonl, and a lock event's "
|
|
1742
|
+
"hidden .<output>.ffskill-audit.jsonl beside the output. Never written under --dry-run; a write that "
|
|
1743
|
+
"fails is a diagnostics.warnings entry (audit_log_unwritable), never a failure"},
|
|
1696
1744
|
"error_kinds": {"input": "missing or unsuitable input, bad arguments", "ffmpeg": "ffmpeg/ffprobe returned an error (message carries the last stderr lines)", "output": "ffmpeg exited 0 but the artifact is missing, empty or unreadable (an empty file is removed)", "missing_tool": "ffmpeg or ffprobe not on PATH", "timeout": "one ffmpeg/ffprobe run exceeded --timeout (default 1800 s, FFMPEG_SKILL_TIMEOUT) and was killed; partial output removed; exit 124", "verification": "the tool ran but its result failed the requested check: check.py platform rows (checks attached), render.py's check stage (output written, check attached), batch.py items (results attached), verify.py steps (files attached), color.py --retag (output written, but ffprobe does not read the requested colour tags on it); exit 1", "interrupted": "the tool received SIGINT or SIGTERM: the running ffmpeg (or sibling script) was stopped and its partial output removed; exit 130 or 143"},
|
|
1697
1745
|
"success_criterion": "exit 0 AND the output exists AND is non-empty AND ffprobe reads a stream from it; only then is status completed printed and the output probe attached",
|
|
1698
1746
|
},
|
package/scripts/render.py
CHANGED
|
@@ -72,7 +72,7 @@ from export import PRESETS, PLATFORM_OF
|
|
|
72
72
|
from _platforms import PLATFORMS, caption_defaults, resolve as resolve_platform
|
|
73
73
|
from _common import STATE, add_common, aspect_ratio, brand_caption_style, load_brand, apply_common, child_args, die, emit, info, probe, require_tool, run, run_tool, place_output, refuse_output_is_input, _check_existing_output, _check_output_path, fingerprint, PLAN_VERSION, ffmpeg_version, ffmpeg_exe
|
|
74
74
|
from _common.runner import HW_ENV, HW_FORCED_ENV, add_hw_orchestrator_args
|
|
75
|
-
from _common.emit import
|
|
75
|
+
from _common.emit import absorb_stage, audit, refuse, warn
|
|
76
76
|
import subprocess
|
|
77
77
|
from _contract import CONTRACT_VERSION
|
|
78
78
|
from batch import file_key
|
|
@@ -364,7 +364,7 @@ def render_pack(names: List[str], args) -> int:
|
|
|
364
364
|
if line.startswith("$ ") or line.startswith("[dry-run]"):
|
|
365
365
|
STATE.commands.append(line[2:] if line.startswith("$ ") else line)
|
|
366
366
|
STATE.commands.extend(str(c) for c in (doc.get("commands") or []))
|
|
367
|
-
|
|
367
|
+
absorb_stage(name, doc)
|
|
368
368
|
out = doc.get("output") or dest_out
|
|
369
369
|
chk = doc.get("check") or {}
|
|
370
370
|
ok = proc.returncode == 0 and doc.get("status") == "completed"
|
|
@@ -580,26 +580,26 @@ def sh(script: str, *argv: Any, extra: List[str] = None, stage: str = None) -> s
|
|
|
580
580
|
# needs (a timeout inside audio.py is a timeout, not an "input" error of render.py).
|
|
581
581
|
err = doc.get("error") or {}
|
|
582
582
|
extra_fields = {"hint": err["hint"]} if err.get("hint") else {}
|
|
583
|
-
|
|
583
|
+
error_extra = absorb_stage(Path(script).stem, doc) # the stage's reason/details and records ride along
|
|
584
|
+
extra_fields.update(_stage_failure_hw(doc))
|
|
584
585
|
die(f"{script} failed: {err.get('message') or (proc.stderr.strip().splitlines() or ['?'])[-1][:300]}",
|
|
585
|
-
code=int(doc.get("exit_code") or 1), kind=err.get("kind") or "input", stage=script, **extra_fields)
|
|
586
|
+
code=int(doc.get("exit_code") or 1), kind=err.get("kind") or "input", stage=script, error_extra=error_extra, **extra_fields)
|
|
586
587
|
_LAST_DOC.clear()
|
|
587
588
|
_LAST_DOC.update(doc if isinstance(doc, dict) else {})
|
|
588
|
-
|
|
589
|
+
absorb_stage(Path(script).stem, doc) # a stage's GPU facts (ran on VideoToolbox, fell back and why) and its diagnostics reach this result
|
|
589
590
|
out_path = str(doc.get("output") or "")
|
|
590
591
|
if key:
|
|
591
592
|
cache_store(stage, key, out_path or (dest or ""), time.time() - started)
|
|
592
593
|
return out_path
|
|
593
594
|
|
|
594
595
|
|
|
595
|
-
def _stage_failure_hw(
|
|
596
|
-
"""A failed stage's
|
|
597
|
-
|
|
598
|
-
|
|
596
|
+
def _stage_failure_hw(doc: Any) -> Dict[str, Any]:
|
|
597
|
+
"""A failed stage's `encoder` -- the command that failed -- for render's own failure document
|
|
598
|
+
(absorb_stage() has already joined the stage's `hw` to the earlier stages'; a fallback before
|
|
599
|
+
the failure is not lost). die() then adds `hw` for a kind: ffmpeg failure, as a tool's own
|
|
599
600
|
run() failure carries it."""
|
|
600
601
|
if not isinstance(doc, dict):
|
|
601
602
|
return {}
|
|
602
|
-
absorb_stage_hw(stage, doc)
|
|
603
603
|
return {"encoder": doc["encoder"]} if doc.get("encoder") else {}
|
|
604
604
|
|
|
605
605
|
|
|
@@ -731,40 +731,94 @@ def cache_store(stage: str, key: str, produced: str, seconds: float) -> None:
|
|
|
731
731
|
info(f"cache: could not store the {stage} artifact ({exc}); the run is unaffected")
|
|
732
732
|
|
|
733
733
|
|
|
734
|
+
def _utc_now() -> str:
|
|
735
|
+
import datetime
|
|
736
|
+
return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
737
|
+
|
|
738
|
+
|
|
739
|
+
def _plan_sha256(path: str) -> Optional[str]:
|
|
740
|
+
try:
|
|
741
|
+
return hashlib.sha256(Path(path).read_bytes()).hexdigest()
|
|
742
|
+
except OSError:
|
|
743
|
+
return None
|
|
744
|
+
|
|
745
|
+
|
|
746
|
+
def _reject_plan(path: str, plan_sha256: Optional[str], reason: str, msg: str, *, hint: Optional[str] = None,
|
|
747
|
+
**details: Any) -> None:
|
|
748
|
+
"""Refuse a plan: one audit record (the plan's `<plan>.audit.jsonl`, or --audit-log), then
|
|
749
|
+
the failure document with `error.reason` and `error.details` -- the plan, its sha256, when it
|
|
750
|
+
was checked, the facts below, and where the audit record went."""
|
|
751
|
+
details = dict({"plan": path, "plan_sha256": plan_sha256, "checked_at": _utc_now()}, **details)
|
|
752
|
+
details["audit_log"] = audit("plan_rejected", dict(details, reason=reason), default_path=path + ".audit.jsonl",
|
|
753
|
+
plan_sha256=plan_sha256)
|
|
754
|
+
refuse("input", reason, msg, hint=hint, extra={"plan": path}, **details)
|
|
755
|
+
|
|
756
|
+
|
|
757
|
+
def plan_input_status(inputs: Sequence[Any]) -> List[Dict[str, Any]]:
|
|
758
|
+
"""Each fingerprinted plan input against the file as it is now: {path, status: unchanged |
|
|
759
|
+
changed | missing, expected: {size, sha256_head_tail}, actual: {size, sha256_head_tail} or
|
|
760
|
+
"missing"}."""
|
|
761
|
+
rows: List[Dict[str, Any]] = []
|
|
762
|
+
for inp in inputs:
|
|
763
|
+
inp = inp if isinstance(inp, dict) else {"path": inp}
|
|
764
|
+
p = inp.get("path")
|
|
765
|
+
expected = {"size": inp.get("size"), "sha256_head_tail": inp.get("sha256_head_tail")}
|
|
766
|
+
if not p or not os.path.isfile(str(p)):
|
|
767
|
+
rows.append({"path": p, "status": "missing", "expected": expected, "actual": "missing"})
|
|
768
|
+
continue
|
|
769
|
+
try:
|
|
770
|
+
now = fingerprint(str(p))
|
|
771
|
+
except OSError:
|
|
772
|
+
rows.append({"path": p, "status": "missing", "expected": expected, "actual": "missing"})
|
|
773
|
+
continue
|
|
774
|
+
actual = {"size": now["size"], "sha256_head_tail": now["sha256_head_tail"]}
|
|
775
|
+
rows.append({"path": p, "status": "unchanged" if actual == expected else "changed", "expected": expected, "actual": actual})
|
|
776
|
+
return rows
|
|
777
|
+
|
|
778
|
+
|
|
734
779
|
def execute_plan(plan: Dict[str, Any], path: str) -> int:
|
|
735
780
|
"""Run a plan written by `<tool> --plan FILE`: refuse if any fingerprinted input changed since
|
|
736
781
|
the plan was made (the plan's commands would then describe a different edit), run the tool
|
|
737
|
-
with the planned argv, then the plan's verify steps. --dry-run prints the planned commands.
|
|
782
|
+
with the planned argv, then the plan's verify steps. --dry-run prints the planned commands.
|
|
783
|
+
|
|
784
|
+
Every refusal is `kind: input` with `error.reason` and `error.details` and appends one audit
|
|
785
|
+
record (docs/design-decisions.md, "Nothing happens silently")."""
|
|
786
|
+
digest = _plan_sha256(path)
|
|
738
787
|
if plan.get("plan_version") != PLAN_VERSION:
|
|
739
|
-
|
|
788
|
+
_reject_plan(path, digest, "plan_version_unsupported",
|
|
789
|
+
f"{path}: plan_version {plan.get('plan_version')!r} is not {PLAN_VERSION}",
|
|
790
|
+
plan_version=plan.get("plan_version"), supported=[PLAN_VERSION])
|
|
740
791
|
tool = str(plan.get("tool") or "")
|
|
741
792
|
script = HERE / f"{tool}.py"
|
|
742
793
|
if not re.fullmatch(r"[a-z][a-z0-9_]*", tool) or not script.exists() or tool == "render":
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
750
|
-
|
|
751
|
-
|
|
752
|
-
|
|
753
|
-
|
|
754
|
-
|
|
755
|
-
|
|
794
|
+
_reject_plan(path, digest, "plan_unknown_tool", f"{path}: unknown tool {tool!r}", tool=tool)
|
|
795
|
+
unbound = [str(u) for u in (plan.get("unbound_inputs") or [])]
|
|
796
|
+
rows = plan_input_status(plan.get("inputs") or [])
|
|
797
|
+
bad = [r for r in rows if r["status"] != "unchanged"]
|
|
798
|
+
if bad:
|
|
799
|
+
lines = [f"{r['path']}: missing" if r["status"] == "missing" else f"{r['path']}: content changed since the plan was made"
|
|
800
|
+
for r in bad]
|
|
801
|
+
_reject_plan(path, digest, "plan_inputs_changed",
|
|
802
|
+
"plan inputs differ from what was planned; re-run the tool with --plan to make a new plan:\n " + "\n ".join(lines),
|
|
803
|
+
hint="plans are bound to the exact input files they were made from",
|
|
804
|
+
inputs=rows, unbound_inputs=unbound)
|
|
805
|
+
if unbound:
|
|
806
|
+
warn(f"{len(unbound)} plan input(s) did not exist when the plan was made, so nothing checks that they are the files "
|
|
807
|
+
"it was made for: " + ", ".join(unbound), "plan_unbound_inputs", paths=unbound)
|
|
756
808
|
if plan.get("cwd"):
|
|
757
809
|
if not os.path.isdir(plan["cwd"]):
|
|
758
|
-
|
|
810
|
+
_reject_plan(path, digest, "plan_cwd_missing",
|
|
811
|
+
f"plan cwd {plan['cwd']} no longer exists; relative paths in the plan would resolve elsewhere -- re-plan",
|
|
812
|
+
cwd=plan["cwd"])
|
|
759
813
|
os.chdir(plan["cwd"])
|
|
760
814
|
if not isinstance(plan.get("argv"), list):
|
|
761
|
-
|
|
815
|
+
_reject_plan(path, digest, "plan_argv_invalid", f"{path}: argv must be a list", argv_type=type(plan.get("argv")).__name__)
|
|
762
816
|
argv = [str(a) for a in plan["argv"]]
|
|
763
817
|
if STATE.dry_run:
|
|
764
818
|
for c in plan.get("commands") or []:
|
|
765
819
|
STATE.commands.append(c)
|
|
766
820
|
info("[dry-run] " + c)
|
|
767
|
-
emit(plan.get("output"), plan=path, tool=tool, stages=[tool], check=None)
|
|
821
|
+
emit(plan.get("output"), plan=path, plan_sha256=digest, tool=tool, stages=[tool], check=None, unbound_inputs=unbound)
|
|
768
822
|
return 0
|
|
769
823
|
info(f"executing plan {path}: {tool} " + " ".join(argv))
|
|
770
824
|
proc = run_tool([str(script)] + argv + child_args() + ["--json"])
|
|
@@ -779,9 +833,10 @@ def execute_plan(plan: Dict[str, Any], path: str) -> int:
|
|
|
779
833
|
doc = {}
|
|
780
834
|
if proc.returncode != 0 or doc.get("status") != "completed":
|
|
781
835
|
err = doc.get("error") or {}
|
|
836
|
+
error_extra = absorb_stage(tool, doc) # the tool's reason/details and records ride along
|
|
782
837
|
die(f"{tool} failed while executing the plan: {err.get('message') or proc.stderr.strip()[-300:]}",
|
|
783
|
-
kind=err.get("kind", "ffmpeg"), plan=path, tool=tool, **_stage_failure_hw(
|
|
784
|
-
|
|
838
|
+
kind=err.get("kind", "ffmpeg"), plan=path, tool=tool, error_extra=error_extra, **_stage_failure_hw(doc))
|
|
839
|
+
absorb_stage(tool, doc) # the tool's GPU facts (fell back and why, ran on VideoToolbox) and diagnostics reach this result
|
|
785
840
|
output = doc.get("output") or plan.get("output")
|
|
786
841
|
check_result = None
|
|
787
842
|
exit_code = 0
|
|
@@ -805,7 +860,7 @@ def execute_plan(plan: Dict[str, Any], path: str) -> int:
|
|
|
805
860
|
info(f"plan done: {output}, but the {check_result.get('platform') or 'content'} check failed" + (f": {', '.join(failed_rows)}" if failed_rows else ""))
|
|
806
861
|
else:
|
|
807
862
|
info(f"plan done: {output}")
|
|
808
|
-
emit(output, plan=path, tool=tool, stages=[tool] + (["check"] if check_result else []), check=check_result, tool_result=doc,
|
|
863
|
+
emit(output, plan=path, plan_sha256=digest, unbound_inputs=unbound, tool=tool, stages=[tool] + (["check"] if check_result else []), check=check_result, tool_result=doc,
|
|
809
864
|
verification=([{"step": "check", "ok": not exit_code, "platform": check_result.get("platform")}] if check_result else [])
|
|
810
865
|
+ [s for s in (doc.get("verification") or []) if s.get("step") != "probe"])
|
|
811
866
|
return 0
|
|
@@ -902,6 +957,10 @@ def main() -> int:
|
|
|
902
957
|
help="write the project's cut as an editor timeline instead of rendering it: .fcpxml (Final Cut, "
|
|
903
958
|
"Resolve), .edl (CMX 3600: Premiere, Resolve, Avid) or .otio (OpenTimelineIO); clips, speed, "
|
|
904
959
|
"transitions, the music bed and chapter markers carry over, everything else is listed as not exported")
|
|
960
|
+
ap.add_argument("--audit-log", metavar="FILE",
|
|
961
|
+
help="append the audit records of this run (a refused plan, a lock taken over, ...) to FILE as JSON lines; "
|
|
962
|
+
"default for a refused plan: <plan>.audit.jsonl next to it. A record that cannot be written is a "
|
|
963
|
+
"warning, never a failure")
|
|
905
964
|
ap.add_argument("--stop-after", choices=["clips", "join", "silence", "fit", "captions", "graphics", "overlays", "audio", "loudness", "export"], help="stop after this stage (for iterating)")
|
|
906
965
|
tpl = ap.add_argument_group("delivery templates (one command per destination)")
|
|
907
966
|
tpl.add_argument("--template", metavar="NAME", help="render INPUT with a shipped template: " + ", ".join(template_names())
|
|
@@ -923,6 +982,7 @@ def main() -> int:
|
|
|
923
982
|
add_hw_orchestrator_args(ap)
|
|
924
983
|
args = ap.parse_args()
|
|
925
984
|
apply_common(args)
|
|
985
|
+
STATE.audit_log = os.path.abspath(args.audit_log) if args.audit_log else None
|
|
926
986
|
|
|
927
987
|
if args.list_templates:
|
|
928
988
|
list_templates()
|