ffmpeg-skill 3.6.3 → 3.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -0
- package/docs/contract.md +2 -2
- package/package.json +1 -1
- package/scripts/_common/asr.py +78 -39
- package/scripts/_common/decision.py +21 -5
- package/scripts/_common/runner.py +225 -11
- package/scripts/_contract.py +2 -0
- package/scripts/batch.py +40 -6
- package/scripts/caption.py +31 -17
- package/scripts/cut.py +7 -3
- package/scripts/export.py +21 -11
- package/scripts/join.py +5 -5
- package/scripts/metadata.py +1 -1
- package/scripts/render.py +35 -5
- package/scripts/report.py +4 -2
- package/scripts/sequence.py +34 -11
- package/scripts/silence.py +11 -6
package/README.md
CHANGED
|
@@ -676,6 +676,8 @@ CI (`.github/workflows/ci.yml`) runs on every pull request and on pushes to `mai
|
|
|
676
676
|
|
|
677
677
|
**Releasing** is fully automated end to end, including the version number itself — a PR doesn't need to touch `package.json`, `docs/contract.md`, or `CHANGELOG.md` at all. Once a PR merges to `main`, `.github/workflows/release.yml` takes it from there: if nobody bumped the version by hand, it resolves the next version ([.github/scripts/resolve_version.py](.github/scripts/resolve_version.py)) from the labels on every PR merged since the last tag (`minor`/`feature`/`enhancement` → minor, `fix`/`bug`/`patch` → patch, an unlabeled PR defaults to `patch`; a PR whose only labels are `chore`, `ci`, `docs` or `dependencies` is not releasable, so a merge that changes no shipped file releases nothing). Most PRs don't need a label added by hand: `release-drafter.yml`'s `autolabel` job applies one automatically from the PR's title/changed files (`Fix ...` → `fix`, `Add ...`/`feat ...` → `feature`, `docs`/`.md`/`.github/`/`build(deps)` changes → `chore`) as soon as it's opened — add a label yourself only to override that. **A major version is never chosen automatically**: no label rule produces `major`, and the workflow refuses to auto-bump across a major boundary even if someone applies that label — a real major release is a deliberate `package.json` bump in a PR, which the manual path below already handles. It bumps `package.json`, `.claude-plugin/plugin.json`, `docs/contract.md` and `docs/roadmap.md`'s released-version sentence, writes a `CHANGELOG.md` section listing those PRs (and any issues they closed), and pushes that commit to `main` itself. Either way — auto-bumped or hand-bumped in the PR — it then creates the `vX.Y.Z` tag, publishes a GitHub Release with notes extracted from `CHANGELOG.md`'s matching section, and publishes the package to npm. A PR that still wants to write its own version bump and `CHANGELOG.md` prose (e.g. to explain the "why" of a release by hand) can — the automation only fills in when nobody made that call already. A push to `main` with nothing new to release is a no-op. npm publishing needs an `NPM_TOKEN` repo secret (an npm access token with publish rights on this package) — without it the tag and GitHub Release still happen, only the npm step is skipped. A repo that depends on this one (an editing skill, an agent) should pin an `ffmpeg-skill` version by tag or npm version, not by tracking `main` — a merged-but-not-yet-released commit on `main` can be ahead of the last published npm version for the few minutes between merge and this workflow completing.
|
|
678
678
|
|
|
679
|
+
**A version that was tagged but never reached npm** (the npm step failed; a re-run of that release sees the tag and skips the publish) is finished by dispatching the same workflow on its tag: `gh workflow run release.yml --ref vX.Y.Z -f version=X.Y.Z`. That runs only the workflow's `recover` job. It refuses a version that is not `X.Y.Z`, has no tag, or that npm already has, and a run whose commit is not the tagged one: the npm provenance attestation names the run's commit, so it must be the commit the package is built from. It publishes with `--provenance` under the `latest` dist-tag when the version is above every stable (non-prerelease) version npm has, and under `backport` otherwise, so `latest` never moves back. Then it publishes the `plugin` branch when no commit there is for this version and the branch's tip is older, and the MCP Registry entry when the registry does not list the version (never for a `backport`). The GitHub Release is left as it is. A tag cut before the job was added (#448; every tag up to v3.6.3) has no `recover` job at its commit and cannot be recovered this way.
|
|
680
|
+
|
|
679
681
|
Contributing a change: see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
680
682
|
|
|
681
683
|
## Docs
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`2.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`3.6.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`3.6.5`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -149,7 +149,7 @@ JSON would have been rewritten for that. The flat keys, typed per tool in `outpu
|
|
|
149
149
|
"deprecated": [],
|
|
150
150
|
"removed": [{"what": "...", "since": "1.10.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"},
|
|
151
151
|
{"what": "running on FFmpeg 5.x (the requirements.ffmpeg floor)", "since": "2.14.0", "replacement": "FFmpeg 6.1 or later; ...", "removed_in": "3.0.0", "where": "behaviour"}],
|
|
152
|
-
"skill": {"id": "ffmpeg-skill", "version": "3.6.
|
|
152
|
+
"skill": {"id": "ffmpeg-skill", "version": "3.6.5", "execution_mode": "local", "kind": "execution",
|
|
153
153
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
154
154
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
155
155
|
"requirements": {"python": ">=3.10 (standard library only)", "ffmpeg": ">=6.1", "ffprobe": ">=6.1"},
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "3.6.
|
|
3
|
+
"version": "3.6.5",
|
|
4
4
|
"mcpName": "io.github.kajisho5/ffmpeg-skill",
|
|
5
5
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
6
6
|
"keywords": [
|
package/scripts/_common/asr.py
CHANGED
|
@@ -9,7 +9,7 @@ why it is its own file.
|
|
|
9
9
|
"""
|
|
10
10
|
from __future__ import annotations
|
|
11
11
|
|
|
12
|
-
import
|
|
12
|
+
import math
|
|
13
13
|
import os
|
|
14
14
|
import re
|
|
15
15
|
import unicodedata
|
|
@@ -18,7 +18,7 @@ from typing import Any, Dict, List, NamedTuple, Optional, Tuple
|
|
|
18
18
|
|
|
19
19
|
from _common.decision import _unspaced_script, fmt_srt_time, parse_time
|
|
20
20
|
from _common.emit import die, info, warn
|
|
21
|
-
from _common.runner import read_text_or_die
|
|
21
|
+
from _common.runner import parse_json, read_json, read_text_or_die
|
|
22
22
|
|
|
23
23
|
# The engines this skill knows how to drive, and how to install each -- one string, so every
|
|
24
24
|
# tool that needs a transcript refuses in the same words. The two Parakeet engines are
|
|
@@ -85,26 +85,60 @@ class Transcription:
|
|
|
85
85
|
f"facts={self.facts!r}, notes={self.notes!r})")
|
|
86
86
|
|
|
87
87
|
|
|
88
|
+
# A line that starts a cue even with no blank line before it: a clock time, "-->", a clock time,
|
|
89
|
+
# then nothing or whitespace (ffmpeg's position fields, X1:.. Y1:..). ffmpeg's SRT demuxer starts
|
|
90
|
+
# an event at every such line; the first line of a block holding "-->" is a timing line whatever
|
|
91
|
+
# its shape.
|
|
92
|
+
_SRT_CUE_START = re.compile(r"^\s*\d+:\d+[\d:.,]*\s*-->\s*\d+:\d+[\d:.,]*(\s|$)")
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
def srt_blocks(text: str) -> List[Tuple[str, List[str]]]:
|
|
96
|
+
"""(timing line, text lines) of every cue in SRT text whose line ends are already `\\n`.
|
|
97
|
+
|
|
98
|
+
A blank (or whitespace-only) line ends a cue, and the first line holding "-->" in a run of
|
|
99
|
+
lines is its timing line; the lines before it (the index) are not text. Inside a run, a later
|
|
100
|
+
line shaped like a timing line (_SRT_CUE_START) starts the next cue, and a bare number just
|
|
101
|
+
before it is that cue's index, so an SRT written without blank lines between cues reads as
|
|
102
|
+
ffmpeg reads it rather than as one cue. A later line that merely contains "-->" is text."""
|
|
103
|
+
out: List[Tuple[str, List[str]]] = []
|
|
104
|
+
for run in re.split(r"\n\s*\n", text):
|
|
105
|
+
lines = [ln for ln in run.split("\n") if ln.strip()]
|
|
106
|
+
first = next((i for i, ln in enumerate(lines) if "-->" in ln), None)
|
|
107
|
+
if first is None:
|
|
108
|
+
continue
|
|
109
|
+
timing, body = lines[first], []
|
|
110
|
+
for ln in lines[first + 1:]:
|
|
111
|
+
if _SRT_CUE_START.match(ln):
|
|
112
|
+
if body and body[-1].strip().isdigit():
|
|
113
|
+
body.pop()
|
|
114
|
+
out.append((timing, body))
|
|
115
|
+
timing, body = ln, []
|
|
116
|
+
else:
|
|
117
|
+
body.append(ln)
|
|
118
|
+
out.append((timing, body))
|
|
119
|
+
return out
|
|
120
|
+
|
|
121
|
+
|
|
122
|
+
def srt_timing_times(line: str) -> Tuple[str, str]:
|
|
123
|
+
"""The start and end tokens of an SRT timing line: what is before the first "-->", and the
|
|
124
|
+
first word after it. ffmpeg reads the two times and ignores the rest (position fields
|
|
125
|
+
`X1:.. X2:.. Y1:.. Y2:..`, or a stray second "-->")."""
|
|
126
|
+
a, _sep, b = line.partition("-->")
|
|
127
|
+
words = b.split()
|
|
128
|
+
return a.strip(), words[0] if words else ""
|
|
129
|
+
|
|
130
|
+
|
|
88
131
|
def parse_srt(path: str) -> List[Tuple[float, float, str]]:
|
|
89
132
|
cues: List[Tuple[float, float, str]] = []
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
times = next((b for b in block if "-->" in b), None)
|
|
98
|
-
if times:
|
|
99
|
-
a, b = times.split("-->")
|
|
100
|
-
text = "\n".join(block[block.index(times) + 1:]).strip()
|
|
101
|
-
try:
|
|
102
|
-
cues.append((parse_time(a), parse_time(b), text))
|
|
103
|
-
except ValueError as e: # includes MissingFpsError: SRT timings are hh:mm:ss,ms, never frames
|
|
104
|
-
die(f"{path}: cannot read the timing line {times.strip()!r}: {e}")
|
|
105
|
-
block = []
|
|
133
|
+
for times, body in srt_blocks(read_text_or_die(path, "--srt")):
|
|
134
|
+
a, b = srt_timing_times(times)
|
|
135
|
+
text = "\n".join(body).strip()
|
|
136
|
+
try:
|
|
137
|
+
cues.append((parse_time(a), parse_time(b), text))
|
|
138
|
+
except ValueError as e: # includes MissingFpsError: SRT timings are hh:mm:ss,ms, never frames
|
|
139
|
+
die(f"{path}: cannot read the timing line {times.strip()!r}: {e}", kind="input")
|
|
106
140
|
if not cues:
|
|
107
|
-
die(f"no cues found in {path}")
|
|
141
|
+
die(f"no cues found in {path}", kind="input")
|
|
108
142
|
return cues
|
|
109
143
|
|
|
110
144
|
|
|
@@ -204,8 +238,9 @@ def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:
|
|
|
204
238
|
# line with two adjacent pipes ("a||b") becomes "a\n\nb" -- and writing that blank line
|
|
205
239
|
# raw would split one cue into two malformed half-blocks (the second missing its own
|
|
206
240
|
# index/timecode). Collapse any run of blank lines within the cue text to a single
|
|
207
|
-
# newline so the cue's own text can never fake the format's block boundary.
|
|
208
|
-
|
|
241
|
+
# newline so the cue's own text can never fake the format's block boundary. A line of
|
|
242
|
+
# only spaces or tabs is blank to every SRT reader too, so it goes the same way.
|
|
243
|
+
t = "\n".join(ln for ln in t.split("\n") if ln.strip())
|
|
209
244
|
fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")
|
|
210
245
|
# under batch.py: this SRT as written is the run's own (an edit made after it is not)
|
|
211
246
|
from _common.runner import report_committed
|
|
@@ -476,20 +511,24 @@ def whisper_word_timings(srt_path: Optional[str]) -> List[Tuple[float, float, st
|
|
|
476
511
|
if not os.path.exists(cand):
|
|
477
512
|
continue
|
|
478
513
|
try:
|
|
479
|
-
data =
|
|
514
|
+
data = read_json(cand)
|
|
480
515
|
except (OSError, ValueError):
|
|
481
516
|
continue
|
|
517
|
+
# the shape is not checked first: a segment that is a string or a list, or "segments" that
|
|
518
|
+
# is a number, is simply no words, as _dicts() reads every engine's JSON
|
|
482
519
|
raw = []
|
|
483
520
|
if isinstance(data, dict):
|
|
484
|
-
raw =
|
|
485
|
-
for seg in data.get("segments")
|
|
486
|
-
raw.extend((seg
|
|
521
|
+
raw = _dicts(data.get("words"))
|
|
522
|
+
for seg in _dicts(data.get("segments")):
|
|
523
|
+
raw.extend(_dicts(seg.get("words")))
|
|
487
524
|
words = []
|
|
488
525
|
for w in raw:
|
|
489
526
|
try:
|
|
490
527
|
text = str(w.get("word") or w.get("text") or "").strip()
|
|
491
|
-
|
|
492
|
-
|
|
528
|
+
start, end = float(w["start"]), float(w["end"])
|
|
529
|
+
# NaN / Infinity are valid to json.loads and place no word anywhere
|
|
530
|
+
if text and math.isfinite(start) and math.isfinite(end):
|
|
531
|
+
words.append((start, end, text))
|
|
493
532
|
except (AttributeError, KeyError, TypeError, ValueError):
|
|
494
533
|
continue
|
|
495
534
|
if words:
|
|
@@ -519,12 +558,12 @@ def _words_from_whisper_cpp_json(path: str) -> "List[Dict[str, Any]]":
|
|
|
519
558
|
boundaries). A token that is part of a character (U+FFFD) is still glued.
|
|
520
559
|
"""
|
|
521
560
|
try:
|
|
522
|
-
doc =
|
|
561
|
+
doc = read_json(path)
|
|
523
562
|
except (OSError, ValueError):
|
|
524
563
|
return []
|
|
525
564
|
out: "List[Dict[str, Any]]" = []
|
|
526
|
-
for seg in (doc.get("transcription")
|
|
527
|
-
for tok in (seg.get("tokens")
|
|
565
|
+
for seg in _dicts(doc.get("transcription")) if isinstance(doc, dict) else []:
|
|
566
|
+
for tok in _dicts(seg.get("tokens")):
|
|
528
567
|
text = str(tok.get("text") or "")
|
|
529
568
|
if not text.strip() or text.strip().startswith("[_"):
|
|
530
569
|
continue
|
|
@@ -552,7 +591,7 @@ def _whisper_json_language(path: str) -> Optional[str]:
|
|
|
552
591
|
"""The language a Whisper engine's JSON says it decoded: whisper.cpp's `result.language`, or
|
|
553
592
|
openai-whisper's top-level `language`. None when the file has neither."""
|
|
554
593
|
try:
|
|
555
|
-
doc =
|
|
594
|
+
doc = read_json(path) or {}
|
|
556
595
|
lang = str(((doc.get("result") or {}).get("language")) or doc.get("language") or "").strip().lower()
|
|
557
596
|
except (OSError, ValueError, AttributeError):
|
|
558
597
|
return None
|
|
@@ -568,12 +607,12 @@ def _heard_language(heard: list) -> Optional[str]:
|
|
|
568
607
|
def _words_from_openai_whisper_json(path: str) -> "List[Dict[str, Any]]":
|
|
569
608
|
"""openai-whisper --word_timestamps True --output_format json: segments[].words[]."""
|
|
570
609
|
try:
|
|
571
|
-
doc =
|
|
610
|
+
doc = read_json(path)
|
|
572
611
|
except (OSError, ValueError):
|
|
573
612
|
return []
|
|
574
613
|
out: "List[Dict[str, Any]]" = []
|
|
575
|
-
for seg in (doc.get("segments")
|
|
576
|
-
for w in (seg.get("words")
|
|
614
|
+
for seg in _dicts(doc.get("segments")) if isinstance(doc, dict) else []:
|
|
615
|
+
for w in _dicts(seg.get("words")):
|
|
577
616
|
try:
|
|
578
617
|
out.append({"word": str(w.get("word") or w.get("text") or "").strip(),
|
|
579
618
|
"start": float(w["start"]), "end": float(w["end"])})
|
|
@@ -592,7 +631,7 @@ def _words_from_parakeet_mlx_json(path: str) -> "List[Dict[str, Any]]":
|
|
|
592
631
|
after it starts a word too (" m" "is" "ter" " " "Q" "u" "il" "ter" -> "mister", "Quilter").
|
|
593
632
|
"""
|
|
594
633
|
try:
|
|
595
|
-
doc =
|
|
634
|
+
doc = read_json(path)
|
|
596
635
|
except (OSError, ValueError):
|
|
597
636
|
return []
|
|
598
637
|
out: "List[Dict[str, Any]]" = []
|
|
@@ -619,7 +658,7 @@ def _words_from_parakeet_mlx_json(path: str) -> "List[Dict[str, Any]]":
|
|
|
619
658
|
def _cues_from_parakeet_mlx_json(path: str) -> List[Tuple[float, float, str]]:
|
|
620
659
|
"""parakeet-mlx's own sentence segmentation, as cues."""
|
|
621
660
|
try:
|
|
622
|
-
doc =
|
|
661
|
+
doc = read_json(path)
|
|
623
662
|
except (OSError, ValueError):
|
|
624
663
|
return []
|
|
625
664
|
cues: List[Tuple[float, float, str]] = []
|
|
@@ -642,7 +681,7 @@ def _dicts(value: Any) -> "List[Dict[str, Any]]":
|
|
|
642
681
|
def _words_from_parakeet_cpp_json(text: str) -> "List[Dict[str, Any]]":
|
|
643
682
|
"""parakeet-cli transcribe --json: {"words": [{"w", "start", "end", "conf"}]}, times in seconds."""
|
|
644
683
|
try:
|
|
645
|
-
doc =
|
|
684
|
+
doc = parse_json(text)
|
|
646
685
|
except ValueError:
|
|
647
686
|
return []
|
|
648
687
|
out: "List[Dict[str, Any]]" = []
|
|
@@ -978,7 +1017,7 @@ def _heard_nothing(text: str, key: str) -> bool:
|
|
|
978
1017
|
Anything else that yields no words -- not JSON, the wrong shape, entries that do not parse --
|
|
979
1018
|
is output that could not be read, not a finding that nobody spoke."""
|
|
980
1019
|
try:
|
|
981
|
-
doc =
|
|
1020
|
+
doc = parse_json(text)
|
|
982
1021
|
except ValueError:
|
|
983
1022
|
return False
|
|
984
1023
|
return isinstance(doc, dict) and doc.get(key) == []
|
|
@@ -990,7 +1029,7 @@ def _whisper_heard_nothing(path: str, key: str) -> bool:
|
|
|
990
1029
|
no speech (#414). A document with text and no word timings is a build without them, not
|
|
991
1030
|
silence, and stays the "no word-level timings" refusal."""
|
|
992
1031
|
try:
|
|
993
|
-
doc =
|
|
1032
|
+
doc = read_json(path)
|
|
994
1033
|
except (OSError, ValueError):
|
|
995
1034
|
return False
|
|
996
1035
|
if not isinstance(doc, dict) or not isinstance(doc.get(key), list):
|
|
@@ -6,7 +6,6 @@ and encoder-selection rules testable on their own.
|
|
|
6
6
|
"""
|
|
7
7
|
from __future__ import annotations
|
|
8
8
|
|
|
9
|
-
import json
|
|
10
9
|
import math
|
|
11
10
|
import os
|
|
12
11
|
import re
|
|
@@ -17,6 +16,7 @@ from typing import Any, Dict, List, Optional, Sequence, Tuple
|
|
|
17
16
|
from _common.color import (as_limited, bt709_label_filter, bt709_matrix_args, bt709_matrix_filter, bt709_tag_args, drawn_stream_format,
|
|
18
17
|
is_neutral_color, limited_range_filter, rgb_island, rgb_to_bt709_filter, _sdr_bt709)
|
|
19
18
|
from _common.emit import die, record_fallback, refuse, warn
|
|
19
|
+
from _common.runner import read_json
|
|
20
20
|
from _common.runner import CODECS, STATE, ffmpeg_encoders, ffmpeg_list_error, ffmpeg_version, hw_platform_reason
|
|
21
21
|
|
|
22
22
|
|
|
@@ -1025,13 +1025,29 @@ def load_brand(path: Optional[str]) -> Dict[str, Any]:
|
|
|
1025
1025
|
if not path:
|
|
1026
1026
|
return brand
|
|
1027
1027
|
if not os.path.exists(path):
|
|
1028
|
-
die(f"brand file not found: {path}")
|
|
1028
|
+
die(f"brand file not found: {path}", kind="input")
|
|
1029
1029
|
try:
|
|
1030
|
-
data =
|
|
1031
|
-
except
|
|
1032
|
-
die(f"brand file
|
|
1030
|
+
data = read_json(path)
|
|
1031
|
+
except OSError as exc:
|
|
1032
|
+
die(f"brand file {path} cannot be read: {exc.strerror or exc}", kind="input")
|
|
1033
|
+
except ValueError as exc: # malformed, not UTF-8, or nested too deep (read_json)
|
|
1034
|
+
die(f"brand file is not valid JSON: {exc}", kind="input")
|
|
1035
|
+
if not isinstance(data, dict):
|
|
1036
|
+
die(f"brand file {path} must be a JSON object {{...}}, got {type(data).__name__}", kind="input")
|
|
1037
|
+
# the shapes the readers below and brand_caption_style() walk: a block that is not an object,
|
|
1038
|
+
# and a path that is not a string, were a traceback there
|
|
1039
|
+
for key, value in (*data.items(), ("styles.caption", (data.get("styles") or {}).get("caption")
|
|
1040
|
+
if isinstance(data.get("styles"), dict) else None)):
|
|
1041
|
+
if value is None:
|
|
1042
|
+
continue
|
|
1043
|
+
if (isinstance(BRAND_DEFAULTS.get(key), dict) or key == "styles.caption") and not isinstance(value, dict):
|
|
1044
|
+
die(f'brand file {path}: "{key}" must be an object {{...}}, got {type(value).__name__}', kind="input")
|
|
1045
|
+
if key in ("logo", "font_file", "font") and not isinstance(value, str):
|
|
1046
|
+
die(f'brand file {path}: "{key}" must be a string, got {type(value).__name__}', kind="input")
|
|
1033
1047
|
base = Path(path).resolve().parent
|
|
1034
1048
|
for k, v in data.items():
|
|
1049
|
+
if v is None and isinstance(brand.get(k), dict):
|
|
1050
|
+
continue # a null block keeps its defaults: readers walk it as an object (#483 review)
|
|
1035
1051
|
if isinstance(v, dict) and isinstance(brand.get(k), dict):
|
|
1036
1052
|
brand[k].update(v)
|
|
1037
1053
|
else:
|
|
@@ -10,6 +10,7 @@ import json
|
|
|
10
10
|
import os
|
|
11
11
|
import platform
|
|
12
12
|
import argparse
|
|
13
|
+
import contextlib
|
|
13
14
|
import re
|
|
14
15
|
import shutil
|
|
15
16
|
import subprocess
|
|
@@ -329,7 +330,7 @@ class Context:
|
|
|
329
330
|
makes it obvious what run()/emit() depend on and lets tests reset it with ``STATE.reset()``.
|
|
330
331
|
"""
|
|
331
332
|
|
|
332
|
-
__slots__ = ("dry_run", "json", "json_brief", "progress", "fast", "duration_hint", "commands", "timeout", "overwrite", "written", "preexisting", "plan", "plan_written", "plan_inputs", "codec", "alpha_line",
|
|
333
|
+
__slots__ = ("dry_run", "json", "json_brief", "progress", "fast", "duration_hint", "commands", "timeout", "overwrite", "written", "preexisting", "checked", "plan", "plan_written", "plan_inputs", "codec", "alpha_line",
|
|
333
334
|
"hw", "hw_source", "hw_notes", "hw_swaps", "hw_fallback", "hw_env_ignored", "hw_stages", "hw_auto",
|
|
334
335
|
"notes", "warnings", "fallbacks", "skipped", "audit_log", "stage_verification", "first_input", "audio_inputs", "video_inputs",
|
|
335
336
|
"source_timecode", "smpte_hint", "delivered", "expected_timecode")
|
|
@@ -356,6 +357,7 @@ class Context:
|
|
|
356
357
|
self.overwrite = False # --overwrite: an existing output may be replaced
|
|
357
358
|
self.written: set = set() # output paths this process has written itself
|
|
358
359
|
self.preexisting: dict = {} # output path -> (size, mtime_ns) of a file that was there before we ran
|
|
360
|
+
self.checked: dict = {} # real output path -> owned_identity() at its output check (None: absent), or as this run left it; refuse_changed_output() compares it at the replace
|
|
359
361
|
self.plan: Optional[str] = None # --plan FILE: write the dry-run as a plan document (implies --dry-run)
|
|
360
362
|
self.plan_written = False # write_plan() ran (emit or the exit hook), so the hook does not write twice
|
|
361
363
|
self.plan_inputs: List[str] = [] # side inputs (srt/ass/lut/font files) a tool named through escape_filter_path
|
|
@@ -1226,6 +1228,9 @@ def refuse_existing_outputs(paths: Sequence[str]) -> None:
|
|
|
1226
1228
|
real = os.path.realpath(output)
|
|
1227
1229
|
except OSError:
|
|
1228
1230
|
continue
|
|
1231
|
+
# what the decision below is about: compared again where the file is replaced
|
|
1232
|
+
# (refuse_changed_output), so one put there during the run is not replaced unasked
|
|
1233
|
+
_record_checked(output, _checked_state(output) if exists else None)
|
|
1229
1234
|
if not exists or real in STATE.written:
|
|
1230
1235
|
continue
|
|
1231
1236
|
size = mtime_ns = None
|
|
@@ -1259,11 +1264,163 @@ def _remember_output(cmd: Sequence[str]) -> None:
|
|
|
1259
1264
|
output = cmd[-1]
|
|
1260
1265
|
if output == "-" or output.startswith("pipe:") or output.startswith("-"):
|
|
1261
1266
|
return
|
|
1267
|
+
_remember_identity(output)
|
|
1268
|
+
report_committed(output)
|
|
1269
|
+
|
|
1270
|
+
|
|
1271
|
+
def _checked_key(path: str) -> str:
|
|
1272
|
+
"""The place a check is about: `path`'s directory resolved, its last component as named, so a
|
|
1273
|
+
symlink created at an absent target since the check is still that target (#480 review)."""
|
|
1274
|
+
full = os.path.abspath(path)
|
|
1275
|
+
return os.path.join(os.path.realpath(os.path.dirname(full)), os.path.basename(full))
|
|
1276
|
+
|
|
1277
|
+
|
|
1278
|
+
def _parent(path: str) -> str:
|
|
1279
|
+
return os.path.realpath(os.path.dirname(os.path.abspath(path)))
|
|
1280
|
+
|
|
1281
|
+
|
|
1282
|
+
def _checked_state(path: str) -> "Optional[Tuple[Optional[Dict[str, int]], bool]]":
|
|
1283
|
+
"""What is at `path` now: (owned_identity, is a symlink), or None when nothing is there."""
|
|
1284
|
+
if not os.path.lexists(path):
|
|
1285
|
+
return None
|
|
1286
|
+
return (owned_identity(path), os.path.islink(path))
|
|
1287
|
+
|
|
1288
|
+
|
|
1289
|
+
def _record_checked(path: str, state: Any) -> None:
|
|
1290
|
+
"""Record what a check (or this run's own write) found at `path`, with the directory the path
|
|
1291
|
+
resolved to then, under the resolved place and under the path as named: a directory symlink
|
|
1292
|
+
switched since is then found by the name and refused, not looked up as a place never checked."""
|
|
1293
|
+
entry = (_parent(path), state)
|
|
1294
|
+
key = _checked_key(path)
|
|
1295
|
+
STATE.checked[key] = entry
|
|
1296
|
+
STATE.checked[os.path.abspath(path)] = entry
|
|
1297
|
+
# the same file reached by another spelling of its path (a write through the resolved name
|
|
1298
|
+
# after a check through a link) is the same record: a later replace through either matches
|
|
1299
|
+
# -- only a record made in the same directory: one whose link has been switched since keeps
|
|
1300
|
+
# what it saw, so the replace through it is still refused
|
|
1301
|
+
for named in [k for k in STATE.checked if k != key]:
|
|
1302
|
+
try:
|
|
1303
|
+
if _checked_key(named) == key and STATE.checked[named][0] == entry[0]:
|
|
1304
|
+
STATE.checked[named] = entry
|
|
1305
|
+
except OSError:
|
|
1306
|
+
pass
|
|
1307
|
+
|
|
1308
|
+
|
|
1309
|
+
def _remember_identity(path: str) -> None:
|
|
1310
|
+
"""Count `path` as this run's own and record the identity this run left it with, which a
|
|
1311
|
+
later replace of it compares (refuse_changed_output)."""
|
|
1262
1312
|
try:
|
|
1263
|
-
|
|
1313
|
+
real = os.path.realpath(path)
|
|
1314
|
+
STATE.written.add(real)
|
|
1264
1315
|
except OSError:
|
|
1265
|
-
|
|
1266
|
-
|
|
1316
|
+
return
|
|
1317
|
+
_record_checked(path, _checked_state(path))
|
|
1318
|
+
|
|
1319
|
+
|
|
1320
|
+
def refuse_changed_output(path: str, cleanup: "Optional[Callable[[], None]]" = None) -> None:
|
|
1321
|
+
"""Refuse to replace `path` when it is no longer the file its output check decided about.
|
|
1322
|
+
|
|
1323
|
+
refuse_existing_outputs() decides before the encode whether an existing target may be
|
|
1324
|
+
replaced (--overwrite, this run's own file, batch.py's own delivery) and records the
|
|
1325
|
+
target's identity (owned_identity(), None when absent); _remember_identity() records it again
|
|
1326
|
+
when this run writes it. Called under the output's lock just before the replace: a file
|
|
1327
|
+
another process put there meanwhile -- a different file at an existing target, or one at a
|
|
1328
|
+
target that was absent -- is refused (`kind: input`, `reason: output_exists`) and `cleanup`
|
|
1329
|
+
removes the new output's temp, unless --overwrite gave consent to replace whatever is at the
|
|
1330
|
+
path (as batch.py's deliver() does). A path with no recorded check is left alone."""
|
|
1331
|
+
if STATE.overwrite or not path or path == "-" or path.startswith("pipe:") or path.startswith("-"):
|
|
1332
|
+
return
|
|
1333
|
+
try:
|
|
1334
|
+
key, named, real = _checked_key(path), os.path.abspath(path), os.path.realpath(path)
|
|
1335
|
+
# the name first: a resolved place another path also reached says nothing about this one
|
|
1336
|
+
recorded = STATE.checked.get(named) or STATE.checked.get(key)
|
|
1337
|
+
if recorded is None:
|
|
1338
|
+
return
|
|
1339
|
+
(parent_then, expected), parent_now, now = recorded, _parent(path), _checked_state(path)
|
|
1340
|
+
except OSError:
|
|
1341
|
+
return
|
|
1342
|
+
if now == expected and parent_now == parent_then:
|
|
1343
|
+
return
|
|
1344
|
+
if cleanup:
|
|
1345
|
+
cleanup()
|
|
1346
|
+
if parent_now != parent_then:
|
|
1347
|
+
why = (f"is now in another directory ({parent_now!r}, checked in {parent_then!r}): a directory "
|
|
1348
|
+
"link on its path was changed during the run")
|
|
1349
|
+
elif expected is None:
|
|
1350
|
+
why = "did not exist when the run checked it and another process created it during the run"
|
|
1351
|
+
elif now is None:
|
|
1352
|
+
why = "was removed or replaced by something that is not a regular file during the run"
|
|
1353
|
+
else:
|
|
1354
|
+
why = "changed during the run (another process wrote or replaced it after the output check)"
|
|
1355
|
+
refuse("input", "output_exists",
|
|
1356
|
+
f"refusing to replace {path!r}: it {why}; the new output was not put there. Pass --overwrite to "
|
|
1357
|
+
"replace it anyway, or choose another -o path",
|
|
1358
|
+
hint="pass --overwrite, or choose another -o path",
|
|
1359
|
+
outputs=[{"path": path, "resolved": real, "changed_during_run": True,
|
|
1360
|
+
"expected": expected, "found": now}])
|
|
1361
|
+
|
|
1362
|
+
|
|
1363
|
+
def _remove_quietly(path: Optional[str]) -> None:
|
|
1364
|
+
if path:
|
|
1365
|
+
try:
|
|
1366
|
+
os.remove(path)
|
|
1367
|
+
except OSError:
|
|
1368
|
+
pass
|
|
1369
|
+
|
|
1370
|
+
|
|
1371
|
+
def replace_checked(tmp: str, final: str) -> None:
|
|
1372
|
+
"""os.replace(tmp, final) for a file a tool writes itself (a time map, a normalised export):
|
|
1373
|
+
under `final`'s output lock, after refuse_changed_output(), which removes `tmp` when it
|
|
1374
|
+
refuses."""
|
|
1375
|
+
replace_checked_together([(tmp, final)])
|
|
1376
|
+
|
|
1377
|
+
|
|
1378
|
+
def replace_checked_together(pairs: "Sequence[Tuple[str, str]]") -> None:
|
|
1379
|
+
"""Replace each (tmp, final) only once every final has passed refuse_changed_output(), all
|
|
1380
|
+
under their output locks: files that belong together (a fill and its key). A file already at a
|
|
1381
|
+
final is moved aside first and put back if a later replace fails, so a failed pair leaves
|
|
1382
|
+
what was there. Every tmp is removed on a refusal, a refused lock or a failed replace (#480
|
|
1383
|
+
review)."""
|
|
1384
|
+
def cleanup() -> None:
|
|
1385
|
+
for tmp, _final in pairs:
|
|
1386
|
+
_remove_quietly(tmp)
|
|
1387
|
+
aside: List[Tuple[str, Optional[str]]] = [] # (final, where its earlier file went, or None)
|
|
1388
|
+
try:
|
|
1389
|
+
with contextlib.ExitStack() as stack:
|
|
1390
|
+
for final in sorted({final for _tmp, final in pairs}):
|
|
1391
|
+
stack.enter_context(_OutputLock(final))
|
|
1392
|
+
for _tmp, final in pairs:
|
|
1393
|
+
refuse_changed_output(final, cleanup)
|
|
1394
|
+
try:
|
|
1395
|
+
for tmp, final in pairs:
|
|
1396
|
+
kept = None
|
|
1397
|
+
if len(pairs) > 1 and os.path.lexists(final):
|
|
1398
|
+
d, base = os.path.split(final)
|
|
1399
|
+
stem, ext = os.path.splitext(base)
|
|
1400
|
+
kept = reserve_sibling(d, stem, ext, "keep")
|
|
1401
|
+
os.replace(final, kept)
|
|
1402
|
+
try:
|
|
1403
|
+
os.replace(tmp, final)
|
|
1404
|
+
except BaseException:
|
|
1405
|
+
if kept:
|
|
1406
|
+
os.replace(kept, final)
|
|
1407
|
+
raise
|
|
1408
|
+
aside.append((final, kept)) # replaced: undone below if a later one fails
|
|
1409
|
+
except BaseException:
|
|
1410
|
+
for final, kept in reversed(aside):
|
|
1411
|
+
if kept:
|
|
1412
|
+
try:
|
|
1413
|
+
os.replace(kept, final)
|
|
1414
|
+
except OSError:
|
|
1415
|
+
pass
|
|
1416
|
+
elif os.path.exists(final):
|
|
1417
|
+
_remove_quietly(final)
|
|
1418
|
+
raise
|
|
1419
|
+
for _final, kept in aside:
|
|
1420
|
+
_remove_quietly(kept)
|
|
1421
|
+
except BaseException:
|
|
1422
|
+
cleanup()
|
|
1423
|
+
raise
|
|
1267
1424
|
|
|
1268
1425
|
|
|
1269
1426
|
# set only by batch.py for the steps it runs: the path of a JSON file {"replaceable": {real path:
|
|
@@ -1376,10 +1533,7 @@ def remember_written(*paths: Optional[str]) -> None:
|
|
|
1376
1533
|
as replacing someone else's file (#415)."""
|
|
1377
1534
|
for path in paths:
|
|
1378
1535
|
if path and os.path.isfile(path):
|
|
1379
|
-
|
|
1380
|
-
STATE.written.add(os.path.realpath(path))
|
|
1381
|
-
except OSError:
|
|
1382
|
-
pass
|
|
1536
|
+
_remember_identity(path)
|
|
1383
1537
|
report_committed(path)
|
|
1384
1538
|
|
|
1385
1539
|
|
|
@@ -1638,6 +1792,9 @@ def run(cmd: Sequence[str], *, quiet: bool = False, check: bool = True, ctx: "Op
|
|
|
1638
1792
|
if is_ffmpeg:
|
|
1639
1793
|
flush_drawtext_textfiles(cmd)
|
|
1640
1794
|
with _OutputLock(cmd[-1] if is_ffmpeg else "-"):
|
|
1795
|
+
if is_ffmpeg:
|
|
1796
|
+
# a file put at the output since the check above is not ours to stage over
|
|
1797
|
+
refuse_changed_output(cmd[-1])
|
|
1641
1798
|
exec_cmd, final, tmp = _stage_existing_output(cmd) if is_ffmpeg else (list(cmd), None, None)
|
|
1642
1799
|
def even_retry(proc: subprocess.CompletedProcess, ran: List[str]) -> "Tuple[subprocess.CompletedProcess, List[str]]":
|
|
1643
1800
|
"""An attempt that failed on odd dimensions, run again with an even scale: after the
|
|
@@ -1694,6 +1851,9 @@ def run(cmd: Sequence[str], *, quiet: bool = False, check: bool = True, ctx: "Op
|
|
|
1694
1851
|
_fail(tried, proc.returncode, proc.stderr or "", ctx=ctx, retried=retried)
|
|
1695
1852
|
if final and tmp:
|
|
1696
1853
|
if proc.returncode == 0:
|
|
1854
|
+
# still under the lock: the target the check let this run replace, not a file
|
|
1855
|
+
# another process put there during the encode
|
|
1856
|
+
refuse_changed_output(final, lambda: _cleanup_partial_output(exec_cmd))
|
|
1697
1857
|
try:
|
|
1698
1858
|
os.replace(tmp, final)
|
|
1699
1859
|
except OSError as e:
|
|
@@ -2104,6 +2264,9 @@ def place_output(src: str, dst: str, move: bool = False) -> None:
|
|
|
2104
2264
|
d, base = os.path.split(dst)
|
|
2105
2265
|
stem, ext = os.path.splitext(base)
|
|
2106
2266
|
with _OutputLock(dst):
|
|
2267
|
+
# under the lock: the target the check above decided about, not one put there since
|
|
2268
|
+
# (`src` is the caller's to remove, as on every other refusal)
|
|
2269
|
+
refuse_changed_output(dst)
|
|
2107
2270
|
if move:
|
|
2108
2271
|
try:
|
|
2109
2272
|
os.replace(src, dst)
|
|
@@ -2114,6 +2277,7 @@ def place_output(src: str, dst: str, move: bool = False) -> None:
|
|
|
2114
2277
|
tmp = None
|
|
2115
2278
|
try:
|
|
2116
2279
|
tmp = copy_to_sibling(src, dst)
|
|
2280
|
+
refuse_changed_output(dst, lambda: _remove_quietly(tmp))
|
|
2117
2281
|
os.replace(tmp, dst)
|
|
2118
2282
|
except OSError as e:
|
|
2119
2283
|
if tmp:
|
|
@@ -2323,15 +2487,27 @@ def _read_ff_list(flag: str, pattern: str) -> set:
|
|
|
2323
2487
|
return names
|
|
2324
2488
|
|
|
2325
2489
|
|
|
2490
|
+
_LINE_END = re.compile(r"\r*\n|\r")
|
|
2491
|
+
|
|
2492
|
+
|
|
2493
|
+
def normalize_text(text: str) -> str:
|
|
2494
|
+
"""A caller's text as every reader here takes it: a leading UTF-8 BOM dropped, and every line
|
|
2495
|
+
ending `\\n`. `\\r\\r\\n` is one line end, as ffmpeg's text demuxers read it (a CRLF file passed
|
|
2496
|
+
through a second CRLF conversion); a lone `\\r` (classic Mac) is one too. Only these: a line is
|
|
2497
|
+
never split at U+2028, U+0085, \\v or \\f, which str.splitlines() also splits at."""
|
|
2498
|
+
return _LINE_END.sub("\n", text.lstrip("\ufeff"))
|
|
2499
|
+
|
|
2500
|
+
|
|
2326
2501
|
def read_text_or_die(path: str, flag: str) -> str:
|
|
2327
2502
|
"""Read a caller-supplied UTF-8 text file (a cue list, chapters, notes) or fail as kind input
|
|
2328
|
-
with the flag named, instead of a FileNotFoundError / UnicodeDecodeError traceback.
|
|
2503
|
+
with the flag named, instead of a FileNotFoundError / UnicodeDecodeError traceback. The text
|
|
2504
|
+
comes back through normalize_text(): no BOM, `\\n` line ends."""
|
|
2329
2505
|
if os.path.isdir(path):
|
|
2330
2506
|
# checked first: Windows raises PermissionError, not IsADirectoryError, for a directory
|
|
2331
2507
|
die(f"{flag}: {path} is a directory, not a text file")
|
|
2332
2508
|
try:
|
|
2333
|
-
with open(path, "r", encoding="utf-8") as fh:
|
|
2334
|
-
return fh.read()
|
|
2509
|
+
with open(path, "r", encoding="utf-8", newline="") as fh:
|
|
2510
|
+
return normalize_text(fh.read())
|
|
2335
2511
|
except FileNotFoundError:
|
|
2336
2512
|
die(f"{flag}: {path} does not exist")
|
|
2337
2513
|
except IsADirectoryError:
|
|
@@ -2343,6 +2519,44 @@ def read_text_or_die(path: str, flag: str) -> str:
|
|
|
2343
2519
|
return "" # unreachable
|
|
2344
2520
|
|
|
2345
2521
|
|
|
2522
|
+
# Deeper than any project, recipe, brand, plan or transcript this skill reads (those are four or
|
|
2523
|
+
# five levels); a file nested past it is refused before code that walks it recursively
|
|
2524
|
+
# (copy.deepcopy, json.dumps) can raise RecursionError.
|
|
2525
|
+
JSON_MAX_DEPTH = 64
|
|
2526
|
+
|
|
2527
|
+
|
|
2528
|
+
class JsonTooDeep(ValueError):
|
|
2529
|
+
"""parse_json()'s refusal of a document nested past JSON_MAX_DEPTH: a ValueError, so callers
|
|
2530
|
+
that catch ValueError refuse it as malformed JSON; one that falls back on a non-JSON file (a
|
|
2531
|
+
media file) can tell this apart, since the file was JSON."""
|
|
2532
|
+
|
|
2533
|
+
|
|
2534
|
+
def parse_json(text: str) -> Any:
|
|
2535
|
+
"""json.loads for a caller's file: a BOM is dropped, and every way the text can fail to be a
|
|
2536
|
+
document this skill can walk raises ValueError -- malformed JSON (json's own ValueError),
|
|
2537
|
+
nesting too deep for the parser (RecursionError) or deeper than JSON_MAX_DEPTH. Callers that
|
|
2538
|
+
already catch ValueError therefore refuse all three as they refuse malformed JSON."""
|
|
2539
|
+
try:
|
|
2540
|
+
data = json.loads(text.lstrip("\ufeff"))
|
|
2541
|
+
except RecursionError:
|
|
2542
|
+
raise JsonTooDeep(f"nested more than {JSON_MAX_DEPTH} levels deep") from None
|
|
2543
|
+
stack = [(data, 1)]
|
|
2544
|
+
while stack:
|
|
2545
|
+
value, depth = stack.pop()
|
|
2546
|
+
if isinstance(value, (dict, list)):
|
|
2547
|
+
if depth > JSON_MAX_DEPTH:
|
|
2548
|
+
raise JsonTooDeep(f"nested more than {JSON_MAX_DEPTH} levels deep")
|
|
2549
|
+
stack.extend((v, depth + 1) for v in (value.values() if isinstance(value, dict) else value))
|
|
2550
|
+
return data
|
|
2551
|
+
|
|
2552
|
+
|
|
2553
|
+
def read_json(path: "os.PathLike[str] | str") -> Any:
|
|
2554
|
+
"""parse_json() of a UTF-8 file. Raises OSError for a file that cannot be read and ValueError
|
|
2555
|
+
(UnicodeDecodeError is one) for anything else, so `except (OSError, ValueError)` covers it."""
|
|
2556
|
+
with open(path, "r", encoding="utf-8") as fh:
|
|
2557
|
+
return parse_json(fh.read())
|
|
2558
|
+
|
|
2559
|
+
|
|
2346
2560
|
# Deferred to the foot of the module on purpose. runner is the first module the package loads and
|
|
2347
2561
|
# emit needs runner's Context/STATE/ERROR_CODE, so the two form a cycle that has to be cut
|
|
2348
2562
|
# somewhere: by the time this line runs every name emit reads from runner above is defined, and
|
package/scripts/_contract.py
CHANGED
|
@@ -443,6 +443,8 @@ ERROR_REASONS = {
|
|
|
443
443
|
"plan_unknown_tool": "render.py PLAN: the plan names no tool render.py can run; details: {plan, plan_sha256, checked_at, tool, audit_log}",
|
|
444
444
|
"plan_cwd_missing": "render.py PLAN: the directory the plan was made in is gone; details: {plan, plan_sha256, checked_at, cwd, audit_log}",
|
|
445
445
|
"plan_argv_invalid": "render.py PLAN: argv is not a list; details: {plan, plan_sha256, checked_at, argv_type, audit_log}",
|
|
446
|
+
"plan_field_invalid": "render.py PLAN: inputs, unbound_inputs, verify (objects), commands (strings) or cwd has the wrong JSON type; "
|
|
447
|
+
"details: {plan, plan_sha256, checked_at, field, field_type, audit_log}",
|
|
446
448
|
"decoder_missing": "color.py / graphics.py / overlay.py: a VP9 source or --video layer stored with an alpha layer, on a build with no libvpx-vp9 decoder (FFmpeg's own VP9 decoder drops the alpha), kind missing_tool; details: {name}",
|
|
447
449
|
"encoder_missing": "--codec av1/prores: the build's encoder list was read and has no encoder for the codec (kind missing_tool; details: {codec, encoders}); or ffmpeg itself reported \"Unknown encoder\" for the encoder the command named (kind ffmpeg; details: {name}; the hint points at doctor)",
|
|
448
450
|
"asr_engine_missing": "kind missing_tool: --transcribe (caption.py; silence.py with --filler or a sentence flag) found no local speech-to-text engine; the message lists the install lines; details: {}",
|
package/scripts/batch.py
CHANGED
|
@@ -33,7 +33,7 @@ from pathlib import Path
|
|
|
33
33
|
from typing import Any, Dict, List, Optional
|
|
34
34
|
|
|
35
35
|
from _common import STATE, add_common, apply_common, child_args, die, emit, info, run_tool, MEDIA_EXT as _MEDIA_EXT
|
|
36
|
-
from _common.runner import (HW_FORCED_ENV, OWNED_ENV, _OutputLock, add_hw_orchestrator_args, copy_to_sibling, env_hw,
|
|
36
|
+
from _common.runner import (HW_FORCED_ENV, OWNED_ENV, _OutputLock, add_hw_orchestrator_args, copy_to_sibling, env_hw, read_json,
|
|
37
37
|
owned_identity, stop_children)
|
|
38
38
|
from _common.emit import absorb_stage_hw, audit, note, shown, steps_encoder_report, warn
|
|
39
39
|
from _common.decision import rebase_project_paths
|
|
@@ -235,6 +235,39 @@ def intermediate_ext(src: Path) -> str:
|
|
|
235
235
|
return src.suffix.lstrip(".") if src.suffix.lower() in (".wav", ".mp3", ".m4a", ".flac") else "mp4"
|
|
236
236
|
|
|
237
237
|
|
|
238
|
+
# the recipe keys the run reads as a string: a path, a glob, a file-name part
|
|
239
|
+
RECIPE_STRINGS = ("glob", "project", "output_dir", "suffix", "ext")
|
|
240
|
+
|
|
241
|
+
|
|
242
|
+
def check_recipe_types(recipe: Dict[str, Any]) -> None:
|
|
243
|
+
"""Refuse a recipe value of the wrong JSON type before anything runs (A18): a number where a
|
|
244
|
+
path or glob goes, an absolute glob (pathlib refuses it), a step that is not a list. Each was
|
|
245
|
+
a traceback, or a failure of every item naming the step's first character."""
|
|
246
|
+
for key in RECIPE_STRINGS:
|
|
247
|
+
value = recipe.get(key)
|
|
248
|
+
if value is not None and not isinstance(value, str):
|
|
249
|
+
die(f'recipe "{key}" must be a string, got {type(value).__name__} {value!r:.60}', kind="input")
|
|
250
|
+
glob = recipe.get("glob")
|
|
251
|
+
if glob and (os.path.isabs(glob) or glob.startswith(("/", "\\"))):
|
|
252
|
+
die(f'recipe "glob" {glob!r} is an absolute path: it matches names inside the batch folder, e.g. "*.mp4" '
|
|
253
|
+
'or "sub/*.mov"', kind="input")
|
|
254
|
+
key = recipe.get("clip_key")
|
|
255
|
+
try:
|
|
256
|
+
index = int(key) if key is not None and not isinstance(key, (bool, float)) else (0 if key is None else -1)
|
|
257
|
+
except (TypeError, ValueError):
|
|
258
|
+
index = -1
|
|
259
|
+
if index < 0:
|
|
260
|
+
die(f'recipe "clip_key" must be a clip index (0, 1, ...), got {key!r:.60}', kind="input")
|
|
261
|
+
steps = recipe.get("steps")
|
|
262
|
+
if steps is not None and not isinstance(steps, list):
|
|
263
|
+
die(f'recipe "steps" must be a list of steps, each ["script.py", args...], got {type(steps).__name__}',
|
|
264
|
+
kind="input")
|
|
265
|
+
for i, st in enumerate(steps or []):
|
|
266
|
+
if not isinstance(st, list) or not st:
|
|
267
|
+
die(f'recipe step {i} must be a list ["script.py", args...], got {type(st).__name__} {st!r:.60}',
|
|
268
|
+
kind="input")
|
|
269
|
+
|
|
270
|
+
|
|
238
271
|
def step_writes(step: Any) -> bool:
|
|
239
272
|
"""Whether a recipe step writes a file: it names {out} (check.py and probe.py do not)."""
|
|
240
273
|
return isinstance(step, list) and any("{out}" in str(a) for a in step[1:])
|
|
@@ -382,14 +415,14 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path,
|
|
|
382
415
|
return max(1.0, deadline - time.monotonic())
|
|
383
416
|
if recipe.get("project"):
|
|
384
417
|
try:
|
|
385
|
-
proj =
|
|
418
|
+
proj = read_json(str(recipe["project"])) # too deep is a ValueError too (A20)
|
|
386
419
|
except (OSError, ValueError) as e:
|
|
387
420
|
raise ItemError({"step": None, "tool": "render.py", "kind": "input", "reason": "recipe_project_unreadable",
|
|
388
421
|
"message": f"recipe.project: {recipe['project']} could not be read as JSON: {e}"})
|
|
389
422
|
if not isinstance(proj, dict):
|
|
390
423
|
raise ItemError({"step": None, "tool": "render.py", "kind": "input", "reason": "recipe_project_unreadable",
|
|
391
424
|
"message": f"recipe.project: {recipe['project']} is not a JSON object"})
|
|
392
|
-
idx = int(recipe.get("clip_key"
|
|
425
|
+
idx = int(recipe.get("clip_key") or 0) # null is clip 0, as check_recipe_types() reads it
|
|
393
426
|
proj.setdefault("clips", [{}])
|
|
394
427
|
while len(proj["clips"]) <= idx:
|
|
395
428
|
proj["clips"].append({})
|
|
@@ -539,11 +572,12 @@ def main() -> int:
|
|
|
539
572
|
if not folder.is_dir():
|
|
540
573
|
die(f"not a folder: {folder}")
|
|
541
574
|
try:
|
|
542
|
-
recipe =
|
|
575
|
+
recipe = read_json(args.recipe) # too deep is a ValueError too (A20)
|
|
543
576
|
except (OSError, ValueError) as exc:
|
|
544
|
-
die(f"cannot read recipe: {exc}")
|
|
577
|
+
die(f"cannot read recipe: {exc}", kind="input")
|
|
545
578
|
if not isinstance(recipe, dict):
|
|
546
|
-
die("cannot read recipe: not a JSON object")
|
|
579
|
+
die("cannot read recipe: not a JSON object", kind="input")
|
|
580
|
+
check_recipe_types(recipe)
|
|
547
581
|
if recipe.get("project") and not os.path.isabs(recipe["project"]):
|
|
548
582
|
recipe["project"] = str((Path(args.recipe).resolve().parent / recipe["project"]))
|
|
549
583
|
# a recipe error is the same for every item: refuse it once, before any item runs, rather than
|
package/scripts/caption.py
CHANGED
|
@@ -59,6 +59,8 @@ from _common.runner import subtitle_codec_for, subtitle_container_holds
|
|
|
59
59
|
from _common.fonts import font_choice, record_font_choice
|
|
60
60
|
from _common import (ASR_INSTALL_HINT, die_no_engine, parse_srt, transcribe, whisper_word_timings,
|
|
61
61
|
write_srt)
|
|
62
|
+
from _common.asr import srt_blocks
|
|
63
|
+
from _common.runner import normalize_text
|
|
62
64
|
from _common import (SAFE_WIDTH_FRACTION, ORPHAN_MIN_EM, WRAP_MODES, wrap_text, wrap_variants, best_break,
|
|
63
65
|
fit_size, line_em_for_size, MIN_CAPTION_FRACTION,
|
|
64
66
|
break_penalty, _is_weak_line, _atoms, _join, _break_spaced, _bare_word, _function_words,
|
|
@@ -83,10 +85,21 @@ ALIGN = {"bottom": 2, "top": 8, "center": 5, "bottom-left": 1, "bottom-right": 3
|
|
|
83
85
|
FORCE_STYLE_ALIGN = {"bottom": 2, "top": 6, "center": 10, "bottom-left": 1, "bottom-right": 3, "top-left": 5, "top-right": 7}
|
|
84
86
|
|
|
85
87
|
TIME_RE = re.compile(
|
|
86
|
-
r"^\s*(?P<a>[\d:.,@]+)\s*(
|
|
88
|
+
r"^\s*(?P<a>[\d:.,@]+)\s*(?P<sep>-->|-|–|to)\s*(?P<b>[\d:.,@]+)\s+(?P<text>.+)$" # @ = the 1.9 @fps suffix
|
|
87
89
|
)
|
|
88
90
|
|
|
89
91
|
|
|
92
|
+
def cue_times_match(line: str) -> "Optional[re.Match[str]]":
|
|
93
|
+
"""TIME_RE's match of a --text line, or None when the line is caption text. The documented
|
|
94
|
+
separators are "-" and "-->", and with them bare seconds are times ("0-1.5 Hello"). "to" is
|
|
95
|
+
also an English word: "3 to 5 apples" is text, so with "to" both sides must be clock times
|
|
96
|
+
(a ':' in each, "0:03 to 0:05 Hello")."""
|
|
97
|
+
m = TIME_RE.match(line)
|
|
98
|
+
if m and m.group("sep") == "to" and not (":" in m.group("a") and ":" in m.group("b")):
|
|
99
|
+
return None
|
|
100
|
+
return m
|
|
101
|
+
|
|
102
|
+
|
|
90
103
|
# an SRT timing line: the times alone, the cue text on the lines after it
|
|
91
104
|
DROP_FRAME_RE = re.compile(r"^\s*\d+:\d{2}:\d{2};\d+")
|
|
92
105
|
SRT_TIMING_RE = re.compile(r"^\s*\d+:\d{2}:\d{2}[,.]\d+\s*-->\s*\d+:\d{2}:\d{2}[,.]\d+\s*$")
|
|
@@ -95,9 +108,11 @@ SRT_TIMING_RE = re.compile(r"^\s*\d+:\d{2}:\d{2}[,.]\d+\s*-->\s*\d+:\d{2}:\d{2}[
|
|
|
95
108
|
def parse_text_cues(path: str, auto_seconds: float, gap: float, fps: Optional[float] = None) -> List[Tuple[float, float, str]]:
|
|
96
109
|
cues: List[Tuple[float, float, str]] = []
|
|
97
110
|
cursor = 0.0
|
|
98
|
-
|
|
111
|
+
# split at "\n" only: splitlines() also breaks a cue at U+2028, U+0085, \v and \f, which
|
|
112
|
+
# made the rest of the line an auto-timed cue of its own
|
|
113
|
+
lines = read_text_or_die(path, "--text").split("\n")
|
|
99
114
|
for n, raw in enumerate(lines, 1):
|
|
100
|
-
if SRT_TIMING_RE.match(raw
|
|
115
|
+
if SRT_TIMING_RE.match(raw):
|
|
101
116
|
# a cue file puts the text on the line with its times; this line has the times alone, so
|
|
102
117
|
# the file is an SRT. Read as a cue file, its numbers and timing lines would be burned in
|
|
103
118
|
# as captions, auto-timed 3 s apart (#396)
|
|
@@ -107,11 +122,10 @@ def parse_text_cues(path: str, auto_seconds: float, gap: float, fps: Optional[fl
|
|
|
107
122
|
# TIME_RE has no ';', so the line would be burned as caption text, auto-timed
|
|
108
123
|
die(f"--text: line {n} has a drop-frame timecode (';'), which is not read: write it as "
|
|
109
124
|
"hh:mm:ss:ff@29.97 or in seconds", kind="input")
|
|
110
|
-
for
|
|
111
|
-
line = raw.rstrip("\n")
|
|
125
|
+
for line in lines:
|
|
112
126
|
if not line.strip():
|
|
113
127
|
continue
|
|
114
|
-
m =
|
|
128
|
+
m = cue_times_match(line)
|
|
115
129
|
if m:
|
|
116
130
|
try:
|
|
117
131
|
start, end = parse_time(m.group("a"), fps), parse_time(m.group("b"), fps)
|
|
@@ -535,7 +549,7 @@ def max_line_em(args, play_w: Optional[int], play_h: Optional[int]) -> Optional[
|
|
|
535
549
|
def parse_ass_dialogue(path: str) -> str:
|
|
536
550
|
"""The spoken text of an ASS file, for script detection -- style/override blocks stripped."""
|
|
537
551
|
text = []
|
|
538
|
-
for line in read_text_or_die(path, "--ass").
|
|
552
|
+
for line in read_text_or_die(path, "--ass").split("\n"):
|
|
539
553
|
if not line.startswith("Dialogue:"):
|
|
540
554
|
continue
|
|
541
555
|
fields = line.split(",", 9)
|
|
@@ -557,7 +571,7 @@ def parse_ass_cues(path: str, stats: Optional[dict] = None) -> List[Tuple[float,
|
|
|
557
571
|
# the [Events] Format line names the columns; Text is always last (libass reads it the same
|
|
558
572
|
# way), and a file without one gets the standard ten
|
|
559
573
|
names = ["layer", "start", "end", "style", "name", "marginl", "marginr", "marginv", "effect", "text"]
|
|
560
|
-
for line in read_text_or_die(path, "--ass").
|
|
574
|
+
for line in read_text_or_die(path, "--ass").split("\n"):
|
|
561
575
|
if line.startswith("Format:") and "text" in line.lower():
|
|
562
576
|
names = [n.strip().lower() for n in line.partition(":")[2].split(",")]
|
|
563
577
|
continue
|
|
@@ -601,7 +615,8 @@ def count_visible_cues(cues, duration) -> Tuple[int, int]:
|
|
|
601
615
|
def shift_ass_file(src: str, dst: str, offset: float) -> Tuple[int, int]:
|
|
602
616
|
"""Copy an ASS file with every Dialogue start/end moved by `offset` seconds; (lines shifted,
|
|
603
617
|
Dialogue lines copied unshifted because their timestamps could not be read)."""
|
|
604
|
-
|
|
618
|
+
# read as the dry run reads it: a file that is not UTF-8 is refused as kind input on both
|
|
619
|
+
text, n, unshifted = shift_ass_text(read_text_or_die(src, "--ass"), offset)
|
|
605
620
|
Path(dst).write_text(text, encoding="utf-8-sig")
|
|
606
621
|
return n, unshifted
|
|
607
622
|
|
|
@@ -621,7 +636,9 @@ def shift_ass_text(source: str, offset: float) -> Tuple[str, int, int]:
|
|
|
621
636
|
|
|
622
637
|
n = unshifted = 0
|
|
623
638
|
out = []
|
|
624
|
-
|
|
639
|
+
# lines end at "\n" only (libass reads them so); a U+2028 inside a Dialogue's text is text
|
|
640
|
+
body = normalize_text(source)
|
|
641
|
+
for line in (body[:-1] if body.endswith("\n") else body).split("\n"):
|
|
625
642
|
if line.startswith("Dialogue:"):
|
|
626
643
|
head, sep, rest = line.partition(":")
|
|
627
644
|
fields = rest.split(",")
|
|
@@ -929,12 +946,9 @@ def srt_timings_shared_grammar(path: str) -> bool:
|
|
|
929
946
|
"""True when every timing line of the SRT at `path` reads the same to ffmpeg's SRT demuxer as
|
|
930
947
|
to parse_srt(). ffmpeg refuses `00:01,000` (no hours) and reads `00:00:01,5` as 1.005 s, where
|
|
931
948
|
parse_srt() reads 1.5 s (#414); such a file is handed to ffmpeg as a rewritten copy."""
|
|
932
|
-
|
|
933
|
-
#
|
|
934
|
-
|
|
935
|
-
blocks = [[ln for ln in b.split("\n") if ln.strip()] for b in re.split(r"\n\s*\n", text)]
|
|
936
|
-
timings = [next((ln for ln in b if "-->" in ln), None) for b in blocks]
|
|
937
|
-
return all(_SHARED_SRT_TIMING.match(line) for line in timings if line is not None)
|
|
949
|
+
# the timing lines parse_srt() reads, grouped as it groups them (srt_blocks()): a later line
|
|
950
|
+
# that only holds "-->" is dialogue, not a timing this check should judge
|
|
951
|
+
return all(_SHARED_SRT_TIMING.match(times) for times, _body in srt_blocks(read_text_or_die(path, "--srt")))
|
|
938
952
|
|
|
939
953
|
|
|
940
954
|
def mux_subtitle_codec(output: str) -> str:
|
|
@@ -1461,7 +1475,7 @@ def main() -> int:
|
|
|
1461
1475
|
if args.ass and args.offset and os.path.exists(args.ass):
|
|
1462
1476
|
shifted = os.path.splitext(output)[0] + "_offset.ass"
|
|
1463
1477
|
if STATE.dry_run:
|
|
1464
|
-
_text, n, unshifted = shift_ass_text(read_text_or_die(args.ass, "--ass")
|
|
1478
|
+
_text, n, unshifted = shift_ass_text(read_text_or_die(args.ass, "--ass"), args.offset)
|
|
1465
1479
|
info(f"[dry-run] would write {shifted} ({n} cues shifted by {args.offset:+g} s)")
|
|
1466
1480
|
else:
|
|
1467
1481
|
n, unshifted = shift_ass_file(args.ass, shifted, args.offset)
|
package/scripts/cut.py
CHANGED
|
@@ -58,7 +58,7 @@ import tempfile
|
|
|
58
58
|
import zlib
|
|
59
59
|
from typing import List, Tuple
|
|
60
60
|
|
|
61
|
-
from _common.runner import fail_if_classified
|
|
61
|
+
from _common.runner import JsonTooDeep, fail_if_classified, read_json
|
|
62
62
|
from _common import (beat_grid, snap_points, decode_pcm_mono, rms_envelope, BEAT_MIN_CONFIDENCE)
|
|
63
63
|
from _common import require_tool
|
|
64
64
|
from _common.probe import TS_EXTS, signatures_match, stream_signature
|
|
@@ -1199,11 +1199,15 @@ def _grid_from_source(path: str, min_confidence: float) -> "dict":
|
|
|
1199
1199
|
"""The beat grid of `path`: a scenes.py --json document if that is what it is, otherwise a
|
|
1200
1200
|
media file to measure. Reading a document is how a caller avoids a second decode."""
|
|
1201
1201
|
try:
|
|
1202
|
-
|
|
1203
|
-
|
|
1202
|
+
doc = read_json(path) # a media file is not UTF-8 or not JSON: a ValueError, then measured
|
|
1203
|
+
except JsonTooDeep as exc:
|
|
1204
|
+
die(f"--snap-source {path}: {exc}", kind="input")
|
|
1204
1205
|
except (OSError, ValueError):
|
|
1205
1206
|
doc = None
|
|
1206
1207
|
if isinstance(doc, dict) and doc.get("beat_grid"):
|
|
1208
|
+
if not isinstance(doc["beat_grid"], dict) or not isinstance(doc.get("beats") or [], list):
|
|
1209
|
+
die(f"--snap-source {path}: beat_grid must be an object and beats a list "
|
|
1210
|
+
"(regenerate it with `scenes.py MUSIC --beats --json`)", kind="input")
|
|
1207
1211
|
grid = dict(doc["beat_grid"])
|
|
1208
1212
|
grid["beats"] = doc.get("beats") or []
|
|
1209
1213
|
# A scenes.py document carries the supported subset since 1.17; one written by an older
|
package/scripts/export.py
CHANGED
|
@@ -51,7 +51,7 @@ from _common import STATE, run_analysis, add_common, apply_common, bt709_label_f
|
|
|
51
51
|
from _common.decision import hw_preset_video, restate_last_swap
|
|
52
52
|
from _common.emit import note, record_fallback, record_skip, warn
|
|
53
53
|
from _common.probe import square_pixels, square_pixels_note
|
|
54
|
-
from _common.runner import ffmpeg_encoders, guard_sidecar, remember_written, reserve_sibling, run_analysis as _run_analysis, subtitle_codec_for
|
|
54
|
+
from _common.runner import ffmpeg_encoders, guard_sidecar, remember_written, replace_checked, replace_checked_together, reserve_sibling, run_analysis as _run_analysis, subtitle_codec_for
|
|
55
55
|
from _common.emit import file_identity
|
|
56
56
|
from check import SPECS as PLATFORMS, measure_loudness
|
|
57
57
|
from _platforms import ALIASES as _ALIASES, PLATFORMS as PLATFORM_TABLE, TP_TOLERANCE_DB, resolve as resolve_platform
|
|
@@ -209,23 +209,32 @@ def _key_fill(args, p: Dict, meta: Dict, output: str, cmd: List[str], vf: List[s
|
|
|
209
209
|
# the fill is always written to a hidden sibling and moved into place only after the encode
|
|
210
210
|
# succeeds: run() stages and cleans up only ffmpeg's last output (the key), so a failed encode
|
|
211
211
|
# left a partial new fill (the retry was refused as output_exists) or truncated an existing one
|
|
212
|
-
|
|
212
|
+
# the key is staged the same way, and both are committed together: a fill refused at its
|
|
213
|
+
# replace never leaves a new key beside it (which a retry would then refuse as existing)
|
|
214
|
+
fill_tmp = key_tmp = None
|
|
213
215
|
if not STATE.dry_run:
|
|
214
216
|
d, base = os.path.split(output)
|
|
215
217
|
stem, ext = os.path.splitext(base)
|
|
216
218
|
fill_tmp = reserve_sibling(d, stem, ext, "fill") # created exclusively: never through a planted link
|
|
219
|
+
kd, kbase = os.path.split(key)
|
|
220
|
+
kstem, kext = os.path.splitext(kbase)
|
|
221
|
+
key_tmp = reserve_sibling(kd, kstem, kext, "key")
|
|
222
|
+
# this run's own placeholder: run()'s output check must not refuse it as someone's file
|
|
223
|
+
STATE.written.add(os.path.realpath(key_tmp))
|
|
217
224
|
cmd += (p["audio"] if has_audio else ["-an"]) + [fill_tmp or output]
|
|
218
|
-
cmd += ["-map", "[key]"] + p["video"] + tc_args + cfr_args(meta) + ["-an", key]
|
|
225
|
+
cmd += ["-map", "[key]"] + p["video"] + tc_args + cfr_args(meta) + ["-an", key_tmp or key]
|
|
219
226
|
try:
|
|
220
227
|
run(cmd)
|
|
221
|
-
if fill_tmp:
|
|
222
|
-
|
|
228
|
+
if fill_tmp and key_tmp:
|
|
229
|
+
# under both locks, and only while each is the file guard_sidecar() decided about
|
|
230
|
+
replace_checked_together([(fill_tmp, output), (key_tmp, key)])
|
|
223
231
|
finally:
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
os.
|
|
232
|
+
for tmp, final in ((fill_tmp, output), (key_tmp, key)):
|
|
233
|
+
if tmp:
|
|
234
|
+
# the command is recorded with the path the caller asked for, not the temp name
|
|
235
|
+
STATE.commands[:] = [c.replace(tmp, final) for c in STATE.commands]
|
|
236
|
+
if os.path.exists(tmp):
|
|
237
|
+
os.remove(tmp)
|
|
229
238
|
remember_written(output, key)
|
|
230
239
|
extra: Dict[str, object] = {"alpha_mode": args.alpha_mode or "straight", "key": {"output": key}}
|
|
231
240
|
# a fill and a key for a switcher carry the picture and the sound; the source's subtitle and
|
|
@@ -746,7 +755,8 @@ def main() -> int:
|
|
|
746
755
|
die(f"--normalize: loudness.py failed: {why} ({kept_clause()})",
|
|
747
756
|
code=child_exit_code(child, proc.returncode) if proc.returncode else 1,
|
|
748
757
|
kind=err.get("kind") or "ffmpeg", output=output, hint=err.get("hint"))
|
|
749
|
-
|
|
758
|
+
# the export this run wrote, unless another process replaced it during the pass
|
|
759
|
+
replace_checked(tmp, output)
|
|
750
760
|
m = measure_loudness(output)
|
|
751
761
|
if not m:
|
|
752
762
|
# the file changed: the measurement before --normalize no longer describes it
|
package/scripts/join.py
CHANGED
|
@@ -44,6 +44,7 @@ from fractions import Fraction
|
|
|
44
44
|
from typing import Any, Dict, List, Optional, Tuple
|
|
45
45
|
|
|
46
46
|
from _common.probe import SILENT_PEAK_DBFS, signatures_match, stream_signature
|
|
47
|
+
from _common.runner import read_text_or_die
|
|
47
48
|
from _common import STATE, AUDIO_MEDIA_EXT, concat_list_line, guard_sidecar, place_output, refuse, sibling_temp, as_limited, draws_chroma, drawn_chain, label_chain, matrix_chain, range_chain, video_args, aac_args, add_common, apply_common, audio_codec_for, default_output, die, dry_run_input_pending, emit, ffmpeg_base, info, measured_level_dbfs, is_audio_output, probe, require_tool, run, validate_color, X264_PRESETS, fmt_secs
|
|
48
49
|
from _common.emit import note, warn
|
|
49
50
|
from _common.decision import av_clip_length, av_trim, stream_starts
|
|
@@ -280,13 +281,12 @@ def read_list(path: str) -> List[str]:
|
|
|
280
281
|
"""--list FILE: one segment per line, in order, relative to the list file's own folder.
|
|
281
282
|
Blank lines and # comments are ignored, and ffmpeg's concat-demuxer spelling
|
|
282
283
|
(`file 'part 01.wav'`) is accepted, so a list written for `ffmpeg -f concat` works as is."""
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
die(f"cannot read --list {path}: {exc}")
|
|
284
|
+
# the shared reader: a BOM, CRLF and a file that is not UTF-8 are read or refused as every
|
|
285
|
+
# other text input is, and a line ends at "\n" only
|
|
286
|
+
text = read_text_or_die(path, "--list")
|
|
287
287
|
base = os.path.dirname(os.path.abspath(path))
|
|
288
288
|
entries: List[str] = []
|
|
289
|
-
for line in text.
|
|
289
|
+
for line in text.split("\n"):
|
|
290
290
|
line = line.strip()
|
|
291
291
|
if not line or line.startswith("#"):
|
|
292
292
|
continue
|
package/scripts/metadata.py
CHANGED
|
@@ -43,7 +43,7 @@ def parse_chapters(path: str, duration: float) -> List[Dict[str, Any]]:
|
|
|
43
43
|
start inside the file, the last chapter running to the file's end."""
|
|
44
44
|
text = read_text_or_die(path, "--chapters")
|
|
45
45
|
entries: List[Dict[str, Any]] = []
|
|
46
|
-
for n, raw in enumerate(text.
|
|
46
|
+
for n, raw in enumerate(text.split("\n"), start=1): # not splitlines(): a title may hold U+2028
|
|
47
47
|
line = raw.strip()
|
|
48
48
|
if not line or line.startswith("#"):
|
|
49
49
|
continue
|
package/scripts/render.py
CHANGED
|
@@ -75,7 +75,7 @@ from typing import Any, Dict, List, Optional, Sequence
|
|
|
75
75
|
from export import PRESETS, PLATFORM_OF
|
|
76
76
|
from _platforms import PLATFORMS, caption_defaults, resolve as resolve_platform
|
|
77
77
|
from _common import guard_sidecar, STATE, add_common, aspect_ratio, brand_caption_style, load_brand, apply_common, child_args, die, emit, info, option_args, probe, require_tool, run, run_tool, place_output, refuse_output_is_input, _check_existing_output, _check_output_path, fingerprint, PLAN_VERSION, ffmpeg_version, X264_PRESETS
|
|
78
|
-
from _common.runner import HW_ENV, HW_FORCED_ENV, add_hw_orchestrator_args, child_exit_code, found_exe, kept_clause, kept_deliverables, mark_delivered, remember_written
|
|
78
|
+
from _common.runner import HW_ENV, HW_FORCED_ENV, add_hw_orchestrator_args, child_exit_code, found_exe, kept_clause, kept_deliverables, mark_delivered, read_json, remember_written
|
|
79
79
|
from _common.emit import absorb_stage, audit, record_fallback, record_skip, refuse, warn
|
|
80
80
|
from _common.decision import rebase_project_paths
|
|
81
81
|
import subprocess
|
|
@@ -182,7 +182,7 @@ def load_template(name: str) -> Dict[str, Any]:
|
|
|
182
182
|
if not path.is_file():
|
|
183
183
|
die(f"unknown template {name!r}", hint="templates: " + ", ".join(template_names()) + " (or 'all')")
|
|
184
184
|
try:
|
|
185
|
-
return
|
|
185
|
+
return read_json(path)
|
|
186
186
|
except (OSError, ValueError) as exc:
|
|
187
187
|
die(f"cannot read template {path}: {exc}")
|
|
188
188
|
return {}
|
|
@@ -504,7 +504,7 @@ def project_input_files(proj: Dict[str, Any], rel: Any) -> List[str]:
|
|
|
504
504
|
# the brand's own logo and font_file are read by the overlay and caption stages, resolved
|
|
505
505
|
# against the brand file as load_brand() resolves them; an unreadable brand is left to them
|
|
506
506
|
try:
|
|
507
|
-
data =
|
|
507
|
+
data = read_json(rel(brand))
|
|
508
508
|
except (OSError, ValueError):
|
|
509
509
|
data = None
|
|
510
510
|
if isinstance(data, dict):
|
|
@@ -565,6 +565,14 @@ def validate_project(proj: Dict[str, Any], timeline: bool = False, stop_after: O
|
|
|
565
565
|
require_object(proj.get(name), name)
|
|
566
566
|
check_keys(proj.get(name), name, name)
|
|
567
567
|
check_keys((proj.get("audio") or {}).get("stems"), "audio.stems", "audio.stems")
|
|
568
|
+
# the values the run reads with .get() or as a dict key: a list there was a traceback (A13)
|
|
569
|
+
audio = proj.get("audio")
|
|
570
|
+
if "audio" in read and isinstance(audio, dict):
|
|
571
|
+
require_object(audio.get("stems"), "audio.stems")
|
|
572
|
+
export = proj.get("export")
|
|
573
|
+
if "export" in read and isinstance(export, dict) and export.get("preset") and not isinstance(export["preset"], str):
|
|
574
|
+
die(f"export.preset: must be a preset name (a string), got {type(export['preset']).__name__} "
|
|
575
|
+
f"{export['preset']!r:.60}")
|
|
568
576
|
if isinstance(proj.get("chapters"), list):
|
|
569
577
|
# Every other project error is raised here, before the first ffmpeg call; a chapter typo
|
|
570
578
|
# found inside the last stage costs a whole render and leaves an unchaptered file behind.
|
|
@@ -590,6 +598,14 @@ def validate_project(proj: Dict[str, Any], timeline: bool = False, stop_after: O
|
|
|
590
598
|
# every clip stage starts from rel(c["src"]): without one it was a KeyError traceback
|
|
591
599
|
if not item.get("src"):
|
|
592
600
|
die(f"clips[{i}]: no src")
|
|
601
|
+
if item.get("speed") and not timeline:
|
|
602
|
+
# the clips stage reads float(speed): "fast" or [2] was a traceback there (A13)
|
|
603
|
+
try:
|
|
604
|
+
spd = float(item["speed"])
|
|
605
|
+
except (TypeError, ValueError):
|
|
606
|
+
spd = float("nan")
|
|
607
|
+
if not (0 < spd < float("inf")):
|
|
608
|
+
die(f"clips[{i}]: speed must be a positive number, got {item['speed']!r:.60}")
|
|
593
609
|
if item.get("snap") is not None:
|
|
594
610
|
# the render reads a clip's snap only on a clip it cuts (in/out), the export never
|
|
595
611
|
if not timeline and (item.get("in") is not None or item.get("out") is not None):
|
|
@@ -984,6 +1000,19 @@ def execute_plan(plan: Dict[str, Any], path: str) -> int:
|
|
|
984
1000
|
_reject_plan(path, digest, "plan_version_unsupported",
|
|
985
1001
|
f"{path}: plan_version {plan.get('plan_version')!r} is not {PLAN_VERSION}",
|
|
986
1002
|
plan_version=plan.get("plan_version"), supported=[PLAN_VERSION])
|
|
1003
|
+
# the fields below are walked as lists of objects or strings: a wrong type was a traceback
|
|
1004
|
+
# with no audit record (A14), and is refused like every other plan error
|
|
1005
|
+
for field, want, item in (("inputs", list, None), ("unbound_inputs", list, None), ("verify", list, dict),
|
|
1006
|
+
("commands", list, str), ("cwd", str, None)):
|
|
1007
|
+
value = plan.get(field)
|
|
1008
|
+
if value is None:
|
|
1009
|
+
continue
|
|
1010
|
+
bad = not isinstance(value, want) or (item is not None and not all(isinstance(v, item) for v in value))
|
|
1011
|
+
if bad:
|
|
1012
|
+
shape = {"inputs": "a list", "unbound_inputs": "a list", "verify": "a list of objects",
|
|
1013
|
+
"commands": "a list of strings", "cwd": "a string"}[field]
|
|
1014
|
+
_reject_plan(path, digest, "plan_field_invalid", f"{path}: {field} must be {shape}, got {type(value).__name__}",
|
|
1015
|
+
field=field, field_type=type(value).__name__)
|
|
987
1016
|
tool = str(plan.get("tool") or "")
|
|
988
1017
|
script = HERE / f"{tool}.py"
|
|
989
1018
|
if not re.fullmatch(r"[a-z][a-z0-9_]*", tool) or not script.exists() or tool == "render":
|
|
@@ -1238,9 +1267,10 @@ def main() -> int:
|
|
|
1238
1267
|
if value:
|
|
1239
1268
|
die(f"{flag} belongs to --template NAME INPUT; a project.json states it in the project itself")
|
|
1240
1269
|
try:
|
|
1241
|
-
|
|
1270
|
+
# read_json(): nesting too deep is a ValueError too, refused here, not a RecursionError
|
|
1271
|
+
proj = read_json(args.project)
|
|
1242
1272
|
except (OSError, ValueError) as exc:
|
|
1243
|
-
die(f"cannot read project: {exc}")
|
|
1273
|
+
die(f"cannot read project: {exc}", kind="input")
|
|
1244
1274
|
if not isinstance(proj, dict):
|
|
1245
1275
|
die(f"{args.project}: not a project or plan object (top level is {type(proj).__name__})")
|
|
1246
1276
|
if "plan_version" in proj:
|
package/scripts/report.py
CHANGED
|
@@ -126,7 +126,7 @@ def pack_report(args) -> int:
|
|
|
126
126
|
text = read_text_or_die(args.pack, "--pack")
|
|
127
127
|
rows: List[List[str]] = []
|
|
128
128
|
header: List[str] = []
|
|
129
|
-
for line in text.
|
|
129
|
+
for line in text.split("\n"):
|
|
130
130
|
line = line.strip()
|
|
131
131
|
if not line.startswith("|"):
|
|
132
132
|
continue
|
|
@@ -233,7 +233,9 @@ def main() -> int:
|
|
|
233
233
|
if meta and meta.get("video"):
|
|
234
234
|
sheets[which], why = sheet_b64(path)
|
|
235
235
|
section("contact_sheet", which, why, f"{which} contact sheet")
|
|
236
|
-
|
|
236
|
+
# one per line, split at "\n" only (a command may hold U+2028 or \f; splitlines() split it)
|
|
237
|
+
command_text = read_text_or_die(args.commands, "--commands") if args.commands else ""
|
|
238
|
+
commands = command_text.removesuffix("\n").split("\n") if command_text else []
|
|
237
239
|
notes = read_text_or_die(args.notes, "--notes") if args.notes else ""
|
|
238
240
|
title = args.title or f"Delivery report — {Path(args.after).name}"
|
|
239
241
|
|
package/scripts/sequence.py
CHANGED
|
@@ -18,11 +18,13 @@ import argparse
|
|
|
18
18
|
import re
|
|
19
19
|
import sys
|
|
20
20
|
import tempfile
|
|
21
|
+
from fractions import Fraction
|
|
21
22
|
from pathlib import Path
|
|
22
23
|
|
|
23
24
|
from _common import add_common, apply_common, default_output, die, emit, ffmpeg_base, info, probe, run, video_args, X264_PRESETS, concat_list_line, fmt_secs
|
|
24
25
|
from _common.emit import note, warn
|
|
25
26
|
from _common.probe import display_size, square_pixels_note
|
|
27
|
+
from _common.timeline import TimelineError, exact_rate
|
|
26
28
|
|
|
27
29
|
|
|
28
30
|
def even(n: float) -> int:
|
|
@@ -34,6 +36,30 @@ def _concat_list_line(path: Path) -> str:
|
|
|
34
36
|
return concat_list_line(str(path))
|
|
35
37
|
|
|
36
38
|
|
|
39
|
+
def frame_durations(count: int, rate: Fraction) -> list:
|
|
40
|
+
"""The concat-list duration of each of `count` frames at `rate`, in whole microseconds: frame
|
|
41
|
+
i lasts round(start(i+1)) - round(start(i)) with start(i) = i/rate, each position rounded
|
|
42
|
+
once, so the demuxer's running sum is frame i's exact start to within half a microsecond at
|
|
43
|
+
every frame (#474: 1/30 written 0.033333 fell 3.3e-7 s behind per frame, half a frame after
|
|
44
|
+
about 50,000 frames)."""
|
|
45
|
+
def at(i: int) -> int:
|
|
46
|
+
return round(i * 1_000_000 / rate)
|
|
47
|
+
return [at(i + 1) - at(i) for i in range(count)]
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
def concat_list(frames: list, rate: Fraction) -> str:
|
|
51
|
+
"""The concat-demuxer list for `frames` (paths) shown one frame each at `rate`."""
|
|
52
|
+
lines = []
|
|
53
|
+
framerate = f"option framerate {rate.numerator}/{rate.denominator}"
|
|
54
|
+
for f, us in zip(frames, frame_durations(len(frames), rate)):
|
|
55
|
+
lines.append(_concat_list_line(f))
|
|
56
|
+
lines.append(framerate)
|
|
57
|
+
lines.append(f"duration {us // 1_000_000}.{us % 1_000_000:06d}")
|
|
58
|
+
lines.append(_concat_list_line(frames[-1])) # concat demuxer: last entry's duration is ignored, so repeat it
|
|
59
|
+
lines.append(framerate)
|
|
60
|
+
return "\n".join(lines)
|
|
61
|
+
|
|
62
|
+
|
|
37
63
|
def natural_key(name: str) -> list:
|
|
38
64
|
"""img2 < img10: digit runs compare as numbers, the rest case-insensitively."""
|
|
39
65
|
return [int(t) if t.isdigit() else t.lower() for t in re.split(r"(\d+)", name)]
|
|
@@ -70,6 +96,10 @@ def main() -> int:
|
|
|
70
96
|
|
|
71
97
|
if args.fps <= 0:
|
|
72
98
|
die("--fps must be > 0")
|
|
99
|
+
try:
|
|
100
|
+
rate = exact_rate(args.fps) # 29.97 is 30000/1001, as every other tool reads it
|
|
101
|
+
except TimelineError as exc:
|
|
102
|
+
die(f"--fps: {exc}")
|
|
73
103
|
directory = Path(args.dir)
|
|
74
104
|
if not directory.is_dir():
|
|
75
105
|
die(f"--dir not found or not a directory: {args.dir}")
|
|
@@ -123,27 +153,20 @@ def main() -> int:
|
|
|
123
153
|
out_w, out_h = even(sw), even(sh)
|
|
124
154
|
|
|
125
155
|
output = args.output or default_output(str(directory).rstrip("/\\") or "sequence", "sequence", "mp4")
|
|
126
|
-
frame_duration = 1.0 / args.fps
|
|
127
|
-
|
|
128
156
|
with tempfile.TemporaryDirectory(prefix="ffmpeg-skill-sequence-") as tmp:
|
|
129
157
|
list_path = Path(tmp) / "frames.txt"
|
|
130
|
-
|
|
131
|
-
for f in frames:
|
|
132
|
-
lines.append(_concat_list_line(f.resolve()))
|
|
133
|
-
lines.append(f"duration {frame_duration:.6f}")
|
|
134
|
-
lines.append(_concat_list_line(frames[-1].resolve())) # concat demuxer: last entry's duration is ignored, so repeat it
|
|
135
|
-
list_path.write_text("\n".join(lines), encoding="utf-8")
|
|
158
|
+
list_path.write_text(concat_list([f.resolve() for f in frames], rate), encoding="utf-8")
|
|
136
159
|
|
|
137
160
|
cmd = ffmpeg_base() + ["-f", "concat", "-safe", "0", "-i", str(list_path)]
|
|
138
|
-
vf = [f"scale={out_w}:{out_h}", "setsar=1", f"fps={
|
|
161
|
+
vf = [f"scale={out_w}:{out_h}", "setsar=1", f"fps={rate.numerator}/{rate.denominator}"]
|
|
139
162
|
cmd += ["-vf", ",".join(vf)]
|
|
140
163
|
# RGB (PNG) frames are converted to the BT.709 the line is tagged with (the first frame's probe)
|
|
141
164
|
cmd += video_args(None, args.crf, args.preset, source=frame_meta)
|
|
142
165
|
# The concat demuxer's trailing repeated-last-file trick (needed so the last real file's
|
|
143
166
|
# duration line takes effect) has been observed to produce an extra frame's worth of
|
|
144
167
|
# duration on some ffmpeg builds -- force the exact intended length rather than trust it.
|
|
145
|
-
|
|
146
|
-
cmd += ["-t", f"{
|
|
168
|
+
total_us = sum(frame_durations(len(frames), rate))
|
|
169
|
+
cmd += ["-t", f"{total_us // 1_000_000}.{total_us % 1_000_000:06d}", "-an", output]
|
|
147
170
|
run(cmd)
|
|
148
171
|
|
|
149
172
|
result = probe(output, role="output")
|
package/scripts/silence.py
CHANGED
|
@@ -28,6 +28,8 @@ from _common import (filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOU
|
|
|
28
28
|
FILLER_PAD, read_text_or_die)
|
|
29
29
|
from _common import asr as _asr
|
|
30
30
|
from _common.emit import warn
|
|
31
|
+
from _common.asr import _dicts
|
|
32
|
+
from _common.runner import parse_json, replace_checked
|
|
31
33
|
from _common import guard_sidecar, refuse_existing_outputs, refuse_output_is_input, _check_output_path
|
|
32
34
|
from _common.emit import refuse
|
|
33
35
|
from _common import detect_silences as detect, STATE, add_common, apply_common, default_output, die, emit, info, is_audio_output, print_json, probe, X264_PRESETS, measured_level_dbfs, fmt_secs
|
|
@@ -114,7 +116,7 @@ def sentence_words(args) -> "Tuple[list, str]":
|
|
|
114
116
|
return usable(args._words, args._words_source)
|
|
115
117
|
if args.words:
|
|
116
118
|
try:
|
|
117
|
-
raw =
|
|
119
|
+
raw = parse_json(read_text_or_die(args.words, "--words"))
|
|
118
120
|
except ValueError as exc:
|
|
119
121
|
die(f"--words {args.words}: not readable JSON ({exc})", kind="input")
|
|
120
122
|
words, _had_segments, _untimed = _words_from_transcript(raw)
|
|
@@ -261,7 +263,8 @@ def write_time_map(path: str, tm: Dict[str, Any]) -> None:
|
|
|
261
263
|
mask = os.umask(0) # mkstemp makes the file 0600; give the map the mode open() would have
|
|
262
264
|
os.umask(mask)
|
|
263
265
|
os.chmod(tmp, 0o666 & ~mask)
|
|
264
|
-
|
|
266
|
+
# under the map's lock, and only while it is the file refuse_existing_outputs() decided about
|
|
267
|
+
replace_checked(tmp, path)
|
|
265
268
|
except BaseException:
|
|
266
269
|
try:
|
|
267
270
|
os.remove(tmp)
|
|
@@ -346,7 +349,7 @@ def resolve_filler(args, meta):
|
|
|
346
349
|
untimed: list = [] # transcript words without a usable timing (a --words file can carry them)
|
|
347
350
|
if args.words:
|
|
348
351
|
try:
|
|
349
|
-
raw =
|
|
352
|
+
raw = parse_json(read_text_or_die(args.words, "--words"))
|
|
350
353
|
except ValueError as exc:
|
|
351
354
|
die(f"--words {args.words}: not readable JSON ({exc})", kind="input")
|
|
352
355
|
source = f"whisper-json:{args.words}"
|
|
@@ -482,11 +485,13 @@ def _words_from_transcript(data):
|
|
|
482
485
|
listed word it could not cut instead of dropping it unseen."""
|
|
483
486
|
raw, had_segments = [], False
|
|
484
487
|
if isinstance(data, dict):
|
|
485
|
-
|
|
486
|
-
|
|
488
|
+
# a "words" or "segments" that is not a list, and a segment that is not an object, carry no
|
|
489
|
+
# words: the file is then refused for having none, not by a traceback
|
|
490
|
+
raw = _dicts(data.get("words"))
|
|
491
|
+
segments = _dicts(data.get("segments"))
|
|
487
492
|
had_segments = bool(segments)
|
|
488
493
|
for seg in segments:
|
|
489
|
-
raw.extend((seg
|
|
494
|
+
raw.extend(_dicts(seg.get("words")))
|
|
490
495
|
elif isinstance(data, list):
|
|
491
496
|
raw = list(data)
|
|
492
497
|
out, untimed = [], []
|