ffmpeg-skill 2.6.0 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/SKILL.md +2 -2
- package/docs/contract.md +40 -7
- package/package.json +1 -1
- package/references/scripts.md +43 -9
- package/scripts/_common/asr.py +747 -94
- package/scripts/_contract.py +43 -7
- package/scripts/caption.py +22 -6
- package/scripts/silence.py +21 -6
package/README.md
CHANGED
|
@@ -82,7 +82,7 @@ An agent that "knows FFmpeg" still guesses: it assumes a frame rate, picks a cod
|
|
|
82
82
|
- **Structured tools, not shell strings.** Each operation is a script with typed arguments. Nothing runs through a shell; no filter graph is accepted from the caller.
|
|
83
83
|
- **A contract the agent can read.** `contract --json` states, for every tool, what it takes, what it writes, which FFmpeg components it needs and how the result is verified. The MCP surface is derived from it.
|
|
84
84
|
- **Verification after execution.** The result is probed, checked against the destination's spec and, when the picture changed, looked at as a contact sheet.
|
|
85
|
-
- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a whisper is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing.
|
|
85
|
+
- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a speech engine (Parakeet or a whisper) is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing. The engine runs on your machine and the audio is never uploaded; parakeet-mlx, faster-whisper and openai-whisper download their model weights the first time they run (parakeet-mlx's default from Hugging Face).
|
|
86
86
|
|
|
87
87
|
## Quick start
|
|
88
88
|
|
|
@@ -371,7 +371,7 @@ The short list for humans. The agent-facing version, with the reasoning, is the
|
|
|
371
371
|
- **Phrase-aware caption breaking (1.16).** `caption.py`/`graphics.py --wrap phrase` (the default) never breaks inside a word or on the wrong side of a hyphen, never leaves a lone digit, kana or punctuation pair on a line, prefers Japanese sentence ends and particles over a mid-word break, and never ends a line on an article or preposition. All four are penalties over break positions that already fit, so no line is widened and the line count never changes; `--wrap measured` restores 1.15's width-only wrap. The text itself is never rewritten or shortened. Since 1.16.1 a Thai run and a katakana word are never broken inside (Thai writes no space inside a phrase and there is no dictionary: the break goes where you put a space or `|`), and a line wider than the safe width is reported as `overlong` with the fix named.
|
|
372
372
|
- **Caption size fitted to the cue (1.17, reachable from the templates since 1.17.1).** At a platform caption size a line holds about six em, so an ordinary sentence needs four lines and `--max-lines 2` used to cut it into consecutive cues — half the sentence arriving late. `caption.py --fit-size` (default `auto`) walks the size down until every cue fits, *before* laying the cues out, with a legibility floor of 4.5 % of the frame height (`--min-size`, default 13 ASS units). An explicit `--size` or a `brand.json` size is a statement about the look and is never overridden — which in 1.17.0 also silenced the fitter on every `render.py --template` run, since a template fills the size from the platform table; 1.17.1 marks that size as the default it is (`"fit_size": "on"` in the filled project), so the type shrinks and no cue is split. `--fit-size off` restores that earlier behaviour byte for byte, and the caption text is still never rewritten to make it fit.
|
|
373
373
|
- **Beat-synced cuts (1.17).** `scenes.py --beats` reports the measured grid — tempo, beat times, and a confidence built from how far the winning autocorrelation lag stands above the others and how many onsets land on it. `cut.py --snap beats` (and a `"snap"` block in a `render.py` project) moves in/out points to the nearest beat within `--snap-tolerance` — and only onto the grid points a measured onset actually marks, never onto the regular grid's continuation through a passage with no music in it. Below `--min-confidence` it **refuses**: a cut point may move to a measured beat and may not appear from one, so speech and ambience get an honest "no steady pulse here" instead of an invented grid.
|
|
374
|
-
- **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Whisper stays optional: `--transcribe` with no engine installed refuses and names the
|
|
374
|
+
- **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Whisper stays optional: `--transcribe` with no engine installed refuses and names the five installs.
|
|
375
375
|
- **Throughput (1.17).** `batch.py --jobs N` runs several files at once, capped at `min(N, cpu_count, 8)` and sharing one `--timeout` budget rather than one per item; the per-item table keeps its order. `render.py --cache DIR` reuses stages whose inputs and arguments did not change, so swapping an export preset re-runs export only. The cache is opt-in with no default directory, and the ffmpeg, skill and contract versions are part of every key, so a cache is never reused across them.
|
|
376
376
|
- **Audiogram (1.16).** `waveform.py --image cover.png` (or `render.py --template audiogram`) puts the waveform over a still plate for an episode that has no picture, with `--platform` for the frame, `--title` and burnt-in captions. The image is a local file you give: nothing is fetched and no cover art is ever invented.
|
|
377
377
|
- **Emoji in captions and titles (1.15).** `caption.py`/`graphics.py --emoji-assets DIR` composites a PNG per emoji (Twemoji/Noto naming, `1f389.png`) on top of the text, because drawtext cannot load a colour emoji font at all and an installed one does not prove libass will draw it in colour — `doctor --json`'s `fonts.emoji` answers that from a render probe. Without assets the run still succeeds and reports `mode: mono`. Nothing is ever downloaded.
|
package/SKILL.md
CHANGED
|
@@ -32,7 +32,7 @@ Ask a short question only when the answer changes the output materially and the
|
|
|
32
32
|
|
|
33
33
|
- **Destination** decides aspect, length limit, loudness and codec; "for Reels" answers all four and names a template. No destination and a plain cut/caption: keep the source format and say so. If the user says "export", "post" or "deliver", ask where.
|
|
34
34
|
- **Duration** ("make it 60 s") without a method: speed up for ≤1.5× changes, trim otherwise, say which. Ask when the content is a talk (trimming loses words) and the change is large.
|
|
35
|
-
- **Captions** without a text source: `--transcribe` if a local
|
|
35
|
+
- **Captions** without a text source: `--transcribe` if a local engine exists, else ask for the text or a timed file; never invent dialogue.
|
|
36
36
|
- **Fonts and brand**: if the user mentions a brand, colours or "our font", ask for or create `brand.json` once, reuse it.
|
|
37
37
|
- **CJK / non-Latin text**: let the tool pick the font by script (`--font` turns it off); `--lang ja|ko` for Han-only text. `doctor --json` `.fonts.scripts` says what renders here. Tofu is a failed job.
|
|
38
38
|
- **Crop position** for `--fit crop`: centre by default, but when the request or the source names an off-centre subject ("keep the product on the right", someone visibly off-centre in the sheet) use `--crop-x`/`--crop-y` (0=left/top, 1=right/bottom) instead of guessing centre, or ask which edge to keep.
|
|
@@ -97,7 +97,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
|
|
|
97
97
|
| "caption it with these lines" (text with times) | `caption.py input.mp4 --text cues.txt` |
|
|
98
98
|
| "keep the subtitles toggleable", "mux in an SRT" | `caption.py input.mp4 --srt subs.srt --mode mux`; repeat `--srt file:lang` for several languages, `.mkv` for more than two |
|
|
99
99
|
| "the captions are tiny / three lines on a Short", "don't chop the sentence" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` keeps `--size` and splits instead, `--min-size` the floor). Keep the template's `--max-lines` (2 on a vertical) and let the size drop; raising it to dodge a shrink stacks two words per line. A word still wider than the column at the floor is sliced at the edge (hyphen preferred), never rewritten, automatic; `broken_inside_word` in the JSON counts it |
|
|
100
|
-
| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (
|
|
100
|
+
| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (Parakeet/whisper; else `--text`) |
|
|
101
101
|
| "TikTok-style captions with the words popping" | `caption.py input.mp4 --text cues.txt --animate pop --karaoke` |
|
|
102
102
|
| "our logo top-right", "a watermark" | `overlay.py input.mp4 --image logo.png --position top-right --scale 200` |
|
|
103
103
|
| "a title for the first 4 seconds" | `overlay.py input.mp4 --text "Title" --position top --start 0 --end 4 --fade 0.4` |
|
package/docs/contract.md
CHANGED
|
@@ -21,7 +21,7 @@ The contract is derived from the code that runs, not maintained beside it:
|
|
|
21
21
|
| Field | Meaning | Changes when |
|
|
22
22
|
|---|---|---|
|
|
23
23
|
| `contract_version` | shape of this document (`1.0`) | a key is renamed, removed or changes meaning |
|
|
24
|
-
| `skill.version` | the npm / package.json version (`2.
|
|
24
|
+
| `skill.version` | the npm / package.json version (`2.7.0`) | any release |
|
|
25
25
|
|
|
26
26
|
A release that adds a tool or a flag keeps `contract_version`; a breaking change to the
|
|
27
27
|
ToolSpec shape bumps it. Consumers pin on `contract_version` and read `skill.version`
|
|
@@ -119,11 +119,15 @@ JSON would have been rewritten for that. The flat keys, typed per tool in `outpu
|
|
|
119
119
|
"contract_version": "1.0",
|
|
120
120
|
"deprecated": [],
|
|
121
121
|
"removed": [{"what": "...", "since": "1.10.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
|
|
122
|
-
"skill": {"id": "ffmpeg-skill", "version": "2.
|
|
122
|
+
"skill": {"id": "ffmpeg-skill", "version": "2.7.0", "execution_mode": "local", "kind": "execution",
|
|
123
123
|
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
|
|
124
124
|
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
|
|
125
125
|
"requirements": {"python": ">=3.9 (standard library only)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
|
|
126
|
-
"execution": {"shell": false, "arbitrary_executables": false, "network": false,
|
|
126
|
+
"execution": {"shell": false, "arbitrary_executables": false, "network": false,
|
|
127
|
+
"model_downloads": {"when": "...", "in_process": {"faster-whisper": "..."},
|
|
128
|
+
"child_process": {"parakeet-mlx": "...", "openai-whisper": "..."},
|
|
129
|
+
"none": ["whisper.cpp", "parakeet.cpp"], "offline": "..."},
|
|
130
|
+
"input_mutation": false}
|
|
127
131
|
}
|
|
128
132
|
```
|
|
129
133
|
|
|
@@ -349,10 +353,10 @@ A project may now carry `"template"` (the name it was filled from) and `"frame":
|
|
|
349
353
|
## Capabilities
|
|
350
354
|
|
|
351
355
|
Names: `ffmpeg`, `ffprobe`, `encoder:<name>`, `filter:<name>`, `bsf:<name>`,
|
|
352
|
-
`external:whisper`. `capabilities.required` is the union of every tool's required list;
|
|
356
|
+
`external:whisper`, `external:parakeet`. `capabilities.required` is the union of every tool's required list;
|
|
353
357
|
`optional` the union of the conditional ones. With detection (the default)
|
|
354
358
|
`available`, `missing` and `missing_optional` are added from `doctor`, which reads
|
|
355
|
-
`ffmpeg -encoders / -filters / -bsfs` and looks for a local whisper. Pass `--static`
|
|
359
|
+
`ffmpeg -encoders / -filters / -bsfs` and looks for a local whisper or Parakeet. Pass `--static`
|
|
356
360
|
to omit detection. Nothing from the environment other than those lists and the
|
|
357
361
|
ffmpeg/ffprobe/python versions is printed; no environment variables, no paths (`doctor`'s
|
|
358
362
|
`hw.default_on` is a boolean derived from `FFMPEG_SKILL_HW`, never its value).
|
|
@@ -360,8 +364,29 @@ ffmpeg/ffprobe/python versions is printed; no environment variables, no paths (`
|
|
|
360
364
|
`external:whisper` is **optional** for two tools since 1.17: `caption.py`
|
|
361
365
|
(`--transcribe`) and `silence.py` (`--filler --transcribe`). Neither requires
|
|
362
366
|
it — both take a transcript the caller already has (`--srt`/`--words`), and
|
|
363
|
-
both refuse with the same
|
|
367
|
+
both refuse with the same install lines when asked to make one with no
|
|
364
368
|
engine present. Whisper is never a dependency of this skill.
|
|
369
|
+
`external:parakeet` is the same kind of optional capability for the same two flags:
|
|
370
|
+
parakeet-mlx, or parakeet-cli with a `.gguf` model it can find. Either engine family
|
|
371
|
+
satisfies `--transcribe`; `--engine auto` (the default, or `FFMPEG_SKILL_ASR_ENGINE`) runs
|
|
372
|
+
Parakeet for English speech and Whisper for every other language. When no `--language` is given
|
|
373
|
+
and no detector can name it, a Whisper engine that can run takes it (whisper.cpp is passed
|
|
374
|
+
`-l auto`: its own default is English). Parakeet then runs only on English *assumed* -- when no
|
|
375
|
+
Whisper engine is installed, or every one failed, or `--model` names an English-only Parakeet
|
|
376
|
+
model -- and the result's top-level `notes` says the transcript is wrong if the speech is not
|
|
377
|
+
English; a multilingual (v3) Parakeet model assumes nothing and gets no note. The result's
|
|
378
|
+
`transcription` says which engine, model and routing decision produced the cues (in every
|
|
379
|
+
`--mode`, `mux` included).
|
|
380
|
+
Every engine transcribes on the machine running the skill, and the audio is never uploaded. The
|
|
381
|
+
skill's own code opens no network connection (`execution.network: false`), but an engine may
|
|
382
|
+
fetch its model weights on first use: parakeet-mlx downloads its default
|
|
383
|
+
`mlx-community/parakeet-tdt-0.6b-v2` from Hugging Face and openai-whisper its model from OpenAI,
|
|
384
|
+
each in its own child process. faster-whisper is a Python library the skill runs inside its own
|
|
385
|
+
process, so its first-run download from Hugging Face is made by the skill's process;
|
|
386
|
+
`HF_HUB_OFFLINE=1` keeps it (and parakeet-mlx) on the cached copy after that. parakeet.cpp and
|
|
387
|
+
whisper.cpp read a model file the user downloaded. `execution.model_downloads` says the same in the
|
|
388
|
+
contract, engine by engine, so a caller that sandboxes the network can tell before a first
|
|
389
|
+
`--transcribe` that faster-whisper's fetch comes from the skill's own process.
|
|
365
390
|
|
|
366
391
|
`doctor` has three states per capability. `available` and `missing` come from a listing
|
|
367
392
|
that was read; `unknown` means the listing that would prove the capability could not be
|
|
@@ -546,11 +571,19 @@ Per-tool keys added after 2.2.2, all additive:
|
|
|
546
571
|
| `short_segments`, `duplicates` | `join.py` (every run) | warnings, never refusals, and the join command is unchanged: `short_segments: [{index, path, duration}]` names each measured input shorter than 2 frames at the join's fps (an audio-only join: shorter than 0.05 s); `duplicates: [{path, indices}]` names a path (compared resolved) listed more than once — repeating a clip can be intended. `[]` when none; each non-empty one adds a `notes` line |
|
|
547
572
|
| `silent` | `waveform.py` (every run) | `true` when the input audio's whole-file peak is at or below `--silence-threshold` (default -50 dBFS), which draws a flat line; `--on-silent warn` (default) renders it anyway with a `notes` line, `fail` refuses (`kind: input`) before ffmpeg runs. `false` when audible, `null` under `--dry-run` (nothing measured). `verified` is unaffected |
|
|
548
573
|
| `encoder`, `hw`, `notes` | every re-encoding tool, `export.py`, `render.py`, `batch.py` (and its rows), `waveform.py` | `encoder`: the video encoder the last command ran (after any GPU→CPU fallback; under `--dry-run`, the planned one), `copy` for a stream copy; absent when no command encoded video. A tool that runs other tools reports its stages': `render.py` its stages and an executed plan's tool, `waveform.py` the title/caption stage that writes its file (its `commands` include that stage's). `hw` is present when VideoToolbox was asked for (`--hw`, `FFMPEG_SKILL_HW=1`, or a stage or `batch.py` item that reported it) and on `export.py` when `FFMPEG_SKILL_HW=1` is set but its delivery preset does not take it: `{requested, source: flag\|env, used, fallback, notes}`. `requested` says whether this run's encodes were asked onto VideoToolbox (`false` for that export), `used` whether the encoder that ran last is VideoToolbox (`null` under `--dry-run`, which ran nothing, and when the tool encoded nothing itself: `batch.py`, whose top-level `hw` gathers its items' `requested`/`source`/`fallback`/`notes` while each `results` row carries the item's own `encoder`, and `hw`/`notes` when VideoToolbox was asked for), `fallback` whether a job VideoToolbox refused was re-encoded on the CPU, and `notes` why. Only a VideoToolbox failure (the encoder could not open, or its compression session failed) is retried on the CPU; any other failure is reported as it happened, once. Every VideoToolbox encode adds a top-level `notes` line: its quality is not CRF-equivalent (`--quality` / a preset's CRF maps to `-q:v` approximately), and `verified` stays what it is for every tool, the measured properties of the output. A `kind: ffmpeg` failure under `--hw` / `FFMPEG_SKILL_HW` carries `encoder` and `hw` for the command that ran last (a stage's, when `render.py`/`waveform.py` re-raise it); when the command itself failed inside the run (not a tool's own check afterwards, such as `cut.py`'s), its message also names that encoder |
|
|
574
|
+
| `transcription` | `caption.py --transcribe` (every `--mode`); inside `filler` for `silence.py --filler --transcribe` | `{routing, detected_language?, engine, model, language}`: the engine and model that made the transcript; `language` is the language the engine was told -- the one named (`--language`, `--filler-lang`), else the one auto's detector named -- and `null` when the engine identified it itself; `detected_language` is present when auto's whisper.cpp detector named one; `routing` says why `--engine auto` chose the engine (`"requested"` for a named engine), ending `, no Parakeet engine transcribed it` when auto chose Parakeet and a Whisper engine made the transcript after every Parakeet engine failed |
|
|
575
|
+
| `notes` | `silence.py --filler --transcribe` (new for `silence.py`; `caption.py` already had it) | present when `--engine auto` ran the English-only Parakeet model on English *assumed* (no language given or detectable, and no Whisper engine installed or none that succeeded): the transcript is wrong if the speech is not English. `caption.py --transcribe` puts the same line in its `notes` |
|
|
549
576
|
|
|
550
577
|
`caption.py --transcribe` and `silence.py --filler --transcribe`: when a local speech engine ran
|
|
551
578
|
and produced no cue, the refusal is `"<engine> found no speech in <input>"` with `kind: input`,
|
|
552
579
|
`reason: "no_speech"` and `engine` in the failure document — distinct from the "no local
|
|
553
|
-
speech-to-text engine found" refusal, which now only means no engine was found.
|
|
580
|
+
speech-to-text engine found" refusal, which now only means no engine was found. When `--engine
|
|
581
|
+
auto` found engines and none transcribed, the refusal is `kind: input` with `reason:
|
|
582
|
+
"engine_failed"` (one ran and failed) or `"english_only"` (the only engines found were Parakeet
|
|
583
|
+
engines auto passed over for a language that is not English), `engine` (the first one it names)
|
|
584
|
+
and `engines: [{engine, reason, detail}]`, where `detail` carries an engine's own error line. A
|
|
585
|
+
named `--engine` that is not installed is `kind: missing_tool`; one that is installed and failed
|
|
586
|
+
is `kind: input`, `reason: "engine_failed"`, with `engine` and `detail`.
|
|
554
587
|
|
|
555
588
|
`check.py` also gains an informational `subtitles` row on **every** platform:
|
|
556
589
|
`PASS` when every soft subtitle stream carries a language tag, `WARN` when one
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ffmpeg-skill",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.7.0",
|
|
4
4
|
"description": "Agent Skill that gives coding agents (Claude Code, Cursor, Codex) a local video editor: 42 FFmpeg tools with a machine-readable contract, contract-derived MCP server, FFmpeg capability detection, probe-first / verify-last workflow. Cut, join, silence removal, fit, captions and karaoke, overlays, motion graphics, HDR to SDR, LUTs, audio clean-up and typed dynamics, sync with drift correction, multicam, loudness, delivery checks, project rendering, batch. No API keys, no cloud, no dependencies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ffmpeg",
|
package/references/scripts.md
CHANGED
|
@@ -490,7 +490,8 @@ each shorter clip's last frame (with silence) out to the longest.
|
|
|
490
490
|
silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [--speech-aware] [-o OUT]
|
|
491
491
|
silence.py INPUT --filler --words transcript.json [--filler-lang auto|en|ja|es|de|fr|pt|it]
|
|
492
492
|
[--filler-words FILE] [--filler-extra W,W] [--filler-keep W,W] [--filler-pad 0.02]
|
|
493
|
-
[--transcribe] [--
|
|
493
|
+
[--transcribe] [--engine auto|parakeet-mlx|parakeet.cpp|whisper.cpp|faster-whisper|openai-whisper]
|
|
494
|
+
[--filler-list] [--max-cuts 400]
|
|
494
495
|
```
|
|
495
496
|
Runs `silencedetect`, keeps `--margin` seconds of air around speech, drops
|
|
496
497
|
gaps shorter than `--min-silence`, and re-encodes once with `select`/`aselect`
|
|
@@ -533,7 +534,10 @@ Three refusals, all `kind: input`, all before any encode:
|
|
|
533
534
|
|
|
534
535
|
- `--filler` with neither `--words` nor `--transcribe` → names both flags.
|
|
535
536
|
- `--transcribe` with no engine on PATH → the same message `caption.py` gives,
|
|
536
|
-
with
|
|
537
|
+
with its install lines (Parakeet and whisper engines). Engines that were found
|
|
538
|
+
but did not transcribe are named instead, with why (`reason: engine_failed`,
|
|
539
|
+
or `english_only` for a Parakeet engine auto passed over for `--filler-lang`
|
|
540
|
+
that is not English; see `caption.py --transcribe` below).
|
|
537
541
|
- `--transcribe` where the engine runs but its build produces no word-level
|
|
538
542
|
timings → names that engine, says some builds do not support word timestamps,
|
|
539
543
|
and points at `--words`. (`--transcribe` drives whichever engine is installed
|
|
@@ -546,7 +550,11 @@ Three refusals, all `kind: input`, all before any encode:
|
|
|
546
550
|
|
|
547
551
|
Results: `filler.removed_count`, `filler.removed_seconds`, `filler.removed`
|
|
548
552
|
(one entry per span), `filler.lang`, `filler.list`, `filler.word_timings` and
|
|
549
|
-
`filler.warnings`.
|
|
553
|
+
`filler.warnings`. With `--transcribe`, `filler.source` is `whisper:ENGINE` or
|
|
554
|
+
`parakeet:ENGINE` and `filler.transcription` says which engine, model, language
|
|
555
|
+
and routing made the words; a top-level `notes` line says when `--engine auto`
|
|
556
|
+
ran the English-only Parakeet model on English assumed (no `--filler-lang`, and
|
|
557
|
+
no Whisper engine that could take it). The existing `removed_seconds` keeps exactly the meaning it
|
|
550
558
|
has always had — the seconds of **silence** this run removed, the figure the
|
|
551
559
|
same run would report without `--filler` — and `removed_seconds_total` is the
|
|
552
560
|
additive sibling covering everything that went. A filler word quiet enough to
|
|
@@ -928,12 +936,38 @@ above the cap is clamped with a note, not refused, and both `jobs` and
|
|
|
928
936
|
- `--watch` composes: each pass is parallel, passes are sequential.
|
|
929
937
|
|
|
930
938
|
### caption.py --transcribe — optional local speech-to-text
|
|
931
|
-
If `
|
|
932
|
-
`
|
|
933
|
-
|
|
934
|
-
|
|
935
|
-
|
|
936
|
-
|
|
939
|
+
If `parakeet-mlx`, `parakeet-cli` (parakeet.cpp, with a `.gguf`), `whisper-cli` (whisper.cpp),
|
|
940
|
+
`faster-whisper` or `whisper` is installed,
|
|
941
|
+
`caption.py input.mp4 --transcribe [--language ja] [--model base] [--engine auto]` writes the
|
|
942
|
+
SRT from the audio and burns it (combine with `--animate pop --karaoke`). `--engine auto` (default,
|
|
943
|
+
or `FFMPEG_SKILL_ASR_ENGINE`) runs Parakeet for English speech -- an English `--language`, else
|
|
944
|
+
English from whisper.cpp's language detector -- and Whisper for any other language. When neither
|
|
945
|
+
can name the language, a Whisper engine that can run goes first; Parakeet runs on English
|
|
946
|
+
*assumed* only when no Whisper engine is installed or every one failed, and the result's
|
|
947
|
+
top-level `notes` then says so: that transcript is wrong for non-English speech, so pass `--language`
|
|
948
|
+
(`--filler-lang` for `silence.py`) when the speech may not be English.
|
|
949
|
+
`--engine parakeet-mlx|parakeet.cpp|whisper.cpp|faster-whisper|openai-whisper` forces one (a
|
|
950
|
+
missing one is `kind: missing_tool`; an installed one that fails is `kind: input`, `reason:
|
|
951
|
+
engine_failed`, with its own error line in `detail`). Without a language whisper.cpp is passed
|
|
952
|
+
`-l auto` (its own default is English), else the named or detected language. `--model` is the
|
|
953
|
+
whisper model; a Parakeet engine takes it only when it names a Parakeet model
|
|
954
|
+
(`mlx-community/parakeet-*` or a `.gguf`), else `PARAKEET_MODEL` / `PARAKEET_CPP_MODEL`, else the
|
|
955
|
+
English `tdt-0.6b-v2`. A non-English `--language` on an English-only Parakeet model is refused; a
|
|
956
|
+
multilingual (v3) model assumes no language, so it gets no English-assumed note. The result's
|
|
957
|
+
`transcription` names the engine, model and routing. `silence.py --filler --transcribe` takes
|
|
958
|
+
the same `--engine`.
|
|
959
|
+
Nothing is required: without an engine it prints install hints and the user can supply `--text`
|
|
960
|
+
cues instead. When engines are installed but none transcribed, the refusal names each and why
|
|
961
|
+
instead (`reason: engine_failed`, or `english_only` for a Parakeet engine auto passed over for
|
|
962
|
+
another language). The skill's own code downloads nothing, and every engine transcribes on this
|
|
963
|
+
machine -- the audio is never uploaded. An engine may fetch its *model weights* the
|
|
964
|
+
first time it runs: parakeet-mlx downloads `mlx-community/parakeet-tdt-0.6b-v2` (or the `--model` /
|
|
965
|
+
`PARAKEET_MODEL` repo) from Hugging Face into its cache; faster-whisper (also from Hugging Face)
|
|
966
|
+
and openai-whisper (from OpenAI) fetch theirs on first use too. faster-whisper is a library that
|
|
967
|
+
runs inside the skill's own process, so its download is made from that process.
|
|
968
|
+
`HF_HUB_OFFLINE=1` keeps the Hugging Face ones on the cached copy afterwards. parakeet.cpp and
|
|
969
|
+
whisper.cpp only read a model file you downloaded. Always tell the user which engine was used,
|
|
970
|
+
and treat the transcript as a draft to review.
|
|
937
971
|
|
|
938
972
|
### MCP server — the toolkit for any MCP client
|
|
939
973
|
`python3 mcp/server.py` speaks MCP over stdio; each script is a tool taking
|