kinescript 1.1.1__tar.gz → 1.3.0__tar.gz

{kinescript-1.1.1 → kinescript-1.3.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: kinescript
-Version: 1.1.1
-Summary: Render motion graphics and video edits from JSON scenes on the GPU, with checks an agent can read.
+Version: 1.3.0
+Summary: Render motion graphics, video edits and voice-over from JSON scenes on the GPU, with checks an agent can read.
 Author: Dr. Know How
 License-Expression: MIT
 Project-URL: Homepage, https://drknowhow.github.io/kinescript/
@@ -9,7 +9,7 @@ Project-URL: Documentation, https://drknowhow.github.io/kinescript/
 Project-URL: Source, https://github.com/drknowhow/kinescript
 Project-URL: Issues, https://github.com/drknowhow/kinescript/issues
 Project-URL: Changelog, https://github.com/drknowhow/kinescript/blob/main/CHANGELOG.md
-Keywords: video,video rendering,video editing,programmatic video,motion graphics,animation,json,skia,gpu,ffmpeg,3d,captions,mcp,ai agents
+Keywords: video,video rendering,video editing,programmatic video,motion graphics,animation,json,skia,gpu,ffmpeg,3d,captions,voice-over,text to speech,audio mixing,loudness,mcp,ai agents
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
 Classifier: Environment :: GPU
@@ -25,6 +25,7 @@ Classifier: Programming Language :: Python :: 3.14
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Topic :: Artistic Software
 Classifier: Topic :: Multimedia :: Graphics
+Classifier: Topic :: Multimedia :: Sound/Audio :: Mixers
 Classifier: Topic :: Multimedia :: Video
 Classifier: Topic :: Multimedia :: Video :: Conversion
 Classifier: Topic :: Multimedia :: Video :: Non-Linear Editor
@@ -46,16 +47,16 @@ Dynamic: license-file
 
 <p align="center"><img src="https://raw.githubusercontent.com/drknowhow/kinescript/main/brand/logo-dark.png" width="460" alt="kinescript"></p>
 
-<p align="center"><b>Video you can diff.</b><br>Motion graphics and video edits written as JSON, checked by agents, rendered on the GPU.</p>
+<p align="center"><b>Video you can diff.</b><br>Motion graphics, video edits and voice-over written as JSON, checked by agents, rendered on the GPU.</p>
 
 <p align="center"><a href="https://drknowhow.github.io/kinescript/">Documentation</a> · <a href="https://github.com/drknowhow/kinescript">Source</a> · <a href="https://github.com/drknowhow/kinescript/blob/main/CHANGELOG.md">Changelog</a> · <a href="https://github.com/drknowhow/kinescript/blob/main/docs/promo.mp4">Launch video</a></p>
 
 <p align="center"><img src="https://raw.githubusercontent.com/drknowhow/kinescript/main/docs/hero.gif" width="720" alt="The Kinescript launch video, rendered by Kinescript"></p>
 
-Kinescript turns a JSON scene into video. Every layer, keyframe and cut is a line of data, so a scene can be
-generated, reviewed in a pull request, and changed by a script or an agent. The renderer is a pure function of
-the frame number: any frame renders on its own, which makes previews instant and lets a render split across
-processes.
+Kinescript turns a JSON scene into video. Every layer, keyframe, cut and sound is a line of data, so a scene can
+be generated, reviewed in a pull request, and changed by a script or an agent. A voice-over can start as a line of
+text. The renderer is a pure function of the frame number: any frame renders on its own, which makes previews
+instant and lets a render split across processes.
 
 ```bash
 pip install kinescript
@@ -66,7 +67,7 @@ It needs Python 3.10 or newer and ffmpeg on your `PATH` ([details](https://githu
 
 The [launch video](https://github.com/drknowhow/kinescript/blob/main/docs/promo.mp4) is a Kinescript scene,
 [`marketing/promo.json`](https://github.com/drknowhow/kinescript/blob/main/marketing/promo.json). It is 35
-seconds of 1080p and renders, with audio, in about 3.5 seconds on the development machine. The
+seconds of 1080p and renders, with audio, in about 3.3 seconds on the development machine. The
 [project site](https://drknowhow.github.io/kinescript/) shows more demos, each next to the scene file it was
 rendered from.
 
@@ -83,7 +84,10 @@ rendered from.
 - **Footage.** Video layers with trim, speed, loop, cover/contain fit, punch-in `zoom` around a `focus` point, and
   rounded picture-in-picture. A `sequence` turns a cut list into clips with fade, wipe, slide, zoom or blur
   transitions. Clip audio and music beds are mixed automatically.
-- **Captions** from `.srt`/`.vtt` or inline cues, with karaoke and pop modes.
+- **Sound on the same timeline.** Voice-over, music and effects are cues like layers, a voice's words are cues
+  too, a voice can be spoken from text, and sounds with a role are levelled and ducked for you
+  ([Sound](https://github.com/drknowhow/kinescript#sound)).
+- **Captions** from `.srt`/`.vtt`, inline cues or a voice's word times, with karaoke and pop modes.
 - **Shaders.** SkSL layers with `mesh`, `aurora`, `grid` and `score` presets, or your own source.
 - **Finishing.** Per-layer blur, shadow, blend modes and colour adjustments; a scene grade with vignette and film
   grain; optional motion blur.
@@ -118,6 +122,60 @@ A 3D title that builds in and turns to face the camera:
  "shine": 0.6, "enter": {"name": "build", "dur": 1.2}}
 ```
 
+## Sound
+
+Sounds are a top-level `audio` list, timed like layers and mixed by ffmpeg while the frames render.
+
+- **Cues both ways.** `"at": "vo1.out+0.3"` starts a layer 0.3 s after the voice ends, and a sound can start at
+  `"title.in"`. A layer's `"sound": "pop.wav"` plays when it appears and moves when it moves. A voice stretches an
+  `auto` duration so it is never cut off.
+- **Words are cues.** A words file next to a voice (`vo1.words.json`, or Whisper's JSON as it is) times each
+  word: `"at": "vo1:render"` starts a layer as the voice says "render", and `{"type": "captions", "voice": "vo1"}`
+  captions it, lighting each word as it is said.
+- **Voice-over from text.** Write what a voice says in `text` and `kine voice` speaks it through a plugin, writing
+  the file and its word times: [Kokoro](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-kokoro)
+  (local, fast),
+  [Chatterbox](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-chatterbox) (local, tags,
+  cloned voices) or [ElevenLabs](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-elevenlabs)
+  (cloud, needs an API key), picked per sound with `backend`. `[[go]]` in the text names the next
+  word, so a cue to `vo1:go` survives a rewrite of the line. Only a line whose text, speaker, rate, backend or
+  options changed is spoken again, and rendering never calls a speech engine.
+- **Delivery tags.** `[whispers]`, `[laughs]` or `[sighs]` in the text tell Chatterbox and ElevenLabs how to say
+  what follows. Tags are not words, so cues, marks and captions skip them, and Kokoro says the line without them.
+- **A mix without listening.** Give a sound a `role` and it is set to that role's level: voice -18 LUFS, music
+  -16, ambience -30, effects peaking at -6 dBFS. Music and ambience dip 14 dB while a voice speaks, an effect's
+  leading silence is skipped so it lands on its cue, and the whole mix ends at -16 LUFS with peaks under
+  -1.5 dBTP. A top-level `mix` changes the targets. Sounds without a role are mixed as given.
+- **Checked and drawn.** `kine check` mixes and measures the sound and reports a voice cut off or masked by the
+  music, an effect that lands late, silence, and a mix off its targets. `kine look --audio` draws the sound under
+  the frames on the same time axis.
+- **Everywhere.** Gain takes keyframes and expressions like any property; sounds loop and pan; the mix goes into
+  MP4, MOV, WebM and `--from/--to` renders, and `-o mix.wav` renders the sound alone.
+
+A narrated beat with a click on a marked word, over music that ducks under the voice:
+
+```json
+"audio": [
+  {"id": "vo", "role": "voice", "at": 0.6, "text": "Each word it says is a [[hit]]cue."},
+  {"id": "bed", "role": "music", "src": "bed.m4a", "loop": true, "until": "end", "fadeOut": 1.5}
+],
+"layers": [
+  {"id": "cue", "type": "text", "text": "Cue", "size": 200, "at": "vo:hit", "sound": "click.wav", "enter": "pop"},
+  {"id": "cc", "type": "captions", "voice": "vo", "mode": "karaoke"}
+]
+```
+
+`kine voice scene.json` speaks it once. The
+[voice demo](https://drknowhow.github.io/kinescript/#sound) on the project site plays the result with sound, next
+to its scene file. The Kokoro plugin needs Python 3.10 to 3.12:
+
+```bash
+pip install kinescript-kokoro
+```
+
+On a newer Python, install it into another one and point `KINE_VOICE_CMD` at it
+([details](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-kokoro)).
+
 ## Built for agents
 
 An agent cannot watch a video, so Kinescript gives it things it can read:
@@ -127,18 +185,23 @@ An agent cannot watch a video, so Kinescript gives it things it can read:
 | `kine docs` | the full scene reference, short enough to keep in context |
 | `kine check scene.json` | every problem at once as JSON, each with its layer, time range and a fix |
 | `kine look scene.json --boxes` | one PNG of many labelled frames, with every layer outlined and the top-level ones named |
+| `kine look scene.json --audio` | the same frames with the sound under them: a lane per role and the mix, the duck and issues, plus each sound's measured loudness |
 | `kine probe scene.json --at 3.2` | resolved in/out of every layer and each visible layer's screen box |
 | `kine still` / `kine render` | a frame, a range (`--from/--to`), a draft (`-q draft --scale 0.5`) or the final file |
 | `kine watch scene.json` | the contact sheet, made again every time the file is saved |
+| `kine voice scene.json` | each sound with text spoken into its file, with its word times, skipping what has not changed |
 
 `kine check` reports unknown keys and bad values, missing media and out-of-range trims, text off the canvas or
 in the safe margin, overlapping text, contrast under 3:1 against what is actually behind the text, tiny text and
-too little reading time. An unknown key names the intended key first. Common names from other tools (`color`,
+too little reading time. When the scene has sound it mixes and measures it, without drawing a frame, while the
+frames are checked, and reports a voice cut off by the scene end, two voices at once, music less than 10 LU under a
+voice, an effect whose first sound comes late, a silent sound, and a mix off its loudness target or at its peak
+ceiling. An unknown key names the intended key first. Common names from other tools (`color`,
 `fontSize`, `duration`, `position`) are read and reported as info notes. It exits with status 1 while there are
 errors.
 
 The same tools are available over MCP (`kine mcp`) as `kine_docs`, `kine_check`, `kine_look`, `kine_still`,
-`kine_probe` and `kine_render`. They take a scene as a file path or as the JSON object itself, and `kine_look`
+`kine_probe`, `kine_render` and `kine_voice`. They take a scene as a file path or as the JSON object itself, and `kine_look`
 and `kine_still` return images inline.
 
 ```json
@@ -205,8 +268,13 @@ More scenes in [`examples/`](https://github.com/drknowhow/kinescript/tree/main/e
   extruded SVG path and the wordmark as extruded text, 5 s.
 - [`import.json`](https://github.com/drknowhow/kinescript/blob/main/examples/import.json): a glTF rocket on a
   ground shadow, 6 s.
+- [`voiceover.json`](https://github.com/drknowhow/kinescript/blob/main/examples/voiceover.json): titles cued from
+  the words of a voice-over, karaoke captions of it, and a music bed that dips under it, 5.4 s.
+- [`narrated.json`](https://github.com/drknowhow/kinescript/blob/main/examples/narrated.json): the same idea
+  with a voice written as text, a marked word and a click on it. Run `kine voice examples/narrated.json` first.
 
-`edit.json` and `import.json` read sample media: run `python examples/make_media.py` first to generate it.
+`edit.json`, `import.json`, `voiceover.json` and `narrated.json` read sample media: run
+`python examples/make_media.py` first to generate it.
 
 ## Performance
 
@@ -215,9 +283,9 @@ Measured on the development machine (Intel Core Ultra 7 265, 20 threads, NVIDIA
 
 | Workload | Result |
 |---|---|
-| Drawing one 1080p frame of the launch video on the GPU, converted to I420 and read back as a render does | median 2.7 ms, p95 3.3 ms (all 1038 frames) |
-| Launch video, 34.6 s at 1080p30, `-q high`, with audio | 3.5 s wall, about 300 frames/s (median of 5 runs) |
-| `examples/edit.json`, 10.4 s with two decoded video clips | 2.05 s wall, 152 frames/s (median of 5 runs) |
+| Drawing one 1080p frame of the launch video on the GPU, converted to I420 and read back as a render does | median 2.8 ms, p95 3.8 ms (all 1038 frames) |
+| Launch video, 34.6 s at 1080p30, `-q high`, with audio | 3.3 s wall, about 320 frames/s (median of 5 runs) |
+| `examples/edit.json`, 10.4 s with two decoded video clips | 2.25 s wall, 139 frames/s (median of 5 runs) |
 
 [`python bench/stages.py`](https://github.com/drknowhow/kinescript/blob/main/bench/stages.py) splits the time
 per frame into drawing, conversion and readback, and writing to ffmpeg for several workloads, and times whole
@@ -235,7 +303,11 @@ Within one process, such as the MCP server, the workers stay warm for two minute
 - 3D is drawn by Skia with a depth sort, not a GL pipeline: models have no image textures, skeletal animation or
   shadows cast on each other, and separate model layers do not hide each other per pixel (merge touching
   objects as `parts` of one model). Draco- and meshopt-compressed model files are not read.
-- Audio is mixing (trim, gain, fades, speed), not editing or effects.
+- Audio is mixing and levelling (trim, gain over time, fades, speed, loop, pan, role levels, ducking, a loudness
+  target), with no effects such as EQ, compression of single sounds or reverb.
+- Kinescript does not speak by itself: `kine voice` needs a plugin or a command. The Kokoro plugin times words
+  for English voices only and needs Python 3.10 to 3.12; the Chatterbox plugin speaks and times English only;
+  the ElevenLabs plugin needs an API key and a network connection.
 - Video is decoded and resampled by ffmpeg to the scene frame rate. Jumping backwards or far ahead reopens the
   decoder, which is slower than sequential playback.
 - Text layout is single-script left-to-right with greedy wrapping. No shaping for complex scripts or bidi.
@@ -248,6 +320,8 @@ kinescript/   engine: timing, props, scene, components, render (Skia), space, re
               mesh and models (3D), output (encoding and the audio mix),
               verify (check, look, probe), schema, docs, mcp, cli
 examples/     example scenes and the generator for sample media
+plugins/      voices for kine voice, each installed on its own: kinescript-kokoro, kinescript-chatterbox,
+              kinescript-elevenlabs
 marketing/    the launch video as a scene and its builder; build_site.py renders the site
 site/         the project site: demo scenes next to the videos rendered from them
 brand/        logo files, generated art and the brand guide

{kinescript-1.1.1 → kinescript-1.3.0}/README.md

@@ -1,15 +1,15 @@
 <p align="center"><img src="https://raw.githubusercontent.com/drknowhow/kinescript/main/brand/logo-dark.png" width="460" alt="kinescript"></p>
 
-<p align="center"><b>Video you can diff.</b><br>Motion graphics and video edits written as JSON, checked by agents, rendered on the GPU.</p>
+<p align="center"><b>Video you can diff.</b><br>Motion graphics, video edits and voice-over written as JSON, checked by agents, rendered on the GPU.</p>
 
 <p align="center"><a href="https://drknowhow.github.io/kinescript/">Documentation</a> · <a href="https://github.com/drknowhow/kinescript">Source</a> · <a href="https://github.com/drknowhow/kinescript/blob/main/CHANGELOG.md">Changelog</a> · <a href="https://github.com/drknowhow/kinescript/blob/main/docs/promo.mp4">Launch video</a></p>
 
 <p align="center"><img src="https://raw.githubusercontent.com/drknowhow/kinescript/main/docs/hero.gif" width="720" alt="The Kinescript launch video, rendered by Kinescript"></p>
 
-Kinescript turns a JSON scene into video. Every layer, keyframe and cut is a line of data, so a scene can be
-generated, reviewed in a pull request, and changed by a script or an agent. The renderer is a pure function of
-the frame number: any frame renders on its own, which makes previews instant and lets a render split across
-processes.
+Kinescript turns a JSON scene into video. Every layer, keyframe, cut and sound is a line of data, so a scene can
+be generated, reviewed in a pull request, and changed by a script or an agent. A voice-over can start as a line of
+text. The renderer is a pure function of the frame number: any frame renders on its own, which makes previews
+instant and lets a render split across processes.
 
 ```bash
 pip install kinescript
@@ -20,7 +20,7 @@ It needs Python 3.10 or newer and ffmpeg on your `PATH` ([details](https://githu
 
 The [launch video](https://github.com/drknowhow/kinescript/blob/main/docs/promo.mp4) is a Kinescript scene,
 [`marketing/promo.json`](https://github.com/drknowhow/kinescript/blob/main/marketing/promo.json). It is 35
-seconds of 1080p and renders, with audio, in about 3.5 seconds on the development machine. The
+seconds of 1080p and renders, with audio, in about 3.3 seconds on the development machine. The
 [project site](https://drknowhow.github.io/kinescript/) shows more demos, each next to the scene file it was
 rendered from.
 
@@ -37,7 +37,10 @@ rendered from.
 - **Footage.** Video layers with trim, speed, loop, cover/contain fit, punch-in `zoom` around a `focus` point, and
   rounded picture-in-picture. A `sequence` turns a cut list into clips with fade, wipe, slide, zoom or blur
   transitions. Clip audio and music beds are mixed automatically.
-- **Captions** from `.srt`/`.vtt` or inline cues, with karaoke and pop modes.
+- **Sound on the same timeline.** Voice-over, music and effects are cues like layers, a voice's words are cues
+  too, a voice can be spoken from text, and sounds with a role are levelled and ducked for you
+  ([Sound](https://github.com/drknowhow/kinescript#sound)).
+- **Captions** from `.srt`/`.vtt`, inline cues or a voice's word times, with karaoke and pop modes.
 - **Shaders.** SkSL layers with `mesh`, `aurora`, `grid` and `score` presets, or your own source.
 - **Finishing.** Per-layer blur, shadow, blend modes and colour adjustments; a scene grade with vignette and film
   grain; optional motion blur.
@@ -72,6 +75,60 @@ A 3D title that builds in and turns to face the camera:
  "shine": 0.6, "enter": {"name": "build", "dur": 1.2}}
 ```
 
+## Sound
+
+Sounds are a top-level `audio` list, timed like layers and mixed by ffmpeg while the frames render.
+
+- **Cues both ways.** `"at": "vo1.out+0.3"` starts a layer 0.3 s after the voice ends, and a sound can start at
+  `"title.in"`. A layer's `"sound": "pop.wav"` plays when it appears and moves when it moves. A voice stretches an
+  `auto` duration so it is never cut off.
+- **Words are cues.** A words file next to a voice (`vo1.words.json`, or Whisper's JSON as it is) times each
+  word: `"at": "vo1:render"` starts a layer as the voice says "render", and `{"type": "captions", "voice": "vo1"}`
+  captions it, lighting each word as it is said.
+- **Voice-over from text.** Write what a voice says in `text` and `kine voice` speaks it through a plugin, writing
+  the file and its word times: [Kokoro](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-kokoro)
+  (local, fast),
+  [Chatterbox](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-chatterbox) (local, tags,
+  cloned voices) or [ElevenLabs](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-elevenlabs)
+  (cloud, needs an API key), picked per sound with `backend`. `[[go]]` in the text names the next
+  word, so a cue to `vo1:go` survives a rewrite of the line. Only a line whose text, speaker, rate, backend or
+  options changed is spoken again, and rendering never calls a speech engine.
+- **Delivery tags.** `[whispers]`, `[laughs]` or `[sighs]` in the text tell Chatterbox and ElevenLabs how to say
+  what follows. Tags are not words, so cues, marks and captions skip them, and Kokoro says the line without them.
+- **A mix without listening.** Give a sound a `role` and it is set to that role's level: voice -18 LUFS, music
+  -16, ambience -30, effects peaking at -6 dBFS. Music and ambience dip 14 dB while a voice speaks, an effect's
+  leading silence is skipped so it lands on its cue, and the whole mix ends at -16 LUFS with peaks under
+  -1.5 dBTP. A top-level `mix` changes the targets. Sounds without a role are mixed as given.
+- **Checked and drawn.** `kine check` mixes and measures the sound and reports a voice cut off or masked by the
+  music, an effect that lands late, silence, and a mix off its targets. `kine look --audio` draws the sound under
+  the frames on the same time axis.
+- **Everywhere.** Gain takes keyframes and expressions like any property; sounds loop and pan; the mix goes into
+  MP4, MOV, WebM and `--from/--to` renders, and `-o mix.wav` renders the sound alone.
+
+A narrated beat with a click on a marked word, over music that ducks under the voice:
+
+```json
+"audio": [
+  {"id": "vo", "role": "voice", "at": 0.6, "text": "Each word it says is a [[hit]]cue."},
+  {"id": "bed", "role": "music", "src": "bed.m4a", "loop": true, "until": "end", "fadeOut": 1.5}
+],
+"layers": [
+  {"id": "cue", "type": "text", "text": "Cue", "size": 200, "at": "vo:hit", "sound": "click.wav", "enter": "pop"},
+  {"id": "cc", "type": "captions", "voice": "vo", "mode": "karaoke"}
+]
+```
+
+`kine voice scene.json` speaks it once. The
+[voice demo](https://drknowhow.github.io/kinescript/#sound) on the project site plays the result with sound, next
+to its scene file. The Kokoro plugin needs Python 3.10 to 3.12:
+
+```bash
+pip install kinescript-kokoro
+```
+
+On a newer Python, install it into another one and point `KINE_VOICE_CMD` at it
+([details](https://github.com/drknowhow/kinescript/tree/main/plugins/kinescript-kokoro)).
+
 ## Built for agents
 
 An agent cannot watch a video, so Kinescript gives it things it can read:
@@ -81,18 +138,23 @@ An agent cannot watch a video, so Kinescript gives it things it can read:
 | `kine docs` | the full scene reference, short enough to keep in context |
 | `kine check scene.json` | every problem at once as JSON, each with its layer, time range and a fix |
 | `kine look scene.json --boxes` | one PNG of many labelled frames, with every layer outlined and the top-level ones named |
+| `kine look scene.json --audio` | the same frames with the sound under them: a lane per role and the mix, the duck and issues, plus each sound's measured loudness |
 | `kine probe scene.json --at 3.2` | resolved in/out of every layer and each visible layer's screen box |
 | `kine still` / `kine render` | a frame, a range (`--from/--to`), a draft (`-q draft --scale 0.5`) or the final file |
 | `kine watch scene.json` | the contact sheet, made again every time the file is saved |
+| `kine voice scene.json` | each sound with text spoken into its file, with its word times, skipping what has not changed |
 
 `kine check` reports unknown keys and bad values, missing media and out-of-range trims, text off the canvas or
 in the safe margin, overlapping text, contrast under 3:1 against what is actually behind the text, tiny text and
-too little reading time. An unknown key names the intended key first. Common names from other tools (`color`,
+too little reading time. When the scene has sound it mixes and measures it, without drawing a frame, while the
+frames are checked, and reports a voice cut off by the scene end, two voices at once, music less than 10 LU under a
+voice, an effect whose first sound comes late, a silent sound, and a mix off its loudness target or at its peak
+ceiling. An unknown key names the intended key first. Common names from other tools (`color`,
 `fontSize`, `duration`, `position`) are read and reported as info notes. It exits with status 1 while there are
 errors.
 
 The same tools are available over MCP (`kine mcp`) as `kine_docs`, `kine_check`, `kine_look`, `kine_still`,
-`kine_probe` and `kine_render`. They take a scene as a file path or as the JSON object itself, and `kine_look`
+`kine_probe`, `kine_render` and `kine_voice`. They take a scene as a file path or as the JSON object itself, and `kine_look`
 and `kine_still` return images inline.
 
 ```json
@@ -159,8 +221,13 @@ More scenes in [`examples/`](https://github.com/drknowhow/kinescript/tree/main/e
   extruded SVG path and the wordmark as extruded text, 5 s.
 - [`import.json`](https://github.com/drknowhow/kinescript/blob/main/examples/import.json): a glTF rocket on a
   ground shadow, 6 s.
+- [`voiceover.json`](https://github.com/drknowhow/kinescript/blob/main/examples/voiceover.json): titles cued from
+  the words of a voice-over, karaoke captions of it, and a music bed that dips under it, 5.4 s.
+- [`narrated.json`](https://github.com/drknowhow/kinescript/blob/main/examples/narrated.json): the same idea
+  with a voice written as text, a marked word and a click on it. Run `kine voice examples/narrated.json` first.
 
-`edit.json` and `import.json` read sample media: run `python examples/make_media.py` first to generate it.
+`edit.json`, `import.json`, `voiceover.json` and `narrated.json` read sample media: run
+`python examples/make_media.py` first to generate it.
 
 ## Performance
 
@@ -169,9 +236,9 @@ Measured on the development machine (Intel Core Ultra 7 265, 20 threads, NVIDIA
 
 | Workload | Result |
 |---|---|
-| Drawing one 1080p frame of the launch video on the GPU, converted to I420 and read back as a render does | median 2.7 ms, p95 3.3 ms (all 1038 frames) |
-| Launch video, 34.6 s at 1080p30, `-q high`, with audio | 3.5 s wall, about 300 frames/s (median of 5 runs) |
-| `examples/edit.json`, 10.4 s with two decoded video clips | 2.05 s wall, 152 frames/s (median of 5 runs) |
+| Drawing one 1080p frame of the launch video on the GPU, converted to I420 and read back as a render does | median 2.8 ms, p95 3.8 ms (all 1038 frames) |
+| Launch video, 34.6 s at 1080p30, `-q high`, with audio | 3.3 s wall, about 320 frames/s (median of 5 runs) |
+| `examples/edit.json`, 10.4 s with two decoded video clips | 2.25 s wall, 139 frames/s (median of 5 runs) |
 
 [`python bench/stages.py`](https://github.com/drknowhow/kinescript/blob/main/bench/stages.py) splits the time
 per frame into drawing, conversion and readback, and writing to ffmpeg for several workloads, and times whole
@@ -189,7 +256,11 @@ Within one process, such as the MCP server, the workers stay warm for two minute
 - 3D is drawn by Skia with a depth sort, not a GL pipeline: models have no image textures, skeletal animation or
   shadows cast on each other, and separate model layers do not hide each other per pixel (merge touching
   objects as `parts` of one model). Draco- and meshopt-compressed model files are not read.
-- Audio is mixing (trim, gain, fades, speed), not editing or effects.
+- Audio is mixing and levelling (trim, gain over time, fades, speed, loop, pan, role levels, ducking, a loudness
+  target), with no effects such as EQ, compression of single sounds or reverb.
+- Kinescript does not speak by itself: `kine voice` needs a plugin or a command. The Kokoro plugin times words
+  for English voices only and needs Python 3.10 to 3.12; the Chatterbox plugin speaks and times English only;
+  the ElevenLabs plugin needs an API key and a network connection.
 - Video is decoded and resampled by ffmpeg to the scene frame rate. Jumping backwards or far ahead reopens the
   decoder, which is slower than sequential playback.
 - Text layout is single-script left-to-right with greedy wrapping. No shaping for complex scripts or bidi.
@@ -202,6 +273,8 @@ kinescript/   engine: timing, props, scene, components, render (Skia), space, re
               mesh and models (3D), output (encoding and the audio mix),
               verify (check, look, probe), schema, docs, mcp, cli
 examples/     example scenes and the generator for sample media
+plugins/      voices for kine voice, each installed on its own: kinescript-kokoro, kinescript-chatterbox,
+              kinescript-elevenlabs
 marketing/    the launch video as a scene and its builder; build_site.py renders the site
 site/         the project site: demo scenes next to the videos rendered from them
 brand/        logo files, generated art and the brand guide

{kinescript-1.1.1 → kinescript-1.3.0}/kinescript/__init__.py

@@ -9,5 +9,5 @@ except ImportError as e:  # on Linux the wheel loads the system's EGL and GL, ev
 from .output import render, still
 from .scene import Scene, SceneError, load
 
-__version__ = "1.1.1"
+__version__ = "1.3.0"
 __all__ = ["Scene", "SceneError", "load", "render", "still", "__version__"]

kinescript-1.3.0/kinescript/audio.py

@@ -0,0 +1,139 @@
+"""The managed mix, used when any sound in a scene has a role: each sound is set to its role's level, music and
+ambience dip while a voice speaks, and the whole mix is brought to one loudness under a peak ceiling.
+
+Every gain here is worked out from the files and the scene, never by listening at render time, so a scene always
+mixes the same way and the numbers can be reported to an agent that cannot hear.
+"""
+from __future__ import annotations
+
+import math
+import re
+import subprocess
+from functools import lru_cache
+
+import numpy as np
+
+from . import media
+from .props import Ctx, compile_prop
+
+ROLES = ("voice", "music", "sfx", "ambience")
+# voice, music and ambience in LUFS; sfx as a sample peak in dBFS, since a short effect has no useful loudness.
+LEVELS = {"voice": -18.0, "music": -16.0, "ambience": -30.0, "sfx": -6.0}
+DUCK = {"by": ["voice"], "under": ["music", "ambience"], "depth": 14.0, "attack": 0.08, "hold": 0.15, "release": 0.4}
+LOUDNESS, PEAK = -16.0, -1.5  # the whole mix: LUFS, and dBTP
+RATE = 1000  # gain curve samples per second
+_AN = 8000  # analysis sample rate
+_WIN = 0.02  # RMS window, seconds
+SPEECH = -45.0  # dBFS: a voice window louder than this, after levelling, ducks the music
+
+
+def settings(spec: dict) -> dict:
+    """The scene's "mix" block over the defaults."""
+    m = spec.get("mix") or {}
+    duck = m.get("duck", {})
+    if duck is not False:
+        duck = dict(DUCK, **(duck if isinstance(duck, dict) else {}))
+        for k in ("by", "under"):
+            duck[k] = [duck[k]] if isinstance(duck[k], str) else list(duck[k])
+    return {"loudness": float(m.get("loudness", LOUDNESS)), "peak": float(m.get("peak", PEAK)),
+            "levels": dict(LEVELS, **(m.get("levels") or {})), "duck": duck or None}
+
+
+def analyze(src: str, start: float = 0.0, end: float | None = None) -> dict:
+    """Loudness and shape of src between start and end (seconds in the file), measured once per file version:
+    lufs, peak (sample, dBFS), true_peak (dBTP), power (mean, dBFS), rms (dBFS per 20 ms window) and lead
+    (seconds of silence before the first sound)."""
+    return _analyze(*media._version(src), round(float(start), 4), None if end is None else round(float(end), 4))
+
+
+@lru_cache(maxsize=256)
+def _analyze(src: str, mtime: int, size: int, start: float, end: float | None) -> dict:
+    return measure(src, start, end)
+
+
+def measure(src: str, start: float = 0.0, end: float | None = None) -> dict:
+    """analyze without the cache, for a file that is made once and measured once, like a finished mix."""
+    cut = f"atrim=start={start}" + ("" if end is None else f":end={end}") + ",asetpts=PTS-STARTPTS,"
+    res = subprocess.run([media.FFMPEG, "-hide_banner", "-nostats", "-i", src, "-af",
+                          cut + f"ebur128=peak=true+sample,aresample={_AN},aformat=sample_fmts=flt:channel_layouts=mono",
+                          "-f", "f32le", "-"], capture_output=True, creationflags=media._NOWIN)
+    if res.returncode:
+        raise media.MediaError(f"could not measure {src}: {res.stderr.decode(errors='replace').strip()[-300:]}")
+    info = loudness(res.stderr.decode(errors="replace"))
+    pcm = np.frombuffer(res.stdout, np.float32).astype(np.float64)
+    w = int(_AN * _WIN)
+    n = len(pcm) // w
+    info["rms"] = 10 * np.log10(np.mean(pcm[: n * w].reshape(n, w) ** 2, axis=1) + 1e-12)
+    info["power"] = float(10 * np.log10(np.mean(pcm ** 2) + 1e-12)) if len(pcm) else -math.inf
+    floor = 10 ** (min(-60.0, info["peak"] - 50) / 20) if math.isfinite(info["peak"]) else 1.0
+    loud = np.abs(pcm) > floor
+    info["lead"] = float(np.argmax(loud)) / _AN if loud.any() else 0.0
+    return info
+
+
+def loudness(log: str) -> dict:
+    """lufs, peak, true_peak and lra (loudness range, LU) from the summary ffmpeg's ebur128 filter writes to its
+    log, and short_max, the loudest 3 s, from the lines it writes every 0.1 s."""
+    at = log.rfind("Summary:")
+    s = log[at:]
+
+    def num(pattern: str) -> float:
+        m = re.search(pattern, s)
+        return float(m.group(1)) if m else -math.inf
+    short = [float(x) for x in re.findall(r"\bS:\s*(-?[\d.]+)", log[:at])]
+    return {"lufs": num(r"I:\s+(-?[\d.]+|-inf) LUFS"), "peak": num(r"Sample peak:\s+Peak:\s+(-?[\d.]+|-inf)"),
+            "true_peak": num(r"True peak:\s+Peak:\s+(-?[\d.]+|-inf)"), "lra": num(r"LRA:\s+(-?[\d.]+) LU"),
+            "short_max": max((x for x in short if x > -70), default=-math.inf)}
+
+
+def level(role: str, info: dict, levels: dict) -> float:
+    """dB that sets a sound to its role's level, between -40 and +24; 0 for a silent file."""
+    if role == "sfx":
+        ref = info["peak"]
+    else:  # under 0.4 s ebur128 reports -70 LUFS: use the mean power instead
+        ref = info["lufs"] if info["lufs"] > -69.9 else info["power"]
+    if not math.isfinite(ref) or ref < -90:
+        return 0.0
+    return max(-40.0, min(24.0, levels[role] - ref))
+
+
+def correction(measured: dict, target: float) -> float:
+    """dB that brings the whole mix to the target loudness, between -20 and +12; 0 for a silent mix."""
+    i = measured["lufs"]
+    return max(-20.0, min(12.0, target - i)) if i > -69.9 else 0.0
+
+
+def duck_curve(voices: list[tuple[float, float, np.ndarray, float]], duration: float, cfg: dict) -> np.ndarray:
+    """Linear gain at RATE over the scene, plus a spare second, for the sounds that duck. It is 1, and -depth dB
+    while a voice speaks: the fall starts `attack` seconds before the voice, so the music is already down for the
+    first syllable; pauses shorter than `hold` stay down; the rise takes `release` seconds.
+    voices: (start, end, RMS in dBFS per 20 ms window from the start, gain in dB applied to the voice)."""
+    n = int(math.ceil(duration * RATE)) + RATE
+    active = np.zeros(n, bool)
+    per = int(RATE * _WIN)
+    for t0, t1, rms, gain in voices:
+        on = np.repeat(rms + gain > SPEECH, per)
+        a, b = max(0, int(round(t0 * RATE))), min(n, int(round(t1 * RATE)))
+        m = min(b - a, len(on))
+        if m > 0:
+            active[a:a + m] |= on[:m]
+    hold = int(cfg["hold"] * RATE)
+    if hold:
+        active = np.convolve(active, np.ones(hold + 1), "full")[:n] > 0
+    depth = float(cfg["depth"])
+    d, idx = np.where(active, depth, 0.0), np.arange(n)
+    up = depth / max(1.0, cfg["release"] * RATE)
+    d = np.maximum.accumulate(d + up * idx) - up * idx  # after each stretch, rise over `release`
+    down = depth / max(1.0, cfg["attack"] * RATE)
+    d = (np.maximum.accumulate(d[::-1] + down * idx) - down * idx)[::-1]  # before each, fall over `attack`
+    return 10 ** (-np.clip(d, 0.0, depth) / 20)
+
+
+def gain_curve(raw, dur: float, start: float, fps: float) -> np.ndarray:
+    """A sound's keyframed or expression gain (dB, timed from the sound's start) as linear gain at RATE, plus a
+    spare second. Sampled at 100 Hz and interpolated: gain moves slowly next to the audio."""
+    prop = compile_prop("gain", raw, Ctx(1, 1, fps))
+    ts = np.linspace(0.0, dur, max(2, int(math.ceil(dur * 100)) + 1))
+    db = np.array([float(prop.at(t, dur, start + t, 0)) for t in ts])
+    n = int(math.ceil(dur * RATE)) + RATE
+    return 10 ** (np.interp(np.arange(n) / RATE, ts, db) / 20)

{kinescript-1.1.1 → kinescript-1.3.0}/kinescript/cli.py

@@ -30,7 +30,8 @@ def main(argv=None) -> int:
     ap.add_argument("--version", action="version", version=f"kinescript {__version__}")
     sub = ap.add_subparsers(dest="cmd", required=True)
 
-    r = sub.add_parser("render", help="render a scene to .mp4/.mov/.webm/.gif or a .png sequence")
+    r = sub.add_parser("render", help="render a scene to .mp4/.mov/.webm/.gif or a .png sequence, or its sound "
+                                      "alone to .wav/.m4a/.mp3/.flac")
     r.add_argument("scene")
     r.add_argument("-o", "--out")
     r.add_argument("-q", "--quality", choices=["draft", "standard", "high"], default="standard")
@@ -53,9 +54,11 @@ def main(argv=None) -> int:
     lk.add_argument("--width", type=int, default=480, help="width of each thumbnail")
     lk.add_argument("--at", type=float, nargs="*", help="explicit times instead of even spacing")
     lk.add_argument("--boxes", action="store_true", help="outline every layer with its id")
+    lk.add_argument("--audio", action="store_true", help="draw the sound under the frames (a lane per role and the "
+                    "mix, with levels, the duck and issues) and print its measurements")
     lk.add_argument("-o", "--out")
 
-    ck = sub.add_parser("check", help="lint layout, timing, contrast and reading time; prints JSON")
+    ck = sub.add_parser("check", help="lint layout, timing, contrast, reading time and sound; prints JSON")
     ck.add_argument("scene")
     ck.add_argument("--step", type=float, default=0.25)
 
@@ -64,6 +67,13 @@ def main(argv=None) -> int:
     pr.add_argument("scene")
     pr.add_argument("--at", type=float)
 
+    v = sub.add_parser("voice", help="speak each sound's text into its file and time its words (only the ones "
+                                     "whose text changed, unless --force)")
+    v.add_argument("scene")
+    v.add_argument("--only", nargs="+", metavar="ID", help="speak just these sounds")
+    v.add_argument("--force", action="store_true", help="speak them again even if their text did not change")
+    v.add_argument("--backend", help="which installed voice backend to use, when there are several")
+
     sub.add_parser("docs", help="print the scene reference")
     nw = sub.add_parser("new", help="write a starter scene")
     nw.add_argument("path", nargs="?", default="scene.json")
@@ -129,7 +139,11 @@ def _run(a) -> int:
     if a.cmd == "look":
         from .verify import contact_sheet
         out = a.out or _default_out(a.scene, "_sheet.png")
-        _emit(dict(ok=True, **contact_sheet(a.scene, out, a.count, a.cols, a.width, a.at, a.boxes)))
+        _emit(dict(ok=True, **contact_sheet(a.scene, out, a.count, a.cols, a.width, a.at, a.boxes, a.audio)))
+        return 0
+    if a.cmd == "voice":
+        from .voice import generate
+        _emit(generate(a.scene, a.only, a.force, a.backend))
         return 0
     if a.cmd == "check":
         from .verify import check

{kinescript-1.1.1 → kinescript-1.3.0}/kinescript/components.py

@@ -129,7 +129,7 @@ COMPONENT_DOCS = {
     "kinetic": "words [list], every [0.6] (seconds per word), size [180], motion [pop], color, x, y [\"50%\"] (centre)",
 }
 _PASS = ("id", "at", "dur", "until", "enter", "exit", "opacity", "scale", "rotate", "hidden", "z", "rotateX", "rotateY",
-         "fixed")
+         "fixed", "sound")
 
 
 class Params(dict):