keepitreel 0.1.0 → 0.1.2

package/README.md

@@ -196,31 +196,38 @@ Renders `start` to `end` of a clip as a 9:16 segment: the landscape picture sits
 |---|---|---|---|
 | `clipPath` | `string` | required | The source video |
 | `start`, `end` | `number` | required | Seconds into the clip |
-| `workDir` | `string` | required | Where the output and the text files ffmpeg reads go |
+| `workDir` | `string` | required | Where the output and the files ffmpeg reads go: the text, and the filter graph (`<name>.filter`), which keeps clips with hundreds of subtitle words within Windows' command-line limit |
 | `name` | `string` | required | The output's base name |
 | `subtitles` | `SubtitleChunk[]` | `[]` | From [`.subtitleChunks()`](#subtitlechunksoptions) |
-| `hook` | `{ band: { y, h, opacity }, lines, outSeconds }` | | A full-width ink band with text, up from the first frame, fading out `outSeconds` before the end |
+| `hook` | `{ band?: { y, h, opacity }, lines, outSeconds? }` | | Text up from the first frame, on a full-width ink band when `band` is given, fading out `outSeconds` before the end. Without `outSeconds` it stays to the end |
 | `tag` | `{ box: { x, y, w, h, opacity }, lines, delay?, outSeconds?, fade? }` | | An ink box with text that fades in after `delay` and out before the end. With `fade: false` it is up for the whole clip |
+| `fill` | `string` | `FILL_9_16` | The filter chain that turns the source picture into the 1080x1920 frame, for a layout of your own (for example the picture cropped and placed higher, over a darker blur) |
 | `hold` | `number` | `0` | Seconds the first frame is held, silent, before the clip plays. Subtitles and mute windows move with it |
 | `mute` | `[number, number][]` | `[]` | Clip-relative windows where the sound is replaced by a beep |
+| `loudness` | `number` | | Target loudness in LUFS, e.g. `-14`. The clip's sound is normalized to it before beeps are mixed in. Without it the sound is left as it is |
 
 `lines` are [text lines](#text-lines).
 
 #### .subtitleChunks(options)
 
-Splits clip-relative words into one-line, centered chunks, breaking at sentence ends. A sentence longer than `maxWords` is cut into the fewest pieces that fit, differing by at most one word ("I heard you / were in town."): seven words become 4 + 3, and twenty-one become 5 + 4 + 4 + 4 + 4 rather than leaving one word over. A piece is split further while it is wider than `maxWidth`, so a word too wide to share a line gets one to itself. Returns an array of `{ start, end, words }`, where each word is placed with its `x`, `font`, `color` and `size`.
+Splits clip-relative words into one-line, centered chunks, breaking at sentence ends. A sentence longer than `maxWords` is cut into the fewest pieces that fit, differing by at most one word ("I heard you / were in town."): seven words become 4 + 3, and twenty-one become 5 + 4 + 4 + 4 + 4 rather than leaving one word over. A piece is split further while it is wider than `maxWidth`, so a word too wide to share a line gets one to itself. Returns an array of `{ start, end, words }`, where each word is placed with its `x`, `font`, `color` and `size`, plus `active: { start, end, color }` when `activeColor` is given.
 
 | Option | Type | Default | |
 |---|---|---|---|
 | `words` | `{ text, start, end }[]` | required | Clip-relative seconds |
 | `duration` | `number` | required | Length of the clip. No chunk stays up past it |
 | `font`, `color` | `string` | required | A font key and an ffmpeg color |
-| `style` | `(word) => { font?, color?, text? }` | | Per-word font, color or text, for a highlighted or blanked-out word |
+| `style` | `(word) => { font?, color?, text?, activeColor? }` | | Per-word font, color or text, for a highlighted or blanked-out word. Its `activeColor` overrides the option for that word, and `null` keeps the word's own color while it is spoken |
+| `activeColor` | `string` | | Karaoke style: each word takes this color while it is spoken, from its start to the next word's start (the line's end for the last word), so Whisper's early end times leave no gap. Without it words keep their color the whole line |
 | `size` | `number` | `56` | |
 | `baseline` | `number` | `1445` | Below the picture of a 4:3 film and above the apps' bottom UI zone |
 | `maxWords` | `number` | `5` | |
 | `hold` | `number` | `0.6` | Seconds a chunk stays up after its last word when the next one is not right after |
 | `maxWidth` | `number` | `TEXT_WIDTH` | |
+| `breakAfter` | `RegExp` | `/[.?!]["']?$/` | Words that end a piece, as sentence ends do. `/[.,?!]["']?$/` also breaks after commas |
+| `pause` | `number` | | A silence longer than this many seconds between two words also ends a piece |
+| `border` | `{ width, color }` | 3 px, the renderer's ink at 85% | The outline around each word |
+| `shadow` | `{ x, y, color }` | | A shadow under each word, offset by `x` and `y` px |
 
 #### .concat(segments, workDir, outputPath)
 
@@ -250,6 +257,18 @@ Returns the width in pixels of `text` in the font with key `font` at `size`.
 
 Returns the largest size up to `size` at which `text` fits in `maxWidth` (default `TEXT_WIDTH`).
 
+#### .wrapText(text, font, size, options?)
+
+Returns `text` wrapped into lines no wider than `maxWidth` (default `TEXT_WIDTH`). With `balance: true` the lines are as even as the fewest lines allow, instead of full lines and a short last one.
+
+#### .printable(text, font)
+
+Returns `text` without the characters the font has no glyph for, such as emoji, with spaces collapsed. ffmpeg draws those as empty boxes.
+
+#### .capHeight(font, size)
+
+Returns the height of the font's capital letters at `size`. `baseline = top + capHeight` places text by the top of its capitals.
+
 #### .drawTexts(lines, out, workDir, name)
 
 Returns ffmpeg `drawtext` filters for [text lines](#text-lines), for filter graphs of your own. Each line fades in, rising by `rise` pixels, from its `delay`. With `out` (`{ end, seconds }`) it fades away over the last `seconds` before `end`. The text goes through files in `workDir`, named after `name`, so it needs no escaping.
@@ -280,6 +299,7 @@ The `lines` of a hook or tag, and the input of `.drawTexts()`:
 | `delay` | `number` | `0` | Seconds before the line fades in. `-0.25` has it fully in at the first frame |
 | `rise` | `number` | `18` | Pixels the line rises while it fades in |
 | `window` | `[number, number]` | | Seconds the line is shown, from the first up to, not including, the second |
+| `box` | `{ color, border }` | | A box behind the text, `border` px around it, or `[vertical, horizontal]` |
 
 In a hook or tag, `.renderClip()` sets `delay` and `rise` itself: the lines follow the band's or box's fade and do not rise.
 

package/dist/video/index.cjs

@@ -42,7 +42,7 @@ function defaultFfmpeg() {
     return "ffmpeg";
   }
 }
-var endsSentence = (text) => /[.?!]["']?$/.test(text);
+var SENTENCE_END = /[.?!]["']?$/;
 var Renderer = class {
   ffmpegPath;
   ink;
@@ -96,6 +96,46 @@ ${stderr.slice(-1500)}`);
     const width = this.textWidth(text, font, size);
     return width <= maxWidth ? size : Math.floor(size * maxWidth / width);
   }
+  /** Height of the font's capital letters at `size`, for placing text by its top. */
+  capHeight(font, size) {
+    const f = this.font(font);
+    return (f.tables.os2?.sCapHeight ?? f.ascender) * size / f.unitsPerEm;
+  }
+  /** `text` without the characters `font` has no glyph for, such as emoji, with spaces collapsed. */
+  printable(text, font) {
+    const f = this.font(font);
+    return [...text].filter((ch) => /\s/.test(ch) || f.charToGlyphIndex(ch) > 0).join("").replace(/\s+/g, " ").trim();
+  }
+  /**
+   * `text` wrapped into lines of at most `maxWidth`. With `balance`, the lines are as even as
+   * the fewest lines allow, instead of full lines and a short last one.
+   */
+  wrapText(text, font, size, { maxWidth = TEXT_WIDTH, balance = false } = {}) {
+    const wrap = (width) => {
+      const lines2 = [];
+      let line = "";
+      for (const word of text.split(" ")) {
+        const next = line ? `${line} ${word}` : word;
+        if (!line || this.textWidth(next, font, size) <= width) line = next;
+        else {
+          lines2.push(line);
+          line = word;
+        }
+      }
+      if (line) lines2.push(line);
+      return lines2;
+    };
+    const lines = wrap(maxWidth);
+    if (!balance || lines.length < 2) return lines;
+    let lo = 0;
+    let hi = maxWidth;
+    while (hi - lo > 1) {
+      const mid = (lo + hi) / 2;
+      if (wrap(mid).length <= lines.length) hi = mid;
+      else lo = mid;
+    }
+    return wrap(hi);
+  }
   /** A solid ink layer at `opacity`, as a filter source with optional alpha fades. */
   inkLayer({ w, h, opacity, duration, fadeIn, fadeOut }) {
     let src = `color=c=${this.ink}:s=${w}x${h}:r=${FPS}:d=${duration},format=rgba,colorchannelmixer=aa=${opacity}`;
@@ -110,7 +150,7 @@ ${stderr.slice(-1500)}`);
    */
   drawTexts(lines, out, workDir, name) {
     return lines.map((line, i) => {
-      const { x = MARGIN, baseline, delay = 0, rise = TEXT_RISE, window } = line;
+      const { x = MARGIN, baseline, delay = 0, rise = TEXT_RISE, window, box } = line;
       const p = `clip((t-${delay})/${TEXT_IN_SECONDS},0,1)`;
       const eased = `(${p}*(2-${p}))`;
       const alpha = out ? `${eased}*clip((${out.end}-t)/${out.seconds},0,1)` : eased;
@@ -120,7 +160,7 @@ ${stderr.slice(-1500)}`);
         `${name}-line${i}.txt`,
         { ...line, x },
         `y='${y}':alpha='${alpha}'` + // Half-open, so lines with back-to-back windows never share a frame.
-        (window ? `:enable='gte(t,${window[0]})*lt(t,${window[1]})'` : "")
+        (window ? `:enable='gte(t,${window[0]})*lt(t,${window[1]})'` : "") + (box ? `:box=1:boxcolor=${box.color}:boxborderw=${[box.border].flat().join("|")}` : "")
       );
     });
   }
@@ -135,15 +175,36 @@ ${stderr.slice(-1500)}`);
    * further while it is wider than `maxWidth`.
    */
   subtitleChunks(options) {
-    const { words, duration, font, color, style, size = 56, baseline = 1445, maxWords = 5, hold = 0.6, maxWidth = TEXT_WIDTH } = options;
+    const {
+      words,
+      duration,
+      font,
+      color,
+      style,
+      activeColor,
+      size = 56,
+      baseline = 1445,
+      maxWords = 5,
+      hold = 0.6,
+      maxWidth = TEXT_WIDTH,
+      breakAfter = SENTENCE_END,
+      pause,
+      border,
+      shadow
+    } = options;
     const styled = words.map((w) => ({ ...w, font, color, ...style?.(w) }));
     const space = this.textWidth(" ", font, size);
     const width = (ws) => ws.reduce((sum, w) => sum + this.textWidth(w.text, w.font, size), 0) + (ws.length - 1) * space;
     const sentences = [];
     let current = [];
     for (const word of styled) {
+      const prev = current[current.length - 1];
+      if (prev && pause != null && word.start - prev.end > pause) {
+        sentences.push(current);
+        current = [];
+      }
       current.push(word);
-      if (endsSentence(word.text)) {
+      if (breakAfter.test(word.text)) {
         sentences.push(current);
         current = [];
       }
@@ -172,39 +233,57 @@ ${stderr.slice(-1500)}`);
       const next = groups[i + 1]?.[0];
       const holdEnd = Math.min(duration, last.end + hold);
       const end = next ? Math.min(next.start, holdEnd) : holdEnd;
+      const start = Math.max(0, first.start);
+      const lineEnd = Math.max(end, first.start + 0.1);
       let x = (WIDTH - width(group)) / 2;
-      const placed = group.map((w) => {
-        const word = { text: w.text, font: w.font, color: w.color, size, baseline, x };
+      const placed = group.map((w, j) => {
+        const word = { text: w.text, font: w.font, color: w.color, size, baseline, x, ...border ? { border } : {}, ...shadow ? { shadow } : {} };
+        const spokenColor = w.activeColor === void 0 ? activeColor : w.activeColor;
+        if (spokenColor) {
+          const from = Math.max(start, w.start);
+          const to = j < group.length - 1 ? group[j + 1].start : lineEnd;
+          word.active = { start: from, end: Math.max(from, to), color: spokenColor };
+        }
         x += this.textWidth(w.text, w.font, size) + space;
         return word;
       });
-      return { start: Math.max(0, first.start), end: Math.max(end, first.start + 0.1), words: placed };
+      return { start, end: lineEnd, words: placed };
     });
   }
   // Each word is its own drawtext so a word can take another color.
   drawSubtitles(chunks, workDir, name) {
     return chunks.flatMap(
-      (chunk, c) => chunk.words.map(
-        (word, w) => this.drawtext(
-          workDir,
-          `${name}-sub${c}-${w}.txt`,
-          word,
-          `y=${word.baseline}-ascent:borderw=3:bordercolor=${this.ink}@0.85:enable='between(t,${chunk.start.toFixed(3)},${chunk.end.toFixed(3)})'`
-        )
-      )
+      (chunk, c) => chunk.words.flatMap((word, w) => {
+        const file = `${name}-sub${c}-${w}.txt`;
+        const { width, color } = word.border ?? { width: 3, color: `${this.ink}@0.85` };
+        const shadow = word.shadow ? `:shadowx=${word.shadow.x}:shadowy=${word.shadow.y}:shadowcolor=${word.shadow.color}` : "";
+        const look = `y=${word.baseline}-ascent:borderw=${width}:bordercolor=${color}${shadow}`;
+        const shown = `between(t,${chunk.start.toFixed(3)},${chunk.end.toFixed(3)})`;
+        if (!word.active) return [this.drawtext(workDir, file, word, `${look}:enable='${shown}'`)];
+        const spoken = `gte(t,${word.active.start.toFixed(3)})*lt(t,${word.active.end.toFixed(3)})`;
+        return [
+          this.drawtext(workDir, file, word, `${look}:enable='${shown}*not(${spoken})'`),
+          this.drawtext(workDir, file, { ...word, color: word.active.color }, `${look}:enable='${spoken}'`)
+        ];
+      })
     );
   }
   /**
    * Renders `start` to `end` of a clip as a 9:16 segment, with optional tag, hook, subtitles
    * and beeps. Returns the file name in `workDir`.
    */
-  async renderClip({ clipPath, start, end, tag, hook, hold = 0, subtitles = [], mute = [], workDir, name }) {
+  async renderClip({ clipPath, start, end, tag, hook, fill = FILL_9_16, hold = 0, subtitles = [], mute = [], loudness, workDir, name }) {
     const duration = end - start + hold;
-    const subs = this.drawSubtitles(subtitles.map((c) => ({ ...c, start: c.start + hold, end: c.end + hold })), workDir, name);
+    const held = subtitles.map((c) => ({
+      start: c.start + hold,
+      end: c.end + hold,
+      words: c.words.map((w) => w.active ? { ...w, active: { ...w.active, start: w.active.start + hold, end: w.active.end + hold } } : w)
+    }));
+    const subs = this.drawSubtitles(held, workDir, name);
     const withSubs = (chain) => subs.length ? `${chain},${subs.join(",")}` : chain;
     const out = hook ? "[pre]" : "[v]";
-    const input = `[0:v]${hold ? `tpad=start_duration=${hold.toFixed(3)}:start_mode=clone,` : ""}${FILL_9_16}`;
-    const sound = `[0:a]aresample=${SAMPLE_RATE}${hold ? `,adelay=${Math.round(hold * 1e3)}:all=1` : ""}`;
+    const input = `[0:v]${hold ? `tpad=start_duration=${hold.toFixed(3)}:start_mode=clone,` : ""}${fill}`;
+    const sound = `[0:a]${loudness == null ? "" : `loudnorm=I=${loudness}:TP=-1.5:LRA=11,`}aresample=${SAMPLE_RATE}${hold ? `,adelay=${Math.round(hold * 1e3)}:all=1` : ""}`;
     let video = `${withSubs(input)}${out}`;
     if (tag) {
       const { box, lines, delay = 0, outSeconds = 0, fade = true } = tag;
@@ -227,18 +306,25 @@ ${stderr.slice(-1500)}`);
     }
     if (hook) {
       const { band, lines, outSeconds } = hook;
-      const ink = this.inkLayer({ w: WIDTH, h: band.h, opacity: band.opacity, duration, fadeOut: { start: duration - outSeconds, seconds: outSeconds } });
-      const texts = this.drawTexts(lines.map((l) => ({ ...l, delay: -TEXT_IN_SECONDS, rise: 0 })), { end: duration, seconds: outSeconds }, workDir, `${name}-hook`);
-      video += `;${ink}[band];[pre][band]overlay=0:${band.y}:format=auto,${texts.join(",")}[v]`;
+      const fadeOut = outSeconds ? { start: duration - outSeconds, seconds: outSeconds } : null;
+      const texts = this.drawTexts(lines.map((l) => ({ ...l, delay: -TEXT_IN_SECONDS, rise: 0 })), fadeOut && { end: duration, seconds: fadeOut.seconds }, workDir, `${name}-hook`);
+      if (band) {
+        const ink = this.inkLayer({ w: WIDTH, h: band.h, opacity: band.opacity, duration, fadeOut });
+        video += `;${ink}[band];[pre][band]overlay=0:${band.y}:format=auto,${texts.join(",")}[v]`;
+      } else {
+        video += `;[pre]${texts.join(",") || "null"}[v]`;
+      }
     }
     let audio = `${sound}[a]`;
     if (mute.length) {
       const muted = `gt(${mute.map(([a, b]) => `between(t,${(a + hold).toFixed(3)},${(b + hold).toFixed(3)})`).join("+")},0)`;
       audio = `${sound},aformat=channel_layouts=stereo,volume=volume=0:enable='${muted}'[dry];aevalsrc='${BEEP_LEVEL}*sin(2*PI*${BEEP_HZ}*t)*${muted}':c=stereo:s=${SAMPLE_RATE}:d=${duration.toFixed(3)}[beep];[dry][beep]amix=inputs=2:normalize=0:duration=first[a]`;
     }
+    const graph = `${name}.filter`;
+    fs.writeFileSync(path.join(workDir, graph), `${video};${audio}`);
     const output = `${name}.mp4`;
     await this.run(
-      ["-ss", String(start), "-to", String(end), "-i", path.resolve(clipPath), "-filter_complex", `${video};${audio}`, "-map", "[v]", "-map", "[a]", ...OUTPUT_ARGS, output],
+      ["-ss", String(start), "-to", String(end), "-i", path.resolve(clipPath), "-filter_complex_script", graph, "-map", "[v]", "-map", "[a]", ...OUTPUT_ARGS, output],
       workDir
     );
     return output;

package/dist/video/index.d.cts

@@ -45,6 +45,22 @@ interface TextLine {
     rise?: number;
     /** [from, to) seconds the line is shown. */
     window?: [number, number];
+    /** A box behind the text, `border` px around it (or [vertical, horizontal]). */
+    box?: {
+        color: string;
+        border: number | [number, number];
+    };
+}
+/** Outline around subtitle words. */
+interface TextBorder {
+    width: number;
+    color: string;
+}
+/** Shadow under subtitle words, offset by `x` and `y` px. */
+interface TextShadow {
+    x: number;
+    y: number;
+    color: string;
 }
 /** Fades the text out over the last `seconds` before `end`. */
 interface FadeOut {
@@ -79,6 +95,15 @@ interface PlacedWord {
     size: number;
     baseline: number;
     x: number;
+    /** While the word is spoken (clip-relative seconds), it is drawn in `color` instead. */
+    active?: {
+        start: number;
+        end: number;
+        color: string;
+    };
+    /** Without it, a 3 px outline in the renderer's ink at 85%. */
+    border?: TextBorder;
+    shadow?: TextShadow;
 }
 /** One line of subtitles, shown from `start` to `end` (clip-relative seconds). */
 interface SubtitleChunk {
@@ -92,12 +117,21 @@ interface SubtitleOptions {
     duration: number;
     font: string;
     color: string;
-    /** Per-word font, color or text, such as a highlighted or blanked-out word. */
+    /**
+     * Per-word font, color or text, such as a highlighted or blanked-out word, and its own
+     * `activeColor` (`null` keeps the word's color while it is spoken).
+     */
     style?: (word: TimedWord) => {
         font?: string;
         color?: string;
         text?: string;
+        activeColor?: string | null;
     };
+    /**
+     * Color of each word while it is spoken, karaoke style: from its start to the next word's
+     * start, so early end times leave no gap, and to the line's end for its last word.
+     */
+    activeColor?: string;
     /** @default 56 */
     size?: number;
     /**
@@ -116,6 +150,18 @@ interface SubtitleOptions {
     hold?: number;
     /** @default TEXT_WIDTH */
     maxWidth?: number;
+    /**
+     * Words that end a piece, like sentence ends do. Pass `/[.,?!]["']?$/` to also break after
+     * commas.
+     *
+     * @default /[.?!]["']?$/
+     */
+    breakAfter?: RegExp;
+    /** A silence longer than this many seconds between two words also ends a piece. */
+    pause?: number;
+    /** @default a 3 px outline in the renderer's ink at 85% */
+    border?: TextBorder;
+    shadow?: TextShadow;
 }
 interface ClipOptions {
     clipPath: string;
@@ -142,16 +188,26 @@ interface ClipOptions {
         outSeconds?: number;
         fade?: boolean;
     };
-    /** A full-width ink band with text, up from the first frame, fading out before the clip ends. */
+    /**
+     * Text up from the first frame, on a full-width ink band when `band` is given, fading out
+     * `outSeconds` before the clip ends. Without `outSeconds` it stays to the end.
+     */
     hook?: {
-        band: {
+        band?: {
             y: number;
             h: number;
             opacity: number;
         };
         lines: TextLine[];
-        outSeconds: number;
+        outSeconds?: number;
     };
+    /**
+     * Filter chain that turns the source picture into the 1080x1920 frame, for a layout of your
+     * own: it reads the source picture and outputs the frame.
+     *
+     * @default FILL_9_16
+     */
+    fill?: string;
     /**
      * Seconds the first frame is held, silent, before the clip plays. Subtitles and mute
      * windows move with it.
@@ -160,6 +216,11 @@ interface ClipOptions {
     subtitles?: SubtitleChunk[];
     /** [start, end] windows (clip-relative seconds) where the sound is replaced by a beep. */
     mute?: [number, number][];
+    /**
+     * Target integrated loudness in LUFS (e.g. -14). The clip's sound is normalized to it
+     * before any beep is mixed in, so beeps stay at the same level in every clip.
+     */
+    loudness?: number;
 }
 /** Renders 1080x1920 reel segments with ffmpeg and joins them. */
 declare class Renderer {
@@ -177,6 +238,18 @@ declare class Renderer {
     textWidth(text: string, font: string, size: number): number;
     /** Largest size up to `size` at which the text fits in `maxWidth`. */
     fitSize(text: string, font: string, size: number, maxWidth?: number): number;
+    /** Height of the font's capital letters at `size`, for placing text by its top. */
+    capHeight(font: string, size: number): number;
+    /** `text` without the characters `font` has no glyph for, such as emoji, with spaces collapsed. */
+    printable(text: string, font: string): string;
+    /**
+     * `text` wrapped into lines of at most `maxWidth`. With `balance`, the lines are as even as
+     * the fewest lines allow, instead of full lines and a short last one.
+     */
+    wrapText(text: string, font: string, size: number, { maxWidth, balance }?: {
+        maxWidth?: number | undefined;
+        balance?: boolean | undefined;
+    }): string[];
     /** A solid ink layer at `opacity`, as a filter source with optional alpha fades. */
     inkLayer({ w, h, opacity, duration, fadeIn, fadeOut }: InkLayerOptions): string;
     /**
@@ -198,7 +271,7 @@ declare class Renderer {
      * Renders `start` to `end` of a clip as a 9:16 segment, with optional tag, hook, subtitles
      * and beeps. Returns the file name in `workDir`.
      */
-    renderClip({ clipPath, start, end, tag, hook, hold, subtitles, mute, workDir, name }: ClipOptions): Promise<string>;
+    renderClip({ clipPath, start, end, tag, hook, fill, hold, subtitles, mute, loudness, workDir, name }: ClipOptions): Promise<string>;
     /**
      * 16 kHz mono 16-bit WAV of `input`, the format keepitreel/transcribe reads. Without
      * `start`/`end`, the whole file.
@@ -224,4 +297,4 @@ declare class Renderer {
     concat(segments: string[], workDir: string, outputPath: string): Promise<void>;
 }
 
-export { type ClipOptions, FILL_9_16, FPS, type FadeOut, HEIGHT, type InkLayerOptions, MARGIN, OUTPUT_ARGS, type PlacedWord, Renderer, type RendererOptions, type SubtitleChunk, type SubtitleOptions, TEXT_WIDTH, type TextLine, type TimedWord, WIDTH };
+export { type ClipOptions, FILL_9_16, FPS, type FadeOut, HEIGHT, type InkLayerOptions, MARGIN, OUTPUT_ARGS, type PlacedWord, Renderer, type RendererOptions, type SubtitleChunk, type SubtitleOptions, TEXT_WIDTH, type TextBorder, type TextLine, type TextShadow, type TimedWord, WIDTH };

package/dist/video/index.d.ts

@@ -45,6 +45,22 @@ interface TextLine {
     rise?: number;
     /** [from, to) seconds the line is shown. */
     window?: [number, number];
+    /** A box behind the text, `border` px around it (or [vertical, horizontal]). */
+    box?: {
+        color: string;
+        border: number | [number, number];
+    };
+}
+/** Outline around subtitle words. */
+interface TextBorder {
+    width: number;
+    color: string;
+}
+/** Shadow under subtitle words, offset by `x` and `y` px. */
+interface TextShadow {
+    x: number;
+    y: number;
+    color: string;
 }
 /** Fades the text out over the last `seconds` before `end`. */
 interface FadeOut {
@@ -79,6 +95,15 @@ interface PlacedWord {
     size: number;
     baseline: number;
     x: number;
+    /** While the word is spoken (clip-relative seconds), it is drawn in `color` instead. */
+    active?: {
+        start: number;
+        end: number;
+        color: string;
+    };
+    /** Without it, a 3 px outline in the renderer's ink at 85%. */
+    border?: TextBorder;
+    shadow?: TextShadow;
 }
 /** One line of subtitles, shown from `start` to `end` (clip-relative seconds). */
 interface SubtitleChunk {
@@ -92,12 +117,21 @@ interface SubtitleOptions {
     duration: number;
     font: string;
     color: string;
-    /** Per-word font, color or text, such as a highlighted or blanked-out word. */
+    /**
+     * Per-word font, color or text, such as a highlighted or blanked-out word, and its own
+     * `activeColor` (`null` keeps the word's color while it is spoken).
+     */
     style?: (word: TimedWord) => {
         font?: string;
         color?: string;
         text?: string;
+        activeColor?: string | null;
     };
+    /**
+     * Color of each word while it is spoken, karaoke style: from its start to the next word's
+     * start, so early end times leave no gap, and to the line's end for its last word.
+     */
+    activeColor?: string;
     /** @default 56 */
     size?: number;
     /**
@@ -116,6 +150,18 @@ interface SubtitleOptions {
     hold?: number;
     /** @default TEXT_WIDTH */
     maxWidth?: number;
+    /**
+     * Words that end a piece, like sentence ends do. Pass `/[.,?!]["']?$/` to also break after
+     * commas.
+     *
+     * @default /[.?!]["']?$/
+     */
+    breakAfter?: RegExp;
+    /** A silence longer than this many seconds between two words also ends a piece. */
+    pause?: number;
+    /** @default a 3 px outline in the renderer's ink at 85% */
+    border?: TextBorder;
+    shadow?: TextShadow;
 }
 interface ClipOptions {
     clipPath: string;
@@ -142,16 +188,26 @@ interface ClipOptions {
         outSeconds?: number;
         fade?: boolean;
     };
-    /** A full-width ink band with text, up from the first frame, fading out before the clip ends. */
+    /**
+     * Text up from the first frame, on a full-width ink band when `band` is given, fading out
+     * `outSeconds` before the clip ends. Without `outSeconds` it stays to the end.
+     */
     hook?: {
-        band: {
+        band?: {
             y: number;
             h: number;
             opacity: number;
         };
         lines: TextLine[];
-        outSeconds: number;
+        outSeconds?: number;
     };
+    /**
+     * Filter chain that turns the source picture into the 1080x1920 frame, for a layout of your
+     * own: it reads the source picture and outputs the frame.
+     *
+     * @default FILL_9_16
+     */
+    fill?: string;
     /**
      * Seconds the first frame is held, silent, before the clip plays. Subtitles and mute
      * windows move with it.
@@ -160,6 +216,11 @@ interface ClipOptions {
     subtitles?: SubtitleChunk[];
     /** [start, end] windows (clip-relative seconds) where the sound is replaced by a beep. */
     mute?: [number, number][];
+    /**
+     * Target integrated loudness in LUFS (e.g. -14). The clip's sound is normalized to it
+     * before any beep is mixed in, so beeps stay at the same level in every clip.
+     */
+    loudness?: number;
 }
 /** Renders 1080x1920 reel segments with ffmpeg and joins them. */
 declare class Renderer {
@@ -177,6 +238,18 @@ declare class Renderer {
     textWidth(text: string, font: string, size: number): number;
     /** Largest size up to `size` at which the text fits in `maxWidth`. */
     fitSize(text: string, font: string, size: number, maxWidth?: number): number;
+    /** Height of the font's capital letters at `size`, for placing text by its top. */
+    capHeight(font: string, size: number): number;
+    /** `text` without the characters `font` has no glyph for, such as emoji, with spaces collapsed. */
+    printable(text: string, font: string): string;
+    /**
+     * `text` wrapped into lines of at most `maxWidth`. With `balance`, the lines are as even as
+     * the fewest lines allow, instead of full lines and a short last one.
+     */
+    wrapText(text: string, font: string, size: number, { maxWidth, balance }?: {
+        maxWidth?: number | undefined;
+        balance?: boolean | undefined;
+    }): string[];
     /** A solid ink layer at `opacity`, as a filter source with optional alpha fades. */
     inkLayer({ w, h, opacity, duration, fadeIn, fadeOut }: InkLayerOptions): string;
     /**
@@ -198,7 +271,7 @@ declare class Renderer {
      * Renders `start` to `end` of a clip as a 9:16 segment, with optional tag, hook, subtitles
      * and beeps. Returns the file name in `workDir`.
      */
-    renderClip({ clipPath, start, end, tag, hook, hold, subtitles, mute, workDir, name }: ClipOptions): Promise<string>;
+    renderClip({ clipPath, start, end, tag, hook, fill, hold, subtitles, mute, loudness, workDir, name }: ClipOptions): Promise<string>;
     /**
      * 16 kHz mono 16-bit WAV of `input`, the format keepitreel/transcribe reads. Without
      * `start`/`end`, the whole file.
@@ -224,4 +297,4 @@ declare class Renderer {
     concat(segments: string[], workDir: string, outputPath: string): Promise<void>;
 }
 
-export { type ClipOptions, FILL_9_16, FPS, type FadeOut, HEIGHT, type InkLayerOptions, MARGIN, OUTPUT_ARGS, type PlacedWord, Renderer, type RendererOptions, type SubtitleChunk, type SubtitleOptions, TEXT_WIDTH, type TextLine, type TimedWord, WIDTH };
+export { type ClipOptions, FILL_9_16, FPS, type FadeOut, HEIGHT, type InkLayerOptions, MARGIN, OUTPUT_ARGS, type PlacedWord, Renderer, type RendererOptions, type SubtitleChunk, type SubtitleOptions, TEXT_WIDTH, type TextBorder, type TextLine, type TextShadow, type TimedWord, WIDTH };

package/dist/video/index.js

@@ -38,7 +38,7 @@ function defaultFfmpeg() {
     return "ffmpeg";
   }
 }
-var endsSentence = (text) => /[.?!]["']?$/.test(text);
+var SENTENCE_END = /[.?!]["']?$/;
 var Renderer = class {
   ffmpegPath;
   ink;
@@ -92,6 +92,46 @@ ${stderr.slice(-1500)}`);
     const width = this.textWidth(text, font, size);
     return width <= maxWidth ? size : Math.floor(size * maxWidth / width);
   }
+  /** Height of the font's capital letters at `size`, for placing text by its top. */
+  capHeight(font, size) {
+    const f = this.font(font);
+    return (f.tables.os2?.sCapHeight ?? f.ascender) * size / f.unitsPerEm;
+  }
+  /** `text` without the characters `font` has no glyph for, such as emoji, with spaces collapsed. */
+  printable(text, font) {
+    const f = this.font(font);
+    return [...text].filter((ch) => /\s/.test(ch) || f.charToGlyphIndex(ch) > 0).join("").replace(/\s+/g, " ").trim();
+  }
+  /**
+   * `text` wrapped into lines of at most `maxWidth`. With `balance`, the lines are as even as
+   * the fewest lines allow, instead of full lines and a short last one.
+   */
+  wrapText(text, font, size, { maxWidth = TEXT_WIDTH, balance = false } = {}) {
+    const wrap = (width) => {
+      const lines2 = [];
+      let line = "";
+      for (const word of text.split(" ")) {
+        const next = line ? `${line} ${word}` : word;
+        if (!line || this.textWidth(next, font, size) <= width) line = next;
+        else {
+          lines2.push(line);
+          line = word;
+        }
+      }
+      if (line) lines2.push(line);
+      return lines2;
+    };
+    const lines = wrap(maxWidth);
+    if (!balance || lines.length < 2) return lines;
+    let lo = 0;
+    let hi = maxWidth;
+    while (hi - lo > 1) {
+      const mid = (lo + hi) / 2;
+      if (wrap(mid).length <= lines.length) hi = mid;
+      else lo = mid;
+    }
+    return wrap(hi);
+  }
   /** A solid ink layer at `opacity`, as a filter source with optional alpha fades. */
   inkLayer({ w, h, opacity, duration, fadeIn, fadeOut }) {
     let src = `color=c=${this.ink}:s=${w}x${h}:r=${FPS}:d=${duration},format=rgba,colorchannelmixer=aa=${opacity}`;
@@ -106,7 +146,7 @@ ${stderr.slice(-1500)}`);
    */
   drawTexts(lines, out, workDir, name) {
     return lines.map((line, i) => {
-      const { x = MARGIN, baseline, delay = 0, rise = TEXT_RISE, window } = line;
+      const { x = MARGIN, baseline, delay = 0, rise = TEXT_RISE, window, box } = line;
       const p = `clip((t-${delay})/${TEXT_IN_SECONDS},0,1)`;
       const eased = `(${p}*(2-${p}))`;
       const alpha = out ? `${eased}*clip((${out.end}-t)/${out.seconds},0,1)` : eased;
@@ -116,7 +156,7 @@ ${stderr.slice(-1500)}`);
         `${name}-line${i}.txt`,
         { ...line, x },
         `y='${y}':alpha='${alpha}'` + // Half-open, so lines with back-to-back windows never share a frame.
-        (window ? `:enable='gte(t,${window[0]})*lt(t,${window[1]})'` : "")
+        (window ? `:enable='gte(t,${window[0]})*lt(t,${window[1]})'` : "") + (box ? `:box=1:boxcolor=${box.color}:boxborderw=${[box.border].flat().join("|")}` : "")
       );
     });
   }
@@ -131,15 +171,36 @@ ${stderr.slice(-1500)}`);
    * further while it is wider than `maxWidth`.
    */
   subtitleChunks(options) {
-    const { words, duration, font, color, style, size = 56, baseline = 1445, maxWords = 5, hold = 0.6, maxWidth = TEXT_WIDTH } = options;
+    const {
+      words,
+      duration,
+      font,
+      color,
+      style,
+      activeColor,
+      size = 56,
+      baseline = 1445,
+      maxWords = 5,
+      hold = 0.6,
+      maxWidth = TEXT_WIDTH,
+      breakAfter = SENTENCE_END,
+      pause,
+      border,
+      shadow
+    } = options;
     const styled = words.map((w) => ({ ...w, font, color, ...style?.(w) }));
     const space = this.textWidth(" ", font, size);
     const width = (ws) => ws.reduce((sum, w) => sum + this.textWidth(w.text, w.font, size), 0) + (ws.length - 1) * space;
     const sentences = [];
     let current = [];
     for (const word of styled) {
+      const prev = current[current.length - 1];
+      if (prev && pause != null && word.start - prev.end > pause) {
+        sentences.push(current);
+        current = [];
+      }
       current.push(word);
-      if (endsSentence(word.text)) {
+      if (breakAfter.test(word.text)) {
         sentences.push(current);
         current = [];
       }
@@ -168,39 +229,57 @@ ${stderr.slice(-1500)}`);
       const next = groups[i + 1]?.[0];
       const holdEnd = Math.min(duration, last.end + hold);
       const end = next ? Math.min(next.start, holdEnd) : holdEnd;
+      const start = Math.max(0, first.start);
+      const lineEnd = Math.max(end, first.start + 0.1);
       let x = (WIDTH - width(group)) / 2;
-      const placed = group.map((w) => {
-        const word = { text: w.text, font: w.font, color: w.color, size, baseline, x };
+      const placed = group.map((w, j) => {
+        const word = { text: w.text, font: w.font, color: w.color, size, baseline, x, ...border ? { border } : {}, ...shadow ? { shadow } : {} };
+        const spokenColor = w.activeColor === void 0 ? activeColor : w.activeColor;
+        if (spokenColor) {
+          const from = Math.max(start, w.start);
+          const to = j < group.length - 1 ? group[j + 1].start : lineEnd;
+          word.active = { start: from, end: Math.max(from, to), color: spokenColor };
+        }
         x += this.textWidth(w.text, w.font, size) + space;
         return word;
       });
-      return { start: Math.max(0, first.start), end: Math.max(end, first.start + 0.1), words: placed };
+      return { start, end: lineEnd, words: placed };
     });
   }
   // Each word is its own drawtext so a word can take another color.
   drawSubtitles(chunks, workDir, name) {
     return chunks.flatMap(
-      (chunk, c) => chunk.words.map(
-        (word, w) => this.drawtext(
-          workDir,
-          `${name}-sub${c}-${w}.txt`,
-          word,
-          `y=${word.baseline}-ascent:borderw=3:bordercolor=${this.ink}@0.85:enable='between(t,${chunk.start.toFixed(3)},${chunk.end.toFixed(3)})'`
-        )
-      )
+      (chunk, c) => chunk.words.flatMap((word, w) => {
+        const file = `${name}-sub${c}-${w}.txt`;
+        const { width, color } = word.border ?? { width: 3, color: `${this.ink}@0.85` };
+        const shadow = word.shadow ? `:shadowx=${word.shadow.x}:shadowy=${word.shadow.y}:shadowcolor=${word.shadow.color}` : "";
+        const look = `y=${word.baseline}-ascent:borderw=${width}:bordercolor=${color}${shadow}`;
+        const shown = `between(t,${chunk.start.toFixed(3)},${chunk.end.toFixed(3)})`;
+        if (!word.active) return [this.drawtext(workDir, file, word, `${look}:enable='${shown}'`)];
+        const spoken = `gte(t,${word.active.start.toFixed(3)})*lt(t,${word.active.end.toFixed(3)})`;
+        return [
+          this.drawtext(workDir, file, word, `${look}:enable='${shown}*not(${spoken})'`),
+          this.drawtext(workDir, file, { ...word, color: word.active.color }, `${look}:enable='${spoken}'`)
+        ];
+      })
     );
   }
   /**
    * Renders `start` to `end` of a clip as a 9:16 segment, with optional tag, hook, subtitles
    * and beeps. Returns the file name in `workDir`.
    */
-  async renderClip({ clipPath, start, end, tag, hook, hold = 0, subtitles = [], mute = [], workDir, name }) {
+  async renderClip({ clipPath, start, end, tag, hook, fill = FILL_9_16, hold = 0, subtitles = [], mute = [], loudness, workDir, name }) {
     const duration = end - start + hold;
-    const subs = this.drawSubtitles(subtitles.map((c) => ({ ...c, start: c.start + hold, end: c.end + hold })), workDir, name);
+    const held = subtitles.map((c) => ({
+      start: c.start + hold,
+      end: c.end + hold,
+      words: c.words.map((w) => w.active ? { ...w, active: { ...w.active, start: w.active.start + hold, end: w.active.end + hold } } : w)
+    }));
+    const subs = this.drawSubtitles(held, workDir, name);
     const withSubs = (chain) => subs.length ? `${chain},${subs.join(",")}` : chain;
     const out = hook ? "[pre]" : "[v]";
-    const input = `[0:v]${hold ? `tpad=start_duration=${hold.toFixed(3)}:start_mode=clone,` : ""}${FILL_9_16}`;
-    const sound = `[0:a]aresample=${SAMPLE_RATE}${hold ? `,adelay=${Math.round(hold * 1e3)}:all=1` : ""}`;
+    const input = `[0:v]${hold ? `tpad=start_duration=${hold.toFixed(3)}:start_mode=clone,` : ""}${fill}`;
+    const sound = `[0:a]${loudness == null ? "" : `loudnorm=I=${loudness}:TP=-1.5:LRA=11,`}aresample=${SAMPLE_RATE}${hold ? `,adelay=${Math.round(hold * 1e3)}:all=1` : ""}`;
     let video = `${withSubs(input)}${out}`;
     if (tag) {
       const { box, lines, delay = 0, outSeconds = 0, fade = true } = tag;
@@ -223,18 +302,25 @@ ${stderr.slice(-1500)}`);
     }
     if (hook) {
       const { band, lines, outSeconds } = hook;
-      const ink = this.inkLayer({ w: WIDTH, h: band.h, opacity: band.opacity, duration, fadeOut: { start: duration - outSeconds, seconds: outSeconds } });
-      const texts = this.drawTexts(lines.map((l) => ({ ...l, delay: -TEXT_IN_SECONDS, rise: 0 })), { end: duration, seconds: outSeconds }, workDir, `${name}-hook`);
-      video += `;${ink}[band];[pre][band]overlay=0:${band.y}:format=auto,${texts.join(",")}[v]`;
+      const fadeOut = outSeconds ? { start: duration - outSeconds, seconds: outSeconds } : null;
+      const texts = this.drawTexts(lines.map((l) => ({ ...l, delay: -TEXT_IN_SECONDS, rise: 0 })), fadeOut && { end: duration, seconds: fadeOut.seconds }, workDir, `${name}-hook`);
+      if (band) {
+        const ink = this.inkLayer({ w: WIDTH, h: band.h, opacity: band.opacity, duration, fadeOut });
+        video += `;${ink}[band];[pre][band]overlay=0:${band.y}:format=auto,${texts.join(",")}[v]`;
+      } else {
+        video += `;[pre]${texts.join(",") || "null"}[v]`;
+      }
     }
     let audio = `${sound}[a]`;
     if (mute.length) {
       const muted = `gt(${mute.map(([a, b]) => `between(t,${(a + hold).toFixed(3)},${(b + hold).toFixed(3)})`).join("+")},0)`;
       audio = `${sound},aformat=channel_layouts=stereo,volume=volume=0:enable='${muted}'[dry];aevalsrc='${BEEP_LEVEL}*sin(2*PI*${BEEP_HZ}*t)*${muted}':c=stereo:s=${SAMPLE_RATE}:d=${duration.toFixed(3)}[beep];[dry][beep]amix=inputs=2:normalize=0:duration=first[a]`;
     }
+    const graph = `${name}.filter`;
+    writeFileSync(join(workDir, graph), `${video};${audio}`);
     const output = `${name}.mp4`;
     await this.run(
-      ["-ss", String(start), "-to", String(end), "-i", resolve(clipPath), "-filter_complex", `${video};${audio}`, "-map", "[v]", "-map", "[a]", ...OUTPUT_ARGS, output],
+      ["-ss", String(start), "-to", String(end), "-i", resolve(clipPath), "-filter_complex_script", graph, "-map", "[v]", "-map", "[a]", ...OUTPUT_ARGS, output],
       workDir
     );
     return output;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "keepitreel",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Building blocks for automated short-form video: Whisper transcription, ffmpeg reel rendering, and publishing to YouTube Shorts, Facebook, Instagram and TikTok.",
   "type": "module",
   "exports": {