npm - @mevdragon/vidfarm-devcli - Versions diffs - 0.13.0 → 0.15.0 - Mend

@mevdragon/vidfarm-devcli 0.13.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (427) hide show

package/.agents/skills/embedded-captions/references/example-renders/memory-wall.html ADDED Viewed

@@ -0,0 +1,334 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=1280, height=720" />
+    <script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
+    <style>
+      * {
+        margin: 0;
+        padding: 0;
+        box-sizing: border-box;
+      }
+      html,
+      body {
+        margin: 0;
+        width: 1280px;
+        height: 720px;
+        overflow: hidden;
+        background: #000;
+        font-family: "Inter", system-ui, sans-serif;
+      }
+      #a-roll,
+      #fg-overlay {
+        position: absolute;
+        inset: 0;
+        width: 100%;
+        height: 100%;
+        object-fit: cover;
+      }
+      /* Foreground matte sits above the captions so the person & mic occlude
+         the text — classic cinematic lower-third layering. */
+      #fg-overlay {
+        z-index: 3;
+        pointer-events: none;
+      }
+      /*
+        Caption stage — perspective container so the text plane lies on the
+        acoustic-foam wall in the upper-right of the frame. Small jitter is
+        applied to the stage by GSAP to fake handheld motion tracking.
+      */
+      #stage {
+        position: absolute;
+        top: 0;
+        left: 0;
+        width: 1280px;
+        height: 720px;
+        perspective: 1400px;
+        perspective-origin: 30% 40%;
+        pointer-events: none;
+        z-index: 2;
+        will-change: transform;
+      }
+      .wall-plane {
+        position: absolute;
+        /* Right-aligned cluster, but widened and pushed slightly left
+           so emphasis words (EVERYTHING, SHARP AGAIN) reach past the
+           subject's silhouette. Short words still hug the wall. */
+        top: 90px;
+        right: 30px;
+        width: 720px;
+        height: 520px;
+        transform-style: preserve-3d;
+        transform: rotateY(-13deg) rotateX(1deg);
+        transform-origin: right center;
+        padding: 24px 32px;
+        display: flex;
+        flex-direction: column;
+        justify-content: center;
+        align-items: flex-end;
+        text-align: right;
+        gap: 14px;
+      }
+      .cap {
+        display: block;
+        position: relative;
+        line-height: 1.05;
+        opacity: 0;
+        /* The "embed in scene" move: overlay blend picks up the wall's
+           luminance and color so the text feels lit BY the scene. */
+        mix-blend-mode: screen;
+        color: #fff4dc;
+        text-shadow:
+          0 0 12px rgba(255, 210, 150, 0.45),
+          0 2px 5px rgba(0, 0, 0, 0.7);
+        filter: brightness(1.1) contrast(1.05);
+        will-change: transform, opacity, filter;
+      }
+      .cap .w {
+        display: inline-block;
+        opacity: 0;
+        will-change: opacity, transform, filter;
+      }
+      /*
+        Unified cinematic typography — no mode switch between soft & sharp.
+        Elegant serif-inflected sans, italic for the dreamy first half,
+        upright for the second half. One look, one feeling, progressing.
+      */
+      .cap-1 {
+        font-size: 78px;
+        font-weight: 600;
+        font-style: italic;
+        letter-spacing: -0.01em;
+      }
+      .cap-2 {
+        font-size: 66px;
+        font-weight: 500;
+        font-style: italic;
+        letter-spacing: -0.005em;
+        padding-right: 44px; /* hanging indent on right for right-aligned flow */
+      }
+      .cap-3 {
+        font-size: 72px;
+        font-weight: 700;
+        font-style: normal;
+        letter-spacing: -0.015em;
+      }
+      .cap-4 {
+        font-size: 90px;
+        font-weight: 900;
+        font-style: normal;
+        letter-spacing: -0.03em;
+        text-transform: uppercase;
+        line-height: 1;
+      }
+    </style>
+  </head>
+  <body>
+    <div
+      id="root"
+      data-composition-id="main"
+      data-start="0"
+      data-duration="8.04"
+      data-width="1280"
+      data-height="720"
+    >
+      <!-- Layer 0: background plate (full video) -->
+      <video
+        id="a-roll"
+        class="clip"
+        src="source.mp4"
+        muted
+        playsinline
+        data-duration="8.04"
+        data-track-index="0"
+        style="z-index: 1"
+      ></video>
+      <!-- Layer 2: caption stage (between background and foreground matte) -->
+      <div id="stage">
+        <div class="wall-plane">
+          <div id="cg-0" class="cap cap-1">
+            <span class="w" data-i="0">Some</span>
+            <span class="w" data-i="1">memories</span>
+            <span class="w" data-i="2">feel</span>
+            <span class="w" data-i="3">soft</span>
+          </div>
+          <div id="cg-1" class="cap cap-2">
+            <span class="w" data-i="0">like</span>
+            <span class="w" data-i="1">old</span>
+            <span class="w" data-i="2">film</span>
+          </div>
+          <div id="cg-2" class="cap cap-3">
+            <span class="w" data-i="0">but</span>
+            <span class="w" data-i="1">suddenly</span>
+          </div>
+          <div id="cg-3" class="cap cap-4">
+            <span class="w" data-i="0">everything</span>
+            <span class="w" data-i="1">is</span>
+            <span class="w" data-i="2">sharp</span>
+            <span class="w" data-i="3">again</span>
+          </div>
+        </div>
+      </div>
+      <!-- Foreground occlusion (person + mic) is applied in post via ffmpeg
+           overlay. Browser-side alpha webm is unreliable across encoders. -->
+      <!-- Audio -->
+      <audio
+        id="a-roll-audio"
+        src="source.mp4"
+        data-start="0"
+        data-duration="8.04"
+        data-track-index="3"
+        data-volume="1"
+      ></audio>
+    </div>
+    <script>
+      window.__timelines = window.__timelines || {};
+      const tl = gsap.timeline({ paused: true });
+      const DUR = 8.04;
+      /* ─────────────────────────────────────────────────────────────────
+         Word-level transcript — ElevenLabs timestamps.
+         ───────────────────────────────────────────────────────────────── */
+      const GROUPS = [
+        {
+          id: "cg-0",
+          words: [
+            { text: "Some", start: 0.24, end: 0.44 },
+            { text: "memories", start: 0.48, end: 0.82 },
+            { text: "feel", start: 0.9, end: 1.14 },
+            { text: "soft", start: 1.2, end: 1.64 },
+          ],
+          in: 0.2,
+          out: 4.85,
+          tone: "soft",
+        },
+        {
+          id: "cg-1",
+          words: [
+            { text: "like", start: 2.66, end: 2.78 },
+            { text: "old", start: 2.86, end: 3.02 },
+            { text: "film", start: 3.14, end: 3.46 },
+          ],
+          in: 2.55,
+          out: 4.85,
+          tone: "soft",
+        },
+        {
+          id: "cg-2",
+          words: [
+            { text: "but", start: 5.02, end: 5.16 },
+            { text: "suddenly", start: 5.22, end: 5.76 },
+          ],
+          in: 4.9,
+          out: 8.04,
+          tone: "present",
+        },
+        {
+          id: "cg-3",
+          words: [
+            { text: "everything", start: 6.8, end: 7.14 },
+            { text: "is", start: 7.16, end: 7.28 },
+            { text: "sharp", start: 7.3, end: 7.52 },
+            { text: "again", start: 7.56, end: 7.92 },
+          ],
+          in: 6.7,
+          out: 8.04,
+          tone: "present",
+        },
+      ];
+      /* ─────────────────────────────────────────────────────────────────
+         Each caption group: cinematic word-by-word reveal. No flashes, no
+         grain — purely text animation. Soft groups drift up gently, later
+         groups resolve firmly into place to match the sentence turn.
+         ───────────────────────────────────────────────────────────────── */
+      GROUPS.forEach(function (g) {
+        var sel = "#" + g.id;
+        var words = g.words.map(function (_, i) {
+          return sel + " .w[data-i='" + i + "']";
+        });
+        var isSoft = g.tone === "soft";
+        /* Container stays fully visible — each word animates its own fade.
+           This avoids a double-fade (container × word) that produced a
+           non-linear pop in the first ~100ms of the first caption. */
+        tl.set(sel, { opacity: 1, y: 0 }, Math.max(0, g.in - 0.01));
+        /* Per-word reveal — opacity + translate only. No letter-spacing
+           or filter:blur here: both can perturb inline-block metrics
+           mid-animation and cause the cap line-box to reflow, which
+           makes words like "Some" visibly shift between rows when a
+           sibling word enters. transform-only = no reflow. */
+        g.words.forEach(function (w, i) {
+          if (isSoft) {
+            tl.fromTo(
+              words[i],
+              { opacity: 0, y: 8 },
+              {
+                opacity: 1,
+                y: 0,
+                duration: 0.45,
+                ease: "power2.out",
+                overwrite: "auto",
+              },
+              w.start,
+            );
+          } else {
+            tl.fromTo(
+              words[i],
+              { opacity: 0, y: 6, scale: 1.04 },
+              {
+                opacity: 1,
+                y: 0,
+                scale: 1.0,
+                duration: 0.22,
+                ease: "power3.out",
+                overwrite: "auto",
+                transformOrigin: "50% 50%",
+              },
+              w.start,
+            );
+          }
+        });
+        /* Group exit — soft groups linger and fade as the sentence turns,
+           present groups hold to the end. Clean fades, no trickery. */
+        if (g.out < DUR) {
+          tl.to(
+            sel,
+            { opacity: 0, duration: 0.5, ease: "power2.in", overwrite: "auto" },
+            g.out - 0.5,
+          );
+        }
+        tl.set(sel, { opacity: 0, visibility: "hidden" }, g.out);
+      });
+      /* Self-lint */
+      GROUPS.forEach(function (g) {
+        var el = document.getElementById(g.id);
+        if (!el) return;
+        tl.seek(g.out + 0.01);
+        var cs = window.getComputedStyle(el);
+        if (cs.opacity !== "0" && cs.visibility !== "hidden") {
+          console.warn("[caption-lint] " + g.id + " visible at t=" + (g.out + 0.01).toFixed(2));
+        }
+      });
+      tl.seek(0);
+      window.__timelines["main"] = tl;
+    </script>
+  </body>
+</html>

package/.agents/skills/embedded-captions/references/failure-modes.md ADDED Viewed

@@ -0,0 +1,145 @@
+# Failure Modes — learned the hard way
+Things that broke during skill development. Check against this list before you ship.
+## Matting
+### CoreML execution provider corrupts face alpha
+Running the matting ONNX through the CoreML EP partitions the graph across providers; the mixed-precision boundary produced alpha ≈ 30 (out of 255) inside the subject's face while the background correctly read 0 (observed with the previous RVM engine — don't re-try). Result: caption text visibly shines THROUGH the face.
+**Fix**: Only ever use `providers=["CPUExecutionProvider"]` for the matting ONNX. Our script does this; don't "optimize" it by adding CoreML.
+### Aspect distortion / sharp channel-stride make the alpha garbage
+(historical — the PP-MattingV2 engine was replaced 2026-06-12 by hyperframes `remove-background`; kept because the lessons generalize) Two earned scars from the ONNX era: (1) fixed-size model exports — squashing a portrait clip into a landscape canvas distorts humans; contain-pad (aspect preserved, centered, alpha cropped back) instead. (2) sharp returns a **3-channel** buffer from `.raw()` after resizing a 1-channel raw input — reading it with a 1-channel stride silently produces a smeared, striped, displaced matte that can _look_ plausible. Always `toBuffer({resolveWithObject:true})` and stride by `info.channels` (other scripts in this skill still use sharp).
+**Fix**: both handled inside `matte.cjs`; preserve them if you touch it.
+### isnet-general-use / u2net_human_seg miss props
+rembg's general/human models capture the person but drop handheld props (mic, notebook). If the caption crosses the mic, it will NOT be occluded by the mic — text floats in front of it unnaturally.
+**Fix**: For scenes where props matter, either (a) accept that the human-matting model captures what it captures, or (b) add a manual mask ROI for the prop region. We haven't productized (b) yet.
+## Layout
+### Flex-stacked captions push hidden ones into gesture zone
+First champion iteration put all cap elements inside a flex column with `display: block`. Hidden captions (visibility: hidden) STILL reserve flex space. Result: cg-4 "dreaming" got pushed to y=700, landing right on the hand gesture zone and getting clipped.
+**Fix**: Inside the caption plane, each `.cap` is `position: absolute` at the same coords. Hidden ones don't occupy layout. Both templates in this skill do it this way.
+### Header-overlay on close-crop shots
+First champion iteration used a full-width header band across the TOP of the frame (`top: 30, left: 40, right: 40, height: 360, rotateX: -3deg, justify-content: flex-start, align-items: center`). Subject's head crosses the center, eating the middle of every caption. "know, for me" became a faint "now, for" with the letters sliced apart.
+**Fix**: On close-crop podcast shots, don't use symmetric full-width bands. Use `corner-column-crown` with caption column in the corner opposite the head.
+### Text right-align + align-items: flex-end + no max-width → flows off-screen
+If you set `align-items: flex-end` on the plane and don't constrain width, each `.cap` shrinks to content width. Long words then extend left beyond the plane's "visible" box without wrapping, clipping into the subject.
+**Fix**: Set `max-width: calc(100% - padding*2)` on each `.cap`. Templates do this.
+## Animation
+### letterSpacing animation causes inline-block reflow jumps
+Had `letterSpacing: "0.04em"` as the "from" value in `tl.fromTo(word, {opacity:0, letterSpacing: ".04em"}, {letterSpacing:"0"}, w.start)`. GSAP snaps the "from" state at w.start. The width of the word then changes over the tween → inline-block reflow → the `.cap` line box recomputes → "Some" visibly jumped between rows as "memories" entered.
+**Fix**: Animate only **transform** properties (opacity, y, scale). NEVER animate letter-spacing, font-size, or filter:blur on `.w` spans. Templates follow this.
+### Double-fade (container × word) produces a non-linear pop
+Fading both the group container AND each word independently multiplies: effective_opacity = container.opacity × word.opacity. Even with matched eases, the combined curve is nonlinear and feels like a snap-in around the 40% mark.
+**Fix**: Keep container at fixed `opacity: 1` during the group's life (set via `tl.set` just before first word). Animate only per-word opacity. Templates do this.
+## Blending
+### mix-blend-mode: overlay vanishes into dark regions
+`overlay(white_text, black_bg) ≈ black`. If part of the caption falls on a dark shadow / dark prop, the letters disappear. The first champion render had `SHARP` become `ARP` because "S" and "H" landed in the dark mic's shadow.
+**Fix**: For scenes with significant dark regions under the caption, use `mix-blend-mode: screen` (which preserves white on dark). `wall-embed` defaults to overlay (for mid-tone walls); `corner-column-crown` defaults to screen (for dark bookshelves).
+### Bright backgrounds wash out screen blend
+Opposite problem: `screen(white, bright) ≈ white` regardless of the text — letters become invisible on bright backgrounds like sunlit windows or white walls.
+**Fix**: If average luminance of the caption region exceeds 180/255, switch CSS to `mix-blend-mode: normal` with an opaque color. Sample the background pixels in the caption bbox across the clip to check.
+## Render pipeline
+### Frame-rate mismatch between bg render and matte PNGs
+If hyperframes renders at 30fps default but the matte was extracted at 24fps source rate, ffmpeg overlay gets a temporal mismatch — alpha from frame N is overlaid on bg content of some fractional frame ≠ N. Result: person's current silhouette and matte silhouette are shifted by a few frames, so the occlusion lags or leads the body.
+**Fix**: Always pass `--fps` to hyperframes render matching the source's native FPS (usually 24). `render-and-composite.sh` reads fps from plan.json and passes it through.
+### WebM alpha (VP8/VP9) flaky in Chromium
+We tried making foreground an alpha-channel WebM. Chromium's support for VP9 alpha is inconsistent; VP8 alpha needs specific metadata the default ffmpeg encode doesn't produce.
+**Fix**: Don't try to put the matte inside hyperframes. Do the overlay in post with ffmpeg against a PNG sequence. `render-and-composite.sh` does this.
+## Crown / centered text
+### Centered crown eaten by body
+Default `.crown-plane` in `corner-column-crown` is `left: 0; right: 0; text-align: center`. That mechanically centers the word at `frame_width/2`. If the subject sits right of center (Jobs at x=1100 in a 1920 frame) OR the word isn't wide enough to poke both clean zones, the body eats 50-70% of the crown and it reads as "THE \_\_\_ S".
+**Fix**: Check subject center_x vs frame center before choosing crown strategy (see [layout-heuristics.md § Crown placement](layout-heuristics.md)). Either size the crown large enough to cross both clean zones, or shift it into the larger clean zone with a narrower width. Keep `text-align: center` within the (re-positioned) box, not the frame.
+### Crown font too small for frame → half-swallowed
+On landscape 1920×1080, a centered crown at 140px is only ~500px wide. If the subject is 700-900px wide, the crown vanishes behind them. Symptom: only "THE" and "S" visible for "THE BEATLES".
+**Fix**: For full-frame centered crown, font must be big enough that `crown_width > subject_width + 400`. See font-size table in [typography-presets.md](typography-presets.md). Prefer 180-260px for landscape centered crowns.
+### Wider caption column + default font size → underweight
+Template defaults (66/78/92/140) assume a ~560px column. Expanding the plane to 700+ px without bumping fonts makes the text feel too small relative to the column.
+**Fix**: Column width 600-760 → phrase ~108px, emph ~128px, crown ~220px centered. See table in [typography-presets.md](typography-presets.md).
+## Scene admission
+### Handheld / fast camera → matte alpha flickers
+The matte is temporally smoothed (EMA in matte.cjs) but moderate motion only. Fast camera pans (vlog selfie) still produce frame-to-frame alpha jitter that shows as flickering caption edges.
+**Fix**: SKILL.md decision gate lists handheld vlog as `⚠️` — degrade or refuse. Production version would add `vidstabdetect` compensation; v1 does not.
+### Multi-person scenes → matte is ambiguous
+The human-matting model segments "foreground" as a single alpha. With two people, it captures both but they're treated as one blob. Captions can't go "behind person A but in front of person B". If the scene has both speakers visible, the skill should split by shot boundaries first.
+**Fix**: Either pre-split with PySceneDetect + per-shot planning, or refuse for `⚠️` multi-subject scenes.
+### Undetected shot cut inside the clip
+Some sources (interviews, broadcast clips) cut to B-roll/archive footage mid-clip. The matte + caption layout assumes the subject is the same person throughout — a hard cut mid-render creates garbage (caption placement relative to old subject applied to a totally different shot).
+**Fix for v1**: Probe the clip at 3-5 timestamps before accepting it. If any frame doesn't contain the primary subject at the same position, trim the clip to end before the cut. `Steve Jobs 60 Minutes` cut to Beatles archival at t≈9s, was manually trimmed to 8.5s.
+**Fix for future**: Run PySceneDetect as part of the admission gate; split into shots automatically.
+### Pillarbox / letterbox content with baked-in graphics
+TV archive clips often have pillarbox (black bars at sides) plus baked lower-third logos ("60 Overtime") and date labels ("2003"). These are part of the video file, can't be matted out, and shrink the usable frame area.
+**Fix**: Detect black-bar margins in the first frame (scan rows/columns for all-black). Constrain caption safe-zone to `[left_margin+pad, right_margin-pad, top_margin+pad, bottom_margin-pad]`. For Jobs, safe content zone was x=280-1640 (not 0-1920). Layout positions must respect this.
+### transcribe.cjs sees an existing transcript and skips
+Transcription is Whisper now, via `transcribe.cjs` (it wraps `hyperframes transcribe` — no API key). `hyperframes init --video <mp4>` may itself auto-write a `transcript.json` in hyperframes' raw whisper shape (a flat word array, no top-level `language_code`). `transcribe.cjs` only treats a transcript as done when it's ALREADY in our normalized schema (`{ words: [...], language_code }`); otherwise it (re-)runs Whisper and normalizes the flat word list into `{ words:[{text,start,end,type:"word"}], language_code }`.
+**Fix / expectation**: If you init via hyperframes first, expect `transcribe.cjs` to normalize that transcript into our schema. Don't hand-leave a half-normalized file (e.g. our `words` shape but no `language_code`) — that's the one state the skip-guard can misread.
+### Transcript gaps > 3s with no speech
+If the original audio has a long silence, leaving the caption plane empty feels wrong mid-video. Either (a) let the prior caption linger, (b) show a title-card crown during the silence, or (c) cut the silence out of the clip.
+**Fix**: During caption-grouping, detect silences > 3s. The skill should prompt the user about this, not silently proceed.