@lalalic/markcut 2.5.1 → 2.7.0

package/AGENTS.md

@@ -214,20 +214,19 @@ node src/render/cli.mjs templates
 
 1. Basic Rendering — empty composition, dimension verification
 2. Image + Subtitle — image rendering, inline/VTT/karaoke subtitles
-3. Built-in Components — AnimatedHeadline, StatCounter, GradientBackground
-4. Effects — fadeIn, bounceIn, custom keyframes
-5. Map Rendering — canvas route visualization
-6. Audio Rendering — audio track, non-silent verification
-7. Include — nested composition via include
-8. Full Feature Combination — multiple features
-9. Scene Node — scene-based rendering
-10. Audio STT — speech-to-text verification via whisper
-11. Multiple Aspect Ratios — 16x9, 9x16, 1x1
-12. Cross-Stream Types — parallel rendering, effect-wrapped children
-13. Frame-Accurate — timeline consistency, frame count
-14. Description Field — schema validation, render preservation
-15. Scene as Folder Alias — scene/folder interchangeability
-16. **Video startFrom/endAt Trimming** — video trimming + photo in series
+3. Effects — fadeIn, bounceIn, custom keyframes
+4. Map Rendering — canvas route visualization
+5. Audio Rendering — audio track, non-silent verification
+6. Include — nested composition via include
+7. Full Feature Combination — multiple features
+8. Scene Node — scene-based rendering
+9. Audio STT — speech-to-text verification via whisper
+10. Multiple Aspect Ratios — 16x9, 9x16, 1x1
+11. Cross-Stream Types — parallel rendering, effect-wrapped children
+12. Frame-Accurate — timeline consistency, frame count
+13. Description Field — schema validation, render preservation
+14. Scene as Folder Alias — scene/folder interchangeability
+15. **Video startFrom/endAt Trimming** — video trimming + photo in series
 
 ### Running Tests
 
@@ -242,9 +241,9 @@ npx vitest run tests/render.test.ts -t "Test Name"  # single test
 1. **TransitionSeries + isBackground**: `TransitionSeries` from `@remotion/transitions` rejects non-Sequence/Transition children. Don't mix `isBackground:true` children with `isSeries:true, transition:"..."` at the same level.
 2. **Video endAt formula**: Must be `startFrom_frames + ((endAt_source - startFrom_source) * fps / playbackRate)` — changing to `endAt_source * fps` breaks slow-motion playback.
 3. **`signalstats` ffmpeg filter**: Doesn't output YAVG on all systems; use file size as proxy for visual content.
-4. **Asset paths**: All `src` paths in markdown are resolved relative to the `.md` file's directory. Preview mode serves files via a multi-directory HTTP server; render mode uses absolute paths directly (bypassing `staticFile()`). Never use a `public/` prefix in markdown `src`.
+4. **Asset paths (unified preview + render)**: All `src` paths in markdown are resolved relative to the `.md` file's directory. Both surfaces store every project-local `src` **relative to the `.md` folder** (e.g. `.markcut/generated/tts/x.mp3`, `assets/pic.png`). Preview: the server's `makePathsRelative` strips the project dir and `resolveAssetPaths`/`toServerUrl` turn them into `/`-rooted URLs served from the project folder. Render: `cli.mjs relativizePaths(streamTree, mdDir)` does the same, then Remotion is invoked with `--public-dir=<mdDir>` so `staticFile()` resolves them (including `fetch()`-based assets like subtitles and includes). Never use a `public/` prefix in markdown `src`.
 5. **Map rendering**: Depends on external tile availability; may produce blank frames offline.
-6. **CLI md render needs two things preview gets for free**: (a) `parseMarkdownVariants(raw)` (returns `{base, variants}`) — `parseMarkdownDescriptive` returns the root directly; (b) bundling the ` ```js imports ``` ` block via `bundleFromEntries` and setting `root.imports` (otherwise jsx components are unregistered → slides render as unstyled black frames). Asset staging (`stageLocalAssets`) was removed — render now keeps absolute paths from `resolve.ts`, bypassing `staticFile()` entirely. All live in `src/render/cli.mjs`; the import-map shim for shared react/remotion instances is `src/utils/component-import-map.ts` (must stay in sync with `player/browser.tsx` + `bundler.mjs getSharedExternals`).
+6. **CLI md render needs two things preview gets for free**: (a) `parseMarkdownVariants(raw)` (returns `{base, variants}`) — `parseMarkdownDescriptive` returns the root directly; (b) bundling the ` ```js imports ``` ` block via `bundleFromEntries` and setting `root.imports` (otherwise jsx components are unregistered → slides render as unstyled black frames). Render and preview now share one folder layout **and** one path scheme: generated artifacts live in `.markcut/generated/{tts,media,includes,components}` (content-addressed, shared caches) and per-variant subtitles/compiled.json in `.markcut/<basename>/<label>/` (label defaults to `default`). Render makes paths relative to the `.md` folder and serves it via `--public-dir` (see gotcha 4) — do NOT revert to absolute paths, which break `fetch()`-based assets (subtitles, includes) that hit the render server origin and 404. All live in `src/render/cli.mjs`; the import-map shim for shared react/remotion instances is `src/utils/component-import-map.ts` (must stay in sync with `player/browser.tsx` + `bundler.mjs getSharedExternals`).
 7. **Publishing**: verify `npx @lalalic/markcut render <md>` from a clean temp dir before/after publish — v1.1.1 shipped with cli.mjs/pipeline.mjs out of sync and md rendering was completely broken. Also beware stale npx caches (`~/.npm/_npx`) silently running old code.
 8. **Clean break from `actions[]`**: leaf timing is now flat base fields (`start`/`end`/`duration`/`startFrom`/`endAt`). Old compiled JSON carrying `actions:[{...}]` no longer parses into a duration — re-compile from markdown/descriptive JSON, or migrate `actions[0]` → flat fields (drop `id`/`volume`/`style`/`effectId`; keep `start`/`end`/`startFrom`/`endAt`/`loop`).
 

package/README.md

@@ -215,6 +215,95 @@ src/
 ```bash
 npm run render      # render a JSON stream tree to MP4
 npm run preview     # open Remotion Studio
+npm run vision      # analyze images/videos in a folder (metadata + AI perception)
 npm run typecheck   # TypeScript type check
 npm test            # run unit + integration tests
 ```
+
+## Vision Pipeline
+
+```bash
+# Full pipeline: extract metadata → normalize → AI perception (images & videos)
+npx markcut vision <folder>
+
+# With interactive labeling step before AI
+npx markcut vision <folder> --label
+
+# Provide context about people/places
+npx markcut vision <folder> --instruct "the girl is Maggie, the man in red is me"
+```
+
+```mermaid
+flowchart LR
+    subgraph Input[Input]
+        A[Media Folder<br/>images + videos]
+    end
+
+    subgraph Metadata["1. Metadata Extraction"]
+        B[exiftool / ffprobe]
+        C[metadata.json<br/>width, height, created,<br/>GPS location, duration]
+    end
+
+    subgraph Normalize["2. Normalization"]
+        D[ffmpeg resize<br/>images → 384px<br/>videos → 360p]
+    end
+
+    subgraph Label["Optional: Labeling"]
+        E[Build preview tree<br/>with seed]
+        F[Label preview server<br/>user annotates scenes]
+        G[Merge labels → userHints<br/>into metadata.json]
+    end
+
+    subgraph Image["3. Image Perception"]
+        H[ITT: image-to-text<br/>mlx-vlm model]
+        I[Perception saved:<br/>description]
+    end
+
+    subgraph Video["4. Video Understanding"]
+        J[VTT: video-to-text<br/>mlx-vlm model]
+        K[STT: speech-to-text<br/>whisper]
+        L[VTT→ overall desc]
+        M[STT → VTT subtitles]
+        N[Build merged segment<br/>boundaries]
+        O[LLM merges boundaries<br/>into final segments]
+        P[Per-segment vision<br/>clip extraction + ITT]
+    end
+
+    subgraph Output[Output]
+        Q[metadata.json<br/>with perception desc,<br/>subtitles & segments]
+    end
+
+    A --> B --> C --> D
+    D --> Image
+    D --> Video
+    C -.->|--label| E --> F --> G --> D
+    Image --> H --> I --> Q
+    Video --> J --> L
+    Video --> K --> M
+    L --> N
+    M --> N
+    N --> O --> P --> Q
+
+    style Input fill:#1a1a2e,stroke:#16213e,color:#fff
+    style Metadata fill:#0f3460,stroke:#16213e,color:#fff
+    style Normalize fill:#533483,stroke:#16213e,color:#fff
+    style Label fill:#e94560,stroke:#16213e,color:#fff
+    style Image fill:#0f3460,stroke:#16213e,color:#fff
+    style Video fill:#0f3460,stroke:#16213e,color:#fff
+    style Output fill:#1a1a2e,stroke:#16213e,color:#fff
+```
+
+### Vision CLI Options
+
+| Option | Description |
+|---|---|
+| `--label` | Add interactive labeling step before AI pipeline (opens browser preview) |
+| `--instruct "text"` | Background context about people/places (injected into AI prompts) |
+| `--prompts-file <path>` | Custom prompts markdown file (default: `src/vision/vision_prompts.md`) |
+| `--vtt-sample-interval <n>` | Sample one video frame every N seconds for VTT (default: 5) |
+| `--pick <files>` | Comma-separated filenames to process (skip others) |
+| `--skip-stt` | Skip speech-to-text for videos |
+| `--dry-run` | Show what would be processed without running AI |
+| `--show-prompts` | Print the prompts template file and exit |
+
+## Architecture

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lalalic/markcut",
-  "version": "2.5.1",
+  "version": "2.7.0",
   "description": "Markdown-to-video engine. Describe scenes in markdown, get a rendered video.",
   "bin": {
     "markcut": "bin/markcut"

package/skills/markcut/SKILL.md

@@ -45,7 +45,7 @@ npx @lalalic/markcut --show-clis # get command specific information
 npx @lalalic/markcut preview <file> # assemble and preview the video with a local server, and chat to edit the video and auto refresh
 npx @lalalic/markcut render <file> # render the video to mp4
 npx @lalalic/markcut vision <folder> # vision understanding medias in folder
-npx @lalalic/markcut vision <folder> --label # full pipeline vision understanding medias in folder, and provide ui to label the medias with text, time ranges
+npx @lalalic/markcut vision <folder> --label # an extra step to provide UI to label the medias with text, time ranges
 
 ```
 

package/skills/markcut/docs/markdown-descriptive.md

@@ -16,7 +16,7 @@ This section defines the markcut Markdown Descriptive grammar. An agent should r
 │   ───...───
 ├─ # video                    ← Root node (REQUIRED, heading level 1)
 │   width:1080 height:1920 fps:30 layout:series
-│   subtitle:{src:"captions.vtt",type:"Typewriter"}
+│   subtitle:{type:"Typewriter"}
 │   ~~~css stylesheet           ← root.stylesheet
 │   .bg { background: #000; }
 │   ~~~
@@ -223,7 +223,7 @@ The line after `# video` holds canvas-level config as space-separated `key:value
 ```markdown
 # video
 width:1920 height:1080 fps:30 layout:series transition:fade transitionTime:1.2
-subtitle:{src:"captions.vtt",type:"Typewriter",fontSize:48}
+subtitle:{type:"Typewriter",fontSize:48}
 ```
 
 Supported root keys: `width`, `height`, `fps`, `layout`, `tts`, `stt`, `tti`, `ttv`, `transition`, `transitionTime`, `instruction`, `metadata`, `stylesheet`, `subtitle`, `voices`.

package/src/components/StoryboardCaption.tsx

@@ -0,0 +1,74 @@
+import * as React from "react";
+
+/**
+ * Built-in component rendered by `--storyboard` preview mode instead of a
+ * TTS-generated audio clip. Shows the script text as a caption-style bar at
+ * the bottom of the frame.
+ *
+ * Usage in compiled JSON:
+ *   <StoryboardCaption script={script} speaker={speaker} />
+ *
+ * No imports or frontmatter registration needed — it's a built-in.
+ */
+export interface StoryboardCaptionProps {
+  /** The narration/dialogue text */
+  script: string;
+  /** Optional speaker name (for multi-turn dialogue) */
+  speaker?: string;
+}
+
+export function StoryboardCaption({ script, speaker }: StoryboardCaptionProps) {
+  return (
+    <div
+      style={{
+        position: "absolute",
+        bottom: "10%",
+        left: "5%",
+        right: "5%",
+        display: "flex",
+        flexDirection: "column",
+        alignItems: "center",
+        pointerEvents: "none",
+      }}
+    >
+      <div
+        style={{
+          maxWidth: "90%",
+          background: "rgba(0,0,0,0.7)",
+          backdropFilter: "blur(4px)",
+          borderRadius: 16,
+          padding: "2.5vh 4vh",
+          textAlign: "center",
+          color: "#fff",
+          fontFamily: "system-ui, -apple-system, Segoe UI, Roboto, sans-serif",
+        }}
+      >
+        {speaker && (
+          <div
+            style={{
+              fontSize: "2vh",
+              fontWeight: 700,
+              color: "#93c5fd",
+              marginBottom: "0.6vh",
+              letterSpacing: "0.04em",
+            }}
+          >
+            {speaker}
+          </div>
+        )}
+        <div
+          style={{
+            fontSize: "2.6vh",
+            lineHeight: 1.5,
+            fontWeight: 500,
+            textShadow: "0 1px 4px rgba(0,0,0,0.4)",
+            whiteSpace: "pre-wrap",
+            wordBreak: "break-word",
+          }}
+        >
+          {script}
+        </div>
+      </div>
+    </div>
+  );
+}

package/src/components/StoryboardInfo.tsx

@@ -0,0 +1,195 @@
+import * as React from "react";
+
+/**
+ * Built-in component rendered by `--storyboard` preview mode as the first
+ * frame of the video. Shows all project frontmatter config and variant info
+ * so the user has full context before reviewing the storyboard scenes.
+ *
+ * All props are strings (the compiler only collects string bindings).
+ *
+ * Usage in compiled JSON:
+ *   <StoryboardInfo config="width: 1080 · height: 1920 · fps: 30 · layout: series"
+ *                   variants="default, zh-tiktok" />
+ *
+ * No imports or frontmatter registration needed — it's a built-in.
+ */
+export interface StoryboardInfoProps {
+  /** Formatted string of all root config fields (separated by " · ") */
+  config?: string;
+  /** Comma-separated list of variant labels */
+  variants?: string;
+  /** Raw YAML frontmatter from the markdown source (if any) */
+  frontmatter?: string;
+}
+
+export function StoryboardInfo({ config, variants, frontmatter }: StoryboardInfoProps) {
+  const parts = (config ?? "").split("  ·  ").filter(Boolean);
+  const variantList = (variants ?? "").split(/,\s*/).filter(Boolean);
+
+  return (
+    <div
+      style={{
+        position: "absolute",
+        inset: 0,
+        display: "flex",
+        flexDirection: "column",
+        alignItems: "center",
+        justifyContent: "center",
+        background: "linear-gradient(135deg,#0f172a,#1e293b,#334155)",
+        color: "#fff",
+        fontFamily: "system-ui, -apple-system, Segoe UI, Roboto, sans-serif",
+        padding: "8%",
+        boxSizing: "border-box",
+      }}
+    >
+      {/* decorative rings */}
+      <div
+        style={{
+          position: "absolute",
+          inset: "5%",
+          border: "2px solid rgba(255,255,255,0.08)",
+          borderRadius: "50%",
+        }}
+      />
+      <div
+        style={{
+          position: "absolute",
+          inset: "12%",
+          border: "1px solid rgba(255,255,255,0.05)",
+          borderRadius: "50%",
+        }}
+      />
+
+      {/* header icon */}
+      <div style={{ fontSize: "6vh", marginBottom: "2.5vh", opacity: 0.7 }}>🎬</div>
+
+      {/* title */}
+      <div
+        style={{
+          fontSize: "4.2vh",
+          fontWeight: 800,
+          letterSpacing: "0.02em",
+          marginBottom: "3vh",
+          textAlign: "center",
+          textShadow: "0 2px 12px rgba(0,0,0,0.5)",
+        }}
+      >
+        Storyboard Preview
+      </div>
+
+      {/* config fields as a table */}
+      {parts.length > 0 && (
+        <div
+          style={{
+            display: "flex",
+            flexWrap: "wrap",
+            justifyContent: "center",
+            gap: "1.2vh",
+            maxWidth: "85%",
+            marginBottom: variantList.length > 0 ? "2.5vh" : 0,
+          }}
+        >
+          {parts.map((entry) => {
+            const colonIdx = entry.indexOf(":");
+            const label = colonIdx > 0 ? entry.slice(0, colonIdx).trim() : "";
+            const value = colonIdx > 0 ? entry.slice(colonIdx + 1).trim() : entry;
+            return (
+              <div
+                key={label || entry}
+                style={{
+                  background: "rgba(255,255,255,0.07)",
+                  border: "1px solid rgba(255,255,255,0.12)",
+                  borderRadius: 12,
+                  padding: "1.2vh 2.2vh",
+                  textAlign: "center",
+                  minWidth: "16vh",
+                }}
+              >
+                {label && (
+                  <div
+                    style={{
+                      fontSize: "1.6vh",
+                      fontWeight: 600,
+                      color: "#93c5fd",
+                      letterSpacing: "0.08em",
+                      textTransform: "uppercase" as const,
+                      marginBottom: "0.4vh",
+                    }}
+                  >
+                    {label}
+                  </div>
+                )}
+                <div
+                  style={{
+                    fontSize: "2.2vh",
+                    fontWeight: 700,
+                    whiteSpace: "pre-wrap",
+                    wordBreak: "break-word",
+                    opacity: value.startsWith("http") ? 0.8 : 1,
+                  }}
+                >
+                  {value}
+                </div>
+              </div>
+            );
+          })}
+        </div>
+      )}
+
+      {/* frontmatter block */}
+      {frontmatter && (
+        <div
+          style={{
+            maxWidth: "85%",
+            background: "rgba(255,255,255,0.05)",
+            border: "1px solid rgba(255,255,255,0.1)",
+            borderRadius: 12,
+            padding: "1.5vh 2.5vh",
+            marginBottom: "2.5vh",
+            fontFamily: "monospace",
+            fontSize: "1.8vh",
+            lineHeight: 1.6,
+            color: "rgba(255,255,255,0.8)",
+            whiteSpace: "pre-wrap",
+            wordBreak: "break-word",
+            textAlign: "left",
+            maxHeight: "28vh",
+            overflow: "auto",
+          }}
+        >
+          {frontmatter}
+        </div>
+      )}
+
+      {/* variants */}
+      {variantList.length > 0 && (
+        <div
+          style={{
+            display: "flex",
+            flexWrap: "wrap",
+            justifyContent: "center",
+            gap: "1vh",
+          }}
+        >
+          {variantList.map((v) => (
+            <div
+              key={v}
+              style={{
+                background: "rgba(251,191,36,0.15)",
+                border: "1px solid rgba(251,191,36,0.35)",
+                borderRadius: 999,
+                padding: "0.8vh 2vh",
+                fontSize: "1.8vh",
+                fontWeight: 600,
+                color: "#fbbf24",
+                letterSpacing: "0.04em",
+              }}
+            >
+              {v}
+            </div>
+          ))}
+        </div>
+      )}
+    </div>
+  );
+}

package/src/components/StoryboardSlot.tsx

@@ -0,0 +1,96 @@
+import * as React from "react";
+
+/**
+ * Built-in component rendered by `--storyboard` preview mode instead of a
+ * generated image (TTI) or video (TTV). Shows the generation prompt with a
+ * clear type indicator so the user can review the story structure fast and
+ * chat to reshape it before committing to slow media generation.
+ *
+ * Usage in markdown / compiled JSON:
+ *   <StoryboardSlot kind="image" prompt={prompt} />
+ *   <StoryboardSlot kind="video" prompt={prompt} />
+ *
+ * The `prompt` variable comes from the component node's data bindings.
+ * No imports or frontmatter registration needed — it's a built-in.
+ */
+export interface StoryboardSlotProps {
+  kind: "image" | "video";
+  prompt?: string;
+}
+
+export function StoryboardSlot({ kind, prompt }: StoryboardSlotProps) {
+  const isVideo = kind === "video";
+
+  const bgGrad = isVideo
+    ? "linear-gradient(135deg,#7c2d12,#b91c1c)"
+    : "linear-gradient(135deg,#1e3a8a,#6d28d9)";
+  const accentColor = isVideo ? "#fbbf24" : "#93c5fd";
+  const label = isVideo ? "Video" : "Image";
+  const emoji = isVideo ? "🎬" : "🖼️";
+
+  return (
+    <div
+      style={{
+        position: "absolute",
+        inset: 0,
+        overflow: "hidden",
+        display: "flex",
+        flexDirection: "column",
+        alignItems: "center",
+        justifyContent: "center",
+        background: bgGrad,
+        color: "#fff",
+        fontFamily: "system-ui, -apple-system, Segoe UI, Roboto, sans-serif",
+        padding: "6%",
+        boxSizing: "border-box",
+      }}
+    >
+      {/* dashed border frame */}
+      <div
+        style={{
+          position: "absolute",
+          inset: "4%",
+          border: "3px dashed rgba(255,255,255,0.35)",
+          borderRadius: 24,
+          pointerEvents: "none",
+        }}
+      />
+
+      {/* type badge */}
+      <div
+        style={{
+          display: "flex",
+          alignItems: "center",
+          gap: "0.5em",
+          background: "rgba(0,0,0,0.35)",
+          border: `2px solid ${accentColor}`,
+          borderRadius: 999,
+          padding: "0.5em 1.1em",
+          fontSize: "2.2vh",
+          fontWeight: 800,
+          letterSpacing: "0.12em",
+          marginBottom: "3vh",
+        }}
+      >
+        <span style={{ fontSize: "3.4vh", lineHeight: 1 }}>{emoji}</span>
+      </div>
+
+      {/* prompt text */}
+      <div
+        style={{
+          position: "relative",
+          maxWidth: "88%",
+          textAlign: "center",
+          fontSize: "3.4vh",
+          lineHeight: 1.4,
+          fontWeight: 600,
+          textShadow: "0 2px 8px rgba(0,0,0,0.5)",
+          whiteSpace: "pre-wrap",
+          wordBreak: "break-word",
+        }}
+      >
+        {prompt ?? (isVideo ? "(no video prompt)" : "(no image prompt)")}
+      </div>
+    </div>
+  );
+}

package/src/config.mjs

@@ -20,7 +20,7 @@ export const args=(function parseArgs(argv) {
     "--tti": "tti",
     "--ttv": "ttv"
   };
-  const args = { command: "", file: "", output: "", forceNew: false, verbose: false, dev: false, label: false, edit: false, noBrowser: false, chat: false, port: 3001, compile: false, cli: false, showClis: false, scriptOutputDir: "", mediaOutputDir: "", variant: [], cliOverrides: {} };
+  const args = { command: "", file: "", output: "", forceNew: false, verbose: false, dev: false, label: false, edit: false, storyboard: false, noBrowser: false, chat: false, port: 3001, compile: false, cli: false, showClis: false, scriptOutputDir: "", mediaOutputDir: "", variant: [], cliOverrides: {} };
   let i = 2;
   if (argv[i]) args.command = argv[i++];
   if (argv[i] && !argv[i].startsWith("--")) args.file = argv[i++];
@@ -37,6 +37,7 @@ export const args=(function parseArgs(argv) {
     else if (flag === "--dev") args.dev = true;
     else if (flag === "--label") args.label = true;
     else if (flag === "--edit") args.edit = true;
+    else if (flag === "--storyboard") args.storyboard = true;
     else if (flag === "--no-browser") args.noBrowser = true;
     else if (flag === "--port" && argv[i]) args.port = parseInt(argv[i], 10);
     else if (flag.startsWith("--port=")) args.port = parseInt(flag.split("=")[1], 10);