@lalalic/markcut 1.0.0

package/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: markcut
+description: >-
+  Compose and render videos from streamline trees. CLI renders stream tree
+  to MP4. Run via `npx markcut` — no install, no code.
+---
+
+## Stream Tree Specs
+
+Everything video is a **stream tree**. see [docs/markdown-descriptive.md](docs/markdown-descriptive.md) for full details.
+
+## .markcut/ Directory Layout
+
+When you run `preview` or `render`, all generated artifacts live under `.markcut/` next to the source file:
+
+```
+.markcut/
+  generated/                    ← shared, content-addressed
+    tts/                        ← TTS audio (content-hash filenames)
+    media/                      ← TTI/TTV media (content-hash filenames)
+    includes/                   ← compiled sub-video JSON (content-hash)
+  <basename>/                   ← per-source-file artifacts
+    components/                 ← component bundles (per-file: imports differ)
+```
+
+- `generated/` — shared artifacts using content-hash filenames, so identical scripts/prompts across different source files reuse the same cached file
+- `<basename>/components/` — per-file component bundles; each source file has its own `imports` block, so bundles live under its basename
+- The server serves `.markcut/` as a document root, so components at `.markcut/slides/components/abc.js` are served as `/slides/components/abc.js`
+
+## Include (Sub-Video Embedding)
+
+The `include` node embeds an external `.md` file as a sub-video. The pipeline fully resolves it recursively:
+
+1. **`resolveIncludes()`** (step 0 of `resolveAll`): reads the referenced `.md`, parses it, resolves its own includes/TTS/media, compiles to a stream tree, writes the compiled JSON to `.markcut/generated/includes/<hash>.json`
+2. **`resolveIncludeImports()`** (server post-compile): if the sub-video has a ` ```js imports ``` ` block, bundles its components independently and writes the bundle URL into the compiled JSON
+3. **`IncludeLeaf`** (render): loads the compiled JSON, dynamically imports the sub-video's component bundle, creates a nested `ComposeContext` with the sub-video's registry merged, and renders via `FolderLeaf`
+
+The sub-video's own components are isolated from the parent's — naming conflicts are resolved by "sub-video wins" priority.
+
+```markdown
+# video
+## Main
+- component duration:3 jsx:"<MainTitle />"
+## Embedded
+- include src:./sub-video.md
+## End
+- component duration:2 jsx:"<Outro />"
+```
+
+```markdown
+# sub-video
+- component duration:3 jsx:"<SubSlide scene='1' />"
+
+```js imports
+export function SubSlide({ scene }) {
+  return <div style={{background:'#667eea', ...}}>Scene {scene}</div>;
+}
+```
+```
+---
+
+## Video Design Best Practice (How-To)
+
+The engine supports four authoring phases. Each has a dedicated doc with full detail.
+
+### Label — annotate clips in a stream tree
+
+Load a stream tree, preview each clip, and attach descriptive labels.
+
+```bash
+npx markcut preview <flat stream tree file> --label
+```
+
+Labels save to `labels.json`. See [docs/label-mode.md](docs/label-mode.md) for:
+- Label player UI (thumbnails, player, label input)
+- Labels persistence format
+- Full workflow diagram
+
+### Storyboard — plan video structure with scene nodes
+
+Use `scene` nodes to organize your video. Each scene has `title`, `instruction`, `script` (TTS narration), `children`. Scenes can nest inside other scenes.
+
+```bash
+see [docs/markdown-descriptive.md](docs/markdown-descriptive.md) for full details.
+
+### 3. Assemble — render stream tree to MP4
+
+```bash
+npx markcut render <stream tree file> --aspect all  # MP4 output
+```
+
+---
+
+## Template Variables
+
+`${width}`, `${height}`, `${fps}`, `${variant}` can be used in `src`,
+`prompt`, and `stylesheet` values. They resolve to the root config values.
+
+```markdown
+# video
+width:1920 height:1080
+## Scene
+- image src:photo_${width}x${height}.jpg
+- component jsx:"<h1>${width}</h1>"    ← NOT resolved (jsx excluded)
+```
+
+> Only `src`, `prompt`, and `stylesheet` support template variables.
+> Root config keys, jsx, script, style, etc. do not.
+
+## Frontmatter
+
+The YAML `---` block is **metadata only**. It does NOT affect video config.
+All configuration comes from the root config line after `# video`:
+
+```markdown
+---
+title: My Campaign
+---
+# video
+width:640 height:480 fps:30 layout:series tts:"edge-tts --voice '...' --text '{input}' --write-media '{output}'"
+```
+
+## AI Media Generation (TTS / STT / TTI(text-to-image) / TTV(text-to-video))
+
+All four pipelines are configured via a **single CLI string** on the root config line. The user/LLM embeds every tool-specific parameter directly in the string — only `{input}` and `{output}` are substituted by the engine.
+
+Default CLI templates (used when not overridden on the root config line):
+
+| Config | Default |
+|--------|---------|
+| `tts` | `uvx edge-tts --voice "en-US-GuyNeural" --text "{input}" --write-media "{output}"` |
+| `stt` | `uvx --from openai-whisper whisper "{input}" --output_format vtt --output_dir "{output}"` |
+| `tti` | `uvx --from mflux mflux-generate-flux2 --model flux2-klein-4b --steps 5 --prompt "{input}" --output "{output}"` |
+| `ttv` | (uses TTI to create an image, then ffmpeg to loop it for 5s) |
+
+Override on the root config line:
+
+```markdown
+# video
+width:1920 height:1080 tts:"uvx edge-tts --voice 'zh-CN-YunxiNeural' --text '{input}' --write-media '{output}'"
+```
+
+---
+
+## 3. CLI
+
+```bash
+npx markcut <command> [options]
+```
+
+### Commands
+
+| Command | Description |
+|---------|-------------|
+| `render <file>` | Render stream tree to MP4 |
+| `preview <file> --label` | Label clips in a simplified stream tree |
+| `preview <file> --edit` | Live editing loop (auto-reload on file change) |
+| `verify <file>` | Validate descriptive file + check imports |
+### Options
+
+| Flag | Values | Default |
+|------|--------|---------|
+| `--aspect` | `16x9` / `9x16` / `1x1` / `all` | `16x9` |
+| `--output` | path | `out/video-{aspect}.mp4` |
+| `--port` | number | `3001` |
+| `--verbose` | flag | `false` (compact progress) |
+| `--compile` | flag | `false` (resolve only) |
+| `--script-output-dir` | dir | `.markcut/generated/tts/` |
+| `--media-output-dir` | dir | `.markcut/generated/media/` |
+
+> Default output dirs auto-resolve to `.markcut/<basename>/<type>/` relative to the source file. You only need `--script-output-dir` / `--media-output-dir` to override.
+
+### Edit Mode for Agents
+
+```bash
+# Start player in background
+npx markcut preview <stream tree file> --edit --port 3001 &
+
+# Player auto-opens browser. Agent edits JSON → player auto-reloads.
+# User clicks ✕ → server exits → agent regains control.
+# Browser feedback input writes to feedback.txt.
+```
+
+---
+
+## self verification
+some common issues (photo or video can't be displayed, audio missing), take below actions to verify
+### preview or render mode
+- screenshot some key frames, and understand image to verify if the image is correct
+### final video
+- screenshot some key frames, and understand image to verify if the image is correct
+- stt the final video audio, and verify if vtt result is correct
+
+
+## Reference
+
+| Topic | File |
+|-------|------|
+| Markdown descriptive format (primary authoring format) | [docs/markdown-descriptive.md](docs/markdown-descriptive.md) |
+| **Video Templates** — ready-to-use markdown for common scenarios | **[docs/templates/](docs/templates/)** |
+| ┣ Courseware / 课件 | [docs/templates/courseware.md](docs/templates/courseware.md) |
+| ┣ Product Ad / 产品广告 | [docs/templates/product-ad.md](docs/templates/product-ad.md) |
+| ┣ Movie Review / 影视讲解 | [docs/templates/movie-review.md](docs/templates/movie-review.md) |
+| ┣ Audiobook / 有声图书 | [docs/templates/audiobook.md](docs/templates/audiobook.md) |
+| ┣ Story Video / 故事视频 | [docs/templates/story-video.md](docs/templates/story-video.md) |
+| ┣ Travel Log / 旅行日志 | [docs/templates/travel-log.md](docs/templates/travel-log.md) |
+| Label system (browse, label, export labels.json) | [docs/label-mode.md](docs/label-mode.md) |
+| Player servers (label + edit mode) | [docs/edit-mode.md](docs/edit-mode.md) |
+| Template overview and TTI/TTV config | [docs/templates.md](docs/templates.md) |

package/docs/dynamic-components.md

@@ -0,0 +1,191 @@
+# Dynamic Components
+
+Three ways to add custom visual content without rebuilding the engine.
+
+## 1. Effects on Any Node (CSS Keyframes)
+
+Add the `effects` key to any node to apply CSS keyframe animations. No wrapper node needed.
+
+```json
+{
+  "id": "animated-scene",
+  "type": "image",
+  "src": "photo.jpg",
+  "duration": 3,
+  "effects": ["bounceIn(1, ease-out, 2)"]
+}
+```
+
+### 25+ Built-in Animation Names
+
+| Fades | Slides | Zooms | Attention | Bounce | Rotations |
+|-------|--------|-------|-----------|--------|-----------|
+| `fadeIn` | `slideInDown` | `zoomIn` | `pulse` | `bounceIn` | `rotateIn` |
+| `fadeOut` | `slideInUp` | `zoomOut` | `flash` | | `rotateOut` |
+| `fadeInDown` | `slideInLeft` | | `bounce` | | |
+| `fadeInUp` | `slideInRight` | | `heartBeat` | | |
+| `fadeInLeft` | | | `rubberBand` | | |
+| `fadeInRight` | | | `shakeX` | | |
+| (8 total) | | | | | |
+
+### Custom Keyframes
+
+```json
+{
+  "type": "effect",
+  "animation": "custom",
+  "customKeyframes": {
+    "0":  { "opacity": "0", "transform": "scale3d(0,0,0) rotate(0deg)" },
+    "50": { "opacity": "0.5", "transform": "scale3d(1.2,1.2,1.2) rotate(180deg)" },
+    "100": { "opacity": "1", "transform": "scale3d(1,1,1) rotate(360deg)" }
+  },
+  "children": [...],
+  "actions": [{ "start": 0, "end": 2 }]
+}
+```
+
+Percentages `"0"`–`"100"` map to action duration. Any numeric CSS property works.
+
+## 2. Remote Components (bundler-based)
+
+Register remote React components via the `~~~js imports` code fence (markdown) or `root.imports` array (JSON descriptive). The player server bundles all registered components into a single ESM module at startup.
+
+```markdown
+~~~js imports
+export { BarChart } from "recharts"
+export { Hello } from "git:user/repo/path/to/Hello.tsx"
+export { Badge } from "https://cdn.example.com/components/badge.js"
+
+export function Slide(props) {
+  return <div style={{color: '#fff'}}>{props.children}</div>;
+}
+~~~
+```
+
+Then use them in component JSX:
+
+```json
+{
+  "type": "component",
+  "jsx": "<Badge text='LIVE' color='#ff0000' />",
+  "actions": [{ "start": 0, "end": 3 }]
+}
+```
+
+### How it works
+
+1. The server extracts the imports block from the source file
+2. `parseImportsBlock` resolves `export { X } from "spec"` and `export function X() {}` patterns
+3. Inline functions' source code is scanned for `import X from "pkg"` statements — those packages are added to dependencies
+4. `bundleFromEntries` creates a temp npm project, installs all packages, and runs `esbuild` to produce a single ESM file
+5. The bundled file URL is set on `root.imports` and served from `/.component-cache/<hash>.js`
+6. At runtime, `MarkCut.useComponentRegistry` does a dynamic `import(url)` to load all named exports
+7. react-jsx-parser resolves component tags from the loaded registry
+
+### Import spec forms
+
+| Pattern | Resolves to | Example |
+|---|---|---|
+| `pkg` or `npm:pkg` | npm package (installed + bundled) | `export { BarChart } from "recharts"` |
+| `git:user/repo` | GitHub repo source | `export { Comp } from "git:user/repo/src/Comp.tsx"` |
+| `https://...` | Raw URL (used as-is) | `export { X } from "https://cdn.example.com/x.js"` |
+| local path | Filesystem path | `export { X } from "./local/Component.tsx"` |
+
+### Inline functions
+
+Define components directly in the imports block. Dependencies used inside the function body are automatically detected and installed:
+
+```markdown
+~~~js imports
+import ReactMarkdown from "react-markdown"
+
+export function Slide({ children }) {
+  return <div className="slide"><ReactMarkdown>{children}</ReactMarkdown></div>;
+}
+~~~
+```
+
+The `import` statements at the top serve dual purpose: they bring packages into scope inside inline functions AND tell the bundler what to install. The packages are added to `package.json` during bundling.
+
+### Component Contract
+
+Props injected automatically:
+- Nothing special required — just standard React props. The engine passes any `data` bindings from the component node as JSX variables.
+
+For frame-accurate animation, use standard Remotion hooks inside inline functions:
+
+```markdown
+~~~js imports
+import { useCurrentFrame } from "remotion"
+
+export function FadeIn({ children }) {
+  const frame = useCurrentFrame();
+  const opacity = Math.min(1, frame / 15);
+  return <div style={{ opacity }}>{children}</div>;
+}
+~~~
+
+Note: `remotion` and `react` are automatically available in the bundle's external scope — they do NOT need to be imported. However, adding `import { useCurrentFrame } from "remotion"` in the imports block is harmless and makes the dependency explicit.
+
+## 3. Custom Components
+
+Create a React component file in `src/components/`, register in `builtinComponents`.
+
+### Component Contract
+
+Props the engine injects automatically:
+- `action: { start: number; end: number }` — action timing from the stream node
+
+Use `useCurrentFrame()` for frame-accurate animation, `useVideoConfig()` for canvas size.
+
+### Example
+
+```tsx
+// src/components/my/Badge.tsx
+import React from "react";
+import { useCurrentFrame, interpolate } from "remotion";
+
+export const Badge: React.FC<{text: string; color?: string; action: any}> =
+  ({ text, color, action }) => {
+    const frame = useCurrentFrame();
+    const local = frame - action.start * 30;
+    const opacity = interpolate(local, [0, 15], [0, 1]);
+    return (
+      <div style={{ opacity, padding: 12, background: color || theme.colors.primary,
+                    color: "white", borderRadius: 8, fontSize: 48 }}>
+        {text}
+      </div>
+    );
+  };
+```
+
+### Register
+
+```tsx
+// src/components/index.ts
+import { Badge } from "./my/Badge";
+export const builtinComponents = { ..., Badge };
+```
+
+### Reference in stream tree
+
+```json
+{
+  "type": "component",
+  "jsx": "<Badge text='NEW' color='#ff6b35' />",
+  "actions": [{ "start": 1, "end": 4 }]
+}
+```
+
+
+# common used components
+- `react-markdown` — render markdown content, use plugins to extend functionality
+  - `remark-gfm` — support GitHub Flavored Markdown (tables, strikethrough, task lists)
+  - `remark-toc` — generate table of contents
+  - `remark-math` — support math formulas
+  - `rehype-katex` — render math formulas with KaTeX
+  - `remark-mermaidjs` — render mermaid diagrams in 
+- `react-markdown-mermaid` — render mermaid diagrams in standalone mode (no need to install mermaid separately)
+
+- `@remotion/shapes` — render shapes like arrows, circles, rectangles, etc
+- `@remotion/starburst` — render starburst animations

package/docs/edit-mode.md

@@ -0,0 +1,220 @@
+# Watch Mode
+
+> `preview <file.md> --edit` — live editing loop with AI-assisted JSON editing.
+
+Edit mode gives you a browser player that **automatically reloads** when the source file changes, plus an **edit input** that calls an AI coding agent (pi) to make changes on your behalf.
+
+Generated artifacts (TTS audio, compiled includes, component bundles) are
+stored under `.markcut/` next to the source file — see [SKILL.md](../SKILL.md) for the full layout.
+
+## Quickstart
+
+```bash
+cd remotion-engine
+npm run preview -- sample-visual.json --edit
+# → Browser opens, terminal blocks until you close it
+```
+
+## Architecture
+
+```
+┌─────────────┐     HTTP      ┌──────────────────┐    spawn     ┌──────────┐
+│   Browser   │◄──────────────►│  player/server   │◄────────────►│  pi CLI  │
+│  (Chrome)   │     SSE        │   .mjs (Node)    │  one-shot    │ (agent)  │
+└──────┬──────┘                └────────┬─────────┘              └──────────┘
+       │                                │
+       │  POST /api/edit                │  fs.watchFile
+       │  {text: "make text bigger"}    │  (poll every 500ms)
+       │                                │
+       │  GET /api/video-data           ▼
+       │  ← stream tree JSON    ┌──────────────┐
+       │                        │ stream tree  │
+       │  EventSource           │ .json (file) │
+       │  /api/events           └──────────────┘
+       │  ← {type:"reload"}           ▲
+       │                              │
+       │  window.dispatchEvent        │ pi edits
+       │  ("refresh-player")          │ this file
+       │                              │
+```
+
+## File: `src/player/server.mjs`
+
+### Server startup (`cli.mjs` → `server.mjs`)
+
+```
+node src/render/cli.mjs preview file.json --edit --port 3031
+```
+
+1. `cli.mjs` spawns `server.mjs` with `--edit` flag
+2. `server.mjs` parses `file.json` to build a recursive tree description
+3. Starts `fs.watchFile(file.json, {interval:500})` — polls every 500ms for changes
+4. Starts HTTP server on specified port
+5. Auto-opens browser via `open http://localhost:PORT`
+
+### Page served
+
+The HTML page loads two scripts:
+
+| Script | Source | Role |
+|--------|--------|------|
+| `player.js` | Bundled esbuild output from `src/player/browser.tsx` | Renders the stream tree using `@remotion/player` with `MarkCut` |
+| Inline script | Generated by `getHtml()` in `server.mjs` | Watch-mode UI controls (close button, edit input, SSE) |
+
+### Edit flow
+
+```
+Browser                          Server                          pi CLI
+  │                                │                              │
+  │── POST /api/edit ────────────► │                              │
+  │   {text: "make text bigger"}   │                              │
+  │                                │── spawn("pi", ["-p", prompt])│
+  │                                │   prompt includes:           │
+  │                                │   • full tree description   │
+  │                                │     (recursive, any depth)  │
+  │                                │   • edit history             │
+  │                                │   • knowledge (all stream   │
+  │                                │     types, components,      │
+  │                                │     schema, styling guide)  │
+  │                                │   • edit request             │
+  │                                │                              │
+  │                                │   pi can edit ANY field on  │
+  │                                │   ANY node in the tree      │
+  │                                │                              ├── edits file.json
+  │                                │                              └── exits
+  │                                │◄── exit code 0              │
+  │◄── {done:true, output:"..."} ──┤                              │
+  │                                │                              │
+  │ suppressReload = true          │                              │
+  │ header shows "Changed text..." │                              │
+  │                                │                              │
+  │  (fs.watchFile fires SSE       │                              │
+  │   but suppressReload=true      │                              │
+  │   so it's ignored)             │                              │
+  │                                │                              │
+  │ after 4s:                      │                              │
+  │ dispatchEvent("refresh-player")│                              │
+  │ → player re-fetches /api/data  │                              │
+  │ → re-renders with new JSON     │                              │
+```
+
+### Key design decisions
+
+**Why in-place refresh instead of `location.reload()`?**
+- Timeline playback continues uninterrupted
+- Status text persists until next edit
+- No page flash
+
+**Why `suppressReload`?**
+- SSE fires ~500ms after pi edits the file
+- API response arrives at the same time
+- `suppressReload` flag ignores the SSE event during edit
+- The client reloads itself after showing the summary
+
+## File: `src/player/browser.tsx`
+
+The bundled player app:
+
+1. Fetches `/api/video-data` on mount
+2. Renders the stream tree using `@remotion/player` with `MarkCut`
+3. Listens for `"refresh-player"` custom DOM event
+4. On refresh event: re-fetches data, re-renders in-place
+
+## File: `src/render/cli.mjs`
+
+The CLI routes `--edit` flag to the player server:
+
+```javascript
+if (args.edit) {
+  const serverArgs = ["src/player/server.mjs", file, "--edit", `--port=${port}`];
+  const child = spawn("node", serverArgs, { cwd: ROOT, stdio: "inherit" });
+  // ...blocks until server exits...
+}
+```
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/video-data` | GET | Returns the raw stream tree JSON file |
+| `/api/scenes` | GET | Returns parsed scenes array with timing |
+| `/api/video-info` | GET | Returns scenes + totalDuration + mode flags |
+| `/api/edit` | POST | Accepts `{text}` → spawns pi to edit the JSON → file watcher triggers → SSE reload |
+| `/api/shutdown` | POST | Kills the server process, unblocking the agent |
+| `/api/events` | GET | SSE stream — sends `{type:"reload"}` when file changes |
+| `/api/labels` | GET/POST | (label mode only) Save/load scene labels |
+
+## The pi prompt
+
+Each edit call sends a structured prompt with a **recursive tree description** instead of a flat scene list:
+
+```
+You are editing sample-visual.json, a Remotion stream tree JSON.
+
+The stream tree (indentation shows nesting; timing in seconds):
+  root "root" (1080×1920, 30fps, series, transition:fade)
+    folder "scene-1" (parallel)
+      component GradientBackground {type:"radial",animated:true,noise:true} [05s, bg]
+      component AnimatedHeadline {"text":"Hello Remotion",...} [0.54.5s]
+    folder "scene-2" (parallel)
+      component GradientBackground {type:"linear",animated:true} [05s, bg]
+      subtitle "This text is styled with CSS" fontSize:52 [0.54.5s]
+    ...
+
+Previous edits:
+1. make scene 1 headline text orange
+
+Edit request: make scene 1 headline text bigger
+
+--- Knowledge ---
+You can edit ANY field on ANY node in the JSON...
+Stream types: root, folder, video, audio, image, subtitle, component, effect, rhythm, map
+...
+IMPORTANT: Read the full existing JSON file before editing. Only edit the JSON file. You can change, add, or remove any field on any node. Output ONLY a one-line summary of what specific change you made.
+```
+
+The tree description walks the JSON recursively, showing:
+- **Nesting depth** via indentation
+- **Type** and **id/name** of each node
+- **Key fields** (dimensions, fps, theme, series/parallel, transitions)
+- **Timing** from actions (`[startend]`)
+- **isBackground** flag (`[bg]`)
+- **Component props** and node-specific fields (fontSize, fit, style, volume, etc.)
+
+This means pi can edit **any aspect of the JSON** ÔÇö not just scene props. Nested folders, effects, rhythms, maps, actions, styles, themes, global stylesheets, audio config ÔÇö everything is visible and editable.
+
+## Session context
+
+The server maintains `editHistory[]` — every edit is appended. On each call, previous edits are included so pi knows what was done before:
+
+```
+Previous edits:
+1. make scene 1 headline text orange
+2. change background from radial to linear gradient
+3. add 3D angle to device mockup in scene 4
+```
+
+This gives pi enough context to make coherent changes across multiple edit rounds.
+
+## Close / tab-close
+
+| Action | What happens |
+|--------|-------------|
+| Click **✕** | `sendBeacon("/api/shutdown")` → server exits → agent unblocks |
+| Close **tab** | SSE connection drops → 3s grace → server exits |
+| Close **browser** | Same as tab close |
+
+## Label mode (`--label`)
+
+`preview <file.json> --label` starts the same player but with a **label input** instead of the edit input. Labels are saved to `labels.json` next to the source file. No AI editing, no file watching.
+
+## Build
+
+The browser bundle is built with esbuild:
+
+```bash
+bash scripts/build-player.sh
+# → src/player/bundle/player.js
+```
+
+The bundle includes React, @remotion/player, MarkCut, all 13 built-in components, themes, templates, and the duration utility. It is served as a static file by the player server.