@amg555/frameforge-mcp 2.3.2 → 2.3.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.mcp.json +2 -4
- package/CHANGELOG.md +24 -1
- package/CLAUDE.md +104 -26
- package/README.md +17 -12
- package/SECURITY.md +1 -3
- package/package.json +7 -3
- package/plugins/video-pipeline/.claude-plugin/plugin.json +19 -0
- package/plugins/video-pipeline/skills/html-renderer/SKILL.md +292 -0
- package/plugins/video-pipeline/skills/video-pipeline/SKILL.md +308 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/ANIMATION_GUIDE.md +257 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/STORYBOARD.md +77 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/index.html +80 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/pipeline/briefings/diagram-author.md +43 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/pipeline/briefings/orchestrator.md +56 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/pipeline/briefings/scene-author.md +42 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/reel/engine.css +140 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/reel/engine.js +870 -0
- package/plugins/video-pipeline/skills/video-pipeline/assets/project/reel/virtual-clock.js +138 -0
- package/plugins/video-pipeline/skills/video-pipeline/references/mcp-tools.md +167 -0
- package/plugins/video-pipeline/skills/video-pipeline/references/providers.md +174 -0
- package/scripts/db-migrate.js +112 -0
- package/scripts/deep-validate.js +326 -0
- package/scripts/generate-initial-assets.js +342 -0
- package/scripts/html-renderer/new-project.mjs +408 -0
- package/scripts/html-renderer/transcribe.py +294 -0
- package/src/asset-library/README.md +2 -2
- package/src/asset-library/asset-library.js +1 -1
- package/src/cli/doctor.js +12 -5
- package/src/cli/make.js +52 -71
- package/src/cli/setup-keys.js +15 -17
- package/src/fact-checker/review-gate.js +5 -1
- package/src/harness/bin/harness.mjs +324 -0
- package/src/harness/lib/browser.mjs +137 -0
- package/src/harness/lib/capture.mjs +161 -0
- package/src/harness/lib/encode.mjs +89 -0
- package/src/harness/lib/jobs.mjs +135 -0
- package/src/harness/lib/pixels.mjs +137 -0
- package/src/harness/lib/sheets.mjs +164 -0
- package/src/harness/lib/verify-purity.mjs +119 -0
- package/src/harness/test/fixtures/broken-scene.html +27 -0
- package/src/harness/test/fixtures/reference-scene.html +23 -0
- package/src/harness/test/fixtures/sample-script.json +18 -0
- package/src/mcp-server/server.js +285 -144
- package/src/pipeline/telemetry.js +1 -0
- package/src/utils/log-redaction.js +5 -3
- package/src/video-renderer/renderer.js +12 -4
- package/src/worker/server-render-free.js +2 -3
- package/src/workflows/assets.js +20 -43
- package/src/workflows/ingest.js +5 -5
|
@@ -0,0 +1,292 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: html-renderer
|
|
3
|
+
description: |
|
|
4
|
+
HTML-as-compositor renderer for FrameForge. Every video frame is a pure
|
|
5
|
+
function of time rendered by a headless browser (Playwright + Chromium)
|
|
6
|
+
and encoded by FFmpeg. Produces exact, pixel-perfect animation — React
|
|
7
|
+
component quality without a React runtime. Use this as an alternative to
|
|
8
|
+
FFmpeg-only zoompan when richer motion graphics are required: kinetic text,
|
|
9
|
+
morphing layouts, spring-physics, counters, progress rings, captions
|
|
10
|
+
synced to audio, and any CSS/SVG animation. Pairs with the FrameForge
|
|
11
|
+
pipeline: script JSON → HTML compositor → frame PNGs → FFmpeg encode.
|
|
12
|
+
Compatible with IBM Bob, Kiro, Cursor, Windsurf, Claude Code, and any
|
|
13
|
+
agentic IDE connected via MCP. Activate when the user asks for richer
|
|
14
|
+
animation, Remotion-style rendering, HTML-based video, or "make it look
|
|
15
|
+
like the reference repo".
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# HTML Renderer — FrameForge Extension
|
|
19
|
+
|
|
20
|
+
Renders video frames as HTML pages (one page = one frame clock) using
|
|
21
|
+
Playwright for screenshot capture and FFmpeg for final encoding.
|
|
22
|
+
|
|
23
|
+
Built on deterministic `seek(t)` — every style is a pure function of time `t`.
|
|
24
|
+
No CSS transitions, no `setTimeout`, no frame state that depends on the
|
|
25
|
+
previous frame (Playwright renders frames out of order for parallelism).
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## When to use this renderer
|
|
30
|
+
|
|
31
|
+
| Use FFmpeg zoompan (default) | Use HTML renderer (this skill) |
|
|
32
|
+
|------------------------------------|--------------------------------------|
|
|
33
|
+
| Batch overnight — speed matters | Animation quality is the priority |
|
|
34
|
+
| Simple Ken-Burns + captions | Kinetic text, counters, data charts |
|
|
35
|
+
| CI/server with no browser | Morphing layouts, spring physics |
|
|
36
|
+
| < 5 min render time required | Creator wants Remotion-style output |
|
|
37
|
+
| Raspberry Pi / low-RAM machine | macOS/Linux with 4GB+ free RAM |
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Requirements
|
|
42
|
+
|
|
43
|
+
- Node.js 18+, FFmpeg, `npx playwright install chromium`
|
|
44
|
+
- macOS or Linux recommended (Windows: WSL2)
|
|
45
|
+
- 4 GB+ free RAM (Playwright renders up to 6 subframes per output frame in
|
|
46
|
+
parallel for motion blur)
|
|
47
|
+
- `uv` (Python) for audio processing scripts: `pip install uv` or
|
|
48
|
+
`curl -LsSf https://astral.sh/uv/install.sh | sh`
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## Setup
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
# Install Playwright chromium (one-time)
|
|
56
|
+
npx playwright install chromium
|
|
57
|
+
|
|
58
|
+
# Scaffold a new HTML-renderer project from a FrameForge script JSON
|
|
59
|
+
node <frameforge-root>/scripts/html-renderer/new-project.mjs \
|
|
60
|
+
--script output/<productionId>-script.json \
|
|
61
|
+
--out output/<productionId>-html/
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
The scaffold copies `index.html` (compositor), starter `edit.js`, the
|
|
65
|
+
`reel/engine.js` runtime, and all render scripts into the project folder.
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## Core concept: `seek(t)`
|
|
70
|
+
|
|
71
|
+
```js
|
|
72
|
+
// edit.js — every element style is computed from t (seconds)
|
|
73
|
+
import { reel, faceAt, layer, text, at, captions } from './reel/engine.js';
|
|
74
|
+
|
|
75
|
+
reel({ width: 1920, height: 1080, fps: 30, duration: 65 });
|
|
76
|
+
|
|
77
|
+
// Title card: fades in at t=0, stays for 3s
|
|
78
|
+
layer('hook').on(t => ({
|
|
79
|
+
opacity: t < 0.3 ? t / 0.3 : t > 2.7 ? (3 - t) / 0.3 : 1,
|
|
80
|
+
}));
|
|
81
|
+
|
|
82
|
+
at(0, text('Why Ancient Humans Started Cooking', {
|
|
83
|
+
font: '72px/1.2 "Archivo"', color: '#fff',
|
|
84
|
+
x: 'center', y: 200,
|
|
85
|
+
}));
|
|
86
|
+
|
|
87
|
+
// Caption synced to FrameForge TTS audio
|
|
88
|
+
captions('audio/outline.txt'); // generated by transcribe.py
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
**Iron rule:** Every style must be a pure function of `t`. If a layer is
|
|
92
|
+
active during frames 5–15 s and you `return` early for t > 15, it must
|
|
93
|
+
explicitly reset every property it drove — otherwise the last frame's value
|
|
94
|
+
sticks on all later frames.
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## Workflow (integrates with FrameForge pipeline)
|
|
99
|
+
|
|
100
|
+
### Step 1: Run FrameForge dry-run to get script JSON
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
npm run make -- --topic "Your Topic" --dry-run --out-dir output/my-topic
|
|
104
|
+
# Produces: output/my-topic/<productionId>-script.json
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
### Step 2: Scaffold the HTML compositor project
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
node scripts/html-renderer/new-project.mjs \
|
|
111
|
+
--script output/my-topic/<productionId>-script.json \
|
|
112
|
+
--out output/my-topic/html-render/
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The scaffolder reads each scene's `narration`, `caption`,
|
|
116
|
+
`visual_description`, `camera_hint`, and `music_mood` from the script JSON
|
|
117
|
+
and generates a starter `edit.js` with one `layer()` per scene, timed to
|
|
118
|
+
the expected narration duration (estimated at 2.5 words/second).
|
|
119
|
+
|
|
120
|
+
### Step 3: Gather assets (optional)
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
# Stock footage
|
|
124
|
+
uv run scripts/stock.py video "ancient cooking fire" --portrait
|
|
125
|
+
|
|
126
|
+
# Website screenshot as b-roll
|
|
127
|
+
node scripts/capture.mjs site https://example.com broll-site
|
|
128
|
+
|
|
129
|
+
# X post card
|
|
130
|
+
node scripts/capture.mjs tweet https://x.com/... post-card
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
### Step 4: Check visually before full render
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
# Contact sheet — one frame every 0.5s
|
|
137
|
+
node scripts/render.mjs sheet
|
|
138
|
+
|
|
139
|
+
# Specific timestamps
|
|
140
|
+
node scripts/render.mjs stills 0 5.5 12 28 45 60
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
### Step 5: Mix audio
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
uv run scripts/mix_audio.py
|
|
147
|
+
# Reads FrameForge TTS outputs, applies -14 LUFS, writes out/audio_mix.m4a
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### Step 6: Full render + verify
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
node scripts/render.mjs full
|
|
154
|
+
uv run scripts/verify.py
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Output: `out/<name>.mp4` (1920x1080 or 1080x1920 for Shorts), verified
|
|
158
|
+
against the same FrameForge quality gates (resolution, codec, LUFS, A/V sync).
|
|
159
|
+
|
|
160
|
+
### Step 7: Re-inject into FrameForge pipeline
|
|
161
|
+
|
|
162
|
+
The rendered MP4 can be injected back into FrameForge via `video_path`:
|
|
163
|
+
|
|
164
|
+
```json
|
|
165
|
+
{
|
|
166
|
+
"scene_id": 1,
|
|
167
|
+
"video_path": "output/my-topic/html-render/out/video.mp4"
|
|
168
|
+
}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Or pass it directly to `publish_video` via MCP:
|
|
172
|
+
|
|
173
|
+
```
|
|
174
|
+
publish_video(productionId, masterPath="output/my-topic/html-render/out/video.mp4", channels=["file","webhook"])
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
179
|
+
## Engine API quick reference
|
|
180
|
+
|
|
181
|
+
| Function | What it does |
|
|
182
|
+
|---|---|
|
|
183
|
+
| `reel({ width, height, fps, duration })` | Configure the compositor |
|
|
184
|
+
| `layer('name').on(t => styles)` | Declare a layer; styles recomputed every frame |
|
|
185
|
+
| `at(t, element)` | Place element at time t |
|
|
186
|
+
| `stagger(t, dt, elements[])` | Place elements with dt spacing |
|
|
187
|
+
| `text(str, opts)` | Animated text with font/color/position |
|
|
188
|
+
| `chip(str, opts)` | Pill-shaped label (tags, captions) |
|
|
189
|
+
| `wallCard(lines[], opts)` | Frosted-glass card over background |
|
|
190
|
+
| `imageCard(src, opts)` | Image with optional caption overlay |
|
|
191
|
+
| `browserCard(url, opts)` | Screenshot-as-card |
|
|
192
|
+
| `terminal(lines[], opts)` | Animated terminal output |
|
|
193
|
+
| `stamp(str, opts)` | Bold diagonal stamp (VERIFIED, SOURCE, etc.) |
|
|
194
|
+
| `codeCard(code, lang, opts)` | Syntax-highlighted code block |
|
|
195
|
+
| `captions(outlinePath)` | Auto-sync captions from whisper outline |
|
|
196
|
+
| `cuePhrase(phrase, offset)` | Word-level trigger with fail-fast transcript check |
|
|
197
|
+
| `cue(sfxName, t, vol)` | Trigger sound effect at time t |
|
|
198
|
+
| `faceAt(layout, t)` | Position talking-head face (FULL, SPLIT, PIP) |
|
|
199
|
+
| `hash(i)` | Deterministic PRNG float [0, 1) for index i |
|
|
200
|
+
| `boilSeed(key, t)` | Deterministic stepped float [0, 1) in 1/12s window |
|
|
201
|
+
| `kf(t, keys, ease)` | Keyframe interpolation with easing curves |
|
|
202
|
+
| `onTwos(t)` | Quantize time to 12 fps (traditional animation on twos) |
|
|
203
|
+
| `seg(t, a, b)` | Clamp t into [0, 1] interval progress |
|
|
204
|
+
| `pulse(t, k)` | Exponential impulse peak at t=0, decaying with rate k |
|
|
205
|
+
| `shots(timeline)` | Time-ordered shot registry fn(t, lt, dur) for parallel render |
|
|
206
|
+
| `on(t => ...)` | Raw per-frame compute (for custom animations) |
|
|
207
|
+
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
## Storyboard Gates, Purity Verification & Review Modes
|
|
211
|
+
|
|
212
|
+
```bash
|
|
213
|
+
# 1. Diagnostic & Setup checks
|
|
214
|
+
node scripts/harness.mjs doctor # Verify Chrome, GPU flags (D3D11/Metal), WebGL context
|
|
215
|
+
node scripts/look.mjs # Pre-render linter (fonts, clipping, NaN transforms)
|
|
216
|
+
|
|
217
|
+
# 2. Purity verification gate (3-point repeat, wander, cold-jump test)
|
|
218
|
+
node scripts/harness.mjs verify # Must pass a===b and a===c across 9 sampled timestamps
|
|
219
|
+
|
|
220
|
+
# 3. Gate stills & review commands
|
|
221
|
+
node scripts/harness.mjs storyboard # 4 keyframes (0%, 33%, 66%, 100%) or validate STORYBOARD.md
|
|
222
|
+
node scripts/harness.mjs sheet # Contact sheet (0.5s interval)
|
|
223
|
+
node scripts/harness.mjs strip 1.0 1.5 # Horizontal frame strip (motion check)
|
|
224
|
+
node scripts/harness.mjs crop 9:16 # Focal-point mobile crop preview
|
|
225
|
+
node scripts/harness.mjs pixels # Blind-spot frame analysis (blank, blown, corner-confined)
|
|
226
|
+
|
|
227
|
+
# 4. Master render & encode
|
|
228
|
+
node scripts/harness.mjs frames # Resumable atomic frame render (.tmp + renameSync)
|
|
229
|
+
node scripts/harness.mjs encode # BT.709 color-tagged master encode + audio mux
|
|
230
|
+
node scripts/harness.mjs all # Complete gate-checked end-to-end production
|
|
231
|
+
node scripts/render.mjs mux # Re-mux audio only (no re-render)
|
|
232
|
+
node scripts/render.mjs kit # Editor kit (Premiere/Resolve XML)
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Full render uses 6-page parallel Playwright pool (motion blur subframes).
|
|
236
|
+
Long videos render in 60s resumable chunks — re-run to continue after crash.
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
## Gotchas
|
|
241
|
+
|
|
242
|
+
- **Pure functions of t.** No CSS transitions, `setTimeout`, or state that
|
|
243
|
+
depends on previous frames. Playwright seeks out of order.
|
|
244
|
+
- **Every style must reset.** If a layer returns early after t > 15s, it
|
|
245
|
+
must explicitly set opacity/display/visibility to 0/none/hidden — or the
|
|
246
|
+
last rendered value bleeds onto all later frames.
|
|
247
|
+
- **HDR iPhone footage** needs tonemapping before compositing
|
|
248
|
+
(`uv run scripts/assemble.py` does this automatically).
|
|
249
|
+
- **Motion blur needs 6 subframes.** Fewer subframes show fast morphs as
|
|
250
|
+
ghost copies. Default is 6 (180° shutter), a 60s video takes ~4 min.
|
|
251
|
+
- **Measure text after fonts load.** Components that need widths (strike-
|
|
252
|
+
through, underline) measure lazily on first `seek()`.
|
|
253
|
+
- **Captions over the mouth.** Use `REEL.captionY` per layout to move them
|
|
254
|
+
to the seam in split-screen mode.
|
|
255
|
+
|
|
256
|
+
---
|
|
257
|
+
|
|
258
|
+
## IBM Bob integration
|
|
259
|
+
|
|
260
|
+
This skill works inside IBM Bob as a `~/.bob/skills/html-renderer/` skill.
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
# Install
|
|
264
|
+
cp -r plugins/video-pipeline/skills/html-renderer/ ~/.bob/skills/html-renderer/
|
|
265
|
+
|
|
266
|
+
# Bob will pick it up automatically — ask:
|
|
267
|
+
# "use the html-renderer skill to animate my FrameForge script"
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
The skill pairs with the existing `video-pipeline` skill: use
|
|
271
|
+
`video-pipeline` for research/script/TTS/publish and `html-renderer` for
|
|
272
|
+
the animation/render stage.
|
|
273
|
+
|
|
274
|
+
---
|
|
275
|
+
|
|
276
|
+
## Files this skill adds
|
|
277
|
+
|
|
278
|
+
```
|
|
279
|
+
plugins/video-pipeline/skills/html-renderer/
|
|
280
|
+
SKILL.md ← this file
|
|
281
|
+
references/engine-api.md ← full engine function reference
|
|
282
|
+
references/design.md ← layouts, safe zones, type scale, pacing
|
|
283
|
+
assets/project/ ← template copied into each render project
|
|
284
|
+
index.html ← compositor page (loads engine.js)
|
|
285
|
+
edit.js ← starter edit (one layer per FrameForge scene)
|
|
286
|
+
reel/engine.js ← springs, layouts, components, captions, seek API
|
|
287
|
+
reel/engine.css ← default fonts + base reset
|
|
288
|
+
scripts/ ← render.mjs, mix_audio.py, verify.py, stock.py,
|
|
289
|
+
capture.mjs, transcribe.py, new-project.mjs
|
|
290
|
+
scripts/html-renderer/
|
|
291
|
+
new-project.mjs ← scaffolds a render project from script JSON
|
|
292
|
+
```
|
|
@@ -0,0 +1,308 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: video-pipeline
|
|
3
|
+
description: >
|
|
4
|
+
Autonomous AI video production pipeline. Given a topic, this skill
|
|
5
|
+
researches, fact-checks, writes the script, synthesises voice, renders
|
|
6
|
+
with FFmpeg, runs quality gates, and saves the output locally or POSTs
|
|
7
|
+
to any webhook — all without human intervention.
|
|
8
|
+
Works in IBM Bob, Codex, opencode, Cursor, Windsurf, Claude Code,
|
|
9
|
+
or any IDE that loads skills from ~/.claude/skills/ or .claude/skills/.
|
|
10
|
+
Trigger phrases: "make a video about", "produce a reel on", "overnight
|
|
11
|
+
batch", "queue videos", "generate video", "run the pipeline", "publish
|
|
12
|
+
video". Also activates when a user pastes a topic list and says
|
|
13
|
+
"make these" or "do the batch".
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
# Video Pipeline Skill
|
|
17
|
+
|
|
18
|
+
End-to-end autonomous video production. The AI proposes every creative
|
|
19
|
+
decision; deterministic systems (FFmpeg, TTS, image gen, MCP) execute them;
|
|
20
|
+
a quality gate blocks bad output before delivery.
|
|
21
|
+
|
|
22
|
+
## Architecture
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
IDE Agent (IBM Bob / Codex / opencode / Cursor / Windsurf / Claude Code)
|
|
26
|
+
│
|
|
27
|
+
▼ Cloudflare Remote MCP (recommended) ─────────────────────────────
|
|
28
|
+
│ https://video-mcp.YOUR_SUBDOMAIN.workers.dev/mcp
|
|
29
|
+
│ 17 tools: create_video_from_script · render_from_scenes
|
|
30
|
+
│ ingest_media · edit_scenes · render_tts · multi_voice_tts
|
|
31
|
+
│ generate_scene_image · assemble_video · publish_video
|
|
32
|
+
│ research_topic · write_script · external_platform_call
|
|
33
|
+
│ queue_full_pipeline · get_job_status
|
|
34
|
+
│ scaffold_animation_project · validate_storyboard · render_animation
|
|
35
|
+
│
|
|
36
|
+
▼ OR — Local stdio / HTTP MCP ────────────────────────────────────
|
|
37
|
+
│ node src/mcp-server/stdio.js (stdio)
|
|
38
|
+
│ node src/mcp-server/server.js (HTTP :3210)
|
|
39
|
+
│
|
|
40
|
+
▼ Execution layer
|
|
41
|
+
│ LLM provider: PROVIDER_LLM=groq|openrouter|anthropic|gemini
|
|
42
|
+
│ |mistral|together|cerebras|openai
|
|
43
|
+
│ |ollama|webhook
|
|
44
|
+
│ TTS provider: PROVIDER_TTS=edge-tts|voxforge|elevenlabs|azure|openai|webhook
|
|
45
|
+
│ Visuals: Code-driven animations (default) + user media (video_path / image_path)
|
|
46
|
+
│
|
|
47
|
+
▼ Optional: queue_full_pipeline
|
|
48
|
+
queue_full_pipeline MCP tool → POST job to worker API (WORKER_API_URL)
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
All secrets come from environment variables — **never** hardcoded.
|
|
52
|
+
See `docs/env.template` for the full reference.
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## Setup
|
|
57
|
+
|
|
58
|
+
### Option A — Cloudflare Remote MCP (Recommended)
|
|
59
|
+
|
|
60
|
+
Deploy once; works in every IDE with no local server process:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
cd cloudflare-mcp
|
|
64
|
+
npm install
|
|
65
|
+
wrangler secret put WORKER_API_URL # https://your-render-app.onrender.com
|
|
66
|
+
wrangler secret put MCP_AUTH_TOKEN # random 32-byte hex
|
|
67
|
+
wrangler secret put INTERNAL_API_TOKEN # shared secret with render node
|
|
68
|
+
wrangler deploy
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Add to any IDE's MCP config:
|
|
72
|
+
|
|
73
|
+
```json
|
|
74
|
+
{
|
|
75
|
+
"mcpServers": {
|
|
76
|
+
"frameforge": {
|
|
77
|
+
"url": "https://video-mcp.YOUR_SUBDOMAIN.workers.dev/mcp",
|
|
78
|
+
"headers": { "Authorization": "Bearer YOUR_MCP_AUTH_TOKEN" }
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### Option B — Local stdio MCP
|
|
85
|
+
|
|
86
|
+
**IBM Bob** (`~/.bob/mcp.json`):
|
|
87
|
+
```json
|
|
88
|
+
{
|
|
89
|
+
"mcpServers": {
|
|
90
|
+
"frameforge": {
|
|
91
|
+
"command": "node",
|
|
92
|
+
"args": ["<absolute-path>/src/mcp-server/stdio.js"],
|
|
93
|
+
"env": { "PROVIDER_LLM": "groq", "GROQ_API_KEY": "gsk_..." }
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**Codex / opencode / Cursor / Windsurf / Claude Code** (`~/.claude/mcp.json` or `.mcp.json`):
|
|
100
|
+
```json
|
|
101
|
+
{
|
|
102
|
+
"mcpServers": {
|
|
103
|
+
"frameforge": {
|
|
104
|
+
"command": "node",
|
|
105
|
+
"args": ["<absolute-path>/src/mcp-server/stdio.js"],
|
|
106
|
+
"env": { "PROVIDER_LLM": "groq", "GROQ_API_KEY": "${GROQ_API_KEY}" }
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### 2. Set environment variables
|
|
113
|
+
|
|
114
|
+
Copy `docs/env.template` → `.env` and fill in. Minimum:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
# LLM (pick one — Groq is free with no credit card)
|
|
118
|
+
PROVIDER_LLM=groq
|
|
119
|
+
GROQ_API_KEY=gsk_...
|
|
120
|
+
|
|
121
|
+
# TTS (edge-tts is free offline)
|
|
122
|
+
PROVIDER_TTS=edge-tts
|
|
123
|
+
|
|
124
|
+
# Visuals are 100% code-driven (kinetic typography/motion graphics) + user media. No image key needed.
|
|
125
|
+
|
|
126
|
+
# Delivery (optional — default is 'file' which copies to N8N_FILES_DIR/published/)
|
|
127
|
+
# PUBLISH_WEBHOOK_URL=https://hooks.example.com/frameforge
|
|
128
|
+
# REVIEW_NOTIFY_URL=https://hooks.example.com/review-gate
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Run the interactive wizard for guided setup:
|
|
132
|
+
```bash
|
|
133
|
+
npm run setup
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## Workflow (step-by-step)
|
|
139
|
+
|
|
140
|
+
### Path 1 — Agent-First (FFmpeg + AI images)
|
|
141
|
+
|
|
142
|
+
**You are already running an LLM — use it.** Write the scenes yourself and hand
|
|
143
|
+
them to FrameForge for rendering. No `GROQ_API_KEY` or any other LLM key needed.
|
|
144
|
+
FrameForge only runs FFmpeg, TTS, and image generation:
|
|
145
|
+
|
|
146
|
+
```
|
|
147
|
+
create_video_from_script({
|
|
148
|
+
scenes: [
|
|
149
|
+
{ narration: "WiFi sends data as radio waves...", caption: "RADIO WAVES",
|
|
150
|
+
visual_description: "router emitting waves, flat illustration" },
|
|
151
|
+
{ narration: "Your device picks up the signal...", caption: "YOUR DEVICE",
|
|
152
|
+
visual_description: "laptop receiving signal, clean diagram" }
|
|
153
|
+
],
|
|
154
|
+
title: "How WiFi Works"
|
|
155
|
+
})
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Then deliver it:
|
|
159
|
+
```
|
|
160
|
+
publish_video({ productionId, masterPath, channels: ["file"] })
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
> **Image generation fallback:** If the configured image provider is unavailable, FrameForge automatically falls back to a dark-background placeholder per scene — the render completes. Switch to `PROVIDER_IMAGE=falai` for a more reliable free provider.
|
|
164
|
+
|
|
165
|
+
### Path 2 — Code-Driven Animated Video (Playwright + ffmpeg, no AI images)
|
|
166
|
+
|
|
167
|
+
Use when: the user wants Remotion-style motion graphics, image gen is unavailable, or they ask for "animated video" or "make it look good". Every pixel is a pure function of time `t`.
|
|
168
|
+
|
|
169
|
+
> **Beat sheet gate:** Reply with a beat sheet and 4 keyframe descriptions first. Stop. Wait for approval before writing any scene code.
|
|
170
|
+
|
|
171
|
+
```
|
|
172
|
+
// 1. Synthesise voice first — audio timing is ground truth
|
|
173
|
+
multi_voice_tts({ text: "Narration.", outputPath: ".frameforge-files/voice_0.m4a" })
|
|
174
|
+
|
|
175
|
+
// 2. Scaffold the compositor project (copies engine + all render scripts + briefings)
|
|
176
|
+
scaffold_animation_project({
|
|
177
|
+
outDir: ".frameforge-files/my-anim/",
|
|
178
|
+
scenes: [ { narration: "...", caption: "...", visual_description: "..." } ],
|
|
179
|
+
audioPath: ".frameforge-files/voice_0.m4a"
|
|
180
|
+
})
|
|
181
|
+
// → returns nextSteps[] — read ANIMATION_GUIDE.md and edit STORYBOARD.md next
|
|
182
|
+
|
|
183
|
+
// 3. Validate storyboard (four hard gates — must pass before scene code)
|
|
184
|
+
validate_storyboard({ storyboardPath: ".frameforge-files/my-anim/STORYBOARD.md" })
|
|
185
|
+
|
|
186
|
+
// 4. Write real animation in edit.js (pure functions of t — read ANIMATION_GUIDE.md)
|
|
187
|
+
|
|
188
|
+
// 5. Pre-flight linter & purity verification (3-point repeat, wander, cold-jump)
|
|
189
|
+
render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "look" })
|
|
190
|
+
render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "verify" })
|
|
191
|
+
|
|
192
|
+
// 6. Review stills (at least one sheet per shot, one strip per key motion)
|
|
193
|
+
render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "sheet" })
|
|
194
|
+
render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "strip",
|
|
195
|
+
timestamps: [2.1, 2.6] }) // every frame of one motion
|
|
196
|
+
render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "crop",
|
|
197
|
+
timestamps: ["9:16", 3.0] }) // mobile crop check
|
|
198
|
+
|
|
199
|
+
// 7. Full render (only after stills pass)
|
|
200
|
+
render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "full",
|
|
201
|
+
audioPath: ".frameforge-files/voice_0.m4a" })
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
Requires one-time: `cd <outDir> && npm install playwright && npx playwright install chromium`
|
|
205
|
+
|
|
206
|
+
### Path 3 — Auto-Research (for unattended / batch runs)
|
|
207
|
+
|
|
208
|
+
Use this when FrameForge runs **without** an IDE in the loop — overnight batch,
|
|
209
|
+
`queue_full_pipeline`, or a standalone worker. The pipeline calls its own LLM
|
|
210
|
+
for research and scripting, so `PROVIDER_LLM` + the matching API key are required:
|
|
211
|
+
|
|
212
|
+
```
|
|
213
|
+
1. research_topic({ topic: "...", style: "educational", format: "short-form reel" })
|
|
214
|
+
2. write_script({ productionId, claims, topic, persona: "tech reviewer" })
|
|
215
|
+
3. render_from_scenes({ productionId })
|
|
216
|
+
4. publish_video({ productionId, masterPath, channels: ["file"] })
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
### Voice Synthesis (per call)
|
|
220
|
+
|
|
221
|
+
```
|
|
222
|
+
multi_voice_tts({
|
|
223
|
+
text: "Narration text here",
|
|
224
|
+
outputPath: "/path/in/files-dir/voice_0.m4a",
|
|
225
|
+
platform: "elevenlabs",
|
|
226
|
+
voice: "your_cloned_voice_id",
|
|
227
|
+
apiKey: "sk_..."
|
|
228
|
+
})
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
### Overnight batch
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
I'm going to sleep. Make as many videos as you can before 9 AM.
|
|
235
|
+
Topics (queue them all):
|
|
236
|
+
1. How does GPS actually work?
|
|
237
|
+
2. Why planes don't fly over the Pacific in a straight line
|
|
238
|
+
3. The engineering inside a modern CPU
|
|
239
|
+
|
|
240
|
+
Rules:
|
|
241
|
+
- Strong hook, double-verify every fact.
|
|
242
|
+
- After each video: run the quality gate, then publish to the output directory.
|
|
243
|
+
- Leave me a summary when done.
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
The skill calls MCP tools sequentially using the StateMachine to track progress.
|
|
247
|
+
If a stage fails it retries with exponential backoff; if it fails terminally it
|
|
248
|
+
marks that production FAILED and moves to the next topic.
|
|
249
|
+
|
|
250
|
+
---
|
|
251
|
+
|
|
252
|
+
## Provider reference
|
|
253
|
+
|
|
254
|
+
| Role | Provider | Key env var | Notes |
|
|
255
|
+
|-------|-------------|--------------------------|-------------------------------|
|
|
256
|
+
| LLM | groq | `GROQ_API_KEY` | Default. Fast, free tier. |
|
|
257
|
+
| LLM | openrouter | `OPENROUTER_API_KEY` | Any model via one key |
|
|
258
|
+
| LLM | anthropic | `ANTHROPIC_API_KEY` | Claude Sonnet/Haiku/Opus |
|
|
259
|
+
| LLM | gemini | `GEMINI_API_KEY` | Gemini 2.0 Flash / 2.5 Pro |
|
|
260
|
+
| LLM | cerebras | `CEREBRAS_API_KEY` | 2000+ tok/s Llama |
|
|
261
|
+
| LLM | openai | `OPENAI_API_KEY` | GPT-4o / o1 |
|
|
262
|
+
| LLM | ollama | _(none)_ | Local, OLLAMA_BASE_URL |
|
|
263
|
+
| TTS | edge-tts | _(none)_ | Free, offline, 300+ voices |
|
|
264
|
+
| TTS | elevenlabs | `ELEVENLABS_API_KEY` | Voice cloning |
|
|
265
|
+
| TTS | azure | `AZURE_SPEECH_KEY` | Enterprise |
|
|
266
|
+
| TTS | openai | `OPENAI_API_KEY` | High quality |
|
|
267
|
+
| Visual| code-driven | _(none)_ | Default — kinetic cards & typography |
|
|
268
|
+
| Visual| user-media | _(none)_ | Caller `image_path` / `video_path` |
|
|
269
|
+
|
|
270
|
+
---
|
|
271
|
+
|
|
272
|
+
## Quality gates (hard — will block delivery)
|
|
273
|
+
|
|
274
|
+
| Gate | Threshold |
|
|
275
|
+
|---------------------|----------------------|
|
|
276
|
+
| File size | > 1 MB |
|
|
277
|
+
| Video stream | present |
|
|
278
|
+
| Resolution (master) | 1920×1080 |
|
|
279
|
+
| Resolution (short) | 1080×1920 |
|
|
280
|
+
| Codec | H.264 |
|
|
281
|
+
| Pixel format | yuv420p |
|
|
282
|
+
| Frame rate | 24–30 fps |
|
|
283
|
+
| Integrated loudness | −14 ±1.5 LUFS |
|
|
284
|
+
| True peak | ≤ −0.5 dBTP |
|
|
285
|
+
| Black frames | none > 0.5 s |
|
|
286
|
+
| Frozen frames | none > 0.5 s |
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## Key files
|
|
291
|
+
|
|
292
|
+
```
|
|
293
|
+
src/mcp-server/server.js MCP HTTP server (17 tools, port 3210)
|
|
294
|
+
src/mcp-server/stdio.js MCP stdio server (17 tools)
|
|
295
|
+
cloudflare-mcp/src/index.js Cloudflare Workers Remote MCP proxy
|
|
296
|
+
src/pipeline/providers.js LLM / TTS / Image abstraction (env-var-switchable providers)
|
|
297
|
+
src/pipeline/state-machine.js File + PostgreSQL production state
|
|
298
|
+
src/pipeline/retry-policy.js Exponential backoff, stage timeouts
|
|
299
|
+
src/pipeline/telemetry.js NDJSON cost tracking
|
|
300
|
+
src/quality-gate/gate.js 9 hard gates
|
|
301
|
+
src/audio-mixer/mixer.js Sidechain ducking, LUFS normalisation
|
|
302
|
+
src/shorts-extractor/extractor.js Silence-based 9:16 crop for Shorts
|
|
303
|
+
src/fact-checker/checker.js Claim risk classification + evidence gates
|
|
304
|
+
src/video-renderer/renderer.js Scene render + master assembly
|
|
305
|
+
src/worker/server-render-free.js Render-optimised worker API (Render free)
|
|
306
|
+
src/worker/server.js Full worker API
|
|
307
|
+
docs/env.template Environment variable reference
|
|
308
|
+
```
|