@amg555/frameforge-mcp 2.7.1 → 2.7.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,384 +1,384 @@
1
- ---
2
- name: video-pipeline
3
- description: >
4
- Autonomous AI video production pipeline. Given a topic, this skill
5
- researches, fact-checks, writes the script, synthesises voice, renders
6
- with FFmpeg, runs quality gates, and saves the output locally or POSTs
7
- to any webhook — all without human intervention.
8
- Works in IBM Bob, Codex, opencode, Cursor, Windsurf, Claude Code,
9
- or any IDE that loads skills from ~/.claude/skills/ or .claude/skills/.
10
- Trigger phrases: "make a video about", "produce a reel on", "overnight
11
- batch", "queue videos", "generate video", "run the pipeline", "publish
12
- video". Also activates when a user pastes a topic list and says
13
- "make these" or "do the batch".
14
- ---
15
-
16
- # Video Pipeline Skill
17
-
18
- End-to-end autonomous video production. The AI proposes every creative
19
- decision; deterministic systems (FFmpeg, TTS, image gen, MCP) execute them;
20
- a quality gate blocks bad output before delivery.
21
-
22
- ## Architecture
23
-
24
- ```
25
- IDE Agent (IBM Bob / Codex / opencode / Cursor / Windsurf / Claude Code)
26
- │
27
- ▼ Cloudflare Remote MCP (recommended) ─────────────────────────────
28
- │ https://video-mcp.YOUR_SUBDOMAIN.workers.dev/mcp
29
- │ 28 tools: create_video_from_script · edit_video · render_from_scenes
30
- │ generate_scene_code · preview_scene · verify_scene_quality
31
- │ ingest_media · edit_scenes · render_tts · multi_voice_tts
32
- │ generate_scene_image · assemble_video · publish_video
33
- │ research_topic · write_script · external_platform_call
34
- │ queue_full_pipeline · get_job_status · export_to_editor
35
- │ scaffold_animation_project · validate_storyboard · render_animation
36
- │ plan_video_edit · sync_upstream_skills · check_upstream_updates
37
- │ transcribe_media · ingest_url · mix_audio_pro
38
- │
39
- ▼ OR — Local stdio / HTTP MCP ────────────────────────────────────
40
- │ node src/mcp-server/stdio.js (stdio)
41
- │ node src/mcp-server/server.js (HTTP :3210)
42
- │
43
- ▼ Execution layer
44
- │ LLM provider: PROVIDER_LLM=groq|openrouter|anthropic|gemini
45
- │ |mistral|together|cerebras|openai
46
- │ |ollama|webhook
47
- │ TTS provider: PROVIDER_TTS=edge-tts|voxforge|elevenlabs|azure|openai|webhook
48
- │ Visuals: Code-driven animations (default) + user media (video_path / image_path)
49
- │
50
- ▼ Optional: queue_full_pipeline
51
- queue_full_pipeline MCP tool → POST job to worker API (WORKER_API_URL)
52
- ```
53
-
54
- All secrets come from environment variables — **never** hardcoded.
55
- See `docs/env.template` for the full reference.
56
-
57
- ---
58
-
59
- ## Setup
60
-
61
- ### Option A — Cloudflare Remote MCP (Recommended)
62
-
63
- Deploy once; works in every IDE with no local server process:
64
-
65
- ```bash
66
- cd cloudflare-mcp
67
- npm install
68
- wrangler secret put WORKER_API_URL # https://your-render-app.onrender.com
69
- wrangler secret put MCP_AUTH_TOKEN # random 32-byte hex
70
- wrangler secret put INTERNAL_API_TOKEN # shared secret with render node
71
- wrangler deploy
72
- ```
73
-
74
- Add to any IDE's MCP config:
75
-
76
- ```json
77
- {
78
- "mcpServers": {
79
- "frameforge": {
80
- "url": "https://video-mcp.YOUR_SUBDOMAIN.workers.dev/mcp",
81
- "headers": { "Authorization": "Bearer YOUR_MCP_AUTH_TOKEN" }
82
- }
83
- }
84
- }
85
- ```
86
-
87
- ### Option B — Local stdio MCP
88
-
89
- **IBM Bob** (`~/.bob/mcp.json`):
90
- ```json
91
- {
92
- "mcpServers": {
93
- "frameforge": {
94
- "command": "node",
95
- "args": ["<absolute-path>/src/mcp-server/stdio.js"],
96
- "env": { "PROVIDER_LLM": "groq", "GROQ_API_KEY": "gsk_..." }
97
- }
98
- }
99
- }
100
- ```
101
-
102
- **Codex / opencode / Cursor / Windsurf / Claude Code** (`~/.claude/mcp.json` or `.mcp.json`):
103
- ```json
104
- {
105
- "mcpServers": {
106
- "frameforge": {
107
- "command": "node",
108
- "args": ["<absolute-path>/src/mcp-server/stdio.js"],
109
- "env": { "PROVIDER_LLM": "groq", "GROQ_API_KEY": "${GROQ_API_KEY}" }
110
- }
111
- }
112
- }
113
- ```
114
-
115
- ### 2. Set environment variables
116
-
117
- Copy `docs/env.template` → `.env` and fill in. Minimum:
118
-
119
- ```bash
120
- # LLM (pick one — Groq is free with no credit card)
121
- PROVIDER_LLM=groq
122
- GROQ_API_KEY=gsk_...
123
-
124
- # TTS (edge-tts is free offline)
125
- PROVIDER_TTS=edge-tts
126
-
127
- # Visuals are 100% code-driven (kinetic typography/motion graphics) + user media. No image key needed.
128
-
129
- # Delivery (optional — default is 'file' which copies to N8N_FILES_DIR/published/)
130
- # PUBLISH_WEBHOOK_URL=https://hooks.example.com/frameforge
131
- # REVIEW_NOTIFY_URL=https://hooks.example.com/review-gate
132
- ```
133
-
134
- Run the interactive wizard for guided setup:
135
- ```bash
136
- npm run setup
137
- ```
138
-
139
- ---
140
-
141
- ## Workflow (step-by-step)
142
-
143
- ### Path 1 — Agent-First (Code-Driven Motion Graphics or Direct Custom Code)
144
-
145
- **You are already running an LLM (Claude Code / Opus 5.5 / Cursor / Bob) — use it.**
146
- Write the scenes yourself and hand them to FrameForge for rendering. No external API key needed.
147
- FrameForge renders full-canvas, broadcast-quality motion graphics (24fps pure function of `t`) with zero slideshow/card patterns:
148
-
149
- - **Option A — Natural Description:** Provide `visual_description` (e.g. "Earth rotating with 4 satellite orbits and microwave beams intersecting"). FrameForge automatically synthesizes a full-canvas procedural 3D/2D animation entity (orbital mechanics, 3D DNA helix, candlestick charts, neural mesh, 3D wireframe polyhedra).
150
- - **Option B — Direct Agent Code:** Provide `html` (complete page with `window.DURATION` & `window.seek(t)`) or `code` (canvas drawing instructions for `t`). FrameForge executes your program directly via headless Chromium.
151
- - **Option C — Generate & Inspect (Layer 3 Loop):** Call `generate_scene_code({ visual_description, style })` to synthesize canvas drawing code. Then call `preview_scene({ html, timestamps: [0.1, 1.5, 2.9] })` to render PNG stills to disk, visually verify the scene in the IDE, tweak if needed, and pass into `create_video_from_script`.
152
- - **Option D — Multilingual & Regional Scripts:** Pass `language: "ml"` (Malayalam), `"hi"` (Hindi), etc. Native DirectWrite/HarfBuzz font stacks (`Nirmala UI`, `Noto Sans Malayalam`) render conjuncts without broken glyphs.
153
-
154
- ```js
155
- create_video_from_script({
156
- topic: "GPS Satellites and Orbital Mechanics",
157
- style: "cosmic_space",
158
- language: "en",
159
- scenes: [
160
- {
161
- narration: "GPS uses trilateration from atomic clock satellites orbiting Earth.",
162
- caption: "TRILATERATION",
163
- visual_description: "3D Earth rotating with 4 satellite orbits and microwave beams converging",
164
- // Optional: you can pass your own Canvas code or complete HTML:
165
- // code: "ctx.fillStyle = '#0c1116'; ctx.fillRect(0,0,W,H); ..."
166
- },
167
- {
168
- narration: "Relativistic time dilation requires nanosecond daily corrections.",
169
- caption: "TIME DILATION",
170
- visual_description: "Quantum atomic clock oscillation with compounding nanosecond curve"
171
- }
172
- ],
173
- title: "How GPS Actually Works"
174
- })
175
- ```
176
-
177
- Then deliver it:
178
- ```
179
- publish_video({ productionId, masterPath, channels: ["file"] })
180
- ```
181
-
182
- ### Path 2 — Code-Driven Animated Video (Playwright + ffmpeg, no AI images)
183
-
184
- Use when: the user wants Remotion-style motion graphics, image gen is unavailable, or they ask for "animated video" or "make it look good". Every pixel is a pure function of time `t`.
185
-
186
- > **Model effort (Opus 5.5 / Claude Code):**
187
- > - Beat sheet + first scene authorship → `effort: xhigh`
188
- > - Storyboard review, strip-range patches, encode-only runs → `effort: medium`
189
- > - `tool_choice` must be `auto` (forced tool use not supported in Opus 5.5)
190
-
191
- > **Beat sheet gate:** Reply with a beat sheet and 4 keyframe descriptions first. Stop. Wait for approval before writing any scene code.
192
-
193
- ```
194
- // 1. Synthesise voice first — audio timing is ground truth
195
- multi_voice_tts({ text: "Narration.", outputPath: ".frameforge-files/voice_0.m4a" })
196
-
197
- // 2. Scaffold the compositor project (copies engine + all render scripts + briefings)
198
- scaffold_animation_project({
199
- outDir: ".frameforge-files/my-anim/",
200
- scenes: [ { narration: "...", caption: "...", visual_description: "..." } ],
201
- audioPath: ".frameforge-files/voice_0.m4a"
202
- })
203
- // → returns nextSteps[] — read ANIMATION_GUIDE.md and edit STORYBOARD.md next
204
-
205
- // 3. Validate storyboard (four hard gates — must pass before scene code)
206
- validate_storyboard({ storyboardPath: ".frameforge-files/my-anim/STORYBOARD.md" })
207
-
208
- // 4. Write real animation in edit.js (pure functions of t — read ANIMATION_GUIDE.md)
209
-
210
- // 5. Pre-flight linter & purity verification (3-point repeat, wander, cold-jump)
211
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "look" })
212
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "verify" })
213
-
214
- // 6. Review stills (at least one sheet per shot, one strip per key motion)
215
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "sheet" })
216
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "strip",
217
- timestamps: [2.1, 2.6] }) // every frame of one motion
218
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "crop",
219
- timestamps: ["9:16", 3.0] }) // mobile crop check
220
-
221
- // 7. Full render (only after stills pass)
222
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "full",
223
- audioPath: ".frameforge-files/voice_0.m4a" })
224
-
225
- // 8. Closed-loop QA fix protocol (mandatory — Opus 5.5 / Claude Code pattern)
226
- render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "quality" })
227
- // Returns: { passed, overallScore, worstFrames: [{time, score, issues}] }
228
- // If passed: false → fix loop (max 3 rounds):
229
- // 1. Read worstFrames[]. 2. Patch edit.js for those timestamps only.
230
- // 3. render_animation({ mode: "strip", timestamps: [failingT - 0.5, failingT + 0.5] })
231
- // 4. render_animation({ mode: "quality" })
232
- // 5. Repeat until passed or 3 rounds — then escalate to user.
233
- // Never re-render full film to fix one timestamp. Use strip only.
234
- ```
235
-
236
- Requires one-time: `cd <outDir> && npm install playwright && npx playwright install chromium`
237
-
238
- ### Path 3 — Auto-Research (for unattended / batch runs)
239
-
240
- Use this when FrameForge runs **without** an IDE in the loop — overnight batch,
241
- `queue_full_pipeline`, or a standalone worker. The pipeline calls its own LLM
242
- for research and scripting, so `PROVIDER_LLM` + the matching API key are required:
243
-
244
- ```
245
- 1. research_topic({ topic: "...", style: "educational", format: "short-form reel" })
246
- 2. write_script({ productionId, claims, topic, persona: "tech reviewer" })
247
- 3. render_from_scenes({ productionId })
248
- 4. publish_video({ productionId, masterPath, channels: ["file"] })
249
- ```
250
-
251
- ### Path 4 — Autonomous Local Video Editing (`edit_video`)
252
-
253
- When the user provides video clips (B-roll, screen recordings, camera footage) and audio/narration, FrameForge autonomously cuts, trims, color-grades, reframes, captions, and stitches them with seamless transitions locally:
254
-
255
- ```js
256
- edit_video({
257
- clips: [
258
- {
259
- video_path: "/path/to/clip1.mp4",
260
- start_time: "00:05.5",
261
- end_time: "00:15.0",
262
- speed: 1.0,
263
- color_grade: "cinematic",
264
- caption: "AUTONOMOUS EDITING",
265
- transition: "dissolve",
266
- transition_duration: 0.5
267
- },
268
- {
269
- video_path: "/path/to/clip2.mp4",
270
- start_time: 0,
271
- duration: 6.0,
272
- speed: 0.8, // subtle slow-mo
273
- color_grade: "cinematic",
274
- caption: "SEAMLESS TRANSITIONS",
275
- clip_volume: 0.3 // duck ambient sound
276
- }
277
- ],
278
- audio_path: "/path/to/master_voiceover.m4a",
279
- music_path: "/path/to/background_track.mp3",
280
- music_volume: 0.07,
281
- orientation: "landscape", // or "portrait" for Reels / TikTok / Shorts
282
- auto_fit: true // automatically slice clips to fit total voiceover duration
283
- })
284
- ```
285
-
286
- Supported features:
287
- - **Trimming & Cutting:** Accurate timestamp parsing (`"01:23.45"`, `"83.45s"`, numeric seconds).
288
- - **Speed Ramping:** `0.25x` to `4.0x` with chained `atempo` filters preserving pitch.
289
- - **Cinematic Color Grading:** `cinematic`, `warm`, `cool`, `vintage`, `bw`, `vibrant`, `cyberpunk`.
290
- - **Canvas Reframe:** `landscape` (1920×1080), `portrait` (1080×1920), `square` (1080×1080).
291
- - **Transitions:** `fade`, `dissolve`, `wipeleft`, `wiperight`, `slideup`, `slidedown`, `circleopen`, `smoothleft`.
292
- - **Audio Mixing & Ducking:** Original ambient audio level control, master voice track, background music with sidechain ducking and LUFS normalisation.
293
-
294
- ### Voice Synthesis (per call)
295
-
296
- ```
297
- multi_voice_tts({
298
- text: "Narration text here",
299
- outputPath: "/path/in/files-dir/voice_0.m4a",
300
- platform: "elevenlabs",
301
- voice: "your_cloned_voice_id",
302
- apiKey: "sk_..."
303
- })
304
- ```
305
-
306
- ### Overnight batch
307
-
308
- ```
309
- I'm going to sleep. Make as many videos as you can before 9 AM.
310
- Topics (queue them all):
311
- 1. How does GPS actually work?
312
- 2. Why planes don't fly over the Pacific in a straight line
313
- 3. The engineering inside a modern CPU
314
-
315
- Rules:
316
- - Strong hook, double-verify every fact.
317
- - After each video: run the quality gate, then publish to the output directory.
318
- - Leave me a summary when done.
319
- ```
320
-
321
- The skill calls MCP tools sequentially using the StateMachine to track progress.
322
- If a stage fails it retries with exponential backoff; if it fails terminally it
323
- marks that production FAILED and moves to the next topic.
324
-
325
- ---
326
-
327
- ## Provider reference
328
-
329
- | Role | Provider | Key env var | Notes |
330
- |-------|-------------|--------------------------|-------------------------------|
331
- | LLM | groq | `GROQ_API_KEY` | Default. Fast, free tier. |
332
- | LLM | openrouter | `OPENROUTER_API_KEY` | Any model via one key |
333
- | LLM | anthropic | `ANTHROPIC_API_KEY` | Claude Sonnet/Haiku/Opus |
334
- | LLM | gemini | `GEMINI_API_KEY` | Gemini 2.0 Flash / 2.5 Pro |
335
- | LLM | cerebras | `CEREBRAS_API_KEY` | 2000+ tok/s Llama |
336
- | LLM | openai | `OPENAI_API_KEY` | GPT-4o / o1 |
337
- | LLM | ollama | _(none)_ | Local, OLLAMA_BASE_URL |
338
- | TTS | edge-tts | _(none)_ | Free, offline, 300+ voices |
339
- | TTS | voxforge | `VOXFORGE_URL` + `_KEY` | Local neural TTS & cloning (Rust) |
340
- | TTS | elevenlabs | `ELEVENLABS_API_KEY` | Voice cloning |
341
- | TTS | azure | `AZURE_SPEECH_KEY` | Enterprise |
342
- | TTS | openai | `OPENAI_API_KEY` | High quality |
343
- | Visual| code-driven | _(none)_ | Default — pure-t canvas animation & kinetic text |
344
- | Visual| user-media | _(none)_ | Caller `image_path` / `video_path` |
345
-
346
- ---
347
-
348
- ## Quality gates (hard — will block delivery)
349
-
350
- | Gate | Threshold |
351
- |---------------------|----------------------|
352
- | File size | > 1 MB |
353
- | Video stream | present |
354
- | Resolution (master) | 1920×1080 |
355
- | Resolution (short) | 1080×1920 |
356
- | Codec | H.264 |
357
- | Pixel format | yuv420p |
358
- | Frame rate | 24–30 fps |
359
- | Integrated loudness | −14 ±1.5 LUFS |
360
- | True peak | ≤ −0.5 dBTP |
361
- | Black frames | none > 0.5 s |
362
- | Frozen frames | none > 0.5 s |
363
-
364
- ---
365
-
366
- ## Key files
367
-
368
- ```
369
- src/mcp-server/server.js MCP HTTP server (21 tools, port 3210)
370
- src/mcp-server/stdio.js MCP stdio server (21 tools)
371
- cloudflare-mcp/src/index.js Cloudflare Workers Remote MCP proxy (21 tools)
372
- src/pipeline/providers.js LLM / TTS / Image abstraction (env-var-switchable providers)
373
- src/pipeline/state-machine.js File + PostgreSQL production state
374
- src/pipeline/retry-policy.js Exponential backoff, stage timeouts
375
- src/pipeline/telemetry.js NDJSON cost tracking
376
- src/quality-gate/gate.js 9 hard gates
377
- src/audio-mixer/mixer.js Sidechain ducking, LUFS normalisation
378
- src/shorts-extractor/extractor.js Silence-based 9:16 crop for Shorts
379
- src/fact-checker/checker.js Claim risk classification + evidence gates
380
- src/video-renderer/renderer.js Scene render + master assembly
381
- src/worker/server-render-free.js Render-optimised worker API (Render free)
382
- src/worker/server.js Full worker API
383
- docs/env.template Environment variable reference
384
- ```
1
+ ---
2
+ name: video-pipeline
3
+ description: >
4
+ Autonomous AI video production pipeline. Given a topic, this skill
5
+ researches, fact-checks, writes the script, synthesises voice, renders
6
+ with FFmpeg, runs quality gates, and saves the output locally or POSTs
7
+ to any webhook — all without human intervention.
8
+ Works in IBM Bob, Codex, opencode, Cursor, Windsurf, Claude Code,
9
+ or any IDE that loads skills from ~/.claude/skills/ or .claude/skills/.
10
+ Trigger phrases: "make a video about", "produce a reel on", "overnight
11
+ batch", "queue videos", "generate video", "run the pipeline", "publish
12
+ video". Also activates when a user pastes a topic list and says
13
+ "make these" or "do the batch".
14
+ ---
15
+
16
+ # Video Pipeline Skill
17
+
18
+ End-to-end autonomous video production. The AI proposes every creative
19
+ decision; deterministic systems (FFmpeg, TTS, image gen, MCP) execute them;
20
+ a quality gate blocks bad output before delivery.
21
+
22
+ ## Architecture
23
+
24
+ ```
25
+ IDE Agent (IBM Bob / Codex / opencode / Cursor / Windsurf / Claude Code)
26
+ │
27
+ ▼ Cloudflare Remote MCP (recommended) ─────────────────────────────
28
+ │ https://video-mcp.YOUR_SUBDOMAIN.workers.dev/mcp
29
+ │ 28 tools: create_video_from_script · edit_video · render_from_scenes
30
+ │ generate_scene_code · preview_scene · verify_scene_quality
31
+ │ ingest_media · edit_scenes · render_tts · multi_voice_tts
32
+ │ generate_scene_image · assemble_video · publish_video
33
+ │ research_topic · write_script · external_platform_call
34
+ │ queue_full_pipeline · get_job_status · export_to_editor
35
+ │ scaffold_animation_project · validate_storyboard · render_animation
36
+ │ plan_video_edit · sync_upstream_skills · check_upstream_updates
37
+ │ transcribe_media · ingest_url · mix_audio_pro
38
+ │
39
+ ▼ OR — Local stdio / HTTP MCP ────────────────────────────────────
40
+ │ node src/mcp-server/stdio.js (stdio)
41
+ │ node src/mcp-server/server.js (HTTP :3210)
42
+ │
43
+ ▼ Execution layer
44
+ │ LLM provider: PROVIDER_LLM=groq|openrouter|anthropic|gemini
45
+ │ |mistral|together|cerebras|openai
46
+ │ |ollama|webhook
47
+ │ TTS provider: PROVIDER_TTS=edge-tts|voxforge|elevenlabs|azure|openai|webhook
48
+ │ Visuals: Code-driven animations (default) + user media (video_path / image_path)
49
+ │
50
+ ▼ Optional: queue_full_pipeline
51
+ queue_full_pipeline MCP tool → POST job to worker API (WORKER_API_URL)
52
+ ```
53
+
54
+ All secrets come from environment variables — **never** hardcoded.
55
+ See `docs/env.template` for the full reference.
56
+
57
+ ---
58
+
59
+ ## Setup
60
+
61
+ ### Option A — Cloudflare Remote MCP (Recommended)
62
+
63
+ Deploy once; works in every IDE with no local server process:
64
+
65
+ ```bash
66
+ cd cloudflare-mcp
67
+ npm install
68
+ wrangler secret put WORKER_API_URL # https://your-render-app.onrender.com
69
+ wrangler secret put MCP_AUTH_TOKEN # random 32-byte hex
70
+ wrangler secret put INTERNAL_API_TOKEN # shared secret with render node
71
+ wrangler deploy
72
+ ```
73
+
74
+ Add to any IDE's MCP config:
75
+
76
+ ```json
77
+ {
78
+ "mcpServers": {
79
+ "frameforge": {
80
+ "url": "https://video-mcp.YOUR_SUBDOMAIN.workers.dev/mcp",
81
+ "headers": { "Authorization": "Bearer YOUR_MCP_AUTH_TOKEN" }
82
+ }
83
+ }
84
+ }
85
+ ```
86
+
87
+ ### Option B — Local stdio MCP
88
+
89
+ **IBM Bob** (`~/.bob/mcp.json`):
90
+ ```json
91
+ {
92
+ "mcpServers": {
93
+ "frameforge": {
94
+ "command": "node",
95
+ "args": ["<absolute-path>/src/mcp-server/stdio.js"],
96
+ "env": { "PROVIDER_LLM": "groq", "GROQ_API_KEY": "gsk_..." }
97
+ }
98
+ }
99
+ }
100
+ ```
101
+
102
+ **Codex / opencode / Cursor / Windsurf / Claude Code** (`~/.claude/mcp.json` or `.mcp.json`):
103
+ ```json
104
+ {
105
+ "mcpServers": {
106
+ "frameforge": {
107
+ "command": "node",
108
+ "args": ["<absolute-path>/src/mcp-server/stdio.js"],
109
+ "env": { "PROVIDER_LLM": "groq", "GROQ_API_KEY": "${GROQ_API_KEY}" }
110
+ }
111
+ }
112
+ }
113
+ ```
114
+
115
+ ### 2. Set environment variables
116
+
117
+ Copy `docs/env.template` → `.env` and fill in. Minimum:
118
+
119
+ ```bash
120
+ # LLM (pick one — Groq is free with no credit card)
121
+ PROVIDER_LLM=groq
122
+ GROQ_API_KEY=gsk_...
123
+
124
+ # TTS (edge-tts is free offline)
125
+ PROVIDER_TTS=edge-tts
126
+
127
+ # Visuals are 100% code-driven (kinetic typography/motion graphics) + user media. No image key needed.
128
+
129
+ # Delivery (optional — default is 'file' which copies to N8N_FILES_DIR/published/)
130
+ # PUBLISH_WEBHOOK_URL=https://hooks.example.com/frameforge
131
+ # REVIEW_NOTIFY_URL=https://hooks.example.com/review-gate
132
+ ```
133
+
134
+ Run the interactive wizard for guided setup:
135
+ ```bash
136
+ npm run setup
137
+ ```
138
+
139
+ ---
140
+
141
+ ## Workflow (step-by-step)
142
+
143
+ ### Path 1 — Agent-First (Code-Driven Motion Graphics or Direct Custom Code)
144
+
145
+ **You are already running an LLM (Claude Code / Opus 5.5 / Cursor / Bob) — use it.**
146
+ Write the scenes yourself and hand them to FrameForge for rendering. No external API key needed.
147
+ FrameForge renders full-canvas, broadcast-quality motion graphics (24fps pure function of `t`) with zero slideshow/card patterns:
148
+
149
+ - **Option A — Natural Description:** Provide `visual_description` (e.g. "Earth rotating with 4 satellite orbits and microwave beams intersecting"). FrameForge automatically synthesizes a full-canvas procedural 3D/2D animation entity (orbital mechanics, 3D DNA helix, candlestick charts, neural mesh, 3D wireframe polyhedra).
150
+ - **Option B — Direct Agent Code:** Provide `html` (complete page with `window.DURATION` & `window.seek(t)`) or `code` (canvas drawing instructions for `t`). FrameForge executes your program directly via headless Chromium.
151
+ - **Option C — Generate & Inspect (Layer 3 Loop):** Call `generate_scene_code({ visual_description, style })` to synthesize canvas drawing code. Then call `preview_scene({ html, timestamps: [0.1, 1.5, 2.9] })` to render PNG stills to disk, visually verify the scene in the IDE, tweak if needed, and pass into `create_video_from_script`.
152
+ - **Option D — Multilingual & Regional Scripts:** Pass `language: "ml"` (Malayalam), `"hi"` (Hindi), etc. Native DirectWrite/HarfBuzz font stacks (`Nirmala UI`, `Noto Sans Malayalam`) render conjuncts without broken glyphs.
153
+
154
+ ```js
155
+ create_video_from_script({
156
+ topic: "GPS Satellites and Orbital Mechanics",
157
+ style: "cosmic_space",
158
+ language: "en",
159
+ scenes: [
160
+ {
161
+ narration: "GPS uses trilateration from atomic clock satellites orbiting Earth.",
162
+ caption: "TRILATERATION",
163
+ visual_description: "3D Earth rotating with 4 satellite orbits and microwave beams converging",
164
+ // Optional: you can pass your own Canvas code or complete HTML:
165
+ // code: "ctx.fillStyle = '#0c1116'; ctx.fillRect(0,0,W,H); ..."
166
+ },
167
+ {
168
+ narration: "Relativistic time dilation requires nanosecond daily corrections.",
169
+ caption: "TIME DILATION",
170
+ visual_description: "Quantum atomic clock oscillation with compounding nanosecond curve"
171
+ }
172
+ ],
173
+ title: "How GPS Actually Works"
174
+ })
175
+ ```
176
+
177
+ Then deliver it:
178
+ ```
179
+ publish_video({ productionId, masterPath, channels: ["file"] })
180
+ ```
181
+
182
+ ### Path 2 — Code-Driven Animated Video (Playwright + ffmpeg, no AI images)
183
+
184
+ Use when: the user wants Remotion-style motion graphics, image gen is unavailable, or they ask for "animated video" or "make it look good". Every pixel is a pure function of time `t`.
185
+
186
+ > **Model effort (Opus 5.5 / Claude Code):**
187
+ > - Beat sheet + first scene authorship → `effort: xhigh`
188
+ > - Storyboard review, strip-range patches, encode-only runs → `effort: medium`
189
+ > - `tool_choice` must be `auto` (forced tool use not supported in Opus 5.5)
190
+
191
+ > **Beat sheet gate:** Reply with a beat sheet and 4 keyframe descriptions first. Stop. Wait for approval before writing any scene code.
192
+
193
+ ```
194
+ // 1. Synthesise voice first — audio timing is ground truth
195
+ multi_voice_tts({ text: "Narration.", outputPath: ".frameforge-files/voice_0.m4a" })
196
+
197
+ // 2. Scaffold the compositor project (copies engine + all render scripts + briefings)
198
+ scaffold_animation_project({
199
+ outDir: ".frameforge-files/my-anim/",
200
+ scenes: [ { narration: "...", caption: "...", visual_description: "..." } ],
201
+ audioPath: ".frameforge-files/voice_0.m4a"
202
+ })
203
+ // → returns nextSteps[] — read ANIMATION_GUIDE.md and edit STORYBOARD.md next
204
+
205
+ // 3. Validate storyboard (four hard gates — must pass before scene code)
206
+ validate_storyboard({ storyboardPath: ".frameforge-files/my-anim/STORYBOARD.md" })
207
+
208
+ // 4. Write real animation in edit.js (pure functions of t — read ANIMATION_GUIDE.md)
209
+
210
+ // 5. Pre-flight linter & purity verification (3-point repeat, wander, cold-jump)
211
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "look" })
212
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "verify" })
213
+
214
+ // 6. Review stills (at least one sheet per shot, one strip per key motion)
215
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "sheet" })
216
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "strip",
217
+ timestamps: [2.1, 2.6] }) // every frame of one motion
218
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "crop",
219
+ timestamps: ["9:16", 3.0] }) // mobile crop check
220
+
221
+ // 7. Full render (only after stills pass)
222
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "full",
223
+ audioPath: ".frameforge-files/voice_0.m4a" })
224
+
225
+ // 8. Closed-loop QA fix protocol (mandatory — Opus 5.5 / Claude Code pattern)
226
+ render_animation({ projectDir: ".frameforge-files/my-anim/", mode: "quality" })
227
+ // Returns: { passed, overallScore, worstFrames: [{time, score, issues}] }
228
+ // If passed: false → fix loop (max 3 rounds):
229
+ // 1. Read worstFrames[]. 2. Patch edit.js for those timestamps only.
230
+ // 3. render_animation({ mode: "strip", timestamps: [failingT - 0.5, failingT + 0.5] })
231
+ // 4. render_animation({ mode: "quality" })
232
+ // 5. Repeat until passed or 3 rounds — then escalate to user.
233
+ // Never re-render full film to fix one timestamp. Use strip only.
234
+ ```
235
+
236
+ Requires one-time: `cd <outDir> && npm install playwright && npx playwright install chromium`
237
+
238
+ ### Path 3 — Auto-Research (for unattended / batch runs)
239
+
240
+ Use this when FrameForge runs **without** an IDE in the loop — overnight batch,
241
+ `queue_full_pipeline`, or a standalone worker. The pipeline calls its own LLM
242
+ for research and scripting, so `PROVIDER_LLM` + the matching API key are required:
243
+
244
+ ```
245
+ 1. research_topic({ topic: "...", style: "educational", format: "short-form reel" })
246
+ 2. write_script({ productionId, claims, topic, persona: "tech reviewer" })
247
+ 3. render_from_scenes({ productionId })
248
+ 4. publish_video({ productionId, masterPath, channels: ["file"] })
249
+ ```
250
+
251
+ ### Path 4 — Autonomous Local Video Editing (`edit_video`)
252
+
253
+ When the user provides video clips (B-roll, screen recordings, camera footage) and audio/narration, FrameForge autonomously cuts, trims, color-grades, reframes, captions, and stitches them with seamless transitions locally:
254
+
255
+ ```js
256
+ edit_video({
257
+ clips: [
258
+ {
259
+ video_path: "/path/to/clip1.mp4",
260
+ start_time: "00:05.5",
261
+ end_time: "00:15.0",
262
+ speed: 1.0,
263
+ color_grade: "cinematic",
264
+ caption: "AUTONOMOUS EDITING",
265
+ transition: "dissolve",
266
+ transition_duration: 0.5
267
+ },
268
+ {
269
+ video_path: "/path/to/clip2.mp4",
270
+ start_time: 0,
271
+ duration: 6.0,
272
+ speed: 0.8, // subtle slow-mo
273
+ color_grade: "cinematic",
274
+ caption: "SEAMLESS TRANSITIONS",
275
+ clip_volume: 0.3 // duck ambient sound
276
+ }
277
+ ],
278
+ audio_path: "/path/to/master_voiceover.m4a",
279
+ music_path: "/path/to/background_track.mp3",
280
+ music_volume: 0.07,
281
+ orientation: "landscape", // or "portrait" for Reels / TikTok / Shorts
282
+ auto_fit: true // automatically slice clips to fit total voiceover duration
283
+ })
284
+ ```
285
+
286
+ Supported features:
287
+ - **Trimming & Cutting:** Accurate timestamp parsing (`"01:23.45"`, `"83.45s"`, numeric seconds).
288
+ - **Speed Ramping:** `0.25x` to `4.0x` with chained `atempo` filters preserving pitch.
289
+ - **Cinematic Color Grading:** `cinematic`, `warm`, `cool`, `vintage`, `bw`, `vibrant`, `cyberpunk`.
290
+ - **Canvas Reframe:** `landscape` (1920×1080), `portrait` (1080×1920), `square` (1080×1080).
291
+ - **Transitions:** `fade`, `dissolve`, `wipeleft`, `wiperight`, `slideup`, `slidedown`, `circleopen`, `smoothleft`.
292
+ - **Audio Mixing & Ducking:** Original ambient audio level control, master voice track, background music with sidechain ducking and LUFS normalisation.
293
+
294
+ ### Voice Synthesis (per call)
295
+
296
+ ```
297
+ multi_voice_tts({
298
+ text: "Narration text here",
299
+ outputPath: "/path/in/files-dir/voice_0.m4a",
300
+ platform: "elevenlabs",
301
+ voice: "your_cloned_voice_id",
302
+ apiKey: "sk_..."
303
+ })
304
+ ```
305
+
306
+ ### Overnight batch
307
+
308
+ ```
309
+ I'm going to sleep. Make as many videos as you can before 9 AM.
310
+ Topics (queue them all):
311
+ 1. How does GPS actually work?
312
+ 2. Why planes don't fly over the Pacific in a straight line
313
+ 3. The engineering inside a modern CPU
314
+
315
+ Rules:
316
+ - Strong hook, double-verify every fact.
317
+ - After each video: run the quality gate, then publish to the output directory.
318
+ - Leave me a summary when done.
319
+ ```
320
+
321
+ The skill calls MCP tools sequentially using the StateMachine to track progress.
322
+ If a stage fails it retries with exponential backoff; if it fails terminally it
323
+ marks that production FAILED and moves to the next topic.
324
+
325
+ ---
326
+
327
+ ## Provider reference
328
+
329
+ | Role | Provider | Key env var | Notes |
330
+ |-------|-------------|--------------------------|-------------------------------|
331
+ | LLM | groq | `GROQ_API_KEY` | Default. Fast, free tier. |
332
+ | LLM | openrouter | `OPENROUTER_API_KEY` | Any model via one key |
333
+ | LLM | anthropic | `ANTHROPIC_API_KEY` | Claude Sonnet/Haiku/Opus |
334
+ | LLM | gemini | `GEMINI_API_KEY` | Gemini 2.0 Flash / 2.5 Pro |
335
+ | LLM | cerebras | `CEREBRAS_API_KEY` | 2000+ tok/s Llama |
336
+ | LLM | openai | `OPENAI_API_KEY` | GPT-4o / o1 |
337
+ | LLM | ollama | _(none)_ | Local, OLLAMA_BASE_URL |
338
+ | TTS | edge-tts | _(none)_ | Free, offline, 300+ voices |
339
+ | TTS | voxforge | `VOXFORGE_URL` + `_KEY` | Local neural TTS & cloning (Rust) |
340
+ | TTS | elevenlabs | `ELEVENLABS_API_KEY` | Voice cloning |
341
+ | TTS | azure | `AZURE_SPEECH_KEY` | Enterprise |
342
+ | TTS | openai | `OPENAI_API_KEY` | High quality |
343
+ | Visual| code-driven | _(none)_ | Default — pure-t canvas animation & kinetic text |
344
+ | Visual| user-media | _(none)_ | Caller `image_path` / `video_path` |
345
+
346
+ ---
347
+
348
+ ## Quality gates (hard — will block delivery)
349
+
350
+ | Gate | Threshold |
351
+ |---------------------|----------------------|
352
+ | File size | > 1 MB |
353
+ | Video stream | present |
354
+ | Resolution (master) | 1920×1080 |
355
+ | Resolution (short) | 1080×1920 |
356
+ | Codec | H.264 |
357
+ | Pixel format | yuv420p |
358
+ | Frame rate | 24–30 fps |
359
+ | Integrated loudness | −14 ±1.5 LUFS |
360
+ | True peak | ≤ −0.5 dBTP |
361
+ | Black frames | none > 0.5 s |
362
+ | Frozen frames | none > 0.5 s |
363
+
364
+ ---
365
+
366
+ ## Key files
367
+
368
+ ```
369
+ src/mcp-server/server.js MCP HTTP server (21 tools, port 3210)
370
+ src/mcp-server/stdio.js MCP stdio server (21 tools)
371
+ cloudflare-mcp/src/index.js Cloudflare Workers Remote MCP proxy (21 tools)
372
+ src/pipeline/providers.js LLM / TTS / Image abstraction (env-var-switchable providers)
373
+ src/pipeline/state-machine.js File + PostgreSQL production state
374
+ src/pipeline/retry-policy.js Exponential backoff, stage timeouts
375
+ src/pipeline/telemetry.js NDJSON cost tracking
376
+ src/quality-gate/gate.js 9 hard gates
377
+ src/audio-mixer/mixer.js Sidechain ducking, LUFS normalisation
378
+ src/shorts-extractor/extractor.js Silence-based 9:16 crop for Shorts
379
+ src/fact-checker/checker.js Claim risk classification + evidence gates
380
+ src/video-renderer/renderer.js Scene render + master assembly
381
+ src/worker/server-render-free.js Render-optimised worker API (Render free)
382
+ src/worker/server.js Full worker API
383
+ docs/env.template Environment variable reference
384
+ ```