coreslopper 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (212) hide show
  1. package/CHANGELOG.md +26 -0
  2. package/LICENSE +21 -0
  3. package/README.md +471 -2
  4. package/dist/cli.js +2487 -0
  5. package/dist/cli.js.map +1 -0
  6. package/package.json +92 -4
  7. package/runtime/.claude/agents/art-director.md +40 -0
  8. package/runtime/.claude/agents/qa-checker.md +31 -0
  9. package/runtime/.claude/agents/stage-runner.md +50 -0
  10. package/runtime/.claude/commands/new-video.md +19 -0
  11. package/runtime/.claude/commands/video-next.md +55 -0
  12. package/runtime/.claude/commands/video-status.md +17 -0
  13. package/runtime/.claude/hooks/validate-artifacts.mjs +88 -0
  14. package/runtime/.claude/settings.json +15 -0
  15. package/runtime/.claude/skills/ai-clips/SKILL.md +88 -0
  16. package/runtime/.claude/skills/assembly-render/SKILL.md +48 -0
  17. package/runtime/.claude/skills/audio-analysis/SKILL.md +49 -0
  18. package/runtime/.claude/skills/character-bible/SKILL.md +44 -0
  19. package/runtime/.claude/skills/domain-modeling/ADR-FORMAT.md +47 -0
  20. package/runtime/.claude/skills/domain-modeling/GLOSSARY-FORMAT.md +60 -0
  21. package/runtime/.claude/skills/domain-modeling/SKILL.md +74 -0
  22. package/runtime/.claude/skills/domain-modeling/agents/openai.yaml +3 -0
  23. package/runtime/.claude/skills/footage/SKILL.md +49 -0
  24. package/runtime/.claude/skills/grill-me/SKILL.md +7 -0
  25. package/runtime/.claude/skills/grill-me/agents/openai.yaml +5 -0
  26. package/runtime/.claude/skills/grilling/SKILL.md +30 -0
  27. package/runtime/.claude/skills/grilling/agents/openai.yaml +3 -0
  28. package/runtime/.claude/skills/lyrics-writer/SKILL.md +43 -0
  29. package/runtime/.claude/skills/remotion-motion/SKILL.md +50 -0
  30. package/runtime/.claude/skills/storyboard/SKILL.md +53 -0
  31. package/runtime/.claude/skills/style-bible/SKILL.md +47 -0
  32. package/runtime/.claude/skills/suno-song/SKILL.md +45 -0
  33. package/runtime/.claude/skills/video-brief/SKILL.md +44 -0
  34. package/runtime/.claude/skills/video-qa/SKILL.md +50 -0
  35. package/runtime/.claude/skills/video-treatment/SKILL.md +44 -0
  36. package/runtime/.claude/skills/writing-for-agents/SKILL-MECHANICS.md +22 -0
  37. package/runtime/.claude/skills/writing-for-agents/SKILL.md +81 -0
  38. package/runtime/.claude/skills/writing-for-agents/agents/openai.yaml +3 -0
  39. package/runtime/.editorconfig +15 -0
  40. package/runtime/.env.example +21 -0
  41. package/runtime/.gitattributes +14 -0
  42. package/runtime/.github/ISSUE_TEMPLATE/bug_report.md +26 -0
  43. package/runtime/.github/ISSUE_TEMPLATE/feature_request.md +22 -0
  44. package/runtime/.github/PULL_REQUEST_TEMPLATE.md +31 -0
  45. package/runtime/.github/workflows/ci.yml +91 -0
  46. package/runtime/.github/workflows/release.yml +78 -0
  47. package/runtime/.mcp.json +8 -0
  48. package/runtime/.nvmrc +1 -0
  49. package/runtime/.release-please-manifest.json +1 -0
  50. package/runtime/AGENTS.md +184 -0
  51. package/runtime/CHANGELOG.md +26 -0
  52. package/runtime/CLAUDE.md +57 -0
  53. package/runtime/CODE_OF_CONDUCT.md +50 -0
  54. package/runtime/CONTRIBUTING.md +132 -0
  55. package/runtime/LICENSE +21 -0
  56. package/runtime/README.md +472 -0
  57. package/runtime/SECURITY.md +41 -0
  58. package/runtime/brand/example/brand.yaml +38 -0
  59. package/runtime/brand/example/fonts/JetBrainsMono.ttf +0 -0
  60. package/runtime/brand/example/fonts/OFL-JetBrainsMono.txt +93 -0
  61. package/runtime/brand/example/fonts/OFL-Orbitron.txt +93 -0
  62. package/runtime/brand/example/fonts/OFL-SpaceGrotesk.txt +93 -0
  63. package/runtime/brand/example/fonts/Orbitron.ttf +0 -0
  64. package/runtime/brand/example/fonts/SpaceGrotesk.ttf +0 -0
  65. package/runtime/brand/example/logo.svg +22 -0
  66. package/runtime/brand/example/styleguide.md +50 -0
  67. package/runtime/brand/example/tokens.json +21 -0
  68. package/runtime/config/defaults.yaml +31 -0
  69. package/runtime/config/models.yaml +253 -0
  70. package/runtime/docs/PLAN.md +663 -0
  71. package/runtime/docs/THIRD_PARTY.md +113 -0
  72. package/runtime/docs/adr/0001-folder-based-projects.md +17 -0
  73. package/runtime/docs/adr/0002-autonomous-pipeline.md +17 -0
  74. package/runtime/docs/adr/0003-stage-isolated-execution.md +17 -0
  75. package/runtime/docs/adr/0004-zod-schema-source.md +17 -0
  76. package/runtime/docs/adr/0005-symbolic-timing-anchors.md +17 -0
  77. package/runtime/docs/adr/0006-fal-adapter-cache-budget.md +19 -0
  78. package/runtime/docs/adr/0007-remotion-deterministic-renderer.md +16 -0
  79. package/runtime/docs/adr/0008-local-audio-analysis.md +17 -0
  80. package/runtime/docs/adr/0009-mit-open-source.md +17 -0
  81. package/runtime/docs/adr/0010-npx-onboarding.md +21 -0
  82. package/runtime/docs/adr/0011-higgsfield-provider.md +23 -0
  83. package/runtime/docs/adr/README.md +19 -0
  84. package/runtime/docs/architecture.md +183 -0
  85. package/runtime/docs/assets/demo-still.png +0 -0
  86. package/runtime/docs/glossary.md +205 -0
  87. package/runtime/docs/mcp.md +51 -0
  88. package/runtime/docs/onboarding.md +169 -0
  89. package/runtime/docs/pipeline.md +186 -0
  90. package/runtime/package-lock.json +6868 -0
  91. package/runtime/package.json +75 -0
  92. package/runtime/packages/cli/package.json +6 -0
  93. package/runtime/packages/cli/src/contact-sheet.ts +124 -0
  94. package/runtime/packages/cli/src/doctor.ts +148 -0
  95. package/runtime/packages/cli/src/generate.ts +233 -0
  96. package/runtime/packages/cli/src/index.ts +453 -0
  97. package/runtime/packages/cli/src/init-command.ts +142 -0
  98. package/runtime/packages/cli/src/init.ts +374 -0
  99. package/runtime/packages/cli/src/media.ts +184 -0
  100. package/runtime/packages/cli/src/new-project.ts +155 -0
  101. package/runtime/packages/cli/src/npx.ts +186 -0
  102. package/runtime/packages/cli/src/onboarding.ts +430 -0
  103. package/runtime/packages/cli/src/qa.ts +183 -0
  104. package/runtime/packages/cli/src/render.ts +177 -0
  105. package/runtime/packages/cli/src/status.ts +23 -0
  106. package/runtime/packages/cli/src/tools.ts +354 -0
  107. package/runtime/packages/cli/src/utils.ts +52 -0
  108. package/runtime/packages/cli/src/validate-file.ts +78 -0
  109. package/runtime/packages/cli/src/validate.ts +65 -0
  110. package/runtime/packages/cli/src/workspace.ts +118 -0
  111. package/runtime/packages/core/package.json +9 -0
  112. package/runtime/packages/core/src/cost.ts +47 -0
  113. package/runtime/packages/core/src/errors.ts +37 -0
  114. package/runtime/packages/core/src/hash.ts +37 -0
  115. package/runtime/packages/core/src/index.ts +8 -0
  116. package/runtime/packages/core/src/models.ts +90 -0
  117. package/runtime/packages/core/src/project.ts +246 -0
  118. package/runtime/packages/core/src/schema.ts +380 -0
  119. package/runtime/packages/core/src/state.ts +237 -0
  120. package/runtime/packages/core/src/timing.ts +419 -0
  121. package/runtime/packages/providers/package.json +9 -0
  122. package/runtime/packages/providers/src/fal.ts +166 -0
  123. package/runtime/packages/providers/src/higgsfield.ts +292 -0
  124. package/runtime/packages/providers/src/http.ts +149 -0
  125. package/runtime/packages/providers/src/index.ts +19 -0
  126. package/runtime/packages/providers/src/mock.ts +71 -0
  127. package/runtime/packages/providers/src/runner.ts +210 -0
  128. package/runtime/packages/providers/src/types.ts +65 -0
  129. package/runtime/packages/remotion/README.md +49 -0
  130. package/runtime/packages/remotion/package.json +6 -0
  131. package/runtime/packages/remotion/remotion.config.ts +5 -0
  132. package/runtime/packages/remotion/src/MusicVideo.tsx +52 -0
  133. package/runtime/packages/remotion/src/Root.tsx +25 -0
  134. package/runtime/packages/remotion/src/components/Backdrops.tsx +230 -0
  135. package/runtime/packages/remotion/src/components/Grain.tsx +21 -0
  136. package/runtime/packages/remotion/src/components/MediaLayer.tsx +130 -0
  137. package/runtime/packages/remotion/src/components/NeonTorus.tsx +71 -0
  138. package/runtime/packages/remotion/src/components/ShotRenderer.tsx +151 -0
  139. package/runtime/packages/remotion/src/components/TextOverlay.tsx +188 -0
  140. package/runtime/packages/remotion/src/components/Transition.tsx +37 -0
  141. package/runtime/packages/remotion/src/fonts.ts +26 -0
  142. package/runtime/packages/remotion/src/hooks/index.ts +1 -0
  143. package/runtime/packages/remotion/src/hooks/useTiming.ts +56 -0
  144. package/runtime/packages/remotion/src/index.ts +4 -0
  145. package/runtime/packages/remotion/src/lib/timing.ts +45 -0
  146. package/runtime/packages/remotion/src/types.ts +85 -0
  147. package/runtime/projects/demo/assets/.gitkeep +0 -0
  148. package/runtime/projects/demo/audio/.gitkeep +0 -0
  149. package/runtime/projects/demo/brief.md +41 -0
  150. package/runtime/projects/demo/characters.yaml +2 -0
  151. package/runtime/projects/demo/lyrics.txt +12 -0
  152. package/runtime/projects/demo/project.yaml +31 -0
  153. package/runtime/projects/demo/renders/.gitkeep +0 -0
  154. package/runtime/projects/demo/review/.gitkeep +0 -0
  155. package/runtime/projects/demo/review/contact-sheet.svg +69 -0
  156. package/runtime/projects/demo/review/report.md +20 -0
  157. package/runtime/projects/demo/song.json +12 -0
  158. package/runtime/projects/demo/state.json +116 -0
  159. package/runtime/projects/demo/storyboard.md +28 -0
  160. package/runtime/projects/demo/storyboard.yaml +101 -0
  161. package/runtime/projects/demo/style.yaml +37 -0
  162. package/runtime/projects/demo/suno-style.txt +1 -0
  163. package/runtime/projects/demo/timing.json +371 -0
  164. package/runtime/projects/demo/timing.overrides.json +6 -0
  165. package/runtime/projects/demo/timing.report.md +18 -0
  166. package/runtime/projects/demo/treatment.md +29 -0
  167. package/runtime/release-please-config.json +25 -0
  168. package/runtime/schema/brand.schema.json +124 -0
  169. package/runtime/schema/characters.schema.json +50 -0
  170. package/runtime/schema/models.schema.json +131 -0
  171. package/runtime/schema/project.schema.json +210 -0
  172. package/runtime/schema/state.schema.json +134 -0
  173. package/runtime/schema/storyboard.schema.json +534 -0
  174. package/runtime/schema/style.schema.json +114 -0
  175. package/runtime/schema/timing-overrides.schema.json +113 -0
  176. package/runtime/schema/timing.schema.json +205 -0
  177. package/runtime/scripts/audio-analysis/README.md +68 -0
  178. package/runtime/scripts/audio-analysis/analyze.py +433 -0
  179. package/runtime/scripts/audio-analysis/conftest.py +4 -0
  180. package/runtime/scripts/audio-analysis/pyproject.toml +30 -0
  181. package/runtime/scripts/audio-analysis/tests/test_analyze_integration.py +60 -0
  182. package/runtime/scripts/audio-analysis/tests/test_timing_core.py +216 -0
  183. package/runtime/scripts/audio-analysis/timing_core.py +674 -0
  184. package/runtime/scripts/audio-analysis/uv.lock +3905 -0
  185. package/runtime/scripts/fixtures/generate-audio.mjs +97 -0
  186. package/runtime/scripts/fixtures/generate-demo-timing.mjs +115 -0
  187. package/runtime/scripts/generate-schema.ts +40 -0
  188. package/runtime/scripts/install-ffmpeg.sh +37 -0
  189. package/runtime/scripts/setup.sh +27 -0
  190. package/runtime/skills-lock.json +29 -0
  191. package/runtime/templates/project/README.md +38 -0
  192. package/runtime/templates/project/assets/.gitkeep +0 -0
  193. package/runtime/templates/project/audio/.gitkeep +0 -0
  194. package/runtime/templates/project/brand/.gitkeep +0 -0
  195. package/runtime/templates/project/brief.md +39 -0
  196. package/runtime/templates/project/characters/.gitkeep +0 -0
  197. package/runtime/templates/project/lyrics.txt +9 -0
  198. package/runtime/templates/project/renders/.gitkeep +0 -0
  199. package/runtime/templates/project/review/.gitkeep +0 -0
  200. package/runtime/templates/project/suno-style.txt +1 -0
  201. package/runtime/templates/project/timing.overrides.json +6 -0
  202. package/runtime/templates/project/treatment.md +21 -0
  203. package/runtime/tests/contact-sheet.test.ts +39 -0
  204. package/runtime/tests/higgsfield.test.ts +243 -0
  205. package/runtime/tests/init.test.ts +161 -0
  206. package/runtime/tests/models.test.ts +90 -0
  207. package/runtime/tests/providers.test.ts +133 -0
  208. package/runtime/tests/schema.test.ts +30 -0
  209. package/runtime/tests/state.test.ts +59 -0
  210. package/runtime/tests/timing.test.ts +114 -0
  211. package/runtime/tsconfig.json +32 -0
  212. package/runtime/vitest.config.ts +18 -0
package/CHANGELOG.md ADDED
@@ -0,0 +1,26 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 (2026-11-08)
4
+
5
+
6
+ ### Features
7
+
8
+ * add example brand kit and brand schema ([08ea9d5](https://github.com/hemi1986/coreslopper/commit/08ea9d579fd48e6ff4a4f13e85d4437b3d3b33f5))
9
+ * add Higgsfield provider and setup quickstarts ([6ced780](https://github.com/hemi1986/coreslopper/commit/6ced780ce9effddfb0272e11eff51939610da2f9))
10
+ * add lip-sync clips for storyboard shots ([3403cf6](https://github.com/hemi1986/coreslopper/commit/3403cf6efde67025dafc7ecdee583c31668027ad))
11
+ * add lip-sync shots and lyrics-first alignment ([d6534a9](https://github.com/hemi1986/coreslopper/commit/d6534a954bf6088f01732a525f629da4426a95d9))
12
+ * add npx onboarding and workspace bootstrap ([2393370](https://github.com/hemi1986/coreslopper/commit/2393370392eacb25d9045fb5b700281a4e88b062))
13
+ * add timing hooks, media helpers, stage reset, and MCP docs ([0b9ef19](https://github.com/hemi1986/coreslopper/commit/0b9ef1917aced6a0647f3dc8baa07222997e2f1a))
14
+ * initial coreslopper workflow and motion-only demo ([a088d10](https://github.com/hemi1986/coreslopper/commit/a088d10e63f3312ef94f8acaa29eb8acc4dfeed4))
15
+ * local ffmpeg bootstrap and media command polish ([ab347af](https://github.com/hemi1986/coreslopper/commit/ab347af580ecd4357bc0e4df8390b3d34735bb1c))
16
+
17
+
18
+ ### Bug Fixes
19
+
20
+ * **audio-analysis:** align lyrics first and keep Unicode text ([ac35d0d](https://github.com/hemi1986/coreslopper/commit/ac35d0d33336e61a9bcc68c515f3a85088a8198f))
21
+ * keep character reference media out of git ([bb253f0](https://github.com/hemi1986/coreslopper/commit/bb253f091e7bcc7e761dce64ea90e6aaf6bed367))
22
+
23
+
24
+ ### Documentation
25
+
26
+ * mark workflow as work in progress ([cd5c1e8](https://github.com/hemi1986/coreslopper/commit/cd5c1e8275ebec06f772eb59a313bf3fc7b97b32))
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 coreslopper contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,472 @@
1
- # Temporary Holding Version
1
+ # coreslopper
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ An open-source, Claude Code-native workflow for making music videos: from a brief, through lyrics and a song, to a validated storyboard and a finished Remotion render.
4
+
5
+ > **Status: work in progress.** This workflow is currently being tested end to end. The pipeline, schemas, skills, and provider integrations may still change, and it is not yet clear that every stage works as documented. Treat it as an experiment, not a production-ready tool.
6
+
7
+ ![A frame from the motion-only demo](docs/assets/demo-still.png)
8
+
9
+ The demo is a 12-second, motion-only project. It runs without API keys and is the first vertical slice of the pipeline.
10
+
11
+ ## Why this exists
12
+
13
+ Most AI music-video workflows are either a pile of prompts or a pile of scripts. coreslopper separates them:
14
+
15
+ - **Creative work** (brief, treatment, lyrics, storyboard, prompts) is done by focused agent skills.
16
+ - **Mechanical work** (audio analysis, schema validation, caching, budget checks, rendering, export) is done by tested TypeScript and Python.
17
+ - **Every shot chooses its source**: motion graphics, an AI clip, real footage, or a mix. The decision lives in the storyboard, not in the project setup.
18
+ - **The pipeline is autonomous by default.** It stops only for budget, missing credentials, or an unrecoverable validation failure. Quality is established before the run through grilling and enforced during the run by automated critics.
19
+ - **It is resumable and thrifty.** `state.json` tracks every stage; generated assets are cached by a hash of model, prompt, seed, references, and params.
20
+
21
+ Read [the plan](docs/PLAN.md) for the full rationale, [the ADRs](docs/adr/) for the decisions, and [the glossary](docs/glossary.md) for the language.
22
+
23
+ ## Features
24
+
25
+ - **Brief → treatment → lyrics → song → timing → style → storyboard → assets → render → QA.**
26
+ - **Storyboard coverage validation.** Shots use symbolic anchors (`section`, `line`, `word`, `beat`, `downbeat`) and must cover the song with no gaps or overlaps.
27
+ - **Local-first audio analysis.** Demucs for stems, WhisperX for word alignment, librosa for BPM/beats/downbeats/onsets/energy. The lyrics are authoritative; only timestamps come from ASR.
28
+ - **Provider-agnostic generation.** `generateImage`, `editImage`, `imageToVideo`, and `lipSync` behind a narrow adapter. fal.ai and Higgsfield are the real adapters; a deterministic mock adapter powers tests and no-account runs.
29
+ - **Cost guard.** Mandatory per-project budget, batch cap, pre-batch estimate, manifest, and cache. Unknown pricing is a hard error.
30
+ - **Character consistency.** Reference images are required for recurring characters; frame chaining is available per storyboard transition.
31
+ - **Remotion renderer.** Frame-driven motion graphics, typography, 3D via `@remotion/three`, and a reusable effect library.
32
+ - **Claude Code layer.** `/new-video`, `/video-next`, `/video-status`, fresh-context stage runners, art-director and QA critics, and schema-validating hooks.
33
+ - **Open-source safe.** MIT for the workflow, no secrets, no copyrighted media, a license-safe demo, and explicit third-party notices.
34
+
35
+ ## Requirements
36
+
37
+ - Node.js 22 or newer.
38
+ - Python 3.12 or newer and [uv](https://docs.astral.sh/uv/).
39
+ - `ffmpeg` and `ffprobe` on `PATH` for footage import and loudness normalization. The motion-only demo does not need them. On Linux x64 without a package manager, run `./scripts/install-ffmpeg.sh`; the media commands check `.tools/ffmpeg` automatically.
40
+ - Optional: a CUDA GPU for WhisperX/Demucs. CPU works but is slower.
41
+ - Optional: a Suno account for song generation, a `FAL_KEY` for fal.ai image/video generation, and a Higgsfield API credential for Higgsfield image/video generation.
42
+
43
+ ## Install with npx
44
+
45
+ Create a self-contained workspace without cloning the repository:
46
+
47
+ ```bash
48
+ npx coreslopper@latest init my-video
49
+ ```
50
+
51
+ The onboarding wizard:
52
+
53
+ - checks Node 22+, npm, git, uv, Python 3.12, ffmpeg, and Claude Code;
54
+ - offers to install uv, Python 3.12, ffmpeg, and Claude Code;
55
+ - asks how the song is made (Suno or import) and how visuals are generated (mock, fal.ai, or Higgsfield);
56
+ - validates optional `FAL_KEY` and Higgsfield credentials and writes them to `.env` (gitignored);
57
+ - configures the Suno MCP server in `.mcp.json`;
58
+ - creates the first project and runs `npm install` plus `uv sync`.
59
+
60
+ Fully non-interactive:
61
+
62
+ ```bash
63
+ npx coreslopper@latest init my-video --yes --song suno --visual higgsfield --budget 50
64
+ ```
65
+
66
+ Set `FAL_KEY` or `HIGGSFIELD_API_KEY` in the environment (or pass `--fal-key` / `--higgsfield-key`) when you choose a real provider. Prefer environment variables; a key passed on the command line can end up in your shell history. Use `--image-provider` and `--video-provider` when the two should differ, for example Higgsfield for keyframes and fal.ai for video.
67
+
68
+ Then open Claude Code in the workspace:
69
+
70
+ ```bash
71
+ cd my-video
72
+ claude
73
+ ```
74
+
75
+ Run `/mcp` once to authenticate Suno, then `/video-next my-video`. The generated workspace contains the full workflow runtime, so every `npm run cli -- ...` command in this README works inside it. Running `npx coreslopper` without arguments inside a workspace prints the available commands. See [docs/onboarding.md](docs/onboarding.md) for all flags and troubleshooting.
76
+
77
+ ## Quickstarts
78
+
79
+ ### 1. Existing Suno.ai song with lyrics and a vocal stem
80
+
81
+ Use this when the song is already finished. The pipeline imports the audio, uses your lyrics as the authoritative text, and takes word timings from the vocal stem.
82
+
83
+ ```bash
84
+ # 1. Create a workspace and project. `import` means "use my song".
85
+ # Use --visual fal instead, or mix providers, if you prefer fal.ai.
86
+ export HIGGSFIELD_API_KEY="key_id:key_secret" # or add it to .env after init
87
+ npx coreslopper@latest init neon-rain --yes --song import --visual higgsfield --budget 50
88
+ cd neon-rain
89
+ ```
90
+
91
+ Mix the providers in `projects/neon-rain/project.yaml` when needed. Lip-sync currently uses the fal adapter, so also set `FAL_KEY` when `video` or `lipsync` is `fal`:
92
+
93
+ ```yaml
94
+ providers:
95
+ image: higgsfield
96
+ video: fal
97
+ lipsync: fal
98
+ song: import
99
+ ```
100
+
101
+ ```bash
102
+ # 2. Put the finished song and the vocal stem into the project.
103
+ mkdir -p projects/neon-rain/audio/stems
104
+ cp ~/Downloads/neon-rain.wav projects/neon-rain/audio/song.wav
105
+ cp ~/Downloads/neon-rain-vocals.wav projects/neon-rain/audio/stems/vocals.wav
106
+
107
+ # If the download is MP3, M4A, or another format, normalize it to WAV instead:
108
+ # npm run cli -- normalize-audio projects/neon-rain --input ~/Downloads/neon-rain.mp3
109
+ ```
110
+
111
+ ```bash
112
+ # 3. Write the lyrics with section tags. Keep the words exactly as sung.
113
+ # The analyzer takes text and order from this file; WhisperX supplies timings.
114
+ $EDITOR projects/neon-rain/lyrics.txt
115
+ ```
116
+
117
+ ```text
118
+ [Verse 1]
119
+ Line one
120
+ Line two
121
+
122
+ [Chorus]
123
+ Line three
124
+ Line four
125
+ ```
126
+
127
+ ```bash
128
+ # 4. Record where the song came from and whether the stem is present.
129
+ $EDITOR projects/neon-rain/song.json
130
+ ```
131
+
132
+ ```json
133
+ {
134
+ "source": "suno",
135
+ "title": "Neon Rain",
136
+ "artist": "your Suno artist",
137
+ "path": "audio/song.wav",
138
+ "duration": 198,
139
+ "bpm": 118,
140
+ "language": "en",
141
+ "hasVocals": true,
142
+ "stems": true,
143
+ "license": "Suno output; check your plan and local law",
144
+ "notes": "Imported from Suno with a vocal stem."
145
+ }
146
+ ```
147
+
148
+ ```bash
149
+ # 5. Mark the lyrics and song stages as already supplied, and write the idea
150
+ # into the brief so the treatment has an anchor.
151
+ $EDITOR projects/neon-rain/brief.md
152
+ npm run cli -- stage-done projects/neon-rain brief --outputs brief.md
153
+ npm run cli -- stage-skip projects/neon-rain lyrics --reason "Imported lyrics"
154
+ npm run cli -- stage-skip projects/neon-rain song --reason "Imported Suno song and vocal stem"
155
+
156
+ # 6. Run the remaining stages in Claude Code.
157
+ claude
158
+ # /video-next neon-rain
159
+ ```
160
+
161
+ The `audio-analysis` stage uses `audio/stems/vocals.wav` when it exists. If you want to run the analysis by hand first:
162
+
163
+ ```bash
164
+ cd scripts/audio-analysis
165
+ uv sync --extra asr --extra stems
166
+ uv run python analyze.py --project ../../projects/neon-rain \
167
+ --vocals ../../projects/neon-rain/audio/stems/vocals.wav --device auto
168
+ # use --device cuda on an NVIDIA machine
169
+ cd ../..
170
+ npm run cli -- stage-done projects/neon-rain audio-analysis --outputs timing.json,timing.report.md
171
+ npm run cli -- validate projects/neon-rain
172
+ ```
173
+
174
+ ### 2. No song yet
175
+
176
+ Use this when the pipeline should write the lyrics, generate the song with Suno, analyze it, and build the video.
177
+
178
+ ```bash
179
+ # 1. Create the workspace with Suno and a visual provider.
180
+ export HIGGSFIELD_API_KEY="key_id:key_secret" # or add it to .env after init
181
+ npx coreslopper@latest init my-video --yes --song suno --visual higgsfield --budget 50
182
+ cd my-video
183
+ ```
184
+
185
+ Use `--visual fal` and `FAL_KEY` for fal.ai. For a mixed setup, start a fresh workspace:
186
+
187
+ ```bash
188
+ npx coreslopper@latest init another-video \
189
+ --yes --song suno \
190
+ --image-provider higgsfield \
191
+ --video-provider fal \
192
+ --budget 50
193
+ ```
194
+
195
+ ```bash
196
+ # 2. Write the one-paragraph idea into the brief, then mark the stage done.
197
+ # Without this, the autonomous brief stage drafts it from the project title.
198
+ $EDITOR projects/my-video/brief.md
199
+ npm run cli -- stage-done projects/my-video brief --outputs brief.md
200
+
201
+ # 3. Open Claude Code, authenticate Suno once, and run the pipeline.
202
+ claude
203
+ # /mcp
204
+ # /video-next my-video
205
+ ```
206
+
207
+ The pipeline writes the treatment and lyrics, generates two Suno variants by default, scores them, analyzes the selected song, builds the style and character bibles, writes and validates the storyboard, produces the assets, and renders the preview and masters. It stops only for budget, missing credentials, or an unrecoverable validation failure.
208
+
209
+ Resume or inspect at any time:
210
+
211
+ ```bash
212
+ npm run cli -- status my-video
213
+ npm run cli -- estimate --role image.keyframe --units 8 --project projects/my-video
214
+ ```
215
+
216
+ In Claude Code, resume one stage with `/video-next my-video --stage storyboard`.
217
+
218
+ ## Work in this repository
219
+
220
+ ```bash
221
+ git clone https://github.com/hemi1986/coreslopper.git
222
+ cd coreslopper
223
+ ./scripts/setup.sh # or: npm install && (cd scripts/audio-analysis && uv sync --group dev)
224
+
225
+ # Generate the license-safe demo audio and validate the project
226
+ npm run demo:audio
227
+ npm run cli -- validate projects/demo
228
+
229
+ # Contact sheet and 16:9 render
230
+ npm run cli -- contact-sheet projects/demo
231
+ npm run cli -- render projects/demo --format 16x9
232
+ ```
233
+
234
+ The render is written to `projects/demo/renders/16x9.mp4`.
235
+
236
+ Or run the full demo (audio, validation, contact sheet, all three formats, and QA):
237
+
238
+ ```bash
239
+ npm run demo
240
+ ```
241
+
242
+ Check your environment at any time:
243
+
244
+ ```bash
245
+ npm run cli -- doctor
246
+ ```
247
+
248
+ ## Create a project
249
+
250
+ ```bash
251
+ npm run cli -- new my-video --budget 50 --visual higgsfield
252
+ # or mix providers:
253
+ npm run cli -- new my-video --budget 50 --image-provider higgsfield --video-provider fal
254
+ ```
255
+
256
+ Budget is mandatory. The command creates `projects/my-video/` from `templates/project/` and initializes `state.json`. In an npx workspace, `npx coreslopper new my-video --budget 50` does the same. Then open Claude Code in the workspace and run:
257
+
258
+ ```text
259
+ /video-next my-video
260
+ ```
261
+
262
+ Add `--stage <stage>` to run one stage, or `--dry-run` to inspect the next stage and budget.
263
+
264
+ ## Pipeline
265
+
266
+ | # | Stage | What happens | Output |
267
+ |---|---|---|---|
268
+ | 1 | `brief` | Grill the idea; write topic, source, audience, length, formats, style, constraints. | `brief.md` |
269
+ | 2 | `treatment` | Core idea, visual world, dramaturgy, shot mix. | `treatment.md` |
270
+ | 3 | `lyrics` | Original lyrics with section tags plus a Suno style prompt. | `lyrics.txt`, `suno-style.txt` |
271
+ | 4 | `song` | Suno MCP (two variants, scored) or import. Stems when available. | `song.json`, `audio/song.wav` |
272
+ | 5 | `audio-analysis` | Demucs + WhisperX + librosa; section table with confidence flags. | `timing.json`, `timing.report.md` |
273
+ | 6 | `style-character` | Palette, typography, look, motion rules, character sheets. | `style.yaml`, `characters.yaml` |
274
+ | 7 | `storyboard` | Anchored shots, coverage validation, contact sheet. | `storyboard.yaml`, `storyboard.md` |
275
+ | 8 | `assets` | Motion components, AI keyframes/clips, footage proxies. | `assets/`, `manifest.jsonl` |
276
+ | 9 | `assembly` | Remotion timeline, preview, master, derivatives. | `renders/` |
277
+ | 10 | `qa-export` | Sync, readability, determinism, budget reconciliation, all formats. | `review/report.md`, renders |
278
+
279
+ No creative approval gate blocks the run. A stage stops only on budget, missing credentials, or an unrecoverable validation failure.
280
+
281
+ ## Audio analysis
282
+
283
+ ```bash
284
+ cd scripts/audio-analysis
285
+ uv sync # core: librosa, numpy, soundfile, jsonschema
286
+ uv sync --extra asr --extra stems # WhisperX + Demucs
287
+ uv run python analyze.py --project ../../projects/my-video --device cuda
288
+ ```
289
+
290
+ The script writes `timing.json` and `timing.report.md`. It never writes `timing.overrides.json`; that file is yours and always wins at load time.
291
+
292
+ For a fixture or a motion-only demo, `--skip-asr` distributes lyric timings without speech recognition. Real lyric videos should use WhisperX.
293
+
294
+ ## AI generation
295
+
296
+ The provider interface is intentionally small:
297
+
298
+ ```ts
299
+ generateImage(input): Promise<ImageResult>;
300
+ editImage(input): Promise<ImageResult>;
301
+ imageToVideo(input): Promise<VideoResult>;
302
+ lipSync(input): Promise<VideoResult>;
303
+ ```
304
+
305
+ fal.ai implements all four operations; Higgsfield implements image and video generation but not lip-sync. The mock adapter returns deterministic SVG placeholders. Model IDs, provider-specific slugs, defaults, and prices live in [`config/models.yaml`](config/models.yaml), not in code. A change of model is a configuration change.
306
+
307
+ Choose providers per project in `project.yaml`:
308
+
309
+ ```yaml
310
+ providers:
311
+ image: higgsfield
312
+ video: fal
313
+ lipsync: fal # optional; defaults to providers.video
314
+ song: import
315
+ ```
316
+
317
+ `image` covers `generateImage` and `editImage`. `video` covers `imageToVideo`. `lipsync` is optional so a project can use Higgsfield for stills and motion while keeping fal.ai for lip-sync. `song` is `suno`, `import`, or `mock`.
318
+
319
+ A generation batch is a JSON array:
320
+
321
+ ```json
322
+ [
323
+ {
324
+ "operation": "editImage",
325
+ "role": "image.keyframe",
326
+ "prompt": "Nova stands in neon rain, cinematic close-up",
327
+ "references": ["characters/nova/sheet.png"],
328
+ "seed": 1234
329
+ },
330
+ {
331
+ "operation": "imageToVideo",
332
+ "role": "video.image_to_video.quality",
333
+ "prompt": "slow push in, rain streaks, neon reflections",
334
+ "references": ["assets/keyframes/shot-004.png"],
335
+ "durationSeconds": 5,
336
+ "seed": 1234
337
+ }
338
+ ]
339
+ ```
340
+
341
+ Estimate and run:
342
+
343
+ ```bash
344
+ npm run cli -- estimate --role image.keyframe --units 8 --project projects/my-video
345
+ npm run cli -- generate projects/my-video --batch assets/batches/scene-01.json
346
+ ```
347
+
348
+ The estimate resolves the role for the project's selected provider. Add `--provider higgsfield` (or `fal`) to compare providers without changing the project.
349
+
350
+ Every call is cached by a SHA-256 hash of the canonical request and recorded in `assets/manifest.jsonl` with provider, model, prompt, seed, references, cost, and cache status. A cache hit costs nothing.
351
+
352
+ ### Credentials
353
+
354
+ ```bash
355
+ # fal.ai
356
+ FAL_KEY=...
357
+
358
+ # Higgsfield: either the complete credential from open.higgsfield.ai ...
359
+ HIGGSFIELD_API_KEY=key_id:key_secret
360
+
361
+ # ... or the split values. HF_API_KEY, HF_KEY, and HF_CREDENTIALS are also accepted.
362
+ HF_API_KEY_ID=...
363
+ HF_API_KEY_SECRET=...
364
+ ```
365
+
366
+ Higgsfield submissions are asynchronous: the adapter uploads local references through the presigned-upload endpoint, polls the request, and downloads the result before it is cached. Its model schemas are strict, so provider-specific params stay in `config/models.yaml`. Three adapter-only options are stripped before the request is sent: `send_seed: true` for models that accept a seed, `send_negative_prompt: true` for models that accept a negative prompt, and `reference_field: <field>` when an edit model uses a name other than `image_urls` (for example `image_reference_url`).
367
+
368
+ Higgsfield does not implement lip-sync in this workflow yet. Set `providers.lipsync: fal` when a storyboard uses lip-sync shots.
369
+
370
+ Without a key, use the mock provider:
371
+
372
+ ```yaml
373
+ providers:
374
+ image: mock
375
+ video: mock
376
+ lipsync: mock
377
+ song: import
378
+ ```
379
+
380
+ ### Provider-specific model roles
381
+
382
+ A role has a base provider and optional alternatives:
383
+
384
+ ```yaml
385
+ roles:
386
+ image.keyframe:
387
+ provider: fal
388
+ model: fal-ai/flux-2-pro
389
+ pricing: { unit: image, usd: 0.05 }
390
+ alternatives:
391
+ higgsfield:
392
+ model: higgsfield-ai/soul/v2/standard
393
+ params: { resolution: 1080p, aspect_ratio: 16:9 }
394
+ pricing: { unit: image, usd: 0.006 }
395
+ ```
396
+
397
+ A project can also override a role explicitly with `models.image.keyframe.model`, `.params`, and `.pricing` in `project.yaml`. The estimator refuses to run a paid batch when the selected provider has no model or price.
398
+
399
+ ## Claude Code layer
400
+
401
+ - **`/new-video`** creates a project from the template. Budget is required.
402
+ - **`/video-next`** runs all open stages, one fresh `stage-runner` subagent per stage. It stops only on hard stops.
403
+ - **`/video-status`** is read-only.
404
+ - **`art-director`** judges stills, keyframes, character sheets, and clip frames against the style bible.
405
+ - **`qa-checker`** runs schema, asset, sync, readability, determinism, and budget checks.
406
+ - **Hooks** validate `storyboard.yaml`, `timing.json`, `timing.overrides.json`, `project.yaml`, `state.json`, `style.yaml`, and `characters.yaml` after edits.
407
+ - **MCP** includes the official Suno server at `https://api.suno.com/mcp`. fal.ai and Higgsfield are used through the adapter, not MCP.
408
+
409
+ See [CLAUDE.md](CLAUDE.md) for the Claude Code specifics.
410
+
411
+ ## Repository layout
412
+
413
+ ```text
414
+ .claude/ skills, commands, agents, hooks, settings
415
+ config/ model roles and defaults
416
+ docs/ plan, glossary, architecture, pipeline, ADRs, third-party notices
417
+ packages/
418
+ core/ Zod schemas, state, timing, hash, cost, project loader
419
+ providers/ fal.ai + Higgsfield + mock, cache, manifest
420
+ cli/ validate, status, new, render, contact-sheet, stage commands
421
+ remotion/ MusicVideo composition, motion components, effects, 3D
422
+ schema/ generated JSON Schema
423
+ scripts/ audio analysis (Python/uv), fixtures
424
+ templates/project project template
425
+ brand/example/ neutral synthwave/cyberpunk kit, OFL fonts
426
+ projects/ video projects; media is gitignored
427
+ ```
428
+
429
+ ## State, caching, and budget
430
+
431
+ - `project.yaml` requires `budget_usd`. `batch_cap_usd` defaults to 30 % of the remaining budget.
432
+ - `state.json` records every stage, attempts, outputs, errors, and spend.
433
+ - `timing.overrides.json` is user-owned and never overwritten.
434
+ - Generated assets live in `.cache/generations/<hash>/` and are recorded in `assets/manifest.jsonl`.
435
+ - `timing.effective.json` is generated by validation; it is derived and gitignored.
436
+
437
+ ## Formats
438
+
439
+ - Master: 16:9, 1920×1080, 30 fps, H.264.
440
+ - Derivatives: 9:16 1080×1920 and 1:1 1080×1080 by default; 4:5 optional.
441
+ - 4K/60 is not a default. Add it as a project-specific format when needed.
442
+
443
+ Derivatives are separate compositions with safe-area-aware layouts, not naive crops.
444
+
445
+ ## Open source and licensing
446
+
447
+ - The workflow is **MIT** licensed. See [LICENSE](LICENSE).
448
+ - **Remotion is source-available, not OSI open source.** It is free for individuals, organizations with up to 3 employees, and non-profits; larger for-profit organizations need a Remotion license. See [docs/THIRD_PARTY.md](docs/THIRD_PARTY.md).
449
+ - The vendored Matt Pocock skills are MIT, pinned and attributed.
450
+ - The example brand uses OFL fonts (Orbitron, Space Grotesk, JetBrains Mono) and an original mark.
451
+ - The demo audio is generated by a script. No Suno/fal.ai/Higgsfield output or copyrighted music is committed.
452
+
453
+ **Do not commit** API keys, `.env`, generated audio/video, or copyrighted media. The repository is open source; keep it license-safe.
454
+
455
+ ## Documentation
456
+
457
+ - [docs/PLAN.md](docs/PLAN.md) — agreed architecture and delivery plan.
458
+ - [docs/onboarding.md](docs/onboarding.md) — npx setup, flags, and troubleshooting.
459
+ - [docs/architecture.md](docs/architecture.md) — data flow, state, cache, providers, rendering.
460
+ - [docs/pipeline.md](docs/pipeline.md) — stage-by-stage runbook.
461
+ - [docs/glossary.md](docs/glossary.md) — ubiquitous language.
462
+ - [docs/adr/](docs/adr/) — architecture decision records.
463
+ - [docs/THIRD_PARTY.md](docs/THIRD_PARTY.md) — licenses and attribution.
464
+ - [docs/mcp.md](docs/mcp.md) — Suno MCP, optional fal.ai exploration, and provider adapter notes.
465
+
466
+ ## Contributing
467
+
468
+ Read [CONTRIBUTING.md](CONTRIBUTING.md), run `npm test`, `npm run typecheck`, and `npm run python:test`, and keep the demo green. For a new skill, use the vendored `writing-for-agents` guidance and keep frontmatter to `name`, `description`, and `license`.
469
+
470
+ ## Security
471
+
472
+ Report vulnerabilities privately as described in [SECURITY.md](SECURITY.md). Never include a real API key in an issue, a screenshot, or a log.