cozyclay 1.2.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +85 -0
- package/README.md +31 -0
- package/THIRD_PARTY_NOTICES.md +37 -1
- package/bin/cozyclay.mjs +51 -2
- package/dist/ai-camera-control/index.html +406 -0
- package/dist/app/index.html +5 -5
- package/dist/assets/app-B3U5aut1.js +4811 -0
- package/dist/assets/app-BrRF0wso.css +1 -0
- package/dist/assets/vision_bundle-jFkh-fIS.js +41 -0
- package/dist/fonts/InstrumentSerif-OFL.txt +93 -0
- package/dist/fonts/Inter-OFL.txt +92 -0
- package/dist/fonts/README.md +15 -0
- package/dist/index.html +60 -16
- package/dist/sitemap.xml +7 -1
- package/mcp/LIVE-PROTOCOL.md +63 -0
- package/mcp/README.md +142 -0
- package/mcp/ardy-prompts.mjs +170 -0
- package/mcp/live-hub.mjs +105 -0
- package/mcp/package.json +24 -0
- package/mcp/server.mjs +1394 -0
- package/package.json +122 -90
- package/src/App.jsx +2957 -514
- package/src/ardy/cskel27.js +7 -2
- package/src/ardy/ik.js +25 -15
- package/src/ardy/npz.js +64 -3
- package/src/ardy/playback.js +31 -1
- package/src/ardy/prompt-clips.js +7 -2
- package/src/ardy/retime.js +211 -0
- package/src/ardy/timeline-coordinates.js +13 -0
- package/src/ardy/timeline.jsx +124 -5
- package/src/ardy/to-cskel27.js +34 -12
- package/src/ardy/trim.js +33 -0
- package/src/asset-pane.jsx +36 -0
- package/src/dualview.jsx +14 -8
- package/src/hierarchy-model.js +95 -13
- package/src/hierarchy-panel.jsx +139 -6
- package/src/live-control.js +122 -0
- package/src/matte-editor.js +543 -0
- package/src/matte.js +503 -0
- package/src/multimodel-ingest.js +344 -0
- package/src/object-gizmo.jsx +43 -15
- package/src/planview.jsx +43 -32
- package/src/pose-extract/detector.js +75 -0
- package/src/pose-extract/index.js +3 -0
- package/src/pose-extract/take.js +87 -0
- package/src/pose-extract/video-frames.js +91 -0
- package/src/pose-thumbs.js +152 -0
- package/src/posestudio.jsx +361 -11
- package/src/project-browser.jsx +135 -0
- package/src/project.js +289 -0
- package/src/props.jsx +69 -3
- package/src/room.jsx +14 -35
- package/src/scene-asset-cache.js +125 -0
- package/src/scene-assets.js +288 -0
- package/src/scene-objects.js +245 -7
- package/src/scenes.js +207 -26
- package/src/shot-authoring.js +55 -13
- package/src/styles.css +1296 -129
- package/tools/ardy/BRIDGE.md +3 -2
- package/tools/ardy/README.md +9 -5
- package/tools/ardy/bridge.mjs +57 -1
- package/tools/ardy/bvh-cskel27.mjs +1209 -0
- package/tools/ardy/cclay_constrained_generate.py +123 -11
- package/tools/ardy/cclay_sequence_generate.py +49 -0
- package/tools/ardy/extract.mjs +367 -0
- package/tools/ardy/footage.mjs +462 -0
- package/tools/ardy/npz.mjs +74 -9
- package/tools/ardy/run-on-box.sh +25 -0
- package/tools/ardy/run-sequence-on-box.sh +15 -0
- package/tools/ardy/runners/remote.mjs +8 -2
- package/dist/assets/app-Cgpk2hwX.js +0 -4803
- package/dist/assets/app-DgZvaAE1.css +0 -1
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 1.3.0
|
|
4
|
+
|
|
5
|
+
The release where the studio stops being something only a person can drive.
|
|
6
|
+
|
|
7
|
+
### Direct it with an AI
|
|
8
|
+
|
|
9
|
+
CozyClay now ships an [MCP](https://modelcontextprotocol.io) server. Point Claude, Cursor, or any
|
|
10
|
+
MCP client at it and ask for a shot in plain language — it places the cast, frames "a low wide
|
|
11
|
+
profile", generates multi-phase motion, and **the viewport moves while you watch**.
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{ "mcpServers": { "cozyclay": { "command": "npx", "args": ["-y", "cozyclay", "mcp"] } } }
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
It computes nothing of its own. Every answer comes from the modules the studio already renders
|
|
18
|
+
with — `shot.js` for film vocabulary, `scenes.js` for the stage, `camera-move.js` for moves — so
|
|
19
|
+
the server and the UI cannot disagree about what a 35 mm medium shot is.
|
|
20
|
+
|
|
21
|
+
- **Live or headless.** With a studio tab open, tool calls drive the real viewport over a local
|
|
22
|
+
socket. With no tab, the same tools run headless: block scenes, derive framing, render prompts,
|
|
23
|
+
and write `.cclayproject` files the studio opens.
|
|
24
|
+
- **Motion from plain beats.** `generate_motion` takes ordinary sentences, rewrites them into the
|
|
25
|
+
shape ARDY was trained to read, splits compound beats, drops language a body cannot perform, and
|
|
26
|
+
lands each phase as a Prompt Block on the timeline.
|
|
27
|
+
- **21 tools**, all documented in [`mcp/README.md`](mcp/README.md).
|
|
28
|
+
|
|
29
|
+
### Cut a photograph out and stand it up
|
|
30
|
+
|
|
31
|
+
Drop an image on Props — or straight onto the shot — and it becomes a card you can place, turn and
|
|
32
|
+
block against. The background editor lives in the card's own inspector: paint or erase the
|
|
33
|
+
selection, zoom and pan with the wheel, and the picture sits on a stage rather than being framed
|
|
34
|
+
by one. Assets are stored once by content hash, so the same picture imported twice costs one copy.
|
|
35
|
+
|
|
36
|
+
### An open stage
|
|
37
|
+
|
|
38
|
+
The two-walled room corner is gone. The set is a 500 m open deck — far enough that no ordinary
|
|
39
|
+
blocking meets its edge — with a two-tier grid (quiet 1 m cells, assertive 10 m lines) and a
|
|
40
|
+
distance falloff that dissolves the floor into the background instead of ending at a horizon.
|
|
41
|
+
Movement clamps moved from ±11 m to ±240 m, so a run or a chase finally has room.
|
|
42
|
+
|
|
43
|
+
### Group props and move them as one
|
|
44
|
+
|
|
45
|
+
Scene objects now carry a parent. Build a rocket from ten primitives, group them, and the whole
|
|
46
|
+
thing drags as one body — through the gizmo, the inspector and the MCP tools alike. Deleting a
|
|
47
|
+
parent promotes its children rather than taking them with it.
|
|
48
|
+
|
|
49
|
+
### Everything else
|
|
50
|
+
|
|
51
|
+
- **Gizmo** — the cast rides the same gizmo as scene objects, with rotate and scale; an XZ pad for
|
|
52
|
+
free ground drags; pick targets balanced so the pad no longer swallows every press.
|
|
53
|
+
- **Mocap** — a video capture panel, takes assigned to cast members, and take trimming.
|
|
54
|
+
- **Timeline** — authoring runs on a 24 fps production clock while ARDY is still spoken to at
|
|
55
|
+
20 fps, and exports record at true motion speed.
|
|
56
|
+
- **Multi-character** — an unbounded cast with per-character motion layers.
|
|
57
|
+
|
|
58
|
+
### Fixes
|
|
59
|
+
|
|
60
|
+
- Props reported `1x1x1` regardless of how they were built: scale never crossed the live protocol,
|
|
61
|
+
and the server reset whatever did arrive. Sizes are now real, and per-axis scale is exposed.
|
|
62
|
+
- A 6 m ceiling survived the walls it belonged to, silently decapitating anything tall.
|
|
63
|
+
- Generating four Prompt Blocks produced one. IK keys from a discarded take made a fresh run look
|
|
64
|
+
like a local edit, and that path ships no schedule. Replacing a take now clears the corrections
|
|
65
|
+
authored against it.
|
|
66
|
+
- A stale service worker served yesterday's bundle to the dev server, so edits appeared to do
|
|
67
|
+
nothing. Dev now serves a self-destructing worker.
|
|
68
|
+
- Prompt blocks are capped at 4 s everywhere, not just in the UI — longer beats chain instead of
|
|
69
|
+
being authored into something the timeline would refuse.
|
|
70
|
+
|
|
71
|
+
### Packaging
|
|
72
|
+
|
|
73
|
+
- `mcp/` ships with the package, so `npx cozyclay mcp` works without a clone.
|
|
74
|
+
- `exports` makes `cozyclay/src/*` a public contract instead of an accident.
|
|
75
|
+
- `three` is declared as an optional peer: the studio bundles its own copy, but anything importing
|
|
76
|
+
`cozyclay/src/*` needs it, and now npm says so.
|
|
77
|
+
|
|
78
|
+
## 1.2.0
|
|
79
|
+
|
|
80
|
+
- `npx cozyclay` opens the studio again.
|
|
81
|
+
- Stopped shipping 26 MB of scratch files.
|
|
82
|
+
|
|
83
|
+
## 0.1.0
|
|
84
|
+
|
|
85
|
+
First public release.
|
package/README.md
CHANGED
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
<a href="https://cozyclay.org/">Demo reel</a> ·
|
|
18
18
|
<a href="#quick-start">Quick start</a> ·
|
|
19
19
|
<a href="#what-you-can-do">Features</a> ·
|
|
20
|
+
<a href="#ai-control-mcp">AI control</a> ·
|
|
20
21
|
<a href="#controls">Controls</a> ·
|
|
21
22
|
<a href="https://github.com/HaD0Yun/CozyClay/issues">Issues</a>
|
|
22
23
|
</p>
|
|
@@ -43,6 +44,7 @@ https://github.com/user-attachments/assets/1d0113e5-6922-443d-affc-1bdabc666247
|
|
|
43
44
|
| **Fly the camera** | Right-drag flies (WASD walks, Q/E cranes), middle-drag pans, Alt+drag orbits the selection, click selects, `F` frames — the muscle memory you already have from a 3D editor. |
|
|
44
45
|
| **Undo anything** | Every scene mutation goes through one history store: a drag, a scrub, an inspector edit is exactly one undo entry. `Esc` cancels an in-flight drag and restores the pre-drag transform. |
|
|
45
46
|
| **Generate motion** | Pose characters and export poses, sequence multi-phase motion as Prompt Blocks on a resizable timeline, send them to ARDY, then play the result back with sparse IK correction where the generated motion needs fixing. |
|
|
47
|
+
| **Direct it with an AI** | Connect Claude — or any MCP client — and ask for a shot in plain language. It places the cast, frames “a low wide profile”, generates multi-phase motion, and the viewport moves in front of you. See [AI control](#ai-control-mcp). |
|
|
46
48
|
|
|
47
49
|
## Requirements
|
|
48
50
|
|
|
@@ -69,6 +71,33 @@ CCLAY_ARDY_HOST=user@your-gpu-box npx cozyclay
|
|
|
69
71
|
|
|
70
72
|
Everything else — staging through camera work and playback — runs without it.
|
|
71
73
|
|
|
74
|
+
## AI control (MCP)
|
|
75
|
+
|
|
76
|
+
The studio ships an [MCP](https://modelcontextprotocol.io) server, so an AI assistant can drive it — the same scene, the same viewport, live:
|
|
77
|
+
|
|
78
|
+
> “Put a detective and a courier in an alley, give me a low wide profile shot,
|
|
79
|
+
> then make her stand up from the chair, sprint, and trip.”
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
{
|
|
83
|
+
"mcpServers": {
|
|
84
|
+
"cozyclay": {
|
|
85
|
+
"command": "npx",
|
|
86
|
+
"args": ["-y", "cozyclay", "mcp"]
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Drop that into `claude_desktop_config.json` (or any MCP client config) and restart the client. The
|
|
93
|
+
first run prints one `npm install` line: the MCP SDK is a 95-package tree, and opening the studio
|
|
94
|
+
should never wait on it, so those dependencies are fetched only when you actually want the server.
|
|
95
|
+
|
|
96
|
+
- **Editor open?** Tool calls move the visible viewport — camera, cast, set, generated motion, prompt blocks on the timeline.
|
|
97
|
+
- **No editor?** The same tools run headless: block scenes, derive film vocabulary (“wide shot · right profile · knee level · 24mm”), render AI video prompts, and write `.cclayproject` files the studio opens.
|
|
98
|
+
|
|
99
|
+
Tools, transports and the live-control protocol are documented in [`mcp/README.md`](mcp/README.md).
|
|
100
|
+
|
|
72
101
|
### From a clone
|
|
73
102
|
|
|
74
103
|
```bash
|
|
@@ -123,6 +152,8 @@ See [`tools/ardy/README.md`](tools/ardy/README.md) for details. This workflow is
|
|
|
123
152
|
| `npm run test:theme` / `test:appearance` / `test:layout` | UI theme, appearance, layout |
|
|
124
153
|
| `npm run test:lifecycle` | Dev-server process lifecycle |
|
|
125
154
|
| `npm run test:ardy` | ARDY conversion, playback, and IK pipeline |
|
|
155
|
+
| `cd mcp && npm run verify` | MCP server over real stdio — all 420 framing combinations |
|
|
156
|
+
| `cd mcp && npm run verify:live` | Live-control protocol against a fake editor |
|
|
126
157
|
| `npm run build` | Production build |
|
|
127
158
|
|
|
128
159
|
Ad-hoc browser QA, while a dev server is available:
|
package/THIRD_PARTY_NOTICES.md
CHANGED
|
@@ -55,6 +55,42 @@ bundled.
|
|
|
55
55
|
- Source: https://github.com/McGill-NLP/llm2vec
|
|
56
56
|
- License text: https://github.com/McGill-NLP/llm2vec/blob/main/LICENSE
|
|
57
57
|
|
|
58
|
+
## Fonts
|
|
59
|
+
|
|
60
|
+
CozyClay bundles subsets of two typefaces. Both are licensed under the SIL Open
|
|
61
|
+
Font License 1.1, which allows them to be redistributed with software as long as
|
|
62
|
+
the copyright notice and the licence travel with the files. The licence texts are
|
|
63
|
+
in `public/fonts/` next to the fonts themselves.
|
|
64
|
+
|
|
65
|
+
### Inter
|
|
66
|
+
|
|
67
|
+
- Copyright (c) 2016 The Inter Project Authors
|
|
68
|
+
- License: SIL Open Font License 1.1
|
|
69
|
+
- Source: https://github.com/rsms/inter
|
|
70
|
+
- License text: `public/fonts/Inter-OFL.txt`
|
|
71
|
+
|
|
72
|
+
### Instrument Serif
|
|
73
|
+
|
|
74
|
+
- Copyright 2022 The Instrument Serif Project Authors
|
|
75
|
+
- License: SIL Open Font License 1.1
|
|
76
|
+
- Source: https://github.com/Instrument/instrument-serif
|
|
77
|
+
- License text: `public/fonts/InstrumentSerif-OFL.txt`
|
|
78
|
+
|
|
79
|
+
## Character models
|
|
80
|
+
|
|
81
|
+
The rigs in `public/models/` are Mixamo characters from Adobe. Adobe permits
|
|
82
|
+
their use in projects but not the distribution of the raw character files, which
|
|
83
|
+
is what shipping them in this repository, the npm package and the hosted site
|
|
84
|
+
amounts to. They are also outside the scope of this repository's GPL-3.0 grant:
|
|
85
|
+
nothing here relicenses them, and a fork does not acquire the right to
|
|
86
|
+
redistribute them.
|
|
87
|
+
|
|
88
|
+
They are being replaced with a CC0 rig. Until that lands, treat these two files
|
|
89
|
+
as third-party content that this licence does not cover.
|
|
90
|
+
|
|
91
|
+
- `x-bot-tpose.fbx`, `y-bot-tpose.fbx` — Adobe Mixamo
|
|
92
|
+
- Terms: https://www.adobe.com/legal/terms.html
|
|
93
|
+
|
|
58
94
|
## CozyClay license scope
|
|
59
95
|
|
|
60
|
-
The CozyClay source code in this repository is licensed under GPL-3.0-or-later. That license does not replace or relicense Three.js, ARDY, ARDY model checkpoints, or any other third-party component.
|
|
96
|
+
The CozyClay source code in this repository is licensed under GPL-3.0-or-later. That license does not replace or relicense Three.js, ARDY, ARDY model checkpoints, the bundled fonts, the character rigs, or any other third-party component.
|
package/bin/cozyclay.mjs
CHANGED
|
@@ -79,6 +79,7 @@ const STAR_AFTER_MS = Number(process.env.COZYCLAY_STAR_AFTER_MS ?? 60_000);
|
|
|
79
79
|
const HELP = `cozyclay - browser-based 3D staging studio
|
|
80
80
|
|
|
81
81
|
npx cozyclay start the studio and open it
|
|
82
|
+
npx cozyclay mcp run the MCP server (for Claude, Cursor, any MCP client)
|
|
82
83
|
npx cozyclay --port 5200 serve on another port
|
|
83
84
|
npx cozyclay --no-ardy skip the optional motion-generation sidecar
|
|
84
85
|
npx cozyclay --no-open do not open a browser
|
|
@@ -189,7 +190,50 @@ function openBrowser(url) {
|
|
|
189
190
|
child.unref();
|
|
190
191
|
}
|
|
191
192
|
|
|
192
|
-
|
|
193
|
+
/**
|
|
194
|
+
* `cozyclay mcp` hands over to the MCP server.
|
|
195
|
+
*
|
|
196
|
+
* Its dependencies are NOT the launcher's: the studio path stays dependency
|
|
197
|
+
* free on purpose, and the MCP SDK is a 95-package tree nobody who only wants
|
|
198
|
+
* a viewport should download. So the server ships with the package, its deps
|
|
199
|
+
* are installed on demand, and a missing one produces an exact instruction
|
|
200
|
+
* rather than a stack trace.
|
|
201
|
+
*/
|
|
202
|
+
async function runMcp(rest) {
|
|
203
|
+
const server = join(PKG_ROOT, "mcp", "server.mjs");
|
|
204
|
+
if (!existsSync(server)) {
|
|
205
|
+
console.error("cozyclay: this build does not include the MCP server.");
|
|
206
|
+
process.exit(1);
|
|
207
|
+
}
|
|
208
|
+
// The deps must land in the PACKAGE root, not in mcp/: server.mjs imports
|
|
209
|
+
// ../src/*.js, and those files resolve `three` from cozyclay/ itself.
|
|
210
|
+
try {
|
|
211
|
+
await import("@modelcontextprotocol/sdk/server/mcp.js");
|
|
212
|
+
await import("three");
|
|
213
|
+
} catch {
|
|
214
|
+
console.error(
|
|
215
|
+
[
|
|
216
|
+
"cozyclay: the MCP server needs four packages the studio does not.",
|
|
217
|
+
"",
|
|
218
|
+
` npm install --prefix ${JSON.stringify(PKG_ROOT)} \\`,
|
|
219
|
+
" @modelcontextprotocol/sdk ws zod three",
|
|
220
|
+
"",
|
|
221
|
+
"Then run `npx cozyclay mcp` again. They are kept out of the default",
|
|
222
|
+
"install so opening the studio never waits on a 95-package tree.",
|
|
223
|
+
].join("\n"),
|
|
224
|
+
);
|
|
225
|
+
process.exit(1);
|
|
226
|
+
}
|
|
227
|
+
const child = spawn(process.execPath, [server, ...rest], { stdio: "inherit" });
|
|
228
|
+
child.on("exit", (code, signal) => process.exit(signal ? 1 : (code ?? 0)));
|
|
229
|
+
}
|
|
230
|
+
|
|
231
|
+
const argv = process.argv.slice(2);
|
|
232
|
+
if (argv[0] === "mcp") {
|
|
233
|
+
await runMcp(argv.slice(1));
|
|
234
|
+
} else {
|
|
235
|
+
|
|
236
|
+
const opts = parseArgs(argv);
|
|
193
237
|
if (opts.help) {
|
|
194
238
|
console.log(HELP);
|
|
195
239
|
process.exit(0);
|
|
@@ -219,7 +263,10 @@ if (opts.ardy && ardyHost && existsSync(BRIDGE)) {
|
|
|
219
263
|
|
|
220
264
|
const server = createServer((req, res) => {
|
|
221
265
|
const url = new URL(req.url ?? "/", "http://localhost");
|
|
222
|
-
|
|
266
|
+
// Only the routes the bridge actually owns: /ardy/ is ALSO a public asset
|
|
267
|
+
// directory (cskel27-rest.json), and those files live in dist/, not behind
|
|
268
|
+
// the sidecar. Same rule as the Vite dev proxy bypass.
|
|
269
|
+
if (/^\/ardy\/(health|bases|generate|footage|extract|motions)(\/|$)/.test(url.pathname)) {
|
|
223
270
|
proxyToBridge(req, res);
|
|
224
271
|
return;
|
|
225
272
|
}
|
|
@@ -286,3 +333,5 @@ server.listen(opts.port, opts.host, () => {
|
|
|
286
333
|
);
|
|
287
334
|
if (opts.open) openBrowser(url);
|
|
288
335
|
});
|
|
336
|
+
|
|
337
|
+
}
|
|
@@ -0,0 +1,406 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en">
|
|
3
|
+
<head>
|
|
4
|
+
<meta charset="UTF-8" />
|
|
5
|
+
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
|
|
6
|
+
<meta name="description" content="Camera control for AI video: why prompting an angle keeps drifting, what Seedance, Veo and Kling actually accept, and how blocking the shot in 3D first gives you the same camera on every take." />
|
|
7
|
+
<meta name="theme-color" content="#232323" />
|
|
8
|
+
<link rel="canonical" href="https://cozyclay.org/ai-camera-control/" />
|
|
9
|
+
<link rel="icon" href="/favicon.ico" sizes="any" />
|
|
10
|
+
<link rel="icon" type="image/png" sizes="32x32" href="/icons/icon-32.png" />
|
|
11
|
+
<link rel="icon" type="image/png" sizes="192x192" href="/icons/icon-192.png" />
|
|
12
|
+
<link rel="apple-touch-icon" href="/icons/apple-touch-icon.png" />
|
|
13
|
+
|
|
14
|
+
<meta property="og:type" content="article" />
|
|
15
|
+
<meta property="og:site_name" content="CozyClay" />
|
|
16
|
+
<meta property="og:title" content="Camera control for AI video: block the shot instead of describing it" />
|
|
17
|
+
<meta property="og:description" content="Why camera prompts drift between takes, and how blocking the angle, lens and dolly in 3D first gets you the same shot every time." />
|
|
18
|
+
<meta property="og:url" content="https://cozyclay.org/ai-camera-control/" />
|
|
19
|
+
<meta property="og:image" content="https://cozyclay.org/media/cozyclay-demo-poster.jpg" />
|
|
20
|
+
<meta property="og:image:width" content="1920" />
|
|
21
|
+
<meta property="og:image:height" content="1080" />
|
|
22
|
+
<meta name="twitter:card" content="summary_large_image" />
|
|
23
|
+
<meta name="twitter:title" content="Camera control for AI video: block the shot instead of describing it" />
|
|
24
|
+
<meta name="twitter:description" content="Why camera prompts drift between takes, and how blocking the angle, lens and dolly in 3D first gets you the same shot every time." />
|
|
25
|
+
<meta name="twitter:image" content="https://cozyclay.org/media/cozyclay-demo-poster.jpg" />
|
|
26
|
+
|
|
27
|
+
<title>Camera control for AI video — block the shot, don't describe it</title>
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
<script type="application/ld+json">
|
|
31
|
+
{
|
|
32
|
+
"@context": "https://schema.org",
|
|
33
|
+
"@type": "Article",
|
|
34
|
+
"headline": "Camera control for AI video: block the shot instead of describing it",
|
|
35
|
+
"description": "Why camera prompts drift between takes, and how blocking the angle, lens and dolly in 3D first gets you the same shot every time.",
|
|
36
|
+
"image": "https://cozyclay.org/media/cozyclay-demo-poster.jpg",
|
|
37
|
+
"mainEntityOfPage": "https://cozyclay.org/ai-camera-control/",
|
|
38
|
+
"author": {
|
|
39
|
+
"@type": "Organization",
|
|
40
|
+
"name": "CozyClay",
|
|
41
|
+
"url": "https://cozyclay.org/"
|
|
42
|
+
},
|
|
43
|
+
"publisher": {
|
|
44
|
+
"@type": "Organization",
|
|
45
|
+
"name": "CozyClay",
|
|
46
|
+
"logo": {
|
|
47
|
+
"@type": "ImageObject",
|
|
48
|
+
"url": "https://cozyclay.org/media/cozyclay-logo.png"
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
</script>
|
|
53
|
+
<script type="application/ld+json">
|
|
54
|
+
{
|
|
55
|
+
"@context": "https://schema.org",
|
|
56
|
+
"@type": "FAQPage",
|
|
57
|
+
"mainEntity": [
|
|
58
|
+
{
|
|
59
|
+
"@type": "Question",
|
|
60
|
+
"name": "Why does my camera angle change between takes?",
|
|
61
|
+
"acceptedAnswer": {
|
|
62
|
+
"@type": "Answer",
|
|
63
|
+
"text": "Because the prompt never fixed it. Words like low angle or close up carry no height, distance or focal length, so the model fills those in and fills them in differently on each run. Pinning composition needs something with numbers behind it - a starting frame, or a camera that exists in a scene."
|
|
64
|
+
}
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
"@type": "Question",
|
|
68
|
+
"name": "Do camera movement prompts work at all?",
|
|
69
|
+
"acceptedAnswer": {
|
|
70
|
+
"@type": "Answer",
|
|
71
|
+
"text": "They work as a nudge. Slow dolly in reliably gets you a push rather than a pan. What it will not do is get you the same push twice, or a push that matches the one in the shot you are cutting to."
|
|
72
|
+
}
|
|
73
|
+
},
|
|
74
|
+
{
|
|
75
|
+
"@type": "Question",
|
|
76
|
+
"name": "What is the highest-leverage thing I can control in AI video?",
|
|
77
|
+
"acceptedAnswer": {
|
|
78
|
+
"@type": "Answer",
|
|
79
|
+
"text": "The first frame. Composition, lens compression and eyeline are all decided the moment that image exists, and the generation inherits them instead of guessing. This is what blocking a shot in 3D gives you."
|
|
80
|
+
}
|
|
81
|
+
},
|
|
82
|
+
{
|
|
83
|
+
"@type": "Question",
|
|
84
|
+
"name": "Do I need Blender or Unreal to block a shot?",
|
|
85
|
+
"acceptedAnswer": {
|
|
86
|
+
"@type": "Answer",
|
|
87
|
+
"text": "No. Both will do it, and both ask you to learn a full DCC to move one camera. CozyClay does the blocking part in a browser tab with Unity-style controls."
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
]
|
|
91
|
+
}
|
|
92
|
+
</script>
|
|
93
|
+
|
|
94
|
+
<script defer src="https://static.cloudflareinsights.com/beacon.min.js" data-cf-beacon='{"token": "6a4384d1375d4ffc8a30a6346a8ad1a0"}'></script>
|
|
95
|
+
<style>
|
|
96
|
+
:root {
|
|
97
|
+
--bg: #1b1b1b;
|
|
98
|
+
--panel: #232323;
|
|
99
|
+
--line: #343434;
|
|
100
|
+
--ink: #ece7de;
|
|
101
|
+
--muted: #a9a29a;
|
|
102
|
+
--clay: #f2eee6;
|
|
103
|
+
--accent: #d9a066;
|
|
104
|
+
}
|
|
105
|
+
* { box-sizing: border-box; }
|
|
106
|
+
html { scroll-behavior: smooth; }
|
|
107
|
+
body {
|
|
108
|
+
margin: 0;
|
|
109
|
+
background: var(--bg);
|
|
110
|
+
color: var(--ink);
|
|
111
|
+
font: 16px/1.65 ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
|
112
|
+
-webkit-font-smoothing: antialiased;
|
|
113
|
+
}
|
|
114
|
+
a { color: var(--accent); }
|
|
115
|
+
.wrap { max-width: 980px; margin: 0 auto; padding: 0 20px; }
|
|
116
|
+
header.site {
|
|
117
|
+
border-bottom: 1px solid var(--line);
|
|
118
|
+
position: sticky; top: 0; background: rgba(27,27,27,.9);
|
|
119
|
+
backdrop-filter: blur(8px); z-index: 10;
|
|
120
|
+
}
|
|
121
|
+
header.site .wrap {
|
|
122
|
+
display: flex; align-items: center; gap: 16px;
|
|
123
|
+
min-height: 60px; flex-wrap: wrap;
|
|
124
|
+
}
|
|
125
|
+
.brand { font-weight: 700; letter-spacing: -.01em; color: var(--clay); text-decoration: none; font-size: 18px; }
|
|
126
|
+
header.site nav { margin-left: auto; display: flex; gap: 18px; flex-wrap: wrap; }
|
|
127
|
+
header.site nav a { color: var(--muted); text-decoration: none; font-size: 14px; }
|
|
128
|
+
header.site nav a:hover { color: var(--ink); }
|
|
129
|
+
|
|
130
|
+
.hero { padding: 64px 0 40px; }
|
|
131
|
+
h1 { font-size: clamp(30px, 5.2vw, 46px); line-height: 1.12; margin: 0 0 18px; letter-spacing: -.02em; color: var(--clay); }
|
|
132
|
+
.lede { font-size: clamp(17px, 2.2vw, 20px); color: var(--muted); margin: 0 0 28px; max-width: 44em; }
|
|
133
|
+
.cta { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 14px; }
|
|
134
|
+
.btn {
|
|
135
|
+
display: inline-block; padding: 12px 20px; border-radius: 8px;
|
|
136
|
+
text-decoration: none; font-weight: 600; font-size: 15px; border: 1px solid var(--line);
|
|
137
|
+
}
|
|
138
|
+
.btn.primary { background: var(--clay); color: #1b1b1b; border-color: var(--clay); }
|
|
139
|
+
.btn.ghost { color: var(--ink); }
|
|
140
|
+
.note { color: var(--muted); font-size: 14px; margin: 0; }
|
|
141
|
+
|
|
142
|
+
figure { margin: 40px 0; }
|
|
143
|
+
figure img { width: 100%; height: auto; display: block; border: 1px solid var(--line); border-radius: 10px; }
|
|
144
|
+
figcaption { color: var(--muted); font-size: 13px; margin-top: 10px; }
|
|
145
|
+
|
|
146
|
+
/* The mark is blue on white and the eyes are knocked out of the fill, so
|
|
147
|
+
it sits on a light plate rather than straight on the dark page. */
|
|
148
|
+
.mark { margin: 0 0 26px; max-width: 190px; }
|
|
149
|
+
.mark img {
|
|
150
|
+
border: 0;
|
|
151
|
+
border-radius: 14px;
|
|
152
|
+
background: var(--clay);
|
|
153
|
+
padding: 14px 12px;
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
.reel {
|
|
157
|
+
margin: 0 0 30px;
|
|
158
|
+
border: 1px solid var(--line);
|
|
159
|
+
border-radius: 12px;
|
|
160
|
+
overflow: hidden;
|
|
161
|
+
background: #000;
|
|
162
|
+
box-shadow: 0 24px 60px rgba(0, 0, 0, .45);
|
|
163
|
+
}
|
|
164
|
+
.reel video { display: block; width: 100%; height: auto; }
|
|
165
|
+
.reel-note { color: var(--muted); font-size: 13px; margin: 12px 0 0; }
|
|
166
|
+
|
|
167
|
+
section { padding: 40px 0; border-top: 1px solid var(--line); }
|
|
168
|
+
h2 { font-size: clamp(22px, 3.2vw, 28px); margin: 0 0 8px; color: var(--clay); letter-spacing: -.01em; }
|
|
169
|
+
h3 { font-size: 17px; margin: 0 0 6px; color: var(--clay); }
|
|
170
|
+
.sub { color: var(--muted); margin: 0 0 26px; max-width: 48em; }
|
|
171
|
+
|
|
172
|
+
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: 18px; }
|
|
173
|
+
.card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 20px; }
|
|
174
|
+
.card p { margin: 0; color: var(--muted); font-size: 15px; }
|
|
175
|
+
|
|
176
|
+
pre {
|
|
177
|
+
background: var(--panel); border: 1px solid var(--line); border-radius: 10px;
|
|
178
|
+
padding: 16px 18px; overflow-x: auto; font-size: 14px; line-height: 1.7;
|
|
179
|
+
}
|
|
180
|
+
code { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
|
|
181
|
+
|
|
182
|
+
table { width: 100%; border-collapse: collapse; font-size: 15px; }
|
|
183
|
+
th, td { text-align: left; padding: 10px 12px; border-bottom: 1px solid var(--line); }
|
|
184
|
+
th { color: var(--clay); font-weight: 600; white-space: nowrap; }
|
|
185
|
+
td { color: var(--muted); }
|
|
186
|
+
.scroller { overflow-x: auto; }
|
|
187
|
+
|
|
188
|
+
details { border-bottom: 1px solid var(--line); padding: 16px 0; }
|
|
189
|
+
details summary { cursor: pointer; font-weight: 600; color: var(--clay); font-size: 16px; }
|
|
190
|
+
details p { color: var(--muted); margin: 12px 0 0; }
|
|
191
|
+
|
|
192
|
+
footer.site { border-top: 1px solid var(--line); padding: 32px 0 56px; color: var(--muted); font-size: 14px; }
|
|
193
|
+
footer.site a { color: var(--muted); }
|
|
194
|
+
|
|
195
|
+
@media (max-width: 620px) {
|
|
196
|
+
.hero { padding: 40px 0 28px; }
|
|
197
|
+
.btn { width: 100%; text-align: center; }
|
|
198
|
+
header.site nav { gap: 14px; width: 100%; margin-left: 0; padding-bottom: 10px; }
|
|
199
|
+
}
|
|
200
|
+
</style>
|
|
201
|
+
</head>
|
|
202
|
+
<body>
|
|
203
|
+
<header class="site">
|
|
204
|
+
<div class="wrap">
|
|
205
|
+
<a class="brand" href="/">CozyClay</a>
|
|
206
|
+
<nav>
|
|
207
|
+
<a href="/#what">What it does</a>
|
|
208
|
+
<a href="/#start">Quick start</a>
|
|
209
|
+
<a href="/ai-camera-control/">AI camera control</a>
|
|
210
|
+
<a href="https://github.com/HaD0Yun/CozyClay">GitHub</a>
|
|
211
|
+
</nav>
|
|
212
|
+
</div>
|
|
213
|
+
</header>
|
|
214
|
+
|
|
215
|
+
<main class="wrap">
|
|
216
|
+
<div class="hero">
|
|
217
|
+
<h1>Camera control for AI video</h1>
|
|
218
|
+
<p class="lede">
|
|
219
|
+
Video models will happily give you a low angle. They will not give you
|
|
220
|
+
<em>the same</em> low angle twice. Here is why the prompt keeps drifting, and what
|
|
221
|
+
to do about it.
|
|
222
|
+
</p>
|
|
223
|
+
</div>
|
|
224
|
+
|
|
225
|
+
<section>
|
|
226
|
+
<h2>The part nobody warns you about</h2>
|
|
227
|
+
<p class="sub">
|
|
228
|
+
A prompt describes a camera in words, and words have no units. "Low angle, close on
|
|
229
|
+
her face, slow push in" is four decisions with no numbers attached: how low, how
|
|
230
|
+
close, what focal length, how far the push travels. The model picks values you never
|
|
231
|
+
specified, and it picks different ones on the next run.
|
|
232
|
+
</p>
|
|
233
|
+
<p class="sub">
|
|
234
|
+
That is fine for a single hero shot. It falls apart the moment you need two shots to
|
|
235
|
+
belong to the same scene. Cut between two takes that each invented their own lens and
|
|
236
|
+
eyeline and the geometry does not agree — the room changes shape across the cut, and
|
|
237
|
+
the audience feels it even when they cannot name it.
|
|
238
|
+
</p>
|
|
239
|
+
<p class="sub">
|
|
240
|
+
The usual response is to write a longer prompt. More adjectives, a reference image,
|
|
241
|
+
a list of shot-type vocabulary. It narrows the spread. It does not close it, because
|
|
242
|
+
the thing being specified was never language in the first place.
|
|
243
|
+
</p>
|
|
244
|
+
</section>
|
|
245
|
+
|
|
246
|
+
<section>
|
|
247
|
+
<h2>What the models actually take</h2>
|
|
248
|
+
<p class="sub">
|
|
249
|
+
The current generation of video models accepts camera intent through some mix of
|
|
250
|
+
three channels, and it is worth knowing which one you are using:
|
|
251
|
+
</p>
|
|
252
|
+
<div class="grid">
|
|
253
|
+
<div class="card">
|
|
254
|
+
<h3>Words</h3>
|
|
255
|
+
<p>
|
|
256
|
+
Shot-type vocabulary in the prompt — wide, medium, close, dolly, crane, 35mm.
|
|
257
|
+
Universally supported, and the loosest. The model interprets; you do not specify.
|
|
258
|
+
</p>
|
|
259
|
+
</div>
|
|
260
|
+
<div class="card">
|
|
261
|
+
<h3>A first frame</h3>
|
|
262
|
+
<p>
|
|
263
|
+
An image the generation starts from. This pins composition hard, because framing,
|
|
264
|
+
lens compression and eyeline are already baked into the pixels. It is the highest
|
|
265
|
+
leverage input most people underuse.
|
|
266
|
+
</p>
|
|
267
|
+
</div>
|
|
268
|
+
<div class="card">
|
|
269
|
+
<h3>A path or reference clip</h3>
|
|
270
|
+
<p>
|
|
271
|
+
Some models take an explicit camera path, or a video whose motion is transferred
|
|
272
|
+
onto a new scene. Strongest control, narrowest support, and it needs the move to
|
|
273
|
+
exist somewhere before you can hand it over.
|
|
274
|
+
</p>
|
|
275
|
+
</div>
|
|
276
|
+
</div>
|
|
277
|
+
</section>
|
|
278
|
+
|
|
279
|
+
<section>
|
|
280
|
+
<h2>Block the shot, then generate</h2>
|
|
281
|
+
<p class="sub">
|
|
282
|
+
Film production solved this before any of these models existed, and the answer was
|
|
283
|
+
previs: build the scene rough in 3D, put a camera in it, and settle the shot while
|
|
284
|
+
it is still cheap to change. The blocking is the specification. Everything downstream
|
|
285
|
+
inherits it.
|
|
286
|
+
</p>
|
|
287
|
+
<p class="sub">
|
|
288
|
+
The same move works here. Put the geometry somewhere real first — a boxed-out set, a
|
|
289
|
+
character on a path, a camera with an actual position and an actual focal length —
|
|
290
|
+
and the questions the prompt was leaving open are already answered. You are no longer
|
|
291
|
+
asking for a low angle. You have one, and you can hand over the frame it produces.
|
|
292
|
+
</p>
|
|
293
|
+
<p class="sub">
|
|
294
|
+
Three things change once the camera lives in a scene instead of a sentence:
|
|
295
|
+
</p>
|
|
296
|
+
<div class="grid">
|
|
297
|
+
<div class="card">
|
|
298
|
+
<h3>Shots agree with each other</h3>
|
|
299
|
+
<p>
|
|
300
|
+
Two angles on one blocked set share a floor plan, so a cut between them holds.
|
|
301
|
+
Two prompts do not share anything.
|
|
302
|
+
</p>
|
|
303
|
+
</div>
|
|
304
|
+
<div class="card">
|
|
305
|
+
<h3>The move is reproducible</h3>
|
|
306
|
+
<p>
|
|
307
|
+
A dolly you drew is the same dolly tomorrow. Regenerate, change the subject, keep
|
|
308
|
+
the camera. Nothing about that survives a reroll of a text prompt.
|
|
309
|
+
</p>
|
|
310
|
+
</div>
|
|
311
|
+
<div class="card">
|
|
312
|
+
<h3>You fail cheaply</h3>
|
|
313
|
+
<p>
|
|
314
|
+
Finding out the angle was wrong costs a drag in a viewport, not a paid generation
|
|
315
|
+
and the wait attached to it.
|
|
316
|
+
</p>
|
|
317
|
+
</div>
|
|
318
|
+
</div>
|
|
319
|
+
</section>
|
|
320
|
+
|
|
321
|
+
<section>
|
|
322
|
+
<h2>Doing it in the browser</h2>
|
|
323
|
+
<p class="sub">
|
|
324
|
+
CozyClay is the previs half of that loop, built to be opened rather than installed.
|
|
325
|
+
Block a set out of primitives, drop a character and draw the path they walk, place
|
|
326
|
+
the camera and set its lens, cut between shots on a timeline, and scrub until the
|
|
327
|
+
timing is right. Then take those frames and those camera moves to whichever model you
|
|
328
|
+
are generating with.
|
|
329
|
+
</p>
|
|
330
|
+
|
|
331
|
+
<div class="reel">
|
|
332
|
+
<video
|
|
333
|
+
src="/media/cozyclay-demo.mp4"
|
|
334
|
+
poster="/media/cozyclay-demo-poster.jpg"
|
|
335
|
+
width="1920" height="1080"
|
|
336
|
+
muted loop playsinline controls preload="none"
|
|
337
|
+
></video>
|
|
338
|
+
</div>
|
|
339
|
+
<p class="reel-note">
|
|
340
|
+
Left of the wipe is the generated result. Right of it is the blocking that produced
|
|
341
|
+
the camera.
|
|
342
|
+
</p>
|
|
343
|
+
|
|
344
|
+
<div class="cta">
|
|
345
|
+
<a class="btn primary" href="/">What CozyClay does</a>
|
|
346
|
+
<a class="btn ghost" href="https://github.com/HaD0Yun/CozyClay">View on GitHub</a>
|
|
347
|
+
</div>
|
|
348
|
+
<p class="note">
|
|
349
|
+
Free and open source. Runs locally with <code>npx cozyclay</code>.
|
|
350
|
+
</p>
|
|
351
|
+
</section>
|
|
352
|
+
|
|
353
|
+
<section id="faq">
|
|
354
|
+
<h2>FAQ</h2>
|
|
355
|
+
<details open>
|
|
356
|
+
<summary>Why does my camera angle change between takes?</summary>
|
|
357
|
+
<p>
|
|
358
|
+
Because the prompt never fixed it. Words like "low angle" or "close up" carry no
|
|
359
|
+
height, distance or focal length, so the model fills those in and fills them in
|
|
360
|
+
differently on each run. Pinning composition needs something with numbers behind it
|
|
361
|
+
— a starting frame, or a camera that exists in a scene.
|
|
362
|
+
</p>
|
|
363
|
+
</details>
|
|
364
|
+
<details>
|
|
365
|
+
<summary>Do camera movement prompts work at all?</summary>
|
|
366
|
+
<p>
|
|
367
|
+
They work as a nudge. "Slow dolly in" reliably gets you a push rather than a pan.
|
|
368
|
+
What it will not do is get you the same push twice, or a push that matches the one
|
|
369
|
+
in the shot you are cutting to. Use the vocabulary, but do not expect it to carry
|
|
370
|
+
continuity.
|
|
371
|
+
</p>
|
|
372
|
+
</details>
|
|
373
|
+
<details>
|
|
374
|
+
<summary>What is the highest-leverage thing I can control?</summary>
|
|
375
|
+
<p>
|
|
376
|
+
The first frame. Composition, lens compression and eyeline are all decided the
|
|
377
|
+
moment that image exists, and the generation inherits them instead of guessing.
|
|
378
|
+
This is exactly what blocking a shot in 3D gives you.
|
|
379
|
+
</p>
|
|
380
|
+
</details>
|
|
381
|
+
<details>
|
|
382
|
+
<summary>Do I need Blender or Unreal to block a shot?</summary>
|
|
383
|
+
<p>
|
|
384
|
+
No. Both will do it, and both ask you to learn a full DCC to move one camera.
|
|
385
|
+
CozyClay does the blocking part in a browser tab with Unity-style controls —
|
|
386
|
+
see <a href="/">what it does</a>.
|
|
387
|
+
</p>
|
|
388
|
+
</details>
|
|
389
|
+
</section>
|
|
390
|
+
</main>
|
|
391
|
+
|
|
392
|
+
<footer class="site">
|
|
393
|
+
<div class="wrap">
|
|
394
|
+
<p>
|
|
395
|
+
<a href="/">CozyClay</a> ·
|
|
396
|
+
<a href="https://github.com/HaD0Yun/CozyClay">GitHub</a> ·
|
|
397
|
+
<a href="https://www.npmjs.com/package/cozyclay">npm</a>
|
|
398
|
+
</p>
|
|
399
|
+
<p>
|
|
400
|
+
CozyClay is not affiliated with or endorsed by NVIDIA, Unity Technologies, ByteDance,
|
|
401
|
+
Google or Kuaishou. Product names are used only to describe interoperability.
|
|
402
|
+
</p>
|
|
403
|
+
</div>
|
|
404
|
+
</footer>
|
|
405
|
+
</body>
|
|
406
|
+
</html>
|