cozyclay 1.1.0 → 1.3.0

package/mcp/LIVE-PROTOCOL.md

@@ -0,0 +1,63 @@
+# CozyClay live-control protocol v1
+
+One WebSocket, JSON text frames. The **MCP server hosts** the socket
+(`ws://127.0.0.1:5184/live`); the **editor is the client** and reconnects
+every 3 s while the page is open. Either side may be absent: the editor works
+exactly as before when nothing is listening, and the MCP server falls back to
+its in-memory scene when no editor is connected.
+
+## Frames
+
+editor -> server, once after connect:
+
+    { "type": "hello", "role": "editor", "version": 1 }
+
+server -> editor, one per command:
+
+    { "type": "cmd", "id": "<opaque string>", "name": "<command>", "args": { ... } }
+
+editor -> server, one per command, echoing `id`:
+
+    { "type": "result", "id": "<same id>", "ok": true,  "value": { ... } }
+    { "type": "result", "id": "<same id>", "ok": false, "error": "<human message>" }
+
+Unknown `name` MUST answer `ok:false`, never silence. The server times a
+command out after 5 s and treats it as failed.
+
+## Commands (v1)
+
+All coordinates are metres, rotations are degrees of yaw, ids are the ids the
+scene document already uses.
+
+| name | args | value | notes |
+| --- | --- | --- | --- |
+| `ping` | `{}` | `{ "pong": true }` | liveness |
+| `describe` | `{}` | `{ sceneName, camera: { x, y, z, focalMm }, characters: [{ id, subject, x, z, rot, hidden }], objects: [{ id, name, x, y, z, rot, scaleX, scaleY, scaleZ, footprint, height }] }` | read the live scene; object scale and footprint travel too, so sizes are reported from what was actually built |
+| `set_camera` | `{ x?, y?, z?, focalMm? }` | `{ camera }` | omitted fields keep their value; the **viewport must visibly move** |
+| `add_character` | `{ subject, x?, z?, rot? }` | `{ id }` | |
+| `update_character` | `{ ref, x?, z?, rot?, subject?, hidden? }` | `{ id }` | `ref` = id, letter (`"A"`) or 1-based slot |
+| `remove_character` | `{ ref }` | `{ id }` | must refuse to empty the cast |
+| `place_object` | `{ kind, x?, z?, y?, rot? }` | `{ id }` | `kind` from OBJECT_LIBRARY |
+| `update_object` | `{ id, x?, y?, z?, rot?, rotX?, rotZ?, scale?, scaleX?, scaleY?, scaleZ?, color? }` | `{ id }` | `scale` sets all three axes; per-axis values override it |
+| `remove_object` | `{ id }` | `{ id }` | |
+| `load_scenes` | `{ document }` | `{ sceneName }` | replace the whole scene document (same shape `serializeSceneDocument` emits); the big hammer that guarantees parity |
+
+## Hard rules for the editor side
+
+- Every mutation MUST go through the same React state paths the UI itself
+  uses (the setters/reducers the gizmo, inspector and panels call). Mutating
+  three.js objects directly is forbidden: anything outside React state is
+  overwritten on the next render.
+- Every mutation MUST land in undo history exactly like the equivalent UI
+  action would, or be explicitly documented as not undoable.
+- The socket client MUST be a no-op in production builds unless explicitly
+  enabled; in dev it may always try. A failed connection must never surface
+  an error to the user - silence and retry.
+
+## Hard rules for the server side
+
+- When an editor is connected, live-capable tools forward to it and answer
+  from its `describe`; when none is connected they fall back to the in-memory
+  scene exactly as today. The tool surface does not change.
+- One editor at a time: a second `hello` replaces the first (last write wins),
+  and the displaced socket is closed.

package/mcp/README.md

@@ -0,0 +1,142 @@
+# CozyClay MCP server
+
+Lets an AI assistant (Claude Desktop, Cursor, any MCP client) block a scene, place the camera,
+generate character motion and read the shot back as film vocabulary — then turn it into an AI
+image/video prompt.
+
+Works two ways, with the same tools:
+
+- **Editor open** — tool calls drive the visible viewport live: camera, cast, set, motion,
+  prompt blocks on the timeline.
+- **No editor** — everything runs headless: no browser, no GPU, no build step.
+
+## Run it locally
+
+```sh
+cd mcp
+npm install
+npm start          # speaks MCP over stdio and hosts ws://127.0.0.1:5184/live
+npm run verify     # drives the no-editor fallback as a client and checks results
+npm run verify:live # drives a fake editor over the live WebSocket protocol
+```
+
+Then point a client at it. For Claude Desktop, in `claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "cozyclay": {
+      "command": "node",
+      "args": ["/absolute/path/to/CozyClay/mcp/server.mjs"]
+    }
+  }
+}
+```
+
+Restart the client; 20 tools appear.
+
+Prefer a long-lived endpoint? `node server.mjs --http 5183` serves Streamable HTTP at
+`http://127.0.0.1:5183/mcp` (one isolated session per client), with a plain status page at `/`.
+
+## What it does
+
+Describe the shot you want. The server does the trigonometry and answers the way a crew would.
+
+> "Put a detective and a courier in an alley, then give me a low wide profile shot of the courier."
+
+```
+WIDE SHOT · RIGHT PROFILE · KNEE LEVEL · 24MM
+
+size      wide shot — the subject fills 40% of frame height
+view      a right-side profile view
+level     a low knee-level angle looking up at the subject
+distance  4.39m
+```
+
+`render_prompt` turns that same geometry into a prompt that carries the real framing, so the
+generated frame matches the blocking instead of drifting off into a generic shot.
+
+## Tools
+
+| tool | what it does |
+| --- | --- |
+| `describe_scene` | the whole state: camera, framing, cast, set |
+| `live_status` | whether an editor tab is connected for live control |
+| `describe_shot` | current camera geometry as film vocabulary |
+| `set_camera` | move the lens / change focal length directly |
+| `frame_shot` | frame by intent — size, view, level, side |
+| `add_character` / `place_character` / `remove_character` | the cast |
+| `focus_character` | choose who the camera frames |
+| `place_object` / `update_object` / `remove_object` | the set |
+| `render_prompt` | the shot as an AI image or video prompt |
+| `generate_motion` | multi-phase character motion through the ARDY bridge — phases land as prompt blocks |
+| `mark_camera_move` / `describe_camera_move` | name a move between two camera positions |
+| `add_scene` / `switch_scene` | multiple scenes per project |
+| `open_project` / `save_project` | read and write `.cclayproject` files |
+
+Coordinates are metres (`x` right, `z` toward the default camera, `y` height above the floor).
+Rotations are degrees of yaw. Characters are addressed by letter (`"A"`), slot (`"2"`) or id
+(`"char-a"`).
+
+## Live editor control
+
+The server hosts `ws://127.0.0.1:5184/live` by default (`COZYCLAY_LIVE_PORT` or
+`--live-port <port>` to change it). A CozyClay editor tab served by `npm run dev` connects on
+its own; `live_status` tells you whether one is attached. With an editor connected, mutations
+forward to it and reads report its real state — the screen you are looking at is the source of
+truth. Without one, every tool keeps its in-memory behaviour. In `--http` mode each session
+child attempts to bind the live port, so one session owns the editor and the others
+intentionally run memory-only.
+
+The wire protocol — one WebSocket, ten commands, editor-side rules — is specified in
+[`LIVE-PROTOCOL.md`](LIVE-PROTOCOL.md).
+
+## Motion generation
+
+`generate_motion` takes plain-language beats and a length:
+
+```
+phases: ["seated on a chair, slowly stands up",
+         "breaks into a sprint",
+         "trips and falls hard to the ground"]
+seconds: 10
+```
+
+It tiles them into contiguous ARDY segments, streams the generation through the local bridge
+(`127.0.0.1:5181`, started by `npm run dev`), and — when an editor is connected — loads the
+result onto the active character with one prompt block per phase on the timeline. Pass a
+previous `motion_url` to reload a clip without generating again.
+
+## Round-trips with the studio
+
+`save_project` writes a real `.cclayproject` file, so a scene blocked here opens in the studio to
+be posed, timed and generated — and a scene built in the studio opens here with `open_project`.
+
+## Design
+
+This server owns no geometry, no film vocabulary and no prompt text. Every answer is computed by
+the same modules the studio renders with, imported straight from the working tree:
+
+| module | responsibility |
+| --- | --- |
+| `../src/shot.js` | geometry → film vocabulary → prompt |
+| `../src/scenes.js` | the scene document and its stage envelope |
+| `../src/scene-objects.js` | the set: create / update / remove, clamped and snapped |
+| `../src/camera-move.js` | two framings → a named camera move |
+| `../src/project.js` | the `.cclayproject` envelope |
+
+That is the whole design. Because the imports are relative, the server always speaks the working
+tree's vocabulary: retune a band in `shot.js` and this server reports the new answer on its next
+start, with nothing to publish or reinstall. The studio and the MCP server cannot disagree about
+what a 35mm medium shot is, because there is only one implementation of it.
+
+`frame_shot` is the one place that inverts `shot.js` rather than calling it — it solves for the
+camera position that produces a requested size. `npm run verify` checks all 420 combinations the
+schema accepts, so a retune in `shot.js` fails loudly here instead of quietly mis-framing.
+
+### Framing conflicts
+
+Shot size is treated as the stronger request. An extreme close-up from an overhead lens is
+geometrically impossible on a wide lens — the camera would sit inside the subject — so the server
+lengthens the lens until the requested angle fits and says so, the way a crew swaps glass rather
+than abandoning the close-up.

package/mcp/ardy-prompts.mjs

@@ -0,0 +1,170 @@
+/**
+ * Writing prompts the way ARDY was trained to read them.
+ *
+ * ARDY's own examples are the specification (nv-tlabs/ardy, scripts/generate.py):
+ *
+ *     python scripts/generate.py "A person walks in a circle."
+ *     python scripts/generate.py "A person jumps." --model core --duration 8.0
+ *     python scripts/generate.py "A person waves." --model g1 --num_samples 4
+ *
+ * Three properties do the work: an explicit subject ("A person"), exactly one
+ * physical action, and a full stop. The demo's preset Prompt List is built the
+ * same way, so this is the distribution the text encoder actually saw.
+ *
+ * The reason one action per prompt matters is architectural, not stylistic:
+ * the interactive demo builds its denoiser with `num_text_tokens=1`
+ * (scripts/interactive_demo/loading.py), so the entire prompt collapses into a
+ * SINGLE conditioning embedding. Two actions in one sentence do not run in
+ * sequence — they average into one confused pose. A sequence of actions is
+ * expressed as a sequence of prompts (CozyClay's Prompt Blocks), never as a
+ * compound sentence.
+ *
+ * What that rules out, and what this module rewrites:
+ *   - no subject            "runs forward"        -> "A person runs forward."
+ *   - compound actions      "stands, then runs"   -> split across phases
+ *   - interior states       "in astonishment"     -> dropped; ARDY animates
+ *                                                    bodies, not feelings
+ *   - camera/scene language "the rocket looms"    -> dropped; the prompt
+ *                                                    describes the BODY only
+ */
+
+/**
+ * Quality policy, mirrored from the studio (App.jsx PROMPT_BLOCK_MAX_FRAMES):
+ * one block never spans more than 4 s. ARDY's trained window is 10 s, but a
+ * long single block drifts — chained 4 s blocks keep each call inside the
+ * model's sweet spot. The studio refuses to generate a longer block, so a tool
+ * that produced one would be authoring something the UI would then reject.
+ */
+export const BLOCK_MAX_SECONDS = 4;
+
+/**
+ * Split a beat that runs longer than the cap into consecutive blocks that do
+ * not. The wording is kept for each piece: continuing the same action is
+ * exactly what the chained-block design is for. Returns whole seconds-ish
+ * spans that sum to the original duration.
+ */
+export function splitLongBeat(seconds, max = BLOCK_MAX_SECONDS) {
+	if (!(seconds > max)) return [seconds];
+	const count = Math.ceil(seconds / max);
+	const even = seconds / count;
+	return Array.from({ length: count }, () => even);
+}
+
+/** Shown in the tool description so a caller writes good phases first time. */
+export const PROMPT_GUIDE = [
+	"ARDY prompt rules (from nv-tlabs/ardy):",
+	'  - One action per phase, phrased like ARDY\'s own examples: "A person walks in a circle."',
+	"  - Subject + single present-tense action + full stop. The prompt becomes ONE embedding",
+	"    (num_text_tokens=1), so two actions in one phase average together instead of playing in order.",
+	"  - Sequence = more phases, never a compound sentence.",
+	`  - A block holds at most ${BLOCK_MAX_SECONDS} s. Longer beats are chained into consecutive blocks,`,
+	"    because a single long block drifts away from its prompt.",
+	"  - Describe the BODY: no emotions, no camera, no scenery, no props the model cannot infer.",
+	"  - WRITE THE AMPLITUDE. A diffusion model regresses toward the mean, so a neutral verb",
+	"    generates a smaller motion than the words suggest. Pick the strong verb and state the",
+	"    magnitude: 'strides forward quickly' over 'walks forward', 'stops abruptly' over 'slows",
+	"    to a stop', 'leans far back and looks straight up' over 'looks up'. Simultaneous detail",
+	"    describing ONE pose is fine; sequential actions still need separate phases.",
+	'  - Good: ["A person strides forward quickly.", "A person stops abruptly.", "A person leans far back and looks straight up."]',
+	'  - Bad:  ["walks forward slowly, then stops abruptly", "staggers back in astonishment"]',
+].join("\n");
+
+/** Interior states and cinematic language ARDY has no body channel for. */
+const UNRENDERABLE = [
+	/\b(in|with)\s+(astonishment|awe|wonder|surprise|fear|joy|excitement|disbelief)\b/gi,
+	/\b(astonished|amazed|awestruck|terrified|delighted|confused|nervous|curious)ly?\b/gi,
+	/\b(as if|like)\s+[^,.]+/gi,
+	/\b(at|toward|towards)\s+(something|the)\s+(enormous|huge|massive|towering|giant)\b/gi,
+	/\b(the\s+)?(camera|shot|frame|rocket|spaceship|building)\b[^,.]*/gi,
+];
+
+/** Connectives that mean "a second action is hiding in this sentence". */
+const SPLIT_ON = /\s*,?\s*\b(?:and then|then|after that|before|while|as)\b\s*/i;
+
+const SUBJECT = /^(a|the)\s+(person|man|woman|character|figure|human)\b/i;
+
+const tidy = (s) =>
+	s
+		.replace(/\s+/g, " ")
+		.replace(/\s+([,.])/g, "$1")
+		.replace(/[,;]+$/, "")
+		.trim();
+
+/**
+ * Rewrite one caller phrase into ARDY's sentence shape.
+ * Returns `{ text, notes }` — notes explain every edit, so a caller can see
+ * why their wording changed rather than silently getting something else.
+ */
+export function normalizePhase(raw) {
+	const notes = [];
+	let s = tidy(String(raw ?? ""));
+	if (!s) return { text: "", notes: ["empty phrase"] };
+
+	for (const pattern of UNRENDERABLE) {
+		if (pattern.test(s)) {
+			s = tidy(s.replace(pattern, ""));
+			notes.push("dropped language ARDY cannot animate (emotion, camera or scenery)");
+			break;
+		}
+	}
+
+	// Keep the first action; a trailing clause belongs in its own phase.
+	const parts = s.split(SPLIT_ON).map(tidy).filter(Boolean);
+	if (parts.length > 1) {
+		s = parts[0];
+		notes.push(`kept the first action; "${parts.slice(1).join(" / ")}" belongs in its own phase`);
+	}
+
+	// A comma usually joins two actions too ("walks forward, stops abruptly").
+	const commaParts = s.split(/\s*,\s*/).map(tidy).filter(Boolean);
+	if (commaParts.length > 1) {
+		s = commaParts[0];
+		notes.push("kept one action per prompt");
+	}
+
+	s = s.replace(/^(and|then|so)\s+/i, "");
+	if (!SUBJECT.test(s)) {
+		s = s.replace(/^[A-Z]/, (c) => c.toLowerCase());
+		s = `A person ${s}`;
+		notes.push('added the subject ARDY expects ("A person ...")');
+	} else {
+		s = s.replace(/^./, (c) => c.toUpperCase());
+	}
+
+	if (!/[.!?]$/.test(s)) {
+		s = `${s}.`;
+		notes.push("closed the sentence");
+	}
+	return { text: tidy(s), notes };
+}
+
+/**
+ * Normalise a whole beat list. A phrase that carried a trailing action gets it
+ * back as its own phase, so "stands up then runs" becomes two real beats
+ * instead of one averaged embedding — capped so a caller cannot blow past the
+ * schema's phase limit.
+ */
+export function normalizePhases(phases, max = 8) {
+	const expanded = [];
+	const sources = [];
+	for (const [index, phase] of phases.entries()) {
+		const pieces = String(phase ?? "")
+			.split(SPLIT_ON)
+			.map(tidy)
+			.filter(Boolean);
+		const parts = pieces.length ? pieces : [phase];
+		expanded.push(...parts);
+		// Which input each output came from, so a caller that attached a duration
+		// to a beat can split that duration across the pieces it became.
+		sources.push(...parts.map(() => index));
+	}
+	const capped = expanded.slice(0, max);
+	const results = capped.map((piece) => normalizePhase(piece));
+	return {
+		texts: results.map((r) => r.text),
+		notes: results.map((r) => r.notes),
+		sources: sources.slice(0, max),
+		expanded: expanded.length > phases.length,
+		dropped: expanded.length - capped.length,
+	};
+}

package/mcp/live-hub.mjs

@@ -0,0 +1,105 @@
+import { randomUUID } from "node:crypto";
+
+import { WebSocket, WebSocketServer } from "ws";
+
+const CLOSE_REPLACED = 4000;
+const COMMAND_TIMEOUT_MS = 5_000;
+
+/**
+ * Transport-only implementation of LIVE-PROTOCOL.md. Scene semantics remain
+ * with the editor and the server's existing CozyClay imports.
+ */
+export class LiveHub {
+	constructor(server = null) {
+		this.server = server;
+		this.editor = null;
+		this.pending = new Map();
+	}
+
+	get connected() {
+		return this.editor?.readyState === WebSocket.OPEN;
+	}
+
+	async command(name, args) {
+		const socket = this.editor;
+		if (!socket || socket.readyState !== WebSocket.OPEN) throw new Error("No live editor is connected.");
+
+		const id = randomUUID();
+		return new Promise((resolve, reject) => {
+			const timer = setTimeout(() => {
+				this.pending.delete(id);
+				reject(new Error(`Live editor timed out running ${name}.`));
+			}, COMMAND_TIMEOUT_MS);
+			this.pending.set(id, { socket, resolve, reject, timer });
+			try {
+				socket.send(JSON.stringify({ type: "cmd", id, name, args }));
+			} catch (error) {
+				clearTimeout(timer);
+				this.pending.delete(id);
+				reject(new Error(`Could not send ${name} to the live editor: ${error.message}`));
+			}
+		});
+	}
+
+	accept(socket) {
+		let greeted = false;
+		socket.on("message", (message, isBinary) => {
+			if (isBinary) return;
+			let frame;
+			try {
+				frame = JSON.parse(message.toString());
+			} catch {
+				return;
+			}
+			if (!greeted) {
+				if (frame?.type !== "hello" || frame.role !== "editor" || frame.version !== 1) {
+					socket.close(1002, "Expected editor hello version 1");
+					return;
+				}
+				greeted = true;
+				const displaced = this.editor;
+				this.editor = socket;
+				if (displaced && displaced !== socket) displaced.close(CLOSE_REPLACED, "Replaced by a newer editor");
+				return;
+			}
+			if (frame?.type !== "result" || typeof frame.id !== "string") return;
+			const pending = this.pending.get(frame.id);
+			if (!pending || pending.socket !== socket) return;
+			clearTimeout(pending.timer);
+			this.pending.delete(frame.id);
+			if (frame.ok === true) pending.resolve(frame.value);
+			else pending.reject(new Error(typeof frame.error === "string" ? frame.error : "Live editor rejected the command."));
+		});
+		socket.on("close", () => this.disconnect(socket));
+		socket.on("error", () => this.disconnect(socket));
+	}
+
+	disconnect(socket) {
+		if (this.editor === socket) this.editor = null;
+		for (const [id, pending] of this.pending) {
+			if (pending.socket !== socket) continue;
+			clearTimeout(pending.timer);
+			this.pending.delete(id);
+			pending.reject(new Error("Live editor disconnected while the command was running."));
+		}
+	}
+}
+
+/** Bind only to loopback. A taken port is an intentional memory-only mode. */
+export async function startLiveHub(port) {
+	let server;
+	try {
+		server = new WebSocketServer({ host: "127.0.0.1", port, path: "/live" });
+		await new Promise((resolve, reject) => {
+			server.once("listening", resolve);
+			server.once("error", reject);
+		});
+	} catch (error) {
+		if (server) server.close();
+		if (error?.code === "EADDRINUSE") return null;
+		throw error;
+	}
+	const hub = new LiveHub(server);
+	server.on("connection", (socket) => hub.accept(socket));
+	return hub;
+}

package/mcp/package.json

@@ -0,0 +1,24 @@
+{
+	"name": "@cozyclay/mcp",
+	"version": "0.1.0",
+	"private": true,
+	"description": "MCP server for CozyClay — block a scene, place the camera, and get film vocabulary and AI prompts back.",
+	"type": "module",
+	"bin": {
+		"cozyclay-mcp": "server.mjs"
+	},
+	"scripts": {
+		"start": "node server.mjs",
+		"verify": "node verify.mjs",
+		"verify:prompts": "node verify-prompts.mjs",
+		"verify:live": "node verify-live.mjs"
+	},
+	"engines": {
+		"node": ">=22"
+	},
+	"dependencies": {
+		"@modelcontextprotocol/sdk": "^1.30.0",
+		"ws": "^8.19.0",
+		"zod": "^3.25.0"
+	}
+}