@gtanczyk/diorama 0.0.0-stage → 0.2.0

package/docs/characters.md

@@ -0,0 +1,58 @@
+# Characters, style and icons
+
+## Characters
+
+`build` also makes the characters, after the places (steps `cast`, `model`, `sprites`; each also runs alone):
+
+- `diorama cast <slug>` paints the hero and every NPC (cast/<key>.png, with what they hold: a lantern or a net SAM's person leaves out stays on, gaps it leaves are filled, and what floats beside them as theirs, a heart or the bubble they are in, comes along) and makes their 2D walk: SAM finds arms, legs and boots to place the joints, then the whole painting is bent by those bones as one fine mesh, not cut into parts, so a coat, an apron or a skirt stretches with the legs and never tears (src/scene/warp.ts). The walk is drawn once into a sheet (cast/<key>/walk.png, 8 frames and the untouched painting for standing; joints in cast/<key>/rig.json) and measured: a walk that opens holes or loses a piece is marked in rig.json `check` and the game slides the painted figure as one piece instead. What the description says they hold or carry (one Spark call, only when the description says they hold something; the person's own list in cast/fixes.json `holds`, Studio agent: `cast_holds`) is found by SAM on its own, or as what stands out from the plain ground and touches them when SAM finds nothing, and kept on the cut-out and the walk however big it is: a slice of toast held over the head in both hands goes with the body and both arms carry it instead of swinging; cast/<key>/cut.json keeps the list and the share of the cut-out's height the body takes, so the game draws the body at its own height with the thing above it (src/scene/held.ts). `diorama rig <slug> [key...]` (Studio agent: `cast_rewalk`) makes the walk again from the paintings already there, with no Muse calls. A drawing SAM does not see as a person is asked as a character; when it still finds none, `cast_status` says why in `rigIssue` and `moves` says how the game moves them instead. Walk sheets whose frames left the pose are listed in `spriteIssues`; `sprites_redo` paints new takes. Spark also says how tall each one stands and whether upright like a person (cast/figures.json, a grown-up 1.62 m): a cat or a child is drawn at its own size, and a creature that is not upright (an animal, a round eye on tiny feet, a living teapot) gets no 2D rig or sprites: it moves as one body, hopping as it goes, squashing as it lands, leaning the way it goes, breathing and blinking (src/play/creature.ts), as its painted figure or, when it is in the sprites plan, as its own 3D model.
+- `diorama model <slug>` gives each character who needs sprites a rigged 3D model: Muse paints them from four sides, TRELLIS turns the views into a mesh and Make-It-Animatable adds a skeleton (both open Hugging Face Spaces; set `HUGGING_FACE_API_KEY` or `HF_TOKEN`, as they run on your account's GPU quota). The model only supplies poses. A creature (not upright) is painted from four sides as it is (no A-pose, no hands) and keeps TRELLIS's mesh with no skeleton (`model.json` `creature: true`): the game hops, squashes, leans and blinks the whole model, so a creature hero or NPC needs no sheets. Studio's Cast **Animate in 3D** (action `cast_animate`) builds a model for anyone the build skips, and the model step keeps building it. The figure is stood up straight in the file, and the poses start from the shape the skin was bound in (the reconstructed figure), not from whatever pose the rig saved its bones in (Make-It-Animatable's T-pose can sit a figure in a long coat down). What the character wears (a satchel, a backpack) is read off the front view and named in every other view's prompt with the side it shows on; Spark then checks all four together, and a view with it on the wrong side, or with anything standing on the ground at the feet, is painted again (up to 2 rounds; a wrong front repaints all four once). Things held in the hands are left out of the views: a suitcase put down beside the feet would be fused to a boot and swing with the leg in every sprite. A view the image safety filter refuses is asked once more in calm words (the looks left to the picture, the hands said plainly); refused again, the model made before stays, or the character walks with its 2D rig, and the build goes on.
+- Optional local Blender (3.6+, found on PATH, in /Applications, or at `DIORAMA_BLENDER`; on/off in Studio Setup, **3D animation**): after the rig, `src/model/blender/relax.py` puts the figure into a calm rest pose. Muse paints the reference with arms raised and feet wide, TRELLIS builds that pose, and the walk then had to turn each arm ~100° down from it, which pinched the shoulders, splayed the hands sideways and kept the legs straddled. Blender poses arms down by the sides with palms to the thighs and feet under the hips, smooths the bent shoulders and hips (Corrective Smooth) and makes that the rest pose (`model.json` `relaxed: true`, the rig as it came kept in `model.source.glb`). The original surface rides along as the `_REST` vertex attribute, so views fitted and painted in the original pose still land where they belong. `diorama model <slug> [key] --relax` (Studio agent: `cast_relax`) does it for existing models and draws their UV sprites again, with no calls. Without Blender nothing changes.
+- `diorama sprites <slug>` draws their walk and talk sheets (cast/<key>/sprites/): the model walks or gestures in a sheet of 8 frames, Muse draws the sheet again in the world's style, and each frame is cut out and lined up with its pose. A sheet that Muse handed back barely changed is drawn again over a different guide. A walk frame Muse painted out of the gait (a crouch, a sit: the painted figure under 88% of the sheet's usual height) is listed as `off` in sprites.json; the walk steps over it and standing still never shows it. Only what the story uses is drawn: the walk facings the hero's routes need, one talk sheet per speaker in the facing of their conversation, and the gesture Spark picks for each line (`--plan` prints the plan without drawing). Rendering the guides needs Playwright's Chromium (`diorama browsers`).
+
+The page draws the best a character has: sprite sheets, else the 2D walk, else the plain cut-out. The bare 3D model (TRELLIS's textures up close: a smeared face, melted hands) only supplies poses for the sprites; the page walks a person with it only with `?model=1` (a creature keeps its model).
+
+## Story stages and optional walking
+
+A character's progress is authored with story flags and conditional dialogue: waiting for an item, received it, then moved to another place. Walks are optional per character; a single event can move someone between connected places without a repeating walk. Model/sprites plan walking assets from these story behaviours, including for a character with no spoken lines. See [NPC story stages, walking and travel](npc-motion.md) for authoring, extension rules and verification.
+
+## Speaking faces
+
+Talking mouths are **off by default**: builds skip the faces step and characters speak with voice and gestures. Turn them on in Studio Setup ("Talking mouths") or with `DIORAMA_FACES=on`; a build then prepares the portrait's mouth shapes last, after every place plays. `diorama faces` run by hand works either way.
+
+`diorama faces <slug> [character...]` prepares reusable mouth shapes and subtle eyelid/brow movement for painted figures, their 2D rigs and visible talk-sprite poses. `build` runs `faces` after cast, sprites and voices. Studio's Cast card has **Prepare speech animation** / **Refresh speech animation**; the same job is `faces_prepare`. `faces_status` reports image/rig/sprite coverage, synchronized recording counts, skipped faces, failed preparation and the model's 3D facial capabilities.
+
+The configured vision model locates the head, lower face and eyes. The image model paints nine mouth shapes for each new visible view; vision validates their identity and articulation and calibrates the cells individually. Rejections identify bad frame indices. Up to three bad frames get individual mouth edits while neighbouring cells remain unchanged; wider or unlocalized failures get a full retry with the diagnostic feedback. The complete atlas is validated again before use. There is only one repair round: one image request normally, at most four including repairs, plus vision calls. When F/V (upper teeth on the lower lip, which the image model keeps painting as a grin) is the only frame still failing after its repair, the B mouth (teeth nearly together) stands in for it with no further image request, and the rest of the atlas is used. Static patches are rejected. The mouth/eye registrations are bound to the source PNG, sprite sheet or rig data; repaints disable stale registrations. A failed preparation retains the previous assets. Back-facing, obscured and unrecognizable faces stay as painted and are reported. An unchanged run reuses its assets and model-call cache.
+
+The actual lip corners in each source pose and the neutral atlas are registered separately. Each atlas frame is aligned separately by its lip centre and upper rim. An uncertain generated lip location gets one magnified lower-face inspection, mapped back into the original cell coordinates; confidence thresholds remain unchanged and unresolved frames retain the previous assets. Patches scale uniformly from that original mouth width and are masked to the lips; the nose, jaw outline and beard are preserved. Closed source poses use the original pixels at rest. A gesture painted with an open mouth (`sourceOpen`) first receives a narrowly masked neutral lip patch, so silence and M/B/P close its original cavity without moving the jaw silhouette. Uncalibrated mouths stay as painted. A single premultiplied crossfade avoids stacking two feathered faces. Painted figures use their original pixel coordinates. A 2D rig uses the actual painted silhouette to map the cropped figure into its padded canvas; baked walk cells follow the torso's rise/fall, and older part rigs keep the face on their torso. Each talk-sprite pose has its own registration. Mirroring, scale, lighting and occlusion are applied to the composed figure by the existing renderer.
+
+The final, already-processed voice files are analysed locally with [Rhubarb Lip Sync](https://github.com/DanielSWolf/rhubarb-lip-sync), cached by SHA-256. The first run finds Rhubarb on PATH or downloads its pinned official 1.14.0 release. Set `DIORAMA_RHUBARB=/path/to/rhubarb` or pass `--rhubarb PATH` to use an installed executable. FFmpeg is also required. English uses PocketSphinx; other languages, including Polish, use the approximate language-independent phonetic recognizer. `--audio-only` refreshes timelines without painting anything. Existing lip-sync is refreshed after Voices or a single voice retry. The recordings themselves are never modified.
+
+The Web Audio source drives mouth timing, independent of simulation speed. Loading, interruption, completion, mute and audio-context suspension return the relaxed mouth. Blinking has a per-character phase. `?lipsync=0` disables facial animation for comparison.
+
+3D rendering consumes the same performance state in two ways. Models with recognized authored facial morphs use ARKit-style jaw/blink/brow/funnel/pucker names and common VRM/Oculus visemes; unrelated morphs are left intact. Reconstructed models with projected painted views use independently calibrated front/left/right facial surfaces. Those canvases update the textures on the actual mesh, following its rotation, skinning, lighting and occlusion; they do not create a mouth cavity or change the jaw geometry. Each view owns its own painter, and unchanged frames avoid redundant GPU uploads. A back-facing head keeps its original paint. Native UV-textured models without facial morphs need facial rigging; the projected-surface fallback does not silently replace their material.
+
+`faces_status.model3d` reports the active method (`morphs` or `painted-surfaces`), its supported views and native morph count. Source hashes cover the model view and its fit settings. `faces_status.spritePoses` separately counts registered, obscured/skipped and pending talk poses. In-progress atlases live in `face-candidate.json` and content-hashed PNGs; a failed calibration keeps the previous live face, and retry resumes the candidate without paying for the same image again. Every atlas mouth is located individually in its full head, cropped at one common scale, then calibrated again in the final crop and aligned by its lip centre and upper edge. Crops extending beyond a generated cell use transparent padding so background rectangles cannot paint over a beard. Source lips are located with the surrounding face visible, then reviewed in close-up. Sprite poses are inspected separately in a cropped upper-body region. Coordinate-analysis resizes explicitly preserve the full input; uncertain lip boxes are saved as proposals, never activated automatically. Partial successful registrations survive a failed pose.
+
+The local review at `/prototype/conductor-lipsync` compares all four characters as an image, 2D rig, sprites or a 3D model in the actual game scene, including a model rotation control. Its close-up disables optional rim/separation and paint smoothing, hides the other hero when reviewing an NPC, and renders the selected 3D model at the enlarged display resolution for a sharp, smooth review. The normal game keeps its existing presentation. Mouth timing in Polish and the visual quality of newly generated faces still need review with the recordings.
+
+## Changing the style
+
+`diorama style <slug> "<style>"` (the Studio's Style button, the `style_set` action) paints a finished world in another style, say from a flat cartoon to a realistic oil painting. Each place is repainted from the painting it has, every fix in it, by one Muse edit that keeps the composition (camera, things, floors, doors where they are and as big), then cut out, measured and checked again from `masks` to `verify`, so the person's fixes (SAM phrases, boxes, brush strokes, floor cells, stand spots) keep lining up. Each character is repainted from their own figure, same face, clothes and pose, and their 3D model and sprite sheets are made again; the hero painted from a photo is painted from the photo again. Children's drawings go with each repaint as the reference, so the children still know their things. Sound and voices stay as they are.
+
+`--place KEY` restyles one place and `--cast KEY` (or `--cast all`) the characters: the style is kept as that place's (`corrections.json`) or character's (`cast/fixes.json`) own, over world.json's; restyling the whole world clears them. `--estimate` only says what it redoes and about what it costs, from the calls this world's own build sent (the Studio and the agent always ask with it first). `styled.json` keeps the style each place and character was last painted in, which is how `paint` and `cast` know a picture is to be repainted rather than painted anew. In the Studio it runs as one job, so History takes it all back at once.
+
+## Inventory icons
+
+`diorama icons <slug> [item...]` (a build step, after `plates`) makes `items/<key>.png` for each item in `story.json`, and nothing for anything else. An item the hero picks up in a place is cut out of that place's painting with the thing's SAM mask, so it costs no model call and looks exactly like what the player saw lying there. One that is too small there to blow up (under 56 px) is redrawn larger by Muse from the cut-out; one that only a person hands over is painted by Muse in the world's style, then cut out by SAM (or, when SAM finds nothing, as what stands out from Muse's plain ground; the shaded wash it stands on is peeled off the edge). `items/items.json` says how each was made. To change one, use the Studio's Cast tab or the agent: `icon_redo` takes a hint in your words, `cut` or `paint`, or another take, and keeps it in `items/fixes.json` for every later build (`icon_fix_clear` takes it back; `icons_status` lists them all).
+
+Speaking animation is an optional build stage: a reported face/audio preparation failure is saved as `faces.status: incomplete` with diagnostics, while other build steps continue. Explicit repair jobs still report failure. This does not mark missing voice recordings or unplayable places as ready. `face-preparation.json` tracks current per-character failures so successful targeted repairs clear their warnings without hiding other characters' failures.
+
+Atlas work is checkpointed before calibration in `cast/<key>/face-work/`. An accepted atlas never gets repainted merely because a later lip detector is uncertain; successful lip detections also survive retry. `faces_status.repairWork` lists source-bound work, stage and failed mouths. `faces_inspect` shows an exact head cell and reference, with a revision token. `face_frame_repair` edits only one inspected cell and revalidates the entire atlas; `face_lips_set` records inspected lip corners in an accepted head cell. Both resume preparation and leave the live face unchanged until the remaining checks pass.
+
+For later calibration stages, `face_calibration_inspect` lists pending source poses or final mouth patches and shows their exact close-up. `face_calibration_set` stores a reviewed normalized lip box and visible-cavity flag against both the candidate revision and inspected pixel hash. These corrections cannot be reused on a repaint or another crop. They do not bypass identity/articulation checks. The agent must inspect the image and cannot invent coordinates when the mouth is ambiguous. New tool definitions require restarting an already-running Studio server.
+
+When the source painting changes, the pipeline retains but stops using older 2D rigs, models and sprite sheets until they are rebuilt. Sprite sheets also track the model they were rendered from. Legacy animation is bound to its current source before a cast or model rebuild, rather than being deleted. Studio reports stale animation as needing a rebuild.
+
+If a generated 2D walk tears, one separate neutral pose is proposed and checked for character identity and held objects before another rig is attempted. The original cast image is not overwritten. The retry is accepted only if its walk passes the same tear check; otherwise the cut-out fallback remains. The neutral-pose and visual checks use model calls, and visual playtesting remains necessary.
+
+New direct UV sheets use 450 × 672 pixel cells (1.5× the former dimensions in each axis) and skip the painterly post-filter to retain texture detail. The larger sheets require more memory; existing sheets remain unchanged until rebuilt.

package/docs/npc-motion.md

@@ -0,0 +1,69 @@
+# NPC story stages, walking and travel
+
+Movement is part of `story.json`. Existing stories keep their stationary NPCs until movement is authored. The story designer can add purposeful patrols; Studio's `story_redo` can request patrols and moves triggered by puzzles or dialogue choices. The Story view displays patrols and travel effects.
+
+## Simple story stages
+
+A character needs only the progress the story uses. For example, a gardener can wait for a key, receive it, then go to the garden. Use ordinary flags with the NPC's key as a prefix and `when` conditions:
+
+| Stage | Dialogue condition | Event that advances it |
+| --- | --- | --- |
+| Waiting for the key | `["!gardener-key-received"]` | Giving the key sets `gardener-key-received` |
+| Key received, still inside | `["gardener-key-received", "!gardener-outside"]` | A goodbye choice sets `gardener-outside` and moves the gardener |
+| In the garden | `["gardener-outside"]`, dialogue in `garden` | The same NPC thanks the hero there |
+
+The event's effects do the work. For example, giving the key can use `[{"take":"garden-key"},{"set":"gardener-key-received"}]`; the departure choice can use `[{"set":"gardener-outside"},{"moveNpc":{"npc":"gardener","to":"garden"}}]`. The key needs an item definition and a reachable way to obtain it. Both places and their connecting exit must already exist. Earlier dialogue branches stop matching after the event, so the NPC no longer asks for a key it has received. List specific branches before a fallback because the first matching branch wins. Use `once` for a one-time object interaction when appropriate.
+
+Flags describe story progress. `state.npcs` records relocated NPCs' actual logical place; `at:place` tests where the player is. Setting a flag alone does not move anyone. Stages can change dialogue while the character stays in one spot. Each character's walking is optional, with no minimum number of moving NPCs and no separate goals or memory system to configure.
+
+The authoring rules are shared by the new-story designer, extension designer and Studio agent in `app/src/story/npc-guidance.ts`. They describe the existing flags/effects schema; there is no additional NPC-stage generation step.
+
+## Studio and rebuilds
+
+Ask, for example: “The gardener waits for my key, thanks me after receiving it, and goes to the garden when I say goodbye. Keep them still until then.” The agent reads `story_status`, then uses `story_redo` for those behavioural changes. It uses `story_text` for wording alone. This rewrites the story and uses the normal model calls. `npc_move` in the place editor changes a standing position; it does not author a story event. `cast_animate` prepares animation assets; it does not give the character a walking route.
+
+After the story job finishes, inspect its checker issues, conditional dialogue and effects. Prepare missing animation with model/sprites for newly walking characters, voices/faces for new spoken lines, and plates for visual object changes. Existing animation can be reused. Reload the game after successful jobs, then run `playthrough` and inspect the changed stages: old requests disappear, the event fires, and the NPC is available in the destination. Inspect a return trip too when authored. A place-motion sample or a passing build alone does not establish that the full sequence works.
+
+For new worlds, the build reads the story before planning models, then uses derived scene geometry for sprites. For extensions, `world_extend` retains existing NPC dialogue, stages, flag names and patrol order, and adds optional stages/walks only for new NPCs. A new travelling NPC is declared only at its original place; its new destination dialogue can name that key. New effects must not change existing NPCs, flags or inventory. A request to change an existing NPC's progression belongs to `story_redo`, rather than an additions-only extension. The combined story is checked before the staged extension is installed.
+
+## Walking within a place
+
+Add an optional `patrols` array:
+
+```json
+{
+  "patrols": [
+    {
+      "npc": "gardener",
+      "place": "courtyard",
+      "via": ["flower-bed", "well"],
+      "when": ["!garden-ready"],
+      "pause": 2
+    }
+  ]
+}
+```
+
+`via` names two to eight fixed things in that place, with at least two different keys. The NPC walks to their approach spots in order, then repeats. `pause` is seconds at each stop (0.5–60, default 2). `when` uses the story's usual facts. The first matching patrol wins. An inactive patrol stops its walker. No patrol means the person stays put.
+
+NPCs share the player's floor grid, obstacle routing, stairs, depth scaling and distance-driven walk cycle. Clicking a moving person stops them while the hero approaches; they remain still for their own speech and conversation, then resume. Hit testing and interaction distance follow their visible body. Positions are retained for the current game session when the hero changes places. Off-screen patrols pause; this does not depend on wall-clock time or introduce timed puzzles.
+
+## Leaving after an event
+
+An interaction's `do`, or a dialogue choice's `do`, can contain:
+
+```json
+{"moveNpc": {"npc": "gardener", "to": "garden"}}
+```
+
+The destination must be a neighbour of the NPC's current place, connected by an existing world exit. Declare the character in exactly one original place in `world.json`; do not add a second character in the destination. Destination dialogues and interactions may address the same key. The checker validates their availability and all endings with the NPC's changing location, including return trips. Use flags and `once` when an event must not repeat.
+
+After the event's lines have finished, the NPC walks to the connecting exit. In the destination it enters at the exit leading back, then walks to its first patrol stop, original standing spot when returning home, or the scene start. A conversation still in progress holds its departure. The source hotspot disappears with the body, and only one clickable instance exists at the destination. If the hero leaves first, the departure completes off screen. Events affecting unseen NPCs work the same way. NPC travel uses world connections; the hero's inventory-based exit locks do not apply to NPCs.
+
+The pure story state stores each relocated NPC's place under `state.npcs`, so story search and serialized states preserve its location. Visual walking is reconciled with that logical destination after speech. An unreachable physical exit produces a browser warning and holds the NPC in place; it does not teleport through an obstacle. Patrols skip unreachable stops after their pause. Check the actual scene's walk grid and play through authored movement.
+
+For structured model output, the travel effect is `{"op":"moveNpc","key":"gardener","to":"garden"}`. Other effects retain their existing `{op,key}` representation.
+
+## Animation assets
+
+`diorama sprites <slug> --plan` includes NPCs with patrols or travel effects, even if they have no spoken lines, and plans their walking facings. Rebuild their model/sprites to obtain full walking animation. Until then, an older NPC with only talk sheets uses its existing 2D rig, model or cut-out while walking. No paid generation is required to use an already available rig or cut-out.

package/docs/scenes.md

@@ -0,0 +1,75 @@
+# Places: depth, weather and the play page
+
+## Plan and painting
+
+Painting includes a numeric composition contract: object boxes, floor polygons and camera horizon. After ordinary region fixes or restyling, the configured vision planner compares the final painting with the block-in. It locates each non-NPC object and checks planned floor coverage, visible connections, frame-edge routes and approaches to items/exits. Code checks completeness and measured displacement/size, rather than accepting an empty list of aesthetic complaints.
+
+A mismatch gets at most two full-frame Muse edits using the original plan and measured problems. Every candidate is checked again. `painting-layout.json` keeps observations and reasons, bound to the exact plan, place and accepted painting. Rejected candidates do not replace the previous `painting.png`; masks refuses a failed or stale report, and current status cannot show an old green verdict. After two rejected repairs, `place_status.paintingLayout` exposes the diagnosis to the agent: `paint_retry` advances the take instead of replaying rejected memo output, or `layout_fix` can simplify an observed rejected composition even when an older painting exists. Legacy worlds are checked when rebuilt from paint; photo worlds retain their photo-based composition flow.
+
+Walk geometry still comes from the accepted painting, not the rough plan. The build no longer invents boardwalks to snapped interaction points and immediately brushes them into the floor. Existing small-gap repairs on observed surfaces remain available. These checks depend on vision observations and are not a substitute for looking at the finished scene.
+
+## Depth: parallax and moving water
+
+`diorama backdrop <slug>` (a build step, after `derive`) turns each place into a little stage, so it moves like a real diorama:
+
+- **Backdrop and cards.** One Muse edit paints the place without its standing things; only their masks are taken from it, and SAM checks each thing is really gone (a leftover gets a smeared fill). The things become cut-out cards. In the page the floor shears with the camera, each card slides with the floor row it stands on, and the upright band behind the walkable floor (a quay wall, a railing) slides as one piece so it does not lean. The hero slides with the floor under his feet and still hides behind cards.
+- **Depth occlusion.** Cards hide people only behind the things the story names. Everything else in the painting (a lamp post, a railing, a bridge's parapet) hides them by the depth map (`app/src/play/depth-occlusion.ts`): a pixel nearer than the feet by 12%+ of their floor row, whose column meets the floor in front of the feet, is put back over the person from the frame as drawn without them. A wall right behind someone (which the depth model reads a little near) never hides them; stairs from `steps` are a raised floor; more than 75% of a person hidden by it means the depth is unsure there, and it hides nothing. `verify` reports `NO_DEPTH` (no depth map, so unnamed things hide no one), `WALK_ON_FACE` (walk cells on an upright face such as a bridge's side) and `DEPTH_UNSURE`; the explorer counts a visible hero over a nearer painting as `drawn-over` (worst `depth`).
+- **No paths up a wall.** When derive joins floors that do not touch, it opens the cheapest corridor of cells between them; that corridor first may not cross an upright surface (the side of a bridge, the wall down to an underpass), told from the depth model before the scale is known (`uprightTest` in `app/src/scene/depthmap.ts`: on a floor the depth changes steadily down the screen, on a wall it hardly does). Only a floor no other path reaches is joined across a wall as before, and derive flags it ("no floor joins … but up a wall"). Cells on such a face, joined or laid by a floor mask, are flagged `WALK_ON_FACE` by verify with their box: the fix is a floor or stair between the two (`floor_cells`, a floor brush), not the wall. A corridor along a one-cell fringe that can't be widened is laid again opening every cell it takes, instead of looping.
+- **Whole cards.** A thing that stands nearer slides more than the wall behind it; that is the diorama. So a card must be the whole thing: SAM by name often takes a container and leaves what is in or on it (a bucket, not the fish heads over its rim), and that part would stay on the backdrop and slide apart. `masks` asks SAM once more by a box around the whole thing (the mask's box joined with Spark's, grown up and sideways) and joins the parts it finds attached to the mask; parts that reach the box's edge are asked by a bigger box. `backdrop` then checks every card's outline: where it cuts through something Muse took out with the thing, the card is flagged `CARD_CUT` (in `backdrop.json` and the verify flags), and the Studio agent fixes it with `mask_fix` and a box around the whole thing.
+- **Clean cut-outs.** SAM 3.1 is asked plainly; three ways its raw answers go wrong are fixed in `masks` without another call. A doorway, arch or gate (a way out set in a wall) often comes back as its frame, posts and lintel with the middle empty: the way out is the opening the frame holds, row by row between the posts, down to the floor (`openingOf`), not a colour cut of the fog behind it. SAM by name can take a sliver of a thing (the boards of a shelf without its frame and tins): the whole-thing box may then grow it up to what Spark boxed, and what it found past Spark's box (the wall above) is left out. A floor is cut around everything lying on it (shavings, pebbles, leaves) and frays into fog: gaps of a few px close and holes it closes on every side up to 1.5% of the frame are filled (`solidFloor`); bigger holes (under a wide table, a pond) stay, and standing things are cut out of the walk by their own masks.
+- **Far flats.** Spark names the bands of scenery above the horizon, farthest first (hills, a town, a castle); SAM cuts each one from the backdrop into `layers/<n>.png`, over a sky with the flats filled in (`layers/sky.png`). Each flat slides a little more than the one behind it.
+- **Water.** SAM finds water on the backdrop (minus the floors and cards, which come out wet otherwise) into `layers/water.jpg` and `layers/water-mask.png`. Spark reads which way it runs on screen, along the channel and into the depth too; that is `water.flow` in `layers.json`, a unit vector downstream, `[0, 0]` for still water like a harbour basin or the sea, which the page sways in slow swells instead of running. Short light glints come and go on every water, drifting with it, so even a dark smooth sea is seen to move. If Spark got the river backwards, tell the Studio agent ("the canal flows toward the bridge"): its `water_flow` action keeps `reverse`, `still` or a hint in words in `places/<key>/depth.json`, which every later build follows.
+- **Painted fog.** SAM finds fog, mist or haze painted on the backdrop (minus the cards; skipped when it is more than half the frame, which is the weather's mist) (when SAM sees no fog, which is often, Spark boxes the fog bands and the fog in them is read from the pixels: pale and greyed) into `layers/mist.png`, a veil of the fog's own colour shaped like its denser and lighter patches, and `layers/mist-mask.png`, where it may show. The page slides two copies of the veil sideways with the wind, cross-fading half a cycle apart, so wisps roll across the land while the land itself stays put; over walkable floors the veil is thin. To change it, tell the Studio agent ("the fog should drift left", "slower", "leave the fog still"): its `fog_drift` action keeps `fog` (auto, left, right, still) and `fogSpeed` in `places/<key>/depth.json` and reruns the backdrop step.
+
+- **Living lights.** The lights Spark placed in the layout (`layout.lights`) come alive in the page, by the words of their names (`src/play/glow.ts`): a window's glow breathes, a lantern, candle or fire flickers, far town lights twinkle, and a lighthouse or beacon turns its beam round, swinging left and right and flaring when it points at us. They sit where the lights were painted, not on Spark's planned points: Muse paints a lamp where the picture wants it (a lighthouse lamp on its tower, well above the point the plan gave). `derive` (`src/scene/glows.ts`) moves each warm light to the strongest bright, warm spot near its point, a lighthouse lamp to the strongest high one anywhere in the upper half when none is near; a light with no painted spot keeps its point only if that is on a painted thing, else it gets no glow. The moved points are `geometry.glows`; a place derived before this keeps the planned points until `diorama derive <slug>` runs again (free: the model answers are remembered). Only light is added over the painting, never covering it. `?glow=0` keeps them as painted (capture mode too, unless `?glow=1`).
+
+`--smear` skips Muse and smears the gaps instead. Places with a `photo.png` stay flat.
+
+- **Separation.** Each person throws a soft shade on what stands just behind them: their silhouette blurred and pushed a little away from the light, only on pixels the depth map (from `derive`; without one, the floor plane with each card on its floor row) puts a short way past their feet. So a hero passing a painted plinth reads as standing in front of it, not pasted on it. `?separate=0` turns it off, `?separate=1.5` darker.
+- **Rim light.** Each person gets a thin warm edge on the side the light comes from (against the way shadows fall; a place with no measured light gets none), as illustrators separate a figure from a busy background. `?rim=0` off, `?rim=1.5` wider.
+- **Focus** (off by default). `?focus=1` softens and pales what is behind the hero with its depth, like a close-up camera. It only separates far things, so it stays opt-in.
+
+## Weather: rain, snow and drips
+
+`diorama weather <slug>` (a build step, after `backdrop`) reads what falls on each place and where water drips, into `places/<key>/weather.json`; the page animates it all, so nothing in the painting is a frozen streak:
+
+- **Fall.** Spark looks at the painting and the place's mood: `rain`, `snow` or `none` (indoors, a cave, wet ground after the rain), how hard (0.3 drizzle, 1 a downpour) and how much the wind slants it. The page drops rain or snow over the whole frame in three depths; each drop ends on the floor row as far away as it is.
+- **Rings and splashes.** SAM finds the water and the puddles (`weather/ripples.png`, minus the floors and cards SAM takes for water). Drops that land there ring; drops that land on a walkable floor splash; on a roof or the sky nothing shows.
+- **Drips.** Spark names up to four edges that drip (eaves, a gutter, an awning, a pipe, a cave ceiling), SAM finds each, and its lowest edge is where drops swell and let go. Each falls under gravity, at the scale of the place, to the floor row under it (the depth map, else the first walkable cell below) and splashes or rings there. A dry place can still drip.
+- **Painting.** When a place's words say rain or snow, the paint step asks Muse for the weather's marks only (wet sheen, puddles, snow lying), no streaks in the air.
+
+To change it, tell the Studio agent ("no rain in the hut", "it should pour", "water drips from the tavern sign"): its `weather_set` action keeps `fall`, `strength`, `fog` (mist drifting across the place, 0..1, which Spark also reads from the picture and the mood) and a hint in words in `places/<key>/weather-wish.json`, which every later build follows. `?weather=0` in the play page keeps every place dry.
+
+## In the play page
+
+- **Walking:** `derive` joins the floors up, but never across painted water (SAM finds it only when a join would cross it), and the game starts on the biggest stretch of floor. The hero routes over the walk grid; the walk phase follows the distance walked, so feet don't slide. Things in front of a person (per-column floor lines from `derive`) are drawn back over them.
+- **Stairs:** `derive` asks SAM for the steps of any floor or walk-on thing named as stairs (stair, step, schody) and keeps each step's band per column (`steps` in geometry.json). A person on them keeps their size from where they stand, but their feet sit on the tread under them and climb only over each riser; a walk never ends on a riser. Fewer than two clear steps keep the stairs a ramp.
+- **Light and shadow:** `derive` measures the painting's light from the shadows things cast (`light` in geometry.json) and masks those shadows (`masks/shadows.png`). A person then casts a matching soft shadow and darkens inside a painted one. A shadow many times its thing's size (stair risers, a tree's shade) is not measured. When Spark sees sun but no thing's own shadow can be measured, the light comes from Spark and nothing is masked. A place with no sun (fog, overcast) gets neither. Each person has one shadow: under measured light the cast shadow replaces the soft blob at the feet (only a thin contact under the soles stays); in shade, fog or without light the blob is the shadow.
+- **Shadows on the ground's shape:** `derive` also writes `depth.png` (`depth` in geometry.json): Depth Anything V2 Small (ONNX on the CPU, about a second per place; the 27 MB model is fetched once into `.diorama/models/`) gives relative depth, and a robust line through the walk cells pins it to the floor the scale already knows, as the floor row at the same distance. The runtime turns each ground pixel into a 3D point and asks whether the person stands between it and the sun (`app/src/play/shadow.ts`), so a shadow breaks over steps, climbs a wall and drops off a quay. It skips what the painting already shades and fades to almost nothing on water. When the depth fits less than 35% of the walk cells (or the model can't run) there is no `depth.png`, the place is flagged `depth: ...`, and shadows lie flat as before.
+- **Fires and lamps:** where the layout has lights, `derive` asks Spark whether one of them (not the sky) lights the ground, and whether it is a living flame; it is stored as `light.at` (and `light.flame`). The layout often names a light by its glow ("lamp-glow", "oven-fire") rather than its thing; such a light goes to the thing near its point that shares a word with it, else the one whose mask holds the point, else the smallest box around it. Each person's shadow then runs straight away from it, fading a few heights away. With the depth map the painted things cast shadows from it too (a bench toward the viewer, a tent away from the fire): each floor point looks back along its ray to the light and is shaded where the ray passes behind a thing. A person's shadow that reaches a thing standing behind them (a tent wall) lies on it. A flame flickers: it sways about 10 cm and its light breathes, so every shadow wavers with it, its bright pixels dance, a warm glow pulses round it and a few sparks rise.
+- **Inventory:** the items the hero carries sit in a paper bag at the bottom, each as its painted icon (step `icons`, below). Click one to take it in hand: it becomes the cursor, with "Use X on Y" over a thing; right click or Esc puts it back. Tried where it does not help, the hero says the item's own line (`nope` in story.json, written with the story). A picked-up item flies from where it lay in the painting into its slot, and the painting shows the spot empty (the `plates` step: Muse removes the thing, and only its mask, grown by any part Muse removed with it that SAM left out, such as a rope tied to it, is patched in, matched in tone; of several takes the one with the least seam and no stain is kept, and `plates.json` gives both).
+- **Touch and keyboard:** on a phone or tablet a tap walks or uses, a tap on a slot takes the item in hand and another puts it back, and a long press looks at a thing (or puts the item back). WASD or the arrows walk the hero and slide along whatever is in the way; the nearest thing in reach lights up and E, Space or Enter uses it; 1 to 9 take an item from the bag.
+- **Gamepad:** any pad works without keys or a mouse, so a small child can play. The stick or d-pad walks, A uses the lit thing (in a conversation it picks the lit answer, at the end it plays again), B puts the item in hand back or looks, X, Y and the shoulder buttons go through the bag one item at a time and back to empty hands. The 8BitDo Micro is read as a standard pad in its D-input mode and in the Mac's MFi profile (buttons by face position, the cross from the hat or the first axes), as the gamedev.pl site does. Other pads the browser does not map work roughly: the two lowest buttons use, the rest go through the bag. Several pads all steer the same hero, or with `?pads=turns` they take turns: a badge in the player's colour shows whose turn it is, and the turn passes to the next pad after a conversation ends or after 60 seconds of play (`&turn=90` for more), never in the middle of a line. With `?pads=party` every pad gets its own hero on screen, the hero's figure dressed and edged in the player's colour (clothes recoloured as the page loads; skin and head keep theirs), on a ring and under a number in that colour: each walks on its own, the bag and the story are shared, and whoever uses or looks at something becomes the hero the story follows (only that player answers in a conversation). When each child draws their own hero, mark the drawings "Another hero (one per gamepad)" in the Studio workshop (kind `player`): `cast` paints and rigs each one, and pad 2 plays the first of them, pad 3 the next (in a party each walks as drawn; in turns the hero turns into whoever plays). They join no place and say nothing in the story. A pad press does not unlock sound in every browser; while sound is held back the page asks for one click. In Studio, Play has a Gamepads choice (one hero, take turns, a hero each).
+- **Talking:** each line knows who says it; the paper bubble sits over the speaker with their name on a tag, and the speaker plays the line's gesture while the other listens. The hero's answers are numbered cards beside his portrait; keys 1 to 4 pick them too.
+- **URL switches:** `place=<key>` start there, `spawn=x,y,left|right` put the hero, `capture=1` fixed 1/60 s steps for recording (flat, no parallax), `parallax=N` parallax strength (`0` flat, default 1 with the mouse), `track=N` a closer camera that pans after the hero, for walks along a place, `sway=1` a slow idle drift for films, `separate=N` the shade a person throws on what is just behind them (`0` off, default 1; off in capture runs unless given), `rim=N` the light edge on each person (`0` off, default 1; off in capture runs unless given), `focus=N` soften what is far behind the hero (default 0), `still` water that does not flow, `grid=1` show the walk grid and occluders, `model=1|0` prefer or skip the 3D figure, `sound=0` silent, `cursor=0` hide the cursor. `window.diorama` drives the page from scripts: `walkTo`, `use`, `act`, `play(steps)`, `wait`, `on(event)`.
+
+## Checking a world
+
+`diorama build` ends with `verify`, which writes `verify.json` per place and prints which places are playable and what was flagged on the way. Places are built side by side and independently, and no place waits for another: each goes through its own steps and is told playable (`place.verdict`) as soon as its verify and free repairs are done, while the rest are still being made. The world's own steps start as soon as what they read is there (`app/src/world/schedule.ts`): the cast is painted from world.json while the places are, voices once the story is, an item's icon once its place is cut out, sprites once every place has its walking routes, sound once every place is painted. A place whose step breaks is set aside with why, and the others go on to the end, characters and sound included. Each model has a cap on requests open at once so the overlap does not run into rate limits (`DIORAMA_LANES="muse=8,spark=12,sam=16"`, the defaults). The report gives each error its playbook (`app/src/scene/playbook.ts`): where to look, the fix tools to try in order, and the costly last resort (a camera too close, `SCALE_NEAR` or a floor cut off once derive shut the front rows for it, is rerun derive first, then relayout and repaint); `place_status` hands the same to the Studio agent, which repairs one place at a time with the fix tools. The `masks` step also checks that each mask holds its thing: one Spark look per place at the painting with every mask box numbered (`app/src/scene/maskcheck.ts`), since a box and its mask can agree and both sit on the neighbour. When Spark points to where the thing really is, that is `MASK_WRONG` in verify (an error for any thing, props too). When it finds the thing nowhere and what it sees in the box is another kind of thing (hay where a horse should be), the thing was never painted: `THING_NOT_PAINTED`, an error whose play is to paint it in. A doubt about details only (a bronze bell for a brass one) is the flag `MASK_DOUBT`. Spark is asked to judge the kind of thing, not its details, and a box someone fixed is taken as given. Before anyone is asked, the build makes the cheap fixes itself (`app/src/scene/autofix.ts`): a gap of up to three 60 px cells with nothing solid in it, or up to six along stairs or a gangway (a walk-on thing) whose mask reaches both floors, is bridged with a floor brush stroke (two passes, as one bridge can bring the next patch within reach; stairs that lead somewhere else, with water between the floors, bridge nothing: that place needs a floor in the layout or a repaint, so it goes to the agent), a stand spot off the floor moves to the nearest reachable point (both without a model call), and a `MASK_WRONG` thing gets the box Spark saw it in (one SAM call). A card flagged `CARD_CUT` gets the cut part brushed onto its mask (a lantern on a boat's gunwale), so the card carries it and the backdrop is made again without it (once per cut). A `THING_NOT_PAINTED` thing left after them gets one Muse edit at its layout box (once per thing, kept as a repaint correction) and the place is rebuilt from paint. They are ordinary corrections, so `corrections_get` lists them and `corrections_clear` takes them back; `build.json` keeps them under `repairs`, and `--repair off` skips them. What the free fixes leave, the Studio hands to the agent at once: a build that ends with places unplayable wakes it with the places and their errors (`budget_set autoRepair:false` turns that off). It repairs with the fix tools in the playbook's order; after three fixes of a place in a row that leave it unplayable the fix tools refuse it and it tells you what it tried, which is your turn. `place_view` is how the agent looks, as a picture it sees (layer `walk`, `masks` or `painting`, all on the 60 px grid), so it has no need to dig through files; after three shell calls in a turn it is reminded of that once. Layer `masks` tints each thing's mask against the box it should be in and flags a mask that cut out a neighbour. Layer `walk` shows why a place cannot be walked: reached floor green, unreached red, each gap as a dashed line naming the 60 px cells to bridge (the ones `floor_cells` takes) and the things in it, and stand spots off the floor (`places/<place>/diagnose.png`; the agent gets it as an image). Every step also leaves a picture to look at: `masks/overlay.png` (what SAM found) and `walk.png` (where the hero can walk, footprints, floor lines and his size at a few depths).
+
+`diorama explore <slug>` puts the hero on every part of each floor in the real play page and checks each frame; it needs Playwright with Chromium (`diorama browsers` downloads it once). `--fix` closes the floor where he looked wrong.
+
+
+## Scene reactions
+
+The `reactions` build step runs after weather and waits for the story. The configured Spark planner reads the painting, floor descriptions, geometry, weather, world style and available story interactions. It chooses restrained dust, splashes, snow, leaves or sparks only where the material and action support them; clean surfaces can have no effects. No model or audio is needed while playing.
+
+`reactions-plan.json` keeps the planner's choices. The compiler binds floor keys and optional local polygons to reachable cells of the actual walk/zone grid; it rejects unknown floors, empty areas, overlapping step recipes, unknown conditions and unreachable interaction targets. `reactions.json` is the validated runtime data. A fingerprint disables stale reactions after the scene, story or author preference changes. Rerun `reactions <slug> [place...]` to update an older world; ordinary builds and restyles include the step automatically.
+
+- **Step:** fires at half-cycle crossings during actual movement, including the last step before stopping. Uses the local material, depth scale and stair height. The foot position is approximate for sprite sheets.
+- **Enter:** fires when a moving actor crosses into the region, including a small patch crossed within one frame. Spawning or standing inside does not fire. Optional story conditions, cooldowns and once-per-play-session entry prevent repetition.
+- **Interaction:** fires for the exact matched use/take story action with the hero in reach. Looking, an unmatched action, an old action after a story edit, and a different refusal branch cannot trigger that recipe. This is a reaction at the object, not a new hand-contact animation.
+
+Particles use seeded randomness, a shared limit of 96, short lifetimes, the scene camera and object occlusion. They clear on place entry; reduced-motion preference suppresses particles. `?reactions=0` disables them for an A/B comparison. Export embeds the validated rules and bundles the renderer without new image assets.
+
+The Studio agent has `reactions_status` and `reactions_set` (a natural-language hint, saved in `reactions-wish.json` across rebuilds). After changing reactions it reloads Play. Verify event-driven motion by walking through the region or using the object; an idle screenshot cannot demonstrate it.

package/docs/setup.md

@@ -0,0 +1,60 @@
+# Setup and sharing
+
+## Setup
+
+Node 22 or newer and a Meta Model API key:
+
+Without a checkout, from npm. Run it in an empty folder: worlds are written to `./worlds` there.
+
+```sh
+mkdir my-worlds && cd my-worlds
+npx -y @gtanczyk/diorama setup     # what this machine still needs; asks for keys, hidden
+npx -y @gtanczyk/diorama doctor    # one live call to each model
+npx -y @gtanczyk/diorama studio    # the GUI
+```
+
+Or install it once, and the command is just `diorama`:
+
+```sh
+npm install -g @gtanczyk/diorama
+diorama setup && diorama studio
+diorama browsers                   # Playwright's Chromium, for explore, playthrough and sprites (once)
+```
+
+`diorama setup` keeps keys encrypted (AES-256-GCM) under `~/.config/diorama/`, with the encryption key in the macOS Keychain or the Linux Secret Service. Environment variables (`META_API_KEY`, `ELEVENLABS_API_KEY`, `HF_TOKEN`) still work and win.
+
+From a checkout, `npm install` once, then the common commands are npm scripts (arguments go after `--`):
+
+```sh
+npm run setup
+npm run doctor
+npm run doctor -- --now        # before building in front of people: is the network and Meta fast enough today
+npm run studio                 # the GUI
+npm run build -- <slug> "<idea>"
+npm run play -- <slug>
+npm run pages -- <slug>        # diorama publish (npm keeps the script name publish for itself)
+npm run export -- <slug> --zip
+npm run browsers               # Playwright's Chromium, for explore, playthrough and sprites
+npm run diorama -- <command>   # any other command, e.g. npm run diorama -- masks <slug>
+npm run check                  # typecheck and tests
+```
+
+Model answers are kept by a hash of their inputs under each world's memo folder, so rerunning a step with the same inputs is free.
+
+### What it costs
+
+Every call sent is logged with its price in `worlds/<slug>/memo/calls.jsonl` (Meta list prices: Muse Image $0.01 an image, SAM 3.1 $0.0025 an image, Muse Spark by tokens). Studio Setup shows the total, today's spend and, once you type the balance your Meta billing page shows, the credits left; the banner warns at $5.
+
+- `DIORAMA_SPARK_TIER`: `auto` (default) sends Spark to Meta's contributor tier (same model, about 1/15 the price, Meta may train on the prompt and answer), except calls that carry your own photos or drawings, which stay `standard`. Set `standard` or `contributor` to force one.
+- Studio Setup has two pickers, Thinking (Spark) and Painting, kept in `~/.config/diorama/models.json`. The environment variables below win over them.
+- Studio Setup → Audio → Audio quality selects Final (the default) or Draft for future sound and voice generation in all worlds. It is saved in `~/.config/diorama/sound.json` and also used by CLI commands. `DIORAMA_SOUND` overrides the saved choice; `--final` / `--draft` override both for a command. Existing recordings need Voices and Sound rerun; changing quality does not rewrite dialogue.
+- `DIORAMA_SPARK_VIA`: which model answers Spark's asks. `meta` (default); `claude` (Anthropic API, `DIORAMA_ANTHROPIC_API_KEY`, model `DIORAMA_CLAUDE_MODEL`, default `claude-sonnet-5-5`); `openrouter:<model id>` (any chat model, `OPENROUTER_API_KEY`).
+- `DIORAMA_IMAGE_VIA`: which model paints. `meta` (Muse Image, default); `gemini-flash`, `gemini-lite`, `nano-banana`, `gemini-pro` (Gemini image models: straight from Google with `GEMINI_API_KEY` or `GOOGLE_API_KEY`, otherwise through OpenRouter), or any OpenRouter image model id. Pictures come back at Muse's size, so every later step works the same. SAM 3.1 always cuts.
+- Answers from another model are kept under their own memo keys and logged with `via`; Setup counts them apart from Meta's bill. Switching back to Meta finds the Meta answers again. A world meant for the hackathon demo stays on Meta (the default).
+
+## Playing and sharing a world
+
+- `diorama play <slug>` serves it on localhost.
+- `diorama publish <slug>` puts it on this repo's GitHub Pages (the `gh-pages` branch, through a worktree under `.diorama/`, so your checkout stays put) and prints a link like `https://gtanczyk.github.io/diorama/<slug>/`. Every published world is listed at `https://gtanczyk.github.io/diorama/`. The first time, turn on Settings > Pages > Deploy from a branch > `gh-pages`. Run it from a fork to publish to your own Pages.
+- `diorama export <slug> --zip` writes `exports/<slug>/` and `exports/<slug>.zip`, a static copy any web host serves; the zip uploads to itch.io as an HTML5 game as is.
+- In the Studio, the top bar's ☰ menu shares a game both ways: Make a zip (then Download it) and Publish a link (it asks first, since anyone can play it there). The agent has them as `world_export` and `world_publish`.

package/docs/sound.md

@@ -0,0 +1,23 @@
+# Sound and voices
+
+`diorama sound <slug>` (also a build step) has Spark plan what the world sounds like from its places, the brief of each painting and the story: a music bed, the ambience of every place, loops on things that sound (a lantern flame, a flag in the wind, a lamp that only roars once the story lights it), the surface of every floor, and one-shots for taking, using, exits, talking and the ending. The plan is `sound.json`; edit it and run the step again to regenerate only what changed.
+
+The Meta Model API has no audio model, so effects come from ElevenLabs (`ELEVENLABS_API_KEY`) and music from Meta's open-weights MusicGen run on your machine (`DIORAMA_PYTHON=/path/to/python` with `torch transformers scipy`; without it, or with `--music eleven`, ElevenLabs makes the music). Code checks every take: a loop that came back as one event is asked for again, loops are crossfaded into themselves so they never click, footstep takes are cut into single steps, and everything is levelled. Needs `ffmpeg`. Without a key the world plays silent.
+
+`diorama voices <slug>` (also a build step) speaks the conversations, the intro and the endings (`--all` also voices the hero's remarks on things; by default those stay in the bubble so the game does not talk at every click). Spark looks at each figure and says who speaks (gender, age, how they sound). Adults get the fitting stock ElevenLabs voice; children, teens and the old get a voice made for them by **ElevenLabs Voice Design**, since no stock voice is a child or truly old (`voices.json`; a designed voice is created on the account only while its lines are read and deleted after, so a world needs one free voice slot; Studio's Cast card or `voice_set` recasts), ElevenLabs reads each line, and **Muse Voice Transcribe** listens to every take: a take that does not say its line is read again, up to three times, and rejected if it still does not.
+
+**Quality and cost.** Final quality is the default: Multilingual v2 speech, up to 3 takes, 20-second ambience and 10-second object loops. `--draft` or `DIORAMA_SOUND=draft` opts into Flash v2.5, up to 2 takes, 10-second ambience and 6-second loops. `--final` or `DIORAMA_SOUND=final` explicitly selects final. These settings affect the next generation; existing recordings need the relevant `sound` or `voices` step rerun. Final generation can use more ElevenLabs credits. The credit floor remains `DIORAMA_ELEVEN_FLOOR` (2000 by default).
+
+**Casting and dialogue.** Voice candidates prefer the story language; designed-voice reuse requires the same language, age, gender and performance brief. Old casts are migrated on the next voices run. Preview text comes from the character's dialogue, including for short non-English samples. Preview selection checks transcription and audio integrity, not highest pitch. Stock fallbacks are reported and are not artificially pitch-shifted, and only an adult gets one: a child, teen or old speaker whose voice Voice Design refused or could not check waits (voice `pending-design`, lines unspoken) until the next voices run or `voice_set` designs it again. A transcriber that does not answer (a busy queue, a dropped socket) is retried with a backoff and never counts against a voice; the paid previews are kept until it answers, so the next run checks them without designing again. Dialogue requests carry unambiguous preceding/following lines through the API context fields. Every speaker has their own recording even when two characters say the same text. New stories also pass an editorial review for natural phrasing, distinct personalities and speaker perspective, in addition to mechanical playability.
+
+**Validation and reuse.** Paid responses remain in the memo. Shared effects require an exact prompt match and sufficient quality; a generic ID such as `shot-ending` cannot substitute another world's finale. Older assets are revalidated on rebuild. Silence, clipping, unusable steps, persistently noisy/noncontinuous loops and failed speech transcripts are rejected. A line with numbers or sums ("10+10=20") is read as words in the story language and checked as said; a take that says another number fails. Crickets, birdsong and other bright tones are not hiss: only noise without a pitch is. A rejected take is forgotten everywhere it was cached, so the next run asks for new takes, and a second take after a nearly silent one asks for it audible, not quieter. `sound_status` lists every sound and why one stayed silent; `sound_redo` (`diorama sound <slug> --redo <id,id>`) makes only those again, with a hint kept in `sound/fixes.json`. A partial result records `failures` in `sound/sounds.json` or `voices.json` and the command fails visibly. Files already generated are retained on disk; rejected recordings are not published in the manifest. Cast cards show unavailable lines and casting warnings. Technical checks do not establish that acting, timbre or artistic sound design is good: audition representative lines and gameplay audio before release.
+
+Studio Setup → Audio → Audio quality saves Final or Draft for future generations in all worlds and CLI commands (`~/.config/diorama/sound.json`). CLI flags override `DIORAMA_SOUND`, which overrides the saved choice. Setup shows and locks the selector when the environment overrides it. A running sound or voices step keeps the quality it started with.
+
+**Transcription outages.** Voice verification sends audio no faster than real time, with one active transcription per process. A busy or disconnected service gets at most four attempts with cooldowns and slower transmission (down to quarter speed); the timeout includes the recording's transmission time. A closed or failed stream stops sending immediately. Voice Design does not add another retry loop. Paid previews remain saved until verification succeeds, and a retry resumes pending voices without recasting working actors. Studio groups blocked lines by speaker and distinguishes pending verification, pending voice creation, rejected audio and generation failures. Its retry button reruns only Voices when no effects are missing.
+
+To rebuild an existing world's audio: `diorama voices <slug>` and `diorama sound <slug>`. Rewriting the script is a separate `story` step; changing voice quality alone does not rewrite existing dialogue.
+
+For one missing line, use `voice_retry` with the speaker key and exact text from `cast_status.voiceFailures`, or `diorama voice-retry <slug> <speaker> "<exact text>"`. Studio's retry button selects this operation when only one line is missing. It keeps the selected voice and every other recording. After a failed audio-quality check, both single-line and full voices retries use a new persisted seed batch rather than replaying the same rejected cache entries. If transcription is unavailable, the next retry checks that batch's saved audio instead of paying for another one. Original cached recordings are retained.
+
+In the page, loops pan and fade with where the hero stands, footsteps follow the floor under him, and `m` mutes. `diorama playthrough --video` records the page's sound too and writes an mp4.

package/docs/studio.md

@@ -0,0 +1,62 @@
+# Studio
+
+`diorama studio [slug]` opens the GUI: describe a world (type it, say it with 🎙, or add photos with ＋ Photo or 📷 Camera, up to 16, one after another: a real room becomes the first place, a person the hero, drawings the characters, things and places they show) and watch it build, place by place, with every step, model call and verify finding live. It listens on 127.0.0.1 only and opens through the one-time link it prints. Everything it does is an action, also runnable as `diorama action <name> '<json>'` (`diorama action` lists them), so a terminal, the page and the Muse Code agent drive the same things. The terminal running the Studio prints what runs (`--verbose` for every job line, `--quiet` for none).
+
+The Studio is an atelier. Its home is one big prompt ("What should the world be?"): one Spark call (action `world_new`) picks the world's title and folder name, how many places the story needs (2 to 5: only what it needs) and the language it is written in. A spoken idea is heard by Muse Voice Transcribe (`idea_hear`). Every world sits in the rail on the left as its painting, with a ring while it builds and a red **!** when it waits for you; the bell gathers what is building, what failed and what finished since you last looked.
+
+**Build it** turns the start screen into a short optional creative conversation (`idea_review`, using the configured Spark provider, Meta by default). There is no modal or dimmed backdrop: the original idea and pictures become the first message, followed by questions and replies. It reads the idea, attached images and sampled saved video observations/transcripts, then picks one worthwhile decision grounded in that idea or the latest reply. There is no fixed character/place/style checklist: it can explore secrets, motives, relationships, world rules, player actions, consequences or twists. Each question chooses its own `presentation`: `pictures` for decisions that benefit from seeing alternatives, or `text` for short verbal choices. Text options generate no images. Original drawings are used directly when a picture choice refers to them. Other picture choices get concept previews in the background (`idea_preview`, using the configured image provider, Muse Image by default). Each generated card costs an image call, cached for reuse. Slow or failed previews do not block choosing or skipping. **Something else…** opens a free-text reply. The reply goes back to the model: it can stop or ask one worthwhile follow-up, with a maximum of two questions in this short flow. Even a clear brief can invite an optional creative direction, but suggestions only become decisions when selected. Questions can refer to a particular image or movie timestamp. **Show me some looks** compares image previews of the same subject in different styles. If every answered question used text and the visual style is still open, the creator sees those style previews automatically at the end. They can choose one, dismiss them, or build immediately; showing a proposal never selects it. These previews use the same cached image calls as manually opening looks. Concrete objects and visual story clues should use picture choices; text is for distinctions that are better expressed in words. **Skip & build** skips the remaining conversation while preserving replies and style choices already made. Calls already sent may still finish and incur their model cost. **Edit idea** restores the original form and attachments without building.
+
+When no questions remain, the conversation says **Your story is ready for its first sketches.** A short hint points to the button below. The primary **Choose cast & places** action opens the visual cast and place selection; selecting a style does not end the creative flow. **Skip & build now** is the secondary shortcut for creators who want to bypass it. The compact translucent action dock stays visible while scrolling. Loading shows a spinner only while work is happening, not while waiting for a choice or after an error.
+
+**My own style…** accepts a written direction, an uploaded/pasted reference, or both. **Preview my style** shows the result before **Use this style** explicitly selects it. Editing the draft invalidates that unselected preview. Written styles are kept verbatim (short names get a harmless “Visual style:” prefix); a reference is interpreted by the configured Spark vision provider and retained separately from story drawings. Its medium, palette and texture guide concept previews, characters and place paintings, without adding the reference’s objects to the story. The reference is copied into both direct and preview-approved worlds, and a later explicit restyle supersedes it. Reference interpretation and image previews use the normal cached model calls.
+
+Answered questions keep their selected picture as a compact card with a check mark, or their chosen text reply without a placeholder image. **Change** (or clicking that card) opens the original alternatives; free-text replies can be edited too. Cancelling or choosing the same answer preserves the current branch, including its selected world-preview takes. Committing a different answer clears only later dependent answers and the old world preview; the next question receives the updated brief. Late responses from the old branch are ignored. Original choice pictures are reused. A selected visual style also remains visible and can be changed from the storyboard; changing it requires a new preview before approving a build.
+
+**Choose cast & places** opens an inline contact sheet of the actual planned hero, NPCs and every place (`idea_sketch`). The normal world-design stage runs once in a hidden draft; there is no gameplay, animation, model, segmentation or audio build yet. `idea_sketch_image` paints one concept per card, two at a time in the UI. Click the picture to enlarge it (up to 1600 px); **Another version** generates just that card from the selected take. **Change…** opens a small inline editor: describe a visual change, paste or attach a reference picture, then **Regenerate**. The selected take and additional reference are passed to the image editor; each revision is saved separately, and earlier takes remain selectable. Retries reuse the same instructions; conflicting instructions cannot overwrite a saved or pending take. Paid images and added references remain on disk. Every ready image carries a UI-only **PREVIEW** badge. These are concept references, not a promise of pixel-identical final gameplay layouts. The normal image provider and its per-image pricing apply; thumbnail dimensions do not imply a cheaper generation tier.
+
+**Add characters or places…** lets the creator describe additions to the preview. Diorama creates a separate expanded plan, keeps the existing selected concept images and sketches new subjects. The revised plan must keep existing subjects and form a connected world; planning and new sketches use model calls. A world can contain up to eight places. If the new plan cannot be validated, the original preview remains available.
+
+**Build this version** passes the exact plan and selected takes to `world_new` via `sketch`. It rejects changed inputs, missing selections or missing images, publishes the approved `world.json` without redesigning it, and starts at `story`. The clean selected images feed the final character and place painters as appearance references; original drawings/photos still accompany them. Scene layout also starts from the approved composition, adapting it where necessary to pass walkability gates; real photos remain authoritative for real-room geometry. Double submissions reuse the same publication. **Skip & build** is always available, including after a failed preview, and uses the ordinary direct-build path with the creator's accepted replies and style. Drafts and images are retained under `.start/sketches/`; approved worlds retain source material, all generated variants, selected references and `approved-sketch.json`.
+
+Only sent replies and an explicitly selected style are added to the build brief. The model's tentative understanding, unselected suggestions and unanswered questions never become build instructions. The original idea and accepted choices are retained in `creative-brief.json`; an explicit style is applied to `world.json`, including worlds from drawings whose default would otherwise be crayons. The world designer can choose the drawn protagonist from the creator's answers instead of always taking the first drawing. Source shapes and identities remain part of the drawing instructions. Agents and CLI clients may call `idea_review` before `world_new`; existing direct `world_new` calls still build immediately, with optional structured `choices` (including an additional note for API clients).
+
+![The atelier](images/studio-atelier.jpg)
+
+A world is a conversation with the agent that looks after it, beside a stage. The conversation tells the world in cards: the idea, the **Story**, the **Places** as small paintings, the **Cast** in a row, **Needs you** when a place is not playable (**Fix it** hands it to the agent), what each turn **Changed**, the build while it runs (click a phase of its bar, or one step's segment, to put what it made on the stage: the story, a place's painting, cut-outs or walk grid, the cast, the game), and **Play**. A card puts what it is about on the stage. Point at a thing in the painting and its SAM mask lights up; click it, or a character in the cast, and it goes into the next message ("What about the fish buns?"), so you talk about *this* instead of naming masks. **＋ Picture** and **Camera** add a photo or a drawing to a message. When the agent needs you to decide (rebuild this place? which of two fixes?), it asks with `ask_person` and the question comes as a card with buttons; a tap is your answer.
+
+A failed layout can be repaired before any painting or place directory exists: `place_status` exposes the failure, `place_fix` resumes that place, and `layout_fix` saves concrete layout guidance in `corrections.json` and rebuilds it through verification. Small positive boxes in generated layouts are enlarged to the 40 px minimum and all geometry checks run again; photo measurements are preserved. Exhausted batches keep their last answer and errors under `memo/layout-retries`, so another attempt uses fresh feedback while retaining paid cached answers. Each batch is limited to three attempts; the usual agent repair limit still applies. Existing paintings are not replaced by `layout_fix`.
+
+The agent can use `job_inspect` to see a job's timestamped recent activity, how long it has been quiet, locally observed model/API operations and their durations, and how many voices or sounds finished or failed. `job_wait` includes a shorter view. An operation still in flight does not say what the remote service is doing, and a quiet log alone is not a diagnosis. `job_cancel` stops a job when requested; already paid calls are not undone.
+
+![A world: the conversation and the stage](images/studio-world.jpg)
+
+**▶ Play**, **↶ Undo**, **Share** and **Film** (the stage alone on the whole screen, for recording) are always in the stage's top bar. The ☰ menu holds History, Style, Build again, Activity (the raw record of what ran), Follow the agent, Setup, and the hand tools: **Fix <place> by hand** opens the place editor (Box, Brush, Stand, Move NPC, Repaint, Note, the thing table) and **Cast details** the cast editor (looks, Repaint, Another take, voices). The game plays on the stage, from the start or a place, with Restart, Full screen, gamepads and **Bot playthrough**; when the world changes it offers a reload, and the agent reloads it itself. Nothing in the Studio scrolls as a page: the conversation and the rail scroll inside themselves, at any window size.
+
+Every fix is kept in a file next to what it fixes and applied again by every later build, so a rebuild never loses it: `places/<place>/corrections.json` (rebuilding only that place from the step it feeds), `story-fixes.json`, `cast/fixes.json`. A place's thing list is edited the same way: Remove takes a prop off it (kept in the picture, or painted out with one Muse edit) and Add makes something painted a thing of its own (`removed` and `added` in `corrections.json`, which every step reads with world.json); exits, items and people belong to the story. A small repaint rebuilds only what it reaches (`app/src/scene/carry.ts`): each place step keeps the painting it last ran on (`places/<place>/.ran/`), compares the new one with it tile by tile, and asks again only about what the changed tiles reach (a thing's or floor's mask, a plate, the backdrop's cards, the light, the stairs); everything else is asked about the old painting, which the memo answers for nothing, and the place keeps its scale. Lighting the lanterns of a harbour takes about 100 s instead of 270 s, and a rebuild on the same painting asks nothing new. A change over a third of the painting rebuilds everything. A place whose steps have not kept their painting (built before this, or put back by an Undo) takes the painting on disk as theirs before a repaint, so that repaint carries too. **History** lists the jobs the world went through, the agent's included, each with who ran it, when and which places it changed, and puts the world back as it was before any of them (Redo takes that back right after); a locked world or place is left alone by the agent.
+
+The conversation is genaicode's chat with Muse Code (default), Claude Code or Codex, picked in its header before the first message (Setup's **Default agent**, or a pick here, is remembered in `~/.config/diorama/studio.json` for new worlds; a world keeps the agent it last talked with, and `DIORAMA_AGENT=codex` overrides both); after it, "Hand over…" passes the conversation to another agent and a line marks where it changed hands. After a Studio restart the earlier turns come back on screen from `studio/sessions/`. The conversation is also kept in words in `studio/conversation.md`: a new agent session (another agent, or after the Studio restarts) is told to read it first, so nothing agreed is lost. The agent works through the same actions as the page, so its steps show live and its changes survive rebuilds; when a build leaves places unplayable it is woken to repair them, three tries a place. Every conversation is recorded in `studio/sessions/`: open one from "Recorded sessions", or `/w/<slug>?replay=<file>&speed=2`, to play it back for a film.
+
+**Setup** shows what this machine still needs, which agents it found, and the model calls sent across all worlds, priced once you enter the per-call prices from your billing pages. Keys are only ever typed in the terminal (`diorama setup`).
+
+## NPC progress and movement
+
+NPCs can have simple story stages: waiting for a key, key received, then in the garden. These are ordinary flags with conditional dialogue and interactions. Ask the agent for the event and the changed response; it reads `story_status` and uses `story_redo` for behaviour or `story_text` for wording alone. Walking is optional for each character. Someone can stay in one place throughout their story, repeat a short purposeful walk, or move to a neighbouring place after one event without having a repeating route.
+
+The Story view shows conditional dialogue, optional walks and departure effects. **Move NPC** in the place editor changes the standing position; **Animate in 3D** prepares assets. Story behaviour is authored separately. After story changes, the agent checks required animation and new voice/face assets, waits for jobs, reloads the game and uses `playthrough` to check the event and the conversation in the destination. A still picture does not verify that sequence. See [NPC story stages, walking and travel](npc-motion.md) for the flag/effect format and build follow-up.
+
+`world_extend` preserves existing NPC stages, flags and walks while adding new content. Changing an existing character's progress uses the story workflow. Existing games gain movement when it is added to their story; there is no automatic requirement to make every NPC walk.
+
+## A game from children's drawings
+
+A world can be made from what children drew and said: `diorama build <slug> --drawings photos/ --told recordings/ --places 3`, with no idea, or an idea of your own instead of the recordings. In the Studio, photograph the drawings on the home screen (📷 Camera keeps taking them; Spark tells drawings from a room or a person) and build, or add them to a world's conversation (**＋ Picture** or **Camera**) and ask the agent; it uses the same workshop actions. Each drawing is kept in `drawings/`, each clip in `told/` with what Muse Voice Transcribe heard. The world keeps every drawing under its key (`drawings.json`): characters are painted from their drawing (`cast`), places and the things drawn in them follow theirs (`paint`), and the style is their crayons come alive. A world with drawings or recordings is for kids (`audience: kids` in world.json): the story is gentle and simple, every line is voiced since the players cannot read, and the text is in the language they spoke (`language`).
+
+To add new drawings to an existing game, the agent saves chat pictures with `kids_from_picture` and starts `world_extend` with the resulting `drawings/` paths and the person's wish. It keeps the existing script and places, adds new places and characters, and rebuilds the connecting place (its painting changes) and the new places. It checks the combined story and builds in a private copy, installing it only on success; the job costs model calls and can be undone in History. A drawing identified as something already in the world cannot yet replace or update that existing thing.
+
+On the start screen, **Video** sits beside **Photo**, **Camera** and **Say it** inside the prompt. Its button opens the native picker; the raw file input and technical instructions are hidden. After selection, the filename, player, progress and **Video tips** appear under the toolbar. It uploads an MP4, MOV, WebM, MKV, M4V or AVI brief (up to 2 GB / 4 hours). Show a drawing or object and explain its role. Uploads use 2 MB chunks; analysis works through 30-second intervals, samples a frame at the start of each 10-second window and pairs it with overlapping speech turns returned by Meta Muse Voice Transcribe’s file endpoint (`POST /v1/asr/transcribe`, DIARIZATION). Each 30-second audio clip is uploaded as mono 24 kHz PCM WAV, with a short silent tail to let the last utterance finish. Movies without audio use pictures alone; silent audio windows skip transcription. Hold each drawing steady for at least ten seconds: quick gestures or drawings between samples can be missed. Ambiguous references are marked for review, not guaranteed to be resolved.
+
+The movie can be played while analysis runs. **Pause** stops after the current interval; **Resume video** reuses completed analysis and transcriptions. After a page refresh, choose the same file again to resume. An outage leaves the interval unfinished, and a rejected interpretation is retried with feedback. Review and edit the accumulated idea before **Build it**, and remove irrelevant pictures. At most 32 useful reference frames spread across the recording are offered; all interval evidence remains saved. The idea must fit the existing 4000-character field, and the combined photo count must fit the displayed limit. Analysis uses configured vision and transcription services; file STT does not deliberately replay audio in real time. Speech evidence is reviewable with the provider, model, approximate turn times and speaker labels. Labels are scoped to each clip, not verified identities across the film.
+
+Uploads and timestamped evidence live in `worlds/.start/videos/<id>/`; a world created from them retains a copy in `videos/<id>/`. Large videos are decoded interval by interval, without loading the entire file or decoded soundtrack into memory. Local tooling requires `ffmpeg` and `ffprobe`. Actions `video_upload`, `video_inspect`, `video_read` and `video_result` expose the same workflow to agents; `world_new` accepts the completed video IDs together with the reviewed idea and photos.
+
+
+File STT is isolated in `app/src/speech/transcribe.ts`: `SttProvider` accepts mono 24 kHz PCM and returns normalized text, timestamp precision (`none`, `turn`, `word`), optional speaker scope and segments in seconds. The default adapter is Meta (`app/src/meta/voice-file.ts`); another adapter can be passed without changing the video workflow. Gemini and OpenAI adapters are not implemented yet. A provider without timestamps must return `timing: none`; its full text is kept as untimed context instead of being assigned to invented frame times. Cache keys include provider, model, hints and audio, and duplicate operations are coalesced. Realtime microphone/voice-validation transport remains separate.

package/package.json

@@ -1,6 +1,72 @@
 {
   "name": "@gtanczyk/diorama",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.2.0",
+  "description": "Turn a story prompt into playable painted point-and-click scenes with Meta Muse and SAM 3.1",
+  "license": "Apache-2.0",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/gtanczyk/diorama.git"
+  },
+  "homepage": "https://github.com/gtanczyk/diorama#readme",
+  "keywords": [
+    "game",
+    "point-and-click",
+    "procedural",
+    "meta",
+    "muse",
+    "sam",
+    "cli"
+  ],
+  "type": "module",
+  "bin": {
+    "diorama": "app/bin/diorama.mjs"
+  },
+  "files": [
+    "app/bin/",
+    "app/src/",
+    "app/studio/",
+    "app/web/",
+    "app/tsconfig.json",
+    "docs/*.md"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "engines": {
+    "node": ">=22"
+  },
+  "scripts": {
+    "diorama": "node app/bin/diorama.mjs",
+    "setup": "node app/bin/diorama.mjs setup",
+    "doctor": "node app/bin/diorama.mjs doctor",
+    "studio": "node app/bin/diorama.mjs studio",
+    "build": "node app/bin/diorama.mjs build",
+    "play": "node app/bin/diorama.mjs play",
+    "pages": "node app/bin/diorama.mjs publish",
+    "export": "node app/bin/diorama.mjs export",
+    "browsers": "node app/bin/diorama.mjs browsers",
+    "typecheck": "tsc --noEmit -p app && tsc --noEmit -p app/web && tsc --noEmit -p app/studio",
+    "test": "vitest run",
+    "check": "npm run typecheck && npm test",
+    "prepublishOnly": "npm run typecheck"
+  },
+  "dependencies": {
+    "@meta-sam/parser": "^0.1.0",
+    "@preact/signals": "^2.11.3",
+    "esbuild": "^0.25.0",
+    "genaicode": "^2.25.0",
+    "onnxruntime-web": "^1.31.1",
+    "playwright": "^1.63.0",
+    "preact": "^10.29.8",
+    "sharp": "^0.33.5",
+    "three": "^0.170.0",
+    "tsx": "^4.19.2",
+    "zod": "^3.25.76"
+  },
+  "devDependencies": {
+    "@types/node": "^22.10.0",
+    "@types/three": "^0.170.0",
+    "typescript": "^5.7.2",
+    "vitest": "^2.1.8"
+  }
+}