@hybridlabor-api/aos 4.3.0 → 4.4.0

package/mcps/tdmcp/docs/guide/physical-installations.md

@@ -1,129 +0,0 @@
----
-description: "How to build reliable tdmcp projects for rooms, projectors, sensors and external hardware without confusing a demo with a validated installation."
----
-
-# Physical installations
-
-Hardware installations fail differently from pure visuals. A network can build
-cleanly, the preview can look right, and the room can still be wrong because the
-sensor sees the wall from another angle, the projector is outside the camera
-frame, the audio interface is running at a different rate, or a device plugin
-crashes TouchDesigner during startup.
-
-The Kinect wall harp turned those failure modes into a reusable tdmcp pattern:
-build the artwork, the diagnostics and the hardware boundary as separate pieces.
-
-## The reliable shape
-
-Use this order for projectors, depth cameras, MIDI/OSC devices and room sensors:
-
-1. **Build a synthetic-safe version first.** The component should render, expose
-   controls and generate plausible test input with no hardware attached.
-2. **Add a diagnostic view before calibration.** Show what the device actually
-   sees: RGB/depth/IR frames, crop boxes, valid-sample ratios and candidate
-   blobs. Do not start by tuning the final artwork.
-3. **Keep crash-prone hardware outside the main `.toe`.** If a plugin or SDK can
-   take TouchDesigner down, run it in a helper process and send normalized OSC,
-   MIDI, UDP, WebSocket or file output back into TD.
-4. **Calibrate on the projected surface.** Put the wizard or target pattern on
-   the same projector output the performer will use. Chat-timed calibration is
-   too fragile for room setup.
-5. **Separate live claims from offline gates.** Typecheck, tests and synthetic
-   previews prove the tool shape. A live room pass proves the actual sensor,
-   projector, audio device and performer distance.
-
-## What the Kinect harp taught us
-
-- **Depth cameras are not magic touch screens.** The Kinect detects depth
-  discontinuities near a wall plane; it does not understand projected lines or
-  colours as touch targets. The software maps tracked blobs into the projected
-  coordinate system.
-- **Projection and sensor coordinate spaces must be explicit.** A hand touching
-  the right side of the wall can trigger the left side if crop, mirror or Y-axis
-  conventions are guessed.
-- **Debug markers can mislead.** Synthetic fallback hands are useful in rehearsal,
-  but debug overlays must clearly switch off when live tracking is absent.
-- **Audio problems are often routing/runtime problems, not volume.** A noisy,
-  glitch-like synth can come from sample-rate mismatch, clipping inside a voice
-  bank, too many overlapping triggers, or the wrong output device.
-- **Restarting a helper is part of the feature.** A stalled depth stream should
-  restart its helper or mark tracking offline; it should not silently freeze the
-  bridge. For the Kinect wall harp bridge, use `--status-json <path>` when an
-  operator or generated TD component needs a machine-readable health surface.
-  Generated `create_kinect_wall_harp` projects read the same file into the
-  `bridge_status` DAT and expose numeric channels through `bridge_status_chop`.
-- **Projector crops should come from RGB plus registered depth.** When a Kinect
-  and projector see the same wall, run the projection calibration pattern and
-  external environment diagnostic first. The diagnostic writes
-  `/tmp/kinect_environment_diagnostic.json`; pass it to
-  `scripts/kinect-wall-harp-bridge.mjs --projection-calibration-json ...` so the
-  bridge uses `registered_projection_bbox` as its depth crop. The OSC contract
-  preserves both crop/projector-space hand points (`x/y`) and full registered
-  depth coordinates (`raw_x/raw_y`) so TD components can reuse the wall-harp
-  `Input*` calibration controls for final alignment.
-
-## Current tools to use
-
-- **`create_kinect_wall_harp`** builds the projected string instrument with
-  synthetic fallback, OSC Kinect input mode, calibration controls and internal
-  synth.
-- **`scripts/create-kinect-projection-calibration-pattern-td.py`** projects the
-  high-contrast room target, and **`scripts/kinect-environment-diagnostic.cpp`**
-  reads Kinect RGB/depth/IR plus libfreenect2 registration to produce the
-  projection crop JSON used by the OSC bridge.
-- **`create_test_pattern`** gives the room a visible alignment target before
-  projection mapping or calibration.
-- **`create_interactive_projection_mapping`** is the rehearsal rig for camera or
-  synthetic motion driving a projector output.
-- **`create_live_source`** builds camera, NDI, Syphon/Spout, screen-grab or
-  network-video inputs and now exposes `source_status` plus
-  `source_status_chop` so the generated project can tell whether the source op
-  is missing, waiting, failed or running.
-- **`create_depth_silhouette`** and **`create_blob_reactive`** are lighter sensor
-  tools when the artwork needs masks or tracked blobs instead of a full custom
-  instrument. `create_depth_silhouette` also exposes `source_status` and
-  `source_status_chop` for its selected synthetic, file or live depth source.
-- **`create_external_io`** is the standard route for OSC, MIDI, DMX, NDI and
-  Syphon/Spout I/O.
-- **`diagnose_hardware_environment`** is the first generic room preflight: it
-  checks bridge reachability, display/projector count and generated
-  `source_status` / `bridge_status` DATs before calibration.
-- **`watch_node`**, **`get_node_state_runtime`** and
-  **`inspect_gpu_and_displays`** help verify that the running TD project is
-  cooking and routed to the intended display.
-
-## Backlog from the Kinect project
-
-The wall harp points to a small reusable installation toolkit:
-
-| Candidate | What it would add | Why it matters |
-|---|---|---|
-| `diagnose_hardware_environment` extensions | Add RGB/depth frame metrics and audio-device checks to the shipped bridge/display/status-DAT preflight. | Artists need to know whether the room is wrong before tuning an artwork. |
-| `create_projection_calibration_wizard` | Refine the existing automatic registered-bbox crop into projected point-pair/ChArUco calibration with stored homography output. | Automatic crop is now the default first pass; precise projector pose still needs a richer calibration model. |
-| `run_external_sensor_bridge` | A reusable helper supervisor for sensor processes, with stale-data detection, restart policy and normalized OSC/WebSocket/status JSON output. | Crash isolation and restart behavior should not be reimplemented per device. |
-| `external_sensor_status_surface` | Shared builder primitive for generated DAT/CHOP health surfaces from external helper status JSON or local TouchDesigner operator status. | Every physical sensor tool should expose the same health shape to panels, overlays and logic. |
-| `diagnose_audio_device` | Output-device, sample-rate, clipping and voice-count checks for TD audio chains. | Glitchy audio is common in interactive instruments and needs a first-class checklist. |
-| `organize_generated_project` | Move, label and prune generated COMPs under `/project1` while preserving useful diagnostics. | Live iteration leaves debris; cleanup should be safe and explain what remains. |
-
-Treat these as next slices, not shipped capabilities. Each one needs its own
-offline test shape and, where hardware is involved, a live validation note.
-
-## Room checklist
-
-Before calling a physical installation "done":
-
-- the final output is on the intended projector or display;
-- the diagnostic view shows live frames or live channels from the real device;
-- synthetic fallback is visibly different from live tracking;
-- crop, mirror and Y-axis conventions are verified with at least left/right and
-  top/bottom touches;
-- audio routes to the intended interface and stays below clipping;
-- helper processes recover from stalls or fail with a visible status;
-- the final component can still render when hardware is unplugged.
-
-## See also
-
-- [Prompt cookbook](/guide/prompt-cookbook#output-mapping)
-- [Layer-1 generators](/guide/generators#installations-studies)
-- [Troubleshooting](/guide/troubleshooting)
-- [Bridge & REST API](/reference/bridge-api)