@hybridlabor-api/aos 4.2.0 → 4.4.0

package/mcps/tdmcp/docs/superpowers/specs/2026-06-23-kinect-wall-harp-design.md

@@ -1,340 +0,0 @@
-# Kinect Wall Harp - Formal Prototype Spec
-
-Date: 2026-06-23
-Status: design approved, ready for implementation planning
-Feature name: `kinect_wall_harp`
-Target runtime: TouchDesigner 2025.32820 on macOS Apple Silicon
-
-## Summary
-
-Build a TouchDesigner prototype for a projected wall harp controlled by a Kinect v2
-mounted above the projector. The Kinect and projector point at the same wall. The
-projection displays eight vertical harp strings. A performer touches or approaches the
-projected wall with either hand; each hand is tracked independently as a depth blob near
-the wall plane. When a hand enters a string zone, the system triggers a short electronic
-pluck synth note and animates the touched string with vibration and glow.
-
-The approved first version prioritizes a reliable, playable wall-interaction prototype:
-
-- Kinect v2 through the already validated FreenectTD `FreenectTOP` plugin.
-- Two independent hand/blob tracks.
-- Eight vertical projected strings.
-- A safe electronic scale for improvisation.
-- Sound generated inside TouchDesigner.
-- Short pluck behavior on entry, not sustained notes.
-- Depth-based wall-touch blob detection rather than skeleton tracking.
-
-Skeleton tracking is explicitly out of scope for this version because the macOS
-FreenectTD path exposes RGB, depth, IR, and point cloud buffers, but does not provide
-body/skeleton joints.
-
-## Physical Setup
-
-- Kinect v2 is mounted above or near the projector.
-- Kinect and projector both point toward the same wall.
-- The projection shows the visible instrument surface.
-- The performer touches, taps, or moves hands very close to the projected wall.
-- The Kinect depth image is calibrated against the wall depth, then filtered for hands
-  that enter a narrow depth band near the wall.
-
-This is treated as a projected wall instrument, not as a free-air harp in the middle of
-the room.
-
-## Target TouchDesigner Network
-
-Create one isolated Base COMP under `/project1`:
-
-```text
-/project1/kinect_wall_harp
-```
-
-The component owns the full prototype and should not modify unrelated nodes. It can reuse
-or replace the earlier Kinect test network only if the operator explicitly requests that
-later. The initial build should leave existing project nodes alone.
-
-### Major Blocks
-
-```text
-kinect input
-  FreenectTOP (Kinect v2)
-        |
-        +--> depth buffer via Render Select TOP
-        |
-        +--> optional RGB/IR debug previews
-
-depth processing
-  crop/fit -> wall depth threshold -> cleanup -> blob extraction
-        |
-        +--> hand_left / hand_right normalized positions
-
-harp logic
-  8 vertical zones
-  per-hand zone state
-  entry-edge detection
-  per-string cooldown
-        |
-        +--> note trigger channels
-        +--> string energy/envelope channels
-
-audio
-  8 electronic pluck voices
-  scale-mapped frequencies
-  summed stereo output
-
-visual
-  8 projected strings
-  hit glow/vibration from string envelopes
-  optional debug overlay for hand blobs and zones
-```
-
-## Tracking Design
-
-### Input
-
-Use `FreenectTOP` with:
-
-- `Hardwareversion = Kinect v2`
-- `Active = On`
-- depth enabled
-
-Depth is read through `Render Select TOP` buffer index `1`. The validated FreenectTD
-mapping is:
-
-- source TOP output: RGB, 1280x720
-- buffer index `1`: depth, 512x424
-- buffer index `2`: point cloud, 512x424
-- buffer index `3`: IR, 512x424
-
-### Wall-Touch Mask
-
-The prototype creates a mask from the depth buffer where pixels are within a configurable
-depth range around the wall-touch band.
-
-Controls:
-
-| Control | Purpose |
-|---|---|
-| `WallDepthCenter` | The depth value representing the wall or touch plane. |
-| `TouchThickness` | Accepted band around the wall-touch plane. |
-| `DepthPolarity` | Allows flipping whether hands are expected in front of or near the wall if needed. |
-| `CropLeft`, `CropRight`, `CropTop`, `CropBottom` | Limits tracking to the projected wall area. |
-| `Sensitivity` | Blob threshold / cleanup aggressiveness. |
-
-The first implementation may use manual calibration sliders instead of automatic wall
-calibration. A later version can add a `Calibrate Wall` pulse that samples the depth image
-when the wall is empty.
-
-### Two-Hand Blob Extraction
-
-Extract up to two hand blobs from the wall-touch mask. The minimum viable approach is:
-
-1. Build a cleaned binary mask from depth.
-2. Reduce the mask to candidate hand regions.
-3. Compute centroid and size for the two strongest candidates.
-4. Sort candidates by X position into left and right hands.
-5. Smooth positions to reduce jitter.
-
-If TouchDesigner-native blob extraction operators are not reliable in the active build,
-use a small Script TOP or Script CHOP helper inside the component to compute centroids from
-the depth mask. This helper should output stable CHOP channels:
-
-```text
-left_present
-left_x
-left_y
-right_present
-right_x
-right_y
-```
-
-Coordinates should be normalized to the calibrated projection region:
-
-- `x = 0` at the left edge of the projected harp area.
-- `x = 1` at the right edge.
-- `y = 0` at the top edge and `y = 1` at the bottom edge, matching image-coordinate
-  convention and making debug overlays easier to align with TOP output.
-
-## Harp Logic
-
-The harp divides the calibrated horizontal range into eight equal string zones. Each hand
-can independently trigger a string.
-
-### Entry Trigger
-
-A note triggers when:
-
-1. The hand is present.
-2. The hand is inside a string's active zone.
-3. That hand was not already inside that string on the previous frame.
-4. The string is not inside its cooldown window.
-
-This avoids repeated retriggers while the hand remains on the line.
-
-### Cooldown
-
-Use a per-string cooldown, default around 120-180 ms. This keeps fast taps playable while
-preventing accidental flams from tracking noise.
-
-### Scale
-
-Use a safe electronic/minor pentatonic style scale across eight strings. Proposed initial
-notes:
-
-```text
-C3, Eb3, F3, G3, Bb3, C4, Eb4, F4
-```
-
-The exact root and octave can be exposed as implementation-time controls if the native TD
-audio network supports it cleanly. The prototype should at least make the frequency table
-easy to edit.
-
-## Audio Design
-
-Generate sound inside TouchDesigner.
-
-Each string triggers a short electronic pluck voice:
-
-- Oscillator: sine, triangle, or simple subtractive waveform.
-- Envelope: fast attack, short decay, no long sustain.
-- Tone: electronic, not acoustic harp.
-- Mix: summed stereo output with master volume.
-
-Controls:
-
-| Control | Purpose |
-|---|---|
-| `MasterVolume` | Overall output gain. |
-| `Decay` | Pluck length. |
-| `Brightness` | Filter or harmonic intensity if available. |
-| `CooldownMs` | Shared or per-string retrigger guard. |
-
-The first build can prioritize reliable note triggers over a polished synth engine. A
-more expressive synth can be layered after the tracking is proven.
-
-## Visual Design
-
-The projected output should be the primary user-facing view:
-
-- Eight vertical strings distributed across the projection area.
-- Idle strings use a cool cyan/blue glow.
-- Touched strings flash warmer or brighter.
-- Touched strings vibrate horizontally for a short time.
-- Optional hand debug dots can be toggled on/off.
-- Optional zone boundaries can be toggled on/off for calibration.
-
-The final output TOP should be clearly named:
-
-```text
-/project1/kinect_wall_harp/out1
-```
-
-Debug outputs should be named predictably:
-
-```text
-depth_debug
-mask_debug
-hands_debug
-audio_debug
-```
-
-## Calibration Controls
-
-The component should expose a custom parameter page with at least:
-
-```text
-Tracking
-  Active
-  WallDepthCenter
-  TouchThickness
-  DepthPolarity
-  Sensitivity
-  Smoothing
-  CropLeft
-  CropRight
-  CropTop
-  CropBottom
-  ShowDebug
-
-Harp
-  StringCount (fixed at 8 for v1, optionally hidden)
-  CooldownMs
-  HitThreshold
-
-Audio
-  MasterVolume
-  Decay
-  Brightness
-
-Visual
-  BaseColor
-  HitColor
-  Glow
-  VibrationAmount
-  VibrationDecay
-```
-
-Manual calibration is acceptable for v1. The operator should be able to tune the wall
-depth and crop live while watching the debug overlay.
-
-## Layout Requirements
-
-All generated TouchDesigner operators must have explicit deterministic coordinates. The
-network should be laid out by role:
-
-- Kinect input and raw buffers on the left.
-- Depth processing and hand tracking in the middle.
-- Harp trigger logic and audio below or to the right.
-- Visual output and final `out1` on the far right.
-- Debug views grouped near their related processing stage.
-
-No generated operators should be stacked at the same coordinates.
-
-## Error Handling and Fallbacks
-
-- If `FreenectTOP` is unavailable, create the component with a clear error/status DAT and
-  a synthetic fallback mode so the visual/audio logic can still be tested.
-- If the Kinect is connected but depth is not cooking, expose the raw FreenectTOP errors
-  and keep the component from silently producing false triggers.
-- If no hands are detected, output zero trigger channels and keep the visual idle state.
-- If only one hand is detected, the detected hand still plays normally.
-- If both hands overlap or merge into one blob, treat it as one blob until they separate.
-
-## Verification Plan
-
-Implementation is done only when these checks pass:
-
-1. TouchDesigner bridge responds at `http://127.0.0.1:9980/api/info`.
-2. `FreenectTOP` exists and cooks in `Kinect v2` mode.
-3. Depth buffer can be selected and cooks without node errors.
-4. `/project1/kinect_wall_harp` exists and has non-overlapping node coordinates.
-5. Network error scan for `/project1/kinect_wall_harp` returns no errors.
-6. Debug channels show left/right hand presence and normalized positions when hands touch
-   the wall region.
-7. Entering a string zone produces one trigger pulse, not continuous retriggering.
-8. Two hands can trigger two different strings independently.
-9. Audio output responds to string triggers.
-10. Visual strings flash/vibrate on triggers.
-11. Debug overlay can be turned off for the clean projected output.
-
-Hardware-dependent checks should be reported separately from offline network checks. Do
-not claim live hand tracking is verified unless it is tested against the real Kinect and
-wall setup.
-
-## Out of Scope for V1
-
-- Skeleton/body joint tracking.
-- MIDI or OSC output to external synths.
-- Automatic projector-camera calibration.
-- Multi-user tracking beyond two hand blobs.
-- Complex gesture vocabulary.
-- Saving a polished reusable `.tox` package.
-- Production-ready audio synthesis.
-
-## Open Implementation Notes
-
-- Prefer built-in TOP/CHOP operators for thresholding, cleanup, and logic where practical.
-- Use a Script CHOP/TOP only where native operators make two-hand centroid extraction
-  unnecessarily brittle.
-- Keep all generated scripts local to the component and small enough to inspect.
-- Run any live Python through the existing tdmcp bridge only during construction and
-  validation. Disable arbitrary exec again after live validation when possible.

package/mcps/tdmcp/docs/superpowers/specs/2026-06-24-kinect-environment-diagnostic-design.md

@@ -1,151 +0,0 @@
-# Kinect Environment Diagnostic - Design
-
-Date: 2026-06-24
-Status: approved for standalone implementation
-Target runtime: TouchDesigner 2025.32820 on macOS with external libfreenect2
-
-## Goal
-
-Create a separate TouchDesigner diagnostic screen at:
-
-```text
-/project1/kinect_environment_diagnostic
-```
-
-The screen helps verify what the Kinect sees before the harp calibration step. It
-must not depend on FreenectTOP because that path has already crashed
-TouchDesigner on this machine. It uses an external libfreenect2 process instead.
-
-## Research Summary
-
-- libfreenect2 exposes Kinect v2 color, IR, depth, and color/depth registration.
-- Microsoft RoomAlive calibrates Kinect/projector setups by requiring both the
-  Kinect color and depth cameras to observe a good portion of the projection.
-- TouchDesigner's KinectCalibration component uses projected checkerboard point
-  pairs: 2D projector points plus 3D points from the Kinect point cloud.
-- Projected checkerboard approaches can extract corners in RGB and pair them
-  with depth samples, but full projector calibration requires careful point order
-  and usually multiple poses.
-- ChArUco boards are a stronger future option than plain checkerboards because
-  they keep subpixel checkerboard accuracy while tolerating partial occlusion.
-- Gray-code/structured-light approaches can produce dense projector-camera
-  correspondences, but they are heavier than needed for the first diagnostic.
-
-## First Implementable Slice
-
-Build a live environment diagnostic, not a full automatic projector calibration.
-
-The external helper opens the Kinect and captures:
-
-- RGB color frame;
-- depth frame;
-- IR frame;
-- per-pixel empty-wall depth background;
-- foreground/candidate statistics using the same depth-band idea as the harp.
-
-It writes a low-rate raw RGBA diagnostic image and JSON status file:
-
-```text
-/tmp/kinect_environment_diagnostic.rgba
-/tmp/kinect_environment_diagnostic.json
-```
-
-TouchDesigner reads the raw RGBA image in a Script TOP and displays it full
-screen. A status DAT records frame counters, valid depth ratio, RGB/depth/IR
-dimensions, calibration state, and candidate counts.
-
-## Visual Layout
-
-The diagnostic image is a 1280x720 four-panel view:
-
-```text
-+----------------------+----------------------+
-| RGB camera           | Depth heatmap         |
-|                      |                      |
-+----------------------+----------------------+
-| IR intensity         | Foreground/mask view  |
-|                      |                      |
-+----------------------+----------------------+
-```
-
-Panel borders and status blocks are drawn into the image so the operator can see
-whether each stream is alive even if the status DAT is not open.
-
-## Calibration Use
-
-This screen answers:
-
-- Is the projector inside the Kinect RGB field of view?
-- Does the Kinect see the wall as a stable depth surface?
-- Are RGB/depth/IR frames alive at the same time?
-- Is the selected wall-touch depth band too strict or too noisy?
-- Does crop need to move left/right/up/down before harping?
-
-It does not yet solve the full projector-camera matrix. The next slice can add a
-projected ChArUco/checkerboard mode that collects point pairs and estimates a
-homography or projector pose.
-
-## Projection Pattern Slice
-
-The second slice adds:
-
-```text
-/project1/kinect_projection_calibration_pattern
-```
-
-This COMP projects a full-screen high-contrast checker/grid pattern with large
-corner fiducials. The external diagnostic helper analyzes the RGB camera frame
-and reports:
-
-- whether a bright projection area is visible;
-- the normalized RGB-camera bounding box of that projected area;
-- detected red, green, blue, yellow, and cyan marker centroids when visible;
-- the existing depth/IR/background statistics.
-
-For the current room, the projection bounding box is the primary calibration
-signal. Marker colors can be partially desaturated by the wall/projector/camera
-pipeline, so marker detections are diagnostic hints rather than the source of
-truth for this slice.
-
-## Registered Projection Slice
-
-The implemented next slice uses `libfreenect2::Registration` to map the RGB
-projection into the Kinect depth frame. This avoids applying a 1920x1080 RGB
-bbox directly to 512x424 depth data.
-
-The helper reports:
-
-- `registered_projection_present`
-- `registered_projection_bright_ratio`
-- `registered_projection_bbox`
-- `projection_depth_samples`
-- `foreground_samples_in_projection`
-- `candidate_samples_in_projection`
-- `candidate_projection_x`
-- `candidate_projection_y`
-
-The bottom-right diagnostic panel tints the registered projection area green and
-draws a green box over the depth-space projection region. Candidate touch pixels
-inside that region are rendered green; candidates outside the projection remain
-amber. The harp should use this registered bbox for its first automatic crop
-proposal.
-
-## Runtime Boundary
-
-Only one process can open the Kinect at a time. While the environment diagnostic
-helper is running, the harp's external Kinect sender is stopped. The harp COMP is
-left intact and can be resumed by restarting its sender.
-
-## References
-
-- OpenKinect libfreenect2 API reference: color, IR, depth, and registration.
-- Microsoft RoomAlive Toolkit ProCamCalibration README: Kinect/projector setup
-  expectations and projector configuration cautions.
-- TouchDesigner forum Kinect2 Projector Calibration component: checkerboard
-  point-pair workflow with color camera and point cloud lookup.
-- Bingyao Huang Kinect/projector calibration writeup: projected checkerboard
-  corners paired with Kinect depth/color captures.
-- OpenCV ChArUco documentation: robust board detection, corner interpolation,
-  and solvePnP pose estimation.
-- Gray-code structured-light project: heavier dense correspondence approach for
-  future calibration refinement.

package/mcps/tdmcp/docs/superpowers/specs/2026-06-24-kinect-wall-harp-external-kinect-design.md

@@ -1,104 +0,0 @@
-# Kinect Wall Harp - External Kinect Input Design
-
-Date: 2026-06-24
-Status: approved for implementation
-Feature name: `kinect_wall_harp_external_kinect`
-Target runtime: TouchDesigner 2025.32820 on macOS Apple Silicon
-
-## Summary
-
-Keep Kinect v2 as the real performer input, but do not activate FreenectTOP in
-the main TouchDesigner project. Prior live validation showed that FreenectTD can
-crash TouchDesigner during Kinect v2 init/cook. The safer architecture is to run
-Kinect depth capture in a separate process, extract up to two wall-touch blobs,
-and send normalized hand channels into TouchDesigner over OSC.
-
-The existing `/project1/kinect_wall_harp` component remains the visual/audio
-instrument. This wave adds a real-Kinect input path that can replace the current
-synthetic hands without putting the main `.toe` at risk.
-
-## Architecture
-
-```text
-Kinect v2 USB 3.0
-  -> external libfreenect2 process
-  -> depth threshold + two blob centroids
-  -> OSC UDP on localhost
-  -> TouchDesigner osc_kinect input CHOP
-  -> kinect_wall_harp hand_tracker
-  -> harp_logic
-  -> strings_visual + pluck_synth + UMC202 audio_out
-```
-
-## Current Evidence
-
-- macOS detects the Kinect as `Xbox NUI Sensor`, Microsoft `1118:708`, USB 3.0
-  link at 5 Gbps, serial `032559434547`.
-- The current machine has `cmake`, `pkg-config`, and Homebrew `libusb`.
-- `libfreenect2`, `Protonect`, and Python freenect bindings are not currently
-  installed on PATH.
-- FreenectTD exists as a TouchDesigner plugin, but crash logs implicate
-  FreenectTOP/libfreenect2 paths, so it must remain disabled in the main project.
-
-## Scope
-
-In scope:
-
-- Build or install `libfreenect2` outside TouchDesigner.
-- Verify Kinect depth capture with `Protonect` or an equivalent test executable.
-- Add an OSC Kinect mode to the harp tool.
-- Create a local bridge process/script that publishes normalized hand channels.
-- Rebuild the live harp in TouchDesigner with `source="osc_kinect"`.
-- Validate that TD receives live channels and stays crash-free.
-
-Out of scope:
-
-- Re-enabling FreenectTOP in the main `.toe`.
-- Skeleton/body tracking.
-- Production-grade calibration UI.
-- External synth/MIDI output.
-
-## OSC Contract
-
-The external bridge sends these localhost OSC addresses:
-
-```text
-/kinect/left/present   0|1
-/kinect/left/x         0..1
-/kinect/left/y         0..1
-/kinect/left/size      0..1
-/kinect/right/present  0|1
-/kinect/right/x        0..1
-/kinect/right/y        0..1
-/kinect/right/size     0..1
-```
-
-TouchDesigner maps these to the same hand storage shape used by the current
-harp:
-
-```text
-left_present, left_x, left_y, left_size
-right_present, right_x, right_y, right_size
-```
-
-## Safety Rules
-
-- Do not activate FreenectTOP in `/project1/kinect_wall_harp`.
-- If the external Kinect process crashes, TouchDesigner should continue running.
-- If OSC packets are absent, the harp should either show no hands or use an
-  explicit fallback mode, never silently claim real Kinect tracking.
-- Keep UMC202 audio routing unchanged.
-
-## Verification
-
-PASS requires:
-
-- Kinect USB visible on macOS.
-- External Kinect test can open the device without crashing TouchDesigner.
-- TouchDesigner bridge stays reachable at `:9980`.
-- `/project1/kinect_wall_harp` has no recursive errors.
-- OSC mode produces live `left_*` / `right_*` hand channels.
-- Preview output renders and audio debug still produces stereo samples.
-
-If `libfreenect2` cannot be built or cannot open the device, mark Kinect input
-BLOCKED with exact build/runtime logs, leaving the existing synthetic harp intact.

package/mcps/tdmcp/training/showintent/data/curated/.gitkeep

@@ -1 +0,0 @@
-

package/mcps/tdmcp/training/showintent/data/curated/curated-approved.jsonl

@@ -1 +0,0 @@
-

package/mcps/tdmcp/training/showintent/data/curation_template.csv

@@ -1 +0,0 @@
-id,input,locale,context,current_mood,expected_intent_json,expected_policy_decision,tags,notes,approved_by_human

package/mcps/tdmcp/training/showintent/data/generated/.gitkeep

@@ -1 +0,0 @@
-