@hybridlabor-api/aos 4.3.0 → 4.4.0

package/mcps/tdmcp/docs/superpowers/plans/2026-06-24-kinect-wall-harp-external-kinect.md

@@ -1,286 +0,0 @@
-# Kinect Wall Harp External Kinect Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Keep Kinect v2 as the real wall-harp input by moving depth capture outside TouchDesigner and feeding normalized hand points back through OSC.
-
-**Architecture:** The main TouchDesigner project remains crash-safe: `/project1/kinect_wall_harp` never activates FreenectTOP. A separate external Kinect process captures depth with `libfreenect2`, extracts two wall-touch blobs, and sends normalized OSC channels to a new `osc_kinect` input mode in the harp tool.
-
-**Tech Stack:** TouchDesigner 2025.32820, tdmcp bridge `0.6.1`, TypeScript/Zod/Vitest, Homebrew, `libfreenect2`, OSC UDP on localhost.
-
----
-
-## Task 1: External Kinect Runtime Probe
-
-**Files:**
-- Modify: `_workspace/kinect-wall-harp/08_external_kinect_runtime.md`
-
-- [ ] **Step 1: Record current USB/runtime state**
-
-Run:
-
-```bash
-rtk proxy sh -lc 'ioreg -p IOUSB -l -w0 | rg -i -A18 -B6 "Xbox NUI Sensor|idVendor\" = 1118|idProduct\" = 708"'
-rtk proxy sh -lc 'command -v Protonect || true; command -v freenect2-glview || true; pkg-config --modversion libfreenect2 2>/dev/null || true'
-rtk proxy sh -lc 'brew list --versions cmake pkg-config libusb 2>/dev/null || true'
-```
-
-Expected:
-
-```text
-Xbox NUI Sensor appears on USB 3.0.
-Protonect/libfreenect2 may be missing before install/build.
-cmake, pkg-config, and libusb are available or installable by Homebrew.
-```
-
-- [ ] **Step 2: Check Homebrew availability**
-
-Run:
-
-```bash
-rtk proxy brew search libfreenect2
-rtk proxy brew info libfreenect2 || true
-```
-
-Expected:
-
-```text
-Either Homebrew has a formula/cask we can install, or we build from OpenKinect source.
-```
-
-- [ ] **Step 3: Install or build libfreenect2 outside TouchDesigner**
-
-Preferred if formula exists:
-
-```bash
-rtk proxy brew install libfreenect2
-```
-
-Fallback source build:
-
-```bash
-rtk proxy mkdir -p _workspace/kinect-wall-harp/external
-rtk proxy git clone https://github.com/OpenKinect/libfreenect2.git _workspace/kinect-wall-harp/external/libfreenect2
-rtk proxy cmake -S _workspace/kinect-wall-harp/external/libfreenect2 -B _workspace/kinect-wall-harp/external/libfreenect2/build -DENABLE_OPENGL=OFF -DENABLE_OPENCL=OFF -DENABLE_CUDA=OFF
-rtk proxy cmake --build _workspace/kinect-wall-harp/external/libfreenect2/build --parallel 4
-```
-
-Expected:
-
-```text
-A Protonect or libfreenect2 test executable exists outside TouchDesigner.
-```
-
-- [ ] **Step 4: Run a short external Kinect open test**
-
-Run the discovered test executable for a short window, for example:
-
-```bash
-rtk proxy timeout 10s _workspace/kinect-wall-harp/external/libfreenect2/build/bin/Protonect
-```
-
-Expected:
-
-```text
-The external process opens the Kinect or reports an actionable USB/firmware/backend error.
-TouchDesigner remains open and bridge :9980 remains reachable.
-```
-
-## Task 2: Add OSC Kinect Mode To Harp Tool
-
-**Files:**
-- Modify: `src/tools/layer1/createKinectWallHarp.ts`
-- Modify: `tests/unit/createKinectWallHarp.test.ts`
-
-- [ ] **Step 1: Write failing schema/payload test**
-
-Add a test case asserting:
-
-```ts
-const parsed = createKinectWallHarpSchema.parse({
-  source: "osc_kinect",
-  osc_port: 7400,
-  fallback_to_synthetic: false,
-});
-expect(parsed.source).toBe("osc_kinect");
-expect(parsed.osc_port).toBe(7400);
-```
-
-Expected before implementation:
-
-```text
-FAIL because source enum does not include osc_kinect and osc_port is not in the schema.
-```
-
-- [ ] **Step 2: Extend schema**
-
-In `createKinectWallHarpSchema`, change:
-
-```ts
-source: z.enum(["freenect", "synthetic"])
-```
-
-to:
-
-```ts
-source: z.enum(["freenect", "synthetic", "osc_kinect"])
-```
-
-Add:
-
-```ts
-osc_port: z.coerce
-  .number()
-  .int()
-  .min(1024)
-  .max(65535)
-  .default(7400)
-  .describe("UDP port for OSC Kinect hand input when source='osc_kinect'."),
-```
-
-- [ ] **Step 3: Add TouchDesigner OSC input nodes**
-
-In the Python payload builder, when `_mode == "osc_kinect"`, create:
-
-```python
-_osc = _create(_cont, ["oscinCHOP"], "osc_kinect_in", -220, -20, "OSC Kinect input")
-_set_par(_osc, ["port"], int(_p["osc_port"]), False)
-_set_par(_osc, ["active"], True, False)
-_osc_select = _create(_cont, ["selectCHOP"], "osc_kinect_select", 20, -20, "OSC hand channels")
-_connect(_osc, _osc_select)
-```
-
-The `hand_tracker` callback receives `osc_path`.
-
-- [ ] **Step 4: Teach `HAND_CHOP_CODE` to read OSC channels**
-
-Add an OSC branch in `onCook`:
-
-```python
-elif mode == "osc_kinect":
-    src = op(CFG.get("osc_path", ""))
-    left = _read_osc_hand(src, "left")
-    right = _read_osc_hand(src, "right")
-```
-
-Implement `_read_osc_hand(src, prefix)` so it accepts both slash-derived and
-underscore channel names:
-
-```python
-present = _read_chop_any(src, [prefix + "_present", prefix + ":present", prefix + "/present"], 0.0)
-x = _read_chop_any(src, [prefix + "_x", prefix + ":x", prefix + "/x"], 0.0)
-y = _read_chop_any(src, [prefix + "_y", prefix + ":y", prefix + "/y"], 0.0)
-size = _read_chop_any(src, [prefix + "_size", prefix + ":size", prefix + "/size"], 0.0)
-return (present, x, y, size)
-```
-
-- [ ] **Step 5: Run focused tests**
-
-Run:
-
-```bash
-rtk proxy npm test -- tests/unit/createKinectWallHarp.test.ts
-rtk proxy npx --yes @biomejs/biome@2.4.15 check src/tools/layer1/createKinectWallHarp.ts tests/unit/createKinectWallHarp.test.ts
-rtk proxy npm run build
-```
-
-Expected:
-
-```text
-Focused tests pass, Biome reports no fixes, TypeScript build passes.
-```
-
-## Task 3: External OSC Bridge Script
-
-**Files:**
-- Create: `scripts/kinect-wall-harp-bridge.mjs`
-- Modify: `package.json` only if adding a script entry is needed.
-
-- [ ] **Step 1: Create a dry-run OSC sender first**
-
-Create a Node script that accepts:
-
-```text
---host 127.0.0.1
---port 7400
---source synthetic|libfreenect2
-```
-
-For `--source synthetic`, emit the eight OSC addresses from the spec at 30 Hz.
-
-- [ ] **Step 2: Verify TouchDesigner receives synthetic OSC**
-
-Run:
-
-```bash
-rtk proxy node scripts/kinect-wall-harp-bridge.mjs --source synthetic --port 7400
-```
-
-Expected:
-
-```text
-TouchDesigner osc_kinect mode sees moving left/right points without Kinect capture.
-```
-
-- [ ] **Step 3: Add libfreenect2 source only after external runtime probe passes**
-
-If `Protonect` works, add a libfreenect2-backed mode or a small helper executable
-wrapper that emits normalized left/right blobs. Keep this separate from the TD
-bridge process.
-
-## Task 4: Live TouchDesigner Rebuild And QA
-
-**Files:**
-- Modify: `_workspace/kinect-wall-harp/08_external_kinect_runtime.md`
-
-- [ ] **Step 1: Rebuild harp in OSC Kinect mode**
-
-Run:
-
-```bash
-rtk proxy node dist/cli/agent.js nodes delete --td-port 9980 --params '{"path":"/project1/kinect_wall_harp"}'
-rtk proxy node dist/cli/agent.js kinect-wall-harp --td-port 9980 --params '{"name":"kinect_wall_harp","source":"osc_kinect","fallback_to_synthetic":false,"osc_port":7400,"show_debug":true,"string_count":8,"master_volume":0.25,"audio_device":"UMC202HD 192k"}'
-```
-
-Expected:
-
-```text
-The harp reports mode osc_kinect, creates osc_kinect_in, and keeps FreenectTOP inactive.
-```
-
-- [ ] **Step 2: Validate live TD health**
-
-Run:
-
-```bash
-rtk proxy node dist/cli/agent.js nodes errors --td-port 9980 --params '{"path":"/project1","recursive":true}'
-rtk proxy node dist/cli/agent.js preview /project1/kinect_wall_harp/out1 --td-port 9980 -o /tmp/kinect_wall_harp_osc_kinect.png
-```
-
-Expected:
-
-```text
-No recursive errors; preview renders eight strings.
-```
-
-- [ ] **Step 3: Validate audio path**
-
-Run a TD Python diagnostic that triggers one pluck and confirms `audio_debug`
-has `left/right` samples. Expected:
-
-```text
-audio_debug contains left/right with nonzero max_abs.
-```
-
-- [ ] **Step 4: Record final PASS/BLOCKED state**
-
-Update `_workspace/kinect-wall-harp/08_external_kinect_runtime.md` with:
-
-```text
-USB: PASS/FAIL
-libfreenect2 external open: PASS/FAIL/BLOCKED
-OSC input mode: PASS/FAIL
-TD errors: PASS/FAIL
-Audio UMC202: PASS/FAIL
-Real Kinect hand tracking: PASS/FAIL/BLOCKED
-```

package/mcps/tdmcp/docs/superpowers/specs/2026-05-31-create-vector-lines-design.md

@@ -1,308 +0,0 @@
-# Create Vector Lines - Formal Feature Spec
-
-Date: 2026-05-31
-Status: design ready for implementation planning
-Harness: `tdmcp-pipeline` / `td-feature-design`
-Feature name: `create_vector_lines`
-
-## Summary
-
-Build a Layer 1 artist tool that captures a still frame from a camera, file, synthetic
-source, or existing TOP, vectorizes the chosen image through a pulse-driven Trace SOP
-pipeline, and composites the rendered vector layer back over the source image. Phase 1 is
-explicitly not realtime: the user presses a `Vectorize` pulse, tdmcp freezes the current
-prepared frame, generates editable SOP geometry, and leaves the vectors ready for later
-animation, modulation, export, or mixing.
-
-The selected direction is hybrid foreground-first:
-
-- Default to a controllable foreground/object workflow, with threshold, blur, invert, and
-  cleanup controls.
-- Keep a `full_frame` mode so the same tool can produce dense abstract line-art from the
-  whole camera image.
-- Do not require MediaPipe, segmentation models, or live tracking in Phase 1.
-
-Phase 2 can add animation over the captured vector geometry. Phase 3 can add throttled
-realtime vectorization once the pulse workflow is proven and live-cook costs are measured.
-
-## Layer + Target Path
-
-- Layer: Layer 1 artist tool.
-- Target file: `src/tools/layer1/createVectorLines.ts`.
-- Tool name: `create_vector_lines`.
-- CLI shorthand: `vector-lines` or `vectorize-lines`.
-- Primary output: `out1` Null TOP, a source image with vector lines composited on top.
-- Secondary outputs:
-  - `vectors_out` Null TOP, the rendered vector layer on transparent/dark background.
-  - `trace1` Trace SOP, the editable vector geometry.
-  - `frozen_frame` Movie File In TOP, the pulse-captured source image used by Trace SOP.
-  - `prep_out` Null TOP, the prepared monochrome/mask image used for capture.
-
-## Input Schema
-
-| Name | Type | Default | Notes |
-|---|---|---:|---|
-| `name` | string | `vector_lines` | Name for the system COMP. |
-| `parent_path` | string | `/project1` | Parent COMP where the system is created. |
-| `source` | enum | `synthetic` | `synthetic`, `camera`, `file`, or `existing_top`. Device capture is opt-in. |
-| `existing_top_path` | string optional | none | Required only when `source="existing_top"`. Pulled through Select TOP. |
-| `movie_file_path` | string optional | none | Used only when `source="file"`. |
-| `camera_device` | string optional | none | Used only when `source="camera"`; omitted means TD default camera. |
-| `mode` | enum | `hybrid_foreground` | `hybrid_foreground`, `foreground_mask`, or `full_frame`. |
-| `analysis_resolution` | tuple number | `[640, 360]` | Resolution for prep/capture/trace. Keeps vectorization cost bounded. |
-| `threshold` | number 0..1 | `0.45` | Brightness/mask cutoff for foreground and Trace SOP. |
-| `pre_blur` | number >=0 | `2` | Blur before threshold/trace to remove camera noise. |
-| `invert` | boolean | `false` | Invert the mask before tracing. |
-| `remove_borders` | boolean | `true` | Trace SOP border cleanup (`delborder`). |
-| `resample` | boolean | `true` | Trace SOP shape resampling (`doresample`). |
-| `step_size` | number >0 | `4` | Trace SOP resample step / simplification control. |
-| `smooth_shapes` | boolean | `true` | Trace SOP smoothing (`dosmooth`). |
-| `fit_curves` | boolean | `false` | Trace SOP Bezier fit (`fitcurve`), probe live before making default true. |
-| `line_color` | string | `#49dcb2` | Hex color for the vector material. |
-| `line_width` | number >0 | `2` | Width for the line/wire material where supported. |
-| `opacity` | number 0..1 | `0.9` | Vector overlay opacity. |
-| `overlay_mode` | enum | `over` | `over`, `add`, `screen`, or `multiply`; maps to Composite TOP operand. |
-| `show_source` | boolean | `true` | Whether final output includes the source image under the vectors. |
-| `expose_controls` | boolean | `true` | Adds the control page and pulse button to the container. |
-
-Device-sourced defaults intentionally favor `synthetic`. Like `create_live_source` and
-`create_motion_reactive`, camera capture can trigger OS permission dialogs and should not
-be the default build path.
-
-## Network Topology
-
-The tool creates one self-contained Base COMP.
-
-```text
-source stage
-  synthetic: noiseTOP/rampTOP test image
-  camera:    videodeviceinTOP
-  file:      moviefileinTOP
-  existing:  selectTOP(existing_top_path)
-        |
-        v
-fit_source (fitTOP, analysis_resolution)
-        |
-        +-------------------------------> source_display (nullTOP)
-        |
-        v
-prep chain
-  monochromeTOP -> blurTOP -> optional edgeTOP -> thresholdTOP -> optional levelTOP invert
-        |
-        v
-prep_out (nullTOP)
-        |
-        |  Pulse "Vectorize"
-        |  saves prep_out to snapshot PNG and reloads frozen_frame
-        v
-frozen_frame (moviefileinTOP)
-        |
-        v
-trace1 (traceSOP, TOP Name points at frozen_frame)
-        |
-        v
-optional simplify/smooth/transform SOP stages
-        |
-        v
-vector_geo (geometryCOMP + wireframe/line material)
-        |
-        v
-render_vectors (renderTOP with orthographic camera)
-        |
-        v
-vectors_opacity (levelTOP)
-        |
-        +---- if show_source=true ---- compositeTOP over source_display ---- out1 (nullTOP)
-        |
-        +---- vectors_out (nullTOP)
-```
-
-### Operator Notes
-
-- `Trace SOP` exists in the knowledge base and is the intended conversion point. The KB
-  confirms parameters for threshold (`thresh`), border removal (`delborder`), resampling
-  (`doresample`, `step`), smoothing (`dosmooth`, `corner`), curve fitting (`fitcurve`,
-  `error`), polygon conversion (`convpoly`, `lod`), and hole filling (`hole`).
-- The exact Trace SOP source parameter should be set defensively. The UI label is `TOP
-  Name`; the implementation should probe likely parameter names in Python and warn rather
-  than fail when a TD build differs.
-- `Wireframe MAT` is the default render material for a plotter/contour look over Trace
-  SOP faces. `Line MAT` is the live-probe fallback if the generated primitives render more
-  cleanly as lines/curves.
-- The vector render should use an orthographic camera and fixed 2D framing so the overlay
-  aligns with the captured source. If exact framing is not reliable on the first pass,
-  include `Scale`, `OffsetX`, and `OffsetY` controls as calibration knobs.
-
-## Bridge / Python Approach
-
-Use the existing Layer 1 orchestration pattern:
-
-- `createSystemContainer(ctx, parent_path, name)` for the Base COMP.
-- `NetworkBuilder.add`, `connect`, `setParams`, and `python` for node creation and
-  fail-forward warnings.
-- `finalize(ctx, { capturePreviewImage: true })` for layout, controls, error check, and
-  preview.
-
-The pulse callback needs a Python DAT inside the container, because the vectorization
-event happens after the tool returns:
-
-1. Add a custom parameter page on the system container:
-   - `Vectorize` pulse.
-   - prep controls: `Mode`, `Threshold`, `PreBlur`, `Invert`, `StepSize`,
-     `SmoothShapes`, `FitCurves`, `RemoveBorders`.
-   - look controls: `LineColor`, `LineWidth`, `Opacity`, `OverlayMode`, `ShowSource`.
-   - optional calibration: `Scale`, `OffsetX`, `OffsetY`.
-2. Add a DAT callback or Execute DAT that handles the pulse.
-3. On pulse:
-   - cook `prep_out`;
-   - write `prep_out` to a PNG under a stable project-local folder such as
-     `<project.folder>/tdmcp_snapshots/vector_lines/<container_name>_latest.png`;
-   - point `frozen_frame.par.file` at that PNG and pulse reload if the par exists;
-   - set Trace SOP params from current controls;
-   - cook `trace1`, `render_vectors`, and `out1`;
-   - write a status table/DAT with timestamp, snapshot path, warning text, and, if easy,
-     point/primitive counts.
-
-Use defensive Python helpers for parameter names, following nearby tools that probe
-platform-dependent parameter names. A wrong parameter name should become a warning, not a
-hard tool failure.
-
-## UI Wireframe
-
-```text
-create_vector_lines COMP
-
-[Input]
-Source: synthetic | camera | file | existing_top
-Mode:   hybrid_foreground | foreground_mask | full_frame
-
-[Capture]
-(Pulse) Vectorize
-Status: Waiting / Captured / Trace warning / Trace failed
-
-[Prep]
-Threshold  [0.00 ---------------- 1.00]
-PreBlur    [0 ---------------------- 32]
-Invert     [toggle]
-RemoveBorders [toggle]
-StepSize   [1 ---------------------- 32]
-SmoothShapes [toggle]
-FitCurves    [toggle]
-
-[Look]
-LineColor  [rgb]
-LineWidth  [0.25 ------------------- 12]
-Opacity    [0.00 ---------------- 1.00]
-OverlayMode: over | add | screen | multiply
-ShowSource [toggle]
-
-[Calibrate]
-Scale   [0.1 --------------------- 4]
-OffsetX [-1 ---------------------- 1]
-OffsetY [-1 ---------------------- 1]
-```
-
-The first implementation can expose the core controls and leave `Status` as a Text DAT if
-custom UI state is too expensive. The pulse button is non-negotiable for Phase 1.
-
-## Phase Boundaries
-
-### Phase 1 - Pulse Vectorization
-
-Ship `create_vector_lines` with pulse capture, Trace SOP vector geometry, rendered overlay,
-and controls above. No realtime loop. No external segmentation dependency.
-
-### Phase 2 - Animate Captured Vectors
-
-Add one or more optional animation tools or extensions:
-
-- point jitter / turbulence over the Trace SOP output;
-- audio or `create_modulators` bindings for line width, opacity, transform, and color;
-- morph between two captured snapshots;
-- SVG/plotter export candidate if the Trace SOP output is stable enough.
-
-### Phase 3 - Throttled Realtime
-
-Add a separate mode that re-vectorizes on a timer or every N frames, never every frame by
-default. This phase needs live TD performance measurements before design is locked.
-
-## Probe-First Risks
-
-| Risk | Why it matters | Probe |
-|---|---|---|
-| Trace SOP source parameter | KB label is `TOP Name`; exact programmatic par name needs live confirmation. | Probe likely names and inspect par list in TD. |
-| Trace cook cost | Dense camera frames can create too many primitives. | Measure at 320p, 480p, 720p with point/primitive counts. |
-| Overlay alignment | SOP coordinate space may not line up with TOP pixel space automatically. | Live validate orthographic camera framing and add calibration controls. |
-| Material choice | Trace may output faces, polygons, or curves depending on params. | Compare Wireframe MAT vs Line MAT on the same capture. |
-| Camera permission | Video Device In TOP can prompt/hang until the OS permission modal is answered. | Keep `synthetic` default; document camera opt-in. |
-| Snapshot writing | Project folders can be read-only or unset in unsaved projects. | Fall back to a temp path and report the chosen path. |
-
-## Test Plan
-
-### Unit / Offline
-
-Add `tests/unit/createVectorLines.test.ts` using the existing mocked TD client / msw
-pattern.
-
-Assert:
-
-- `create_vector_lines` registers with a Zod schema and annotations
-  `{ readOnlyHint: false, destructiveHint: false, openWorldHint: true }`.
-- Default invocation creates a Base COMP, synthetic source, Fit TOP, monochrome/blur/mask
-  prep chain, Movie File In TOP for `frozen_frame`, Trace SOP, geometry/render/composite
-  chain, and `out1` Null TOP.
-- `source="existing_top"` uses Select TOP and does not create camera/file input nodes.
-- `source="camera"` creates `videodeviceinTOP` but is never the default.
-- Args map to intended parameters: `threshold`, `pre_blur`, `step_size`,
-  `remove_borders`, `smooth_shapes`, `opacity`, and `overlay_mode`.
-- Returned JSON includes `container`, `source_path`, `prep_path`, `snapshot_path`,
-  `trace_sop`, `vectors_output`, `output_path`, `controls`, and `warnings`.
-- The pulse callback DAT contains the expected high-level actions: save `prep_out`,
-  reload `frozen_frame`, set Trace SOP params, cook trace/render/output, update status.
-
-### Live / Manual
-
-Run after implementation with a reachable TD bridge:
-
-1. Build with synthetic source and press `Vectorize`. Confirm `out1` previews line-art
-   overlay and `trace1` contains geometry.
-2. Build with `existing_top_path` from a known image/movie TOP. Confirm pulse captures
-   the current prepared frame, not a continuously changing feed.
-3. Toggle `full_frame` vs `foreground_mask` / `hybrid_foreground` and confirm density
-   changes as expected.
-4. Adjust threshold, blur, step size, line width, and opacity, then re-pulse and confirm
-   visible changes.
-5. Try camera source only with user permission available; confirm no crash if the camera
-   produces no image.
-6. Capture `get_td_node_errors` and a preview after each live probe.
-
-## Integration Notes
-
-Builder creates only:
-
-- `src/tools/layer1/createVectorLines.ts`
-- `tests/unit/createVectorLines.test.ts`
-
-Integrator edits shared files:
-
-- `src/tools/layer1/index.ts` to export/register the tool.
-- `src/cli/agent.ts` to add a `vector-lines` or `vectorize-lines` shorthand command.
-- `CHANGELOG.md` and generated docs only after implementation is accepted.
-
-Docs:
-
-- `docs/reference/tools.md` is generated; do not hand-edit.
-- Add a short guide/example only after live validation, because screenshots and exact Trace
-  controls should be based on the verified TD build.
-
-## Acceptance Criteria
-
-- A non-realtime pulse workflow exists and is the only automatic vectorization path in
-  Phase 1.
-- The tool can build and preview without a camera.
-- The user can vectorize an existing TOP or camera frame into editable SOP geometry.
-- The final output composites vector lines over the source and exposes a stable output TOP.
-- The implementation reports warnings for unverified Trace/camera/snapshot details instead
-  of throwing.
-- Unit tests pass, typecheck passes, and live validation is either PASS or explicitly
-  marked `UNVERIFIED-pending` if the TD bridge is unavailable.