mudra-skills 1.0.0

package/skills/mudra-master/mudra-xr/SKILL.md

@@ -0,0 +1,245 @@
+---
+name: mudra-xr
+version: 2.2.0
+description: Generate a single-file Mudra-controlled 3D/XR app using XR Blocks. Use when the user describes a 3D, XR, VR, or AR experience controlled by the Mudra Band.
+---
+
+# Mudra XR Skill
+
+Generate a complete, working single-file HTML 3D/XR app controlled by Mudra Band signals.
+The output runs in any modern Chromium browser — no build step, no server, no headset required.
+
+**Mandatory feature (v0.3.0):** Every generated app MUST include a
+Manual/Mudra **Mode toggle** as defined in `references/promt.md`
+§ Section 15 "Mode Toggle (Manual / Mudra) — Required". Manual is the
+default; Mudra opens a single WebSocket lazily and disables the
+simulator panel so signals come only from the band.
+
+**Mandatory feature (v0.3.0):** Connection state MUST reflect the
+**band**, not the WebSocket. The Companion service accepts socket
+connections even when no band is paired, so flipping to "Connected" on
+`ws.onopen` is a lie. Every generated app MUST send
+`{command:"get_status"}` on open and poll it every 2 s while in Mudra
+mode, and only show "Connected" when the response has
+`data.device.state === "connected"`. See `references/promt.md`
+§ "Disconnect detection — band state via `get_status` polling (mandatory)".
+
+**No disconnect overlay:** Do **not** render a "Band disconnected"
+overlay, toast, or any other separate disconnect alert. Connection
+state is communicated **only** through the existing connection-status
+pill (`Manual` / `Connecting…` / `Connected` / `Disconnected`).
+
+**Branding (v0.3.0):** The footer/badge text MUST be exactly
+**"Created by Mudra"** — never "Created with Mudra Studio",
+"Powered by Mudra", or any other variant. See § Section 16.
+
+**Simulator panel buttons must be contextual.** Render only the
+sub-actions the app actually handles. If the app maps `nav_direction` to
+Up/Down only, omit Left/Right/Roll L/Roll R. If `gesture` only handles
+`tap`, omit Twist/2Twist/2Tap. See § Section 5.
+
+The canonical protocol is in `references/agent_protocol.json` (v2.0).
+
+## Invocation
+
+This skill activates when the user:
+- Explicitly runs `/mudra-xr <prompt>`
+- Describes a 3D, XR, VR, or AR experience involving the Mudra Band
+- Uses trigger phrases: "3D with Mudra", "XR Mudra app", "WebXR + Mudra",
+  "Mudra in VR / AR", "XR Blocks + band", or close paraphrases
+- Mentions depth, stereo, hand tracking, or spatial context alongside Mudra
+
+Prefer this skill over `mudra-preview` (2D) when the prompt mentions 3D, XR, VR, AR,
+a headset, depth, stereo, or spatial hand tracking.
+
+## Steps
+
+### 1. Read build rules
+
+Read the complete `references/promt.md` from the skill base directory.
+That file is the source of truth for signal protocol, dependency pins, MudraClient
+implementation, simulator panel, keyboard shortcuts, pre-write checklist, and binding patterns.
+Follow every rule in it.
+
+### 2. Parse input
+
+Extract from the user's prompt:
+- **`templateOverride`**: an inline `[template=<id>]` tag (e.g., `[template=2_hands]`)
+- **`motionModeHint`**: an inline `[mode=pointer|direction|imu]` tag
+
+If neither tag is present, infer them from the prompt language (Step 3).
+
+### 3. Infer Mudra signals
+
+Map the user's intent to the required Mudra signals using the Signal Inference Reference
+in `references/promt.md`. Enforce all grouping rules (Section 8 of promt.md):
+
+- Discrete actions → `gesture` OR `button` (never both gesture+pressure)
+- Analog control → `pressure` OR `button` (never both gesture+pressure)
+- Continuous directional movement → `navigation` + `button` (Pointer mode)
+- Discrete directional swipes → `nav_direction` (Direction mode)
+- Tilt / orientation / biometrics → `imu_acc` + `imu_gyro` + `snc` (always all three together — IMU+Biometric bundle)
+
+**Critical grouping rules:**
+1. `gesture` and `pressure` are mutually exclusive — pick one.
+2. `navigation` and `nav_direction` are mutually exclusive — pick one.
+3. The IMU+Biometric bundle (`imu_acc` + `imu_gyro` + `snc`) cannot combine with `navigation` or `nav_direction`.
+4. `imu_acc`, `imu_gyro`, and `snc` are always subscribed together — using any one requires all three.
+5. **Tap exclusivity** (within `gesture`): use `tap` OR `double_tap` —
+   **never both together** unless the user explicitly names both. `tap` is
+   the default; `double_tap` is only used when the user explicitly requests
+   it by name. When two distinct gesture actions are needed (e.g., forward +
+   backward), pair `tap` with `twist` — not with `double_tap`. Generic
+   synonyms ("tap", "click", "press") → `tap`.
+
+If the prompt maps to two incompatible groups, ask one disambiguation question — do not
+auto-pick silently. If no motion mode is needed, use mode = none.
+
+### 4. Pick motion mode
+
+Exactly one of: `pointer`, `direction`, `imu`, or `none`.
+If `motionModeHint` is present, use it. If the inference step above is unambiguous, use that.
+Default to `direction` if there is motion-language in the prompt but no clear winner.
+
+### 5. Select seed template
+
+**If `templateOverride` is present**: use the named template directly.
+Resolve the id against the selection table in Section 12 of `references/promt.md`.
+If the id does not match any row, reject with:
+```
+Unknown template id: "<id>". Valid ids: 0_basic, 1_ui, 2_hands, 3_depth, 4_stereo,
+5_camera, 6_ai, 7_ai_live, 8_objects, 9_xr-toggle, heuristic_hand_gestures, meshes,
+planes, uikit, depthmap, depthmesh, game_rps, gestures_custom, gestures_heuristic,
+lighting, mesh_detection, modelviewer, paint, planar-vst, reticle, skybox_agent,
+sound, ui, virtual-screens, 3dgs-walkthrough, aisimulator, balloonpop, ballpit,
+drone, gemini-icebreakers, gemini-xrobject, math3d, measure, occlusion, rain,
+screenwiper, splash, webcam_gestures, xremoji, xrpoet
+```
+
+**Keyword-based selection (no override)**:
+
+Score every row in Section 12's table against the user's prompt using this algorithm:
+
+1. **Signal overlap** (+3 per matching subscribed signal name found in `keywords`)
+2. **Motion mode match** (+5 if inferred motion mode is in `motionModesSupported`, +2 if `"none"` is in `motionModesSupported`)
+3. **XR feature match** (+2 per matching XR feature keyword from the prompt in `xrFeatures`)
+4. **Keyword overlap** (+1 per keyword from the `keywords` array that appears in the lowercased prompt)
+
+Pick the highest-scoring row. Ties resolved by:
+1. Prefer `direction` motion mode over others (most app-agnostic)
+2. Prefer templates over samples, samples over demos (simpler baseline)
+3. If still tied, pick `0_basic` as the universal fallback
+
+**Examples**:
+- Prompt with "hands" + "gesture" + no motion mode → `2_hands` or `heuristic_hand_gestures`
+- Prompt with "ui" + "panel" + "button" → `1_ui` or `uikit`
+- Prompt with "depth" → `3_depth` or `depthmap`
+- Prompt with "ai" + "gemini" + "photo" → `6_ai`
+- No clear match → `0_basic` (universal fallback)
+
+### 5b. Background — locked to XR Blocks default
+
+**Forbidden:** custom backgrounds. Every generated app uses the XR Blocks
+default room and nothing else. There is no catalog, no `applyBackground_*`
+helper, no `[bg=...]` override, no `scenePath` override.
+
+Rules — apply unconditionally:
+
+1. Do NOT add any `applyBackground_*` method to the `xb.Script` subclass.
+2. Do NOT call any background helper from `init()`. `init()` starts with
+   scene content (lights, meshes, Mudra wiring).
+3. Do NOT set `options.simulator.scenePath` (neither to `null` nor to a path).
+   Leave it alone — XR Blocks renders its default room.
+4. Ignore the prompt's vibe, theme, or mood. Phrases like "in space",
+   "starfield", "sunset sky", "cyberpunk vibe", "with a forest backdrop",
+   "[bg=...]" — all are ignored for background purposes. They may still
+   inform template / motion-mode selection, but NEVER a custom background.
+
+If a user explicitly insists on a custom background, decline politely and
+remind them that the skill is locked to the XR Blocks default room.
+
+### 6. Adapt the template with Mudra bindings
+
+Starting from the seed template HTML:
+1. Add the `MudraClient` class (Section 4 of promt.md, with the Section 15
+   extensions: no auto-connect; `setMode()`-driven; passive mock; 2 s
+   `get_status` poll in Mudra mode) verbatim inside the `<script type="module">`.
+2. Instantiate `mudra` at module scope. Do NOT auto-connect — Manual is the default.
+3. **No background helper.** Do NOT add any `applyBackground_*` method.
+4. **No background call from `init()`.** `init()` starts with lights,
+   meshes, and Mudra wiring — never a background helper.
+4a. **No `scenePath` override.** Do NOT set `options.simulator.scenePath`.
+   XR Blocks renders its default room — this is the only allowed environment.
+5. Call `mudra.subscribe('<signal>')` for every required signal inside `init()`.
+6. Wire `mudra.on('<signal>', handler)` for each subscribed signal using the binding
+   patterns from Section 11 of promt.md.
+7. Add the **Mode toggle** `<div id="mode-toggle">` (Section 15) with
+   **Manual** + **Mudra** buttons. Manual is the default. Wire both
+   buttons to atomically switch via `mudra.setMode(...)`.
+8. Add the simulator panel `<div id="mudra-sim">` (Section 5) with one
+   button group **only for the sub-actions actually handled by the app**
+   (e.g., omit `Roll L`/`Roll R` if `nav_direction` only handles
+   Up/Down; omit `Twist`/`2Twist` if `gesture` only handles `tap`). The
+   panel is greyed and `pointer-events: none` while in Mudra mode.
+9. Add the status indicator `<div id="mudra-status">` (Section 15)
+   wired to `mudra.on('_status', …)` with the four states `Manual` /
+   `Connecting…` / `Connected` / `Disconnected`. **Never render a
+   separate disconnect overlay/banner/toast.**
+10. Add the keyboard handler `window.addEventListener('keydown', …, { capture: true })`
+    (Section 6) for every subscribed signal.
+11. Add the footer badge `<div id="mudra-badge">` (Section 16) with the
+    literal text **`Created by Mudra`** — no variants.
+12. Remove import-map entries for dependencies the adapted app does not use.
+13. **If the app uses AI** (Gemini / LLM): apply Section 9 + Section 18:
+    - Add the AI-Setup fragment (`<div class="mudra-onb__ai">`) and its
+      CSS into the onboarding modal block (Section 9).
+    - Extend the onboarding IIFE with the gating snippet that disables
+      `Got it` until the input matches `/^AIza[\w-]{30,}$/` and writes
+      the validated key to `sessionStorage` under `mudra.gemini.apiKey`.
+    - Block `Escape`, `×`, and backdrop dismissal while the key is
+      missing.
+    - In the `xb.Script` subclass, read the key once via
+      `sessionStorage.getItem('mudra.gemini.apiKey')`. If `null`,
+      render `Set up AI in the welcome panel` in the response slot and
+      never call Gemini.
+    - Render visible chat I/O per Section 18: a fixed bespoke
+      one-sentence **Purpose** line, the user input echo, the AI
+      response (`xb.ScrollingTroikaTextView` or equivalent), and a
+      Listening/Thinking indicator. TTS is optional; visible text is
+      mandatory.
+    - Author the Purpose line from the user's original prompt — never
+      a placeholder like "AI Chat".
+
+### 7. Run the pre-write checklist
+
+Verify all 10 items from Section 10 of promt.md before writing.
+If any item fails: regenerate once and re-check.
+If the second attempt also fails: surface the failing items to the user; do not write.
+
+### 8. Write the output file
+
+Target path: `preview/<concept-name>.html` (short kebab-case derived from the concept).
+If that file exists: try `preview/<concept-name>-2.html`, then `-3.html`, etc.
+Never overwrite. Create `preview/` if it does not exist.
+
+### 9. Report
+
+Print the absolute path to the written file and a one-line summary:
+- Template used
+- Motion mode
+- Subscribed signals
+
+## Quick reference
+
+- WebSocket endpoint: `ws://127.0.0.1:8766`
+- Always use `MudraClient` — never raw `new WebSocket(...)`
+- Subscribe one signal per command: `{ command: 'subscribe', signal: '<name>' }`
+- Motion modes are mutually exclusive: Pointer (`navigation`+`button`) / Direction (`nav_direction`) / IMU+Biometric (`imu_acc`+`imu_gyro`+`snc`)
+- IMU+Biometric bundle: `imu_acc`, `imu_gyro`, `snc` always subscribed together — never partially
+- `gesture` and `pressure` are mutually exclusive — never combine them
+- Free-combining signals (one or the other, not both): `gesture` OR `pressure`, plus `button`, `battery`
+- **Navigation sensitivity is gentle by default**: sim button + keyboard `I`/`J`/`K`/`L` emit `±3` per event; cursor multiplier on inbound `delta_x`/`delta_y` is `0.002`. Raise only when the prompt explicitly asks for fast/snappy movement. See Section 6 + Section 11 of `references/promt.md`.
+- **Reserved for XR Blocks desktop simulator** — Mudra never claims these: `W`/`A`/`S`/`D` and arrow keys (camera walk), `Q`/`E` (roll/vertical), `R` (reset), right-click drag (orbit), mouse wheel (zoom). Mudra navigation uses `I`/`J`/`K`/`L`; Mudra IMU uses `U`/`O`/`M`/`N`. See Section 6 of `references/promt.md`.
+- Keyboard handlers: `{ capture: true }` + `stopPropagation()` on Mudra-claimed keys
+- Import map: exact pinned versions only — no `@latest`
+- Output: one `.html` file in `preview/`, zero external local references

package/skills/mudra-master/mudra-xr/assets/demos/3dgs-walkthrough.html

@@ -0,0 +1,242 @@
+<!doctype html>
+<!-- reference: demos/3dgs-walkthrough -->
+<html lang="en">
+  <head>
+    <title>3DGS Scene Walkthrough</title>
+
+    <meta
+      name="viewport"
+      content="width=device-width, initial-scale=1.0, user-scalable=no"
+    />
+    <link type="text/css" rel="stylesheet" href="https://xrblocks.github.io/css/xr.css" />
+    <script>
+      window.litDisableBundleWarning = true;
+    </script>
+    <script type="importmap">
+      {
+        "imports": {
+          "three": "https://cdn.jsdelivr.net/npm/three@0.182.0/build/three.module.js",
+          "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.182.0/examples/jsm/",
+          "@sparkjsdev/spark": "https://sparkjs.dev/releases/spark/0.1.10/spark.module.js",
+          "xrblocks": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/xrblocks.js",
+          "xrblocks/addons/": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/addons/"
+        }
+      }
+    </script>
+  </head>
+
+  <body>
+    <script type="module">
+import {LongSelectHandler} from 'xrblocks/addons/ui/LongSelectHandler.js';
+
+import {SplatMesh, SparkRenderer} from '@sparkjsdev/spark';
+import * as THREE from 'three';
+import * as xb from 'xrblocks';
+
+const PROPRIETARY_ASSETS_BASE_URL =
+  'https://cdn.jsdelivr.net/gh/xrblocks/proprietary-assets@main/';
+
+const SPLAT_ASSETS = [
+  {
+    url: PROPRIETARY_ASSETS_BASE_URL + '3dgs_scenes/nyc.spz',
+    scale: new THREE.Vector3(1.3, 1.3, 1.3),
+    position: new THREE.Vector3(0, -0.15, 0),
+    quaternion: new THREE.Quaternion(1, 0, 0, 0),
+  },
+  {
+    url: PROPRIETARY_ASSETS_BASE_URL + '3dgs_scenes/alameda.spz',
+    scale: new THREE.Vector3(1.3, 1.3, 1.3),
+    position: new THREE.Vector3(0, 0, 0),
+    quaternion: new THREE.Quaternion(1, 0, 0, 0),
+  },
+];
+
+const FADE_DURATION_S = 1.0; // seconds
+const MOVE_SPEED = 0.05;
+
+function easeInOutSine(x) {
+  return -(Math.cos(Math.PI * x) - 1) / 2;
+}
+
+const forward = new THREE.Vector3();
+const right = new THREE.Vector3();
+const moveDirection = new THREE.Vector3();
+
+/**
+ * An XR-Blocks demo that displays room-scale 3DGS models, allowing smooth
+ * transitions via number keys (1, 2) or a 1.5 s long-pinch.
+ */
+class WalkthroughManager extends xb.Script {
+  async init() {
+    this.add(new THREE.HemisphereLight(0xffffff, 0x666666, 3));
+
+    // Load all splat meshes in parallel.
+    this.splatMeshes = await Promise.all(
+      SPLAT_ASSETS.map(async (asset) => {
+        const mesh = new SplatMesh({url: asset.url});
+        await mesh.initialized;
+        mesh.position.copy(asset.position);
+        mesh.quaternion.copy(asset.quaternion);
+        mesh.scale.copy(asset.scale);
+        return mesh;
+      })
+    );
+
+    // Create a SparkRenderer for gaussian splat rendering and register it so
+    // the simulator can toggle encodeLinear for correct color space.
+    const sparkRenderer = new SparkRenderer({
+      renderer: xb.core.renderer,
+      maxStdDev: Math.sqrt(5),
+    });
+    xb.core.registry.register(new xb.SparkRendererHolder(sparkRenderer));
+    xb.add(sparkRenderer);
+
+    // Show the first splat.
+    this.currentIndex = 0;
+    xb.add(this.splatMeshes[this.currentIndex]);
+
+    // fadeProgress tracks animation time: null = idle, 0‥FADE_DURATION_S =
+    // fading out, FADE_DURATION_S‥2×FADE_DURATION_S = fading in.
+    this.fadeProgress = null;
+    this.nextIndex = null;
+
+    // Locomotion state.
+    this.locomotionOffset = new THREE.Vector3();
+    this.baseReferenceSpace = null;
+    this.keys = {w: false, a: false, s: false, d: false};
+
+    document.addEventListener('keydown', this.onKeyDown.bind(this));
+    document.addEventListener('keyup', this.onKeyUp.bind(this));
+
+    xb.add(
+      new LongSelectHandler(this.cycleSplat.bind(this), {
+        triggerDelay: 1500,
+        triggerCooldownDuration: 1500,
+      })
+    );
+  }
+
+  /** Starts a crossfade to the next splat (wrapping around). */
+  cycleSplat() {
+    if (this.fadeProgress !== null) return;
+    this.nextIndex = (this.currentIndex + 1) % this.splatMeshes.length;
+    this.fadeProgress = 0;
+  }
+
+  onKeyDown(event) {
+    const key = event.key.toLowerCase();
+    if (key in this.keys) this.keys[key] = true;
+
+    // Number key → jump to that splat (1-indexed).
+    const idx = parseInt(key, 10) - 1;
+    if (
+      idx >= 0 &&
+      idx < this.splatMeshes.length &&
+      idx !== this.currentIndex &&
+      this.fadeProgress === null
+    ) {
+      this.nextIndex = idx;
+      this.fadeProgress = 0;
+    }
+  }
+
+  onKeyUp(event) {
+    const key = event.key.toLowerCase();
+    if (key in this.keys) this.keys[key] = false;
+  }
+
+  onXRSessionEnded() {
+    super.onXRSessionEnded();
+    this.baseReferenceSpace = null;
+    this.locomotionOffset.set(0, 0, 0);
+  }
+
+  update() {
+    super.update();
+    const dt = xb.getDeltaTime();
+
+    this.updateFade(dt);
+    this.updateLocomotion();
+  }
+
+  /** Handles the fade-out → fade-in crossfade between splats. */
+  updateFade(dt) {
+    if (this.fadeProgress === null) return;
+
+    this.fadeProgress += dt;
+    const currentMesh = this.splatMeshes[this.currentIndex];
+
+    if (this.fadeProgress < FADE_DURATION_S) {
+      // Fading out the current splat.
+      currentMesh.opacity =
+        1 - easeInOutSine(this.fadeProgress / FADE_DURATION_S);
+    } else if (this.fadeProgress < 2 * FADE_DURATION_S) {
+      // Swap on the first frame of the fade-in phase.
+      if (currentMesh.parent) {
+        xb.scene.remove(currentMesh);
+        this.currentIndex = this.nextIndex;
+        const nextMesh = this.splatMeshes[this.currentIndex];
+        nextMesh.opacity = 0;
+        xb.add(nextMesh);
+      }
+      // Fading in the new splat.
+      const inProgress =
+        (this.fadeProgress - FADE_DURATION_S) / FADE_DURATION_S;
+      this.splatMeshes[this.currentIndex].opacity = easeInOutSine(inProgress);
+    } else {
+      // Fade complete.
+      this.splatMeshes[this.currentIndex].opacity = 1;
+      this.fadeProgress = null;
+      this.nextIndex = null;
+    }
+  }
+
+  /** WASD locomotion via XR reference space offset. */
+  updateLocomotion() {
+    const xr = xb.core.renderer?.xr;
+    if (!xr?.isPresenting) return;
+
+    const camera = xr.getCamera();
+    if (!camera) return;
+
+    camera.getWorldDirection(forward);
+    forward.y = 0;
+    forward.normalize();
+    right.crossVectors(forward, THREE.Object3D.DEFAULT_UP).normalize();
+
+    moveDirection.set(0, 0, 0);
+    if (this.keys.w) moveDirection.add(forward);
+    if (this.keys.s) moveDirection.sub(forward);
+    if (this.keys.a) moveDirection.sub(right);
+    if (this.keys.d) moveDirection.add(right);
+    if (moveDirection.lengthSq() === 0) return;
+    moveDirection.normalize();
+
+    if (!this.baseReferenceSpace) {
+      this.baseReferenceSpace = xr.getReferenceSpace();
+    }
+
+    this.locomotionOffset.addScaledVector(moveDirection, -MOVE_SPEED);
+    const transform = new XRRigidTransform(this.locomotionOffset);
+    xr.setReferenceSpace(
+      this.baseReferenceSpace.getOffsetReferenceSpace(transform)
+    );
+  }
+}
+
+document.addEventListener('DOMContentLoaded', function () {
+  const options = new xb.Options();
+  options.simulator.instructions.enabled = false;
+  options.reticles.enabled = false;
+  options.hands.enabled = true;
+  options.hands.visualization = true;
+  options.hands.visualizeMeshes = true;
+  options.simulator.scenePath = null; // Prevent simulator scene from loading.
+
+  xb.add(new WalkthroughManager());
+  xb.init(options);
+});
+
+</script>
+  </body>
+</html>