mudra-skills 3.0.2 → 3.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/mudra-master/SKILL.md +0 -196
- package/skills/mudra-master/mudra-preview/SKILL.md +0 -150
- package/skills/mudra-master/mudra-preview/assets/ar-menu.html +0 -382
- package/skills/mudra-master/mudra-preview/assets/document-scroller.html +0 -468
- package/skills/mudra-master/mudra-preview/assets/drum-machine.html +0 -788
- package/skills/mudra-master/mudra-preview/assets/emg-visualizer.html +0 -390
- package/skills/mudra-master/mudra-preview/assets/generative-art.html +0 -408
- package/skills/mudra-master/mudra-preview/assets/gesture-assistant.html +0 -342
- package/skills/mudra-master/mudra-preview/assets/gesture-speech.html +0 -444
- package/skills/mudra-master/mudra-preview/assets/hands-free-desktop.html +0 -376
- package/skills/mudra-master/mudra-preview/assets/model-rotator.html +0 -365
- package/skills/mudra-master/mudra-preview/assets/mudra-duel.html +0 -3200
- package/skills/mudra-master/mudra-preview/assets/mudra-monitor.html +0 -1451
- package/skills/mudra-master/mudra-preview/assets/mudra-ultimate-template.html +0 -891
- package/skills/mudra-master/mudra-preview/assets/music-sequencer.html +0 -455
- package/skills/mudra-master/mudra-preview/assets/neural-pong.html +0 -380
- package/skills/mudra-master/mudra-preview/assets/neural-snake.html +0 -364
- package/skills/mudra-master/mudra-preview/assets/presentation-controller.html +0 -345
- package/skills/mudra-master/mudra-preview/assets/pressure-painter.html +0 -285
- package/skills/mudra-master/mudra-preview/assets/runner.html +0 -1812
- package/skills/mudra-master/mudra-preview/assets/smart-home.html +0 -461
- package/skills/mudra-master/mudra-preview/assets/space-invaders.html +0 -1154
- package/skills/mudra-master/mudra-preview/assets/waterful-ring-toss.html +0 -1624
- package/skills/mudra-master/mudra-preview/references/agent_protocol.json +0 -634
- package/skills/mudra-master/mudra-preview/references/prompt.md +0 -16746
- package/skills/mudra-master/mudra-xr/SKILL.md +0 -245
- package/skills/mudra-master/mudra-xr/assets/demos/3dgs-walkthrough.html +0 -242
- package/skills/mudra-master/mudra-xr/assets/demos/aisimulator.html +0 -792
- package/skills/mudra-master/mudra-xr/assets/demos/balloonpop.html +0 -422
- package/skills/mudra-master/mudra-xr/assets/demos/ballpit.html +0 -284
- package/skills/mudra-master/mudra-xr/assets/demos/drone.html +0 -41
- package/skills/mudra-master/mudra-xr/assets/demos/gemini-icebreakers.html +0 -338
- package/skills/mudra-master/mudra-xr/assets/demos/gemini-xrobject.html +0 -302
- package/skills/mudra-master/mudra-xr/assets/demos/math3d.html +0 -188
- package/skills/mudra-master/mudra-xr/assets/demos/measure.html +0 -247
- package/skills/mudra-master/mudra-xr/assets/demos/occlusion.html +0 -251
- package/skills/mudra-master/mudra-xr/assets/demos/rain.html +0 -498
- package/skills/mudra-master/mudra-xr/assets/demos/screenwiper.html +0 -432
- package/skills/mudra-master/mudra-xr/assets/demos/splash.html +0 -519
- package/skills/mudra-master/mudra-xr/assets/demos/webcam_gestures.html +0 -253
- package/skills/mudra-master/mudra-xr/assets/demos/xremoji.html +0 -833
- package/skills/mudra-master/mudra-xr/assets/demos/xrpoet.html +0 -151
- package/skills/mudra-master/mudra-xr/assets/samples/depthmap.html +0 -266
- package/skills/mudra-master/mudra-xr/assets/samples/depthmesh.html +0 -100
- package/skills/mudra-master/mudra-xr/assets/samples/game_rps.html +0 -1099
- package/skills/mudra-master/mudra-xr/assets/samples/gestures_custom.html +0 -467
- package/skills/mudra-master/mudra-xr/assets/samples/gestures_heuristic.html +0 -256
- package/skills/mudra-master/mudra-xr/assets/samples/lighting.html +0 -163
- package/skills/mudra-master/mudra-xr/assets/samples/mesh_detection.html +0 -46
- package/skills/mudra-master/mudra-xr/assets/samples/modelviewer.html +0 -183
- package/skills/mudra-master/mudra-xr/assets/samples/paint.html +0 -133
- package/skills/mudra-master/mudra-xr/assets/samples/planar-vst.html +0 -45
- package/skills/mudra-master/mudra-xr/assets/samples/reticle.html +0 -95
- package/skills/mudra-master/mudra-xr/assets/samples/skybox_agent.html +0 -384
- package/skills/mudra-master/mudra-xr/assets/samples/sound.html +0 -390
- package/skills/mudra-master/mudra-xr/assets/samples/ui.html +0 -496
- package/skills/mudra-master/mudra-xr/assets/samples/virtual-screens.html +0 -808
- package/skills/mudra-master/mudra-xr/assets/templates/0_basic.html +0 -64
- package/skills/mudra-master/mudra-xr/assets/templates/1_ui.html +0 -107
- package/skills/mudra-master/mudra-xr/assets/templates/2_hands.html +0 -183
- package/skills/mudra-master/mudra-xr/assets/templates/3_depth.html +0 -89
- package/skills/mudra-master/mudra-xr/assets/templates/4_stereo.html +0 -84
- package/skills/mudra-master/mudra-xr/assets/templates/5_camera.html +0 -117
- package/skills/mudra-master/mudra-xr/assets/templates/6_ai.html +0 -154
- package/skills/mudra-master/mudra-xr/assets/templates/7_ai_live.html +0 -253
- package/skills/mudra-master/mudra-xr/assets/templates/8_objects.html +0 -95
- package/skills/mudra-master/mudra-xr/assets/templates/9_xr-toggle.html +0 -90
- package/skills/mudra-master/mudra-xr/assets/templates/heuristic_hand_gestures.html +0 -102
- package/skills/mudra-master/mudra-xr/assets/templates/meshes.html +0 -47
- package/skills/mudra-master/mudra-xr/assets/templates/planes.html +0 -47
- package/skills/mudra-master/mudra-xr/assets/templates/uikit.html +0 -187
- package/skills/mudra-master/mudra-xr/references/agent_protocol.json +0 -865
- package/skills/mudra-master/mudra-xr/references/prompt.md +0 -1605
- package/skills/mudra-preview/SKILL.md +0 -159
- package/skills/mudra-preview/assets/ar-menu.html +0 -382
- package/skills/mudra-preview/assets/document-scroller.html +0 -468
- package/skills/mudra-preview/assets/drum-machine.html +0 -788
- package/skills/mudra-preview/assets/emg-visualizer.html +0 -390
- package/skills/mudra-preview/assets/generative-art.html +0 -408
- package/skills/mudra-preview/assets/gesture-assistant.html +0 -342
- package/skills/mudra-preview/assets/gesture-speech.html +0 -444
- package/skills/mudra-preview/assets/hands-free-desktop.html +0 -376
- package/skills/mudra-preview/assets/model-rotator.html +0 -365
- package/skills/mudra-preview/assets/mudra-duel.html +0 -3200
- package/skills/mudra-preview/assets/mudra-monitor.html +0 -1451
- package/skills/mudra-preview/assets/mudra-ultimate-template.html +0 -997
- package/skills/mudra-preview/assets/music-sequencer.html +0 -455
- package/skills/mudra-preview/assets/neural-pong.html +0 -380
- package/skills/mudra-preview/assets/neural-snake.html +0 -364
- package/skills/mudra-preview/assets/presentation-controller.html +0 -345
- package/skills/mudra-preview/assets/pressure-painter.html +0 -285
- package/skills/mudra-preview/assets/runner.html +0 -1812
- package/skills/mudra-preview/assets/smart-home.html +0 -461
- package/skills/mudra-preview/assets/space-invaders.html +0 -1154
- package/skills/mudra-preview/assets/waterful-ring-toss.html +0 -1624
- package/skills/mudra-preview/references/agent_protocol.json +0 -701
- package/skills/mudra-preview/references/prompt.md +0 -16731
- package/skills/mudra-xr/SKILL.md +0 -245
- package/skills/mudra-xr/assets/demos/3dgs-walkthrough.html +0 -242
- package/skills/mudra-xr/assets/demos/aisimulator.html +0 -792
- package/skills/mudra-xr/assets/demos/balloonpop.html +0 -422
- package/skills/mudra-xr/assets/demos/ballpit.html +0 -284
- package/skills/mudra-xr/assets/demos/drone.html +0 -41
- package/skills/mudra-xr/assets/demos/gemini-icebreakers.html +0 -338
- package/skills/mudra-xr/assets/demos/gemini-xrobject.html +0 -302
- package/skills/mudra-xr/assets/demos/math3d.html +0 -188
- package/skills/mudra-xr/assets/demos/measure.html +0 -247
- package/skills/mudra-xr/assets/demos/occlusion.html +0 -251
- package/skills/mudra-xr/assets/demos/rain.html +0 -498
- package/skills/mudra-xr/assets/demos/screenwiper.html +0 -432
- package/skills/mudra-xr/assets/demos/splash.html +0 -519
- package/skills/mudra-xr/assets/demos/webcam_gestures.html +0 -253
- package/skills/mudra-xr/assets/demos/xremoji.html +0 -833
- package/skills/mudra-xr/assets/demos/xrpoet.html +0 -151
- package/skills/mudra-xr/assets/samples/depthmap.html +0 -266
- package/skills/mudra-xr/assets/samples/depthmesh.html +0 -100
- package/skills/mudra-xr/assets/samples/game_rps.html +0 -1099
- package/skills/mudra-xr/assets/samples/gestures_custom.html +0 -467
- package/skills/mudra-xr/assets/samples/gestures_heuristic.html +0 -256
- package/skills/mudra-xr/assets/samples/lighting.html +0 -163
- package/skills/mudra-xr/assets/samples/mesh_detection.html +0 -46
- package/skills/mudra-xr/assets/samples/modelviewer.html +0 -183
- package/skills/mudra-xr/assets/samples/paint.html +0 -133
- package/skills/mudra-xr/assets/samples/planar-vst.html +0 -45
- package/skills/mudra-xr/assets/samples/reticle.html +0 -95
- package/skills/mudra-xr/assets/samples/skybox_agent.html +0 -384
- package/skills/mudra-xr/assets/samples/sound.html +0 -390
- package/skills/mudra-xr/assets/samples/ui.html +0 -496
- package/skills/mudra-xr/assets/samples/virtual-screens.html +0 -808
- package/skills/mudra-xr/assets/templates/0_basic.html +0 -64
- package/skills/mudra-xr/assets/templates/1_ui.html +0 -107
- package/skills/mudra-xr/assets/templates/2_hands.html +0 -183
- package/skills/mudra-xr/assets/templates/3_depth.html +0 -89
- package/skills/mudra-xr/assets/templates/4_stereo.html +0 -84
- package/skills/mudra-xr/assets/templates/5_camera.html +0 -117
- package/skills/mudra-xr/assets/templates/6_ai.html +0 -154
- package/skills/mudra-xr/assets/templates/7_ai_live.html +0 -253
- package/skills/mudra-xr/assets/templates/8_objects.html +0 -95
- package/skills/mudra-xr/assets/templates/9_xr-toggle.html +0 -90
- package/skills/mudra-xr/assets/templates/heuristic_hand_gestures.html +0 -102
- package/skills/mudra-xr/assets/templates/meshes.html +0 -47
- package/skills/mudra-xr/assets/templates/planes.html +0 -47
- package/skills/mudra-xr/assets/templates/uikit.html +0 -187
- package/skills/mudra-xr/references/agent_protocol.json +0 -865
- package/skills/mudra-xr/references/prompt.md +0 -1605
|
@@ -1,1605 +0,0 @@
|
|
|
1
|
-
# Mudra XR Skill — Build Rules
|
|
2
|
-
|
|
3
|
-
This document is the canonical reference for the `mudra-xr` Claude Code skill.
|
|
4
|
-
Read it in full before generating any app. Every rule below is non-negotiable
|
|
5
|
-
unless explicitly marked as configurable.
|
|
6
|
-
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
## Section 1 — Mudra Protocol
|
|
10
|
-
|
|
11
|
-
### WebSocket endpoint
|
|
12
|
-
|
|
13
|
-
```
|
|
14
|
-
ws://127.0.0.1:8766
|
|
15
|
-
```
|
|
16
|
-
|
|
17
|
-
Always construct the connection through `MudraClient` (Section 4).
|
|
18
|
-
Never use raw `new WebSocket(...)`.
|
|
19
|
-
|
|
20
|
-
### Eight canonical signals
|
|
21
|
-
|
|
22
|
-
| Signal | Category | Description |
|
|
23
|
-
|--------|----------|-------------|
|
|
24
|
-
| `gesture` | Discrete | Hand gesture events (tap, double_tap, twist, double_twist) |
|
|
25
|
-
| `button` | Discrete | Button hold / release |
|
|
26
|
-
| `pressure` | Analog | Finger pressure 0–100, normalized 0–1 |
|
|
27
|
-
| `navigation` | Motion (Pointer) | Continuous delta_x / delta_y cursor movement |
|
|
28
|
-
| `nav_direction` | Motion (Direction) | Discrete directional swipes: None, Right, Left, Up, Down, Roll Left, Roll Right |
|
|
29
|
-
| `imu_acc` | Motion (IMU) | Accelerometer values [x, y, z] m/s², frequency 1125 Hz |
|
|
30
|
-
| `imu_gyro` | Motion (IMU) | Gyroscope values [x, y, z] deg/s, frequency 1125 Hz |
|
|
31
|
-
| `snc` | Biometric | 3 de-interleaved channel arrays [[ch1], [ch2], [ch3]] |
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
### Subscription handshake
|
|
35
|
-
|
|
36
|
-
Send one command per signal — never use plural `signals`, arrays, or batch commands:
|
|
37
|
-
|
|
38
|
-
```js
|
|
39
|
-
// CORRECT
|
|
40
|
-
ws.send(JSON.stringify({ command: 'subscribe', signal: 'gesture' }));
|
|
41
|
-
ws.send(JSON.stringify({ command: 'subscribe', signal: 'pressure' }));
|
|
42
|
-
|
|
43
|
-
// WRONG — never do this
|
|
44
|
-
ws.send(JSON.stringify({ command: 'subscribe', signals: ['gesture', 'pressure'] }));
|
|
45
|
-
ws.send(JSON.stringify({ command: 'subscribe', signal: ['gesture', 'pressure'] }));
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
### Full command surface
|
|
49
|
-
|
|
50
|
-
`subscribe`, `unsubscribe`, `get_subscriptions`, `get_status`, `trigger_gesture`
|
|
51
|
-
|
|
52
|
-
**Removed**: `enable`, `disable`, `get_docs` — the new Dart server rejects them with `unknown_command`.
|
|
53
|
-
|
|
54
|
-
### Inbound message payload shapes
|
|
55
|
-
|
|
56
|
-
```js
|
|
57
|
-
// gesture — no confidence field
|
|
58
|
-
{ type: 'gesture', data: { type: 'tap'|'double_tap'|'twist'|'double_twist', timestamp }, timestamp }
|
|
59
|
-
|
|
60
|
-
// button
|
|
61
|
-
{ type: 'button', data: { state: 'pressed'|'released', timestamp }, timestamp }
|
|
62
|
-
|
|
63
|
-
// pressure
|
|
64
|
-
{ type: 'pressure', data: { value: 0–100, normalized: 0–1, timestamp }, timestamp }
|
|
65
|
-
|
|
66
|
-
// navigation
|
|
67
|
-
{ type: 'navigation', data: { delta_x: number, delta_y: number, timestamp }, timestamp }
|
|
68
|
-
|
|
69
|
-
// nav_direction
|
|
70
|
-
{ type: 'nav_direction', data: { direction: 'Right'|'Left'|'Up'|'Down'|'Roll Left'|'Roll Right'|'None', timestamp }, timestamp }
|
|
71
|
-
|
|
72
|
-
// imu_acc
|
|
73
|
-
{ type: 'imu_acc', data: { values: [x, y, z], frequency: number, frequency_std: number, timestamp }, timestamp }
|
|
74
|
-
|
|
75
|
-
// imu_gyro
|
|
76
|
-
{ type: 'imu_gyro', data: { values: [x, y, z], frequency: number, frequency_std: number, timestamp }, timestamp }
|
|
77
|
-
|
|
78
|
-
// snc — extend rolling buffers (500 samples/channel) with all samples per callback
|
|
79
|
-
{ type: 'snc', data: { values: [[ch1_samples], [ch2_samples], [ch3_samples]], frequency: number, frequency_std: number, timestamp }, timestamp }
|
|
80
|
-
|
|
81
|
-
// status — response to get_status command
|
|
82
|
-
{ type: 'status', data: { device: { name, address, battery, charging, firmware, serial_number, hand, state, firmware_config: { target, active } }, subscriptions: { snc, imu_acc, imu_gyro, pressure, gesture, navigation, nav_direction, button } }, timestamp }
|
|
83
|
-
|
|
84
|
-
// subscription_status — response to subscribe/unsubscribe
|
|
85
|
-
{ type: 'subscription_status', data: { signal: string, subscribed: boolean }, timestamp }
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
**Removed**: `confidence` field from gesture (not emitted by Dart server).
|
|
89
|
-
|
|
90
|
-
**Legacy / not emitted by Dart server**: `connection_status` frames were sent by the old Python server on connect. The new Dart server sends **no unsolicited frames** on open — do NOT gate any UI on receiving `connection_status`.
|
|
91
|
-
|
|
92
|
-
### Error frames
|
|
93
|
-
|
|
94
|
-
The server emits `{type:"error", data:{error:"<code>", message:"..."}, timestamp}` for protocol violations. Codes generated apps may receive:
|
|
95
|
-
|
|
96
|
-
| `data.error` code | Meaning | App must do |
|
|
97
|
-
|---|---|---|
|
|
98
|
-
| `invalid_json` | Malformed JSON sent | Log; never expected from generated apps |
|
|
99
|
-
| `missing_command` | No `command` field | Log; never expected |
|
|
100
|
-
| `unknown_command` | Command not in the supported set | Log; never expected after spec 001 |
|
|
101
|
-
| `missing_signal` | `subscribe`/`unsubscribe` without `signal` | Log; never expected |
|
|
102
|
-
| `invalid_signal` | Unrecognised signal name | Log; never expected after spec 001 |
|
|
103
|
-
| `client_already_connected` | Another tab/process owns the single client slot | **Show terminal "close the other tab" message; do NOT retry** |
|
|
104
|
-
|
|
105
|
-
For any unexpected error code: log and remain in current state. Do not crash the scene.
|
|
106
|
-
|
|
107
|
-
---
|
|
108
|
-
|
|
109
|
-
## Section 2 — Canonical Dependency Pins
|
|
110
|
-
|
|
111
|
-
Use the exact versions below. Never use `@latest` or version ranges.
|
|
112
|
-
Import map `<script type="importmap">` must contain **only** the dependencies
|
|
113
|
-
the app actually uses — no unused entries.
|
|
114
|
-
|
|
115
|
-
```json
|
|
116
|
-
{
|
|
117
|
-
"imports": {
|
|
118
|
-
"three": "https://cdn.jsdelivr.net/npm/three@0.182.0/build/three.module.js",
|
|
119
|
-
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.182.0/examples/jsm/",
|
|
120
|
-
"troika-three-text": "https://cdn.jsdelivr.net/gh/protectwise/troika@028b81cf308f0f22e5aa8e78196be56ec1997af5/packages/troika-three-text/src/index.js",
|
|
121
|
-
"troika-three-utils": "https://cdn.jsdelivr.net/gh/protectwise/troika@v0.52.4/packages/troika-three-utils/src/index.js",
|
|
122
|
-
"troika-worker-utils": "https://cdn.jsdelivr.net/gh/protectwise/troika@v0.52.4/packages/troika-worker-utils/src/index.js",
|
|
123
|
-
"bidi-js": "https://esm.sh/bidi-js@%5E1.0.2?target=es2022",
|
|
124
|
-
"webgl-sdf-generator": "https://esm.sh/webgl-sdf-generator@1.1.1/es2022/webgl-sdf-generator.mjs",
|
|
125
|
-
"lit": "https://cdn.jsdelivr.net/gh/lit/dist@3/core/lit-core.min.js",
|
|
126
|
-
"lit/": "https://esm.run/lit@3/",
|
|
127
|
-
"xrblocks": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/xrblocks.js",
|
|
128
|
-
"xrblocks/addons/": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/addons/"
|
|
129
|
-
}
|
|
130
|
-
}
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
XR Blocks stylesheet (always `<link>` in `<head>`):
|
|
134
|
-
|
|
135
|
-
```html
|
|
136
|
-
<link type="text/css" rel="stylesheet" href="https://xrblocks.github.io/css/xr.css" />
|
|
137
|
-
```
|
|
138
|
-
|
|
139
|
-
Lit bundle warning suppression (place before import map):
|
|
140
|
-
|
|
141
|
-
```html
|
|
142
|
-
<script>window.litDisableBundleWarning = true;</script>
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
---
|
|
146
|
-
|
|
147
|
-
## Section 3 — XR Blocks Lifecycle Composition
|
|
148
|
-
|
|
149
|
-
### Module-scope MudraClient
|
|
150
|
-
|
|
151
|
-
Instantiate `MudraClient` exactly once at module scope — not inside `init()`.
|
|
152
|
-
The constructor does NOT open a WebSocket; connection is driven by `mudra.setMode("mudra")` only (Section 15). Module-scope instantiation ensures the `MudraClient` object is ready before the XR scene initializes.
|
|
153
|
-
|
|
154
|
-
```js
|
|
155
|
-
// Module scope — outside any class (no URL arg; URL is baked into MudraClient per Section 4)
|
|
156
|
-
const mudra = new MudraClient();
|
|
157
|
-
|
|
158
|
-
class MainScript extends xb.Script {
|
|
159
|
-
init() {
|
|
160
|
-
// Register signal handlers and subscribe here, AFTER scene objects are created
|
|
161
|
-
mudra.on('gesture', (data) => { /* handle tap / twist / etc. */ });
|
|
162
|
-
mudra.on('pressure', (data) => { /* handle finger pressure */ });
|
|
163
|
-
mudra.subscribe('gesture');
|
|
164
|
-
mudra.subscribe('pressure');
|
|
165
|
-
}
|
|
166
|
-
update() { /* called every frame by xb */ }
|
|
167
|
-
}
|
|
168
|
-
```
|
|
169
|
-
|
|
170
|
-
### xb.Script class structure
|
|
171
|
-
|
|
172
|
-
Every generated app's top-level logic lives inside a class extending `xb.Script`.
|
|
173
|
-
The class name may be anything (`MainScript`, `CubeApp`, etc.) but must extend `xb.Script`.
|
|
174
|
-
|
|
175
|
-
```js
|
|
176
|
-
class MainScript extends xb.Script {
|
|
177
|
-
/** Called once after XR Blocks and the WebGL context are ready. */
|
|
178
|
-
init() {
|
|
179
|
-
// Add lights, meshes, Mudra bindings here.
|
|
180
|
-
this.add(new THREE.HemisphereLight(0xffffff, 0x666666, 3));
|
|
181
|
-
// Place objects at xb.user.objectDistance in front of viewer:
|
|
182
|
-
this.mesh.position.set(0, xb.user.height - 0.5, -xb.user.objectDistance);
|
|
183
|
-
}
|
|
184
|
-
|
|
185
|
-
/** Called every animation frame. Keep cheap — no allocations. */
|
|
186
|
-
update() { }
|
|
187
|
-
|
|
188
|
-
/** Fires when a pinch/controller-trigger starts in XR. */
|
|
189
|
-
onSelectStart(event) { }
|
|
190
|
-
|
|
191
|
-
/** Fires when a pinch/controller-trigger ends in XR. */
|
|
192
|
-
onSelectEnd(event) { }
|
|
193
|
-
|
|
194
|
-
/** Fires every frame while a pinch is held in XR. */
|
|
195
|
-
onSelecting(event) { }
|
|
196
|
-
}
|
|
197
|
-
```
|
|
198
|
-
|
|
199
|
-
### Entry point
|
|
200
|
-
|
|
201
|
-
```js
|
|
202
|
-
document.addEventListener('DOMContentLoaded', function () {
|
|
203
|
-
xb.add(new MainScript());
|
|
204
|
-
xb.init(new xb.Options());
|
|
205
|
-
});
|
|
206
|
-
```
|
|
207
|
-
|
|
208
|
-
### Key spatial constants
|
|
209
|
-
|
|
210
|
-
- `xb.user.height` — floor-relative eye height (approx. 1.6 m)
|
|
211
|
-
- `xb.user.objectDistance` — recommended arm-length distance for objects (approx. 0.8 m)
|
|
212
|
-
- Y-up coordinate system; Z is toward the viewer (negative Z = in front of you)
|
|
213
|
-
|
|
214
|
-
---
|
|
215
|
-
|
|
216
|
-
## Section 4 — MudraClient (WebSocket + Mode Toggle glue)
|
|
217
|
-
|
|
218
|
-
### Policy
|
|
219
|
-
|
|
220
|
-
- The constructor does **NOT** open a WebSocket. Connection is driven by `setMode("mudra")` only.
|
|
221
|
-
- In Manual mode the sim panel and keyboard shortcuts inject synthetic signals through `_emit()` — the exact same handler path as real WebSocket messages. No auto-firing timers. No mock auto-activation.
|
|
222
|
-
- In Mudra mode a real WebSocket opens to `ws://127.0.0.1:8766`. On successful open: replay the subscription record + send `get_status` + start the 2 s status-poll.
|
|
223
|
-
- If the socket closes for a non-conflict reason: schedule reconnect with backoff `[1 s, 2 s, 5 s, 10 s]` (capped). The `_suppressReconnect` flag prevents retry on `client_already_connected`.
|
|
224
|
-
- If `client_already_connected` error arrives: transition to `already-in-use`; set `_suppressReconnect = true`; do not retry.
|
|
225
|
-
- The 6-state `ConnectionState` vocabulary and status-pill wiring live in **Section 15** (canonical). This section documents only the `MudraClient` class itself.
|
|
226
|
-
|
|
227
|
-
### Connection-status state machine
|
|
228
|
-
|
|
229
|
-
See Section 15 for the full 6-state transition diagram. Vocabulary:
|
|
230
|
-
|
|
231
|
-
```text
|
|
232
|
-
idle | connecting | connected | ws-only | reconnecting | already-in-use
|
|
233
|
-
```
|
|
234
|
-
|
|
235
|
-
`simulated` and `disconnected-simulated` are **removed**. No auto-mock activation.
|
|
236
|
-
|
|
237
|
-
### Required MudraClient implementation
|
|
238
|
-
|
|
239
|
-
Include this class verbatim in every generated app:
|
|
240
|
-
|
|
241
|
-
```js
|
|
242
|
-
class MudraClient {
|
|
243
|
-
static _WS_URL = 'ws://127.0.0.1:8766';
|
|
244
|
-
static _BACKOFF = [1000, 2000, 5000, 10000]; // ms, last value is the cap
|
|
245
|
-
|
|
246
|
-
constructor() {
|
|
247
|
-
this._handlers = {};
|
|
248
|
-
this._subscriptions = new Set(); // subscription record — replayed on reconnect
|
|
249
|
-
this._ws = null;
|
|
250
|
-
this._mode = 'manual';
|
|
251
|
-
this._connState = 'idle';
|
|
252
|
-
this._pollTimer = null;
|
|
253
|
-
this._reconnectTimer = null;
|
|
254
|
-
this._backoffIdx = 0;
|
|
255
|
-
this._suppressReconnect = false;
|
|
256
|
-
this._connToken = 0; // incremented on every openSocket() to cancel stale callbacks
|
|
257
|
-
}
|
|
258
|
-
|
|
259
|
-
/** Register a handler for a signal type or _status events. Call before subscribe(). */
|
|
260
|
-
on(signal, fn) { this._handlers[signal] = fn; }
|
|
261
|
-
|
|
262
|
-
/** Add signal to subscription record. If WS is open, send subscribe immediately. */
|
|
263
|
-
subscribe(signal) {
|
|
264
|
-
this._subscriptions.add(signal);
|
|
265
|
-
if (this._ws && this._ws.readyState === WebSocket.OPEN) {
|
|
266
|
-
this._ws.send(JSON.stringify({ command: 'subscribe', signal }));
|
|
267
|
-
}
|
|
268
|
-
}
|
|
269
|
-
|
|
270
|
-
/** Remove signal from subscription record. If WS is open, send unsubscribe. */
|
|
271
|
-
unsubscribe(signal) {
|
|
272
|
-
this._subscriptions.delete(signal);
|
|
273
|
-
if (this._ws && this._ws.readyState === WebSocket.OPEN) {
|
|
274
|
-
this._ws.send(JSON.stringify({ command: 'unsubscribe', signal }));
|
|
275
|
-
}
|
|
276
|
-
}
|
|
277
|
-
|
|
278
|
-
/** Send an arbitrary command. In Manual mode, trigger_gesture dispatches via _emit(). */
|
|
279
|
-
send(cmd) {
|
|
280
|
-
if (this._ws && this._ws.readyState === WebSocket.OPEN) {
|
|
281
|
-
this._ws.send(JSON.stringify(cmd));
|
|
282
|
-
} else if (cmd.command === 'trigger_gesture') {
|
|
283
|
-
this._dispatchMockGesture(cmd.data?.type ?? 'tap');
|
|
284
|
-
}
|
|
285
|
-
}
|
|
286
|
-
|
|
287
|
-
/** Switch mode. setMode("mudra") opens the socket; setMode("manual") closes it. */
|
|
288
|
-
setMode(mode) {
|
|
289
|
-
if (this._mode === mode) return;
|
|
290
|
-
this._mode = mode;
|
|
291
|
-
if (mode === 'manual') {
|
|
292
|
-
this._closeSocket();
|
|
293
|
-
this._setState('idle');
|
|
294
|
-
} else {
|
|
295
|
-
this._suppressReconnect = false;
|
|
296
|
-
this._backoffIdx = 0;
|
|
297
|
-
this._openSocket();
|
|
298
|
-
}
|
|
299
|
-
}
|
|
300
|
-
|
|
301
|
-
get status() { return this._connState; }
|
|
302
|
-
|
|
303
|
-
_setState(s) {
|
|
304
|
-
this._connState = s;
|
|
305
|
-
if (this._handlers['_status']) this._handlers['_status'](s);
|
|
306
|
-
}
|
|
307
|
-
|
|
308
|
-
_emit(payload) {
|
|
309
|
-
if (this._handlers[payload.type]) this._handlers[payload.type](payload.data);
|
|
310
|
-
}
|
|
311
|
-
|
|
312
|
-
/** Called by sim-panel gesture buttons and keyboard Space shortcut. */
|
|
313
|
-
_dispatchMockGesture(type) {
|
|
314
|
-
this._emit({ type: 'gesture', data: { type, timestamp: Date.now() } });
|
|
315
|
-
}
|
|
316
|
-
|
|
317
|
-
_openSocket() {
|
|
318
|
-
const token = ++this._connToken;
|
|
319
|
-
this._setState('connecting');
|
|
320
|
-
try {
|
|
321
|
-
const ws = new WebSocket(MudraClient._WS_URL);
|
|
322
|
-
this._ws = ws;
|
|
323
|
-
|
|
324
|
-
ws.onopen = () => {
|
|
325
|
-
if (this._connToken !== token) return;
|
|
326
|
-
this._backoffIdx = 0;
|
|
327
|
-
this._replaySubscriptions();
|
|
328
|
-
ws.send(JSON.stringify({ command: 'get_status' }));
|
|
329
|
-
this._startPoll();
|
|
330
|
-
};
|
|
331
|
-
|
|
332
|
-
ws.onmessage = (e) => {
|
|
333
|
-
if (this._connToken !== token) return;
|
|
334
|
-
let msg;
|
|
335
|
-
try { msg = JSON.parse(e.data); } catch { return; }
|
|
336
|
-
|
|
337
|
-
if (msg.type === 'error' && msg.data?.error === 'client_already_connected') {
|
|
338
|
-
this._suppressReconnect = true;
|
|
339
|
-
this._stopPoll();
|
|
340
|
-
this._setState('already-in-use');
|
|
341
|
-
return;
|
|
342
|
-
}
|
|
343
|
-
|
|
344
|
-
if (msg.type === 'status') {
|
|
345
|
-
const d = msg.data?.device;
|
|
346
|
-
const bandConnected = !!(d?.firmware && d?.serial_number);
|
|
347
|
-
if (bandConnected) {
|
|
348
|
-
if (this._connState !== 'connected') {
|
|
349
|
-
this._setState('connected');
|
|
350
|
-
this._replaySubscriptions(); // FR-012: re-subscribe on band reconnect
|
|
351
|
-
}
|
|
352
|
-
// expose hand via a synthetic 'hand' event for the UI chip
|
|
353
|
-
if (this._handlers['_hand']) {
|
|
354
|
-
const raw = d.hand ?? '';
|
|
355
|
-
const hand = /^(LEFT|RIGHT)$/i.test(raw) ? raw.toUpperCase() : 'None';
|
|
356
|
-
this._handlers['_hand'](hand);
|
|
357
|
-
}
|
|
358
|
-
} else {
|
|
359
|
-
if (this._connState !== 'ws-only') this._setState('ws-only');
|
|
360
|
-
if (this._handlers['_hand']) this._handlers['_hand']('None');
|
|
361
|
-
}
|
|
362
|
-
return;
|
|
363
|
-
}
|
|
364
|
-
|
|
365
|
-
if (this._handlers[msg.type]) this._handlers[msg.type](msg.data);
|
|
366
|
-
};
|
|
367
|
-
|
|
368
|
-
ws.onerror = () => { /* onclose fires after onerror; handle there */ };
|
|
369
|
-
|
|
370
|
-
ws.onclose = () => {
|
|
371
|
-
if (this._connToken !== token) return;
|
|
372
|
-
this._stopPoll();
|
|
373
|
-
if (this._suppressReconnect || this._mode !== 'mudra') return;
|
|
374
|
-
this._setState('reconnecting');
|
|
375
|
-
const delay = MudraClient._BACKOFF[Math.min(this._backoffIdx, MudraClient._BACKOFF.length - 1)];
|
|
376
|
-
this._backoffIdx = Math.min(this._backoffIdx + 1, MudraClient._BACKOFF.length - 1);
|
|
377
|
-
this._reconnectTimer = setTimeout(() => {
|
|
378
|
-
if (this._mode === 'mudra' && !this._suppressReconnect) this._openSocket();
|
|
379
|
-
}, delay);
|
|
380
|
-
};
|
|
381
|
-
} catch (_) {
|
|
382
|
-
this._setState('reconnecting');
|
|
383
|
-
}
|
|
384
|
-
}
|
|
385
|
-
|
|
386
|
-
_closeSocket() {
|
|
387
|
-
this._stopPoll();
|
|
388
|
-
clearTimeout(this._reconnectTimer);
|
|
389
|
-
this._reconnectTimer = null;
|
|
390
|
-
this._connToken++; // invalidate all callbacks from the old socket
|
|
391
|
-
if (this._ws) { try { this._ws.close(); } catch (_) {} this._ws = null; }
|
|
392
|
-
}
|
|
393
|
-
|
|
394
|
-
_startPoll() {
|
|
395
|
-
this._stopPoll();
|
|
396
|
-
this._pollTimer = setInterval(() => {
|
|
397
|
-
if (this._ws && this._ws.readyState === WebSocket.OPEN) {
|
|
398
|
-
this._ws.send(JSON.stringify({ command: 'get_status' }));
|
|
399
|
-
}
|
|
400
|
-
}, 2000);
|
|
401
|
-
}
|
|
402
|
-
|
|
403
|
-
_stopPoll() {
|
|
404
|
-
clearInterval(this._pollTimer);
|
|
405
|
-
this._pollTimer = null;
|
|
406
|
-
}
|
|
407
|
-
|
|
408
|
-
/** Replay the subscription record. Called on open and on band reconnect. */
|
|
409
|
-
_replaySubscriptions() {
|
|
410
|
-
if (!this._ws || this._ws.readyState !== WebSocket.OPEN) return;
|
|
411
|
-
this._subscriptions.forEach(sig =>
|
|
412
|
-
this._ws.send(JSON.stringify({ command: 'subscribe', signal: sig }))
|
|
413
|
-
);
|
|
414
|
-
}
|
|
415
|
-
|
|
416
|
-
destroy() {
|
|
417
|
-
this._closeSocket();
|
|
418
|
-
}
|
|
419
|
-
}
|
|
420
|
-
```
|
|
421
|
-
|
|
422
|
-
### Section 4 note on "no fallback"
|
|
423
|
-
|
|
424
|
-
**"No fallback"** means: no automatic transition to a "Simulated" or "Disconnected — simulated" state when the WebSocket is unavailable. If the socket fails, the pill shows `Reconnecting…` and retries — it never auto-activates synthetic data injection.
|
|
425
|
-
|
|
426
|
-
Manual mode signal injection (sim panel + keyboard shortcuts calling `_emit()`) is **preserved** — this is Principle IV compliance, not a fallback. Sim-panel signals use the same `_emit()` path as real WebSocket messages, so the app does not branch on whether it is simulated or not.
|
|
427
|
-
|
|
428
|
-
---
|
|
429
|
-
|
|
430
|
-
## Section 5 — Simulator Panel
|
|
431
|
-
|
|
432
|
-
### Purpose
|
|
433
|
-
|
|
434
|
-
The simulator panel lets a user exercise every subscribed Mudra signal without
|
|
435
|
-
a band or XR headset. It is a **2D DOM overlay** (not spatial XR UI) and must
|
|
436
|
-
always be visible in flat-screen mode. It disappears automatically when the
|
|
437
|
-
browser transitions to an immersive WebXR session (native WebXR DOM suppression).
|
|
438
|
-
|
|
439
|
-
### DOM structure
|
|
440
|
-
|
|
441
|
-
```html
|
|
442
|
-
<div id="mudra-sim" style="
|
|
443
|
-
position: fixed; bottom: 0; left: 0; right: 0;
|
|
444
|
-
background: rgba(0,0,0,0.75); backdrop-filter: blur(4px);
|
|
445
|
-
padding: 8px 12px; display: flex; flex-wrap: wrap; gap: 8px;
|
|
446
|
-
z-index: 9999; font-family: system-ui, sans-serif;">
|
|
447
|
-
<!-- One button group per subscribed signal -->
|
|
448
|
-
</div>
|
|
449
|
-
```
|
|
450
|
-
|
|
451
|
-
### Button groups per signal (render ONLY the sub-actions the app actually uses)
|
|
452
|
-
|
|
453
|
-
The catalog below is the **maximal** set per signal. The simulator panel
|
|
454
|
-
MUST render **only** the sub-actions the generated app handles — never
|
|
455
|
-
extras. Examples:
|
|
456
|
-
|
|
457
|
-
- App maps `gesture.tap` only → render `Tap`. Omit `2Tap`, `Twist`, `2Twist`.
|
|
458
|
-
- App maps `nav_direction` to Up/Down only → render `↑`, `↓`. Omit
|
|
459
|
-
`←`, `→`, `Roll L`, `Roll R`.
|
|
460
|
-
- App uses `imu_acc` for X-axis tilt only → render `Tilt X+`, `Tilt X−`.
|
|
461
|
-
Omit `Tilt Y+`, `Tilt Y−`.
|
|
462
|
-
|
|
463
|
-
When in doubt, walk every `mudra.on('<signal>', …)` handler and emit a
|
|
464
|
-
button only for sub-actions referenced inside it. Unused buttons are a
|
|
465
|
-
checklist failure (see Section 10, item 6).
|
|
466
|
-
|
|
467
|
-
| Signal | Maximal buttons (subset based on app handlers) |
|
|
468
|
-
|--------|---------|
|
|
469
|
-
| `gesture` | `Tap`, `2Tap`, `Twist`, `2Twist` |
|
|
470
|
-
| `nav_direction` | `↑`, `↓`, `←`, `→`, `Roll L`, `Roll R` |
|
|
471
|
-
| `navigation` | `↑`, `↓`, `←`, `→` (each emits one delta event of ±3 — gentle/low-sensitivity default; raise per-app only if the prompt explicitly asks for fast/snappy movement) |
|
|
472
|
-
| `pressure` | Slider `0–100` (label shows current value) |
|
|
473
|
-
| `button` | `Press`, `Release` |
|
|
474
|
-
| `imu_acc` | `Tilt X+`, `Tilt X−`, `Tilt Y+`, `Tilt Y−` (5-frame burst at ±2 m/s²) |
|
|
475
|
-
| `imu_gyro` | `Rot X+`, `Rot X−`, `Rot Y+`, `Rot Y−` (5-frame burst at ±10 deg/s) |
|
|
476
|
-
| `snc` | `Spike` (burst of elevated samples on all 3 channels) |
|
|
477
|
-
|
|
478
|
-
### Button firing rules
|
|
479
|
-
|
|
480
|
-
- Every button fires via the **same code path** as a real Mudra signal.
|
|
481
|
-
Call the same handler that `mudra.on(signal, handler)` would invoke.
|
|
482
|
-
- For `gesture` buttons: also call `mudra.send({ command: 'trigger_gesture', data: { type } })`
|
|
483
|
-
so the round-trip works when a real band is connected.
|
|
484
|
-
- Never duplicate logic between the simulator path and the real-signal path.
|
|
485
|
-
|
|
486
|
-
```js
|
|
487
|
-
// Example: gesture sim button wires to the same handler as real signals
|
|
488
|
-
function simGesture(type) {
|
|
489
|
-
mudra.send({ command: 'trigger_gesture', data: { type } });
|
|
490
|
-
handleGesture({ type, timestamp: Date.now() });
|
|
491
|
-
}
|
|
492
|
-
|
|
493
|
-
// Example: pressure slider
|
|
494
|
-
pressureSlider.addEventListener('input', () => {
|
|
495
|
-
const norm = pressureSlider.value / 100;
|
|
496
|
-
handlePressure({ value: +pressureSlider.value, normalized: norm, timestamp: Date.now() });
|
|
497
|
-
});
|
|
498
|
-
```
|
|
499
|
-
|
|
500
|
-
### Visibility
|
|
501
|
-
|
|
502
|
-
- `#mudra-sim` is always visible in flat-screen mode.
|
|
503
|
-
- Do NOT conditionally hide it when the band is connected (both sources are valid per FR-014).
|
|
504
|
-
- The 2D DOM disappears automatically in immersive WebXR — no JS required.
|
|
505
|
-
|
|
506
|
-
---
|
|
507
|
-
|
|
508
|
-
## Section 6 — Keyboard Shortcuts + XR Blocks Precedence
|
|
509
|
-
|
|
510
|
-
### Canonical keyboard map
|
|
511
|
-
|
|
512
|
-
| Key | Signal / Action |
|
|
513
|
-
|-----|----------------|
|
|
514
|
-
| `Space` | `gesture` → tap |
|
|
515
|
-
| `Shift` | `button` → press (keydown) / release (keyup) |
|
|
516
|
-
| `[` | `pressure` decrease (−10, min 0) |
|
|
517
|
-
| `]` | `pressure` increase (+10, max 100) |
|
|
518
|
-
| `i` | `nav_direction` → Up OR `navigation` delta_y +3 |
|
|
519
|
-
| `k` | `nav_direction` → Down OR `navigation` delta_y −3 |
|
|
520
|
-
| `j` | `nav_direction` → Left OR `navigation` delta_x −3 |
|
|
521
|
-
| `l` | `nav_direction` → Right OR `navigation` delta_x +3 |
|
|
522
|
-
|
|
523
|
-
**Navigation sensitivity default**: keyboard `I`/`J`/`K`/`L` and sim panel
|
|
524
|
-
buttons emit deltas of **±3** per event (gentle / low-sensitivity
|
|
525
|
-
baseline). This keeps cursor/pan motion calm and predictable. Only raise
|
|
526
|
-
the magnitude when the prompt explicitly calls for fast/snappy movement
|
|
527
|
-
(racing, arcade-twitch, etc.).
|
|
528
|
-
| `u` | `imu_acc` tilt X+ burst |
|
|
529
|
-
| `o` | `imu_acc` tilt X− burst |
|
|
530
|
-
| `m` | `imu_gyro` rot Y+ burst |
|
|
531
|
-
| `n` | `imu_gyro` rot Y− burst |
|
|
532
|
-
|
|
533
|
-
For `imu` bursts: fire 5 synthetic frames at ±[2, 0, 9.81] m/s² (acc) or ±[10, 0, 0.5] deg/s (gyro).
|
|
534
|
-
|
|
535
|
-
### Reserved keys & mouse — owned by XR Blocks desktop simulator
|
|
536
|
-
|
|
537
|
-
The XR Blocks desktop simulator (active whenever the app runs in a flat
|
|
538
|
-
browser without a WebXR session) owns the keys and mouse gestures that
|
|
539
|
-
let the user navigate the 3D scene. Mudra MUST NOT claim or
|
|
540
|
-
`stopPropagation()` any of these — they are how the user orbits, walks,
|
|
541
|
-
and zooms while previewing the app:
|
|
542
|
-
|
|
543
|
-
| Input | XR Blocks role |
|
|
544
|
-
|-------|----------------|
|
|
545
|
-
| `W` `A` `S` `D` | Walk camera forward / left / back / right |
|
|
546
|
-
| `ArrowUp` `ArrowDown` `ArrowLeft` `ArrowRight` | Camera nav (alt to WASD) |
|
|
547
|
-
| `Q` `E` | Camera roll / vertical |
|
|
548
|
-
| `R` | Reset camera pose |
|
|
549
|
-
| Right-click drag | Orbit / look around |
|
|
550
|
-
| Mouse wheel | Zoom in / out |
|
|
551
|
-
|
|
552
|
-
**Why Mudra uses `I`/`J`/`K`/`L` and `U`/`O`/`M`/`N`**: these keys are
|
|
553
|
-
explicitly off the XR Blocks reserved set, so the desktop simulator
|
|
554
|
-
keeps full camera control while the band-driven keyboard shortcuts
|
|
555
|
-
remain available for testing without a band. Never reassign a Mudra
|
|
556
|
-
shortcut onto a reserved key, even when the app does not subscribe to a
|
|
557
|
-
navigation signal.
|
|
558
|
-
|
|
559
|
-
### Attachment rule (critical)
|
|
560
|
-
|
|
561
|
-
Mudra keyboard handlers MUST attach with `capture: true` and call
|
|
562
|
-
`event.stopPropagation()` on every key the app subscribes to. This prevents
|
|
563
|
-
XR Blocks' desktop-simulator bubble-phase listeners from double-firing.
|
|
564
|
-
|
|
565
|
-
```js
|
|
566
|
-
window.addEventListener('keydown', (e) => {
|
|
567
|
-
switch (e.code) {
|
|
568
|
-
case 'Space':
|
|
569
|
-
e.stopPropagation();
|
|
570
|
-
simGesture('tap');
|
|
571
|
-
break;
|
|
572
|
-
case 'BracketLeft':
|
|
573
|
-
e.stopPropagation();
|
|
574
|
-
adjustPressure(-10);
|
|
575
|
-
break;
|
|
576
|
-
case 'BracketRight':
|
|
577
|
-
e.stopPropagation();
|
|
578
|
-
adjustPressure(+10);
|
|
579
|
-
break;
|
|
580
|
-
case 'KeyI':
|
|
581
|
-
e.stopPropagation();
|
|
582
|
-
handleNavDirection({ direction: 'Up', timestamp: Date.now() });
|
|
583
|
-
break;
|
|
584
|
-
// ... etc for subscribed keys only — using I/J/K/L for nav and U/O/M/N for IMU
|
|
585
|
-
}
|
|
586
|
-
}, { capture: true });
|
|
587
|
-
```
|
|
588
|
-
|
|
589
|
-
Only intercept keys for signals the app actually subscribes to. Do NOT
|
|
590
|
-
`stopPropagation` on keys that no Mudra signal handles — XR Blocks needs
|
|
591
|
-
those, especially the reserved set above (WASD, arrows, Q/E/R, mouse).
|
|
592
|
-
|
|
593
|
-
---
|
|
594
|
-
|
|
595
|
-
## Section 7 — Connection-Status Indicator
|
|
596
|
-
|
|
597
|
-
> **Superseded by Section 15.** The status pill is now part of the Mode Toggle chrome and follows the 6-state machine defined there. Read Section 15 for the canonical DOM sketch (`#mudra-status` + `#mudra-hand`), state labels, colour hints, and wiring.
|
|
598
|
-
>
|
|
599
|
-
> The 4-state vocabulary (`connecting` / `connected` / `simulated` / `disconnected-simulated`) is **removed**. New vocabulary: `idle` (Manual) / `Connecting…` / `Connected` / `WebSocket Only` (orange) / `Reconnecting…` / `Companion already in use`.
|
|
600
|
-
>
|
|
601
|
-
> The `#mudra-status` `<div>` still lives at `position: fixed; top: 8px; right: 12px` and disappears automatically in immersive WebXR. See Section 15 for full wiring.
|
|
602
|
-
|
|
603
|
-
---
|
|
604
|
-
|
|
605
|
-
## Section 8 — Signal Grouping Rules (Non-Negotiable)
|
|
606
|
-
|
|
607
|
-
### Signal group classification
|
|
608
|
-
|
|
609
|
-
| Group | Signals | Combine freely with |
|
|
610
|
-
|-------|---------|---------------------|
|
|
611
|
-
| **Pointer** | `navigation`, `button` | `gesture` OR `button` (not `pressure` if `gesture` is used) |
|
|
612
|
-
| **Direction** | `nav_direction` | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
|
|
613
|
-
| **IMU+Biometric** | `imu_acc`, `imu_gyro`, `snc` | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
|
|
614
|
-
| *(none)* | — | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
|
|
615
|
-
|
|
616
|
-
### Bundling rule — IMU+Biometric (CRITICAL)
|
|
617
|
-
|
|
618
|
-
`imu_acc`, `imu_gyro`, and `snc` are an **inseparable bundle**. If the user's prompt
|
|
619
|
-
implies any one of them, subscribe to **all three**. Never subscribe to only one or
|
|
620
|
-
two of them.
|
|
621
|
-
|
|
622
|
-
```js
|
|
623
|
-
// CORRECT — all three always together
|
|
624
|
-
mudra.subscribe('imu_acc');
|
|
625
|
-
mudra.subscribe('imu_gyro');
|
|
626
|
-
mudra.subscribe('snc');
|
|
627
|
-
|
|
628
|
-
// WRONG — partial subscriptions
|
|
629
|
-
mudra.subscribe('snc'); // missing imu_acc and imu_gyro
|
|
630
|
-
mudra.subscribe('imu_acc'); // missing imu_gyro and snc
|
|
631
|
-
```
|
|
632
|
-
|
|
633
|
-
### XOR rules (all non-negotiable)
|
|
634
|
-
|
|
635
|
-
1. **Gesture ⊕ Pressure** — an app may use `gesture` OR `pressure`, never both.
|
|
636
|
-
2. **Navigation ⊕ Nav_direction** — an app may use `navigation` OR `nav_direction`, never both.
|
|
637
|
-
3. **Pointer/Direction ⊕ IMU+Biometric** — `navigation` and `nav_direction` cannot be combined with the IMU+Biometric bundle (`imu_acc`/`imu_gyro`/`snc`).
|
|
638
|
-
|
|
639
|
-
### Illegal combinations
|
|
640
|
-
|
|
641
|
-
```
|
|
642
|
-
// REJECT these signal sets
|
|
643
|
-
gesture + pressure
|
|
644
|
-
navigation + nav_direction
|
|
645
|
-
navigation + imu_acc
|
|
646
|
-
navigation + imu_gyro
|
|
647
|
-
navigation + snc
|
|
648
|
-
nav_direction + imu_acc
|
|
649
|
-
nav_direction + imu_gyro
|
|
650
|
-
nav_direction + snc
|
|
651
|
-
button + nav_direction // button belongs to Pointer mode only
|
|
652
|
-
```
|
|
653
|
-
|
|
654
|
-
### Valid signal sets (examples)
|
|
655
|
-
|
|
656
|
-
```
|
|
657
|
-
gesture + button
|
|
658
|
-
pressure + button
|
|
659
|
-
navigation + button
|
|
660
|
-
navigation + button + gesture
|
|
661
|
-
nav_direction
|
|
662
|
-
nav_direction + pressure + button
|
|
663
|
-
imu_acc + imu_gyro + snc
|
|
664
|
-
imu_acc + imu_gyro + snc + gesture
|
|
665
|
-
imu_acc + imu_gyro + snc + button
|
|
666
|
-
imu_acc + imu_gyro + snc + pressure + button
|
|
667
|
-
```
|
|
668
|
-
|
|
669
|
-
### Inference priority for ties
|
|
670
|
-
|
|
671
|
-
When the user prompt maps to multiple modes equally, ask one disambiguation
|
|
672
|
-
question — do not silently pick one.
|
|
673
|
-
|
|
674
|
-
Default to `direction` mode (`nav_direction`) when there is no navigation
|
|
675
|
-
language in the prompt at all and a motion mode is required by the template.
|
|
676
|
-
|
|
677
|
-
---
|
|
678
|
-
|
|
679
|
-
## Section 9 — AI API Key Handling (onboarding-gated, mandatory for AI apps)
|
|
680
|
-
|
|
681
|
-
### Lifecycle: `sessionStorage` keyed by `mudra.gemini.apiKey`
|
|
682
|
-
|
|
683
|
-
For any generated app that calls a Gemini / LLM endpoint, the API key MUST
|
|
684
|
-
be entered through the **onboarding modal** before the user can use the
|
|
685
|
-
app. The key is stored in `sessionStorage` under the literal key
|
|
686
|
-
`mudra.gemini.apiKey` — it persists across reloads in the same tab and
|
|
687
|
-
clears when the tab closes. **Do NOT use `localStorage`. Do NOT use a
|
|
688
|
-
`prompt()` popup. Do NOT prompt on first AI call.**
|
|
689
|
-
|
|
690
|
-
```js
|
|
691
|
-
// Read at app start
|
|
692
|
-
const apiKey = sessionStorage.getItem('mudra.gemini.apiKey');
|
|
693
|
-
|
|
694
|
-
// Write only from the onboarding "AI Setup" step
|
|
695
|
-
sessionStorage.setItem('mudra.gemini.apiKey', enteredKey);
|
|
696
|
-
```
|
|
697
|
-
|
|
698
|
-
### Required onboarding "AI Setup" fragment
|
|
699
|
-
|
|
700
|
-
For AI apps only, the onboarding modal MUST include the following
|
|
701
|
-
fragment inside `<section class="mudra-onb__body">`, placed AFTER the
|
|
702
|
-
actions table and BEFORE `</section>`:
|
|
703
|
-
|
|
704
|
-
```html
|
|
705
|
-
<div class="mudra-onb__ai" data-uses-ai>
|
|
706
|
-
<h3 class="mudra-onb__ai-title">AI Setup</h3>
|
|
707
|
-
<p class="mudra-onb__ai-lede">
|
|
708
|
-
This app uses Google Gemini. Paste your API key to continue —
|
|
709
|
-
it's stored only in this browser tab (<code>sessionStorage</code>)
|
|
710
|
-
and is never sent anywhere except Google's API.
|
|
711
|
-
<a href="https://aistudio.google.com/" target="_blank" rel="noopener">Get a key →</a>
|
|
712
|
-
</p>
|
|
713
|
-
<input
|
|
714
|
-
id="mudra-onb-ai-key"
|
|
715
|
-
class="mudra-onb__ai-input"
|
|
716
|
-
type="password"
|
|
717
|
-
autocomplete="off"
|
|
718
|
-
spellcheck="false"
|
|
719
|
-
placeholder="Paste Gemini API key (starts with AIza…)"
|
|
720
|
-
aria-label="Gemini API key"
|
|
721
|
-
/>
|
|
722
|
-
<p class="mudra-onb__ai-hint" data-role="hint"></p>
|
|
723
|
-
</div>
|
|
724
|
-
```
|
|
725
|
-
|
|
726
|
-
And the matching CSS (added to the existing `<style>`):
|
|
727
|
-
|
|
728
|
-
```css
|
|
729
|
-
.mudra-onb__ai { margin-top: 14px; padding-top: 12px; border-top: 1px solid #eee; }
|
|
730
|
-
.mudra-onb__ai-title { margin: 0 0 6px; font-size: 0.95rem; font-weight: 700; color: #111; }
|
|
731
|
-
.mudra-onb__ai-lede { margin: 0 0 10px; color: #555; font-size: 0.85rem; }
|
|
732
|
-
.mudra-onb__ai-lede a { color: #0d9488; }
|
|
733
|
-
.mudra-onb__ai-input {
|
|
734
|
-
width: 100%; box-sizing: border-box;
|
|
735
|
-
padding: 9px 12px; border: 1px solid #d0d0d0; border-radius: 8px;
|
|
736
|
-
font: 13px/1.4 ui-monospace, SFMono-Regular, Menlo, monospace;
|
|
737
|
-
background: #f8f9fb; color: #111;
|
|
738
|
-
}
|
|
739
|
-
.mudra-onb__ai-input:focus { outline: 2px solid #14b8a6; border-color: #14b8a6; background: #fff; }
|
|
740
|
-
.mudra-onb__ai-hint { margin: 6px 2px 0; font-size: 0.75rem; color: #b91c1c; min-height: 1em; }
|
|
741
|
-
.mudra-onb__continue:disabled { opacity: 0.45; cursor: not-allowed; }
|
|
742
|
-
```
|
|
743
|
-
|
|
744
|
-
### Behaviour rules (AI apps only)
|
|
745
|
-
|
|
746
|
-
1. **`Got it` button starts disabled.** The IIFE that wires the modal
|
|
747
|
-
reads `dialog.querySelector('.mudra-onb__ai')` — if it exists, the
|
|
748
|
-
`.mudra-onb__continue` button is disabled until
|
|
749
|
-
`.mudra-onb__ai-input` is non-empty AND matches `/^AIza[\w-]{30,}$/`
|
|
750
|
-
(Google API-key prefix sanity check).
|
|
751
|
-
2. **On click of `Got it`**, write the trimmed value to
|
|
752
|
-
`sessionStorage.setItem('mudra.gemini.apiKey', value)`, then close
|
|
753
|
-
the modal.
|
|
754
|
-
3. **At every page load**, the IIFE checks `sessionStorage` first:
|
|
755
|
-
- If `mudra.gemini.apiKey` is present and matches the prefix regex,
|
|
756
|
-
the AI-Setup fragment is hidden (the user already provided a key
|
|
757
|
-
this session) and `Got it` is enabled immediately.
|
|
758
|
-
- If absent or malformed, the AI-Setup fragment is shown and the
|
|
759
|
-
modal CANNOT be dismissed by `Escape`, the `×` close button, or
|
|
760
|
-
the reopen `?` button without entering a valid key. `dialog.close()`
|
|
761
|
-
paths called from those handlers are no-ops while the key is
|
|
762
|
-
missing.
|
|
763
|
-
4. **The `?` reopen button** for AI apps re-runs the gating logic. If
|
|
764
|
-
the user clears `sessionStorage` mid-session and reopens, the
|
|
765
|
-
AI-Setup fragment renders again.
|
|
766
|
-
5. **Reading the key in app code:** the `xb.Script` subclass reads
|
|
767
|
-
`sessionStorage.getItem('mudra.gemini.apiKey')` in `init()`. If the
|
|
768
|
-
key is `null`, the AI portion of the app stays inert (no calls to
|
|
769
|
-
Gemini) and the visible chat panel (Section 18) renders a `Set up
|
|
770
|
-
AI in the welcome panel` placeholder.
|
|
771
|
-
6. **Never bake a key into the HTML source.** Pre-write regex scan
|
|
772
|
-
(`/AIza[A-Za-z0-9_-]{30,}|sk-[A-Za-z0-9_-]{32,}/` excluding the
|
|
773
|
-
literal placeholder string `AIza…`) must return zero matches.
|
|
774
|
-
7. **Never auto-read from URL params, `localStorage`, or `prompt()`.**
|
|
775
|
-
8. **Non-AI apps** ignore this section entirely. The AI-Setup fragment
|
|
776
|
-
is omitted; the modal works as defined in Section 17 unchanged.
|
|
777
|
-
|
|
778
|
-
### Canonical Gemini model — `gemini-2.5-flash` only
|
|
779
|
-
|
|
780
|
-
For any generated app that calls Gemini via the **REST `generateContent`
|
|
781
|
-
endpoint**, the model ID MUST be exactly `gemini-2.5-flash`. No other
|
|
782
|
-
model IDs are permitted for REST text/chat/vision generation. This is a
|
|
783
|
-
hard pin — preview aliases get retired by Google and the app then 404s.
|
|
784
|
-
|
|
785
|
-
```js
|
|
786
|
-
// CORRECT — the only permitted REST model for text/chat/vision
|
|
787
|
-
const url = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=${encodeURIComponent(apiKey)}`;
|
|
788
|
-
|
|
789
|
-
// FORBIDDEN — preview aliases, dated aliases, retired families
|
|
790
|
-
gemini-1.5-flash, gemini-1.5-flash-latest, gemini-1.5-pro
|
|
791
|
-
gemini-2.5-flash-preview-09-2025, gemini-2.5-flash-preview-04-2025
|
|
792
|
-
gemini-flash-latest, gemini-pro, gemini-2.5-flash-002 (any -NNN suffix)
|
|
793
|
-
```
|
|
794
|
-
|
|
795
|
-
Pre-write regex scan — every generated HTML file MUST satisfy:
|
|
796
|
-
|
|
797
|
-
- `/generativelanguage\.googleapis\.com\/v1beta\/models\/([a-z0-9-]+):generateContent/`
|
|
798
|
-
→ captured model ID MUST equal `gemini-2.5-flash`. Any other capture
|
|
799
|
-
fails the pre-write checklist.
|
|
800
|
-
|
|
801
|
-
Out-of-scope use cases (allowed exceptions, REST-only rule does NOT apply):
|
|
802
|
-
|
|
803
|
-
- **Live API** (WebSocket / xrblocks `xb.core.ai.startLiveSession()`):
|
|
804
|
-
the Live endpoint uses its own model set (`gemini-2.0-flash-live-001`,
|
|
805
|
-
`gemini-2.5-flash-native-audio-preview-12-2025`). Apps that genuinely
|
|
806
|
-
need streaming audio/video may use those, but every text-chat app
|
|
807
|
-
must use the REST pin above.
|
|
808
|
-
- **Image generation** via `:generateContent`: `gemini-2.5-flash-image`
|
|
809
|
-
is permitted only when the app's purpose is image output. Default to
|
|
810
|
-
the text pin otherwise.
|
|
811
|
-
|
|
812
|
-
If an app needs a different model, raise it to the user before writing —
|
|
813
|
-
do not silently swap in a preview alias.
|
|
814
|
-
|
|
815
|
-
---
|
|
816
|
-
|
|
817
|
-
## Section 10 — Pre-Write Checklist + Collision Handling
|
|
818
|
-
|
|
819
|
-
Before calling `Write` to emit a generated app, verify all items:
|
|
820
|
-
|
|
821
|
-
| # | Check | Pass condition |
|
|
822
|
-
|---|-------|----------------|
|
|
823
|
-
| 1 | Single file | Exactly one `<html>` document; all CSS in `<style>`; all JS in `<script>` or `<script type="module">` |
|
|
824
|
-
| 2 | Import map | One `<script type="importmap">` block; contents match canonical pins (Section 2) exactly; no unused entries |
|
|
825
|
-
| 3 | xb.Script entry | Top-level logic inside `class <Name> extends xb.Script`; `xb.add(new <Name>())` + `xb.init(new xb.Options())` on `DOMContentLoaded` |
|
|
826
|
-
| 4 | MudraClient | One `MudraClient` instance at module scope; constructor does NOT auto-connect; `setMode()` drives connect/disconnect; no `setTimeout(_, 1500)` timer; no `_startMock()` auto-firing; no `'simulated'` / `'disconnected-simulated'` strings |
|
|
827
|
-
| 5 | Subscribe commands | Every used signal has exactly one `mudra.subscribe('<signal>')` call; none outside the signal set |
|
|
828
|
-
| 6 | Simulator panel | `<div id="mudra-sim">` present; ONLY buttons for sub-actions actually handled by the app (no extras like Roll L/R or Twist if unused); buttons fire via handler, not inline `onclick` |
|
|
829
|
-
| 7 | Keyboard bindings | `window.addEventListener('keydown', …, { capture: true })` present; `event.stopPropagation()` on every Mudra-claimed key |
|
|
830
|
-
| 8 | Status indicator | `<div id="mudra-status">` + `<span id="mudra-hand">` present; text states are `Manual` / `Connecting…` / `Connected` / `WebSocket Only` / `Reconnecting…` / `Companion already in use` (Section 15); no `simulated` / `disconnected-simulated` strings; `#mudra-hand` shows `LEFT`/`RIGHT` when connected, `None` otherwise |
|
|
831
|
-
| 9 | AI key gating | If `usesAI`: the AI-Setup fragment (Section 9) is present inside `.mudra-onb__body`; key is read from `sessionStorage.getItem('mudra.gemini.apiKey')` only; ZERO `prompt(` calls for the key; ZERO `localStorage` references; ZERO baked keys (regex scan) |
|
|
832
|
-
| 9a | Visible AI chat I/O | If `usesAI`: the scene renders BOTH the latest user input AND the AI response as visible text (xb.ScrollingTroikaTextView, troika `Text`, or xb.SpatialPanel rows). The visible "Purpose" line states what the app does in one sentence. TTS may exist but is never the only output (Section 18) |
|
|
833
|
-
| 10 | Background lockdown | ZERO `applyBackground_*` methods in the class; ZERO calls to a background helper from `init()`; no `options.simulator.scenePath` line anywhere. Generated apps use the XR Blocks default room only (Section 14) |
|
|
834
|
-
| 11 | Mode toggle | `<div id="mode-toggle">` with **Manual** + **Mudra** buttons; Manual is the default on load; toggle remains clickable when disconnected; flipping atomically opens/closes the socket per Section 15 |
|
|
835
|
-
| 12 | Band-state polling | In Mudra mode the app sends `{command:"get_status"}` on `ws.onopen` and every 2000 ms thereafter; pill flips to `Connected` ONLY when `data.device.firmware && data.device.serial_number` (both truthy); `WebSocket Only` (orange) when socket open but band not connected |
|
|
836
|
-
| 13 | No disconnect overlay | No banner / toast / modal / inline alert ever rendered for disconnect — pill is the only indicator |
|
|
837
|
-
| 14 | Footer | Exactly one `<div id="mudra-badge">` containing the literal text `Created by Mudra` (no variants) |
|
|
838
|
-
| 15 | Mock is passive | No `setTimeout(_, 1500)` timer; no `_startMock()` auto-firing; Manual-mode signals come only from sim-panel clicks and keyboard shortcuts calling `_emit()` directly |
|
|
839
|
-
| 16 | Gemini model pin | If the app calls `generativelanguage.googleapis.com/v1beta/models/<id>:generateContent`, the captured `<id>` MUST equal `gemini-2.5-flash`. No preview / dated / latest aliases. Live-API and image-gen exceptions per Section 9 |
|
|
840
|
-
| 17 | Subscription record | `MudraClient._subscriptions` is a `Set<string>` that tracks subscribed signals; `_replaySubscriptions()` fires on `ws.onopen` AND on any `ws-only` → `connected` transition; signals in the record are never resurrected if `unsubscribe()` was called explicitly |
|
|
841
|
-
| 18 | client_already_connected | The `{type:"error",data:{error:"client_already_connected"}}` error frame is handled: pill transitions to `Companion already in use`; `_suppressReconnect = true`; the backoff timer does NOT fire on the subsequent `onclose` |
|
|
842
|
-
| 19 | Reconnect backoff | WS `onclose` (non-conflict) schedules reconnect with `[1000, 2000, 5000, 10000]` ms schedule; `_backoffIdx` resets to 0 on every successful `onopen` |
|
|
843
|
-
| 20 | No forbidden strings | Generated source contains zero occurrences of: `confidence` as a gesture field, `enable`/`disable`/`get_docs` as commands, `'simulated'`/`'disconnected-simulated'` as status strings, `8765` as a port |
|
|
844
|
-
|
|
845
|
-
### Retry policy
|
|
846
|
-
|
|
847
|
-
If any check fails:
|
|
848
|
-
1. Regenerate once and re-run the full checklist.
|
|
849
|
-
2. If the second attempt also fails, surface the specific failing check(s)
|
|
850
|
-
to the user and do **not** write the file.
|
|
851
|
-
|
|
852
|
-
### File collision / auto-suffix rule
|
|
853
|
-
|
|
854
|
-
- Target filename: `preview/<name>.html`
|
|
855
|
-
- If that file already exists, try `preview/<name>-2.html`, then `-3.html`, etc.
|
|
856
|
-
- Never overwrite an existing file.
|
|
857
|
-
- Always report the final written path to the user.
|
|
858
|
-
|
|
859
|
-
---
|
|
860
|
-
|
|
861
|
-
## Section 11 — Signal → XR Binding Patterns
|
|
862
|
-
|
|
863
|
-
Use these as code snippets when adapting a template.
|
|
864
|
-
Copy and adapt — do not invent new patterns from scratch.
|
|
865
|
-
|
|
866
|
-
### gesture.tap → onSelectStart forwarding
|
|
867
|
-
|
|
868
|
-
```js
|
|
869
|
-
// In init():
|
|
870
|
-
mudra.on('gesture', (data) => {
|
|
871
|
-
if (data.type === 'tap') this.onSelectStart({ source: 'mudra' });
|
|
872
|
-
if (data.type === 'double_tap') this.onSelectEnd({ source: 'mudra' });
|
|
873
|
-
});
|
|
874
|
-
mudra.subscribe('gesture');
|
|
875
|
-
```
|
|
876
|
-
|
|
877
|
-
### pressure → scale / color mapping
|
|
878
|
-
|
|
879
|
-
```js
|
|
880
|
-
// In init():
|
|
881
|
-
mudra.on('pressure', (data) => {
|
|
882
|
-
const s = 0.5 + data.normalized * 1.5; // scale 0.5 … 2.0
|
|
883
|
-
this.mesh.scale.setScalar(s);
|
|
884
|
-
const hue = data.normalized * 0.8; // hue 0 (red) … 0.8 (blue)
|
|
885
|
-
this.mesh.material.color.setHSL(hue, 0.9, 0.5);
|
|
886
|
-
});
|
|
887
|
-
mudra.subscribe('pressure');
|
|
888
|
-
```
|
|
889
|
-
|
|
890
|
-
### nav_direction → spatial menu step
|
|
891
|
-
|
|
892
|
-
```js
|
|
893
|
-
// In init():
|
|
894
|
-
mudra.on('nav_direction', (data) => {
|
|
895
|
-
switch (data.direction) {
|
|
896
|
-
case 'Up': this.menuIndex = Math.max(0, this.menuIndex - 1); break;
|
|
897
|
-
case 'Down': this.menuIndex = Math.min(this.items.length - 1, this.menuIndex + 1); break;
|
|
898
|
-
case 'Right': this.selectItem(this.menuIndex); break;
|
|
899
|
-
case 'Left': this.goBack(); break;
|
|
900
|
-
}
|
|
901
|
-
this.updateMenuHighlight();
|
|
902
|
-
});
|
|
903
|
-
mudra.subscribe('nav_direction');
|
|
904
|
-
```
|
|
905
|
-
|
|
906
|
-
### imu_acc + imu_gyro + snc bundle → subscribe all three together
|
|
907
|
-
|
|
908
|
-
`imu_acc`, `imu_gyro`, and `snc` are always subscribed together. Register handlers
|
|
909
|
-
for each signal you actually use in the app, but always send all three subscribe commands.
|
|
910
|
-
|
|
911
|
-
```js
|
|
912
|
-
// In init():
|
|
913
|
-
mudra.on('imu_acc', (data) => {
|
|
914
|
-
const [ax, ay] = data.values;
|
|
915
|
-
this.mesh.rotation.x = THREE.MathUtils.clamp(ax * 0.1, -Math.PI / 4, Math.PI / 4);
|
|
916
|
-
this.mesh.rotation.z = THREE.MathUtils.clamp(ay * 0.1, -Math.PI / 4, Math.PI / 4);
|
|
917
|
-
});
|
|
918
|
-
mudra.on('imu_gyro', (data) => {
|
|
919
|
-
const [gx] = data.values;
|
|
920
|
-
this.mesh.rotation.y += gx * 0.001;
|
|
921
|
-
});
|
|
922
|
-
mudra.on('snc', (data) => {
|
|
923
|
-
const ch1 = data.values[0];
|
|
924
|
-
const latest = ch1[ch1.length - 1]; // most recent sample
|
|
925
|
-
const norm = Math.min(1, Math.abs(latest) / 500); // normalize
|
|
926
|
-
this.overlay.material.opacity = norm * 0.7;
|
|
927
|
-
});
|
|
928
|
-
// ALWAYS subscribe all three — they form an inseparable bundle
|
|
929
|
-
mudra.subscribe('imu_acc');
|
|
930
|
-
mudra.subscribe('imu_gyro');
|
|
931
|
-
mudra.subscribe('snc');
|
|
932
|
-
```
|
|
933
|
-
|
|
934
|
-
### navigation → continuous cursor / pan
|
|
935
|
-
|
|
936
|
-
**Default sensitivity multiplier: `0.002`** (gentle / low-sensitivity).
|
|
937
|
-
Use this baseline for all navigation/cursor/pan bindings. Only raise it
|
|
938
|
-
when the prompt explicitly asks for fast/snappy movement.
|
|
939
|
-
|
|
940
|
-
```js
|
|
941
|
-
// In init():
|
|
942
|
-
const NAV_SENSITIVITY = 0.002; // gentle default — slow, predictable
|
|
943
|
-
mudra.on('navigation', (data) => {
|
|
944
|
-
this.cursorX = THREE.MathUtils.clamp(this.cursorX + data.delta_x * NAV_SENSITIVITY, -1, 1);
|
|
945
|
-
this.cursorY = THREE.MathUtils.clamp(this.cursorY - data.delta_y * NAV_SENSITIVITY, -1, 1);
|
|
946
|
-
this.cursor.position.set(this.cursorX, this.cursorY, -xb.user.objectDistance);
|
|
947
|
-
});
|
|
948
|
-
mudra.subscribe('navigation');
|
|
949
|
-
```
|
|
950
|
-
|
|
951
|
-
---
|
|
952
|
-
|
|
953
|
-
## Section 12 — Template Selection Table
|
|
954
|
-
|
|
955
|
-
One row per asset in `assets/`. The skill's keyword-based selector scores each row
|
|
956
|
-
against the user's prompt (signal names, motion keywords, XR feature words).
|
|
957
|
-
|
|
958
|
-
| id | path | keywords | motionModesSupported | xrFeatures |
|
|
959
|
-
|----|------|----------|---------------------|------------|
|
|
960
|
-
| `0_basic` | `assets/templates/0_basic.html` | `["basic","cylinder","pinch","color","simple","starter"]` | `["none","pointer"]` | `["input"]` |
|
|
961
|
-
| `1_ui` | `assets/templates/1_ui.html` | `["ui","spatial","panel","text","sdf","font","button","draggable","troika"]` | `["pointer"]` | `["input","ui"]` |
|
|
962
|
-
| `2_hands` | `assets/templates/2_hands.html` | `["hands","hand","pinch","gesture","joints","finger","hand-tracking"]` | `["pointer"]` | `["hands","input"]` |
|
|
963
|
-
| `3_depth` | `assets/templates/3_depth.html` | `["depth","mesh","depth-sensing","plane","environment","occlusion"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
|
|
964
|
-
| `4_stereo` | `assets/templates/4_stereo.html` | `["stereo","video","passthrough","camera","background","environment","feed"]` | `["none"]` | `["camera","passthrough"]` |
|
|
965
|
-
| `5_camera` | `assets/templates/5_camera.html` | `["camera","video","passthrough","texture","scene","background"]` | `["none","pointer"]` | `["camera","input"]` |
|
|
966
|
-
| `6_ai` | `assets/templates/6_ai.html` | `["ai","gemini","query","vision","photo","capture","llm","multimodal"]` | `["pointer"]` | `["input","camera"]` |
|
|
967
|
-
| `7_ai_live` | `assets/templates/7_ai_live.html` | `["ai","gemini","live","speech","transcription","voice","microphone","audio","real-time"]` | `["pointer"]` | `["input"]` |
|
|
968
|
-
| `8_objects` | `assets/templates/8_objects.html` | `["objects","detection","model","3d","place","anchor","ar","environment"]` | `["pointer"]` | `["mesh-detection","input"]` |
|
|
969
|
-
| `9_xr-toggle` | `assets/templates/9_xr-toggle.html` | `["toggle","xr","enter","exit","session","button","transition"]` | `["none","pointer"]` | `["input"]` |
|
|
970
|
-
| `heuristic_hand_gestures` | `assets/templates/heuristic_hand_gestures.html` | `["gesture","heuristic","hand","recognition","custom","pattern","hand-tracking"]` | `["none"]` | `["hands"]` |
|
|
971
|
-
| `meshes` | `assets/templates/meshes.html` | `["mesh","environment","scan","plane","floor","wall","surface"]` | `["none"]` | `["mesh-detection"]` |
|
|
972
|
-
| `planes` | `assets/templates/planes.html` | `["plane","floor","wall","surface","anchor","environment","horizontal","vertical"]` | `["none"]` | `["plane-detection"]` |
|
|
973
|
-
| `uikit` | `assets/templates/uikit.html` | `["ui","kit","component","widget","button","icon","material","text","panel","menu"]` | `["pointer"]` | `["input","ui"]` |
|
|
974
|
-
| `depthmap` | `assets/samples/depthmap.html` | `["depth","map","visualization","color","gradient","environment","scan"]` | `["none"]` | `["depth-sensing"]` |
|
|
975
|
-
| `depthmesh` | `assets/samples/depthmesh.html` | `["depth","mesh","wireframe","environment","scan","geometry"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
|
|
976
|
-
| `game_rps` | `assets/samples/game_rps.html` | `["game","rps","rock","paper","scissors","gesture","compete","fun","hand"]` | `["none"]` | `["hands"]` |
|
|
977
|
-
| `gestures_custom` | `assets/samples/gestures_custom.html` | `["gesture","custom","recognize","train","pose","hand-tracking","hands"]` | `["none"]` | `["hands"]` |
|
|
978
|
-
| `gestures_heuristic` | `assets/samples/gestures_heuristic.html` | `["gesture","heuristic","recognize","hand","pose","pinch","open","fist"]` | `["none"]` | `["hands"]` |
|
|
979
|
-
| `lighting` | `assets/samples/lighting.html` | `["lighting","light","shadow","scene","animals","3d","models","environment"]` | `["pointer"]` | `["input"]` |
|
|
980
|
-
| `mesh_detection` | `assets/samples/mesh_detection.html` | `["mesh","detection","environment","scan","ar","surface"]` | `["none"]` | `["mesh-detection"]` |
|
|
981
|
-
| `modelviewer` | `assets/samples/modelviewer.html` | `["model","viewer","3d","gltf","glb","object","rotate","inspect","load"]` | `["pointer"]` | `["input"]` |
|
|
982
|
-
| `paint` | `assets/samples/paint.html` | `["paint","draw","brush","stroke","canvas","art","color","gesture"]` | `["pointer"]` | `["input","hands"]` |
|
|
983
|
-
| `planar-vst` | `assets/samples/planar-vst.html` | `["plane","vst","passthrough","video","portal","ar","surface"]` | `["none"]` | `["plane-detection","camera"]` |
|
|
984
|
-
| `reticle` | `assets/samples/reticle.html` | `["reticle","cursor","pointer","aim","gaze","target","floor","placement"]` | `["none","pointer"]` | `["plane-detection","input"]` |
|
|
985
|
-
| `skybox_agent` | `assets/samples/skybox_agent.html` | `["skybox","sky","background","ai","gemini","generate","environment","image"]` | `["pointer"]` | `["input"]` |
|
|
986
|
-
| `sound` | `assets/samples/sound.html` | `["sound","audio","music","spatial","3d-audio","positional","play"]` | `["pointer"]` | `["input"]` |
|
|
987
|
-
| `ui` | `assets/samples/ui.html` | `["ui","panel","button","menu","list","text","interface","spatial"]` | `["pointer"]` | `["input","ui"]` |
|
|
988
|
-
| `virtual-screens` | `assets/samples/virtual-screens.html` | `["screen","virtual","window","share","stream","desktop","browser","display"]` | `["pointer"]` | `["input"]` |
|
|
989
|
-
| `3dgs-walkthrough` | `assets/demos/3dgs-walkthrough.html` | `["gaussian","splat","3dgs","scene","walkthrough","room","photo","realistic"]` | `["imu","direction"]` | `["input"]` |
|
|
990
|
-
| `aisimulator` | `assets/demos/aisimulator.html` | `["ai","simulate","gemini","agent","roleplay","character","conversation","npc"]` | `["pointer"]` | `["input"]` |
|
|
991
|
-
| `balloonpop` | `assets/demos/balloonpop.html` | `["balloon","pop","game","gesture","fun","pinch","particle","explosion"]` | `["none"]` | `["hands","input"]` |
|
|
992
|
-
| `ballpit` | `assets/demos/ballpit.html` | `["ball","physics","pit","throw","gravity","interact","fun","ammo"]` | `["pointer"]` | `["input","hands"]` |
|
|
993
|
-
| `drone` | `assets/demos/drone.html` | `["drone","fly","navigate","control","imu","tilt","direction","pilot"]` | `["imu","direction"]` | `["input"]` |
|
|
994
|
-
| `gemini-icebreakers` | `assets/demos/gemini-icebreakers.html` | `["gemini","icebreaker","question","conversation","ai","social","fun"]` | `["pointer"]` | `["input"]` |
|
|
995
|
-
| `gemini-xrobject` | `assets/demos/gemini-xrobject.html` | `["gemini","object","recognize","camera","ar","label","identify","vision"]` | `["pointer"]` | `["input","camera"]` |
|
|
996
|
-
| `math3d` | `assets/demos/math3d.html` | `["math","3d","formula","graph","equation","plot","visualization","education"]` | `["pointer"]` | `["input","ui"]` |
|
|
997
|
-
| `measure` | `assets/demos/measure.html` | `["measure","ruler","tape","distance","ar","depth","spatial","length"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
|
|
998
|
-
| `occlusion` | `assets/demos/occlusion.html` | `["occlusion","depth","animal","model","shadow","ar","realistic","environment"]` | `["pointer"]` | `["input","depth-sensing"]` |
|
|
999
|
-
| `rain` | `assets/demos/rain.html` | `["rain","particle","weather","atmosphere","shader","visual","instanced"]` | `["none","pointer"]` | `["input"]` |
|
|
1000
|
-
| `screenwiper` | `assets/demos/screenwiper.html` | `["wiper","screen","wipe","clear","gesture","brush","clean","effect"]` | `["pointer"]` | `["input","hands"]` |
|
|
1001
|
-
| `splash` | `assets/demos/splash.html` | `["splash","paint","decal","physics","shoot","color","ball","impact"]` | `["pointer"]` | `["input"]` |
|
|
1002
|
-
| `webcam_gestures` | `assets/demos/webcam_gestures.html` | `["webcam","mediapipe","gesture","hand","camera","tracking","no-headset","flat"]` | `["none"]` | `[]` |
|
|
1003
|
-
| `xremoji` | `assets/demos/xremoji.html` | `["emoji","expression","face","gesture","fun","balloon","tensorflow","social"]` | `["none"]` | `["hands"]` |
|
|
1004
|
-
| `xrpoet` | `assets/demos/xrpoet.html` | `["poem","poetry","ai","gemini","generate","creative","camera","writing"]` | `["pointer"]` | `["input","camera"]` |
|
|
1005
|
-
|
|
1006
|
-
---
|
|
1007
|
-
|
|
1008
|
-
## Section 13 — Template Inference + Override
|
|
1009
|
-
|
|
1010
|
-
*(Populated after Phase 5 / T032.)*
|
|
1011
|
-
|
|
1012
|
-
---
|
|
1013
|
-
|
|
1014
|
-
## Section 14 — Background Lockdown (XR Blocks default only)
|
|
1015
|
-
|
|
1016
|
-
**Custom backgrounds are forbidden.** Every generated XR app uses the XR
|
|
1017
|
-
Blocks default room and nothing else. There is no catalog, no helper, no
|
|
1018
|
-
override, no `[bg=...]` tag, no `scenePath` override.
|
|
1019
|
-
|
|
1020
|
-
### Hard rules — apply unconditionally
|
|
1021
|
-
|
|
1022
|
-
1. The `xb.Script` subclass MUST NOT contain any `applyBackground_*` method.
|
|
1023
|
-
2. `init()` MUST NOT call any background helper. It starts with lights,
|
|
1024
|
-
meshes, and Mudra wiring.
|
|
1025
|
-
3. The entry point MUST NOT set `options.simulator.scenePath` — not to
|
|
1026
|
-
`null`, not to a path. Leave it alone so XR Blocks renders its default
|
|
1027
|
-
room.
|
|
1028
|
-
4. No `THREE.SphereGeometry` dome, no custom skybox `Mesh`, no
|
|
1029
|
-
`THREE.Points` starfield, no `THREE.GridHelper` floor, no equirectangular
|
|
1030
|
-
`TextureLoader().load(...)` for background purposes. (Per-scene
|
|
1031
|
-
geometry that the app actually needs is fine — the ban is on standalone
|
|
1032
|
-
environment domes/floors/skies.)
|
|
1033
|
-
5. Prompt cues like "in space", "starfield", "sunset sky", "cyberpunk
|
|
1034
|
-
vibe", "with a forest backdrop", or even literal `[bg=<id>]` tags MUST
|
|
1035
|
-
be IGNORED for background purposes. They may still inform template /
|
|
1036
|
-
motion-mode selection.
|
|
1037
|
-
6. If the user explicitly insists on a custom background, decline and
|
|
1038
|
-
remind them that this skill is locked to the XR Blocks default room.
|
|
1039
|
-
|
|
1040
|
-
### Pre-write regex (Section 10 check #10)
|
|
1041
|
-
|
|
1042
|
-
The generated source MUST satisfy ALL of the following:
|
|
1043
|
-
|
|
1044
|
-
- `/applyBackground_/` → zero matches.
|
|
1045
|
-
- `/options\.simulator\.scenePath/` → zero matches.
|
|
1046
|
-
|
|
1047
|
-
If either pattern matches, the file fails the pre-write checklist and is
|
|
1048
|
-
not written.
|
|
1049
|
-
|
|
1050
|
-
---
|
|
1051
|
-
## Section 15 — Mode Toggle (Manual / Mudra) — Required
|
|
1052
|
-
|
|
1053
|
-
**This section is the canonical source for the connection-status state machine and status pill wiring.** Section 4 defines the `MudraClient` class; Section 7 is superseded by this section.
|
|
1054
|
-
|
|
1055
|
-
Every XR app generated by this skill MUST implement the Mode Toggle exactly as specified here.
|
|
1056
|
-
|
|
1057
|
-
### Summary
|
|
1058
|
-
|
|
1059
|
-
The user picks how the app is driven via a visible **Mode** control,
|
|
1060
|
-
rendered as a 2D DOM overlay (it disappears automatically in immersive
|
|
1061
|
-
WebXR sessions, like the simulator panel and status pill):
|
|
1062
|
-
|
|
1063
|
-
- **Manual** (default on load): the simulator panel is fully interactive.
|
|
1064
|
-
Sim-panel actions inject synthetic signal messages into the same handler
|
|
1065
|
-
pipeline that real WebSocket messages would flow through. **No WebSocket
|
|
1066
|
-
connection is opened in Manual mode.** The XR scene reacts only to
|
|
1067
|
-
signal handlers — never to direct DOM clicks bypassing the signal path.
|
|
1068
|
-
- **Mudra**: the app opens one WebSocket to `ws://127.0.0.1:8766` and
|
|
1069
|
-
subscribes to its signals one-at-a-time. The simulator panel is
|
|
1070
|
-
visually disabled (greyed-out, `pointer-events: none`,
|
|
1071
|
-
`aria-disabled="true"`) and emits no synthetic signals. The
|
|
1072
|
-
connection-status pill reflects band-pairing state via `get_status`
|
|
1073
|
-
polling — **no separate overlay, toast, or banner is rendered.**
|
|
1074
|
-
|
|
1075
|
-
**The Mode toggle MUST remain fully clickable and keyboard-focusable at
|
|
1076
|
-
all times** — including while the band is disconnected. Manual is the
|
|
1077
|
-
default on first load.
|
|
1078
|
-
|
|
1079
|
-
### State machine
|
|
1080
|
-
|
|
1081
|
-
```text
|
|
1082
|
-
type Mode = "manual" | "mudra" // default "manual"
|
|
1083
|
-
type ConnectionState =
|
|
1084
|
-
| "idle" // Manual mode. No socket open.
|
|
1085
|
-
| "connecting" // Socket opening, OR socket open but first get_status not yet received.
|
|
1086
|
-
| "connected" // Socket open AND device.firmware && device.serial_number both truthy.
|
|
1087
|
-
| "ws-only" // Socket open BUT device.firmware or device.serial_number is null/falsy (band not paired).
|
|
1088
|
-
| "reconnecting" // Socket closed for non-conflict reason; backoff retry in progress.
|
|
1089
|
-
| "already-in-use" // Server sent client_already_connected; terminal until mode→Manual or page reload.
|
|
1090
|
-
```
|
|
1091
|
-
|
|
1092
|
-
### Lazy-WS lifecycle (mandatory)
|
|
1093
|
-
|
|
1094
|
-
| Transition | Action |
|
|
1095
|
-
|------------|--------|
|
|
1096
|
-
| page load | `mode = "manual"`, `connectionState = "idle"`, NO socket |
|
|
1097
|
-
| Manual → Mudra | call `_openSocket()`; `connectionState = "connecting"` |
|
|
1098
|
-
| Mudra → Manual | call `_closeSocket()`; cancel reconnect + poll timers; `connectionState = "idle"` |
|
|
1099
|
-
| WS `onopen` (in Mudra) | reset backoff; replay subscription record; send `{command:"get_status"}`; start 2 s poll timer; `connectionState` stays `"connecting"` until first status response |
|
|
1100
|
-
| inbound `{type:"status"}` — `device.firmware && device.serial_number` truthy (in Mudra) | `connectionState = "connected"`; update hand chip from `device.hand`; if transitioning from `"ws-only"`, call `_replaySubscriptions()` (FR-012) |
|
|
1101
|
-
| inbound `{type:"status"}` — firmware or serial_number falsy (in Mudra) | `connectionState = "ws-only"`; hand chip = `None` |
|
|
1102
|
-
| inbound `{type:"error", data:{error:"client_already_connected"}}` (in Mudra) | `_suppressReconnect = true`; stop poll; `connectionState = "already-in-use"` |
|
|
1103
|
-
| WS `onclose` when `_suppressReconnect = true` | no-op (the error frame already set state) |
|
|
1104
|
-
| WS `onerror` / `onclose` (in Mudra, non-conflict) | stop poll; `connectionState = "reconnecting"`; advance backoff; schedule `_openSocket()` after backoff delay |
|
|
1105
|
-
| reconnect tick fires (in Mudra, non-conflict) | call `_openSocket()` → `"connecting"` |
|
|
1106
|
-
| Mudra → Manual at any point | call `_closeSocket()`; clear `_suppressReconnect`; `connectionState = "idle"` |
|
|
1107
|
-
|
|
1108
|
-
**Single-socket guarantee:** The `_connToken` counter (Section 4) ensures callbacks from a stale socket are discarded if mode is toggled rapidly.
|
|
1109
|
-
|
|
1110
|
-
### Band-connected predicate (mandatory)
|
|
1111
|
-
|
|
1112
|
-
**The WebSocket handshake to `127.0.0.1:8766` only proves the Companion service is up. It does NOT prove the Mudra Band is paired.**
|
|
1113
|
-
|
|
1114
|
-
The band-connected check uses both `device.firmware` AND `device.serial_number`:
|
|
1115
|
-
|
|
1116
|
-
```js
|
|
1117
|
-
const bandConnected = !!(msg.data?.device?.firmware && msg.data?.device?.serial_number);
|
|
1118
|
-
```
|
|
1119
|
-
|
|
1120
|
-
- Both truthy → `connectionState = "connected"`.
|
|
1121
|
-
- Either null/falsy → `connectionState = "ws-only"`.
|
|
1122
|
-
|
|
1123
|
-
Poll every **2000 ms** while the socket is `OPEN`. The pill MAY sit on `Connecting…` for up to one poll cycle (~2 s) after entering Mudra mode — that is correct behaviour.
|
|
1124
|
-
|
|
1125
|
-
### Hand chip
|
|
1126
|
-
|
|
1127
|
-
Every generated app MUST render a hand chip (`<span id="mudra-hand">`) beside the status pill:
|
|
1128
|
-
|
|
1129
|
-
- Shows `LEFT` or `RIGHT` (raw value from `device.hand`, uppercased) when `connectionState === "connected"`.
|
|
1130
|
-
- Shows `None` in all other states.
|
|
1131
|
-
- Normalization: if `device.hand` arrives as an enum string (e.g., `HandType.right`), strip the type prefix and uppercase. If normalization fails, display `None`.
|
|
1132
|
-
|
|
1133
|
-
```js
|
|
1134
|
-
mudra.on('_hand', (hand) => {
|
|
1135
|
-
document.getElementById('mudra-hand').textContent = hand;
|
|
1136
|
-
});
|
|
1137
|
-
```
|
|
1138
|
-
|
|
1139
|
-
### Reconnect backoff
|
|
1140
|
-
|
|
1141
|
-
Non-conflict WS close while `mode === "mudra"`:
|
|
1142
|
-
|
|
1143
|
-
```js
|
|
1144
|
-
// MudraClient._BACKOFF = [1000, 2000, 5000, 10000] // ms; last value is the cap
|
|
1145
|
-
```
|
|
1146
|
-
|
|
1147
|
-
- `_backoffIdx` advances on every non-conflict close; resets to 0 on every `onopen`.
|
|
1148
|
-
- The `client_already_connected` path sets `_suppressReconnect = true` which bypasses the backoff timer entirely.
|
|
1149
|
-
|
|
1150
|
-
### Status pill text states
|
|
1151
|
-
|
|
1152
|
-
| connectionState | `#mudra-status` text | `#mudra-hand` text | Colour hint |
|
|
1153
|
-
|---|---|---|---|
|
|
1154
|
-
| `idle` (Manual) | `Manual` | `None` | neutral |
|
|
1155
|
-
| `connecting` | `Connecting…` | `None` | amber |
|
|
1156
|
-
| `connected` | `Connected` | `LEFT` or `RIGHT` | green |
|
|
1157
|
-
| `ws-only` | `WebSocket Only` | `None` | **orange** (`#eab308`) |
|
|
1158
|
-
| `reconnecting` | `Reconnecting…` | `None` | amber (muted) |
|
|
1159
|
-
| `already-in-use` | `Companion already in use — close the other tab first` | `None` | red/error |
|
|
1160
|
-
|
|
1161
|
-
The pill + hand chip are the **only** connection indicators. No banner, toast, or modal.
|
|
1162
|
-
|
|
1163
|
-
The simulator panel is greyed (`pointer-events: none`, reduced opacity) in every state except `idle` (Manual).
|
|
1164
|
-
|
|
1165
|
-
### Mode-toggle DOM sketch
|
|
1166
|
-
|
|
1167
|
-
```html
|
|
1168
|
-
<div id="mode-toggle" role="tablist" aria-label="Mode">
|
|
1169
|
-
<button id="mode-manual" role="tab" aria-selected="true">Manual</button>
|
|
1170
|
-
<button id="mode-mudra" role="tab" aria-selected="false">Mudra</button>
|
|
1171
|
-
</div>
|
|
1172
|
-
<div style="display:flex;align-items:center;gap:6px;position:fixed;top:8px;right:12px;z-index:9999;">
|
|
1173
|
-
<div id="mudra-status" style="padding:4px 10px;border-radius:999px;font-size:0.8rem;font-family:system-ui,sans-serif;background:rgba(0,0,0,0.6);color:#fff;">Manual</div>
|
|
1174
|
-
<span id="mudra-hand" style="padding:2px 8px;border-radius:999px;font-size:0.75rem;font-family:system-ui,sans-serif;background:rgba(255,255,255,0.12);color:#fff;">None</span>
|
|
1175
|
-
</div>
|
|
1176
|
-
```
|
|
1177
|
-
|
|
1178
|
-
Both `#mudra-status` and `#mudra-hand` are 2D DOM elements — they disappear automatically in immersive WebXR sessions via native DOM suppression. No extra JS wiring needed for this.
|
|
1179
|
-
|
|
1180
|
-
### Status pill wiring
|
|
1181
|
-
|
|
1182
|
-
```js
|
|
1183
|
-
mudra.on('_status', (s) => {
|
|
1184
|
-
const pill = document.getElementById('mudra-status');
|
|
1185
|
-
const labels = {
|
|
1186
|
-
'idle': 'Manual',
|
|
1187
|
-
'connecting': 'Connecting…',
|
|
1188
|
-
'connected': 'Connected',
|
|
1189
|
-
'ws-only': 'WebSocket Only',
|
|
1190
|
-
'reconnecting': 'Reconnecting…',
|
|
1191
|
-
'already-in-use':'Companion already in use — close the other tab first',
|
|
1192
|
-
};
|
|
1193
|
-
const colors = {
|
|
1194
|
-
'idle': 'rgba(0,0,0,0.6)',
|
|
1195
|
-
'connecting': 'rgba(180,120,0,0.8)',
|
|
1196
|
-
'connected': 'rgba(22,163,74,0.8)',
|
|
1197
|
-
'ws-only': '#eab308',
|
|
1198
|
-
'reconnecting': 'rgba(120,80,0,0.7)',
|
|
1199
|
-
'already-in-use':'rgba(185,28,28,0.85)',
|
|
1200
|
-
};
|
|
1201
|
-
pill.textContent = labels[s] ?? s;
|
|
1202
|
-
pill.style.background = colors[s] ?? 'rgba(0,0,0,0.6)';
|
|
1203
|
-
|
|
1204
|
-
// Grey the simulator panel in every non-Manual state
|
|
1205
|
-
const sim = document.getElementById('mudra-sim');
|
|
1206
|
-
if (sim) {
|
|
1207
|
-
sim.style.opacity = s === 'idle' ? '1' : '0.35';
|
|
1208
|
-
sim.style.pointerEvents = s === 'idle' ? '' : 'none';
|
|
1209
|
-
}
|
|
1210
|
-
|
|
1211
|
-
// Manage mode-toggle button aria-selected
|
|
1212
|
-
document.getElementById('mode-manual').setAttribute('aria-selected', s === 'idle' ? 'true' : 'false');
|
|
1213
|
-
document.getElementById('mode-mudra').setAttribute('aria-selected', s !== 'idle' ? 'true' : 'false');
|
|
1214
|
-
});
|
|
1215
|
-
|
|
1216
|
-
mudra.on('_hand', (hand) => {
|
|
1217
|
-
document.getElementById('mudra-hand').textContent = hand;
|
|
1218
|
-
});
|
|
1219
|
-
```
|
|
1220
|
-
|
|
1221
|
-
On every mode button click, atomically: call `mudra.setMode(newMode)`. `MudraClient.setMode()` handles all internal cleanup (cancel timers, close socket, reset tokens).
|
|
1222
|
-
|
|
1223
|
-
---
|
|
1224
|
-
|
|
1225
|
-
## Section 16 — Footer Branding — "Created by Mudra"
|
|
1226
|
-
|
|
1227
|
-
Every generated app MUST render a small footer/badge with the **exact**
|
|
1228
|
-
text **`Created by Mudra`**. Never "Created with Mudra Studio", never
|
|
1229
|
-
"Powered by Mudra", never any other variant. The badge is a 2D DOM
|
|
1230
|
-
overlay (disappears in immersive WebXR like the rest of the chrome).
|
|
1231
|
-
|
|
1232
|
-
```html
|
|
1233
|
-
<div id="mudra-badge" style="
|
|
1234
|
-
position: fixed; bottom: 8px; right: 12px;
|
|
1235
|
-
padding: 4px 10px; border-radius: 999px;
|
|
1236
|
-
font-size: 0.75rem; font-family: system-ui, sans-serif;
|
|
1237
|
-
background: rgba(0,0,0,0.5); color: #fff; opacity: 0.85;
|
|
1238
|
-
z-index: 9999; pointer-events: none;">Created by Mudra</div>
|
|
1239
|
-
```
|
|
1240
|
-
|
|
1241
|
-
Position is flexible if it would overlap the simulator panel — keep the
|
|
1242
|
-
text identical.
|
|
1243
|
-
|
|
1244
|
-
---
|
|
1245
|
-
|
|
1246
|
-
## Section 17 — Onboarding Modal (mandatory, STRICT) — feature 008-strict-onboarding-templates
|
|
1247
|
-
|
|
1248
|
-
> **Supersedes feature 005's loose modal rules.** The locked layout is
|
|
1249
|
-
> **Template 3 — split-card**. The DOM, CSS, and JS below MUST be emitted
|
|
1250
|
-
> **verbatim**. Only the per-app content slots may vary.
|
|
1251
|
-
|
|
1252
|
-
Every generated XR app MUST ship a first-run onboarding modal that greets
|
|
1253
|
-
the user and lists every action the app supports, with paired **Mudra** and
|
|
1254
|
-
**Manual** controls per action. The modal closes via `×` (skip), the
|
|
1255
|
-
**Continue** button, or `Escape`. It re-opens via a small floating `?` icon
|
|
1256
|
-
**only outside immersive XR** — in-XR re-onboarding is out of scope for v1
|
|
1257
|
-
(per feature 005 clarification Q3).
|
|
1258
|
-
|
|
1259
|
-
The binding contract is `specs/008-strict-onboarding-templates/contracts/onboarding-block.md`. The blocks below are the verbatim copies emitted into every app.
|
|
1260
|
-
|
|
1261
|
-
### XR-specific behavior (read carefully — unchanged from feature 005)
|
|
1262
|
-
|
|
1263
|
-
The onboarding modal is a **2D HTML overlay** shown before immersive entry.
|
|
1264
|
-
|
|
1265
|
-
- **MANDATORY: Disable the XR Blocks default Welcome overlay.** When the
|
|
1266
|
-
Simulator addon is imported (`import 'xrblocks/addons/simulator/SimulatorAddons.js';`),
|
|
1267
|
-
XR Blocks injects its own "Welcome to XR Blocks!" intro modal. This
|
|
1268
|
-
overlay competes visually with the Mudra onboarding modal and must be
|
|
1269
|
-
suppressed. Immediately after constructing the options object, set:
|
|
1270
|
-
|
|
1271
|
-
```js
|
|
1272
|
-
const options = new xb.Options();
|
|
1273
|
-
options.simulator.instructions.enabled = false; // suppress XR Blocks default Welcome overlay; the Mudra onboarding modal replaces it
|
|
1274
|
-
// ...rest of options config
|
|
1275
|
-
xb.init(options);
|
|
1276
|
-
```
|
|
1277
|
-
|
|
1278
|
-
- **Do NOT** mirror the Mudra modal as a 3D panel inside the XR scene. In-XR re-onboarding is explicitly out of scope for v1.
|
|
1279
|
-
- The inline `<script>` already listens for `xrsession-start` / `vr-session-start` / `ar-session-start` events and hides both the modal and the `?` icon during immersive sessions. Do NOT edit that wiring.
|
|
1280
|
-
- The modal layers above the XR canvas in 2D mode (`z-index: 100`). The Mudra badge from Section 16 and the simulator panel sit beneath it, by design.
|
|
1281
|
-
|
|
1282
|
-
### Required palette addition
|
|
1283
|
-
|
|
1284
|
-
Every generated app's `:root` MUST add one variable beyond the canonical palette:
|
|
1285
|
-
|
|
1286
|
-
```css
|
|
1287
|
-
--on-primary: #0c0d10; /* dark text on the primary-blue Continue button */
|
|
1288
|
-
```
|
|
1289
|
-
|
|
1290
|
-
### Locked DOM (paste at end of `<body>`)
|
|
1291
|
-
|
|
1292
|
-
```html
|
|
1293
|
-
<!-- === BEGIN onboarding-block === (Template 3 — Split Card, feature 008) -->
|
|
1294
|
-
<!-- IMPORTANT: do NOT add the `open` attribute. In 3D apps this matters
|
|
1295
|
-
particularly — XR Blocks' canvas is appended to <body> AFTER this
|
|
1296
|
-
dialog, and a non-modal dialog (HTML `open`) does NOT enter the
|
|
1297
|
-
browser's top layer. The dialog looks visible but the canvas can
|
|
1298
|
-
intercept clicks on Continue / ×. The IIFE below calls showModal()
|
|
1299
|
-
on load, which elevates the dialog above XR Blocks' canvas
|
|
1300
|
-
regardless of z-index. -->
|
|
1301
|
-
<dialog id="mudra-onboarding" data-mudra-onboarding data-app-name="{APP_NAME}">
|
|
1302
|
-
<div class="ob-card">
|
|
1303
|
-
<button class="ob-x" aria-label="Skip onboarding" data-ob-close>×</button>
|
|
1304
|
-
<div class="ob-left">
|
|
1305
|
-
<div class="ob-brand-block">
|
|
1306
|
-
<span class="ob-brand-mark">Mudra Studio</span>
|
|
1307
|
-
<h2 class="ob-brand-name">{APP_NAME_HEAD} <em>{APP_NAME_TAIL}</em></h2>
|
|
1308
|
-
<p class="ob-tagline">{APP_TAGLINE}</p>
|
|
1309
|
-
</div>
|
|
1310
|
-
<span class="ob-brand-footer">Created by Mudra</span>
|
|
1311
|
-
</div>
|
|
1312
|
-
<div class="ob-right">
|
|
1313
|
-
<h3 class="ob-section-title">How to use this app</h3>
|
|
1314
|
-
<div class="ob-chip-grid" id="ob-rows"></div>
|
|
1315
|
-
<div class="ob-continue-row"><button class="ob-continue" data-ob-close>Continue</button></div>
|
|
1316
|
-
</div>
|
|
1317
|
-
</div>
|
|
1318
|
-
</dialog>
|
|
1319
|
-
<button id="mudra-onboarding-help" class="ob-help-btn" aria-label="Reopen onboarding" hidden>?</button>
|
|
1320
|
-
<!-- === END onboarding-block === -->
|
|
1321
|
-
```
|
|
1322
|
-
|
|
1323
|
-
### Locked CSS (paste inside the existing `<style>` block in `<head>` — add one if the template does not have one)
|
|
1324
|
-
|
|
1325
|
-
```css
|
|
1326
|
-
/* === BEGIN onboarding-block === (Template 3 — Split Card, feature 008) */
|
|
1327
|
-
#mudra-onboarding{position:fixed;inset:0;border:0;padding:0;background:transparent;width:100%;height:100%;max-width:none;max-height:none;display:grid;place-items:center;z-index:100;color:var(--text);}
|
|
1328
|
-
#mudra-onboarding::backdrop{background:rgba(0,0,0,0.55);backdrop-filter:blur(6px);}
|
|
1329
|
-
#mudra-onboarding[hidden],#mudra-onboarding:not([open]){display:none;}
|
|
1330
|
-
.ob-card{position:relative;background:var(--card);backdrop-filter:blur(10px);border:1px solid rgba(255,255,255,0.08);border-radius:18px;width:min(720px,94vw);max-height:88vh;overflow:auto;display:grid;grid-template-columns:1fr 1fr;gap:0;box-shadow:0 20px 60px rgba(0,0,0,0.5);font-family:'Poppins',system-ui,sans-serif;}
|
|
1331
|
-
.ob-x{position:absolute;top:14px;right:14px;width:32px;height:32px;border-radius:50%;appearance:none;border:0;background:rgba(255,255,255,0.08);color:var(--text);font-size:18px;cursor:pointer;z-index:2;}
|
|
1332
|
-
.ob-x:hover{background:rgba(255,255,255,0.16);}
|
|
1333
|
-
.ob-left{padding:36px 28px;background:linear-gradient(135deg,rgba(108,140,255,0.16),rgba(185,124,255,0.12));border-right:1px solid rgba(255,255,255,0.06);display:flex;flex-direction:column;justify-content:space-between;border-radius:18px 0 0 18px;}
|
|
1334
|
-
.ob-brand-block{display:flex;flex-direction:column;gap:14px;}
|
|
1335
|
-
.ob-brand-mark{display:flex;align-items:center;gap:10px;font-size:13px;letter-spacing:0.16em;text-transform:uppercase;color:var(--text-secondary);}
|
|
1336
|
-
.ob-brand-mark::before{content:"";display:inline-block;width:24px;height:2px;background:var(--primary);}
|
|
1337
|
-
.ob-brand-name{font-size:34px;font-weight:700;line-height:1.05;margin:0;}
|
|
1338
|
-
.ob-brand-name em{font-style:normal;color:var(--accent);}
|
|
1339
|
-
.ob-tagline{color:var(--text-secondary);font-size:15px;margin:6px 0 0;line-height:1.5;}
|
|
1340
|
-
.ob-brand-footer{font-size:11px;letter-spacing:0.18em;text-transform:uppercase;color:var(--text-secondary);}
|
|
1341
|
-
.ob-right{padding:32px 28px 24px;display:flex;flex-direction:column;}
|
|
1342
|
-
.ob-section-title{font-size:12px;letter-spacing:0.1em;text-transform:uppercase;color:var(--text-secondary);margin:0 0 14px;}
|
|
1343
|
-
.ob-chip-grid{display:grid;grid-template-columns:1fr;gap:8px;flex:1;}
|
|
1344
|
-
.ob-chip{display:grid;grid-template-columns:1fr auto auto;gap:10px;align-items:center;background:rgba(255,255,255,0.04);border:1px solid rgba(255,255,255,0.05);border-radius:10px;padding:10px 12px;font-size:14px;}
|
|
1345
|
-
.ob-chip .nm{font-weight:500;}
|
|
1346
|
-
.ob-chip .mu{font-size:12px;padding:3px 9px;border-radius:999px;background:rgba(108,140,255,0.2);color:var(--primary);font-weight:600;}
|
|
1347
|
-
.ob-chip .mn{font-size:12px;color:var(--text-secondary);}
|
|
1348
|
-
.ob-continue-row{display:flex;justify-content:flex-end;margin-top:18px;}
|
|
1349
|
-
.ob-continue{appearance:none;border:0;background:var(--primary);color:var(--on-primary);font:inherit;font-weight:600;padding:10px 22px;border-radius:10px;cursor:pointer;}
|
|
1350
|
-
.ob-continue:hover{filter:brightness(1.1);}
|
|
1351
|
-
.ob-help-btn{position:fixed;bottom:16px;right:16px;appearance:none;border:0;width:36px;height:36px;border-radius:50%;background:var(--card);color:var(--text);font-size:18px;cursor:pointer;backdrop-filter:blur(10px);border:1px solid rgba(255,255,255,0.12);z-index:50;}
|
|
1352
|
-
@media (max-width:640px){
|
|
1353
|
-
.ob-card{grid-template-columns:1fr;}
|
|
1354
|
-
.ob-left{border-right:0;border-bottom:1px solid rgba(255,255,255,0.06);border-radius:18px 18px 0 0;padding:24px;}
|
|
1355
|
-
.ob-brand-name{font-size:24px;}
|
|
1356
|
-
.ob-right{padding:20px;}
|
|
1357
|
-
.ob-continue{width:100%;}
|
|
1358
|
-
.ob-continue-row{justify-content:stretch;}
|
|
1359
|
-
}
|
|
1360
|
-
/* === END onboarding-block === */
|
|
1361
|
-
```
|
|
1362
|
-
|
|
1363
|
-
### Locked JS (paste inside an inline `<script>` — NOT `type="module"` — at end of `<body>`)
|
|
1364
|
-
|
|
1365
|
-
```js
|
|
1366
|
-
// === BEGIN onboarding-block === (Template 3 — Split Card, feature 008)
|
|
1367
|
-
window.MUDRA_ONBOARDING_ACTIONS = [
|
|
1368
|
-
// Filled by the skill from the app's subscribed signals. See "App-aware filter — strict" below.
|
|
1369
|
-
];
|
|
1370
|
-
|
|
1371
|
-
(function () {
|
|
1372
|
-
const root = document.getElementById('mudra-onboarding');
|
|
1373
|
-
const help = document.getElementById('mudra-onboarding-help');
|
|
1374
|
-
const grid = document.getElementById('ob-rows');
|
|
1375
|
-
|
|
1376
|
-
grid.innerHTML = window.MUDRA_ONBOARDING_ACTIONS.map(r => `
|
|
1377
|
-
<div class="ob-chip">
|
|
1378
|
-
<span class="nm">${r.action}</span>
|
|
1379
|
-
<span class="mu">${r.mudra}</span>
|
|
1380
|
-
<span class="mn">${r.manual}</span>
|
|
1381
|
-
</div>`).join('');
|
|
1382
|
-
|
|
1383
|
-
function isInImmersiveXR() {
|
|
1384
|
-
return !!(window.xb && window.xb.session && window.xb.session.isImmersive);
|
|
1385
|
-
}
|
|
1386
|
-
function openOb() { if (!root.open) root.showModal(); help.hidden = true; }
|
|
1387
|
-
function closeOb() { if (root.open) root.close(); help.hidden = false; }
|
|
1388
|
-
|
|
1389
|
-
// Close wiring — BOTH `.ob-x` and `.ob-continue` carry [data-ob-close].
|
|
1390
|
-
// This single querySelectorAll attaches the same `closeOb` to each;
|
|
1391
|
-
// do not add separate listeners or split the behavior.
|
|
1392
|
-
root.querySelectorAll('[data-ob-close]').forEach(b => b.addEventListener('click', closeOb));
|
|
1393
|
-
|
|
1394
|
-
// Re-open wiring — floating help-pill (mouse/touch) and the `?` key.
|
|
1395
|
-
// `?` is gated on isInImmersiveXR() so it never re-opens inside VR/AR.
|
|
1396
|
-
help.addEventListener('click', openOb);
|
|
1397
|
-
document.addEventListener('keydown', (e) => {
|
|
1398
|
-
if (e.key === 'Escape' && root.open) closeOb();
|
|
1399
|
-
if (e.key === '?' && !root.open && !isInImmersiveXR()) openOb();
|
|
1400
|
-
});
|
|
1401
|
-
|
|
1402
|
-
// Hide modal + help during immersive XR sessions (preserves feature 005's contract).
|
|
1403
|
-
['xrsession-start','vr-session-start','ar-session-start'].forEach(ev =>
|
|
1404
|
-
window.addEventListener(ev, () => { if (root.open) root.close(); help.hidden = true; })
|
|
1405
|
-
);
|
|
1406
|
-
|
|
1407
|
-
// Open on load — UNCONDITIONAL showModal(). The dialog enters the
|
|
1408
|
-
// browser's top layer so XR Blocks' canvas (appended after this
|
|
1409
|
-
// script) can never intercept clicks on Continue / ×. Do not gate on
|
|
1410
|
-
// `root.open`; do not use `root.show()`.
|
|
1411
|
-
root.showModal();
|
|
1412
|
-
})();
|
|
1413
|
-
// === END onboarding-block ===
|
|
1414
|
-
```
|
|
1415
|
-
|
|
1416
|
-
### Close behavior — what each control does (verbatim explanation for the model)
|
|
1417
|
-
|
|
1418
|
-
| Control | When it fires | Effect |
|
|
1419
|
-
|---|---|---|
|
|
1420
|
-
| **Continue** (`.ob-continue`) | User clicks / activates the primary CTA in the right column | `root.close()`. Help-pill `?` becomes visible bottom-right. User starts using the app (or enters XR via the XR Blocks button). |
|
|
1421
|
-
| **×** (`.ob-x`) | User clicks / activates the circular X in the top-right of the card | Same as Continue — `root.close()` + show `?` pill. The two paths are intentionally equivalent. X is the visual "skip" affordance. (Exception: AI-Setup extension blocks both Continue and × until a valid API key is entered — see "AI-app extension" below.) |
|
|
1422
|
-
| **Escape** | User presses Esc while modal is open | Native `<dialog>` close + our `keydown` handler unhides the `?` pill. (AI extension also intercepts this via the `cancel` event when no valid key is set.) |
|
|
1423
|
-
| **`?` key** | User presses `?` while modal is closed AND NOT inside immersive XR | `root.showModal()` reopens. Inside VR/AR the key is a no-op so the user is never yanked out of the scene. |
|
|
1424
|
-
| **Help-pill** (`#mudra-onboarding-help`) | User clicks the floating `?` button bottom-right (visible only after the modal has been closed once, and hidden during immersive XR) | `root.showModal()` reopens. |
|
|
1425
|
-
| **`xrsession-start` / `vr-session-start` / `ar-session-start`** | User enters VR/AR via the XR Blocks Enter button | Force-close the modal; hide the help-pill so it does not appear in-world. |
|
|
1426
|
-
| **Page load** | Every fresh render | `root.showModal()` runs unconditionally inside the IIFE. Help-pill starts hidden. |
|
|
1427
|
-
|
|
1428
|
-
### How "close" is wired (REQUIRED — do not refactor)
|
|
1429
|
-
|
|
1430
|
-
1. Both close buttons carry the `data-ob-close` attribute. The single line `root.querySelectorAll('[data-ob-close]').forEach(b => b.addEventListener('click', closeOb))` is the entire mouse/touch close wiring. The AI-Setup extension below adds its own `capture:true` listener on top of these to gate close when no API key is present — it does not replace the underlying wiring.
|
|
1431
|
-
2. Keyboard close is handled by the `document`-level `keydown` listener so Escape fires even when focus is outside the dialog.
|
|
1432
|
-
3. Opening on load uses `root.showModal()` (modal mode, top layer). Never `root.show()` (non-modal) and never the `open` attribute in HTML. In 3D apps this is REQUIRED — non-modal dialogs do not enter the top layer and XR Blocks' canvas can intercept clicks behind the modal.
|
|
1433
|
-
|
|
1434
|
-
### Anti-patterns — will fail review
|
|
1435
|
-
|
|
1436
|
-
- ❌ `<dialog ... open>` in HTML. Open via `showModal()` only.
|
|
1437
|
-
- ❌ Calling `root.show()` instead of `root.showModal()`.
|
|
1438
|
-
- ❌ Omitting `options.simulator.instructions.enabled = false;` — XR Blocks' Welcome overlay then competes with the Mudra modal.
|
|
1439
|
-
- ❌ Adding click-outside-to-close, time-out auto-close, or any extra close path.
|
|
1440
|
-
- ❌ Hiding `#mudra-onboarding-help` permanently after first close — it must reappear (and remain hidden ONLY during immersive XR).
|
|
1441
|
-
- ❌ Wiring `closeOb` to a button that lacks the `data-ob-close` attribute.
|
|
1442
|
-
- ❌ Differentiating × from Continue behaviorally (e.g., "X means skip, Continue means save"). They are the same close path.
|
|
1443
|
-
|
|
1444
|
-
### Per-app content slots — the ONLY things you may vary
|
|
1445
|
-
|
|
1446
|
-
| Slot | Source | Notes |
|
|
1447
|
-
|---|---|---|
|
|
1448
|
-
| `{APP_NAME}` (`data-app-name`) | Derived from generated HTML filename | Override only to fix acronym capitalization (e.g., `data-app-name="AR Menu"` for `ar-menu.html`). |
|
|
1449
|
-
| `{APP_NAME_HEAD}` / `{APP_NAME_TAIL}` | App name split into leading word + trailing word(s). Trailing word gets the `<em>` accent. | Single-word names: HEAD is whole word, TAIL is empty (emit `<em></em>`). |
|
|
1450
|
-
| `{APP_TAGLINE}` | One-line description. MUST end with a period. MUST NOT exceed 90 characters. | Example: "Stack blocks with a flick of the wrist." |
|
|
1451
|
-
| `MUDRA_ONBOARDING_ACTIONS` | Per `actions-array.md`. | See filter rule below. |
|
|
1452
|
-
|
|
1453
|
-
### `MUDRA_ONBOARDING_ACTIONS` shape (renamed from feature-005 `ACTIONS`)
|
|
1454
|
-
|
|
1455
|
-
```js
|
|
1456
|
-
window.MUDRA_ONBOARDING_ACTIONS = [
|
|
1457
|
-
{ action: "Toggle XR session", mudra: "Thumb tap", manual: "Enter", mode: "gesture" },
|
|
1458
|
-
{ action: "Look around", mudra: "Lift wrist", manual: "Right-click + drag", mode: "imu_acc" },
|
|
1459
|
-
{ action: "Move", mudra: "Swipe", manual: "W A S D", mode: "nav_direction" }
|
|
1460
|
-
];
|
|
1461
|
-
```
|
|
1462
|
-
|
|
1463
|
-
Each row has four required fields:
|
|
1464
|
-
|
|
1465
|
-
- **`action`** — the behavior in plain English (NOT the control name).
|
|
1466
|
-
- **`mudra`** — the Mudra control prose (`"Tap"`, `"Twist"`, `"Press 70%"`, `"Tilt left"`).
|
|
1467
|
-
- **`manual`** — keyboard / mouse fallback. Use `"—"` (em dash) if there is no Manual equivalent (common in XR for camera / look controls).
|
|
1468
|
-
- **`mode`** — one of: `gesture` | `button` | `pressure` | `navigation` | `nav_direction` | `imu_acc` | `imu_gyro` | `snc`.
|
|
1469
|
-
|
|
1470
|
-
### App-aware filter — STRICT (feature 008, FR-010)
|
|
1471
|
-
|
|
1472
|
-
Before emitting `MUDRA_ONBOARDING_ACTIONS`, the skill MUST filter:
|
|
1473
|
-
|
|
1474
|
-
1. **Build the subscribed set** — every canonical signal this specific app subscribes to.
|
|
1475
|
-
2. **Drop every row** whose `mode` is not in the subscribed set. **Forbidden** to emit a row for an unsubscribed signal — that is a generation-time bug, not a runtime filter.
|
|
1476
|
-
3. **Verify exactly one motion mode** (`navigation` | `nav_direction` | `imu_acc`/`imu_gyro` family) across all rows (Constitution Principle III).
|
|
1477
|
-
4. **Verify Manual ↔ Mudra parity** — every row has both a Mudra and a Manual cell; `"—"` is the only acceptable Manual placeholder.
|
|
1478
|
-
5. **No collisions** — no two rows share the same `manual` value or the same effective `mudra` value.
|
|
1479
|
-
|
|
1480
|
-
### Anti-patterns (will fail review)
|
|
1481
|
-
|
|
1482
|
-
- ❌ Omitting `options.simulator.instructions.enabled = false;` — the XR Blocks default Welcome overlay then duplicates / overrides the Mudra onboarding modal.
|
|
1483
|
-
- ❌ Mirroring the modal as a 3D panel inside the XR scene (out of scope v1).
|
|
1484
|
-
- ❌ Editing the `xrsession-start` / `xrsession-end` hide-show wiring.
|
|
1485
|
-
- ❌ Emitting a row whose `mode` is not in the app's subscribed set.
|
|
1486
|
-
- ❌ Mixing two motion modes in the same array.
|
|
1487
|
-
- ❌ A row with `manual: null` or `manual: ""` — use `"—"`.
|
|
1488
|
-
- ❌ Two rows with the same `manual` value — keyboard collision.
|
|
1489
|
-
- ❌ Branding strings other than literal `Created by Mudra`. ("Mudra Studio" stays only as the `.ob-brand-mark` line above the app name.)
|
|
1490
|
-
- ❌ CTA labels other than `Continue`.
|
|
1491
|
-
|
|
1492
|
-
Binding contracts: `specs/008-strict-onboarding-templates/contracts/onboarding-block.md` and `specs/008-strict-onboarding-templates/contracts/actions-array.md`.
|
|
1493
|
-
|
|
1494
|
-
### AI-app extension (mandatory when `usesAI`)
|
|
1495
|
-
|
|
1496
|
-
The AI-Setup fragment is rendered as an **extra row inside `.ob-chip-grid`** — not as a separate section. The fragment goes at the **top** of the chip grid (above all action chips) so the user sees it before any controls.
|
|
1497
|
-
|
|
1498
|
-
```html
|
|
1499
|
-
<!-- Inject AS THE FIRST CHILD of #ob-rows when usesAI is true -->
|
|
1500
|
-
<div class="ob-chip ob-chip-ai" id="ob-chip-ai" data-role="ai-setup">
|
|
1501
|
-
<span class="nm">AI key</span>
|
|
1502
|
-
<input class="mudra-onb__ai-input" type="password" placeholder="AIza…" aria-label="Gemini API key" autocomplete="off" />
|
|
1503
|
-
<span class="mn" data-role="hint" aria-live="polite"></span>
|
|
1504
|
-
</div>
|
|
1505
|
-
```
|
|
1506
|
-
|
|
1507
|
-
```css
|
|
1508
|
-
/* Append to the locked onboarding-block CSS for AI-using apps only */
|
|
1509
|
-
.ob-chip-ai{grid-template-columns:auto 1fr auto;}
|
|
1510
|
-
.ob-chip-ai .mudra-onb__ai-input{appearance:none;background:rgba(255,255,255,0.04);color:var(--text);border:1px solid rgba(255,255,255,0.12);border-radius:8px;padding:6px 10px;font:inherit;font-size:13px;}
|
|
1511
|
-
.ob-chip-ai .mudra-onb__ai-input:focus{outline:2px solid var(--primary);outline-offset:0;}
|
|
1512
|
-
.ob-chip-ai .mn[data-role="hint"]{color:var(--warning);}
|
|
1513
|
-
```
|
|
1514
|
-
|
|
1515
|
-
The IIFE that wires the modal MUST be extended with this gating block, inserted just before the final unconditional `root.showModal();`:
|
|
1516
|
-
|
|
1517
|
-
```js
|
|
1518
|
-
// AI-Setup gating — only runs when the fragment is present (usesAI === true)
|
|
1519
|
-
const aiFragment = document.getElementById('ob-chip-ai');
|
|
1520
|
-
const continueBtn = root.querySelector('.ob-continue');
|
|
1521
|
-
const closeBtn = root.querySelector('.ob-x');
|
|
1522
|
-
const KEY_NAME = "mudra.gemini.apiKey";
|
|
1523
|
-
const KEY_REGEX = /^AIza[\w-]{30,}$/;
|
|
1524
|
-
|
|
1525
|
-
const hasValidStoredKey = () => {
|
|
1526
|
-
const k = sessionStorage.getItem(KEY_NAME);
|
|
1527
|
-
return typeof k === "string" && KEY_REGEX.test(k.trim());
|
|
1528
|
-
};
|
|
1529
|
-
|
|
1530
|
-
if (aiFragment) {
|
|
1531
|
-
const input = aiFragment.querySelector(".mudra-onb__ai-input");
|
|
1532
|
-
const hint = aiFragment.querySelector('[data-role="hint"]');
|
|
1533
|
-
|
|
1534
|
-
const refreshGate = () => {
|
|
1535
|
-
if (hasValidStoredKey()) {
|
|
1536
|
-
aiFragment.hidden = true;
|
|
1537
|
-
continueBtn.disabled = false;
|
|
1538
|
-
return;
|
|
1539
|
-
}
|
|
1540
|
-
aiFragment.hidden = false;
|
|
1541
|
-
const v = input.value.trim();
|
|
1542
|
-
const ok = KEY_REGEX.test(v);
|
|
1543
|
-
continueBtn.disabled = !ok;
|
|
1544
|
-
hint.textContent = (!v || ok) ? "" : "Key should start with \"AIza\" and be ~39 chars.";
|
|
1545
|
-
};
|
|
1546
|
-
|
|
1547
|
-
input.addEventListener("input", refreshGate);
|
|
1548
|
-
refreshGate();
|
|
1549
|
-
|
|
1550
|
-
continueBtn.addEventListener("click", () => {
|
|
1551
|
-
const v = input.value.trim();
|
|
1552
|
-
if (KEY_REGEX.test(v)) sessionStorage.setItem(KEY_NAME, v);
|
|
1553
|
-
}, { capture: true });
|
|
1554
|
-
|
|
1555
|
-
const blockClose = (e) => {
|
|
1556
|
-
if (hasValidStoredKey()) return;
|
|
1557
|
-
e.preventDefault();
|
|
1558
|
-
e.stopPropagation();
|
|
1559
|
-
input.focus();
|
|
1560
|
-
};
|
|
1561
|
-
closeBtn.addEventListener("click", blockClose, { capture: true });
|
|
1562
|
-
root.addEventListener("cancel", blockClose); // Escape
|
|
1563
|
-
}
|
|
1564
|
-
```
|
|
1565
|
-
|
|
1566
|
-
When the AI fragment is **absent** (non-AI app), the existing wiring runs untouched — `Continue` enables immediately, `Escape` / `×` dismiss as normal.
|
|
1567
|
-
|
|
1568
|
-
> **Migration note (feature 008, 2026-05-14):** the legacy `ACTIONS` variable is renamed to `MUDRA_ONBOARDING_ACTIONS`. The legacy `mudra-onb__*` class names from feature 005 are replaced by `ob-*` (the locked block's class names). The `Got it` CTA label is renamed to `Continue`. The AI-Setup fragment moves from `.mudra-onb__body` (separate section) into `.ob-chip-grid` (first chip).
|
|
1569
|
-
|
|
1570
|
-
---
|
|
1571
|
-
|
|
1572
|
-
## Section 18 — Visible AI Chat I/O (mandatory when `usesAI`)
|
|
1573
|
-
|
|
1574
|
-
Every AI-using app MUST render the conversation as on-screen text in the
|
|
1575
|
-
3D scene — TTS / speech synthesis is optional, never a substitute.
|
|
1576
|
-
|
|
1577
|
-
### Required visible elements
|
|
1578
|
-
|
|
1579
|
-
| Element | What it shows | How to render |
|
|
1580
|
-
|---------|---------------|---------------|
|
|
1581
|
-
| **Purpose line** | One short sentence stating what the app does ("Ask anything — I'll answer.", "Tell me the date.", etc.). Visible at all times. | Troika `Text` or a top row in an `xb.SpatialPanel`. |
|
|
1582
|
-
| **User input echo** | The latest user message (transcript from speech recognition OR typed text). Updates as soon as input is captured. | A bordered/highlighted row in the panel, e.g. prefixed `💬 You: …`. |
|
|
1583
|
-
| **AI response** | The most recent AI reply in full readable text. Scrollable / wrapping. | `xb.ScrollingTroikaTextView` or a tall row in the panel, prefixed `🤖 AI: …`. |
|
|
1584
|
-
| **Listening / Thinking indicator** | Distinguishes idle / listening / thinking states. | Avatar pulse + a single-word status line ("Listening…", "Thinking…", "Tap to talk"). |
|
|
1585
|
-
|
|
1586
|
-
### Rules
|
|
1587
|
-
|
|
1588
|
-
1. **Both sides of every exchange must be visible.** Voice-only output
|
|
1589
|
-
is a checklist failure. The chat panel must accumulate at least the
|
|
1590
|
-
last user turn AND the last AI turn at the same time.
|
|
1591
|
-
2. **The Purpose line is fixed** for the lifetime of the app. Author it
|
|
1592
|
-
from the user's prompt — e.g. `"Create 3d AI I can ask the date"` →
|
|
1593
|
-
Purpose `"Voice assistant — ask anything, tap to talk."`. Never use
|
|
1594
|
-
a placeholder like `"AI Chat"` alone.
|
|
1595
|
-
3. **Show typed input when speech recognition is unavailable.** If
|
|
1596
|
-
`webkitSpeechRecognition` / `SpeechRecognition` are missing, render
|
|
1597
|
-
an `<input type="text">` inside an XR Blocks panel OR a 2D overlay
|
|
1598
|
-
below the simulator panel. Both echo and reply still render in the
|
|
1599
|
-
3D scene.
|
|
1600
|
-
4. **When no API key is set yet**, the chat panel renders the literal
|
|
1601
|
-
text `Set up AI in the welcome panel` in the response slot — do not
|
|
1602
|
-
attempt any API call.
|
|
1603
|
-
5. **TTS** (`speechSynthesis`) is allowed but optional. If present, it
|
|
1604
|
-
speaks the AI response in addition to displaying it. Never as a
|
|
1605
|
-
replacement.
|