mudra-skills 3.0.2 → 3.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (146) hide show
  1. package/package.json +1 -1
  2. package/skills/mudra-master/SKILL.md +0 -196
  3. package/skills/mudra-master/mudra-preview/SKILL.md +0 -150
  4. package/skills/mudra-master/mudra-preview/assets/ar-menu.html +0 -382
  5. package/skills/mudra-master/mudra-preview/assets/document-scroller.html +0 -468
  6. package/skills/mudra-master/mudra-preview/assets/drum-machine.html +0 -788
  7. package/skills/mudra-master/mudra-preview/assets/emg-visualizer.html +0 -390
  8. package/skills/mudra-master/mudra-preview/assets/generative-art.html +0 -408
  9. package/skills/mudra-master/mudra-preview/assets/gesture-assistant.html +0 -342
  10. package/skills/mudra-master/mudra-preview/assets/gesture-speech.html +0 -444
  11. package/skills/mudra-master/mudra-preview/assets/hands-free-desktop.html +0 -376
  12. package/skills/mudra-master/mudra-preview/assets/model-rotator.html +0 -365
  13. package/skills/mudra-master/mudra-preview/assets/mudra-duel.html +0 -3200
  14. package/skills/mudra-master/mudra-preview/assets/mudra-monitor.html +0 -1451
  15. package/skills/mudra-master/mudra-preview/assets/mudra-ultimate-template.html +0 -891
  16. package/skills/mudra-master/mudra-preview/assets/music-sequencer.html +0 -455
  17. package/skills/mudra-master/mudra-preview/assets/neural-pong.html +0 -380
  18. package/skills/mudra-master/mudra-preview/assets/neural-snake.html +0 -364
  19. package/skills/mudra-master/mudra-preview/assets/presentation-controller.html +0 -345
  20. package/skills/mudra-master/mudra-preview/assets/pressure-painter.html +0 -285
  21. package/skills/mudra-master/mudra-preview/assets/runner.html +0 -1812
  22. package/skills/mudra-master/mudra-preview/assets/smart-home.html +0 -461
  23. package/skills/mudra-master/mudra-preview/assets/space-invaders.html +0 -1154
  24. package/skills/mudra-master/mudra-preview/assets/waterful-ring-toss.html +0 -1624
  25. package/skills/mudra-master/mudra-preview/references/agent_protocol.json +0 -634
  26. package/skills/mudra-master/mudra-preview/references/prompt.md +0 -16746
  27. package/skills/mudra-master/mudra-xr/SKILL.md +0 -245
  28. package/skills/mudra-master/mudra-xr/assets/demos/3dgs-walkthrough.html +0 -242
  29. package/skills/mudra-master/mudra-xr/assets/demos/aisimulator.html +0 -792
  30. package/skills/mudra-master/mudra-xr/assets/demos/balloonpop.html +0 -422
  31. package/skills/mudra-master/mudra-xr/assets/demos/ballpit.html +0 -284
  32. package/skills/mudra-master/mudra-xr/assets/demos/drone.html +0 -41
  33. package/skills/mudra-master/mudra-xr/assets/demos/gemini-icebreakers.html +0 -338
  34. package/skills/mudra-master/mudra-xr/assets/demos/gemini-xrobject.html +0 -302
  35. package/skills/mudra-master/mudra-xr/assets/demos/math3d.html +0 -188
  36. package/skills/mudra-master/mudra-xr/assets/demos/measure.html +0 -247
  37. package/skills/mudra-master/mudra-xr/assets/demos/occlusion.html +0 -251
  38. package/skills/mudra-master/mudra-xr/assets/demos/rain.html +0 -498
  39. package/skills/mudra-master/mudra-xr/assets/demos/screenwiper.html +0 -432
  40. package/skills/mudra-master/mudra-xr/assets/demos/splash.html +0 -519
  41. package/skills/mudra-master/mudra-xr/assets/demos/webcam_gestures.html +0 -253
  42. package/skills/mudra-master/mudra-xr/assets/demos/xremoji.html +0 -833
  43. package/skills/mudra-master/mudra-xr/assets/demos/xrpoet.html +0 -151
  44. package/skills/mudra-master/mudra-xr/assets/samples/depthmap.html +0 -266
  45. package/skills/mudra-master/mudra-xr/assets/samples/depthmesh.html +0 -100
  46. package/skills/mudra-master/mudra-xr/assets/samples/game_rps.html +0 -1099
  47. package/skills/mudra-master/mudra-xr/assets/samples/gestures_custom.html +0 -467
  48. package/skills/mudra-master/mudra-xr/assets/samples/gestures_heuristic.html +0 -256
  49. package/skills/mudra-master/mudra-xr/assets/samples/lighting.html +0 -163
  50. package/skills/mudra-master/mudra-xr/assets/samples/mesh_detection.html +0 -46
  51. package/skills/mudra-master/mudra-xr/assets/samples/modelviewer.html +0 -183
  52. package/skills/mudra-master/mudra-xr/assets/samples/paint.html +0 -133
  53. package/skills/mudra-master/mudra-xr/assets/samples/planar-vst.html +0 -45
  54. package/skills/mudra-master/mudra-xr/assets/samples/reticle.html +0 -95
  55. package/skills/mudra-master/mudra-xr/assets/samples/skybox_agent.html +0 -384
  56. package/skills/mudra-master/mudra-xr/assets/samples/sound.html +0 -390
  57. package/skills/mudra-master/mudra-xr/assets/samples/ui.html +0 -496
  58. package/skills/mudra-master/mudra-xr/assets/samples/virtual-screens.html +0 -808
  59. package/skills/mudra-master/mudra-xr/assets/templates/0_basic.html +0 -64
  60. package/skills/mudra-master/mudra-xr/assets/templates/1_ui.html +0 -107
  61. package/skills/mudra-master/mudra-xr/assets/templates/2_hands.html +0 -183
  62. package/skills/mudra-master/mudra-xr/assets/templates/3_depth.html +0 -89
  63. package/skills/mudra-master/mudra-xr/assets/templates/4_stereo.html +0 -84
  64. package/skills/mudra-master/mudra-xr/assets/templates/5_camera.html +0 -117
  65. package/skills/mudra-master/mudra-xr/assets/templates/6_ai.html +0 -154
  66. package/skills/mudra-master/mudra-xr/assets/templates/7_ai_live.html +0 -253
  67. package/skills/mudra-master/mudra-xr/assets/templates/8_objects.html +0 -95
  68. package/skills/mudra-master/mudra-xr/assets/templates/9_xr-toggle.html +0 -90
  69. package/skills/mudra-master/mudra-xr/assets/templates/heuristic_hand_gestures.html +0 -102
  70. package/skills/mudra-master/mudra-xr/assets/templates/meshes.html +0 -47
  71. package/skills/mudra-master/mudra-xr/assets/templates/planes.html +0 -47
  72. package/skills/mudra-master/mudra-xr/assets/templates/uikit.html +0 -187
  73. package/skills/mudra-master/mudra-xr/references/agent_protocol.json +0 -865
  74. package/skills/mudra-master/mudra-xr/references/prompt.md +0 -1605
  75. package/skills/mudra-preview/SKILL.md +0 -159
  76. package/skills/mudra-preview/assets/ar-menu.html +0 -382
  77. package/skills/mudra-preview/assets/document-scroller.html +0 -468
  78. package/skills/mudra-preview/assets/drum-machine.html +0 -788
  79. package/skills/mudra-preview/assets/emg-visualizer.html +0 -390
  80. package/skills/mudra-preview/assets/generative-art.html +0 -408
  81. package/skills/mudra-preview/assets/gesture-assistant.html +0 -342
  82. package/skills/mudra-preview/assets/gesture-speech.html +0 -444
  83. package/skills/mudra-preview/assets/hands-free-desktop.html +0 -376
  84. package/skills/mudra-preview/assets/model-rotator.html +0 -365
  85. package/skills/mudra-preview/assets/mudra-duel.html +0 -3200
  86. package/skills/mudra-preview/assets/mudra-monitor.html +0 -1451
  87. package/skills/mudra-preview/assets/mudra-ultimate-template.html +0 -997
  88. package/skills/mudra-preview/assets/music-sequencer.html +0 -455
  89. package/skills/mudra-preview/assets/neural-pong.html +0 -380
  90. package/skills/mudra-preview/assets/neural-snake.html +0 -364
  91. package/skills/mudra-preview/assets/presentation-controller.html +0 -345
  92. package/skills/mudra-preview/assets/pressure-painter.html +0 -285
  93. package/skills/mudra-preview/assets/runner.html +0 -1812
  94. package/skills/mudra-preview/assets/smart-home.html +0 -461
  95. package/skills/mudra-preview/assets/space-invaders.html +0 -1154
  96. package/skills/mudra-preview/assets/waterful-ring-toss.html +0 -1624
  97. package/skills/mudra-preview/references/agent_protocol.json +0 -701
  98. package/skills/mudra-preview/references/prompt.md +0 -16731
  99. package/skills/mudra-xr/SKILL.md +0 -245
  100. package/skills/mudra-xr/assets/demos/3dgs-walkthrough.html +0 -242
  101. package/skills/mudra-xr/assets/demos/aisimulator.html +0 -792
  102. package/skills/mudra-xr/assets/demos/balloonpop.html +0 -422
  103. package/skills/mudra-xr/assets/demos/ballpit.html +0 -284
  104. package/skills/mudra-xr/assets/demos/drone.html +0 -41
  105. package/skills/mudra-xr/assets/demos/gemini-icebreakers.html +0 -338
  106. package/skills/mudra-xr/assets/demos/gemini-xrobject.html +0 -302
  107. package/skills/mudra-xr/assets/demos/math3d.html +0 -188
  108. package/skills/mudra-xr/assets/demos/measure.html +0 -247
  109. package/skills/mudra-xr/assets/demos/occlusion.html +0 -251
  110. package/skills/mudra-xr/assets/demos/rain.html +0 -498
  111. package/skills/mudra-xr/assets/demos/screenwiper.html +0 -432
  112. package/skills/mudra-xr/assets/demos/splash.html +0 -519
  113. package/skills/mudra-xr/assets/demos/webcam_gestures.html +0 -253
  114. package/skills/mudra-xr/assets/demos/xremoji.html +0 -833
  115. package/skills/mudra-xr/assets/demos/xrpoet.html +0 -151
  116. package/skills/mudra-xr/assets/samples/depthmap.html +0 -266
  117. package/skills/mudra-xr/assets/samples/depthmesh.html +0 -100
  118. package/skills/mudra-xr/assets/samples/game_rps.html +0 -1099
  119. package/skills/mudra-xr/assets/samples/gestures_custom.html +0 -467
  120. package/skills/mudra-xr/assets/samples/gestures_heuristic.html +0 -256
  121. package/skills/mudra-xr/assets/samples/lighting.html +0 -163
  122. package/skills/mudra-xr/assets/samples/mesh_detection.html +0 -46
  123. package/skills/mudra-xr/assets/samples/modelviewer.html +0 -183
  124. package/skills/mudra-xr/assets/samples/paint.html +0 -133
  125. package/skills/mudra-xr/assets/samples/planar-vst.html +0 -45
  126. package/skills/mudra-xr/assets/samples/reticle.html +0 -95
  127. package/skills/mudra-xr/assets/samples/skybox_agent.html +0 -384
  128. package/skills/mudra-xr/assets/samples/sound.html +0 -390
  129. package/skills/mudra-xr/assets/samples/ui.html +0 -496
  130. package/skills/mudra-xr/assets/samples/virtual-screens.html +0 -808
  131. package/skills/mudra-xr/assets/templates/0_basic.html +0 -64
  132. package/skills/mudra-xr/assets/templates/1_ui.html +0 -107
  133. package/skills/mudra-xr/assets/templates/2_hands.html +0 -183
  134. package/skills/mudra-xr/assets/templates/3_depth.html +0 -89
  135. package/skills/mudra-xr/assets/templates/4_stereo.html +0 -84
  136. package/skills/mudra-xr/assets/templates/5_camera.html +0 -117
  137. package/skills/mudra-xr/assets/templates/6_ai.html +0 -154
  138. package/skills/mudra-xr/assets/templates/7_ai_live.html +0 -253
  139. package/skills/mudra-xr/assets/templates/8_objects.html +0 -95
  140. package/skills/mudra-xr/assets/templates/9_xr-toggle.html +0 -90
  141. package/skills/mudra-xr/assets/templates/heuristic_hand_gestures.html +0 -102
  142. package/skills/mudra-xr/assets/templates/meshes.html +0 -47
  143. package/skills/mudra-xr/assets/templates/planes.html +0 -47
  144. package/skills/mudra-xr/assets/templates/uikit.html +0 -187
  145. package/skills/mudra-xr/references/agent_protocol.json +0 -865
  146. package/skills/mudra-xr/references/prompt.md +0 -1605
@@ -1,1605 +0,0 @@
1
- # Mudra XR Skill — Build Rules
2
-
3
- This document is the canonical reference for the `mudra-xr` Claude Code skill.
4
- Read it in full before generating any app. Every rule below is non-negotiable
5
- unless explicitly marked as configurable.
6
-
7
- ---
8
-
9
- ## Section 1 — Mudra Protocol
10
-
11
- ### WebSocket endpoint
12
-
13
- ```
14
- ws://127.0.0.1:8766
15
- ```
16
-
17
- Always construct the connection through `MudraClient` (Section 4).
18
- Never use raw `new WebSocket(...)`.
19
-
20
- ### Eight canonical signals
21
-
22
- | Signal | Category | Description |
23
- |--------|----------|-------------|
24
- | `gesture` | Discrete | Hand gesture events (tap, double_tap, twist, double_twist) |
25
- | `button` | Discrete | Button hold / release |
26
- | `pressure` | Analog | Finger pressure 0–100, normalized 0–1 |
27
- | `navigation` | Motion (Pointer) | Continuous delta_x / delta_y cursor movement |
28
- | `nav_direction` | Motion (Direction) | Discrete directional swipes: None, Right, Left, Up, Down, Roll Left, Roll Right |
29
- | `imu_acc` | Motion (IMU) | Accelerometer values [x, y, z] m/s², frequency 1125 Hz |
30
- | `imu_gyro` | Motion (IMU) | Gyroscope values [x, y, z] deg/s, frequency 1125 Hz |
31
- | `snc` | Biometric | 3 de-interleaved channel arrays [[ch1], [ch2], [ch3]] |
32
-
33
-
34
- ### Subscription handshake
35
-
36
- Send one command per signal — never use plural `signals`, arrays, or batch commands:
37
-
38
- ```js
39
- // CORRECT
40
- ws.send(JSON.stringify({ command: 'subscribe', signal: 'gesture' }));
41
- ws.send(JSON.stringify({ command: 'subscribe', signal: 'pressure' }));
42
-
43
- // WRONG — never do this
44
- ws.send(JSON.stringify({ command: 'subscribe', signals: ['gesture', 'pressure'] }));
45
- ws.send(JSON.stringify({ command: 'subscribe', signal: ['gesture', 'pressure'] }));
46
- ```
47
-
48
- ### Full command surface
49
-
50
- `subscribe`, `unsubscribe`, `get_subscriptions`, `get_status`, `trigger_gesture`
51
-
52
- **Removed**: `enable`, `disable`, `get_docs` — the new Dart server rejects them with `unknown_command`.
53
-
54
- ### Inbound message payload shapes
55
-
56
- ```js
57
- // gesture — no confidence field
58
- { type: 'gesture', data: { type: 'tap'|'double_tap'|'twist'|'double_twist', timestamp }, timestamp }
59
-
60
- // button
61
- { type: 'button', data: { state: 'pressed'|'released', timestamp }, timestamp }
62
-
63
- // pressure
64
- { type: 'pressure', data: { value: 0–100, normalized: 0–1, timestamp }, timestamp }
65
-
66
- // navigation
67
- { type: 'navigation', data: { delta_x: number, delta_y: number, timestamp }, timestamp }
68
-
69
- // nav_direction
70
- { type: 'nav_direction', data: { direction: 'Right'|'Left'|'Up'|'Down'|'Roll Left'|'Roll Right'|'None', timestamp }, timestamp }
71
-
72
- // imu_acc
73
- { type: 'imu_acc', data: { values: [x, y, z], frequency: number, frequency_std: number, timestamp }, timestamp }
74
-
75
- // imu_gyro
76
- { type: 'imu_gyro', data: { values: [x, y, z], frequency: number, frequency_std: number, timestamp }, timestamp }
77
-
78
- // snc — extend rolling buffers (500 samples/channel) with all samples per callback
79
- { type: 'snc', data: { values: [[ch1_samples], [ch2_samples], [ch3_samples]], frequency: number, frequency_std: number, timestamp }, timestamp }
80
-
81
- // status — response to get_status command
82
- { type: 'status', data: { device: { name, address, battery, charging, firmware, serial_number, hand, state, firmware_config: { target, active } }, subscriptions: { snc, imu_acc, imu_gyro, pressure, gesture, navigation, nav_direction, button } }, timestamp }
83
-
84
- // subscription_status — response to subscribe/unsubscribe
85
- { type: 'subscription_status', data: { signal: string, subscribed: boolean }, timestamp }
86
- ```
87
-
88
- **Removed**: `confidence` field from gesture (not emitted by Dart server).
89
-
90
- **Legacy / not emitted by Dart server**: `connection_status` frames were sent by the old Python server on connect. The new Dart server sends **no unsolicited frames** on open — do NOT gate any UI on receiving `connection_status`.
91
-
92
- ### Error frames
93
-
94
- The server emits `{type:"error", data:{error:"<code>", message:"..."}, timestamp}` for protocol violations. Codes generated apps may receive:
95
-
96
- | `data.error` code | Meaning | App must do |
97
- |---|---|---|
98
- | `invalid_json` | Malformed JSON sent | Log; never expected from generated apps |
99
- | `missing_command` | No `command` field | Log; never expected |
100
- | `unknown_command` | Command not in the supported set | Log; never expected after spec 001 |
101
- | `missing_signal` | `subscribe`/`unsubscribe` without `signal` | Log; never expected |
102
- | `invalid_signal` | Unrecognised signal name | Log; never expected after spec 001 |
103
- | `client_already_connected` | Another tab/process owns the single client slot | **Show terminal "close the other tab" message; do NOT retry** |
104
-
105
- For any unexpected error code: log and remain in current state. Do not crash the scene.
106
-
107
- ---
108
-
109
- ## Section 2 — Canonical Dependency Pins
110
-
111
- Use the exact versions below. Never use `@latest` or version ranges.
112
- Import map `<script type="importmap">` must contain **only** the dependencies
113
- the app actually uses — no unused entries.
114
-
115
- ```json
116
- {
117
- "imports": {
118
- "three": "https://cdn.jsdelivr.net/npm/three@0.182.0/build/three.module.js",
119
- "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.182.0/examples/jsm/",
120
- "troika-three-text": "https://cdn.jsdelivr.net/gh/protectwise/troika@028b81cf308f0f22e5aa8e78196be56ec1997af5/packages/troika-three-text/src/index.js",
121
- "troika-three-utils": "https://cdn.jsdelivr.net/gh/protectwise/troika@v0.52.4/packages/troika-three-utils/src/index.js",
122
- "troika-worker-utils": "https://cdn.jsdelivr.net/gh/protectwise/troika@v0.52.4/packages/troika-worker-utils/src/index.js",
123
- "bidi-js": "https://esm.sh/bidi-js@%5E1.0.2?target=es2022",
124
- "webgl-sdf-generator": "https://esm.sh/webgl-sdf-generator@1.1.1/es2022/webgl-sdf-generator.mjs",
125
- "lit": "https://cdn.jsdelivr.net/gh/lit/dist@3/core/lit-core.min.js",
126
- "lit/": "https://esm.run/lit@3/",
127
- "xrblocks": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/xrblocks.js",
128
- "xrblocks/addons/": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/addons/"
129
- }
130
- }
131
- ```
132
-
133
- XR Blocks stylesheet (always `<link>` in `<head>`):
134
-
135
- ```html
136
- <link type="text/css" rel="stylesheet" href="https://xrblocks.github.io/css/xr.css" />
137
- ```
138
-
139
- Lit bundle warning suppression (place before import map):
140
-
141
- ```html
142
- <script>window.litDisableBundleWarning = true;</script>
143
- ```
144
-
145
- ---
146
-
147
- ## Section 3 — XR Blocks Lifecycle Composition
148
-
149
- ### Module-scope MudraClient
150
-
151
- Instantiate `MudraClient` exactly once at module scope — not inside `init()`.
152
- The constructor does NOT open a WebSocket; connection is driven by `mudra.setMode("mudra")` only (Section 15). Module-scope instantiation ensures the `MudraClient` object is ready before the XR scene initializes.
153
-
154
- ```js
155
- // Module scope — outside any class (no URL arg; URL is baked into MudraClient per Section 4)
156
- const mudra = new MudraClient();
157
-
158
- class MainScript extends xb.Script {
159
- init() {
160
- // Register signal handlers and subscribe here, AFTER scene objects are created
161
- mudra.on('gesture', (data) => { /* handle tap / twist / etc. */ });
162
- mudra.on('pressure', (data) => { /* handle finger pressure */ });
163
- mudra.subscribe('gesture');
164
- mudra.subscribe('pressure');
165
- }
166
- update() { /* called every frame by xb */ }
167
- }
168
- ```
169
-
170
- ### xb.Script class structure
171
-
172
- Every generated app's top-level logic lives inside a class extending `xb.Script`.
173
- The class name may be anything (`MainScript`, `CubeApp`, etc.) but must extend `xb.Script`.
174
-
175
- ```js
176
- class MainScript extends xb.Script {
177
- /** Called once after XR Blocks and the WebGL context are ready. */
178
- init() {
179
- // Add lights, meshes, Mudra bindings here.
180
- this.add(new THREE.HemisphereLight(0xffffff, 0x666666, 3));
181
- // Place objects at xb.user.objectDistance in front of viewer:
182
- this.mesh.position.set(0, xb.user.height - 0.5, -xb.user.objectDistance);
183
- }
184
-
185
- /** Called every animation frame. Keep cheap — no allocations. */
186
- update() { }
187
-
188
- /** Fires when a pinch/controller-trigger starts in XR. */
189
- onSelectStart(event) { }
190
-
191
- /** Fires when a pinch/controller-trigger ends in XR. */
192
- onSelectEnd(event) { }
193
-
194
- /** Fires every frame while a pinch is held in XR. */
195
- onSelecting(event) { }
196
- }
197
- ```
198
-
199
- ### Entry point
200
-
201
- ```js
202
- document.addEventListener('DOMContentLoaded', function () {
203
- xb.add(new MainScript());
204
- xb.init(new xb.Options());
205
- });
206
- ```
207
-
208
- ### Key spatial constants
209
-
210
- - `xb.user.height` — floor-relative eye height (approx. 1.6 m)
211
- - `xb.user.objectDistance` — recommended arm-length distance for objects (approx. 0.8 m)
212
- - Y-up coordinate system; Z is toward the viewer (negative Z = in front of you)
213
-
214
- ---
215
-
216
- ## Section 4 — MudraClient (WebSocket + Mode Toggle glue)
217
-
218
- ### Policy
219
-
220
- - The constructor does **NOT** open a WebSocket. Connection is driven by `setMode("mudra")` only.
221
- - In Manual mode the sim panel and keyboard shortcuts inject synthetic signals through `_emit()` — the exact same handler path as real WebSocket messages. No auto-firing timers. No mock auto-activation.
222
- - In Mudra mode a real WebSocket opens to `ws://127.0.0.1:8766`. On successful open: replay the subscription record + send `get_status` + start the 2 s status-poll.
223
- - If the socket closes for a non-conflict reason: schedule reconnect with backoff `[1 s, 2 s, 5 s, 10 s]` (capped). The `_suppressReconnect` flag prevents retry on `client_already_connected`.
224
- - If `client_already_connected` error arrives: transition to `already-in-use`; set `_suppressReconnect = true`; do not retry.
225
- - The 6-state `ConnectionState` vocabulary and status-pill wiring live in **Section 15** (canonical). This section documents only the `MudraClient` class itself.
226
-
227
- ### Connection-status state machine
228
-
229
- See Section 15 for the full 6-state transition diagram. Vocabulary:
230
-
231
- ```text
232
- idle | connecting | connected | ws-only | reconnecting | already-in-use
233
- ```
234
-
235
- `simulated` and `disconnected-simulated` are **removed**. No auto-mock activation.
236
-
237
- ### Required MudraClient implementation
238
-
239
- Include this class verbatim in every generated app:
240
-
241
- ```js
242
- class MudraClient {
243
- static _WS_URL = 'ws://127.0.0.1:8766';
244
- static _BACKOFF = [1000, 2000, 5000, 10000]; // ms, last value is the cap
245
-
246
- constructor() {
247
- this._handlers = {};
248
- this._subscriptions = new Set(); // subscription record — replayed on reconnect
249
- this._ws = null;
250
- this._mode = 'manual';
251
- this._connState = 'idle';
252
- this._pollTimer = null;
253
- this._reconnectTimer = null;
254
- this._backoffIdx = 0;
255
- this._suppressReconnect = false;
256
- this._connToken = 0; // incremented on every openSocket() to cancel stale callbacks
257
- }
258
-
259
- /** Register a handler for a signal type or _status events. Call before subscribe(). */
260
- on(signal, fn) { this._handlers[signal] = fn; }
261
-
262
- /** Add signal to subscription record. If WS is open, send subscribe immediately. */
263
- subscribe(signal) {
264
- this._subscriptions.add(signal);
265
- if (this._ws && this._ws.readyState === WebSocket.OPEN) {
266
- this._ws.send(JSON.stringify({ command: 'subscribe', signal }));
267
- }
268
- }
269
-
270
- /** Remove signal from subscription record. If WS is open, send unsubscribe. */
271
- unsubscribe(signal) {
272
- this._subscriptions.delete(signal);
273
- if (this._ws && this._ws.readyState === WebSocket.OPEN) {
274
- this._ws.send(JSON.stringify({ command: 'unsubscribe', signal }));
275
- }
276
- }
277
-
278
- /** Send an arbitrary command. In Manual mode, trigger_gesture dispatches via _emit(). */
279
- send(cmd) {
280
- if (this._ws && this._ws.readyState === WebSocket.OPEN) {
281
- this._ws.send(JSON.stringify(cmd));
282
- } else if (cmd.command === 'trigger_gesture') {
283
- this._dispatchMockGesture(cmd.data?.type ?? 'tap');
284
- }
285
- }
286
-
287
- /** Switch mode. setMode("mudra") opens the socket; setMode("manual") closes it. */
288
- setMode(mode) {
289
- if (this._mode === mode) return;
290
- this._mode = mode;
291
- if (mode === 'manual') {
292
- this._closeSocket();
293
- this._setState('idle');
294
- } else {
295
- this._suppressReconnect = false;
296
- this._backoffIdx = 0;
297
- this._openSocket();
298
- }
299
- }
300
-
301
- get status() { return this._connState; }
302
-
303
- _setState(s) {
304
- this._connState = s;
305
- if (this._handlers['_status']) this._handlers['_status'](s);
306
- }
307
-
308
- _emit(payload) {
309
- if (this._handlers[payload.type]) this._handlers[payload.type](payload.data);
310
- }
311
-
312
- /** Called by sim-panel gesture buttons and keyboard Space shortcut. */
313
- _dispatchMockGesture(type) {
314
- this._emit({ type: 'gesture', data: { type, timestamp: Date.now() } });
315
- }
316
-
317
- _openSocket() {
318
- const token = ++this._connToken;
319
- this._setState('connecting');
320
- try {
321
- const ws = new WebSocket(MudraClient._WS_URL);
322
- this._ws = ws;
323
-
324
- ws.onopen = () => {
325
- if (this._connToken !== token) return;
326
- this._backoffIdx = 0;
327
- this._replaySubscriptions();
328
- ws.send(JSON.stringify({ command: 'get_status' }));
329
- this._startPoll();
330
- };
331
-
332
- ws.onmessage = (e) => {
333
- if (this._connToken !== token) return;
334
- let msg;
335
- try { msg = JSON.parse(e.data); } catch { return; }
336
-
337
- if (msg.type === 'error' && msg.data?.error === 'client_already_connected') {
338
- this._suppressReconnect = true;
339
- this._stopPoll();
340
- this._setState('already-in-use');
341
- return;
342
- }
343
-
344
- if (msg.type === 'status') {
345
- const d = msg.data?.device;
346
- const bandConnected = !!(d?.firmware && d?.serial_number);
347
- if (bandConnected) {
348
- if (this._connState !== 'connected') {
349
- this._setState('connected');
350
- this._replaySubscriptions(); // FR-012: re-subscribe on band reconnect
351
- }
352
- // expose hand via a synthetic 'hand' event for the UI chip
353
- if (this._handlers['_hand']) {
354
- const raw = d.hand ?? '';
355
- const hand = /^(LEFT|RIGHT)$/i.test(raw) ? raw.toUpperCase() : 'None';
356
- this._handlers['_hand'](hand);
357
- }
358
- } else {
359
- if (this._connState !== 'ws-only') this._setState('ws-only');
360
- if (this._handlers['_hand']) this._handlers['_hand']('None');
361
- }
362
- return;
363
- }
364
-
365
- if (this._handlers[msg.type]) this._handlers[msg.type](msg.data);
366
- };
367
-
368
- ws.onerror = () => { /* onclose fires after onerror; handle there */ };
369
-
370
- ws.onclose = () => {
371
- if (this._connToken !== token) return;
372
- this._stopPoll();
373
- if (this._suppressReconnect || this._mode !== 'mudra') return;
374
- this._setState('reconnecting');
375
- const delay = MudraClient._BACKOFF[Math.min(this._backoffIdx, MudraClient._BACKOFF.length - 1)];
376
- this._backoffIdx = Math.min(this._backoffIdx + 1, MudraClient._BACKOFF.length - 1);
377
- this._reconnectTimer = setTimeout(() => {
378
- if (this._mode === 'mudra' && !this._suppressReconnect) this._openSocket();
379
- }, delay);
380
- };
381
- } catch (_) {
382
- this._setState('reconnecting');
383
- }
384
- }
385
-
386
- _closeSocket() {
387
- this._stopPoll();
388
- clearTimeout(this._reconnectTimer);
389
- this._reconnectTimer = null;
390
- this._connToken++; // invalidate all callbacks from the old socket
391
- if (this._ws) { try { this._ws.close(); } catch (_) {} this._ws = null; }
392
- }
393
-
394
- _startPoll() {
395
- this._stopPoll();
396
- this._pollTimer = setInterval(() => {
397
- if (this._ws && this._ws.readyState === WebSocket.OPEN) {
398
- this._ws.send(JSON.stringify({ command: 'get_status' }));
399
- }
400
- }, 2000);
401
- }
402
-
403
- _stopPoll() {
404
- clearInterval(this._pollTimer);
405
- this._pollTimer = null;
406
- }
407
-
408
- /** Replay the subscription record. Called on open and on band reconnect. */
409
- _replaySubscriptions() {
410
- if (!this._ws || this._ws.readyState !== WebSocket.OPEN) return;
411
- this._subscriptions.forEach(sig =>
412
- this._ws.send(JSON.stringify({ command: 'subscribe', signal: sig }))
413
- );
414
- }
415
-
416
- destroy() {
417
- this._closeSocket();
418
- }
419
- }
420
- ```
421
-
422
- ### Section 4 note on "no fallback"
423
-
424
- **"No fallback"** means: no automatic transition to a "Simulated" or "Disconnected — simulated" state when the WebSocket is unavailable. If the socket fails, the pill shows `Reconnecting…` and retries — it never auto-activates synthetic data injection.
425
-
426
- Manual mode signal injection (sim panel + keyboard shortcuts calling `_emit()`) is **preserved** — this is Principle IV compliance, not a fallback. Sim-panel signals use the same `_emit()` path as real WebSocket messages, so the app does not branch on whether it is simulated or not.
427
-
428
- ---
429
-
430
- ## Section 5 — Simulator Panel
431
-
432
- ### Purpose
433
-
434
- The simulator panel lets a user exercise every subscribed Mudra signal without
435
- a band or XR headset. It is a **2D DOM overlay** (not spatial XR UI) and must
436
- always be visible in flat-screen mode. It disappears automatically when the
437
- browser transitions to an immersive WebXR session (native WebXR DOM suppression).
438
-
439
- ### DOM structure
440
-
441
- ```html
442
- <div id="mudra-sim" style="
443
- position: fixed; bottom: 0; left: 0; right: 0;
444
- background: rgba(0,0,0,0.75); backdrop-filter: blur(4px);
445
- padding: 8px 12px; display: flex; flex-wrap: wrap; gap: 8px;
446
- z-index: 9999; font-family: system-ui, sans-serif;">
447
- <!-- One button group per subscribed signal -->
448
- </div>
449
- ```
450
-
451
- ### Button groups per signal (render ONLY the sub-actions the app actually uses)
452
-
453
- The catalog below is the **maximal** set per signal. The simulator panel
454
- MUST render **only** the sub-actions the generated app handles — never
455
- extras. Examples:
456
-
457
- - App maps `gesture.tap` only → render `Tap`. Omit `2Tap`, `Twist`, `2Twist`.
458
- - App maps `nav_direction` to Up/Down only → render `↑`, `↓`. Omit
459
- `←`, `→`, `Roll L`, `Roll R`.
460
- - App uses `imu_acc` for X-axis tilt only → render `Tilt X+`, `Tilt X−`.
461
- Omit `Tilt Y+`, `Tilt Y−`.
462
-
463
- When in doubt, walk every `mudra.on('<signal>', …)` handler and emit a
464
- button only for sub-actions referenced inside it. Unused buttons are a
465
- checklist failure (see Section 10, item 6).
466
-
467
- | Signal | Maximal buttons (subset based on app handlers) |
468
- |--------|---------|
469
- | `gesture` | `Tap`, `2Tap`, `Twist`, `2Twist` |
470
- | `nav_direction` | `↑`, `↓`, `←`, `→`, `Roll L`, `Roll R` |
471
- | `navigation` | `↑`, `↓`, `←`, `→` (each emits one delta event of ±3 — gentle/low-sensitivity default; raise per-app only if the prompt explicitly asks for fast/snappy movement) |
472
- | `pressure` | Slider `0–100` (label shows current value) |
473
- | `button` | `Press`, `Release` |
474
- | `imu_acc` | `Tilt X+`, `Tilt X−`, `Tilt Y+`, `Tilt Y−` (5-frame burst at ±2 m/s²) |
475
- | `imu_gyro` | `Rot X+`, `Rot X−`, `Rot Y+`, `Rot Y−` (5-frame burst at ±10 deg/s) |
476
- | `snc` | `Spike` (burst of elevated samples on all 3 channels) |
477
-
478
- ### Button firing rules
479
-
480
- - Every button fires via the **same code path** as a real Mudra signal.
481
- Call the same handler that `mudra.on(signal, handler)` would invoke.
482
- - For `gesture` buttons: also call `mudra.send({ command: 'trigger_gesture', data: { type } })`
483
- so the round-trip works when a real band is connected.
484
- - Never duplicate logic between the simulator path and the real-signal path.
485
-
486
- ```js
487
- // Example: gesture sim button wires to the same handler as real signals
488
- function simGesture(type) {
489
- mudra.send({ command: 'trigger_gesture', data: { type } });
490
- handleGesture({ type, timestamp: Date.now() });
491
- }
492
-
493
- // Example: pressure slider
494
- pressureSlider.addEventListener('input', () => {
495
- const norm = pressureSlider.value / 100;
496
- handlePressure({ value: +pressureSlider.value, normalized: norm, timestamp: Date.now() });
497
- });
498
- ```
499
-
500
- ### Visibility
501
-
502
- - `#mudra-sim` is always visible in flat-screen mode.
503
- - Do NOT conditionally hide it when the band is connected (both sources are valid per FR-014).
504
- - The 2D DOM disappears automatically in immersive WebXR — no JS required.
505
-
506
- ---
507
-
508
- ## Section 6 — Keyboard Shortcuts + XR Blocks Precedence
509
-
510
- ### Canonical keyboard map
511
-
512
- | Key | Signal / Action |
513
- |-----|----------------|
514
- | `Space` | `gesture` → tap |
515
- | `Shift` | `button` → press (keydown) / release (keyup) |
516
- | `[` | `pressure` decrease (−10, min 0) |
517
- | `]` | `pressure` increase (+10, max 100) |
518
- | `i` | `nav_direction` → Up OR `navigation` delta_y +3 |
519
- | `k` | `nav_direction` → Down OR `navigation` delta_y −3 |
520
- | `j` | `nav_direction` → Left OR `navigation` delta_x −3 |
521
- | `l` | `nav_direction` → Right OR `navigation` delta_x +3 |
522
-
523
- **Navigation sensitivity default**: keyboard `I`/`J`/`K`/`L` and sim panel
524
- buttons emit deltas of **±3** per event (gentle / low-sensitivity
525
- baseline). This keeps cursor/pan motion calm and predictable. Only raise
526
- the magnitude when the prompt explicitly calls for fast/snappy movement
527
- (racing, arcade-twitch, etc.).
528
- | `u` | `imu_acc` tilt X+ burst |
529
- | `o` | `imu_acc` tilt X− burst |
530
- | `m` | `imu_gyro` rot Y+ burst |
531
- | `n` | `imu_gyro` rot Y− burst |
532
-
533
- For `imu` bursts: fire 5 synthetic frames at ±[2, 0, 9.81] m/s² (acc) or ±[10, 0, 0.5] deg/s (gyro).
534
-
535
- ### Reserved keys & mouse — owned by XR Blocks desktop simulator
536
-
537
- The XR Blocks desktop simulator (active whenever the app runs in a flat
538
- browser without a WebXR session) owns the keys and mouse gestures that
539
- let the user navigate the 3D scene. Mudra MUST NOT claim or
540
- `stopPropagation()` any of these — they are how the user orbits, walks,
541
- and zooms while previewing the app:
542
-
543
- | Input | XR Blocks role |
544
- |-------|----------------|
545
- | `W` `A` `S` `D` | Walk camera forward / left / back / right |
546
- | `ArrowUp` `ArrowDown` `ArrowLeft` `ArrowRight` | Camera nav (alt to WASD) |
547
- | `Q` `E` | Camera roll / vertical |
548
- | `R` | Reset camera pose |
549
- | Right-click drag | Orbit / look around |
550
- | Mouse wheel | Zoom in / out |
551
-
552
- **Why Mudra uses `I`/`J`/`K`/`L` and `U`/`O`/`M`/`N`**: these keys are
553
- explicitly off the XR Blocks reserved set, so the desktop simulator
554
- keeps full camera control while the band-driven keyboard shortcuts
555
- remain available for testing without a band. Never reassign a Mudra
556
- shortcut onto a reserved key, even when the app does not subscribe to a
557
- navigation signal.
558
-
559
- ### Attachment rule (critical)
560
-
561
- Mudra keyboard handlers MUST attach with `capture: true` and call
562
- `event.stopPropagation()` on every key the app subscribes to. This prevents
563
- XR Blocks' desktop-simulator bubble-phase listeners from double-firing.
564
-
565
- ```js
566
- window.addEventListener('keydown', (e) => {
567
- switch (e.code) {
568
- case 'Space':
569
- e.stopPropagation();
570
- simGesture('tap');
571
- break;
572
- case 'BracketLeft':
573
- e.stopPropagation();
574
- adjustPressure(-10);
575
- break;
576
- case 'BracketRight':
577
- e.stopPropagation();
578
- adjustPressure(+10);
579
- break;
580
- case 'KeyI':
581
- e.stopPropagation();
582
- handleNavDirection({ direction: 'Up', timestamp: Date.now() });
583
- break;
584
- // ... etc for subscribed keys only — using I/J/K/L for nav and U/O/M/N for IMU
585
- }
586
- }, { capture: true });
587
- ```
588
-
589
- Only intercept keys for signals the app actually subscribes to. Do NOT
590
- `stopPropagation` on keys that no Mudra signal handles — XR Blocks needs
591
- those, especially the reserved set above (WASD, arrows, Q/E/R, mouse).
592
-
593
- ---
594
-
595
- ## Section 7 — Connection-Status Indicator
596
-
597
- > **Superseded by Section 15.** The status pill is now part of the Mode Toggle chrome and follows the 6-state machine defined there. Read Section 15 for the canonical DOM sketch (`#mudra-status` + `#mudra-hand`), state labels, colour hints, and wiring.
598
- >
599
- > The 4-state vocabulary (`connecting` / `connected` / `simulated` / `disconnected-simulated`) is **removed**. New vocabulary: `idle` (Manual) / `Connecting…` / `Connected` / `WebSocket Only` (orange) / `Reconnecting…` / `Companion already in use`.
600
- >
601
- > The `#mudra-status` `<div>` still lives at `position: fixed; top: 8px; right: 12px` and disappears automatically in immersive WebXR. See Section 15 for full wiring.
602
-
603
- ---
604
-
605
- ## Section 8 — Signal Grouping Rules (Non-Negotiable)
606
-
607
- ### Signal group classification
608
-
609
- | Group | Signals | Combine freely with |
610
- |-------|---------|---------------------|
611
- | **Pointer** | `navigation`, `button` | `gesture` OR `button` (not `pressure` if `gesture` is used) |
612
- | **Direction** | `nav_direction` | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
613
- | **IMU+Biometric** | `imu_acc`, `imu_gyro`, `snc` | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
614
- | *(none)* | — | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
615
-
616
- ### Bundling rule — IMU+Biometric (CRITICAL)
617
-
618
- `imu_acc`, `imu_gyro`, and `snc` are an **inseparable bundle**. If the user's prompt
619
- implies any one of them, subscribe to **all three**. Never subscribe to only one or
620
- two of them.
621
-
622
- ```js
623
- // CORRECT — all three always together
624
- mudra.subscribe('imu_acc');
625
- mudra.subscribe('imu_gyro');
626
- mudra.subscribe('snc');
627
-
628
- // WRONG — partial subscriptions
629
- mudra.subscribe('snc'); // missing imu_acc and imu_gyro
630
- mudra.subscribe('imu_acc'); // missing imu_gyro and snc
631
- ```
632
-
633
- ### XOR rules (all non-negotiable)
634
-
635
- 1. **Gesture ⊕ Pressure** — an app may use `gesture` OR `pressure`, never both.
636
- 2. **Navigation ⊕ Nav_direction** — an app may use `navigation` OR `nav_direction`, never both.
637
- 3. **Pointer/Direction ⊕ IMU+Biometric** — `navigation` and `nav_direction` cannot be combined with the IMU+Biometric bundle (`imu_acc`/`imu_gyro`/`snc`).
638
-
639
- ### Illegal combinations
640
-
641
- ```
642
- // REJECT these signal sets
643
- gesture + pressure
644
- navigation + nav_direction
645
- navigation + imu_acc
646
- navigation + imu_gyro
647
- navigation + snc
648
- nav_direction + imu_acc
649
- nav_direction + imu_gyro
650
- nav_direction + snc
651
- button + nav_direction // button belongs to Pointer mode only
652
- ```
653
-
654
- ### Valid signal sets (examples)
655
-
656
- ```
657
- gesture + button
658
- pressure + button
659
- navigation + button
660
- navigation + button + gesture
661
- nav_direction
662
- nav_direction + pressure + button
663
- imu_acc + imu_gyro + snc
664
- imu_acc + imu_gyro + snc + gesture
665
- imu_acc + imu_gyro + snc + button
666
- imu_acc + imu_gyro + snc + pressure + button
667
- ```
668
-
669
- ### Inference priority for ties
670
-
671
- When the user prompt maps to multiple modes equally, ask one disambiguation
672
- question — do not silently pick one.
673
-
674
- Default to `direction` mode (`nav_direction`) when there is no navigation
675
- language in the prompt at all and a motion mode is required by the template.
676
-
677
- ---
678
-
679
- ## Section 9 — AI API Key Handling (onboarding-gated, mandatory for AI apps)
680
-
681
- ### Lifecycle: `sessionStorage` keyed by `mudra.gemini.apiKey`
682
-
683
- For any generated app that calls a Gemini / LLM endpoint, the API key MUST
684
- be entered through the **onboarding modal** before the user can use the
685
- app. The key is stored in `sessionStorage` under the literal key
686
- `mudra.gemini.apiKey` — it persists across reloads in the same tab and
687
- clears when the tab closes. **Do NOT use `localStorage`. Do NOT use a
688
- `prompt()` popup. Do NOT prompt on first AI call.**
689
-
690
- ```js
691
- // Read at app start
692
- const apiKey = sessionStorage.getItem('mudra.gemini.apiKey');
693
-
694
- // Write only from the onboarding "AI Setup" step
695
- sessionStorage.setItem('mudra.gemini.apiKey', enteredKey);
696
- ```
697
-
698
- ### Required onboarding "AI Setup" fragment
699
-
700
- For AI apps only, the onboarding modal MUST include the following
701
- fragment inside `<section class="mudra-onb__body">`, placed AFTER the
702
- actions table and BEFORE `</section>`:
703
-
704
- ```html
705
- <div class="mudra-onb__ai" data-uses-ai>
706
- <h3 class="mudra-onb__ai-title">AI Setup</h3>
707
- <p class="mudra-onb__ai-lede">
708
- This app uses Google Gemini. Paste your API key to continue —
709
- it's stored only in this browser tab (<code>sessionStorage</code>)
710
- and is never sent anywhere except Google's API.
711
- <a href="https://aistudio.google.com/" target="_blank" rel="noopener">Get a key →</a>
712
- </p>
713
- <input
714
- id="mudra-onb-ai-key"
715
- class="mudra-onb__ai-input"
716
- type="password"
717
- autocomplete="off"
718
- spellcheck="false"
719
- placeholder="Paste Gemini API key (starts with AIza…)"
720
- aria-label="Gemini API key"
721
- />
722
- <p class="mudra-onb__ai-hint" data-role="hint"></p>
723
- </div>
724
- ```
725
-
726
- And the matching CSS (added to the existing `<style>`):
727
-
728
- ```css
729
- .mudra-onb__ai { margin-top: 14px; padding-top: 12px; border-top: 1px solid #eee; }
730
- .mudra-onb__ai-title { margin: 0 0 6px; font-size: 0.95rem; font-weight: 700; color: #111; }
731
- .mudra-onb__ai-lede { margin: 0 0 10px; color: #555; font-size: 0.85rem; }
732
- .mudra-onb__ai-lede a { color: #0d9488; }
733
- .mudra-onb__ai-input {
734
- width: 100%; box-sizing: border-box;
735
- padding: 9px 12px; border: 1px solid #d0d0d0; border-radius: 8px;
736
- font: 13px/1.4 ui-monospace, SFMono-Regular, Menlo, monospace;
737
- background: #f8f9fb; color: #111;
738
- }
739
- .mudra-onb__ai-input:focus { outline: 2px solid #14b8a6; border-color: #14b8a6; background: #fff; }
740
- .mudra-onb__ai-hint { margin: 6px 2px 0; font-size: 0.75rem; color: #b91c1c; min-height: 1em; }
741
- .mudra-onb__continue:disabled { opacity: 0.45; cursor: not-allowed; }
742
- ```
743
-
744
- ### Behaviour rules (AI apps only)
745
-
746
- 1. **`Got it` button starts disabled.** The IIFE that wires the modal
747
- reads `dialog.querySelector('.mudra-onb__ai')` — if it exists, the
748
- `.mudra-onb__continue` button is disabled until
749
- `.mudra-onb__ai-input` is non-empty AND matches `/^AIza[\w-]{30,}$/`
750
- (Google API-key prefix sanity check).
751
- 2. **On click of `Got it`**, write the trimmed value to
752
- `sessionStorage.setItem('mudra.gemini.apiKey', value)`, then close
753
- the modal.
754
- 3. **At every page load**, the IIFE checks `sessionStorage` first:
755
- - If `mudra.gemini.apiKey` is present and matches the prefix regex,
756
- the AI-Setup fragment is hidden (the user already provided a key
757
- this session) and `Got it` is enabled immediately.
758
- - If absent or malformed, the AI-Setup fragment is shown and the
759
- modal CANNOT be dismissed by `Escape`, the `×` close button, or
760
- the reopen `?` button without entering a valid key. `dialog.close()`
761
- paths called from those handlers are no-ops while the key is
762
- missing.
763
- 4. **The `?` reopen button** for AI apps re-runs the gating logic. If
764
- the user clears `sessionStorage` mid-session and reopens, the
765
- AI-Setup fragment renders again.
766
- 5. **Reading the key in app code:** the `xb.Script` subclass reads
767
- `sessionStorage.getItem('mudra.gemini.apiKey')` in `init()`. If the
768
- key is `null`, the AI portion of the app stays inert (no calls to
769
- Gemini) and the visible chat panel (Section 18) renders a `Set up
770
- AI in the welcome panel` placeholder.
771
- 6. **Never bake a key into the HTML source.** Pre-write regex scan
772
- (`/AIza[A-Za-z0-9_-]{30,}|sk-[A-Za-z0-9_-]{32,}/` excluding the
773
- literal placeholder string `AIza…`) must return zero matches.
774
- 7. **Never auto-read from URL params, `localStorage`, or `prompt()`.**
775
- 8. **Non-AI apps** ignore this section entirely. The AI-Setup fragment
776
- is omitted; the modal works as defined in Section 17 unchanged.
777
-
778
- ### Canonical Gemini model — `gemini-2.5-flash` only
779
-
780
- For any generated app that calls Gemini via the **REST `generateContent`
781
- endpoint**, the model ID MUST be exactly `gemini-2.5-flash`. No other
782
- model IDs are permitted for REST text/chat/vision generation. This is a
783
- hard pin — preview aliases get retired by Google and the app then 404s.
784
-
785
- ```js
786
- // CORRECT — the only permitted REST model for text/chat/vision
787
- const url = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=${encodeURIComponent(apiKey)}`;
788
-
789
- // FORBIDDEN — preview aliases, dated aliases, retired families
790
- gemini-1.5-flash, gemini-1.5-flash-latest, gemini-1.5-pro
791
- gemini-2.5-flash-preview-09-2025, gemini-2.5-flash-preview-04-2025
792
- gemini-flash-latest, gemini-pro, gemini-2.5-flash-002 (any -NNN suffix)
793
- ```
794
-
795
- Pre-write regex scan — every generated HTML file MUST satisfy:
796
-
797
- - `/generativelanguage\.googleapis\.com\/v1beta\/models\/([a-z0-9-]+):generateContent/`
798
- → captured model ID MUST equal `gemini-2.5-flash`. Any other capture
799
- fails the pre-write checklist.
800
-
801
- Out-of-scope use cases (allowed exceptions, REST-only rule does NOT apply):
802
-
803
- - **Live API** (WebSocket / xrblocks `xb.core.ai.startLiveSession()`):
804
- the Live endpoint uses its own model set (`gemini-2.0-flash-live-001`,
805
- `gemini-2.5-flash-native-audio-preview-12-2025`). Apps that genuinely
806
- need streaming audio/video may use those, but every text-chat app
807
- must use the REST pin above.
808
- - **Image generation** via `:generateContent`: `gemini-2.5-flash-image`
809
- is permitted only when the app's purpose is image output. Default to
810
- the text pin otherwise.
811
-
812
- If an app needs a different model, raise it to the user before writing —
813
- do not silently swap in a preview alias.
814
-
815
- ---
816
-
817
- ## Section 10 — Pre-Write Checklist + Collision Handling
818
-
819
- Before calling `Write` to emit a generated app, verify all items:
820
-
821
- | # | Check | Pass condition |
822
- |---|-------|----------------|
823
- | 1 | Single file | Exactly one `<html>` document; all CSS in `<style>`; all JS in `<script>` or `<script type="module">` |
824
- | 2 | Import map | One `<script type="importmap">` block; contents match canonical pins (Section 2) exactly; no unused entries |
825
- | 3 | xb.Script entry | Top-level logic inside `class <Name> extends xb.Script`; `xb.add(new <Name>())` + `xb.init(new xb.Options())` on `DOMContentLoaded` |
826
- | 4 | MudraClient | One `MudraClient` instance at module scope; constructor does NOT auto-connect; `setMode()` drives connect/disconnect; no `setTimeout(_, 1500)` timer; no `_startMock()` auto-firing; no `'simulated'` / `'disconnected-simulated'` strings |
827
- | 5 | Subscribe commands | Every used signal has exactly one `mudra.subscribe('<signal>')` call; none outside the signal set |
828
- | 6 | Simulator panel | `<div id="mudra-sim">` present; ONLY buttons for sub-actions actually handled by the app (no extras like Roll L/R or Twist if unused); buttons fire via handler, not inline `onclick` |
829
- | 7 | Keyboard bindings | `window.addEventListener('keydown', …, { capture: true })` present; `event.stopPropagation()` on every Mudra-claimed key |
830
- | 8 | Status indicator | `<div id="mudra-status">` + `<span id="mudra-hand">` present; text states are `Manual` / `Connecting…` / `Connected` / `WebSocket Only` / `Reconnecting…` / `Companion already in use` (Section 15); no `simulated` / `disconnected-simulated` strings; `#mudra-hand` shows `LEFT`/`RIGHT` when connected, `None` otherwise |
831
- | 9 | AI key gating | If `usesAI`: the AI-Setup fragment (Section 9) is present inside `.mudra-onb__body`; key is read from `sessionStorage.getItem('mudra.gemini.apiKey')` only; ZERO `prompt(` calls for the key; ZERO `localStorage` references; ZERO baked keys (regex scan) |
832
- | 9a | Visible AI chat I/O | If `usesAI`: the scene renders BOTH the latest user input AND the AI response as visible text (xb.ScrollingTroikaTextView, troika `Text`, or xb.SpatialPanel rows). The visible "Purpose" line states what the app does in one sentence. TTS may exist but is never the only output (Section 18) |
833
- | 10 | Background lockdown | ZERO `applyBackground_*` methods in the class; ZERO calls to a background helper from `init()`; no `options.simulator.scenePath` line anywhere. Generated apps use the XR Blocks default room only (Section 14) |
834
- | 11 | Mode toggle | `<div id="mode-toggle">` with **Manual** + **Mudra** buttons; Manual is the default on load; toggle remains clickable when disconnected; flipping atomically opens/closes the socket per Section 15 |
835
- | 12 | Band-state polling | In Mudra mode the app sends `{command:"get_status"}` on `ws.onopen` and every 2000 ms thereafter; pill flips to `Connected` ONLY when `data.device.firmware && data.device.serial_number` (both truthy); `WebSocket Only` (orange) when socket open but band not connected |
836
- | 13 | No disconnect overlay | No banner / toast / modal / inline alert ever rendered for disconnect — pill is the only indicator |
837
- | 14 | Footer | Exactly one `<div id="mudra-badge">` containing the literal text `Created by Mudra` (no variants) |
838
- | 15 | Mock is passive | No `setTimeout(_, 1500)` timer; no `_startMock()` auto-firing; Manual-mode signals come only from sim-panel clicks and keyboard shortcuts calling `_emit()` directly |
839
- | 16 | Gemini model pin | If the app calls `generativelanguage.googleapis.com/v1beta/models/<id>:generateContent`, the captured `<id>` MUST equal `gemini-2.5-flash`. No preview / dated / latest aliases. Live-API and image-gen exceptions per Section 9 |
840
- | 17 | Subscription record | `MudraClient._subscriptions` is a `Set<string>` that tracks subscribed signals; `_replaySubscriptions()` fires on `ws.onopen` AND on any `ws-only` → `connected` transition; signals in the record are never resurrected if `unsubscribe()` was called explicitly |
841
- | 18 | client_already_connected | The `{type:"error",data:{error:"client_already_connected"}}` error frame is handled: pill transitions to `Companion already in use`; `_suppressReconnect = true`; the backoff timer does NOT fire on the subsequent `onclose` |
842
- | 19 | Reconnect backoff | WS `onclose` (non-conflict) schedules reconnect with `[1000, 2000, 5000, 10000]` ms schedule; `_backoffIdx` resets to 0 on every successful `onopen` |
843
- | 20 | No forbidden strings | Generated source contains zero occurrences of: `confidence` as a gesture field, `enable`/`disable`/`get_docs` as commands, `'simulated'`/`'disconnected-simulated'` as status strings, `8765` as a port |
844
-
845
- ### Retry policy
846
-
847
- If any check fails:
848
- 1. Regenerate once and re-run the full checklist.
849
- 2. If the second attempt also fails, surface the specific failing check(s)
850
- to the user and do **not** write the file.
851
-
852
- ### File collision / auto-suffix rule
853
-
854
- - Target filename: `preview/<name>.html`
855
- - If that file already exists, try `preview/<name>-2.html`, then `-3.html`, etc.
856
- - Never overwrite an existing file.
857
- - Always report the final written path to the user.
858
-
859
- ---
860
-
861
- ## Section 11 — Signal → XR Binding Patterns
862
-
863
- Use these as code snippets when adapting a template.
864
- Copy and adapt — do not invent new patterns from scratch.
865
-
866
- ### gesture.tap → onSelectStart forwarding
867
-
868
- ```js
869
- // In init():
870
- mudra.on('gesture', (data) => {
871
- if (data.type === 'tap') this.onSelectStart({ source: 'mudra' });
872
- if (data.type === 'double_tap') this.onSelectEnd({ source: 'mudra' });
873
- });
874
- mudra.subscribe('gesture');
875
- ```
876
-
877
- ### pressure → scale / color mapping
878
-
879
- ```js
880
- // In init():
881
- mudra.on('pressure', (data) => {
882
- const s = 0.5 + data.normalized * 1.5; // scale 0.5 … 2.0
883
- this.mesh.scale.setScalar(s);
884
- const hue = data.normalized * 0.8; // hue 0 (red) … 0.8 (blue)
885
- this.mesh.material.color.setHSL(hue, 0.9, 0.5);
886
- });
887
- mudra.subscribe('pressure');
888
- ```
889
-
890
- ### nav_direction → spatial menu step
891
-
892
- ```js
893
- // In init():
894
- mudra.on('nav_direction', (data) => {
895
- switch (data.direction) {
896
- case 'Up': this.menuIndex = Math.max(0, this.menuIndex - 1); break;
897
- case 'Down': this.menuIndex = Math.min(this.items.length - 1, this.menuIndex + 1); break;
898
- case 'Right': this.selectItem(this.menuIndex); break;
899
- case 'Left': this.goBack(); break;
900
- }
901
- this.updateMenuHighlight();
902
- });
903
- mudra.subscribe('nav_direction');
904
- ```
905
-
906
- ### imu_acc + imu_gyro + snc bundle → subscribe all three together
907
-
908
- `imu_acc`, `imu_gyro`, and `snc` are always subscribed together. Register handlers
909
- for each signal you actually use in the app, but always send all three subscribe commands.
910
-
911
- ```js
912
- // In init():
913
- mudra.on('imu_acc', (data) => {
914
- const [ax, ay] = data.values;
915
- this.mesh.rotation.x = THREE.MathUtils.clamp(ax * 0.1, -Math.PI / 4, Math.PI / 4);
916
- this.mesh.rotation.z = THREE.MathUtils.clamp(ay * 0.1, -Math.PI / 4, Math.PI / 4);
917
- });
918
- mudra.on('imu_gyro', (data) => {
919
- const [gx] = data.values;
920
- this.mesh.rotation.y += gx * 0.001;
921
- });
922
- mudra.on('snc', (data) => {
923
- const ch1 = data.values[0];
924
- const latest = ch1[ch1.length - 1]; // most recent sample
925
- const norm = Math.min(1, Math.abs(latest) / 500); // normalize
926
- this.overlay.material.opacity = norm * 0.7;
927
- });
928
- // ALWAYS subscribe all three — they form an inseparable bundle
929
- mudra.subscribe('imu_acc');
930
- mudra.subscribe('imu_gyro');
931
- mudra.subscribe('snc');
932
- ```
933
-
934
- ### navigation → continuous cursor / pan
935
-
936
- **Default sensitivity multiplier: `0.002`** (gentle / low-sensitivity).
937
- Use this baseline for all navigation/cursor/pan bindings. Only raise it
938
- when the prompt explicitly asks for fast/snappy movement.
939
-
940
- ```js
941
- // In init():
942
- const NAV_SENSITIVITY = 0.002; // gentle default — slow, predictable
943
- mudra.on('navigation', (data) => {
944
- this.cursorX = THREE.MathUtils.clamp(this.cursorX + data.delta_x * NAV_SENSITIVITY, -1, 1);
945
- this.cursorY = THREE.MathUtils.clamp(this.cursorY - data.delta_y * NAV_SENSITIVITY, -1, 1);
946
- this.cursor.position.set(this.cursorX, this.cursorY, -xb.user.objectDistance);
947
- });
948
- mudra.subscribe('navigation');
949
- ```
950
-
951
- ---
952
-
953
- ## Section 12 — Template Selection Table
954
-
955
- One row per asset in `assets/`. The skill's keyword-based selector scores each row
956
- against the user's prompt (signal names, motion keywords, XR feature words).
957
-
958
- | id | path | keywords | motionModesSupported | xrFeatures |
959
- |----|------|----------|---------------------|------------|
960
- | `0_basic` | `assets/templates/0_basic.html` | `["basic","cylinder","pinch","color","simple","starter"]` | `["none","pointer"]` | `["input"]` |
961
- | `1_ui` | `assets/templates/1_ui.html` | `["ui","spatial","panel","text","sdf","font","button","draggable","troika"]` | `["pointer"]` | `["input","ui"]` |
962
- | `2_hands` | `assets/templates/2_hands.html` | `["hands","hand","pinch","gesture","joints","finger","hand-tracking"]` | `["pointer"]` | `["hands","input"]` |
963
- | `3_depth` | `assets/templates/3_depth.html` | `["depth","mesh","depth-sensing","plane","environment","occlusion"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
964
- | `4_stereo` | `assets/templates/4_stereo.html` | `["stereo","video","passthrough","camera","background","environment","feed"]` | `["none"]` | `["camera","passthrough"]` |
965
- | `5_camera` | `assets/templates/5_camera.html` | `["camera","video","passthrough","texture","scene","background"]` | `["none","pointer"]` | `["camera","input"]` |
966
- | `6_ai` | `assets/templates/6_ai.html` | `["ai","gemini","query","vision","photo","capture","llm","multimodal"]` | `["pointer"]` | `["input","camera"]` |
967
- | `7_ai_live` | `assets/templates/7_ai_live.html` | `["ai","gemini","live","speech","transcription","voice","microphone","audio","real-time"]` | `["pointer"]` | `["input"]` |
968
- | `8_objects` | `assets/templates/8_objects.html` | `["objects","detection","model","3d","place","anchor","ar","environment"]` | `["pointer"]` | `["mesh-detection","input"]` |
969
- | `9_xr-toggle` | `assets/templates/9_xr-toggle.html` | `["toggle","xr","enter","exit","session","button","transition"]` | `["none","pointer"]` | `["input"]` |
970
- | `heuristic_hand_gestures` | `assets/templates/heuristic_hand_gestures.html` | `["gesture","heuristic","hand","recognition","custom","pattern","hand-tracking"]` | `["none"]` | `["hands"]` |
971
- | `meshes` | `assets/templates/meshes.html` | `["mesh","environment","scan","plane","floor","wall","surface"]` | `["none"]` | `["mesh-detection"]` |
972
- | `planes` | `assets/templates/planes.html` | `["plane","floor","wall","surface","anchor","environment","horizontal","vertical"]` | `["none"]` | `["plane-detection"]` |
973
- | `uikit` | `assets/templates/uikit.html` | `["ui","kit","component","widget","button","icon","material","text","panel","menu"]` | `["pointer"]` | `["input","ui"]` |
974
- | `depthmap` | `assets/samples/depthmap.html` | `["depth","map","visualization","color","gradient","environment","scan"]` | `["none"]` | `["depth-sensing"]` |
975
- | `depthmesh` | `assets/samples/depthmesh.html` | `["depth","mesh","wireframe","environment","scan","geometry"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
976
- | `game_rps` | `assets/samples/game_rps.html` | `["game","rps","rock","paper","scissors","gesture","compete","fun","hand"]` | `["none"]` | `["hands"]` |
977
- | `gestures_custom` | `assets/samples/gestures_custom.html` | `["gesture","custom","recognize","train","pose","hand-tracking","hands"]` | `["none"]` | `["hands"]` |
978
- | `gestures_heuristic` | `assets/samples/gestures_heuristic.html` | `["gesture","heuristic","recognize","hand","pose","pinch","open","fist"]` | `["none"]` | `["hands"]` |
979
- | `lighting` | `assets/samples/lighting.html` | `["lighting","light","shadow","scene","animals","3d","models","environment"]` | `["pointer"]` | `["input"]` |
980
- | `mesh_detection` | `assets/samples/mesh_detection.html` | `["mesh","detection","environment","scan","ar","surface"]` | `["none"]` | `["mesh-detection"]` |
981
- | `modelviewer` | `assets/samples/modelviewer.html` | `["model","viewer","3d","gltf","glb","object","rotate","inspect","load"]` | `["pointer"]` | `["input"]` |
982
- | `paint` | `assets/samples/paint.html` | `["paint","draw","brush","stroke","canvas","art","color","gesture"]` | `["pointer"]` | `["input","hands"]` |
983
- | `planar-vst` | `assets/samples/planar-vst.html` | `["plane","vst","passthrough","video","portal","ar","surface"]` | `["none"]` | `["plane-detection","camera"]` |
984
- | `reticle` | `assets/samples/reticle.html` | `["reticle","cursor","pointer","aim","gaze","target","floor","placement"]` | `["none","pointer"]` | `["plane-detection","input"]` |
985
- | `skybox_agent` | `assets/samples/skybox_agent.html` | `["skybox","sky","background","ai","gemini","generate","environment","image"]` | `["pointer"]` | `["input"]` |
986
- | `sound` | `assets/samples/sound.html` | `["sound","audio","music","spatial","3d-audio","positional","play"]` | `["pointer"]` | `["input"]` |
987
- | `ui` | `assets/samples/ui.html` | `["ui","panel","button","menu","list","text","interface","spatial"]` | `["pointer"]` | `["input","ui"]` |
988
- | `virtual-screens` | `assets/samples/virtual-screens.html` | `["screen","virtual","window","share","stream","desktop","browser","display"]` | `["pointer"]` | `["input"]` |
989
- | `3dgs-walkthrough` | `assets/demos/3dgs-walkthrough.html` | `["gaussian","splat","3dgs","scene","walkthrough","room","photo","realistic"]` | `["imu","direction"]` | `["input"]` |
990
- | `aisimulator` | `assets/demos/aisimulator.html` | `["ai","simulate","gemini","agent","roleplay","character","conversation","npc"]` | `["pointer"]` | `["input"]` |
991
- | `balloonpop` | `assets/demos/balloonpop.html` | `["balloon","pop","game","gesture","fun","pinch","particle","explosion"]` | `["none"]` | `["hands","input"]` |
992
- | `ballpit` | `assets/demos/ballpit.html` | `["ball","physics","pit","throw","gravity","interact","fun","ammo"]` | `["pointer"]` | `["input","hands"]` |
993
- | `drone` | `assets/demos/drone.html` | `["drone","fly","navigate","control","imu","tilt","direction","pilot"]` | `["imu","direction"]` | `["input"]` |
994
- | `gemini-icebreakers` | `assets/demos/gemini-icebreakers.html` | `["gemini","icebreaker","question","conversation","ai","social","fun"]` | `["pointer"]` | `["input"]` |
995
- | `gemini-xrobject` | `assets/demos/gemini-xrobject.html` | `["gemini","object","recognize","camera","ar","label","identify","vision"]` | `["pointer"]` | `["input","camera"]` |
996
- | `math3d` | `assets/demos/math3d.html` | `["math","3d","formula","graph","equation","plot","visualization","education"]` | `["pointer"]` | `["input","ui"]` |
997
- | `measure` | `assets/demos/measure.html` | `["measure","ruler","tape","distance","ar","depth","spatial","length"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
998
- | `occlusion` | `assets/demos/occlusion.html` | `["occlusion","depth","animal","model","shadow","ar","realistic","environment"]` | `["pointer"]` | `["input","depth-sensing"]` |
999
- | `rain` | `assets/demos/rain.html` | `["rain","particle","weather","atmosphere","shader","visual","instanced"]` | `["none","pointer"]` | `["input"]` |
1000
- | `screenwiper` | `assets/demos/screenwiper.html` | `["wiper","screen","wipe","clear","gesture","brush","clean","effect"]` | `["pointer"]` | `["input","hands"]` |
1001
- | `splash` | `assets/demos/splash.html` | `["splash","paint","decal","physics","shoot","color","ball","impact"]` | `["pointer"]` | `["input"]` |
1002
- | `webcam_gestures` | `assets/demos/webcam_gestures.html` | `["webcam","mediapipe","gesture","hand","camera","tracking","no-headset","flat"]` | `["none"]` | `[]` |
1003
- | `xremoji` | `assets/demos/xremoji.html` | `["emoji","expression","face","gesture","fun","balloon","tensorflow","social"]` | `["none"]` | `["hands"]` |
1004
- | `xrpoet` | `assets/demos/xrpoet.html` | `["poem","poetry","ai","gemini","generate","creative","camera","writing"]` | `["pointer"]` | `["input","camera"]` |
1005
-
1006
- ---
1007
-
1008
- ## Section 13 — Template Inference + Override
1009
-
1010
- *(Populated after Phase 5 / T032.)*
1011
-
1012
- ---
1013
-
1014
- ## Section 14 — Background Lockdown (XR Blocks default only)
1015
-
1016
- **Custom backgrounds are forbidden.** Every generated XR app uses the XR
1017
- Blocks default room and nothing else. There is no catalog, no helper, no
1018
- override, no `[bg=...]` tag, no `scenePath` override.
1019
-
1020
- ### Hard rules — apply unconditionally
1021
-
1022
- 1. The `xb.Script` subclass MUST NOT contain any `applyBackground_*` method.
1023
- 2. `init()` MUST NOT call any background helper. It starts with lights,
1024
- meshes, and Mudra wiring.
1025
- 3. The entry point MUST NOT set `options.simulator.scenePath` — not to
1026
- `null`, not to a path. Leave it alone so XR Blocks renders its default
1027
- room.
1028
- 4. No `THREE.SphereGeometry` dome, no custom skybox `Mesh`, no
1029
- `THREE.Points` starfield, no `THREE.GridHelper` floor, no equirectangular
1030
- `TextureLoader().load(...)` for background purposes. (Per-scene
1031
- geometry that the app actually needs is fine — the ban is on standalone
1032
- environment domes/floors/skies.)
1033
- 5. Prompt cues like "in space", "starfield", "sunset sky", "cyberpunk
1034
- vibe", "with a forest backdrop", or even literal `[bg=<id>]` tags MUST
1035
- be IGNORED for background purposes. They may still inform template /
1036
- motion-mode selection.
1037
- 6. If the user explicitly insists on a custom background, decline and
1038
- remind them that this skill is locked to the XR Blocks default room.
1039
-
1040
- ### Pre-write regex (Section 10 check #10)
1041
-
1042
- The generated source MUST satisfy ALL of the following:
1043
-
1044
- - `/applyBackground_/` → zero matches.
1045
- - `/options\.simulator\.scenePath/` → zero matches.
1046
-
1047
- If either pattern matches, the file fails the pre-write checklist and is
1048
- not written.
1049
-
1050
- ---
1051
- ## Section 15 — Mode Toggle (Manual / Mudra) — Required
1052
-
1053
- **This section is the canonical source for the connection-status state machine and status pill wiring.** Section 4 defines the `MudraClient` class; Section 7 is superseded by this section.
1054
-
1055
- Every XR app generated by this skill MUST implement the Mode Toggle exactly as specified here.
1056
-
1057
- ### Summary
1058
-
1059
- The user picks how the app is driven via a visible **Mode** control,
1060
- rendered as a 2D DOM overlay (it disappears automatically in immersive
1061
- WebXR sessions, like the simulator panel and status pill):
1062
-
1063
- - **Manual** (default on load): the simulator panel is fully interactive.
1064
- Sim-panel actions inject synthetic signal messages into the same handler
1065
- pipeline that real WebSocket messages would flow through. **No WebSocket
1066
- connection is opened in Manual mode.** The XR scene reacts only to
1067
- signal handlers — never to direct DOM clicks bypassing the signal path.
1068
- - **Mudra**: the app opens one WebSocket to `ws://127.0.0.1:8766` and
1069
- subscribes to its signals one-at-a-time. The simulator panel is
1070
- visually disabled (greyed-out, `pointer-events: none`,
1071
- `aria-disabled="true"`) and emits no synthetic signals. The
1072
- connection-status pill reflects band-pairing state via `get_status`
1073
- polling — **no separate overlay, toast, or banner is rendered.**
1074
-
1075
- **The Mode toggle MUST remain fully clickable and keyboard-focusable at
1076
- all times** — including while the band is disconnected. Manual is the
1077
- default on first load.
1078
-
1079
- ### State machine
1080
-
1081
- ```text
1082
- type Mode = "manual" | "mudra" // default "manual"
1083
- type ConnectionState =
1084
- | "idle" // Manual mode. No socket open.
1085
- | "connecting" // Socket opening, OR socket open but first get_status not yet received.
1086
- | "connected" // Socket open AND device.firmware && device.serial_number both truthy.
1087
- | "ws-only" // Socket open BUT device.firmware or device.serial_number is null/falsy (band not paired).
1088
- | "reconnecting" // Socket closed for non-conflict reason; backoff retry in progress.
1089
- | "already-in-use" // Server sent client_already_connected; terminal until mode→Manual or page reload.
1090
- ```
1091
-
1092
- ### Lazy-WS lifecycle (mandatory)
1093
-
1094
- | Transition | Action |
1095
- |------------|--------|
1096
- | page load | `mode = "manual"`, `connectionState = "idle"`, NO socket |
1097
- | Manual → Mudra | call `_openSocket()`; `connectionState = "connecting"` |
1098
- | Mudra → Manual | call `_closeSocket()`; cancel reconnect + poll timers; `connectionState = "idle"` |
1099
- | WS `onopen` (in Mudra) | reset backoff; replay subscription record; send `{command:"get_status"}`; start 2 s poll timer; `connectionState` stays `"connecting"` until first status response |
1100
- | inbound `{type:"status"}` — `device.firmware && device.serial_number` truthy (in Mudra) | `connectionState = "connected"`; update hand chip from `device.hand`; if transitioning from `"ws-only"`, call `_replaySubscriptions()` (FR-012) |
1101
- | inbound `{type:"status"}` — firmware or serial_number falsy (in Mudra) | `connectionState = "ws-only"`; hand chip = `None` |
1102
- | inbound `{type:"error", data:{error:"client_already_connected"}}` (in Mudra) | `_suppressReconnect = true`; stop poll; `connectionState = "already-in-use"` |
1103
- | WS `onclose` when `_suppressReconnect = true` | no-op (the error frame already set state) |
1104
- | WS `onerror` / `onclose` (in Mudra, non-conflict) | stop poll; `connectionState = "reconnecting"`; advance backoff; schedule `_openSocket()` after backoff delay |
1105
- | reconnect tick fires (in Mudra, non-conflict) | call `_openSocket()` → `"connecting"` |
1106
- | Mudra → Manual at any point | call `_closeSocket()`; clear `_suppressReconnect`; `connectionState = "idle"` |
1107
-
1108
- **Single-socket guarantee:** The `_connToken` counter (Section 4) ensures callbacks from a stale socket are discarded if mode is toggled rapidly.
1109
-
1110
- ### Band-connected predicate (mandatory)
1111
-
1112
- **The WebSocket handshake to `127.0.0.1:8766` only proves the Companion service is up. It does NOT prove the Mudra Band is paired.**
1113
-
1114
- The band-connected check uses both `device.firmware` AND `device.serial_number`:
1115
-
1116
- ```js
1117
- const bandConnected = !!(msg.data?.device?.firmware && msg.data?.device?.serial_number);
1118
- ```
1119
-
1120
- - Both truthy → `connectionState = "connected"`.
1121
- - Either null/falsy → `connectionState = "ws-only"`.
1122
-
1123
- Poll every **2000 ms** while the socket is `OPEN`. The pill MAY sit on `Connecting…` for up to one poll cycle (~2 s) after entering Mudra mode — that is correct behaviour.
1124
-
1125
- ### Hand chip
1126
-
1127
- Every generated app MUST render a hand chip (`<span id="mudra-hand">`) beside the status pill:
1128
-
1129
- - Shows `LEFT` or `RIGHT` (raw value from `device.hand`, uppercased) when `connectionState === "connected"`.
1130
- - Shows `None` in all other states.
1131
- - Normalization: if `device.hand` arrives as an enum string (e.g., `HandType.right`), strip the type prefix and uppercase. If normalization fails, display `None`.
1132
-
1133
- ```js
1134
- mudra.on('_hand', (hand) => {
1135
- document.getElementById('mudra-hand').textContent = hand;
1136
- });
1137
- ```
1138
-
1139
- ### Reconnect backoff
1140
-
1141
- Non-conflict WS close while `mode === "mudra"`:
1142
-
1143
- ```js
1144
- // MudraClient._BACKOFF = [1000, 2000, 5000, 10000] // ms; last value is the cap
1145
- ```
1146
-
1147
- - `_backoffIdx` advances on every non-conflict close; resets to 0 on every `onopen`.
1148
- - The `client_already_connected` path sets `_suppressReconnect = true` which bypasses the backoff timer entirely.
1149
-
1150
- ### Status pill text states
1151
-
1152
- | connectionState | `#mudra-status` text | `#mudra-hand` text | Colour hint |
1153
- |---|---|---|---|
1154
- | `idle` (Manual) | `Manual` | `None` | neutral |
1155
- | `connecting` | `Connecting…` | `None` | amber |
1156
- | `connected` | `Connected` | `LEFT` or `RIGHT` | green |
1157
- | `ws-only` | `WebSocket Only` | `None` | **orange** (`#eab308`) |
1158
- | `reconnecting` | `Reconnecting…` | `None` | amber (muted) |
1159
- | `already-in-use` | `Companion already in use — close the other tab first` | `None` | red/error |
1160
-
1161
- The pill + hand chip are the **only** connection indicators. No banner, toast, or modal.
1162
-
1163
- The simulator panel is greyed (`pointer-events: none`, reduced opacity) in every state except `idle` (Manual).
1164
-
1165
- ### Mode-toggle DOM sketch
1166
-
1167
- ```html
1168
- <div id="mode-toggle" role="tablist" aria-label="Mode">
1169
- <button id="mode-manual" role="tab" aria-selected="true">Manual</button>
1170
- <button id="mode-mudra" role="tab" aria-selected="false">Mudra</button>
1171
- </div>
1172
- <div style="display:flex;align-items:center;gap:6px;position:fixed;top:8px;right:12px;z-index:9999;">
1173
- <div id="mudra-status" style="padding:4px 10px;border-radius:999px;font-size:0.8rem;font-family:system-ui,sans-serif;background:rgba(0,0,0,0.6);color:#fff;">Manual</div>
1174
- <span id="mudra-hand" style="padding:2px 8px;border-radius:999px;font-size:0.75rem;font-family:system-ui,sans-serif;background:rgba(255,255,255,0.12);color:#fff;">None</span>
1175
- </div>
1176
- ```
1177
-
1178
- Both `#mudra-status` and `#mudra-hand` are 2D DOM elements — they disappear automatically in immersive WebXR sessions via native DOM suppression. No extra JS wiring needed for this.
1179
-
1180
- ### Status pill wiring
1181
-
1182
- ```js
1183
- mudra.on('_status', (s) => {
1184
- const pill = document.getElementById('mudra-status');
1185
- const labels = {
1186
- 'idle': 'Manual',
1187
- 'connecting': 'Connecting…',
1188
- 'connected': 'Connected',
1189
- 'ws-only': 'WebSocket Only',
1190
- 'reconnecting': 'Reconnecting…',
1191
- 'already-in-use':'Companion already in use — close the other tab first',
1192
- };
1193
- const colors = {
1194
- 'idle': 'rgba(0,0,0,0.6)',
1195
- 'connecting': 'rgba(180,120,0,0.8)',
1196
- 'connected': 'rgba(22,163,74,0.8)',
1197
- 'ws-only': '#eab308',
1198
- 'reconnecting': 'rgba(120,80,0,0.7)',
1199
- 'already-in-use':'rgba(185,28,28,0.85)',
1200
- };
1201
- pill.textContent = labels[s] ?? s;
1202
- pill.style.background = colors[s] ?? 'rgba(0,0,0,0.6)';
1203
-
1204
- // Grey the simulator panel in every non-Manual state
1205
- const sim = document.getElementById('mudra-sim');
1206
- if (sim) {
1207
- sim.style.opacity = s === 'idle' ? '1' : '0.35';
1208
- sim.style.pointerEvents = s === 'idle' ? '' : 'none';
1209
- }
1210
-
1211
- // Manage mode-toggle button aria-selected
1212
- document.getElementById('mode-manual').setAttribute('aria-selected', s === 'idle' ? 'true' : 'false');
1213
- document.getElementById('mode-mudra').setAttribute('aria-selected', s !== 'idle' ? 'true' : 'false');
1214
- });
1215
-
1216
- mudra.on('_hand', (hand) => {
1217
- document.getElementById('mudra-hand').textContent = hand;
1218
- });
1219
- ```
1220
-
1221
- On every mode button click, atomically: call `mudra.setMode(newMode)`. `MudraClient.setMode()` handles all internal cleanup (cancel timers, close socket, reset tokens).
1222
-
1223
- ---
1224
-
1225
- ## Section 16 — Footer Branding — "Created by Mudra"
1226
-
1227
- Every generated app MUST render a small footer/badge with the **exact**
1228
- text **`Created by Mudra`**. Never "Created with Mudra Studio", never
1229
- "Powered by Mudra", never any other variant. The badge is a 2D DOM
1230
- overlay (disappears in immersive WebXR like the rest of the chrome).
1231
-
1232
- ```html
1233
- <div id="mudra-badge" style="
1234
- position: fixed; bottom: 8px; right: 12px;
1235
- padding: 4px 10px; border-radius: 999px;
1236
- font-size: 0.75rem; font-family: system-ui, sans-serif;
1237
- background: rgba(0,0,0,0.5); color: #fff; opacity: 0.85;
1238
- z-index: 9999; pointer-events: none;">Created by Mudra</div>
1239
- ```
1240
-
1241
- Position is flexible if it would overlap the simulator panel — keep the
1242
- text identical.
1243
-
1244
- ---
1245
-
1246
- ## Section 17 — Onboarding Modal (mandatory, STRICT) — feature 008-strict-onboarding-templates
1247
-
1248
- > **Supersedes feature 005's loose modal rules.** The locked layout is
1249
- > **Template 3 — split-card**. The DOM, CSS, and JS below MUST be emitted
1250
- > **verbatim**. Only the per-app content slots may vary.
1251
-
1252
- Every generated XR app MUST ship a first-run onboarding modal that greets
1253
- the user and lists every action the app supports, with paired **Mudra** and
1254
- **Manual** controls per action. The modal closes via `×` (skip), the
1255
- **Continue** button, or `Escape`. It re-opens via a small floating `?` icon
1256
- **only outside immersive XR** — in-XR re-onboarding is out of scope for v1
1257
- (per feature 005 clarification Q3).
1258
-
1259
- The binding contract is `specs/008-strict-onboarding-templates/contracts/onboarding-block.md`. The blocks below are the verbatim copies emitted into every app.
1260
-
1261
- ### XR-specific behavior (read carefully — unchanged from feature 005)
1262
-
1263
- The onboarding modal is a **2D HTML overlay** shown before immersive entry.
1264
-
1265
- - **MANDATORY: Disable the XR Blocks default Welcome overlay.** When the
1266
- Simulator addon is imported (`import 'xrblocks/addons/simulator/SimulatorAddons.js';`),
1267
- XR Blocks injects its own "Welcome to XR Blocks!" intro modal. This
1268
- overlay competes visually with the Mudra onboarding modal and must be
1269
- suppressed. Immediately after constructing the options object, set:
1270
-
1271
- ```js
1272
- const options = new xb.Options();
1273
- options.simulator.instructions.enabled = false; // suppress XR Blocks default Welcome overlay; the Mudra onboarding modal replaces it
1274
- // ...rest of options config
1275
- xb.init(options);
1276
- ```
1277
-
1278
- - **Do NOT** mirror the Mudra modal as a 3D panel inside the XR scene. In-XR re-onboarding is explicitly out of scope for v1.
1279
- - The inline `<script>` already listens for `xrsession-start` / `vr-session-start` / `ar-session-start` events and hides both the modal and the `?` icon during immersive sessions. Do NOT edit that wiring.
1280
- - The modal layers above the XR canvas in 2D mode (`z-index: 100`). The Mudra badge from Section 16 and the simulator panel sit beneath it, by design.
1281
-
1282
- ### Required palette addition
1283
-
1284
- Every generated app's `:root` MUST add one variable beyond the canonical palette:
1285
-
1286
- ```css
1287
- --on-primary: #0c0d10; /* dark text on the primary-blue Continue button */
1288
- ```
1289
-
1290
- ### Locked DOM (paste at end of `<body>`)
1291
-
1292
- ```html
1293
- <!-- === BEGIN onboarding-block === (Template 3 — Split Card, feature 008) -->
1294
- <!-- IMPORTANT: do NOT add the `open` attribute. In 3D apps this matters
1295
- particularly — XR Blocks' canvas is appended to <body> AFTER this
1296
- dialog, and a non-modal dialog (HTML `open`) does NOT enter the
1297
- browser's top layer. The dialog looks visible but the canvas can
1298
- intercept clicks on Continue / ×. The IIFE below calls showModal()
1299
- on load, which elevates the dialog above XR Blocks' canvas
1300
- regardless of z-index. -->
1301
- <dialog id="mudra-onboarding" data-mudra-onboarding data-app-name="{APP_NAME}">
1302
- <div class="ob-card">
1303
- <button class="ob-x" aria-label="Skip onboarding" data-ob-close>×</button>
1304
- <div class="ob-left">
1305
- <div class="ob-brand-block">
1306
- <span class="ob-brand-mark">Mudra Studio</span>
1307
- <h2 class="ob-brand-name">{APP_NAME_HEAD} <em>{APP_NAME_TAIL}</em></h2>
1308
- <p class="ob-tagline">{APP_TAGLINE}</p>
1309
- </div>
1310
- <span class="ob-brand-footer">Created by Mudra</span>
1311
- </div>
1312
- <div class="ob-right">
1313
- <h3 class="ob-section-title">How to use this app</h3>
1314
- <div class="ob-chip-grid" id="ob-rows"></div>
1315
- <div class="ob-continue-row"><button class="ob-continue" data-ob-close>Continue</button></div>
1316
- </div>
1317
- </div>
1318
- </dialog>
1319
- <button id="mudra-onboarding-help" class="ob-help-btn" aria-label="Reopen onboarding" hidden>?</button>
1320
- <!-- === END onboarding-block === -->
1321
- ```
1322
-
1323
- ### Locked CSS (paste inside the existing `<style>` block in `<head>` — add one if the template does not have one)
1324
-
1325
- ```css
1326
- /* === BEGIN onboarding-block === (Template 3 — Split Card, feature 008) */
1327
- #mudra-onboarding{position:fixed;inset:0;border:0;padding:0;background:transparent;width:100%;height:100%;max-width:none;max-height:none;display:grid;place-items:center;z-index:100;color:var(--text);}
1328
- #mudra-onboarding::backdrop{background:rgba(0,0,0,0.55);backdrop-filter:blur(6px);}
1329
- #mudra-onboarding[hidden],#mudra-onboarding:not([open]){display:none;}
1330
- .ob-card{position:relative;background:var(--card);backdrop-filter:blur(10px);border:1px solid rgba(255,255,255,0.08);border-radius:18px;width:min(720px,94vw);max-height:88vh;overflow:auto;display:grid;grid-template-columns:1fr 1fr;gap:0;box-shadow:0 20px 60px rgba(0,0,0,0.5);font-family:'Poppins',system-ui,sans-serif;}
1331
- .ob-x{position:absolute;top:14px;right:14px;width:32px;height:32px;border-radius:50%;appearance:none;border:0;background:rgba(255,255,255,0.08);color:var(--text);font-size:18px;cursor:pointer;z-index:2;}
1332
- .ob-x:hover{background:rgba(255,255,255,0.16);}
1333
- .ob-left{padding:36px 28px;background:linear-gradient(135deg,rgba(108,140,255,0.16),rgba(185,124,255,0.12));border-right:1px solid rgba(255,255,255,0.06);display:flex;flex-direction:column;justify-content:space-between;border-radius:18px 0 0 18px;}
1334
- .ob-brand-block{display:flex;flex-direction:column;gap:14px;}
1335
- .ob-brand-mark{display:flex;align-items:center;gap:10px;font-size:13px;letter-spacing:0.16em;text-transform:uppercase;color:var(--text-secondary);}
1336
- .ob-brand-mark::before{content:"";display:inline-block;width:24px;height:2px;background:var(--primary);}
1337
- .ob-brand-name{font-size:34px;font-weight:700;line-height:1.05;margin:0;}
1338
- .ob-brand-name em{font-style:normal;color:var(--accent);}
1339
- .ob-tagline{color:var(--text-secondary);font-size:15px;margin:6px 0 0;line-height:1.5;}
1340
- .ob-brand-footer{font-size:11px;letter-spacing:0.18em;text-transform:uppercase;color:var(--text-secondary);}
1341
- .ob-right{padding:32px 28px 24px;display:flex;flex-direction:column;}
1342
- .ob-section-title{font-size:12px;letter-spacing:0.1em;text-transform:uppercase;color:var(--text-secondary);margin:0 0 14px;}
1343
- .ob-chip-grid{display:grid;grid-template-columns:1fr;gap:8px;flex:1;}
1344
- .ob-chip{display:grid;grid-template-columns:1fr auto auto;gap:10px;align-items:center;background:rgba(255,255,255,0.04);border:1px solid rgba(255,255,255,0.05);border-radius:10px;padding:10px 12px;font-size:14px;}
1345
- .ob-chip .nm{font-weight:500;}
1346
- .ob-chip .mu{font-size:12px;padding:3px 9px;border-radius:999px;background:rgba(108,140,255,0.2);color:var(--primary);font-weight:600;}
1347
- .ob-chip .mn{font-size:12px;color:var(--text-secondary);}
1348
- .ob-continue-row{display:flex;justify-content:flex-end;margin-top:18px;}
1349
- .ob-continue{appearance:none;border:0;background:var(--primary);color:var(--on-primary);font:inherit;font-weight:600;padding:10px 22px;border-radius:10px;cursor:pointer;}
1350
- .ob-continue:hover{filter:brightness(1.1);}
1351
- .ob-help-btn{position:fixed;bottom:16px;right:16px;appearance:none;border:0;width:36px;height:36px;border-radius:50%;background:var(--card);color:var(--text);font-size:18px;cursor:pointer;backdrop-filter:blur(10px);border:1px solid rgba(255,255,255,0.12);z-index:50;}
1352
- @media (max-width:640px){
1353
- .ob-card{grid-template-columns:1fr;}
1354
- .ob-left{border-right:0;border-bottom:1px solid rgba(255,255,255,0.06);border-radius:18px 18px 0 0;padding:24px;}
1355
- .ob-brand-name{font-size:24px;}
1356
- .ob-right{padding:20px;}
1357
- .ob-continue{width:100%;}
1358
- .ob-continue-row{justify-content:stretch;}
1359
- }
1360
- /* === END onboarding-block === */
1361
- ```
1362
-
1363
- ### Locked JS (paste inside an inline `<script>` — NOT `type="module"` — at end of `<body>`)
1364
-
1365
- ```js
1366
- // === BEGIN onboarding-block === (Template 3 — Split Card, feature 008)
1367
- window.MUDRA_ONBOARDING_ACTIONS = [
1368
- // Filled by the skill from the app's subscribed signals. See "App-aware filter — strict" below.
1369
- ];
1370
-
1371
- (function () {
1372
- const root = document.getElementById('mudra-onboarding');
1373
- const help = document.getElementById('mudra-onboarding-help');
1374
- const grid = document.getElementById('ob-rows');
1375
-
1376
- grid.innerHTML = window.MUDRA_ONBOARDING_ACTIONS.map(r => `
1377
- <div class="ob-chip">
1378
- <span class="nm">${r.action}</span>
1379
- <span class="mu">${r.mudra}</span>
1380
- <span class="mn">${r.manual}</span>
1381
- </div>`).join('');
1382
-
1383
- function isInImmersiveXR() {
1384
- return !!(window.xb && window.xb.session && window.xb.session.isImmersive);
1385
- }
1386
- function openOb() { if (!root.open) root.showModal(); help.hidden = true; }
1387
- function closeOb() { if (root.open) root.close(); help.hidden = false; }
1388
-
1389
- // Close wiring — BOTH `.ob-x` and `.ob-continue` carry [data-ob-close].
1390
- // This single querySelectorAll attaches the same `closeOb` to each;
1391
- // do not add separate listeners or split the behavior.
1392
- root.querySelectorAll('[data-ob-close]').forEach(b => b.addEventListener('click', closeOb));
1393
-
1394
- // Re-open wiring — floating help-pill (mouse/touch) and the `?` key.
1395
- // `?` is gated on isInImmersiveXR() so it never re-opens inside VR/AR.
1396
- help.addEventListener('click', openOb);
1397
- document.addEventListener('keydown', (e) => {
1398
- if (e.key === 'Escape' && root.open) closeOb();
1399
- if (e.key === '?' && !root.open && !isInImmersiveXR()) openOb();
1400
- });
1401
-
1402
- // Hide modal + help during immersive XR sessions (preserves feature 005's contract).
1403
- ['xrsession-start','vr-session-start','ar-session-start'].forEach(ev =>
1404
- window.addEventListener(ev, () => { if (root.open) root.close(); help.hidden = true; })
1405
- );
1406
-
1407
- // Open on load — UNCONDITIONAL showModal(). The dialog enters the
1408
- // browser's top layer so XR Blocks' canvas (appended after this
1409
- // script) can never intercept clicks on Continue / ×. Do not gate on
1410
- // `root.open`; do not use `root.show()`.
1411
- root.showModal();
1412
- })();
1413
- // === END onboarding-block ===
1414
- ```
1415
-
1416
- ### Close behavior — what each control does (verbatim explanation for the model)
1417
-
1418
- | Control | When it fires | Effect |
1419
- |---|---|---|
1420
- | **Continue** (`.ob-continue`) | User clicks / activates the primary CTA in the right column | `root.close()`. Help-pill `?` becomes visible bottom-right. User starts using the app (or enters XR via the XR Blocks button). |
1421
- | **×** (`.ob-x`) | User clicks / activates the circular X in the top-right of the card | Same as Continue — `root.close()` + show `?` pill. The two paths are intentionally equivalent. X is the visual "skip" affordance. (Exception: AI-Setup extension blocks both Continue and × until a valid API key is entered — see "AI-app extension" below.) |
1422
- | **Escape** | User presses Esc while modal is open | Native `<dialog>` close + our `keydown` handler unhides the `?` pill. (AI extension also intercepts this via the `cancel` event when no valid key is set.) |
1423
- | **`?` key** | User presses `?` while modal is closed AND NOT inside immersive XR | `root.showModal()` reopens. Inside VR/AR the key is a no-op so the user is never yanked out of the scene. |
1424
- | **Help-pill** (`#mudra-onboarding-help`) | User clicks the floating `?` button bottom-right (visible only after the modal has been closed once, and hidden during immersive XR) | `root.showModal()` reopens. |
1425
- | **`xrsession-start` / `vr-session-start` / `ar-session-start`** | User enters VR/AR via the XR Blocks Enter button | Force-close the modal; hide the help-pill so it does not appear in-world. |
1426
- | **Page load** | Every fresh render | `root.showModal()` runs unconditionally inside the IIFE. Help-pill starts hidden. |
1427
-
1428
- ### How "close" is wired (REQUIRED — do not refactor)
1429
-
1430
- 1. Both close buttons carry the `data-ob-close` attribute. The single line `root.querySelectorAll('[data-ob-close]').forEach(b => b.addEventListener('click', closeOb))` is the entire mouse/touch close wiring. The AI-Setup extension below adds its own `capture:true` listener on top of these to gate close when no API key is present — it does not replace the underlying wiring.
1431
- 2. Keyboard close is handled by the `document`-level `keydown` listener so Escape fires even when focus is outside the dialog.
1432
- 3. Opening on load uses `root.showModal()` (modal mode, top layer). Never `root.show()` (non-modal) and never the `open` attribute in HTML. In 3D apps this is REQUIRED — non-modal dialogs do not enter the top layer and XR Blocks' canvas can intercept clicks behind the modal.
1433
-
1434
- ### Anti-patterns — will fail review
1435
-
1436
- - ❌ `<dialog ... open>` in HTML. Open via `showModal()` only.
1437
- - ❌ Calling `root.show()` instead of `root.showModal()`.
1438
- - ❌ Omitting `options.simulator.instructions.enabled = false;` — XR Blocks' Welcome overlay then competes with the Mudra modal.
1439
- - ❌ Adding click-outside-to-close, time-out auto-close, or any extra close path.
1440
- - ❌ Hiding `#mudra-onboarding-help` permanently after first close — it must reappear (and remain hidden ONLY during immersive XR).
1441
- - ❌ Wiring `closeOb` to a button that lacks the `data-ob-close` attribute.
1442
- - ❌ Differentiating × from Continue behaviorally (e.g., "X means skip, Continue means save"). They are the same close path.
1443
-
1444
- ### Per-app content slots — the ONLY things you may vary
1445
-
1446
- | Slot | Source | Notes |
1447
- |---|---|---|
1448
- | `{APP_NAME}` (`data-app-name`) | Derived from generated HTML filename | Override only to fix acronym capitalization (e.g., `data-app-name="AR Menu"` for `ar-menu.html`). |
1449
- | `{APP_NAME_HEAD}` / `{APP_NAME_TAIL}` | App name split into leading word + trailing word(s). Trailing word gets the `<em>` accent. | Single-word names: HEAD is whole word, TAIL is empty (emit `<em></em>`). |
1450
- | `{APP_TAGLINE}` | One-line description. MUST end with a period. MUST NOT exceed 90 characters. | Example: "Stack blocks with a flick of the wrist." |
1451
- | `MUDRA_ONBOARDING_ACTIONS` | Per `actions-array.md`. | See filter rule below. |
1452
-
1453
- ### `MUDRA_ONBOARDING_ACTIONS` shape (renamed from feature-005 `ACTIONS`)
1454
-
1455
- ```js
1456
- window.MUDRA_ONBOARDING_ACTIONS = [
1457
- { action: "Toggle XR session", mudra: "Thumb tap", manual: "Enter", mode: "gesture" },
1458
- { action: "Look around", mudra: "Lift wrist", manual: "Right-click + drag", mode: "imu_acc" },
1459
- { action: "Move", mudra: "Swipe", manual: "W A S D", mode: "nav_direction" }
1460
- ];
1461
- ```
1462
-
1463
- Each row has four required fields:
1464
-
1465
- - **`action`** — the behavior in plain English (NOT the control name).
1466
- - **`mudra`** — the Mudra control prose (`"Tap"`, `"Twist"`, `"Press 70%"`, `"Tilt left"`).
1467
- - **`manual`** — keyboard / mouse fallback. Use `"—"` (em dash) if there is no Manual equivalent (common in XR for camera / look controls).
1468
- - **`mode`** — one of: `gesture` | `button` | `pressure` | `navigation` | `nav_direction` | `imu_acc` | `imu_gyro` | `snc`.
1469
-
1470
- ### App-aware filter — STRICT (feature 008, FR-010)
1471
-
1472
- Before emitting `MUDRA_ONBOARDING_ACTIONS`, the skill MUST filter:
1473
-
1474
- 1. **Build the subscribed set** — every canonical signal this specific app subscribes to.
1475
- 2. **Drop every row** whose `mode` is not in the subscribed set. **Forbidden** to emit a row for an unsubscribed signal — that is a generation-time bug, not a runtime filter.
1476
- 3. **Verify exactly one motion mode** (`navigation` | `nav_direction` | `imu_acc`/`imu_gyro` family) across all rows (Constitution Principle III).
1477
- 4. **Verify Manual ↔ Mudra parity** — every row has both a Mudra and a Manual cell; `"—"` is the only acceptable Manual placeholder.
1478
- 5. **No collisions** — no two rows share the same `manual` value or the same effective `mudra` value.
1479
-
1480
- ### Anti-patterns (will fail review)
1481
-
1482
- - ❌ Omitting `options.simulator.instructions.enabled = false;` — the XR Blocks default Welcome overlay then duplicates / overrides the Mudra onboarding modal.
1483
- - ❌ Mirroring the modal as a 3D panel inside the XR scene (out of scope v1).
1484
- - ❌ Editing the `xrsession-start` / `xrsession-end` hide-show wiring.
1485
- - ❌ Emitting a row whose `mode` is not in the app's subscribed set.
1486
- - ❌ Mixing two motion modes in the same array.
1487
- - ❌ A row with `manual: null` or `manual: ""` — use `"—"`.
1488
- - ❌ Two rows with the same `manual` value — keyboard collision.
1489
- - ❌ Branding strings other than literal `Created by Mudra`. ("Mudra Studio" stays only as the `.ob-brand-mark` line above the app name.)
1490
- - ❌ CTA labels other than `Continue`.
1491
-
1492
- Binding contracts: `specs/008-strict-onboarding-templates/contracts/onboarding-block.md` and `specs/008-strict-onboarding-templates/contracts/actions-array.md`.
1493
-
1494
- ### AI-app extension (mandatory when `usesAI`)
1495
-
1496
- The AI-Setup fragment is rendered as an **extra row inside `.ob-chip-grid`** — not as a separate section. The fragment goes at the **top** of the chip grid (above all action chips) so the user sees it before any controls.
1497
-
1498
- ```html
1499
- <!-- Inject AS THE FIRST CHILD of #ob-rows when usesAI is true -->
1500
- <div class="ob-chip ob-chip-ai" id="ob-chip-ai" data-role="ai-setup">
1501
- <span class="nm">AI key</span>
1502
- <input class="mudra-onb__ai-input" type="password" placeholder="AIza…" aria-label="Gemini API key" autocomplete="off" />
1503
- <span class="mn" data-role="hint" aria-live="polite"></span>
1504
- </div>
1505
- ```
1506
-
1507
- ```css
1508
- /* Append to the locked onboarding-block CSS for AI-using apps only */
1509
- .ob-chip-ai{grid-template-columns:auto 1fr auto;}
1510
- .ob-chip-ai .mudra-onb__ai-input{appearance:none;background:rgba(255,255,255,0.04);color:var(--text);border:1px solid rgba(255,255,255,0.12);border-radius:8px;padding:6px 10px;font:inherit;font-size:13px;}
1511
- .ob-chip-ai .mudra-onb__ai-input:focus{outline:2px solid var(--primary);outline-offset:0;}
1512
- .ob-chip-ai .mn[data-role="hint"]{color:var(--warning);}
1513
- ```
1514
-
1515
- The IIFE that wires the modal MUST be extended with this gating block, inserted just before the final unconditional `root.showModal();`:
1516
-
1517
- ```js
1518
- // AI-Setup gating — only runs when the fragment is present (usesAI === true)
1519
- const aiFragment = document.getElementById('ob-chip-ai');
1520
- const continueBtn = root.querySelector('.ob-continue');
1521
- const closeBtn = root.querySelector('.ob-x');
1522
- const KEY_NAME = "mudra.gemini.apiKey";
1523
- const KEY_REGEX = /^AIza[\w-]{30,}$/;
1524
-
1525
- const hasValidStoredKey = () => {
1526
- const k = sessionStorage.getItem(KEY_NAME);
1527
- return typeof k === "string" && KEY_REGEX.test(k.trim());
1528
- };
1529
-
1530
- if (aiFragment) {
1531
- const input = aiFragment.querySelector(".mudra-onb__ai-input");
1532
- const hint = aiFragment.querySelector('[data-role="hint"]');
1533
-
1534
- const refreshGate = () => {
1535
- if (hasValidStoredKey()) {
1536
- aiFragment.hidden = true;
1537
- continueBtn.disabled = false;
1538
- return;
1539
- }
1540
- aiFragment.hidden = false;
1541
- const v = input.value.trim();
1542
- const ok = KEY_REGEX.test(v);
1543
- continueBtn.disabled = !ok;
1544
- hint.textContent = (!v || ok) ? "" : "Key should start with \"AIza\" and be ~39 chars.";
1545
- };
1546
-
1547
- input.addEventListener("input", refreshGate);
1548
- refreshGate();
1549
-
1550
- continueBtn.addEventListener("click", () => {
1551
- const v = input.value.trim();
1552
- if (KEY_REGEX.test(v)) sessionStorage.setItem(KEY_NAME, v);
1553
- }, { capture: true });
1554
-
1555
- const blockClose = (e) => {
1556
- if (hasValidStoredKey()) return;
1557
- e.preventDefault();
1558
- e.stopPropagation();
1559
- input.focus();
1560
- };
1561
- closeBtn.addEventListener("click", blockClose, { capture: true });
1562
- root.addEventListener("cancel", blockClose); // Escape
1563
- }
1564
- ```
1565
-
1566
- When the AI fragment is **absent** (non-AI app), the existing wiring runs untouched — `Continue` enables immediately, `Escape` / `×` dismiss as normal.
1567
-
1568
- > **Migration note (feature 008, 2026-05-14):** the legacy `ACTIONS` variable is renamed to `MUDRA_ONBOARDING_ACTIONS`. The legacy `mudra-onb__*` class names from feature 005 are replaced by `ob-*` (the locked block's class names). The `Got it` CTA label is renamed to `Continue`. The AI-Setup fragment moves from `.mudra-onb__body` (separate section) into `.ob-chip-grid` (first chip).
1569
-
1570
- ---
1571
-
1572
- ## Section 18 — Visible AI Chat I/O (mandatory when `usesAI`)
1573
-
1574
- Every AI-using app MUST render the conversation as on-screen text in the
1575
- 3D scene — TTS / speech synthesis is optional, never a substitute.
1576
-
1577
- ### Required visible elements
1578
-
1579
- | Element | What it shows | How to render |
1580
- |---------|---------------|---------------|
1581
- | **Purpose line** | One short sentence stating what the app does ("Ask anything — I'll answer.", "Tell me the date.", etc.). Visible at all times. | Troika `Text` or a top row in an `xb.SpatialPanel`. |
1582
- | **User input echo** | The latest user message (transcript from speech recognition OR typed text). Updates as soon as input is captured. | A bordered/highlighted row in the panel, e.g. prefixed `💬 You: …`. |
1583
- | **AI response** | The most recent AI reply in full readable text. Scrollable / wrapping. | `xb.ScrollingTroikaTextView` or a tall row in the panel, prefixed `🤖 AI: …`. |
1584
- | **Listening / Thinking indicator** | Distinguishes idle / listening / thinking states. | Avatar pulse + a single-word status line ("Listening…", "Thinking…", "Tap to talk"). |
1585
-
1586
- ### Rules
1587
-
1588
- 1. **Both sides of every exchange must be visible.** Voice-only output
1589
- is a checklist failure. The chat panel must accumulate at least the
1590
- last user turn AND the last AI turn at the same time.
1591
- 2. **The Purpose line is fixed** for the lifetime of the app. Author it
1592
- from the user's prompt — e.g. `"Create 3d AI I can ask the date"` →
1593
- Purpose `"Voice assistant — ask anything, tap to talk."`. Never use
1594
- a placeholder like `"AI Chat"` alone.
1595
- 3. **Show typed input when speech recognition is unavailable.** If
1596
- `webkitSpeechRecognition` / `SpeechRecognition` are missing, render
1597
- an `<input type="text">` inside an XR Blocks panel OR a 2D overlay
1598
- below the simulator panel. Both echo and reply still render in the
1599
- 3D scene.
1600
- 4. **When no API key is set yet**, the chat panel renders the literal
1601
- text `Set up AI in the welcome panel` in the response slot — do not
1602
- attempt any API call.
1603
- 5. **TTS** (`speechSynthesis`) is allowed but optional. If present, it
1604
- speaks the AI response in addition to displaying it. Never as a
1605
- replacement.