mudra-skills 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +19 -0
- package/CLAUDE.md +35 -0
- package/bin/install.js +25 -0
- package/package.json +28 -0
- package/skills/mudra-master/SKILL.md +202 -0
- package/skills/mudra-master/mudra-preview/SKILL.md +151 -0
- package/skills/mudra-master/mudra-preview/assets/ar-menu.html +382 -0
- package/skills/mudra-master/mudra-preview/assets/document-scroller.html +468 -0
- package/skills/mudra-master/mudra-preview/assets/drum-machine.html +788 -0
- package/skills/mudra-master/mudra-preview/assets/emg-visualizer.html +390 -0
- package/skills/mudra-master/mudra-preview/assets/generative-art.html +408 -0
- package/skills/mudra-master/mudra-preview/assets/gesture-assistant.html +342 -0
- package/skills/mudra-master/mudra-preview/assets/gesture-speech.html +444 -0
- package/skills/mudra-master/mudra-preview/assets/hands-free-desktop.html +376 -0
- package/skills/mudra-master/mudra-preview/assets/model-rotator.html +365 -0
- package/skills/mudra-master/mudra-preview/assets/mudra-duel.html +3200 -0
- package/skills/mudra-master/mudra-preview/assets/mudra-monitor.html +1451 -0
- package/skills/mudra-master/mudra-preview/assets/mudra-ultimate-template.html +891 -0
- package/skills/mudra-master/mudra-preview/assets/music-sequencer.html +455 -0
- package/skills/mudra-master/mudra-preview/assets/neural-pong.html +380 -0
- package/skills/mudra-master/mudra-preview/assets/neural-snake.html +364 -0
- package/skills/mudra-master/mudra-preview/assets/presentation-controller.html +345 -0
- package/skills/mudra-master/mudra-preview/assets/pressure-painter.html +285 -0
- package/skills/mudra-master/mudra-preview/assets/runner.html +1812 -0
- package/skills/mudra-master/mudra-preview/assets/smart-home.html +461 -0
- package/skills/mudra-master/mudra-preview/assets/space-invaders.html +1154 -0
- package/skills/mudra-master/mudra-preview/assets/waterful-ring-toss.html +1624 -0
- package/skills/mudra-master/mudra-preview/references/agent_protocol.json +646 -0
- package/skills/mudra-master/mudra-preview/references/promt.md +16829 -0
- package/skills/mudra-master/mudra-xr/SKILL.md +245 -0
- package/skills/mudra-master/mudra-xr/assets/demos/3dgs-walkthrough.html +242 -0
- package/skills/mudra-master/mudra-xr/assets/demos/aisimulator.html +792 -0
- package/skills/mudra-master/mudra-xr/assets/demos/balloonpop.html +422 -0
- package/skills/mudra-master/mudra-xr/assets/demos/ballpit.html +284 -0
- package/skills/mudra-master/mudra-xr/assets/demos/drone.html +41 -0
- package/skills/mudra-master/mudra-xr/assets/demos/gemini-icebreakers.html +338 -0
- package/skills/mudra-master/mudra-xr/assets/demos/gemini-xrobject.html +302 -0
- package/skills/mudra-master/mudra-xr/assets/demos/math3d.html +188 -0
- package/skills/mudra-master/mudra-xr/assets/demos/measure.html +247 -0
- package/skills/mudra-master/mudra-xr/assets/demos/occlusion.html +251 -0
- package/skills/mudra-master/mudra-xr/assets/demos/rain.html +498 -0
- package/skills/mudra-master/mudra-xr/assets/demos/screenwiper.html +432 -0
- package/skills/mudra-master/mudra-xr/assets/demos/splash.html +519 -0
- package/skills/mudra-master/mudra-xr/assets/demos/webcam_gestures.html +253 -0
- package/skills/mudra-master/mudra-xr/assets/demos/xremoji.html +833 -0
- package/skills/mudra-master/mudra-xr/assets/demos/xrpoet.html +151 -0
- package/skills/mudra-master/mudra-xr/assets/samples/depthmap.html +266 -0
- package/skills/mudra-master/mudra-xr/assets/samples/depthmesh.html +100 -0
- package/skills/mudra-master/mudra-xr/assets/samples/game_rps.html +1099 -0
- package/skills/mudra-master/mudra-xr/assets/samples/gestures_custom.html +467 -0
- package/skills/mudra-master/mudra-xr/assets/samples/gestures_heuristic.html +256 -0
- package/skills/mudra-master/mudra-xr/assets/samples/lighting.html +163 -0
- package/skills/mudra-master/mudra-xr/assets/samples/mesh_detection.html +46 -0
- package/skills/mudra-master/mudra-xr/assets/samples/modelviewer.html +183 -0
- package/skills/mudra-master/mudra-xr/assets/samples/paint.html +133 -0
- package/skills/mudra-master/mudra-xr/assets/samples/planar-vst.html +45 -0
- package/skills/mudra-master/mudra-xr/assets/samples/reticle.html +95 -0
- package/skills/mudra-master/mudra-xr/assets/samples/skybox_agent.html +384 -0
- package/skills/mudra-master/mudra-xr/assets/samples/sound.html +390 -0
- package/skills/mudra-master/mudra-xr/assets/samples/ui.html +496 -0
- package/skills/mudra-master/mudra-xr/assets/samples/virtual-screens.html +808 -0
- package/skills/mudra-master/mudra-xr/assets/templates/0_basic.html +64 -0
- package/skills/mudra-master/mudra-xr/assets/templates/1_ui.html +107 -0
- package/skills/mudra-master/mudra-xr/assets/templates/2_hands.html +183 -0
- package/skills/mudra-master/mudra-xr/assets/templates/3_depth.html +89 -0
- package/skills/mudra-master/mudra-xr/assets/templates/4_stereo.html +84 -0
- package/skills/mudra-master/mudra-xr/assets/templates/5_camera.html +117 -0
- package/skills/mudra-master/mudra-xr/assets/templates/6_ai.html +154 -0
- package/skills/mudra-master/mudra-xr/assets/templates/7_ai_live.html +253 -0
- package/skills/mudra-master/mudra-xr/assets/templates/8_objects.html +95 -0
- package/skills/mudra-master/mudra-xr/assets/templates/9_xr-toggle.html +90 -0
- package/skills/mudra-master/mudra-xr/assets/templates/heuristic_hand_gestures.html +102 -0
- package/skills/mudra-master/mudra-xr/assets/templates/meshes.html +47 -0
- package/skills/mudra-master/mudra-xr/assets/templates/planes.html +47 -0
- package/skills/mudra-master/mudra-xr/assets/templates/uikit.html +187 -0
- package/skills/mudra-master/mudra-xr/references/agent_protocol.json +878 -0
- package/skills/mudra-master/mudra-xr/references/promt.md +1516 -0
- package/skills/mudra-preview/SKILL.md +151 -0
- package/skills/mudra-preview/assets/ar-menu.html +382 -0
- package/skills/mudra-preview/assets/document-scroller.html +468 -0
- package/skills/mudra-preview/assets/drum-machine.html +788 -0
- package/skills/mudra-preview/assets/emg-visualizer.html +390 -0
- package/skills/mudra-preview/assets/generative-art.html +408 -0
- package/skills/mudra-preview/assets/gesture-assistant.html +342 -0
- package/skills/mudra-preview/assets/gesture-speech.html +444 -0
- package/skills/mudra-preview/assets/hands-free-desktop.html +376 -0
- package/skills/mudra-preview/assets/model-rotator.html +365 -0
- package/skills/mudra-preview/assets/mudra-duel.html +3200 -0
- package/skills/mudra-preview/assets/mudra-monitor.html +1451 -0
- package/skills/mudra-preview/assets/mudra-ultimate-template.html +891 -0
- package/skills/mudra-preview/assets/music-sequencer.html +455 -0
- package/skills/mudra-preview/assets/neural-pong.html +380 -0
- package/skills/mudra-preview/assets/neural-snake.html +364 -0
- package/skills/mudra-preview/assets/presentation-controller.html +345 -0
- package/skills/mudra-preview/assets/pressure-painter.html +285 -0
- package/skills/mudra-preview/assets/runner.html +1812 -0
- package/skills/mudra-preview/assets/smart-home.html +461 -0
- package/skills/mudra-preview/assets/space-invaders.html +1154 -0
- package/skills/mudra-preview/assets/waterful-ring-toss.html +1624 -0
- package/skills/mudra-preview/references/agent_protocol.json +646 -0
- package/skills/mudra-preview/references/promt.md +16829 -0
- package/skills/mudra-xr/SKILL.md +245 -0
- package/skills/mudra-xr/assets/demos/3dgs-walkthrough.html +242 -0
- package/skills/mudra-xr/assets/demos/aisimulator.html +792 -0
- package/skills/mudra-xr/assets/demos/balloonpop.html +422 -0
- package/skills/mudra-xr/assets/demos/ballpit.html +284 -0
- package/skills/mudra-xr/assets/demos/drone.html +41 -0
- package/skills/mudra-xr/assets/demos/gemini-icebreakers.html +338 -0
- package/skills/mudra-xr/assets/demos/gemini-xrobject.html +302 -0
- package/skills/mudra-xr/assets/demos/math3d.html +188 -0
- package/skills/mudra-xr/assets/demos/measure.html +247 -0
- package/skills/mudra-xr/assets/demos/occlusion.html +251 -0
- package/skills/mudra-xr/assets/demos/rain.html +498 -0
- package/skills/mudra-xr/assets/demos/screenwiper.html +432 -0
- package/skills/mudra-xr/assets/demos/splash.html +519 -0
- package/skills/mudra-xr/assets/demos/webcam_gestures.html +253 -0
- package/skills/mudra-xr/assets/demos/xremoji.html +833 -0
- package/skills/mudra-xr/assets/demos/xrpoet.html +151 -0
- package/skills/mudra-xr/assets/samples/depthmap.html +266 -0
- package/skills/mudra-xr/assets/samples/depthmesh.html +100 -0
- package/skills/mudra-xr/assets/samples/game_rps.html +1099 -0
- package/skills/mudra-xr/assets/samples/gestures_custom.html +467 -0
- package/skills/mudra-xr/assets/samples/gestures_heuristic.html +256 -0
- package/skills/mudra-xr/assets/samples/lighting.html +163 -0
- package/skills/mudra-xr/assets/samples/mesh_detection.html +46 -0
- package/skills/mudra-xr/assets/samples/modelviewer.html +183 -0
- package/skills/mudra-xr/assets/samples/paint.html +133 -0
- package/skills/mudra-xr/assets/samples/planar-vst.html +45 -0
- package/skills/mudra-xr/assets/samples/reticle.html +95 -0
- package/skills/mudra-xr/assets/samples/skybox_agent.html +384 -0
- package/skills/mudra-xr/assets/samples/sound.html +390 -0
- package/skills/mudra-xr/assets/samples/ui.html +496 -0
- package/skills/mudra-xr/assets/samples/virtual-screens.html +808 -0
- package/skills/mudra-xr/assets/templates/0_basic.html +64 -0
- package/skills/mudra-xr/assets/templates/1_ui.html +107 -0
- package/skills/mudra-xr/assets/templates/2_hands.html +183 -0
- package/skills/mudra-xr/assets/templates/3_depth.html +89 -0
- package/skills/mudra-xr/assets/templates/4_stereo.html +84 -0
- package/skills/mudra-xr/assets/templates/5_camera.html +117 -0
- package/skills/mudra-xr/assets/templates/6_ai.html +154 -0
- package/skills/mudra-xr/assets/templates/7_ai_live.html +253 -0
- package/skills/mudra-xr/assets/templates/8_objects.html +95 -0
- package/skills/mudra-xr/assets/templates/9_xr-toggle.html +90 -0
- package/skills/mudra-xr/assets/templates/heuristic_hand_gestures.html +102 -0
- package/skills/mudra-xr/assets/templates/meshes.html +47 -0
- package/skills/mudra-xr/assets/templates/planes.html +47 -0
- package/skills/mudra-xr/assets/templates/uikit.html +187 -0
- package/skills/mudra-xr/references/agent_protocol.json +878 -0
- package/skills/mudra-xr/references/promt.md +1516 -0
|
@@ -0,0 +1,1516 @@
|
|
|
1
|
+
# Mudra XR Skill — Build Rules
|
|
2
|
+
|
|
3
|
+
This document is the canonical reference for the `mudra-xr` Claude Code skill.
|
|
4
|
+
Read it in full before generating any app. Every rule below is non-negotiable
|
|
5
|
+
unless explicitly marked as configurable.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Section 1 — Mudra Protocol
|
|
10
|
+
|
|
11
|
+
### WebSocket endpoint
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
ws://127.0.0.1:8766
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Always construct the connection through `MudraClient` (Section 4).
|
|
18
|
+
Never use raw `new WebSocket(...)`.
|
|
19
|
+
|
|
20
|
+
### Nine canonical signals
|
|
21
|
+
|
|
22
|
+
| Signal | Category | Description |
|
|
23
|
+
|--------|----------|-------------|
|
|
24
|
+
| `gesture` | Discrete | Hand gesture events (tap, double_tap, twist, double_twist) |
|
|
25
|
+
| `button` | Discrete | Button hold / release |
|
|
26
|
+
| `pressure` | Analog | Finger pressure 0–100, normalized 0–1 |
|
|
27
|
+
| `navigation` | Motion (Pointer) | Continuous delta_x / delta_y cursor movement |
|
|
28
|
+
| `nav_direction` | Motion (Direction) | Discrete directional swipes: None, Right, Left, Up, Down, Roll Left, Roll Right |
|
|
29
|
+
| `imu_acc` | Motion (IMU) | Accelerometer values [x, y, z] m/s², frequency 1125 Hz |
|
|
30
|
+
| `imu_gyro` | Motion (IMU) | Gyroscope values [x, y, z] deg/s, frequency 1125 Hz |
|
|
31
|
+
| `snc` | Biometric | 3 de-interleaved channel arrays [[ch1], [ch2], [ch3]] |
|
|
32
|
+
| `battery` | Status | Battery level 0–100, charging boolean |
|
|
33
|
+
|
|
34
|
+
### Subscription handshake
|
|
35
|
+
|
|
36
|
+
Send one command per signal — never use plural `signals`, arrays, or batch commands:
|
|
37
|
+
|
|
38
|
+
```js
|
|
39
|
+
// CORRECT
|
|
40
|
+
ws.send(JSON.stringify({ command: 'subscribe', signal: 'gesture' }));
|
|
41
|
+
ws.send(JSON.stringify({ command: 'subscribe', signal: 'pressure' }));
|
|
42
|
+
|
|
43
|
+
// WRONG — never do this
|
|
44
|
+
ws.send(JSON.stringify({ command: 'subscribe', signals: ['gesture', 'pressure'] }));
|
|
45
|
+
ws.send(JSON.stringify({ command: 'subscribe', signal: ['gesture', 'pressure'] }));
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
### Full command surface
|
|
49
|
+
|
|
50
|
+
`subscribe`, `unsubscribe`, `get_subscriptions`, `enable`, `disable`,
|
|
51
|
+
`get_status`, `get_docs`, `trigger_gesture`
|
|
52
|
+
|
|
53
|
+
### Inbound message payload shapes
|
|
54
|
+
|
|
55
|
+
```js
|
|
56
|
+
// gesture
|
|
57
|
+
{ type: 'gesture', data: { type: 'tap'|'double_tap'|'twist'|'double_twist', confidence: 0–1, timestamp }, timestamp }
|
|
58
|
+
|
|
59
|
+
// button
|
|
60
|
+
{ type: 'button', data: { state: 'pressed'|'released', timestamp }, timestamp }
|
|
61
|
+
|
|
62
|
+
// pressure
|
|
63
|
+
{ type: 'pressure', data: { value: 0–100, normalized: 0–1, timestamp }, timestamp }
|
|
64
|
+
|
|
65
|
+
// navigation
|
|
66
|
+
{ type: 'navigation', data: { delta_x: number, delta_y: number, timestamp }, timestamp }
|
|
67
|
+
|
|
68
|
+
// nav_direction
|
|
69
|
+
{ type: 'nav_direction', data: { direction: 'Right'|'Left'|'Up'|'Down'|'Roll Left'|'Roll Right'|'None', timestamp }, timestamp }
|
|
70
|
+
|
|
71
|
+
// imu_acc
|
|
72
|
+
{ type: 'imu_acc', data: { values: [x, y, z], frequency: 1125, timestamp }, timestamp }
|
|
73
|
+
|
|
74
|
+
// imu_gyro
|
|
75
|
+
{ type: 'imu_gyro', data: { values: [x, y, z], frequency: 1125, timestamp }, timestamp }
|
|
76
|
+
|
|
77
|
+
// snc — extend rolling buffers (500 samples/channel) with all samples per callback
|
|
78
|
+
{ type: 'snc', data: { values: [[ch1_samples], [ch2_samples], [ch3_samples]], timestamp }, timestamp }
|
|
79
|
+
|
|
80
|
+
// battery
|
|
81
|
+
{ type: 'battery', data: { level: 0–100, charging: boolean, timestamp }, timestamp }
|
|
82
|
+
|
|
83
|
+
// connection_status
|
|
84
|
+
{ type: 'connection_status', data: { status: 'connected'|'disconnected', message: string }, timestamp }
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
## Section 2 — Canonical Dependency Pins
|
|
90
|
+
|
|
91
|
+
Use the exact versions below. Never use `@latest` or version ranges.
|
|
92
|
+
Import map `<script type="importmap">` must contain **only** the dependencies
|
|
93
|
+
the app actually uses — no unused entries.
|
|
94
|
+
|
|
95
|
+
```json
|
|
96
|
+
{
|
|
97
|
+
"imports": {
|
|
98
|
+
"three": "https://cdn.jsdelivr.net/npm/three@0.182.0/build/three.module.js",
|
|
99
|
+
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.182.0/examples/jsm/",
|
|
100
|
+
"troika-three-text": "https://cdn.jsdelivr.net/gh/protectwise/troika@028b81cf308f0f22e5aa8e78196be56ec1997af5/packages/troika-three-text/src/index.js",
|
|
101
|
+
"troika-three-utils": "https://cdn.jsdelivr.net/gh/protectwise/troika@v0.52.4/packages/troika-three-utils/src/index.js",
|
|
102
|
+
"troika-worker-utils": "https://cdn.jsdelivr.net/gh/protectwise/troika@v0.52.4/packages/troika-worker-utils/src/index.js",
|
|
103
|
+
"bidi-js": "https://esm.sh/bidi-js@%5E1.0.2?target=es2022",
|
|
104
|
+
"webgl-sdf-generator": "https://esm.sh/webgl-sdf-generator@1.1.1/es2022/webgl-sdf-generator.mjs",
|
|
105
|
+
"lit": "https://cdn.jsdelivr.net/gh/lit/dist@3/core/lit-core.min.js",
|
|
106
|
+
"lit/": "https://esm.run/lit@3/",
|
|
107
|
+
"xrblocks": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/xrblocks.js",
|
|
108
|
+
"xrblocks/addons/": "https://cdn.jsdelivr.net/npm/xrblocks@0.11.0/build/addons/"
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
XR Blocks stylesheet (always `<link>` in `<head>`):
|
|
114
|
+
|
|
115
|
+
```html
|
|
116
|
+
<link type="text/css" rel="stylesheet" href="https://xrblocks.github.io/css/xr.css" />
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Lit bundle warning suppression (place before import map):
|
|
120
|
+
|
|
121
|
+
```html
|
|
122
|
+
<script>window.litDisableBundleWarning = true;</script>
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## Section 3 — XR Blocks Lifecycle Composition
|
|
128
|
+
|
|
129
|
+
### Module-scope MudraClient
|
|
130
|
+
|
|
131
|
+
Instantiate `MudraClient` exactly once at module scope — not inside `init()`.
|
|
132
|
+
This ensures the WebSocket open-attempt starts immediately on page load so the
|
|
133
|
+
1500 ms timeout (Section 4) begins counting before the XR scene initializes.
|
|
134
|
+
|
|
135
|
+
```js
|
|
136
|
+
// Module scope — outside any class
|
|
137
|
+
const mudra = new MudraClient('ws://127.0.0.1:8766');
|
|
138
|
+
|
|
139
|
+
class MainScript extends xb.Script {
|
|
140
|
+
init() {
|
|
141
|
+
// Wire mudra handlers here after scene objects are created
|
|
142
|
+
mudra.on('gesture', (data) => { ... });
|
|
143
|
+
mudra.on('pressure', (data) => { ... });
|
|
144
|
+
}
|
|
145
|
+
update() { /* called every frame by xb */ }
|
|
146
|
+
}
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
### xb.Script class structure
|
|
150
|
+
|
|
151
|
+
Every generated app's top-level logic lives inside a class extending `xb.Script`.
|
|
152
|
+
The class name may be anything (`MainScript`, `CubeApp`, etc.) but must extend `xb.Script`.
|
|
153
|
+
|
|
154
|
+
```js
|
|
155
|
+
class MainScript extends xb.Script {
|
|
156
|
+
/** Called once after XR Blocks and the WebGL context are ready. */
|
|
157
|
+
init() {
|
|
158
|
+
// Add lights, meshes, Mudra bindings here.
|
|
159
|
+
this.add(new THREE.HemisphereLight(0xffffff, 0x666666, 3));
|
|
160
|
+
// Place objects at xb.user.objectDistance in front of viewer:
|
|
161
|
+
this.mesh.position.set(0, xb.user.height - 0.5, -xb.user.objectDistance);
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
/** Called every animation frame. Keep cheap — no allocations. */
|
|
165
|
+
update() { }
|
|
166
|
+
|
|
167
|
+
/** Fires when a pinch/controller-trigger starts in XR. */
|
|
168
|
+
onSelectStart(event) { }
|
|
169
|
+
|
|
170
|
+
/** Fires when a pinch/controller-trigger ends in XR. */
|
|
171
|
+
onSelectEnd(event) { }
|
|
172
|
+
|
|
173
|
+
/** Fires every frame while a pinch is held in XR. */
|
|
174
|
+
onSelecting(event) { }
|
|
175
|
+
}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Entry point
|
|
179
|
+
|
|
180
|
+
```js
|
|
181
|
+
document.addEventListener('DOMContentLoaded', function () {
|
|
182
|
+
xb.add(new MainScript());
|
|
183
|
+
xb.init(new xb.Options());
|
|
184
|
+
});
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
### Key spatial constants
|
|
188
|
+
|
|
189
|
+
- `xb.user.height` — floor-relative eye height (approx. 1.6 m)
|
|
190
|
+
- `xb.user.objectDistance` — recommended arm-length distance for objects (approx. 0.8 m)
|
|
191
|
+
- Y-up coordinate system; Z is toward the viewer (negative Z = in front of you)
|
|
192
|
+
|
|
193
|
+
---
|
|
194
|
+
|
|
195
|
+
## Section 4 — Mock WebSocket Fallback (MudraClient)
|
|
196
|
+
|
|
197
|
+
### Policy
|
|
198
|
+
|
|
199
|
+
- Start the WebSocket open attempt immediately on page load.
|
|
200
|
+
- If the WebSocket does not open within **1500 ms**, activate the mock automatically.
|
|
201
|
+
- If the WebSocket closes mid-session (band disconnect), flip to mock without a page reload.
|
|
202
|
+
- The mock fires exactly the same message format as the real device.
|
|
203
|
+
- App code must not branch on `_useMock` — it must receive the same messages either way.
|
|
204
|
+
|
|
205
|
+
### Connection-status state machine
|
|
206
|
+
|
|
207
|
+
```
|
|
208
|
+
[page load]
|
|
209
|
+
│
|
|
210
|
+
▼
|
|
211
|
+
connecting ──── ws opens within 1500 ms ──→ connected
|
|
212
|
+
│
|
|
213
|
+
└── timeout or error ──────────────────→ simulated
|
|
214
|
+
│
|
|
215
|
+
connected ──── ws.onclose fires ───────────────→ disconnected-simulated
|
|
216
|
+
simulated ──── ws.onclose fires ───────────────→ (already simulated, no change)
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
### Required MudraClient implementation
|
|
220
|
+
|
|
221
|
+
Include this class verbatim in every generated app:
|
|
222
|
+
|
|
223
|
+
```js
|
|
224
|
+
class MudraClient {
|
|
225
|
+
constructor(url) {
|
|
226
|
+
this._handlers = {};
|
|
227
|
+
this._subscriptions = new Set();
|
|
228
|
+
this._timers = [];
|
|
229
|
+
this._status = 'connecting';
|
|
230
|
+
this._notifyStatus('connecting');
|
|
231
|
+
|
|
232
|
+
const timeout = setTimeout(() => this._startMock(), 1500);
|
|
233
|
+
|
|
234
|
+
try {
|
|
235
|
+
this._ws = new WebSocket(url);
|
|
236
|
+
this._ws.onopen = () => {
|
|
237
|
+
clearTimeout(timeout);
|
|
238
|
+
this._status = 'connected';
|
|
239
|
+
this._notifyStatus('connected');
|
|
240
|
+
this._subscriptions.forEach(sig =>
|
|
241
|
+
this._ws.send(JSON.stringify({ command: 'subscribe', signal: sig }))
|
|
242
|
+
);
|
|
243
|
+
};
|
|
244
|
+
this._ws.onmessage = (e) => {
|
|
245
|
+
const msg = JSON.parse(e.data);
|
|
246
|
+
if (this._handlers[msg.type]) this._handlers[msg.type](msg.data);
|
|
247
|
+
};
|
|
248
|
+
this._ws.onclose = () => {
|
|
249
|
+
clearTimeout(timeout);
|
|
250
|
+
if (this._status === 'connected') {
|
|
251
|
+
this._status = 'disconnected-simulated';
|
|
252
|
+
this._notifyStatus('disconnected-simulated');
|
|
253
|
+
this._startMock();
|
|
254
|
+
}
|
|
255
|
+
};
|
|
256
|
+
this._ws.onerror = () => {
|
|
257
|
+
clearTimeout(timeout);
|
|
258
|
+
this._startMock();
|
|
259
|
+
};
|
|
260
|
+
} catch (_) {
|
|
261
|
+
clearTimeout(timeout);
|
|
262
|
+
this._startMock();
|
|
263
|
+
}
|
|
264
|
+
}
|
|
265
|
+
|
|
266
|
+
/** Register a handler for a signal type. Call before subscribe(). */
|
|
267
|
+
on(signal, fn) {
|
|
268
|
+
this._handlers[signal] = fn;
|
|
269
|
+
}
|
|
270
|
+
|
|
271
|
+
/** Subscribe to a signal. Safe to call before the WebSocket opens. */
|
|
272
|
+
subscribe(signal) {
|
|
273
|
+
this._subscriptions.add(signal);
|
|
274
|
+
if (this._ws && this._ws.readyState === WebSocket.OPEN) {
|
|
275
|
+
this._ws.send(JSON.stringify({ command: 'subscribe', signal }));
|
|
276
|
+
}
|
|
277
|
+
}
|
|
278
|
+
|
|
279
|
+
/** Send an arbitrary command to the band service. */
|
|
280
|
+
send(cmd) {
|
|
281
|
+
if (this._ws && this._ws.readyState === WebSocket.OPEN) {
|
|
282
|
+
this._ws.send(JSON.stringify(cmd));
|
|
283
|
+
} else if (cmd.command === 'trigger_gesture') {
|
|
284
|
+
this._dispatchMockGesture(cmd.data.type);
|
|
285
|
+
}
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
/** Current connection status string. */
|
|
289
|
+
get status() { return this._status; }
|
|
290
|
+
|
|
291
|
+
_notifyStatus(s) {
|
|
292
|
+
if (this._handlers['_status']) this._handlers['_status'](s);
|
|
293
|
+
}
|
|
294
|
+
|
|
295
|
+
_emit(payload) {
|
|
296
|
+
if (this._handlers[payload.type]) this._handlers[payload.type](payload.data);
|
|
297
|
+
}
|
|
298
|
+
|
|
299
|
+
_dispatchMockGesture(type) {
|
|
300
|
+
this._emit({ type: 'gesture', data: { type, confidence: 0.99, timestamp: Date.now() } });
|
|
301
|
+
}
|
|
302
|
+
|
|
303
|
+
_startMock() {
|
|
304
|
+
if (this._status === 'simulated' || this._status === 'disconnected-simulated') return;
|
|
305
|
+
const wasPreviouslyConnected = this._status === 'connected';
|
|
306
|
+
this._status = wasPreviouslyConnected ? 'disconnected-simulated' : 'simulated';
|
|
307
|
+
this._notifyStatus(this._status);
|
|
308
|
+
// Passive mock: no auto-firing. Signals fire ONLY from sim-panel clicks,
|
|
309
|
+
// keyboard shortcuts, or real WebSocket messages. This keeps the scene
|
|
310
|
+
// stable and makes the sim panel the single, explicit source of motion.
|
|
311
|
+
}
|
|
312
|
+
|
|
313
|
+
destroy() {
|
|
314
|
+
this._timers.forEach(t => clearInterval(t));
|
|
315
|
+
if (this._ws) this._ws.close();
|
|
316
|
+
}
|
|
317
|
+
}
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
---
|
|
321
|
+
|
|
322
|
+
## Section 5 — Simulator Panel
|
|
323
|
+
|
|
324
|
+
### Purpose
|
|
325
|
+
|
|
326
|
+
The simulator panel lets a user exercise every subscribed Mudra signal without
|
|
327
|
+
a band or XR headset. It is a **2D DOM overlay** (not spatial XR UI) and must
|
|
328
|
+
always be visible in flat-screen mode. It disappears automatically when the
|
|
329
|
+
browser transitions to an immersive WebXR session (native WebXR DOM suppression).
|
|
330
|
+
|
|
331
|
+
### DOM structure
|
|
332
|
+
|
|
333
|
+
```html
|
|
334
|
+
<div id="mudra-sim" style="
|
|
335
|
+
position: fixed; bottom: 0; left: 0; right: 0;
|
|
336
|
+
background: rgba(0,0,0,0.75); backdrop-filter: blur(4px);
|
|
337
|
+
padding: 8px 12px; display: flex; flex-wrap: wrap; gap: 8px;
|
|
338
|
+
z-index: 9999; font-family: system-ui, sans-serif;">
|
|
339
|
+
<!-- One button group per subscribed signal -->
|
|
340
|
+
</div>
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
### Button groups per signal (render ONLY the sub-actions the app actually uses)
|
|
344
|
+
|
|
345
|
+
The catalog below is the **maximal** set per signal. The simulator panel
|
|
346
|
+
MUST render **only** the sub-actions the generated app handles — never
|
|
347
|
+
extras. Examples:
|
|
348
|
+
|
|
349
|
+
- App maps `gesture.tap` only → render `Tap`. Omit `2Tap`, `Twist`, `2Twist`.
|
|
350
|
+
- App maps `nav_direction` to Up/Down only → render `↑`, `↓`. Omit
|
|
351
|
+
`←`, `→`, `Roll L`, `Roll R`.
|
|
352
|
+
- App uses `imu_acc` for X-axis tilt only → render `Tilt X+`, `Tilt X−`.
|
|
353
|
+
Omit `Tilt Y+`, `Tilt Y−`.
|
|
354
|
+
|
|
355
|
+
When in doubt, walk every `mudra.on('<signal>', …)` handler and emit a
|
|
356
|
+
button only for sub-actions referenced inside it. Unused buttons are a
|
|
357
|
+
checklist failure (see Section 10, item 6).
|
|
358
|
+
|
|
359
|
+
| Signal | Maximal buttons (subset based on app handlers) |
|
|
360
|
+
|--------|---------|
|
|
361
|
+
| `gesture` | `Tap`, `2Tap`, `Twist`, `2Twist` |
|
|
362
|
+
| `nav_direction` | `↑`, `↓`, `←`, `→`, `Roll L`, `Roll R` |
|
|
363
|
+
| `navigation` | `↑`, `↓`, `←`, `→` (each emits one delta event of ±3 — gentle/low-sensitivity default; raise per-app only if the prompt explicitly asks for fast/snappy movement) |
|
|
364
|
+
| `pressure` | Slider `0–100` (label shows current value) |
|
|
365
|
+
| `button` | `Press`, `Release` |
|
|
366
|
+
| `imu_acc` | `Tilt X+`, `Tilt X−`, `Tilt Y+`, `Tilt Y−` (5-frame burst at ±2 m/s²) |
|
|
367
|
+
| `imu_gyro` | `Rot X+`, `Rot X−`, `Rot Y+`, `Rot Y−` (5-frame burst at ±10 deg/s) |
|
|
368
|
+
| `snc` | `Spike` (burst of elevated samples on all 3 channels) |
|
|
369
|
+
|
|
370
|
+
### Button firing rules
|
|
371
|
+
|
|
372
|
+
- Every button fires via the **same code path** as a real Mudra signal.
|
|
373
|
+
Call the same handler that `mudra.on(signal, handler)` would invoke.
|
|
374
|
+
- For `gesture` buttons: also call `mudra.send({ command: 'trigger_gesture', data: { type } })`
|
|
375
|
+
so the round-trip works when a real band is connected.
|
|
376
|
+
- Never duplicate logic between the simulator path and the real-signal path.
|
|
377
|
+
|
|
378
|
+
```js
|
|
379
|
+
// Example: gesture sim button wires to the same handler as real signals
|
|
380
|
+
function simGesture(type) {
|
|
381
|
+
mudra.send({ command: 'trigger_gesture', data: { type } });
|
|
382
|
+
handleGesture({ type, confidence: 1.0, timestamp: Date.now() });
|
|
383
|
+
}
|
|
384
|
+
|
|
385
|
+
// Example: pressure slider
|
|
386
|
+
pressureSlider.addEventListener('input', () => {
|
|
387
|
+
const norm = pressureSlider.value / 100;
|
|
388
|
+
handlePressure({ value: +pressureSlider.value, normalized: norm, timestamp: Date.now() });
|
|
389
|
+
});
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
### Visibility
|
|
393
|
+
|
|
394
|
+
- `#mudra-sim` is always visible in flat-screen mode.
|
|
395
|
+
- Do NOT conditionally hide it when the band is connected (both sources are valid per FR-014).
|
|
396
|
+
- The 2D DOM disappears automatically in immersive WebXR — no JS required.
|
|
397
|
+
|
|
398
|
+
---
|
|
399
|
+
|
|
400
|
+
## Section 6 — Keyboard Shortcuts + XR Blocks Precedence
|
|
401
|
+
|
|
402
|
+
### Canonical keyboard map
|
|
403
|
+
|
|
404
|
+
| Key | Signal / Action |
|
|
405
|
+
|-----|----------------|
|
|
406
|
+
| `Space` | `gesture` → tap |
|
|
407
|
+
| `Shift` | `button` → press (keydown) / release (keyup) |
|
|
408
|
+
| `[` | `pressure` decrease (−10, min 0) |
|
|
409
|
+
| `]` | `pressure` increase (+10, max 100) |
|
|
410
|
+
| `i` | `nav_direction` → Up OR `navigation` delta_y +3 |
|
|
411
|
+
| `k` | `nav_direction` → Down OR `navigation` delta_y −3 |
|
|
412
|
+
| `j` | `nav_direction` → Left OR `navigation` delta_x −3 |
|
|
413
|
+
| `l` | `nav_direction` → Right OR `navigation` delta_x +3 |
|
|
414
|
+
|
|
415
|
+
**Navigation sensitivity default**: keyboard `I`/`J`/`K`/`L` and sim panel
|
|
416
|
+
buttons emit deltas of **±3** per event (gentle / low-sensitivity
|
|
417
|
+
baseline). This keeps cursor/pan motion calm and predictable. Only raise
|
|
418
|
+
the magnitude when the prompt explicitly calls for fast/snappy movement
|
|
419
|
+
(racing, arcade-twitch, etc.).
|
|
420
|
+
| `u` | `imu_acc` tilt X+ burst |
|
|
421
|
+
| `o` | `imu_acc` tilt X− burst |
|
|
422
|
+
| `m` | `imu_gyro` rot Y+ burst |
|
|
423
|
+
| `n` | `imu_gyro` rot Y− burst |
|
|
424
|
+
|
|
425
|
+
For `imu` bursts: fire 5 synthetic frames at ±[2, 0, 9.81] m/s² (acc) or ±[10, 0, 0.5] deg/s (gyro).
|
|
426
|
+
|
|
427
|
+
### Reserved keys & mouse — owned by XR Blocks desktop simulator
|
|
428
|
+
|
|
429
|
+
The XR Blocks desktop simulator (active whenever the app runs in a flat
|
|
430
|
+
browser without a WebXR session) owns the keys and mouse gestures that
|
|
431
|
+
let the user navigate the 3D scene. Mudra MUST NOT claim or
|
|
432
|
+
`stopPropagation()` any of these — they are how the user orbits, walks,
|
|
433
|
+
and zooms while previewing the app:
|
|
434
|
+
|
|
435
|
+
| Input | XR Blocks role |
|
|
436
|
+
|-------|----------------|
|
|
437
|
+
| `W` `A` `S` `D` | Walk camera forward / left / back / right |
|
|
438
|
+
| `ArrowUp` `ArrowDown` `ArrowLeft` `ArrowRight` | Camera nav (alt to WASD) |
|
|
439
|
+
| `Q` `E` | Camera roll / vertical |
|
|
440
|
+
| `R` | Reset camera pose |
|
|
441
|
+
| Right-click drag | Orbit / look around |
|
|
442
|
+
| Mouse wheel | Zoom in / out |
|
|
443
|
+
|
|
444
|
+
**Why Mudra uses `I`/`J`/`K`/`L` and `U`/`O`/`M`/`N`**: these keys are
|
|
445
|
+
explicitly off the XR Blocks reserved set, so the desktop simulator
|
|
446
|
+
keeps full camera control while the band-driven keyboard shortcuts
|
|
447
|
+
remain available for testing without a band. Never reassign a Mudra
|
|
448
|
+
shortcut onto a reserved key, even when the app does not subscribe to a
|
|
449
|
+
navigation signal.
|
|
450
|
+
|
|
451
|
+
### Attachment rule (critical)
|
|
452
|
+
|
|
453
|
+
Mudra keyboard handlers MUST attach with `capture: true` and call
|
|
454
|
+
`event.stopPropagation()` on every key the app subscribes to. This prevents
|
|
455
|
+
XR Blocks' desktop-simulator bubble-phase listeners from double-firing.
|
|
456
|
+
|
|
457
|
+
```js
|
|
458
|
+
window.addEventListener('keydown', (e) => {
|
|
459
|
+
switch (e.code) {
|
|
460
|
+
case 'Space':
|
|
461
|
+
e.stopPropagation();
|
|
462
|
+
simGesture('tap');
|
|
463
|
+
break;
|
|
464
|
+
case 'BracketLeft':
|
|
465
|
+
e.stopPropagation();
|
|
466
|
+
adjustPressure(-10);
|
|
467
|
+
break;
|
|
468
|
+
case 'BracketRight':
|
|
469
|
+
e.stopPropagation();
|
|
470
|
+
adjustPressure(+10);
|
|
471
|
+
break;
|
|
472
|
+
case 'KeyI':
|
|
473
|
+
e.stopPropagation();
|
|
474
|
+
handleNavDirection({ direction: 'Up', timestamp: Date.now() });
|
|
475
|
+
break;
|
|
476
|
+
// ... etc for subscribed keys only — using I/J/K/L for nav and U/O/M/N for IMU
|
|
477
|
+
}
|
|
478
|
+
}, { capture: true });
|
|
479
|
+
```
|
|
480
|
+
|
|
481
|
+
Only intercept keys for signals the app actually subscribes to. Do NOT
|
|
482
|
+
`stopPropagation` on keys that no Mudra signal handles — XR Blocks needs
|
|
483
|
+
those, especially the reserved set above (WASD, arrows, Q/E/R, mouse).
|
|
484
|
+
|
|
485
|
+
---
|
|
486
|
+
|
|
487
|
+
## Section 7 — Connection-Status Indicator
|
|
488
|
+
|
|
489
|
+
### Required DOM element
|
|
490
|
+
|
|
491
|
+
Every generated app must include exactly one visible element that reflects
|
|
492
|
+
the current `MudraClient` status:
|
|
493
|
+
|
|
494
|
+
```html
|
|
495
|
+
<div id="mudra-status" style="
|
|
496
|
+
position: fixed; top: 8px; right: 12px;
|
|
497
|
+
padding: 4px 10px; border-radius: 999px;
|
|
498
|
+
font-size: 0.8rem; font-family: system-ui, sans-serif;
|
|
499
|
+
background: rgba(0,0,0,0.6); color: #fff;
|
|
500
|
+
z-index: 9999;">Connecting…</div>
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
### Text states
|
|
504
|
+
|
|
505
|
+
| MudraClient status | textContent |
|
|
506
|
+
|-------------------|-------------|
|
|
507
|
+
| `connecting` | `Connecting…` |
|
|
508
|
+
| `connected` | `Connected` |
|
|
509
|
+
| `simulated` | `Simulated` |
|
|
510
|
+
| `disconnected-simulated` | `Disconnected — simulated` |
|
|
511
|
+
|
|
512
|
+
### Wiring
|
|
513
|
+
|
|
514
|
+
```js
|
|
515
|
+
mudra.on('_status', (s) => {
|
|
516
|
+
const chip = document.getElementById('mudra-status');
|
|
517
|
+
const labels = {
|
|
518
|
+
'connecting': 'Connecting…',
|
|
519
|
+
'connected': 'Connected',
|
|
520
|
+
'simulated': 'Simulated',
|
|
521
|
+
'disconnected-simulated': 'Disconnected — simulated',
|
|
522
|
+
};
|
|
523
|
+
chip.textContent = labels[s] ?? s;
|
|
524
|
+
});
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
### Visibility
|
|
528
|
+
|
|
529
|
+
- Visible in flat-screen mode at all times.
|
|
530
|
+
- Disappears automatically in immersive XR (native DOM suppression).
|
|
531
|
+
- Place in the top-right corner by default; adapt if the template uses that space.
|
|
532
|
+
|
|
533
|
+
---
|
|
534
|
+
|
|
535
|
+
## Section 8 — Signal Grouping Rules (Non-Negotiable)
|
|
536
|
+
|
|
537
|
+
### Signal group classification
|
|
538
|
+
|
|
539
|
+
| Group | Signals | Combine freely with |
|
|
540
|
+
|-------|---------|---------------------|
|
|
541
|
+
| **Pointer** | `navigation`, `button` | `gesture` OR `button` (not `pressure` if `gesture` is used) |
|
|
542
|
+
| **Direction** | `nav_direction` | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
|
|
543
|
+
| **IMU+Biometric** | `imu_acc`, `imu_gyro`, `snc` | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
|
|
544
|
+
| *(none)* | — | `gesture` OR `pressure` OR `button` (but not `gesture`+`pressure` together) |
|
|
545
|
+
|
|
546
|
+
### Bundling rule — IMU+Biometric (CRITICAL)
|
|
547
|
+
|
|
548
|
+
`imu_acc`, `imu_gyro`, and `snc` are an **inseparable bundle**. If the user's prompt
|
|
549
|
+
implies any one of them, subscribe to **all three**. Never subscribe to only one or
|
|
550
|
+
two of them.
|
|
551
|
+
|
|
552
|
+
```js
|
|
553
|
+
// CORRECT — all three always together
|
|
554
|
+
mudra.subscribe('imu_acc');
|
|
555
|
+
mudra.subscribe('imu_gyro');
|
|
556
|
+
mudra.subscribe('snc');
|
|
557
|
+
|
|
558
|
+
// WRONG — partial subscriptions
|
|
559
|
+
mudra.subscribe('snc'); // missing imu_acc and imu_gyro
|
|
560
|
+
mudra.subscribe('imu_acc'); // missing imu_gyro and snc
|
|
561
|
+
```
|
|
562
|
+
|
|
563
|
+
### XOR rules (all non-negotiable)
|
|
564
|
+
|
|
565
|
+
1. **Gesture ⊕ Pressure** — an app may use `gesture` OR `pressure`, never both.
|
|
566
|
+
2. **Navigation ⊕ Nav_direction** — an app may use `navigation` OR `nav_direction`, never both.
|
|
567
|
+
3. **Pointer/Direction ⊕ IMU+Biometric** — `navigation` and `nav_direction` cannot be combined with the IMU+Biometric bundle (`imu_acc`/`imu_gyro`/`snc`).
|
|
568
|
+
|
|
569
|
+
### Illegal combinations
|
|
570
|
+
|
|
571
|
+
```
|
|
572
|
+
// REJECT these signal sets
|
|
573
|
+
gesture + pressure
|
|
574
|
+
navigation + nav_direction
|
|
575
|
+
navigation + imu_acc
|
|
576
|
+
navigation + imu_gyro
|
|
577
|
+
navigation + snc
|
|
578
|
+
nav_direction + imu_acc
|
|
579
|
+
nav_direction + imu_gyro
|
|
580
|
+
nav_direction + snc
|
|
581
|
+
button + nav_direction // button belongs to Pointer mode only
|
|
582
|
+
```
|
|
583
|
+
|
|
584
|
+
### Valid signal sets (examples)
|
|
585
|
+
|
|
586
|
+
```
|
|
587
|
+
gesture + button
|
|
588
|
+
pressure + button
|
|
589
|
+
navigation + button
|
|
590
|
+
navigation + button + gesture
|
|
591
|
+
nav_direction
|
|
592
|
+
nav_direction + pressure + button
|
|
593
|
+
imu_acc + imu_gyro + snc
|
|
594
|
+
imu_acc + imu_gyro + snc + gesture
|
|
595
|
+
imu_acc + imu_gyro + snc + button
|
|
596
|
+
imu_acc + imu_gyro + snc + pressure + button
|
|
597
|
+
```
|
|
598
|
+
|
|
599
|
+
### Inference priority for ties
|
|
600
|
+
|
|
601
|
+
When the user prompt maps to multiple modes equally, ask one disambiguation
|
|
602
|
+
question — do not silently pick one.
|
|
603
|
+
|
|
604
|
+
Default to `direction` mode (`nav_direction`) when there is no navigation
|
|
605
|
+
language in the prompt at all and a motion mode is required by the template.
|
|
606
|
+
|
|
607
|
+
---
|
|
608
|
+
|
|
609
|
+
## Section 9 — AI API Key Handling (onboarding-gated, mandatory for AI apps)
|
|
610
|
+
|
|
611
|
+
### Lifecycle: `sessionStorage` keyed by `mudra.gemini.apiKey`
|
|
612
|
+
|
|
613
|
+
For any generated app that calls a Gemini / LLM endpoint, the API key MUST
|
|
614
|
+
be entered through the **onboarding modal** before the user can use the
|
|
615
|
+
app. The key is stored in `sessionStorage` under the literal key
|
|
616
|
+
`mudra.gemini.apiKey` — it persists across reloads in the same tab and
|
|
617
|
+
clears when the tab closes. **Do NOT use `localStorage`. Do NOT use a
|
|
618
|
+
`prompt()` popup. Do NOT prompt on first AI call.**
|
|
619
|
+
|
|
620
|
+
```js
|
|
621
|
+
// Read at app start
|
|
622
|
+
const apiKey = sessionStorage.getItem('mudra.gemini.apiKey');
|
|
623
|
+
|
|
624
|
+
// Write only from the onboarding "AI Setup" step
|
|
625
|
+
sessionStorage.setItem('mudra.gemini.apiKey', enteredKey);
|
|
626
|
+
```
|
|
627
|
+
|
|
628
|
+
### Required onboarding "AI Setup" fragment
|
|
629
|
+
|
|
630
|
+
For AI apps only, the onboarding modal MUST include the following
|
|
631
|
+
fragment inside `<section class="mudra-onb__body">`, placed AFTER the
|
|
632
|
+
actions table and BEFORE `</section>`:
|
|
633
|
+
|
|
634
|
+
```html
|
|
635
|
+
<div class="mudra-onb__ai" data-uses-ai>
|
|
636
|
+
<h3 class="mudra-onb__ai-title">AI Setup</h3>
|
|
637
|
+
<p class="mudra-onb__ai-lede">
|
|
638
|
+
This app uses Google Gemini. Paste your API key to continue —
|
|
639
|
+
it's stored only in this browser tab (<code>sessionStorage</code>)
|
|
640
|
+
and is never sent anywhere except Google's API.
|
|
641
|
+
<a href="https://aistudio.google.com/" target="_blank" rel="noopener">Get a key →</a>
|
|
642
|
+
</p>
|
|
643
|
+
<input
|
|
644
|
+
id="mudra-onb-ai-key"
|
|
645
|
+
class="mudra-onb__ai-input"
|
|
646
|
+
type="password"
|
|
647
|
+
autocomplete="off"
|
|
648
|
+
spellcheck="false"
|
|
649
|
+
placeholder="Paste Gemini API key (starts with AIza…)"
|
|
650
|
+
aria-label="Gemini API key"
|
|
651
|
+
/>
|
|
652
|
+
<p class="mudra-onb__ai-hint" data-role="hint"></p>
|
|
653
|
+
</div>
|
|
654
|
+
```
|
|
655
|
+
|
|
656
|
+
And the matching CSS (added to the existing `<style>`):
|
|
657
|
+
|
|
658
|
+
```css
|
|
659
|
+
.mudra-onb__ai { margin-top: 14px; padding-top: 12px; border-top: 1px solid #eee; }
|
|
660
|
+
.mudra-onb__ai-title { margin: 0 0 6px; font-size: 0.95rem; font-weight: 700; color: #111; }
|
|
661
|
+
.mudra-onb__ai-lede { margin: 0 0 10px; color: #555; font-size: 0.85rem; }
|
|
662
|
+
.mudra-onb__ai-lede a { color: #0d9488; }
|
|
663
|
+
.mudra-onb__ai-input {
|
|
664
|
+
width: 100%; box-sizing: border-box;
|
|
665
|
+
padding: 9px 12px; border: 1px solid #d0d0d0; border-radius: 8px;
|
|
666
|
+
font: 13px/1.4 ui-monospace, SFMono-Regular, Menlo, monospace;
|
|
667
|
+
background: #f8f9fb; color: #111;
|
|
668
|
+
}
|
|
669
|
+
.mudra-onb__ai-input:focus { outline: 2px solid #14b8a6; border-color: #14b8a6; background: #fff; }
|
|
670
|
+
.mudra-onb__ai-hint { margin: 6px 2px 0; font-size: 0.75rem; color: #b91c1c; min-height: 1em; }
|
|
671
|
+
.mudra-onb__continue:disabled { opacity: 0.45; cursor: not-allowed; }
|
|
672
|
+
```
|
|
673
|
+
|
|
674
|
+
### Behaviour rules (AI apps only)
|
|
675
|
+
|
|
676
|
+
1. **`Got it` button starts disabled.** The IIFE that wires the modal
|
|
677
|
+
reads `dialog.querySelector('.mudra-onb__ai')` — if it exists, the
|
|
678
|
+
`.mudra-onb__continue` button is disabled until
|
|
679
|
+
`.mudra-onb__ai-input` is non-empty AND matches `/^AIza[\w-]{30,}$/`
|
|
680
|
+
(Google API-key prefix sanity check).
|
|
681
|
+
2. **On click of `Got it`**, write the trimmed value to
|
|
682
|
+
`sessionStorage.setItem('mudra.gemini.apiKey', value)`, then close
|
|
683
|
+
the modal.
|
|
684
|
+
3. **At every page load**, the IIFE checks `sessionStorage` first:
|
|
685
|
+
- If `mudra.gemini.apiKey` is present and matches the prefix regex,
|
|
686
|
+
the AI-Setup fragment is hidden (the user already provided a key
|
|
687
|
+
this session) and `Got it` is enabled immediately.
|
|
688
|
+
- If absent or malformed, the AI-Setup fragment is shown and the
|
|
689
|
+
modal CANNOT be dismissed by `Escape`, the `×` close button, or
|
|
690
|
+
the reopen `?` button without entering a valid key. `dialog.close()`
|
|
691
|
+
paths called from those handlers are no-ops while the key is
|
|
692
|
+
missing.
|
|
693
|
+
4. **The `?` reopen button** for AI apps re-runs the gating logic. If
|
|
694
|
+
the user clears `sessionStorage` mid-session and reopens, the
|
|
695
|
+
AI-Setup fragment renders again.
|
|
696
|
+
5. **Reading the key in app code:** the `xb.Script` subclass reads
|
|
697
|
+
`sessionStorage.getItem('mudra.gemini.apiKey')` in `init()`. If the
|
|
698
|
+
key is `null`, the AI portion of the app stays inert (no calls to
|
|
699
|
+
Gemini) and the visible chat panel (Section 18) renders a `Set up
|
|
700
|
+
AI in the welcome panel` placeholder.
|
|
701
|
+
6. **Never bake a key into the HTML source.** Pre-write regex scan
|
|
702
|
+
(`/AIza[A-Za-z0-9_-]{30,}|sk-[A-Za-z0-9_-]{32,}/` excluding the
|
|
703
|
+
literal placeholder string `AIza…`) must return zero matches.
|
|
704
|
+
7. **Never auto-read from URL params, `localStorage`, or `prompt()`.**
|
|
705
|
+
8. **Non-AI apps** ignore this section entirely. The AI-Setup fragment
|
|
706
|
+
is omitted; the modal works as defined in Section 17 unchanged.
|
|
707
|
+
|
|
708
|
+
### Canonical Gemini model — `gemini-2.5-flash` only
|
|
709
|
+
|
|
710
|
+
For any generated app that calls Gemini via the **REST `generateContent`
|
|
711
|
+
endpoint**, the model ID MUST be exactly `gemini-2.5-flash`. No other
|
|
712
|
+
model IDs are permitted for REST text/chat/vision generation. This is a
|
|
713
|
+
hard pin — preview aliases get retired by Google and the app then 404s.
|
|
714
|
+
|
|
715
|
+
```js
|
|
716
|
+
// CORRECT — the only permitted REST model for text/chat/vision
|
|
717
|
+
const url = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=${encodeURIComponent(apiKey)}`;
|
|
718
|
+
|
|
719
|
+
// FORBIDDEN — preview aliases, dated aliases, retired families
|
|
720
|
+
gemini-1.5-flash, gemini-1.5-flash-latest, gemini-1.5-pro
|
|
721
|
+
gemini-2.5-flash-preview-09-2025, gemini-2.5-flash-preview-04-2025
|
|
722
|
+
gemini-flash-latest, gemini-pro, gemini-2.5-flash-002 (any -NNN suffix)
|
|
723
|
+
```
|
|
724
|
+
|
|
725
|
+
Pre-write regex scan — every generated HTML file MUST satisfy:
|
|
726
|
+
|
|
727
|
+
- `/generativelanguage\.googleapis\.com\/v1beta\/models\/([a-z0-9-]+):generateContent/`
|
|
728
|
+
→ captured model ID MUST equal `gemini-2.5-flash`. Any other capture
|
|
729
|
+
fails the pre-write checklist.
|
|
730
|
+
|
|
731
|
+
Out-of-scope use cases (allowed exceptions, REST-only rule does NOT apply):
|
|
732
|
+
|
|
733
|
+
- **Live API** (WebSocket / xrblocks `xb.core.ai.startLiveSession()`):
|
|
734
|
+
the Live endpoint uses its own model set (`gemini-2.0-flash-live-001`,
|
|
735
|
+
`gemini-2.5-flash-native-audio-preview-12-2025`). Apps that genuinely
|
|
736
|
+
need streaming audio/video may use those, but every text-chat app
|
|
737
|
+
must use the REST pin above.
|
|
738
|
+
- **Image generation** via `:generateContent`: `gemini-2.5-flash-image`
|
|
739
|
+
is permitted only when the app's purpose is image output. Default to
|
|
740
|
+
the text pin otherwise.
|
|
741
|
+
|
|
742
|
+
If an app needs a different model, raise it to the user before writing —
|
|
743
|
+
do not silently swap in a preview alias.
|
|
744
|
+
|
|
745
|
+
---
|
|
746
|
+
|
|
747
|
+
## Section 10 — Pre-Write Checklist + Collision Handling
|
|
748
|
+
|
|
749
|
+
Before calling `Write` to emit a generated app, verify all items:
|
|
750
|
+
|
|
751
|
+
| # | Check | Pass condition |
|
|
752
|
+
|---|-------|----------------|
|
|
753
|
+
| 1 | Single file | Exactly one `<html>` document; all CSS in `<style>`; all JS in `<script>` or `<script type="module">` |
|
|
754
|
+
| 2 | Import map | One `<script type="importmap">` block; contents match canonical pins (Section 2) exactly; no unused entries |
|
|
755
|
+
| 3 | xb.Script entry | Top-level logic inside `class <Name> extends xb.Script`; `xb.add(new <Name>())` + `xb.init(new xb.Options())` on `DOMContentLoaded` |
|
|
756
|
+
| 4 | MudraClient | One `MudraClient` instance at module scope; URL = `ws://127.0.0.1:8766`; does NOT auto-connect; `setMode()` drives connect/disconnect |
|
|
757
|
+
| 5 | Subscribe commands | Every used signal has exactly one `mudra.subscribe('<signal>')` call; none outside the signal set |
|
|
758
|
+
| 6 | Simulator panel | `<div id="mudra-sim">` present; ONLY buttons for sub-actions actually handled by the app (no extras like Roll L/R or Twist if unused); buttons fire via handler, not inline `onclick` |
|
|
759
|
+
| 7 | Keyboard bindings | `window.addEventListener('keydown', …, { capture: true })` present; `event.stopPropagation()` on every Mudra-claimed key |
|
|
760
|
+
| 8 | Status indicator | `<div id="mudra-status">` present; text states are `Manual` / `Connecting…` / `Connected` / `Disconnected` (Section 15); no `simulated` strings |
|
|
761
|
+
| 9 | AI key gating | If `usesAI`: the AI-Setup fragment (Section 9) is present inside `.mudra-onb__body`; key is read from `sessionStorage.getItem('mudra.gemini.apiKey')` only; ZERO `prompt(` calls for the key; ZERO `localStorage` references; ZERO baked keys (regex scan) |
|
|
762
|
+
| 9a | Visible AI chat I/O | If `usesAI`: the scene renders BOTH the latest user input AND the AI response as visible text (xb.ScrollingTroikaTextView, troika `Text`, or xb.SpatialPanel rows). The visible "Purpose" line states what the app does in one sentence. TTS may exist but is never the only output (Section 18) |
|
|
763
|
+
| 10 | Background lockdown | ZERO `applyBackground_*` methods in the class; ZERO calls to a background helper from `init()`; no `options.simulator.scenePath` line anywhere. Generated apps use the XR Blocks default room only (Section 14) |
|
|
764
|
+
| 11 | Mode toggle | `<div id="mode-toggle">` with **Manual** + **Mudra** buttons; Manual is the default on load; toggle remains clickable when disconnected; flipping atomically opens/closes the socket per Section 15 |
|
|
765
|
+
| 12 | Band-state polling | In Mudra mode the app sends `{command:"get_status"}` on `ws.onopen` and every 2000 ms thereafter; pill flips to `Connected` ONLY when `data.device.state === "connected"` |
|
|
766
|
+
| 13 | No disconnect overlay | No banner / toast / modal / inline alert ever rendered for disconnect — pill is the only indicator |
|
|
767
|
+
| 14 | Footer | Exactly one `<div id="mudra-badge">` containing the literal text `Created by Mudra` (no variants) |
|
|
768
|
+
| 15 | Mock is passive | `MudraClient._startMock()` (or equivalent) starts NO intervals — synthetic signals come only from sim-panel clicks and keyboard shortcuts |
|
|
769
|
+
| 16 | Gemini model pin | If the app calls `generativelanguage.googleapis.com/v1beta/models/<id>:generateContent`, the captured `<id>` MUST equal `gemini-2.5-flash`. No preview / dated / latest aliases. Live-API and image-gen exceptions per Section 9 |
|
|
770
|
+
|
|
771
|
+
### Retry policy
|
|
772
|
+
|
|
773
|
+
If any check fails:
|
|
774
|
+
1. Regenerate once and re-run the full checklist.
|
|
775
|
+
2. If the second attempt also fails, surface the specific failing check(s)
|
|
776
|
+
to the user and do **not** write the file.
|
|
777
|
+
|
|
778
|
+
### File collision / auto-suffix rule
|
|
779
|
+
|
|
780
|
+
- Target filename: `preview/<name>.html`
|
|
781
|
+
- If that file already exists, try `preview/<name>-2.html`, then `-3.html`, etc.
|
|
782
|
+
- Never overwrite an existing file.
|
|
783
|
+
- Always report the final written path to the user.
|
|
784
|
+
|
|
785
|
+
---
|
|
786
|
+
|
|
787
|
+
## Section 11 — Signal → XR Binding Patterns
|
|
788
|
+
|
|
789
|
+
Use these as code snippets when adapting a template.
|
|
790
|
+
Copy and adapt — do not invent new patterns from scratch.
|
|
791
|
+
|
|
792
|
+
### gesture.tap → onSelectStart forwarding
|
|
793
|
+
|
|
794
|
+
```js
|
|
795
|
+
// In init():
|
|
796
|
+
mudra.on('gesture', (data) => {
|
|
797
|
+
if (data.type === 'tap') this.onSelectStart({ source: 'mudra' });
|
|
798
|
+
if (data.type === 'double_tap') this.onSelectEnd({ source: 'mudra' });
|
|
799
|
+
});
|
|
800
|
+
mudra.subscribe('gesture');
|
|
801
|
+
```
|
|
802
|
+
|
|
803
|
+
### pressure → scale / color mapping
|
|
804
|
+
|
|
805
|
+
```js
|
|
806
|
+
// In init():
|
|
807
|
+
mudra.on('pressure', (data) => {
|
|
808
|
+
const s = 0.5 + data.normalized * 1.5; // scale 0.5 … 2.0
|
|
809
|
+
this.mesh.scale.setScalar(s);
|
|
810
|
+
const hue = data.normalized * 0.8; // hue 0 (red) … 0.8 (blue)
|
|
811
|
+
this.mesh.material.color.setHSL(hue, 0.9, 0.5);
|
|
812
|
+
});
|
|
813
|
+
mudra.subscribe('pressure');
|
|
814
|
+
```
|
|
815
|
+
|
|
816
|
+
### nav_direction → spatial menu step
|
|
817
|
+
|
|
818
|
+
```js
|
|
819
|
+
// In init():
|
|
820
|
+
mudra.on('nav_direction', (data) => {
|
|
821
|
+
switch (data.direction) {
|
|
822
|
+
case 'Up': this.menuIndex = Math.max(0, this.menuIndex - 1); break;
|
|
823
|
+
case 'Down': this.menuIndex = Math.min(this.items.length - 1, this.menuIndex + 1); break;
|
|
824
|
+
case 'Right': this.selectItem(this.menuIndex); break;
|
|
825
|
+
case 'Left': this.goBack(); break;
|
|
826
|
+
}
|
|
827
|
+
this.updateMenuHighlight();
|
|
828
|
+
});
|
|
829
|
+
mudra.subscribe('nav_direction');
|
|
830
|
+
```
|
|
831
|
+
|
|
832
|
+
### imu_acc + imu_gyro + snc bundle → subscribe all three together
|
|
833
|
+
|
|
834
|
+
`imu_acc`, `imu_gyro`, and `snc` are always subscribed together. Register handlers
|
|
835
|
+
for each signal you actually use in the app, but always send all three subscribe commands.
|
|
836
|
+
|
|
837
|
+
```js
|
|
838
|
+
// In init():
|
|
839
|
+
mudra.on('imu_acc', (data) => {
|
|
840
|
+
const [ax, ay] = data.values;
|
|
841
|
+
this.mesh.rotation.x = THREE.MathUtils.clamp(ax * 0.1, -Math.PI / 4, Math.PI / 4);
|
|
842
|
+
this.mesh.rotation.z = THREE.MathUtils.clamp(ay * 0.1, -Math.PI / 4, Math.PI / 4);
|
|
843
|
+
});
|
|
844
|
+
mudra.on('imu_gyro', (data) => {
|
|
845
|
+
const [gx] = data.values;
|
|
846
|
+
this.mesh.rotation.y += gx * 0.001;
|
|
847
|
+
});
|
|
848
|
+
mudra.on('snc', (data) => {
|
|
849
|
+
const ch1 = data.values[0];
|
|
850
|
+
const latest = ch1[ch1.length - 1]; // most recent sample
|
|
851
|
+
const norm = Math.min(1, Math.abs(latest) / 500); // normalize
|
|
852
|
+
this.overlay.material.opacity = norm * 0.7;
|
|
853
|
+
});
|
|
854
|
+
// ALWAYS subscribe all three — they form an inseparable bundle
|
|
855
|
+
mudra.subscribe('imu_acc');
|
|
856
|
+
mudra.subscribe('imu_gyro');
|
|
857
|
+
mudra.subscribe('snc');
|
|
858
|
+
```
|
|
859
|
+
|
|
860
|
+
### navigation → continuous cursor / pan
|
|
861
|
+
|
|
862
|
+
**Default sensitivity multiplier: `0.002`** (gentle / low-sensitivity).
|
|
863
|
+
Use this baseline for all navigation/cursor/pan bindings. Only raise it
|
|
864
|
+
when the prompt explicitly asks for fast/snappy movement.
|
|
865
|
+
|
|
866
|
+
```js
|
|
867
|
+
// In init():
|
|
868
|
+
const NAV_SENSITIVITY = 0.002; // gentle default — slow, predictable
|
|
869
|
+
mudra.on('navigation', (data) => {
|
|
870
|
+
this.cursorX = THREE.MathUtils.clamp(this.cursorX + data.delta_x * NAV_SENSITIVITY, -1, 1);
|
|
871
|
+
this.cursorY = THREE.MathUtils.clamp(this.cursorY - data.delta_y * NAV_SENSITIVITY, -1, 1);
|
|
872
|
+
this.cursor.position.set(this.cursorX, this.cursorY, -xb.user.objectDistance);
|
|
873
|
+
});
|
|
874
|
+
mudra.subscribe('navigation');
|
|
875
|
+
```
|
|
876
|
+
|
|
877
|
+
---
|
|
878
|
+
|
|
879
|
+
## Section 12 — Template Selection Table
|
|
880
|
+
|
|
881
|
+
One row per asset in `assets/`. The skill's keyword-based selector scores each row
|
|
882
|
+
against the user's prompt (signal names, motion keywords, XR feature words).
|
|
883
|
+
|
|
884
|
+
| id | path | keywords | motionModesSupported | xrFeatures |
|
|
885
|
+
|----|------|----------|---------------------|------------|
|
|
886
|
+
| `0_basic` | `assets/templates/0_basic.html` | `["basic","cylinder","pinch","color","simple","starter"]` | `["none","pointer"]` | `["input"]` |
|
|
887
|
+
| `1_ui` | `assets/templates/1_ui.html` | `["ui","spatial","panel","text","sdf","font","button","draggable","troika"]` | `["pointer"]` | `["input","ui"]` |
|
|
888
|
+
| `2_hands` | `assets/templates/2_hands.html` | `["hands","hand","pinch","gesture","joints","finger","hand-tracking"]` | `["pointer"]` | `["hands","input"]` |
|
|
889
|
+
| `3_depth` | `assets/templates/3_depth.html` | `["depth","mesh","depth-sensing","plane","environment","occlusion"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
|
|
890
|
+
| `4_stereo` | `assets/templates/4_stereo.html` | `["stereo","video","passthrough","camera","background","environment","feed"]` | `["none"]` | `["camera","passthrough"]` |
|
|
891
|
+
| `5_camera` | `assets/templates/5_camera.html` | `["camera","video","passthrough","texture","scene","background"]` | `["none","pointer"]` | `["camera","input"]` |
|
|
892
|
+
| `6_ai` | `assets/templates/6_ai.html` | `["ai","gemini","query","vision","photo","capture","llm","multimodal"]` | `["pointer"]` | `["input","camera"]` |
|
|
893
|
+
| `7_ai_live` | `assets/templates/7_ai_live.html` | `["ai","gemini","live","speech","transcription","voice","microphone","audio","real-time"]` | `["pointer"]` | `["input"]` |
|
|
894
|
+
| `8_objects` | `assets/templates/8_objects.html` | `["objects","detection","model","3d","place","anchor","ar","environment"]` | `["pointer"]` | `["mesh-detection","input"]` |
|
|
895
|
+
| `9_xr-toggle` | `assets/templates/9_xr-toggle.html` | `["toggle","xr","enter","exit","session","button","transition"]` | `["none","pointer"]` | `["input"]` |
|
|
896
|
+
| `heuristic_hand_gestures` | `assets/templates/heuristic_hand_gestures.html` | `["gesture","heuristic","hand","recognition","custom","pattern","hand-tracking"]` | `["none"]` | `["hands"]` |
|
|
897
|
+
| `meshes` | `assets/templates/meshes.html` | `["mesh","environment","scan","plane","floor","wall","surface"]` | `["none"]` | `["mesh-detection"]` |
|
|
898
|
+
| `planes` | `assets/templates/planes.html` | `["plane","floor","wall","surface","anchor","environment","horizontal","vertical"]` | `["none"]` | `["plane-detection"]` |
|
|
899
|
+
| `uikit` | `assets/templates/uikit.html` | `["ui","kit","component","widget","button","icon","material","text","panel","menu"]` | `["pointer"]` | `["input","ui"]` |
|
|
900
|
+
| `depthmap` | `assets/samples/depthmap.html` | `["depth","map","visualization","color","gradient","environment","scan"]` | `["none"]` | `["depth-sensing"]` |
|
|
901
|
+
| `depthmesh` | `assets/samples/depthmesh.html` | `["depth","mesh","wireframe","environment","scan","geometry"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
|
|
902
|
+
| `game_rps` | `assets/samples/game_rps.html` | `["game","rps","rock","paper","scissors","gesture","compete","fun","hand"]` | `["none"]` | `["hands"]` |
|
|
903
|
+
| `gestures_custom` | `assets/samples/gestures_custom.html` | `["gesture","custom","recognize","train","pose","hand-tracking","hands"]` | `["none"]` | `["hands"]` |
|
|
904
|
+
| `gestures_heuristic` | `assets/samples/gestures_heuristic.html` | `["gesture","heuristic","recognize","hand","pose","pinch","open","fist"]` | `["none"]` | `["hands"]` |
|
|
905
|
+
| `lighting` | `assets/samples/lighting.html` | `["lighting","light","shadow","scene","animals","3d","models","environment"]` | `["pointer"]` | `["input"]` |
|
|
906
|
+
| `mesh_detection` | `assets/samples/mesh_detection.html` | `["mesh","detection","environment","scan","ar","surface"]` | `["none"]` | `["mesh-detection"]` |
|
|
907
|
+
| `modelviewer` | `assets/samples/modelviewer.html` | `["model","viewer","3d","gltf","glb","object","rotate","inspect","load"]` | `["pointer"]` | `["input"]` |
|
|
908
|
+
| `paint` | `assets/samples/paint.html` | `["paint","draw","brush","stroke","canvas","art","color","gesture"]` | `["pointer"]` | `["input","hands"]` |
|
|
909
|
+
| `planar-vst` | `assets/samples/planar-vst.html` | `["plane","vst","passthrough","video","portal","ar","surface"]` | `["none"]` | `["plane-detection","camera"]` |
|
|
910
|
+
| `reticle` | `assets/samples/reticle.html` | `["reticle","cursor","pointer","aim","gaze","target","floor","placement"]` | `["none","pointer"]` | `["plane-detection","input"]` |
|
|
911
|
+
| `skybox_agent` | `assets/samples/skybox_agent.html` | `["skybox","sky","background","ai","gemini","generate","environment","image"]` | `["pointer"]` | `["input"]` |
|
|
912
|
+
| `sound` | `assets/samples/sound.html` | `["sound","audio","music","spatial","3d-audio","positional","play"]` | `["pointer"]` | `["input"]` |
|
|
913
|
+
| `ui` | `assets/samples/ui.html` | `["ui","panel","button","menu","list","text","interface","spatial"]` | `["pointer"]` | `["input","ui"]` |
|
|
914
|
+
| `virtual-screens` | `assets/samples/virtual-screens.html` | `["screen","virtual","window","share","stream","desktop","browser","display"]` | `["pointer"]` | `["input"]` |
|
|
915
|
+
| `3dgs-walkthrough` | `assets/demos/3dgs-walkthrough.html` | `["gaussian","splat","3dgs","scene","walkthrough","room","photo","realistic"]` | `["imu","direction"]` | `["input"]` |
|
|
916
|
+
| `aisimulator` | `assets/demos/aisimulator.html` | `["ai","simulate","gemini","agent","roleplay","character","conversation","npc"]` | `["pointer"]` | `["input"]` |
|
|
917
|
+
| `balloonpop` | `assets/demos/balloonpop.html` | `["balloon","pop","game","gesture","fun","pinch","particle","explosion"]` | `["none"]` | `["hands","input"]` |
|
|
918
|
+
| `ballpit` | `assets/demos/ballpit.html` | `["ball","physics","pit","throw","gravity","interact","fun","ammo"]` | `["pointer"]` | `["input","hands"]` |
|
|
919
|
+
| `drone` | `assets/demos/drone.html` | `["drone","fly","navigate","control","imu","tilt","direction","pilot"]` | `["imu","direction"]` | `["input"]` |
|
|
920
|
+
| `gemini-icebreakers` | `assets/demos/gemini-icebreakers.html` | `["gemini","icebreaker","question","conversation","ai","social","fun"]` | `["pointer"]` | `["input"]` |
|
|
921
|
+
| `gemini-xrobject` | `assets/demos/gemini-xrobject.html` | `["gemini","object","recognize","camera","ar","label","identify","vision"]` | `["pointer"]` | `["input","camera"]` |
|
|
922
|
+
| `math3d` | `assets/demos/math3d.html` | `["math","3d","formula","graph","equation","plot","visualization","education"]` | `["pointer"]` | `["input","ui"]` |
|
|
923
|
+
| `measure` | `assets/demos/measure.html` | `["measure","ruler","tape","distance","ar","depth","spatial","length"]` | `["none"]` | `["depth-sensing","mesh-detection"]` |
|
|
924
|
+
| `occlusion` | `assets/demos/occlusion.html` | `["occlusion","depth","animal","model","shadow","ar","realistic","environment"]` | `["pointer"]` | `["input","depth-sensing"]` |
|
|
925
|
+
| `rain` | `assets/demos/rain.html` | `["rain","particle","weather","atmosphere","shader","visual","instanced"]` | `["none","pointer"]` | `["input"]` |
|
|
926
|
+
| `screenwiper` | `assets/demos/screenwiper.html` | `["wiper","screen","wipe","clear","gesture","brush","clean","effect"]` | `["pointer"]` | `["input","hands"]` |
|
|
927
|
+
| `splash` | `assets/demos/splash.html` | `["splash","paint","decal","physics","shoot","color","ball","impact"]` | `["pointer"]` | `["input"]` |
|
|
928
|
+
| `webcam_gestures` | `assets/demos/webcam_gestures.html` | `["webcam","mediapipe","gesture","hand","camera","tracking","no-headset","flat"]` | `["none"]` | `[]` |
|
|
929
|
+
| `xremoji` | `assets/demos/xremoji.html` | `["emoji","expression","face","gesture","fun","balloon","tensorflow","social"]` | `["none"]` | `["hands"]` |
|
|
930
|
+
| `xrpoet` | `assets/demos/xrpoet.html` | `["poem","poetry","ai","gemini","generate","creative","camera","writing"]` | `["pointer"]` | `["input","camera"]` |
|
|
931
|
+
|
|
932
|
+
---
|
|
933
|
+
|
|
934
|
+
## Section 13 — Template Inference + Override
|
|
935
|
+
|
|
936
|
+
*(Populated after Phase 5 / T032.)*
|
|
937
|
+
|
|
938
|
+
---
|
|
939
|
+
|
|
940
|
+
## Section 14 — Background Lockdown (XR Blocks default only)
|
|
941
|
+
|
|
942
|
+
**Custom backgrounds are forbidden.** Every generated XR app uses the XR
|
|
943
|
+
Blocks default room and nothing else. There is no catalog, no helper, no
|
|
944
|
+
override, no `[bg=...]` tag, no `scenePath` override.
|
|
945
|
+
|
|
946
|
+
### Hard rules — apply unconditionally
|
|
947
|
+
|
|
948
|
+
1. The `xb.Script` subclass MUST NOT contain any `applyBackground_*` method.
|
|
949
|
+
2. `init()` MUST NOT call any background helper. It starts with lights,
|
|
950
|
+
meshes, and Mudra wiring.
|
|
951
|
+
3. The entry point MUST NOT set `options.simulator.scenePath` — not to
|
|
952
|
+
`null`, not to a path. Leave it alone so XR Blocks renders its default
|
|
953
|
+
room.
|
|
954
|
+
4. No `THREE.SphereGeometry` dome, no custom skybox `Mesh`, no
|
|
955
|
+
`THREE.Points` starfield, no `THREE.GridHelper` floor, no equirectangular
|
|
956
|
+
`TextureLoader().load(...)` for background purposes. (Per-scene
|
|
957
|
+
geometry that the app actually needs is fine — the ban is on standalone
|
|
958
|
+
environment domes/floors/skies.)
|
|
959
|
+
5. Prompt cues like "in space", "starfield", "sunset sky", "cyberpunk
|
|
960
|
+
vibe", "with a forest backdrop", or even literal `[bg=<id>]` tags MUST
|
|
961
|
+
be IGNORED for background purposes. They may still inform template /
|
|
962
|
+
motion-mode selection.
|
|
963
|
+
6. If the user explicitly insists on a custom background, decline and
|
|
964
|
+
remind them that this skill is locked to the XR Blocks default room.
|
|
965
|
+
|
|
966
|
+
### Pre-write regex (Section 10 check #10)
|
|
967
|
+
|
|
968
|
+
The generated source MUST satisfy ALL of the following:
|
|
969
|
+
|
|
970
|
+
- `/applyBackground_/` → zero matches.
|
|
971
|
+
- `/options\.simulator\.scenePath/` → zero matches.
|
|
972
|
+
|
|
973
|
+
If either pattern matches, the file fails the pre-write checklist and is
|
|
974
|
+
not written.
|
|
975
|
+
|
|
976
|
+
---
|
|
977
|
+
## Section 15 — Mode Toggle (Manual / Mudra) — Required
|
|
978
|
+
|
|
979
|
+
**This section supersedes Section 4's auto-fallback "simulated" status and
|
|
980
|
+
Section 7's `simulated` / `disconnected-simulated` states for all new
|
|
981
|
+
apps.** Every XR app generated by this skill MUST implement the Mode
|
|
982
|
+
Toggle exactly as specified here. Canonical protocol:
|
|
983
|
+
`references/agent_protocol.json` (v2.0).
|
|
984
|
+
|
|
985
|
+
### Summary
|
|
986
|
+
|
|
987
|
+
The user picks how the app is driven via a visible **Mode** control,
|
|
988
|
+
rendered as a 2D DOM overlay (it disappears automatically in immersive
|
|
989
|
+
WebXR sessions, like the simulator panel and status pill):
|
|
990
|
+
|
|
991
|
+
- **Manual** (default on load): the simulator panel is fully interactive.
|
|
992
|
+
Sim-panel actions inject synthetic signal messages into the same handler
|
|
993
|
+
pipeline that real WebSocket messages would flow through. **No WebSocket
|
|
994
|
+
connection is opened in Manual mode.** The XR scene reacts only to
|
|
995
|
+
signal handlers — never to direct DOM clicks bypassing the signal path.
|
|
996
|
+
- **Mudra**: the app opens one WebSocket to `ws://127.0.0.1:8766` and
|
|
997
|
+
subscribes to its signals one-at-a-time. The simulator panel is
|
|
998
|
+
visually disabled (greyed-out, `pointer-events: none`,
|
|
999
|
+
`aria-disabled="true"`) and emits no synthetic signals. The
|
|
1000
|
+
connection-status pill reflects band-pairing state via `get_status`
|
|
1001
|
+
polling — **no separate overlay, toast, or banner is rendered.**
|
|
1002
|
+
|
|
1003
|
+
**The Mode toggle MUST remain fully clickable and keyboard-focusable at
|
|
1004
|
+
all times** — including while the band is disconnected. Manual is the
|
|
1005
|
+
default on first load.
|
|
1006
|
+
|
|
1007
|
+
### State machine
|
|
1008
|
+
|
|
1009
|
+
```text
|
|
1010
|
+
type Mode = "manual" | "mudra" // default "manual"
|
|
1011
|
+
type ConnectionState =
|
|
1012
|
+
| "idle" // No socket open. Always the case in Manual.
|
|
1013
|
+
| "connecting" // Socket opening, OR socket open but band-pairing not yet confirmed via get_status.
|
|
1014
|
+
| "connected" // Socket open AND last get_status response had data.device.state === "connected".
|
|
1015
|
+
| "disconnected" // Socket closed/errored, OR socket open but data.device.state !== "connected".
|
|
1016
|
+
```
|
|
1017
|
+
|
|
1018
|
+
Lazy-WS lifecycle (mandatory):
|
|
1019
|
+
|
|
1020
|
+
| Transition | Action |
|
|
1021
|
+
|------------|--------|
|
|
1022
|
+
| page load | `mode = "manual"`, `connectionState = "idle"`, NO socket |
|
|
1023
|
+
| Manual → Mudra | open new socket, set `connectionState = "connecting"` |
|
|
1024
|
+
| Mudra → Manual | close socket, cancel any in-flight reconnect timer, set `connectionState = "idle"` |
|
|
1025
|
+
| WS `open` (in Mudra) | keep `connectionState = "connecting"`, send all `subscribe` commands, send `{command:"get_status"}`, start status-poll timer |
|
|
1026
|
+
| inbound `status` with `data.device.state === "connected"` (in Mudra) | `connectionState = "connected"` |
|
|
1027
|
+
| inbound `status` with `data.device.state !== "connected"` (in Mudra) | `connectionState = "disconnected"`, **keep socket open** — do NOT closeSocket, do NOT schedule WS reconnect; status-poll will surface the band coming back |
|
|
1028
|
+
| inbound `connection_status: connected` (in Mudra) | request a fresh `get_status`; do not flip the pill on this alone |
|
|
1029
|
+
| inbound `connection_status: disconnected` (in Mudra) | `connectionState = "disconnected"` |
|
|
1030
|
+
| WS error / WS close (in Mudra) | `connectionState = "disconnected"`, stop status-poll, schedule socket reconnect |
|
|
1031
|
+
| reconnect tick (in Mudra & socket dead) | open new socket → `connecting` |
|
|
1032
|
+
|
|
1033
|
+
**Single-socket guarantee:** never have two `WebSocket` instances open at
|
|
1034
|
+
once. Use a connection token to neutralise rapid-toggle races (see the
|
|
1035
|
+
`MudraClient` extensions below).
|
|
1036
|
+
|
|
1037
|
+
### MudraClient changes vs. Section 4
|
|
1038
|
+
|
|
1039
|
+
The `MudraClient` from Section 4 must be extended (or replaced) so it:
|
|
1040
|
+
|
|
1041
|
+
1. Does NOT auto-connect in its constructor. Connection is driven by
|
|
1042
|
+
`setMode("mudra")` only.
|
|
1043
|
+
2. Exposes `setMode(mode)` to flip between `"manual"` and `"mudra"`.
|
|
1044
|
+
3. In Manual mode, the `_startMock()` interval generators (if any) are
|
|
1045
|
+
NOT started. Mock signals must be **passive**: emitted only when the
|
|
1046
|
+
sim panel is clicked or a keyboard shortcut fires. (Memory:
|
|
1047
|
+
`MudraClient mock must be passive` — strip auto-firing intervals from
|
|
1048
|
+
`_startMock()`; sim panel clicks and keys are the only signal source.)
|
|
1049
|
+
4. On entering Mudra mode, opens one WebSocket, sends `subscribe` for
|
|
1050
|
+
every signal in its subscription list, sends `{command:"get_status"}`
|
|
1051
|
+
immediately, and starts a 2 s `get_status` poll while the socket is
|
|
1052
|
+
`OPEN`.
|
|
1053
|
+
5. Emits `_status` events with the new four-state vocabulary:
|
|
1054
|
+
`"idle" | "connecting" | "connected" | "disconnected"`. The legacy
|
|
1055
|
+
`"simulated"` / `"disconnected-simulated"` strings are removed.
|
|
1056
|
+
|
|
1057
|
+
### Disconnect detection — band state via `get_status` polling (mandatory)
|
|
1058
|
+
|
|
1059
|
+
**The WebSocket handshake to `127.0.0.1:8766` only proves the Companion
|
|
1060
|
+
service is up. It does NOT prove the user's Mudra Band is paired.** The
|
|
1061
|
+
Companion accepts socket connections even when no band is bonded — so
|
|
1062
|
+
flipping the pill to "Connected" on `ws.onopen` is wrong. The pill MUST
|
|
1063
|
+
reflect the band itself, not the socket.
|
|
1064
|
+
|
|
1065
|
+
The source of truth is the `status` response to `{command:"get_status"}`:
|
|
1066
|
+
|
|
1067
|
+
```json
|
|
1068
|
+
> {"command":"get_status"}
|
|
1069
|
+
< {"type":"status","data":{"device":{"state":"connected", ... }, ...}, "timestamp": ...}
|
|
1070
|
+
< {"type":"status","data":{"device":{"state":"disconnected", ...}, ...}, "timestamp": ...}
|
|
1071
|
+
```
|
|
1072
|
+
|
|
1073
|
+
Rules:
|
|
1074
|
+
|
|
1075
|
+
1. On `ws.onopen` (in Mudra mode): stay in `connecting`; send all
|
|
1076
|
+
`subscribe` commands; send `{command:"get_status"}`; start a
|
|
1077
|
+
**status-poll timer** that sends `{command:"get_status"}` every
|
|
1078
|
+
**2000 ms** while `mode === "mudra"` and the socket is `OPEN`.
|
|
1079
|
+
2. On inbound `{type:"status"}` (in Mudra mode):
|
|
1080
|
+
- `data?.device?.state === "connected"` → `setState("connected")`.
|
|
1081
|
+
- Else → `setState("disconnected")`. Keep the socket open. Do NOT
|
|
1082
|
+
`closeSocket()`. The next poll tick picks the band up after pairing.
|
|
1083
|
+
3. On inbound `{type:"connection_status"}`: hint only. On `disconnected`,
|
|
1084
|
+
flip the pill. On `connected`, send a fresh `{command:"get_status"}`
|
|
1085
|
+
and let the `status` handler do the actual transition.
|
|
1086
|
+
4. On WS `error` / `close` (in Mudra mode): `setState("disconnected")`,
|
|
1087
|
+
stop the status-poll timer, `scheduleReconnect()`.
|
|
1088
|
+
5. On Manual mode: stop the status-poll timer in `closeSocket()`.
|
|
1089
|
+
|
|
1090
|
+
Do not poll faster than 1 s; do not poll slower than 5 s. 2 s is
|
|
1091
|
+
mandated. The pill MAY sit on `Connecting…` for up to one poll cycle
|
|
1092
|
+
(~2 s) after entering Mudra mode while the first `status` round-trips —
|
|
1093
|
+
that is correct behaviour.
|
|
1094
|
+
|
|
1095
|
+
### Reconnect backoff
|
|
1096
|
+
|
|
1097
|
+
While `mode === "mudra" && connectionState === "disconnected"` AND the
|
|
1098
|
+
socket itself is dead (not just the band):
|
|
1099
|
+
|
|
1100
|
+
```js
|
|
1101
|
+
const RECONNECT_DELAYS_MS = [1000, 2000, 5000, 5000, 5000]; // capped at 5s
|
|
1102
|
+
```
|
|
1103
|
+
|
|
1104
|
+
Reset the index on every successful `connected` transition.
|
|
1105
|
+
|
|
1106
|
+
### Status pill text states (replaces Section 7)
|
|
1107
|
+
|
|
1108
|
+
| connectionState | textContent | colour hint |
|
|
1109
|
+
|-----------------|-------------|-------------|
|
|
1110
|
+
| `idle` (Manual) | `Manual` | neutral |
|
|
1111
|
+
| `connecting` | `Connecting…` | amber |
|
|
1112
|
+
| `connected` | `Connected` | green |
|
|
1113
|
+
| `disconnected` | `Disconnected` | red |
|
|
1114
|
+
|
|
1115
|
+
The pill is the **only** disconnect indicator. No banner, toast, or
|
|
1116
|
+
modal. The simulator panel is greyed (reduced opacity,
|
|
1117
|
+
`pointer-events: none`) when in Mudra + `disconnected`, but the pill is
|
|
1118
|
+
still the only textual disconnect cue.
|
|
1119
|
+
|
|
1120
|
+
### Mode-toggle DOM sketch
|
|
1121
|
+
|
|
1122
|
+
```html
|
|
1123
|
+
<div id="mode-toggle" role="tablist" aria-label="Mode">
|
|
1124
|
+
<button id="mode-manual" role="tab" aria-selected="true">Manual</button>
|
|
1125
|
+
<button id="mode-mudra" role="tab" aria-selected="false">Mudra</button>
|
|
1126
|
+
</div>
|
|
1127
|
+
<div id="mudra-status" class="conn-manual">Manual</div>
|
|
1128
|
+
```
|
|
1129
|
+
|
|
1130
|
+
On every mode change, atomically: cancel any reconnect timer, stop the
|
|
1131
|
+
status-poll, close any open socket, reset `connToken`, then either
|
|
1132
|
+
(Manual) leave `connectionState = "idle"` OR (Mudra) call `openSocket()`.
|
|
1133
|
+
|
|
1134
|
+
---
|
|
1135
|
+
|
|
1136
|
+
## Section 16 — Footer Branding — "Created by Mudra"
|
|
1137
|
+
|
|
1138
|
+
Every generated app MUST render a small footer/badge with the **exact**
|
|
1139
|
+
text **`Created by Mudra`**. Never "Created with Mudra Studio", never
|
|
1140
|
+
"Powered by Mudra", never any other variant. The badge is a 2D DOM
|
|
1141
|
+
overlay (disappears in immersive WebXR like the rest of the chrome).
|
|
1142
|
+
|
|
1143
|
+
```html
|
|
1144
|
+
<div id="mudra-badge" style="
|
|
1145
|
+
position: fixed; bottom: 8px; right: 12px;
|
|
1146
|
+
padding: 4px 10px; border-radius: 999px;
|
|
1147
|
+
font-size: 0.75rem; font-family: system-ui, sans-serif;
|
|
1148
|
+
background: rgba(0,0,0,0.5); color: #fff; opacity: 0.85;
|
|
1149
|
+
z-index: 9999; pointer-events: none;">Created by Mudra</div>
|
|
1150
|
+
```
|
|
1151
|
+
|
|
1152
|
+
Position is flexible if it would overlap the simulator panel — keep the
|
|
1153
|
+
text identical.
|
|
1154
|
+
|
|
1155
|
+
---
|
|
1156
|
+
|
|
1157
|
+
## Section 17 — Onboarding Modal (mandatory, STRICT) — feature 008-strict-onboarding-templates
|
|
1158
|
+
|
|
1159
|
+
> **Supersedes feature 005's loose modal rules.** The locked layout is
|
|
1160
|
+
> **Template 3 — split-card**. The DOM, CSS, and JS below MUST be emitted
|
|
1161
|
+
> **verbatim**. Only the per-app content slots may vary.
|
|
1162
|
+
|
|
1163
|
+
Every generated XR app MUST ship a first-run onboarding modal that greets
|
|
1164
|
+
the user and lists every action the app supports, with paired **Mudra** and
|
|
1165
|
+
**Manual** controls per action. The modal closes via `×` (skip), the
|
|
1166
|
+
**Continue** button, or `Escape`. It re-opens via a small floating `?` icon
|
|
1167
|
+
**only outside immersive XR** — in-XR re-onboarding is out of scope for v1
|
|
1168
|
+
(per feature 005 clarification Q3).
|
|
1169
|
+
|
|
1170
|
+
The binding contract is `specs/008-strict-onboarding-templates/contracts/onboarding-block.md`. The blocks below are the verbatim copies emitted into every app.
|
|
1171
|
+
|
|
1172
|
+
### XR-specific behavior (read carefully — unchanged from feature 005)
|
|
1173
|
+
|
|
1174
|
+
The onboarding modal is a **2D HTML overlay** shown before immersive entry.
|
|
1175
|
+
|
|
1176
|
+
- **MANDATORY: Disable the XR Blocks default Welcome overlay.** When the
|
|
1177
|
+
Simulator addon is imported (`import 'xrblocks/addons/simulator/SimulatorAddons.js';`),
|
|
1178
|
+
XR Blocks injects its own "Welcome to XR Blocks!" intro modal. This
|
|
1179
|
+
overlay competes visually with the Mudra onboarding modal and must be
|
|
1180
|
+
suppressed. Immediately after constructing the options object, set:
|
|
1181
|
+
|
|
1182
|
+
```js
|
|
1183
|
+
const options = new xb.Options();
|
|
1184
|
+
options.simulator.instructions.enabled = false; // suppress XR Blocks default Welcome overlay; the Mudra onboarding modal replaces it
|
|
1185
|
+
// ...rest of options config
|
|
1186
|
+
xb.init(options);
|
|
1187
|
+
```
|
|
1188
|
+
|
|
1189
|
+
- **Do NOT** mirror the Mudra modal as a 3D panel inside the XR scene. In-XR re-onboarding is explicitly out of scope for v1.
|
|
1190
|
+
- The inline `<script>` already listens for `xrsession-start` / `vr-session-start` / `ar-session-start` events and hides both the modal and the `?` icon during immersive sessions. Do NOT edit that wiring.
|
|
1191
|
+
- The modal layers above the XR canvas in 2D mode (`z-index: 100`). The Mudra badge from Section 16 and the simulator panel sit beneath it, by design.
|
|
1192
|
+
|
|
1193
|
+
### Required palette addition
|
|
1194
|
+
|
|
1195
|
+
Every generated app's `:root` MUST add one variable beyond the canonical palette:
|
|
1196
|
+
|
|
1197
|
+
```css
|
|
1198
|
+
--on-primary: #0c0d10; /* dark text on the primary-blue Continue button */
|
|
1199
|
+
```
|
|
1200
|
+
|
|
1201
|
+
### Locked DOM (paste at end of `<body>`)
|
|
1202
|
+
|
|
1203
|
+
```html
|
|
1204
|
+
<!-- === BEGIN onboarding-block === (Template 3 — Split Card, feature 008) -->
|
|
1205
|
+
<!-- IMPORTANT: do NOT add the `open` attribute. In 3D apps this matters
|
|
1206
|
+
particularly — XR Blocks' canvas is appended to <body> AFTER this
|
|
1207
|
+
dialog, and a non-modal dialog (HTML `open`) does NOT enter the
|
|
1208
|
+
browser's top layer. The dialog looks visible but the canvas can
|
|
1209
|
+
intercept clicks on Continue / ×. The IIFE below calls showModal()
|
|
1210
|
+
on load, which elevates the dialog above XR Blocks' canvas
|
|
1211
|
+
regardless of z-index. -->
|
|
1212
|
+
<dialog id="mudra-onboarding" data-mudra-onboarding data-app-name="{APP_NAME}">
|
|
1213
|
+
<div class="ob-card">
|
|
1214
|
+
<button class="ob-x" aria-label="Skip onboarding" data-ob-close>×</button>
|
|
1215
|
+
<div class="ob-left">
|
|
1216
|
+
<div class="ob-brand-block">
|
|
1217
|
+
<span class="ob-brand-mark">Mudra Studio</span>
|
|
1218
|
+
<h2 class="ob-brand-name">{APP_NAME_HEAD} <em>{APP_NAME_TAIL}</em></h2>
|
|
1219
|
+
<p class="ob-tagline">{APP_TAGLINE}</p>
|
|
1220
|
+
</div>
|
|
1221
|
+
<span class="ob-brand-footer">Created by Mudra</span>
|
|
1222
|
+
</div>
|
|
1223
|
+
<div class="ob-right">
|
|
1224
|
+
<h3 class="ob-section-title">How to use this app</h3>
|
|
1225
|
+
<div class="ob-chip-grid" id="ob-rows"></div>
|
|
1226
|
+
<div class="ob-continue-row"><button class="ob-continue" data-ob-close>Continue</button></div>
|
|
1227
|
+
</div>
|
|
1228
|
+
</div>
|
|
1229
|
+
</dialog>
|
|
1230
|
+
<button id="mudra-onboarding-help" class="ob-help-btn" aria-label="Reopen onboarding" hidden>?</button>
|
|
1231
|
+
<!-- === END onboarding-block === -->
|
|
1232
|
+
```
|
|
1233
|
+
|
|
1234
|
+
### Locked CSS (paste inside the existing `<style>` block in `<head>` — add one if the template does not have one)
|
|
1235
|
+
|
|
1236
|
+
```css
|
|
1237
|
+
/* === BEGIN onboarding-block === (Template 3 — Split Card, feature 008) */
|
|
1238
|
+
#mudra-onboarding{position:fixed;inset:0;border:0;padding:0;background:transparent;width:100%;height:100%;max-width:none;max-height:none;display:grid;place-items:center;z-index:100;color:var(--text);}
|
|
1239
|
+
#mudra-onboarding::backdrop{background:rgba(0,0,0,0.55);backdrop-filter:blur(6px);}
|
|
1240
|
+
#mudra-onboarding[hidden],#mudra-onboarding:not([open]){display:none;}
|
|
1241
|
+
.ob-card{position:relative;background:var(--card);backdrop-filter:blur(10px);border:1px solid rgba(255,255,255,0.08);border-radius:18px;width:min(720px,94vw);max-height:88vh;overflow:auto;display:grid;grid-template-columns:1fr 1fr;gap:0;box-shadow:0 20px 60px rgba(0,0,0,0.5);font-family:'Poppins',system-ui,sans-serif;}
|
|
1242
|
+
.ob-x{position:absolute;top:14px;right:14px;width:32px;height:32px;border-radius:50%;appearance:none;border:0;background:rgba(255,255,255,0.08);color:var(--text);font-size:18px;cursor:pointer;z-index:2;}
|
|
1243
|
+
.ob-x:hover{background:rgba(255,255,255,0.16);}
|
|
1244
|
+
.ob-left{padding:36px 28px;background:linear-gradient(135deg,rgba(108,140,255,0.16),rgba(185,124,255,0.12));border-right:1px solid rgba(255,255,255,0.06);display:flex;flex-direction:column;justify-content:space-between;border-radius:18px 0 0 18px;}
|
|
1245
|
+
.ob-brand-block{display:flex;flex-direction:column;gap:14px;}
|
|
1246
|
+
.ob-brand-mark{display:flex;align-items:center;gap:10px;font-size:13px;letter-spacing:0.16em;text-transform:uppercase;color:var(--text-secondary);}
|
|
1247
|
+
.ob-brand-mark::before{content:"";display:inline-block;width:24px;height:2px;background:var(--primary);}
|
|
1248
|
+
.ob-brand-name{font-size:34px;font-weight:700;line-height:1.05;margin:0;}
|
|
1249
|
+
.ob-brand-name em{font-style:normal;color:var(--accent);}
|
|
1250
|
+
.ob-tagline{color:var(--text-secondary);font-size:15px;margin:6px 0 0;line-height:1.5;}
|
|
1251
|
+
.ob-brand-footer{font-size:11px;letter-spacing:0.18em;text-transform:uppercase;color:var(--text-secondary);}
|
|
1252
|
+
.ob-right{padding:32px 28px 24px;display:flex;flex-direction:column;}
|
|
1253
|
+
.ob-section-title{font-size:12px;letter-spacing:0.1em;text-transform:uppercase;color:var(--text-secondary);margin:0 0 14px;}
|
|
1254
|
+
.ob-chip-grid{display:grid;grid-template-columns:1fr;gap:8px;flex:1;}
|
|
1255
|
+
.ob-chip{display:grid;grid-template-columns:1fr auto auto;gap:10px;align-items:center;background:rgba(255,255,255,0.04);border:1px solid rgba(255,255,255,0.05);border-radius:10px;padding:10px 12px;font-size:14px;}
|
|
1256
|
+
.ob-chip .nm{font-weight:500;}
|
|
1257
|
+
.ob-chip .mu{font-size:12px;padding:3px 9px;border-radius:999px;background:rgba(108,140,255,0.2);color:var(--primary);font-weight:600;}
|
|
1258
|
+
.ob-chip .mn{font-size:12px;color:var(--text-secondary);}
|
|
1259
|
+
.ob-continue-row{display:flex;justify-content:flex-end;margin-top:18px;}
|
|
1260
|
+
.ob-continue{appearance:none;border:0;background:var(--primary);color:var(--on-primary);font:inherit;font-weight:600;padding:10px 22px;border-radius:10px;cursor:pointer;}
|
|
1261
|
+
.ob-continue:hover{filter:brightness(1.1);}
|
|
1262
|
+
.ob-help-btn{position:fixed;bottom:16px;right:16px;appearance:none;border:0;width:36px;height:36px;border-radius:50%;background:var(--card);color:var(--text);font-size:18px;cursor:pointer;backdrop-filter:blur(10px);border:1px solid rgba(255,255,255,0.12);z-index:50;}
|
|
1263
|
+
@media (max-width:640px){
|
|
1264
|
+
.ob-card{grid-template-columns:1fr;}
|
|
1265
|
+
.ob-left{border-right:0;border-bottom:1px solid rgba(255,255,255,0.06);border-radius:18px 18px 0 0;padding:24px;}
|
|
1266
|
+
.ob-brand-name{font-size:24px;}
|
|
1267
|
+
.ob-right{padding:20px;}
|
|
1268
|
+
.ob-continue{width:100%;}
|
|
1269
|
+
.ob-continue-row{justify-content:stretch;}
|
|
1270
|
+
}
|
|
1271
|
+
/* === END onboarding-block === */
|
|
1272
|
+
```
|
|
1273
|
+
|
|
1274
|
+
### Locked JS (paste inside an inline `<script>` — NOT `type="module"` — at end of `<body>`)
|
|
1275
|
+
|
|
1276
|
+
```js
|
|
1277
|
+
// === BEGIN onboarding-block === (Template 3 — Split Card, feature 008)
|
|
1278
|
+
window.MUDRA_ONBOARDING_ACTIONS = [
|
|
1279
|
+
// Filled by the skill from the app's subscribed signals. See "App-aware filter — strict" below.
|
|
1280
|
+
];
|
|
1281
|
+
|
|
1282
|
+
(function () {
|
|
1283
|
+
const root = document.getElementById('mudra-onboarding');
|
|
1284
|
+
const help = document.getElementById('mudra-onboarding-help');
|
|
1285
|
+
const grid = document.getElementById('ob-rows');
|
|
1286
|
+
|
|
1287
|
+
grid.innerHTML = window.MUDRA_ONBOARDING_ACTIONS.map(r => `
|
|
1288
|
+
<div class="ob-chip">
|
|
1289
|
+
<span class="nm">${r.action}</span>
|
|
1290
|
+
<span class="mu">${r.mudra}</span>
|
|
1291
|
+
<span class="mn">${r.manual}</span>
|
|
1292
|
+
</div>`).join('');
|
|
1293
|
+
|
|
1294
|
+
function isInImmersiveXR() {
|
|
1295
|
+
return !!(window.xb && window.xb.session && window.xb.session.isImmersive);
|
|
1296
|
+
}
|
|
1297
|
+
function openOb() { if (!root.open) root.showModal(); help.hidden = true; }
|
|
1298
|
+
function closeOb() { if (root.open) root.close(); help.hidden = false; }
|
|
1299
|
+
|
|
1300
|
+
// Close wiring — BOTH `.ob-x` and `.ob-continue` carry [data-ob-close].
|
|
1301
|
+
// This single querySelectorAll attaches the same `closeOb` to each;
|
|
1302
|
+
// do not add separate listeners or split the behavior.
|
|
1303
|
+
root.querySelectorAll('[data-ob-close]').forEach(b => b.addEventListener('click', closeOb));
|
|
1304
|
+
|
|
1305
|
+
// Re-open wiring — floating help-pill (mouse/touch) and the `?` key.
|
|
1306
|
+
// `?` is gated on isInImmersiveXR() so it never re-opens inside VR/AR.
|
|
1307
|
+
help.addEventListener('click', openOb);
|
|
1308
|
+
document.addEventListener('keydown', (e) => {
|
|
1309
|
+
if (e.key === 'Escape' && root.open) closeOb();
|
|
1310
|
+
if (e.key === '?' && !root.open && !isInImmersiveXR()) openOb();
|
|
1311
|
+
});
|
|
1312
|
+
|
|
1313
|
+
// Hide modal + help during immersive XR sessions (preserves feature 005's contract).
|
|
1314
|
+
['xrsession-start','vr-session-start','ar-session-start'].forEach(ev =>
|
|
1315
|
+
window.addEventListener(ev, () => { if (root.open) root.close(); help.hidden = true; })
|
|
1316
|
+
);
|
|
1317
|
+
|
|
1318
|
+
// Open on load — UNCONDITIONAL showModal(). The dialog enters the
|
|
1319
|
+
// browser's top layer so XR Blocks' canvas (appended after this
|
|
1320
|
+
// script) can never intercept clicks on Continue / ×. Do not gate on
|
|
1321
|
+
// `root.open`; do not use `root.show()`.
|
|
1322
|
+
root.showModal();
|
|
1323
|
+
})();
|
|
1324
|
+
// === END onboarding-block ===
|
|
1325
|
+
```
|
|
1326
|
+
|
|
1327
|
+
### Close behavior — what each control does (verbatim explanation for the model)
|
|
1328
|
+
|
|
1329
|
+
| Control | When it fires | Effect |
|
|
1330
|
+
|---|---|---|
|
|
1331
|
+
| **Continue** (`.ob-continue`) | User clicks / activates the primary CTA in the right column | `root.close()`. Help-pill `?` becomes visible bottom-right. User starts using the app (or enters XR via the XR Blocks button). |
|
|
1332
|
+
| **×** (`.ob-x`) | User clicks / activates the circular X in the top-right of the card | Same as Continue — `root.close()` + show `?` pill. The two paths are intentionally equivalent. X is the visual "skip" affordance. (Exception: AI-Setup extension blocks both Continue and × until a valid API key is entered — see "AI-app extension" below.) |
|
|
1333
|
+
| **Escape** | User presses Esc while modal is open | Native `<dialog>` close + our `keydown` handler unhides the `?` pill. (AI extension also intercepts this via the `cancel` event when no valid key is set.) |
|
|
1334
|
+
| **`?` key** | User presses `?` while modal is closed AND NOT inside immersive XR | `root.showModal()` reopens. Inside VR/AR the key is a no-op so the user is never yanked out of the scene. |
|
|
1335
|
+
| **Help-pill** (`#mudra-onboarding-help`) | User clicks the floating `?` button bottom-right (visible only after the modal has been closed once, and hidden during immersive XR) | `root.showModal()` reopens. |
|
|
1336
|
+
| **`xrsession-start` / `vr-session-start` / `ar-session-start`** | User enters VR/AR via the XR Blocks Enter button | Force-close the modal; hide the help-pill so it does not appear in-world. |
|
|
1337
|
+
| **Page load** | Every fresh render | `root.showModal()` runs unconditionally inside the IIFE. Help-pill starts hidden. |
|
|
1338
|
+
|
|
1339
|
+
### How "close" is wired (REQUIRED — do not refactor)
|
|
1340
|
+
|
|
1341
|
+
1. Both close buttons carry the `data-ob-close` attribute. The single line `root.querySelectorAll('[data-ob-close]').forEach(b => b.addEventListener('click', closeOb))` is the entire mouse/touch close wiring. The AI-Setup extension below adds its own `capture:true` listener on top of these to gate close when no API key is present — it does not replace the underlying wiring.
|
|
1342
|
+
2. Keyboard close is handled by the `document`-level `keydown` listener so Escape fires even when focus is outside the dialog.
|
|
1343
|
+
3. Opening on load uses `root.showModal()` (modal mode, top layer). Never `root.show()` (non-modal) and never the `open` attribute in HTML. In 3D apps this is REQUIRED — non-modal dialogs do not enter the top layer and XR Blocks' canvas can intercept clicks behind the modal.
|
|
1344
|
+
|
|
1345
|
+
### Anti-patterns — will fail review
|
|
1346
|
+
|
|
1347
|
+
- ❌ `<dialog ... open>` in HTML. Open via `showModal()` only.
|
|
1348
|
+
- ❌ Calling `root.show()` instead of `root.showModal()`.
|
|
1349
|
+
- ❌ Omitting `options.simulator.instructions.enabled = false;` — XR Blocks' Welcome overlay then competes with the Mudra modal.
|
|
1350
|
+
- ❌ Adding click-outside-to-close, time-out auto-close, or any extra close path.
|
|
1351
|
+
- ❌ Hiding `#mudra-onboarding-help` permanently after first close — it must reappear (and remain hidden ONLY during immersive XR).
|
|
1352
|
+
- ❌ Wiring `closeOb` to a button that lacks the `data-ob-close` attribute.
|
|
1353
|
+
- ❌ Differentiating × from Continue behaviorally (e.g., "X means skip, Continue means save"). They are the same close path.
|
|
1354
|
+
|
|
1355
|
+
### Per-app content slots — the ONLY things you may vary
|
|
1356
|
+
|
|
1357
|
+
| Slot | Source | Notes |
|
|
1358
|
+
|---|---|---|
|
|
1359
|
+
| `{APP_NAME}` (`data-app-name`) | Derived from generated HTML filename | Override only to fix acronym capitalization (e.g., `data-app-name="AR Menu"` for `ar-menu.html`). |
|
|
1360
|
+
| `{APP_NAME_HEAD}` / `{APP_NAME_TAIL}` | App name split into leading word + trailing word(s). Trailing word gets the `<em>` accent. | Single-word names: HEAD is whole word, TAIL is empty (emit `<em></em>`). |
|
|
1361
|
+
| `{APP_TAGLINE}` | One-line description. MUST end with a period. MUST NOT exceed 90 characters. | Example: "Stack blocks with a flick of the wrist." |
|
|
1362
|
+
| `MUDRA_ONBOARDING_ACTIONS` | Per `actions-array.md`. | See filter rule below. |
|
|
1363
|
+
|
|
1364
|
+
### `MUDRA_ONBOARDING_ACTIONS` shape (renamed from feature-005 `ACTIONS`)
|
|
1365
|
+
|
|
1366
|
+
```js
|
|
1367
|
+
window.MUDRA_ONBOARDING_ACTIONS = [
|
|
1368
|
+
{ action: "Toggle XR session", mudra: "Thumb tap", manual: "Enter", mode: "gesture" },
|
|
1369
|
+
{ action: "Look around", mudra: "Lift wrist", manual: "Right-click + drag", mode: "imu_acc" },
|
|
1370
|
+
{ action: "Move", mudra: "Swipe", manual: "W A S D", mode: "nav_direction" }
|
|
1371
|
+
];
|
|
1372
|
+
```
|
|
1373
|
+
|
|
1374
|
+
Each row has four required fields:
|
|
1375
|
+
|
|
1376
|
+
- **`action`** — the behavior in plain English (NOT the control name).
|
|
1377
|
+
- **`mudra`** — the Mudra control prose (`"Tap"`, `"Twist"`, `"Press 70%"`, `"Tilt left"`).
|
|
1378
|
+
- **`manual`** — keyboard / mouse fallback. Use `"—"` (em dash) if there is no Manual equivalent (common in XR for camera / look controls).
|
|
1379
|
+
- **`mode`** — one of: `gesture` | `button` | `pressure` | `navigation` | `nav_direction` | `imu_acc` | `imu_gyro` | `snc`.
|
|
1380
|
+
|
|
1381
|
+
### App-aware filter — STRICT (feature 008, FR-010)
|
|
1382
|
+
|
|
1383
|
+
Before emitting `MUDRA_ONBOARDING_ACTIONS`, the skill MUST filter:
|
|
1384
|
+
|
|
1385
|
+
1. **Build the subscribed set** — every canonical signal this specific app subscribes to.
|
|
1386
|
+
2. **Drop every row** whose `mode` is not in the subscribed set. **Forbidden** to emit a row for an unsubscribed signal — that is a generation-time bug, not a runtime filter.
|
|
1387
|
+
3. **Verify exactly one motion mode** (`navigation` | `nav_direction` | `imu_acc`/`imu_gyro` family) across all rows (Constitution Principle III).
|
|
1388
|
+
4. **Verify Manual ↔ Mudra parity** — every row has both a Mudra and a Manual cell; `"—"` is the only acceptable Manual placeholder.
|
|
1389
|
+
5. **No collisions** — no two rows share the same `manual` value or the same effective `mudra` value.
|
|
1390
|
+
|
|
1391
|
+
### Anti-patterns (will fail review)
|
|
1392
|
+
|
|
1393
|
+
- ❌ Omitting `options.simulator.instructions.enabled = false;` — the XR Blocks default Welcome overlay then duplicates / overrides the Mudra onboarding modal.
|
|
1394
|
+
- ❌ Mirroring the modal as a 3D panel inside the XR scene (out of scope v1).
|
|
1395
|
+
- ❌ Editing the `xrsession-start` / `xrsession-end` hide-show wiring.
|
|
1396
|
+
- ❌ Emitting a row whose `mode` is not in the app's subscribed set.
|
|
1397
|
+
- ❌ Mixing two motion modes in the same array.
|
|
1398
|
+
- ❌ A row with `manual: null` or `manual: ""` — use `"—"`.
|
|
1399
|
+
- ❌ Two rows with the same `manual` value — keyboard collision.
|
|
1400
|
+
- ❌ Branding strings other than literal `Created by Mudra`. ("Mudra Studio" stays only as the `.ob-brand-mark` line above the app name.)
|
|
1401
|
+
- ❌ CTA labels other than `Continue`.
|
|
1402
|
+
|
|
1403
|
+
Binding contracts: `specs/008-strict-onboarding-templates/contracts/onboarding-block.md` and `specs/008-strict-onboarding-templates/contracts/actions-array.md`.
|
|
1404
|
+
|
|
1405
|
+
### AI-app extension (mandatory when `usesAI`)
|
|
1406
|
+
|
|
1407
|
+
The AI-Setup fragment is rendered as an **extra row inside `.ob-chip-grid`** — not as a separate section. The fragment goes at the **top** of the chip grid (above all action chips) so the user sees it before any controls.
|
|
1408
|
+
|
|
1409
|
+
```html
|
|
1410
|
+
<!-- Inject AS THE FIRST CHILD of #ob-rows when usesAI is true -->
|
|
1411
|
+
<div class="ob-chip ob-chip-ai" id="ob-chip-ai" data-role="ai-setup">
|
|
1412
|
+
<span class="nm">AI key</span>
|
|
1413
|
+
<input class="mudra-onb__ai-input" type="password" placeholder="AIza…" aria-label="Gemini API key" autocomplete="off" />
|
|
1414
|
+
<span class="mn" data-role="hint" aria-live="polite"></span>
|
|
1415
|
+
</div>
|
|
1416
|
+
```
|
|
1417
|
+
|
|
1418
|
+
```css
|
|
1419
|
+
/* Append to the locked onboarding-block CSS for AI-using apps only */
|
|
1420
|
+
.ob-chip-ai{grid-template-columns:auto 1fr auto;}
|
|
1421
|
+
.ob-chip-ai .mudra-onb__ai-input{appearance:none;background:rgba(255,255,255,0.04);color:var(--text);border:1px solid rgba(255,255,255,0.12);border-radius:8px;padding:6px 10px;font:inherit;font-size:13px;}
|
|
1422
|
+
.ob-chip-ai .mudra-onb__ai-input:focus{outline:2px solid var(--primary);outline-offset:0;}
|
|
1423
|
+
.ob-chip-ai .mn[data-role="hint"]{color:var(--warning);}
|
|
1424
|
+
```
|
|
1425
|
+
|
|
1426
|
+
The IIFE that wires the modal MUST be extended with this gating block, inserted just before the final unconditional `root.showModal();`:
|
|
1427
|
+
|
|
1428
|
+
```js
|
|
1429
|
+
// AI-Setup gating — only runs when the fragment is present (usesAI === true)
|
|
1430
|
+
const aiFragment = document.getElementById('ob-chip-ai');
|
|
1431
|
+
const continueBtn = root.querySelector('.ob-continue');
|
|
1432
|
+
const closeBtn = root.querySelector('.ob-x');
|
|
1433
|
+
const KEY_NAME = "mudra.gemini.apiKey";
|
|
1434
|
+
const KEY_REGEX = /^AIza[\w-]{30,}$/;
|
|
1435
|
+
|
|
1436
|
+
const hasValidStoredKey = () => {
|
|
1437
|
+
const k = sessionStorage.getItem(KEY_NAME);
|
|
1438
|
+
return typeof k === "string" && KEY_REGEX.test(k.trim());
|
|
1439
|
+
};
|
|
1440
|
+
|
|
1441
|
+
if (aiFragment) {
|
|
1442
|
+
const input = aiFragment.querySelector(".mudra-onb__ai-input");
|
|
1443
|
+
const hint = aiFragment.querySelector('[data-role="hint"]');
|
|
1444
|
+
|
|
1445
|
+
const refreshGate = () => {
|
|
1446
|
+
if (hasValidStoredKey()) {
|
|
1447
|
+
aiFragment.hidden = true;
|
|
1448
|
+
continueBtn.disabled = false;
|
|
1449
|
+
return;
|
|
1450
|
+
}
|
|
1451
|
+
aiFragment.hidden = false;
|
|
1452
|
+
const v = input.value.trim();
|
|
1453
|
+
const ok = KEY_REGEX.test(v);
|
|
1454
|
+
continueBtn.disabled = !ok;
|
|
1455
|
+
hint.textContent = (!v || ok) ? "" : "Key should start with \"AIza\" and be ~39 chars.";
|
|
1456
|
+
};
|
|
1457
|
+
|
|
1458
|
+
input.addEventListener("input", refreshGate);
|
|
1459
|
+
refreshGate();
|
|
1460
|
+
|
|
1461
|
+
continueBtn.addEventListener("click", () => {
|
|
1462
|
+
const v = input.value.trim();
|
|
1463
|
+
if (KEY_REGEX.test(v)) sessionStorage.setItem(KEY_NAME, v);
|
|
1464
|
+
}, { capture: true });
|
|
1465
|
+
|
|
1466
|
+
const blockClose = (e) => {
|
|
1467
|
+
if (hasValidStoredKey()) return;
|
|
1468
|
+
e.preventDefault();
|
|
1469
|
+
e.stopPropagation();
|
|
1470
|
+
input.focus();
|
|
1471
|
+
};
|
|
1472
|
+
closeBtn.addEventListener("click", blockClose, { capture: true });
|
|
1473
|
+
root.addEventListener("cancel", blockClose); // Escape
|
|
1474
|
+
}
|
|
1475
|
+
```
|
|
1476
|
+
|
|
1477
|
+
When the AI fragment is **absent** (non-AI app), the existing wiring runs untouched — `Continue` enables immediately, `Escape` / `×` dismiss as normal.
|
|
1478
|
+
|
|
1479
|
+
> **Migration note (feature 008, 2026-05-14):** the legacy `ACTIONS` variable is renamed to `MUDRA_ONBOARDING_ACTIONS`. The legacy `mudra-onb__*` class names from feature 005 are replaced by `ob-*` (the locked block's class names). The `Got it` CTA label is renamed to `Continue`. The AI-Setup fragment moves from `.mudra-onb__body` (separate section) into `.ob-chip-grid` (first chip).
|
|
1480
|
+
|
|
1481
|
+
---
|
|
1482
|
+
|
|
1483
|
+
## Section 18 — Visible AI Chat I/O (mandatory when `usesAI`)
|
|
1484
|
+
|
|
1485
|
+
Every AI-using app MUST render the conversation as on-screen text in the
|
|
1486
|
+
3D scene — TTS / speech synthesis is optional, never a substitute.
|
|
1487
|
+
|
|
1488
|
+
### Required visible elements
|
|
1489
|
+
|
|
1490
|
+
| Element | What it shows | How to render |
|
|
1491
|
+
|---------|---------------|---------------|
|
|
1492
|
+
| **Purpose line** | One short sentence stating what the app does ("Ask anything — I'll answer.", "Tell me the date.", etc.). Visible at all times. | Troika `Text` or a top row in an `xb.SpatialPanel`. |
|
|
1493
|
+
| **User input echo** | The latest user message (transcript from speech recognition OR typed text). Updates as soon as input is captured. | A bordered/highlighted row in the panel, e.g. prefixed `💬 You: …`. |
|
|
1494
|
+
| **AI response** | The most recent AI reply in full readable text. Scrollable / wrapping. | `xb.ScrollingTroikaTextView` or a tall row in the panel, prefixed `🤖 AI: …`. |
|
|
1495
|
+
| **Listening / Thinking indicator** | Distinguishes idle / listening / thinking states. | Avatar pulse + a single-word status line ("Listening…", "Thinking…", "Tap to talk"). |
|
|
1496
|
+
|
|
1497
|
+
### Rules
|
|
1498
|
+
|
|
1499
|
+
1. **Both sides of every exchange must be visible.** Voice-only output
|
|
1500
|
+
is a checklist failure. The chat panel must accumulate at least the
|
|
1501
|
+
last user turn AND the last AI turn at the same time.
|
|
1502
|
+
2. **The Purpose line is fixed** for the lifetime of the app. Author it
|
|
1503
|
+
from the user's prompt — e.g. `"Create 3d AI I can ask the date"` →
|
|
1504
|
+
Purpose `"Voice assistant — ask anything, tap to talk."`. Never use
|
|
1505
|
+
a placeholder like `"AI Chat"` alone.
|
|
1506
|
+
3. **Show typed input when speech recognition is unavailable.** If
|
|
1507
|
+
`webkitSpeechRecognition` / `SpeechRecognition` are missing, render
|
|
1508
|
+
an `<input type="text">` inside an XR Blocks panel OR a 2D overlay
|
|
1509
|
+
below the simulator panel. Both echo and reply still render in the
|
|
1510
|
+
3D scene.
|
|
1511
|
+
4. **When no API key is set yet**, the chat panel renders the literal
|
|
1512
|
+
text `Set up AI in the welcome panel` in the response slot — do not
|
|
1513
|
+
attempt any API call.
|
|
1514
|
+
5. **TTS** (`speechSynthesis`) is allowed but optional. If present, it
|
|
1515
|
+
speaks the AI response in addition to displaying it. Never as a
|
|
1516
|
+
replacement.
|