mudra-skills 3.1.2 → 3.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -1
- package/package.json +1 -1
- package/skills/mudra-master/SKILL.md +1 -1
- package/skills/mudra-master/mudra-preview/SKILL.md +39 -11
- package/skills/mudra-master/mudra-preview/references/agent_protocol.json +193 -27
- package/skills/mudra-master/mudra-preview/references/prompt.md +199 -37
- package/skills/mudra-master/mudra-xr/SKILL.md +30 -6
- package/skills/mudra-master/mudra-xr/references/agent_protocol.json +162 -21
- package/skills/mudra-master/mudra-xr/references/prompt.md +43 -8
- package/skills/mudra-preview/SKILL.md +39 -11
- package/skills/mudra-preview/references/agent_protocol.json +193 -27
- package/skills/mudra-preview/references/prompt.md +199 -37
- package/skills/mudra-xr/SKILL.md +30 -6
- package/skills/mudra-xr/references/agent_protocol.json +162 -21
- package/skills/mudra-xr/references/prompt.md +43 -8
|
@@ -26,10 +26,26 @@ UX feel, correct protocol usage, and a fast testing loop.
|
|
|
26
26
|
|
|
27
27
|
2. Select signals and enforce compatibility:
|
|
28
28
|
- Map discrete actions → `gesture` or `button`
|
|
29
|
-
- Map analog control → `
|
|
29
|
+
- Map analog control → `direct_pressure` (default) or
|
|
30
|
+
`pinch_pressure` — **exactly one**. There is no bare `pressure`
|
|
31
|
+
signal. Both are Finger pressure 0–100, normalized 0–1.
|
|
32
|
+
`direct_pressure` is the **new** continuous ungated stream —
|
|
33
|
+
always on, no gating. Requires firmware 6.0.12.11 and above.
|
|
34
|
+
`pinch_pressure` is the **original** tap-to-release filtered
|
|
35
|
+
stream: values stream only between tap and release (starts on tap,
|
|
36
|
+
falls off on release, stops until the next tap). Works on older
|
|
37
|
+
firmware. Pick `pinch_pressure` only for explicit commit-then-modulate
|
|
38
|
+
interactions (grab-and-scale, pinch-to-zoom, hold-to-charge).
|
|
30
39
|
- Map directional control → `navigation`
|
|
31
40
|
- Map directional gestures → `nav_direction`
|
|
32
|
-
- Map orientation/
|
|
41
|
+
- Map **hand orientation / aiming / heading / 1:1 rotation** →
|
|
42
|
+
`imu_quaternion`. Requires firmware 6.0.12.11 and above.
|
|
43
|
+
**Note:** `data.values` is a **list of samples**,
|
|
44
|
+
each a 4-element `[w, x, y, z]` unit quaternion. This is one nesting
|
|
45
|
+
level deeper than `imu_acc`/`imu_gyro`. Read the latest with
|
|
46
|
+
`values.at(-1)`. Standalone signal — combines with anything,
|
|
47
|
+
including `navigation` and `nav_direction`.
|
|
48
|
+
- Map raw inertial data / shake / acceleration → `imu_acc` + `imu_gyro`
|
|
33
49
|
- Map biometric use cases → `emg` — **note**: EMG data arrives
|
|
34
50
|
as 3 de-interleaved channel arrays
|
|
35
51
|
`[[ch1_samples], [ch2_samples], [ch3_samples]]`, not a flat
|
|
@@ -44,9 +60,16 @@ UX feel, correct protocol usage, and a fast testing loop.
|
|
|
44
60
|
- Subscribe **one signal per command**, using the key `signal` (singular):
|
|
45
61
|
`{ "command": "subscribe", "signal": "<name>" }`
|
|
46
62
|
- **NEVER** use `signals` (plural), arrays, or batch subscribe commands
|
|
47
|
-
- Valid subscribable signals (
|
|
48
|
-
`gesture`, `button`, `
|
|
49
|
-
`nav_direction`, `imu_acc`, `imu_gyro`,
|
|
63
|
+
- Valid subscribable signals (10 total):
|
|
64
|
+
`gesture`, `button`, `direct_pressure`, `pinch_pressure`,
|
|
65
|
+
`navigation`, `nav_direction`, `imu_acc`, `imu_gyro`,
|
|
66
|
+
`imu_quaternion`, `emg`
|
|
67
|
+
- **`pressure` is NOT a valid signal name.** It was the old name for
|
|
68
|
+
`pinch_pressure`. `direct_pressure` is a **new** continuous stream,
|
|
69
|
+
not a rename of `pressure`. Sending `pressure` returns
|
|
70
|
+
`error: invalid_signal` and the app receives nothing. The frame
|
|
71
|
+
`type` mirrors the signal name, so handlers must match
|
|
72
|
+
`direct_pressure` / `pinch_pressure` too.
|
|
50
73
|
- Full command surface: `subscribe`, `unsubscribe`,
|
|
51
74
|
`get_subscriptions`, `get_status`, `status`, `get_device_info`,
|
|
52
75
|
`trigger_gesture`
|
|
@@ -73,20 +96,41 @@ UX feel, correct protocol usage, and a fast testing loop.
|
|
|
73
96
|
Unless the user explicitly asks for a different signal, every generated
|
|
74
97
|
app MUST restrict itself to **at most these four signals**:
|
|
75
98
|
|
|
76
|
-
1. `pressure
|
|
99
|
+
1. `direct_pressure` (or `pinch_pressure` — exactly one pressure mode;
|
|
100
|
+
see the Pressure mode rule below)
|
|
77
101
|
2. `gesture` filtered to **`tap`** only
|
|
78
102
|
3. `gesture` filtered to **`double_tap`** only
|
|
79
103
|
4. **One** directional signal — either `nav_direction` **or** `navigation`,
|
|
80
104
|
never both in the same app
|
|
81
105
|
|
|
106
|
+
**Pressure mode rule.** There is no signal called `pressure`; that was
|
|
107
|
+
the old name for `pinch_pressure` and now returns `invalid_signal`.
|
|
108
|
+
`direct_pressure` is a **new** signal, not a split of the old one.
|
|
109
|
+
|
|
110
|
+
- `direct_pressure` — Finger pressure 0–100, normalized 0–1. **New**
|
|
111
|
+
continuous ungated stream — always on, no tap/release gating.
|
|
112
|
+
**This is the default.** Requires firmware 6.0.12.11 and above. Use
|
|
113
|
+
it whenever the user just says "pressure" or names an analog synonym
|
|
114
|
+
(volume, brush, throttle, zoom, intensity, opacity).
|
|
115
|
+
- `pinch_pressure` — Finger pressure 0–100, normalized 0–1. The
|
|
116
|
+
**original** tap-to-release filtered stream. Values stream only
|
|
117
|
+
between tap and release: streaming starts on tap, the value falls off
|
|
118
|
+
on release, and streaming stops until the next tap. Works on older
|
|
119
|
+
firmware. Use it only when the interaction is explicitly
|
|
120
|
+
commit-then-modulate: grab-and-scale, pinch-to-zoom,
|
|
121
|
+
pinch-and-hold-to-charge.
|
|
122
|
+
|
|
123
|
+
Never subscribe to both — they are mutually exclusive. Do not ask the
|
|
124
|
+
user which mode they want; pick the default and say so in one clause.
|
|
125
|
+
|
|
82
126
|
Drop any of the four when the concept does not need it (e.g. a pure
|
|
83
127
|
tap-counter subscribes to `gesture` only). All other signals
|
|
84
|
-
(`button`, `imu_acc`, `imu_gyro`, `emg`) and other gesture subtypes
|
|
128
|
+
(`button`, `imu_acc`, `imu_gyro`, `imu_quaternion`, `emg`) and other gesture subtypes
|
|
85
129
|
(`twist`, `double_twist`, …) are **off by default** — only include them
|
|
86
130
|
when the user names them, names a synonym from the Signal Inference
|
|
87
131
|
Reference below, or describes an interaction that genuinely cannot be
|
|
88
132
|
expressed with the default four (e.g. "tilt to steer" → IMU,
|
|
89
|
-
"hold to charge" → `button`).
|
|
133
|
+
"aim at the target" → `imu_quaternion`, "hold to charge" → `button`).
|
|
90
134
|
|
|
91
135
|
The simulator panel must mirror whichever subset the app actually
|
|
92
136
|
subscribes to — never render buttons for signals that are not wired.
|
|
@@ -99,6 +143,56 @@ subscribes to — never render buttons for signals that are not wired.
|
|
|
99
143
|
- **Direction mode**: `nav_direction`
|
|
100
144
|
- **IMU+Biometric bundle**: `imu_acc` + `imu_gyro` + `emg` (always all three)
|
|
101
145
|
|
|
146
|
+
`imu_quaternion` (Hand Orientation) sits **outside** this grouping — see
|
|
147
|
+
"Hand Orientation" below. It is not a mode and does not participate in
|
|
148
|
+
any XOR.
|
|
149
|
+
|
|
150
|
+
### Hand Orientation — `imu_quaternion` (standalone)
|
|
151
|
+
|
|
152
|
+
Absolute, drift-free hand orientation as a stream of unit quaternions.
|
|
153
|
+
Use it for aiming, heading, pose gating, and 1:1 rotation of a 3D or 2D
|
|
154
|
+
object. Prefer it over integrating `imu_gyro`: no drift correction, no
|
|
155
|
+
sensor fusion on your side, and it does **not** drag in the biometric
|
|
156
|
+
bundle.
|
|
157
|
+
|
|
158
|
+
Requires firmware 6.0.12.11 and above.
|
|
159
|
+
|
|
160
|
+
It belongs to no bundle and no mode. It combines freely with every other
|
|
161
|
+
signal — including `navigation` and `nav_direction`, which the
|
|
162
|
+
IMU+Biometric bundle cannot.
|
|
163
|
+
|
|
164
|
+
**Payload shape — read this carefully.** `data.values` is a **list of
|
|
165
|
+
samples**, each sample a 4-element array `[w, x, y, z]`. That is one
|
|
166
|
+
nesting level deeper than `imu_acc`/`imu_gyro`, whose `values` is three
|
|
167
|
+
flat per-axis arrays. Do not assume the two shapes match.
|
|
168
|
+
|
|
169
|
+
```js
|
|
170
|
+
// CORRECT — take a sample first, then destructure it
|
|
171
|
+
if (msg.type === 'imu_quaternion') {
|
|
172
|
+
const [w, x, y, z] = msg.data.values.at(-1); // latest sample
|
|
173
|
+
applyOrientation(w, x, y, z);
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
// WRONG — treats values as one flat quaternion
|
|
177
|
+
const [w, x, y, z] = msg.data.values; // these are whole samples, not components
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Each sample is a unit quaternion, so `w² + x² + y² + z² ≈ 1.0`. If your
|
|
181
|
+
norms scatter away from 1, you are reading the array at the wrong depth.
|
|
182
|
+
|
|
183
|
+
Implementation notes:
|
|
184
|
+
|
|
185
|
+
- Read the latest sample per animation frame; use all samples only when
|
|
186
|
+
you need the full motion trace.
|
|
187
|
+
- Slerp toward the incoming value (factor ~0.2) rather than snapping, to
|
|
188
|
+
absorb packet jitter.
|
|
189
|
+
- Latch a reference quaternion on a tap so the user can zero the pose in
|
|
190
|
+
a comfortable position, then show rotation relative to it.
|
|
191
|
+
- Converting to a three.js quaternion swaps the argument order:
|
|
192
|
+
`new THREE.Quaternion(x, y, z, w)`.
|
|
193
|
+
- For a 2D readout, convert to Euler angles at render time only — never
|
|
194
|
+
store Euler state.
|
|
195
|
+
|
|
102
196
|
### Bundling rule — IMU+Biometric (CRITICAL)
|
|
103
197
|
|
|
104
198
|
`imu_acc`, `imu_gyro`, and `emg` are an **inseparable bundle**. If the
|
|
@@ -118,14 +212,21 @@ ws.send(JSON.stringify({ command: 'subscribe', signal: 'imu_acc' })); // miss
|
|
|
118
212
|
|
|
119
213
|
### XOR rules (all non-negotiable)
|
|
120
214
|
|
|
121
|
-
1. **`gesture` ⊕
|
|
122
|
-
|
|
123
|
-
|
|
215
|
+
1. **`gesture` ⊕ pressure** — pick one; never combine `gesture` with
|
|
216
|
+
`direct_pressure` or `pinch_pressure`.
|
|
217
|
+
2. **`direct_pressure` ⊕ `pinch_pressure`** — exactly one pressure
|
|
218
|
+
signal per app; they are mutually exclusive.
|
|
219
|
+
3. **`navigation` ⊕ `nav_direction`** — pick one; never combine them.
|
|
220
|
+
4. **(`navigation` or `nav_direction`) ⊕ IMU+Biometric bundle** — directional
|
|
124
221
|
motion signals cannot be combined with the IMU+Biometric bundle (`imu_acc`/`imu_gyro`/`emg`).
|
|
125
222
|
|
|
223
|
+
`imu_quaternion` appears in none of these rules — it is exempt from all
|
|
224
|
+
of them.
|
|
225
|
+
|
|
126
226
|
### Never combine
|
|
127
227
|
|
|
128
|
-
- `gesture` + `
|
|
228
|
+
- `gesture` + `direct_pressure` / `pinch_pressure`
|
|
229
|
+
- `direct_pressure` + `pinch_pressure`
|
|
129
230
|
- `navigation` + `nav_direction`
|
|
130
231
|
- `navigation` + `imu_acc` / `imu_gyro` / `emg`
|
|
131
232
|
- `nav_direction` + `imu_acc` / `imu_gyro` / `emg`
|
|
@@ -137,6 +238,9 @@ ws.send(JSON.stringify({ command: 'subscribe', signal: 'imu_acc' })); // miss
|
|
|
137
238
|
above — `button` belongs to Pointer mode and never combines with
|
|
138
239
|
`nav_direction`).
|
|
139
240
|
|
|
241
|
+
`imu_quaternion` combines freely with everything, with no exceptions —
|
|
242
|
+
including `navigation` and `nav_direction`.
|
|
243
|
+
|
|
140
244
|
When a conflict appears, explain the limitation and recommend one path.
|
|
141
245
|
|
|
142
246
|
## Sample Catalog
|
|
@@ -219,15 +323,20 @@ real connection fails. This is not optional — without it the app is broken in
|
|
|
219
323
|
|
|
220
324
|
```
|
|
221
325
|
// WRONG — plural key
|
|
222
|
-
{ "command": "subscribe", "signals": ["gesture", "
|
|
326
|
+
{ "command": "subscribe", "signals": ["gesture", "navigation"] }
|
|
223
327
|
|
|
224
328
|
// WRONG — array value
|
|
225
|
-
{ "command": "subscribe", "signal": ["gesture", "
|
|
329
|
+
{ "command": "subscribe", "signal": ["gesture", "navigation"] }
|
|
226
330
|
|
|
227
331
|
// WRONG — enable/disable commands do not exist in the new server
|
|
228
|
-
{ "command": "enable", "data": { "signals": ["gesture", "
|
|
332
|
+
{ "command": "enable", "data": { "signals": ["gesture", "navigation"] } }
|
|
229
333
|
// → Use { "command": "subscribe", "signal": "gesture" } instead
|
|
230
334
|
|
|
335
|
+
// WRONG — 'pressure' is the old name for pinch_pressure; it is not a signal
|
|
336
|
+
{ "command": "subscribe", "signal": "pressure" }
|
|
337
|
+
// → error: invalid_signal. Use "direct_pressure" (new continuous stream)
|
|
338
|
+
// or "pinch_pressure" (original tap-to-release stream).
|
|
339
|
+
|
|
231
340
|
// WRONG — raw WebSocket
|
|
232
341
|
const ws = new WebSocket('ws://127.0.0.1:8766');
|
|
233
342
|
```
|
|
@@ -235,7 +344,8 @@ const ws = new WebSocket('ws://127.0.0.1:8766');
|
|
|
235
344
|
```
|
|
236
345
|
// CORRECT — one subscribe per signal
|
|
237
346
|
ws.send(JSON.stringify({ "command": "subscribe", "signal": "gesture" }));
|
|
238
|
-
ws.send(JSON.stringify({ "command": "subscribe", "signal": "
|
|
347
|
+
ws.send(JSON.stringify({ "command": "subscribe", "signal": "direct_pressure" }));
|
|
348
|
+
ws.send(JSON.stringify({ "command": "subscribe", "signal": "imu_quaternion" }));
|
|
239
349
|
|
|
240
350
|
// CORRECT — always use MudraWebSocket
|
|
241
351
|
const ws = new MudraWebSocket('ws://127.0.0.1:8766');
|
|
@@ -299,7 +409,7 @@ class MudraWebSocket {
|
|
|
299
409
|
device: { name: 'Mudra Band (sim)', address: '00:00:00:00', battery: 85, charging: false,
|
|
300
410
|
firmware: '6.0.0.0', serial_number: 1000000, hand: 'RIGHT', state: 'connected',
|
|
301
411
|
firmware_config: { target: 'BandMode.mudraLink', active: false } },
|
|
302
|
-
subscriptions: Object.fromEntries(['emg','imu_acc','imu_gyro','
|
|
412
|
+
subscriptions: Object.fromEntries(['emg','imu_acc','imu_gyro','imu_quaternion','direct_pressure','pinch_pressure','gesture','navigation','nav_direction','button'].map(s => [s, this._subscriptions.has(s)]))
|
|
303
413
|
}, timestamp: Date.now() });
|
|
304
414
|
}
|
|
305
415
|
}
|
|
@@ -319,13 +429,25 @@ class MudraWebSocket {
|
|
|
319
429
|
this._emit({ type: 'gesture', data: { type, timestamp: Date.now() }, timestamp: Date.now() });
|
|
320
430
|
}, 3000));
|
|
321
431
|
|
|
322
|
-
// Pressure: sine wave at 20 Hz
|
|
432
|
+
// Pressure: sine wave at 20 Hz. Emits under whichever pressure mode is subscribed
|
|
433
|
+
// ('direct_pressure' or 'pinch_pressure') — never both, they are mutually exclusive.
|
|
323
434
|
let t = 0;
|
|
324
435
|
this._timers.push(setInterval(() => {
|
|
325
|
-
|
|
436
|
+
const mode = ['direct_pressure', 'pinch_pressure'].find(s => this._subscriptions.has(s));
|
|
437
|
+
if (!mode) return;
|
|
326
438
|
t += 0.05;
|
|
327
439
|
const norm = (Math.sin(t) + 1) / 2;
|
|
328
|
-
this._emit({ type:
|
|
440
|
+
this._emit({ type: mode, data: { value: Math.round(norm * 100), normalized: norm, timestamp: Date.now() }, timestamp: Date.now() });
|
|
441
|
+
}, 50));
|
|
442
|
+
|
|
443
|
+
// Hand orientation: slow yaw sweep at 20 Hz.
|
|
444
|
+
// NOTE the shape — `values` is a LIST of [w, x, y, z] samples, not a flat quaternion.
|
|
445
|
+
let yaw = 0;
|
|
446
|
+
this._timers.push(setInterval(() => {
|
|
447
|
+
if (!this._subscriptions.has('imu_quaternion')) return;
|
|
448
|
+
yaw = (yaw + 0.02) % (Math.PI * 2);
|
|
449
|
+
const w = Math.cos(yaw / 2), y = Math.sin(yaw / 2);
|
|
450
|
+
this._emit({ type: 'imu_quaternion', data: { values: [[w, 0, y, 0]], frequency: 50, frequency_std: 0.4, timestamp: Date.now() }, timestamp: Date.now() });
|
|
329
451
|
}, 50));
|
|
330
452
|
|
|
331
453
|
// Navigation: small random deltas at 20 Hz
|
|
@@ -487,9 +609,12 @@ Outbound:
|
|
|
487
609
|
- Re-issue ALL subscribes on every (re-)connect.
|
|
488
610
|
|
|
489
611
|
Inbound shape: `{ "type": "...", "data": { ... }, "timestamp": <ms> }`.
|
|
490
|
-
Handle types: `gesture`, `
|
|
491
|
-
`
|
|
612
|
+
Handle types: `gesture`, `direct_pressure`, `pinch_pressure`,
|
|
613
|
+
`navigation`, `nav_direction`, `button`, `imu_acc`, `imu_gyro`,
|
|
614
|
+
`imu_quaternion`, `emg`, `status`, `device_info`,
|
|
492
615
|
`subscription_status`, `subscriptions`, `airtouch_state`, `error`.
|
|
616
|
+
The frame `type` always mirrors the subscribed signal name — there is no
|
|
617
|
+
`pressure` frame type.
|
|
493
618
|
The server sends NO `connection_status` frame. Anything else: log + ignore.
|
|
494
619
|
|
|
495
620
|
### Disconnect detection — band state via `get_status` polling (mandatory)
|
|
@@ -615,8 +740,10 @@ Every generated app MUST render these three elements at all times:
|
|
|
615
740
|
```js
|
|
616
741
|
function dispatch(msg) {
|
|
617
742
|
switch (msg.type) {
|
|
618
|
-
case "gesture":
|
|
619
|
-
case "
|
|
743
|
+
case "gesture": handleGesture(msg.data); break;
|
|
744
|
+
case "direct_pressure":
|
|
745
|
+
case "pinch_pressure": handlePressure(msg.data); break; // one mode only per app
|
|
746
|
+
case "imu_quaternion": handleOrientation(msg.data.values.at(-1)); break; // [w,x,y,z]
|
|
620
747
|
case "navigation": handleNavigation(msg.data); break;
|
|
621
748
|
case "nav_direction": handleNavDirection(msg.data); break;
|
|
622
749
|
case "button": handleButton(msg.data); break;
|
|
@@ -703,9 +830,10 @@ direction (Manual→Mudra and Mudra→Manual):
|
|
|
703
830
|
|
|
704
831
|
| Signal | Reset value |
|
|
705
832
|
|--------|-------------|
|
|
706
|
-
| `
|
|
833
|
+
| `direct_pressure` / `pinch_pressure` | `{ value: 0, normalized: 0.0 }` |
|
|
707
834
|
| `imu_acc` | `[0, 0, 9.81]` (gravity at rest) |
|
|
708
835
|
| `imu_gyro` | `[0, 0, 0]` |
|
|
836
|
+
| `imu_quaternion` | identity quaternion `[1, 0, 0, 0]` (no rotation) |
|
|
709
837
|
| `navigation` accumulated cursor | app-defined origin (e.g. canvas centre) |
|
|
710
838
|
| `emg` rolling buffers | cleared (`[[], [], []]`) |
|
|
711
839
|
|
|
@@ -786,7 +914,9 @@ side-effect ordering intact:
|
|
|
786
914
|
|
|
787
915
|
```javascript
|
|
788
916
|
// ── State ─────────────────────────────────────────────────────────────────
|
|
789
|
-
const SUBSCRIBED_SIGNALS = ["gesture", "
|
|
917
|
+
const SUBSCRIBED_SIGNALS = ["gesture", "imu_quaternion"]; // adapt per app
|
|
918
|
+
// Pressure apps use "direct_pressure" (default) or "pinch_pressure" — never both,
|
|
919
|
+
// and never alongside "gesture". There is no bare "pressure" signal.
|
|
790
920
|
let mode = "manual";
|
|
791
921
|
let connectionState = "idle";
|
|
792
922
|
let socket = null;
|
|
@@ -833,7 +963,7 @@ function stopStatusPoll() {
|
|
|
833
963
|
// ── Continuous-state reset ───────────────────────────────────────────────
|
|
834
964
|
function resetContinuousState() {
|
|
835
965
|
// adapt per app
|
|
836
|
-
// pressure, imu, cursor, emg buffers → neutral
|
|
966
|
+
// pressure, imu, quaternion (→ identity [1,0,0,0]), cursor, emg buffers → neutral
|
|
837
967
|
}
|
|
838
968
|
|
|
839
969
|
// ── Mode change ──────────────────────────────────────────────────────────
|
|
@@ -910,8 +1040,10 @@ Pick exactly ONE motion mode per app: **Pointer** (`navigation` + `button`)
|
|
|
910
1040
|
**XOR** **Direction** (`nav_direction`) **XOR** **IMU+Biometric**
|
|
911
1041
|
(`imu_acc` + `imu_gyro` + `emg`, always all three together). The Mode
|
|
912
1042
|
toggle does NOT relax this rule. Additional XOR rules: `gesture` and
|
|
913
|
-
|
|
1043
|
+
pressure are mutually exclusive — never combine them; `direct_pressure`
|
|
1044
|
+
and `pinch_pressure` are mutually exclusive with each other. `button`
|
|
914
1045
|
combines freely (subject to the Pointer/Direction/IMU XOR).
|
|
1046
|
+
`imu_quaternion` is exempt from every XOR and combines with anything. Requires firmware 6.0.12.11 and above. `direct_pressure` Requires firmware 6.0.12.11 and above.
|
|
915
1047
|
|
|
916
1048
|
---
|
|
917
1049
|
|
|
@@ -1116,7 +1248,7 @@ window.MUDRA_ONBOARDING_ACTIONS = [
|
|
|
1116
1248
|
```js
|
|
1117
1249
|
window.MUDRA_ONBOARDING_ACTIONS = [
|
|
1118
1250
|
{ action: "Trigger pad", mudra: "Tap", manual: "Space", mode: "gesture" },
|
|
1119
|
-
{ action: "Adjust volume", mudra: "Press 70%", manual: "[ / ]", mode: "
|
|
1251
|
+
{ action: "Adjust volume", mudra: "Press 70%", manual: "[ / ]", mode: "direct_pressure" },
|
|
1120
1252
|
{ action: "Cycle pad bank", mudra: "Twist", manual: "Tab", mode: "gesture" }
|
|
1121
1253
|
];
|
|
1122
1254
|
```
|
|
@@ -1126,7 +1258,7 @@ Each row has four required fields:
|
|
|
1126
1258
|
- **`action`** — the behavior in plain English. NOT the control name.
|
|
1127
1259
|
- **`mudra`** — the Mudra-control prose (e.g., `"Tap"`, `"Twist"`, `"Press 70%"`, `"Tilt left"`).
|
|
1128
1260
|
- **`manual`** — the keyboard / mouse fallback (`"Space"`, `"Shift + ←"`, `"[ / ]"`). Use `"—"` (em dash) if no Manual equivalent exists.
|
|
1129
|
-
- **`mode`** — one of the
|
|
1261
|
+
- **`mode`** — one of the ten canonical signal names: `gesture` | `button` | `direct_pressure` | `pinch_pressure` | `navigation` | `nav_direction` | `imu_acc` | `imu_gyro` | `imu_quaternion` | `emg`. The skill uses this for the filter rule below. `pressure` is **not** a canonical name.
|
|
1130
1262
|
|
|
1131
1263
|
### App-aware filter — STRICT (feature 008, FR-010)
|
|
1132
1264
|
|
|
@@ -1140,11 +1272,12 @@ Before emitting `MUDRA_ONBOARDING_ACTIONS`, the skill MUST filter:
|
|
|
1140
1272
|
|
|
1141
1273
|
#### Anti-patterns (will fail review)
|
|
1142
1274
|
|
|
1143
|
-
- ❌ Emitting a row with `mode: "
|
|
1275
|
+
- ❌ Emitting a row with `mode: "direct_pressure"` when the app does not subscribe to `direct_pressure`.
|
|
1276
|
+
- ❌ Emitting `mode: "pressure"` at all — it is not a canonical signal name. Use `direct_pressure` or `pinch_pressure`.
|
|
1144
1277
|
- ❌ Mixing two motion modes (e.g., `nav_direction` AND `imu_acc`) in the same array.
|
|
1145
1278
|
- ❌ A row with `manual: null` or `manual: ""` — use `"—"` if no fallback exists.
|
|
1146
1279
|
- ❌ Two rows with `manual: "Space"` — keyboard collision.
|
|
1147
|
-
- ❌ Renaming a canonical signal in `mode` (e.g., `"squeeze"` instead of `"
|
|
1280
|
+
- ❌ Renaming a canonical signal in `mode` (e.g., `"squeeze"` instead of `"pinch_pressure"`, or `"orientation"` instead of `"imu_quaternion"`).
|
|
1148
1281
|
- ❌ `action: "Press Space"` — that's a control, not a behavior. Use `action: "Fire"`.
|
|
1149
1282
|
|
|
1150
1283
|
### Forbidden — never emit
|
|
@@ -1194,32 +1327,61 @@ Use this as the default behavior for intent-to-signal mapping.
|
|
|
1194
1327
|
|
|
1195
1328
|
- `gesture`: tap, click, trigger, action, button press, drum, hit, select
|
|
1196
1329
|
- `button`: hold, press and hold, drag, push-to-talk, sprint, charge
|
|
1197
|
-
- `pressure
|
|
1330
|
+
- `direct_pressure` (Finger pressure 0–100, normalized 0–1; **new** continuous ungated stream; default). Requires firmware 6.0.12.11 and above. Synonyms: pressure, slide, volume, size, intensity, throttle, opacity, brush, zoom, analog, force, press harder
|
|
1331
|
+
- `pinch_pressure` (Finger pressure 0–100, normalized 0–1; **original** tap-to-release filtered stream; works on older firmware): pinch, squeeze, pinch and hold, tap then squeeze, grab and scale, pinch-to-zoom, hold to charge
|
|
1198
1332
|
- `navigation`: move, up/down, left/right, steer, cursor, pan, scroll, direction, arrow
|
|
1199
1333
|
- `nav_direction`: swipe, directional gesture, menu direction, card swipe, flick — directions: None, Right, Left, Up, Down, Roll Left, Roll Right (+ reverse variants)
|
|
1200
|
-
- `
|
|
1334
|
+
- `imu_quaternion` (**Hand Orientation** — standalone, combines with anything). Requires firmware 6.0.12.11 and above. Synonyms: hand orientation, wrist orientation, absolute orientation, aim, point at, heading, which way the hand is pointing, roll/pitch/yaw, quaternion, 1:1 rotation
|
|
1335
|
+
- `imu_acc + imu_gyro + emg` (single bundle — always subscribe to all three): tilt, shake, acceleration, balance, level, muscle, EMG, biometric, fatigue, nerve
|
|
1201
1336
|
|
|
1202
1337
|
### Bundling Rule
|
|
1203
1338
|
|
|
1204
1339
|
`imu_acc`, `imu_gyro`, and `emg` are an inseparable bundle. If the user
|
|
1205
1340
|
wants any one of them, subscribe to all three.
|
|
1206
1341
|
|
|
1342
|
+
`imu_quaternion` is **not** part of that bundle. It is independently
|
|
1343
|
+
subscribable and exempt from every XOR rule. Requires firmware 6.0.12.11 and above.
|
|
1344
|
+
|
|
1345
|
+
### Pressure Mode Rule
|
|
1346
|
+
|
|
1347
|
+
There is no `pressure` signal — that was the old name for
|
|
1348
|
+
`pinch_pressure` and sending it returns `invalid_signal`.
|
|
1349
|
+
Both signals are Finger pressure 0–100, normalized 0–1.
|
|
1350
|
+
`direct_pressure` is a **new** continuous ungated stream (default).
|
|
1351
|
+
Requires firmware 6.0.12.11 and above. `pinch_pressure` is the
|
|
1352
|
+
**original** tap-to-release filtered stream and works on older
|
|
1353
|
+
firmware. Pick exactly one. Do not ask the user which; infer it and
|
|
1354
|
+
state the choice in one clause.
|
|
1355
|
+
|
|
1207
1356
|
### Ambiguity Rules
|
|
1208
1357
|
|
|
1209
1358
|
When concept could map to either `navigation` or the IMU+Biometric bundle (`imu_acc + imu_gyro + emg`), ask one clarifying question and recommend the better fit:
|
|
1210
1359
|
- use `navigation` (+`button`) for continuous directional movement/cursor/panning/drag
|
|
1211
|
-
- use the IMU+Biometric bundle (`imu_acc + imu_gyro + emg`) for
|
|
1360
|
+
- use the IMU+Biometric bundle (`imu_acc + imu_gyro + emg`) for shake/acceleration/biometrics
|
|
1361
|
+
|
|
1362
|
+
**Check for the orientation escape hatch first.** If the user only needs
|
|
1363
|
+
to know *where the hand is pointing* — aiming, heading, 1:1 rotation,
|
|
1364
|
+
pose gating — this is not a conflict at all: use `imu_quaternion`, which
|
|
1365
|
+
combines with `navigation` and `nav_direction` and needs no bundle. Only
|
|
1366
|
+
fall back to the clarifying question when the concept genuinely wants raw
|
|
1367
|
+
acceleration or muscle data alongside directional movement.
|
|
1212
1368
|
|
|
1213
1369
|
When concept could use either `navigation` or `nav_direction`, pick based on control style:
|
|
1214
1370
|
- use `navigation` (+`button`) for **continuous** pointer/cursor control (smooth deltas)
|
|
1215
1371
|
- use `nav_direction` for **discrete** directional gestures (swipe-like, menu selection)
|
|
1216
1372
|
- these cannot be combined (same physical hand movement)
|
|
1217
1373
|
|
|
1218
|
-
When concept could use either `gesture` or
|
|
1374
|
+
When concept could use either `gesture` or a pressure signal, pick based on control style:
|
|
1219
1375
|
- use `gesture` for **discrete** actions (tap, double-tap, twist)
|
|
1220
|
-
- use `
|
|
1376
|
+
- use `direct_pressure` for **analog** control (volume, brush size, throttle)
|
|
1221
1377
|
- these cannot be combined — pick one interaction model per app
|
|
1222
1378
|
|
|
1379
|
+
When the concept is analog, pick the pressure mode without asking:
|
|
1380
|
+
- **commit-then-modulate** ("tap to grab, then squeeze to resize",
|
|
1381
|
+
"pinch and hold to charge", "pinch to zoom") → `pinch_pressure`
|
|
1382
|
+
- everything else → `direct_pressure`
|
|
1383
|
+
- never both — they are mutually exclusive
|
|
1384
|
+
|
|
1223
1385
|
---
|
|
1224
1386
|
|
|
1225
1387
|
## Sample Apps
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: mudra-xr
|
|
3
|
-
version: 3.
|
|
3
|
+
version: 3.2.0
|
|
4
4
|
description: Generate a single-file Mudra-controlled 3D/XR app using XR Blocks. Use when the user describes a 3D, XR, VR, or AR experience controlled by the Mudra Band.
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -75,16 +75,38 @@ Map the user's intent to the required Mudra signals using the Signal Inference R
|
|
|
75
75
|
in `references/promt.md`. Enforce all grouping rules (Section 8 of promt.md):
|
|
76
76
|
|
|
77
77
|
- Discrete actions → `gesture` OR `button` (never both gesture+pressure)
|
|
78
|
-
- Analog control → `pressure` OR `button` (never both gesture+pressure)
|
|
78
|
+
- Analog control → `direct_pressure` (Finger pressure 0–100, normalized 0–1; new continuous ungated stream; default) OR `pinch_pressure` (Finger pressure 0–100, normalized 0–1; original tap-to-release filtered stream) OR `button` (never both gesture+pressure). `direct_pressure` Requires firmware 6.0.12.11 and above.
|
|
79
|
+
- Hand orientation / aiming / 1:1 rotation → `imu_quaternion` (standalone — no motion mode required). Requires firmware 6.0.12.11 and above.
|
|
79
80
|
- Continuous directional movement → `navigation` + `button` (Pointer mode)
|
|
80
81
|
- Discrete directional swipes → `nav_direction` (Direction mode)
|
|
81
|
-
-
|
|
82
|
+
- Shake / acceleration / biometrics → `imu_acc` + `imu_gyro` + `emg` (always all three together — IMU+Biometric bundle)
|
|
82
83
|
|
|
83
84
|
**Critical grouping rules:**
|
|
84
|
-
1. `gesture` and
|
|
85
|
+
1. `gesture` and pressure are mutually exclusive — pick one.
|
|
85
86
|
2. `navigation` and `nav_direction` are mutually exclusive — pick one.
|
|
86
87
|
3. The IMU+Biometric bundle (`imu_acc` + `imu_gyro` + `emg`) cannot combine with `navigation` or `nav_direction`.
|
|
87
88
|
4. `imu_acc`, `imu_gyro`, and `emg` are always subscribed together — using any one requires all three.
|
|
89
|
+
4a. **Pressure: pick exactly one; there is no bare `pressure` signal.**
|
|
90
|
+
Both are Finger pressure 0–100, normalized 0–1.
|
|
91
|
+
`direct_pressure` is the **new** continuous ungated stream —
|
|
92
|
+
always on, no tap/release gating. **Default.** Requires firmware
|
|
93
|
+
6.0.12.11 and above. `pinch_pressure` is the **original**
|
|
94
|
+
tap-to-release filtered stream (formerly named `pressure`): values
|
|
95
|
+
stream only between tap and release (starts on tap, falls off on
|
|
96
|
+
release, stops until the next tap). Works on older firmware. Use
|
|
97
|
+
`pinch_pressure` only for explicit commit-then-modulate
|
|
98
|
+
(grab-and-scale / pinch-to-zoom / hold-to-charge). Do not ask which
|
|
99
|
+
— infer it and state the choice in one clause. Sending `pressure`
|
|
100
|
+
returns `invalid_signal`.
|
|
101
|
+
4b. **`imu_quaternion` (Hand Orientation) is exempt from rules 1–4.** It
|
|
102
|
+
is standalone, belongs to no bundle, requires no motion mode, and
|
|
103
|
+
combines freely with everything — including `navigation` and
|
|
104
|
+
`nav_direction`, which the IMU+Biometric bundle cannot. Use it for
|
|
105
|
+
aiming, ray direction, 1:1 mesh rotation, pose gating, and heading.
|
|
106
|
+
Requires firmware 6.0.12.11 and above. Its payload nests one level
|
|
107
|
+
deeper than the other IMU signals: `data.values` is a **list of
|
|
108
|
+
`[w, x, y, z]` samples** — read the latest with `values.at(-1)`, and
|
|
109
|
+
note three.js swaps the order to `new THREE.Quaternion(x, y, z, w)`.
|
|
88
110
|
5. **Tap exclusivity** (within `gesture`): use `tap` OR `double_tap` —
|
|
89
111
|
**never both together** unless the user explicitly names both. `tap` is
|
|
90
112
|
the default; `double_tap` is only used when the user explicitly requests
|
|
@@ -236,8 +258,10 @@ Print the absolute path to the written file and a one-line summary:
|
|
|
236
258
|
- Subscribe one signal per command: `{ command: 'subscribe', signal: '<name>' }`
|
|
237
259
|
- Motion modes are mutually exclusive: Pointer (`navigation`+`button`) / Direction (`nav_direction`) / IMU+Biometric (`imu_acc`+`imu_gyro`+`emg`)
|
|
238
260
|
- IMU+Biometric bundle: `imu_acc`, `imu_gyro`, `emg` always subscribed together — never partially
|
|
239
|
-
- `
|
|
240
|
-
-
|
|
261
|
+
- Hand Orientation (`imu_quaternion`) is **outside** the motion-mode XOR — standalone, combines with anything including `navigation` / `nav_direction`. Requires firmware 6.0.12.11 and above. `data.values` is a list of `[w, x, y, z]` samples; read `values.at(-1)`, and build with `new THREE.Quaternion(x, y, z, w)`
|
|
262
|
+
- Pressure: pick exactly one. Both are Finger pressure 0–100, normalized 0–1. `direct_pressure` is the **new** continuous ungated stream (default). Requires firmware 6.0.12.11 and above. `pinch_pressure` is the **original** tap-to-release filtered stream and works on older firmware. There is no bare `pressure` signal.
|
|
263
|
+
- `gesture` and pressure are mutually exclusive — never combine them
|
|
264
|
+
- Free-combining signals (one or the other, not both): `gesture` OR one pressure mode, plus `button`; `imu_quaternion` combines with all of them
|
|
241
265
|
- **Navigation sensitivity is gentle by default**: sim button + keyboard `I`/`J`/`K`/`L` emit `±3` per event; cursor multiplier on inbound `delta_x`/`delta_y` is `0.002`. Raise only when the prompt explicitly asks for fast/snappy movement. See Section 6 + Section 11 of `references/promt.md`.
|
|
242
266
|
- **Reserved for XR Blocks desktop simulator** — Mudra never claims these: `W`/`A`/`S`/`D` and arrow keys (camera walk), `Q`/`E` (roll/vertical), `R` (reset), right-click drag (orbit), mouse wheel (zoom). Mudra navigation uses `I`/`J`/`K`/`L`; Mudra IMU uses `U`/`O`/`M`/`N`. See Section 6 of `references/promt.md`.
|
|
243
267
|
- Keyboard handlers: `{ capture: true }` + `stopPropagation()` on Mudra-claimed keys
|