@iwsdk/create 1.1.0-rc.0 → 1.1.0-rc.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli.js +8 -13
- package/dist/guidance/agents/.agents/skills/iwsdk-compose-scene/SKILL.md +16 -2
- package/dist/guidance/agents/.agents/skills/iwsdk-compose-scene/references/environment-recipes.md +234 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-compose-scene/references/scene-format.md +6 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-dev/references/api-reference.md +56 -33
- package/dist/guidance/agents/.agents/skills/iwsdk-dev/references/iterate.md +24 -5
- package/dist/guidance/agents/.agents/skills/iwsdk-glasses-test/SKILL.md +281 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-grab/SKILL.md +24 -1
- package/dist/guidance/agents/.agents/skills/iwsdk-ray/SKILL.md +3 -1
- package/dist/guidance/agents/.agents/skills/iwsdk-ui/SKILL.md +25 -2
- package/dist/guidance/claude/.claude/agents/iwsdk-project-code-reviewer.md +73 -6
- package/dist/guidance/claude/.claude/rules/assets-and-manifest.md +2 -0
- package/dist/guidance/claude/.claude/rules/ecs-api.md +1 -1
- package/dist/guidance/claude/.claude/rules/scene-json.md +3 -1
- package/dist/guidance/claude/.claude/rules/uikitml.md +5 -0
- package/dist/guidance/claude/.claude/settings.json +1 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-compose-scene/SKILL.md +16 -2
- package/dist/guidance/claude/.claude/skills/iwsdk-compose-scene/references/environment-recipes.md +234 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-compose-scene/references/scene-format.md +6 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-dev/references/api-reference.md +56 -33
- package/dist/guidance/claude/.claude/skills/iwsdk-dev/references/iterate.md +24 -5
- package/dist/guidance/claude/.claude/skills/iwsdk-glasses-test/SKILL.md +281 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-grab/SKILL.md +24 -1
- package/dist/guidance/claude/.claude/skills/iwsdk-ray/SKILL.md +3 -1
- package/dist/guidance/claude/.claude/skills/iwsdk-ui/SKILL.md +25 -2
- package/dist/guidance/common/AGENTS.md +25 -12
- package/dist/guidance/copilot/.github/instructions/assets-and-manifest.instructions.md +2 -0
- package/dist/guidance/copilot/.github/instructions/ecs-api.instructions.md +1 -1
- package/dist/guidance/copilot/.github/instructions/scene-json.instructions.md +3 -1
- package/dist/guidance/copilot/.github/instructions/uikitml.instructions.md +5 -0
- package/dist/guidance/cursor/.cursor/rules/assets-and-manifest.mdc +2 -0
- package/dist/guidance/cursor/.cursor/rules/ecs-api.mdc +1 -1
- package/dist/guidance/cursor/.cursor/rules/scene-json.mdc +3 -1
- package/dist/guidance/cursor/.cursor/rules/uikitml.mdc +5 -0
- package/dist/guidance/scoped-agents/public/scenes/AGENTS.md +3 -1
- package/dist/guidance/scoped-agents/public/ui/AGENTS.md +5 -0
- package/dist/guidance/scoped-agents/src/AGENTS.md +3 -1
- package/dist/template/common/README.md +19 -0
- package/dist/template/common/public/ui/label-grab.uikitml +57 -0
- package/dist/template/common/public/ui/label-select.uikitml +57 -0
- package/dist/template/common/public/ui/label-spin.uikitml +57 -0
- package/dist/template/common/public/ui/welcome.uikitml +33 -32
- package/dist/template/common/src/assets.ts +32 -23
- package/dist/template/common/src/scene-assets/decor.scene-asset.ts +781 -0
- package/dist/template/common/src/scene-assets/studio.scene-asset.ts +298 -0
- package/dist/template/common-js/README.md +19 -0
- package/dist/template/common-js/public/ui/label-grab.uikitml +57 -0
- package/dist/template/common-js/public/ui/label-select.uikitml +57 -0
- package/dist/template/common-js/public/ui/label-spin.uikitml +57 -0
- package/dist/template/common-js/public/ui/welcome.uikitml +33 -32
- package/dist/template/common-js/src/assets.js +20 -23
- package/dist/template/common-js/src/scene-assets/decor.scene-asset.js +554 -0
- package/dist/template/common-js/src/scene-assets/studio.scene-asset.js +236 -0
- package/dist/template/scenes/ar.iwsdk.scene.json +354 -58
- package/dist/template/scenes/immersive.iwsdk.scene.json +435 -54
- package/package.json +4 -4
- package/dist/template/common/public/gltf/webxr-banner/banner.gltf +0 -68
- package/dist/template/common/public/textures/webxr.png +0 -0
- package/dist/template/common-js/public/gltf/webxr-banner/banner.gltf +0 -68
- package/dist/template/common-js/public/textures/webxr.png +0 -0
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iwsdk-glasses-test
|
|
3
|
+
description: Test an IWSDK app for Meta VR Glasses — gaze-and-pinch input, the glasses' narrower field of view, and gaze preview on a Quest without eye tracking. Use when the user targets Meta VR Glasses, asks for gaze or eye-tracking input, or wants to check that content fits the glasses' view.
|
|
4
|
+
argument-hint: '[test scenario]'
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Test for Meta VR Glasses
|
|
8
|
+
|
|
9
|
+
Meta VR Glasses run the same WebXR app as Quest, with two differences that
|
|
10
|
+
break untested apps: a much narrower field of view, and gaze-and-pinch input
|
|
11
|
+
(look at a target, pinch either hand to select it). Prove both with the
|
|
12
|
+
desktop emulator first; use a Quest only for what the emulator cannot show.
|
|
13
|
+
|
|
14
|
+
The requested scenario is in `$ARGUMENTS`.
|
|
15
|
+
|
|
16
|
+
## What differs from Quest
|
|
17
|
+
|
|
18
|
+
| Property | Meta VR Glasses |
|
|
19
|
+
| ------------- | ------------------------------------------------------------------------------------------------ |
|
|
20
|
+
| Field of view | 37° left and right, 25° up, 43° down from the view direction (74° × 68°), the same for both eyes |
|
|
21
|
+
| Primary input | One gaze ray (`targetRayMode: "gaze"`) picks the target; a pinch from either hand selects |
|
|
22
|
+
| Other input | Hands; hand and controller far rays take over whenever gaze is disconnected |
|
|
23
|
+
| Frame rates | 72, 90, 96, 100, 120 Hz |
|
|
24
|
+
| Session modes | `immersive-vr` and `immersive-ar`, with the same features as Quest 3 plus `gaze-tracking` |
|
|
25
|
+
| Gaze cursor | None in production: each target must show its own hover and pressed feedback |
|
|
26
|
+
|
|
27
|
+
Because the view is short above eye level (25°) and tall below it (43°), put
|
|
28
|
+
UI and key targets level with or slightly below the eyes and near the middle.
|
|
29
|
+
Content that Quest shows comfortably at 40° to the side or 30° up is out of
|
|
30
|
+
view on the glasses.
|
|
31
|
+
|
|
32
|
+
## 1. Configure the project
|
|
33
|
+
|
|
34
|
+
In `iwsdk.config.json`, request gaze and hands, and select the glasses
|
|
35
|
+
emulator profile:
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
{
|
|
39
|
+
"world": {
|
|
40
|
+
"xr": {
|
|
41
|
+
"features": {
|
|
42
|
+
"handTracking": true,
|
|
43
|
+
"gazeTracking": true
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
},
|
|
47
|
+
"dev": {
|
|
48
|
+
"emulator": { "device": "metaVRGlasses" }
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
- Keep `gazeTracking` optional (`true`). `{ "required": true }` makes a
|
|
54
|
+
runtime without gaze reject the whole session. The same holds for
|
|
55
|
+
`handTracking`: keep it `true` unless the app cannot work without hands.
|
|
56
|
+
- `gazeTracking` is what registers `GazeSystem`. `world.features.gaze` only
|
|
57
|
+
tunes it; it does not enable gaze on its own.
|
|
58
|
+
- Targets need no gaze-specific code: `RayInteractable`, `DistanceGrabbable`,
|
|
59
|
+
UIKit controls and the `Hovered` and `Pressed` tags work for gaze as they do
|
|
60
|
+
for hand rays.
|
|
61
|
+
- While testing, `world.features.gaze.showDebugReticle: true` draws a white
|
|
62
|
+
hit reticle. Remove it before shipping.
|
|
63
|
+
|
|
64
|
+
Editing `iwsdk.config.json` restarts Vite in place and reloads the page, which
|
|
65
|
+
ends any XR session: enter XR again afterwards. Run
|
|
66
|
+
`npm run --if-present typecheck`, then `npx @iwsdk/cli dev up` (or wait for
|
|
67
|
+
`npx @iwsdk/cli dev status` to report `browserCommandReady: true` if it is
|
|
68
|
+
already running).
|
|
69
|
+
|
|
70
|
+
## 2. Confirm the glasses profile
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
npx @iwsdk/cli xr status
|
|
74
|
+
npx @iwsdk/cli xr enter # only when no session is active
|
|
75
|
+
npx @iwsdk/cli xr status # sessionActive must now be true
|
|
76
|
+
npx @iwsdk/cli xr get-device-state
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Continue only when `xr status` reports `deviceName: "Meta VR Glasses"` and,
|
|
80
|
+
with the session active, `enabledFeatures` contains `"gaze-tracking"`, and
|
|
81
|
+
`xr get-device-state` has a `gaze` entry with `connected: true`. With the
|
|
82
|
+
Quest 3 profile, `deviceName` is `Meta Quest 3` and there is no `gaze` entry:
|
|
83
|
+
fix `dev.emulator.device` and re-check. `xr enter` can return
|
|
84
|
+
`success: true` even when the session request fails (for example, a required
|
|
85
|
+
feature the profile lacks), so always confirm `sessionActive`.
|
|
86
|
+
|
|
87
|
+
Neither command reports the glasses' field of view. The `fov` value in
|
|
88
|
+
`xr get-device-state` (90) is the emulator preview's vertical angle and is the
|
|
89
|
+
same for every profile. The glasses' view shows up as a black mask around the
|
|
90
|
+
runtime screenshot in step 3.
|
|
91
|
+
|
|
92
|
+
## 3. Check the field of view
|
|
93
|
+
|
|
94
|
+
List every element the user must see or reach (panels, buttons, targets,
|
|
95
|
+
instructions). `npx @iwsdk/cli xr list-targets` lists the interactables with
|
|
96
|
+
box centres and sizes; `npx @iwsdk/cli scene runtime-hierarchy` lists every
|
|
97
|
+
named object with its `uuid`, including labels and panels. Read each
|
|
98
|
+
element's position with
|
|
99
|
+
`npx @iwsdk/cli scene transform --input-json '{"uuid":"<uuid>"}'` and use
|
|
100
|
+
`positionRelativeToXROrigin`, which is in the same frame as the headset pose
|
|
101
|
+
in `xr get-device-state`. (`ecs query` returns the `Transform` relative to the
|
|
102
|
+
parent entity, not a world position.)
|
|
103
|
+
|
|
104
|
+
With the headset looking straight ahead (identity orientation, down −Z) and
|
|
105
|
+
the element at offset `(dx, dy, dz)` from the headset (`dz` negative, in
|
|
106
|
+
front):
|
|
107
|
+
|
|
108
|
+
- horizontal angle `atan2(dx, -dz)` must be within ±37°;
|
|
109
|
+
- vertical angle `atan2(dy, -dz)` must be between −43° and +25°.
|
|
110
|
+
|
|
111
|
+
Both angles are measured against the view axis, as the glasses' projection
|
|
112
|
+
and the emulator's mask are. Do not use the elevation
|
|
113
|
+
`atan2(dy, hypot(dx, dz))`: it under-reports the vertical angle of anything
|
|
114
|
+
off to the side. A point 35° to the side and 46° below the view axis is out
|
|
115
|
+
of view, but its elevation is only −40°.
|
|
116
|
+
|
|
117
|
+
Check corners, not just centres, and leave a few degrees of margin. For
|
|
118
|
+
objects, use the box corners from `xr list-targets`. For a UIKitML panel,
|
|
119
|
+
`npx @iwsdk/cli ui inspect --input-json '{"entityIndex":<n>}'` reports
|
|
120
|
+
`panel.computedSize` in centimetres; multiply by the panel's `globalScale`
|
|
121
|
+
from `scene transform` (or use `targetSize`, in metres, when it is not 0×0),
|
|
122
|
+
then place the rectangle at the panel's position with its rotation.
|
|
123
|
+
`xr list-targets` boxes for UIKit panels can be inflated, so do not use them
|
|
124
|
+
for panel corners.
|
|
125
|
+
|
|
126
|
+
Then capture one runtime image as visual evidence:
|
|
127
|
+
|
|
128
|
+
```bash
|
|
129
|
+
npx @iwsdk/cli browser screenshot --output-file /tmp/glasses-fov.png
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Everything outside the glasses' view is masked black in the image. In the
|
|
133
|
+
default managed session, IWER's control panels cover the left and right of
|
|
134
|
+
the image and can hide content near the edges of the view. For an
|
|
135
|
+
unobstructed image, restart with `npx @iwsdk/cli dev restart --ai-mode agent`
|
|
136
|
+
(headless, no control panels). Tell the developer before switching modes.
|
|
137
|
+
|
|
138
|
+
Use the runtime screenshot, not `scene_screenshot`: the editor camera does not
|
|
139
|
+
use the glasses' view. Content clipped in the image or outside the angles
|
|
140
|
+
above must move closer to the view centre (or lower, rather than higher).
|
|
141
|
+
|
|
142
|
+
## 4. Test gaze and pinch
|
|
143
|
+
|
|
144
|
+
Gaze picks the target; hands only pinch. Switch to hands, and record what is
|
|
145
|
+
hovered before aiming so the check shows a change:
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
npx @iwsdk/cli xr set-input-mode --input-json '{"mode":"hand"}'
|
|
149
|
+
npx @iwsdk/cli ecs find --input-json '{"withComponents":["Hovered"]}'
|
|
150
|
+
npx @iwsdk/cli xr look-at --input-json '{"device":"gaze","target":{"x":0,"y":1.2,"z":-2}}'
|
|
151
|
+
npx @iwsdk/cli ecs find --input-json '{"withComponents":["Hovered"]}'
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Aim at a point on the target's visible surface. A box centre from
|
|
155
|
+
`xr list-targets` works for solid objects, but the centre of a ring or frame
|
|
156
|
+
is empty and the gaze ray passes through it. After aiming, `ecs find` must
|
|
157
|
+
list the target entity with `Hovered` (or the app must show its own hover
|
|
158
|
+
state). The default gaze looks straight ahead, so whatever is in front may
|
|
159
|
+
already be hovered: aim at another target first and show `Hovered` move.
|
|
160
|
+
|
|
161
|
+
- **Click or UIKit button:** `xr select` with `{"device":"hand-right"}`,
|
|
162
|
+
keeping the default `duration`. A click needs press and release within
|
|
163
|
+
300 ms: `xr select` with a long `duration` (for example 1 s), or
|
|
164
|
+
`xr set-select-value` 1 then 0, shows `Pressed` (and the button's `active`
|
|
165
|
+
state in `ui inspect`) but never clicks. Query the resulting app state
|
|
166
|
+
afterwards. A select that ends the session, such as an Exit button, returns
|
|
167
|
+
`runtime_command_failed` ("The XR session ended.") even though it worked:
|
|
168
|
+
confirm with `xr status`. For a target that reacts to `Pressed` rather than
|
|
169
|
+
a click, the tag only lasts for the press: run
|
|
170
|
+
`xr select` with `"duration":2` in the background and `ecs find` `Pressed`
|
|
171
|
+
during the hold.
|
|
172
|
+
- **Hold, grab or drag:** `xr set-select-value` with
|
|
173
|
+
`{"device":"hand-right","value":1}`, move the hand with `xr set-transform`
|
|
174
|
+
or `xr animate-to`, query while held (`Pressed`, `Grabbed`, the transform),
|
|
175
|
+
then release with `value: 0` and query again. With the default
|
|
176
|
+
`pointerTransformFollowsHand`, a gaze-started grab follows the pinching
|
|
177
|
+
hand's movement, not the gaze.
|
|
178
|
+
- **Either hand:** repeat one selection with `hand-left`; both hands commit.
|
|
179
|
+
|
|
180
|
+
Gaze reports `pointerType === "ray"` with `source: "gaze"` in its pointer
|
|
181
|
+
state, so the app's ordinary ray handlers receive it.
|
|
182
|
+
|
|
183
|
+
## 5. Test fallback and priority
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
npx @iwsdk/cli xr set-connected --input-json '{"device":"gaze","connected":false}'
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
With gaze disconnected, hand far rays must target and select again (aim a hand
|
|
190
|
+
with `xr look-at` and select). Reconnect gaze with `"connected":true` and
|
|
191
|
+
confirm it owns targeting again: the hand ray's hover is dropped and the gaze
|
|
192
|
+
target is hovered. A near touch or grab with either hand suppresses gaze while
|
|
193
|
+
it lasts; test that when the app mixes near and far interaction.
|
|
194
|
+
|
|
195
|
+
Read the console with `npx @iwsdk/cli browser logs` (no level filter) and keep
|
|
196
|
+
the `[iwsdk][gaze]` lines. `browser logs` works on the managed browser only: a
|
|
197
|
+
physical headset target returns `unsupported_on_target`, so on a Quest read the
|
|
198
|
+
console through Chrome remote debugging
|
|
199
|
+
(`adb -s <serial> forward tcp:9222 localabstract:chrome_devtools_remote`, then
|
|
200
|
+
`chrome://inspect`).
|
|
201
|
+
|
|
202
|
+
| Message | Meaning |
|
|
203
|
+
| ------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
|
204
|
+
| `'gaze-tracking' granted for this session` | The runtime granted the feature |
|
|
205
|
+
| `'gaze-tracking' was NOT granted` | The runtime refused the feature |
|
|
206
|
+
| `gaze XRInputSource appeared` | A gaze source is present (also after a reconnect) |
|
|
207
|
+
| `eye pose is valid — real gaze data is flowing` | Real eye tracking is delivering poses |
|
|
208
|
+
| `emulated gaze pose is valid` | The emulator or a Quest mode supplies synthetic gaze: expected in testing |
|
|
209
|
+
| `the gaze XRInputSource disappeared` | Gaze disconnected; hand and controller far rays target |
|
|
210
|
+
| `no XRInputSource with targetRayMode === "gaze"` | No gaze source appeared within 3 s of session start |
|
|
211
|
+
| `getPose(targetRaySpace) returned null` | The source has no valid pose |
|
|
212
|
+
| `mapping XRTargetRaySpace onto the head pose` | On real glasses, the runtime returned head-directed gaze: a runtime problem, not an app one |
|
|
213
|
+
| `emulated gaze still points where the head does` | Emulated gaze has not been aimed yet: aim it with `xr look-at` |
|
|
214
|
+
| `target-device preview is intentionally providing ...` | The hands-on Quest preview (step 6) is running: expected |
|
|
215
|
+
| `renderer has no XRFrame or reference space yet` | Printed once at session start on Quest; a problem only if it repeats |
|
|
216
|
+
|
|
217
|
+
## 6. On a Quest headset (optional)
|
|
218
|
+
|
|
219
|
+
Quest 3 and 3S have no eye tracking, so a Quest checks real rendering, frame rate
|
|
220
|
+
and native hands, not gaze accuracy. In both modes below the headset renders
|
|
221
|
+
its own, wider field of view (about 45° to 49° to each side, 48° up and 50°
|
|
222
|
+
down on a Quest 3S), and `xr get-device-state` reports the emulator's
|
|
223
|
+
synthetic `fov: 90`. Keep using step 3 for framing.
|
|
224
|
+
|
|
225
|
+
- **Hands-on preview:** add `"targetDevicePreview": { "gazeSimulation": "head" }`
|
|
226
|
+
under `dev` (the app must request `gazeTracking`) and run
|
|
227
|
+
`npx @iwsdk/cli dev up` without `--native-xr-control`. On Quest Browser,
|
|
228
|
+
the dev server keeps the browser's own viewer, hands and controllers and
|
|
229
|
+
adds a gaze source that follows the head; while it is connected, hand and
|
|
230
|
+
controller far rays are off. The user looks at a target and pinches with a
|
|
231
|
+
real hand. It is dev-server only and left out of production builds.
|
|
232
|
+
- Opening the app: the dev server uses HTTPS with a self-signed
|
|
233
|
+
certificate. On the same Wi-Fi, open a `data.runtimeUrls.network` URL and
|
|
234
|
+
accept the certificate warning. Over USB, run
|
|
235
|
+
`adb -s <serial> reverse tcp:<port> tcp:<port>` and open
|
|
236
|
+
`https://localhost:<port>/` (`http://` does not connect) and accept the
|
|
237
|
+
warning, or set `iwsdkDev({ https: false })` in `vite.config.ts` to serve
|
|
238
|
+
HTTP: `http://localhost` through `adb reverse` is a secure context, so
|
|
239
|
+
there is no warning.
|
|
240
|
+
- Enter XR with the app's Enter XR button or the browser's Enter VR
|
|
241
|
+
button; this mode does not enter on its own. The first entry for each
|
|
242
|
+
origin (`http://` and `https://` are separate) shows Quest Browser's
|
|
243
|
+
hand-tracking permission prompt: ask the user to choose Allow.
|
|
244
|
+
- To read state from the CLI, pair the page first: open the URL from
|
|
245
|
+
`runtime pair-headset` and use its physical `runtimeTarget` as in
|
|
246
|
+
`iwsdk-native-xr-test`. `ecs find`, `ecs query`, `xr status` and
|
|
247
|
+
`xr look-at` with `"device":"gaze"` work (look-at sets the gaze direction
|
|
248
|
+
relative to the head). Hand, controller and headset commands such as
|
|
249
|
+
`xr select` and `xr set-transform` report success but do nothing in this
|
|
250
|
+
mode, and `xr get-device-state` lists hands and controllers the session
|
|
251
|
+
does not have: only a real pinch selects.
|
|
252
|
+
- **Scripted run:** follow the `iwsdk-native-xr-test` skill with
|
|
253
|
+
`dev.emulator.device` set to `metaVRGlasses`. `dev up --native-xr-control`
|
|
254
|
+
replaces the headset's inputs with the glasses profile (synthetic
|
|
255
|
+
controllers or hands, plus a gaze source), so steps 4 and 5 run on the
|
|
256
|
+
device with each command's `runtimeTarget`; only `browser logs` is
|
|
257
|
+
unavailable there (see step 5). `targetDevicePreview` is ignored in that
|
|
258
|
+
mode.
|
|
259
|
+
|
|
260
|
+
## Rules
|
|
261
|
+
|
|
262
|
+
- Do not treat head-directed gaze (the preview, or the emulator's **Head**
|
|
263
|
+
gaze mode) as proof of eye-tracking accuracy. It proves the interaction
|
|
264
|
+
plumbing only.
|
|
265
|
+
- Do not add a head-pose fallback for missing gaze in production code: when
|
|
266
|
+
gaze is unavailable, IWSDK hands targeting back to hand and controller rays.
|
|
267
|
+
- Do not move gaze with `set-transform` position: gaze takes orientation only
|
|
268
|
+
and its origin follows the headset.
|
|
269
|
+
- Do not claim a fit from a Quest screenshot or the editor view; only the
|
|
270
|
+
glasses profile in the desktop emulator has the narrower field of view.
|
|
271
|
+
|
|
272
|
+
## Report
|
|
273
|
+
|
|
274
|
+
- emulator device and enabled XR features (gaze granted or not);
|
|
275
|
+
- each checked element with its horizontal and vertical angles, and any that
|
|
276
|
+
fall outside the glasses' view;
|
|
277
|
+
- each gaze target: hover evidence, the pinch used, and the state change;
|
|
278
|
+
- fallback result with gaze disconnected;
|
|
279
|
+
- `[iwsdk][gaze]` console lines and other errors;
|
|
280
|
+
- for a headset run, its serial, which Quest mode was used, and whether the
|
|
281
|
+
page was paired.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: iwsdk-grab
|
|
3
|
-
description: Use emulated XR controllers to verify proximity grabbing for OneHandGrabbable and TwoHandsGrabbable objects. Use for pick up, move, release, or regression-test requests involving near-hand grab with the squeeze button.
|
|
3
|
+
description: Use emulated XR controllers or hands to verify proximity grabbing for OneHandGrabbable and TwoHandsGrabbable objects. Use for pick up, move, release, or regression-test requests involving near-hand grab with the squeeze button or a hand pinch.
|
|
4
4
|
argument-hint: '(use the current request)'
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -36,6 +36,29 @@ Do not use trigger index `0`, `xr select`, or teleport-style device mutation for
|
|
|
36
36
|
proximity grab. Do not edit source merely to make an existing named entity easier
|
|
37
37
|
to locate; report missing runtime identity when that is the actual defect.
|
|
38
38
|
|
|
39
|
+
## Hand pinch grab
|
|
40
|
+
|
|
41
|
+
When the user asks for hands or a pinch, grab with the emulated hand instead of
|
|
42
|
+
squeeze. This needs `grabbing.useHandPinchForGrab`, which is on unless the project
|
|
43
|
+
manifest sets it to `false`.
|
|
44
|
+
Switch input once, then grab and carry in one sequence, by the entity's name
|
|
45
|
+
from `xr list-targets`:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
npx @iwsdk/cli xr set-input-mode --input-json '{"mode":"hand"}'
|
|
49
|
+
npx @iwsdk/cli xr run-sequence --input-json '{"device":"hand-right","steps":[{"target":"Requested Name","approach":{"distance":0.06},"pinch":{"value":1,"at":"arrival"}},{"position":{"x":0.3,"y":1,"z":-1.2},"depart":{"distance":0.08}}]}' --timeout 45000
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
While the hand holds something, a step `position` is where the object's base
|
|
53
|
+
goes. Query `Grabbed` while held, then let go and query the final transform:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
npx @iwsdk/cli xr run-sequence --input-json '{"device":"hand-right","steps":[{"pinch":0,"duration":0.15}]}'
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The sequence result lists the select events the application saw; a missing
|
|
60
|
+
`selectstart` means the pinch never reached the object.
|
|
61
|
+
|
|
39
62
|
## Evidence and stopping
|
|
40
63
|
|
|
41
64
|
Prefer measured ECS state over screenshots. Capture one runtime image only when
|
|
@@ -17,7 +17,9 @@ distance grabs share aiming but not input duration or movement behavior.
|
|
|
17
17
|
`requireElementById`; use a project test hook only when no addressable runtime
|
|
18
18
|
target exists.
|
|
19
19
|
3. Aim the requested controller with `xr look-at`. Do not translate the
|
|
20
|
-
controller toward a ray target.
|
|
20
|
+
controller toward a ray target. The transform of an open shape (a ring, a
|
|
21
|
+
knot) can sit in empty space: aim at its surface, such as the rim at
|
|
22
|
+
transform + (0, radius, 0), and confirm `Hovered` before pressing.
|
|
21
23
|
|
|
22
24
|
## Select exactly one branch
|
|
23
25
|
|
|
@@ -54,8 +54,17 @@ JSON representation.
|
|
|
54
54
|
- Prefer supported explicit properties over CSS shorthand.
|
|
55
55
|
- Reuse the configured Horizon kit and bundled Lucide icons instead of
|
|
56
56
|
registering replacements in application code.
|
|
57
|
-
-
|
|
58
|
-
|
|
57
|
+
- Production builds keep only the Lucide icons named in static
|
|
58
|
+
`public/**/*.uikitml` files. UIKitML built or fetched at runtime (a string in
|
|
59
|
+
`src/`, a file outside `public/`, a remote URL) works in `npm run dev` but
|
|
60
|
+
fails after `npm run build` with `Unknown component <Settings>` and a hint.
|
|
61
|
+
List those icons by component name in `vite.config.ts`:
|
|
62
|
+
`iwsdkDev({ bundle: { icons: ['Settings'] } })` (`'all'` keeps all ~1,600,
|
|
63
|
+
about 4 MB more). Check dynamic UIKitML in a production build.
|
|
64
|
+
- In the stock Horizon kit, `<Panel>` renders an opaque surface that follows
|
|
65
|
+
the color scheme: dark at runtime when `spatialUI.preferredColorScheme` is
|
|
66
|
+
`dark` (the scaffold default), white in editor previews. Give text a dark
|
|
67
|
+
color plus a light `:dark` variant, as `public/ui/welcome.uikitml` does.
|
|
59
68
|
- Remote fonts can reflow layout. Use a local font when offline availability is
|
|
60
69
|
a product requirement.
|
|
61
70
|
- Use plain ASCII punctuation in runtime status copy. Do not use em dashes,
|
|
@@ -245,6 +254,20 @@ query the entity while held to observe `Grabbed`, then release with the same
|
|
|
245
254
|
command and `"value":0`. Movement is unnecessary unless the request asks for
|
|
246
255
|
it.
|
|
247
256
|
|
|
257
|
+
When the request asks for a poke (near touch) on a control, press it with the
|
|
258
|
+
emulated hand's index fingertip by its element ID. The panel entity needs
|
|
259
|
+
`PokeInteractable`:
|
|
260
|
+
|
|
261
|
+
```bash
|
|
262
|
+
npx @iwsdk/cli xr set-input-mode --input-json '{"mode":"hand"}'
|
|
263
|
+
npx @iwsdk/cli xr run-sequence --input-json '{"device":"hand-right","steps":[{"target":"Btn_Action","press":{}}]}' --timeout 45000
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
The press clicked when `presses[0].contactMs` is below 800; at 800 or more a
|
|
267
|
+
slow headless frame held the touch too long, so run the press once more. Confirm
|
|
268
|
+
the state change with `ui inspect`. `xr list-targets` lists the element IDs that
|
|
269
|
+
can be pressed.
|
|
270
|
+
|
|
248
271
|
The commands in this section are the supported path. Do not call broad
|
|
249
272
|
`iwsdk`, `ui`, `browser`, `ecs`, or `xr` help. If a shown command fails, use its
|
|
250
273
|
error output and the exact forms above to correct it instead of invoking
|
|
@@ -377,17 +377,20 @@ const root = world.activeLevel.value;
|
|
|
377
377
|
root.addComponent(DomeGradient, { ... });
|
|
378
378
|
```
|
|
379
379
|
|
|
380
|
-
### 19. Missing `_needsUpdate` on environment changes
|
|
380
|
+
### 19. Missing `_needsUpdate` on environment texture changes
|
|
381
381
|
|
|
382
|
-
Flag: Setting properties on `
|
|
382
|
+
Flag: Setting properties on `DomeTexture`/`IBLTexture` without setting `_needsUpdate: true`. `DomeGradient` and `IBLGradient` apply edits on the next frame, so do not flag them for a missing `_needsUpdate`.
|
|
383
383
|
|
|
384
384
|
```typescript
|
|
385
|
-
// ❌ BAD -
|
|
386
|
-
root.setValue(
|
|
385
|
+
// ❌ BAD - DomeTexture changes are silently ignored
|
|
386
|
+
root.setValue(DomeTexture, 'intensity', 0.6);
|
|
387
387
|
|
|
388
|
-
// ✅ GOOD -
|
|
388
|
+
// ✅ GOOD - _needsUpdate is scalar
|
|
389
|
+
root.setValue(DomeTexture, 'intensity', 0.6);
|
|
390
|
+
root.setValue(DomeTexture, '_needsUpdate', true);
|
|
391
|
+
|
|
392
|
+
// ✅ ALSO GOOD - gradient edits apply without the flag
|
|
389
393
|
root.getVectorView(DomeGradient, 'sky').set([0.1, 0.2, 0.8, 1.0]);
|
|
390
|
-
root.setValue(DomeGradient, '_needsUpdate', true);
|
|
391
394
|
```
|
|
392
395
|
|
|
393
396
|
### 20. Using `destroy()` instead of `dispose()` for GPU objects
|
|
@@ -402,6 +405,70 @@ entity.destroy();
|
|
|
402
405
|
entity.dispose();
|
|
403
406
|
```
|
|
404
407
|
|
|
408
|
+
### 21. Hand-built sky or sky fade
|
|
409
|
+
|
|
410
|
+
Flag: a `ShaderMaterial` sky sphere, `BackSide` sky meshes, or an `onBeforeCompile` that pastes sky gradient GLSL into a floor material. Point to the built-ins: `DomeGradient` on the level root (with `elevationDeg`/`azimuthDeg` to tilt it and `dither` against banding) plus `DomeFade` on the floor node.
|
|
411
|
+
|
|
412
|
+
```typescript
|
|
413
|
+
// ❌ BAD - a second sky that the environment system, editor and AR policy know nothing about
|
|
414
|
+
const sky = new Mesh(
|
|
415
|
+
new SphereGeometry(60),
|
|
416
|
+
new ShaderMaterial({ side: BackSide, fragmentShader: SKY_GLSL }),
|
|
417
|
+
);
|
|
418
|
+
floorMaterial.onBeforeCompile = (shader) => {
|
|
419
|
+
/* paste SKY_GLSL, mix by distance */
|
|
420
|
+
};
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
```jsonc
|
|
424
|
+
// ✅ GOOD - scene JSON: the root sky plus a fade on the floor node
|
|
425
|
+
{
|
|
426
|
+
"components": {
|
|
427
|
+
"com.iwsdk.components.DomeGradient": {
|
|
428
|
+
"elevationDeg": 24,
|
|
429
|
+
"azimuthDeg": -31,
|
|
430
|
+
"dither": true,
|
|
431
|
+
},
|
|
432
|
+
},
|
|
433
|
+
"nodes": [
|
|
434
|
+
{
|
|
435
|
+
"id": "floor",
|
|
436
|
+
"components": {
|
|
437
|
+
"com.iwsdk.components.DomeFade": { "start": 4, "end": 10 },
|
|
438
|
+
},
|
|
439
|
+
},
|
|
440
|
+
],
|
|
441
|
+
}
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
### 22. Hand-built ground shadows
|
|
445
|
+
|
|
446
|
+
Flag: a canvas-painted radial blob card under a prop, or a `ShadowMaterial` plane
|
|
447
|
+
added to ground content in AR. Both are built in: author `ContactShadow` and
|
|
448
|
+
`ShadowCatcher` node components in the scene instead.
|
|
449
|
+
|
|
450
|
+
```typescript
|
|
451
|
+
// ❌ BAD - blob card and AR catcher built by hand (DOM canvas, extra assets)
|
|
452
|
+
const blob = new Mesh(
|
|
453
|
+
plane,
|
|
454
|
+
new MeshBasicMaterial({ map: canvasBlob, transparent: true }),
|
|
455
|
+
);
|
|
456
|
+
const catcher = new Mesh(plane, new ShadowMaterial({ opacity: 0.25 }));
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
✅ GOOD - a node component in the scene JSON; for AR, add one floor node with
|
|
460
|
+
`com.iwsdk.components.ShadowCatcher`:
|
|
461
|
+
|
|
462
|
+
```json
|
|
463
|
+
{
|
|
464
|
+
"id": "plinth",
|
|
465
|
+
"content": { "type": "asset", "asset": "plinth" },
|
|
466
|
+
"components": {
|
|
467
|
+
"com.iwsdk.components.ContactShadow": { "size": [0.9, 0.9] }
|
|
468
|
+
}
|
|
469
|
+
}
|
|
470
|
+
```
|
|
471
|
+
|
|
405
472
|
---
|
|
406
473
|
|
|
407
474
|
## Confidence-Based Reporting
|
|
@@ -53,3 +53,5 @@ pattern for repetition; a pattern whose prefab resolves to a single component-fr
|
|
|
53
53
|
mesh lowers to one `InstancedMesh`.
|
|
54
54
|
|
|
55
55
|
Start assets at `castShadow: false` and opt in only for grounded focal masses.
|
|
56
|
+
Don't build contact-shadow blob cards or `ShadowMaterial` catcher planes here: put
|
|
57
|
+
`ContactShadow` or `ShadowCatcher` node components on the scene nodes instead.
|
|
@@ -71,7 +71,7 @@ and XR rays. Grab components: `OneHandGrabbable` and `TwoHandsGrabbable` are
|
|
|
71
71
|
proximity-based and respond to **squeeze**, while `DistanceGrabbable` is
|
|
72
72
|
ray-based and responds to **trigger**. `Grabbed` is a transient tag managed by
|
|
73
73
|
`GrabSystem` — read it, never add or remove it. `GrabSystem.useHandPinchForGrab`
|
|
74
|
-
defaults to **
|
|
74
|
+
defaults to **true**, so a hand pinch grabs; set it to `false` to require squeeze.
|
|
75
75
|
|
|
76
76
|
`DistanceGrabbable` fields: `movementMode` (`MoveTowardsTarget`, `MoveFromTarget`,
|
|
77
77
|
`MoveAtSource`, `RotateAtSource`), `returnToOrigin`, `moveSpeedFactor`,
|
|
@@ -39,6 +39,8 @@ player eye height to prove it.
|
|
|
39
39
|
did not appear — even when the image looks plausible and `valid` is `true`. Check
|
|
40
40
|
it against what you expect rather than trusting the picture. Single-sided
|
|
41
41
|
surfaces facing away from the camera are the usual cause, and they report nothing.
|
|
42
|
+
`ContactShadow` and `ShadowCatcher` meshes take no rays, so a node whose only
|
|
43
|
+
content is one of them is never listed; check the object it grounds instead.
|
|
42
44
|
|
|
43
45
|
## Component keys come in two forms
|
|
44
46
|
|
|
@@ -65,5 +67,5 @@ view near `[0, 1.6, 0]`) before accepting XR framing.
|
|
|
65
67
|
open document. A result is usable only when `valid: true` **and** a PNG comes
|
|
66
68
|
back. Render each changed module before the root.
|
|
67
69
|
|
|
68
|
-
For composing a scene rather than editing one, use the `iwsdk-scene
|
|
70
|
+
For composing a scene rather than editing one, use the `iwsdk-compose-scene`
|
|
69
71
|
skill — it owns the intake, review and stop-rule procedure this file does not.
|
|
@@ -26,6 +26,11 @@ Expand shorthands (`padding: 12 18` → four properties), use unitless
|
|
|
26
26
|
line-height multipliers, and prefer uniform `border-color`/`border-width` over
|
|
27
27
|
directional variants. Parser errors include source locations.
|
|
28
28
|
|
|
29
|
+
**Production builds drop unused Lucide icons.** They keep only icons named in
|
|
30
|
+
`public/**/*.uikitml`. Icons named only by UIKitML built at runtime or loaded
|
|
31
|
+
from elsewhere must be listed in `iwsdkDev({ bundle: { icons: [...] } })`, or
|
|
32
|
+
they fail after `npm run build`.
|
|
33
|
+
|
|
29
34
|
## Runtime API
|
|
30
35
|
|
|
31
36
|
Resolve the placed panel by **scene node id**, then elements by their `id`
|
|
@@ -242,12 +242,26 @@ Directional lights emit along local `-Z`. Start from the shown transform or anot
|
|
|
242
242
|
simple intentional angle and refine it from the render; do not write a look-at solver.
|
|
243
243
|
The common light payloads are `AmbientLight: {color, intensity}`,
|
|
244
244
|
`HemisphereLight: {skyColor, groundColor, intensity}` (there is no `color` field),
|
|
245
|
-
and `PointLight: {color, intensity, distance, decay, castShadow}`. A
|
|
245
|
+
and `PointLight: {color, intensity, distance, decay, castShadow}`. A floor that should
|
|
246
|
+
have no visible edge gets `"com.iwsdk.components.DomeFade": {"start": 4, "end": 10}`
|
|
247
|
+
on the floor node itself: it fades that node's meshes into the root `DomeGradient`
|
|
248
|
+
between those horizontal distances from the node origin, and does nothing without a
|
|
249
|
+
`DomeGradient`. Do not paste sky GLSL into an `onBeforeCompile`. A clean state after
|
|
246
250
|
the correction is the finish gate; do not query capabilities again merely to confirm
|
|
247
251
|
these established fields.
|
|
248
252
|
|
|
253
|
+
Ground props with node components, not blob or catcher assets. Add
|
|
254
|
+
`"com.iwsdk.components.ContactShadow": {"size": [0.9, 0.9]}` to each node that stands
|
|
255
|
+
on the floor (origin at floor contact, size a little wider than the footprint). For
|
|
256
|
+
AR, add one component-only node with `ShadowCatcher: {size, opacity}` at floor height;
|
|
257
|
+
it shows the shadows of `castShadow` content only when `environment.shadows` is true
|
|
258
|
+
and a light has `castShadow`. `ContactShadow` and `ShadowCatcher` are established
|
|
259
|
+
schema too. For a complete look to copy (the starter's tilted clay-studio sky
|
|
260
|
+
with a fading floor, a default sky, or AR passthrough grounding), use
|
|
261
|
+
[environment-recipes.md](references/environment-recipes.md).
|
|
262
|
+
|
|
249
263
|
For this common path, do not read the references or call
|
|
250
|
-
`scene_get_capabilities`: `DomeGradient`, `IBLGradient`, `AmbientLight`,
|
|
264
|
+
`scene_get_capabilities`: `DomeGradient`, `IBLGradient`, `DomeFade`, `AmbientLight`,
|
|
251
265
|
`HemisphereLight`, `DirectionalLight`, `PointLight`, asset nodes,
|
|
252
266
|
`player.transform`, and perspective/orthographic `authoring.views` are established
|
|
253
267
|
schema. Do not write a separate projection or light-aim solver. Choose the initial
|