@iwsdk/create 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +103 -41
- package/dist/cli.js +1003 -3418
- package/dist/guidance/agents/.agents/skills/iwsdk-debug/SKILL.md +96 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-depth-occlusion/SKILL.md +135 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-grab/SKILL.md +129 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/SKILL.md +92 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/references/component-reference.md +154 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/references/tuning-and-config.md +88 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/references/workflows.md +303 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/SKILL.md +364 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/api-reference.md +1489 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/build-milestones.md +241 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/design-deck.md +101 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/grounding.md +176 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/ideation.md +156 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/verification.md +199 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-ray/SKILL.md +159 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/SKILL.md +255 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/agents/openai.yaml +5 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/asset-authoring.md +295 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/composition-patterns.md +243 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/image-intake.md +191 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/review-and-stop.md +99 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/scene-format.md +458 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/text-intake.md +82 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-ui/SKILL.md +208 -0
- package/dist/guidance/claude/.claude/agents/iwsdk-project-code-reviewer.md +450 -0
- package/dist/guidance/claude/.claude/rules/assets-and-manifest.md +55 -0
- package/dist/guidance/claude/.claude/rules/ecs-api.md +102 -0
- package/dist/guidance/claude/.claude/rules/scene-json.md +64 -0
- package/dist/guidance/claude/.claude/rules/uikitml.md +60 -0
- package/dist/guidance/claude/.claude/settings.json +19 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-debug/SKILL.md +96 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-depth-occlusion/SKILL.md +135 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-grab/SKILL.md +129 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/SKILL.md +92 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/references/component-reference.md +154 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/references/tuning-and-config.md +88 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/references/workflows.md +303 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/SKILL.md +364 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/api-reference.md +1489 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/build-milestones.md +241 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/design-deck.md +101 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/grounding.md +176 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/ideation.md +156 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/verification.md +199 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-ray/SKILL.md +159 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/SKILL.md +255 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/agents/openai.yaml +5 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/asset-authoring.md +295 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/composition-patterns.md +243 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/image-intake.md +191 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/review-and-stop.md +99 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/scene-format.md +458 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/text-intake.md +82 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-ui/SKILL.md +208 -0
- package/dist/guidance/claude/CLAUDE.md +142 -0
- package/dist/guidance/codex/.codex/config.toml +2 -0
- package/dist/guidance/common/AGENTS.md +151 -0
- package/dist/guidance/copilot/.github/instructions/assets-and-manifest.instructions.md +52 -0
- package/dist/guidance/copilot/.github/instructions/ecs-api.instructions.md +100 -0
- package/dist/guidance/copilot/.github/instructions/scene-json.instructions.md +63 -0
- package/dist/guidance/copilot/.github/instructions/uikitml.instructions.md +59 -0
- package/dist/guidance/cursor/.cursor/rules/assets-and-manifest.mdc +57 -0
- package/dist/guidance/cursor/.cursor/rules/ecs-api.mdc +104 -0
- package/dist/guidance/cursor/.cursor/rules/scene-json.mdc +66 -0
- package/dist/guidance/cursor/.cursor/rules/uikitml.mdc +62 -0
- package/dist/guidance/scoped-agents/public/scenes/AGENTS.md +63 -0
- package/dist/guidance/scoped-agents/public/ui/AGENTS.md +59 -0
- package/dist/guidance/scoped-agents/src/AGENTS.md +149 -0
- package/dist/template/common/README.md +13 -0
- package/dist/template/common/gitignore.template +45 -0
- package/dist/template/common/index.html +16 -0
- package/dist/template/common/nvmrc.template +1 -0
- package/dist/template/common/public/audio/chime.mp3 +0 -0
- package/dist/template/common/public/gltf/webxr-banner/banner.gltf +68 -0
- package/dist/template/common/public/textures/webxr.png +0 -0
- package/dist/template/common/public/ui/welcome.uikitml +98 -0
- package/dist/template/common/src/assets.ts +54 -0
- package/dist/template/common/src/components.ts +11 -0
- package/dist/template/common/src/index.ts +19 -0
- package/dist/template/common/src/panel.ts +38 -0
- package/dist/template/common/src/robot-component.ts +10 -0
- package/dist/template/common/src/robot.ts +38 -0
- package/dist/template/common/src/vite-env.d.ts +2 -0
- package/dist/template/common/tsconfig.json +15 -0
- package/dist/template/common/vite.config.ts +27 -0
- package/dist/template/common-js/README.md +13 -0
- package/dist/template/common-js/gitignore.template +45 -0
- package/dist/template/common-js/index.html +16 -0
- package/dist/template/common-js/nvmrc.template +1 -0
- package/dist/template/common-js/public/audio/chime.mp3 +0 -0
- package/dist/template/common-js/public/gltf/webxr-banner/banner.gltf +68 -0
- package/dist/template/common-js/public/textures/webxr.png +0 -0
- package/dist/template/common-js/public/ui/welcome.uikitml +98 -0
- package/dist/template/common-js/src/assets.js +45 -0
- package/dist/template/common-js/src/components.js +9 -0
- package/dist/template/common-js/src/index.js +14 -0
- package/dist/template/common-js/src/panel.js +31 -0
- package/dist/template/common-js/src/robot-component.js +8 -0
- package/dist/template/common-js/src/robot.js +37 -0
- package/dist/template/common-js/vite.config.js +25 -0
- package/dist/template/scenes/ar.iwsdk.scene.json +96 -0
- package/dist/template/scenes/immersive.iwsdk.scene.json +109 -0
- package/package.json +23 -17
- package/scripts/verify-release-ready.mjs +76 -0
- package/LICENSE +0 -21
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# Phases 5–7 Playbook — Runtime Verification
|
|
2
|
+
|
|
3
|
+
Everything here uses the `iwsdk` CLI (works in every harness). MCP
|
|
4
|
+
equivalents exist for every command but several names differ: `xr enter` =
|
|
5
|
+
`xr_accept_session`, `xr status` = `xr_get_session_status`, `ecs find` =
|
|
6
|
+
`ecs_find_entities`, `ecs query` = `ecs_query_entity`, `browser logs` =
|
|
7
|
+
`browser_get_console_logs`, `browser reload` = `browser_reload_page`,
|
|
8
|
+
`scene state` = `scene_get_state`, and `scene render-file` =
|
|
9
|
+
`scene_render_file`; the rest map 1:1
|
|
10
|
+
(`xr_select` → `xr select`, `ecs_pause` → `ecs pause`, …). Use MCP tools only
|
|
11
|
+
if your harness clearly has them connected. The sibling `iwsdk-grab` /
|
|
12
|
+
`iwsdk-ray` / `iwsdk-debug` / `iwsdk-ui` skills cover the same runtime
|
|
13
|
+
operations as focused per-interaction recipes — consistent with this file.
|
|
14
|
+
|
|
15
|
+
**Ground rules.** Run every `npx iwsdk …` from **inside the app directory**
|
|
16
|
+
(the app must depend on `@iwsdk/cli`; a bare `npx iwsdk` elsewhere resolves
|
|
17
|
+
to an unrelated npm package). One CLI call per shell command — no `&&`
|
|
18
|
+
chains between CLI calls; use separate `sleep N` calls for waits. Every
|
|
19
|
+
command prints a JSON envelope `{ok, data|error}` on stdout — parse it and
|
|
20
|
+
check the assertion **before** the next command. Add `--timeout 20000` to
|
|
21
|
+
slow ops (reload, xr enter, animate-to, screenshot).
|
|
22
|
+
|
|
23
|
+
## The Standard Loop
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
npx tsc --noEmit # 0. types clean first — always
|
|
27
|
+
npx iwsdk dev status # 1. already running? if state.running AND
|
|
28
|
+
# state.browserCommandReady are true,
|
|
29
|
+
# REUSE it — never start a second server
|
|
30
|
+
npx iwsdk dev up --timeout 60000 # 2. only if not running: detached daemon;
|
|
31
|
+
# returns once the browser is command-ready.
|
|
32
|
+
# (`npm run dev` = `iwsdk dev up --open
|
|
33
|
+
# --foreground` — never returns; use it only
|
|
34
|
+
# where your harness manages background
|
|
35
|
+
# processes. NEVER a bare '&'.)
|
|
36
|
+
npx iwsdk ecs systems # 3. connectivity: returns system list
|
|
37
|
+
npx iwsdk browser reload --timeout 20000 # 4. fresh page state
|
|
38
|
+
sleep 3
|
|
39
|
+
npx iwsdk browser screenshot --output-file design/verify/base.png --timeout 20000
|
|
40
|
+
# 5. runtime renders? (not black/empty).
|
|
41
|
+
# Browser screenshots are runtime-only and
|
|
42
|
+
# auto-switch away from editor view. Without
|
|
43
|
+
# --output-file the PNG lands in the system
|
|
44
|
+
# temp dir (path in the JSON envelope)
|
|
45
|
+
npx iwsdk xr enter --timeout 20000 # 6. enter emulated XR session
|
|
46
|
+
sleep 2
|
|
47
|
+
npx iwsdk browser logs --input-json '{"count":30}'
|
|
48
|
+
# 7. console clean — scan ALL levels (a
|
|
49
|
+
# level filter can miss important errors);
|
|
50
|
+
# fail the gate on any error-looking line
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
If `dev up` times out, poll yourself: run `npx iwsdk dev status`, `sleep 5`,
|
|
54
|
+
repeat up to ~60 s (a shell `until` loop around the _single_ CLI call is fine
|
|
55
|
+
— the no-chaining rule is about chaining different CLI calls), then inspect
|
|
56
|
+
`npx iwsdk dev logs --tail 100`.
|
|
57
|
+
|
|
58
|
+
Then per-scenario: discover → simulate → assert. Finish with
|
|
59
|
+
`npx iwsdk dev down` when done for the session.
|
|
60
|
+
|
|
61
|
+
The dev server owns one managed headed Playwright Chromium window, with IWER enabled
|
|
62
|
+
for XR starters. It starts with the dev server; do not launch a second browser or turn
|
|
63
|
+
Vite's independent `server.open` back on. Use `scene screenshot` / `scene render-file`
|
|
64
|
+
for editor evidence and `browser screenshot` for the application runtime.
|
|
65
|
+
|
|
66
|
+
## Discover (after every reload — entity indices are NOT stable)
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
npx iwsdk ecs find --input-json '{"withComponents":["OneHandGrabbable"]}'
|
|
70
|
+
npx iwsdk ecs query --input-json '{"entityIndex":3}' # full entity dump
|
|
71
|
+
npx iwsdk scene state --raw # active scene/runtime state
|
|
72
|
+
npx iwsdk ecs components # component registry
|
|
73
|
+
npx iwsdk ecs systems # system registry
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Simulate input (XR emulation cheat sheet)
|
|
77
|
+
|
|
78
|
+
Devices: `headset`, `controller-left`, `controller-right`, `hand-left`,
|
|
79
|
+
`hand-right`.
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
# Aim/move devices
|
|
83
|
+
npx iwsdk xr set-transform --input-json '{"device":"headset","position":{"x":0,"y":1.6,"z":0}}'
|
|
84
|
+
npx iwsdk xr look-at --input-json '{"device":"controller-right","target":{"x":0,"y":1,"z":-2}}'
|
|
85
|
+
npx iwsdk xr animate-to --input-json '{"device":"controller-right","position":{"x":0.2,"y":1.1,"z":-0.4},"duration":0.5}' --timeout 20000
|
|
86
|
+
|
|
87
|
+
# Buttons — WRONG BUTTON FAILS SILENTLY, know which mechanic uses which:
|
|
88
|
+
npx iwsdk xr set-select-value --input-json '{"device":"controller-right","value":1}' # press TRIGGER — UI pointerdown or distance-grab engage
|
|
89
|
+
npx iwsdk xr set-select-value --input-json '{"device":"controller-right","value":0}' # release TRIGGER — UI pointerup or distance-grab release
|
|
90
|
+
npx iwsdk xr set-gamepad-state --input-json '{"device":"controller-right","buttons":[{"index":1,"value":1}]}' # SQUEEZE — near grab (one/two-hand)
|
|
91
|
+
npx iwsdk xr set-gamepad-state --input-json '{"device":"controller-right","buttons":[{"index":3,"value":1}]}' # A/X button (jump default)
|
|
92
|
+
|
|
93
|
+
# Thumbsticks (axes index 0=X, 1=Y):
|
|
94
|
+
npx iwsdk xr set-gamepad-state --input-json '{"device":"controller-left","axes":[{"index":1,"value":-1}]}' # slide forward (0 to stop)
|
|
95
|
+
npx iwsdk xr set-gamepad-state --input-json '{"device":"controller-right","axes":[{"index":0,"value":1}]}' # snap turn — EDGE-TRIGGERED: return to 0 before turning again
|
|
96
|
+
npx iwsdk xr set-gamepad-state --input-json '{"device":"controller-right","axes":[{"index":1,"value":1}]}' # hold then release to 0 = teleport
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Synthetic gamepad indices for `set-gamepad-state`: 0=trigger, 1=squeeze,
|
|
100
|
+
2=thumbstick-press, 3=A/X, 4=B/Y, 5=thumbrest. (App code uses
|
|
101
|
+
`InputComponent` enums; these raw indices are only for the emulator CLI.)
|
|
102
|
+
|
|
103
|
+
## Assert
|
|
104
|
+
|
|
105
|
+
Pick the right instrument per behavior:
|
|
106
|
+
|
|
107
|
+
| Behavior | Assertion |
|
|
108
|
+
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
|
|
109
|
+
| hover/click state | `ecs query` for `Hovered`/`Pressed` on the entity |
|
|
110
|
+
| object moved/scaled/spawned | `ecs snapshot` (label before) → act → `ecs snapshot` (after) → `ecs diff` |
|
|
111
|
+
| physics settled/fell/collided | snapshot/diff on Transform position, or pause+step deterministically |
|
|
112
|
+
| **locomotion** (player moved) | **screenshots before/after** — the XR origin moves, headset transform stays constant; never assert on headset transform |
|
|
113
|
+
| **near-field grab** | snapshot/diff — `Hovered`/`Pressed` are NOT applied on near grabs |
|
|
114
|
+
| UI reacted | `browser logs` with a pattern: instrument handlers with distinctive `console.log('[GAME] score=…')` and assert `{"pattern":"\\[GAME\\]"}` |
|
|
115
|
+
| game logic over time | `ecs pause` → `ecs step --input-json '{"count":N}'` → query → `ecs resume` |
|
|
116
|
+
| looks right | `browser screenshot` and actually look at the image |
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
npx iwsdk ecs snapshot --input-json '{"label":"before"}'
|
|
120
|
+
# …act…
|
|
121
|
+
npx iwsdk ecs snapshot --input-json '{"label":"after"}'
|
|
122
|
+
npx iwsdk ecs diff --input-json '{"from":"before","to":"after"}'
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Only the two most recent snapshot labels are retained.
|
|
126
|
+
|
|
127
|
+
## Known traps
|
|
128
|
+
|
|
129
|
+
- Entity indices change on every reload — re-run `ecs find`; never reuse.
|
|
130
|
+
- JSON values must be real types: `true` not `"true"`, numbers not strings
|
|
131
|
+
(`ecs set-component` silently no-ops on wrong types).
|
|
132
|
+
- Trigger vs squeeze mix-ups produce _no error and no effect_ — check the
|
|
133
|
+
mechanic's grab type first (`DistanceGrabbable` → trigger/select-value;
|
|
134
|
+
`OneHandGrabbable`/`TwoHandsGrabbable` → squeeze button 1).
|
|
135
|
+
- Teleport is blocked while the ray hovers an `Interactable`.
|
|
136
|
+
- Snap turn re-fires only after the stick returns to 0.
|
|
137
|
+
- `--input-json` takes one single-quoted JSON argument; malformed JSON errors
|
|
138
|
+
out — build it with a heredoc if it contains quotes.
|
|
139
|
+
- UIKitML files live under `public/ui`, are registered in the asset manifest with
|
|
140
|
+
`AssetType.UIKitML`, and are parsed directly at runtime. Scene JSON references the
|
|
141
|
+
manifest asset ID; application code uses the placed scene node ID and element IDs.
|
|
142
|
+
- After deploying: verify the live HTML references the newly built hashed
|
|
143
|
+
assets (stale-dist deploys are the classic ship failure).
|
|
144
|
+
|
|
145
|
+
## Managed-runtime hardening
|
|
146
|
+
|
|
147
|
+
- Prefer `xr set-select-value` 1→0 when a test needs observable press and release
|
|
148
|
+
phases. Use `xr set-gamepad-state` for explicit raw gamepad buttons. Application UI
|
|
149
|
+
should use its normal `click` handlers unless the product itself needs lower-level
|
|
150
|
+
pointer phases.
|
|
151
|
+
- **Physics steps on the raw frame delta.** Reloads, screenshots, and
|
|
152
|
+
`ecs snapshot` (full-state serialize) stall the tab for hundreds of ms to
|
|
153
|
+
seconds; a single huge step tunnels thin/falling bodies and pops stacked
|
|
154
|
+
contacts. Don't run heavy CLI commands mid-interaction; snapshot between
|
|
155
|
+
scenarios, not during them. Build apps defensively (see
|
|
156
|
+
build-milestones.md runtime-hitch hardening defaults).
|
|
157
|
+
- **Parse the right payload path**: `browser logs` returns the entries under
|
|
158
|
+
`data.result` (a LIST of {level, message}); `browser screenshot` returns
|
|
159
|
+
`data.screenshotPath` directly under `data`. Shapes vary per command —
|
|
160
|
+
check one real response before scripting assertions.
|
|
161
|
+
- **Log-window noise**: `[IWSDK-MCP-TRACE]` lines flood small `count`
|
|
162
|
+
windows; use `count` 60–100 or a `pattern` filter for game logs.
|
|
163
|
+
- **Queries and screenshots disagreeing** (state changes you can't explain)
|
|
164
|
+
usually means **two browser tabs** are connected — the relay is
|
|
165
|
+
first-response-wins. Check `dev status` → `connectedClientCount`; if >1,
|
|
166
|
+
`npx iwsdk dev restart` to converge on one tab.
|
|
167
|
+
|
|
168
|
+
## Recovery ladder (runtime unresponsive / weird)
|
|
169
|
+
|
|
170
|
+
1. `npx iwsdk xr status` and `npx iwsdk dev status` — what state is it in?
|
|
171
|
+
2. `npx iwsdk browser logs` — crashed app code shows here.
|
|
172
|
+
3. `npx iwsdk browser reload --timeout 20000` — clears bad page state.
|
|
173
|
+
4. `npx iwsdk dev restart` — bad transport/server state.
|
|
174
|
+
5. Re-enter XR (`xr enter`) and re-discover entities after any of the above.
|
|
175
|
+
6. Dev server won't start: check `npx iwsdk dev logs --tail 100`; commonest
|
|
176
|
+
causes are a missing `dev:runtime` script, port conflict from an orphaned
|
|
177
|
+
server (`iwsdk dev down` in that app dir), or Playwright browser revision
|
|
178
|
+
mismatch (`npx playwright@<pinned> install chromium`).
|
|
179
|
+
|
|
180
|
+
## VERIFICATION.md format (Phase 6 output)
|
|
181
|
+
|
|
182
|
+
```markdown
|
|
183
|
+
# <Title> — Verification Report (<date>, commit <sha>)
|
|
184
|
+
|
|
185
|
+
| # | Success criterion | Result | Evidence |
|
|
186
|
+
| --- | ----------------- | ------ | ------------------------------------------ |
|
|
187
|
+
| S1 | throw knocks pins | PASS | diff: pin-3 rot Δ87°; design/verify/s1.png |
|
|
188
|
+
|
|
189
|
+
## Scenario transcripts
|
|
190
|
+
|
|
191
|
+
### S1
|
|
192
|
+
|
|
193
|
+
<the exact commands run + the assertion output snippets>
|
|
194
|
+
|
|
195
|
+
## Deferred / known gaps
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Keep screenshots in `design/verify/`. A criterion without evidence is not a
|
|
199
|
+
PASS.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iwsdk-ray
|
|
3
|
+
description: Ray-based interactions in the WebXR scene — click objects, press UI buttons, or distance-grab with DistanceGrabbable. Use when the user wants to point at and interact with something at a distance, click a UI button, or test ray-based selection.
|
|
4
|
+
argument-hint: <target> [action]
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Ray Interaction
|
|
8
|
+
|
|
9
|
+
Point at and interact with objects or UI elements in the XR scene using the controller ray. The workflow has a **required core** (steps 1-4) and **optional extensions** that depend on whether the target is an object or UI element, and what kind of interaction is needed.
|
|
10
|
+
|
|
11
|
+
User request is in `$ARGUMENTS`.
|
|
12
|
+
|
|
13
|
+
## Required Core
|
|
14
|
+
|
|
15
|
+
These steps always execute in order.
|
|
16
|
+
|
|
17
|
+
### Step 1: Enter XR
|
|
18
|
+
|
|
19
|
+
Check session status. If not in an active XR session, accept and enter.
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
xr_get_session_status → if not sessionActive → xr_accept_session
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
### Step 2: Locate the target
|
|
26
|
+
|
|
27
|
+
**For scene objects:** Find a live entity by name or interaction component, then
|
|
28
|
+
query its component data.
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
ecs_find_entities({namePattern: target}) → ecs_query_entity(entityIndex)
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
**For UI elements:** start from the stable UIKitML identities instead of trying to
|
|
35
|
+
discover an internal panel component:
|
|
36
|
+
|
|
37
|
+
1. Read the UIKITML source file to find the element's `id` (e.g., `<button id="xr-button">`)
|
|
38
|
+
2. Find the manifest asset ID and the placed scene node whose `content.asset` uses it
|
|
39
|
+
3. In application code, resolve the placed `UIKitMLAsset` with
|
|
40
|
+
`world.requireSceneObject(sceneNodeId)` and resolve the element with
|
|
41
|
+
`requireElementById(elementId)`
|
|
42
|
+
4. Add an application-level test hook exposing the element's world target when the
|
|
43
|
+
runtime tools need a precise point to aim at
|
|
44
|
+
|
|
45
|
+
If the element already has a name in the hierarchy, skip straight to getting its transform.
|
|
46
|
+
|
|
47
|
+
If the object is not found, report the available named objects and stop.
|
|
48
|
+
|
|
49
|
+
### Step 3: Get its transform
|
|
50
|
+
|
|
51
|
+
Read the target's Transform position from the queried entity or application test
|
|
52
|
+
hook from step 2.
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
ecs_query_entity(entityIndex) → Transform.position
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### Step 4: Aim the controller
|
|
59
|
+
|
|
60
|
+
Point the controller at the target. Default to `"controller-right"` unless the user specified left. Do NOT move the controller — only rotate it.
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
xr_look_at({ device: "controller-right", target: { x, y, z } })
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The controller ray is now pointing at the target. What happens next depends on the interaction type.
|
|
67
|
+
|
|
68
|
+
## Interaction Branches
|
|
69
|
+
|
|
70
|
+
Based on the user's intent and the target's components, choose ONE of the following.
|
|
71
|
+
|
|
72
|
+
### Branch A: Click / Select (objects with RayInteractable, or UI buttons)
|
|
73
|
+
|
|
74
|
+
For simple clicks, send an explicit trigger press and release. Use for UI buttons and objects that respond to the Pressed component.
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
xr_set_select_value({ device: "controller-right", value: 1 })
|
|
78
|
+
xr_set_select_value({ device: "controller-right", value: 0 })
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
This explicit press/release sequence is deterministic and makes the held state visible
|
|
82
|
+
between the two commands.
|
|
83
|
+
|
|
84
|
+
### Branch B: Distance Grab (objects with DistanceGrabbable)
|
|
85
|
+
|
|
86
|
+
DistanceGrabbable requires **press and hold** on the trigger (button index 0), not a quick select.
|
|
87
|
+
|
|
88
|
+
#### B1: Engage trigger
|
|
89
|
+
|
|
90
|
+
```
|
|
91
|
+
xr_set_gamepad_state({
|
|
92
|
+
device: "controller-right",
|
|
93
|
+
buttons: [{ index: 0, value: 1 }],
|
|
94
|
+
})
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
The object is now distance-grabbed. Behavior depends on the `movementMode`:
|
|
98
|
+
|
|
99
|
+
- **MoveFromTarget / MoveAtSource / RotateAtSource** — object stays remote, moves relative to controller movement
|
|
100
|
+
- **MoveTowardsTarget** — object flies into the controller's hand, then behaves like a proximity grab
|
|
101
|
+
|
|
102
|
+
#### B2: Move to destination (optional)
|
|
103
|
+
|
|
104
|
+
If the user wants to move the object somewhere, animate the controller to the destination.
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
xr_animate_to({
|
|
108
|
+
device: "controller-right",
|
|
109
|
+
position: { x, y, z },
|
|
110
|
+
duration: 0.5,
|
|
111
|
+
})
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
If no destination specified but user asked to "move" or "bring" the object, animate to in front of the headset: `xr_get_transform({ "device": "headset" })` → place at `(head.x, head.y - 0.2, head.z - 0.5)`.
|
|
115
|
+
|
|
116
|
+
#### B3: Release trigger
|
|
117
|
+
|
|
118
|
+
```
|
|
119
|
+
xr_set_gamepad_state({
|
|
120
|
+
device: "controller-right",
|
|
121
|
+
buttons: [{ index: 0, value: 0 }],
|
|
122
|
+
})
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
#### B4: Return controller
|
|
126
|
+
|
|
127
|
+
Animate back to resting position.
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
xr_animate_to({
|
|
131
|
+
device: "controller-right",
|
|
132
|
+
position: { x: 0.2, y: 1.4, z: -0.3 },
|
|
133
|
+
duration: 0.5,
|
|
134
|
+
})
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Default resting positions: right `(0.2, 1.4, -0.3)`, left `(-0.2, 1.4, -0.3)`.
|
|
138
|
+
|
|
139
|
+
### Step 5: Verify (optional)
|
|
140
|
+
|
|
141
|
+
Take a screenshot to confirm the result.
|
|
142
|
+
|
|
143
|
+
```
|
|
144
|
+
browser_screenshot
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
`browser_screenshot` is runtime-only. If the editor is visible, the managed workspace
|
|
148
|
+
switches to runtime before capture.
|
|
149
|
+
|
|
150
|
+
## Notes
|
|
151
|
+
|
|
152
|
+
- **Click vs hold:** use `xr_set_select_value` 1→0 for quick clicks. Hold it at 1 while moving a DistanceGrabbable, then release with 0.
|
|
153
|
+
- **Trigger vs squeeze:** Ray interactions use the **trigger (button index 0)**. Proximity grabs (OneHandGrabbable/TwoHandsGrabbable) use **squeeze (button index 1)**. Don't mix them up.
|
|
154
|
+
- **UI element discovery:** use manifest asset ID → stable scene node ID → UIKitML
|
|
155
|
+
element ID. Expose a test hook for the resolved element when a world-space aim target
|
|
156
|
+
is needed. Guessing from panel offsets or transient entity indices is fragile.
|
|
157
|
+
- **DistanceGrabbable movement modes:** Check the entity's DistanceGrabbable component to see which `movementMode` is set. Use `ecs_query_entity` if unsure.
|
|
158
|
+
- **Don't move the controller to the target** — ray interactions work at a distance. Only rotate via `xr_look_at`, don't translate.
|
|
159
|
+
|
|
@@ -0,0 +1,255 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iwsdk-scene-composer
|
|
3
|
+
description: Compose editable static IWSDK scenes from text, images, or hybrid references using a shared application asset manifest, v1 scene JSON, modular scene files, and the managed editor's validation and rendering tools. Use for 3D environments, props, architecture, staged scenes, procedural Three.js assets, custom PBR or shader materials, glTF assets, prefabs, patterns, lighting, camera matching, or visual review.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# IWSDK Scene Composer
|
|
7
|
+
|
|
8
|
+
Author native IWSDK assets and scene files. Application code owns geometry and
|
|
9
|
+
materials; scene JSON owns composition; the managed editor provides visual feedback
|
|
10
|
+
and human transform/component adjustment.
|
|
11
|
+
|
|
12
|
+
Read these references when relevant:
|
|
13
|
+
|
|
14
|
+
- [scene-format.md](references/scene-format.md) before editing scene JSON;
|
|
15
|
+
- [asset-authoring.md](references/asset-authoring.md) before creating or changing
|
|
16
|
+
glTF or procedural assets;
|
|
17
|
+
- [text-intake.md](references/text-intake.md) for text-only requests;
|
|
18
|
+
- [image-intake.md](references/image-intake.md) for image or hybrid requests;
|
|
19
|
+
- [composition-patterns.md](references/composition-patterns.md) for decomposition
|
|
20
|
+
and repetition strategies;
|
|
21
|
+
- [review-and-stop.md](references/review-and-stop.md) before final review.
|
|
22
|
+
|
|
23
|
+
## Fixed Boundaries
|
|
24
|
+
|
|
25
|
+
- Use only `iwsdk.scene.v1`. There is no compatibility schema.
|
|
26
|
+
- Scene files are the composition source of truth. Create and edit them with normal
|
|
27
|
+
filesystem tools under `public/scenes/`.
|
|
28
|
+
- Scene JSON has one renderable content kind: `asset`. It does not define models,
|
|
29
|
+
primitive geometry, material resources, or material overrides.
|
|
30
|
+
- The default export of the configured application asset manifest is the asset source
|
|
31
|
+
of truth. It may contain URL-backed glTF and UIKitML entries plus parentless
|
|
32
|
+
`Object3D` prototypes with arbitrary Three.js geometry and materials.
|
|
33
|
+
- The application runtime and editor import the same manifest module independently.
|
|
34
|
+
Never depend on shared object identity, iframe messaging, DOM state, or a live
|
|
35
|
+
runtime world when defining assets.
|
|
36
|
+
- Humans use the editor for selection, hierarchy, transforms, components, root
|
|
37
|
+
lighting, and preview visibility. They do not edit geometry or materials there.
|
|
38
|
+
- Agents may edit asset TypeScript and scene JSON, then use the editor to validate and
|
|
39
|
+
render the result.
|
|
40
|
+
|
|
41
|
+
The public scene MCP surface is intentionally small:
|
|
42
|
+
|
|
43
|
+
```text
|
|
44
|
+
scene_open
|
|
45
|
+
scene_render_file
|
|
46
|
+
scene_flatten_file
|
|
47
|
+
scene_get_state
|
|
48
|
+
scene_get_capabilities
|
|
49
|
+
scene_screenshot
|
|
50
|
+
scene_select
|
|
51
|
+
scene_set_camera
|
|
52
|
+
scene_set_preview_visibility
|
|
53
|
+
scene_measure_image_regions
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Document creation and mutation happen through direct file edits. Do not look for MCP
|
|
57
|
+
create/add/update/remove/patch/save/compose/review/publish tools.
|
|
58
|
+
|
|
59
|
+
When MCP is unavailable, use the CLI equivalents:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
npx iwsdk dev status
|
|
63
|
+
npx iwsdk dev up
|
|
64
|
+
npx iwsdk scene capabilities --raw
|
|
65
|
+
npx iwsdk scene render-file \
|
|
66
|
+
--input-json '{"path":"public/scenes/room.iwsdk.scene.json","viewId":"hero"}' \
|
|
67
|
+
--output-file artifacts/room.png
|
|
68
|
+
npx iwsdk scene flatten \
|
|
69
|
+
--input-json '{"path":"public/scenes/room.composition.iwsdk.scene.json","outputPath":"public/scenes/room.iwsdk.scene.json"}' --raw
|
|
70
|
+
npx iwsdk scene open \
|
|
71
|
+
--input-json '{"path":"public/scenes/room.iwsdk.scene.json"}' --raw
|
|
72
|
+
npx iwsdk scene state --raw
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
`iwsdk dev up` starts the server in the background, launches the configured
|
|
76
|
+
managed editor browser, and waits for the command bridge. Do not edit
|
|
77
|
+
`vite.config.ts` to change browser mode as an ad hoc startup workaround.
|
|
78
|
+
|
|
79
|
+
`scene_render_file` renders a file without replacing the editor's active document,
|
|
80
|
+
but it still uses the managed editor browser for manifest evaluation and WebGL.
|
|
81
|
+
If startup reports `dev_browser_not_ready`, inspect `iwsdk dev status` and
|
|
82
|
+
`iwsdk dev logs --tail 100`. Retry only when the diagnostics indicate a transient
|
|
83
|
+
startup failure. Do not invent a custom CPU or Playwright renderer and present it as
|
|
84
|
+
authoritative editor evidence. Preserve the structured failure, continue
|
|
85
|
+
type/schema/build checks that remain meaningful, and report the visual-verification
|
|
86
|
+
gate as blocked.
|
|
87
|
+
|
|
88
|
+
Camera parameters are intentionally distinct: `view` accepts only the built-in
|
|
89
|
+
presets (`current`, `top`, `front`, `back`, `left`, `right`, `quarter`, `orbit`),
|
|
90
|
+
while `viewId` selects an exact camera declared in `authoring.views`. Once a level
|
|
91
|
+
loads, its saved hero view owns runtime framing and supersedes the initial
|
|
92
|
+
`World.create({ render: { camera } })` pose.
|
|
93
|
+
|
|
94
|
+
## Workflow
|
|
95
|
+
|
|
96
|
+
### 1. Specify
|
|
97
|
+
|
|
98
|
+
Turn the request into a compact implementation brief:
|
|
99
|
+
|
|
100
|
+
- required and optional features;
|
|
101
|
+
- source evidence regions for image input;
|
|
102
|
+
- silhouette, proportions, parts, negative space, contacts, and material response;
|
|
103
|
+
- hero and diagnostic views;
|
|
104
|
+
- measurable acceptance criteria;
|
|
105
|
+
- assumptions, uncertainty, and fidelity ceiling.
|
|
106
|
+
|
|
107
|
+
A single image proves visible composition, not hidden geometry. Do not silently invent
|
|
108
|
+
occluded detail or lower requested fidelity.
|
|
109
|
+
|
|
110
|
+
### 2. Plan Assets And Modules
|
|
111
|
+
|
|
112
|
+
Call `scene_get_capabilities` once. Inspect `src/assets.ts` and existing asset modules.
|
|
113
|
+
|
|
114
|
+
Choose an external asset source deliberately:
|
|
115
|
+
|
|
116
|
+
- Use the configured MetaVR asset search, or
|
|
117
|
+
`npx @meta-quest/metavr --json asset search "<query>"`, for ready-made static
|
|
118
|
+
props and background dressing. Results provide previews plus GLB/FBX downloads,
|
|
119
|
+
but do not promise semantic subparts, rigging, articulation, or independently
|
|
120
|
+
editable pieces. Inspect the downloaded hierarchy, and copy selected files into
|
|
121
|
+
project-owned storage instead of persisting a returned CDN URL.
|
|
122
|
+
- Use `npx @drawcall/market skill` and `npx @drawcall/market types`, then search a
|
|
123
|
+
concrete need with `npx @drawcall/market search "<query>" --type <type> --limit 3`,
|
|
124
|
+
when an installable reusable asset, template, or provider-generated result is a
|
|
125
|
+
better starting point. Preview finalists and install the exact printed
|
|
126
|
+
`name@version`; trust the install output for consumer paths.
|
|
127
|
+
- Drawcall Market is a marketplace/install/generation CLI, not a universal
|
|
128
|
+
procedural-geometry engine. When the request depends on controllable parts,
|
|
129
|
+
parametric dimensions, articulation, or code-driven variation and no suitable
|
|
130
|
+
code-backed template exists, author a deterministic Three.js `Object3D` prototype.
|
|
131
|
+
|
|
132
|
+
For every visible form, choose one of:
|
|
133
|
+
|
|
134
|
+
1. reuse an existing manifest asset;
|
|
135
|
+
2. add a glTF entry to the manifest;
|
|
136
|
+
3. register a UIKitML file with `AssetType.UIKitML`;
|
|
137
|
+
4. create a deterministic parentless `Object3D` prototype in code and register it;
|
|
138
|
+
5. assemble existing assets with a scene prefab or module.
|
|
139
|
+
|
|
140
|
+
Create custom geometry and materials in asset code, not JSON. Prefer separate
|
|
141
|
+
`*.scene-asset.ts` modules for substantial procedural assets and import their
|
|
142
|
+
prototypes into `src/assets.ts`.
|
|
143
|
+
|
|
144
|
+
For initial construction, plan independent semantic groups as standalone scratch
|
|
145
|
+
scene modules. Give each module a local origin, size envelope, attachment points,
|
|
146
|
+
required views, and asset IDs. Asset and component IDs are application-global;
|
|
147
|
+
imported node and prefab IDs are namespaced. Imports are an authoring-only assembly
|
|
148
|
+
mechanism, never a runtime or editable-project format.
|
|
149
|
+
|
|
150
|
+
### 3. Build
|
|
151
|
+
|
|
152
|
+
Author assets first, then scene JSON. Build in dependency order:
|
|
153
|
+
|
|
154
|
+
1. support/stage and representative lighting;
|
|
155
|
+
2. large composition masses;
|
|
156
|
+
3. identity-critical groups;
|
|
157
|
+
4. repeated secondary detail;
|
|
158
|
+
5. hero camera and final environment.
|
|
159
|
+
|
|
160
|
+
Use meters, stable descriptive IDs, deterministic ordering, and explicit transforms.
|
|
161
|
+
Groups supply hierarchy, never visible mass. Use `castShadow` and `receiveShadow` on
|
|
162
|
+
asset nodes only when needed. Use prefabs and patterns for repetition; keep repeated
|
|
163
|
+
asset prototypes resource-sharing friendly.
|
|
164
|
+
|
|
165
|
+
### 4. Validate And Materialize
|
|
166
|
+
|
|
167
|
+
With the managed editor command-ready, call `scene_render_file` on every changed
|
|
168
|
+
scratch module, then the composition root. It resolves imports for authoring preview,
|
|
169
|
+
validates schema and manifest references, lowers the scene, and returns a PNG plus
|
|
170
|
+
diagnostics without changing the active document. Fix failures in the owning asset or
|
|
171
|
+
scratch file.
|
|
172
|
+
|
|
173
|
+
After the composition root passes, run `scene_flatten_file` / `iwsdk scene flatten`
|
|
174
|
+
once to materialize an import-free final scene. The command preserves import wrapper
|
|
175
|
+
groups, validates the output, and refuses to write if its runtime hash differs from
|
|
176
|
+
the composed source. This is a one-way publication boundary: the flat file becomes
|
|
177
|
+
the sole source of truth, and later scratch-module changes must not be re-flattened
|
|
178
|
+
over human edits.
|
|
179
|
+
|
|
180
|
+
Call `scene_open` only on the flattened file for live collaboration. Import-bearing
|
|
181
|
+
files remain renderable composition previews but are never opened as editable scenes
|
|
182
|
+
and never load in the application runtime.
|
|
183
|
+
|
|
184
|
+
Use `scene_get_state` for selection, hashes, diagnostics, dirty/conflict state, runtime
|
|
185
|
+
readiness, and render statistics. Use camera, screenshot, selection, and preview
|
|
186
|
+
visibility tools only when their live-editor context is useful.
|
|
187
|
+
|
|
188
|
+
### 5. Review And Refine
|
|
189
|
+
|
|
190
|
+
Review in three passes:
|
|
191
|
+
|
|
192
|
+
1. **Layout**: hierarchy, scale, support contacts, and arrangement.
|
|
193
|
+
2. **Geometry**: silhouette, proportions, parts, negative space, and alternate views.
|
|
194
|
+
3. **Final**: material response, color, lighting, environment, and hero framing.
|
|
195
|
+
|
|
196
|
+
Keep review orchestration and evidence outside the editor. The editor supplies
|
|
197
|
+
authoritative screenshots, hashes, camera state, diagnostics, and render measurements.
|
|
198
|
+
Derive comparisons, defect lists, lineage, and stop decisions in ordinary task files.
|
|
199
|
+
|
|
200
|
+
Fix the highest-impact defect in its owning asset or scene module, rerender that file,
|
|
201
|
+
then rerender the root. Default to two focused correction rounds. Stop earlier on a
|
|
202
|
+
repeated defect, oscillation, plateau, missing input/asset, or representation ceiling.
|
|
203
|
+
|
|
204
|
+
### 6. Finish
|
|
205
|
+
|
|
206
|
+
Finish only when:
|
|
207
|
+
|
|
208
|
+
- every scratch module and the composition root validates and renders;
|
|
209
|
+
- the final editable scene is flattened and contains no `imports`;
|
|
210
|
+
- the active editor state is clean and conflict-free;
|
|
211
|
+
- required views are nonblank and correctly framed;
|
|
212
|
+
- required features pass measurable and visual checks;
|
|
213
|
+
- manifest asset IDs resolve in both editor and application runtime;
|
|
214
|
+
- the application build and selected scene load without blocking errors.
|
|
215
|
+
|
|
216
|
+
If a required gate is unavailable, finish with an explicit blocked or
|
|
217
|
+
accepted-with-gaps result. Passing a local schema check, production build, or custom
|
|
218
|
+
diagnostic image does not substitute for authoritative editor renders and state.
|
|
219
|
+
|
|
220
|
+
## Modular Composition
|
|
221
|
+
|
|
222
|
+
```json
|
|
223
|
+
{
|
|
224
|
+
"version": "iwsdk.scene.v1",
|
|
225
|
+
"units": "meters",
|
|
226
|
+
"imports": [
|
|
227
|
+
{
|
|
228
|
+
"id": "reading-nook",
|
|
229
|
+
"src": "./modules/reading-nook.iwsdk.scene.json",
|
|
230
|
+
"transform": { "position": [1.8, 0, -0.6] }
|
|
231
|
+
}
|
|
232
|
+
],
|
|
233
|
+
"resources": {},
|
|
234
|
+
"nodes": []
|
|
235
|
+
}
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
Each scratch module must be valid by itself. Imports resolve recursively in
|
|
239
|
+
declaration order. The import entry becomes a transform group. The composition root
|
|
240
|
+
owns global components, environment, metadata, and authoring settings. Cycles, unsafe
|
|
241
|
+
IDs, missing files, duplicate namespaced IDs, and invalid modules fail composition.
|
|
242
|
+
|
|
243
|
+
For parallel initial construction, assign one scratch module file per worker. Never
|
|
244
|
+
let two workers edit one file. Render modules independently, import only passing
|
|
245
|
+
modules, correct cross-module scale, contact, occlusion, lighting, and framing at the
|
|
246
|
+
composition root, then flatten exactly once. Parallel module iteration ends at that
|
|
247
|
+
boundary; continue all later edits in the flat file.
|
|
248
|
+
|
|
249
|
+
## Regeneration And Provenance
|
|
250
|
+
|
|
251
|
+
Preserve stable IDs when revising the flat file. Never overwrite unrelated
|
|
252
|
+
human-authored files or re-flatten over editor changes. Record the skill/runtime
|
|
253
|
+
versions, input hashes, composition/final/module paths, capability hash,
|
|
254
|
+
source/composed/runtime hashes, assumptions, and fidelity ceiling in authoring
|
|
255
|
+
metadata or adjacent task evidence.
|