@iwsdk/create 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +103 -41
- package/dist/cli.js +1003 -3418
- package/dist/guidance/agents/.agents/skills/iwsdk-debug/SKILL.md +96 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-depth-occlusion/SKILL.md +135 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-grab/SKILL.md +129 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/SKILL.md +92 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/references/component-reference.md +154 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/references/tuning-and-config.md +88 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-physics/references/workflows.md +303 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/SKILL.md +364 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/api-reference.md +1489 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/build-milestones.md +241 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/design-deck.md +101 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/grounding.md +176 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/ideation.md +156 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-planner/references/verification.md +199 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-ray/SKILL.md +159 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/SKILL.md +255 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/agents/openai.yaml +5 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/asset-authoring.md +295 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/composition-patterns.md +243 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/image-intake.md +191 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/review-and-stop.md +99 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/scene-format.md +458 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-scene-composer/references/text-intake.md +82 -0
- package/dist/guidance/agents/.agents/skills/iwsdk-ui/SKILL.md +208 -0
- package/dist/guidance/claude/.claude/agents/iwsdk-project-code-reviewer.md +450 -0
- package/dist/guidance/claude/.claude/rules/assets-and-manifest.md +55 -0
- package/dist/guidance/claude/.claude/rules/ecs-api.md +102 -0
- package/dist/guidance/claude/.claude/rules/scene-json.md +64 -0
- package/dist/guidance/claude/.claude/rules/uikitml.md +60 -0
- package/dist/guidance/claude/.claude/settings.json +19 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-debug/SKILL.md +96 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-depth-occlusion/SKILL.md +135 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-grab/SKILL.md +129 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/SKILL.md +92 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/references/component-reference.md +154 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/references/tuning-and-config.md +88 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-physics/references/workflows.md +303 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/SKILL.md +364 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/api-reference.md +1489 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/build-milestones.md +241 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/design-deck.md +101 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/grounding.md +176 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/ideation.md +156 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-planner/references/verification.md +199 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-ray/SKILL.md +159 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/SKILL.md +255 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/agents/openai.yaml +5 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/asset-authoring.md +295 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/composition-patterns.md +243 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/image-intake.md +191 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/review-and-stop.md +99 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/scene-format.md +458 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-scene-composer/references/text-intake.md +82 -0
- package/dist/guidance/claude/.claude/skills/iwsdk-ui/SKILL.md +208 -0
- package/dist/guidance/claude/CLAUDE.md +142 -0
- package/dist/guidance/codex/.codex/config.toml +2 -0
- package/dist/guidance/common/AGENTS.md +151 -0
- package/dist/guidance/copilot/.github/instructions/assets-and-manifest.instructions.md +52 -0
- package/dist/guidance/copilot/.github/instructions/ecs-api.instructions.md +100 -0
- package/dist/guidance/copilot/.github/instructions/scene-json.instructions.md +63 -0
- package/dist/guidance/copilot/.github/instructions/uikitml.instructions.md +59 -0
- package/dist/guidance/cursor/.cursor/rules/assets-and-manifest.mdc +57 -0
- package/dist/guidance/cursor/.cursor/rules/ecs-api.mdc +104 -0
- package/dist/guidance/cursor/.cursor/rules/scene-json.mdc +66 -0
- package/dist/guidance/cursor/.cursor/rules/uikitml.mdc +62 -0
- package/dist/guidance/scoped-agents/public/scenes/AGENTS.md +63 -0
- package/dist/guidance/scoped-agents/public/ui/AGENTS.md +59 -0
- package/dist/guidance/scoped-agents/src/AGENTS.md +149 -0
- package/dist/template/common/README.md +13 -0
- package/dist/template/common/gitignore.template +45 -0
- package/dist/template/common/index.html +16 -0
- package/dist/template/common/nvmrc.template +1 -0
- package/dist/template/common/public/audio/chime.mp3 +0 -0
- package/dist/template/common/public/gltf/webxr-banner/banner.gltf +68 -0
- package/dist/template/common/public/textures/webxr.png +0 -0
- package/dist/template/common/public/ui/welcome.uikitml +98 -0
- package/dist/template/common/src/assets.ts +54 -0
- package/dist/template/common/src/components.ts +11 -0
- package/dist/template/common/src/index.ts +19 -0
- package/dist/template/common/src/panel.ts +38 -0
- package/dist/template/common/src/robot-component.ts +10 -0
- package/dist/template/common/src/robot.ts +38 -0
- package/dist/template/common/src/vite-env.d.ts +2 -0
- package/dist/template/common/tsconfig.json +15 -0
- package/dist/template/common/vite.config.ts +27 -0
- package/dist/template/common-js/README.md +13 -0
- package/dist/template/common-js/gitignore.template +45 -0
- package/dist/template/common-js/index.html +16 -0
- package/dist/template/common-js/nvmrc.template +1 -0
- package/dist/template/common-js/public/audio/chime.mp3 +0 -0
- package/dist/template/common-js/public/gltf/webxr-banner/banner.gltf +68 -0
- package/dist/template/common-js/public/textures/webxr.png +0 -0
- package/dist/template/common-js/public/ui/welcome.uikitml +98 -0
- package/dist/template/common-js/src/assets.js +45 -0
- package/dist/template/common-js/src/components.js +9 -0
- package/dist/template/common-js/src/index.js +14 -0
- package/dist/template/common-js/src/panel.js +31 -0
- package/dist/template/common-js/src/robot-component.js +8 -0
- package/dist/template/common-js/src/robot.js +37 -0
- package/dist/template/common-js/vite.config.js +25 -0
- package/dist/template/scenes/ar.iwsdk.scene.json +96 -0
- package/dist/template/scenes/immersive.iwsdk.scene.json +109 -0
- package/package.json +23 -17
- package/scripts/verify-release-ready.mjs +76 -0
- package/LICENSE +0 -21
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
---
|
|
2
|
+
paths:
|
|
3
|
+
- "src/**/*.ts"
|
|
4
|
+
- "src/**/*.js"
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# ECS API reference
|
|
8
|
+
|
|
9
|
+
## Field types
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
Types.Float32 Float64 Int8 Int16 Int32 Boolean String FilePath
|
|
13
|
+
Types.Vec2 Vec3 Vec4 Color(RGBA) Entity Enum
|
|
14
|
+
Types.Object // avoid — not optimised
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Component
|
|
18
|
+
|
|
19
|
+
Declare in a system-free module, then list it in `src/components.ts` via
|
|
20
|
+
`defineComponents()` or the editor cannot author it.
|
|
21
|
+
|
|
22
|
+
```ts
|
|
23
|
+
export const MyComponent = createComponent('MyComponent', {
|
|
24
|
+
speed: { type: Types.Float32, default: 1 },
|
|
25
|
+
tint: { type: Types.Color, default: [1, 1, 1, 1] }, // RGBA
|
|
26
|
+
});
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Optional metadata per field — `label`, `step`, `min`, `max`, `enum`, `help`,
|
|
30
|
+
`widget` — drives the editor inspector.
|
|
31
|
+
|
|
32
|
+
## System
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
export class MySystem extends createSystem(
|
|
36
|
+
{
|
|
37
|
+
items: { required: [MyComponent] },
|
|
38
|
+
active: { required: [MyComponent], excluded: [Grabbed] },
|
|
39
|
+
},
|
|
40
|
+
{ speed: { type: Types.Float32, default: 1 } }, // optional config
|
|
41
|
+
) {
|
|
42
|
+
private temp!: Vector3;
|
|
43
|
+
|
|
44
|
+
init() {
|
|
45
|
+
this.temp = new Vector3();
|
|
46
|
+
this.queries.items.subscribe('qualify', (entity) => { /* matched */ });
|
|
47
|
+
this.cleanupFuncs.push(this.config.speed.subscribe((v) => { /* … */ }));
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
update(delta: number, time: number) {
|
|
51
|
+
for (const entity of this.queries.active.entities) { /* no allocations */ }
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Queries take `required`, `excluded` and `where` (value predicates via `eq`, `ne`,
|
|
57
|
+
`lt`, `le`, `gt`, `ge`, `isin`, `nin`). `excluded` is the clean way to skip
|
|
58
|
+
entities in a transient state — e.g. excluding `Grabbed` so a held object is left
|
|
59
|
+
alone.
|
|
60
|
+
|
|
61
|
+
Register explicitly in `src/index.ts` with `world.registerSystem(MySystem)`.
|
|
62
|
+
|
|
63
|
+
Read fields with `entity.getValue(Component, 'field')`. For vector fields use
|
|
64
|
+
`entity.getVectorView(Component, 'field')` — it returns a `Float32Array` view with
|
|
65
|
+
no allocation, and `setValue` **throws** on vector fields in elics 3.4.x.
|
|
66
|
+
|
|
67
|
+
## Interaction components
|
|
68
|
+
|
|
69
|
+
`RayInteractable` plus `Hovered`/`Pressed` covers both mouse/touch canvas input
|
|
70
|
+
and XR rays. Grab components: `OneHandGrabbable` and `TwoHandsGrabbable` are
|
|
71
|
+
proximity-based and respond to **squeeze**, while `DistanceGrabbable` is
|
|
72
|
+
ray-based and responds to **trigger**. `Grabbed` is a transient tag managed by
|
|
73
|
+
`GrabSystem` — read it, never add or remove it. `GrabSystem.useHandPinchForGrab`
|
|
74
|
+
defaults to **false**, so hand pinch does not grab unless you enable it.
|
|
75
|
+
|
|
76
|
+
`DistanceGrabbable` fields: `movementMode` (`MoveTowardsTarget`, `MoveFromTarget`,
|
|
77
|
+
`MoveAtSource`, `RotateAtSource`), `returnToOrigin`, `moveSpeedFactor`,
|
|
78
|
+
`detachOnGrab`, `translate`, `rotate`, `scale`, `targetPositionOffset`,
|
|
79
|
+
`targetQuaternionOffset`. `returnToOrigin: true` gives spring-back on release with
|
|
80
|
+
no custom code.
|
|
81
|
+
|
|
82
|
+
For a held object use `GrabSystem`'s public methods — `forceRelease(entity)`,
|
|
83
|
+
`getHolderHand(entity)` — never deep-import `Handle`.
|
|
84
|
+
|
|
85
|
+
## XR input
|
|
86
|
+
|
|
87
|
+
```ts
|
|
88
|
+
const pad = this.input.xr.gamepads.right;
|
|
89
|
+
pad?.getButtonDown(InputComponent.Trigger); // just pressed
|
|
90
|
+
pad?.getAxesValues(InputComponent.Thumbstick);
|
|
91
|
+
this.player.head; this.player.raySpaces.left; this.player.gripSpaces.right;
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Prefer `world.input.actions` for reusable intent (`locomotion.move`) over raw
|
|
95
|
+
buttons. Browser controls live at `world.input.keyboard` and
|
|
96
|
+
`world.input.browserGamepads`.
|
|
97
|
+
|
|
98
|
+
## Lifecycle
|
|
99
|
+
|
|
100
|
+
`world.visibilityState` reports `NonImmersive`, `Visible`, `VisibleBlurred` —
|
|
101
|
+
pause simulation on blur. `world.camera.position` is local to `world.player`; use
|
|
102
|
+
`getWorldPosition()` when logic needs the true viewer position.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
---
|
|
2
|
+
paths:
|
|
3
|
+
- "public/scenes/**"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Scene JSON — things that fail silently
|
|
7
|
+
|
|
8
|
+
Format reference lives in
|
|
9
|
+
`.claude/skills/iwsdk-scene-composer/references/scene-format.md`: the document
|
|
10
|
+
boundary, content kinds, prefabs, patterns, light fields, authoring views. Read
|
|
11
|
+
it before composing. This file is only the short list of behaviours that produce
|
|
12
|
+
no error when you get them wrong.
|
|
13
|
+
|
|
14
|
+
Only `iwsdk.scene.v1` exists, units are always meters, and structural objects are
|
|
15
|
+
closed — an unknown field fails validation.
|
|
16
|
+
|
|
17
|
+
## Imports stop at the authoring boundary
|
|
18
|
+
|
|
19
|
+
Authoring tools can validate and render a root with `imports`, so modular scratch
|
|
20
|
+
scenes remain useful for composition. The editable scene session and application
|
|
21
|
+
runtime deliberately reject imports. Run `scene_flatten_file` or
|
|
22
|
+
`npx iwsdk scene flatten` once, verify the source and output runtime hashes match,
|
|
23
|
+
then make the flat output the sole editable source of truth. Never re-flatten over
|
|
24
|
+
later human editor changes.
|
|
25
|
+
|
|
26
|
+
## The player materialises at the scene origin
|
|
27
|
+
|
|
28
|
+
`0,0,0` unless the document authors `player.transform`. Nothing warns you. A
|
|
29
|
+
scene can pass every authored camera view and still spawn the player inside the
|
|
30
|
+
geometry, because review cameras are not the viewpoint that ships. Keep that
|
|
31
|
+
volume clear or place the player deliberately, and author a diagnostic view at
|
|
32
|
+
player eye height to prove it.
|
|
33
|
+
|
|
34
|
+
## `visibleNodeIds` is the silent-failure detector
|
|
35
|
+
|
|
36
|
+
`renderStats.visibleNodeIds` lists what actually rendered. A node missing from it
|
|
37
|
+
did not appear — even when the image looks plausible and `valid` is `true`. Check
|
|
38
|
+
it against what you expect rather than trusting the picture. Single-sided
|
|
39
|
+
surfaces facing away from the camera are the usual cause, and they report nothing.
|
|
40
|
+
|
|
41
|
+
## Component keys come in two forms
|
|
42
|
+
|
|
43
|
+
Root-level components use the namespaced form
|
|
44
|
+
(`com.iwsdk.components.DomeGradient`). Node-level components accept the bare name
|
|
45
|
+
(`RayInteractable`, `LocomotionEnvironment`). **The editor writes the namespaced
|
|
46
|
+
form on nodes**, so a file touched by both hands contains both. Match whatever the
|
|
47
|
+
node already uses instead of normalising.
|
|
48
|
+
|
|
49
|
+
`LevelRoot`, `LevelTag` and `Transform` are runtime-owned; never author them.
|
|
50
|
+
|
|
51
|
+
## The scene owns the runtime camera
|
|
52
|
+
|
|
53
|
+
The hero authoring view supersedes `world.render.camera` from
|
|
54
|
+
`iwsdk.config.json` once the level initialises. Changing the config camera to fix
|
|
55
|
+
runtime framing will appear to do nothing — change the hero view instead.
|
|
56
|
+
|
|
57
|
+
## Validation
|
|
58
|
+
|
|
59
|
+
`scene_render_file` validates, composes, lowers and renders without touching the
|
|
60
|
+
open document. A result is usable only when `valid: true` **and** a PNG comes
|
|
61
|
+
back. Render each changed module before the root.
|
|
62
|
+
|
|
63
|
+
For composing a scene rather than editing one, use the `iwsdk-scene-composer`
|
|
64
|
+
skill — it owns the intake, review and stop-rule procedure this file does not.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
---
|
|
2
|
+
paths:
|
|
3
|
+
- "public/ui/**"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# UIKitML panels
|
|
7
|
+
|
|
8
|
+
HTML/CSS-shaped, not a browser engine. Validate rather than assume support.
|
|
9
|
+
|
|
10
|
+
**Numeric sizes are UIKit units (centimetres), not pixels.** `width: 95` is a
|
|
11
|
+
0.95 m panel. Font sizes follow the same scale: `font-size: 3` is 3 cm tall.
|
|
12
|
+
|
|
13
|
+
**Panels are single-sided and face +Z.** To aim one at a camera, yaw by
|
|
14
|
+
`atan2(dx, dz)` toward it — there is no 180° term. A back-facing panel renders
|
|
15
|
+
nothing, reports no error, and still returns `valid: true` from
|
|
16
|
+
`scene_render_file`; the only signal is its absence from
|
|
17
|
+
`renderStats.visibleNodeIds`.
|
|
18
|
+
|
|
19
|
+
Verified working: `display: flex`, `flex-direction`, `align-items`,
|
|
20
|
+
`justify-content: space-between`, `gap`, `margin-*`, `padding-*`, `width`,
|
|
21
|
+
`height`, `background-color`, `border-color`, `border-width`, `border-radius`,
|
|
22
|
+
`font-size`, `font-weight`, `color`, `text-align`, `cursor`. Tags `div`, `span`,
|
|
23
|
+
`button` all work.
|
|
24
|
+
|
|
25
|
+
Expand shorthands (`padding: 12 18` → four properties), use unitless
|
|
26
|
+
line-height multipliers, and prefer uniform `border-color`/`border-width` over
|
|
27
|
+
directional variants. Parser errors include source locations.
|
|
28
|
+
|
|
29
|
+
## Runtime API
|
|
30
|
+
|
|
31
|
+
Resolve the placed panel by **scene node id**, then elements by their `id`
|
|
32
|
+
attribute:
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
const panel = world.getSceneObject<UIKitMLAsset>('config-panel');
|
|
36
|
+
const chip = panel?.getElementById<UIKit.Text>('finish-gold');
|
|
37
|
+
chip?.addEventListener('click', onClick);
|
|
38
|
+
chip?.setProperties({ backgroundColor: '#d8e6f5', color: '#0e1116' });
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
`setProperties` updates both text (`{ text }`) and style
|
|
42
|
+
(`backgroundColor`, `color`, `borderColor`, `display`) — style changes do apply,
|
|
43
|
+
including over class-based styles from the `<style>` block. `UIKit.Text` is the
|
|
44
|
+
public element type; no deep import from `@pmndrs/uikit` is needed.
|
|
45
|
+
|
|
46
|
+
Three distinct identities, don't conflate them: the **manifest ID** names the
|
|
47
|
+
reusable asset, the **scene node ID** names the placed instance, and the element
|
|
48
|
+
`id` names a field inside it. Never depend on a transient entity index.
|
|
49
|
+
|
|
50
|
+
Give the panel node `RayInteractable` so clicks land.
|
|
51
|
+
|
|
52
|
+
The editor render does not run application systems, so `scene_screenshot` can
|
|
53
|
+
never prove a panel *behaves* — only `browser_screenshot`, against the runtime,
|
|
54
|
+
can.
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
For the authoring procedure — isolated preview, placement, editing, wiring
|
|
59
|
+
runtime behavior, and the completion checklist — use the `iwsdk-ui` skill rather
|
|
60
|
+
than improvising. This file is the format reference it relies on.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json.schemastore.org/claude-code-settings.json",
|
|
3
|
+
"permissions": {
|
|
4
|
+
"allow": [
|
|
5
|
+
"mcp__iwsdk-runtime__*",
|
|
6
|
+
"mcp__iwsdk-reference__*",
|
|
7
|
+
"mcp__metavr__*",
|
|
8
|
+
"Skill(iwsdk-debug)",
|
|
9
|
+
"Skill(iwsdk-depth-occlusion)",
|
|
10
|
+
"Skill(iwsdk-grab)",
|
|
11
|
+
"Skill(iwsdk-physics)",
|
|
12
|
+
"Skill(iwsdk-scene-composer)",
|
|
13
|
+
"Skill(iwsdk-planner)",
|
|
14
|
+
"Skill(iwsdk-ray)",
|
|
15
|
+
"Skill(iwsdk-ui)"
|
|
16
|
+
]
|
|
17
|
+
},
|
|
18
|
+
"enabledMcpjsonServers": ["iwsdk-runtime", "iwsdk-reference", "metavr"]
|
|
19
|
+
}
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iwsdk-debug
|
|
3
|
+
description: Debug continuous behavior in WebXR scenes — physics, animations, collisions, game loops, or any real-time interaction that happens too fast for an agent to observe. Uses ECS pause/step/snapshot/diff to freeze time and inspect state frame by frame.
|
|
4
|
+
argument-hint: '[description of behavior to debug]'
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Debug Continuous Behavior
|
|
8
|
+
|
|
9
|
+
Real-time behavior (physics, animations, collisions, game loops) happens too fast for an agent to observe directly. By the time you take a screenshot, the action is over. This skill uses ECS time-control tools to freeze, step, and diff state frame by frame.
|
|
10
|
+
|
|
11
|
+
User request is in `$ARGUMENTS`.
|
|
12
|
+
|
|
13
|
+
## Core Workflow
|
|
14
|
+
|
|
15
|
+
Every debugging session follows this pattern:
|
|
16
|
+
|
|
17
|
+
1. **Set up** the scenario (position objects, aim controllers, etc.)
|
|
18
|
+
2. **`ecs_pause`** — freeze ECS updates right before the interesting moment
|
|
19
|
+
3. **`ecs_snapshot({ "label": "before" })`** — capture state before the action
|
|
20
|
+
4. **Trigger** the action (release grip, apply force, start animation, etc.)
|
|
21
|
+
5. **`ecs_step(count, delta)`** — advance a few frames at fixed timestep
|
|
22
|
+
6. **`browser_screenshot`** — visually verify what happened
|
|
23
|
+
7. **`ecs_snapshot({ "label": "after" })`** — capture state after stepping
|
|
24
|
+
8. **`ecs_diff({ "from": "before", "to": "after" })`** — see exactly what changed
|
|
25
|
+
9. **Repeat** steps 5-8, stepping further until the behavior completes
|
|
26
|
+
10. **`ecs_resume`** — return to normal execution when done
|
|
27
|
+
|
|
28
|
+
The key insight: **pause BEFORE triggering the action**, not after. If you pause after, you've already missed the first frames.
|
|
29
|
+
|
|
30
|
+
## Tool Reference
|
|
31
|
+
|
|
32
|
+
| Tool | Purpose |
|
|
33
|
+
| ------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
34
|
+
| `ecs_pause` | Freeze all ECS system updates. Render loop continues — screenshots still work. |
|
|
35
|
+
| `ecs_step({"count":N,"delta":SECONDS})` | Advance N frames with fixed timestep (seconds). Must pause first. |
|
|
36
|
+
| `ecs_resume` | Resume normal execution. First frame uses capped delta to avoid physics explosions. |
|
|
37
|
+
| `ecs_snapshot({"label":"..."})` | Capture full ECS state. Stores up to 2 snapshots. |
|
|
38
|
+
| `ecs_diff({"from":"...","to":"..."})` | Compare two snapshots. Shows added/removed entities and field-level value changes. |
|
|
39
|
+
| `ecs_toggle_system({"name":"...","paused":true})` | Pause/resume a single system. Use `ecs_list_systems` to discover names. |
|
|
40
|
+
| `browser_screenshot` | Runtime visual verification — works while paused and auto-switches from editor view. |
|
|
41
|
+
|
|
42
|
+
## Stepping Guidelines
|
|
43
|
+
|
|
44
|
+
- **`delta`** is in seconds. Common values: `0.016` (60fps), `0.0139` (72fps/Quest refresh rate).
|
|
45
|
+
- **Start small** — step 1-3 frames first to catch the initial moment, then step more.
|
|
46
|
+
- **Don't overshoot** — stepping 100 frames at once defeats the purpose. Step in batches of 5-20.
|
|
47
|
+
|
|
48
|
+
## Patterns
|
|
49
|
+
|
|
50
|
+
Short domain-specific tips. Apply the core workflow above, plus these hints.
|
|
51
|
+
|
|
52
|
+
### Physics (falling, bouncing, collisions)
|
|
53
|
+
|
|
54
|
+
- Pause BEFORE releasing the object or applying force.
|
|
55
|
+
- Step 1-3 frames at `delta: 0.016` to catch initial acceleration.
|
|
56
|
+
- In diffs, check `PhysicsBody._linearVelocity` and `PhysicsBody._angularVelocity` to see motion direction and speed.
|
|
57
|
+
- Check `Transform.position` to track movement.
|
|
58
|
+
- If an object falls through a surface: verify the surface entity has both `PhysicsBody` (Static) and `PhysicsShape` (TriMesh for complex geometry).
|
|
59
|
+
- Use `ecs_query_entity` to inspect `PhysicsShape` and `PhysicsBody` on both the falling object and the surface.
|
|
60
|
+
|
|
61
|
+
### Grab and Throw
|
|
62
|
+
|
|
63
|
+
- Pause while the object is still held (trigger/grip engaged).
|
|
64
|
+
- Release the input (set button value to 0) while paused.
|
|
65
|
+
- Step frame by frame to observe the release velocity.
|
|
66
|
+
- In diffs, `PhysicsBody._linearVelocity` shows the throw direction and speed.
|
|
67
|
+
- If the object doesn't move after release: check that it has `PhysicsBody` with `state: Dynamic`.
|
|
68
|
+
|
|
69
|
+
### Animations and Transitions
|
|
70
|
+
|
|
71
|
+
- Step 1 frame at a time with `delta` matching your target framerate.
|
|
72
|
+
- Compare `Transform.position`, `Transform.orientation`, and `Transform.scale` across snapshots to track interpolation.
|
|
73
|
+
- Use screenshots between steps to observe visual progression.
|
|
74
|
+
|
|
75
|
+
### Collision Detection
|
|
76
|
+
|
|
77
|
+
- Pause just before two objects meet.
|
|
78
|
+
- Step 1 frame at a time.
|
|
79
|
+
- Watch for `PhysicsBody._linearVelocity` sign changes (indicates bounce/impact).
|
|
80
|
+
- Watch for `PhysicsBody._angularVelocity` spikes (indicates tumbling from impact).
|
|
81
|
+
- If objects pass through each other: check `PhysicsShape` exists on both entities, and verify shape types are appropriate (use `TriMesh` for complex static geometry).
|
|
82
|
+
|
|
83
|
+
### System Isolation
|
|
84
|
+
|
|
85
|
+
- Use `ecs_list_systems` to see all systems and their priorities.
|
|
86
|
+
- Use `ecs_toggle_system({ "name": "SystemName", "paused": true })` to pause a suspect system while others run.
|
|
87
|
+
- Step forward and observe — if the bug disappears, that system is the cause.
|
|
88
|
+
- Remember to unpause the system when done.
|
|
89
|
+
|
|
90
|
+
## Notes
|
|
91
|
+
|
|
92
|
+
- **Snapshots overwrite** — only 2 are stored. Label them clearly ("before"/"after") and diff before taking new ones.
|
|
93
|
+
- **Resume is safe** — the first frame after resume uses a capped delta to prevent physics explosions from accumulated time.
|
|
94
|
+
- **Render loop continues while paused** — screenshots always work, and the XR session stays alive.
|
|
95
|
+
- **Stepping requires pause** — `ecs_step` will fail if you haven't called `ecs_pause` first.
|
|
96
|
+
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iwsdk-depth-occlusion
|
|
3
|
+
description: Guide for implementing depth sensing and occlusion in IWSDK projects. Use when adding depth-based occlusion to hide virtual objects behind real-world surfaces, configuring DepthSensingSystem, choosing occlusion modes, or troubleshooting objects that disappear or fail to occlude.
|
|
4
|
+
argument-hint: '[description of depth occlusion task]'
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Depth Occlusion
|
|
8
|
+
|
|
9
|
+
Hide virtual objects behind real-world surfaces using WebXR depth sensing. The system samples a per-pixel depth texture from the XR device and compares it against each virtual fragment's depth — if the real surface is closer, the fragment is faded out.
|
|
10
|
+
|
|
11
|
+
## Setup
|
|
12
|
+
|
|
13
|
+
Three things are required: XR session depth config, the system, and the component.
|
|
14
|
+
|
|
15
|
+
### 1. Enable depth sensing on the XR session
|
|
16
|
+
|
|
17
|
+
```jsonc
|
|
18
|
+
// iwsdk.config.json
|
|
19
|
+
{
|
|
20
|
+
"world": {
|
|
21
|
+
"xr": {
|
|
22
|
+
"mode": "ar",
|
|
23
|
+
"referenceSpace": "unbounded",
|
|
24
|
+
"features": {
|
|
25
|
+
"depthSensing": {
|
|
26
|
+
"required": true,
|
|
27
|
+
"usage": "gpu-optimized",
|
|
28
|
+
"format": "float32"
|
|
29
|
+
},
|
|
30
|
+
"hitTest": { "required": true },
|
|
31
|
+
"anchors": { "required": true },
|
|
32
|
+
"unbounded": { "required": true }
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
}
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
### 2. Register `DepthSensingSystem` and `DepthOccludable`
|
|
40
|
+
|
|
41
|
+
```typescript
|
|
42
|
+
import { DepthSensingSystem, DepthOccludable } from '@iwsdk/core';
|
|
43
|
+
|
|
44
|
+
world
|
|
45
|
+
.registerSystem(DepthSensingSystem, {
|
|
46
|
+
configData: {
|
|
47
|
+
enableDepthTexture: true,
|
|
48
|
+
enableOcclusion: true,
|
|
49
|
+
useFloat32: true,
|
|
50
|
+
blurRadius: 20.0,
|
|
51
|
+
},
|
|
52
|
+
});
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### 3. Add `DepthOccludable` to entities
|
|
56
|
+
|
|
57
|
+
```typescript
|
|
58
|
+
import { DepthOccludable, OcclusionShadersMode } from '@iwsdk/core';
|
|
59
|
+
|
|
60
|
+
// Soft occlusion (default) — smooth edges via 13-tap blur
|
|
61
|
+
entity.addComponent(DepthOccludable);
|
|
62
|
+
|
|
63
|
+
// Hard occlusion — sharp edges, single depth sample
|
|
64
|
+
entity.addComponent(DepthOccludable, {
|
|
65
|
+
mode: OcclusionShadersMode.HardOcclusion,
|
|
66
|
+
});
|
|
67
|
+
|
|
68
|
+
// MinMax occlusion — best quality, extra preprocessing pass
|
|
69
|
+
entity.addComponent(DepthOccludable, {
|
|
70
|
+
mode: OcclusionShadersMode.MinMaxSoftOcclusion,
|
|
71
|
+
});
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
The material must have `transparent: true`. The system sets this automatically, but verify it on custom materials.
|
|
75
|
+
|
|
76
|
+
## Occlusion Modes
|
|
77
|
+
|
|
78
|
+
| Mode | Quality | Cost | Best For |
|
|
79
|
+
| --------------------- | ------- | ------ | ----------------------------------------------------------------- |
|
|
80
|
+
| `SoftOcclusion` | Good | Low | Most objects — smooth edges, hides depth aliasing |
|
|
81
|
+
| `HardOcclusion` | Basic | Lowest | Small objects or when sharp edges are acceptable |
|
|
82
|
+
| `MinMaxSoftOcclusion` | Best | Medium | Large objects with complex silhouettes against varied backgrounds |
|
|
83
|
+
|
|
84
|
+
## DepthSensingSystem Config
|
|
85
|
+
|
|
86
|
+
| Property | Type | Default | Description |
|
|
87
|
+
| -------------------- | ------- | ------- | ----------------------------------------- |
|
|
88
|
+
| `enableOcclusion` | Boolean | `true` | Master switch for all occlusion |
|
|
89
|
+
| `enableDepthTexture` | Boolean | `true` | Create GPU textures from depth data |
|
|
90
|
+
| `useFloat32` | Boolean | `true` | Float32 depth textures (higher precision) |
|
|
91
|
+
| `blurRadius` | Float32 | `20.0` | Blur radius for soft occlusion (pixels) |
|
|
92
|
+
|
|
93
|
+
## Depth Sensing Modes
|
|
94
|
+
|
|
95
|
+
| Mode | When to use |
|
|
96
|
+
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
97
|
+
| `cpu-optimized` | Simpler, works everywhere. Depth as linear meters in a DataArrayTexture. |
|
|
98
|
+
| `gpu-optimized` | **Recommended.** Matches Quest hardware format. Depth as reverse-Z inverse depth in an ExternalTexture. Required for production parity with on-device behavior. Note that Quest devices only support this mode. |
|
|
99
|
+
|
|
100
|
+
## AR Session Requirements
|
|
101
|
+
|
|
102
|
+
Depth occlusion only works in AR mode. The scene background must be `null` for passthrough:
|
|
103
|
+
|
|
104
|
+
```typescript
|
|
105
|
+
scene.background = null;
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
## Troubleshooting
|
|
109
|
+
|
|
110
|
+
**Objects never occlude (always visible on top)**
|
|
111
|
+
|
|
112
|
+
- Verify the entity has `DepthOccludable` component
|
|
113
|
+
- Verify `DepthSensingSystem` is registered with `enableOcclusion: true`
|
|
114
|
+
- Check that `depthSensing` is in the XR features config
|
|
115
|
+
|
|
116
|
+
**Objects always invisible in IWER**
|
|
117
|
+
|
|
118
|
+
- In the IWER emulator, the SEM must have loaded environment geometry. If no room is loaded, no depth data is produced.
|
|
119
|
+
- Check the console for "Warning: depth-sensing feature not enabled"
|
|
120
|
+
|
|
121
|
+
**Flickering or noisy occlusion edges**
|
|
122
|
+
|
|
123
|
+
- Increase `blurRadius` (try 30-40)
|
|
124
|
+
- Switch from `HardOcclusion` to `SoftOcclusion`
|
|
125
|
+
- Use `MinMaxSoftOcclusion` for best edge quality
|
|
126
|
+
|
|
127
|
+
## Notes
|
|
128
|
+
|
|
129
|
+
- **Only works in AR mode** — set `world.xr.mode` to `"ar"` and configure
|
|
130
|
+
`world.xr.features.depthSensing` in `iwsdk.config.json`.
|
|
131
|
+
- **`DepthOccludable` may be incompatible with custom shaders** that override `diffuse` or `fog_vertex` includes, since the occlusion code is injected at those shader hook points.
|
|
132
|
+
- **Non-occludable objects** are simply entities without `DepthOccludable` — they render normally on top of everything.
|
|
133
|
+
- **`DepthOccludable` is built into IWSDK** and is already registered — it appears
|
|
134
|
+
in `scene_get_capabilities` under `registeredComponents`. Do not call
|
|
135
|
+
`world.registerComponent()` for it; only the system needs registering.
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iwsdk-grab
|
|
3
|
+
description: Grab an object in the WebXR scene using emulated controllers. Use when the user wants to pick up, move, or test grabbing an object. Supports OneHandGrabbable and TwoHandsGrabbable components which use proximity-based grip (squeeze button), not trigger.
|
|
4
|
+
argument-hint: <object-name> [destination]
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Grab Object
|
|
8
|
+
|
|
9
|
+
Grab an object in the XR scene using the IWER emulated controllers. The workflow has a **required core** (steps 1-5) that must always execute, and **optional extensions** (steps 6-9) that depend on the user's intent.
|
|
10
|
+
|
|
11
|
+
User request is in `$ARGUMENTS`.
|
|
12
|
+
|
|
13
|
+
## Required Core
|
|
14
|
+
|
|
15
|
+
These steps always execute in order. A grab cannot succeed without them.
|
|
16
|
+
|
|
17
|
+
### Step 1: Enter XR
|
|
18
|
+
|
|
19
|
+
Check session status. If not in an active XR session, accept and enter.
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
xr_get_session_status → if not sessionActive → xr_accept_session
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
### Step 2: Locate the target
|
|
26
|
+
|
|
27
|
+
Find the live entity by name or grabbable component, then query its component data.
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
ecs_find_entities({namePattern: target}) → ecs_query_entity(entityIndex)
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
If the object is not found, search `OneHandGrabbable` and `TwoHandsGrabbable`
|
|
34
|
+
entities, report the available matches, and stop.
|
|
35
|
+
|
|
36
|
+
### Step 3: Get its transform
|
|
37
|
+
|
|
38
|
+
Read the object's Transform position from the queried entity.
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
ecs_query_entity(entityIndex) → Transform.position
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
### Step 4: Animate controller to target
|
|
45
|
+
|
|
46
|
+
Animate the controller to the object's position. Default to `"controller-right"` unless the user specified left.
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
xr_animate_to({
|
|
50
|
+
device: "controller-right",
|
|
51
|
+
position: { x, y, z },
|
|
52
|
+
duration: 0.5,
|
|
53
|
+
})
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Step 5: Engage grip
|
|
57
|
+
|
|
58
|
+
OneHandGrabbable and TwoHandsGrabbable are proximity-based and use the **squeeze/grip button (index 1)**, not the trigger.
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
xr_set_gamepad_state({
|
|
62
|
+
device: "controller-right",
|
|
63
|
+
buttons: [{ index: 1, value: 1 }],
|
|
64
|
+
})
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The object is now grabbed. If the user only asked to grab (not move), stop here.
|
|
68
|
+
|
|
69
|
+
## Optional Extensions
|
|
70
|
+
|
|
71
|
+
Apply these based on the user's request.
|
|
72
|
+
|
|
73
|
+
### Step 6: Move to destination
|
|
74
|
+
|
|
75
|
+
If the user specified a destination position, animate the controller there. If no position was given but the user asked to "move" the object, animate it to in front of the headset.
|
|
76
|
+
|
|
77
|
+
To find "in front of headset": `xr_get_transform({ "device": "headset" })` → place at `(head.x, head.y - 0.2, head.z - 0.5)` adjusted for head orientation.
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
xr_animate_to({
|
|
81
|
+
device: "controller-right",
|
|
82
|
+
position: { x, y, z },
|
|
83
|
+
duration: 0.5,
|
|
84
|
+
})
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
### Step 7: Release grip
|
|
88
|
+
|
|
89
|
+
Release the squeeze button to drop the object.
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
xr_set_gamepad_state({
|
|
93
|
+
device: "controller-right",
|
|
94
|
+
buttons: [{ index: 1, value: 0 }],
|
|
95
|
+
})
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Step 8: Return controller
|
|
99
|
+
|
|
100
|
+
Animate the controller back to its resting position so it's not overlapping the dropped object.
|
|
101
|
+
|
|
102
|
+
```
|
|
103
|
+
xr_animate_to({
|
|
104
|
+
device: "controller-right",
|
|
105
|
+
position: { x: 0.2, y: 1.4, z: -0.3 },
|
|
106
|
+
duration: 0.5,
|
|
107
|
+
})
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Default resting positions: right `(0.2, 1.4, -0.3)`, left `(-0.2, 1.4, -0.3)`.
|
|
111
|
+
|
|
112
|
+
### Step 9: Verify
|
|
113
|
+
|
|
114
|
+
Take a screenshot to confirm the result.
|
|
115
|
+
|
|
116
|
+
```
|
|
117
|
+
browser_screenshot
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The screenshot command is runtime-only. If the editor is visible, the managed
|
|
121
|
+
workspace switches to runtime before capture.
|
|
122
|
+
|
|
123
|
+
## Notes
|
|
124
|
+
|
|
125
|
+
- **Never use `xr_set_device_state` to move controllers** — it teleports instead of animating, which can break grab state.
|
|
126
|
+
- **Never use `xr_select` or trigger (button index 0) for grabs** — OneHandGrabbable/TwoHandsGrabbable respond to squeeze (button index 1).
|
|
127
|
+
- **DistanceGrabbable is different** — it uses ray-based selection, not proximity. This skill does not cover DistanceGrabbable.
|
|
128
|
+
- If the object lacks a name in the hierarchy, suggest adding `mesh.name = "MyObject"` in code before `createTransformEntity`.
|
|
129
|
+
|