@dcl-regenesislabs/opendcl 0.1.4-22555336781.commit-12e2bc1 → 0.1.4-22809399016.commit-8d4acb4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -3
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/extensions/dcl-screenshot.ts +482 -0
- package/extensions/permissions/index.ts +56 -25
- package/extensions/permissions/utils.ts +15 -1
- package/package.json +4 -3
- package/prompts/system.md +23 -0
- package/skills/visual-feedback/SKILL.md +137 -0
package/README.md
CHANGED
|
@@ -28,11 +28,11 @@ The result: **more creators building more scenes, faster.**
|
|
|
28
28
|
- **Branded header** — on startup, displays a block-character "Decentraland" ASCII art banner with version and working directory. Falls back to a compact text header on narrow terminals
|
|
29
29
|
- **Multi-provider LLM support** — works with Claude, OpenAI, Google, Ollama (free/local), OpenRouter, and more
|
|
30
30
|
- **Scene-aware** — automatically detects your project's `scene.json`, SDK version, and entry points
|
|
31
|
-
- **
|
|
31
|
+
- **19 built-in skills** — scaffolding, 3D models, interactivity, UI, animations, multiplayer, authoritative server, audio/video, deployment (Genesis City & Worlds), optimization, camera control, lighting, player/avatar, NFT/blockchain, advanced rendering, advanced input, scene runtime, visual feedback
|
|
32
32
|
- **Integrated commands** — `/init` to scaffold, `/preview` to launch the dev server, `/tasks` to manage running processes, `/review` to audit code
|
|
33
33
|
- **TypeScript validation** — catches type errors immediately after writing code
|
|
34
34
|
- **Free asset catalogs** — 2,700+ Creator Hub 3D models, 900+ CC0-licensed models, and 50 audio files the agent proactively suggests when building scenes
|
|
35
|
-
- **Permission gate** — prompts for confirmation before destructive bash commands
|
|
35
|
+
- **Permission gate** — prompts for confirmation before destructive bash commands, writes to sensitive files, or any file access outside the working directory
|
|
36
36
|
- **Compact tool output** — write shows path + size instead of file content, read shows a 5-line preview instead of 10
|
|
37
37
|
- **Session persistence** — pick up where you left off across sessions
|
|
38
38
|
|
|
@@ -102,6 +102,8 @@ This uses the open [skills](https://github.com/vercel-labs/skills) CLI to copy S
|
|
|
102
102
|
| `/review` | Review scene code for quality, performance, and SDK7 best practices |
|
|
103
103
|
| `/explain <concept>` | Explain a Decentraland SDK7 concept (e.g., `/explain tweens`) |
|
|
104
104
|
|
|
105
|
+
The agent also has a `screenshot` tool it can call automatically to see the running preview. On first use it asks for your permission to open a headless browser. The browser stays open for the entire session — no repeated logins.
|
|
106
|
+
|
|
105
107
|
## Skills
|
|
106
108
|
|
|
107
109
|
OpenDCL loads domain-specific skills on demand based on what you're asking:
|
|
@@ -126,6 +128,7 @@ OpenDCL loads domain-specific skills on demand based on what you're asking:
|
|
|
126
128
|
| `advanced-rendering` | Billboards, 3D text, materials, transparency |
|
|
127
129
|
| `advanced-input` | Cursor state, movement restriction, WASD patterns |
|
|
128
130
|
| `scene-runtime` | Async tasks, fetch, timers, realm info, restricted actions, testing |
|
|
131
|
+
| `visual-feedback` | Use the screenshot tool to see your scene, verify changes, iterate visually |
|
|
129
132
|
|
|
130
133
|
## How It Works
|
|
131
134
|
|
|
@@ -155,10 +158,11 @@ opendcl/
|
|
|
155
158
|
│ ├── dcl-status.ts # Thinking/streaming status (elapsed time + tokens)
|
|
156
159
|
│ ├── dcl-update-check.ts # Checks npm for newer OpenDCL versions
|
|
157
160
|
│ ├── dcl-validate.ts # Post-write TypeScript validation
|
|
161
|
+
│ ├── dcl-screenshot.ts # screenshot tool (headless Chrome, persistent browser)
|
|
158
162
|
│ ├── dcl-tasks.ts # /tasks command (process manager)
|
|
159
163
|
│ ├── process-registry.ts # Shared background process registry
|
|
160
164
|
│ └── permissions/ # Permission gate for dangerous operations
|
|
161
|
-
├── skills/ #
|
|
165
|
+
├── skills/ # 19 SKILL.md files (domain expertise)
|
|
162
166
|
├── prompts/ # System prompt + command templates
|
|
163
167
|
├── context/ # SDK7 reference docs + asset catalog
|
|
164
168
|
└── tests/ # Vitest test suites
|
package/dist/index.js
CHANGED
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAEA;;;;GAIG;AAEH,OAAO,EAAE,IAAI,EAAE,eAAe,EAAE,MAAM,+BAA+B,CAAC;AACtE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,EAAE,wBAAwB,EAAE,MAAM,6BAA6B,CAAC;AACvE,OAAO,EAAE,UAAU,EAAE,SAAS,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAC7E,OAAO,EAAE,OAAO,EAAE,MAAM,SAAS,CAAC;AAClC,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AAE1C,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAClD,MAAM,SAAS,GAAG,OAAO,CAAC,UAAU,CAAC,CAAC;AACtC,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;AAEzC,uFAAuF;AACvF,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,EAAE,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC;AACtD,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,EAAE,CAAC;IACrC,OAAO,CAAC,GAAG,CAAC,mBAAmB,GAAG,QAAQ,CAAC;AAC7C,CAAC;AAED,wEAAwE;AACxE,MAAM,YAAY,GAAG,IAAI,CAAC,QAAQ,EAAE,eAAe,CAAC,CAAC;AACrD,IAAI,CAAC,UAAU,CAAC,YAAY,CAAC,EAAE,CAAC;IAC9B,SAAS,CAAC,QAAQ,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACzC,aAAa,CAAC,YAAY,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,iBAAiB,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC;AAC3F,CAAC;AAED,yCAAyC;AACzC,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AAEnC,wFAAwF;AACxF,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,iBAAiB,CAAC,EAAE,CAAC;IACtC,MAAM,GAAG,GAAG,YAAY,CAAC,IAAI,CAAC,UAAU,EAAE,mBAAmB,CAAC,EAAE,OAAO,CAAC,CAAC;IACzE,MAAM,UAAU,GAAG,IAAI,CAAC,UAAU,EAAE,SAAS,CAAC,CAAC;IAC/C,MAAM,YAAY,GAAG,GAAG;SACrB,OAAO,CAAC,uBAAuB,EAAE,EAAE,CAAC;SACpC,OAAO,CAAC,wBAAwB,EAAE,CAAC,CAAC,EAAE,QAAQ,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,CAAC;SAC9E,IAAI,EAAE,CAAC;IACV,IAAI,CAAC,IAAI,CAAC,iBAAiB,EAAE,YAAY,CAAC,CAAC;AAC7C,CAAC;AAED,sBAAsB;AACtB,MAAM,MAAM,GAAG,IAAI,CAAC,UAAU,EAAE,YAAY,CAAC,CAAC;AAC9C,MAAM,UAAU,GAAG;IACjB,gBAAgB;IAChB,gBAAgB;IAChB,aAAa;IACb,eAAe;IACf,cAAc;IACd,qBAAqB;IACrB,iBAAiB;IACjB,eAAe;IACf,qBAAqB;IACrB,eAAe;IACf,cAAc;IACd,mBAAmB;CACpB,CAAC;AACF,KAAK,MAAM,GAAG,IAAI,UAAU,EAAE,CAAC;IAC7B,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,CAAC;AACrC,CAAC;AACD,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,oBAAoB,CAAC,CAAC,CAAC;AACpD,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,sBAAsB,CAAC,CAAC,CAAC;AAEtD,6BAA6B;AAC7B,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,CAAC,CAAC;AAEjD,yFAAyF;AACzF,IAAI,CAAC,IAAI,CAAC,mBAAmB,EAAE,IAAI,CAAC,UAAU,EAAE,mBAAmB,CAAC,CAAC,CAAC;AACtE,IAAI,CAAC,IAAI,CAAC,mBAAmB,EAAE,IAAI,CAAC,UAAU,EAAE,oBAAoB,CAAC,CAAC,CAAC;AAEvE,gFAAgF;AAChF,mEAAmE;AACnE,IAAI,CAAC,KAAK,EAAE,EAAE,CAAC;IACb,OAAO,CAAC,GAAG,CAAC,qBAAqB,GAAG,GAAG,CAAC;AAC1C,CAAC;AAED,6EAA6E;AAC7E,sEAAsE;AACtE,MAAM,YAAY,GAAG,eAAe,CAAC,SAAS,CAAC,WAAW,CAAC;AAC3D,eAAe,CAAC,SAAS,CAAC,WAAW,GAAG,UAAU,GAAW;IAC3D,IAAI,GAAG,CAAC,UAAU,CAAC,qBAAqB,CAAC;QAAE,OAAO;IAClD,YAAY,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;AAC/B,CAAC,CAAC;AAEF,+EAA+E;AAC/E,qDAAqD;AACpD,eAAe,CAAC,SAAiB,CAAC,sBAAsB,GAAG;IAC1D,OAAO,SAAS,CAAC;AACnB,CAAC,CAAC;AAEF,0EAA0E;AAC1E,iFAAiF;AACjF,gEAAgE;AAChE,MAAM,cAAc,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,IAAI,CAAC,UAAU,EAAE,cAAc,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;AAClG,eAAe,CAAC,SAAiB,CAAC,sBAAsB,GAAG;IACzD,IAAY,CAAC,UAAU,CACtB,YAAY,cAAc,gEAAgE,cAAc,EAAE,CAC3G,CAAC;AACJ,CAAC,CAAC;AAEF,yFAAyF;AACzF,uFAAuF;AACvF,8FAA8F;AAC9F,2EAA2E;AAC3E,MAAM,WAAW,GAAI,eAAe,CAAC,SAAiB,CAAC,2BAA2B,CAAC;AAClF,eAAe,CAAC,SAAiB,CAAC,2BAA2B,GAAG,UAAU,QAAgB;IACzF,MAAM,QAAQ,GAAG,WAAW,CAAC,IAAI,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IAClD,IAAI,QAAQ;QAAE,OAAO,QAAQ,CAAC;IAC9B,OAAO,wBAAwB,CAAC,QAAQ,CAAC,CAAC;AAC5C,CAAC,CAAC;AAEF,IAAI,CAAC,IAAI,CAAC,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE;IACvB,OAAO,CAAC,KAAK,CAAC,sBAAsB,EAAE,GAAG,CAAC,CAAC;IAC3C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAEA;;;;GAIG;AAEH,OAAO,EAAE,IAAI,EAAE,eAAe,EAAE,MAAM,+BAA+B,CAAC;AACtE,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,EAAE,wBAAwB,EAAE,MAAM,6BAA6B,CAAC;AACvE,OAAO,EAAE,UAAU,EAAE,SAAS,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AAC7E,OAAO,EAAE,OAAO,EAAE,MAAM,SAAS,CAAC;AAClC,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AAE1C,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAClD,MAAM,SAAS,GAAG,OAAO,CAAC,UAAU,CAAC,CAAC;AACtC,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;AAEzC,uFAAuF;AACvF,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,EAAE,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC;AACtD,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,EAAE,CAAC;IACrC,OAAO,CAAC,GAAG,CAAC,mBAAmB,GAAG,QAAQ,CAAC;AAC7C,CAAC;AAED,wEAAwE;AACxE,MAAM,YAAY,GAAG,IAAI,CAAC,QAAQ,EAAE,eAAe,CAAC,CAAC;AACrD,IAAI,CAAC,UAAU,CAAC,YAAY,CAAC,EAAE,CAAC;IAC9B,SAAS,CAAC,QAAQ,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACzC,aAAa,CAAC,YAAY,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,iBAAiB,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC;AAC3F,CAAC;AAED,yCAAyC;AACzC,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AAEnC,wFAAwF;AACxF,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,iBAAiB,CAAC,EAAE,CAAC;IACtC,MAAM,GAAG,GAAG,YAAY,CAAC,IAAI,CAAC,UAAU,EAAE,mBAAmB,CAAC,EAAE,OAAO,CAAC,CAAC;IACzE,MAAM,UAAU,GAAG,IAAI,CAAC,UAAU,EAAE,SAAS,CAAC,CAAC;IAC/C,MAAM,YAAY,GAAG,GAAG;SACrB,OAAO,CAAC,uBAAuB,EAAE,EAAE,CAAC;SACpC,OAAO,CAAC,wBAAwB,EAAE,CAAC,CAAC,EAAE,QAAQ,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,CAAC;SAC9E,IAAI,EAAE,CAAC;IACV,IAAI,CAAC,IAAI,CAAC,iBAAiB,EAAE,YAAY,CAAC,CAAC;AAC7C,CAAC;AAED,sBAAsB;AACtB,MAAM,MAAM,GAAG,IAAI,CAAC,UAAU,EAAE,YAAY,CAAC,CAAC;AAC9C,MAAM,UAAU,GAAG;IACjB,gBAAgB;IAChB,gBAAgB;IAChB,aAAa;IACb,eAAe;IACf,cAAc;IACd,qBAAqB;IACrB,iBAAiB;IACjB,eAAe;IACf,qBAAqB;IACrB,eAAe;IACf,cAAc;IACd,mBAAmB;IACnB,mBAAmB;CACpB,CAAC;AACF,KAAK,MAAM,GAAG,IAAI,UAAU,EAAE,CAAC;IAC7B,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,CAAC;AACrC,CAAC;AACD,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,oBAAoB,CAAC,CAAC,CAAC;AACpD,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,sBAAsB,CAAC,CAAC,CAAC;AAEtD,6BAA6B;AAC7B,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,CAAC,CAAC;AAEjD,yFAAyF;AACzF,IAAI,CAAC,IAAI,CAAC,mBAAmB,EAAE,IAAI,CAAC,UAAU,EAAE,mBAAmB,CAAC,CAAC,CAAC;AACtE,IAAI,CAAC,IAAI,CAAC,mBAAmB,EAAE,IAAI,CAAC,UAAU,EAAE,oBAAoB,CAAC,CAAC,CAAC;AAEvE,gFAAgF;AAChF,mEAAmE;AACnE,IAAI,CAAC,KAAK,EAAE,EAAE,CAAC;IACb,OAAO,CAAC,GAAG,CAAC,qBAAqB,GAAG,GAAG,CAAC;AAC1C,CAAC;AAED,6EAA6E;AAC7E,sEAAsE;AACtE,MAAM,YAAY,GAAG,eAAe,CAAC,SAAS,CAAC,WAAW,CAAC;AAC3D,eAAe,CAAC,SAAS,CAAC,WAAW,GAAG,UAAU,GAAW;IAC3D,IAAI,GAAG,CAAC,UAAU,CAAC,qBAAqB,CAAC;QAAE,OAAO;IAClD,YAAY,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;AAC/B,CAAC,CAAC;AAEF,+EAA+E;AAC/E,qDAAqD;AACpD,eAAe,CAAC,SAAiB,CAAC,sBAAsB,GAAG;IAC1D,OAAO,SAAS,CAAC;AACnB,CAAC,CAAC;AAEF,0EAA0E;AAC1E,iFAAiF;AACjF,gEAAgE;AAChE,MAAM,cAAc,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,IAAI,CAAC,UAAU,EAAE,cAAc,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC;AAClG,eAAe,CAAC,SAAiB,CAAC,sBAAsB,GAAG;IACzD,IAAY,CAAC,UAAU,CACtB,YAAY,cAAc,gEAAgE,cAAc,EAAE,CAC3G,CAAC;AACJ,CAAC,CAAC;AAEF,yFAAyF;AACzF,uFAAuF;AACvF,8FAA8F;AAC9F,2EAA2E;AAC3E,MAAM,WAAW,GAAI,eAAe,CAAC,SAAiB,CAAC,2BAA2B,CAAC;AAClF,eAAe,CAAC,SAAiB,CAAC,2BAA2B,GAAG,UAAU,QAAgB;IACzF,MAAM,QAAQ,GAAG,WAAW,CAAC,IAAI,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC;IAClD,IAAI,QAAQ;QAAE,OAAO,QAAQ,CAAC;IAC9B,OAAO,wBAAwB,CAAC,QAAQ,CAAC,CAAC;AAC5C,CAAC,CAAC;AAEF,IAAI,CAAC,IAAI,CAAC,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE;IACvB,OAAO,CAAC,KAAK,CAAC,sBAAsB,EAAE,GAAG,CAAC,CAAC;IAC3C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}
|
|
@@ -0,0 +1,482 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DCL Screenshot Extension
|
|
3
|
+
*
|
|
4
|
+
* Registers the `screenshot` tool (LLM-callable) that captures a screenshot
|
|
5
|
+
* of the running Decentraland preview. Uses playwright-core with system Chrome
|
|
6
|
+
* for headless browser automation.
|
|
7
|
+
*
|
|
8
|
+
* Features:
|
|
9
|
+
* - Persistent browser window — launch once, reuse forever (no repeated logins)
|
|
10
|
+
* - User consent prompt before first use
|
|
11
|
+
* - Auto-detects preview URL from the process registry (/preview)
|
|
12
|
+
* - Auto-dismisses the "Explore as Guest" welcome screen on first load
|
|
13
|
+
* - Supports input actions (click, key, move, look, drag, wait) before capture
|
|
14
|
+
* - Returns screenshot as base64 image directly in tool result
|
|
15
|
+
*
|
|
16
|
+
* Chrome flags for Bevy-Web renderer (WebGPU via Metal on macOS):
|
|
17
|
+
* --enable-gpu --use-gl=angle --use-angle=metal --ignore-gpu-blocklist
|
|
18
|
+
*/
|
|
19
|
+
|
|
20
|
+
import type { ExtensionFactory } from "@mariozechner/pi-coding-agent";
|
|
21
|
+
import { Type } from "@sinclair/typebox";
|
|
22
|
+
import { processes } from "./process-registry.js";
|
|
23
|
+
|
|
24
|
+
// ── Lazy-loaded playwright types ───────────────────────────────────────────
|
|
25
|
+
|
|
26
|
+
type Browser = import("playwright-core").Browser;
|
|
27
|
+
type Page = import("playwright-core").Page;
|
|
28
|
+
type ConsoleMessage = import("playwright-core").ConsoleMessage;
|
|
29
|
+
|
|
30
|
+
// ── Action types ───────────────────────────────────────────────────────────
|
|
31
|
+
|
|
32
|
+
interface Action {
|
|
33
|
+
type:
|
|
34
|
+
| "click"
|
|
35
|
+
| "key"
|
|
36
|
+
| "mouse"
|
|
37
|
+
| "wait"
|
|
38
|
+
| "lookLeft"
|
|
39
|
+
| "lookRight"
|
|
40
|
+
| "lookUp"
|
|
41
|
+
| "lookDown"
|
|
42
|
+
| "moveForward"
|
|
43
|
+
| "moveBack"
|
|
44
|
+
| "moveLeft"
|
|
45
|
+
| "moveRight";
|
|
46
|
+
x?: number;
|
|
47
|
+
y?: number;
|
|
48
|
+
key?: string;
|
|
49
|
+
holdMs?: number;
|
|
50
|
+
dx?: number;
|
|
51
|
+
dy?: number;
|
|
52
|
+
ms?: number;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/** Default pixels to drag for look actions. */
|
|
56
|
+
const LOOK_DRAG_PX = 200;
|
|
57
|
+
/** Default duration (ms) to hold movement keys. */
|
|
58
|
+
const MOVE_HOLD_MS = 500;
|
|
59
|
+
/** Process registry key for the screenshot browser. */
|
|
60
|
+
const PROCESS_NAME = "screenshot-browser";
|
|
61
|
+
/** Relative position of the "Explore as Guest" button on the Bevy-Web renderer welcome canvas.
|
|
62
|
+
* Calibrated for the Bevy-Web renderer layout — may need updating if the welcome screen changes. */
|
|
63
|
+
const WELCOME_BUTTON_POS = { x: 0.237, y: 0.583 };
|
|
64
|
+
|
|
65
|
+
// ── Chrome flags for Bevy-Web (WebGPU via Metal) ──────────────────────────
|
|
66
|
+
|
|
67
|
+
const CHROME_FLAGS = [
|
|
68
|
+
// The local dev server (localhost) serves assets and WASM from different ports/origins.
|
|
69
|
+
// Without these flags, CORS blocks the renderer from loading scene content.
|
|
70
|
+
"--disable-web-security",
|
|
71
|
+
"--allow-insecure-localhost",
|
|
72
|
+
"--disable-features=PrivateNetworkAccessPermissionPrompt",
|
|
73
|
+
"--enable-gpu",
|
|
74
|
+
"--enable-webgl",
|
|
75
|
+
"--ignore-gpu-blocklist",
|
|
76
|
+
"--enable-features=Vulkan",
|
|
77
|
+
"--use-gl=angle",
|
|
78
|
+
"--use-angle=metal",
|
|
79
|
+
];
|
|
80
|
+
|
|
81
|
+
// ── Viewport helpers ──────────────────────────────────────────────────────
|
|
82
|
+
|
|
83
|
+
function getViewportCenter(page: Page): { x: number; y: number } {
|
|
84
|
+
const size = page.viewportSize();
|
|
85
|
+
if (size) return { x: Math.round(size.width / 2), y: Math.round(size.height / 2) };
|
|
86
|
+
return { x: 640, y: 360 };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
// ── Execute input actions ─────────────────────────────────────────────────
|
|
90
|
+
|
|
91
|
+
async function executeActions(page: Page, actions: Action[]): Promise<void> {
|
|
92
|
+
for (const action of actions) {
|
|
93
|
+
switch (action.type) {
|
|
94
|
+
case "click":
|
|
95
|
+
await page.mouse.click(action.x ?? 0, action.y ?? 0);
|
|
96
|
+
break;
|
|
97
|
+
|
|
98
|
+
case "key":
|
|
99
|
+
if (!action.key) throw new Error('key action requires a "key" parameter');
|
|
100
|
+
if (action.holdMs) {
|
|
101
|
+
await page.keyboard.down(action.key);
|
|
102
|
+
await page.waitForTimeout(action.holdMs);
|
|
103
|
+
await page.keyboard.up(action.key);
|
|
104
|
+
} else {
|
|
105
|
+
await page.keyboard.press(action.key);
|
|
106
|
+
}
|
|
107
|
+
break;
|
|
108
|
+
|
|
109
|
+
case "mouse": {
|
|
110
|
+
const center = getViewportCenter(page);
|
|
111
|
+
await page.mouse.move(center.x, center.y);
|
|
112
|
+
await page.mouse.down();
|
|
113
|
+
await page.mouse.move(
|
|
114
|
+
center.x + (action.dx ?? 0),
|
|
115
|
+
center.y + (action.dy ?? 0),
|
|
116
|
+
{ steps: 10 },
|
|
117
|
+
);
|
|
118
|
+
await page.mouse.up();
|
|
119
|
+
break;
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
case "wait":
|
|
123
|
+
await page.waitForTimeout(action.ms ?? 1000);
|
|
124
|
+
break;
|
|
125
|
+
|
|
126
|
+
// Camera look (mouse drags from center)
|
|
127
|
+
case "lookLeft":
|
|
128
|
+
case "lookRight":
|
|
129
|
+
case "lookUp":
|
|
130
|
+
case "lookDown": {
|
|
131
|
+
const c = getViewportCenter(page);
|
|
132
|
+
const px = action.dx ? Math.abs(action.dx) : LOOK_DRAG_PX;
|
|
133
|
+
const targets: Record<string, { x: number; y: number }> = {
|
|
134
|
+
lookLeft: { x: c.x - px, y: c.y },
|
|
135
|
+
lookRight: { x: c.x + px, y: c.y },
|
|
136
|
+
lookUp: { x: c.x, y: c.y - px },
|
|
137
|
+
lookDown: { x: c.x, y: c.y + px },
|
|
138
|
+
};
|
|
139
|
+
const target = targets[action.type];
|
|
140
|
+
await page.mouse.move(c.x, c.y);
|
|
141
|
+
await page.mouse.down();
|
|
142
|
+
await page.mouse.move(target.x, target.y, { steps: 10 });
|
|
143
|
+
await page.mouse.up();
|
|
144
|
+
break;
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
// WASD movement (key holds)
|
|
148
|
+
case "moveForward":
|
|
149
|
+
case "moveBack":
|
|
150
|
+
case "moveLeft":
|
|
151
|
+
case "moveRight": {
|
|
152
|
+
const keyMap: Record<string, string> = {
|
|
153
|
+
moveForward: "w",
|
|
154
|
+
moveBack: "s",
|
|
155
|
+
moveLeft: "a",
|
|
156
|
+
moveRight: "d",
|
|
157
|
+
};
|
|
158
|
+
const ms = action.holdMs ?? MOVE_HOLD_MS;
|
|
159
|
+
await page.keyboard.down(keyMap[action.type]);
|
|
160
|
+
await page.waitForTimeout(ms);
|
|
161
|
+
await page.keyboard.up(keyMap[action.type]);
|
|
162
|
+
break;
|
|
163
|
+
}
|
|
164
|
+
}
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
|
|
168
|
+
// ── Extension ─────────────────────────────────────────────────────────────
|
|
169
|
+
|
|
170
|
+
const extension: ExtensionFactory = (pi) => {
|
|
171
|
+
let browser: Browser | null = null;
|
|
172
|
+
let persistentPage: Page | null = null;
|
|
173
|
+
let currentPreviewUrl: string | null = null;
|
|
174
|
+
let sceneEntered = false;
|
|
175
|
+
let userConsented: boolean | null = null;
|
|
176
|
+
|
|
177
|
+
function resetPageState(): void {
|
|
178
|
+
persistentPage = null;
|
|
179
|
+
currentPreviewUrl = null;
|
|
180
|
+
sceneEntered = false;
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
async function launchBrowser(): Promise<Browser> {
|
|
184
|
+
if (browser?.isConnected()) return browser;
|
|
185
|
+
|
|
186
|
+
const pw = await import("playwright-core");
|
|
187
|
+
browser = await pw.chromium.launch({
|
|
188
|
+
channel: "chrome",
|
|
189
|
+
headless: true,
|
|
190
|
+
args: CHROME_FLAGS,
|
|
191
|
+
});
|
|
192
|
+
|
|
193
|
+
// Register in process registry so /tasks can manage it
|
|
194
|
+
processes.set(PROCESS_NAME, {
|
|
195
|
+
name: "Screenshot browser",
|
|
196
|
+
info: "headless Chrome for preview screenshots",
|
|
197
|
+
kill: () => closeBrowser(),
|
|
198
|
+
});
|
|
199
|
+
|
|
200
|
+
return browser;
|
|
201
|
+
}
|
|
202
|
+
|
|
203
|
+
async function closeBrowser(): Promise<void> {
|
|
204
|
+
resetPageState();
|
|
205
|
+
if (browser) {
|
|
206
|
+
try { await browser.close(); } catch { /* already closed */ }
|
|
207
|
+
browser = null;
|
|
208
|
+
}
|
|
209
|
+
processes.delete(PROCESS_NAME);
|
|
210
|
+
}
|
|
211
|
+
|
|
212
|
+
/**
|
|
213
|
+
* Enter the Decentraland scene — dismiss the "Explore as Guest" welcome screen.
|
|
214
|
+
* Uses fixed coordinate click (24% x, 58% y) which works with Bevy-Web renderer.
|
|
215
|
+
*/
|
|
216
|
+
async function enterScene(page: Page): Promise<void> {
|
|
217
|
+
// Set up asset listener before entering
|
|
218
|
+
let assetsReady = false;
|
|
219
|
+
const onConsole = (msg: ConsoleMessage) => {
|
|
220
|
+
if (msg.text().includes("pendingAssets: 0")) assetsReady = true;
|
|
221
|
+
};
|
|
222
|
+
page.on("console", onConsole);
|
|
223
|
+
|
|
224
|
+
try {
|
|
225
|
+
// Wait for canvas to appear
|
|
226
|
+
const canvas = page.locator("canvas").first();
|
|
227
|
+
await canvas.waitFor({ timeout: 30_000 });
|
|
228
|
+
await page.waitForTimeout(4_000);
|
|
229
|
+
|
|
230
|
+
const box = await canvas.boundingBox();
|
|
231
|
+
if (!box) throw new Error("No canvas found after page load");
|
|
232
|
+
|
|
233
|
+
// Click "EXPLORE AS GUEST" button
|
|
234
|
+
for (let attempt = 0; attempt < 3; attempt++) {
|
|
235
|
+
await page.mouse.click(
|
|
236
|
+
box.x + box.width * WELCOME_BUTTON_POS.x,
|
|
237
|
+
box.y + box.height * WELCOME_BUTTON_POS.y,
|
|
238
|
+
);
|
|
239
|
+
await page.waitForTimeout(400);
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
// Wait for scene assets to load
|
|
243
|
+
for (let i = 0; i < 20; i++) {
|
|
244
|
+
if (assetsReady) break;
|
|
245
|
+
await page.waitForTimeout(1000);
|
|
246
|
+
}
|
|
247
|
+
if (!assetsReady) {
|
|
248
|
+
console.warn("[screenshot] Asset loading timed out (20s) — capturing anyway");
|
|
249
|
+
}
|
|
250
|
+
await page.waitForTimeout(1_000);
|
|
251
|
+
|
|
252
|
+
// Click canvas center for pointer lock / focus
|
|
253
|
+
await page.mouse.click(box.x + box.width / 2, box.y + box.height / 2);
|
|
254
|
+
await page.waitForTimeout(300);
|
|
255
|
+
} finally {
|
|
256
|
+
page.removeListener("console", onConsole);
|
|
257
|
+
}
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
/**
|
|
261
|
+
* Get the persistent page, navigating + entering scene only when needed.
|
|
262
|
+
* Subsequent calls with the same preview URL skip all setup.
|
|
263
|
+
*/
|
|
264
|
+
async function getPage(
|
|
265
|
+
previewUrl: string,
|
|
266
|
+
onUpdate?: (msg: string) => void,
|
|
267
|
+
): Promise<Page> {
|
|
268
|
+
const instance = await launchBrowser();
|
|
269
|
+
|
|
270
|
+
// Reuse existing page if still alive and URL matches
|
|
271
|
+
if (persistentPage && currentPreviewUrl === previewUrl) {
|
|
272
|
+
try {
|
|
273
|
+
await persistentPage.evaluate(() => true);
|
|
274
|
+
return persistentPage;
|
|
275
|
+
} catch {
|
|
276
|
+
// Page crashed — recreate
|
|
277
|
+
resetPageState();
|
|
278
|
+
}
|
|
279
|
+
}
|
|
280
|
+
|
|
281
|
+
// Close stale page if URL changed
|
|
282
|
+
if (persistentPage) {
|
|
283
|
+
try { await persistentPage.close(); } catch { /* already closed */ }
|
|
284
|
+
resetPageState();
|
|
285
|
+
}
|
|
286
|
+
|
|
287
|
+
onUpdate?.("Launching browser and navigating to preview...");
|
|
288
|
+
const page = await instance.newPage({ viewport: { width: 1280, height: 720 } });
|
|
289
|
+
await page.goto(previewUrl, { waitUntil: "domcontentloaded", timeout: 60_000 });
|
|
290
|
+
|
|
291
|
+
// Enter scene (welcome screen + asset loading).
|
|
292
|
+
// Set persistentPage only after success — if enterScene throws, the page
|
|
293
|
+
// won't be cached and the next call will start fresh instead of reusing
|
|
294
|
+
// a page that never entered the scene.
|
|
295
|
+
if (!sceneEntered) {
|
|
296
|
+
onUpdate?.("Entering scene (dismissing welcome screen, loading assets)...");
|
|
297
|
+
try {
|
|
298
|
+
await enterScene(page);
|
|
299
|
+
sceneEntered = true;
|
|
300
|
+
} catch (err) {
|
|
301
|
+
try { await page.close(); } catch { /* ignore */ }
|
|
302
|
+
throw err;
|
|
303
|
+
}
|
|
304
|
+
}
|
|
305
|
+
|
|
306
|
+
persistentPage = page;
|
|
307
|
+
currentPreviewUrl = previewUrl;
|
|
308
|
+
return page;
|
|
309
|
+
}
|
|
310
|
+
|
|
311
|
+
// ── Register the screenshot tool ──────────────────────────────────────
|
|
312
|
+
|
|
313
|
+
pi.registerTool({
|
|
314
|
+
name: "screenshot",
|
|
315
|
+
label: "Screenshot",
|
|
316
|
+
description: `Capture a screenshot of the running Decentraland preview. Start the preview first with /preview.
|
|
317
|
+
|
|
318
|
+
The browser stays open between calls — only the first screenshot navigates and enters the scene (~15s). Subsequent screenshots are instant.
|
|
319
|
+
|
|
320
|
+
## Actions (optional, performed before capture)
|
|
321
|
+
Low-level: click (x,y coords), key (press/hold), mouse (relative drag), wait
|
|
322
|
+
High-level helpers:
|
|
323
|
+
- lookLeft / lookRight / lookUp / lookDown — camera rotation (dx/dy pixels, default 200)
|
|
324
|
+
- moveForward / moveBack / moveLeft / moveRight — WASD movement (holdMs duration, default 500ms)
|
|
325
|
+
|
|
326
|
+
Movement speed is ~6m/s. Camera always faces north in headless mode.`,
|
|
327
|
+
promptGuidelines: [
|
|
328
|
+
"After writing scene code with the preview running, proactively use `screenshot` (with wait: 2000 for hot-reload) to verify your changes visually. Don't wait for the user to check.",
|
|
329
|
+
"If the screenshot tool fails (no Chrome, browser crash, user declined), continue working normally without vision. Tell the user what happened and suggest they check the preview manually.",
|
|
330
|
+
"Don't retry screenshot more than once if it fails — fall back to asking the user to verify visually.",
|
|
331
|
+
],
|
|
332
|
+
parameters: Type.Object({
|
|
333
|
+
wait: Type.Optional(
|
|
334
|
+
Type.Number({
|
|
335
|
+
description:
|
|
336
|
+
"Extra ms to wait before capturing (for hot-reload settle). Default: 1000. First call auto-waits for scene load.",
|
|
337
|
+
default: 1000,
|
|
338
|
+
}),
|
|
339
|
+
),
|
|
340
|
+
actions: Type.Optional(
|
|
341
|
+
Type.Array(
|
|
342
|
+
Type.Object({
|
|
343
|
+
type: Type.Union([
|
|
344
|
+
Type.Literal("click"),
|
|
345
|
+
Type.Literal("key"),
|
|
346
|
+
Type.Literal("mouse"),
|
|
347
|
+
Type.Literal("wait"),
|
|
348
|
+
Type.Literal("lookLeft"),
|
|
349
|
+
Type.Literal("lookRight"),
|
|
350
|
+
Type.Literal("lookUp"),
|
|
351
|
+
Type.Literal("lookDown"),
|
|
352
|
+
Type.Literal("moveForward"),
|
|
353
|
+
Type.Literal("moveBack"),
|
|
354
|
+
Type.Literal("moveLeft"),
|
|
355
|
+
Type.Literal("moveRight"),
|
|
356
|
+
]),
|
|
357
|
+
x: Type.Optional(Type.Number({ description: "X pixel coordinate for click" })),
|
|
358
|
+
y: Type.Optional(Type.Number({ description: "Y pixel coordinate for click" })),
|
|
359
|
+
key: Type.Optional(Type.String({ description: "Key to press (for key action)" })),
|
|
360
|
+
holdMs: Type.Optional(
|
|
361
|
+
Type.Number({ description: "Hold duration in ms (key or movement, default 500ms)" }),
|
|
362
|
+
),
|
|
363
|
+
dx: Type.Optional(
|
|
364
|
+
Type.Number({ description: "Relative X pixels (mouse drag or look, default 200)" }),
|
|
365
|
+
),
|
|
366
|
+
dy: Type.Optional(
|
|
367
|
+
Type.Number({ description: "Relative Y pixels (mouse drag or look, default 200)" }),
|
|
368
|
+
),
|
|
369
|
+
ms: Type.Optional(
|
|
370
|
+
Type.Number({ description: "Wait duration in ms (for wait action)" }),
|
|
371
|
+
),
|
|
372
|
+
}),
|
|
373
|
+
{ description: "Input actions to perform before taking the screenshot" },
|
|
374
|
+
),
|
|
375
|
+
),
|
|
376
|
+
}),
|
|
377
|
+
async execute(_id, params, _signal, _onUpdate, ctx) {
|
|
378
|
+
const { wait: waitMs = 1000, actions } = params as {
|
|
379
|
+
wait?: number;
|
|
380
|
+
actions?: Action[];
|
|
381
|
+
};
|
|
382
|
+
|
|
383
|
+
const onUpdate = (msg: string) => _onUpdate?.({
|
|
384
|
+
content: [{ type: "text" as const, text: msg }],
|
|
385
|
+
details: undefined,
|
|
386
|
+
});
|
|
387
|
+
|
|
388
|
+
// Check if preview is running
|
|
389
|
+
const preview = processes.get("preview");
|
|
390
|
+
if (!preview?.info) {
|
|
391
|
+
return {
|
|
392
|
+
content: [{
|
|
393
|
+
type: "text" as const,
|
|
394
|
+
text: preview
|
|
395
|
+
? "Preview server is still starting — URL not ready yet. Wait a moment and try again."
|
|
396
|
+
: "No preview server running. Start one first with /preview.",
|
|
397
|
+
}],
|
|
398
|
+
details: undefined,
|
|
399
|
+
};
|
|
400
|
+
}
|
|
401
|
+
|
|
402
|
+
// Ask user for consent on first use
|
|
403
|
+
if (userConsented === null) {
|
|
404
|
+
const agreed = await ctx.ui.confirm(
|
|
405
|
+
"Enable Screenshot Tool",
|
|
406
|
+
"OpenDCL wants to open a headless browser to capture screenshots of your scene preview. This lets the AI see what your scene looks like and iterate visually.\n\nAllow?",
|
|
407
|
+
);
|
|
408
|
+
userConsented = agreed;
|
|
409
|
+
if (!agreed) {
|
|
410
|
+
return {
|
|
411
|
+
content: [{
|
|
412
|
+
type: "text" as const,
|
|
413
|
+
text: "Screenshot tool declined by user. The AI cannot see the preview.",
|
|
414
|
+
}],
|
|
415
|
+
details: undefined,
|
|
416
|
+
};
|
|
417
|
+
}
|
|
418
|
+
}
|
|
419
|
+
|
|
420
|
+
if (!userConsented) {
|
|
421
|
+
return {
|
|
422
|
+
content: [{
|
|
423
|
+
type: "text" as const,
|
|
424
|
+
text: "Screenshot tool was previously declined. Restart the session to re-enable.",
|
|
425
|
+
}],
|
|
426
|
+
details: undefined,
|
|
427
|
+
};
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
const previewUrl = preview.info;
|
|
431
|
+
|
|
432
|
+
try {
|
|
433
|
+
const page = await getPage(previewUrl, onUpdate);
|
|
434
|
+
|
|
435
|
+
// Execute actions before screenshot
|
|
436
|
+
if (actions?.length) {
|
|
437
|
+
onUpdate("Performing actions...");
|
|
438
|
+
await executeActions(page, actions);
|
|
439
|
+
}
|
|
440
|
+
|
|
441
|
+
// Wait for hot-reload / render settle
|
|
442
|
+
if (waitMs > 0) {
|
|
443
|
+
await page.waitForTimeout(waitMs);
|
|
444
|
+
}
|
|
445
|
+
|
|
446
|
+
onUpdate("Capturing screenshot...");
|
|
447
|
+
const buffer = await page.screenshot({ type: "png", timeout: 30_000 });
|
|
448
|
+
const base64 = buffer.toString("base64");
|
|
449
|
+
|
|
450
|
+
return {
|
|
451
|
+
content: [
|
|
452
|
+
{
|
|
453
|
+
type: "text" as const,
|
|
454
|
+
text: `Screenshot captured (1280×720) from ${previewUrl}`,
|
|
455
|
+
},
|
|
456
|
+
{ type: "image" as const, data: base64, mimeType: "image/png" },
|
|
457
|
+
],
|
|
458
|
+
details: undefined,
|
|
459
|
+
};
|
|
460
|
+
} catch (err) {
|
|
461
|
+
// Reset on failure so next call retries fresh
|
|
462
|
+
resetPageState();
|
|
463
|
+
const errorMsg = err instanceof Error ? err.message : String(err);
|
|
464
|
+
return {
|
|
465
|
+
content: [{
|
|
466
|
+
type: "text" as const,
|
|
467
|
+
text: `Screenshot failed: ${errorMsg}\n\nContinue working without visual feedback. Ask the user to check the preview in their browser and describe what they see.`,
|
|
468
|
+
}],
|
|
469
|
+
details: undefined,
|
|
470
|
+
};
|
|
471
|
+
}
|
|
472
|
+
},
|
|
473
|
+
});
|
|
474
|
+
|
|
475
|
+
// ── Cleanup on shutdown ──────────────────────────────────────────────
|
|
476
|
+
|
|
477
|
+
pi.on("session_shutdown", async () => {
|
|
478
|
+
await closeBrowser();
|
|
479
|
+
});
|
|
480
|
+
};
|
|
481
|
+
|
|
482
|
+
export default extension;
|
|
@@ -7,25 +7,53 @@
|
|
|
7
7
|
*/
|
|
8
8
|
|
|
9
9
|
import type { ExtensionFactory } from "@mariozechner/pi-coding-agent";
|
|
10
|
-
import { classifyBashCommand, classifyFilePath } from "./utils.js";
|
|
10
|
+
import { classifyBashCommand, classifyFilePath, isOutsideCwd, OUTSIDE_CWD_REASON } from "./utils.js";
|
|
11
|
+
import { resolve } from "node:path";
|
|
11
12
|
|
|
12
|
-
|
|
13
|
+
type BlockResult = { block: true; reason: string };
|
|
14
|
+
|
|
15
|
+
function blockResult(reason: string, detail: string): BlockResult {
|
|
13
16
|
return {
|
|
14
17
|
block: true,
|
|
15
18
|
reason: `Blocked: ${reason}\n${detail}\nUse --no-permissions to allow in non-interactive mode.`,
|
|
16
19
|
};
|
|
17
20
|
}
|
|
18
21
|
|
|
19
|
-
function denyResult(reason: string):
|
|
22
|
+
function denyResult(reason: string): BlockResult {
|
|
20
23
|
return { block: true, reason: `User denied: ${reason}` };
|
|
21
24
|
}
|
|
22
25
|
|
|
23
|
-
const
|
|
24
|
-
const ALWAYS = "Always allow";
|
|
25
|
-
const DENY = "Deny";
|
|
26
|
+
const CHOICES = ["Allow", "Always allow", "Deny"] as const;
|
|
26
27
|
|
|
27
28
|
const extension: ExtensionFactory = (pi) => {
|
|
28
29
|
const sessionAllow = new Set<string>();
|
|
30
|
+
const allowedPaths = new Set<string>();
|
|
31
|
+
|
|
32
|
+
function isPathAllowed(resolvedPath: string): boolean {
|
|
33
|
+
for (const allowed of allowedPaths) {
|
|
34
|
+
if (resolvedPath === allowed || resolvedPath.startsWith(allowed + "/")) return true;
|
|
35
|
+
}
|
|
36
|
+
return false;
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
/**
|
|
40
|
+
* Prompts the user for confirmation and handles their choice.
|
|
41
|
+
* Returns a BlockResult to deny, or undefined to allow.
|
|
42
|
+
*/
|
|
43
|
+
async function promptOrBlock(
|
|
44
|
+
ctx: { hasUI: boolean; ui: { select: (title: string, options: string[]) => Promise<string | null> } },
|
|
45
|
+
reason: string,
|
|
46
|
+
detail: string,
|
|
47
|
+
onAlways: () => void,
|
|
48
|
+
): Promise<BlockResult | undefined> {
|
|
49
|
+
if (!ctx.hasUI) return blockResult(reason, detail);
|
|
50
|
+
|
|
51
|
+
const choice = await ctx.ui.select(`${reason}\n${detail}`, [...CHOICES]);
|
|
52
|
+
|
|
53
|
+
if (choice === "Always allow") { onAlways(); return; }
|
|
54
|
+
if (choice === "Allow") return;
|
|
55
|
+
return denyResult(reason);
|
|
56
|
+
}
|
|
29
57
|
|
|
30
58
|
pi.registerFlag("no-permissions", {
|
|
31
59
|
description: "Disable permission gate (skip confirmation prompts for dangerous operations)",
|
|
@@ -43,33 +71,36 @@ const extension: ExtensionFactory = (pi) => {
|
|
|
43
71
|
const reason = classifyBashCommand(command);
|
|
44
72
|
if (!reason || sessionAllow.has(reason)) return;
|
|
45
73
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
const choice = await ctx.ui.select(
|
|
49
|
-
`${reason}\nCommand: ${command}`,
|
|
50
|
-
[ALLOW, ALWAYS, DENY],
|
|
51
|
-
);
|
|
52
|
-
|
|
53
|
-
if (choice === ALWAYS) { sessionAllow.add(reason); return; }
|
|
54
|
-
if (choice === ALLOW) return;
|
|
55
|
-
return denyResult(reason);
|
|
74
|
+
return promptOrBlock(ctx, reason, `Command: ${command}`, () => sessionAllow.add(reason));
|
|
56
75
|
}
|
|
57
76
|
|
|
58
77
|
if (toolName === "write" || toolName === "edit") {
|
|
59
78
|
const filePath = (event.input as { path?: string }).path ?? "";
|
|
60
79
|
const reason = filePath ? classifyFilePath(filePath, ctx.cwd) : null;
|
|
61
|
-
if (!reason
|
|
80
|
+
if (!reason) return;
|
|
62
81
|
|
|
63
|
-
if (
|
|
82
|
+
if (reason === OUTSIDE_CWD_REASON) {
|
|
83
|
+
const resolved = resolve(ctx.cwd, filePath);
|
|
84
|
+
if (isPathAllowed(resolved)) return;
|
|
85
|
+
|
|
86
|
+
return promptOrBlock(ctx, reason, `File: ${filePath}`, () => allowedPaths.add(resolved));
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
if (sessionAllow.has(reason)) return;
|
|
90
|
+
|
|
91
|
+
return promptOrBlock(ctx, reason, `Path: ${filePath}`, () => sessionAllow.add(reason));
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
if (toolName === "read" || toolName === "grep" || toolName === "find" || toolName === "ls") {
|
|
95
|
+
const filePath = (event.input as { path?: string }).path ?? "";
|
|
96
|
+
if (!filePath) return;
|
|
64
97
|
|
|
65
|
-
const
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
);
|
|
98
|
+
const resolved = resolve(ctx.cwd, filePath);
|
|
99
|
+
const reason = isOutsideCwd(filePath, ctx.cwd);
|
|
100
|
+
if (!reason) return;
|
|
101
|
+
if (isPathAllowed(resolved)) return;
|
|
69
102
|
|
|
70
|
-
|
|
71
|
-
if (choice === ALLOW) return;
|
|
72
|
-
return denyResult(reason);
|
|
103
|
+
return promptOrBlock(ctx, reason, `Path: ${filePath}`, () => allowedPaths.add(resolved));
|
|
73
104
|
}
|
|
74
105
|
});
|
|
75
106
|
};
|
|
@@ -5,6 +5,8 @@
|
|
|
5
5
|
|
|
6
6
|
import { resolve, relative } from "node:path";
|
|
7
7
|
|
|
8
|
+
export const OUTSIDE_CWD_REASON = "Accesses path outside working directory";
|
|
9
|
+
|
|
8
10
|
interface DenylistEntry {
|
|
9
11
|
pattern: RegExp;
|
|
10
12
|
reason: string;
|
|
@@ -75,8 +77,20 @@ export function classifyFilePath(filePath: string, projectRoot: string): string
|
|
|
75
77
|
const rel = relative(projectRoot, resolved);
|
|
76
78
|
|
|
77
79
|
if (rel.startsWith("..")) {
|
|
78
|
-
return
|
|
80
|
+
return OUTSIDE_CWD_REASON;
|
|
79
81
|
}
|
|
80
82
|
|
|
81
83
|
return findMatchingReason(SENSITIVE_FILE_PATTERNS, filePath);
|
|
82
84
|
}
|
|
85
|
+
|
|
86
|
+
/**
|
|
87
|
+
* Returns a reason string if the file path resolves outside the given cwd,
|
|
88
|
+
* or null if inside (or empty).
|
|
89
|
+
*/
|
|
90
|
+
export function isOutsideCwd(filePath: string, cwd: string): string | null {
|
|
91
|
+
if (!filePath) return null;
|
|
92
|
+
const resolved = resolve(cwd, filePath);
|
|
93
|
+
const rel = relative(cwd, resolved);
|
|
94
|
+
if (rel.startsWith("..")) return OUTSIDE_CWD_REASON;
|
|
95
|
+
return null;
|
|
96
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@dcl-regenesislabs/opendcl",
|
|
3
|
-
"version": "0.1.4-
|
|
3
|
+
"version": "0.1.4-22809399016.commit-8d4acb4",
|
|
4
4
|
"description": "AI coding assistant for Decentraland SDK7 scene development",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -44,7 +44,8 @@
|
|
|
44
44
|
"author": "Decentraland",
|
|
45
45
|
"license": "Apache-2.0",
|
|
46
46
|
"dependencies": {
|
|
47
|
-
"@mariozechner/pi-coding-agent": "^0.54.0"
|
|
47
|
+
"@mariozechner/pi-coding-agent": "^0.54.0",
|
|
48
|
+
"playwright-core": "^1.58.2"
|
|
48
49
|
},
|
|
49
50
|
"devDependencies": {
|
|
50
51
|
"@types/node": "^22.0.0",
|
|
@@ -66,5 +67,5 @@
|
|
|
66
67
|
"prompts/",
|
|
67
68
|
"context/"
|
|
68
69
|
],
|
|
69
|
-
"commit": "
|
|
70
|
+
"commit": "8d4acb4ad164fa52bed5cbd4ce94f269b5e027f2"
|
|
70
71
|
}
|
package/prompts/system.md
CHANGED
|
@@ -115,11 +115,34 @@ scene-project/
|
|
|
115
115
|
- `context/asset-packs-catalog.md` — 2,700+ models from the official Decentraland Creator Hub (furniture, structures, decorations, etc.)
|
|
116
116
|
- Download matching models with `curl -o models/filename.glb "URL"` before referencing them in code.
|
|
117
117
|
|
|
118
|
+
### Visual Iteration Workflow
|
|
119
|
+
|
|
120
|
+
When the preview server is running, **proactively use the `screenshot` tool after making scene changes**. Don't wait for the user to check — verify your own work:
|
|
121
|
+
|
|
122
|
+
1. Write code or modify the scene.
|
|
123
|
+
2. Use `screenshot` (with a `wait` of ~2000ms for hot-reload) to see the result.
|
|
124
|
+
3. Describe what you see honestly — what's working, what's missing, what looks wrong.
|
|
125
|
+
4. If something is off, fix it and screenshot again.
|
|
126
|
+
|
|
127
|
+
This way the user gets a working scene without having to open a browser and report issues back to you.
|
|
128
|
+
|
|
129
|
+
The screenshot tool supports actions before capture — move around (moveForward, moveLeft, etc.), look around (lookLeft, lookUp), click objects, press keys. Use these to explore from different angles or test interactivity.
|
|
130
|
+
|
|
131
|
+
The browser stays open between calls — only the first screenshot takes ~15s (launch + enter scene). Subsequent ones are instant.
|
|
132
|
+
|
|
133
|
+
If the user asks you to iterate autonomously (e.g., "keep going until it looks right"):
|
|
134
|
+
1. Make code changes.
|
|
135
|
+
2. Wait for hot reload (~2s), then take a screenshot.
|
|
136
|
+
3. Analyze whether the result matches the goal.
|
|
137
|
+
4. If not, make targeted fixes and screenshot again.
|
|
138
|
+
5. Repeat (up to 5 iterations) until done.
|
|
139
|
+
|
|
118
140
|
## Tools & Commands
|
|
119
141
|
|
|
120
142
|
You have these Decentraland-specific tools — **use them directly** when the user's request matches:
|
|
121
143
|
- `init` — Scaffold a new scene (**always use this first** in an empty folder)
|
|
122
144
|
- `preview` — Start the Bevy-web preview server
|
|
145
|
+
- `screenshot` — Capture a screenshot of the running preview. Supports movement and interaction actions before capture.
|
|
123
146
|
- `deploy` — Deploy to Genesis City or a World (auto-detects from scene.json)
|
|
124
147
|
- `tasks` — List or stop running background processes
|
|
125
148
|
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: visual-feedback
|
|
3
|
+
description: Use the screenshot tool to see the running Decentraland preview, verify scene changes visually, explore from different angles, and iterate until the scene looks right. Use when the preview is running and you need to check what the scene looks like, debug visual issues, verify placement, or iterate on appearance.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Visual Feedback — Seeing Your Scene
|
|
7
|
+
|
|
8
|
+
The `screenshot` tool lets you capture what the Decentraland preview looks like right now. The browser stays open between calls — only the first screenshot takes ~15s (launch + enter scene). After that, screenshots are instant.
|
|
9
|
+
|
|
10
|
+
**Prerequisites:** The preview server must be running (`/preview`). The tool auto-detects the preview URL.
|
|
11
|
+
|
|
12
|
+
## When to Use Screenshots
|
|
13
|
+
|
|
14
|
+
- **After placing objects** — verify they're positioned correctly, not floating or buried
|
|
15
|
+
- **After changing materials/colors** — confirm the visual result matches intent
|
|
16
|
+
- **After downloading 3D models** — check they loaded and look right
|
|
17
|
+
- **When the user asks "how does it look?"** — show them and describe what you see
|
|
18
|
+
- **When debugging** — "the tree is invisible" → screenshot to see what's actually rendering
|
|
19
|
+
- **When iterating** — code → screenshot → fix → screenshot until it's right
|
|
20
|
+
|
|
21
|
+
## Basic Usage
|
|
22
|
+
|
|
23
|
+
Take a screenshot of the current view:
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
Use the screenshot tool with no actions to capture the current scene view.
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
The tool returns the image directly — you'll see it and can describe what's visible.
|
|
30
|
+
|
|
31
|
+
## Movement & Exploration
|
|
32
|
+
|
|
33
|
+
The scene camera **always faces north** in headless mode. Movement is relative to compass direction, not camera:
|
|
34
|
+
|
|
35
|
+
| Action | Direction | Key |
|
|
36
|
+
|--------|-----------|-----|
|
|
37
|
+
| `moveForward` | North (toward top of screen) | W |
|
|
38
|
+
| `moveBack` | South (toward bottom) | S |
|
|
39
|
+
| `moveRight` | East (toward right) | D |
|
|
40
|
+
| `moveLeft` | West (toward left) | A |
|
|
41
|
+
|
|
42
|
+
Movement speed is ~6 meters/second. Default `holdMs` is 500ms (~3 meters).
|
|
43
|
+
|
|
44
|
+
### Exploring a scene
|
|
45
|
+
|
|
46
|
+
To see objects from different angles, chain movement actions before capturing:
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
screenshot with actions:
|
|
50
|
+
1. moveForward (holdMs: 1000) — walk 6m north
|
|
51
|
+
2. moveRight (holdMs: 500) — strafe 3m east
|
|
52
|
+
→ captures screenshot from the new position
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Camera rotation
|
|
56
|
+
|
|
57
|
+
Use look actions to rotate the camera view:
|
|
58
|
+
|
|
59
|
+
| Action | Effect |
|
|
60
|
+
|--------|--------|
|
|
61
|
+
| `lookLeft` | Rotate camera left |
|
|
62
|
+
| `lookRight` | Rotate camera right |
|
|
63
|
+
| `lookUp` | Tilt camera up |
|
|
64
|
+
| `lookDown` | Tilt camera down |
|
|
65
|
+
|
|
66
|
+
Default rotation is 200 pixels of mouse drag. Use `dx`/`dy` for more or less.
|
|
67
|
+
|
|
68
|
+
## Interacting Before Capture
|
|
69
|
+
|
|
70
|
+
### Click on objects
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
screenshot with actions:
|
|
74
|
+
1. click (x: 640, y: 400) — click center of viewport
|
|
75
|
+
→ captures the result (e.g., object selected, door opened)
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Coordinates are in pixels (viewport is 1280×720).
|
|
79
|
+
|
|
80
|
+
### Press keys
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
screenshot with actions:
|
|
84
|
+
1. key (key: "1") — toggle editor camera
|
|
85
|
+
2. wait (ms: 500) — let camera settle
|
|
86
|
+
→ captures the overhead editor view
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
## Visual Iteration Pattern
|
|
90
|
+
|
|
91
|
+
When building or modifying a scene, use this loop:
|
|
92
|
+
|
|
93
|
+
1. **Make code changes** (write to `src/index.ts`)
|
|
94
|
+
2. **Wait for hot reload** — use `wait` action with ~2000ms
|
|
95
|
+
3. **Take screenshot** — see what changed
|
|
96
|
+
4. **Evaluate** — describe honestly: what works, what's wrong, what's missing
|
|
97
|
+
5. **Fix and repeat** — up to 5 iterations
|
|
98
|
+
|
|
99
|
+
Example flow:
|
|
100
|
+
```
|
|
101
|
+
1. Write code to add a red cube at (8, 1, 8)
|
|
102
|
+
2. screenshot with actions: [wait 2000ms]
|
|
103
|
+
→ "I can see a red cube floating 1m above the ground at the center. It looks correct."
|
|
104
|
+
3. Write code to add a blue sphere next to it
|
|
105
|
+
4. screenshot with actions: [wait 2000ms]
|
|
106
|
+
→ "The blue sphere is there but it's intersecting the cube. Let me adjust the position."
|
|
107
|
+
5. Fix the position, screenshot again
|
|
108
|
+
→ "Both objects are now properly placed side by side."
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## Scene Layout Awareness
|
|
112
|
+
|
|
113
|
+
- Each **parcel** is 16×16 meters. A 1×1 scene has coordinates 0-16 in X and Z.
|
|
114
|
+
- **Y is up**. Ground level is Y=0.
|
|
115
|
+
- The avatar **spawns near the south-west corner** (low X, low Z).
|
|
116
|
+
- Objects at the **center** of a 1×1 scene are at roughly (8, 0, 8).
|
|
117
|
+
- The **minimap** in the top-left corner shows coordinates and a compass.
|
|
118
|
+
|
|
119
|
+
For a 2×2 scene (32×32m), center is (16, 0, 16) and the avatar needs to walk ~14m north and east to reach it.
|
|
120
|
+
|
|
121
|
+
## Troubleshooting
|
|
122
|
+
|
|
123
|
+
| Problem | Solution |
|
|
124
|
+
|---------|----------|
|
|
125
|
+
| Screenshot shows welcome screen | The scene hasn't loaded yet — increase `wait` time |
|
|
126
|
+
| Black/empty screenshot | Preview server may have crashed — check `/tasks` |
|
|
127
|
+
| Objects not visible | They may be behind the camera (south of avatar) — use `moveBack` or `lookLeft`/`lookRight` to find them |
|
|
128
|
+
| Scene looks different after code change | Hot reload takes ~1-2s — add a `wait` action of 2000ms |
|
|
129
|
+
| "No preview server running" | Start it with `/preview` first |
|
|
130
|
+
|
|
131
|
+
## Tips
|
|
132
|
+
|
|
133
|
+
- **First screenshot is slow** (~15s) because it launches a browser and enters the scene. After that, screenshots are instant.
|
|
134
|
+
- **The browser persists** across all screenshot calls in the session — no repeated logins.
|
|
135
|
+
- **Don't over-move** — keep `holdMs` values short (300-800ms) to avoid overshooting targets.
|
|
136
|
+
- **Hot reload** — after writing code, wait ~2s before screenshotting to let the scene update.
|
|
137
|
+
- **Describe honestly** — if something looks wrong, say so. The user trusts your visual assessment.
|