effing-use 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +136 -24
- package/package.json +1 -1
- package/skills/effing-use/SKILL.md +10 -7
- package/src/browser/delta.ts +41 -4
- package/src/browser/engine.ts +116 -79
- package/src/browser/evidence.ts +30 -8
- package/src/browser/identity.ts +142 -0
- package/src/browser/macro.ts +21 -5
- package/src/browser/record.ts +33 -1
- package/src/browser/refs.ts +143 -109
- package/src/http.ts +1 -1
- package/src/tools/act.ts +1 -5
package/README.md
CHANGED
|
@@ -8,14 +8,14 @@ Full Chromium automation in **3 tools, 3.9 KB**. Same pages, same clicks, same s
|
|
|
8
8
|
|
|
9
9
|
## Install (Bun-only)
|
|
10
10
|
|
|
11
|
-
Requires [Bun](https://bun.sh) 1.4+. Installs from npm in seconds — the tarball is ~16 kB ([`effing-use` v0.
|
|
11
|
+
Requires [Bun](https://bun.sh) 1.4+. Installs from npm in seconds — the tarball is ~16 kB ([`effing-use` v0.2.1](https://www.npmjs.com/package/effing-use)):
|
|
12
12
|
|
|
13
13
|
```bash
|
|
14
14
|
# No install needed — bunx fetches from npm on first run
|
|
15
|
-
bunx effing-use # CLI (needs running server)
|
|
16
|
-
bunx effing-use-http # HTTP MCP on :3123 (/mcp) — shared across editors
|
|
17
|
-
bunx effing-use-stdio # STDIO MCP (single editor)
|
|
18
|
-
bunx playwright install chromium --only-shell
|
|
15
|
+
bunx --package effing-use effing-use # CLI (needs running server)
|
|
16
|
+
bunx --package effing-use effing-use-http # HTTP MCP on :3123 (/mcp) — shared across editors
|
|
17
|
+
bunx --package effing-use effing-use-stdio # STDIO MCP (single editor)
|
|
18
|
+
bunx playwright install chromium --only-shell # one-time Chromium download (~150 MB)
|
|
19
19
|
```
|
|
20
20
|
|
|
21
21
|
Optional — install globally so `effing-use` is on your PATH:
|
|
@@ -41,41 +41,74 @@ Re-run `bunx playwright install chromium --only-shell` whenever you bump the `pl
|
|
|
41
41
|
|
|
42
42
|
**Why Chromium is separate:** Playwright supports multiple browsers and updates its pinned builds every release, so the binary can't live inside the npm tarball (ours is 16 kB). Docker users skip this — Chromium is baked into the `mcr.microsoft.com/playwright` base image.
|
|
43
43
|
|
|
44
|
+
### npm vs source
|
|
45
|
+
|
|
46
|
+
| Source | Command | When to use |
|
|
47
|
+
| ------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
|
48
|
+
| **npm** (`bunx --package effing-use`) | `effing-use`, `effing-use-http`, `effing-use-stdio` | Recommended — always matches published `package.json` version, no clone needed |
|
|
49
|
+
| **source** (`bun src/*.ts`) | `bun src/cli.ts`, `bun src/http.ts`, `bun src/index.ts` | Local dev / unreleased changes |
|
|
50
|
+
|
|
51
|
+
Both expose the same 3 bins: `effing-use` = CLI (HTTP client), `effing-use-stdio` = MCP stdio, `effing-use-http` = MCP http. Don't mix them — `effing-use` alone is **not** an MCP server (it prints CLI help and exits, causing `Failed to parse message` / `MCP server has stopped`).
|
|
52
|
+
|
|
44
53
|
## Run it in 60 seconds (recommended path)
|
|
45
54
|
|
|
46
55
|
**Step 1/3 — Start the server.** One server, every local editor:
|
|
47
56
|
|
|
48
57
|
```bash
|
|
58
|
+
# Docker (recommended — Chromium baked in, no local install)
|
|
59
|
+
docker compose up --build -d
|
|
60
|
+
curl http://localhost:3123/healthz # {"ok":true,"name":"effing-use"}
|
|
61
|
+
|
|
62
|
+
# Or without Docker
|
|
49
63
|
bun install
|
|
50
64
|
bunx playwright install chromium --only-shell
|
|
51
|
-
|
|
52
|
-
|
|
65
|
+
bun src/http.ts # or: bunx --package effing-use effing-use-http
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
The HTTP server binds `0.0.0.0:3123` inside Docker (so forwarded ports work) and sets `idleTimeout: 0` so the MCP SSE stream isn't killed after 10s of idle time. Plain HTTP on loopback is intentional — add TLS at the edge for remote use.
|
|
69
|
+
|
|
70
|
+
**Step 2/3 — Connect.** Pick one transport:
|
|
71
|
+
|
|
72
|
+
**HTTP (shared — recommended for multiple VS Code windows):**
|
|
73
|
+
|
|
74
|
+
```json
|
|
75
|
+
// ~/.config/Code/User/mcp.json (global, all workspaces) or .vscode/mcp.json
|
|
76
|
+
{
|
|
77
|
+
"servers": {
|
|
78
|
+
"effing-use": { "type": "http", "url": "http://localhost:3123/mcp" }
|
|
79
|
+
}
|
|
80
|
+
}
|
|
53
81
|
```
|
|
54
82
|
|
|
55
|
-
|
|
83
|
+
One Docker/bun process serves every window. Use same `sessionId` to share tabs, different `sessionId` to isolate.
|
|
56
84
|
|
|
57
|
-
**
|
|
85
|
+
**STDIO (isolated — one browser per window):**
|
|
58
86
|
|
|
59
87
|
```json
|
|
60
88
|
{
|
|
61
|
-
"
|
|
89
|
+
"servers": {
|
|
62
90
|
"effing-use": {
|
|
91
|
+
"type": "stdio",
|
|
63
92
|
"command": "bunx",
|
|
64
|
-
"args": ["effing-use-stdio"]
|
|
93
|
+
"args": ["--package", "effing-use", "effing-use-stdio"],
|
|
94
|
+
"env": {
|
|
95
|
+
"BROWSER_HEADLESS": "true",
|
|
96
|
+
"OUTPUT_DIR": "${workspaceFolder}/.browser-use"
|
|
97
|
+
}
|
|
65
98
|
}
|
|
66
99
|
}
|
|
67
100
|
}
|
|
68
101
|
```
|
|
69
102
|
|
|
70
|
-
|
|
103
|
+
From source: `command: "bun"`, `args: ["/absolute/path/to/effing-use/src/index.ts"]`.
|
|
71
104
|
|
|
72
105
|
**CLI vs MCP — when to use which:**
|
|
73
106
|
|
|
74
|
-
| Surface
|
|
75
|
-
|
|
76
|
-
| **MCP (stdio)** | `effing-use-stdio` / `bun src/index.ts` | No (spawns own browser)
|
|
77
|
-
| **MCP (http)**
|
|
78
|
-
| **CLI**
|
|
107
|
+
| Surface | Command | Needs server? | Best for |
|
|
108
|
+
| --------------- | --------------------------------------- | ------------------------------- | ------------------------------- |
|
|
109
|
+
| **MCP (stdio)** | `effing-use-stdio` / `bun src/index.ts` | No (spawns own browser) | Single VS Code / Claude Desktop |
|
|
110
|
+
| **MCP (http)** | `effing-use-http` / `bun src/http.ts` | Yes (`:3123`) | Shared across editors, Docker |
|
|
111
|
+
| **CLI** | `effing-use` / `bun src/cli.ts` | Yes (`:3123`, `EFFING_USE_URL`) | Terminal agents, scripts, CI |
|
|
79
112
|
|
|
80
113
|
The CLI is a thin HTTP client over the same engine — `effing-use observe --kind snapshot --mode delta` and `browser_observe kind=snapshot mode=delta` hit the same code. Start the server once (`bun src/http.ts` or `docker compose up`), then use either face.
|
|
81
114
|
|
|
@@ -94,6 +127,46 @@ The CLI is a thin HTTP client over the same engine — `effing-use observe --kin
|
|
|
94
127
|
|
|
95
128
|
No Docker? `bun src/cli.ts --help` (CLI), `bun src/index.ts` (STDIO MCP), or `bun src/http.ts` (`:3123` `/mcp`) works directly.
|
|
96
129
|
|
|
130
|
+
### CLI usage
|
|
131
|
+
|
|
132
|
+
The CLI needs a running HTTP server (`EFFING_USE_URL` defaults to `http://localhost:3123/mcp`):
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
# Observe
|
|
136
|
+
effing-use observe --kind snapshot --mode delta --session dev
|
|
137
|
+
effing-use observe --kind title --session dev
|
|
138
|
+
effing-use observe --kind screenshot --session dev
|
|
139
|
+
|
|
140
|
+
# Act
|
|
141
|
+
effing-use act --action open --value https://example.com --session dev
|
|
142
|
+
effing-use act --action click --target e5 --expect 'url~/dashboard' --session dev
|
|
143
|
+
effing-use act --action batch --file steps.json --session dev
|
|
144
|
+
|
|
145
|
+
# Extract
|
|
146
|
+
effing-use extract --kind text --selector main --session dev
|
|
147
|
+
effing-use extract --kind state --session dev
|
|
148
|
+
|
|
149
|
+
# With custom server URL
|
|
150
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use observe --kind snapshot
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
From npm without global install: `bunx --package effing-use effing-use observe --kind snapshot`.
|
|
154
|
+
|
|
155
|
+
### Docker
|
|
156
|
+
|
|
157
|
+
```bash
|
|
158
|
+
docker compose up --build -d # build + start (0.0.0.0:3123, idleTimeout:0)
|
|
159
|
+
docker compose logs -f effing-use # tail logs
|
|
160
|
+
curl http://localhost:3123/healthz # health check
|
|
161
|
+
docker compose down # stop (add -v to remove volumes)
|
|
162
|
+
# Rebuild after pulling new code
|
|
163
|
+
docker compose up --build -d
|
|
164
|
+
# Orphaned old container holding :3123? (renamed service)
|
|
165
|
+
docker rm -f effing-use-computer-use-1 && docker compose up -d
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Compose details: `restart: unless-stopped`, `init: true` + `ipc: host` (Playwright flags), `.browser-use/` bind-mounted, `EFFING_PORT` overrides host port (`EFFING_PORT=4000 docker compose up -d`).
|
|
169
|
+
|
|
97
170
|
## Why agents prefer 3 tools
|
|
98
171
|
|
|
99
172
|
- 🪶 **Tiny handshake, full surface** — `browser_act` (32 actions), `browser_observe` (8 kinds + `mode`/`scope`), `browser_extract` (8 kinds incl. `state`). No schema bloat, no guessing which of 24 tools to call.
|
|
@@ -102,6 +175,36 @@ No Docker? `bun src/cli.ts --help` (CLI), `bun src/index.ts` (STDIO MCP), or `bu
|
|
|
102
175
|
- ⚡ **One call, not five** — `batch` runs fill+press flows in one turn (max 20 steps, stops on first error). `goal` plans or returns `E_GOAL_UNCLEAR` + `suggestedSteps` instead of hallucinating.
|
|
103
176
|
- 🐳 **Shared, not spawned** — one Docker server serves every local VS Code instance. No per-client `npx` spawn.
|
|
104
177
|
|
|
178
|
+
## Harness — verify, delta, state, replay
|
|
179
|
+
|
|
180
|
+
v2 turns the browser into a harness: AI actions are verifiable, diffable, and replayable — not just fire-and-forget clicks.
|
|
181
|
+
|
|
182
|
+
**1. Verify — fingerprints + expectations + failure contract**
|
|
183
|
+
|
|
184
|
+
- **Fingerprint registry** (`src/browser/identity.ts`): every `eN` ref is fingerprinted (role, accessibleName, textHash, box, pathHash). Stale refs rebind only on an UNAMBIGUOUS identity match (`rebound:true`); if several elements match equally they fail with `E_STALE` + hint — the engine never guesses a target.
|
|
185
|
+
- **Expect mini-language** (`expect` on any `browser_act`): `url~/dashboard` | `text~/Saved/` | `visible=.modal` | `gone=.spinner` — evaluated server-side, returns `E_EXPECT` / `E_BAD_EXPECT` on mismatch instead of hallucinated success. ReDoS-capped and regex-validated.
|
|
186
|
+
- **Failure contract**: after an uncertain mutation the engine sets `mustObserve:true` — next mutation fails with `E_MUST_OBSERVE` until you re-snapshot. No blind chains.
|
|
187
|
+
- **Evidence envelope**: every `browser_act` returns `effect: { urlChanged, urlBefore/After, domChanged, consoleErrors, networkFailures }` capped at `EFFECT_MAX_CHARS` (800) so the agent sees what actually happened.
|
|
188
|
+
|
|
189
|
+
**2. Delta — pay only for what changed**
|
|
190
|
+
|
|
191
|
+
- `browser_observe kind=snapshot mode=delta` (default) — MutationObserver dirty flag + baseline diff. Returns only `[changed]` lines or `unchanged:true` on stable pages (~90% token saving). `mode=full` for complete dump, `scope="<css>"` for subtree.
|
|
192
|
+
- `DELTA_DEFAULT=true` — flip to `false` to default to full snapshots.
|
|
193
|
+
|
|
194
|
+
**3. State — per-session memory**
|
|
195
|
+
|
|
196
|
+
- `browser_act action=note value="..."` appends to `notes` (capped `STATE_MAX_LINES=40`), `browser_extract kind=state` reads `notes` + `lastActions` ring (last 10). Persisted to `.browser-use/state/<session>.md` so agents survive context compaction.
|
|
197
|
+
|
|
198
|
+
**4. Record → Compile → Replay — deterministic macros**
|
|
199
|
+
|
|
200
|
+
- `record_start` / `record_stop` captures every step with resolved selectors + fingerprints. Secrets auto-redacted (`RECORD_REDACT=true`, `«redacted»` for password/otp/token fields).
|
|
201
|
+
- `compile` generates `.browser-use/macros/<name>.{ts,md}` — a Playwright `run(page)` function + a `SKILL.md` doc. Irreversible steps (`submit`/`pay`/`delete`/etc.) are flagged `requiresApproval`.
|
|
202
|
+
- `replay` replays deterministically; pauses with `E_APPROVAL_REQUIRED` until `approve:true` if any irreversible step exists.
|
|
203
|
+
|
|
204
|
+
**5. CLI — same engine, no MCP client**
|
|
205
|
+
|
|
206
|
+
- `effing-use observe/act/extract` over `EFFING_USE_URL` (`http://localhost:3123/mcp`) — for terminal agents, scripts, and CI. See [CLI usage](#cli-usage).
|
|
207
|
+
|
|
105
208
|
## The loop (agents: follow this order)
|
|
106
209
|
|
|
107
210
|
1. `browser_observe` kind=`snapshot` → get `[eN]` refs (default `mode: delta` — only changed lines; `mode: full` for complete dump; `scope: "<css>"` for subtree). Never guess refs, re-snapshot after navigation.
|
|
@@ -130,18 +233,27 @@ Errors are always `{ ok: false, code, message, hint }` with `E_NOT_FOUND | E_TIM
|
|
|
130
233
|
|
|
131
234
|
Need Firefox/WebKit, device emulation, or persistent profiles? Use Playwright MCP. Need context budget for Chromium work? Stay here. Full breakdown in [`docs/COMPARISON.md`](docs/COMPARISON.md).
|
|
132
235
|
|
|
133
|
-
## Record → replay
|
|
236
|
+
## Record → replay (harness)
|
|
134
237
|
|
|
135
238
|
```bash
|
|
136
|
-
# MCP
|
|
239
|
+
# MCP — capture any flow, compile to code + skill, replay deterministically
|
|
137
240
|
browser_act action=record_start value=my-flow
|
|
138
|
-
# ... do the flow ...
|
|
139
|
-
browser_act action=record_stop
|
|
140
|
-
browser_act action=compile value=my-flow
|
|
141
|
-
browser_act action=replay value=my-flow
|
|
142
|
-
browser_act action=replay value=my-flow approve=true
|
|
241
|
+
# ... do the flow (clicks, fills, etc.) ...
|
|
242
|
+
browser_act action=record_stop # -> .browser-use/recordings/my-flow.json (secrets redacted)
|
|
243
|
+
browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
|
|
244
|
+
browser_act action=replay value=my-flow # pauses with E_APPROVAL_REQUIRED if irreversible
|
|
245
|
+
browser_act action=replay value=my-flow approve=true # replay with approval
|
|
246
|
+
|
|
247
|
+
# CLI — same flow over HTTP
|
|
248
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action record_start --value my-flow
|
|
249
|
+
# ... do the flow via CLI or MCP ...
|
|
250
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action record_stop
|
|
251
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action compile --value my-flow
|
|
252
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action replay --value my-flow --approve true
|
|
143
253
|
```
|
|
144
254
|
|
|
255
|
+
Artifacts: `recordings/*.json` (raw steps), `macros/*.ts` (Playwright `run(page)`), `macros/*.md` (skill doc). Redaction and approval gates are on by default (`RECORD_REDACT=true`).
|
|
256
|
+
|
|
145
257
|
## Config
|
|
146
258
|
|
|
147
259
|
Env defaults in [`.env.example`](.env.example): `BROWSER_HEADLESS`, `BROWSER_VIEWPORT_W/H`, `BROWSER_TIMEOUT_MS`, `OUTPUT_DIR` (`.browser-use/`, gitignored), `OUTPUT_MAX_CHARS`, `ALLOW_EVAL`, `DELTA_DEFAULT`, `EFFECT_MAX_CHARS`, `STATE_MAX_LINES`, `RECORD_REDACT`, `EFFING_USE_URL`.
|
package/package.json
CHANGED
|
@@ -15,14 +15,14 @@ cost than 21-tool browser servers. `tools/list` is ~3.9KB.
|
|
|
15
15
|
|
|
16
16
|
## The loop (always follow this order)
|
|
17
17
|
|
|
18
|
-
1. `browser_observe` with `kind: "snapshot"` → get `[eN]` refs. Default is `mode: "delta"` (
|
|
18
|
+
1. `browser_observe` with `kind: "snapshot"` → get `[eN]` refs. Default is `mode: "delta"` (`[changed]`/`[removed]` lines since last observe; `unchanged:true` when the page is quiet — navigation automatically forces a fresh `mode: "full"`). Use `mode: "full"` for a complete dump, `scope: "<css>"` to observe a subtree.
|
|
19
19
|
2. `browser_act` to interact (`open`, `click`, `fill`, `type`, `press`, `select`, `check`, `wait`, …). Add `expect: "url~/dashboard"` or `text~/Saved/` for deterministic post-conditions.
|
|
20
|
-
3. `browser_observe` with `kind: "snapshot"` again — delta returns `unchanged:true` if nothing changed, or `[changed]` lines.
|
|
20
|
+
3. `browser_observe` with `kind: "snapshot"` again — delta returns `unchanged:true` if nothing changed, or `[changed]`/`[removed]` lines.
|
|
21
21
|
4. `browser_extract` with `kind: "text" | "table" | "query" | "state"` → scrape or read task state.
|
|
22
22
|
|
|
23
23
|
Rules:
|
|
24
24
|
|
|
25
|
-
- Never guess refs. Re-snapshot after every navigation
|
|
25
|
+
- Never guess refs. Re-snapshot after every navigation (refs are page-keyed — a cross-page stale ref returns `E_NOT_FOUND`). On a same-page re-render the engine rebinds only an UNAMBIGUOUS identity match (result carries `rebound:true`); if the fingerprint matches several elements it fails with `E_STALE` instead of guessing — re-observe.
|
|
26
26
|
- Prefer `batch`: one `browser_act` with `steps[]` for fill+press flows (max 20 steps, stops on first error).
|
|
27
27
|
- Large outputs are files under `.browser-use/` (gitignored). Read the path, not the preview.
|
|
28
28
|
- Snapshots are capped at `OUTPUT_MAX_CHARS` (default 4000) with `…[truncated N chars, see file]`.
|
|
@@ -37,8 +37,8 @@ Rules:
|
|
|
37
37
|
`scroll` (`up`/`down`/`top`/`bottom` or a target), `back`/`forward`/`reload`,
|
|
38
38
|
`wait` (`ms:500`, `text:Saved`, or a ref), `dialog_accept`/`dialog_dismiss` (arm before the triggering step),
|
|
39
39
|
`resize` (`1280x800` in `value`), `tab_new`/`tab_select`/`tab_close`, `close`,
|
|
40
|
-
`goal` (deterministic add-todo/search planner, else `E_GOAL_UNCLEAR` + `suggestedSteps`),
|
|
41
|
-
`batch` (needs `steps[]`), `note` (append to task state), `record_start`/`record_stop` (capture flow), `compile` (
|
|
40
|
+
`goal` (deterministic add-todo/search planner, else `E_GOAL_UNCLEAR` + `suggestedSteps`; counts as a mutation — subject to the `mustObserve` guard, evidence, and recording),
|
|
41
|
+
`batch` (needs `steps[]`), `note` (append to task state), `record_start`/`record_stop` (capture flow), `compile` (standalone `.ts` + SKILL.md; selectors resolved id → name → ARIA → data-\* → placeholder → text), `replay` (deterministic replay; runs benign steps, halts AT approval-gated steps).
|
|
42
42
|
`expect` mini-language: `url~<regex>` | `text~<regex>` | `visible=<css>` | `gone=<css>` — evaluated in code, returns `E_EXPECT` or `E_BAD_EXPECT`.
|
|
43
43
|
|
|
44
44
|
**browser_observe** — read-only: `snapshot` (e-refs, `mode: full|delta` default delta, `scope: <css>`), `screenshot` (file path,
|
|
@@ -70,8 +70,11 @@ browser_act action=record_start value=my-flow
|
|
|
70
70
|
# ... do the flow ...
|
|
71
71
|
browser_act action=record_stop
|
|
72
72
|
browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
|
|
73
|
-
browser_act action=replay value=my-flow #
|
|
74
|
-
|
|
73
|
+
browser_act action=replay value=my-flow # runs benign steps, halts AT the first
|
|
74
|
+
# approval-gated step → E_APPROVAL_REQUIRED
|
|
75
|
+
# { step: N (1-indexed), completed: C }
|
|
76
|
+
browser_act action=replay value=my-flow approve=true # resumes at the gate; the
|
|
77
|
+
# benign prefix is NOT re-run (cursor)
|
|
75
78
|
```
|
|
76
79
|
|
|
77
80
|
## Setup
|
package/src/browser/delta.ts
CHANGED
|
@@ -44,6 +44,11 @@ export function getBaseline(sessionId: string): string | null {
|
|
|
44
44
|
return ensureState(sessionId).baseline;
|
|
45
45
|
}
|
|
46
46
|
|
|
47
|
+
/** URL the current baseline was captured on — forces mode:full after navigation (plan §5.1). */
|
|
48
|
+
export function getBaselineUrl(sessionId: string): string | null {
|
|
49
|
+
return pageStates.get(key(sessionId))?.baselineUrl ?? null;
|
|
50
|
+
}
|
|
51
|
+
|
|
47
52
|
export function isDirty(sessionId: string): boolean {
|
|
48
53
|
return ensureState(sessionId).dirty;
|
|
49
54
|
}
|
|
@@ -66,8 +71,32 @@ export async function injectDirtyObserver(
|
|
|
66
71
|
w.__effDirty = false;
|
|
67
72
|
w.__effSid = sid;
|
|
68
73
|
if (w.__effObs) w.__effObs.disconnect();
|
|
69
|
-
|
|
70
|
-
|
|
74
|
+
// Attribute spam that never changes the snapshot's ref lines (class,
|
|
75
|
+
// style, expand/animation state) used to mark every live SPA permanently
|
|
76
|
+
// dirty — the cheap "unchanged" fast path never fired. Ignore those;
|
|
77
|
+
// content changes still arrive via childList/characterData/input/change.
|
|
78
|
+
const NOISE_ATTRS = [
|
|
79
|
+
"class",
|
|
80
|
+
"style",
|
|
81
|
+
"aria-expanded",
|
|
82
|
+
"data-state",
|
|
83
|
+
"data-orientation",
|
|
84
|
+
"data-scroll-state",
|
|
85
|
+
"data-highlighted",
|
|
86
|
+
"data-hovered",
|
|
87
|
+
"data-dragging",
|
|
88
|
+
"data-resizing",
|
|
89
|
+
"data-index",
|
|
90
|
+
];
|
|
91
|
+
const obs = new MutationObserver((muts) => {
|
|
92
|
+
for (const m of muts) {
|
|
93
|
+
if (m.type === "attributes") {
|
|
94
|
+
const name = m.attributeName || "";
|
|
95
|
+
if (NOISE_ATTRS.indexOf(name) !== -1) continue;
|
|
96
|
+
}
|
|
97
|
+
w.__effDirty = true;
|
|
98
|
+
return;
|
|
99
|
+
}
|
|
71
100
|
});
|
|
72
101
|
obs.observe(document.documentElement, {
|
|
73
102
|
childList: true,
|
|
@@ -127,10 +156,18 @@ export function computeDelta(
|
|
|
127
156
|
const baseLines = baseline.split("\n");
|
|
128
157
|
const curLines = current.split("\n");
|
|
129
158
|
const baseSet = new Set(baseLines);
|
|
159
|
+
const curSet = new Set(curLines);
|
|
130
160
|
const changed = curLines.filter((l) => !baseSet.has(l));
|
|
161
|
+
const removed = baseLines.filter((l) => !curSet.has(l));
|
|
162
|
+
if (changed.length === 0 && removed.length === 0)
|
|
163
|
+
return { delta: "", unchanged: true };
|
|
131
164
|
const unchangedCount = curLines.length - changed.length;
|
|
132
|
-
if (changed.length === 0) return { delta: "", unchanged: true };
|
|
133
165
|
const header = `…${unchangedCount} unchanged lines…`;
|
|
134
|
-
const delta = [
|
|
166
|
+
const delta = [
|
|
167
|
+
header,
|
|
168
|
+
...changed.map((l) => `[changed] ${l}`),
|
|
169
|
+
// plan §5.1: removed nodes are reported explicitly
|
|
170
|
+
...removed.map((l) => `[removed] ${l}`),
|
|
171
|
+
].join("\n");
|
|
135
172
|
return { delta, unchanged: false };
|
|
136
173
|
}
|
package/src/browser/engine.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import type { Page, BrowserContext } from "playwright";
|
|
2
2
|
import type { Config } from "../config.js";
|
|
3
|
-
import { EngineError, resolveLocator } from "./refs.js";
|
|
3
|
+
import { EngineError, resolveLocator, stableSelectorFor } from "./refs.js";
|
|
4
4
|
import { cap, saveText, stamp } from "./output.js";
|
|
5
5
|
import {
|
|
6
6
|
getConsoleLogs,
|
|
@@ -13,6 +13,7 @@ import {
|
|
|
13
13
|
import { collectEffect, evaluateExpect, parseExpect } from "./evidence.js";
|
|
14
14
|
import {
|
|
15
15
|
getBaseline,
|
|
16
|
+
getBaselineUrl,
|
|
16
17
|
setBaseline,
|
|
17
18
|
computeDelta,
|
|
18
19
|
injectDirtyObserver,
|
|
@@ -28,9 +29,17 @@ import {
|
|
|
28
29
|
loadRecording,
|
|
29
30
|
startRecording,
|
|
30
31
|
stopRecording,
|
|
32
|
+
getReplayCursor,
|
|
33
|
+
setReplayCursor,
|
|
34
|
+
clearReplayCursor,
|
|
31
35
|
} from "./record.js";
|
|
32
36
|
import { compileMacro, isIrreversible } from "./macro.js";
|
|
33
|
-
import {
|
|
37
|
+
import {
|
|
38
|
+
registerFingerprints,
|
|
39
|
+
clearRegistry,
|
|
40
|
+
getFingerprint,
|
|
41
|
+
extractFingerprints as collectFingerprints,
|
|
42
|
+
} from "./identity.js";
|
|
34
43
|
|
|
35
44
|
export type ActAction =
|
|
36
45
|
| "open"
|
|
@@ -85,6 +94,19 @@ function timeoutOf(config: Config): number {
|
|
|
85
94
|
async function liteState(page: Page): Promise<{ url: string; title: string }> {
|
|
86
95
|
return { url: page.url(), title: await page.title().catch(() => "") };
|
|
87
96
|
}
|
|
97
|
+
|
|
98
|
+
/**
|
|
99
|
+
* SPA navigations commit after the action returns (client router awaits data),
|
|
100
|
+
* so evidence collected immediately can report urlChanged:false for a click
|
|
101
|
+
* that DOES navigate (observed: kikitai "Try paste & read" → /read).
|
|
102
|
+
* Brief settle: one tick, then a second if the URL moved.
|
|
103
|
+
*/
|
|
104
|
+
async function settleSpa(page: Page, action: string): Promise<void> {
|
|
105
|
+
if (!["click", "dblclick", "press", "goal"].includes(action)) return;
|
|
106
|
+
const u0 = page.url();
|
|
107
|
+
await page.waitForTimeout(60);
|
|
108
|
+
if (page.url() !== u0) await page.waitForTimeout(60);
|
|
109
|
+
}
|
|
88
110
|
function toEngineError(e: unknown, fallbackHint: string): EngineError {
|
|
89
111
|
if (e instanceof EngineError) return e;
|
|
90
112
|
const msg = e instanceof Error ? e.message : String(e);
|
|
@@ -116,6 +138,7 @@ const MUTATING = new Set<string>([
|
|
|
116
138
|
"open",
|
|
117
139
|
"goto",
|
|
118
140
|
"resize",
|
|
141
|
+
"goal",
|
|
119
142
|
"tab_new",
|
|
120
143
|
"tab_select",
|
|
121
144
|
"tab_close",
|
|
@@ -136,6 +159,10 @@ export async function doAct(
|
|
|
136
159
|
const timeout = timeoutOf(config);
|
|
137
160
|
const sid = opts?.sessionId ?? "default";
|
|
138
161
|
|
|
162
|
+
// Per-action stash must never leak from a previous (possibly failed) act
|
|
163
|
+
const pageBag = page as unknown as Record<string, unknown>;
|
|
164
|
+
delete pageBag.__rebound;
|
|
165
|
+
|
|
139
166
|
// Non-page meta actions (no mustObserve guard, no evidence)
|
|
140
167
|
if (action === "note") {
|
|
141
168
|
if (!value && !target)
|
|
@@ -209,24 +236,28 @@ export async function doAct(
|
|
|
209
236
|
);
|
|
210
237
|
});
|
|
211
238
|
const approve = opts?.approve === true;
|
|
212
|
-
//
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
);
|
|
216
|
-
if (needsApproval.length > 0 && !approve) {
|
|
217
|
-
const first = needsApproval[0];
|
|
218
|
-
return {
|
|
219
|
-
ok: false,
|
|
220
|
-
code: "E_APPROVAL_REQUIRED",
|
|
221
|
-
message: `Step ${first.seq} requires approval (${first.op}).`,
|
|
222
|
-
hint: "Re-run with approve:true.",
|
|
223
|
-
step: first.seq,
|
|
224
|
-
description: `${first.op} ${first.target ?? ""}`,
|
|
225
|
-
} as any;
|
|
226
|
-
}
|
|
227
|
-
// Deterministic replay
|
|
239
|
+
// Plan §6.3: run benign steps, pause AT each approval-gated step, and keep
|
|
240
|
+
// a cursor so approve:true resumes here instead of re-running the prefix.
|
|
241
|
+
const stepsArr = rec.steps as any[];
|
|
242
|
+
const start = getReplayCursor(sid, name) ?? 0;
|
|
228
243
|
const results: any[] = [];
|
|
229
|
-
for (
|
|
244
|
+
for (let i = start; i < stepsArr.length; i++) {
|
|
245
|
+
const s = stepsArr[i];
|
|
246
|
+
if (isIrreversible(s) && !approve) {
|
|
247
|
+
setReplayCursor(sid, name, i);
|
|
248
|
+
const ran = i - start;
|
|
249
|
+
return {
|
|
250
|
+
ok: false,
|
|
251
|
+
code: "E_APPROVAL_REQUIRED",
|
|
252
|
+
message:
|
|
253
|
+
`Step ${i + 1} requires approval (${s.op}).` +
|
|
254
|
+
(ran > 0 ? ` Ran ${ran} prior step(s).` : ""),
|
|
255
|
+
hint: "Re-run with approve:true to resume from this step.",
|
|
256
|
+
step: i + 1,
|
|
257
|
+
completed: results.length,
|
|
258
|
+
description: `${s.op} ${s.target ?? ""}`,
|
|
259
|
+
} as any;
|
|
260
|
+
}
|
|
230
261
|
try {
|
|
231
262
|
const r = await doAct(
|
|
232
263
|
page,
|
|
@@ -238,6 +269,7 @@ export async function doAct(
|
|
|
238
269
|
);
|
|
239
270
|
results.push({ ok: true, seq: s.seq, ...r });
|
|
240
271
|
} catch (e) {
|
|
272
|
+
clearReplayCursor(sid, name);
|
|
241
273
|
const err = toEngineError(e, "Replay failed.");
|
|
242
274
|
results.push({
|
|
243
275
|
ok: false,
|
|
@@ -248,7 +280,14 @@ export async function doAct(
|
|
|
248
280
|
return { replayed: false, name, failSeq: s.seq, results } as any;
|
|
249
281
|
}
|
|
250
282
|
}
|
|
251
|
-
|
|
283
|
+
clearReplayCursor(sid, name);
|
|
284
|
+
return {
|
|
285
|
+
replayed: true,
|
|
286
|
+
name,
|
|
287
|
+
steps: results.length,
|
|
288
|
+
resumedFrom: start,
|
|
289
|
+
results,
|
|
290
|
+
} as any;
|
|
252
291
|
}
|
|
253
292
|
|
|
254
293
|
// Failure contract guard
|
|
@@ -275,8 +314,10 @@ export async function doAct(
|
|
|
275
314
|
const urlBefore = page.url();
|
|
276
315
|
let domBefore: string | null = null;
|
|
277
316
|
try {
|
|
317
|
+
// Full innerText (50k cap) — evidence diffs need the whole page, not just
|
|
318
|
+
// the first 200 chars (v0.2: below-fold changes looked like no-ops).
|
|
278
319
|
domBefore = await page.evaluate(
|
|
279
|
-
() => document.body?.innerText?.slice(0,
|
|
320
|
+
() => document.body?.innerText?.slice(0, 50000) ?? "",
|
|
280
321
|
);
|
|
281
322
|
} catch {
|
|
282
323
|
domBefore = null;
|
|
@@ -579,7 +620,12 @@ export async function doAct(
|
|
|
579
620
|
result = { closed: true };
|
|
580
621
|
break;
|
|
581
622
|
case "goal":
|
|
582
|
-
|
|
623
|
+
// Route through the normal post-action pipeline (guard, evidence,
|
|
624
|
+
// dirty flag, recording) — v0.2 returned here and skipped all of it.
|
|
625
|
+
result = {
|
|
626
|
+
...(await doGoal(page, config, value ?? target ?? "", opts)),
|
|
627
|
+
};
|
|
628
|
+
break;
|
|
583
629
|
case "batch":
|
|
584
630
|
throw new EngineError(
|
|
585
631
|
"E_BAD_INPUT",
|
|
@@ -608,6 +654,7 @@ export async function doAct(
|
|
|
608
654
|
// Post-action: evidence, expect, state, dirty, recording
|
|
609
655
|
let effect: any = undefined;
|
|
610
656
|
if (isMutating(action)) {
|
|
657
|
+
await settleSpa(page, action);
|
|
611
658
|
effect = await collectEffect(
|
|
612
659
|
page,
|
|
613
660
|
sid,
|
|
@@ -657,8 +704,15 @@ export async function doAct(
|
|
|
657
704
|
`${actionLine} -> ${effect ? JSON.stringify(effect).slice(0, 80) : "ok"}`,
|
|
658
705
|
);
|
|
659
706
|
|
|
660
|
-
// Recording capture
|
|
707
|
+
// Recording capture — resolve a portable selector (id → name → ARIA →
|
|
708
|
+
// data-* → placeholder → text) so compiled macros don't depend on e-refs.
|
|
661
709
|
if (isRecording(sid) && isMutating(action)) {
|
|
710
|
+
let resolved = target;
|
|
711
|
+
if (target) {
|
|
712
|
+
resolved =
|
|
713
|
+
(await stableSelectorFor(page, target, sid).catch(() => null)) ??
|
|
714
|
+
target;
|
|
715
|
+
}
|
|
662
716
|
captureStep(
|
|
663
717
|
sid,
|
|
664
718
|
{
|
|
@@ -666,7 +720,8 @@ export async function doAct(
|
|
|
666
720
|
target,
|
|
667
721
|
value,
|
|
668
722
|
expect: opts?.expect,
|
|
669
|
-
resolvedSelector:
|
|
723
|
+
resolvedSelector: resolved,
|
|
724
|
+
targetFingerprint: target ? getFingerprint(sid, target) : undefined,
|
|
670
725
|
},
|
|
671
726
|
config,
|
|
672
727
|
);
|
|
@@ -795,9 +850,22 @@ export async function doObserve(
|
|
|
795
850
|
| "full"
|
|
796
851
|
| "delta";
|
|
797
852
|
const scope = opts?.scope;
|
|
853
|
+
// Navigation forces a fresh full baseline (plan §5.1) and needs a settle
|
|
854
|
+
// beat: SPA routes hydrate after the URL changes — snapshots taken too
|
|
855
|
+
// early miss the header/nav entirely (observed on kikitai /read).
|
|
856
|
+
const prevUrl = getBaselineUrl(sessionId);
|
|
857
|
+
const urlChanged = prevUrl !== null && prevUrl !== page.url();
|
|
858
|
+
const settleHydration = async () => {
|
|
859
|
+
if (!urlChanged) return;
|
|
860
|
+
await page
|
|
861
|
+
.waitForLoadState("networkidle", { timeout: 1500 })
|
|
862
|
+
.catch(() => {});
|
|
863
|
+
await page.waitForTimeout(80);
|
|
864
|
+
};
|
|
798
865
|
// Scoped snapshot: only within selector
|
|
799
866
|
let yaml: string;
|
|
800
867
|
if (scope) {
|
|
868
|
+
await settleHydration();
|
|
801
869
|
yaml = await buildSnapshot(page, scope);
|
|
802
870
|
const path = await saveText(config, stamp("snapshot", "yaml"), yaml);
|
|
803
871
|
const { text, truncated } = cap(yaml, config.outputMaxChars);
|
|
@@ -815,8 +883,9 @@ export async function doObserve(
|
|
|
815
883
|
// Delta path
|
|
816
884
|
if (mode === "delta") {
|
|
817
885
|
const baseline = getBaseline(sessionId);
|
|
818
|
-
//
|
|
819
|
-
if (!baseline) {
|
|
886
|
+
// No baseline, or the page navigated since the baseline → forced full
|
|
887
|
+
if (!baseline || urlChanged) {
|
|
888
|
+
await settleHydration();
|
|
820
889
|
yaml = await buildSnapshot(page);
|
|
821
890
|
setBaseline(sessionId, yaml, page.url());
|
|
822
891
|
await injectDirtyObserver(page, sessionId);
|
|
@@ -879,6 +948,7 @@ export async function doObserve(
|
|
|
879
948
|
};
|
|
880
949
|
}
|
|
881
950
|
// Full mode
|
|
951
|
+
await settleHydration();
|
|
882
952
|
yaml = await buildSnapshot(page);
|
|
883
953
|
setBaseline(sessionId, yaml, page.url());
|
|
884
954
|
await injectDirtyObserver(page, sessionId);
|
|
@@ -960,10 +1030,28 @@ async function buildSnapshot(page: Page, scope?: string): Promise<string> {
|
|
|
960
1030
|
}
|
|
961
1031
|
return els.slice(0, 200).map((el) => {
|
|
962
1032
|
const tag = el.tagName.toLowerCase();
|
|
963
|
-
|
|
1033
|
+
let text = (el.textContent ?? "")
|
|
964
1034
|
.trim()
|
|
965
1035
|
.replace(/\s+/g, " ")
|
|
966
1036
|
.slice(0, 80);
|
|
1037
|
+
// Form controls carry no text — surface placeholder/label so refs stop
|
|
1038
|
+
// showing as `input ""`, and mark checked state (real state changes that
|
|
1039
|
+
// delta should surface).
|
|
1040
|
+
const input = el as HTMLInputElement;
|
|
1041
|
+
if (!text) {
|
|
1042
|
+
text = (
|
|
1043
|
+
input.placeholder ||
|
|
1044
|
+
(input.labels && input.labels[0]?.textContent) ||
|
|
1045
|
+
""
|
|
1046
|
+
)
|
|
1047
|
+
.trim()
|
|
1048
|
+
.replace(/\s+/g, " ")
|
|
1049
|
+
.slice(0, 80);
|
|
1050
|
+
}
|
|
1051
|
+
const checked =
|
|
1052
|
+
(tag === "input" || tag === "select") && input.checked
|
|
1053
|
+
? " [checked]"
|
|
1054
|
+
: "";
|
|
967
1055
|
const aria = el.getAttribute("aria-label") ?? "";
|
|
968
1056
|
const name = el.getAttribute("name") ?? "";
|
|
969
1057
|
const id = el.getAttribute("id") ?? "";
|
|
@@ -974,64 +1062,13 @@ async function buildSnapshot(page: Page, scope?: string): Promise<string> {
|
|
|
974
1062
|
]
|
|
975
1063
|
.filter(Boolean)
|
|
976
1064
|
.join(" ");
|
|
977
|
-
return `${tag} "${text}"${extra ? " " + extra : ""}`;
|
|
1065
|
+
return `${tag} "${text}"${extra ? " " + extra : ""}${checked}`;
|
|
978
1066
|
});
|
|
979
1067
|
}, scope ?? null);
|
|
980
1068
|
const lines = items.map((line, i) => `[e${i}] ${line}`);
|
|
981
1069
|
return `${header}\n${lines.join("\n") || "(no interactive elements)"}`;
|
|
982
1070
|
}
|
|
983
1071
|
|
|
984
|
-
async function collectFingerprints(
|
|
985
|
-
page: Page,
|
|
986
|
-
): Promise<Array<{ ref: string; fp: import("./identity.js").Fingerprint }>> {
|
|
987
|
-
const raw = await page
|
|
988
|
-
.evaluate(() => {
|
|
989
|
-
const els = [
|
|
990
|
-
...document.querySelectorAll(
|
|
991
|
-
"button, a, input, select, textarea, [role=button], [tabindex]",
|
|
992
|
-
),
|
|
993
|
-
].slice(0, 200);
|
|
994
|
-
return els.map((el) => {
|
|
995
|
-
const text = (el.textContent ?? "")
|
|
996
|
-
.trim()
|
|
997
|
-
.replace(/\s+/g, " ")
|
|
998
|
-
.slice(0, 80);
|
|
999
|
-
const aria = el.getAttribute("aria-label") ?? "";
|
|
1000
|
-
const role = el.getAttribute("role") ?? el.tagName.toLowerCase();
|
|
1001
|
-
let cur: Element | null = el;
|
|
1002
|
-
const parts: string[] = [];
|
|
1003
|
-
while (cur && parts.length < 6) {
|
|
1004
|
-
parts.push(cur.tagName.toLowerCase());
|
|
1005
|
-
cur = cur.parentElement;
|
|
1006
|
-
}
|
|
1007
|
-
const rect = el.getBoundingClientRect();
|
|
1008
|
-
return {
|
|
1009
|
-
role,
|
|
1010
|
-
accessibleName: aria,
|
|
1011
|
-
text,
|
|
1012
|
-
pathHash: parts.join(">"),
|
|
1013
|
-
box: { x: rect.x, y: rect.y, w: rect.width, h: rect.height },
|
|
1014
|
-
};
|
|
1015
|
-
});
|
|
1016
|
-
})
|
|
1017
|
-
.catch(() => [] as any[]);
|
|
1018
|
-
return raw.map((r: any, i: number) => ({
|
|
1019
|
-
ref: `e${i}`,
|
|
1020
|
-
fp: {
|
|
1021
|
-
role: r.role || "generic",
|
|
1022
|
-
accessibleName: (r.accessibleName || "").trim().slice(0, 80),
|
|
1023
|
-
textHash: (r.text || "").trim().replace(/\s+/g, " ").slice(0, 80),
|
|
1024
|
-
box: {
|
|
1025
|
-
x: Math.round(r.box.x),
|
|
1026
|
-
y: Math.round(r.box.y),
|
|
1027
|
-
w: Math.round(r.box.w),
|
|
1028
|
-
h: Math.round(r.box.h),
|
|
1029
|
-
},
|
|
1030
|
-
pathHash: r.pathHash || "",
|
|
1031
|
-
},
|
|
1032
|
-
}));
|
|
1033
|
-
}
|
|
1034
|
-
|
|
1035
1072
|
// ---------------------------------------------------------------- extract
|
|
1036
1073
|
|
|
1037
1074
|
export async function doExtract(
|
package/src/browser/evidence.ts
CHANGED
|
@@ -46,16 +46,35 @@ export async function collectEffect(
|
|
|
46
46
|
const urlAfter = page.url();
|
|
47
47
|
const urlChanged = urlBefore !== urlAfter;
|
|
48
48
|
|
|
49
|
-
// domChanged:
|
|
49
|
+
// domChanged: line-level diff of body.innerText (50k cap). v0.2 only
|
|
50
|
+
// compared the first 200 chars, so any below-fold change looked like a no-op
|
|
51
|
+
// (and fed the failure contract wrong "no DOM change" signals).
|
|
50
52
|
let domChanged: string[] = [];
|
|
51
53
|
try {
|
|
52
54
|
const currentDom = await page.evaluate(
|
|
53
|
-
() => document.body?.innerText?.slice(0,
|
|
55
|
+
() => document.body?.innerText?.slice(0, 50000) ?? "",
|
|
54
56
|
);
|
|
55
57
|
if (domBeforeHash !== null && currentDom !== domBeforeHash) {
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
58
|
+
const before = new Set(
|
|
59
|
+
domBeforeHash
|
|
60
|
+
.split("\n")
|
|
61
|
+
.map((l) => l.trim())
|
|
62
|
+
.filter(Boolean),
|
|
63
|
+
);
|
|
64
|
+
const after = currentDom
|
|
65
|
+
.split("\n")
|
|
66
|
+
.map((l) => l.trim())
|
|
67
|
+
.filter(Boolean);
|
|
68
|
+
const afterSet = new Set(after);
|
|
69
|
+
const changed = after
|
|
70
|
+
.filter((l) => !before.has(l))
|
|
71
|
+
.slice(0, 5)
|
|
72
|
+
.map((l) => l.slice(0, 140));
|
|
73
|
+
const removed = [...before]
|
|
74
|
+
.filter((l) => !afterSet.has(l))
|
|
75
|
+
.slice(0, 3)
|
|
76
|
+
.map((l) => `- ${l.slice(0, 138)}`);
|
|
77
|
+
domChanged = [...changed, ...removed].slice(0, 5);
|
|
59
78
|
}
|
|
60
79
|
} catch {
|
|
61
80
|
// ignore
|
|
@@ -96,9 +115,12 @@ export function parseExpect(
|
|
|
96
115
|
const s = expect.trim();
|
|
97
116
|
let parsed: { kind: string; pattern: string } | null = null;
|
|
98
117
|
if (s.startsWith("url~")) parsed = { kind: "url", pattern: s.slice(4) };
|
|
99
|
-
else if (s.startsWith("text~"))
|
|
100
|
-
|
|
101
|
-
else if (s.startsWith("
|
|
118
|
+
else if (s.startsWith("text~"))
|
|
119
|
+
parsed = { kind: "text", pattern: s.slice(5) };
|
|
120
|
+
else if (s.startsWith("visible="))
|
|
121
|
+
parsed = { kind: "visible", pattern: s.slice(8) };
|
|
122
|
+
else if (s.startsWith("gone="))
|
|
123
|
+
parsed = { kind: "gone", pattern: s.slice(5) };
|
|
102
124
|
else return null;
|
|
103
125
|
if (parsed.pattern.length > EXPECT_MAX_PATTERN) return null;
|
|
104
126
|
// Validate regex patterns early for url~/text~
|
package/src/browser/identity.ts
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
import type { Page } from "playwright";
|
|
2
|
+
|
|
1
3
|
export type Fingerprint = {
|
|
2
4
|
role: string;
|
|
3
5
|
accessibleName: string;
|
|
@@ -6,6 +8,114 @@ export type Fingerprint = {
|
|
|
6
8
|
pathHash: string;
|
|
7
9
|
};
|
|
8
10
|
|
|
11
|
+
/** Raw per-element data as extracted inside the page (before normalization). */
|
|
12
|
+
export type RawFp = {
|
|
13
|
+
role: string;
|
|
14
|
+
accessibleName: string;
|
|
15
|
+
text: string;
|
|
16
|
+
pathHash: string;
|
|
17
|
+
box: { x: number; y: number; w: number; h: number };
|
|
18
|
+
};
|
|
19
|
+
|
|
20
|
+
/**
|
|
21
|
+
* Browser-side fingerprint extractor — SINGLE SOURCE OF TRUTH shared by
|
|
22
|
+
* snapshot registration, stale-ref validation and rebind search. Serialized
|
|
23
|
+
* into the page as a string so it never closes over module scope.
|
|
24
|
+
*
|
|
25
|
+
* v0.2 bug fixed here: fingerprints used tag-name roles ("input"), aria-label
|
|
26
|
+
* only, and own textContent (always "" for form controls), so every bare
|
|
27
|
+
* checkbox shared one identity — rebind then clicked the wrong element
|
|
28
|
+
* (TodoMVC: stale item toggle rebound to toggle-all). Now:
|
|
29
|
+
* - role: implicit ARIA role (input[type=checkbox] → "checkbox")
|
|
30
|
+
* - accessibleName: aria-label → <label> → alt → title → placeholder → text
|
|
31
|
+
* - textHash: own text, else contextual text (nearest li/label/td/row), else
|
|
32
|
+
* placeholder — so item toggles carry their todo text
|
|
33
|
+
*/
|
|
34
|
+
export const RAW_FP_SCRIPT = `(() => {
|
|
35
|
+
const SEL = "button, a, input, select, textarea, [role=button], [tabindex]";
|
|
36
|
+
const norm = (s) => (s || "").trim().replace(/\\s+/g, " ").slice(0, 80);
|
|
37
|
+
const implicitRole = (el) => {
|
|
38
|
+
const r = el.getAttribute("role");
|
|
39
|
+
if (r) return r;
|
|
40
|
+
const tag = el.tagName.toLowerCase();
|
|
41
|
+
if (tag === "a") return el.hasAttribute("href") ? "link" : "generic";
|
|
42
|
+
if (tag === "button") return "button";
|
|
43
|
+
if (tag === "textarea") return "textbox";
|
|
44
|
+
if (tag === "select") return "combobox";
|
|
45
|
+
if (tag === "summary") return "button";
|
|
46
|
+
if (tag === "input") {
|
|
47
|
+
const t = (el.getAttribute("type") || "text").toLowerCase();
|
|
48
|
+
if (t === "checkbox") return "checkbox";
|
|
49
|
+
if (t === "radio") return "radio";
|
|
50
|
+
if (t === "submit" || t === "button" || t === "reset" || t === "image") return "button";
|
|
51
|
+
if (t === "range") return "slider";
|
|
52
|
+
if (t === "hidden") return "hidden";
|
|
53
|
+
return "textbox";
|
|
54
|
+
}
|
|
55
|
+
return tag;
|
|
56
|
+
};
|
|
57
|
+
const accName = (el) => {
|
|
58
|
+
const aria = el.getAttribute("aria-label");
|
|
59
|
+
if (aria && aria.trim()) return norm(aria);
|
|
60
|
+
if (el.labels && el.labels.length > 0) {
|
|
61
|
+
const t = norm(el.labels[0].textContent);
|
|
62
|
+
if (t) return t;
|
|
63
|
+
}
|
|
64
|
+
const alt = el.getAttribute("alt");
|
|
65
|
+
if (alt && alt.trim()) return norm(alt);
|
|
66
|
+
const title = el.getAttribute("title");
|
|
67
|
+
if (title && title.trim()) return norm(title);
|
|
68
|
+
const ph = el.getAttribute("placeholder");
|
|
69
|
+
if (ph && ph.trim()) return norm(ph);
|
|
70
|
+
return norm(el.textContent);
|
|
71
|
+
};
|
|
72
|
+
const textOf = (el) => {
|
|
73
|
+
const own = norm(el.textContent);
|
|
74
|
+
if (own) return own;
|
|
75
|
+
const ctx = el.closest && el.closest("li, label, td, [role=listitem], [role=row]");
|
|
76
|
+
if (ctx) {
|
|
77
|
+
const t = norm(ctx.textContent);
|
|
78
|
+
if (t) return t;
|
|
79
|
+
}
|
|
80
|
+
return norm(el.getAttribute("placeholder"));
|
|
81
|
+
};
|
|
82
|
+
const pathOf = (el) => {
|
|
83
|
+
const parts = [];
|
|
84
|
+
let cur = el;
|
|
85
|
+
while (cur && parts.length < 6) {
|
|
86
|
+
parts.push(cur.tagName.toLowerCase());
|
|
87
|
+
cur = cur.parentElement;
|
|
88
|
+
}
|
|
89
|
+
return parts.join(">");
|
|
90
|
+
};
|
|
91
|
+
const els = Array.prototype.slice.call(document.querySelectorAll(SEL), 0, 200);
|
|
92
|
+
return els.map((el) => {
|
|
93
|
+
const rect = el.getBoundingClientRect();
|
|
94
|
+
return {
|
|
95
|
+
role: implicitRole(el),
|
|
96
|
+
accessibleName: accName(el),
|
|
97
|
+
text: textOf(el),
|
|
98
|
+
pathHash: pathOf(el),
|
|
99
|
+
box: {
|
|
100
|
+
x: Math.round(rect.x),
|
|
101
|
+
y: Math.round(rect.y),
|
|
102
|
+
w: Math.round(rect.width),
|
|
103
|
+
h: Math.round(rect.height)
|
|
104
|
+
}
|
|
105
|
+
};
|
|
106
|
+
});
|
|
107
|
+
})()`;
|
|
108
|
+
|
|
109
|
+
/** Extract fingerprints for every snapshot-visible element, as e-refs. */
|
|
110
|
+
export async function extractFingerprints(
|
|
111
|
+
page: Page,
|
|
112
|
+
): Promise<Array<{ ref: string; fp: Fingerprint }>> {
|
|
113
|
+
const raw = (await page
|
|
114
|
+
.evaluate(RAW_FP_SCRIPT)
|
|
115
|
+
.catch(() => [] as RawFp[])) as RawFp[];
|
|
116
|
+
return raw.map((r, i) => ({ ref: `e${i}`, fp: buildFingerprint(r) }));
|
|
117
|
+
}
|
|
118
|
+
|
|
9
119
|
function hashText(s: string): string {
|
|
10
120
|
return s.trim().replace(/\s+/g, " ").slice(0, 80);
|
|
11
121
|
}
|
|
@@ -41,6 +151,38 @@ export function clearRegistry(sessionId: string): void {
|
|
|
41
151
|
registry.delete(sessionId);
|
|
42
152
|
}
|
|
43
153
|
|
|
154
|
+
export type RebindCandidate = { index: number; fp: Fingerprint };
|
|
155
|
+
|
|
156
|
+
/**
|
|
157
|
+
* Conservative rebind ladder (P0 fix): pick an element only when the match is
|
|
158
|
+
* UNAMBIGUOUS — never guess among equals.
|
|
159
|
+
*
|
|
160
|
+
* 1. loose match (role + accessibleName + textHash) — unique → take it
|
|
161
|
+
* 2. multiple loose → narrow by identical pathHash; still >1 → give up
|
|
162
|
+
* 3. no loose → fuzzy (role + name/text contains) under the same rules
|
|
163
|
+
*
|
|
164
|
+
* Returns the chosen candidate index, or null when zero or more than one
|
|
165
|
+
* candidate survives — callers turn null into E_STALE (fail loud) instead of
|
|
166
|
+
* clicking the wrong element.
|
|
167
|
+
*/
|
|
168
|
+
export function chooseRebindIndex(
|
|
169
|
+
expected: Fingerprint,
|
|
170
|
+
candidates: RebindCandidate[],
|
|
171
|
+
): number | null {
|
|
172
|
+
const narrow = (pool: RebindCandidate[]): number | null => {
|
|
173
|
+
if (pool.length === 1) return pool[0].index;
|
|
174
|
+
if (pool.length > 1 && expected.pathHash) {
|
|
175
|
+
const byPath = pool.filter((c) => c.fp.pathHash === expected.pathHash);
|
|
176
|
+
if (byPath.length === 1) return byPath[0].index;
|
|
177
|
+
}
|
|
178
|
+
return null;
|
|
179
|
+
};
|
|
180
|
+
const loose = candidates.filter((c) => looseMatch(expected, c.fp));
|
|
181
|
+
if (loose.length > 0) return narrow(loose);
|
|
182
|
+
const fuzzy = candidates.filter((c) => fuzzyMatch(expected, c.fp));
|
|
183
|
+
return narrow(fuzzy);
|
|
184
|
+
}
|
|
185
|
+
|
|
44
186
|
export function fingerprintEquals(a: Fingerprint, b: Fingerprint): boolean {
|
|
45
187
|
return (
|
|
46
188
|
a.role === b.role &&
|
package/src/browser/macro.ts
CHANGED
|
@@ -32,8 +32,11 @@ export async function compileMacro(
|
|
|
32
32
|
const warnings: string[] = [];
|
|
33
33
|
const irreversibleSteps = steps.filter(isIrreversible);
|
|
34
34
|
if (irreversibleSteps.length > 0) {
|
|
35
|
+
// Steps are human-facing here — 1-indexed, matching the .md step list
|
|
35
36
|
warnings.push(
|
|
36
|
-
`${irreversibleSteps.length} step(s) require approval: ${irreversibleSteps
|
|
37
|
+
`${irreversibleSteps.length} step(s) require approval: steps ${irreversibleSteps
|
|
38
|
+
.map((s) => s.seq + 1)
|
|
39
|
+
.join(", ")}`,
|
|
37
40
|
);
|
|
38
41
|
}
|
|
39
42
|
|
|
@@ -49,12 +52,21 @@ export async function compileMacro(
|
|
|
49
52
|
const sel = selectorFor(s);
|
|
50
53
|
const val = (s.value ?? "").slice(0, 500);
|
|
51
54
|
const flag = isIrreversible(s) ? " // requiresApproval" : "";
|
|
55
|
+
// Recordings made before stable-selector resolution store session-scoped
|
|
56
|
+
// e-refs — flag them so nobody runs the .ts standalone expecting it to work.
|
|
57
|
+
const staleNote = /^e\d+$/.test(sel)
|
|
58
|
+
? " /* session-scoped e-ref — not a standalone selector */"
|
|
59
|
+
: "";
|
|
52
60
|
switch (s.op) {
|
|
53
61
|
case "click":
|
|
54
|
-
tsLines.push(
|
|
62
|
+
tsLines.push(
|
|
63
|
+
` await page.locator(${JSON.stringify(sel)}).click();${flag}${staleNote}`,
|
|
64
|
+
);
|
|
55
65
|
break;
|
|
56
66
|
case "fill":
|
|
57
|
-
tsLines.push(
|
|
67
|
+
tsLines.push(
|
|
68
|
+
` await page.locator(${JSON.stringify(sel)}).fill(${JSON.stringify(val)});${flag}${staleNote}`,
|
|
69
|
+
);
|
|
58
70
|
break;
|
|
59
71
|
case "press":
|
|
60
72
|
tsLines.push(
|
|
@@ -63,10 +75,14 @@ export async function compileMacro(
|
|
|
63
75
|
break;
|
|
64
76
|
case "goto":
|
|
65
77
|
case "open":
|
|
66
|
-
tsLines.push(
|
|
78
|
+
tsLines.push(
|
|
79
|
+
` await page.goto(${JSON.stringify(val || sel)});${flag}`,
|
|
80
|
+
);
|
|
67
81
|
break;
|
|
68
82
|
default:
|
|
69
|
-
tsLines.push(
|
|
83
|
+
tsLines.push(
|
|
84
|
+
` // ${JSON.stringify(s.op)} ${JSON.stringify(sel)} ${JSON.stringify(val)}${flag}`,
|
|
85
|
+
);
|
|
70
86
|
break;
|
|
71
87
|
}
|
|
72
88
|
}
|
package/src/browser/record.ts
CHANGED
|
@@ -97,7 +97,11 @@ export async function saveRecording(
|
|
|
97
97
|
const dir = join(config.outputDir, "recordings");
|
|
98
98
|
await mkdir(dir, { recursive: true });
|
|
99
99
|
const path = join(dir, `${safe}.json`);
|
|
100
|
-
const rec: Recording = {
|
|
100
|
+
const rec: Recording = {
|
|
101
|
+
name: safe,
|
|
102
|
+
createdAt: new Date().toISOString(),
|
|
103
|
+
steps,
|
|
104
|
+
};
|
|
101
105
|
await writeFile(path, JSON.stringify(rec, null, 2), "utf-8");
|
|
102
106
|
return path;
|
|
103
107
|
}
|
|
@@ -115,3 +119,31 @@ export async function loadRecording(
|
|
|
115
119
|
export function getActiveName(sessionId: string): string | null {
|
|
116
120
|
return activeRecordings.get(sessionId)?.name ?? null;
|
|
117
121
|
}
|
|
122
|
+
|
|
123
|
+
// ---------------------------------------------------------------- replay cursor
|
|
124
|
+
// Plan §6.3: pause AT an approval-gated step and resume with approve:true —
|
|
125
|
+
// the cursor keeps the benign prefix from running twice (non-idempotent steps).
|
|
126
|
+
const replayCursors = new Map<string, number>();
|
|
127
|
+
|
|
128
|
+
function cursorKey(sessionId: string, name: string): string {
|
|
129
|
+
return `${sessionId}:${name}`;
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
export function getReplayCursor(
|
|
133
|
+
sessionId: string,
|
|
134
|
+
name: string,
|
|
135
|
+
): number | undefined {
|
|
136
|
+
return replayCursors.get(cursorKey(sessionId, name));
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
export function setReplayCursor(
|
|
140
|
+
sessionId: string,
|
|
141
|
+
name: string,
|
|
142
|
+
step: number,
|
|
143
|
+
): void {
|
|
144
|
+
replayCursors.set(cursorKey(sessionId, name), step);
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
export function clearReplayCursor(sessionId: string, name: string): void {
|
|
148
|
+
replayCursors.delete(cursorKey(sessionId, name));
|
|
149
|
+
}
|
package/src/browser/refs.ts
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
1
|
import type { Page, Locator } from "playwright";
|
|
2
2
|
import {
|
|
3
3
|
getFingerprint,
|
|
4
|
-
|
|
5
|
-
|
|
4
|
+
buildFingerprint,
|
|
5
|
+
chooseRebindIndex,
|
|
6
|
+
RAW_FP_SCRIPT,
|
|
6
7
|
type Fingerprint,
|
|
8
|
+
type RawFp,
|
|
9
|
+
type RebindCandidate,
|
|
7
10
|
} from "./identity.js";
|
|
8
11
|
|
|
9
12
|
export class EngineError extends Error {
|
|
@@ -52,15 +55,20 @@ export async function resolveLocator(
|
|
|
52
55
|
if (sessionId) {
|
|
53
56
|
const expected = getFingerprint(sessionId, t);
|
|
54
57
|
if (expected) {
|
|
55
|
-
|
|
58
|
+
// Validate against the SAME extractor that registered the snapshot
|
|
59
|
+
const raws = (await page.evaluate(RAW_FP_SCRIPT).catch(() => null)) as
|
|
60
|
+
| RawFp[]
|
|
61
|
+
| null;
|
|
62
|
+
const idx = Number(t.slice(1));
|
|
63
|
+
const actual = raws && raws[idx] ? buildFingerprint(raws[idx]) : null;
|
|
56
64
|
if (actual) {
|
|
57
65
|
const exact =
|
|
58
66
|
expected.role === actual.role &&
|
|
59
67
|
expected.accessibleName === actual.accessibleName &&
|
|
60
68
|
expected.textHash === actual.textHash;
|
|
61
69
|
if (!exact) {
|
|
62
|
-
//
|
|
63
|
-
const rebound = await findRebound(page, expected);
|
|
70
|
+
// Conservative rebind — ambiguous → E_STALE, never guess
|
|
71
|
+
const rebound = await findRebound(page, expected, raws);
|
|
64
72
|
if (rebound) {
|
|
65
73
|
// Return rebound locator; caller can check rebound flag via session
|
|
66
74
|
(page as unknown as Record<string, unknown>).__rebound = true;
|
|
@@ -110,117 +118,143 @@ export async function resolveLocator(
|
|
|
110
118
|
);
|
|
111
119
|
}
|
|
112
120
|
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
const role = el.getAttribute("role") ?? el.tagName.toLowerCase();
|
|
120
|
-
// simple path hash: tag chain
|
|
121
|
-
let cur: Element | null = el;
|
|
122
|
-
const parts: string[] = [];
|
|
123
|
-
while (cur && parts.length < 6) {
|
|
124
|
-
parts.push(cur.tagName.toLowerCase());
|
|
125
|
-
cur = cur.parentElement;
|
|
126
|
-
}
|
|
127
|
-
return { role, accessibleName: aria, text, pathHash: parts.join(">") };
|
|
128
|
-
})
|
|
129
|
-
.catch(() => ({
|
|
130
|
-
role: "generic",
|
|
131
|
-
accessibleName: "",
|
|
132
|
-
text: "",
|
|
133
|
-
pathHash: "",
|
|
134
|
-
}));
|
|
135
|
-
return {
|
|
136
|
-
role: data.role || "generic",
|
|
137
|
-
accessibleName: (data.accessibleName || "").trim().slice(0, 80),
|
|
138
|
-
textHash: (data.text || "").trim().replace(/\s+/g, " ").slice(0, 80),
|
|
139
|
-
box: {
|
|
140
|
-
x: Math.round(box?.x ?? 0),
|
|
141
|
-
y: Math.round(box?.y ?? 0),
|
|
142
|
-
w: Math.round(box?.width ?? 0),
|
|
143
|
-
h: Math.round(box?.height ?? 0),
|
|
144
|
-
},
|
|
145
|
-
pathHash: data.pathHash || "",
|
|
146
|
-
};
|
|
147
|
-
}
|
|
148
|
-
|
|
121
|
+
/**
|
|
122
|
+
* Find where the element `expected` used to describe lives now.
|
|
123
|
+
* Uses the shared extractor + conservative chooser: only an UNAMBIGUOUS match
|
|
124
|
+
* is returned; zero or several equally-good candidates → null → E_STALE
|
|
125
|
+
* upstream (v0.2 bug: first-match rebind clicked toggle-all).
|
|
126
|
+
*/
|
|
149
127
|
async function findRebound(
|
|
150
128
|
page: Page,
|
|
151
129
|
expected: Fingerprint,
|
|
130
|
+
raws: RawFp[] | null,
|
|
152
131
|
): Promise<Locator | null> {
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
132
|
+
const list =
|
|
133
|
+
raws ?? ((await page.evaluate(RAW_FP_SCRIPT).catch(() => [])) as RawFp[]);
|
|
134
|
+
const candidates: RebindCandidate[] = list.map((r, i) => ({
|
|
135
|
+
index: i,
|
|
136
|
+
fp: buildFingerprint(r),
|
|
137
|
+
}));
|
|
138
|
+
const pick = chooseRebindIndex(expected, candidates);
|
|
139
|
+
if (pick === null) return null;
|
|
140
|
+
return page.locator(INTERACTIVE_SELECTOR).nth(pick);
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
/**
|
|
144
|
+
* Resolve a portable, standalone selector for a target (plan §6.2 preference:
|
|
145
|
+
* id → name → ARIA label → stable data-* → placeholder → recorded text).
|
|
146
|
+
* Never returns nth-index or session-scoped e-refs; null when nothing unique
|
|
147
|
+
* can be found (caller falls back to the raw target).
|
|
148
|
+
*/
|
|
149
|
+
export async function stableSelectorFor(
|
|
150
|
+
page: Page,
|
|
151
|
+
target: string,
|
|
152
|
+
sessionId?: string,
|
|
153
|
+
): Promise<string | null> {
|
|
154
|
+
try {
|
|
155
|
+
const loc = await resolveLocator(page, target, sessionId);
|
|
156
|
+
const cands = await loc
|
|
157
|
+
.first()
|
|
158
|
+
.evaluate((el) => {
|
|
159
|
+
const out: string[] = [];
|
|
160
|
+
const tag = el.tagName.toLowerCase();
|
|
161
|
+
const esc = (s: string) => {
|
|
162
|
+
const c = (
|
|
163
|
+
globalThis as unknown as {
|
|
164
|
+
CSS?: { escape?: (x: string) => string };
|
|
165
|
+
}
|
|
166
|
+
).CSS;
|
|
167
|
+
return c && c.escape ? c.escape(s) : s;
|
|
168
|
+
};
|
|
169
|
+
const av = (s: string): string | null => {
|
|
170
|
+
const bs = String.fromCharCode(92);
|
|
171
|
+
if (s.indexOf('"') !== -1 || s.indexOf(bs) !== -1) return null;
|
|
172
|
+
return '"' + s + '"';
|
|
173
|
+
};
|
|
174
|
+
const uniq = (s: string): boolean => {
|
|
175
|
+
try {
|
|
176
|
+
return document.querySelectorAll(s).length === 1;
|
|
177
|
+
} catch {
|
|
178
|
+
return false;
|
|
179
|
+
}
|
|
180
|
+
};
|
|
181
|
+
// 1. stable id (skip framework-generated/dynamic ids)
|
|
182
|
+
const id = el.getAttribute("id");
|
|
183
|
+
if (
|
|
184
|
+
id &&
|
|
185
|
+
!/^(bits|radix|headlessui|react-aria|ember|react-|:)/.test(id) &&
|
|
186
|
+
!/[-_]\d+$/.test(id) &&
|
|
187
|
+
uniq("#" + esc(id))
|
|
188
|
+
) {
|
|
189
|
+
out.push("#" + esc(id));
|
|
190
|
+
}
|
|
191
|
+
// 2. name attribute
|
|
192
|
+
const name = el.getAttribute("name");
|
|
193
|
+
if (name) {
|
|
194
|
+
const a = av(name);
|
|
195
|
+
if (a && uniq("[name=" + a + "]")) out.push("[name=" + a + "]");
|
|
196
|
+
}
|
|
197
|
+
// 3. ARIA label on the element itself
|
|
198
|
+
const aria = el.getAttribute("aria-label");
|
|
199
|
+
if (aria) {
|
|
200
|
+
const a = av(aria);
|
|
201
|
+
if (a && uniq(tag + "[aria-label=" + a + "]"))
|
|
202
|
+
out.push(tag + "[aria-label=" + a + "]");
|
|
203
|
+
}
|
|
204
|
+
// 4. stable data-* (test attrs preferred; state/animation attrs skipped)
|
|
205
|
+
const attrs = Array.prototype.slice.call(el.attributes) as Array<{
|
|
206
|
+
name: string;
|
|
207
|
+
value: string;
|
|
208
|
+
}>;
|
|
209
|
+
const dataAttrs = attrs.filter(
|
|
210
|
+
(a) =>
|
|
211
|
+
a.name.startsWith("data-") &&
|
|
212
|
+
!/^data-(state|orientation|index|selected|active|highlighted|value|radix|size|style|theme)/.test(
|
|
213
|
+
a.name,
|
|
214
|
+
),
|
|
215
|
+
);
|
|
216
|
+
dataAttrs.sort((a, b) => {
|
|
217
|
+
const score = (n: string) =>
|
|
218
|
+
/^data-(test|testid|cy)/.test(n) ? 0 : 1;
|
|
219
|
+
return score(a.name) - score(b.name);
|
|
220
|
+
});
|
|
221
|
+
for (const a of dataAttrs) {
|
|
222
|
+
const v = av(a.value);
|
|
223
|
+
if (!v) continue;
|
|
224
|
+
const s = tag + "[" + a.name + "=" + v + "]";
|
|
225
|
+
if (uniq(s)) {
|
|
226
|
+
out.push(s);
|
|
227
|
+
break;
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
// 5. placeholder (form controls with no name/id)
|
|
231
|
+
const ph = el.getAttribute("placeholder");
|
|
232
|
+
if (ph) {
|
|
233
|
+
const a = av(ph);
|
|
234
|
+
if (a && uniq(tag + "[placeholder=" + a + "]"))
|
|
235
|
+
out.push(tag + "[placeholder=" + a + "]");
|
|
236
|
+
}
|
|
237
|
+
// 6. recorded text (Playwright exact text engine)
|
|
238
|
+
const text = (el.textContent || "")
|
|
163
239
|
.trim()
|
|
164
240
|
.replace(/\s+/g, " ")
|
|
165
|
-
.slice(0,
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
const parts: string[] = [];
|
|
170
|
-
while (cur && parts.length < 6) {
|
|
171
|
-
parts.push(cur.tagName.toLowerCase());
|
|
172
|
-
cur = cur.parentElement;
|
|
241
|
+
.slice(0, 60);
|
|
242
|
+
if (text && (tag === "button" || tag === "a")) {
|
|
243
|
+
const a = av(text);
|
|
244
|
+
if (a) out.push("text=" + a);
|
|
173
245
|
}
|
|
174
|
-
return
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
.catch(
|
|
184
|
-
() =>
|
|
185
|
-
[] as Array<{
|
|
186
|
-
i: number;
|
|
187
|
-
role: string;
|
|
188
|
-
accessibleName: string;
|
|
189
|
-
textHash: string;
|
|
190
|
-
pathHash: string;
|
|
191
|
-
}>,
|
|
192
|
-
);
|
|
193
|
-
for (const e of els) {
|
|
194
|
-
const cand: Fingerprint = {
|
|
195
|
-
role: e.role,
|
|
196
|
-
accessibleName: e.accessibleName,
|
|
197
|
-
textHash: e.textHash,
|
|
198
|
-
box: { x: 0, y: 0, w: 0, h: 0 },
|
|
199
|
-
pathHash: e.pathHash,
|
|
200
|
-
};
|
|
201
|
-
if (looseMatch(expected, cand)) {
|
|
202
|
-
return page
|
|
203
|
-
.locator(
|
|
204
|
-
"button, a, input, select, textarea, [role=button], [tabindex]",
|
|
205
|
-
)
|
|
206
|
-
.nth(e.i);
|
|
207
|
-
}
|
|
208
|
-
}
|
|
209
|
-
for (const e of els) {
|
|
210
|
-
const cand: Fingerprint = {
|
|
211
|
-
role: e.role,
|
|
212
|
-
accessibleName: e.accessibleName,
|
|
213
|
-
textHash: e.textHash,
|
|
214
|
-
box: { x: 0, y: 0, w: 0, h: 0 },
|
|
215
|
-
pathHash: e.pathHash,
|
|
216
|
-
};
|
|
217
|
-
if (fuzzyMatch(expected, cand)) {
|
|
218
|
-
return page
|
|
219
|
-
.locator(
|
|
220
|
-
"button, a, input, select, textarea, [role=button], [tabindex]",
|
|
221
|
-
)
|
|
222
|
-
.nth(e.i);
|
|
246
|
+
return out;
|
|
247
|
+
})
|
|
248
|
+
.catch(() => [] as string[]);
|
|
249
|
+
for (const c of cands) {
|
|
250
|
+
const n = await page
|
|
251
|
+
.locator(c)
|
|
252
|
+
.count()
|
|
253
|
+
.catch(() => 0);
|
|
254
|
+
if (n === 1) return c;
|
|
223
255
|
}
|
|
256
|
+
return null;
|
|
257
|
+
} catch {
|
|
258
|
+
return null;
|
|
224
259
|
}
|
|
225
|
-
return null;
|
|
226
260
|
}
|
package/src/http.ts
CHANGED
|
@@ -11,7 +11,7 @@ const transport = new HttpTransport(server, { path: "/mcp" });
|
|
|
11
11
|
|
|
12
12
|
Bun.serve({
|
|
13
13
|
port,
|
|
14
|
-
hostname: "
|
|
14
|
+
hostname: "0.0.0.0",
|
|
15
15
|
// Bun closes idle connections after 10s by default (idleTimeout), and the
|
|
16
16
|
// timer applies even while a response is being streamed. The MCP
|
|
17
17
|
// Streamable-HTTP SSE notification stream sits idle between server->client
|
package/src/tools/act.ts
CHANGED
|
@@ -3,7 +3,7 @@ import { tool } from "tmcp/utils";
|
|
|
3
3
|
import * as v from "valibot";
|
|
4
4
|
import { loadConfig } from "../config.js";
|
|
5
5
|
import { getPage, getContext } from "../browser/session.js";
|
|
6
|
-
import { doAct, doBatch,
|
|
6
|
+
import { doAct, doBatch, type ActAction } from "../browser/engine.js";
|
|
7
7
|
import { EngineError } from "../browser/refs.js";
|
|
8
8
|
|
|
9
9
|
export const ACT_ACTIONS = [
|
|
@@ -101,10 +101,6 @@ export const actTool = defineTool(
|
|
|
101
101
|
JSON.stringify({ ok: true, action, url, title, ...result }),
|
|
102
102
|
);
|
|
103
103
|
}
|
|
104
|
-
if (action === "goal") {
|
|
105
|
-
const result = await doGoal(page, config, value ?? target ?? "", opts);
|
|
106
|
-
return tool.text(JSON.stringify({ ok: true, action, ...result }));
|
|
107
|
-
}
|
|
108
104
|
const result = await doAct(
|
|
109
105
|
page,
|
|
110
106
|
config,
|