effing-use 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -8,14 +8,14 @@ Full Chromium automation in **3 tools, 3.9 KB**. Same pages, same clicks, same s
8
8
 
9
9
  ## Install (Bun-only)
10
10
 
11
- Requires [Bun](https://bun.sh) 1.4+. Installs from npm in seconds — the tarball is ~16 kB ([`effing-use` v0.1.1](https://www.npmjs.com/package/effing-use)):
11
+ Requires [Bun](https://bun.sh) 1.4+. Installs from npm in seconds — the tarball is ~16 kB ([`effing-use` v0.2.1](https://www.npmjs.com/package/effing-use)):
12
12
 
13
13
  ```bash
14
14
  # No install needed — bunx fetches from npm on first run
15
- bunx effing-use # CLI (needs running server) — effing-use observe/act/extract
16
- bunx effing-use-http # HTTP MCP on :3123 (/mcp) — shared across editors
17
- bunx effing-use-stdio # STDIO MCP (single editor) — legacy bin
18
- bunx playwright install chromium --only-shell # one-time Chromium download (~150 MB)
15
+ bunx --package effing-use effing-use # CLI (needs running server)
16
+ bunx --package effing-use effing-use-http # HTTP MCP on :3123 (/mcp) — shared across editors
17
+ bunx --package effing-use effing-use-stdio # STDIO MCP (single editor)
18
+ bunx playwright install chromium --only-shell # one-time Chromium download (~150 MB)
19
19
  ```
20
20
 
21
21
  Optional — install globally so `effing-use` is on your PATH:
@@ -41,41 +41,74 @@ Re-run `bunx playwright install chromium --only-shell` whenever you bump the `pl
41
41
 
42
42
  **Why Chromium is separate:** Playwright supports multiple browsers and updates its pinned builds every release, so the binary can't live inside the npm tarball (ours is 16 kB). Docker users skip this — Chromium is baked into the `mcr.microsoft.com/playwright` base image.
43
43
 
44
+ ### npm vs source
45
+
46
+ | Source | Command | When to use |
47
+ | ------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------ |
48
+ | **npm** (`bunx --package effing-use`) | `effing-use`, `effing-use-http`, `effing-use-stdio` | Recommended — always matches published `package.json` version, no clone needed |
49
+ | **source** (`bun src/*.ts`) | `bun src/cli.ts`, `bun src/http.ts`, `bun src/index.ts` | Local dev / unreleased changes |
50
+
51
+ Both expose the same 3 bins: `effing-use` = CLI (HTTP client), `effing-use-stdio` = MCP stdio, `effing-use-http` = MCP http. Don't mix them — `effing-use` alone is **not** an MCP server (it prints CLI help and exits, causing `Failed to parse message` / `MCP server has stopped`).
52
+
44
53
  ## Run it in 60 seconds (recommended path)
45
54
 
46
55
  **Step 1/3 — Start the server.** One server, every local editor:
47
56
 
48
57
  ```bash
58
+ # Docker (recommended — Chromium baked in, no local install)
59
+ docker compose up --build -d
60
+ curl http://localhost:3123/healthz # {"ok":true,"name":"effing-use"}
61
+
62
+ # Or without Docker
49
63
  bun install
50
64
  bunx playwright install chromium --only-shell
51
- docker compose up --build -d
52
- curl http://localhost:3123/healthz
65
+ bun src/http.ts # or: bunx --package effing-use effing-use-http
66
+ ```
67
+
68
+ The HTTP server binds `0.0.0.0:3123` inside Docker (so forwarded ports work) and sets `idleTimeout: 0` so the MCP SSE stream isn't killed after 10s of idle time. Plain HTTP on loopback is intentional — add TLS at the edge for remote use.
69
+
70
+ **Step 2/3 — Connect.** Pick one transport:
71
+
72
+ **HTTP (shared — recommended for multiple VS Code windows):**
73
+
74
+ ```json
75
+ // ~/.config/Code/User/mcp.json (global, all workspaces) or .vscode/mcp.json
76
+ {
77
+ "servers": {
78
+ "effing-use": { "type": "http", "url": "http://localhost:3123/mcp" }
79
+ }
80
+ }
53
81
  ```
54
82
 
55
- Point any VS Code instance at `http://localhost:3123/mcp` (see `.vscode/mcp.json` → `effing-use (http)`). Plain HTTP on loopback is intentional — add TLS at the edge for remote use.
83
+ One Docker/bun process serves every window. Use same `sessionId` to share tabs, different `sessionId` to isolate.
56
84
 
57
- **Step 2/3 — Connect.** STDIO for one editor, HTTP for all of them:
85
+ **STDIO (isolated — one browser per window):**
58
86
 
59
87
  ```json
60
88
  {
61
- "mcpServers": {
89
+ "servers": {
62
90
  "effing-use": {
91
+ "type": "stdio",
63
92
  "command": "bunx",
64
- "args": ["effing-use-stdio"]
93
+ "args": ["--package", "effing-use", "effing-use-stdio"],
94
+ "env": {
95
+ "BROWSER_HEADLESS": "true",
96
+ "OUTPUT_DIR": "${workspaceFolder}/.browser-use"
97
+ }
65
98
  }
66
99
  }
67
100
  }
68
101
  ```
69
102
 
70
- Or HTTP: `http://localhost:3123/mcp` (via `bunx effing-use-http` or Docker). From source instead: `command: "bun"`, `args: ["/path/to/effing-use/src/index.ts"]`.
103
+ From source: `command: "bun"`, `args: ["/absolute/path/to/effing-use/src/index.ts"]`.
71
104
 
72
105
  **CLI vs MCP — when to use which:**
73
106
 
74
- | Surface | Command | Needs server? | Best for |
75
- |---------|---------|---------------|----------|
76
- | **MCP (stdio)** | `effing-use-stdio` / `bun src/index.ts` | No (spawns own browser) | Single VS Code / Claude Desktop |
77
- | **MCP (http)** | `effing-use-http` / `bun src/http.ts` | Yes (`:3123`) | Shared across editors, Docker |
78
- | **CLI** | `effing-use` / `bun src/cli.ts` | Yes (`:3123`, `EFFING_USE_URL`) | Terminal agents, scripts, CI |
107
+ | Surface | Command | Needs server? | Best for |
108
+ | --------------- | --------------------------------------- | ------------------------------- | ------------------------------- |
109
+ | **MCP (stdio)** | `effing-use-stdio` / `bun src/index.ts` | No (spawns own browser) | Single VS Code / Claude Desktop |
110
+ | **MCP (http)** | `effing-use-http` / `bun src/http.ts` | Yes (`:3123`) | Shared across editors, Docker |
111
+ | **CLI** | `effing-use` / `bun src/cli.ts` | Yes (`:3123`, `EFFING_USE_URL`) | Terminal agents, scripts, CI |
79
112
 
80
113
  The CLI is a thin HTTP client over the same engine — `effing-use observe --kind snapshot --mode delta` and `browser_observe kind=snapshot mode=delta` hit the same code. Start the server once (`bun src/http.ts` or `docker compose up`), then use either face.
81
114
 
@@ -94,6 +127,46 @@ The CLI is a thin HTTP client over the same engine — `effing-use observe --kin
94
127
 
95
128
  No Docker? `bun src/cli.ts --help` (CLI), `bun src/index.ts` (STDIO MCP), or `bun src/http.ts` (`:3123` `/mcp`) works directly.
96
129
 
130
+ ### CLI usage
131
+
132
+ The CLI needs a running HTTP server (`EFFING_USE_URL` defaults to `http://localhost:3123/mcp`):
133
+
134
+ ```bash
135
+ # Observe
136
+ effing-use observe --kind snapshot --mode delta --session dev
137
+ effing-use observe --kind title --session dev
138
+ effing-use observe --kind screenshot --session dev
139
+
140
+ # Act
141
+ effing-use act --action open --value https://example.com --session dev
142
+ effing-use act --action click --target e5 --expect 'url~/dashboard' --session dev
143
+ effing-use act --action batch --file steps.json --session dev
144
+
145
+ # Extract
146
+ effing-use extract --kind text --selector main --session dev
147
+ effing-use extract --kind state --session dev
148
+
149
+ # With custom server URL
150
+ EFFING_USE_URL=http://localhost:3123/mcp effing-use observe --kind snapshot
151
+ ```
152
+
153
+ From npm without global install: `bunx --package effing-use effing-use observe --kind snapshot`.
154
+
155
+ ### Docker
156
+
157
+ ```bash
158
+ docker compose up --build -d # build + start (0.0.0.0:3123, idleTimeout:0)
159
+ docker compose logs -f effing-use # tail logs
160
+ curl http://localhost:3123/healthz # health check
161
+ docker compose down # stop (add -v to remove volumes)
162
+ # Rebuild after pulling new code
163
+ docker compose up --build -d
164
+ # Orphaned old container holding :3123? (renamed service)
165
+ docker rm -f effing-use-computer-use-1 && docker compose up -d
166
+ ```
167
+
168
+ Compose details: `restart: unless-stopped`, `init: true` + `ipc: host` (Playwright flags), `.browser-use/` bind-mounted, `EFFING_PORT` overrides host port (`EFFING_PORT=4000 docker compose up -d`).
169
+
97
170
  ## Why agents prefer 3 tools
98
171
 
99
172
  - 🪶 **Tiny handshake, full surface** — `browser_act` (32 actions), `browser_observe` (8 kinds + `mode`/`scope`), `browser_extract` (8 kinds incl. `state`). No schema bloat, no guessing which of 24 tools to call.
@@ -102,6 +175,36 @@ No Docker? `bun src/cli.ts --help` (CLI), `bun src/index.ts` (STDIO MCP), or `bu
102
175
  - ⚡ **One call, not five** — `batch` runs fill+press flows in one turn (max 20 steps, stops on first error). `goal` plans or returns `E_GOAL_UNCLEAR` + `suggestedSteps` instead of hallucinating.
103
176
  - 🐳 **Shared, not spawned** — one Docker server serves every local VS Code instance. No per-client `npx` spawn.
104
177
 
178
+ ## Harness — verify, delta, state, replay
179
+
180
+ v2 turns the browser into a harness: AI actions are verifiable, diffable, and replayable — not just fire-and-forget clicks.
181
+
182
+ **1. Verify — fingerprints + expectations + failure contract**
183
+
184
+ - **Fingerprint registry** (`src/browser/identity.ts`): every `eN` ref is fingerprinted (role, accessibleName, textHash, box, pathHash). Stale refs rebind only on an UNAMBIGUOUS identity match (`rebound:true`); if several elements match equally they fail with `E_STALE` + hint — the engine never guesses a target.
185
+ - **Expect mini-language** (`expect` on any `browser_act`): `url~/dashboard` | `text~/Saved/` | `visible=.modal` | `gone=.spinner` — evaluated server-side, returns `E_EXPECT` / `E_BAD_EXPECT` on mismatch instead of hallucinated success. ReDoS-capped and regex-validated.
186
+ - **Failure contract**: after an uncertain mutation the engine sets `mustObserve:true` — next mutation fails with `E_MUST_OBSERVE` until you re-snapshot. No blind chains.
187
+ - **Evidence envelope**: every `browser_act` returns `effect: { urlChanged, urlBefore/After, domChanged, consoleErrors, networkFailures }` capped at `EFFECT_MAX_CHARS` (800) so the agent sees what actually happened.
188
+
189
+ **2. Delta — pay only for what changed**
190
+
191
+ - `browser_observe kind=snapshot mode=delta` (default) — MutationObserver dirty flag + baseline diff. Returns only `[changed]` lines or `unchanged:true` on stable pages (~90% token saving). `mode=full` for complete dump, `scope="<css>"` for subtree.
192
+ - `DELTA_DEFAULT=true` — flip to `false` to default to full snapshots.
193
+
194
+ **3. State — per-session memory**
195
+
196
+ - `browser_act action=note value="..."` appends to `notes` (capped `STATE_MAX_LINES=40`), `browser_extract kind=state` reads `notes` + `lastActions` ring (last 10). Persisted to `.browser-use/state/<session>.md` so agents survive context compaction.
197
+
198
+ **4. Record → Compile → Replay — deterministic macros**
199
+
200
+ - `record_start` / `record_stop` captures every step with resolved selectors + fingerprints. Secrets auto-redacted (`RECORD_REDACT=true`, `«redacted»` for password/otp/token fields).
201
+ - `compile` generates `.browser-use/macros/<name>.{ts,md}` — a Playwright `run(page)` function + a `SKILL.md` doc. Irreversible steps (`submit`/`pay`/`delete`/etc.) are flagged `requiresApproval`.
202
+ - `replay` replays deterministically; pauses with `E_APPROVAL_REQUIRED` until `approve:true` if any irreversible step exists.
203
+
204
+ **5. CLI — same engine, no MCP client**
205
+
206
+ - `effing-use observe/act/extract` over `EFFING_USE_URL` (`http://localhost:3123/mcp`) — for terminal agents, scripts, and CI. See [CLI usage](#cli-usage).
207
+
105
208
  ## The loop (agents: follow this order)
106
209
 
107
210
  1. `browser_observe` kind=`snapshot` → get `[eN]` refs (default `mode: delta` — only changed lines; `mode: full` for complete dump; `scope: "<css>"` for subtree). Never guess refs, re-snapshot after navigation.
@@ -130,18 +233,27 @@ Errors are always `{ ok: false, code, message, hint }` with `E_NOT_FOUND | E_TIM
130
233
 
131
234
  Need Firefox/WebKit, device emulation, or persistent profiles? Use Playwright MCP. Need context budget for Chromium work? Stay here. Full breakdown in [`docs/COMPARISON.md`](docs/COMPARISON.md).
132
235
 
133
- ## Record → replay
236
+ ## Record → replay (harness)
134
237
 
135
238
  ```bash
136
- # MCP
239
+ # MCP — capture any flow, compile to code + skill, replay deterministically
137
240
  browser_act action=record_start value=my-flow
138
- # ... do the flow ...
139
- browser_act action=record_stop
140
- browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
141
- browser_act action=replay value=my-flow # pauses with E_APPROVAL_REQUIRED if irreversible
142
- browser_act action=replay value=my-flow approve=true
241
+ # ... do the flow (clicks, fills, etc.) ...
242
+ browser_act action=record_stop # -> .browser-use/recordings/my-flow.json (secrets redacted)
243
+ browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
244
+ browser_act action=replay value=my-flow # pauses with E_APPROVAL_REQUIRED if irreversible
245
+ browser_act action=replay value=my-flow approve=true # replay with approval
246
+
247
+ # CLI — same flow over HTTP
248
+ EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action record_start --value my-flow
249
+ # ... do the flow via CLI or MCP ...
250
+ EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action record_stop
251
+ EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action compile --value my-flow
252
+ EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action replay --value my-flow --approve true
143
253
  ```
144
254
 
255
+ Artifacts: `recordings/*.json` (raw steps), `macros/*.ts` (Playwright `run(page)`), `macros/*.md` (skill doc). Redaction and approval gates are on by default (`RECORD_REDACT=true`).
256
+
145
257
  ## Config
146
258
 
147
259
  Env defaults in [`.env.example`](.env.example): `BROWSER_HEADLESS`, `BROWSER_VIEWPORT_W/H`, `BROWSER_TIMEOUT_MS`, `OUTPUT_DIR` (`.browser-use/`, gitignored), `OUTPUT_MAX_CHARS`, `ALLOW_EVAL`, `DELTA_DEFAULT`, `EFFECT_MAX_CHARS`, `STATE_MAX_LINES`, `RECORD_REDACT`, `EFFING_USE_URL`.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "effing-use",
3
- "version": "0.2.0",
3
+ "version": "0.2.1",
4
4
  "description": "Token-efficient browser control: 3 tools (act, observe, extract) built with tmcp + Bun + Playwright",
5
5
  "license": "MIT",
6
6
  "author": "Michael Obubelebra Amachree",
@@ -15,14 +15,14 @@ cost than 21-tool browser servers. `tools/list` is ~3.9KB.
15
15
 
16
16
  ## The loop (always follow this order)
17
17
 
18
- 1. `browser_observe` with `kind: "snapshot"` → get `[eN]` refs. Default is `mode: "delta"` (only changes since last observe); use `mode: "full"` for a complete dump. Use `scope: "<css>"` to observe a subtree.
18
+ 1. `browser_observe` with `kind: "snapshot"` → get `[eN]` refs. Default is `mode: "delta"` (`[changed]`/`[removed]` lines since last observe; `unchanged:true` when the page is quiet — navigation automatically forces a fresh `mode: "full"`). Use `mode: "full"` for a complete dump, `scope: "<css>"` to observe a subtree.
19
19
  2. `browser_act` to interact (`open`, `click`, `fill`, `type`, `press`, `select`, `check`, `wait`, …). Add `expect: "url~/dashboard"` or `text~/Saved/` for deterministic post-conditions.
20
- 3. `browser_observe` with `kind: "snapshot"` again — delta returns `unchanged:true` if nothing changed, or `[changed]` lines.
20
+ 3. `browser_observe` with `kind: "snapshot"` again — delta returns `unchanged:true` if nothing changed, or `[changed]`/`[removed]` lines.
21
21
  4. `browser_extract` with `kind: "text" | "table" | "query" | "state"` → scrape or read task state.
22
22
 
23
23
  Rules:
24
24
 
25
- - Never guess refs. Re-snapshot after every navigation. Stale refs return `E_STALE` or auto-rebind with `rebound:true`.
25
+ - Never guess refs. Re-snapshot after every navigation (refs are page-keyed — a cross-page stale ref returns `E_NOT_FOUND`). On a same-page re-render the engine rebinds only an UNAMBIGUOUS identity match (result carries `rebound:true`); if the fingerprint matches several elements it fails with `E_STALE` instead of guessing — re-observe.
26
26
  - Prefer `batch`: one `browser_act` with `steps[]` for fill+press flows (max 20 steps, stops on first error).
27
27
  - Large outputs are files under `.browser-use/` (gitignored). Read the path, not the preview.
28
28
  - Snapshots are capped at `OUTPUT_MAX_CHARS` (default 4000) with `…[truncated N chars, see file]`.
@@ -37,8 +37,8 @@ Rules:
37
37
  `scroll` (`up`/`down`/`top`/`bottom` or a target), `back`/`forward`/`reload`,
38
38
  `wait` (`ms:500`, `text:Saved`, or a ref), `dialog_accept`/`dialog_dismiss` (arm before the triggering step),
39
39
  `resize` (`1280x800` in `value`), `tab_new`/`tab_select`/`tab_close`, `close`,
40
- `goal` (deterministic add-todo/search planner, else `E_GOAL_UNCLEAR` + `suggestedSteps`),
41
- `batch` (needs `steps[]`), `note` (append to task state), `record_start`/`record_stop` (capture flow), `compile` (macro+SKILL.md), `replay` (deterministic replay, needs `approve:true` for irreversible steps).
40
+ `goal` (deterministic add-todo/search planner, else `E_GOAL_UNCLEAR` + `suggestedSteps`; counts as a mutation — subject to the `mustObserve` guard, evidence, and recording),
41
+ `batch` (needs `steps[]`), `note` (append to task state), `record_start`/`record_stop` (capture flow), `compile` (standalone `.ts` + SKILL.md; selectors resolved id → name → ARIA → data-\* → placeholder → text), `replay` (deterministic replay; runs benign steps, halts AT approval-gated steps).
42
42
  `expect` mini-language: `url~<regex>` | `text~<regex>` | `visible=<css>` | `gone=<css>` — evaluated in code, returns `E_EXPECT` or `E_BAD_EXPECT`.
43
43
 
44
44
  **browser_observe** — read-only: `snapshot` (e-refs, `mode: full|delta` default delta, `scope: <css>`), `screenshot` (file path,
@@ -70,8 +70,11 @@ browser_act action=record_start value=my-flow
70
70
  # ... do the flow ...
71
71
  browser_act action=record_stop
72
72
  browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
73
- browser_act action=replay value=my-flow # pauses with E_APPROVAL_REQUIRED if irreversible
74
- browser_act action=replay value=my-flow approve=true
73
+ browser_act action=replay value=my-flow # runs benign steps, halts AT the first
74
+ # approval-gated step → E_APPROVAL_REQUIRED
75
+ # { step: N (1-indexed), completed: C }
76
+ browser_act action=replay value=my-flow approve=true # resumes at the gate; the
77
+ # benign prefix is NOT re-run (cursor)
75
78
  ```
76
79
 
77
80
  ## Setup
@@ -44,6 +44,11 @@ export function getBaseline(sessionId: string): string | null {
44
44
  return ensureState(sessionId).baseline;
45
45
  }
46
46
 
47
+ /** URL the current baseline was captured on — forces mode:full after navigation (plan §5.1). */
48
+ export function getBaselineUrl(sessionId: string): string | null {
49
+ return pageStates.get(key(sessionId))?.baselineUrl ?? null;
50
+ }
51
+
47
52
  export function isDirty(sessionId: string): boolean {
48
53
  return ensureState(sessionId).dirty;
49
54
  }
@@ -66,8 +71,32 @@ export async function injectDirtyObserver(
66
71
  w.__effDirty = false;
67
72
  w.__effSid = sid;
68
73
  if (w.__effObs) w.__effObs.disconnect();
69
- const obs = new MutationObserver(() => {
70
- w.__effDirty = true;
74
+ // Attribute spam that never changes the snapshot's ref lines (class,
75
+ // style, expand/animation state) used to mark every live SPA permanently
76
+ // dirty — the cheap "unchanged" fast path never fired. Ignore those;
77
+ // content changes still arrive via childList/characterData/input/change.
78
+ const NOISE_ATTRS = [
79
+ "class",
80
+ "style",
81
+ "aria-expanded",
82
+ "data-state",
83
+ "data-orientation",
84
+ "data-scroll-state",
85
+ "data-highlighted",
86
+ "data-hovered",
87
+ "data-dragging",
88
+ "data-resizing",
89
+ "data-index",
90
+ ];
91
+ const obs = new MutationObserver((muts) => {
92
+ for (const m of muts) {
93
+ if (m.type === "attributes") {
94
+ const name = m.attributeName || "";
95
+ if (NOISE_ATTRS.indexOf(name) !== -1) continue;
96
+ }
97
+ w.__effDirty = true;
98
+ return;
99
+ }
71
100
  });
72
101
  obs.observe(document.documentElement, {
73
102
  childList: true,
@@ -127,10 +156,18 @@ export function computeDelta(
127
156
  const baseLines = baseline.split("\n");
128
157
  const curLines = current.split("\n");
129
158
  const baseSet = new Set(baseLines);
159
+ const curSet = new Set(curLines);
130
160
  const changed = curLines.filter((l) => !baseSet.has(l));
161
+ const removed = baseLines.filter((l) => !curSet.has(l));
162
+ if (changed.length === 0 && removed.length === 0)
163
+ return { delta: "", unchanged: true };
131
164
  const unchangedCount = curLines.length - changed.length;
132
- if (changed.length === 0) return { delta: "", unchanged: true };
133
165
  const header = `…${unchangedCount} unchanged lines…`;
134
- const delta = [header, ...changed.map((l) => `[changed] ${l}`)].join("\n");
166
+ const delta = [
167
+ header,
168
+ ...changed.map((l) => `[changed] ${l}`),
169
+ // plan §5.1: removed nodes are reported explicitly
170
+ ...removed.map((l) => `[removed] ${l}`),
171
+ ].join("\n");
135
172
  return { delta, unchanged: false };
136
173
  }
@@ -1,6 +1,6 @@
1
1
  import type { Page, BrowserContext } from "playwright";
2
2
  import type { Config } from "../config.js";
3
- import { EngineError, resolveLocator } from "./refs.js";
3
+ import { EngineError, resolveLocator, stableSelectorFor } from "./refs.js";
4
4
  import { cap, saveText, stamp } from "./output.js";
5
5
  import {
6
6
  getConsoleLogs,
@@ -13,6 +13,7 @@ import {
13
13
  import { collectEffect, evaluateExpect, parseExpect } from "./evidence.js";
14
14
  import {
15
15
  getBaseline,
16
+ getBaselineUrl,
16
17
  setBaseline,
17
18
  computeDelta,
18
19
  injectDirtyObserver,
@@ -28,9 +29,17 @@ import {
28
29
  loadRecording,
29
30
  startRecording,
30
31
  stopRecording,
32
+ getReplayCursor,
33
+ setReplayCursor,
34
+ clearReplayCursor,
31
35
  } from "./record.js";
32
36
  import { compileMacro, isIrreversible } from "./macro.js";
33
- import { registerFingerprints, clearRegistry } from "./identity.js";
37
+ import {
38
+ registerFingerprints,
39
+ clearRegistry,
40
+ getFingerprint,
41
+ extractFingerprints as collectFingerprints,
42
+ } from "./identity.js";
34
43
 
35
44
  export type ActAction =
36
45
  | "open"
@@ -85,6 +94,19 @@ function timeoutOf(config: Config): number {
85
94
  async function liteState(page: Page): Promise<{ url: string; title: string }> {
86
95
  return { url: page.url(), title: await page.title().catch(() => "") };
87
96
  }
97
+
98
+ /**
99
+ * SPA navigations commit after the action returns (client router awaits data),
100
+ * so evidence collected immediately can report urlChanged:false for a click
101
+ * that DOES navigate (observed: kikitai "Try paste & read" → /read).
102
+ * Brief settle: one tick, then a second if the URL moved.
103
+ */
104
+ async function settleSpa(page: Page, action: string): Promise<void> {
105
+ if (!["click", "dblclick", "press", "goal"].includes(action)) return;
106
+ const u0 = page.url();
107
+ await page.waitForTimeout(60);
108
+ if (page.url() !== u0) await page.waitForTimeout(60);
109
+ }
88
110
  function toEngineError(e: unknown, fallbackHint: string): EngineError {
89
111
  if (e instanceof EngineError) return e;
90
112
  const msg = e instanceof Error ? e.message : String(e);
@@ -116,6 +138,7 @@ const MUTATING = new Set<string>([
116
138
  "open",
117
139
  "goto",
118
140
  "resize",
141
+ "goal",
119
142
  "tab_new",
120
143
  "tab_select",
121
144
  "tab_close",
@@ -136,6 +159,10 @@ export async function doAct(
136
159
  const timeout = timeoutOf(config);
137
160
  const sid = opts?.sessionId ?? "default";
138
161
 
162
+ // Per-action stash must never leak from a previous (possibly failed) act
163
+ const pageBag = page as unknown as Record<string, unknown>;
164
+ delete pageBag.__rebound;
165
+
139
166
  // Non-page meta actions (no mustObserve guard, no evidence)
140
167
  if (action === "note") {
141
168
  if (!value && !target)
@@ -209,24 +236,28 @@ export async function doAct(
209
236
  );
210
237
  });
211
238
  const approve = opts?.approve === true;
212
- // Check approval gate
213
- const needsApproval = (rec.steps as any[]).filter((s: any) =>
214
- isIrreversible(s),
215
- );
216
- if (needsApproval.length > 0 && !approve) {
217
- const first = needsApproval[0];
218
- return {
219
- ok: false,
220
- code: "E_APPROVAL_REQUIRED",
221
- message: `Step ${first.seq} requires approval (${first.op}).`,
222
- hint: "Re-run with approve:true.",
223
- step: first.seq,
224
- description: `${first.op} ${first.target ?? ""}`,
225
- } as any;
226
- }
227
- // Deterministic replay
239
+ // Plan §6.3: run benign steps, pause AT each approval-gated step, and keep
240
+ // a cursor so approve:true resumes here instead of re-running the prefix.
241
+ const stepsArr = rec.steps as any[];
242
+ const start = getReplayCursor(sid, name) ?? 0;
228
243
  const results: any[] = [];
229
- for (const s of rec.steps as any[]) {
244
+ for (let i = start; i < stepsArr.length; i++) {
245
+ const s = stepsArr[i];
246
+ if (isIrreversible(s) && !approve) {
247
+ setReplayCursor(sid, name, i);
248
+ const ran = i - start;
249
+ return {
250
+ ok: false,
251
+ code: "E_APPROVAL_REQUIRED",
252
+ message:
253
+ `Step ${i + 1} requires approval (${s.op}).` +
254
+ (ran > 0 ? ` Ran ${ran} prior step(s).` : ""),
255
+ hint: "Re-run with approve:true to resume from this step.",
256
+ step: i + 1,
257
+ completed: results.length,
258
+ description: `${s.op} ${s.target ?? ""}`,
259
+ } as any;
260
+ }
230
261
  try {
231
262
  const r = await doAct(
232
263
  page,
@@ -238,6 +269,7 @@ export async function doAct(
238
269
  );
239
270
  results.push({ ok: true, seq: s.seq, ...r });
240
271
  } catch (e) {
272
+ clearReplayCursor(sid, name);
241
273
  const err = toEngineError(e, "Replay failed.");
242
274
  results.push({
243
275
  ok: false,
@@ -248,7 +280,14 @@ export async function doAct(
248
280
  return { replayed: false, name, failSeq: s.seq, results } as any;
249
281
  }
250
282
  }
251
- return { replayed: true, name, steps: results.length, results } as any;
283
+ clearReplayCursor(sid, name);
284
+ return {
285
+ replayed: true,
286
+ name,
287
+ steps: results.length,
288
+ resumedFrom: start,
289
+ results,
290
+ } as any;
252
291
  }
253
292
 
254
293
  // Failure contract guard
@@ -275,8 +314,10 @@ export async function doAct(
275
314
  const urlBefore = page.url();
276
315
  let domBefore: string | null = null;
277
316
  try {
317
+ // Full innerText (50k cap) — evidence diffs need the whole page, not just
318
+ // the first 200 chars (v0.2: below-fold changes looked like no-ops).
278
319
  domBefore = await page.evaluate(
279
- () => document.body?.innerText?.slice(0, 200) ?? "",
320
+ () => document.body?.innerText?.slice(0, 50000) ?? "",
280
321
  );
281
322
  } catch {
282
323
  domBefore = null;
@@ -579,7 +620,12 @@ export async function doAct(
579
620
  result = { closed: true };
580
621
  break;
581
622
  case "goal":
582
- return doGoal(page, config, value ?? target ?? "", opts);
623
+ // Route through the normal post-action pipeline (guard, evidence,
624
+ // dirty flag, recording) — v0.2 returned here and skipped all of it.
625
+ result = {
626
+ ...(await doGoal(page, config, value ?? target ?? "", opts)),
627
+ };
628
+ break;
583
629
  case "batch":
584
630
  throw new EngineError(
585
631
  "E_BAD_INPUT",
@@ -608,6 +654,7 @@ export async function doAct(
608
654
  // Post-action: evidence, expect, state, dirty, recording
609
655
  let effect: any = undefined;
610
656
  if (isMutating(action)) {
657
+ await settleSpa(page, action);
611
658
  effect = await collectEffect(
612
659
  page,
613
660
  sid,
@@ -657,8 +704,15 @@ export async function doAct(
657
704
  `${actionLine} -> ${effect ? JSON.stringify(effect).slice(0, 80) : "ok"}`,
658
705
  );
659
706
 
660
- // Recording capture
707
+ // Recording capture — resolve a portable selector (id → name → ARIA →
708
+ // data-* → placeholder → text) so compiled macros don't depend on e-refs.
661
709
  if (isRecording(sid) && isMutating(action)) {
710
+ let resolved = target;
711
+ if (target) {
712
+ resolved =
713
+ (await stableSelectorFor(page, target, sid).catch(() => null)) ??
714
+ target;
715
+ }
662
716
  captureStep(
663
717
  sid,
664
718
  {
@@ -666,7 +720,8 @@ export async function doAct(
666
720
  target,
667
721
  value,
668
722
  expect: opts?.expect,
669
- resolvedSelector: target,
723
+ resolvedSelector: resolved,
724
+ targetFingerprint: target ? getFingerprint(sid, target) : undefined,
670
725
  },
671
726
  config,
672
727
  );
@@ -795,9 +850,22 @@ export async function doObserve(
795
850
  | "full"
796
851
  | "delta";
797
852
  const scope = opts?.scope;
853
+ // Navigation forces a fresh full baseline (plan §5.1) and needs a settle
854
+ // beat: SPA routes hydrate after the URL changes — snapshots taken too
855
+ // early miss the header/nav entirely (observed on kikitai /read).
856
+ const prevUrl = getBaselineUrl(sessionId);
857
+ const urlChanged = prevUrl !== null && prevUrl !== page.url();
858
+ const settleHydration = async () => {
859
+ if (!urlChanged) return;
860
+ await page
861
+ .waitForLoadState("networkidle", { timeout: 1500 })
862
+ .catch(() => {});
863
+ await page.waitForTimeout(80);
864
+ };
798
865
  // Scoped snapshot: only within selector
799
866
  let yaml: string;
800
867
  if (scope) {
868
+ await settleHydration();
801
869
  yaml = await buildSnapshot(page, scope);
802
870
  const path = await saveText(config, stamp("snapshot", "yaml"), yaml);
803
871
  const { text, truncated } = cap(yaml, config.outputMaxChars);
@@ -815,8 +883,9 @@ export async function doObserve(
815
883
  // Delta path
816
884
  if (mode === "delta") {
817
885
  const baseline = getBaseline(sessionId);
818
- // If no baseline, do full
819
- if (!baseline) {
886
+ // No baseline, or the page navigated since the baseline → forced full
887
+ if (!baseline || urlChanged) {
888
+ await settleHydration();
820
889
  yaml = await buildSnapshot(page);
821
890
  setBaseline(sessionId, yaml, page.url());
822
891
  await injectDirtyObserver(page, sessionId);
@@ -879,6 +948,7 @@ export async function doObserve(
879
948
  };
880
949
  }
881
950
  // Full mode
951
+ await settleHydration();
882
952
  yaml = await buildSnapshot(page);
883
953
  setBaseline(sessionId, yaml, page.url());
884
954
  await injectDirtyObserver(page, sessionId);
@@ -960,10 +1030,28 @@ async function buildSnapshot(page: Page, scope?: string): Promise<string> {
960
1030
  }
961
1031
  return els.slice(0, 200).map((el) => {
962
1032
  const tag = el.tagName.toLowerCase();
963
- const text = (el.textContent ?? "")
1033
+ let text = (el.textContent ?? "")
964
1034
  .trim()
965
1035
  .replace(/\s+/g, " ")
966
1036
  .slice(0, 80);
1037
+ // Form controls carry no text — surface placeholder/label so refs stop
1038
+ // showing as `input ""`, and mark checked state (real state changes that
1039
+ // delta should surface).
1040
+ const input = el as HTMLInputElement;
1041
+ if (!text) {
1042
+ text = (
1043
+ input.placeholder ||
1044
+ (input.labels && input.labels[0]?.textContent) ||
1045
+ ""
1046
+ )
1047
+ .trim()
1048
+ .replace(/\s+/g, " ")
1049
+ .slice(0, 80);
1050
+ }
1051
+ const checked =
1052
+ (tag === "input" || tag === "select") && input.checked
1053
+ ? " [checked]"
1054
+ : "";
967
1055
  const aria = el.getAttribute("aria-label") ?? "";
968
1056
  const name = el.getAttribute("name") ?? "";
969
1057
  const id = el.getAttribute("id") ?? "";
@@ -974,64 +1062,13 @@ async function buildSnapshot(page: Page, scope?: string): Promise<string> {
974
1062
  ]
975
1063
  .filter(Boolean)
976
1064
  .join(" ");
977
- return `${tag} "${text}"${extra ? " " + extra : ""}`;
1065
+ return `${tag} "${text}"${extra ? " " + extra : ""}${checked}`;
978
1066
  });
979
1067
  }, scope ?? null);
980
1068
  const lines = items.map((line, i) => `[e${i}] ${line}`);
981
1069
  return `${header}\n${lines.join("\n") || "(no interactive elements)"}`;
982
1070
  }
983
1071
 
984
- async function collectFingerprints(
985
- page: Page,
986
- ): Promise<Array<{ ref: string; fp: import("./identity.js").Fingerprint }>> {
987
- const raw = await page
988
- .evaluate(() => {
989
- const els = [
990
- ...document.querySelectorAll(
991
- "button, a, input, select, textarea, [role=button], [tabindex]",
992
- ),
993
- ].slice(0, 200);
994
- return els.map((el) => {
995
- const text = (el.textContent ?? "")
996
- .trim()
997
- .replace(/\s+/g, " ")
998
- .slice(0, 80);
999
- const aria = el.getAttribute("aria-label") ?? "";
1000
- const role = el.getAttribute("role") ?? el.tagName.toLowerCase();
1001
- let cur: Element | null = el;
1002
- const parts: string[] = [];
1003
- while (cur && parts.length < 6) {
1004
- parts.push(cur.tagName.toLowerCase());
1005
- cur = cur.parentElement;
1006
- }
1007
- const rect = el.getBoundingClientRect();
1008
- return {
1009
- role,
1010
- accessibleName: aria,
1011
- text,
1012
- pathHash: parts.join(">"),
1013
- box: { x: rect.x, y: rect.y, w: rect.width, h: rect.height },
1014
- };
1015
- });
1016
- })
1017
- .catch(() => [] as any[]);
1018
- return raw.map((r: any, i: number) => ({
1019
- ref: `e${i}`,
1020
- fp: {
1021
- role: r.role || "generic",
1022
- accessibleName: (r.accessibleName || "").trim().slice(0, 80),
1023
- textHash: (r.text || "").trim().replace(/\s+/g, " ").slice(0, 80),
1024
- box: {
1025
- x: Math.round(r.box.x),
1026
- y: Math.round(r.box.y),
1027
- w: Math.round(r.box.w),
1028
- h: Math.round(r.box.h),
1029
- },
1030
- pathHash: r.pathHash || "",
1031
- },
1032
- }));
1033
- }
1034
-
1035
1072
  // ---------------------------------------------------------------- extract
1036
1073
 
1037
1074
  export async function doExtract(
@@ -46,16 +46,35 @@ export async function collectEffect(
46
46
  const urlAfter = page.url();
47
47
  const urlChanged = urlBefore !== urlAfter;
48
48
 
49
- // domChanged: try to get diff via delta module if available, else simple heuristic
49
+ // domChanged: line-level diff of body.innerText (50k cap). v0.2 only
50
+ // compared the first 200 chars, so any below-fold change looked like a no-op
51
+ // (and fed the failure contract wrong "no DOM change" signals).
50
52
  let domChanged: string[] = [];
51
53
  try {
52
54
  const currentDom = await page.evaluate(
53
- () => document.body?.innerText?.slice(0, 200) ?? "",
55
+ () => document.body?.innerText?.slice(0, 50000) ?? "",
54
56
  );
55
57
  if (domBeforeHash !== null && currentDom !== domBeforeHash) {
56
- domChanged = [
57
- currentDom.slice(0, 120).replace(/\s+/g, " ").trim(),
58
- ].filter(Boolean);
58
+ const before = new Set(
59
+ domBeforeHash
60
+ .split("\n")
61
+ .map((l) => l.trim())
62
+ .filter(Boolean),
63
+ );
64
+ const after = currentDom
65
+ .split("\n")
66
+ .map((l) => l.trim())
67
+ .filter(Boolean);
68
+ const afterSet = new Set(after);
69
+ const changed = after
70
+ .filter((l) => !before.has(l))
71
+ .slice(0, 5)
72
+ .map((l) => l.slice(0, 140));
73
+ const removed = [...before]
74
+ .filter((l) => !afterSet.has(l))
75
+ .slice(0, 3)
76
+ .map((l) => `- ${l.slice(0, 138)}`);
77
+ domChanged = [...changed, ...removed].slice(0, 5);
59
78
  }
60
79
  } catch {
61
80
  // ignore
@@ -96,9 +115,12 @@ export function parseExpect(
96
115
  const s = expect.trim();
97
116
  let parsed: { kind: string; pattern: string } | null = null;
98
117
  if (s.startsWith("url~")) parsed = { kind: "url", pattern: s.slice(4) };
99
- else if (s.startsWith("text~")) parsed = { kind: "text", pattern: s.slice(5) };
100
- else if (s.startsWith("visible=")) parsed = { kind: "visible", pattern: s.slice(8) };
101
- else if (s.startsWith("gone=")) parsed = { kind: "gone", pattern: s.slice(5) };
118
+ else if (s.startsWith("text~"))
119
+ parsed = { kind: "text", pattern: s.slice(5) };
120
+ else if (s.startsWith("visible="))
121
+ parsed = { kind: "visible", pattern: s.slice(8) };
122
+ else if (s.startsWith("gone="))
123
+ parsed = { kind: "gone", pattern: s.slice(5) };
102
124
  else return null;
103
125
  if (parsed.pattern.length > EXPECT_MAX_PATTERN) return null;
104
126
  // Validate regex patterns early for url~/text~
@@ -1,3 +1,5 @@
1
+ import type { Page } from "playwright";
2
+
1
3
  export type Fingerprint = {
2
4
  role: string;
3
5
  accessibleName: string;
@@ -6,6 +8,114 @@ export type Fingerprint = {
6
8
  pathHash: string;
7
9
  };
8
10
 
11
+ /** Raw per-element data as extracted inside the page (before normalization). */
12
+ export type RawFp = {
13
+ role: string;
14
+ accessibleName: string;
15
+ text: string;
16
+ pathHash: string;
17
+ box: { x: number; y: number; w: number; h: number };
18
+ };
19
+
20
+ /**
21
+ * Browser-side fingerprint extractor — SINGLE SOURCE OF TRUTH shared by
22
+ * snapshot registration, stale-ref validation and rebind search. Serialized
23
+ * into the page as a string so it never closes over module scope.
24
+ *
25
+ * v0.2 bug fixed here: fingerprints used tag-name roles ("input"), aria-label
26
+ * only, and own textContent (always "" for form controls), so every bare
27
+ * checkbox shared one identity — rebind then clicked the wrong element
28
+ * (TodoMVC: stale item toggle rebound to toggle-all). Now:
29
+ * - role: implicit ARIA role (input[type=checkbox] → "checkbox")
30
+ * - accessibleName: aria-label → <label> → alt → title → placeholder → text
31
+ * - textHash: own text, else contextual text (nearest li/label/td/row), else
32
+ * placeholder — so item toggles carry their todo text
33
+ */
34
+ export const RAW_FP_SCRIPT = `(() => {
35
+ const SEL = "button, a, input, select, textarea, [role=button], [tabindex]";
36
+ const norm = (s) => (s || "").trim().replace(/\\s+/g, " ").slice(0, 80);
37
+ const implicitRole = (el) => {
38
+ const r = el.getAttribute("role");
39
+ if (r) return r;
40
+ const tag = el.tagName.toLowerCase();
41
+ if (tag === "a") return el.hasAttribute("href") ? "link" : "generic";
42
+ if (tag === "button") return "button";
43
+ if (tag === "textarea") return "textbox";
44
+ if (tag === "select") return "combobox";
45
+ if (tag === "summary") return "button";
46
+ if (tag === "input") {
47
+ const t = (el.getAttribute("type") || "text").toLowerCase();
48
+ if (t === "checkbox") return "checkbox";
49
+ if (t === "radio") return "radio";
50
+ if (t === "submit" || t === "button" || t === "reset" || t === "image") return "button";
51
+ if (t === "range") return "slider";
52
+ if (t === "hidden") return "hidden";
53
+ return "textbox";
54
+ }
55
+ return tag;
56
+ };
57
+ const accName = (el) => {
58
+ const aria = el.getAttribute("aria-label");
59
+ if (aria && aria.trim()) return norm(aria);
60
+ if (el.labels && el.labels.length > 0) {
61
+ const t = norm(el.labels[0].textContent);
62
+ if (t) return t;
63
+ }
64
+ const alt = el.getAttribute("alt");
65
+ if (alt && alt.trim()) return norm(alt);
66
+ const title = el.getAttribute("title");
67
+ if (title && title.trim()) return norm(title);
68
+ const ph = el.getAttribute("placeholder");
69
+ if (ph && ph.trim()) return norm(ph);
70
+ return norm(el.textContent);
71
+ };
72
+ const textOf = (el) => {
73
+ const own = norm(el.textContent);
74
+ if (own) return own;
75
+ const ctx = el.closest && el.closest("li, label, td, [role=listitem], [role=row]");
76
+ if (ctx) {
77
+ const t = norm(ctx.textContent);
78
+ if (t) return t;
79
+ }
80
+ return norm(el.getAttribute("placeholder"));
81
+ };
82
+ const pathOf = (el) => {
83
+ const parts = [];
84
+ let cur = el;
85
+ while (cur && parts.length < 6) {
86
+ parts.push(cur.tagName.toLowerCase());
87
+ cur = cur.parentElement;
88
+ }
89
+ return parts.join(">");
90
+ };
91
+ const els = Array.prototype.slice.call(document.querySelectorAll(SEL), 0, 200);
92
+ return els.map((el) => {
93
+ const rect = el.getBoundingClientRect();
94
+ return {
95
+ role: implicitRole(el),
96
+ accessibleName: accName(el),
97
+ text: textOf(el),
98
+ pathHash: pathOf(el),
99
+ box: {
100
+ x: Math.round(rect.x),
101
+ y: Math.round(rect.y),
102
+ w: Math.round(rect.width),
103
+ h: Math.round(rect.height)
104
+ }
105
+ };
106
+ });
107
+ })()`;
108
+
109
+ /** Extract fingerprints for every snapshot-visible element, as e-refs. */
110
+ export async function extractFingerprints(
111
+ page: Page,
112
+ ): Promise<Array<{ ref: string; fp: Fingerprint }>> {
113
+ const raw = (await page
114
+ .evaluate(RAW_FP_SCRIPT)
115
+ .catch(() => [] as RawFp[])) as RawFp[];
116
+ return raw.map((r, i) => ({ ref: `e${i}`, fp: buildFingerprint(r) }));
117
+ }
118
+
9
119
  function hashText(s: string): string {
10
120
  return s.trim().replace(/\s+/g, " ").slice(0, 80);
11
121
  }
@@ -41,6 +151,38 @@ export function clearRegistry(sessionId: string): void {
41
151
  registry.delete(sessionId);
42
152
  }
43
153
 
154
+ export type RebindCandidate = { index: number; fp: Fingerprint };
155
+
156
+ /**
157
+ * Conservative rebind ladder (P0 fix): pick an element only when the match is
158
+ * UNAMBIGUOUS — never guess among equals.
159
+ *
160
+ * 1. loose match (role + accessibleName + textHash) — unique → take it
161
+ * 2. multiple loose → narrow by identical pathHash; still >1 → give up
162
+ * 3. no loose → fuzzy (role + name/text contains) under the same rules
163
+ *
164
+ * Returns the chosen candidate index, or null when zero or more than one
165
+ * candidate survives — callers turn null into E_STALE (fail loud) instead of
166
+ * clicking the wrong element.
167
+ */
168
+ export function chooseRebindIndex(
169
+ expected: Fingerprint,
170
+ candidates: RebindCandidate[],
171
+ ): number | null {
172
+ const narrow = (pool: RebindCandidate[]): number | null => {
173
+ if (pool.length === 1) return pool[0].index;
174
+ if (pool.length > 1 && expected.pathHash) {
175
+ const byPath = pool.filter((c) => c.fp.pathHash === expected.pathHash);
176
+ if (byPath.length === 1) return byPath[0].index;
177
+ }
178
+ return null;
179
+ };
180
+ const loose = candidates.filter((c) => looseMatch(expected, c.fp));
181
+ if (loose.length > 0) return narrow(loose);
182
+ const fuzzy = candidates.filter((c) => fuzzyMatch(expected, c.fp));
183
+ return narrow(fuzzy);
184
+ }
185
+
44
186
  export function fingerprintEquals(a: Fingerprint, b: Fingerprint): boolean {
45
187
  return (
46
188
  a.role === b.role &&
@@ -32,8 +32,11 @@ export async function compileMacro(
32
32
  const warnings: string[] = [];
33
33
  const irreversibleSteps = steps.filter(isIrreversible);
34
34
  if (irreversibleSteps.length > 0) {
35
+ // Steps are human-facing here — 1-indexed, matching the .md step list
35
36
  warnings.push(
36
- `${irreversibleSteps.length} step(s) require approval: ${irreversibleSteps.map((s) => s.seq).join(", ")}`,
37
+ `${irreversibleSteps.length} step(s) require approval: steps ${irreversibleSteps
38
+ .map((s) => s.seq + 1)
39
+ .join(", ")}`,
37
40
  );
38
41
  }
39
42
 
@@ -49,12 +52,21 @@ export async function compileMacro(
49
52
  const sel = selectorFor(s);
50
53
  const val = (s.value ?? "").slice(0, 500);
51
54
  const flag = isIrreversible(s) ? " // requiresApproval" : "";
55
+ // Recordings made before stable-selector resolution store session-scoped
56
+ // e-refs — flag them so nobody runs the .ts standalone expecting it to work.
57
+ const staleNote = /^e\d+$/.test(sel)
58
+ ? " /* session-scoped e-ref — not a standalone selector */"
59
+ : "";
52
60
  switch (s.op) {
53
61
  case "click":
54
- tsLines.push(` await page.locator(${JSON.stringify(sel)}).click();${flag}`);
62
+ tsLines.push(
63
+ ` await page.locator(${JSON.stringify(sel)}).click();${flag}${staleNote}`,
64
+ );
55
65
  break;
56
66
  case "fill":
57
- tsLines.push(` await page.locator(${JSON.stringify(sel)}).fill(${JSON.stringify(val)});${flag}`);
67
+ tsLines.push(
68
+ ` await page.locator(${JSON.stringify(sel)}).fill(${JSON.stringify(val)});${flag}${staleNote}`,
69
+ );
58
70
  break;
59
71
  case "press":
60
72
  tsLines.push(
@@ -63,10 +75,14 @@ export async function compileMacro(
63
75
  break;
64
76
  case "goto":
65
77
  case "open":
66
- tsLines.push(` await page.goto(${JSON.stringify(val || sel)});${flag}`);
78
+ tsLines.push(
79
+ ` await page.goto(${JSON.stringify(val || sel)});${flag}`,
80
+ );
67
81
  break;
68
82
  default:
69
- tsLines.push(` // ${JSON.stringify(s.op)} ${JSON.stringify(sel)} ${JSON.stringify(val)}${flag}`);
83
+ tsLines.push(
84
+ ` // ${JSON.stringify(s.op)} ${JSON.stringify(sel)} ${JSON.stringify(val)}${flag}`,
85
+ );
70
86
  break;
71
87
  }
72
88
  }
@@ -97,7 +97,11 @@ export async function saveRecording(
97
97
  const dir = join(config.outputDir, "recordings");
98
98
  await mkdir(dir, { recursive: true });
99
99
  const path = join(dir, `${safe}.json`);
100
- const rec: Recording = { name: safe, createdAt: new Date().toISOString(), steps };
100
+ const rec: Recording = {
101
+ name: safe,
102
+ createdAt: new Date().toISOString(),
103
+ steps,
104
+ };
101
105
  await writeFile(path, JSON.stringify(rec, null, 2), "utf-8");
102
106
  return path;
103
107
  }
@@ -115,3 +119,31 @@ export async function loadRecording(
115
119
  export function getActiveName(sessionId: string): string | null {
116
120
  return activeRecordings.get(sessionId)?.name ?? null;
117
121
  }
122
+
123
+ // ---------------------------------------------------------------- replay cursor
124
+ // Plan §6.3: pause AT an approval-gated step and resume with approve:true —
125
+ // the cursor keeps the benign prefix from running twice (non-idempotent steps).
126
+ const replayCursors = new Map<string, number>();
127
+
128
+ function cursorKey(sessionId: string, name: string): string {
129
+ return `${sessionId}:${name}`;
130
+ }
131
+
132
+ export function getReplayCursor(
133
+ sessionId: string,
134
+ name: string,
135
+ ): number | undefined {
136
+ return replayCursors.get(cursorKey(sessionId, name));
137
+ }
138
+
139
+ export function setReplayCursor(
140
+ sessionId: string,
141
+ name: string,
142
+ step: number,
143
+ ): void {
144
+ replayCursors.set(cursorKey(sessionId, name), step);
145
+ }
146
+
147
+ export function clearReplayCursor(sessionId: string, name: string): void {
148
+ replayCursors.delete(cursorKey(sessionId, name));
149
+ }
@@ -1,9 +1,12 @@
1
1
  import type { Page, Locator } from "playwright";
2
2
  import {
3
3
  getFingerprint,
4
- looseMatch,
5
- fuzzyMatch,
4
+ buildFingerprint,
5
+ chooseRebindIndex,
6
+ RAW_FP_SCRIPT,
6
7
  type Fingerprint,
8
+ type RawFp,
9
+ type RebindCandidate,
7
10
  } from "./identity.js";
8
11
 
9
12
  export class EngineError extends Error {
@@ -52,15 +55,20 @@ export async function resolveLocator(
52
55
  if (sessionId) {
53
56
  const expected = getFingerprint(sessionId, t);
54
57
  if (expected) {
55
- const actual = await fingerprintAt(page, loc).catch(() => null);
58
+ // Validate against the SAME extractor that registered the snapshot
59
+ const raws = (await page.evaluate(RAW_FP_SCRIPT).catch(() => null)) as
60
+ | RawFp[]
61
+ | null;
62
+ const idx = Number(t.slice(1));
63
+ const actual = raws && raws[idx] ? buildFingerprint(raws[idx]) : null;
56
64
  if (actual) {
57
65
  const exact =
58
66
  expected.role === actual.role &&
59
67
  expected.accessibleName === actual.accessibleName &&
60
68
  expected.textHash === actual.textHash;
61
69
  if (!exact) {
62
- // Try rebind: search tree for loose/fuzzy match
63
- const rebound = await findRebound(page, expected);
70
+ // Conservative rebind — ambiguous → E_STALE, never guess
71
+ const rebound = await findRebound(page, expected, raws);
64
72
  if (rebound) {
65
73
  // Return rebound locator; caller can check rebound flag via session
66
74
  (page as unknown as Record<string, unknown>).__rebound = true;
@@ -110,117 +118,143 @@ export async function resolveLocator(
110
118
  );
111
119
  }
112
120
 
113
- async function fingerprintAt(page: Page, loc: Locator): Promise<Fingerprint> {
114
- const box = await loc.boundingBox().catch(() => null);
115
- const data = await loc
116
- .evaluate((el) => {
117
- const text = (el.textContent ?? "").trim().slice(0, 80);
118
- const aria = el.getAttribute("aria-label") ?? "";
119
- const role = el.getAttribute("role") ?? el.tagName.toLowerCase();
120
- // simple path hash: tag chain
121
- let cur: Element | null = el;
122
- const parts: string[] = [];
123
- while (cur && parts.length < 6) {
124
- parts.push(cur.tagName.toLowerCase());
125
- cur = cur.parentElement;
126
- }
127
- return { role, accessibleName: aria, text, pathHash: parts.join(">") };
128
- })
129
- .catch(() => ({
130
- role: "generic",
131
- accessibleName: "",
132
- text: "",
133
- pathHash: "",
134
- }));
135
- return {
136
- role: data.role || "generic",
137
- accessibleName: (data.accessibleName || "").trim().slice(0, 80),
138
- textHash: (data.text || "").trim().replace(/\s+/g, " ").slice(0, 80),
139
- box: {
140
- x: Math.round(box?.x ?? 0),
141
- y: Math.round(box?.y ?? 0),
142
- w: Math.round(box?.width ?? 0),
143
- h: Math.round(box?.height ?? 0),
144
- },
145
- pathHash: data.pathHash || "",
146
- };
147
- }
148
-
121
+ /**
122
+ * Find where the element `expected` used to describe lives now.
123
+ * Uses the shared extractor + conservative chooser: only an UNAMBIGUOUS match
124
+ * is returned; zero or several equally-good candidates → null → E_STALE
125
+ * upstream (v0.2 bug: first-match rebind clicked toggle-all).
126
+ */
149
127
  async function findRebound(
150
128
  page: Page,
151
129
  expected: Fingerprint,
130
+ raws: RawFp[] | null,
152
131
  ): Promise<Locator | null> {
153
- // Scan interactive elements for loose/fuzzy match
154
- const els = await page
155
- .evaluate(() => {
156
- const nodes = [
157
- ...document.querySelectorAll(
158
- "button, a, input, select, textarea, [role=button], [tabindex]",
159
- ),
160
- ];
161
- return nodes.slice(0, 200).map((el, i) => {
162
- const text = (el.textContent ?? "")
132
+ const list =
133
+ raws ?? ((await page.evaluate(RAW_FP_SCRIPT).catch(() => [])) as RawFp[]);
134
+ const candidates: RebindCandidate[] = list.map((r, i) => ({
135
+ index: i,
136
+ fp: buildFingerprint(r),
137
+ }));
138
+ const pick = chooseRebindIndex(expected, candidates);
139
+ if (pick === null) return null;
140
+ return page.locator(INTERACTIVE_SELECTOR).nth(pick);
141
+ }
142
+
143
+ /**
144
+ * Resolve a portable, standalone selector for a target (plan §6.2 preference:
145
+ * id → name → ARIA label → stable data-* → placeholder → recorded text).
146
+ * Never returns nth-index or session-scoped e-refs; null when nothing unique
147
+ * can be found (caller falls back to the raw target).
148
+ */
149
+ export async function stableSelectorFor(
150
+ page: Page,
151
+ target: string,
152
+ sessionId?: string,
153
+ ): Promise<string | null> {
154
+ try {
155
+ const loc = await resolveLocator(page, target, sessionId);
156
+ const cands = await loc
157
+ .first()
158
+ .evaluate((el) => {
159
+ const out: string[] = [];
160
+ const tag = el.tagName.toLowerCase();
161
+ const esc = (s: string) => {
162
+ const c = (
163
+ globalThis as unknown as {
164
+ CSS?: { escape?: (x: string) => string };
165
+ }
166
+ ).CSS;
167
+ return c && c.escape ? c.escape(s) : s;
168
+ };
169
+ const av = (s: string): string | null => {
170
+ const bs = String.fromCharCode(92);
171
+ if (s.indexOf('"') !== -1 || s.indexOf(bs) !== -1) return null;
172
+ return '"' + s + '"';
173
+ };
174
+ const uniq = (s: string): boolean => {
175
+ try {
176
+ return document.querySelectorAll(s).length === 1;
177
+ } catch {
178
+ return false;
179
+ }
180
+ };
181
+ // 1. stable id (skip framework-generated/dynamic ids)
182
+ const id = el.getAttribute("id");
183
+ if (
184
+ id &&
185
+ !/^(bits|radix|headlessui|react-aria|ember|react-|:)/.test(id) &&
186
+ !/[-_]\d+$/.test(id) &&
187
+ uniq("#" + esc(id))
188
+ ) {
189
+ out.push("#" + esc(id));
190
+ }
191
+ // 2. name attribute
192
+ const name = el.getAttribute("name");
193
+ if (name) {
194
+ const a = av(name);
195
+ if (a && uniq("[name=" + a + "]")) out.push("[name=" + a + "]");
196
+ }
197
+ // 3. ARIA label on the element itself
198
+ const aria = el.getAttribute("aria-label");
199
+ if (aria) {
200
+ const a = av(aria);
201
+ if (a && uniq(tag + "[aria-label=" + a + "]"))
202
+ out.push(tag + "[aria-label=" + a + "]");
203
+ }
204
+ // 4. stable data-* (test attrs preferred; state/animation attrs skipped)
205
+ const attrs = Array.prototype.slice.call(el.attributes) as Array<{
206
+ name: string;
207
+ value: string;
208
+ }>;
209
+ const dataAttrs = attrs.filter(
210
+ (a) =>
211
+ a.name.startsWith("data-") &&
212
+ !/^data-(state|orientation|index|selected|active|highlighted|value|radix|size|style|theme)/.test(
213
+ a.name,
214
+ ),
215
+ );
216
+ dataAttrs.sort((a, b) => {
217
+ const score = (n: string) =>
218
+ /^data-(test|testid|cy)/.test(n) ? 0 : 1;
219
+ return score(a.name) - score(b.name);
220
+ });
221
+ for (const a of dataAttrs) {
222
+ const v = av(a.value);
223
+ if (!v) continue;
224
+ const s = tag + "[" + a.name + "=" + v + "]";
225
+ if (uniq(s)) {
226
+ out.push(s);
227
+ break;
228
+ }
229
+ }
230
+ // 5. placeholder (form controls with no name/id)
231
+ const ph = el.getAttribute("placeholder");
232
+ if (ph) {
233
+ const a = av(ph);
234
+ if (a && uniq(tag + "[placeholder=" + a + "]"))
235
+ out.push(tag + "[placeholder=" + a + "]");
236
+ }
237
+ // 6. recorded text (Playwright exact text engine)
238
+ const text = (el.textContent || "")
163
239
  .trim()
164
240
  .replace(/\s+/g, " ")
165
- .slice(0, 80);
166
- const aria = el.getAttribute("aria-label") ?? "";
167
- const role = el.getAttribute("role") ?? el.tagName.toLowerCase();
168
- let cur: Element | null = el;
169
- const parts: string[] = [];
170
- while (cur && parts.length < 6) {
171
- parts.push(cur.tagName.toLowerCase());
172
- cur = cur.parentElement;
241
+ .slice(0, 60);
242
+ if (text && (tag === "button" || tag === "a")) {
243
+ const a = av(text);
244
+ if (a) out.push("text=" + a);
173
245
  }
174
- return {
175
- i,
176
- role,
177
- accessibleName: aria,
178
- textHash: text,
179
- pathHash: parts.join(">"),
180
- };
181
- });
182
- })
183
- .catch(
184
- () =>
185
- [] as Array<{
186
- i: number;
187
- role: string;
188
- accessibleName: string;
189
- textHash: string;
190
- pathHash: string;
191
- }>,
192
- );
193
- for (const e of els) {
194
- const cand: Fingerprint = {
195
- role: e.role,
196
- accessibleName: e.accessibleName,
197
- textHash: e.textHash,
198
- box: { x: 0, y: 0, w: 0, h: 0 },
199
- pathHash: e.pathHash,
200
- };
201
- if (looseMatch(expected, cand)) {
202
- return page
203
- .locator(
204
- "button, a, input, select, textarea, [role=button], [tabindex]",
205
- )
206
- .nth(e.i);
207
- }
208
- }
209
- for (const e of els) {
210
- const cand: Fingerprint = {
211
- role: e.role,
212
- accessibleName: e.accessibleName,
213
- textHash: e.textHash,
214
- box: { x: 0, y: 0, w: 0, h: 0 },
215
- pathHash: e.pathHash,
216
- };
217
- if (fuzzyMatch(expected, cand)) {
218
- return page
219
- .locator(
220
- "button, a, input, select, textarea, [role=button], [tabindex]",
221
- )
222
- .nth(e.i);
246
+ return out;
247
+ })
248
+ .catch(() => [] as string[]);
249
+ for (const c of cands) {
250
+ const n = await page
251
+ .locator(c)
252
+ .count()
253
+ .catch(() => 0);
254
+ if (n === 1) return c;
223
255
  }
256
+ return null;
257
+ } catch {
258
+ return null;
224
259
  }
225
- return null;
226
260
  }
package/src/http.ts CHANGED
@@ -11,7 +11,7 @@ const transport = new HttpTransport(server, { path: "/mcp" });
11
11
 
12
12
  Bun.serve({
13
13
  port,
14
- hostname: "127.0.0.1",
14
+ hostname: "0.0.0.0",
15
15
  // Bun closes idle connections after 10s by default (idleTimeout), and the
16
16
  // timer applies even while a response is being streamed. The MCP
17
17
  // Streamable-HTTP SSE notification stream sits idle between server->client
package/src/tools/act.ts CHANGED
@@ -3,7 +3,7 @@ import { tool } from "tmcp/utils";
3
3
  import * as v from "valibot";
4
4
  import { loadConfig } from "../config.js";
5
5
  import { getPage, getContext } from "../browser/session.js";
6
- import { doAct, doBatch, doGoal, type ActAction } from "../browser/engine.js";
6
+ import { doAct, doBatch, type ActAction } from "../browser/engine.js";
7
7
  import { EngineError } from "../browser/refs.js";
8
8
 
9
9
  export const ACT_ACTIONS = [
@@ -101,10 +101,6 @@ export const actTool = defineTool(
101
101
  JSON.stringify({ ok: true, action, url, title, ...result }),
102
102
  );
103
103
  }
104
- if (action === "goal") {
105
- const result = await doGoal(page, config, value ?? target ?? "", opts);
106
- return tool.text(JSON.stringify({ ok: true, action, ...result }));
107
- }
108
104
  const result = await doAct(
109
105
  page,
110
106
  config,