effing-use 0.1.1 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +165 -26
- package/package.json +4 -3
- package/skills/effing-use/SKILL.md +38 -11
- package/src/browser/delta.ts +173 -0
- package/src/browser/engine.ts +581 -58
- package/src/browser/evidence.ts +208 -0
- package/src/browser/identity.ts +237 -0
- package/src/browser/macro.ts +118 -0
- package/src/browser/record.ts +149 -0
- package/src/browser/refs.ts +192 -7
- package/src/browser/session.ts +16 -0
- package/src/browser/state.ts +100 -0
- package/src/cli.ts +173 -0
- package/src/config.ts +10 -0
- package/src/http.ts +1 -0
- package/src/tools/act.ts +10 -7
- package/src/tools/extract.ts +1 -0
- package/src/tools/observe.ts +5 -1
package/README.md
CHANGED
|
@@ -8,21 +8,23 @@ Full Chromium automation in **3 tools, 3.9 KB**. Same pages, same clicks, same s
|
|
|
8
8
|
|
|
9
9
|
## Install (Bun-only)
|
|
10
10
|
|
|
11
|
-
Requires [Bun](https://bun.sh) 1.4+. Installs from npm in seconds — the tarball is ~16 kB ([`effing-use` v0.
|
|
11
|
+
Requires [Bun](https://bun.sh) 1.4+. Installs from npm in seconds — the tarball is ~16 kB ([`effing-use` v0.2.1](https://www.npmjs.com/package/effing-use)):
|
|
12
12
|
|
|
13
13
|
```bash
|
|
14
14
|
# No install needed — bunx fetches from npm on first run
|
|
15
|
-
bunx effing-use #
|
|
16
|
-
bunx effing-use-http # HTTP on :3123 (/mcp) — shared across editors
|
|
17
|
-
bunx
|
|
15
|
+
bunx --package effing-use effing-use # CLI (needs running server)
|
|
16
|
+
bunx --package effing-use effing-use-http # HTTP MCP on :3123 (/mcp) — shared across editors
|
|
17
|
+
bunx --package effing-use effing-use-stdio # STDIO MCP (single editor)
|
|
18
|
+
bunx playwright install chromium --only-shell # one-time Chromium download (~150 MB)
|
|
18
19
|
```
|
|
19
20
|
|
|
20
21
|
Optional — install globally so `effing-use` is on your PATH:
|
|
21
22
|
|
|
22
23
|
```bash
|
|
23
24
|
bun install -g effing-use
|
|
24
|
-
effing-use
|
|
25
|
-
effing-use-http # HTTP on :3123 (/mcp)
|
|
25
|
+
effing-use --help # CLI
|
|
26
|
+
effing-use-http # HTTP MCP on :3123 (/mcp)
|
|
27
|
+
effing-use-stdio # STDIO MCP
|
|
26
28
|
```
|
|
27
29
|
|
|
28
30
|
The `playwright` npm package ships the driver, **not** the browser — every user needs Playwright's version-pinned Chromium once. From source instead:
|
|
@@ -30,41 +32,85 @@ The `playwright` npm package ships the driver, **not** the browser — every use
|
|
|
30
32
|
```bash
|
|
31
33
|
bun install
|
|
32
34
|
bunx playwright install chromium --only-shell
|
|
33
|
-
bun src/
|
|
34
|
-
bun src/http.ts # HTTP on :3123 (/mcp)
|
|
35
|
+
bun src/cli.ts --help # CLI (needs running server)
|
|
36
|
+
bun src/http.ts # HTTP MCP on :3123 (/mcp)
|
|
37
|
+
bun src/index.ts # STDIO MCP
|
|
35
38
|
```
|
|
36
39
|
|
|
37
40
|
Re-run `bunx playwright install chromium --only-shell` whenever you bump the `playwright` dependency (each Playwright version pins its own browser build). Verify with `bunx playwright install --dry-run chromium` or check `~/.cache/ms-playwright/`.
|
|
38
41
|
|
|
39
42
|
**Why Chromium is separate:** Playwright supports multiple browsers and updates its pinned builds every release, so the binary can't live inside the npm tarball (ours is 16 kB). Docker users skip this — Chromium is baked into the `mcr.microsoft.com/playwright` base image.
|
|
40
43
|
|
|
44
|
+
### npm vs source
|
|
45
|
+
|
|
46
|
+
| Source | Command | When to use |
|
|
47
|
+
| ------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
|
48
|
+
| **npm** (`bunx --package effing-use`) | `effing-use`, `effing-use-http`, `effing-use-stdio` | Recommended — always matches published `package.json` version, no clone needed |
|
|
49
|
+
| **source** (`bun src/*.ts`) | `bun src/cli.ts`, `bun src/http.ts`, `bun src/index.ts` | Local dev / unreleased changes |
|
|
50
|
+
|
|
51
|
+
Both expose the same 3 bins: `effing-use` = CLI (HTTP client), `effing-use-stdio` = MCP stdio, `effing-use-http` = MCP http. Don't mix them — `effing-use` alone is **not** an MCP server (it prints CLI help and exits, causing `Failed to parse message` / `MCP server has stopped`).
|
|
52
|
+
|
|
41
53
|
## Run it in 60 seconds (recommended path)
|
|
42
54
|
|
|
43
55
|
**Step 1/3 — Start the server.** One server, every local editor:
|
|
44
56
|
|
|
45
57
|
```bash
|
|
58
|
+
# Docker (recommended — Chromium baked in, no local install)
|
|
59
|
+
docker compose up --build -d
|
|
60
|
+
curl http://localhost:3123/healthz # {"ok":true,"name":"effing-use"}
|
|
61
|
+
|
|
62
|
+
# Or without Docker
|
|
46
63
|
bun install
|
|
47
64
|
bunx playwright install chromium --only-shell
|
|
48
|
-
|
|
49
|
-
|
|
65
|
+
bun src/http.ts # or: bunx --package effing-use effing-use-http
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
The HTTP server binds `0.0.0.0:3123` inside Docker (so forwarded ports work) and sets `idleTimeout: 0` so the MCP SSE stream isn't killed after 10s of idle time. Plain HTTP on loopback is intentional — add TLS at the edge for remote use.
|
|
69
|
+
|
|
70
|
+
**Step 2/3 — Connect.** Pick one transport:
|
|
71
|
+
|
|
72
|
+
**HTTP (shared — recommended for multiple VS Code windows):**
|
|
73
|
+
|
|
74
|
+
```json
|
|
75
|
+
// ~/.config/Code/User/mcp.json (global, all workspaces) or .vscode/mcp.json
|
|
76
|
+
{
|
|
77
|
+
"servers": {
|
|
78
|
+
"effing-use": { "type": "http", "url": "http://localhost:3123/mcp" }
|
|
79
|
+
}
|
|
80
|
+
}
|
|
50
81
|
```
|
|
51
82
|
|
|
52
|
-
|
|
83
|
+
One Docker/bun process serves every window. Use same `sessionId` to share tabs, different `sessionId` to isolate.
|
|
53
84
|
|
|
54
|
-
**
|
|
85
|
+
**STDIO (isolated — one browser per window):**
|
|
55
86
|
|
|
56
87
|
```json
|
|
57
88
|
{
|
|
58
|
-
"
|
|
89
|
+
"servers": {
|
|
59
90
|
"effing-use": {
|
|
91
|
+
"type": "stdio",
|
|
60
92
|
"command": "bunx",
|
|
61
|
-
"args": ["effing-use"]
|
|
93
|
+
"args": ["--package", "effing-use", "effing-use-stdio"],
|
|
94
|
+
"env": {
|
|
95
|
+
"BROWSER_HEADLESS": "true",
|
|
96
|
+
"OUTPUT_DIR": "${workspaceFolder}/.browser-use"
|
|
97
|
+
}
|
|
62
98
|
}
|
|
63
99
|
}
|
|
64
100
|
}
|
|
65
101
|
```
|
|
66
102
|
|
|
67
|
-
|
|
103
|
+
From source: `command: "bun"`, `args: ["/absolute/path/to/effing-use/src/index.ts"]`.
|
|
104
|
+
|
|
105
|
+
**CLI vs MCP — when to use which:**
|
|
106
|
+
|
|
107
|
+
| Surface | Command | Needs server? | Best for |
|
|
108
|
+
| --------------- | --------------------------------------- | ------------------------------- | ------------------------------- |
|
|
109
|
+
| **MCP (stdio)** | `effing-use-stdio` / `bun src/index.ts` | No (spawns own browser) | Single VS Code / Claude Desktop |
|
|
110
|
+
| **MCP (http)** | `effing-use-http` / `bun src/http.ts` | Yes (`:3123`) | Shared across editors, Docker |
|
|
111
|
+
| **CLI** | `effing-use` / `bun src/cli.ts` | Yes (`:3123`, `EFFING_USE_URL`) | Terminal agents, scripts, CI |
|
|
112
|
+
|
|
113
|
+
The CLI is a thin HTTP client over the same engine — `effing-use observe --kind snapshot --mode delta` and `browser_observe kind=snapshot mode=delta` hit the same code. Start the server once (`bun src/http.ts` or `docker compose up`), then use either face.
|
|
68
114
|
|
|
69
115
|
**Step 3/3 — Drive.** Search Wikipedia in one call instead of two round-trips:
|
|
70
116
|
|
|
@@ -79,30 +125,102 @@ Or HTTP: `http://localhost:3123/mcp` (via `bunx effing-use-http` or Docker). Fro
|
|
|
79
125
|
}
|
|
80
126
|
```
|
|
81
127
|
|
|
82
|
-
No Docker? `bun src/index.ts` (STDIO) or `bun src/http.ts` (`:3123` `/mcp`) works directly.
|
|
128
|
+
No Docker? `bun src/cli.ts --help` (CLI), `bun src/index.ts` (STDIO MCP), or `bun src/http.ts` (`:3123` `/mcp`) works directly.
|
|
129
|
+
|
|
130
|
+
### CLI usage
|
|
131
|
+
|
|
132
|
+
The CLI needs a running HTTP server (`EFFING_USE_URL` defaults to `http://localhost:3123/mcp`):
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
# Observe
|
|
136
|
+
effing-use observe --kind snapshot --mode delta --session dev
|
|
137
|
+
effing-use observe --kind title --session dev
|
|
138
|
+
effing-use observe --kind screenshot --session dev
|
|
139
|
+
|
|
140
|
+
# Act
|
|
141
|
+
effing-use act --action open --value https://example.com --session dev
|
|
142
|
+
effing-use act --action click --target e5 --expect 'url~/dashboard' --session dev
|
|
143
|
+
effing-use act --action batch --file steps.json --session dev
|
|
144
|
+
|
|
145
|
+
# Extract
|
|
146
|
+
effing-use extract --kind text --selector main --session dev
|
|
147
|
+
effing-use extract --kind state --session dev
|
|
148
|
+
|
|
149
|
+
# With custom server URL
|
|
150
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use observe --kind snapshot
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
From npm without global install: `bunx --package effing-use effing-use observe --kind snapshot`.
|
|
154
|
+
|
|
155
|
+
### Docker
|
|
156
|
+
|
|
157
|
+
```bash
|
|
158
|
+
docker compose up --build -d # build + start (0.0.0.0:3123, idleTimeout:0)
|
|
159
|
+
docker compose logs -f effing-use # tail logs
|
|
160
|
+
curl http://localhost:3123/healthz # health check
|
|
161
|
+
docker compose down # stop (add -v to remove volumes)
|
|
162
|
+
# Rebuild after pulling new code
|
|
163
|
+
docker compose up --build -d
|
|
164
|
+
# Orphaned old container holding :3123? (renamed service)
|
|
165
|
+
docker rm -f effing-use-computer-use-1 && docker compose up -d
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Compose details: `restart: unless-stopped`, `init: true` + `ipc: host` (Playwright flags), `.browser-use/` bind-mounted, `EFFING_PORT` overrides host port (`EFFING_PORT=4000 docker compose up -d`).
|
|
83
169
|
|
|
84
170
|
## Why agents prefer 3 tools
|
|
85
171
|
|
|
86
|
-
- 🪶 **Tiny handshake, full surface** — `browser_act` (
|
|
172
|
+
- 🪶 **Tiny handshake, full surface** — `browser_act` (32 actions), `browser_observe` (8 kinds + `mode`/`scope`), `browser_extract` (8 kinds incl. `state`). No schema bloat, no guessing which of 24 tools to call.
|
|
87
173
|
- 🧠 **Context-safe by default** — snapshots capped at `OUTPUT_MAX_CHARS` (default 4000); full YAML/text saved under `.browser-use/`, never dumped inline. HN snapshot: 4,034-char preview, full file on disk.
|
|
88
174
|
- 🖼️ **File paths, not base64** — screenshots, PDFs, traces return paths like `.browser-use/shot-*.png`. Read the file only when needed.
|
|
89
175
|
- ⚡ **One call, not five** — `batch` runs fill+press flows in one turn (max 20 steps, stops on first error). `goal` plans or returns `E_GOAL_UNCLEAR` + `suggestedSteps` instead of hallucinating.
|
|
90
176
|
- 🐳 **Shared, not spawned** — one Docker server serves every local VS Code instance. No per-client `npx` spawn.
|
|
91
177
|
|
|
178
|
+
## Harness — verify, delta, state, replay
|
|
179
|
+
|
|
180
|
+
v2 turns the browser into a harness: AI actions are verifiable, diffable, and replayable — not just fire-and-forget clicks.
|
|
181
|
+
|
|
182
|
+
**1. Verify — fingerprints + expectations + failure contract**
|
|
183
|
+
|
|
184
|
+
- **Fingerprint registry** (`src/browser/identity.ts`): every `eN` ref is fingerprinted (role, accessibleName, textHash, box, pathHash). Stale refs rebind only on an UNAMBIGUOUS identity match (`rebound:true`); if several elements match equally they fail with `E_STALE` + hint — the engine never guesses a target.
|
|
185
|
+
- **Expect mini-language** (`expect` on any `browser_act`): `url~/dashboard` | `text~/Saved/` | `visible=.modal` | `gone=.spinner` — evaluated server-side, returns `E_EXPECT` / `E_BAD_EXPECT` on mismatch instead of hallucinated success. ReDoS-capped and regex-validated.
|
|
186
|
+
- **Failure contract**: after an uncertain mutation the engine sets `mustObserve:true` — next mutation fails with `E_MUST_OBSERVE` until you re-snapshot. No blind chains.
|
|
187
|
+
- **Evidence envelope**: every `browser_act` returns `effect: { urlChanged, urlBefore/After, domChanged, consoleErrors, networkFailures }` capped at `EFFECT_MAX_CHARS` (800) so the agent sees what actually happened.
|
|
188
|
+
|
|
189
|
+
**2. Delta — pay only for what changed**
|
|
190
|
+
|
|
191
|
+
- `browser_observe kind=snapshot mode=delta` (default) — MutationObserver dirty flag + baseline diff. Returns only `[changed]` lines or `unchanged:true` on stable pages (~90% token saving). `mode=full` for complete dump, `scope="<css>"` for subtree.
|
|
192
|
+
- `DELTA_DEFAULT=true` — flip to `false` to default to full snapshots.
|
|
193
|
+
|
|
194
|
+
**3. State — per-session memory**
|
|
195
|
+
|
|
196
|
+
- `browser_act action=note value="..."` appends to `notes` (capped `STATE_MAX_LINES=40`), `browser_extract kind=state` reads `notes` + `lastActions` ring (last 10). Persisted to `.browser-use/state/<session>.md` so agents survive context compaction.
|
|
197
|
+
|
|
198
|
+
**4. Record → Compile → Replay — deterministic macros**
|
|
199
|
+
|
|
200
|
+
- `record_start` / `record_stop` captures every step with resolved selectors + fingerprints. Secrets auto-redacted (`RECORD_REDACT=true`, `«redacted»` for password/otp/token fields).
|
|
201
|
+
- `compile` generates `.browser-use/macros/<name>.{ts,md}` — a Playwright `run(page)` function + a `SKILL.md` doc. Irreversible steps (`submit`/`pay`/`delete`/etc.) are flagged `requiresApproval`.
|
|
202
|
+
- `replay` replays deterministically; pauses with `E_APPROVAL_REQUIRED` until `approve:true` if any irreversible step exists.
|
|
203
|
+
|
|
204
|
+
**5. CLI — same engine, no MCP client**
|
|
205
|
+
|
|
206
|
+
- `effing-use observe/act/extract` over `EFFING_USE_URL` (`http://localhost:3123/mcp`) — for terminal agents, scripts, and CI. See [CLI usage](#cli-usage).
|
|
207
|
+
|
|
92
208
|
## The loop (agents: follow this order)
|
|
93
209
|
|
|
94
|
-
1. `browser_observe` kind=`snapshot` → get `[eN]` refs (
|
|
95
|
-
2. `browser_act` to interact — prefer `batch` with `steps[]`
|
|
96
|
-
3. `browser_observe` kind=`
|
|
97
|
-
4. `browser_extract` kind=`text`|`table`|`query` → scrape
|
|
210
|
+
1. `browser_observe` kind=`snapshot` → get `[eN]` refs (default `mode: delta` — only changed lines; `mode: full` for complete dump; `scope: "<css>"` for subtree). Never guess refs, re-snapshot after navigation.
|
|
211
|
+
2. `browser_act` to interact — prefer `batch` with `steps[]`; add `expect: "url~/dashboard"` for deterministic post-conditions
|
|
212
|
+
3. `browser_observe` kind=`snapshot` again — delta returns `unchanged:true` or `[changed]` lines
|
|
213
|
+
4. `browser_extract` kind=`text`|`table`|`query`|`state` → scrape or read task state (`notes` + `lastActions`)
|
|
214
|
+
|
|
215
|
+
CLI equivalent: `effing-use observe --kind snapshot --mode delta` / `effing-use act --action click --target e5 --expect 'url~/dashboard'` / `effing-use extract --kind state`
|
|
98
216
|
|
|
99
217
|
## Tools
|
|
100
218
|
|
|
101
|
-
- `browser_act` — open/goto, click, dblclick, fill, type, press, select, check/uncheck, hover, drag, upload, scroll, back/forward/reload, wait, dialog_accept/dismiss, tabs (new/select/close), resize, close, `goal`, `batch`
|
|
102
|
-
- `browser_observe` — snapshot (e-refs), screenshot (path), url, title, console (last N), network (method/url/status ring), tabs, focused
|
|
103
|
-
- `browser_extract` — text, html, table (≤100 rows JSON), query (text|href|json), pdf, trace_start/stop
|
|
219
|
+
- `browser_act` — open/goto, click, dblclick, fill, type, press, select, check/uncheck, hover, drag, upload, scroll, back/forward/reload, wait, dialog_accept/dismiss, tabs (new/select/close), resize, close, `goal`, `batch`, `note`, `record_start`/`record_stop`, `compile`, `replay` (+ `expect` + `approve`)
|
|
220
|
+
- `browser_observe` — snapshot (e-refs, `mode: full|delta` default delta, `scope`), screenshot (path), url, title, console (last N), network (method/url/status ring), tabs, focused
|
|
221
|
+
- `browser_extract` — text, html, table (≤100 rows JSON), query (text|href|json), pdf, trace_start/stop, `state`
|
|
104
222
|
|
|
105
|
-
Errors are always `{ ok: false, code, message, hint }` with `E_NOT_FOUND | E_TIMEOUT | E_NO_PAGE | E_BAD_INPUT | E_GOAL_UNCLEAR` — never a stack trace.
|
|
223
|
+
Errors are always `{ ok: false, code, message, hint }` with `E_NOT_FOUND | E_TIMEOUT | E_NO_PAGE | E_BAD_INPUT | E_GOAL_UNCLEAR | E_STALE | E_EXPECT | E_BAD_EXPECT | E_MUST_OBSERVE | E_APPROVAL_REQUIRED` — never a stack trace.
|
|
106
224
|
|
|
107
225
|
## effing-use vs Playwright MCP
|
|
108
226
|
|
|
@@ -115,9 +233,30 @@ Errors are always `{ ok: false, code, message, hint }` with `E_NOT_FOUND | E_TIM
|
|
|
115
233
|
|
|
116
234
|
Need Firefox/WebKit, device emulation, or persistent profiles? Use Playwright MCP. Need context budget for Chromium work? Stay here. Full breakdown in [`docs/COMPARISON.md`](docs/COMPARISON.md).
|
|
117
235
|
|
|
236
|
+
## Record → replay (harness)
|
|
237
|
+
|
|
238
|
+
```bash
|
|
239
|
+
# MCP — capture any flow, compile to code + skill, replay deterministically
|
|
240
|
+
browser_act action=record_start value=my-flow
|
|
241
|
+
# ... do the flow (clicks, fills, etc.) ...
|
|
242
|
+
browser_act action=record_stop # -> .browser-use/recordings/my-flow.json (secrets redacted)
|
|
243
|
+
browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
|
|
244
|
+
browser_act action=replay value=my-flow # pauses with E_APPROVAL_REQUIRED if irreversible
|
|
245
|
+
browser_act action=replay value=my-flow approve=true # replay with approval
|
|
246
|
+
|
|
247
|
+
# CLI — same flow over HTTP
|
|
248
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action record_start --value my-flow
|
|
249
|
+
# ... do the flow via CLI or MCP ...
|
|
250
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action record_stop
|
|
251
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action compile --value my-flow
|
|
252
|
+
EFFING_USE_URL=http://localhost:3123/mcp effing-use act --action replay --value my-flow --approve true
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
Artifacts: `recordings/*.json` (raw steps), `macros/*.ts` (Playwright `run(page)`), `macros/*.md` (skill doc). Redaction and approval gates are on by default (`RECORD_REDACT=true`).
|
|
256
|
+
|
|
118
257
|
## Config
|
|
119
258
|
|
|
120
|
-
Env defaults in [`.env.example`](.env.example): `BROWSER_HEADLESS`, `BROWSER_VIEWPORT_W/H`, `BROWSER_TIMEOUT_MS`, `OUTPUT_DIR` (`.browser-use/`, gitignored), `OUTPUT_MAX_CHARS`, `ALLOW_EVAL`.
|
|
259
|
+
Env defaults in [`.env.example`](.env.example): `BROWSER_HEADLESS`, `BROWSER_VIEWPORT_W/H`, `BROWSER_TIMEOUT_MS`, `OUTPUT_DIR` (`.browser-use/`, gitignored), `OUTPUT_MAX_CHARS`, `ALLOW_EVAL`, `DELTA_DEFAULT`, `EFFECT_MAX_CHARS`, `STATE_MAX_LINES`, `RECORD_REDACT`, `EFFING_USE_URL`.
|
|
121
260
|
|
|
122
261
|
Agent skill: `skills/effing-use/SKILL.md` (skills.sh-ready).
|
|
123
262
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "effing-use",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.1",
|
|
4
4
|
"description": "Token-efficient browser control: 3 tools (act, observe, extract) built with tmcp + Bun + Playwright",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Michael Obubelebra Amachree",
|
|
@@ -12,8 +12,9 @@
|
|
|
12
12
|
"type": "module",
|
|
13
13
|
"main": "src/index.ts",
|
|
14
14
|
"bin": {
|
|
15
|
-
"effing-use": "src/
|
|
16
|
-
"effing-use-http": "src/http.ts"
|
|
15
|
+
"effing-use": "src/cli.ts",
|
|
16
|
+
"effing-use-http": "src/http.ts",
|
|
17
|
+
"effing-use-stdio": "src/index.ts"
|
|
17
18
|
},
|
|
18
19
|
"files": [
|
|
19
20
|
"src",
|
|
@@ -15,40 +15,67 @@ cost than 21-tool browser servers. `tools/list` is ~3.9KB.
|
|
|
15
15
|
|
|
16
16
|
## The loop (always follow this order)
|
|
17
17
|
|
|
18
|
-
1. `browser_observe` with `kind: "snapshot"` → get `[eN]` refs.
|
|
19
|
-
2. `browser_act` to interact (`open`, `click`, `fill`, `type`, `press`, `select`, `check`, `wait`, …).
|
|
20
|
-
3. `browser_observe` with `kind: "
|
|
21
|
-
4. `browser_extract` with `kind: "text" | "table" | "query"` → scrape.
|
|
18
|
+
1. `browser_observe` with `kind: "snapshot"` → get `[eN]` refs. Default is `mode: "delta"` (`[changed]`/`[removed]` lines since last observe; `unchanged:true` when the page is quiet — navigation automatically forces a fresh `mode: "full"`). Use `mode: "full"` for a complete dump, `scope: "<css>"` to observe a subtree.
|
|
19
|
+
2. `browser_act` to interact (`open`, `click`, `fill`, `type`, `press`, `select`, `check`, `wait`, …). Add `expect: "url~/dashboard"` or `text~/Saved/` for deterministic post-conditions.
|
|
20
|
+
3. `browser_observe` with `kind: "snapshot"` again — delta returns `unchanged:true` if nothing changed, or `[changed]`/`[removed]` lines.
|
|
21
|
+
4. `browser_extract` with `kind: "text" | "table" | "query" | "state"` → scrape or read task state.
|
|
22
22
|
|
|
23
23
|
Rules:
|
|
24
24
|
|
|
25
|
-
- Never guess refs. Re-snapshot after every navigation.
|
|
25
|
+
- Never guess refs. Re-snapshot after every navigation (refs are page-keyed — a cross-page stale ref returns `E_NOT_FOUND`). On a same-page re-render the engine rebinds only an UNAMBIGUOUS identity match (result carries `rebound:true`); if the fingerprint matches several elements it fails with `E_STALE` instead of guessing — re-observe.
|
|
26
26
|
- Prefer `batch`: one `browser_act` with `steps[]` for fill+press flows (max 20 steps, stops on first error).
|
|
27
27
|
- Large outputs are files under `.browser-use/` (gitignored). Read the path, not the preview.
|
|
28
28
|
- Snapshots are capped at `OUTPUT_MAX_CHARS` (default 4000) with `…[truncated N chars, see file]`.
|
|
29
|
+
- After an uncertain mutation the engine sets `mustObserve:true` — next mutation fails with `E_MUST_OBSERVE` until you observe.
|
|
29
30
|
|
|
30
31
|
## Tool cheat sheet
|
|
31
32
|
|
|
32
|
-
**browser_act** — `action` + optional `target` (e-ref, `role=` selector, or CSS) + `value`:
|
|
33
|
+
**browser_act** — `action` + optional `target` (e-ref, `role=` selector, or CSS) + `value` + `expect` + `approve`:
|
|
33
34
|
`open`/`goto` (URL in `value`), `click`, `dblclick`, `fill`, `type`,
|
|
34
35
|
`press` (key like `Enter`), `select`, `check`/`uncheck`, `hover`,
|
|
35
36
|
`drag` (start in `target`, end in `value`), `upload` (comma-separated paths in `value`),
|
|
36
37
|
`scroll` (`up`/`down`/`top`/`bottom` or a target), `back`/`forward`/`reload`,
|
|
37
38
|
`wait` (`ms:500`, `text:Saved`, or a ref), `dialog_accept`/`dialog_dismiss` (arm before the triggering step),
|
|
38
39
|
`resize` (`1280x800` in `value`), `tab_new`/`tab_select`/`tab_close`, `close`,
|
|
39
|
-
`goal` (deterministic add-todo/search planner, else `E_GOAL_UNCLEAR` + `suggestedSteps`),
|
|
40
|
-
`batch` (needs `steps[]`).
|
|
40
|
+
`goal` (deterministic add-todo/search planner, else `E_GOAL_UNCLEAR` + `suggestedSteps`; counts as a mutation — subject to the `mustObserve` guard, evidence, and recording),
|
|
41
|
+
`batch` (needs `steps[]`), `note` (append to task state), `record_start`/`record_stop` (capture flow), `compile` (standalone `.ts` + SKILL.md; selectors resolved id → name → ARIA → data-\* → placeholder → text), `replay` (deterministic replay; runs benign steps, halts AT approval-gated steps).
|
|
42
|
+
`expect` mini-language: `url~<regex>` | `text~<regex>` | `visible=<css>` | `gone=<css>` — evaluated in code, returns `E_EXPECT` or `E_BAD_EXPECT`.
|
|
41
43
|
|
|
42
|
-
**browser_observe** — read-only: `snapshot` (e-refs), `screenshot` (file path,
|
|
44
|
+
**browser_observe** — read-only: `snapshot` (e-refs, `mode: full|delta` default delta, `scope: <css>`), `screenshot` (file path,
|
|
43
45
|
`full` page by default), `url`, `title`, `console` (last N, `limit`),
|
|
44
46
|
`network` (method/url/status ring), `tabs`, `focused` (activeElement HTML).
|
|
45
47
|
|
|
46
48
|
**browser_extract** — `text`, `html`, `table` (≤100 rows as JSON),
|
|
47
49
|
`query` (`selector` + `mode: text|href|json`), `pdf` (headless Chromium only),
|
|
48
|
-
`trace_start`/`trace_stop` (Playwright trace zip).
|
|
50
|
+
`trace_start`/`trace_stop` (Playwright trace zip), `state` (task notes + lastActions ring).
|
|
49
51
|
|
|
50
52
|
Errors always come back as `{ ok: false, code, message, hint }` with codes
|
|
51
|
-
`E_NOT_FOUND | E_TIMEOUT | E_NO_PAGE | E_BAD_INPUT | E_GOAL_UNCLEAR` — never a stack trace.
|
|
53
|
+
`E_NOT_FOUND | E_TIMEOUT | E_NO_PAGE | E_BAD_INPUT | E_GOAL_UNCLEAR | E_STALE | E_EXPECT | E_BAD_EXPECT | E_MUST_OBSERVE | E_APPROVAL_REQUIRED` — never a stack trace.
|
|
54
|
+
|
|
55
|
+
## CLI face (same engine, no MCP client needed)
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
effing-use observe --kind snapshot --mode delta --session dev
|
|
59
|
+
effing-use act --action click --target e5 --expect 'url~/dashboard' --session dev
|
|
60
|
+
effing-use extract --kind state --session dev
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Set `EFFING_USE_URL` (default `http://localhost:3123/mcp`). Requires a running `bun src/http.ts` server.
|
|
64
|
+
|
|
65
|
+
## Record → replay
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
# via MCP
|
|
69
|
+
browser_act action=record_start value=my-flow
|
|
70
|
+
# ... do the flow ...
|
|
71
|
+
browser_act action=record_stop
|
|
72
|
+
browser_act action=compile value=my-flow # -> .browser-use/macros/my-flow.{ts,md}
|
|
73
|
+
browser_act action=replay value=my-flow # runs benign steps, halts AT the first
|
|
74
|
+
# approval-gated step → E_APPROVAL_REQUIRED
|
|
75
|
+
# { step: N (1-indexed), completed: C }
|
|
76
|
+
browser_act action=replay value=my-flow approve=true # resumes at the gate; the
|
|
77
|
+
# benign prefix is NOT re-run (cursor)
|
|
78
|
+
```
|
|
52
79
|
|
|
53
80
|
## Setup
|
|
54
81
|
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
import type { Page } from "playwright";
|
|
2
|
+
|
|
3
|
+
type PageState = {
|
|
4
|
+
dirty: boolean;
|
|
5
|
+
baseline: string | null;
|
|
6
|
+
baselineUrl: string | null;
|
|
7
|
+
};
|
|
8
|
+
|
|
9
|
+
const pageStates = new Map<string, PageState>();
|
|
10
|
+
|
|
11
|
+
function key(sessionId: string): string {
|
|
12
|
+
return sessionId;
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
export function ensureState(sessionId: string): PageState {
|
|
16
|
+
let s = pageStates.get(key(sessionId));
|
|
17
|
+
if (!s) {
|
|
18
|
+
s = { dirty: true, baseline: null, baselineUrl: null };
|
|
19
|
+
pageStates.set(key(sessionId), s);
|
|
20
|
+
}
|
|
21
|
+
return s;
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
export function markDirty(sessionId: string): void {
|
|
25
|
+
ensureState(sessionId).dirty = true;
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
export function markClean(sessionId: string): void {
|
|
29
|
+
ensureState(sessionId).dirty = false;
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
export function setBaseline(
|
|
33
|
+
sessionId: string,
|
|
34
|
+
snapshot: string,
|
|
35
|
+
url: string,
|
|
36
|
+
): void {
|
|
37
|
+
const s = ensureState(sessionId);
|
|
38
|
+
s.baseline = snapshot;
|
|
39
|
+
s.baselineUrl = url;
|
|
40
|
+
s.dirty = false;
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
export function getBaseline(sessionId: string): string | null {
|
|
44
|
+
return ensureState(sessionId).baseline;
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/** URL the current baseline was captured on — forces mode:full after navigation (plan §5.1). */
|
|
48
|
+
export function getBaselineUrl(sessionId: string): string | null {
|
|
49
|
+
return pageStates.get(key(sessionId))?.baselineUrl ?? null;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
export function isDirty(sessionId: string): boolean {
|
|
53
|
+
return ensureState(sessionId).dirty;
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
export function clearDelta(sessionId: string): void {
|
|
57
|
+
pageStates.delete(key(sessionId));
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
export async function injectDirtyObserver(
|
|
61
|
+
page: Page,
|
|
62
|
+
sessionId: string,
|
|
63
|
+
): Promise<void> {
|
|
64
|
+
try {
|
|
65
|
+
await page.evaluate((sid) => {
|
|
66
|
+
const w = window as unknown as {
|
|
67
|
+
__effDirty?: boolean;
|
|
68
|
+
__effSid?: string;
|
|
69
|
+
__effObs?: MutationObserver;
|
|
70
|
+
};
|
|
71
|
+
w.__effDirty = false;
|
|
72
|
+
w.__effSid = sid;
|
|
73
|
+
if (w.__effObs) w.__effObs.disconnect();
|
|
74
|
+
// Attribute spam that never changes the snapshot's ref lines (class,
|
|
75
|
+
// style, expand/animation state) used to mark every live SPA permanently
|
|
76
|
+
// dirty — the cheap "unchanged" fast path never fired. Ignore those;
|
|
77
|
+
// content changes still arrive via childList/characterData/input/change.
|
|
78
|
+
const NOISE_ATTRS = [
|
|
79
|
+
"class",
|
|
80
|
+
"style",
|
|
81
|
+
"aria-expanded",
|
|
82
|
+
"data-state",
|
|
83
|
+
"data-orientation",
|
|
84
|
+
"data-scroll-state",
|
|
85
|
+
"data-highlighted",
|
|
86
|
+
"data-hovered",
|
|
87
|
+
"data-dragging",
|
|
88
|
+
"data-resizing",
|
|
89
|
+
"data-index",
|
|
90
|
+
];
|
|
91
|
+
const obs = new MutationObserver((muts) => {
|
|
92
|
+
for (const m of muts) {
|
|
93
|
+
if (m.type === "attributes") {
|
|
94
|
+
const name = m.attributeName || "";
|
|
95
|
+
if (NOISE_ATTRS.indexOf(name) !== -1) continue;
|
|
96
|
+
}
|
|
97
|
+
w.__effDirty = true;
|
|
98
|
+
return;
|
|
99
|
+
}
|
|
100
|
+
});
|
|
101
|
+
obs.observe(document.documentElement, {
|
|
102
|
+
childList: true,
|
|
103
|
+
subtree: true,
|
|
104
|
+
attributes: true,
|
|
105
|
+
characterData: true,
|
|
106
|
+
});
|
|
107
|
+
document.addEventListener(
|
|
108
|
+
"input",
|
|
109
|
+
() => {
|
|
110
|
+
w.__effDirty = true;
|
|
111
|
+
},
|
|
112
|
+
true,
|
|
113
|
+
);
|
|
114
|
+
document.addEventListener(
|
|
115
|
+
"change",
|
|
116
|
+
() => {
|
|
117
|
+
w.__effDirty = true;
|
|
118
|
+
},
|
|
119
|
+
true,
|
|
120
|
+
);
|
|
121
|
+
w.__effObs = obs;
|
|
122
|
+
}, sessionId);
|
|
123
|
+
} catch {
|
|
124
|
+
// ignore injection failures (e.g. page not ready)
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
export async function checkDirty(page: Page): Promise<boolean> {
|
|
129
|
+
try {
|
|
130
|
+
const d = await page.evaluate(
|
|
131
|
+
() => (window as unknown as { __effDirty?: boolean }).__effDirty,
|
|
132
|
+
);
|
|
133
|
+
return Boolean(d);
|
|
134
|
+
} catch {
|
|
135
|
+
return true;
|
|
136
|
+
}
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
export async function clearDirtyFlag(page: Page): Promise<void> {
|
|
140
|
+
try {
|
|
141
|
+
await page.evaluate(() => {
|
|
142
|
+
(window as unknown as { __effDirty?: boolean }).__effDirty = false;
|
|
143
|
+
});
|
|
144
|
+
} catch {
|
|
145
|
+
/* ignore */
|
|
146
|
+
}
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
export function computeDelta(
|
|
150
|
+
baseline: string | null,
|
|
151
|
+
current: string,
|
|
152
|
+
): { delta: string; unchanged: boolean } {
|
|
153
|
+
if (!baseline) return { delta: current, unchanged: false };
|
|
154
|
+
if (baseline === current) return { delta: "", unchanged: true };
|
|
155
|
+
// Simple line diff: emit only changed lines with [changed] markers
|
|
156
|
+
const baseLines = baseline.split("\n");
|
|
157
|
+
const curLines = current.split("\n");
|
|
158
|
+
const baseSet = new Set(baseLines);
|
|
159
|
+
const curSet = new Set(curLines);
|
|
160
|
+
const changed = curLines.filter((l) => !baseSet.has(l));
|
|
161
|
+
const removed = baseLines.filter((l) => !curSet.has(l));
|
|
162
|
+
if (changed.length === 0 && removed.length === 0)
|
|
163
|
+
return { delta: "", unchanged: true };
|
|
164
|
+
const unchangedCount = curLines.length - changed.length;
|
|
165
|
+
const header = `…${unchangedCount} unchanged lines…`;
|
|
166
|
+
const delta = [
|
|
167
|
+
header,
|
|
168
|
+
...changed.map((l) => `[changed] ${l}`),
|
|
169
|
+
// plan §5.1: removed nodes are reported explicitly
|
|
170
|
+
...removed.map((l) => `[removed] ${l}`),
|
|
171
|
+
].join("\n");
|
|
172
|
+
return { delta, unchanged: false };
|
|
173
|
+
}
|