pi-browser-harness 0.14.1 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +15 -8
- package/package.json +1 -1
- package/skills/pi-browser-harness/SKILL.md +1 -2
- package/src/cdp/commands.ts +28 -0
- package/src/client.ts +10 -3
- package/src/daemon/bridge.ts +4 -3
- package/src/daemon/protocol.ts +11 -0
- package/src/daemon/transport.ts +1 -0
- package/src/domains/cdp-call.ts +2 -1
- package/src/domains/click.ts +5 -2
- package/src/domains/form.ts +40 -31
- package/src/domains/js.ts +20 -2
- package/src/domains/keyboard.ts +171 -35
- package/src/domains/page/click.page.ts +22 -0
- package/src/domains/page/fill.page.ts +70 -1
- package/src/domains/page/serp.page.ts +29 -16
- package/src/domains/page.ts +1 -1
- package/src/domains/screenshot.ts +21 -1
- package/src/domains/search/engines/bing.ts +47 -0
- package/src/domains/search/engines/brave.ts +23 -0
- package/src/domains/search/engines/duckduckgo.ts +42 -0
- package/src/domains/search/engines/google.ts +95 -0
- package/src/domains/search/engines/index.ts +13 -0
- package/src/domains/search/engines/registry.ts +29 -0
- package/src/domains/search/redirect-resolver.ts +1 -1
- package/src/domains/search/serp.ts +157 -0
- package/src/domains/search/web-search.ts +235 -105
- package/src/domains/snapshot.ts +1 -1
- package/src/domains/target.ts +83 -7
- package/src/recipes/recordable.ts +4 -3
- package/src/registry.ts +1 -2
- package/src/util/image-size.ts +16 -0
- package/src/domains/search/google-serp.ts +0 -155
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,31 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to pi-browser-harness will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## 0.16.0 — 2026-11-07
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **`browser_web_search` falls back across search engines.** With no `engine` given it tries Google, then Bing, DuckDuckGo and Brave, until one returns results, so a Google CAPTCHA or markup change no longer breaks search. `details.engine` says which engine answered and `details.fallbacks` lists the ones that failed first. When every engine fails the error is `all_engines_failed` with each engine's reason; when none found anything it stays `no_results`.
|
|
10
|
+
- **`engine` parameter.** Name one engine (`google`, `bing`, `duckduckgo`, `brave`) to search it alone, with no fallback.
|
|
11
|
+
- **Pluggable engines.** Each engine is a self-contained module mounted into a registry (`createEngineRegistry().mount(engine)`) that is injected into `createWebSearchTool`; mount order is the fallback order. Adding an engine is one module and one `mount`.
|
|
12
|
+
|
|
13
|
+
## 0.15.1 — 2026-11-07
|
|
14
|
+
|
|
15
|
+
- Re-release of 0.15.0, whose npm publish failed (the provenance log returned 502, then rejected the retry as a duplicate). No code changes.
|
|
16
|
+
|
|
17
|
+
## 0.15.0 — 2026-11-07
|
|
18
|
+
|
|
19
|
+
### Changed
|
|
20
|
+
|
|
21
|
+
- **`browser_fill` and `browser_type` are one tool: `browser_type`.** Pass a `ref` (or `selector`) and it focuses the field, replaces its value (`clear:false` appends; `text: ""` empties it), types with real input events so autocomplete lists open, and reports the value read back. When typing does not produce the value (date or masked inputs) it sets the value directly and says so. With no ref it types into the focused element, appending as before. Saved recipes with `fill` steps still replay.
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
|
|
25
|
+
- **`browser_click` no longer clicks into nothing.** A ref is scrolled into view (nested scrolling lists included) and measured at click time; a point still outside the viewport, or an `x`/`y` outside it, is refused with the reason. When something covers the element (an overlay, a sticky header) the result says what was clicked instead. An `x`/`y` click names the element it landed on.
|
|
26
|
+
- **`browser_press_key` runs Cmd/Ctrl shortcuts.** Cmd/Ctrl + A, C, X, V, Z, Y now carry Chrome's editing command, so select-all, copy, paste and undo work. Letter keys send the right `code` (`KeyA`) and key code (65).
|
|
27
|
+
- **`browser_execute_js` waits up to 30s by default, and `timeoutMs` (max 120s) raises it.** It used to stop at 10s, mid-way through scripts that loop with waits. A timeout now says the script may have partly run.
|
|
28
|
+
- **`browser_screenshot` says how its pixels map to click coordinates** when the image is not 1:1 with CSS pixels (2× on Retina, or after `maxDim`).
|
|
29
|
+
|
|
5
30
|
## 0.14.1 — 2026-11-07
|
|
6
31
|
|
|
7
32
|
### Fixed
|
package/README.md
CHANGED
|
@@ -18,7 +18,7 @@ no MCP hop, no second browser, no extra API keys.
|
|
|
18
18
|
| **Browser** | Your running Chrome — real profile, live sessions | Launches a clean instance | Launches a clean instance | Launches a clean instance |
|
|
19
19
|
| **Element targeting** | AX-tree refs (`[eN]`) that re-resolve at action time; coordinate and selector fallbacks | Selectors / AX locators | CSS selectors | LLM-inferred selectors |
|
|
20
20
|
| **Input dispatch** | Compositor-level CDP input — works through iframes, shadow DOM, cross-origin frames | DOM-level | DOM-level | DOM-level |
|
|
21
|
-
| **Framework-safe writes** | `
|
|
21
|
+
| **Framework-safe writes** | `browser_type` types with real input events, reads the value back, and falls back to the native setter when typing doesn't take | Native fill | Manual | Native fill |
|
|
22
22
|
| **Transport** | Native pi extension, in-process | MCP / JSON-RPC round trip | MCP / JSON-RPC round trip | In-process + its own LLM calls |
|
|
23
23
|
| **Extra credentials** | None | None | None | Its own model API key |
|
|
24
24
|
| **Concurrency** | Reads run in parallel; mutations auto-serialize through a shared mutex | Sequential | Sequential | Sequential |
|
|
@@ -76,10 +76,12 @@ interactive element carries a stable ref (`[eN]`) plus click coordinates `@(x,y)
|
|
|
76
76
|
[e12] button "Create account" @(456,412)
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
-
Pass the ref — not the coordinates — to `browser_click`, `
|
|
79
|
+
Pass the ref — not the coordinates — to `browser_click`, `browser_type`, `browser_select_option`,
|
|
80
80
|
`browser_focus`, `browser_upload_file`, or `browser_dispatch_key`. Refs are keyed to element
|
|
81
81
|
identity, so they re-resolve position at action time and survive the re-renders that SPA forms
|
|
82
|
-
trigger on every keystroke. Coordinates go stale the moment the layout reflows.
|
|
82
|
+
trigger on every keystroke. Coordinates go stale the moment the layout reflows. `browser_click` by ref
|
|
83
|
+
scrolls the element into view first (inside scrolling lists too) and warns when something covers it;
|
|
84
|
+
an `x`/`y` outside the viewport is refused instead of clicking nothing.
|
|
83
85
|
|
|
84
86
|
Three things follow from that:
|
|
85
87
|
|
|
@@ -117,7 +119,7 @@ Three things follow from that:
|
|
|
117
119
|
|
|
118
120
|
| Tool | Purpose |
|
|
119
121
|
|---|---|
|
|
120
|
-
| `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real
|
|
122
|
+
| `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real search results page in your Chrome. No API key. Links only. Walks the fallback ladder Google → Bing → DuckDuckGo → Brave until one answers, or searches only the `engine` you name. |
|
|
121
123
|
| `browser_read_page` | Reader mode. A `url` (opened and closed in its own tab) or an owned `targetId` → clean article text, boilerplate stripped. |
|
|
122
124
|
|
|
123
125
|
Both run in isolated tabs and never disturb your current page.
|
|
@@ -127,10 +129,9 @@ Both run in isolated tabs and never disturb your current page.
|
|
|
127
129
|
| Tool | Purpose |
|
|
128
130
|
|---|---|
|
|
129
131
|
| `browser_click` | Click by `ref` (preferred) or viewport `x`/`y`. Compositor-level, so iframes and shadow DOM work. |
|
|
130
|
-
| `browser_fill` | Set an input, textarea, or contenteditable. Fires bubbling `input`/`change`, returns the value written. |
|
|
131
132
|
| `browser_select_option` | Choose in a native `<select>` by value, label, or index. Returns available options on no match. |
|
|
132
133
|
| `browser_focus` | Deterministic `.focus()` — no coordinate accuracy needed. |
|
|
133
|
-
| `browser_type` |
|
|
134
|
+
| `browser_type` | Enter text into a field by `ref`/selector (replaces its value; `clear:false` appends) or into the focused element. Real input events, so autocomplete opens; returns the value read back. |
|
|
134
135
|
| `browser_press_key` | Press a key with an optional modifier bitfield. |
|
|
135
136
|
| `browser_dispatch_key` | Synthetic DOM `KeyboardEvent` on one element, for pages that ignore raw CDP keys. |
|
|
136
137
|
| `browser_scroll` | Scroll by delta. W3C wheel convention: positive `deltaY` = down. |
|
|
@@ -202,7 +203,7 @@ Wait, snapshot once, then act by ref. No screenshots anywhere in this loop.
|
|
|
202
203
|
```
|
|
203
204
|
browser_wait_for({ selector: "#email" })
|
|
204
205
|
browser_snapshot() // → [e7] textbox, [e9] combobox, [e12] button
|
|
205
|
-
|
|
206
|
+
browser_type({ ref: "e7", text: "a@b.com" })
|
|
206
207
|
browser_select_option({ ref: "e9", label: "India" })
|
|
207
208
|
browser_click({ ref: "e12" }) // re-resolves position even after reflow
|
|
208
209
|
```
|
|
@@ -244,8 +245,14 @@ browser_network_requests({ // did a request 4xx/5xx?
|
|
|
244
245
|
```
|
|
245
246
|
browser_web_search({ query: "chrome devtools protocol overview", limit: 5 })
|
|
246
247
|
browser_read_page({ url: "<the best result's url>" })
|
|
248
|
+
|
|
249
|
+
// Search one engine only, with no fallback:
|
|
250
|
+
browser_web_search({ query: "chrome devtools protocol overview", engine: "bing" })
|
|
247
251
|
```
|
|
248
252
|
|
|
253
|
+
`details.engine` names the engine that answered, and `details.fallbacks` lists any engines that failed
|
|
254
|
+
first (CAPTCHA, no results, markup change, load failure).
|
|
255
|
+
|
|
249
256
|
Both tools open their own isolated tab, so neither disturbs the tab you are working in.
|
|
250
257
|
|
|
251
258
|
### Keyboard modifiers
|
|
@@ -352,7 +359,7 @@ re-runnable.
|
|
|
352
359
|
|
|
353
360
|
Another extension can add its own `browser_*` tools to this harness. Contributed tools share the
|
|
354
361
|
same client, mutation mutex, tab ownership, and lane prompt as the built-ins, so a `serialized` tool
|
|
355
|
-
you add queues behind `browser_click` exactly as `
|
|
362
|
+
you add queues behind `browser_click` exactly as `browser_type` does.
|
|
356
363
|
|
|
357
364
|
Define the tool with `defineBrowserTool` and hand it to `registerBrowserTools` at your extension's
|
|
358
365
|
load time, before pi emits `session_start`. The harness registers it with pi on the next
|
package/package.json
CHANGED
|
@@ -61,7 +61,6 @@ Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
|
|
|
61
61
|
| `browser_download` | serialized | Configure download directory |
|
|
62
62
|
| `browser_drag_and_drop` | serialized | Drag and drop between two coordinates |
|
|
63
63
|
| `browser_execute_js` | parallel | Execute JS in the page — surgical DOM reads (preferred over screenshot) |
|
|
64
|
-
| `browser_fill` | serialized | Fill a form field by ref (preferred) or selector (works with React/Vue controlled inputs) |
|
|
65
64
|
| `browser_fill_form` | serialized | Fill many form fields at once by ref — React-safe, self-confirming |
|
|
66
65
|
| `browser_focus` | serialized | Focus an element by ref (preferred) or selector |
|
|
67
66
|
| `browser_go_back` | serialized | Go back one page |
|
|
@@ -89,7 +88,7 @@ Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
|
|
|
89
88
|
| `browser_setup` | parallel | Initialize browser connection |
|
|
90
89
|
| `browser_snapshot` | parallel | Get accessibility-tree snapshot with stable element refs (default for page inspection) |
|
|
91
90
|
| `browser_switch_tab` | serialized | Switch tabs |
|
|
92
|
-
| `browser_type` | serialized | Type text into the focused element |
|
|
91
|
+
| `browser_type` | serialized | Type text into a field by ref (preferred), selector, or the focused element |
|
|
93
92
|
| `browser_upload_file` | serialized | Upload a file to a file input by ref (preferred) or selector |
|
|
94
93
|
| `browser_viewport_resize` | serialized | Resize the viewport (responsive testing) |
|
|
95
94
|
| `browser_wait` | parallel | Wait N seconds |
|
package/src/cdp/commands.ts
CHANGED
|
@@ -271,6 +271,18 @@ export const COMMANDS = {
|
|
|
271
271
|
}),
|
|
272
272
|
Type.Object({ data: Type.String(), stream: Type.Optional(Type.String()) }),
|
|
273
273
|
),
|
|
274
|
+
"Page.getLayoutMetrics": cmd(
|
|
275
|
+
Empty,
|
|
276
|
+
Type.Object(
|
|
277
|
+
{
|
|
278
|
+
cssLayoutViewport: Type.Object(
|
|
279
|
+
{ clientWidth: Type.Number(), clientHeight: Type.Number() },
|
|
280
|
+
{ additionalProperties: true },
|
|
281
|
+
),
|
|
282
|
+
},
|
|
283
|
+
{ additionalProperties: true },
|
|
284
|
+
),
|
|
285
|
+
),
|
|
274
286
|
"Page.handleJavaScriptDialog": cmd(
|
|
275
287
|
Type.Object({ accept: Type.Boolean(), promptText: Type.Optional(Type.String()) }),
|
|
276
288
|
Empty,
|
|
@@ -322,6 +334,22 @@ export const COMMANDS = {
|
|
|
322
334
|
),
|
|
323
335
|
}),
|
|
324
336
|
),
|
|
337
|
+
"DOM.scrollIntoViewIfNeeded": cmd(
|
|
338
|
+
Type.Object({ nodeId: Type.Optional(Type.Number()), backendNodeId: Type.Optional(Type.Number()) }),
|
|
339
|
+
Empty,
|
|
340
|
+
),
|
|
341
|
+
"DOM.getNodeForLocation": cmd(
|
|
342
|
+
Type.Object({
|
|
343
|
+
x: Type.Integer(),
|
|
344
|
+
y: Type.Integer(),
|
|
345
|
+
includeUserAgentShadowDOM: Type.Optional(Type.Boolean()),
|
|
346
|
+
ignorePointerEventsNone: Type.Optional(Type.Boolean()),
|
|
347
|
+
}),
|
|
348
|
+
Type.Object(
|
|
349
|
+
{ backendNodeId: Type.Number(), frameId: Type.Optional(Type.String()), nodeId: Type.Optional(Type.Number()) },
|
|
350
|
+
{ additionalProperties: true },
|
|
351
|
+
),
|
|
352
|
+
),
|
|
325
353
|
"DOM.resolveNode": cmd(
|
|
326
354
|
Type.Object({ nodeId: Type.Optional(Type.Number()), backendNodeId: Type.Optional(Type.Number()) }),
|
|
327
355
|
Type.Object({ object: RemoteObject }),
|
package/src/client.ts
CHANGED
|
@@ -59,7 +59,7 @@ export type BrowserClient = {
|
|
|
59
59
|
stop(): Promise<void>;
|
|
60
60
|
detach(): Promise<void>;
|
|
61
61
|
closeOwnedTabs(): Promise<void>;
|
|
62
|
-
evaluateJs(expression: string, sessionId?: string): Promise<Result<unknown, CdpError>>;
|
|
62
|
+
evaluateJs(expression: string, sessionId?: string, opts?: { readonly timeoutMs?: number }): Promise<Result<unknown, CdpError>>;
|
|
63
63
|
pageInfo(): Promise<Result<PageInfoResult, CdpError>>;
|
|
64
64
|
takeDialog(): DialogInfo | null;
|
|
65
65
|
listTabs(includeInternal?: boolean): Promise<Result<ReadonlyArray<TabInfo>, CdpError>>;
|
|
@@ -260,9 +260,16 @@ export const createBrowserClient = (opts: BrowserClientOptions): BrowserClient =
|
|
|
260
260
|
return running;
|
|
261
261
|
};
|
|
262
262
|
|
|
263
|
-
const evaluateJs = async (
|
|
263
|
+
const evaluateJs = async (
|
|
264
|
+
expression: string,
|
|
265
|
+
sessionId?: string,
|
|
266
|
+
opts?: { readonly timeoutMs?: number },
|
|
267
|
+
): Promise<Result<unknown, CdpError>> => {
|
|
264
268
|
const params = evaluateParams(expression);
|
|
265
|
-
const
|
|
269
|
+
const callOpts = opts?.timeoutMs !== undefined ? { timeoutMs: opts.timeoutMs } : {};
|
|
270
|
+
const r = sessionId
|
|
271
|
+
? await session.callOnTarget("Runtime.evaluate", params, sessionId, callOpts)
|
|
272
|
+
: await session.call("Runtime.evaluate", params, callOpts);
|
|
266
273
|
if (!r.success) return r;
|
|
267
274
|
if (r.data.exceptionDetails !== undefined) {
|
|
268
275
|
return err(cdpError("remote_error", `JS evaluation failed: ${JSON.stringify(r.data.exceptionDetails)}`, "Runtime.evaluate"));
|
package/src/daemon/bridge.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { isCdpRawMessage } from "../cdp/types";
|
|
2
2
|
import type { CdpRawMessage } from "../cdp/types";
|
|
3
3
|
import type { WireRequest, WireResponse, WireEvent } from "./protocol";
|
|
4
|
-
import {
|
|
4
|
+
import { DAEMON_RECONNECT_METHOD, commandTimeoutFor } from "./protocol";
|
|
5
5
|
import { asString, isRecord } from "../util/guards";
|
|
6
6
|
import { type ChromeConnectionDependencies, type ConnectionStatus, createChromeConnection } from "./connection";
|
|
7
7
|
|
|
@@ -246,6 +246,7 @@ export const createCdpBridge = (options: ChromeConnectionDependencies = {}): Cdp
|
|
|
246
246
|
if (sessionId !== undefined) router.release(sessionId);
|
|
247
247
|
}
|
|
248
248
|
|
|
249
|
+
const timeoutMs = commandTimeoutFor(req);
|
|
249
250
|
const daemonId = mux.allocate(
|
|
250
251
|
{
|
|
251
252
|
clientId,
|
|
@@ -257,8 +258,8 @@ export const createCdpBridge = (options: ChromeConnectionDependencies = {}): Cdp
|
|
|
257
258
|
(id) =>
|
|
258
259
|
setTimeout(() => {
|
|
259
260
|
mux.take(id);
|
|
260
|
-
send(clientId, errorResponse(req.id, `Timeout after ${
|
|
261
|
-
},
|
|
261
|
+
send(clientId, errorResponse(req.id, `Timeout after ${timeoutMs}ms: ${req.method}`));
|
|
262
|
+
}, timeoutMs),
|
|
262
263
|
);
|
|
263
264
|
|
|
264
265
|
const payload: Record<string, unknown> = {
|
package/src/daemon/protocol.ts
CHANGED
|
@@ -6,6 +6,8 @@ export type WireRequest = {
|
|
|
6
6
|
readonly method: string;
|
|
7
7
|
readonly params?: Record<string, unknown>;
|
|
8
8
|
readonly sessionId?: string;
|
|
9
|
+
// How long the daemon waits for Chrome's answer; see commandTimeoutFor.
|
|
10
|
+
readonly timeoutMs?: number;
|
|
9
11
|
};
|
|
10
12
|
|
|
11
13
|
export type WireResponse = {
|
|
@@ -71,6 +73,7 @@ const isWireMessage = (v: unknown): v is WireMessage => {
|
|
|
71
73
|
if (typeof v["id"] !== "number") return false;
|
|
72
74
|
if (typeof v["method"] !== "string") return false;
|
|
73
75
|
if (!isOptionalRecord(v["params"])) return false;
|
|
76
|
+
if (v["timeoutMs"] !== undefined && typeof v["timeoutMs"] !== "number") return false;
|
|
74
77
|
return isOptionalString(v["sessionId"]);
|
|
75
78
|
}
|
|
76
79
|
case "response": {
|
|
@@ -138,3 +141,11 @@ export const CDP_RECONNECT_COOLDOWN_MAX_MS = 5 * 60 * 1000;
|
|
|
138
141
|
export const DAEMON_RECONNECT_METHOD = "Daemon.reconnect";
|
|
139
142
|
|
|
140
143
|
export const CDP_COMMAND_TIMEOUT_MS = 10_000;
|
|
144
|
+
|
|
145
|
+
// The longest a caller may ask the daemon to wait for one command (browser_execute_js's timeoutMs).
|
|
146
|
+
export const CDP_COMMAND_TIMEOUT_MAX_MS = 120_000;
|
|
147
|
+
|
|
148
|
+
// A request may lengthen the daemon's wait (a long-running script) but never shorten it below the default: short
|
|
149
|
+
// client-side probes already give up on their own, and a shorter daemon timer would only race them.
|
|
150
|
+
export const commandTimeoutFor = (req: { readonly timeoutMs?: number }): number =>
|
|
151
|
+
Math.min(Math.max(req.timeoutMs ?? CDP_COMMAND_TIMEOUT_MS, CDP_COMMAND_TIMEOUT_MS), CDP_COMMAND_TIMEOUT_MAX_MS);
|
package/src/daemon/transport.ts
CHANGED
|
@@ -194,6 +194,7 @@ export const createDaemonTransport = (clientId: string, socketPath: string = DAE
|
|
|
194
194
|
method,
|
|
195
195
|
params,
|
|
196
196
|
...(sessionId !== null ? { sessionId } : {}),
|
|
197
|
+
...(opts?.timeoutMs !== undefined ? { timeoutMs: opts.timeoutMs } : {}),
|
|
197
198
|
};
|
|
198
199
|
|
|
199
200
|
return sendWithTimeout(pending, id, method, timeoutMs, "Daemon", () => sock.write(serialize(req) + "\n"));
|
package/src/domains/cdp-call.ts
CHANGED
|
@@ -55,8 +55,9 @@ export const evalJs = async (
|
|
|
55
55
|
client: BrowserClient,
|
|
56
56
|
expression: string,
|
|
57
57
|
sessionId?: string,
|
|
58
|
+
opts?: { readonly timeoutMs?: number },
|
|
58
59
|
): Promise<Result<unknown, ToolErr>> => {
|
|
59
|
-
const r = await client.evaluateJs(expression, sessionId);
|
|
60
|
+
const r = await client.evaluateJs(expression, sessionId, opts);
|
|
60
61
|
if (!r.success) return err(cdpErrToToolErr(r.error, "Runtime.evaluate"));
|
|
61
62
|
return r;
|
|
62
63
|
};
|
package/src/domains/click.ts
CHANGED
|
@@ -55,7 +55,7 @@ export const clickTool = defineBrowserTool({
|
|
|
55
55
|
name: "browser_click",
|
|
56
56
|
label: "Browser Click",
|
|
57
57
|
description:
|
|
58
|
-
"Click an element. PREFERRED: pass `ref` (e.g. 'e12') from browser_snapshot — it
|
|
58
|
+
"Click an element. PREFERRED: pass `ref` (e.g. 'e12') from browser_snapshot — it scrolls the element into view (inside scrolling lists too), measures it at click time, and warns if something covers it. Fallback: pass viewport CSS-pixel `x`/`y` (refused when outside the viewport; the result names the element that was hit). Compositor-level click works through iframes, shadow DOM, and cross-origin content. After clicking, a compact diff of page changes is appended.",
|
|
59
59
|
promptSnippet: "Click an element by ref (preferred) or pixel coordinates",
|
|
60
60
|
promptGuidelines: [
|
|
61
61
|
"If a click doesn't register, run pi with --browser-debug-clicks (or set BH_DEBUG_CLICKS=1) to get annotated screenshots. For React/Vue components ignoring clicks, try browser_dispatch_key.",
|
|
@@ -69,7 +69,10 @@ export const clickTool = defineBrowserTool({
|
|
|
69
69
|
|
|
70
70
|
const resolved = await resolveTarget(client, args);
|
|
71
71
|
if (!resolved.success) return resolved;
|
|
72
|
-
const { x, y, label
|
|
72
|
+
const { x, y, label, covered, hit } = resolved.data;
|
|
73
|
+
const target = label
|
|
74
|
+
+ (hit !== undefined ? ` on ${hit}` : "")
|
|
75
|
+
+ (covered !== undefined ? `\n⚠ The point was covered by ${covered}, so the click went to that element, not [${args.ref ?? ""}]. Close the overlay or scroll, then re-snapshot.` : "");
|
|
73
76
|
|
|
74
77
|
const clicked = await dispatchClick(client, x, y, button, count);
|
|
75
78
|
if (!clicked.success) return clicked;
|
package/src/domains/form.ts
CHANGED
|
@@ -2,7 +2,8 @@ import { Type } from "typebox";
|
|
|
2
2
|
import { type Result, err, ok } from "../util/result";
|
|
3
3
|
import { defineBrowserTool, type JsonObject, type ToolErr, type ToolOk } from "../util/tool";
|
|
4
4
|
import { detailsOf, type ElementOutcome, isPageResult, resolveAndCall, runOnElement } from "./element-call";
|
|
5
|
-
import { fillField, type FillOptions } from "./page/fill.page";
|
|
5
|
+
import { fillField, type FillOptions, type PageResult } from "./page/fill.page";
|
|
6
|
+
import type { BrowserClient } from "../client";
|
|
6
7
|
import { type FocusResult, focusElement, type SelectOption, type SelectResult, type SelectWanted, selectOption, setChecked } from "./page/form.page";
|
|
7
8
|
import { elementFunctionSource } from "./page/inject";
|
|
8
9
|
import { interactiveDiff } from "./ref-resolve";
|
|
@@ -56,45 +57,53 @@ const FillArgs = Type.Object({
|
|
|
56
57
|
const FILL_FN = elementFunctionSource(fillField);
|
|
57
58
|
const FILL_OPTIONS: FillOptions = { rejectSelect: true, focusFirst: false };
|
|
58
59
|
|
|
60
|
+
type FilledField = Extract<PageResult, { readonly ok: true }>;
|
|
61
|
+
|
|
62
|
+
// Writes `value` through the page's native value setter (the fill engine in page/fill.page.ts) and returns what the
|
|
63
|
+
// field holds afterwards. browser_type falls back to it when typing did not produce the value; recipe "fill" steps
|
|
64
|
+
// (recorded by the old browser_fill and by browser_fill_form) replay through it.
|
|
65
|
+
export const nativeFill = async (
|
|
66
|
+
client: BrowserClient,
|
|
67
|
+
t: Target,
|
|
68
|
+
value: string,
|
|
69
|
+
): Promise<Result<FilledField, ToolErr>> => {
|
|
70
|
+
const r = await runOnElement(client, { ref: t.ref, selector: t.selector, functionDeclaration: FILL_FN, args: [value, FILL_OPTIONS] });
|
|
71
|
+
if (!r.success) return r;
|
|
72
|
+
const narrowed = narrow(r.data, t, "browser_fill", isPageResult);
|
|
73
|
+
if (!narrowed.success) return narrowed;
|
|
74
|
+
const res = narrowed.data;
|
|
75
|
+
if (res.ok) return ok(res);
|
|
76
|
+
const target = targetLabel(t);
|
|
77
|
+
const details = { ...targetDetails(t), ...(res.tag !== undefined ? { tag: res.tag } : {}) };
|
|
78
|
+
if (res.code === "select_rejected") {
|
|
79
|
+
return err({
|
|
80
|
+
kind: "invalid_state",
|
|
81
|
+
message: `Element <select> is not fillable (not an input, textarea, or contenteditable): ${target}. For <select> use browser_select_option.`,
|
|
82
|
+
details,
|
|
83
|
+
});
|
|
84
|
+
}
|
|
85
|
+
return err({ kind: "invalid_state", message: `Could not fill ${target}: ${res.reason}`, details });
|
|
86
|
+
};
|
|
87
|
+
|
|
88
|
+
// Not registered for the agent any more (browser_type replaces it); kept so saved recipes' "fill" steps replay.
|
|
59
89
|
export const fillTool = defineBrowserTool({
|
|
60
90
|
name: "browser_fill",
|
|
61
91
|
label: "Browser Fill",
|
|
62
92
|
description:
|
|
63
|
-
"
|
|
64
|
-
promptSnippet: "
|
|
65
|
-
promptGuidelines: [
|
|
66
|
-
"Pass the desired value; no browser_click is needed first.",
|
|
67
|
-
"For a checkbox or radio pass 'true' or 'false' — it toggles `checked`, not the submit value. browser_set_checked does the same thing with a boolean.",
|
|
68
|
-
"Use browser_type instead only for keystroke-sensitive widgets (autocomplete, masked/segmented inputs).",
|
|
69
|
-
"Fill sets the value but does not focus or submit. To submit a tag/autocomplete input, follow with browser_dispatch_key({ ref, key: 'Enter' }) — not browser_press_key, which targets the focused element and may miss this field.",
|
|
70
|
-
"For a custom (div-based) dropdown, browser_fill won't validate against its option list — open it, run browser_snapshot, then browser_click the option's ref.",
|
|
71
|
-
"Confirm the field value browser_fill returns matches what you intended.",
|
|
72
|
-
],
|
|
93
|
+
"Recipe replay only: set a form field's value through the native setter, or tick a checkbox/radio with 'true'/'false'.",
|
|
94
|
+
promptSnippet: "Set a form field's value (recipe replay)",
|
|
95
|
+
promptGuidelines: [],
|
|
73
96
|
parameters: FillArgs,
|
|
74
97
|
concurrency: "serialized",
|
|
75
98
|
async handler(args, { client }): Promise<Result<ToolOk, ToolErr>> {
|
|
76
|
-
const
|
|
77
|
-
if (!
|
|
78
|
-
const
|
|
79
|
-
if (!narrowed.success) return narrowed;
|
|
80
|
-
const res = narrowed.data;
|
|
81
|
-
const target = targetLabel(args);
|
|
82
|
-
if (!res.ok) {
|
|
83
|
-
const details = { ref: args.ref, selector: args.selector, ...(res.tag !== undefined ? { tag: res.tag } : {}) };
|
|
84
|
-
if (res.code === "select_rejected") {
|
|
85
|
-
return err({
|
|
86
|
-
kind: "invalid_state",
|
|
87
|
-
message: `Element <select> is not fillable (not an input, textarea, or contenteditable): ${target}. For <select> use browser_select_option.`,
|
|
88
|
-
details,
|
|
89
|
-
});
|
|
90
|
-
}
|
|
91
|
-
return err({ kind: "invalid_state", message: `Could not fill ${target}: ${res.reason}`, details });
|
|
92
|
-
}
|
|
99
|
+
const filled = await nativeFill(client, args, args.value);
|
|
100
|
+
if (!filled.success) return filled;
|
|
101
|
+
const res = filled.data;
|
|
93
102
|
const verified = "checked" in res ? res.checked : res.value;
|
|
94
103
|
const shown = "checked" in res ? `checked=${res.checked}` : JSON.stringify(res.value);
|
|
95
104
|
const diff = await interactiveDiff(client);
|
|
96
105
|
return ok({
|
|
97
|
-
text: `Filled ${
|
|
106
|
+
text: `Filled ${targetLabel(args)} = ${shown}${diff}`,
|
|
98
107
|
details: { ...targetDetails(args), value: args.value, verified, tag: res.tag, kind: res.kind },
|
|
99
108
|
});
|
|
100
109
|
},
|
|
@@ -119,10 +128,10 @@ export const focusTool = defineBrowserTool({
|
|
|
119
128
|
name: "browser_focus",
|
|
120
129
|
label: "Browser Focus",
|
|
121
130
|
description:
|
|
122
|
-
"Focus an element via the DOM .focus() method — deterministic, no coordinate accuracy needed. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`.
|
|
131
|
+
"Focus an element via the DOM .focus() method — deterministic, no coordinate accuracy needed. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`.",
|
|
123
132
|
promptSnippet: "Focus an element by ref (preferred) or selector",
|
|
124
133
|
promptGuidelines: [
|
|
125
|
-
"
|
|
134
|
+
"To enter text, pass the ref straight to browser_type — it focuses the field itself.",
|
|
126
135
|
],
|
|
127
136
|
parameters: FocusArgs,
|
|
128
137
|
concurrency: "serialized",
|
package/src/domains/js.ts
CHANGED
|
@@ -40,6 +40,13 @@ const prettyJson = (text: string): string => {
|
|
|
40
40
|
const ExecuteJsArgs = Type.Object({
|
|
41
41
|
expression: Type.String({ description: "A JavaScript expression, or a function body: a body with a top-level `return` (after any statements) runs inside an async function and returns that value." }),
|
|
42
42
|
targetId: Type.Optional(Type.String({ description: "Optional iframe targetId; default = current page." })),
|
|
43
|
+
timeoutMs: Type.Optional(
|
|
44
|
+
Type.Integer({
|
|
45
|
+
minimum: 1_000,
|
|
46
|
+
maximum: 120_000,
|
|
47
|
+
description: "How long to wait for the result (an awaited promise included). Default 30000, max 120000. Raise it for scripts that loop with waits.",
|
|
48
|
+
}),
|
|
49
|
+
),
|
|
43
50
|
});
|
|
44
51
|
|
|
45
52
|
const ExecuteJsDetailsSchema = Type.Object({
|
|
@@ -51,6 +58,8 @@ const ExecuteJsDetailsSchema = Type.Object({
|
|
|
51
58
|
|
|
52
59
|
export type ExecuteJsDetails = DetailsOf<typeof ExecuteJsDetailsSchema>;
|
|
53
60
|
|
|
61
|
+
const EXECUTE_JS_DEFAULT_TIMEOUT_MS = 30_000;
|
|
62
|
+
|
|
54
63
|
export const executeJsTool = defineBrowserTool({
|
|
55
64
|
name: "browser_execute_js",
|
|
56
65
|
label: "Browser Execute JS",
|
|
@@ -63,13 +72,22 @@ export const executeJsTool = defineBrowserTool({
|
|
|
63
72
|
"For 'is this element visible / enabled / checked / focused', a one-liner here beats a screenshot every time.",
|
|
64
73
|
"For iframes, first run `Object.values(document.querySelectorAll('iframe'))` to find the iframe, then pass its targetId.",
|
|
65
74
|
"Result must be JSON-serializable (Runtime.evaluate returnByValue=true).",
|
|
75
|
+
"A script that awaits (loops with setTimeout, waits for UI) gets 30s by default; pass timeoutMs (max 120000) for longer ones.",
|
|
66
76
|
],
|
|
67
77
|
parameters: ExecuteJsArgs,
|
|
68
78
|
details: ExecuteJsDetailsSchema,
|
|
69
79
|
concurrency: "parallel",
|
|
70
80
|
async handler(args, { client }): Promise<Result<ToolOk<ExecuteJsDetails>, ToolErr>> {
|
|
71
|
-
const
|
|
72
|
-
|
|
81
|
+
const timeoutMs = args.timeoutMs ?? EXECUTE_JS_DEFAULT_TIMEOUT_MS;
|
|
82
|
+
const r = await evalJs(client, args.expression, args.targetId, { timeoutMs });
|
|
83
|
+
if (!r.success) {
|
|
84
|
+
if (r.error.kind !== "timeout" && !/timeout/i.test(r.error.message)) return r;
|
|
85
|
+
return err({
|
|
86
|
+
kind: "timeout",
|
|
87
|
+
message: `${r.error.message}. The script may have partly run before the wait ran out — check the page's state before retrying. Pass a larger timeoutMs (max 120000) or split the work into several calls.`,
|
|
88
|
+
details: { ...(r.error.details ?? {}), timeoutMs },
|
|
89
|
+
});
|
|
90
|
+
}
|
|
73
91
|
const valueStr = r.data === undefined ? "undefined" : JSON.stringify(r.data);
|
|
74
92
|
// The full value lives in the text (and in fullOutputPath when truncated); details carry only what the
|
|
75
93
|
// renderer needs so a large result is not persisted twice.
|