pi-browser-harness 0.14.1 → 0.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,31 @@
2
2
 
3
3
  All notable changes to pi-browser-harness will be documented in this file.
4
4
 
5
+ ## 0.16.0 — 2026-11-07
6
+
7
+ ### Added
8
+
9
+ - **`browser_web_search` falls back across search engines.** With no `engine` given it tries Google, then Bing, DuckDuckGo and Brave, until one returns results, so a Google CAPTCHA or markup change no longer breaks search. `details.engine` says which engine answered and `details.fallbacks` lists the ones that failed first. When every engine fails the error is `all_engines_failed` with each engine's reason; when none found anything it stays `no_results`.
10
+ - **`engine` parameter.** Name one engine (`google`, `bing`, `duckduckgo`, `brave`) to search it alone, with no fallback.
11
+ - **Pluggable engines.** Each engine is a self-contained module mounted into a registry (`createEngineRegistry().mount(engine)`) that is injected into `createWebSearchTool`; mount order is the fallback order. Adding an engine is one module and one `mount`.
12
+
13
+ ## 0.15.1 — 2026-11-07
14
+
15
+ - Re-release of 0.15.0, whose npm publish failed (the provenance log returned 502, then rejected the retry as a duplicate). No code changes.
16
+
17
+ ## 0.15.0 — 2026-11-07
18
+
19
+ ### Changed
20
+
21
+ - **`browser_fill` and `browser_type` are one tool: `browser_type`.** Pass a `ref` (or `selector`) and it focuses the field, replaces its value (`clear:false` appends; `text: ""` empties it), types with real input events so autocomplete lists open, and reports the value read back. When typing does not produce the value (date or masked inputs) it sets the value directly and says so. With no ref it types into the focused element, appending as before. Saved recipes with `fill` steps still replay.
22
+
23
+ ### Fixed
24
+
25
+ - **`browser_click` no longer clicks into nothing.** A ref is scrolled into view (nested scrolling lists included) and measured at click time; a point still outside the viewport, or an `x`/`y` outside it, is refused with the reason. When something covers the element (an overlay, a sticky header) the result says what was clicked instead. An `x`/`y` click names the element it landed on.
26
+ - **`browser_press_key` runs Cmd/Ctrl shortcuts.** Cmd/Ctrl + A, C, X, V, Z, Y now carry Chrome's editing command, so select-all, copy, paste and undo work. Letter keys send the right `code` (`KeyA`) and key code (65).
27
+ - **`browser_execute_js` waits up to 30s by default, and `timeoutMs` (max 120s) raises it.** It used to stop at 10s, mid-way through scripts that loop with waits. A timeout now says the script may have partly run.
28
+ - **`browser_screenshot` says how its pixels map to click coordinates** when the image is not 1:1 with CSS pixels (2× on Retina, or after `maxDim`).
29
+
5
30
  ## 0.14.1 — 2026-11-07
6
31
 
7
32
  ### Fixed
package/README.md CHANGED
@@ -18,7 +18,7 @@ no MCP hop, no second browser, no extra API keys.
18
18
  | **Browser** | Your running Chrome — real profile, live sessions | Launches a clean instance | Launches a clean instance | Launches a clean instance |
19
19
  | **Element targeting** | AX-tree refs (`[eN]`) that re-resolve at action time; coordinate and selector fallbacks | Selectors / AX locators | CSS selectors | LLM-inferred selectors |
20
20
  | **Input dispatch** | Compositor-level CDP input — works through iframes, shadow DOM, cross-origin frames | DOM-level | DOM-level | DOM-level |
21
- | **Framework-safe writes** | `browser_fill` writes via the native setter and fires bubbling events, so controlled inputs keep the value | Native fill | Manual | Native fill |
21
+ | **Framework-safe writes** | `browser_type` types with real input events, reads the value back, and falls back to the native setter when typing doesn't take | Native fill | Manual | Native fill |
22
22
  | **Transport** | Native pi extension, in-process | MCP / JSON-RPC round trip | MCP / JSON-RPC round trip | In-process + its own LLM calls |
23
23
  | **Extra credentials** | None | None | None | Its own model API key |
24
24
  | **Concurrency** | Reads run in parallel; mutations auto-serialize through a shared mutex | Sequential | Sequential | Sequential |
@@ -76,10 +76,12 @@ interactive element carries a stable ref (`[eN]`) plus click coordinates `@(x,y)
76
76
  [e12] button "Create account" @(456,412)
77
77
  ```
78
78
 
79
- Pass the ref — not the coordinates — to `browser_click`, `browser_fill`, `browser_select_option`,
79
+ Pass the ref — not the coordinates — to `browser_click`, `browser_type`, `browser_select_option`,
80
80
  `browser_focus`, `browser_upload_file`, or `browser_dispatch_key`. Refs are keyed to element
81
81
  identity, so they re-resolve position at action time and survive the re-renders that SPA forms
82
- trigger on every keystroke. Coordinates go stale the moment the layout reflows.
82
+ trigger on every keystroke. Coordinates go stale the moment the layout reflows. `browser_click` by ref
83
+ scrolls the element into view first (inside scrolling lists too) and warns when something covers it;
84
+ an `x`/`y` outside the viewport is refused instead of clicking nothing.
83
85
 
84
86
  Three things follow from that:
85
87
 
@@ -117,7 +119,7 @@ Three things follow from that:
117
119
 
118
120
  | Tool | Purpose |
119
121
  |---|---|
120
- | `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real Google SERP in your Chrome. No API key. Links only. |
122
+ | `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real search results page in your Chrome. No API key. Links only. Walks the fallback ladder Google → Bing → DuckDuckGo → Brave until one answers, or searches only the `engine` you name. |
121
123
  | `browser_read_page` | Reader mode. A `url` (opened and closed in its own tab) or an owned `targetId` → clean article text, boilerplate stripped. |
122
124
 
123
125
  Both run in isolated tabs and never disturb your current page.
@@ -127,10 +129,9 @@ Both run in isolated tabs and never disturb your current page.
127
129
  | Tool | Purpose |
128
130
  |---|---|
129
131
  | `browser_click` | Click by `ref` (preferred) or viewport `x`/`y`. Compositor-level, so iframes and shadow DOM work. |
130
- | `browser_fill` | Set an input, textarea, or contenteditable. Fires bubbling `input`/`change`, returns the value written. |
131
132
  | `browser_select_option` | Choose in a native `<select>` by value, label, or index. Returns available options on no match. |
132
133
  | `browser_focus` | Deterministic `.focus()` — no coordinate accuracy needed. |
133
- | `browser_type` | Type into the focused element. For keystroke-sensitive widgets; prefer `browser_fill` otherwise. |
134
+ | `browser_type` | Enter text into a field by `ref`/selector (replaces its value; `clear:false` appends) or into the focused element. Real input events, so autocomplete opens; returns the value read back. |
134
135
  | `browser_press_key` | Press a key with an optional modifier bitfield. |
135
136
  | `browser_dispatch_key` | Synthetic DOM `KeyboardEvent` on one element, for pages that ignore raw CDP keys. |
136
137
  | `browser_scroll` | Scroll by delta. W3C wheel convention: positive `deltaY` = down. |
@@ -202,7 +203,7 @@ Wait, snapshot once, then act by ref. No screenshots anywhere in this loop.
202
203
  ```
203
204
  browser_wait_for({ selector: "#email" })
204
205
  browser_snapshot() // → [e7] textbox, [e9] combobox, [e12] button
205
- browser_fill({ ref: "e7", value: "a@b.com" })
206
+ browser_type({ ref: "e7", text: "a@b.com" })
206
207
  browser_select_option({ ref: "e9", label: "India" })
207
208
  browser_click({ ref: "e12" }) // re-resolves position even after reflow
208
209
  ```
@@ -244,8 +245,14 @@ browser_network_requests({ // did a request 4xx/5xx?
244
245
  ```
245
246
  browser_web_search({ query: "chrome devtools protocol overview", limit: 5 })
246
247
  browser_read_page({ url: "<the best result's url>" })
248
+
249
+ // Search one engine only, with no fallback:
250
+ browser_web_search({ query: "chrome devtools protocol overview", engine: "bing" })
247
251
  ```
248
252
 
253
+ `details.engine` names the engine that answered, and `details.fallbacks` lists any engines that failed
254
+ first (CAPTCHA, no results, markup change, load failure).
255
+
249
256
  Both tools open their own isolated tab, so neither disturbs the tab you are working in.
250
257
 
251
258
  ### Keyboard modifiers
@@ -352,7 +359,7 @@ re-runnable.
352
359
 
353
360
  Another extension can add its own `browser_*` tools to this harness. Contributed tools share the
354
361
  same client, mutation mutex, tab ownership, and lane prompt as the built-ins, so a `serialized` tool
355
- you add queues behind `browser_click` exactly as `browser_fill` does.
362
+ you add queues behind `browser_click` exactly as `browser_type` does.
356
363
 
357
364
  Define the tool with `defineBrowserTool` and hand it to `registerBrowserTools` at your extension's
358
365
  load time, before pi emits `session_start`. The harness registers it with pi on the next
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-browser-harness",
3
- "version": "0.14.1",
3
+ "version": "0.16.0",
4
4
  "description": "Browser control extension for pi — navigate, click, type, screenshot, and extract data from real Chrome via CDP",
5
5
  "keywords": [
6
6
  "pi-package",
@@ -61,7 +61,6 @@ Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
61
61
  | `browser_download` | serialized | Configure download directory |
62
62
  | `browser_drag_and_drop` | serialized | Drag and drop between two coordinates |
63
63
  | `browser_execute_js` | parallel | Execute JS in the page — surgical DOM reads (preferred over screenshot) |
64
- | `browser_fill` | serialized | Fill a form field by ref (preferred) or selector (works with React/Vue controlled inputs) |
65
64
  | `browser_fill_form` | serialized | Fill many form fields at once by ref — React-safe, self-confirming |
66
65
  | `browser_focus` | serialized | Focus an element by ref (preferred) or selector |
67
66
  | `browser_go_back` | serialized | Go back one page |
@@ -89,7 +88,7 @@ Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
89
88
  | `browser_setup` | parallel | Initialize browser connection |
90
89
  | `browser_snapshot` | parallel | Get accessibility-tree snapshot with stable element refs (default for page inspection) |
91
90
  | `browser_switch_tab` | serialized | Switch tabs |
92
- | `browser_type` | serialized | Type text into the focused element |
91
+ | `browser_type` | serialized | Type text into a field by ref (preferred), selector, or the focused element |
93
92
  | `browser_upload_file` | serialized | Upload a file to a file input by ref (preferred) or selector |
94
93
  | `browser_viewport_resize` | serialized | Resize the viewport (responsive testing) |
95
94
  | `browser_wait` | parallel | Wait N seconds |
@@ -271,6 +271,18 @@ export const COMMANDS = {
271
271
  }),
272
272
  Type.Object({ data: Type.String(), stream: Type.Optional(Type.String()) }),
273
273
  ),
274
+ "Page.getLayoutMetrics": cmd(
275
+ Empty,
276
+ Type.Object(
277
+ {
278
+ cssLayoutViewport: Type.Object(
279
+ { clientWidth: Type.Number(), clientHeight: Type.Number() },
280
+ { additionalProperties: true },
281
+ ),
282
+ },
283
+ { additionalProperties: true },
284
+ ),
285
+ ),
274
286
  "Page.handleJavaScriptDialog": cmd(
275
287
  Type.Object({ accept: Type.Boolean(), promptText: Type.Optional(Type.String()) }),
276
288
  Empty,
@@ -322,6 +334,22 @@ export const COMMANDS = {
322
334
  ),
323
335
  }),
324
336
  ),
337
+ "DOM.scrollIntoViewIfNeeded": cmd(
338
+ Type.Object({ nodeId: Type.Optional(Type.Number()), backendNodeId: Type.Optional(Type.Number()) }),
339
+ Empty,
340
+ ),
341
+ "DOM.getNodeForLocation": cmd(
342
+ Type.Object({
343
+ x: Type.Integer(),
344
+ y: Type.Integer(),
345
+ includeUserAgentShadowDOM: Type.Optional(Type.Boolean()),
346
+ ignorePointerEventsNone: Type.Optional(Type.Boolean()),
347
+ }),
348
+ Type.Object(
349
+ { backendNodeId: Type.Number(), frameId: Type.Optional(Type.String()), nodeId: Type.Optional(Type.Number()) },
350
+ { additionalProperties: true },
351
+ ),
352
+ ),
325
353
  "DOM.resolveNode": cmd(
326
354
  Type.Object({ nodeId: Type.Optional(Type.Number()), backendNodeId: Type.Optional(Type.Number()) }),
327
355
  Type.Object({ object: RemoteObject }),
package/src/client.ts CHANGED
@@ -59,7 +59,7 @@ export type BrowserClient = {
59
59
  stop(): Promise<void>;
60
60
  detach(): Promise<void>;
61
61
  closeOwnedTabs(): Promise<void>;
62
- evaluateJs(expression: string, sessionId?: string): Promise<Result<unknown, CdpError>>;
62
+ evaluateJs(expression: string, sessionId?: string, opts?: { readonly timeoutMs?: number }): Promise<Result<unknown, CdpError>>;
63
63
  pageInfo(): Promise<Result<PageInfoResult, CdpError>>;
64
64
  takeDialog(): DialogInfo | null;
65
65
  listTabs(includeInternal?: boolean): Promise<Result<ReadonlyArray<TabInfo>, CdpError>>;
@@ -260,9 +260,16 @@ export const createBrowserClient = (opts: BrowserClientOptions): BrowserClient =
260
260
  return running;
261
261
  };
262
262
 
263
- const evaluateJs = async (expression: string, sessionId?: string): Promise<Result<unknown, CdpError>> => {
263
+ const evaluateJs = async (
264
+ expression: string,
265
+ sessionId?: string,
266
+ opts?: { readonly timeoutMs?: number },
267
+ ): Promise<Result<unknown, CdpError>> => {
264
268
  const params = evaluateParams(expression);
265
- const r = sessionId ? await session.callOnTarget("Runtime.evaluate", params, sessionId) : await session.call("Runtime.evaluate", params);
269
+ const callOpts = opts?.timeoutMs !== undefined ? { timeoutMs: opts.timeoutMs } : {};
270
+ const r = sessionId
271
+ ? await session.callOnTarget("Runtime.evaluate", params, sessionId, callOpts)
272
+ : await session.call("Runtime.evaluate", params, callOpts);
266
273
  if (!r.success) return r;
267
274
  if (r.data.exceptionDetails !== undefined) {
268
275
  return err(cdpError("remote_error", `JS evaluation failed: ${JSON.stringify(r.data.exceptionDetails)}`, "Runtime.evaluate"));
@@ -1,7 +1,7 @@
1
1
  import { isCdpRawMessage } from "../cdp/types";
2
2
  import type { CdpRawMessage } from "../cdp/types";
3
3
  import type { WireRequest, WireResponse, WireEvent } from "./protocol";
4
- import { CDP_COMMAND_TIMEOUT_MS, DAEMON_RECONNECT_METHOD } from "./protocol";
4
+ import { DAEMON_RECONNECT_METHOD, commandTimeoutFor } from "./protocol";
5
5
  import { asString, isRecord } from "../util/guards";
6
6
  import { type ChromeConnectionDependencies, type ConnectionStatus, createChromeConnection } from "./connection";
7
7
 
@@ -246,6 +246,7 @@ export const createCdpBridge = (options: ChromeConnectionDependencies = {}): Cdp
246
246
  if (sessionId !== undefined) router.release(sessionId);
247
247
  }
248
248
 
249
+ const timeoutMs = commandTimeoutFor(req);
249
250
  const daemonId = mux.allocate(
250
251
  {
251
252
  clientId,
@@ -257,8 +258,8 @@ export const createCdpBridge = (options: ChromeConnectionDependencies = {}): Cdp
257
258
  (id) =>
258
259
  setTimeout(() => {
259
260
  mux.take(id);
260
- send(clientId, errorResponse(req.id, `Timeout after ${CDP_COMMAND_TIMEOUT_MS}ms: ${req.method}`));
261
- }, CDP_COMMAND_TIMEOUT_MS),
261
+ send(clientId, errorResponse(req.id, `Timeout after ${timeoutMs}ms: ${req.method}`));
262
+ }, timeoutMs),
262
263
  );
263
264
 
264
265
  const payload: Record<string, unknown> = {
@@ -6,6 +6,8 @@ export type WireRequest = {
6
6
  readonly method: string;
7
7
  readonly params?: Record<string, unknown>;
8
8
  readonly sessionId?: string;
9
+ // How long the daemon waits for Chrome's answer; see commandTimeoutFor.
10
+ readonly timeoutMs?: number;
9
11
  };
10
12
 
11
13
  export type WireResponse = {
@@ -71,6 +73,7 @@ const isWireMessage = (v: unknown): v is WireMessage => {
71
73
  if (typeof v["id"] !== "number") return false;
72
74
  if (typeof v["method"] !== "string") return false;
73
75
  if (!isOptionalRecord(v["params"])) return false;
76
+ if (v["timeoutMs"] !== undefined && typeof v["timeoutMs"] !== "number") return false;
74
77
  return isOptionalString(v["sessionId"]);
75
78
  }
76
79
  case "response": {
@@ -138,3 +141,11 @@ export const CDP_RECONNECT_COOLDOWN_MAX_MS = 5 * 60 * 1000;
138
141
  export const DAEMON_RECONNECT_METHOD = "Daemon.reconnect";
139
142
 
140
143
  export const CDP_COMMAND_TIMEOUT_MS = 10_000;
144
+
145
+ // The longest a caller may ask the daemon to wait for one command (browser_execute_js's timeoutMs).
146
+ export const CDP_COMMAND_TIMEOUT_MAX_MS = 120_000;
147
+
148
+ // A request may lengthen the daemon's wait (a long-running script) but never shorten it below the default: short
149
+ // client-side probes already give up on their own, and a shorter daemon timer would only race them.
150
+ export const commandTimeoutFor = (req: { readonly timeoutMs?: number }): number =>
151
+ Math.min(Math.max(req.timeoutMs ?? CDP_COMMAND_TIMEOUT_MS, CDP_COMMAND_TIMEOUT_MS), CDP_COMMAND_TIMEOUT_MAX_MS);
@@ -194,6 +194,7 @@ export const createDaemonTransport = (clientId: string, socketPath: string = DAE
194
194
  method,
195
195
  params,
196
196
  ...(sessionId !== null ? { sessionId } : {}),
197
+ ...(opts?.timeoutMs !== undefined ? { timeoutMs: opts.timeoutMs } : {}),
197
198
  };
198
199
 
199
200
  return sendWithTimeout(pending, id, method, timeoutMs, "Daemon", () => sock.write(serialize(req) + "\n"));
@@ -55,8 +55,9 @@ export const evalJs = async (
55
55
  client: BrowserClient,
56
56
  expression: string,
57
57
  sessionId?: string,
58
+ opts?: { readonly timeoutMs?: number },
58
59
  ): Promise<Result<unknown, ToolErr>> => {
59
- const r = await client.evaluateJs(expression, sessionId);
60
+ const r = await client.evaluateJs(expression, sessionId, opts);
60
61
  if (!r.success) return err(cdpErrToToolErr(r.error, "Runtime.evaluate"));
61
62
  return r;
62
63
  };
@@ -55,7 +55,7 @@ export const clickTool = defineBrowserTool({
55
55
  name: "browser_click",
56
56
  label: "Browser Click",
57
57
  description:
58
- "Click an element. PREFERRED: pass `ref` (e.g. 'e12') from browser_snapshot — it re-resolves the element's position at click time, so it works even after the page re-renders and moves things. Fallback: pass viewport CSS-pixel `x`/`y`. Compositor-level click works through iframes, shadow DOM, and cross-origin content. After clicking, a compact diff of page changes is appended.",
58
+ "Click an element. PREFERRED: pass `ref` (e.g. 'e12') from browser_snapshot — it scrolls the element into view (inside scrolling lists too), measures it at click time, and warns if something covers it. Fallback: pass viewport CSS-pixel `x`/`y` (refused when outside the viewport; the result names the element that was hit). Compositor-level click works through iframes, shadow DOM, and cross-origin content. After clicking, a compact diff of page changes is appended.",
59
59
  promptSnippet: "Click an element by ref (preferred) or pixel coordinates",
60
60
  promptGuidelines: [
61
61
  "If a click doesn't register, run pi with --browser-debug-clicks (or set BH_DEBUG_CLICKS=1) to get annotated screenshots. For React/Vue components ignoring clicks, try browser_dispatch_key.",
@@ -69,7 +69,10 @@ export const clickTool = defineBrowserTool({
69
69
 
70
70
  const resolved = await resolveTarget(client, args);
71
71
  if (!resolved.success) return resolved;
72
- const { x, y, label: target } = resolved.data;
72
+ const { x, y, label, covered, hit } = resolved.data;
73
+ const target = label
74
+ + (hit !== undefined ? ` on ${hit}` : "")
75
+ + (covered !== undefined ? `\n⚠ The point was covered by ${covered}, so the click went to that element, not [${args.ref ?? ""}]. Close the overlay or scroll, then re-snapshot.` : "");
73
76
 
74
77
  const clicked = await dispatchClick(client, x, y, button, count);
75
78
  if (!clicked.success) return clicked;
@@ -2,7 +2,8 @@ import { Type } from "typebox";
2
2
  import { type Result, err, ok } from "../util/result";
3
3
  import { defineBrowserTool, type JsonObject, type ToolErr, type ToolOk } from "../util/tool";
4
4
  import { detailsOf, type ElementOutcome, isPageResult, resolveAndCall, runOnElement } from "./element-call";
5
- import { fillField, type FillOptions } from "./page/fill.page";
5
+ import { fillField, type FillOptions, type PageResult } from "./page/fill.page";
6
+ import type { BrowserClient } from "../client";
6
7
  import { type FocusResult, focusElement, type SelectOption, type SelectResult, type SelectWanted, selectOption, setChecked } from "./page/form.page";
7
8
  import { elementFunctionSource } from "./page/inject";
8
9
  import { interactiveDiff } from "./ref-resolve";
@@ -56,45 +57,53 @@ const FillArgs = Type.Object({
56
57
  const FILL_FN = elementFunctionSource(fillField);
57
58
  const FILL_OPTIONS: FillOptions = { rejectSelect: true, focusFirst: false };
58
59
 
60
+ type FilledField = Extract<PageResult, { readonly ok: true }>;
61
+
62
+ // Writes `value` through the page's native value setter (the fill engine in page/fill.page.ts) and returns what the
63
+ // field holds afterwards. browser_type falls back to it when typing did not produce the value; recipe "fill" steps
64
+ // (recorded by the old browser_fill and by browser_fill_form) replay through it.
65
+ export const nativeFill = async (
66
+ client: BrowserClient,
67
+ t: Target,
68
+ value: string,
69
+ ): Promise<Result<FilledField, ToolErr>> => {
70
+ const r = await runOnElement(client, { ref: t.ref, selector: t.selector, functionDeclaration: FILL_FN, args: [value, FILL_OPTIONS] });
71
+ if (!r.success) return r;
72
+ const narrowed = narrow(r.data, t, "browser_fill", isPageResult);
73
+ if (!narrowed.success) return narrowed;
74
+ const res = narrowed.data;
75
+ if (res.ok) return ok(res);
76
+ const target = targetLabel(t);
77
+ const details = { ...targetDetails(t), ...(res.tag !== undefined ? { tag: res.tag } : {}) };
78
+ if (res.code === "select_rejected") {
79
+ return err({
80
+ kind: "invalid_state",
81
+ message: `Element <select> is not fillable (not an input, textarea, or contenteditable): ${target}. For <select> use browser_select_option.`,
82
+ details,
83
+ });
84
+ }
85
+ return err({ kind: "invalid_state", message: `Could not fill ${target}: ${res.reason}`, details });
86
+ };
87
+
88
+ // Not registered for the agent any more (browser_type replaces it); kept so saved recipes' "fill" steps replay.
59
89
  export const fillTool = defineBrowserTool({
60
90
  name: "browser_fill",
61
91
  label: "Browser Fill",
62
92
  description:
63
- "Fill a form field (input/textarea/contenteditable), or tick a checkbox/radio by passing 'true'/'false'. PREFERRED: pass `ref` (e.g. 'e7') from browser_snapshot — survives re-renders. Fallback: a CSS `selector`. Writes through the native value setter and fires bubbling 'input'/'change' events, so React/Vue/Angular controlled components and rich-text editors update correctly — unlike browser_type. Refuses a disabled field rather than writing a value the page will ignore. Returns the field's value after writing, plus a compact diff of page changes.",
64
- promptSnippet: "Fill a form field by ref (preferred) or selector (works with React/Vue controlled inputs)",
65
- promptGuidelines: [
66
- "Pass the desired value; no browser_click is needed first.",
67
- "For a checkbox or radio pass 'true' or 'false' — it toggles `checked`, not the submit value. browser_set_checked does the same thing with a boolean.",
68
- "Use browser_type instead only for keystroke-sensitive widgets (autocomplete, masked/segmented inputs).",
69
- "Fill sets the value but does not focus or submit. To submit a tag/autocomplete input, follow with browser_dispatch_key({ ref, key: 'Enter' }) — not browser_press_key, which targets the focused element and may miss this field.",
70
- "For a custom (div-based) dropdown, browser_fill won't validate against its option list — open it, run browser_snapshot, then browser_click the option's ref.",
71
- "Confirm the field value browser_fill returns matches what you intended.",
72
- ],
93
+ "Recipe replay only: set a form field's value through the native setter, or tick a checkbox/radio with 'true'/'false'.",
94
+ promptSnippet: "Set a form field's value (recipe replay)",
95
+ promptGuidelines: [],
73
96
  parameters: FillArgs,
74
97
  concurrency: "serialized",
75
98
  async handler(args, { client }): Promise<Result<ToolOk, ToolErr>> {
76
- const r = await runOnElement(client, { ref: args.ref, selector: args.selector, functionDeclaration: FILL_FN, args: [args.value, FILL_OPTIONS] });
77
- if (!r.success) return r;
78
- const narrowed = narrow(r.data, args, "browser_fill", isPageResult);
79
- if (!narrowed.success) return narrowed;
80
- const res = narrowed.data;
81
- const target = targetLabel(args);
82
- if (!res.ok) {
83
- const details = { ref: args.ref, selector: args.selector, ...(res.tag !== undefined ? { tag: res.tag } : {}) };
84
- if (res.code === "select_rejected") {
85
- return err({
86
- kind: "invalid_state",
87
- message: `Element <select> is not fillable (not an input, textarea, or contenteditable): ${target}. For <select> use browser_select_option.`,
88
- details,
89
- });
90
- }
91
- return err({ kind: "invalid_state", message: `Could not fill ${target}: ${res.reason}`, details });
92
- }
99
+ const filled = await nativeFill(client, args, args.value);
100
+ if (!filled.success) return filled;
101
+ const res = filled.data;
93
102
  const verified = "checked" in res ? res.checked : res.value;
94
103
  const shown = "checked" in res ? `checked=${res.checked}` : JSON.stringify(res.value);
95
104
  const diff = await interactiveDiff(client);
96
105
  return ok({
97
- text: `Filled ${target} = ${shown}${diff}`,
106
+ text: `Filled ${targetLabel(args)} = ${shown}${diff}`,
98
107
  details: { ...targetDetails(args), value: args.value, verified, tag: res.tag, kind: res.kind },
99
108
  });
100
109
  },
@@ -119,10 +128,10 @@ export const focusTool = defineBrowserTool({
119
128
  name: "browser_focus",
120
129
  label: "Browser Focus",
121
130
  description:
122
- "Focus an element via the DOM .focus() method — deterministic, no coordinate accuracy needed. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`. Use before browser_type when a click might miss the field.",
131
+ "Focus an element via the DOM .focus() method — deterministic, no coordinate accuracy needed. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`.",
123
132
  promptSnippet: "Focus an element by ref (preferred) or selector",
124
133
  promptGuidelines: [
125
- "For simply setting a value, prefer browser_fill (it doesn't need a separate focus step).",
134
+ "To enter text, pass the ref straight to browser_type — it focuses the field itself.",
126
135
  ],
127
136
  parameters: FocusArgs,
128
137
  concurrency: "serialized",
package/src/domains/js.ts CHANGED
@@ -40,6 +40,13 @@ const prettyJson = (text: string): string => {
40
40
  const ExecuteJsArgs = Type.Object({
41
41
  expression: Type.String({ description: "A JavaScript expression, or a function body: a body with a top-level `return` (after any statements) runs inside an async function and returns that value." }),
42
42
  targetId: Type.Optional(Type.String({ description: "Optional iframe targetId; default = current page." })),
43
+ timeoutMs: Type.Optional(
44
+ Type.Integer({
45
+ minimum: 1_000,
46
+ maximum: 120_000,
47
+ description: "How long to wait for the result (an awaited promise included). Default 30000, max 120000. Raise it for scripts that loop with waits.",
48
+ }),
49
+ ),
43
50
  });
44
51
 
45
52
  const ExecuteJsDetailsSchema = Type.Object({
@@ -51,6 +58,8 @@ const ExecuteJsDetailsSchema = Type.Object({
51
58
 
52
59
  export type ExecuteJsDetails = DetailsOf<typeof ExecuteJsDetailsSchema>;
53
60
 
61
+ const EXECUTE_JS_DEFAULT_TIMEOUT_MS = 30_000;
62
+
54
63
  export const executeJsTool = defineBrowserTool({
55
64
  name: "browser_execute_js",
56
65
  label: "Browser Execute JS",
@@ -63,13 +72,22 @@ export const executeJsTool = defineBrowserTool({
63
72
  "For 'is this element visible / enabled / checked / focused', a one-liner here beats a screenshot every time.",
64
73
  "For iframes, first run `Object.values(document.querySelectorAll('iframe'))` to find the iframe, then pass its targetId.",
65
74
  "Result must be JSON-serializable (Runtime.evaluate returnByValue=true).",
75
+ "A script that awaits (loops with setTimeout, waits for UI) gets 30s by default; pass timeoutMs (max 120000) for longer ones.",
66
76
  ],
67
77
  parameters: ExecuteJsArgs,
68
78
  details: ExecuteJsDetailsSchema,
69
79
  concurrency: "parallel",
70
80
  async handler(args, { client }): Promise<Result<ToolOk<ExecuteJsDetails>, ToolErr>> {
71
- const r = await evalJs(client, args.expression, args.targetId);
72
- if (!r.success) return r;
81
+ const timeoutMs = args.timeoutMs ?? EXECUTE_JS_DEFAULT_TIMEOUT_MS;
82
+ const r = await evalJs(client, args.expression, args.targetId, { timeoutMs });
83
+ if (!r.success) {
84
+ if (r.error.kind !== "timeout" && !/timeout/i.test(r.error.message)) return r;
85
+ return err({
86
+ kind: "timeout",
87
+ message: `${r.error.message}. The script may have partly run before the wait ran out — check the page's state before retrying. Pass a larger timeoutMs (max 120000) or split the work into several calls.`,
88
+ details: { ...(r.error.details ?? {}), timeoutMs },
89
+ });
90
+ }
73
91
  const valueStr = r.data === undefined ? "undefined" : JSON.stringify(r.data);
74
92
  // The full value lives in the text (and in fullOutputPath when truncated); details carry only what the
75
93
  // renderer needs so a large result is not persisted twice.