pi-browser-harness 0.14.1 → 0.15.1

package/CHANGELOG.md

@@ -2,6 +2,23 @@
 
 All notable changes to pi-browser-harness will be documented in this file.
 
+## 0.15.1 — 2026-11-07
+
+- Re-release of 0.15.0, whose npm publish failed (the provenance log returned 502, then rejected the retry as a duplicate). No code changes.
+
+## 0.15.0 — 2026-11-07
+
+### Changed
+
+- **`browser_fill` and `browser_type` are one tool: `browser_type`.** Pass a `ref` (or `selector`) and it focuses the field, replaces its value (`clear:false` appends; `text: ""` empties it), types with real input events so autocomplete lists open, and reports the value read back. When typing does not produce the value (date or masked inputs) it sets the value directly and says so. With no ref it types into the focused element, appending as before. Saved recipes with `fill` steps still replay.
+
+### Fixed
+
+- **`browser_click` no longer clicks into nothing.** A ref is scrolled into view (nested scrolling lists included) and measured at click time; a point still outside the viewport, or an `x`/`y` outside it, is refused with the reason. When something covers the element (an overlay, a sticky header) the result says what was clicked instead. An `x`/`y` click names the element it landed on.
+- **`browser_press_key` runs Cmd/Ctrl shortcuts.** Cmd/Ctrl + A, C, X, V, Z, Y now carry Chrome's editing command, so select-all, copy, paste and undo work. Letter keys send the right `code` (`KeyA`) and key code (65).
+- **`browser_execute_js` waits up to 30s by default, and `timeoutMs` (max 120s) raises it.** It used to stop at 10s, mid-way through scripts that loop with waits. A timeout now says the script may have partly run.
+- **`browser_screenshot` says how its pixels map to click coordinates** when the image is not 1:1 with CSS pixels (2× on Retina, or after `maxDim`).
+
 ## 0.14.1 — 2026-11-07
 
 ### Fixed

package/README.md

@@ -18,7 +18,7 @@ no MCP hop, no second browser, no extra API keys.
 | **Browser** | Your running Chrome — real profile, live sessions | Launches a clean instance | Launches a clean instance | Launches a clean instance |
 | **Element targeting** | AX-tree refs (`[eN]`) that re-resolve at action time; coordinate and selector fallbacks | Selectors / AX locators | CSS selectors | LLM-inferred selectors |
 | **Input dispatch** | Compositor-level CDP input — works through iframes, shadow DOM, cross-origin frames | DOM-level | DOM-level | DOM-level |
-| **Framework-safe writes** | `browser_fill` writes via the native setter and fires bubbling events, so controlled inputs keep the value | Native fill | Manual | Native fill |
+| **Framework-safe writes** | `browser_type` types with real input events, reads the value back, and falls back to the native setter when typing doesn't take | Native fill | Manual | Native fill |
 | **Transport** | Native pi extension, in-process | MCP / JSON-RPC round trip | MCP / JSON-RPC round trip | In-process + its own LLM calls |
 | **Extra credentials** | None | None | None | Its own model API key |
 | **Concurrency** | Reads run in parallel; mutations auto-serialize through a shared mutex | Sequential | Sequential | Sequential |
@@ -76,10 +76,12 @@ interactive element carries a stable ref (`[eN]`) plus click coordinates `@(x,y)
 [e12] button   "Create account" @(456,412)
 ```
 
-Pass the ref — not the coordinates — to `browser_click`, `browser_fill`, `browser_select_option`,
+Pass the ref — not the coordinates — to `browser_click`, `browser_type`, `browser_select_option`,
 `browser_focus`, `browser_upload_file`, or `browser_dispatch_key`. Refs are keyed to element
 identity, so they re-resolve position at action time and survive the re-renders that SPA forms
-trigger on every keystroke. Coordinates go stale the moment the layout reflows.
+trigger on every keystroke. Coordinates go stale the moment the layout reflows. `browser_click` by ref
+scrolls the element into view first (inside scrolling lists too) and warns when something covers it;
+an `x`/`y` outside the viewport is refused instead of clicking nothing.
 
 Three things follow from that:
 
@@ -127,10 +129,9 @@ Both run in isolated tabs and never disturb your current page.
 | Tool | Purpose |
 |---|---|
 | `browser_click` | Click by `ref` (preferred) or viewport `x`/`y`. Compositor-level, so iframes and shadow DOM work. |
-| `browser_fill` | Set an input, textarea, or contenteditable. Fires bubbling `input`/`change`, returns the value written. |
 | `browser_select_option` | Choose in a native `<select>` by value, label, or index. Returns available options on no match. |
 | `browser_focus` | Deterministic `.focus()` — no coordinate accuracy needed. |
-| `browser_type` | Type into the focused element. For keystroke-sensitive widgets; prefer `browser_fill` otherwise. |
+| `browser_type` | Enter text into a field by `ref`/selector (replaces its value; `clear:false` appends) or into the focused element. Real input events, so autocomplete opens; returns the value read back. |
 | `browser_press_key` | Press a key with an optional modifier bitfield. |
 | `browser_dispatch_key` | Synthetic DOM `KeyboardEvent` on one element, for pages that ignore raw CDP keys. |
 | `browser_scroll` | Scroll by delta. W3C wheel convention: positive `deltaY` = down. |
@@ -202,7 +203,7 @@ Wait, snapshot once, then act by ref. No screenshots anywhere in this loop.
 ```
 browser_wait_for({ selector: "#email" })
 browser_snapshot()                                  // → [e7] textbox, [e9] combobox, [e12] button
-browser_fill({ ref: "e7", value: "a@b.com" })
+browser_type({ ref: "e7", text: "a@b.com" })
 browser_select_option({ ref: "e9", label: "India" })
 browser_click({ ref: "e12" })                       // re-resolves position even after reflow
 ```
@@ -352,7 +353,7 @@ re-runnable.
 
 Another extension can add its own `browser_*` tools to this harness. Contributed tools share the
 same client, mutation mutex, tab ownership, and lane prompt as the built-ins, so a `serialized` tool
-you add queues behind `browser_click` exactly as `browser_fill` does.
+you add queues behind `browser_click` exactly as `browser_type` does.
 
 Define the tool with `defineBrowserTool` and hand it to `registerBrowserTools` at your extension's
 load time, before pi emits `session_start`. The harness registers it with pi on the next

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-browser-harness",
-  "version": "0.14.1",
+  "version": "0.15.1",
   "description": "Browser control extension for pi — navigate, click, type, screenshot, and extract data from real Chrome via CDP",
   "keywords": [
     "pi-package",

package/skills/pi-browser-harness/SKILL.md

@@ -61,7 +61,6 @@ Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
 | `browser_download` | serialized | Configure download directory |
 | `browser_drag_and_drop` | serialized | Drag and drop between two coordinates |
 | `browser_execute_js` | parallel | Execute JS in the page — surgical DOM reads (preferred over screenshot) |
-| `browser_fill` | serialized | Fill a form field by ref (preferred) or selector (works with React/Vue controlled inputs) |
 | `browser_fill_form` | serialized | Fill many form fields at once by ref — React-safe, self-confirming |
 | `browser_focus` | serialized | Focus an element by ref (preferred) or selector |
 | `browser_go_back` | serialized | Go back one page |
@@ -89,7 +88,7 @@ Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
 | `browser_setup` | parallel | Initialize browser connection |
 | `browser_snapshot` | parallel | Get accessibility-tree snapshot with stable element refs (default for page inspection) |
 | `browser_switch_tab` | serialized | Switch tabs |
-| `browser_type` | serialized | Type text into the focused element |
+| `browser_type` | serialized | Type text into a field by ref (preferred), selector, or the focused element |
 | `browser_upload_file` | serialized | Upload a file to a file input by ref (preferred) or selector |
 | `browser_viewport_resize` | serialized | Resize the viewport (responsive testing) |
 | `browser_wait` | parallel | Wait N seconds |

package/src/cdp/commands.ts

@@ -271,6 +271,18 @@ export const COMMANDS = {
     }),
     Type.Object({ data: Type.String(), stream: Type.Optional(Type.String()) }),
   ),
+  "Page.getLayoutMetrics": cmd(
+    Empty,
+    Type.Object(
+      {
+        cssLayoutViewport: Type.Object(
+          { clientWidth: Type.Number(), clientHeight: Type.Number() },
+          { additionalProperties: true },
+        ),
+      },
+      { additionalProperties: true },
+    ),
+  ),
   "Page.handleJavaScriptDialog": cmd(
     Type.Object({ accept: Type.Boolean(), promptText: Type.Optional(Type.String()) }),
     Empty,
@@ -322,6 +334,22 @@ export const COMMANDS = {
       ),
     }),
   ),
+  "DOM.scrollIntoViewIfNeeded": cmd(
+    Type.Object({ nodeId: Type.Optional(Type.Number()), backendNodeId: Type.Optional(Type.Number()) }),
+    Empty,
+  ),
+  "DOM.getNodeForLocation": cmd(
+    Type.Object({
+      x: Type.Integer(),
+      y: Type.Integer(),
+      includeUserAgentShadowDOM: Type.Optional(Type.Boolean()),
+      ignorePointerEventsNone: Type.Optional(Type.Boolean()),
+    }),
+    Type.Object(
+      { backendNodeId: Type.Number(), frameId: Type.Optional(Type.String()), nodeId: Type.Optional(Type.Number()) },
+      { additionalProperties: true },
+    ),
+  ),
   "DOM.resolveNode": cmd(
     Type.Object({ nodeId: Type.Optional(Type.Number()), backendNodeId: Type.Optional(Type.Number()) }),
     Type.Object({ object: RemoteObject }),

package/src/client.ts

@@ -59,7 +59,7 @@ export type BrowserClient = {
   stop(): Promise<void>;
   detach(): Promise<void>;
   closeOwnedTabs(): Promise<void>;
-  evaluateJs(expression: string, sessionId?: string): Promise<Result<unknown, CdpError>>;
+  evaluateJs(expression: string, sessionId?: string, opts?: { readonly timeoutMs?: number }): Promise<Result<unknown, CdpError>>;
   pageInfo(): Promise<Result<PageInfoResult, CdpError>>;
   takeDialog(): DialogInfo | null;
   listTabs(includeInternal?: boolean): Promise<Result<ReadonlyArray<TabInfo>, CdpError>>;
@@ -260,9 +260,16 @@ export const createBrowserClient = (opts: BrowserClientOptions): BrowserClient =
     return running;
   };
 
-  const evaluateJs = async (expression: string, sessionId?: string): Promise<Result<unknown, CdpError>> => {
+  const evaluateJs = async (
+    expression: string,
+    sessionId?: string,
+    opts?: { readonly timeoutMs?: number },
+  ): Promise<Result<unknown, CdpError>> => {
     const params = evaluateParams(expression);
-    const r = sessionId ? await session.callOnTarget("Runtime.evaluate", params, sessionId) : await session.call("Runtime.evaluate", params);
+    const callOpts = opts?.timeoutMs !== undefined ? { timeoutMs: opts.timeoutMs } : {};
+    const r = sessionId
+      ? await session.callOnTarget("Runtime.evaluate", params, sessionId, callOpts)
+      : await session.call("Runtime.evaluate", params, callOpts);
     if (!r.success) return r;
     if (r.data.exceptionDetails !== undefined) {
       return err(cdpError("remote_error", `JS evaluation failed: ${JSON.stringify(r.data.exceptionDetails)}`, "Runtime.evaluate"));

package/src/daemon/bridge.ts

@@ -1,7 +1,7 @@
 import { isCdpRawMessage } from "../cdp/types";
 import type { CdpRawMessage } from "../cdp/types";
 import type { WireRequest, WireResponse, WireEvent } from "./protocol";
-import { CDP_COMMAND_TIMEOUT_MS, DAEMON_RECONNECT_METHOD } from "./protocol";
+import { DAEMON_RECONNECT_METHOD, commandTimeoutFor } from "./protocol";
 import { asString, isRecord } from "../util/guards";
 import { type ChromeConnectionDependencies, type ConnectionStatus, createChromeConnection } from "./connection";
 
@@ -246,6 +246,7 @@ export const createCdpBridge = (options: ChromeConnectionDependencies = {}): Cdp
       if (sessionId !== undefined) router.release(sessionId);
     }
 
+    const timeoutMs = commandTimeoutFor(req);
     const daemonId = mux.allocate(
       {
         clientId,
@@ -257,8 +258,8 @@ export const createCdpBridge = (options: ChromeConnectionDependencies = {}): Cdp
       (id) =>
         setTimeout(() => {
           mux.take(id);
-          send(clientId, errorResponse(req.id, `Timeout after ${CDP_COMMAND_TIMEOUT_MS}ms: ${req.method}`));
-        }, CDP_COMMAND_TIMEOUT_MS),
+          send(clientId, errorResponse(req.id, `Timeout after ${timeoutMs}ms: ${req.method}`));
+        }, timeoutMs),
     );
 
     const payload: Record<string, unknown> = {

package/src/daemon/protocol.ts

@@ -6,6 +6,8 @@ export type WireRequest = {
   readonly method: string;
   readonly params?: Record<string, unknown>;
   readonly sessionId?: string;
+  // How long the daemon waits for Chrome's answer; see commandTimeoutFor.
+  readonly timeoutMs?: number;
 };
 
 export type WireResponse = {
@@ -71,6 +73,7 @@ const isWireMessage = (v: unknown): v is WireMessage => {
       if (typeof v["id"] !== "number") return false;
       if (typeof v["method"] !== "string") return false;
       if (!isOptionalRecord(v["params"])) return false;
+      if (v["timeoutMs"] !== undefined && typeof v["timeoutMs"] !== "number") return false;
       return isOptionalString(v["sessionId"]);
     }
     case "response": {
@@ -138,3 +141,11 @@ export const CDP_RECONNECT_COOLDOWN_MAX_MS = 5 * 60 * 1000;
 export const DAEMON_RECONNECT_METHOD = "Daemon.reconnect";
 
 export const CDP_COMMAND_TIMEOUT_MS = 10_000;
+
+// The longest a caller may ask the daemon to wait for one command (browser_execute_js's timeoutMs).
+export const CDP_COMMAND_TIMEOUT_MAX_MS = 120_000;
+
+// A request may lengthen the daemon's wait (a long-running script) but never shorten it below the default: short
+// client-side probes already give up on their own, and a shorter daemon timer would only race them.
+export const commandTimeoutFor = (req: { readonly timeoutMs?: number }): number =>
+  Math.min(Math.max(req.timeoutMs ?? CDP_COMMAND_TIMEOUT_MS, CDP_COMMAND_TIMEOUT_MS), CDP_COMMAND_TIMEOUT_MAX_MS);

package/src/daemon/transport.ts

@@ -194,6 +194,7 @@ export const createDaemonTransport = (clientId: string, socketPath: string = DAE
       method,
       params,
       ...(sessionId !== null ? { sessionId } : {}),
+      ...(opts?.timeoutMs !== undefined ? { timeoutMs: opts.timeoutMs } : {}),
     };
 
     return sendWithTimeout(pending, id, method, timeoutMs, "Daemon", () => sock.write(serialize(req) + "\n"));

package/src/domains/cdp-call.ts

@@ -55,8 +55,9 @@ export const evalJs = async (
   client: BrowserClient,
   expression: string,
   sessionId?: string,
+  opts?: { readonly timeoutMs?: number },
 ): Promise<Result<unknown, ToolErr>> => {
-  const r = await client.evaluateJs(expression, sessionId);
+  const r = await client.evaluateJs(expression, sessionId, opts);
   if (!r.success) return err(cdpErrToToolErr(r.error, "Runtime.evaluate"));
   return r;
 };

package/src/domains/click.ts

@@ -55,7 +55,7 @@ export const clickTool = defineBrowserTool({
   name: "browser_click",
   label: "Browser Click",
   description:
-    "Click an element. PREFERRED: pass `ref` (e.g. 'e12') from browser_snapshot — it re-resolves the element's position at click time, so it works even after the page re-renders and moves things. Fallback: pass viewport CSS-pixel `x`/`y`. Compositor-level click works through iframes, shadow DOM, and cross-origin content. After clicking, a compact diff of page changes is appended.",
+    "Click an element. PREFERRED: pass `ref` (e.g. 'e12') from browser_snapshot — it scrolls the element into view (inside scrolling lists too), measures it at click time, and warns if something covers it. Fallback: pass viewport CSS-pixel `x`/`y` (refused when outside the viewport; the result names the element that was hit). Compositor-level click works through iframes, shadow DOM, and cross-origin content. After clicking, a compact diff of page changes is appended.",
   promptSnippet: "Click an element by ref (preferred) or pixel coordinates",
   promptGuidelines: [
     "If a click doesn't register, run pi with --browser-debug-clicks (or set BH_DEBUG_CLICKS=1) to get annotated screenshots. For React/Vue components ignoring clicks, try browser_dispatch_key.",
@@ -69,7 +69,10 @@ export const clickTool = defineBrowserTool({
 
     const resolved = await resolveTarget(client, args);
     if (!resolved.success) return resolved;
-    const { x, y, label: target } = resolved.data;
+    const { x, y, label, covered, hit } = resolved.data;
+    const target = label
+      + (hit !== undefined ? ` on ${hit}` : "")
+      + (covered !== undefined ? `\n⚠ The point was covered by ${covered}, so the click went to that element, not [${args.ref ?? ""}]. Close the overlay or scroll, then re-snapshot.` : "");
 
     const clicked = await dispatchClick(client, x, y, button, count);
     if (!clicked.success) return clicked;

package/src/domains/form.ts

@@ -2,7 +2,8 @@ import { Type } from "typebox";
 import { type Result, err, ok } from "../util/result";
 import { defineBrowserTool, type JsonObject, type ToolErr, type ToolOk } from "../util/tool";
 import { detailsOf, type ElementOutcome, isPageResult, resolveAndCall, runOnElement } from "./element-call";
-import { fillField, type FillOptions } from "./page/fill.page";
+import { fillField, type FillOptions, type PageResult } from "./page/fill.page";
+import type { BrowserClient } from "../client";
 import { type FocusResult, focusElement, type SelectOption, type SelectResult, type SelectWanted, selectOption, setChecked } from "./page/form.page";
 import { elementFunctionSource } from "./page/inject";
 import { interactiveDiff } from "./ref-resolve";
@@ -56,45 +57,53 @@ const FillArgs = Type.Object({
 const FILL_FN = elementFunctionSource(fillField);
 const FILL_OPTIONS: FillOptions = { rejectSelect: true, focusFirst: false };
 
+type FilledField = Extract<PageResult, { readonly ok: true }>;
+
+// Writes `value` through the page's native value setter (the fill engine in page/fill.page.ts) and returns what the
+// field holds afterwards. browser_type falls back to it when typing did not produce the value; recipe "fill" steps
+// (recorded by the old browser_fill and by browser_fill_form) replay through it.
+export const nativeFill = async (
+  client: BrowserClient,
+  t: Target,
+  value: string,
+): Promise<Result<FilledField, ToolErr>> => {
+  const r = await runOnElement(client, { ref: t.ref, selector: t.selector, functionDeclaration: FILL_FN, args: [value, FILL_OPTIONS] });
+  if (!r.success) return r;
+  const narrowed = narrow(r.data, t, "browser_fill", isPageResult);
+  if (!narrowed.success) return narrowed;
+  const res = narrowed.data;
+  if (res.ok) return ok(res);
+  const target = targetLabel(t);
+  const details = { ...targetDetails(t), ...(res.tag !== undefined ? { tag: res.tag } : {}) };
+  if (res.code === "select_rejected") {
+    return err({
+      kind: "invalid_state",
+      message: `Element <select> is not fillable (not an input, textarea, or contenteditable): ${target}. For <select> use browser_select_option.`,
+      details,
+    });
+  }
+  return err({ kind: "invalid_state", message: `Could not fill ${target}: ${res.reason}`, details });
+};
+
+// Not registered for the agent any more (browser_type replaces it); kept so saved recipes' "fill" steps replay.
 export const fillTool = defineBrowserTool({
   name: "browser_fill",
   label: "Browser Fill",
   description:
-    "Fill a form field (input/textarea/contenteditable), or tick a checkbox/radio by passing 'true'/'false'. PREFERRED: pass `ref` (e.g. 'e7') from browser_snapshot — survives re-renders. Fallback: a CSS `selector`. Writes through the native value setter and fires bubbling 'input'/'change' events, so React/Vue/Angular controlled components and rich-text editors update correctly — unlike browser_type. Refuses a disabled field rather than writing a value the page will ignore. Returns the field's value after writing, plus a compact diff of page changes.",
-  promptSnippet: "Fill a form field by ref (preferred) or selector (works with React/Vue controlled inputs)",
-  promptGuidelines: [
-    "Pass the desired value; no browser_click is needed first.",
-    "For a checkbox or radio pass 'true' or 'false' — it toggles `checked`, not the submit value. browser_set_checked does the same thing with a boolean.",
-    "Use browser_type instead only for keystroke-sensitive widgets (autocomplete, masked/segmented inputs).",
-    "Fill sets the value but does not focus or submit. To submit a tag/autocomplete input, follow with browser_dispatch_key({ ref, key: 'Enter' }) — not browser_press_key, which targets the focused element and may miss this field.",
-    "For a custom (div-based) dropdown, browser_fill won't validate against its option list — open it, run browser_snapshot, then browser_click the option's ref.",
-    "Confirm the field value browser_fill returns matches what you intended.",
-  ],
+    "Recipe replay only: set a form field's value through the native setter, or tick a checkbox/radio with 'true'/'false'.",
+  promptSnippet: "Set a form field's value (recipe replay)",
+  promptGuidelines: [],
   parameters: FillArgs,
   concurrency: "serialized",
   async handler(args, { client }): Promise<Result<ToolOk, ToolErr>> {
-    const r = await runOnElement(client, { ref: args.ref, selector: args.selector, functionDeclaration: FILL_FN, args: [args.value, FILL_OPTIONS] });
-    if (!r.success) return r;
-    const narrowed = narrow(r.data, args, "browser_fill", isPageResult);
-    if (!narrowed.success) return narrowed;
-    const res = narrowed.data;
-    const target = targetLabel(args);
-    if (!res.ok) {
-      const details = { ref: args.ref, selector: args.selector, ...(res.tag !== undefined ? { tag: res.tag } : {}) };
-      if (res.code === "select_rejected") {
-        return err({
-          kind: "invalid_state",
-          message: `Element <select> is not fillable (not an input, textarea, or contenteditable): ${target}. For <select> use browser_select_option.`,
-          details,
-        });
-      }
-      return err({ kind: "invalid_state", message: `Could not fill ${target}: ${res.reason}`, details });
-    }
+    const filled = await nativeFill(client, args, args.value);
+    if (!filled.success) return filled;
+    const res = filled.data;
     const verified = "checked" in res ? res.checked : res.value;
     const shown = "checked" in res ? `checked=${res.checked}` : JSON.stringify(res.value);
     const diff = await interactiveDiff(client);
     return ok({
-      text: `Filled ${target} = ${shown}${diff}`,
+      text: `Filled ${targetLabel(args)} = ${shown}${diff}`,
       details: { ...targetDetails(args), value: args.value, verified, tag: res.tag, kind: res.kind },
     });
   },
@@ -119,10 +128,10 @@ export const focusTool = defineBrowserTool({
   name: "browser_focus",
   label: "Browser Focus",
   description:
-    "Focus an element via the DOM .focus() method — deterministic, no coordinate accuracy needed. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`. Use before browser_type when a click might miss the field.",
+    "Focus an element via the DOM .focus() method — deterministic, no coordinate accuracy needed. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`.",
   promptSnippet: "Focus an element by ref (preferred) or selector",
   promptGuidelines: [
-    "For simply setting a value, prefer browser_fill (it doesn't need a separate focus step).",
+    "To enter text, pass the ref straight to browser_type — it focuses the field itself.",
   ],
   parameters: FocusArgs,
   concurrency: "serialized",

package/src/domains/js.ts

@@ -40,6 +40,13 @@ const prettyJson = (text: string): string => {
 const ExecuteJsArgs = Type.Object({
   expression: Type.String({ description: "A JavaScript expression, or a function body: a body with a top-level `return` (after any statements) runs inside an async function and returns that value." }),
   targetId: Type.Optional(Type.String({ description: "Optional iframe targetId; default = current page." })),
+  timeoutMs: Type.Optional(
+    Type.Integer({
+      minimum: 1_000,
+      maximum: 120_000,
+      description: "How long to wait for the result (an awaited promise included). Default 30000, max 120000. Raise it for scripts that loop with waits.",
+    }),
+  ),
 });
 
 const ExecuteJsDetailsSchema = Type.Object({
@@ -51,6 +58,8 @@ const ExecuteJsDetailsSchema = Type.Object({
 
 export type ExecuteJsDetails = DetailsOf<typeof ExecuteJsDetailsSchema>;
 
+const EXECUTE_JS_DEFAULT_TIMEOUT_MS = 30_000;
+
 export const executeJsTool = defineBrowserTool({
   name: "browser_execute_js",
   label: "Browser Execute JS",
@@ -63,13 +72,22 @@ export const executeJsTool = defineBrowserTool({
     "For 'is this element visible / enabled / checked / focused', a one-liner here beats a screenshot every time.",
     "For iframes, first run `Object.values(document.querySelectorAll('iframe'))` to find the iframe, then pass its targetId.",
     "Result must be JSON-serializable (Runtime.evaluate returnByValue=true).",
+    "A script that awaits (loops with setTimeout, waits for UI) gets 30s by default; pass timeoutMs (max 120000) for longer ones.",
   ],
   parameters: ExecuteJsArgs,
   details: ExecuteJsDetailsSchema,
   concurrency: "parallel",
   async handler(args, { client }): Promise<Result<ToolOk<ExecuteJsDetails>, ToolErr>> {
-    const r = await evalJs(client, args.expression, args.targetId);
-    if (!r.success) return r;
+    const timeoutMs = args.timeoutMs ?? EXECUTE_JS_DEFAULT_TIMEOUT_MS;
+    const r = await evalJs(client, args.expression, args.targetId, { timeoutMs });
+    if (!r.success) {
+      if (r.error.kind !== "timeout" && !/timeout/i.test(r.error.message)) return r;
+      return err({
+        kind: "timeout",
+        message: `${r.error.message}. The script may have partly run before the wait ran out — check the page's state before retrying. Pass a larger timeoutMs (max 120000) or split the work into several calls.`,
+        details: { ...(r.error.details ?? {}), timeoutMs },
+      });
+    }
     const valueStr = r.data === undefined ? "undefined" : JSON.stringify(r.data);
     // The full value lives in the text (and in fullOutputPath when truncated); details carry only what the
     // renderer needs so a large result is not persisted twice.

package/src/domains/keyboard.ts

@@ -7,6 +7,11 @@ import { cdpCall, evalJs } from "./cdp-call";
 import { runOnElement } from "./element-call";
 import { elementFunctionSource } from "./page/inject";
 import { dispatchKeyEvent, type KeyEventSpec } from "./page/keyboard.page";
+import { prepareTyping, readTypedValue, type TypePrep, type TypedValue } from "./page/fill.page";
+import { nativeFill } from "./form";
+import { interactiveDiff } from "./ref-resolve";
+import { isRecord } from "../util/guards";
+import type { BrowserClient } from "../client";
 
 const DISPATCH_KEY_FN = elementFunctionSource(dispatchKeyEvent);
 
@@ -19,53 +24,180 @@ const VIRTUAL_KEY_CODES: Readonly<Record<string, number>> = {
   F7: 118, F8: 119, F9: 120, F10: 121, F11: 122, F12: 123,
 };
 
-const virtualKeyCode = (key: string): number =>
-  VIRTUAL_KEY_CODES[key] ?? (key.length === 1 ? key.charCodeAt(0) : 0);
+// Windows virtual-key codes for letters and digits are the upper-case ASCII code ("a" and "A" are both 65).
+export const virtualKeyCode = (key: string): number => {
+  const named = VIRTUAL_KEY_CODES[key];
+  if (named !== undefined) return named;
+  if (/^[a-z0-9]$/i.test(key)) return key.toUpperCase().charCodeAt(0);
+  return key.length === 1 ? key.charCodeAt(0) : 0;
+};
+
+// KeyboardEvent.code: the physical key. Pages that check `code` (shortcut handlers) need "KeyA", not "a".
+export const physicalCode = (key: string): string => {
+  if (/^[a-z]$/i.test(key)) return `Key${key.toUpperCase()}`;
+  if (/^[0-9]$/.test(key)) return `Digit${key}`;
+  if (key === " ") return "Space";
+  return key;
+};
+
+// Chrome only runs an editing shortcut sent over CDP when the event names the command (macOS does not map Cmd+A to
+// select-all for synthetic events), so Ctrl/Cmd + A/C/X/V/Z/Y carry it explicitly.
+export const editingCommands = (key: string, modifiers: number): ReadonlyArray<string> => {
+  const ctrlOrMeta = (modifiers & (2 | 4)) !== 0;
+  if (!ctrlOrMeta || (modifiers & 1) !== 0) return [];
+  const shift = (modifiers & 8) !== 0;
+  switch (key.toLowerCase()) {
+    case "a": return shift ? [] : ["selectAll"];
+    case "c": return shift ? [] : ["copy"];
+    case "x": return shift ? [] : ["cut"];
+    case "v": return shift ? [] : ["paste"];
+    case "z": return shift ? ["redo"] : ["undo"];
+    case "y": return shift ? [] : ["redo"];
+    default: return [];
+  }
+};
 
 const TypeArgs = Type.Object({
-  text: Type.String({ description: "Text to type" }),
+  ref: Type.Optional(
+    Type.String({ description: "Stable element ref from browser_snapshot (e.g. 'e7'). PREFERRED. The field is focused for you." }),
+  ),
+  selector: Type.Optional(Type.String({ description: "CSS selector of the field (fallback when no ref)." })),
+  text: Type.String({ description: "Text to enter. With `clear` (the default when a field is named) it replaces the field's value; \"\" empties it." }),
+  clear: Type.Optional(
+    Type.Boolean({
+      description:
+        "Replace the field's current value (true) or add to it at the cursor (false). Default: true when ref/selector is given, false when typing into the already-focused element.",
+    }),
+  ),
 });
 
-type FocusProbe = { readonly ok: boolean; readonly tag: string | null };
+type TypeTarget = { readonly ref?: string | undefined; readonly selector?: string | undefined };
 
-const isFocusProbe = (v: unknown): v is FocusProbe =>
-  typeof v === "object" && v !== null
-  && "ok" in v && typeof v.ok === "boolean"
-  && "tag" in v && (typeof v.tag === "string" || v.tag === null);
+const PREP_FN = elementFunctionSource(prepareTyping);
+const READ_FN = elementFunctionSource(readTypedValue);
+
+const isTypePrep = (v: unknown): v is TypePrep => {
+  if (!isRecord(v) || typeof v["ok"] !== "boolean") return false;
+  if (v["ok"]) return typeof v["kind"] === "string" && typeof v["value"] === "string" && typeof v["tag"] === "string";
+  return typeof v["code"] === "string" && typeof v["reason"] === "string";
+};
+
+const isTypedValue = (v: unknown): v is TypedValue =>
+  isRecord(v) && typeof v["value"] === "string" && typeof v["focused"] === "boolean";
+
+const hasTarget = (t: TypeTarget): boolean => t.ref !== undefined || t.selector !== undefined;
+
+const labelOf = (t: TypeTarget): string => (t.ref !== undefined ? `[${t.ref}]` : t.selector ?? "the focused element");
+
+// Runs an element function on the named field, or on document.activeElement when none is named.
+const onField = async (
+  client: BrowserClient,
+  t: TypeTarget,
+  fn: string,
+  args: ReadonlyArray<unknown>,
+): Promise<Result<unknown, ToolErr>> => {
+  if (hasTarget(t)) {
+    const r = await runOnElement(client, { ref: t.ref, selector: t.selector, functionDeclaration: fn, args });
+    if (!r.success) return r;
+    if (!r.data.found) return err({ kind: "invalid_state", message: `No element matched: ${labelOf(t)}` });
+    return ok(r.data.value);
+  }
+  return evalJs(client, "(" + fn + ").apply(document.activeElement, " + safeJs`${args}` + ")");
+};
+
+const prepErr = (t: TypeTarget, prep: Extract<TypePrep, { readonly ok: false }>): ToolErr => {
+  const where = labelOf(t);
+  const hint =
+    prep.code === "select" ? " For a <select> use browser_select_option."
+    : prep.code === "checkable" ? " For a checkbox or radio use browser_set_checked."
+    : prep.code === "not_element" ? " Pass the field's ref, or click it first so it has focus."
+    : "";
+  return { kind: "invalid_state", message: `Cannot type into ${where}: ${prep.reason}.${hint}`, details: { code: prep.code, ...(prep.tag !== undefined ? { tag: prep.tag } : {}) } };
+};
+
+// Backspace with the field's content selected deletes it, the way a user empties a field.
+const pressBackspace = async (client: BrowserClient): Promise<Result<void, ToolErr>> => {
+  const key = { key: "Backspace", code: "Backspace", windowsVirtualKeyCode: 8, nativeVirtualKeyCode: 8 };
+  const down = await cdpCall(client, "Input.dispatchKeyEvent", { type: "keyDown", ...key });
+  if (!down.success) return down;
+  const up = await cdpCall(client, "Input.dispatchKeyEvent", { type: "keyUp", ...key });
+  if (!up.success) return up;
+  return ok(undefined);
+};
 
 export const typeTool = defineBrowserTool({
   name: "browser_type",
   label: "Browser Type",
-  description: "Type text into the currently focused element. Use browser_click first to focus an input field.",
-  promptSnippet: "Type text into the focused element",
+  description:
+    "Enter text into a field the way a user types: focuses it, replaces its value (or appends with clear:false), fires real input events so autocomplete/suggestion lists open and React/Vue controlled inputs update, then reads the value back. PREFERRED: pass `ref` from browser_snapshot; fallback: a CSS `selector`; with neither, types into the focused element. If typing does not produce the value (date/masked inputs), it falls back to setting the value directly and says so.",
+  promptSnippet: "Type text into a field by ref (preferred), selector, or the focused element",
   promptGuidelines: [
-    "For special keys (Enter, Tab, Escape, arrows), use browser_press_key instead.",
+    "One tool for all text entry: browser_type({ ref, text }) replaces the field's value; pass clear:false to append; text \"\" empties the field.",
+    "Check the value it reports — a page can reformat or reject what was typed.",
+    "Suggestion lists open as you type: after browser_type, run browser_snapshot and browser_click the option's ref.",
+    "For a <select> use browser_select_option; for a checkbox/radio use browser_set_checked.",
+    "For special keys (Enter, Tab, Escape, arrows) use browser_press_key.",
   ],
   parameters: TypeArgs,
   concurrency: "serialized",
   async handler(args, { client }): Promise<Result<ToolOk, ToolErr>> {
-    const focused = await evalJs(client, safeJs`
-      (() => {
-        const el = document.activeElement;
-        const ok = !!el && el !== document.body &&
-          (el.tagName === "INPUT" || el.tagName === "TEXTAREA" || el.isContentEditable);
-        return { ok, tag: el ? el.tagName : null };
-      })()
-    `);
-    if (focused.success) {
-      const f: FocusProbe = isFocusProbe(focused.data) ? focused.data : { ok: false, tag: null };
-      if (!f.ok) {
+    const t: TypeTarget = { ref: args.ref, selector: args.selector };
+    const named = hasTarget(t);
+    const clear = args.clear ?? named;
+    const where = labelOf(t);
+
+    const prepRaw = await onField(client, t, PREP_FN, [clear]);
+    if (!prepRaw.success) return prepRaw;
+    if (!isTypePrep(prepRaw.data)) {
+      if (named) return err({ kind: "internal", message: "unexpected page result shape from browser_type", details: { target: where } });
+      return err({
+        kind: "invalid_state",
+        message: "No editable element is focused — typed text would be lost. Pass the field's ref, or click it first.",
+      });
+    }
+    const prep = prepRaw.data;
+    if (!prep.ok) {
+      if (!named && prep.code !== "disabled" && prep.code !== "readonly") {
         return err({
           kind: "invalid_state",
-          message:
-            "No editable element is focused — typed text would be lost. Click the field with browser_click first, or use browser_fill with a selector.",
-          details: { activeElement: f.tag },
+          message: "No editable element is focused — typed text would be lost. Pass the field's ref, or click it first.",
+          details: { activeElement: prep.tag ?? null },
         });
       }
+      return err(prepErr(t, prep));
     }
-    const r = await cdpCall(client, "Input.insertText", { text: args.text });
-    if (!r.success) return r;
-    return ok({ text: `Typed: "${args.text}"` });
+
+    const typed = clear && args.text === "" ? await pressBackspace(client) : await cdpCall(client, "Input.insertText", { text: args.text });
+    if (!typed.success) return typed;
+
+    const readRaw = await onField(client, t, READ_FN, []);
+    const after = readRaw.success && isTypedValue(readRaw.data) ? readRaw.data : undefined;
+    const expected = clear ? args.text : prep.value + args.text;
+    let value = after?.value;
+    let note = "";
+    if (value !== undefined && value !== expected && clear && named && prep.kind !== "contenteditable") {
+      // Typing did not produce the value (a date or masked input, or a handler that swallowed the keys): set it.
+      const filled = await nativeFill(client, t, args.text);
+      if (filled.success && "value" in filled.data) {
+        value = filled.data.value;
+        note = " (typing did not take; value set directly)";
+      }
+    }
+    if (value !== undefined && value !== expected) note += ` ⚠ the field holds ${JSON.stringify(value)}, not what was typed — the page reformatted or rejected it`;
+    if (after !== undefined && !after.focused && note === "") note = " (the field lost focus after typing)";
+
+    const diff = await interactiveDiff(client);
+    const shown = value !== undefined ? ` → value ${JSON.stringify(value)}` : "";
+    return ok({
+      text: `Typed "${args.text}" into ${where}${shown}${note}${diff}`,
+      details: {
+        ...(args.ref !== undefined ? { ref: args.ref } : {}),
+        ...(args.selector !== undefined ? { selector: args.selector } : {}),
+        text: args.text,
+        clear,
+        ...(value !== undefined ? { value } : {}),
+      },
+    });
   },
 });
 
@@ -100,23 +232,26 @@ export const pressKeyTool = defineBrowserTool({
     const code = virtualKeyCode(k);
     const modifiers = args.modifiers ?? 0;
     const isChar = k.length === 1;
+    const physical = physicalCode(k);
+    // A printable character needs a `char` event for the page to receive keypress/textInput and actually insert it; Shift (8) still counts as printable.
+    const hasCommandModifier = (modifiers & (1 | 2 | 4)) !== 0;
+    const commands = editingCommands(k, modifiers);
     const downParams: ParamsOf<"Input.dispatchKeyEvent"> = {
-      type: "keyDown",
+      type: isChar && hasCommandModifier ? "rawKeyDown" : "keyDown",
       key: k,
-      code: k,
+      code: physical,
       modifiers,
       ...(code ? { windowsVirtualKeyCode: code, nativeVirtualKeyCode: code } : {}),
-      ...(isChar ? { text: k, unmodifiedText: k } : {}),
+      ...(isChar && !hasCommandModifier ? { text: k, unmodifiedText: k } : {}),
+      ...(commands.length > 0 ? { commands: [...commands] } : {}),
     };
     const down = await cdpCall(client, "Input.dispatchKeyEvent", downParams);
     if (!down.success) return down;
-    // A printable character needs a `char` event for the page to receive keypress/textInput and actually insert it; Shift (8) still counts as printable.
-    const hasCommandModifier = (modifiers & (1 | 2 | 4)) !== 0;
     if (isChar && !hasCommandModifier) {
       const charEv = await cdpCall(client, "Input.dispatchKeyEvent", { type: "char", text: k, key: k });
       if (!charEv.success) return charEv;
     }
-    const up = await cdpCall(client, "Input.dispatchKeyEvent", { type: "keyUp", key: k, code: k, modifiers });
+    const up = await cdpCall(client, "Input.dispatchKeyEvent", { type: "keyUp", key: k, code: physical, modifiers });
     if (!up.success) return up;
     return ok({ text: `Pressed: ${k}${modifiers ? ` (modifiers=${modifiers})` : ""}` });
   },
@@ -153,7 +288,8 @@ export const dispatchKeyTool = defineBrowserTool({
     const eventType = args.eventType ?? "keydown";
     const target = args.ref ?? args.selector ?? "";
     // Legacy handlers branch on e.keyCode/e.which rather than e.key, so populate both.
-    const code = virtualKeyCode(args.key);
+    // keypress carries the character code ("a" = 97); keydown/keyup carry the key code ("a" = 65).
+    const code = eventType === "keypress" && args.key.length === 1 ? args.key.charCodeAt(0) : virtualKeyCode(args.key);
     if (args.ref !== undefined) {
       // Through runOnElement so the resolved RemoteObject is released after the call.
       const spec: KeyEventSpec = { type: eventType, key: args.key, code };

package/src/domains/page/click.page.ts

@@ -0,0 +1,22 @@
+// Page-side code for browser_click; see fill.page.ts for the rules every `*.page.ts` module follows.
+
+// What sits under the click point, relative to the element the click was aimed at.
+export type HitInfo = {
+  // The point lands on the target, inside it, or on an ancestor that wraps it (a label around an icon).
+  readonly onTarget: boolean;
+  // A short description of the element actually under the point, e.g. `div.modal-overlay "Close"`.
+  readonly hit: string;
+};
+
+// `target` is the element the click aims at (Runtime.callFunctionOn's `this`), `hit` the element hit-testing found.
+export function hitInfo(target: Element | null, hit: Element | null): HitInfo {
+  if (hit?.nodeType !== 1) return { onTarget: false, hit: "nothing" };
+  const onTarget = target !== null && (target === hit || target.contains(hit) || hit.contains(target));
+  const tag = hit.tagName.toLowerCase();
+  const id = hit.id ? "#" + hit.id : "";
+  const cls = typeof hit.className === "string" && hit.className.trim() !== ""
+    ? "." + hit.className.trim().split(/\s+/).slice(0, 2).join(".")
+    : "";
+  const label = (hit.getAttribute("aria-label") ?? hit.textContent).replace(/\s+/g, " ").trim().slice(0, 40);
+  return { onTarget, hit: tag + id + cls + (label !== "" ? ' "' + label + '"' : "") };
+}

package/src/domains/page/fill.page.ts

@@ -10,7 +10,7 @@
 export type PageOption = { readonly value: string; readonly text: string };
 
 export type FillOptions = {
-  // browser_fill refuses a <select> (its documented error points at browser_select_option); browser_fill_form fills it.
+  // The recipe-only browser_fill refuses a <select> (its documented error points at browser_select_option); browser_fill_form fills it.
   readonly rejectSelect: boolean;
   readonly focusFirst: boolean;
 };
@@ -37,6 +37,8 @@ export type FormControl = HTMLElement & {
   value?: string;
   checked?: boolean;
   disabled?: boolean;
+  readOnly?: boolean;
+  select?: () => void;
   options?: HTMLOptionsCollection;
 };
 
@@ -137,3 +139,70 @@ export function fillField(el: FormControl | null, value: unknown, opts: FillOpti
 
   return { ok: false, reason: "element is not a fillable field (tag=" + tag + ")", tag: tagName };
 }
+
+// What browser_type learns about its field before typing: whether it can take text, and its value right now.
+export type TypePrep =
+  | { readonly ok: true; readonly kind: "input" | "textarea" | "contenteditable"; readonly value: string; readonly tag: string }
+  | {
+      readonly ok: false;
+      readonly code: "not_element" | "disabled" | "readonly" | "select" | "checkable" | "not_editable";
+      readonly reason: string;
+      readonly tag?: string;
+    };
+
+// Focuses `el` and, when `clear` is set, selects everything in it, so the text typed next replaces the old value the
+// way select-all + typing does for a user. Refuses fields that cannot take typed text, naming the tool that can.
+export function prepareTyping(el: FormControl | null, clear: boolean): TypePrep {
+  if (el?.nodeType !== 1) return { ok: false, code: "not_element", reason: "no element to type into" };
+  const tagName = el.tagName;
+  const tag = tagName.toLowerCase();
+  const type = (el.type ?? "").toLowerCase();
+  if (tag === "select") return { ok: false, code: "select", reason: "element is a <select>", tag: tagName };
+  if (tag === "input" && (type === "checkbox" || type === "radio")) {
+    return { ok: false, code: "checkable", reason: "element is a " + type, tag: tagName };
+  }
+  const editableTag = (tag === "input" && type !== "button" && type !== "submit" && type !== "file" && type !== "image" && type !== "reset")
+    || tag === "textarea";
+  if (!editableTag && !el.isContentEditable) {
+    return { ok: false, code: "not_editable", reason: "element is not a text field (tag=" + tag + ")", tag: tagName };
+  }
+  if (el.disabled) return { ok: false, code: "disabled", reason: "element is disabled", tag: tagName };
+  if (editableTag && el.readOnly === true) {
+    return { ok: false, code: "readonly", reason: "element is read-only", tag: tagName };
+  }
+  try {
+    el.focus();
+  } catch {
+    /* focus throws on some detached or shadow nodes; the caller checks where the text landed */
+  }
+  if (editableTag) {
+    if (clear) {
+      try {
+        el.select?.();
+      } catch {
+        /* select() is a no-op or throws on a few input types (date, color); the caller verifies and falls back */
+      }
+    }
+    return { ok: true, kind: tag === "textarea" ? "textarea" : "input", value: el.value ?? "", tag: tagName };
+  }
+  if (clear) {
+    const sel = window.getSelection();
+    if (sel) {
+      sel.removeAllRanges();
+      const rng = document.createRange();
+      rng.selectNodeContents(el);
+      sel.addRange(rng);
+    }
+  }
+  return { ok: true, kind: "contenteditable", value: el.textContent, tag: tagName };
+}
+
+// The field's value after typing, and whether it still has focus (a blur handler may have moved it elsewhere).
+export type TypedValue = { readonly value: string; readonly focused: boolean };
+
+export function readTypedValue(el: FormControl | null): TypedValue {
+  if (el?.nodeType !== 1) return { value: "", focused: false };
+  const tag = el.tagName.toLowerCase();
+  const value = tag === "input" || tag === "textarea" ? (el.value ?? "") : el.textContent;
+  return { value, focused: document.activeElement === el };
+}

package/src/domains/page.ts

@@ -91,7 +91,7 @@ export const waitForTool = defineBrowserTool({
     "Poll until a condition holds: a selector appears (or disappears with gone:true), or text appears in the page. Returns a typed timeout error if the deadline elapses. Use to wait for SPA content to render before interacting.",
   promptSnippet: "Wait for an element or text to appear (or an element to disappear)",
   promptGuidelines: [
-    "Use before browser_fill / browser_click on dynamic pages so you act after the element renders.",
+    "Use before browser_type / browser_click on dynamic pages so you act after the element renders.",
   ],
   parameters: WaitForArgs,
   concurrency: "parallel",

package/src/domains/screenshot.ts

@@ -1,4 +1,8 @@
 import { readFileSync } from "node:fs";
+import { readFile } from "node:fs/promises";
+import type { BrowserClient } from "../client";
+import { imageWidth } from "../util/image-size";
+import { evalJs } from "./cdp-call";
 import { Type } from "typebox";
 import { Image, type ImageTheme, Text, truncateToWidth, visibleWidth } from "@mariozechner/pi-tui";
 import { type Result, ok } from "../util/result";
@@ -20,6 +24,21 @@ const ScreenshotDetailsSchema = Type.Object({
 
 export type ScreenshotDetails = DetailsOf<typeof ScreenshotDetailsSchema>;
 
+// Screenshots come out in device pixels (2x on a Retina display) and maxDim may shrink them again, so a point read
+// off the image is not a browser_click coordinate. Say how to convert, whenever the two differ.
+const scaleNote = async (client: BrowserClient, path: string, fullPage: boolean): Promise<string> => {
+  try {
+    const width = imageWidth(await readFile(path));
+    const css = await evalJs(client, fullPage ? "document.documentElement.scrollWidth" : "window.innerWidth");
+    if (width === undefined || !css.success || typeof css.data !== "number" || css.data <= 0) return "";
+    const ratio = width / css.data;
+    if (Math.abs(ratio - 1) < 0.05) return "";
+    return `\nImage scale: ${ratio.toFixed(2)} image px per CSS px — divide coordinates read off this image by ${ratio.toFixed(2)} before passing them to browser_click (or use refs from browser_snapshot).`;
+  } catch {
+    return "";
+  }
+};
+
 export const screenshotTool = defineBrowserTool({
   name: "browser_screenshot",
   label: "Browser Screenshot",
@@ -40,8 +59,9 @@ export const screenshotTool = defineBrowserTool({
       const resized = await resizeIfNeeded(cap.data.path, args.maxDim);
       note = resized.note;
     }
+    const scale = await scaleNote(client, cap.data.path, args.fullPage ?? false);
     return ok({
-      text: `Screenshot saved: ${cap.data.path}${note}`,
+      text: `Screenshot saved: ${cap.data.path}${note}${scale}`,
       details: { path: cap.data.path, format: cap.data.format, attached: false },
     });
   },

package/src/domains/snapshot.ts

@@ -115,7 +115,7 @@ export const snapshotTool = defineBrowserTool({
   name: "browser_snapshot",
   label: "Browser Snapshot",
   description:
-    "DEFAULT tool for understanding what is on the page. Returns the structured accessibility tree (roles, names, states, hierarchy). Every interactive element gets a stable ref shown as [eN] plus click coordinates @(x,y). Pass the ref to browser_click/browser_fill/etc. — refs survive re-renders, coordinates don't. Use this BEFORE deciding whether you need a screenshot. Pair with browser_execute_js for surgical reads of specific element values.",
+    "DEFAULT tool for understanding what is on the page. Returns the structured accessibility tree (roles, names, states, hierarchy). Every interactive element gets a stable ref shown as [eN] plus click coordinates @(x,y). Pass the ref to browser_click/browser_type/etc. — refs survive re-renders, coordinates don't. Use this BEFORE deciding whether you need a screenshot. Pair with browser_execute_js for surgical reads of specific element values.",
   promptSnippet: "Get accessibility-tree snapshot with stable element refs (default for page inspection)",
   promptGuidelines: [
     "Re-run browser_snapshot after a navigation or a major re-render to get fresh refs.",

package/src/domains/target.ts

@@ -1,7 +1,12 @@
 import type { BrowserClient } from "../client";
+import { isRecord } from "../util/guards";
 import { type Result, err, ok } from "../util/result";
 import type { ToolErr } from "../util/tool";
-import { resolveRefToBox } from "./ref-resolve";
+import { boxOf } from "./box";
+import { cdpCall } from "./cdp-call";
+import { hitInfo } from "./page/click.page";
+import { elementFunctionSource } from "./page/inject";
+import { resolveRefToBackendId } from "./ref-resolve";
 
 export type PointArgs = {
   readonly ref?: string | undefined;
@@ -13,23 +18,94 @@ export type ResolvedTarget = {
   readonly x: number;
   readonly y: number;
   readonly label: string;
+  // Set when hit-testing found something other than the ref's element under the point (an overlay, a sticky header).
+  readonly covered?: string;
+  // For an x/y click: what is under the point, so the agent can tell whether it hit what it meant to.
+  readonly hit?: string;
 };
 
+type Viewport = { readonly width: number; readonly height: number };
+
+// The layout viewport in CSS pixels, or undefined when Chrome would not say (then no bounds check is made).
+export const viewportOf = async (client: BrowserClient): Promise<Viewport | undefined> => {
+  const r = await cdpCall(client, "Page.getLayoutMetrics", {});
+  if (!r.success) return undefined;
+  const { clientWidth, clientHeight } = r.data.cssLayoutViewport;
+  return clientWidth > 0 && clientHeight > 0 ? { width: clientWidth, height: clientHeight } : undefined;
+};
+
+const inside = (vp: Viewport, x: number, y: number): boolean => x >= 0 && y >= 0 && x < vp.width && y < vp.height;
+
+const outsideErr = (x: number, y: number, vp: Viewport, ref?: string): ToolErr => ({
+  kind: "invalid_state",
+  message: ref !== undefined
+    ? `[${ref}] is at (${x}, ${y}), outside the ${vp.width}x${vp.height} viewport even after scrolling it into view — it may be clipped by a scroll container or positioned off-screen. Scroll its container (browser_scroll) and re-snapshot.`
+    : `Point (${x}, ${y}) is outside the ${vp.width}x${vp.height} viewport, so the click would hit nothing. Use the element's ref from browser_snapshot instead — browser_click scrolls it into view.`,
+  details: { x, y, viewportWidth: vp.width, viewportHeight: vp.height, ...(ref !== undefined ? { ref } : {}) },
+});
+
+const HIT_FN = elementFunctionSource(hitInfo);
+
+const isHitInfo = (v: unknown): v is { readonly onTarget: boolean; readonly hit: string } =>
+  isRecord(v) && typeof v["onTarget"] === "boolean" && typeof v["hit"] === "string";
+
+// What hit-testing finds at the point, relative to the target element (or just what is there, with no target);
+// undefined when it cannot tell.
+const hitTest = async (client: BrowserClient, backendId: number | undefined, x: number, y: number): Promise<{ readonly onTarget: boolean; readonly hit: string } | undefined> => {
+  const loc = await cdpCall(client, "DOM.getNodeForLocation", { x: Math.round(x), y: Math.round(y), ignorePointerEventsNone: true });
+  if (!loc.success) return undefined;
+  const target = await cdpCall(client, "DOM.resolveNode", { backendNodeId: backendId ?? loc.data.backendNodeId });
+  const hit = await cdpCall(client, "DOM.resolveNode", { backendNodeId: loc.data.backendNodeId });
+  const targetId = target.success ? target.data.object.objectId : undefined;
+  const hitId = hit.success ? hit.data.object.objectId : undefined;
+  try {
+    if (targetId === undefined || hitId === undefined) return undefined;
+    const called = await cdpCall(client, "Runtime.callFunctionOn", {
+      objectId: targetId,
+      functionDeclaration: HIT_FN,
+      arguments: [{ objectId: hitId }],
+      returnByValue: true,
+    });
+    if (!called.success || called.data.exceptionDetails !== undefined) return undefined;
+    const v: unknown = called.data.result.value;
+    return isHitInfo(v) ? v : undefined;
+  } finally {
+    for (const id of [targetId, hitId]) if (id !== undefined) await cdpCall(client, "Runtime.releaseObject", { objectId: id });
+  }
+};
+
+// Turns a ref (or x/y) into the point a trusted mouse event should land on. For a ref: scroll the element into view
+// (nested scroll containers included), measure it now, refuse a point still outside the viewport, and hit-test it so
+// a covered element is reported instead of silently clicking whatever is on top.
 export const resolveTarget = async (
   client: BrowserClient,
   args: PointArgs,
 ): Promise<Result<ResolvedTarget, ToolErr>> => {
   if (args.ref !== undefined) {
-    const box = await resolveRefToBox(client, args.ref);
-    if (!box.success) return box;
+    const backendId = resolveRefToBackendId(client, args.ref);
+    if (!backendId.success) return backendId;
+    // A failed scroll is not fatal: the bounds check below catches an element that is still out of view.
+    await cdpCall(client, "DOM.scrollIntoViewIfNeeded", { backendNodeId: backendId.data });
+    const box = await boxOf(client, backendId.data);
+    if (!box.success) {
+      return err({ kind: "invalid_state", message: `Ref ${args.ref} is unknown or stale — re-run browser_snapshot to get fresh refs.`, details: { ref: args.ref } });
+    }
+    const { cx, cy } = box.data;
+    const vp = await viewportOf(client);
+    if (vp !== undefined && !inside(vp, cx, cy)) return err(outsideErr(cx, cy, vp, args.ref));
+    const hit = await hitTest(client, backendId.data, cx, cy);
     return ok({
-      x: box.data.cx,
-      y: box.data.cy,
-      label: `[${args.ref}] (${box.data.cx}, ${box.data.cy})`,
+      x: cx,
+      y: cy,
+      label: `[${args.ref}] (${cx}, ${cy})`,
+      ...(hit !== undefined && !hit.onTarget ? { covered: hit.hit } : {}),
     });
   }
   if (args.x !== undefined && args.y !== undefined) {
-    return ok({ x: args.x, y: args.y, label: `(${args.x}, ${args.y})` });
+    const vp = await viewportOf(client);
+    if (vp !== undefined && !inside(vp, args.x, args.y)) return err(outsideErr(args.x, args.y, vp));
+    const hit = await hitTest(client, undefined, args.x, args.y);
+    return ok({ x: args.x, y: args.y, label: `(${args.x}, ${args.y})`, ...(hit !== undefined ? { hit: hit.hit } : {}) });
   }
   return err({ kind: "invalid_args", message: "Provide either `ref` or both `x` and `y`." });
 };

package/src/recipes/recordable.ts

@@ -20,7 +20,7 @@ export const REPLAYABLE: ReadonlyMap<string, "ref" | "fields" | "none"> = new Ma
   ["browser_dispatch_key", "ref"],
   ["browser_upload_file", "ref"],
   ["browser_fill_form", "fields"],
-  ["browser_type", "none"],
+  ["browser_type", "ref"],
   ["browser_press_key", "none"],
   ["browser_navigate", "none"],
   ["browser_go_back", "none"],
@@ -303,8 +303,9 @@ export const observeCall: CallObserver = async ({ name, concurrency, args, clien
     }
     const fields = mode === "fields" ? await fieldsOf(client, args) : undefined;
     let password = fields?.some((f) => f.password) ?? false;
-    if (name === "browser_fill") password = await isPasswordTarget(client, ref, selector);
-    if (name === "browser_type") {
+    if (name === "browser_fill" || (name === "browser_type" && (ref !== undefined || selector !== undefined))) {
+      password = await isPasswordTarget(client, ref, selector);
+    } else if (name === "browser_type") {
       const r = await evalJs(client, FOCUSED_IS_PASSWORD_EXPR);
       password = r.success && r.data === true;
     }

package/src/registry.ts

@@ -4,7 +4,7 @@ import type { BrowserTool } from "./util/tool";
 import { type Result, err, ok } from "./util/result";
 import { clickTool } from "./domains/click";
 import { typeTool, pressKeyTool, dispatchKeyTool } from "./domains/keyboard";
-import { fillTool, selectOptionTool, focusTool, setCheckedTool } from "./domains/form";
+import { selectOptionTool, focusTool, setCheckedTool } from "./domains/form";
 import { fillFormTool } from "./domains/form-batch";
 import { pageInfoTool, waitTool, waitForLoadTool, waitForTool } from "./domains/page";
 import { scrollTool } from "./domains/scroll";
@@ -31,7 +31,6 @@ export const ALL_TOOLS: ReadonlyArray<BrowserTool> = [
   setupTool,
   clickTool,
   typeTool,
-  fillTool,
   fillFormTool,
   selectOptionTool,
   setCheckedTool,

package/src/util/image-size.ts

@@ -0,0 +1,16 @@
+// Reads an image's pixel width from its header, without decoding it (sharp is optional). PNG keeps it in IHDR; JPEG
+// in the first start-of-frame marker. Returns undefined for anything else or a truncated file.
+export const imageWidth = (buf: Buffer): number | undefined => {
+  if (buf.length >= 24 && buf.readUInt32BE(0) === 0x89504e47) return buf.readUInt32BE(16);
+  if (buf.length < 4 || buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
+  let i = 2;
+  while (i + 9 < buf.length) {
+    if (buf[i] !== 0xff) return undefined;
+    const marker = buf[i + 1] ?? 0;
+    const len = buf.readUInt16BE(i + 2);
+    // SOF0..SOF15, minus DHT (C4), JPG (C8) and DAC (CC), carry the frame size: height then width.
+    if (marker >= 0xc0 && marker <= 0xcf && marker !== 0xc4 && marker !== 0xc8 && marker !== 0xcc) return buf.readUInt16BE(i + 7);
+    i += 2 + len;
+  }
+  return undefined;
+};