replicas-cli 0.2.355 → 0.2.356

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/dist/index.mjs +1352 -161
  2. package/package.json +1 -1
package/dist/index.mjs CHANGED
@@ -8010,13 +8010,12 @@ replicas computer info
8010
8010
  # 2) Launch a browser on the workspace display.
8011
8011
  replicas computer launch chrome https://news.ycombinator.com
8012
8012
 
8013
- # 3) Observe the settled screen and browser tab state before clicking.
8014
- replicas computer observe /tmp/state.png
8015
- replicas computer browser --snapshot
8013
+ # 3) Capture the settled viewport and native accessibility tree together.
8014
+ replicas computer browser-state /tmp/state.png --full
8016
8015
 
8017
- # 4) Drive the UI.
8018
- replicas computer click 521 700 # click coordinates from the screenshot
8019
- replicas computer scroll down --amount 5
8016
+ # 4) Batch actions that use refs from the same state, then capture the semantic diff.
8017
+ replicas computer browser-batch '[{"action":"fill","ref":17,"value":"hello"},{"action":"click","ref":42}]'
8018
+ replicas computer browser-state /tmp/after-click.png
8020
8019
 
8021
8020
  # 5) (Optional) Record a screencap to share back.
8022
8021
  replicas computer record start /tmp/demo.mp4 --fps 60
@@ -8045,21 +8044,33 @@ Waits briefly for the screen to stop changing, saves a 1:1 screenshot, and print
8045
8044
 
8046
8045
  Use this instead of hand-written \`sleep && screenshot\` loops after clicks, navigation, typing, or page loads. By default it saves a 100px coordinate-grid screenshot and waits up to 3s for 600ms of visual stability. Pass \`--raw\` if you need an unannotated 1:1 screenshot.
8047
8046
 
8048
- ### \`replicas computer browser\`
8049
- Prints JSON for Chrome tabs launched through Replicas, including page titles and URLs. Pass \`--snapshot\` to include visible page text and interactive controls with DOM viewport bounding boxes.
8047
+ ### \`replicas computer browser-state <path> [--full] [--timeout MS] [--stable-ms MS]\`
8048
+ Waits for the selected Chrome page to settle, saves its viewport screenshot, and prints a compact native Chrome accessibility tree in the same coordinate space. Actionable nodes have DOM-backed refs such as \`[ref=42] button "Save"\`; the tree includes accessible roles, names, values, states, visible bounds, and iframe content.
8049
+
8050
+ The first call returns a full tree. Later calls for the same target and document return semantic changes; navigation or reload forces a new full tree. Pass \`--full\` to reset the baseline. Refs are valid for the current document.
8051
+
8052
+ ### \`replicas computer browser [--snapshot]\`
8053
+ Lists Chrome tabs. \`--snapshot\` returns the compatibility accessibility snapshot. Use \`browser-state\` for the primary action/observe loop.
8054
+
8055
+ ### \`replicas computer browser-batch '<actions-json>'\`
8056
+ Runs up to 100 ordered browser actions through one Chrome session and stops at the first failure. Supported actions are \`click\`, \`fill\`, \`key\`, \`type\`, \`scroll\`, and \`wait\`. Prefer this when several actions use refs from the same \`browser-state\`; it avoids repeated process, tab-selection, and connection setup. Do not carry refs across navigation\u2014capture fresh state and start a new batch.
8057
+
8058
+ ### \`replicas computer browser-click --ref ID [--button left|middle|right] [--double]\`
8059
+ Scrolls the referenced node into view and sends trusted Chrome mouse input at its current visible bounds. Prefer refs from \`browser-state\`. \`browser-click <text> [--exact] [--index N]\` remains a fallback when no fresh ref is available.
8050
8060
 
8051
- Use this alongside \`observe\` when testing web apps so you do not infer navigation, page state, or click targets from pixels alone. Snapshot coordinates are DOM viewport coordinates, not desktop click coordinates.
8061
+ ### \`replicas computer browser-fill --ref ID <value>\`
8062
+ Focuses the referenced field, fills it with trusted Chrome input, and verifies the resulting value. Select controls use their semantic option value. \`browser-fill <field> <value> [--exact] [--index N]\` remains a text-matching fallback.
8052
8063
 
8053
- ### \`replicas computer browser-click <text> [--exact] [--index N]\`
8054
- Clicks the first visible Chrome control whose text, label, placeholder, or href matches \`<text>\`. Use this for web buttons and links found via \`browser --snapshot\`; it is faster and less error-prone than converting DOM coordinates to desktop pixels.
8064
+ ### \`replicas computer browser-scroll <up|down|left|right> [--ref ID] [--amount PX]\`
8065
+ Sends trusted wheel input at a referenced element or the center of the viewport.
8055
8066
 
8056
- ### \`replicas computer browser-fill <field> <value> [--exact] [--index N]\`
8057
- Fills the first visible Chrome field whose label, placeholder, name, or text matches \`<field>\`, then dispatches input/change events. Use this for web forms instead of clicking a field and typing through the desktop.
8067
+ ### \`replicas computer browser-key <combo>\` / \`browser-type <text>\`
8068
+ Sends trusted keyboard input to the focused element in the selected Chrome target. Prefer \`browser-fill\` when setting a known field value.
8058
8069
 
8059
8070
  ### \`replicas computer browser-wait <text> [--mode any|text|title|url|control] [--exact] [--timeout MS]\`
8060
8071
  Waits until the active Chrome page matches text in the title, URL, body text, or visible controls. Use this after \`browser-click\` / \`browser-fill\` when you need web app state to settle without screenshot polling.
8061
8072
 
8062
- For \`browser-click\`, \`browser-fill\`, and \`browser-wait\`, pass \`--id <id>\`, \`--title <text>\`, \`--url <text>\`, or \`--page <n>\` when multiple Chrome tabs are open. Run \`replicas computer browser\` first to list tabs. Prefer \`--id\` when a click may change the page title or URL.
8073
+ For browser state and actions, pass \`--target-id <id>\`, \`--title <text>\`, \`--url <text>\`, or \`--page <n>\` when multiple Chrome tabs are open. Prefer \`--target-id\` because titles and URLs can change after an action. The matched tab is brought to the foreground before each action, so driving multiple tabs or recording the screen switches which tab is visible.
8063
8074
 
8064
8075
  ### \`replicas computer click <x> <y> [--button N] [--double] [--modifiers ctrl+shift]\`
8065
8076
  Move to (x, y) and click. Coordinates can be absolute pixels or percentages such as \`50%\` \`50%\`. Default is left-click (button 1); pass \`--button 3\` for right-click. \`--modifiers\` holds keys during the click (e.g. ctrl-click a link to open in a new tab).
@@ -8092,26 +8103,29 @@ Spawns an app on the workspace display. Built-in aliases:
8092
8103
  Anything else gets \`exec\`'d verbatim, so \`replicas computer launch xeyes\` works if xeyes is installed.
8093
8104
  When opening a known page, prefer passing the URL directly: \`replicas computer launch chrome http://localhost:3000/\`.
8094
8105
 
8106
+ Chrome launches do not return until the requested page is controllable. The output includes its page ID; carry that ID with \`--target-id\` into \`browser-state\` and browser actions so unrelated tabs cannot steal task context.
8107
+
8095
8108
  ### \`replicas computer record start <path> [--fps N]\`
8096
- Starts an ffmpeg screen recorder. Output is post-processed when stopped: action windows stay at normal speed, idle gaps accelerate, click moments get eased camera zoom, and a synthetic cursor animates between logged mouse positions. The raw capture is fragmented MP4 while active (still playable if the workspace dies mid-record). Default 60fps; drop to 30 if the workspace is CPU-constrained.
8109
+ Starts an ffmpeg screen recorder and returns only after ffmpeg has written the first bytes. Output is post-processed when stopped: action windows stay at normal speed, idle gaps accelerate, click moments get eased camera zoom, and a synthetic cursor animates between logged mouse positions. The raw capture is fragmented MP4 while active (still playable if the workspace dies mid-record). Default 60fps; drop to 30 if the workspace is CPU-constrained.
8097
8110
 
8098
8111
  Only one recording at a time. Re-running \`start\` while one is active fails - call \`stop\` first.
8099
8112
 
8100
8113
  ### \`replicas computer record stop\`
8101
- SIGINTs ffmpeg, waits for it to finalize the MP4, prints the output path. Upload it with \`replicas media upload <path>\` to share it.
8114
+ SIGINTs ffmpeg, verifies that the process has finalized the MP4, then prints the output path. A finalization timeout preserves the recording state so \`stop\` can be retried safely. Upload it with \`replicas media upload <path>\` to share it.
8102
8115
 
8103
8116
  ## Patterns
8104
8117
 
8105
- ### Action / observe loop
8106
- You are blind between tool calls. After any action that changes the screen, observe before deciding the next coordinate:
8118
+ ### Browser action / state loop
8119
+ For Chrome, use the native accessibility state before interpreting pixels or guessing coordinates:
8107
8120
 
8108
8121
  \`\`\`bash
8109
- replicas computer click 521 700
8110
- replicas computer observe /tmp/after-click.png
8111
- # read /tmp/after-click.png and the JSON output, decide next click
8122
+ replicas computer launch chrome https://example.com
8123
+ replicas computer browser-state /tmp/before.png --full --target-id <page-id>
8124
+ replicas computer browser-batch '[{"action":"fill","ref":17,"value":"hello"},{"action":"click","ref":23}]' --target-id <page-id>
8125
+ replicas computer browser-state /tmp/after.png --target-id <page-id>
8112
8126
  \`\`\`
8113
8127
 
8114
- The JSON \`stable\`, \`frames\`, and \`changes\` fields tell you whether something changed while you were waiting. If \`stable\` is false, observe again or increase \`--timeout\` before acting on coordinates.
8128
+ Read the screenshot and tree as one state. Batch ordered actions whose refs all come from that state, then inspect the next semantic diff. A batch is fail-fast and validates each ref against the cached document. Split at navigation or whenever a later action depends on newly rendered refs. Fall back to \`observe\` plus desktop coordinates for canvas content, browser chrome, non-Chrome apps, or controls missing from the accessibility tree.
8115
8129
 
8116
8130
  ### Typing into an address bar
8117
8131
  \`\`\`bash
@@ -8120,10 +8134,9 @@ replicas computer observe /tmp/browser-open.png
8120
8134
  replicas computer key ctrl+l
8121
8135
  replicas computer type "https://example.com"
8122
8136
  replicas computer key Return
8123
- replicas computer observe /tmp/loaded.png
8124
- replicas computer browser --snapshot
8125
- replicas computer browser-fill "Search" "replicas"
8126
- replicas computer browser-click "More information" --title "Example"
8137
+ replicas computer browser-state /tmp/loaded.png --full
8138
+ replicas computer browser-fill --ref 17 "replicas"
8139
+ replicas computer browser-click --ref 23 --title "Example"
8127
8140
  replicas computer browser-wait "Example Domain" --mode title --title "Example"
8128
8141
  \`\`\`
8129
8142
 
@@ -9679,7 +9692,7 @@ var HOOK_EXEC_MAX_BUFFER_BYTES = 10 * 1024 * 1024;
9679
9692
  var REPLICAS_CONFIG_FILENAMES = ["replicas.json", "replicas.yaml", "replicas.yml"];
9680
9693
 
9681
9694
  // ../shared/src/cli-version.ts
9682
- var CLI_VERSION = "0.2.355";
9695
+ var CLI_VERSION = "0.2.356";
9683
9696
 
9684
9697
  // ../shared/src/version.ts
9685
9698
  function compareVersions(v1, v2) {
@@ -15035,7 +15048,7 @@ var sleep = (ms) => new Promise((resolve2) => setTimeout(resolve2, ms));
15035
15048
 
15036
15049
  // src/commands/computer/recording.ts
15037
15050
  import { spawn as spawn3 } from "child_process";
15038
- import { appendFileSync, existsSync as existsSync2, mkdirSync as mkdirSync2, readFileSync as readFileSync2, rmSync as rmSync2, writeFileSync as writeFileSync2 } from "fs";
15051
+ import { appendFileSync, existsSync as existsSync2, mkdirSync as mkdirSync2, readFileSync as readFileSync2, rmSync as rmSync2, statSync, writeFileSync as writeFileSync2 } from "fs";
15039
15052
  import { dirname as dirname2 } from "path";
15040
15053
 
15041
15054
  // src/commands/computer/recording/render.ts
@@ -15516,6 +15529,18 @@ var RECORD_STARTED_AT_FILE = `${STATE_DIR}/recording-started-at.txt`;
15516
15529
  var RECORD_FPS_FILE = `${STATE_DIR}/recording-fps.txt`;
15517
15530
  var RECORD_DIMENSIONS_FILE = `${STATE_DIR}/recording-dimensions.json`;
15518
15531
  var RECORD_ACTIONS_FILE = `${STATE_DIR}/recording-actions.jsonl`;
15532
+ var RECORD_STATE_FILES = [
15533
+ RECORD_PID_FILE,
15534
+ RECORD_PATH_FILE,
15535
+ RECORD_RAW_PATH_FILE,
15536
+ RECORD_STARTED_AT_FILE,
15537
+ RECORD_FPS_FILE,
15538
+ RECORD_DIMENSIONS_FILE,
15539
+ RECORD_ACTIONS_FILE
15540
+ ];
15541
+ function clearRecordingState() {
15542
+ for (const file of RECORD_STATE_FILES) rmSync2(file, { force: true });
15543
+ }
15519
15544
  function recordingStartedAt() {
15520
15545
  if (!existsSync2(RECORD_STARTED_AT_FILE)) return null;
15521
15546
  const startedAt = Number.parseInt(readFileSync2(RECORD_STARTED_AT_FILE, "utf8").trim(), 10);
@@ -15573,13 +15598,13 @@ async function computerRecordStartCommand(path6, options) {
15573
15598
  if (existsSync2(RECORD_PID_FILE)) {
15574
15599
  const pid = parseInt(readFileSync2(RECORD_PID_FILE, "utf8").trim(), 10);
15575
15600
  if (Number.isFinite(pid)) {
15576
- let alive = false;
15601
+ let alive2 = false;
15577
15602
  try {
15578
15603
  process.kill(pid, 0);
15579
- alive = true;
15604
+ alive2 = true;
15580
15605
  } catch {
15581
15606
  }
15582
- if (alive) fail(`recording already in progress (pid ${pid}). run \`replicas computer record stop\` first.`);
15607
+ if (alive2) fail(`recording already in progress (pid ${pid}). run \`replicas computer record stop\` first.`);
15583
15608
  }
15584
15609
  }
15585
15610
  const target = resolvePath(path6);
@@ -15626,7 +15651,29 @@ async function computerRecordStartCommand(path6, options) {
15626
15651
  writeFileSync2(RECORD_STARTED_AT_FILE, String(Date.now()));
15627
15652
  writeFileSync2(RECORD_FPS_FILE, String(fps));
15628
15653
  writeFileSync2(RECORD_DIMENSIONS_FILE, JSON.stringify({ width, height }));
15629
- console.log(target);
15654
+ const startedAt = Date.now();
15655
+ while (Date.now() - startedAt < 5e3) {
15656
+ try {
15657
+ process.kill(child.pid, 0);
15658
+ if (existsSync2(rawTarget) && statSync(rawTarget).size > 0) {
15659
+ console.log(`${target} (recording ready in ${Date.now() - startedAt}ms)`);
15660
+ return;
15661
+ }
15662
+ } catch {
15663
+ break;
15664
+ }
15665
+ await sleep(100);
15666
+ }
15667
+ let alive = false;
15668
+ try {
15669
+ process.kill(child.pid, 0);
15670
+ alive = true;
15671
+ } catch {
15672
+ }
15673
+ if (alive) fail("ffmpeg is running but did not produce recording output within 5 seconds; run `replicas computer record stop` to finalize or retry");
15674
+ clearRecordingState();
15675
+ rmSync2(rawTarget, { force: true });
15676
+ fail("ffmpeg exited before screen recording became ready");
15630
15677
  }
15631
15678
  async function computerRecordStopCommand() {
15632
15679
  if (!existsSync2(RECORD_PID_FILE) && !existsSync2(RECORD_PATH_FILE)) fail("no recording in progress");
@@ -15637,15 +15684,17 @@ async function computerRecordStopCommand() {
15637
15684
  process.kill(pid, "SIGINT");
15638
15685
  } catch {
15639
15686
  }
15640
- for (let i = 0; i < 30; i++) {
15687
+ let alive = true;
15688
+ for (let i = 0; i < 150; i++) {
15641
15689
  try {
15642
15690
  process.kill(pid, 0);
15643
15691
  } catch {
15692
+ alive = false;
15644
15693
  break;
15645
15694
  }
15646
- await new Promise((r) => setTimeout(r, 200));
15695
+ await sleep(200);
15647
15696
  }
15648
- rmSync2(RECORD_PID_FILE, { force: true });
15697
+ if (alive) fail(`ffmpeg did not finalize recording within 30 seconds (pid ${pid})`);
15649
15698
  }
15650
15699
  if (existsSync2(RECORD_PATH_FILE)) {
15651
15700
  const target = readFileSync2(RECORD_PATH_FILE, "utf8").trim();
@@ -15658,13 +15707,8 @@ async function computerRecordStopCommand() {
15658
15707
  rmSync2(rawPath, { force: true });
15659
15708
  }
15660
15709
  console.log(target);
15661
- rmSync2(RECORD_PATH_FILE, { force: true });
15662
- rmSync2(RECORD_RAW_PATH_FILE, { force: true });
15663
- rmSync2(RECORD_STARTED_AT_FILE, { force: true });
15664
- rmSync2(RECORD_FPS_FILE, { force: true });
15665
- rmSync2(RECORD_DIMENSIONS_FILE, { force: true });
15666
- rmSync2(RECORD_ACTIONS_FILE, { force: true });
15667
15710
  }
15711
+ clearRecordingState();
15668
15712
  }
15669
15713
 
15670
15714
  // src/commands/computer/index.ts
@@ -16003,8 +16047,8 @@ async function getChromePages() {
16003
16047
  async function selectChromePage(options = {}) {
16004
16048
  const pages = await getChromePages();
16005
16049
  let matches = pages;
16006
- if (options.id) {
16007
- matches = matches.filter((page2) => page2.id === options.id);
16050
+ if (options.targetId) {
16051
+ matches = matches.filter((page2) => page2.id === options.targetId);
16008
16052
  }
16009
16053
  if (options.title) {
16010
16054
  const needle = options.title.toLowerCase();
@@ -16023,121 +16067,528 @@ async function selectChromePage(options = {}) {
16023
16067
  `No matching debuggable Chrome page found (${matches.length} match${matches.length === 1 ? "" : "es"}, index ${index}). Use \`replicas computer browser\` to list pages.`
16024
16068
  );
16025
16069
  }
16070
+ await sendChromeCommand(webSocketDebuggerUrl, "Page.bringToFront", {});
16026
16071
  return { ...page, webSocketDebuggerUrl };
16027
16072
  }
16028
- function buildBrowserSnapshotExpression(textLimit, elementLimit) {
16029
- return `(() => {
16030
- const clip = (value, limit) => String(value || '').replace(/\\s+/g, ' ').trim().slice(0, limit);
16031
- const escape = (value) => globalThis.CSS && CSS.escape ? CSS.escape(value) : String(value).replace(/["\\\\]/g, '\\\\$&');
16073
+ var VISIBLE_JS = `
16032
16074
  const visible = (el) => {
16033
16075
  const rect = el.getBoundingClientRect();
16034
16076
  const style = getComputedStyle(el);
16035
16077
  return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
16036
- };
16037
- const labels = (el) => {
16038
- const values = [];
16039
- const id = el.getAttribute('id');
16040
- if (id) values.push(...Array.from(document.querySelectorAll('label[for="' + escape(id) + '"]')).map((label) => label.innerText));
16041
- const wrappingLabel = el.closest('label');
16042
- if (wrappingLabel) values.push(wrappingLabel.innerText);
16043
- values.push(el.innerText, el.value, el.getAttribute('aria-label'), el.getAttribute('title'), el.getAttribute('placeholder'), el.href, el.id, el.name, el.tagName);
16044
- return values.map((value) => clip(value, 160)).filter(Boolean);
16045
- };
16046
- const name = (el) => labels(el)[0] || clip(el.tagName, 160);
16047
- const selector = 'a,button,input,textarea,select,[role="button"],[role="link"],[role="textbox"],[contenteditable="true"]';
16048
- const controls = Array.from(document.querySelectorAll(selector))
16049
- .filter(visible)
16050
- .slice(0, ${elementLimit})
16051
- .map((el) => {
16052
- const rect = el.getBoundingClientRect();
16053
- return {
16054
- tag: el.tagName.toLowerCase(),
16055
- role: el.getAttribute('role') || null,
16056
- type: el.getAttribute('type') || null,
16057
- text: name(el),
16058
- href: el.href || null,
16059
- disabled: !!el.disabled || el.getAttribute('aria-disabled') === 'true',
16060
- rect: {
16061
- x: Math.round(rect.x),
16062
- y: Math.round(rect.y),
16063
- width: Math.round(rect.width),
16064
- height: Math.round(rect.height),
16065
- centerX: Math.round(rect.x + rect.width / 2),
16066
- centerY: Math.round(rect.y + rect.height / 2),
16067
- },
16068
- };
16069
- });
16070
- return {
16071
- title: document.title,
16072
- url: location.href,
16073
- text: clip(document.body ? document.body.innerText : '', ${textLimit}),
16074
- controls,
16075
- };
16076
- })()`;
16078
+ };`;
16079
+ async function sendChromeSessionCommand(session, method, params) {
16080
+ return chromeResult(await session.send({ method, params }), method);
16077
16081
  }
16078
- async function evaluateChromeTarget(webSocketDebuggerUrl, expression) {
16082
+ async function withChromeSession(webSocketDebuggerUrl, callback) {
16079
16083
  return await new Promise((resolve2, reject) => {
16080
16084
  const ws = new WebSocket(webSocketDebuggerUrl);
16085
+ const pending = /* @__PURE__ */ new Map();
16086
+ let nextId = 1;
16081
16087
  let settled = false;
16082
- const finish = (callback) => {
16088
+ const failSession = (error) => {
16083
16089
  if (settled) return;
16084
16090
  settled = true;
16085
- clearTimeout(timeout);
16091
+ for (const request of pending.values()) {
16092
+ clearTimeout(request.timeout);
16093
+ request.reject(error);
16094
+ }
16095
+ pending.clear();
16086
16096
  ws.close();
16087
- callback();
16097
+ reject(error);
16088
16098
  };
16089
- const timeout = setTimeout(() => {
16090
- finish(() => reject(new Error("Chrome DevTools evaluation timed out")));
16091
- }, 5e3);
16092
- ws.addEventListener("open", () => {
16093
- ws.send(JSON.stringify({
16094
- id: 1,
16095
- method: "Runtime.evaluate",
16096
- params: {
16097
- expression,
16098
- awaitPromise: true,
16099
- returnByValue: true
16100
- }
16101
- }));
16099
+ const finishSession = (value) => {
16100
+ if (settled) return;
16101
+ settled = true;
16102
+ for (const request of pending.values()) clearTimeout(request.timeout);
16103
+ pending.clear();
16104
+ ws.close();
16105
+ resolve2(value);
16106
+ };
16107
+ const session = {
16108
+ send: ({ method, params }) => new Promise((resolveCommand, rejectCommand) => {
16109
+ const id = nextId++;
16110
+ const timeout = setTimeout(() => {
16111
+ pending.delete(id);
16112
+ rejectCommand(new Error(`Chrome DevTools ${method} timed out`));
16113
+ }, 1e4);
16114
+ pending.set(id, { method, resolve: resolveCommand, reject: rejectCommand, timeout });
16115
+ ws.send(JSON.stringify({ id, method, params }));
16116
+ })
16117
+ };
16118
+ ws.addEventListener("open", async () => {
16119
+ try {
16120
+ finishSession(await callback(session));
16121
+ } catch (error) {
16122
+ failSession(error instanceof Error ? error : new Error(String(error)));
16123
+ }
16102
16124
  });
16103
16125
  ws.addEventListener("message", (event) => {
16104
16126
  const data = typeof event.data === "string" ? event.data : Buffer.isBuffer(event.data) ? event.data.toString("utf8") : Buffer.from(event.data).toString("utf8");
16105
- let parsed;
16127
+ let message;
16106
16128
  try {
16107
- parsed = JSON.parse(data);
16129
+ message = JSON.parse(data);
16108
16130
  } catch (error) {
16109
- const reason = error instanceof Error ? error.message : String(error);
16110
- finish(() => reject(new Error(`Chrome DevTools returned invalid WebSocket JSON: ${reason}`)));
16111
- return;
16112
- }
16113
- if (typeof parsed !== "object" || parsed === null) {
16114
- finish(() => reject(new Error("Chrome DevTools returned an invalid WebSocket message")));
16115
- return;
16116
- }
16117
- const message = parsed;
16118
- if (message.id !== 1) return;
16119
- if (message.error) {
16120
- finish(() => reject(new Error(message.error?.message || "Chrome DevTools evaluation failed")));
16121
- return;
16122
- }
16123
- if (message.result?.exceptionDetails) {
16124
- finish(() => reject(new Error(message.result.exceptionDetails.text || "Chrome DevTools evaluation threw")));
16131
+ failSession(new Error(`Chrome DevTools returned invalid WebSocket JSON: ${error instanceof Error ? error.message : String(error)}`));
16125
16132
  return;
16126
16133
  }
16127
- finish(() => resolve2(message.result?.result?.value ?? null));
16134
+ if (!isRecord(message) || typeof message.id !== "number") return;
16135
+ const request = pending.get(message.id);
16136
+ if (!request) return;
16137
+ clearTimeout(request.timeout);
16138
+ pending.delete(message.id);
16139
+ const detail = isRecord(message.error) && typeof message.error.message === "string" ? message.error.message : void 0;
16140
+ request.resolve(message.error ? { error: { message: detail } } : { result: isRecord(message.result) ? message.result : {} });
16128
16141
  });
16129
- ws.addEventListener("error", () => {
16130
- finish(() => reject(new Error("Chrome DevTools WebSocket failed")));
16142
+ ws.addEventListener("error", () => failSession(new Error("Chrome DevTools WebSocket failed")));
16143
+ ws.addEventListener("close", () => {
16144
+ if (!settled) failSession(new Error("Chrome DevTools WebSocket closed unexpectedly"));
16131
16145
  });
16132
16146
  });
16133
16147
  }
16148
+ async function sendChromeCommands(webSocketDebuggerUrl, commands) {
16149
+ return await withChromeSession(webSocketDebuggerUrl, async (session) => await Promise.all(commands.map((command) => session.send(command))));
16150
+ }
16151
+ async function sendChromeCommand(webSocketDebuggerUrl, method, params) {
16152
+ const response = (await sendChromeCommands(webSocketDebuggerUrl, [{ method, params }]))[0];
16153
+ if (response.error) throw new Error(response.error.message || `Chrome DevTools ${method} failed`);
16154
+ return response.result ?? {};
16155
+ }
16156
+ async function callFunctionOnChromeNodeInSession(session, backendNodeId, functionDeclaration, args = []) {
16157
+ const resolved = await sendChromeSessionCommand(session, "DOM.resolveNode", { backendNodeId });
16158
+ const objectId = isRecord(resolved.object) ? resolved.object.objectId : void 0;
16159
+ if (typeof objectId !== "string") throw new Error(`Could not resolve browser element ref ${backendNodeId}`);
16160
+ return await sendChromeSessionCommand(session, "Runtime.callFunctionOn", {
16161
+ objectId,
16162
+ functionDeclaration,
16163
+ arguments: args.map((value) => ({ value })),
16164
+ returnByValue: true
16165
+ });
16166
+ }
16167
+ function chromeCallValue(response) {
16168
+ return isRecord(response.result) ? response.result.value : void 0;
16169
+ }
16170
+ async function evaluateChromeTarget(webSocketDebuggerUrl, expression) {
16171
+ return await withChromeSession(webSocketDebuggerUrl, async (session) => await evaluateChromeSession(session, expression));
16172
+ }
16173
+ async function evaluateChromeSession(session, expression) {
16174
+ const response = await sendChromeSessionCommand(session, "Runtime.evaluate", {
16175
+ expression,
16176
+ awaitPromise: true,
16177
+ returnByValue: true
16178
+ });
16179
+ const exception = response.exceptionDetails;
16180
+ if (exception) {
16181
+ const text = typeof exception === "object" && exception !== null && "text" in exception && typeof exception.text === "string" ? exception.text : "";
16182
+ throw new Error(text || "Chrome DevTools evaluation threw");
16183
+ }
16184
+ const evaluated = response.result;
16185
+ return typeof evaluated === "object" && evaluated !== null && "value" in evaluated ? evaluated.value ?? null : null;
16186
+ }
16187
+ function isRawAXNode(value) {
16188
+ return isRecord(value);
16189
+ }
16190
+ function isDOMSnapshotDocument(value) {
16191
+ return isRecord(value);
16192
+ }
16193
+ function chromeVisualViewport(layout) {
16194
+ const viewport = isRecord(layout.cssVisualViewport) ? layout.cssVisualViewport : {};
16195
+ return {
16196
+ width: typeof viewport.clientWidth === "number" ? viewport.clientWidth : 0,
16197
+ height: typeof viewport.clientHeight === "number" ? viewport.clientHeight : 0
16198
+ };
16199
+ }
16200
+ function roundBrowserRect(rect) {
16201
+ return {
16202
+ x: Math.round(rect.x),
16203
+ y: Math.round(rect.y),
16204
+ width: Math.round(rect.width),
16205
+ height: Math.round(rect.height),
16206
+ centerX: Math.round(rect.centerX),
16207
+ centerY: Math.round(rect.centerY)
16208
+ };
16209
+ }
16210
+ function browserStateProperties(node) {
16211
+ const properties = [];
16212
+ for (const { name, value } of node.properties ?? []) {
16213
+ const state = value?.value;
16214
+ if (!name || !BROWSER_STATE_PROPERTIES.has(name) || typeof state !== "string" && typeof state !== "number" && typeof state !== "boolean") continue;
16215
+ properties.push([name, name === "url" ? clippedText(state, 300) : state]);
16216
+ }
16217
+ return Object.fromEntries(properties);
16218
+ }
16219
+ function readBrowserStateCache(path6) {
16220
+ let value;
16221
+ try {
16222
+ value = JSON.parse(readFileSync3(path6, "utf8"));
16223
+ } catch {
16224
+ return null;
16225
+ }
16226
+ if (!isRecord(value) || !Array.isArray(value.entries)) return null;
16227
+ const entries = [];
16228
+ for (const entry of value.entries) {
16229
+ if (!isRecord(entry) || typeof entry.key !== "string" || typeof entry.semantic !== "string" || typeof entry.line !== "string") return null;
16230
+ entries.push({ key: entry.key, semantic: entry.semantic, line: entry.line });
16231
+ }
16232
+ return {
16233
+ url: typeof value.url === "string" ? value.url : void 0,
16234
+ documentId: typeof value.documentId === "string" || value.documentId === null ? value.documentId : void 0,
16235
+ entries
16236
+ };
16237
+ }
16238
+ function chromeResult(response, method) {
16239
+ if (response.error) throw new Error(response.error.message || `Chrome DevTools ${method} failed`);
16240
+ return response.result ?? {};
16241
+ }
16242
+ function collectFrameIds(frameTree) {
16243
+ const ids = [];
16244
+ const visit = (value) => {
16245
+ if (!isRecord(value)) return;
16246
+ const frame = isRecord(value.frame) ? value.frame : {};
16247
+ if (typeof frame.id === "string") ids.push(frame.id);
16248
+ if (Array.isArray(value.childFrames)) value.childFrames.forEach(visit);
16249
+ };
16250
+ visit(frameTree);
16251
+ return ids;
16252
+ }
16253
+ function browserDocumentId(frameTree) {
16254
+ if (!isRecord(frameTree) || !isRecord(frameTree.frame)) return null;
16255
+ const loaderId = frameTree.frame.loaderId;
16256
+ return typeof loaderId === "string" ? loaderId : null;
16257
+ }
16258
+ function intersectRects(a, b) {
16259
+ const x = Math.max(a.x, b.x);
16260
+ const y = Math.max(a.y, b.y);
16261
+ const right = Math.min(a.x + a.width, b.x + b.width);
16262
+ const bottom = Math.min(a.y + a.height, b.y + b.height);
16263
+ if (right <= x || bottom <= y) return null;
16264
+ return {
16265
+ x,
16266
+ y,
16267
+ width: right - x,
16268
+ height: bottom - y,
16269
+ centerX: x + (right - x) / 2,
16270
+ centerY: y + (bottom - y) / 2
16271
+ };
16272
+ }
16273
+ function browserContentQuadRects(quads, viewportRect) {
16274
+ if (!Array.isArray(quads)) return [];
16275
+ return quads.filter((quad) => Array.isArray(quad) && quad.length >= 8 && quad.every(Number.isFinite)).map((quad) => {
16276
+ const xs = [quad[0], quad[2], quad[4], quad[6]];
16277
+ const ys = [quad[1], quad[3], quad[5], quad[7]];
16278
+ const x = Math.min(...xs);
16279
+ const y = Math.min(...ys);
16280
+ const width = Math.max(...xs) - x;
16281
+ const height = Math.max(...ys) - y;
16282
+ return { x, y, width, height, centerX: x + width / 2, centerY: y + height / 2 };
16283
+ }).map((rect) => intersectRects(rect, viewportRect)).filter((rect) => !!rect).sort((a, b) => b.width * b.height - a.width * a.height);
16284
+ }
16285
+ function buildBrowserBounds(result, viewport, targetId) {
16286
+ const strings = Array.isArray(result.strings) ? result.strings : [];
16287
+ const documents = Array.isArray(result.documents) ? result.documents.filter(isDOMSnapshotDocument) : [];
16288
+ const localRects = documents.map((document) => {
16289
+ const rects = /* @__PURE__ */ new Map();
16290
+ const nodeIndexes = document.layout?.nodeIndex ?? [];
16291
+ const bounds = document.layout?.bounds ?? [];
16292
+ const scrollX = document.scrollOffsetX ?? 0;
16293
+ const scrollY = document.scrollOffsetY ?? 0;
16294
+ nodeIndexes.forEach((nodeIndex, index) => {
16295
+ const bound = bounds[index];
16296
+ if (!Array.isArray(bound) || bound.length < 4 || !bound.every(Number.isFinite)) return;
16297
+ const [rawX, rawY, width, height] = bound;
16298
+ rects.set(nodeIndex, {
16299
+ x: rawX - scrollX,
16300
+ y: rawY - scrollY,
16301
+ width,
16302
+ height,
16303
+ centerX: rawX - scrollX + width / 2,
16304
+ centerY: rawY - scrollY + height / 2
16305
+ });
16306
+ });
16307
+ return rects;
16308
+ });
16309
+ const parents = /* @__PURE__ */ new Map();
16310
+ documents.forEach((document, documentIndex) => {
16311
+ const sparse = document.nodes?.contentDocumentIndex;
16312
+ sparse?.index?.forEach((nodeIndex, index) => {
16313
+ const childDocumentIndex = sparse.value?.[index];
16314
+ if (typeof childDocumentIndex === "number") parents.set(childDocumentIndex, { documentIndex, nodeIndex });
16315
+ });
16316
+ });
16317
+ const targetDocumentIndex = documents.findIndex(
16318
+ (document) => typeof document.frameId === "number" && strings[document.frameId] === targetId
16319
+ );
16320
+ const rootIndex = targetDocumentIndex >= 0 ? targetDocumentIndex : documents.findIndex((_, index) => !parents.has(index));
16321
+ const viewportRect = {
16322
+ x: 0,
16323
+ y: 0,
16324
+ width: viewport.width,
16325
+ height: viewport.height,
16326
+ centerX: viewport.width / 2,
16327
+ centerY: viewport.height / 2
16328
+ };
16329
+ const origins = /* @__PURE__ */ new Map();
16330
+ origins.set(rootIndex < 0 ? 0 : rootIndex, { x: 0, y: 0, clip: viewportRect });
16331
+ for (let pass = 0; pass < documents.length; pass++) {
16332
+ let changed = false;
16333
+ for (const [childIndex, parent] of parents) {
16334
+ if (origins.has(childIndex)) continue;
16335
+ const parentOrigin = origins.get(parent.documentIndex);
16336
+ const frameRect = localRects[parent.documentIndex]?.get(parent.nodeIndex);
16337
+ if (!parentOrigin || !frameRect) continue;
16338
+ const globalFrameRect = {
16339
+ ...frameRect,
16340
+ x: parentOrigin.x + frameRect.x,
16341
+ y: parentOrigin.y + frameRect.y,
16342
+ centerX: parentOrigin.x + frameRect.centerX,
16343
+ centerY: parentOrigin.y + frameRect.centerY
16344
+ };
16345
+ const childDocument = documents[childIndex];
16346
+ const insetX = Math.max(0, (globalFrameRect.width - (childDocument?.contentWidth ?? globalFrameRect.width)) / 2);
16347
+ const insetY = Math.max(0, (globalFrameRect.height - (childDocument?.contentHeight ?? globalFrameRect.height)) / 2);
16348
+ const contentRect = {
16349
+ x: globalFrameRect.x + insetX,
16350
+ y: globalFrameRect.y + insetY,
16351
+ width: Math.max(0, globalFrameRect.width - insetX * 2),
16352
+ height: Math.max(0, globalFrameRect.height - insetY * 2),
16353
+ centerX: globalFrameRect.centerX,
16354
+ centerY: globalFrameRect.centerY
16355
+ };
16356
+ const clip = intersectRects(parentOrigin.clip, contentRect);
16357
+ if (!clip) continue;
16358
+ origins.set(childIndex, { x: contentRect.x, y: contentRect.y, clip });
16359
+ changed = true;
16360
+ }
16361
+ if (!changed) break;
16362
+ }
16363
+ const byBackendNodeId = /* @__PURE__ */ new Map();
16364
+ documents.forEach((document, documentIndex) => {
16365
+ const origin = origins.get(documentIndex);
16366
+ if (!origin) return;
16367
+ const backendNodeIds = document.nodes?.backendNodeId ?? [];
16368
+ for (const [nodeIndex, local] of localRects[documentIndex] ?? []) {
16369
+ const backendNodeId = backendNodeIds[nodeIndex];
16370
+ if (!Number.isInteger(backendNodeId)) continue;
16371
+ const rect = {
16372
+ ...local,
16373
+ x: origin.x + local.x,
16374
+ y: origin.y + local.y,
16375
+ centerX: origin.x + local.centerX,
16376
+ centerY: origin.y + local.centerY
16377
+ };
16378
+ if (intersectRects(rect, origin.clip)) byBackendNodeId.set(backendNodeId, { rect, clip: origin.clip });
16379
+ }
16380
+ });
16381
+ return byBackendNodeId;
16382
+ }
16383
+ var BROWSER_INTERACTIVE_ROLES = /* @__PURE__ */ new Set([
16384
+ "button",
16385
+ "checkbox",
16386
+ "combobox",
16387
+ "link",
16388
+ "listbox",
16389
+ "menuitem",
16390
+ "menuitemcheckbox",
16391
+ "menuitemradio",
16392
+ "option",
16393
+ "radio",
16394
+ "scrollbar",
16395
+ "searchbox",
16396
+ "slider",
16397
+ "spinbutton",
16398
+ "switch",
16399
+ "tab",
16400
+ "textbox",
16401
+ "treeitem"
16402
+ ]);
16403
+ var BROWSER_STRUCTURAL_ROLES = /* @__PURE__ */ new Set([
16404
+ "alert",
16405
+ "cell",
16406
+ "columnheader",
16407
+ "dialog",
16408
+ "figure",
16409
+ "gridcell",
16410
+ "heading",
16411
+ "image",
16412
+ "listitem",
16413
+ "main",
16414
+ "paragraph",
16415
+ "region",
16416
+ "row",
16417
+ "rowheader",
16418
+ "status",
16419
+ "StaticText"
16420
+ ]);
16421
+ var BROWSER_STATE_PROPERTIES = /* @__PURE__ */ new Set([
16422
+ "autocomplete",
16423
+ "checked",
16424
+ "disabled",
16425
+ "editable",
16426
+ "expanded",
16427
+ "focusable",
16428
+ "focused",
16429
+ "haspopup",
16430
+ "invalid",
16431
+ "level",
16432
+ "multiselectable",
16433
+ "orientation",
16434
+ "pressed",
16435
+ "readonly",
16436
+ "protected",
16437
+ "required",
16438
+ "selected",
16439
+ "url"
16440
+ ]);
16441
+ function clippedText(value, limit = 240) {
16442
+ return String(value ?? "").replace(/\s+/g, " ").trim().slice(0, limit);
16443
+ }
16444
+ function browserTargetOutput(page) {
16445
+ return {
16446
+ targetId: page.id ?? null,
16447
+ title: (page.title ?? "").slice(0, 1e3),
16448
+ url: (page.url ?? "").slice(0, 2e3)
16449
+ };
16450
+ }
16451
+ function browserNodeActionable(role, states) {
16452
+ return BROWSER_INTERACTIVE_ROLES.has(role) || states.editable === true || states.focusable === true && role !== "RootWebArea";
16453
+ }
16454
+ async function captureBrowserSnapshot(page, options) {
16455
+ const initialCommands = [
16456
+ { method: "Page.getFrameTree", params: {} },
16457
+ { method: "Page.getLayoutMetrics", params: {} },
16458
+ { method: "DOMSnapshot.captureSnapshot", params: { computedStyles: [], includeDOMRects: true, includePaintOrder: true } }
16459
+ ];
16460
+ const initial = await sendChromeCommands(page.webSocketDebuggerUrl, initialCommands);
16461
+ const frameTree = chromeResult(initial[0], initialCommands[0].method).frameTree;
16462
+ const layout = chromeResult(initial[1], initialCommands[1].method);
16463
+ const domSnapshot = chromeResult(initial[2], initialCommands[2].method);
16464
+ const viewport = chromeVisualViewport(layout);
16465
+ const frameIds = collectFrameIds(frameTree);
16466
+ if (frameIds.length === 0 && page.id) frameIds.push(page.id);
16467
+ const axCommands = frameIds.map((frameId) => ({
16468
+ method: "Accessibility.getFullAXTree",
16469
+ params: { frameId }
16470
+ }));
16471
+ const axResults = await sendChromeCommands(page.webSocketDebuggerUrl, axCommands);
16472
+ const nodes = [];
16473
+ axResults.forEach((response, index) => {
16474
+ if (response.error) return;
16475
+ const responseNodes = response.result?.nodes;
16476
+ if (!Array.isArray(responseNodes)) return;
16477
+ for (const node of responseNodes) {
16478
+ if (!isRawAXNode(node)) continue;
16479
+ nodes.push({ ...node, sourceFrameId: frameIds[index] });
16480
+ }
16481
+ });
16482
+ const bounds = buildBrowserBounds(domSnapshot, viewport, page.id);
16483
+ const nodesByKey = new Map(nodes.map((node) => [`${node.sourceFrameId}:${node.nodeId}`, node]));
16484
+ const propertyMap = browserStateProperties;
16485
+ const roleOf = (node) => clippedText(node.role?.value, 80);
16486
+ const nameOf = (node) => clippedText(node.name?.value);
16487
+ const valueOf = (node) => {
16488
+ if (propertyMap(node).protected === true) return null;
16489
+ const value = node.value?.value;
16490
+ if (typeof value === "number" || typeof value === "boolean") return value;
16491
+ const text = clippedText(value);
16492
+ if (/^[•●*]+$/.test(text)) return "[protected]";
16493
+ return text || null;
16494
+ };
16495
+ const meaningful = nodes.filter((node) => !node.ignored && Number.isInteger(node.backendDOMNodeId)).map((node) => ({
16496
+ node,
16497
+ role: roleOf(node),
16498
+ bound: typeof node.backendDOMNodeId === "number" ? bounds.get(node.backendDOMNodeId) : void 0
16499
+ })).filter(({ node, role, bound }) => {
16500
+ if (!bound || bound.rect.width <= 0 || bound.rect.height <= 0) return false;
16501
+ if (role === "InlineTextBox" || role === "none" || role === "generic") return false;
16502
+ if (BROWSER_INTERACTIVE_ROLES.has(role) || BROWSER_STRUCTURAL_ROLES.has(role)) return true;
16503
+ return !!nameOf(node) || valueOf(node) !== null || propertyMap(node).focusable === true;
16504
+ }).filter(({ node, role }) => {
16505
+ if (role !== "StaticText") return true;
16506
+ const parent = node.parentId ? nodesByKey.get(`${node.sourceFrameId}:${node.parentId}`) : void 0;
16507
+ return !parent || nameOf(parent) !== nameOf(node);
16508
+ }).sort((a, b) => a.bound.rect.y - b.bound.rect.y || a.bound.rect.x - b.bound.rect.x);
16509
+ const elements = [];
16510
+ const entries = [];
16511
+ const meaningfulKeys = new Set(meaningful.map(({ node }) => `${node.sourceFrameId}:${node.nodeId}`));
16512
+ const visibleText = [];
16513
+ let visibleTextLength = 0;
16514
+ let treeLength = 0;
16515
+ const treeLimit = Math.min(5e4, Math.max(4e3, options.textLimit * 2));
16516
+ for (const { node, role, bound } of meaningful) {
16517
+ const name = nameOf(node);
16518
+ const description = clippedText(node.description?.value) || null;
16519
+ const value = valueOf(node);
16520
+ const states = propertyMap(node);
16521
+ const actionable = browserNodeActionable(role, states);
16522
+ const ref = actionable && elements.length < options.elementLimit ? String(node.backendDOMNodeId) : null;
16523
+ const visibleRect = role === "RootWebArea" && node.sourceFrameId === page.id ? { x: 0, y: 0, width: viewport.width, height: viewport.height, centerX: viewport.width / 2, centerY: viewport.height / 2 } : intersectRects(bound.rect, bound.clip) ?? bound.rect;
16524
+ const rect = roundBrowserRect(visibleRect);
16525
+ const actions = [
16526
+ ...actionable ? ["click"] : [],
16527
+ ...["textbox", "searchbox", "combobox", "spinbutton", "slider"].includes(role) || states.editable === true ? ["set_value"] : [],
16528
+ "scroll"
16529
+ ];
16530
+ if (ref) {
16531
+ elements.push({
16532
+ ref,
16533
+ role,
16534
+ name: name || null,
16535
+ description,
16536
+ value,
16537
+ states,
16538
+ actions,
16539
+ rect,
16540
+ frameId: node.sourceFrameId
16541
+ });
16542
+ }
16543
+ const stateText = Object.entries(states).filter(([key]) => key !== "url" && key !== "focusable").map(([key, state]) => `${key}=${JSON.stringify(state)}`);
16544
+ const url = typeof states.url === "string" ? clippedText(states.url, 300) : "";
16545
+ const semantic = [role, name && JSON.stringify(name), value !== null && `value=${JSON.stringify(value)}`, description && `description=${JSON.stringify(description)}`, url && `url=${JSON.stringify(url)}`, ...stateText].filter(Boolean).join(" ");
16546
+ let depth = 0;
16547
+ let parentId = node.parentId;
16548
+ const visited = /* @__PURE__ */ new Set();
16549
+ while (parentId && depth < 8) {
16550
+ const parentKey = `${node.sourceFrameId}:${parentId}`;
16551
+ if (visited.has(parentKey)) break;
16552
+ visited.add(parentKey);
16553
+ if (meaningfulKeys.has(parentKey)) depth++;
16554
+ parentId = nodesByKey.get(parentKey)?.parentId;
16555
+ }
16556
+ const line = `${" ".repeat(depth)}${ref ? `[ref=${ref}]` : "-"} ${semantic} (${rect.x},${rect.y} ${rect.width}x${rect.height})`;
16557
+ if (treeLength + line.length + 1 <= treeLimit) {
16558
+ const key = Number.isInteger(node.backendDOMNodeId) ? `dom:${node.sourceFrameId}:${node.backendDOMNodeId}` : `ax:${node.sourceFrameId}:${node.nodeId}`;
16559
+ entries.push({ key, semantic, line });
16560
+ treeLength += line.length + 1;
16561
+ }
16562
+ if (name && visibleTextLength < options.textLimit && (role === "StaticText" || role === "heading" || BROWSER_INTERACTIVE_ROLES.has(role))) {
16563
+ const remaining = options.textLimit - visibleTextLength;
16564
+ visibleText.push(name.slice(0, remaining));
16565
+ visibleTextLength += Math.min(name.length, remaining) + 1;
16566
+ }
16567
+ }
16568
+ const revision = createHash2("sha256").update(JSON.stringify([page.url, entries.map(({ key, semantic }) => [key, semantic])])).digest("hex").slice(0, 16);
16569
+ const snapshot = {
16570
+ title: (page.title ?? "").slice(0, 1e3),
16571
+ url: (page.url ?? "").slice(0, 2e3),
16572
+ documentId: browserDocumentId(frameTree),
16573
+ revision,
16574
+ viewport,
16575
+ text: visibleText.join(" ").slice(0, options.textLimit),
16576
+ tree: entries.map(({ line }) => line).join("\n"),
16577
+ elements,
16578
+ controls: elements,
16579
+ nodeCount: entries.length,
16580
+ elementCount: elements.length,
16581
+ truncated: entries.length < meaningful.length || elements.length < meaningful.filter(({ node, role }) => browserNodeActionable(role, propertyMap(node))).length
16582
+ };
16583
+ return { snapshot, entries };
16584
+ }
16134
16585
  async function computerBrowserCommand(options = {}) {
16135
16586
  const textLimit = options.limit ? parseCoord(options.limit, "--limit") : 4e3;
16136
16587
  const elementLimit = options.elementLimit ? parseCoord(options.elementLimit, "--element-limit") : 80;
16137
16588
  if (textLimit < 0 || textLimit > 5e4) fail("--limit must be between 0 and 50000");
16138
16589
  if (elementLimit < 0 || elementLimit > 500) fail("--element-limit must be between 0 and 500");
16139
- const expression = buildBrowserSnapshotExpression(textLimit, elementLimit);
16140
- const pages = (await getChromePages()).map((target) => ({
16590
+ const targets = options.targetId || options.page || options.title || options.url ? [await selectChromePage(options)] : await getChromePages();
16591
+ const pages = targets.map((target) => ({
16141
16592
  target,
16142
16593
  page: {
16143
16594
  id: target.id ?? null,
@@ -16150,7 +16601,11 @@ async function computerBrowserCommand(options = {}) {
16150
16601
  for (const { target, page } of pages) {
16151
16602
  if (options.snapshot && target.webSocketDebuggerUrl) {
16152
16603
  try {
16153
- pageResults.push({ ...page, snapshot: await evaluateChromeTarget(target.webSocketDebuggerUrl, expression) });
16604
+ const { snapshot } = await captureBrowserSnapshot(
16605
+ { ...target, webSocketDebuggerUrl: target.webSocketDebuggerUrl },
16606
+ { textLimit, elementLimit }
16607
+ );
16608
+ pageResults.push({ ...page, snapshot });
16154
16609
  } catch (error) {
16155
16610
  pageResults.push({ ...page, snapshotError: error instanceof Error ? error.message : "snapshot failed" });
16156
16611
  }
@@ -16164,6 +16619,188 @@ async function computerBrowserCommand(options = {}) {
16164
16619
  pages: pageResults
16165
16620
  }, null, 2));
16166
16621
  }
16622
+ function buildBrowserStabilityExpression() {
16623
+ return `(() => {
16624
+ const key = '__replicasComputerUseObserver';
16625
+ if (!globalThis[key]) {
16626
+ const state = { revision: 0 };
16627
+ const observer = new MutationObserver(() => state.revision++);
16628
+ globalThis[key] = { state, observer, observing: false };
16629
+ }
16630
+ if (document.documentElement && !globalThis[key].observing) {
16631
+ globalThis[key].observer.observe(document.documentElement, {
16632
+ subtree: true,
16633
+ childList: true,
16634
+ attributes: true,
16635
+ characterData: true,
16636
+ });
16637
+ globalThis[key].observing = true;
16638
+ }
16639
+ const controls = Array.from(document.querySelectorAll('input,textarea,select,[contenteditable="true"],[role="checkbox"],[role="combobox"],[role="slider"],[role="switch"],[role="textbox"]')).slice(0, 200);
16640
+ const active = document.activeElement;
16641
+ const controlState = controls.map((element, index) => {
16642
+ const value = 'value' in element
16643
+ ? element instanceof HTMLInputElement && element.type === 'password'
16644
+ ? String(element.value).length
16645
+ : String(element.value)
16646
+ : element.textContent;
16647
+ return [
16648
+ index,
16649
+ element === active,
16650
+ value,
16651
+ 'checked' in element ? element.checked : null,
16652
+ 'selectedIndex' in element ? element.selectedIndex : null,
16653
+ 'disabled' in element ? element.disabled : null,
16654
+ element.getAttribute('aria-checked'),
16655
+ element.getAttribute('aria-expanded'),
16656
+ element.getAttribute('aria-selected'),
16657
+ ].join('|');
16658
+ }).join('\\u001f');
16659
+ let controlRevision = 2166136261;
16660
+ for (let index = 0; index < controlState.length; index++) {
16661
+ controlRevision ^= controlState.charCodeAt(index);
16662
+ controlRevision = Math.imul(controlRevision, 16777619);
16663
+ }
16664
+ return {
16665
+ revision: globalThis[key].state.revision,
16666
+ controlRevision: (controlRevision >>> 0).toString(16),
16667
+ readyState: document.readyState,
16668
+ busy: !!document.querySelector('[aria-busy="true"]'),
16669
+ url: location.href.slice(0, 2000),
16670
+ scrollX: Math.round(scrollX),
16671
+ scrollY: Math.round(scrollY),
16672
+ innerWidth,
16673
+ innerHeight,
16674
+ devicePixelRatio,
16675
+ };
16676
+ })()`;
16677
+ }
16678
+ async function waitForBrowserStability(page, options) {
16679
+ const startedAt = Date.now();
16680
+ let lastSignature = null;
16681
+ let lastChangeAt = startedAt;
16682
+ let samples = 0;
16683
+ let changes = 0;
16684
+ let state = null;
16685
+ while (Date.now() - startedAt <= options.timeoutMs) {
16686
+ try {
16687
+ state = await evaluateChromeTarget(page.webSocketDebuggerUrl, buildBrowserStabilityExpression());
16688
+ samples++;
16689
+ const signature = JSON.stringify(state);
16690
+ const now = Date.now();
16691
+ if (lastSignature === null || signature !== lastSignature) {
16692
+ if (lastSignature !== null) changes++;
16693
+ lastSignature = signature;
16694
+ lastChangeAt = now;
16695
+ }
16696
+ const ready = typeof state === "object" && state !== null && "readyState" in state && state.readyState !== "loading" && "busy" in state && state.busy !== true;
16697
+ if (samples > 1 && ready && now - lastChangeAt >= options.stableMs) {
16698
+ return { stable: true, elapsedMs: now - startedAt, samples, changes, state };
16699
+ }
16700
+ } catch {
16701
+ }
16702
+ await sleep(options.pollMs);
16703
+ }
16704
+ return { stable: false, elapsedMs: Date.now() - startedAt, samples, changes, state };
16705
+ }
16706
+ function browserStateCachePath(targetId) {
16707
+ return `${STATE_DIR}/browser-state-${targetId.replace(/[^a-zA-Z0-9_-]/g, "_")}.json`;
16708
+ }
16709
+ function browserStateDiff(previous, current) {
16710
+ const before = new Map(previous.map((entry) => [entry.key, entry]));
16711
+ const after = new Map(current.map((entry) => [entry.key, entry]));
16712
+ const lines = [];
16713
+ let added = 0;
16714
+ let changed = 0;
16715
+ let removed = 0;
16716
+ for (const entry of current) {
16717
+ const old = before.get(entry.key);
16718
+ if (!old) {
16719
+ added++;
16720
+ lines.push(`+ ${entry.line}`);
16721
+ } else if (old.semantic !== entry.semantic) {
16722
+ changed++;
16723
+ lines.push(`~ ${old.line}
16724
+ ${entry.line}`);
16725
+ }
16726
+ }
16727
+ for (const entry of previous) {
16728
+ if (after.has(entry.key)) continue;
16729
+ removed++;
16730
+ lines.push(`- ${entry.line}`);
16731
+ }
16732
+ return { tree: lines.join("\n") || "No accessibility changes.", added, changed, removed };
16733
+ }
16734
+ async function computerBrowserStateCommand(path6, options = {}) {
16735
+ const textLimit = options.limit ? parseCoord(options.limit, "--limit") : 8e3;
16736
+ const elementLimit = options.elementLimit ? parseCoord(options.elementLimit, "--element-limit") : 120;
16737
+ const timeoutMs = options.timeout ? parseCoord(options.timeout, "--timeout") : 5e3;
16738
+ const stableMs = options.stableMs ? parseCoord(options.stableMs, "--stable-ms") : 500;
16739
+ const pollMs = options.pollMs ? parseCoord(options.pollMs, "--poll-ms") : 100;
16740
+ if (textLimit < 0 || textLimit > 5e4) fail("--limit must be between 0 and 50000");
16741
+ if (elementLimit < 0 || elementLimit > 500) fail("--element-limit must be between 0 and 500");
16742
+ if (timeoutMs < 0) fail("--timeout must be >= 0");
16743
+ if (stableMs < 0) fail("--stable-ms must be >= 0");
16744
+ if (pollMs < 50 || pollMs > 2e3) fail("--poll-ms must be between 50 and 2000");
16745
+ const page = await selectChromePage(options);
16746
+ const stability = await waitForBrowserStability(page, { timeoutMs, stableMs, pollMs });
16747
+ const target = resolvePath(path6);
16748
+ mkdirSync3(dirname3(target), { recursive: true });
16749
+ const [{ snapshot, entries }, screenshotResult] = await Promise.all([
16750
+ captureBrowserSnapshot(page, { textLimit, elementLimit }),
16751
+ sendChromeCommand(page.webSocketDebuggerUrl, "Page.captureScreenshot", {
16752
+ format: "png",
16753
+ fromSurface: true,
16754
+ captureBeyondViewport: false
16755
+ })
16756
+ ]);
16757
+ const data = screenshotResult.data;
16758
+ if (typeof data !== "string") fail("Chrome did not return screenshot data");
16759
+ writeFileSync3(target, Buffer.from(data, "base64"));
16760
+ const screenshot = readPngDimensions(target);
16761
+ const targetId = page.id ?? "unknown";
16762
+ const cachePath = browserStateCachePath(targetId);
16763
+ let mode = "full";
16764
+ let tree = snapshot.tree;
16765
+ let changes = { added: snapshot.nodeCount, changed: 0, removed: 0 };
16766
+ if (!options.full && existsSync3(cachePath)) {
16767
+ const cached = readBrowserStateCache(cachePath);
16768
+ if (cached?.url === snapshot.url && cached.documentId === snapshot.documentId) {
16769
+ mode = "diff";
16770
+ const diff = browserStateDiff(cached.entries, entries);
16771
+ tree = diff.tree;
16772
+ changes = { added: diff.added, changed: diff.changed, removed: diff.removed };
16773
+ }
16774
+ }
16775
+ mkdirSync3(STATE_DIR, { recursive: true });
16776
+ writeFileSync3(cachePath, JSON.stringify({ url: snapshot.url, documentId: snapshot.documentId, entries }));
16777
+ const stableState = isRecord(stability.state) ? stability.state : {};
16778
+ const state = {
16779
+ title: snapshot.title,
16780
+ url: snapshot.url,
16781
+ documentId: snapshot.documentId,
16782
+ revision: snapshot.revision,
16783
+ viewport: snapshot.viewport,
16784
+ text: snapshot.text,
16785
+ tree,
16786
+ nodeCount: snapshot.nodeCount,
16787
+ elementCount: snapshot.elementCount,
16788
+ truncated: snapshot.truncated,
16789
+ mode,
16790
+ changes
16791
+ };
16792
+ console.log(JSON.stringify({
16793
+ targetId,
16794
+ screenshot: {
16795
+ path: target,
16796
+ width: screenshot.width,
16797
+ height: screenshot.height,
16798
+ devicePixelRatio: typeof stableState.devicePixelRatio === "number" ? stableState.devicePixelRatio : 1
16799
+ },
16800
+ stability,
16801
+ state
16802
+ }, null, 2));
16803
+ }
16167
16804
  var DESKTOP_POINT_JS = `
16168
16805
  const desktopPoint = (rect) => {
16169
16806
  const borderX = Math.max(0, (window.outerWidth - window.innerWidth) / 2);
@@ -16173,17 +16810,95 @@ var DESKTOP_POINT_JS = `
16173
16810
  y: Math.round(window.screenY + topChrome + rect.y + rect.height / 2),
16174
16811
  };
16175
16812
  };`;
16813
+ function parseBrowserRef(value) {
16814
+ const ref = parseCoord(value.replace(/^ref=/, ""), "--ref");
16815
+ if (ref <= 0) fail("--ref must be a positive backend DOM node ID from the latest browser state");
16816
+ return ref;
16817
+ }
16818
+ function readBrowserRefCache(page) {
16819
+ const targetId = page.id;
16820
+ if (!targetId) fail("Chrome target has no stable ID. Capture fresh browser state and retry.");
16821
+ const cachePath = browserStateCachePath(targetId);
16822
+ if (!existsSync3(cachePath)) fail(`No browser state is cached for target ${targetId}. Run replicas computer browser-state <path> --target-id ${targetId} first.`);
16823
+ const cached = readBrowserStateCache(cachePath);
16824
+ if (!cached) fail(`Browser state for target ${targetId} is invalid. Capture fresh browser state and retry.`);
16825
+ if (cached.url !== (page.url ?? "").slice(0, 2e3)) fail("Cached browser state belongs to a different URL. Capture fresh browser state and retry.");
16826
+ return { targetId, cached };
16827
+ }
16828
+ function browserViewportRect(layout) {
16829
+ const viewport = chromeVisualViewport(layout);
16830
+ return {
16831
+ x: 0,
16832
+ y: 0,
16833
+ width: viewport.width,
16834
+ height: viewport.height,
16835
+ centerX: viewport.width / 2,
16836
+ centerY: viewport.height / 2
16837
+ };
16838
+ }
16839
+ async function prepareBrowserElement(page, refValue) {
16840
+ const backendNodeId = parseBrowserRef(refValue);
16841
+ const { cached } = readBrowserRefCache(page);
16842
+ return await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
16843
+ const layout = await sendChromeSessionCommand(session, "Page.getLayoutMetrics", {});
16844
+ const rect = await prepareBrowserRefInSession(session, backendNodeId, cached, browserViewportRect(layout));
16845
+ const axNodes = (await sendChromeSessionCommand(session, "Accessibility.getPartialAXTree", {
16846
+ backendNodeId,
16847
+ fetchRelatives: false
16848
+ })).nodes;
16849
+ const rawNode = Array.isArray(axNodes) && isRawAXNode(axNodes[0]) ? axNodes[0] : null;
16850
+ const states = rawNode ? browserStateProperties(rawNode) : {};
16851
+ const role = clippedText(rawNode?.role?.value, 80);
16852
+ const element = rawNode ? {
16853
+ ref: String(backendNodeId),
16854
+ role,
16855
+ name: clippedText(rawNode.name?.value) || null,
16856
+ description: clippedText(rawNode.description?.value) || null,
16857
+ value: typeof rawNode.value?.value === "string" ? clippedText(rawNode.value.value) : typeof rawNode.value?.value === "number" || typeof rawNode.value?.value === "boolean" ? rawNode.value.value : null,
16858
+ states,
16859
+ actions: [
16860
+ "click",
16861
+ ...["textbox", "searchbox", "combobox", "spinbutton", "slider"].includes(role) || states.editable === true ? ["set_value"] : [],
16862
+ "scroll"
16863
+ ]
16864
+ } : null;
16865
+ const desktopPoint = await evaluateChromeSession(session, `(() => {
16866
+ ${DESKTOP_POINT_JS}
16867
+ return desktopPoint({ x: ${rect.centerX}, y: ${rect.centerY}, width: 0, height: 0 });
16868
+ })()`);
16869
+ return {
16870
+ ref: String(backendNodeId),
16871
+ rect,
16872
+ element,
16873
+ desktopPoint: typeof desktopPoint === "object" && desktopPoint !== null && "x" in desktopPoint && typeof desktopPoint.x === "number" && "y" in desktopPoint && typeof desktopPoint.y === "number" ? { x: desktopPoint.x, y: desktopPoint.y } : null
16874
+ };
16875
+ });
16876
+ }
16877
+ async function prepareBrowserRefInSession(session, backendNodeId, cached, viewportRect) {
16878
+ if (!cached.entries.some(({ line }) => line.trimStart().startsWith(`[ref=${backendNodeId}]`))) {
16879
+ fail(`Browser element ref ${backendNodeId} is not in the latest state. Capture fresh browser state and retry.`);
16880
+ }
16881
+ const frameTree = (await sendChromeSessionCommand(session, "Page.getFrameTree", {})).frameTree;
16882
+ if (cached.documentId !== browserDocumentId(frameTree)) {
16883
+ fail(`Browser element ref ${backendNodeId} belongs to a previous document. Capture fresh browser state and retry.`);
16884
+ }
16885
+ try {
16886
+ await sendChromeSessionCommand(session, "DOM.scrollIntoViewIfNeeded", { backendNodeId });
16887
+ } catch (error) {
16888
+ fail(`Browser element ref ${backendNodeId} is stale or cannot be scrolled into view: ${error instanceof Error ? error.message : String(error)}`);
16889
+ }
16890
+ const quads = (await sendChromeSessionCommand(session, "DOM.getContentQuads", { backendNodeId })).quads;
16891
+ const candidates = browserContentQuadRects(quads, viewportRect);
16892
+ if (!candidates[0]) fail(`Browser element ref ${backendNodeId} has no visible content quad. Fetch fresh browser state and retry.`);
16893
+ return roundBrowserRect(candidates[0]);
16894
+ }
16176
16895
  function buildBrowserClickExpression(query, options) {
16177
16896
  return `(() => {
16178
16897
  const query = ${JSON.stringify(query)};
16179
16898
  const exact = ${JSON.stringify(options.exact)};
16180
16899
  const index = ${options.index};
16181
16900
  const normalize = (value) => String(value || '').replace(/\\s+/g, ' ').trim();
16182
- const visible = (el) => {
16183
- const rect = el.getBoundingClientRect();
16184
- const style = getComputedStyle(el);
16185
- return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
16186
- };
16901
+ ${VISIBLE_JS}
16187
16902
  const label = (el) => normalize(el.innerText || el.value || el.getAttribute('aria-label') || el.getAttribute('title') || el.getAttribute('placeholder') || el.href || el.id || el.name || el.tagName);
16188
16903
  ${DESKTOP_POINT_JS}
16189
16904
  const matches = (text) => {
@@ -16208,8 +16923,7 @@ ${DESKTOP_POINT_JS}
16208
16923
  const { el, text } = candidate;
16209
16924
  el.scrollIntoView({ block: 'center', inline: 'center' });
16210
16925
  const rect = el.getBoundingClientRect();
16211
- if (typeof el.focus === 'function') el.focus();
16212
- el.click();
16926
+ if (typeof el.focus === 'function') el.focus({ preventScroll: true });
16213
16927
  return {
16214
16928
  clicked: true,
16215
16929
  query,
@@ -16232,6 +16946,15 @@ ${DESKTOP_POINT_JS}
16232
16946
  };
16233
16947
  })()`;
16234
16948
  }
16949
+ function browserActionRectCenter(result) {
16950
+ if (typeof result !== "object" || result === null || !("rect" in result)) return null;
16951
+ const rect = result.rect;
16952
+ if (typeof rect !== "object" || rect === null) return null;
16953
+ const x = "centerX" in rect ? rect.centerX : void 0;
16954
+ const y = "centerY" in rect ? rect.centerY : void 0;
16955
+ if (typeof x !== "number" || typeof y !== "number" || !Number.isFinite(x) || !Number.isFinite(y)) return null;
16956
+ return { x, y };
16957
+ }
16235
16958
  function browserActionPoint(result) {
16236
16959
  if (typeof result !== "object" || result === null) return null;
16237
16960
  const action = "result" in result ? result.result : result;
@@ -16247,19 +16970,149 @@ function browserActionPoint(result) {
16247
16970
  y: clamp(Math.round(y), 0, dimensions.height)
16248
16971
  };
16249
16972
  }
16973
+ async function dispatchChromeClick(page, x, y, button, clicks) {
16974
+ await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
16975
+ await dispatchChromeClickInSession(session, x, y, button, clicks);
16976
+ });
16977
+ }
16978
+ async function dispatchChromeClickInSession(session, x, y, button, clicks) {
16979
+ for (let clickCount = 1; clickCount <= clicks; clickCount++) {
16980
+ await sendChromeSessionCommand(session, "Input.dispatchMouseEvent", {
16981
+ type: "mousePressed",
16982
+ x,
16983
+ y,
16984
+ button,
16985
+ clickCount
16986
+ });
16987
+ await sendChromeSessionCommand(session, "Input.dispatchMouseEvent", {
16988
+ type: "mouseReleased",
16989
+ x,
16990
+ y,
16991
+ button,
16992
+ clickCount
16993
+ });
16994
+ }
16995
+ }
16996
+ async function replaceChromeText(page, text) {
16997
+ await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
16998
+ await replaceChromeTextInSession(session, text);
16999
+ });
17000
+ }
17001
+ async function replaceChromeTextInSession(session, text) {
17002
+ await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
17003
+ type: "rawKeyDown",
17004
+ key: "a",
17005
+ code: "KeyA",
17006
+ windowsVirtualKeyCode: 65,
17007
+ nativeVirtualKeyCode: 65,
17008
+ modifiers: 2
17009
+ });
17010
+ await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
17011
+ type: "keyUp",
17012
+ key: "a",
17013
+ code: "KeyA",
17014
+ windowsVirtualKeyCode: 65,
17015
+ nativeVirtualKeyCode: 65,
17016
+ modifiers: 2
17017
+ });
17018
+ await sendChromeSessionCommand(session, "Input.insertText", { text });
17019
+ }
17020
+ var BROWSER_DIRECT_VALUE_INPUT_TYPES = /* @__PURE__ */ new Set(["number", "range", "date", "datetime-local", "month", "week", "time", "color"]);
17021
+ function browserFillInvalid(actual, expected, fieldType, semanticSelection = false, selectionMatched = false) {
17022
+ if (semanticSelection) return !selectionMatched;
17023
+ if (["number", "range"].includes(fieldType)) {
17024
+ const actualNumber = typeof actual === "number" ? actual : Number(actual);
17025
+ const expectedNumber = Number(expected);
17026
+ return !Number.isFinite(actualNumber) || !Number.isFinite(expectedNumber) || actualNumber !== expectedNumber;
17027
+ }
17028
+ if (fieldType === "color") return String(actual).toLowerCase() !== expected.toLowerCase();
17029
+ return actual !== expected;
17030
+ }
17031
+ async function fillChromeNodeInSession(session, backendNodeId, value) {
17032
+ await sendChromeSessionCommand(session, "DOM.focus", { backendNodeId });
17033
+ const directResult = await callFunctionOnChromeNodeInSession(
17034
+ session,
17035
+ backendNodeId,
17036
+ `function(value) {
17037
+ const emit = () => {
17038
+ this.dispatchEvent(new Event('input', { bubbles: true }));
17039
+ this.dispatchEvent(new Event('change', { bubbles: true }));
17040
+ };
17041
+ if (this instanceof HTMLSelectElement) {
17042
+ const option = Array.from(this.options).find((item) => item.value === value || item.text.trim() === value);
17043
+ if (option) this.value = option.value;
17044
+ else this.value = value;
17045
+ emit();
17046
+ return { handled: true, semanticSelection: true, matched: !!option, value: this.value };
17047
+ }
17048
+ if (this instanceof HTMLInputElement && ${JSON.stringify([...BROWSER_DIRECT_VALUE_INPUT_TYPES])}.includes(this.type.toLowerCase())) {
17049
+ const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')?.set;
17050
+ if (!setter) return { handled: false };
17051
+ setter.call(this, value);
17052
+ emit();
17053
+ return { handled: true, semanticSelection: false, matched: true, value: this.value };
17054
+ }
17055
+ return { handled: false };
17056
+ }`,
17057
+ [value]
17058
+ );
17059
+ const directValue = chromeCallValue(directResult);
17060
+ const directHandled = isRecord(directValue) && directValue.handled === true;
17061
+ if (!directHandled) await replaceChromeTextInSession(session, value);
17062
+ const verification = await callFunctionOnChromeNodeInSession(
17063
+ session,
17064
+ backendNodeId,
17065
+ `function() {
17066
+ const value = this instanceof HTMLSelectElement || 'value' in this ? this.value : this.textContent;
17067
+ const type = this instanceof HTMLSelectElement
17068
+ ? 'select'
17069
+ : this instanceof HTMLInputElement
17070
+ ? this.type.toLowerCase()
17071
+ : this.isContentEditable
17072
+ ? 'contenteditable'
17073
+ : 'text';
17074
+ return { value, valueLength: String(value ?? '').length, type };
17075
+ }`
17076
+ );
17077
+ const actual = chromeCallValue(verification);
17078
+ if (!isRecord(actual)) fail(`Chrome did not return a value for browser element ref ${backendNodeId}.`);
17079
+ const fieldType = typeof actual.type === "string" ? actual.type : "";
17080
+ const semanticSelection = isRecord(directValue) && directValue.semanticSelection === true;
17081
+ const selectionMatched = isRecord(directValue) && directValue.matched === true;
17082
+ if (browserFillInvalid(actual.value, value, fieldType, semanticSelection, selectionMatched)) {
17083
+ fail(`Chrome reported that ref ${backendNodeId} contains ${JSON.stringify(actual.value)} after filling, expected ${JSON.stringify(value)}.`);
17084
+ }
17085
+ return { value: actual.value, valueLength: actual.valueLength, type: fieldType };
17086
+ }
16250
17087
  async function computerBrowserClickCommand(query, options = {}) {
16251
17088
  const page = await selectChromePage(options);
17089
+ if (options.ref) {
17090
+ const target = await prepareBrowserElement(page, options.ref);
17091
+ const buttonNames = { "1": "left", "2": "middle", "3": "right", left: "left", middle: "middle", right: "right" };
17092
+ const button = buttonNames[(options.button ?? "left").toLowerCase()];
17093
+ if (!button) fail("--button must be one of left|middle|right|1|2|3");
17094
+ const clicks = options.double ? 2 : 1;
17095
+ await dispatchChromeClick(page, target.rect.centerX, target.rect.centerY, button, clicks);
17096
+ if (target.desktopPoint) logRecordingAction({ type: "click", ...target.desktopPoint });
17097
+ console.log(JSON.stringify({
17098
+ ...browserTargetOutput(page),
17099
+ result: { clicked: true, ref: target.ref, button, clickCount: clicks, rect: target.rect, element: target.element }
17100
+ }, null, 2));
17101
+ return;
17102
+ }
17103
+ if (!query) fail("Provide control text or --ref <id> from the latest browser state");
16252
17104
  const index = options.index ? parseCoord(options.index, "--index") : 0;
16253
17105
  if (index < 0) fail("--index must be >= 0");
16254
17106
  const result = await evaluateChromeTarget(
16255
17107
  page.webSocketDebuggerUrl,
16256
17108
  buildBrowserClickExpression(query, { exact: !!options.exact, index })
16257
17109
  );
17110
+ const center = browserActionRectCenter(result);
17111
+ if (center) await dispatchChromeClick(page, center.x, center.y, "left", 1);
16258
17112
  const point = browserActionPoint(result);
16259
17113
  if (point) logRecordingAction({ type: "click", ...point });
16260
17114
  console.log(JSON.stringify({
16261
- title: page.title ?? "",
16262
- url: page.url ?? "",
17115
+ ...browserTargetOutput(page),
16263
17116
  result
16264
17117
  }, null, 2));
16265
17118
  }
@@ -16271,11 +17124,7 @@ function buildBrowserFillExpression(query, value, options) {
16271
17124
  const index = ${options.index};
16272
17125
  const normalize = (text) => String(text || '').replace(/\\s+/g, ' ').trim();
16273
17126
  const escape = (text) => globalThis.CSS && CSS.escape ? CSS.escape(text) : String(text).replace(/["\\\\]/g, '\\\\$&');
16274
- const visible = (el) => {
16275
- const rect = el.getBoundingClientRect();
16276
- const style = getComputedStyle(el);
16277
- return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
16278
- };
17127
+ ${VISIBLE_JS}
16279
17128
  const matches = (text) => {
16280
17129
  const haystack = text.toLowerCase();
16281
17130
  const needle = query.toLowerCase();
@@ -16320,18 +17169,42 @@ ${DESKTOP_POINT_JS}
16320
17169
  }
16321
17170
  const { el, text } = candidate;
16322
17171
  el.scrollIntoView({ block: 'center', inline: 'center' });
16323
- if (typeof el.focus === 'function') el.focus();
17172
+ if (typeof el.focus === 'function') el.focus({ preventScroll: true });
17173
+ const fieldType = el.tagName.toLowerCase() === 'select' ? 'select' : String(el.getAttribute('type') || 'text').toLowerCase();
17174
+ let nativeInputRequired = false;
17175
+ let semanticSelection = false;
17176
+ let selectionMatched = false;
16324
17177
  if (el.tagName.toLowerCase() === 'select') {
16325
17178
  const option = Array.from(el.options).find((item) => item.value === value || normalize(item.text) === value);
16326
17179
  if (option) el.value = option.value;
16327
17180
  else el.value = value;
17181
+ semanticSelection = true;
17182
+ selectionMatched = !!option;
17183
+ el.dispatchEvent(new Event('input', { bubbles: true }));
17184
+ el.dispatchEvent(new Event('change', { bubbles: true }));
17185
+ } else if (el instanceof HTMLInputElement && ${JSON.stringify([...BROWSER_DIRECT_VALUE_INPUT_TYPES])}.includes(fieldType)) {
17186
+ const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')?.set;
17187
+ if (setter) {
17188
+ setter.call(el, value);
17189
+ el.dispatchEvent(new Event('input', { bubbles: true }));
17190
+ el.dispatchEvent(new Event('change', { bubbles: true }));
17191
+ } else {
17192
+ nativeInputRequired = true;
17193
+ }
16328
17194
  } else if (el.isContentEditable) {
16329
- el.textContent = value;
17195
+ const selection = getSelection();
17196
+ const range = document.createRange();
17197
+ range.selectNodeContents(el);
17198
+ selection.removeAllRanges();
17199
+ selection.addRange(range);
17200
+ nativeInputRequired = true;
17201
+ } else if (typeof el.select === 'function') {
17202
+ el.select();
17203
+ nativeInputRequired = true;
16330
17204
  } else {
16331
- el.value = value;
17205
+ el.setSelectionRange(0, String(el.value || '').length);
17206
+ nativeInputRequired = true;
16332
17207
  }
16333
- el.dispatchEvent(new InputEvent('input', { bubbles: true, inputType: 'insertText', data: value }));
16334
- el.dispatchEvent(new Event('change', { bubbles: true }));
16335
17208
  const rect = el.getBoundingClientRect();
16336
17209
  return {
16337
17210
  filled: true,
@@ -16340,9 +17213,13 @@ ${DESKTOP_POINT_JS}
16340
17213
  index,
16341
17214
  tag: el.tagName.toLowerCase(),
16342
17215
  role: el.getAttribute('role') || null,
16343
- type: el.getAttribute('type') || null,
17216
+ type: fieldType,
16344
17217
  text,
16345
- value: el.isContentEditable ? el.textContent : el.value,
17218
+ actualValue: el.isContentEditable ? el.textContent : 'value' in el ? el.value : null,
17219
+ valueLength: String(el.isContentEditable ? el.textContent : 'value' in el ? el.value : '').length,
17220
+ nativeInputRequired,
17221
+ semanticSelection,
17222
+ selectionMatched,
16346
17223
  desktopPoint: desktopPoint(rect),
16347
17224
  rect: {
16348
17225
  x: Math.round(rect.x),
@@ -16357,6 +17234,18 @@ ${DESKTOP_POINT_JS}
16357
17234
  }
16358
17235
  async function computerBrowserFillCommand(query, value, options = {}) {
16359
17236
  const page = await selectChromePage(options);
17237
+ if (options.ref) {
17238
+ const target = await prepareBrowserElement(page, options.ref);
17239
+ const backendNodeId = parseBrowserRef(options.ref);
17240
+ const actual = await withChromeSession(page.webSocketDebuggerUrl, async (session) => await fillChromeNodeInSession(session, backendNodeId, value));
17241
+ if (target.desktopPoint) logRecordingAction({ type: "type", ...target.desktopPoint });
17242
+ console.log(JSON.stringify({
17243
+ ...browserTargetOutput(page),
17244
+ result: { filled: true, ref: target.ref, valueLength: actual.valueLength, rect: target.rect, element: target.element }
17245
+ }, null, 2));
17246
+ return;
17247
+ }
17248
+ if (!query) fail("Provide field text or --ref <id> from the latest browser state");
16360
17249
  const index = options.index ? parseCoord(options.index, "--index") : 0;
16361
17250
  if (index < 0) fail("--index must be >= 0");
16362
17251
  const result = await evaluateChromeTarget(
@@ -16364,10 +17253,28 @@ async function computerBrowserFillCommand(query, value, options = {}) {
16364
17253
  buildBrowserFillExpression(query, value, { exact: !!options.exact, index })
16365
17254
  );
16366
17255
  const point = browserActionPoint(result);
17256
+ if (typeof result === "object" && result !== null && "nativeInputRequired" in result && result.nativeInputRequired === true) {
17257
+ await replaceChromeText(page, value);
17258
+ const actualValue = await evaluateChromeTarget(page.webSocketDebuggerUrl, `(() => {
17259
+ const el = document.activeElement;
17260
+ return el && el.isContentEditable ? el.textContent : el && 'value' in el ? el.value : null;
17261
+ })()`);
17262
+ const fieldType = "type" in result && typeof result.type === "string" ? result.type.toLowerCase() : "";
17263
+ if (browserFillInvalid(actualValue, value, fieldType)) {
17264
+ fail(`Chrome reported that ${JSON.stringify(query)} contains ${JSON.stringify(actualValue)} after filling, expected ${JSON.stringify(value)}.`);
17265
+ }
17266
+ if (isRecord(result)) {
17267
+ Object.assign(result, { actualValue, valueLength: String(actualValue ?? "").length });
17268
+ }
17269
+ } else if (isRecord(result) && result.filled === true) {
17270
+ const fieldType = typeof result.type === "string" ? result.type : "";
17271
+ if (browserFillInvalid(result.actualValue, value, fieldType, result.semanticSelection === true, result.selectionMatched === true)) {
17272
+ fail(`Chrome reported that ${JSON.stringify(query)} contains ${JSON.stringify(result.actualValue)} after filling, expected ${JSON.stringify(value)}.`);
17273
+ }
17274
+ }
16367
17275
  if (point) logRecordingAction({ type: "type", ...point });
16368
17276
  console.log(JSON.stringify({
16369
- title: page.title ?? "",
16370
- url: page.url ?? "",
17277
+ ...browserTargetOutput(page),
16371
17278
  result
16372
17279
  }, null, 2));
16373
17280
  }
@@ -16377,11 +17284,7 @@ function buildBrowserWaitExpression(query, options) {
16377
17284
  const mode = ${JSON.stringify(options.mode)};
16378
17285
  const exact = ${JSON.stringify(options.exact)};
16379
17286
  const normalize = (text) => String(text || '').replace(/\\s+/g, ' ').trim();
16380
- const visible = (el) => {
16381
- const rect = el.getBoundingClientRect();
16382
- const style = getComputedStyle(el);
16383
- return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
16384
- };
17287
+ ${VISIBLE_JS}
16385
17288
  const matches = (text) => {
16386
17289
  const haystack = normalize(text).toLowerCase();
16387
17290
  const needle = query.toLowerCase();
@@ -16436,6 +17339,241 @@ async function computerBrowserWaitCommand(query, options = {}) {
16436
17339
  console.log(JSON.stringify({ ok: false, elapsedMs: Date.now() - start, attempts, result: lastResult }, null, 2));
16437
17340
  process.exitCode = 1;
16438
17341
  }
17342
+ async function computerBrowserScrollCommand(direction, options = {}) {
17343
+ const page = await selectChromePage(options);
17344
+ const normalized = direction.toLowerCase();
17345
+ if (!["up", "down", "left", "right"].includes(normalized)) fail("direction must be one of up|down|left|right");
17346
+ const amount = options.amount ? parseCoord(options.amount, "--amount") : 600;
17347
+ if (amount <= 0 || amount > 1e4) fail("--amount must be between 1 and 10000 CSS pixels");
17348
+ const layout = await sendChromeCommand(page.webSocketDebuggerUrl, "Page.getLayoutMetrics", {});
17349
+ const viewport = chromeVisualViewport(layout);
17350
+ const target = options.ref ? await prepareBrowserElement(page, options.ref) : null;
17351
+ const x = target?.rect.centerX ?? viewport.width / 2;
17352
+ const y = target?.rect.centerY ?? viewport.height / 2;
17353
+ const deltaX = normalized === "left" ? -amount : normalized === "right" ? amount : 0;
17354
+ const deltaY = normalized === "up" ? -amount : normalized === "down" ? amount : 0;
17355
+ await sendChromeCommand(page.webSocketDebuggerUrl, "Input.dispatchMouseEvent", {
17356
+ type: "mouseWheel",
17357
+ x,
17358
+ y,
17359
+ deltaX,
17360
+ deltaY
17361
+ });
17362
+ logRecordingAction({ type: "scroll", ...target?.desktopPoint ?? {} });
17363
+ console.log(JSON.stringify({
17364
+ ...browserTargetOutput(page),
17365
+ result: { scrolled: true, direction: normalized, amount, ref: target?.ref ?? null, x: Math.round(x), y: Math.round(y) }
17366
+ }, null, 2));
17367
+ }
17368
+ function browserKeyDetails(combo) {
17369
+ const tokens = combo.split("+").map((token) => token.trim()).filter(Boolean);
17370
+ if (tokens.length === 0) fail("key combination cannot be empty");
17371
+ let modifiers = 0;
17372
+ for (const token of tokens.slice(0, -1)) {
17373
+ const normalized2 = token.toLowerCase();
17374
+ if (["alt", "option"].includes(normalized2)) modifiers |= 1;
17375
+ else if (["ctrl", "control"].includes(normalized2)) modifiers |= 2;
17376
+ else if (["meta", "cmd", "command", "super"].includes(normalized2)) modifiers |= 4;
17377
+ else if (normalized2 === "shift") modifiers |= 8;
17378
+ else fail(`Unsupported browser key modifier ${JSON.stringify(token)}`);
17379
+ }
17380
+ const requested = tokens[tokens.length - 1];
17381
+ const aliases = {
17382
+ enter: { key: "Enter", code: "Enter", windowsVirtualKeyCode: 13 },
17383
+ return: { key: "Enter", code: "Enter", windowsVirtualKeyCode: 13 },
17384
+ tab: { key: "Tab", code: "Tab", windowsVirtualKeyCode: 9 },
17385
+ escape: { key: "Escape", code: "Escape", windowsVirtualKeyCode: 27 },
17386
+ esc: { key: "Escape", code: "Escape", windowsVirtualKeyCode: 27 },
17387
+ backspace: { key: "Backspace", code: "Backspace", windowsVirtualKeyCode: 8 },
17388
+ delete: { key: "Delete", code: "Delete", windowsVirtualKeyCode: 46 },
17389
+ space: { key: " ", code: "Space", windowsVirtualKeyCode: 32 },
17390
+ up: { key: "ArrowUp", code: "ArrowUp", windowsVirtualKeyCode: 38 },
17391
+ arrowup: { key: "ArrowUp", code: "ArrowUp", windowsVirtualKeyCode: 38 },
17392
+ down: { key: "ArrowDown", code: "ArrowDown", windowsVirtualKeyCode: 40 },
17393
+ arrowdown: { key: "ArrowDown", code: "ArrowDown", windowsVirtualKeyCode: 40 },
17394
+ left: { key: "ArrowLeft", code: "ArrowLeft", windowsVirtualKeyCode: 37 },
17395
+ arrowleft: { key: "ArrowLeft", code: "ArrowLeft", windowsVirtualKeyCode: 37 },
17396
+ right: { key: "ArrowRight", code: "ArrowRight", windowsVirtualKeyCode: 39 },
17397
+ arrowright: { key: "ArrowRight", code: "ArrowRight", windowsVirtualKeyCode: 39 },
17398
+ home: { key: "Home", code: "Home", windowsVirtualKeyCode: 36 },
17399
+ end: { key: "End", code: "End", windowsVirtualKeyCode: 35 },
17400
+ pageup: { key: "PageUp", code: "PageUp", windowsVirtualKeyCode: 33 },
17401
+ pagedown: { key: "PageDown", code: "PageDown", windowsVirtualKeyCode: 34 }
17402
+ };
17403
+ const normalized = requested.toLowerCase();
17404
+ const alias = aliases[normalized];
17405
+ if (alias) return { ...alias, modifiers, ...alias.key === " " && modifiers === 0 ? { text: " " } : {} };
17406
+ if (/^[a-z]$/i.test(requested)) {
17407
+ const upper = requested.toUpperCase();
17408
+ const key = modifiers & 8 ? upper : requested.toLowerCase();
17409
+ return { key, code: `Key${upper}`, windowsVirtualKeyCode: upper.charCodeAt(0), modifiers, ...modifiers === 0 ? { text: key } : {} };
17410
+ }
17411
+ if (/^[0-9]$/.test(requested)) {
17412
+ return { key: requested, code: `Digit${requested}`, windowsVirtualKeyCode: requested.charCodeAt(0), modifiers, ...modifiers === 0 ? { text: requested } : {} };
17413
+ }
17414
+ fail(`Unsupported browser key ${JSON.stringify(requested)}`);
17415
+ }
17416
+ async function computerBrowserKeyCommand(combo, options = {}) {
17417
+ const page = await selectChromePage(options);
17418
+ const key = browserKeyDetails(combo);
17419
+ await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
17420
+ await dispatchChromeKeyInSession(session, key);
17421
+ });
17422
+ logRecordingAction({ type: "key" });
17423
+ console.log(JSON.stringify({ targetId: page.id ?? null, result: { pressed: combo } }, null, 2));
17424
+ }
17425
+ async function dispatchChromeKeyInSession(session, key) {
17426
+ await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
17427
+ type: "keyDown",
17428
+ ...key,
17429
+ nativeVirtualKeyCode: key.windowsVirtualKeyCode
17430
+ });
17431
+ await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
17432
+ type: "keyUp",
17433
+ key: key.key,
17434
+ code: key.code,
17435
+ modifiers: key.modifiers,
17436
+ windowsVirtualKeyCode: key.windowsVirtualKeyCode,
17437
+ nativeVirtualKeyCode: key.windowsVirtualKeyCode
17438
+ });
17439
+ }
17440
+ async function computerBrowserTypeCommand(text, options = {}) {
17441
+ const page = await selectChromePage(options);
17442
+ await sendChromeCommand(page.webSocketDebuggerUrl, "Input.insertText", { text });
17443
+ logRecordingAction({ type: "type" });
17444
+ console.log(JSON.stringify({ targetId: page.id ?? null, result: { typed: true, length: text.length } }, null, 2));
17445
+ }
17446
+ function parseBrowserBatchActions(value) {
17447
+ let input;
17448
+ try {
17449
+ input = JSON.parse(value);
17450
+ } catch (error) {
17451
+ fail(`Batch actions must be valid JSON: ${error instanceof Error ? error.message : String(error)}`);
17452
+ }
17453
+ if (!Array.isArray(input) || input.length === 0 || input.length > 100) {
17454
+ fail("Batch actions must be a JSON array containing 1 to 100 actions");
17455
+ }
17456
+ return input.map((item, index) => {
17457
+ if (!isRecord(item) || typeof item.action !== "string") fail(`Batch action ${index} must be an object with an action`);
17458
+ if (item.action === "click") {
17459
+ if (typeof item.ref !== "string" && typeof item.ref !== "number") fail(`Batch click ${index} requires ref`);
17460
+ const buttons = { "1": "left", "2": "middle", "3": "right", left: "left", middle: "middle", right: "right" };
17461
+ const requestedButton = item.button === void 0 ? "left" : String(item.button).toLowerCase();
17462
+ const button = buttons[requestedButton];
17463
+ if (!button) fail(`Batch click ${index} button must be left, middle, or right`);
17464
+ return { action: "click", ref: String(item.ref), button, clicks: item.double === true ? 2 : 1 };
17465
+ }
17466
+ if (item.action === "fill") {
17467
+ if (typeof item.ref !== "string" && typeof item.ref !== "number" || typeof item.value !== "string") {
17468
+ fail(`Batch fill ${index} requires ref and string value`);
17469
+ }
17470
+ return { action: "fill", ref: String(item.ref), value: item.value };
17471
+ }
17472
+ if (item.action === "key") {
17473
+ if (typeof item.combo !== "string") fail(`Batch key ${index} requires combo`);
17474
+ return { action: "key", combo: item.combo };
17475
+ }
17476
+ if (item.action === "type") {
17477
+ if (typeof item.text !== "string") fail(`Batch type ${index} requires text`);
17478
+ return { action: "type", text: item.text };
17479
+ }
17480
+ if (item.action === "scroll") {
17481
+ const directions = { up: "up", down: "down", left: "left", right: "right" };
17482
+ const direction = typeof item.direction === "string" ? directions[item.direction] : void 0;
17483
+ if (!direction) {
17484
+ fail(`Batch scroll ${index} direction must be up, down, left, or right`);
17485
+ }
17486
+ const amount = item.amount === void 0 ? 600 : Number(item.amount);
17487
+ if (!Number.isFinite(amount) || amount <= 0 || amount > 1e4) fail(`Batch scroll ${index} amount must be between 1 and 10000`);
17488
+ if (item.ref !== void 0 && typeof item.ref !== "string" && typeof item.ref !== "number") fail(`Batch scroll ${index} ref is invalid`);
17489
+ return {
17490
+ action: "scroll",
17491
+ direction,
17492
+ amount,
17493
+ ...item.ref === void 0 ? {} : { ref: String(item.ref) }
17494
+ };
17495
+ }
17496
+ if (item.action === "wait") {
17497
+ if (typeof item.text !== "string") fail(`Batch wait ${index} requires text`);
17498
+ const mode = item.mode === void 0 ? "any" : String(item.mode);
17499
+ if (!["any", "text", "title", "url", "control"].includes(mode)) fail(`Batch wait ${index} mode is invalid`);
17500
+ const timeoutMs = item.timeoutMs === void 0 ? 1e4 : Number(item.timeoutMs);
17501
+ const pollMs = item.pollMs === void 0 ? 250 : Number(item.pollMs);
17502
+ if (!Number.isFinite(timeoutMs) || timeoutMs < 0) fail(`Batch wait ${index} timeoutMs must be >= 0`);
17503
+ if (!Number.isFinite(pollMs) || pollMs < 50 || pollMs > 2e3) fail(`Batch wait ${index} pollMs must be between 50 and 2000`);
17504
+ return { action: "wait", text: item.text, mode, exact: item.exact === true, timeoutMs, pollMs };
17505
+ }
17506
+ fail(`Batch action ${index} has unsupported action ${JSON.stringify(item.action)}`);
17507
+ });
17508
+ }
17509
+ async function computerBrowserBatchCommand(actionsJson, options = {}) {
17510
+ const actions = parseBrowserBatchActions(actionsJson);
17511
+ const page = await selectChromePage(options);
17512
+ const { targetId, cached } = readBrowserRefCache(page);
17513
+ const startedAt = Date.now();
17514
+ const completed = [];
17515
+ try {
17516
+ await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
17517
+ const layout = await sendChromeSessionCommand(session, "Page.getLayoutMetrics", {});
17518
+ const viewportRect = browserViewportRect(layout);
17519
+ for (const [index, action] of actions.entries()) {
17520
+ const actionStartedAt = Date.now();
17521
+ if (action.action === "click") {
17522
+ const ref = parseBrowserRef(action.ref);
17523
+ const rect = await prepareBrowserRefInSession(session, ref, cached, viewportRect);
17524
+ await dispatchChromeClickInSession(session, rect.centerX, rect.centerY, action.button, action.clicks);
17525
+ logRecordingAction({ type: "click" });
17526
+ completed.push({ index, action: action.action, ref: String(ref), elapsedMs: Date.now() - actionStartedAt });
17527
+ } else if (action.action === "fill") {
17528
+ const ref = parseBrowserRef(action.ref);
17529
+ await prepareBrowserRefInSession(session, ref, cached, viewportRect);
17530
+ const result = await fillChromeNodeInSession(session, ref, action.value);
17531
+ logRecordingAction({ type: "type" });
17532
+ completed.push({ index, action: action.action, ref: String(ref), valueLength: result.valueLength, elapsedMs: Date.now() - actionStartedAt });
17533
+ } else if (action.action === "key") {
17534
+ await dispatchChromeKeyInSession(session, browserKeyDetails(action.combo));
17535
+ logRecordingAction({ type: "key" });
17536
+ completed.push({ index, action: action.action, elapsedMs: Date.now() - actionStartedAt });
17537
+ } else if (action.action === "type") {
17538
+ await sendChromeSessionCommand(session, "Input.insertText", { text: action.text });
17539
+ logRecordingAction({ type: "type" });
17540
+ completed.push({ index, action: action.action, length: action.text.length, elapsedMs: Date.now() - actionStartedAt });
17541
+ } else if (action.action === "scroll") {
17542
+ const ref = action.ref ? parseBrowserRef(action.ref) : null;
17543
+ const rect = ref ? await prepareBrowserRefInSession(session, ref, cached, viewportRect) : viewportRect;
17544
+ const deltaX = action.direction === "left" ? -action.amount : action.direction === "right" ? action.amount : 0;
17545
+ const deltaY = action.direction === "up" ? -action.amount : action.direction === "down" ? action.amount : 0;
17546
+ await sendChromeSessionCommand(session, "Input.dispatchMouseEvent", {
17547
+ type: "mouseWheel",
17548
+ x: rect.centerX,
17549
+ y: rect.centerY,
17550
+ deltaX,
17551
+ deltaY
17552
+ });
17553
+ logRecordingAction({ type: "scroll" });
17554
+ completed.push({ index, action: action.action, direction: action.direction, amount: action.amount, elapsedMs: Date.now() - actionStartedAt });
17555
+ } else {
17556
+ const expression = buildBrowserWaitExpression(action.text, { mode: action.mode, exact: action.exact });
17557
+ let attempts = 0;
17558
+ let result = null;
17559
+ while (Date.now() - actionStartedAt <= action.timeoutMs) {
17560
+ attempts++;
17561
+ result = await evaluateChromeSession(session, expression);
17562
+ if (isRecord(result) && result.matched === true) break;
17563
+ await sleep(action.pollMs);
17564
+ }
17565
+ if (!isRecord(result) || result.matched !== true) fail(`Batch wait ${index} timed out after ${Date.now() - actionStartedAt}ms`);
17566
+ completed.push({ index, action: action.action, attempts, elapsedMs: Date.now() - actionStartedAt });
17567
+ }
17568
+ }
17569
+ });
17570
+ } catch (error) {
17571
+ const message = error instanceof Error ? error.message : String(error);
17572
+ console.log(JSON.stringify({ ok: false, targetId, elapsedMs: Date.now() - startedAt, completed, failedIndex: completed.length, error: message }, null, 2));
17573
+ throw error;
17574
+ }
17575
+ console.log(JSON.stringify({ ok: true, targetId, elapsedMs: Date.now() - startedAt, completed }, null, 2));
17576
+ }
16439
17577
  async function computerClickCommand(xStr, yStr, options) {
16440
17578
  const dimensions = getDisplayDimensions();
16441
17579
  const x = parseScreenCoord(xStr, "x", dimensions.width);
@@ -16558,12 +17696,55 @@ async function computerLaunchCommand(app, args) {
16558
17696
  const baseArgs = app === "chrome" && !existsSync3(CHROME_WRAPPER) ? LEGACY_CHROME_ALIAS : APP_ALIASES[app] ?? [app];
16559
17697
  const bin = baseArgs[0];
16560
17698
  const fullArgs = [...baseArgs.slice(1), ...args];
17699
+ const expectsNewChromePage = args.some((arg) => !arg.startsWith("-"));
17700
+ const existingChromePageIds = /* @__PURE__ */ new Set();
17701
+ if (app === "chrome") {
17702
+ try {
17703
+ for (const page of await getChromePages()) {
17704
+ if (page.id) existingChromePageIds.add(page.id);
17705
+ }
17706
+ } catch {
17707
+ }
17708
+ }
16561
17709
  const child = spawn4(bin, fullArgs, {
16562
17710
  env: withDisplay(),
16563
17711
  detached: true,
16564
17712
  stdio: "ignore"
16565
17713
  });
16566
17714
  child.unref();
17715
+ if (app === "chrome") {
17716
+ const startedAt = Date.now();
17717
+ let pageCount = 0;
17718
+ let readyPage = null;
17719
+ while (Date.now() - startedAt < 15e3) {
17720
+ let pages = [];
17721
+ try {
17722
+ pages = await getChromePages();
17723
+ } catch {
17724
+ }
17725
+ pageCount = pages.length;
17726
+ const candidates = expectsNewChromePage && existingChromePageIds.size > 0 ? pages.filter((page) => page.id && !existingChromePageIds.has(page.id)) : pages;
17727
+ for (const page of candidates) {
17728
+ if (!page.webSocketDebuggerUrl || !page.url || page.url === "about:blank") continue;
17729
+ const readyState = await evaluateChromeTarget(page.webSocketDebuggerUrl, "document.readyState");
17730
+ if (readyState === "interactive" || readyState === "complete") {
17731
+ readyPage = page;
17732
+ break;
17733
+ }
17734
+ }
17735
+ if (readyPage) break;
17736
+ try {
17737
+ if (child.pid) process.kill(child.pid, 0);
17738
+ } catch {
17739
+ fail(`Chrome exited before its desktop control channel became ready.`);
17740
+ }
17741
+ await sleep(100);
17742
+ }
17743
+ if (!readyPage?.webSocketDebuggerUrl) fail(`Chrome launched but its requested page was not controllable after 15 seconds.`);
17744
+ await sendChromeCommand(readyPage.webSocketDebuggerUrl, "Page.bringToFront", {});
17745
+ console.log(`launched ${bin} (pid ${child.pid}, ready in ${Date.now() - startedAt}ms, page ${readyPage.id}, ${pageCount} open)`);
17746
+ return;
17747
+ }
16567
17748
  console.log(`launched ${bin} (pid ${child.pid})`);
16568
17749
  }
16569
17750
 
@@ -20990,10 +22171,20 @@ if (isAgentMode()) {
20990
22171
  computer.command("status").description("Show which desktop services are running and the active preview URL").action(wrap(() => computerStatusCommand()));
20991
22172
  computer.command("screenshot <path>").description("Capture the current desktop to a PNG file. Use --raw or --grid for agent click planning; omit both for a branded shareable image.").option("--raw", "Save a 1:1 desktop capture with no branding, padding, or rounded corners").option("--grid [px]", "Overlay a coordinate grid on a 1:1 desktop capture. Default 100px.").action(wrap((path6, options) => computerScreenshotCommand(path6, options)));
20992
22173
  computer.command("observe <path>").description("Wait briefly for the desktop to settle, save a 1:1 screenshot, and print JSON screen context for agents.").option("--raw", "Save a 1:1 desktop capture with no coordinate grid").option("--grid [px]", "Overlay a coordinate grid on the 1:1 desktop capture. Default 100px.").option("--timeout <ms>", "Maximum time to wait for visual stability. Default 3000.").option("--stable-ms <ms>", "Required unchanged time before the screen is considered stable. Default 600.").option("--poll-ms <ms>", "Screenshot polling interval while waiting. Default 200.").action(wrap((path6, options) => computerObserveCommand(path6, options)));
20993
- computer.command("browser").description("Print JSON for Chrome tabs launched through Replicas, including page titles and URLs.").option("--snapshot", "Include visible page text and interactive controls with bounding boxes").option("--limit <chars>", "Maximum body text characters per page snapshot. Default 4000.").option("--element-limit <n>", "Maximum controls per page snapshot. Default 80.").action(wrap((options) => computerBrowserCommand(options)));
20994
- computer.command("browser-click <text>").description("Click the first visible Chrome control whose text, label, placeholder, or href matches <text>.").option("--exact", "Require an exact text match instead of substring matching").option("--index <n>", "When multiple controls match, click the nth match. Default 0.").option("--id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserClickCommand(text, options)));
20995
- computer.command("browser-fill <field> <value>").description("Fill the first visible Chrome field whose label, placeholder, name, or text matches <field>.").option("--exact", "Require an exact field match instead of substring matching").option("--index <n>", "When multiple fields match, fill the nth match. Default 0.").option("--id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((field, value, options) => computerBrowserFillCommand(field, value, options)));
20996
- computer.command("browser-wait <text>").description("Wait until Chrome page title, URL, body text, or controls match <text>.").option("--mode <mode>", "Where to match: any, text, title, url, or control. Default any.").option("--exact", "Require an exact match instead of substring matching").option("--timeout <ms>", "Maximum time to wait. Default 10000.").option("--poll-ms <ms>", "Polling interval. Default 250.").option("--id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserWaitCommand(text, options)));
22174
+ computer.command("browser").description("Print JSON for Chrome tabs launched through Replicas, including page titles and URLs.").option("--snapshot", "Include visible page text and interactive controls with bounding boxes").option("--limit <chars>", "Maximum body text characters per page snapshot. Default 4000.").option("--element-limit <n>", "Maximum controls per page snapshot. Default 80.").option("--target-id <id>", "Only include the Chrome page with this target id").option("--page <n>", "When multiple pages match, include the nth page. Default 0.").option("--title <text>", "Only include pages whose title contains text").option("--url <text>", "Only include pages whose URL contains text").action(wrap((options) => computerBrowserCommand(options)));
22175
+ computer.command("browser-state <path>").description("Wait for a Chrome page to settle, capture its viewport, and print a bounded accessibility state with actionable refs.").option("--full", "Return the full accessibility tree instead of a diff from the previous state").option("--limit <chars>", "Maximum visible text characters. Default 8000.").option("--element-limit <n>", "Maximum actionable elements. Default 120.").option("--timeout <ms>", "Maximum time to wait for state stability. Default 5000.").option("--stable-ms <ms>", "Required unchanged time before capture. Default 500.").option("--poll-ms <ms>", "State polling interval. Default 100.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((path6, options) => computerBrowserStateCommand(path6, options)));
22176
+ computer.command("browser-batch <actions>").description("Run an ordered, fail-fast JSON array of browser actions through one Chrome session.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((actions, options) => computerBrowserBatchCommand(actions, options)));
22177
+ computer.command("browser-click [text]").description("Click a Chrome element by ref from browser-state, or fall back to visible control text.").option("--ref <id>", "Click the backend DOM node ref from the latest browser state").option("--exact", "Require an exact text match instead of substring matching").option("--index <n>", "When multiple controls match, click the nth match. Default 0.").option("--button <button>", "Mouse button for ref clicks: left, middle, or right. Default left.").option("--double", "Double-click a ref target").option("--target-id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserClickCommand(text, options)));
22178
+ computer.command("browser-fill [field] [value]").description("Fill a Chrome field by ref from browser-state, or fall back to visible field text.").option("--ref <id>", "Fill the backend DOM node ref from the latest browser state").option("--exact", "Require an exact field match instead of substring matching").option("--index <n>", "When multiple fields match, fill the nth match. Default 0.").option("--target-id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((field, value, options) => {
22179
+ const resolvedValue = value ?? (options.ref ? field : void 0);
22180
+ const resolvedField = value === void 0 && options.ref ? void 0 : field;
22181
+ if (resolvedValue === void 0) throw new Error("Provide <field> <value>, or --ref <id> <value>");
22182
+ return computerBrowserFillCommand(resolvedField, resolvedValue, options);
22183
+ }));
22184
+ computer.command("browser-scroll <direction>").description("Send trusted wheel input to a Chrome element ref or the center of the target page.").option("--ref <id>", "Scroll from the backend DOM node ref from the latest browser state").option("--amount <px>", "Scroll distance in CSS pixels. Default 600.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((direction, options) => computerBrowserScrollCommand(direction, options)));
22185
+ computer.command("browser-key <combo>").description("Press a key or modifier chord in the target Chrome page with trusted browser input.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((combo, options) => computerBrowserKeyCommand(combo, options)));
22186
+ computer.command("browser-type <text>").description("Type literal text into the focused field in the target Chrome page with trusted browser input.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserTypeCommand(text, options)));
22187
+ computer.command("browser-wait <text>").description("Wait until Chrome page title, URL, body text, or controls match <text>.").option("--mode <mode>", "Where to match: any, text, title, url, or control. Default any.").option("--exact", "Require an exact match instead of substring matching").option("--timeout <ms>", "Maximum time to wait. Default 10000.").option("--poll-ms <ms>", "Polling interval. Default 250.").option("--target-id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserWaitCommand(text, options)));
20997
22188
  computer.command("click <x> <y>").description("Move to (x, y) and click. Coordinates are pixels or percentages like 50% 50%.").option("-b, --button <n>", "Mouse button (1=left, 2=middle, 3=right). Default 1.").option("--double", "Double-click instead of single-click").option("--modifiers <mods>", "Hold modifier keys during the click, e.g. ctrl or ctrl+shift").action(wrap((x, y, options) => computerClickCommand(x, y, options)));
20998
22189
  computer.command("move <x> <y>").description("Move the mouse to (x, y) without clicking. Coordinates are pixels or percentages like 50% 50%.").action(wrap((x, y) => computerMoveCommand(x, y)));
20999
22190
  computer.command("type <text>").description("Type a literal string into the focused field. Use `key` for key combos like ctrl+l.").option("--delay <ms>", "Per-character delay in ms (default 12 \u2248 80 wpm)").action(wrap((text, options) => computerTypeCommand(text, options)));