replicas-cli 0.2.354 → 0.2.356
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/index.mjs +1377 -161
- package/package.json +1 -1
package/dist/index.mjs
CHANGED
|
@@ -8010,13 +8010,12 @@ replicas computer info
|
|
|
8010
8010
|
# 2) Launch a browser on the workspace display.
|
|
8011
8011
|
replicas computer launch chrome https://news.ycombinator.com
|
|
8012
8012
|
|
|
8013
|
-
# 3)
|
|
8014
|
-
replicas computer
|
|
8015
|
-
replicas computer browser --snapshot
|
|
8013
|
+
# 3) Capture the settled viewport and native accessibility tree together.
|
|
8014
|
+
replicas computer browser-state /tmp/state.png --full
|
|
8016
8015
|
|
|
8017
|
-
# 4)
|
|
8018
|
-
replicas computer
|
|
8019
|
-
replicas computer
|
|
8016
|
+
# 4) Batch actions that use refs from the same state, then capture the semantic diff.
|
|
8017
|
+
replicas computer browser-batch '[{"action":"fill","ref":17,"value":"hello"},{"action":"click","ref":42}]'
|
|
8018
|
+
replicas computer browser-state /tmp/after-click.png
|
|
8020
8019
|
|
|
8021
8020
|
# 5) (Optional) Record a screencap to share back.
|
|
8022
8021
|
replicas computer record start /tmp/demo.mp4 --fps 60
|
|
@@ -8045,21 +8044,33 @@ Waits briefly for the screen to stop changing, saves a 1:1 screenshot, and print
|
|
|
8045
8044
|
|
|
8046
8045
|
Use this instead of hand-written \`sleep && screenshot\` loops after clicks, navigation, typing, or page loads. By default it saves a 100px coordinate-grid screenshot and waits up to 3s for 600ms of visual stability. Pass \`--raw\` if you need an unannotated 1:1 screenshot.
|
|
8047
8046
|
|
|
8048
|
-
### \`replicas computer browser\`
|
|
8049
|
-
|
|
8047
|
+
### \`replicas computer browser-state <path> [--full] [--timeout MS] [--stable-ms MS]\`
|
|
8048
|
+
Waits for the selected Chrome page to settle, saves its viewport screenshot, and prints a compact native Chrome accessibility tree in the same coordinate space. Actionable nodes have DOM-backed refs such as \`[ref=42] button "Save"\`; the tree includes accessible roles, names, values, states, visible bounds, and iframe content.
|
|
8049
|
+
|
|
8050
|
+
The first call returns a full tree. Later calls for the same target and document return semantic changes; navigation or reload forces a new full tree. Pass \`--full\` to reset the baseline. Refs are valid for the current document.
|
|
8051
|
+
|
|
8052
|
+
### \`replicas computer browser [--snapshot]\`
|
|
8053
|
+
Lists Chrome tabs. \`--snapshot\` returns the compatibility accessibility snapshot. Use \`browser-state\` for the primary action/observe loop.
|
|
8054
|
+
|
|
8055
|
+
### \`replicas computer browser-batch '<actions-json>'\`
|
|
8056
|
+
Runs up to 100 ordered browser actions through one Chrome session and stops at the first failure. Supported actions are \`click\`, \`fill\`, \`key\`, \`type\`, \`scroll\`, and \`wait\`. Prefer this when several actions use refs from the same \`browser-state\`; it avoids repeated process, tab-selection, and connection setup. Do not carry refs across navigation\u2014capture fresh state and start a new batch.
|
|
8057
|
+
|
|
8058
|
+
### \`replicas computer browser-click --ref ID [--button left|middle|right] [--double]\`
|
|
8059
|
+
Scrolls the referenced node into view and sends trusted Chrome mouse input at its current visible bounds. Prefer refs from \`browser-state\`. \`browser-click <text> [--exact] [--index N]\` remains a fallback when no fresh ref is available.
|
|
8050
8060
|
|
|
8051
|
-
|
|
8061
|
+
### \`replicas computer browser-fill --ref ID <value>\`
|
|
8062
|
+
Focuses the referenced field, fills it with trusted Chrome input, and verifies the resulting value. Select controls use their semantic option value. \`browser-fill <field> <value> [--exact] [--index N]\` remains a text-matching fallback.
|
|
8052
8063
|
|
|
8053
|
-
### \`replicas computer browser-
|
|
8054
|
-
|
|
8064
|
+
### \`replicas computer browser-scroll <up|down|left|right> [--ref ID] [--amount PX]\`
|
|
8065
|
+
Sends trusted wheel input at a referenced element or the center of the viewport.
|
|
8055
8066
|
|
|
8056
|
-
### \`replicas computer browser-
|
|
8057
|
-
|
|
8067
|
+
### \`replicas computer browser-key <combo>\` / \`browser-type <text>\`
|
|
8068
|
+
Sends trusted keyboard input to the focused element in the selected Chrome target. Prefer \`browser-fill\` when setting a known field value.
|
|
8058
8069
|
|
|
8059
8070
|
### \`replicas computer browser-wait <text> [--mode any|text|title|url|control] [--exact] [--timeout MS]\`
|
|
8060
8071
|
Waits until the active Chrome page matches text in the title, URL, body text, or visible controls. Use this after \`browser-click\` / \`browser-fill\` when you need web app state to settle without screenshot polling.
|
|
8061
8072
|
|
|
8062
|
-
For
|
|
8073
|
+
For browser state and actions, pass \`--target-id <id>\`, \`--title <text>\`, \`--url <text>\`, or \`--page <n>\` when multiple Chrome tabs are open. Prefer \`--target-id\` because titles and URLs can change after an action. The matched tab is brought to the foreground before each action, so driving multiple tabs or recording the screen switches which tab is visible.
|
|
8063
8074
|
|
|
8064
8075
|
### \`replicas computer click <x> <y> [--button N] [--double] [--modifiers ctrl+shift]\`
|
|
8065
8076
|
Move to (x, y) and click. Coordinates can be absolute pixels or percentages such as \`50%\` \`50%\`. Default is left-click (button 1); pass \`--button 3\` for right-click. \`--modifiers\` holds keys during the click (e.g. ctrl-click a link to open in a new tab).
|
|
@@ -8092,26 +8103,29 @@ Spawns an app on the workspace display. Built-in aliases:
|
|
|
8092
8103
|
Anything else gets \`exec\`'d verbatim, so \`replicas computer launch xeyes\` works if xeyes is installed.
|
|
8093
8104
|
When opening a known page, prefer passing the URL directly: \`replicas computer launch chrome http://localhost:3000/\`.
|
|
8094
8105
|
|
|
8106
|
+
Chrome launches do not return until the requested page is controllable. The output includes its page ID; carry that ID with \`--target-id\` into \`browser-state\` and browser actions so unrelated tabs cannot steal task context.
|
|
8107
|
+
|
|
8095
8108
|
### \`replicas computer record start <path> [--fps N]\`
|
|
8096
|
-
Starts an ffmpeg screen recorder. Output is post-processed when stopped: action windows stay at normal speed, idle gaps accelerate, click moments get eased camera zoom, and a synthetic cursor animates between logged mouse positions. The raw capture is fragmented MP4 while active (still playable if the workspace dies mid-record). Default 60fps; drop to 30 if the workspace is CPU-constrained.
|
|
8109
|
+
Starts an ffmpeg screen recorder and returns only after ffmpeg has written the first bytes. Output is post-processed when stopped: action windows stay at normal speed, idle gaps accelerate, click moments get eased camera zoom, and a synthetic cursor animates between logged mouse positions. The raw capture is fragmented MP4 while active (still playable if the workspace dies mid-record). Default 60fps; drop to 30 if the workspace is CPU-constrained.
|
|
8097
8110
|
|
|
8098
8111
|
Only one recording at a time. Re-running \`start\` while one is active fails - call \`stop\` first.
|
|
8099
8112
|
|
|
8100
8113
|
### \`replicas computer record stop\`
|
|
8101
|
-
SIGINTs ffmpeg,
|
|
8114
|
+
SIGINTs ffmpeg, verifies that the process has finalized the MP4, then prints the output path. A finalization timeout preserves the recording state so \`stop\` can be retried safely. Upload it with \`replicas media upload <path>\` to share it.
|
|
8102
8115
|
|
|
8103
8116
|
## Patterns
|
|
8104
8117
|
|
|
8105
|
-
###
|
|
8106
|
-
|
|
8118
|
+
### Browser action / state loop
|
|
8119
|
+
For Chrome, use the native accessibility state before interpreting pixels or guessing coordinates:
|
|
8107
8120
|
|
|
8108
8121
|
\`\`\`bash
|
|
8109
|
-
replicas computer
|
|
8110
|
-
replicas computer
|
|
8111
|
-
|
|
8122
|
+
replicas computer launch chrome https://example.com
|
|
8123
|
+
replicas computer browser-state /tmp/before.png --full --target-id <page-id>
|
|
8124
|
+
replicas computer browser-batch '[{"action":"fill","ref":17,"value":"hello"},{"action":"click","ref":23}]' --target-id <page-id>
|
|
8125
|
+
replicas computer browser-state /tmp/after.png --target-id <page-id>
|
|
8112
8126
|
\`\`\`
|
|
8113
8127
|
|
|
8114
|
-
|
|
8128
|
+
Read the screenshot and tree as one state. Batch ordered actions whose refs all come from that state, then inspect the next semantic diff. A batch is fail-fast and validates each ref against the cached document. Split at navigation or whenever a later action depends on newly rendered refs. Fall back to \`observe\` plus desktop coordinates for canvas content, browser chrome, non-Chrome apps, or controls missing from the accessibility tree.
|
|
8115
8129
|
|
|
8116
8130
|
### Typing into an address bar
|
|
8117
8131
|
\`\`\`bash
|
|
@@ -8120,10 +8134,9 @@ replicas computer observe /tmp/browser-open.png
|
|
|
8120
8134
|
replicas computer key ctrl+l
|
|
8121
8135
|
replicas computer type "https://example.com"
|
|
8122
8136
|
replicas computer key Return
|
|
8123
|
-
replicas computer
|
|
8124
|
-
replicas computer browser --
|
|
8125
|
-
replicas computer browser-
|
|
8126
|
-
replicas computer browser-click "More information" --title "Example"
|
|
8137
|
+
replicas computer browser-state /tmp/loaded.png --full
|
|
8138
|
+
replicas computer browser-fill --ref 17 "replicas"
|
|
8139
|
+
replicas computer browser-click --ref 23 --title "Example"
|
|
8127
8140
|
replicas computer browser-wait "Example Domain" --mode title --title "Example"
|
|
8128
8141
|
\`\`\`
|
|
8129
8142
|
|
|
@@ -9283,6 +9296,7 @@ In agent mode the CLI hides commands that don't make sense for in-workspace agen
|
|
|
9283
9296
|
| \`replicas automation ...\` | Manage automations (cron + GitHub/GitLab event triggers) |
|
|
9284
9297
|
| \`replicas preview ...\` | Register / list preview URLs (covered in \`PREVIEWS.md\`) |
|
|
9285
9298
|
| \`replicas media ...\` | Upload screenshots, videos, audio (covered in \`MEDIA.md\`) |
|
|
9299
|
+
| \`replicas slack thread ...\` | Attach or switch Slack thread routing (covered in \`SLACK.md\`) |
|
|
9286
9300
|
|
|
9287
9301
|
\`replicas <command> --help\` is always the source of truth for flags.
|
|
9288
9302
|
|
|
@@ -9516,6 +9530,30 @@ curl -s -X POST "https://slack.com/api/chat.postMessage" \\
|
|
|
9516
9530
|
|
|
9517
9531
|
Omit \`thread_ts\` to post a new message to the channel. Include it to reply in a thread.
|
|
9518
9532
|
|
|
9533
|
+
#### Attach every new conversation
|
|
9534
|
+
|
|
9535
|
+
Whenever you send a Slack message outside the thread that already routes to the current workspace, immediately attach the destination thread so future replies are delivered to this workspace:
|
|
9536
|
+
|
|
9537
|
+
\`\`\`bash
|
|
9538
|
+
replicas slack thread attach --channel <channel-id> --thread-ts <thread-root-ts>
|
|
9539
|
+
\`\`\`
|
|
9540
|
+
|
|
9541
|
+
- For a new top-level message or DM, use the \`channel\` and \`ts\` from the successful \`chat.postMessage\` response. That message's \`ts\` is the new thread root.
|
|
9542
|
+
- For a reply in another existing thread, use its root \`thread_ts\`, not the reply's \`ts\`.
|
|
9543
|
+
- Do not attach the originating thread again when it already routes to the current workspace.
|
|
9544
|
+
- Treat sending and attaching as one operation: check that both API calls succeed, and report an attachment failure instead of implying future replies will reach you.
|
|
9545
|
+
|
|
9546
|
+
#### Switch a thread to an existing workspace
|
|
9547
|
+
|
|
9548
|
+
When the user wants a Slack thread to continue in a particular existing workspace, resolve the target and switch the thread:
|
|
9549
|
+
|
|
9550
|
+
\`\`\`bash
|
|
9551
|
+
replicas list
|
|
9552
|
+
replicas slack thread switch <workspace-id-or-name> --channel <channel-id> --thread-ts <thread-root-ts>
|
|
9553
|
+
\`\`\`
|
|
9554
|
+
|
|
9555
|
+
The channel and thread flags default to \`SLACK_CHANNEL_ID\` and \`SLACK_THREAD_TS\` for a Slack-triggered workspace. Workspace-authenticated agents may target only their current workspace; switching to a different workspace requires running the command with user authentication outside agent mode. If that restriction applies, give the user the exact command to run rather than claiming the switch succeeded.
|
|
9556
|
+
|
|
9519
9557
|
### Searching Messages
|
|
9520
9558
|
|
|
9521
9559
|
\`\`\`bash
|
|
@@ -9654,7 +9692,7 @@ var HOOK_EXEC_MAX_BUFFER_BYTES = 10 * 1024 * 1024;
|
|
|
9654
9692
|
var REPLICAS_CONFIG_FILENAMES = ["replicas.json", "replicas.yaml", "replicas.yml"];
|
|
9655
9693
|
|
|
9656
9694
|
// ../shared/src/cli-version.ts
|
|
9657
|
-
var CLI_VERSION = "0.2.
|
|
9695
|
+
var CLI_VERSION = "0.2.356";
|
|
9658
9696
|
|
|
9659
9697
|
// ../shared/src/version.ts
|
|
9660
9698
|
function compareVersions(v1, v2) {
|
|
@@ -15010,7 +15048,7 @@ var sleep = (ms) => new Promise((resolve2) => setTimeout(resolve2, ms));
|
|
|
15010
15048
|
|
|
15011
15049
|
// src/commands/computer/recording.ts
|
|
15012
15050
|
import { spawn as spawn3 } from "child_process";
|
|
15013
|
-
import { appendFileSync, existsSync as existsSync2, mkdirSync as mkdirSync2, readFileSync as readFileSync2, rmSync as rmSync2, writeFileSync as writeFileSync2 } from "fs";
|
|
15051
|
+
import { appendFileSync, existsSync as existsSync2, mkdirSync as mkdirSync2, readFileSync as readFileSync2, rmSync as rmSync2, statSync, writeFileSync as writeFileSync2 } from "fs";
|
|
15014
15052
|
import { dirname as dirname2 } from "path";
|
|
15015
15053
|
|
|
15016
15054
|
// src/commands/computer/recording/render.ts
|
|
@@ -15491,6 +15529,18 @@ var RECORD_STARTED_AT_FILE = `${STATE_DIR}/recording-started-at.txt`;
|
|
|
15491
15529
|
var RECORD_FPS_FILE = `${STATE_DIR}/recording-fps.txt`;
|
|
15492
15530
|
var RECORD_DIMENSIONS_FILE = `${STATE_DIR}/recording-dimensions.json`;
|
|
15493
15531
|
var RECORD_ACTIONS_FILE = `${STATE_DIR}/recording-actions.jsonl`;
|
|
15532
|
+
var RECORD_STATE_FILES = [
|
|
15533
|
+
RECORD_PID_FILE,
|
|
15534
|
+
RECORD_PATH_FILE,
|
|
15535
|
+
RECORD_RAW_PATH_FILE,
|
|
15536
|
+
RECORD_STARTED_AT_FILE,
|
|
15537
|
+
RECORD_FPS_FILE,
|
|
15538
|
+
RECORD_DIMENSIONS_FILE,
|
|
15539
|
+
RECORD_ACTIONS_FILE
|
|
15540
|
+
];
|
|
15541
|
+
function clearRecordingState() {
|
|
15542
|
+
for (const file of RECORD_STATE_FILES) rmSync2(file, { force: true });
|
|
15543
|
+
}
|
|
15494
15544
|
function recordingStartedAt() {
|
|
15495
15545
|
if (!existsSync2(RECORD_STARTED_AT_FILE)) return null;
|
|
15496
15546
|
const startedAt = Number.parseInt(readFileSync2(RECORD_STARTED_AT_FILE, "utf8").trim(), 10);
|
|
@@ -15548,13 +15598,13 @@ async function computerRecordStartCommand(path6, options) {
|
|
|
15548
15598
|
if (existsSync2(RECORD_PID_FILE)) {
|
|
15549
15599
|
const pid = parseInt(readFileSync2(RECORD_PID_FILE, "utf8").trim(), 10);
|
|
15550
15600
|
if (Number.isFinite(pid)) {
|
|
15551
|
-
let
|
|
15601
|
+
let alive2 = false;
|
|
15552
15602
|
try {
|
|
15553
15603
|
process.kill(pid, 0);
|
|
15554
|
-
|
|
15604
|
+
alive2 = true;
|
|
15555
15605
|
} catch {
|
|
15556
15606
|
}
|
|
15557
|
-
if (
|
|
15607
|
+
if (alive2) fail(`recording already in progress (pid ${pid}). run \`replicas computer record stop\` first.`);
|
|
15558
15608
|
}
|
|
15559
15609
|
}
|
|
15560
15610
|
const target = resolvePath(path6);
|
|
@@ -15601,7 +15651,29 @@ async function computerRecordStartCommand(path6, options) {
|
|
|
15601
15651
|
writeFileSync2(RECORD_STARTED_AT_FILE, String(Date.now()));
|
|
15602
15652
|
writeFileSync2(RECORD_FPS_FILE, String(fps));
|
|
15603
15653
|
writeFileSync2(RECORD_DIMENSIONS_FILE, JSON.stringify({ width, height }));
|
|
15604
|
-
|
|
15654
|
+
const startedAt = Date.now();
|
|
15655
|
+
while (Date.now() - startedAt < 5e3) {
|
|
15656
|
+
try {
|
|
15657
|
+
process.kill(child.pid, 0);
|
|
15658
|
+
if (existsSync2(rawTarget) && statSync(rawTarget).size > 0) {
|
|
15659
|
+
console.log(`${target} (recording ready in ${Date.now() - startedAt}ms)`);
|
|
15660
|
+
return;
|
|
15661
|
+
}
|
|
15662
|
+
} catch {
|
|
15663
|
+
break;
|
|
15664
|
+
}
|
|
15665
|
+
await sleep(100);
|
|
15666
|
+
}
|
|
15667
|
+
let alive = false;
|
|
15668
|
+
try {
|
|
15669
|
+
process.kill(child.pid, 0);
|
|
15670
|
+
alive = true;
|
|
15671
|
+
} catch {
|
|
15672
|
+
}
|
|
15673
|
+
if (alive) fail("ffmpeg is running but did not produce recording output within 5 seconds; run `replicas computer record stop` to finalize or retry");
|
|
15674
|
+
clearRecordingState();
|
|
15675
|
+
rmSync2(rawTarget, { force: true });
|
|
15676
|
+
fail("ffmpeg exited before screen recording became ready");
|
|
15605
15677
|
}
|
|
15606
15678
|
async function computerRecordStopCommand() {
|
|
15607
15679
|
if (!existsSync2(RECORD_PID_FILE) && !existsSync2(RECORD_PATH_FILE)) fail("no recording in progress");
|
|
@@ -15612,15 +15684,17 @@ async function computerRecordStopCommand() {
|
|
|
15612
15684
|
process.kill(pid, "SIGINT");
|
|
15613
15685
|
} catch {
|
|
15614
15686
|
}
|
|
15615
|
-
|
|
15687
|
+
let alive = true;
|
|
15688
|
+
for (let i = 0; i < 150; i++) {
|
|
15616
15689
|
try {
|
|
15617
15690
|
process.kill(pid, 0);
|
|
15618
15691
|
} catch {
|
|
15692
|
+
alive = false;
|
|
15619
15693
|
break;
|
|
15620
15694
|
}
|
|
15621
|
-
await
|
|
15695
|
+
await sleep(200);
|
|
15622
15696
|
}
|
|
15623
|
-
|
|
15697
|
+
if (alive) fail(`ffmpeg did not finalize recording within 30 seconds (pid ${pid})`);
|
|
15624
15698
|
}
|
|
15625
15699
|
if (existsSync2(RECORD_PATH_FILE)) {
|
|
15626
15700
|
const target = readFileSync2(RECORD_PATH_FILE, "utf8").trim();
|
|
@@ -15633,13 +15707,8 @@ async function computerRecordStopCommand() {
|
|
|
15633
15707
|
rmSync2(rawPath, { force: true });
|
|
15634
15708
|
}
|
|
15635
15709
|
console.log(target);
|
|
15636
|
-
rmSync2(RECORD_PATH_FILE, { force: true });
|
|
15637
|
-
rmSync2(RECORD_RAW_PATH_FILE, { force: true });
|
|
15638
|
-
rmSync2(RECORD_STARTED_AT_FILE, { force: true });
|
|
15639
|
-
rmSync2(RECORD_FPS_FILE, { force: true });
|
|
15640
|
-
rmSync2(RECORD_DIMENSIONS_FILE, { force: true });
|
|
15641
|
-
rmSync2(RECORD_ACTIONS_FILE, { force: true });
|
|
15642
15710
|
}
|
|
15711
|
+
clearRecordingState();
|
|
15643
15712
|
}
|
|
15644
15713
|
|
|
15645
15714
|
// src/commands/computer/index.ts
|
|
@@ -15978,8 +16047,8 @@ async function getChromePages() {
|
|
|
15978
16047
|
async function selectChromePage(options = {}) {
|
|
15979
16048
|
const pages = await getChromePages();
|
|
15980
16049
|
let matches = pages;
|
|
15981
|
-
if (options.
|
|
15982
|
-
matches = matches.filter((page2) => page2.id === options.
|
|
16050
|
+
if (options.targetId) {
|
|
16051
|
+
matches = matches.filter((page2) => page2.id === options.targetId);
|
|
15983
16052
|
}
|
|
15984
16053
|
if (options.title) {
|
|
15985
16054
|
const needle = options.title.toLowerCase();
|
|
@@ -15998,121 +16067,528 @@ async function selectChromePage(options = {}) {
|
|
|
15998
16067
|
`No matching debuggable Chrome page found (${matches.length} match${matches.length === 1 ? "" : "es"}, index ${index}). Use \`replicas computer browser\` to list pages.`
|
|
15999
16068
|
);
|
|
16000
16069
|
}
|
|
16070
|
+
await sendChromeCommand(webSocketDebuggerUrl, "Page.bringToFront", {});
|
|
16001
16071
|
return { ...page, webSocketDebuggerUrl };
|
|
16002
16072
|
}
|
|
16003
|
-
|
|
16004
|
-
return `(() => {
|
|
16005
|
-
const clip = (value, limit) => String(value || '').replace(/\\s+/g, ' ').trim().slice(0, limit);
|
|
16006
|
-
const escape = (value) => globalThis.CSS && CSS.escape ? CSS.escape(value) : String(value).replace(/["\\\\]/g, '\\\\$&');
|
|
16073
|
+
var VISIBLE_JS = `
|
|
16007
16074
|
const visible = (el) => {
|
|
16008
16075
|
const rect = el.getBoundingClientRect();
|
|
16009
16076
|
const style = getComputedStyle(el);
|
|
16010
16077
|
return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
|
|
16011
|
-
}
|
|
16012
|
-
|
|
16013
|
-
|
|
16014
|
-
const id = el.getAttribute('id');
|
|
16015
|
-
if (id) values.push(...Array.from(document.querySelectorAll('label[for="' + escape(id) + '"]')).map((label) => label.innerText));
|
|
16016
|
-
const wrappingLabel = el.closest('label');
|
|
16017
|
-
if (wrappingLabel) values.push(wrappingLabel.innerText);
|
|
16018
|
-
values.push(el.innerText, el.value, el.getAttribute('aria-label'), el.getAttribute('title'), el.getAttribute('placeholder'), el.href, el.id, el.name, el.tagName);
|
|
16019
|
-
return values.map((value) => clip(value, 160)).filter(Boolean);
|
|
16020
|
-
};
|
|
16021
|
-
const name = (el) => labels(el)[0] || clip(el.tagName, 160);
|
|
16022
|
-
const selector = 'a,button,input,textarea,select,[role="button"],[role="link"],[role="textbox"],[contenteditable="true"]';
|
|
16023
|
-
const controls = Array.from(document.querySelectorAll(selector))
|
|
16024
|
-
.filter(visible)
|
|
16025
|
-
.slice(0, ${elementLimit})
|
|
16026
|
-
.map((el) => {
|
|
16027
|
-
const rect = el.getBoundingClientRect();
|
|
16028
|
-
return {
|
|
16029
|
-
tag: el.tagName.toLowerCase(),
|
|
16030
|
-
role: el.getAttribute('role') || null,
|
|
16031
|
-
type: el.getAttribute('type') || null,
|
|
16032
|
-
text: name(el),
|
|
16033
|
-
href: el.href || null,
|
|
16034
|
-
disabled: !!el.disabled || el.getAttribute('aria-disabled') === 'true',
|
|
16035
|
-
rect: {
|
|
16036
|
-
x: Math.round(rect.x),
|
|
16037
|
-
y: Math.round(rect.y),
|
|
16038
|
-
width: Math.round(rect.width),
|
|
16039
|
-
height: Math.round(rect.height),
|
|
16040
|
-
centerX: Math.round(rect.x + rect.width / 2),
|
|
16041
|
-
centerY: Math.round(rect.y + rect.height / 2),
|
|
16042
|
-
},
|
|
16043
|
-
};
|
|
16044
|
-
});
|
|
16045
|
-
return {
|
|
16046
|
-
title: document.title,
|
|
16047
|
-
url: location.href,
|
|
16048
|
-
text: clip(document.body ? document.body.innerText : '', ${textLimit}),
|
|
16049
|
-
controls,
|
|
16050
|
-
};
|
|
16051
|
-
})()`;
|
|
16078
|
+
};`;
|
|
16079
|
+
async function sendChromeSessionCommand(session, method, params) {
|
|
16080
|
+
return chromeResult(await session.send({ method, params }), method);
|
|
16052
16081
|
}
|
|
16053
|
-
async function
|
|
16082
|
+
async function withChromeSession(webSocketDebuggerUrl, callback) {
|
|
16054
16083
|
return await new Promise((resolve2, reject) => {
|
|
16055
16084
|
const ws = new WebSocket(webSocketDebuggerUrl);
|
|
16085
|
+
const pending = /* @__PURE__ */ new Map();
|
|
16086
|
+
let nextId = 1;
|
|
16056
16087
|
let settled = false;
|
|
16057
|
-
const
|
|
16088
|
+
const failSession = (error) => {
|
|
16058
16089
|
if (settled) return;
|
|
16059
16090
|
settled = true;
|
|
16060
|
-
|
|
16091
|
+
for (const request of pending.values()) {
|
|
16092
|
+
clearTimeout(request.timeout);
|
|
16093
|
+
request.reject(error);
|
|
16094
|
+
}
|
|
16095
|
+
pending.clear();
|
|
16061
16096
|
ws.close();
|
|
16062
|
-
|
|
16097
|
+
reject(error);
|
|
16063
16098
|
};
|
|
16064
|
-
const
|
|
16065
|
-
|
|
16066
|
-
|
|
16067
|
-
|
|
16068
|
-
|
|
16069
|
-
|
|
16070
|
-
|
|
16071
|
-
|
|
16072
|
-
|
|
16073
|
-
|
|
16074
|
-
|
|
16075
|
-
|
|
16076
|
-
|
|
16099
|
+
const finishSession = (value) => {
|
|
16100
|
+
if (settled) return;
|
|
16101
|
+
settled = true;
|
|
16102
|
+
for (const request of pending.values()) clearTimeout(request.timeout);
|
|
16103
|
+
pending.clear();
|
|
16104
|
+
ws.close();
|
|
16105
|
+
resolve2(value);
|
|
16106
|
+
};
|
|
16107
|
+
const session = {
|
|
16108
|
+
send: ({ method, params }) => new Promise((resolveCommand, rejectCommand) => {
|
|
16109
|
+
const id = nextId++;
|
|
16110
|
+
const timeout = setTimeout(() => {
|
|
16111
|
+
pending.delete(id);
|
|
16112
|
+
rejectCommand(new Error(`Chrome DevTools ${method} timed out`));
|
|
16113
|
+
}, 1e4);
|
|
16114
|
+
pending.set(id, { method, resolve: resolveCommand, reject: rejectCommand, timeout });
|
|
16115
|
+
ws.send(JSON.stringify({ id, method, params }));
|
|
16116
|
+
})
|
|
16117
|
+
};
|
|
16118
|
+
ws.addEventListener("open", async () => {
|
|
16119
|
+
try {
|
|
16120
|
+
finishSession(await callback(session));
|
|
16121
|
+
} catch (error) {
|
|
16122
|
+
failSession(error instanceof Error ? error : new Error(String(error)));
|
|
16123
|
+
}
|
|
16077
16124
|
});
|
|
16078
16125
|
ws.addEventListener("message", (event) => {
|
|
16079
16126
|
const data = typeof event.data === "string" ? event.data : Buffer.isBuffer(event.data) ? event.data.toString("utf8") : Buffer.from(event.data).toString("utf8");
|
|
16080
|
-
let
|
|
16127
|
+
let message;
|
|
16081
16128
|
try {
|
|
16082
|
-
|
|
16129
|
+
message = JSON.parse(data);
|
|
16083
16130
|
} catch (error) {
|
|
16084
|
-
|
|
16085
|
-
finish(() => reject(new Error(`Chrome DevTools returned invalid WebSocket JSON: ${reason}`)));
|
|
16086
|
-
return;
|
|
16087
|
-
}
|
|
16088
|
-
if (typeof parsed !== "object" || parsed === null) {
|
|
16089
|
-
finish(() => reject(new Error("Chrome DevTools returned an invalid WebSocket message")));
|
|
16131
|
+
failSession(new Error(`Chrome DevTools returned invalid WebSocket JSON: ${error instanceof Error ? error.message : String(error)}`));
|
|
16090
16132
|
return;
|
|
16091
16133
|
}
|
|
16092
|
-
|
|
16093
|
-
|
|
16094
|
-
if (
|
|
16095
|
-
|
|
16096
|
-
|
|
16097
|
-
|
|
16098
|
-
|
|
16099
|
-
finish(() => reject(new Error(message.result.exceptionDetails.text || "Chrome DevTools evaluation threw")));
|
|
16100
|
-
return;
|
|
16101
|
-
}
|
|
16102
|
-
finish(() => resolve2(message.result?.result?.value ?? null));
|
|
16134
|
+
if (!isRecord(message) || typeof message.id !== "number") return;
|
|
16135
|
+
const request = pending.get(message.id);
|
|
16136
|
+
if (!request) return;
|
|
16137
|
+
clearTimeout(request.timeout);
|
|
16138
|
+
pending.delete(message.id);
|
|
16139
|
+
const detail = isRecord(message.error) && typeof message.error.message === "string" ? message.error.message : void 0;
|
|
16140
|
+
request.resolve(message.error ? { error: { message: detail } } : { result: isRecord(message.result) ? message.result : {} });
|
|
16103
16141
|
});
|
|
16104
|
-
ws.addEventListener("error", () =>
|
|
16105
|
-
|
|
16142
|
+
ws.addEventListener("error", () => failSession(new Error("Chrome DevTools WebSocket failed")));
|
|
16143
|
+
ws.addEventListener("close", () => {
|
|
16144
|
+
if (!settled) failSession(new Error("Chrome DevTools WebSocket closed unexpectedly"));
|
|
16106
16145
|
});
|
|
16107
16146
|
});
|
|
16108
16147
|
}
|
|
16148
|
+
async function sendChromeCommands(webSocketDebuggerUrl, commands) {
|
|
16149
|
+
return await withChromeSession(webSocketDebuggerUrl, async (session) => await Promise.all(commands.map((command) => session.send(command))));
|
|
16150
|
+
}
|
|
16151
|
+
async function sendChromeCommand(webSocketDebuggerUrl, method, params) {
|
|
16152
|
+
const response = (await sendChromeCommands(webSocketDebuggerUrl, [{ method, params }]))[0];
|
|
16153
|
+
if (response.error) throw new Error(response.error.message || `Chrome DevTools ${method} failed`);
|
|
16154
|
+
return response.result ?? {};
|
|
16155
|
+
}
|
|
16156
|
+
async function callFunctionOnChromeNodeInSession(session, backendNodeId, functionDeclaration, args = []) {
|
|
16157
|
+
const resolved = await sendChromeSessionCommand(session, "DOM.resolveNode", { backendNodeId });
|
|
16158
|
+
const objectId = isRecord(resolved.object) ? resolved.object.objectId : void 0;
|
|
16159
|
+
if (typeof objectId !== "string") throw new Error(`Could not resolve browser element ref ${backendNodeId}`);
|
|
16160
|
+
return await sendChromeSessionCommand(session, "Runtime.callFunctionOn", {
|
|
16161
|
+
objectId,
|
|
16162
|
+
functionDeclaration,
|
|
16163
|
+
arguments: args.map((value) => ({ value })),
|
|
16164
|
+
returnByValue: true
|
|
16165
|
+
});
|
|
16166
|
+
}
|
|
16167
|
+
function chromeCallValue(response) {
|
|
16168
|
+
return isRecord(response.result) ? response.result.value : void 0;
|
|
16169
|
+
}
|
|
16170
|
+
async function evaluateChromeTarget(webSocketDebuggerUrl, expression) {
|
|
16171
|
+
return await withChromeSession(webSocketDebuggerUrl, async (session) => await evaluateChromeSession(session, expression));
|
|
16172
|
+
}
|
|
16173
|
+
async function evaluateChromeSession(session, expression) {
|
|
16174
|
+
const response = await sendChromeSessionCommand(session, "Runtime.evaluate", {
|
|
16175
|
+
expression,
|
|
16176
|
+
awaitPromise: true,
|
|
16177
|
+
returnByValue: true
|
|
16178
|
+
});
|
|
16179
|
+
const exception = response.exceptionDetails;
|
|
16180
|
+
if (exception) {
|
|
16181
|
+
const text = typeof exception === "object" && exception !== null && "text" in exception && typeof exception.text === "string" ? exception.text : "";
|
|
16182
|
+
throw new Error(text || "Chrome DevTools evaluation threw");
|
|
16183
|
+
}
|
|
16184
|
+
const evaluated = response.result;
|
|
16185
|
+
return typeof evaluated === "object" && evaluated !== null && "value" in evaluated ? evaluated.value ?? null : null;
|
|
16186
|
+
}
|
|
16187
|
+
function isRawAXNode(value) {
|
|
16188
|
+
return isRecord(value);
|
|
16189
|
+
}
|
|
16190
|
+
function isDOMSnapshotDocument(value) {
|
|
16191
|
+
return isRecord(value);
|
|
16192
|
+
}
|
|
16193
|
+
function chromeVisualViewport(layout) {
|
|
16194
|
+
const viewport = isRecord(layout.cssVisualViewport) ? layout.cssVisualViewport : {};
|
|
16195
|
+
return {
|
|
16196
|
+
width: typeof viewport.clientWidth === "number" ? viewport.clientWidth : 0,
|
|
16197
|
+
height: typeof viewport.clientHeight === "number" ? viewport.clientHeight : 0
|
|
16198
|
+
};
|
|
16199
|
+
}
|
|
16200
|
+
function roundBrowserRect(rect) {
|
|
16201
|
+
return {
|
|
16202
|
+
x: Math.round(rect.x),
|
|
16203
|
+
y: Math.round(rect.y),
|
|
16204
|
+
width: Math.round(rect.width),
|
|
16205
|
+
height: Math.round(rect.height),
|
|
16206
|
+
centerX: Math.round(rect.centerX),
|
|
16207
|
+
centerY: Math.round(rect.centerY)
|
|
16208
|
+
};
|
|
16209
|
+
}
|
|
16210
|
+
function browserStateProperties(node) {
|
|
16211
|
+
const properties = [];
|
|
16212
|
+
for (const { name, value } of node.properties ?? []) {
|
|
16213
|
+
const state = value?.value;
|
|
16214
|
+
if (!name || !BROWSER_STATE_PROPERTIES.has(name) || typeof state !== "string" && typeof state !== "number" && typeof state !== "boolean") continue;
|
|
16215
|
+
properties.push([name, name === "url" ? clippedText(state, 300) : state]);
|
|
16216
|
+
}
|
|
16217
|
+
return Object.fromEntries(properties);
|
|
16218
|
+
}
|
|
16219
|
+
function readBrowserStateCache(path6) {
|
|
16220
|
+
let value;
|
|
16221
|
+
try {
|
|
16222
|
+
value = JSON.parse(readFileSync3(path6, "utf8"));
|
|
16223
|
+
} catch {
|
|
16224
|
+
return null;
|
|
16225
|
+
}
|
|
16226
|
+
if (!isRecord(value) || !Array.isArray(value.entries)) return null;
|
|
16227
|
+
const entries = [];
|
|
16228
|
+
for (const entry of value.entries) {
|
|
16229
|
+
if (!isRecord(entry) || typeof entry.key !== "string" || typeof entry.semantic !== "string" || typeof entry.line !== "string") return null;
|
|
16230
|
+
entries.push({ key: entry.key, semantic: entry.semantic, line: entry.line });
|
|
16231
|
+
}
|
|
16232
|
+
return {
|
|
16233
|
+
url: typeof value.url === "string" ? value.url : void 0,
|
|
16234
|
+
documentId: typeof value.documentId === "string" || value.documentId === null ? value.documentId : void 0,
|
|
16235
|
+
entries
|
|
16236
|
+
};
|
|
16237
|
+
}
|
|
16238
|
+
function chromeResult(response, method) {
|
|
16239
|
+
if (response.error) throw new Error(response.error.message || `Chrome DevTools ${method} failed`);
|
|
16240
|
+
return response.result ?? {};
|
|
16241
|
+
}
|
|
16242
|
+
function collectFrameIds(frameTree) {
|
|
16243
|
+
const ids = [];
|
|
16244
|
+
const visit = (value) => {
|
|
16245
|
+
if (!isRecord(value)) return;
|
|
16246
|
+
const frame = isRecord(value.frame) ? value.frame : {};
|
|
16247
|
+
if (typeof frame.id === "string") ids.push(frame.id);
|
|
16248
|
+
if (Array.isArray(value.childFrames)) value.childFrames.forEach(visit);
|
|
16249
|
+
};
|
|
16250
|
+
visit(frameTree);
|
|
16251
|
+
return ids;
|
|
16252
|
+
}
|
|
16253
|
+
function browserDocumentId(frameTree) {
|
|
16254
|
+
if (!isRecord(frameTree) || !isRecord(frameTree.frame)) return null;
|
|
16255
|
+
const loaderId = frameTree.frame.loaderId;
|
|
16256
|
+
return typeof loaderId === "string" ? loaderId : null;
|
|
16257
|
+
}
|
|
16258
|
+
function intersectRects(a, b) {
|
|
16259
|
+
const x = Math.max(a.x, b.x);
|
|
16260
|
+
const y = Math.max(a.y, b.y);
|
|
16261
|
+
const right = Math.min(a.x + a.width, b.x + b.width);
|
|
16262
|
+
const bottom = Math.min(a.y + a.height, b.y + b.height);
|
|
16263
|
+
if (right <= x || bottom <= y) return null;
|
|
16264
|
+
return {
|
|
16265
|
+
x,
|
|
16266
|
+
y,
|
|
16267
|
+
width: right - x,
|
|
16268
|
+
height: bottom - y,
|
|
16269
|
+
centerX: x + (right - x) / 2,
|
|
16270
|
+
centerY: y + (bottom - y) / 2
|
|
16271
|
+
};
|
|
16272
|
+
}
|
|
16273
|
+
function browserContentQuadRects(quads, viewportRect) {
|
|
16274
|
+
if (!Array.isArray(quads)) return [];
|
|
16275
|
+
return quads.filter((quad) => Array.isArray(quad) && quad.length >= 8 && quad.every(Number.isFinite)).map((quad) => {
|
|
16276
|
+
const xs = [quad[0], quad[2], quad[4], quad[6]];
|
|
16277
|
+
const ys = [quad[1], quad[3], quad[5], quad[7]];
|
|
16278
|
+
const x = Math.min(...xs);
|
|
16279
|
+
const y = Math.min(...ys);
|
|
16280
|
+
const width = Math.max(...xs) - x;
|
|
16281
|
+
const height = Math.max(...ys) - y;
|
|
16282
|
+
return { x, y, width, height, centerX: x + width / 2, centerY: y + height / 2 };
|
|
16283
|
+
}).map((rect) => intersectRects(rect, viewportRect)).filter((rect) => !!rect).sort((a, b) => b.width * b.height - a.width * a.height);
|
|
16284
|
+
}
|
|
16285
|
+
function buildBrowserBounds(result, viewport, targetId) {
|
|
16286
|
+
const strings = Array.isArray(result.strings) ? result.strings : [];
|
|
16287
|
+
const documents = Array.isArray(result.documents) ? result.documents.filter(isDOMSnapshotDocument) : [];
|
|
16288
|
+
const localRects = documents.map((document) => {
|
|
16289
|
+
const rects = /* @__PURE__ */ new Map();
|
|
16290
|
+
const nodeIndexes = document.layout?.nodeIndex ?? [];
|
|
16291
|
+
const bounds = document.layout?.bounds ?? [];
|
|
16292
|
+
const scrollX = document.scrollOffsetX ?? 0;
|
|
16293
|
+
const scrollY = document.scrollOffsetY ?? 0;
|
|
16294
|
+
nodeIndexes.forEach((nodeIndex, index) => {
|
|
16295
|
+
const bound = bounds[index];
|
|
16296
|
+
if (!Array.isArray(bound) || bound.length < 4 || !bound.every(Number.isFinite)) return;
|
|
16297
|
+
const [rawX, rawY, width, height] = bound;
|
|
16298
|
+
rects.set(nodeIndex, {
|
|
16299
|
+
x: rawX - scrollX,
|
|
16300
|
+
y: rawY - scrollY,
|
|
16301
|
+
width,
|
|
16302
|
+
height,
|
|
16303
|
+
centerX: rawX - scrollX + width / 2,
|
|
16304
|
+
centerY: rawY - scrollY + height / 2
|
|
16305
|
+
});
|
|
16306
|
+
});
|
|
16307
|
+
return rects;
|
|
16308
|
+
});
|
|
16309
|
+
const parents = /* @__PURE__ */ new Map();
|
|
16310
|
+
documents.forEach((document, documentIndex) => {
|
|
16311
|
+
const sparse = document.nodes?.contentDocumentIndex;
|
|
16312
|
+
sparse?.index?.forEach((nodeIndex, index) => {
|
|
16313
|
+
const childDocumentIndex = sparse.value?.[index];
|
|
16314
|
+
if (typeof childDocumentIndex === "number") parents.set(childDocumentIndex, { documentIndex, nodeIndex });
|
|
16315
|
+
});
|
|
16316
|
+
});
|
|
16317
|
+
const targetDocumentIndex = documents.findIndex(
|
|
16318
|
+
(document) => typeof document.frameId === "number" && strings[document.frameId] === targetId
|
|
16319
|
+
);
|
|
16320
|
+
const rootIndex = targetDocumentIndex >= 0 ? targetDocumentIndex : documents.findIndex((_, index) => !parents.has(index));
|
|
16321
|
+
const viewportRect = {
|
|
16322
|
+
x: 0,
|
|
16323
|
+
y: 0,
|
|
16324
|
+
width: viewport.width,
|
|
16325
|
+
height: viewport.height,
|
|
16326
|
+
centerX: viewport.width / 2,
|
|
16327
|
+
centerY: viewport.height / 2
|
|
16328
|
+
};
|
|
16329
|
+
const origins = /* @__PURE__ */ new Map();
|
|
16330
|
+
origins.set(rootIndex < 0 ? 0 : rootIndex, { x: 0, y: 0, clip: viewportRect });
|
|
16331
|
+
for (let pass = 0; pass < documents.length; pass++) {
|
|
16332
|
+
let changed = false;
|
|
16333
|
+
for (const [childIndex, parent] of parents) {
|
|
16334
|
+
if (origins.has(childIndex)) continue;
|
|
16335
|
+
const parentOrigin = origins.get(parent.documentIndex);
|
|
16336
|
+
const frameRect = localRects[parent.documentIndex]?.get(parent.nodeIndex);
|
|
16337
|
+
if (!parentOrigin || !frameRect) continue;
|
|
16338
|
+
const globalFrameRect = {
|
|
16339
|
+
...frameRect,
|
|
16340
|
+
x: parentOrigin.x + frameRect.x,
|
|
16341
|
+
y: parentOrigin.y + frameRect.y,
|
|
16342
|
+
centerX: parentOrigin.x + frameRect.centerX,
|
|
16343
|
+
centerY: parentOrigin.y + frameRect.centerY
|
|
16344
|
+
};
|
|
16345
|
+
const childDocument = documents[childIndex];
|
|
16346
|
+
const insetX = Math.max(0, (globalFrameRect.width - (childDocument?.contentWidth ?? globalFrameRect.width)) / 2);
|
|
16347
|
+
const insetY = Math.max(0, (globalFrameRect.height - (childDocument?.contentHeight ?? globalFrameRect.height)) / 2);
|
|
16348
|
+
const contentRect = {
|
|
16349
|
+
x: globalFrameRect.x + insetX,
|
|
16350
|
+
y: globalFrameRect.y + insetY,
|
|
16351
|
+
width: Math.max(0, globalFrameRect.width - insetX * 2),
|
|
16352
|
+
height: Math.max(0, globalFrameRect.height - insetY * 2),
|
|
16353
|
+
centerX: globalFrameRect.centerX,
|
|
16354
|
+
centerY: globalFrameRect.centerY
|
|
16355
|
+
};
|
|
16356
|
+
const clip = intersectRects(parentOrigin.clip, contentRect);
|
|
16357
|
+
if (!clip) continue;
|
|
16358
|
+
origins.set(childIndex, { x: contentRect.x, y: contentRect.y, clip });
|
|
16359
|
+
changed = true;
|
|
16360
|
+
}
|
|
16361
|
+
if (!changed) break;
|
|
16362
|
+
}
|
|
16363
|
+
const byBackendNodeId = /* @__PURE__ */ new Map();
|
|
16364
|
+
documents.forEach((document, documentIndex) => {
|
|
16365
|
+
const origin = origins.get(documentIndex);
|
|
16366
|
+
if (!origin) return;
|
|
16367
|
+
const backendNodeIds = document.nodes?.backendNodeId ?? [];
|
|
16368
|
+
for (const [nodeIndex, local] of localRects[documentIndex] ?? []) {
|
|
16369
|
+
const backendNodeId = backendNodeIds[nodeIndex];
|
|
16370
|
+
if (!Number.isInteger(backendNodeId)) continue;
|
|
16371
|
+
const rect = {
|
|
16372
|
+
...local,
|
|
16373
|
+
x: origin.x + local.x,
|
|
16374
|
+
y: origin.y + local.y,
|
|
16375
|
+
centerX: origin.x + local.centerX,
|
|
16376
|
+
centerY: origin.y + local.centerY
|
|
16377
|
+
};
|
|
16378
|
+
if (intersectRects(rect, origin.clip)) byBackendNodeId.set(backendNodeId, { rect, clip: origin.clip });
|
|
16379
|
+
}
|
|
16380
|
+
});
|
|
16381
|
+
return byBackendNodeId;
|
|
16382
|
+
}
|
|
16383
|
+
var BROWSER_INTERACTIVE_ROLES = /* @__PURE__ */ new Set([
|
|
16384
|
+
"button",
|
|
16385
|
+
"checkbox",
|
|
16386
|
+
"combobox",
|
|
16387
|
+
"link",
|
|
16388
|
+
"listbox",
|
|
16389
|
+
"menuitem",
|
|
16390
|
+
"menuitemcheckbox",
|
|
16391
|
+
"menuitemradio",
|
|
16392
|
+
"option",
|
|
16393
|
+
"radio",
|
|
16394
|
+
"scrollbar",
|
|
16395
|
+
"searchbox",
|
|
16396
|
+
"slider",
|
|
16397
|
+
"spinbutton",
|
|
16398
|
+
"switch",
|
|
16399
|
+
"tab",
|
|
16400
|
+
"textbox",
|
|
16401
|
+
"treeitem"
|
|
16402
|
+
]);
|
|
16403
|
+
var BROWSER_STRUCTURAL_ROLES = /* @__PURE__ */ new Set([
|
|
16404
|
+
"alert",
|
|
16405
|
+
"cell",
|
|
16406
|
+
"columnheader",
|
|
16407
|
+
"dialog",
|
|
16408
|
+
"figure",
|
|
16409
|
+
"gridcell",
|
|
16410
|
+
"heading",
|
|
16411
|
+
"image",
|
|
16412
|
+
"listitem",
|
|
16413
|
+
"main",
|
|
16414
|
+
"paragraph",
|
|
16415
|
+
"region",
|
|
16416
|
+
"row",
|
|
16417
|
+
"rowheader",
|
|
16418
|
+
"status",
|
|
16419
|
+
"StaticText"
|
|
16420
|
+
]);
|
|
16421
|
+
var BROWSER_STATE_PROPERTIES = /* @__PURE__ */ new Set([
|
|
16422
|
+
"autocomplete",
|
|
16423
|
+
"checked",
|
|
16424
|
+
"disabled",
|
|
16425
|
+
"editable",
|
|
16426
|
+
"expanded",
|
|
16427
|
+
"focusable",
|
|
16428
|
+
"focused",
|
|
16429
|
+
"haspopup",
|
|
16430
|
+
"invalid",
|
|
16431
|
+
"level",
|
|
16432
|
+
"multiselectable",
|
|
16433
|
+
"orientation",
|
|
16434
|
+
"pressed",
|
|
16435
|
+
"readonly",
|
|
16436
|
+
"protected",
|
|
16437
|
+
"required",
|
|
16438
|
+
"selected",
|
|
16439
|
+
"url"
|
|
16440
|
+
]);
|
|
16441
|
+
function clippedText(value, limit = 240) {
|
|
16442
|
+
return String(value ?? "").replace(/\s+/g, " ").trim().slice(0, limit);
|
|
16443
|
+
}
|
|
16444
|
+
function browserTargetOutput(page) {
|
|
16445
|
+
return {
|
|
16446
|
+
targetId: page.id ?? null,
|
|
16447
|
+
title: (page.title ?? "").slice(0, 1e3),
|
|
16448
|
+
url: (page.url ?? "").slice(0, 2e3)
|
|
16449
|
+
};
|
|
16450
|
+
}
|
|
16451
|
+
function browserNodeActionable(role, states) {
|
|
16452
|
+
return BROWSER_INTERACTIVE_ROLES.has(role) || states.editable === true || states.focusable === true && role !== "RootWebArea";
|
|
16453
|
+
}
|
|
16454
|
+
async function captureBrowserSnapshot(page, options) {
|
|
16455
|
+
const initialCommands = [
|
|
16456
|
+
{ method: "Page.getFrameTree", params: {} },
|
|
16457
|
+
{ method: "Page.getLayoutMetrics", params: {} },
|
|
16458
|
+
{ method: "DOMSnapshot.captureSnapshot", params: { computedStyles: [], includeDOMRects: true, includePaintOrder: true } }
|
|
16459
|
+
];
|
|
16460
|
+
const initial = await sendChromeCommands(page.webSocketDebuggerUrl, initialCommands);
|
|
16461
|
+
const frameTree = chromeResult(initial[0], initialCommands[0].method).frameTree;
|
|
16462
|
+
const layout = chromeResult(initial[1], initialCommands[1].method);
|
|
16463
|
+
const domSnapshot = chromeResult(initial[2], initialCommands[2].method);
|
|
16464
|
+
const viewport = chromeVisualViewport(layout);
|
|
16465
|
+
const frameIds = collectFrameIds(frameTree);
|
|
16466
|
+
if (frameIds.length === 0 && page.id) frameIds.push(page.id);
|
|
16467
|
+
const axCommands = frameIds.map((frameId) => ({
|
|
16468
|
+
method: "Accessibility.getFullAXTree",
|
|
16469
|
+
params: { frameId }
|
|
16470
|
+
}));
|
|
16471
|
+
const axResults = await sendChromeCommands(page.webSocketDebuggerUrl, axCommands);
|
|
16472
|
+
const nodes = [];
|
|
16473
|
+
axResults.forEach((response, index) => {
|
|
16474
|
+
if (response.error) return;
|
|
16475
|
+
const responseNodes = response.result?.nodes;
|
|
16476
|
+
if (!Array.isArray(responseNodes)) return;
|
|
16477
|
+
for (const node of responseNodes) {
|
|
16478
|
+
if (!isRawAXNode(node)) continue;
|
|
16479
|
+
nodes.push({ ...node, sourceFrameId: frameIds[index] });
|
|
16480
|
+
}
|
|
16481
|
+
});
|
|
16482
|
+
const bounds = buildBrowserBounds(domSnapshot, viewport, page.id);
|
|
16483
|
+
const nodesByKey = new Map(nodes.map((node) => [`${node.sourceFrameId}:${node.nodeId}`, node]));
|
|
16484
|
+
const propertyMap = browserStateProperties;
|
|
16485
|
+
const roleOf = (node) => clippedText(node.role?.value, 80);
|
|
16486
|
+
const nameOf = (node) => clippedText(node.name?.value);
|
|
16487
|
+
const valueOf = (node) => {
|
|
16488
|
+
if (propertyMap(node).protected === true) return null;
|
|
16489
|
+
const value = node.value?.value;
|
|
16490
|
+
if (typeof value === "number" || typeof value === "boolean") return value;
|
|
16491
|
+
const text = clippedText(value);
|
|
16492
|
+
if (/^[•●*]+$/.test(text)) return "[protected]";
|
|
16493
|
+
return text || null;
|
|
16494
|
+
};
|
|
16495
|
+
const meaningful = nodes.filter((node) => !node.ignored && Number.isInteger(node.backendDOMNodeId)).map((node) => ({
|
|
16496
|
+
node,
|
|
16497
|
+
role: roleOf(node),
|
|
16498
|
+
bound: typeof node.backendDOMNodeId === "number" ? bounds.get(node.backendDOMNodeId) : void 0
|
|
16499
|
+
})).filter(({ node, role, bound }) => {
|
|
16500
|
+
if (!bound || bound.rect.width <= 0 || bound.rect.height <= 0) return false;
|
|
16501
|
+
if (role === "InlineTextBox" || role === "none" || role === "generic") return false;
|
|
16502
|
+
if (BROWSER_INTERACTIVE_ROLES.has(role) || BROWSER_STRUCTURAL_ROLES.has(role)) return true;
|
|
16503
|
+
return !!nameOf(node) || valueOf(node) !== null || propertyMap(node).focusable === true;
|
|
16504
|
+
}).filter(({ node, role }) => {
|
|
16505
|
+
if (role !== "StaticText") return true;
|
|
16506
|
+
const parent = node.parentId ? nodesByKey.get(`${node.sourceFrameId}:${node.parentId}`) : void 0;
|
|
16507
|
+
return !parent || nameOf(parent) !== nameOf(node);
|
|
16508
|
+
}).sort((a, b) => a.bound.rect.y - b.bound.rect.y || a.bound.rect.x - b.bound.rect.x);
|
|
16509
|
+
const elements = [];
|
|
16510
|
+
const entries = [];
|
|
16511
|
+
const meaningfulKeys = new Set(meaningful.map(({ node }) => `${node.sourceFrameId}:${node.nodeId}`));
|
|
16512
|
+
const visibleText = [];
|
|
16513
|
+
let visibleTextLength = 0;
|
|
16514
|
+
let treeLength = 0;
|
|
16515
|
+
const treeLimit = Math.min(5e4, Math.max(4e3, options.textLimit * 2));
|
|
16516
|
+
for (const { node, role, bound } of meaningful) {
|
|
16517
|
+
const name = nameOf(node);
|
|
16518
|
+
const description = clippedText(node.description?.value) || null;
|
|
16519
|
+
const value = valueOf(node);
|
|
16520
|
+
const states = propertyMap(node);
|
|
16521
|
+
const actionable = browserNodeActionable(role, states);
|
|
16522
|
+
const ref = actionable && elements.length < options.elementLimit ? String(node.backendDOMNodeId) : null;
|
|
16523
|
+
const visibleRect = role === "RootWebArea" && node.sourceFrameId === page.id ? { x: 0, y: 0, width: viewport.width, height: viewport.height, centerX: viewport.width / 2, centerY: viewport.height / 2 } : intersectRects(bound.rect, bound.clip) ?? bound.rect;
|
|
16524
|
+
const rect = roundBrowserRect(visibleRect);
|
|
16525
|
+
const actions = [
|
|
16526
|
+
...actionable ? ["click"] : [],
|
|
16527
|
+
...["textbox", "searchbox", "combobox", "spinbutton", "slider"].includes(role) || states.editable === true ? ["set_value"] : [],
|
|
16528
|
+
"scroll"
|
|
16529
|
+
];
|
|
16530
|
+
if (ref) {
|
|
16531
|
+
elements.push({
|
|
16532
|
+
ref,
|
|
16533
|
+
role,
|
|
16534
|
+
name: name || null,
|
|
16535
|
+
description,
|
|
16536
|
+
value,
|
|
16537
|
+
states,
|
|
16538
|
+
actions,
|
|
16539
|
+
rect,
|
|
16540
|
+
frameId: node.sourceFrameId
|
|
16541
|
+
});
|
|
16542
|
+
}
|
|
16543
|
+
const stateText = Object.entries(states).filter(([key]) => key !== "url" && key !== "focusable").map(([key, state]) => `${key}=${JSON.stringify(state)}`);
|
|
16544
|
+
const url = typeof states.url === "string" ? clippedText(states.url, 300) : "";
|
|
16545
|
+
const semantic = [role, name && JSON.stringify(name), value !== null && `value=${JSON.stringify(value)}`, description && `description=${JSON.stringify(description)}`, url && `url=${JSON.stringify(url)}`, ...stateText].filter(Boolean).join(" ");
|
|
16546
|
+
let depth = 0;
|
|
16547
|
+
let parentId = node.parentId;
|
|
16548
|
+
const visited = /* @__PURE__ */ new Set();
|
|
16549
|
+
while (parentId && depth < 8) {
|
|
16550
|
+
const parentKey = `${node.sourceFrameId}:${parentId}`;
|
|
16551
|
+
if (visited.has(parentKey)) break;
|
|
16552
|
+
visited.add(parentKey);
|
|
16553
|
+
if (meaningfulKeys.has(parentKey)) depth++;
|
|
16554
|
+
parentId = nodesByKey.get(parentKey)?.parentId;
|
|
16555
|
+
}
|
|
16556
|
+
const line = `${" ".repeat(depth)}${ref ? `[ref=${ref}]` : "-"} ${semantic} (${rect.x},${rect.y} ${rect.width}x${rect.height})`;
|
|
16557
|
+
if (treeLength + line.length + 1 <= treeLimit) {
|
|
16558
|
+
const key = Number.isInteger(node.backendDOMNodeId) ? `dom:${node.sourceFrameId}:${node.backendDOMNodeId}` : `ax:${node.sourceFrameId}:${node.nodeId}`;
|
|
16559
|
+
entries.push({ key, semantic, line });
|
|
16560
|
+
treeLength += line.length + 1;
|
|
16561
|
+
}
|
|
16562
|
+
if (name && visibleTextLength < options.textLimit && (role === "StaticText" || role === "heading" || BROWSER_INTERACTIVE_ROLES.has(role))) {
|
|
16563
|
+
const remaining = options.textLimit - visibleTextLength;
|
|
16564
|
+
visibleText.push(name.slice(0, remaining));
|
|
16565
|
+
visibleTextLength += Math.min(name.length, remaining) + 1;
|
|
16566
|
+
}
|
|
16567
|
+
}
|
|
16568
|
+
const revision = createHash2("sha256").update(JSON.stringify([page.url, entries.map(({ key, semantic }) => [key, semantic])])).digest("hex").slice(0, 16);
|
|
16569
|
+
const snapshot = {
|
|
16570
|
+
title: (page.title ?? "").slice(0, 1e3),
|
|
16571
|
+
url: (page.url ?? "").slice(0, 2e3),
|
|
16572
|
+
documentId: browserDocumentId(frameTree),
|
|
16573
|
+
revision,
|
|
16574
|
+
viewport,
|
|
16575
|
+
text: visibleText.join(" ").slice(0, options.textLimit),
|
|
16576
|
+
tree: entries.map(({ line }) => line).join("\n"),
|
|
16577
|
+
elements,
|
|
16578
|
+
controls: elements,
|
|
16579
|
+
nodeCount: entries.length,
|
|
16580
|
+
elementCount: elements.length,
|
|
16581
|
+
truncated: entries.length < meaningful.length || elements.length < meaningful.filter(({ node, role }) => browserNodeActionable(role, propertyMap(node))).length
|
|
16582
|
+
};
|
|
16583
|
+
return { snapshot, entries };
|
|
16584
|
+
}
|
|
16109
16585
|
async function computerBrowserCommand(options = {}) {
|
|
16110
16586
|
const textLimit = options.limit ? parseCoord(options.limit, "--limit") : 4e3;
|
|
16111
16587
|
const elementLimit = options.elementLimit ? parseCoord(options.elementLimit, "--element-limit") : 80;
|
|
16112
16588
|
if (textLimit < 0 || textLimit > 5e4) fail("--limit must be between 0 and 50000");
|
|
16113
16589
|
if (elementLimit < 0 || elementLimit > 500) fail("--element-limit must be between 0 and 500");
|
|
16114
|
-
const
|
|
16115
|
-
const pages =
|
|
16590
|
+
const targets = options.targetId || options.page || options.title || options.url ? [await selectChromePage(options)] : await getChromePages();
|
|
16591
|
+
const pages = targets.map((target) => ({
|
|
16116
16592
|
target,
|
|
16117
16593
|
page: {
|
|
16118
16594
|
id: target.id ?? null,
|
|
@@ -16125,7 +16601,11 @@ async function computerBrowserCommand(options = {}) {
|
|
|
16125
16601
|
for (const { target, page } of pages) {
|
|
16126
16602
|
if (options.snapshot && target.webSocketDebuggerUrl) {
|
|
16127
16603
|
try {
|
|
16128
|
-
|
|
16604
|
+
const { snapshot } = await captureBrowserSnapshot(
|
|
16605
|
+
{ ...target, webSocketDebuggerUrl: target.webSocketDebuggerUrl },
|
|
16606
|
+
{ textLimit, elementLimit }
|
|
16607
|
+
);
|
|
16608
|
+
pageResults.push({ ...page, snapshot });
|
|
16129
16609
|
} catch (error) {
|
|
16130
16610
|
pageResults.push({ ...page, snapshotError: error instanceof Error ? error.message : "snapshot failed" });
|
|
16131
16611
|
}
|
|
@@ -16139,6 +16619,188 @@ async function computerBrowserCommand(options = {}) {
|
|
|
16139
16619
|
pages: pageResults
|
|
16140
16620
|
}, null, 2));
|
|
16141
16621
|
}
|
|
16622
|
+
function buildBrowserStabilityExpression() {
|
|
16623
|
+
return `(() => {
|
|
16624
|
+
const key = '__replicasComputerUseObserver';
|
|
16625
|
+
if (!globalThis[key]) {
|
|
16626
|
+
const state = { revision: 0 };
|
|
16627
|
+
const observer = new MutationObserver(() => state.revision++);
|
|
16628
|
+
globalThis[key] = { state, observer, observing: false };
|
|
16629
|
+
}
|
|
16630
|
+
if (document.documentElement && !globalThis[key].observing) {
|
|
16631
|
+
globalThis[key].observer.observe(document.documentElement, {
|
|
16632
|
+
subtree: true,
|
|
16633
|
+
childList: true,
|
|
16634
|
+
attributes: true,
|
|
16635
|
+
characterData: true,
|
|
16636
|
+
});
|
|
16637
|
+
globalThis[key].observing = true;
|
|
16638
|
+
}
|
|
16639
|
+
const controls = Array.from(document.querySelectorAll('input,textarea,select,[contenteditable="true"],[role="checkbox"],[role="combobox"],[role="slider"],[role="switch"],[role="textbox"]')).slice(0, 200);
|
|
16640
|
+
const active = document.activeElement;
|
|
16641
|
+
const controlState = controls.map((element, index) => {
|
|
16642
|
+
const value = 'value' in element
|
|
16643
|
+
? element instanceof HTMLInputElement && element.type === 'password'
|
|
16644
|
+
? String(element.value).length
|
|
16645
|
+
: String(element.value)
|
|
16646
|
+
: element.textContent;
|
|
16647
|
+
return [
|
|
16648
|
+
index,
|
|
16649
|
+
element === active,
|
|
16650
|
+
value,
|
|
16651
|
+
'checked' in element ? element.checked : null,
|
|
16652
|
+
'selectedIndex' in element ? element.selectedIndex : null,
|
|
16653
|
+
'disabled' in element ? element.disabled : null,
|
|
16654
|
+
element.getAttribute('aria-checked'),
|
|
16655
|
+
element.getAttribute('aria-expanded'),
|
|
16656
|
+
element.getAttribute('aria-selected'),
|
|
16657
|
+
].join('|');
|
|
16658
|
+
}).join('\\u001f');
|
|
16659
|
+
let controlRevision = 2166136261;
|
|
16660
|
+
for (let index = 0; index < controlState.length; index++) {
|
|
16661
|
+
controlRevision ^= controlState.charCodeAt(index);
|
|
16662
|
+
controlRevision = Math.imul(controlRevision, 16777619);
|
|
16663
|
+
}
|
|
16664
|
+
return {
|
|
16665
|
+
revision: globalThis[key].state.revision,
|
|
16666
|
+
controlRevision: (controlRevision >>> 0).toString(16),
|
|
16667
|
+
readyState: document.readyState,
|
|
16668
|
+
busy: !!document.querySelector('[aria-busy="true"]'),
|
|
16669
|
+
url: location.href.slice(0, 2000),
|
|
16670
|
+
scrollX: Math.round(scrollX),
|
|
16671
|
+
scrollY: Math.round(scrollY),
|
|
16672
|
+
innerWidth,
|
|
16673
|
+
innerHeight,
|
|
16674
|
+
devicePixelRatio,
|
|
16675
|
+
};
|
|
16676
|
+
})()`;
|
|
16677
|
+
}
|
|
16678
|
+
async function waitForBrowserStability(page, options) {
|
|
16679
|
+
const startedAt = Date.now();
|
|
16680
|
+
let lastSignature = null;
|
|
16681
|
+
let lastChangeAt = startedAt;
|
|
16682
|
+
let samples = 0;
|
|
16683
|
+
let changes = 0;
|
|
16684
|
+
let state = null;
|
|
16685
|
+
while (Date.now() - startedAt <= options.timeoutMs) {
|
|
16686
|
+
try {
|
|
16687
|
+
state = await evaluateChromeTarget(page.webSocketDebuggerUrl, buildBrowserStabilityExpression());
|
|
16688
|
+
samples++;
|
|
16689
|
+
const signature = JSON.stringify(state);
|
|
16690
|
+
const now = Date.now();
|
|
16691
|
+
if (lastSignature === null || signature !== lastSignature) {
|
|
16692
|
+
if (lastSignature !== null) changes++;
|
|
16693
|
+
lastSignature = signature;
|
|
16694
|
+
lastChangeAt = now;
|
|
16695
|
+
}
|
|
16696
|
+
const ready = typeof state === "object" && state !== null && "readyState" in state && state.readyState !== "loading" && "busy" in state && state.busy !== true;
|
|
16697
|
+
if (samples > 1 && ready && now - lastChangeAt >= options.stableMs) {
|
|
16698
|
+
return { stable: true, elapsedMs: now - startedAt, samples, changes, state };
|
|
16699
|
+
}
|
|
16700
|
+
} catch {
|
|
16701
|
+
}
|
|
16702
|
+
await sleep(options.pollMs);
|
|
16703
|
+
}
|
|
16704
|
+
return { stable: false, elapsedMs: Date.now() - startedAt, samples, changes, state };
|
|
16705
|
+
}
|
|
16706
|
+
function browserStateCachePath(targetId) {
|
|
16707
|
+
return `${STATE_DIR}/browser-state-${targetId.replace(/[^a-zA-Z0-9_-]/g, "_")}.json`;
|
|
16708
|
+
}
|
|
16709
|
+
function browserStateDiff(previous, current) {
|
|
16710
|
+
const before = new Map(previous.map((entry) => [entry.key, entry]));
|
|
16711
|
+
const after = new Map(current.map((entry) => [entry.key, entry]));
|
|
16712
|
+
const lines = [];
|
|
16713
|
+
let added = 0;
|
|
16714
|
+
let changed = 0;
|
|
16715
|
+
let removed = 0;
|
|
16716
|
+
for (const entry of current) {
|
|
16717
|
+
const old = before.get(entry.key);
|
|
16718
|
+
if (!old) {
|
|
16719
|
+
added++;
|
|
16720
|
+
lines.push(`+ ${entry.line}`);
|
|
16721
|
+
} else if (old.semantic !== entry.semantic) {
|
|
16722
|
+
changed++;
|
|
16723
|
+
lines.push(`~ ${old.line}
|
|
16724
|
+
${entry.line}`);
|
|
16725
|
+
}
|
|
16726
|
+
}
|
|
16727
|
+
for (const entry of previous) {
|
|
16728
|
+
if (after.has(entry.key)) continue;
|
|
16729
|
+
removed++;
|
|
16730
|
+
lines.push(`- ${entry.line}`);
|
|
16731
|
+
}
|
|
16732
|
+
return { tree: lines.join("\n") || "No accessibility changes.", added, changed, removed };
|
|
16733
|
+
}
|
|
16734
|
+
async function computerBrowserStateCommand(path6, options = {}) {
|
|
16735
|
+
const textLimit = options.limit ? parseCoord(options.limit, "--limit") : 8e3;
|
|
16736
|
+
const elementLimit = options.elementLimit ? parseCoord(options.elementLimit, "--element-limit") : 120;
|
|
16737
|
+
const timeoutMs = options.timeout ? parseCoord(options.timeout, "--timeout") : 5e3;
|
|
16738
|
+
const stableMs = options.stableMs ? parseCoord(options.stableMs, "--stable-ms") : 500;
|
|
16739
|
+
const pollMs = options.pollMs ? parseCoord(options.pollMs, "--poll-ms") : 100;
|
|
16740
|
+
if (textLimit < 0 || textLimit > 5e4) fail("--limit must be between 0 and 50000");
|
|
16741
|
+
if (elementLimit < 0 || elementLimit > 500) fail("--element-limit must be between 0 and 500");
|
|
16742
|
+
if (timeoutMs < 0) fail("--timeout must be >= 0");
|
|
16743
|
+
if (stableMs < 0) fail("--stable-ms must be >= 0");
|
|
16744
|
+
if (pollMs < 50 || pollMs > 2e3) fail("--poll-ms must be between 50 and 2000");
|
|
16745
|
+
const page = await selectChromePage(options);
|
|
16746
|
+
const stability = await waitForBrowserStability(page, { timeoutMs, stableMs, pollMs });
|
|
16747
|
+
const target = resolvePath(path6);
|
|
16748
|
+
mkdirSync3(dirname3(target), { recursive: true });
|
|
16749
|
+
const [{ snapshot, entries }, screenshotResult] = await Promise.all([
|
|
16750
|
+
captureBrowserSnapshot(page, { textLimit, elementLimit }),
|
|
16751
|
+
sendChromeCommand(page.webSocketDebuggerUrl, "Page.captureScreenshot", {
|
|
16752
|
+
format: "png",
|
|
16753
|
+
fromSurface: true,
|
|
16754
|
+
captureBeyondViewport: false
|
|
16755
|
+
})
|
|
16756
|
+
]);
|
|
16757
|
+
const data = screenshotResult.data;
|
|
16758
|
+
if (typeof data !== "string") fail("Chrome did not return screenshot data");
|
|
16759
|
+
writeFileSync3(target, Buffer.from(data, "base64"));
|
|
16760
|
+
const screenshot = readPngDimensions(target);
|
|
16761
|
+
const targetId = page.id ?? "unknown";
|
|
16762
|
+
const cachePath = browserStateCachePath(targetId);
|
|
16763
|
+
let mode = "full";
|
|
16764
|
+
let tree = snapshot.tree;
|
|
16765
|
+
let changes = { added: snapshot.nodeCount, changed: 0, removed: 0 };
|
|
16766
|
+
if (!options.full && existsSync3(cachePath)) {
|
|
16767
|
+
const cached = readBrowserStateCache(cachePath);
|
|
16768
|
+
if (cached?.url === snapshot.url && cached.documentId === snapshot.documentId) {
|
|
16769
|
+
mode = "diff";
|
|
16770
|
+
const diff = browserStateDiff(cached.entries, entries);
|
|
16771
|
+
tree = diff.tree;
|
|
16772
|
+
changes = { added: diff.added, changed: diff.changed, removed: diff.removed };
|
|
16773
|
+
}
|
|
16774
|
+
}
|
|
16775
|
+
mkdirSync3(STATE_DIR, { recursive: true });
|
|
16776
|
+
writeFileSync3(cachePath, JSON.stringify({ url: snapshot.url, documentId: snapshot.documentId, entries }));
|
|
16777
|
+
const stableState = isRecord(stability.state) ? stability.state : {};
|
|
16778
|
+
const state = {
|
|
16779
|
+
title: snapshot.title,
|
|
16780
|
+
url: snapshot.url,
|
|
16781
|
+
documentId: snapshot.documentId,
|
|
16782
|
+
revision: snapshot.revision,
|
|
16783
|
+
viewport: snapshot.viewport,
|
|
16784
|
+
text: snapshot.text,
|
|
16785
|
+
tree,
|
|
16786
|
+
nodeCount: snapshot.nodeCount,
|
|
16787
|
+
elementCount: snapshot.elementCount,
|
|
16788
|
+
truncated: snapshot.truncated,
|
|
16789
|
+
mode,
|
|
16790
|
+
changes
|
|
16791
|
+
};
|
|
16792
|
+
console.log(JSON.stringify({
|
|
16793
|
+
targetId,
|
|
16794
|
+
screenshot: {
|
|
16795
|
+
path: target,
|
|
16796
|
+
width: screenshot.width,
|
|
16797
|
+
height: screenshot.height,
|
|
16798
|
+
devicePixelRatio: typeof stableState.devicePixelRatio === "number" ? stableState.devicePixelRatio : 1
|
|
16799
|
+
},
|
|
16800
|
+
stability,
|
|
16801
|
+
state
|
|
16802
|
+
}, null, 2));
|
|
16803
|
+
}
|
|
16142
16804
|
var DESKTOP_POINT_JS = `
|
|
16143
16805
|
const desktopPoint = (rect) => {
|
|
16144
16806
|
const borderX = Math.max(0, (window.outerWidth - window.innerWidth) / 2);
|
|
@@ -16148,17 +16810,95 @@ var DESKTOP_POINT_JS = `
|
|
|
16148
16810
|
y: Math.round(window.screenY + topChrome + rect.y + rect.height / 2),
|
|
16149
16811
|
};
|
|
16150
16812
|
};`;
|
|
16813
|
+
function parseBrowserRef(value) {
|
|
16814
|
+
const ref = parseCoord(value.replace(/^ref=/, ""), "--ref");
|
|
16815
|
+
if (ref <= 0) fail("--ref must be a positive backend DOM node ID from the latest browser state");
|
|
16816
|
+
return ref;
|
|
16817
|
+
}
|
|
16818
|
+
function readBrowserRefCache(page) {
|
|
16819
|
+
const targetId = page.id;
|
|
16820
|
+
if (!targetId) fail("Chrome target has no stable ID. Capture fresh browser state and retry.");
|
|
16821
|
+
const cachePath = browserStateCachePath(targetId);
|
|
16822
|
+
if (!existsSync3(cachePath)) fail(`No browser state is cached for target ${targetId}. Run replicas computer browser-state <path> --target-id ${targetId} first.`);
|
|
16823
|
+
const cached = readBrowserStateCache(cachePath);
|
|
16824
|
+
if (!cached) fail(`Browser state for target ${targetId} is invalid. Capture fresh browser state and retry.`);
|
|
16825
|
+
if (cached.url !== (page.url ?? "").slice(0, 2e3)) fail("Cached browser state belongs to a different URL. Capture fresh browser state and retry.");
|
|
16826
|
+
return { targetId, cached };
|
|
16827
|
+
}
|
|
16828
|
+
function browserViewportRect(layout) {
|
|
16829
|
+
const viewport = chromeVisualViewport(layout);
|
|
16830
|
+
return {
|
|
16831
|
+
x: 0,
|
|
16832
|
+
y: 0,
|
|
16833
|
+
width: viewport.width,
|
|
16834
|
+
height: viewport.height,
|
|
16835
|
+
centerX: viewport.width / 2,
|
|
16836
|
+
centerY: viewport.height / 2
|
|
16837
|
+
};
|
|
16838
|
+
}
|
|
16839
|
+
async function prepareBrowserElement(page, refValue) {
|
|
16840
|
+
const backendNodeId = parseBrowserRef(refValue);
|
|
16841
|
+
const { cached } = readBrowserRefCache(page);
|
|
16842
|
+
return await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
|
|
16843
|
+
const layout = await sendChromeSessionCommand(session, "Page.getLayoutMetrics", {});
|
|
16844
|
+
const rect = await prepareBrowserRefInSession(session, backendNodeId, cached, browserViewportRect(layout));
|
|
16845
|
+
const axNodes = (await sendChromeSessionCommand(session, "Accessibility.getPartialAXTree", {
|
|
16846
|
+
backendNodeId,
|
|
16847
|
+
fetchRelatives: false
|
|
16848
|
+
})).nodes;
|
|
16849
|
+
const rawNode = Array.isArray(axNodes) && isRawAXNode(axNodes[0]) ? axNodes[0] : null;
|
|
16850
|
+
const states = rawNode ? browserStateProperties(rawNode) : {};
|
|
16851
|
+
const role = clippedText(rawNode?.role?.value, 80);
|
|
16852
|
+
const element = rawNode ? {
|
|
16853
|
+
ref: String(backendNodeId),
|
|
16854
|
+
role,
|
|
16855
|
+
name: clippedText(rawNode.name?.value) || null,
|
|
16856
|
+
description: clippedText(rawNode.description?.value) || null,
|
|
16857
|
+
value: typeof rawNode.value?.value === "string" ? clippedText(rawNode.value.value) : typeof rawNode.value?.value === "number" || typeof rawNode.value?.value === "boolean" ? rawNode.value.value : null,
|
|
16858
|
+
states,
|
|
16859
|
+
actions: [
|
|
16860
|
+
"click",
|
|
16861
|
+
...["textbox", "searchbox", "combobox", "spinbutton", "slider"].includes(role) || states.editable === true ? ["set_value"] : [],
|
|
16862
|
+
"scroll"
|
|
16863
|
+
]
|
|
16864
|
+
} : null;
|
|
16865
|
+
const desktopPoint = await evaluateChromeSession(session, `(() => {
|
|
16866
|
+
${DESKTOP_POINT_JS}
|
|
16867
|
+
return desktopPoint({ x: ${rect.centerX}, y: ${rect.centerY}, width: 0, height: 0 });
|
|
16868
|
+
})()`);
|
|
16869
|
+
return {
|
|
16870
|
+
ref: String(backendNodeId),
|
|
16871
|
+
rect,
|
|
16872
|
+
element,
|
|
16873
|
+
desktopPoint: typeof desktopPoint === "object" && desktopPoint !== null && "x" in desktopPoint && typeof desktopPoint.x === "number" && "y" in desktopPoint && typeof desktopPoint.y === "number" ? { x: desktopPoint.x, y: desktopPoint.y } : null
|
|
16874
|
+
};
|
|
16875
|
+
});
|
|
16876
|
+
}
|
|
16877
|
+
async function prepareBrowserRefInSession(session, backendNodeId, cached, viewportRect) {
|
|
16878
|
+
if (!cached.entries.some(({ line }) => line.trimStart().startsWith(`[ref=${backendNodeId}]`))) {
|
|
16879
|
+
fail(`Browser element ref ${backendNodeId} is not in the latest state. Capture fresh browser state and retry.`);
|
|
16880
|
+
}
|
|
16881
|
+
const frameTree = (await sendChromeSessionCommand(session, "Page.getFrameTree", {})).frameTree;
|
|
16882
|
+
if (cached.documentId !== browserDocumentId(frameTree)) {
|
|
16883
|
+
fail(`Browser element ref ${backendNodeId} belongs to a previous document. Capture fresh browser state and retry.`);
|
|
16884
|
+
}
|
|
16885
|
+
try {
|
|
16886
|
+
await sendChromeSessionCommand(session, "DOM.scrollIntoViewIfNeeded", { backendNodeId });
|
|
16887
|
+
} catch (error) {
|
|
16888
|
+
fail(`Browser element ref ${backendNodeId} is stale or cannot be scrolled into view: ${error instanceof Error ? error.message : String(error)}`);
|
|
16889
|
+
}
|
|
16890
|
+
const quads = (await sendChromeSessionCommand(session, "DOM.getContentQuads", { backendNodeId })).quads;
|
|
16891
|
+
const candidates = browserContentQuadRects(quads, viewportRect);
|
|
16892
|
+
if (!candidates[0]) fail(`Browser element ref ${backendNodeId} has no visible content quad. Fetch fresh browser state and retry.`);
|
|
16893
|
+
return roundBrowserRect(candidates[0]);
|
|
16894
|
+
}
|
|
16151
16895
|
function buildBrowserClickExpression(query, options) {
|
|
16152
16896
|
return `(() => {
|
|
16153
16897
|
const query = ${JSON.stringify(query)};
|
|
16154
16898
|
const exact = ${JSON.stringify(options.exact)};
|
|
16155
16899
|
const index = ${options.index};
|
|
16156
16900
|
const normalize = (value) => String(value || '').replace(/\\s+/g, ' ').trim();
|
|
16157
|
-
|
|
16158
|
-
const rect = el.getBoundingClientRect();
|
|
16159
|
-
const style = getComputedStyle(el);
|
|
16160
|
-
return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
|
|
16161
|
-
};
|
|
16901
|
+
${VISIBLE_JS}
|
|
16162
16902
|
const label = (el) => normalize(el.innerText || el.value || el.getAttribute('aria-label') || el.getAttribute('title') || el.getAttribute('placeholder') || el.href || el.id || el.name || el.tagName);
|
|
16163
16903
|
${DESKTOP_POINT_JS}
|
|
16164
16904
|
const matches = (text) => {
|
|
@@ -16183,8 +16923,7 @@ ${DESKTOP_POINT_JS}
|
|
|
16183
16923
|
const { el, text } = candidate;
|
|
16184
16924
|
el.scrollIntoView({ block: 'center', inline: 'center' });
|
|
16185
16925
|
const rect = el.getBoundingClientRect();
|
|
16186
|
-
if (typeof el.focus === 'function') el.focus();
|
|
16187
|
-
el.click();
|
|
16926
|
+
if (typeof el.focus === 'function') el.focus({ preventScroll: true });
|
|
16188
16927
|
return {
|
|
16189
16928
|
clicked: true,
|
|
16190
16929
|
query,
|
|
@@ -16207,6 +16946,15 @@ ${DESKTOP_POINT_JS}
|
|
|
16207
16946
|
};
|
|
16208
16947
|
})()`;
|
|
16209
16948
|
}
|
|
16949
|
+
function browserActionRectCenter(result) {
|
|
16950
|
+
if (typeof result !== "object" || result === null || !("rect" in result)) return null;
|
|
16951
|
+
const rect = result.rect;
|
|
16952
|
+
if (typeof rect !== "object" || rect === null) return null;
|
|
16953
|
+
const x = "centerX" in rect ? rect.centerX : void 0;
|
|
16954
|
+
const y = "centerY" in rect ? rect.centerY : void 0;
|
|
16955
|
+
if (typeof x !== "number" || typeof y !== "number" || !Number.isFinite(x) || !Number.isFinite(y)) return null;
|
|
16956
|
+
return { x, y };
|
|
16957
|
+
}
|
|
16210
16958
|
function browserActionPoint(result) {
|
|
16211
16959
|
if (typeof result !== "object" || result === null) return null;
|
|
16212
16960
|
const action = "result" in result ? result.result : result;
|
|
@@ -16222,19 +16970,149 @@ function browserActionPoint(result) {
|
|
|
16222
16970
|
y: clamp(Math.round(y), 0, dimensions.height)
|
|
16223
16971
|
};
|
|
16224
16972
|
}
|
|
16973
|
+
async function dispatchChromeClick(page, x, y, button, clicks) {
|
|
16974
|
+
await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
|
|
16975
|
+
await dispatchChromeClickInSession(session, x, y, button, clicks);
|
|
16976
|
+
});
|
|
16977
|
+
}
|
|
16978
|
+
async function dispatchChromeClickInSession(session, x, y, button, clicks) {
|
|
16979
|
+
for (let clickCount = 1; clickCount <= clicks; clickCount++) {
|
|
16980
|
+
await sendChromeSessionCommand(session, "Input.dispatchMouseEvent", {
|
|
16981
|
+
type: "mousePressed",
|
|
16982
|
+
x,
|
|
16983
|
+
y,
|
|
16984
|
+
button,
|
|
16985
|
+
clickCount
|
|
16986
|
+
});
|
|
16987
|
+
await sendChromeSessionCommand(session, "Input.dispatchMouseEvent", {
|
|
16988
|
+
type: "mouseReleased",
|
|
16989
|
+
x,
|
|
16990
|
+
y,
|
|
16991
|
+
button,
|
|
16992
|
+
clickCount
|
|
16993
|
+
});
|
|
16994
|
+
}
|
|
16995
|
+
}
|
|
16996
|
+
async function replaceChromeText(page, text) {
|
|
16997
|
+
await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
|
|
16998
|
+
await replaceChromeTextInSession(session, text);
|
|
16999
|
+
});
|
|
17000
|
+
}
|
|
17001
|
+
async function replaceChromeTextInSession(session, text) {
|
|
17002
|
+
await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
|
|
17003
|
+
type: "rawKeyDown",
|
|
17004
|
+
key: "a",
|
|
17005
|
+
code: "KeyA",
|
|
17006
|
+
windowsVirtualKeyCode: 65,
|
|
17007
|
+
nativeVirtualKeyCode: 65,
|
|
17008
|
+
modifiers: 2
|
|
17009
|
+
});
|
|
17010
|
+
await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
|
|
17011
|
+
type: "keyUp",
|
|
17012
|
+
key: "a",
|
|
17013
|
+
code: "KeyA",
|
|
17014
|
+
windowsVirtualKeyCode: 65,
|
|
17015
|
+
nativeVirtualKeyCode: 65,
|
|
17016
|
+
modifiers: 2
|
|
17017
|
+
});
|
|
17018
|
+
await sendChromeSessionCommand(session, "Input.insertText", { text });
|
|
17019
|
+
}
|
|
17020
|
+
var BROWSER_DIRECT_VALUE_INPUT_TYPES = /* @__PURE__ */ new Set(["number", "range", "date", "datetime-local", "month", "week", "time", "color"]);
|
|
17021
|
+
function browserFillInvalid(actual, expected, fieldType, semanticSelection = false, selectionMatched = false) {
|
|
17022
|
+
if (semanticSelection) return !selectionMatched;
|
|
17023
|
+
if (["number", "range"].includes(fieldType)) {
|
|
17024
|
+
const actualNumber = typeof actual === "number" ? actual : Number(actual);
|
|
17025
|
+
const expectedNumber = Number(expected);
|
|
17026
|
+
return !Number.isFinite(actualNumber) || !Number.isFinite(expectedNumber) || actualNumber !== expectedNumber;
|
|
17027
|
+
}
|
|
17028
|
+
if (fieldType === "color") return String(actual).toLowerCase() !== expected.toLowerCase();
|
|
17029
|
+
return actual !== expected;
|
|
17030
|
+
}
|
|
17031
|
+
async function fillChromeNodeInSession(session, backendNodeId, value) {
|
|
17032
|
+
await sendChromeSessionCommand(session, "DOM.focus", { backendNodeId });
|
|
17033
|
+
const directResult = await callFunctionOnChromeNodeInSession(
|
|
17034
|
+
session,
|
|
17035
|
+
backendNodeId,
|
|
17036
|
+
`function(value) {
|
|
17037
|
+
const emit = () => {
|
|
17038
|
+
this.dispatchEvent(new Event('input', { bubbles: true }));
|
|
17039
|
+
this.dispatchEvent(new Event('change', { bubbles: true }));
|
|
17040
|
+
};
|
|
17041
|
+
if (this instanceof HTMLSelectElement) {
|
|
17042
|
+
const option = Array.from(this.options).find((item) => item.value === value || item.text.trim() === value);
|
|
17043
|
+
if (option) this.value = option.value;
|
|
17044
|
+
else this.value = value;
|
|
17045
|
+
emit();
|
|
17046
|
+
return { handled: true, semanticSelection: true, matched: !!option, value: this.value };
|
|
17047
|
+
}
|
|
17048
|
+
if (this instanceof HTMLInputElement && ${JSON.stringify([...BROWSER_DIRECT_VALUE_INPUT_TYPES])}.includes(this.type.toLowerCase())) {
|
|
17049
|
+
const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')?.set;
|
|
17050
|
+
if (!setter) return { handled: false };
|
|
17051
|
+
setter.call(this, value);
|
|
17052
|
+
emit();
|
|
17053
|
+
return { handled: true, semanticSelection: false, matched: true, value: this.value };
|
|
17054
|
+
}
|
|
17055
|
+
return { handled: false };
|
|
17056
|
+
}`,
|
|
17057
|
+
[value]
|
|
17058
|
+
);
|
|
17059
|
+
const directValue = chromeCallValue(directResult);
|
|
17060
|
+
const directHandled = isRecord(directValue) && directValue.handled === true;
|
|
17061
|
+
if (!directHandled) await replaceChromeTextInSession(session, value);
|
|
17062
|
+
const verification = await callFunctionOnChromeNodeInSession(
|
|
17063
|
+
session,
|
|
17064
|
+
backendNodeId,
|
|
17065
|
+
`function() {
|
|
17066
|
+
const value = this instanceof HTMLSelectElement || 'value' in this ? this.value : this.textContent;
|
|
17067
|
+
const type = this instanceof HTMLSelectElement
|
|
17068
|
+
? 'select'
|
|
17069
|
+
: this instanceof HTMLInputElement
|
|
17070
|
+
? this.type.toLowerCase()
|
|
17071
|
+
: this.isContentEditable
|
|
17072
|
+
? 'contenteditable'
|
|
17073
|
+
: 'text';
|
|
17074
|
+
return { value, valueLength: String(value ?? '').length, type };
|
|
17075
|
+
}`
|
|
17076
|
+
);
|
|
17077
|
+
const actual = chromeCallValue(verification);
|
|
17078
|
+
if (!isRecord(actual)) fail(`Chrome did not return a value for browser element ref ${backendNodeId}.`);
|
|
17079
|
+
const fieldType = typeof actual.type === "string" ? actual.type : "";
|
|
17080
|
+
const semanticSelection = isRecord(directValue) && directValue.semanticSelection === true;
|
|
17081
|
+
const selectionMatched = isRecord(directValue) && directValue.matched === true;
|
|
17082
|
+
if (browserFillInvalid(actual.value, value, fieldType, semanticSelection, selectionMatched)) {
|
|
17083
|
+
fail(`Chrome reported that ref ${backendNodeId} contains ${JSON.stringify(actual.value)} after filling, expected ${JSON.stringify(value)}.`);
|
|
17084
|
+
}
|
|
17085
|
+
return { value: actual.value, valueLength: actual.valueLength, type: fieldType };
|
|
17086
|
+
}
|
|
16225
17087
|
async function computerBrowserClickCommand(query, options = {}) {
|
|
16226
17088
|
const page = await selectChromePage(options);
|
|
17089
|
+
if (options.ref) {
|
|
17090
|
+
const target = await prepareBrowserElement(page, options.ref);
|
|
17091
|
+
const buttonNames = { "1": "left", "2": "middle", "3": "right", left: "left", middle: "middle", right: "right" };
|
|
17092
|
+
const button = buttonNames[(options.button ?? "left").toLowerCase()];
|
|
17093
|
+
if (!button) fail("--button must be one of left|middle|right|1|2|3");
|
|
17094
|
+
const clicks = options.double ? 2 : 1;
|
|
17095
|
+
await dispatchChromeClick(page, target.rect.centerX, target.rect.centerY, button, clicks);
|
|
17096
|
+
if (target.desktopPoint) logRecordingAction({ type: "click", ...target.desktopPoint });
|
|
17097
|
+
console.log(JSON.stringify({
|
|
17098
|
+
...browserTargetOutput(page),
|
|
17099
|
+
result: { clicked: true, ref: target.ref, button, clickCount: clicks, rect: target.rect, element: target.element }
|
|
17100
|
+
}, null, 2));
|
|
17101
|
+
return;
|
|
17102
|
+
}
|
|
17103
|
+
if (!query) fail("Provide control text or --ref <id> from the latest browser state");
|
|
16227
17104
|
const index = options.index ? parseCoord(options.index, "--index") : 0;
|
|
16228
17105
|
if (index < 0) fail("--index must be >= 0");
|
|
16229
17106
|
const result = await evaluateChromeTarget(
|
|
16230
17107
|
page.webSocketDebuggerUrl,
|
|
16231
17108
|
buildBrowserClickExpression(query, { exact: !!options.exact, index })
|
|
16232
17109
|
);
|
|
17110
|
+
const center = browserActionRectCenter(result);
|
|
17111
|
+
if (center) await dispatchChromeClick(page, center.x, center.y, "left", 1);
|
|
16233
17112
|
const point = browserActionPoint(result);
|
|
16234
17113
|
if (point) logRecordingAction({ type: "click", ...point });
|
|
16235
17114
|
console.log(JSON.stringify({
|
|
16236
|
-
|
|
16237
|
-
url: page.url ?? "",
|
|
17115
|
+
...browserTargetOutput(page),
|
|
16238
17116
|
result
|
|
16239
17117
|
}, null, 2));
|
|
16240
17118
|
}
|
|
@@ -16246,11 +17124,7 @@ function buildBrowserFillExpression(query, value, options) {
|
|
|
16246
17124
|
const index = ${options.index};
|
|
16247
17125
|
const normalize = (text) => String(text || '').replace(/\\s+/g, ' ').trim();
|
|
16248
17126
|
const escape = (text) => globalThis.CSS && CSS.escape ? CSS.escape(text) : String(text).replace(/["\\\\]/g, '\\\\$&');
|
|
16249
|
-
|
|
16250
|
-
const rect = el.getBoundingClientRect();
|
|
16251
|
-
const style = getComputedStyle(el);
|
|
16252
|
-
return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
|
|
16253
|
-
};
|
|
17127
|
+
${VISIBLE_JS}
|
|
16254
17128
|
const matches = (text) => {
|
|
16255
17129
|
const haystack = text.toLowerCase();
|
|
16256
17130
|
const needle = query.toLowerCase();
|
|
@@ -16295,18 +17169,42 @@ ${DESKTOP_POINT_JS}
|
|
|
16295
17169
|
}
|
|
16296
17170
|
const { el, text } = candidate;
|
|
16297
17171
|
el.scrollIntoView({ block: 'center', inline: 'center' });
|
|
16298
|
-
if (typeof el.focus === 'function') el.focus();
|
|
17172
|
+
if (typeof el.focus === 'function') el.focus({ preventScroll: true });
|
|
17173
|
+
const fieldType = el.tagName.toLowerCase() === 'select' ? 'select' : String(el.getAttribute('type') || 'text').toLowerCase();
|
|
17174
|
+
let nativeInputRequired = false;
|
|
17175
|
+
let semanticSelection = false;
|
|
17176
|
+
let selectionMatched = false;
|
|
16299
17177
|
if (el.tagName.toLowerCase() === 'select') {
|
|
16300
17178
|
const option = Array.from(el.options).find((item) => item.value === value || normalize(item.text) === value);
|
|
16301
17179
|
if (option) el.value = option.value;
|
|
16302
17180
|
else el.value = value;
|
|
17181
|
+
semanticSelection = true;
|
|
17182
|
+
selectionMatched = !!option;
|
|
17183
|
+
el.dispatchEvent(new Event('input', { bubbles: true }));
|
|
17184
|
+
el.dispatchEvent(new Event('change', { bubbles: true }));
|
|
17185
|
+
} else if (el instanceof HTMLInputElement && ${JSON.stringify([...BROWSER_DIRECT_VALUE_INPUT_TYPES])}.includes(fieldType)) {
|
|
17186
|
+
const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')?.set;
|
|
17187
|
+
if (setter) {
|
|
17188
|
+
setter.call(el, value);
|
|
17189
|
+
el.dispatchEvent(new Event('input', { bubbles: true }));
|
|
17190
|
+
el.dispatchEvent(new Event('change', { bubbles: true }));
|
|
17191
|
+
} else {
|
|
17192
|
+
nativeInputRequired = true;
|
|
17193
|
+
}
|
|
16303
17194
|
} else if (el.isContentEditable) {
|
|
16304
|
-
|
|
17195
|
+
const selection = getSelection();
|
|
17196
|
+
const range = document.createRange();
|
|
17197
|
+
range.selectNodeContents(el);
|
|
17198
|
+
selection.removeAllRanges();
|
|
17199
|
+
selection.addRange(range);
|
|
17200
|
+
nativeInputRequired = true;
|
|
17201
|
+
} else if (typeof el.select === 'function') {
|
|
17202
|
+
el.select();
|
|
17203
|
+
nativeInputRequired = true;
|
|
16305
17204
|
} else {
|
|
16306
|
-
el.value
|
|
17205
|
+
el.setSelectionRange(0, String(el.value || '').length);
|
|
17206
|
+
nativeInputRequired = true;
|
|
16307
17207
|
}
|
|
16308
|
-
el.dispatchEvent(new InputEvent('input', { bubbles: true, inputType: 'insertText', data: value }));
|
|
16309
|
-
el.dispatchEvent(new Event('change', { bubbles: true }));
|
|
16310
17208
|
const rect = el.getBoundingClientRect();
|
|
16311
17209
|
return {
|
|
16312
17210
|
filled: true,
|
|
@@ -16315,9 +17213,13 @@ ${DESKTOP_POINT_JS}
|
|
|
16315
17213
|
index,
|
|
16316
17214
|
tag: el.tagName.toLowerCase(),
|
|
16317
17215
|
role: el.getAttribute('role') || null,
|
|
16318
|
-
type:
|
|
17216
|
+
type: fieldType,
|
|
16319
17217
|
text,
|
|
16320
|
-
|
|
17218
|
+
actualValue: el.isContentEditable ? el.textContent : 'value' in el ? el.value : null,
|
|
17219
|
+
valueLength: String(el.isContentEditable ? el.textContent : 'value' in el ? el.value : '').length,
|
|
17220
|
+
nativeInputRequired,
|
|
17221
|
+
semanticSelection,
|
|
17222
|
+
selectionMatched,
|
|
16321
17223
|
desktopPoint: desktopPoint(rect),
|
|
16322
17224
|
rect: {
|
|
16323
17225
|
x: Math.round(rect.x),
|
|
@@ -16332,6 +17234,18 @@ ${DESKTOP_POINT_JS}
|
|
|
16332
17234
|
}
|
|
16333
17235
|
async function computerBrowserFillCommand(query, value, options = {}) {
|
|
16334
17236
|
const page = await selectChromePage(options);
|
|
17237
|
+
if (options.ref) {
|
|
17238
|
+
const target = await prepareBrowserElement(page, options.ref);
|
|
17239
|
+
const backendNodeId = parseBrowserRef(options.ref);
|
|
17240
|
+
const actual = await withChromeSession(page.webSocketDebuggerUrl, async (session) => await fillChromeNodeInSession(session, backendNodeId, value));
|
|
17241
|
+
if (target.desktopPoint) logRecordingAction({ type: "type", ...target.desktopPoint });
|
|
17242
|
+
console.log(JSON.stringify({
|
|
17243
|
+
...browserTargetOutput(page),
|
|
17244
|
+
result: { filled: true, ref: target.ref, valueLength: actual.valueLength, rect: target.rect, element: target.element }
|
|
17245
|
+
}, null, 2));
|
|
17246
|
+
return;
|
|
17247
|
+
}
|
|
17248
|
+
if (!query) fail("Provide field text or --ref <id> from the latest browser state");
|
|
16335
17249
|
const index = options.index ? parseCoord(options.index, "--index") : 0;
|
|
16336
17250
|
if (index < 0) fail("--index must be >= 0");
|
|
16337
17251
|
const result = await evaluateChromeTarget(
|
|
@@ -16339,10 +17253,28 @@ async function computerBrowserFillCommand(query, value, options = {}) {
|
|
|
16339
17253
|
buildBrowserFillExpression(query, value, { exact: !!options.exact, index })
|
|
16340
17254
|
);
|
|
16341
17255
|
const point = browserActionPoint(result);
|
|
17256
|
+
if (typeof result === "object" && result !== null && "nativeInputRequired" in result && result.nativeInputRequired === true) {
|
|
17257
|
+
await replaceChromeText(page, value);
|
|
17258
|
+
const actualValue = await evaluateChromeTarget(page.webSocketDebuggerUrl, `(() => {
|
|
17259
|
+
const el = document.activeElement;
|
|
17260
|
+
return el && el.isContentEditable ? el.textContent : el && 'value' in el ? el.value : null;
|
|
17261
|
+
})()`);
|
|
17262
|
+
const fieldType = "type" in result && typeof result.type === "string" ? result.type.toLowerCase() : "";
|
|
17263
|
+
if (browserFillInvalid(actualValue, value, fieldType)) {
|
|
17264
|
+
fail(`Chrome reported that ${JSON.stringify(query)} contains ${JSON.stringify(actualValue)} after filling, expected ${JSON.stringify(value)}.`);
|
|
17265
|
+
}
|
|
17266
|
+
if (isRecord(result)) {
|
|
17267
|
+
Object.assign(result, { actualValue, valueLength: String(actualValue ?? "").length });
|
|
17268
|
+
}
|
|
17269
|
+
} else if (isRecord(result) && result.filled === true) {
|
|
17270
|
+
const fieldType = typeof result.type === "string" ? result.type : "";
|
|
17271
|
+
if (browserFillInvalid(result.actualValue, value, fieldType, result.semanticSelection === true, result.selectionMatched === true)) {
|
|
17272
|
+
fail(`Chrome reported that ${JSON.stringify(query)} contains ${JSON.stringify(result.actualValue)} after filling, expected ${JSON.stringify(value)}.`);
|
|
17273
|
+
}
|
|
17274
|
+
}
|
|
16342
17275
|
if (point) logRecordingAction({ type: "type", ...point });
|
|
16343
17276
|
console.log(JSON.stringify({
|
|
16344
|
-
|
|
16345
|
-
url: page.url ?? "",
|
|
17277
|
+
...browserTargetOutput(page),
|
|
16346
17278
|
result
|
|
16347
17279
|
}, null, 2));
|
|
16348
17280
|
}
|
|
@@ -16352,11 +17284,7 @@ function buildBrowserWaitExpression(query, options) {
|
|
|
16352
17284
|
const mode = ${JSON.stringify(options.mode)};
|
|
16353
17285
|
const exact = ${JSON.stringify(options.exact)};
|
|
16354
17286
|
const normalize = (text) => String(text || '').replace(/\\s+/g, ' ').trim();
|
|
16355
|
-
|
|
16356
|
-
const rect = el.getBoundingClientRect();
|
|
16357
|
-
const style = getComputedStyle(el);
|
|
16358
|
-
return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none';
|
|
16359
|
-
};
|
|
17287
|
+
${VISIBLE_JS}
|
|
16360
17288
|
const matches = (text) => {
|
|
16361
17289
|
const haystack = normalize(text).toLowerCase();
|
|
16362
17290
|
const needle = query.toLowerCase();
|
|
@@ -16411,6 +17339,241 @@ async function computerBrowserWaitCommand(query, options = {}) {
|
|
|
16411
17339
|
console.log(JSON.stringify({ ok: false, elapsedMs: Date.now() - start, attempts, result: lastResult }, null, 2));
|
|
16412
17340
|
process.exitCode = 1;
|
|
16413
17341
|
}
|
|
17342
|
+
async function computerBrowserScrollCommand(direction, options = {}) {
|
|
17343
|
+
const page = await selectChromePage(options);
|
|
17344
|
+
const normalized = direction.toLowerCase();
|
|
17345
|
+
if (!["up", "down", "left", "right"].includes(normalized)) fail("direction must be one of up|down|left|right");
|
|
17346
|
+
const amount = options.amount ? parseCoord(options.amount, "--amount") : 600;
|
|
17347
|
+
if (amount <= 0 || amount > 1e4) fail("--amount must be between 1 and 10000 CSS pixels");
|
|
17348
|
+
const layout = await sendChromeCommand(page.webSocketDebuggerUrl, "Page.getLayoutMetrics", {});
|
|
17349
|
+
const viewport = chromeVisualViewport(layout);
|
|
17350
|
+
const target = options.ref ? await prepareBrowserElement(page, options.ref) : null;
|
|
17351
|
+
const x = target?.rect.centerX ?? viewport.width / 2;
|
|
17352
|
+
const y = target?.rect.centerY ?? viewport.height / 2;
|
|
17353
|
+
const deltaX = normalized === "left" ? -amount : normalized === "right" ? amount : 0;
|
|
17354
|
+
const deltaY = normalized === "up" ? -amount : normalized === "down" ? amount : 0;
|
|
17355
|
+
await sendChromeCommand(page.webSocketDebuggerUrl, "Input.dispatchMouseEvent", {
|
|
17356
|
+
type: "mouseWheel",
|
|
17357
|
+
x,
|
|
17358
|
+
y,
|
|
17359
|
+
deltaX,
|
|
17360
|
+
deltaY
|
|
17361
|
+
});
|
|
17362
|
+
logRecordingAction({ type: "scroll", ...target?.desktopPoint ?? {} });
|
|
17363
|
+
console.log(JSON.stringify({
|
|
17364
|
+
...browserTargetOutput(page),
|
|
17365
|
+
result: { scrolled: true, direction: normalized, amount, ref: target?.ref ?? null, x: Math.round(x), y: Math.round(y) }
|
|
17366
|
+
}, null, 2));
|
|
17367
|
+
}
|
|
17368
|
+
function browserKeyDetails(combo) {
|
|
17369
|
+
const tokens = combo.split("+").map((token) => token.trim()).filter(Boolean);
|
|
17370
|
+
if (tokens.length === 0) fail("key combination cannot be empty");
|
|
17371
|
+
let modifiers = 0;
|
|
17372
|
+
for (const token of tokens.slice(0, -1)) {
|
|
17373
|
+
const normalized2 = token.toLowerCase();
|
|
17374
|
+
if (["alt", "option"].includes(normalized2)) modifiers |= 1;
|
|
17375
|
+
else if (["ctrl", "control"].includes(normalized2)) modifiers |= 2;
|
|
17376
|
+
else if (["meta", "cmd", "command", "super"].includes(normalized2)) modifiers |= 4;
|
|
17377
|
+
else if (normalized2 === "shift") modifiers |= 8;
|
|
17378
|
+
else fail(`Unsupported browser key modifier ${JSON.stringify(token)}`);
|
|
17379
|
+
}
|
|
17380
|
+
const requested = tokens[tokens.length - 1];
|
|
17381
|
+
const aliases = {
|
|
17382
|
+
enter: { key: "Enter", code: "Enter", windowsVirtualKeyCode: 13 },
|
|
17383
|
+
return: { key: "Enter", code: "Enter", windowsVirtualKeyCode: 13 },
|
|
17384
|
+
tab: { key: "Tab", code: "Tab", windowsVirtualKeyCode: 9 },
|
|
17385
|
+
escape: { key: "Escape", code: "Escape", windowsVirtualKeyCode: 27 },
|
|
17386
|
+
esc: { key: "Escape", code: "Escape", windowsVirtualKeyCode: 27 },
|
|
17387
|
+
backspace: { key: "Backspace", code: "Backspace", windowsVirtualKeyCode: 8 },
|
|
17388
|
+
delete: { key: "Delete", code: "Delete", windowsVirtualKeyCode: 46 },
|
|
17389
|
+
space: { key: " ", code: "Space", windowsVirtualKeyCode: 32 },
|
|
17390
|
+
up: { key: "ArrowUp", code: "ArrowUp", windowsVirtualKeyCode: 38 },
|
|
17391
|
+
arrowup: { key: "ArrowUp", code: "ArrowUp", windowsVirtualKeyCode: 38 },
|
|
17392
|
+
down: { key: "ArrowDown", code: "ArrowDown", windowsVirtualKeyCode: 40 },
|
|
17393
|
+
arrowdown: { key: "ArrowDown", code: "ArrowDown", windowsVirtualKeyCode: 40 },
|
|
17394
|
+
left: { key: "ArrowLeft", code: "ArrowLeft", windowsVirtualKeyCode: 37 },
|
|
17395
|
+
arrowleft: { key: "ArrowLeft", code: "ArrowLeft", windowsVirtualKeyCode: 37 },
|
|
17396
|
+
right: { key: "ArrowRight", code: "ArrowRight", windowsVirtualKeyCode: 39 },
|
|
17397
|
+
arrowright: { key: "ArrowRight", code: "ArrowRight", windowsVirtualKeyCode: 39 },
|
|
17398
|
+
home: { key: "Home", code: "Home", windowsVirtualKeyCode: 36 },
|
|
17399
|
+
end: { key: "End", code: "End", windowsVirtualKeyCode: 35 },
|
|
17400
|
+
pageup: { key: "PageUp", code: "PageUp", windowsVirtualKeyCode: 33 },
|
|
17401
|
+
pagedown: { key: "PageDown", code: "PageDown", windowsVirtualKeyCode: 34 }
|
|
17402
|
+
};
|
|
17403
|
+
const normalized = requested.toLowerCase();
|
|
17404
|
+
const alias = aliases[normalized];
|
|
17405
|
+
if (alias) return { ...alias, modifiers, ...alias.key === " " && modifiers === 0 ? { text: " " } : {} };
|
|
17406
|
+
if (/^[a-z]$/i.test(requested)) {
|
|
17407
|
+
const upper = requested.toUpperCase();
|
|
17408
|
+
const key = modifiers & 8 ? upper : requested.toLowerCase();
|
|
17409
|
+
return { key, code: `Key${upper}`, windowsVirtualKeyCode: upper.charCodeAt(0), modifiers, ...modifiers === 0 ? { text: key } : {} };
|
|
17410
|
+
}
|
|
17411
|
+
if (/^[0-9]$/.test(requested)) {
|
|
17412
|
+
return { key: requested, code: `Digit${requested}`, windowsVirtualKeyCode: requested.charCodeAt(0), modifiers, ...modifiers === 0 ? { text: requested } : {} };
|
|
17413
|
+
}
|
|
17414
|
+
fail(`Unsupported browser key ${JSON.stringify(requested)}`);
|
|
17415
|
+
}
|
|
17416
|
+
async function computerBrowserKeyCommand(combo, options = {}) {
|
|
17417
|
+
const page = await selectChromePage(options);
|
|
17418
|
+
const key = browserKeyDetails(combo);
|
|
17419
|
+
await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
|
|
17420
|
+
await dispatchChromeKeyInSession(session, key);
|
|
17421
|
+
});
|
|
17422
|
+
logRecordingAction({ type: "key" });
|
|
17423
|
+
console.log(JSON.stringify({ targetId: page.id ?? null, result: { pressed: combo } }, null, 2));
|
|
17424
|
+
}
|
|
17425
|
+
async function dispatchChromeKeyInSession(session, key) {
|
|
17426
|
+
await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
|
|
17427
|
+
type: "keyDown",
|
|
17428
|
+
...key,
|
|
17429
|
+
nativeVirtualKeyCode: key.windowsVirtualKeyCode
|
|
17430
|
+
});
|
|
17431
|
+
await sendChromeSessionCommand(session, "Input.dispatchKeyEvent", {
|
|
17432
|
+
type: "keyUp",
|
|
17433
|
+
key: key.key,
|
|
17434
|
+
code: key.code,
|
|
17435
|
+
modifiers: key.modifiers,
|
|
17436
|
+
windowsVirtualKeyCode: key.windowsVirtualKeyCode,
|
|
17437
|
+
nativeVirtualKeyCode: key.windowsVirtualKeyCode
|
|
17438
|
+
});
|
|
17439
|
+
}
|
|
17440
|
+
async function computerBrowserTypeCommand(text, options = {}) {
|
|
17441
|
+
const page = await selectChromePage(options);
|
|
17442
|
+
await sendChromeCommand(page.webSocketDebuggerUrl, "Input.insertText", { text });
|
|
17443
|
+
logRecordingAction({ type: "type" });
|
|
17444
|
+
console.log(JSON.stringify({ targetId: page.id ?? null, result: { typed: true, length: text.length } }, null, 2));
|
|
17445
|
+
}
|
|
17446
|
+
function parseBrowserBatchActions(value) {
|
|
17447
|
+
let input;
|
|
17448
|
+
try {
|
|
17449
|
+
input = JSON.parse(value);
|
|
17450
|
+
} catch (error) {
|
|
17451
|
+
fail(`Batch actions must be valid JSON: ${error instanceof Error ? error.message : String(error)}`);
|
|
17452
|
+
}
|
|
17453
|
+
if (!Array.isArray(input) || input.length === 0 || input.length > 100) {
|
|
17454
|
+
fail("Batch actions must be a JSON array containing 1 to 100 actions");
|
|
17455
|
+
}
|
|
17456
|
+
return input.map((item, index) => {
|
|
17457
|
+
if (!isRecord(item) || typeof item.action !== "string") fail(`Batch action ${index} must be an object with an action`);
|
|
17458
|
+
if (item.action === "click") {
|
|
17459
|
+
if (typeof item.ref !== "string" && typeof item.ref !== "number") fail(`Batch click ${index} requires ref`);
|
|
17460
|
+
const buttons = { "1": "left", "2": "middle", "3": "right", left: "left", middle: "middle", right: "right" };
|
|
17461
|
+
const requestedButton = item.button === void 0 ? "left" : String(item.button).toLowerCase();
|
|
17462
|
+
const button = buttons[requestedButton];
|
|
17463
|
+
if (!button) fail(`Batch click ${index} button must be left, middle, or right`);
|
|
17464
|
+
return { action: "click", ref: String(item.ref), button, clicks: item.double === true ? 2 : 1 };
|
|
17465
|
+
}
|
|
17466
|
+
if (item.action === "fill") {
|
|
17467
|
+
if (typeof item.ref !== "string" && typeof item.ref !== "number" || typeof item.value !== "string") {
|
|
17468
|
+
fail(`Batch fill ${index} requires ref and string value`);
|
|
17469
|
+
}
|
|
17470
|
+
return { action: "fill", ref: String(item.ref), value: item.value };
|
|
17471
|
+
}
|
|
17472
|
+
if (item.action === "key") {
|
|
17473
|
+
if (typeof item.combo !== "string") fail(`Batch key ${index} requires combo`);
|
|
17474
|
+
return { action: "key", combo: item.combo };
|
|
17475
|
+
}
|
|
17476
|
+
if (item.action === "type") {
|
|
17477
|
+
if (typeof item.text !== "string") fail(`Batch type ${index} requires text`);
|
|
17478
|
+
return { action: "type", text: item.text };
|
|
17479
|
+
}
|
|
17480
|
+
if (item.action === "scroll") {
|
|
17481
|
+
const directions = { up: "up", down: "down", left: "left", right: "right" };
|
|
17482
|
+
const direction = typeof item.direction === "string" ? directions[item.direction] : void 0;
|
|
17483
|
+
if (!direction) {
|
|
17484
|
+
fail(`Batch scroll ${index} direction must be up, down, left, or right`);
|
|
17485
|
+
}
|
|
17486
|
+
const amount = item.amount === void 0 ? 600 : Number(item.amount);
|
|
17487
|
+
if (!Number.isFinite(amount) || amount <= 0 || amount > 1e4) fail(`Batch scroll ${index} amount must be between 1 and 10000`);
|
|
17488
|
+
if (item.ref !== void 0 && typeof item.ref !== "string" && typeof item.ref !== "number") fail(`Batch scroll ${index} ref is invalid`);
|
|
17489
|
+
return {
|
|
17490
|
+
action: "scroll",
|
|
17491
|
+
direction,
|
|
17492
|
+
amount,
|
|
17493
|
+
...item.ref === void 0 ? {} : { ref: String(item.ref) }
|
|
17494
|
+
};
|
|
17495
|
+
}
|
|
17496
|
+
if (item.action === "wait") {
|
|
17497
|
+
if (typeof item.text !== "string") fail(`Batch wait ${index} requires text`);
|
|
17498
|
+
const mode = item.mode === void 0 ? "any" : String(item.mode);
|
|
17499
|
+
if (!["any", "text", "title", "url", "control"].includes(mode)) fail(`Batch wait ${index} mode is invalid`);
|
|
17500
|
+
const timeoutMs = item.timeoutMs === void 0 ? 1e4 : Number(item.timeoutMs);
|
|
17501
|
+
const pollMs = item.pollMs === void 0 ? 250 : Number(item.pollMs);
|
|
17502
|
+
if (!Number.isFinite(timeoutMs) || timeoutMs < 0) fail(`Batch wait ${index} timeoutMs must be >= 0`);
|
|
17503
|
+
if (!Number.isFinite(pollMs) || pollMs < 50 || pollMs > 2e3) fail(`Batch wait ${index} pollMs must be between 50 and 2000`);
|
|
17504
|
+
return { action: "wait", text: item.text, mode, exact: item.exact === true, timeoutMs, pollMs };
|
|
17505
|
+
}
|
|
17506
|
+
fail(`Batch action ${index} has unsupported action ${JSON.stringify(item.action)}`);
|
|
17507
|
+
});
|
|
17508
|
+
}
|
|
17509
|
+
async function computerBrowserBatchCommand(actionsJson, options = {}) {
|
|
17510
|
+
const actions = parseBrowserBatchActions(actionsJson);
|
|
17511
|
+
const page = await selectChromePage(options);
|
|
17512
|
+
const { targetId, cached } = readBrowserRefCache(page);
|
|
17513
|
+
const startedAt = Date.now();
|
|
17514
|
+
const completed = [];
|
|
17515
|
+
try {
|
|
17516
|
+
await withChromeSession(page.webSocketDebuggerUrl, async (session) => {
|
|
17517
|
+
const layout = await sendChromeSessionCommand(session, "Page.getLayoutMetrics", {});
|
|
17518
|
+
const viewportRect = browserViewportRect(layout);
|
|
17519
|
+
for (const [index, action] of actions.entries()) {
|
|
17520
|
+
const actionStartedAt = Date.now();
|
|
17521
|
+
if (action.action === "click") {
|
|
17522
|
+
const ref = parseBrowserRef(action.ref);
|
|
17523
|
+
const rect = await prepareBrowserRefInSession(session, ref, cached, viewportRect);
|
|
17524
|
+
await dispatchChromeClickInSession(session, rect.centerX, rect.centerY, action.button, action.clicks);
|
|
17525
|
+
logRecordingAction({ type: "click" });
|
|
17526
|
+
completed.push({ index, action: action.action, ref: String(ref), elapsedMs: Date.now() - actionStartedAt });
|
|
17527
|
+
} else if (action.action === "fill") {
|
|
17528
|
+
const ref = parseBrowserRef(action.ref);
|
|
17529
|
+
await prepareBrowserRefInSession(session, ref, cached, viewportRect);
|
|
17530
|
+
const result = await fillChromeNodeInSession(session, ref, action.value);
|
|
17531
|
+
logRecordingAction({ type: "type" });
|
|
17532
|
+
completed.push({ index, action: action.action, ref: String(ref), valueLength: result.valueLength, elapsedMs: Date.now() - actionStartedAt });
|
|
17533
|
+
} else if (action.action === "key") {
|
|
17534
|
+
await dispatchChromeKeyInSession(session, browserKeyDetails(action.combo));
|
|
17535
|
+
logRecordingAction({ type: "key" });
|
|
17536
|
+
completed.push({ index, action: action.action, elapsedMs: Date.now() - actionStartedAt });
|
|
17537
|
+
} else if (action.action === "type") {
|
|
17538
|
+
await sendChromeSessionCommand(session, "Input.insertText", { text: action.text });
|
|
17539
|
+
logRecordingAction({ type: "type" });
|
|
17540
|
+
completed.push({ index, action: action.action, length: action.text.length, elapsedMs: Date.now() - actionStartedAt });
|
|
17541
|
+
} else if (action.action === "scroll") {
|
|
17542
|
+
const ref = action.ref ? parseBrowserRef(action.ref) : null;
|
|
17543
|
+
const rect = ref ? await prepareBrowserRefInSession(session, ref, cached, viewportRect) : viewportRect;
|
|
17544
|
+
const deltaX = action.direction === "left" ? -action.amount : action.direction === "right" ? action.amount : 0;
|
|
17545
|
+
const deltaY = action.direction === "up" ? -action.amount : action.direction === "down" ? action.amount : 0;
|
|
17546
|
+
await sendChromeSessionCommand(session, "Input.dispatchMouseEvent", {
|
|
17547
|
+
type: "mouseWheel",
|
|
17548
|
+
x: rect.centerX,
|
|
17549
|
+
y: rect.centerY,
|
|
17550
|
+
deltaX,
|
|
17551
|
+
deltaY
|
|
17552
|
+
});
|
|
17553
|
+
logRecordingAction({ type: "scroll" });
|
|
17554
|
+
completed.push({ index, action: action.action, direction: action.direction, amount: action.amount, elapsedMs: Date.now() - actionStartedAt });
|
|
17555
|
+
} else {
|
|
17556
|
+
const expression = buildBrowserWaitExpression(action.text, { mode: action.mode, exact: action.exact });
|
|
17557
|
+
let attempts = 0;
|
|
17558
|
+
let result = null;
|
|
17559
|
+
while (Date.now() - actionStartedAt <= action.timeoutMs) {
|
|
17560
|
+
attempts++;
|
|
17561
|
+
result = await evaluateChromeSession(session, expression);
|
|
17562
|
+
if (isRecord(result) && result.matched === true) break;
|
|
17563
|
+
await sleep(action.pollMs);
|
|
17564
|
+
}
|
|
17565
|
+
if (!isRecord(result) || result.matched !== true) fail(`Batch wait ${index} timed out after ${Date.now() - actionStartedAt}ms`);
|
|
17566
|
+
completed.push({ index, action: action.action, attempts, elapsedMs: Date.now() - actionStartedAt });
|
|
17567
|
+
}
|
|
17568
|
+
}
|
|
17569
|
+
});
|
|
17570
|
+
} catch (error) {
|
|
17571
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
17572
|
+
console.log(JSON.stringify({ ok: false, targetId, elapsedMs: Date.now() - startedAt, completed, failedIndex: completed.length, error: message }, null, 2));
|
|
17573
|
+
throw error;
|
|
17574
|
+
}
|
|
17575
|
+
console.log(JSON.stringify({ ok: true, targetId, elapsedMs: Date.now() - startedAt, completed }, null, 2));
|
|
17576
|
+
}
|
|
16414
17577
|
async function computerClickCommand(xStr, yStr, options) {
|
|
16415
17578
|
const dimensions = getDisplayDimensions();
|
|
16416
17579
|
const x = parseScreenCoord(xStr, "x", dimensions.width);
|
|
@@ -16533,12 +17696,55 @@ async function computerLaunchCommand(app, args) {
|
|
|
16533
17696
|
const baseArgs = app === "chrome" && !existsSync3(CHROME_WRAPPER) ? LEGACY_CHROME_ALIAS : APP_ALIASES[app] ?? [app];
|
|
16534
17697
|
const bin = baseArgs[0];
|
|
16535
17698
|
const fullArgs = [...baseArgs.slice(1), ...args];
|
|
17699
|
+
const expectsNewChromePage = args.some((arg) => !arg.startsWith("-"));
|
|
17700
|
+
const existingChromePageIds = /* @__PURE__ */ new Set();
|
|
17701
|
+
if (app === "chrome") {
|
|
17702
|
+
try {
|
|
17703
|
+
for (const page of await getChromePages()) {
|
|
17704
|
+
if (page.id) existingChromePageIds.add(page.id);
|
|
17705
|
+
}
|
|
17706
|
+
} catch {
|
|
17707
|
+
}
|
|
17708
|
+
}
|
|
16536
17709
|
const child = spawn4(bin, fullArgs, {
|
|
16537
17710
|
env: withDisplay(),
|
|
16538
17711
|
detached: true,
|
|
16539
17712
|
stdio: "ignore"
|
|
16540
17713
|
});
|
|
16541
17714
|
child.unref();
|
|
17715
|
+
if (app === "chrome") {
|
|
17716
|
+
const startedAt = Date.now();
|
|
17717
|
+
let pageCount = 0;
|
|
17718
|
+
let readyPage = null;
|
|
17719
|
+
while (Date.now() - startedAt < 15e3) {
|
|
17720
|
+
let pages = [];
|
|
17721
|
+
try {
|
|
17722
|
+
pages = await getChromePages();
|
|
17723
|
+
} catch {
|
|
17724
|
+
}
|
|
17725
|
+
pageCount = pages.length;
|
|
17726
|
+
const candidates = expectsNewChromePage && existingChromePageIds.size > 0 ? pages.filter((page) => page.id && !existingChromePageIds.has(page.id)) : pages;
|
|
17727
|
+
for (const page of candidates) {
|
|
17728
|
+
if (!page.webSocketDebuggerUrl || !page.url || page.url === "about:blank") continue;
|
|
17729
|
+
const readyState = await evaluateChromeTarget(page.webSocketDebuggerUrl, "document.readyState");
|
|
17730
|
+
if (readyState === "interactive" || readyState === "complete") {
|
|
17731
|
+
readyPage = page;
|
|
17732
|
+
break;
|
|
17733
|
+
}
|
|
17734
|
+
}
|
|
17735
|
+
if (readyPage) break;
|
|
17736
|
+
try {
|
|
17737
|
+
if (child.pid) process.kill(child.pid, 0);
|
|
17738
|
+
} catch {
|
|
17739
|
+
fail(`Chrome exited before its desktop control channel became ready.`);
|
|
17740
|
+
}
|
|
17741
|
+
await sleep(100);
|
|
17742
|
+
}
|
|
17743
|
+
if (!readyPage?.webSocketDebuggerUrl) fail(`Chrome launched but its requested page was not controllable after 15 seconds.`);
|
|
17744
|
+
await sendChromeCommand(readyPage.webSocketDebuggerUrl, "Page.bringToFront", {});
|
|
17745
|
+
console.log(`launched ${bin} (pid ${child.pid}, ready in ${Date.now() - startedAt}ms, page ${readyPage.id}, ${pageCount} open)`);
|
|
17746
|
+
return;
|
|
17747
|
+
}
|
|
16542
17748
|
console.log(`launched ${bin} (pid ${child.pid})`);
|
|
16543
17749
|
}
|
|
16544
17750
|
|
|
@@ -20965,10 +22171,20 @@ if (isAgentMode()) {
|
|
|
20965
22171
|
computer.command("status").description("Show which desktop services are running and the active preview URL").action(wrap(() => computerStatusCommand()));
|
|
20966
22172
|
computer.command("screenshot <path>").description("Capture the current desktop to a PNG file. Use --raw or --grid for agent click planning; omit both for a branded shareable image.").option("--raw", "Save a 1:1 desktop capture with no branding, padding, or rounded corners").option("--grid [px]", "Overlay a coordinate grid on a 1:1 desktop capture. Default 100px.").action(wrap((path6, options) => computerScreenshotCommand(path6, options)));
|
|
20967
22173
|
computer.command("observe <path>").description("Wait briefly for the desktop to settle, save a 1:1 screenshot, and print JSON screen context for agents.").option("--raw", "Save a 1:1 desktop capture with no coordinate grid").option("--grid [px]", "Overlay a coordinate grid on the 1:1 desktop capture. Default 100px.").option("--timeout <ms>", "Maximum time to wait for visual stability. Default 3000.").option("--stable-ms <ms>", "Required unchanged time before the screen is considered stable. Default 600.").option("--poll-ms <ms>", "Screenshot polling interval while waiting. Default 200.").action(wrap((path6, options) => computerObserveCommand(path6, options)));
|
|
20968
|
-
computer.command("browser").description("Print JSON for Chrome tabs launched through Replicas, including page titles and URLs.").option("--snapshot", "Include visible page text and interactive controls with bounding boxes").option("--limit <chars>", "Maximum body text characters per page snapshot. Default 4000.").option("--element-limit <n>", "Maximum controls per page snapshot. Default 80.").action(wrap((options) => computerBrowserCommand(options)));
|
|
20969
|
-
computer.command("browser-
|
|
20970
|
-
computer.command("browser-
|
|
20971
|
-
computer.command("browser-
|
|
22174
|
+
computer.command("browser").description("Print JSON for Chrome tabs launched through Replicas, including page titles and URLs.").option("--snapshot", "Include visible page text and interactive controls with bounding boxes").option("--limit <chars>", "Maximum body text characters per page snapshot. Default 4000.").option("--element-limit <n>", "Maximum controls per page snapshot. Default 80.").option("--target-id <id>", "Only include the Chrome page with this target id").option("--page <n>", "When multiple pages match, include the nth page. Default 0.").option("--title <text>", "Only include pages whose title contains text").option("--url <text>", "Only include pages whose URL contains text").action(wrap((options) => computerBrowserCommand(options)));
|
|
22175
|
+
computer.command("browser-state <path>").description("Wait for a Chrome page to settle, capture its viewport, and print a bounded accessibility state with actionable refs.").option("--full", "Return the full accessibility tree instead of a diff from the previous state").option("--limit <chars>", "Maximum visible text characters. Default 8000.").option("--element-limit <n>", "Maximum actionable elements. Default 120.").option("--timeout <ms>", "Maximum time to wait for state stability. Default 5000.").option("--stable-ms <ms>", "Required unchanged time before capture. Default 500.").option("--poll-ms <ms>", "State polling interval. Default 100.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((path6, options) => computerBrowserStateCommand(path6, options)));
|
|
22176
|
+
computer.command("browser-batch <actions>").description("Run an ordered, fail-fast JSON array of browser actions through one Chrome session.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((actions, options) => computerBrowserBatchCommand(actions, options)));
|
|
22177
|
+
computer.command("browser-click [text]").description("Click a Chrome element by ref from browser-state, or fall back to visible control text.").option("--ref <id>", "Click the backend DOM node ref from the latest browser state").option("--exact", "Require an exact text match instead of substring matching").option("--index <n>", "When multiple controls match, click the nth match. Default 0.").option("--button <button>", "Mouse button for ref clicks: left, middle, or right. Default left.").option("--double", "Double-click a ref target").option("--target-id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserClickCommand(text, options)));
|
|
22178
|
+
computer.command("browser-fill [field] [value]").description("Fill a Chrome field by ref from browser-state, or fall back to visible field text.").option("--ref <id>", "Fill the backend DOM node ref from the latest browser state").option("--exact", "Require an exact field match instead of substring matching").option("--index <n>", "When multiple fields match, fill the nth match. Default 0.").option("--target-id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((field, value, options) => {
|
|
22179
|
+
const resolvedValue = value ?? (options.ref ? field : void 0);
|
|
22180
|
+
const resolvedField = value === void 0 && options.ref ? void 0 : field;
|
|
22181
|
+
if (resolvedValue === void 0) throw new Error("Provide <field> <value>, or --ref <id> <value>");
|
|
22182
|
+
return computerBrowserFillCommand(resolvedField, resolvedValue, options);
|
|
22183
|
+
}));
|
|
22184
|
+
computer.command("browser-scroll <direction>").description("Send trusted wheel input to a Chrome element ref or the center of the target page.").option("--ref <id>", "Scroll from the backend DOM node ref from the latest browser state").option("--amount <px>", "Scroll distance in CSS pixels. Default 600.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((direction, options) => computerBrowserScrollCommand(direction, options)));
|
|
22185
|
+
computer.command("browser-key <combo>").description("Press a key or modifier chord in the target Chrome page with trusted browser input.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((combo, options) => computerBrowserKeyCommand(combo, options)));
|
|
22186
|
+
computer.command("browser-type <text>").description("Type literal text into the focused field in the target Chrome page with trusted browser input.").option("--target-id <id>", "Target the Chrome page with this stable target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserTypeCommand(text, options)));
|
|
22187
|
+
computer.command("browser-wait <text>").description("Wait until Chrome page title, URL, body text, or controls match <text>.").option("--mode <mode>", "Where to match: any, text, title, url, or control. Default any.").option("--exact", "Require an exact match instead of substring matching").option("--timeout <ms>", "Maximum time to wait. Default 10000.").option("--poll-ms <ms>", "Polling interval. Default 250.").option("--target-id <id>", "Only target the Chrome page with this target id").option("--page <n>", "When multiple pages match, target the nth page. Default 0.").option("--title <text>", "Only target pages whose title contains text").option("--url <text>", "Only target pages whose URL contains text").action(wrap((text, options) => computerBrowserWaitCommand(text, options)));
|
|
20972
22188
|
computer.command("click <x> <y>").description("Move to (x, y) and click. Coordinates are pixels or percentages like 50% 50%.").option("-b, --button <n>", "Mouse button (1=left, 2=middle, 3=right). Default 1.").option("--double", "Double-click instead of single-click").option("--modifiers <mods>", "Hold modifier keys during the click, e.g. ctrl or ctrl+shift").action(wrap((x, y, options) => computerClickCommand(x, y, options)));
|
|
20973
22189
|
computer.command("move <x> <y>").description("Move the mouse to (x, y) without clicking. Coordinates are pixels or percentages like 50% 50%.").action(wrap((x, y) => computerMoveCommand(x, y)));
|
|
20974
22190
|
computer.command("type <text>").description("Type a literal string into the focused field. Use `key` for key combos like ctrl+l.").option("--delay <ms>", "Per-character delay in ms (default 12 \u2248 80 wpm)").action(wrap((text, options) => computerTypeCommand(text, options)));
|