@glittering0717/ai-browser 1.3.1 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +56 -5
- package/bin/ai-browser.cjs +41 -4
- package/examples/AGENT_INSTRUCTIONS.md +49 -164
- package/llms-install.md +67 -5
- package/package.json +3 -1
- package/scripts/cli_call.cjs +222 -0
- package/scripts/ensure_running.cjs +195 -0
- package/scripts/skill.cjs +133 -0
- package/skills/ai-browser/SKILL.md +56 -0
- package/skills/ai-browser/references/cli.md +81 -0
- package/skills/ai-browser/references/hard-pages.md +52 -0
- package/skills/ai-browser/references/network.md +39 -0
- package/skills/ai-browser/references/tasks.md +44 -0
- package/skills/ai-browser/references/troubleshooting.md +31 -0
- package/skills/content-publish/SKILL.md +61 -0
- package/skills/read-webpage/SKILL.md +52 -0
- package/skills/web-network-monitor/SKILL.md +52 -0
- package/src/main/axExtractor.js +50 -6
- package/src/main/capability_probe.js +93 -5
- package/src/main/index.js +99 -6
- package/src/main/mcp_args.js +122 -0
- package/src/main/mcp_server.js +72 -159
- package/src/main/mcp_tools.js +127 -86
- package/src/main/mcp_ws.js +7 -1
- package/src/main/page_manager.js +481 -71
- package/src/main/window_open_shim_source.js +117 -0
- package/src/preload/bridge.cjs +22 -4
- package/src/preload/extractor.cjs +161 -17
- package/src/shared/doc_coverage.js +264 -0
- package/src/shared/effect_state.js +160 -0
- package/src/shared/frame_probe.cjs +91 -1
- package/src/shared/frame_read_source.js +260 -0
- package/src/shared/opened_tab.js +71 -0
- package/src/shared/ref_epoch.js +99 -0
- package/src/shared/tab_hooks.js +33 -0
- package/src/shared/version.js +1 -1
- package/tools/coverage_probe.cjs +111 -9
package/README.md
CHANGED
|
@@ -205,13 +205,38 @@ small surface, since every tool schema is injected into your agent's context on
|
|
|
205
205
|
| `browse_evaluate` | Run a JS *expression* in page context (5000-char cap, Node identifiers rejected). |
|
|
206
206
|
| `browse_scroll` | Scroll the page, or scroll an element into view. |
|
|
207
207
|
| `browse_wait` | Poll until `button_enabled` / `modal_appeared` / `text_contains` / `url_contains`. |
|
|
208
|
-
| `
|
|
209
|
-
| `browse_new_tab` | Open a new tab, optionally at a URL. |
|
|
210
|
-
| `browse_close_tab` | Close a tab by id. |
|
|
211
|
-
| `browse_set_active_tab` | Switch which tab subsequent calls act on. |
|
|
208
|
+
| `browse_tabs` | Tabs: `list` / `new` / `close` / `switch`. `switch` sets the default target for later calls. |
|
|
212
209
|
| `browse_network` | Chrome-DevTools-style network inspection: `list` (filter by method / URL / status / type), `get` (request + response headers and bodies), `clear`, `configure`. |
|
|
213
210
|
| `browse_quit` | Shut the whole browser down and release the process. |
|
|
214
211
|
|
|
212
|
+
### What every act result already tells you: `effect_state`
|
|
213
|
+
|
|
214
|
+
`browse_act` no longer leaves you guessing whether your input reached the page. Every result
|
|
215
|
+
carries an explicit verdict:
|
|
216
|
+
|
|
217
|
+
| `effect_state` | Meaning | `retry` |
|
|
218
|
+
|---|---|---|
|
|
219
|
+
| `committed` | dispatched and acked by the page | `no` |
|
|
220
|
+
| `none` | provably nothing happened (no such target, bad params, CDP down) | `safe` |
|
|
221
|
+
| `unknown` | cannot prove nothing happened — timeout, paste dispatched but never consumed, half a key sequence | `verify_first` |
|
|
222
|
+
|
|
223
|
+
The default is `unknown`, not `none`: when the browser cannot tell, it says so. Retrying an
|
|
224
|
+
`unknown` blindly is how a form gets submitted twice — read the page first. `unknown` results
|
|
225
|
+
also carry a one-sentence `effect_hint` saying exactly that.
|
|
226
|
+
|
|
227
|
+
When a `target` cannot be found, the result adds `stale_ref` / `ref_epoch` / `tree_epoch` /
|
|
228
|
+
`ref_diagnosis`, because "no such target" hides three very different situations:
|
|
229
|
+
|
|
230
|
+
- `ref_diagnosis: "stale"` — the page was re-read after you took that ref (`ref_epoch` <
|
|
231
|
+
`tree_epoch`). Get a fresh one; retrying the same ref will keep failing.
|
|
232
|
+
- `ref_diagnosis: "never_seen"` — the ref has never been in any tree read so far. Re-reading
|
|
233
|
+
will not help: wrong value (visible text instead of a `data-ai-id`), a typo, or another tab's ref.
|
|
234
|
+
- `ref_diagnosis: "present_in_tree"` — in the current tree but not found in the DOM: iframe,
|
|
235
|
+
shadow root, or a detached node.
|
|
236
|
+
|
|
237
|
+
All three report `effect_state: "none"` (nothing happened) with `retry: "verify_first"` — no side
|
|
238
|
+
effects, but repeating the identical call is pointless.
|
|
239
|
+
|
|
215
240
|
### Prove it happened: `browse_diff`
|
|
216
241
|
|
|
217
242
|
A tool reporting `success` is **not** evidence that anything changed — the browser will happily
|
|
@@ -396,6 +421,22 @@ MCP is a thin stdio adapter. Anything MCP can do is reachable directly over JSON
|
|
|
396
421
|
`ws://localhost:9223` from any language. Source of truth: the method switch in
|
|
397
422
|
[`src/main/ws_server.js`](src/main/ws_server.js).
|
|
398
423
|
|
|
424
|
+
You do not have to write a client to do this. The CLI speaks the same protocol:
|
|
425
|
+
|
|
426
|
+
```bash
|
|
427
|
+
ai-browser status # running? pid / port / tabs
|
|
428
|
+
ai-browser call ui.list_tabs
|
|
429
|
+
ai-browser call ui.get_tree '{"subset":"interactive"}'
|
|
430
|
+
ai-browser call ui.navigate '{"url":"https://example.com"}'
|
|
431
|
+
echo '{"url":"https://example.com"}' | ai-browser call ui.navigate -
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
`call` prints the raw `result` as one line of JSON (pipe it into `jq`), exits `0` on success,
|
|
435
|
+
`1` when the call failed, `2` on a bad command line. `--port`, `--timeout`, `--pretty` are available.
|
|
436
|
+
|
|
437
|
+
So there are three doors to one implementation — MCP stdio (`ai-browser mcp`), JSON-RPC over
|
|
438
|
+
WebSocket, and this CLI. None of them is a separate code path.
|
|
439
|
+
|
|
399
440
|
| Method | Purpose | Maps to |
|
|
400
441
|
|---|---|---|
|
|
401
442
|
| `ui.get_tree` | Semantic tree (+ context). Params: `focusedOnly`, `ax`, `subset`, `mode`, `tab` | `browse_get_tree` |
|
|
@@ -407,7 +448,7 @@ MCP is a thin stdio adapter. Anything MCP can do is reachable directly over JSON
|
|
|
407
448
|
| `ui.evaluate` | Run JS in page context, same guard as MCP | `browse_evaluate` |
|
|
408
449
|
| `ui.scroll` | Scroll page or element (`direction`, `amount`, `target`). **Document scrolling only** — for canvas apps use `ui.act{action:'wheel'}`. | `browse_scroll` |
|
|
409
450
|
| `ui.wait` | Poll until a condition or timeout | `browse_wait` |
|
|
410
|
-
| `ui.new_tab` / `ui.close_tab` / `ui.list_tabs` / `ui.set_active_tab` | Tab management | `
|
|
451
|
+
| `ui.new_tab` / `ui.close_tab` / `ui.list_tabs` / `ui.set_active_tab` | Tab management (the four are merged into `browse_tabs{op}` on the MCP side) | `browse_tabs` |
|
|
411
452
|
| `ui.subscribe` / `ui.unsubscribe` | Event subscription; `params.events`, `"*"` for all | — (WS only — see below) |
|
|
412
453
|
| `ui.downloads` | This tab's downloads with on-disk paths (`state`, `received_bytes`, `dir`) | `browse_downloads` |
|
|
413
454
|
| `ui.get_focused` | The focused element subtree | — |
|
|
@@ -660,6 +701,16 @@ It exits non-zero below 80% so you can drop it into CI. Measured on 2026-10-17 (
|
|
|
660
701
|
CSDN): **82% / 97% / 88%** before the click-boundary fix, **100% / 100% / 100%** after, at ~40ms
|
|
661
702
|
per `ui.get_tree`.
|
|
662
703
|
|
|
704
|
+
> **Those three numbers were measuring the wrong denominator.** Until 2026-10-19 the tool only
|
|
705
|
+
> counted the top document: on a page whose body lives in a cross-process iframe it reported
|
|
706
|
+
> "4 click targets, 100% usable, 0 gaps" while the entire course catalogue was invisible. It now
|
|
707
|
+
> measures every `frame-N` it can reach and **prints how many frames it could not** — an
|
|
708
|
+
> unmeasured area inside the denominator is worse than a wrong measurement. Re-measured
|
|
709
|
+
> 2026-10-19: Baidu 100%, CSDN 99% (8 cross-process frames now counted, 2 residual gaps),
|
|
710
|
+
> and the 超星 course page 82% with the 2 remaining gaps named explicitly. The write-up of this
|
|
711
|
+
> class of mistake — a metric that silently excludes the part that matters — is
|
|
712
|
+
> [`docs/POSTMORTEM_FRAME_VISIBILITY.md`](docs/POSTMORTEM_FRAME_VISIBILITY.md).
|
|
713
|
+
|
|
663
714
|
Two things the numbers do **not** cover, so you do not chase them: an element the page gives no
|
|
664
715
|
readable name (image-only links with no `alt` — 53% of Bilibili's clickable nodes) can never get a
|
|
665
716
|
label without inventing one, and a canvas has no DOM to read at all. Those are genuine boundaries,
|
package/bin/ai-browser.cjs
CHANGED
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
// ai-browser launch the visible browser window (foreground)
|
|
5
5
|
// ai-browser start same
|
|
6
6
|
// ai-browser mcp run the MCP stdio server (used by MCP clients)
|
|
7
|
+
// ai-browser skill list / install bundled agent skills
|
|
7
8
|
// ai-browser --version
|
|
8
9
|
// ai-browser --help
|
|
9
10
|
//
|
|
@@ -19,13 +20,24 @@ const HELP = `ai-browser ${pkg.version}
|
|
|
19
20
|
|
|
20
21
|
Usage:
|
|
21
22
|
ai-browser [start] Launch the visible AI Browser window
|
|
22
|
-
ai-browser mcp Run the MCP stdio server (
|
|
23
|
+
ai-browser mcp Run the MCP stdio server (17 browse_* tools)
|
|
24
|
+
ai-browser skill [list|install|print]
|
|
25
|
+
Install the bundled agent skills into your client
|
|
26
|
+
ai-browser call <method> [json|-]
|
|
27
|
+
Call one JSON-RPC method and print the result
|
|
28
|
+
(starts the browser first if it is not running)
|
|
29
|
+
ai-browser ensure Start the browser if needed, then exit (idempotent)
|
|
30
|
+
ai-browser status Is the browser running? pid / port / tabs
|
|
23
31
|
ai-browser --version Print version
|
|
24
32
|
ai-browser --help Show this help
|
|
25
33
|
|
|
26
|
-
The browser exposes:
|
|
27
|
-
• MCP server over stdio (
|
|
28
|
-
•
|
|
34
|
+
The browser exposes one implementation through three thin doors:
|
|
35
|
+
• MCP server over stdio → ai-browser mcp (for agent clients)
|
|
36
|
+
• WebSocket JSON-RPC → ws://127.0.0.1:9223 (ui.* methods)
|
|
37
|
+
• This CLI → ai-browser call ui.list_tabs
|
|
38
|
+
|
|
39
|
+
Agent skills ship in this package. Install them once with "ai-browser skill install"
|
|
40
|
+
and every new session picks them up automatically — no manual instruction pasting.
|
|
29
41
|
|
|
30
42
|
Docs: https://github.com/Glittering/ai-browser
|
|
31
43
|
`;
|
|
@@ -55,6 +67,31 @@ async function main() {
|
|
|
55
67
|
await import(pathToFileURL(serverPath).href);
|
|
56
68
|
break;
|
|
57
69
|
}
|
|
70
|
+
case 'skill': {
|
|
71
|
+
// 技能装进客户端的技能目录 —— 装一次,之后每次新会话自动发现。
|
|
72
|
+
const { run } = require('../scripts/skill.cjs');
|
|
73
|
+
process.exit(run(argv.slice(1)));
|
|
74
|
+
break;
|
|
75
|
+
}
|
|
76
|
+
case 'call': {
|
|
77
|
+
// 命令行直接调一条 JSON-RPC。存在的理由:以前想从 shell 问一句"有哪些标签页"
|
|
78
|
+
// 都得先 npm i ws 再写脚本。见 scripts/cli_call.cjs 顶部注释。
|
|
79
|
+
const { runCall } = require('../scripts/cli_call.cjs');
|
|
80
|
+
process.exit(await runCall(argv.slice(1)));
|
|
81
|
+
break;
|
|
82
|
+
}
|
|
83
|
+
case 'status': {
|
|
84
|
+
const { runStatus } = require('../scripts/cli_call.cjs');
|
|
85
|
+
process.exit(await runStatus(argv.slice(1)));
|
|
86
|
+
break;
|
|
87
|
+
}
|
|
88
|
+
case 'ensure': {
|
|
89
|
+
// 只把浏览器弄到"能连"的状态,然后退出。存在的理由:让 `call` 之外还有一条
|
|
90
|
+
// **显式**的启动动作,脚本/CI 里想先起再调时用它(幂等,已在跑就是 no-op)。
|
|
91
|
+
const { runEnsure } = require('../scripts/cli_call.cjs');
|
|
92
|
+
process.exit(await runEnsure(argv.slice(1)));
|
|
93
|
+
break;
|
|
94
|
+
}
|
|
58
95
|
case '--version':
|
|
59
96
|
case '-v':
|
|
60
97
|
case 'version':
|
|
@@ -1,186 +1,71 @@
|
|
|
1
|
-
# AI Browser —
|
|
1
|
+
# AI Browser — agent 接入
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
(Claude Code 的 `CLAUDE.md`、Cursor Rules、Codex `AGENTS.md`、或任意 MCP 客户端的自定义指令)。
|
|
5
|
-
不需要改任何东西——浏览器由 agent 自己在第一次调用时拉起。
|
|
3
|
+
## 先别贴这份文件
|
|
6
4
|
|
|
7
|
-
|
|
8
|
-
|
|
5
|
+
```bash
|
|
6
|
+
npx -y @glittering0717/ai-browser skill install
|
|
7
|
+
```
|
|
9
8
|
|
|
10
|
-
|
|
11
|
-
|
|
9
|
+
装一次,之后**每次新会话客户端会自动发现**这些技能,不需要你在对话里重新教一遍、也不用把
|
|
10
|
+
下面任何内容粘进 system prompt:
|
|
12
11
|
|
|
13
|
-
|
|
12
|
+
| 技能 | 干什么 |
|
|
13
|
+
|---|---|
|
|
14
|
+
| `ai-browser` | 最小循环、卡住时的第一反应、难搞页面的处理 |
|
|
15
|
+
| `content-publish` | 把文章发布到 CSDN / 知乎 / 掘金 |
|
|
16
|
+
| `read-webpage` | 读一篇文章正文并总结 |
|
|
17
|
+
| `web-network-monitor` | 长时间盯某个页面的请求 |
|
|
14
18
|
|
|
15
|
-
|
|
19
|
+
不装技能也能用——20 个 `browse_*` 工具的 schema 本身自带说明。技能只是把"怎么组合"讲清楚。
|
|
16
20
|
|
|
17
|
-
|
|
18
|
-
我可以用 AI Browser 浏览网页。它提供 MCP 服务(20 个 browse_* 工具),第一次调用时会自动
|
|
19
|
-
拉起一个真实可见的浏览器窗口,用完调用 browse_quit 释放。页面以结构化语义树返回,不是截图。
|
|
20
|
-
|
|
21
|
-
标准循环:
|
|
22
|
-
1. browse_navigate { url } 打开页面
|
|
23
|
-
2. browse_get_tree 读页面,返回语义树:每个节点有 role / label / value /
|
|
24
|
-
states / actions
|
|
25
|
-
3. browse_act { action, target } 操作元素,target 填第 2 步节点里的 id(即 data-ai-id)
|
|
26
|
-
action 可选:click / type(光标处追加)/ setContent
|
|
27
|
-
(整体替换)/ clear / focus / hover / scroll_to /
|
|
28
|
-
upload / get_value(只读回读全文)/ drag / press / wheel
|
|
29
|
-
4. browse_wait { condition } 等待条件成立:button_enabled / modal_appeared /
|
|
30
|
-
text_contains / url_contains
|
|
31
|
-
5. browse_evaluate { js } 在页面上下文跑 JS 表达式(上限 5000 字符,
|
|
32
|
-
禁用 process. / require / child_process)
|
|
33
|
-
6. browse_errors 出事了先看这里:主进程异常 / 崩溃 / 导航失败 /
|
|
34
|
-
RPC 报错的最近记录(含调用上下文与日志路径)
|
|
35
|
-
|
|
36
|
-
省 token(browse_get_tree 参数):
|
|
37
|
-
- { "subset": "interactive" } 只保留可交互节点,砍掉纯布局分支 —— 默认就该带上
|
|
38
|
-
- { "ax": true } 走 AX 语义层,链接会带 url
|
|
39
|
-
- { "mode": "diff" } 只看上次读取以来新增 / 消失的可交互节点
|
|
40
|
-
|
|
41
|
-
验证写操作(success 不等于做成了):browse_diff 先 snapshot 基线,写操作之后再 diff,
|
|
42
|
-
verdict 显式给出 changed / unchanged。
|
|
43
|
-
|
|
44
|
-
找不到元素 / 点了没反应时的排查顺序(**别一上来就写站点专用选择器**):
|
|
45
|
-
1. 重读 browse_get_tree,或先 browse_wait —— SPA 常常只是还没渲染完
|
|
46
|
-
2. 节点在树里但操作失败 → 是**动作层**问题(跨进程 iframe、canvas、需要受信输入、
|
|
47
|
-
上传控件未挂载),写选择器解决不了
|
|
48
|
-
3. 树里没有、页面上明明有 → 是**读层**缺口,属于浏览器该修的东西;先跑
|
|
49
|
-
tools/coverage_probe.cjs <url> 量化,再考虑绕过
|
|
50
|
-
4. 以上都不成立,才在自己脚本里用选择器兜底;别把"某个站点的选择器"记成通用知识
|
|
51
|
-
|
|
52
|
-
其他:browse_scroll 滚动文档(画布类页面用 browse_act 的 wheel);browse_list_tabs /
|
|
53
|
-
browse_new_tab / browse_close_tab / browse_set_active_tab 管标签页;browse_network 拉取网络
|
|
54
|
-
请求(含 POST body,免订阅);browse_capabilities 一次探明页面该走哪条路(结构化数据 /
|
|
55
|
-
语义树 / 指针手势);browse_canvas 读 canvas 绘制调用里的文本(不是 OCR);
|
|
56
|
-
browse_downloads 查下载文件落盘路径;browse_repaint 在窗口画面陈旧时强制重绘。
|
|
57
|
-
|
|
58
|
-
需要人帮忙时(登录、扫码、短信验证码、滑块 / 人机验证,以及任何我做不了的操作):
|
|
59
|
-
浏览器窗口是真实可见的,人随时可以用鼠标键盘操作。我应该在对话里直接说一句人话请用户去窗口里
|
|
60
|
-
完成,例如"请在浏览器窗口里扫码登录,完成后回复我'好了'",然后停下来等用户回复;收到"好了"之后
|
|
61
|
-
重新 browse_get_tree 确认状态再继续。不要编造登录凭据,也不要反复重试那些注定失败的点击。
|
|
62
|
-
浏览器侧没有任何"交接"接口,也不需要——人机协作就发生在我们的对话里。
|
|
63
|
-
|
|
64
|
-
收尾:任务完成后调用 browse_quit(会关闭整个浏览器窗口和进程,释放资源)。
|
|
65
|
-
```
|
|
21
|
+
---
|
|
66
22
|
|
|
67
|
-
##
|
|
23
|
+
## 兜底:客户端不支持技能时
|
|
24
|
+
|
|
25
|
+
把下面这段贴进 `CLAUDE.md` / `AGENTS.md` / 自定义指令。**只需要这几行**,其余按需查技能里的
|
|
26
|
+
`references/`。
|
|
68
27
|
|
|
69
28
|
```text
|
|
70
|
-
|
|
71
|
-
real, visible browser window on my first call, and I release it with browse_quit when done. Pages come
|
|
72
|
-
back as a structured semantic tree — no screenshots, no OCR.
|
|
73
|
-
|
|
74
|
-
Standard loop:
|
|
75
|
-
1. browse_navigate { url } load a page
|
|
76
|
-
2. browse_get_tree read the page: nodes carry role / label / value / states /
|
|
77
|
-
actions
|
|
78
|
-
3. browse_act { action, target } act on an element; target is the node's id (its data-ai-id)
|
|
79
|
-
action: click / type (append at caret) / setContent
|
|
80
|
-
(replace all) / clear / focus / hover / scroll_to /
|
|
81
|
-
upload / get_value (read-only) / drag / press / wheel
|
|
82
|
-
4. browse_wait { condition } block until: button_enabled / modal_appeared /
|
|
83
|
-
text_contains / url_contains
|
|
84
|
-
5. browse_evaluate { js } run a JS expression in page context (5000-char cap;
|
|
85
|
-
process. / require / child_process are rejected)
|
|
86
|
-
|
|
87
|
-
Saving tokens (browse_get_tree params):
|
|
88
|
-
- { "subset": "interactive" } drop layout-only branches — use this by default
|
|
89
|
-
- { "ax": true } AX semantic layer; links carry url
|
|
90
|
-
- { "mode": "diff" } only interactive nodes added/hidden since my last read
|
|
91
|
-
|
|
92
|
-
Verifying writes (success is not evidence): browse_diff snapshots a baseline, then diffs after the
|
|
93
|
-
write and returns an explicit verdict of changed / unchanged.
|
|
94
|
-
|
|
95
|
-
When an element cannot be found, or a click does nothing — work down this list, do **not** reach for a
|
|
96
|
-
site-specific selector first:
|
|
97
|
-
1. re-read with browse_get_tree, or browse_wait — an SPA usually just has not rendered yet
|
|
98
|
-
2. the node is in the tree but the action fails → an **action-layer** gap (cross-process iframe,
|
|
99
|
-
canvas, trusted input, unmounted upload input); a selector will not fix it
|
|
100
|
-
3. the node is missing though the page clearly renders it → a **read-layer** gap, i.e. something the
|
|
101
|
-
browser should fix; quantify it with `tools/coverage_probe.cjs <url>` before working around it
|
|
102
|
-
4. only then pin it with a selector inside your own script — and never record "the selector for site X"
|
|
103
|
-
as general knowledge
|
|
104
|
-
|
|
105
|
-
Also: browse_scroll (document scrolling; use the wheel action on canvas apps); browse_list_tabs /
|
|
106
|
-
browse_new_tab / browse_close_tab / browse_set_active_tab for tabs; browse_network for captured
|
|
107
|
-
requests including POST bodies (no subscription); browse_capabilities to pick a route on an
|
|
108
|
-
unfamiliar page (structured data / semantic tree / pointer gestures); browse_canvas reads text
|
|
109
|
-
straight from canvas draw calls (no OCR); browse_downloads for on-disk download paths;
|
|
110
|
-
browse_repaint when the window shows a stale page.
|
|
111
|
-
|
|
112
|
-
When I need a human (login, QR code, SMS code, slider/CAPTCHA, or anything I genuinely cannot do):
|
|
113
|
-
the browser window is real and visible — the user can just use their mouse and keyboard in it. I should
|
|
114
|
-
say so plainly in the conversation (e.g. "please scan the QR code in the browser window and reply
|
|
115
|
-
'done'"), then stop and wait for that reply. After "done", I re-read with browse_get_tree and continue.
|
|
116
|
-
I never invent credentials or loop on clicks that cannot work. There is no hand-off API on the browser
|
|
117
|
-
side, and none is needed — the collaboration happens right here in our conversation.
|
|
118
|
-
|
|
119
|
-
Wrap up: call browse_quit when the task is finished (it closes the window and exits the process).
|
|
120
|
-
```
|
|
29
|
+
我可以用 ai-browser 操作一个真实可见的浏览器(MCP: browse_* 工具)。
|
|
121
30
|
|
|
122
|
-
|
|
31
|
+
循环:browse_navigate {url} → browse_get_tree {subset:"interactive"} 取节点 id →
|
|
32
|
+
browse_act {action:"click"|"type", target:id, text?}
|
|
123
33
|
|
|
124
|
-
|
|
34
|
+
success:true 只代表动作发出去了,不代表页面变了。要证明生效:browse_diff 先 snapshot、
|
|
35
|
+
操作后 diff,看 verdict。unchanged 说明要么点错了、要么页面没响应,别接着瞎点。
|
|
125
36
|
|
|
126
|
-
|
|
127
|
-
|
|
37
|
+
卡住时的顺序:节点没有 → browse_wait 再重读树;有节点但操作失败 → 动作层问题(iframe /
|
|
38
|
+
canvas / 富文本),换动作不是换选择器;点了没反应 → browse_screenshot 看视觉 或
|
|
39
|
+
browse_network 看请求;怀疑掉登录 → browse_auth_state;都不是 → browse_errors。
|
|
128
40
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
"command": "npx",
|
|
134
|
-
"args": ["-y", "@glittering0717/ai-browser", "mcp"]
|
|
135
|
-
}
|
|
136
|
-
}
|
|
137
|
-
}
|
|
41
|
+
需要人(登录/扫码/滑块/支付):在对话里直接请用户在窗口里操作,收到"好了"再 browse_get_tree
|
|
42
|
+
确认。不要编造凭据,不要重试注定失败的点击。窗口是真的,人随时能抢鼠标。
|
|
43
|
+
|
|
44
|
+
收尾:browse_quit 关掉窗口和进程(人正在操作时不要调)。
|
|
138
45
|
```
|
|
139
46
|
|
|
140
|
-
|
|
141
|
-
`/你的/绝对路径/ai-browser/src/main/mcp_server.js`(必须是绝对路径)。
|
|
47
|
+
---
|
|
142
48
|
|
|
143
|
-
|
|
144
|
-
`ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/`。
|
|
145
|
-
* 无需事先启动浏览器:MCP 服务器会探测 `9223` 端口,没人在监听就自己 spawn Electron(最多等 30 秒)。
|
|
146
|
-
* 想常驻浏览器也可以:在项目根目录跑 `npm start`,之后所有客户端共用这一个窗口。
|
|
147
|
-
* 不装 MCP 也能用:原始 WebSocket JSON-RPC 在 `ws://localhost:9223`,见 `raw-ws.mjs`。
|
|
49
|
+
## 详细内容在哪
|
|
148
50
|
|
|
149
|
-
|
|
51
|
+
按需读,不要一次读完:
|
|
150
52
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
|
154
|
-
|
|
155
|
-
|
|
|
156
|
-
|
|
|
157
|
-
|
|
|
158
|
-
| `browse_evaluate` | `js`, `tab?` | 表达式(非语句),5000 字符上限 |
|
|
159
|
-
| `browse_scroll` | `direction`, `amount`, `target`, `tab` | 滚页面或滚到某元素 |
|
|
160
|
-
| `browse_wait` | `condition`, `target?`, `text?`, `timeout_ms?` | 轮询直到成立或超时 |
|
|
161
|
-
| `browse_diff` | `operation` (`snapshot`/`diff`), `js?`, `snapshot?`, `rearm?`, `forget?` | 快照基线 + 比对,证明写操作真的生效 |
|
|
162
|
-
| `browse_capabilities` | `include_network?`, `network_limit?` | 探明页面路线:L1 结构化数据 / L2 语义树 / L3 指针手势 |
|
|
163
|
-
| `browse_network` | `operation` (`list`/`get`/`clear`/`configure`), `method?`, `url_contains?`, `status_min/max?` | 网络请求检查,含 POST body,默认免订阅 |
|
|
164
|
-
| `browse_canvas` | `operation` (`list`/`read`/`configure`/`capture`), `canvas_id?` | 读 canvas 绘制调用文本;WebGL 才用 `capture` |
|
|
165
|
-
| `browse_downloads` | — | 查下载项与落盘路径(下载异步,点完后轮询) |
|
|
166
|
-
| `browse_repaint` | — | 窗口画面陈旧时强制合成器重绘,返回 `repaint_count` |
|
|
167
|
-
| `browse_list_tabs` / `browse_new_tab` / `browse_close_tab` / `browse_set_active_tab` | `tab?`, `url?` | 标签页管理 |
|
|
168
|
-
| `browse_quit` | — | 关闭窗口并退出进程 |
|
|
169
|
-
|
|
170
|
-
> 实时事件订阅(`dom_change` / `network_response` / `js_error` 等)只在原始 WebSocket 层
|
|
171
|
-
> 提供(`ui.subscribe`),MCP stdio 不支持 server→client 通知,所以没有 `browse_subscribe`;
|
|
172
|
-
> 拉取式的检查一律用 `browse_network`。
|
|
53
|
+
| 想知道 | 读 |
|
|
54
|
+
|---|---|
|
|
55
|
+
| 失败排查顺序、四条别做、省 token 的读树参数 | `skills/ai-browser/references/troubleshooting.md` |
|
|
56
|
+
| canvas / 富文本 / 跨进程 iframe / 上传 | `skills/ai-browser/references/hard-pages.md` |
|
|
57
|
+
| 抓请求、从页面取结构化数据 | `skills/ai-browser/references/network.md` |
|
|
58
|
+
| 登录、表单、多标签、证明生效 | `skills/ai-browser/references/tasks.md` |
|
|
59
|
+
| 全部参数 | 工具的 `inputSchema`(每轮都在,不用另读) |
|
|
173
60
|
|
|
174
|
-
|
|
61
|
+
## 配置 MCP 服务器
|
|
175
62
|
|
|
176
|
-
|
|
63
|
+
见 `examples/mcp-config.json`,或:
|
|
177
64
|
|
|
178
|
-
|
|
65
|
+
```json
|
|
66
|
+
{ "mcpServers": { "ai-browser": {
|
|
67
|
+
"command": "npx", "args": ["-y", "@glittering0717/ai-browser", "mcp"]
|
|
68
|
+
} } }
|
|
69
|
+
```
|
|
179
70
|
|
|
180
|
-
|
|
181
|
-
2. agent 做不了的事,**在对话里说一句人话请求用户去做**,然后停下来等回复。
|
|
182
|
-
3. 用户做完回一句"好了",agent 重新 `browse_get_tree` 确认状态,继续往下走。
|
|
183
|
-
4. **浏览器侧没有任何"交接"接口,也不需要。** 交接是一个对话协议,不是一个 API。
|
|
184
|
-
脚本版的等价实现见 `with-human.mjs`:它只是 `readline` 打印一句提示 + 等回车,
|
|
185
|
-
浏览器对此毫不知情。
|
|
186
|
-
5. 登录态存在 `persist:` partition 里,重启浏览器也还在,通常只需人工介入一次。
|
|
71
|
+
浏览器由 agent 在第一次调用时自己拉起,不需要你手动开。
|
package/llms-install.md
CHANGED
|
@@ -11,8 +11,9 @@ semantic tree (roles / labels / values / actions, stable `data-ai-id` per elemen
|
|
|
11
11
|
not screenshots. Input is trusted CDP keyboard/mouse events. A human can use the
|
|
12
12
|
window's mouse and keyboard at any time (login, QR scan, CAPTCHA, payment).
|
|
13
13
|
|
|
14
|
-
Two equivalent interfaces: **MCP stdio** (
|
|
15
|
-
JSON-RPC API at `ws://localhost:9223` (`ui.*` methods).
|
|
14
|
+
Two equivalent interfaces: **MCP stdio** (17 `browse_*` tools) and a raw WebSocket
|
|
15
|
+
JSON-RPC API at `ws://localhost:9223` (`ui.*` methods). A CLI (`ai-browser call ui.xxx`) speaks
|
|
16
|
+
the same JSON-RPC and needs **no registration at all** — see "Fastest path" below.
|
|
16
17
|
|
|
17
18
|
## Requirements
|
|
18
19
|
|
|
@@ -65,6 +66,28 @@ ls node_modules/electron/dist # must contain Electron.app (macOS)
|
|
|
65
66
|
Optional end-to-end check: `npx -y @glittering0717/ai-browser start` (or `npm start` from source) opens
|
|
66
67
|
a 1280×800 window and logs `AI Browser WS server listening on ws://localhost:9223`.
|
|
67
68
|
|
|
69
|
+
## Fastest path — works with zero registration
|
|
70
|
+
|
|
71
|
+
If you just need to drive the browser now, skip the MCP section entirely. The CLI reaches the
|
|
72
|
+
same engine, and **starts the browser itself if it is not running** (idempotent; no separate
|
|
73
|
+
`start` step):
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
npm i -g @glittering0717/ai-browser # or: npm link (from a source checkout)
|
|
77
|
+
ai-browser call ui.navigate '{"url":"https://example.com"}'
|
|
78
|
+
ai-browser call ui.get_tree '{"subset":"interactive"}'
|
|
79
|
+
ai-browser call ui.act '{"action":"click","target":"e:a-3"}'
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
No global install? Prefix every command with `npx -y @glittering0717/ai-browser`.
|
|
83
|
+
|
|
84
|
+
Why this matters more than it looks: registering an MCP server passes through **three gates the
|
|
85
|
+
agent does not control** — the config file, a one-time trust/approval for executing a local
|
|
86
|
+
program, and the client's tool snapshot taken at session start. If any one of them has not
|
|
87
|
+
completed, `browse_*` simply does not exist in the tool list, and the symptom is identical to
|
|
88
|
+
"the capability is not installed". A weaker model then spends a long time concluding it is
|
|
89
|
+
misconfigured. The CLI has none of those gates: it only needs the command to run.
|
|
90
|
+
|
|
68
91
|
## Register as an MCP server
|
|
69
92
|
|
|
70
93
|
Add this to the client config (Claude Desktop config, `~/.cursor/mcp.json`, project `.mcp.json`,
|
|
@@ -84,13 +107,28 @@ Cline, etc.):
|
|
|
84
107
|
From-source alternative: `"command": "node"` with the absolute path to
|
|
85
108
|
`src/main/mcp_server.js`.
|
|
86
109
|
|
|
110
|
+
> **Match the version, or nothing lines up.** The skill shipped in this package documents the
|
|
111
|
+
> tool surface of the *same* version. Before pasting `npx` above, compare
|
|
112
|
+
> `npm view @glittering0717/ai-browser version` with this repo's `package.json`. If they differ
|
|
113
|
+
> (they have: npm 1.3.1 vs repo 2.0.0), use the from-source form with the source path — a 1.x
|
|
114
|
+
> server behind a 2.x skill means every tool name and parameter in the skill is wrong, and a
|
|
115
|
+
> small model cannot tell that apart from "the MCP server is down". A concrete field report:
|
|
116
|
+
> a 35B local model spent a long session concluding "the skill is installed but the MCP server
|
|
117
|
+
> is not mounted", correctly diagnosing a missing config entry while the real trap was that
|
|
118
|
+
> nothing had been registered at all — see the `ai-browser` entry in the client's `mcp.json`.
|
|
119
|
+
>
|
|
120
|
+
> **No MCP at all?** The CLI speaks the same JSON-RPC and needs no registration:
|
|
121
|
+
> `ai-browser status`, `ai-browser call ui.navigate '{"url":"..."}'`. Installing the package
|
|
122
|
+
> globally (`npm i -g @glittering0717/ai-browser`) puts that binary on PATH; from a source
|
|
123
|
+
> checkout, `npm link` does the same.
|
|
124
|
+
|
|
87
125
|
Do **not** launch the browser yourself first: on the first `browse_*` call the MCP server probes
|
|
88
126
|
port 9223, spawns Electron detached if nothing is listening, and waits up to 30 s (the very first
|
|
89
127
|
call can take longer if the engine download hasn't happened yet — the progress goes to stderr).
|
|
90
128
|
A long-lived shared window is also fine — run the `start` command manually; a second launch only
|
|
91
129
|
focuses the existing window (single instance lock).
|
|
92
130
|
|
|
93
|
-
## The
|
|
131
|
+
## The 17 tools
|
|
94
132
|
|
|
95
133
|
- `browse_navigate` — load a URL in the active or given tab.
|
|
96
134
|
- `browse_get_tree` — semantic tree + context (modals, toasts, required-field
|
|
@@ -100,6 +138,11 @@ focuses the existing window (single instance lock).
|
|
|
100
138
|
`setContent` (replace all), `clear`, `focus`, `hover`, `scroll_to`, `upload`,
|
|
101
139
|
`get_value` (read-only full value; tree truncates at 200 chars), `drag`,
|
|
102
140
|
`press`, `wheel`.
|
|
141
|
+
Every result carries `effect_state: "committed" | "none" | "unknown"` plus a
|
|
142
|
+
`retry` verdict (`no` / `safe` / `verify_first`). Treat `unknown` as "read the
|
|
143
|
+
page before touching it again" — a blind retry can submit twice. A missing
|
|
144
|
+
target adds `stale_ref` + `ref_diagnosis` (`stale` → re-read the tree;
|
|
145
|
+
`never_seen` → the ref itself is wrong; `present_in_tree` → iframe/shadow DOM).
|
|
103
146
|
- `browse_diff` — `snapshot` a baseline, then `diff`; returns an explicit
|
|
104
147
|
`verdict: "changed" | "unchanged"` with before/after JSON paths. Wrap every
|
|
105
148
|
write with this — a success return is not evidence the write worked.
|
|
@@ -118,10 +161,29 @@ focuses the existing window (single instance lock).
|
|
|
118
161
|
- `browse_repaint` — force compositor redraw when the window shows a stale page.
|
|
119
162
|
- `browse_downloads` — list downloads with on-disk paths (poll after clicking
|
|
120
163
|
export/download; downloads are asynchronous).
|
|
121
|
-
- `
|
|
122
|
-
`
|
|
164
|
+
- `browse_tabs` — tab management: `{op: "list" | "new" | "close" | "switch"}`.
|
|
165
|
+
`switch` sets the default target for later calls, so you rarely pass `tab`.
|
|
123
166
|
- `browse_quit` — shut the whole browser down and release the process.
|
|
124
167
|
|
|
168
|
+
**Which door should you use?** Prefer the **CLI** unless the tool list already contains
|
|
169
|
+
`browse_*`. MCP is a nicer interface (schemas are self-describing) but it is not under your
|
|
170
|
+
control; the CLI is. Both hit one implementation (`page_manager`) — there is no capability that
|
|
171
|
+
only exists over MCP.
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
ai-browser status # is it running? pid / port / tabs
|
|
175
|
+
ai-browser ensure # start it if needed, then exit
|
|
176
|
+
ai-browser call ui.list_tabs
|
|
177
|
+
ai-browser call ui.navigate '{"url":"..."}'
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
CLI method names are the tool names with `browse_` → `ui.` (`get_tree`, `act`, `navigate`,
|
|
181
|
+
`evaluate`, `scroll`, `wait`, `screenshot`, `errors`, `capabilities`, `downloads`, `quit`,
|
|
182
|
+
`auth_state`, `repaint`). The four that do not map one-to-one: `tabs` → `ui.list_tabs` /
|
|
183
|
+
`ui.new_tab` / `ui.close_tab` / `ui.set_active_tab`; `network` → `ui.network_list` /
|
|
184
|
+
`ui.network_get`; `canvas` → `ui.canvas_list` / `ui.canvas_read` / `ui.canvas_capture` /
|
|
185
|
+
`ui.canvas_configure`; `diff` → `ui.snapshot` / `ui.diff`.
|
|
186
|
+
|
|
125
187
|
There is no `browse_subscribe` and no `browse_network_body`: real-time events are
|
|
126
188
|
raw-WebSocket only (`ui.subscribe`), because MCP stdio drops server→client
|
|
127
189
|
notifications. Use `browse_network` for pull-based inspection.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@glittering0717/ai-browser",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "2.1.0",
|
|
4
4
|
"description": "AI-native browser — renders pixels for humans, structured semantic-tree API for agents. No screenshots, no OCR, no coordinate guessing.",
|
|
5
5
|
"main": "src/main/index.js",
|
|
6
6
|
"bin": {
|
|
@@ -10,6 +10,7 @@
|
|
|
10
10
|
"bin",
|
|
11
11
|
"scripts",
|
|
12
12
|
"src",
|
|
13
|
+
"skills",
|
|
13
14
|
"tools/coverage_probe.cjs",
|
|
14
15
|
"examples/AGENT_INSTRUCTIONS.md",
|
|
15
16
|
"examples/mcp-config.json",
|
|
@@ -70,6 +71,7 @@
|
|
|
70
71
|
"nav": "node e2e/nav_ws.cjs",
|
|
71
72
|
"audit": "node e2e/audit_ws.cjs",
|
|
72
73
|
"e2e:all": "node e2e/run_all.cjs",
|
|
74
|
+
"readcov": "node e2e/readcov_ws.cjs",
|
|
73
75
|
"coverage": "node tools/coverage_probe.cjs"
|
|
74
76
|
},
|
|
75
77
|
"dependencies": {
|