@agentruntime-labs/browser-agent 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +202 -0
- package/NOTICE +6 -0
- package/README.md +64 -34
- package/bin/browser-agent.js +1 -0
- package/cli/control.mjs +477 -0
- package/cli/daemon.mjs +143 -0
- package/cli/hosts.mjs +1 -0
- package/cli/main.mjs +24 -0
- package/cli/mcp.mjs +1 -0
- package/docs/ARCHITECTURE.md +73 -0
- package/docs/DESKTOP_MCP.md +41 -0
- package/docs/EXTENSION_MANIFEST.md +163 -0
- package/docs/HYBRID.md +12 -0
- package/docs/MANAGED_BROWSER.md +111 -0
- package/docs/MCP_TOOLS.md +486 -0
- package/docs/MESSAGES.md +272 -0
- package/docs/MVP.md +76 -0
- package/docs/PRIVACY_POLICY.md +50 -0
- package/docs/README.md +27 -0
- package/docs/REFERENCE_PROJECTS.md +24 -0
- package/docs/RELEASE_CHECKLIST.md +49 -0
- package/docs/REPEATED_DOCS.md +27 -0
- package/docs/SAFETY.md +92 -0
- package/docs/SHELL_QUOTING.md +30 -0
- package/docs/SPLIT_REPOS.md +46 -0
- package/docs/THIRD_PARTY_LICENSES.md +182 -0
- package/docs/VISION.md +38 -0
- package/docs/WEB_STORE_SUBMISSION.md +117 -0
- package/extension/approval.js +1 -0
- package/extension/assets/agentruntime-logo.svg +77 -0
- package/extension/assets/agentruntime-mark-512.png +0 -0
- package/extension/background.js +776 -62
- package/extension/cdp.js +577 -0
- package/extension/constants.js +1 -0
- package/extension/content.js +69 -9
- package/extension/devtools-grant.html +31 -0
- package/extension/devtools-grant.js +29 -0
- package/extension/icons/icon128.png +0 -0
- package/extension/icons/icon16.png +0 -0
- package/extension/icons/icon32.png +0 -0
- package/extension/icons/icon48.png +0 -0
- package/extension/manifest.json +50 -23
- package/extension/policy.js +102 -0
- package/extension/popup.html +190 -56
- package/extension/popup.js +376 -166
- package/extension/soft.js +301 -0
- package/extension/welcome.html +91 -0
- package/extension/welcome.js +13 -0
- package/package.json +6 -2
- package/server/dist/arBrowserPaths.d.ts.map +1 -1
- package/server/dist/arBrowserPaths.js +1 -0
- package/server/dist/arBrowserPaths.js.map +1 -1
- package/server/dist/automationBackend.d.ts +46 -0
- package/server/dist/automationBackend.d.ts.map +1 -0
- package/server/dist/automationBackend.js +20 -0
- package/server/dist/automationBackend.js.map +1 -0
- package/server/dist/automationSession.d.ts.map +1 -1
- package/server/dist/automationSession.js +1 -0
- package/server/dist/automationSession.js.map +1 -1
- package/server/dist/bridge.d.ts +17 -4
- package/server/dist/bridge.d.ts.map +1 -1
- package/server/dist/bridge.js +132 -33
- package/server/dist/bridge.js.map +1 -1
- package/server/dist/browserToolsExtras.d.ts +2 -2
- package/server/dist/browserToolsExtras.d.ts.map +1 -1
- package/server/dist/browserToolsExtras.js +70 -1
- package/server/dist/browserToolsExtras.js.map +1 -1
- package/server/dist/cdpTools.d.ts +5 -0
- package/server/dist/cdpTools.d.ts.map +1 -0
- package/server/dist/cdpTools.js +411 -0
- package/server/dist/cdpTools.js.map +1 -0
- package/server/dist/compositeTools.d.ts +2 -2
- package/server/dist/compositeTools.d.ts.map +1 -1
- package/server/dist/compositeTools.js.map +1 -1
- package/server/dist/debugSnapshot.d.ts +2 -2
- package/server/dist/debugSnapshot.d.ts.map +1 -1
- package/server/dist/debugSnapshot.js.map +1 -1
- package/server/dist/guardrails.d.ts.map +1 -1
- package/server/dist/guardrails.js +1 -0
- package/server/dist/guardrails.js.map +1 -1
- package/server/dist/humanLike.d.ts.map +1 -1
- package/server/dist/humanLike.js +1 -0
- package/server/dist/humanLike.js.map +1 -1
- package/server/dist/main.js +56 -15
- package/server/dist/main.js.map +1 -1
- package/server/dist/managedBridge.d.ts +58 -0
- package/server/dist/managedBridge.d.ts.map +1 -0
- package/server/dist/managedBridge.js +619 -0
- package/server/dist/managedBridge.js.map +1 -0
- package/server/dist/managedDom.d.ts +58 -0
- package/server/dist/managedDom.d.ts.map +1 -0
- package/server/dist/managedDom.js +310 -0
- package/server/dist/managedDom.js.map +1 -0
- package/server/dist/policyTiers.d.ts.map +1 -1
- package/server/dist/policyTiers.js +3 -1
- package/server/dist/policyTiers.js.map +1 -1
- package/server/dist/protocol.d.ts +4 -0
- package/server/dist/protocol.d.ts.map +1 -1
- package/server/dist/protocol.js +1 -0
- package/server/dist/protocol.js.map +1 -1
- package/server/dist/recorder.d.ts +22 -0
- package/server/dist/recorder.d.ts.map +1 -0
- package/server/dist/recorder.js +82 -0
- package/server/dist/recorder.js.map +1 -0
- package/server/dist/richEditorFill.d.ts.map +1 -1
- package/server/dist/richEditorFill.js +1 -0
- package/server/dist/richEditorFill.js.map +1 -1
- package/server/dist/screenshotCrop.d.ts.map +1 -1
- package/server/dist/screenshotCrop.js +1 -0
- package/server/dist/screenshotCrop.js.map +1 -1
- package/server/dist/sessionLog.d.ts +1 -0
- package/server/dist/sessionLog.d.ts.map +1 -1
- package/server/dist/sessionLog.js +1 -0
- package/server/dist/sessionLog.js.map +1 -1
- package/server/dist/sessionPhase.d.ts.map +1 -1
- package/server/dist/sessionPhase.js.map +1 -1
- package/server/dist/sessionSummary.d.ts.map +1 -1
- package/server/dist/sessionSummary.js +1 -0
- package/server/dist/sessionSummary.js.map +1 -1
- package/server/dist/shotUtil.d.ts +44 -0
- package/server/dist/shotUtil.d.ts.map +1 -0
- package/server/dist/shotUtil.js +89 -0
- package/server/dist/shotUtil.js.map +1 -0
- package/server/dist/skillsInstructions.d.ts.map +1 -1
- package/server/dist/skillsInstructions.js +1 -0
- package/server/dist/skillsInstructions.js.map +1 -1
- package/server/dist/snapshotText.d.ts +47 -0
- package/server/dist/snapshotText.d.ts.map +1 -0
- package/server/dist/snapshotText.js +99 -0
- package/server/dist/snapshotText.js.map +1 -0
- package/server/dist/structureFilter.d.ts.map +1 -1
- package/server/dist/structureFilter.js.map +1 -1
- package/server/dist/surveyCatalog.d.ts.map +1 -1
- package/server/dist/surveyCatalog.js +1 -0
- package/server/dist/surveyCatalog.js.map +1 -1
- package/server/dist/surveyRunner.d.ts.map +1 -1
- package/server/dist/surveyRunner.js.map +1 -1
- package/server/dist/toolHelpers.d.ts +27 -2
- package/server/dist/toolHelpers.d.ts.map +1 -1
- package/server/dist/toolHelpers.js +67 -1
- package/server/dist/toolHelpers.js.map +1 -1
- package/server/dist/toolRuntime.d.ts +4 -0
- package/server/dist/toolRuntime.d.ts.map +1 -1
- package/server/dist/toolRuntime.js +12 -6
- package/server/dist/toolRuntime.js.map +1 -1
- package/server/dist/toolSchemas.d.ts +37 -1
- package/server/dist/toolSchemas.d.ts.map +1 -1
- package/server/dist/toolSchemas.js +33 -1
- package/server/dist/toolSchemas.js.map +1 -1
- package/server/dist/tools.d.ts +2 -2
- package/server/dist/tools.d.ts.map +1 -1
- package/server/dist/tools.js +65 -11
- package/server/dist/tools.js.map +1 -1
- package/server/dist/verdictExtract.d.ts.map +1 -1
- package/server/dist/verdictExtract.js +1 -0
- package/server/dist/verdictExtract.js.map +1 -1
- package/server/dist/workflowTools.d.ts +2 -2
- package/server/dist/workflowTools.d.ts.map +1 -1
- package/server/dist/workflowTools.js +16 -4
- package/server/dist/workflowTools.js.map +1 -1
- package/skills/browser-agent/SKILL.md +89 -0
|
@@ -0,0 +1,486 @@
|
|
|
1
|
+
# MCP tools (v0 schemas)
|
|
2
|
+
|
|
3
|
+
This document specifies **MCP tool definitions** exposed by the local stdio MCP server: `name`, `description`, and **`inputSchema`** (JSON Schema; MCP defaults to **JSON Schema 2020-12** when no `$schema` is set—see [MCP tools in the spec](https://modelcontextprotocol.io/specification/2024-11-05/server/tools)).
|
|
4
|
+
|
|
5
|
+
**Implementation:** The server also registers a Zod **`outputSchema`** per tool and returns **`structuredContent`** validated at runtime (`server/src/toolSchemas.ts`). Prefer consuming **`structuredContent`** in MCP clients when available; the `text` content mirrors it as JSON for convenience.
|
|
6
|
+
|
|
7
|
+
**Cursor:** Hosts like Cursor require MCP tool `name`s to be **letters, digits, and underscores only** (no dots). Names use the `browser_*` pattern, not `browser.*`.
|
|
8
|
+
|
|
9
|
+
**v0 rule:** No arbitrary `evaluate` unless the MCP host sets **`BROWSER_AUTOMATION_ALLOW_EVAL=1`**. Console and network tools are **metadata-only** (no response bodies). Tools operate on a **connected tab** unless explicitly noted; several DOM tools accept optional **`frameId`** for subframe targeting.
|
|
10
|
+
|
|
11
|
+
## Bridge mapping
|
|
12
|
+
|
|
13
|
+
The MCP server maps each tool to a WebSocket **`request.method`** (short name) for the extension. Use this table as the single source of truth.
|
|
14
|
+
|
|
15
|
+
| MCP tool name | Bridge `method` |
|
|
16
|
+
|---------------|-----------------|
|
|
17
|
+
| `browser_navigate` | `navigate` |
|
|
18
|
+
| `browser_get_current_tab` | `get_current_tab` |
|
|
19
|
+
| `browser_get_tabs` | `get_tabs` |
|
|
20
|
+
| `browser_get_page_text` | `get_page_text` |
|
|
21
|
+
| `browser_get_page_structure` | `get_page_structure` |
|
|
22
|
+
| `browser_click_element` | `click_element` |
|
|
23
|
+
| `browser_fill_input` | `fill_input` |
|
|
24
|
+
| `browser_press_key` | `press_key` |
|
|
25
|
+
| `browser_scroll` | `scroll` |
|
|
26
|
+
| `browser_select_option` | `select_option` |
|
|
27
|
+
| `browser_hover` | `hover` |
|
|
28
|
+
| `browser_go_back` | `go_back` |
|
|
29
|
+
| `browser_reload` | `reload` |
|
|
30
|
+
| `browser_tab_open` | `tab_open` |
|
|
31
|
+
| `browser_tab_close` | `tab_close` |
|
|
32
|
+
| `browser_tab_activate` | `tab_activate` |
|
|
33
|
+
| `browser_screenshot_visible` | `screenshot_visible` |
|
|
34
|
+
| `browser_get_selected_text` | `get_selected_text` |
|
|
35
|
+
| `browser_wait` | `wait` |
|
|
36
|
+
| `browser_wait_for` | `wait_for` |
|
|
37
|
+
| `browser_ask_confirmation` | `ask_confirmation` |
|
|
38
|
+
| `browser_find_nodes` | *(server composes `get_page_structure`)* |
|
|
39
|
+
| `browser_screenshot_element` | `screenshot_element` |
|
|
40
|
+
| `browser_console_recent` | `console_recent` |
|
|
41
|
+
| `browser_network_recent` | `get_network_log` |
|
|
42
|
+
| `browser_set_file_input` | `set_file_input` |
|
|
43
|
+
| `browser_get_element_screen_rect` | `get_ref_screen_rect` |
|
|
44
|
+
| `browser_evaluate` | `evaluate_expression` |
|
|
45
|
+
| `browser_assert_url_contains` | *(server composes `get_current_tab`)* |
|
|
46
|
+
| `browser_assert_text_contains` | *(server composes `get_page_text`)* |
|
|
47
|
+
|
|
48
|
+
Parameters **`tabId`** and **`frameId`** (where supported) are passed through to the extension; omit **`tabId`** to use the user-connected tab from the popup.
|
|
49
|
+
|
|
50
|
+
## Shared types (informal)
|
|
51
|
+
|
|
52
|
+
These appear inside `inputSchema` / results; document as conventions until you add `outputSchema`.
|
|
53
|
+
|
|
54
|
+
| Type | Meaning |
|
|
55
|
+
|------|--------|
|
|
56
|
+
| `TabId` | Chrome internal tab id (integer). |
|
|
57
|
+
| `ElementRef` | Opaque string from latest snapshot for **this** tab (invalid after major navigation unless implementation refreshes). |
|
|
58
|
+
| `Url` | string (absolute URL of loaded page). |
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## Tool definitions
|
|
63
|
+
|
|
64
|
+
### `browser_navigate`
|
|
65
|
+
|
|
66
|
+
Navigates the **connected** tab to an absolute **http** or **https** URL (rejects `javascript:`, `file:`, etc. at the extension).
|
|
67
|
+
|
|
68
|
+
**inputSchema:**
|
|
69
|
+
|
|
70
|
+
```json
|
|
71
|
+
{
|
|
72
|
+
"type": "object",
|
|
73
|
+
"properties": {
|
|
74
|
+
"tabId": { "type": "integer", "description": "Optional; defaults to connected tab." },
|
|
75
|
+
"url": { "type": "string", "minLength": 1 }
|
|
76
|
+
},
|
|
77
|
+
"required": ["url"],
|
|
78
|
+
"additionalProperties": false
|
|
79
|
+
}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
**Result (informal):** `{ "navigated": true, "tabId": 1, "url": "https://...", "pendingUrl": "..." }`
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
### `browser_get_current_tab`
|
|
87
|
+
|
|
88
|
+
Returns metadata for the **connected** tab.
|
|
89
|
+
|
|
90
|
+
**inputSchema:**
|
|
91
|
+
|
|
92
|
+
```json
|
|
93
|
+
{
|
|
94
|
+
"type": "object",
|
|
95
|
+
"properties": {},
|
|
96
|
+
"additionalProperties": false
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
**Result (informal JSON):**
|
|
101
|
+
|
|
102
|
+
```json
|
|
103
|
+
{
|
|
104
|
+
"tabId": 1234567890,
|
|
105
|
+
"title": "Page title",
|
|
106
|
+
"url": "https://example.com/",
|
|
107
|
+
"windowId": 1,
|
|
108
|
+
"active": true
|
|
109
|
+
}
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
### `browser_get_tabs`
|
|
115
|
+
|
|
116
|
+
**Optional in v0.** If implemented, return **minimal** metadata only (no full DOM); consider omitting from first release per [MVP](./MVP.md).
|
|
117
|
+
|
|
118
|
+
**inputSchema:**
|
|
119
|
+
|
|
120
|
+
```json
|
|
121
|
+
{
|
|
122
|
+
"type": "object",
|
|
123
|
+
"properties": {
|
|
124
|
+
"activeOnly": {
|
|
125
|
+
"type": "boolean",
|
|
126
|
+
"description": "If true, only the active tab in the current window.",
|
|
127
|
+
"default": false
|
|
128
|
+
}
|
|
129
|
+
},
|
|
130
|
+
"additionalProperties": false
|
|
131
|
+
}
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
**Result:** `{ "tabs": [ { "tabId", "title", "url", "active" } ] }`
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
### `browser_get_page_text`
|
|
139
|
+
|
|
140
|
+
Extract **visible** text; server/extension must **truncate** to `maxChars`.
|
|
141
|
+
|
|
142
|
+
**inputSchema:**
|
|
143
|
+
|
|
144
|
+
```json
|
|
145
|
+
{
|
|
146
|
+
"type": "object",
|
|
147
|
+
"properties": {
|
|
148
|
+
"tabId": {
|
|
149
|
+
"type": "integer",
|
|
150
|
+
"description": "Target tab. Omit to use the connected tab."
|
|
151
|
+
},
|
|
152
|
+
"maxChars": {
|
|
153
|
+
"type": "integer",
|
|
154
|
+
"minimum": 1000,
|
|
155
|
+
"maximum": 500000,
|
|
156
|
+
"default": 50000,
|
|
157
|
+
"description": "Hard cap for returned text length."
|
|
158
|
+
}
|
|
159
|
+
},
|
|
160
|
+
"additionalProperties": false
|
|
161
|
+
}
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
**Result:** `{ "url", "text", "truncated": true|false }`
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
### `browser_get_page_structure`
|
|
169
|
+
|
|
170
|
+
Structured snapshot for modeling + stable refs for actions. Exact node shape is implementation-defined but should stay **accessibility-oriented** (role, name, hints) rather than dumping raw HTML.
|
|
171
|
+
|
|
172
|
+
**inputSchema:**
|
|
173
|
+
|
|
174
|
+
```json
|
|
175
|
+
{
|
|
176
|
+
"type": "object",
|
|
177
|
+
"properties": {
|
|
178
|
+
"tabId": { "type": "integer" },
|
|
179
|
+
"maxNodes": {
|
|
180
|
+
"type": "integer",
|
|
181
|
+
"minimum": 10,
|
|
182
|
+
"maximum": 5000,
|
|
183
|
+
"default": 500
|
|
184
|
+
},
|
|
185
|
+
"includeRects": {
|
|
186
|
+
"type": "boolean",
|
|
187
|
+
"default": true,
|
|
188
|
+
"description": "Include viewport-relative bounding boxes for disambiguation."
|
|
189
|
+
}
|
|
190
|
+
},
|
|
191
|
+
"additionalProperties": false
|
|
192
|
+
}
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
**Result (example shape):**
|
|
196
|
+
|
|
197
|
+
```json
|
|
198
|
+
{
|
|
199
|
+
"url": "https://example.com/",
|
|
200
|
+
"viewport": { "width": 1200, "height": 800, "scrollX": 0, "scrollY": 0 },
|
|
201
|
+
"nodes": [
|
|
202
|
+
{
|
|
203
|
+
"ref": "e12",
|
|
204
|
+
"role": "button",
|
|
205
|
+
"name": "Submit",
|
|
206
|
+
"tag": "button",
|
|
207
|
+
"hint": "Submit · primary",
|
|
208
|
+
"rect": { "x": 10, "y": 200, "w": 80, "h": 32 },
|
|
209
|
+
"disabled": false
|
|
210
|
+
}
|
|
211
|
+
]
|
|
212
|
+
}
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
---
|
|
216
|
+
|
|
217
|
+
### `browser_click_element`
|
|
218
|
+
|
|
219
|
+
Click element identified by **`ref`** from the latest structure snapshot.
|
|
220
|
+
|
|
221
|
+
**inputSchema:**
|
|
222
|
+
|
|
223
|
+
```json
|
|
224
|
+
{
|
|
225
|
+
"type": "object",
|
|
226
|
+
"properties": {
|
|
227
|
+
"tabId": { "type": "integer" },
|
|
228
|
+
"ref": { "type": "string", "minLength": 1 },
|
|
229
|
+
"button": {
|
|
230
|
+
"type": "string",
|
|
231
|
+
"enum": ["left", "middle", "right"],
|
|
232
|
+
"default": "left"
|
|
233
|
+
}
|
|
234
|
+
},
|
|
235
|
+
"required": ["ref"],
|
|
236
|
+
"additionalProperties": false
|
|
237
|
+
}
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
**Result:** `{ "clicked": true, "ref": "e12" }`
|
|
241
|
+
On ambiguous or stale ref: bridge returns `error` with `invalid_params` / `internal` as appropriate.
|
|
242
|
+
|
|
243
|
+
**Policy:** If classified as sensitive (e.g. role + name heuristics / form submit context), server must **not** forward** until `browser_ask_confirmation` succeeds—see [SAFETY](./SAFETY.md).
|
|
244
|
+
|
|
245
|
+
---
|
|
246
|
+
|
|
247
|
+
### `browser_fill_input`
|
|
248
|
+
|
|
249
|
+
Set value on an input/textarea/contenteditable identified by **`ref`**. **Does not submit** the form.
|
|
250
|
+
|
|
251
|
+
**inputSchema:**
|
|
252
|
+
|
|
253
|
+
```json
|
|
254
|
+
{
|
|
255
|
+
"type": "object",
|
|
256
|
+
"properties": {
|
|
257
|
+
"tabId": { "type": "integer" },
|
|
258
|
+
"ref": { "type": "string", "minLength": 1 },
|
|
259
|
+
"value": { "type": "string" },
|
|
260
|
+
"clearFirst": { "type": "boolean", "default": true }
|
|
261
|
+
},
|
|
262
|
+
"required": ["ref", "value"],
|
|
263
|
+
"additionalProperties": false
|
|
264
|
+
}
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
**Result:** `{ "filled": true, "ref": "e3" }`
|
|
268
|
+
|
|
269
|
+
---
|
|
270
|
+
|
|
271
|
+
### `browser_screenshot_visible`
|
|
272
|
+
|
|
273
|
+
Capture **viewport** screenshot of the connected (or specified) tab.
|
|
274
|
+
|
|
275
|
+
**inputSchema:**
|
|
276
|
+
|
|
277
|
+
```json
|
|
278
|
+
{
|
|
279
|
+
"type": "object",
|
|
280
|
+
"properties": {
|
|
281
|
+
"tabId": { "type": "integer" },
|
|
282
|
+
"format": {
|
|
283
|
+
"type": "string",
|
|
284
|
+
"enum": ["png", "jpeg"],
|
|
285
|
+
"default": "jpeg"
|
|
286
|
+
},
|
|
287
|
+
"quality": {
|
|
288
|
+
"type": "integer",
|
|
289
|
+
"minimum": 40,
|
|
290
|
+
"maximum": 95,
|
|
291
|
+
"default": 80,
|
|
292
|
+
"description": "JPEG quality only; ignored for png."
|
|
293
|
+
},
|
|
294
|
+
"maxWidth": {
|
|
295
|
+
"type": "integer",
|
|
296
|
+
"minimum": 200,
|
|
297
|
+
"maximum": 4096,
|
|
298
|
+
"default": 1280
|
|
299
|
+
}
|
|
300
|
+
},
|
|
301
|
+
"additionalProperties": false
|
|
302
|
+
}
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
**Result:** `{ "mimeType": "image/jpeg", "dataBase64": "..." }`
|
|
306
|
+
MCP servers often return images as resources or structured content depending on SDK—document your chosen mapping in the implementation README.
|
|
307
|
+
|
|
308
|
+
---
|
|
309
|
+
|
|
310
|
+
### `browser_get_selected_text`
|
|
311
|
+
|
|
312
|
+
**inputSchema:**
|
|
313
|
+
|
|
314
|
+
```json
|
|
315
|
+
{
|
|
316
|
+
"type": "object",
|
|
317
|
+
"properties": {
|
|
318
|
+
"tabId": { "type": "integer" }
|
|
319
|
+
},
|
|
320
|
+
"additionalProperties": false
|
|
321
|
+
}
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
**Result:** `{ "text": "...", "empty": false }`
|
|
325
|
+
|
|
326
|
+
---
|
|
327
|
+
|
|
328
|
+
### `browser_wait`
|
|
329
|
+
|
|
330
|
+
Bounded wait; keep semantics **explicit** to avoid agent spin-loops.
|
|
331
|
+
|
|
332
|
+
**inputSchema:**
|
|
333
|
+
|
|
334
|
+
```json
|
|
335
|
+
{
|
|
336
|
+
"type": "object",
|
|
337
|
+
"properties": {
|
|
338
|
+
"tabId": { "type": "integer" },
|
|
339
|
+
"ms": {
|
|
340
|
+
"type": "integer",
|
|
341
|
+
"minimum": 0,
|
|
342
|
+
"maximum": 30000,
|
|
343
|
+
"default": 500,
|
|
344
|
+
"description": "Sleep duration in milliseconds (extension-side timer)."
|
|
345
|
+
},
|
|
346
|
+
"until": {
|
|
347
|
+
"type": "string",
|
|
348
|
+
"enum": ["time", "navigation_idle"],
|
|
349
|
+
"default": "time",
|
|
350
|
+
"description": "time = simple delay; navigation_idle = optional best-effort idle heuristic if implemented."
|
|
351
|
+
}
|
|
352
|
+
},
|
|
353
|
+
"additionalProperties": false
|
|
354
|
+
}
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
**Result:** `{ "ok": true, "waitedMs": 500 }`
|
|
358
|
+
|
|
359
|
+
---
|
|
360
|
+
|
|
361
|
+
### `browser_ask_confirmation`
|
|
362
|
+
|
|
363
|
+
Human-in-the-loop gate for sensitive operations. Two-phase pattern:
|
|
364
|
+
|
|
365
|
+
1. Agent calls `browser_ask_confirmation` with **summary** + **risk** + optional `correlationId`.
|
|
366
|
+
2. Extension shows native UI; user approves or denies.
|
|
367
|
+
3. MCP returns outcome to the agent; subsequent **guarded** tool may require passing the returned **`approvalToken`** (implementation option).
|
|
368
|
+
|
|
369
|
+
**inputSchema:**
|
|
370
|
+
|
|
371
|
+
```json
|
|
372
|
+
{
|
|
373
|
+
"type": "object",
|
|
374
|
+
"properties": {
|
|
375
|
+
"title": { "type": "string", "maxLength": 120 },
|
|
376
|
+
"summary": { "type": "string", "maxLength": 4000 },
|
|
377
|
+
"risk": {
|
|
378
|
+
"type": "string",
|
|
379
|
+
"enum": ["low", "medium", "high"],
|
|
380
|
+
"description": "Agent-declared risk; server may upgrade but not downgrade."
|
|
381
|
+
},
|
|
382
|
+
"suggestedAction": {
|
|
383
|
+
"type": "string",
|
|
384
|
+
"description": "e.g. click Submit on example.com",
|
|
385
|
+
"maxLength": 500
|
|
386
|
+
},
|
|
387
|
+
"timeoutMs": {
|
|
388
|
+
"type": "integer",
|
|
389
|
+
"minimum": 5000,
|
|
390
|
+
"maximum": 600000,
|
|
391
|
+
"default": 120000
|
|
392
|
+
}
|
|
393
|
+
},
|
|
394
|
+
"required": ["title", "summary", "risk"],
|
|
395
|
+
"additionalProperties": false
|
|
396
|
+
}
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
**Result (approve):**
|
|
400
|
+
|
|
401
|
+
```json
|
|
402
|
+
{
|
|
403
|
+
"decision": "approved",
|
|
404
|
+
"approvalToken": "opaque",
|
|
405
|
+
"approvedAt": "2026-05-18T12:34:56.000Z"
|
|
406
|
+
}
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
**Result (deny/timeout):**
|
|
410
|
+
|
|
411
|
+
```json
|
|
412
|
+
{
|
|
413
|
+
"decision": "denied",
|
|
414
|
+
"reason": "user_rejected"
|
|
415
|
+
}
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
**Note:** If you add `approvalToken`, the server must **verify** it before forwarding a guarded `click_element` / `fill_input` that mutates sensitive UIs.
|
|
419
|
+
|
|
420
|
+
---
|
|
421
|
+
|
|
422
|
+
## Tools explicitly gated or omitted
|
|
423
|
+
|
|
424
|
+
- **`browser_evaluate`** — requires **`BROWSER_AUTOMATION_ALLOW_EVAL=1`** on the server process; still avoid untrusted expression strings.
|
|
425
|
+
- **Cookie/storage/header manipulation**, broad **debugger** attachment, and **dialog** override without user visibility remain out of scope for this connector.
|
|
426
|
+
|
|
427
|
+
---
|
|
428
|
+
|
|
429
|
+
## Evolution
|
|
430
|
+
|
|
431
|
+
- Add **`outputSchema`** when your SDK supports it; helps clients validate results.
|
|
432
|
+
- Add **`annotations`** (read-only hints, etc.) per MCP spec as tooling matures.
|
|
433
|
+
|
|
434
|
+
---
|
|
435
|
+
|
|
436
|
+
## DevTools (CDP) tools — extension 0.2.0+
|
|
437
|
+
|
|
438
|
+
These need the `debugger` permission (the extension adds it). While a tab is attached Chrome shows the "started debugging this browser" bar. The extension attaches per call and detaches again unless emulation or a recording is active.
|
|
439
|
+
|
|
440
|
+
| MCP tool | Bridge `method` | What it does |
|
|
441
|
+
|---|---|---|
|
|
442
|
+
| `browser_screenshot_visible` (`mode`, `fullPage`) | `screenshot_visible` | `auto` uses tab capture only for an active tab in a focused window, otherwise a DevTools capture that works hidden or covered. `fullPage` captures the whole scrollable page. |
|
|
443
|
+
| `browser_set_viewport` | `set_viewport` | Presets `mobile` 375x812, `mobile-small`, `tablet` 768x1024, `laptop`, `desktop`, or width and height. Touch and mobile user agent for mobile presets. |
|
|
444
|
+
| `browser_set_color_scheme` | `set_color_scheme` | Emulates `prefers-color-scheme` light, dark or none. |
|
|
445
|
+
| `browser_reset_emulation` | `reset_emulation` | Clears viewport, touch, user agent and color scheme. |
|
|
446
|
+
| `browser_mouse_move`, `browser_double_click`, `browser_right_click`, `browser_drag` | `pointer` | Real mouse events at a ref or at viewport x,y. Drag takes `toRef` or `toX`,`toY`, `steps`, `holdMs`. |
|
|
447
|
+
| `browser_clipboard_read`, `browser_clipboard_write` | `clipboard_read`, `clipboard_write` | System clipboard text via the page. |
|
|
448
|
+
| `browser_record_start`, `browser_record_stop` | `record_start`, `record_stop`, `record_fetch` | DevTools screencast. Stop writes frames and encodes an MP4 (and optional GIF) with ffmpeg to `~/.ar_browser/recordings/<name>/`. Frames come only when the page repaints; gaps are capped at 4 s. |
|
|
449
|
+
| `browser_window_list`, `browser_window_open`, `browser_window_resize`, `browser_window_focus` | `window_*` | Find the user's window, open a clean fixed-size window for recording, resize or focus. |
|
|
450
|
+
| `browser_downloads` | `get_downloads` | Now served by the extension through `chrome.downloads` (list and wait for in-progress files). |
|
|
451
|
+
|
|
452
|
+
Not supported by the managed (Playwright) driver. Requires `ffmpeg` on PATH for recordings.
|
|
453
|
+
|
|
454
|
+
### DevTools permission model (extension 0.9.0+)
|
|
455
|
+
|
|
456
|
+
Chrome does not allow `debugger` as an optional permission, so it is declared in the manifest. The extension gates it itself: it never attaches the debugger unless the user allowed DevTools features, and Chrome shows its "started debugging this browser" bar only while a call that needs it has the debugger attached.
|
|
457
|
+
|
|
458
|
+
- Default engine for viewport, mouse, clipboard and recording uses no debugger (page zoom, synthetic events, tab capture). Pass `engine: "cdp"` to force the DevTools version.
|
|
459
|
+
- Only these need DevTools: `browser_set_color_scheme`, `browser_screenshot_visible` with `fullPage`, `clip` or `mode: "cdp"`, and anything with `engine: "cdp"`.
|
|
460
|
+
- When one of them runs and DevTools features are not allowed, the extension opens a small window: **Allow for 10 minutes**, **Allow until I turn it off**, or **Not now**. The popup has a matching switch, and the choice is stored in the extension, not granted by Chrome.
|
|
461
|
+
- The popup has **Ask me when a tool needs it** (on by default). Unticked, such calls fail with `cdp_not_enabled` instead.
|
|
462
|
+
- A normal Chrome window cannot be narrower than about 516 px. A popup-type window can: `browser-agent window open <url> --phone` gives a real 375 px wide page with no zoom and no DevTools (desktop user agent, no touch).
|
|
463
|
+
- `browser_screenshot_visible` with `restoreWindow: true` briefly un-minimizes a minimized window to capture it.
|
|
464
|
+
|
|
465
|
+
### Several Chrome profiles (extension 0.5.0+)
|
|
466
|
+
|
|
467
|
+
Each Chrome profile with the extension connects as its own extension, named in the popup (**This Chrome profile is called**). A server now accepts many extensions at once.
|
|
468
|
+
|
|
469
|
+
- `browser_profiles` lists them and shows which one receives calls.
|
|
470
|
+
- `browser_use_profile {profile}` routes following calls to that profile (label, case-insensitive, or id prefix); an empty string goes back to the default, which is the earliest connected.
|
|
471
|
+
- Tab and window ids belong to one profile; call `browser_get_tabs` after switching.
|
|
472
|
+
- CLI: `browser-agent profiles`, `browser-agent use <label>`, or `--profile <label>` on any command.
|
|
473
|
+
- A reconnect from the same install (same instance id) replaces its old socket.
|
|
474
|
+
|
|
475
|
+
### Canvas apps and games: coordinates, regions, waiting (extension 0.7.1+)
|
|
476
|
+
|
|
477
|
+
| Tool | What it does |
|
|
478
|
+
|---|---|
|
|
479
|
+
| `browser_click_at` | Single click (left or right, optional hold) at x,y in page CSS pixels, or in the last screenshot's pixels with `onShot:true`. `audit:true` saves before/after screenshots and a JSON line under `~/.ar_browser/audit/<date>/`. |
|
|
480
|
+
| `browser_wheel` | Mouse wheel at x,y (no element needed). |
|
|
481
|
+
| `browser_wait_change` | Waits until the screen settles (default) or changes after an action; optional `region`. |
|
|
482
|
+
| `browser_screenshot_visible` | Now takes `region` (page pixels) and `scale` to read small text, and returns `imageScale` and `viewport` (CSS size, pixel ratio, scroll) so a position in the picture maps back to the page. |
|
|
483
|
+
|
|
484
|
+
- The default click and wheel are untrusted page events (no Chrome debugging bar). Some canvases ignore them. `engine:"cdp"` sends trusted input through DevTools; the user must allow DevTools access first (the Allow window, or the popup switch).
|
|
485
|
+
- The default profile for commands that name none is the one whose window the user focused last (extensions report focus), else the earliest connected.
|
|
486
|
+
- CLI: `click --x 400 --y 300 [--on-shot] [--right] [--hold ms] [--audit] [--engine cdp]`, `wheel --x --y --dy`, `wait-change [--change] [--region x,y,w,h]`, `shot --region x,y,w,h --scale 3`.
|