@agentruntime-labs/browser-agent 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (161) hide show
  1. package/LICENSE +202 -0
  2. package/NOTICE +6 -0
  3. package/README.md +64 -34
  4. package/bin/browser-agent.js +1 -0
  5. package/cli/control.mjs +477 -0
  6. package/cli/daemon.mjs +143 -0
  7. package/cli/hosts.mjs +1 -0
  8. package/cli/main.mjs +24 -0
  9. package/cli/mcp.mjs +1 -0
  10. package/docs/ARCHITECTURE.md +73 -0
  11. package/docs/DESKTOP_MCP.md +41 -0
  12. package/docs/EXTENSION_MANIFEST.md +163 -0
  13. package/docs/HYBRID.md +12 -0
  14. package/docs/MANAGED_BROWSER.md +111 -0
  15. package/docs/MCP_TOOLS.md +486 -0
  16. package/docs/MESSAGES.md +272 -0
  17. package/docs/MVP.md +76 -0
  18. package/docs/PRIVACY_POLICY.md +50 -0
  19. package/docs/README.md +27 -0
  20. package/docs/REFERENCE_PROJECTS.md +24 -0
  21. package/docs/RELEASE_CHECKLIST.md +49 -0
  22. package/docs/REPEATED_DOCS.md +27 -0
  23. package/docs/SAFETY.md +92 -0
  24. package/docs/SHELL_QUOTING.md +30 -0
  25. package/docs/SPLIT_REPOS.md +46 -0
  26. package/docs/THIRD_PARTY_LICENSES.md +182 -0
  27. package/docs/VISION.md +38 -0
  28. package/docs/WEB_STORE_SUBMISSION.md +117 -0
  29. package/extension/approval.js +1 -0
  30. package/extension/assets/agentruntime-logo.svg +77 -0
  31. package/extension/assets/agentruntime-mark-512.png +0 -0
  32. package/extension/background.js +776 -62
  33. package/extension/cdp.js +577 -0
  34. package/extension/constants.js +1 -0
  35. package/extension/content.js +69 -9
  36. package/extension/devtools-grant.html +31 -0
  37. package/extension/devtools-grant.js +29 -0
  38. package/extension/icons/icon128.png +0 -0
  39. package/extension/icons/icon16.png +0 -0
  40. package/extension/icons/icon32.png +0 -0
  41. package/extension/icons/icon48.png +0 -0
  42. package/extension/manifest.json +50 -23
  43. package/extension/policy.js +102 -0
  44. package/extension/popup.html +190 -56
  45. package/extension/popup.js +376 -166
  46. package/extension/soft.js +301 -0
  47. package/extension/welcome.html +91 -0
  48. package/extension/welcome.js +13 -0
  49. package/package.json +6 -2
  50. package/server/dist/arBrowserPaths.d.ts.map +1 -1
  51. package/server/dist/arBrowserPaths.js +1 -0
  52. package/server/dist/arBrowserPaths.js.map +1 -1
  53. package/server/dist/automationBackend.d.ts +46 -0
  54. package/server/dist/automationBackend.d.ts.map +1 -0
  55. package/server/dist/automationBackend.js +20 -0
  56. package/server/dist/automationBackend.js.map +1 -0
  57. package/server/dist/automationSession.d.ts.map +1 -1
  58. package/server/dist/automationSession.js +1 -0
  59. package/server/dist/automationSession.js.map +1 -1
  60. package/server/dist/bridge.d.ts +17 -4
  61. package/server/dist/bridge.d.ts.map +1 -1
  62. package/server/dist/bridge.js +132 -33
  63. package/server/dist/bridge.js.map +1 -1
  64. package/server/dist/browserToolsExtras.d.ts +2 -2
  65. package/server/dist/browserToolsExtras.d.ts.map +1 -1
  66. package/server/dist/browserToolsExtras.js +70 -1
  67. package/server/dist/browserToolsExtras.js.map +1 -1
  68. package/server/dist/cdpTools.d.ts +5 -0
  69. package/server/dist/cdpTools.d.ts.map +1 -0
  70. package/server/dist/cdpTools.js +411 -0
  71. package/server/dist/cdpTools.js.map +1 -0
  72. package/server/dist/compositeTools.d.ts +2 -2
  73. package/server/dist/compositeTools.d.ts.map +1 -1
  74. package/server/dist/compositeTools.js.map +1 -1
  75. package/server/dist/debugSnapshot.d.ts +2 -2
  76. package/server/dist/debugSnapshot.d.ts.map +1 -1
  77. package/server/dist/debugSnapshot.js.map +1 -1
  78. package/server/dist/guardrails.d.ts.map +1 -1
  79. package/server/dist/guardrails.js +1 -0
  80. package/server/dist/guardrails.js.map +1 -1
  81. package/server/dist/humanLike.d.ts.map +1 -1
  82. package/server/dist/humanLike.js +1 -0
  83. package/server/dist/humanLike.js.map +1 -1
  84. package/server/dist/main.js +56 -15
  85. package/server/dist/main.js.map +1 -1
  86. package/server/dist/managedBridge.d.ts +58 -0
  87. package/server/dist/managedBridge.d.ts.map +1 -0
  88. package/server/dist/managedBridge.js +619 -0
  89. package/server/dist/managedBridge.js.map +1 -0
  90. package/server/dist/managedDom.d.ts +58 -0
  91. package/server/dist/managedDom.d.ts.map +1 -0
  92. package/server/dist/managedDom.js +310 -0
  93. package/server/dist/managedDom.js.map +1 -0
  94. package/server/dist/policyTiers.d.ts.map +1 -1
  95. package/server/dist/policyTiers.js +3 -1
  96. package/server/dist/policyTiers.js.map +1 -1
  97. package/server/dist/protocol.d.ts +4 -0
  98. package/server/dist/protocol.d.ts.map +1 -1
  99. package/server/dist/protocol.js +1 -0
  100. package/server/dist/protocol.js.map +1 -1
  101. package/server/dist/recorder.d.ts +22 -0
  102. package/server/dist/recorder.d.ts.map +1 -0
  103. package/server/dist/recorder.js +82 -0
  104. package/server/dist/recorder.js.map +1 -0
  105. package/server/dist/richEditorFill.d.ts.map +1 -1
  106. package/server/dist/richEditorFill.js +1 -0
  107. package/server/dist/richEditorFill.js.map +1 -1
  108. package/server/dist/screenshotCrop.d.ts.map +1 -1
  109. package/server/dist/screenshotCrop.js +1 -0
  110. package/server/dist/screenshotCrop.js.map +1 -1
  111. package/server/dist/sessionLog.d.ts +1 -0
  112. package/server/dist/sessionLog.d.ts.map +1 -1
  113. package/server/dist/sessionLog.js +1 -0
  114. package/server/dist/sessionLog.js.map +1 -1
  115. package/server/dist/sessionPhase.d.ts.map +1 -1
  116. package/server/dist/sessionPhase.js.map +1 -1
  117. package/server/dist/sessionSummary.d.ts.map +1 -1
  118. package/server/dist/sessionSummary.js +1 -0
  119. package/server/dist/sessionSummary.js.map +1 -1
  120. package/server/dist/shotUtil.d.ts +44 -0
  121. package/server/dist/shotUtil.d.ts.map +1 -0
  122. package/server/dist/shotUtil.js +89 -0
  123. package/server/dist/shotUtil.js.map +1 -0
  124. package/server/dist/skillsInstructions.d.ts.map +1 -1
  125. package/server/dist/skillsInstructions.js +1 -0
  126. package/server/dist/skillsInstructions.js.map +1 -1
  127. package/server/dist/snapshotText.d.ts +47 -0
  128. package/server/dist/snapshotText.d.ts.map +1 -0
  129. package/server/dist/snapshotText.js +99 -0
  130. package/server/dist/snapshotText.js.map +1 -0
  131. package/server/dist/structureFilter.d.ts.map +1 -1
  132. package/server/dist/structureFilter.js.map +1 -1
  133. package/server/dist/surveyCatalog.d.ts.map +1 -1
  134. package/server/dist/surveyCatalog.js +1 -0
  135. package/server/dist/surveyCatalog.js.map +1 -1
  136. package/server/dist/surveyRunner.d.ts.map +1 -1
  137. package/server/dist/surveyRunner.js.map +1 -1
  138. package/server/dist/toolHelpers.d.ts +27 -2
  139. package/server/dist/toolHelpers.d.ts.map +1 -1
  140. package/server/dist/toolHelpers.js +67 -1
  141. package/server/dist/toolHelpers.js.map +1 -1
  142. package/server/dist/toolRuntime.d.ts +4 -0
  143. package/server/dist/toolRuntime.d.ts.map +1 -1
  144. package/server/dist/toolRuntime.js +12 -6
  145. package/server/dist/toolRuntime.js.map +1 -1
  146. package/server/dist/toolSchemas.d.ts +37 -1
  147. package/server/dist/toolSchemas.d.ts.map +1 -1
  148. package/server/dist/toolSchemas.js +33 -1
  149. package/server/dist/toolSchemas.js.map +1 -1
  150. package/server/dist/tools.d.ts +2 -2
  151. package/server/dist/tools.d.ts.map +1 -1
  152. package/server/dist/tools.js +65 -11
  153. package/server/dist/tools.js.map +1 -1
  154. package/server/dist/verdictExtract.d.ts.map +1 -1
  155. package/server/dist/verdictExtract.js +1 -0
  156. package/server/dist/verdictExtract.js.map +1 -1
  157. package/server/dist/workflowTools.d.ts +2 -2
  158. package/server/dist/workflowTools.d.ts.map +1 -1
  159. package/server/dist/workflowTools.js +16 -4
  160. package/server/dist/workflowTools.js.map +1 -1
  161. package/skills/browser-agent/SKILL.md +89 -0
@@ -0,0 +1,486 @@
1
+ # MCP tools (v0 schemas)
2
+
3
+ This document specifies **MCP tool definitions** exposed by the local stdio MCP server: `name`, `description`, and **`inputSchema`** (JSON Schema; MCP defaults to **JSON Schema 2020-12** when no `$schema` is set—see [MCP tools in the spec](https://modelcontextprotocol.io/specification/2024-11-05/server/tools)).
4
+
5
+ **Implementation:** The server also registers a Zod **`outputSchema`** per tool and returns **`structuredContent`** validated at runtime (`server/src/toolSchemas.ts`). Prefer consuming **`structuredContent`** in MCP clients when available; the `text` content mirrors it as JSON for convenience.
6
+
7
+ **Cursor:** Hosts like Cursor require MCP tool `name`s to be **letters, digits, and underscores only** (no dots). Names use the `browser_*` pattern, not `browser.*`.
8
+
9
+ **v0 rule:** No arbitrary `evaluate` unless the MCP host sets **`BROWSER_AUTOMATION_ALLOW_EVAL=1`**. Console and network tools are **metadata-only** (no response bodies). Tools operate on a **connected tab** unless explicitly noted; several DOM tools accept optional **`frameId`** for subframe targeting.
10
+
11
+ ## Bridge mapping
12
+
13
+ The MCP server maps each tool to a WebSocket **`request.method`** (short name) for the extension. Use this table as the single source of truth.
14
+
15
+ | MCP tool name | Bridge `method` |
16
+ |---------------|-----------------|
17
+ | `browser_navigate` | `navigate` |
18
+ | `browser_get_current_tab` | `get_current_tab` |
19
+ | `browser_get_tabs` | `get_tabs` |
20
+ | `browser_get_page_text` | `get_page_text` |
21
+ | `browser_get_page_structure` | `get_page_structure` |
22
+ | `browser_click_element` | `click_element` |
23
+ | `browser_fill_input` | `fill_input` |
24
+ | `browser_press_key` | `press_key` |
25
+ | `browser_scroll` | `scroll` |
26
+ | `browser_select_option` | `select_option` |
27
+ | `browser_hover` | `hover` |
28
+ | `browser_go_back` | `go_back` |
29
+ | `browser_reload` | `reload` |
30
+ | `browser_tab_open` | `tab_open` |
31
+ | `browser_tab_close` | `tab_close` |
32
+ | `browser_tab_activate` | `tab_activate` |
33
+ | `browser_screenshot_visible` | `screenshot_visible` |
34
+ | `browser_get_selected_text` | `get_selected_text` |
35
+ | `browser_wait` | `wait` |
36
+ | `browser_wait_for` | `wait_for` |
37
+ | `browser_ask_confirmation` | `ask_confirmation` |
38
+ | `browser_find_nodes` | *(server composes `get_page_structure`)* |
39
+ | `browser_screenshot_element` | `screenshot_element` |
40
+ | `browser_console_recent` | `console_recent` |
41
+ | `browser_network_recent` | `get_network_log` |
42
+ | `browser_set_file_input` | `set_file_input` |
43
+ | `browser_get_element_screen_rect` | `get_ref_screen_rect` |
44
+ | `browser_evaluate` | `evaluate_expression` |
45
+ | `browser_assert_url_contains` | *(server composes `get_current_tab`)* |
46
+ | `browser_assert_text_contains` | *(server composes `get_page_text`)* |
47
+
48
+ Parameters **`tabId`** and **`frameId`** (where supported) are passed through to the extension; omit **`tabId`** to use the user-connected tab from the popup.
49
+
50
+ ## Shared types (informal)
51
+
52
+ These appear inside `inputSchema` / results; document as conventions until you add `outputSchema`.
53
+
54
+ | Type | Meaning |
55
+ |------|--------|
56
+ | `TabId` | Chrome internal tab id (integer). |
57
+ | `ElementRef` | Opaque string from latest snapshot for **this** tab (invalid after major navigation unless implementation refreshes). |
58
+ | `Url` | string (absolute URL of loaded page). |
59
+
60
+ ---
61
+
62
+ ## Tool definitions
63
+
64
+ ### `browser_navigate`
65
+
66
+ Navigates the **connected** tab to an absolute **http** or **https** URL (rejects `javascript:`, `file:`, etc. at the extension).
67
+
68
+ **inputSchema:**
69
+
70
+ ```json
71
+ {
72
+ "type": "object",
73
+ "properties": {
74
+ "tabId": { "type": "integer", "description": "Optional; defaults to connected tab." },
75
+ "url": { "type": "string", "minLength": 1 }
76
+ },
77
+ "required": ["url"],
78
+ "additionalProperties": false
79
+ }
80
+ ```
81
+
82
+ **Result (informal):** `{ "navigated": true, "tabId": 1, "url": "https://...", "pendingUrl": "..." }`
83
+
84
+ ---
85
+
86
+ ### `browser_get_current_tab`
87
+
88
+ Returns metadata for the **connected** tab.
89
+
90
+ **inputSchema:**
91
+
92
+ ```json
93
+ {
94
+ "type": "object",
95
+ "properties": {},
96
+ "additionalProperties": false
97
+ }
98
+ ```
99
+
100
+ **Result (informal JSON):**
101
+
102
+ ```json
103
+ {
104
+ "tabId": 1234567890,
105
+ "title": "Page title",
106
+ "url": "https://example.com/",
107
+ "windowId": 1,
108
+ "active": true
109
+ }
110
+ ```
111
+
112
+ ---
113
+
114
+ ### `browser_get_tabs`
115
+
116
+ **Optional in v0.** If implemented, return **minimal** metadata only (no full DOM); consider omitting from first release per [MVP](./MVP.md).
117
+
118
+ **inputSchema:**
119
+
120
+ ```json
121
+ {
122
+ "type": "object",
123
+ "properties": {
124
+ "activeOnly": {
125
+ "type": "boolean",
126
+ "description": "If true, only the active tab in the current window.",
127
+ "default": false
128
+ }
129
+ },
130
+ "additionalProperties": false
131
+ }
132
+ ```
133
+
134
+ **Result:** `{ "tabs": [ { "tabId", "title", "url", "active" } ] }`
135
+
136
+ ---
137
+
138
+ ### `browser_get_page_text`
139
+
140
+ Extract **visible** text; server/extension must **truncate** to `maxChars`.
141
+
142
+ **inputSchema:**
143
+
144
+ ```json
145
+ {
146
+ "type": "object",
147
+ "properties": {
148
+ "tabId": {
149
+ "type": "integer",
150
+ "description": "Target tab. Omit to use the connected tab."
151
+ },
152
+ "maxChars": {
153
+ "type": "integer",
154
+ "minimum": 1000,
155
+ "maximum": 500000,
156
+ "default": 50000,
157
+ "description": "Hard cap for returned text length."
158
+ }
159
+ },
160
+ "additionalProperties": false
161
+ }
162
+ ```
163
+
164
+ **Result:** `{ "url", "text", "truncated": true|false }`
165
+
166
+ ---
167
+
168
+ ### `browser_get_page_structure`
169
+
170
+ Structured snapshot for modeling + stable refs for actions. Exact node shape is implementation-defined but should stay **accessibility-oriented** (role, name, hints) rather than dumping raw HTML.
171
+
172
+ **inputSchema:**
173
+
174
+ ```json
175
+ {
176
+ "type": "object",
177
+ "properties": {
178
+ "tabId": { "type": "integer" },
179
+ "maxNodes": {
180
+ "type": "integer",
181
+ "minimum": 10,
182
+ "maximum": 5000,
183
+ "default": 500
184
+ },
185
+ "includeRects": {
186
+ "type": "boolean",
187
+ "default": true,
188
+ "description": "Include viewport-relative bounding boxes for disambiguation."
189
+ }
190
+ },
191
+ "additionalProperties": false
192
+ }
193
+ ```
194
+
195
+ **Result (example shape):**
196
+
197
+ ```json
198
+ {
199
+ "url": "https://example.com/",
200
+ "viewport": { "width": 1200, "height": 800, "scrollX": 0, "scrollY": 0 },
201
+ "nodes": [
202
+ {
203
+ "ref": "e12",
204
+ "role": "button",
205
+ "name": "Submit",
206
+ "tag": "button",
207
+ "hint": "Submit · primary",
208
+ "rect": { "x": 10, "y": 200, "w": 80, "h": 32 },
209
+ "disabled": false
210
+ }
211
+ ]
212
+ }
213
+ ```
214
+
215
+ ---
216
+
217
+ ### `browser_click_element`
218
+
219
+ Click element identified by **`ref`** from the latest structure snapshot.
220
+
221
+ **inputSchema:**
222
+
223
+ ```json
224
+ {
225
+ "type": "object",
226
+ "properties": {
227
+ "tabId": { "type": "integer" },
228
+ "ref": { "type": "string", "minLength": 1 },
229
+ "button": {
230
+ "type": "string",
231
+ "enum": ["left", "middle", "right"],
232
+ "default": "left"
233
+ }
234
+ },
235
+ "required": ["ref"],
236
+ "additionalProperties": false
237
+ }
238
+ ```
239
+
240
+ **Result:** `{ "clicked": true, "ref": "e12" }`
241
+ On ambiguous or stale ref: bridge returns `error` with `invalid_params` / `internal` as appropriate.
242
+
243
+ **Policy:** If classified as sensitive (e.g. role + name heuristics / form submit context), server must **not** forward** until `browser_ask_confirmation` succeeds—see [SAFETY](./SAFETY.md).
244
+
245
+ ---
246
+
247
+ ### `browser_fill_input`
248
+
249
+ Set value on an input/textarea/contenteditable identified by **`ref`**. **Does not submit** the form.
250
+
251
+ **inputSchema:**
252
+
253
+ ```json
254
+ {
255
+ "type": "object",
256
+ "properties": {
257
+ "tabId": { "type": "integer" },
258
+ "ref": { "type": "string", "minLength": 1 },
259
+ "value": { "type": "string" },
260
+ "clearFirst": { "type": "boolean", "default": true }
261
+ },
262
+ "required": ["ref", "value"],
263
+ "additionalProperties": false
264
+ }
265
+ ```
266
+
267
+ **Result:** `{ "filled": true, "ref": "e3" }`
268
+
269
+ ---
270
+
271
+ ### `browser_screenshot_visible`
272
+
273
+ Capture **viewport** screenshot of the connected (or specified) tab.
274
+
275
+ **inputSchema:**
276
+
277
+ ```json
278
+ {
279
+ "type": "object",
280
+ "properties": {
281
+ "tabId": { "type": "integer" },
282
+ "format": {
283
+ "type": "string",
284
+ "enum": ["png", "jpeg"],
285
+ "default": "jpeg"
286
+ },
287
+ "quality": {
288
+ "type": "integer",
289
+ "minimum": 40,
290
+ "maximum": 95,
291
+ "default": 80,
292
+ "description": "JPEG quality only; ignored for png."
293
+ },
294
+ "maxWidth": {
295
+ "type": "integer",
296
+ "minimum": 200,
297
+ "maximum": 4096,
298
+ "default": 1280
299
+ }
300
+ },
301
+ "additionalProperties": false
302
+ }
303
+ ```
304
+
305
+ **Result:** `{ "mimeType": "image/jpeg", "dataBase64": "..." }`
306
+ MCP servers often return images as resources or structured content depending on SDK—document your chosen mapping in the implementation README.
307
+
308
+ ---
309
+
310
+ ### `browser_get_selected_text`
311
+
312
+ **inputSchema:**
313
+
314
+ ```json
315
+ {
316
+ "type": "object",
317
+ "properties": {
318
+ "tabId": { "type": "integer" }
319
+ },
320
+ "additionalProperties": false
321
+ }
322
+ ```
323
+
324
+ **Result:** `{ "text": "...", "empty": false }`
325
+
326
+ ---
327
+
328
+ ### `browser_wait`
329
+
330
+ Bounded wait; keep semantics **explicit** to avoid agent spin-loops.
331
+
332
+ **inputSchema:**
333
+
334
+ ```json
335
+ {
336
+ "type": "object",
337
+ "properties": {
338
+ "tabId": { "type": "integer" },
339
+ "ms": {
340
+ "type": "integer",
341
+ "minimum": 0,
342
+ "maximum": 30000,
343
+ "default": 500,
344
+ "description": "Sleep duration in milliseconds (extension-side timer)."
345
+ },
346
+ "until": {
347
+ "type": "string",
348
+ "enum": ["time", "navigation_idle"],
349
+ "default": "time",
350
+ "description": "time = simple delay; navigation_idle = optional best-effort idle heuristic if implemented."
351
+ }
352
+ },
353
+ "additionalProperties": false
354
+ }
355
+ ```
356
+
357
+ **Result:** `{ "ok": true, "waitedMs": 500 }`
358
+
359
+ ---
360
+
361
+ ### `browser_ask_confirmation`
362
+
363
+ Human-in-the-loop gate for sensitive operations. Two-phase pattern:
364
+
365
+ 1. Agent calls `browser_ask_confirmation` with **summary** + **risk** + optional `correlationId`.
366
+ 2. Extension shows native UI; user approves or denies.
367
+ 3. MCP returns outcome to the agent; subsequent **guarded** tool may require passing the returned **`approvalToken`** (implementation option).
368
+
369
+ **inputSchema:**
370
+
371
+ ```json
372
+ {
373
+ "type": "object",
374
+ "properties": {
375
+ "title": { "type": "string", "maxLength": 120 },
376
+ "summary": { "type": "string", "maxLength": 4000 },
377
+ "risk": {
378
+ "type": "string",
379
+ "enum": ["low", "medium", "high"],
380
+ "description": "Agent-declared risk; server may upgrade but not downgrade."
381
+ },
382
+ "suggestedAction": {
383
+ "type": "string",
384
+ "description": "e.g. click Submit on example.com",
385
+ "maxLength": 500
386
+ },
387
+ "timeoutMs": {
388
+ "type": "integer",
389
+ "minimum": 5000,
390
+ "maximum": 600000,
391
+ "default": 120000
392
+ }
393
+ },
394
+ "required": ["title", "summary", "risk"],
395
+ "additionalProperties": false
396
+ }
397
+ ```
398
+
399
+ **Result (approve):**
400
+
401
+ ```json
402
+ {
403
+ "decision": "approved",
404
+ "approvalToken": "opaque",
405
+ "approvedAt": "2026-05-18T12:34:56.000Z"
406
+ }
407
+ ```
408
+
409
+ **Result (deny/timeout):**
410
+
411
+ ```json
412
+ {
413
+ "decision": "denied",
414
+ "reason": "user_rejected"
415
+ }
416
+ ```
417
+
418
+ **Note:** If you add `approvalToken`, the server must **verify** it before forwarding a guarded `click_element` / `fill_input` that mutates sensitive UIs.
419
+
420
+ ---
421
+
422
+ ## Tools explicitly gated or omitted
423
+
424
+ - **`browser_evaluate`** — requires **`BROWSER_AUTOMATION_ALLOW_EVAL=1`** on the server process; still avoid untrusted expression strings.
425
+ - **Cookie/storage/header manipulation**, broad **debugger** attachment, and **dialog** override without user visibility remain out of scope for this connector.
426
+
427
+ ---
428
+
429
+ ## Evolution
430
+
431
+ - Add **`outputSchema`** when your SDK supports it; helps clients validate results.
432
+ - Add **`annotations`** (read-only hints, etc.) per MCP spec as tooling matures.
433
+
434
+ ---
435
+
436
+ ## DevTools (CDP) tools — extension 0.2.0+
437
+
438
+ These need the `debugger` permission (the extension adds it). While a tab is attached Chrome shows the "started debugging this browser" bar. The extension attaches per call and detaches again unless emulation or a recording is active.
439
+
440
+ | MCP tool | Bridge `method` | What it does |
441
+ |---|---|---|
442
+ | `browser_screenshot_visible` (`mode`, `fullPage`) | `screenshot_visible` | `auto` uses tab capture only for an active tab in a focused window, otherwise a DevTools capture that works hidden or covered. `fullPage` captures the whole scrollable page. |
443
+ | `browser_set_viewport` | `set_viewport` | Presets `mobile` 375x812, `mobile-small`, `tablet` 768x1024, `laptop`, `desktop`, or width and height. Touch and mobile user agent for mobile presets. |
444
+ | `browser_set_color_scheme` | `set_color_scheme` | Emulates `prefers-color-scheme` light, dark or none. |
445
+ | `browser_reset_emulation` | `reset_emulation` | Clears viewport, touch, user agent and color scheme. |
446
+ | `browser_mouse_move`, `browser_double_click`, `browser_right_click`, `browser_drag` | `pointer` | Real mouse events at a ref or at viewport x,y. Drag takes `toRef` or `toX`,`toY`, `steps`, `holdMs`. |
447
+ | `browser_clipboard_read`, `browser_clipboard_write` | `clipboard_read`, `clipboard_write` | System clipboard text via the page. |
448
+ | `browser_record_start`, `browser_record_stop` | `record_start`, `record_stop`, `record_fetch` | DevTools screencast. Stop writes frames and encodes an MP4 (and optional GIF) with ffmpeg to `~/.ar_browser/recordings/<name>/`. Frames come only when the page repaints; gaps are capped at 4 s. |
449
+ | `browser_window_list`, `browser_window_open`, `browser_window_resize`, `browser_window_focus` | `window_*` | Find the user's window, open a clean fixed-size window for recording, resize or focus. |
450
+ | `browser_downloads` | `get_downloads` | Now served by the extension through `chrome.downloads` (list and wait for in-progress files). |
451
+
452
+ Not supported by the managed (Playwright) driver. Requires `ffmpeg` on PATH for recordings.
453
+
454
+ ### DevTools permission model (extension 0.9.0+)
455
+
456
+ Chrome does not allow `debugger` as an optional permission, so it is declared in the manifest. The extension gates it itself: it never attaches the debugger unless the user allowed DevTools features, and Chrome shows its "started debugging this browser" bar only while a call that needs it has the debugger attached.
457
+
458
+ - Default engine for viewport, mouse, clipboard and recording uses no debugger (page zoom, synthetic events, tab capture). Pass `engine: "cdp"` to force the DevTools version.
459
+ - Only these need DevTools: `browser_set_color_scheme`, `browser_screenshot_visible` with `fullPage`, `clip` or `mode: "cdp"`, and anything with `engine: "cdp"`.
460
+ - When one of them runs and DevTools features are not allowed, the extension opens a small window: **Allow for 10 minutes**, **Allow until I turn it off**, or **Not now**. The popup has a matching switch, and the choice is stored in the extension, not granted by Chrome.
461
+ - The popup has **Ask me when a tool needs it** (on by default). Unticked, such calls fail with `cdp_not_enabled` instead.
462
+ - A normal Chrome window cannot be narrower than about 516 px. A popup-type window can: `browser-agent window open <url> --phone` gives a real 375 px wide page with no zoom and no DevTools (desktop user agent, no touch).
463
+ - `browser_screenshot_visible` with `restoreWindow: true` briefly un-minimizes a minimized window to capture it.
464
+
465
+ ### Several Chrome profiles (extension 0.5.0+)
466
+
467
+ Each Chrome profile with the extension connects as its own extension, named in the popup (**This Chrome profile is called**). A server now accepts many extensions at once.
468
+
469
+ - `browser_profiles` lists them and shows which one receives calls.
470
+ - `browser_use_profile {profile}` routes following calls to that profile (label, case-insensitive, or id prefix); an empty string goes back to the default, which is the earliest connected.
471
+ - Tab and window ids belong to one profile; call `browser_get_tabs` after switching.
472
+ - CLI: `browser-agent profiles`, `browser-agent use <label>`, or `--profile <label>` on any command.
473
+ - A reconnect from the same install (same instance id) replaces its old socket.
474
+
475
+ ### Canvas apps and games: coordinates, regions, waiting (extension 0.7.1+)
476
+
477
+ | Tool | What it does |
478
+ |---|---|
479
+ | `browser_click_at` | Single click (left or right, optional hold) at x,y in page CSS pixels, or in the last screenshot's pixels with `onShot:true`. `audit:true` saves before/after screenshots and a JSON line under `~/.ar_browser/audit/<date>/`. |
480
+ | `browser_wheel` | Mouse wheel at x,y (no element needed). |
481
+ | `browser_wait_change` | Waits until the screen settles (default) or changes after an action; optional `region`. |
482
+ | `browser_screenshot_visible` | Now takes `region` (page pixels) and `scale` to read small text, and returns `imageScale` and `viewport` (CSS size, pixel ratio, scroll) so a position in the picture maps back to the page. |
483
+
484
+ - The default click and wheel are untrusted page events (no Chrome debugging bar). Some canvases ignore them. `engine:"cdp"` sends trusted input through DevTools; the user must allow DevTools access first (the Allow window, or the popup switch).
485
+ - The default profile for commands that name none is the one whose window the user focused last (extensions report focus), else the earliest connected.
486
+ - CLI: `click --x 400 --y 300 [--on-shot] [--right] [--hold ms] [--audit] [--engine cdp]`, `wheel --x --y --dy`, `wait-change [--change] [--region x,y,w,h]`, `shot --region x,y,w,h --scale 3`.