@agentruntime-labs/browser-agent 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (161) hide show
  1. package/LICENSE +202 -0
  2. package/NOTICE +6 -0
  3. package/README.md +64 -34
  4. package/bin/browser-agent.js +1 -0
  5. package/cli/control.mjs +477 -0
  6. package/cli/daemon.mjs +143 -0
  7. package/cli/hosts.mjs +1 -0
  8. package/cli/main.mjs +24 -0
  9. package/cli/mcp.mjs +1 -0
  10. package/docs/ARCHITECTURE.md +73 -0
  11. package/docs/DESKTOP_MCP.md +41 -0
  12. package/docs/EXTENSION_MANIFEST.md +163 -0
  13. package/docs/HYBRID.md +12 -0
  14. package/docs/MANAGED_BROWSER.md +111 -0
  15. package/docs/MCP_TOOLS.md +486 -0
  16. package/docs/MESSAGES.md +272 -0
  17. package/docs/MVP.md +76 -0
  18. package/docs/PRIVACY_POLICY.md +50 -0
  19. package/docs/README.md +27 -0
  20. package/docs/REFERENCE_PROJECTS.md +24 -0
  21. package/docs/RELEASE_CHECKLIST.md +49 -0
  22. package/docs/REPEATED_DOCS.md +27 -0
  23. package/docs/SAFETY.md +92 -0
  24. package/docs/SHELL_QUOTING.md +30 -0
  25. package/docs/SPLIT_REPOS.md +46 -0
  26. package/docs/THIRD_PARTY_LICENSES.md +182 -0
  27. package/docs/VISION.md +38 -0
  28. package/docs/WEB_STORE_SUBMISSION.md +117 -0
  29. package/extension/approval.js +1 -0
  30. package/extension/assets/agentruntime-logo.svg +77 -0
  31. package/extension/assets/agentruntime-mark-512.png +0 -0
  32. package/extension/background.js +776 -62
  33. package/extension/cdp.js +577 -0
  34. package/extension/constants.js +1 -0
  35. package/extension/content.js +69 -9
  36. package/extension/devtools-grant.html +31 -0
  37. package/extension/devtools-grant.js +29 -0
  38. package/extension/icons/icon128.png +0 -0
  39. package/extension/icons/icon16.png +0 -0
  40. package/extension/icons/icon32.png +0 -0
  41. package/extension/icons/icon48.png +0 -0
  42. package/extension/manifest.json +50 -23
  43. package/extension/policy.js +102 -0
  44. package/extension/popup.html +190 -56
  45. package/extension/popup.js +376 -166
  46. package/extension/soft.js +301 -0
  47. package/extension/welcome.html +91 -0
  48. package/extension/welcome.js +13 -0
  49. package/package.json +6 -2
  50. package/server/dist/arBrowserPaths.d.ts.map +1 -1
  51. package/server/dist/arBrowserPaths.js +1 -0
  52. package/server/dist/arBrowserPaths.js.map +1 -1
  53. package/server/dist/automationBackend.d.ts +46 -0
  54. package/server/dist/automationBackend.d.ts.map +1 -0
  55. package/server/dist/automationBackend.js +20 -0
  56. package/server/dist/automationBackend.js.map +1 -0
  57. package/server/dist/automationSession.d.ts.map +1 -1
  58. package/server/dist/automationSession.js +1 -0
  59. package/server/dist/automationSession.js.map +1 -1
  60. package/server/dist/bridge.d.ts +17 -4
  61. package/server/dist/bridge.d.ts.map +1 -1
  62. package/server/dist/bridge.js +132 -33
  63. package/server/dist/bridge.js.map +1 -1
  64. package/server/dist/browserToolsExtras.d.ts +2 -2
  65. package/server/dist/browserToolsExtras.d.ts.map +1 -1
  66. package/server/dist/browserToolsExtras.js +70 -1
  67. package/server/dist/browserToolsExtras.js.map +1 -1
  68. package/server/dist/cdpTools.d.ts +5 -0
  69. package/server/dist/cdpTools.d.ts.map +1 -0
  70. package/server/dist/cdpTools.js +411 -0
  71. package/server/dist/cdpTools.js.map +1 -0
  72. package/server/dist/compositeTools.d.ts +2 -2
  73. package/server/dist/compositeTools.d.ts.map +1 -1
  74. package/server/dist/compositeTools.js.map +1 -1
  75. package/server/dist/debugSnapshot.d.ts +2 -2
  76. package/server/dist/debugSnapshot.d.ts.map +1 -1
  77. package/server/dist/debugSnapshot.js.map +1 -1
  78. package/server/dist/guardrails.d.ts.map +1 -1
  79. package/server/dist/guardrails.js +1 -0
  80. package/server/dist/guardrails.js.map +1 -1
  81. package/server/dist/humanLike.d.ts.map +1 -1
  82. package/server/dist/humanLike.js +1 -0
  83. package/server/dist/humanLike.js.map +1 -1
  84. package/server/dist/main.js +56 -15
  85. package/server/dist/main.js.map +1 -1
  86. package/server/dist/managedBridge.d.ts +58 -0
  87. package/server/dist/managedBridge.d.ts.map +1 -0
  88. package/server/dist/managedBridge.js +619 -0
  89. package/server/dist/managedBridge.js.map +1 -0
  90. package/server/dist/managedDom.d.ts +58 -0
  91. package/server/dist/managedDom.d.ts.map +1 -0
  92. package/server/dist/managedDom.js +310 -0
  93. package/server/dist/managedDom.js.map +1 -0
  94. package/server/dist/policyTiers.d.ts.map +1 -1
  95. package/server/dist/policyTiers.js +3 -1
  96. package/server/dist/policyTiers.js.map +1 -1
  97. package/server/dist/protocol.d.ts +4 -0
  98. package/server/dist/protocol.d.ts.map +1 -1
  99. package/server/dist/protocol.js +1 -0
  100. package/server/dist/protocol.js.map +1 -1
  101. package/server/dist/recorder.d.ts +22 -0
  102. package/server/dist/recorder.d.ts.map +1 -0
  103. package/server/dist/recorder.js +82 -0
  104. package/server/dist/recorder.js.map +1 -0
  105. package/server/dist/richEditorFill.d.ts.map +1 -1
  106. package/server/dist/richEditorFill.js +1 -0
  107. package/server/dist/richEditorFill.js.map +1 -1
  108. package/server/dist/screenshotCrop.d.ts.map +1 -1
  109. package/server/dist/screenshotCrop.js +1 -0
  110. package/server/dist/screenshotCrop.js.map +1 -1
  111. package/server/dist/sessionLog.d.ts +1 -0
  112. package/server/dist/sessionLog.d.ts.map +1 -1
  113. package/server/dist/sessionLog.js +1 -0
  114. package/server/dist/sessionLog.js.map +1 -1
  115. package/server/dist/sessionPhase.d.ts.map +1 -1
  116. package/server/dist/sessionPhase.js.map +1 -1
  117. package/server/dist/sessionSummary.d.ts.map +1 -1
  118. package/server/dist/sessionSummary.js +1 -0
  119. package/server/dist/sessionSummary.js.map +1 -1
  120. package/server/dist/shotUtil.d.ts +44 -0
  121. package/server/dist/shotUtil.d.ts.map +1 -0
  122. package/server/dist/shotUtil.js +89 -0
  123. package/server/dist/shotUtil.js.map +1 -0
  124. package/server/dist/skillsInstructions.d.ts.map +1 -1
  125. package/server/dist/skillsInstructions.js +1 -0
  126. package/server/dist/skillsInstructions.js.map +1 -1
  127. package/server/dist/snapshotText.d.ts +47 -0
  128. package/server/dist/snapshotText.d.ts.map +1 -0
  129. package/server/dist/snapshotText.js +99 -0
  130. package/server/dist/snapshotText.js.map +1 -0
  131. package/server/dist/structureFilter.d.ts.map +1 -1
  132. package/server/dist/structureFilter.js.map +1 -1
  133. package/server/dist/surveyCatalog.d.ts.map +1 -1
  134. package/server/dist/surveyCatalog.js +1 -0
  135. package/server/dist/surveyCatalog.js.map +1 -1
  136. package/server/dist/surveyRunner.d.ts.map +1 -1
  137. package/server/dist/surveyRunner.js.map +1 -1
  138. package/server/dist/toolHelpers.d.ts +27 -2
  139. package/server/dist/toolHelpers.d.ts.map +1 -1
  140. package/server/dist/toolHelpers.js +67 -1
  141. package/server/dist/toolHelpers.js.map +1 -1
  142. package/server/dist/toolRuntime.d.ts +4 -0
  143. package/server/dist/toolRuntime.d.ts.map +1 -1
  144. package/server/dist/toolRuntime.js +12 -6
  145. package/server/dist/toolRuntime.js.map +1 -1
  146. package/server/dist/toolSchemas.d.ts +37 -1
  147. package/server/dist/toolSchemas.d.ts.map +1 -1
  148. package/server/dist/toolSchemas.js +33 -1
  149. package/server/dist/toolSchemas.js.map +1 -1
  150. package/server/dist/tools.d.ts +2 -2
  151. package/server/dist/tools.d.ts.map +1 -1
  152. package/server/dist/tools.js +65 -11
  153. package/server/dist/tools.js.map +1 -1
  154. package/server/dist/verdictExtract.d.ts.map +1 -1
  155. package/server/dist/verdictExtract.js +1 -0
  156. package/server/dist/verdictExtract.js.map +1 -1
  157. package/server/dist/workflowTools.d.ts +2 -2
  158. package/server/dist/workflowTools.d.ts.map +1 -1
  159. package/server/dist/workflowTools.js +16 -4
  160. package/server/dist/workflowTools.js.map +1 -1
  161. package/skills/browser-agent/SKILL.md +89 -0
@@ -0,0 +1,73 @@
1
+ # Architecture
2
+
3
+ ## High-level diagram
4
+
5
+ ```mermaid
6
+ flowchart LR
7
+ subgraph client
8
+ Agent[MCP client e.g. Cursor / AgentRuntime]
9
+ end
10
+ subgraph local
11
+ MCP[Local MCP server stdio]
12
+ BR[Session / policy / logs]
13
+ WS[WebSocket server 127.0.0.1]
14
+ MCP --- BR
15
+ MCP --- WS
16
+ end
17
+ subgraph browser
18
+ SW[MV3 service worker]
19
+ CS[Isolated content scripts]
20
+ Page[Web page DOM]
21
+ SW <--> WS
22
+ SW --> CS
23
+ CS --> Page
24
+ end
25
+ Agent <-- stdio --> MCP
26
+ ```
27
+
28
+ ## Components
29
+
30
+ ### 1. Browser extension (MV3)
31
+
32
+ | Piece | Responsibility |
33
+ |-------|------------------|
34
+ | **Service worker** | WebSocket client to localhost; message routing; `tabs` / `scripting` orchestration; optional “connected tab” state. |
35
+ | **Content script(s)** | Read structured access to the **active** document: visible text bounds, interactive elements, selected text; perform **user-consented** UI actions. |
36
+ | **UI** | Minimal: connect/disconnect, show linked port/session, optional action log—can grow later (side panel). |
37
+
38
+ **Isolation:** Prefer **isolated** content scripts; avoid MAIN-world injection as a default (see [Safety](./SAFETY.md)).
39
+
40
+ ### 2. Localhost bridge
41
+
42
+ | Option | Pros | Cons |
43
+ |--------|------|------|
44
+ | **WebSocket (`ws://127.0.0.1`)** | Fast iteration; simple framing (JSON-RPC / envelope). | Must authenticate/session-bind the socket so random local processes cannot drive the extension. |
45
+ | **Native messaging** | Stronger process boundary vs random localhost peers. | More packaging friction; host app install. |
46
+
47
+ **MVP default:** WebSocket with a **session token** established when the user clicks “Connect” in the extension and/or server prints a one-time secret.
48
+
49
+ **Spec:** Wire format, pairing, and error codes are defined in [MESSAGES](./MESSAGES.md). MCP-facing tool schemas and bridge method names are in [MCP_TOOLS](./MCP_TOOLS.md).
50
+
51
+ ### 3. MCP server (local)
52
+
53
+ - Implements **stdio MCP** for the host agent.
54
+ - Holds **policy**: allowed origins, connected tab id, confirmation rules.
55
+ - Logs requests/responses **locally** (redact secrets; configurable verbosity).
56
+ - Maps MCP tools → bridge messages → extension responses.
57
+
58
+ Language choice (Go, Rust, Node, etc.) is an implementation detail; the **contract** is MCP in, WebSocket out.
59
+
60
+ ## Message flow (typical)
61
+
62
+ 1. User connects extension to server (explicit action).
63
+ 2. Agent calls MCP tool, e.g. `browser_get_page_structure` / `browser_click_element`.
64
+ 3. Server validates policy (tab, origin, sensitivity).
65
+ 4. Server sends bridge message to extension.
66
+ 5. Content script executes action **in-page**; returns structured result + optional screenshot metadata.
67
+ 6. Server returns MCP result to agent.
68
+
69
+ ## What we deliberately do *not* depend on
70
+
71
+ - Remote debugging port open on the user’s daily browser
72
+ - Launching a second browser binary as the “real” session
73
+ - CDP as the **primary** control plane for v0 (optional future for **debug** tools only, gated)
@@ -0,0 +1,41 @@
1
+ > **Repeated doc (repo independence).** Canonical copy: [desktop-agent `docs/DESKTOP_MCP.md`](https://github.com/agentruntime-io/desktop-agent/blob/main/docs/DESKTOP_MCP.md). Prefer editing the canonical repo, then `split/assemble-repos.ps1`; or edit both copies deliberately.
2
+
3
+ # Desktop MCP (sibling server)
4
+
5
+ Win32 stdio MCP: **real** mouse, keyboard, clipboard, screenshots. Repo: [desktop-agent](https://github.com/agentruntime-io/desktop-agent).
6
+
7
+ ## Requirements
8
+
9
+ - Windows 10/11, Node 18+
10
+ - `DESKTOP_AUTOMATION_ENABLED=1` in MCP env (mutating tools off by default)
11
+
12
+ ## Entrypoint
13
+
14
+ ```json
15
+ "desktop-automation": {
16
+ "command": "node",
17
+ "args": ["${workspaceFolder}/dist/main.js"],
18
+ "env": {
19
+ "DESKTOP_AUTOMATION_ENABLED": "1"
20
+ }
21
+ }
22
+ ```
23
+
24
+ (Path is for **desktop-agent** clone root; browser-agent uses only the browser server in its `.cursor/mcp.json`.)
25
+
26
+ ## Tools (summary)
27
+
28
+ | Tool | Role |
29
+ |------|------|
30
+ | `desktop_session_open` / `desktop_session_resume` | Session ids |
31
+ | `desktop_screenshot`, `desktop_resolve_grid_cell` | Screen + grid targeting |
32
+ | `desktop_probe_click`, `desktop_click`, `desktop_drag`, `desktop_scroll` | Pointer |
33
+ | `desktop_type_text`, `desktop_press_key` | Keyboard |
34
+ | `desktop_focus_window`, `desktop_list_windows` | Window focus |
35
+ | `desktop_clipboard_*` | Clipboard path imports |
36
+
37
+ Full table, env vars, and native-app recipes: **`README.md`** at this repo root.
38
+
39
+ ## When to use with browser-agent
40
+
41
+ See **`docs/HYBRID.md`**.
@@ -0,0 +1,163 @@
1
+ # Extension manifest (MV3)
2
+
3
+ This document records **Chrome Manifest V3** choices for the connector extension: required **permissions**, **host access**, and **content scripts**. It should stay aligned with [SAFETY](./SAFETY.md) (minimize scope) and [MVP](./MVP.md) (connected tab).
4
+
5
+ **References:**
6
+
7
+ - [Declare permissions](https://developer.chrome.com/docs/extensions/develop/concepts/declare-permissions)
8
+ - [Match patterns](https://developer.chrome.com/docs/extensions/develop/concepts/match-patterns)
9
+ - [Content scripts](https://developer.chrome.com/docs/extensions/develop/concepts/content-scripts)
10
+ - [chrome.scripting](https://developer.chrome.com/docs/extensions/reference/api/scripting)
11
+ - [chrome.tabs](https://developer.chrome.com/docs/extensions/reference/api/tabs)
12
+
13
+ ---
14
+
15
+ ## Principles
16
+
17
+ 1. **Default least privilege** — Prefer `activeTab` + explicit user gestures over `<all_urls>`.
18
+ 2. **Connected tab** — Automation targets **one** tab id at a time; reject bridge requests for other tabs until user reconnects.
19
+ 3. **MAIN-world injection** — Isolated content scripts are the default. **`browser_console_recent`** and **`browser_evaluate`** inject short MAIN-world hooks only for diagnostics / opt-in eval; keep eval off unless `BROWSER_AUTOMATION_ALLOW_EVAL=1`.
20
+ 4. **Justify every permission** — If you add one, append a row here.
21
+
22
+ ---
23
+
24
+ ## Baseline `manifest.json` (template)
25
+
26
+ > **Note:** Version numbers and asset paths are placeholders until code exists.
27
+
28
+ ```json
29
+ {
30
+ "manifest_version": 3,
31
+ "name": "AgentRuntime Browser Agent",
32
+ "version": "0.1.0",
33
+ "description": "Local MCP bridge for the active browser tab (user-connected).",
34
+ "minimum_chrome_version": "114",
35
+ "action": {
36
+ "default_title": "AgentRuntime Browser Agent",
37
+ "default_popup": "popup.html"
38
+ },
39
+ "background": {
40
+ "service_worker": "background.js",
41
+ "type": "module"
42
+ },
43
+ "permissions": [
44
+ "storage",
45
+ "scripting",
46
+ "tabs",
47
+ "activeTab"
48
+ ],
49
+ "host_permissions": [],
50
+ "content_scripts": []
51
+ }
52
+ ```
53
+
54
+ **Why `activeTab` *and* `tabs`:**
55
+
56
+ - **`activeTab`** grants **temporary** host access for the **current tab** when the user invokes the extension (toolbar). Useful for “Connect this tab” flows.
57
+ - **`tabs`** lets the service worker read **`tabId`**, `url`, `title` for the **connected** tab and inject programmatically without repeated user clicks—still subject to your **host_permissions** / origin policy.
58
+
59
+ If you can make “connect” always originate from a user action and only ever touch that tab’s origin via `activeTab`, you may **try** to ship with only `activeTab` + `scripting` + `storage`—but programmatic injection on an arbitrary **user-picked** tab id usually expects **`tabs`** in practice.
60
+
61
+ ---
62
+
63
+ ## Permissions table (v0)
64
+
65
+ | Permission | Required | Rationale |
66
+ |------------|----------|-----------|
67
+ | `storage` | Yes | Persist WebSocket port, session prefs, last connected `tabId`, pairing token fingerprint (not the secret itself). |
68
+ | `scripting` | Yes | Inject isolated content scripts; `chrome.scripting.executeScript` / registered CS. |
69
+ | `tabs` | Yes (recommended) | Resolve connected tab metadata; coordinate injections by `tabId`. |
70
+ | `activeTab` | Yes (recommended) | Minimize broad host access; pair with user-invoked “Connect”. |
71
+ | `notifications` | No | Optional for confirmation prompts; prefer **popup/side panel** for trust UX. |
72
+ | `offscreen` | Maybe | Only if you need offscreen document APIs for capture—prefer `chrome.tabs.captureVisibleTab` first. |
73
+ | `debugger` | **No (v0)** | CDP power; violates “boring, strict” safety posture unless separately gated and audited. |
74
+ | `webRequest` | Yes (current) | Capture **completed** request metadata only (URL, status, method) into a small ring buffer for `browser_network_recent` — **no bodies**. |
75
+ | `history` / `bookmarks` | **No** | Unrelated to connector MVP. |
76
+ | `nativeMessaging` | Optional alt | If you replace WebSocket with native host—different deployment. |
77
+
78
+ ---
79
+
80
+ ## Host permissions (`host_permissions`)
81
+
82
+ ### Option A — Tightest (recommended first)
83
+
84
+ Start with **empty** `host_permissions: []` and rely on **`activeTab`** for host access **when the user invokes** the extension (e.g. clicks Connect on the current tab).
85
+
86
+ **Limitation:** Long-lived “connected tab” automation across navigations may require **re-invoking** user gesture policies depending on navigation and API use—validate in implementation.
87
+
88
+ ### Option B — Explicit schemes
89
+
90
+ Grant only what you need, e.g. internal tooling:
91
+
92
+ ```json
93
+ "host_permissions": [
94
+ "https://*.company.test/*",
95
+ "http://localhost/*"
96
+ ]
97
+ ```
98
+
99
+ ### Option C — Broad access (discouraged for v0)
100
+
101
+ ```json
102
+ "host_permissions": [
103
+ "<all_urls>"
104
+ ]
105
+ ```
106
+
107
+ Use only if you **cannot** ship otherwise—and pair with **strict** origin allowlists in the **MCP server** + extension (double gate). Document the business reason here before merging.
108
+
109
+ ---
110
+
111
+ ## Content scripts
112
+
113
+ **Isolated world (default):** Register static content scripts only for origins you explicitly support, or inject **dynamically** with `chrome.scripting.executeScript` on the connected tab.
114
+
115
+ Example **dynamic** injection (conceptual):
116
+
117
+ - Service worker receives bridge `request`.
118
+ - Calls `chrome.scripting.executeScript({ target: { tabId }, files: ["content.js"] })` **or** `func` injection with **no** user secrets passed from the page.
119
+
120
+ **Do not** expose `postMessage` bridges to the page that carry MCP commands.
121
+
122
+ **`web_accessible_resources`:** Avoid unless required for explicitly trusted UI assets; **never** expose internals to arbitrary pages in v0.
123
+
124
+ ---
125
+
126
+ ## Content Security Policy / MV3 caveats
127
+
128
+ - Service worker has **no DOM**; keep WebSocket + message routing thin.
129
+ - Avoid `eval`, remote code, or remotely loaded logic in extension pages.
130
+
131
+ ---
132
+
133
+ ## WebSocket / network
134
+
135
+ The extension **initiates** outbound connections to `ws://127.0.0.1:*`. This is generally allowed; **do not** add **broad** `host_permissions` “for WebSocket”—loopback is not a match-pattern host permission in the same way; validate during implementation on Chrome/Brave.
136
+
137
+ If a browser build blocks loopback from extensions, document the workaround (flag / enterprise policy) — rare for `127.0.0.1`.
138
+
139
+ ---
140
+
141
+ ## `externally_connectable`
142
+
143
+ **Default:** Omit (not externally connectable).
144
+
145
+ You do **not** want random websites to message your extension. Pairing via **HTTP localhost** pages messaging the extension is a **later** hardening topic.
146
+
147
+ ---
148
+
149
+ ## Update / packaging
150
+
151
+ - **Unpacked dev:** Document `chrome://extensions` → Load unpacked.
152
+ - **Store:** Requires privacy policy if you collect anything; **this design** should collect **nothing** remote.
153
+
154
+ ---
155
+
156
+ ## Change log (manifest)
157
+
158
+ | Date | Change | Author |
159
+ |------|--------|--------|
160
+ | 2026-05-18 | Initial MVP manifest template + permission rationale | docs |
161
+ | 2026-05-18 | Ship `host_permissions: ["<all_urls>"]` so `captureVisibleTab` / element screenshots work on **loopback and non‑matching** URLs during automation (E2E + real use). Tighten later if product requires a narrower allowlist. | docs |
162
+
163
+ When you change permissions in code, **update this file in the same PR**.
package/docs/HYBRID.md ADDED
@@ -0,0 +1,12 @@
1
+ > **Repeated doc (repo independence).** Canonical copy: [desktop-agent `docs/HYBRID.md`](https://github.com/agentruntime-io/desktop-agent/blob/main/docs/HYBRID.md). Prefer editing the canonical repo, then `split/assemble-repos.ps1`; or edit both copies deliberately.
2
+
3
+ # Hybrid browser + desktop
4
+
5
+ This MCP server sends **real OS input** (Win32). It does not see the DOM.
6
+
7
+ For web automation, use **[browser-agent](https://github.com/agentruntime-io/browser-agent)** for structure, refs, and navigation. When `browser_click_element` fails or the page requires trusted events:
8
+
9
+ 1. `browser_get_element_screen_rect` (browser MCP) → `centerX` / `centerY`
10
+ 2. `desktop_probe_click` → `desktop_click` (this MCP)
11
+
12
+ Enable both servers in Cursor (see each repo’s `.cursor/mcp.json`).
@@ -0,0 +1,111 @@
1
+ # Managed browser (Playwright) — optional driver
2
+
3
+ **Default unchanged:** `BROWSER_AUTOMATION_DRIVER` unset → MV3 extension + WebSocket bridge (your real Chrome).
4
+
5
+ **Opt-in:** spawn **Chromium via Playwright** (headless by default, or headed) — Vercel-like “spin up a browser” without the extension.
6
+
7
+ ## MCP / CLI
8
+
9
+ ```bash
10
+ # Headless managed Chromium
11
+ BROWSER_AUTOMATION_DRIVER=managed node server/dist/main.js
12
+
13
+ # Visible window (debugging)
14
+ BROWSER_AUTOMATION_DRIVER=managed BROWSER_AUTOMATION_HEADED=1 node server/dist/main.js
15
+
16
+ # CLI flag
17
+ node server/dist/main.js --managed
18
+ ```
19
+
20
+ Cursor `mcp.json` example:
21
+
22
+ ```json
23
+ "browser-automation-managed": {
24
+ "command": "node",
25
+ "args": [".../server/dist/main.js", "--managed"],
26
+ "env": {
27
+ "BROWSER_AUTOMATION_DRIVER": "managed",
28
+ "BROWSER_AUTOMATION_HEADED": "1"
29
+ }
30
+ }
31
+ ```
32
+
33
+ ## One-time setup
34
+
35
+ ```bash
36
+ cd server
37
+ npm install
38
+ npx playwright install chromium
39
+ ```
40
+
41
+ ## Supported tools
42
+
43
+ Verified by live runs (headless **and** headed) via `server/scripts/live-managed-test.mjs`.
44
+
45
+ | Tool | Managed |
46
+ |------|---------|
47
+ | `navigate`, `get_current_tab`, `get_tabs`, `tab_open` / `tab_close` / `tab_activate`, `go_back`, `reload` | Yes (multi-tab; `tabId` routing; popups auto-become the active tab) |
48
+ | `snapshot` (**preferred**) | Yes — compact text `@e3 button "Submit"`, ~3-5x fewer tokens than the JSON structure; flags captchas |
49
+ | `get_page_structure`, `get_page_text`, `find_nodes`, `get_selected_text` | Yes (DOM walk + refs `e1`…; labels from `<label>`, aria, button text). **Pierces open shadow DOM and iframes** (frame refs `f2.e1`) |
50
+ | `downloads` | Yes — files from clicked download links are saved to `BROWSER_AUTOMATION_DOWNLOAD_DIR` |
51
+ | `click_element`, `fill_input`, `select_option`, `hover`, `scroll`, `press_key` (with modifiers / `ref`), `set_file_input` | Yes |
52
+ | `wait`, `wait_for` | Yes (same semantics as extension; follows popup tab) |
53
+ | `screenshot_visible` (png/jpeg), `screenshot_element` | Yes |
54
+ | `console_recent`, `network_recent`, `evaluate` (needs `BROWSER_AUTOMATION_ALLOW_EVAL=1`) | Yes |
55
+ | `get_element_screen_rect` (hybrid desktop click) | **Headed only** (needs a real window; headless returns a clear `unsupported` error). Verified with a real OS click |
56
+ | `ask_confirmation` | Extension driver only (readable `unsupported` error) |
57
+
58
+ Alert/confirm/prompt dialogs are auto-accepted (and logged to `console_recent`) so pages never freeze.
59
+ The "one navigate until evidence" gate is **off by default** in managed mode (Playwright `goto` is awaited, so there is no race);
60
+ set `BROWSER_AUTOMATION_ENFORCE_NAV_DISCIPLINE=1` to re-enable.
61
+
62
+ Check `browser_extension_status` → `driver: "managed"`, `managedReady: true`.
63
+
64
+ ## Options (env)
65
+
66
+ | Variable | Effect |
67
+ |----------|--------|
68
+ | `BROWSER_AUTOMATION_HEADED=1` | Visible window |
69
+ | `BROWSER_AUTOMATION_USER_DATA_DIR=<dir>` | Persistent profile: cookies / logins survive restarts (log in once headed, reuse headless) |
70
+ | `BROWSER_AUTOMATION_CHANNEL=chrome\|msedge` | Use installed Chrome/Edge instead of bundled Chromium (no `playwright install` needed) |
71
+ | `BROWSER_AUTOMATION_VIEWPORT=1440x900` | Viewport size (default `1280x720`) |
72
+ | `BROWSER_AUTOMATION_DOWNLOAD_DIR=<dir>` | Where downloads are saved (default `~/.ar_browser/downloads`) |
73
+ | `BROWSER_AUTOMATION_BLOCK_RESOURCES=image,media,font` | Abort these resource types for faster page loads |
74
+
75
+ Headless uses Playwright's *new headless* mode (`channel: chromium`): real plugins and `window.chrome` (the old headless shell has 0 plugins and no `window.chrome`), a normalized (non-`HeadlessChrome`) User-Agent, and `--disable-blink-features=AutomationControlled`.
76
+
77
+ `browser_navigate` accepts `waitUntil: commit | domcontentloaded | load` (managed). Most slow navigations are network/DNS latency (e.g. httpbin.org spends ~7s in DNS), not driver overhead; use `commit` + `browser_wait_for` or `BLOCK_RESOURCES` when speed matters.
78
+
79
+ ### Iframes, shadow DOM, captchas
80
+
81
+ - Controls inside iframes (including cross-origin and nested) and **open** shadow roots get refs; iframe refs are `f<N>.e<M>`. Refs are only valid until the next snapshot.
82
+ - Closed shadow roots are invisible to the managed driver (the extension driver can pierce them).
83
+ - Captcha widgets (reCAPTCHA, hCaptcha, Turnstile, Arkose, GeeTest) are *detected and flagged* (`captcha` + `humanRequired`), never solved. Use headed mode so a human can complete them.
84
+
85
+ ## Live test
86
+
87
+ ```bash
88
+ cd server && npm run build
89
+ node scripts/live-managed-test.mjs # headless
90
+ node scripts/live-managed-test.mjs --headed # visible
91
+
92
+ # shadow DOM, iframes, downloads, login + persistent profile, captcha, fingerprint (+ real OS click when --headed)
93
+ node scripts/live-advanced-test.mjs [--headed] [--no-network]
94
+
95
+ # every remaining tool (sessions, survey, hover, keys, file input, tabs, screenshots...); fails if a tool is untested
96
+ node scripts/live-coverage-test.mjs [--headed]
97
+ ```
98
+
99
+ **Screenshots** (`browser_screenshot_visible`, `browser_screenshot_element`, `browser_debug_snapshot`) return a real MCP `image` content block plus a small text block (base64 replaced by a marker). `structuredContent` still carries `dataBase64` for programmatic consumers.
100
+
101
+ Drives the real MCP server over stdio with a strict MCP client (validates every result against its outputSchema). Needs network access.
102
+ Artifacts (screenshots, JSON results) go to `.live-test/` (gitignored).
103
+ ## When to use which
104
+
105
+ | Driver | Use |
106
+ |--------|-----|
107
+ | **extension** | User logins, SSO, existing Chrome profile, production dogfooding |
108
+ | **managed** | CI, agents, clean sessions, headless loops, no “Connect extension” step |
109
+
110
+ Refs use the same `e1` pattern, but **snapshots are not interchangeable** across drivers or browser instances.
111
+