@agentruntime-labs/browser-agent 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (161) hide show
  1. package/LICENSE +202 -0
  2. package/NOTICE +6 -0
  3. package/README.md +64 -34
  4. package/bin/browser-agent.js +1 -0
  5. package/cli/control.mjs +477 -0
  6. package/cli/daemon.mjs +143 -0
  7. package/cli/hosts.mjs +1 -0
  8. package/cli/main.mjs +24 -0
  9. package/cli/mcp.mjs +1 -0
  10. package/docs/ARCHITECTURE.md +73 -0
  11. package/docs/DESKTOP_MCP.md +41 -0
  12. package/docs/EXTENSION_MANIFEST.md +163 -0
  13. package/docs/HYBRID.md +12 -0
  14. package/docs/MANAGED_BROWSER.md +111 -0
  15. package/docs/MCP_TOOLS.md +486 -0
  16. package/docs/MESSAGES.md +272 -0
  17. package/docs/MVP.md +76 -0
  18. package/docs/PRIVACY_POLICY.md +50 -0
  19. package/docs/README.md +27 -0
  20. package/docs/REFERENCE_PROJECTS.md +24 -0
  21. package/docs/RELEASE_CHECKLIST.md +49 -0
  22. package/docs/REPEATED_DOCS.md +27 -0
  23. package/docs/SAFETY.md +92 -0
  24. package/docs/SHELL_QUOTING.md +30 -0
  25. package/docs/SPLIT_REPOS.md +46 -0
  26. package/docs/THIRD_PARTY_LICENSES.md +182 -0
  27. package/docs/VISION.md +38 -0
  28. package/docs/WEB_STORE_SUBMISSION.md +117 -0
  29. package/extension/approval.js +1 -0
  30. package/extension/assets/agentruntime-logo.svg +77 -0
  31. package/extension/assets/agentruntime-mark-512.png +0 -0
  32. package/extension/background.js +776 -62
  33. package/extension/cdp.js +577 -0
  34. package/extension/constants.js +1 -0
  35. package/extension/content.js +69 -9
  36. package/extension/devtools-grant.html +31 -0
  37. package/extension/devtools-grant.js +29 -0
  38. package/extension/icons/icon128.png +0 -0
  39. package/extension/icons/icon16.png +0 -0
  40. package/extension/icons/icon32.png +0 -0
  41. package/extension/icons/icon48.png +0 -0
  42. package/extension/manifest.json +50 -23
  43. package/extension/policy.js +102 -0
  44. package/extension/popup.html +190 -56
  45. package/extension/popup.js +376 -166
  46. package/extension/soft.js +301 -0
  47. package/extension/welcome.html +91 -0
  48. package/extension/welcome.js +13 -0
  49. package/package.json +6 -2
  50. package/server/dist/arBrowserPaths.d.ts.map +1 -1
  51. package/server/dist/arBrowserPaths.js +1 -0
  52. package/server/dist/arBrowserPaths.js.map +1 -1
  53. package/server/dist/automationBackend.d.ts +46 -0
  54. package/server/dist/automationBackend.d.ts.map +1 -0
  55. package/server/dist/automationBackend.js +20 -0
  56. package/server/dist/automationBackend.js.map +1 -0
  57. package/server/dist/automationSession.d.ts.map +1 -1
  58. package/server/dist/automationSession.js +1 -0
  59. package/server/dist/automationSession.js.map +1 -1
  60. package/server/dist/bridge.d.ts +17 -4
  61. package/server/dist/bridge.d.ts.map +1 -1
  62. package/server/dist/bridge.js +132 -33
  63. package/server/dist/bridge.js.map +1 -1
  64. package/server/dist/browserToolsExtras.d.ts +2 -2
  65. package/server/dist/browserToolsExtras.d.ts.map +1 -1
  66. package/server/dist/browserToolsExtras.js +70 -1
  67. package/server/dist/browserToolsExtras.js.map +1 -1
  68. package/server/dist/cdpTools.d.ts +5 -0
  69. package/server/dist/cdpTools.d.ts.map +1 -0
  70. package/server/dist/cdpTools.js +411 -0
  71. package/server/dist/cdpTools.js.map +1 -0
  72. package/server/dist/compositeTools.d.ts +2 -2
  73. package/server/dist/compositeTools.d.ts.map +1 -1
  74. package/server/dist/compositeTools.js.map +1 -1
  75. package/server/dist/debugSnapshot.d.ts +2 -2
  76. package/server/dist/debugSnapshot.d.ts.map +1 -1
  77. package/server/dist/debugSnapshot.js.map +1 -1
  78. package/server/dist/guardrails.d.ts.map +1 -1
  79. package/server/dist/guardrails.js +1 -0
  80. package/server/dist/guardrails.js.map +1 -1
  81. package/server/dist/humanLike.d.ts.map +1 -1
  82. package/server/dist/humanLike.js +1 -0
  83. package/server/dist/humanLike.js.map +1 -1
  84. package/server/dist/main.js +56 -15
  85. package/server/dist/main.js.map +1 -1
  86. package/server/dist/managedBridge.d.ts +58 -0
  87. package/server/dist/managedBridge.d.ts.map +1 -0
  88. package/server/dist/managedBridge.js +619 -0
  89. package/server/dist/managedBridge.js.map +1 -0
  90. package/server/dist/managedDom.d.ts +58 -0
  91. package/server/dist/managedDom.d.ts.map +1 -0
  92. package/server/dist/managedDom.js +310 -0
  93. package/server/dist/managedDom.js.map +1 -0
  94. package/server/dist/policyTiers.d.ts.map +1 -1
  95. package/server/dist/policyTiers.js +3 -1
  96. package/server/dist/policyTiers.js.map +1 -1
  97. package/server/dist/protocol.d.ts +4 -0
  98. package/server/dist/protocol.d.ts.map +1 -1
  99. package/server/dist/protocol.js +1 -0
  100. package/server/dist/protocol.js.map +1 -1
  101. package/server/dist/recorder.d.ts +22 -0
  102. package/server/dist/recorder.d.ts.map +1 -0
  103. package/server/dist/recorder.js +82 -0
  104. package/server/dist/recorder.js.map +1 -0
  105. package/server/dist/richEditorFill.d.ts.map +1 -1
  106. package/server/dist/richEditorFill.js +1 -0
  107. package/server/dist/richEditorFill.js.map +1 -1
  108. package/server/dist/screenshotCrop.d.ts.map +1 -1
  109. package/server/dist/screenshotCrop.js +1 -0
  110. package/server/dist/screenshotCrop.js.map +1 -1
  111. package/server/dist/sessionLog.d.ts +1 -0
  112. package/server/dist/sessionLog.d.ts.map +1 -1
  113. package/server/dist/sessionLog.js +1 -0
  114. package/server/dist/sessionLog.js.map +1 -1
  115. package/server/dist/sessionPhase.d.ts.map +1 -1
  116. package/server/dist/sessionPhase.js.map +1 -1
  117. package/server/dist/sessionSummary.d.ts.map +1 -1
  118. package/server/dist/sessionSummary.js +1 -0
  119. package/server/dist/sessionSummary.js.map +1 -1
  120. package/server/dist/shotUtil.d.ts +44 -0
  121. package/server/dist/shotUtil.d.ts.map +1 -0
  122. package/server/dist/shotUtil.js +89 -0
  123. package/server/dist/shotUtil.js.map +1 -0
  124. package/server/dist/skillsInstructions.d.ts.map +1 -1
  125. package/server/dist/skillsInstructions.js +1 -0
  126. package/server/dist/skillsInstructions.js.map +1 -1
  127. package/server/dist/snapshotText.d.ts +47 -0
  128. package/server/dist/snapshotText.d.ts.map +1 -0
  129. package/server/dist/snapshotText.js +99 -0
  130. package/server/dist/snapshotText.js.map +1 -0
  131. package/server/dist/structureFilter.d.ts.map +1 -1
  132. package/server/dist/structureFilter.js.map +1 -1
  133. package/server/dist/surveyCatalog.d.ts.map +1 -1
  134. package/server/dist/surveyCatalog.js +1 -0
  135. package/server/dist/surveyCatalog.js.map +1 -1
  136. package/server/dist/surveyRunner.d.ts.map +1 -1
  137. package/server/dist/surveyRunner.js.map +1 -1
  138. package/server/dist/toolHelpers.d.ts +27 -2
  139. package/server/dist/toolHelpers.d.ts.map +1 -1
  140. package/server/dist/toolHelpers.js +67 -1
  141. package/server/dist/toolHelpers.js.map +1 -1
  142. package/server/dist/toolRuntime.d.ts +4 -0
  143. package/server/dist/toolRuntime.d.ts.map +1 -1
  144. package/server/dist/toolRuntime.js +12 -6
  145. package/server/dist/toolRuntime.js.map +1 -1
  146. package/server/dist/toolSchemas.d.ts +37 -1
  147. package/server/dist/toolSchemas.d.ts.map +1 -1
  148. package/server/dist/toolSchemas.js +33 -1
  149. package/server/dist/toolSchemas.js.map +1 -1
  150. package/server/dist/tools.d.ts +2 -2
  151. package/server/dist/tools.d.ts.map +1 -1
  152. package/server/dist/tools.js +65 -11
  153. package/server/dist/tools.js.map +1 -1
  154. package/server/dist/verdictExtract.d.ts.map +1 -1
  155. package/server/dist/verdictExtract.js +1 -0
  156. package/server/dist/verdictExtract.js.map +1 -1
  157. package/server/dist/workflowTools.d.ts +2 -2
  158. package/server/dist/workflowTools.d.ts.map +1 -1
  159. package/server/dist/workflowTools.js +16 -4
  160. package/server/dist/workflowTools.js.map +1 -1
  161. package/skills/browser-agent/SKILL.md +89 -0
@@ -0,0 +1,272 @@
1
+ # WebSocket bridge protocol
2
+
3
+ This document defines the **localhost WebSocket** contract between the **MCP server’s bridge** (server side) and the **browser extension** (client side). It is **project-specific**—intentionally stricter than ad hoc bridges (e.g. no wire versioning and weak origin checks in some reference projects).
4
+
5
+ **Goals:**
6
+
7
+ - **Versioned** envelopes so extension and server can evolve.
8
+ - **Session-bound** connections: not every local process may drive the browser.
9
+ - **Bounded** payloads (anti-footgun for huge DOM / screenshots).
10
+ - **JSON** text frames only (MVP).
11
+
12
+ **Non-goals:** Binary frames, multiplexed streams, remote peers.
13
+
14
+ ---
15
+
16
+ ## Transport
17
+
18
+ | Item | Value |
19
+ |------|--------|
20
+ | URL | `ws://127.0.0.1:{port}/` (default port TBD; configurable) |
21
+ | Bind | Listen on **loopback only** (`127.0.0.1`), never `0.0.0.0` |
22
+ | Frame | Text frames, UTF-8 JSON, **one object per frame** |
23
+
24
+ ---
25
+
26
+ ## Prior art (research note)
27
+
28
+ [ofershap/real-browser-mcp](https://github.com/ofershap/real-browser-mcp) uses JSON with `id`, `tool`, `params` from server → extension, and `{ id, success, result|error }` back; **ping/pong** keepalives; **no** wire protocol version or app-level token. **This spec adds** `proto`, `type`, **`session` pairing**, and structured **`error`** codes—recommended for a **trusted** daily-driver connector.
29
+
30
+ ---
31
+
32
+ ## Pairing & session
33
+
34
+ ### Startup (MCP / bridge)
35
+
36
+ 1. Bridge generates a random **`sessionSecret`** (≥128 bits), stores it in memory.
37
+ 2. Bridge prints **one line** to stderr for the human operator, e.g.
38
+ `SESSION=<base64url secret>; connect extension with this token.`
39
+ (Exact UX TBD: copy button in CLI, QR, etc.)
40
+ 3. Extension user pastes token into popup **or** first WebSocket message sends it.
41
+
42
+ ### First message after connect (extension → bridge)
43
+
44
+ **Required:** extension sends `session.open` **before** any `request` (except optional `session.open` retry).
45
+
46
+ ```json
47
+ {
48
+ "proto": 1,
49
+ "type": "session.open",
50
+ "id": "550e8400-e29b-41d4-a716-446655440000",
51
+ "payload": {
52
+ "token": "<sessionSecret>",
53
+ "extensionId": "<chrome.runtime.id>",
54
+ "client": "browser_automation_ext",
55
+ "clientVersion": "0.1.0"
56
+ }
57
+ }
58
+ ```
59
+
60
+ **Bridge → extension:**
61
+
62
+ ```json
63
+ {
64
+ "proto": 1,
65
+ "type": "session.opened",
66
+ "id": "550e8400-e29b-41d4-a716-446655440000",
67
+ "payload": {
68
+ "sessionId": "opaque-session-handle",
69
+ "serverVersion": "0.1.0",
70
+ "bridgePort": 18791,
71
+ "limits": { "maxPayloadBytes": 10485760, "maxNodes": 5000 }
72
+ }
73
+ }
74
+ ```
75
+
76
+ **Errors:** wrong token → close socket with **policy violation** (see errors below). Max failed opens → optional rate limit.
77
+
78
+ **Why:** Without this, **any local process** that can speak WebSocket to loopback may be able to drive the extension unless gated solely by `Origin` checks (which are [inconsistent or absent for some clients](https://github.com/ofershap/real-browser-mcp/blob/main/mcp-server/src/bridge.ts)—**do not rely** on Origin alone).
79
+
80
+ ---
81
+
82
+ ## Envelope (all messages)
83
+
84
+ | Field | Type | Required | Notes |
85
+ |-------|------|----------|--------|
86
+ | `proto` | integer | Yes | **Wire protocol version.** Only `1` in this document. |
87
+ | `type` | string | Yes | `session.open`, `session.opened`, `request`, `response`, `event`, `ping`, `pong`, `error` |
88
+ | `id` | string | Usually | Correlation id: UUID v4 **recommended**; required for `request`/`response`. |
89
+ | `payload` | object | Sometimes | Type-specific body (preferred). |
90
+ | | | | Legacy fields may exist during migration—**avoid** top-level `tool` without `type`. |
91
+
92
+ **Max frame size:** enforce `maxPayloadBytes` (default **10 MiB**) **before** `JSON.parse`; reject with `payload_too_large`.
93
+
94
+ ---
95
+
96
+ ## Request (bridge → extension)
97
+
98
+ Sent after `session.opened`. Maps 1:1 to MCP tool handler invoking extension.
99
+
100
+ ```json
101
+ {
102
+ "proto": 1,
103
+ "type": "request",
104
+ "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
105
+ "payload": {
106
+ "method": "get_page_structure",
107
+ "params": {
108
+ "tabId": 123,
109
+ "maxNodes": 500,
110
+ "includeRects": true
111
+ },
112
+ "meta": {
113
+ "deadlineMs": 30000,
114
+ "mcpTool": "browser_get_page_structure",
115
+ "mcpRequestId": "opaque"
116
+ }
117
+ }
118
+ }
119
+ ```
120
+
121
+ | Field | Meaning |
122
+ |-------|--------|
123
+ | `method` | Short bridge method name; see [MCP_TOOLS](./MCP_TOOLS.md). |
124
+ | `params` | Method arguments; must be a JSON object. |
125
+ | `meta` | Optional tracing; **must not** include secrets or page credentials. |
126
+
127
+ ---
128
+
129
+ ## Response (extension → bridge)
130
+
131
+ ### Success
132
+
133
+ ```json
134
+ {
135
+ "proto": 1,
136
+ "type": "response",
137
+ "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
138
+ "payload": {
139
+ "ok": true,
140
+ "result": { }
141
+ }
142
+ }
143
+ ```
144
+
145
+ ### Application error (handled)
146
+
147
+ ```json
148
+ {
149
+ "proto": 1,
150
+ "type": "response",
151
+ "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
152
+ "payload": {
153
+ "ok": false,
154
+ "error": {
155
+ "code": "stale_ref",
156
+ "message": "Element ref e12 not found after navigation",
157
+ "details": { "ref": "e12" }
158
+ }
159
+ }
160
+ }
161
+ ```
162
+
163
+ **Bridge behavior:** map `error.code` to MCP `isError` / `-32602` style messages as your SDK requires.
164
+
165
+ ---
166
+
167
+ ## Events (extension → bridge, optional)
168
+
169
+ Use for **navigation**, tab closed, user denied confirmation—so MCP server can cancel waits.
170
+
171
+ ```json
172
+ {
173
+ "proto": 1,
174
+ "type": "event",
175
+ "id": "b4b7b5c3-2a0e-4c8f-9d2a-7f1e12345678",
176
+ "payload": {
177
+ "name": "tab.navigated",
178
+ "tabId": 123,
179
+ "url": "https://example.com/next"
180
+ }
181
+ }
182
+ ```
183
+
184
+ **Ordering:** events are **not** guaranteed ordered with responses unless documented; consumers must tolerate interleaving.
185
+
186
+ ---
187
+
188
+ ## Keepalive
189
+
190
+ Optional but recommended (timeouts configurable).
191
+
192
+ **Bridge → extension:**
193
+
194
+ ```json
195
+ { "proto": 1, "type": "ping", "id": "018f4624-d3a8-7f3a-9b2c-111122223333" }
196
+ ```
197
+
198
+ **Extension → bridge:**
199
+
200
+ ```json
201
+ { "proto": 1, "type": "pong", "id": "018f4624-d3a8-7f3a-9b2c-111122223333" }
202
+ ```
203
+
204
+ Implementations may use **null `id`** for heartbeat-only pings if both sides agree—prefer explicit ids to simplify logs.
205
+
206
+ ---
207
+
208
+ ## Error codes (application `error.code`)
209
+
210
+ | Code | Meaning |
211
+ |------|--------|
212
+ | `unauthorized` | Bad/missing session token. |
213
+ | `unknown_method` | `method` not implemented. |
214
+ | `invalid_params` | Params failed validation. |
215
+ | `tab_not_found` | `tabId` invalid or closed. |
216
+ | `not_connected` | No user-connected tab for this session. |
217
+ | `policy_denied` | Origin/action blocked by allowlist or sensitivity gate. |
218
+ | `stale_ref` | Element `ref` missing or outdated. |
219
+ | `timeout` | Exceeded `deadlineMs`. |
220
+ | `internal` | Unclassified failure; log stack in server only. |
221
+
222
+ ---
223
+
224
+ ## Wire protocol negotiation
225
+
226
+ - If **bridge** receives `proto` **unsupported**, reply once with `type: "error"` and close.
227
+ - If **extension** receives `proto` > supported, close and show UI “update server”.
228
+
229
+ Example **fatal wire error (bridge → extension):**
230
+
231
+ ```json
232
+ {
233
+ "proto": 1,
234
+ "type": "error",
235
+ "id": null,
236
+ "payload": {
237
+ "code": "unsupported_proto",
238
+ "message": "server max proto is 1, client sent 9"
239
+ }
240
+ }
241
+ ```
242
+
243
+ ---
244
+
245
+ ## Size & abuse limits
246
+
247
+ | Limit | Suggested default |
248
+ |--------|-------------------|
249
+ | Max JSON frame | 10 MiB |
250
+ | Max `get_page_text` chars | Enforced server-side per [MCP_TOOLS](./MCP_TOOLS.md) |
251
+ | Max structure nodes | 5000 (reject or trim with `truncated: true`) |
252
+ | Concurrent requests | 1–2 per tab (avoid re-entrancy bugs); queue rest |
253
+
254
+ ---
255
+
256
+ ## Mapping to MCP
257
+
258
+ 1. MCP `tools/call` hits tool handler in server.
259
+ 2. Handler validates policy + **tabId**.
260
+ 3. Sends WebSocket `request` with `meta.mcpTool` for logs.
261
+ 4. Awaits matching `response` by `id`.
262
+ 5. Converts `result` / `error` to MCP tool result.
263
+
264
+ ---
265
+
266
+ ## Change log
267
+
268
+ | Date | Change |
269
+ |------|--------|
270
+ | 2026-05-18 | Initial protocol v1 (stricter than reference OSS bridge) |
271
+
272
+ When `proto` increments, append a **new section** to this doc (do not overwrite v1).
package/docs/MVP.md ADDED
@@ -0,0 +1,76 @@
1
+ # MVP scope
2
+
3
+ ## Status
4
+
5
+ **v0 shipped.** The loop is proven in production dogfooding:
6
+
7
+ `MCP client → local MCP server → localhost bridge → extension → page → structured reply`
8
+
9
+ The openai-test-client + session JSONL logs validate multi-step flows (e.g. Google SERP → five bot-detector sites sequentially).
10
+
11
+ ## Goal (original)
12
+
13
+ Ship a **boring**, reviewable connector with **strict** tab scope and **gated** dangerous capabilities.
14
+
15
+ ## Deliverables
16
+
17
+ | Deliverable | Status |
18
+ |-------------|--------|
19
+ | **Extension** | Connects to local server; target tab; content script snapshot + actions. |
20
+ | **Bridge** | Localhost WebSocket + session binding; versioned wire (`v1`). |
21
+ | **MCP server** | stdio MCP; tools below + extras; structured outputs (Zod). |
22
+ | **openai-test-client** | Skills, session log, navigate discipline, gpt-5.1 default. |
23
+ | **Docs** | [MCP_TOOLS](./MCP_TOOLS.md), [MESSAGES](./MESSAGES.md), [SAFETY](./SAFETY.md) — keep aligned with shipped behavior. |
24
+
25
+ ## MCP tools (shipped)
26
+
27
+ Canonical names and JSON Schemas: [MCP_TOOLS](./MCP_TOOLS.md).
28
+
29
+ ### Core (v0)
30
+
31
+ | Tool | Purpose |
32
+ |------|---------|
33
+ | `browser_navigate` | Load http(s) URL in connected tab. |
34
+ | `browser_get_current_tab` | Title, URL, tab id. |
35
+ | `browser_get_page_text` | Visible text (bounded). |
36
+ | `browser_get_page_structure` | Snapshot with stable refs; viewportOnly + stats. |
37
+ | `browser_click_element` | Click by ref. |
38
+ | `browser_fill_input` | Fill by ref. |
39
+ | `browser_screenshot_visible` | Viewport capture. |
40
+ | `browser_wait` | Short DOM settle wait. |
41
+
42
+ ### Post-v0 (shipped)
43
+
44
+ | Tool | Purpose |
45
+ |------|---------|
46
+ | `browser_wait_for` | Poll URL/text (replaces blind `browser_wait` after navigate). |
47
+ | `browser_find_nodes` | Filter structure by role/name/tag. |
48
+ | `browser_debug_snapshot` | Tab + structure + text preview + optional screenshot/network. |
49
+ | `browser_screenshot_element` | Crop element by ref. |
50
+ | `browser_console_recent` / `browser_network_recent` | Bounded diagnostics. |
51
+ | `browser_set_file_input` | File upload via ref. |
52
+ | `browser_evaluate` | Gated (`BROWSER_AUTOMATION_ALLOW_EVAL=1`). |
53
+ | `browser_assert_*` | URL/text gates for agent workflows. |
54
+ | Tab / navigation extras | `browser_go_back`, `browser_reload`, `browser_tab_*`, etc. |
55
+
56
+ ## Explicitly still out of scope
57
+
58
+ - Stealth / anti-bot / fingerprint spoofing
59
+ - Remote/cloud relay
60
+ - Autonomous multi-tab workflows without user intent
61
+ - Cookie/storage editing without explicit product design
62
+
63
+ ## Definition of done (v0) — met
64
+
65
+ - E2E: snapshot → click → screenshot (Playwright + real extension).
66
+ - CI: unit + E2E under xvfb.
67
+ - Safety doc matches gated eval and diagnostics tiers.
68
+ - Dogfood: multi-site survey with JSONL audit trail.
69
+
70
+ ## Suggested next (v0.1)
71
+
72
+ 1. Client navigate discipline + depth (done in openai-test-client).
73
+ 2. SERP hub patterns in client skills (done).
74
+ 3. Optional: two-tab SERP workflow; Amazon/captcha human-in-the-loop.
75
+
76
+ See [`openai-test-client/README.md`](openai-test-client/README.md) for client env, skills, and tests.
@@ -0,0 +1,50 @@
1
+ # Privacy policy: AgentRuntime Browser Agent
2
+
3
+ Effective date: replace with the publication date. Publisher: AgentRuntime Labs (contact: replace with the support address).
4
+
5
+ ## Summary
6
+
7
+ The extension works only between your Chrome browser and software that runs on your own computer. We do not receive,
8
+ collect, store or sell any information about you or your browsing.
9
+
10
+ ## What the extension handles
11
+
12
+ When a local AI agent that you run asks for it, the extension reads from the tab that agent targets:
13
+
14
+ - the tab's address (URL) and title,
15
+ - the visible text and structure of the page, and screenshots of it,
16
+ - recent network request metadata (address, status, method, type; never bodies or headers),
17
+ - recent download names and states.
18
+
19
+ It can also perform actions in that tab (click, type, scroll, open or close tabs) and, only if you allow it, run a
20
+ script that your agent sends.
21
+
22
+ ## Where the data goes
23
+
24
+ Only to a server listening on `127.0.0.1` on your own computer, over a local WebSocket connection that you start
25
+ yourself (`browser-agent`). That server hands the data to the AI application on your computer that you chose to
26
+ connect, such as Claude Desktop or Cursor. What that application does with the data is governed by that application's
27
+ own terms and privacy policy, not by this extension.
28
+
29
+ The extension makes **no** connection to AgentRuntime or to any other internet server, includes no analytics or
30
+ advertising code, and does not load code from the internet.
31
+
32
+ ## What is stored
33
+
34
+ In Chrome's local extension storage on your device: the server port, an optional session token you typed, a random id
35
+ and a name for this Chrome profile, your choice of pinned tab, and your DevTools and script-evaluation settings. A short
36
+ list of recent network metadata is kept in memory only and is lost when the browser or extension restarts.
37
+
38
+ ## Your controls
39
+
40
+ - Remove the extension to delete everything it stores.
41
+ - Press **Disconnect** in the popup to stop all agent access immediately.
42
+ - DevTools features and script evaluation are off until you switch them on, and you can switch them off again.
43
+
44
+ ## Changes
45
+
46
+ If this policy changes, the new version will be published at this address with a new effective date.
47
+
48
+ ## Contact
49
+
50
+ Replace with the support address (for example support@agentruntime.io).
package/docs/README.md ADDED
@@ -0,0 +1,27 @@
1
+ # Documentation
2
+
3
+ Docs for the **browser automation** project: a small, auditable stack that connects your **real** browser profile to local agents via MCP—without stealth, fingerprint games, or a parallel “automation browser.”
4
+
5
+ **Run / build / test:** see the repository [README.md](../README.md).
6
+
7
+ ## How to read this
8
+
9
+ 1. [Vision](./VISION.md) — what we are building and what “legit” means here
10
+ 2. [Architecture](./ARCHITECTURE.md) — components and data flow
11
+ 3. [Safety](./SAFETY.md) — threat model, gates, and non-goals
12
+ 4. [MVP](./MVP.md) — first shippable slice and tool list
13
+ 5. [MESSAGES](./MESSAGES.md) — WebSocket wire protocol (pairing, envelopes, errors)
14
+ 6. [MCP tools](./MCP_TOOLS.md) — MCP tool names + JSON Schemas + bridge mapping
15
+ 7. [Extension manifest](./EXTENSION_MANIFEST.md) — MV3 permissions and host access
16
+ 8. [Reference projects](./REFERENCE_PROJECTS.md) — Phase 1 study list (read, don’t blindly fork)
17
+ 9. [Repeated docs](./REPEATED_DOCS.md) — which files are copied across browser-agent / desktop-agent
18
+ 10. [Hybrid flows](./HYBRID.md) — in **desktop-agent** canonical; repeated in browser-agent after assemble
19
+ 11. [Desktop MCP summary](./DESKTOP_MCP.md) — in **desktop-agent** canonical; repeated in browser-agent after assemble
20
+
21
+ ## Glossary
22
+
23
+ | Term | Meaning |
24
+ |------|--------|
25
+ | **Legit browser (this repo)** | Legitimate, user-trustworthy control of **your existing** Chrome/Brave (etc.) session—not a new browser engine and not a spoofed automation profile. |
26
+ | **Connector** | The extension + localhost bridge + MCP server that exposes a **minimal** tool surface. |
27
+ | **Connected tab** | The tab (or explicit tab id) the user/agent has scoped for the current session. |
@@ -0,0 +1,24 @@
1
+ # Reference projects (Phase 1 study)
2
+
3
+ Use these to **learn patterns and tool semantics**. Prefer **copying ideas** into a **small** codebase you own; avoid adopting large permission or injection models wholesale.
4
+
5
+ | Project | Use for | Caution |
6
+ |---------|---------|---------|
7
+ | **[real-browser-mcp](https://github.com/ofershap/real-browser-mcp)** | Extension + localhost + MCP loop; real profile. | Audit manifest & tools; gate `evaluate`/network if present. |
8
+ | **[chrome-devtools-mcp](https://github.com/ChromeDevTools/chrome-devtools-mcp)** | Tool naming/behavior reference. | Puppeteer/debug automation—not your daily-profile trust model. |
9
+ | **[mcp-chrome](https://github.com/hangwin/mcp-chrome)** | Feature inventory (tabs, extraction). | Broad permissions; MAIN-world injection patterns—do not mirror for v0. |
10
+ | **BrowserMCP-class products** | UX/setup benchmarks. | Treat as **opaque** until fully source-audited. |
11
+ | **[OpenChrome](https://github.com/shaun0927/openchrome)** | Optional engineering ideas (structure, retries). | **Not** a foundation if you reject stealth/anti-bot posture. |
12
+ | **[Nanobrowser](https://github.com/nanobrowser/nanobrowser)** | Side panel, task UX, confirmations. | Heavy product surface—not the MCP bridge reference. |
13
+
14
+ ## Study outputs (checklist)
15
+
16
+ For each reference you actually clone/run, write down:
17
+
18
+ 1. **Transport** — WebSocket vs native messaging; auth story.
19
+ 2. **Permissions** — manifest entries and why.
20
+ 3. **Snapshot model** — how refs are generated and kept stable.
21
+ 4. **Risky tools** — eval, injection, network, debugger.
22
+ 5. **What you will **not** port** — explicit list.
23
+
24
+ Store notes under `docs/notes/` (optional, per your workflow) so design decisions stay traceable.
@@ -0,0 +1,49 @@
1
+ # Release checklist (browser-agent)
2
+
3
+ Licence: Apache-2.0 (`LICENSE`, `NOTICE`, SPDX headers, `docs/THIRD_PARTY_LICENSES.md`).
4
+
5
+ ## Before any release
6
+
7
+ - [ ] `node scripts/gen-third-party.mjs` exits 0 (it fails if a dependency licence needs review) and the regenerated
8
+ `docs/THIRD_PARTY_LICENSES.md` is committed.
9
+ - [ ] `cd server && npm run build && npx vitest run` green.
10
+ - [ ] Live pass on a real Chrome with the extension reloaded (see "Live test script" below).
11
+ - [ ] Versions agree: `package.json`, `extension/manifest.json`. Bump both (manifest versions must increase for the store).
12
+ - [ ] `npm pack --dry-run` shows only `bin cli docs extension server/dist LICENSE NOTICE README.md package.json`. No
13
+ `node_modules`, tokens, logs or test output.
14
+ - [ ] README and `docs/MCP_TOOLS.md` match the tools in `server/dist`.
15
+
16
+ ## npm (CLI and MCP server)
17
+
18
+ 1. Publish `@agentruntime-labs/browser-agent` first. The store reviewer instructions and the README use `npx`, so the
19
+ version with the `daemon` and `shell` commands must be on npm before the extension is submitted.
20
+ 2. Tag the repo `vX.Y.Z`.
21
+
22
+ See also `RELEASE-NPM.md`.
23
+
24
+ ## Chrome Web Store
25
+
26
+ 1. Fill in the placeholders in `docs/PRIVACY_POLICY.md` (date, contact) and publish it at a public URL.
27
+ 2. Capture screenshots (1280x800) and a 440x280 promo tile. Decide whether to replace the placeholder icons.
28
+ 3. `dev_tools/build-extension-zip.ps1` to create the zip. Check it contains `icons/` and every `.js`/`.html` listed in
29
+ the manifest.
30
+ 4. Developer dashboard (one-time registration fee applies): upload the zip, paste the text from
31
+ `docs/WEB_STORE_SUBMISSION.md`, set visibility (Public or Unlisted for a soft launch), submit for review.
32
+ 5. After approval, put the store link in the README and in the installer's "Next:" message.
33
+
34
+ ## Live test script (about 10 minutes)
35
+
36
+ 1. Reload the extension in each Chrome profile; give each a name in the popup.
37
+ 2. `browser-agent wait`, `browser-agent profiles` (all profiles listed), `browser-agent tabs`.
38
+ 3. `open <url>`, `use-tab <text>`, `shot`, `shot --full` (expects the Allow window the first time), `viewport mobile`,
39
+ `reset`, `scheme dark` (needs Allow), `dblclick`, `rclick`, `drag` on a test page, `clip write` then `clip read`,
40
+ `record start` then `record stop demo --gif`.
41
+ 4. Two profiles: `--profile A tabs` and `--profile B tabs` return different tab ids.
42
+ 5. Disconnect in the popup: commands fail with "not connected", and no reconnect happens until Connect is pressed.
43
+ 6. Close the Chrome window with the agent's default tab: the next command falls back to the popup's tab choice.
44
+
45
+ ## Known limits to state in release notes
46
+
47
+ - A tab in a minimized window cannot be captured unless `--restore` is used.
48
+ - Phone and tablet widths use page zoom by default (no touch, no mobile user agent). `engine: cdp` gives true emulation.
49
+ - Recording without DevTools is about 2 frames per second.
@@ -0,0 +1,27 @@
1
+ # Repeated docs (repo independence)
2
+
3
+ Some markdown is **copied into both** [browser-agent](https://github.com/agentruntime-io/browser-agent) and [desktop-agent](https://github.com/agentruntime-io/desktop-agent) so each clone is usable without the other. Each repeated file starts with a **Repeated doc** callout naming the **canonical** repo/path.
4
+
5
+ **Source of truth for assembly:** this monorepo’s `docs/` plus `split/*/docs/` overlays. Re-sync with `.\split\assemble-repos.ps1`.
6
+
7
+ ## Matrix
8
+
9
+ | Document | Canonical repo | Repeated in | Notes |
10
+ |----------|----------------|-------------|--------|
11
+ | `docs/ARCHITECTURE.md` | browser-agent | desktop-agent | Extension + bridge architecture |
12
+ | `docs/EXTENSION_MANIFEST.md` | browser-agent | desktop-agent | MV3; relevant for hybrid users |
13
+ | `docs/MCP_TOOLS.md` | browser-agent | desktop-agent | Browser MCP tools + bridge |
14
+ | `docs/MESSAGES.md` | browser-agent | desktop-agent | WebSocket wire protocol |
15
+ | `docs/MVP.md` | browser-agent | desktop-agent | |
16
+ | `docs/README.md` | browser-agent | desktop-agent | Doc index (desktop intro added on repeat) |
17
+ | `docs/REFERENCE_PROJECTS.md` | browser-agent | desktop-agent | |
18
+ | `docs/SAFETY.md` | browser-agent | desktop-agent | |
19
+ | `docs/VISION.md` | browser-agent | desktop-agent | |
20
+ | `docs/SPLIT_REPOS.md` | browser-agent | desktop-agent | Split/history notes |
21
+ | `docs/HYBRID.md` | desktop-agent | browser-agent | Browser + desktop pairing |
22
+ | `docs/DESKTOP_MCP.md` | desktop-agent | browser-agent | Desktop MCP summary (full detail: desktop-agent `README.md`) |
23
+
24
+ ## Editing workflow
25
+
26
+ 1. **Prefer editing canonical side** (table above), then run assemble and push both repos.
27
+ 2. **Emergency edit in one GitHub repo only:** update the other copy or drift is intentional—remove/update the repeated callout if you fork content permanently.
package/docs/SAFETY.md ADDED
@@ -0,0 +1,92 @@
1
+ # Safety and threat model
2
+
3
+ ## Assets we protect
4
+
5
+ - **User browser session** — cookies, tokens, logged-in state.
6
+ - **Page content** — PII, email, health/financial data visible in DOM or screenshots.
7
+ - **Action integrity** — no silent submits, purchases, or message sends.
8
+
9
+ ## Actors
10
+
11
+ | Actor | Capability | Concern |
12
+ |--------|------------|---------|
13
+ | **Local agent (LLM + MCP client)** | Calls tools with high autonomy | Constrained by policy, confirmations, and client-side navigate discipline (openai-test-client). |
14
+ | **Local MCP server** | Sees tool args and results | Must not phone home; logs controllable. |
15
+ | **Other local processes** | May try `127.0.0.1` | WebSocket requires session token (except `--insecure-no-token` dev mode). |
16
+ | **Web pages** | Cannot call MCP directly | Extension must not expose a page-invokable automation API. |
17
+
18
+ ## Tool tiers (as shipped)
19
+
20
+ ### Tier A — Default (read + low-risk act)
21
+
22
+ - Tab metadata; connected-tab scope.
23
+ - Visible / accessibility-oriented **structure** (interactables + refs).
24
+ - Read visible text / selected text.
25
+ - Screenshot (viewport or element crop).
26
+ - Click / fill on **explicit** refs from snapshot.
27
+ - **browser_wait** / **browser_wait_for**.
28
+ - **browser_find_nodes**, **browser_debug_snapshot** (read-heavy bundles).
29
+
30
+ ### Tier B — Bounded diagnostics (shipped)
31
+
32
+ - **browser_console_recent** — ring buffer, capped entries.
33
+ - **browser_network_recent** — recent requests (extension `webRequest`; documented in manifest).
34
+ - **browser_set_file_input** — explicit path + ref; user chose file on disk.
35
+ - **browser_assert_*** — read-only gates for agent workflows.
36
+
37
+ ### Tier C — Gated / avoid by default
38
+
39
+ - **browser_evaluate** — MAIN-world `eval`; requires `BROWSER_AUTOMATION_ALLOW_EVAL=1` on server.
40
+ - Header/cookie/storage **modification** (not exposed).
41
+ - Request interception / MITM.
42
+ - Dialog override without user visibility.
43
+ - **Stealth / anti-bot / fingerprint tweaks** — explicitly out of product scope.
44
+
45
+ ## Sensitive actions (require confirmation)
46
+
47
+ Examples: submit application, send message/email, checkout/pay, delete data, change security settings, OAuth consent.
48
+
49
+ **Server env (optional):**
50
+
51
+ - `BROWSER_AUTOMATION_CONFIRM_CROSS_ORIGIN=1` — in-browser prompt before cross-origin navigate.
52
+ - `BROWSER_AUTOMATION_CONFIRM_TAB_CLOSE=1` — prompt before tab close.
53
+
54
+ **Rule:** Irreversible or account/security impact should block until the user approves in extension UI, not only LLM text.
55
+
56
+ ## MCP server guardrails (all clients)
57
+
58
+ | Mechanism | Purpose |
59
+ |-----------|---------|
60
+ | **~/.ar_browser/** | Logs, session state, skills, bot-detector catalog (override: `BROWSER_AUTOMATION_HOME`). |
61
+ | **`browser_session_open` / `browser_session_resume`** | `automationSessionId` for navigate guard + resume across runs. |
62
+ | **Navigate discipline** (`BROWSER_AUTOMATION_ENFORCE_NAV_DISCIPLINE`, default on) | Evidence before next navigate; blocks batched navigates within a session. |
63
+ | **Human-like mode** (`BROWSER_AUTOMATION_HUMAN_LIKE=1`, default off) | Random pacing between act tools; extension hover-before-click and char-by-char fill; CAPTCHA/bot-wall detection on page text reads. |
64
+ | **Session JSONL** (`BROWSER_AUTOMATION_SESSION_LOG`, default on) | Audit tool sequence under `~/.ar_browser/logs/`; set `0` to disable. |
65
+ | **Skills** (`~/.ar_browser/skills/`) | Workflow guides bootstrapped from `openai-test-client/skills/`. |
66
+
67
+ ## openai-test-client guardrails (optional harness)
68
+
69
+ | Mechanism | Purpose |
70
+ |-----------|---------|
71
+ | **Client skills** | Same skill files injected into the OpenAI system prompt. |
72
+ | **Client navigate discipline** (`BROWSER_AUTOMATION_CLIENT_ENFORCE_NAV=1`) | Duplicate of server guard — off by default. |
73
+ | **Client session JSONL** (`BROWSER_AUTOMATION_CLIENT_SESSION_LOG=1`) | Optional duplicate log under `./logs/`. |
74
+ | **Max tool depth** (`BROWSER_AUTOMATION_MAX_TOOL_DEPTH`, default 64) | Room for multi-site surveys without `"continue"`. |
75
+
76
+ ## Permission minimization (extension)
77
+
78
+ MVP uses broad `http(s)://*/*` host permission so one target tab works across sites. **webRequest** is used for bounded network_recent. Justify tightening in [EXTENSION_MANIFEST](./EXTENSION_MANIFEST.md) when adding optional host-scoped permissions.
79
+
80
+ ## Validation checklist (before “daily driver”)
81
+
82
+ Run with and without the extension on a fixed test page and compare (no intentional delta from our code):
83
+
84
+ - `navigator.webdriver` (no **extra** spoofing from us).
85
+ - `navigator.userAgent`, `platform`, language, timezone.
86
+ - Viewport / screen accessors.
87
+
88
+ **Expectation:** No deliberate identity spoofing. Extension + automation may still be detected by third-party bot tests — that is informational, not a bug to “fix” with stealth.
89
+
90
+ ## “Full source audit” bar (third-party)
91
+
92
+ If you integrate external packages that touch the bridge or extension, trace every path from tool call to network/browser API.
@@ -0,0 +1,30 @@
1
+ # Quoting URLs and text on the command line
2
+
3
+ The shell reads your line before `browser-agent` does. A URL such as
4
+ `https://console.agentruntime.io/billing?tab=plans&x=1` contains characters the shell treats as syntax, so the CLI
5
+ can receive a damaged argument, or never run at all. This is the shell's behaviour, not the CLI's.
6
+
7
+ Characters that cause trouble: `&` `?` `%` `#` `;` `|` `*` `$` quotes and spaces.
8
+
9
+ ## Per shell
10
+
11
+ | Shell | Quoting that works | Escape character | Traps |
12
+ |---|---|---|---|
13
+ | **PowerShell** | Single quotes `'...'` are fully literal. A `'` inside is doubled: `'it''s'`. | Backtick, for example `` `& `` | An unquoted `&` is a parse error (it is the call operator). Double quotes expand `$var`. In Windows PowerShell 5.1 a `"` inside an argument passed to a program is mangled, so avoid inner double quotes. |
14
+ | **cmd.exe** | Double quotes `"..."`. Single quotes are not quotes. | Caret `^`, for example `^&` | An unquoted `&` splits the line into two commands. `%NAME%` is expanded (write `%%` inside batch files). |
15
+ | **bash / Git Bash / zsh** | Single quotes `'...'` are literal. A `'` inside is written `'\''`. Double quotes still expand `$`, backtick and (bash) `!`. | Backslash, for example `\&` | An unquoted `&` sends the command to the background. In zsh an unquoted `?` fails with "no matches found". `#` after a space starts a comment. |
16
+
17
+ Rule of thumb: put the URL in single quotes in PowerShell, bash and zsh, and in double quotes in cmd.
18
+
19
+ ## Ways that avoid quoting in any shell
20
+
21
+ 1. **Interactive shell.** `browser-agent shell` gives a `browser-agent>` prompt with its own parser. Type
22
+ `open https://example.com/a?x=1&y=2` as plain text. Double or single quotes group words, and a backslash escapes
23
+ a space or quote. Windows paths such as `C:\temp\x.jpg` are left alone.
24
+ 2. **Clipboard.** `browser-agent open --clip` and `browser-agent nav --clip` read the URL from the clipboard.
25
+ 3. **Stdin.** `browser-agent open -` and `nav -` read the URL from standard input. Feed it from a literal here-string
26
+ so nothing is interpreted:
27
+ - PowerShell: `@'` newline URL newline `'@ | browser-agent open -`
28
+ - bash: `browser-agent open - <<'EOF'` newline URL newline `EOF`
29
+
30
+ Do not "fix" a URL by encoding `&` as `%26`: that changes its meaning (the `&` separates query parameters).