barebrowse 0.25.0 → 0.26.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,43 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.26.0] - 2026-10-27
4
+
5
+ ### Added
6
+
7
+ - **Opt-in link URLs in snapshots.** `links: true` adds each link's absolute URL
8
+ after its name, in every mode: `link "name" <absolute-url> [ref=N]`. Headings
9
+ that wrap a link keep its URL, and in data mode so do group items and table
10
+ cells. `javascript:` links and same-page `#fragment` links are skipped, and
11
+ spaces and `<>` in a URL are percent-encoded. It is available as
12
+ `snapshot({ links: true })`, in `browse()` and `connect()`, as the `links`
13
+ argument of the MCP `browse` and `snapshot` tools, as `--links` on
14
+ `barebrowse snapshot` and `barebrowse browse`, and in the bareagent tools.
15
+ Off by default, and the output is unchanged when it is off. The data-mode
16
+ legend gains a links note only when links are on.
17
+ - **`goto()` now reports where it landed.** It resolves `{ url, status }`: the
18
+ final URL after redirects and the main document's HTTP status, on both
19
+ engines. `page.lastNavigation` holds the same value and is reset to `null`
20
+ after `goBack()`, `goForward()`, `reload()` and a tab switch. After a non-2xx
21
+ `goto()`, the next `snapshot()` and the `browse()` output start with
22
+ `[HTTP n — the page may be an error page]`. An HTTP 4xx or 5xx is reported,
23
+ never thrown, including an error response with an empty body. The MCP and
24
+ CLI `goto` print `ok (HTTP n)` for a non-2xx page.
25
+ - **Network failures reject with a typed error.** An unreachable host (DNS
26
+ failure, connection refused, TLS error and the like) rejects with
27
+ `err.code === 'NAV_FAILED'` and the browser's raw message in `err.reason`.
28
+ Behaviour change: on Chromium, `goto()` and `browse()` to an unreachable host
29
+ used to resolve with the browser's own error page; they now reject. Firefox
30
+ already rejected, with an untyped error; it is now the same typed error.
31
+
32
+ ### Fixed
33
+
34
+ - **A download no longer makes `goto()` wait 30s and time out on Chromium.**
35
+ Navigating to a URL that turns into a download, or to a 204 response, returns
36
+ at once. A 204 reports status 204 on both engines.
37
+ - **`barebrowse install --skill` works from an npm install.** The skill file
38
+ was missing from the published package, so the command failed outside a
39
+ checkout.
40
+
3
41
  ## [0.25.0] - 2026-10-26
4
42
 
5
43
  ### Added
package/README.md CHANGED
@@ -104,6 +104,8 @@ MCP tools: `browse`, `goto`, `snapshot`, `readable`, `click`, `type`, `press`, `
104
104
 
105
105
  `browse` and `snapshot` accept `pruneMode: 'act'|'read'|'data'`. `act` (default) keeps interactive elements — best for clicking/filling. `read` keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. If act-mode collapses a content-heavy page near-totally, the snapshot includes a `hint: …` line suggesting `pruneMode='read'` so the agent doesn't bail to a separate HTTP fetch. `data` is one line per record — tables with headers, repeated items grouped; for data pages, not articles. Output starts with a one-line legend (`# data mode: …`): table rows read `row "<label>": <column>: <value> · …`, groups read `<label>: a, b, c`, and a title line followed by detail lines is one block, up to the next title line. It covers the main landmark only (nav, header and pagination outside `<main>` are dropped): use `act` for menus and page controls.
106
106
 
107
+ `links: true` (opt-in, `snapshot({ links: true })`, `browse(url, { links: true })`, or the `connect({ links: true })` default; MCP/bareagent `links` param, CLI `snapshot --links` / `browse <url> --links`) adds each link's absolute URL right after its quoted name: `- link "Fix login bug" <https://github.com/o/r/issues/42> [ref=12]`. Works in every mode; a heading that wraps a link keeps the URL on its own line (`heading "title" <url> [level=3] [ref=N]`); data mode also puts it after link cells in table rows and each item of a collapsed group (`Labels: bug <url> [ref=1], docs <url> [ref=2]`) and its legend gains ` Links: "name" <url>.`. `javascript:` and same-page `#fragment` links get no URL, every link line, row cell and group item carries its own URL (nothing is deduped across them, even when two links share a URL), and URLs never contain a raw space, `<` or `>`. Off by default: output is unchanged and costs nothing (about +23% chars on a GitHub issue list in data mode).
108
+
107
109
  `npx barebrowse doctor` reports the runtime environment first — default engine/mode, installed Chromium/Firefox binaries, cookie sources, and any live session's engine + capabilities — then scans every known MCP config location and flags scope conflicts. `npx barebrowse install --force` overwrites an existing entry pointing at a different endpoint.
108
110
 
109
111
  ### 3. Library -- for agentic automation
@@ -138,6 +140,25 @@ const snap = await page.snapshot();
138
140
  await page.close(); // closes only the tab barebrowse opened — your browser keeps running
139
141
  ```
140
142
 
143
+ ### Did the navigation work?
144
+
145
+ `goto()` resolves to `{ url, status }` — the final URL after redirects and the
146
+ main document's HTTP status (`0` when no HTTP response was observed, e.g. an aborted navigation).
147
+ `page.lastNavigation` returns the same object (`null` before any goto, and reset
148
+ to `null` by `goBack`/`goForward`/`reload`/`switchTab` — status is reported for
149
+ `goto()` only, so those never leave a stale warning). A
150
+ non-2xx page still loads, and its snapshot (and `browse()` output) carries a
151
+ `[HTTP 404 — the page may be an error page]` line. If the browser cannot
152
+ reach a document at all (DNS failure, connection refused, TLS error) `goto()`
153
+ and `browse()` reject with an Error whose `code` is `'NAV_FAILED'` and `reason`
154
+ the browser's raw string (`net::ERR_NAME_NOT_RESOLVED`, `NS_ERROR_UNKNOWN_HOST`).
155
+ A download or a 204 is not a failure. Through MCP and the CLI, `goto` prints
156
+ `ok` for 2xx and `ok (HTTP 404)` otherwise.
157
+
158
+ ```js
159
+ const { url, status } = await page.goto('https://example.com/gone');
160
+ ```
161
+
141
162
  ### Firefox (WebDriver BiDi)
142
163
 
143
164
  CDP is deprecated in Firefox, so barebrowse drives it over the W3C WebDriver
@@ -50,6 +50,7 @@ const snapshot = await browse('https://example.com', {
50
50
  browser: 'firefox', // cookie source: 'firefox' | 'chromium' (auto-detected)
51
51
  prune: true, // apply ARIA pruning (47-95% token reduction)
52
52
  pruneMode: 'act', // 'act' (interactive elements) | 'read' (all content) | 'data' (one line per record)
53
+ links: false, // default for snapshot({ links }): link URLs after each link name
53
54
  consent: true, // auto-dismiss cookie consent dialogs
54
55
  blockAds: true, // block 128 ad/tracker URL patterns (default on for owned browsers)
55
56
  blockUrls: [], // extra URL globs to block (merged with the default)
@@ -65,11 +66,12 @@ const snapshot = await browse('https://example.com', {
65
66
 
66
67
  | Method | Args | Returns | Notes |
67
68
  |---|---|---|---|
68
- | `goto(url, timeout?)` | url: string, timeout: number (default 30000) | void | Navigate + wait for load + dismiss consent |
69
+ | `goto(url, timeout?)` | url: string, timeout: number (default 30000) | `{ url, status }` | Navigate + wait for load + dismiss consent. Resolves to the final URL (after redirects) and the main document's HTTP status (`0` = no HTTP response observed, e.g. an aborted navigation). Non-2xx still resolves; `snapshot()` then carries `[HTTP 404 — the page may be an error page]`. Network failure (DNS, refused, TLS) rejects with `err.code === 'NAV_FAILED'` and `err.reason` (raw browser string, e.g. `net::ERR_NAME_NOT_RESOLVED` / `NS_ERROR_UNKNOWN_HOST`). A download or 204 (`ERR_ABORTED` / `NS_BINDING_ABORTED`) is not a failure and returns without waiting for a load event. `browse()` still returns a string (warning line prepended when non-2xx) and rejects with the same `NAV_FAILED`. |
70
+ | `lastNavigation` | -- | `{ url, status }` or null | Outcome of the latest `goto()`; null before any goto and while a goto is failing. Reset to null by `reload`/`goBack`/`goForward`/`switchTab` (status is reported for `goto()` only) |
69
71
  | `goBack()` | -- | void | Navigate back in browser history |
70
72
  | `goForward()` | -- | void | Navigate forward in browser history |
71
73
  | `reload(opts?)` | { ignoreCache?: boolean, timeout?: number } | void | Reload the current page. Clears refMap (refs from pre-reload reject). |
72
- | `snapshot(pruneOpts?)` | false or { mode: 'act'\|'read'\|'data' } | string | ARIA tree with `[ref=N]` markers. Pass `false` for raw. |
74
+ | `snapshot(pruneOpts?)` | false or { mode: 'act'\|'read'\|'data', links?: boolean } | string | ARIA tree with `[ref=N]` markers. Pass `false` for raw. `links: true` adds `<absolute-url>` after each link name (see below). |
73
75
  | `readable()` | -- | object | Clean article text (Reader-View engine). `{ ok, title, byline, text, length, confidence: 'high'\|'low', readerable, hint? }` or `{ ok: false, hint }`. For *reading*, not interacting — see note below. |
74
76
  | `click(ref)` | ref: string | void | Scroll into view + mouse press+release at center |
75
77
  | `type(ref, text, opts?)` | ref: string, text: string, opts: { clear?, keyEvents? } | void | Focus + insert text. `clear: true` replaces existing. |
@@ -304,7 +306,9 @@ Action tools return `'ok'` -- the agent calls `snapshot` explicitly to observe.
304
306
 
305
307
  `browse` and `snapshot` also accept `pruneMode: 'act'|'read'|'data'`. `act` (the default) keeps interactive elements and short labels — best for clicking/filling. `read` keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. `data` (opt-in) is one line per record — tables with headers (`row "Label": Header: value · …`), repeated items grouped (`Label: a, b, c`) — for data pages (timetables, listings, results), not articles; every data snapshot starts with ONE legend line (`# data mode: one line per record. Tables: row "<label>": <column>: <value> · … Groups: <label>: a, b, c. A title line followed by detail lines is one block, up to the next title line. Main content only; use mode "act" for menus and page controls.`) so you can read it without docs; it keeps `[ref=N]` on interactive nodes (clickable like act) and is *larger* than act on prose pages — use `read`/`readable()` there. Same surface on the bareagent adapter. If act mode collapses a content-heavy page (raw > 5 KB → pruned < 500 chars AND < 5% of raw), the result includes a `hint: act mode dropped most of the page — retry with pruneMode='read' …` line between the stats and the tree so the caller knows to re-snapshot in read mode instead of bailing to a separate HTTP fetch.
306
308
 
307
- Session runs in hybrid mode (headless with automatic headed fallback on bot detection). `goto` injects cookies from the user's browser before navigation for authenticated access.
309
+ `links: true` (opt-in, `snapshot({ links: true })`, `browse(url, { links: true })`, or the `connect({ links: true })` default; MCP/bareagent `links` param, CLI `snapshot --links` / `browse <url> --links`) adds each link's absolute URL right after its quoted name: `- link "Fix login bug" <https://github.com/o/r/issues/42> [ref=12]`. Works in every mode; a heading that wraps a link keeps the URL on its own line (`heading "title" <url> [level=3] [ref=N]`); data mode also puts it after link cells in table rows and each item of a collapsed group (`Labels: bug <url> [ref=1], docs <url> [ref=2]`) and its legend gains ` Links: "name" <url>.`. `javascript:` and same-page `#fragment` links get no URL, every link line, row cell and group item carries its own URL (nothing is deduped across them, even when two links share a URL), and URLs never contain a raw space, `<` or `>`. Off by default: output is unchanged and costs nothing (about +23% chars on a GitHub issue list in data mode).
310
+
311
+ `goto` answers `ok` for 2xx and `ok (HTTP 404)` for other statuses (the next `snapshot` starts with an `[HTTP 404 …]` warning line); an unreachable host is an error. Session runs in hybrid mode (headless with automatic headed fallback on bot detection). `goto` injects cookies from the user's browser before navigation for authenticated access.
308
312
 
309
313
  Session tools share a singleton page, lazy-created on first use. All session tools have auto-retry on transient failures (browser crash, WebSocket close, navigation timeout) on a per-tool deadline (v0.9.0 H5): `goto`/`reload`/`wait_for` 60s, `back`/`forward` 30s, interactive ops (`click`/`type`/`press`/`scroll`/`hover`/`select`/`drag`/`snapshot`/`eval`) 15s, `tabs` 5s, heavy I/O (`pdf`/`screenshot`/`upload`) 45s — replaces the prior blanket 30s. Session resets between attempts. Idempotent tools retry once; mutating tools (`click`/`type`/`upload`/etc.) `{ retry: false }` so partial first attempts don't replay on a fresh page. Scroll accepts `direction: "up"/"down"` in addition to numeric `deltaY`. Click falls back to JS `.click()` when elements have no layout. `browse` has a 60s timeout (no retry — stateless). Assess tries headless first; if bot-blocked, retries headed. Browser OOM/crash auto-recovers (session resets, server stays alive).
310
314
 
package/cli.js CHANGED
@@ -37,7 +37,7 @@ if (args.includes('--daemon-internal')) {
37
37
  } else if (cmd === 'goto' && args[1]) {
38
38
  await cmdProxy('goto', { url: args[1], timeout: parseFlag('--timeout') });
39
39
  } else if (cmd === 'snapshot') {
40
- await cmdProxy('snapshot', { mode: parseFlag('--mode') });
40
+ await cmdProxy('snapshot', { mode: parseFlag('--mode'), links: hasFlag('--links') || undefined });
41
41
  } else if (cmd === 'readable') {
42
42
  await cmdProxy('readable');
43
43
  } else if (cmd === 'screenshot') {
@@ -185,7 +185,7 @@ async function cmdProxy(command, cmdArgs) {
185
185
  try { unlinkSync(sessionPath); } catch { /* already gone */ }
186
186
  process.stdout.write('Session closed.\n');
187
187
  } else {
188
- process.stdout.write('ok\n');
188
+ process.stdout.write(result.status ? `ok (HTTP ${result.status})\n` : 'ok\n');
189
189
  }
190
190
  } catch (err) {
191
191
  if (command === 'close') {
@@ -203,9 +203,9 @@ async function cmdProxy(command, cmdArgs) {
203
203
  async function oneShot() {
204
204
  const { browse } = await import('./src/index.js');
205
205
  const url = args[1];
206
- const mode = args[2] || 'headless';
206
+ const mode = args[2] && !args[2].startsWith('--') ? args[2] : 'headless';
207
207
  try {
208
- const snapshot = await browse(url, { mode, incognito: hasFlag('--incognito') || undefined });
208
+ const snapshot = await browse(url, { mode, incognito: hasFlag('--incognito') || undefined, links: hasFlag('--links') || undefined });
209
209
  process.stdout.write(snapshot + '\n');
210
210
  process.exit(0);
211
211
  } catch (err) {
@@ -585,7 +585,7 @@ Navigation:
585
585
  barebrowse back Go back in history
586
586
  barebrowse forward Go forward in history
587
587
  barebrowse reload [--no-cache] Reload current page
588
- barebrowse snapshot [--mode=M] ARIA snapshot -> .barebrowse/page-*.yml
588
+ barebrowse snapshot [--mode=M] [--links] ARIA snapshot -> .barebrowse/page-*.yml (--links: link URLs)
589
589
  barebrowse readable Clean article text -> .barebrowse/article-*.txt
590
590
  barebrowse screenshot [--format] Screenshot -> .barebrowse/screenshot-*.png
591
591
  barebrowse pdf [--landscape] PDF export -> .barebrowse/page-*.pdf
@@ -620,7 +620,7 @@ Debugging:
620
620
  barebrowse switch-engine <chromium|firefox> Swap the browser in place (page, tabs, refs are lost)
621
621
 
622
622
  One-shot:
623
- barebrowse browse <url> [mode] Browse + print snapshot to stdout
623
+ barebrowse browse <url> [mode] [--links] Browse + print snapshot to stdout
624
624
 
625
625
  MCP:
626
626
  barebrowse mcp Start MCP server (JSON-RPC over stdio)
@@ -0,0 +1,141 @@
1
+ ---
2
+ name: barebrowse
3
+ description: Browser automation using the user's real browser with real cookies. Handles consent walls, login sessions, and bot detection automatically.
4
+ allowed-tools: Bash(barebrowse:*)
5
+ ---
6
+
7
+ # barebrowse CLI — Browser Automation for Agents
8
+
9
+ Browse any URL using the user's real browser with real cookies. Returns pruned ARIA snapshots (40-90% smaller than raw) with `[ref=N]` markers for interaction. Handles cookie consent, login sessions, JS dialogs, and bot detection automatically.
10
+
11
+ ## Quick Start
12
+
13
+ ```bash
14
+ barebrowse open https://example.com # Start session + navigate
15
+ barebrowse snapshot # Get ARIA snapshot → .barebrowse/page-*.yml
16
+ barebrowse click 8 # Click element with ref=8
17
+ barebrowse snapshot # See result
18
+ barebrowse close # End session
19
+ ```
20
+
21
+ All output files go to `.barebrowse/` in the current directory. Read them with the Read tool when needed. Output files older than 7 days are removed automatically at `open`; saved state files (`state-*.json`) are kept.
22
+
23
+ ## Commands
24
+
25
+ ### Session Lifecycle
26
+
27
+ | Command | Description |
28
+ |---------|-------------|
29
+ | `barebrowse open [url] [flags]` | Start browser session. Optionally navigate to URL. |
30
+ | `barebrowse close` | Close session and kill browser. |
31
+ | `barebrowse status` | Check if session is running. |
32
+
33
+ **Open flags:**
34
+ - `--mode=headless|headed|hybrid` — Browser mode (default: headless)
35
+ - `--no-cookies` — Skip cookie injection
36
+ - `--browser=firefox|chromium` — Cookie source
37
+ - `--prune-mode=act|read|data` — Default pruning mode
38
+ - `--timeout=N` — Navigation timeout in ms
39
+ - `--proxy=URL` — HTTP/SOCKS proxy server
40
+ - `--viewport=WxH` — Viewport size (e.g. 1280x720)
41
+ - `--storage-state=FILE` — Load cookies/localStorage from JSON file
42
+ - `--block-private-network` — SSRF guard: refuse loopback / RFC-1918 / link-local / cloud-metadata hosts (v0.11.0)
43
+ - `--upload-dir=DIR` — Sandbox uploads to DIR; reject files outside it (v0.11.0)
44
+
45
+ > Security (v0.11.0): `file:`/`chrome:`/etc. navigation is blocked by default, and the daemon requires a per-session token (handled transparently by the CLI). Snapshots and saved state are written owner-only (`0600`).
46
+
47
+ ### Navigation
48
+
49
+ | Command | Output |
50
+ |---------|--------|
51
+ | `barebrowse goto <url>` | Navigates, waits for load, dismisses consent. Prints "ok", or "ok (HTTP 404)" for a non-2xx page (the next snapshot starts with an `[HTTP 404 …]` warning). Unreachable host: error. |
52
+ | `barebrowse back` | Go back in browser history. |
53
+ | `barebrowse forward` | Go forward in browser history. |
54
+ | `barebrowse snapshot` | ARIA snapshot → `.barebrowse/page-<timestamp>.yml` |
55
+ | `barebrowse snapshot --mode=read` | Read mode: keeps all text (for content extraction) |
56
+ | `barebrowse snapshot --mode=data` | Data mode: one line per record (table rows with headers, repeated items grouped). For data pages, not articles. Main content only; use `act` for menus and page controls. |
57
+ | `barebrowse snapshot --links` | Add each link's absolute URL after its name: `link "title" <https://…> [ref=N]` (combine with `--mode`; also on `browse <url> --links`). `javascript:` and same-page `#` links get none. |
58
+ | `barebrowse readable` | Clean article text → `.barebrowse/article-<timestamp>.txt`. For reading/summarising articles (news, blogs, docs); falls back to snapshot on non-articles. |
59
+ | `barebrowse screenshot` | Screenshot → `.barebrowse/screenshot-<timestamp>.png` |
60
+ | `barebrowse pdf [--landscape]` | PDF export → `.barebrowse/page-<timestamp>.pdf` |
61
+
62
+ ### Interaction
63
+
64
+ | Command | Description |
65
+ |---------|-------------|
66
+ | `barebrowse click <ref>` | Click element (scrolls into view first) |
67
+ | `barebrowse type <ref> <text>` | Type text into element |
68
+ | `barebrowse fill <ref> <text>` | Clear existing content + type new text |
69
+ | `barebrowse press <key>` | Press key: Enter, Tab, Escape, Backspace, Delete, arrows, Space |
70
+ | `barebrowse scroll <deltaY>` | Scroll page (positive=down, negative=up) |
71
+ | `barebrowse hover <ref>` | Hover over element (triggers tooltips) |
72
+ | `barebrowse select <ref> <value>` | Select dropdown option |
73
+ | `barebrowse drag <fromRef> <toRef>` | Drag element to another element |
74
+ | `barebrowse upload <ref> <files..>` | Upload file(s) to a file input element |
75
+
76
+ ### Tabs
77
+
78
+ | Command | Description |
79
+ |---------|-------------|
80
+ | `barebrowse tabs` | List open tabs (index, url, title) |
81
+ | `barebrowse tab <index>` | Switch to tab by index |
82
+
83
+ ### Debugging
84
+
85
+ | Command | Output |
86
+ |---------|--------|
87
+ | `barebrowse eval <expression>` | Evaluate JS in page, print result |
88
+ | `barebrowse wait-idle` | Wait for network idle (no requests for 500ms) |
89
+ | `barebrowse wait-for [opts]` | Wait for content to appear on page |
90
+ | `barebrowse console-logs` | Console logs → `.barebrowse/console-<timestamp>.json` |
91
+ | `barebrowse network-log` | Network log → `.barebrowse/network-<timestamp>.json` |
92
+ | `barebrowse network-log --failed` | Only failed/4xx/5xx requests |
93
+ | `barebrowse dialog-log` | JS dialog log → `.barebrowse/dialogs-<timestamp>.json` |
94
+ | `barebrowse save-state` | Cookies + localStorage → `.barebrowse/state-<timestamp>.json` |
95
+ | `barebrowse switch-engine <chromium\|firefox>` | Swap the session's browser engine in place; page, tabs and refs are lost, so `goto` again |
96
+
97
+ **wait-for flags:**
98
+ - `--text=STRING` — Wait for text to appear in page body
99
+ - `--selector=CSS` — Wait for CSS selector to match
100
+ - `--timeout=N` — Max wait time in ms (default: 30000)
101
+
102
+ ## Snapshot Format
103
+
104
+ The snapshot is a YAML-like ARIA tree. Each line is one node:
105
+
106
+ ```
107
+ # https://example.com/
108
+ # 379 chars → 45 chars (88% pruned)
109
+ - heading "Example Domain" [level=1] [ref=3]
110
+ ```
111
+
112
+ - `[ref=N]` — Use this number with click, type, fill, hover, select, drag, upload
113
+ - Refs change on every snapshot — always take a fresh snapshot before interacting
114
+ - **act mode** (default): interactive elements + labels — for clicking, typing, navigating
115
+ - **read mode**: all text content — for reading articles, extracting data
116
+
117
+ ## Workflow Pattern
118
+
119
+ 1. `barebrowse open <url>` — start session
120
+ 2. `barebrowse snapshot` — observe page (read the .yml file)
121
+ 3. Decide action based on snapshot content
122
+ 4. `barebrowse click/type/fill/press/scroll/drag/upload <ref>` — act
123
+ 5. `barebrowse snapshot` — observe result (refs are now different!)
124
+ 6. Repeat 3-5 until goal achieved
125
+ 7. `barebrowse close` — clean up
126
+
127
+ ## Tips
128
+
129
+ - **Always snapshot before interacting** — refs are ephemeral and change every time
130
+ - **Use `fill` instead of `type`** when replacing existing text in input fields
131
+ - **Use `--mode=read`** for snapshot when you need to extract article content or data
132
+ - **Use `back`/`forward`** to navigate browser history instead of re-entering URLs
133
+ - **Use `upload`** for file inputs — pass absolute paths to the files
134
+ - **Use `wait-for`** when content loads asynchronously — more reliable than `wait-idle`
135
+ - **Check `dialog-log`** if JS alerts/confirms were auto-dismissed during your session
136
+ - **Use `save-state`** to persist cookies/localStorage for later sessions via `--storage-state`
137
+ - **Check `console-logs`** when page behavior seems wrong — JS errors show up there
138
+ - **Check `network-log --failed`** to debug missing content or broken API calls
139
+ - **Use `eval`** as an escape hatch when ARIA tree doesn't show what you need
140
+ - **One session per project** — `.barebrowse/` is project-scoped
141
+ - For bot-detected sites, use `--mode=headed` (requires browser with `--remote-debugging-port=9222`)
@@ -0,0 +1,140 @@
1
+ ---
2
+ name: barebrowse
3
+ description: Browser automation using the user's real browser with real cookies. Handles consent walls, login sessions, and bot detection automatically.
4
+ allowed-tools: Bash(barebrowse:*)
5
+ ---
6
+ # barebrowse CLI — Browser Automation for Agents
7
+
8
+ Browse any URL using the user's real browser with real cookies. Returns pruned ARIA snapshots (40-90% smaller than raw) with `[ref=N]` markers for interaction. Handles cookie consent, login sessions, JS dialogs, and bot detection automatically.
9
+
10
+ ## Quick Start
11
+
12
+ ```bash
13
+ barebrowse open https://example.com # Start session + navigate
14
+ barebrowse snapshot # Get ARIA snapshot → .barebrowse/page-*.yml
15
+ barebrowse click 8 # Click element with ref=8
16
+ barebrowse snapshot # See result
17
+ barebrowse close # End session
18
+ ```
19
+
20
+ All output files go to `.barebrowse/` in the current directory. Read them with the Read tool when needed. Output files older than 7 days are removed automatically at `open`; saved state files (`state-*.json`) are kept.
21
+
22
+ ## Commands
23
+
24
+ ### Session Lifecycle
25
+
26
+ | Command | Description |
27
+ |---------|-------------|
28
+ | `barebrowse open [url] [flags]` | Start browser session. Optionally navigate to URL. |
29
+ | `barebrowse close` | Close session and kill browser. |
30
+ | `barebrowse status` | Check if session is running. |
31
+
32
+ **Open flags:**
33
+ - `--mode=headless|headed|hybrid` — Browser mode (default: headless)
34
+ - `--no-cookies` — Skip cookie injection
35
+ - `--browser=firefox|chromium` — Cookie source
36
+ - `--prune-mode=act|read|data` — Default pruning mode
37
+ - `--timeout=N` — Navigation timeout in ms
38
+ - `--proxy=URL` — HTTP/SOCKS proxy server
39
+ - `--viewport=WxH` — Viewport size (e.g. 1280x720)
40
+ - `--storage-state=FILE` — Load cookies/localStorage from JSON file
41
+ - `--block-private-network` — SSRF guard: refuse loopback / RFC-1918 / link-local / cloud-metadata hosts (v0.11.0)
42
+ - `--upload-dir=DIR` — Sandbox uploads to DIR; reject files outside it (v0.11.0)
43
+
44
+ > Security (v0.11.0): `file:`/`chrome:`/etc. navigation is blocked by default, and the daemon requires a per-session token (handled transparently by the CLI). Snapshots and saved state are written owner-only (`0600`).
45
+
46
+ ### Navigation
47
+
48
+ | Command | Output |
49
+ |---------|--------|
50
+ | `barebrowse goto <url>` | Navigates, waits for load, dismisses consent. Prints "ok", or "ok (HTTP 404)" for a non-2xx page (the next snapshot starts with an `[HTTP 404 …]` warning). Unreachable host: error. |
51
+ | `barebrowse back` | Go back in browser history. |
52
+ | `barebrowse forward` | Go forward in browser history. |
53
+ | `barebrowse snapshot` | ARIA snapshot → `.barebrowse/page-<timestamp>.yml` |
54
+ | `barebrowse snapshot --mode=read` | Read mode: keeps all text (for content extraction) |
55
+ | `barebrowse snapshot --mode=data` | Data mode: one line per record (table rows with headers, repeated items grouped). For data pages, not articles. Main content only; use `act` for menus and page controls. |
56
+ | `barebrowse snapshot --links` | Add each link's absolute URL after its name: `link "title" <https://…> [ref=N]` (combine with `--mode`; also on `browse <url> --links`). `javascript:` and same-page `#` links get none. |
57
+ | `barebrowse readable` | Clean article text → `.barebrowse/article-<timestamp>.txt`. For reading/summarising articles (news, blogs, docs); falls back to snapshot on non-articles. |
58
+ | `barebrowse screenshot` | Screenshot → `.barebrowse/screenshot-<timestamp>.png` |
59
+ | `barebrowse pdf [--landscape]` | PDF export → `.barebrowse/page-<timestamp>.pdf` |
60
+
61
+ ### Interaction
62
+
63
+ | Command | Description |
64
+ |---------|-------------|
65
+ | `barebrowse click <ref>` | Click element (scrolls into view first) |
66
+ | `barebrowse type <ref> <text>` | Type text into element |
67
+ | `barebrowse fill <ref> <text>` | Clear existing content + type new text |
68
+ | `barebrowse press <key>` | Press key: Enter, Tab, Escape, Backspace, Delete, arrows, Space |
69
+ | `barebrowse scroll <deltaY>` | Scroll page (positive=down, negative=up) |
70
+ | `barebrowse hover <ref>` | Hover over element (triggers tooltips) |
71
+ | `barebrowse select <ref> <value>` | Select dropdown option |
72
+ | `barebrowse drag <fromRef> <toRef>` | Drag element to another element |
73
+ | `barebrowse upload <ref> <files..>` | Upload file(s) to a file input element |
74
+
75
+ ### Tabs
76
+
77
+ | Command | Description |
78
+ |---------|-------------|
79
+ | `barebrowse tabs` | List open tabs (index, url, title) |
80
+ | `barebrowse tab <index>` | Switch to tab by index |
81
+
82
+ ### Debugging
83
+
84
+ | Command | Output |
85
+ |---------|--------|
86
+ | `barebrowse eval <expression>` | Evaluate JS in page, print result |
87
+ | `barebrowse wait-idle` | Wait for network idle (no requests for 500ms) |
88
+ | `barebrowse wait-for [opts]` | Wait for content to appear on page |
89
+ | `barebrowse console-logs` | Console logs → `.barebrowse/console-<timestamp>.json` |
90
+ | `barebrowse network-log` | Network log → `.barebrowse/network-<timestamp>.json` |
91
+ | `barebrowse network-log --failed` | Only failed/4xx/5xx requests |
92
+ | `barebrowse dialog-log` | JS dialog log → `.barebrowse/dialogs-<timestamp>.json` |
93
+ | `barebrowse save-state` | Cookies + localStorage → `.barebrowse/state-<timestamp>.json` |
94
+ | `barebrowse switch-engine <chromium\|firefox>` | Swap the session's browser engine in place; page, tabs and refs are lost, so `goto` again |
95
+
96
+ **wait-for flags:**
97
+ - `--text=STRING` — Wait for text to appear in page body
98
+ - `--selector=CSS` — Wait for CSS selector to match
99
+ - `--timeout=N` — Max wait time in ms (default: 30000)
100
+
101
+ ## Snapshot Format
102
+
103
+ The snapshot is a YAML-like ARIA tree. Each line is one node:
104
+
105
+ ```
106
+ # https://example.com/
107
+ # 379 chars → 45 chars (88% pruned)
108
+ - heading "Example Domain" [level=1] [ref=3]
109
+ ```
110
+
111
+ - `[ref=N]` — Use this number with click, type, fill, hover, select, drag, upload
112
+ - Refs change on every snapshot — always take a fresh snapshot before interacting
113
+ - **act mode** (default): interactive elements + labels — for clicking, typing, navigating
114
+ - **read mode**: all text content — for reading articles, extracting data
115
+
116
+ ## Workflow Pattern
117
+
118
+ 1. `barebrowse open <url>` — start session
119
+ 2. `barebrowse snapshot` — observe page (read the .yml file)
120
+ 3. Decide action based on snapshot content
121
+ 4. `barebrowse click/type/fill/press/scroll/drag/upload <ref>` — act
122
+ 5. `barebrowse snapshot` — observe result (refs are now different!)
123
+ 6. Repeat 3-5 until goal achieved
124
+ 7. `barebrowse close` — clean up
125
+
126
+ ## Tips
127
+
128
+ - **Always snapshot before interacting** — refs are ephemeral and change every time
129
+ - **Use `fill` instead of `type`** when replacing existing text in input fields
130
+ - **Use `--mode=read`** for snapshot when you need to extract article content or data
131
+ - **Use `back`/`forward`** to navigate browser history instead of re-entering URLs
132
+ - **Use `upload`** for file inputs — pass absolute paths to the files
133
+ - **Use `wait-for`** when content loads asynchronously — more reliable than `wait-idle`
134
+ - **Check `dialog-log`** if JS alerts/confirms were auto-dismissed during your session
135
+ - **Use `save-state`** to persist cookies/localStorage for later sessions via `--storage-state`
136
+ - **Check `console-logs`** when page behavior seems wrong — JS errors show up there
137
+ - **Check `network-log --failed`** to debug missing content or broken API calls
138
+ - **Use `eval`** as an escape hatch when ARIA tree doesn't show what you need
139
+ - **One session per project** — `.barebrowse/` is project-scoped
140
+ - For bot-detected sites, use `--mode=headed` (requires browser with `--remote-debugging-port=9222`)
package/mcp-server.js CHANGED
@@ -14,6 +14,7 @@ import { browse, connect } from './src/index.js';
14
14
  import { formatReadable } from './src/readable.js';
15
15
  import { cleanOld } from './src/output-clean.js';
16
16
  import { scopedCookiesForUrl } from './src/auth.js';
17
+ import { isHttpError } from './src/navigation.js';
17
18
  import { mkdirSync, writeFileSync, readFileSync } from 'node:fs';
18
19
  import { join, dirname } from 'node:path';
19
20
  import { pathToFileURL, fileURLToPath } from 'node:url';
@@ -268,6 +269,7 @@ export const TOOLS = [
268
269
  mode: { type: 'string', enum: ['headless', 'headed', 'hybrid'], description: 'Browser mode (default: headless)' },
269
270
  pruneMode: { type: 'string', enum: ['act', 'read', 'data'], description: 'Pruning mode. "act" (default) keeps interactive elements and short labels — best for clicking/filling. "read" keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. If the page is content-heavy and act-mode returns mostly empty, retry with "read". "data": one line per record — tables with headers, repeated items grouped; for data pages, not articles. Output starts with a legend line; read it as: table rows `row "<label>": <column>: <value> · …`, groups `<label>: a, b, c`, and a title line followed by detail lines is one block up to the next title line. Main content only: use "act" for menus and page controls.' },
270
271
  incognito: { type: 'boolean', description: 'Clean, unauthenticated session: skip cookie injection so the page loads logged-out (default: false).' },
272
+ links: { type: 'boolean', description: 'Show each link\'s absolute URL after its name: link "name" <url> [ref=N]. Default off.' },
271
273
  maxChars: { type: 'number', description: 'Max chars to return inline. Larger snapshots are saved to .barebrowse/ and a file path is returned instead. Default: 30000.' },
272
274
  },
273
275
  required: ['url'],
@@ -275,7 +277,7 @@ export const TOOLS = [
275
277
  },
276
278
  {
277
279
  name: 'goto',
278
- description: 'Open URL in a persistent interactive browser session (pair with snapshot/click/type/press for multi-step flows). Use when the task needs clicking, typing, or form submission. Injects auth cookies. Returns ok — call snapshot to observe.',
280
+ description: 'Open URL in a persistent interactive browser session (pair with snapshot/click/type/press for multi-step flows). Use when the task needs clicking, typing, or form submission. Injects auth cookies. Returns ok — call snapshot to observe. A non-2xx page answers `ok (HTTP 404)` and the snapshot starts with an [HTTP …] warning line; an unreachable host (DNS, refused, TLS) is an error.',
279
281
  inputSchema: {
280
282
  type: 'object',
281
283
  properties: {
@@ -291,6 +293,7 @@ export const TOOLS = [
291
293
  type: 'object',
292
294
  properties: {
293
295
  pruneMode: { type: 'string', enum: ['act', 'read', 'data'], description: 'Pruning mode. "act" (default) keeps interactive elements and short labels — best for clicking/filling. "read" keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. If a previous snapshot looked empty on a content-heavy page, retry with "read". "data": one line per record — tables with headers, repeated items grouped; for data pages, not articles. Output starts with a legend line; read it as: table rows `row "<label>": <column>: <value> · …`, groups `<label>: a, b, c`, and a title line followed by detail lines is one block up to the next title line. Main content only: use "act" for menus and page controls.' },
296
+ links: { type: 'boolean', description: 'Show each link\'s absolute URL after its name: link "name" <url> [ref=N]. Default off.' },
294
297
  maxChars: { type: 'number', description: 'Max chars to return inline. Larger snapshots are saved to .barebrowse/ and a file path is returned instead. Default: 30000.' },
295
298
  },
296
299
  },
@@ -544,7 +547,7 @@ async function handleToolCall(name, args) {
544
547
  case 'browse': {
545
548
  let timer;
546
549
  const text = await Promise.race([
547
- browse(args.url, { mode: args.mode, pruneMode: args.pruneMode, incognito: args.incognito }),
550
+ browse(args.url, { mode: args.mode, pruneMode: args.pruneMode, incognito: args.incognito, links: args.links }),
548
551
  new Promise((_, rej) => { timer = setTimeout(() => rej(new Error('browse timed out after 60s')), 60000); }),
549
552
  ]);
550
553
  clearTimeout(timer);
@@ -566,12 +569,12 @@ async function handleToolCall(name, args) {
566
569
  try { await page.injectCookies(args.url); } catch (e) { error ??= e.message; }
567
570
  _lastCookies = cookieReport(args.url, { cookies, error });
568
571
  }
569
- await page.goto(args.url);
570
- return 'ok';
572
+ const { status } = await page.goto(args.url);
573
+ return isHttpError(status) ? `ok (HTTP ${status})` : 'ok';
571
574
  }, TIMEOUTS.goto);
572
575
  case 'snapshot': return withRetry(async () => {
573
576
  const page = await getPage();
574
- const text = await page.snapshot(args.pruneMode ? { mode: args.pruneMode } : undefined);
577
+ const text = await page.snapshot(args.pruneMode || args.links ? { mode: args.pruneMode, links: args.links } : undefined);
575
578
  const limit = args.maxChars ?? MAX_CHARS_DEFAULT;
576
579
  if (text.length > limit) {
577
580
  const file = saveSnapshot(text);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "barebrowse",
3
- "version": "0.25.0",
3
+ "version": "0.26.0",
4
4
  "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
5
5
  "repository": {
6
6
  "type": "git",
@@ -28,6 +28,7 @@
28
28
  "src/",
29
29
  "types/",
30
30
  "cli.js",
31
+ "commands/",
31
32
  "mcp-server.js",
32
33
  "barebrowse.context.md",
33
34
  "README.md",
package/src/aria.js CHANGED
@@ -5,6 +5,8 @@
5
5
  * and formats it as readable YAML-like text, similar to Playwright's ariaSnapshot.
6
6
  */
7
7
 
8
+ import { urlsText } from './links.js';
9
+
8
10
  /**
9
11
  * Format a nested ARIA tree as readable text output.
10
12
  *
@@ -37,6 +39,9 @@ export function formatTree(node, depth = 0) {
37
39
  line += ` "${node.name}"`;
38
40
  }
39
41
 
42
+ // Link URLs (opt-in links: true): right after the name, before props and ref
43
+ line += urlsText(node.urls);
44
+
40
45
  // Notable properties that agents care about
41
46
  const props = node.properties || {};
42
47
  const propParts = [];
@@ -166,6 +166,7 @@ const AX_FN = function reconstructAX(base, REF_ATTR) {
166
166
  const exp = el.getAttribute('aria-expanded');
167
167
  if (exp !== null) p.expanded = exp === 'true';
168
168
  if (role === 'heading') p.level = Number(el.getAttribute('aria-level')) || Number(el.tagName[1]) || 2;
169
+ if (role === 'link' && el.href) p.url = el.href; // browser-serialized absolute URL (CDP's `url` property)
169
170
  if (el.getAttribute('aria-selected') === 'true') p.selected = true;
170
171
  if (el.required || el.getAttribute('aria-required') === 'true') p.required = true;
171
172
  if ((role === 'textbox' || role === 'searchbox' || role === 'combobox' || role === 'spinbutton') && el.value) {
package/src/bareagent.js CHANGED
@@ -63,10 +63,11 @@ export function createBrowseTools(opts = {}) {
63
63
  properties: {
64
64
  url: { type: 'string', description: 'URL to browse' },
65
65
  pruneMode: { type: 'string', enum: ['act', 'read', 'data'], description: '"act" (default) for interactive elements only; "read" for paragraphs and long text (articles/docs); "data": one line per record — tables with headers, repeated items grouped; for data pages, not articles. Output starts with a legend line; read it as: table rows `row "<label>": <column>: <value> · …`, groups `<label>: a, b, c`, and a title line followed by detail lines is one block up to the next title line. Main content only: use "act" for menus and page controls.' },
66
+ links: { type: 'boolean', description: 'Show each link\'s absolute URL after its name: link "name" <url> [ref=N]. Default off.' },
66
67
  },
67
68
  required: ['url'],
68
69
  },
69
- execute: async ({ url, pruneMode }) => await browse(url, pruneMode ? { ...opts, pruneMode } : opts),
70
+ execute: async ({ url, pruneMode, links }) => await browse(url, pruneMode || links ? { ...opts, pruneMode, links } : opts),
70
71
  },
71
72
  {
72
73
  name: 'goto',
@@ -87,11 +88,12 @@ export function createBrowseTools(opts = {}) {
87
88
  type: 'object',
88
89
  properties: {
89
90
  pruneMode: { type: 'string', enum: ['act', 'read', 'data'], description: '"act" (default) for interactive elements only; "read" for paragraphs and long text (articles/docs); "data": one line per record — tables with headers, repeated items grouped; for data pages, not articles. Output starts with a legend line; read it as: table rows `row "<label>": <column>: <value> · …`, groups `<label>: a, b, c`, and a title line followed by detail lines is one block up to the next title line. Main content only: use "act" for menus and page controls.' },
91
+ links: { type: 'boolean', description: 'Show each link\'s absolute URL after its name: link "name" <url> [ref=N]. Default off.' },
90
92
  },
91
93
  },
92
- execute: async (/** @type {{ pruneMode?: string }} */ { pruneMode } = {}) => {
94
+ execute: async (/** @type {{ pruneMode?: string, links?: boolean }} */ { pruneMode, links } = {}) => {
93
95
  const page = await getPage();
94
- return await page.snapshot(pruneMode ? { mode: pruneMode } : undefined);
96
+ return await page.snapshot(pruneMode || links ? { mode: pruneMode, links } : undefined);
95
97
  },
96
98
  },
97
99
  {
package/src/daemon.js CHANGED
@@ -13,6 +13,7 @@ import { join, resolve } from 'node:path';
13
13
  import { connect } from './index.js';
14
14
  import { formatReadable } from './readable.js';
15
15
  import { cleanOld } from './output-clean.js';
16
+ import { isHttpError } from './navigation.js';
16
17
 
17
18
  /** Owner-only file write helper — daemon artifacts can hold authenticated content. */
18
19
  function writeFilePrivate(path, data) {
@@ -313,13 +314,13 @@ export async function runDaemon(opts, outputDir, initialUrl) {
313
314
  // Command handlers
314
315
  const handlers = {
315
316
  async goto({ url, timeout }) {
316
- await page.goto(url, timeout || 30000);
317
- return { ok: true };
317
+ const { status } = await page.goto(url, timeout || 30000);
318
+ return isHttpError(status) ? { ok: true, status } : { ok: true };
318
319
  },
319
320
 
320
- async snapshot({ mode }) {
321
+ async snapshot({ mode, links }) {
321
322
  const pruneMode = mode || defaultPruneMode;
322
- const text = await page.snapshot({ mode: pruneMode });
323
+ const text = await page.snapshot({ mode: pruneMode, links });
323
324
  const ts = new Date().toISOString().replace(/[:.]/g, '-');
324
325
  const file = join(absDir, `page-${ts}.yml`);
325
326
  writeFilePrivate(file, text);