barebrowse 0.27.0 → 0.28.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,75 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.28.0] - 2026-11-16
4
+
5
+ ### Added
6
+
7
+ - **Map + zoom for `snapshot()`.** `snapshot({ map: true })` returns an outline of
8
+ the page's big boxes (landmarks, lists, tables, grids, feeds, tabpanels,
9
+ dialogs) with a ref and a size each; `snapshot({ within })` returns one box, or
10
+ several in page order, as if it were the whole page: a ref from the map, a
11
+ role, `role "name"`, or `role after "heading"` for an unnamed box. Works in
12
+ every mode and with `links`, on both engines. Two or more matches return the
13
+ map with the matches marked; none, or a ref whose element is gone, throws. The
14
+ library default is unchanged (byte for byte). MCP `snapshot` gets `map`,
15
+ `within` and `full`; a snapshot over `maxChars` now also returns the map
16
+ inline (the full snapshot is still saved to a file). CLI: `snapshot --map`,
17
+ `--within=<box>` (repeatable), `--full`, `--max-chars=N`.
18
+ - **Map lists card groups.** A grid of alike cards (products, listings, videos)
19
+ that no landmark, list or table role marks now shows in the map as
20
+ `- group ×18 (6.3k chars) [ref=N]`, nested where it sits; `within: <ref>` zooms
21
+ to it and `within: 'group'` selects the groups. Found by structure only (4 or
22
+ more siblings of one role with similar shape and real text), never inside
23
+ banner/navigation/contentinfo, never for a list or table. Only the map and
24
+ `within` output change; default, act, read and data snapshots are unchanged.
25
+ - **MCP and CLI settle once before the first snapshot after a goto.** A page's
26
+ load event is not its last request (Airbnb on Firefox was incomplete 5 of 5
27
+ times; Flights about 1 in 5). The first `snapshot` after a `goto` now waits for
28
+ the network to go quiet (idle 500ms, capped at 3s; about 0.5s on a finished
29
+ page), best-effort: a timeout is not an error. A goto followed by a click pays
30
+ nothing. MCP `snapshot` takes `settle: false` and the CLI `snapshot --no-settle`
31
+ to skip it. The library (`connect()`, `browse()`) is unchanged.
32
+
33
+ ### Fixed
34
+
35
+ - **Chromium: `goto()` to a hash-only change took 30s.** `goto('/ok#a')` then
36
+ `goto('/ok#b')` waited for a load event that never fires; it now returns at once
37
+ with `{ url, status: 0 }`, as Firefox does.
38
+ - **Chromium: refs die when a click or key press navigates to a new document.**
39
+ An old ref after a link click used to hit whatever element now had that number;
40
+ it now throws, like after `goto`. Hash links and `pushState` keep refs valid.
41
+ - **Firefox: clicks on elements below the fold.** `click()`, `hover()` and
42
+ `drag()` threw `move target out of bounds`, so Google's consent interstitial
43
+ was never dismissed on Firefox. The target is scrolled into view first.
44
+ - **Chromium: `goBack()` / `goForward()` took 30s** when the traversal restored
45
+ the page from the back/forward cache or landed on a same-document history
46
+ entry. They now return when the traversal lands.
47
+ - **Chromium: numeric refs.** `click`, `type`, `hover`, `select`, `drag` and
48
+ `upload` accept a ref as a number as well as a string.
49
+ - **Firefox: refs from before a navigation no longer resolve to another
50
+ element.** A stale ref now throws, as on Chromium, including after a page
51
+ script's `history.back()` restores the old document from the bfcache. The
52
+ token that tracks this lives in an isolated BiDi sandbox realm, out of reach
53
+ of page script.
54
+ - **Firefox: table cells and image-only links in act/read snapshots.** Cell,
55
+ column-header and row-header text was dropped, so tables pruned to their name;
56
+ an image-only link had no name (its `alt` was ignored), so data mode dropped
57
+ it. Both now match Chromium.
58
+ - **Firefox: consent accept that navigates.** The snapshot after the click no
59
+ longer reads the consent interstitial before the next page lands (1s settle,
60
+ as on Chromium).
61
+ - **Consent: `goto()` reports the page behind an accept that navigated.** The
62
+ resolved `{ url, status }` is the landed page, not the interstitial.
63
+ - **MCP and CLI: a zoom over `maxChars` says why it came back as a file path**,
64
+ instead of returning the path with no explanation.
65
+ - **Chromium cookie reuse works on KDE.** The KWallet password was
66
+ base64-decoded before key derivation, which made every encrypted (`v11`)
67
+ cookie fail to decrypt and get dropped (0 of 17 on a real jar). It is now used
68
+ as stored, as Chromium does.
69
+ - **Partitioned Chromium cookies are skipped.** A partitioned (CHIPS) cookie
70
+ sharing name, domain and path with the top-level one could shadow it on merge,
71
+ as the Firefox reader already avoided.
72
+
3
73
  ## [0.27.0] - 2026-10-30
4
74
 
5
75
  ### Added
package/README.md CHANGED
@@ -106,6 +106,8 @@ MCP tools: `browse`, `goto`, `snapshot`, `readable`, `click`, `type`, `press`, `
106
106
 
107
107
  `links: true` (opt-in, `snapshot({ links: true })`, `browse(url, { links: true })`, or the `connect({ links: true })` default; MCP/bareagent `links` param, CLI `snapshot --links` / `browse <url> --links`) adds each link's absolute URL right after its quoted name: `- link "Fix login bug" <https://github.com/o/r/issues/42> [ref=12]`. Works in every mode; a heading that wraps a link keeps the URL on its own line (`heading "title" <url> [level=3] [ref=N]`); data mode also puts it after link cells in table rows and each item of a collapsed group (`Labels: bug <url> [ref=1], docs <url> [ref=2]`) and its legend gains ` Links: "name" <url>.`. `javascript:` and same-page `#fragment` links get no URL, every link line, row cell and group item carries its own URL (nothing is deduped across them, even when two links share a URL), and URLs never contain a raw space, `<` or `>`. Off by default: output is unchanged and costs nothing (about +23% chars on a GitHub issue list in data mode).
108
108
 
109
+ **Map + zoom** (big or cluttered pages). `snapshot({ map: true })` returns an outline of the page's big boxes instead of content, one line each, nested, with a ref and the size of its text in data mode: `- main [ref=5] (21.3k chars)`, ` - list "Issues" [ref=40] (14.9k chars)`, ` - list after "Other flights" [ref=88] (3.1k chars)` (an unnamed box shows the heading before it). Boxes: banner, navigation, main, complementary, contentinfo, region, form, search, list, table, grid, treegrid, feed, tablist, tabpanel, dialog; only those of at least 800 chars are listed. Card groups are listed too: a parent whose substantial children are mostly alike (a grid of product, listing or video cards that no role marks; found by structure only: 4 or more cards of one role and similar shape, each with real text) shows as `- group ×18 (6.3k chars) [ref=N]` where it sits (size = the cards' text; ref = their parent's, so `within: N` zooms the cards and anything beside them in that parent; when the cards sit straight in `main` that is main's ref). Not looked for inside banner, navigation or contentinfo, and not for a list or table (already boxes). A bare `within: 'group'` selects the card groups (two or more: the map with `<- match`; with none on the page, it matches real `group` nodes as before). No name or position selector: use the ref. `snapshot({ within })` then returns just one box, as if it were the whole page (prune's landmark step and data mode's main-only scope are off inside it; every mode and `links` work): `within: 40` (a ref from the map), `'table'` (role), `'list "Issues"'` (role + exact accessible name), `'list after "Other flights"'` (an unnamed box by the heading right before it, which survives a reload), or an array of those for several boxes, output in page order. Two or more matches return the map with the matches marked `<- match` (never picks one); none match, or a ref whose element is gone, throws. A ref in `within` is a ref from your previous snapshot (refs renumber on every snapshot and reset on navigation), found again by its element, not its number; refs printed in a zoom are clickable. Refs only hold until the next snapshot or navigation (including one a click or key press caused: Chromium checks the page's document identity at every ref use, so a hash link or `pushState` keeps refs, a new page kills them), and a ref from an older map can point at a different box; for anything you store or hardcode, use names (`list "Issues"`, `list after "Top flights"`), which are stable across reloads. `within` wins over `map`. The library default is unchanged: no `map`/`within` gives today's output byte for byte (so a consumer that freezes the whole page text is unaffected). MCP `snapshot` takes `map`, `within` (string, number or array) and `full`, and a snapshot over `maxChars` returns the map inline while still saving the full snapshot to `.barebrowse/` (`full: true` keeps the path-only answer); MCP and the CLI daemon wait once for the network to go quiet (idle 500ms, up to 3s) before the first snapshot after a `goto`, so late-loading content is in it (`settle: false` / `--no-settle` skip it; the library is unchanged). CLI `snapshot --map`, `--within=<box>` (repeat the flag for several boxes; quote a name: `--within='list "Issues"'`), `--full`, `--max-chars=N` (default 30000: over it, the map is printed after the file path). `browse()` and the bareagent adapter are unchanged. A table keeps its cell text in act/read mode on both engines.
110
+
109
111
  `npx barebrowse doctor` reports the runtime environment first — default engine/mode, installed Chromium/Firefox binaries, cookie sources, and any live session's engine + capabilities — then scans every known MCP config location and flags scope conflicts. `npx barebrowse install --force` overwrites an existing entry pointing at a different endpoint.
110
112
 
111
113
  ### 3. Library -- for agentic automation
@@ -71,7 +71,7 @@ const snapshot = await browse('https://example.com', {
71
71
  | `goBack()` | -- | void | Navigate back in browser history |
72
72
  | `goForward()` | -- | void | Navigate forward in browser history |
73
73
  | `reload(opts?)` | { ignoreCache?: boolean, timeout?: number } | void | Reload the current page. Clears refMap (refs from pre-reload reject). |
74
- | `snapshot(pruneOpts?)` | false or { mode: 'act'\|'read'\|'data', links?: boolean } | string | ARIA tree with `[ref=N]` markers. Pass `false` for raw. `links: true` adds `<absolute-url>` after each link name (see below). |
74
+ | `snapshot(pruneOpts?)` | false or { mode: 'act'\|'read'\|'data', links?: boolean, map?: boolean, within?: string\|number\|Array } | string | ARIA tree with `[ref=N]` markers. Pass `false` for raw. `links: true` adds `<absolute-url>` after each link name (see below). `map: true` returns an outline of the page's big boxes (ref + size); `within` zooms to one box or several (see Map + zoom below). |
75
75
  | `readable()` | -- | object | Clean article text (Reader-View engine). `{ ok, title, byline, text, length, confidence: 'high'\|'low', readerable, hint? }` or `{ ok: false, hint }`. For *reading*, not interacting — see note below. |
76
76
  | `click(ref)` | ref: string | void | Scroll into view + mouse press+release at center |
77
77
  | `type(ref, text, opts?)` | ref: string, text: string, opts: { clear?, keyEvents? } | void | Focus + insert text. `clear: true` replaces existing. |
@@ -133,6 +133,28 @@ Key rules:
133
133
  - `click(ref)` / `type(ref, text)` / `hover(ref)` / `select(ref, value)` use these ref strings
134
134
  - Pruning removes noise (~47-95% token reduction) while keeping all interactive elements
135
135
 
136
+ ## Map + zoom — big pages
137
+
138
+ Order for a big or cluttered page: map, then zoom, then mode. The library default is unchanged (no `map`/`within` = today's output, byte for byte).
139
+
140
+ ```js
141
+ await page.snapshot({ map: true }); // outline of the big boxes, no content
142
+ await page.snapshot({ within: 40 }); // one box by ref from the map
143
+ await page.snapshot({ within: 'table' }); // by role
144
+ await page.snapshot({ within: 'list "Issues"' }); // role + exact accessible name
145
+ await page.snapshot({ within: 'list after "Other flights"' }); // unnamed box: the heading right before it (survives a reload)
146
+ await page.snapshot({ within: [430, 'table'], mode: 'data', links: true }); // several boxes, page order; any mode/links
147
+ ```
148
+
149
+ - Map line: `- role "name" [ref=N] (14.9k chars)`, nested; unnamed boxes show `after "<heading>"`. Size = the box's text in data mode. Boxes: banner, navigation, main, complementary, contentinfo, region, form, search, list, table, grid, treegrid, feed, tablist, tabpanel, dialog; listed from 800 chars.
150
+ - Inside a zoom the box is the whole page: no landmark pruning, no main-only scope in data mode; everything else prunes as usual.
151
+ - Card groups: a grid of alike cards no role marks (products, listings, videos; structural detection, 4+ alike siblings) is listed as `- group ×18 (6.3k chars) [ref=N]`; zoom it by that ref (it is the cards' parent, so cards straight in `main` give main's ref). `within: 'group'` selects the card groups (several: map with `<- match`). Not inside banner/navigation/contentinfo, not for a list or table. No name selector.
152
+ - Match is exact on role and accessible name (or the box's preceding sibling heading text). Two or more matches: the map comes back with `<- match` on them, nothing is picked. Zero: throws, naming what was asked.
153
+ - A ref in `within` is a ref from your previous snapshot; it is found again by its element (refs renumber every snapshot). Gone element or a ref from before a navigation: throws. Refs printed in a zoom work with `click()` etc. Refs only hold until the next snapshot or navigation, and a ref from an older map can point at a different box; for anything you store or hardcode, use names (`list "Issues"`, `list after "Top flights"`), which are stable across reloads. `within` wins over `map`.
154
+ - Firefox act/read keep a table's cells (text under each cell), as Chromium does; `mode: 'data'` still gives one line per row.
155
+ - `browse()` and `createBrowseTools()` (bareagent) have no map/zoom.
156
+ - MCP: `snapshot` takes `map`, `within`, `full`; over `maxChars` it returns the map inline and saves the full snapshot to a file (`full: true` = path only). CLI: `snapshot --map`, `--within=<box>` (repeatable; quote names), `--full`, `--max-chars=N`. MCP and the CLI daemon wait once (network idle 500ms, up to 3s) before the first snapshot after a `goto` so late content is in it; `settle: false` / `--no-settle` skip it; the library is unchanged.
157
+
136
158
  ## readable() vs snapshot() — which to use
137
159
 
138
160
  Two different jobs:
@@ -254,6 +276,8 @@ For coding agents (Claude Code, Copilot, Cursor) and quick interactive testing.
254
276
  ```bash
255
277
  barebrowse open https://example.com # Start daemon + navigate
256
278
  barebrowse snapshot # → .barebrowse/page-<timestamp>.yml
279
+ barebrowse snapshot --map # outline of the page's big boxes (ref + size)
280
+ barebrowse snapshot --within='list "Issues"' # one box only (ref, role, role "name", role after "heading"; repeat for several)
257
281
  barebrowse readable # → .barebrowse/article-<timestamp>.txt (clean article text)
258
282
  barebrowse click 8 # Click element ref=8
259
283
  barebrowse type 12 hello world # Type into element ref=12
@@ -303,7 +327,7 @@ barebrowse ships an MCP server for direct use with Claude Desktop, Cursor, or an
303
327
 
304
328
  Action tools return `'ok'` -- the agent calls `snapshot` explicitly to observe. This avoids double-token output since MCP tool calls are cheap to chain.
305
329
 
306
- `browse`, `snapshot`, and `readable` accept a `maxChars` param (default 30000). If the output exceeds the limit it's saved to `.barebrowse/` and a short message with the file path is returned instead (`page-<timestamp>.yml` for snapshots, `article-<timestamp>.txt` for `readable`). `screenshot` always saves to `.barebrowse/screenshot-<timestamp>.{png,jpeg,webp}` and returns the file path (raw base64 in a JSON-RPC response would blow `maxChars`). `tabs` returns the JSON array, or with `switchTo: N` it switches and returns `'ok'`. All files MCP writes are owner-only (`0600` in a `0700` dir) — they can hold authenticated page content, so they're not world-readable on a shared host.
330
+ `browse`, `snapshot`, and `readable` accept a `maxChars` param (default 30000). If the output exceeds the limit it's saved to `.barebrowse/` and a short message with the file path is returned instead (`page-<timestamp>.yml` for snapshots, `article-<timestamp>.txt` for `readable`). The exception is `snapshot` with neither `within`, `map` nor `full`: it also returns the page map inline, to zoom from with `within` (see Map + zoom); `full: true` gives the path alone. `browse` is unchanged (one-shot: nothing to zoom afterwards). `screenshot` always saves to `.barebrowse/screenshot-<timestamp>.{png,jpeg,webp}` and returns the file path (raw base64 in a JSON-RPC response would blow `maxChars`). `tabs` returns the JSON array, or with `switchTo: N` it switches and returns `'ok'`. All files MCP writes are owner-only (`0600` in a `0700` dir) — they can hold authenticated page content, so they're not world-readable on a shared host.
307
331
 
308
332
  `browse` and `snapshot` also accept `pruneMode: 'act'|'read'|'data'`. `act` (the default) keeps interactive elements and short labels — best for clicking/filling. `read` keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. `data` (opt-in) is one line per record — tables with headers (`row "Label": Header: value · …`), repeated items grouped (`Label: a, b, c`) — for data pages (timetables, listings, results), not articles; every data snapshot starts with ONE legend line (`# data mode: one line per record. Tables: row "<label>": <column>: <value> · … Groups: <label>: a, b, c. A title line followed by detail lines is one block, up to the next title line. Main content only; use mode "act" for menus and page controls.`) so you can read it without docs; it keeps `[ref=N]` on interactive nodes (clickable like act) and is *larger* than act on prose pages — use `read`/`readable()` there. Same surface on the bareagent adapter. If act mode collapses a content-heavy page (raw > 5 KB → pruned < 500 chars AND < 5% of raw), the result includes a `hint: act mode dropped most of the page — retry with pruneMode='read' …` line between the stats and the tree so the caller knows to re-snapshot in read mode instead of bailing to a separate HTTP fetch.
309
333
 
package/cli.js CHANGED
@@ -38,7 +38,14 @@ if (args.includes('--daemon-internal')) {
38
38
  } else if (cmd === 'goto' && args[1]) {
39
39
  await cmdProxy('goto', { url: args[1], timeout: parseFlag('--timeout') });
40
40
  } else if (cmd === 'snapshot') {
41
- await cmdProxy('snapshot', { mode: parseFlag('--mode'), links: hasFlag('--links') || undefined });
41
+ const within = parseFlagAll('--within');
42
+ await cmdProxy('snapshot', {
43
+ mode: parseFlag('--mode'), links: hasFlag('--links') || undefined,
44
+ within: within && (within.length === 1 ? within[0] : within),
45
+ map: hasFlag('--map') || undefined, full: hasFlag('--full') || undefined,
46
+ maxChars: parseFlag('--max-chars') && Number(parseFlag('--max-chars')),
47
+ settle: hasFlag('--no-settle') ? false : undefined,
48
+ });
42
49
  } else if (cmd === 'readable') {
43
50
  await cmdProxy('readable');
44
51
  } else if (cmd === 'screenshot') {
@@ -177,6 +184,10 @@ async function cmdProxy(command, cmdArgs) {
177
184
  // Print result
178
185
  if (result.file && result.count !== undefined) {
179
186
  process.stdout.write(`${result.file} (${result.count} entries)\n`);
187
+ } else if (result.file && result.map) {
188
+ process.stdout.write(`${result.file}\nOver --max-chars, so here is the map of the page; zoom with: snapshot --within=<ref or name> (repeat the flag for several boxes), or --full for the path alone.\n${result.map}\n`);
189
+ } else if (result.file && result.note) {
190
+ process.stdout.write(`${result.file}\n${result.note}\n`);
180
191
  } else if (result.file) {
181
192
  process.stdout.write(`${result.file}\n`);
182
193
  } else if (result.value !== undefined) {
@@ -604,6 +615,13 @@ Navigation:
604
615
  barebrowse forward Go forward in history
605
616
  barebrowse reload [--no-cache] Reload current page
606
617
  barebrowse snapshot [--mode=M] [--links] ARIA snapshot -> .barebrowse/page-*.yml (--links: link URLs)
618
+ --map Outline of the page's big boxes (ref + size) instead of content
619
+ --within=<box> Zoom to one box, as if it were the page: a ref from the map, a role (table),
620
+ role "name" (--within='list "Issues"'), or role after "heading". Repeat the flag
621
+ for several boxes (page order). Works with --mode and --links.
622
+ --full Skip the auto map. A snapshot over --max-chars (default 30000) is saved
623
+ in full and the map is printed after the path, to zoom from.
624
+ --no-settle Don't wait for the network to go quiet (up to 3s) before the first snapshot after a goto
607
625
  barebrowse readable Clean article text -> .barebrowse/article-*.txt
608
626
  barebrowse screenshot [--format] Screenshot -> .barebrowse/screenshot-*.png
609
627
  barebrowse pdf [--landscape] PDF export -> .barebrowse/page-*.pdf
@@ -56,6 +56,10 @@ All output files go to `.barebrowse/` in the current directory. Read them with t
56
56
  | `barebrowse snapshot --mode=read` | Read mode: keeps all text (for content extraction) |
57
57
  | `barebrowse snapshot --mode=data` | Data mode: one line per record (table rows with headers, repeated items grouped). For data pages, not articles. Main content only; use `act` for menus and page controls. |
58
58
  | `barebrowse snapshot --links` | Add each link's absolute URL after its name: `link "title" <https://…> [ref=N]` (combine with `--mode`; also on `browse <url> --links`). `javascript:` and same-page `#` links get none. |
59
+ | `barebrowse snapshot --map` | Outline of the page's big boxes (lists, tables, main, …), each with a ref and a size. Cheap (1-2% of the page): use it on big pages, then zoom. A grid of alike cards (products, listings, videos) shows as `group ×18 (6.3k chars) [ref=N]`: zoom it with `--within=<that ref>`. |
60
+ | `barebrowse snapshot --within=<box>` | Only that box, as if it were the page. `<box>` = a ref from the map, a role (`table`), `role "name"` (`--within='list "Issues"'`), or an unnamed box by its heading (`--within='list after "Other flights"'`). Repeat the flag for several boxes (page order). Combine with `--mode` / `--links`. Several matches: the map comes back with the matches marked; none: error. |
61
+ | `barebrowse snapshot --no-settle` | The first snapshot after a `goto` waits once for the network to go quiet (up to 3s) so late-loading content is in it; this skips the wait. |
62
+ | `barebrowse snapshot --full` | A snapshot over `--max-chars` (default 30000) is saved in full and the map is printed after the path; `--full` prints the path alone. |
59
63
  | `barebrowse readable` | Clean article text → `.barebrowse/article-<timestamp>.txt`. For reading/summarising articles (news, blogs, docs); falls back to snapshot on non-articles. |
60
64
  | `barebrowse screenshot` | Screenshot → `.barebrowse/screenshot-<timestamp>.png` |
61
65
  | `barebrowse pdf [--landscape]` | PDF export → `.barebrowse/page-<timestamp>.pdf` |
@@ -129,6 +133,7 @@ The snapshot is a YAML-like ARIA tree. Each line is one node:
129
133
 
130
134
  - **Always snapshot before interacting** — refs are ephemeral and change every time
131
135
  - **Use `fill` instead of `type`** when replacing existing text in input fields
136
+ - **On a big page, `snapshot --map` then `snapshot --within=<ref or name>`** instead of reading the whole file; refs in `--within` come from the last snapshot
132
137
  - **Use `--mode=read`** for snapshot when you need to extract article content or data
133
138
  - **Use `back`/`forward`** to navigate browser history instead of re-entering URLs
134
139
  - **Use `upload`** for file inputs — pass absolute paths to the files
package/mcp-server.js CHANGED
@@ -15,6 +15,7 @@ import { formatReadable } from './src/readable.js';
15
15
  import { cleanOld } from './src/output-clean.js';
16
16
  import { scopedCookiesForUrl } from './src/auth.js';
17
17
  import { isHttpError } from './src/navigation.js';
18
+ import { overNote } from './src/zoom.js';
18
19
  import { parseAllowHosts } from './src/site-lock.js';
19
20
  import { mkdirSync, writeFileSync, readFileSync } from 'node:fs';
20
21
  import { join, dirname } from 'node:path';
@@ -122,6 +123,16 @@ export function saveSnapshot(text, { prefix = 'page', ext = 'yml' } = {}) {
122
123
  }
123
124
 
124
125
  let _page = null;
126
+ // A goto's load event is not the page's last request: the first snapshot after it waits
127
+ // once for the network to go quiet (idle 500ms, capped 3s), like consentRecheck does once
128
+ // per navigation. A goto followed by a click pays nothing. Library timing is unchanged.
129
+ let _settlePending = false;
130
+ async function settleOnce(page, settle) {
131
+ if (!_settlePending) return;
132
+ _settlePending = false;
133
+ if (settle === false) return;
134
+ try { await page.waitForNetworkIdle({ idle: 500, timeout: 3000 }); } catch { /* best-effort: a busy page is snapshotted as it is */ }
135
+ }
125
136
  let _pageConnecting = null;
126
137
  // Active engine: starts from BAREBROWSE_ENGINE, changed at runtime by switch_engine.
127
138
  let _engine = process.env.BAREBROWSE_ENGINE === 'firefox' ? 'firefox' : 'chromium';
@@ -233,6 +244,11 @@ export const TOOL_TIER = {
233
244
  upload: 'rwx', eval: 'rwx',
234
245
  };
235
246
 
247
+ // MCP annotation hints, classified next to the tier map: read tools whose repeat call is a no-op,
248
+ // and state-changing tools that are still non-destructive.
249
+ const IDEMPOTENT_READS = new Set(['snapshot', 'readable', 'screenshot', 'pdf', 'tabs', 'capabilities', 'wait_for']);
250
+ const NON_DESTRUCTIVE = new Set(['hover', 'switch_engine']);
251
+
236
252
  /** @param {string} name */
237
253
  export function toolTier(name) { return TOOL_TIER[name] ?? 'rwx'; }
238
254
 
@@ -259,10 +275,10 @@ function readAccess() {
259
275
  function annotationsFor(name) {
260
276
  if (toolTier(name) === 'r') {
261
277
  const a = { readOnlyHint: true, openWorldHint: name !== 'capabilities' };
262
- if (['snapshot', 'readable', 'screenshot', 'pdf', 'tabs', 'capabilities', 'wait_for'].includes(name)) a.idempotentHint = true;
278
+ if (IDEMPOTENT_READS.has(name)) a.idempotentHint = true;
263
279
  return a;
264
280
  }
265
- return { readOnlyHint: false, destructiveHint: !['hover', 'switch_engine'].includes(name), openWorldHint: true };
281
+ return { readOnlyHint: false, destructiveHint: !NON_DESTRUCTIVE.has(name), openWorldHint: true };
266
282
  }
267
283
 
268
284
  async function getPage() {
@@ -337,13 +353,20 @@ export const TOOLS = [
337
353
  },
338
354
  {
339
355
  name: 'snapshot',
340
- description: 'Get the current ARIA snapshot of the session page. Returns a YAML-like tree with [ref=N] markers on interactive elements.',
356
+ description: 'Get the current ARIA snapshot of the session page. Returns a YAML-like tree with [ref=N] markers on interactive elements. Big pages: start with map:true (an outline of the page\'s boxes: header, main, lists, tables… each with a ref and a size), then call again with within=<ref or name> to read just the box you need, in any pruneMode. A page over maxChars returns that map for you instead of the content (the full snapshot is also saved to a file); pass full:true to get the file path alone.',
341
357
  inputSchema: {
342
358
  type: 'object',
343
359
  properties: {
344
360
  pruneMode: { type: 'string', enum: ['act', 'read', 'data'], description: 'Pruning mode. "act" (default) keeps interactive elements and short labels — best for clicking/filling. "read" keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. If a previous snapshot looked empty on a content-heavy page, retry with "read". "data": one line per record — tables with headers, repeated items grouped; for data pages, not articles. Output starts with a legend line; read it as: table rows `row "<label>": <column>: <value> · …`, groups `<label>: a, b, c`, and a title line followed by detail lines is one block up to the next title line. Main content only: use "act" for menus and page controls.' },
345
361
  links: { type: 'boolean', description: 'Show each link\'s absolute URL after its name: link "name" <url> [ref=N]. Default off.' },
346
- maxChars: { type: 'number', description: 'Max chars to return inline. Larger snapshots are saved to .barebrowse/ and a file path is returned instead. Default: 30000.' },
362
+ map: { type: 'boolean', description: 'Return an outline of the page\'s big boxes instead of content: `- list "Issues" [ref=40] (14.9k chars)`, nested, one line each. Unnamed boxes show the heading before them: `list after "Other flights"`. Size is the box\'s text in "data" mode. A grid of alike cards (products, listings, videos) shows as `group ×18 (6.3k chars) [ref=N]`; within=<that ref> zooms it. Costs 1-2% of the page; use it to pick a box for within.' },
363
+ within: {
364
+ anyOf: [{ type: 'string' }, { type: 'number' }, { type: 'array', items: { anyOf: [{ type: 'string' }, { type: 'number' }] } }],
365
+ description: 'Zoom: return only this box, as if it were the whole page (works with every pruneMode and links). A ref from the map (e.g. 40; refs change on every snapshot, so use one from the latest map), a role (table), a role and its exact name (list "Issues"), or an unnamed box by the heading right before it (list after "Other flights"). An array returns several boxes, in page order. If two or more boxes match you get the map back with the matches marked; if none match, an error. A ref whose element is gone is an error: take a fresh map. Refs only hold until the next snapshot or navigation, and a ref from an older map can point at a different box; for anything you store or hardcode, use names (list "Issues", list after "Top flights"), which are stable across reloads.',
366
+ },
367
+ settle: { type: 'boolean', description: 'The first snapshot after a goto waits once for the network to go quiet (up to 3s, about 0.5s on a finished page) so late-loading content is in it. Pass false to snapshot immediately. Default: true.' },
368
+ full: { type: 'boolean', description: 'Skip the automatic map: a snapshot over maxChars is saved to a file and only the path is returned (the old behavior). Ignored when within or map is set.' },
369
+ maxChars: { type: 'number', description: 'Max chars to return inline. A larger snapshot is saved to .barebrowse/ in full and the map of the page is returned instead (or just the file path with full:true). Default: 30000.' },
347
370
  },
348
371
  },
349
372
  },
@@ -624,15 +647,25 @@ async function handleToolCall(name, args) {
624
647
  _lastCookies = cookieReport(args.url, { cookies, error });
625
648
  }
626
649
  const { status } = await page.goto(args.url);
650
+ _settlePending = true;
627
651
  return isHttpError(status) ? `ok (HTTP ${status})` : 'ok';
628
652
  }, TIMEOUTS.goto);
629
653
  case 'snapshot': return withRetry(async () => {
630
654
  const page = await getPage();
631
- const text = await page.snapshot(args.pruneMode || args.links ? { mode: args.pruneMode, links: args.links } : undefined);
655
+ await settleOnce(page, args.settle);
656
+ const zoomed = args.map || args.within != null;
657
+ const text = await page.snapshot(args.pruneMode || args.links || zoomed
658
+ ? { mode: args.pruneMode, links: args.links, map: args.map, within: args.within } : undefined);
632
659
  const limit = args.maxChars ?? MAX_CHARS_DEFAULT;
633
660
  if (text.length > limit) {
634
661
  const file = saveSnapshot(text);
635
- return `Snapshot (${text.length} chars) saved to ${file}`;
662
+ const saved = `Snapshot (${text.length} chars) saved to ${file}`;
663
+ if (zoomed && !args.full) return `${saved}. ${overNote(args.within != null ? 'zoom' : 'map', text.length, limit)}`;
664
+ if (zoomed || args.full) return saved;
665
+ // Over the cap and the agent chose nothing: hand back the map (a few % of the page) to zoom from.
666
+ const map = await page.snapshot({ links: args.links, map: true });
667
+ if (map.length > limit) return saved;
668
+ return `${saved}. Over maxChars (${limit}), so here is the map of the page; call snapshot with within=<ref or name> to read one box (an array takes several), or full:true for the path alone.\n${map}`;
636
669
  }
637
670
  return text;
638
671
  }, TIMEOUTS.snapshot);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "barebrowse",
3
- "version": "0.27.0",
3
+ "version": "0.28.0",
4
4
  "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
5
5
  "repository": {
6
6
  "type": "git",
package/src/auth.js CHANGED
@@ -56,6 +56,16 @@ export function findFirefoxCookieDb() {
56
56
 
57
57
  // --- Chromium cookie decryption (Linux) ---
58
58
 
59
+ /**
60
+ * kwallet-query output -> Chromium keyring password. Chromium feeds the stored
61
+ * string to PBKDF2 as-is (it is not base64 to decode).
62
+ * @param {string} stdout
63
+ * @returns {string}
64
+ */
65
+ export function kwalletPassword(stdout) {
66
+ return stdout.trim();
67
+ }
68
+
59
69
  /**
60
70
  * Get Chromium encryption password from OS keyring.
61
71
  * Tries KWallet (KDE) first, then GNOME keyring, then fallback.
@@ -63,11 +73,11 @@ export function findFirefoxCookieDb() {
63
73
  function getChromiumPassword() {
64
74
  // KDE / KWallet
65
75
  try {
66
- const b64 = execSync(
76
+ const pw = kwalletPassword(execSync(
67
77
  'kwallet-query -r "Chromium Safe Storage" -f "Chromium Keys" kdewallet',
68
78
  { encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }
69
- ).trim();
70
- if (b64) return Buffer.from(b64, 'base64').toString('binary');
79
+ ));
80
+ if (pw) return pw;
71
81
  } catch { /* not KDE or no entry */ }
72
82
 
73
83
  // GNOME keyring / libsecret
@@ -87,7 +97,7 @@ function getChromiumPassword() {
87
97
  * Derive AES key from Chromium keyring password.
88
98
  * Chrome Linux: PBKDF2-SHA1, salt='saltysalt', 1 iteration, 16-byte key.
89
99
  */
90
- function deriveKey(password) {
100
+ export function deriveKey(password) {
91
101
  return pbkdf2Sync(password, 'saltysalt', 1, 16, 'sha1');
92
102
  }
93
103
 
@@ -97,7 +107,7 @@ function deriveKey(password) {
97
107
  * @param {Buffer} aesKey - Derived AES key
98
108
  * @returns {string} Decrypted cookie value
99
109
  */
100
- function decryptCookie(encrypted, aesKey) {
110
+ export function decryptCookie(encrypted, aesKey) {
101
111
  const buf = Buffer.from(encrypted);
102
112
  if (buf.length === 0) return '';
103
113
 
@@ -121,7 +131,7 @@ function decryptCookie(encrypted, aesKey) {
121
131
  * @param {string} [domain] - Filter by domain (e.g. '.github.com')
122
132
  * @returns {Array<object>} Cookies in CDP Network.setCookie format
123
133
  */
124
- function extractChromiumCookies(dbPath, domain) {
134
+ export function extractChromiumCookies(dbPath, domain) {
125
135
  const password = getChromiumPassword();
126
136
  const aesKey = deriveKey(password);
127
137
 
@@ -132,10 +142,18 @@ function extractChromiumCookies(dbPath, domain) {
132
142
  CAST(expires_utc AS TEXT) AS expires_utc, is_secure, is_httponly, samesite
133
143
  FROM cookies`;
134
144
  const params = [];
145
+ // Skip partitioned (CHIPS) rows: a non-empty top_frame_site_key marks a cookie
146
+ // scoped to a top-level site, and it can share name+domain+path with the real
147
+ // one. Older Chromium schemas lack the column, so only filter when it exists.
148
+ const where = [];
149
+ if (db.prepare('PRAGMA table_info(cookies)').all().some((c) => c.name === 'top_frame_site_key')) {
150
+ where.push(`top_frame_site_key = ''`);
151
+ }
135
152
  if (domain) {
136
- sql += ` WHERE host_key LIKE ?`;
153
+ where.push(`host_key LIKE ?`);
137
154
  params.push(`%${domain}%`);
138
155
  }
156
+ if (where.length) sql += ` WHERE ${where.join(' AND ')}`;
139
157
 
140
158
  const stmt = db.prepare(sql);
141
159
  const rows = params.length ? stmt.all(...params) : stmt.all();
@@ -49,6 +49,10 @@ const AX_FN = function reconstructAX(base, REF_ATTR) {
49
49
  'tab', 'menuitem', 'option', 'treeitem', 'switch', 'checkbox', 'radio',
50
50
  ]);
51
51
 
52
+ // Table cells name from content like the roles above, but CDP also keeps
53
+ // their text as StaticText children; without them act/read prune the cell away.
54
+ const KEEP_TEXT_CHILDREN = new Set(['cell', 'columnheader', 'rowheader']);
55
+
52
56
  // HTML tag → implicit ARIA role (the common, high-value subset).
53
57
  const TAG_ROLE = {
54
58
  A: (el) => (el.hasAttribute('href') ? 'link' : 'generic'),
@@ -159,6 +163,23 @@ const AX_FN = function reconstructAX(base, REF_ATTR) {
159
163
  return '';
160
164
  }
161
165
 
166
+ // Descendant text for name-from-content, with an <img>'s alt standing in for
167
+ // the image (accname step 2F), so an image-only link is named like in CDP.
168
+ function contentName(el) {
169
+ let out = '';
170
+ const add = (n) => {
171
+ for (const c of n.childNodes) {
172
+ if (c.nodeType === 3) out += c.textContent;
173
+ else if (c.nodeType === 1) {
174
+ if (c.tagName === 'IMG') out += ' ' + (c.getAttribute('alt') || '') + ' ';
175
+ else add(c);
176
+ }
177
+ }
178
+ };
179
+ add(el);
180
+ return out.replace(/\s+/g, ' ').trim();
181
+ }
182
+
162
183
  function props(el, role) {
163
184
  const p = {};
164
185
  if (role === 'checkbox' || role === 'radio' || role === 'switch') p.checked = !!el.checked;
@@ -190,9 +211,7 @@ const AX_FN = function reconstructAX(base, REF_ATTR) {
190
211
  // text for roles that name from content (button/link/heading/…), matching
191
212
  // CDP — otherwise those nodes come back nameless.
192
213
  let name = accName(el);
193
- if (!name && NAME_FROM_CONTENT.has(role)) {
194
- name = el.textContent.replace(/\s+/g, ' ').trim();
195
- }
214
+ if (!name && NAME_FROM_CONTENT.has(role)) name = contentName(el);
196
215
  const node = makeNode(role, name, props(el, role));
197
216
  el.setAttribute(REF_ATTR, node.nodeId);
198
217
 
@@ -201,7 +220,7 @@ const AX_FN = function reconstructAX(base, REF_ATTR) {
201
220
  // would otherwise flood every snapshot). A multi/size listbox keeps them.
202
221
  if (el.tagName === 'SELECT' && !el.multiple) return node;
203
222
 
204
- if (!NAME_FROM_CONTENT.has(role)) {
223
+ if (!NAME_FROM_CONTENT.has(role) || KEEP_TEXT_CHILDREN.has(role)) {
205
224
  for (const child of kidsOf(el)) {
206
225
  if (child.nodeType === 3) {
207
226
  const t = child.textContent.replace(/\s+/g, ' ').trim();
@@ -213,7 +232,7 @@ const AX_FN = function reconstructAX(base, REF_ATTR) {
213
232
  }
214
233
  } else {
215
234
  // name-from-content: still recurse into element children for nested
216
- // interactives (e.g. a link inside a heading), but drop bare text.
235
+ // interactives (e.g. a link inside a heading), but drop bare text (cells excepted, above).
217
236
  for (const child of kidsOf(el)) {
218
237
  if (child.nodeType !== 1) continue;
219
238
  const c = walk(child);
package/src/bidi.js CHANGED
@@ -98,11 +98,12 @@ export async function createBiDi(wsUrl) {
98
98
  * @param {string} context - browsing-context id
99
99
  * @param {string} expression - JS source to evaluate
100
100
  * @param {boolean} [awaitPromise=true]
101
+ * @param {string} [sandbox] - run in this isolated realm (page script cannot see or alter it)
101
102
  * @returns {Promise<*>} the deserialized primitive value (result.value)
102
103
  */
103
- async evaluate(context, expression, awaitPromise = true) {
104
+ async evaluate(context, expression, awaitPromise = true, sandbox) {
104
105
  const res = await client.send('script.evaluate', {
105
- expression, target: { context }, awaitPromise,
106
+ expression, target: sandbox ? { context, sandbox } : { context }, awaitPromise,
106
107
  });
107
108
  if (res.type === 'exception') {
108
109
  const d = res.exceptionDetails;
package/src/daemon.js CHANGED
@@ -11,6 +11,7 @@ import { writeFileSync, mkdirSync, existsSync, readFileSync, unlinkSync } from '
11
11
  import { randomBytes, timingSafeEqual } from 'node:crypto';
12
12
  import { join, resolve } from 'node:path';
13
13
  import { connect } from './index.js';
14
+ import { overNote } from './zoom.js';
14
15
  import { formatReadable } from './readable.js';
15
16
  import { cleanOld } from './output-clean.js';
16
17
  import { isHttpError } from './navigation.js';
@@ -302,12 +303,23 @@ export async function runDaemon(opts, outputDir, initialUrl) {
302
303
  }
303
304
  await attachCapture(page);
304
305
 
306
+ // The first snapshot after a goto waits once for the network to go quiet (idle 500ms,
307
+ // capped 3s): a page's load event is not its last request. Library timing is unchanged.
308
+ let settlePending = false;
309
+ async function settleOnce(settle) {
310
+ if (!settlePending) return;
311
+ settlePending = false;
312
+ if (settle === false) return;
313
+ try { await page.waitForNetworkIdle({ idle: 500, timeout: 3000 }); } catch { /* best-effort */ }
314
+ }
315
+
305
316
  // Navigate to initial URL if provided
306
317
  if (initialUrl) {
307
318
  if (opts.cookies !== false) {
308
319
  try { await page.injectCookies(initialUrl, { browser: opts.browser }); } catch { /* no cookies */ }
309
320
  }
310
321
  await page.goto(initialUrl, opts.timeout ? Number(opts.timeout) : 30000);
322
+ settlePending = true;
311
323
  }
312
324
 
313
325
  // Default prune mode
@@ -317,15 +329,26 @@ export async function runDaemon(opts, outputDir, initialUrl) {
317
329
  const handlers = {
318
330
  async goto({ url, timeout }) {
319
331
  const { status } = await page.goto(url, timeout || 30000);
332
+ settlePending = true;
320
333
  return isHttpError(status) ? { ok: true, status } : { ok: true };
321
334
  },
322
335
 
323
- async snapshot({ mode, links }) {
336
+ async snapshot({ mode, links, within, map, full, maxChars, settle }) {
337
+ await settleOnce(settle);
324
338
  const pruneMode = mode || defaultPruneMode;
325
- const text = await page.snapshot({ mode: pruneMode, links });
339
+ const text = await page.snapshot({ mode: pruneMode, links, within, map });
326
340
  const ts = new Date().toISOString().replace(/[:.]/g, '-');
327
341
  const file = join(absDir, `page-${ts}.yml`);
328
342
  writeFilePrivate(file, text);
343
+ // Over the cap with no zoom asked for: the file stays complete, and the
344
+ // map (a few % of the page) comes back too, to zoom from. Same as MCP.
345
+ if (within == null && !map && !full && text.length > (maxChars ?? 30000)) {
346
+ const outline = await page.snapshot({ links, map: true });
347
+ if (outline.length <= (maxChars ?? 30000)) return { ok: true, file, map: outline };
348
+ }
349
+ if ((within != null || map) && !full && text.length > (maxChars ?? 30000)) {
350
+ return { ok: true, file, note: overNote(within != null ? 'zoom' : 'map', text.length, maxChars ?? 30000, true) };
351
+ }
329
352
  return { ok: true, file };
330
353
  },
331
354
 
package/src/data.js CHANGED
@@ -88,7 +88,7 @@ export const stripLegend = (s) => (s.startsWith(LEGEND) ? s.slice(s.indexOf('\n'
88
88
  * Render a raw ARIA tree in data mode.
89
89
  *
90
90
  * @param {object} root - Root node from buildTree() (CDP vocabulary)
91
- * @param {{ links?: boolean }} [opts] - links: the tree's link nodes carry `urls` (annotateLinks); adds the legend note
91
+ * @param {{ links?: boolean, whole?: boolean }} [opts] - links: the tree's link nodes carry `urls` (annotateLinks); adds the legend note. whole: root is a zoom box (snapshot `within`), so no main-only scope
92
92
  * @returns {string} LEGEND line, then one line per record, YAML-ish like formatTree(); '' when there are no records
93
93
  */
94
94
  export function dataSnapshot(root, opts = {}) {
@@ -283,7 +283,7 @@ export function dataSnapshot(root, opts = {}) {
283
283
  if (tops.length === 1 && (tops[0].role === 'RootWebArea' || tops[0].role === 'WebArea')) tops = tops[0].children;
284
284
  const find = (ns, role) => { for (const n of ns) { if (n.role === role) return n; const f = find(n.children, role); if (f) return f; } return null; };
285
285
  const main = find(tops, 'main');
286
- if (main && main.children.length) tops = [main];
286
+ if (!opts.whole && main && main.children.length) tops = [main];
287
287
  const res = xform({ role: 'root', name: '', children: tops, props: {} }, []);
288
288
 
289
289
  const lines = [];