barebrowse 0.18.0 → 0.19.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,84 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.19.2] - 2026-07-11
4
+
5
+ ### Fixed
6
+
7
+ - **Firefox `saveState()` now writes `sameSite` capitalized** (`Lax`/`Strict`/`None`,
8
+ was BiDi's lowercase `lax`/`strict`/`none`). The only loader for a state file is
9
+ `connect({ storageState })`, which is CDP-only (`Network.setCookies`) and rejects
10
+ a lowercase enum — so the whole batch threw and every cookie was silently dropped,
11
+ making a Firefox-saved state useless for auth restore. Found by `/code-review` and
12
+ confirmed against real Firefox.
13
+ - **Firefox CLI/daemon `eval` accepts statement-form expressions again**
14
+ (`var x = 2; x * 3`). A Phase-4 wrapper (`await (${expr})`, added to flatten
15
+ objects) turned every statement list into a `SyntaxError` — a regression versus
16
+ the CDP `Runtime.evaluate` path. Now evaluates the raw expression (BiDi
17
+ `script.evaluate` has eval-semantics) and deserializes the `{type,value}` remote
18
+ value back to the plain value CDP's `returnByValue` yields, so deep objects stay
19
+ clean *and* statements work.
20
+ - **Firefox daemon console capture no longer dumps nested serialization** for
21
+ object/array console arguments — it falls back to the type name, matching the CDP
22
+ capture's `RemoteObject.description` ("Object").
23
+
24
+ ## [0.19.1] - 2026-07-11
25
+
26
+ ### Changed
27
+
28
+ - **MCP: Firefox sessions now default to `hybrid`** (was `headless`), matching the
29
+ Chromium default — Phase 4 gave Firefox the same headless→headed challenge
30
+ fallback, so there's no reason to hold it back. A failed headed relaunch (no
31
+ display) falls back to the headless result. Override with `BAREBROWSE_MODE`;
32
+ the Chromium path now honors `BAREBROWSE_MODE` too (was hardcoded `hybrid`).
33
+
34
+ ### Docs
35
+
36
+ - **README** refocused on primitives and value — Firefox stated as a capability
37
+ set rather than a version-by-version log; stale counts/versions and a redundant
38
+ section removed.
39
+ - **Single PRD** — the Firefox parity plan is merged into `docs/01-product/prd.md`
40
+ as a "Firefox Parity" section (capability matrix + goal/non-goals + forward
41
+ plan); the standalone `firefox-parity-plan.md` is removed.
42
+ - Fixed stale "hybrid is chromium-only" strings in the CLI help and MCP comments.
43
+
44
+ ## [0.19.0] - 2026-07-11
45
+
46
+ ### Added
47
+
48
+ - **Firefox/BiDi resilience + remaining methods (parity plan Phase 4).** The
49
+ Firefox engine reaches practical parity with the Chromium/CDP engine — four
50
+ CDP-only surfaces now work the same over WebDriver BiDi:
51
+ - **Hybrid mode** (`mode: 'hybrid'`). On a bot-challenge page (Cloudflare "Just
52
+ a moment", hCaptcha, etc.) the Firefox engine now relaunches **headed** and
53
+ retries, exactly like the CDP path — a real display often clears an
54
+ interstitial headless can't. The relaunch tears down the headless
55
+ Firefox+BiDi and hands a fresh connection back to the page, which rebinds to
56
+ it; a failed relaunch (no display) keeps the headless result. Cookies
57
+ injected before the challenge are **re-injected into the fresh headed
58
+ profile** so the retry stays authenticated. The challenge heuristic
59
+ (`isChallengePage`) is now shared by both engines (`challenge.js`).
60
+ - **`page.saveState(file)`** — persists cookies (`storage.getCookies`) +
61
+ localStorage (`script.evaluate`) to a `0600` JSON file, flattened to the
62
+ same shape the CDP path writes (was a throwing stub).
63
+ - **`page.waitForNavigation()`** — resolves on the next top-context
64
+ `browsingContext.load` event, with an SPA settle fallback (was a stub).
65
+ - **`page.downloads`** — real download tracking via
66
+ `browsingContext.downloadWillBegin`/`downloadEnd`. Downloads land in a
67
+ throwaway dir (Firefox `browser.download.*` prefs, parity with CDP's
68
+ `setDownloadBehavior`) and each record carries `{url, suggestedFilename,
69
+ savedPath, state}` with BiDi's `complete` normalized to CDP's `completed`
70
+ (was always empty).
71
+
72
+ BiDi mechanics were POC-measured against real Firefox before wiring (cookie/
73
+ localStorage shapes on real vs opaque origins; the `load` event sequence; that
74
+ Firefox emits `downloadWillBegin`/`downloadEnd` and honors the download-dir
75
+ prefs). The shared JS-dialog decision core was also extracted to `dialog.js`
76
+ (both engines — Phase-3 review finding #5). Guarded by
77
+ `test/unit/firefox-hybrid.test.js` (5, fake-BiDi orchestration) and Firefox
78
+ integration tests (saveState, waitForNavigation ×2, downloads).
79
+ Still Firefox-limited: `reload({ignoreCache})` (upstream BiDi gap, no local
80
+ fix).
81
+
3
82
  ## [0.18.0] - 2026-07-11
4
83
 
5
84
  ### Added
package/README.md CHANGED
@@ -96,15 +96,15 @@ Or manually add to your config (`claude_desktop_config.json`, `.cursor/mcp.json`
96
96
  }
97
97
  ```
98
98
 
99
- 19 tools: `browse`, `goto`, `snapshot`, `readable`, `click`, `type`, `press`, `scroll`, `hover`, `select`, `back`, `forward`, `reload`, `drag`, `upload`, `pdf`, `screenshot`, `wait_for`, `tabs`. Plus `assess` (privacy scan) if [wearehere](https://github.com/hamr0/wearehere) is installed. Plus opt-in `eval` (`BAREBROWSE_MCP_EVAL=1`) — runs JS in the authenticated session, off by default because it can read cookies/localStorage. Session runs in hybrid mode with automatic cookie injection. Per-tool timeouts (goto/reload/wait_for 60s, back/forward 30s, interactive ops 15s, pdf/screenshot/upload 45s) with auto-retry on transient failures (idempotent only — mutating tools fail loudly to avoid double-submits).
99
+ MCP tools: `browse`, `goto`, `snapshot`, `readable`, `click`, `type`, `press`, `scroll`, `hover`, `select`, `back`, `forward`, `reload`, `drag`, `upload`, `pdf`, `screenshot`, `wait_for`, `tabs`. Plus `assess` (privacy scan) if [wearehere](https://github.com/hamr0/wearehere) is installed. Plus opt-in `eval` (`BAREBROWSE_MCP_EVAL=1`) — runs JS in the authenticated session, off by default because it can read cookies/localStorage. Session runs in hybrid mode with automatic cookie injection. Per-tool timeouts (goto/reload/wait_for 60s, back/forward 30s, interactive ops 15s, pdf/screenshot/upload 45s) with auto-retry on transient failures (idempotent only — mutating tools fail loudly to avoid double-submits).
100
100
 
101
- `browse` and `snapshot` accept `pruneMode: 'act'|'read'` (v0.9.1). `act` (default) keeps interactive elements — best for clicking/filling. `read` keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. If act-mode collapses a content-heavy page near-totally, the snapshot includes a `hint: …` line suggesting `pruneMode='read'` so the agent doesn't bail to a separate HTTP fetch.
101
+ `browse` and `snapshot` accept `pruneMode: 'act'|'read'`. `act` (default) keeps interactive elements — best for clicking/filling. `read` keeps paragraphs, headings, and long text — best for articles, docs, and content extraction. If act-mode collapses a content-heavy page near-totally, the snapshot includes a `hint: …` line suggesting `pruneMode='read'` so the agent doesn't bail to a separate HTTP fetch.
102
102
 
103
103
  Troubleshooting MCP setup: `npx barebrowse doctor` scans every known config location and flags scope conflicts. `npx barebrowse install --force` overwrites an existing entry pointing at a different endpoint.
104
104
 
105
105
  ### 3. Library -- for agentic automation
106
106
 
107
- Import barebrowse in your agent code. One-shot reads, interactive sessions, full observe-think-act loops. Works with any LLM orchestration library. Ships with a ready-made adapter for [bareagent](https://www.npmjs.com/package/bare-agent) (18 tools, auto-snapshot after every action).
107
+ Import barebrowse in your agent code. One-shot reads, interactive sessions, full observe-think-act loops. Works with any LLM orchestration library. Ships with a ready-made adapter for [bareagent](https://www.npmjs.com/package/bare-agent) (auto-snapshot after every action).
108
108
 
109
109
  For code examples, API reference, and wiring instructions, see **[barebrowse.context.md](barebrowse.context.md)** -- the full integration guide.
110
110
 
@@ -112,10 +112,12 @@ For code examples, API reference, and wiring instructions, see **[barebrowse.con
112
112
 
113
113
  | Mode | What happens | Best for |
114
114
  |------|-------------|----------|
115
- | **Headless** (default) | Launches a fresh Chromium, no UI | Fast automation, scraping, reading pages |
116
- | **Headed** | Auto-launches a visible Chromium window | Bot-detected sites, visual debugging, CAPTCHAs |
115
+ | **Headless** (default) | Launches a fresh browser, no UI | Fast automation, scraping, reading pages |
116
+ | **Headed** | Auto-launches a visible browser window | Bot-detected sites, visual debugging, CAPTCHAs |
117
117
  | **Hybrid** | Tries headless first, auto-launches headed if blocked | General-purpose agent browsing |
118
118
 
119
+ All three work on both engines (Chromium/CDP and Firefox/BiDi).
120
+
119
121
  ### Attach to your already-running browser
120
122
 
121
123
  Start Chromium yourself with a debug port, then drive your real logged-in session:
@@ -135,23 +137,22 @@ await page.close(); // closes only the tab barebrowse opened — your browser ke
135
137
  ### Firefox (WebDriver BiDi)
136
138
 
137
139
  CDP is deprecated in Firefox, so barebrowse drives it over the W3C WebDriver
138
- BiDi protocol instead — a second transport over the same `ws` dependency, no
139
- extra download. Same `page.*` API (the ARIA snapshot is reconstructed in-page
140
- since BiDi has no `getFullAXTree`):
140
+ BiDi protocol — a second transport over the same `ws` dependency, no extra
141
+ download. Same `page.*` API (the ARIA snapshot is reconstructed in-page since
142
+ BiDi has no `getFullAXTree`):
141
143
 
142
144
  ```js
143
- const page = await connect({ engine: 'firefox' }); // headless by default
145
+ const page = await connect({ engine: 'firefox' });
144
146
  ```
145
147
 
146
- From the CLI: `barebrowse open <url> --engine firefox`. From MCP: set
147
- `BAREBROWSE_ENGINE=firefox`. Firefox cookies (plaintext) reuse into the same
148
- engine. Consent auto-dismiss and headless stealth (v0.16.0), the daemon's
149
- console/network capture and `waitForNetworkIdle` (v0.17.0), and ad/tracker
150
- blocking plus JS dialog handling (`dialogLog`/`onDialog`, v0.18.0) all work on
151
- Firefox too. Chromium (CDP) remains the default; only `hybrid` mode is still
152
- Chromium-only.
148
+ CLI: `barebrowse open <url> --engine firefox`. MCP: `BAREBROWSE_ENGINE=firefox`.
149
+ Firefox cookies (plaintext) reuse into the same engine.
153
150
 
154
- No clone profile, no fresh cookies the agent sees what you see.
151
+ Firefox is at practical parity with Chromium: stealth, consent auto-dismiss,
152
+ ad/tracker blocking, JS dialogs, console/network capture, hybrid fallback,
153
+ `saveState`, `waitForNavigation`, and download tracking all work the same way.
154
+ Chromium (CDP) stays the default; the only remaining gap is
155
+ `reload({ignoreCache})` (upstream BiDi limitation).
155
156
 
156
157
  ### Incognito (clean session)
157
158
 
@@ -170,9 +171,9 @@ From the CLI: `barebrowse open <url> --incognito`. From MCP: set
170
171
 
171
172
  ## What it handles automatically
172
173
 
173
- Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (ARIA node count heuristic + stealth patches + automatic headed fallback — snapshot shows `[BOT CHALLENGE DETECTED]` warning when blocked), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
174
+ Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (challenge-phrase + node-count heuristic + stealth patches + automatic headed fallback — snapshot shows `[BOT CHALLENGE DETECTED]` warning when blocked), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
174
175
 
175
- ## Safe by default (v0.11.0)
176
+ ## Safe by default
176
177
 
177
178
  barebrowse hands an autonomous — and therefore prompt-injectable — agent an *authenticated* browser, so the defaults are calibrated for that threat:
178
179
 
@@ -223,8 +224,8 @@ Everything the agent can do through barebrowse:
223
224
  | **Wait for network idle** | Resolve when no pending requests for 500ms |
224
225
  | **Dialog handling** | Auto-dismiss JS alert/confirm/prompt dialogs |
225
226
  | **Save state** | Export cookies + localStorage to JSON |
226
- | **Inject cookies** | Extract from Firefox/Chromium and inject via CDP |
227
- | **Raw CDP** | Escape hatch for any Chrome DevTools Protocol command |
227
+ | **Inject cookies** | Extract from Firefox/Chromium and inject into the session (CDP or BiDi) |
228
+ | **Raw CDP / BiDi** | Escape hatch: `page.cdp` (Chromium) or `page.bidi` (Firefox) for any low-level command |
228
229
 
229
230
  ## Tested against
230
231
 
@@ -232,10 +233,6 @@ Everything the agent can do through barebrowse:
232
233
 
233
234
  Google, YouTube, BBC, Wikipedia, GitHub, DuckDuckGo, Hacker News, Amazon DE, The Guardian, Spiegel, Le Monde, El Pais, Corriere, NOS, Bild, Nu.nl, Booking, NYT, Stack Overflow, CNN, Reddit
234
235
 
235
- ## Context file
236
-
237
- **[barebrowse.context.md](barebrowse.context.md)** is the full integration guide. Feed it to an AI assistant or read it yourself -- it covers the complete API, snapshot format, interaction loop, auth options, bareagent wiring, MCP setup, and gotchas. Everything you need to wire barebrowse into a project.
238
-
239
236
  ## How it works
240
237
 
241
238
  ```
@@ -247,12 +244,15 @@ URL -> find/launch browser (chromium.js)
247
244
  -> navigate to URL, wait for load
248
245
  -> detect + dismiss cookie consent dialogs (consent.js)
249
246
  -> get full ARIA accessibility tree (aria.js)
250
- -> 9-step pruning pipeline from mcprune (prune.js)
247
+ -> pruning pipeline from mcprune (prune.js)
251
248
  -> dispatch real input events: click/type/scroll (interact.js)
252
249
  -> agent-ready snapshot with [ref=N] markers
253
250
  ```
254
251
 
255
- 14 modules, ~3,000 lines, two small dependencies (`ws`, `@mozilla/readability`).
252
+ Firefox follows the same flow over WebDriver BiDi (`bidi.js`), with the AX tree
253
+ reconstructed in-page (`ax-snapshot.js`) and pruning/formatting shared verbatim.
254
+
255
+ A small set of focused modules, two small dependencies (`ws`, `@mozilla/readability`).
256
256
 
257
257
  ## Requirements
258
258
 
@@ -95,7 +95,7 @@ const snapshot = await browse('https://example.com', {
95
95
  | `close()` | -- | void | Close page, disconnect CDP, kill browser (if headless) |
96
96
 
97
97
  **connect() options** (in addition to mode/port/consent):
98
- - `engine: 'chromium'|'firefox'` — Browser engine. Default `chromium` (CDP). `firefox` drives Firefox over WebDriver BiDi (CDP is deprecated there) — a separate transport with the same `page.*` API; the ARIA snapshot is reconstructed in-page since BiDi has no `getFullAXTree`. Consent auto-dismiss and stealth (headless anti-detection) work on Firefox as of v0.16.0; the CLI daemon's console/network capture and `waitForNetworkIdle` work as of v0.17.0; ad/tracker blocking (`blockAds`/`blockUrls`) and JS dialog handling (`dialogLog`/`onDialog`) work as of Phase 3. Still Chromium-only: `hybrid` mode. `reload()` ignores `ignoreCache` on Firefox (BiDi limitation).
98
+ - `engine: 'chromium'|'firefox'` — Browser engine. Default `chromium` (CDP). `firefox` drives Firefox over WebDriver BiDi (CDP is deprecated there) — a separate transport with the same `page.*` API; the ARIA snapshot is reconstructed in-page since BiDi has no `getFullAXTree`. Consent auto-dismiss and stealth (headless anti-detection) work on Firefox as of v0.16.0; the CLI daemon's console/network capture and `waitForNetworkIdle` work as of v0.17.0; ad/tracker blocking (`blockAds`/`blockUrls`) and JS dialog handling (`dialogLog`/`onDialog`) work as of Phase 3; `hybrid` mode, `saveState`, `waitForNavigation`, and download tracking (`page.downloads`) work as of Phase 4. The only remaining Firefox gap is `reload()` ignoring `ignoreCache` (upstream BiDi limitation).
99
99
  - `port: 9222` — Attach to a Chromium already running with `--remote-debugging-port=N` instead of spawning one. The browser keeps running on `close()`. Stealth + permission denial + download capture are skipped to avoid mutating the user's running browser.
100
100
  - `proxy: 'http://...'` — HTTP/SOCKS proxy for browser
101
101
  - `viewport: '1280x720'` — Set viewport dimensions
package/cli.js CHANGED
@@ -487,7 +487,7 @@ Session:
487
487
  --engine=chromium|firefox Browser engine (default: chromium/CDP;
488
488
  firefox drives over WebDriver BiDi)
489
489
  --mode=headless|headed|hybrid Browser mode (default: headless;
490
- hybrid is chromium-only)
490
+ hybrid works on both engines)
491
491
  --port=N CDP port for headed mode
492
492
  --no-cookies Skip cookie injection
493
493
  --incognito Clean, unauthenticated session (skip all auth injection)
package/mcp-server.js CHANGED
@@ -123,15 +123,17 @@ async function getPage() {
123
123
  if (_pageConnecting) return _pageConnecting;
124
124
  // Engine selection is a launch-time setting (the browser persists across
125
125
  // tool calls), so it's an env var, not a per-request flag. Firefox is driven
126
- // over WebDriver BiDi and has no hybrid fallback default it to headless
127
- // (override with BAREBROWSE_MODE=headed).
126
+ // over WebDriver BiDi and, as of parity Phase 4, has the same hybrid fallback
127
+ // as Chromium (relaunch headed on a bot-challenge page) — so it defaults to
128
+ // hybrid too, matching the Chromium default (override with BAREBROWSE_MODE).
129
+ // A failed headed relaunch (no display) falls back to the headless result.
128
130
  // Incognito is a launch-time setting like engine/mode: the session persists
129
131
  // across tool calls, so it's an env var (BAREBROWSE_INCOGNITO=1), not a
130
132
  // per-request flag. Skips all auth injection for a clean, logged-out session.
131
133
  const incognito = process.env.BAREBROWSE_INCOGNITO === '1';
132
134
  const connectOpts = process.env.BAREBROWSE_ENGINE === 'firefox'
133
- ? { engine: 'firefox', mode: process.env.BAREBROWSE_MODE || 'headless', incognito }
134
- : { mode: 'hybrid', incognito };
135
+ ? { engine: 'firefox', mode: process.env.BAREBROWSE_MODE || 'hybrid', incognito }
136
+ : { mode: process.env.BAREBROWSE_MODE || 'hybrid', incognito };
135
137
  _pageConnecting = connect(connectOpts);
136
138
  try {
137
139
  _page = await _pageConnecting;
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "barebrowse",
3
- "version": "0.18.0",
3
+ "version": "0.19.2",
4
4
  "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
5
5
  "repository": {
6
6
  "type": "git",
@@ -0,0 +1,77 @@
1
+ /**
2
+ * challenge.js — Bot-challenge / interstitial detection for the hybrid fallback.
3
+ *
4
+ * Extracted from index.js so both engines can share it without a circular
5
+ * import (index.js drives CDP; firefox-page.js drives BiDi, and both need to
6
+ * decide "is this a Cloudflare/hCaptcha wall we should retry headed?"). Pure —
7
+ * operates on the nested ARIA tree that buildTree() / ariaTree() produce, which
8
+ * is identical in shape across engines.
9
+ */
10
+
11
+ /**
12
+ * Heuristic: does this ARIA tree look like a bot-challenge / block interstitial
13
+ * (Cloudflare "Just a moment", hCaptcha, Akamai) rather than real content?
14
+ *
15
+ * H9 split: STRONG_PHRASES are essentially-unambiguous challenge UI and fire
16
+ * regardless of page size; WEAK_PHRASES only fire when the page is ALSO tiny
17
+ * (so a legitimate-looking error page with "access denied" in its body doesn't
18
+ * trip the fallback).
19
+ *
20
+ * @param {object} tree - Nested ARIA tree (from buildTree / ariaTree)
21
+ * @param {number} [nodeCount] - Node count (CDP: getFullAXTree; BiDi: tree walk)
22
+ * @returns {boolean}
23
+ */
24
+ export function isChallengePage(tree, nodeCount) {
25
+ if (!tree) return true; // truly empty AX tree — something went wrong fetching the page
26
+
27
+ const text = flattenTreeText(tree);
28
+ const lower = text.toLowerCase();
29
+
30
+ // Strong phrases — distinctive enough to identify the challenge product
31
+ // by name. Fire on their own regardless of node count.
32
+ const STRONG_PHRASES = [
33
+ 'just a moment', // Cloudflare interstitial
34
+ 'checking if the site connection is secure', // Cloudflare
35
+ 'checking your browser', // Various JS challenges
36
+ 'verify you are human', // hCaptcha / reCAPTCHA
37
+ 'prove your humanity',
38
+ 'attention required', // Cloudflare block page
39
+ 'enable javascript and cookies to continue', // Cloudflare
40
+ 'please complete the security check', // Cloudflare/Akamai
41
+ ];
42
+ if (STRONG_PHRASES.some((p) => lower.includes(p))) return true;
43
+
44
+ // Weak phrases — show up on real challenge pages but ALSO on legitimate
45
+ // small error pages. Only count when the page is itself tiny (low node
46
+ // count or near-empty text), which is the corroborating signal that
47
+ // separates a real error UI from a challenge skeleton.
48
+ const WEAK_PHRASES = [
49
+ 'please wait',
50
+ 'request blocked',
51
+ 'access denied',
52
+ 'permission denied',
53
+ 'unknown error',
54
+ 'file a ticket',
55
+ ];
56
+ const tinyPage = (nodeCount !== undefined && nodeCount < 30) || text.trim().length < 50;
57
+ if (tinyPage && WEAK_PHRASES.some((p) => lower.includes(p))) return true;
58
+
59
+ return false;
60
+ }
61
+
62
+ /** Count every node in a nested ARIA tree (shared node-count for the tinyPage test). */
63
+ export function countNodes(node) {
64
+ if (!node) return 0;
65
+ let n = 1;
66
+ for (const c of node.children || []) n += countNodes(c);
67
+ return n;
68
+ }
69
+
70
+ function flattenTreeText(node) {
71
+ if (!node) return '';
72
+ let text = node.name || '';
73
+ for (const child of node.children || []) {
74
+ text += ' ' + flattenTreeText(child);
75
+ }
76
+ return text;
77
+ }
package/src/daemon.js CHANGED
@@ -67,6 +67,31 @@ export function buildDaemonArgs(opts, absDir, initialUrl, cliPath) {
67
67
  return args;
68
68
  }
69
69
 
70
+ /**
71
+ * Convert a BiDi remote value ({type,value} — see script.evaluate) into the
72
+ * plain JS value CDP's `Runtime.evaluate({returnByValue:true})` yields, so an
73
+ * `eval` returns the same shape on both engines. BiDi serializes objects/arrays
74
+ * as entry lists ([[k,v],…] / [v,…]) rather than plain values; walk them back.
75
+ * @param {object} r - BiDi remote value
76
+ * @returns {*}
77
+ */
78
+ export function deserializeBiDi(r) {
79
+ if (!r || typeof r !== 'object') return r;
80
+ switch (r.type) {
81
+ case 'undefined': return undefined;
82
+ case 'null': return null;
83
+ case 'string': case 'number': case 'boolean': case 'bigint': return r.value;
84
+ case 'array': case 'set': return (r.value || []).map(deserializeBiDi);
85
+ case 'object': case 'map':
86
+ return Object.fromEntries(
87
+ (r.value || []).map(([k, v]) => [typeof k === 'object' ? deserializeBiDi(k) : k, deserializeBiDi(v)]),
88
+ );
89
+ // date/regexp/node/function/etc. — no plain-value equivalent; use the raw
90
+ // value if BiDi provided one, else the type name (matches CDP's coarseness).
91
+ default: return r.value !== undefined ? r.value : r.type;
92
+ }
93
+ }
94
+
70
95
  /**
71
96
  * Wire Firefox/BiDi console + network capture onto the daemon's log arrays.
72
97
  * The BiDi analogue of the CDP `Runtime.consoleAPICalled` / `Network.*`
@@ -99,8 +124,12 @@ export async function attachBiDiCapture(bidi, { consoleLogs, networkLogs, pendin
99
124
  // engines — BiDi says 'warn', CDP says 'warning'.
100
125
  type: e.method === 'warn' ? 'warning' : (e.method || e.level),
101
126
  timestamp: new Date().toISOString(),
127
+ // Primitive args carry a usable `.value`; objects/arrays/functions carry a
128
+ // nested {type,value} serialization we must NOT splat into the log (it's
129
+ // deep junk) — fall back to the type name, mirroring the CDP capture which
130
+ // logs a RemoteObject's `.description` ("Object") for non-primitives.
102
131
  args: Array.isArray(e.args) && e.args.length
103
- ? e.args.map((a) => a.value ?? a.type)
132
+ ? e.args.map((a) => (a.value !== undefined && typeof a.value !== 'object' ? a.value : a.type))
104
133
  : [e.text],
105
134
  });
106
135
  });
@@ -416,13 +445,20 @@ export async function runDaemon(opts, outputDir, initialUrl) {
416
445
  // Firefox/BiDi has no page.cdp — evaluate over BiDi in the active context.
417
446
  if (!page.cdp) {
418
447
  try {
419
- // BiDi's raw result.value serializes objects into a {type,value}
420
- // entries structure, not a plain object (CDP's returnByValue does).
421
- // Stringify + parse in-page so an object/array eval matches the CDP
422
- // shape (same trick readable() uses). `undefined` JSON `null`.
423
- const wrapped = `(async () => { const v = await (${expression}); return JSON.stringify(v === undefined ? null : v); })()`;
424
- const raw = await page.bidi.evaluate(page.context, wrapped, true);
425
- return { ok: true, value: raw == null ? null : JSON.parse(raw) };
448
+ // Evaluate the RAW expression: script.evaluate has eval-semantics, so
449
+ // statement lists ("var x = 2; x * 3") work and yield the completion
450
+ // value matching the CDP Runtime.evaluate path. (An earlier
451
+ // `await (${expression})` wrapper flattened objects but broke every
452
+ // statement-form expression with a SyntaxError.) BiDi returns a
453
+ // {type,value} remote value; deserializeBiDi walks it back to the plain
454
+ // JS value CDP's returnByValue produces.
455
+ const res = await page.bidi.send('script.evaluate', {
456
+ expression, target: { context: page.context }, awaitPromise: true,
457
+ });
458
+ if (res.type === 'exception') {
459
+ return { ok: false, error: res.exceptionDetails?.text || 'eval error' };
460
+ }
461
+ return { ok: true, value: deserializeBiDi(res.result) };
426
462
  } catch (err) {
427
463
  return { ok: false, error: err.message || 'eval error' };
428
464
  }
package/src/dialog.js ADDED
@@ -0,0 +1,52 @@
1
+ /**
2
+ * dialog.js — Shared JS-dialog decision core for both engines.
3
+ *
4
+ * A JS dialog (alert/confirm/prompt/beforeunload) surfaces differently per
5
+ * protocol — CDP's `Page.javascriptDialogOpening` vs BiDi's
6
+ * `browsingContext.userPromptOpened` — but the *decision* (accept? what prompt
7
+ * text?) and the log entry are identical. This module single-sources both so
8
+ * the CDP path (index.js) and the BiDi path (firefox-page.js) can't drift
9
+ * (code-review Phase-3 finding #5).
10
+ */
11
+
12
+ /**
13
+ * Build a dialogLog entry from a normalized dialog descriptor.
14
+ * @param {string} type - 'alert' | 'confirm' | 'prompt' | 'beforeunload'
15
+ * @param {string} [message]
16
+ * @returns {{type: string, message: string, timestamp: string}}
17
+ */
18
+ export function dialogLogEntry(type, message) {
19
+ return { type, message: message || '', timestamp: new Date().toISOString() };
20
+ }
21
+
22
+ /**
23
+ * Decide how to answer a JS dialog. Default policy (both engines): accept
24
+ * everything except `beforeunload` (dismiss = stay on page); a `prompt`
25
+ * returns its default text. A caller-installed handler may override via
26
+ * `{ accept, promptText }`; if the handler throws we keep the defaults so the
27
+ * page never hangs waiting for a reply that never arrives.
28
+ *
29
+ * @param {{type: string, message?: string, defaultPrompt?: string}} dialog
30
+ * @param {?(function({type,message,defaultPrompt}): (object|Promise<object>))} handler
31
+ * @returns {Promise<{accept: boolean, promptText: string}>}
32
+ */
33
+ export async function decideDialog({ type, message, defaultPrompt }, handler) {
34
+ let accept = type !== 'beforeunload';
35
+ let promptText = defaultPrompt || '';
36
+ if (handler) {
37
+ try {
38
+ const decision = await handler({
39
+ type,
40
+ message: message || '',
41
+ defaultPrompt: defaultPrompt || '',
42
+ });
43
+ if (decision && typeof decision === 'object') {
44
+ if (typeof decision.accept === 'boolean') accept = decision.accept;
45
+ if (typeof decision.promptText === 'string') promptText = decision.promptText;
46
+ }
47
+ } catch {
48
+ // Handler threw — keep defaults so the page doesn't hang.
49
+ }
50
+ }
51
+ return { accept, promptText };
52
+ }
@@ -23,6 +23,8 @@ import { scopedCookiesForUrl } from './auth.js';
23
23
  import { assertNavigable, assertUploadAllowed } from './url-guard.js';
24
24
  import { dismissConsentFirefox } from './consent-firefox.js';
25
25
  import { waitForNetworkIdleBiDi } from './network-idle.js';
26
+ import { decideDialog, dialogLogEntry } from './dialog.js';
27
+ import { isChallengePage, countNodes } from './challenge.js';
26
28
 
27
29
  /** BiDi/WebDriver normalized key values for named keys (U+E000 block). */
28
30
  const BIDI_KEYS = {
@@ -41,6 +43,9 @@ const BIDI_KEYS = {
41
43
  * @param {string} [opts.uploadDir] - When set, upload() rejects files outside this dir.
42
44
  * @param {boolean} [opts.incognito=false] - Clean session: injectCookies() is a no-op.
43
45
  * @param {boolean} [opts.consent=true] - Auto-dismiss cookie consent dialogs after goto().
46
+ * @param {boolean} [opts.hybrid=false] - Relaunch headed on a bot-challenge page and retry.
47
+ * @param {boolean} [opts.headed=false] - Whether the browser was launched headed (skips hybrid fallback).
48
+ * @param {?function(): Promise<{bidi: object, topContext: string}>} [opts.relaunchHeaded] - Hybrid relaunch hook (from connectFirefox).
44
49
  * @returns {Promise<object>} page object
45
50
  */
46
51
  export async function createFirefoxPage(bidi, opts = {}) {
@@ -49,6 +54,17 @@ export async function createFirefoxPage(bidi, opts = {}) {
49
54
  const uploadDir = opts.uploadDir || null;
50
55
  const incognito = !!opts.incognito;
51
56
  const consent = opts.consent !== false;
57
+ // Hybrid mode: on a bot-challenge page, relaunch headed and retry. The
58
+ // relaunch tears down this Firefox+BiDi and returns a fresh one, so the
59
+ // browser lifecycle is owned by connectFirefox, which supplies relaunchHeaded.
60
+ const hybrid = !!opts.hybrid;
61
+ const relaunchHeaded = opts.relaunchHeaded || null;
62
+ let currentlyHeaded = !!opts.headed; // already headed → no fallback needed
63
+ let botBlocked = false;
64
+ // Last injectCookies(url, opts) args, so a hybrid relaunch (a brand-new
65
+ // Firefox profile) can re-inject them — otherwise the headed retry loads
66
+ // unauthenticated, defeating hybrid on an auth-gated challenge page.
67
+ let lastInject = null;
52
68
  // The active browsing context. Starts at the initial tab; switchTab() points
53
69
  // it at another top-level context, so it's mutable and read via a getter.
54
70
  const { contexts } = await bidi.send('browsingContext.getTree', {});
@@ -170,6 +186,37 @@ export async function createFirefoxPage(bidi, opts = {}) {
170
186
  return Promise.race([p, t]).finally(() => clearTimeout(timer));
171
187
  }
172
188
 
189
+ // Download tracking, parity with the CDP `page.downloads` array. Firefox
190
+ // downloads land in `downloadDir` (a throwaway dir the caller sets via launch
191
+ // prefs — see connectFirefox / firefox.js), so `savedPath` is a real path the
192
+ // caller can read. BiDi emits browsingContext.downloadWillBegin (start) +
193
+ // downloadEnd (finish, with the actual filepath + status) — measured against
194
+ // real Firefox. Records mirror the CDP shape:
195
+ // { url, suggestedFilename, savedPath, state, totalBytes, receivedBytes }.
196
+ const downloads = [];
197
+ async function setupDownloads() {
198
+ await bidi.subscribe(['browsingContext.downloadWillBegin', 'browsingContext.downloadEnd']);
199
+ bidi.on('browsingContext.downloadWillBegin', (e) => {
200
+ downloads.push({
201
+ url: e.url,
202
+ suggestedFilename: e.suggestedFilename || '',
203
+ savedPath: null,
204
+ state: 'inProgress',
205
+ totalBytes: 0,
206
+ receivedBytes: 0,
207
+ });
208
+ });
209
+ bidi.on('browsingContext.downloadEnd', (e) => {
210
+ // Correlate to the most recent still-in-progress record for this URL.
211
+ const d = [...downloads].reverse().find((x) => x.url === e.url && x.state === 'inProgress');
212
+ if (!d) return;
213
+ // BiDi status is 'complete' | 'canceled'; normalize 'complete' to CDP's
214
+ // 'completed' so callers can branch on one vocabulary across engines.
215
+ d.state = e.status === 'complete' ? 'completed' : e.status || 'completed';
216
+ d.savedPath = e.filepath || null;
217
+ });
218
+ }
219
+
173
220
  // JS dialog handling (alert/confirm/prompt/beforeunload), parity with the
174
221
  // CDP path in index.js. The BiDi session is created with
175
222
  // unhandledPromptBehavior:'ignore' (see bidi.js) so prompts stay open until
@@ -182,32 +229,16 @@ export async function createFirefoxPage(bidi, opts = {}) {
182
229
  async function setupDialogs() {
183
230
  await bidi.subscribe(['browsingContext.userPromptOpened']);
184
231
  bidi.on('browsingContext.userPromptOpened', async (e) => {
185
- dialogLog.push({
186
- type: e.type,
187
- message: e.message || '',
188
- timestamp: new Date().toISOString(),
189
- });
190
- let accept = e.type !== 'beforeunload';
191
- let userText = e.defaultValue || '';
192
- if (onDialogHandler) {
193
- try {
194
- const decision = await onDialogHandler({
195
- type: e.type,
196
- message: e.message || '',
197
- defaultPrompt: e.defaultValue || '',
198
- });
199
- if (decision && typeof decision === 'object') {
200
- if (typeof decision.accept === 'boolean') accept = decision.accept;
201
- if (typeof decision.promptText === 'string') userText = decision.promptText;
202
- }
203
- } catch {
204
- // Handler threw — fall back to defaults so the prompt is still
205
- // answered and the page doesn't hang on an 'ignore' dialog.
206
- }
207
- }
232
+ dialogLog.push(dialogLogEntry(e.type, e.message));
233
+ // Shared decision core with the CDP path (dialog.js). BiDi's userText is
234
+ // the CDP promptText; its defaultValue is the CDP defaultPrompt.
235
+ const { accept, promptText } = await decideDialog(
236
+ { type: e.type, message: e.message, defaultPrompt: e.defaultValue },
237
+ onDialogHandler,
238
+ );
208
239
  try {
209
240
  await bidi.send('browsingContext.handleUserPrompt', {
210
- context: e.context, accept, userText,
241
+ context: e.context, accept, userText: promptText,
211
242
  });
212
243
  } catch {
213
244
  // Prompt already gone (closed by page JS / navigation). Nothing to do.
@@ -215,29 +246,83 @@ export async function createFirefoxPage(bidi, opts = {}) {
215
246
  });
216
247
  }
217
248
 
249
+ /**
250
+ * Wire all event subscriptions (dialogs, downloads, load) on the CURRENT
251
+ * bidi connection. Run once at construction, and again after a hybrid
252
+ * relaunch swaps in a fresh connection. Dialogs must be wired before any
253
+ * navigation (the 'ignore' capability would otherwise hang a prompt).
254
+ */
255
+ async function setupSubscriptions() {
256
+ await setupDialogs();
257
+ await setupDownloads();
258
+ await bidi.subscribe(['browsingContext.load']);
259
+ }
260
+
261
+ /**
262
+ * Post-navigation routine shared by goto() and the hybrid retry: settle for
263
+ * dynamic content, auto-dismiss consent (best-effort), then report whether
264
+ * the resulting page looks like a bot challenge.
265
+ */
266
+ async function afterNavigate() {
267
+ await new Promise((r) => setTimeout(r, 500));
268
+ // Build the tree ONCE and use it for both consent dismissal and challenge
269
+ // detection. A challenge page (Cloudflare/hCaptcha) never carries a consent
270
+ // banner, so dismissing consent can't flip the challenge verdict — reusing
271
+ // the pre-dismiss tree avoids a second full AX reconstruction per goto.
272
+ let root = null;
273
+ try { root = await buildTree(); } catch { /* null → treated as challenge */ }
274
+ if (consent && root) {
275
+ try { await dismissConsentFirefox(root, (ref) => pointerClick(ref)); }
276
+ catch { /* consent dismissal is best-effort */ }
277
+ }
278
+ return isChallengePage(root, countNodes(root));
279
+ }
280
+
218
281
  const page = {
219
282
  /** The BiDi escape hatch, analogous to connect()'s page.cdp. */
220
283
  get bidi() { return bidi; },
221
284
  /** The active browsing-context id (getter so it tracks switchTab). */
222
285
  get context() { return topContext; },
286
+ /** Whether the last goto() landed on a bot-challenge page (parity w/ CDP). */
287
+ get botBlocked() { return botBlocked; },
223
288
 
224
289
  async goto(url, timeout = 30000) {
225
290
  // Same navigation guard the CDP path enforces — block file:/chrome:/
226
291
  // view-source: and (optionally) private-network hosts before navigating.
227
292
  assertNavigable(url, urlGuard);
293
+ refContexts = new Map(); // refs from the previous page are now stale
228
294
  await withTimeout(
229
295
  bidi.send('browsingContext.navigate', { context: topContext, url, wait: 'complete' }),
230
296
  timeout, `goto(${url})`);
231
- // Brief settle for dynamic/SPA content, matching the CDP path.
232
- await new Promise((r) => setTimeout(r, 500));
233
- // Auto-dismiss cookie consent, mirroring the CDP path (index.js runs
234
- // dismissConsent right after navigate). Best-effort: a failure here must
235
- // never fail the navigation.
236
- if (consent) {
297
+ // Settle + consent + challenge detection (mirrors the CDP path).
298
+ botBlocked = await afterNavigate();
299
+
300
+ // Hybrid fallback: on a bot-challenge page, relaunch headed once and
301
+ // retry a real display often clears an interstitial headless can't.
302
+ // Only when hybrid mode and we're still headless. relaunchHeaded (from
303
+ // connectFirefox) tears down this Firefox+BiDi and returns a fresh one.
304
+ if (botBlocked && hybrid && !currentlyHeaded && relaunchHeaded) {
237
305
  try {
238
- const root = await buildTree();
239
- if (root) await dismissConsentFirefox(root, (ref) => pointerClick(ref));
240
- } catch { /* consent dismissal is best-effort */ }
306
+ const relaunched = await relaunchHeaded();
307
+ bidi = relaunched.bidi; // rebind the closure — all inner fns follow
308
+ topContext = relaunched.topContext;
309
+ currentlyHeaded = true;
310
+ refContexts = new Map();
311
+ await setupSubscriptions(); // re-wire dialogs/downloads/load on the new bidi
312
+ // Re-inject cookies into the fresh profile BEFORE navigating, so the
313
+ // headed retry is authenticated (the relaunch is a new Firefox profile
314
+ // — the pre-relaunch session's cookies are gone).
315
+ if (lastInject) {
316
+ try { await page.injectCookies(lastInject.url, lastInject.cookieOpts); } catch { /* best-effort */ }
317
+ }
318
+ await withTimeout(
319
+ bidi.send('browsingContext.navigate', { context: topContext, url, wait: 'complete' }),
320
+ timeout, `goto(${url})`);
321
+ botBlocked = await afterNavigate();
322
+ } catch {
323
+ // Headed relaunch failed (no display?) — keep the headless result;
324
+ // botBlocked stays true so the caller can see it didn't clear.
325
+ }
241
326
  }
242
327
  },
243
328
 
@@ -308,6 +393,7 @@ export async function createFirefoxPage(bidi, opts = {}) {
308
393
  */
309
394
  async injectCookies(url, cookieOpts) {
310
395
  if (incognito) return 0;
396
+ lastInject = { url, cookieOpts }; // remember for a hybrid re-inject
311
397
  const cookies = scopedCookiesForUrl(url, { browser: cookieOpts?.browser });
312
398
  let injected = 0;
313
399
  for (const c of cookies) {
@@ -456,14 +542,8 @@ export async function createFirefoxPage(bidi, opts = {}) {
456
542
  return waitForNetworkIdleBiDi(bidi, idleOpts);
457
543
  },
458
544
 
459
- // --- CDP-only surfaces, stubbed for daemon parity ---------------------
460
- // The daemon (src/daemon.js) dispatches these unconditionally. On the
461
- // Firefox/BiDi engine download tracking isn't wired (Phase 4), so downloads
462
- // is genuinely empty; saveState/waitForNavigation are CDP-only and fail with
463
- // a clear, intentional message instead of an incidental TypeError.
464
- // (dialogLog + onDialog ARE wired — Phase 3, see setupDialogs above.)
465
- // Documented as a known gap in CHANGELOG.
466
- get downloads() { return []; },
545
+ /** Downloads that began in this session (Phase 4 — see setupDownloads). */
546
+ get downloads() { return downloads; },
467
547
 
468
548
  dialogLog,
469
549
 
@@ -477,11 +557,64 @@ export async function createFirefoxPage(bidi, opts = {}) {
477
557
  onDialogHandler = handler;
478
558
  },
479
559
 
480
- async saveState() {
481
- throw new Error('saveState() is not supported on the Firefox/BiDi engine (CDP-only)');
560
+ /**
561
+ * Persist cookies + localStorage to a JSON file (parity with the CDP
562
+ * saveState). Cookies come from BiDi storage.getCookies (whose value is a
563
+ * `{type,value}` object) and are flattened to the CDP-symmetric shape so the
564
+ * file format matches across engines. Written 0600 — it holds session
565
+ * tokens, so a multi-user host must not be able to read another user's.
566
+ */
567
+ async saveState(filePath) {
568
+ const { cookies } = await bidi.send('storage.getCookies', {});
569
+ const flat = cookies.map((c) => {
570
+ const out = {
571
+ name: c.name,
572
+ value: c.value && typeof c.value === 'object' ? c.value.value : c.value,
573
+ domain: c.domain,
574
+ path: c.path,
575
+ secure: !!c.secure,
576
+ httpOnly: !!c.httpOnly,
577
+ };
578
+ // BiDi reports sameSite lowercase (strict|lax|none); the CDP saveState /
579
+ // Network.setCookies vocabulary is capitalized (Strict|Lax|None). The only
580
+ // loader for a state file is connect()'s storageState (CDP-only), which
581
+ // rejects a lowercase enum and silently drops every cookie — so capitalize
582
+ // here to keep the format truly symmetric and the round-trip working.
583
+ if (c.sameSite && c.sameSite !== 'default') {
584
+ out.sameSite = c.sameSite.charAt(0).toUpperCase() + c.sameSite.slice(1);
585
+ }
586
+ if (c.expiry && c.expiry > 0) out.expires = c.expiry;
587
+ return out;
588
+ });
589
+ const lsRaw = await bidi
590
+ .evaluate(topContext, 'JSON.stringify(Object.fromEntries(Object.entries(localStorage)))', false)
591
+ .catch(() => '{}'); // opaque-origin pages (about:blank) throw — treat as empty
592
+ const state = { cookies: flat, localStorage: JSON.parse(lsRaw || '{}') };
593
+ const { writeFileSync, chmodSync } = await import('node:fs');
594
+ writeFileSync(filePath, JSON.stringify(state, null, 2), { mode: 0o600 });
595
+ try { chmodSync(filePath, 0o600); } catch { /* best effort if pre-existing */ }
482
596
  },
483
- async waitForNavigation() {
484
- throw new Error('waitForNavigation() is not supported on the Firefox/BiDi engine (CDP-only)');
597
+
598
+ /**
599
+ * Resolve when the active tab fires its next load event (parity with the
600
+ * CDP waitForNavigation → Page.loadEventFired). Scoped to topContext so a
601
+ * subframe load can't resolve it early. Falls back to a short settle delay
602
+ * for SPA navigations that fire no load event.
603
+ */
604
+ async waitForNavigation(timeout = 30000) {
605
+ try {
606
+ await /** @type {Promise<void>} */ (new Promise((resolve, reject) => {
607
+ const timer = setTimeout(() => { unsub(); reject(new Error('waitForNavigation timed out')); }, timeout);
608
+ const unsub = bidi.on('browsingContext.load', (e) => {
609
+ if (e.context !== topContext) return;
610
+ clearTimeout(timer); unsub(); resolve();
611
+ });
612
+ }));
613
+ refContexts = new Map(); // the DOM changed — old refs are stale
614
+ } catch {
615
+ // No load event (SPA pushState/replaceState) — settle for DOM updates.
616
+ await new Promise((r) => setTimeout(r, 500));
617
+ }
485
618
  },
486
619
 
487
620
  async close() {
@@ -499,9 +632,9 @@ export async function createFirefoxPage(bidi, opts = {}) {
499
632
  await new Promise((r) => setTimeout(r, 500));
500
633
  }
501
634
 
502
- // Wire the dialog handler before returning — must precede any navigation so
503
- // an 'ignore' prompt (see bidi.js capability) is never left hanging.
504
- await setupDialogs();
635
+ // Wire event subscriptions before returning — dialogs must precede any
636
+ // navigation so an 'ignore' prompt (see bidi.js capability) is never hung.
637
+ await setupSubscriptions();
505
638
 
506
639
  return page;
507
640
  }
package/src/firefox.js CHANGED
@@ -97,6 +97,7 @@ export function findFirefox() {
97
97
  * @param {string} [opts.proxy] - Proxy 'host:port' or 'scheme://host:port'
98
98
  * (http/https → HTTP+SSL proxy; socks/socks5/socks4 → SOCKS), via prefs
99
99
  * @param {{width:number,height:number}} [opts.viewport] - Initial window size
100
+ * @param {string} [opts.downloadDir] - Route downloads here (throwaway dir), no prompt
100
101
  * @returns {Promise<{wsUrl: string, process: import('node:child_process').ChildProcess, port: number, ownedProfileDir: string}>}
101
102
  */
102
103
  export async function launchFirefox(opts = {}) {
@@ -114,6 +115,20 @@ export async function launchFirefox(opts = {}) {
114
115
  'user_pref("geo.prompt.testing", true);',
115
116
  'user_pref("geo.prompt.testing.allow", true);',
116
117
  ];
118
+ if (opts.downloadDir) {
119
+ // Route downloads to a caller-owned throwaway dir (parity with CDP's
120
+ // Browser.setDownloadBehavior) and never prompt — folderList:2 = custom dir.
121
+ // JSON.stringify emits a fully-escaped JS string literal (quotes, backslashes
122
+ // AND newlines/control chars), so a path can't inject extra user_pref lines.
123
+ const dir = JSON.stringify(String(opts.downloadDir));
124
+ prefs.push(
125
+ 'user_pref("browser.download.folderList", 2);',
126
+ `user_pref("browser.download.dir", ${dir});`,
127
+ 'user_pref("browser.download.useDownloadDir", true);',
128
+ 'user_pref("browser.download.manager.showWhenStarting", false);',
129
+ 'user_pref("browser.download.always_ask_before_handling_new_types", false);',
130
+ );
131
+ }
117
132
  if (opts.proxy) {
118
133
  // Honor the scheme: a socks:// proxy must be wired as SOCKS, not HTTP —
119
134
  // otherwise SOCKS traffic is silently sent to an HTTP proxy and fails.
package/src/index.js CHANGED
@@ -23,6 +23,8 @@ import { applyFirefoxStealth } from './stealth-firefox.js';
23
23
  import { applyFirefoxBlocklist } from './blocklist-firefox.js';
24
24
  import { resolveBlocklistPatterns } from './blocklist.js';
25
25
  import { waitForNetworkIdle } from './network-idle.js';
26
+ import { decideDialog, dialogLogEntry } from './dialog.js';
27
+ import { isChallengePage } from './challenge.js';
26
28
  import { readable as extractReadable } from './readable.js';
27
29
  import { assertNavigable, assertUploadAllowed } from './url-guard.js';
28
30
  import { join as pathJoin } from 'node:path';
@@ -352,29 +354,11 @@ export async function connect(opts = {}) {
352
354
  let onDialogHandler = null;
353
355
  function setupDialogHandler(session) {
354
356
  session.on('Page.javascriptDialogOpening', async (params) => {
355
- dialogLog.push({
356
- type: params.type,
357
- message: params.message,
358
- timestamp: new Date().toISOString(),
359
- });
360
- let accept = params.type !== 'beforeunload';
361
- let promptText = params.defaultPrompt || '';
362
- if (onDialogHandler) {
363
- try {
364
- const decision = await onDialogHandler({
365
- type: params.type,
366
- message: params.message,
367
- defaultPrompt: params.defaultPrompt || '',
368
- });
369
- if (decision && typeof decision === 'object') {
370
- if (typeof decision.accept === 'boolean') accept = decision.accept;
371
- if (typeof decision.promptText === 'string') promptText = decision.promptText;
372
- }
373
- } catch {
374
- // Handler threw — fall back to defaults so the page doesn't hang
375
- // waiting for a never-arriving handleJavaScriptDialog reply.
376
- }
377
- }
357
+ dialogLog.push(dialogLogEntry(params.type, params.message));
358
+ const { accept, promptText } = await decideDialog(
359
+ { type: params.type, message: params.message, defaultPrompt: params.defaultPrompt },
360
+ onDialogHandler,
361
+ );
378
362
  await session.send('Page.handleJavaScriptDialog', { accept, promptText });
379
363
  });
380
364
  }
@@ -748,56 +732,89 @@ async function suppressPermissions(cdp) {
748
732
  * ref model, and AX-tree source all differ (see firefox-page.js). close() is
749
733
  * wrapped to also reap the Firefox process + temp profile.
750
734
  *
751
- * Chromium-only options NOT applied on this path: `hybrid` mode. `saveState`,
752
- * `waitForNavigation`, and download capture are CDP-only too (the page object
753
- * stubs them — see firefox-page.js). `blockAds`/`blockUrls` and JS dialog
754
- * handling (`dialogLog`/`onDialog`) DO apply as of Phase 3 (ad-block via a
755
- * catch-all `network.addIntercept` + in-process glob match; dialogs via
756
- * `browsingContext.userPromptOpened` `handleUserPrompt`).
757
- * `waitForNetworkIdle` and daemon console/network capture DO apply as of Phase
758
- * 2 (over BiDi `network.*` and `log.entryAdded` events). `consent`
759
- * (auto-dismiss) and
760
- * stealth patches DO apply here as of v0.16.0 (stealth headless-only, via BiDi
761
- * preload script see stealth-firefox.js / consent-firefox.js), alongside the
762
- * navigation guard
763
- * (`allowLocalUrls`/`blockPrivateNetwork`), `uploadDir` sandbox, `incognito`,
764
- * `proxy`, `viewport`, and `pruneMode`.
765
- * @param {object} opts - connect() options ({ mode, proxy, binary, viewport, pruneMode, urlGuard, uploadDir, incognito })
735
+ * As of Phase 4, the Firefox path reaches practical parity with the CDP path:
736
+ * `hybrid` mode (relaunch headed on a bot-challenge page), `saveState`
737
+ * (`storage.getCookies` + localStorage), `waitForNavigation`
738
+ * (`browsingContext.load`), and download capture (`page.downloads` via
739
+ * `browsingContext.downloadWillBegin`/`downloadEnd`) all work here now.
740
+ * `blockAds`/`blockUrls` + JS dialog handling arrived in Phase 3; console/
741
+ * network capture + `waitForNetworkIdle` in Phase 2; stealth + consent in
742
+ * v0.16.0. Still Firefox-limited: `reload({ignoreCache})` (upstream BiDi gap).
743
+ * The navigation guard (`allowLocalUrls`/`blockPrivateNetwork`), `uploadDir`
744
+ * sandbox, `incognito`, `proxy`, `viewport`, and `pruneMode` all apply.
745
+ * @param {object} opts - connect() options ({ mode, proxy, binary, viewport, pruneMode, urlGuard, uploadDir, incognito, downloadPath, blockAds, blockUrls })
766
746
  * @returns {Promise<object>} Firefox page object
767
747
  */
768
748
  async function connectFirefox(opts) {
769
- const browser = await launchFirefox({
770
- headed: opts.mode === 'headed',
771
- proxy: opts.proxy,
772
- binary: opts.binary,
773
- viewport: opts.viewport,
774
- });
775
- const bidi = await createBiDi(browser.wsUrl);
776
- // Stealth patches (headless only, mirroring the CDP path's `mode !== 'headed'`
777
- // gate). Registered before the first navigation via script.addPreloadScript.
778
- // Firefox needs a much smaller script than Chromium — see stealth-firefox.js.
779
- if (opts.mode !== 'headed') {
780
- await applyFirefoxStealth(bidi);
781
- }
782
- // Ad/tracker blocking (parity with the CDP applyBlocklist). Firefox is always
783
- // a launched, throwaway profile (never attach mode), so the default is on —
784
- // unlike the CDP path, which defaults off in attach mode. Must run before the
785
- // first navigation so the catch-all intercept is live when the page loads.
786
- await applyFirefoxBlocklist(bidi, {
749
+ const hybrid = opts.mode === 'hybrid';
750
+ const blockOpts = {
787
751
  blockAds: opts.blockAds !== undefined ? opts.blockAds : true,
788
752
  blockUrls: opts.blockUrls,
789
- });
753
+ };
754
+
755
+ // Downloads land in a throwaway dir (parity with CDP's setDownloadBehavior).
756
+ // A caller-supplied opts.downloadPath is honored and left alone; otherwise we
757
+ // own + reap a temp dir on close.
758
+ let ownedDownloadDir = null;
759
+ let downloadDir = opts.downloadPath || null;
760
+ if (!downloadDir) {
761
+ const { mkdtempSync } = await import('node:fs');
762
+ const { tmpdir } = await import('node:os');
763
+ ownedDownloadDir = mkdtempSync(pathJoin(tmpdir(), 'barebrowse-ff-dl-'));
764
+ downloadDir = ownedDownloadDir;
765
+ }
766
+
767
+ // Launch Firefox, open a BiDi session, and apply stealth (headless-only,
768
+ // mirroring the CDP `mode !== 'headed'` gate) + ad/tracker blocking. Shared
769
+ // by the initial launch and the hybrid headed relaunch so both are wired
770
+ // identically. Blocking defaults on — Firefox is always a throwaway profile,
771
+ // never attach mode — and must be live before the first navigation.
772
+ async function launchAndSetup(headed) {
773
+ const browser = await launchFirefox({
774
+ headed, proxy: opts.proxy, binary: opts.binary, viewport: opts.viewport, downloadDir,
775
+ });
776
+ const bidi = await createBiDi(browser.wsUrl);
777
+ if (!headed) await applyFirefoxStealth(bidi);
778
+ await applyFirefoxBlocklist(bidi, blockOpts);
779
+ const { contexts } = await bidi.send('browsingContext.getTree', {});
780
+ return { browser, bidi, topContext: contexts[0].context };
781
+ }
782
+
783
+ let { browser, bidi } = await launchAndSetup(opts.mode === 'headed');
784
+
785
+ // Hybrid relaunch: launch a headed Firefox FIRST (so a failure leaves the
786
+ // current session intact), then tear down the old headless one and hand the
787
+ // fresh bidi/topContext back to the page, which rebinds to it.
788
+ async function relaunchHeaded() {
789
+ const next = await launchAndSetup(true);
790
+ const old = browser;
791
+ browser = next.browser;
792
+ try { bidi.close(); } catch {}
793
+ try { await cleanupFirefox(old); } catch {}
794
+ bidi = next.bidi;
795
+ return { bidi: next.bidi, topContext: next.topContext };
796
+ }
797
+
790
798
  const page = await createFirefoxPage(bidi, {
791
799
  pruneMode: opts.pruneMode,
792
800
  urlGuard: { allowLocalUrls: opts.allowLocalUrls, blockPrivateNetwork: opts.blockPrivateNetwork },
793
801
  uploadDir: opts.uploadDir || null,
794
802
  incognito: !!opts.incognito,
795
803
  consent: opts.consent,
804
+ hybrid,
805
+ headed: opts.mode === 'headed',
806
+ relaunchHeaded: hybrid ? relaunchHeaded : null,
796
807
  });
797
808
  const closePage = page.close.bind(page);
798
809
  page.close = async () => {
799
810
  await closePage();
800
- await cleanupFirefox(browser);
811
+ await cleanupFirefox(browser); // `browser` tracks the relaunched one if hybrid fired
812
+ if (ownedDownloadDir) {
813
+ try {
814
+ const { rmSync } = await import('node:fs');
815
+ rmSync(ownedDownloadDir, { recursive: true, force: true });
816
+ } catch { /* best-effort reap */ }
817
+ }
801
818
  };
802
819
  return page;
803
820
  }
@@ -1105,66 +1122,8 @@ function extractProps(props) {
1105
1122
  return result;
1106
1123
  }
1107
1124
 
1108
- /**
1109
- * Detect if a page is a bot-challenge page (Cloudflare, hCaptcha, etc.).
1110
- *
1111
- * Pre-H9 this was over-aggressive: `nodeCount < 50` alone fired on any
1112
- * legitimate small page (404s, simple landings, error pages), and generic
1113
- * phrases like "access denied" / "unknown error" / "permission denied"
1114
- * triggered on real HTTP 4xx/5xx pages, kicking hybrid mode into a costly
1115
- * headed fallback for nothing.
1116
- *
1117
- * H9 split: STRONG_PHRASES are essentially-unambiguous challenge UI and
1118
- * fire regardless of page size; WEAK_PHRASES only fire when the page is
1119
- * ALSO tiny (so a legitimate-looking error page with "access denied" in
1120
- * its body doesn't trip the fallback).
1121
- *
1122
- * @param {object} tree - Nested ARIA tree (from buildTree)
1123
- * @param {number} [nodeCount] - Raw CDP node count (from Accessibility.getFullAXTree)
1124
- */
1125
- export function isChallengePage(tree, nodeCount) {
1126
- if (!tree) return true; // truly empty AX tree — something went wrong fetching the page
1127
-
1128
- const text = flattenTreeText(tree);
1129
- const lower = text.toLowerCase();
1130
-
1131
- // Strong phrases — distinctive enough to identify the challenge product
1132
- // by name. Fire on their own regardless of node count.
1133
- const STRONG_PHRASES = [
1134
- 'just a moment', // Cloudflare interstitial
1135
- 'checking if the site connection is secure', // Cloudflare
1136
- 'checking your browser', // Various JS challenges
1137
- 'verify you are human', // hCaptcha / reCAPTCHA
1138
- 'prove your humanity',
1139
- 'attention required', // Cloudflare block page
1140
- 'enable javascript and cookies to continue', // Cloudflare
1141
- 'please complete the security check', // Cloudflare/Akamai
1142
- ];
1143
- if (STRONG_PHRASES.some((p) => lower.includes(p))) return true;
1144
-
1145
- // Weak phrases — show up on real challenge pages but ALSO on legitimate
1146
- // small error pages. Only count when the page is itself tiny (low node
1147
- // count or near-empty text), which is the corroborating signal that
1148
- // separates a real error UI from a challenge skeleton.
1149
- const WEAK_PHRASES = [
1150
- 'please wait',
1151
- 'request blocked',
1152
- 'access denied',
1153
- 'permission denied',
1154
- 'unknown error',
1155
- 'file a ticket',
1156
- ];
1157
- const tinyPage = (nodeCount !== undefined && nodeCount < 30) || text.trim().length < 50;
1158
- if (tinyPage && WEAK_PHRASES.some((p) => lower.includes(p))) return true;
1159
-
1160
- return false;
1161
- }
1162
-
1163
- function flattenTreeText(node) {
1164
- if (!node) return '';
1165
- let text = node.name || '';
1166
- for (const child of node.children || []) {
1167
- text += ' ' + flattenTreeText(child);
1168
- }
1169
- return text;
1170
- }
1125
+ // isChallengePage (the hybrid-fallback gate) now lives in challenge.js so both
1126
+ // the CDP path here and the BiDi path (firefox-page.js) share it without a
1127
+ // circular import. Re-exported (from the local import above) so existing
1128
+ // importers of `src/index.js` and the public API keep working unchanged.
1129
+ export { isChallengePage };
@@ -0,0 +1,25 @@
1
+ /**
2
+ * challenge.js — Bot-challenge / interstitial detection for the hybrid fallback.
3
+ *
4
+ * Extracted from index.js so both engines can share it without a circular
5
+ * import (index.js drives CDP; firefox-page.js drives BiDi, and both need to
6
+ * decide "is this a Cloudflare/hCaptcha wall we should retry headed?"). Pure —
7
+ * operates on the nested ARIA tree that buildTree() / ariaTree() produce, which
8
+ * is identical in shape across engines.
9
+ */
10
+ /**
11
+ * Heuristic: does this ARIA tree look like a bot-challenge / block interstitial
12
+ * (Cloudflare "Just a moment", hCaptcha, Akamai) rather than real content?
13
+ *
14
+ * H9 split: STRONG_PHRASES are essentially-unambiguous challenge UI and fire
15
+ * regardless of page size; WEAK_PHRASES only fire when the page is ALSO tiny
16
+ * (so a legitimate-looking error page with "access denied" in its body doesn't
17
+ * trip the fallback).
18
+ *
19
+ * @param {object} tree - Nested ARIA tree (from buildTree / ariaTree)
20
+ * @param {number} [nodeCount] - Node count (CDP: getFullAXTree; BiDi: tree walk)
21
+ * @returns {boolean}
22
+ */
23
+ export function isChallengePage(tree: object, nodeCount?: number): boolean;
24
+ /** Count every node in a nested ARIA tree (shared node-count for the tinyPage test). */
25
+ export function countNodes(node: any): number;
package/types/daemon.d.ts CHANGED
@@ -10,6 +10,15 @@
10
10
  * @returns {string[]}
11
11
  */
12
12
  export function buildDaemonArgs(opts: object, absDir: string, initialUrl: string | undefined, cliPath: string): string[];
13
+ /**
14
+ * Convert a BiDi remote value ({type,value} — see script.evaluate) into the
15
+ * plain JS value CDP's `Runtime.evaluate({returnByValue:true})` yields, so an
16
+ * `eval` returns the same shape on both engines. BiDi serializes objects/arrays
17
+ * as entry lists ([[k,v],…] / [v,…]) rather than plain values; walk them back.
18
+ * @param {object} r - BiDi remote value
19
+ * @returns {*}
20
+ */
21
+ export function deserializeBiDi(r: object): any;
13
22
  /**
14
23
  * Wire Firefox/BiDi console + network capture onto the daemon's log arrays.
15
24
  * The BiDi analogue of the CDP `Runtime.consoleAPICalled` / `Network.*`
@@ -0,0 +1,44 @@
1
+ /**
2
+ * dialog.js — Shared JS-dialog decision core for both engines.
3
+ *
4
+ * A JS dialog (alert/confirm/prompt/beforeunload) surfaces differently per
5
+ * protocol — CDP's `Page.javascriptDialogOpening` vs BiDi's
6
+ * `browsingContext.userPromptOpened` — but the *decision* (accept? what prompt
7
+ * text?) and the log entry are identical. This module single-sources both so
8
+ * the CDP path (index.js) and the BiDi path (firefox-page.js) can't drift
9
+ * (code-review Phase-3 finding #5).
10
+ */
11
+ /**
12
+ * Build a dialogLog entry from a normalized dialog descriptor.
13
+ * @param {string} type - 'alert' | 'confirm' | 'prompt' | 'beforeunload'
14
+ * @param {string} [message]
15
+ * @returns {{type: string, message: string, timestamp: string}}
16
+ */
17
+ export function dialogLogEntry(type: string, message?: string): {
18
+ type: string;
19
+ message: string;
20
+ timestamp: string;
21
+ };
22
+ /**
23
+ * Decide how to answer a JS dialog. Default policy (both engines): accept
24
+ * everything except `beforeunload` (dismiss = stay on page); a `prompt`
25
+ * returns its default text. A caller-installed handler may override via
26
+ * `{ accept, promptText }`; if the handler throws we keep the defaults so the
27
+ * page never hangs waiting for a reply that never arrives.
28
+ *
29
+ * @param {{type: string, message?: string, defaultPrompt?: string}} dialog
30
+ * @param {?(function({type,message,defaultPrompt}): (object|Promise<object>))} handler
31
+ * @returns {Promise<{accept: boolean, promptText: string}>}
32
+ */
33
+ export function decideDialog({ type, message, defaultPrompt }: {
34
+ type: string;
35
+ message?: string;
36
+ defaultPrompt?: string;
37
+ }, handler: ((arg0: {
38
+ type: any;
39
+ message: any;
40
+ defaultPrompt: any;
41
+ }) => (object | Promise<object>)) | null): Promise<{
42
+ accept: boolean;
43
+ promptText: string;
44
+ }>;
@@ -7,6 +7,9 @@
7
7
  * @param {string} [opts.uploadDir] - When set, upload() rejects files outside this dir.
8
8
  * @param {boolean} [opts.incognito=false] - Clean session: injectCookies() is a no-op.
9
9
  * @param {boolean} [opts.consent=true] - Auto-dismiss cookie consent dialogs after goto().
10
+ * @param {boolean} [opts.hybrid=false] - Relaunch headed on a bot-challenge page and retry.
11
+ * @param {boolean} [opts.headed=false] - Whether the browser was launched headed (skips hybrid fallback).
12
+ * @param {?function(): Promise<{bidi: object, topContext: string}>} [opts.relaunchHeaded] - Hybrid relaunch hook (from connectFirefox).
10
13
  * @returns {Promise<object>} page object
11
14
  */
12
15
  export function createFirefoxPage(bidi: object, opts?: {
@@ -18,4 +21,10 @@ export function createFirefoxPage(bidi: object, opts?: {
18
21
  uploadDir?: string | undefined;
19
22
  incognito?: boolean | undefined;
20
23
  consent?: boolean | undefined;
24
+ hybrid?: boolean | undefined;
25
+ headed?: boolean | undefined;
26
+ relaunchHeaded?: (() => Promise<{
27
+ bidi: object;
28
+ topContext: string;
29
+ }>) | null | undefined;
21
30
  }): Promise<object>;
@@ -14,6 +14,7 @@ export function findFirefox(): string;
14
14
  * @param {string} [opts.proxy] - Proxy 'host:port' or 'scheme://host:port'
15
15
  * (http/https → HTTP+SSL proxy; socks/socks5/socks4 → SOCKS), via prefs
16
16
  * @param {{width:number,height:number}} [opts.viewport] - Initial window size
17
+ * @param {string} [opts.downloadDir] - Route downloads here (throwaway dir), no prompt
17
18
  * @returns {Promise<{wsUrl: string, process: import('node:child_process').ChildProcess, port: number, ownedProfileDir: string}>}
18
19
  */
19
20
  export function launchFirefox(opts?: {
@@ -25,6 +26,7 @@ export function launchFirefox(opts?: {
25
26
  width: number;
26
27
  height: number;
27
28
  } | undefined;
29
+ downloadDir?: string | undefined;
28
30
  }): Promise<{
29
31
  wsUrl: string;
30
32
  process: import("node:child_process").ChildProcess;
package/types/index.d.ts CHANGED
@@ -129,21 +129,5 @@ export function connect(opts?: {
129
129
  export function applyBlocklist(session: any, pageOpts: any): Promise<void>;
130
130
  /** Test-only: reset the warn-once flag. Not part of the public API. */
131
131
  export function _resetBlocklistWarning(): void;
132
- /**
133
- * Detect if a page is a bot-challenge page (Cloudflare, hCaptcha, etc.).
134
- *
135
- * Pre-H9 this was over-aggressive: `nodeCount < 50` alone fired on any
136
- * legitimate small page (404s, simple landings, error pages), and generic
137
- * phrases like "access denied" / "unknown error" / "permission denied"
138
- * triggered on real HTTP 4xx/5xx pages, kicking hybrid mode into a costly
139
- * headed fallback for nothing.
140
- *
141
- * H9 split: STRONG_PHRASES are essentially-unambiguous challenge UI and
142
- * fire regardless of page size; WEAK_PHRASES only fire when the page is
143
- * ALSO tiny (so a legitimate-looking error page with "access denied" in
144
- * its body doesn't trip the fallback).
145
- *
146
- * @param {object} tree - Nested ARIA tree (from buildTree)
147
- * @param {number} [nodeCount] - Raw CDP node count (from Accessibility.getFullAXTree)
148
- */
149
- export function isChallengePage(tree: object, nodeCount?: number): boolean;
132
+ export { isChallengePage };
133
+ import { isChallengePage } from './challenge.js';