barebrowse 0.15.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +140 -1
- package/README.md +20 -1
- package/barebrowse.context.md +3 -3
- package/package.json +1 -1
- package/src/bidi.js +8 -2
- package/src/blocklist-firefox.js +60 -0
- package/src/blocklist.js +49 -0
- package/src/consent-firefox.js +125 -0
- package/src/consent-patterns.js +154 -0
- package/src/consent.js +5 -144
- package/src/daemon.js +74 -4
- package/src/firefox-page.js +136 -45
- package/src/index.js +33 -11
- package/src/network-idle.js +66 -28
- package/src/stealth-firefox.js +42 -0
- package/src/stealth.js +107 -69
- package/types/blocklist-firefox.d.ts +15 -0
- package/types/blocklist.d.ts +34 -0
- package/types/consent-firefox.d.ts +9 -0
- package/types/consent-patterns.d.ts +12 -0
- package/types/daemon.d.ts +22 -0
- package/types/firefox-page.d.ts +2 -0
- package/types/network-idle.d.ts +16 -8
- package/types/stealth-firefox.d.ts +9 -0
- package/types/stealth.d.ts +31 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,138 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.18.0] - 2026-07-11
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- **Firefox/BiDi noise-reduction + dialogs (parity plan Phase 3).** Two more
|
|
8
|
+
Chromium/CDP capabilities now work the same on the Firefox engine:
|
|
9
|
+
- **Ad/tracker blocking** (`blockAds`, on by default; `blockUrls` to extend).
|
|
10
|
+
BiDi's `network.addIntercept` can't express our glob patterns — its
|
|
11
|
+
`urlPatterns` reject `*` outright and have no subdomain wildcard — so the
|
|
12
|
+
Firefox path registers a *catch-all* `beforeRequestSent` intercept and
|
|
13
|
+
matches each request URL in-process against the shared `blocklist.js`
|
|
14
|
+
(new `makeBlockMatcher` compiles the CDP globs to a predicate, single-sourced
|
|
15
|
+
across engines). Matches are failed like CDP's `ERR_BLOCKED_BY_CLIENT`; the
|
|
16
|
+
rest continue. Default is **on** (Firefox is always a launched throwaway
|
|
17
|
+
profile, never attach mode). `src/blocklist-firefox.js`.
|
|
18
|
+
- **JS dialog handling** (`alert`/`confirm`/`prompt`/`beforeunload`). The BiDi
|
|
19
|
+
session is created with `unhandledPromptBehavior:'ignore'` so Firefox no
|
|
20
|
+
longer auto-dismisses prompts before we can act; `page.dialogLog` records
|
|
21
|
+
every dialog and `page.onDialog(handler)` overrides the default (accept all
|
|
22
|
+
except `beforeunload`), mirroring the CDP surface exactly.
|
|
23
|
+
|
|
24
|
+
BiDi mechanics were POC-measured against real Firefox before wiring (the "no
|
|
25
|
+
such alert" auto-dismiss race, and that `urlPatterns` reject `*`). Guarded by
|
|
26
|
+
`test/unit/blocklist-firefox.test.js` (10) and Firefox integration tests
|
|
27
|
+
(ad-block via a hermetic CORS server so the `blockAds:false` control genuinely
|
|
28
|
+
passes through; dialog auto-accept + `dialogLog` + custom `onDialog`).
|
|
29
|
+
Remaining Firefox gaps (Phase 4+): hybrid mode, `saveState`,
|
|
30
|
+
`waitForNavigation`, downloads, `reload({ignoreCache})`.
|
|
31
|
+
|
|
32
|
+
## [0.17.0] - 2026-07-10
|
|
33
|
+
|
|
34
|
+
### Added
|
|
35
|
+
|
|
36
|
+
- **Firefox/BiDi observability parity (parity plan Phase 2).** The Firefox
|
|
37
|
+
engine now backs the same console/network capture and network-idle wait the
|
|
38
|
+
Chromium/CDP engine has, over WebDriver BiDi events instead of CDP:
|
|
39
|
+
- **`page.waitForNetworkIdle()`** works on Firefox (was a CDP-only stub that
|
|
40
|
+
threw). Wired to `network.beforeRequestSent` / `responseCompleted` /
|
|
41
|
+
`fetchError`, reusing the same orphan-resilient idle core as the CDP path
|
|
42
|
+
(`waitForNetworkIdleBiDi` in `network-idle.js`).
|
|
43
|
+
- **Daemon console + network log capture** on Firefox: the `console-logs`,
|
|
44
|
+
`network-log`, and `wait-idle` CLI commands now return real data for a
|
|
45
|
+
`--engine firefox` session (previously empty / unsupported). Backed by
|
|
46
|
+
`log.entryAdded` + `network.*` events (`attachBiDiCapture` in `daemon.js`),
|
|
47
|
+
mirroring the CDP `Runtime.consoleAPICalled` / `Network.*` capture. BiDi's
|
|
48
|
+
`warn` console level is normalized to CDP's `warning` so `--level` filters
|
|
49
|
+
match cross-engine.
|
|
50
|
+
|
|
51
|
+
Event shapes were measured against real Firefox before wiring (per the Phase 1
|
|
52
|
+
lesson). Unit-tested against those shapes in `test/unit/network-idle.test.js`
|
|
53
|
+
and `test/unit/daemon.test.js`; live-verified in `test/integration/firefox.test.js`.
|
|
54
|
+
Still Firefox gaps (Phase 3+): ad/tracker block, dialog capture, `saveState`,
|
|
55
|
+
`waitForNavigation`, hybrid mode.
|
|
56
|
+
|
|
57
|
+
## [0.16.1] - 2026-07-10
|
|
58
|
+
|
|
59
|
+
### Fixed
|
|
60
|
+
|
|
61
|
+
- **Firefox consent auto-dismiss no longer risks clicking an unrelated button.**
|
|
62
|
+
A code review of v0.16.0 found (and reproduced) a false-positive: when a
|
|
63
|
+
consent *dialog* was detected but contained no in-dialog accept button, the
|
|
64
|
+
Firefox walker fell back to a page-wide "accept" scan that could auto-click an
|
|
65
|
+
unrelated `Accept all …` control elsewhere on the page (e.g. a ToS/signup
|
|
66
|
+
button) during `goto()`. The page-wide scan now runs **only** for banner-style
|
|
67
|
+
consent (no dialog container at all). Trade-off: an accept button rendered
|
|
68
|
+
*outside* its own dialog (some SourcePoint deployments) is no longer
|
|
69
|
+
auto-dismissed on Firefox — we accept that miss rather than risk a wrong
|
|
70
|
+
click. Regression-tested in `test/unit/consent-firefox.test.js`.
|
|
71
|
+
|
|
72
|
+
### Documentation
|
|
73
|
+
|
|
74
|
+
- **Firefox stealth + consent known limitations** captured in the PRD
|
|
75
|
+
(`docs/01-product/prd.md`), slated for the Phase 5 cross-engine fidelity
|
|
76
|
+
harness: the consent double AX-build per navigate (bounded latency, parity
|
|
77
|
+
with CDP), the single-click-without-re-verify behavior (`consent: false` to
|
|
78
|
+
opt out), and the latent `navigator.webdriver` own-property fallback. All
|
|
79
|
+
three were validated in the same review; none is a correctness defect on
|
|
80
|
+
current engines.
|
|
81
|
+
|
|
82
|
+
## [0.16.0] - 2026-07-10
|
|
83
|
+
|
|
84
|
+
### Added
|
|
85
|
+
|
|
86
|
+
- **Firefox anti-detection + consent auto-dismiss (BiDi parity Phase 1).** The
|
|
87
|
+
two most agent-breaking Firefox gaps are closed: headless bot-detection and
|
|
88
|
+
cookie-consent dialogs that were never dismissed. Both now work on
|
|
89
|
+
`connect({ engine: 'firefox' })` (and therefore the CLI/daemon/MCP Firefox
|
|
90
|
+
paths), behind the same flags as the CDP engine.
|
|
91
|
+
- **Stealth** (`src/stealth-firefox.js`, headless only, via BiDi
|
|
92
|
+
`script.addPreloadScript`). Deliberately *not* a port of Chromium's
|
|
93
|
+
`STEALTH_SCRIPT`: a POC measured stock Firefox-under-BiDi and found it
|
|
94
|
+
already exposes a realistic surface (no `window.chrome`, a real plugin set,
|
|
95
|
+
the real GPU, a UA with no "Headless" marker) — injecting Chromium's
|
|
96
|
+
Chrome-shims would have *created* a spoof tell worse than the one removed.
|
|
97
|
+
Firefox's only real tell is `navigator.webdriver` (`true`), so it gets a
|
|
98
|
+
minimal script: `navigator.webdriver` hiding + canvas-fingerprint noise.
|
|
99
|
+
`WEBDRIVER_PATCH` and `CANVAS_NOISE_PATCH` are now exported from
|
|
100
|
+
`stealth.js` and shared by both engines (single-sourcing the canvas
|
|
101
|
+
double-XOR fix).
|
|
102
|
+
- **Hardened `navigator.webdriver` hiding on both engines.** The shared
|
|
103
|
+
`WEBDRIVER_PATCH` now *deletes* the getter off `Navigator.prototype` instead
|
|
104
|
+
of shadowing it with an own property. A naive override hid the value but left
|
|
105
|
+
`navigator.hasOwnProperty('webdriver') === true` — a tell that advanced
|
|
106
|
+
anti-bot checks (e.g. sannysoft's "WebDriver New") detect. After the change
|
|
107
|
+
`navigator.webdriver` is `undefined` and both `hasOwnProperty` and
|
|
108
|
+
`'webdriver' in navigator` read `false`, matching a stock browser
|
|
109
|
+
(POC-verified on Chromium and Firefox). This improves the Chromium engine
|
|
110
|
+
too, not just Firefox.
|
|
111
|
+
- **Consent** (`src/consent-firefox.js`). A walker over the reconstructed
|
|
112
|
+
nested AX tree that clicks the "accept" button via the existing BiDi
|
|
113
|
+
`pointerClick`. The multilingual accept/dialog patterns were extracted to
|
|
114
|
+
`src/consent-patterns.js`, now imported by both the CDP (`consent.js`) and
|
|
115
|
+
BiDi walkers, so language coverage stays single-sourced. Runs after
|
|
116
|
+
`goto()` behind the same `consent !== false` flag as CDP.
|
|
117
|
+
- **Tests:** `test/unit/consent-firefox.test.js` (pure walker: dialog
|
|
118
|
+
scoping, pattern priority, non-English, banner fallback, false-positive
|
|
119
|
+
guard) plus Firefox integration tests (webdriver hidden at page-parse time,
|
|
120
|
+
no `window.chrome` spoof, consent auto-dismiss + a `consent: false` control
|
|
121
|
+
that proves the test can fail).
|
|
122
|
+
|
|
123
|
+
### Known gaps (Firefox engine)
|
|
124
|
+
|
|
125
|
+
- `hybrid` mode, ad/tracker blocking, and console/network log capture remain
|
|
126
|
+
Chromium-only (BiDi parity Phases 2–4). `saveState`, `waitForNavigation`,
|
|
127
|
+
and `waitForNetworkIdle` still throw a clear "not supported on the
|
|
128
|
+
Firefox/BiDi engine" error; download/dialog logs read empty.
|
|
129
|
+
- `reload()` still cannot honour `ignoreCache` (Firefox BiDi does not support
|
|
130
|
+
it yet).
|
|
131
|
+
- Consent auto-dismiss clicks once without re-verifying dismissal (the CDP
|
|
132
|
+
path's synthetic→real retry is unnecessary here — BiDi `pointerClick` is
|
|
133
|
+
already a real pointer event). Accessible-name computation remains a
|
|
134
|
+
high-value subset of the full W3C accname spec.
|
|
135
|
+
|
|
3
136
|
## [0.15.0] - 2026-07-10
|
|
4
137
|
|
|
5
138
|
### Added
|
|
@@ -23,7 +156,13 @@
|
|
|
23
156
|
parity with CDP.
|
|
24
157
|
- Fidelity validated against real CDP snapshots; iframes (incl. nested +
|
|
25
158
|
multi-tab), shadow DOM, CSP, and SPA timing covered in
|
|
26
|
-
`test/integration/firefox.test.js` (
|
|
159
|
+
`test/integration/firefox.test.js` (18 tests).
|
|
160
|
+
- `goto`/`reload`/history navigation honour a timeout (a page that never
|
|
161
|
+
finishes loading rejects instead of hanging). Proxy honours the URL scheme
|
|
162
|
+
(`http`/`https` → HTTP+SSL, `socks`/`socks5`/`socks4` → SOCKS). Cookie
|
|
163
|
+
injection is host-scoped like CDP (shared `scopedCookiesForUrl`, not the
|
|
164
|
+
whole jar). Snapshot fidelity: a collapsed `<select>` surfaces as its value
|
|
165
|
+
(not every `<option>`), and bare text directly under `<body>` is kept.
|
|
27
166
|
|
|
28
167
|
- **Incognito mode — a clean, unauthenticated session (`incognito: true`).**
|
|
29
168
|
Skips ALL auth injection: no cookie extraction/injection and no
|
package/README.md
CHANGED
|
@@ -145,10 +145,29 @@ const page = await connect({ engine: 'firefox' }); // headless by default
|
|
|
145
145
|
|
|
146
146
|
From the CLI: `barebrowse open <url> --engine firefox`. From MCP: set
|
|
147
147
|
`BAREBROWSE_ENGINE=firefox`. Firefox cookies (plaintext) reuse into the same
|
|
148
|
-
engine.
|
|
148
|
+
engine. Consent auto-dismiss and headless stealth (v0.16.0), the daemon's
|
|
149
|
+
console/network capture and `waitForNetworkIdle` (v0.17.0), and ad/tracker
|
|
150
|
+
blocking plus JS dialog handling (`dialogLog`/`onDialog`, v0.18.0) all work on
|
|
151
|
+
Firefox too. Chromium (CDP) remains the default; only `hybrid` mode is still
|
|
152
|
+
Chromium-only.
|
|
149
153
|
|
|
150
154
|
No clone profile, no fresh cookies — the agent sees what you see.
|
|
151
155
|
|
|
156
|
+
### Incognito (clean session)
|
|
157
|
+
|
|
158
|
+
Pass `incognito: true` for a clean, **unauthenticated** session — barebrowse
|
|
159
|
+
skips *all* auth injection (cookies + storage state), so the agent browses
|
|
160
|
+
logged out. It is not Chrome's `--incognito` flag: the session already runs in
|
|
161
|
+
a throwaway profile, so this gates the *other* auth source — your real browser
|
|
162
|
+
cookies. Works on both engines:
|
|
163
|
+
|
|
164
|
+
```js
|
|
165
|
+
const page = await connect({ incognito: true });
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
From the CLI: `barebrowse open <url> --incognito`. From MCP: set
|
|
169
|
+
`BAREBROWSE_INCOGNITO=1`.
|
|
170
|
+
|
|
152
171
|
## What it handles automatically
|
|
153
172
|
|
|
154
173
|
Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (ARIA node count heuristic + stealth patches + automatic headed fallback — snapshot shows `[BOT CHALLENGE DETECTED]` warning when blocked), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
|
package/barebrowse.context.md
CHANGED
|
@@ -95,7 +95,7 @@ const snapshot = await browse('https://example.com', {
|
|
|
95
95
|
| `close()` | -- | void | Close page, disconnect CDP, kill browser (if headless) |
|
|
96
96
|
|
|
97
97
|
**connect() options** (in addition to mode/port/consent):
|
|
98
|
-
- `engine: 'chromium'|'firefox'` — Browser engine. Default `chromium` (CDP). `firefox` drives Firefox over WebDriver BiDi (CDP is deprecated there) — a separate transport with the same `page.*` API; the ARIA snapshot is reconstructed in-page since BiDi has no `getFullAXTree`.
|
|
98
|
+
- `engine: 'chromium'|'firefox'` — Browser engine. Default `chromium` (CDP). `firefox` drives Firefox over WebDriver BiDi (CDP is deprecated there) — a separate transport with the same `page.*` API; the ARIA snapshot is reconstructed in-page since BiDi has no `getFullAXTree`. Consent auto-dismiss and stealth (headless anti-detection) work on Firefox as of v0.16.0; the CLI daemon's console/network capture and `waitForNetworkIdle` work as of v0.17.0; ad/tracker blocking (`blockAds`/`blockUrls`) and JS dialog handling (`dialogLog`/`onDialog`) work as of Phase 3. Still Chromium-only: `hybrid` mode. `reload()` ignores `ignoreCache` on Firefox (BiDi limitation).
|
|
99
99
|
- `port: 9222` — Attach to a Chromium already running with `--remote-debugging-port=N` instead of spawning one. The browser keeps running on `close()`. Stealth + permission denial + download capture are skipped to avoid mutating the user's running browser.
|
|
100
100
|
- `proxy: 'http://...'` — HTTP/SOCKS proxy for browser
|
|
101
101
|
- `viewport: '1280x720'` — Set viewport dimensions
|
|
@@ -189,8 +189,8 @@ barebrowse can inject cookies from the user's real browser sessions, bypassing l
|
|
|
189
189
|
|
|
190
190
|
| Obstacle | How | Mode |
|
|
191
191
|
|---|---|---|
|
|
192
|
-
| Cookie consent | ARIA scan
|
|
193
|
-
| Consent behind iframes |
|
|
192
|
+
| Cookie consent | ARIA scan for the accept button, 29 languages. CDP: `jsClick` (overlay-bypassing) then a real-mouse-click fallback for CMPs that ignore synthetic clicks. Firefox/BiDi: a single real pointer click. A page-wide accept scan runs only for banner-style consent (no dialog) — when a dialog is detected but has no in-dialog accept button, no page-wide scan runs, to avoid clicking an unrelated `Accept all …` control | Both engines |
|
|
193
|
+
| Consent behind iframes | Scanned/clicked in the owning frame (CDP `DOM.resolveNode`; Firefox resolves the ref in its browsing context) | Both engines |
|
|
194
194
|
| Permission prompts | Launch flags + CDP Browser.setPermission auto-deny | Both |
|
|
195
195
|
| Media autoplay blocked | `--autoplay-policy=no-user-gesture-required` | Both |
|
|
196
196
|
| Login walls | Cookie extraction from Firefox/Chromium + CDP injection | Both |
|
package/package.json
CHANGED
package/src/bidi.js
CHANGED
|
@@ -135,8 +135,14 @@ export async function createBiDi(wsUrl) {
|
|
|
135
135
|
close() { ws.close(); },
|
|
136
136
|
};
|
|
137
137
|
|
|
138
|
-
// Open the session.
|
|
139
|
-
|
|
138
|
+
// Open the session. unhandledPromptBehavior:'ignore' stops Firefox from
|
|
139
|
+
// auto-dismissing JS dialogs (alert/confirm/prompt) before we can handle them
|
|
140
|
+
// — without it, browsingContext.handleUserPrompt loses the race and fails
|
|
141
|
+
// with "no such alert" (measured). firefox-page.js wires a userPromptOpened
|
|
142
|
+
// handler before any navigation, so no dialog is ever left hanging.
|
|
143
|
+
const session = await client.send('session.new', {
|
|
144
|
+
capabilities: { alwaysMatch: { unhandledPromptBehavior: 'ignore' } },
|
|
145
|
+
});
|
|
140
146
|
client.sessionId = session.sessionId;
|
|
141
147
|
client.capabilities = session.capabilities;
|
|
142
148
|
return client;
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* blocklist-firefox.js — Ad/tracker blocking on the Firefox/BiDi engine.
|
|
3
|
+
*
|
|
4
|
+
* The BiDi analogue of index.js's applyBlocklist (CDP Network.setBlockedURLs).
|
|
5
|
+
* BiDi's network.addIntercept can't express our glob patterns — its urlPatterns
|
|
6
|
+
* reject '*' ("forbidden character *") and have no subdomain wildcard — so we
|
|
7
|
+
* register a *catch-all* intercept (empty urlPatterns) at the beforeRequestSent
|
|
8
|
+
* phase and decide per request in-process, matching each URL against the shared
|
|
9
|
+
* blocklist (makeBlockMatcher). Matches are failed (net error, like CDP's
|
|
10
|
+
* ERR_BLOCKED_BY_CLIENT); everything else is continued immediately.
|
|
11
|
+
*
|
|
12
|
+
* Cost vs. CDP: CDP matches browser-side with zero round-trips. Here every
|
|
13
|
+
* request pauses for one continue/fail round-trip to Node. On a typical page
|
|
14
|
+
* (tens–low-hundreds of requests over a local socket) that's negligible, and
|
|
15
|
+
* it's the only route that preserves glob parity without a second list.
|
|
16
|
+
*/
|
|
17
|
+
|
|
18
|
+
import { resolveBlocklistPatterns, makeBlockMatcher } from './blocklist.js';
|
|
19
|
+
|
|
20
|
+
/**
|
|
21
|
+
* Install ad/tracker blocking on a BiDi session. Idempotent per session is the
|
|
22
|
+
* caller's concern; call once at connect time before the first navigation.
|
|
23
|
+
*
|
|
24
|
+
* @param {object} bidi - BiDi client (send, subscribe, on).
|
|
25
|
+
* @param {object} pageOpts
|
|
26
|
+
* @param {boolean} [pageOpts.blockAds] - false disables the default list.
|
|
27
|
+
* @param {string[]} [pageOpts.blockUrls] - extra CDP-format globs; extend the
|
|
28
|
+
* default unless blockAds is false, in which case they're the whole list.
|
|
29
|
+
* @returns {Promise<void>}
|
|
30
|
+
*/
|
|
31
|
+
export async function applyFirefoxBlocklist(bidi, pageOpts = {}) {
|
|
32
|
+
const patterns = resolveBlocklistPatterns(pageOpts);
|
|
33
|
+
if (!patterns.length) return;
|
|
34
|
+
|
|
35
|
+
const isBlocked = makeBlockMatcher(patterns);
|
|
36
|
+
|
|
37
|
+
await bidi.send('network.addIntercept', {
|
|
38
|
+
phases: ['beforeRequestSent'],
|
|
39
|
+
urlPatterns: [],
|
|
40
|
+
});
|
|
41
|
+
await bidi.subscribe(['network.beforeRequestSent']);
|
|
42
|
+
|
|
43
|
+
bidi.on('network.beforeRequestSent', async (e) => {
|
|
44
|
+
// Only paused (intercepted) events need a decision; capture-only listeners
|
|
45
|
+
// see the same event with isBlocked=false and must be ignored here.
|
|
46
|
+
if (!e.isBlocked) return;
|
|
47
|
+
const id = e.request?.request;
|
|
48
|
+
if (!id) return;
|
|
49
|
+
try {
|
|
50
|
+
if (isBlocked(e.request.url)) {
|
|
51
|
+
await bidi.send('network.failRequest', { request: id });
|
|
52
|
+
} else {
|
|
53
|
+
await bidi.send('network.continueRequest', { request: id });
|
|
54
|
+
}
|
|
55
|
+
} catch {
|
|
56
|
+
// "no such request" — the request completed or was cancelled between the
|
|
57
|
+
// event and our reply (redundant nav requests, aborted fetches). Harmless.
|
|
58
|
+
}
|
|
59
|
+
});
|
|
60
|
+
}
|
package/src/blocklist.js
CHANGED
|
@@ -200,3 +200,52 @@ export const DEFAULT_BLOCKLIST = [
|
|
|
200
200
|
'*://*.bluekai.com/*', // Oracle Data Cloud
|
|
201
201
|
'*://*.krxd.net/*', // Salesforce / Krux
|
|
202
202
|
];
|
|
203
|
+
|
|
204
|
+
/**
|
|
205
|
+
* Resolve the effective blocklist from a page's blockAds/blockUrls options.
|
|
206
|
+
* Single-sourced across engines (CDP applyBlocklist + BiDi applyFirefoxBlocklist)
|
|
207
|
+
* so the merge/extend rule can't drift between them:
|
|
208
|
+
* - blockAds !== false → DEFAULT_BLOCKLIST plus any blockUrls (extend);
|
|
209
|
+
* - blockAds === false → only blockUrls (the default list is dropped);
|
|
210
|
+
* - neither → empty (blocking disabled).
|
|
211
|
+
*
|
|
212
|
+
* @param {object} [pageOpts]
|
|
213
|
+
* @param {boolean} [pageOpts.blockAds] - false drops the default list.
|
|
214
|
+
* @param {string[]} [pageOpts.blockUrls] - extra CDP-format globs.
|
|
215
|
+
* @returns {string[]} the patterns to block (possibly empty).
|
|
216
|
+
*/
|
|
217
|
+
export function resolveBlocklistPatterns(pageOpts = {}) {
|
|
218
|
+
return pageOpts.blockAds === false
|
|
219
|
+
? (pageOpts.blockUrls || [])
|
|
220
|
+
: [...DEFAULT_BLOCKLIST, ...(pageOpts.blockUrls || [])];
|
|
221
|
+
}
|
|
222
|
+
|
|
223
|
+
/**
|
|
224
|
+
* Compile CDP-format glob patterns into a single URL-matching predicate.
|
|
225
|
+
*
|
|
226
|
+
* CDP blocks natively via Network.setBlockedURLs, which does the glob matching
|
|
227
|
+
* browser-side. WebDriver BiDi has no glob-capable equivalent — network.
|
|
228
|
+
* addIntercept's urlPatterns reject '*' outright ("forbidden character *") and
|
|
229
|
+
* can't express subdomain wildcards like *.doubleclick.net. So the Firefox
|
|
230
|
+
* path intercepts *all* requests and matches each URL here, in-process, against
|
|
231
|
+
* the same patterns — keeping the blocklist single-sourced across engines.
|
|
232
|
+
*
|
|
233
|
+
* Matches CDP's glob semantics: '*' = any run of characters, '?' = exactly
|
|
234
|
+
* one character, whole-URL (anchored) match; every other character is literal.
|
|
235
|
+
*
|
|
236
|
+
* @param {string[]} patterns - CDP-format globs (e.g. DEFAULT_BLOCKLIST).
|
|
237
|
+
* @returns {(url: string) => boolean} true when `url` matches any pattern.
|
|
238
|
+
*/
|
|
239
|
+
export function makeBlockMatcher(patterns) {
|
|
240
|
+
const regexes = patterns.map((p) => {
|
|
241
|
+
// Escape every regex metachar EXCEPT the two glob wildcards, then expand
|
|
242
|
+
// them: '*' -> '.*', '?' -> '.'. (Escaping runs first so it never touches
|
|
243
|
+
// the '.' / '*' we insert next.)
|
|
244
|
+
const escaped = p
|
|
245
|
+
.replace(/[.+^${}()|[\]\\]/g, '\\$&')
|
|
246
|
+
.replace(/\*/g, '.*')
|
|
247
|
+
.replace(/\?/g, '.');
|
|
248
|
+
return new RegExp('^' + escaped + '$');
|
|
249
|
+
});
|
|
250
|
+
return (url) => regexes.some((re) => re.test(url));
|
|
251
|
+
}
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* consent-firefox.js — Auto-dismiss cookie consent dialogs on the Firefox/BiDi
|
|
3
|
+
* engine, by walking the reconstructed AX tree (ax-snapshot.js).
|
|
4
|
+
*
|
|
5
|
+
* The CDP walker (consent.js) can't be reused verbatim: it consumes CDP's flat
|
|
6
|
+
* getFullAXTree (nodes[] with parentId / role.value / backendDOMNodeId) and
|
|
7
|
+
* clicks via DOM.resolveNode + Input.dispatchMouseEvent. The Firefox tree is a
|
|
8
|
+
* DIFFERENT shape — nested `children`, string `role`/`name`, and each node's
|
|
9
|
+
* `nodeId` IS its ref (stamped as data-bb-ref) — and clicks go through
|
|
10
|
+
* input.performActions. So this is a parallel walker sharing only the language
|
|
11
|
+
* patterns (consent-patterns.js).
|
|
12
|
+
*
|
|
13
|
+
* It's written as a PURE function over (root tree, click(ref)) so it can be
|
|
14
|
+
* unit-tested against fixture trees with no browser: the caller (firefox-page)
|
|
15
|
+
* injects a real click that routes ref → pointerClick. The nested tree also
|
|
16
|
+
* makes "descendant of dialog" a plain subtree walk — no parentMap needed.
|
|
17
|
+
*/
|
|
18
|
+
|
|
19
|
+
import { ACCEPT_PATTERNS, DIALOG_ROLES, CONSENT_DIALOG_HINTS } from './consent-patterns.js';
|
|
20
|
+
|
|
21
|
+
/** Roles whose text confirms a container is a consent dialog. */
|
|
22
|
+
const CONSENT_TEXT_ROLES = new Set(['heading', 'StaticText', 'generic']);
|
|
23
|
+
|
|
24
|
+
/** Depth-first walk yielding every node in the subtree rooted at `node`. */
|
|
25
|
+
function* walk(node) {
|
|
26
|
+
if (!node) return;
|
|
27
|
+
yield node;
|
|
28
|
+
for (const child of node.children || []) yield* walk(child);
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
/** True if any node in this subtree carries consent-hint text. */
|
|
32
|
+
function hasConsentContent(dialog) {
|
|
33
|
+
for (const node of walk(dialog)) {
|
|
34
|
+
if (node === dialog) continue;
|
|
35
|
+
if (CONSENT_TEXT_ROLES.has(node.role) && CONSENT_DIALOG_HINTS.some((p) => p.test(node.name || ''))) {
|
|
36
|
+
return true;
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
return false;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
/**
|
|
43
|
+
* Find the best "accept" button inside a dialog subtree, honouring
|
|
44
|
+
* ACCEPT_PATTERNS priority (most specific first).
|
|
45
|
+
*/
|
|
46
|
+
function findAcceptButton(dialog) {
|
|
47
|
+
for (const pattern of ACCEPT_PATTERNS) {
|
|
48
|
+
for (const node of walk(dialog)) {
|
|
49
|
+
if (node.role === 'button' && node.name && pattern.test(node.name)) return node;
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
return null;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* Fallback when no consent dialog container is found: scan the whole tree for a
|
|
57
|
+
* button matching a STRONG (multi-word) accept pattern. Excludes the bare
|
|
58
|
+
* ^accept$/^agree$/^ok$ fallbacks (last 3) so we don't false-match unrelated
|
|
59
|
+
* page buttons — mirrors consent.js's tryGlobalConsentButton.
|
|
60
|
+
*/
|
|
61
|
+
function findGlobalAcceptButton(root) {
|
|
62
|
+
const safePatterns = ACCEPT_PATTERNS.slice(0, -3);
|
|
63
|
+
for (const pattern of safePatterns) {
|
|
64
|
+
for (const node of walk(root)) {
|
|
65
|
+
if (node.role === 'button' && node.name && pattern.test(node.name)) return node;
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
return null;
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
/**
|
|
72
|
+
* Try to dismiss a cookie consent dialog in a reconstructed Firefox AX tree.
|
|
73
|
+
*
|
|
74
|
+
* @param {object} root - Spliced AX tree root (from firefox-page's buildTree).
|
|
75
|
+
* @param {(ref: string) => Promise<void>} click - Clicks the element for a ref
|
|
76
|
+
* (firefox-page injects one that routes to pointerClick / performActions).
|
|
77
|
+
* @returns {Promise<boolean>} true if an accept button was found and clicked.
|
|
78
|
+
*/
|
|
79
|
+
export async function dismissConsentFirefox(root, click) {
|
|
80
|
+
if (!root) return false;
|
|
81
|
+
|
|
82
|
+
// Find dialog containers that look like consent dialogs.
|
|
83
|
+
const consentDialogs = [];
|
|
84
|
+
for (const node of walk(root)) {
|
|
85
|
+
if (!DIALOG_ROLES.has(node.role)) continue;
|
|
86
|
+
const name = node.name || '';
|
|
87
|
+
if (CONSENT_DIALOG_HINTS.some((p) => p.test(name)) || hasConsentContent(node)) {
|
|
88
|
+
consentDialogs.push(node);
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// Accept button inside a consent dialog (preferred — scoped, low false-positive).
|
|
93
|
+
for (const dialog of consentDialogs) {
|
|
94
|
+
const button = findAcceptButton(dialog);
|
|
95
|
+
if (button?.nodeId) {
|
|
96
|
+
try {
|
|
97
|
+
await click(button.nodeId);
|
|
98
|
+
return true;
|
|
99
|
+
} catch {
|
|
100
|
+
// Click failed (stale ref / navigated) — try the next dialog.
|
|
101
|
+
}
|
|
102
|
+
}
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
// Banner-style consent (no dialog container at all): scan the page for a
|
|
106
|
+
// strong accept button. We deliberately DO NOT run this page-wide scan when a
|
|
107
|
+
// consent dialog WAS detected but had no in-scope accept button — a page-wide
|
|
108
|
+
// match there can click an UNRELATED "Accept all …" control elsewhere (e.g. a
|
|
109
|
+
// ToS/signup button), an automatic wrong mutation on goto(). The trade-off is
|
|
110
|
+
// losing the rare "accept button rendered outside its own dialog" pattern; we
|
|
111
|
+
// accept that miss rather than risk a wrong click. See the PRD "known
|
|
112
|
+
// limitations" note for the validated false-positive this guards against.
|
|
113
|
+
if (consentDialogs.length === 0) {
|
|
114
|
+
const global = findGlobalAcceptButton(root);
|
|
115
|
+
if (global?.nodeId) {
|
|
116
|
+
try {
|
|
117
|
+
await click(global.nodeId);
|
|
118
|
+
return true;
|
|
119
|
+
} catch {
|
|
120
|
+
return false;
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
return false;
|
|
125
|
+
}
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* consent-patterns.js — Shared cookie-consent detection data.
|
|
3
|
+
*
|
|
4
|
+
* The engine-agnostic half of consent handling: the multilingual regex sets
|
|
5
|
+
* and dialog roles used to recognise a consent dialog and its "accept" button.
|
|
6
|
+
* Both the CDP walker (consent.js) and the BiDi walker (consent-firefox.js)
|
|
7
|
+
* import these so the language coverage is single-sourced — a new language or
|
|
8
|
+
* pattern is added once and both engines get it.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
// Button text patterns that mean "accept all" / "I agree" across common languages.
|
|
12
|
+
// Order matters: more specific patterns first to avoid false positives.
|
|
13
|
+
export const ACCEPT_PATTERNS = [
|
|
14
|
+
// English
|
|
15
|
+
/\baccept\s*all\b/i,
|
|
16
|
+
/\ballow\s*all\b/i,
|
|
17
|
+
/\bagree\s*to\s*all\b/i,
|
|
18
|
+
/\byes,?\s*i\s*agree\b/i,
|
|
19
|
+
/\bi\s*agree\b/i,
|
|
20
|
+
/\baccept\s*cookies?\b/i,
|
|
21
|
+
/\ballow\s*cookies?\b/i,
|
|
22
|
+
/\bgot\s*it\b/i,
|
|
23
|
+
// Dutch
|
|
24
|
+
/\balles\s*accepteren\b/i,
|
|
25
|
+
/\balles\s*toestaan\b/i,
|
|
26
|
+
/\baccepteren\b/i,
|
|
27
|
+
/\bakkoord\b/i,
|
|
28
|
+
// German
|
|
29
|
+
/\balle\s*akzeptieren\b/i,
|
|
30
|
+
/\ballem\s*zustimmen\b/i,
|
|
31
|
+
/\balle\s*cookies?\s*akzeptieren\b/i,
|
|
32
|
+
// French
|
|
33
|
+
/\btout\s*accepter\b/i,
|
|
34
|
+
/\baccepter\s*tout\b/i,
|
|
35
|
+
/\bj['']accepte\b/i,
|
|
36
|
+
// Spanish
|
|
37
|
+
/\baceptar\s*todo\b/i,
|
|
38
|
+
/\baceptar\s*todas?\b/i,
|
|
39
|
+
// Italian
|
|
40
|
+
/\baccetta\s*tutto\b/i,
|
|
41
|
+
/\baccetto\b/i,
|
|
42
|
+
// Portuguese
|
|
43
|
+
/\baceitar\s*tudo\b/i,
|
|
44
|
+
// Russian
|
|
45
|
+
/принять\s*все/i,
|
|
46
|
+
/принять/i,
|
|
47
|
+
/согласен/i,
|
|
48
|
+
// Ukrainian
|
|
49
|
+
/прийняти\s*все/i,
|
|
50
|
+
/прийняти/i,
|
|
51
|
+
// Polish
|
|
52
|
+
/zaakceptuj\s*wszystk/i,
|
|
53
|
+
/akceptuj\s*wszystk/i,
|
|
54
|
+
/zgadzam\s*się/i,
|
|
55
|
+
// Czech
|
|
56
|
+
/přijmout\s*vše/i,
|
|
57
|
+
/souhlasím/i,
|
|
58
|
+
// Turkish
|
|
59
|
+
/tümünü\s*kabul\s*et/i,
|
|
60
|
+
/kabul\s*et/i,
|
|
61
|
+
/kabul\s*ediyorum/i,
|
|
62
|
+
// Romanian
|
|
63
|
+
/acceptă\s*tot/i,
|
|
64
|
+
/accept\s*toate/i,
|
|
65
|
+
// Hungarian
|
|
66
|
+
/összes\s*elfogadás/i,
|
|
67
|
+
/elfogad/i,
|
|
68
|
+
// Greek
|
|
69
|
+
/αποδοχή\s*όλων/i,
|
|
70
|
+
/αποδέχομαι/i,
|
|
71
|
+
// Swedish
|
|
72
|
+
/acceptera\s*alla/i,
|
|
73
|
+
/godkänn\s*alla/i,
|
|
74
|
+
// Danish
|
|
75
|
+
/accepter\s*alle/i,
|
|
76
|
+
/acceptér\s*alle/i,
|
|
77
|
+
// Norwegian
|
|
78
|
+
/godta\s*alle/i,
|
|
79
|
+
/aksepter\s*alle/i,
|
|
80
|
+
// Finnish
|
|
81
|
+
/hyväksy\s*kaikki/i,
|
|
82
|
+
/hyväksyn/i,
|
|
83
|
+
// Arabic
|
|
84
|
+
/قبول\s*الكل/,
|
|
85
|
+
/قبول\s*الجميع/,
|
|
86
|
+
/موافق/,
|
|
87
|
+
/قبول/,
|
|
88
|
+
// Persian
|
|
89
|
+
/پذیرش\s*همه/,
|
|
90
|
+
/موافقم/,
|
|
91
|
+
/پذیرش/,
|
|
92
|
+
// Chinese (Simplified + Traditional)
|
|
93
|
+
/全部接受/,
|
|
94
|
+
/接受所有/,
|
|
95
|
+
/接受全部/,
|
|
96
|
+
/同意并继续/,
|
|
97
|
+
/全部接受/,
|
|
98
|
+
/接受/,
|
|
99
|
+
/同意/,
|
|
100
|
+
// Japanese
|
|
101
|
+
/すべて受け入れ/,
|
|
102
|
+
/すべて許可/,
|
|
103
|
+
/同意する/,
|
|
104
|
+
/同意します/,
|
|
105
|
+
// Korean
|
|
106
|
+
/모두\s*수락/,
|
|
107
|
+
/모두\s*동의/,
|
|
108
|
+
/동의합니다/,
|
|
109
|
+
/수락/,
|
|
110
|
+
// Vietnamese
|
|
111
|
+
/chấp\s*nhận\s*tất\s*cả/i,
|
|
112
|
+
/đồng\s*ý\s*tất\s*cả/i,
|
|
113
|
+
/đồng\s*ý/i,
|
|
114
|
+
// Thai
|
|
115
|
+
/ยอมรับทั้งหมด/,
|
|
116
|
+
/ยอมรับ/,
|
|
117
|
+
// Hindi
|
|
118
|
+
/सभी\s*स्वीकार/,
|
|
119
|
+
/स्वीकार\s*करें/,
|
|
120
|
+
/सहमत/,
|
|
121
|
+
// Indonesian / Malay
|
|
122
|
+
/terima\s*semua/i,
|
|
123
|
+
/setuju/i,
|
|
124
|
+
// Generic single-word fallbacks (only matched inside dialogs)
|
|
125
|
+
/^accept$/i,
|
|
126
|
+
/^agree$/i,
|
|
127
|
+
/^ok$/i,
|
|
128
|
+
];
|
|
129
|
+
|
|
130
|
+
// Roles that indicate a consent dialog container.
|
|
131
|
+
export const DIALOG_ROLES = new Set(['dialog', 'alertdialog']);
|
|
132
|
+
|
|
133
|
+
// Text patterns in dialog names/headings that confirm it's about consent.
|
|
134
|
+
export const CONSENT_DIALOG_HINTS = [
|
|
135
|
+
/cookie/i,
|
|
136
|
+
/consent/i,
|
|
137
|
+
/privacy/i,
|
|
138
|
+
/before\s*you\s*continue/i,
|
|
139
|
+
/voordat\s*je\s*verdergaat/i, // Dutch
|
|
140
|
+
/bevor\s*du\s*fortf/i, // German
|
|
141
|
+
/avant\s*de\s*continuer/i, // French
|
|
142
|
+
/antes\s*de\s*continuar/i, // Spanish / Portuguese
|
|
143
|
+
/prima\s*di\s*continuare/i, // Italian
|
|
144
|
+
/zanim\s*przejdziesz/i, // Polish
|
|
145
|
+
/прежде\s*чем\s*продолжить/i, // Russian
|
|
146
|
+
/devam\s*etmeden\s*önce/i, // Turkish
|
|
147
|
+
/続行する前に/, // Japanese
|
|
148
|
+
/继续前/, // Chinese Simplified
|
|
149
|
+
/繼續前/, // Chinese Traditional
|
|
150
|
+
/계속하기\s*전에/, // Korean
|
|
151
|
+
/trước\s*khi\s*tiếp\s*tục/i, // Vietnamese
|
|
152
|
+
/ملفات\s*تعريف\s*الارتباط/, // Arabic: cookies
|
|
153
|
+
/คุกกี้/, // Thai: cookies
|
|
154
|
+
];
|