@fanboynz/network-scanner 3.2.0 → 3.4.0

package/CHANGELOG.md

@@ -2,6 +2,50 @@
 
 All notable changes to the Network Scanner (nwss.js) project.
 
+## [3.4.0] - 2026-06-13
+
+### Added
+- **`redirect_first_party` site option** (default `true`) — by default a redirect's destination domains (and chain hops) are registered first-party so the landed site's own resources aren't captured as third-party. Set `false` to keep redirect targets **third-party**, so `filterRegex`/`dig` apply to them under `thirdParty: true` — e.g. capturing the end domain of an ad/cloak redirect chain (which Chrome reaches via the `ERR_TOO_MANY_REDIRECTS` curl-resolve recovery). The originally-scanned domain stays first-party either way.
+- **`ERR_TOO_MANY_REDIRECTS` is recovered, not hard-failed** — a redirect-cloaking chain (rotating throwaway domains) can exceed Chrome's ~20-hop ceiling. The scanner now recovers via two complementary paths: **(1)** it first waits briefly for the browser to *ride through* on its own — a JS/meta hop on a committed page resets Chrome's hop counter and often carries the page to the end site for free; **(2)** if the page parked on `chrome-error://` instead **and the site has `curl: true`**, it resolves the chain endpoint with `curl` (which, unlike headless Chrome, isn't served the endless-loop variant) and navigates there directly — a short hop that lands on the real end site. Either way it captures the end site's ad/tracker requests; falls back to the captured chain requests when neither path lands. The curl step is opt-in via the existing `curl` site option (so `curl: false`/unset never shells out to curl) and is also **skipped under a proxy/VPN** (curl runs direct and would leak the real IP / resolve from the wrong network); the free ride-through always applies.
+- **`click_elements` site option** — after a page loads, click a list of CSS selectors **in order** (searched across the main frame and any iframe) (e.g. `["a[href*='/movie/']", ".play"]` to click a movie link then a play button). Reaches content via organic navigation/gesture instead of a direct deep-load, which some sites JS-redirect away, and triggers click-only content like video players. Each selector is `waitForSelector`-ed (visible) up to `click_wait` before clicking, so JS-rendered targets like video players aren't missed by racing ahead of them. The request interceptor stays attached, so the post-click page's requests run through the same `filterRegex`/`dig` matching; a click that navigates is followed and later selectors query the resulting page. Honors `realistic_click` (genuine trusted gesture) and `cursor_mode: "ghost"` (Bezier travel to the element); missing elements are skipped and never fail the scan. Settle/nav wait per click via `click_wait` (default 5000ms, capped at half the per-URL timeout).
+- **`--dns` now also pins Chrome's page-navigation resolver via DoH.** Chrome ignores `--dns` for navigation and reads `/etc/resolv.conf` directly, so a broken or filtering system resolver could `ERR_NAME_NOT_RESOLVED` a domain the pre-check had already resolved. When the `--dns` servers map to a known public DoH provider — **Google, Cloudflare, Quad9, OpenDNS, AdGuard, CleanBrowsing, DNS.SB, Mullvad** (incl. malware/family/unfiltered variants) — Chrome is launched with secure-DNS `automatic` mode pointed at that provider, so page navigation resolves through the same resolver as the pre-check. `automatic` (not `secure`) keeps a system-DNS fallback if DoH is unreachable rather than failing the batch. **Applied to direct connections only** — skipped when a proxy (`--proxy-server`) or VPN is active, since the exit/tunnel does the resolution and local DoH would be redundant or resolve geo-split domains to the wrong region. Unmapped resolvers (custom/ISP, per-account providers like NextDNS, IPv6) fall back to system DNS with a warning naming the supported providers.
+- **`--doh-disable`** site/CLI option (`doh_disable` in `.nwssconfig`), default off — opt out of the Chrome-navigation DoH pinning entirely. Chrome then resolves page navigation via the system `resolv.conf` even when `--dns` maps to a known provider, while the pre-check and `dig` still honor `--dns`. For networks where DoH adds latency or is blocked, or when system-path resolution is specifically wanted.
+
+### Changed
+- **A clamped `delay` is now logged (`--debug`)** — when `delay` exceeds its ceiling (the default 2s cap, or `timeout/2` under `delay_uncapped: true`) it was silently reduced, so `delay: 48000` quietly running as 29000ms looked like the flag was ignored. A debug line now reports the clamp and which ceiling applied (raise `timeout`, or set `delay_uncapped: true`, to lift it). The per-URL budget already reserves the full configured `delay`; this only surfaces the post-load dwell clamp.
+- **DNS pre-check is paced and more tolerant under concurrency** — a concurrent scan fired up to `max_concurrent` simultaneous c-ares UDP queries at the pinned `--dns` servers; the burst (rough on WSL2's UDP-through-NAT path, and rate-limited by public resolvers) produced timeouts / `EREFUSED` that tripped the circuit breaker (`resolver errors N/M — suspending DNS pre-check`) and lost the dead-host-skip optimization. The pre-check timeout is raised 2s → 4s (a clean NXDOMAIN still returns fast, so the higher ceiling only costs time when the resolver is genuinely slow), and `createRotatingResolver` now caps in-flight queries with a counting semaphore (default 6) so the burst is paced and excess callers queue and drain quickly. The circuit breaker itself is unchanged — these reduce the error rate so it stops tripping on healthy resolvers.
+
+### Fixed
+- **`whois` availability probe is now platform-aware** — the fallback used `which whois` (Unix-only), which on native Windows would false-negative an installed `whois.exe` whose `whois --version` errors (e.g. Sysinternals whois). Uses `where` on Windows, `which` elsewhere. No change on Linux/macOS/WSL.
+
+## [3.3.0] - 2026-06-06
+
+### Added
+- **DNS dead-domain skip + corroborated persistence** — within a scan, once a host resolves NXDOMAIN/ENODATA it is remembered and repeat URLs on that host are skipped without re-resolving. With `--dns-cache`, a host that *also* fails navigation (`ERR_NAME_NOT_RESOLVED` / `ERR_ADDRESS_UNREACHABLE`) is corroborated and persisted to the negative cache (`.dnsnegcache`, 12h TTL) so it is skipped on the next run too. Only definitive non-existence is cached — resolver errors fail open and never poison a live host.
+- **`acceptInsecureCerts` on browser launch** — TLS/cert errors (expired, self-signed, name-mismatch) no longer abort navigation, so streaming/pirate domains with broken certs are still scanned.
+- **`--disable-popup-blocking` when a site uses `capture_popups`** — Chrome's pop-up blocker (`chrome://settings/content/popups`) is turned off only for popup-capture scans, so non-gesture popunders (document-level `onclick` / timer SDKs) fire and get captured too. Non-popup scans keep the blocker on (stealthier — a real browser blocks non-gesture `window.open()`); gesture-triggered popups already worked via the synthetic-click path.
+
+### Changed
+- **The main-frame document is never blocked** — the scanned page (and any main-frame redirect target) is exempt from adblock / `blocked` / `blockDomainsByUrl` aborts. Aborting it made the navigation never commit (`about:blank` → timeout), silently breaking scanned URLs that matched our own filter lists (common on adult/pirate/stream domains). The request still flows through the matcher, so a main-frame redirect destination (e.g. a filecrypt → ad-domain hop) is still captured; sub-frame / ad iframes stay blockable.
+- **Navigation timeouts are recovered, not discarded** — on a nav timeout the scanner retries leniently and proceeds with the partially-loaded page instead of dropping the URL (a page still at `about:blank` is still treated as a failure).
+- **whois disk-cache TTL raised to 36h** (dig stays 20h) — registrar data is stable and whois servers rate-limit aggressively, so a longer TTL cuts repeat queries; dig keeps its 20h TTL.
+- **VPN is Linux-only with a clear guard** — `vpn` / `openvpn` on macOS/Windows now returns an explicit "Linux-only" error instead of cryptic `ip` / `/proc` failures.
+
+### Performance
+- **`psl.parse` memoized by hostname** in the request hot path — both per-request handlers (main page + popup capture) parsed the root domain of *every* request, while a page hammers the same handful of hosts (CDN, analytics, ad domains). A hostname-keyed memo turns almost all of those into `Map` hits, replacing the URL-keyed cache (fewer + shorter keys, far higher hit rate).
+- **Lower per-request overhead** — the iframe-loop guard's `frame().url()` lookup is now gated behind a cheap URL string test instead of running on every request.
+- **Removed redundant disk I/O** — a leaked adblock combined-list temp file in `tmpdir` is now cleaned up, and a redundant `existsSync` before each forced screenshot's recursive `mkdir` was dropped.
+
+### Fixed
+- **Periodic debug/`--dumpurls` log flush is now synchronous** — the 2s timer used async `fs.writeFile({flag:'a'})` with no in-flight guard, so two ticks could append to the same file concurrently and interleave lines, and it cleared the buffer *before* the write confirmed (silently dropping entries on a failed write). It now uses `appendFileSync`, clears only after a successful write (transient failures retry next tick), and is bounded so a permanently-unwritable path can't grow memory.
+- **Dead-domain skip works without `--show-dead-domains`** — the in-scan skip recorded into the dead set only when the report flag was on, which made the skip dead code; recording is now unconditional and the flag gates only the end-of-scan report. Transient DNS errors were also dropped from the dead-domain match so only `ERR_NAME_NOT_RESOLVED` / `ERR_ADDRESS_UNREACHABLE` mark a host dead.
+
+### Removed
+- **Hardcoded `dmzjmp` iframe-loop guard** — the domain-specific abort for a `creative.dmzjmp.com` frame requesting `go.dmzjmp.com/api/models` (added mid-2025 to stop a runaway request loop) has not recurred and was removed from the request hot path; the per-URL timeout remains the backstop. Recoverable from git history — prefer a config-driven `iframe_loop_guards` entry if it ever returns.
+
+### Documentation
+- **README + man page now document `--block-ads` and `--adblock-engine`** — blocking ads/trackers *during* the scan with EasyList-format list(s) (comma-separated), and the `js` (default, native parser) vs `rust` (Brave `adblock-rs`) matcher backends.
+
 ## [3.2.0] - 2026-06-04
 
 ### Added

package/README.md

@@ -66,9 +66,10 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--use-puppeteer-core`      | Use `puppeteer-core` with system Chrome instead of bundled Chromium |
 | `--use-obscura`             | Connect to running Obscura CDP server (`ws://127.0.0.1:9222` or `OBSCURA_WS` env). Skips fingerprint injection — Obscura provides built-in stealth |
 | `--load-extension <path>`   | Load unpacked Chrome extension from directory (can be used multiple times) |
-| `--dns-cache`               | Persist dig/whois results to disk between runs (20hr TTL, 2000-entry cap each, `.digcache`/`.whoiscache`), **plus** the DNS pre-check negative cache (NXDOMAIN/ENODATA only — never resolver errors — 12h TTL, `.dnsnegcache`) so known-dead hosts aren't re-resolved next run. Disk writes are atomic (tmp + rename); corrupt cache files are detected on load with a `[dns-cache]` warn line and reset cleanly. |
-| `--no-dns-precheck`         | Disable per-URL DNS resolution check before page navigation. By default, hosts that dig/whois have already proven live (within the 20hr cache TTL) skip their c-ares pre-check via a positive-resolution index. |
-| `--block-ads=<files>`       | Block ads using EasyList format rules (comma-separated: `easylist.txt,easyprivacy.txt`) |
+| `--dns-cache`               | Persist dig/whois results to disk between runs (dig 20hr / whois 36hr TTL, 2000-entry cap each, `.digcache`/`.whoiscache`), **plus** the DNS pre-check negative cache (NXDOMAIN/ENODATA only — never resolver errors — 12h TTL, `.dnsnegcache`) so known-dead hosts aren't re-resolved next run. Disk writes are atomic (tmp + rename); corrupt cache files are detected on load with a `[dns-cache]` warn line and reset cleanly. |
+| `--no-dns-precheck`         | Disable per-URL DNS resolution check before page navigation. By default, hosts that dig/whois have already proven live (within the dig/whois cache TTL) skip their c-ares pre-check via a positive-resolution index. |
+| `--block-ads=<files>`       | Block ads/trackers **during the scan** using EasyList-format filter list(s) (`\|\|domain^`, `/ads/*`, etc.). Comma-separated for multiple: `--block-ads=easylist.txt,easyprivacy.txt`. See [Blocking ads during the scan](#blocking-ads-during-the-scan). |
+| `--adblock-engine=<js\|rust>` | Matcher backend for `--block-ads` (default: `js`). `rust` uses Brave's `adblock-rs` (much faster on large lists) and requires `npm i adblock-rs`. |
 | `--cdp`                     | Enable Chrome DevTools Protocol logging (now per-page if enabled) |
 | `--remove-dupes`            | Remove duplicate domains from output (only with `-o`) |
 | `--dry-run`                 | Console output only: show matching regex, titles, whois/dig/searchstring results, and adblock rules |
@@ -76,7 +77,8 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--help`, `-h`              | Show this help menu |
 | `--version`                 | Show script version |
 | `--max-concurrent <number>` | Maximum concurrent site processing (1-50, overrides config/default) |
-| `--dns <ip[,ip,...]>` | Resolver(s) for the DNS pre-check **and** nettools' `dig` (one pins, several rotate per query; overrides `/etc/resolv.conf`). Does not affect Chrome navigation or `whois`. Useful when the system resolver is flaky and `dig`-gated domains time out |
+| `--dns <ip[,ip,...]>` | Resolver(s) for the DNS pre-check, nettools' `dig`, **and** — when they map to a known public DoH provider — Chrome's page navigation via DNS-over-HTTPS on direct connections (one pins, several rotate per query; overrides `/etc/resolv.conf`; does not affect `whois`). Chrome normally ignores `--dns` and reads `resolv.conf` directly, so a broken system resolver could fail a domain the pre-check already resolved; pinning Chrome's secure-DNS (`automatic` mode) to the matching provider closes that gap. Mapped providers: Google, Cloudflare, Quad9, OpenDNS, AdGuard, CleanBrowsing, DNS.SB, Mullvad (incl. malware/family/unfiltered variants). **Skipped under a proxy/VPN** (the exit/tunnel resolves); unmapped resolvers (custom/ISP, per-account, IPv6) fall back to system DNS with a warning |
+| `--doh-disable` | Opt out of the Chrome-navigation DoH pinning (default: off). Chrome resolves page navigation via system `resolv.conf` even when `--dns` maps to a known provider; the pre-check and `dig` still honor `--dns`. Use when DoH adds latency, is blocked on the network, or you want system-path resolution |
 | `--show-dead-domains` | At end of scan, list hostnames that did not resolve / were unreachable (`NXDOMAIN`/`ENODATA` + `ERR_NAME_NOT_RESOLVED`/`ERR_ADDRESS_UNREACHABLE`). Excludes blocks/timeouts (those mean the domain is alive). For pruning dead URLs. |
 | `--cleanup-interval <number>` | Browser restart interval in URLs processed (1-1000, overrides config/default) |
 
@@ -92,6 +94,37 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--clear-cache`             | Clear persistent cache before scanning (improves fresh start performance) |
 | `--ignore-cache`            | Bypass all smart caching functionality during scanning |
 
+### Blocking ads during the scan
+
+`--block-ads` makes the scanner **block** matching requests *during* the scan (separate from capturing rules) — to keep ad/tracker noise out of the page, speed up loads, or test that a list catches what it should.
+
+**Adding lists.** Pass one or more EasyList-format filter lists (same syntax as uBlock Origin / EasyList):
+
+```bash
+# Single list
+node nwss.js --block-ads=easylist.txt
+
+# Multiple lists — comma-separated, no spaces
+node nwss.js --block-ads=easylist.txt,easyprivacy.txt,mylist.txt
+```
+
+Lists are plain-text **network** rules — `||doubleclick.net^`, `/ads/*`, `||example.com^$script`, etc. Element-hiding/cosmetic rules (`##…`) don't apply to request blocking and are ignored. The scanned page's own top-level document is never blocked (only sub-resources), so a site whose own domain is in a list still loads.
+
+**Engine — `js` vs `rust`** (`--adblock-engine`, default `js`):
+
+| Engine | Flag | Backend | When |
+|---|---|---|---|
+| **js** (default) | `--adblock-engine=js` | `lib/adblock.js` — pure-JS, no extra deps | Default; fine for small/medium lists, works everywhere |
+| **rust** | `--adblock-engine=rust` | `lib/adblock-rust.js` — Brave's [`adblock-rs`](https://github.com/brave/adblock-rust) | Large lists (full EasyList + EasyPrivacy + …); much faster matching. Drop-in (same rules, same results). Requires `npm install adblock-rs` (needs a Rust toolchain) |
+
+The two engines are interchangeable — same rule format, same blocking result; `rust` is purely a speed option for big lists. If you pass `--adblock-engine=rust` without `adblock-rs` installed, install it (`npm i adblock-rs`) or drop the flag to use `js`.
+
+```bash
+# Fast matching over big lists with the Rust engine
+npm install adblock-rs
+node nwss.js --block-ads=easylist.txt,easyprivacy.txt --adblock-engine=rust
+```
+
 ---
 
 ## config.json Format
@@ -165,6 +198,7 @@ Example:
 | `interact`           | `true` or `false` | `false` | Simulate user interaction (hover, click) |
 | `firstParty`         | `0` or `1` | `0` | Match first-party requests |
 | `thirdParty`         | `0` or `1` | `1` | Match third-party requests |
+| `redirect_first_party` | Boolean | `true` | Whether redirect-destination domains count as first-party. Set `false` to keep redirect targets (and chain hops) **third-party**, so `filterRegex`/`dig` can match them under `thirdParty:true` — e.g. capturing the end domain of an ad/cloak redirect. The originally-scanned domain stays first-party either way. Note: this also un-excludes the redirect target's own same-domain resources (more captured) |
 | `subDomains`         | `0` or `1` | `0` | 1 = preserve subdomains in output |
 | `blocked`            | Array | - | Domains or regexes to block during scanning |
 | `even_blocked`       | Boolean | `false` | Add matching rules even if requests are blocked |
@@ -284,6 +318,8 @@ When a page redirects to a new domain, first-party/third-party detection is base
 | `interact_click_count` | Integer | `3` | Number of random content-zone clicks per load (capped at 20). Default 3 = primary + 2 backups, since ad SDKs sometimes suppress the 1st/2nd click as warmup |
 | `realistic_click`    | Boolean | `false` | Higher click fidelity: denser mouse approach (15 steps), ±1px hand-tremor micro-moves during the press, and ±1.5px mouseup drift (so mousedown≠mouseup coords) — for sites that score click realism. Costs ~80–120ms/click |
 | `interact_typing`    | Boolean | `false` | Enable typing simulation |
+| `click_elements`     | String[] | - | After load, click these CSS selectors **in order**, searched across the **main frame and any iframe** — reach content via organic navigation/gesture instead of a direct load (e.g. `["a[href*='/movie/']", ".play"]` to click a link then a play button). The request interceptor stays attached, so the post-click page's requests are matched against `filterRegex`/`dig` as usual. A click that navigates is followed; later selectors query the resulting page. Honors `realistic_click` and `cursor_mode: "ghost"` (Bezier travel to the element); missing elements are skipped (never fails the scan) |
+| `click_wait`         | Integer | `5000` | Per click: max time (ms) to wait for the element to appear/be visible (`waitForSelector`) **and** the settle/navigation wait after it; capped at half the per-URL timeout |
 | `interact_intensity` | String | `"medium"` | Interaction simulation intensity: "low", "medium", "high" |
 | `cursor_mode`        | `"ghost"` | - | Use ghost-cursor Bezier mouse movements (requires `npm i ghost-cursor`) |
 | `ghost_cursor_speed` | Number | auto | Ghost-cursor movement speed multiplier |

package/lib/dns.js

@@ -117,6 +117,29 @@ function createRotatingResolver(opts = {}) {
   // distribution stays exactly even (no locking needed).
   const nextResolver = () => (pool ? pool[cursor++ % pool.length] : dnsPromises);
 
+  // Concurrency cap (counting semaphore). The scanner runs up to max_concurrent
+  // navigations, each firing a pre-check; bursting that many simultaneous c-ares
+  // UDP queries at one resolver provokes timeouts / EREFUSED rate-limiting (and
+  // is rough on WSL2's UDP-through-NAT path), which then trips the pre-check
+  // circuit breaker. Capping in-flight queries paces the burst so the resolver
+  // can keep up. Excess callers queue and drain quickly (resolutions are short).
+  const maxInFlight = Math.max(1, opts.maxConcurrent || 6);
+  let inFlight = 0;
+  const waiters = [];
+  function acquire() {
+    return new Promise(resolve => {
+      if (inFlight < maxInFlight) { inFlight++; resolve(); }
+      else waiters.push(resolve);
+    });
+  }
+  function release() {
+    inFlight--;
+    if (waiters.length > 0 && inFlight < maxInFlight) {
+      inFlight++;
+      waiters.shift()();
+    }
+  }
+
   // One resolution attempt: rotate the lead server, resolve4 first, and on
   // no-IPv4 (ENODATA/ENOTFOUND) fall back to resolve6 so IPv6-only hosts aren't
   // wrongly skipped. Any OTHER code propagates unchanged so the caller sees the
@@ -147,16 +170,21 @@ function createRotatingResolver(opts = {}) {
    * if one REFUSES, the retry hits another).
    */
   async function resolveHost(hostname, timeoutMs) {
+    await acquire();
     try {
-      await attempt(hostname, timeoutMs);
-    } catch (firstErr) {
-      const code = firstErr && firstErr.code;
-      if (DNS_TRANSIENT_ERRORS.has(code) || (firstErr && firstErr.message === 'DNS timeout')) {
-        if (forceDebug) console.log(formatLogMessage('debug', `DNS pre-check transient (${code || 'timeout'}) for ${hostname}, retrying once`));
+      try {
         await attempt(hostname, timeoutMs);
-      } else {
-        throw firstErr;
+      } catch (firstErr) {
+        const code = firstErr && firstErr.code;
+        if (DNS_TRANSIENT_ERRORS.has(code) || (firstErr && firstErr.message === 'DNS timeout')) {
+          if (forceDebug) console.log(formatLogMessage('debug', `DNS pre-check transient (${code || 'timeout'}) for ${hostname}, retrying once`));
+          await attempt(hostname, timeoutMs);
+        } else {
+          throw firstErr;
+        }
       }
+    } finally {
+      release();
     }
   }
 
@@ -230,9 +258,91 @@ function createDnsCircuitBreaker(opts = {}) {
   };
 }
 
+// Map well-known public resolver IPs (IPv4) to their DNS-over-HTTPS (DoH)
+// endpoint templates. Chrome's page-navigation resolver ignores --dns and reads
+// /etc/resolv.conf directly; pointing Chrome's DoH at the same provider the
+// --dns pre-check uses closes that gap, so a broken or filtering system
+// resolv.conf can't fail navigations the pre-check already passed.
+//
+// Only providers whose DoH endpoint is derivable from a fixed anycast IP are
+// listed. Resolvers with per-account templates (NextDNS, ControlD, AdGuard
+// personal) can't be mapped from an IP and fall through to `unmapped` — Chrome
+// keeps system DNS for those (a warning is logged). IPv6 is intentionally not
+// mapped here; an IPv6 --dns server simply falls back to system DNS.
+const DOH_PROVIDER_TEMPLATES = {
+  // Google
+  '8.8.8.8': 'https://dns.google/dns-query',
+  '8.8.4.4': 'https://dns.google/dns-query',
+  // Cloudflare — standard / malware-blocking / malware+adult
+  '1.1.1.1': 'https://cloudflare-dns.com/dns-query',
+  '1.0.0.1': 'https://cloudflare-dns.com/dns-query',
+  '1.1.1.2': 'https://security.cloudflare-dns.com/dns-query',
+  '1.0.0.2': 'https://security.cloudflare-dns.com/dns-query',
+  '1.1.1.3': 'https://family.cloudflare-dns.com/dns-query',
+  '1.0.0.3': 'https://family.cloudflare-dns.com/dns-query',
+  // Quad9 — secured (default) / unsecured / secured+ECS
+  '9.9.9.9': 'https://dns.quad9.net/dns-query',
+  '149.112.112.112': 'https://dns.quad9.net/dns-query',
+  '9.9.9.10': 'https://dns10.quad9.net/dns-query',
+  '149.112.112.10': 'https://dns10.quad9.net/dns-query',
+  '9.9.9.11': 'https://dns11.quad9.net/dns-query',
+  '149.112.112.11': 'https://dns11.quad9.net/dns-query',
+  // OpenDNS — standard / FamilyShield
+  '208.67.222.222': 'https://doh.opendns.com/dns-query',
+  '208.67.220.220': 'https://doh.opendns.com/dns-query',
+  '208.67.222.123': 'https://doh.familyshield.opendns.com/dns-query',
+  '208.67.220.123': 'https://doh.familyshield.opendns.com/dns-query',
+  // AdGuard DNS — default (ad/tracker block) / non-filtering / family
+  '94.140.14.14': 'https://dns.adguard-dns.com/dns-query',
+  '94.140.15.15': 'https://dns.adguard-dns.com/dns-query',
+  '94.140.14.140': 'https://unfiltered.adguard-dns.com/dns-query',
+  '94.140.14.141': 'https://unfiltered.adguard-dns.com/dns-query',
+  '94.140.14.15': 'https://family.adguard-dns.com/dns-query',
+  '94.140.15.16': 'https://family.adguard-dns.com/dns-query',
+  // CleanBrowsing — security / family / adult
+  '185.228.168.9': 'https://doh.cleanbrowsing.org/doh/security-filter/',
+  '185.228.169.9': 'https://doh.cleanbrowsing.org/doh/security-filter/',
+  '185.228.168.168': 'https://doh.cleanbrowsing.org/doh/family-filter/',
+  '185.228.169.168': 'https://doh.cleanbrowsing.org/doh/family-filter/',
+  '185.228.168.10': 'https://doh.cleanbrowsing.org/doh/adult-filter/',
+  '185.228.169.11': 'https://doh.cleanbrowsing.org/doh/adult-filter/',
+  // DNS.SB
+  '185.222.222.222': 'https://doh.dns.sb/dns-query',
+  '45.11.45.11': 'https://doh.dns.sb/dns-query',
+  // Mullvad (non-filtering)
+  '194.242.2.2': 'https://dns.mullvad.net/dns-query',
+};
+
+/**
+ * Resolve a list of --dns resolver specs to Chrome DoH templates.
+ * Strips any :port (DoH is always 443) and dedupes. Returns the
+ * space-joined template string Chrome's --dns-over-https-templates wants,
+ * plus which inputs were mapped vs had no known DoH endpoint.
+ * @param {string[]} servers - resolver IPs (optionally ip:port) from --dns
+ * @returns {{ templates: string, mapped: string[], unmapped: string[] }}
+ */
+function dohTemplatesForResolvers(servers) {
+  const templates = [];
+  const mapped = [];
+  const unmapped = [];
+  for (const raw of (servers || [])) {
+    const ip = String(raw).trim().replace(/:\d+$/, ''); // drop :port — DoH is 443
+    if (!ip) continue;
+    const tpl = DOH_PROVIDER_TEMPLATES[ip];
+    if (tpl) {
+      if (!templates.includes(tpl)) templates.push(tpl);
+      mapped.push(ip);
+    } else {
+      unmapped.push(ip);
+    }
+  }
+  return { templates: templates.join(' '), mapped, unmapped };
+}
+
 module.exports = {
   createRotatingResolver,
   createDnsCircuitBreaker,
   parseDnsServers,
   isNonExistenceError,
+  dohTemplatesForResolvers,
 };

package/lib/fingerprint.js

@@ -2489,44 +2489,47 @@ async function applyUserAgentSpoofing(page, siteConfig, forceDebug, currentUrl)
         }, 'enhanced mouse/pointer spoofing');
 
         safeExecute(() => {
-          // Neutralize CDP fingerprinting traps and filter DevTools traces
-          // CDP's Runtime.enable causes the inspector to read properties on console-logged objects.
-          // Detection scripts exploit this via console.debug with Error objects (custom .stack getters)
-          // or objects with Proxy prototypes. Only override console.debug — safest, minimal footprint.
-
-          const originalConsoleDebug = console.debug;
-          console.debug = function(...args) {
-            // Filter DevTools-related messages
-            const message = args.join(' ');
-            if (typeof message === 'string' && (
-                message.includes('DevTools') ||
-                message.includes('Runtime.evaluate') ||
-                message.includes('Page.addScriptToEvaluateOnNewDocument') ||
-                message.includes('Protocol error'))) {
-              return;
+          // Neutralize CDP "inspector reads logged object" traps across the
+          // common console methods. CDP's Runtime.enable serializes console
+          // arguments, reading getters / walking prototypes on logged objects —
+          // disable-devtool-style scripts exploit this (log an object with a
+          // getter, then check if it fired) to detect the inspector and redirect
+          // away. The previous version guarded ONLY console.debug; detectors
+          // overwhelmingly use console.log (and dir/table), so the trap still
+          // fired. Sanitize args so getter/proxy traps can't trigger, and drop
+          // DevTools-protocol noise. Covers log/info/debug/dir/table — the
+          // methods detection uses; warn/error stay native (legit usage, and
+          // not the common vector).
+          const sanitizeArg = (arg) => {
+            // Strip Error objects with custom .stack getters (inspector reads .stack)
+            if (arg instanceof Error) {
+              const desc = Object.getOwnPropertyDescriptor(arg, 'stack');
+              if (desc && desc.get) return `${arg.name}: ${arg.message}`;
             }
-            // Sanitize args to neutralize CDP fingerprinting traps
-            const sanitized = args.map(arg => {
-              // Strip Error objects with custom .stack getters (CDP inspector reads .stack)
-              if (arg instanceof Error) {
-                const desc = Object.getOwnPropertyDescriptor(arg, 'stack');
-                if (desc && desc.get) return `${arg.name}: ${arg.message}`;
-              }
-              // Neutralize Proxy prototype traps (CDP inspector walks prototype chain)
-              if (arg !== null && typeof arg === 'object') {
-                try {
-                  const proto = Object.getPrototypeOf(arg);
-                  if (proto && proto !== Object.prototype && proto !== Array.prototype) {
-                    try { Object.keys(proto); } catch { return '[object Object]'; }
-                  }
-                } catch { return '[object Object]'; }
-              }
-              return arg;
-            });
-            return originalConsoleDebug.apply(this, sanitized);
+            // Neutralize Proxy/getter prototype traps (inspector walks the chain)
+            if (arg !== null && typeof arg === 'object') {
+              try {
+                const proto = Object.getPrototypeOf(arg);
+                if (proto && proto !== Object.prototype && proto !== Array.prototype) {
+                  try { Object.keys(proto); } catch { return '[object Object]'; }
+                }
+              } catch { return '[object Object]'; }
+            }
+            return arg;
           };
-
-        }, 'console error suppression');
+          const DEVTOOLS_NOISE = ['DevTools', 'Runtime.evaluate', 'Page.addScriptToEvaluateOnNewDocument', 'Protocol error'];
+          for (const method of ['log', 'info', 'debug', 'dir', 'table']) {
+            const original = console[method];
+            if (typeof original !== 'function') continue;
+            console[method] = function(...args) {
+              const message = args.join(' ');
+              if (typeof message === 'string' && DEVTOOLS_NOISE.some(n => message.includes(n))) {
+                return;
+              }
+              return original.apply(this, args.map(sanitizeArg));
+            };
+          }
+        }, 'console trap neutralization');
         
         // NOTE: The previous `location URL masking` Proxy was removed.
         // It wrapped window.location in a Proxy to return 'about:blank' when

package/lib/interaction.js

@@ -1322,6 +1322,154 @@ function createInteractionConfig(url, siteConfig = {}) {
  * const { generateRandomCoordinates } = require('./lib/interaction');
  * const pos = generateRandomCoordinates(1920, 1080, { preferEdges: true });
  */
+
+/**
+ * Find the first VISIBLE match for a selector across the main frame AND every
+ * child frame (iframes), polling until found or timeoutMs. page.waitForSelector
+ * only searches the main frame; players/ads commonly live in an iframe, so we
+ * walk page.frames() (main frame first, so a main-frame match wins). "Visible"
+ * is approximated by a non-null boundingBox (rules out display:none/zero-size).
+ * Returns an ElementHandle bound to its frame, or null on timeout.
+ */
+async function findVisibleInAnyFrame(page, selector, timeoutMs) {
+  const deadline = Date.now() + Math.max(0, timeoutMs);
+  for (;;) {
+    if (page.isClosed()) return null;
+    const mf = page.mainFrame();
+    const frames = [mf, ...page.frames().filter(f => f !== mf)]; // main frame first
+    for (const frame of frames) {
+      let el = null;
+      try {
+        el = await frame.$(selector);
+      } catch (e) {
+        // An invalid CSS selector (config typo) throws "... is not a valid
+        // selector" — a permanent error, so surface it instead of polling to a
+        // confusing not-found. Frame-detached / cross-origin hiccups have other
+        // messages and fall through to keep polling.
+        if (/is not a valid selector|not a valid or unsupported selector/i.test((e && e.message) || '')) {
+          throw new Error(`invalid CSS selector: ${(e && e.message) || e}`);
+        }
+        el = null;
+      }
+      if (el) {
+        let box = null;
+        try { box = await el.boundingBox(); } catch (_) { box = null; }
+        if (box) return el;            // present AND rendered
+        try { await el.dispose(); } catch (_) { /* not rendered yet — keep polling */ }
+      }
+    }
+    if (Date.now() >= deadline) return null;
+    await new Promise(r => setTimeout(r, 250));
+  }
+}
+
+/**
+ * Click a list of CSS selectors in order, reaching content via organic
+ * gesture/navigation instead of a direct page load. Each selector's first
+ * match is clicked; if the click navigates (an <a href> / form submit), we wait
+ * for it to commit, otherwise we wait a settle window for in-page actions
+ * (e.g. a player starting). The page's request interceptor stays attached
+ * throughout, so the post-click requests flow into the caller's normal
+ * filterRegex/dig matching — this function only performs the clicks.
+ *
+ * Missing elements are skipped (sites change markup); a click error never
+ * throws out of here. After a navigation, later selectors are queried against
+ * the NEW page (so e.g. "movie link" then "play button" works).
+ *
+ * @param {import('puppeteer').Page} page
+ * @param {string[]} selectors - CSS selectors, clicked in order
+ * @param {object} [options]
+ * @param {boolean} [options.realistic=false] - use humanClick (hover/tremor) vs elementHandle.click
+ * @param {number}  [options.waitMs=5000] - per click: max wait for the element to
+ *   appear+be visible (waitForSelector), AND the settle/nav window after the click
+ * @param {function} [options.ghostClick] - optional (x,y)=>Promise that performs a
+ *   ghost-cursor click (Bezier travel + press) at the element centre. Injected by the
+ *   caller so this module needn't depend on ghost-cursor.js (which depends on this one).
+ *   When provided it takes precedence over the humanClick/el.click paths.
+ * @param {boolean} [options.forceDebug=false]
+ * @returns {Promise<Array<{selector:string, clicked:boolean, reason?:string}>>}
+ */
+async function performTargetedClicks(page, selectors, options = {}) {
+  const { realistic = false, waitMs = 5000, forceDebug = false, ghostClick = null } = options;
+  const results = [];
+  if (!Array.isArray(selectors)) return results;
+
+  for (const raw of selectors) {
+    const selector = typeof raw === 'string' ? raw.trim() : '';
+    if (!selector) continue;
+    if (page.isClosed()) break;
+
+    // Wait for the element to appear AND be visible (up to waitMs), searching
+    // the main frame and every iframe — many targets (video players, lazy
+    // menus, post-consent buttons) are injected by JS after DOMContentLoaded
+    // and/or live inside an iframe, so a single main-frame query would miss
+    // them. Timeout → treat as not-found, skip.
+    let el;
+    try {
+      el = await findVisibleInAnyFrame(page, selector, waitMs);
+    } catch (selErr) {
+      const msg = (selErr && selErr.message) || '';
+      if (/invalid CSS selector|not a valid selector/i.test(msg)) {
+        // Config typo, not a transient miss — warn (visible without --debug).
+        console.warn(formatLogMessage('warn', `${INTERACTION_TAG} click_elements: invalid selector "${selector}" — skipping (${msg})`));
+        results.push({ selector, clicked: false, reason: 'invalid-selector' });
+      } else {
+        // Defensive: findVisibleInAnyFrame swallows transient/detached-frame
+        // errors internally, so this shouldn't fire — but never let an
+        // unexpected find error abort the remaining selectors.
+        if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: find failed for "${selector}": ${msg} — skipping`));
+        results.push({ selector, clicked: false, reason: msg || 'find-error' });
+      }
+      continue;
+    }
+    if (!el) {
+      if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: "${selector}" not visible (any frame) within ${waitMs}ms — skipping`));
+      results.push({ selector, clicked: false, reason: 'not-found' });
+      continue;
+    }
+
+    try {
+      // Bring it into view so coordinate clicks land (elementHandle.click also
+      // auto-scrolls, but humanClick clicks raw coordinates).
+      try { await el.evaluate(e => e.scrollIntoView({ block: 'center', inline: 'center' })); } catch (_) { /* detached/odd element */ }
+
+      // Arm a navigation wait BEFORE clicking so a link/submit is caught.
+      const navP = page.waitForNavigation({ waitUntil: 'domcontentloaded', timeout: waitMs + 3000 }).catch(() => {});
+
+      let box = null;
+      try { box = await el.boundingBox(); } catch (_) { box = null; }
+      if (typeof ghostClick === 'function' && box) {
+        // Ghost-cursor path: Bezier travel to the element centre + realistic
+        // press, matching the interact phase (caller-injected).
+        await ghostClick(box.x + box.width / 2, box.y + box.height / 2);
+      } else if (realistic && box) {
+        await humanClick(page, box.x + box.width / 2, box.y + box.height / 2, { realistic: true, forceDebug });
+      } else {
+        await el.click({ delay: 30 }); // trusted gesture; auto-scrolls + handles non-visible coords
+      }
+
+      // Resolve on whichever comes first: a committed navigation, or the settle
+      // window (in-page actions). Either way, requests fired in between are
+      // already captured by the caller's interceptor. Capture-and-clear the
+      // settle timer (cdp.js / interact pattern): when navP wins, an uncleared
+      // setTimeout would keep the event loop + closure alive for the full waitMs.
+      let settleTimer;
+      await Promise.race([
+        navP,
+        new Promise(r => { settleTimer = setTimeout(r, waitMs); })
+      ]).finally(() => clearTimeout(settleTimer));
+      results.push({ selector, clicked: true });
+      if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: clicked "${selector}"`));
+    } catch (err) {
+      if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: click failed for "${selector}": ${err.message}`));
+      results.push({ selector, clicked: false, reason: err.message });
+    } finally {
+      try { await el.dispose(); } catch (_) { /* detached after navigation — fine */ }
+    }
+  }
+  return results;
+}
+
 module.exports = {
   // Main interaction functions
   performPageInteraction,
@@ -1337,5 +1485,8 @@ module.exports = {
   // hand-tremor + mouseup drift). Reused by lib/ghost-cursor.js so the ghost
   // coordinate click gets the same press realism as built-in content clicks.
   humanClick,
+  // Click specific CSS selectors in order (organic navigation / play-button /
+  // link clicking) — site config `click_elements`.
+  performTargetedClicks,
   generateRandomCoordinates
 };

package/lib/nettools.js

@@ -30,7 +30,7 @@ const GLOBAL_DIG_CACHE_MAX = 2000;
 // Global whois result cache — shared across ALL handler instances and processUrl calls
 // Whois data is per root domain and doesn't change based on search terms
 const globalWhoisResultCache = new Map();
-const GLOBAL_WHOIS_CACHE_TTL = 72000000; // 20 hours (persisted to disk between runs)
+const GLOBAL_WHOIS_CACHE_TTL = 129600000; // 36 hours (persisted to disk between runs). Longer than dig's 20h: registrar data is very stable and whois servers rate-limit aggressively, so caching longer cuts repeat queries.
 const GLOBAL_WHOIS_CACHE_MAX = 2000;
 
 // Persistent disk cache file paths
@@ -40,8 +40,8 @@ const WHOIS_CACHE_FILE = path.join(__dirname, '..', '.whoiscache');
 // Index of hostnames known to resolve, populated as a side effect of
 // positive dig/whois cache writes AND cache hits. nwss.js's DNS pre-check
 // reads this via domainKnownToResolve() so it can skip its own resolve4
-// call on hosts that dig or whois have already proven live within the
-// 20-hour TTL window. Populating on cache HITS (not just writes) handles
+// call on hosts that dig or whois have already proven live within their
+// cache TTL window (dig 20h / whois 36h). Populating on cache HITS (not just writes) handles
 // the --dns-cache disk-load case where entries arrive without going
 // through the in-process write path. Stale entries -- hostname in Set but
 // the dig/whois entry has since been evicted -- are harmless: worst case
@@ -366,7 +366,10 @@ function validateWhoisAvailability() {
     };
   } catch (error) {
     try {
-      execSync('which whois', { encoding: 'utf8' });
+      // `which` is Unix-only; Windows uses `where`. Without this, an installed
+      // whois.exe whose `whois --version` errors (e.g. Sysinternals whois)
+      // would be false-negatived as unavailable on native Windows.
+      execSync(process.platform === 'win32' ? 'where whois' : 'which whois', { encoding: 'utf8' });
       validateWhoisAvailability._cached = {
         isAvailable: true,
         version: 'whois (version unknown)'

package/lib/openvpn_vpn.js

@@ -778,6 +778,14 @@ function validateOvpnConfig(ovpnConfig) {
  * @returns {Promise<Object>} { success, connection, tunDevice, error }
  */
 async function connectForSite(siteConfig, forceDebug = false) {
+  // Platform guard: OpenVPN routing here reads /proc and uses the iproute2 `ip`
+  // command, both Linux-only. Fail clearly instead of a cryptic /proc or `ip`
+  // error on macOS/Windows. WSL2 reports 'linux' and passes (TUN is checked
+  // separately below via isWSL/checkTunDevice).
+  if (process.platform !== 'linux') {
+    return { success: false, error: `OpenVPN routing is currently Linux-only (needs /proc + the iproute2 'ip' command; not available on ${process.platform}). Run on Linux/WSL2, or remove the 'openvpn' option from the site config.` };
+  }
+
   const ovpnConfig = normalizeOvpnConfig(siteConfig.openvpn);
   if (!ovpnConfig) {
     return { success: false, error: 'Invalid OpenVPN configuration' };

package/lib/validate_rules.js

@@ -1102,7 +1102,7 @@ function testDomainValidation() {
 const KNOWN_SITE_CONFIG_KEYS = new Set([
   'adblock_rules', 'blocked', 'bypass_cache', 'capture_popups',
   'capture_popups_max_depth', 'capture_popups_window_ms', 'cdp', 'cdp_specific',
-  'clear_sitedata', 'clear_sitedata_full_on_reload',
+  'clear_sitedata', 'clear_sitedata_full_on_reload', 'click_elements', 'click_wait',
   'cloudflare_bypass', 'cloudflare_max_retries', 'comments',
   'cloudflare_parallel_detection', 'cloudflare_phish', 'cloudflare_retry_on_error',
   'css_blocked', 'curl', 'cursor_mode', 'custom_headers', 'delay',
@@ -1119,7 +1119,7 @@ const KNOWN_SITE_CONFIG_KEYS = new Set([
   'js_redirect_timeout', 'localhost', 'max_redirects', 'openvpn', 'pihole',
   'output_regex',
   'plain', 'privoxy', 'proxy', 'proxy_bypass', 'proxy_debug', 'proxy_remote_dns',
-  'realistic_click', 'referrer_disable', 'referrer_headers', 'regex_and',
+  'realistic_click', 'redirect_first_party', 'referrer_disable', 'referrer_headers', 'regex_and',
   'reload', 'resourceTypes', 'screenshot', 'searchstring', 'searchstring_and',
   'socks5_bypass', 'socks5_debug', 'socks5_proxy', 'socks5_remote_dns',
   'subDomains',
@@ -1151,7 +1151,7 @@ const BOOLEAN_SITE_CONFIG_FIELDS = new Set([
   'grep', 'headful', 'ignore_similar', 'ignore_similar_ignored_domains',
   'interact', 'interact_clicks', 'interact_scrolling', 'isBrave', 'localhost',
   'pihole', 'plain', 'privoxy', 'proxy_debug', 'proxy_remote_dns',
-  'realistic_click', 'referrer_disable', 'regex_and', 'screenshot',
+  'realistic_click', 'redirect_first_party', 'referrer_disable', 'regex_and', 'screenshot',
   'searchstring_and', 'socks5_debug', 'socks5_remote_dns', 'thirdParty',
   'unbound', 'whois_retry_on_error', 'whois_retry_on_timeout', 'whois_use_fallback',
 ]);