@fanboynz/network-scanner 3.3.0 → 3.4.0

package/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 All notable changes to the Network Scanner (nwss.js) project.
 
+## [3.4.0] - 2026-06-13
+
+### Added
+- **`redirect_first_party` site option** (default `true`) — by default a redirect's destination domains (and chain hops) are registered first-party so the landed site's own resources aren't captured as third-party. Set `false` to keep redirect targets **third-party**, so `filterRegex`/`dig` apply to them under `thirdParty: true` — e.g. capturing the end domain of an ad/cloak redirect chain (which Chrome reaches via the `ERR_TOO_MANY_REDIRECTS` curl-resolve recovery). The originally-scanned domain stays first-party either way.
+- **`ERR_TOO_MANY_REDIRECTS` is recovered, not hard-failed** — a redirect-cloaking chain (rotating throwaway domains) can exceed Chrome's ~20-hop ceiling. The scanner now recovers via two complementary paths: **(1)** it first waits briefly for the browser to *ride through* on its own — a JS/meta hop on a committed page resets Chrome's hop counter and often carries the page to the end site for free; **(2)** if the page parked on `chrome-error://` instead **and the site has `curl: true`**, it resolves the chain endpoint with `curl` (which, unlike headless Chrome, isn't served the endless-loop variant) and navigates there directly — a short hop that lands on the real end site. Either way it captures the end site's ad/tracker requests; falls back to the captured chain requests when neither path lands. The curl step is opt-in via the existing `curl` site option (so `curl: false`/unset never shells out to curl) and is also **skipped under a proxy/VPN** (curl runs direct and would leak the real IP / resolve from the wrong network); the free ride-through always applies.
+- **`click_elements` site option** — after a page loads, click a list of CSS selectors **in order** (searched across the main frame and any iframe) (e.g. `["a[href*='/movie/']", ".play"]` to click a movie link then a play button). Reaches content via organic navigation/gesture instead of a direct deep-load, which some sites JS-redirect away, and triggers click-only content like video players. Each selector is `waitForSelector`-ed (visible) up to `click_wait` before clicking, so JS-rendered targets like video players aren't missed by racing ahead of them. The request interceptor stays attached, so the post-click page's requests run through the same `filterRegex`/`dig` matching; a click that navigates is followed and later selectors query the resulting page. Honors `realistic_click` (genuine trusted gesture) and `cursor_mode: "ghost"` (Bezier travel to the element); missing elements are skipped and never fail the scan. Settle/nav wait per click via `click_wait` (default 5000ms, capped at half the per-URL timeout).
+- **`--dns` now also pins Chrome's page-navigation resolver via DoH.** Chrome ignores `--dns` for navigation and reads `/etc/resolv.conf` directly, so a broken or filtering system resolver could `ERR_NAME_NOT_RESOLVED` a domain the pre-check had already resolved. When the `--dns` servers map to a known public DoH provider — **Google, Cloudflare, Quad9, OpenDNS, AdGuard, CleanBrowsing, DNS.SB, Mullvad** (incl. malware/family/unfiltered variants) — Chrome is launched with secure-DNS `automatic` mode pointed at that provider, so page navigation resolves through the same resolver as the pre-check. `automatic` (not `secure`) keeps a system-DNS fallback if DoH is unreachable rather than failing the batch. **Applied to direct connections only** — skipped when a proxy (`--proxy-server`) or VPN is active, since the exit/tunnel does the resolution and local DoH would be redundant or resolve geo-split domains to the wrong region. Unmapped resolvers (custom/ISP, per-account providers like NextDNS, IPv6) fall back to system DNS with a warning naming the supported providers.
+- **`--doh-disable`** site/CLI option (`doh_disable` in `.nwssconfig`), default off — opt out of the Chrome-navigation DoH pinning entirely. Chrome then resolves page navigation via the system `resolv.conf` even when `--dns` maps to a known provider, while the pre-check and `dig` still honor `--dns`. For networks where DoH adds latency or is blocked, or when system-path resolution is specifically wanted.
+
+### Changed
+- **A clamped `delay` is now logged (`--debug`)** — when `delay` exceeds its ceiling (the default 2s cap, or `timeout/2` under `delay_uncapped: true`) it was silently reduced, so `delay: 48000` quietly running as 29000ms looked like the flag was ignored. A debug line now reports the clamp and which ceiling applied (raise `timeout`, or set `delay_uncapped: true`, to lift it). The per-URL budget already reserves the full configured `delay`; this only surfaces the post-load dwell clamp.
+- **DNS pre-check is paced and more tolerant under concurrency** — a concurrent scan fired up to `max_concurrent` simultaneous c-ares UDP queries at the pinned `--dns` servers; the burst (rough on WSL2's UDP-through-NAT path, and rate-limited by public resolvers) produced timeouts / `EREFUSED` that tripped the circuit breaker (`resolver errors N/M — suspending DNS pre-check`) and lost the dead-host-skip optimization. The pre-check timeout is raised 2s → 4s (a clean NXDOMAIN still returns fast, so the higher ceiling only costs time when the resolver is genuinely slow), and `createRotatingResolver` now caps in-flight queries with a counting semaphore (default 6) so the burst is paced and excess callers queue and drain quickly. The circuit breaker itself is unchanged — these reduce the error rate so it stops tripping on healthy resolvers.
+
+### Fixed
+- **`whois` availability probe is now platform-aware** — the fallback used `which whois` (Unix-only), which on native Windows would false-negative an installed `whois.exe` whose `whois --version` errors (e.g. Sysinternals whois). Uses `where` on Windows, `which` elsewhere. No change on Linux/macOS/WSL.
+
 ## [3.3.0] - 2026-06-06
 
 ### Added

package/README.md

@@ -77,7 +77,8 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--help`, `-h`              | Show this help menu |
 | `--version`                 | Show script version |
 | `--max-concurrent <number>` | Maximum concurrent site processing (1-50, overrides config/default) |
-| `--dns <ip[,ip,...]>` | Resolver(s) for the DNS pre-check **and** nettools' `dig` (one pins, several rotate per query; overrides `/etc/resolv.conf`). Does not affect Chrome navigation or `whois`. Useful when the system resolver is flaky and `dig`-gated domains time out |
+| `--dns <ip[,ip,...]>` | Resolver(s) for the DNS pre-check, nettools' `dig`, **and** — when they map to a known public DoH provider — Chrome's page navigation via DNS-over-HTTPS on direct connections (one pins, several rotate per query; overrides `/etc/resolv.conf`; does not affect `whois`). Chrome normally ignores `--dns` and reads `resolv.conf` directly, so a broken system resolver could fail a domain the pre-check already resolved; pinning Chrome's secure-DNS (`automatic` mode) to the matching provider closes that gap. Mapped providers: Google, Cloudflare, Quad9, OpenDNS, AdGuard, CleanBrowsing, DNS.SB, Mullvad (incl. malware/family/unfiltered variants). **Skipped under a proxy/VPN** (the exit/tunnel resolves); unmapped resolvers (custom/ISP, per-account, IPv6) fall back to system DNS with a warning |
+| `--doh-disable` | Opt out of the Chrome-navigation DoH pinning (default: off). Chrome resolves page navigation via system `resolv.conf` even when `--dns` maps to a known provider; the pre-check and `dig` still honor `--dns`. Use when DoH adds latency, is blocked on the network, or you want system-path resolution |
 | `--show-dead-domains` | At end of scan, list hostnames that did not resolve / were unreachable (`NXDOMAIN`/`ENODATA` + `ERR_NAME_NOT_RESOLVED`/`ERR_ADDRESS_UNREACHABLE`). Excludes blocks/timeouts (those mean the domain is alive). For pruning dead URLs. |
 | `--cleanup-interval <number>` | Browser restart interval in URLs processed (1-1000, overrides config/default) |
 
@@ -197,6 +198,7 @@ Example:
 | `interact`           | `true` or `false` | `false` | Simulate user interaction (hover, click) |
 | `firstParty`         | `0` or `1` | `0` | Match first-party requests |
 | `thirdParty`         | `0` or `1` | `1` | Match third-party requests |
+| `redirect_first_party` | Boolean | `true` | Whether redirect-destination domains count as first-party. Set `false` to keep redirect targets (and chain hops) **third-party**, so `filterRegex`/`dig` can match them under `thirdParty:true` — e.g. capturing the end domain of an ad/cloak redirect. The originally-scanned domain stays first-party either way. Note: this also un-excludes the redirect target's own same-domain resources (more captured) |
 | `subDomains`         | `0` or `1` | `0` | 1 = preserve subdomains in output |
 | `blocked`            | Array | - | Domains or regexes to block during scanning |
 | `even_blocked`       | Boolean | `false` | Add matching rules even if requests are blocked |
@@ -316,6 +318,8 @@ When a page redirects to a new domain, first-party/third-party detection is base
 | `interact_click_count` | Integer | `3` | Number of random content-zone clicks per load (capped at 20). Default 3 = primary + 2 backups, since ad SDKs sometimes suppress the 1st/2nd click as warmup |
 | `realistic_click`    | Boolean | `false` | Higher click fidelity: denser mouse approach (15 steps), ±1px hand-tremor micro-moves during the press, and ±1.5px mouseup drift (so mousedown≠mouseup coords) — for sites that score click realism. Costs ~80–120ms/click |
 | `interact_typing`    | Boolean | `false` | Enable typing simulation |
+| `click_elements`     | String[] | - | After load, click these CSS selectors **in order**, searched across the **main frame and any iframe** — reach content via organic navigation/gesture instead of a direct load (e.g. `["a[href*='/movie/']", ".play"]` to click a link then a play button). The request interceptor stays attached, so the post-click page's requests are matched against `filterRegex`/`dig` as usual. A click that navigates is followed; later selectors query the resulting page. Honors `realistic_click` and `cursor_mode: "ghost"` (Bezier travel to the element); missing elements are skipped (never fails the scan) |
+| `click_wait`         | Integer | `5000` | Per click: max time (ms) to wait for the element to appear/be visible (`waitForSelector`) **and** the settle/navigation wait after it; capped at half the per-URL timeout |
 | `interact_intensity` | String | `"medium"` | Interaction simulation intensity: "low", "medium", "high" |
 | `cursor_mode`        | `"ghost"` | - | Use ghost-cursor Bezier mouse movements (requires `npm i ghost-cursor`) |
 | `ghost_cursor_speed` | Number | auto | Ghost-cursor movement speed multiplier |

package/lib/dns.js

@@ -117,6 +117,29 @@ function createRotatingResolver(opts = {}) {
   // distribution stays exactly even (no locking needed).
   const nextResolver = () => (pool ? pool[cursor++ % pool.length] : dnsPromises);
 
+  // Concurrency cap (counting semaphore). The scanner runs up to max_concurrent
+  // navigations, each firing a pre-check; bursting that many simultaneous c-ares
+  // UDP queries at one resolver provokes timeouts / EREFUSED rate-limiting (and
+  // is rough on WSL2's UDP-through-NAT path), which then trips the pre-check
+  // circuit breaker. Capping in-flight queries paces the burst so the resolver
+  // can keep up. Excess callers queue and drain quickly (resolutions are short).
+  const maxInFlight = Math.max(1, opts.maxConcurrent || 6);
+  let inFlight = 0;
+  const waiters = [];
+  function acquire() {
+    return new Promise(resolve => {
+      if (inFlight < maxInFlight) { inFlight++; resolve(); }
+      else waiters.push(resolve);
+    });
+  }
+  function release() {
+    inFlight--;
+    if (waiters.length > 0 && inFlight < maxInFlight) {
+      inFlight++;
+      waiters.shift()();
+    }
+  }
+
   // One resolution attempt: rotate the lead server, resolve4 first, and on
   // no-IPv4 (ENODATA/ENOTFOUND) fall back to resolve6 so IPv6-only hosts aren't
   // wrongly skipped. Any OTHER code propagates unchanged so the caller sees the
@@ -147,16 +170,21 @@ function createRotatingResolver(opts = {}) {
    * if one REFUSES, the retry hits another).
    */
   async function resolveHost(hostname, timeoutMs) {
+    await acquire();
     try {
-      await attempt(hostname, timeoutMs);
-    } catch (firstErr) {
-      const code = firstErr && firstErr.code;
-      if (DNS_TRANSIENT_ERRORS.has(code) || (firstErr && firstErr.message === 'DNS timeout')) {
-        if (forceDebug) console.log(formatLogMessage('debug', `DNS pre-check transient (${code || 'timeout'}) for ${hostname}, retrying once`));
+      try {
         await attempt(hostname, timeoutMs);
-      } else {
-        throw firstErr;
+      } catch (firstErr) {
+        const code = firstErr && firstErr.code;
+        if (DNS_TRANSIENT_ERRORS.has(code) || (firstErr && firstErr.message === 'DNS timeout')) {
+          if (forceDebug) console.log(formatLogMessage('debug', `DNS pre-check transient (${code || 'timeout'}) for ${hostname}, retrying once`));
+          await attempt(hostname, timeoutMs);
+        } else {
+          throw firstErr;
+        }
       }
+    } finally {
+      release();
     }
   }
 
@@ -230,9 +258,91 @@ function createDnsCircuitBreaker(opts = {}) {
   };
 }
 
+// Map well-known public resolver IPs (IPv4) to their DNS-over-HTTPS (DoH)
+// endpoint templates. Chrome's page-navigation resolver ignores --dns and reads
+// /etc/resolv.conf directly; pointing Chrome's DoH at the same provider the
+// --dns pre-check uses closes that gap, so a broken or filtering system
+// resolv.conf can't fail navigations the pre-check already passed.
+//
+// Only providers whose DoH endpoint is derivable from a fixed anycast IP are
+// listed. Resolvers with per-account templates (NextDNS, ControlD, AdGuard
+// personal) can't be mapped from an IP and fall through to `unmapped` — Chrome
+// keeps system DNS for those (a warning is logged). IPv6 is intentionally not
+// mapped here; an IPv6 --dns server simply falls back to system DNS.
+const DOH_PROVIDER_TEMPLATES = {
+  // Google
+  '8.8.8.8': 'https://dns.google/dns-query',
+  '8.8.4.4': 'https://dns.google/dns-query',
+  // Cloudflare — standard / malware-blocking / malware+adult
+  '1.1.1.1': 'https://cloudflare-dns.com/dns-query',
+  '1.0.0.1': 'https://cloudflare-dns.com/dns-query',
+  '1.1.1.2': 'https://security.cloudflare-dns.com/dns-query',
+  '1.0.0.2': 'https://security.cloudflare-dns.com/dns-query',
+  '1.1.1.3': 'https://family.cloudflare-dns.com/dns-query',
+  '1.0.0.3': 'https://family.cloudflare-dns.com/dns-query',
+  // Quad9 — secured (default) / unsecured / secured+ECS
+  '9.9.9.9': 'https://dns.quad9.net/dns-query',
+  '149.112.112.112': 'https://dns.quad9.net/dns-query',
+  '9.9.9.10': 'https://dns10.quad9.net/dns-query',
+  '149.112.112.10': 'https://dns10.quad9.net/dns-query',
+  '9.9.9.11': 'https://dns11.quad9.net/dns-query',
+  '149.112.112.11': 'https://dns11.quad9.net/dns-query',
+  // OpenDNS — standard / FamilyShield
+  '208.67.222.222': 'https://doh.opendns.com/dns-query',
+  '208.67.220.220': 'https://doh.opendns.com/dns-query',
+  '208.67.222.123': 'https://doh.familyshield.opendns.com/dns-query',
+  '208.67.220.123': 'https://doh.familyshield.opendns.com/dns-query',
+  // AdGuard DNS — default (ad/tracker block) / non-filtering / family
+  '94.140.14.14': 'https://dns.adguard-dns.com/dns-query',
+  '94.140.15.15': 'https://dns.adguard-dns.com/dns-query',
+  '94.140.14.140': 'https://unfiltered.adguard-dns.com/dns-query',
+  '94.140.14.141': 'https://unfiltered.adguard-dns.com/dns-query',
+  '94.140.14.15': 'https://family.adguard-dns.com/dns-query',
+  '94.140.15.16': 'https://family.adguard-dns.com/dns-query',
+  // CleanBrowsing — security / family / adult
+  '185.228.168.9': 'https://doh.cleanbrowsing.org/doh/security-filter/',
+  '185.228.169.9': 'https://doh.cleanbrowsing.org/doh/security-filter/',
+  '185.228.168.168': 'https://doh.cleanbrowsing.org/doh/family-filter/',
+  '185.228.169.168': 'https://doh.cleanbrowsing.org/doh/family-filter/',
+  '185.228.168.10': 'https://doh.cleanbrowsing.org/doh/adult-filter/',
+  '185.228.169.11': 'https://doh.cleanbrowsing.org/doh/adult-filter/',
+  // DNS.SB
+  '185.222.222.222': 'https://doh.dns.sb/dns-query',
+  '45.11.45.11': 'https://doh.dns.sb/dns-query',
+  // Mullvad (non-filtering)
+  '194.242.2.2': 'https://dns.mullvad.net/dns-query',
+};
+
+/**
+ * Resolve a list of --dns resolver specs to Chrome DoH templates.
+ * Strips any :port (DoH is always 443) and dedupes. Returns the
+ * space-joined template string Chrome's --dns-over-https-templates wants,
+ * plus which inputs were mapped vs had no known DoH endpoint.
+ * @param {string[]} servers - resolver IPs (optionally ip:port) from --dns
+ * @returns {{ templates: string, mapped: string[], unmapped: string[] }}
+ */
+function dohTemplatesForResolvers(servers) {
+  const templates = [];
+  const mapped = [];
+  const unmapped = [];
+  for (const raw of (servers || [])) {
+    const ip = String(raw).trim().replace(/:\d+$/, ''); // drop :port — DoH is 443
+    if (!ip) continue;
+    const tpl = DOH_PROVIDER_TEMPLATES[ip];
+    if (tpl) {
+      if (!templates.includes(tpl)) templates.push(tpl);
+      mapped.push(ip);
+    } else {
+      unmapped.push(ip);
+    }
+  }
+  return { templates: templates.join(' '), mapped, unmapped };
+}
+
 module.exports = {
   createRotatingResolver,
   createDnsCircuitBreaker,
   parseDnsServers,
   isNonExistenceError,
+  dohTemplatesForResolvers,
 };

package/lib/fingerprint.js

@@ -2489,44 +2489,47 @@ async function applyUserAgentSpoofing(page, siteConfig, forceDebug, currentUrl)
         }, 'enhanced mouse/pointer spoofing');
 
         safeExecute(() => {
-          // Neutralize CDP fingerprinting traps and filter DevTools traces
-          // CDP's Runtime.enable causes the inspector to read properties on console-logged objects.
-          // Detection scripts exploit this via console.debug with Error objects (custom .stack getters)
-          // or objects with Proxy prototypes. Only override console.debug — safest, minimal footprint.
-
-          const originalConsoleDebug = console.debug;
-          console.debug = function(...args) {
-            // Filter DevTools-related messages
-            const message = args.join(' ');
-            if (typeof message === 'string' && (
-                message.includes('DevTools') ||
-                message.includes('Runtime.evaluate') ||
-                message.includes('Page.addScriptToEvaluateOnNewDocument') ||
-                message.includes('Protocol error'))) {
-              return;
+          // Neutralize CDP "inspector reads logged object" traps across the
+          // common console methods. CDP's Runtime.enable serializes console
+          // arguments, reading getters / walking prototypes on logged objects —
+          // disable-devtool-style scripts exploit this (log an object with a
+          // getter, then check if it fired) to detect the inspector and redirect
+          // away. The previous version guarded ONLY console.debug; detectors
+          // overwhelmingly use console.log (and dir/table), so the trap still
+          // fired. Sanitize args so getter/proxy traps can't trigger, and drop
+          // DevTools-protocol noise. Covers log/info/debug/dir/table — the
+          // methods detection uses; warn/error stay native (legit usage, and
+          // not the common vector).
+          const sanitizeArg = (arg) => {
+            // Strip Error objects with custom .stack getters (inspector reads .stack)
+            if (arg instanceof Error) {
+              const desc = Object.getOwnPropertyDescriptor(arg, 'stack');
+              if (desc && desc.get) return `${arg.name}: ${arg.message}`;
             }
-            // Sanitize args to neutralize CDP fingerprinting traps
-            const sanitized = args.map(arg => {
-              // Strip Error objects with custom .stack getters (CDP inspector reads .stack)
-              if (arg instanceof Error) {
-                const desc = Object.getOwnPropertyDescriptor(arg, 'stack');
-                if (desc && desc.get) return `${arg.name}: ${arg.message}`;
-              }
-              // Neutralize Proxy prototype traps (CDP inspector walks prototype chain)
-              if (arg !== null && typeof arg === 'object') {
-                try {
-                  const proto = Object.getPrototypeOf(arg);
-                  if (proto && proto !== Object.prototype && proto !== Array.prototype) {
-                    try { Object.keys(proto); } catch { return '[object Object]'; }
-                  }
-                } catch { return '[object Object]'; }
-              }
-              return arg;
-            });
-            return originalConsoleDebug.apply(this, sanitized);
+            // Neutralize Proxy/getter prototype traps (inspector walks the chain)
+            if (arg !== null && typeof arg === 'object') {
+              try {
+                const proto = Object.getPrototypeOf(arg);
+                if (proto && proto !== Object.prototype && proto !== Array.prototype) {
+                  try { Object.keys(proto); } catch { return '[object Object]'; }
+                }
+              } catch { return '[object Object]'; }
+            }
+            return arg;
           };
-
-        }, 'console error suppression');
+          const DEVTOOLS_NOISE = ['DevTools', 'Runtime.evaluate', 'Page.addScriptToEvaluateOnNewDocument', 'Protocol error'];
+          for (const method of ['log', 'info', 'debug', 'dir', 'table']) {
+            const original = console[method];
+            if (typeof original !== 'function') continue;
+            console[method] = function(...args) {
+              const message = args.join(' ');
+              if (typeof message === 'string' && DEVTOOLS_NOISE.some(n => message.includes(n))) {
+                return;
+              }
+              return original.apply(this, args.map(sanitizeArg));
+            };
+          }
+        }, 'console trap neutralization');
         
         // NOTE: The previous `location URL masking` Proxy was removed.
         // It wrapped window.location in a Proxy to return 'about:blank' when

package/lib/interaction.js

@@ -1322,6 +1322,154 @@ function createInteractionConfig(url, siteConfig = {}) {
  * const { generateRandomCoordinates } = require('./lib/interaction');
  * const pos = generateRandomCoordinates(1920, 1080, { preferEdges: true });
  */
+
+/**
+ * Find the first VISIBLE match for a selector across the main frame AND every
+ * child frame (iframes), polling until found or timeoutMs. page.waitForSelector
+ * only searches the main frame; players/ads commonly live in an iframe, so we
+ * walk page.frames() (main frame first, so a main-frame match wins). "Visible"
+ * is approximated by a non-null boundingBox (rules out display:none/zero-size).
+ * Returns an ElementHandle bound to its frame, or null on timeout.
+ */
+async function findVisibleInAnyFrame(page, selector, timeoutMs) {
+  const deadline = Date.now() + Math.max(0, timeoutMs);
+  for (;;) {
+    if (page.isClosed()) return null;
+    const mf = page.mainFrame();
+    const frames = [mf, ...page.frames().filter(f => f !== mf)]; // main frame first
+    for (const frame of frames) {
+      let el = null;
+      try {
+        el = await frame.$(selector);
+      } catch (e) {
+        // An invalid CSS selector (config typo) throws "... is not a valid
+        // selector" — a permanent error, so surface it instead of polling to a
+        // confusing not-found. Frame-detached / cross-origin hiccups have other
+        // messages and fall through to keep polling.
+        if (/is not a valid selector|not a valid or unsupported selector/i.test((e && e.message) || '')) {
+          throw new Error(`invalid CSS selector: ${(e && e.message) || e}`);
+        }
+        el = null;
+      }
+      if (el) {
+        let box = null;
+        try { box = await el.boundingBox(); } catch (_) { box = null; }
+        if (box) return el;            // present AND rendered
+        try { await el.dispose(); } catch (_) { /* not rendered yet — keep polling */ }
+      }
+    }
+    if (Date.now() >= deadline) return null;
+    await new Promise(r => setTimeout(r, 250));
+  }
+}
+
+/**
+ * Click a list of CSS selectors in order, reaching content via organic
+ * gesture/navigation instead of a direct page load. Each selector's first
+ * match is clicked; if the click navigates (an <a href> / form submit), we wait
+ * for it to commit, otherwise we wait a settle window for in-page actions
+ * (e.g. a player starting). The page's request interceptor stays attached
+ * throughout, so the post-click requests flow into the caller's normal
+ * filterRegex/dig matching — this function only performs the clicks.
+ *
+ * Missing elements are skipped (sites change markup); a click error never
+ * throws out of here. After a navigation, later selectors are queried against
+ * the NEW page (so e.g. "movie link" then "play button" works).
+ *
+ * @param {import('puppeteer').Page} page
+ * @param {string[]} selectors - CSS selectors, clicked in order
+ * @param {object} [options]
+ * @param {boolean} [options.realistic=false] - use humanClick (hover/tremor) vs elementHandle.click
+ * @param {number}  [options.waitMs=5000] - per click: max wait for the element to
+ *   appear+be visible (waitForSelector), AND the settle/nav window after the click
+ * @param {function} [options.ghostClick] - optional (x,y)=>Promise that performs a
+ *   ghost-cursor click (Bezier travel + press) at the element centre. Injected by the
+ *   caller so this module needn't depend on ghost-cursor.js (which depends on this one).
+ *   When provided it takes precedence over the humanClick/el.click paths.
+ * @param {boolean} [options.forceDebug=false]
+ * @returns {Promise<Array<{selector:string, clicked:boolean, reason?:string}>>}
+ */
+async function performTargetedClicks(page, selectors, options = {}) {
+  const { realistic = false, waitMs = 5000, forceDebug = false, ghostClick = null } = options;
+  const results = [];
+  if (!Array.isArray(selectors)) return results;
+
+  for (const raw of selectors) {
+    const selector = typeof raw === 'string' ? raw.trim() : '';
+    if (!selector) continue;
+    if (page.isClosed()) break;
+
+    // Wait for the element to appear AND be visible (up to waitMs), searching
+    // the main frame and every iframe — many targets (video players, lazy
+    // menus, post-consent buttons) are injected by JS after DOMContentLoaded
+    // and/or live inside an iframe, so a single main-frame query would miss
+    // them. Timeout → treat as not-found, skip.
+    let el;
+    try {
+      el = await findVisibleInAnyFrame(page, selector, waitMs);
+    } catch (selErr) {
+      const msg = (selErr && selErr.message) || '';
+      if (/invalid CSS selector|not a valid selector/i.test(msg)) {
+        // Config typo, not a transient miss — warn (visible without --debug).
+        console.warn(formatLogMessage('warn', `${INTERACTION_TAG} click_elements: invalid selector "${selector}" — skipping (${msg})`));
+        results.push({ selector, clicked: false, reason: 'invalid-selector' });
+      } else {
+        // Defensive: findVisibleInAnyFrame swallows transient/detached-frame
+        // errors internally, so this shouldn't fire — but never let an
+        // unexpected find error abort the remaining selectors.
+        if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: find failed for "${selector}": ${msg} — skipping`));
+        results.push({ selector, clicked: false, reason: msg || 'find-error' });
+      }
+      continue;
+    }
+    if (!el) {
+      if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: "${selector}" not visible (any frame) within ${waitMs}ms — skipping`));
+      results.push({ selector, clicked: false, reason: 'not-found' });
+      continue;
+    }
+
+    try {
+      // Bring it into view so coordinate clicks land (elementHandle.click also
+      // auto-scrolls, but humanClick clicks raw coordinates).
+      try { await el.evaluate(e => e.scrollIntoView({ block: 'center', inline: 'center' })); } catch (_) { /* detached/odd element */ }
+
+      // Arm a navigation wait BEFORE clicking so a link/submit is caught.
+      const navP = page.waitForNavigation({ waitUntil: 'domcontentloaded', timeout: waitMs + 3000 }).catch(() => {});
+
+      let box = null;
+      try { box = await el.boundingBox(); } catch (_) { box = null; }
+      if (typeof ghostClick === 'function' && box) {
+        // Ghost-cursor path: Bezier travel to the element centre + realistic
+        // press, matching the interact phase (caller-injected).
+        await ghostClick(box.x + box.width / 2, box.y + box.height / 2);
+      } else if (realistic && box) {
+        await humanClick(page, box.x + box.width / 2, box.y + box.height / 2, { realistic: true, forceDebug });
+      } else {
+        await el.click({ delay: 30 }); // trusted gesture; auto-scrolls + handles non-visible coords
+      }
+
+      // Resolve on whichever comes first: a committed navigation, or the settle
+      // window (in-page actions). Either way, requests fired in between are
+      // already captured by the caller's interceptor. Capture-and-clear the
+      // settle timer (cdp.js / interact pattern): when navP wins, an uncleared
+      // setTimeout would keep the event loop + closure alive for the full waitMs.
+      let settleTimer;
+      await Promise.race([
+        navP,
+        new Promise(r => { settleTimer = setTimeout(r, waitMs); })
+      ]).finally(() => clearTimeout(settleTimer));
+      results.push({ selector, clicked: true });
+      if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: clicked "${selector}"`));
+    } catch (err) {
+      if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements: click failed for "${selector}": ${err.message}`));
+      results.push({ selector, clicked: false, reason: err.message });
+    } finally {
+      try { await el.dispose(); } catch (_) { /* detached after navigation — fine */ }
+    }
+  }
+  return results;
+}
+
 module.exports = {
   // Main interaction functions
   performPageInteraction,
@@ -1337,5 +1485,8 @@ module.exports = {
   // hand-tremor + mouseup drift). Reused by lib/ghost-cursor.js so the ghost
   // coordinate click gets the same press realism as built-in content clicks.
   humanClick,
+  // Click specific CSS selectors in order (organic navigation / play-button /
+  // link clicking) — site config `click_elements`.
+  performTargetedClicks,
   generateRandomCoordinates
 };

package/lib/nettools.js

@@ -366,7 +366,10 @@ function validateWhoisAvailability() {
     };
   } catch (error) {
     try {
-      execSync('which whois', { encoding: 'utf8' });
+      // `which` is Unix-only; Windows uses `where`. Without this, an installed
+      // whois.exe whose `whois --version` errors (e.g. Sysinternals whois)
+      // would be false-negatived as unavailable on native Windows.
+      execSync(process.platform === 'win32' ? 'where whois' : 'which whois', { encoding: 'utf8' });
       validateWhoisAvailability._cached = {
         isAvailable: true,
         version: 'whois (version unknown)'

package/lib/validate_rules.js

@@ -1102,7 +1102,7 @@ function testDomainValidation() {
 const KNOWN_SITE_CONFIG_KEYS = new Set([
   'adblock_rules', 'blocked', 'bypass_cache', 'capture_popups',
   'capture_popups_max_depth', 'capture_popups_window_ms', 'cdp', 'cdp_specific',
-  'clear_sitedata', 'clear_sitedata_full_on_reload',
+  'clear_sitedata', 'clear_sitedata_full_on_reload', 'click_elements', 'click_wait',
   'cloudflare_bypass', 'cloudflare_max_retries', 'comments',
   'cloudflare_parallel_detection', 'cloudflare_phish', 'cloudflare_retry_on_error',
   'css_blocked', 'curl', 'cursor_mode', 'custom_headers', 'delay',
@@ -1119,7 +1119,7 @@ const KNOWN_SITE_CONFIG_KEYS = new Set([
   'js_redirect_timeout', 'localhost', 'max_redirects', 'openvpn', 'pihole',
   'output_regex',
   'plain', 'privoxy', 'proxy', 'proxy_bypass', 'proxy_debug', 'proxy_remote_dns',
-  'realistic_click', 'referrer_disable', 'referrer_headers', 'regex_and',
+  'realistic_click', 'redirect_first_party', 'referrer_disable', 'referrer_headers', 'regex_and',
   'reload', 'resourceTypes', 'screenshot', 'searchstring', 'searchstring_and',
   'socks5_bypass', 'socks5_debug', 'socks5_proxy', 'socks5_remote_dns',
   'subDomains',
@@ -1151,7 +1151,7 @@ const BOOLEAN_SITE_CONFIG_FIELDS = new Set([
   'grep', 'headful', 'ignore_similar', 'ignore_similar_ignored_domains',
   'interact', 'interact_clicks', 'interact_scrolling', 'isBrave', 'localhost',
   'pihole', 'plain', 'privoxy', 'proxy_debug', 'proxy_remote_dns',
-  'realistic_click', 'referrer_disable', 'regex_and', 'screenshot',
+  'realistic_click', 'redirect_first_party', 'referrer_disable', 'regex_and', 'screenshot',
   'searchstring_and', 'socks5_debug', 'socks5_remote_dns', 'thirdParty',
   'unbound', 'whois_retry_on_error', 'whois_retry_on_timeout', 'whois_use_fallback',
 ]);

package/nwss.1

@@ -138,12 +138,32 @@ Maximum concurrent site processing (1-50, overrides config/default).
 
 .TP
 .BR \--dns " \fIIP\fR[,\fIIP\fR...]"
-Nameserver(s) for the DNS pre-check AND nettools' dig \(em does not affect Chrome
-navigation or whois. A single address pins all queries to it; several are
-rotated per query (each leading once, the rest as failover) to spread the
-load. Routing dig through these avoids dig timeouts on a flaky system resolver
-silently dropping dig-gated domains. Overrides /etc/resolv.conf. Invalid
-entries are warned and dropped.
+Nameserver(s) for the DNS pre-check, nettools' dig, and \(em when they map to a
+known public DoH provider \(em Chrome's page navigation via DNS-over-HTTPS on
+direct connections (does not affect whois). A single address pins all queries
+to it; several are rotated per query (each leading once, the rest as failover)
+to spread the load. Routing dig through these avoids dig timeouts on a flaky
+system resolver silently dropping dig-gated domains. Overrides
+/etc/resolv.conf. Invalid entries are warned and dropped.
+.IP
+Chrome ignores \fB\--dns\fR for navigation and reads /etc/resolv.conf directly,
+so a broken system resolver could fail a domain the pre-check already resolved.
+When the \fB\--dns\fR servers match a known DoH provider (Google, Cloudflare,
+Quad9, OpenDNS, AdGuard, CleanBrowsing, DNS.SB, Mullvad \(em including
+malware/family/unfiltered variants), Chrome is launched with secure-DNS
+\fIautomatic\fR mode pointed at that provider, so navigation resolves through
+the same resolver. \fIautomatic\fR (not \fIsecure\fR) keeps a system-DNS
+fallback if DoH is unreachable. Skipped under a proxy or VPN (the exit/tunnel
+resolves); unmapped resolvers (custom/ISP, per-account providers, IPv6) fall
+back to system DNS with a warning.
+
+.TP
+.B \--doh-disable
+Opt out of the Chrome-navigation DoH pinning (default: off). Chrome then
+resolves page navigation via the system /etc/resolv.conf even when \fB\--dns\fR
+maps to a known provider; the pre-check and dig still honor \fB\--dns\fR. Use
+when DoH adds latency, is blocked on the network, or system-path resolution is
+specifically wanted.
 
 .TP
 .BR \--cleanup-interval " \fINUMBER\fR"
@@ -326,6 +346,14 @@ Integer. Number of random content-zone clicks per load, capped at 20 (default: 3
 .B realistic_click
 Boolean. Higher click fidelity: denser mouse approach (15 steps), sub-pixel hand-tremor micro-moves during the press, and a small mouseup drift so the mousedown and mouseup coordinates differ. For sites that score click realism. Costs roughly 80-120ms per click (default: false).
 
+.TP
+.B click_elements
+Array of CSS selectors. After the page loads, click each selector's first match \fBin order\fR (searched across the main frame and any iframe) \(em reaching content via organic navigation/gesture instead of a direct deep-load (which some sites JS-redirect away). Example: \fB["a[href*='/movie/']", ".play"]\fR clicks a movie link then a play button. The request interceptor stays attached, so the post-click page's requests are matched against \fBfilterRegex\fR/\fBdig\fR as usual; a click that navigates is followed and later selectors query the resulting page. Honors \fBrealistic_click\fR and \fBcursor_mode: "ghost"\fR (Bezier travel to the element); missing elements are skipped and never fail the scan.
+
+.TP
+.B click_wait
+Per click: maximum time in milliseconds to wait for the element to appear and be visible (\fBwaitForSelector\fR) AND the navigation/settle wait after the click (default: 5000; capped at half the per-URL timeout).
+
 .TP
 .B delay
 Milliseconds to wait after page load (default: 4000).
@@ -350,6 +378,10 @@ Boolean. Allow first-party request matching (default: false).
 .B thirdParty
 Boolean. Allow third-party request matching (default: true).
 
+.TP
+.B redirect_first_party
+Boolean (default: true). Whether redirect-destination domains (and chain hops) are treated as first-party. Set to \fBfalse\fR to keep redirect targets \fBthird-party\fR so \fBfilterRegex\fR/\fBdig\fR can match them under \fBthirdParty: true\fR \(em e.g. capturing the end domain of an ad/cloak redirect. The originally-scanned domain stays first-party either way; note this also un-excludes the redirect target's own same-domain resources.
+
 .TP
 .B fingerprint_protection
 Boolean or \fB"random"\fR. Enable browser fingerprint spoofing.

package/nwss.js

@@ -9,7 +9,7 @@ const fs = require('fs');
 const os = require('os');
 const psl = require('psl');
 const path = require('path');
-const { createRotatingResolver, createDnsCircuitBreaker, parseDnsServers, isNonExistenceError } = require('./lib/dns');
+const { createRotatingResolver, createDnsCircuitBreaker, parseDnsServers, isNonExistenceError, dohTemplatesForResolvers } = require('./lib/dns');
 const { createGrepHandler, validateGrepAvailability } = require('./lib/grep');
 const { compressMultipleFiles } = require('./lib/compress');
 const { parseSearchStrings, createResponseHandler } = require('./lib/searchstring');
@@ -17,6 +17,7 @@ const { applyAllFingerprintSpoofing, USER_AGENT_COLLECTIONS, CHROME_BUILD, CHROM
 const { formatRules, handleOutput, getFormatDescription } = require('./lib/output');
 // Curl functionality (replace searchstring curl handler)
 const { validateCurlAvailability, createCurlHandler: createCurlModuleHandler } = require('./lib/curl');
+const { runProcess } = require('./lib/spawn-async');
 // Rule validation
 const { validateRulesetFile, validateFullConfig, testDomainValidation, cleanRulesetFile, normalizeSiteConfig } = require('./lib/validate_rules');
 // CF Bypass
@@ -65,7 +66,7 @@ const SMART_CACHE_TAG = messageColors.processing('[SmartCache]');
 // log lines (start/completed). Same cyan as the other monitoring tags.
 const CONCURRENCY_TAG = messageColors.processing('[CONCURRENCY]');
 // Enhanced mouse interaction and page simulation
-const { performPageInteraction, createInteractionConfig, computeInteractionCeilingMs, performContentClicks, humanLikeMouseMove } = require('./lib/interaction');
+const { performPageInteraction, createInteractionConfig, computeInteractionCeilingMs, performContentClicks, humanLikeMouseMove, performTargetedClicks } = require('./lib/interaction');
 // Optional ghost-cursor support for advanced Bezier-based mouse movements
 const { createGhostCursor, ghostMove, ghostClick, ghostRandomMove, resolveGhostCursorConfig } = require('./lib/ghost-cursor');
 // Domain detection cache for performance optimization
@@ -241,6 +242,7 @@ if (fs.existsSync(NWSSCONFIG_PATH)) {
         resource_cleanup_interval: ['--cleanup-interval'],
         dns: ['--dns'],
         dns_cache: ['--dns-cache'],
+        doh_disable: ['--doh-disable'],
         cache_requests: ['--cache-requests'],
         dumpurls: ['--dumpurls'],
         remove_tempfiles: ['--remove-tempfiles'],
@@ -377,7 +379,13 @@ if (dnsCacheMode) enableDiskCache();
 // Filters NXDOMAIN / unresolvable hostnames in <100ms before paying the
 // ~5-15s Puppeteer + Cloudflare detection round-trip on each.
 const dnsPrecheckEnabled = !args.includes('--no-dns-precheck');
-const dnsPrecheckTimeoutMs = 2000;
+// 4s (was 2s): under a concurrent scan the c-ares UDP burst against the pinned
+// resolvers can take >2s to answer — a tight timeout false-counted those as
+// resolver errors and tripped the circuit breaker. A clean NXDOMAIN still
+// returns fast (the resolver answers immediately), so the higher ceiling only
+// costs time when the resolver is genuinely slow — exactly when we want to wait
+// rather than false-fail. Paired with the resolver's concurrency cap below.
+const dnsPrecheckTimeoutMs = 4000;
 
 // --show-dead-domains: collect hostnames that are definitively DEAD (do not
 // exist / unreachable) and print them at the end of the scan so they can be
@@ -442,6 +450,31 @@ const dnsResolver = createRotatingResolver({ servers: dnsServersOverride, forceD
 // system /etc/resolv.conf, which on a flaky setup times out and silently drops
 // dig-gated domains). Only when --dns is explicitly set.
 if (dnsServersOverride.length > 0) setDigResolvers(dnsServersOverride);
+// Pin Chrome's NAVIGATION resolver to the same providers via DoH. Chrome
+// ignores --dns for page loads and reads /etc/resolv.conf directly, so a broken
+// system resolver (e.g. one returning REFUSED) can ERR_NAME_NOT_RESOLVED a
+// domain the pre-check already resolved. Mapping --dns to the matching DoH
+// endpoint makes navigation use the pinned provider instead of resolv.conf.
+// 'automatic' mode (not 'secure') so Chrome still falls back to system DNS if
+// DoH is unreachable rather than failing the whole batch. Empty templates when
+// --dns is absent or maps to no known DoH provider — Chrome keeps system DNS.
+//
+// Applied ONLY to direct connections (see createBrowser): when a proxy or VPN
+// is active, the exit/tunnel does the resolution (remote DNS / pushed DNS), so
+// pinning local DoH would be redundant and could resolve geo-split domains to
+// the wrong region. In those modes Chrome defers to the proxy/VPN as before.
+// --doh-disable (default false) opts out of the Chrome DoH pinning entirely —
+// navigation falls back to system resolv.conf even when --dns maps to a known
+// provider. The pre-check and dig still honor --dns. Use it if DoH adds
+// unwanted latency, is blocked on the network, or you specifically want Chrome
+// to resolve via the system path.
+const dohDisabled = args.includes('--doh-disable');
+const chromeDoh = dnsServersOverride.length > 0
+  ? dohTemplatesForResolvers(dnsServersOverride)
+  : { templates: '', mapped: [], unmapped: [] };
+// anyVpnConfigured and the DoH startup log live inside the main IIFE below:
+// `sites` is destructured from the config later in module load, so referencing
+// it at this point in top-level evaluation would TDZ-throw.
 // Circuit breaker: if resolver errors dominate, suspend the pre-check for a
 // cooldown so a refusal storm doesn't keep hammering a broken resolver (sites
 // still load — a suspended pre-check just proceeds to navigation).
@@ -818,8 +851,12 @@ General Options:
 
 Validation Options:
   --cache-requests               Cache HTTP requests to avoid re-requesting same URLs within scan
-  --dns <ip[,ip,...]>            Resolver(s) for the DNS pre-check AND nettools' dig (not Chrome nav / whois).
-                                 One pins all queries to it; several rotate per query. Overrides /etc/resolv.conf.
+  --dns <ip[,ip,...]>            Resolver(s) for the DNS pre-check, nettools' dig, and — when they map to a
+                                 known DoH provider — Chrome's page navigation via DoH on direct connections
+                                 (skipped under proxy/VPN; not whois). Overrides /etc/resolv.conf.
+                                 One pins all queries to it; several rotate per query.
+  --doh-disable                  Opt out of the Chrome-navigation DoH pinning (default: off). Chrome then
+                                 resolves via system resolv.conf; --dns still pins the pre-check and dig.
   --dns-cache                    Persist dig/whois results to disk between runs (dig 20h / whois 36h TTL, 2000-entry cap each),
                                  plus the DNS pre-check negative cache (NXDOMAIN only, 12h TTL, .dnsnegcache)
   --no-dns-precheck              Disable per-URL DNS resolution check before page navigation.
@@ -894,6 +931,9 @@ Redirect Handling Options:
   source: true/false                           Save page source HTML after load
   firstParty: true/false                       Allow first-party matches (default: false)
   thirdParty: true/false                       Allow third-party matches (default: true)
+  redirect_first_party: true/false             Treat redirect-destination domains as first-party (default: true).
+                                              false keeps redirect targets third-party so filterRegex/dig can match
+                                              them (e.g. capturing an ad/cloak redirect's end domain)
   screenshot: true/false/\"force\"                Capture screenshot (true=on failure, \"force\"=always)
   headful: true/false                          Launch browser with GUI for this site
   fingerprint_protection: true/false/"random" Enable fingerprint spoofing: true/false/"random"
@@ -931,6 +971,9 @@ Advanced Options:
   interact_scrolling: true/false              Enable scrolling simulation (default: true)
   interact_clicks: true/false                 Enable element clicking simulation (default: false)
   interact_typing: true/false                 Enable typing simulation (default: false)
+  click_elements: ["sel1","sel2"]             After load, click these CSS selectors in order, main frame + iframes
+                                              (organic nav / play button). Honors realistic_click + cursor_mode "ghost"; missing skipped
+  click_wait: <milliseconds>                  Per-click: max wait for the element to appear + settle/nav after (default: 5000)
   cursor_mode: "ghost"                        Use ghost-cursor Bezier mouse (requires: npm i ghost-cursor)
   ghost_cursor_speed: <number>                Ghost-cursor speed multiplier (default: auto)
   ghost_cursor_hesitate: <milliseconds>       Delay before ghost-cursor clicks (default: 50)
@@ -1878,6 +1921,20 @@ function setupFrameHandling(page, forceDebug) {
     console.log(formatLogMessage('debug', `${POPUP_TAG} capture_popups set — launching with --disable-popup-blocking (non-gesture popunders allowed)`));
   }
 
+  // DoH gate: any VPN site disables Chrome DoH (the tunnel resolves). Computed
+  // here (not at module top) because `sites` is only initialized by this point.
+  // Read by createBrowser's launch args; the startup log reports the decision.
+  const anyVpnConfigured = Array.isArray(sites) && sites.some(s => s && (s.vpn || s.openvpn));
+  if (dnsServersOverride.length > 0 && !silentMode) {
+    if (dohDisabled) {
+      console.log(formatLogMessage('info', `Chrome DoH disabled via --doh-disable — navigation uses system resolv.conf; --dns still pins the pre-check and dig.`));
+    } else if (chromeDoh.templates) {
+      console.log(formatLogMessage('info', `Chrome navigation will use DoH (automatic) on direct connections: ${chromeDoh.templates}${anyVpnConfigured ? ' — VPN configured, so it defers to VPN resolution' : ' — deferred to proxy resolution on proxied sites'}`));
+    } else {
+      console.warn(formatLogMessage('warn', `--dns servers (${chromeDoh.unmapped.join(', ')}) have no known DoH endpoint — Chrome navigation stays on system resolv.conf; only the pre-check and dig are pinned. Known providers: Google, Cloudflare, Quad9, OpenDNS, AdGuard, CleanBrowsing, DNS.SB, Mullvad.`));
+    }
+  }
+
   /**
    * Creates a new browser instance with consistent configuration
    * Uses system Chrome and temporary directories to minimize disk usage
@@ -1985,6 +2042,19 @@ function setupFrameHandling(page, forceDebug) {
         '--use-mock-keychain',
         '--disable-client-side-phishing-detection',
         '--enable-features=NetworkService',
+        // DoH for Chrome's navigation resolver when --dns maps to a known
+        // provider — but ONLY on direct connections. A proxied launch carries
+        // a --proxy-server in extraArgs and does its own (remote) DNS; a VPN
+        // tunnels resolution. In both cases local DoH is redundant and could
+        // resolve geo-split domains to the wrong region, so it's skipped and
+        // Chrome defers to the proxy/VPN. 'automatic' keeps a system-DNS
+        // fallback if DoH is unreachable. Flags omitted when not applicable.
+        ...((chromeDoh.templates
+             && !dohDisabled
+             && !anyVpnConfigured
+             && !extraArgs.some(a => typeof a === 'string' && a.startsWith('--proxy-server')))
+          ? ['--dns-over-https-mode=automatic', `--dns-over-https-templates=${chromeDoh.templates}`]
+          : []),
         // Disk space controls - minimal cache for scanning workloads
         `--disk-cache-size=${CACHE_LIMITS.DISK_CACHE_SIZE}`,
         `--media-cache-size=${CACHE_LIMITS.MEDIA_CACHE_SIZE}`,
@@ -2468,10 +2538,18 @@ function setupFrameHandling(page, forceDebug) {
       page.setDefaultNavigationTimeout(Math.min(timeout, TIMEOUTS.DEFAULT_NAVIGATION));
       // Aggressive timeouts prevent hanging in Puppeteer 23.x while maintaining speed
       
-      page.on('console', (msg) => {
-        if (forceDebug && msg.type() === 'error') console.log(formatLogMessage('debug', `Console error: ${msg.text()}`));
-      });
-      
+      // Only attach a console listener under --debug. Registering ANY 'console'
+      // listener makes Puppeteer enable the CDP Runtime domain, which arms
+      // console-based automation/DevTools traps (e.g. disable-devtool logs an
+      // object with a getter and detects the inspector reading it → redirects
+      // away). The body is a no-op without forceDebug, so attaching it
+      // unconditionally armed that trap for zero benefit.
+      if (forceDebug) {
+        page.on('console', (msg) => {
+          if (msg.type() === 'error') console.log(formatLogMessage('debug', `Console error: ${msg.text()}`));
+        });
+      }
+
       // Add page crash handler
       page.on('error', (err) => {
         if (forceDebug) console.log(formatLogMessage('debug', `Page crashed: ${err.message}`));
@@ -3356,12 +3434,18 @@ function setupFrameHandling(page, forceDebug) {
         // (normalizeSiteConfig now coerces interact: 1 → true with a warning,
         // so by the time we get here both should be booleans — but keep the
         // diagnostic accurate for the truly-missing case.)
+        const hasClickElements = Array.isArray(siteConfig.click_elements) && siteConfig.click_elements.length > 0;
         const interactOn = siteConfig.interact === true;
         const clicksOn = siteConfig.interact_clicks === true;
-        if (!interactOn && !clicksOn) {
-          console.log(formatLogMessage('debug', `[popup] capture_popups is enabled but neither 'interact' nor 'interact_clicks' is — set BOTH to true to fire user-gesture clicks; without them, only popups opened via in-page redirects will capture`));
+        if (hasClickElements && (!interactOn || !clicksOn)) {
+          // click_elements fires its own trusted gesture clicks, so popups it
+          // triggers capture regardless of interact/interact_clicks. Don't warn
+          // "no clicks fire" — surface the random-click coverage gap instead.
+          console.log(formatLogMessage('debug', `[popup] capture_popups: click_elements supplies targeted gesture clicks (popups they trigger WILL capture). interact=${interactOn}, interact_clicks=${clicksOn} — enable both for random content-zone click coverage of overlay popunders too`));
+        } else if (!interactOn && !clicksOn) {
+          console.log(formatLogMessage('debug', `[popup] capture_popups is enabled but neither 'interact' nor 'interact_clicks' is — set BOTH to true to fire user-gesture clicks; without them, only popups opened via in-page redirects (or click_elements) will capture`));
         } else if (!interactOn) {
-          console.log(formatLogMessage('debug', `[popup] capture_popups is enabled but 'interact' is not — set interact: true to enable the interaction loop (interact_clicks is already set); without it, no fake clicks fire`));
+          console.log(formatLogMessage('debug', `[popup] capture_popups is enabled but 'interact' is not — set interact: true to enable the interaction loop (interact_clicks is already set); without it, no random fake clicks fire`));
         } else if (!clicksOn) {
           console.log(formatLogMessage('debug', `[popup] capture_popups is enabled but 'interact_clicks' is not — set interact_clicks: true to enable element-targeted clicks; without it, only random content-zone clicks fire and may miss overlay-based popunders`));
         }
@@ -4433,6 +4517,77 @@ function setupFrameHandling(page, forceDebug) {
               if (forceDebug) console.log(formatLogMessage('debug', `Navigation timeout — proceeding with partially-loaded page for ${currentUrl}`));
               navigationResult = { finalUrl: partialUrl, redirected: false, redirectChain: [currentUrl], originalUrl: currentUrl, redirectDomains: [], httpStatus: null, cfRay: null };
             }
+          } else if (navErr.message.includes('ERR_TOO_MANY_REDIRECTS')) {
+            // Redirect-cloaking chain exceeded Chrome's ~20-hop per-navigation
+            // ceiling, so goto() rejected. Two recovery paths — they cover
+            // opposite cases run-to-run, so try both:
+            //   1. Browser ride-through (free): a JS/meta hop on a committed
+            //      intermediate page resets Chrome's counter and carries the page
+            //      to the end site on its own. Check if it already happened, else
+            //      wait briefly for it.
+            //   2. curl-resolve (fallback, only if the page parked on
+            //      chrome-error): curl follows the chain (it gets the real chain,
+            //      not headless Chrome's endless loop) to the JS-handoff page;
+            //      navigating there directly is a SHORT hop that reaches the end
+            //      site. Skipped under proxy/VPN — curl runs DIRECT from the host
+            //      and would leak the real IP / resolve from the wrong network.
+            // If neither reaches a real page, keep the chain requests already
+            // captured (grouped under the original URL, never chrome-error).
+            let landedUrl = '';
+            const isRealPage = (u) => !!u && /^https?:\/\//.test(u) && !u.startsWith('chrome-error://') && u !== currentUrl;
+
+            // 1) Browser ride-through — may have completed during goto(); if not,
+            //    wait for the next navigation(s) to carry it through.
+            try { if (!page.isClosed() && isRealPage(page.url())) landedUrl = page.url(); } catch {}
+            for (let r = 0; r < 3 && !landedUrl; r++) {
+              try {
+                await page.waitForNavigation({ waitUntil: 'domcontentloaded', timeout: 8000 });
+                if (!page.isClosed() && isRealPage(page.url())) landedUrl = page.url();
+              } catch { break; } // no further navigation — stop waiting
+            }
+            if (landedUrl && forceDebug) console.log(formatLogMessage('debug', `Too many redirects — browser rode through to ${landedUrl} for ${currentUrl}`));
+
+            // 2) curl-resolve fallback — only if still parked (no ride-through).
+            //    Opt-in via the site's `curl` option: if you didn't enable curl
+            //    in the config, the scanner won't shell out to it here either
+            //    (consistent with the content-analysis `curl` gate).
+            if (!landedUrl) {
+              const curlResolveOk = siteConfig.curl === true && !needsProxy(siteConfig) && !anyVpnConfigured && validateCurlAvailability().isAvailable;
+              if (curlResolveOk) {
+                let resolvedUrl = '';
+                try {
+                  const curlUa = USER_AGENT_COLLECTIONS.get((siteConfig.userAgent || 'chrome').toLowerCase())
+                    || 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36';
+                  const cr = await runProcess('curl', ['-sL', '--max-redirs', '50', '--max-time', '20', '-o', '/dev/null', '-A', curlUa, '-w', '%{url_effective}', currentUrl], { timeout: 22000, maxStdout: 4096 });
+                  const u = (cr.stdout || '').trim();
+                  if (cr.code === 0 && /^https?:\/\//.test(u) && u !== currentUrl) resolvedUrl = u;
+                } catch (_) { /* curl failed */ }
+                if (resolvedUrl) {
+                  if (forceDebug) console.log(formatLogMessage('debug', `Too many redirects — curl resolved the chain to ${resolvedUrl}; navigating there directly for ${currentUrl}`));
+                  // Navigate to the resolved endpoint; the streaming/embed end page
+                  // often never reaches DOM-ready, so the goto may throw — either
+                  // way it navigated, so adopt page.url().
+                  try { navigationResult = await navigateWithRedirectHandling(page, resolvedUrl, siteConfig, gotoOptions, forceDebug, formatLogMessage); } catch (_) { /* timed out — use page.url() below */ }
+                  try { if (!page.isClosed() && page.url() && !page.url().startsWith('chrome-error://')) landedUrl = page.url(); } catch {}
+                } else if (forceDebug) {
+                  console.log(formatLogMessage('debug', `Too many redirects — no ride-through and curl could not resolve; keeping chain captures for ${currentUrl}`));
+                }
+              } else if (forceDebug) {
+                const why = siteConfig.curl !== true ? 'curl not enabled (curl:false)'
+                  : (needsProxy(siteConfig) || anyVpnConfigured) ? 'proxy/VPN active'
+                  : 'curl unavailable';
+                console.log(formatLogMessage('debug', `Too many redirects — no ride-through and curl-resolve skipped (${why}); keeping chain captures for ${currentUrl}`));
+              }
+            }
+
+            // navigateWithRedirectHandling may already have set navigationResult
+            // (clean curl path). Otherwise build a partial from where we landed —
+            // the end site if we rode through / curl'd, else the original URL with
+            // the chain requests already captured.
+            if (!navigationResult) {
+              const fu = landedUrl || currentUrl;
+              navigationResult = { finalUrl: fu, redirected: fu !== currentUrl, redirectChain: [currentUrl, fu], originalUrl: currentUrl, redirectDomains: [], httpStatus: null, cfRay: null };
+            }
           } else {
             throw navErr;
           }
@@ -4478,17 +4633,26 @@ function setupFrameHandling(page, forceDebug) {
           redirectHistory.add(currentUrl);
           redirectHistory.add(finalUrl);
 
-          // Add redirect destination to first-party domains immediately
-          if (finalDomain) {
-            firstPartyDomains.add(finalDomain);
-          }
-          
-          // Also add any intermediate redirect domains as first-party
-          if (redirectDomains && redirectDomains.length > 0) {
-            redirectDomains.forEach(domain => {
-              const rootDomain = safeGetDomain(`http://${domain}`, false);
-              if (rootDomain) firstPartyDomains.add(rootDomain);
-            });
+          // Add redirect destination (and intermediates) to first-party domains
+          // so the landed site's own resources aren't captured as third-party.
+          // Opt out with redirect_first_party:false — then redirect targets stay
+          // THIRD-PARTY and become eligible for filterRegex/dig under
+          // thirdParty:true (e.g. capturing an ad/cloak redirect's end domain).
+          // The originally-scanned domain (added earlier) stays first-party.
+          const redirectsAreFirstParty = siteConfig.redirect_first_party !== false;
+          if (redirectsAreFirstParty) {
+            if (finalDomain) {
+              firstPartyDomains.add(finalDomain);
+            }
+            // Also add any intermediate redirect domains as first-party
+            if (redirectDomains && redirectDomains.length > 0) {
+              redirectDomains.forEach(domain => {
+                const rootDomain = safeGetDomain(`http://${domain}`, false);
+                if (rootDomain) firstPartyDomains.add(rootDomain);
+              });
+            }
+          } else if (forceDebug) {
+            console.log(formatLogMessage('debug', `redirect_first_party:false — keeping redirect target ${finalDomain} third-party for ${currentUrl}`));
           }
           
           if (originalDomain !== finalDomain) {
@@ -4745,6 +4909,45 @@ function setupFrameHandling(page, forceDebug) {
       }
       }
 
+      // Targeted clicks: after load, click configured CSS selectors in order
+      // (e.g. a movie link, then a play button) to reach content via organic
+      // navigation/gesture instead of a direct deep-load (which some sites
+      // JS-redirect away). The request interceptor stays attached, so the
+      // post-click page's requests flow into the same filterRegex/dig matching.
+      // Reuses realistic_click for a genuine trusted gesture. Runs before the
+      // delay/interact phase so those operate on the resulting page.
+      if (Array.isArray(siteConfig.click_elements) && siteConfig.click_elements.length > 0 && page && !page.isClosed()) {
+        // If ghost-cursor is enabled for this site (cursor_mode:"ghost" or
+        // --ghost-cursor), route the targeted clicks through it — Bezier travel
+        // to the element + realistic press — matching the interact phase.
+        // Injected so interaction.js needn't require ghost-cursor.js (circular).
+        // Falls back to performTargetedClicks' humanClick/el.click when ghost is
+        // off or the package isn't installed (resolveGhostCursorConfig → null).
+        let ghostClicker = null;
+        const tcGhostCfg = resolveGhostCursorConfig(siteConfig, globalGhostCursor, forceDebug);
+        if (tcGhostCfg) {
+          const tcCursor = createGhostCursor(page, { forceDebug });
+          if (tcCursor) {
+            ghostClicker = (x, y) => ghostClick(tcCursor, { x, y }, {
+              hesitate: tcGhostCfg.hesitate,
+              page,
+              realistic: siteConfig.realistic_click === true,
+              forceDebug
+            });
+          }
+        }
+        try {
+          await performTargetedClicks(page, siteConfig.click_elements, {
+            realistic: siteConfig.realistic_click === true,
+            waitMs: Math.min(Number(siteConfig.click_wait) || 5000, Math.floor(timeout / 2)),
+            ghostClick: ghostClicker,
+            forceDebug
+          });
+        } catch (clickErr) {
+          if (forceDebug) console.log(formatLogMessage('debug', `${INTERACTION_TAG} click_elements phase failed for ${currentUrl}: ${clickErr.message}`));
+        }
+      }
+
       const delayMs = siteConfig.delay || TIMEOUTS.DEFAULT_DELAY;
 
       // Optimized delays for Puppeteer 23.x performance
@@ -4761,6 +4964,13 @@ function setupFrameHandling(page, forceDebug) {
       const actualDelay = siteConfig.delay_uncapped === true
         ? Math.min(delayMs, Math.floor(timeout / 2))
         : Math.min(delayMs, TIMEOUTS.NETWORK_IDLE);
+      // Surface the clamp — otherwise `delay: 48000` silently running as 29000
+      // (timeout/2) looks like the flag was ignored. The per-URL budget already
+      // reserves the full `delay`, so the lever to honor it is a larger timeout.
+      if (forceDebug && actualDelay < delayMs) {
+        const ceiling = siteConfig.delay_uncapped === true ? 'timeout/2; raise timeout to lift' : 'default 2s cap; set delay_uncapped:true to lift';
+        console.log(formatLogMessage('debug', `delay ${delayMs}ms clamped to ${actualDelay}ms (${ceiling}) for ${currentUrl}`));
+      }
 
       // Build delay promise (networkIdle + delay + optional flowProxy delay)
       const delayPromise = (async () => {
@@ -5033,6 +5243,21 @@ function setupFrameHandling(page, forceDebug) {
         
       let reloadSuccess = false;
 
+      // page.reload() can't carry a referer; when referrer_headers is set,
+      // re-navigate to the current URL with it so referer-gated embeds keep
+      // serving across the reload:N loop (the initial goto carries the referer,
+      // but reload() drops it). Nav-only scope — subresources keep their normal
+      // page-origin referer (unlike setExtraHTTPHeaders, which would force the
+      // referer onto every request and can break embeds whose subresources
+      // expect own-origin). A static referrer_headers string is identical each
+      // reload; random/mixed modes pick a fresh value per reload.
+      const reloadReferer = siteConfig.referrer_headers
+        ? getReferrerForUrl(currentUrl, siteConfig.referrer_headers, siteConfig.referrer_disable, forceDebug)
+        : '';
+      const reloadOrReferredGoto = (opts) => reloadReferer
+        ? page.goto(page.url(), { ...opts, referer: reloadReferer })
+        : page.reload(opts);
+
   // Skip force reload if browser seems unhealthy
   const skipForceReload = i > 2; // After 2 attempts, skip force reload
       
@@ -5055,7 +5280,7 @@ function setupFrameHandling(page, forceDebug) {
           await raceWithTimer(page.setCacheEnabled(false), 'Cache disable timeout', 8000);
 
             // Use networkidle2 for force reload to better detect when page is actually loaded
-            await page.reload({ waitUntil: 'networkidle2', timeout: Math.min(timeout, 15000) });
+            await reloadOrReferredGoto({ waitUntil: 'networkidle2', timeout: Math.min(timeout, 15000) });
 
           // Timeout-protected cache enable
           await raceWithTimer(page.setCacheEnabled(true), 'Cache enable timeout', 8000);
@@ -5094,7 +5319,7 @@ function setupFrameHandling(page, forceDebug) {
         ? { waitUntil: 'domcontentloaded', timeout: 10000 }  // Simpler after failures
         : { waitUntil: 'networkidle2', timeout: 15000 };     // Full wait first time
       
-      await page.reload(reloadOptions);
+      await reloadOrReferredGoto(reloadOptions);
             
           if (forceDebug) console.log(formatLogMessage('debug', `Standard reload #${i} completed for ${currentUrl}`));
         } catch (standardReloadErr) {
@@ -5954,10 +6179,24 @@ function setupFrameHandling(page, forceDebug) {
      const INTERACTION_OVERHEAD_MS = interactionOnForTask
        ? computeInteractionCeilingMs(createInteractionConfig(task.url, task.config))
        : 0;
+     // click_elements runs ONCE after load (before the delay/interact/reload
+     // phases): N selectors, each a settle/nav wait (click_wait, capped at
+     // timeout/2 — mirror the call site) plus ~2s for scroll + the click action
+     // (ghost Bezier travel is the slowest). Budget it so a heavy click chain
+     // can't trip the per-URL ceiling before the work that follows it. Not
+     // multiplied by reloadCount — the click phase is one-time.
+     const clickEls = Array.isArray(task.config.click_elements)
+       ? task.config.click_elements.filter(s => typeof s === 'string' && s.trim())
+       : [];
+     const clickWaitMs = clickEls.length
+       ? Math.min(Number(task.config.click_wait) || 5000, Math.floor((task.config.timeout || 35000) / 2))
+       : 0;
+     const CLICK_ELEMENTS_OVERHEAD_MS = clickEls.length * (clickWaitMs + 2000);
      const PER_URL_TIMEOUT_MS = Math.max(
        75000,
        (task.config.timeout || 35000)
          + ((task.config.delay || 0) + INTERACTION_OVERHEAD_MS) * (1 + reloadCount)
+         + CLICK_ELEMENTS_OVERHEAD_MS
          + 30000
      );
      // Feed the hang-check restart so it never escalates before this URL's own

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fanboynz/network-scanner",
-  "version": "3.3.0",
+  "version": "3.4.0",
   "description": "A Puppeteer-based network scanner for analyzing web traffic, generating adblock filter rules, and identifying third-party requests. Features include fingerprint spoofing, Cloudflare bypass, content analysis with curl/grep, and multiple output formats.",
   "main": "nwss.js",
   "scripts": {