@fanboynz/network-scanner 3.1.2 → 3.2.0

package/CHANGELOG.md

@@ -2,7 +2,30 @@
 
 All notable changes to the Network Scanner (nwss.js) project.
 
-## [3.1.1] - 2026-05-30
+## [3.2.0] - 2026-06-04
+
+### Added
+- **`output_regex`** site option — a per-site regex whose capture group 1 (or whole match) becomes the rule body, so output can be a path-prefix rule like `||host/script/` instead of `||host^`. Collapses randomized filenames under a stable path into one rule and lets you block a folder on a host that also serves legit content; falls back to `||host^` when the regex doesn't match. Adblock-only — domain-based formats (dnsmasq/unbound/pi-hole/hosts/plain) emit the bare host. Compiled once per pattern (memoized) and validated at config load.
+- **dig resolver failover** — `digLookup` now fails over through the `--dns` resolvers on timeout / no-reply / `REFUSED` / `SERVFAIL` (up to 3 attempts, `+time=2 +tries=1` each), matching the resilience the whois retry and DNS pre-check rotation already had. With no `--dns`, the system-resolver path keeps dig's native `resolv.conf` rotation unchanged.
+
+### Changed
+- **Ghost-cursor coordinate clicks now use the same realistic press as the built-in content clicks** (`humanClick`): hover dwell + mousedown/hold/mouseup, plus hand-tremor during the hold and a mouseup drift (so mousedown ≠ mouseup coordinates) when `realistic_click` is set — replacing a 0ms `page.mouse.click`.
+- **Ghost-cursor clicks honor `interact_click_count`** (default 3, cap 20) instead of firing a single click — ad SDKs often swallow the 1st/2nd click as warmup. The bezier movement loop reserves part of `ghost_cursor_duration` for the clicks (raise the duration to fit more; the default 2000ms fits ~1 realistic click).
+- **`dig` success is judged by RCODE, not stderr** — a dig that prints a transient `communications error` warning but still returns a valid `ANSWER SECTION` is no longer discarded.
+- **dig-only configs skip the whois root-domain parse** per request (small per-request saving when no `whois`/`whois-or` is configured).
+
+### Fixed
+- **`max_redirects: 0`** now means "follow none" instead of silently becoming 10 (the `|| 10` falsy-zero bug in `nwss.js` and `lib/redirect.js`).
+- **A `REFUSED`/`SERVFAIL` dig that exhausts all resolvers returns failure** so it isn't cached — a transient resolver-side error no longer poisons a domain for the cache TTL.
+- **Ghost-cursor coordinate click no longer reports false success** — it returned `true` (and logged "Clicked") even when the click was silently skipped for lack of a page; it now returns `false` and logs the skip.
+
+### Removed
+- **`follow_redirects`** site option — documented in `--help`, the man page, the README, and example configs but never wired to any runtime behavior; removed from the docs. Use `max_redirects` instead (`0` = follow none).
+
+### Security
+- **dig argv-injection guard** — `digLookup` rejects non-hostname-shaped input before shelling out. `dig` has no `--` end-of-options marker (unlike whois) and parses `@`/`-`/`+`-leading argv tokens as options, so a crafted "domain" like `@evil-resolver` (redirects the query to an arbitrary server) or `-f /path` (reads a file as a query batch) is now rejected — out-of-charset or dash-leading values fall back to no-match.
+
+## [3.1.2] - 2026-05-30
 
 ### Changed
 - **Fingerprint identity pinned to Stable Chrome 148**, not whatever Chrome-for-Testing puppeteer bundles (currently 149, ahead of Stable). The spoof must blend with the real-world population; claiming an unreleased build is itself a tell. The Chrome major + build (`CHROME_BUILD`) + GREASE brand (`CHROME_GREASE_BRAND`) are now single constants — see `lib/fingerprint.md`.

package/CLAUDE.md

@@ -6,7 +6,7 @@ Puppeteer-based network scanner for analyzing web traffic, generating adblock fi
 
 - `nwss.js` — Main entry point (~5,800 lines). CLI args, URL processing, orchestration.
 - `config.json` — Default scan configuration (sites, filters, options).
-- `lib/` — 32 focused, single-purpose modules:
+- `lib/` — 33 focused, single-purpose modules:
   - `fingerprint.js` — Bot detection evasion (device/GPU/timezone spoofing)
   - `cloudflare.js` — Cloudflare challenge detection and solving
   - `browserhealth.js` — Memory management and browser lifecycle
@@ -14,6 +14,7 @@ Puppeteer-based network scanner for analyzing web traffic, generating adblock fi
   - `ghost-cursor.js` — Bezier-curve cursor pathing for human-like mouse movement
   - `smart-cache.js` — Multi-layer caching with persistence
   - `nettools.js` — WHOIS/dig integration
+  - `dns.js` — DNS pre-check resolver: multi-nameserver rotation + `--dns` override (pre-check only; not Chrome/dig)
   - `output.js` — Multi-format rule output (adblock, dnsmasq, unbound, pihole, etc.)
   - `proxy.js` — SOCKS5/HTTP proxy support
   - `socks-relay.js` — Local SOCKS proxy relay/chain helper

package/README.md

@@ -17,6 +17,7 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 - Subdomain handling (collapse to root or full subdomain)
 - Optionally match only first-party, third-party, or both
 - Enhanced redirect handling with JavaScript and meta refresh detection
+- Capture and drive popup/popunder chains (`capture_popups` + `interact_popups`) so domains reachable only via a clicked popup still match
 - Per-site proxy routing (SOCKS5, SOCKS4, HTTP, HTTPS) with pre-flight health checks
 
 ---
@@ -50,7 +51,6 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 
 | Argument                  | Description |
 |:---------------------------|:------------|
-| `--verbose`                 | Force verbose mode globally |
 | `--debug`                   | Force debug mode globally |
 | `--silent`                  | Suppress normal console logs |
 | `--titles`                  | Add `! <url>` title before each site's group |
@@ -66,7 +66,7 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--use-puppeteer-core`      | Use `puppeteer-core` with system Chrome instead of bundled Chromium |
 | `--use-obscura`             | Connect to running Obscura CDP server (`ws://127.0.0.1:9222` or `OBSCURA_WS` env). Skips fingerprint injection — Obscura provides built-in stealth |
 | `--load-extension <path>`   | Load unpacked Chrome extension from directory (can be used multiple times) |
-| `--dns-cache`               | Persist dig/whois results to disk between runs (20hr TTL, 2000-entry cap each, `.digcache`/`.whoiscache`). Disk writes are atomic (tmp + rename); corrupt cache files are detected on load with a `[dns-cache]` warn line and reset cleanly. |
+| `--dns-cache`               | Persist dig/whois results to disk between runs (20hr TTL, 2000-entry cap each, `.digcache`/`.whoiscache`), **plus** the DNS pre-check negative cache (NXDOMAIN/ENODATA only — never resolver errors — 12h TTL, `.dnsnegcache`) so known-dead hosts aren't re-resolved next run. Disk writes are atomic (tmp + rename); corrupt cache files are detected on load with a `[dns-cache]` warn line and reset cleanly. |
 | `--no-dns-precheck`         | Disable per-URL DNS resolution check before page navigation. By default, hosts that dig/whois have already proven live (within the 20hr cache TTL) skip their c-ares pre-check via a positive-resolution index. |
 | `--block-ads=<files>`       | Block ads using EasyList format rules (comma-separated: `easylist.txt,easyprivacy.txt`) |
 | `--cdp`                     | Enable Chrome DevTools Protocol logging (now per-page if enabled) |
@@ -76,6 +76,8 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--help`, `-h`              | Show this help menu |
 | `--version`                 | Show script version |
 | `--max-concurrent <number>` | Maximum concurrent site processing (1-50, overrides config/default) |
+| `--dns <ip[,ip,...]>` | Resolver(s) for the DNS pre-check **and** nettools' `dig` (one pins, several rotate per query; overrides `/etc/resolv.conf`). Does not affect Chrome navigation or `whois`. Useful when the system resolver is flaky and `dig`-gated domains time out |
+| `--show-dead-domains` | At end of scan, list hostnames that did not resolve / were unreachable (`NXDOMAIN`/`ENODATA` + `ERR_NAME_NOT_RESOLVED`/`ERR_ADDRESS_UNREACHABLE`). Excludes blocks/timeouts (those mean the domain is alive). For pruning dead URLs. |
 | `--cleanup-interval <number>` | Browser restart interval in URLs processed (1-1000, overrides config/default) |
 
 ### Validation Options
@@ -152,6 +154,7 @@ Example:
 | `userAgent`          | `chrome`, `chrome_mac`, `chrome_linux`, `firefox`, `firefox_mac`, `firefox_linux`, `safari` | - | User agent for page |
 | `filterRegex`        | String or Array | `.*` | Regex or list of regexes to match requests |
 | `regex_and`          | Boolean | `false` | Use AND logic for multiple filterRegex patterns - ALL patterns must match the same URL |
+| `output_regex`       | String | — | Regex applied to each matched URL to build the rule body: capture group 1 (or whole match) becomes `\|\|<capture>` instead of `\|\|host^`. E.g. `^https?:\/\/([^\/]+\/[^\/]+\/)` turns `https://host.com/script/abc.js` into `\|\|host.com/script/`. The capture must include the host. No match → falls back to `\|\|host^`. Adblock-only; domain formats (dnsmasq/pihole/hosts/plain) emit the bare host |
 | `comments`           | String or Array | - | String of comments or references |
 | `resourceTypes`      | Array | `["script", "xhr", "image", "stylesheet"]` | What resource types to monitor |
 | `reload`             | Integer | `1` | Number of times to reload page |
@@ -176,8 +179,7 @@ Example:
 
 | Field                | Values | Default | Description |
 |:---------------------|:-------|:-------:|:------------|
-| `follow_redirects`   | Boolean | `true` | Follow redirects to new domains |
-| `max_redirects`      | Integer | `10` | Maximum number of redirects to follow |
+| `max_redirects`      | Integer | `10` | Maximum number of redirects to follow (`0` = follow none) |
 | `js_redirect_timeout` | Milliseconds | `5000` | Time to wait for JavaScript redirects |
 | `detect_js_patterns` | Boolean | `true` | Analyze page source for redirect patterns |
 | `redirect_timeout_multiplier` | Number | `1.5` | Increase timeout for redirected URLs |
@@ -279,6 +281,8 @@ When a page redirects to a new domain, first-party/third-party detection is base
 | `interact_duration`  | Milliseconds | `2000` | Duration of interaction simulation |
 | `interact_scrolling` | Boolean | `true` | Enable scrolling simulation |
 | `interact_clicks`    | Boolean | `false` | Enable element clicking simulation |
+| `interact_click_count` | Integer | `3` | Number of random content-zone clicks per load (capped at 20). Default 3 = primary + 2 backups, since ad SDKs sometimes suppress the 1st/2nd click as warmup |
+| `realistic_click`    | Boolean | `false` | Higher click fidelity: denser mouse approach (15 steps), ±1px hand-tremor micro-moves during the press, and ±1.5px mouseup drift (so mousedown≠mouseup coords) — for sites that score click realism. Costs ~80–120ms/click |
 | `interact_typing`    | Boolean | `false` | Enable typing simulation |
 | `interact_intensity` | String | `"medium"` | Interaction simulation intensity: "low", "medium", "high" |
 | `cursor_mode`        | `"ghost"` | - | Use ghost-cursor Bezier mouse movements (requires `npm i ghost-cursor`) |
@@ -295,6 +299,21 @@ When a page redirects to a new domain, first-party/third-party detection is base
 | `ignore_similar_threshold` | Integer | - | Override global similarity threshold for this site |
 | `ignore_similar_ignored_domains` | Boolean | - | Override global `ignore_similar_ignored_domains` for this site |
 
+### Popup Capture Options
+
+Capture (and optionally drive) the popup/popunder windows that ad and redirect
+scripts open, so domains reachable only via a popup chain still match `filterRegex`.
+The same `filterRegex` applies to the whole chain — it must contain every pattern
+you expect along it. Popup capture only fires when the main page is actually
+clicking, so set `interact: true` **and** `interact_clicks: true` as well.
+
+| Field                | Values | Default | Description |
+|:---------------------|:-------|:-------:|:------------|
+| `capture_popups`     | Boolean | `false` | Capture popup windows opened during the scan and evaluate their landing URL + in-popup requests against `filterRegex`/`dig`/`whois` (requires `interact` + `interact_clicks` to fire user-gesture clicks) |
+| `interact_popups`    | Boolean | `false` | Mouse-click inside captured popups (3 content-zone clicks) so the chain cascades to its next redirect/ad. Requires `capture_popups`. Clicks popups up to `capture_popups_max_depth − 1` (the deepest captured popup is observed, not clicked) |
+| `capture_popups_max_depth` | Integer | `4` | Max popup-chain depth to capture (`site → p1 → p2 → p3 → destination`). Each extra level multiplies popups + time |
+| `capture_popups_window_ms` | Integer | `5000` | Per-popup capture window (ms) before the popup is auto-closed |
+
 ### VPN Options
 
 Route traffic through a VPN for specific sites. Requires `sudo` privileges. The VPN connection is established before scanning and torn down after the site completes.
@@ -596,8 +615,11 @@ node nwss.js --max-concurrent 12 --cleanup-interval 300 -o rules.txt
 {
   "url": "https://anti-bot-site.com",
   "interact": true,
+  "interact_clicks": true,
   "cursor_mode": "ghost",
-  "ghost_cursor_duration": 3000,
+  "realistic_click": true,
+  "interact_click_count": 3,
+  "ghost_cursor_duration": 5000,
   "ghost_cursor_speed": 1.2,
   "fingerprint_protection": "random",
   "filterRegex": "tracking|analytics",
@@ -610,6 +632,12 @@ Or enable globally via CLI:
 node nwss.js --ghost-cursor --debug -o rules.txt
 ```
 
+**Ghost-cursor clicks.** The cursor moves with `cursor_mode: "ghost"`, but it only *clicks* when both `interact: true` **and** `interact_clicks: true` are set (same rule as the built-in path). Click behavior:
+
+- `realistic_click: true` — each press adds hand-tremor during the hold and a mouseup drift, so `mousedown` ≠ `mouseup` coordinates (the press is routed through the same `humanClick` the built-in content clicks use).
+- `interact_click_count` — number of clicks per load (default `3`, capped at `20`). The default of 3 matters because some ad SDKs swallow the 1st/2nd click as warmup.
+- **Duration vs. clicks:** realistic clicks take ~600–700ms each, and the bezier movement loop reserves up to **half** of `ghost_cursor_duration` for them. So the default `ghost_cursor_duration: 2000` only fits **~1 click** — raise it to roughly `interact_click_count × 700 + movement` (e.g. `5000`–`8000`) to fit all of them.
+
 > **Note:** ghost-cursor is an optional dependency. Install with `npm install ghost-cursor`. If not installed, the scanner falls back to the built-in mouse simulation automatically.
 
 #### E-commerce Site Scanning

package/eslint.config.mjs

@@ -2,5 +2,17 @@ import globals from "globals";
 import { defineConfig } from "eslint/config";
 
 export default defineConfig([
-  { files: ["**/*.{js,mjs,cjs}"], languageOptions: { globals: globals.browser } },
+  {
+    files: ["**/*.{js,mjs,cjs}"],
+    // Node globals (require/module/process/Buffer/...) plus browser globals
+    // (document/window/navigator) — the latter are referenced inside
+    // page.evaluate() callbacks that eslint parses as part of the file.
+    languageOptions: { globals: { ...globals.node, ...globals.browser } },
+    // Catch undefined-variable references statically. node --check only
+    // validates syntax, so an orphaned identifier (e.g. a const that was
+    // removed while a usage remained) passes parsing but throws
+    // ReferenceError at runtime only when that branch executes. no-undef
+    // turns that whole class into a build-time failure.
+    rules: { "no-undef": "error" },
+  },
 ]);

package/lib/browserhealth.js

@@ -7,12 +7,11 @@ const { formatLogMessage, messageColors } = require('./colorize');
 const IS_PAGE_FROM_PREVIOUS_SCAN_TAG = messageColors.processing('[isPageFromPreviousScan]');
 const REALTIME_CLEANUP_TAG = messageColors.processing('[realtime_cleanup]');
 const GROUP_WINDOW_CLEANUP_TAG = messageColors.processing('[group_window_cleanup]');
-const { execSync, execFile } = require('child_process');
+const { execFile } = require('child_process');
 
 // Window cleanup delay constant
 const WINDOW_CLEANUP_DELAY_MS = 15000;
 // window_clean REALTIME
-const REALTIME_CLEANUP_BUFFER_MS = 25000; // Additional buffer time after site delay (increased for Cloudflare)
 const REALTIME_CLEANUP_THRESHOLD = 12; // Default number of pages to keep
 const REALTIME_CLEANUP_MIN_PAGES = 6; // Minimum pages before cleanup kicks in
 
@@ -380,7 +379,30 @@ async function performRealtimeWindowCleanup(browserInstance, threshold = REALTIM
 
     // Use the provided total delay (already includes appropriate buffer)
     const cleanupDelay = totalDelay;
-    
+
+    // Pre-wait short-circuit. The only pages this pass can ever close are popups
+    // (untracked) and idle pages — active main pages are protected by
+    // isPageSafeToClose. When concurrency exceeds the threshold the page count is
+    // dominated by active main pages, so without this we'd wait the full
+    // cleanupDelay and then close nothing (e.g. max_concurrent 30 vs threshold 8
+    // = a ~36s no-op on every task). If nothing is even a candidate, skip the
+    // wait. A main task that finishes during the skipped wait closes its OWN page,
+    // so realtime cleanup never needed to wait for it.
+    const hasCloseCandidate = quickPages.some(p => {
+      if (p.isClosed()) return false;
+      const usage = pageUsageTracker.get(p);
+      return !usage || !usage.isProcessing; // untracked popup, or a tracked-idle page
+    });
+    if (!hasCloseCandidate) {
+      if (forceDebug) {
+        console.log(formatLogMessage('debug', `${REALTIME_CLEANUP_TAG} ${quickPages.length} pages but all actively processing — skipping ${cleanupDelay}ms wait (nothing closeable)`));
+      }
+      result.success = true;
+      result.totalPages = quickPages.length;
+      result.reason = 'all_active';
+      return result;
+    }
+
     if (forceDebug) {
       console.log(formatLogMessage('debug', `${REALTIME_CLEANUP_TAG} Waiting ${cleanupDelay}ms before cleanup (threshold: ${threshold})`));
     }

package/lib/dns.js

@@ -0,0 +1,238 @@
+/**
+ * DNS pre-check resolver with multi-nameserver rotation.
+ *
+ * Owns nameserver selection and robust resolution for the scan's DNS
+ * pre-check. The default global resolver leads EVERY query with the FIRST
+ * nameserver in /etc/resolv.conf, so under scan concurrency one server
+ * (typically the ISP resolver) takes the whole c-ares burst and starts
+ * answering REFUSED while the other configured servers (e.g. 8.8.8.8/8.8.4.4)
+ * sit idle. This module builds one Resolver per nameserver — each leading with
+ * a different server, the rest kept as failover order — and round-robins them
+ * per resolve attempt so the lead spreads across all servers (and across the
+ * retry). A `--dns` override pins/rotates an explicit list instead of
+ * resolv.conf.
+ *
+ * Scope: this affects the pre-check resolver only. Chrome's navigation DNS
+ * (OS resolver) and nettools' dig/whois are separate paths and unaffected.
+ */
+const net = require('node:net');
+const dnsPromises = require('node:dns/promises');
+const { getServers: getSystemDnsServers } = require('node:dns');
+const { Resolver: DnsPromiseResolver } = require('node:dns/promises');
+const { formatLogMessage } = require('./colorize');
+
+// c-ares codes that mean "resolver problem" (retry-worthy / fail-open), not
+// "the host does not exist".
+const DNS_TRANSIENT_ERRORS = new Set(['ETIMEOUT', 'ESERVFAIL', 'EREFUSED', 'ECONNREFUSED']);
+
+/**
+ * True only for a definitive "host does not exist / has no address" answer —
+ * the only case that justifies skipping a URL in the pre-check. Everything
+ * else (EREFUSED, ESERVFAIL, ETIMEOUT, ECONNREFUSED, timeout) is a resolver
+ * problem the caller should fail open on.
+ * @param {string} code
+ * @returns {boolean}
+ */
+function isNonExistenceError(code) {
+  return code === 'ENOTFOUND' || code === 'ENODATA';
+}
+
+// Accept a bare IPv4/IPv6 address, or an address with a port in the exact form
+// Resolver.setServers() understands: `ipv4:port` or `[ipv6]:port`.
+function isResolverSpec(s) {
+  if (net.isIP(s)) return true;
+  const bracketed = s.match(/^\[([0-9a-fA-F:]+)\](?::\d{1,5})?$/);
+  if (bracketed) return net.isIP(bracketed[1]) === 6;
+  const v4port = s.match(/^(\d{1,3}(?:\.\d{1,3}){3}):\d{1,5}$/);
+  if (v4port) return net.isIP(v4port[1]) === 4;
+  return false;
+}
+
+/**
+ * Parse + validate a `--dns` / config value into a clean, de-duplicated server
+ * list. Accepts a comma-separated string or an array. Each entry may be a bare
+ * IPv4/IPv6 address or an address with a port (`8.8.8.8:5353`,
+ * `[2001:db8::1]:5353`) — the form setServers() accepts. Invalid entries are
+ * warned and dropped; duplicates are collapsed so the rotation stays even.
+ * @param {string|string[]|undefined} raw
+ * @returns {string[]} validated server specs (possibly empty)
+ */
+function parseDnsServers(raw) {
+  if (!raw) return [];
+  const parts = (Array.isArray(raw) ? raw : String(raw).split(','))
+    .map(s => String(s).trim())
+    .filter(Boolean);
+  const valid = [];
+  const seen = new Set();
+  for (const p of parts) {
+    if (!isResolverSpec(p)) {
+      console.warn(`⚠ --dns: ignoring invalid server "${p}" (expected IPv4/IPv6, optionally with :port)`);
+      continue;
+    }
+    if (!seen.has(p)) { seen.add(p); valid.push(p); }
+  }
+  return valid;
+}
+
+/**
+ * Build a rotating pre-check resolver.
+ * @param {object} [opts]
+ * @param {string[]} [opts.servers] - explicit servers (from --dns). When empty,
+ *   the system resolv.conf servers are used.
+ * @param {boolean} [opts.forceDebug] - emit a debug line on the retry path.
+ * @returns {{ resolveHost: (hostname:string, timeoutMs:number)=>Promise<void>,
+ *   servers: string[], rotates: boolean, pinned: boolean }}
+ *   resolveHost resolves on success and rejects with the final error
+ *   (err.code intact) on failure.
+ */
+function createRotatingResolver(opts = {}) {
+  const forceDebug = !!opts.forceDebug;
+  const override = Array.isArray(opts.servers) && opts.servers.length > 0 ? opts.servers : null;
+
+  let systemServers = [];
+  try { systemServers = getSystemDnsServers(); } catch { systemServers = []; }
+  const servers = override || systemServers;
+
+  // Pin/rotate an explicit --dns list (even a single server — never fall back
+  // to the OS resolver in that case). For resolv.conf, only build a pool when
+  // there is more than one server to rotate; otherwise use the global API
+  // (which already reads resolv.conf).
+  const shouldPool = override ? servers.length >= 1 : servers.length > 1;
+  let pool = null;
+  if (shouldPool) {
+    pool = servers.map((_, i) => {
+      const r = new DnsPromiseResolver();
+      // setServers accepts exactly what we hold here: getServers()'s own output
+      // (system path) or net-validated specs incl. ip:port (override path).
+      // Keep the resolver's default servers if an entry is somehow rejected.
+      try { r.setServers([...servers.slice(i), ...servers.slice(0, i)]); } catch { /* keep default */ }
+      return r;
+    });
+  }
+
+  let cursor = 0;
+  // Resolver for the next attempt: rotated when a pool exists, else the global
+  // promises API. `cursor++` is a synchronous single-threaded increment, so even
+  // under heavy concurrency every caller gets a distinct slot and the lead
+  // distribution stays exactly even (no locking needed).
+  const nextResolver = () => (pool ? pool[cursor++ % pool.length] : dnsPromises);
+
+  // One resolution attempt: rotate the lead server, resolve4 first, and on
+  // no-IPv4 (ENODATA/ENOTFOUND) fall back to resolve6 so IPv6-only hosts aren't
+  // wrongly skipped. Any OTHER code propagates unchanged so the caller sees the
+  // real resolver error. A timeout is kept as a safety net — with c-ares off
+  // the libuv threadpool it should rarely fire.
+  async function attempt(hostname, timeoutMs) {
+    const resolver = nextResolver();
+    let timer;
+    try {
+      const timeoutP = new Promise((_, reject) => {
+        timer = setTimeout(() => reject(new Error('DNS timeout')), timeoutMs);
+      });
+      const chain = resolver.resolve4(hostname).catch(err => {
+        if (err && (err.code === 'ENODATA' || err.code === 'ENOTFOUND')) {
+          return resolver.resolve6(hostname);
+        }
+        throw err;
+      });
+      await Promise.race([chain, timeoutP]);
+    } finally {
+      if (timer) clearTimeout(timer);
+    }
+  }
+
+  /**
+   * Resolve a hostname, rotating the lead server per attempt and retrying once
+   * on a transient/resolver error (so the retry leads with the next server —
+   * if one REFUSES, the retry hits another).
+   */
+  async function resolveHost(hostname, timeoutMs) {
+    try {
+      await attempt(hostname, timeoutMs);
+    } catch (firstErr) {
+      const code = firstErr && firstErr.code;
+      if (DNS_TRANSIENT_ERRORS.has(code) || (firstErr && firstErr.message === 'DNS timeout')) {
+        if (forceDebug) console.log(formatLogMessage('debug', `DNS pre-check transient (${code || 'timeout'}) for ${hostname}, retrying once`));
+        await attempt(hostname, timeoutMs);
+      } else {
+        throw firstErr;
+      }
+    }
+  }
+
+  return { resolveHost, servers, rotates: !!pool, pinned: !!override };
+}
+
+/**
+ * Circuit breaker for the DNS pre-check. During a resolver-refusal storm the
+ * pre-check is worthless (every host fails open and proceeds anyway) and
+ * actively harmful (it piles ~2× the queries — with the retry — onto an
+ * already-refusing resolver). This trips when resolver errors dominate a recent
+ * window of attempts and suspends pre-checking for a cooldown so the resolver
+ * gets breathing room; sites still load (a suspended pre-check just proceeds to
+ * navigation, exactly like a single fail-open). NXDOMAIN and success count as
+ * HEALTHY (the resolver answered) — only resolver errors (EREFUSED / ESERVFAIL
+ * / ETIMEOUT / ECONNREFUSED / timeout) count against it.
+ *
+ * @param {object} [opts]
+ * @param {number} [opts.window=20]        attempts kept in the rolling window
+ * @param {number} [opts.threshold=10]     resolver-errors in the window to trip
+ * @param {number} [opts.cooldownMs=30000] how long to stay suspended once tripped
+ * @param {boolean} [opts.forceDebug]
+ * @param {function} [opts.now]            clock injection (tests); defaults to Date.now
+ * @returns {{ record:(isResolverError:boolean)=>void, isTripped:()=>boolean,
+ *   stats:()=>{tripped:boolean,errorCount:number,windowFill:number,trips:number} }}
+ */
+function createDnsCircuitBreaker(opts = {}) {
+  const windowSize = opts.window || 20;
+  const threshold = opts.threshold || 10;
+  const cooldownMs = opts.cooldownMs != null ? opts.cooldownMs : 30000;
+  const forceDebug = !!opts.forceDebug;
+  const now = opts.now || Date.now;
+
+  const recent = [];   // booleans, true = resolver error
+  let errorCount = 0;
+  let openUntil = 0;   // suspended while now() < openUntil
+  let trips = 0;
+
+  // Feed one resolve outcome. Only ever called while closed (a suspended
+  // pre-check skips the resolve, so no outcome is produced).
+  function record(isResolverError) {
+    recent.push(!!isResolverError);
+    if (isResolverError) errorCount++;
+    if (recent.length > windowSize && recent.shift()) errorCount--;
+
+    if (now() >= openUntil && errorCount >= threshold) {
+      openUntil = now() + cooldownMs;
+      trips++;
+      console.log(formatLogMessage('warn', `[dns-precheck] resolver errors ${errorCount}/${recent.length} — suspending DNS pre-check ${Math.round(cooldownMs / 1000)}s (sites still load; backing off the resolver)`));
+    }
+  }
+
+  // True while suspended. On the first call after the cooldown elapses, resume
+  // with a clean window so the storm is re-measured fresh rather than re-tripping
+  // on stale errors.
+  function isTripped() {
+    if (now() < openUntil) return true;
+    if (openUntil !== 0) {
+      openUntil = 0;
+      recent.length = 0;
+      errorCount = 0;
+      if (forceDebug) console.log(formatLogMessage('debug', '[dns-precheck] cooldown elapsed — resuming DNS pre-check'));
+    }
+    return false;
+  }
+
+  return {
+    record,
+    isTripped,
+    stats: () => ({ tripped: now() < openUntil, errorCount, windowFill: recent.length, trips }),
+  };
+}
+
+module.exports = {
+  createRotatingResolver,
+  createDnsCircuitBreaker,
+  parseDnsServers,
+  isNonExistenceError,
+};

package/lib/domain-cache.js

@@ -103,54 +103,6 @@ class DomainCache {
     return wasNew;
   }
 
-  /**
-   * Combined check-and-mark in one pass. Functionally equivalent to
-   * isDomainAlreadyDetected() followed by markDomainAsDetected(), but with
-   * one Set.has() call instead of two. (JS is single-threaded so all three
-   * variants are individually atomic; this one is just cheaper.)
-   * @param {string} domain - Domain to check and potentially mark
-   * @returns {boolean} True if domain was ALREADY detected (should skip), false if NEW (should process)
-   */
-  checkAndMark(domain) {
-    if (!domain || typeof domain !== 'string') {
-      return false;
-    }
-
-    const wasAlreadyDetected = this.cache.has(domain);
-    
-    if (wasAlreadyDetected) {
-      // Domain already exists - update skip stats and return true (should skip)
-      this.stats.totalSkipped++;
-      this.stats.cacheHits++;
-      
-      if (this.enableLogging) {
-        console.log(formatLogMessage('debug', `${this.logPrefix} Cache HIT: ${domain} (skipped)`));
-      }
-      return true; // Already detected, should skip
-    }
-    
-    // Domain is NEW - mark it as detected
-    this.stats.cacheMisses++;
-    
-    this.cache.add(domain);
-    this.stats.totalDetected++;
-    
-    if (this.enableLogging) {
-      console.log(formatLogMessage('debug', `${this.logPrefix} Cache MISS: ${domain} (processing and marked, cache size: ${this.cache.size})`));
-    }
-    
-    // Check size after the add so an overflow only fires eviction once per
-    // overflowing call (using targetCacheSize precomputed in the constructor).
-    if (this.cache.size > this.maxCacheSize) {
-      const toRemove = this.cache.size - this.targetCacheSize;
-      if (toRemove > 0) {
-        this.clearOldestEntries(toRemove);
-      }
-    }
-
-    return false; // New domain, should process
-  }
-
   /**
    * Clear oldest entries from cache (FIFO eviction). Set iteration order is
    * guaranteed insertion order per ES2015, so this genuinely evicts oldest-
@@ -208,45 +160,6 @@ class DomainCache {
     return this.cache.has(domain);
   }
 
-  /**
-   * Add multiple domains to cache at once. Uses a single .size delta to
-   * count actually-new entries (skipping per-domain .has() calls), and
-   * runs the size-overflow eviction check once after the batch instead of
-   * per-domain. For a batch of N domains this is N .has() calls saved and
-   * up to N redundant cap checks collapsed to one.
-   * @param {Array<string>} domains - Array of domains to add
-   * @returns {number} Number of domains actually added (excludes duplicates)
-   */
-  markMultipleDomainsAsDetected(domains) {
-    if (!Array.isArray(domains) || domains.length === 0) {
-      return 0;
-    }
-
-    const startSize = this.cache.size;
-    for (let i = 0; i < domains.length; i++) {
-      const d = domains[i];
-      if (d && typeof d === 'string') {
-        this.cache.add(d);
-      }
-    }
-    const addedCount = this.cache.size - startSize;
-    this.stats.totalDetected += addedCount;
-
-    if (this.enableLogging && addedCount > 0) {
-      console.log(formatLogMessage('debug', `${this.logPrefix} Batch added ${addedCount} new domains (cache size: ${this.cache.size})`));
-    }
-
-    // One eviction sweep at the end, mirroring the single-add overflow check.
-    if (this.cache.size > this.maxCacheSize) {
-      const toRemove = this.cache.size - this.targetCacheSize;
-      if (toRemove > 0) {
-        this.clearOldestEntries(toRemove);
-      }
-    }
-
-    return addedCount;
-  }
-
   /**
    * Create bound helper functions for easy integration with existing code
    * @returns {object} Object with bound helper functions
@@ -255,7 +168,6 @@ class DomainCache {
     return {
       isDomainAlreadyDetected: this.isDomainAlreadyDetected.bind(this),
       markDomainAsDetected: this.markDomainAsDetected.bind(this),
-      checkAndMark: this.checkAndMark.bind(this),
       getSkippedCount: () => this.stats.totalSkipped,
       getCacheSize: () => this.cache.size,
       getStats: this.getStats.bind(this)
@@ -273,8 +185,7 @@ let globalDomainCache = null;
  *
  * NOTE: `options` is honored ONLY on the first call (the call that actually
  * constructs the singleton). Subsequent calls return the existing instance
- * regardless of what's passed. If you need different settings, call
- * resetGlobalCache() first or use `new DomainCache(options)` directly.
+ * regardless of what's passed; options are fixed at first construction.
  *
  * Under debug logging, a warning fires if a later caller passes options
  * that don't match the live instance — silent drift is a recurring source
@@ -295,7 +206,7 @@ function getGlobalDomainCache(options = {}) {
       (options.enableLogging !== undefined && options.enableLogging !== globalDomainCache.enableLogging) ||
       (options.logPrefix !== undefined && options.logPrefix !== globalDomainCache.logPrefix);
     if (drifted) {
-      console.log(formatLogMessage('debug', `${globalDomainCache.logPrefix} getGlobalDomainCache called with options that differ from the live singleton; ignored (call resetGlobalCache() first to apply new options)`));
+      console.log(formatLogMessage('debug', `${globalDomainCache.logPrefix} getGlobalDomainCache called with options that differ from the live singleton; ignored (options are fixed at first construction)`));
     }
   }
   return globalDomainCache;
@@ -312,36 +223,17 @@ function createGlobalHelpers(options = {}) {
 }
 
 /**
- * Reset the global cache (useful for testing or manual resets)
- */
-function resetGlobalCache() {
-  if (globalDomainCache) {
-    globalDomainCache.clear();
-  }
-  globalDomainCache = null;
-}
-
-/**
- * Legacy wrapper functions for backward compatibility
- * These match the original function signatures from nwss.js
+ * Legacy wrapper for backward compatibility.
  *
- * NOTE: getTotalDomainsSkipped and getDetectedDomainsCount are the only
- * ones kept — they're used directly by nwss.js for end-of-scan stats.
- * Previously-defined isDomainAlreadyDetected / markDomainAsDetected /
- * checkAndMark wrappers were removed: nwss.js calls those via
- * createGlobalHelpers() now and repo-wide grep confirmed zero remaining
- * external callers of the legacy wrappers.
+ * getDetectedDomainsCount is the only one kept — nwss.js reads it for the
+ * end-of-scan "unique domains cached" stat. getTotalDomainsSkipped was
+ * removed: its value was always 0 because the global cache's skip-check
+ * (isDomainAlreadyDetected) is never called — cross-URL dedup is handled by
+ * nettools' processed-domain sets / smart-cache / the per-URL set — so the
+ * stat was misleading. The isDomainAlreadyDetected / markDomainAsDetected /
+ * checkAndMark wrappers were likewise removed; nwss.js uses createGlobalHelpers().
  */
 
-/**
- * Get total domains skipped (legacy wrapper)
- * @returns {number} Number of domains skipped
- */
-function getTotalDomainsSkipped() {
-  const cache = getGlobalDomainCache();
-  return cache.stats.totalSkipped;
-}
-
 /**
  * Get detected domains cache size (legacy wrapper)
  * @returns {number} Size of the detected domains cache
@@ -352,15 +244,10 @@ function getDetectedDomainsCount() {
 }
 
 module.exports = {
-  // Main class
-  DomainCache,
-  
-  // Global cache functions
-  getGlobalDomainCache,
+  // Global cache helpers — createGlobalHelpers feeds nwss.js's per-domain
+  // marking; getDetectedDomainsCount feeds the end-of-scan "unique domains
+  // cached" stat. (DomainCache / getGlobalDomainCache stay internal — no
+  // external consumer; construct via createGlobalHelpers.)
   createGlobalHelpers,
-  resetGlobalCache,
-  
-  // Legacy wrappers still used by nwss.js for end-of-scan stats
-  getTotalDomainsSkipped,
   getDetectedDomainsCount
 };