@fanboynz/network-scanner 2.0.59 → 2.0.61

package/CHANGELOG.md

@@ -2,6 +2,57 @@
 
 All notable changes to the Network Scanner (nwss.js) project.
 
+## [2.0.61] - 2026-03-17
+
+### Added
+- `.nwssconfig` file for per-config-file CLI settings — define output, concurrency, flags per JSON config
+- `--no-color` / `--no-colour` flag to disable colors (colors now enabled by default)
+- Navigation timeout fallback — retries with `waitUntil: networkidle2` on timeout, 10s cap
+- Skip domains after 3 consecutive timeouts in the same scan to avoid wasting time on down sites
+- Fingerprint cache capped at 500 entries with LRU eviction
+
+### Fixed
+- `chrome-error://` popup redirects no longer throw errors — continue processing captured requests
+- Suppressed noisy `about:blank` and `chrome-error://` redirect warnings (visible with `--debug` only)
+- Fallback retry skipped for `chrome-error://` redirects (instant failure, not genuine timeout)
+- Page URL checked before fallback retry to detect already-failed state
+- `.nwssconfig` keys support both hyphens and underscores (`dns-cache` and `dns_cache` both work)
+
+### Improved
+- Colors enabled by default — no need for `--color` flag or `color: true` in `.nwssconfig`
+- Chrome UA bumped to 146, Firefox UA bumped to 148
+- Sec-CH-UA headers updated to match Chrome 146
+
+## [2.0.60] - 2026-03-16
+
+### Added
+- `--dns-cache` flag for persistent dig/whois disk caching between runs (`.digcache`, `.whoiscache`)
+- `--load-extension <path>` flag to load unpacked Chrome extensions (supports multiple)
+- `--block-ads` now supports comma-separated list files (`--block-ads=easylist.txt,easyprivacy.txt`)
+- `disable_ad_tagging` config option to control Chrome AdTagging (default: true)
+- DNS cache hit/miss statistics in scan summary output with fresh domain names listed
+- Concurrent dig/whois deduplication — multiple pages requesting the same domain share one lookup
+- SIGINT/SIGTERM handlers for `--keep-open` to prevent orphaned Chrome processes
+
+### Fixed
+- Adblock pipe (`|`) character handling — mid-pattern pipes were incorrectly treated as anchors, causing broad false positives on EasyList rules like `/addyn|*|adtech;`
+- Domain Map fast path was skipping resource type checks — `$ping`, `$script` etc. now correctly enforced
+- Domain extraction for `||domain.com/path` rules — path was incorrectly included in domain name
+- `--keep-open` now skips extension-blocking Chrome flags so Chrome Web Store and extensions work
+- Corrupt disk cache files are deleted instead of persisted
+- `getBaseDomain()` now uses `psl` for correct multi-part TLD handling (`.co.uk`, `.com.au`)
+- Merged 7 separate `--disable-features` flags into one — Chrome only reads the last occurrence
+
+### Improved
+- `$document` rules treated as full domain blocks (matches all resource types)
+- `adblock.js`: regex cache for compiled patterns, Set for resource type lookups, lazy parentDomains, two-level result cache with LRU eviction (32K), hoisted constants, freed parsed options after rule parsing
+- `output.js`: capped wildcard regex cache at 500, simplified `*.domain.com` suffix matching, hoisted resource type map
+- `compare.js`: pre-compiled and deduplicated 6 normalization regexes
+- `grep.js`: build grep args once outside pattern loop
+- `domain-cache.js`: use Set iterator for eviction instead of full array copy
+- `nettools.js`: hoisted ANSI strip regex, disk cache flushes once on exit instead of per-lookup
+- Dig/whois cache: 14-hour TTL, 1000 entry limit, pretty-printed JSON files
+
 ## [2.0.59] - 2026-03-15
 
 ### Added

package/README.md

@@ -29,7 +29,7 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 |:---------------------------|:------------|
 | `-o, --output <file>`       | Output file for rules. If omitted, prints to console |
 | `--compare <file>`          | Remove rules that already exist in this file before output |
-| `--color, --colour`         | Enable colored console output for status messages |
+| `--no-color, --no-colour`   | Disable colored console output (colors enabled by default) |
 | `--append`                  | Append new rules to output file instead of overwriting (requires `-o`) |
 
 ### Output Format Options
@@ -64,6 +64,9 @@ A Puppeteer-based tool for scanning websites to find third-party (or optionally
 | `--headful`                 | Launch browser with GUI (not headless) |
 | `--keep-open`               | Keep browser and tabs open after scan completes (use with `--headful` for debugging) |
 | `--use-puppeteer-core`      | Use `puppeteer-core` with system Chrome instead of bundled Chromium |
+| `--load-extension <path>`   | Load unpacked Chrome extension from directory (can be used multiple times) |
+| `--dns-cache`               | Persist dig/whois results to disk between runs (14hr TTL, `.digcache`/`.whoiscache`) |
+| `--block-ads=<files>`       | Block ads using EasyList format rules (comma-separated: `easylist.txt,easyprivacy.txt`) |
 | `--cdp`                     | Enable Chrome DevTools Protocol logging (now per-page if enabled) |
 | `--remove-dupes`            | Remove duplicate domains from output (only with `-o`) |
 | `--dry-run`                 | Console output only: show matching regex, titles, whois/dig/searchstring results, and adblock rules |
@@ -404,6 +407,53 @@ If a proxy fails mid-scan, Chromium's error code is detected and diagnosed:
 
 Detected error codes: `ERR_PROXY_CONNECTION_FAILED`, `ERR_SOCKS_CONNECTION_FAILED`, `ERR_TUNNEL_CONNECTION_FAILED`, `ERR_PROXY_AUTH_UNSUPPORTED`, `ERR_PROXY_AUTH_REQUESTED`, `ERR_SOCKS_CONNECTION_HOST_UNREACHABLE`, `ERR_PROXY_CERTIFICATE_INVALID`, `ERR_NO_SUPPORTED_PROXIES`.
 
+---
+
+## .nwssconfig — Per-Config Settings
+
+Create a `.nwssconfig` file in the project root to define CLI settings per config file. When a config filename matches a key, those settings are automatically applied. CLI flags merge with and override `.nwssconfig` settings.
+
+```json
+{
+  "configs": {
+    "config-clean1.json": {
+      "output": "outputfile.txt",
+      "max_concurrent": 30,
+      "dns_cache": true,
+      "cache_requests": true,
+      "dumpurls": true,
+      "remove_tempfiles": true,
+      "color": true
+    },
+    "config-clean2.json": {
+      "output": "outputfile.txt",
+      "max_concurrent": 15,
+      "dns_cache": true,
+      "cache_requests": true,
+      "dumpurls": true,
+      "remove_tempfiles": true,
+      "color": true,
+      "debug": true,
+      "block_ads": "easylist.txt,easyprivacy.txt"
+    }
+  }
+}
+```
+
+**Usage:**
+
+```bash
+node nwss.js config-clean1.json                    # uses .nwssconfig settings
+node nwss.js config-clean2.json --debug             # .nwssconfig + debug override
+node nwss.js config-other.json --max-concurrent 5   # no match in .nwssconfig, uses CLI flags
+```
+
+**Supported settings:** `output`, `max_concurrent`, `dns_cache`, `cache_requests`, `dumpurls`, `remove_tempfiles`, `color`, `remove_dupes`, `compress_logs`, `debug`, `silent`, `verbose`, `headful`, `keep_open`, `dry_run`, `titles`, `sub_domains`, `no_interact`, `ghost_cursor`, `plain`, `cdp`, `dnsmasq`, `unbound`, `privoxy`, `pihole`, `eval_on_doc`, `use_puppeteer_core`, `ignore_cache`, `clear_cache`, `block_ads`, `compare`, `localhost`, `append`.
+
+**Priority:** CLI flags > `.nwssconfig` > hardcoded defaults.
+
+---
+
 ### Global Configuration Options
 
 These options go at the root level of your config.json: