@govtechsg/oobee 0.10.98 → 0.11.1

package/AGENTS.md

@@ -144,6 +144,8 @@ The `constants` default export object holds runtime state:
 | `OOBEE_SCAN_PRODUCT` | Adds `scanProduct` tag to Sentry events |
 | `OOBEE_CONSECUTIVE_MAX_RETRIES` | Max consecutive HTTP failures before circuit breaker aborts crawl (default 100) |
 | `OOBEE_VALIDATE_URL` | If set, exit after URL validation without scanning |
+| `OOBEE_SAVE_DOM` | `1` or `true` = save full-page DOM HTML to `pageDOMs/` in results directory. Supported scan types: Website, Sitemap, Intelligent, LocalFile, Custom |
+| `OOBEE_SAVE_PAGE_SCREENSHOT` | `1` or `true` = save full-page desktop + mobile viewport screenshots to `pageDOMs/desktopPageScreenshots/` and `pageDOMs/mobilePageScreenshots/`. Mobile viewport uses iPhone 11 width programmatically. Supported scan types: Website, Sitemap, Intelligent, LocalFile, Custom |
 | `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` | Proxy configuration |
 | `NO_PROXY` / `INCLUDE_PROXY` | Proxy bypass/include lists |
 
@@ -212,6 +214,7 @@ docker run oobee node dist/cli.js ...
 
     - If Chrome/Edge profile cloning fails (for example `EBUSY` while copying locked cookie/state files on Windows), Oobee now falls back to an empty cloned profile directory for that scan. This keeps browser launch stable, but authenticated session cookies may not be available.
     - Crawlee's browser pool retires and re-launches browser instances after ~4 minutes. On Windows, reusing the same `--user-data-dir` causes Chrome exit code 21 (stale lock contention). `getPreLaunchHook()` in `commonCrawlerFunc.ts` assigns unique `_pool{N}` directories for each re-launch and performs a best-effort async clone of the base profile. Cleanup must glob `_pool*` directories alongside the base `oobee-{token}` dir.
+    - On Windows, Chrome writes files asynchronously during its shutdown sequence (`first_party_sets.db`, `optimization_guide_model_store/`, `segmentation_platform/`, `Local State`, `Profile N/`). Pool directory cleanup uses a 5s initial delay (vs 2s on other platforms) and retries up to 3 times in `getPostPageCloseHook()`. The final sweep in `cleanUp()` also retries after a 3s delay on Windows.
 
 4. **`constants.launcher` mutation** — When webkit is the fallback, `constants.launcher` is reassigned globally. This affects all subsequent browser launches in the same process.
 
@@ -252,6 +255,14 @@ When making changes, validate these areas which have well-established edge cases
 - Crawlee's async lock-file operations (`.json.lock` mkdir) can fire after the crawl finishes. On Windows, this triggers uncaughtException EPERM during report generation. A scoped exception handler suppresses these. The cleanup delay is 5s on Windows, 3s on others.
 - The crawlee dataset folder and `tmp-items` (intermediate JSONL store) must be deleted BEFORE zipping results. `zipResults` must be the last step in `generateArtifacts()` — any cleanup or processing that removes temp files from `storagePath` must happen earlier. The dataset deletion uses an awaited delay (not fire-and-forget setTimeout) to let lingering Crawlee I/O flush.
 - Errors must only be recorded in `failedRequestHandler` (after all retries exhausted), not in the `requestHandler` catch block. Crawlee retries up to 3 times, so recording in the catch block creates duplicates and false positives for URLs that succeed on retry.
+- **"Download is starting" navigation errors**: URLs that trigger file downloads (e.g. Salesforce `/download/` endpoints without `.pdf` extension) cause `page.goto()` to throw. Crawlee retries 3 times (all fail the same way). In `failedRequestHandler`, detect via `request.errorMessages.includes('Download is starting')`. If `isScanPdfs`: re-enqueue with `skipNavigation: true` and unique key `download_${url}` so the requestHandler's PDF download path handles it via `sendRequest`. If not scanning PDFs: classify as `STATUS_CODE_METADATA[1]` ("Not A Supported Document").
+- **403 rate-limit retry**: In `failedRequestHandler`, 403 URLs are re-enqueued once with `userData.rateLimitRetried = true` and unique key `ratelimit_${url}`. This gives them a fresh attempt cycle after adaptive concurrency recovers from the rate-limit burst. Do NOT call `rateController.onFailure()` on the first pass — this avoids double-counting toward the circuit breaker. If the retry also fails, it falls through to the normal `onFailure` + circuit breaker path.
+- **Non-HTML document classification ("Unsupported Documents")**: URLs serving non-HTML content (images, PDFs, media, binary files) must be classified as `STATUS_CODE_METADATA[1]` ("Not A Supported Document") with `httpStatusCode: 1` in both crawlers. This applies whether the content-type is detected pre-navigation (via file extension in `preNavigationHooks`) or post-navigation (via response `content-type` header). The report UI identifies unsupported documents by `httpStatusCode === 1` — any other status code (even 0) with incorrect metadata may land in "Pages Not Scanned" instead. Both crawlDomain and crawlSitemap now report these consistently — crawlDomain no longer silently drops them.
+  - **`blackListedFileExtensions` (shared constant in `constants.ts`)**: Single source of truth for non-scannable file extensions. Used in both crawlers' `preNavigationHooks` to skip navigation, and in crawlDomain's post-navigation `isBlacklistedFileExtensions()` check. When adding new non-scannable types, update this one list. Does NOT include `.pdf` — PDFs are handled by the separate PDF scan path.
+  - **Both crawlers' preNavigationHooks**: Check URL pathname extension against `blackListedFileExtensions`. If matched, set `skipNavigation=true` (crawlDomain) or `skipNavigation=true` + `isNotSupportedDocument=true` (crawlSitemap). This avoids wasting browser resources on non-HTML URLs.
+  - **crawlSitemap else block**: when navigation succeeds but `content-type` is non-HTML (e.g. `image/png`, `application/pdf` after a 302 redirect), classifies as unsupported instead of using the HTTP status code. The check: `contentType && !contentType.startsWith('text/html') && !contentType.includes('html')`. Empty content-type falls through to existing logic (preserves behavior for pages that don't send content-type headers).
+  - **crawlDomain**: `shouldSkipDueToUnsupportedContent()` and `isBlacklistedFileExtensions()` now push to `urlsCrawled.userExcluded` with `httpStatusCode: 1`. Previously these silently returned with no classification. The preNavigationHook catches blacklisted extensions before navigation; for URLs without obvious extensions (e.g. redirects to non-HTML), the post-navigation content-type check is the safety net.
+  - **No impact to runAxeScript**: HTML pages never trigger these paths — `shouldSkipDueToUnsupportedContent` returns false for `text/html`, `blackListedFileExtensions` doesn't match `.html`/no-extension URLs, and `isWhitelistedContentType` passes for `text/html`. The full page lifecycle (waitForPageLoaded → DOM mutation observer → runAxeScript) is unchanged for scannable pages.
 
 ### URL & Redirect Handling
 - `https://example.com` and `https://example.com/` must be treated as the same page. Use `normUrl()` (wrapping `@apify/utilities normalizeUrl`) for all dedup sets.
@@ -298,6 +309,22 @@ When making changes, validate these areas which have well-established edge cases
 - In intelligent crawl, each phase (sitemap then domain) creates its own `CrawlRateController` instance — transitioning from sitemap to domain crawl starts fresh.
 - Without the circuit breaker, a rate-limited crawl with thousands of enqueued URLs would run indefinitely, never hit the success threshold, and never generate a report.
 - When enqueuing all sitemap URLs (which we do for accurate `totalLinksFetchedFromSitemaps` reporting), always ensure either a scan duration (`-d`) or the circuit breaker is in place as a safety net.
+- **WAF behavior differs by source IP**: Datacenter/cloud IPs (Linux servers) are often rate-limited more aggressively by WAFs than residential/ISP IPs (Windows desktops). This means the circuit breaker (100 consecutive failures) fires on Linux but NOT on Windows for the same site — Windows successfully recovers concurrency between rate-limit bursts, keeping the consecutive failure counter below 100. This difference affects whether `isAbortingScanNow` is set, which gates the second-pass click-discovery loop.
+
+### Click-Discovery Second Pass in crawlDomain
+- After `crawler.run()` completes, `crawlDomain` has a second-pass loop that re-visits all scanned same-hostname pages for `customEnqueueLinksByClickingElements` — a sequential loop that clicks every interactive element with 1s delay between clicks.
+- **This loop is skipped when `fromCrawlIntelligentSitemap` is true** — the domain phase of intelligent crawl is only meant to discover new pages via `<a>` link extraction, not exhaustively click 3000+ already-scanned pages.
+- **This loop is skipped when `isAbortingScanNow` is true** — if the circuit breaker fired or the rate controller hit its page limit, the second pass is skipped. This is why Linux scans (where the circuit breaker fires due to stricter WAF) don't get stuck, but Windows scans (where recovery succeeds) enter the loop.
+- Without `scanDuration` or the `fromCrawlIntelligentSitemap` guard, the second pass on a 3000+ page site can take 10+ hours (each page × 90s `requestHandlerTimeoutSecs` at concurrency 1).
+- The second pass produces no log output — `__clickpass__` handlers call `enqueueProcess` and return without `guiInfoLog` or rate controller interaction, making it appear as if the scan is hung.
+
+### Intelligent Sitemap Link Discovery Optimization
+- In intelligent crawl mode, `crawlSitemap` now performs `enqueueLinks` on each successfully scanned page (gated by `fromCrawlIntelligentSitemap`). This discovers `<a>` links from sitemap pages without any additional page loads — just a DOM query on the already-loaded page.
+- Discovered URLs go into a `RequestQueue`. Crawlee processes `RequestList` (sitemap URLs) first, then `RequestQueue` items after. So discovered links are scanned after all sitemap URLs complete, within the same crawlSitemap phase.
+- This eliminates most of the work the subsequent `crawlDomain` supplement phase would otherwise do. The domain phase still runs (to discover pages reachable only from the entry URL), but finds almost everything already in `scannedUrlSet` and finishes quickly.
+- **No behavior change for standalone scans**: The `enqueueLinks` block is gated by `fromCrawlIntelligentSitemap` — standalone sitemap scans (`-s sitemap`) and standalone website scans (`-s website`) are unaffected.
+- The `transformRequestFunction` in sitemap `enqueueLinks` filters robots.txt-disallowed URLs and marks PDFs for `skipNavigation`, matching `crawlDomain`'s behavior.
+- `crawlSitemap` always creates a `RequestQueue` (even in standalone mode) to enable download re-enqueue and 403 retry handling. An empty queue has zero impact on crawl behavior.
 
 ### Scan Consistency Between crawlDomain and crawlSitemap
 - Both crawlers must produce equivalent axe scan results for the same page. Any difference in how the page is observed/stabilized before `runAxeScript()` will cause inconsistent accessibility findings between scan types.
@@ -327,5 +354,17 @@ When making changes, validate these areas which have well-established edge cases
 ├── scanPagesDetail.json    # Per-page breakdown
 ├── scanPagesSummary.json   # Page-level summary
 ├── sitemap.xml             # Discovered URLs
-└── screenshots/            # Violation screenshots (if enabled)
+├── screenshots/            # Violation screenshots (if enabled)
+└── pageDOMs/              # Page capture (if OOBEE_SAVE_DOM or OOBEE_SAVE_PAGE_SCREENSHOT)
+    ├── {hash}-{truncated_path}.html          # Full DOM (OOBEE_SAVE_DOM)
+    ├── domManifest.json                     # Maps URLs → hash, file paths, errors
+    ├── desktopPageScreenshots/             # Desktop viewport PNGs (OOBEE_SAVE_PAGE_SCREENSHOT)
+    │   └── {hash}-{truncated_path}.png
+    └── mobilePageScreenshots/              # Mobile viewport PNGs (OOBEE_SAVE_PAGE_SCREENSHOT)
+        └── {hash}-{truncated_path}.png
 ```
+
+- `{hash}` is a 7-character SHA-256 of the page URL (consistent per URL, like git short hashes)
+- `{truncated_path}` is the URL path with `/` replaced by `_`, max 80 characters
+- If filename collisions occur, a `-2`, `-3` etc. suffix is added before the extension
+- `domManifest.json` is written when either env var is enabled, containing URL-to-file mappings

package/CRAWL.md

@@ -0,0 +1,260 @@
+# How Oobee Scanning Works
+
+## Table of Contents
+
+**Part 1: How Scanning Affects Accessibility Results**
+- [What Oobee Does](#what-oobee-does)
+- [What Gets Scanned](#what-gets-scanned)
+- [Scan Strategy](#scan-strategy)
+- [Why Some Pages Are Not Scanned](#why-some-pages-are-not-scanned)
+- [What Affects Result Accuracy](#what-affects-result-accuracy)
+- [Understanding the Report](#understanding-the-report)
+- [Choosing a Page Budget](#choosing-a-page-budget)
+- [Tips for Better Scan Results](#tips-for-better-scan-results)
+- [Recommended Hardware](#recommended-hardware)
+
+**Part 2: Technical Details**
+- [Scan Modes](#scan-modes)
+- [Page Discovery Mechanics](#page-discovery-mechanics)
+- [Concurrency and Ordering](#concurrency-and-ordering)
+- [Adaptive Concurrency and Rate Limiting](#adaptive-concurrency-and-rate-limiting)
+- [Error Handling Pipeline](#error-handling-pipeline)
+- [Page Classification](#page-classification)
+- [Page Budget](#page-budget)
+- [Browser Pool and Session State](#browser-pool-and-session-state)
+
+---
+
+## Part 1: How Scanning Affects Accessibility Results
+
+### What Oobee Does
+
+Oobee visits web pages using a real browser (Chrome/Chromium) and runs automated accessibility checks against each page. It produces a report of issues found, organized by severity (must fix, good to fix, needs review).
+
+### What Gets Scanned
+
+Oobee supports three main scan modes:
+
+- **Intelligent** (recommended): Automatically finds your sitemap via robots.txt, scans all sitemap pages, then follows links to discover any pages the sitemap missed. This gives the most complete and predictable coverage — the sitemap ensures known pages are scanned in a consistent order, while link discovery catches pages the sitemap doesn't list.
+- **Sitemap**: Scans only the pages listed in your website's sitemap XML file. You provide the sitemap URL, and Oobee visits each page listed in it. Good when you know your sitemap is comprehensive.
+- **Website**: Starts at one page and follows links to discover more pages on the same site. Discovers pages organically by extracting links from each page it visits. Coverage depends heavily on site navigation structure.
+
+Not all pages on a site will necessarily be scanned. The scanner stops when it reaches the page limit, time limit, or runs out of discoverable pages.
+
+### Scan Strategy
+
+The **strategy** (`-s`) controls which discovered links are followed. It is supported in Intelligent, Website, and Sitemap modes:
+
+- **`same-domain`** (default): Follows links on the same registered domain, including subdomains.
+- **`same-hostname`**: Only follows links on the exact same hostname (after stripping `www.`).
+- **`ignore`** (sitemap mode only): No URL filtering — all URLs in the sitemap are scanned regardless of domain. This is the default for standalone sitemap scans.
+
+**Example**: Scanning `https://www.acme.gov.sg/services/home`
+
+| Discovered Link | `same-domain` | `same-hostname` |
+|-----------------|:---:|:---:|
+| `https://www.acme.gov.sg/services/faq` | Followed | Followed |
+| `https://acme.gov.sg/about` | Followed | Followed (www. stripped) |
+| `https://employer.acme.gov.sg/dashboard` | Followed (same domain `acme.gov.sg`) | Skipped (different hostname) |
+| `https://blog.acme.gov.sg/news` | Followed (same domain `acme.gov.sg`) | Skipped (different hostname) |
+| `https://www.other.gov.sg/resources` | Skipped (different domain) | Skipped (different hostname) |
+
+Choosing the right strategy affects coverage:
+- Use `same-domain` when your site spans multiple subdomains and you want a holistic view.
+- Use `same-hostname` when you only care about one specific subdomain, or when following subdomains would waste budget on unrelated content (e.g., a separate blog platform).
+- The strategy only filters links discovered dynamically during the crawl — it does NOT affect which pages are listed in the sitemap.
+
+### Why Some Pages Are Not Scanned
+
+Pages may be skipped for several reasons:
+
+- **Not an HTML document**: The URL points to an image, PDF, spreadsheet, video, or other file type that cannot be checked for web accessibility.
+- **Blocked by the server**: The website's firewall or rate limiter returned an error (commonly 403 Forbidden). This is especially common on large scans where the server detects rapid automated requests.
+- **Requires authentication the scanner doesn't have**: Download endpoints, APIs, or pages behind a different login system than the one whose cookies were provided.
+- **Took too long to respond**: Pages that don't load within 30 seconds are skipped.
+- **Budget exhausted**: The scanner reached its page limit or time limit before getting to this page.
+- **Redirected out of scope**: The page redirected to a different domain that's outside the scan's strategy filter.
+
+### What Affects Result Accuracy
+
+- **Dynamic content**: Pages with heavy JavaScript may produce slightly different results on different runs because content loads asynchronously. The scanner waits for the page to stabilize, but some elements may appear after the check completes.
+- **Rate limiting**: When a site blocks the scanner, it reduces its speed and retries, but some pages may be permanently lost if the site continues blocking. Accuracy improves with higher page budgets and longer scan durations.
+- **Authentication**: The scanner sees pages as a logged-out visitor unless you provide authenticated browser cookies. Some page content is only visible when logged in.
+- **Non-deterministic discovery** (website mode): Pages are discovered by following links. Because multiple pages load simultaneously, the order in which links are found varies between runs. A page discovered on one run may not be found on another if the scanner hits its limit first.
+- **Mid-scan redirects**: If a page redirects while being scanned (JavaScript redirect, meta refresh), the accessibility check may run on the redirect target rather than the original page.
+
+### Understanding the Report
+
+The scan report categorizes URLs into three groups:
+
+- **Pages Scanned**: Pages where the accessibility check completed successfully. These have full results.
+- **Pages Not Scanned**: Pages that were found but couldn't be checked. The report shows why (e.g., "403 - Forbidden", "Web Crawler Errored", "access restrictions").
+- **Unsupported Documents**: URLs that point to non-web-page content — images, PDFs (when PDF scanning is disabled), downloads, media files. These are not accessibility-checkable.
+
+A higher "Pages Scanned" count relative to total discovered URLs means more comprehensive coverage.
+
+### Choosing a Page Budget
+
+The page budget (`-p`) determines how many pages are scanned. Choosing the right number is important for getting a representative picture of your site's accessibility:
+
+- **Small sites (< 100 pages)**: Set the budget equal to or above your total page count. You want full coverage.
+- **Medium sites (100-1000 pages)**: A budget of 500-1000 pages gives a good representative sample. Most accessibility issues repeat across page templates, so scanning a subset often reveals the same patterns as scanning everything.
+- **Large sites (1000+ pages)**: A budget of 2000-5000 pages typically captures all unique page templates and their issues. Beyond this, you'll mostly find the same issue types on more pages.
+
+A representative scan means enough pages have been checked that the issues found reflect the overall accessibility posture of the site. If your site has 20 page templates, scanning 100 pages is likely sufficient. If it has hundreds of unique layouts, you need a larger budget.
+
+### Tips for Better Scan Results
+
+- **Use Intelligent mode** for the most complete coverage — it combines sitemap accuracy with link-discovery thoroughness. This is the recommended mode for all production scans.
+- **Set an appropriate page budget** (`-p`) — too low and you miss page templates; too high and you waste time scanning duplicate layouts. Start with 2000 for large sites and adjust.
+- **Provide authenticated cookies** for sites behind login (clone your Chrome profile with `-u`).
+- **Increase scan duration** (`-d 3600`) for large sites to give the scanner enough time.
+- **Use `same-hostname` strategy** (`-s same-hostname`) to avoid following links to subdomains or external sites.
+- **Run from a residential IP** if possible — datacenter IPs are more likely to be rate-limited by firewalls.
+
+### Recommended Hardware
+
+Scanning is CPU and memory intensive — each page runs in a real browser with full JavaScript execution.  The time estimates illustrated below for scans that do not include screenshots.
+
+| Scan Size | Hardware | Max Concurrency | Expected Duration |
+|-----------|----------|-----------------|-------------------|
+| 1,000 pages | ECS Fargate, 2 vCPU / 4 GB RAM | 10 (`-t 10`) | 2-3 hours |
+| 5,000 pages | Recent Core Ultra or Snapdragon X Series Laptop/Desktop, 8 cores / 12 threads, 24 GB RAM | 25 (`-t 25`) | 2-3 hours |
+
+**Oobee Desktop** runs with `OOBEE_FAST_CRAWLER=true`, which means concurrency scales up aggressively to the maximum (25) as fast as possible. This is suitable for desktop machines with adequate CPU and RAM, but may cause stability issues on low-powered devices.
+
+Setting concurrency above 25 (e.g. `-t 50`) is possible but generally provides no speed improvement — either the target website rate-limits the extra requests, or the machine itself becomes the bottleneck (CPU saturation, memory pressure). In practice, 25 is the sweet spot for most hardware and most sites.
+
+For smaller environments (Fargate, low-spec VMs), reduce concurrency to 10 to avoid overwhelming the container's limited CPU and memory. Higher concurrency on constrained hardware causes thrashing, not faster scans.
+
+**Slow scan mode** (`-c website` with click discovery): The second-pass click-discovery loop visits every scanned page sequentially and clicks each interactive element with delays. On large sites (1000+ pages), this can add many hours to the scan. Use Intelligent mode instead — it discovers links via `<a>` tag extraction during the sitemap phase without the expensive click loop.
+
+These estimates assume the target site doesn't aggressively rate-limit. Actual times depend on page complexity, server response speed, and WAF behavior.
+
+**Disk space**: Ensure at least 20 GB free for a 5,000-page scan. This accounts for browser pool directories (~30-50 MB active at any time, but accumulates if cleanup is delayed), intermediate per-page JSON results (~2-5 KB each, ~10-25 MB total), uncompressed report artifacts (HTML report with embedded data can reach 500 MB+ before compression), and temporary PDF/screenshot storage if enabled. Smaller scans (1,000 pages) can get by with 5-10 GB free. Running out of disk space mid-scan causes hard failures (ENOSPC).
+
+**Slow machines degrade both coverage and accuracy**: Oobee has fixed timeouts — 30 seconds for a page to start loading, 90 seconds total for the page to load and be scanned. On an underpowered machine, Chromium itself runs slowly, which means:
+
+- **Dropped pages**: Pages that would load fine on a fast machine may hit the 30s navigation timeout simply because the CPU can't parse JavaScript fast enough. These pages are retried 3 times and then recorded as errors.
+- **Inaccurate DOM state**: Poorly-coded websites that rely on JavaScript to render content (SPAs, lazy-loaded components, deferred widgets) may not finish rendering before the scanner checks them. On a fast machine, the DOM mutation observer (5s quiet window) catches most dynamic content. On a slow machine, JavaScript execution is delayed — the DOM appears "settled" (no mutations detected) even though rendering hasn't started yet. This means the accessibility scan runs against an incomplete page, producing false negatives (missing issues that exist on the fully-rendered page) or false positives (flagging placeholder content that would normally be replaced).
+- **Thermal throttling cascade**: Sustained CPU load triggers thermal throttling on laptops, which makes subsequent pages even slower, leading to more timeouts and increasingly stale DOM snapshots. A scan that starts fine can degrade badly in its second hour.
+
+Desktop machines or cloud instances with adequate cooling and dedicated CPU cores are strongly preferred for large scans.
+
+**Docker deployments**: Ensure the container has access to the recommended CPU and memory — Chromium under-performs significantly when CPU-throttled or memory-constrained (swapping). Allocate sufficient storage volume and avoid thin-provisioned storage that can exhaust mid-scan.
+
+---
+
+## Part 2: Technical Details
+
+### Scan Modes
+
+| Mode | Entry Point | Discovery Method | Page Order |
+|------|-------------|------------------|------------|
+| Website | `crawlDomain.ts` | BFS via `enqueueLinks` + click discovery | Non-deterministic |
+| Sitemap | `crawlSitemap.ts` | URLs from sitemap XML | Mostly FIFO from XML |
+| Intelligent | `crawlIntelligentSitemap.ts` | Sitemap → `enqueueLinks` → domain supplement | Sitemap-ordered, then non-deterministic |
+
+### Page Discovery Mechanics
+
+**Sitemap mode:**
+- URLs parsed from sitemap XML into a Crawlee `RequestList` (ordered array)
+- In intelligent mode: URLs sorted by closeness to user URL + last-modified date
+- `RequestQueue` created alongside for discovered `<a>` links (intelligent mode) and download/403 re-enqueue
+- Crawlee processes `RequestList` first, then `RequestQueue` items
+
+**Website (domain) mode:**
+- Seed URL added to `RequestQueue`
+- Each scanned page runs `enqueueLinks` to extract `<a>` tags matching the strategy filter
+- Click-discovery pass: re-visits seed-hostname pages, clicks interactive elements, captures popup/navigation URLs
+- Second-pass click-discovery is skipped in intelligent mode (already covered by sitemap + link extraction)
+
+**Strategy filter** restricts which discovered URLs are followed:
+- `same-hostname`: Only URLs on the same hostname (after stripping `www.`)
+- `same-domain`: Same registered domain (includes subdomains)
+- `all`: Follow all URLs regardless of domain
+
+**Pre-navigation filtering:**
+- `blackListedFileExtensions`: Skips known non-HTML extensions (images, media, documents) before browser navigates
+- `disallowedListOfPatterns`: Skips non-HTTP protocols (mailto:, tel:, javascript:, etc.)
+- `robots.txt` disallow rules: Checked before enqueue
+
+### Concurrency and Ordering
+
+- Default max concurrency: 25 simultaneous pages (configurable via `-t`)
+- Pages process in parallel — completion order depends on server response times, not queue order
+- **Sitemap mode**: `RequestList` provides FIFO order to workers, but with 25 workers, pages 1-25 start together and complete in arbitrary order
+- **Domain mode**: Entirely non-deterministic — faster pages discover and enqueue links first, which get processed before slower pages' links
+- **Practical implication**: Two runs of the same sitemap scan produce the same set of pages (deterministic input), but a domain scan may find different pages depending on timing
+
+### Adaptive Concurrency and Rate Limiting
+
+The `CrawlRateController` manages concurrency dynamically:
+
+- **On HTTP 4xx/5xx**: Concurrency halved (floor 1). Consecutive failure counter incremented.
+- **On success**: After 10 consecutive successes, concurrency increases by 2 (up to original max). Counter reset.
+- **Circuit breaker**: After 100 consecutive failures (`OOBEE_CONSECUTIVE_MAX_RETRIES`), the crawl aborts gracefully.
+- **403 retry**: First 403 re-enqueued with `rateLimitRetried` flag (doesn't count toward circuit breaker). Second 403 is permanent failure.
+
+Crawlee's `retryOnBlocked: true` detects blocked responses (403, 429) and rotates sessions automatically before the request reaches the handler.
+
+### Error Handling Pipeline
+
+```
+Page Request
+    ↓
+[Navigation: 30s timeout]
+    ↓ success                    ↓ failure (timeout, network error, blocked)
+requestHandler                   Crawlee retries (up to 3 times)
+    ↓                                ↓ all retries exhausted
+[waitForPageLoaded: 10s]         failedRequestHandler
+[DOM mutation observer: 5s]          ↓
+[runAxeScript: axe-core scan]    Final classification:
+    ↓                            - "Download is starting" → Unsupported Document
+Results pushed to dataset        - 403 (first time) → re-enqueue for retry
+                                 - 403 (second time) → circuit breaker check → error
+                                 - Other → "Web Crawler Errored"
+```
+
+- `requestHandler` processes successful navigations. Errors thrown here are caught by Crawlee and the page is retried.
+- `failedRequestHandler` fires only after all retries are exhausted. This is where final error classification happens.
+- `requestHandlerTimeoutSecs: 90` — if the entire handler (navigation + scan) takes longer, Crawlee kills and retries.
+- Errors are never recorded in the `requestHandler` catch block — only in `failedRequestHandler` — to avoid duplicates for pages that succeed on retry.
+
+### Page Classification
+
+| Array | Report Category | Trigger |
+|-------|----------------|---------|
+| `urlsCrawled.scanned` | Pages Scanned | Successful axe scan |
+| `urlsCrawled.error` | Pages Not Scanned | All retries exhausted |
+| `urlsCrawled.invalid` | Pages Not Scanned | HTTP 3xx+ status or non-whitelisted content-type |
+| `urlsCrawled.forbidden` | Pages Not Scanned | (reserved) |
+| `urlsCrawled.userExcluded` | Unsupported Documents (if `httpStatusCode=1`) or Pages Not Scanned | Blacklisted extension, non-HTML content-type, excluded pattern, download URL |
+
+The report splits "Pages Not Scanned" further:
+- Items with `httpStatusCode === 1` → **Unsupported Documents** tab
+- Everything else → **Pages Not Scanned** tab
+
+Key `STATUS_CODE_METADATA` values:
+- `[1]` = "Not A Supported Document"
+- `[2]` = "Web Crawler Errored"
+- `[200]` = "Oobee was not able to scan the page due to access restrictions or compatibility issues"
+- `[403]` = "403 - Forbidden"
+- `[599]` = "Uncommon Response Status Code Received"
+
+### Page Budget
+
+`maxRequestsPerCrawl` counts **successful scans only**, not total requests:
+- A page that errors 3 times and succeeds on the 4th still counts as 1
+- A page that errors permanently doesn't consume budget
+- In intelligent mode, budget is shared across phases: `remaining = max - sitemapPhaseScanned`
+- `scanDuration` is a hard time limit (seconds) that aborts all phases when exceeded
+
+### Browser Pool and Session State
+
+- Browser instance retired after 500 pages → new browser launched
+- Each browser gets its own `_pool{N}` directory with cookies cloned fresh from the pristine profile
+- The base `userDataDirectory` is never modified by a running browser — always treated as read-only source
+- Pool directories cleaned mid-scan (when new browser launches) and at scan end
+- `--disk-cache-size=10485760` (10MB) caps per-browser cache to prevent storage bloat
+- Browser rotation means session cookies stay fresh (cloned from original) but any server-side session tracking may see the crawler as a "new visitor" after rotation

package/README.md

@@ -18,6 +18,10 @@ For the **user-friendly desktop application**, check out [Oobee](https://go.gov.
 6. [Corretto](https://aws.amazon.com/corretto)
 7. [VeraPDF](https://github.com/veraPDF/veraPDF-apps)
 
+## How Scanning Works
+
+For a detailed explanation of how Oobee scans websites, what affects result accuracy, recommended hardware specs, and tips for getting the best coverage, see [CRAWL.md](./CRAWL.md). Recommended reading before running large scans.
+
 ## Using Oobee as a NodeJS module
 
 If you wish to use Oobee as a NodeJS module that can be integrated with end-to-end testing frameworks, refer to the [integration guide](./INTEGRATION.md) on how you can do so.
@@ -96,6 +100,8 @@ verapdf --version
 | OOBEE_SCAN_METADATA | Overrides the `entryUrl` tag sent to telemetry. | |
 | OOBEE_SCAN_PRODUCT | Adds a `scanProduct` tag to telemetry events. | |
 | OOBEE_CONSECUTIVE_MAX_RETRIES | Max consecutive HTTP failures before the circuit breaker aborts the crawl. | `100` |
+| OOBEE_SAVE_DOM | Set to `1` or `true` to save full-page DOM HTML for each scanned page to the results directory. Works with all scan types (Website, Sitemap, Intelligent, LocalFile, Custom). | |
+| OOBEE_SAVE_PAGE_SCREENSHOT | Set to `1` or `true` to save full-page desktop and mobile viewport screenshots for each scanned page. Mobile viewport width is determined from iPhone 11 device profile. Works with all scan types (Website, Sitemap, Intelligent, LocalFile, Custom). | |
 | HTTP_PROXY | URL of the proxy server to be used for HTTP requests (e.g. `http://proxy.example.com:8080`). | |
 | HTTPS_PROXY | URL of the proxy server to be used for HTTPS requests (e.g. `https://proxy.example.com:8080`). | |
 | ALL_PROXY | URL of the proxy server to be used for all requests, typically used for SOCKS5 proxies (e.g. `socks5://proxy.example.com:1080`. Note: IPv6 direct connections may still continue even though socks5 proxy is specified due to a known issue with Chrome/Chromium. (Recommended workaround is to turn off IPv6 at host-level). | |

package/dist/combine.js

@@ -12,6 +12,7 @@ import { consoleLogger } from './logs.js';
 import runCustom from './crawlers/runCustom.js';
 import { alertMessageOptions } from './constants/cliFunctions.js';
 import { isS3UploadEnabled, getS3MetadataFromEnv, getS3UploadPrefix, uploadFolderToS3, } from './services/s3Uploader.js';
+import { writeManifest, resetCaptureEntries, isPageCaptureEnabled } from './crawlers/pageCapture.js';
 // Class exports
 export class ViewportSettingsClass {
     constructor(deviceChosen, customDevice, viewportWidth, playwrightDeviceDetailsObject) {
@@ -32,6 +33,9 @@ const combineRun = async (details, deviceToScan) => {
     process.env.CRAWLEE_LOG_LEVEL = 'ERROR';
     process.env.CRAWLEE_STORAGE_DIR = randomToken;
     constants.sitemapFetchedLinks = null;
+    if (isPageCaptureEnabled() && !process.env.OOBEE_SCAN_PRODUCT) {
+        process.env.OOBEE_SCAN_PRODUCT = 'U&A';
+    }
     if (process.env.CRAWLEE_SYSTEM_INFO_V2 === undefined) {
         // Set the environment variable to enable system info v2
         // Resolves issue with when wmic is not installed on Windows
@@ -52,6 +56,13 @@ const combineRun = async (details, deviceToScan) => {
             consoleLogger.info(`Suppressed Playwright post-close connection error: ${err.message}`);
             return;
         }
+        // Suppress EPERM errors from Crawlee's async lock-file operations that fire
+        // after a crawl phase finishes. On Windows, the sitemap phase's async lock
+        // cleanup can race with the domain phase starting in the same directory.
+        if (err.message?.includes('EPERM') && err.message?.includes('.json.lock')) {
+            consoleLogger.info(`Suppressed Crawlee lock-file EPERM (stale async cleanup): ${err.message}`);
+            return;
+        }
         throw err;
     };
     process.on('uncaughtException', psTreeHandler);
@@ -186,6 +197,8 @@ const combineRun = async (details, deviceToScan) => {
     if (scanDetails.urlsCrawled) {
         if (scanDetails.urlsCrawled.scanned.length > 0) {
             await createAndUpdateResultsFolders(randomToken);
+            await writeManifest(randomToken);
+            resetCaptureEntries();
             const pagesNotScanned = [
                 ...urlsCrawledObj.error,
                 ...urlsCrawledObj.invalid,

package/dist/constants/common.js

@@ -1806,6 +1806,13 @@ export const getPlaywrightLaunchOptions = (browser) => {
         !arg.startsWith('--user-agent=') &&
         arg !== '--mute-audio' &&
         !(browser === BrowserTypes.CHROME && arg === '--edge-skip-compat-layer-relaunch'));
+    // Cap browser disk cache to 10MB per instance to prevent storage bloat
+    // during long crawls with multiple pool rotations
+    finalArgs.push('--disk-cache-size=10485760');
+    // Prevent Windows from throttling background Chromium processes
+    if (os.platform() === 'win32') {
+        finalArgs.push('--disable-features=UseEcoQoSForBackgroundProcess');
+    }
     // Headless flags (unchanged)
     if (process.env.CRAWLEE_HEADLESS === '1') {
         if (!finalArgs.includes('--mute-audio'))

package/dist/constants/constants.js

@@ -25,10 +25,27 @@ export const blackListedFileExtensions = [
     'svg',
     'gif',
     'woff',
+    'woff2',
     'zip',
     'webp',
     'json',
     'xml',
+    'ico',
+    'bmp',
+    'tiff',
+    'tif',
+    'avi',
+    'mov',
+    'wmv',
+    'flv',
+    'ogg',
+    'wav',
+    'doc',
+    'docx',
+    'xls',
+    'xlsx',
+    'ppt',
+    'pptx',
 ];
 export const getIntermediateScreenshotsPath = (datasetsPath) => `${datasetsPath}/screenshots`;
 export const destinationPath = (storagePath) => `${storagePath}/screenshots`;

package/dist/crawlers/commonCrawlerFunc.js

@@ -890,9 +890,9 @@ export const runAxeScript = async ({ includeScreenshots, page, randomToken, cust
     return filterAxeResults(results, pageTitle, customFlowDetails);
 };
 export const createCrawleeSubFolders = async (randomToken) => {
-    const crawleeDir = path.join(getStoragePath(randomToken), "crawlee");
-    const dataset = await Dataset.open(crawleeDir);
-    const requestQueue = await RequestQueue.open(crawleeDir);
+    const baseDir = path.join(getStoragePath(randomToken), "crawlee");
+    const dataset = await Dataset.open(baseDir);
+    const requestQueue = await RequestQueue.open(`${baseDir}_rq`);
     return { dataset, requestQueue };
 };
 export const preNavigationHooks = (extraHTTPHeaders) => {
@@ -964,60 +964,74 @@ export const postNavigationHooks = [
 ];
 export const getPreLaunchHook = (userDataDirectory) => {
     let launchCount = 0;
+    let previousPoolDir = null;
     return async (_pageId, launchContext) => {
         const fsp = await import('fs/promises').then(m => m.default);
         launchCount += 1;
-        // First launch uses the base directory; subsequent launches get a unique
-        // directory so that lingering file handles from a retired browser don't
-        // cause Chrome exit code 21 on Windows.
-        const effectiveDir = launchCount === 1
-            ? userDataDirectory
-            : `${userDataDirectory}_pool${launchCount}`;
-        await fsp.mkdir(effectiveDir, { recursive: true });
-        // For pool re-launches, copy only auth-relevant files from the base
-        // directory so authenticated sessions are preserved without cloning the
-        // entire profile (which grows with caches, IndexedDB, etc. during the crawl).
-        if (launchCount > 1) {
-            try {
-                // Copy top-level Local State (cookie encryption keys on Windows)
-                const localStateSrc = path.join(userDataDirectory, 'Local State');
-                if (await fsp.stat(localStateSrc).catch(() => null)) {
-                    await fsp.copyFile(localStateSrc, path.join(effectiveDir, 'Local State')).catch(() => { });
-                }
-                // Find profile directories (Default, Profile 1, Profile 2, etc.)
-                const entries = await fsp.readdir(userDataDirectory, { withFileTypes: true }).catch(() => []);
-                const profileDirs = entries.filter((e) => e.isDirectory() && /^(Default|Profile \d+)$/i.test(e.name));
-                for (const profile of profileDirs) {
-                    const srcProfile = path.join(userDataDirectory, profile.name);
-                    const destProfile = path.join(effectiveDir, profile.name);
-                    await fsp.mkdir(destProfile, { recursive: true }).catch(() => { });
-                    // Cookies (macOS layout: <Profile>/Cookies)
-                    const cookiesSrc = path.join(srcProfile, 'Cookies');
-                    if (await fsp.stat(cookiesSrc).catch(() => null)) {
-                        await fsp.copyFile(cookiesSrc, path.join(destProfile, 'Cookies')).catch(() => { });
-                    }
-                    // Cookies (Windows layout: <Profile>/Network/Cookies)
-                    const networkCookiesSrc = path.join(srcProfile, 'Network', 'Cookies');
-                    if (await fsp.stat(networkCookiesSrc).catch(() => null)) {
-                        const destNetwork = path.join(destProfile, 'Network');
-                        await fsp.mkdir(destNetwork, { recursive: true }).catch(() => { });
-                        await fsp.copyFile(networkCookiesSrc, path.join(destNetwork, 'Cookies')).catch(() => { });
+        // Clean up the previous pool directory. When preLaunchHooks fires, the
+        // previous browser has been retired and is finishing its last pages.
+        // Schedule cleanup with enough delay for Chrome to fully exit —
+        // closeInactiveBrowserAfterSecs (30s) + Windows file-lock grace.
+        if (previousPoolDir) {
+            const dirToClean = previousPoolDir;
+            const isWin = process.platform === 'win32';
+            const delay = isWin ? 40000 : 35000;
+            setTimeout(async () => {
+                for (let attempt = 0; attempt < 3; attempt++) {
+                    try {
+                        await fsp.rm(dirToClean, { recursive: true, force: true });
+                        return;
                     }
-                    // Local Storage (auth tokens, session data)
-                    const localStorageSrc = path.join(srcProfile, 'Local Storage');
-                    const localStorageStat = await fsp.stat(localStorageSrc).catch(() => null);
-                    if (localStorageStat && localStorageStat.isDirectory()) {
-                        const destLocalStorage = path.join(destProfile, 'Local Storage');
-                        await fsp.mkdir(destLocalStorage, { recursive: true }).catch(() => { });
-                        const lsFiles = await fsp.readdir(localStorageSrc).catch(() => []);
-                        await Promise.all(lsFiles.map(f => fsp.copyFile(path.join(localStorageSrc, f), path.join(destLocalStorage, f)).catch(() => { })));
+                    catch {
+                        await new Promise(r => setTimeout(r, 5000));
                     }
                 }
+            }, delay);
+        }
+        // Every browser gets its own directory. The base userDataDirectory is
+        // treated as a read-only cookie source (the pristine clone of the user's
+        // profile) and is never used directly by a running browser.
+        const effectiveDir = `${userDataDirectory}_pool${launchCount}`;
+        await fsp.mkdir(effectiveDir, { recursive: true });
+        // Copy auth-relevant files from the pristine base directory so
+        // authenticated sessions are preserved across pool rotations.
+        try {
+            const localStateSrc = path.join(userDataDirectory, 'Local State');
+            if (await fsp.stat(localStateSrc).catch(() => null)) {
+                await fsp.copyFile(localStateSrc, path.join(effectiveDir, 'Local State')).catch(() => { });
             }
-            catch {
-                // Silent fallback: use empty profile if clone fails
+            const entries = await fsp.readdir(userDataDirectory, { withFileTypes: true }).catch(() => []);
+            const profileDirs = entries.filter((e) => e.isDirectory() && /^(Default|Profile \d+)$/i.test(e.name));
+            for (const profile of profileDirs) {
+                const srcProfile = path.join(userDataDirectory, profile.name);
+                const destProfile = path.join(effectiveDir, profile.name);
+                await fsp.mkdir(destProfile, { recursive: true }).catch(() => { });
+                // Cookies (macOS layout: <Profile>/Cookies)
+                const cookiesSrc = path.join(srcProfile, 'Cookies');
+                if (await fsp.stat(cookiesSrc).catch(() => null)) {
+                    await fsp.copyFile(cookiesSrc, path.join(destProfile, 'Cookies')).catch(() => { });
+                }
+                // Cookies (Windows layout: <Profile>/Network/Cookies)
+                const networkCookiesSrc = path.join(srcProfile, 'Network', 'Cookies');
+                if (await fsp.stat(networkCookiesSrc).catch(() => null)) {
+                    const destNetwork = path.join(destProfile, 'Network');
+                    await fsp.mkdir(destNetwork, { recursive: true }).catch(() => { });
+                    await fsp.copyFile(networkCookiesSrc, path.join(destNetwork, 'Cookies')).catch(() => { });
+                }
+                // Local Storage (auth tokens, session data)
+                const localStorageSrc = path.join(srcProfile, 'Local Storage');
+                const localStorageStat = await fsp.stat(localStorageSrc).catch(() => null);
+                if (localStorageStat && localStorageStat.isDirectory()) {
+                    const destLocalStorage = path.join(destProfile, 'Local Storage');
+                    await fsp.mkdir(destLocalStorage, { recursive: true }).catch(() => { });
+                    const lsFiles = await fsp.readdir(localStorageSrc).catch(() => []);
+                    await Promise.all(lsFiles.map(f => fsp.copyFile(path.join(localStorageSrc, f), path.join(destLocalStorage, f)).catch(() => { })));
+                }
             }
         }
+        catch {
+            // Silent fallback: use empty profile if clone fails
+        }
         // Clean any stale lock files that may block browser launches on Windows
         const lockFiles = [
             path.join(effectiveDir, 'SingletonLock'),
@@ -1028,10 +1042,37 @@ export const getPreLaunchHook = (userDataDirectory) => {
             path.join(effectiveDir, 'Default', 'Network', 'LOCK'),
         ];
         await Promise.all(lockFiles.map(f => fsp.rm(f, { force: true }).catch(() => { })));
+        previousPoolDir = effectiveDir;
         // eslint-disable-next-line no-param-reassign
         launchContext.userDataDir = effectiveDir;
     };
 };
+export const getPostPageCloseHook = (userDataDirectory) => {
+    return async (_pageId, browserController) => {
+        if (browserController.activePages === 0) {
+            const dir = browserController.launchContext?.userDataDir;
+            if (dir && dir !== userDataDirectory && dir.includes('_pool')) {
+                const fsp = await import('fs/promises').then(m => m.default);
+                // Chrome on Windows writes optimization/segmentation data during shutdown;
+                // wait longer (5s) for the process to fully release file handles.
+                const isWin = process.platform === 'win32';
+                const initialDelay = isWin ? 5000 : 2000;
+                setTimeout(async () => {
+                    for (let attempt = 0; attempt < 3; attempt++) {
+                        try {
+                            await fsp.rm(dir, { recursive: true, force: true });
+                            return;
+                        }
+                        catch {
+                            // Retry after a short back-off (Windows file locks)
+                            await new Promise(r => setTimeout(r, 2000));
+                        }
+                    }
+                }, initialDelay);
+            }
+        }
+    };
+};
 export const failedRequestHandler = async ({ request }) => {
     guiInfoLog(guiInfoStatusTypes.ERROR, { numScanned: 0, urlScanned: request.url });
     log.error(`Failed Request - ${request.url}: ${request.errorMessages}`);