mcp-scraper 0.35.1 → 0.36.0

package/docs/specs/kernel-stealth-captcha-test-matrix.md

@@ -0,0 +1,278 @@
+# Kernel stealth, CAPTCHA, proxy, and localization test matrix
+
+Status: executable direct-Kernel smoke runner implemented; product-contract gates specified below.
+
+## Objective
+
+Determine which Kernel configuration reliably reaches and extracts localized Google Search and Maps results, while proving that MCP Scraper preserves the requested egress mode, waits for the managed solver correctly, reports evidence honestly, charges correctly, and cleans up every resource.
+
+This is deliberately two test systems:
+
+1. A direct Kernel capability matrix isolates browser, proxy, and solver behavior. It has one attempt per cell and no product fallback.
+2. An MCP Scraper product-contract matrix verifies retry policy, extraction, diagnostics, billing, and cleanup after the direct matrix identifies promising configurations.
+
+Do not use a production retry result to compare proxy families. Today, `harvest` can switch an explicit `location` request to `none` after two failures. That contaminates the experiment.
+
+## What counts as proof
+
+- A CAPTCHA was solved only when Kernel emits a terminal `captcha_solve_result` with `status: success` **and** the page becomes usable afterward.
+- A client-side wait expiring is not a solver failure. Record it separately from Kernel `failure`, `timeout`, and `abandoned` outcomes.
+- A Google Sorry page is a challenge outcome, never a ranking result.
+- A new proxy ID does not prove a new exit IP. Record both the proxy resource and observed egress.
+- A residential cell passes only if the requested mode remained effective. Direct or ISP fallback cannot be credited to residential.
+- One successful probabilistic run is a smoke result. Promotion requires repeated success.
+
+## Test layers
+
+### Layer 0 — deterministic release gates
+
+These tests are mocked or fixture-driven, incur no Kernel spend, and must gate every release.
+
+| ID | Situation | Test to create | Required assertion |
+|---|---|---|---|
+| U01 | Explicit residential fails | `explicit_location_never_downgrades_without_allowProxyFallback` | No attempt changes to `none` unless the caller explicitly enabled fallback. |
+| U02 | Resolver returns no location proxy | `location_resolution_without_proxy_id_fails_before_browser_launch` | Browser launch never occurs and the terminal code is `proxy_unavailable`. |
+| U03 | Authorized fallback occurs | `fallback_response_reports_requested_mode_effective_mode_and_fallback_used` | Requested, effective, and fallback fields remain distinct. |
+| U04 | Configured fallback is used | `configured_fallback_does_not_claim_residential` | Result cannot be labeled location/residential. |
+| U05 | Managed solver clears a challenge | `challenge_clearance_records_terminal_success_and_stable_page` | Two consecutive clear checks plus successful solver telemetry are required. |
+| U06 | Client waits expire | `challenge_wait_matrix_preserves_client_and_kernel_outcomes` | Exercise 2.5s, 10s, 30s, 60s, and abort; no outcome collapsing. |
+| U07 | Challenge briefly clears and returns | `transient_clear_does_not_pass_stability_check` | Reappearing challenge remains blocked. |
+| U08 | Maps is challenged | `maps_challenge_waits_for_managed_solver_before_rotation` | Maps uses the shared solver wait rather than throwing immediately. |
+| U09 | Maps solver succeeds | `maps_solver_clearance_continues_in_same_session` | Extraction continues without creating a fresh session. |
+| U10 | Maps wait is aborted | `maps_abort_during_solver_wait_closes_session_once` | One close, one delete, no retry after abort. |
+| U11 | Proxy resources differ but exit does not | `distinct_proxy_ids_with_same_observed_ip_set_exit_reused` | Cell is flagged `exitReused`; it is not counted as a fresh exit. |
+| U12 | Egress evidence is missing | `missing_egress_probe_makes_proxy_verdict_unknown_not_pass` | Proxy verdict is unknown. |
+| U13 | Residential geo is wrong | `residential_success_requires_observed_geo_match` | Location mismatch fails the location cell. |
+| U14 | Kernel default proxy is requested | `kernel_default_mode_does_not_disable_default_proxy` | Browser update never sends `disable_default_proxy: true`. |
+| U15 | Direct stealth is requested | `direct_stealth_disables_default_proxy_and_keeps_solver` | Update happens before navigation and solver remains enabled. |
+| U16 | Headless is requested | `requested_browser_mode_matches_kernel_create_and_response` | Request, returned Kernel mode, telemetry, and billing agree. |
+| U17 | A challenged session times out | `solver_timeout_preserves_session_when_handoff_enabled` | Session remains active for a bounded signed handoff instead of immediate deletion. |
+| U18 | Handoff is resumed | `resume_after_handoff_uses_original_session_and_cleans_up` | Original session is used, then deleted exactly once. |
+| U19 | Handoff is unauthorized or expired | `expired_or_wrong_owner_handoff_is_rejected` | No session URL or token is leaked. |
+| U20 | Visible desktop pack says Businesses | `extracts_businesses_local_pack_fixture` | Cards, CIDs, URLs, ratings, and positions are structured. |
+| U21 | Visible desktop pack says Places | `extracts_places_local_pack_fixture` | Parser does not depend on one English heading. |
+| U22 | Visible mobile pack uses different markup | `extracts_mobile_local_pack_fixture` | Mobile cards are structured. |
+| U23 | Pack is visible but parser returns zero | `visible_pack_with_zero_parsed_cards_emits_parser_miss` | A parser miss is explicit, not a silent empty pack. |
+| U24 | SERP succeeds and optional snapshots throw | `successful_serp_survives_page_snapshot_transport_failure` | SERP is returned with a snapshot warning. |
+| U25 | One snapshot fails | `partial_success_charges_only_started_snapshots` | SERP plus attempted snapshots are settled; no blanket extraction failure. |
+| U26 | Direct session billing retries | `browser_session_settlement_is_atomic_and_idempotent` | Retry cannot double-charge after debit succeeds and row update fails. |
+| U27 | Challenge diagnostics are formatted | `outbound_challenge_diagnostics_redact_tokens_and_full_ip` | Sorry query tokens, task IDs, session IDs, and full IPs stay internal. |
+| U28 | Matrix artifact repeats a cell | `matrix_recorder_preserves_every_repetition` | Key is `runId + cellId + repetition`; no last-write collapse. |
+| U29 | Budget abort fires | `matrix_budget_abort_still_cleans_every_resource` | Browser, session, proxy, profile, and terminal artifact are all finalized. |
+
+Required local-pack fixtures:
+
+- `tests/fixtures/serp/local-pack-businesses-desktop.html`
+- `tests/fixtures/serp/local-pack-places-desktop.html`
+- `tests/fixtures/serp/local-pack-mobile.html`
+
+### Layer 1 — direct Kernel capability matrix
+
+Run sequentially. Each cell gets one browser, one target navigation, no retry, and no fallback. Enable only Kernel's default operational telemetry set so CAPTCHA events are captured without recording page content or credentials.
+
+For the standard reCAPTCHA demo, the runner attempts one checkbox click to trigger the challenge when the widget permits it. If Kernel's automatic solver has already locked the widget in a loading state, it does not take over or solve manually: it waits and requires a response token plus a usable submitted demo page.
+
+#### Capability controls on Google's reCAPTCHA demo
+
+| Cell | Stealth | Egress | Solver expected | Purpose |
+|---|---:|---|---:|---|
+| A | off | direct | no | Direct negative control. |
+| B | on | Kernel default static ISP | yes | Kernel's recommended bundled mode. |
+| C | off | explicit ISP | no | Network-matched negative control for D. |
+| D | on | same explicit ISP | yes | Isolates solver effect on a stable ISP exit. |
+| E | off | explicit residential | no | Network-matched negative control for F. |
+| F | on | same explicit residential | yes | Tests managed solver with residential egress. |
+| G | on | direct after `disable_default_proxy` | yes | Network-matched solver comparison against A. |
+
+Cell pass rules:
+
+- Negative control: the test is valid if the challenge remains and no managed-solver success is claimed.
+- Solver cell: `captcha_solve_result.status === "success"`, response token appears, and submitted demo becomes usable.
+- No challenge exercised: `not_challenged`, not a solver pass.
+- hCaptcha entitlement absence: `unsupported_for_org`, not solver failure.
+
+#### Initial Google Search cells
+
+Fixed input:
+
+- query: `florist Nashville TN`
+- location/UULE: `Nashville, TN`
+- target: `genysflowers.com`
+- device: desktop
+- wait cap: 30 seconds for smoke; longer caps are separate cells
+
+| Cell | Egress | Browser | Session | Purpose |
+|---|---|---|---|---|
+| S01 | direct + managed solver | headless | fresh | Known clean-egress baseline. |
+| S02 | Kernel default ISP + managed solver | headless | fresh | Test the configuration Kernel says generally has better pass rates. |
+| S03 | ZIP 37203 residential + managed solver | headless | fresh | Compare exact residential targeting. |
+| S04 | direct + managed solver | headful | fresh | Isolate headless detection on direct egress. |
+| S05 | Kernel default ISP + managed solver | headful | fresh | Isolate headless detection on Kernel's bundled proxy. |
+| S06 | ZIP 37203 residential + managed solver | headful | fresh | Isolate headless detection for residential. |
+| S07 | ZIP 37209 residential + managed solver | headless | separate run | Detect ZIP-specific proxy reputation or inventory. |
+| S08 | winning proxy | headless | second navigation in same session | Measure session warming. |
+| S09 | winning proxy | headless | new session with warmed profile | Measure persisted state. |
+
+For every cell capture:
+
+- requested and returned stealth/headless values;
+- whether Kernel's default proxy was disabled;
+- proxy type, resolution source, and masked resource suffix;
+- diagnostic egress IP, city, region, postal code, country, and organization;
+- target-domain challenge classification;
+- every terminal `captcha_solve_result` field;
+- client wait duration versus Kernel solver duration;
+- organic count, local-pack count, Geny's organic ranks, and location evidence;
+- browser close, session deletion, proxy deletion, profile deletion;
+- elapsed session time and modeled browser cost.
+
+Residential IPs can rotate per connection. An IP diagnostic against another host is supporting evidence, not proof of the exact IP Google saw.
+
+### Layer 2 — direct Google Maps matrix
+
+Use `florist` + `Nashville, TN`, target business `Geny's Flowers`.
+
+| Cell | Egress | Browser | ZIP |
+|---|---|---|---|
+| M01 | direct + managed solver | headless | — |
+| M02 | Kernel default ISP + managed solver | headless | — |
+| M03 | residential + managed solver | headless | 37203 |
+| M04 | residential + managed solver | headless | 37209 |
+| M05 | winning egress | headful | winner |
+| M06 | winning egress | headless | same-session second search |
+
+The same challenge state machine and telemetry requirements apply to Search and Maps.
+
+### Layer 3 — MCP Scraper product-contract matrix
+
+Run only after Layer 1 identifies viable configurations.
+
+| Cell | Product surface | Requested mode | Hard assertions |
+|---|---|---|---|
+| P01 | `/serp-intelligence/capture` | `none` | Effective direct, UULE present, cleanup true, no fallback. |
+| P02 | `/serp-intelligence/capture` | `location`, 37203 | Effective residential, geo evidence, no silent direct success. |
+| P03 | `/maps/search` | `none` | Shared solver wait, structured results or explicit terminal challenge. |
+| P04 | `/maps/search` | `location`, 37203 | Effective residential, no mode downgrade, complete attempt evidence. |
+| P05 | SERP plus snapshots | winner | Optional snapshot failure preserves the SERP. |
+| P06 | challenged handoff | winner | Signed first-party handoff resumes original session and settles once. |
+
+Billing assertions:
+
+- Search holds 14 Credits, settles to 4 when headless or 14 when headful, and refunds a failed capture.
+- Maps Search costs 5 Credits and refunds failure.
+- Direct Kernel experiments do not touch the MCP Scraper wallet.
+- Browser mode used for billing must equal the mode Kernel reports.
+
+## Adaptive replication
+
+Do not run the full Cartesian product immediately.
+
+1. Run A–G once as capability smoke.
+2. Run S01–S06 once, then run S07 as a separate ZIP-targeted comparison.
+3. Repeat the two best solver-enabled configurations three times on:
+   - `florist Nashville TN`
+   - `wedding florist Nashville TN`
+   - `flower delivery Nashville TN`
+4. Cross only the winner with headful/headless, same/fresh session, and cold/warm profile.
+5. Run Search and Maps ten times each at controlled cadence before promotion.
+6. Run a 24-hour soak only after the winner is stable.
+
+Randomize configuration order within each query block. Do not run Google cells concurrently; concurrency itself changes rate-limit pressure and invalidates the comparison.
+
+## Promotion gates
+
+A configuration is promotable only when:
+
+- every successful run's effective egress equals its requested egress;
+- there are zero silent fallbacks and zero cleanup leaks;
+- solver success has terminal Kernel telemetry plus usable page evidence;
+- requested location is matched or the result is explicitly unknown/failing;
+- a one-off winner reproduces at least three times;
+- the final soak shows at least a 20 percentage-point success-rate improvement over direct control;
+- median and p95 latency remain within the product timeout budget;
+- Search and Maps both use the same challenge state machine;
+- billing settlement matches the effective browser mode.
+
+## Cost and stop controls
+
+Phase 1 defaults:
+
+- concurrency: 1;
+- maximum sessions: 13 for the executable smoke runner, 18 for the expanded phase;
+- per-session wall cap: 60 seconds;
+- no automatic cell retry;
+- no automatic proxy-family fallback;
+- modeled browser spend cap: $0.05 for one expanded phase, excluding proxy/bandwidth charges;
+- full IPs only in local mode-0600 JSON; rollups mask IPs and resource IDs.
+
+Stop immediately on authentication/plan/quota failure, two failed session deletions, an effective-mode mismatch, three consecutive proxy-family setup failures, or the session/time/spend cap.
+
+## Executable runner
+
+Implemented files:
+
+- `tests/live/harness/kernel-captcha-matrix.ts`
+- `tests/live/serp/kernel-stealth-captcha-matrix.live.test.ts`
+
+Run the direct smoke matrix only:
+
+```bash
+LIVE=1 \
+KERNEL_CAPTCHA_MATRIX=1 \
+KERNEL_CAPTCHA_MATRIX_SCOPE=smoke \
+KERNEL_CAPTCHA_MATRIX_WAIT_MS=30000 \
+npx vitest run --reporter=basic --config vitest.live.config.ts \
+  tests/live/serp/kernel-stealth-captcha-matrix.live.test.ts
+```
+
+The runner creates isolated ISP and residential proxy resources, runs cells sequentially, reads CAPTCHA telemetry through Kernel's current REST endpoint, writes a mode-0600 JSON artifact and masked Markdown rollup, and deletes every browser session and disposable proxy.
+
+## Exploratory evidence from 2026-07-17
+
+- The seven capability controls produced no attributable managed-solver proof. Several stealth cells emitted `captcha_solve_result` events marked `other:success`, but the target challenge remained visible and the event host/path fields were null. The runner correctly does not count those events as a solve.
+- S01–S03 (headless direct, Kernel-default ISP, and ZIP-targeted residential) all reached Google's unusual-traffic challenge in the initial run.
+- S04 and S05 (headful direct and headful Kernel-default ISP) returned usable localized Search pages without encountering a challenge.
+- S06 (headful residential) used Nashville diagnostic egress but remained challenged after the 30-second wait despite repeated unattributed `other:success` events.
+- These are one-run exploratory outcomes, not promotion evidence. No configuration has yet proven managed CAPTCHA solving, and any apparent winner must be replicated under randomized order and controlled cadence.
+
+Masked evidence:
+
+- `output/kernel-stealth-captcha-matrix-2026-07-17.md`
+- `output/kernel-stealth-captcha-matrix-2026-07-17-headful.md`
+
+## Concurrent default-ISP versus direct evidence from 2026-07-17
+
+One 20-request headful, stealth-enabled burst prepared ten city pairs and released every Google navigation within 3 milliseconds of one shared barrier. The Kernel-default lane omitted `proxy_id` and kept the default proxy enabled; the direct lane omitted `proxy_id` and explicitly sent `disable_default_proxy: true`.
+
+- Clean extracted-SERP success: default 0/10; direct 7/10.
+- Google `/search` access without a challenge: default 2/10; direct 9/10.
+- Google Sorry-page blocks: default 8/10; direct 1/10.
+- Every cell reported its intended effective mode; there was no retry, fallback, setup failure, or cleanup leak.
+- Four unblocked pages had no results at the first post-`domcontentloaded` inspection, so this one burst reports access and extraction separately. The runner now waits an identical bounded 1.5 seconds before inspection on future runs.
+
+This is evidence for this account's simultaneous 20-session workload, not a general claim about all Kernel default ISP exits. See `output/kernel-default-vs-direct-serp-burst-2026-07-17.md` and `docs/decisions/2026-07-17-kernel-default-isp-serp-burst-experiment.md`.
+
+## Current known gaps the matrix must expose
+
+- Search waits only 2.5 seconds by default for Kernel's solver.
+- Maps does not call the shared solver-wait path.
+- Explicit location can silently switch to direct egress.
+- BrowserDriver cannot represent Kernel's default stealth ISP because it disables the default whenever no explicit proxy ID is present.
+- `config.headless` can disagree with the mode actually sent to Kernel.
+- Current Kernel SDK dependency is 0.52.0; the current npm release is 0.79.0, and 0.52.0 exposes no telemetry surface in its types. The runner uses Kernel's documented REST telemetry endpoint until the SDK is upgraded.
+- Existing live rollups key by fallback path and collapse repeated cells.
+- Local-pack extraction has no real DOM fixture tests.
+- Optional page-snapshot transport failure can discard an otherwise successful SERP.
+
+## Official Kernel references
+
+- [Stealth mode and CAPTCHA handling](https://www.kernel.sh/docs/browsers/bot-detection/stealth)
+- [Bot-detection recommendations](https://www.kernel.sh/docs/browsers/bot-detection/overview)
+- [Telemetry overview](https://kernel.sh/docs/browsers/telemetry/overview.md)
+- [Telemetry categories](https://kernel.sh/docs/browsers/telemetry/categories.md)
+- [CAPTCHA telemetry schema](https://kernel.sh/docs/api-reference/browser-telemetry/stream-telemetry-events-via-sse.md)
+- [Telemetry readback](https://kernel.sh/docs/api-reference/browser-telemetry/read-telemetry-events-for-a-browser-session.md)
+- [ISP proxies](https://www.kernel.sh/docs/proxies/isp)
+- [Residential proxies](https://www.kernel.sh/docs/proxies/residential)
+- [Profiles](https://www.kernel.sh/docs/auth/profiles)
+- [Browser pools](https://www.kernel.sh/docs/browsers/pools/overview)