switchback 0.4.0__tar.gz → 0.6.0__tar.gz

{switchback-0.4.0 → switchback-0.6.0}/.env.example

@@ -11,7 +11,7 @@
 OTEL_SERVICE_NAME=switchback
 OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 
-# ── Search (Tier-0 SearXNG, query → URLs) ───────────────────────────────────
+# ── Search (SearXNG, query → URLs — separate from the fetch cascade) ─────────
 SEARXNG_URL=http://localhost:8888
 
 # ── Output format ───────────────────────────────────────────────────────────
@@ -27,29 +27,38 @@ SEARXNG_URL=http://localhost:8888
 # fall back to their text for those sources.
 SCRAPER_OUTPUT_FORMAT=markdown
 
-# ── Tier 2 · Cloudflare solver (cloudscraper) ───────────────────────────────
+# ── tier_3 · Cloudflare solver (cloudscraper) ───────────────────────────────
 # Needs the 3.x Enhanced Edition fork (see README); with the frozen PyPI build
 # the tier reports `unavailable` and fails fast. Wall-clock cap on a single solve
 # so an unsolvable challenge can't eat the per-URL deadline before the browser
-# tier runs. Lower (e.g. 12) if Tier 2 rarely wins on your hosts.
-SCRAPER_CLOUDSCRAPER_TIMEOUT_S=25
+# tier runs. Lower (e.g. 12) if tier_3 rarely wins on your hosts.
+SCRAPER_TIER_3_TIMEOUT_S=25
 
-# ── Tier 2.5 · Jina Reader (r.jina.ai) ──────────────────────────────────────
-# Optional: keyless works at 20 RPM. A key gives 500 RPM + a 10M-token grant.
-JINA_API_KEY=
-SCRAPER_JINA_TIMEOUT_S=20
-
-# ── Tier 3b · Camoufox (Firefox stealth) ────────────────────────────────────
+# ── tier_5 · Camoufox (Firefox stealth) ─────────────────────────────────────
 # ON by default. Needs: pip install camoufox && camoufox fetch
 # Set to 1 to turn the tier off entirely.
 SCRAPER_DISABLE_CAMOUFOX=
-SCRAPER_CAMOUFOX_TIMEOUT_MS=45000
+SCRAPER_TIER_5_TIMEOUT_S=45
 
-# ── Tier 4 · Firecrawl (paid, last resort) ──────────────────────────────────
+# ── tier_7 · Firecrawl (paid, last resort) ──────────────────────────────────
 # Required only if this tier runs. Set SCRAPER_DISABLE_FIRECRAWL=1 to skip it.
 FIRECRAWL_API_KEY=
 SCRAPER_DISABLE_FIRECRAWL=
 
+# ── Per-tier timeouts (seconds) ─────────────────────────────────────────────
+# Each tier's wall-clock/socket cap; override any of them. Defaults shown below.
+# tier_3 (=25) and tier_5 (=45) are set live in their sections above; the rest
+# fall back to these defaults. The pre-0.5.0 names are still honored when the new
+# var is unset: SCRAPER_CLOUDSCRAPER_TIMEOUT_S → tier_3,
+# SCRAPER_CAMOUFOX_TIMEOUT_MS → tier_5, SCRAPER_RESIDENTIAL_TIMEOUT_MS → tier_6.
+#SCRAPER_TIER_1_TIMEOUT_S=15    # direct APIs / open mirrors
+#SCRAPER_TIER_2_TIMEOUT_S=15    # plain HTTP + TLS impersonation
+#SCRAPER_TIER_3_TIMEOUT_S=25    # cloudscraper (Cloudflare solver)
+#SCRAPER_TIER_4_TIMEOUT_S=15    # stealth headless browser (patchright)
+#SCRAPER_TIER_5_TIMEOUT_S=45    # camoufox (slowest rung; hard CF solves ~40s)
+#SCRAPER_TIER_6_TIMEOUT_S=30    # residential-IP CDP browser
+#SCRAPER_TIER_7_TIMEOUT_S=15    # Firecrawl (paid) — was unbounded; raise if scrapes get cut off
+
 # ── Orchestrator ────────────────────────────────────────────────────────────
 # Per-URL wall-clock budget (s), checked between tiers. 45s balances latency vs
 # coverage — roughly fits a Camoufox solve (~40s) that starts after the cheaper

{switchback-0.4.0 → switchback-0.6.0}/CHANGELOG.md

@@ -4,7 +4,72 @@ All notable changes to this project are documented here. Format loosely follows
 [Keep a Changelog](https://keepachangelog.com/); this project uses semantic-ish
 versioning while pre-1.0.
 
-## [Unreleased]
+## [0.6.0] - 2026-10-14
+
+### Added
+- **`SCRAPER_DISABLE_CLOUDSCRAPER`** — opt-out for tier_3. Solving a JS challenge
+  runs site-supplied JavaScript in js2py or Node's `vm`, neither of which is a
+  security boundary; deployments that scrape URLs from untrusted users should turn
+  it off and let the browser tiers clear Cloudflare. Unset = unchanged. A host whose
+  learned winner was tier_3 falls through to the next tier rather than stalling,
+  and `--doctor` shows tier_3 as `off` without failing the healthcheck.
+- **`SCRAPER_BROWSER_BLOCK_PRIVATE`** — opt-in guard for tiers 4 and 5: every
+  request and WebSocket a page makes to a loopback, private, link-local (incl.
+  cloud metadata) or CGNAT address is refused, and service workers are blocked so
+  they can't bypass it. Unset = unchanged (no routing is installed at all).
+- **Agent-ready extras, all opt-in** (`scrape(..., state=, appendix=, query=)`,
+  the same flags on `scrape_detailed`, `--state` / `--appendix` / `--query` on the
+  CLI, and `state` / `appendix` / `query` in the HTTP `/scrape` body). Without them
+  results, CLI JSON and HTTP JSON are unchanged: `ScrapeResult` and `ScrapeOutcome`
+  keep exactly their fields. With them, results are the new subclasses
+  `ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras`.
+  - `state` — the page's embedded JSON (Next.js `__NEXT_DATA__`, Nuxt,
+    `window.__X__` globals, JSON-LD).
+  - `appendix` — nav/header/footer/aside as separate markdown regions.
+  - `query` → `highlights` — the passages that answer the question, each with a
+    text-fragment citation link. Graded by local BM25 by default, or by TypeSafe's
+    Jev model with `SCRAPER_HIGHLIGHTS_GRADER=jev` + `TYPESAFE_API_KEY` (falls back
+    to BM25 on any Jev error).
+- **MCP server** — `switchback-mcp` (stdio) exposes `scrape` and `search` as tools.
+  Install with `switchback[mcp]`.
+- **YouTube transcripts** in tier_1 (title, channel, timestamped transcript linked
+  to the moment in the video). Opt in with `SCRAPER_YOUTUBE=1` plus
+  `switchback[youtube]`; otherwise, or for videos with no captions, YouTube URLs
+  go through the cascade as before.
+  YouTube blocks most cloud IPs; a blocked request is retried once through
+  `SCRAPER_EGRESS_PROXY` when one is configured. Transcript highlights cite the
+  timestamp in the video.
+- **Office and ebook files** (.docx, .pptx, .xlsx, .xls, .epub) converted to
+  markdown in tier_2 via markitdown. Opt in with `SCRAPER_DOCUMENTS=1` plus
+  `switchback[documents]`; otherwise these URLs are handled exactly as before.
+
+The new extras are deliberately not added to `[all]`, so upgrading an existing
+`[all]` install pulls no new dependencies.
+
+## [0.5.0] - 2026-06-30
+
+### Changed
+- **Tiers renamed to plain `tier_1`…`tier_7`** (cost-ordered, contiguous) in place
+  of the old mixed scheme (`tier0_apis`, `tier1_http`, `tier2_cloudscraper`,
+  `tier3_browser`, `tier3b_camoufox`, `tier_residential`, `tier4_firecrawl`). The
+  mapping is positional: `tier_1`=apis, `tier_2`=http, `tier_3`=cloudscraper,
+  `tier_4`=browser, `tier_5`=camoufox, `tier_6`=residential, `tier_7`=firecrawl.
+  **Backwards-compatible:** an existing `state/botwall_db.json` is migrated on load
+  (a host's learned `winning_tier` / `tier_stats` keys are remapped to the new
+  names), so routing survives the upgrade instead of re-probing from scratch.
+
+### Added
+- **Per-tier timeout knobs** — every tier now reads `SCRAPER_TIER_<N>_TIMEOUT_S`
+  (seconds, `N` = 1–7). Defaults: `15` for tiers without a prior budget
+  (apis/http/browser), and the existing budgets are preserved — `tier_3`=25,
+  `tier_5`=45, `tier_6`=30. The previously-unconfigurable/unbounded tiers (apis,
+  http, browser, **firecrawl**) are now bounded and overridable. The pre-rename
+  `SCRAPER_CLOUDSCRAPER_TIMEOUT_S` / `SCRAPER_CAMOUFOX_TIMEOUT_MS` /
+  `SCRAPER_RESIDENTIAL_TIMEOUT_MS` are still honored when the new var is unset.
+  Note: `tier_7` (paid Firecrawl) was previously unbounded; its new `15`s default
+  bounds it — raise `SCRAPER_TIER_7_TIMEOUT_S` if hard hosts get cut off (a scrape
+  killed at the cap may still be billed). `SCRAPER_TIER_RETRIES_<TIER>` overrides
+  follow the new names (e.g. `SCRAPER_TIER_RETRIES_TIER_4`).
 
 ## [0.4.0] - 2026-06-29
 

switchback-0.6.0/CONTRIBUTING.md

@@ -0,0 +1,67 @@
+# Contributing
+
+Thanks for your interest in improving switchback.
+
+## Development setup
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[all]"
+patchright install chromium && camoufox fetch   # for the browser tiers
+```
+
+## Architecture
+The engine is a cost-ordered cascade (`switchback/tiers/`) governed by a per-host
+policy (`switchback/policy/`). Start with the cascade runner in
+`switchback/orchestrator.py`.
+
+## Guidelines
+- **Keep the core small.** Each tier imports its deps lazily and a missing dep is
+  just a tier miss — keep heavy/paid/optional pieces behind extras in
+  `pyproject.toml`.
+- **Make new behavior configurable and off-safe.** New features should be gated by
+  an env var that defaults to current behavior (see the existing `SCRAPER_*` vars).
+- **Match the surrounding style** — terse, comment-the-why, no speculative
+  abstractions.
+- **Don't commit secrets or run artifacts.** `.env`, `state/`, and `*.csv` are
+  gitignored; keep it that way.
+
+## Backward compatibility comes first
+Upgrading must never change what an existing call returns. Breaking a user's app
+is worse than any feature is good.
+
+What went wrong once: the agent extras added `state` / `appendix` / `highlights`
+as defaulted fields on `ScrapeResult` and `ScrapeOutcome`. That looked additive,
+but `asdict(r)` (how people serialise results) gained three `null` keys and
+`astuple(r)` grew from 4 values to 7. The CLI and HTTP JSON were fine; the Python
+objects were not. It was caught before merge and fixed with opt-in subclasses.
+
+Rules for any new feature:
+- **Never add, remove, rename or reorder fields on a public result type**
+  (`ScrapeResult`, `ScrapeOutcome`, `TierAttempt`, `SearchResult`). Put new data
+  on a subclass that is returned only when the caller opts in (see
+  `ScrapeResultWithExtras` in `switchback/orchestrator.py`).
+- **New parameters are keyword-only and default to the old behaviour.** Existing
+  parameters keep their names, order and defaults.
+- **CLI and HTTP JSON gain a key only when the request asked for it** (see
+  `result_dict` in `switchback/api.py`).
+- **A new dependency gets its own extra and stays out of `[all]`**, and a missing
+  dependency means the old behaviour. Installing a package is not opt-in: users
+  may already have it for something else. Anything that changes what an existing
+  call returns stays off until a `SCRAPER_*` variable turns it on (see
+  `SCRAPER_YOUTUBE`, `SCRAPER_DOCUMENTS`).
+- **The pinned tests are the contract.** The "backward compatibility" tests in
+  `tests/test_extras.py` pin the result fields and the CLI/HTTP JSON keys, and CI
+  runs them. If one fails, the change is breaking: redesign the change, don't
+  edit the pin.
+- **Before opening the PR, compare against `main` live:** scrape a few pages
+  with the default call on both; the output should be byte-identical.
+
+## Tests
+`pytest tests` runs the offline tests (fake tiers, no network, a few seconds);
+CI runs them on every PR. `tests/test_suite.py` exercises the cascade across the anti-bot difficulty
+spectrum (needs network + browser tiers): `python tests/test_suite.py --quick`
+for a fast tier-0/1 pass.
+
+## Pull requests
+Keep PRs focused; describe what changed and why, and note any new env var or
+endpoint in the README and CHANGELOG.

{switchback-0.4.0 → switchback-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: switchback
-Version: 0.4.0
+Version: 0.6.0
 Summary: One cost-ordered scrape cascade (HTTP → stealth browser → paid), shared by every tool.
 Author-email: Akash Kodavuru <akash@theaklabs.com>
 License: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/akash-kr/switchback
 Project-URL: Repository, https://github.com/akash-kr/switchback
 Project-URL: Issues, https://github.com/akash-kr/switchback/issues
 Project-URL: Changelog, https://github.com/akash-kr/switchback/blob/main/CHANGELOG.md
-Keywords: scraping,crawler,cloudflare,markdown,cascade,anti-bot,stealth
+Keywords: web-scraping,scraper,crawler,html-to-markdown,markdown,cloudflare-bypass,anti-bot,stealth,playwright,patchright,camoufox,curl-cffi,cloudscraper,firecrawl,headless-browser,residential-proxy,llm,rag,ai-agents,opentelemetry
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -41,6 +41,12 @@ Requires-Dist: firecrawl-py; extra == "firecrawl"
 Provides-Extra: server
 Requires-Dist: fastapi; extra == "server"
 Requires-Dist: uvicorn; extra == "server"
+Provides-Extra: mcp
+Requires-Dist: mcp; extra == "mcp"
+Provides-Extra: youtube
+Requires-Dist: youtube-transcript-api>=1.0; extra == "youtube"
+Provides-Extra: documents
+Requires-Dist: markitdown[docx,pptx,xls,xlsx]; extra == "documents"
 Provides-Extra: tracing
 Requires-Dist: opentelemetry-sdk; extra == "tracing"
 Requires-Dist: opentelemetry-exporter-otlp-proto-grpc; extra == "tracing"
@@ -70,6 +76,8 @@ Dynamic: license-file
 
 <div align="center">
 
+# switchback — Python web scraper: any URL → clean Markdown
+
 **One cost-ordered scrape cascade — HTTP → stealth browser → paid — shared by every tool.**
 
 Give it a URL; it tries the cheapest way to get clean Markdown first and only escalates
@@ -80,6 +88,8 @@ to a heavier (slower, costlier) tier when the cheap one is walled. Stops at the
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![CI](https://github.com/akash-kr/switchback/actions/workflows/ci.yml/badge.svg)](https://github.com/akash-kr/switchback/actions/workflows/ci.yml)
 
+⭐ If switchback saved you a paid-API bill, a star helps others find it.
+
 </div>
 
 ---
@@ -117,17 +127,25 @@ python -m switchback https://example.com/article    # JSON on stdout — bridge
 
 That's the whole loop. Add tiers as you need them (see [Install](#install)).
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/akash-kr/switchback/main/docs/assets/demo.gif" width="880"
+       alt="switchback scraping a Cloudflare-walled Quora page: tier_1 to tier_4 come up empty or walled, tier_5 (Camoufox) returns clean Markdown in 20.2s; on the second run botwall remembers the host and goes straight to tier_5">
+</p>
+
+<sub>A real run, time-compressed. Run 1 climbs the cascade until a tier gets through. Run 2 hits the
+same host, and botwall starts at the tier that won last time. The paid tier is never touched.</sub>
+
 ## The cascade (stop at first success)
 
 | Tier | Strategy | Cost |
 |---|---|---|
-| 0 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC; extend: job boards) | free, cleanest |
-| 1 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs | cheap |
-| 2 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) | cheap-ish (~5s/host) |
-| 3 | Stealth headless browser (`patchright`, Chromium) | heavy |
-| 3b | Camoufox (Firefox stealth) — **on by default** (opt out: `SCRAPER_DISABLE_CAMOUFOX`) | heavy + slow (~40s on hard CF) |
-| 3c | Residential-IP browser over CDP (`BU_CDP_URL`) — off unless configured | heavy (remote egress) |
-| 4 | Firecrawl (paid, env-gated, audited) | paid, last resort |
+| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC, YouTube transcripts; extend: job boards) | free, cleanest |
+| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs and office/ebook files | cheap |
+| tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) — on by default (opt out: `SCRAPER_DISABLE_CLOUDSCRAPER`) | cheap-ish (~5s/host) |
+| tier_4 | Stealth headless browser (`patchright`, Chromium) | heavy |
+| tier_5 | Camoufox (Firefox stealth) — **on by default** (opt out: `SCRAPER_DISABLE_CAMOUFOX`) | heavy + slow (~40s on hard CF) |
+| tier_6 | Residential-IP browser over CDP (`BU_CDP_URL`) — off unless configured | heavy (remote egress) |
+| tier_7 | Firecrawl (paid, env-gated, audited) | paid, last resort |
 
 Every URL has a wall-clock budget (`SCRAPER_DEADLINE_S`, default 45s) checked between
 tiers so one URL can't run the whole cascade of timeouts. Each tier attempt records
@@ -142,17 +160,26 @@ Search (query → URLs) is separate from the scrape cascade: `switchback.search(
 ## Install
 
 ```bash
-pip install switchback                 # core: normalization + cheap tiers (0/1) + search
-pip install "switchback[cloudflare]"   # + Tier 2 Cloudflare/anti-bot solver (cloudscraper)
+pip install switchback                 # core: normalization + cheap tiers (tier_1/tier_2) + search
+pip install "switchback[cloudflare]"   # + tier_3 Cloudflare/anti-bot solver (cloudscraper)
 pip install "switchback[server]"       # + HTTP service (fastapi, uvicorn) incl. /metrics + /traces
-pip install "switchback[browser]" && patchright install chromium   # + Tier 3 stealth Chromium
-pip install "switchback[camoufox]" && camoufox fetch               # + Tier 3b Firefox stealth
-pip install "switchback[firecrawl]"    # + Tier 4 paid API (needs FIRECRAWL_API_KEY)
+pip install "switchback[browser]" && patchright install chromium   # + tier_4 stealth Chromium
+pip install "switchback[camoufox]" && camoufox fetch               # + tier_5 Firefox stealth
+pip install "switchback[firecrawl]"    # + tier_7 paid API (needs FIRECRAWL_API_KEY)
 pip install "switchback[tracing]"      # + OpenTelemetry -> any OTLP backend
-pip install "switchback[all]"          # everything
+pip install "switchback[all]"          # all of the above
+pip install "switchback[mcp]"          # + MCP server for agents (switchback-mcp)
+pip install "switchback[youtube]"      # + YouTube transcripts in tier_1 (and set SCRAPER_YOUTUBE=1)
+pip install "switchback[documents]"    # + .docx/.pptx/.xlsx/.xls/.epub → markdown in tier_2 (and set SCRAPER_DOCUMENTS=1)
 ```
 
-For Tier 2's **full** v3 JS-VM + Turnstile + stealth, install the Enhanced Edition
+`mcp`, `youtube` and `documents` are not part of `[all]`, so upgrading an existing
+`[all]` install doesn't pull new dependencies (`documents` brings markitdown and
+onnxruntime). Add them by name. YouTube transcripts and office files also need
+their switch (`SCRAPER_YOUTUBE=1`, `SCRAPER_DOCUMENTS=1`), because they change what
+those URLs return; without it they behave exactly as before, installed or not.
+
+For tier_3's **full** v3 JS-VM + Turnstile + stealth, install the Enhanced Edition
 3.x fork (PyPI's `cloudscraper` is the older v1/v2 — PyPI forbids pinning a
 git-URL dep inside a published package, so install it alongside):
 
@@ -170,19 +197,44 @@ and land *after* boot (e.g. an async install thread on Azure). Until they're
 ready, those tiers report **`unavailable`** (a distinct outcome carrying the exact
 fix) and the cascade falls through — they are never silently skipped. Checklist:
 
-- **Tier 3 is the real workhorse for Cloudflare/JS sites** — make sure its browser
+- **tier_4 is the real workhorse for Cloudflare/JS sites** — make sure its browser
   is installed: `patchright install chromium` (note: **patchright**, not vanilla
   `playwright`). On a cold start, run this in your post-boot install step/thread;
-  Tier 3 flips to ready once it finishes.
-- **Tier 2 needs the cloudscraper 3.x fork** (above) to attempt stealth. With the
+  tier_4 flips to ready once it finishes.
+- **tier_3 needs the cloudscraper 3.x fork** (above) to attempt stealth. With the
   frozen PyPI `cloudscraper` it reports `unavailable` and fails fast (no wasted
-  solve budget) instead of erroring mid-cascade. Tier 2 is a *weak* solver for
+  solve budget) instead of erroring mid-cascade. tier_3 is a *weak* solver for
   modern Cloudflare — treat it as a cheap try before the browser, not the primary.
-- **Install Node.js** for Tier 2's v3 JS-VM challenges — faster and thread-safe
+- **Install Node.js** for tier_3's v3 JS-VM challenges — faster and thread-safe
   vs. the pure-Python js2py fallback (relevant under concurrent load).
-- **Bound Tier 2's solve budget** with `SCRAPER_CLOUDSCRAPER_TIMEOUT_S` (default
-  `25`) so an unsolvable challenge can't eat the per-URL deadline before the
-  browser tier runs. Lower it (e.g. `12`) if Tier 2 rarely wins on your hosts.
+- **Bound tier_3's solve budget** with `SCRAPER_TIER_3_TIMEOUT_S` (default `25`;
+  the old `SCRAPER_CLOUDSCRAPER_TIMEOUT_S` is still honored) so an unsolvable
+  challenge can't eat the per-URL deadline before the browser tier runs. Lower it
+  (e.g. `12`) if tier_3 rarely wins on your hosts.
+
+### Scraping URLs other people submit
+
+If the URLs come from users you don't fully trust (a "save this link" box, a
+feed reader, a public API), two opt-in switches harden the engine. Both default
+to off, so nothing changes unless you set them:
+
+```bash
+export SCRAPER_DISABLE_CLOUDSCRAPER=1   # tier_3 runs site JavaScript outside a browser sandbox
+export SCRAPER_BROWSER_BLOCK_PRIVATE=1  # browser tiers can't reach your machine's or network's private services
+```
+
+- **Why tier_3:** solving a Cloudflare JS challenge means running JavaScript the
+  site sent, in js2py or Node's `vm` module, and neither is a security boundary.
+  The browser tiers run page JavaScript inside the browser's own sandbox instead.
+- **Install tier_4's browser first** (`patchright install chromium`): with tier_3
+  off, it is what clears Cloudflare. `--doctor` shows tier_3 as `off` and still
+  exits 0.
+- **Why the browser guard:** a scraped page can redirect, frame or `fetch()` an
+  address like `http://127.0.0.1:…` or a cloud metadata endpoint, and whatever
+  answers comes back as scraped content. With the guard, those requests (and
+  WebSockets) are refused and service workers are disabled so they can't route
+  around it. Check the URL you pass in yourself too: the guard covers what the
+  *page* asks for in the browser tiers, not the HTTP tiers' own requests.
 
 **Verify readiness on the box** with the preflight check (doubles as a healthcheck
 — exit 0 when the capable tiers are ready):
@@ -226,6 +278,61 @@ Non-Python callers: see [clients/node_bridge.md](clients/node_bridge.md). Python
 callers that want HTTP-with-CLI-fallback can drop in
 [clients/python_client.py](clients/python_client.py).
 
+**MCP server** (for agents: Claude Code, Cursor, any MCP client)
+```bash
+pip install "switchback[mcp]"
+claude mcp add switchback -- switchback-mcp           # stdio; tools: scrape, search
+```
+
+## Agent-ready extras
+
+Off by default: without them every result is exactly what it was before, the same
+`ScrapeResult` / `ScrapeOutcome` with the same fields, and the same CLI and HTTP
+JSON. Ask for any combination per call; results then come back as
+`ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras` (subclasses, so existing
+`isinstance` checks still hold) with the extras you asked for filled in and the
+rest `None`. The CLI and HTTP JSON add only the keys you asked for.
+
+```python
+from switchback import scrape
+[r] = scrape("https://example.com/product", state=True, appendix=True,
+             query="what does it cost?")
+r.state       # {"__NEXT_DATA__": {...}, "json_ld": [...], "__INITIAL_STATE__": {...}}
+r.appendix    # [{"region": "nav", "markdown": "..."}, {"region": "footer", ...}]
+r.highlights  # [{"text": "...", "score": 7.1, "grader": "bm25",
+              #   "link": "https://example.com/product#:~:text=Price%20from,per%20month"}]
+```
+
+```bash
+switchback --state --appendix --query "what does it cost?" https://example.com/product
+curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"query":"pricing","state":true}'
+```
+
+| extra | what you get |
+| --- | --- |
+| `state` | The JSON the page embeds for its own frontend: Next.js `__NEXT_DATA__`, Nuxt, `window.__X__` globals (incl. `JSON.parse("…")`), schema.org JSON-LD. Often cleaner than the text (prices, ids, dates). Only valid JSON is returned. `{}` when the winning tier had no HTML (API mirrors, PDFs, docs, Firecrawl markdown). |
+| `appendix` | The menus, header, footer and sidebars that are stripped from the main content, as separate markdown, one entry per region. |
+| `query` → `highlights` | The passages that answer the question, best first, each with a citation `link`: a [text fragment](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments) URL that opens the page scrolled to, and highlighting, that passage (for a YouTube transcript, the video at that moment). If the text isn't visible on the page (e.g. it came from the `<title>` or image alt text) the browser simply opens the page at the top. |
+
+`state` and `appendix` need the fetched HTML, so they skip the content cache
+(`SCRAPER_CONTENT_TTL_S`) and fetch; `highlights` work on cached content too.
+
+**Grading highlights.** The default grader is local BM25 keyword scoring: free,
+offline, no data leaves the box. For better relevance, opt into
+[TypeSafe](https://typesafe.ai)'s Jev model, which answers "does this passage
+answer the question?" for every passage and returns a calibrated probability
+(passages under `SCRAPER_HIGHLIGHTS_MIN_SCORE`, default 0.5, are dropped, so an
+off-topic page honestly returns `[]`):
+
+```bash
+export SCRAPER_HIGHLIGHTS_GRADER=jev
+export TYPESAFE_API_KEY=...        # sends the page passages + query to api.typesafe.ai
+```
+
+Jev is billed per input token (about $0.0004 for a 10k-token page). If the call
+fails, that request falls back to BM25 (`"grader": "bm25"`); the scrape never fails
+because of highlights.
+
 ## Cost-scoped residential egress
 
 The dominant reason hard hosts wall you is the **datacenter IP**, not the
@@ -282,16 +389,18 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 <details>
 <summary><b>Env gates</b> — enable/disable tiers and integrations</summary>
 
-- `SCRAPER_DISABLE_FIRECRAWL` — skip Tier 4
-- `FIRECRAWL_API_KEY` — enable Tier 4
-- `SCRAPER_DISABLE_CAMOUFOX` — turn off Tier 3b (on by default; needs `pip install camoufox` + `camoufox fetch`)
-- `BU_CDP_URL` — enable Tier 3c residential browser by pointing at a CDP endpoint
+- `SCRAPER_DISABLE_FIRECRAWL` — skip tier_7
+- `FIRECRAWL_API_KEY` — enable tier_7
+- `SCRAPER_DISABLE_CAMOUFOX` — turn off tier_5 (on by default; needs `pip install camoufox` + `camoufox fetch`)
+- `SCRAPER_DISABLE_CLOUDSCRAPER` — turn off tier_3 (on by default). Recommended when you scrape URLs other people submit; see [Scraping URLs other people submit](#scraping-urls-other-people-submit)
+- `SCRAPER_BROWSER_BLOCK_PRIVATE` — opt-in, off by default: tiers 4 and 5 refuse any request a page makes to a loopback, private, link-local (incl. cloud metadata) or CGNAT address
+- `BU_CDP_URL` — enable tier_6 residential browser by pointing at a CDP endpoint
 - `SCRAPER_PROXY` — route *all* tiers/URLs through a proxy
 - `SCRAPER_EGRESS_PROXY` — route only walled hosts through a proxy (see [Cost-scoped residential egress](#cost-scoped-residential-egress))
 - `SEARXNG_URL` — defaults to `http://localhost:8888`
 - `SCRAPER_STATE_DIR` — where the botwall DB/event log + session cache live
 - `SCRAPER_COOKIES_FILE` — Netscape `cookies.txt` to scrape login-gated hosts (injected into the HTTP and browser tiers)
-- `SCRAPER_CAPTCHA_PROVIDER` + `SCRAPER_CAPTCHA_API_KEY` — opt-in, off by default: wire a third-party solver (2captcha/capsolver/capmonster/anticaptcha/deathbycaptcha/9kw) into Tier 2 for Turnstile/reCAPTCHA/hCaptcha on CF hosts. **Paid**, billed per solve by the provider.
+- `SCRAPER_CAPTCHA_PROVIDER` + `SCRAPER_CAPTCHA_API_KEY` — opt-in, off by default: wire a third-party solver (2captcha/capsolver/capmonster/anticaptcha/deathbycaptcha/9kw) into tier_3 for Turnstile/reCAPTCHA/hCaptcha on CF hosts. **Paid**, billed per solve by the provider.
 </details>
 
 <details>
@@ -300,7 +409,7 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_OUTPUT_FORMAT` — output shape: `markdown` (default) · `markdown_trimmed` · `html` · `html_selectors` (see [Output formats](#output-formats))
 - `SCRAPER_DEADLINE_S` — per-URL budget (45s)
 - `SCRAPER_FIRECRAWL_FALLBACK_AFTER_S` — after this many seconds on a URL, stop trying the local tiers and fall back to Firecrawl, so a hard host doesn't burn the whole deadline before the paid last resort gets a turn (25s; 0 = off)
-- `SCRAPER_CAMOUFOX_TIMEOUT_MS` — (45000)
+- `SCRAPER_TIER_<N>_TIMEOUT_S` — per-tier timeout in seconds, `N` = 1–7 (tier_1 apis · tier_2 http · tier_3 cloudscraper · tier_4 browser · tier_5 camoufox · tier_6 residential · tier_7 firecrawl). Defaults: 15/15/**25**/15/**45**/**30**/15 (the three bold ones keep their prior budgets; everything else is 15s). The pre-0.5.0 `SCRAPER_CLOUDSCRAPER_TIMEOUT_S` / `SCRAPER_CAMOUFOX_TIMEOUT_MS` / `SCRAPER_RESIDENTIAL_TIMEOUT_MS` are still honored when the new var is unset. Note: `tier_7` (Firecrawl) was previously unbounded — its 15s default now bounds the paid tier, so raise `SCRAPER_TIER_7_TIMEOUT_S` if slow hosts get cut off
 - `SCRAPER_BROWSER_CONCURRENCY` — max simultaneous headless browsers (default 1)
 - `SCRAPER_BOTWALL_URL_SKIP_COOLDOWN_H` — auto-skip re-test window (24h; 0 = never)
 - `SCRAPER_BOTWALL_EGRESS_AFTER` — local-tier failures before a host escalates to the residential tier (default 2)
@@ -308,10 +417,17 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_DISABLE_SESSION_CACHE` — turn off cf_clearance reuse
 - `SCRAPER_CONTENT_TTL_S` — URL→result cache TTL (**0 = off**; set e.g. 86400 to skip re-scraping a page within a day)
 - `SCRAPER_BACKOFF_BASE_MS` / `SCRAPER_BACKOFF_MAX_MS` — exponential backoff between tiers after a rate-limit/timeout (base 0 = off)
-- `SCRAPER_TIER_RETRIES` — same-tier retries before falling through (default 0 = off; `N` → up to `1+N` tries per tier), with per-tier overrides `SCRAPER_TIER_RETRIES_<TIER>` (e.g. `SCRAPER_TIER_RETRIES_TIER3_BROWSER=2`)
+- `SCRAPER_TIER_RETRIES` — same-tier retries before falling through (default 0 = off; `N` → up to `1+N` tries per tier), with per-tier overrides `SCRAPER_TIER_RETRIES_<TIER>` (e.g. `SCRAPER_TIER_RETRIES_TIER_4=2`)
 - `SCRAPER_TIER_RETRY_ON` — failure classes eligible for a same-tier retry (default `timeout,rate_limited,connection`; widen to include `botwall,http_block` behind a rotating residential proxy so each retry gets a fresh IP). Retries are bounded by `SCRAPER_DEADLINE_S`; enabling them on the paid Firecrawl tier bills per attempt
 - `SCRAPER_LOGIN_HOOK` — `pkg.module:func` returning `{cookie: value}` for a host (see [Logged-in sessions](#logged-in-sessions))
 - `SCRAPER_EXTRACTION_FILE` — per-domain extraction prefs JSON (default `config/extraction.json`)
+- `SCRAPER_HIGHLIGHTS_GRADER` — `bm25` (default) or `jev` (needs `TYPESAFE_API_KEY`; see [Agent-ready extras](#agent-ready-extras))
+- `SCRAPER_HIGHLIGHTS_TOP_K` — highlights returned per URL (5)
+- `SCRAPER_HIGHLIGHTS_MIN_SCORE` — Jev probability below which a passage is dropped (0.5)
+- `SCRAPER_JEV_MODEL` / `SCRAPER_JEV_TIMEOUT_S` — Jev model alias (`jev-latest`) and per-request timeout (20s)
+- `SCRAPER_YOUTUBE` — opt-in: `1` returns YouTube transcripts from tier_1 (needs `switchback[youtube]`)
+- `SCRAPER_DOCUMENTS` — opt-in: `1` converts .docx/.pptx/.xlsx/.xls/.epub to markdown in tier_2 (needs `switchback[documents]`)
+- `SCRAPER_YOUTUBE_LANGS` — transcript languages to try, in order (`en`; e.g. `en,hi`). YouTube blocks most cloud IPs; when it does, the transcript is retried once through `SCRAPER_EGRESS_PROXY` if set
 - `SCRAPER_TRACE_SESSION` — opt-in: capture a Playwright trace (screenshots + DOM + network) per browser-tier attempt, written to `state/traces/`
 - `BENCH_FIRECRAWL_USD` / `BENCH_FIRECRAWL_HARD_MULT` — cost model for the savings report
 </details>
@@ -363,6 +479,20 @@ strip site-specific noise, declare prefs per host in `config/extraction.json`
 (see [config/extraction.example.json](config/extraction.example.json)); every
 tier's normalize step picks them up automatically.
 
+## Roadmap
+
+Not built yet:
+
+- **Layout-aware markdown** — keep pricing grids and card layouts built from
+  `<div>`s (not `<table>`s) as tables, using the browser tiers' rendered layout.
+- **Scanned pages and images** — OCR / vision-model text for PDFs with no text
+  layer and for images.
+- **Audio and caption-less video** — speech-to-text transcription.
+- **Hosted service** — managed browsers at high concurrency, stored results and
+  shareable result links. switchback stays a library you run yourself.
+- Smaller gaps: Next.js App Router flight data (`self.__next_f`) in `state`, and
+  YouTube chapters in transcripts.
+
 ## Contributing
 
 Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Start with the