switchback 0.5.0__tar.gz → 0.6.0__tar.gz

{switchback-0.5.0 → switchback-0.6.0}/CHANGELOG.md

@@ -4,7 +4,47 @@ All notable changes to this project are documented here. Format loosely follows
 [Keep a Changelog](https://keepachangelog.com/); this project uses semantic-ish
 versioning while pre-1.0.
 
-## [Unreleased]
+## [0.6.0] - 2026-10-14
+
+### Added
+- **`SCRAPER_DISABLE_CLOUDSCRAPER`** — opt-out for tier_3. Solving a JS challenge
+  runs site-supplied JavaScript in js2py or Node's `vm`, neither of which is a
+  security boundary; deployments that scrape URLs from untrusted users should turn
+  it off and let the browser tiers clear Cloudflare. Unset = unchanged. A host whose
+  learned winner was tier_3 falls through to the next tier rather than stalling,
+  and `--doctor` shows tier_3 as `off` without failing the healthcheck.
+- **`SCRAPER_BROWSER_BLOCK_PRIVATE`** — opt-in guard for tiers 4 and 5: every
+  request and WebSocket a page makes to a loopback, private, link-local (incl.
+  cloud metadata) or CGNAT address is refused, and service workers are blocked so
+  they can't bypass it. Unset = unchanged (no routing is installed at all).
+- **Agent-ready extras, all opt-in** (`scrape(..., state=, appendix=, query=)`,
+  the same flags on `scrape_detailed`, `--state` / `--appendix` / `--query` on the
+  CLI, and `state` / `appendix` / `query` in the HTTP `/scrape` body). Without them
+  results, CLI JSON and HTTP JSON are unchanged: `ScrapeResult` and `ScrapeOutcome`
+  keep exactly their fields. With them, results are the new subclasses
+  `ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras`.
+  - `state` — the page's embedded JSON (Next.js `__NEXT_DATA__`, Nuxt,
+    `window.__X__` globals, JSON-LD).
+  - `appendix` — nav/header/footer/aside as separate markdown regions.
+  - `query` → `highlights` — the passages that answer the question, each with a
+    text-fragment citation link. Graded by local BM25 by default, or by TypeSafe's
+    Jev model with `SCRAPER_HIGHLIGHTS_GRADER=jev` + `TYPESAFE_API_KEY` (falls back
+    to BM25 on any Jev error).
+- **MCP server** — `switchback-mcp` (stdio) exposes `scrape` and `search` as tools.
+  Install with `switchback[mcp]`.
+- **YouTube transcripts** in tier_1 (title, channel, timestamped transcript linked
+  to the moment in the video). Opt in with `SCRAPER_YOUTUBE=1` plus
+  `switchback[youtube]`; otherwise, or for videos with no captions, YouTube URLs
+  go through the cascade as before.
+  YouTube blocks most cloud IPs; a blocked request is retried once through
+  `SCRAPER_EGRESS_PROXY` when one is configured. Transcript highlights cite the
+  timestamp in the video.
+- **Office and ebook files** (.docx, .pptx, .xlsx, .xls, .epub) converted to
+  markdown in tier_2 via markitdown. Opt in with `SCRAPER_DOCUMENTS=1` plus
+  `switchback[documents]`; otherwise these URLs are handled exactly as before.
+
+The new extras are deliberately not added to `[all]`, so upgrading an existing
+`[all]` install pulls no new dependencies.
 
 ## [0.5.0] - 2026-06-30
 

switchback-0.6.0/CONTRIBUTING.md

@@ -0,0 +1,67 @@
+# Contributing
+
+Thanks for your interest in improving switchback.
+
+## Development setup
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[all]"
+patchright install chromium && camoufox fetch   # for the browser tiers
+```
+
+## Architecture
+The engine is a cost-ordered cascade (`switchback/tiers/`) governed by a per-host
+policy (`switchback/policy/`). Start with the cascade runner in
+`switchback/orchestrator.py`.
+
+## Guidelines
+- **Keep the core small.** Each tier imports its deps lazily and a missing dep is
+  just a tier miss — keep heavy/paid/optional pieces behind extras in
+  `pyproject.toml`.
+- **Make new behavior configurable and off-safe.** New features should be gated by
+  an env var that defaults to current behavior (see the existing `SCRAPER_*` vars).
+- **Match the surrounding style** — terse, comment-the-why, no speculative
+  abstractions.
+- **Don't commit secrets or run artifacts.** `.env`, `state/`, and `*.csv` are
+  gitignored; keep it that way.
+
+## Backward compatibility comes first
+Upgrading must never change what an existing call returns. Breaking a user's app
+is worse than any feature is good.
+
+What went wrong once: the agent extras added `state` / `appendix` / `highlights`
+as defaulted fields on `ScrapeResult` and `ScrapeOutcome`. That looked additive,
+but `asdict(r)` (how people serialise results) gained three `null` keys and
+`astuple(r)` grew from 4 values to 7. The CLI and HTTP JSON were fine; the Python
+objects were not. It was caught before merge and fixed with opt-in subclasses.
+
+Rules for any new feature:
+- **Never add, remove, rename or reorder fields on a public result type**
+  (`ScrapeResult`, `ScrapeOutcome`, `TierAttempt`, `SearchResult`). Put new data
+  on a subclass that is returned only when the caller opts in (see
+  `ScrapeResultWithExtras` in `switchback/orchestrator.py`).
+- **New parameters are keyword-only and default to the old behaviour.** Existing
+  parameters keep their names, order and defaults.
+- **CLI and HTTP JSON gain a key only when the request asked for it** (see
+  `result_dict` in `switchback/api.py`).
+- **A new dependency gets its own extra and stays out of `[all]`**, and a missing
+  dependency means the old behaviour. Installing a package is not opt-in: users
+  may already have it for something else. Anything that changes what an existing
+  call returns stays off until a `SCRAPER_*` variable turns it on (see
+  `SCRAPER_YOUTUBE`, `SCRAPER_DOCUMENTS`).
+- **The pinned tests are the contract.** The "backward compatibility" tests in
+  `tests/test_extras.py` pin the result fields and the CLI/HTTP JSON keys, and CI
+  runs them. If one fails, the change is breaking: redesign the change, don't
+  edit the pin.
+- **Before opening the PR, compare against `main` live:** scrape a few pages
+  with the default call on both; the output should be byte-identical.
+
+## Tests
+`pytest tests` runs the offline tests (fake tiers, no network, a few seconds);
+CI runs them on every PR. `tests/test_suite.py` exercises the cascade across the anti-bot difficulty
+spectrum (needs network + browser tiers): `python tests/test_suite.py --quick`
+for a fast tier-0/1 pass.
+
+## Pull requests
+Keep PRs focused; describe what changed and why, and note any new env var or
+endpoint in the README and CHANGELOG.

{switchback-0.5.0 → switchback-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: switchback
-Version: 0.5.0
+Version: 0.6.0
 Summary: One cost-ordered scrape cascade (HTTP → stealth browser → paid), shared by every tool.
 Author-email: Akash Kodavuru <akash@theaklabs.com>
 License: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/akash-kr/switchback
 Project-URL: Repository, https://github.com/akash-kr/switchback
 Project-URL: Issues, https://github.com/akash-kr/switchback/issues
 Project-URL: Changelog, https://github.com/akash-kr/switchback/blob/main/CHANGELOG.md
-Keywords: scraping,crawler,cloudflare,markdown,cascade,anti-bot,stealth
+Keywords: web-scraping,scraper,crawler,html-to-markdown,markdown,cloudflare-bypass,anti-bot,stealth,playwright,patchright,camoufox,curl-cffi,cloudscraper,firecrawl,headless-browser,residential-proxy,llm,rag,ai-agents,opentelemetry
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -41,6 +41,12 @@ Requires-Dist: firecrawl-py; extra == "firecrawl"
 Provides-Extra: server
 Requires-Dist: fastapi; extra == "server"
 Requires-Dist: uvicorn; extra == "server"
+Provides-Extra: mcp
+Requires-Dist: mcp; extra == "mcp"
+Provides-Extra: youtube
+Requires-Dist: youtube-transcript-api>=1.0; extra == "youtube"
+Provides-Extra: documents
+Requires-Dist: markitdown[docx,pptx,xls,xlsx]; extra == "documents"
 Provides-Extra: tracing
 Requires-Dist: opentelemetry-sdk; extra == "tracing"
 Requires-Dist: opentelemetry-exporter-otlp-proto-grpc; extra == "tracing"
@@ -70,6 +76,8 @@ Dynamic: license-file
 
 <div align="center">
 
+# switchback — Python web scraper: any URL → clean Markdown
+
 **One cost-ordered scrape cascade — HTTP → stealth browser → paid — shared by every tool.**
 
 Give it a URL; it tries the cheapest way to get clean Markdown first and only escalates
@@ -80,6 +88,8 @@ to a heavier (slower, costlier) tier when the cheap one is walled. Stops at the
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![CI](https://github.com/akash-kr/switchback/actions/workflows/ci.yml/badge.svg)](https://github.com/akash-kr/switchback/actions/workflows/ci.yml)
 
+⭐ If switchback saved you a paid-API bill, a star helps others find it.
+
 </div>
 
 ---
@@ -117,13 +127,21 @@ python -m switchback https://example.com/article    # JSON on stdout — bridge
 
 That's the whole loop. Add tiers as you need them (see [Install](#install)).
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/akash-kr/switchback/main/docs/assets/demo.gif" width="880"
+       alt="switchback scraping a Cloudflare-walled Quora page: tier_1 to tier_4 come up empty or walled, tier_5 (Camoufox) returns clean Markdown in 20.2s; on the second run botwall remembers the host and goes straight to tier_5">
+</p>
+
+<sub>A real run, time-compressed. Run 1 climbs the cascade until a tier gets through. Run 2 hits the
+same host, and botwall starts at the tier that won last time. The paid tier is never touched.</sub>
+
 ## The cascade (stop at first success)
 
 | Tier | Strategy | Cost |
 |---|---|---|
-| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC; extend: job boards) | free, cleanest |
-| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs | cheap |
-| tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) | cheap-ish (~5s/host) |
+| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC, YouTube transcripts; extend: job boards) | free, cleanest |
+| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs and office/ebook files | cheap |
+| tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) — on by default (opt out: `SCRAPER_DISABLE_CLOUDSCRAPER`) | cheap-ish (~5s/host) |
 | tier_4 | Stealth headless browser (`patchright`, Chromium) | heavy |
 | tier_5 | Camoufox (Firefox stealth) — **on by default** (opt out: `SCRAPER_DISABLE_CAMOUFOX`) | heavy + slow (~40s on hard CF) |
 | tier_6 | Residential-IP browser over CDP (`BU_CDP_URL`) — off unless configured | heavy (remote egress) |
@@ -149,9 +167,18 @@ pip install "switchback[browser]" && patchright install chromium   # + tier_4 st
 pip install "switchback[camoufox]" && camoufox fetch               # + tier_5 Firefox stealth
 pip install "switchback[firecrawl]"    # + tier_7 paid API (needs FIRECRAWL_API_KEY)
 pip install "switchback[tracing]"      # + OpenTelemetry -> any OTLP backend
-pip install "switchback[all]"          # everything
+pip install "switchback[all]"          # all of the above
+pip install "switchback[mcp]"          # + MCP server for agents (switchback-mcp)
+pip install "switchback[youtube]"      # + YouTube transcripts in tier_1 (and set SCRAPER_YOUTUBE=1)
+pip install "switchback[documents]"    # + .docx/.pptx/.xlsx/.xls/.epub → markdown in tier_2 (and set SCRAPER_DOCUMENTS=1)
 ```
 
+`mcp`, `youtube` and `documents` are not part of `[all]`, so upgrading an existing
+`[all]` install doesn't pull new dependencies (`documents` brings markitdown and
+onnxruntime). Add them by name. YouTube transcripts and office files also need
+their switch (`SCRAPER_YOUTUBE=1`, `SCRAPER_DOCUMENTS=1`), because they change what
+those URLs return; without it they behave exactly as before, installed or not.
+
 For tier_3's **full** v3 JS-VM + Turnstile + stealth, install the Enhanced Edition
 3.x fork (PyPI's `cloudscraper` is the older v1/v2 — PyPI forbids pinning a
 git-URL dep inside a published package, so install it alongside):
@@ -185,6 +212,30 @@ fix) and the cascade falls through — they are never silently skipped. Checklis
   challenge can't eat the per-URL deadline before the browser tier runs. Lower it
   (e.g. `12`) if tier_3 rarely wins on your hosts.
 
+### Scraping URLs other people submit
+
+If the URLs come from users you don't fully trust (a "save this link" box, a
+feed reader, a public API), two opt-in switches harden the engine. Both default
+to off, so nothing changes unless you set them:
+
+```bash
+export SCRAPER_DISABLE_CLOUDSCRAPER=1   # tier_3 runs site JavaScript outside a browser sandbox
+export SCRAPER_BROWSER_BLOCK_PRIVATE=1  # browser tiers can't reach your machine's or network's private services
+```
+
+- **Why tier_3:** solving a Cloudflare JS challenge means running JavaScript the
+  site sent, in js2py or Node's `vm` module, and neither is a security boundary.
+  The browser tiers run page JavaScript inside the browser's own sandbox instead.
+- **Install tier_4's browser first** (`patchright install chromium`): with tier_3
+  off, it is what clears Cloudflare. `--doctor` shows tier_3 as `off` and still
+  exits 0.
+- **Why the browser guard:** a scraped page can redirect, frame or `fetch()` an
+  address like `http://127.0.0.1:…` or a cloud metadata endpoint, and whatever
+  answers comes back as scraped content. With the guard, those requests (and
+  WebSockets) are refused and service workers are disabled so they can't route
+  around it. Check the URL you pass in yourself too: the guard covers what the
+  *page* asks for in the browser tiers, not the HTTP tiers' own requests.
+
 **Verify readiness on the box** with the preflight check (doubles as a healthcheck
 — exit 0 when the capable tiers are ready):
 
@@ -227,6 +278,61 @@ Non-Python callers: see [clients/node_bridge.md](clients/node_bridge.md). Python
 callers that want HTTP-with-CLI-fallback can drop in
 [clients/python_client.py](clients/python_client.py).
 
+**MCP server** (for agents: Claude Code, Cursor, any MCP client)
+```bash
+pip install "switchback[mcp]"
+claude mcp add switchback -- switchback-mcp           # stdio; tools: scrape, search
+```
+
+## Agent-ready extras
+
+Off by default: without them every result is exactly what it was before, the same
+`ScrapeResult` / `ScrapeOutcome` with the same fields, and the same CLI and HTTP
+JSON. Ask for any combination per call; results then come back as
+`ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras` (subclasses, so existing
+`isinstance` checks still hold) with the extras you asked for filled in and the
+rest `None`. The CLI and HTTP JSON add only the keys you asked for.
+
+```python
+from switchback import scrape
+[r] = scrape("https://example.com/product", state=True, appendix=True,
+             query="what does it cost?")
+r.state       # {"__NEXT_DATA__": {...}, "json_ld": [...], "__INITIAL_STATE__": {...}}
+r.appendix    # [{"region": "nav", "markdown": "..."}, {"region": "footer", ...}]
+r.highlights  # [{"text": "...", "score": 7.1, "grader": "bm25",
+              #   "link": "https://example.com/product#:~:text=Price%20from,per%20month"}]
+```
+
+```bash
+switchback --state --appendix --query "what does it cost?" https://example.com/product
+curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"query":"pricing","state":true}'
+```
+
+| extra | what you get |
+| --- | --- |
+| `state` | The JSON the page embeds for its own frontend: Next.js `__NEXT_DATA__`, Nuxt, `window.__X__` globals (incl. `JSON.parse("…")`), schema.org JSON-LD. Often cleaner than the text (prices, ids, dates). Only valid JSON is returned. `{}` when the winning tier had no HTML (API mirrors, PDFs, docs, Firecrawl markdown). |
+| `appendix` | The menus, header, footer and sidebars that are stripped from the main content, as separate markdown, one entry per region. |
+| `query` → `highlights` | The passages that answer the question, best first, each with a citation `link`: a [text fragment](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments) URL that opens the page scrolled to, and highlighting, that passage (for a YouTube transcript, the video at that moment). If the text isn't visible on the page (e.g. it came from the `<title>` or image alt text) the browser simply opens the page at the top. |
+
+`state` and `appendix` need the fetched HTML, so they skip the content cache
+(`SCRAPER_CONTENT_TTL_S`) and fetch; `highlights` work on cached content too.
+
+**Grading highlights.** The default grader is local BM25 keyword scoring: free,
+offline, no data leaves the box. For better relevance, opt into
+[TypeSafe](https://typesafe.ai)'s Jev model, which answers "does this passage
+answer the question?" for every passage and returns a calibrated probability
+(passages under `SCRAPER_HIGHLIGHTS_MIN_SCORE`, default 0.5, are dropped, so an
+off-topic page honestly returns `[]`):
+
+```bash
+export SCRAPER_HIGHLIGHTS_GRADER=jev
+export TYPESAFE_API_KEY=...        # sends the page passages + query to api.typesafe.ai
+```
+
+Jev is billed per input token (about $0.0004 for a 10k-token page). If the call
+fails, that request falls back to BM25 (`"grader": "bm25"`); the scrape never fails
+because of highlights.
+
 ## Cost-scoped residential egress
 
 The dominant reason hard hosts wall you is the **datacenter IP**, not the
@@ -286,6 +392,8 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_DISABLE_FIRECRAWL` — skip tier_7
 - `FIRECRAWL_API_KEY` — enable tier_7
 - `SCRAPER_DISABLE_CAMOUFOX` — turn off tier_5 (on by default; needs `pip install camoufox` + `camoufox fetch`)
+- `SCRAPER_DISABLE_CLOUDSCRAPER` — turn off tier_3 (on by default). Recommended when you scrape URLs other people submit; see [Scraping URLs other people submit](#scraping-urls-other-people-submit)
+- `SCRAPER_BROWSER_BLOCK_PRIVATE` — opt-in, off by default: tiers 4 and 5 refuse any request a page makes to a loopback, private, link-local (incl. cloud metadata) or CGNAT address
 - `BU_CDP_URL` — enable tier_6 residential browser by pointing at a CDP endpoint
 - `SCRAPER_PROXY` — route *all* tiers/URLs through a proxy
 - `SCRAPER_EGRESS_PROXY` — route only walled hosts through a proxy (see [Cost-scoped residential egress](#cost-scoped-residential-egress))
@@ -313,6 +421,13 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_TIER_RETRY_ON` — failure classes eligible for a same-tier retry (default `timeout,rate_limited,connection`; widen to include `botwall,http_block` behind a rotating residential proxy so each retry gets a fresh IP). Retries are bounded by `SCRAPER_DEADLINE_S`; enabling them on the paid Firecrawl tier bills per attempt
 - `SCRAPER_LOGIN_HOOK` — `pkg.module:func` returning `{cookie: value}` for a host (see [Logged-in sessions](#logged-in-sessions))
 - `SCRAPER_EXTRACTION_FILE` — per-domain extraction prefs JSON (default `config/extraction.json`)
+- `SCRAPER_HIGHLIGHTS_GRADER` — `bm25` (default) or `jev` (needs `TYPESAFE_API_KEY`; see [Agent-ready extras](#agent-ready-extras))
+- `SCRAPER_HIGHLIGHTS_TOP_K` — highlights returned per URL (5)
+- `SCRAPER_HIGHLIGHTS_MIN_SCORE` — Jev probability below which a passage is dropped (0.5)
+- `SCRAPER_JEV_MODEL` / `SCRAPER_JEV_TIMEOUT_S` — Jev model alias (`jev-latest`) and per-request timeout (20s)
+- `SCRAPER_YOUTUBE` — opt-in: `1` returns YouTube transcripts from tier_1 (needs `switchback[youtube]`)
+- `SCRAPER_DOCUMENTS` — opt-in: `1` converts .docx/.pptx/.xlsx/.xls/.epub to markdown in tier_2 (needs `switchback[documents]`)
+- `SCRAPER_YOUTUBE_LANGS` — transcript languages to try, in order (`en`; e.g. `en,hi`). YouTube blocks most cloud IPs; when it does, the transcript is retried once through `SCRAPER_EGRESS_PROXY` if set
 - `SCRAPER_TRACE_SESSION` — opt-in: capture a Playwright trace (screenshots + DOM + network) per browser-tier attempt, written to `state/traces/`
 - `BENCH_FIRECRAWL_USD` / `BENCH_FIRECRAWL_HARD_MULT` — cost model for the savings report
 </details>
@@ -364,6 +479,20 @@ strip site-specific noise, declare prefs per host in `config/extraction.json`
 (see [config/extraction.example.json](config/extraction.example.json)); every
 tier's normalize step picks them up automatically.
 
+## Roadmap
+
+Not built yet:
+
+- **Layout-aware markdown** — keep pricing grids and card layouts built from
+  `<div>`s (not `<table>`s) as tables, using the browser tiers' rendered layout.
+- **Scanned pages and images** — OCR / vision-model text for PDFs with no text
+  layer and for images.
+- **Audio and caption-less video** — speech-to-text transcription.
+- **Hosted service** — managed browsers at high concurrency, stored results and
+  shareable result links. switchback stays a library you run yourself.
+- Smaller gaps: Next.js App Router flight data (`self.__next_f`) in `state`, and
+  YouTube chapters in transcripts.
+
 ## Contributing
 
 Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Start with the

{switchback-0.5.0 → switchback-0.6.0}/README.md

@@ -11,6 +11,8 @@
 
 <div align="center">
 
+# switchback — Python web scraper: any URL → clean Markdown
+
 **One cost-ordered scrape cascade — HTTP → stealth browser → paid — shared by every tool.**
 
 Give it a URL; it tries the cheapest way to get clean Markdown first and only escalates
@@ -21,6 +23,8 @@ to a heavier (slower, costlier) tier when the cheap one is walled. Stops at the
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 [![CI](https://github.com/akash-kr/switchback/actions/workflows/ci.yml/badge.svg)](https://github.com/akash-kr/switchback/actions/workflows/ci.yml)
 
+⭐ If switchback saved you a paid-API bill, a star helps others find it.
+
 </div>
 
 ---
@@ -58,13 +62,21 @@ python -m switchback https://example.com/article    # JSON on stdout — bridge
 
 That's the whole loop. Add tiers as you need them (see [Install](#install)).
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/akash-kr/switchback/main/docs/assets/demo.gif" width="880"
+       alt="switchback scraping a Cloudflare-walled Quora page: tier_1 to tier_4 come up empty or walled, tier_5 (Camoufox) returns clean Markdown in 20.2s; on the second run botwall remembers the host and goes straight to tier_5">
+</p>
+
+<sub>A real run, time-compressed. Run 1 climbs the cascade until a tier gets through. Run 2 hits the
+same host, and botwall starts at the tier that won last time. The paid tier is never touched.</sub>
+
 ## The cascade (stop at first success)
 
 | Tier | Strategy | Cost |
 |---|---|---|
-| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC; extend: job boards) | free, cleanest |
-| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs | cheap |
-| tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) | cheap-ish (~5s/host) |
+| tier_1 | Direct APIs / mirrors (arxiv, wikipedia, EuropePMC, YouTube transcripts; extend: job boards) | free, cleanest |
+| tier_2 | Plain HTTP + TLS impersonation (`curl_cffi`), incl. PDFs and office/ebook files | cheap |
+| tier_3 | Cloudflare / anti-bot solver (`cloudscraper`, install `.[cloudflare]`) — on by default (opt out: `SCRAPER_DISABLE_CLOUDSCRAPER`) | cheap-ish (~5s/host) |
 | tier_4 | Stealth headless browser (`patchright`, Chromium) | heavy |
 | tier_5 | Camoufox (Firefox stealth) — **on by default** (opt out: `SCRAPER_DISABLE_CAMOUFOX`) | heavy + slow (~40s on hard CF) |
 | tier_6 | Residential-IP browser over CDP (`BU_CDP_URL`) — off unless configured | heavy (remote egress) |
@@ -90,9 +102,18 @@ pip install "switchback[browser]" && patchright install chromium   # + tier_4 st
 pip install "switchback[camoufox]" && camoufox fetch               # + tier_5 Firefox stealth
 pip install "switchback[firecrawl]"    # + tier_7 paid API (needs FIRECRAWL_API_KEY)
 pip install "switchback[tracing]"      # + OpenTelemetry -> any OTLP backend
-pip install "switchback[all]"          # everything
+pip install "switchback[all]"          # all of the above
+pip install "switchback[mcp]"          # + MCP server for agents (switchback-mcp)
+pip install "switchback[youtube]"      # + YouTube transcripts in tier_1 (and set SCRAPER_YOUTUBE=1)
+pip install "switchback[documents]"    # + .docx/.pptx/.xlsx/.xls/.epub → markdown in tier_2 (and set SCRAPER_DOCUMENTS=1)
 ```
 
+`mcp`, `youtube` and `documents` are not part of `[all]`, so upgrading an existing
+`[all]` install doesn't pull new dependencies (`documents` brings markitdown and
+onnxruntime). Add them by name. YouTube transcripts and office files also need
+their switch (`SCRAPER_YOUTUBE=1`, `SCRAPER_DOCUMENTS=1`), because they change what
+those URLs return; without it they behave exactly as before, installed or not.
+
 For tier_3's **full** v3 JS-VM + Turnstile + stealth, install the Enhanced Edition
 3.x fork (PyPI's `cloudscraper` is the older v1/v2 — PyPI forbids pinning a
 git-URL dep inside a published package, so install it alongside):
@@ -126,6 +147,30 @@ fix) and the cascade falls through — they are never silently skipped. Checklis
   challenge can't eat the per-URL deadline before the browser tier runs. Lower it
   (e.g. `12`) if tier_3 rarely wins on your hosts.
 
+### Scraping URLs other people submit
+
+If the URLs come from users you don't fully trust (a "save this link" box, a
+feed reader, a public API), two opt-in switches harden the engine. Both default
+to off, so nothing changes unless you set them:
+
+```bash
+export SCRAPER_DISABLE_CLOUDSCRAPER=1   # tier_3 runs site JavaScript outside a browser sandbox
+export SCRAPER_BROWSER_BLOCK_PRIVATE=1  # browser tiers can't reach your machine's or network's private services
+```
+
+- **Why tier_3:** solving a Cloudflare JS challenge means running JavaScript the
+  site sent, in js2py or Node's `vm` module, and neither is a security boundary.
+  The browser tiers run page JavaScript inside the browser's own sandbox instead.
+- **Install tier_4's browser first** (`patchright install chromium`): with tier_3
+  off, it is what clears Cloudflare. `--doctor` shows tier_3 as `off` and still
+  exits 0.
+- **Why the browser guard:** a scraped page can redirect, frame or `fetch()` an
+  address like `http://127.0.0.1:…` or a cloud metadata endpoint, and whatever
+  answers comes back as scraped content. With the guard, those requests (and
+  WebSockets) are refused and service workers are disabled so they can't route
+  around it. Check the URL you pass in yourself too: the guard covers what the
+  *page* asks for in the browser tiers, not the HTTP tiers' own requests.
+
 **Verify readiness on the box** with the preflight check (doubles as a healthcheck
 — exit 0 when the capable tiers are ready):
 
@@ -168,6 +213,61 @@ Non-Python callers: see [clients/node_bridge.md](clients/node_bridge.md). Python
 callers that want HTTP-with-CLI-fallback can drop in
 [clients/python_client.py](clients/python_client.py).
 
+**MCP server** (for agents: Claude Code, Cursor, any MCP client)
+```bash
+pip install "switchback[mcp]"
+claude mcp add switchback -- switchback-mcp           # stdio; tools: scrape, search
+```
+
+## Agent-ready extras
+
+Off by default: without them every result is exactly what it was before, the same
+`ScrapeResult` / `ScrapeOutcome` with the same fields, and the same CLI and HTTP
+JSON. Ask for any combination per call; results then come back as
+`ScrapeResultWithExtras` / `ScrapeOutcomeWithExtras` (subclasses, so existing
+`isinstance` checks still hold) with the extras you asked for filled in and the
+rest `None`. The CLI and HTTP JSON add only the keys you asked for.
+
+```python
+from switchback import scrape
+[r] = scrape("https://example.com/product", state=True, appendix=True,
+             query="what does it cost?")
+r.state       # {"__NEXT_DATA__": {...}, "json_ld": [...], "__INITIAL_STATE__": {...}}
+r.appendix    # [{"region": "nav", "markdown": "..."}, {"region": "footer", ...}]
+r.highlights  # [{"text": "...", "score": 7.1, "grader": "bm25",
+              #   "link": "https://example.com/product#:~:text=Price%20from,per%20month"}]
+```
+
+```bash
+switchback --state --appendix --query "what does it cost?" https://example.com/product
+curl -s localhost:8799/scrape -d '{"urls":["https://example.com"],"query":"pricing","state":true}'
+```
+
+| extra | what you get |
+| --- | --- |
+| `state` | The JSON the page embeds for its own frontend: Next.js `__NEXT_DATA__`, Nuxt, `window.__X__` globals (incl. `JSON.parse("…")`), schema.org JSON-LD. Often cleaner than the text (prices, ids, dates). Only valid JSON is returned. `{}` when the winning tier had no HTML (API mirrors, PDFs, docs, Firecrawl markdown). |
+| `appendix` | The menus, header, footer and sidebars that are stripped from the main content, as separate markdown, one entry per region. |
+| `query` → `highlights` | The passages that answer the question, best first, each with a citation `link`: a [text fragment](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments) URL that opens the page scrolled to, and highlighting, that passage (for a YouTube transcript, the video at that moment). If the text isn't visible on the page (e.g. it came from the `<title>` or image alt text) the browser simply opens the page at the top. |
+
+`state` and `appendix` need the fetched HTML, so they skip the content cache
+(`SCRAPER_CONTENT_TTL_S`) and fetch; `highlights` work on cached content too.
+
+**Grading highlights.** The default grader is local BM25 keyword scoring: free,
+offline, no data leaves the box. For better relevance, opt into
+[TypeSafe](https://typesafe.ai)'s Jev model, which answers "does this passage
+answer the question?" for every passage and returns a calibrated probability
+(passages under `SCRAPER_HIGHLIGHTS_MIN_SCORE`, default 0.5, are dropped, so an
+off-topic page honestly returns `[]`):
+
+```bash
+export SCRAPER_HIGHLIGHTS_GRADER=jev
+export TYPESAFE_API_KEY=...        # sends the page passages + query to api.typesafe.ai
+```
+
+Jev is billed per input token (about $0.0004 for a 10k-token page). If the call
+fails, that request falls back to BM25 (`"grader": "bm25"`); the scrape never fails
+because of highlights.
+
 ## Cost-scoped residential egress
 
 The dominant reason hard hosts wall you is the **datacenter IP**, not the
@@ -227,6 +327,8 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_DISABLE_FIRECRAWL` — skip tier_7
 - `FIRECRAWL_API_KEY` — enable tier_7
 - `SCRAPER_DISABLE_CAMOUFOX` — turn off tier_5 (on by default; needs `pip install camoufox` + `camoufox fetch`)
+- `SCRAPER_DISABLE_CLOUDSCRAPER` — turn off tier_3 (on by default). Recommended when you scrape URLs other people submit; see [Scraping URLs other people submit](#scraping-urls-other-people-submit)
+- `SCRAPER_BROWSER_BLOCK_PRIVATE` — opt-in, off by default: tiers 4 and 5 refuse any request a page makes to a loopback, private, link-local (incl. cloud metadata) or CGNAT address
 - `BU_CDP_URL` — enable tier_6 residential browser by pointing at a CDP endpoint
 - `SCRAPER_PROXY` — route *all* tiers/URLs through a proxy
 - `SCRAPER_EGRESS_PROXY` — route only walled hosts through a proxy (see [Cost-scoped residential egress](#cost-scoped-residential-egress))
@@ -254,6 +356,13 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
 - `SCRAPER_TIER_RETRY_ON` — failure classes eligible for a same-tier retry (default `timeout,rate_limited,connection`; widen to include `botwall,http_block` behind a rotating residential proxy so each retry gets a fresh IP). Retries are bounded by `SCRAPER_DEADLINE_S`; enabling them on the paid Firecrawl tier bills per attempt
 - `SCRAPER_LOGIN_HOOK` — `pkg.module:func` returning `{cookie: value}` for a host (see [Logged-in sessions](#logged-in-sessions))
 - `SCRAPER_EXTRACTION_FILE` — per-domain extraction prefs JSON (default `config/extraction.json`)
+- `SCRAPER_HIGHLIGHTS_GRADER` — `bm25` (default) or `jev` (needs `TYPESAFE_API_KEY`; see [Agent-ready extras](#agent-ready-extras))
+- `SCRAPER_HIGHLIGHTS_TOP_K` — highlights returned per URL (5)
+- `SCRAPER_HIGHLIGHTS_MIN_SCORE` — Jev probability below which a passage is dropped (0.5)
+- `SCRAPER_JEV_MODEL` / `SCRAPER_JEV_TIMEOUT_S` — Jev model alias (`jev-latest`) and per-request timeout (20s)
+- `SCRAPER_YOUTUBE` — opt-in: `1` returns YouTube transcripts from tier_1 (needs `switchback[youtube]`)
+- `SCRAPER_DOCUMENTS` — opt-in: `1` converts .docx/.pptx/.xlsx/.xls/.epub to markdown in tier_2 (needs `switchback[documents]`)
+- `SCRAPER_YOUTUBE_LANGS` — transcript languages to try, in order (`en`; e.g. `en,hi`). YouTube blocks most cloud IPs; when it does, the transcript is retried once through `SCRAPER_EGRESS_PROXY` if set
 - `SCRAPER_TRACE_SESSION` — opt-in: capture a Playwright trace (screenshots + DOM + network) per browser-tier attempt, written to `state/traces/`
 - `BENCH_FIRECRAWL_USD` / `BENCH_FIRECRAWL_HARD_MULT` — cost model for the savings report
 </details>
@@ -305,6 +414,20 @@ strip site-specific noise, declare prefs per host in `config/extraction.json`
 (see [config/extraction.example.json](config/extraction.example.json)); every
 tier's normalize step picks them up automatically.
 
+## Roadmap
+
+Not built yet:
+
+- **Layout-aware markdown** — keep pricing grids and card layouts built from
+  `<div>`s (not `<table>`s) as tables, using the browser tiers' rendered layout.
+- **Scanned pages and images** — OCR / vision-model text for PDFs with no text
+  layer and for images.
+- **Audio and caption-less video** — speech-to-text transcription.
+- **Hosted service** — managed browsers at high concurrency, stored results and
+  shareable result links. switchback stays a library you run yourself.
+- Smaller gaps: Next.js App Router flight data (`self.__next_f`) in `state`, and
+  YouTube chapters in transcripts.
+
 ## Contributing
 
 Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Start with the

{switchback-0.5.0 → switchback-0.6.0}/pyproject.toml

@@ -4,13 +4,17 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "switchback"
-version = "0.5.0"
+version = "0.6.0"
 description = "One cost-ordered scrape cascade (HTTP → stealth browser → paid), shared by every tool."
 readme = "README.md"
 requires-python = ">=3.10"
 license = { text = "MIT" }
 authors = [{ name = "Akash Kodavuru", email = "akash@theaklabs.com" }]
-keywords = ["scraping", "crawler", "cloudflare", "markdown", "cascade", "anti-bot", "stealth"]
+keywords = [
+    "web-scraping", "scraper", "crawler", "html-to-markdown", "markdown", "cloudflare-bypass",
+    "anti-bot", "stealth", "playwright", "patchright", "camoufox", "curl-cffi", "cloudscraper",
+    "firecrawl", "headless-browser", "residential-proxy", "llm", "rag", "ai-agents", "opentelemetry",
+]
 classifiers = [
     "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
@@ -49,6 +53,9 @@ browser = ["patchright"]                 # tier3: stealth headless Chromium
 camoufox = ["camoufox[geoip]"]           # tier3b: Firefox stealth (geoip: match locale/tz to proxy IP)
 firecrawl = ["firecrawl-py"]             # tier4: paid last-resort API
 server = ["fastapi", "uvicorn"]          # HTTP service (switchback.server) + /metrics + /traces
+mcp = ["mcp"]                            # MCP server for agents (switchback-mcp)
+youtube = ["youtube-transcript-api>=1.0"]   # tier_1: YouTube transcripts
+documents = ["markitdown[docx,pptx,xlsx,xls]"]  # tier_2: office/ebook files → markdown
 tracing = [                              # OTel → any OTLP backend
     "opentelemetry-sdk",
     "opentelemetry-exporter-otlp-proto-grpc",
@@ -58,11 +65,15 @@ all = [
     "fastapi", "uvicorn",
     "opentelemetry-sdk", "opentelemetry-exporter-otlp-proto-grpc",
 ]
+# Not in `all` on purpose: added in 0.6, kept out so existing `[all]` installs
+# don't suddenly pull markitdown's onnxruntime (via magika) or the MCP SDK.
+# Install them by name: "switchback[mcp,youtube,documents]".
 dev = ["pytest"]
 
 [project.scripts]
 switchback = "switchback.api:_main"
 switchback-server = "switchback.server:main"
+switchback-mcp = "switchback.mcp_server:main"
 
 [project.urls]
 Homepage = "https://github.com/akash-kr/switchback"

{switchback-0.5.0 → switchback-0.6.0}/switchback/__init__.py

@@ -5,8 +5,10 @@ Public API:
     results = scrape(["https://example.com/article"])
     hits = search("web scraping")            # query → URLs (SearXNG)
 """
-from .api import scrape, scrape_detailed, ScrapeOutcome, ScrapeResult, TierAttempt
+from .api import (scrape, scrape_detailed, ScrapeOutcome, ScrapeResult, TierAttempt,
+                  ScrapeOutcomeWithExtras, ScrapeResultWithExtras)
 from .search import search, SearchResult
 
 __all__ = ["scrape", "scrape_detailed", "ScrapeOutcome", "ScrapeResult",
-           "TierAttempt", "search", "SearchResult"]
+           "TierAttempt", "search", "SearchResult",
+           "ScrapeOutcomeWithExtras", "ScrapeResultWithExtras"]