wintergrab 0.2.0__tar.gz

wintergrab-0.2.0/.gitignore

@@ -0,0 +1,31 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.python-version
+
+# Tooling
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Crawl output and state
+.crawl/
+crawls/
+*.jsonl
+!tests/data/*.jsonl
+
+# Editors / OS
+.idea/
+.vscode/
+.DS_Store
+
+# Default HTTP cache directory
+.wintergrab-cache/

wintergrab-0.2.0/CHANGELOG.md

@@ -0,0 +1,61 @@
+# Changelog
+
+## 0.2.0
+
+First release on PyPI.
+
+- **Zero-selector extraction**: `structured_data()` (JSON-LD, microdata,
+  OpenGraph, Twitter, meta), `embedded_json()` (`__NEXT_DATA__`,
+  `window.__STATE__`, `JSON.parse` payloads), `find_json()`, `tables()`,
+  `next_page()`/`follow_next()`, `detect_records()`/`auto_extract()` and
+  learn-by-example `learn()` → reusable `LearnedSchema`.
+- **HTTP cache** with `revalidate`/`prefer`/`offline`/`refresh` modes for all
+  fetchers and spiders (offline replay of whole crawls).
+- **Disk frontier** (`frontier="disk"`): SQLite queue + scalable Bloom filter,
+  flat memory, at-least-once crash recovery.
+- **Browser**: `capture=` records XHR/fetch API responses; `export_cookies()` /
+  `add_cookies()`; spiders share browser cookies with HTTP sessions.
+- **Sitemaps**: `wg.sitemap()`, `Spider.sitemap_urls/rules/follow/since`.
+- **Output**: SQLite exporter with upserts, `unique_key` de-duplication,
+  buffered writers, orjson.
+- **Speed**: uvloop when installed (`[speed]` extra), leaner crawl loop, and
+  profile-guided hot-path shortcuts (URL joining and canonicalisation,
+  cached request host/fingerprint, compiled XPath cache, byte-level block
+  check). Each shortcut is tested to return exactly what the code it
+  bypasses returns.
+- Live terminal progress line; `wintergrab doctor`; many new CLI flags.
+- Engine robustness: no lost requests on cancel/force-stop/fatal errors,
+  crash-safe JSON output, correct Retry-After/429 pacing, signal handlers
+  restored, Ctrl+C in Jupyter.
+- **Block detection** recognises Fastly's "Client Challenge" (served with
+  status 200) and DataDome, and a spider that gives up on a block page says
+  so instead of reporting a bare "HTTP 200".
+- **Browser**: after a bot check passes, the capture waits until the real
+  page is fully parsed. Behind pypi.org's check, the browser used to return
+  a half-loaded page.
+- **Security**: a page on an allowed domain that redirects elsewhere
+  (possibly to an internal address) is no longer passed to the callbacks
+  (`offsite_redirects` stat).
+- The CLI writes UTF-8 when its output is redirected (Windows pipes default
+  to cp1252). Tested on Linux, macOS and Windows, Python 3.10-3.14.
+
+## 0.1.0
+
+Internal milestone, never published to PyPI.
+
+- **Fetching**: `get`/`post`/`aget`/`apost` shortcuts; `Fetcher` and `AsyncFetcher`
+  sessions with browser TLS/HTTP2 impersonation (curl_cffi), retries with
+  exponential backoff and `Retry-After`, proxy rotation, charset detection.
+- **Browser**: `render`/`arender`, `BrowserFetcher` and `AsyncBrowserFetcher`
+  (Playwright) with stealth patches, resource blocking, `wait_for`, scrolling,
+  page actions, screenshots, challenge-page waiting and per-proxy contexts.
+- **Parsing**: `Selector` with CSS (`::text`, `::attr()`) and XPath, extraction
+  schemas (`Field`), `find_by_text`, `find_by_regex`, `find_similar`, generated
+  selectors, links, text and Markdown conversion.
+- **Adaptive selectors**: element fingerprints stored in SQLite; broken
+  selectors relocate their elements by similarity.
+- **Spiders**: asyncio crawler with per-domain scheduling, AutoThrottle,
+  multiple sessions, session fallback for blocked pages, proxy rotation,
+  robots.txt, limits, JSON/JSONL/CSV output, streaming, and pause/resume with
+  checkpoints.
+- **CLI**: `wintergrab get`, `wintergrab crawl`, `wintergrab shell`.

wintergrab-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 opensourcewinter and wintergrab contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wintergrab-0.2.0/PKG-INFO

@@ -0,0 +1,303 @@
+Metadata-Version: 2.5
+Name: wintergrab
+Version: 0.2.0
+Summary: Friendly web scraping that scales from one page to big crawls: browser-like fetching, adaptive selectors, resumable async spiders.
+Project-URL: Homepage, https://github.com/opensourcewinter/wintergrab
+Project-URL: Documentation, https://github.com/opensourcewinter/wintergrab/tree/main/docs
+Project-URL: Changelog, https://github.com/opensourcewinter/wintergrab/blob/main/CHANGELOG.md
+Project-URL: Source, https://github.com/opensourcewinter/wintergrab
+Project-URL: Issues, https://github.com/opensourcewinter/wintergrab/issues
+Author: wintergrab contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: crawler,css-selectors,curl-cffi,playwright,scraping,spider,web-scraping,xpath
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: cssselect>=1.2
+Requires-Dist: curl-cffi>=0.10
+Requires-Dist: lxml>=5.0
+Provides-Extra: all
+Requires-Dist: orjson>=3.9; extra == 'all'
+Requires-Dist: playwright>=1.45; extra == 'all'
+Requires-Dist: uvloop>=0.19; (sys_platform != 'win32') and extra == 'all'
+Provides-Extra: browser
+Requires-Dist: playwright>=1.45; extra == 'browser'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: playwright>=1.45; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: speed
+Requires-Dist: orjson>=3.9; extra == 'speed'
+Requires-Dist: uvloop>=0.19; (sys_platform != 'win32') and extra == 'speed'
+Description-Content-Type: text/markdown
+
+# wintergrab
+
+**Friendly web scraping that scales from one page to big crawls.**
+
+```python
+import wintergrab as wg
+
+page = wg.get("https://quotes.toscrape.com/")
+for quote in page.css(".quote"):
+    print(quote.css(".text::text").get(), "-", quote.css(".author::text").get())
+```
+
+wintergrab is a Python toolkit for grabbing data from websites. Simple things
+are one line. When a site is harder (JavaScript, bot checks, rate limits,
+thousands of pages), the same API scales up.
+
+- **Fetch like a real browser.** HTTP requests carry Chrome/Firefox/Safari
+  TLS and HTTP/2 fingerprints (via [curl_cffi](https://github.com/lexiforest/curl_cffi)).
+  A headless Chromium (via [Playwright](https://playwright.dev/python/)) is
+  one flag away for JavaScript pages. It hides common automation tells and
+  waits out "checking your browser" interstitials.
+- **Parse with CSS or XPath.** Scrapy-style `::text` / `::attr(href)`,
+  extraction schemas, search by text, "find similar elements", and
+  HTML → Markdown/text conversion.
+- **Selectors that adapt.** With `adaptive=True`, wintergrab remembers what a
+  selector matched. After a redesign that breaks it, it finds the most
+  similar elements on the new page.
+- **Crawl at scale.** Async spiders with concurrency limits, multiple
+  sessions (HTTP + browser, several accounts…), proxy rotation with health
+  checks, **AutoThrottle** that backs off when a site pushes back,
+  robots.txt support, and **pause/resume** (Ctrl+C, then run again).
+- **Scrape without selectors.** Pull JSON-LD/microdata/OpenGraph, the JSON
+  state that React/Next/Vue apps embed in their HTML, and every table.
+  `auto_extract()` finds a page's product grid or result list and names the
+  fields. `learn({"title": "…", "price": "…"})` writes the selectors for you
+  from values you can see on the page.
+- **Built for big, long crawls.** An HTTP cache that revalidates with `304`s
+  and replays whole crawls offline. A disk-backed queue with a Bloom filter
+  that keeps memory flat at millions of URLs and survives `kill -9`. Sitemap
+  crawling, SQLite output with upserts, and a live progress line.
+- **Fast.** In a [reproducible benchmark](https://github.com/opensourcewinter/wintergrab/blob/main/benchmarks/README.md) against a
+  local test shop, a wintergrab spider crawled about 1,000 pages/s on one
+  core. That is 1.8× Crawlee and 3.8× Scrapy at the same concurrency, with
+  under half their memory. On real sites, the site and your politeness
+  settings usually set the pace, not the crawler.
+- **Browser superpowers.** Capture the JSON API calls a page makes while it
+  renders. Clear a login or JS check once in the browser, then continue over
+  fast HTTP with the same cookies.
+- **A small CLI.** `wintergrab get` and `wintergrab crawl` cover the common
+  jobs with no code at all, including `--auto`, `--learn` and `--offline`.
+
+## Install
+
+```bash
+pip install wintergrab                 # HTTP fetching, parsing, spiders, CLI
+pip install "wintergrab[browser]"      # + headless browser support
+pip install "wintergrab[speed]"        # + uvloop and orjson
+playwright install chromium            # one-time browser download (browser extra only)
+wintergrab doctor                      # check what is installed
+```
+
+Python 3.10+ on Linux, macOS and Windows. On a fresh Linux machine, use
+`playwright install --with-deps chromium` to get the browser's system
+libraries too. The development version installs straight from GitHub:
+`pip install "wintergrab @ git+https://github.com/opensourcewinter/wintergrab"`.
+
+## A quick tour
+
+### Fetch and parse
+
+```python
+import wintergrab as wg
+
+page = wg.get("https://books.toscrape.com/")      # looks like Chrome, retries hiccups
+page.status, page.title                           # (200, 'All products | Books to Scrape')
+
+page.css("h3 a::attr(title)").getall()            # every title
+page.css(".price_color::text").get()              # first price: '£51.77'
+page.xpath("//p[contains(@class, 'star-rating')]/@class").get()
+
+for book in page.css("article.product_pod"):      # loop and query inside
+    print(book.css("h3 a").attr("title"), book.css(".price_color").text)
+
+page.links(".pager")                              # absolute URLs of links in the pager
+page.find_by_text("Tipping the Velvet")           # search by visible text
+page.markdown(main_content=True)                  # the page as Markdown
+```
+
+### Pull out structured data
+
+```python
+from wintergrab import Field
+
+books = page.extract_all("article.product_pod", {
+    "title": "h3 a::attr(title)",
+    "price": Field(".price_color::text", transform=lambda p: float(p.lstrip("£"))),
+    "rating": Field("p.star-rating", attr="class", regex=r"star-rating (\w+)"),
+})
+page.extract({"titles": ["h3 a::attr(title)"]})       # a one-item list = all matches
+```
+
+### Survive layout changes
+
+```python
+products = page.css(".product-card", adaptive=True)
+```
+
+The first time, wintergrab saves a fingerprint of what matched: tag,
+attributes, text, position, parent and neighbours. If the site later renames
+`.product-card` or wraps it in new containers, the same call scores every
+element on the new page and returns the closest matches. It logs a warning
+so you know to update the selector. See [docs/adaptive-selectors.md](https://github.com/opensourcewinter/wintergrab/blob/main/docs/adaptive-selectors.md).
+
+### Scrape without writing selectors
+
+```python
+page.auto_extract()          # [{"title", "url", "image", "price", "rating"...}, ...] from the main record list
+schema = page.learn({"title": "A Light in the Attic", "price": "£51.77"})
+schema.extract(other_page)   # the learned selectors work on every page of that template
+page.structured_data()       # JSON-LD, microdata, OpenGraph, meta tags
+page.embedded_json()         # __NEXT_DATA__, window.__INITIAL_STATE__, ... (SPAs without a browser)
+page.tables()                # every table as records
+page.next_page()             # pagination, auto-detected
+```
+
+### JavaScript pages
+
+```python
+page = wg.render("https://quotes.toscrape.com/js/", wait_for=".quote")
+
+with wg.BrowserFetcher(headless=True) as browser:     # reuse one browser
+    page = browser.get(url, scroll=True, screenshot="page.png")
+```
+
+### Many pages at once
+
+```python
+async with wg.AsyncFetcher() as fetcher:
+    pages = await fetcher.get_many(urls, concurrency=10)
+```
+
+### Crawl a site
+
+```python
+from wintergrab import Spider
+
+class BooksSpider(Spider):
+    start_urls = ["https://books.toscrape.com/"]
+    allowed_domains = ["books.toscrape.com"]
+    concurrency = 16                 # AutoThrottle adapts the real speed per domain
+    crawl_dir = ".crawl/books"       # makes it resumable: Ctrl+C pauses, re-run resumes
+    output = "books.jsonl"           # items stream here (.jsonl / .json / .csv)
+
+    def parse(self, response):
+        for link in response.css("article.product_pod h3 a"):
+            yield response.follow(link, callback=self.parse_book)
+        yield from response.follow_all("li.next a")
+
+    def parse_book(self, response):
+        yield {
+            "title": response.css("h1::text").get(),
+            "price": response.css(".product_main .price_color::text").get(),
+        }
+
+result = BooksSpider().run()
+print(result.status, result.stats["pages"], result.stats["items"])
+```
+
+Scaling up is a few attributes away:
+
+```python
+class BigCrawl(Spider):
+    sitemap_urls = ["https://shop.example/robots.txt"]   # discover pages from sitemaps
+    frontier = "disk"            # flat memory for millions of URLs, crash-safe queue
+    crawl_dir = ".crawl/big"
+    cache = ".cache/big"         # revalidating HTTP cache; cache_mode="offline" replays the crawl
+    output = "catalog.db"        # SQLite...
+    unique_key = "url"           # ...with upserts: re-crawls update rows in place
+    fallback_session = "browser" # blocked page? retry it in a headless browser, share its cookies
+```
+
+Spiders also give you:
+
+- **Sessions.** Route requests through different fetchers with
+  `Request(url, session="browser")`. Set `fallback_session="browser"` to
+  retry blocked pages in a headless browser automatically.
+- **Proxy rotation.** `proxies = [...]` (or a `ProxyRotator`). Proxies that
+  keep failing are benched for a while.
+- **Speed control.** Per-domain concurrency and delays that back off on
+  429/503/block pages, honour `Retry-After` and robots.txt `Crawl-delay`,
+  and recover gradually.
+- **Limits and hooks.** `max_pages`, `max_items`, `max_depth`,
+  `process_item()`, `on_error()`, `on_start()` / `on_close()`, and
+  `async for item in spider.stream()`.
+
+### Command line
+
+```bash
+wintergrab get https://quotes.toscrape.com                          # page as Markdown
+wintergrab get https://quotes.toscrape.com --css ".quote .text::text"
+wintergrab get https://books.toscrape.com --each article.product_pod \
+    --field title="h3 a::attr(title)" --field price=.price_color::text -o books.csv
+wintergrab get https://quotes.toscrape.com/js/ --browser --wait-for .quote
+
+wintergrab get https://books.toscrape.com --auto                     # records, no selectors
+wintergrab get https://books.toscrape.com --learn "title=A Light in the Attic" --save-schema books.json
+wintergrab crawl https://books.toscrape.com --schema books.json --paginate -o books.jsonl
+wintergrab get https://shop.example/p/1 --structured                # JSON-LD, OpenGraph...
+
+wintergrab crawl my_spider.py -o items.jsonl --crawl-dir .crawl/mine   # run a spider file
+wintergrab crawl https://books.toscrape.com --follow "li.next a" --follow "h3 a" \
+    --each ".product_main" --field title=h1::text --max-pages 50 -o books.jsonl
+
+wintergrab shell https://quotes.toscrape.com                        # explore interactively
+```
+
+## Documentation
+
+| Guide | What's inside |
+|---|---|
+| [Getting started](https://github.com/opensourcewinter/wintergrab/blob/main/docs/getting-started.md) | Install, first scrape, first spider, in 10 minutes |
+| [Fetching](https://github.com/opensourcewinter/wintergrab/blob/main/docs/fetching.md) | `get`/`Fetcher`/`AsyncFetcher`/`BrowserFetcher`, options, errors |
+| [Parsing](https://github.com/opensourcewinter/wintergrab/blob/main/docs/parsing.md) | Selectors, extraction schemas, text search, Markdown |
+| [Adaptive selectors](https://github.com/opensourcewinter/wintergrab/blob/main/docs/adaptive-selectors.md) | How relocation works and how to tune it |
+| [Spiders](https://github.com/opensourcewinter/wintergrab/blob/main/docs/spiders.md) | Crawling, sessions, pause/resume, output, every setting |
+| [Power features](https://github.com/opensourcewinter/wintergrab/blob/main/docs/power-features.md) | Zero-selector extraction, cache & offline replay, API capture, cookie handoff, sitemaps, disk frontier, SQLite |
+| [Tough sites](https://github.com/opensourcewinter/wintergrab/blob/main/docs/anti-blocking.md) | Impersonation, browsers, proxies, AutoThrottle, etiquette |
+| [CLI](https://github.com/opensourcewinter/wintergrab/blob/main/docs/cli.md) | `get`, `crawl` and `shell` reference |
+| [Examples](https://github.com/opensourcewinter/wintergrab/tree/main/examples/) | Runnable scripts for every feature |
+
+## Scrape responsibly
+
+wintergrab makes polite crawling the default. Spiders obey robots.txt,
+adapt their speed to each site, and back off when asked. Stealth features
+exist so legitimate automation isn't misclassified. They don't make it OK
+to ignore a site's terms, hammer servers, or collect personal data you
+have no right to. Check the rules of each site you scrape.
+
+## Development
+
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+playwright install chromium        # for the browser tests (skipped otherwise)
+pytest                             # runs against a local test site; no internet needed
+ruff check . && ruff format --check .
+```
+
+See [CONTRIBUTING.md](https://github.com/opensourcewinter/wintergrab/blob/main/CONTRIBUTING.md) for the live tests and the release
+process, and [SECURITY.md](https://github.com/opensourcewinter/wintergrab/blob/main/SECURITY.md) to report a vulnerability.
+
+## License
+
+[MIT](https://github.com/opensourcewinter/wintergrab/blob/main/LICENSE)

wintergrab-0.2.0/README.md

@@ -0,0 +1,254 @@
+# wintergrab
+
+**Friendly web scraping that scales from one page to big crawls.**
+
+```python
+import wintergrab as wg
+
+page = wg.get("https://quotes.toscrape.com/")
+for quote in page.css(".quote"):
+    print(quote.css(".text::text").get(), "-", quote.css(".author::text").get())
+```
+
+wintergrab is a Python toolkit for grabbing data from websites. Simple things
+are one line. When a site is harder (JavaScript, bot checks, rate limits,
+thousands of pages), the same API scales up.
+
+- **Fetch like a real browser.** HTTP requests carry Chrome/Firefox/Safari
+  TLS and HTTP/2 fingerprints (via [curl_cffi](https://github.com/lexiforest/curl_cffi)).
+  A headless Chromium (via [Playwright](https://playwright.dev/python/)) is
+  one flag away for JavaScript pages. It hides common automation tells and
+  waits out "checking your browser" interstitials.
+- **Parse with CSS or XPath.** Scrapy-style `::text` / `::attr(href)`,
+  extraction schemas, search by text, "find similar elements", and
+  HTML → Markdown/text conversion.
+- **Selectors that adapt.** With `adaptive=True`, wintergrab remembers what a
+  selector matched. After a redesign that breaks it, it finds the most
+  similar elements on the new page.
+- **Crawl at scale.** Async spiders with concurrency limits, multiple
+  sessions (HTTP + browser, several accounts…), proxy rotation with health
+  checks, **AutoThrottle** that backs off when a site pushes back,
+  robots.txt support, and **pause/resume** (Ctrl+C, then run again).
+- **Scrape without selectors.** Pull JSON-LD/microdata/OpenGraph, the JSON
+  state that React/Next/Vue apps embed in their HTML, and every table.
+  `auto_extract()` finds a page's product grid or result list and names the
+  fields. `learn({"title": "…", "price": "…"})` writes the selectors for you
+  from values you can see on the page.
+- **Built for big, long crawls.** An HTTP cache that revalidates with `304`s
+  and replays whole crawls offline. A disk-backed queue with a Bloom filter
+  that keeps memory flat at millions of URLs and survives `kill -9`. Sitemap
+  crawling, SQLite output with upserts, and a live progress line.
+- **Fast.** In a [reproducible benchmark](https://github.com/opensourcewinter/wintergrab/blob/main/benchmarks/README.md) against a
+  local test shop, a wintergrab spider crawled about 1,000 pages/s on one
+  core. That is 1.8× Crawlee and 3.8× Scrapy at the same concurrency, with
+  under half their memory. On real sites, the site and your politeness
+  settings usually set the pace, not the crawler.
+- **Browser superpowers.** Capture the JSON API calls a page makes while it
+  renders. Clear a login or JS check once in the browser, then continue over
+  fast HTTP with the same cookies.
+- **A small CLI.** `wintergrab get` and `wintergrab crawl` cover the common
+  jobs with no code at all, including `--auto`, `--learn` and `--offline`.
+
+## Install
+
+```bash
+pip install wintergrab                 # HTTP fetching, parsing, spiders, CLI
+pip install "wintergrab[browser]"      # + headless browser support
+pip install "wintergrab[speed]"        # + uvloop and orjson
+playwright install chromium            # one-time browser download (browser extra only)
+wintergrab doctor                      # check what is installed
+```
+
+Python 3.10+ on Linux, macOS and Windows. On a fresh Linux machine, use
+`playwright install --with-deps chromium` to get the browser's system
+libraries too. The development version installs straight from GitHub:
+`pip install "wintergrab @ git+https://github.com/opensourcewinter/wintergrab"`.
+
+## A quick tour
+
+### Fetch and parse
+
+```python
+import wintergrab as wg
+
+page = wg.get("https://books.toscrape.com/")      # looks like Chrome, retries hiccups
+page.status, page.title                           # (200, 'All products | Books to Scrape')
+
+page.css("h3 a::attr(title)").getall()            # every title
+page.css(".price_color::text").get()              # first price: '£51.77'
+page.xpath("//p[contains(@class, 'star-rating')]/@class").get()
+
+for book in page.css("article.product_pod"):      # loop and query inside
+    print(book.css("h3 a").attr("title"), book.css(".price_color").text)
+
+page.links(".pager")                              # absolute URLs of links in the pager
+page.find_by_text("Tipping the Velvet")           # search by visible text
+page.markdown(main_content=True)                  # the page as Markdown
+```
+
+### Pull out structured data
+
+```python
+from wintergrab import Field
+
+books = page.extract_all("article.product_pod", {
+    "title": "h3 a::attr(title)",
+    "price": Field(".price_color::text", transform=lambda p: float(p.lstrip("£"))),
+    "rating": Field("p.star-rating", attr="class", regex=r"star-rating (\w+)"),
+})
+page.extract({"titles": ["h3 a::attr(title)"]})       # a one-item list = all matches
+```
+
+### Survive layout changes
+
+```python
+products = page.css(".product-card", adaptive=True)
+```
+
+The first time, wintergrab saves a fingerprint of what matched: tag,
+attributes, text, position, parent and neighbours. If the site later renames
+`.product-card` or wraps it in new containers, the same call scores every
+element on the new page and returns the closest matches. It logs a warning
+so you know to update the selector. See [docs/adaptive-selectors.md](https://github.com/opensourcewinter/wintergrab/blob/main/docs/adaptive-selectors.md).
+
+### Scrape without writing selectors
+
+```python
+page.auto_extract()          # [{"title", "url", "image", "price", "rating"...}, ...] from the main record list
+schema = page.learn({"title": "A Light in the Attic", "price": "£51.77"})
+schema.extract(other_page)   # the learned selectors work on every page of that template
+page.structured_data()       # JSON-LD, microdata, OpenGraph, meta tags
+page.embedded_json()         # __NEXT_DATA__, window.__INITIAL_STATE__, ... (SPAs without a browser)
+page.tables()                # every table as records
+page.next_page()             # pagination, auto-detected
+```
+
+### JavaScript pages
+
+```python
+page = wg.render("https://quotes.toscrape.com/js/", wait_for=".quote")
+
+with wg.BrowserFetcher(headless=True) as browser:     # reuse one browser
+    page = browser.get(url, scroll=True, screenshot="page.png")
+```
+
+### Many pages at once
+
+```python
+async with wg.AsyncFetcher() as fetcher:
+    pages = await fetcher.get_many(urls, concurrency=10)
+```
+
+### Crawl a site
+
+```python
+from wintergrab import Spider
+
+class BooksSpider(Spider):
+    start_urls = ["https://books.toscrape.com/"]
+    allowed_domains = ["books.toscrape.com"]
+    concurrency = 16                 # AutoThrottle adapts the real speed per domain
+    crawl_dir = ".crawl/books"       # makes it resumable: Ctrl+C pauses, re-run resumes
+    output = "books.jsonl"           # items stream here (.jsonl / .json / .csv)
+
+    def parse(self, response):
+        for link in response.css("article.product_pod h3 a"):
+            yield response.follow(link, callback=self.parse_book)
+        yield from response.follow_all("li.next a")
+
+    def parse_book(self, response):
+        yield {
+            "title": response.css("h1::text").get(),
+            "price": response.css(".product_main .price_color::text").get(),
+        }
+
+result = BooksSpider().run()
+print(result.status, result.stats["pages"], result.stats["items"])
+```
+
+Scaling up is a few attributes away:
+
+```python
+class BigCrawl(Spider):
+    sitemap_urls = ["https://shop.example/robots.txt"]   # discover pages from sitemaps
+    frontier = "disk"            # flat memory for millions of URLs, crash-safe queue
+    crawl_dir = ".crawl/big"
+    cache = ".cache/big"         # revalidating HTTP cache; cache_mode="offline" replays the crawl
+    output = "catalog.db"        # SQLite...
+    unique_key = "url"           # ...with upserts: re-crawls update rows in place
+    fallback_session = "browser" # blocked page? retry it in a headless browser, share its cookies
+```
+
+Spiders also give you:
+
+- **Sessions.** Route requests through different fetchers with
+  `Request(url, session="browser")`. Set `fallback_session="browser"` to
+  retry blocked pages in a headless browser automatically.
+- **Proxy rotation.** `proxies = [...]` (or a `ProxyRotator`). Proxies that
+  keep failing are benched for a while.
+- **Speed control.** Per-domain concurrency and delays that back off on
+  429/503/block pages, honour `Retry-After` and robots.txt `Crawl-delay`,
+  and recover gradually.
+- **Limits and hooks.** `max_pages`, `max_items`, `max_depth`,
+  `process_item()`, `on_error()`, `on_start()` / `on_close()`, and
+  `async for item in spider.stream()`.
+
+### Command line
+
+```bash
+wintergrab get https://quotes.toscrape.com                          # page as Markdown
+wintergrab get https://quotes.toscrape.com --css ".quote .text::text"
+wintergrab get https://books.toscrape.com --each article.product_pod \
+    --field title="h3 a::attr(title)" --field price=.price_color::text -o books.csv
+wintergrab get https://quotes.toscrape.com/js/ --browser --wait-for .quote
+
+wintergrab get https://books.toscrape.com --auto                     # records, no selectors
+wintergrab get https://books.toscrape.com --learn "title=A Light in the Attic" --save-schema books.json
+wintergrab crawl https://books.toscrape.com --schema books.json --paginate -o books.jsonl
+wintergrab get https://shop.example/p/1 --structured                # JSON-LD, OpenGraph...
+
+wintergrab crawl my_spider.py -o items.jsonl --crawl-dir .crawl/mine   # run a spider file
+wintergrab crawl https://books.toscrape.com --follow "li.next a" --follow "h3 a" \
+    --each ".product_main" --field title=h1::text --max-pages 50 -o books.jsonl
+
+wintergrab shell https://quotes.toscrape.com                        # explore interactively
+```
+
+## Documentation
+
+| Guide | What's inside |
+|---|---|
+| [Getting started](https://github.com/opensourcewinter/wintergrab/blob/main/docs/getting-started.md) | Install, first scrape, first spider, in 10 minutes |
+| [Fetching](https://github.com/opensourcewinter/wintergrab/blob/main/docs/fetching.md) | `get`/`Fetcher`/`AsyncFetcher`/`BrowserFetcher`, options, errors |
+| [Parsing](https://github.com/opensourcewinter/wintergrab/blob/main/docs/parsing.md) | Selectors, extraction schemas, text search, Markdown |
+| [Adaptive selectors](https://github.com/opensourcewinter/wintergrab/blob/main/docs/adaptive-selectors.md) | How relocation works and how to tune it |
+| [Spiders](https://github.com/opensourcewinter/wintergrab/blob/main/docs/spiders.md) | Crawling, sessions, pause/resume, output, every setting |
+| [Power features](https://github.com/opensourcewinter/wintergrab/blob/main/docs/power-features.md) | Zero-selector extraction, cache & offline replay, API capture, cookie handoff, sitemaps, disk frontier, SQLite |
+| [Tough sites](https://github.com/opensourcewinter/wintergrab/blob/main/docs/anti-blocking.md) | Impersonation, browsers, proxies, AutoThrottle, etiquette |
+| [CLI](https://github.com/opensourcewinter/wintergrab/blob/main/docs/cli.md) | `get`, `crawl` and `shell` reference |
+| [Examples](https://github.com/opensourcewinter/wintergrab/tree/main/examples/) | Runnable scripts for every feature |
+
+## Scrape responsibly
+
+wintergrab makes polite crawling the default. Spiders obey robots.txt,
+adapt their speed to each site, and back off when asked. Stealth features
+exist so legitimate automation isn't misclassified. They don't make it OK
+to ignore a site's terms, hammer servers, or collect personal data you
+have no right to. Check the rules of each site you scrape.
+
+## Development
+
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+playwright install chromium        # for the browser tests (skipped otherwise)
+pytest                             # runs against a local test site; no internet needed
+ruff check . && ruff format --check .
+```
+
+See [CONTRIBUTING.md](https://github.com/opensourcewinter/wintergrab/blob/main/CONTRIBUTING.md) for the live tests and the release
+process, and [SECURITY.md](https://github.com/opensourcewinter/wintergrab/blob/main/SECURITY.md) to report a vulnerability.
+
+## License
+
+[MIT](https://github.com/opensourcewinter/wintergrab/blob/main/LICENSE)