scrapper-tool 0.1.0__tar.gz

scrapper_tool-0.1.0/.gitignore

@@ -0,0 +1,44 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+.Python
+build/
+dist/
+*.egg-info/
+*.egg
+.eggs/
+
+# Unit test / coverage reports
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+.mypy_cache/
+.ruff_cache/
+
+# Environments
+.env
+.venv
+venv/
+ENV/
+
+# IDEs
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# uv
+.python-version
+
+# Live-probe artifacts (gitignored — kept local for diagnostics)
+reports/
+docs/_evidence/
+
+# OS
+.DS_Store
+Thumbs.db

scrapper_tool-0.1.0/CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+All notable changes to `scrapper-tool` are recorded here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows [SemVer](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-04-30
+
+First public release. Covers Pattern A/B/C/D extraction primitives, the four-profile anti-bot impersonation ladder, deterministic fixture-replay testing, the generic `Adapter` Protocol, and a `scrapper-tool canary` CLI.
+
+### Added
+- M0 — repo bootstrap: `pyproject.toml`, MIT `LICENSE`, README, governance files (`CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`), CI workflow (`.github/workflows/ci.yml` — ruff + mypy --strict + pytest + pip-audit on py3.12/3.13/3.14 matrix), tag-triggered PyPI release workflow (`.github/workflows/release.yml`, OIDC trusted-publisher).
+- `[project.optional-dependencies]` placeholders for `hostile` (Scrapling) and `agent` (MCP — populated in M13).
+- M1 — HTTP core extracted from PartsPilot's `affiliate-service`: `scrapper_tool.http.vendor_client()` (httpx + curl_cffi backends, async context manager) and `scrapper_tool.http.request_with_retry()` (3 attempts, exponential backoff with ±25% jitter, retries 429/5xx/transport errors, no-retry on 4xx ≠ 429, X-Request-ID injection).
+- M1 — Exception hierarchy: `ScrapingError` (base), `VendorHTTPError`, `VendorUnavailable` (alias), `BlockedError`, `ParseError`. `BlockedError` and `ParseError` deliberately do NOT inherit from `VendorHTTPError` — circuit breakers should catch one but not the others.
+- M1 — Optional `structlog` integration via `scrapper_tool._logging.get_logger()`; falls back to a stdlib `logging` adapter that accepts the same `key=value` kwarg shape.
+- M1 — Top-level re-exports: `scrapper_tool.{vendor_client, request_with_retry, VendorHTTPError, BlockedError, ParseError, ScrapingError, VendorUnavailable, VendorHTTPClient}`.
+- M2 — Anti-bot impersonation ladder (`scrapper_tool.ladder`): `IMPERSONATE_LADDER = ("chrome133a", "chrome124", "safari18_0", "firefox135")` and `request_with_ladder(method, url, ...)` walking it top-to-bottom on 403/503. First profile to return ≠403/503 wins; all-403 raises `BlockedError` with a "escalate to Pattern D" message. Each ladder step opens a fresh `curl_cffi.AsyncSession` (one-shot per profile, sessions pinned to a single fingerprint). Logs winning profile via the structured logger (`ladder.profile_won` / `ladder.profile_blocked`).
+- M2 — Re-exported at top level: `scrapper_tool.{IMPERSONATE_LADDER, request_with_ladder}`.
+- M2 — 9 ladder unit tests (happy path, 403→200 fallback, 503 rotate-like-403, safari wins when chrome burns, all-403 raises, custom ladder, empty ladder ValueError, default-ladder shape, header propagation). Uses an inline `_FakeCurlSession` lifted to `scrapper_tool.testing` in M6.
+- M3 — Pattern B helper (`scrapper_tool.patterns.b`): `extract_product_offer(html, base_url=None)` returns a normalised `ProductOffer` Pydantic model from any of JSON-LD / microdata / RDFa Product blocks. Handles top-level Products, Products nested inside `@graph`, multi-offer lists (takes first), price/currency nested inside `priceSpecification`, brand-as-dict-or-string, image-as-list-or-dict, all `gtin{,8,12,13,14}` variants. Powered by `extruct.extract(..., uniform=True)` so one walker covers all three syntaxes.
+- M3 — `ProductOffer` model fields: `name`, `sku`, `mpn` (often the OEM in automotive use cases), `gtin`, `brand`, `description`, `image`, `price` (Decimal), `currency` (ISO 4217), `availability` (raw schema.org URI), `url`. `model_config = {"extra": "ignore"}` so vendors adding fields don't break parsing.
+- M3 — 10 Pattern B unit tests (JSON-LD top-level, JSON-LD inside @graph, offers as list, priceSpecification fallback, microdata, brand-as-string, no-Product-block returns None, plain HTML returns None, base_url propagation, extra-keys ignored).
+- M4 — Pattern C helper (`scrapper_tool.patterns.c`): `extract_microdata_price(html) -> tuple[Decimal, str] | None` for sites that ship `<meta itemprop="price"> + <meta itemprop="priceCurrency">` schema.org microdata anchors (preferred — stable across CSS reshuffles); `extract_via_selectors(html, *, price_selector, currency_selector=None, default_currency=None)` for last-resort bespoke CSS selectors. Backed by `selectolax` (lexbor backend; 30-40× faster than BeautifulSoup at our fetch volumes).
+- M4 — Internal `_coerce_decimal` strips common currency glyphs (`$`, `€`, `£`, `₪`, `¥`) and US/UK thousands-separator commas before parsing. European decimal-comma is NOT supported by default — vendor-specific normalisation is the consumer's job.
+- M4 — 21 Pattern C unit tests (microdata via `<meta>` content attribute, microdata via text fallback, price-without-currency returns None, missing microdata returns None, selector with default_currency, selector with currency_selector, selector with `data-price` attribute preferred, missing element returns None, ValueError on no-currency-source, glyph stripping for 6 currency symbols, thousands-separator stripping, unparseable input returns None).
+- M5 — Pattern D helper (`scrapper_tool.patterns.d.hostile_client`): async context manager wrapping Scrapling's `StealthyFetcher` for Cloudflare Turnstile / Akamai EVA / Distil-class hostile sites. Lazy-imports `scrapling` so consumers without the `[hostile]` extra installed see a useful `ImportError` with install hint rather than `ModuleNotFoundError` at import time. Forwards `headless`, `block_resources`, `timeout`, and arbitrary `extra_kwargs` to the fetcher; supports both async (`aclose`) and sync (`close`) lifecycle on exit.
+- M5 — 5 Pattern D unit tests (`ImportError` raised when `[hostile]` not installed, fetcher yielded + closed on exit, `extra_kwargs` propagate, sync-close fallback for older Scrapling versions, module docstring readable without scrapling installed). Real Scrapling integration deferred to live-probe tests (`tests/integration/test_live_probes.py`, `live` marker, opt-in).
+- M6 — Test helpers (`scrapper_tool.testing`): `FakeCurlSession` (drop-in mock for `curl_cffi.AsyncSession` because `respx` doesn't intercept it), `FakeResponse` (minimal duck-typed response), `replay_fixture(path, parser)` (load fixture file from disk and feed to a parser), `assert_pydantic_snapshot(obj, path, *, write_if_missing=True)` (golden-snapshot diff for Pydantic models with first-run seeding).
+- M6 — Refactored `tests/unit/test_ladder.py` to use the canonical `FakeCurlSession` (M2's inline mock removed; replaced with the import).
+- M6 — 12 meta-tests in `tests/unit/test_testing_helpers.py` covering FakeResponse construction, FakeCurlSession reset/configuration/calls-tracking, replay_fixture text loading, snapshot first-run-write / pass-on-match / fail-on-drift / write_if_missing=False semantics. 100% coverage on `testing.py`.
+- M5.5 — Filled `docs/research/2026-04-30-landscape.md` (~250 lines, 19 numbered sources). Eight sections: TLS-impersonation libraries, browser-stealth tools, anti-bot platforms in 2026, LLM-assisted scraping, HTML parsing libraries, structured-data extraction, what's deliberately missing from the lib, and a refresh policy that makes successor landscape docs append-only history rather than edits-in-place.
+
+- M7 — Generic `Adapter[QueryT, ResultT]` Protocol (`scrapper_tool.adapter`). Structural typing with `runtime_checkable` so `isinstance(obj, Adapter)` works without inheritance. Required surface: `vendor_id: str` attribute + `async search(query)` + `async fetch_detail(url)`. Doc-strings codify the error-bubbling contract (VendorHTTPError → breaker trips; BlockedError → escalate to Pattern D; ParseError → don't trip breaker, parser drift bug). Re-exported as `scrapper_tool.Adapter`.
+- M7 — 6 Protocol tests: complete impl satisfies isinstance, missing method fails, missing field fails, search round-trip, fetch_detail round-trip, fetch_detail returns None for missing URL.
+- M8 — `scrapper-tool canary` CLI (`scrapper_tool.canary` module + `[project.scripts]` entry). Walks the impersonation ladder against a target URL, reports which profile won (or all-blocked). Designed for cron / GitHub Actions to surface "chrome133a is starting to 403" before any consumer adapter notices. Flags: `--profiles chrome133a,chrome124,...` (custom ladder), `--timeout` (per-request), `--proxy`, `--json` (machine-readable output). Exit codes: 0 success, 1 all-blocked, 2 error. Public API: `run_canary()` (programmatic) + `probe_profile()` (single-profile probe).
+- M8 — 12 canary unit tests covering happy-path (first profile wins, others skipped), 403 fallback (rotates), all-blocked (exit_code=1), empty ladder ValueError, custom ladder, text mode, JSON mode parseable, --profiles override, exit codes, --help, no-subcommand argparse error, malformed --profiles flag.
+
+### Fixed
+- CI: `pip-audit --skip-editable` so the build doesn't try to look up `scrapper-tool` itself on PyPI before v0.1.0 ships.
+
+### Notes
+- Initial milestone scope and decision log live in [`docs/research/2026-04-30-landscape.md`](docs/research/2026-04-30-landscape.md) (filled in during M5.5).
+
+### Notes
+- Initial milestone scope and decision log live in [`docs/research/2026-04-30-landscape.md`](docs/research/2026-04-30-landscape.md) (filled in during M5.5).

scrapper_tool-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ValeroK
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scrapper_tool-0.1.0/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.4
+Name: scrapper-tool
+Version: 0.1.0
+Summary: Reusable web-scraping toolkit — Pattern A/B/C/D ladder, TLS-impersonation fallback chain, deterministic fixture-replay testing, and an optional MCP server for LLM agents.
+Project-URL: Homepage, https://github.com/ValeroK/scrapper-tool
+Project-URL: Repository, https://github.com/ValeroK/scrapper-tool
+Project-URL: Issues, https://github.com/ValeroK/scrapper-tool/issues
+Project-URL: Changelog, https://github.com/ValeroK/scrapper-tool/blob/main/CHANGELOG.md
+Author: ValeroK
+License-Expression: MIT
+License-File: LICENSE
+Keywords: anti-bot,curl-cffi,extruct,html-parsing,llm-agents,mcp,web-scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: curl-cffi>=0.7
+Requires-Dist: extruct>=0.18
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.5
+Requires-Dist: selectolax>=0.3.21
+Provides-Extra: agent
+Provides-Extra: dev
+Requires-Dist: mypy>=1.13; extra == 'dev'
+Requires-Dist: pip-audit>=2.7; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: respx>=0.21; extra == 'dev'
+Requires-Dist: ruff>=0.9; extra == 'dev'
+Provides-Extra: hostile
+Requires-Dist: scrapling>=0.3; extra == 'hostile'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# scrapper-tool
+
+**A reusable Python web-scraping toolkit — production-grade primitives, anti-bot ladder, fixture-replay testing.**
+
+Built from the scraping core behind [PartsPilot](https://github.com/ValeroK/affiliate-service), extracted as an open-source library so other projects (and LLM agents) can pick up the same patterns without redoing the reverse-engineering work.
+
+<br />
+
+[![CI](https://github.com/ValeroK/scrapper-tool/actions/workflows/ci.yml/badge.svg)](https://github.com/ValeroK/scrapper-tool/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
+[![Python versions](https://img.shields.io/pypi/pyversions/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
+[![Downloads](https://img.shields.io/pypi/dm/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230.svg)](https://github.com/astral-sh/ruff)
+[![Type-checked: mypy](https://img.shields.io/badge/type--checked-mypy-1f5082.svg)](https://mypy-lang.org/)
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
+[![GitHub Stars](https://img.shields.io/github/stars/ValeroK/scrapper-tool?style=social)](https://github.com/ValeroK/scrapper-tool/stargazers)
+[![GitHub Forks](https://img.shields.io/github/forks/ValeroK/scrapper-tool?style=social)](https://github.com/ValeroK/scrapper-tool/network/members)
+
+[**Quickstart**](#quickstart) · [**Documentation**](docs/index.md) · [**Recon playbook**](docs/recon.md) · [**Changelog**](CHANGELOG.md) · [**Contributing**](CONTRIBUTING.md)
+
+</div>
+
+---
+
+> **Status (2026-04-30):** alpha. `v0.1.0` covers the core pattern ladder, anti-bot helpers, and deterministic fixture-replay testing. `v0.2.0` adds an MCP server for LLM agents (Claude, OpenClaw, Hermes Agent, AutoGen, LangChain).
+
+## Table of contents
+
+- [Why scrapper-tool](#why-scrapper-tool)
+- [The four scraping patterns](#the-four-scraping-patterns)
+- [Architecture](#architecture)
+- [Install](#install)
+- [Quickstart](#quickstart)
+- [Documentation](#documentation)
+- [Why these tools?](#why-these-tools)
+- [Roadmap](#roadmap)
+- [Contributing](#contributing)
+- [Contributors](#contributors)
+- [Acknowledgements](#acknowledgements)
+- [License](#license)
+
+## Why scrapper-tool
+
+Most scrapers are written from scratch every time, even though 90% of the work is the same: pick the right extraction pattern, survive the TLS fingerprint, retry/backoff sanely, and write tests that don't drift the moment a site updates.
+
+`scrapper-tool` packages the parts that don't change per vendor, so you only write the parts that do.
+
+- **Pattern-first design.** Four named, documented extraction patterns (A–D) — pick the one DevTools points at, skip the rest.
+- **Anti-bot ladder built in.** Auto-walks `chrome133a → chrome124 → safari18_0 → firefox135` when a profile gets fingerprinted.
+- **Deterministic tests.** Fixture-replay (`FakeCurlSession`, `replay_fixture`, golden snapshots) — no live HTTP in CI.
+- **Optional hostile mode.** Cloudflare Turnstile / Akamai EVA defeat path via [Scrapling](https://github.com/D4Vinci/Scrapling) — opt-in extra, no Playwright bloat by default.
+- **LLM-agent ready.** `v0.2.0+` ships an MCP server so Claude, AutoGen, LangChain, etc. can drive the scraper directly.
+- **Boring stack.** `httpx`, `curl_cffi`, `selectolax`, `extruct`. No managed SaaS bundled — your code, your egress.
+
+## The four scraping patterns
+
+Web scraping in 2026 is dominated by four recurring patterns. This lib gives each pattern a documented helper plus the surrounding infrastructure (HTTP client with TLS-impersonation fallback, retry/backoff, fixture-replay testing) so you don't reinvent them per vendor.
+
+| Pattern | When to use | Helper | Cost |
+|---|---|---|---|
+| **A — JSON API** | DevTools shows an XHR returning the price-bearing JSON. Anonymous or OAuth. | `vendor_client()` + your own response model | Lowest — parse, validate, done. |
+| **B — Embedded JSON** | Document HTML carries `<script type="application/ld+json">`, `__NEXT_DATA__`, `__NUXT__`, or `self.__next_f.push(...)`. | `patterns.b.extract_product_offer()` (via [`extruct`](https://github.com/scrapinghub/extruct)) | Low — one call, broad markup coverage. |
+| **C — CSS / microdata** | Price visible in HTML, no embedded JSON. Prefer `itemprop="price"` schema.org microdata. | `patterns.c.extract_microdata_price()` (via [`selectolax`](https://github.com/rushter/selectolax)) | Medium — selectors break on ancestor reshuffles. |
+| **D — Hostile** | Cloudflare Turnstile, Akamai EVA, etc. defeat both default `httpx` and `curl_cffi`. | `patterns.d.hostile_client()` (via [Scrapling](https://github.com/D4Vinci/Scrapling)) — `pip install scrapper-tool[hostile]` | Highest — Playwright runtime, ≈400 MB image bloat. |
+
+Plus a four-profile **anti-bot ladder** (`chrome133a → chrome124 → safari18_0 → firefox135`) that auto-walks when a profile gets fingerprinted, and a `scrapper-tool canary` CLI for nightly fingerprint-health probes.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    A[Your scraper code] --> B[vendor_client / request_with_retry]
+    B --> C{TLS-sensitive?}
+    C -- no --> D[httpx]
+    C -- yes --> E[curl_cffi ladder]
+    E --> E1[chrome133a] --> E2[chrome124] --> E3[safari18_0] --> E4[firefox135]
+    D --> F[Response]
+    E4 --> F
+    F --> G{Pattern}
+    G -- A --> H[JSON API model]
+    G -- B --> I[extruct: ld+json / next_data / nuxt]
+    G -- C --> J[selectolax: microdata / CSS]
+    G -- D --> K["Scrapling (Playwright + Turnstile)"]
+    H --> L[Validated product data]
+    I --> L
+    J --> L
+    K --> L
+```
+
+## Install
+
+```bash
+pip install scrapper-tool                # core: httpx + curl_cffi + selectolax + extruct
+pip install scrapper-tool[hostile]       # adds Scrapling for Cloudflare Turnstile
+pip install scrapper-tool[agent]         # adds the MCP server (v0.2.0+) for LLM agents
+```
+
+> **Tip.** The `[hostile]` extra pulls Playwright (~400 MB). Don't install it unless you actually need pattern D.
+
+## Quickstart
+
+```python
+import asyncio
+from scrapper_tool import vendor_client, request_with_retry
+from scrapper_tool.patterns.b import extract_product_offer
+
+async def main() -> None:
+    async with vendor_client() as client:
+        resp = await request_with_retry(client, "GET", "https://example-shop.test/product/123")
+        product = extract_product_offer(resp.text, base_url=str(resp.url))
+        print(product)
+
+asyncio.run(main())
+```
+
+For TLS-sensitive vendors, flip one switch:
+
+```python
+async with vendor_client(use_curl_cffi=True) as client:
+    ...   # walks chrome133a → chrome124 → safari → firefox until one returns 200
+```
+
+See **[`docs/quickstart.md`](docs/quickstart.md)** for a 5-minute on-ramp covering all four patterns.
+
+## Documentation
+
+| | |
+|---|---|
+| **[Quickstart](docs/quickstart.md)** | 5-minute on-ramp. |
+| **[Recon playbook](docs/recon.md)** | DevTools-driven reverse-engineering of a new vendor site. |
+| **[Pattern A — JSON API](docs/patterns/a-json-api.md)** | Vendor exposes an XHR / JSON endpoint. |
+| **[Pattern B — Embedded JSON](docs/patterns/b-embedded-json.md)** | `ld+json`, `__NEXT_DATA__`, `__NUXT__`, RSC payloads. |
+| **[Pattern C — CSS / microdata](docs/patterns/c-css-microdata.md)** | `itemprop="price"`, fallback selectors. |
+| **[Pattern D — Hostile](docs/patterns/d-hostile.md)** | Cloudflare Turnstile, Akamai EVA. |
+| **[Anti-bot ladder reference](docs/reference/ladder.md)** | How the ladder walks, when to bump the primary profile. |
+| **[Test helpers](docs/reference/testing.md)** | `FakeCurlSession`, `replay_fixture`, golden-snapshot pattern. |
+| **[Agent integration](docs/agent-integration.md)** | MCP wiring for Claude, OpenClaw, Hermes Agent, AutoGen, LangChain. *(v0.2.0+)* |
+| **[2026-04-30 landscape research](docs/research/2026-04-30-landscape.md)** | Why these tools, sourced. |
+
+## Why these tools?
+
+Short version: `curl_cffi` is the only actively-maintained TLS-impersonation lib with `chrome131+`/`chrome133a`/`chrome142`/`chrome146` profiles; `puppeteer-stealth` and `playwright-extra` were deprecated in 2025-02; Scrapling is the only OSS Playwright-based stack with a working Turnstile auto-solve as of 2026; managed SaaS (Firecrawl, ZenRows, Bright Data) is deliberately not bundled.
+
+Full sourced rationale: **[`docs/research/2026-04-30-landscape.md`](docs/research/2026-04-30-landscape.md)**.
+
+## Roadmap
+
+- [x] **v0.1.0** — Core HTTP client, retry/backoff, anti-bot ladder, patterns A–D, fixture-replay test helpers.
+- [ ] **v0.2.0** — MCP server for LLM agents; canary CLI for nightly fingerprint-health probes.
+- [ ] **v0.3.0** — Pluggable rate-limit / robots.txt policies; per-vendor profile presets.
+- [ ] **v1.0.0** — API stability guarantee; broader pattern-D backends.
+
+See [`CHANGELOG.md`](CHANGELOG.md) for landed changes and [open issues](https://github.com/ValeroK/scrapper-tool/issues) for what's in flight.
+
+## Contributing
+
+PRs and issues are welcome. Every PR that meaningfully changes how we scrape lands a `CHANGELOG.md` row.
+
+- Read **[`CONTRIBUTING.md`](CONTRIBUTING.md)** for the maintenance contract.
+- Read **[`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)** before opening a discussion.
+- Good first issues live under the [`good first issue`](https://github.com/ValeroK/scrapper-tool/labels/good%20first%20issue) label.
+
+## Contributors
+
+<a href="https://github.com/ValeroK/scrapper-tool/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=ValeroK/scrapper-tool" alt="Contributors" />
+</a>
+
+Want to see your avatar here? Check [CONTRIBUTING.md](CONTRIBUTING.md) and open a PR.
+
+## Acknowledgements
+
+`scrapper-tool` stands on the shoulders of these projects:
+
+- [`httpx`](https://github.com/encode/httpx) — async HTTP client
+- [`curl_cffi`](https://github.com/lexiforest/curl_cffi) — TLS / JA3 impersonation
+- [`selectolax`](https://github.com/rushter/selectolax) — fast HTML parsing
+- [`extruct`](https://github.com/scrapinghub/extruct) — `ld+json`, microdata, RDFa extraction
+- [`Scrapling`](https://github.com/D4Vinci/Scrapling) — Playwright-based hostile-site backend
+
+## License
+
+[MIT](LICENSE) © scrapper-tool contributors.
+
+<div align="center">
+
+If `scrapper-tool` saves you time, consider [starring the repo](https://github.com/ValeroK/scrapper-tool) — it helps others find it.
+
+</div>

scrapper_tool-0.1.0/README.md

@@ -0,0 +1,192 @@
+<div align="center">
+
+# scrapper-tool
+
+**A reusable Python web-scraping toolkit — production-grade primitives, anti-bot ladder, fixture-replay testing.**
+
+Built from the scraping core behind [PartsPilot](https://github.com/ValeroK/affiliate-service), extracted as an open-source library so other projects (and LLM agents) can pick up the same patterns without redoing the reverse-engineering work.
+
+<br />
+
+[![CI](https://github.com/ValeroK/scrapper-tool/actions/workflows/ci.yml/badge.svg)](https://github.com/ValeroK/scrapper-tool/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
+[![Python versions](https://img.shields.io/pypi/pyversions/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
+[![Downloads](https://img.shields.io/pypi/dm/scrapper-tool.svg)](https://pypi.org/project/scrapper-tool/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230.svg)](https://github.com/astral-sh/ruff)
+[![Type-checked: mypy](https://img.shields.io/badge/type--checked-mypy-1f5082.svg)](https://mypy-lang.org/)
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
+[![GitHub Stars](https://img.shields.io/github/stars/ValeroK/scrapper-tool?style=social)](https://github.com/ValeroK/scrapper-tool/stargazers)
+[![GitHub Forks](https://img.shields.io/github/forks/ValeroK/scrapper-tool?style=social)](https://github.com/ValeroK/scrapper-tool/network/members)
+
+[**Quickstart**](#quickstart) · [**Documentation**](docs/index.md) · [**Recon playbook**](docs/recon.md) · [**Changelog**](CHANGELOG.md) · [**Contributing**](CONTRIBUTING.md)
+
+</div>
+
+---
+
+> **Status (2026-04-30):** alpha. `v0.1.0` covers the core pattern ladder, anti-bot helpers, and deterministic fixture-replay testing. `v0.2.0` adds an MCP server for LLM agents (Claude, OpenClaw, Hermes Agent, AutoGen, LangChain).
+
+## Table of contents
+
+- [Why scrapper-tool](#why-scrapper-tool)
+- [The four scraping patterns](#the-four-scraping-patterns)
+- [Architecture](#architecture)
+- [Install](#install)
+- [Quickstart](#quickstart)
+- [Documentation](#documentation)
+- [Why these tools?](#why-these-tools)
+- [Roadmap](#roadmap)
+- [Contributing](#contributing)
+- [Contributors](#contributors)
+- [Acknowledgements](#acknowledgements)
+- [License](#license)
+
+## Why scrapper-tool
+
+Most scrapers are written from scratch every time, even though 90% of the work is the same: pick the right extraction pattern, survive the TLS fingerprint, retry/backoff sanely, and write tests that don't drift the moment a site updates.
+
+`scrapper-tool` packages the parts that don't change per vendor, so you only write the parts that do.
+
+- **Pattern-first design.** Four named, documented extraction patterns (A–D) — pick the one DevTools points at, skip the rest.
+- **Anti-bot ladder built in.** Auto-walks `chrome133a → chrome124 → safari18_0 → firefox135` when a profile gets fingerprinted.
+- **Deterministic tests.** Fixture-replay (`FakeCurlSession`, `replay_fixture`, golden snapshots) — no live HTTP in CI.
+- **Optional hostile mode.** Cloudflare Turnstile / Akamai EVA defeat path via [Scrapling](https://github.com/D4Vinci/Scrapling) — opt-in extra, no Playwright bloat by default.
+- **LLM-agent ready.** `v0.2.0+` ships an MCP server so Claude, AutoGen, LangChain, etc. can drive the scraper directly.
+- **Boring stack.** `httpx`, `curl_cffi`, `selectolax`, `extruct`. No managed SaaS bundled — your code, your egress.
+
+## The four scraping patterns
+
+Web scraping in 2026 is dominated by four recurring patterns. This lib gives each pattern a documented helper plus the surrounding infrastructure (HTTP client with TLS-impersonation fallback, retry/backoff, fixture-replay testing) so you don't reinvent them per vendor.
+
+| Pattern | When to use | Helper | Cost |
+|---|---|---|---|
+| **A — JSON API** | DevTools shows an XHR returning the price-bearing JSON. Anonymous or OAuth. | `vendor_client()` + your own response model | Lowest — parse, validate, done. |
+| **B — Embedded JSON** | Document HTML carries `<script type="application/ld+json">`, `__NEXT_DATA__`, `__NUXT__`, or `self.__next_f.push(...)`. | `patterns.b.extract_product_offer()` (via [`extruct`](https://github.com/scrapinghub/extruct)) | Low — one call, broad markup coverage. |
+| **C — CSS / microdata** | Price visible in HTML, no embedded JSON. Prefer `itemprop="price"` schema.org microdata. | `patterns.c.extract_microdata_price()` (via [`selectolax`](https://github.com/rushter/selectolax)) | Medium — selectors break on ancestor reshuffles. |
+| **D — Hostile** | Cloudflare Turnstile, Akamai EVA, etc. defeat both default `httpx` and `curl_cffi`. | `patterns.d.hostile_client()` (via [Scrapling](https://github.com/D4Vinci/Scrapling)) — `pip install scrapper-tool[hostile]` | Highest — Playwright runtime, ≈400 MB image bloat. |
+
+Plus a four-profile **anti-bot ladder** (`chrome133a → chrome124 → safari18_0 → firefox135`) that auto-walks when a profile gets fingerprinted, and a `scrapper-tool canary` CLI for nightly fingerprint-health probes.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    A[Your scraper code] --> B[vendor_client / request_with_retry]
+    B --> C{TLS-sensitive?}
+    C -- no --> D[httpx]
+    C -- yes --> E[curl_cffi ladder]
+    E --> E1[chrome133a] --> E2[chrome124] --> E3[safari18_0] --> E4[firefox135]
+    D --> F[Response]
+    E4 --> F
+    F --> G{Pattern}
+    G -- A --> H[JSON API model]
+    G -- B --> I[extruct: ld+json / next_data / nuxt]
+    G -- C --> J[selectolax: microdata / CSS]
+    G -- D --> K["Scrapling (Playwright + Turnstile)"]
+    H --> L[Validated product data]
+    I --> L
+    J --> L
+    K --> L
+```
+
+## Install
+
+```bash
+pip install scrapper-tool                # core: httpx + curl_cffi + selectolax + extruct
+pip install scrapper-tool[hostile]       # adds Scrapling for Cloudflare Turnstile
+pip install scrapper-tool[agent]         # adds the MCP server (v0.2.0+) for LLM agents
+```
+
+> **Tip.** The `[hostile]` extra pulls Playwright (~400 MB). Don't install it unless you actually need pattern D.
+
+## Quickstart
+
+```python
+import asyncio
+from scrapper_tool import vendor_client, request_with_retry
+from scrapper_tool.patterns.b import extract_product_offer
+
+async def main() -> None:
+    async with vendor_client() as client:
+        resp = await request_with_retry(client, "GET", "https://example-shop.test/product/123")
+        product = extract_product_offer(resp.text, base_url=str(resp.url))
+        print(product)
+
+asyncio.run(main())
+```
+
+For TLS-sensitive vendors, flip one switch:
+
+```python
+async with vendor_client(use_curl_cffi=True) as client:
+    ...   # walks chrome133a → chrome124 → safari → firefox until one returns 200
+```
+
+See **[`docs/quickstart.md`](docs/quickstart.md)** for a 5-minute on-ramp covering all four patterns.
+
+## Documentation
+
+| | |
+|---|---|
+| **[Quickstart](docs/quickstart.md)** | 5-minute on-ramp. |
+| **[Recon playbook](docs/recon.md)** | DevTools-driven reverse-engineering of a new vendor site. |
+| **[Pattern A — JSON API](docs/patterns/a-json-api.md)** | Vendor exposes an XHR / JSON endpoint. |
+| **[Pattern B — Embedded JSON](docs/patterns/b-embedded-json.md)** | `ld+json`, `__NEXT_DATA__`, `__NUXT__`, RSC payloads. |
+| **[Pattern C — CSS / microdata](docs/patterns/c-css-microdata.md)** | `itemprop="price"`, fallback selectors. |
+| **[Pattern D — Hostile](docs/patterns/d-hostile.md)** | Cloudflare Turnstile, Akamai EVA. |
+| **[Anti-bot ladder reference](docs/reference/ladder.md)** | How the ladder walks, when to bump the primary profile. |
+| **[Test helpers](docs/reference/testing.md)** | `FakeCurlSession`, `replay_fixture`, golden-snapshot pattern. |
+| **[Agent integration](docs/agent-integration.md)** | MCP wiring for Claude, OpenClaw, Hermes Agent, AutoGen, LangChain. *(v0.2.0+)* |
+| **[2026-04-30 landscape research](docs/research/2026-04-30-landscape.md)** | Why these tools, sourced. |
+
+## Why these tools?
+
+Short version: `curl_cffi` is the only actively-maintained TLS-impersonation lib with `chrome131+`/`chrome133a`/`chrome142`/`chrome146` profiles; `puppeteer-stealth` and `playwright-extra` were deprecated in 2025-02; Scrapling is the only OSS Playwright-based stack with a working Turnstile auto-solve as of 2026; managed SaaS (Firecrawl, ZenRows, Bright Data) is deliberately not bundled.
+
+Full sourced rationale: **[`docs/research/2026-04-30-landscape.md`](docs/research/2026-04-30-landscape.md)**.
+
+## Roadmap
+
+- [x] **v0.1.0** — Core HTTP client, retry/backoff, anti-bot ladder, patterns A–D, fixture-replay test helpers.
+- [ ] **v0.2.0** — MCP server for LLM agents; canary CLI for nightly fingerprint-health probes.
+- [ ] **v0.3.0** — Pluggable rate-limit / robots.txt policies; per-vendor profile presets.
+- [ ] **v1.0.0** — API stability guarantee; broader pattern-D backends.
+
+See [`CHANGELOG.md`](CHANGELOG.md) for landed changes and [open issues](https://github.com/ValeroK/scrapper-tool/issues) for what's in flight.
+
+## Contributing
+
+PRs and issues are welcome. Every PR that meaningfully changes how we scrape lands a `CHANGELOG.md` row.
+
+- Read **[`CONTRIBUTING.md`](CONTRIBUTING.md)** for the maintenance contract.
+- Read **[`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)** before opening a discussion.
+- Good first issues live under the [`good first issue`](https://github.com/ValeroK/scrapper-tool/labels/good%20first%20issue) label.
+
+## Contributors
+
+<a href="https://github.com/ValeroK/scrapper-tool/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=ValeroK/scrapper-tool" alt="Contributors" />
+</a>
+
+Want to see your avatar here? Check [CONTRIBUTING.md](CONTRIBUTING.md) and open a PR.
+
+## Acknowledgements
+
+`scrapper-tool` stands on the shoulders of these projects:
+
+- [`httpx`](https://github.com/encode/httpx) — async HTTP client
+- [`curl_cffi`](https://github.com/lexiforest/curl_cffi) — TLS / JA3 impersonation
+- [`selectolax`](https://github.com/rushter/selectolax) — fast HTML parsing
+- [`extruct`](https://github.com/scrapinghub/extruct) — `ld+json`, microdata, RDFa extraction
+- [`Scrapling`](https://github.com/D4Vinci/Scrapling) — Playwright-based hostile-site backend
+
+## License
+
+[MIT](LICENSE) © scrapper-tool contributors.
+
+<div align="center">
+
+If `scrapper-tool` saves you time, consider [starring the repo](https://github.com/ValeroK/scrapper-tool) — it helps others find it.
+
+</div>

scrapper_tool-0.1.0/docs/agent-integration/.gitkeep

@@ -0,0 +1 @@
+# Agent integration manifests + examples (populated in M13).

scrapper_tool-0.1.0/docs/agent-integration.md

@@ -0,0 +1,33 @@
+# Agent integration
+
+> **Status: stub for v0.0.x. The MCP server lands in v0.2.0 (M13).** This doc is reserved at the top-level docs IA so the slot is visible from M0 — no retrofitting required when M13 ships.
+
+## What this will cover
+
+`scrapper-tool` ships an optional MCP server (`pip install scrapper-tool[agent]`) that exposes the lib's helpers as tools any [Model Context Protocol](https://modelcontextprotocol.io)-compatible agent can call:
+
+- `fetch_with_ladder` — issue an HTTP request through the four-profile impersonation ladder; report which profile won.
+- `extract_product` — parse a schema.org Product+Offer block from HTML (Pattern B).
+- `extract_microdata_price` — parse `itemprop="price"` schema.org microdata from HTML (Pattern C).
+- `recon_classify` — classify a URL into Pattern A/B/C/D programmatically.
+- `hostile_fetch` — Scrapling-backed fetch with auto-Turnstile-solve (requires `[hostile]` extra).
+- `canary` — fire one probe per impersonation profile against a URL, report 200/403/timeout per profile.
+
+Sections planned for M13:
+
+1. **MCP via Claude Code / Claude Desktop** — `.mcp.json` snippet.
+2. **MCP via the Anthropic SDK + `mcp-use`** — Python snippet.
+3. **OpenClaw integration** — manifest at `docs/agent-integration/openclaw.json`.
+4. **Hermes Agent integration** — manifest at `docs/agent-integration/hermes.yaml`.
+5. **AutoGen / LangChain via `mcp-use`** — code snippets.
+6. **Security note** — the MCP server runs in the user's trust boundary; tool docstrings call out unsafe inputs (raw URLs from untrusted content); the consuming agent's own permission model gates user-data-bearing fetches.
+
+## Why MCP
+
+The 2026 agent stacks (Hermes Agent's plugin system, OpenClaw's tool registry, Composio connectors) all consume MCP servers. Reference implementations — Microsoft's [Playwright MCP](https://playwright.dev/python/agents) (~29k stars), Browserbase's Stagehand MCP (~21k stars), MaitreyaM's Crawl4AI MCP, luminati-io's web-scraping-with-mcp — all converge on the same shape.
+
+The official Python MCP SDK (`mcp` package) is the canonical path; `mcp-use` bridges it to LangChain/AutoGen/the Anthropic SDK directly.
+
+---
+
+*Detailed integration docs land in v0.2.0 (M13). Track [#TBD](https://github.com/ValeroK/scrapper-tool/issues) for the milestone.*

scrapper_tool-0.1.0/docs/index.md

@@ -0,0 +1,22 @@
+# scrapper-tool documentation
+
+A reusable Python web-scraping toolkit. See the [README](../README.md) for the elevator pitch.
+
+## Table of contents
+
+- **[Quickstart](quickstart.md)** — 5-minute on-ramp.
+- **[Recon playbook](recon.md)** — DevTools-driven reverse-engineering of a new vendor site (the methodology behind PartsPilot's `scraping-vendor-recon` skill, generalised).
+- **Pattern guides**:
+  - [Pattern A — JSON API](patterns/a-json-api.md)
+  - [Pattern B — Embedded JSON](patterns/b-embedded-json.md)
+  - [Pattern C — CSS / microdata](patterns/c-css-microdata.md)
+  - [Pattern D — Hostile (Cloudflare Turnstile, Akamai EVA, …)](patterns/d-hostile.md)
+- **Reference**:
+  - [HTTP client](reference/http.md)
+  - [Anti-bot ladder](reference/ladder.md)
+  - [Test helpers](reference/testing.md)
+- **Research**:
+  - [2026-04-30 landscape snapshot](research/2026-04-30-landscape.md) — why these tools, sourced.
+  - [Tool catalog](research/tool-catalog.md) — adopted / candidate / rejected matrix.
+  - [Do-not-adopt list](research/do-not-adopt.md) — append-only rejects with dates + reasons.
+- **[Agent integration](agent-integration.md)** — MCP wiring for LLM agents (v0.2.0+).

scrapper_tool-0.1.0/docs/patterns/a-json-api.md

@@ -0,0 +1,9 @@
+# Pattern A — JSON API
+
+> *Stub — populated in M9.*
+
+**Signals**: DevTools Network → Fetch/XHR shows a request returning JSON with the price/availability fields you need. Endpoint may be anonymous, OAuth-gated, or behind a session cookie. Anonymous + stable is best.
+
+**Helper**: there isn't one — Pattern A is just `vendor_client()` + `request_with_retry()` + your own Pydantic response model. The lib's job is to provide a polite HTTP client; the response shape is yours.
+
+**Cost**: lowest. JSON parsing is faster + more robust than HTML scraping.

scrapper_tool-0.1.0/docs/patterns/b-embedded-json.md

@@ -0,0 +1,17 @@
+# Pattern B — Embedded JSON
+
+> *Stub — populated in M3.*
+
+**Signals**: the document response is HTML, but the price lives inside a `<script>` tag as JSON. Common markers:
+
+| Marker | What it is |
+|---|---|
+| `<script type="application/ld+json">` with `"@type": "Product"` | schema.org Product/Offer block. **Most common modern pattern.** |
+| `__NEXT_DATA__` | Next.js page data. |
+| `window.__INITIAL_STATE__` | Older Vue/React SSR pattern. |
+| `self.__next_f.push(...)` | Next.js 13+ App Router streaming. |
+| `<script>window.__NUXT__ = ...</script>` | Nuxt SSR data. |
+
+**Helper**: `scrapper_tool.patterns.b.extract_product_offer(html)` (via [extruct](https://github.com/scrapinghub/extruct)).
+
+**Cost**: low. One call, broad markup coverage (handles RDFa, microdata, JSON-LD with multiple `@graph` nesting shapes).