3tears-scrape 0.18.0__tar.gz

3tears_scrape-0.18.0/.gitignore

@@ -0,0 +1,216 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Claude Code local state
+.claude/
+
+# prawduct session evidence (local governance artifacts, never shipped)
+.prawduct/
+
+# macOS folder metadata
+.DS_Store

3tears_scrape-0.18.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mark Pace
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

3tears_scrape-0.18.0/PKG-INFO

@@ -0,0 +1,221 @@
+Metadata-Version: 2.4
+Name: 3tears-scrape
+Version: 0.18.0
+Summary: AI-driven, self-healing web scraping: pluggable render backends + LLM-proposed extraction candidates validated and judged against real page content, never hand-rolled per-site parsers
+Project-URL: Repository, https://github.com/pacepace/3tears
+Author: pace
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Typing :: Typed
+Requires-Python: >=3.14
+Requires-Dist: 3tears
+Requires-Dist: 3tears-agent-tools[document]
+Requires-Dist: 3tears-models
+Requires-Dist: 3tears-observe
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: camoufox>=0.4
+Requires-Dist: httpx[socks]>=0.27
+Requires-Dist: langchain-core>=0.3
+Requires-Dist: playwright<1.61
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Description-Content-Type: text/markdown
+
+# 3tears-scrape
+
+AI-driven, self-healing web scraping. Onboarding a new site is a config addition (a `ScrapeTarget` row), never a hand-written parser: an LLM proposes extraction candidates, each is structurally validated against the real page, an LLM judge picks the winner by comparing candidate values against real page content, and the winning strategy is persisted as a recipe reused on every later fetch until it stops working.
+
+```bash
+pip install 3tears-scrape
+```
+
+## What you get
+
+- **`ScrapeDriver`** -- a pluggable, backend-agnostic render contract (`RenderedPage`, `NavStep`, `NetworkCall`). Five implementations ship: `NodriverSidecarDriver` (real headless Chromium via an isolated HTTP sidecar -- see `sidecar/`), `CamoufoxDriver` (in-process stealth Firefox, no sidecar needed), `DocumentDriver` (PDF/DOCX/XLSX/CSV/TXT/MD/LaTeX via `3tears-agent-tools`' `parse_document`), `ApiDriver` (stateless JSON API GET), and `NetworkCaptureDriver` (authenticated in-session XHR capture for pages whose real data never reaches a statelessly-fetchable API).
+- **Two extraction-strategy shapes** -- CSS-selector candidates for real (or synthesized) HTML tables, and regex candidates for text-block/prose pages with no table structure at all. Both run through the identical propose -> structurally-validate -> judge -> persist cycle; the judge only ever sees extracted values and real page content, never which mechanism produced them.
+- **`run_eval_loop` / `run_eval_loop_multi_row`** -- the self-healing cycle itself: reuse a target's existing `ScrapeRecipe` with zero LLM calls while it keeps validating; only regenerate candidates once `consecutive_validation_failures` crosses a threshold.
+- **`ScrapeTarget` / `ScrapeRecipe` / `ScrapeExtraction`** -- the domain-agnostic persisted entities (three-tier L1/L2/L3-backed collections). The core never learns what a field *means* -- only that whatever a candidate extracts for a caller-supplied field name parses as its declared type.
+- **Pluggable target config** -- `TargetSource` (`StaticTargetSource` / `YamlTargetSource` / `CollectionTargetSource`) and `bootstrap_targets()` for seeding a database-backed target collection from a git-tracked YAML file, never overwriting a row already present.
+- **`ScrapeTool`** -- an ad-hoc, single-call MCP-exposed entry point: give it a URL and a field schema, get back real extracted data through the same eval loop, no target pre-configuration required.
+- **`enrichment.py`** -- a secondary, separate LLM pass capturing free-form context a structured schema has no field for, kept distinct from validated structured data.
+- **`find_target_page()`** -- a bounded-turn research agent: give it a plain-language query ("Ohio WARN Act notices"), it searches and fetches candidate pages (WebSearch/WebFetch, capped turns), deterministically verifies the winner has real extractable structure (a table, a document link, a JSON API response -- never a browser fetch, so it never needs the sidecar running), and returns a `PageFinderResult` (`url`, a `driver_backend` guess, `verified`). Independently callable -- it never persists a `ScrapeTarget` or forces extraction to follow; a caller with a known URL never needs this at all.
+- **`discover_candidates()` / `discover_row_candidates()`** -- "capture every variable on this page": the inverse of `generate_candidates`/`generate_row_candidates`, no caller-supplied `field_schema` required. An LLM proposes field names/types/selectors it finds on the page, each validated by the exact same `validate_candidate`/`validate_row_candidate` every schema-driven candidate already goes through -- a discovered field that doesn't structurally validate is dropped, never included. Returns a `DiscoverySchemaResult` whose `field_schema`/`strategy` are already shaped exactly as `ScrapeTarget.field_schema`/`ScrapeRecipe.extraction_strategy` expect. Never persists anything -- a caller decides what to do with the result.
+
+## Quickstart
+
+```python
+from threetears.scrape.drivers.nodriver_sidecar import NodriverSidecarDriver
+from threetears.scrape.eval_loop import run_eval_loop_multi_row
+from threetears.scrape.collections import ScrapeRecipeCollection, ScrapeExtractionCollection
+
+driver = NodriverSidecarDriver("http://localhost:8088")
+page = await driver.render("https://example.gov/warn-notices")
+
+extraction = await run_eval_loop_multi_row(
+    "example_warn_notices", page.html, page.final_url,
+    schema={"employer": str, "county": str, "affected_count": int},
+    recipe_collection=ScrapeRecipeCollection(registry, config),
+    extraction_collection=ScrapeExtractionCollection(registry, config),
+    api_key=openrouter_api_key,
+)
+print(extraction.validation_status, extraction.structured_fields["records"])
+```
+
+## The nodriver sidecar
+
+`NodriverSidecarDriver` talks HTTP to a genuinely separate process/container -- this is the AGPL isolation boundary nodriver (AGPL-3.0) requires, since this package itself is MIT-licensed. Build and run it via `docker buildx bake nodriver-sidecar` from the repo root; see `sidecar/README.md` for details. If you don't want the sidecar dependency at all, `CamoufoxDriver` is a fully in-process, MPL-2.0-safe alternative.
+
+## Architecture
+
+> Lifted from faidh's `src/faidh/scrape/` into this package (`scrape-task-01`, 2026-07-15) as a directory move -- zero logic changes. faidh's WARN Act plugin is used throughout as the running example of a consumer; it remains a faidh-side module (`faidh/src/faidh/intake/plugins/warn_act.py`), not part of this package, and is the only place WARN-domain meaning exists.
+
+### Why this exists
+
+WARN Act notices are published by ~24 US state labor departments in wildly inconsistent formats -- real HTML tables, PDF/DOCX/XLSX filings, prose text blocks, JSON APIs, and at least one state gated behind an authenticated search form with no stable URLs. The naive answer is "write a scraper per state." That's 24+ bespoke, brittle, un-composable parsers that all rot independently.
+
+The actual answer: **two orthogonal, config-selected axes** -- how to *fetch* a page and how to *extract* from it -- cover every state as a combination of the two, with zero jurisdiction-specific code in the core.
+
+### 1. Data flow: source → Postgres (faidh's WARN Act consumer, as an example)
+
+```mermaid
+flowchart TD
+    A[ScrapeTarget config<br/>YAML seed row] --> B["ScrapeDriver.render(url, ...)"]
+    B --> C[RenderedPage<br/>html, status, final_url, timing_ms, network_calls]
+    C --> D{run_eval_loop /<br/>run_eval_loop_multi_row}
+    D -->|reuse existing recipe| E[ScrapeRecipe]
+    D -->|regenerate via LLM candidates + judge| E
+    E --> F[ScrapeExtraction persisted<br/>structured_fields = records]
+    F --> G["WarnActPlugin._produce()<br/>generic records → Tier-2 fields"]
+    G --> H[ArbitrarySignalEntity.create<br/>signal_type=warn_act_notice]
+    H --> I["PluginBase.collect()<br/>throttle-gate → _produce() → save_entity() → yield"]
+    I --> J[(Postgres:<br/>faidh_arbitrary_signals)]
+    I --> K["publish_arbitrary_signals()"]
+    K --> L[NATS subject: arbitrary_signals]
+    L --> M[Downstream: scoreboard, retrospect, ...]
+
+    F -.-> N[(Postgres:<br/>scrape_targets / scrape_recipes /<br/>scrape_extractions — provenance)]
+```
+
+Two Postgres-writing paths run **in parallel, not sequentially**:
+
+- **Provenance** (`scrape_targets`, `scrape_recipes`, `scrape_extractions`) -- what was fetched, what strategy won, when. Domain-agnostic, lives in this package.
+- **Signal** (`faidh_arbitrary_signals`) -- what faidh's forecasting actually reads. Faidh-specific, mapped by `WarnActPlugin._produce()`, stays in faidh.
+
+### 2. The abstraction: `ScrapeTarget` is the entire adapter surface
+
+Zero jurisdiction-specific Python exists anywhere in this package. In faidh's WARN Act consumer, every state is a `ScrapeTarget` row (`faidh/src/faidh/intake/plugins/seeds/warn_act_targets.yaml`), bootstrapped into the DB.
+
+| Field | Controls |
+|---|---|
+| `url` | what to fetch |
+| `driver_backend` | which of 5 `ScrapeDriver` implementations renders it |
+| `wait_for` | CSS selector to settle on (async-loading pages) |
+| `nav_steps` | ordered click/fill/wait_for/wait_ms/scroll_into_view/scroll_page/evaluate actions (search-form-gated, lazy-render, or in-page-JS-state targets) |
+| `timeout_seconds` | per-target render budget |
+| `field_schema` | field name → Python type, e.g. `{"employer": str, "county": str}` |
+| `extraction_strategy_type` | `"css"` or `"regex"` |
+| `api_results_path` / `api_fragment_field` | JSON traversal (API driver only) |
+| `multi_row` | one record vs. many per page |
+
+The core (`driver.py`, `extraction.py`, `eval_loop.py`, `collections.py`) never branches on target identity -- only on `ScrapeTarget` fields and a caller-supplied `field_schema`. It doesn't know what `employer` *means*; it only enforces that whatever a candidate extracts for that field name parses as the declared type. Stated in `extraction.py`'s own docstring: *"this module never hardcodes what a field means."*
+
+### 3. Fetch × Extract = many combinations, not one scraper per site
+
+```mermaid
+flowchart LR
+    subgraph Fetch["Axis 1 — driver_backend (drivers/)"]
+        D1[nodriver_sidecar<br/>headless Chromium, sidecar]
+        D2[camoufox<br/>in-process stealth Firefox]
+        D3[document<br/>PDF/DOCX/XLSX/CSV → parse_document]
+        D4[api<br/>stateless JSON GET]
+        D5[network_capture<br/>authenticated XHR capture]
+    end
+    subgraph Converge["Everything converges here"]
+        R[RenderedPage.html<br/>real or synthetic — identical downstream]
+    end
+    subgraph Extract["Axis 2 — extraction_strategy_type"]
+        S1[css<br/>real/synthesized table → LLM proposes selectors]
+        S2[regex<br/>prose text block → LLM proposes named-group patterns]
+    end
+    D1 & D2 & D3 & D4 & D5 --> R
+    R --> S1
+    R --> S2
+    S1 & S2 --> J[Identical propose → validate →<br/>judge → persist cycle, eval_loop.py]
+```
+
+In faidh's WARN Act consumer, every one of the ~24 onboarded states is one config row picking a combination from this matrix. A new driver/strategy is only written when a target needs a *fetch mechanism* or *extraction shape* that doesn't exist yet -- five times across the WARN Act build: regex strategy, `ApiDriver`, native CSV support, `NetworkCaptureDriver`, plus the original CSS/nodriver baseline.
+
+### 4. "The signal must carry the value; the model must never infer it" -- where the rule actually lives
+
+| Checkpoint | Mechanism | Proof (faidh's WARN Act consumer) |
+|---|---|---|
+| Structural validation, all-or-nothing per record | `validate_candidate` / `validate_row_candidate` (+ regex variants), `extraction.py` -- no coercion, no defaults, a record missing even one requested field is **dropped entirely** | Oklahoma: 217 real records exist, only 128 survived -- the other 89 genuinely lack a workforce-region value on the page |
+| Schema omits what a page can't provide | `field_schema` per target -- never forces a value that isn't there | New York schema doesn't request `affected_count`/`effective_date` because its page has neither |
+| Judge does semantic verification, not just structural plausibility | `_judge_candidates` / `_judge_row_candidates`, `eval_loop.py` -- sees real page HTML + candidate values, told structural validity is already checked, job is *semantic correctness* | Georgia: an earlier schema mistakenly asked for `county`; judge rejected the resulting hallucinated-mapping candidate rather than accepting it |
+| Confirmed vs. best-guess survives to the signal | `ScrapeExtraction.validation_status` ∈ `validated` / `needs_review` / `failed` | `WarnActPlugin._produce()` logs a distinct warning for `needs_review` yields rather than treating them as confirmed |
+
+**Open tension, not fully resolved:** a `needs_review` record still gets yielded as a real signal -- "surface for review, never silently drop" beats "silently withhold," by deliberate design. But nothing downstream currently filters on `validation_status`. The signal *carries* the distinction; no consumer currently *reads* it. A consumer treating every signal as equally trustworthy today would be wrong to.
+
+### 5. Where new capabilities plug in
+
+```mermaid
+flowchart TD
+    Core["This package<br/>driver.py / extraction.py / eval_loop.py / collections.py"]
+    T01["scrape-task-01 (done, 2026-07-15)<br/>Lift: moved core here from faidh<br/>package-boundary move only, zero logic change"]
+    T02["scrape-task-02 (planned)<br/>Page-finder agent<br/>outputs ScrapeTarget-shaped guess<br/>(URL + driver_backend/wait_for)"]
+    T03["scrape-task-03 (planned)<br/>Schema-discovery mode<br/>flag on generate_candidates —<br/>schema OUT instead of schema IN"]
+    T01 --> Core
+    Core -.plain data in/out, no hidden state.-> T02
+    Core -.plain data in/out, no hidden state.-> T03
+    T02 --> BT["bootstrap_targets()<br/>target_source.py"]
+    BT --> Core
+```
+
+`task-02`/`task-03` are new front-end stages that will produce/consume the same plain data shapes (`RenderedPage`, `FieldSchema`, `ScrapeExtraction`) the core already uses -- neither requires touching `driver.py`, `eval_loop.py`, or `collections.py`.
+
+### 6. Module map
+
+```
+packages/scrape/src/threetears/scrape/
+├── driver.py                 ScrapeDriver ABC, RenderedPage, NavStep, NetworkCall
+├── drivers/
+│   ├── nodriver_sidecar.py   HTTP → headless Chromium sidecar (AGPL boundary)
+│   ├── camoufox.py           in-process stealth Firefox (MPL-2.0-safe)
+│   ├── document.py           parse_document() + document_text_to_html()
+│   ├── api.py                stateless JSON GET, _resolve_path()
+│   └── network_capture.py    wraps another driver, _find_largest_record_list()
+├── extraction.py             generate_candidates/generate_row_candidates (+ regex),
+│                              validate_candidate/validate_row_candidate (+ regex),
+│                              html_to_text(), strip_boilerplate()
+├── eval_loop.py               run_eval_loop()/run_eval_loop_multi_row(),
+│                              _judge_candidates()/_judge_row_candidates(),
+│                              _persist_extraction(), _save_recipe()
+├── collections.py             ScrapeTarget/ScrapeRecipe/ScrapeExtraction (BaseEntity)
+│                              + BaseCollection subclasses (L1/L2/L3 via
+│                              threetears.core.backends.protocol.DurableStore)
+├── migrations.py               v001-v008, PACKAGE_NAME="3tears_scrape"
+├── target_source.py            TargetSource ABC, YamlTargetSource, CollectionTargetSource,
+│                              StaticTargetSource, bootstrap_targets()
+├── tool.py                     ScrapeTool — ad-hoc single-call MCP entry point
+├── enrichment.py                secondary free-form LLM notes pass (separate from structured_fields)
+└── llm_retry.py                 bounded_retry_structured_call() — shared by extraction.py + eval_loop.py
+
+Example consumer (faidh repo, not part of this package):
+faidh/src/faidh/intake/plugins/warn_act.py               WarnActPlugin — the ONLY place WARN-domain meaning exists
+faidh/src/faidh/intake/plugins/seeds/warn_act_targets.yaml   the ScrapeTarget config rows
+faidh/src/faidh/intake/runner.py                          publish_observations(), publish_arbitrary_signals(), poll_scrape_targets()
+faidh/src/faidh/intake/plugins/__init__.py                PluginBase.collect() — persist-then-yield template method
+faidh/src/faidh/intake/signals/arbitrary.py               ArbitrarySignalEntity/Collection — the actual Postgres sink
+```
+
+**Call chain, one live poll cycle (faidh's WARN Act consumer):** `runner.publish_observations()` → `WarnActPlugin.collect()` (inherited `PluginBase`) → `WarnActPlugin._produce()` → resolves `self._drivers[target.driver_backend]` → `driver.render(...)` → `run_eval_loop_multi_row(...)` (or `run_eval_loop` if `multi_row=False`) → `_reuse_row_recipe`/`_regenerate_row_recipe` (or `_regex_` variants) → `_persist_extraction()` writes `scrape_extractions` → back in `_produce()`, each record becomes an `ArbitrarySignalEntity` → `collect()`'s `save_entity()` writes `faidh_arbitrary_signals` → `publish_arbitrary_signals()` re-drives `collect()` and publishes each yielded entity to `arbitrary_signals()` on NATS.
+
+## License
+
+MIT. See [LICENSE](LICENSE). The bundled sidecar (`sidecar/`) wraps nodriver (AGPL-3.0) as a genuinely separate process -- see `sidecar/README.md`.

3tears_scrape-0.18.0/README.md

@@ -0,0 +1,192 @@
+# 3tears-scrape
+
+AI-driven, self-healing web scraping. Onboarding a new site is a config addition (a `ScrapeTarget` row), never a hand-written parser: an LLM proposes extraction candidates, each is structurally validated against the real page, an LLM judge picks the winner by comparing candidate values against real page content, and the winning strategy is persisted as a recipe reused on every later fetch until it stops working.
+
+```bash
+pip install 3tears-scrape
+```
+
+## What you get
+
+- **`ScrapeDriver`** -- a pluggable, backend-agnostic render contract (`RenderedPage`, `NavStep`, `NetworkCall`). Five implementations ship: `NodriverSidecarDriver` (real headless Chromium via an isolated HTTP sidecar -- see `sidecar/`), `CamoufoxDriver` (in-process stealth Firefox, no sidecar needed), `DocumentDriver` (PDF/DOCX/XLSX/CSV/TXT/MD/LaTeX via `3tears-agent-tools`' `parse_document`), `ApiDriver` (stateless JSON API GET), and `NetworkCaptureDriver` (authenticated in-session XHR capture for pages whose real data never reaches a statelessly-fetchable API).
+- **Two extraction-strategy shapes** -- CSS-selector candidates for real (or synthesized) HTML tables, and regex candidates for text-block/prose pages with no table structure at all. Both run through the identical propose -> structurally-validate -> judge -> persist cycle; the judge only ever sees extracted values and real page content, never which mechanism produced them.
+- **`run_eval_loop` / `run_eval_loop_multi_row`** -- the self-healing cycle itself: reuse a target's existing `ScrapeRecipe` with zero LLM calls while it keeps validating; only regenerate candidates once `consecutive_validation_failures` crosses a threshold.
+- **`ScrapeTarget` / `ScrapeRecipe` / `ScrapeExtraction`** -- the domain-agnostic persisted entities (three-tier L1/L2/L3-backed collections). The core never learns what a field *means* -- only that whatever a candidate extracts for a caller-supplied field name parses as its declared type.
+- **Pluggable target config** -- `TargetSource` (`StaticTargetSource` / `YamlTargetSource` / `CollectionTargetSource`) and `bootstrap_targets()` for seeding a database-backed target collection from a git-tracked YAML file, never overwriting a row already present.
+- **`ScrapeTool`** -- an ad-hoc, single-call MCP-exposed entry point: give it a URL and a field schema, get back real extracted data through the same eval loop, no target pre-configuration required.
+- **`enrichment.py`** -- a secondary, separate LLM pass capturing free-form context a structured schema has no field for, kept distinct from validated structured data.
+- **`find_target_page()`** -- a bounded-turn research agent: give it a plain-language query ("Ohio WARN Act notices"), it searches and fetches candidate pages (WebSearch/WebFetch, capped turns), deterministically verifies the winner has real extractable structure (a table, a document link, a JSON API response -- never a browser fetch, so it never needs the sidecar running), and returns a `PageFinderResult` (`url`, a `driver_backend` guess, `verified`). Independently callable -- it never persists a `ScrapeTarget` or forces extraction to follow; a caller with a known URL never needs this at all.
+- **`discover_candidates()` / `discover_row_candidates()`** -- "capture every variable on this page": the inverse of `generate_candidates`/`generate_row_candidates`, no caller-supplied `field_schema` required. An LLM proposes field names/types/selectors it finds on the page, each validated by the exact same `validate_candidate`/`validate_row_candidate` every schema-driven candidate already goes through -- a discovered field that doesn't structurally validate is dropped, never included. Returns a `DiscoverySchemaResult` whose `field_schema`/`strategy` are already shaped exactly as `ScrapeTarget.field_schema`/`ScrapeRecipe.extraction_strategy` expect. Never persists anything -- a caller decides what to do with the result.
+
+## Quickstart
+
+```python
+from threetears.scrape.drivers.nodriver_sidecar import NodriverSidecarDriver
+from threetears.scrape.eval_loop import run_eval_loop_multi_row
+from threetears.scrape.collections import ScrapeRecipeCollection, ScrapeExtractionCollection
+
+driver = NodriverSidecarDriver("http://localhost:8088")
+page = await driver.render("https://example.gov/warn-notices")
+
+extraction = await run_eval_loop_multi_row(
+    "example_warn_notices", page.html, page.final_url,
+    schema={"employer": str, "county": str, "affected_count": int},
+    recipe_collection=ScrapeRecipeCollection(registry, config),
+    extraction_collection=ScrapeExtractionCollection(registry, config),
+    api_key=openrouter_api_key,
+)
+print(extraction.validation_status, extraction.structured_fields["records"])
+```
+
+## The nodriver sidecar
+
+`NodriverSidecarDriver` talks HTTP to a genuinely separate process/container -- this is the AGPL isolation boundary nodriver (AGPL-3.0) requires, since this package itself is MIT-licensed. Build and run it via `docker buildx bake nodriver-sidecar` from the repo root; see `sidecar/README.md` for details. If you don't want the sidecar dependency at all, `CamoufoxDriver` is a fully in-process, MPL-2.0-safe alternative.
+
+## Architecture
+
+> Lifted from faidh's `src/faidh/scrape/` into this package (`scrape-task-01`, 2026-07-15) as a directory move -- zero logic changes. faidh's WARN Act plugin is used throughout as the running example of a consumer; it remains a faidh-side module (`faidh/src/faidh/intake/plugins/warn_act.py`), not part of this package, and is the only place WARN-domain meaning exists.
+
+### Why this exists
+
+WARN Act notices are published by ~24 US state labor departments in wildly inconsistent formats -- real HTML tables, PDF/DOCX/XLSX filings, prose text blocks, JSON APIs, and at least one state gated behind an authenticated search form with no stable URLs. The naive answer is "write a scraper per state." That's 24+ bespoke, brittle, un-composable parsers that all rot independently.
+
+The actual answer: **two orthogonal, config-selected axes** -- how to *fetch* a page and how to *extract* from it -- cover every state as a combination of the two, with zero jurisdiction-specific code in the core.
+
+### 1. Data flow: source → Postgres (faidh's WARN Act consumer, as an example)
+
+```mermaid
+flowchart TD
+    A[ScrapeTarget config<br/>YAML seed row] --> B["ScrapeDriver.render(url, ...)"]
+    B --> C[RenderedPage<br/>html, status, final_url, timing_ms, network_calls]
+    C --> D{run_eval_loop /<br/>run_eval_loop_multi_row}
+    D -->|reuse existing recipe| E[ScrapeRecipe]
+    D -->|regenerate via LLM candidates + judge| E
+    E --> F[ScrapeExtraction persisted<br/>structured_fields = records]
+    F --> G["WarnActPlugin._produce()<br/>generic records → Tier-2 fields"]
+    G --> H[ArbitrarySignalEntity.create<br/>signal_type=warn_act_notice]
+    H --> I["PluginBase.collect()<br/>throttle-gate → _produce() → save_entity() → yield"]
+    I --> J[(Postgres:<br/>faidh_arbitrary_signals)]
+    I --> K["publish_arbitrary_signals()"]
+    K --> L[NATS subject: arbitrary_signals]
+    L --> M[Downstream: scoreboard, retrospect, ...]
+
+    F -.-> N[(Postgres:<br/>scrape_targets / scrape_recipes /<br/>scrape_extractions — provenance)]
+```
+
+Two Postgres-writing paths run **in parallel, not sequentially**:
+
+- **Provenance** (`scrape_targets`, `scrape_recipes`, `scrape_extractions`) -- what was fetched, what strategy won, when. Domain-agnostic, lives in this package.
+- **Signal** (`faidh_arbitrary_signals`) -- what faidh's forecasting actually reads. Faidh-specific, mapped by `WarnActPlugin._produce()`, stays in faidh.
+
+### 2. The abstraction: `ScrapeTarget` is the entire adapter surface
+
+Zero jurisdiction-specific Python exists anywhere in this package. In faidh's WARN Act consumer, every state is a `ScrapeTarget` row (`faidh/src/faidh/intake/plugins/seeds/warn_act_targets.yaml`), bootstrapped into the DB.
+
+| Field | Controls |
+|---|---|
+| `url` | what to fetch |
+| `driver_backend` | which of 5 `ScrapeDriver` implementations renders it |
+| `wait_for` | CSS selector to settle on (async-loading pages) |
+| `nav_steps` | ordered click/fill/wait_for/wait_ms/scroll_into_view/scroll_page/evaluate actions (search-form-gated, lazy-render, or in-page-JS-state targets) |
+| `timeout_seconds` | per-target render budget |
+| `field_schema` | field name → Python type, e.g. `{"employer": str, "county": str}` |
+| `extraction_strategy_type` | `"css"` or `"regex"` |
+| `api_results_path` / `api_fragment_field` | JSON traversal (API driver only) |
+| `multi_row` | one record vs. many per page |
+
+The core (`driver.py`, `extraction.py`, `eval_loop.py`, `collections.py`) never branches on target identity -- only on `ScrapeTarget` fields and a caller-supplied `field_schema`. It doesn't know what `employer` *means*; it only enforces that whatever a candidate extracts for that field name parses as the declared type. Stated in `extraction.py`'s own docstring: *"this module never hardcodes what a field means."*
+
+### 3. Fetch × Extract = many combinations, not one scraper per site
+
+```mermaid
+flowchart LR
+    subgraph Fetch["Axis 1 — driver_backend (drivers/)"]
+        D1[nodriver_sidecar<br/>headless Chromium, sidecar]
+        D2[camoufox<br/>in-process stealth Firefox]
+        D3[document<br/>PDF/DOCX/XLSX/CSV → parse_document]
+        D4[api<br/>stateless JSON GET]
+        D5[network_capture<br/>authenticated XHR capture]
+    end
+    subgraph Converge["Everything converges here"]
+        R[RenderedPage.html<br/>real or synthetic — identical downstream]
+    end
+    subgraph Extract["Axis 2 — extraction_strategy_type"]
+        S1[css<br/>real/synthesized table → LLM proposes selectors]
+        S2[regex<br/>prose text block → LLM proposes named-group patterns]
+    end
+    D1 & D2 & D3 & D4 & D5 --> R
+    R --> S1
+    R --> S2
+    S1 & S2 --> J[Identical propose → validate →<br/>judge → persist cycle, eval_loop.py]
+```
+
+In faidh's WARN Act consumer, every one of the ~24 onboarded states is one config row picking a combination from this matrix. A new driver/strategy is only written when a target needs a *fetch mechanism* or *extraction shape* that doesn't exist yet -- five times across the WARN Act build: regex strategy, `ApiDriver`, native CSV support, `NetworkCaptureDriver`, plus the original CSS/nodriver baseline.
+
+### 4. "The signal must carry the value; the model must never infer it" -- where the rule actually lives
+
+| Checkpoint | Mechanism | Proof (faidh's WARN Act consumer) |
+|---|---|---|
+| Structural validation, all-or-nothing per record | `validate_candidate` / `validate_row_candidate` (+ regex variants), `extraction.py` -- no coercion, no defaults, a record missing even one requested field is **dropped entirely** | Oklahoma: 217 real records exist, only 128 survived -- the other 89 genuinely lack a workforce-region value on the page |
+| Schema omits what a page can't provide | `field_schema` per target -- never forces a value that isn't there | New York schema doesn't request `affected_count`/`effective_date` because its page has neither |
+| Judge does semantic verification, not just structural plausibility | `_judge_candidates` / `_judge_row_candidates`, `eval_loop.py` -- sees real page HTML + candidate values, told structural validity is already checked, job is *semantic correctness* | Georgia: an earlier schema mistakenly asked for `county`; judge rejected the resulting hallucinated-mapping candidate rather than accepting it |
+| Confirmed vs. best-guess survives to the signal | `ScrapeExtraction.validation_status` ∈ `validated` / `needs_review` / `failed` | `WarnActPlugin._produce()` logs a distinct warning for `needs_review` yields rather than treating them as confirmed |
+
+**Open tension, not fully resolved:** a `needs_review` record still gets yielded as a real signal -- "surface for review, never silently drop" beats "silently withhold," by deliberate design. But nothing downstream currently filters on `validation_status`. The signal *carries* the distinction; no consumer currently *reads* it. A consumer treating every signal as equally trustworthy today would be wrong to.
+
+### 5. Where new capabilities plug in
+
+```mermaid
+flowchart TD
+    Core["This package<br/>driver.py / extraction.py / eval_loop.py / collections.py"]
+    T01["scrape-task-01 (done, 2026-07-15)<br/>Lift: moved core here from faidh<br/>package-boundary move only, zero logic change"]
+    T02["scrape-task-02 (planned)<br/>Page-finder agent<br/>outputs ScrapeTarget-shaped guess<br/>(URL + driver_backend/wait_for)"]
+    T03["scrape-task-03 (planned)<br/>Schema-discovery mode<br/>flag on generate_candidates —<br/>schema OUT instead of schema IN"]
+    T01 --> Core
+    Core -.plain data in/out, no hidden state.-> T02
+    Core -.plain data in/out, no hidden state.-> T03
+    T02 --> BT["bootstrap_targets()<br/>target_source.py"]
+    BT --> Core
+```
+
+`task-02`/`task-03` are new front-end stages that will produce/consume the same plain data shapes (`RenderedPage`, `FieldSchema`, `ScrapeExtraction`) the core already uses -- neither requires touching `driver.py`, `eval_loop.py`, or `collections.py`.
+
+### 6. Module map
+
+```
+packages/scrape/src/threetears/scrape/
+├── driver.py                 ScrapeDriver ABC, RenderedPage, NavStep, NetworkCall
+├── drivers/
+│   ├── nodriver_sidecar.py   HTTP → headless Chromium sidecar (AGPL boundary)
+│   ├── camoufox.py           in-process stealth Firefox (MPL-2.0-safe)
+│   ├── document.py           parse_document() + document_text_to_html()
+│   ├── api.py                stateless JSON GET, _resolve_path()
+│   └── network_capture.py    wraps another driver, _find_largest_record_list()
+├── extraction.py             generate_candidates/generate_row_candidates (+ regex),
+│                              validate_candidate/validate_row_candidate (+ regex),
+│                              html_to_text(), strip_boilerplate()
+├── eval_loop.py               run_eval_loop()/run_eval_loop_multi_row(),
+│                              _judge_candidates()/_judge_row_candidates(),
+│                              _persist_extraction(), _save_recipe()
+├── collections.py             ScrapeTarget/ScrapeRecipe/ScrapeExtraction (BaseEntity)
+│                              + BaseCollection subclasses (L1/L2/L3 via
+│                              threetears.core.backends.protocol.DurableStore)
+├── migrations.py               v001-v008, PACKAGE_NAME="3tears_scrape"
+├── target_source.py            TargetSource ABC, YamlTargetSource, CollectionTargetSource,
+│                              StaticTargetSource, bootstrap_targets()
+├── tool.py                     ScrapeTool — ad-hoc single-call MCP entry point
+├── enrichment.py                secondary free-form LLM notes pass (separate from structured_fields)
+└── llm_retry.py                 bounded_retry_structured_call() — shared by extraction.py + eval_loop.py
+
+Example consumer (faidh repo, not part of this package):
+faidh/src/faidh/intake/plugins/warn_act.py               WarnActPlugin — the ONLY place WARN-domain meaning exists
+faidh/src/faidh/intake/plugins/seeds/warn_act_targets.yaml   the ScrapeTarget config rows
+faidh/src/faidh/intake/runner.py                          publish_observations(), publish_arbitrary_signals(), poll_scrape_targets()
+faidh/src/faidh/intake/plugins/__init__.py                PluginBase.collect() — persist-then-yield template method
+faidh/src/faidh/intake/signals/arbitrary.py               ArbitrarySignalEntity/Collection — the actual Postgres sink
+```
+
+**Call chain, one live poll cycle (faidh's WARN Act consumer):** `runner.publish_observations()` → `WarnActPlugin.collect()` (inherited `PluginBase`) → `WarnActPlugin._produce()` → resolves `self._drivers[target.driver_backend]` → `driver.render(...)` → `run_eval_loop_multi_row(...)` (or `run_eval_loop` if `multi_row=False`) → `_reuse_row_recipe`/`_regenerate_row_recipe` (or `_regex_` variants) → `_persist_extraction()` writes `scrape_extractions` → back in `_produce()`, each record becomes an `ArbitrarySignalEntity` → `collect()`'s `save_entity()` writes `faidh_arbitrary_signals` → `publish_arbitrary_signals()` re-drives `collect()` and publishes each yielded entity to `arbitrary_signals()` on NATS.
+
+## License
+
+MIT. See [LICENSE](LICENSE). The bundled sidecar (`sidecar/`) wraps nodriver (AGPL-3.0) as a genuinely separate process -- see `sidecar/README.md`.