matrx-scraper 0.1.0__tar.gz

matrx_scraper-0.1.0/.env.example

@@ -0,0 +1,34 @@
+# =============================================================================
+# Matrx Scraper — Standalone Server Configuration
+# =============================================================================
+# Copy this file to .env and fill in the values.
+
+# --- Required ---
+DATABASE_URL=postgresql://user:password@host:5432/dbname
+SUPABASE_JWT_SECRET=your-supabase-jwt-secret
+
+# --- Optional auth ---
+# ADMIN_API_TOKEN is accepted by AuthMiddleware but the static-token escape
+# hatch was removed in 2026-05 — it is ignored. Leave unset.
+ADMIN_API_TOKEN=
+
+# --- Search ---
+BRAVE_API_KEY=
+
+# --- Proxy ---
+PROXY_DATACENTER_URL=
+PROXY_RESIDENTIAL_URL=
+
+# --- Browser pool ---
+BROWSER_POOL_SIZE=3
+ENABLE_BROWSER_POOL=true
+
+# --- Feature toggles ---
+ENABLE_CACHE=true
+ENABLE_DOMAIN_CONFIG=true
+
+# --- Server ---
+HOST=0.0.0.0
+PORT=8000
+WORKERS=1
+LOG_LEVEL=info

matrx_scraper-0.1.0/.gitignore

@@ -0,0 +1,257 @@
+*.pyc
+secrets/
+ignore/
+temp/
+logs/
+todo
+text_notes/
+aidream/secrets/2.env
+automation_matrix/matrix_processing/temp/*
+cd
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+.venv/
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+# The blanket lib/ rule above is from the standard Python .gitignore template
+# and was silently swallowing TS source under the SPA `src/lib/` folders.
+# Re-allow them explicitly so frontend builds don't ship without their lib layer.
+!dashboard/src/lib/
+!dashboard/src/lib/**
+!workflow-studio/src/lib/
+!workflow-studio/src/lib/**
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+ai/tests/clean_response.json
+ai/tests/cx_storage_response.json
+ai/tests/execution_test.py
+ai/tests/final_response.json
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.env_remote
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+.env.armanonly
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# random armani files
+/armani_dev/secrets/
+/armani/
+/_armani/
+
+
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+.idea/
+.vscode/
+/node_modules/
+
+dump.rdb
+
+frontend/
+
+# AME Temp Files and directory structure
+# Ignore all files in the temp directory and its subdirectories
+/temp/**/*
+/tmp/**/*
+
+# Allow .gitkeep files to retain directory structure
+!/temp/**/.gitkeep
+!/tmp/**/.gitkeep
+
+# Armani
+.history*
+.history/
+local_data/
+local_reports_data/
+webscraper/quick_scrapes/temp/
+automation_matrix/ai_apis/fireworks/_dev/*
+automation_matrix/ai_apis/fireworks/_dev/fireworks_sample.py
+*.pdf
+*.flac
+*.mp3
+*.wav
+miniconda.sh
+/database/python_sql/temp_data/
+ .history*
+ .history/
+.history/
+
+_dev/
+/_dev/
+requirements_filtered.txt
+
+# matrx-dev-tools backups
+.env-backups/
+# Matrx Ship config (contains API key)
+.matrx-ship.json
+
+# Matrx config (contains API keys)
+.matrx.json
+.matrx-tools.conf
+
+# Claude Code local worktrees and per-user settings
+.claude/worktrees/
+.claude/settings.local.json
+
+# Append-only snapshots from matrx_utils.update_history (unbounded; do not commit)
+common/utils/data_in_code/data_history.json
+packages/matrx-utils/matrx_utils/data_in_code/data_history.json
+
+# Tool-dispatch debug logs — one file per server start, never committed
+.matrx-debug/
+
+# macOS Finder metadata
+.DS_Store
+**/.DS_Store
+
+# Environment files
+.env
+.env.*
+*.env
+*.env.*
+
+# Keep safe templates trackable
+!.env.example
+!.env.sample
+!.env.template

matrx_scraper-0.1.0/CLAUDE.md

@@ -0,0 +1,104 @@
+# CLAUDE.md — matrx-scraper
+
+> **Operating Principle: Build the platform, not the artifact.** Every task is a probe that exposes a missing capability — build it, then consume it. Code that only serves one artifact is forbidden. Full doctrine: [/PRINCIPLES.md](../../PRINCIPLES.md).
+
+**Package:** `matrx-scraper` (PyPI) — Python 3.12+ — currently v0.1.0 (alpha)
+**Role in the graph:** Tier 2. Depends on `matrx-utils`. Optionally depends on `matrx-connect` (for streaming/events integration) and `matrx-orm` (for the Postgres-backed domain-config backend). Used by the aidream app and potentially by matrx-graph nodes.
+
+---
+
+## Read this first
+
+matrx-scraper is the canonical web scraping + HTML parsing + site crawling + search + rendering pipeline for the Matrx family. It **replaces** the legacy root-level `scraper/` directory and parts of `research/`. Do not add new scraping logic to those folders.
+
+This package is also a strong example of optional-sibling integration: it works standalone (just `pip install matrx-scraper`), degrades gracefully when matrx-connect isn't present, and uses an `_ext.py` registry for host-provided hooks.
+
+---
+
+## What this package provides
+
+- **Scraping** (`matrx_scraper.scraper`, `matrx_scraper.orchestrator`): `scrape(url, **opts)`, `scrape_many(urls, ...)`, `scrape_many_stream(...)`, `ScrapeResult`, `ScrapeOptions`, `ScrapeService`.
+- **Parsing pipeline** (`matrx_scraper.parser`): 8-stage HTML pipeline — normalize → `NoiseRemover` → `ScrapeFilter` → `ElementExtractor` → `LinkExtractor` → metadata (extruct) → hashing (MinHash/SimHash) → markdownify. Entry point: `parse_html(html, **opts)` and `ParserOrchestrator`.
+- **Crawling** (`matrx_scraper.crawler`): `crawl_site(base_url)`, `SiteCrawler` — async BFS traversal, robots.txt-aware.
+- **Search** (`matrx_scraper.search`): `BraveSearchClient`.
+- **Caching** (`matrx_scraper.cache`): `CacheBackend` with `MemoryCache`, `TwoTierCache`.
+- **Per-URL / per-domain config** (`matrx_scraper.domain_config`): `DomainConfigBackend`. Default is static; a Postgres-backed backend is available behind an optional `matrx-orm` extra.
+- **Browser automation** (optional): `PlaywrightBrowserPool` behind the `[browser]` extra.
+- **FastAPI server** (optional, `[server]` extra): `matrx-scraper` CLI / `server/__main__.py`, plus routers under `api/`.
+
+Public API is the ~20 symbols in `matrx_scraper/__init__.py`. Keep stable.
+
+---
+
+## The `_ext` / `configure_ext` injection pattern
+
+Because matrx-scraper wants to integrate with matrx-connect's event system when present, but also run standalone, it uses the `_ext` registry pattern:
+
+```python
+# Host wires it at startup, passing matrx-connect types in:
+import matrx_scraper
+from matrx_connect.context.events import InfoPayload, WarningPayload
+from matrx_connect.context.data_types import …
+
+matrx_scraper.configure_ext(
+    info_payload_cls=InfoPayload,
+    warning_payload_cls=WarningPayload,
+    # …
+)
+```
+
+Inside this package:
+
+- All matrx-connect imports are **conditional** — wrapped in `has_ext("connect")` checks or lazy imports behind functions.
+- `ScrapeService` (the FastAPI adapter in `service.py`) accepts an injected `AppContext` shape; it does not import matrx-connect eagerly.
+- Cache / domain-config / browser-pool backends are all injectable — pass them to the orchestrator or leave as defaults.
+
+---
+
+## Dependency rules specific to this package
+
+- ✅ `from matrx_utils import …` — declared hard dep.
+- ✅ `from matrx_connect import …` — **only** inside conditionally-imported code paths, and only when `has_ext("connect")` confirms it was configured. Never at module top level of a core scraping file.
+- ✅ `from matrx_orm import …` — **only** inside the Postgres domain-config backend module, behind the `[postgres]` extra.
+- ❌ No `from matrx_ai import …` (matrx-ai depends on graph, not scraper; keep it that way).
+- ❌ No `from matrx_graph import …` at module top level. If a scraping step wants to run a graph, the host composes them; matrx-scraper doesn't orchestrate graphs internally.
+- ❌ No `from aidream import …`. No imports from root `scraper/`, `research/`, `common/`, `config/`, `api_management/`.
+
+---
+
+## Python standards (same as root)
+
+- Full type hints. Parser/scraper result types are public contracts — Pydantic.
+- No docstrings except on the public API (`scrape`, `parse_html`, `ScrapeResult`, `ScrapeService`, `crawl_site`). One line.
+- Hot paths: HTML parsing (selectolax over bs4 where both are available), noise removal, link classification. Avoid repeated DOM traversals; do one pass and reuse structures.
+- Explicit exception handling — every network op has retry + timeout, and errors surface through the emitter when one is available.
+
+---
+
+## Testing this package in isolation
+
+```bash
+uv run pytest packages/matrx-scraper/tests
+```
+
+Tests must run with only matrx-utils installed. Browser + Postgres-backed tests live behind extras and skip when the extras aren't installed.
+
+---
+
+## Relationship to legacy `scraper/` and `research/`
+
+Both are being **migrated out**. The internal docs in this package (`MIGRATION_STATUS.md`, `GAPS_TO_FIX.md`, `LEGACY_AUDIT.md`, `MIGRATION_GUIDE.md`) track specifics. High-level rules:
+
+- Consumers in `research/` should import from `matrx_scraper`, not from the old `scraper/` directory.
+- Features still missing from matrx-scraper (MCP tool adapters, auto-clicker, shadow-DOM walker, stateful browser sessions, disk cache, StackBlitz preprocessor, etc.) should be ported into this package, not extended inside `scraper/`.
+- When you port a feature, delete the corresponding code in `scraper/` after confirming no remaining consumers.
+
+See the package-internal migration docs **and** root `PACKAGES_MIGRATION_PLAN.md`.
+
+---
+
+## Known issues
+
+- Some advanced features in `MIGRATION_STATUS.md` not yet ported.
+- `scrape_many_stream` was added late; consumers may still be using buffered `scrape_many`. Prefer the streaming path.
+- This package's code itself has no known aidream-coupling violations.

matrx_scraper-0.1.0/Dockerfile

@@ -0,0 +1,50 @@
+# Standalone scraper microservice.
+# Build context: monorepo root (so we can access pyproject.toml + uv.lock + sibling workspace packages).
+# Coolify config: base_directory=/, dockerfile_location=/packages/matrx-scraper/Dockerfile.
+
+FROM python:3.13-slim AS base
+
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1 \
+    UV_LINK_MODE=copy \
+    UV_COMPILE_BYTECODE=1
+
+# System deps: curl for healthcheck. Playwright browser deps are installed
+# later by `playwright install --with-deps`.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        curl \
+        ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip install --no-cache-dir uv
+
+WORKDIR /app
+
+# uv sync needs the workspace root pyproject.toml + uv.lock and ALL workspace
+# members listed in [tool.uv.workspace] members. We copy the whole packages/
+# directory (~10 MB) — partial copies break workspace resolution.
+COPY pyproject.toml uv.lock ./
+COPY packages/ ./packages/
+
+# Install matrx-scraper plus its [server] extras, using the frozen uv.lock
+# from the workspace root. `--package matrx-scraper` focuses sync on this
+# workspace member only — the heavy aidream-current root project is NOT
+# installed. --no-dev skips dev-only tooling.
+RUN uv sync --frozen --no-dev --package matrx-scraper --extra server
+
+# Playwright Chromium + its OS deps. Largest layer; isolated for caching.
+RUN uv run --no-sync playwright install --with-deps chromium
+
+# Default to 8001 to match the existing Coolify Traefik labels and avoid
+# proxy reconfiguration. ServerConfig.from_env() respects PORT.
+ENV PORT=8001 \
+    HOST=0.0.0.0
+
+EXPOSE 8001
+
+# /health/ready confirms DB pool + cache are wired before traffic flows.
+HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
+    CMD curl -fsS http://localhost:8001/health/ready >/dev/null || exit 1
+
+CMD ["uv", "run", "--no-sync", "python", "-m", "matrx_scraper.server"]

matrx_scraper-0.1.0/GAPS_TO_FIX.md

@@ -0,0 +1,220 @@
+# Gaps to Fix Before Switching the Router to `matrx-scraper`
+
+This document lists only the things we **lose** by switching the FastAPI router
+(`aidream/api/routers/scraper.py`) from the old `scraper/services_v2/service.py`
+to the new `matrx_scraper` package. It is a to-do list for the new package, not
+a comparison of the two systems.
+
+Improvements in the new system are not listed here — they are already done and we
+keep them. Frontend breaking changes are flagged so the React team can be notified.
+
+---
+
+## 1. Streaming — `scrape_many()` Does Not Stream
+
+**Severity: BLOCKER**
+
+The old system yields each result to the frontend *as it finishes*, so the user
+sees pages arrive one by one. The new `scrape_many()` uses `asyncio.gather` and
+returns a batch only after every URL in the list completes.
+
+**What needs to be added to `orchestrator.py`:**
+
+An async-generator companion:
+
+```python
+async def scrape_many_stream(
+    urls: List[str],
+    use_proxy: bool = True,
+    concurrency: int = 5,
+) -> AsyncGenerator[ScrapeResult, None]:
+    semaphore = asyncio.Semaphore(concurrency)
+
+    async def _bounded(url: str) -> ScrapeResult:
+        async with semaphore:
+            return await scrape(url, use_proxy=use_proxy)
+
+    for coro in asyncio.as_completed([_bounded(u) for u in urls]):
+        yield await coro
+```
+
+The `ScrapeService` adapter (Priority 2 in `MIGRATION_STATUS.md`) must use this
+generator and call `emitter.send_data()` once per result, identical to how
+`quick_scrape_stream()` works today.
+
+---
+
+## 2. Frontend-Breaking Output Schema Changes
+
+The table below lists every field difference that will cause the frontend to break
+or silently lose data. Fields the frontend never uses (new additions) are not listed.
+
+| Old field | Status in new `ScrapeResult` | Action needed |
+|---|---|---|
+| `status` (`"success"` / `"error"` string) | Renamed to `success` (bool) | **React team must update**: change all `result.status === "success"` checks to `result.success === true` |
+| `error` (error string or null) | Renamed to `failure_reason` | **React team must update**: rename `result.error` → `result.failure_reason` |
+| `scraped_at` (ISO datetime string) | **Missing** | Add `scraped_at: str` to `ScrapeResult` and set it to `datetime.now(timezone.utc).isoformat()` inside `_build_result_from_response()` |
+| `hashes` (list of strings) | Present, but now a `dict` with keys `minhash`, `simhash`, `outline_simhash` | **React team must update** hashes display. Old frontend typed this as `string[]` — it is now `{ minhash: number[], simhash: number, outline_simhash: number }`. This is an improvement; just needs a frontend update. |
+| `overview.char_count_formatted` | **Missing from overview dict** | Add `char_count_formatted` to the overview dict built inside `element_extractor.py`. This was the raw text length (with formatting markers). |
+| `overview.page_title` | Moved to top-level `title` field | The `overview` dict still has `page_title` via the pipeline, so this should already be present inside `overview`. Confirm `overview["page_title"]` is still populated. If so, **no action needed** — the old frontend read from `overview.page_title`, not the flat `title`. |
+| `overview.outline` | Moved to top-level `document_outline` | Same as above — confirm `overview["outline"]` is still populated by the pipeline. The flat `document_outline` is additional, not a replacement. |
+| `metadata` (was inside `overview`) | Moved to top-level `metadata` | Old frontend read from `overview.metadata`. The `overview` dict in the new pipeline still carries `metadata` inside it (via `element_extractor`). The new flat `result.metadata` is an extra top-level copy. **No action needed** as long as `overview["metadata"]` is still present. Verify this. |
+
+### Envelope shape
+
+The service adapter must still emit the exact same SSE envelope the frontend expects:
+
+```json
+{
+  "response_type": "fetch_results",
+  "metadata": { "execution_time_ms": 123.4 },
+  "results": [ ...per-page results... ]
+}
+```
+
+`scrape_many()` returns a bare `List[ScrapeResult]` with no envelope. The adapter
+layer is responsible for wrapping it. This is not a gap in the package itself — it
+belongs in the `ScrapeService` adapter noted in `MIGRATION_STATUS.md`.
+
+---
+
+## 3. Per-Field Output Selection (Boolean Flags)
+
+**Severity: MEDIUM — no crash, but wastes bandwidth**
+
+The old system respects boolean flags from the request (`get_overview`,
+`get_structured_data`, `get_organized_data`, `get_links`, `get_text_data`,
+`get_main_image`, `get_content_filter_removal_details`) and omits fields that are
+`False`. The new `ScrapeResult.to_dict()` emits every non-None field unconditionally.
+
+The `ScrapeService` adapter should apply the flags as a post-processing filter:
+
+```python
+def apply_field_flags(result: dict, options: ScrapeOptionsBase) -> dict:
+    if not options.get_overview:
+        result.pop("overview", None)
+    if not options.get_organized_data:
+        result.pop("organized_data", None)
+    # ... etc.
+    return result
+```
+
+This does not need to be in the `matrx_scraper` package — it belongs in the adapter.
+
+---
+
+## 4. Per-Domain Proxy / Permission Gating
+
+**Severity: LOW for now — the feature existed but was rarely triggered**
+
+The old `QuickScrapeManager` checked three DB tables before scraping each URL:
+
+1. `scrape_domain.scrape_allowed` — global on/off per domain
+2. `scrape_domain_quick_scrape_settings.enabled` — quick-scrape-specific on/off
+3. `scrape_domain_quick_scrape_settings.proxy_type` — which proxy tier to use
+
+The new `scrape()` accepts only `use_proxy: bool` and `request_type: RequestType`.
+There is no DB lookup; it's a binary on/off.
+
+**What to build (when needed):** A lightweight domain-config resolver that reads
+those same DB tables and translates them into a `use_proxy` bool and
+`request_type` value before calling `scrape()`. This can live in the adapter layer,
+not in the package itself.
+
+**For now:** Default to `use_proxy=True` in the adapter. This matches the old
+default behavior for domains with no explicit settings.
+
+---
+
+## 5. `scraped_at` Field Missing from `ScrapeResult`
+
+This is also called out in item 2 above, but isolated here for clarity because it
+requires a code change inside the package.
+
+**File:** `packages/matrx-scraper/matrx_scraper/orchestrator.py`
+
+**Fix:** Add `scraped_at: Optional[str] = None` to `ScrapeResult` and set it in
+`_build_result_from_response()`:
+
+```python
+from datetime import datetime, timezone
+
+# inside _build_result_from_response(), after building `result`:
+result.scraped_at = datetime.now(timezone.utc).isoformat()
+```
+
+---
+
+## Summary Checklist
+
+```
+[x] Add scrape_many_stream() async generator to orchestrator.py        (DONE)
+[x] Add scraped_at field to ScrapeResult                               (DONE)
+[x] Confirm overview["page_title"] still populated in pipeline output  (CONFIRMED — core.py line ~165)
+[x] Confirm overview["outline"] still populated in pipeline output     (CONFIRMED — core.py line ~170)
+[x] Confirm overview["metadata"] still populated in pipeline output    (CONFIRMED — core.py line ~163)
+[x] Add char_count_formatted back to overview dict                     (DONE — core.py)
+[x] Build ScrapeService adapter (matrx_scraper/service.py)             (DONE)
+[x]   ↳ adapter wraps scrape_many_stream(), emits per-result send_data()
+[x]   ↳ adapter builds fetch_results envelope with execution_time_ms
+[x]   ↳ adapter applies boolean field-flag filtering post-scrape
+[x]   ↳ adapter defaults proxy=True (DB lookup deferred — low priority)
+[x] Router (aidream/api/routers/scraper.py) wired to matrx_scraper     (DONE)
+[ ] Notify React team:
+[ ]   ↳ result.status ("success"/"error") → result.success (bool)
+[ ]   ↳ result.error (string) → result.failure_reason (string)
+[ ]   ↳ result.hashes type changed: string[] → { minhash, simhash, outline_simhash }
+```
+
+---
+
+## 5. Domain-Config SQL Has No Canonical Schema
+
+**Severity: LOW (no crash, but silently broken)**
+
+`PostgresDomainConfigStore._load_all_domains()` ships hand-written SQL against
+three tables — `scrape_domain`, `scrape_domain_settings`, `scrape_path_pattern`,
+`scrape_path_override` — but the package does not own any DDL for those tables.
+The column names are inferred from whatever the host's migrations happened to
+create. We hit this in the wild on 2026-05-01: the package SQL referenced
+`scrape_path_pattern.domain_id` and `scrape_path_pattern.pattern`, while the
+actual columns are `scrape_domain_id` and `path_pattern`. The `_refresh()`
+exception was caught and logged, so domain configs silently fell back to empty
+and proxy/permission rules never engaged — this is the exact silent-degradation
+mode the platform principle ("aggressively detected, not just patched") tells
+us to eliminate.
+
+**Two complementary platform fixes:**
+
+1. **Ship the DDL.** Add `matrx_scraper/sql/domain_config.sql` (or a tiny
+   migration set) that creates the four tables with the column names the
+   package's queries actually use. A standalone consumer runs it once; aidream
+   is a pre-existing host whose migrations already created compatible tables.
+2. **Pre-flight schema check.** On `PostgresDomainConfigStore.start()`, run a
+   `LIMIT 0` of each query inside a single connection. If any column is
+   missing, raise `RuntimeError` with the exact column-vs-table that doesn't
+   match — fail loudly at startup instead of silently caching `{}` for the
+   life of the process.
+
+The hot-fix on 2026-05-01 just renamed the columns in-query (using SQL
+aliases so the Python `row[...]` access is unchanged). That removes the
+immediate failure but leaves the class of failure for the next drift.
+
+---
+
+## ✅ MCP Tools & Research — Migrated (done separately)
+
+All active callers of the old `scraper_enhanced` parser have been switched to
+`matrx_scraper`. The `ai_research_with_images` field and all concurrent patterns
+are unchanged — only the import source changed.
+
+| File | Change |
+|---|---|
+| `scraper/scraper_enhanced/features/read_page.py` | `scraper_enhanced.{scraper,parser.parser,content_extractors}` → `matrx_scraper.{scraper,parser.core,extractors}` |
+| `scraper/scraper_enhanced/features/mcp_tool_helpers.py` | same |
+| `scraper/scraper_enhanced/features/top_n_brave_results.py` | same |
+| `research/multisource.py` | `scraper_enhanced.parser.parser.parse_html` → `matrx_scraper.parser.core.parse_html` |
+| `research/scraper.py` | already on `matrx_scraper` — no change needed |
+
+See `LEGACY_AUDIT.md` for the full inventory of remaining old-scraper references
+(deprecated Socket.IO stack + dead scripts).