scrapemend 0.0.1__tar.gz

scrapemend-0.0.1/.demo-cache.json

@@ -0,0 +1,7 @@
+{
+  "product@shop.demo": {
+    "title": "div.product__name",
+    "price": "div.product__price",
+    "stock": "span.product__availability"
+  }
+}

scrapemend-0.0.1/.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: pip install -e ".[dev,openai]"
+      - name: Lint
+        run: ruff check .
+      - name: Test
+        run: pytest -q

scrapemend-0.0.1/.gitignore

@@ -0,0 +1,26 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+
+# Tooling
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+# scrapemend runtime artifacts
+.scrapemend-cache.json
+
+# Secrets — never commit API keys
+.env
+*.key
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/

scrapemend-0.0.1/LAUNCH.md

@@ -0,0 +1,65 @@
+# scrapemend 冷启动手册 —— 「别人怎么看到你」的落地版
+
+这份文档只回答一个问题：**一个没名气、没粉丝、不发视频的 17 岁开发者，怎么让全球开发者看到并用上这个库。** 全部动作都是文字驱动，不需要剪任何视频。
+
+---
+
+## 核心逻辑
+
+开源工具的传播靠两股力量，都不是视频：
+
+1. **搜索**：开发者遇到问题 → 搜 Google / PyPI / GitHub → 找到你。
+2. **社区分享**：开发者在论坛/聚合站看到你 → 点 star → 帮你转。
+
+你要做的就是把这两条路铺好，然后让项目自己滚。
+
+---
+
+## 第一阶段：让项目「可被搜到」（发布前必须做完）
+
+- [ ] **README 里堆对关键词**（不是堆砌，是自然出现）：`web scraping`、`self-healing`、
+      `scraper breaks`、`LLM`、`AI`、`CSS selector`、`site redesign`。开发者就是用这些词搜的。
+- [ ] **一句话定位清晰**：首屏第一行必须让人 3 秒看懂「这是干嘛的、解决什么痛」。
+      现在的 "Self-healing web scraping for Python" 就是干这个的。
+- [ ] **一个能跑的 Quickstart**：别人复制粘贴 10 行就能看到效果，是留住人的关键。
+- [ ] **一张流程图 / 一个对比**（纯文字 ASCII 也行，已在 README 里）：说明你和普通爬虫的差别。
+- [ ] 发布到 **PyPI**：`pip install scrapemend` 能装，是可信度的门槛。
+
+## 第二阶段：第一批 star 从哪来（发布当天~第一周）
+
+发布不是「push 上去就完事」，而是**主动把它端到人面前**。按优先级：
+
+- [ ] **Reddit**：`r/Python`（周四有 "What's everyone working on" 帖）、`r/webscraping`
+      （这里的人天天被选择器失效折磨，是你最精准的受众）。发帖讲**痛点故事**，不要硬广：
+      「我受够了每次网站改版就重写爬虫，于是做了个选择器失效时用 LLM 自愈的库」。
+- [ ] **Hacker News**（Show HN）：标题写 `Show HN: scrapemend – self-healing web scraping that repairs itself when sites change`。
+      发帖时间选美西早上。第一条评论自己补充技术细节和 roadmap。
+- [ ] **V2EX `分享创造` 节点**（中文开发者，你母语主场，先练手拿第一波反馈）。
+- [ ] **dev.to / 掘金 写一篇文章**：标题如「为什么你的爬虫总在网站改版后崩，以及怎么让它自愈」。
+      文章本身长期吃搜索流量，文末自然引到项目。这是你最擅长的文字载体。
+- [ ] **Lobsters**（技术人密度高，质量反馈多）。
+
+**发帖纪律**：讲痛点、讲故事、讲取舍，别吹。开发者极度反感营销腔，但很吃「我也被这个坑过」的共鸣。
+
+## 第三阶段：让它滚起来（第一个月）
+
+- [ ] 认真回每一个 issue 和评论，24 小时内响应。早期每个用户都是火种。
+- [ ] 把真实用户的问题变成 README 的 FAQ / 新 example。
+- [ ] 做一个 OpenAI adapter 的**真实网站 demo**（gif 或纯文字输出都行），证明「真能自愈」。
+- [ ] 每解决一个真实痛点，发一条简短更新到原帖/社区。
+- [ ] star 过 100 后，在 README 顶部加 badge，信任感滚雪球。
+
+## 变现铺垫（有流量之后，别急）
+
+- 开源核心永远免费（这是流量来源，别动）。
+- 当有人问「能不能批量 / 团队共享缓存 / 私有部署支持」——那就是付费信号。
+- 收款走 **GitHub Sponsors / Polar / Lemon Squeezy**（支持个人、全球、信用卡，
+  比对公/PayPal 商用更适合你当前处境；具体注册仍需家人协助实名）。
+
+---
+
+## 一句话总结
+
+**你不用让世界来找你，你把项目端到「正在被爬虫折磨的开发者」面前。**
+Reddit 的 r/webscraping、HN 的 Show HN、一篇讲痛点的文章——这三样就是你的「被看到」，
+一秒视频都不用剪。

scrapemend-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 OriginTown
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scrapemend-0.0.1/PKG-INFO

@@ -0,0 +1,118 @@
+Metadata-Version: 2.5
+Name: scrapemend
+Version: 0.0.1
+Summary: Self-healing web scraping for Python. Describe what you want in plain language; when a site changes and your selectors break, an LLM repairs them automatically.
+Project-URL: Homepage, https://github.com/FengYue427/scrapemend
+Project-URL: Repository, https://github.com/FengYue427/scrapemend
+Project-URL: Issues, https://github.com/FengYue427/scrapemend/issues
+Author: FengYue427
+License: MIT
+License-File: LICENSE
+Keywords: ai,crawler,data-extraction,llm,scraping,self-healing,web-scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.27
+Requires-Dist: selectolax>=0.3.21
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: openai
+Requires-Dist: openai>=1.40; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# scrapemend
+
+**Self-healing web scraping for Python.** Describe the data you want once.
+scrapemend extracts it with cached CSS selectors — and when a site changes
+its markup and those selectors break, a language model inspects the new page,
+repairs the selectors, and caches the fix. The next run is free again.
+
+> Traditional scrapers break every time a site is redesigned. scrapemend
+> treats a broken selector as a cache miss, not a crash.
+
+---
+
+## Why
+
+Every scraper hits the same wall: you write selectors, the site ships a
+redesign, your selectors silently return nothing, and you go rewrite them by
+hand. Cloud "AI scraping" services solve the robustness problem but are
+expensive, send your target data through someone else's servers, and run an
+LLM call on *every* request.
+
+scrapemend takes the middle path:
+
+- **Cache first.** Known selectors run with zero LLM cost — it's just parsing.
+- **Heal on miss.** The LLM is only called when selectors stop matching.
+- **Self-hosted.** It's a library, not a service. Your keys, your data.
+
+## Install
+
+```bash
+pip install scrapemend
+```
+
+## Quickstart
+
+```python
+from scrapemend import Scraper, Schema
+
+schema = (
+    Schema("product")
+    .add("title", "the product name")
+    .add("price", "the product price with currency")
+)
+
+scraper = Scraper()  # offline stub by default; plug in a real LLM to heal
+result = scraper.extract("https://shop.example/p/1", schema)
+
+print(result["title"], result["price"])
+print("healed:", result.healed, "tokens:", result.tokens)
+```
+
+You describe *what* you want in plain language. You never hand-write a CSS
+selector — scrapemend discovers them, and re-discovers them when they break.
+
+## How it works
+
+```
+extract(url, schema)
+      │
+      ├─ cached selectors exist? ──► apply them
+      │                               ├─ all fields matched ──► return (free)
+      │                               └─ some field missed ──┐
+      │                                                      ▼
+      └─ no cache ─────────────────────────────────► LLM heals selectors
+                                                      apply, cache, return
+```
+
+Because healing only runs on a miss, steady-state scraping costs nothing in
+tokens. You pay the model exactly when a site actually changes.
+
+## Status
+
+Early alpha (`0.0.1`). The core pipeline — cache, parse, heal, re-cache — works
+and is covered by tests that run fully offline. A real OpenAI adapter is the
+next step; the LLM layer is provider-agnostic by design.
+
+## Roadmap
+
+- [x] Cache-first / heal-on-miss pipeline
+- [x] Offline, key-free test path
+- [ ] OpenAI adapter
+- [ ] Local-model adapter
+- [x] HTML condensing for cheaper prompts
+- [ ] Pluggable cost/budget guards
+
+## License
+
+MIT

scrapemend-0.0.1/README.md

@@ -0,0 +1,87 @@
+# scrapemend
+
+**Self-healing web scraping for Python.** Describe the data you want once.
+scrapemend extracts it with cached CSS selectors — and when a site changes
+its markup and those selectors break, a language model inspects the new page,
+repairs the selectors, and caches the fix. The next run is free again.
+
+> Traditional scrapers break every time a site is redesigned. scrapemend
+> treats a broken selector as a cache miss, not a crash.
+
+---
+
+## Why
+
+Every scraper hits the same wall: you write selectors, the site ships a
+redesign, your selectors silently return nothing, and you go rewrite them by
+hand. Cloud "AI scraping" services solve the robustness problem but are
+expensive, send your target data through someone else's servers, and run an
+LLM call on *every* request.
+
+scrapemend takes the middle path:
+
+- **Cache first.** Known selectors run with zero LLM cost — it's just parsing.
+- **Heal on miss.** The LLM is only called when selectors stop matching.
+- **Self-hosted.** It's a library, not a service. Your keys, your data.
+
+## Install
+
+```bash
+pip install scrapemend
+```
+
+## Quickstart
+
+```python
+from scrapemend import Scraper, Schema
+
+schema = (
+    Schema("product")
+    .add("title", "the product name")
+    .add("price", "the product price with currency")
+)
+
+scraper = Scraper()  # offline stub by default; plug in a real LLM to heal
+result = scraper.extract("https://shop.example/p/1", schema)
+
+print(result["title"], result["price"])
+print("healed:", result.healed, "tokens:", result.tokens)
+```
+
+You describe *what* you want in plain language. You never hand-write a CSS
+selector — scrapemend discovers them, and re-discovers them when they break.
+
+## How it works
+
+```
+extract(url, schema)
+      │
+      ├─ cached selectors exist? ──► apply them
+      │                               ├─ all fields matched ──► return (free)
+      │                               └─ some field missed ──┐
+      │                                                      ▼
+      └─ no cache ─────────────────────────────────► LLM heals selectors
+                                                      apply, cache, return
+```
+
+Because healing only runs on a miss, steady-state scraping costs nothing in
+tokens. You pay the model exactly when a site actually changes.
+
+## Status
+
+Early alpha (`0.0.1`). The core pipeline — cache, parse, heal, re-cache — works
+and is covered by tests that run fully offline. A real OpenAI adapter is the
+next step; the LLM layer is provider-agnostic by design.
+
+## Roadmap
+
+- [x] Cache-first / heal-on-miss pipeline
+- [x] Offline, key-free test path
+- [ ] OpenAI adapter
+- [ ] Local-model adapter
+- [x] HTML condensing for cheaper prompts
+- [ ] Pluggable cost/budget guards
+
+## License
+
+MIT

scrapemend-0.0.1/examples/self_heal_demo.py

@@ -0,0 +1,108 @@
+"""Live self-healing demo using DeepSeek.
+
+Run it to watch scrapemend survive a site "redesign": the first page
+teaches it selectors, a cached run costs nothing, then the markup changes
+and the LLM repairs the selectors automatically.
+
+Setup:
+    pip install "scrapemend[openai]"
+    set DEEPSEEK_API_KEY=sk-...        # PowerShell: $env:DEEPSEEK_API_KEY="sk-..."
+    python examples/self_heal_demo.py
+
+No key? The script falls back to an offline stub so it still runs end to
+end (with canned selectors) and you can see the flow without spending a cent.
+"""
+
+from __future__ import annotations
+
+import os
+
+from scrapemend import Schema, Scraper
+from scrapemend.cache import SelectorCache
+from scrapemend.llm import EchoClient
+
+URL = "https://shop.demo/product/42"
+
+# Version 1 of the page.
+PAGE_V1 = """
+<html><body>
+  <h1 class="title">Mechanical Keyboard</h1>
+  <span class="price">$89.00</span>
+  <div class="stock">In stock</div>
+</body></html>
+"""
+
+# Same product, redesigned markup — every selector from V1 now misses.
+PAGE_V2 = """
+<html><body>
+  <div class="product__name">Mechanical Keyboard</div>
+  <div class="product__price">$89.00</div>
+  <span class="product__availability">In stock</span>
+</body></html>
+"""
+
+
+def build_client():
+    """Prefer real DeepSeek healing; fall back to an offline stub."""
+    if os.environ.get("DEEPSEEK_API_KEY"):
+        from scrapemend.providers.deepseek_client import DeepSeekClient
+
+        print(">> Using DeepSeek for live healing.\n")
+        return DeepSeekClient()
+
+    print(">> No DEEPSEEK_API_KEY found — running offline with a stub.\n")
+    # The stub returns whatever we set as .reply; the demo updates it to
+    # mimic what a model would return for each page version.
+    return EchoClient()
+
+
+def main() -> None:
+    schema = (
+        Schema("product")
+        .add("title", "the product name / title")
+        .add("price", "the product price including the currency symbol")
+        .add("stock", "the availability or stock status text")
+    )
+
+    client = build_client()
+    offline = isinstance(client, EchoClient)
+    scraper = Scraper(client=client, cache=SelectorCache(".demo-cache.json"))
+
+    print("=" * 60)
+    print("Run 1 — first ever visit (no cache). Expect: HEAL.")
+    if offline:
+        client.reply = '{"title":"h1.title","price":"span.price","stock":"div.stock"}'
+    r1 = scraper.extract(URL, schema, html=PAGE_V1)
+    _report(r1)
+
+    print("=" * 60)
+    print("Run 2 — same markup again. Expect: CACHE HIT, 0 tokens.")
+    r2 = scraper.extract(URL, schema, html=PAGE_V1)
+    _report(r2)
+
+    print("=" * 60)
+    print("Run 3 — the site got redesigned (new markup). Expect: RE-HEAL.")
+    if offline:
+        client.reply = (
+            '{"title":"div.product__name","price":"div.product__price",'
+            '"stock":"span.product__availability"}'
+        )
+    r3 = scraper.extract(URL, schema, html=PAGE_V2)
+    _report(r3)
+
+    print("=" * 60)
+    print("Done. Same Schema, two different layouts, zero hand-written selectors.")
+
+
+def _report(result) -> None:
+    status = "CACHE HIT" if result.from_cache else ("HEALED" if result.healed else "parsed")
+    print(f"  status : {status}")
+    print(f"  tokens : {result.tokens}")
+    print(f"  data   : {result.data}")
+    if result.notes:
+        print(f"  notes  : {result.notes}")
+    print()
+
+
+if __name__ == "__main__":
+    main()

scrapemend-0.0.1/pyproject.toml

@@ -0,0 +1,53 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "scrapemend"
+version = "0.0.1"
+description = "Self-healing web scraping for Python. Describe what you want in plain language; when a site changes and your selectors break, an LLM repairs them automatically."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "FengYue427" }]
+keywords = [
+    "scraping",
+    "web-scraping",
+    "crawler",
+    "llm",
+    "ai",
+    "self-healing",
+    "data-extraction",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "httpx>=0.27",
+    "selectolax>=0.3.21",
+]
+
+[project.optional-dependencies]
+openai = ["openai>=1.40"]
+dev = ["pytest>=8", "ruff>=0.6"]
+
+[project.urls]
+Homepage = "https://github.com/FengYue427/scrapemend"
+Repository = "https://github.com/FengYue427/scrapemend"
+Issues = "https://github.com/FengYue427/scrapemend/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/scrapemend"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"

scrapemend-0.0.1/src/scrapemend/__init__.py

@@ -0,0 +1,14 @@
+"""scrapemend — self-healing web scraping for Python.
+
+Describe the data you want once. scrapemend extracts it with cached
+selectors; when a site changes its markup and those selectors stop
+matching, a language model inspects the new page and repairs them
+automatically, then caches the fix so the next run is free again.
+"""
+
+from scrapemend.result import ExtractionResult
+from scrapemend.schema import Field, Schema
+from scrapemend.scraper import Scraper
+
+__all__ = ["ExtractionResult", "Field", "Schema", "Scraper"]
+__version__ = "0.0.1"

scrapemend-0.0.1/src/scrapemend/cache.py

@@ -0,0 +1,57 @@
+"""On-disk selector cache keyed by (schema, site).
+
+The cache is what makes the common path free: once selectors are known
+for a given schema and host, extraction is pure parsing with no LLM call.
+Healing only writes here when the cached selectors stop matching.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from urllib.parse import urlparse
+
+
+class SelectorCache:
+    """Maps a schema+host to the CSS selectors last known to work.
+
+    Stored as a single JSON file so it is trivial to inspect, diff, and
+    commit to a repo if a team wants to share healed selectors. (Shared,
+    hosted cache sync is a candidate Pro feature later.)
+    """
+
+    def __init__(self, path: str | Path = ".scrapemend-cache.json") -> None:
+        self.path = Path(path)
+        self._data: dict[str, dict[str, str]] = {}
+        self._load()
+
+    def _load(self) -> None:
+        if self.path.exists():
+            try:
+                self._data = json.loads(self.path.read_text(encoding="utf-8"))
+            except (json.JSONDecodeError, OSError):
+                self._data = {}
+
+    def _flush(self) -> None:
+        self.path.write_text(
+            json.dumps(self._data, indent=2, ensure_ascii=False), encoding="utf-8"
+        )
+
+    @staticmethod
+    def _key(schema_name: str, url: str) -> str:
+        host = urlparse(url).netloc or "unknown"
+        return f"{schema_name}@{host}"
+
+    def get(self, schema_name: str, url: str) -> dict[str, str] | None:
+        """Return cached {field_name: css_selector} or None if unknown."""
+        return self._data.get(self._key(schema_name, url))
+
+    def put(self, schema_name: str, url: str, selectors: dict[str, str]) -> None:
+        """Store healed selectors for this schema+host and persist."""
+        self._data[self._key(schema_name, url)] = selectors
+        self._flush()
+
+    def drop(self, schema_name: str, url: str) -> None:
+        """Forget selectors for a schema+host (forces a re-heal next run)."""
+        self._data.pop(self._key(schema_name, url), None)
+        self._flush()

scrapemend-0.0.1/src/scrapemend/condense.py

@@ -0,0 +1,60 @@
+"""Shrink HTML before it goes into a healing prompt.
+
+The model only needs the page's *structure* to pick CSS selectors, not
+its scripts, styles, or inline data payloads. Stripping those cuts prompt
+tokens sharply (cheaper healing) and removes noise that can mislead the
+model (more accurate selectors). This directly serves the library's
+"robust but cheap" promise.
+"""
+
+from __future__ import annotations
+
+import re
+
+from selectolax.lexbor import LexborHTMLParser
+
+#: Tags that carry no structural signal for selector discovery.
+_DROP_TAGS = ("script", "style", "noscript", "template", "svg", "iframe")
+
+#: The only attributes that matter for building CSS selectors.
+_KEEP_ATTRS = ("class", "id")
+
+_WS = re.compile(r"\s+")
+
+
+def condense(html: str, limit: int = 6000) -> str:
+    """Return a stripped-down version of ``html`` for prompting.
+
+    Removes non-structural tags, drops every attribute except class/id,
+    collapses whitespace, and finally caps length as a hard token guard.
+
+    Falls back to a plain truncation if parsing fails for any reason, so
+    healing never breaks just because a page is malformed.
+    """
+    try:
+        tree = LexborHTMLParser(html)
+    except Exception:  # noqa: BLE001  # pragma: no cover - last-resort guard; never let healing crash on a weird page
+        return _WS.sub(" ", html)[:limit]
+
+    for tag in _DROP_TAGS:
+        for node in tree.css(tag):
+            node.decompose()
+
+    body = tree.body if tree.body is not None else tree.root
+    if body is None:
+        return _WS.sub(" ", html)[:limit]
+
+    for node in body.traverse(include_text=False):
+        attrs = getattr(node, "attributes", None)
+        if not attrs:
+            continue
+        for name in list(attrs.keys()):
+            if name not in _KEEP_ATTRS:
+                try:
+                    del node.attrs[name]
+                except (KeyError, TypeError):  # pragma: no cover - defensive
+                    pass
+
+    cleaned = body.html or ""
+    cleaned = _WS.sub(" ", cleaned).strip()
+    return cleaned[:limit]

scrapemend-0.0.1/src/scrapemend/healer.py

@@ -0,0 +1,49 @@
+"""Selector discovery/repair via the LLM.
+
+Given a page and a schema, ask the model for one CSS selector per field.
+The healer is only invoked when cached selectors are missing or have
+stopped matching, so its token cost is paid rarely, not per request.
+"""
+
+from __future__ import annotations
+
+import json
+
+from scrapemend.condense import condense
+from scrapemend.llm import LLMClient
+from scrapemend.schema import Schema
+
+_SYSTEM = (
+    "You map data fields to CSS selectors for a web page. "
+    "Return ONLY a JSON object mapping each field name to a single CSS "
+    "selector string that selects the element holding that field's value. "
+    "No prose, no code fences."
+)
+
+
+def build_prompt(schema: Schema, html: str) -> str:
+    lines = [f"- {f.name}: {f.describe}" for f in schema.fields]
+    return (
+        "Fields to locate:\n"
+        + "\n".join(lines)
+        + "\n\nPage HTML (condensed, possibly truncated):\n"
+        + condense(html)
+    )
+
+
+def heal(schema: Schema, html: str, client: LLMClient) -> dict[str, str]:
+    """Ask the LLM for fresh selectors. Returns {field_name: selector}.
+
+    Unknown or malformed replies yield an empty mapping rather than
+    raising, so the caller can record a clean failure and move on.
+    """
+    reply = client.complete(_SYSTEM, build_prompt(schema, html))
+    try:
+        parsed = json.loads(reply)
+    except json.JSONDecodeError:
+        return {}
+    if not isinstance(parsed, dict):
+        return {}
+    # Keep only known fields with string selectors.
+    valid = set(schema.field_names())
+    return {k: v for k, v in parsed.items() if k in valid and isinstance(v, str)}

scrapemend-0.0.1/src/scrapemend/llm.py

@@ -0,0 +1,60 @@
+"""The shared LLM layer.
+
+This module is deliberately generic: a provider-agnostic client plus a
+token/cost ledger. scrapemend uses it only for selector healing, but it
+carries no scraping concepts, so the planned LLM-developer-tooling product
+can reuse it wholesale. Keep it free of any scraping imports.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Protocol, runtime_checkable
+
+
+@dataclass
+class Usage:
+    """Running tally of tokens spent, so callers can report real cost."""
+
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+
+    @property
+    def total(self) -> int:
+        return self.prompt_tokens + self.completion_tokens
+
+    def add(self, prompt: int, completion: int) -> None:
+        self.prompt_tokens += prompt
+        self.completion_tokens += completion
+
+
+@runtime_checkable
+class LLMClient(Protocol):
+    """Minimal contract any provider adapter must satisfy.
+
+    Kept tiny on purpose: one text-in/text-out call. Concrete adapters
+    (OpenAI, local models, etc.) live in submodules and record usage.
+    """
+
+    usage: Usage
+
+    def complete(self, system: str, user: str) -> str:
+        """Return the model's text response to a system+user prompt."""
+        ...
+
+
+@dataclass
+class EchoClient:
+    """A no-network stub used in tests and offline development.
+
+    It never contacts a provider; it just returns a fixed string. This
+    lets the whole heal/extract pipeline run in CI without API keys.
+    """
+
+    reply: str = "{}"
+    usage: Usage = field(default_factory=Usage)
+
+    def complete(self, system: str, user: str) -> str:
+        # Pretend we spent a few tokens so cost tracking has something to show.
+        self.usage.add(prompt=len(user) // 4, completion=len(self.reply) // 4)
+        return self.reply

scrapemend-0.0.1/src/scrapemend/providers/__init__.py

@@ -0,0 +1,6 @@
+"""Concrete LLM provider adapters.
+
+Each adapter implements the :class:`scrapemend.llm.LLMClient` protocol
+(a ``usage`` ledger plus a ``complete(system, user)`` call) and is kept
+import-light so the core library never hard-depends on any one provider.
+"""

scrapemend-0.0.1/src/scrapemend/providers/deepseek_client.py

@@ -0,0 +1,83 @@
+"""DeepSeek-backed adapter for selector healing.
+
+DeepSeek exposes an OpenAI-compatible API, so this adapter reuses the
+``openai`` SDK and only swaps the base URL, key source, and model name.
+It supports JSON output mode, which is exactly what the healer wants
+(a strict JSON object mapping fields to selectors).
+
+DeepSeek is a strong default here: it is reachable from mainland China
+without extra setup and is cheap enough that infrequent healing calls
+are effectively free. Install the SDK with:
+
+    pip install "scrapemend[openai]"
+
+and set DEEPSEEK_API_KEY in your environment.
+"""
+
+from __future__ import annotations
+
+import os
+
+from scrapemend.llm import Usage
+
+#: DeepSeek's OpenAI-compatible endpoint.
+DEEPSEEK_BASE_URL = "https://api.deepseek.com"
+
+#: Alias that always points at DeepSeek's current main chat model, so this
+#: adapter keeps working across model version bumps.
+DEEPSEEK_DEFAULT_MODEL = "deepseek-chat"
+
+
+class DeepSeekClient:
+    """Calls DeepSeek's chat API (OpenAI-compatible) and tracks usage.
+
+    Args:
+        model: Chat model name. Defaults to the ``deepseek-chat`` alias.
+        api_key: Overrides the ``DEEPSEEK_API_KEY`` environment variable.
+        base_url: Overrides the default DeepSeek endpoint (useful for
+            proxies or OpenAI-compatible gateways).
+        temperature: Kept low for deterministic, parseable selector JSON.
+    """
+
+    def __init__(
+        self,
+        model: str = DEEPSEEK_DEFAULT_MODEL,
+        api_key: str | None = None,
+        base_url: str = DEEPSEEK_BASE_URL,
+        temperature: float = 0.0,
+    ) -> None:
+        try:
+            from openai import OpenAI
+        except ImportError as exc:  # pragma: no cover - import guard
+            raise ImportError(
+                "The DeepSeek adapter uses the OpenAI SDK. "
+                'Install it with: pip install "scrapemend[openai]"'
+            ) from exc
+
+        key = api_key or os.environ.get("DEEPSEEK_API_KEY")
+        if not key:
+            raise ValueError(
+                "No API key. Pass api_key=... or set DEEPSEEK_API_KEY in the environment."
+            )
+
+        self._client = OpenAI(api_key=key, base_url=base_url)
+        self.model = model
+        self.temperature = temperature
+        self.usage = Usage()
+
+    def complete(self, system: str, user: str) -> str:
+        resp = self._client.chat.completions.create(
+            model=self.model,
+            temperature=self.temperature,
+            response_format={"type": "json_object"},
+            messages=[
+                {"role": "system", "content": system},
+                {"role": "user", "content": user},
+            ],
+        )
+        if resp.usage is not None:
+            self.usage.add(
+                prompt=resp.usage.prompt_tokens,
+                completion=resp.usage.completion_tokens,
+            )
+        return resp.choices[0].message.content or "{}"

scrapemend-0.0.1/src/scrapemend/providers/openai_client.py

@@ -0,0 +1,67 @@
+"""OpenAI-backed adapter for selector healing.
+
+Implements the LLMClient protocol. The ``openai`` package is imported
+lazily so that merely importing scrapemend never requires it — only
+constructing an :class:`OpenAIClient` does. Install with:
+
+    pip install "scrapemend[openai]"
+"""
+
+from __future__ import annotations
+
+import os
+
+from scrapemend.llm import Usage
+
+
+class OpenAIClient:
+    """Calls an OpenAI chat model and tracks token usage.
+
+    Args:
+        model: Chat model name. A small, cheap model is the sensible
+            default since healing is infrequent and the task is narrow.
+        api_key: Overrides the ``OPENAI_API_KEY`` environment variable.
+        temperature: Kept low for deterministic, parseable selector JSON.
+    """
+
+    def __init__(
+        self,
+        model: str = "gpt-4o-mini",
+        api_key: str | None = None,
+        temperature: float = 0.0,
+    ) -> None:
+        try:
+            from openai import OpenAI
+        except ImportError as exc:  # pragma: no cover - import guard
+            raise ImportError(
+                "The OpenAI adapter needs the 'openai' package. "
+                'Install it with: pip install "scrapemend[openai]"'
+            ) from exc
+
+        key = api_key or os.environ.get("OPENAI_API_KEY")
+        if not key:
+            raise ValueError(
+                "No API key. Pass api_key=... or set OPENAI_API_KEY in the environment."
+            )
+
+        self._client = OpenAI(api_key=key)
+        self.model = model
+        self.temperature = temperature
+        self.usage = Usage()
+
+    def complete(self, system: str, user: str) -> str:
+        resp = self._client.chat.completions.create(
+            model=self.model,
+            temperature=self.temperature,
+            response_format={"type": "json_object"},
+            messages=[
+                {"role": "system", "content": system},
+                {"role": "user", "content": user},
+            ],
+        )
+        if resp.usage is not None:
+            self.usage.add(
+                prompt=resp.usage.prompt_tokens,
+                completion=resp.usage.completion_tokens,
+            )
+        return resp.choices[0].message.content or "{}"

scrapemend-0.0.1/src/scrapemend/result.py

@@ -0,0 +1,37 @@
+"""Return type carrying extracted data plus a little provenance.
+
+Beyond the data itself, each extraction reports *how* it was produced:
+whether cached selectors worked or the LLM had to heal them, and the
+token cost of any healing. That provenance is what later powers cost
+tracking — the shared layer the planned LLM-tooling product reuses.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+
+@dataclass
+class ExtractionResult:
+    """Outcome of a single extraction.
+
+    Attributes:
+        data: Mapping of field name to extracted value(s).
+        healed: True when selectors had to be repaired by the LLM this run.
+        from_cache: True when every field was served from cached selectors.
+        tokens: Tokens spent on healing this run (0 when nothing healed).
+        notes: Human-readable diagnostics, e.g. which fields failed.
+    """
+
+    data: dict[str, Any] = field(default_factory=dict)
+    healed: bool = False
+    from_cache: bool = True
+    tokens: int = 0
+    notes: list[str] = field(default_factory=list)
+
+    def __getitem__(self, key: str) -> Any:
+        return self.data[key]
+
+    def get(self, key: str, default: Any = None) -> Any:
+        return self.data.get(key, default)

scrapemend-0.0.1/src/scrapemend/schema.py

@@ -0,0 +1,51 @@
+"""User-facing description of *what* to extract.
+
+A ``Schema`` is a plain, declarative description of the fields the user
+wants off a page. It carries no selectors — selectors are discovered and
+healed at runtime and live in the cache, not here. This keeps the thing
+the user writes stable even as target sites change their markup.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+
+@dataclass
+class Field:
+    """One piece of data to pull off a page.
+
+    Args:
+        name: Key the extracted value appears under in the result.
+        describe: Plain-language description used by the LLM to locate the
+            value when selectors need to be (re)discovered. Example:
+            "the product price including currency symbol".
+        many: If True, collect every match as a list instead of the first.
+        attr: Optional HTML attribute to read (e.g. "href"). When None, the
+            element's text content is used.
+    """
+
+    name: str
+    describe: str
+    many: bool = False
+    attr: str | None = None
+
+
+@dataclass
+class Schema:
+    """A named collection of :class:`Field` objects.
+
+    The ``name`` is used as the cache namespace so that selectors healed
+    for one schema/site pair do not collide with another.
+    """
+
+    name: str
+    fields: list[Field] = field(default_factory=list)
+
+    def add(self, name: str, describe: str, *, many: bool = False, attr: str | None = None) -> Schema:
+        """Add a field and return self so calls can be chained."""
+        self.fields.append(Field(name=name, describe=describe, many=many, attr=attr))
+        return self
+
+    def field_names(self) -> list[str]:
+        return [f.name for f in self.fields]

scrapemend-0.0.1/src/scrapemend/scraper.py

@@ -0,0 +1,99 @@
+"""The public entry point tying cache, parsing, and healing together.
+
+Flow per extraction:
+    1. Try cached selectors (free, fast).
+    2. If any required field fails to match, the page likely changed —
+       invoke the LLM healer, re-extract, and cache the repaired selectors.
+
+This "cache first, heal only on miss" policy is the whole point: you get
+AI robustness without paying an LLM call on every request.
+"""
+
+from __future__ import annotations
+
+import httpx
+from selectolax.lexbor import LexborHTMLParser as HTMLParser
+
+from scrapemend.cache import SelectorCache
+from scrapemend.healer import heal
+from scrapemend.llm import EchoClient, LLMClient
+from scrapemend.result import ExtractionResult
+from scrapemend.schema import Schema
+
+
+class Scraper:
+    """Extracts data described by a :class:`Schema` from web pages.
+
+    Args:
+        client: LLM adapter used for healing. Defaults to an offline stub
+            so the library imports and runs without any API key; pass a
+            real provider adapter to enable actual healing.
+        cache: Selector cache. Defaults to a JSON file in the cwd.
+    """
+
+    def __init__(self, client: LLMClient | None = None, cache: SelectorCache | None = None) -> None:
+        self.client = client or EchoClient()
+        self.cache = cache or SelectorCache()
+
+    def fetch(self, url: str) -> str:
+        resp = httpx.get(url, follow_redirects=True, timeout=30)
+        resp.raise_for_status()
+        return resp.text
+
+    def extract(self, url: str, schema: Schema, *, html: str | None = None) -> ExtractionResult:
+        """Extract ``schema`` from ``url`` (or supplied ``html``)."""
+        page = html if html is not None else self.fetch(url)
+        tree = HTMLParser(page)
+
+        selectors = self.cache.get(schema.name, url)
+        if selectors is not None:
+            result = self._apply(tree, schema, selectors)
+            if not result.notes:  # every field matched
+                result.from_cache = True
+                return result
+            # Cached selectors went stale — fall through to healing.
+
+        return self._heal_and_extract(url, schema, page, tree)
+
+    def _heal_and_extract(
+        self, url: str, schema: Schema, page: str, tree: HTMLParser
+    ) -> ExtractionResult:
+        before = self.client.usage.total
+        selectors = heal(schema, page, self.client)
+        spent = self.client.usage.total - before
+
+        if not selectors:
+            return ExtractionResult(
+                from_cache=False,
+                healed=True,
+                tokens=spent,
+                notes=["healing failed: model returned no usable selectors"],
+            )
+
+        result = self._apply(tree, schema, selectors)
+        result.from_cache = False
+        result.healed = True
+        result.tokens = spent
+        if not any("no match" in n for n in result.notes):
+            self.cache.put(schema.name, url, selectors)
+        return result
+
+    @staticmethod
+    def _apply(tree: HTMLParser, schema: Schema, selectors: dict[str, str]) -> ExtractionResult:
+        out = ExtractionResult()
+        for f in schema.fields:
+            sel = selectors.get(f.name)
+            nodes = tree.css(sel) if sel else []
+            if not nodes:
+                out.notes.append(f"{f.name}: no match")
+                out.data[f.name] = [] if f.many else None
+                continue
+            values = [_read(n, f.attr) for n in nodes]
+            out.data[f.name] = values if f.many else values[0]
+        return out
+
+
+def _read(node, attr: str | None) -> str:
+    if attr is not None:
+        return node.attributes.get(attr, "") or ""
+    return node.text(strip=True)

scrapemend-0.0.1/tests/test_condense.py

@@ -0,0 +1,58 @@
+"""Tests for HTML condensing — the token-saving pre-pass before healing."""
+
+from __future__ import annotations
+
+from scrapemend.condense import condense
+
+NOISY = """
+<html>
+  <head>
+    <style>.x { color: red; }</style>
+    <script>var tracking = 1; doStuff();</script>
+  </head>
+  <body>
+    <div class="product" data-track="abc" style="margin:0" onclick="go()">
+      <h1 id="name" class="title">Widget</h1>
+      <span class="price" data-currency="usd">$5.00</span>
+    </div>
+    <noscript>enable js</noscript>
+  </body>
+</html>
+"""
+
+
+def test_drops_script_and_style_content():
+    out = condense(NOISY)
+    assert "tracking" not in out
+    assert "color: red" not in out
+    assert "enable js" not in out
+
+
+def test_keeps_class_and_id_drops_noise_attrs():
+    out = condense(NOISY)
+    # Structural hooks the model needs must survive.
+    assert "product" in out
+    assert "title" in out
+    assert 'id="name"' in out or "name" in out
+    # Noise attributes must be gone.
+    assert "data-track" not in out
+    assert "onclick" not in out
+    assert "margin:0" not in out
+
+
+def test_preserves_the_actual_data_text():
+    out = condense(NOISY)
+    assert "Widget" in out
+    assert "$5.00" in out
+
+
+def test_respects_length_limit():
+    big = "<body>" + "<div class='a'>x</div>" * 10000 + "</body>"
+    out = condense(big, limit=500)
+    assert len(out) <= 500
+
+
+def test_malformed_html_does_not_crash():
+    out = condense("<div class='a'><span>unclosed")
+    assert isinstance(out, str)
+    assert "unclosed" in out

scrapemend-0.0.1/tests/test_deepseek_adapter.py

@@ -0,0 +1,44 @@
+"""Structural tests for the DeepSeek adapter — no network, no spend.
+
+Mirrors the OpenAI adapter tests: verify protocol conformance, lazy
+import, early failure on a missing key, and that the DeepSeek endpoint
+defaults are wired up — without ever calling the real API.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from scrapemend.llm import LLMClient
+
+
+def test_core_import_does_not_require_sdk():
+    import scrapemend  # noqa: F401
+    from scrapemend.providers import deepseek_client  # noqa: F401
+
+
+def test_missing_key_raises_before_any_call(monkeypatch):
+    monkeypatch.delenv("DEEPSEEK_API_KEY", raising=False)
+    from scrapemend.providers.deepseek_client import DeepSeekClient
+
+    with pytest.raises(ValueError, match="No API key"):
+        DeepSeekClient()
+
+
+def test_defaults_point_at_deepseek(monkeypatch):
+    monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-test-not-real")
+    from scrapemend.providers.deepseek_client import (
+        DEEPSEEK_BASE_URL,
+        DEEPSEEK_DEFAULT_MODEL,
+        DeepSeekClient,
+    )
+
+    try:
+        client = DeepSeekClient()
+    except ImportError:
+        pytest.skip("openai package not installed in this environment")
+
+    assert isinstance(client, LLMClient)
+    assert client.model == DEEPSEEK_DEFAULT_MODEL == "deepseek-chat"
+    assert DEEPSEEK_BASE_URL == "https://api.deepseek.com"
+    assert client.usage.total == 0

scrapemend-0.0.1/tests/test_openai_adapter.py

@@ -0,0 +1,41 @@
+"""Structural tests for the OpenAI adapter — no network, no spend.
+
+We never call the real API here. We only verify that the adapter obeys
+the LLMClient protocol and fails loudly and early when misconfigured.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from scrapemend.llm import LLMClient
+
+
+def test_importing_scrapemend_does_not_require_openai():
+    # Core import must stay dependency-light; adapter import is separate.
+    import scrapemend  # noqa: F401
+    from scrapemend.providers import openai_client  # noqa: F401
+
+
+def test_missing_key_raises_before_any_call(monkeypatch):
+    monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+    from scrapemend.providers.openai_client import OpenAIClient
+
+    with pytest.raises(ValueError, match="No API key"):
+        OpenAIClient()
+
+
+def test_adapter_satisfies_protocol_shape(monkeypatch):
+    # With a key present, construction should wire up a usage ledger and a
+    # complete() method, matching the structural LLMClient protocol.
+    monkeypatch.setenv("OPENAI_API_KEY", "sk-test-not-real")
+    from scrapemend.providers.openai_client import OpenAIClient
+
+    try:
+        client = OpenAIClient()
+    except ImportError:
+        pytest.skip("openai package not installed in this environment")
+
+    assert isinstance(client, LLMClient)  # runtime_checkable protocol
+    assert client.usage.total == 0
+    assert hasattr(client, "complete")

scrapemend-0.0.1/tests/test_pipeline.py

@@ -0,0 +1,72 @@
+"""End-to-end check of the cache-first / heal-on-miss flow.
+
+Uses the offline EchoClient so the whole pipeline runs with no network
+and no API key — the same way CI will run it.
+"""
+
+from __future__ import annotations
+
+from scrapemend import Schema, Scraper
+from scrapemend.cache import SelectorCache
+from scrapemend.llm import EchoClient
+
+PAGE_V1 = """
+<html><body>
+  <h1 class="title">Wireless Mouse</h1>
+  <span class="price">$19.99</span>
+</body></html>
+"""
+
+# Same data, different markup — simulates the site getting redesigned.
+PAGE_V2 = """
+<html><body>
+  <div class="product-name">Wireless Mouse</div>
+  <div class="cost">$19.99</div>
+</body></html>
+"""
+
+
+def _schema() -> Schema:
+    return (
+        Schema("product")
+        .add("title", "the product name")
+        .add("price", "the product price with currency")
+    )
+
+
+def test_heal_then_cache_then_reheal(tmp_path):
+    cache = SelectorCache(tmp_path / "cache.json")
+
+    # First run: no cache -> heal. Model "discovers" v1 selectors.
+    client = EchoClient(reply='{"title": "h1.title", "price": "span.price"}')
+    scraper = Scraper(client=client, cache=cache)
+
+    r1 = scraper.extract("https://shop.example/p/1", _schema(), html=PAGE_V1)
+    assert r1.healed is True
+    assert r1.from_cache is False
+    assert r1["title"] == "Wireless Mouse"
+    assert r1["price"] == "$19.99"
+    assert r1.tokens > 0
+
+    # Second run on the SAME markup: cached selectors work, no healing.
+    r2 = scraper.extract("https://shop.example/p/1", _schema(), html=PAGE_V1)
+    assert r2.from_cache is True
+    assert r2.healed is False
+    assert r2.tokens == 0
+    assert r2["title"] == "Wireless Mouse"
+
+    # Site redesigns (PAGE_V2). Cached selectors miss -> heal again.
+    client.reply = '{"title": "div.product-name", "price": "div.cost"}'
+    r3 = scraper.extract("https://shop.example/p/1", _schema(), html=PAGE_V2)
+    assert r3.healed is True
+    assert r3.from_cache is False
+    assert r3["title"] == "Wireless Mouse"
+    assert r3["price"] == "$19.99"
+
+
+def test_healing_failure_is_graceful():
+    # Model returns junk; pipeline should not crash, just report it.
+    scraper = Scraper(client=EchoClient(reply="not json"))
+    r = scraper.extract("https://x.example", _schema(), html=PAGE_V1)
+    assert r.healed is True
+    assert "healing failed" in r.notes[0]