@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/data-cleaning/evals/cases.yaml

@@ -0,0 +1,57 @@
+skill: data-cleaning
+
+should_trigger:
+  - prompt: "Clean and dedupe this customer CSV and give me something I can re-run every month with the same result."
+    why: Core ask — reproducible cleaning pipeline plus dedupe; the headline use case.
+  - prompt: "My integer ID column came in as 1001.0 floats and a bunch of rows are duplicated. Fix it."
+    why: Non-obvious dtype footgun — only this skill explains read-typing (dtype_backend) before deterministic dedupe.
+  - prompt: "Normalize these country values — USA, U.S., United States all mean the same thing."
+    why: Category normalization via a mapping table; the unmapped→quarantine pattern is this skill's.
+  - prompt: "Add a validation gate so bad rows fail loudly before they ever reach my model."
+    why: Schema-contract / pandera framing — declare types, ranges, allowed values; fail loud.
+  - prompt: "Quarantine the rows that don't pass instead of silently dropping them."
+    why: Non-obvious quarantine pattern unique to this skill; not a generic cleaning ask.
+  - prompt: "Limpiar y deduplicar este dataset y dejar un pipeline repetible."
+    why: Spanish trigger — same reproducible clean+dedupe intent, no English keyword present.
+  - prompt: "Why do my dates parse wrong sometimes and a few rows look duplicated after import?"
+    why: Symptom-led, never says 'clean' — ambiguous date format + dedupe both live here.
+
+should_not_trigger:
+  - prompt: "Export this cleaned DataFrame to Excel with a summary chart and conditional formatting."
+    route_to: spreadsheet-ops
+    why: Output is a styled sheet/formulas/chart; cleaning already ended at the clean table.
+  - prompt: "Scrape all the product prices off this catalog website."
+    route_to: data-scraper
+    why: Acquisition — the rows aren't held yet; cleaning only starts once you have tabular rows.
+  - prompt: "Pull the line items out of these invoice PDFs into a table."
+    route_to: structured-extraction
+    why: Parsing fields out of unstructured blobs into rows, not cleaning rows you already hold.
+  - prompt: "Aggregate this 5GB CSV with SQL fast on my laptop."
+    route_to: duckdb
+    why: OLAP speed over multi-GB files is the bottleneck, not correctness/normalization.
+  - prompt: "Build the KPI dashboard from this already-clean table."
+    route_to: business-intelligence
+    why: Consumes the clean output; the boundary is the validated table.
+  - prompt: "Forecast next quarter's revenue from this monthly clean series."
+    route_to: forecasting
+    why: Modeling on top of clean data, not producing the clean data.
+
+capability:
+  - scenario: >
+      Given a messy customers CSV — an id column read as floats with .0, "N/A" and 999 sentinels in age,
+      duplicate rows per customer, country values USA / U.S. / United States, and malformed dates — produce
+      a reproducible clean() pipeline plus a pandera schema that types, normalizes, dedupes deterministically,
+      validates, and quarantines failures rather than dropping them.
+    must_include:
+      - Reads with explicit dtype control (dtype_backend="pyarrow" or numpy_nullable + na_values) so ids stay int, not float
+      - Profiles first (nulls %, dup count, per-column dtype, cardinality) to drive the plan
+      - Normalizes strings with strip/casefold/NFKC before dedupe
+      - Maps categories with a mapping table (dict), unmapped → NA/quarantine, not a regex pile
+      - Turns sentinels (-1/999/"N/A") into NA, with an explicit clip-vs-reject range decision
+      - Parses dates with an explicit format and errors="coerce", then checks the coerced-NaT count
+      - Deduplicates deterministically — explicit subset key + stable sort_values + keep
+      - Handles missing per column; no bare fillna(0); impute adds a _was_missing flag
+      - Defines a pandera DataFrameModel with coerce, nullable, Field checks (isin/ge/le/unique), and lazy=True
+      - Quarantines failing rows to an artifact instead of silently dropping them
+      - Writes to a NEW output artifact, leaving the raw source untouched
+      - Logs a row-count in/out/coerced/quarantined diff and pins library versions

package/skills/shared/data-cleaning/references/normalization-recipes.md

@@ -0,0 +1,136 @@
+# Normalization recipes — copy-paste
+
+Each recipe is deterministic and re-runnable. Drop them into your `normalize(df)` step. Polars equivalents
+at the end for when pandas runs out of RAM.
+
+## Category mapping (unmapped → quarantine, never silent pass)
+
+A dict is auditable; a regex tower is not. Anything not in the map becomes NA so the schema gate catches it.
+
+```python
+COUNTRY = {
+    "usa": "US", "u.s.": "US", "u.s.a.": "US", "united states": "US", "us": "US",
+    "españa": "ES", "espana": "ES", "spain": "ES", "es": "ES",
+    "france": "FR", "fr": "FR",
+}
+
+def map_category(s, mapping):
+    key = s.str.strip().str.casefold().str.normalize("NFKC")
+    mapped = key.map(mapping)
+    unmapped = key.notna() & mapped.isna()
+    if unmapped.any():
+        # surface the new values so you extend the map deliberately, not silently
+        print("UNMAPPED categories:", sorted(key[unmapped].unique()))
+    return mapped     # unmapped rows are NA -> the schema's isin/nullable=False check rejects them
+
+df["country"] = map_category(df["country"], COUNTRY)
+```
+
+## Robust date parsing (explicit format, count the casualties)
+
+```python
+def parse_dates(s, fmt="%Y-%m-%d", utc=True):
+    parsed = pd.to_datetime(s, format=fmt, utc=utc, errors="coerce")
+    failed = parsed.isna() & s.notna()        # coerced to NaT but original wasn't null = parse failure
+    if failed.any():
+        raise ValueError(f"{failed.sum()} values did not match {fmt!r}: {s[failed].head().tolist()}")
+    return parsed
+```
+
+If a single source genuinely mixes formats, parse each known format and `combine_first`, then assert no NaT
+remains. Never fall back to dayfirst inference — `03/04/2026` is ambiguous and pandas picks silently.
+
+```python
+def parse_mixed(s, formats=("%Y-%m-%d", "%d/%m/%Y")):
+    out = pd.Series(pd.NaT, index=s.index, dtype="datetime64[ns, UTC]")
+    for fmt in formats:
+        try_ = pd.to_datetime(s, format=fmt, utc=True, errors="coerce")
+        out = out.fillna(try_)
+    remaining = out.isna() & s.notna()
+    assert not remaining.any(), f"{remaining.sum()} dates matched none of {formats}"
+    return out
+```
+
+## Unicode / encoding repair
+
+```python
+# 1. Read with the RIGHT encoding (don't let pandas/locale guess). If you see mojibake like "Rodríguez",
+#    the file is utf-8 misread as latin-1 — re-read as utf-8.
+df = pd.read_csv("raw.csv", encoding="utf-8")
+
+# 2. Normalize text: NFKC folds full-width/compatibility forms; strip control chars and zero-width junk.
+import re
+ZERO_WIDTH = re.compile(r"[​‌‍]")
+def clean_text(s):
+    return (s.str.normalize("NFKC")
+             .str.replace(ZERO_WIDTH, "", regex=True)
+             .str.strip())
+
+# 3. Last-resort repair of already-corrupted text (needs `pip install ftfy`):
+#    from ftfy import fix_text; df["name"] = df["name"].map(fix_text)
+```
+
+## Sentinel → NA
+
+Map the per-column sentinels to real NA *before* any numeric check, so means and ranges aren't poisoned.
+
+```python
+SENTINELS = {
+    "age":     [-1, 999],
+    "income":  [-1, 0, 999999],     # 0 is a sentinel here only if 0 income is impossible in this dataset
+    "rating":  [-1],
+}
+for col, vals in SENTINELS.items():
+    df[col] = df[col].mask(df[col].isin(vals))     # -> NA; decide drop/impute/quarantine in handle_missing
+```
+
+## Numeric range — clip vs reject
+
+```python
+# CLIP: out-of-range is plausibly a recording cap → pull to the bound (keeps the row).
+df["age"] = df["age"].clip(lower=0, upper=120)
+
+# REJECT: out-of-range is impossible → set NA so the gate quarantines the row (don't fabricate a value).
+df["age"] = df["age"].mask(~df["age"].between(0, 120))
+```
+
+Decide per column and write the choice down — clipping a fraud-amount silently hides outliers; rejecting a
+sensor reading silently loses data. The right call depends on the field's meaning.
+
+## Polars equivalents (clean-at-scale)
+
+Same profile→normalize→dedupe→validate shape, lazy and parallel. Pin Polars and stay deterministic.
+
+```python
+import polars as pl
+
+lf = pl.scan_csv("raw.csv", infer_schema_length=10_000)     # lazy: nothing reads until .collect()
+
+clean = (
+    lf
+    # strings: strip + lowercase + (NFKC via map if needed)
+    .with_columns(pl.col("country").str.strip_chars().str.to_lowercase().alias("country"))
+    # category mapping table
+    .with_columns(pl.col("country").replace_strict(COUNTRY, default=None))
+    # sentinels -> null
+    .with_columns(pl.when(pl.col("age").is_in([-1, 999])).then(None).otherwise(pl.col("age")).alias("age"))
+    # dates: explicit format, strict=False coerces failures to null (then check the count)
+    .with_columns(pl.col("signup").str.to_datetime("%Y-%m-%d", strict=False, time_zone="UTC"))
+    # missing: drop rows missing the required key
+    .drop_nulls(subset=["customer_id"])
+    # dedupe: deterministic — sort then keep first per key
+    .sort(["customer_id", "updated_at"], descending=[False, True])
+    .unique(subset=["customer_id"], keep="first")
+    .collect()                                              # execute the lazy plan
+)
+
+# fill_null deliberately (with a flag column), never a blind 0:
+clean = clean.with_columns(
+    pl.col("income").is_null().alias("income_was_missing"),
+    pl.col("income").fill_null(strategy="mean"),
+)
+```
+
+pandera validates Polars frames too (`import pandera.polars as pa`), so the gate from
+[validation-patterns.md](validation-patterns.md) carries over. Hand multi-GB analytical SQL to
+[`duckdb`](../../duckdb/SKILL.md) — the engine changes, the discipline does not.

package/skills/shared/data-cleaning/references/validation-patterns.md

@@ -0,0 +1,134 @@
+# Validation patterns — pandera, GX Core, dbt
+
+The gate is what makes cleaned data *trustworthy*: it asserts the contract holds and fails loud when next
+month's file drifts. Pick the lightest tool that covers your blast radius.
+
+## Which validator
+
+| Situation | Use | Why |
+| --- | --- | --- |
+| One pipeline, validate a DataFrame in Python | **pandera** | Lightweight, in-process, lives next to `clean()`; pin `pandera==0.31.1` (supports pandas ≥ 3) |
+| Many datasets/teams, shared data-quality platform, results store + docs | **GX Core 1.0** | Heavier; Data Context / Suites / Checkpoints are infrastructure, overkill for one script |
+| Cleaning lives in a SQL warehouse, not Python | **dbt** model contracts + tests | Contract enforced at build; tests run post-materialization |
+
+Default to pandera. Escalate only when the heavier tool's features are actually needed.
+
+## Pandera — the full DataFrameModel
+
+```python
+import pandera.pandas as pa
+from pandera.typing import Series
+
+class CustomerSchema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(unique=True, ge=1)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])
+    age:         Series[float] = pa.Field(ge=0, le=120, nullable=True)
+    email:       Series[str]   = pa.Field(str_matches=r"^[^@\s]+@[^@\s]+\.[^@\s]+$", nullable=True)
+    signup:      Series[pa.DateTime] = pa.Field(nullable=False)
+
+    class Config:
+        strict = True        # reject unexpected columns — schema drift fails the build
+        coerce = True        # coerce to declared dtype; raise if a value can't be coerced
+        ordered = False      # set True if column order is part of the contract
+
+    @pa.check("email", name="lowercase_email")
+    def email_is_lower(cls, s: Series[str]) -> Series[bool]:
+        # custom row-wise check: returns a boolean Series, one per row
+        return s.isna() | (s == s.str.lower())
+
+    @pa.dataframe_check
+    def row_budget(cls, df) -> bool:
+        # whole-frame invariant: catch a truncated/exploded extract
+        return 1 <= len(df) <= 5_000_000
+```
+
+`coerce` fixes types the contract expects. `nullable` declares which columns may hold NA. Field checks
+(`ge`, `le`, `isin`, `unique`, `str_matches`) are allowed-value rules. `@pa.check` is a custom column
+check; `@pa.dataframe_check` is a whole-frame invariant.
+
+## Lazy errors + the quarantine split
+
+`lazy=True` collects **every** violation across the frame instead of dying on the first. Use it to split
+valid rows from failing rows — the failures become the quarantine, never a silent drop.
+
+```python
+def validate_and_split(df, schema):
+    """Return (valid_df, quarantine_df). Quarantine is kept, never discarded."""
+    try:
+        valid = schema.validate(df, lazy=True)
+        return valid, df.iloc[0:0].copy()          # empty quarantine — everything passed
+    except pa.errors.SchemaErrors as e:
+        # failure_cases columns: schema_context, column, check, failure_case, index
+        bad_idx = e.failure_cases["index"].dropna().unique()
+        quarantine = df.loc[df.index.isin(bad_idx)].copy()
+        quarantine["_failed_checks"] = (
+            e.failure_cases.dropna(subset=["index"])
+             .groupby("index")["check"].agg(lambda c: "; ".join(map(str, c)))
+        )
+        valid = df.loc[~df.index.isin(bad_idx)]
+        return valid, quarantine
+
+valid, quarantine = validate_and_split(df, CustomerSchema)
+quarantine.to_parquet("quarantine.parquet")        # someone investigates these
+print(f"valid={len(valid)} quarantined={len(quarantine)}")   # the audit diff
+```
+
+Validate **output** always. Validate **input expectations** too when you can state them (e.g. raw must have
+these columns before you start) — it turns a confusing mid-pipeline crash into a clear "the source changed".
+
+## GX Core 1.0 — checkpoint sketch
+
+GX rebranded its OSS to **GX Core** and shipped 1.0 GA with a simplified API. Reach for it when validation
+is a shared platform across many datasets, not a single script.
+
+```python
+import great_expectations as gx
+
+context = gx.get_context()                                   # Data Context
+ds  = context.data_sources.add_pandas("customers_src")       # Data Source
+asset = ds.add_dataframe_asset(name="customers")
+batch = asset.add_batch_definition_whole_dataframe("batch")
+
+suite = context.suites.add(gx.ExpectationSuite(name="customers_suite"))
+suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(column="age", min_value=0, max_value=120))
+suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(column="country", value_set=["US", "ES", "FR"]))
+
+vd = context.validation_definitions.add(
+    gx.ValidationDefinition(name="customers_vd", data=batch, suite=suite)
+)
+checkpoint = context.checkpoints.add(gx.Checkpoint(name="customers_cp", validation_definitions=[vd]))
+result = checkpoint.run(batch_parameters={"dataframe": df})
+assert result.success, "GX checkpoint failed — see Data Docs"
+```
+
+The flow is always **Data Context → Data Source → Expectation Suite → Validation Definition → Checkpoint**.
+You get a results store and generated Data Docs — worth the weight only when many people consume them.
+
+## dbt — contracts vs tests (cleaning in the warehouse)
+
+When cleaning is SQL in a warehouse, the contract lives in the model's YAML. **Contracts** are enforced at
+*build* (the model fails to build if the output is off-contract). **Tests** validate *after*
+materialization.
+
+```yaml
+# models/staging/_stg_customers.yml
+models:
+  - name: stg_customers
+    config:
+      contract: { enforced: true }          # build fails if columns/types drift — the contract
+    columns:
+      - name: customer_id
+        data_type: bigint
+        constraints: [{ type: not_null }, { type: unique }]
+      - name: country
+        data_type: varchar
+        data_tests:                          # validated post-materialization — the tests
+          - accepted_values: { values: ["US", "ES", "FR"] }
+      - name: age
+        data_type: integer
+        data_tests:
+          - dbt_utils.accepted_range: { min_value: 0, max_value: 120 }
+```
+
+Contract = "this shape is guaranteed at build"; tests = "these values held after the run". Use both: the
+contract stops schema drift, the tests catch data drift.

package/skills/shared/data-cleaning/scripts/verify.sh

@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# verify.sh — data-cleaning skill check. Run from anywhere. Read-only by default.
+#
+# Two layers:
+#   STATIC (always): SKILL.md exists, frontmatter has the required keys, description is one
+#     physical line, and both references/ files exist and are non-empty.
+#   RUNTIME (only if pandas + pandera importable): build a tiny fixture with ONE clearly-good
+#     row and ONE clearly-bad row (negative age / unmapped country), run a pandera schema with
+#     lazy=True, and assert the good row PASSES while the bad row is FLAGGED — proving the gate
+#     is not a no-op. If pandas/pandera are absent, print SKIP and still pass the static checks.
+#
+# Writes only inside a private mktemp dir, removed on exit. No network.
+# Exit: 0 on pass or skip; non-zero ONLY on a real failure (missing artifact, malformed
+# frontmatter, or a gate that fails to reject a known-bad row).
+
+YELLOW=$'\033[33m'; GREEN=$'\033[32m'; RED=$'\033[31m'; NC=$'\033[0m'
+skip() { printf '%s[skip]%s %s\n' "$YELLOW" "$NC" "$*"; }
+ok()   { printf '%s[ok]%s %s\n'   "$GREEN"  "$NC" "$*"; }
+err()  { printf '%s[fail]%s %s\n' "$RED"    "$NC" "$*"; }
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SKILL_DIR="$(dirname "$SCRIPT_DIR")"
+SKILL="$SKILL_DIR/SKILL.md"
+fail=0
+
+# ---- STATIC CHECKS (always run) ----
+if [ ! -f "$SKILL" ]; then
+  err "SKILL.md not found at $SKILL"; exit 1
+fi
+ok "SKILL.md present"
+
+for key in "name:" "description:" "tags:" "recommends:" "origin:"; do
+  if grep -q "^${key}" "$SKILL"; then
+    ok "frontmatter has ${key%:}"
+  else
+    err "frontmatter missing ${key%:}"; fail=1
+  fi
+done
+
+# description must be exactly one physical line
+desc_lines="$(grep -c '^description:' "$SKILL" || true)"
+if [ "$desc_lines" = "1" ]; then
+  ok "description present as a single key line"
+else
+  err "expected exactly one 'description:' line, found $desc_lines"; fail=1
+fi
+
+for ref in "references/validation-patterns.md" "references/normalization-recipes.md"; do
+  if [ -s "$SKILL_DIR/$ref" ]; then
+    ok "$ref present and non-empty"
+  else
+    err "$ref missing or empty"; fail=1
+  fi
+done
+
+if [ "$fail" -ne 0 ]; then
+  err "static checks failed"; exit 1
+fi
+
+# ---- RUNTIME GATE CHECK (only if deps present) ----
+if ! command -v python3 >/dev/null 2>&1; then
+  skip "python3 not found — ran static checks only"; exit 0
+fi
+if ! python3 -c "import pandas, pandera" >/dev/null 2>&1; then
+  skip "pandas/pandera not importable — ran static checks only"; exit 0
+fi
+
+TMP="$(mktemp -d 2>/dev/null || mktemp -d -t dc_verify)"
+trap 'rm -rf "$TMP"' EXIT
+
+python3 - <<'PY'
+import sys
+import pandas as pd
+import pandera.pandas as pa
+from pandera.typing import Series
+
+# Fixture: row 1 is clearly GOOD; row 2 is clearly BAD (negative age + unmapped country -> NA).
+COUNTRY = {"usa": "US", "u.s.": "US", "united states": "US", "es": "ES"}
+raw = pd.DataFrame({
+    "customer_id": [1, 2],
+    "country":     ["U.S.", "Atlantis"],   # row2 unmapped -> NA -> fails isin
+    "age":         [42, -5],               # row2 out of range -> fails ge(0)
+})
+raw["country"] = raw["country"].str.strip().str.casefold().map(COUNTRY)
+
+class Schema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(ge=1, unique=True)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])
+    age:         Series[float] = pa.Field(ge=0, le=120)
+    class Config:
+        coerce = True
+        strict = True
+
+try:
+    Schema.validate(raw, lazy=True)
+    print("GATE-NOOP: schema accepted a known-bad row")  # gate did nothing -> failure
+    sys.exit(1)
+except pa.errors.SchemaErrors as e:
+    bad_idx = set(e.failure_cases["index"].dropna().astype(int).unique())
+    # The bad row (index 1) must be flagged; the good row (index 0) must NOT be.
+    if 1 in bad_idx and 0 not in bad_idx:
+        sys.exit(0)
+    print(f"GATE-WRONG: flagged indices were {sorted(bad_idx)}; expected {{1}}")
+    sys.exit(1)
+PY
+rc=$?
+if [ "$rc" -eq 0 ]; then
+  ok "validation gate rejected the bad row and passed the good row (gate is not a no-op)"
+  exit 0
+else
+  err "validation gate did not behave as documented"
+  exit 1
+fi

package/skills/shared/data-scraper/SKILL.md

@@ -0,0 +1,130 @@
+---
+name: data-scraper
+description: "Use when data lives on a website with no usable API — listings, prices, public records — and the scrape must stay legal and not get blocked: legal gate, extraction path, durable selectors, pacing, resilience. NOT parsing bytes you already hold into fields (that is structured-extraction), NOT a documented API or key (that is api-connector-builder)."
+tags: [web-scraping, crawler, anti-bot, robots-txt, gdpr-compliance, playwright, rate-limiting]
+recommends: [api-connector-builder, structured-extraction, data-cleaning, gdpr-privacy, webhooks]
+origin: ai-hub
+---
+
+# Data scraper
+
+You get bytes off websites you do not control — legally, and without getting blocked. You pick the cheapest extraction path that works, build selectors that survive a redesign, pace requests so the host neither bans nor sues you, and you write down the legal basis *before* the first request goes out.
+
+One rule above all the others: **scraping is the fallback, not the default.** It is what you reach for only when no API serves the data. If the site has a documented API or you hold a key, stop — that is `../api-connector-builder/SKILL.md`. And once you have the bytes, parsing them into fields is `../structured-extraction/SKILL.md`, normalizing the rows is `../data-cleaning/SKILL.md`. This skill ends the moment you hold the bytes.
+
+In 2025-2026 scrapers do not fail on parsing. They fail on a terms-of-service breach, on GDPR exposure, or on being blocked after hammering a host. So the work runs in this order: legal gate → extraction path → tool → selectors → politeness → resilience. Skipping the gate is how you end up in *Meta v. Bright Data*.
+
+## The legal gate — run this before any request
+
+Walk every item. Each ends in **proceed**, **proceed-narrowed**, or **stop**. One item at stop means the whole scrape stops until you resolve it. Depth and the case law are in `references/legal-compliance.md`.
+
+The gates below are the ones with real legal exposure — contract, data-protection law, and anti-circumvention. **robots.txt is not one of them**: it is a voluntary convention with no statutory force, so it informs the decision and never blocks it on its own.
+
+- [ ] **Is there an API?** If yes, you are in the wrong skill — never scrape what an API serves. → otherwise proceed.
+- [ ] **Did you read the ToS, and does login/auth apply?** Scraping is most exposed as *breach of contract* when you accepted terms — typically by logging in (*Meta v. Bright Data*, 2024). Logged-out public data weakens that claim. **Prefer logged-out public pages; never bypass auth.** → narrow to public, or stop.
+- [ ] **Is it personal data?** Names, emails, photos, reviews, IP addresses all count. Scraping public personal data for a *new* purpose — aggregation, resale, AI training — is a severe GDPR breach with fines into the tens of millions of EUR. You need a lawful basis (usually legitimate interest) and data minimization. Filter out special categories at the source. → proceed-narrowed (basis + minimization, see `../gdpr-privacy/SKILL.md`), or stop.
+- [ ] **robots.txt and ai.txt?** Read them — they are a *convention*, not law, so this item never stops a scrape on its own. A `Disallow` tells you the host would rather you did not, and `Crawl-delay` tells you its tolerance; both are useful intelligence about where you are likely to get blocked. Weigh it: honoring robots is the low-friction default and the cleanest evidence of good faith, but scraping your own site, one you have permission for, or public pages for research is legitimate whether or not robots allows it. → **advisory: note the decision and move on.**
+- [ ] **Are you about to bypass a control you were shown?** A CAPTCHA, a hard block, an enforced rate limit. *Reddit v. Perplexity AI* (2025) turns precisely on whether anti-bot measures were circumvented — a materially worse position than respectfully pacing public pages. Pacing public data is defensible; defeating a control is the frontier where you lose. → **stop. Do not solve the CAPTCHA. Do not bypass the block.**
+
+`hiQ v. LinkedIn` established that scraping *public* data is not automatically a CFAA violation — but that is the floor, not a license. Public + logged-out + no-personal-data is the defensible quadrant, and honoring robots keeps it tidy without being what makes it lawful. Anything outside it, document why and get a human to sign off.
+
+## Extraction-path decision table
+
+Walk *down* only when the rung above is unavailable. ~94% of modern sites are client-side rendered, yet most still ship machine-readable structured data — parse that before you launch a browser.
+
+| Path | Use when | Cost | Detectability |
+|------|----------|------|---------------|
+| **API** (incl. internal XHR/JSON endpoints) | Any documented API, or the page fetches its own JSON you can call directly | Lowest | Lowest — looks like normal traffic |
+| **sitemap.xml + JSON-LD** | `/sitemap.xml` lists the URLs; `<script type="application/ld+json">` carries the records (JSON-LD is Google's preferred structured format, so it is everywhere) | Low — one HTTP GET, parse JSON | Low |
+| **HTML selectors** | Data is in server-rendered HTML, no JSON-LD, no XHR JSON | Medium — selectors drift on redesign | Medium |
+| **Headless browser** | The data only exists after JS executes (SPA, lazy-load, infinite scroll) and there is no callable XHR endpoint | Highest — CPU, RAM, time, easiest to fingerprint | Highest |
+
+Before you reach for a browser, open DevTools → Network and look for the XHR/fetch that already returns JSON. Calling that endpoint directly is faster, stabler, and less detectable than rendering the whole page to read what the page itself fetched.
+
+## Tool picker
+
+| Site profile | Tool | Version (2026) | The one reason |
+|--------------|------|----------------|----------------|
+| Static HTML, no fingerprint wall | `httpx` + `selectolax` (or BeautifulSoup) | current | Fast, no browser; `selectolax` parses far quicker than `lxml` |
+| Static HTML but TLS/JA3 fingerprint blocks you | `curl_cffi` (profile `chrome131`) | current | Impersonates a real browser's TLS/JA3/HTTP2 fingerprint, not just the User-Agent |
+| JS-rendered SPA | Playwright | 1.60.0 (1.59 shipped 2026-04-01) | First-class async, auto-wait, the maintained headless standard |
+| Scalable, resilient, recurring crawler | Crawlee (JS or Python) | actively maintained 2026 | Wraps Playwright + proxy rotation + browserforge fingerprints + a disk-persisted `RequestQueue` that resumes after a crash |
+| Pure-Python static, legacy codebase | Scrapy | current | Battle-tested for static targets — but its Twisted core lags the asyncio ecosystem; pick Crawlee for new work |
+
+Default new builds to **Crawlee** when the job is recurring or must not break; reach for plain `httpx`/`curl_cffi` only when the target is static and one-shot.
+
+## Robust selectors
+
+Selectors break on redesign because they ride on layout, not meaning. Anchor on what is *semantically* stable — `data-*` attributes, ARIA roles, microdata, visible text — never on `nth-child` chains or generated CSS class hashes (`.css-1a2b3c`), which change on every build.
+
+```html
+<!-- the page you are scraping -->
+<article data-testid="listing-card">
+  <h2 class="css-1a2b3c">Acme Drill 9000</h2>
+  <span data-price="129.00">€129,00</span>
+</article>
+```
+
+```python
+# Bad — rides on layout and a build-generated hash; dies on the next deploy
+title = page.query_selector("div:nth-child(3) > article > .css-1a2b3c").inner_text()
+
+# Good — anchor on stable semantics, with a fallback chain, and fail loud
+def text_or_raise(card, selectors, field):
+    for sel in selectors:           # try each selector in priority order
+        el = card.query_selector(sel)
+        if el and el.inner_text().strip():
+            return el.inner_text().strip()
+    raise LookupError(f"required field {field!r} not found via {selectors}")
+
+card  = page.query_selector('[data-testid="listing-card"]')
+title = text_or_raise(card, ["h2", '[itemprop="name"]'], "title")
+price = text_or_raise(card, ["[data-price]", "span:has-text('€')"], "price")
+```
+
+Two rules baked into that snippet:
+
+- **Fallback selector chains.** Give every required field 2-3 ordered selectors. A single redesign rarely breaks all of them at once, so the scraper degrades instead of dying.
+- **Fail loud on a missing required field.** Raise — never silently write `null`. A silent null is a corrupted dataset you discover three months later. A raised error is a fix you make today.
+
+## Politeness and anti-block
+
+Concrete numbers beat "be respectful." Depth — fingerprint profiles, header sets, proxy taxonomy — is in `references/anti-bot.md`.
+
+- **Pace.** Start at **1 request per 1-3 seconds per host**, concurrency capped at **2-5 per host**. Tune *down* if you see 429s, never up to chase speed.
+- **Backoff with jitter.** On a transient error, exponential backoff *plus* random jitter — `delay = base * 2**attempt + random(0, base)`. Without jitter, every worker retries in lockstep and you self-DDoS the host.
+- **Honor 429 and `Retry-After`.** A 429 with `Retry-After: 30` means wait 30 seconds, not retry immediately. Ignoring it is the fastest route to an IP ban.
+- **Conditional GET.** Send `If-Modified-Since` / `If-None-Match` (ETag) on re-crawls. A `304 Not Modified` costs nothing and tells you the page is unchanged — cheaper for you, lighter on the host.
+- **Realistic headers + TLS fingerprint.** A bare `python-requests` User-Agent is an instant tell. Send a full, current browser header set; when JA3/TLS fingerprinting blocks you, switch to `curl_cffi` (`chrome131`) — see the references.
+- **Proxies only when justified.** Residential/mobile proxies are for geo-gating and IP-rate distribution on a target that *permits* the scrape — never to evade a block you were explicitly shown. Budget for them on large recurring jobs; do not reach for them to defeat a hard block (that is the legal line you do not cross).
+
+> **Resilience beats speed.** A scraper that runs 50% slower but never breaks is infinitely more valuable than a fast one that dies weekly. Pace for survival, not throughput.
+
+## Resilience
+
+A recurring crawler must survive crashes, redeploys, and the target's schema drift. Patterns and copy-paste starters are in `references/frameworks.md`.
+
+- **Resumable queue.** Use a disk-persisted request queue (Crawlee's `RequestQueue`) so a crash resumes from where it stopped, not from zero. Re-crawling 10k pages because the box rebooted is wasted budget and extra load on the host.
+- **Idempotent writes.** Upsert keyed on a stable id (the source URL or a record id), never blind append — so a re-run repairs rows instead of duplicating them.
+- **Checkpoint every N records.** Flush progress periodically so an interrupt loses minutes, not hours.
+- **Change detection by hash.** Store a content hash per record; only re-process when it changes. This pairs with conditional GET to keep recurring crawls cheap.
+- **Monitor and alert.** Alert on a spike in missing-field errors (schema drift — the site redesigned) and on a spike in 403/429 (you are being blocked). Both are silent-failure modes that rot a dataset until someone checks.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Scraping behind a login, then calling it "public data" | Accepting ToS at login is the breach-of-contract hook (*Meta v. Bright Data*) | Stay logged-out on public pages; never bypass auth |
+| Not even reading robots.txt / ai.txt | You lose free intelligence on where you will get blocked, and any good-faith story later | Parse both; honor by default, override deliberately and write down why |
+| No delay, unbounded concurrency | Hammers the host → IP ban, possible CFAA-style exposure | 1 req / 1-3s, cap 2-5 concurrent per host |
+| Selectors on `nth-child` / `.css-1a2b3c` hashes | Break on the next deploy; silent data loss | Anchor on `data-*` / semantic / text, with fallbacks |
+| Silently writing `null` on a missing field | Corrupts the dataset; discovered months later | Raise on a missing *required* field — fail loud |
+| Solving a CAPTCHA / bypassing a hard block | Circumventing a shown control (*Reddit v. Perplexity*) — the worst legal posture | Stop. That control is a "no." |
+| Scraping personal data with no lawful basis | GDPR fines into tens of millions EUR | Establish basis + minimize + filter special categories (`../gdpr-privacy/SKILL.md`) |
+| Storing everything "just in case" | Defeats minimization; expands breach blast radius | Keep only fields the purpose needs; set retention |
+| Launching a browser when JSON-LD was right there | Slowest, most detectable, most expensive path | Check XHR/JSON-LD/sitemap first; browser is last |
+| No resume — a crash restarts from zero | Wastes budget, doubles load on the host | Disk-persisted resumable queue |
+| Hardcoding one User-Agent forever | Stale UA is an easy bot tell | Current full header set; rotate when justified |
+| Retrying with no backoff | Lockstep retries self-DDoS the host | Exponential backoff + jitter; honor `Retry-After` |
+
+When in doubt about whether a scrape is defensible, the answer is the gate. Run it, write down the outcome, and only then send a request.

package/skills/shared/data-scraper/evals/README.md

@@ -0,0 +1,3 @@
+# Evals: data-scraper
+
+These cases are routing and coverage checks, not an automated pass/fail suite. Read `cases.yaml` against `SKILL.md`. For each `should_trigger` prompt, confirm the description and body would plausibly fire this skill rather than a sibling — note the two non-obvious symptom cases (429/IP-ban and selectors-break-on-redesign, which read as ops problems but are core here) and the Spanish and Catalan triggers. For each `should_not_trigger` prompt, confirm the named `route_to` sibling is the better home and that the boundary in the description would defer to it (structured-extraction once you hold bytes, api-connector-builder when an API exists, data-cleaning for normalization, e2e-testing for UI flow automation, gdpr-privacy for a pure lawful-basis question). For the `capability` case, have a human or a harness LLM produce the response and check it hits every `must_include` item — above all that it runs the legal gate first, walks the extraction path before reaching for a browser, paces per host with backoff and honors 429, uses a resumable queue, fails loud on missing fields, and refuses to bypass a CAPTCHA or hard block. There is no scripted assertion here. The separate `scripts/verify.sh` is the only automated check, and it statically lints an emitted scraper file for a compliance preflight, a rate limit/concurrency cap, retry/backoff, and the absence of CAPTCHA-solver / auth-bypass calls — not these cases.

package/skills/shared/data-scraper/evals/cases.yaml

@@ -0,0 +1,46 @@
+skill: data-scraper
+
+should_trigger:
+  - prompt: "I need to pull all product prices off this competitor's catalog site — they have no API."
+    why: Core case — structured data off a site with no API, the canonical scrape.
+  - prompt: "Build me a crawler that grabs new public tender listings every morning."
+    why: Recurring resilient crawler — routes to the resilience + tool-picker sections.
+  - prompt: "My scraper keeps getting 429s and then the IP gets banned after about 200 pages."
+    why: Non-obvious — the symptom is a politeness/anti-block failure, not a parsing one.
+  - prompt: "The CSS selectors in my scraper break every time the site redesigns."
+    why: Non-obvious — points at the robust-selectors section (data-attr / semantic / fallback chains).
+  - prompt: "Is it legal for me to scrape this site, and how do I do it without getting sued?"
+    why: Legal-gate first — ToS, robots.txt, personal-data, circumvention before any request.
+  - prompt: "Vull extreure les dades de contacte d'un directori d'empreses públic."
+    why: Catalan trigger — public directory scrape that also nudges the GDPR personal-data gate.
+  - prompt: "Necesito scrapear una página de anuncios sin que me bloqueen."
+    why: Spanish trigger — extraction plus the anti-block concern.
+
+should_not_trigger:
+  - prompt: "Parse these 500 saved HTML invoices into a fields table."
+    route_to: structured-extraction
+    why: Bytes are already in hand; the task is field extraction from documents, not fetching.
+  - prompt: "The site publishes a documented REST API with a key — pull the orders into our DB."
+    route_to: api-connector-builder
+    why: A real API exists; never scrape what an API serves.
+  - prompt: "Dedup and normalize this messy scraped CSV — fix the types and nulls."
+    route_to: data-cleaning
+    why: Downstream of having the rows; normalization, not extraction.
+  - prompt: "Automate clicking through our checkout flow to test it end to end."
+    route_to: e2e-testing
+    why: Browser UI automation for testing a flow, not data extraction.
+  - prompt: "What's my GDPR lawful basis for processing this customer list we already hold?"
+    route_to: gdpr-privacy
+    why: Pure legal-basis question with no scrape — the privacy artifact layer.
+
+capability:
+  - scenario: "Stand up a resilient daily crawler for a public, JS-rendered listings site that has no API. It contains business names and contact emails. Tell me how to build it so it stays legal and doesn't get blocked."
+    must_include:
+      - Runs the legal gate before any request — reads ToS, checks robots.txt/ai.txt, and flags the contact emails as personal data needing a lawful basis + minimization (defers basis detail to gdpr-privacy)
+      - Walks the extraction path — checks for a callable XHR/JSON endpoint, sitemap.xml, and JSON-LD before committing to a headless browser
+      - Picks a tool with a reason — Playwright (1.60) or Crawlee for the JS-rendered SPA, not plain httpx
+      - Builds selectors on stable attributes (data-* / semantic / text) with a fallback chain, and fails loud on a missing required field rather than writing null
+      - Sets a per-host rate limit (≈1 req / 1-3s, concurrency cap 2-5), exponential backoff with jitter, and honors 429 / Retry-After
+      - Uses a resumable disk-persisted queue (Crawlee RequestQueue) and idempotent upserts keyed on a stable id so a crash resumes instead of restarting
+      - Refuses to bypass any CAPTCHA or hard block, treating it as a stop and re-running the gate
+      - Adds monitoring/alerts on missing-field spikes (schema drift) and 403/429 spikes (blocking)