@damiangilgonzalez/harness 1.3.7

package/skills/data-policy/references/consent-and-ropa.md

@@ -0,0 +1,97 @@
+# Consent matrix, ROPA, and the legitimate-interest worksheet
+
+The copy-ready artifacts behind the consent and ROPA sections. Validate with a
+DPO before adoption; nothing here is legal advice.
+
+## ROPA template (Art. 30)
+
+One row per processing activity. Art. 30 columns are mandatory; the lawful-basis
+line is recommended even though Art. 30 doesn't strictly require it — it speeds
+audits, DPIAs, and notice updates.
+
+```text
+Activity:      <name of the processing activity>
+Purpose:       <why you process this data>
+Data cats:     <categories of personal data>        | Subjects: <categories of data subjects>
+Recipients:    <internal teams + processors + third parties>
+Transfers:     <country + transfer mechanism, e.g. SCCs; or "none">
+Retention:     <period + expiry action — must match the retention schedule>
+Security:      <RBAC, encryption, logging, ...>
+Lawful basis:  <Art. 6 sub-paragraph>               ← recommended, not strictly required
+```
+
+### Worked rows
+
+```text
+Activity:      Customer support ticketing
+Purpose:       resolve and track support requests
+Data cats:     name, email, account ID, message content | Subjects: customers
+Recipients:    internal support team; Zendesk (processor)
+Transfers:     US (SCCs in place)
+Retention:     2 years after ticket closed, then delete
+Security:      RBAC, encryption at rest, access logging
+Lawful basis:  Art. 6(1)(b) contract
+
+Activity:      Product analytics
+Purpose:       understand feature usage to improve the product
+Data cats:     pseudonymous event data, device, coarse location | Subjects: users
+Recipients:    internal product team; analytics processor
+Transfers:     none (EU-hosted)
+Retention:     14 months, then anonymize
+Security:      pseudonymization, RBAC, encryption at rest
+Lawful basis:  Art. 6(1)(a) consent (non-essential analytics)
+```
+
+## Consent matrix template
+
+```text
+| Purpose | Lawful basis (Art. 6(1)(a)) | Capture point | Proof fields stored | Withdrawal mechanism | Refresh cadence |
+|---------|-----------------------------|---------------|---------------------|----------------------|-----------------|
+|         |                             |               | timestamp, text version, scope, method | | ~12 mo / on material change |
+```
+
+### Worked example
+
+```text
+| Purpose           | Lawful basis | Capture point   | Proof fields stored                | Withdrawal       | Refresh             |
+|-------------------|--------------|-----------------|------------------------------------|------------------|---------------------|
+| Marketing email   | Art. 6(1)(a) | signup checkbox | ts, text v2.1, scope, method       | one-click unsub  | 12 mo / on change   |
+| Product analytics | Art. 6(1)(a) | cookie banner   | ts, banner vN, categories, method  | re-open banner   | 12 mo / on change   |
+```
+
+Capture rules (Art. 4(11) / Art. 7): affirmative opt-in, never pre-ticked;
+granular per purpose; reject as easy as accept (no dark patterns); withdrawal as
+easy as giving. The ePrivacy Regulation was withdrawn Feb 2025 — the ePrivacy
+Directive still governs cookies and trackers.
+
+## Consent withdrawal / refresh workflow
+
+1. **Withdraw** — subject clicks the one-click control (unsubscribe link, banner
+   re-open). No friction, no retention-by-dark-pattern.
+2. **Stop processing** — the purpose tied to that consent halts immediately.
+3. **Log** — write the withdrawal event (timestamp, scope withdrawn, method)
+   to the consent log; keep proof of withdrawal as you keep proof of consent.
+4. **Honor downstream** — propagate to processors and suppress in the CDP/ESP.
+5. **Refresh** — re-request consent after ~12 months or whenever the
+   consent text materially changes; re-prompting resets the proof record.
+
+## Legitimate-interest balancing test (Art. 6(1)(f))
+
+Run and document this before relying on legitimate interest. Anchor to EDPB
+Guidelines 1/2024 (Oct 2024).
+
+1. **Purpose test** — Is there a legitimate interest, clearly articulated?
+   (e.g. fraud prevention, network security, direct B2B marketing.)
+2. **Necessity test** — Is the processing actually necessary for that interest,
+   or would less data / a less intrusive method achieve it?
+3. **Balancing test** — Do the individual's interests, rights, and reasonable
+   expectations override the interest? Consider the relationship, the data
+   sensitivity, and whether the subject would expect this use.
+
+If the balance tips against you, fall back to consent or contract — or don't
+process. Record the outcome; a regulator can ask to see the assessment.
+
+---
+
+**Not legal advice.** A qualified DPO or privacy counsel must validate the
+lawful-basis register, ROPA, and consent model before adoption.

package/skills/data-policy/references/retention-schedule.md

@@ -0,0 +1,83 @@
+# Retention schedule — template + worked example
+
+This is the copy-ready artifact the body points to. Fill the template, validate
+every period against local + sector law, and end the document with the DPO /
+counsel sign-off line. Nothing here is legal advice.
+
+## The fillable template
+
+One row per personal-data category. No row ships without a period, a lawful
+basis, and an expiry action.
+
+```text
+| Data category | Purpose | Lawful basis (Art. 6) | Retention period | Expiry action | System of record | Legal-hold flag | Review date |
+|---------------|---------|-----------------------|------------------|---------------|------------------|-----------------|-------------|
+|               |         |                       |                  | delete/anon/archive |            | yes/no          | YYYY-MM-DD  |
+```
+
+Column rules:
+
+- **Retention period** — a concrete duration or named criteria ("36 months
+  after last order"), never "as long as necessary" alone.
+- **Lawful basis** — the specific Art. 6 sub-paragraph, not just "GDPR".
+- **Expiry action** — exactly one of delete / anonymize / archive (see note).
+- **System of record** — the actual store(s) the deletion job must touch,
+  including the warehouse and backups, not just the live table.
+- **Legal-hold flag** — whether rows here can be put under hold (litigation /
+  regulatory), which the deletion job must skip.
+- **Review date** — when this row is re-validated against current law.
+
+## Populated example schedule
+
+Periods below are common working defaults — **starting points, validate against
+local + sector law** (Usercentrics; Secure Privacy, 2026). They are not asserted
+as universally lawful.
+
+```text
+| Data category        | Purpose                  | Lawful basis        | Retention period            | Expiry action | System of record            | Legal-hold | Review date |
+|----------------------|--------------------------|---------------------|-----------------------------|---------------|-----------------------------|------------|-------------|
+| HR / payroll records | employment + statutory   | Art. 6(1)(b)+(c)    | 6 years after employment ends | archive     | HRIS + payroll backups      | yes        | 2027-01-01  |
+| Accounting / tax     | statutory bookkeeping    | Art. 6(1)(c)        | 10 years (statutory, EU)    | archive       | ERP + cold storage          | yes        | 2027-01-01  |
+| Customer / CRM       | fulfil + relationship    | Art. 6(1)(b)+(f)    | 3 years after last interaction | anonymize  | Postgres `customers` + DWH  | yes        | 2027-01-01  |
+| Marketing consent    | email campaigns          | Art. 6(1)(a)        | life of consent + proof     | delete        | CDP + consent log           | no         | 2027-01-01  |
+| Support tickets      | resolve + audit          | Art. 6(1)(b)+(f)    | 2 years after ticket closed | delete        | Zendesk (processor)         | yes        | 2027-01-01  |
+| Server / access logs | security + debugging     | Art. 6(1)(f)        | 90 days                     | delete        | log store + backup rotation | no         | 2027-01-01  |
+```
+
+## delete vs anonymize vs archive
+
+Pick the expiry action deliberately:
+
+- **delete** — destroy the record everywhere it lives. The default when no
+  lawful reason to keep remains.
+- **anonymize** — strip identifiers so the data is no longer personal data;
+  storage limitation then no longer applies. Valid only if re-identification is
+  genuinely infeasible (a pseudonym you can reverse is *not* anonymization).
+  Useful when you want aggregate analytics after the personal purpose ends.
+- **archive** — retain under Art. 89(1) safeguards for archival, statistical,
+  scientific, or statutory purposes (e.g. tax records). Move out of live
+  systems, restrict access, document the lawful long-term purpose.
+
+## Local + sector law validation checklist
+
+Run this before adopting any schedule. A copied default can be unlawful in a
+given jurisdiction or sector.
+
+- [ ] Each period checked against the relevant **statutory minimum** (tax,
+      employment, healthcare records) for every country you operate in.
+- [ ] Each period checked against the relevant **maximum** / minimization duty —
+      keeping longer than necessary is itself a violation.
+- [ ] Sector rules applied (healthcare 10y DE to 20y FR; financial-services and
+      regulated industries carry their own keep-periods).
+- [ ] CPRA / US-state rules handled where applicable: disclose the retention
+      period (or criteria) at or before collection, and don't keep longer than
+      reasonably necessary for the disclosed purpose.
+- [ ] AI reuse stated explicitly (purpose limitation, Art. 5(1)(b)).
+- [ ] Backups and archives covered by the same periods as live data.
+- [ ] DPO / privacy counsel sign-off obtained before adoption.
+
+---
+
+**Not legal advice.** Retention periods are jurisdiction- and sector-specific. A
+qualified DPO or privacy counsel must validate this schedule before it is
+adopted.

package/skills/data-policy/scripts/verify.sh

@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+# verify.sh — data-policy
+# Read-only, offline, deterministic lint of a GENERATED governance artifact
+# (a finished retention schedule / ROPA / consent matrix / data policy).
+# Exits 0 on a clean OR empty/missing target (no false failure).
+# Non-zero only on a real violation. grep/awk only, no deps. bash 3.2-safe.
+#
+# Checks (per the plan):
+#   1. A retention/schedule table row must not be missing its period or lawful
+#      basis (empty or dash cell under a period/retention/basis column).
+#   2. Vague period language ("as long as necessary", "indefinitely", "forever",
+#      "permanently", "until no longer needed") must NOT be the only period —
+#      fail if present and no concrete period (number+unit, or "criteria") exists.
+#   3. A full policy must carry a disclaimer: "DPO" / "counsel" /
+#      "legal review" / "not legal advice".
+#   4. If the text mentions deletion it must also mention "backup" or "archive"
+#      (the enforcement gap).
+#
+# This lints FINISHED artifacts only. Blank-celled templates (a file whose
+# heading or text marks it a "template", or that carries empty `| |` fill-in
+# rows) are skipped — an unfilled template is not a failed policy.
+#
+# Usage: verify.sh [FILE_OR_DIR]   (default: current directory)
+#        verify.sh -               (read from stdin)
+
+set -uo pipefail
+
+TARGET="${1:-.}"
+
+TMP=""
+cleanup() { [ -n "$TMP" ] && rm -f "$TMP"; }
+trap cleanup EXIT
+
+# --- gather candidate files (or stdin), bash-3.2 safe ------------------------
+LIST="$(mktemp)"
+trap 'cleanup; rm -f "$LIST"' EXIT
+
+if [ "$TARGET" = "-" ]; then
+  TMP="$(mktemp)"
+  cat > "$TMP"
+  printf '%s\n' "$TMP" > "$LIST"
+elif [ -d "$TARGET" ]; then
+  find "$TARGET" -type f \( -name '*.md' -o -name '*.markdown' -o -name '*.txt' \) 2>/dev/null > "$LIST"
+elif [ -f "$TARGET" ]; then
+  printf '%s\n' "$TARGET" > "$LIST"
+else
+  echo "verify: target '$TARGET' not found; nothing to check."
+  exit 0
+fi
+
+if [ ! -s "$LIST" ]; then
+  echo "verify: no candidate files under '$TARGET'; clean by default."
+  exit 0
+fi
+
+fail=0
+checked=0
+note_fail() { echo "FAIL: $1"; fail=1; }
+
+VAGUE='as long as necessary|indefinitely|forever|permanently|until no longer needed'
+CONCRETE='[0-9]+[[:space:]]*(day|days|week|weeks|month|months|year|years|yr|yrs|mo)|criteria'
+DISCLAIMER='DPO|counsel|legal review|not legal advice'
+DELETION='delet|erase|purge'
+BACKUPS='backup|archive'
+ARTIFACT='retention|lawful basis|record of processing|ropa|consent matrix|storage limitation|expiry'
+
+while IFS= read -r f; do
+  [ -n "$f" ] || continue
+  [ -f "$f" ] || continue
+  label="$f"; [ "$f" = "$TMP" ] && label="<stdin>"
+
+  # Only lint files that look like a governance ARTIFACT, not prose docs that
+  # merely name retention. Require an artifact signal AND a markdown table
+  # (a line with two or more pipes) — schedules/ROPAs/consent matrices are tabular.
+  grep -qiE "$ARTIFACT" "$f" 2>/dev/null || continue
+  grep -qE '\|.*\|' "$f" 2>/dev/null || continue
+
+  # Skip unfilled TEMPLATES: a file that announces itself a template, or carries
+  # an empty markdown fill-in row like "|  |  |  |". Those legitimately have
+  # blank cells and are not a failed policy.
+  if grep -qiE 'template|fill-?able|fill-?in|copy-?ready' "$f" 2>/dev/null \
+     || grep -qE '^\|[[:space:]]*\|([[:space:]]*\|)+[[:space:]]*$' "$f" 2>/dev/null; then
+    continue
+  fi
+
+  checked=$((checked+1))
+
+  # --- Check 1: table rows missing period or lawful basis --------------------
+  awk -v FNAME="$label" '
+    function trim(s){ gsub(/^[ \t]+|[ \t]+$/,"",s); return s }
+    BEGIN{ FS="|"; rc=0 }
+    /\|/ && /[Pp]eriod|[Rr]etention|[Ll]awful basis|[Bb]asis/ {
+      delete col
+      for (i=1;i<=NF;i++){
+        c=tolower(trim($i))
+        if (c ~ /period|retention/ || c ~ /lawful basis|^basis$/){ col[i]=c }
+      }
+      n=0; for (k in col) n++
+      if (n>0){ inhdr=NR; next }
+    }
+    # an optional markdown separator row (---|---) right after the header
+    inhdr && NR==inhdr+1 && /^[ \t]*\|?[ \t:|-]+$/ { next }
+    inhdr && NR>inhdr {
+      if ($0 !~ /\|/){ inhdr=0; next }
+      for (i in col){
+        cell=trim($i)
+        if (cell=="" || cell ~ /^-+$/){
+          printf("FAIL: %s: table row %d missing %s cell\n", FNAME, NR, col[i]); rc=1
+        }
+      }
+    }
+    END{ exit rc }
+  ' "$f" || fail=1
+
+  # --- Check 2: vague period as the ONLY period ------------------------------
+  if grep -qiE "$VAGUE" "$f" 2>/dev/null; then
+    grep -qiE "$CONCRETE" "$f" 2>/dev/null \
+      || note_fail "$label: vague period with no concrete period or criteria"
+  fi
+
+  # --- Check 3: missing disclaimer -------------------------------------------
+  grep -qiE "$DISCLAIMER" "$f" 2>/dev/null \
+    || note_fail "$label: no DPO / counsel / legal-review / 'not legal advice' disclaimer"
+
+  # --- Check 4: deletion without backups/archives ----------------------------
+  if grep -qiE "$DELETION" "$f" 2>/dev/null; then
+    grep -qiE "$BACKUPS" "$f" 2>/dev/null \
+      || note_fail "$label: mentions deletion but never backups/archives (the enforcement gap)"
+  fi
+done < "$LIST"
+
+if [ "$fail" -ne 0 ]; then
+  echo "verify: violations found."
+  exit 1
+fi
+
+if [ "$checked" -eq 0 ]; then
+  echo "verify: no finished policy artifacts to lint (templates/non-artifacts only); clean by default."
+  exit 0
+fi
+
+echo "verify: OK — periods + bases present, no vague-only periods, disclaimer present, backups covered."
+exit 0

package/skills/data-scraper/SKILL.md

@@ -0,0 +1,130 @@
+---
+name: data-scraper
+description: "Use when data lives on a website with no usable API — listings, prices, public records — and the scrape must stay legal and not get blocked: legal gate, extraction path, durable selectors, pacing, resilience. NOT parsing bytes you already hold into fields (that is structured-extraction), NOT a documented API or key (that is api-connector-builder)."
+tags: [web-scraping, crawler, anti-bot, robots-txt, gdpr-compliance, playwright, rate-limiting]
+recommends: [api-connector-builder, structured-extraction, data-cleaning, gdpr-privacy, webhooks]
+origin: risco
+---
+
+# Data scraper
+
+You get bytes off websites you do not control — legally, and without getting blocked. You pick the cheapest extraction path that works, build selectors that survive a redesign, pace requests so the host neither bans nor sues you, and you write down the legal basis *before* the first request goes out.
+
+One rule above all the others: **scraping is the fallback, not the default.** It is what you reach for only when no API serves the data. If the site has a documented API or you hold a key, stop — that is `../api-connector-builder/SKILL.md`. And once you have the bytes, parsing them into fields is `../structured-extraction/SKILL.md`, normalizing the rows is `../data-cleaning/SKILL.md`. This skill ends the moment you hold the bytes.
+
+In 2025-2026 scrapers do not fail on parsing. They fail on a terms-of-service breach, on GDPR exposure, or on being blocked after hammering a host. So the work runs in this order: legal gate → extraction path → tool → selectors → politeness → resilience. Skipping the gate is how you end up in *Meta v. Bright Data*.
+
+## The legal gate — run this before any request
+
+Walk every item. Each ends in **proceed**, **proceed-narrowed**, or **stop**. One item at stop means the whole scrape stops until you resolve it. Depth and the case law are in `references/legal-compliance.md`.
+
+The gates below are the ones with real legal exposure — contract, data-protection law, and anti-circumvention. **robots.txt is not one of them**: it is a voluntary convention with no statutory force, so it informs the decision and never blocks it on its own.
+
+- [ ] **Is there an API?** If yes, you are in the wrong skill — never scrape what an API serves. → otherwise proceed.
+- [ ] **Did you read the ToS, and does login/auth apply?** Scraping is most exposed as *breach of contract* when you accepted terms — typically by logging in (*Meta v. Bright Data*, 2024). Logged-out public data weakens that claim. **Prefer logged-out public pages; never bypass auth.** → narrow to public, or stop.
+- [ ] **Is it personal data?** Names, emails, photos, reviews, IP addresses all count. Scraping public personal data for a *new* purpose — aggregation, resale, AI training — is a severe GDPR breach with fines into the tens of millions of EUR. You need a lawful basis (usually legitimate interest) and data minimization. Filter out special categories at the source. → proceed-narrowed (basis + minimization, see `../gdpr-privacy/SKILL.md`), or stop.
+- [ ] **robots.txt and ai.txt?** Read them — they are a *convention*, not law, so this item never stops a scrape on its own. A `Disallow` tells you the host would rather you did not, and `Crawl-delay` tells you its tolerance; both are useful intelligence about where you are likely to get blocked. Weigh it: honoring robots is the low-friction default and the cleanest evidence of good faith, but scraping your own site, one you have permission for, or public pages for research is legitimate whether or not robots allows it. → **advisory: note the decision and move on.**
+- [ ] **Are you about to bypass a control you were shown?** A CAPTCHA, a hard block, an enforced rate limit. *Reddit v. Perplexity AI* (2025) turns precisely on whether anti-bot measures were circumvented — a materially worse position than respectfully pacing public pages. Pacing public data is defensible; defeating a control is the frontier where you lose. → **stop. Do not solve the CAPTCHA. Do not bypass the block.**
+
+`hiQ v. LinkedIn` established that scraping *public* data is not automatically a CFAA violation — but that is the floor, not a license. Public + logged-out + no-personal-data is the defensible quadrant, and honoring robots keeps it tidy without being what makes it lawful. Anything outside it, document why and get a human to sign off.
+
+## Extraction-path decision table
+
+Walk *down* only when the rung above is unavailable. ~94% of modern sites are client-side rendered, yet most still ship machine-readable structured data — parse that before you launch a browser.
+
+| Path | Use when | Cost | Detectability |
+|------|----------|------|---------------|
+| **API** (incl. internal XHR/JSON endpoints) | Any documented API, or the page fetches its own JSON you can call directly | Lowest | Lowest — looks like normal traffic |
+| **sitemap.xml + JSON-LD** | `/sitemap.xml` lists the URLs; `<script type="application/ld+json">` carries the records (JSON-LD is Google's preferred structured format, so it is everywhere) | Low — one HTTP GET, parse JSON | Low |
+| **HTML selectors** | Data is in server-rendered HTML, no JSON-LD, no XHR JSON | Medium — selectors drift on redesign | Medium |
+| **Headless browser** | The data only exists after JS executes (SPA, lazy-load, infinite scroll) and there is no callable XHR endpoint | Highest — CPU, RAM, time, easiest to fingerprint | Highest |
+
+Before you reach for a browser, open DevTools → Network and look for the XHR/fetch that already returns JSON. Calling that endpoint directly is faster, stabler, and less detectable than rendering the whole page to read what the page itself fetched.
+
+## Tool picker
+
+| Site profile | Tool | Version (2026) | The one reason |
+|--------------|------|----------------|----------------|
+| Static HTML, no fingerprint wall | `httpx` + `selectolax` (or BeautifulSoup) | current | Fast, no browser; `selectolax` parses far quicker than `lxml` |
+| Static HTML but TLS/JA3 fingerprint blocks you | `curl_cffi` (profile `chrome131`) | current | Impersonates a real browser's TLS/JA3/HTTP2 fingerprint, not just the User-Agent |
+| JS-rendered SPA | Playwright | 1.60.0 (1.59 shipped 2026-04-01) | First-class async, auto-wait, the maintained headless standard |
+| Scalable, resilient, recurring crawler | Crawlee (JS or Python) | actively maintained 2026 | Wraps Playwright + proxy rotation + browserforge fingerprints + a disk-persisted `RequestQueue` that resumes after a crash |
+| Pure-Python static, legacy codebase | Scrapy | current | Battle-tested for static targets — but its Twisted core lags the asyncio ecosystem; pick Crawlee for new work |
+
+Default new builds to **Crawlee** when the job is recurring or must not break; reach for plain `httpx`/`curl_cffi` only when the target is static and one-shot.
+
+## Robust selectors
+
+Selectors break on redesign because they ride on layout, not meaning. Anchor on what is *semantically* stable — `data-*` attributes, ARIA roles, microdata, visible text — never on `nth-child` chains or generated CSS class hashes (`.css-1a2b3c`), which change on every build.
+
+```html
+<!-- the page you are scraping -->
+<article data-testid="listing-card">
+  <h2 class="css-1a2b3c">Acme Drill 9000</h2>
+  <span data-price="129.00">€129,00</span>
+</article>
+```
+
+```python
+# Bad — rides on layout and a build-generated hash; dies on the next deploy
+title = page.query_selector("div:nth-child(3) > article > .css-1a2b3c").inner_text()
+
+# Good — anchor on stable semantics, with a fallback chain, and fail loud
+def text_or_raise(card, selectors, field):
+    for sel in selectors:           # try each selector in priority order
+        el = card.query_selector(sel)
+        if el and el.inner_text().strip():
+            return el.inner_text().strip()
+    raise LookupError(f"required field {field!r} not found via {selectors}")
+
+card  = page.query_selector('[data-testid="listing-card"]')
+title = text_or_raise(card, ["h2", '[itemprop="name"]'], "title")
+price = text_or_raise(card, ["[data-price]", "span:has-text('€')"], "price")
+```
+
+Two rules baked into that snippet:
+
+- **Fallback selector chains.** Give every required field 2-3 ordered selectors. A single redesign rarely breaks all of them at once, so the scraper degrades instead of dying.
+- **Fail loud on a missing required field.** Raise — never silently write `null`. A silent null is a corrupted dataset you discover three months later. A raised error is a fix you make today.
+
+## Politeness and anti-block
+
+Concrete numbers beat "be respectful." Depth — fingerprint profiles, header sets, proxy taxonomy — is in `references/anti-bot.md`.
+
+- **Pace.** Start at **1 request per 1-3 seconds per host**, concurrency capped at **2-5 per host**. Tune *down* if you see 429s, never up to chase speed.
+- **Backoff with jitter.** On a transient error, exponential backoff *plus* random jitter — `delay = base * 2**attempt + random(0, base)`. Without jitter, every worker retries in lockstep and you self-DDoS the host.
+- **Honor 429 and `Retry-After`.** A 429 with `Retry-After: 30` means wait 30 seconds, not retry immediately. Ignoring it is the fastest route to an IP ban.
+- **Conditional GET.** Send `If-Modified-Since` / `If-None-Match` (ETag) on re-crawls. A `304 Not Modified` costs nothing and tells you the page is unchanged — cheaper for you, lighter on the host.
+- **Realistic headers + TLS fingerprint.** A bare `python-requests` User-Agent is an instant tell. Send a full, current browser header set; when JA3/TLS fingerprinting blocks you, switch to `curl_cffi` (`chrome131`) — see the references.
+- **Proxies only when justified.** Residential/mobile proxies are for geo-gating and IP-rate distribution on a target that *permits* the scrape — never to evade a block you were explicitly shown. Budget for them on large recurring jobs; do not reach for them to defeat a hard block (that is the legal line you do not cross).
+
+> **Resilience beats speed.** A scraper that runs 50% slower but never breaks is infinitely more valuable than a fast one that dies weekly. Pace for survival, not throughput.
+
+## Resilience
+
+A recurring crawler must survive crashes, redeploys, and the target's schema drift. Patterns and copy-paste starters are in `references/frameworks.md`.
+
+- **Resumable queue.** Use a disk-persisted request queue (Crawlee's `RequestQueue`) so a crash resumes from where it stopped, not from zero. Re-crawling 10k pages because the box rebooted is wasted budget and extra load on the host.
+- **Idempotent writes.** Upsert keyed on a stable id (the source URL or a record id), never blind append — so a re-run repairs rows instead of duplicating them.
+- **Checkpoint every N records.** Flush progress periodically so an interrupt loses minutes, not hours.
+- **Change detection by hash.** Store a content hash per record; only re-process when it changes. This pairs with conditional GET to keep recurring crawls cheap.
+- **Monitor and alert.** Alert on a spike in missing-field errors (schema drift — the site redesigned) and on a spike in 403/429 (you are being blocked). Both are silent-failure modes that rot a dataset until someone checks.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Scraping behind a login, then calling it "public data" | Accepting ToS at login is the breach-of-contract hook (*Meta v. Bright Data*) | Stay logged-out on public pages; never bypass auth |
+| Not even reading robots.txt / ai.txt | You lose free intelligence on where you will get blocked, and any good-faith story later | Parse both; honor by default, override deliberately and write down why |
+| No delay, unbounded concurrency | Hammers the host → IP ban, possible CFAA-style exposure | 1 req / 1-3s, cap 2-5 concurrent per host |
+| Selectors on `nth-child` / `.css-1a2b3c` hashes | Break on the next deploy; silent data loss | Anchor on `data-*` / semantic / text, with fallbacks |
+| Silently writing `null` on a missing field | Corrupts the dataset; discovered months later | Raise on a missing *required* field — fail loud |
+| Solving a CAPTCHA / bypassing a hard block | Circumventing a shown control (*Reddit v. Perplexity*) — the worst legal posture | Stop. That control is a "no." |
+| Scraping personal data with no lawful basis | GDPR fines into tens of millions EUR | Establish basis + minimize + filter special categories (`../gdpr-privacy/SKILL.md`) |
+| Storing everything "just in case" | Defeats minimization; expands breach blast radius | Keep only fields the purpose needs; set retention |
+| Launching a browser when JSON-LD was right there | Slowest, most detectable, most expensive path | Check XHR/JSON-LD/sitemap first; browser is last |
+| No resume — a crash restarts from zero | Wastes budget, doubles load on the host | Disk-persisted resumable queue |
+| Hardcoding one User-Agent forever | Stale UA is an easy bot tell | Current full header set; rotate when justified |
+| Retrying with no backoff | Lockstep retries self-DDoS the host | Exponential backoff + jitter; honor `Retry-After` |
+
+When in doubt about whether a scrape is defensible, the answer is the gate. Run it, write down the outcome, and only then send a request.

package/skills/data-scraper/evals/README.md

@@ -0,0 +1,3 @@
+# Evals: data-scraper
+
+These cases are routing and coverage checks, not an automated pass/fail suite. Read `cases.yaml` against `SKILL.md`. For each `should_trigger` prompt, confirm the description and body would plausibly fire this skill rather than a sibling — note the two non-obvious symptom cases (429/IP-ban and selectors-break-on-redesign, which read as ops problems but are core here) and the Spanish and Catalan triggers. For each `should_not_trigger` prompt, confirm the named `route_to` sibling is the better home and that the boundary in the description would defer to it (structured-extraction once you hold bytes, api-connector-builder when an API exists, data-cleaning for normalization, e2e-testing for UI flow automation, gdpr-privacy for a pure lawful-basis question). For the `capability` case, have a human or a harness LLM produce the response and check it hits every `must_include` item — above all that it runs the legal gate first, walks the extraction path before reaching for a browser, paces per host with backoff and honors 429, uses a resumable queue, fails loud on missing fields, and refuses to bypass a CAPTCHA or hard block. There is no scripted assertion here. The separate `scripts/verify.sh` is the only automated check, and it statically lints an emitted scraper file for a compliance preflight, a rate limit/concurrency cap, retry/backoff, and the absence of CAPTCHA-solver / auth-bypass calls — not these cases.

package/skills/data-scraper/evals/cases.yaml

@@ -0,0 +1,46 @@
+skill: data-scraper
+
+should_trigger:
+  - prompt: "I need to pull all product prices off this competitor's catalog site — they have no API."
+    why: Core case — structured data off a site with no API, the canonical scrape.
+  - prompt: "Build me a crawler that grabs new public tender listings every morning."
+    why: Recurring resilient crawler — routes to the resilience + tool-picker sections.
+  - prompt: "My scraper keeps getting 429s and then the IP gets banned after about 200 pages."
+    why: Non-obvious — the symptom is a politeness/anti-block failure, not a parsing one.
+  - prompt: "The CSS selectors in my scraper break every time the site redesigns."
+    why: Non-obvious — points at the robust-selectors section (data-attr / semantic / fallback chains).
+  - prompt: "Is it legal for me to scrape this site, and how do I do it without getting sued?"
+    why: Legal-gate first — ToS, robots.txt, personal-data, circumvention before any request.
+  - prompt: "Vull extreure les dades de contacte d'un directori d'empreses públic."
+    why: Catalan trigger — public directory scrape that also nudges the GDPR personal-data gate.
+  - prompt: "Necesito scrapear una página de anuncios sin que me bloqueen."
+    why: Spanish trigger — extraction plus the anti-block concern.
+
+should_not_trigger:
+  - prompt: "Parse these 500 saved HTML invoices into a fields table."
+    route_to: structured-extraction
+    why: Bytes are already in hand; the task is field extraction from documents, not fetching.
+  - prompt: "The site publishes a documented REST API with a key — pull the orders into our DB."
+    route_to: api-connector-builder
+    why: A real API exists; never scrape what an API serves.
+  - prompt: "Dedup and normalize this messy scraped CSV — fix the types and nulls."
+    route_to: data-cleaning
+    why: Downstream of having the rows; normalization, not extraction.
+  - prompt: "Automate clicking through our checkout flow to test it end to end."
+    route_to: e2e-testing
+    why: Browser UI automation for testing a flow, not data extraction.
+  - prompt: "What's my GDPR lawful basis for processing this customer list we already hold?"
+    route_to: gdpr-privacy
+    why: Pure legal-basis question with no scrape — the privacy artifact layer.
+
+capability:
+  - scenario: "Stand up a resilient daily crawler for a public, JS-rendered listings site that has no API. It contains business names and contact emails. Tell me how to build it so it stays legal and doesn't get blocked."
+    must_include:
+      - Runs the legal gate before any request — reads ToS, checks robots.txt/ai.txt, and flags the contact emails as personal data needing a lawful basis + minimization (defers basis detail to gdpr-privacy)
+      - Walks the extraction path — checks for a callable XHR/JSON endpoint, sitemap.xml, and JSON-LD before committing to a headless browser
+      - Picks a tool with a reason — Playwright (1.60) or Crawlee for the JS-rendered SPA, not plain httpx
+      - Builds selectors on stable attributes (data-* / semantic / text) with a fallback chain, and fails loud on a missing required field rather than writing null
+      - Sets a per-host rate limit (≈1 req / 1-3s, concurrency cap 2-5), exponential backoff with jitter, and honors 429 / Retry-After
+      - Uses a resumable disk-persisted queue (Crawlee RequestQueue) and idempotent upserts keyed on a stable id so a crash resumes instead of restarting
+      - Refuses to bypass any CAPTCHA or hard block, treating it as a stop and re-running the gate
+      - Adds monitoring/alerts on missing-field spikes (schema drift) and 403/429 spikes (blocking)

package/skills/data-scraper/references/anti-bot.md

@@ -0,0 +1,85 @@
+# Anti-bot reference
+
+Anti-bot handling is mandatory in 2026, not optional — basic Puppeteer worked in 2024, but every major site now ships detection. This covers the *legitimate* techniques: realistic fingerprints and pacing on a scrape you are entitled to run. It does **not** cover defeating a control you were shown — that is the legal line from `legal-compliance.md`, and you do not cross it.
+
+## TLS / JA3 fingerprinting with curl_cffi
+
+Many blocks key on the TLS/JA3/HTTP2 fingerprint, not the User-Agent. A real Chrome handshake looks different from a Python `requests` handshake regardless of headers, so spoofing the UA alone fails. `curl_cffi` impersonates a real browser's full fingerprint.
+
+```python
+from curl_cffi import requests
+
+# impersonate="chrome131" sends Chrome's real TLS/JA3 + HTTP2 fingerprint
+r = requests.get(
+    "https://example.com/listings",
+    impersonate="chrome131",
+    timeout=20,
+)
+r.raise_for_status()
+html = r.text
+```
+
+Use this the moment a static target returns 403 to `httpx` but 200 in a real browser — that gap is almost always fingerprinting.
+
+## Realistic header set
+
+Send a full, current browser header set, not a lone User-Agent. A bare `python-requests/2.x` UA is an instant tell.
+
+```python
+HEADERS = {
+    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+                  "AppleWebKit/537.36 (KHTML, like Gecko) "
+                  "Chrome/131.0.0.0 Safari/537.36",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.9",
+    "Accept-Encoding": "gzip, deflate, br",
+    "Sec-Fetch-Dest": "document",
+    "Sec-Fetch-Mode": "navigate",
+    "Sec-Fetch-Site": "none",
+    "Upgrade-Insecure-Requests": "1",
+}
+```
+
+Keep the UA current — a stale Chrome version is its own tell. Refresh it when the major version moves.
+
+## Playwright launch config
+
+```python
+from playwright.async_api import async_playwright
+
+async def fetch(url: str) -> str:
+    async with async_playwright() as p:
+        browser = await p.chromium.launch(headless=True)
+        ctx = await browser.new_context(
+            user_agent=HEADERS["User-Agent"],
+            locale="en-US",
+            viewport={"width": 1366, "height": 768},
+        )
+        page = await ctx.new_page()
+        await page.goto(url, wait_until="networkidle", timeout=30_000)
+        html = await page.content()
+        await browser.close()
+        return html
+```
+
+For scale, prefer Crawlee's `AdaptivePlaywrightCrawler` with browserforge fingerprints over hand-rolled stealth — it picks the cheapest mode (HTTP vs browser) per request and ships coherent fingerprints. See `frameworks.md`.
+
+## Proxy taxonomy — when each is justified
+
+Proxies distribute IP load and resolve geo-gating on a scrape you are entitled to run. They are not for evading a block you were explicitly shown.
+
+| Type | Cost | Justified when |
+|------|------|----------------|
+| Datacenter | Lowest | High-volume scrape of a permissive target; easiest to detect/block |
+| Residential | Medium | Geo-gated content; target blocks datacenter ranges but permits the scrape |
+| Mobile | Highest | Mobile-only content or the strictest IP reputation requirements |
+
+Rule: reach for a proxy to spread load or reach a geo, never to defeat a hard block. The latter is circumvention.
+
+## CAPTCHA — the line you do not cross
+
+A CAPTCHA is the site explicitly saying "prove you are human or do not proceed." Routing it to a solver service to keep scraping is circumventing a shown access control — exactly the posture at issue in *Reddit v. Perplexity AI* (2025), and the worst place to be legally.
+
+When you hit a CAPTCHA or a hard block: **stop.** Re-run the legal gate. Slow down, narrow the path set, or seek permission/an API. Do not solve it.
+
+Sources: brightdata.com "Web Scraping With curl_cffi 2026"; curl-cffi.readthedocs.io; dataresearchtools.com TLS mimicry 2026; crawlee.dev/python docs. All accessed 2026-06-02.