@damiangilgonzalez/harness 1.3.7

package/skills/data-cleaning/evals/cases.yaml

@@ -0,0 +1,57 @@
+skill: data-cleaning
+
+should_trigger:
+  - prompt: "Clean and dedupe this customer CSV and give me something I can re-run every month with the same result."
+    why: Core ask — reproducible cleaning pipeline plus dedupe; the headline use case.
+  - prompt: "My integer ID column came in as 1001.0 floats and a bunch of rows are duplicated. Fix it."
+    why: Non-obvious dtype footgun — only this skill explains read-typing (dtype_backend) before deterministic dedupe.
+  - prompt: "Normalize these country values — USA, U.S., United States all mean the same thing."
+    why: Category normalization via a mapping table; the unmapped→quarantine pattern is this skill's.
+  - prompt: "Add a validation gate so bad rows fail loudly before they ever reach my model."
+    why: Schema-contract / pandera framing — declare types, ranges, allowed values; fail loud.
+  - prompt: "Quarantine the rows that don't pass instead of silently dropping them."
+    why: Non-obvious quarantine pattern unique to this skill; not a generic cleaning ask.
+  - prompt: "Limpiar y deduplicar este dataset y dejar un pipeline repetible."
+    why: Spanish trigger — same reproducible clean+dedupe intent, no English keyword present.
+  - prompt: "Why do my dates parse wrong sometimes and a few rows look duplicated after import?"
+    why: Symptom-led, never says 'clean' — ambiguous date format + dedupe both live here.
+
+should_not_trigger:
+  - prompt: "Export this cleaned DataFrame to Excel with a summary chart and conditional formatting."
+    route_to: spreadsheet-ops
+    why: Output is a styled sheet/formulas/chart; cleaning already ended at the clean table.
+  - prompt: "Scrape all the product prices off this catalog website."
+    route_to: data-scraper
+    why: Acquisition — the rows aren't held yet; cleaning only starts once you have tabular rows.
+  - prompt: "Pull the line items out of these invoice PDFs into a table."
+    route_to: structured-extraction
+    why: Parsing fields out of unstructured blobs into rows, not cleaning rows you already hold.
+  - prompt: "Aggregate this 5GB CSV with SQL fast on my laptop."
+    route_to: duckdb
+    why: OLAP speed over multi-GB files is the bottleneck, not correctness/normalization.
+  - prompt: "Build the KPI dashboard from this already-clean table."
+    route_to: business-intelligence
+    why: Consumes the clean output; the boundary is the validated table.
+  - prompt: "Forecast next quarter's revenue from this monthly clean series."
+    route_to: forecasting
+    why: Modeling on top of clean data, not producing the clean data.
+
+capability:
+  - scenario: >
+      Given a messy customers CSV — an id column read as floats with .0, "N/A" and 999 sentinels in age,
+      duplicate rows per customer, country values USA / U.S. / United States, and malformed dates — produce
+      a reproducible clean() pipeline plus a pandera schema that types, normalizes, dedupes deterministically,
+      validates, and quarantines failures rather than dropping them.
+    must_include:
+      - Reads with explicit dtype control (dtype_backend="pyarrow" or numpy_nullable + na_values) so ids stay int, not float
+      - Profiles first (nulls %, dup count, per-column dtype, cardinality) to drive the plan
+      - Normalizes strings with strip/casefold/NFKC before dedupe
+      - Maps categories with a mapping table (dict), unmapped → NA/quarantine, not a regex pile
+      - Turns sentinels (-1/999/"N/A") into NA, with an explicit clip-vs-reject range decision
+      - Parses dates with an explicit format and errors="coerce", then checks the coerced-NaT count
+      - Deduplicates deterministically — explicit subset key + stable sort_values + keep
+      - Handles missing per column; no bare fillna(0); impute adds a _was_missing flag
+      - Defines a pandera DataFrameModel with coerce, nullable, Field checks (isin/ge/le/unique), and lazy=True
+      - Quarantines failing rows to an artifact instead of silently dropping them
+      - Writes to a NEW output artifact, leaving the raw source untouched
+      - Logs a row-count in/out/coerced/quarantined diff and pins library versions

package/skills/data-cleaning/references/normalization-recipes.md

@@ -0,0 +1,136 @@
+# Normalization recipes — copy-paste
+
+Each recipe is deterministic and re-runnable. Drop them into your `normalize(df)` step. Polars equivalents
+at the end for when pandas runs out of RAM.
+
+## Category mapping (unmapped → quarantine, never silent pass)
+
+A dict is auditable; a regex tower is not. Anything not in the map becomes NA so the schema gate catches it.
+
+```python
+COUNTRY = {
+    "usa": "US", "u.s.": "US", "u.s.a.": "US", "united states": "US", "us": "US",
+    "españa": "ES", "espana": "ES", "spain": "ES", "es": "ES",
+    "france": "FR", "fr": "FR",
+}
+
+def map_category(s, mapping):
+    key = s.str.strip().str.casefold().str.normalize("NFKC")
+    mapped = key.map(mapping)
+    unmapped = key.notna() & mapped.isna()
+    if unmapped.any():
+        # surface the new values so you extend the map deliberately, not silently
+        print("UNMAPPED categories:", sorted(key[unmapped].unique()))
+    return mapped     # unmapped rows are NA -> the schema's isin/nullable=False check rejects them
+
+df["country"] = map_category(df["country"], COUNTRY)
+```
+
+## Robust date parsing (explicit format, count the casualties)
+
+```python
+def parse_dates(s, fmt="%Y-%m-%d", utc=True):
+    parsed = pd.to_datetime(s, format=fmt, utc=utc, errors="coerce")
+    failed = parsed.isna() & s.notna()        # coerced to NaT but original wasn't null = parse failure
+    if failed.any():
+        raise ValueError(f"{failed.sum()} values did not match {fmt!r}: {s[failed].head().tolist()}")
+    return parsed
+```
+
+If a single source genuinely mixes formats, parse each known format and `combine_first`, then assert no NaT
+remains. Never fall back to dayfirst inference — `03/04/2026` is ambiguous and pandas picks silently.
+
+```python
+def parse_mixed(s, formats=("%Y-%m-%d", "%d/%m/%Y")):
+    out = pd.Series(pd.NaT, index=s.index, dtype="datetime64[ns, UTC]")
+    for fmt in formats:
+        try_ = pd.to_datetime(s, format=fmt, utc=True, errors="coerce")
+        out = out.fillna(try_)
+    remaining = out.isna() & s.notna()
+    assert not remaining.any(), f"{remaining.sum()} dates matched none of {formats}"
+    return out
+```
+
+## Unicode / encoding repair
+
+```python
+# 1. Read with the RIGHT encoding (don't let pandas/locale guess). If you see mojibake like "Rodríguez",
+#    the file is utf-8 misread as latin-1 — re-read as utf-8.
+df = pd.read_csv("raw.csv", encoding="utf-8")
+
+# 2. Normalize text: NFKC folds full-width/compatibility forms; strip control chars and zero-width junk.
+import re
+ZERO_WIDTH = re.compile(r"[​‌‍]")
+def clean_text(s):
+    return (s.str.normalize("NFKC")
+             .str.replace(ZERO_WIDTH, "", regex=True)
+             .str.strip())
+
+# 3. Last-resort repair of already-corrupted text (needs `pip install ftfy`):
+#    from ftfy import fix_text; df["name"] = df["name"].map(fix_text)
+```
+
+## Sentinel → NA
+
+Map the per-column sentinels to real NA *before* any numeric check, so means and ranges aren't poisoned.
+
+```python
+SENTINELS = {
+    "age":     [-1, 999],
+    "income":  [-1, 0, 999999],     # 0 is a sentinel here only if 0 income is impossible in this dataset
+    "rating":  [-1],
+}
+for col, vals in SENTINELS.items():
+    df[col] = df[col].mask(df[col].isin(vals))     # -> NA; decide drop/impute/quarantine in handle_missing
+```
+
+## Numeric range — clip vs reject
+
+```python
+# CLIP: out-of-range is plausibly a recording cap → pull to the bound (keeps the row).
+df["age"] = df["age"].clip(lower=0, upper=120)
+
+# REJECT: out-of-range is impossible → set NA so the gate quarantines the row (don't fabricate a value).
+df["age"] = df["age"].mask(~df["age"].between(0, 120))
+```
+
+Decide per column and write the choice down — clipping a fraud-amount silently hides outliers; rejecting a
+sensor reading silently loses data. The right call depends on the field's meaning.
+
+## Polars equivalents (clean-at-scale)
+
+Same profile→normalize→dedupe→validate shape, lazy and parallel. Pin Polars and stay deterministic.
+
+```python
+import polars as pl
+
+lf = pl.scan_csv("raw.csv", infer_schema_length=10_000)     # lazy: nothing reads until .collect()
+
+clean = (
+    lf
+    # strings: strip + lowercase + (NFKC via map if needed)
+    .with_columns(pl.col("country").str.strip_chars().str.to_lowercase().alias("country"))
+    # category mapping table
+    .with_columns(pl.col("country").replace_strict(COUNTRY, default=None))
+    # sentinels -> null
+    .with_columns(pl.when(pl.col("age").is_in([-1, 999])).then(None).otherwise(pl.col("age")).alias("age"))
+    # dates: explicit format, strict=False coerces failures to null (then check the count)
+    .with_columns(pl.col("signup").str.to_datetime("%Y-%m-%d", strict=False, time_zone="UTC"))
+    # missing: drop rows missing the required key
+    .drop_nulls(subset=["customer_id"])
+    # dedupe: deterministic — sort then keep first per key
+    .sort(["customer_id", "updated_at"], descending=[False, True])
+    .unique(subset=["customer_id"], keep="first")
+    .collect()                                              # execute the lazy plan
+)
+
+# fill_null deliberately (with a flag column), never a blind 0:
+clean = clean.with_columns(
+    pl.col("income").is_null().alias("income_was_missing"),
+    pl.col("income").fill_null(strategy="mean"),
+)
+```
+
+pandera validates Polars frames too (`import pandera.polars as pa`), so the gate from
+[validation-patterns.md](validation-patterns.md) carries over. Hand multi-GB analytical SQL to
+[`duckdb`](../../duckdb/SKILL.md) — the engine changes, the discipline does not.

package/skills/data-cleaning/references/validation-patterns.md

@@ -0,0 +1,134 @@
+# Validation patterns — pandera, GX Core, dbt
+
+The gate is what makes cleaned data *trustworthy*: it asserts the contract holds and fails loud when next
+month's file drifts. Pick the lightest tool that covers your blast radius.
+
+## Which validator
+
+| Situation | Use | Why |
+| --- | --- | --- |
+| One pipeline, validate a DataFrame in Python | **pandera** | Lightweight, in-process, lives next to `clean()`; pin `pandera==0.31.1` (supports pandas ≥ 3) |
+| Many datasets/teams, shared data-quality platform, results store + docs | **GX Core 1.0** | Heavier; Data Context / Suites / Checkpoints are infrastructure, overkill for one script |
+| Cleaning lives in a SQL warehouse, not Python | **dbt** model contracts + tests | Contract enforced at build; tests run post-materialization |
+
+Default to pandera. Escalate only when the heavier tool's features are actually needed.
+
+## Pandera — the full DataFrameModel
+
+```python
+import pandera.pandas as pa
+from pandera.typing import Series
+
+class CustomerSchema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(unique=True, ge=1)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])
+    age:         Series[float] = pa.Field(ge=0, le=120, nullable=True)
+    email:       Series[str]   = pa.Field(str_matches=r"^[^@\s]+@[^@\s]+\.[^@\s]+$", nullable=True)
+    signup:      Series[pa.DateTime] = pa.Field(nullable=False)
+
+    class Config:
+        strict = True        # reject unexpected columns — schema drift fails the build
+        coerce = True        # coerce to declared dtype; raise if a value can't be coerced
+        ordered = False      # set True if column order is part of the contract
+
+    @pa.check("email", name="lowercase_email")
+    def email_is_lower(cls, s: Series[str]) -> Series[bool]:
+        # custom row-wise check: returns a boolean Series, one per row
+        return s.isna() | (s == s.str.lower())
+
+    @pa.dataframe_check
+    def row_budget(cls, df) -> bool:
+        # whole-frame invariant: catch a truncated/exploded extract
+        return 1 <= len(df) <= 5_000_000
+```
+
+`coerce` fixes types the contract expects. `nullable` declares which columns may hold NA. Field checks
+(`ge`, `le`, `isin`, `unique`, `str_matches`) are allowed-value rules. `@pa.check` is a custom column
+check; `@pa.dataframe_check` is a whole-frame invariant.
+
+## Lazy errors + the quarantine split
+
+`lazy=True` collects **every** violation across the frame instead of dying on the first. Use it to split
+valid rows from failing rows — the failures become the quarantine, never a silent drop.
+
+```python
+def validate_and_split(df, schema):
+    """Return (valid_df, quarantine_df). Quarantine is kept, never discarded."""
+    try:
+        valid = schema.validate(df, lazy=True)
+        return valid, df.iloc[0:0].copy()          # empty quarantine — everything passed
+    except pa.errors.SchemaErrors as e:
+        # failure_cases columns: schema_context, column, check, failure_case, index
+        bad_idx = e.failure_cases["index"].dropna().unique()
+        quarantine = df.loc[df.index.isin(bad_idx)].copy()
+        quarantine["_failed_checks"] = (
+            e.failure_cases.dropna(subset=["index"])
+             .groupby("index")["check"].agg(lambda c: "; ".join(map(str, c)))
+        )
+        valid = df.loc[~df.index.isin(bad_idx)]
+        return valid, quarantine
+
+valid, quarantine = validate_and_split(df, CustomerSchema)
+quarantine.to_parquet("quarantine.parquet")        # someone investigates these
+print(f"valid={len(valid)} quarantined={len(quarantine)}")   # the audit diff
+```
+
+Validate **output** always. Validate **input expectations** too when you can state them (e.g. raw must have
+these columns before you start) — it turns a confusing mid-pipeline crash into a clear "the source changed".
+
+## GX Core 1.0 — checkpoint sketch
+
+GX rebranded its OSS to **GX Core** and shipped 1.0 GA with a simplified API. Reach for it when validation
+is a shared platform across many datasets, not a single script.
+
+```python
+import great_expectations as gx
+
+context = gx.get_context()                                   # Data Context
+ds  = context.data_sources.add_pandas("customers_src")       # Data Source
+asset = ds.add_dataframe_asset(name="customers")
+batch = asset.add_batch_definition_whole_dataframe("batch")
+
+suite = context.suites.add(gx.ExpectationSuite(name="customers_suite"))
+suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(column="age", min_value=0, max_value=120))
+suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(column="country", value_set=["US", "ES", "FR"]))
+
+vd = context.validation_definitions.add(
+    gx.ValidationDefinition(name="customers_vd", data=batch, suite=suite)
+)
+checkpoint = context.checkpoints.add(gx.Checkpoint(name="customers_cp", validation_definitions=[vd]))
+result = checkpoint.run(batch_parameters={"dataframe": df})
+assert result.success, "GX checkpoint failed — see Data Docs"
+```
+
+The flow is always **Data Context → Data Source → Expectation Suite → Validation Definition → Checkpoint**.
+You get a results store and generated Data Docs — worth the weight only when many people consume them.
+
+## dbt — contracts vs tests (cleaning in the warehouse)
+
+When cleaning is SQL in a warehouse, the contract lives in the model's YAML. **Contracts** are enforced at
+*build* (the model fails to build if the output is off-contract). **Tests** validate *after*
+materialization.
+
+```yaml
+# models/staging/_stg_customers.yml
+models:
+  - name: stg_customers
+    config:
+      contract: { enforced: true }          # build fails if columns/types drift — the contract
+    columns:
+      - name: customer_id
+        data_type: bigint
+        constraints: [{ type: not_null }, { type: unique }]
+      - name: country
+        data_type: varchar
+        data_tests:                          # validated post-materialization — the tests
+          - accepted_values: { values: ["US", "ES", "FR"] }
+      - name: age
+        data_type: integer
+        data_tests:
+          - dbt_utils.accepted_range: { min_value: 0, max_value: 120 }
+```
+
+Contract = "this shape is guaranteed at build"; tests = "these values held after the run". Use both: the
+contract stops schema drift, the tests catch data drift.

package/skills/data-cleaning/scripts/verify.sh

@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# verify.sh — data-cleaning skill check. Run from anywhere. Read-only by default.
+#
+# Two layers:
+#   STATIC (always): SKILL.md exists, frontmatter has the required keys, description is one
+#     physical line, and both references/ files exist and are non-empty.
+#   RUNTIME (only if pandas + pandera importable): build a tiny fixture with ONE clearly-good
+#     row and ONE clearly-bad row (negative age / unmapped country), run a pandera schema with
+#     lazy=True, and assert the good row PASSES while the bad row is FLAGGED — proving the gate
+#     is not a no-op. If pandas/pandera are absent, print SKIP and still pass the static checks.
+#
+# Writes only inside a private mktemp dir, removed on exit. No network.
+# Exit: 0 on pass or skip; non-zero ONLY on a real failure (missing artifact, malformed
+# frontmatter, or a gate that fails to reject a known-bad row).
+
+YELLOW=$'\033[33m'; GREEN=$'\033[32m'; RED=$'\033[31m'; NC=$'\033[0m'
+skip() { printf '%s[skip]%s %s\n' "$YELLOW" "$NC" "$*"; }
+ok()   { printf '%s[ok]%s %s\n'   "$GREEN"  "$NC" "$*"; }
+err()  { printf '%s[fail]%s %s\n' "$RED"    "$NC" "$*"; }
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SKILL_DIR="$(dirname "$SCRIPT_DIR")"
+SKILL="$SKILL_DIR/SKILL.md"
+fail=0
+
+# ---- STATIC CHECKS (always run) ----
+if [ ! -f "$SKILL" ]; then
+  err "SKILL.md not found at $SKILL"; exit 1
+fi
+ok "SKILL.md present"
+
+for key in "name:" "description:" "tags:" "recommends:" "origin:"; do
+  if grep -q "^${key}" "$SKILL"; then
+    ok "frontmatter has ${key%:}"
+  else
+    err "frontmatter missing ${key%:}"; fail=1
+  fi
+done
+
+# description must be exactly one physical line
+desc_lines="$(grep -c '^description:' "$SKILL" || true)"
+if [ "$desc_lines" = "1" ]; then
+  ok "description present as a single key line"
+else
+  err "expected exactly one 'description:' line, found $desc_lines"; fail=1
+fi
+
+for ref in "references/validation-patterns.md" "references/normalization-recipes.md"; do
+  if [ -s "$SKILL_DIR/$ref" ]; then
+    ok "$ref present and non-empty"
+  else
+    err "$ref missing or empty"; fail=1
+  fi
+done
+
+if [ "$fail" -ne 0 ]; then
+  err "static checks failed"; exit 1
+fi
+
+# ---- RUNTIME GATE CHECK (only if deps present) ----
+if ! command -v python3 >/dev/null 2>&1; then
+  skip "python3 not found — ran static checks only"; exit 0
+fi
+if ! python3 -c "import pandas, pandera" >/dev/null 2>&1; then
+  skip "pandas/pandera not importable — ran static checks only"; exit 0
+fi
+
+TMP="$(mktemp -d 2>/dev/null || mktemp -d -t dc_verify)"
+trap 'rm -rf "$TMP"' EXIT
+
+python3 - <<'PY'
+import sys
+import pandas as pd
+import pandera.pandas as pa
+from pandera.typing import Series
+
+# Fixture: row 1 is clearly GOOD; row 2 is clearly BAD (negative age + unmapped country -> NA).
+COUNTRY = {"usa": "US", "u.s.": "US", "united states": "US", "es": "ES"}
+raw = pd.DataFrame({
+    "customer_id": [1, 2],
+    "country":     ["U.S.", "Atlantis"],   # row2 unmapped -> NA -> fails isin
+    "age":         [42, -5],               # row2 out of range -> fails ge(0)
+})
+raw["country"] = raw["country"].str.strip().str.casefold().map(COUNTRY)
+
+class Schema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(ge=1, unique=True)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])
+    age:         Series[float] = pa.Field(ge=0, le=120)
+    class Config:
+        coerce = True
+        strict = True
+
+try:
+    Schema.validate(raw, lazy=True)
+    print("GATE-NOOP: schema accepted a known-bad row")  # gate did nothing -> failure
+    sys.exit(1)
+except pa.errors.SchemaErrors as e:
+    bad_idx = set(e.failure_cases["index"].dropna().astype(int).unique())
+    # The bad row (index 1) must be flagged; the good row (index 0) must NOT be.
+    if 1 in bad_idx and 0 not in bad_idx:
+        sys.exit(0)
+    print(f"GATE-WRONG: flagged indices were {sorted(bad_idx)}; expected {{1}}")
+    sys.exit(1)
+PY
+rc=$?
+if [ "$rc" -eq 0 ]; then
+  ok "validation gate rejected the bad row and passed the good row (gate is not a no-op)"
+  exit 0
+else
+  err "validation gate did not behave as documented"
+  exit 1
+fi

package/skills/data-policy/SKILL.md

@@ -0,0 +1,150 @@
+---
+name: data-policy
+description: "Use when building internal data-governance machinery: a retention schedule (period, lawful basis, expiry action, system where deletion runs), an Art. 6 lawful-basis register, an Art. 30 ROPA, or a consent capture/withdrawal model. NOT the public privacy notice or DSAR handling (that is `gdpr-privacy`), NOT SOC 2 posture (that is `compliance`)."
+tags: [data-governance, retention, gdpr, ropa, consent, lawful-basis, privacy, data-minimization]
+recommends: [gdpr-privacy, compliance, contracts, secure-coding, db-migrations, postgresdb]
+origin: risco
+---
+
+# Data policy
+
+You produce the structured governance artifacts engineering and ops implement — a retention schedule, a lawful-basis register, a Record of Processing Activities (ROPA), a consent model — not the public-facing notice users read (that is `../gdpr-privacy/SKILL.md`). You are not a DPO and you never claim to be one.
+
+**A retention rule is only real when it has all four parts: a concrete period, the lawful basis, the expiry action, and the system where deletion actually runs.** A policy that names a period but never deletes anything is a paper policy — and a paper policy is precisely what regulators fine. Cumulative GDPR fines hit ~EUR 5.65B across ~2,245 actions by March 2025, and the two failures that recur are *no systematic data classification* and *no automated deletion capability* (Secure Privacy / CMS Enforcement Tracker, 2025). Every schedule you emit ends with the DPO/counsel sign-off boundary below.
+
+## First move: which artifact does the operator need?
+
+Map the request to one artifact before writing anything. Each routes to a section.
+
+| Operator says | Artifact | Go to |
+|---|---|---|
+| "How long do we keep X / write our retention policy" | Retention schedule | Build the retention schedule |
+| "Is our basis consent or legitimate interest?" | Lawful-basis register | Pick the lawful basis |
+| "Set up a ROPA / Article 30 record" | ROPA row | The ROPA |
+| "Design consent capture / withdrawal" | Consent matrix | Consent model |
+| "Auto-delete but keep legal holds / backups still have data" | Deletion workflow | Make it real in systems |
+
+If they want the public privacy notice, DPA clauses, or SOC 2 readiness instead, stop and route them — see the boundary below.
+
+## Build the retention schedule
+
+This is the core artifact. For every category of personal data, walk five columns in order: **data category -> purpose -> lawful basis -> retention period -> expiry action -> system of record**. GDPR's storage-limitation principle (Art. 5(1)(e)) requires data be held in identifiable form *no longer than necessary for the purpose it was collected for*; GDPR sets no fixed periods — duration is driven by purpose plus sector law (gdpr-info.eu Art. 5; Usercentrics, 2026).
+
+The expiry action is one of three, and you must pick one explicitly:
+
+- **delete** — the row is gone.
+- **anonymize** — identifiers stripped so the record is no longer personal data (then storage limitation no longer bites); valid only if re-identification is genuinely infeasible.
+- **archive** — kept under Art. 89(1) safeguards for a lawful long-term purpose (statutory, archival, statistical).
+
+Worked example. The Bad version is what gets fined; the Good version is enforceable.
+
+```text
+Bad:  Customer data — kept as long as necessary.
+Good: | Category        | Purpose        | Lawful basis      | Period                  | Expiry   | System of record       |
+      | Customer orders | fulfil + tax   | Art. 6(1)(b) +(c) | 36 mo after last order  | anonymize| Postgres `orders` + DWH|
+```
+
+Working default periods — **starting points, never asserted as universally lawful; validate against local + sector law** (Usercentrics; Secure Privacy, 2026):
+
+| Category | Common default | Basis it usually rides on |
+|---|---|---|
+| Accounting / tax records | ~10 years (statutory in most EU states) | Art. 6(1)(c) legal obligation |
+| HR records (post-employment) | ~3–6 years | Art. 6(1)(b)/(c) |
+| Customer / CRM | ~3 years after last interaction | Art. 6(1)(b)/(f) |
+| Marketing consent records | life of consent + proof | Art. 6(1)(a) consent |
+| Support tickets | 1–3 years | Art. 6(1)(b)/(f) |
+| Server / access logs | short (30–180 days typical) | Art. 6(1)(f) legitimate interest |
+
+Every processing activity in the ROPA should appear as a row here. The full fillable template with the delete-vs-anonymize-vs-archive note and the validation checklist lives in `references/retention-schedule.md`.
+
+## Pick the lawful basis
+
+Art. 6 gives **six** lawful bases, and you must identify one *before* processing starts: consent, contract, legal obligation, vital interests, public task, legitimate interests (gdpr-info.eu Art. 6; IAPP). Consent is one of six and is often the *weakest* choice for operational data.
+
+The trap: defaulting everything to consent. Consent is revocable at any time, so building contract-essential processing on it means a withdrawal can leave you unable to deliver the service. Use **contract** (Art. 6(1)(b)) for what the service requires, **legal obligation** (Art. 6(1)(c)) for statutory keep-periods, and **legitimate interest** (Art. 6(1)(f)) for fraud prevention, security logging, and most analytics. Reserve **consent** (Art. 6(1)(a)) for marketing and non-essential cookies/trackers.
+
+When you lean on legitimate interest, run the three-part balancing test and write it down:
+
+1. **Purpose** — is the interest legitimate and clearly stated?
+2. **Necessity** — is the processing actually needed, or would less-intrusive data do?
+3. **Balancing** — does it override the data subject's rights and reasonable expectations?
+
+Anchor this to the EDPB Guidelines 1/2024 on legitimate interest (Oct 2024). The worksheet is in `references/consent-and-ropa.md`.
+
+## The ROPA
+
+A ROPA (Art. 30) is the central inventory: one row per processing activity. Minimum columns: **activity, purpose, data categories + data subjects, recipients, transfers, retention period, security measures.** Art. 30 does not strictly require logging the Art. 6 basis — but record it per row anyway; it speeds audits, DPIAs, and notice updates (TermsFeed; Legiscope, 2026).
+
+```text
+Activity:    Customer support ticketing
+Purpose:     resolve and track support requests
+Data cats:   name, email, account ID, message content | Subjects: customers
+Recipients:  internal support team; Zendesk (processor)
+Transfers:   US (SCCs in place) — point to gdpr-privacy for the mechanism
+Retention:   2 years after ticket closed, then delete
+Security:    RBAC, encryption at rest, access logging
+Lawful basis: Art. 6(1)(b) contract  ← log it even though Art. 30 doesn't demand it
+```
+
+The full ROPA template with a second worked row — plus the consent-matrix template and the withdrawal/refresh workflow — is in `references/consent-and-ropa.md`.
+
+## Consent model
+
+Where consent *is* the basis, it must be valid under Art. 4(11) / Art. 7: **freely given, specific, informed, and unambiguous** — a positive opt-in act (EDPB).
+
+- **Capture:** an affirmative action, never a pre-ticked box. Granular per purpose (marketing email != product analytics). Reject must be as easy as accept — no dark patterns.
+- **Proof / logging:** store enough to prove consent later — timestamp, the consent-text *version*, the scope/purposes granted, and the capture method.
+- **Withdrawal:** must be as easy as giving it. One click, no retention-by-friction.
+- **Refresh:** EDPB recommends refreshing after ~12 months or on a material change.
+
+```text
+| Purpose          | Basis        | Capture point      | Proof fields stored              | Withdrawal      |
+| Marketing email  | Art. 6(1)(a) | signup checkbox    | ts, text v2.1, scope, method     | one-click unsub |
+| Product analytics| Art. 6(1)(a) | cookie banner      | ts, banner vN, categories, method| banner re-open  |
+```
+
+One note so you don't over-promise on cookies: the **ePrivacy Regulation was formally withdrawn by the European Commission in February 2025**, so the ePrivacy Directive (and its national implementations) still governs cookies and trackers (Hunton; Clym, 2026). Don't cite a Regulation that does not exist.
+
+## Make it real in systems
+
+The policy is worthless until deletion runs in the systems that actually hold the data — *including backups and archives*, which is exactly where regulators find data that should be gone.
+
+Checklist:
+
+- [ ] **Classify** the data first — you can't apply a period to a category you haven't mapped.
+- [ ] **Automate deletion** — a scheduled job, not a human promising to remember.
+- [ ] **Cover backups and archives**, not just live tables. Retention limits apply everywhere a copy lives.
+- [ ] **Legal-hold exception path** — a row under litigation/regulatory hold is *skipped* by the deletion job, and the basis for the hold is documented.
+- [ ] **Immutable audit log** — every deletion writes a record (what category, when, by which job) you can show a regulator.
+
+```text
+Bad:  A nightly cron deletes expired rows from the prod database.
+Good: The deletion job covers prod + the data warehouse + backup snapshots;
+      it skips any row flagged under legal hold; and it writes a deletion
+      audit record (category, count, timestamp, job id) for every run.
+```
+
+The deletion *mechanics* — TTL columns, partition drops, soft-delete schema — belong to `../db-migrations/SKILL.md` and `../postgresdb/SKILL.md`. You write the *policy* that those mechanics must satisfy.
+
+## AI reuse and cross-border transfers
+
+State explicitly in the policy whether production data may be reused for **AI/model training**. GDPR purpose limitation (Art. 5(1)(b)) restricts reusing data collected for one purpose to train a model — that is a new purpose needing its own basis. The EU AI Act adds documentation and logging-retention duties, with high-risk obligations applying from **2 Aug 2026**; the Commission's Digital Omnibus proposal would let AI providers lean on legitimate interest for development with enhanced safeguards and an unconditional opt-out (TechGDPR; IAPP, 2026). Practical rule: the retention policy must say whether AI reuse is allowed, on what basis, and how a subject opts out.
+
+For cross-border transfers, name the mechanism in the ROPA row (e.g. SCCs) and point to `../gdpr-privacy/SKILL.md` for the SCC/notice depth — that is its territory, not yours.
+
+## The boundary
+
+Retention periods are jurisdiction- and sector-specific, so a period you assert as final is legal advice you are not qualified to give — that is why this line has no exceptions. **You produce governance drafts, not legal sign-off.** Every policy you emit ends with a statement that a qualified DPO or privacy counsel must validate the schedule and lawful-basis register before adoption, and that this is not legal advice. You never assert a period is universally lawful.
+
+Hand off the edges: public-facing privacy notice + data-subject access/erasure (DSAR) handling -> `../gdpr-privacy/SKILL.md`; audit posture, SOC 2 / ISO 27001, control mapping -> `../compliance/SKILL.md`; a negotiated DPA's contractual clauses or a two-party data contract -> `../contracts/SKILL.md`; encryption, access hardening, threat controls on the systems -> `../secure-coding/SKILL.md`; the actual deletion mechanics in the database -> `../db-migrations/SKILL.md` / `../postgresdb/SKILL.md`.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Consent as the default basis for everything | Consent is revocable; a withdrawal breaks contract-essential processing | Use contract / legal obligation / legitimate interest for operational data; reserve consent for marketing |
+| "As long as necessary" / "indefinitely" as the only period | No concrete clock means nothing ever deletes — the classic paper policy | Give months/years or named criteria per category, validated against local law |
+| Delete from prod but leave backups/archives untouched | The data regulators find is the copy you forgot | Deletion job must cover prod + warehouse + backups |
+| No legal-hold exception in the auto-deletion job | The job destroys data under litigation hold — spoliation | Flag held rows; skip them; document the hold basis |
+| Copy a generic retention template unchanged | Periods are jurisdiction/sector-specific; a copied period can be unlawful | Tag every period "validate vs local + sector law"; adjust |
+| Emit the policy as final / "compliant" | Crosses into legal advice you can't give | End with DPO/counsel sign-off + not-legal-advice line |

package/skills/data-policy/evals/README.md

@@ -0,0 +1,15 @@
+# Evals — data-policy
+
+These cases are read by a human or an LLM grader; there is no automated runner in
+this repo. The `should_trigger` and `should_not_trigger` lists check routing
+intuition — does the description fire on the right prompts (including the
+non-obvious "fix retention in backups" and the Spanish phrasing) and defer to the
+named sibling on near-misses like a public privacy notice (`gdpr-privacy`), DPA
+clauses (`contracts`), SOC 2 readiness (`compliance`), encryption
+(`secure-coding`), or TTL migrations (`db-migrations`). The `capability` case is
+graded by generating a real retention schedule for the scenario and checking it
+against the `must_include` rubric by hand. To run: paste a prompt to the skill,
+read the output, and confirm the rubric items are present. The `scripts/verify.sh`
+lint can be pointed at any generated schedule/ROPA/consent file to mechanically
+catch missing periods, vague-only periods, a missing disclaimer, and a
+deletion-without-backups gap.

package/skills/data-policy/evals/cases.yaml

@@ -0,0 +1,44 @@
+skill: data-policy
+
+should_trigger:
+  - prompt: "Write our data retention policy with periods for HR, customer, and finance data."
+    why: "Core artifact — a retention schedule mapping categories to periods, the skill's primary job."
+  - prompt: "How long are we allowed to keep customer support tickets and server logs?"
+    why: "Everyday entry point — a retention-period question per data category."
+  - prompt: "Set up a Record of Processing Activities for our SaaS."
+    why: "ROPA / Art. 30 inventory, explicitly in scope."
+  - prompt: "Is our lawful basis for the newsletter consent or legitimate interest?"
+    why: "Non-obvious — looks like a marketing question, but it is a lawful-basis governance call (Art. 6) this skill owns."
+  - prompt: "We auto-delete users after 2 years but our backups still have them — how do we fix retention in backups?"
+    why: "Non-obvious — the backups/enforcement gap is squarely this skill's 'make it real in systems' section, not db mechanics."
+  - prompt: "Define cuánto tiempo guardamos los datos de clientes y qué borramos al expirar."
+    why: "Spanish trigger — retention period + expiry action, the core artifact."
+  - prompt: "Design our consent capture and withdrawal flow and log proof of consent."
+    why: "Consent model — capture, withdrawal, proof logging, the consent-matrix part of the skill."
+
+should_not_trigger:
+  - prompt: "Write the privacy policy our users see on the website."
+    route_to: gdpr-privacy
+    why: "Public-facing notice + DSAR handling, not the internal retention/governance machinery."
+  - prompt: "Draft the Data Processing Addendum clauses in our vendor contract."
+    route_to: contracts
+    why: "Negotiated two-party contractual clauses, not an internal policy artifact."
+  - prompt: "Get us SOC 2 audit-ready and map our security controls."
+    route_to: compliance
+    why: "Audit posture / framework control mapping, not data-lifecycle rules."
+  - prompt: "Encrypt the PII at rest and harden the database access."
+    route_to: secure-coding
+    why: "Security controls on the systems, not the governance policy they enforce."
+  - prompt: "Write the TTL migration to actually drop the old partitions."
+    route_to: db-migrations
+    why: "Deletion mechanics in the database, not the retention policy those mechanics satisfy."
+
+capability:
+  - scenario: "A 20-person SaaS asks for a retention schedule covering customer accounts, support tickets, marketing consents, and access logs — with the lawful basis for each and what runs at expiry."
+    must_include:
+      - "A table with one row per category, each having all of: retention period, lawful basis (specific Art. 6 sub-paragraph), expiry action (delete/anonymize/archive), and system of record."
+      - "At least one row using a lawful basis other than consent (e.g. Art. 6(1)(b) contract or 6(1)(f) legitimate interest for logs)."
+      - "Explicit handling of backups/archives, not just live data."
+      - "A legal-hold exception note for the auto-deletion workflow."
+      - "The DPO/counsel sign-off boundary and a not-legal-advice statement."
+      - "A 'validate against local + sector law' caveat — no period asserted as universally lawful."