rsc-universal 0.1.1

package/skills/data-cleaning/references/validation-patterns.md

@@ -0,0 +1,134 @@
+# Validation patterns — pandera, GX Core, dbt
+
+The gate is what makes cleaned data *trustworthy*: it asserts the contract holds and fails loud when next
+month's file drifts. Pick the lightest tool that covers your blast radius.
+
+## Which validator
+
+| Situation | Use | Why |
+| --- | --- | --- |
+| One pipeline, validate a DataFrame in Python | **pandera** | Lightweight, in-process, lives next to `clean()`; pin `pandera==0.31.1` (supports pandas ≥ 3) |
+| Many datasets/teams, shared data-quality platform, results store + docs | **GX Core 1.0** | Heavier; Data Context / Suites / Checkpoints are infrastructure, overkill for one script |
+| Cleaning lives in a SQL warehouse, not Python | **dbt** model contracts + tests | Contract enforced at build; tests run post-materialization |
+
+Default to pandera. Escalate only when the heavier tool's features are actually needed.
+
+## Pandera — the full DataFrameModel
+
+```python
+import pandera.pandas as pa
+from pandera.typing import Series
+
+class CustomerSchema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(unique=True, ge=1)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])
+    age:         Series[float] = pa.Field(ge=0, le=120, nullable=True)
+    email:       Series[str]   = pa.Field(str_matches=r"^[^@\s]+@[^@\s]+\.[^@\s]+$", nullable=True)
+    signup:      Series[pa.DateTime] = pa.Field(nullable=False)
+
+    class Config:
+        strict = True        # reject unexpected columns — schema drift fails the build
+        coerce = True        # coerce to declared dtype; raise if a value can't be coerced
+        ordered = False      # set True if column order is part of the contract
+
+    @pa.check("email", name="lowercase_email")
+    def email_is_lower(cls, s: Series[str]) -> Series[bool]:
+        # custom row-wise check: returns a boolean Series, one per row
+        return s.isna() | (s == s.str.lower())
+
+    @pa.dataframe_check
+    def row_budget(cls, df) -> bool:
+        # whole-frame invariant: catch a truncated/exploded extract
+        return 1 <= len(df) <= 5_000_000
+```
+
+`coerce` fixes types the contract expects. `nullable` declares which columns may hold NA. Field checks
+(`ge`, `le`, `isin`, `unique`, `str_matches`) are allowed-value rules. `@pa.check` is a custom column
+check; `@pa.dataframe_check` is a whole-frame invariant.
+
+## Lazy errors + the quarantine split
+
+`lazy=True` collects **every** violation across the frame instead of dying on the first. Use it to split
+valid rows from failing rows — the failures become the quarantine, never a silent drop.
+
+```python
+def validate_and_split(df, schema):
+    """Return (valid_df, quarantine_df). Quarantine is kept, never discarded."""
+    try:
+        valid = schema.validate(df, lazy=True)
+        return valid, df.iloc[0:0].copy()          # empty quarantine — everything passed
+    except pa.errors.SchemaErrors as e:
+        # failure_cases columns: schema_context, column, check, failure_case, index
+        bad_idx = e.failure_cases["index"].dropna().unique()
+        quarantine = df.loc[df.index.isin(bad_idx)].copy()
+        quarantine["_failed_checks"] = (
+            e.failure_cases.dropna(subset=["index"])
+             .groupby("index")["check"].agg(lambda c: "; ".join(map(str, c)))
+        )
+        valid = df.loc[~df.index.isin(bad_idx)]
+        return valid, quarantine
+
+valid, quarantine = validate_and_split(df, CustomerSchema)
+quarantine.to_parquet("quarantine.parquet")        # someone investigates these
+print(f"valid={len(valid)} quarantined={len(quarantine)}")   # the audit diff
+```
+
+Validate **output** always. Validate **input expectations** too when you can state them (e.g. raw must have
+these columns before you start) — it turns a confusing mid-pipeline crash into a clear "the source changed".
+
+## GX Core 1.0 — checkpoint sketch
+
+GX rebranded its OSS to **GX Core** and shipped 1.0 GA with a simplified API. Reach for it when validation
+is a shared platform across many datasets, not a single script.
+
+```python
+import great_expectations as gx
+
+context = gx.get_context()                                   # Data Context
+ds  = context.data_sources.add_pandas("customers_src")       # Data Source
+asset = ds.add_dataframe_asset(name="customers")
+batch = asset.add_batch_definition_whole_dataframe("batch")
+
+suite = context.suites.add(gx.ExpectationSuite(name="customers_suite"))
+suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(column="age", min_value=0, max_value=120))
+suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(column="country", value_set=["US", "ES", "FR"]))
+
+vd = context.validation_definitions.add(
+    gx.ValidationDefinition(name="customers_vd", data=batch, suite=suite)
+)
+checkpoint = context.checkpoints.add(gx.Checkpoint(name="customers_cp", validation_definitions=[vd]))
+result = checkpoint.run(batch_parameters={"dataframe": df})
+assert result.success, "GX checkpoint failed — see Data Docs"
+```
+
+The flow is always **Data Context → Data Source → Expectation Suite → Validation Definition → Checkpoint**.
+You get a results store and generated Data Docs — worth the weight only when many people consume them.
+
+## dbt — contracts vs tests (cleaning in the warehouse)
+
+When cleaning is SQL in a warehouse, the contract lives in the model's YAML. **Contracts** are enforced at
+*build* (the model fails to build if the output is off-contract). **Tests** validate *after*
+materialization.
+
+```yaml
+# models/staging/_stg_customers.yml
+models:
+  - name: stg_customers
+    config:
+      contract: { enforced: true }          # build fails if columns/types drift — the contract
+    columns:
+      - name: customer_id
+        data_type: bigint
+        constraints: [{ type: not_null }, { type: unique }]
+      - name: country
+        data_type: varchar
+        data_tests:                          # validated post-materialization — the tests
+          - accepted_values: { values: ["US", "ES", "FR"] }
+      - name: age
+        data_type: integer
+        data_tests:
+          - dbt_utils.accepted_range: { min_value: 0, max_value: 120 }
+```
+
+Contract = "this shape is guaranteed at build"; tests = "these values held after the run". Use both: the
+contract stops schema drift, the tests catch data drift.

package/skills/data-cleaning/scripts/verify.sh

@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# verify.sh — data-cleaning skill check. Run from anywhere. Read-only by default.
+#
+# Two layers:
+#   STATIC (always): SKILL.md exists, frontmatter has the required keys, description is one
+#     physical line, and both references/ files exist and are non-empty.
+#   RUNTIME (only if pandas + pandera importable): build a tiny fixture with ONE clearly-good
+#     row and ONE clearly-bad row (negative age / unmapped country), run a pandera schema with
+#     lazy=True, and assert the good row PASSES while the bad row is FLAGGED — proving the gate
+#     is not a no-op. If pandas/pandera are absent, print SKIP and still pass the static checks.
+#
+# Writes only inside a private mktemp dir, removed on exit. No network.
+# Exit: 0 on pass or skip; non-zero ONLY on a real failure (missing artifact, malformed
+# frontmatter, or a gate that fails to reject a known-bad row).
+
+YELLOW=$'\033[33m'; GREEN=$'\033[32m'; RED=$'\033[31m'; NC=$'\033[0m'
+skip() { printf '%s[skip]%s %s\n' "$YELLOW" "$NC" "$*"; }
+ok()   { printf '%s[ok]%s %s\n'   "$GREEN"  "$NC" "$*"; }
+err()  { printf '%s[fail]%s %s\n' "$RED"    "$NC" "$*"; }
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SKILL_DIR="$(dirname "$SCRIPT_DIR")"
+SKILL="$SKILL_DIR/SKILL.md"
+fail=0
+
+# ---- STATIC CHECKS (always run) ----
+if [ ! -f "$SKILL" ]; then
+  err "SKILL.md not found at $SKILL"; exit 1
+fi
+ok "SKILL.md present"
+
+for key in "name:" "description:" "tags:" "recommends:" "origin:"; do
+  if grep -q "^${key}" "$SKILL"; then
+    ok "frontmatter has ${key%:}"
+  else
+    err "frontmatter missing ${key%:}"; fail=1
+  fi
+done
+
+# description must be exactly one physical line
+desc_lines="$(grep -c '^description:' "$SKILL" || true)"
+if [ "$desc_lines" = "1" ]; then
+  ok "description present as a single key line"
+else
+  err "expected exactly one 'description:' line, found $desc_lines"; fail=1
+fi
+
+for ref in "references/validation-patterns.md" "references/normalization-recipes.md"; do
+  if [ -s "$SKILL_DIR/$ref" ]; then
+    ok "$ref present and non-empty"
+  else
+    err "$ref missing or empty"; fail=1
+  fi
+done
+
+if [ "$fail" -ne 0 ]; then
+  err "static checks failed"; exit 1
+fi
+
+# ---- RUNTIME GATE CHECK (only if deps present) ----
+if ! command -v python3 >/dev/null 2>&1; then
+  skip "python3 not found — ran static checks only"; exit 0
+fi
+if ! python3 -c "import pandas, pandera" >/dev/null 2>&1; then
+  skip "pandas/pandera not importable — ran static checks only"; exit 0
+fi
+
+TMP="$(mktemp -d 2>/dev/null || mktemp -d -t dc_verify)"
+trap 'rm -rf "$TMP"' EXIT
+
+python3 - <<'PY'
+import sys
+import pandas as pd
+import pandera.pandas as pa
+from pandera.typing import Series
+
+# Fixture: row 1 is clearly GOOD; row 2 is clearly BAD (negative age + unmapped country -> NA).
+COUNTRY = {"usa": "US", "u.s.": "US", "united states": "US", "es": "ES"}
+raw = pd.DataFrame({
+    "customer_id": [1, 2],
+    "country":     ["U.S.", "Atlantis"],   # row2 unmapped -> NA -> fails isin
+    "age":         [42, -5],               # row2 out of range -> fails ge(0)
+})
+raw["country"] = raw["country"].str.strip().str.casefold().map(COUNTRY)
+
+class Schema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(ge=1, unique=True)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])
+    age:         Series[float] = pa.Field(ge=0, le=120)
+    class Config:
+        coerce = True
+        strict = True
+
+try:
+    Schema.validate(raw, lazy=True)
+    print("GATE-NOOP: schema accepted a known-bad row")  # gate did nothing -> failure
+    sys.exit(1)
+except pa.errors.SchemaErrors as e:
+    bad_idx = set(e.failure_cases["index"].dropna().astype(int).unique())
+    # The bad row (index 1) must be flagged; the good row (index 0) must NOT be.
+    if 1 in bad_idx and 0 not in bad_idx:
+        sys.exit(0)
+    print(f"GATE-WRONG: flagged indices were {sorted(bad_idx)}; expected {{1}}")
+    sys.exit(1)
+PY
+rc=$?
+if [ "$rc" -eq 0 ]; then
+  ok "validation gate rejected the bad row and passed the good row (gate is not a no-op)"
+  exit 0
+else
+  err "validation gate did not behave as documented"
+  exit 1
+fi

package/skills/data-policy/SKILL.md

@@ -0,0 +1,163 @@
+---
+name: data-policy
+description: "Use when an operator needs the internal data-governance machinery a business runs on — a retention schedule (how long each data category is kept and what happens at expiry), the lawful basis per activity, a Record of Processing Activities (ROPA), or a consent capture/withdrawal model; symptoms include 'how long can we keep this', a paper policy that never runs in systems, or data still in backups past its retention. Triggers: 'write our data retention policy', 'how long can we keep customer/HR/support/log data', 'set up a ROPA', 'lawful basis consent or legitimate interest', 'design our consent capture and withdrawal flow', 'auto-deletion with legal-hold exceptions', 'we delete from prod but backups still have it', 'política de conservación de datos', 'cuánto tiempo guardamos los datos'. NOT the public-facing privacy notice or DSAR handling (that is gdpr-privacy), NOT SOC 2 audit posture (that is compliance), and NOT the database TTL/partition-drop mechanics (that is db-migrations)."
+tags: [data-governance, retention, gdpr, ropa, consent, lawful-basis, privacy, data-minimization]
+recommends: [gdpr-privacy, compliance, contracts, secure-coding, db-migrations, postgresdb]
+origin: risco
+---
+
+# Data policy
+
+You build the **internal governance machinery** a business runs on: a retention schedule, a lawful-basis register, a Record of Processing Activities (ROPA), and a consent model. You produce structured artifacts that engineering and ops can implement — not the public-facing privacy notice users read (that is `../gdpr-privacy/SKILL.md`). You are not a DPO and you never claim to be one.
+
+One rule sits above everything else.
+
+**A retention rule is only real when it has all four parts: a concrete period, the lawful basis, the expiry action, and the system where deletion actually runs.** A policy that names a period but never deletes anything is a paper policy — and a paper policy is precisely what regulators fine. Cumulative GDPR fines hit ~EUR 5.65B across ~2,245 actions by March 2025, and the two failures that recur are *no systematic data classification* and *no automated deletion capability* (Secure Privacy / CMS Enforcement Tracker, 2025). Every schedule you emit ends with the DPO/counsel sign-off boundary below.
+
+## First move: which artifact does the operator need?
+
+Map the request to one artifact before writing anything. Each routes to a section.
+
+| Operator says | Artifact | Go to |
+|---|---|---|
+| "How long do we keep X / write our retention policy" | Retention schedule | Build the retention schedule |
+| "Is our basis consent or legitimate interest?" | Lawful-basis register | Pick the lawful basis |
+| "Set up a ROPA / Article 30 record" | ROPA row | The ROPA |
+| "Design consent capture / withdrawal" | Consent matrix | Consent model |
+| "Auto-delete but keep legal holds / backups still have data" | Deletion workflow | Make it real in systems |
+
+If the operator wants the public privacy notice, the DPA contract clauses, or SOC 2 readiness, stop and route them — see the boundary section. Those are different skills.
+
+## Build the retention schedule
+
+This is the core artifact. For every category of personal data, walk five columns in order: **data category -> purpose -> lawful basis -> retention period -> expiry action -> system of record**. GDPR's storage-limitation principle (Art. 5(1)(e)) requires data be held in identifiable form *no longer than necessary for the purpose it was collected for*; GDPR sets no fixed periods — duration is driven by purpose plus sector law (gdpr-info.eu Art. 5; Usercentrics, 2026).
+
+The expiry action is one of three, and you must pick one explicitly:
+
+- **delete** — the row is gone.
+- **anonymize** — identifiers stripped so the record is no longer personal data (then storage limitation no longer bites); valid only if re-identification is genuinely infeasible.
+- **archive** — kept under Art. 89(1) safeguards for a lawful long-term purpose (statutory, archival, statistical).
+
+Worked example. The Bad version is what gets fined; the Good version is enforceable.
+
+```text
+Bad:  Customer data — kept as long as necessary.
+Good: | Category        | Purpose        | Lawful basis      | Period                  | Expiry   | System of record       |
+      | Customer orders | fulfil + tax   | Art. 6(1)(b) +(c) | 36 mo after last order  | anonymize| Postgres `orders` + DWH|
+```
+
+Working default periods — **starting points, never asserted as universally lawful; validate against local + sector law** (Usercentrics; Secure Privacy, 2026):
+
+| Category | Common default | Basis it usually rides on |
+|---|---|---|
+| Accounting / tax records | ~10 years (statutory in most EU states) | Art. 6(1)(c) legal obligation |
+| HR records (post-employment) | ~3–6 years | Art. 6(1)(b)/(c) |
+| Customer / CRM | ~3 years after last interaction | Art. 6(1)(b)/(f) |
+| Marketing consent records | life of consent + proof | Art. 6(1)(a) consent |
+| Support tickets | 1–3 years | Art. 6(1)(b)/(f) |
+| Server / access logs | short (30–180 days typical) | Art. 6(1)(f) legitimate interest |
+
+Every processing activity in the ROPA should appear as a row here. The full fillable template with the delete-vs-anonymize-vs-archive note and the validation checklist lives in `references/retention-schedule.md`.
+
+## Pick the lawful basis
+
+Art. 6 gives **six** lawful bases, and you must identify one *before* processing starts: consent, contract, legal obligation, vital interests, public task, legitimate interests (gdpr-info.eu Art. 6; IAPP). Consent is one of six and is often the *weakest* choice for operational data.
+
+The trap: defaulting everything to consent. Consent is revocable at any time, so building contract-essential processing on it means a withdrawal can leave you unable to deliver the service. Use **contract** (Art. 6(1)(b)) for what the service requires, **legal obligation** (Art. 6(1)(c)) for statutory keep-periods, and **legitimate interest** (Art. 6(1)(f)) for fraud prevention, security logging, and most analytics. Reserve **consent** (Art. 6(1)(a)) for marketing and non-essential cookies/trackers.
+
+When you lean on legitimate interest, run the three-part balancing test and write it down:
+
+1. **Purpose** — is the interest legitimate and clearly stated?
+2. **Necessity** — is the processing actually needed, or would less-intrusive data do?
+3. **Balancing** — does it override the data subject's rights and reasonable expectations?
+
+Anchor this to the EDPB Guidelines 1/2024 on legitimate interest (Oct 2024). The worksheet is in `references/consent-and-ropa.md`.
+
+## The ROPA
+
+A ROPA (Art. 30) is the central inventory: one row per processing activity. Minimum columns: **activity, purpose, data categories + data subjects, recipients, transfers, retention period, security measures.** Art. 30 does not strictly require logging the Art. 6 basis — but record it per row anyway; it speeds audits, DPIAs, and notice updates (TermsFeed; Legiscope, 2026).
+
+```text
+Activity:    Customer support ticketing
+Purpose:     resolve and track support requests
+Data cats:   name, email, account ID, message content | Subjects: customers
+Recipients:  internal support team; Zendesk (processor)
+Transfers:   US (SCCs in place) — point to gdpr-privacy for the mechanism
+Retention:   2 years after ticket closed, then delete
+Security:    RBAC, encryption at rest, access logging
+Lawful basis: Art. 6(1)(b) contract  ← log it even though Art. 30 doesn't demand it
+```
+
+The full template with a second worked row is in `references/consent-and-ropa.md`.
+
+## Consent model
+
+Where consent *is* the basis, it must be valid under Art. 4(11) / Art. 7: **freely given, specific, informed, and unambiguous** — a positive opt-in act (EDPB).
+
+- **Capture:** an affirmative action, never a pre-ticked box. Granular per purpose (marketing email != product analytics). Reject must be as easy as accept — no dark patterns.
+- **Proof / logging:** store enough to prove consent later — timestamp, the consent-text *version*, the scope/purposes granted, and the capture method.
+- **Withdrawal:** must be as easy as giving it. One click, no retention-by-friction.
+- **Refresh:** EDPB recommends refreshing after ~12 months or on a material change.
+
+```text
+| Purpose          | Basis        | Capture point      | Proof fields stored              | Withdrawal      |
+| Marketing email  | Art. 6(1)(a) | signup checkbox    | ts, text v2.1, scope, method     | one-click unsub |
+| Product analytics| Art. 6(1)(a) | cookie banner      | ts, banner vN, categories, method| banner re-open  |
+```
+
+One note so you don't over-promise on cookies: the **ePrivacy Regulation was formally withdrawn by the European Commission in February 2025**, so the ePrivacy Directive (and its national implementations) still governs cookies and trackers (Hunton; Clym, 2026). Don't cite a Regulation that does not exist.
+
+## Make it real in systems
+
+This is the section that earns the skill. The policy is worthless until deletion runs in the systems that actually hold the data — *including backups and archives*, which is exactly where regulators find data that should be gone.
+
+Checklist:
+
+- [ ] **Classify** the data first — you can't apply a period to a category you haven't mapped.
+- [ ] **Automate deletion** — a scheduled job, not a human promising to remember.
+- [ ] **Cover backups and archives**, not just live tables. Retention limits apply everywhere a copy lives.
+- [ ] **Legal-hold exception path** — a row under litigation/regulatory hold is *skipped* by the deletion job, and the basis for the hold is documented.
+- [ ] **Immutable audit log** — every deletion writes a record (what category, when, by which job) you can show a regulator.
+
+```text
+Bad:  A nightly cron deletes expired rows from the prod database.
+Good: The deletion job covers prod + the data warehouse + backup snapshots;
+      it skips any row flagged under legal hold; and it writes a deletion
+      audit record (category, count, timestamp, job id) for every run.
+```
+
+The deletion *mechanics* — TTL columns, partition drops, soft-delete schema — belong to `../db-migrations/SKILL.md` and `../postgresdb/SKILL.md`. You write the *policy* that those mechanics must satisfy.
+
+## AI reuse and cross-border transfers
+
+State explicitly in the policy whether production data may be reused for **AI/model training**. GDPR purpose limitation (Art. 5(1)(b)) restricts reusing data collected for one purpose to train a model — that is a new purpose needing its own basis. The EU AI Act adds documentation and logging-retention duties, with high-risk obligations applying from **2 Aug 2026**; the Commission's Digital Omnibus proposal would let AI providers lean on legitimate interest for development with enhanced safeguards and an unconditional opt-out (TechGDPR; IAPP, 2026). Practical rule: the retention policy must say whether AI reuse is allowed, on what basis, and how a subject opts out.
+
+For cross-border transfers, name the mechanism in the ROPA row (e.g. SCCs) and point to `../gdpr-privacy/SKILL.md` for the SCC/notice depth — that is its territory, not yours.
+
+## The boundary (non-negotiable)
+
+Retention periods are jurisdiction- and sector-specific. **You produce governance drafts, not legal sign-off.** Every policy you emit ends with a line stating a qualified DPO or privacy counsel must validate the schedule and lawful-basis register before adoption, and that this is not legal advice. You never assert a period is universally lawful.
+
+Hand off the edges:
+
+- Public-facing privacy notice + data-subject access/erasure (DSAR) handling -> `../gdpr-privacy/SKILL.md`.
+- Audit posture, SOC 2 / ISO 27001, control mapping -> `../compliance/SKILL.md`.
+- A negotiated DPA's contractual clauses or a two-party data contract -> `../contracts/SKILL.md`.
+- Encryption, access hardening, threat controls on the systems -> `../secure-coding/SKILL.md`.
+- The actual deletion mechanics in the database -> `../db-migrations/SKILL.md` / `../postgresdb/SKILL.md`.
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Do instead |
+|---|---|---|
+| Consent as the default basis for everything | Consent is revocable; a withdrawal breaks contract-essential processing | Use contract / legal obligation / legitimate interest for operational data; reserve consent for marketing |
+| "As long as necessary" / "indefinitely" as the only period | No concrete clock means nothing ever deletes — the classic paper policy | Give months/years or named criteria per category, validated against local law |
+| Delete from prod but leave backups/archives untouched | The data regulators find is the copy you forgot | Deletion job must cover prod + warehouse + backups |
+| No legal-hold exception in the auto-deletion job | The job destroys data under litigation hold — spoliation | Flag held rows; skip them; document the hold basis |
+| Copy a generic retention template unchanged | Periods are jurisdiction/sector-specific; a copied period can be unlawful | Tag every period "validate vs local + sector law"; adjust |
+| Emit the policy as final / "compliant" | Crosses into legal advice you can't give | End with DPO/counsel sign-off + not-legal-advice line |
+
+## References
+
+- `references/retention-schedule.md` — fillable schedule template (data category, purpose, lawful basis, period, expiry action, system of record, legal-hold flag, review date) + a populated example across HR, finance/tax, CRM, marketing, support, logs, with the delete-vs-anonymize-vs-archive note and the local + sector law validation checklist.
+- `references/consent-and-ropa.md` — consent matrix template + worked example, the Art. 30 ROPA template + worked rows, the legitimate-interest balancing-test worksheet, and the consent withdrawal/refresh workflow.

package/skills/data-policy/evals/README.md

@@ -0,0 +1,15 @@
+# Evals — data-policy
+
+These cases are read by a human or an LLM grader; there is no automated runner in
+this repo. The `should_trigger` and `should_not_trigger` lists check routing
+intuition — does the description fire on the right prompts (including the
+non-obvious "fix retention in backups" and the Spanish phrasing) and defer to the
+named sibling on near-misses like a public privacy notice (`gdpr-privacy`), DPA
+clauses (`contracts`), SOC 2 readiness (`compliance`), encryption
+(`secure-coding`), or TTL migrations (`db-migrations`). The `capability` case is
+graded by generating a real retention schedule for the scenario and checking it
+against the `must_include` rubric by hand. To run: paste a prompt to the skill,
+read the output, and confirm the rubric items are present. The `scripts/verify.sh`
+lint can be pointed at any generated schedule/ROPA/consent file to mechanically
+catch missing periods, vague-only periods, a missing disclaimer, and a
+deletion-without-backups gap.

package/skills/data-policy/evals/cases.yaml

@@ -0,0 +1,44 @@
+skill: data-policy
+
+should_trigger:
+  - prompt: "Write our data retention policy with periods for HR, customer, and finance data."
+    why: "Core artifact — a retention schedule mapping categories to periods, the skill's primary job."
+  - prompt: "How long are we allowed to keep customer support tickets and server logs?"
+    why: "Everyday entry point — a retention-period question per data category."
+  - prompt: "Set up a Record of Processing Activities for our SaaS."
+    why: "ROPA / Art. 30 inventory, explicitly in scope."
+  - prompt: "Is our lawful basis for the newsletter consent or legitimate interest?"
+    why: "Non-obvious — looks like a marketing question, but it is a lawful-basis governance call (Art. 6) this skill owns."
+  - prompt: "We auto-delete users after 2 years but our backups still have them — how do we fix retention in backups?"
+    why: "Non-obvious — the backups/enforcement gap is squarely this skill's 'make it real in systems' section, not db mechanics."
+  - prompt: "Define cuánto tiempo guardamos los datos de clientes y qué borramos al expirar."
+    why: "Spanish trigger — retention period + expiry action, the core artifact."
+  - prompt: "Design our consent capture and withdrawal flow and log proof of consent."
+    why: "Consent model — capture, withdrawal, proof logging, the consent-matrix part of the skill."
+
+should_not_trigger:
+  - prompt: "Write the privacy policy our users see on the website."
+    route_to: gdpr-privacy
+    why: "Public-facing notice + DSAR handling, not the internal retention/governance machinery."
+  - prompt: "Draft the Data Processing Addendum clauses in our vendor contract."
+    route_to: contracts
+    why: "Negotiated two-party contractual clauses, not an internal policy artifact."
+  - prompt: "Get us SOC 2 audit-ready and map our security controls."
+    route_to: compliance
+    why: "Audit posture / framework control mapping, not data-lifecycle rules."
+  - prompt: "Encrypt the PII at rest and harden the database access."
+    route_to: secure-coding
+    why: "Security controls on the systems, not the governance policy they enforce."
+  - prompt: "Write the TTL migration to actually drop the old partitions."
+    route_to: db-migrations
+    why: "Deletion mechanics in the database, not the retention policy those mechanics satisfy."
+
+capability:
+  - scenario: "A 20-person SaaS asks for a retention schedule covering customer accounts, support tickets, marketing consents, and access logs — with the lawful basis for each and what runs at expiry."
+    must_include:
+      - "A table with one row per category, each having all of: retention period, lawful basis (specific Art. 6 sub-paragraph), expiry action (delete/anonymize/archive), and system of record."
+      - "At least one row using a lawful basis other than consent (e.g. Art. 6(1)(b) contract or 6(1)(f) legitimate interest for logs)."
+      - "Explicit handling of backups/archives, not just live data."
+      - "A legal-hold exception note for the auto-deletion workflow."
+      - "The DPO/counsel sign-off boundary and a not-legal-advice statement."
+      - "A 'validate against local + sector law' caveat — no period asserted as universally lawful."

package/skills/data-policy/references/consent-and-ropa.md

@@ -0,0 +1,97 @@
+# Consent matrix, ROPA, and the legitimate-interest worksheet
+
+The copy-ready artifacts behind the consent and ROPA sections. Validate with a
+DPO before adoption; nothing here is legal advice.
+
+## ROPA template (Art. 30)
+
+One row per processing activity. Art. 30 columns are mandatory; the lawful-basis
+line is recommended even though Art. 30 doesn't strictly require it — it speeds
+audits, DPIAs, and notice updates.
+
+```text
+Activity:      <name of the processing activity>
+Purpose:       <why you process this data>
+Data cats:     <categories of personal data>        | Subjects: <categories of data subjects>
+Recipients:    <internal teams + processors + third parties>
+Transfers:     <country + transfer mechanism, e.g. SCCs; or "none">
+Retention:     <period + expiry action — must match the retention schedule>
+Security:      <RBAC, encryption, logging, ...>
+Lawful basis:  <Art. 6 sub-paragraph>               ← recommended, not strictly required
+```
+
+### Worked rows
+
+```text
+Activity:      Customer support ticketing
+Purpose:       resolve and track support requests
+Data cats:     name, email, account ID, message content | Subjects: customers
+Recipients:    internal support team; Zendesk (processor)
+Transfers:     US (SCCs in place)
+Retention:     2 years after ticket closed, then delete
+Security:      RBAC, encryption at rest, access logging
+Lawful basis:  Art. 6(1)(b) contract
+
+Activity:      Product analytics
+Purpose:       understand feature usage to improve the product
+Data cats:     pseudonymous event data, device, coarse location | Subjects: users
+Recipients:    internal product team; analytics processor
+Transfers:     none (EU-hosted)
+Retention:     14 months, then anonymize
+Security:      pseudonymization, RBAC, encryption at rest
+Lawful basis:  Art. 6(1)(a) consent (non-essential analytics)
+```
+
+## Consent matrix template
+
+```text
+| Purpose | Lawful basis (Art. 6(1)(a)) | Capture point | Proof fields stored | Withdrawal mechanism | Refresh cadence |
+|---------|-----------------------------|---------------|---------------------|----------------------|-----------------|
+|         |                             |               | timestamp, text version, scope, method | | ~12 mo / on material change |
+```
+
+### Worked example
+
+```text
+| Purpose           | Lawful basis | Capture point   | Proof fields stored                | Withdrawal       | Refresh             |
+|-------------------|--------------|-----------------|------------------------------------|------------------|---------------------|
+| Marketing email   | Art. 6(1)(a) | signup checkbox | ts, text v2.1, scope, method       | one-click unsub  | 12 mo / on change   |
+| Product analytics | Art. 6(1)(a) | cookie banner   | ts, banner vN, categories, method  | re-open banner   | 12 mo / on change   |
+```
+
+Capture rules (Art. 4(11) / Art. 7): affirmative opt-in, never pre-ticked;
+granular per purpose; reject as easy as accept (no dark patterns); withdrawal as
+easy as giving. The ePrivacy Regulation was withdrawn Feb 2025 — the ePrivacy
+Directive still governs cookies and trackers.
+
+## Consent withdrawal / refresh workflow
+
+1. **Withdraw** — subject clicks the one-click control (unsubscribe link, banner
+   re-open). No friction, no retention-by-dark-pattern.
+2. **Stop processing** — the purpose tied to that consent halts immediately.
+3. **Log** — write the withdrawal event (timestamp, scope withdrawn, method)
+   to the consent log; keep proof of withdrawal as you keep proof of consent.
+4. **Honor downstream** — propagate to processors and suppress in the CDP/ESP.
+5. **Refresh** — re-request consent after ~12 months or whenever the
+   consent text materially changes; re-prompting resets the proof record.
+
+## Legitimate-interest balancing test (Art. 6(1)(f))
+
+Run and document this before relying on legitimate interest. Anchor to EDPB
+Guidelines 1/2024 (Oct 2024).
+
+1. **Purpose test** — Is there a legitimate interest, clearly articulated?
+   (e.g. fraud prevention, network security, direct B2B marketing.)
+2. **Necessity test** — Is the processing actually necessary for that interest,
+   or would less data / a less intrusive method achieve it?
+3. **Balancing test** — Do the individual's interests, rights, and reasonable
+   expectations override the interest? Consider the relationship, the data
+   sensitivity, and whether the subject would expect this use.
+
+If the balance tips against you, fall back to consent or contract — or don't
+process. Record the outcome; a regulator can ask to see the assessment.
+
+---
+
+**Not legal advice.** A qualified DPO or privacy counsel must validate the
+lawful-basis register, ROPA, and consent model before adoption.

package/skills/data-policy/references/retention-schedule.md

@@ -0,0 +1,83 @@
+# Retention schedule — template + worked example
+
+This is the copy-ready artifact the body points to. Fill the template, validate
+every period against local + sector law, and end the document with the DPO /
+counsel sign-off line. Nothing here is legal advice.
+
+## The fillable template
+
+One row per personal-data category. No row ships without a period, a lawful
+basis, and an expiry action.
+
+```text
+| Data category | Purpose | Lawful basis (Art. 6) | Retention period | Expiry action | System of record | Legal-hold flag | Review date |
+|---------------|---------|-----------------------|------------------|---------------|------------------|-----------------|-------------|
+|               |         |                       |                  | delete/anon/archive |            | yes/no          | YYYY-MM-DD  |
+```
+
+Column rules:
+
+- **Retention period** — a concrete duration or named criteria ("36 months
+  after last order"), never "as long as necessary" alone.
+- **Lawful basis** — the specific Art. 6 sub-paragraph, not just "GDPR".
+- **Expiry action** — exactly one of delete / anonymize / archive (see note).
+- **System of record** — the actual store(s) the deletion job must touch,
+  including the warehouse and backups, not just the live table.
+- **Legal-hold flag** — whether rows here can be put under hold (litigation /
+  regulatory), which the deletion job must skip.
+- **Review date** — when this row is re-validated against current law.
+
+## Populated example schedule
+
+Periods below are common working defaults — **starting points, validate against
+local + sector law** (Usercentrics; Secure Privacy, 2026). They are not asserted
+as universally lawful.
+
+```text
+| Data category        | Purpose                  | Lawful basis        | Retention period            | Expiry action | System of record            | Legal-hold | Review date |
+|----------------------|--------------------------|---------------------|-----------------------------|---------------|-----------------------------|------------|-------------|
+| HR / payroll records | employment + statutory   | Art. 6(1)(b)+(c)    | 6 years after employment ends | archive     | HRIS + payroll backups      | yes        | 2027-01-01  |
+| Accounting / tax     | statutory bookkeeping    | Art. 6(1)(c)        | 10 years (statutory, EU)    | archive       | ERP + cold storage          | yes        | 2027-01-01  |
+| Customer / CRM       | fulfil + relationship    | Art. 6(1)(b)+(f)    | 3 years after last interaction | anonymize  | Postgres `customers` + DWH  | yes        | 2027-01-01  |
+| Marketing consent    | email campaigns          | Art. 6(1)(a)        | life of consent + proof     | delete        | CDP + consent log           | no         | 2027-01-01  |
+| Support tickets      | resolve + audit          | Art. 6(1)(b)+(f)    | 2 years after ticket closed | delete        | Zendesk (processor)         | yes        | 2027-01-01  |
+| Server / access logs | security + debugging     | Art. 6(1)(f)        | 90 days                     | delete        | log store + backup rotation | no         | 2027-01-01  |
+```
+
+## delete vs anonymize vs archive
+
+Pick the expiry action deliberately:
+
+- **delete** — destroy the record everywhere it lives. The default when no
+  lawful reason to keep remains.
+- **anonymize** — strip identifiers so the data is no longer personal data;
+  storage limitation then no longer applies. Valid only if re-identification is
+  genuinely infeasible (a pseudonym you can reverse is *not* anonymization).
+  Useful when you want aggregate analytics after the personal purpose ends.
+- **archive** — retain under Art. 89(1) safeguards for archival, statistical,
+  scientific, or statutory purposes (e.g. tax records). Move out of live
+  systems, restrict access, document the lawful long-term purpose.
+
+## Local + sector law validation checklist
+
+Run this before adopting any schedule. A copied default can be unlawful in a
+given jurisdiction or sector.
+
+- [ ] Each period checked against the relevant **statutory minimum** (tax,
+      employment, healthcare records) for every country you operate in.
+- [ ] Each period checked against the relevant **maximum** / minimization duty —
+      keeping longer than necessary is itself a violation.
+- [ ] Sector rules applied (healthcare 10y DE to 20y FR; financial-services and
+      regulated industries carry their own keep-periods).
+- [ ] CPRA / US-state rules handled where applicable: disclose the retention
+      period (or criteria) at or before collection, and don't keep longer than
+      reasonably necessary for the disclosed purpose.
+- [ ] AI reuse stated explicitly (purpose limitation, Art. 5(1)(b)).
+- [ ] Backups and archives covered by the same periods as live data.
+- [ ] DPO / privacy counsel sign-off obtained before adoption.
+
+---
+
+**Not legal advice.** Retention periods are jurisdiction- and sector-specific. A
+qualified DPO or privacy counsel must validate this schedule before it is
+adopted.