@damiangilgonzalez/harness 1.3.7

package/skills/dashboard/evals/README.md

@@ -0,0 +1,3 @@
+# Evals — dashboard
+
+`cases.yaml` is a human/LLM-graded trigger and capability set, not an auto-runner. To use it: read each `should_trigger` prompt and confirm an agent routing skills would reach into `dashboard`, then read each `should_not_trigger` prompt and confirm it routes to the named sibling instead (the boundary against `kpi-framework`, `analytics`, `reporting`, `forecasting`, `business-intelligence`, and `ab-testing`) — eyeball the routing, don't just trust the labels. For the `capability` case, have the agent actually produce a `dashboard.yaml` from the 14-metric scenario, then grade the output against the `must_include` rubric and run `scripts/verify.sh path/to/dashboard.yaml`; a real pass means the rubric items are all present AND verify.sh exits 0. verify.sh needs `python3` with PyYAML and is read-only, so it is safe to run on any candidate artifact.

package/skills/dashboard/evals/cases.yaml

@@ -0,0 +1,50 @@
+skill: dashboard
+
+should_trigger:
+  - prompt: "Build a KPI dashboard for our leadership team that fits on one screen."
+    why: Core build job — arrange a metric set into a scannable one-screen layout.
+  - prompt: "Nobody opens the dashboard we built — it has too many charts and you can't tell if we're healthy."
+    why: Non-obvious audit/fix symptom; the tile-budget + 5-second-read failure mode this skill owns.
+  - prompt: "Which chart should each of these 8 metrics use? Some are trends, some are target-vs-actual."
+    why: Chart-per-metric selection is a core capability (line vs bullet vs sorted bar).
+  - prompt: "Munta un dashboard de KPIs per a direcció amb el north-star a dalt."
+    why: Catalan phrasing for the build job, including the north-star placement rule.
+  - prompt: "El dashboard tiene 20 gráficos y nadie lo mira — recórtalo a lo que de verdad importa."
+    why: Spanish phrasing; tile-budget + Decision Test (cutting vanity tiles).
+  - prompt: "Turn this list of metrics into a tiered, scannable layout with one north-star tile."
+    why: Layout hierarchy / Z-F scan with a single entry point.
+  - prompt: "Give me a dashboard.yaml manifest a developer can render for our exec metrics."
+    why: Producing the checkable artifact (tile manifest) is the literal output of this skill.
+
+should_not_trigger:
+  - prompt: "Decide our north-star metric and build the metric tree with targets and owners."
+    route_to: kpi-framework
+    why: Defining which metrics matter and their targets — dashboard displays a metric set, it does not choose it.
+  - prompt: "Set up event tracking and funnels in PostHog so we can measure activation."
+    route_to: analytics
+    why: Instrumentation and data production — analytics produces the numbers the dashboard arranges.
+  - prompt: "Write the monthly board report with commentary on why churn rose."
+    route_to: reporting
+    why: A narrative document with prose, not a live at-a-glance screen.
+  - prompt: "Project our MRR for the next 4 quarters under three scenarios."
+    route_to: forecasting
+    why: Predicting a metric forward, not displaying current status.
+  - prompt: "Build a self-serve BI semantic layer so anyone can slice revenue by any dimension."
+    route_to: business-intelligence
+    why: Exploration / semantic modeling platform, not a fixed at-a-glance screen.
+  - prompt: "Read out our pricing A/B test — is variant B's lift significant?"
+    route_to: ab-testing
+    why: Experiment readout / variant significance, not dashboard layout.
+
+capability:
+  - scenario: >
+      Given a raw list of 14 candidate metrics for a SaaS exec team — including vanity
+      ones like total signups, total page views, and total followers — produce a
+      dashboard.yaml manifest plus a one-screen layout.
+    must_include:
+      - Applies the Decision Test and cuts or converts vanity metrics (total signups → activation rate; page views → conversion rate by source; total followers → follower-growth rate).
+      - Final manifest has 5-9 tiles with exactly one tile flagged north_star true.
+      - Every tile carries a comparison (target | prior | benchmark), an owner, a refresh cadence, and a non-empty decision field.
+      - Correct chart per metric shape (trend → line, target-vs-actual → bullet, category comparison → sorted bar, headline → big_number); no pie >5 slices and no gauge/dial.
+      - A layout block placing the north-star top-left with a Z or F scan and grouped tiles.
+      - Valid YAML that passes scripts/verify.sh (parses, budget, one north-star, required keys, no banned charts).

package/skills/dashboard/references/chart-selection.md

@@ -0,0 +1,34 @@
+# Chart selection — full matrix and edge cases
+
+Pick the chart from the **shape of the question the metric answers**, not from novelty. The body table covers the common cases; this is the full matrix plus the situations where the obvious choice is wrong.
+
+## Full metric-shape → chart matrix
+
+| Metric shape | Primary chart | Acceptable alternative | Avoid |
+| --- | --- | --- | --- |
+| One headline number, "where are we now" | Big-number tile + delta | Big number + sparkline | Gauge, dial |
+| Value over time, "which way is it going" | Line | Area (only if part-to-whole over time) | Many overlapping lines (>4) |
+| Compare a category across items | Sorted horizontal bar | Dot plot | Pie, vertical bars with long labels |
+| Actual vs a single target | Bullet chart | Bar with a target reference line | Gauge, dial, donut |
+| Part-to-whole, ≤5 parts | Stacked bar | Treemap, pie (3-5 slices only) | Pie >5, nested donuts |
+| Part-to-whole over time | 100% stacked area/bar | Small multiples of stacked bars | 3D area |
+| Relationship between two measures | Scatter | Bubble (adds a 3rd measure via size) | Dual-axis line |
+| Distribution of a measure | Histogram | Box plot | Single average number alone |
+| Progress toward a deadline/goal | Bullet or progress bar | Burn-up line | Gauge |
+| Geographic spread | Choropleth map | Sorted bar by region | 3D map, bubble-on-map clutter |
+
+## Edge cases where the obvious chart is wrong
+
+- **Sparkline inside a tile.** A north-star big number reads better with a tiny inline trend than with a separate chart tile. Use a sparkline when the *value* is the message and the *trend* is supporting context.
+- **Small multiples beat one busy chart.** Comparing the same trend across 6 segments? Don't stack 6 lines on one axis — render 6 small identical line charts in a grid. The eye compares shapes faster than it untangles a spaghetti chart.
+- **When a plain table beats every chart.** If the reader needs to look up exact values, compare more than ~3 precise numbers per row, or the data is inherently tabular (a ranked leaderboard with several columns), a clean table with right-aligned numbers and inline RYG cells wins. A chart that forces the reader back to a tooltip for the real number has failed.
+- **Log scale.** Use only when values span orders of magnitude (e.g. error counts from 1 to 100,000) AND the audience understands log axes. For an exec dashboard, prefer splitting the metric or using a different view — a log axis silently understates big swings to a non-technical reader.
+- **Annotating inflection points.** A trend line earns a one-line annotation at the moment something changed ("v2 launch", "price increase"). Why: the reader's first question at any bend is "what happened here"; answer it on the chart, not in a footnote.
+- **Sorted bars for "who missed target".** When the question is accountability ("which regions are below plan"), sort bars by gap-to-target and color the misses. Don't sort alphabetically — sorting *is* the answer.
+- **Dual axis: almost never.** Two y-axes on one chart invent correlations by letting you scale each line to "look related". If two metrics must be compared, normalize both to an index (base 100) and plot on one axis, or use two stacked charts.
+
+## Color discipline
+
+- Reserve red/yellow/green strictly for **status**. If everything is colored, nothing signals.
+- Use a single neutral hue for non-status series; differentiate by position and labels, not by a rainbow.
+- Ensure status is not encoded by color alone — pair RYG with an icon or shape so the screen stays readable for color-blind viewers (visual polish lives in [design](../../design/SKILL.md)).

package/skills/dashboard/references/tile-schema.md

@@ -0,0 +1,164 @@
+# Tile schema — `dashboard.yaml` field reference
+
+The artifact `dashboard.yaml` is a manifest a BI tool, a web app, or a human can render. This is the full field-by-field reference, two filled examples, and exactly what `scripts/verify.sh` enforces.
+
+## Top-level keys
+
+| Key | Required | Type | Notes |
+| --- | --- | --- | --- |
+| `title` | yes | string | Dashboard name shown to the reader |
+| `date_range` | yes | string | Active window, e.g. `"2026-05"` or `"last 90d"` |
+| `tiles` | yes | list | 1-9 entries (budget rule; verify.sh fails >9) |
+| `layout` | yes | map | Placement + scan pattern (below) |
+
+## Tile fields
+
+| Field | Required | Type | Notes |
+| --- | --- | --- | --- |
+| `id` | yes | string | Unique slug, referenced by drill-downs |
+| `north_star` | one tile only | bool | Exactly one tile sets `true`; verify.sh enforces |
+| `metric` | yes | string | Human label of the KPI |
+| `chart` | yes | enum | `big_number`, `line`, `sorted_bar`, `bullet`, `stacked_bar`, `treemap`, `scatter`, `histogram`. Banned: `pie` >5 cats, `gauge`, `dial` |
+| `value_source` | recommended | string | Where the number comes from (table/query/sheet) — analytics owns this |
+| `units` | recommended | enum/string | `percent`, `currency`, `count`, `months`, etc. |
+| `comparison` | yes | map | `{ type: target\|prior\|benchmark, value: <n or ref> }` |
+| `delta` | recommended | string | What the change is measured against, e.g. `vs_prior_week` |
+| `status_logic` | recommended | string | The RYG rule, e.g. `"green ≥ target; yellow ≥ 90%; red < 90%"` |
+| `owner` | yes | string | Accountable human/role |
+| `refresh` | yes | enum | `realtime`, `hourly`, `daily`, `weekly`, `monthly` — match to decision cadence |
+| `decision` | yes | string | The Decision Test answer: what action this number drives. Must be non-empty |
+
+## `layout` block
+
+| Field | Type | Notes |
+| --- | --- | --- |
+| `north_star` | enum | `top_left` (recommended) — where the entry-point tile sits |
+| `scan` | enum | `Z` or `F` reading pattern |
+| `groups` | list | Named clusters of related tiles, e.g. `[growth, revenue, health]` |
+
+## Example A — SaaS exec (the body's worked example, extended)
+
+```yaml
+title: SaaS Exec Dashboard
+date_range: "2026-05"
+tiles:
+  - id: waa
+    north_star: true
+    metric: Weekly Active Accounts
+    chart: big_number
+    units: accounts
+    comparison: { type: target, value: 1200 }
+    delta: vs_prior_week
+    status_logic: "green ≥ target; yellow ≥ 90%; red < 90%"
+    owner: VP Product
+    refresh: daily
+    decision: "Below 90% of target two weeks running escalates activation work."
+  - id: nrr
+    metric: Net Revenue Retention
+    chart: big_number
+    units: percent
+    comparison: { type: benchmark, value: 110 }
+    owner: VP Sales
+    refresh: monthly
+    decision: "Below 100% triggers a save/expansion review."
+  - id: churn
+    metric: Monthly Logo Churn
+    chart: line
+    units: percent
+    comparison: { type: target, value: 3.0 }
+    owner: Head of CS
+    refresh: daily
+    decision: "Above target investigates cohort + reason codes."
+  - id: runway
+    metric: Cash Runway
+    chart: bullet
+    units: months
+    comparison: { type: target, value: 18 }
+    owner: CFO
+    refresh: monthly
+    decision: "Under 12 months opens fundraise / cost review."
+  - id: pipeline
+    metric: New Pipeline by Source
+    chart: sorted_bar
+    units: currency
+    comparison: { type: prior, value: last_month }
+    owner: Head of Marketing
+    refresh: weekly
+    decision: "Shift spend toward the top-converting source."
+layout:
+  north_star: top_left
+  scan: Z
+  groups: [growth, revenue, health]
+```
+
+## Example B — E-commerce ops
+
+```yaml
+title: E-commerce Ops Dashboard
+date_range: "last 28d"
+tiles:
+  - id: rev
+    north_star: true
+    metric: Net Revenue
+    chart: big_number
+    units: currency
+    comparison: { type: prior, value: prev_28d }
+    delta: vs_prior_period
+    status_logic: "green ≥ prior; yellow within -5%; red < -5%"
+    owner: Head of Ecom
+    refresh: daily
+    decision: "A red week triggers a promo / merchandising review."
+  - id: cvr
+    metric: Conversion Rate by Channel
+    chart: sorted_bar
+    units: percent
+    comparison: { type: benchmark, value: 2.5 }
+    owner: Growth Lead
+    refresh: daily
+    decision: "Reallocate ad spend toward channels above benchmark."
+  - id: aov
+    metric: Average Order Value
+    chart: line
+    units: currency
+    comparison: { type: prior, value: prev_28d }
+    owner: Merchandising
+    refresh: daily
+    decision: "Falling AOV prompts a bundle / upsell test."
+  - id: cac
+    metric: CAC vs Target
+    chart: bullet
+    units: currency
+    comparison: { type: target, value: 35 }
+    owner: Growth Lead
+    refresh: weekly
+    decision: "Over target pauses the worst-performing campaign."
+  - id: returns
+    metric: Return Rate
+    chart: line
+    units: percent
+    comparison: { type: target, value: 8.0 }
+    owner: Ops Manager
+    refresh: weekly
+    decision: "A rising trend opens a sizing/quality investigation."
+layout:
+  north_star: top_left
+  scan: F
+  groups: [acquisition, revenue, fulfillment]
+```
+
+## What `verify.sh` enforces (and what it does not)
+
+Enforces — **shape and discipline**:
+
+1. File parses as YAML.
+2. `tiles:` exists and holds 1-9 entries (fails if >9 — the budget rule).
+3. Exactly one tile has `north_star: true`.
+4. Every tile has non-empty `metric`, `chart`, `comparison`, `owner`, `refresh`, `decision`.
+5. No `chart: pie` with more than 5 categories; no `chart: gauge` or `chart: dial`.
+
+Does **not** judge:
+
+- Whether the chosen metrics are the *right* metrics — that is [kpi-framework](../../kpi-framework/SKILL.md).
+- Whether the numbers/values are correct — that is the data layer ([analytics](../../analytics/SKILL.md)).
+
+So a `dashboard.yaml` can pass verify.sh and still display the wrong metrics. verify.sh guards the craft of the dashboard, not the choice of what to measure.

package/skills/dashboard/scripts/verify.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# verify.sh — check a dashboard.yaml for SHAPE and DISCIPLINE, not correctness.
+# Read-only. Exits 0 on a clean/empty target (no false failures).
+#
+# Checks:
+#   1. file parses as YAML
+#   2. tiles: exists and holds 1-9 entries (fail if >9 — the budget rule)
+#   3. exactly one tile has north_star: true
+#   4. every tile has non-empty: metric, chart, comparison, owner, refresh, decision
+#   5. banned charts: no pie with >5 categories; no gauge/dial
+#
+# Does NOT judge metric choice or numeric values.
+#
+# Usage: verify.sh [path/to/dashboard.yaml]   (default: ./dashboard.yaml)
+
+set -euo pipefail
+
+TARGET="${1:-./dashboard.yaml}"
+
+if [[ ! -e "$TARGET" ]]; then
+  echo "verify: no dashboard.yaml at '$TARGET' — nothing to check. PASS (empty target)."
+  exit 0
+fi
+
+if [[ ! -s "$TARGET" ]]; then
+  echo "verify: '$TARGET' is empty — nothing to check. PASS (empty target)."
+  exit 0
+fi
+
+# Need a YAML parser. Prefer python3; fall back gracefully if absent.
+if ! command -v python3 >/dev/null 2>&1; then
+  echo "verify: python3 not found — cannot parse YAML on this machine."
+  echo "        Install python3 (with PyYAML) to run shape checks. SKIP (no false failure)."
+  exit 0
+fi
+
+python3 - "$TARGET" <<'PY'
+import sys
+
+path = sys.argv[1]
+
+try:
+    import yaml
+except ModuleNotFoundError:
+    print("verify: PyYAML not installed — cannot parse YAML.")
+    print("        Run: python3 -m pip install pyyaml  then re-run. SKIP (no false failure).")
+    sys.exit(0)
+
+errors = []
+warnings = []
+
+try:
+    with open(path) as fh:
+        doc = yaml.safe_load(fh)
+except yaml.YAMLError as e:
+    print(f"verify: FAIL — '{path}' is not valid YAML:\n{e}")
+    sys.exit(1)
+
+if doc is None:
+    print(f"verify: '{path}' parsed as empty. PASS (empty target).")
+    sys.exit(0)
+
+if not isinstance(doc, dict):
+    print("verify: FAIL — top level must be a mapping with a 'tiles:' key.")
+    sys.exit(1)
+
+tiles = doc.get("tiles")
+if not isinstance(tiles, list) or not tiles:
+    print("verify: FAIL — 'tiles:' must be a non-empty list.")
+    sys.exit(1)
+
+# Tile budget
+n = len(tiles)
+if n > 9:
+    errors.append(f"tile budget: {n} tiles exceeds the 9-tile ceiling — tier the extras to a drill-down.")
+elif n < 5:
+    warnings.append(f"tile budget: only {n} tiles — fine, but exec dashboards usually carry 5-9.")
+
+REQUIRED = ["metric", "chart", "comparison", "owner", "refresh", "decision"]
+BANNED_CHARTS = {"gauge", "dial"}
+
+north_star_count = 0
+for i, tile in enumerate(tiles):
+    label = f"tile[{i}]"
+    if not isinstance(tile, dict):
+        errors.append(f"{label}: not a mapping.")
+        continue
+    tid = tile.get("id", label)
+    label = f"tile '{tid}'"
+
+    if tile.get("north_star") is True:
+        north_star_count += 1
+
+    for key in REQUIRED:
+        val = tile.get(key)
+        if val is None or (isinstance(val, str) and not val.strip()) or val == {} or val == []:
+            errors.append(f"{label}: missing or empty required field '{key}'.")
+
+    chart = tile.get("chart")
+    if isinstance(chart, str):
+        c = chart.strip().lower()
+        if c in BANNED_CHARTS:
+            errors.append(f"{label}: chart '{chart}' is banned — use a bullet chart instead.")
+        if c == "pie":
+            cats = tile.get("categories")
+            if isinstance(cats, int) and cats > 5:
+                errors.append(f"{label}: pie with {cats} categories — over 5 is unreadable; use a sorted bar.")
+            elif isinstance(cats, list) and len(cats) > 5:
+                errors.append(f"{label}: pie with {len(cats)} categories — over 5 is unreadable; use a sorted bar.")
+            else:
+                warnings.append(f"{label}: pie chart — keep it to 3-5 slices or switch to a sorted bar.")
+
+if north_star_count == 0:
+    errors.append("north_star: no tile flagged 'north_star: true' — a dashboard needs one entry point.")
+elif north_star_count > 1:
+    errors.append(f"north_star: {north_star_count} tiles flagged 'north_star: true' — there must be exactly one.")
+
+for w in warnings:
+    print(f"verify: warn — {w}")
+
+if errors:
+    print()
+    for e in errors:
+        print(f"verify: FAIL — {e}")
+    print(f"\nverify: {len(errors)} problem(s) found in '{path}'.")
+    sys.exit(1)
+
+print(f"verify: PASS — '{path}' has {n} tile(s), one north-star, all required fields, no banned charts.")
+sys.exit(0)
+PY

package/skills/data-cleaning/SKILL.md

@@ -0,0 +1,259 @@
+---
+name: data-cleaning
+description: "Use when a raw table is too dirty to trust — nulls, sentinels, duplicate rows, category sprawl, mixed types, bad dates — and you need a re-runnable clean() plus a schema gate that fails loud. NOT emitting .xlsx (that is spreadsheet-ops), NOT acquiring rows (that is data-scraper), NOT parsing PDF/HTML into rows (that is structured-extraction)."
+tags: [data-cleaning, data-quality, pandas, pandera, validation, deduplication, reproducibility, polars]
+recommends: [duckdb, spreadsheet-ops, structured-extraction, data-scraper, analytics, business-intelligence, forecasting, python]
+profiles: []
+origin: risco
+---
+
+# Data cleaning — make dirty data trustworthy, and make the cleaning auditable
+
+A clean table is **typed + deduped + normalized + validated + reproducible**. The deliverable here is
+never "I opened a notebook and fixed some rows by hand." It is a re-runnable function `clean(raw) -> df`
+plus a **schema gate** that fails loud when next month's file violates the contract. Reproducible means
+the same input always yields the same output: versions pinned, sorts deterministic, nothing random without
+a seed. If you can't re-run it tomorrow and get the identical result, you haven't cleaned the data — you've
+edited a snapshot.
+
+Cleaning **starts** once you hold tabular rows and **ends** at a validated table/DataFrame/Parquet. Before
+that boundary the job is acquisition ([`data-scraper`](../data-scraper/SKILL.md),
+[`structured-extraction`](../structured-extraction/SKILL.md)); after it, consumption
+([`spreadsheet-ops`](../spreadsheet-ops/SKILL.md), [`analytics`](../analytics/SKILL.md),
+[`business-intelligence`](../business-intelligence/SKILL.md),
+[`forecasting`](../forecasting/SKILL.md)). Multi-GB analytical SQL is an engine choice, not a cleaning one
+— [`duckdb`](../duckdb/SKILL.md).
+
+Current stack (verified 2026-06-02): **pandas 3.0.x** (3.0.0 shipped 2026-01-21) and **pandera 0.31.1**
+(supports pandas ≥ 3) for in-pipeline schema validation; **Polars** and **DuckDB** when pandas runs out of
+RAM. Pin them: `pandas==3.0.3`, `pandera==0.31.1`.
+
+## The pipeline shape
+
+One canonical order. Each step is positioned for a reason, not by habit.
+
+```python
+import pandas as pd
+
+def clean(raw_path: str) -> pd.DataFrame:
+    df = read_typed(raw_path)     # 1. read with explicit dtypes — never let pandas guess
+    df = normalize(df)            # 2. strings/categories/numbers/dates — collapse invisible variance
+    df = dedupe(df)               # 3. AFTER normalize+type, so "1"/1 and "US "/"US" actually collapse
+    df = handle_missing(df)       # 4. decide per column: drop / impute+flag / leave NA / quarantine
+    df = Schema.validate(df, lazy=True)  # 5. the GATE — fail loud, surface every violation at once
+    return df
+```
+
+- **Type before dedupe** — otherwise `"1"` (string) and `1` (int) survive as two distinct keys.
+- **Normalize before dedupe** — `"US "` and `"US"` are the same customer; dedupe can't see that until
+  whitespace/case are collapsed.
+- **Validate last** — it is the gate, not a cleaning step. It asserts the contract holds *after* all fixes.
+- **Write to a NEW artifact** — the raw file is read-only; you never overwrite your only source.
+
+## Read it right
+
+The single most common reproducibility footgun: pandas' legacy numpy path silently casts an integer column
+containing one NaN to `float64`, so your `id` becomes `1001.0`. Control the dtype on read.
+
+```python
+# BAD — pandas guesses: ids become floats, "N/A" stays a string, "" is sometimes NaN sometimes ""
+df = pd.read_csv("raw.csv")
+
+# GOOD — explicit, deterministic, real nullable types
+df = pd.read_csv(
+    "raw.csv",
+    dtype_backend="pyarrow",          # real nullable ints/strings; no silent float-cast
+    na_values=["", "N/A", "NA", "null", "-1", "999"],  # YOUR sentinels become real NA
+    keep_default_na=True,             # keep pandas' default NA tokens too
+    encoding="utf-8",                 # state it; don't let locale decide
+)
+```
+
+Two pandas 3.0 facts that read depends on. `dtype_backend="pyarrow"` only works if pyarrow is actually
+installed — PDEP-14 deliberately kept a NumPy-object fallback so PyArrow stays *recommended, not required*
+— so `pip install pyarrow` for the faster backed path, or pass `dtype_backend="numpy_nullable"` when it is
+absent. And the default `str` dtype (PyArrow-backed when pyarrow is present, NumPy-object-backed otherwise)
+uses **`NaN` missing-value semantics** like every other default dtype: test for null with `pd.isna()`, never
+by comparing against whichever null token happened to appear.
+
+## Profile before you fix
+
+Let the numbers drive the plan, not a glance at `df.head()`. Run this first, every time.
+
+```python
+def profile(df: pd.DataFrame) -> pd.DataFrame:
+    return pd.DataFrame({
+        "dtype":     df.dtypes.astype(str),
+        "null_pct":  (df.isna().mean() * 100).round(1),
+        "n_unique":  df.nunique(dropna=True),       # cardinality — catches category sprawl
+        "sample":    df.apply(lambda s: s.dropna().unique()[:3].tolist()),
+    })
+
+print(profile(df))
+print("rows:", len(df), "exact dupes:", df.duplicated().sum())
+```
+
+A column at 90% null is a drop candidate; one with 400 distinct "countries" needs a mapping table; an
+"age" with min `-1`/max `999` has sentinels to map. The profile is your TODO list.
+
+## Normalize
+
+Each fix below: **Bad → Good**, with a one-line why.
+
+**Strings** — invisible variance (trailing space, mixed case, lookalike unicode) silently breaks joins
+and dedupe.
+
+```python
+# BAD: "US ", "us", "ｕｓ" all look different to a join
+# GOOD:
+s = df["country"].str.strip().str.casefold().str.normalize("NFKC")
+```
+
+**Categories** — use a **mapping table**, never a tower of regex. A dict is auditable and an *unmapped*
+value gets quarantined instead of silently passing through.
+
+```python
+COUNTRY = {"usa": "US", "u.s.": "US", "united states": "US", "u.s.a.": "US", "es": "ES", "españa": "ES"}
+key = df["country"].str.strip().str.casefold()
+df["country"] = key.map(COUNTRY)            # unmapped -> NA, which the gate below will catch (no silent pass)
+```
+
+**Numbers** — turn sentinels into NA, then choose a range policy explicitly: *clip* (cap to bound) when
+out-of-range is plausibly a recording cap, *reject* (→ NA / quarantine) when it is impossible.
+
+```python
+df["age"] = df["age"].mask(df["age"].isin([-1, 999]))   # sentinels -> NA
+df["age"] = df["age"].clip(lower=0, upper=120)          # clip policy; or .mask(~df["age"].between(0,120)) to reject
+```
+
+**Dates** — state the `format`, coerce, then **count the casualties**. Never trust dayfirst inference;
+`03/04/2026` is ambiguous and pandas will pick silently.
+
+```python
+parsed = pd.to_datetime(df["signup"], format="%Y-%m-%d", utc=True, errors="coerce")
+bad = parsed.isna() & df["signup"].notna()
+assert bad.sum() == 0, f"{bad.sum()} dates failed the expected format — inspect before proceeding"
+df["signup"] = parsed
+```
+
+Copy-paste versions of all of these — category mapping with unmapped→quarantine, a robust date parser,
+unicode/encoding repair, a sentinel→NA table, numeric clip-vs-reject, plus Polars equivalents — are in
+[references/normalization-recipes.md](references/normalization-recipes.md).
+
+## Dedupe
+
+`drop_duplicates(keep="first")` is meaningless without a defined key and a stable sort — "first" of what
+order? Define both.
+
+```python
+key = ["customer_id"]                                   # the BUSINESS key, stated explicitly
+df = (df.sort_values(["customer_id", "updated_at"], ascending=[True, False], kind="stable")
+        .drop_duplicates(subset=key, keep="first"))     # keep most-recent per customer, deterministically
+```
+
+Near-duplicates (`"Acme Inc"` vs `"Acme, Inc."`) are a *normalization* problem — collapse them in the
+normalize step first; only then does exact dedupe catch them. Fuzzy matching is a separate, riskier
+decision — make it visible, never automatic.
+
+## Missing values — decide per column
+
+No silent `fillna(0)`: a zero is a value, and treating "unknown" as zero poisons every mean, sum, and model
+downstream. Pick deliberately.
+
+| Situation | Action | Why |
+| --- | --- | --- |
+| Column is mostly null (e.g. >70%) and not load-bearing | Drop the **column** | Imputing it invents signal that isn't there |
+| A few rows missing a *required* key (id, date) | Drop the **row** (and log/quarantine) | Can't dedupe or join without the key |
+| Numeric gap you must fill for a model | Impute **and add a `_was_missing` flag** | The model can learn "was missing"; you keep the audit trail |
+| Genuinely optional field | **Leave NA** | NA is information; don't fabricate a value |
+| Value is present but *invalid* (unmapped category, bad date) | **Quarantine the row** | Don't drop silently and don't let it pass the gate |
+
+```python
+df["income_was_missing"] = df["income"].isna()
+df["income"] = df["income"].fillna(df["income"].median())   # impute + flag, never bare fillna(0)
+```
+
+## Validate — the gate
+
+This is where cleaning becomes *trustworthy*. Declare the contract as a pandera `DataFrameModel`, validate
+**output** (and input expectations where they exist), and split valid rows from failures instead of
+crashing — the failures become your quarantine.
+
+```python
+import pandera.pandas as pa
+from pandera.typing import Series
+
+class CustomerSchema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(unique=True, ge=1)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])     # only mapped categories survive
+    age:         Series[float] = pa.Field(ge=0, le=120, nullable=True)
+    signup:      Series[pa.DateTime] = pa.Field(nullable=False)
+
+    class Config:
+        strict = True       # reject unexpected columns
+        coerce = True       # coerce to declared dtype, fail loud if impossible
+
+# lazy=True collects EVERY violation at once instead of dying on the first
+try:
+    valid = CustomerSchema.validate(df, lazy=True)
+except pa.errors.SchemaErrors as e:
+    failures = e.failure_cases          # dataframe of exactly which rows/checks failed
+    failures.to_parquet("quarantine.parquet")   # keep, don't drop — someone investigates these
+    valid = df.drop(index=e.failure_cases["index"].dropna().unique())  # proceed with the clean subset
+```
+
+`coerce=True` fixes types the contract expects; `nullable` states which columns may hold NA; field
+`Check`s (`ge`, `le`, `isin`, `unique`) are the allowed-value rules. `strict` catches columns that
+shouldn't be there. Together they are the data contract in code. Log the row-count diff on every run —
+`in`, `out`, coerced, quarantined — so what the pipeline changed is an auditable record, not an assumption.
+
+When to escalate beyond pandera: reach for **GX Core 1.0** (Great Expectations' rebranded OSS — Data
+Context → Data Source → Expectation Suite → Validation Definition → Checkpoint) when you need a *shared
+data-quality platform* across many datasets and teams with a results store and docs. Use **dbt model
+contracts** (enforced at build) plus **dbt tests** (post-materialization) when the cleaning lives in a
+SQL warehouse, not Python. The full `DataFrameModel` (custom `@pa.check`, lazy `SchemaErrors` report,
+valid/quarantine split helper), the GX checkpoint sketch, the dbt model-contract + `data_tests` YAML, and
+the "which validator" chooser are in [references/validation-patterns.md](references/validation-patterns.md).
+
+## Scale — when pandas hurts
+
+Heuristic: pandas is fine while the data fits comfortably in RAM (roughly ≤ 1–2 GB working set). Beyond
+that, or when a groupby/join dominates the runtime, switch the *mechanics* (not the principles):
+
+- **Polars** for clean-at-scale: `pl.scan_csv(...)` (lazy, parallel, Rust), then `.unique()`,
+  `.drop_nulls()`, `.fill_null(...)`, `.str.*` — the same profile→normalize→dedupe→validate shape, faster.
+  pandera validates Polars frames too, and the
+  [recipes reference](references/normalization-recipes.md) has the Polars equivalent of every fix above.
+- **DuckDB** when the bottleneck is analytical SQL over multi-GB files — point heavy joins/aggregations
+  there: [`duckdb`](../duckdb/SKILL.md). It is an *engine* choice; correctness/normalization is still this
+  skill's job.
+
+## Anti-patterns
+
+| Anti-pattern | Why it breaks |
+| --- | --- |
+| "`fillna(0)` to get rid of the nulls" | Zero is a value; it distorts every mean/sum/model. Impute deliberately and add a `_was_missing` flag. |
+| "`drop_duplicates()` — done" | No `subset`, no sort → which row survives is nondeterministic. Define the key, `sort_values(kind="stable")`, set `keep`. |
+| "`pd.read_csv(path)` and start cleaning" | pandas guesses: ids become floats, dates become strings. Pass `dtype_backend` + `na_values`. |
+| "I fixed the rows in a notebook cell" | Not reproducible — next month's file gets nothing. Wrap it in `clean(raw) -> df`. |
+| "Drop the rows that look wrong" | Silent data loss with no audit trail. Quarantine to a file; someone investigates. |
+| "A few regexes will normalize the countries" | Unmaintainable and silent on new values. Use a mapping dict; unmapped → NA → caught by the gate. |
+| "`pd.to_datetime` figures out the format" | Ambiguous dates parse silently wrong. State `format=`, `errors="coerce"`, then assert the NaT count. |
+| "Validation passed, so we're good" | A gate that never fails is a no-op. Feed it a known-bad row and confirm it *rejects*. |
+| "It's slow, rewrite everything in Polars" | Switch the engine, not the discipline — profile→normalize→dedupe→validate still applies. |
+
+## Verify
+
+`scripts/verify.sh` runs from anywhere, no network. It does static structure checks on this skill
+(frontmatter keys, references present) always, and — when pandas + pandera are installed — extracts the
+documented pattern, feeds it one clearly-good row and one clearly-bad row, and asserts the good row PASSES
+validation while the bad row is FLAGGED/quarantined, proving the gate is not a no-op. Without
+pandas/pandera it prints SKIP for the runtime check and still passes the static checks.
+
+## Project grounding (docs + CLAUDE.md)
+
+In a project with a `docs/` layer (the [`harness`](../harness/SKILL.md) wiki), record this dataset's
+cleaning decisions — the schema/contract, the category mapping tables, the dedupe key, the quarantine
+location, version pins — in `docs/wiki/data/<dataset>.md`, link it from the root `CLAUDE.md`
+`## Knowledge map`, and read it first on every re-run so the contract stays consistent. No `docs/`? Skip
+silently. Conventions are recorded, never gated.

package/skills/data-cleaning/evals/README.md

@@ -0,0 +1,16 @@
+# Eval harness — `data-cleaning`
+
+`cases.yaml` is a human/LLM-graded rubric, not an automated runner. It grades two things: **triggering**
+(does the skill fire on real messy-data asks and stay quiet on near-misses that belong to a sibling?) and
+**capability** (does loading the skill produce a measurably better cleaning pipeline?). The mechanical
+check — that the documented validation gate actually rejects a bad row — lives separately in
+`scripts/verify.sh`, which these cases do not duplicate.
+
+To run triggering: start a fresh agent with only `data-cleaning` discoverable, paste each `should_trigger`
+prompt verbatim, and confirm it reaches for this skill (3–5 trials each, since the decision is stochastic;
+pass if it fires in the majority). For each `should_not_trigger`, confirm it does **not** fire and would
+plausibly route to the named `route_to` sibling (some siblings may not be built yet — the routing intent is
+what's graded). To run capability: give the scenario to a clean agent WITHOUT the skill and again WITH it,
+then grade each transcript against the `must_include` list, one point per bullet that is specifically and
+correctly covered (not merely name-dropped). WITH should clearly beat WITHOUT. Record trial counts and
+per-bullet verdicts so a reviewer can audit; don't report a bare pass/fail.