@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/csharp-dotnet/references/aspnetcore.md

@@ -0,0 +1,99 @@
+# ASP.NET Core 10 — full catalog
+
+The body has the decision and the canonical shape; this is the depth.
+
+## Minimal APIs vs controllers
+
+| Pick minimal APIs when | Pick controllers when |
+|---|---|
+| New service, REST/JSON endpoints, route groups suffice | You need view rendering (MVC/Razor) |
+| You want the least ceremony and AOT-friendliness | You rely on the full filter pipeline, conventions, or `ApiController` model-binding niceties at scale |
+| Endpoints map cleanly to delegates | Large teams already standardized on controllers |
+
+They coexist in one app. Default to minimal for new code; state the reason if you choose controllers.
+
+## Route groups + endpoint filters
+
+```csharp
+var products = app.MapGroup("/products")
+    .WithTags("Products")
+    .RequireAuthorization()                 // applies to the whole group
+    .AddEndpointFilter<ValidationFilter>(); // cross-cutting check per request
+
+products.MapGet("/{id:int}", GetById);
+products.MapPost("/", Create);
+```
+
+An endpoint filter runs around the handler — use it for validation, logging, or shaping results without repeating code per endpoint.
+
+```csharp
+public sealed class ValidationFilter : IEndpointFilter
+{
+    public async ValueTask<object?> InvokeAsync(EndpointFilterInvocationContext ctx, EndpointFilterDelegate next)
+    {
+        // inspect ctx.Arguments, short-circuit with Results.ValidationProblem(...) if invalid
+        return await next(ctx);
+    }
+}
+```
+
+## Validation (.NET 10)
+
+In .NET 10, DataAnnotations on minimal API parameters — query, header, and body, **including records** — are validated automatically; a failure produces a ProblemDetails 400 without manual code. Annotate the DTO and you are done for simple cases; use an endpoint filter or FluentValidation for cross-field rules.
+
+```csharp
+public record CreateProduct(
+    [property: Required, MaxLength(120)] string Name,
+    [property: Range(0.0, 1_000_000.0)] decimal Price);
+```
+
+## ProblemDetails (RFC 9457)
+
+`AddProblemDetails()` + `UseExceptionHandler()` turn unhandled exceptions and status-code responses into a consistent machine-readable error body. Never return raw exception text to clients. Use `Results.Problem(...)` / `Results.ValidationProblem(...)` for explicit error shaping.
+
+## OpenAPI 3.1
+
+`Microsoft.AspNetCore.OpenAPI` generates an **OpenAPI 3.1** document (JSON Schema draft 2020-12) at build/runtime:
+
+```csharp
+builder.Services.AddOpenApi();
+app.MapOpenApi();               // serves /openapi/v1.json
+```
+
+Annotate with `.WithTags(...)`, `.WithName(...)`, `.Produces<T>(...)`, `.WithSummary(...)` for a richer document. Pair with a UI (Scalar, Swagger UI) only in non-prod by default.
+
+## Middleware ordering
+
+Order matters — each component sees the request on the way in and the response on the way out. A workable baseline:
+
+```text
+UseExceptionHandler  ->  HSTS / HttpsRedirection  ->  static assets
+  ->  routing  ->  CORS  ->  Authentication  ->  Authorization  ->  endpoints
+```
+
+Authentication must precede Authorization. Exception handling goes first so it wraps everything below it.
+
+## DI lifetimes
+
+| Lifetime | Instances | Use for |
+|---|---|---|
+| `Singleton` | one per app | stateless services, caches, `IHttpClientFactory` |
+| `Scoped` | one per request | `DbContext`, per-request state |
+| `Transient` | one per resolve | lightweight stateless helpers |
+
+**Captive dependency**: injecting a `Scoped` (or `Transient` holding scoped state) into a `Singleton` captures it for the app lifetime — a reused `DbContext` across requests corrupts change tracking. Inject `IServiceScopeFactory` and create a scope per unit of work instead, or make the consumer scoped. The DI container's scope validation catches many of these at startup in development.
+
+## Configuration / options pattern
+
+Bind config sections to typed options instead of reading magic strings:
+
+```csharp
+builder.Services.Configure<ShopOptions>(builder.Configuration.GetSection("Shop"));
+// inject IOptions<ShopOptions> (singleton snapshot) or IOptionsSnapshot<ShopOptions> (per-request reload)
+```
+
+Use `IOptionsSnapshot<T>` in scoped services when values can change at runtime; `IOptions<T>` otherwise.
+
+## Auth defaults
+
+ASP.NET Core 10 is secure-by-default. Still: configure authentication then authorization in the right middleware order, prefer policy-based authorization (`RequireAuthorization("policy")`) over scattered role checks, enforce HTTPS redirection + HSTS in production, and keep token/cookie protection on the Data Protection key ring (persisted and shared across instances). Threat modeling beyond these knobs is `secure-coding`.

package/skills/shared/csharp-dotnet/references/async.md

@@ -0,0 +1,82 @@
+# async/await in .NET — full catalog
+
+The body has the rules; this is the depth. Read before any non-trivial async review.
+
+## Task vs ValueTask
+
+- Default to `Task`/`Task<T>`. It composes everywhere and is hard to misuse.
+- `ValueTask<T>` avoids an allocation when the method usually completes synchronously
+  (a cache hit, a buffered read). Constraints: **await it exactly once**, never block on
+  it, never store it. Convert with `.AsTask()` if you need to await twice or `WhenAll` it.
+
+```csharp
+// Good: hot path that is usually a cache hit
+public ValueTask<User> GetAsync(int id, CancellationToken ct) =>
+    _cache.TryGetValue(id, out var u) ? new ValueTask<User>(u) : new ValueTask<User>(LoadAsync(id, ct));
+```
+
+## ConfigureAwait
+
+- **Libraries**: `ConfigureAwait(false)` on every await. Why: do not capture and force a
+  resumption context onto callers; avoids deadlocks if a caller blocks.
+- **ASP.NET Core app code**: there is no `SynchronizationContext`, so `ConfigureAwait`
+  is a no-op — do not litter app code with it. The deadlock risk comes from *blocking*,
+  not from omitting `ConfigureAwait`.
+
+## CancellationToken propagation
+
+Flow one token from the entry point through every async call. Pass it last by convention.
+
+```csharp
+public async Task<Report> BuildAsync(int id, CancellationToken ct)
+{
+    var data = await _repo.LoadAsync(id, ct);          // DB
+    var enriched = await _http.GetAsync(data.Url, ct); // outbound HTTP
+    ct.ThrowIfCancellationRequested();                 // explicit check before expensive CPU work
+    return Render(enriched);
+}
+```
+
+`CancellationToken.None` is the explicit "this genuinely cannot be cancelled" signal —
+use it deliberately, never as a shortcut to avoid plumbing.
+
+## Deadlock catalog
+
+| Trigger | Mechanism | Fix |
+|---|---|---|
+| `.Result` / `.Wait()` blocking on async | Thread blocks waiting for a continuation that needs that thread (classic context deadlock) or starves the thread pool under load | `await` |
+| `Task.Run` to "make sync code async" then await | Burns a thread pool thread for I/O-bound work | Use a truly async API |
+| Sync-over-async deep in a call chain | One blocking call poisons the whole request | Make the whole chain async |
+
+The symptom users report: "my web request hangs / the app freezes under load." It is
+almost always a blocking call on an async method. Search for `.Result`, `.Wait(`,
+`.GetAwaiter().GetResult()`.
+
+## IAsyncEnumerable
+
+Stream rows instead of materializing a giant `List<T>`:
+
+```csharp
+public async IAsyncEnumerable<OrderRow> StreamAsync([EnumeratorCancellation] CancellationToken ct)
+{
+    await foreach (var o in _db.Orders.AsAsyncEnumerable().WithCancellation(ct))
+        yield return new OrderRow(o.Id, o.Total);
+}
+```
+
+`[EnumeratorCancellation]` lets `await foreach (... .WithCancellation(ct))` flow the token in.
+
+## Parallelism
+
+- **Independent awaits**: `await Task.WhenAll(a, b, c)` — runs concurrently, surfaces the
+  first exception (inspect `Task.Exception` for all).
+- **Bounded concurrency over a collection**: `Parallel.ForEachAsync` with
+  `MaxDegreeOfParallelism`. Do not fire 10k tasks at once.
+- **Producer/consumer pipelines**: `System.Threading.Channels` (`Channel<T>`).
+- **Synchronization** (C# 13+): use the `System.Threading.Lock` type instead of locking
+  on an arbitrary `object`; it has a clearer `EnterScope()` API and better diagnostics.
+
+```csharp
+private readonly System.Threading.Lock _gate = new();   // C# 13+
+public void Touch() { lock (_gate) { _count++; } }
+```

package/skills/shared/csharp-dotnet/references/efcore.md

@@ -0,0 +1,96 @@
+# EF Core 10 — full catalog
+
+The body has the rules; this is the depth. Raw DB schema/index tuning is `postgresdb`.
+
+## DbContext lifetime
+
+- Register with `AddDbContext<T>` — **scoped** (one per request). It is not thread-safe.
+- Never share a `DbContext` across concurrent awaits or background threads. For parallel
+  work, create a context per unit via `IDbContextFactory<T>` (`AddDbContextFactory`).
+- For short-lived background jobs, resolve a fresh scope and a fresh context per job.
+
+```csharp
+builder.Services.AddDbContext<ShopDb>(o => o.UseNpgsql(cs));
+// parallel/background work:
+builder.Services.AddDbContextFactory<ShopDb>(o => o.UseNpgsql(cs));
+```
+
+## Query patterns
+
+```csharp
+// Read path: no tracking, project straight to a DTO (selects only needed columns).
+var rows = await db.Orders.AsNoTracking()
+    .Where(o => o.Status == OrderStatus.Open)
+    .Select(o => new OrderRow(o.Id, o.Customer.Name, o.Lines.Count))
+    .ToListAsync(ct);
+
+// Write path: tracked entity, mutate, SaveChangesAsync with the token.
+var order = await db.Orders.FindAsync([id], ct);
+order!.Status = OrderStatus.Closed;
+await db.SaveChangesAsync(ct);
+```
+
+- `AsNoTracking()` on every read-only query — drops change-tracking memory/CPU.
+- Use the async query operators (`ToListAsync`, `FirstOrDefaultAsync`, `AnyAsync`,
+  `SaveChangesAsync`) and flow the token into all of them.
+- Projection (`Select` to a DTO) is usually better than loading whole entities: less data
+  over the wire, no accidental over-fetch.
+
+## Tracking and N+1
+
+The N+1 trap: load a list, then touch a navigation property inside a loop — EF issues one
+extra query per row. Symptom: "my page does hundreds of tiny queries."
+
+```csharp
+// Bad: N+1
+var orders = await db.Orders.ToListAsync(ct);
+foreach (var o in orders) { var c = o.Customer.Name; } // one query per order
+
+// Good: projection (single JOIN) or Include
+var orders = await db.Orders.Include(o => o.Customer).ToListAsync(ct);
+```
+
+Use `AsSplitQuery()` when a single query with multiple collection `Include`s causes a
+cartesian-product explosion (rows multiply across joins). Trade-off: more round-trips,
+smaller result sets.
+
+## Migrations workflow
+
+```bash
+dotnet ef migrations add AddProductPrice -p Shop.Api
+# review the generated Up()/Down() — they are real code you ship
+dotnet ef database update -p Shop.Api
+dotnet ef migrations script --idempotent -o migrate.sql   # for controlled prod deploys
+```
+
+- Always review the generated migration before applying. EF guesses; you confirm.
+- Never hand-edit a migration that has already been applied to any shared environment —
+  add a new one.
+- For production, prefer an idempotent SQL script run by your deploy pipeline over
+  `database update` at app startup. Deploy mechanics are `deployment`.
+
+## EF Core 10 highlights
+
+- **Native `json` column type** — map a property to a real JSON column with first-class
+  querying, instead of serializing to a string manually.
+- **`vector` type + `VECTOR_DISTANCE()`** — store embeddings and run similarity search in
+  the database for RAG/semantic features (provider support required).
+- **`LeftJoin` / `RightJoin` LINQ operators** — express outer joins directly instead of
+  the old `GroupJoin` + `SelectMany` + `DefaultIfEmpty` dance.
+- **Named query filters** — multiple global query filters per entity, each named so you
+  can selectively disable one (e.g. ignore soft-delete but keep tenant filter) via
+  `IgnoreQueryFilters([...])`.
+
+## Raw SQL safety
+
+Use only parameterized interpolation — EF turns the holes into parameters:
+
+```csharp
+// Good: parameterized, injection-safe
+var products = await db.Products
+    .FromSql($"SELECT * FROM products WHERE name = {name}")   // {name} becomes a parameter
+    .ToListAsync(ct);
+```
+
+Never `FromSqlRaw($"... WHERE name = '{name}'")` with string interpolation/concatenation —
+that is SQL injection. Reserve raw SQL for cases LINQ cannot express; otherwise stay in LINQ.

package/skills/shared/csharp-dotnet/scripts/verify.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+#
+# verify.sh - local quality gate for a .NET solution or project (superset of CI).
+#
+# Usage:
+#   cd <your-solution-or-project-root>   # dir with a .sln or .csproj
+#   ./verify.sh
+#
+# Runs: dotnet format --verify-no-changes (style), dotnet build -warnaserror
+# (compile + analyzers + NRT warnings as errors), dotnet test (xUnit/NUnit),
+# and dotnet list package --vulnerable --include-transitive (NuGet vuln scan).
+# A missing `dotnet` is skipped with a yellow warning (not a failure).
+# Real problems (unformatted code, build/test failures, known vulnerabilities)
+# exit non-zero. Read-only: never mutates your source.
+#
+# Portability: runs on stock macOS bash 3.2 (no mapfile, no associative arrays).
+
+set -euo pipefail
+
+# Colors only when stdout is a TTY (keeps logs/CI output clean).
+if [ -t 1 ]; then
+  YELLOW=$'\033[33m'; RED=$'\033[31m'; GREEN=$'\033[32m'; RESET=$'\033[0m'
+else
+  YELLOW=''; RED=''; GREEN=''; RESET=''
+fi
+
+failed=0
+
+have() { command -v "$1" >/dev/null 2>&1; }
+warn() { printf '%s[skip]%s %s\n' "$YELLOW" "$RESET" "$*"; }
+fail() { printf '%s[fail]%s %s\n' "$RED" "$RESET" "$*"; failed=1; }
+ok()   { printf '%s[ ok ]%s %s\n' "$GREEN" "$RESET" "$*"; }
+info() { printf -- '----- %s\n' "$*"; }
+
+# Must run from a solution or project root. An empty/clean dir with no project
+# is not a failure - nothing to check, exit 0.
+sln_count=$( (ls ./*.sln ./*.slnx 2>/dev/null || true) | wc -l | tr -d ' ')
+csproj_count=$( (ls ./*.csproj 2>/dev/null || true) | wc -l | tr -d ' ')
+if [ "$sln_count" = "0" ] && [ "$csproj_count" = "0" ]; then
+  warn "no .sln/.slnx or .csproj in $(pwd) - nothing to verify."
+  exit 0
+fi
+
+# The whole pipeline needs the dotnet SDK; without it, skip cleanly.
+if ! have dotnet; then
+  warn "dotnet SDK not found - install .NET 10 SDK to run any checks. Skipping."
+  exit 0
+fi
+
+# 1. dotnet format --verify-no-changes - style/whitespace/import order, read-only.
+info "dotnet format --verify-no-changes"
+if dotnet format --verify-no-changes; then
+  ok "format clean"
+else
+  fail "formatting issues (run: dotnet format)"
+fi
+
+# 2. dotnet build -warnaserror - compile + analyzers + nullable warnings as errors.
+info "dotnet build -warnaserror"
+if dotnet build -warnaserror --nologo; then
+  ok "build clean (analyzers + NRT)"
+else
+  fail "build failed or emitted warnings (analyzers/NRT)"
+fi
+
+# 3. dotnet test - unit + integration tests.
+info "dotnet test"
+if dotnet test --nologo; then
+  ok "tests pass"
+else
+  fail "tests failed"
+fi
+
+# 4. dotnet list package --vulnerable - known NuGet vulnerabilities are a real failure.
+#    The command exits 0 even when vulns exist, so grep the output to decide.
+info "dotnet list package --vulnerable --include-transitive"
+vuln_out="$(dotnet list package --vulnerable --include-transitive 2>&1 || true)"
+if printf '%s' "$vuln_out" | grep -qiE 'has the following vulnerable|Severity'; then
+  fail "vulnerable NuGet packages found:"
+  printf '%s\n' "$vuln_out"
+else
+  ok "no known vulnerable packages"
+fi
+
+echo
+if [ "$failed" -ne 0 ]; then
+  printf '%sFAIL:%s one or more checks failed.\n' "$RED" "$RESET"
+  exit 1
+fi
+printf '%sPASS:%s all checks passed.\n' "$GREEN" "$RESET"

package/skills/shared/data-cleaning/SKILL.md

@@ -0,0 +1,259 @@
+---
+name: data-cleaning
+description: "Use when a raw table is too dirty to trust — nulls, sentinels, duplicate rows, category sprawl, mixed types, bad dates — and you need a re-runnable clean() plus a schema gate that fails loud. NOT emitting .xlsx (that is spreadsheet-ops), NOT acquiring rows (that is data-scraper), NOT parsing PDF/HTML into rows (that is structured-extraction)."
+tags: [data-cleaning, data-quality, pandas, pandera, validation, deduplication, reproducibility, polars]
+recommends: [duckdb, spreadsheet-ops, structured-extraction, data-scraper, analytics, business-intelligence, forecasting, python]
+profiles: []
+origin: ai-hub
+---
+
+# Data cleaning — make dirty data trustworthy, and make the cleaning auditable
+
+A clean table is **typed + deduped + normalized + validated + reproducible**. The deliverable here is
+never "I opened a notebook and fixed some rows by hand." It is a re-runnable function `clean(raw) -> df`
+plus a **schema gate** that fails loud when next month's file violates the contract. Reproducible means
+the same input always yields the same output: versions pinned, sorts deterministic, nothing random without
+a seed. If you can't re-run it tomorrow and get the identical result, you haven't cleaned the data — you've
+edited a snapshot.
+
+Cleaning **starts** once you hold tabular rows and **ends** at a validated table/DataFrame/Parquet. Before
+that boundary the job is acquisition ([`data-scraper`](../data-scraper/SKILL.md),
+[`structured-extraction`](../structured-extraction/SKILL.md)); after it, consumption
+([`spreadsheet-ops`](../spreadsheet-ops/SKILL.md), [`analytics`](../analytics/SKILL.md),
+[`business-intelligence`](../business-intelligence/SKILL.md),
+[`forecasting`](../forecasting/SKILL.md)). Multi-GB analytical SQL is an engine choice, not a cleaning one
+— [`duckdb`](../duckdb/SKILL.md).
+
+Current stack (verified 2026-06-02): **pandas 3.0.x** (3.0.0 shipped 2026-01-21) and **pandera 0.31.1**
+(supports pandas ≥ 3) for in-pipeline schema validation; **Polars** and **DuckDB** when pandas runs out of
+RAM. Pin them: `pandas==3.0.3`, `pandera==0.31.1`.
+
+## The pipeline shape
+
+One canonical order. Each step is positioned for a reason, not by habit.
+
+```python
+import pandas as pd
+
+def clean(raw_path: str) -> pd.DataFrame:
+    df = read_typed(raw_path)     # 1. read with explicit dtypes — never let pandas guess
+    df = normalize(df)            # 2. strings/categories/numbers/dates — collapse invisible variance
+    df = dedupe(df)               # 3. AFTER normalize+type, so "1"/1 and "US "/"US" actually collapse
+    df = handle_missing(df)       # 4. decide per column: drop / impute+flag / leave NA / quarantine
+    df = Schema.validate(df, lazy=True)  # 5. the GATE — fail loud, surface every violation at once
+    return df
+```
+
+- **Type before dedupe** — otherwise `"1"` (string) and `1` (int) survive as two distinct keys.
+- **Normalize before dedupe** — `"US "` and `"US"` are the same customer; dedupe can't see that until
+  whitespace/case are collapsed.
+- **Validate last** — it is the gate, not a cleaning step. It asserts the contract holds *after* all fixes.
+- **Write to a NEW artifact** — the raw file is read-only; you never overwrite your only source.
+
+## Read it right
+
+The single most common reproducibility footgun: pandas' legacy numpy path silently casts an integer column
+containing one NaN to `float64`, so your `id` becomes `1001.0`. Control the dtype on read.
+
+```python
+# BAD — pandas guesses: ids become floats, "N/A" stays a string, "" is sometimes NaN sometimes ""
+df = pd.read_csv("raw.csv")
+
+# GOOD — explicit, deterministic, real nullable types
+df = pd.read_csv(
+    "raw.csv",
+    dtype_backend="pyarrow",          # real nullable ints/strings; no silent float-cast
+    na_values=["", "N/A", "NA", "null", "-1", "999"],  # YOUR sentinels become real NA
+    keep_default_na=True,             # keep pandas' default NA tokens too
+    encoding="utf-8",                 # state it; don't let locale decide
+)
+```
+
+Two pandas 3.0 facts that read depends on. `dtype_backend="pyarrow"` only works if pyarrow is actually
+installed — PDEP-14 deliberately kept a NumPy-object fallback so PyArrow stays *recommended, not required*
+— so `pip install pyarrow` for the faster backed path, or pass `dtype_backend="numpy_nullable"` when it is
+absent. And the default `str` dtype (PyArrow-backed when pyarrow is present, NumPy-object-backed otherwise)
+uses **`NaN` missing-value semantics** like every other default dtype: test for null with `pd.isna()`, never
+by comparing against whichever null token happened to appear.
+
+## Profile before you fix
+
+Let the numbers drive the plan, not a glance at `df.head()`. Run this first, every time.
+
+```python
+def profile(df: pd.DataFrame) -> pd.DataFrame:
+    return pd.DataFrame({
+        "dtype":     df.dtypes.astype(str),
+        "null_pct":  (df.isna().mean() * 100).round(1),
+        "n_unique":  df.nunique(dropna=True),       # cardinality — catches category sprawl
+        "sample":    df.apply(lambda s: s.dropna().unique()[:3].tolist()),
+    })
+
+print(profile(df))
+print("rows:", len(df), "exact dupes:", df.duplicated().sum())
+```
+
+A column at 90% null is a drop candidate; one with 400 distinct "countries" needs a mapping table; an
+"age" with min `-1`/max `999` has sentinels to map. The profile is your TODO list.
+
+## Normalize
+
+Each fix below: **Bad → Good**, with a one-line why.
+
+**Strings** — invisible variance (trailing space, mixed case, lookalike unicode) silently breaks joins
+and dedupe.
+
+```python
+# BAD: "US ", "us", "ｕｓ" all look different to a join
+# GOOD:
+s = df["country"].str.strip().str.casefold().str.normalize("NFKC")
+```
+
+**Categories** — use a **mapping table**, never a tower of regex. A dict is auditable and an *unmapped*
+value gets quarantined instead of silently passing through.
+
+```python
+COUNTRY = {"usa": "US", "u.s.": "US", "united states": "US", "u.s.a.": "US", "es": "ES", "españa": "ES"}
+key = df["country"].str.strip().str.casefold()
+df["country"] = key.map(COUNTRY)            # unmapped -> NA, which the gate below will catch (no silent pass)
+```
+
+**Numbers** — turn sentinels into NA, then choose a range policy explicitly: *clip* (cap to bound) when
+out-of-range is plausibly a recording cap, *reject* (→ NA / quarantine) when it is impossible.
+
+```python
+df["age"] = df["age"].mask(df["age"].isin([-1, 999]))   # sentinels -> NA
+df["age"] = df["age"].clip(lower=0, upper=120)          # clip policy; or .mask(~df["age"].between(0,120)) to reject
+```
+
+**Dates** — state the `format`, coerce, then **count the casualties**. Never trust dayfirst inference;
+`03/04/2026` is ambiguous and pandas will pick silently.
+
+```python
+parsed = pd.to_datetime(df["signup"], format="%Y-%m-%d", utc=True, errors="coerce")
+bad = parsed.isna() & df["signup"].notna()
+assert bad.sum() == 0, f"{bad.sum()} dates failed the expected format — inspect before proceeding"
+df["signup"] = parsed
+```
+
+Copy-paste versions of all of these — category mapping with unmapped→quarantine, a robust date parser,
+unicode/encoding repair, a sentinel→NA table, numeric clip-vs-reject, plus Polars equivalents — are in
+[references/normalization-recipes.md](references/normalization-recipes.md).
+
+## Dedupe
+
+`drop_duplicates(keep="first")` is meaningless without a defined key and a stable sort — "first" of what
+order? Define both.
+
+```python
+key = ["customer_id"]                                   # the BUSINESS key, stated explicitly
+df = (df.sort_values(["customer_id", "updated_at"], ascending=[True, False], kind="stable")
+        .drop_duplicates(subset=key, keep="first"))     # keep most-recent per customer, deterministically
+```
+
+Near-duplicates (`"Acme Inc"` vs `"Acme, Inc."`) are a *normalization* problem — collapse them in the
+normalize step first; only then does exact dedupe catch them. Fuzzy matching is a separate, riskier
+decision — make it visible, never automatic.
+
+## Missing values — decide per column
+
+No silent `fillna(0)`: a zero is a value, and treating "unknown" as zero poisons every mean, sum, and model
+downstream. Pick deliberately.
+
+| Situation | Action | Why |
+| --- | --- | --- |
+| Column is mostly null (e.g. >70%) and not load-bearing | Drop the **column** | Imputing it invents signal that isn't there |
+| A few rows missing a *required* key (id, date) | Drop the **row** (and log/quarantine) | Can't dedupe or join without the key |
+| Numeric gap you must fill for a model | Impute **and add a `_was_missing` flag** | The model can learn "was missing"; you keep the audit trail |
+| Genuinely optional field | **Leave NA** | NA is information; don't fabricate a value |
+| Value is present but *invalid* (unmapped category, bad date) | **Quarantine the row** | Don't drop silently and don't let it pass the gate |
+
+```python
+df["income_was_missing"] = df["income"].isna()
+df["income"] = df["income"].fillna(df["income"].median())   # impute + flag, never bare fillna(0)
+```
+
+## Validate — the gate
+
+This is where cleaning becomes *trustworthy*. Declare the contract as a pandera `DataFrameModel`, validate
+**output** (and input expectations where they exist), and split valid rows from failures instead of
+crashing — the failures become your quarantine.
+
+```python
+import pandera.pandas as pa
+from pandera.typing import Series
+
+class CustomerSchema(pa.DataFrameModel):
+    customer_id: Series[int]   = pa.Field(unique=True, ge=1)
+    country:     Series[str]   = pa.Field(isin=["US", "ES", "FR"])     # only mapped categories survive
+    age:         Series[float] = pa.Field(ge=0, le=120, nullable=True)
+    signup:      Series[pa.DateTime] = pa.Field(nullable=False)
+
+    class Config:
+        strict = True       # reject unexpected columns
+        coerce = True       # coerce to declared dtype, fail loud if impossible
+
+# lazy=True collects EVERY violation at once instead of dying on the first
+try:
+    valid = CustomerSchema.validate(df, lazy=True)
+except pa.errors.SchemaErrors as e:
+    failures = e.failure_cases          # dataframe of exactly which rows/checks failed
+    failures.to_parquet("quarantine.parquet")   # keep, don't drop — someone investigates these
+    valid = df.drop(index=e.failure_cases["index"].dropna().unique())  # proceed with the clean subset
+```
+
+`coerce=True` fixes types the contract expects; `nullable` states which columns may hold NA; field
+`Check`s (`ge`, `le`, `isin`, `unique`) are the allowed-value rules. `strict` catches columns that
+shouldn't be there. Together they are the data contract in code. Log the row-count diff on every run —
+`in`, `out`, coerced, quarantined — so what the pipeline changed is an auditable record, not an assumption.
+
+When to escalate beyond pandera: reach for **GX Core 1.0** (Great Expectations' rebranded OSS — Data
+Context → Data Source → Expectation Suite → Validation Definition → Checkpoint) when you need a *shared
+data-quality platform* across many datasets and teams with a results store and docs. Use **dbt model
+contracts** (enforced at build) plus **dbt tests** (post-materialization) when the cleaning lives in a
+SQL warehouse, not Python. The full `DataFrameModel` (custom `@pa.check`, lazy `SchemaErrors` report,
+valid/quarantine split helper), the GX checkpoint sketch, the dbt model-contract + `data_tests` YAML, and
+the "which validator" chooser are in [references/validation-patterns.md](references/validation-patterns.md).
+
+## Scale — when pandas hurts
+
+Heuristic: pandas is fine while the data fits comfortably in RAM (roughly ≤ 1–2 GB working set). Beyond
+that, or when a groupby/join dominates the runtime, switch the *mechanics* (not the principles):
+
+- **Polars** for clean-at-scale: `pl.scan_csv(...)` (lazy, parallel, Rust), then `.unique()`,
+  `.drop_nulls()`, `.fill_null(...)`, `.str.*` — the same profile→normalize→dedupe→validate shape, faster.
+  pandera validates Polars frames too, and the
+  [recipes reference](references/normalization-recipes.md) has the Polars equivalent of every fix above.
+- **DuckDB** when the bottleneck is analytical SQL over multi-GB files — point heavy joins/aggregations
+  there: [`duckdb`](../duckdb/SKILL.md). It is an *engine* choice; correctness/normalization is still this
+  skill's job.
+
+## Anti-patterns
+
+| Anti-pattern | Why it breaks |
+| --- | --- |
+| "`fillna(0)` to get rid of the nulls" | Zero is a value; it distorts every mean/sum/model. Impute deliberately and add a `_was_missing` flag. |
+| "`drop_duplicates()` — done" | No `subset`, no sort → which row survives is nondeterministic. Define the key, `sort_values(kind="stable")`, set `keep`. |
+| "`pd.read_csv(path)` and start cleaning" | pandas guesses: ids become floats, dates become strings. Pass `dtype_backend` + `na_values`. |
+| "I fixed the rows in a notebook cell" | Not reproducible — next month's file gets nothing. Wrap it in `clean(raw) -> df`. |
+| "Drop the rows that look wrong" | Silent data loss with no audit trail. Quarantine to a file; someone investigates. |
+| "A few regexes will normalize the countries" | Unmaintainable and silent on new values. Use a mapping dict; unmapped → NA → caught by the gate. |
+| "`pd.to_datetime` figures out the format" | Ambiguous dates parse silently wrong. State `format=`, `errors="coerce"`, then assert the NaT count. |
+| "Validation passed, so we're good" | A gate that never fails is a no-op. Feed it a known-bad row and confirm it *rejects*. |
+| "It's slow, rewrite everything in Polars" | Switch the engine, not the discipline — profile→normalize→dedupe→validate still applies. |
+
+## Verify
+
+`scripts/verify.sh` runs from anywhere, no network. It does static structure checks on this skill
+(frontmatter keys, references present) always, and — when pandas + pandera are installed — extracts the
+documented pattern, feeds it one clearly-good row and one clearly-bad row, and asserts the good row PASSES
+validation while the bad row is FLAGGED/quarantined, proving the gate is not a no-op. Without
+pandas/pandera it prints SKIP for the runtime check and still passes the static checks.
+
+## Project grounding (02-DOCS + CLAUDE.md)
+
+In a project with a `02-DOCS/` layer (the [`harness`](../harness/SKILL.md) wiki), record this dataset's
+cleaning decisions — the schema/contract, the category mapping tables, the dedupe key, the quarantine
+location, version pins — in `02-DOCS/wiki/data/<dataset>.md`, link it from the root `CLAUDE.md`
+`## Knowledge map`, and read it first on every re-run so the contract stays consistent. No `02-DOCS/`? Skip
+silently. Conventions are recorded, never gated.

package/skills/shared/data-cleaning/evals/README.md

@@ -0,0 +1,16 @@
+# Eval harness — `data-cleaning`
+
+`cases.yaml` is a human/LLM-graded rubric, not an automated runner. It grades two things: **triggering**
+(does the skill fire on real messy-data asks and stay quiet on near-misses that belong to a sibling?) and
+**capability** (does loading the skill produce a measurably better cleaning pipeline?). The mechanical
+check — that the documented validation gate actually rejects a bad row — lives separately in
+`scripts/verify.sh`, which these cases do not duplicate.
+
+To run triggering: start a fresh agent with only `data-cleaning` discoverable, paste each `should_trigger`
+prompt verbatim, and confirm it reaches for this skill (3–5 trials each, since the decision is stochastic;
+pass if it fires in the majority). For each `should_not_trigger`, confirm it does **not** fire and would
+plausibly route to the named `route_to` sibling (some siblings may not be built yet — the routing intent is
+what's graded). To run capability: give the scenario to a clean agent WITHOUT the skill and again WITH it,
+then grade each transcript against the `must_include` list, one point per bullet that is specifically and
+correctly covered (not merely name-dropped). WITH should clearly beat WITHOUT. Record trial counts and
+per-bullet verdicts so a reviewer can audit; don't report a bare pass/fail.