@1aboveio/skills 1.0.0 → 1.0.2

package/skills/app-dev/data-quality/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: data-quality
+description: "Check a published warehouse table's data against its data-catalog definitions with read-only SELECTs: columns never filled or stuck at one value, counts below zero, ratios outside 0–1, declared values that never appear and undeclared values that do. Then classify each finding as a wrong definition, wrong code, or wrong data. Use when asked whether a table's data is right, to profile or sanity-check a dwd/ads table, to find dead or constant columns, to check a column's values against its documented list, after a pipeline change ships, or when a data-catalog definition and the data might disagree. NOT for validating a Spark change before it ships (pyspark canary and reconciliation), writing definitions (data-catalog), or monitoring dashboards."
+release-version: 0.39.0
+---
+**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
+# Data quality
+
+Ask the data whether it means what its definitions say. The definitions come from the
+data-catalog skill; this skill turns them into read-only queries, runs them, and
+reports where table, code, and documentation disagree.
+
+## Find your task
+
+| Task | Do |
+|---|---|
+| The repo has no `data-catalog.json` or the table has no definitions | Set the repo up and document the table with the data-catalog skill first |
+| Check one or more tables | [Run the checks](#run-the-checks) |
+| Decide what a finding means | [Classify each finding](#classify-each-finding) |
+| What each check does, and how to make a column checkable | [references/checks.md](references/checks.md) |
+| Connect to Presto, Trino, Spark SQL, or another engine | [references/checks.md#runners](references/checks.md#runners) |
+
+`<engine>` is `<skill-base-dir>/scripts/data_quality.py`. It reads definitions through
+the repo's own `scripts/data-catalog/data_catalog.py`.
+
+## Rules
+
+1. **Read only.** Every query is one SELECT built by the engine; the SQLite runner
+   opens the file read-only. Never add a write to a runner, and never run the checks
+   with credentials that can write when read-only ones exist.
+2. **Bound the scan.** Pass `--where` on the partition column for a recent range, and
+   widen only when a finding needs it.
+   Example: `--where "dt >= DATE '2026-10-01'"` scanned 333K rows in 18 seconds.
+3. **Aggregates only.** The checks return counts, distinct counts, minimums, maximums,
+   and counts per declared value. Never add a query that returns rows, and never print
+   a column's actual value when it could identify a customer.
+4. **A finding is a lead until you classify it.** Read the code that writes the column
+   before saying the data, the code, or the definition is wrong.
+   Example: `exchange_share` reached 25 while its unit says ratio: read the formula
+   before deciding whether the unit or the code is wrong.
+5. **Report; never fix in the same step.** List each finding with its classification
+   and evidence. Fixing a definition is a data-catalog change; fixing code or data is
+   the owner's call. File an issue only when the user asks.
+6. **Make columns checkable at the source.** A finding that needs a fact the catalog
+   lacks (allowed values, a unit) is fixed by adding it to the `Field`, not by
+   special-casing the check.
+   Example: add `values=("是", "否", "无法判定")` to `agent_risk_status`, then rerun.
+
+## Run the checks
+
+```bash
+python3 <engine> run --table usr.dwd_orders --where "dt >= DATE '2026-10-01'" \
+  --key dt --key order_id --runner "python3 tools/presto_query.py"   # or --sqlite warehouse.db
+python3 <engine> plan --table usr.dwd_orders --where "..."   # just the SQL, to run elsewhere
+python3 <engine> judge --results results.json               # judge rows you ran yourself
+```
+
+- Run from the repo root, or pass `--root`.
+- Pass the table's grain as `--key` (one per column): a join that repeats rows breaks
+  it, and nothing else catches that.
+- One table per run when tables use different partition columns.
+- Engine errors such as "exceeds the allowed maximum stages": lower `--batch`.
+- A column type the engine cannot count distinct values of (maps, some arrays): `--skip`.
+- Exit 0: no problems. Exit 1: problems found. Exit 2: setup or usage error.
+
+## Classify each finding
+
+For each problem, read the column's definition and the code that writes it, then give
+one of:
+
+| Class | Means | Example | Next step |
+|---|---|---|---|
+| definition | the code and data agree; the definition is wrong or incomplete | a retention ratio documented as 0–1 that is by design unbounded | fix the `Field` (data-catalog) |
+| code | the code does not compute what the definition says | `transaction_result` hard-coded to `SUCCESS` | report to the owner with the code location |
+| data | the code is right; the source data is wrong or missing | a source that never fills a certificate type | report to the source owner |
+| expected | correct as is, and the definition should say so | a payout-only column that is 0 in a collections-only period | add the reason to the definition so the next run marks it documented |
+
+A column documented as always empty whose `source` names a real upstream column is
+still a question for the owner: is the placeholder meant to stay, or is the code
+missing the read?
+
+Findings marked `(documented)` already match what the definition says (for example
+"always empty"), version and hash columns, and missing-flags that are 0 on every row:
+mention them, do not chase them.
+
+## Report
+
+Give, per table: the range checked, the row count, then each problem as
+`column — finding — class — evidence (file:line or query)`, most serious first.
+End with what you could not classify and why. Never claim a table is clean when a
+check was `not-run` or a column was skipped.
+
+## Done when
+
+- Every problem finding has a class and evidence, or is listed as unclassified.
+- Nothing was written to the warehouse, and no row-level value was printed.

package/skills/app-dev/data-quality/agents/openai.yaml

@@ -0,0 +1,3 @@
+interface:
+  display_name: "Data quality"
+  short_description: "Read-only checks of a warehouse table's data against its data-catalog definitions, with each finding classified"

package/skills/app-dev/data-quality/references/checks.md

@@ -0,0 +1,56 @@
+# Checks, findings, and runners
+
+## What each check reads
+
+The engine builds up to three kinds of query per documented table:
+
+- **Profile** (one per 40 columns, `--batch`): `COUNT(*)`, and per column `COUNT(col)`
+  and `COUNT(DISTINCT col)`; `MIN`/`MAX` for columns whose `unit` is numeric (count,
+  ratio, percent, a currency, seconds, days, score) or whose name looks like a flag
+  (`*_missing`, `*_flag`, `is_*`, `has_*`).
+- **Key** (with `--key`): counts the grain values that repeat, with a GROUP BY …
+  HAVING COUNT(*) > 1 subquery.
+- **Values** (one per column with `values=(...)`): `SELECT col AS value, COUNT(*) AS n …
+  GROUP BY col`. Only declare `values` on enumerated columns, never on identifiers.
+
+## Findings
+
+| Finding | Fires when | Marked documented when |
+|---|---|---|
+| `empty-table` | no rows in the `--where` range; nothing else is judged | — |
+| `never-filled` | the column is empty in every row | the definition says always empty / 始终为空 |
+| `mostly-empty` | filled in under 1% of rows | — |
+| `constant` | one distinct value in every filled row, with at least 30 rows (`--constant-min-rows`) and the column not named in `--where` | the definition says always/constant/恒为; the name is a version, hash, or source table; a flag column is constant at 0 or N |
+| `negative-value` | minimum below 0 for unit count, seconds, or days | — |
+| `out-of-range` | values outside 0–1 for unit `ratio`, or 0–100 for unit `percent` | — |
+| `unseen-value` | a declared value never appears (and the column has rows) | — |
+| `undeclared-value` | a value appears that `values` does not list (empty values are not counted) | — |
+| `duplicate-key` | a `--key` combination appears in more than one row | — |
+| `not-run` | a query has no result | — |
+
+## Making a column checkable
+
+Each check needs a fact in the column's data-catalog `Field`:
+
+| To check | Write in the `Field` |
+|---|---|
+| allowed values | `values=("是", "否", "无法判定")`, or `--   values: 是; 否; 无法判定` in SQL |
+| 0–1 range | `unit="ratio"`; use `"percent"` for 0–100; leave a ratio unbounded by design without that unit |
+| never negative | `unit="count"` (or seconds, days) |
+| expected empty or constant | say so in the text: "Always empty until the source fills it." |
+
+## Runners
+
+`run --runner CMD` sends one SELECT on stdin per query and reads the rows from stdout:
+a JSON array of objects, JSON lines (one object per line), or CSV with a header row.
+Exit non-zero with a message on stderr when a query fails.
+
+| Engine | Runner command shape |
+|---|---|
+| Trino / Presto CLI | `trino --server <url> --catalog hive --output-format JSON` (reads stdin) |
+| Presto REST | a small script that POSTs to `/v1/statement`, follows `nextUri`, prints a JSON array |
+| Spark SQL | a script that runs `spark.sql(stdin).toPandas().to_json(orient="records")` |
+| SQLite (local, tests) | built in: `--sqlite <file>`, opened read-only |
+
+Use `--dialect spark` with Spark SQL (backtick identifiers); the default quotes with
+double quotes. Use read-only credentials where the engine offers them.

package/skills/app-dev/data-quality/release-check.mjs

@@ -0,0 +1,93 @@
+#!/usr/bin/env node
+import { spawnSync } from 'node:child_process';
+import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { homedir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+const WEEK_MS = 7 * DAY_MS;
+const here = dirname(fileURLToPath(import.meta.url));
+
+function readJson(path, fallback = {}) {
+  try { return JSON.parse(readFileSync(path, 'utf8')); } catch { return fallback; }
+}
+
+function frontmatterValue(text, key) {
+  const frontmatter = /^---\n([\s\S]*?)\n---/.exec(text)?.[1] ?? '';
+  const match = new RegExp('^' + key + ':\\s*([^\\n]+)$', 'm').exec(frontmatter);
+  return match?.[1].trim().replace(/^['"]|['"]$/g, '') ?? null;
+}
+
+function repositoryOf(source) {
+  const value = String(source ?? '').trim();
+  const match = /^(?:git@github\.com:|https:\/\/github\.com\/)?([A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+?)(?:\.git)?$/.exec(value);
+  return match?.[1] ?? null;
+}
+
+function stableVersion(value) {
+  const match = /^v?(\d+)\.(\d+)\.(\d+)$/.exec(String(value ?? '').trim());
+  return match ? match.slice(1).map(Number) : null;
+}
+
+function newer(left, right) {
+  for (let index = 0; index < 3; index += 1) {
+    if (left[index] !== right[index]) return left[index] > right[index];
+  }
+  return false;
+}
+
+function writeCache(path, value) {
+  try {
+    mkdirSync(dirname(path), { recursive: true });
+    writeFileSync(path, JSON.stringify(value));
+  } catch { /* A read-only cache must not block skill use. */ }
+}
+
+function main(now = Date.now()) {
+  if (process.env.ONEABOVEIO_NO_UPDATE_CHECK) return;
+  const skill = readFileSync(join(here, 'SKILL.md'), 'utf8');
+  const name = frontmatterValue(skill, 'name');
+  const localVersion = stableVersion(frontmatterValue(skill, 'release-version'));
+  if (!name || !localVersion) return;
+
+  const lock = readJson(join(process.env.HOME || homedir(), '.agents', '.skill-lock.json'));
+  const entry = lock.skills?.[name];
+  const repository = repositoryOf(entry?.sourceUrl || entry?.source);
+  if (!repository) return;
+
+  const cacheRoot = process.env.XDG_CACHE_HOME || join(process.env.HOME || homedir(), '.cache');
+  const cachePath = join(cacheRoot, '1aboveio-skills', 'release-check.json');
+  const all = readJson(cachePath);
+  const cache = all[repository] && typeof all[repository] === 'object' ? all[repository] : {};
+
+  if (!cache.lastCheck || now - cache.lastCheck >= DAY_MS) {
+    const result = spawnSync('gh', ['api', 'repos/' + repository + '/releases/latest', '--jq', '.tag_name'], {
+      encoding: 'utf8',
+      stdio: ['ignore', 'pipe', 'ignore'],
+      timeout: 1500,
+    });
+    cache.lastCheck = now;
+    if (result.status === 0 && stableVersion(result.stdout)) cache.latestTag = result.stdout.trim();
+    else delete cache.latestTag;
+    all[repository] = cache;
+    writeCache(cachePath, all);
+  }
+
+  const latestVersion = stableVersion(cache.latestTag);
+  if (!latestVersion || !newer(latestVersion, localVersion)) return;
+  if (cache.notifiedTag === cache.latestTag && cache.notifiedAt && now - cache.notifiedAt < WEEK_MS) return;
+
+  cache.notifiedTag = cache.latestTag;
+  cache.notifiedAt = now;
+  all[repository] = cache;
+  writeCache(cachePath, all);
+  const command = `npx skills@1.5.23 add '${repository}#${cache.latestTag}' --global --agent universal claude-code --skill ${name} --yes`;
+  process.stdout.write(
+    `UPDATE_AVAILABLE: A newer 1AboveIO skills release (${cache.latestTag}) is available; this skill is `
+      + `v${localVersion.join('.')}. Ask once whether to refresh ${name}. On approval run \`${command}\`. `
+      + 'The refreshed instructions apply next session. Either way, continue the current task and do not ask again this session.\n',
+  );
+}
+
+try { main(); } catch { /* Update checks are advisory and always fail silent. */ }

package/skills/app-dev/data-quality/scripts/data-quality.test.mjs

@@ -0,0 +1,12 @@
+import assert from 'node:assert/strict';
+import { spawnSync } from 'node:child_process';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import test from 'node:test';
+
+const scriptsDir = dirname(fileURLToPath(import.meta.url));
+
+test('data-quality test_data_quality.py', () => {
+  const result = spawnSync('python3', [join(scriptsDir, 'test_data_quality.py')], { encoding: 'utf8' });
+  assert.equal(result.status, 0, `${result.stdout}\n${result.stderr}`);
+});

package/skills/app-dev/data-quality/scripts/data_quality.py

@@ -0,0 +1,405 @@
+#!/usr/bin/env python3
+"""Data quality engine: check a published table's data against its data-catalog definitions.
+
+Commands:
+  plan     write the read-only SELECT checks for documented tables (JSON)
+  judge    turn query results into findings
+  run      plan, execute through a runner, and judge in one step
+  version  print the engine identity
+
+Definitions come from the repo's data-catalog engine (scripts/data-catalog/data_catalog.py),
+so a table is checkable once its script carries CATALOG_TABLE / CATALOG_FIELDS or a
+`-- @field` block. Every query is a single SELECT; nothing is ever written.
+
+Runner contract (run --runner CMD): CMD reads one SQL statement on stdin and prints the
+result rows on stdout as a JSON array of objects, or as CSV with a header row.
+
+Standard library only.
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import hashlib
+import importlib.util
+import io
+import json
+import re
+import shlex
+import sqlite3
+import subprocess
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+
+ENGINE_VERSION = "1.0.0"
+CATALOG_ENGINE = "scripts/data-catalog/data_catalog.py"
+NUMERIC_UNITS = {"count", "usd", "cny", "eur", "gbp", "hkd", "amount", "ratio", "percent", "seconds", "days", "score"}
+NON_NEGATIVE_UNITS = {"count", "seconds", "days"}
+RANGES = {"ratio": (0.0, 1.0), "percent": (0.0, 100.0)}
+DOCUMENTED_CONSTANT_RE = re.compile(r"始终|恒为|目前.{0,8}(为空|不填)|always|constant|never (filled|set)|not filled", re.I)
+PROVENANCE_RE = re.compile(r"(^|_)(version|sha\d*|hash|checksum|source_table|compute_time|etl_time)$", re.I)
+FLAG_RE = re.compile(r"(_missing|_flag)$|^(is|has)_", re.I)   # healthy when constant at 0 / N
+CLEAR_FLAG = {"0", "0.0", "false", "n", "no"}
+PROFILE_BATCH = 40  # columns per profile query: each COUNT(DISTINCT) is a stage, and engines cap stages
+MOSTLY_EMPTY = 0.01
+CONSTANT_MIN_ROWS = 30  # below this, one distinct value says little
+NULL_KEY = "<null>"
+
+
+class QualityError(Exception):
+    """A usage or setup problem; exit code 2."""
+
+
+# ---------------------------------------------------------------------------
+# Definitions, through the repo's data-catalog engine
+# ---------------------------------------------------------------------------
+
+def load_catalog_engine(root: Path, path: str | None):
+    engine = Path(path) if path else root / CATALOG_ENGINE
+    if not engine.is_file():
+        raise QualityError(f"no data-catalog engine at {engine}; set the repo up with the data-catalog skill first "
+                           "(its init copies the engine), or pass --catalog-engine")
+    spec = importlib.util.spec_from_file_location("data_catalog_for_quality", engine)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module  # dataclasses resolve string annotations through sys.modules
+    spec.loader.exec_module(module)
+    return module
+
+
+def documented_tables(root: Path, engine_path: str | None, wanted: list) -> list:
+    dc = load_catalog_engine(root, engine_path)
+    try:
+        config = dc.load_config(root)
+        tables, errors, _ = dc.collect(root, config)
+    except dc.CatalogError as exc:
+        raise QualityError(f"data-catalog: {exc}") from exc
+    if errors:
+        raise QualityError("data-catalog metadata has errors; run its generate first:\n  " + "\n  ".join(errors))
+    if wanted:
+        missing = [w for w in wanted if w not in {t.name for t in tables}]
+        if missing:
+            raise QualityError(f"not documented in the catalog: {', '.join(missing)}")
+        tables = [t for t in tables if t.name in wanted]
+    if not tables:
+        raise QualityError("no documented tables to check")
+    return tables
+
+
+# ---------------------------------------------------------------------------
+# Plan: one profile query per table, one value query per enumerated column
+# ---------------------------------------------------------------------------
+
+def quote(name: str, dialect: str) -> str:
+    if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name):
+        raise QualityError(f"column name {name!r} cannot be quoted safely")
+    return f"`{name}`" if dialect == "spark" else f'"{name}"'
+
+
+def table_ref(name: str) -> str:
+    if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*){0,2}", name):
+        raise QualityError(f"table name {name!r} is not a plain identifier")
+    return name
+
+
+def allowed(f) -> tuple:
+    """Declared values; empty for engine copies older than `Field.values`."""
+    return tuple(getattr(f, "values", ()) or ())
+
+
+def is_numeric(f) -> bool:
+    return (f.unit or "").strip().lower() in NUMERIC_UNITS
+
+
+def plan_queries(tables: list, where: str | None, dialect: str, skip: set, batch: int = PROFILE_BATCH,
+                 key: list | None = None) -> list:
+    cond = f" WHERE {where}" if where else ""
+    queries = []
+    for t in tables:
+        if key:
+            unknown = [k for k in key if k not in {f.name for f in t.fields}]
+            if unknown:
+                raise QualityError(f"--key column(s) not documented in {t.name}: {', '.join(unknown)}")
+            cols = ", ".join(quote(k, dialect) for k in key)
+            sql = (f"SELECT COUNT(*) AS dup_keys, COALESCE(SUM(c), 0) AS dup_rows FROM "
+                   f"(SELECT COUNT(*) AS c FROM {table_ref(t.name)}{cond} GROUP BY {cols} HAVING COUNT(*) > 1) dup")
+            queries.append({"id": f"key:{t.name}", "table": t.name, "kind": "key", "columns": list(key), "sql": sql})
+        cols = [f for f in t.fields if f.name not in skip]
+        chunks = [cols[i:i + batch] for i in range(0, len(cols), batch)] or [[]]
+        for k, chunk in enumerate(chunks):
+            parts = ["COUNT(*) AS n"]
+            for i, f in enumerate(chunk):
+                q = quote(f.name, dialect)
+                parts += [f"COUNT({q}) AS nn_{i}", f"COUNT(DISTINCT {q}) AS d_{i}"]
+                if is_numeric(f) or FLAG_RE.search(f.name):
+                    parts += [f"MIN({q}) AS mn_{i}", f"MAX({q}) AS mx_{i}"]
+            sql = f"SELECT {', '.join(parts)} FROM {table_ref(t.name)}{cond}"
+            suffix = f"#{k + 1}" if len(chunks) > 1 else ""
+            queries.append({"id": f"profile:{t.name}{suffix}", "table": t.name, "kind": "profile", "part": k,
+                            "columns": [f.name for f in chunk], "sql": sql})
+        for f in cols:
+            if allowed(f):
+                q = quote(f.name, dialect)
+                sql = f"SELECT {q} AS value, COUNT(*) AS n FROM {table_ref(t.name)}{cond} GROUP BY {q}"
+                queries.append({"id": f"values:{t.name}.{f.name}", "table": t.name, "kind": "values",
+                                "column": f.name, "sql": sql})
+    for q in queries:
+        assert_read_only(q["sql"])
+    return queries
+
+
+def assert_read_only(sql: str) -> None:
+    if not re.match(r"\s*SELECT\b", sql, re.I) or ";" in sql:
+        raise QualityError(f"refusing to run a statement that is not a single SELECT: {sql[:80]}")
+
+
+# ---------------------------------------------------------------------------
+# Judge: results -> findings
+# ---------------------------------------------------------------------------
+
+@dataclass
+class Finding:
+    table: str
+    column: str | None
+    kind: str
+    message: str
+    documented: bool = False   # the definition already says so: report as info, not a problem
+
+
+def number(value):
+    if value is None or value == "":
+        return None
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def where_columns(where: str | None) -> set:
+    """Identifiers a --where condition names: those columns are constant by construction."""
+    if not where:
+        return set()
+    bare = re.sub(r"'(?:[^']|'')*'", " ", where)
+    return {w.strip('`"').lower() for w in re.findall(r'[`"]?[A-Za-z_][A-Za-z0-9_]*[`"]?', bare)}
+
+
+def judge(tables: list, queries: list, results: dict, where: str | None = None,
+          constant_min_rows: int = CONSTANT_MIN_ROWS) -> list:
+    by_table = {t.name: t for t in tables}
+    pinned = where_columns(where)
+    findings = []
+    for q in queries:
+        rows = results.get(q["id"])
+        if rows is None:
+            findings.append(Finding(q["table"], q.get("column"), "not-run", f"no result for {q['id']}"))
+            continue
+        t = by_table[q["table"]]
+        if q["kind"] == "key":
+            row = {k.lower(): v for k, v in (rows[0] if rows else {}).items()}
+            keys = number(row.get("dup_keys")) or 0
+            if keys:
+                findings.append(Finding(t.name, None, "duplicate-key",
+                                        f"{int(keys)} ({', '.join(q['columns'])}) value(s) repeat across "
+                                        f"{int(number(row.get('dup_rows')) or 0)} rows; the grain allows one each"))
+            continue
+        if q["kind"] == "profile":
+            part = judge_profile(t, q["columns"], rows, pinned, constant_min_rows)
+            if q.get("part", 0) > 0:  # every batch sees the same row count; report an empty range once
+                part = [f for f in part if f.kind != "empty-table"]
+            findings += part
+        else:
+            findings += judge_values(t, q["column"], rows)
+    return findings
+
+
+def judge_profile(t, columns: list, rows: list, pinned: set, constant_min_rows: int) -> list:
+    if len(rows) != 1:
+        return [Finding(t.name, None, "not-run", f"profile returned {len(rows)} rows, expected 1")]
+    row = {k.lower(): v for k, v in rows[0].items()}
+    n = number(row.get("n")) or 0
+    if n == 0:
+        return [Finding(t.name, None, "empty-table", "no rows in the checked range; nothing else can be judged")]
+    fields = {f.name: f for f in t.fields}
+    out = []
+    for i, name in enumerate(columns):
+        f = fields[name]
+        text = " ".join([*f.text.values(), f.source])
+        documented = bool(DOCUMENTED_CONSTANT_RE.search(text))
+        filled = number(row.get(f"nn_{i}")) or 0
+        distinct = number(row.get(f"d_{i}")) or 0
+        lo, hi = number(row.get(f"mn_{i}")), number(row.get(f"mx_{i}"))
+        if filled == 0:
+            out.append(Finding(t.name, name, "never-filled", f"empty in all {int(n)} rows", documented))
+            continue
+        if filled / n < MOSTLY_EMPTY:
+            out.append(Finding(t.name, name, "mostly-empty", f"filled in {int(filled)} of {int(n)} rows"))
+        if distinct == 1 and n >= constant_min_rows and name.lower() not in pinned:
+            value = row.get(f"mn_{i}")
+            shown = f" ({value})" if value is not None else ""
+            clear_flag = FLAG_RE.search(name) and str(row.get(f"mn_{i}")).strip().lower() in CLEAR_FLAG
+            # a build's version is the same on every row; a missing-flag at 0 means nothing was missing
+            expected = documented or bool(PROVENANCE_RE.search(name)) or bool(clear_flag)
+            out.append(Finding(t.name, name, "constant", f"one value{shown} in all {int(filled)} filled rows", expected))
+        unit = (f.unit or "").strip().lower()
+        if unit in NON_NEGATIVE_UNITS and lo is not None and lo < 0:
+            out.append(Finding(t.name, name, "negative-value", f"minimum {lo:g}, but the unit is {f.unit}"))
+        if unit in RANGES and lo is not None and hi is not None:
+            low, high = RANGES[unit]
+            if lo < low or hi > high:
+                out.append(Finding(t.name, name, "out-of-range",
+                                   f"values from {lo:g} to {hi:g}, outside {low:g}–{high:g} for unit {f.unit}"))
+    return out
+
+
+def judge_values(t, column: str, rows: list) -> list:
+    f = next(f for f in t.fields if f.name == column)
+    seen = {}
+    for r in rows:
+        r = {k.lower(): v for k, v in r.items()}
+        value = r.get("value")
+        empty = value is None or (value == "" and "" not in allowed(f))
+        key = NULL_KEY if empty else str(value)
+        seen[key] = seen.get(key, 0) + (number(r.get("n")) or 0)
+    declared = set(allowed(f))
+    out = []
+    unseen = [v for v in allowed(f) if v not in seen]
+    if unseen and seen:
+        out.append(Finding(t.name, column, "unseen-value", f"declared but never present: {', '.join(unseen)}"))
+    extra = sorted(k for k in seen if k not in declared and k != NULL_KEY)
+    if extra:
+        counts = ", ".join(f"{k} ({int(seen[k])})" for k in extra[:10])
+        out.append(Finding(t.name, column, "undeclared-value", f"present but not declared: {counts}"))
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Runners
+# ---------------------------------------------------------------------------
+
+def parse_rows(text: str) -> list:
+    text = text.strip()
+    if not text:
+        return []
+    if text[0] == "[":
+        return json.loads(text)
+    if text[0] == "{":  # one object, or JSON lines (trino/presto --output-format JSON)
+        return [json.loads(line) for line in text.splitlines() if line.strip()]
+    return [dict(r) for r in csv.DictReader(io.StringIO(text))]
+
+
+def run_command(cmd: str, sql: str) -> list:
+    proc = subprocess.run(shlex.split(cmd), input=sql, capture_output=True, text=True)
+    if proc.returncode != 0:
+        raise QualityError(f"runner failed ({proc.returncode}) on: {sql[:80]}\n{proc.stderr.strip()[-400:]}")
+    return parse_rows(proc.stdout)
+
+
+def run_sqlite(db: str, sql: str) -> list:
+    con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)  # read-only by construction
+    try:
+        cur = con.execute(sql)
+        names = [d[0] for d in cur.description]
+        return [dict(zip(names, row)) for row in cur.fetchall()]
+    finally:
+        con.close()
+
+
+def execute(queries: list, runner: str | None, sqlite_db: str | None) -> dict:
+    results = {}
+    for q in queries:
+        assert_read_only(q["sql"])
+        results[q["id"]] = run_sqlite(sqlite_db, q["sql"]) if sqlite_db else run_command(runner, q["sql"])
+    return results
+
+
+# ---------------------------------------------------------------------------
+# Report and CLI
+# ---------------------------------------------------------------------------
+
+def report(findings: list, tables: list, where: str | None, as_json: bool) -> int:
+    problems = [f for f in findings if not f.documented]
+    if as_json:
+        print(json.dumps({"engine": ENGINE_VERSION, "where": where, "tables": [t.name for t in tables],
+                          "problems": len(problems),
+                          "findings": [vars(f) for f in findings]}, ensure_ascii=False, indent=2))
+        return 1 if problems else 0
+    scope = f" where {where}" if where else " (all rows)"
+    print(f"data-quality: {len(tables)} table(s){scope}; {len(problems)} problem(s), "
+          f"{len(findings) - len(problems)} already documented")
+    for t in tables:
+        mine = [f for f in findings if f.table == t.name]
+        if not mine:
+            continue
+        print(f"\n{t.name}")
+        for f in mine:
+            tag = " (documented)" if f.documented else ""
+            print(f"  - {f.kind}{tag}: {f.column + ': ' if f.column else ''}{f.message}")
+    return 1 if problems else 0
+
+
+def engine_identity() -> str:
+    return f"{ENGINE_VERSION}+{hashlib.sha256(Path(__file__).read_bytes()).hexdigest()[:12]}"
+
+
+def find_root(start: Path) -> Path:
+    for candidate in [start, *start.parents]:
+        if (candidate / "data-catalog.json").exists() or (candidate / ".git").exists():
+            return candidate
+    return start
+
+
+def main(argv: list | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="data_quality.py", description=__doc__.split("\n\n")[0])
+    parser.add_argument("--root", type=Path, help="repo root (default: nearest dir with data-catalog.json or .git)")
+    parser.add_argument("--catalog-engine", help=f"path to data_catalog.py (default <root>/{CATALOG_ENGINE})")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    def scoped(p):
+        p.add_argument("--table", action="append", default=[], help="documented table to check; repeatable (default all)")
+        p.add_argument("--where", help="SQL condition that bounds the scan, e.g. \"dt >= '2026-10-01'\"")
+        p.add_argument("--dialect", choices=["ansi", "spark"], default="ansi", help="identifier quoting: ansi \"x\", spark `x`")
+        p.add_argument("--skip", action="append", default=[], help="column to leave out (e.g. arrays); repeatable")
+        p.add_argument("--key", action="append", default=[],
+                       help="grain column; repeat for a composite key: --key dt --key cust_id")
+        p.add_argument("--batch", type=int, default=PROFILE_BATCH,
+                       help=f"columns per profile query (default {PROFILE_BATCH}); lower it if the engine caps stages")
+        p.add_argument("--constant-min-rows", type=int, default=CONSTANT_MIN_ROWS,
+                       help=f"rows needed before a single value counts as constant (default {CONSTANT_MIN_ROWS})")
+
+    p_plan = sub.add_parser("plan", help="print the SELECT checks as JSON")
+    scoped(p_plan)
+    p_judge = sub.add_parser("judge", help="judge results; --results maps query id to rows")
+    scoped(p_judge)
+    p_judge.add_argument("--results", required=True, help="JSON file: {query id: [row objects]}")
+    p_judge.add_argument("--json", action="store_true")
+    p_run = sub.add_parser("run", help="plan, execute, judge")
+    scoped(p_run)
+    group = p_run.add_mutually_exclusive_group(required=True)
+    group.add_argument("--runner", help="command that reads SQL on stdin and prints JSON or CSV rows")
+    group.add_argument("--sqlite", help="SQLite database file (opened read-only)")
+    p_run.add_argument("--json", action="store_true")
+    sub.add_parser("version", help="print the engine identity")
+    args = parser.parse_args(argv)
+    if args.command == "version":
+        print(engine_identity())
+        return 0
+    root = (args.root or find_root(Path.cwd())).resolve()
+    try:
+        tables = documented_tables(root, args.catalog_engine, args.table)
+        queries = plan_queries(tables, args.where, args.dialect, set(args.skip), args.batch, args.key)
+        if args.command == "plan":
+            print(json.dumps(queries, ensure_ascii=False, indent=2))
+            return 0
+        if args.command == "judge":
+            results = json.loads(Path(args.results).read_text(encoding="utf-8"))
+        else:
+            results = execute(queries, args.runner, args.sqlite)
+        found = judge(tables, queries, results, args.where, args.constant_min_rows)
+        return report(found, tables, args.where, args.json)
+    except QualityError as exc:
+        print(f"data-quality: {exc}", file=sys.stderr)
+        return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main())