@1aboveio/app-dev-skills 0.1.1 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -17,6 +17,7 @@ pi install npm:@1aboveio/app-dev-skills
17
17
  - `app-debug`
18
18
  - `better-auth`
19
19
  - `data-catalog`
20
+ - `data-quality`
20
21
  - `ios-foundation`
21
22
  - `monorepo`
22
23
  - `nextjs-fullstack`
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@1aboveio/app-dev-skills",
3
- "version": "0.1.1",
3
+ "version": "0.1.2",
4
4
  "description": "Next.js, React Native, auth, Prisma, shadcn, Python backends, Airflow and PySpark.",
5
5
  "license": "MIT",
6
6
  "repository": {
@@ -27,7 +27,7 @@
27
27
  "id": "app-dev",
28
28
  "core": "@1aboveio/core-skills",
29
29
  "coreSkills": [],
30
- "digest": "9d998b32d00efcf94a8fd20756a8acf62064226329e05fc9d23f4e68b5cc548b"
30
+ "digest": "de7c7e63b8cf033c471b81777170064bef838b906cc332885b0a3a652f2ff96a"
31
31
  },
32
32
  "publishConfig": {
33
33
  "registry": "https://registry.npmjs.org/",
@@ -2,7 +2,7 @@
2
2
  name: airflow
3
3
  description: Airflow DAG architecture and conventions. Use when designing, writing, reviewing, testing, or debugging Airflow DAGs and tasks, including TaskGroups, data intervals, timezone-aware date parameters, CLI validation, environment-based scheduling, and pytest integrity tests.
4
4
  user-invocable: true
5
- release-version: 0.38.0
5
+ release-version: 0.39.0
6
6
  ---
7
7
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
8
8
  # Airflow DAG Development Patterns
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: android-foundation
3
3
  description: "Prove an Expo / React Native app on a native Android emulator, and fix it when that proof fails. Use before you fix and re-run or re-dispatch a failed Android CI job (`gh workflow run android.yml`, `gh run rerun`): a re-run is not a debugger, so reproduce the failing journey checkpoint off CI first. Also use when a UiAutomator journey or its Python harness tests fail (a CHECK/FAIL line, a wait that timed out, a control not found, adb offline, a lost Metro bridge, a fixture failing on the runner), to build the debug APK and run the device journey, to set up a fresh Android workspace (SDK/NDK/JDK, Gradle caches, private AVD, disposable PostgreSQL), or to add the emulator job to CI. NOT for workflow YAML, CI speed or runner design (github-actions), browser tests (e2e-test), iOS (ios-foundation), plain Jest component tests, or signing and store release."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Android foundation
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: app-debug
3
3
  description: "Diagnose runtime failures in deployed Cloud Run services — 5xx/422 errors, tracebacks, Next.js SSR/hydration or Server Action failures, DB/Secret Manager connection errors, cold-start timeouts, container OOM. NOT for build or deploy failures (cloud-debug)."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # App Debug
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: better-auth
3
3
  description: "Add or debug Better Auth for Next.js App Router: login, sessions, protected routes, middleware, OAuth (including Feishu/Lark), cookies, Prisma, and Auth.js migration."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Better Auth for Next.js
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: data-catalog
3
3
  description: "Column definitions for warehouse tables, written inside the Spark/PySpark or SQL script that builds each table (CATALOG_TABLE / CATALOG_FIELDS, or a `-- @field` comment block), with a generated markdown catalog in docs/catalog/ and a CI check that fails a PR whose catalog is stale. Use when writing or changing a table script that adds, renames, removes, or changes the meaning of an output column or feature; when asked what a warehouse column or feature means; to set a repo up ('init the data catalog', 'generate the data dictionary', '字段字典', 'add catalog CI'); or to audit existing SQL or Spark scripts for undocumented columns. NOT for implementing or tuning the Spark job itself (pyspark), DAG wiring (airflow), or general workflow YAML (github-actions)."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Data catalog
@@ -42,6 +42,7 @@ Never document tables during setup unless the task asks for it.
42
42
  | CI says the catalog is stale | `python3 <engine> generate`, commit the pages |
43
43
  | Before committing a table-script change, or reviewing one | `python3 <engine> drift` (`--base origin/main` for a whole branch) |
44
44
  | `Field`/`Table` arguments, the SQL block, config keys, exit codes | [references/format.md](references/format.md) |
45
+ | Does the table's data match its definitions? | the data-quality skill |
45
46
 
46
47
  `<engine>` is `scripts/data-catalog/data_catalog.py` in the repo, or
47
48
  `<skill-base-dir>/scripts/data_catalog.py` before the repo has a copy. When
@@ -52,7 +52,8 @@ table. `docs/catalog/` is generated from those definitions; never edit it by han
52
52
  collections.", not "0 when none".
53
53
  9. Never restate the column name or write "same as above".
54
54
  10. Cover every case: what the column holds for each kind of row, including when it
55
- is empty; list an enumerated column's values; say so if it is always empty today.
55
+ is empty; list an enumerated column's values in the text and in `values=(...)`;
56
+ say so if it is always empty today.
56
57
  11. Write related columns as a set: a count and its amount, a value and its missing
57
58
  flag, a legacy column and its replacement use the same scope and wording, and any
58
59
  difference is stated in both rows.
@@ -50,7 +50,8 @@ column that says only what is particular to that column.
50
50
  `"同上,30 天窗口"` tell the reader nothing.
51
51
  10. **Cover every case.** Say what the column holds for every kind of row (each
52
52
  transaction type, each branch of the code), including when it is empty. List
53
- the values an enumerated column can take, with a Chinese gloss for English codes.
53
+ the values an enumerated column can take, with a Chinese gloss for English codes,
54
+ and put the same list in `values=(...)` so the data can be checked against it.
54
55
  A column that is always empty or constant today says so.
55
56
  Bad: `"换汇交易为 COMPLETED,内部转账为空。"` (collections and payouts unsaid)
56
57
  Good: `"换汇交易为 COMPLETED;收款、付款和内部转账为空。"`
@@ -41,7 +41,10 @@ CATALOG_FIELDS = (
41
41
  PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)
42
42
  ```
43
43
 
44
- - `Field(name, <one keyword per language>, source, window=None, unit=None)`.
44
+ - `Field(name, <one keyword per language>, source, window=None, unit=None, values=())`.
45
+ `values` lists the values an enumerated column can take, as distinct strings
46
+ (`values=("是", "否", "无法判定")`); the data-quality skill checks the data against it.
47
+ Never use it on identifiers or free text.
45
48
  - `Table(name, <one keyword per language>, grain_<lang>=..., notes_<lang>=(...))`: a
46
49
  grain for every language, and the same number of notes in every language (notes are
47
50
  optional).
@@ -77,7 +80,8 @@ SELECT ...
77
80
  ```
78
81
 
79
82
  - A directive line is `-- @table <name>` or `-- @field <name>`.
80
- - A key line is exactly two spaces after `-- `, then `key: value`.
83
+ - A key line is exactly two spaces after `-- `, then `key: value`. `values:` takes a
84
+ list split on `;` or `;`.
81
85
  - A continuation line has four or more spaces and joins the previous key with a space.
82
86
  - `notes_<lang>` on `@table` splits on `;` or `;`.
83
87
  - The block ends at the first non-comment line. One `@table` per file.
@@ -49,7 +49,7 @@ DEFAULT_CONFIG = {
49
49
  "tableNames": [],
50
50
  "schemaNames": [],
51
51
  }
52
- FIELD_EXTRAS = ("source", "window", "unit")
52
+ FIELD_EXTRAS = ("source", "window", "unit", "values")
53
53
  LANGUAGE_LABELS = {"en": "English", "zh": "中文", "zh_hant": "繁體中文", "ja": "日本語", "ko": "한국어"}
54
54
  LANGUAGE_RE = re.compile(r"^[a-z]{2}(_[a-z]+)?$")
55
55
  SKIP_DIRS = {"node_modules", "__pycache__", "venv"}
@@ -151,6 +151,7 @@ class FieldDoc:
151
151
  window: str | None
152
152
  unit: str | None
153
153
  line: int
154
+ values: tuple = () # the values an enumerated column can take; checked by data-quality
154
155
 
155
156
 
156
157
  @dataclass
@@ -258,7 +259,14 @@ def build_field(values: dict, languages: list, where: str, line: int) -> FieldDo
258
259
  f"languages configured: {languages}")
259
260
  text = {lang: require_text(values, lang, where, line, f"Field {name!r}") for lang in languages}
260
261
  source = require_text(values, "source", where, line, f"Field {name!r}")
261
- return FieldDoc(name, text, source, (values.get("window") or None), (values.get("unit") or None), line)
262
+ allowed = values.get("values") or ()
263
+ if isinstance(allowed, str):
264
+ allowed = tuple(v.strip() for v in re.split(r"[;;]", allowed))
265
+ if not isinstance(allowed, tuple) or not all(isinstance(v, str) and v != "" for v in allowed):
266
+ raise CatalogError(f"{where}:{line}: Field {name!r}: values must be a tuple of non-empty strings")
267
+ if len(set(allowed)) != len(allowed):
268
+ raise CatalogError(f"{where}:{line}: Field {name!r}: values lists a value twice")
269
+ return FieldDoc(name, text, source, (values.get("window") or None), (values.get("unit") or None), line, allowed)
262
270
 
263
271
 
264
272
  def build_table(values: dict, fields: list, languages: list, where: str, line: int, script: str) -> TableDoc:
@@ -545,15 +553,18 @@ def render_table(table: TableDoc, root: Path, config: dict) -> str:
545
553
  out += ["## Fields", ""]
546
554
  has_window = any(f.window for f in table.fields)
547
555
  has_unit = any(f.unit for f in table.fields)
556
+ has_values = any(f.values for f in table.fields)
548
557
  head = ["Field", *(label(lang) for lang in langs), "Source"]
549
558
  head += ["Window"] if has_window else []
550
559
  head += ["Unit"] if has_unit else []
560
+ head += ["Values"] if has_values else []
551
561
  out.append("| " + " | ".join(head) + " |")
552
562
  out.append("|" + "---|" * len(head))
553
563
  for f in table.fields:
554
564
  row = [f"`{f.name}`", *(cell(f.text[lang]) for lang in langs), cell(f.source)]
555
565
  row += [cell(f.window)] if has_window else []
556
566
  row += [cell(f.unit)] if has_unit else []
567
+ row += [cell(", ".join(f"`{v}`" for v in f.values))] if has_values else []
557
568
  out.append("| " + " | ".join(row) + " |")
558
569
  return "\n".join(out) + "\n"
559
570
 
@@ -1175,7 +1186,7 @@ def code_without_metadata(text: str, suffix: str) -> str:
1175
1186
  def definitions(table: TableDoc | None) -> tuple[dict, tuple]:
1176
1187
  if table is None:
1177
1188
  return {}, ()
1178
- fields = {f.name: (tuple(sorted(f.text.items())), f.source, f.window, f.unit) for f in table.fields}
1189
+ fields = {f.name: (tuple(sorted(f.text.items())), f.source, f.window, f.unit, f.values) for f in table.fields}
1179
1190
  notes = tuple((lang, tuple(n)) for lang, n in sorted(table.notes.items()))
1180
1191
  return fields, (tuple(sorted(table.grain.items())), tuple(sorted(table.text.items())), notes)
1181
1192
 
@@ -1274,6 +1285,7 @@ class Field:
1274
1285
  source: str
1275
1286
  window: str | None = None
1276
1287
  unit: str | None = None
1288
+ values: tuple[str, ...] = ()
1277
1289
 
1278
1290
 
1279
1291
  @dataclass(frozen=True)
@@ -124,6 +124,21 @@ class GenerateTest(unittest.TestCase):
124
124
  self.assertIn("| [`dwd.customer_daily`](dwd.customer_daily.md) | Daily customer collection rollup. | "
125
125
  "one row per (dt, cust_id) | 2 |", index)
126
126
 
127
+ def test_values_render_and_must_be_distinct_strings(self):
128
+ script = PY_SCRIPT.replace('source="Run parameter `dt`."),', 'source="Run parameter `dt`.", values=("a", "b")),')
129
+ root = make_repo({"jobs/a.py": script}, BILINGUAL)
130
+ self.assertEqual(run(root, "generate")[0], 0)
131
+ page = (root / "docs/catalog/dwd.customer_daily.md").read_text(encoding="utf-8")
132
+ self.assertIn("| Unit | Values |", page)
133
+ self.assertIn("| `a`, `b` |", page)
134
+ twice = script.replace('values=("a", "b")', 'values=("a", "a")')
135
+ root = make_repo({"jobs/a.py": twice}, BILINGUAL)
136
+ self.assertIn("values lists a value twice", run(root, "generate")[2])
137
+ sql = SQL_SCRIPT.replace("-- source: dim_merchant.merchant_id", "-- source: dim_merchant.merchant_id\n-- values: m1;m2")
138
+ root = make_repo({"sql/m.sql": sql}, BILINGUAL)
139
+ self.assertEqual(run(root, "generate")[0], 0)
140
+ self.assertIn("`m1`, `m2`", (root / "docs/catalog/ads.merchant_monthly.md").read_text(encoding="utf-8"))
141
+
127
142
  def test_check_passes_when_current_and_fails_after_a_definition_changes(self):
128
143
  root = make_repo({"jobs/dwd/customer_daily.py": PY_SCRIPT}, BILINGUAL)
129
144
  self.assertEqual(run(root, "generate")[0], 0)
@@ -0,0 +1,3 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 1AboveIO
@@ -0,0 +1,96 @@
1
+ ---
2
+ name: data-quality
3
+ description: "Check a published warehouse table's data against its data-catalog definitions with read-only SELECTs: columns never filled or stuck at one value, counts below zero, ratios outside 0–1, declared values that never appear and undeclared values that do. Then classify each finding as a wrong definition, wrong code, or wrong data. Use when asked whether a table's data is right, to profile or sanity-check a dwd/ads table, to find dead or constant columns, to check a column's values against its documented list, after a pipeline change ships, or when a data-catalog definition and the data might disagree. NOT for validating a Spark change before it ships (pyspark canary and reconciliation), writing definitions (data-catalog), or monitoring dashboards."
4
+ release-version: 0.39.0
5
+ ---
6
+ **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
+ # Data quality
8
+
9
+ Ask the data whether it means what its definitions say. The definitions come from the
10
+ data-catalog skill; this skill turns them into read-only queries, runs them, and
11
+ reports where table, code, and documentation disagree.
12
+
13
+ ## Find your task
14
+
15
+ | Task | Do |
16
+ |---|---|
17
+ | The repo has no `data-catalog.json` or the table has no definitions | Set the repo up and document the table with the data-catalog skill first |
18
+ | Check one or more tables | [Run the checks](#run-the-checks) |
19
+ | Decide what a finding means | [Classify each finding](#classify-each-finding) |
20
+ | What each check does, and how to make a column checkable | [references/checks.md](references/checks.md) |
21
+ | Connect to Presto, Trino, Spark SQL, or another engine | [references/checks.md#runners](references/checks.md#runners) |
22
+
23
+ `<engine>` is `<skill-base-dir>/scripts/data_quality.py`. It reads definitions through
24
+ the repo's own `scripts/data-catalog/data_catalog.py`.
25
+
26
+ ## Rules
27
+
28
+ 1. **Read only.** Every query is one SELECT built by the engine; the SQLite runner
29
+ opens the file read-only. Never add a write to a runner, and never run the checks
30
+ with credentials that can write when read-only ones exist.
31
+ 2. **Bound the scan.** Pass `--where` on the partition column for a recent range, and
32
+ widen only when a finding needs it.
33
+ Example: `--where "dt >= DATE '2026-10-01'"` scanned 333K rows in 18 seconds.
34
+ 3. **Aggregates only.** The checks return counts, distinct counts, minimums, maximums,
35
+ and counts per declared value. Never add a query that returns rows, and never print
36
+ a column's actual value when it could identify a customer.
37
+ 4. **A finding is a lead until you classify it.** Read the code that writes the column
38
+ before saying the data, the code, or the definition is wrong.
39
+ Example: `exchange_share` reached 25 while its unit says ratio: read the formula
40
+ before deciding whether the unit or the code is wrong.
41
+ 5. **Report; never fix in the same step.** List each finding with its classification
42
+ and evidence. Fixing a definition is a data-catalog change; fixing code or data is
43
+ the owner's call. File an issue only when the user asks.
44
+ 6. **Make columns checkable at the source.** A finding that needs a fact the catalog
45
+ lacks (allowed values, a unit) is fixed by adding it to the `Field`, not by
46
+ special-casing the check.
47
+ Example: add `values=("是", "否", "无法判定")` to `agent_risk_status`, then rerun.
48
+
49
+ ## Run the checks
50
+
51
+ ```bash
52
+ python3 <engine> run --table usr.dwd_orders --where "dt >= DATE '2026-10-01'" \
53
+ --key dt --key order_id --runner "python3 tools/presto_query.py" # or --sqlite warehouse.db
54
+ python3 <engine> plan --table usr.dwd_orders --where "..." # just the SQL, to run elsewhere
55
+ python3 <engine> judge --results results.json # judge rows you ran yourself
56
+ ```
57
+
58
+ - Run from the repo root, or pass `--root`.
59
+ - Pass the table's grain as `--key` (one per column): a join that repeats rows breaks
60
+ it, and nothing else catches that.
61
+ - One table per run when tables use different partition columns.
62
+ - Engine errors such as "exceeds the allowed maximum stages": lower `--batch`.
63
+ - A column type the engine cannot count distinct values of (maps, some arrays): `--skip`.
64
+ - Exit 0: no problems. Exit 1: problems found. Exit 2: setup or usage error.
65
+
66
+ ## Classify each finding
67
+
68
+ For each problem, read the column's definition and the code that writes it, then give
69
+ one of:
70
+
71
+ | Class | Means | Example | Next step |
72
+ |---|---|---|---|
73
+ | definition | the code and data agree; the definition is wrong or incomplete | a retention ratio documented as 0–1 that is by design unbounded | fix the `Field` (data-catalog) |
74
+ | code | the code does not compute what the definition says | `transaction_result` hard-coded to `SUCCESS` | report to the owner with the code location |
75
+ | data | the code is right; the source data is wrong or missing | a source that never fills a certificate type | report to the source owner |
76
+ | expected | correct as is, and the definition should say so | a payout-only column that is 0 in a collections-only period | add the reason to the definition so the next run marks it documented |
77
+
78
+ A column documented as always empty whose `source` names a real upstream column is
79
+ still a question for the owner: is the placeholder meant to stay, or is the code
80
+ missing the read?
81
+
82
+ Findings marked `(documented)` already match what the definition says (for example
83
+ "always empty"), version and hash columns, and missing-flags that are 0 on every row:
84
+ mention them, do not chase them.
85
+
86
+ ## Report
87
+
88
+ Give, per table: the range checked, the row count, then each problem as
89
+ `column — finding — class — evidence (file:line or query)`, most serious first.
90
+ End with what you could not classify and why. Never claim a table is clean when a
91
+ check was `not-run` or a column was skipped.
92
+
93
+ ## Done when
94
+
95
+ - Every problem finding has a class and evidence, or is listed as unclassified.
96
+ - Nothing was written to the warehouse, and no row-level value was printed.
@@ -0,0 +1,3 @@
1
+ interface:
2
+ display_name: "Data quality"
3
+ short_description: "Read-only checks of a warehouse table's data against its data-catalog definitions, with each finding classified"
@@ -0,0 +1,56 @@
1
+ # Checks, findings, and runners
2
+
3
+ ## What each check reads
4
+
5
+ The engine builds up to three kinds of query per documented table:
6
+
7
+ - **Profile** (one per 40 columns, `--batch`): `COUNT(*)`, and per column `COUNT(col)`
8
+ and `COUNT(DISTINCT col)`; `MIN`/`MAX` for columns whose `unit` is numeric (count,
9
+ ratio, percent, a currency, seconds, days, score) or whose name looks like a flag
10
+ (`*_missing`, `*_flag`, `is_*`, `has_*`).
11
+ - **Key** (with `--key`): counts the grain values that repeat, with a GROUP BY …
12
+ HAVING COUNT(*) > 1 subquery.
13
+ - **Values** (one per column with `values=(...)`): `SELECT col AS value, COUNT(*) AS n …
14
+ GROUP BY col`. Only declare `values` on enumerated columns, never on identifiers.
15
+
16
+ ## Findings
17
+
18
+ | Finding | Fires when | Marked documented when |
19
+ |---|---|---|
20
+ | `empty-table` | no rows in the `--where` range; nothing else is judged | — |
21
+ | `never-filled` | the column is empty in every row | the definition says always empty / 始终为空 |
22
+ | `mostly-empty` | filled in under 1% of rows | — |
23
+ | `constant` | one distinct value in every filled row, with at least 30 rows (`--constant-min-rows`) and the column not named in `--where` | the definition says always/constant/恒为; the name is a version, hash, or source table; a flag column is constant at 0 or N |
24
+ | `negative-value` | minimum below 0 for unit count, seconds, or days | — |
25
+ | `out-of-range` | values outside 0–1 for unit `ratio`, or 0–100 for unit `percent` | — |
26
+ | `unseen-value` | a declared value never appears (and the column has rows) | — |
27
+ | `undeclared-value` | a value appears that `values` does not list (empty values are not counted) | — |
28
+ | `duplicate-key` | a `--key` combination appears in more than one row | — |
29
+ | `not-run` | a query has no result | — |
30
+
31
+ ## Making a column checkable
32
+
33
+ Each check needs a fact in the column's data-catalog `Field`:
34
+
35
+ | To check | Write in the `Field` |
36
+ |---|---|
37
+ | allowed values | `values=("是", "否", "无法判定")`, or `-- values: 是; 否; 无法判定` in SQL |
38
+ | 0–1 range | `unit="ratio"`; use `"percent"` for 0–100; leave a ratio unbounded by design without that unit |
39
+ | never negative | `unit="count"` (or seconds, days) |
40
+ | expected empty or constant | say so in the text: "Always empty until the source fills it." |
41
+
42
+ ## Runners
43
+
44
+ `run --runner CMD` sends one SELECT on stdin per query and reads the rows from stdout:
45
+ a JSON array of objects, JSON lines (one object per line), or CSV with a header row.
46
+ Exit non-zero with a message on stderr when a query fails.
47
+
48
+ | Engine | Runner command shape |
49
+ |---|---|
50
+ | Trino / Presto CLI | `trino --server <url> --catalog hive --output-format JSON` (reads stdin) |
51
+ | Presto REST | a small script that POSTs to `/v1/statement`, follows `nextUri`, prints a JSON array |
52
+ | Spark SQL | a script that runs `spark.sql(stdin).toPandas().to_json(orient="records")` |
53
+ | SQLite (local, tests) | built in: `--sqlite <file>`, opened read-only |
54
+
55
+ Use `--dialect spark` with Spark SQL (backtick identifiers); the default quotes with
56
+ double quotes. Use read-only credentials where the engine offers them.
@@ -0,0 +1,93 @@
1
+ #!/usr/bin/env node
2
+ import { spawnSync } from 'node:child_process';
3
+ import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
4
+ import { homedir } from 'node:os';
5
+ import { dirname, join } from 'node:path';
6
+ import { fileURLToPath } from 'node:url';
7
+
8
+ const DAY_MS = 24 * 60 * 60 * 1000;
9
+ const WEEK_MS = 7 * DAY_MS;
10
+ const here = dirname(fileURLToPath(import.meta.url));
11
+
12
+ function readJson(path, fallback = {}) {
13
+ try { return JSON.parse(readFileSync(path, 'utf8')); } catch { return fallback; }
14
+ }
15
+
16
+ function frontmatterValue(text, key) {
17
+ const frontmatter = /^---\n([\s\S]*?)\n---/.exec(text)?.[1] ?? '';
18
+ const match = new RegExp('^' + key + ':\\s*([^\\n]+)$', 'm').exec(frontmatter);
19
+ return match?.[1].trim().replace(/^['"]|['"]$/g, '') ?? null;
20
+ }
21
+
22
+ function repositoryOf(source) {
23
+ const value = String(source ?? '').trim();
24
+ const match = /^(?:git@github\.com:|https:\/\/github\.com\/)?([A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+?)(?:\.git)?$/.exec(value);
25
+ return match?.[1] ?? null;
26
+ }
27
+
28
+ function stableVersion(value) {
29
+ const match = /^v?(\d+)\.(\d+)\.(\d+)$/.exec(String(value ?? '').trim());
30
+ return match ? match.slice(1).map(Number) : null;
31
+ }
32
+
33
+ function newer(left, right) {
34
+ for (let index = 0; index < 3; index += 1) {
35
+ if (left[index] !== right[index]) return left[index] > right[index];
36
+ }
37
+ return false;
38
+ }
39
+
40
+ function writeCache(path, value) {
41
+ try {
42
+ mkdirSync(dirname(path), { recursive: true });
43
+ writeFileSync(path, JSON.stringify(value));
44
+ } catch { /* A read-only cache must not block skill use. */ }
45
+ }
46
+
47
+ function main(now = Date.now()) {
48
+ if (process.env.ONEABOVEIO_NO_UPDATE_CHECK) return;
49
+ const skill = readFileSync(join(here, 'SKILL.md'), 'utf8');
50
+ const name = frontmatterValue(skill, 'name');
51
+ const localVersion = stableVersion(frontmatterValue(skill, 'release-version'));
52
+ if (!name || !localVersion) return;
53
+
54
+ const lock = readJson(join(process.env.HOME || homedir(), '.agents', '.skill-lock.json'));
55
+ const entry = lock.skills?.[name];
56
+ const repository = repositoryOf(entry?.sourceUrl || entry?.source);
57
+ if (!repository) return;
58
+
59
+ const cacheRoot = process.env.XDG_CACHE_HOME || join(process.env.HOME || homedir(), '.cache');
60
+ const cachePath = join(cacheRoot, '1aboveio-skills', 'release-check.json');
61
+ const all = readJson(cachePath);
62
+ const cache = all[repository] && typeof all[repository] === 'object' ? all[repository] : {};
63
+
64
+ if (!cache.lastCheck || now - cache.lastCheck >= DAY_MS) {
65
+ const result = spawnSync('gh', ['api', 'repos/' + repository + '/releases/latest', '--jq', '.tag_name'], {
66
+ encoding: 'utf8',
67
+ stdio: ['ignore', 'pipe', 'ignore'],
68
+ timeout: 1500,
69
+ });
70
+ cache.lastCheck = now;
71
+ if (result.status === 0 && stableVersion(result.stdout)) cache.latestTag = result.stdout.trim();
72
+ else delete cache.latestTag;
73
+ all[repository] = cache;
74
+ writeCache(cachePath, all);
75
+ }
76
+
77
+ const latestVersion = stableVersion(cache.latestTag);
78
+ if (!latestVersion || !newer(latestVersion, localVersion)) return;
79
+ if (cache.notifiedTag === cache.latestTag && cache.notifiedAt && now - cache.notifiedAt < WEEK_MS) return;
80
+
81
+ cache.notifiedTag = cache.latestTag;
82
+ cache.notifiedAt = now;
83
+ all[repository] = cache;
84
+ writeCache(cachePath, all);
85
+ const command = `npx skills@1.5.23 add '${repository}#${cache.latestTag}' --global --agent universal claude-code --skill ${name} --yes`;
86
+ process.stdout.write(
87
+ `UPDATE_AVAILABLE: A newer 1AboveIO skills release (${cache.latestTag}) is available; this skill is `
88
+ + `v${localVersion.join('.')}. Ask once whether to refresh ${name}. On approval run \`${command}\`. `
89
+ + 'The refreshed instructions apply next session. Either way, continue the current task and do not ask again this session.\n',
90
+ );
91
+ }
92
+
93
+ try { main(); } catch { /* Update checks are advisory and always fail silent. */ }
@@ -0,0 +1,12 @@
1
+ import assert from 'node:assert/strict';
2
+ import { spawnSync } from 'node:child_process';
3
+ import { dirname, join } from 'node:path';
4
+ import { fileURLToPath } from 'node:url';
5
+ import test from 'node:test';
6
+
7
+ const scriptsDir = dirname(fileURLToPath(import.meta.url));
8
+
9
+ test('data-quality test_data_quality.py', () => {
10
+ const result = spawnSync('python3', [join(scriptsDir, 'test_data_quality.py')], { encoding: 'utf8' });
11
+ assert.equal(result.status, 0, `${result.stdout}\n${result.stderr}`);
12
+ });
@@ -0,0 +1,405 @@
1
+ #!/usr/bin/env python3
2
+ """Data quality engine: check a published table's data against its data-catalog definitions.
3
+
4
+ Commands:
5
+ plan write the read-only SELECT checks for documented tables (JSON)
6
+ judge turn query results into findings
7
+ run plan, execute through a runner, and judge in one step
8
+ version print the engine identity
9
+
10
+ Definitions come from the repo's data-catalog engine (scripts/data-catalog/data_catalog.py),
11
+ so a table is checkable once its script carries CATALOG_TABLE / CATALOG_FIELDS or a
12
+ `-- @field` block. Every query is a single SELECT; nothing is ever written.
13
+
14
+ Runner contract (run --runner CMD): CMD reads one SQL statement on stdin and prints the
15
+ result rows on stdout as a JSON array of objects, or as CSV with a header row.
16
+
17
+ Standard library only.
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import argparse
22
+ import csv
23
+ import hashlib
24
+ import importlib.util
25
+ import io
26
+ import json
27
+ import re
28
+ import shlex
29
+ import sqlite3
30
+ import subprocess
31
+ import sys
32
+ from dataclasses import dataclass, field
33
+ from pathlib import Path
34
+
35
+ ENGINE_VERSION = "1.0.0"
36
+ CATALOG_ENGINE = "scripts/data-catalog/data_catalog.py"
37
+ NUMERIC_UNITS = {"count", "usd", "cny", "eur", "gbp", "hkd", "amount", "ratio", "percent", "seconds", "days", "score"}
38
+ NON_NEGATIVE_UNITS = {"count", "seconds", "days"}
39
+ RANGES = {"ratio": (0.0, 1.0), "percent": (0.0, 100.0)}
40
+ DOCUMENTED_CONSTANT_RE = re.compile(r"始终|恒为|目前.{0,8}(为空|不填)|always|constant|never (filled|set)|not filled", re.I)
41
+ PROVENANCE_RE = re.compile(r"(^|_)(version|sha\d*|hash|checksum|source_table|compute_time|etl_time)$", re.I)
42
+ FLAG_RE = re.compile(r"(_missing|_flag)$|^(is|has)_", re.I) # healthy when constant at 0 / N
43
+ CLEAR_FLAG = {"0", "0.0", "false", "n", "no"}
44
+ PROFILE_BATCH = 40 # columns per profile query: each COUNT(DISTINCT) is a stage, and engines cap stages
45
+ MOSTLY_EMPTY = 0.01
46
+ CONSTANT_MIN_ROWS = 30 # below this, one distinct value says little
47
+ NULL_KEY = "<null>"
48
+
49
+
50
+ class QualityError(Exception):
51
+ """A usage or setup problem; exit code 2."""
52
+
53
+
54
+ # ---------------------------------------------------------------------------
55
+ # Definitions, through the repo's data-catalog engine
56
+ # ---------------------------------------------------------------------------
57
+
58
+ def load_catalog_engine(root: Path, path: str | None):
59
+ engine = Path(path) if path else root / CATALOG_ENGINE
60
+ if not engine.is_file():
61
+ raise QualityError(f"no data-catalog engine at {engine}; set the repo up with the data-catalog skill first "
62
+ "(its init copies the engine), or pass --catalog-engine")
63
+ spec = importlib.util.spec_from_file_location("data_catalog_for_quality", engine)
64
+ module = importlib.util.module_from_spec(spec)
65
+ sys.modules[spec.name] = module # dataclasses resolve string annotations through sys.modules
66
+ spec.loader.exec_module(module)
67
+ return module
68
+
69
+
70
+ def documented_tables(root: Path, engine_path: str | None, wanted: list) -> list:
71
+ dc = load_catalog_engine(root, engine_path)
72
+ try:
73
+ config = dc.load_config(root)
74
+ tables, errors, _ = dc.collect(root, config)
75
+ except dc.CatalogError as exc:
76
+ raise QualityError(f"data-catalog: {exc}") from exc
77
+ if errors:
78
+ raise QualityError("data-catalog metadata has errors; run its generate first:\n " + "\n ".join(errors))
79
+ if wanted:
80
+ missing = [w for w in wanted if w not in {t.name for t in tables}]
81
+ if missing:
82
+ raise QualityError(f"not documented in the catalog: {', '.join(missing)}")
83
+ tables = [t for t in tables if t.name in wanted]
84
+ if not tables:
85
+ raise QualityError("no documented tables to check")
86
+ return tables
87
+
88
+
89
+ # ---------------------------------------------------------------------------
90
+ # Plan: one profile query per table, one value query per enumerated column
91
+ # ---------------------------------------------------------------------------
92
+
93
+ def quote(name: str, dialect: str) -> str:
94
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name):
95
+ raise QualityError(f"column name {name!r} cannot be quoted safely")
96
+ return f"`{name}`" if dialect == "spark" else f'"{name}"'
97
+
98
+
99
+ def table_ref(name: str) -> str:
100
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*){0,2}", name):
101
+ raise QualityError(f"table name {name!r} is not a plain identifier")
102
+ return name
103
+
104
+
105
+ def allowed(f) -> tuple:
106
+ """Declared values; empty for engine copies older than `Field.values`."""
107
+ return tuple(getattr(f, "values", ()) or ())
108
+
109
+
110
+ def is_numeric(f) -> bool:
111
+ return (f.unit or "").strip().lower() in NUMERIC_UNITS
112
+
113
+
114
+ def plan_queries(tables: list, where: str | None, dialect: str, skip: set, batch: int = PROFILE_BATCH,
115
+ key: list | None = None) -> list:
116
+ cond = f" WHERE {where}" if where else ""
117
+ queries = []
118
+ for t in tables:
119
+ if key:
120
+ unknown = [k for k in key if k not in {f.name for f in t.fields}]
121
+ if unknown:
122
+ raise QualityError(f"--key column(s) not documented in {t.name}: {', '.join(unknown)}")
123
+ cols = ", ".join(quote(k, dialect) for k in key)
124
+ sql = (f"SELECT COUNT(*) AS dup_keys, COALESCE(SUM(c), 0) AS dup_rows FROM "
125
+ f"(SELECT COUNT(*) AS c FROM {table_ref(t.name)}{cond} GROUP BY {cols} HAVING COUNT(*) > 1) dup")
126
+ queries.append({"id": f"key:{t.name}", "table": t.name, "kind": "key", "columns": list(key), "sql": sql})
127
+ cols = [f for f in t.fields if f.name not in skip]
128
+ chunks = [cols[i:i + batch] for i in range(0, len(cols), batch)] or [[]]
129
+ for k, chunk in enumerate(chunks):
130
+ parts = ["COUNT(*) AS n"]
131
+ for i, f in enumerate(chunk):
132
+ q = quote(f.name, dialect)
133
+ parts += [f"COUNT({q}) AS nn_{i}", f"COUNT(DISTINCT {q}) AS d_{i}"]
134
+ if is_numeric(f) or FLAG_RE.search(f.name):
135
+ parts += [f"MIN({q}) AS mn_{i}", f"MAX({q}) AS mx_{i}"]
136
+ sql = f"SELECT {', '.join(parts)} FROM {table_ref(t.name)}{cond}"
137
+ suffix = f"#{k + 1}" if len(chunks) > 1 else ""
138
+ queries.append({"id": f"profile:{t.name}{suffix}", "table": t.name, "kind": "profile", "part": k,
139
+ "columns": [f.name for f in chunk], "sql": sql})
140
+ for f in cols:
141
+ if allowed(f):
142
+ q = quote(f.name, dialect)
143
+ sql = f"SELECT {q} AS value, COUNT(*) AS n FROM {table_ref(t.name)}{cond} GROUP BY {q}"
144
+ queries.append({"id": f"values:{t.name}.{f.name}", "table": t.name, "kind": "values",
145
+ "column": f.name, "sql": sql})
146
+ for q in queries:
147
+ assert_read_only(q["sql"])
148
+ return queries
149
+
150
+
151
+ def assert_read_only(sql: str) -> None:
152
+ if not re.match(r"\s*SELECT\b", sql, re.I) or ";" in sql:
153
+ raise QualityError(f"refusing to run a statement that is not a single SELECT: {sql[:80]}")
154
+
155
+
156
+ # ---------------------------------------------------------------------------
157
+ # Judge: results -> findings
158
+ # ---------------------------------------------------------------------------
159
+
160
+ @dataclass
161
+ class Finding:
162
+ table: str
163
+ column: str | None
164
+ kind: str
165
+ message: str
166
+ documented: bool = False # the definition already says so: report as info, not a problem
167
+
168
+
169
+ def number(value):
170
+ if value is None or value == "":
171
+ return None
172
+ try:
173
+ return float(value)
174
+ except (TypeError, ValueError):
175
+ return None
176
+
177
+
178
+ def where_columns(where: str | None) -> set:
179
+ """Identifiers a --where condition names: those columns are constant by construction."""
180
+ if not where:
181
+ return set()
182
+ bare = re.sub(r"'(?:[^']|'')*'", " ", where)
183
+ return {w.strip('`"').lower() for w in re.findall(r'[`"]?[A-Za-z_][A-Za-z0-9_]*[`"]?', bare)}
184
+
185
+
186
+ def judge(tables: list, queries: list, results: dict, where: str | None = None,
187
+ constant_min_rows: int = CONSTANT_MIN_ROWS) -> list:
188
+ by_table = {t.name: t for t in tables}
189
+ pinned = where_columns(where)
190
+ findings = []
191
+ for q in queries:
192
+ rows = results.get(q["id"])
193
+ if rows is None:
194
+ findings.append(Finding(q["table"], q.get("column"), "not-run", f"no result for {q['id']}"))
195
+ continue
196
+ t = by_table[q["table"]]
197
+ if q["kind"] == "key":
198
+ row = {k.lower(): v for k, v in (rows[0] if rows else {}).items()}
199
+ keys = number(row.get("dup_keys")) or 0
200
+ if keys:
201
+ findings.append(Finding(t.name, None, "duplicate-key",
202
+ f"{int(keys)} ({', '.join(q['columns'])}) value(s) repeat across "
203
+ f"{int(number(row.get('dup_rows')) or 0)} rows; the grain allows one each"))
204
+ continue
205
+ if q["kind"] == "profile":
206
+ part = judge_profile(t, q["columns"], rows, pinned, constant_min_rows)
207
+ if q.get("part", 0) > 0: # every batch sees the same row count; report an empty range once
208
+ part = [f for f in part if f.kind != "empty-table"]
209
+ findings += part
210
+ else:
211
+ findings += judge_values(t, q["column"], rows)
212
+ return findings
213
+
214
+
215
+ def judge_profile(t, columns: list, rows: list, pinned: set, constant_min_rows: int) -> list:
216
+ if len(rows) != 1:
217
+ return [Finding(t.name, None, "not-run", f"profile returned {len(rows)} rows, expected 1")]
218
+ row = {k.lower(): v for k, v in rows[0].items()}
219
+ n = number(row.get("n")) or 0
220
+ if n == 0:
221
+ return [Finding(t.name, None, "empty-table", "no rows in the checked range; nothing else can be judged")]
222
+ fields = {f.name: f for f in t.fields}
223
+ out = []
224
+ for i, name in enumerate(columns):
225
+ f = fields[name]
226
+ text = " ".join([*f.text.values(), f.source])
227
+ documented = bool(DOCUMENTED_CONSTANT_RE.search(text))
228
+ filled = number(row.get(f"nn_{i}")) or 0
229
+ distinct = number(row.get(f"d_{i}")) or 0
230
+ lo, hi = number(row.get(f"mn_{i}")), number(row.get(f"mx_{i}"))
231
+ if filled == 0:
232
+ out.append(Finding(t.name, name, "never-filled", f"empty in all {int(n)} rows", documented))
233
+ continue
234
+ if filled / n < MOSTLY_EMPTY:
235
+ out.append(Finding(t.name, name, "mostly-empty", f"filled in {int(filled)} of {int(n)} rows"))
236
+ if distinct == 1 and n >= constant_min_rows and name.lower() not in pinned:
237
+ value = row.get(f"mn_{i}")
238
+ shown = f" ({value})" if value is not None else ""
239
+ clear_flag = FLAG_RE.search(name) and str(row.get(f"mn_{i}")).strip().lower() in CLEAR_FLAG
240
+ # a build's version is the same on every row; a missing-flag at 0 means nothing was missing
241
+ expected = documented or bool(PROVENANCE_RE.search(name)) or bool(clear_flag)
242
+ out.append(Finding(t.name, name, "constant", f"one value{shown} in all {int(filled)} filled rows", expected))
243
+ unit = (f.unit or "").strip().lower()
244
+ if unit in NON_NEGATIVE_UNITS and lo is not None and lo < 0:
245
+ out.append(Finding(t.name, name, "negative-value", f"minimum {lo:g}, but the unit is {f.unit}"))
246
+ if unit in RANGES and lo is not None and hi is not None:
247
+ low, high = RANGES[unit]
248
+ if lo < low or hi > high:
249
+ out.append(Finding(t.name, name, "out-of-range",
250
+ f"values from {lo:g} to {hi:g}, outside {low:g}–{high:g} for unit {f.unit}"))
251
+ return out
252
+
253
+
254
+ def judge_values(t, column: str, rows: list) -> list:
255
+ f = next(f for f in t.fields if f.name == column)
256
+ seen = {}
257
+ for r in rows:
258
+ r = {k.lower(): v for k, v in r.items()}
259
+ value = r.get("value")
260
+ empty = value is None or (value == "" and "" not in allowed(f))
261
+ key = NULL_KEY if empty else str(value)
262
+ seen[key] = seen.get(key, 0) + (number(r.get("n")) or 0)
263
+ declared = set(allowed(f))
264
+ out = []
265
+ unseen = [v for v in allowed(f) if v not in seen]
266
+ if unseen and seen:
267
+ out.append(Finding(t.name, column, "unseen-value", f"declared but never present: {', '.join(unseen)}"))
268
+ extra = sorted(k for k in seen if k not in declared and k != NULL_KEY)
269
+ if extra:
270
+ counts = ", ".join(f"{k} ({int(seen[k])})" for k in extra[:10])
271
+ out.append(Finding(t.name, column, "undeclared-value", f"present but not declared: {counts}"))
272
+ return out
273
+
274
+
275
+ # ---------------------------------------------------------------------------
276
+ # Runners
277
+ # ---------------------------------------------------------------------------
278
+
279
+ def parse_rows(text: str) -> list:
280
+ text = text.strip()
281
+ if not text:
282
+ return []
283
+ if text[0] == "[":
284
+ return json.loads(text)
285
+ if text[0] == "{": # one object, or JSON lines (trino/presto --output-format JSON)
286
+ return [json.loads(line) for line in text.splitlines() if line.strip()]
287
+ return [dict(r) for r in csv.DictReader(io.StringIO(text))]
288
+
289
+
290
+ def run_command(cmd: str, sql: str) -> list:
291
+ proc = subprocess.run(shlex.split(cmd), input=sql, capture_output=True, text=True)
292
+ if proc.returncode != 0:
293
+ raise QualityError(f"runner failed ({proc.returncode}) on: {sql[:80]}\n{proc.stderr.strip()[-400:]}")
294
+ return parse_rows(proc.stdout)
295
+
296
+
297
+ def run_sqlite(db: str, sql: str) -> list:
298
+ con = sqlite3.connect(f"file:{db}?mode=ro", uri=True) # read-only by construction
299
+ try:
300
+ cur = con.execute(sql)
301
+ names = [d[0] for d in cur.description]
302
+ return [dict(zip(names, row)) for row in cur.fetchall()]
303
+ finally:
304
+ con.close()
305
+
306
+
307
+ def execute(queries: list, runner: str | None, sqlite_db: str | None) -> dict:
308
+ results = {}
309
+ for q in queries:
310
+ assert_read_only(q["sql"])
311
+ results[q["id"]] = run_sqlite(sqlite_db, q["sql"]) if sqlite_db else run_command(runner, q["sql"])
312
+ return results
313
+
314
+
315
+ # ---------------------------------------------------------------------------
316
+ # Report and CLI
317
+ # ---------------------------------------------------------------------------
318
+
319
+ def report(findings: list, tables: list, where: str | None, as_json: bool) -> int:
320
+ problems = [f for f in findings if not f.documented]
321
+ if as_json:
322
+ print(json.dumps({"engine": ENGINE_VERSION, "where": where, "tables": [t.name for t in tables],
323
+ "problems": len(problems),
324
+ "findings": [vars(f) for f in findings]}, ensure_ascii=False, indent=2))
325
+ return 1 if problems else 0
326
+ scope = f" where {where}" if where else " (all rows)"
327
+ print(f"data-quality: {len(tables)} table(s){scope}; {len(problems)} problem(s), "
328
+ f"{len(findings) - len(problems)} already documented")
329
+ for t in tables:
330
+ mine = [f for f in findings if f.table == t.name]
331
+ if not mine:
332
+ continue
333
+ print(f"\n{t.name}")
334
+ for f in mine:
335
+ tag = " (documented)" if f.documented else ""
336
+ print(f" - {f.kind}{tag}: {f.column + ': ' if f.column else ''}{f.message}")
337
+ return 1 if problems else 0
338
+
339
+
340
+ def engine_identity() -> str:
341
+ return f"{ENGINE_VERSION}+{hashlib.sha256(Path(__file__).read_bytes()).hexdigest()[:12]}"
342
+
343
+
344
+ def find_root(start: Path) -> Path:
345
+ for candidate in [start, *start.parents]:
346
+ if (candidate / "data-catalog.json").exists() or (candidate / ".git").exists():
347
+ return candidate
348
+ return start
349
+
350
+
351
+ def main(argv: list | None = None) -> int:
352
+ parser = argparse.ArgumentParser(prog="data_quality.py", description=__doc__.split("\n\n")[0])
353
+ parser.add_argument("--root", type=Path, help="repo root (default: nearest dir with data-catalog.json or .git)")
354
+ parser.add_argument("--catalog-engine", help=f"path to data_catalog.py (default <root>/{CATALOG_ENGINE})")
355
+ sub = parser.add_subparsers(dest="command", required=True)
356
+
357
+ def scoped(p):
358
+ p.add_argument("--table", action="append", default=[], help="documented table to check; repeatable (default all)")
359
+ p.add_argument("--where", help="SQL condition that bounds the scan, e.g. \"dt >= '2026-10-01'\"")
360
+ p.add_argument("--dialect", choices=["ansi", "spark"], default="ansi", help="identifier quoting: ansi \"x\", spark `x`")
361
+ p.add_argument("--skip", action="append", default=[], help="column to leave out (e.g. arrays); repeatable")
362
+ p.add_argument("--key", action="append", default=[],
363
+ help="grain column; repeat for a composite key: --key dt --key cust_id")
364
+ p.add_argument("--batch", type=int, default=PROFILE_BATCH,
365
+ help=f"columns per profile query (default {PROFILE_BATCH}); lower it if the engine caps stages")
366
+ p.add_argument("--constant-min-rows", type=int, default=CONSTANT_MIN_ROWS,
367
+ help=f"rows needed before a single value counts as constant (default {CONSTANT_MIN_ROWS})")
368
+
369
+ p_plan = sub.add_parser("plan", help="print the SELECT checks as JSON")
370
+ scoped(p_plan)
371
+ p_judge = sub.add_parser("judge", help="judge results; --results maps query id to rows")
372
+ scoped(p_judge)
373
+ p_judge.add_argument("--results", required=True, help="JSON file: {query id: [row objects]}")
374
+ p_judge.add_argument("--json", action="store_true")
375
+ p_run = sub.add_parser("run", help="plan, execute, judge")
376
+ scoped(p_run)
377
+ group = p_run.add_mutually_exclusive_group(required=True)
378
+ group.add_argument("--runner", help="command that reads SQL on stdin and prints JSON or CSV rows")
379
+ group.add_argument("--sqlite", help="SQLite database file (opened read-only)")
380
+ p_run.add_argument("--json", action="store_true")
381
+ sub.add_parser("version", help="print the engine identity")
382
+ args = parser.parse_args(argv)
383
+ if args.command == "version":
384
+ print(engine_identity())
385
+ return 0
386
+ root = (args.root or find_root(Path.cwd())).resolve()
387
+ try:
388
+ tables = documented_tables(root, args.catalog_engine, args.table)
389
+ queries = plan_queries(tables, args.where, args.dialect, set(args.skip), args.batch, args.key)
390
+ if args.command == "plan":
391
+ print(json.dumps(queries, ensure_ascii=False, indent=2))
392
+ return 0
393
+ if args.command == "judge":
394
+ results = json.loads(Path(args.results).read_text(encoding="utf-8"))
395
+ else:
396
+ results = execute(queries, args.runner, args.sqlite)
397
+ found = judge(tables, queries, results, args.where, args.constant_min_rows)
398
+ return report(found, tables, args.where, args.json)
399
+ except QualityError as exc:
400
+ print(f"data-quality: {exc}", file=sys.stderr)
401
+ return 2
402
+
403
+
404
+ if __name__ == "__main__":
405
+ sys.exit(main())
@@ -0,0 +1,277 @@
1
+ """Tests for data_quality.py against a real repo and a real SQLite warehouse. Run: python3 test_data_quality.py"""
2
+ from __future__ import annotations
3
+
4
+ import contextlib
5
+ import io
6
+ import json
7
+ import shutil
8
+ import sqlite3
9
+ import sys
10
+ import tempfile
11
+ import textwrap
12
+ import unittest
13
+ from pathlib import Path
14
+
15
+ HERE = Path(__file__).resolve().parent
16
+ CATALOG_ENGINE = HERE.parent.parent / "data-catalog" / "scripts" / "data_catalog.py"
17
+ sys.path.insert(0, str(HERE))
18
+ import data_quality as dq # noqa: E402
19
+
20
+ SCRIPT = '''
21
+ from catalog_fields import Field, Table
22
+
23
+ CATALOG_TABLE = Table(
24
+ "customer_daily",
25
+ en="Daily customer metrics.",
26
+ grain_en="One row per (dt, cust_id).",
27
+ )
28
+ CATALOG_FIELDS = (
29
+ Field("dt", en="Snapshot date.", source="run parameter dt"),
30
+ Field("cust_id", en="Customer id.", source="dwd.transaction.cust_id"),
31
+ Field("coll_cnt_7d", en="Number of collections in the last 7 days.", source="count of rows", unit="count"),
32
+ Field("recall_ratio", en="Recalled share of collected amount, 0-1.", source="recall / collected", unit="ratio"),
33
+ Field("agent_risk", en="Agent risk: yes, no or undetermined.", source="agent list and recalls",
34
+ values=("是", "否", "无法判定")),
35
+ Field("result", en="Transaction result.", source="constant in v1", values=("SUCCESS", "FAILED")),
36
+ Field("cert_type", en="Payer certificate type.", source="dwd.transaction.debtor_cert_type"),
37
+ Field("legacy_flag", en="Legacy flag; always empty since v2.", source="none"),
38
+ Field("channel", en="Collection channel.", source="dwd.transaction.channel", values=("bank", "card")),
39
+ )
40
+ PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)
41
+ '''
42
+
43
+ ROWS = [
44
+ # dt, cust_id, coll_cnt_7d, recall_ratio, agent_risk, result, cert_type, legacy_flag, channel
45
+ ("2026-10-01", "c1", 3, 0.1, "是", "SUCCESS", None, None, "bank"),
46
+ ("2026-10-01", "c2", 0, 1.4, "无法判定", "SUCCESS", None, None, "card"),
47
+ ("2026-10-02", "c1", -1, 0.0, "无法判定", "SUCCESS", None, None, "wallet"),
48
+ ("2026-10-02", "c3", 5, None, "是", "SUCCESS", None, None, "bank"),
49
+ ]
50
+
51
+
52
+ def make_repo(rows=ROWS, script=SCRIPT, extra_column=None) -> tuple[Path, Path]:
53
+ root = Path(tempfile.mkdtemp(prefix="data-quality-test-"))
54
+ (root / ".git").mkdir()
55
+ (root / "scripts/data-catalog").mkdir(parents=True)
56
+ shutil.copy(CATALOG_ENGINE, root / "scripts/data-catalog/data_catalog.py")
57
+ (root / "jobs").mkdir()
58
+ (root / "jobs/customer_daily.py").write_text(textwrap.dedent(script), encoding="utf-8")
59
+ (root / "data-catalog.json").write_text(json.dumps({"version": 1, "scripts": ["jobs/*.py"], "languages": ["en"]}),
60
+ encoding="utf-8")
61
+ db = root / "warehouse.db"
62
+ con = sqlite3.connect(db)
63
+ columns = ["dt TEXT", "cust_id TEXT", "coll_cnt_7d INTEGER", "recall_ratio REAL", "agent_risk TEXT", "result TEXT",
64
+ "cert_type TEXT", "legacy_flag TEXT", "channel TEXT"]
65
+ if extra_column:
66
+ columns.insert(6, extra_column)
67
+ con.execute(f"CREATE TABLE customer_daily ({', '.join(columns)})")
68
+ con.executemany(f"INSERT INTO customer_daily VALUES ({', '.join('?' * len(columns))})", rows)
69
+ con.commit()
70
+ con.close()
71
+ return root, db
72
+
73
+
74
+ def run(root: Path, *argv: str) -> tuple[int, str, str]:
75
+ out, err = io.StringIO(), io.StringIO()
76
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
77
+ code = dq.main(["--root", str(root), *argv])
78
+ return code, out.getvalue(), err.getvalue()
79
+
80
+
81
+ def findings(root: Path, db: Path, *extra: str) -> tuple[int, dict]:
82
+ code, out, err = run(root, "run", "--sqlite", str(db), "--json", "--constant-min-rows", "2", *extra)
83
+ assert out, err
84
+ report = json.loads(out)
85
+ return code, {(f["column"], f["kind"]): f for f in report["findings"]}
86
+
87
+
88
+ class RunTest(unittest.TestCase):
89
+ def test_each_check_fires_on_the_column_that_breaks_it(self):
90
+ root, db = make_repo()
91
+ code, found = findings(root, db)
92
+ self.assertEqual(code, 1)
93
+ self.assertEqual(found[("coll_cnt_7d", "negative-value")]["message"], "minimum -1, but the unit is count")
94
+ self.assertEqual(found[("recall_ratio", "out-of-range")]["message"],
95
+ "values from 0 to 1.4, outside 0–1 for unit ratio")
96
+ self.assertEqual(found[("agent_risk", "unseen-value")]["message"], "declared but never present: 否")
97
+ self.assertEqual(found[("channel", "undeclared-value")]["message"], "present but not declared: wallet (1)")
98
+ self.assertEqual(found[("result", "constant")]["message"], "one value in all 4 filled rows")
99
+ self.assertEqual(found[("result", "unseen-value")]["message"], "declared but never present: FAILED")
100
+ self.assertEqual(found[("cert_type", "never-filled")]["message"], "empty in all 4 rows")
101
+ self.assertFalse(found[("cert_type", "never-filled")]["documented"])
102
+ # the definition already says "always empty": reported, but as documented
103
+ self.assertTrue(found[("legacy_flag", "never-filled")]["documented"])
104
+ # clean columns raise nothing
105
+ self.assertFalse([k for k in found if k[0] in ("dt", "cust_id")])
106
+
107
+ def test_a_missing_flag_constant_at_zero_is_healthy_but_at_one_is_not(self):
108
+ script = SCRIPT.replace('Field("cert_type"', 'Field("fx_missing", en="1 if any amount lacked a rate.", source="flag"),\n Field("cert_type"')
109
+ for value, healthy in ((0, True), (1, False)):
110
+ rows = [r[:6] + (value,) + r[6:] for r in ROWS]
111
+ root, db = make_repo(rows, script, extra_column="fx_missing INTEGER")
112
+ _, found = findings(root, db)
113
+ self.assertEqual(found[("fx_missing", "constant")]["documented"], healthy, value)
114
+
115
+ def test_version_and_hash_columns_are_expected_to_be_constant(self):
116
+ script = SCRIPT.replace('Field("cert_type"', 'Field("rule_version", en="Rule set version.", source="RULE_VERSION"),\n Field("cert_type"')
117
+ rows = [r[:6] + ("v2",) + r[6:] for r in ROWS]
118
+ root = Path(tempfile.mkdtemp(prefix="data-quality-test-"))
119
+ root, db = make_repo(rows, script, extra_column="rule_version TEXT")
120
+ _, found = findings(root, db)
121
+ self.assertTrue(found[("rule_version", "constant")]["documented"])
122
+
123
+ def test_duplicate_grain_keys_are_counted(self):
124
+ rows = ROWS + [ROWS[0]] # (2026-10-01, c1) twice
125
+ root, db = make_repo(rows)
126
+ _, found = findings(root, db, "--key", "dt", "--key", "cust_id")
127
+ self.assertEqual(found[(None, "duplicate-key")]["message"],
128
+ "1 (dt, cust_id) value(s) repeat across 2 rows; the grain allows one each")
129
+ root, db = make_repo()
130
+ _, clean = findings(root, db, "--key", "dt", "--key", "cust_id")
131
+ self.assertNotIn((None, "duplicate-key"), clean)
132
+ code, _, err = run(root, "plan", "--key", "nope")
133
+ self.assertEqual(code, 2)
134
+ self.assertIn("--key column(s) not documented in customer_daily: nope", err)
135
+
136
+ def test_where_bounds_the_scan(self):
137
+ root, db = make_repo()
138
+ _, found = findings(root, db, "--where", "dt = '2026-10-01'")
139
+ self.assertNotIn(("coll_cnt_7d", "negative-value"), found)
140
+ self.assertNotIn(("channel", "undeclared-value"), found)
141
+ self.assertIn(("recall_ratio", "out-of-range"), found)
142
+
143
+ def test_where_columns_and_small_samples_are_not_called_constant(self):
144
+ root, db = make_repo()
145
+ _, found = findings(root, db, "--where", "dt = '2026-10-01'")
146
+ self.assertNotIn(("dt", "constant"), found)
147
+ code, out, _ = run(root, "run", "--sqlite", str(db), "--json") # default minimum: 30 rows
148
+ self.assertNotIn("constant", {f["kind"] for f in json.loads(out)["findings"]})
149
+
150
+ def test_an_empty_range_is_reported_once_and_judges_nothing_else(self):
151
+ root, db = make_repo()
152
+ code, found = findings(root, db, "--where", "dt = '1999-01-01'")
153
+ self.assertEqual(code, 1)
154
+ self.assertEqual(list(found), [(None, "empty-table")])
155
+
156
+ def test_clean_data_exits_zero(self):
157
+ clean = [(dt, c, n, ratio, a, r, cert, None, ch) for dt, c, n, ratio, a, r, cert, ch in [
158
+ ("2026-10-01", "c1", 3, 0.1, "是", "SUCCESS", "ID", "bank"),
159
+ ("2026-10-02", "c2", 0, 0.0, "否", "FAILED", "PASSPORT", "card"),
160
+ ("2026-10-03", "c3", 7, 1.0, "无法判定", "SUCCESS", "ID", "bank")]]
161
+ root, db = make_repo(clean)
162
+ code, found = findings(root, db)
163
+ self.assertEqual(code, 0, found)
164
+ self.assertEqual([k for k, f in found.items() if not f["documented"]], [])
165
+
166
+ def test_external_runner_gets_only_selects_and_its_csv_or_json_is_read(self):
167
+ root, db = make_repo()
168
+ runner = root / "runner.py"
169
+ log = root / "sql.log"
170
+ runner.write_text(textwrap.dedent(f'''
171
+ import csv, sqlite3, sys
172
+ sql = sys.stdin.read()
173
+ open({str(log)!r}, "a").write(sql + "\\n")
174
+ con = sqlite3.connect({str(db)!r})
175
+ cur = con.execute(sql)
176
+ w = csv.writer(sys.stdout)
177
+ w.writerow([d[0] for d in cur.description])
178
+ w.writerows(cur.fetchall())
179
+ '''), encoding="utf-8")
180
+ code, out, err = run(root, "run", "--runner", f"{sys.executable} {runner}", "--json")
181
+ self.assertEqual(code, 1, err)
182
+ kinds = {(f["column"], f["kind"]) for f in json.loads(out)["findings"]}
183
+ self.assertIn(("coll_cnt_7d", "negative-value"), kinds)
184
+ statements = [s for s in log.read_text(encoding="utf-8").split("\n") if s.strip()]
185
+ self.assertTrue(statements and all(s.startswith("SELECT ") for s in statements))
186
+
187
+
188
+ class PlanJudgeTest(unittest.TestCase):
189
+ def test_plan_then_judge_matches_run(self):
190
+ root, db = make_repo()
191
+ code, out, _ = run(root, "plan", "--dialect", "spark", "--skip", "channel")
192
+ self.assertEqual(code, 0)
193
+ queries = json.loads(out)
194
+ self.assertEqual([q["id"] for q in queries],
195
+ ["profile:customer_daily", "values:customer_daily.agent_risk", "values:customer_daily.result"])
196
+ self.assertIn("COUNT(DISTINCT `coll_cnt_7d`) AS d_2", queries[0]["sql"])
197
+ self.assertIn("MIN(`recall_ratio`) AS mn_3", queries[0]["sql"])
198
+ self.assertNotIn("channel", queries[0]["sql"])
199
+ con = sqlite3.connect(db)
200
+ results = {}
201
+ for q in queries:
202
+ cur = con.execute(q["sql"].replace("`", '"'))
203
+ results[q["id"]] = [dict(zip([d[0] for d in cur.description], r)) for r in cur.fetchall()]
204
+ path = root / "results.json"
205
+ path.write_text(json.dumps(results), encoding="utf-8")
206
+ code, out, _ = run(root, "judge", "--skip", "channel", "--results", str(path), "--json", "--constant-min-rows", "2")
207
+ kinds = {(f["column"], f["kind"]) for f in json.loads(out)["findings"]}
208
+ self.assertIn(("agent_risk", "unseen-value"), kinds)
209
+ self.assertNotIn(("channel", "undeclared-value"), kinds)
210
+
211
+ def test_profile_batches_cover_every_column_and_report_an_empty_range_once(self):
212
+ root, db = make_repo()
213
+ code, out, _ = run(root, "plan", "--batch", "4")
214
+ ids = [q["id"] for q in json.loads(out) if q["kind"] == "profile"]
215
+ self.assertEqual(ids, ["profile:customer_daily#1", "profile:customer_daily#2", "profile:customer_daily#3"])
216
+ cols = [c for q in json.loads(out) if q["kind"] == "profile" for c in q["columns"]]
217
+ self.assertEqual(len(cols), 9)
218
+ _, found = findings(root, db, "--batch", "4")
219
+ self.assertIn(("recall_ratio", "out-of-range"), found)
220
+ self.assertIn(("legacy_flag", "never-filled"), found)
221
+ _, empty = findings(root, db, "--batch", "4", "--where", "dt = 'none'")
222
+ self.assertEqual(list(empty), [(None, "empty-table")])
223
+
224
+ def test_a_missing_result_is_reported_not_skipped(self):
225
+ root, _ = make_repo()
226
+ path = root / "results.json"
227
+ path.write_text("{}", encoding="utf-8")
228
+ code, out, _ = run(root, "judge", "--results", str(path), "--json")
229
+ self.assertEqual(code, 1)
230
+ self.assertIn("not-run", {f["kind"] for f in json.loads(out)["findings"]})
231
+
232
+
233
+ class OldEngineTest(unittest.TestCase):
234
+ def test_fields_from_an_engine_copy_without_values_still_plan(self):
235
+ from types import SimpleNamespace as NS
236
+ old = NS(name="t", fields=[NS(name="a", text={"en": "x"}, source="s", window=None, unit="count", line=1)])
237
+ [query] = dq.plan_queries([old], None, "ansi", set())
238
+ self.assertEqual(query["sql"], 'SELECT COUNT(*) AS n, COUNT("a") AS nn_0, COUNT(DISTINCT "a") AS d_0, '
239
+ 'MIN("a") AS mn_0, MAX("a") AS mx_0 FROM t')
240
+
241
+
242
+ class RowsTest(unittest.TestCase):
243
+ def test_runner_output_formats(self):
244
+ self.assertEqual(dq.parse_rows('[{"n": 1}]'), [{"n": 1}])
245
+ self.assertEqual(dq.parse_rows('{"value": "a", "n": 2}\n{"value": "b", "n": 3}\n'),
246
+ [{"value": "a", "n": 2}, {"value": "b", "n": 3}])
247
+ self.assertEqual(dq.parse_rows("value,n\na,2\n"), [{"value": "a", "n": "2"}])
248
+ self.assertEqual(dq.parse_rows(""), [])
249
+
250
+
251
+ class SetupTest(unittest.TestCase):
252
+ def test_without_the_catalog_engine_it_explains_the_setup(self):
253
+ root, _ = make_repo()
254
+ (root / "scripts/data-catalog/data_catalog.py").unlink()
255
+ code, _, err = run(root, "plan")
256
+ self.assertEqual(code, 2)
257
+ self.assertIn("set the repo up with the data-catalog skill first", err)
258
+
259
+ def test_an_unknown_table_is_refused(self):
260
+ root, _ = make_repo()
261
+ code, _, err = run(root, "plan", "--table", "nope")
262
+ self.assertEqual(code, 2)
263
+ self.assertIn("not documented in the catalog: nope", err)
264
+
265
+ def test_statements_other_than_one_select_are_refused(self):
266
+ for sql in ("DELETE FROM t", "SELECT 1; DROP TABLE t"):
267
+ with self.assertRaises(dq.QualityError):
268
+ dq.assert_read_only(sql)
269
+
270
+ def test_the_sqlite_runner_cannot_write(self):
271
+ root, db = make_repo()
272
+ with self.assertRaises(sqlite3.OperationalError):
273
+ dq.run_sqlite(str(db), "INSERT INTO customer_daily (dt) VALUES ('x')")
274
+
275
+
276
+ if __name__ == "__main__":
277
+ unittest.main()
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: ios-foundation
3
3
  description: "Set up and prove an Expo / React Native native iOS workspace from a clean checkout to a manual Login smoke on an owned iOS Simulator: isolated worktree and private caches, Xcode/runtime/CocoaPods preflight, frozen install with `pod install --deployment`, platform-scoped prebuild, an ad-hoc-signed Debug simulator build, a private simulator by exact UDID, Metro on a free port, a disposable PostgreSQL backend with owned cleanup, and the evidence record (setup proof only; feature work reports the smoke result). Use for 'set up iOS', 'build the iOS app', 'run the app on the iOS simulator', 'add iOS to the React Native app', 'prove the iOS build', or when a pod install, xcodebuild, simulator boot, Metro connection or Keychain session fails. NOT for Android (android-foundation), signing, TestFlight or App Store release, or general workflow YAML (github-actions)."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # iOS foundation
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: monorepo
3
3
  description: Turborepo architecture for shared packages, pnpm workspaces, DAL boundaries, and pipelines.
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Turborepo Monorepo Architecture
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: nextjs-fullstack
3
3
  description: "Build or debug Next.js App Router features with React 19 — pages/layouts/route handlers, Server/Client Components, Server Actions, forms, middleware, DAL, strict TypeScript, Zod — plus our conventions: nuqs URL state, SWR cache, Skeleton/Progress + Sonner feedback, drag-and-drop upload, the portal app shell, list pages as filter/sort/paginate tables. shadcn component mechanics: see the shadcn skill."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Next.js 16 Fullstack Development
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: prisma-setup
3
3
  description: Use when adding or debugging Prisma with PostgreSQL, Next.js, or Turborepo, including schema.prisma, prisma.config.ts, client generation, migrations, seeds, relations, adapters, and DAL/database-layer setup. Triggers on phrases like "Prisma schema", "prisma migrate", "schema.prisma", "seed the database", "PrismaClient error", "relation issue", or "database layer".
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Prisma v7 Setup Guide
@@ -12,7 +12,7 @@ description: >
12
12
  Connect, spark-submit, Hudi commits, backfills, validate-only canaries,
13
13
  cluster reconciliation, or Spark event-log profiling — including warehouse
14
14
  job timeouts, empty Hudi writes, or EMR/Dataproc performance regressions.
15
- release-version: 0.38.0
15
+ release-version: 0.39.0
16
16
  ---
17
17
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
18
18
  # PySpark
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: python-backend
3
3
  description: "Python backend architecture for FastAPI, Tortoise ORM, Pydantic Settings, loguru, and uv. Also: uv.lock installs in CI/Docker via a proxy or mirror (`uv sync --frozen` bypasses it), slow uv installs on runners, relocking."
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Python Backend Conventions
@@ -2,7 +2,7 @@
2
2
  name: shadcn
3
3
  description: "Add or debug shadcn/ui in a React/Next.js project: components.json, shadcn init/add, presets, registries, Tailwind styling, base/radix/aria composition, component fixes, and list-index pages — where the default is the DiceUI data table with filter/sort/pagination in the URL via nuqs, not a bare shadcn Table."
4
4
  allowed-tools: Bash(npx shadcn@latest *), Bash(pnpm dlx shadcn@latest *), Bash(bunx --bun shadcn@latest *)
5
- release-version: 0.38.0
5
+ release-version: 0.39.0
6
6
  ---
7
7
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
8
8
  # shadcn/ui
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: zod-v4
3
3
  description: Zod v4 schema conventions and v3 migration for errors, validators, objects, enums, records, and async parsing.
4
- release-version: 0.38.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Zod v3 to v4 Migration