@1aboveio/app-dev-skills 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -0
- package/package.json +2 -2
- package/skills/airflow/SKILL.md +1 -1
- package/skills/android-foundation/SKILL.md +1 -1
- package/skills/app-debug/SKILL.md +1 -1
- package/skills/better-auth/SKILL.md +1 -1
- package/skills/data-catalog/SKILL.md +2 -1
- package/skills/data-catalog/references/data-catalog-conventions.md +2 -1
- package/skills/data-catalog/references/field-definitions.md +2 -1
- package/skills/data-catalog/references/format.md +6 -2
- package/skills/data-catalog/scripts/data_catalog.py +15 -3
- package/skills/data-catalog/scripts/test_data_catalog.py +15 -0
- package/skills/data-quality/LICENSE +3 -0
- package/skills/data-quality/SKILL.md +96 -0
- package/skills/data-quality/agents/openai.yaml +3 -0
- package/skills/data-quality/references/checks.md +56 -0
- package/skills/data-quality/release-check.mjs +93 -0
- package/skills/data-quality/scripts/data-quality.test.mjs +12 -0
- package/skills/data-quality/scripts/data_quality.py +405 -0
- package/skills/data-quality/scripts/test_data_quality.py +277 -0
- package/skills/ios-foundation/SKILL.md +1 -1
- package/skills/monorepo/SKILL.md +1 -1
- package/skills/nextjs-fullstack/SKILL.md +1 -1
- package/skills/prisma-setup/SKILL.md +1 -1
- package/skills/pyspark/SKILL.md +1 -1
- package/skills/python-backend/SKILL.md +1 -1
- package/skills/shadcn/SKILL.md +1 -1
- package/skills/zod-v4/SKILL.md +1 -1
package/README.md
CHANGED
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@1aboveio/app-dev-skills",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.2",
|
|
4
4
|
"description": "Next.js, React Native, auth, Prisma, shadcn, Python backends, Airflow and PySpark.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -27,7 +27,7 @@
|
|
|
27
27
|
"id": "app-dev",
|
|
28
28
|
"core": "@1aboveio/core-skills",
|
|
29
29
|
"coreSkills": [],
|
|
30
|
-
"digest": "
|
|
30
|
+
"digest": "de7c7e63b8cf033c471b81777170064bef838b906cc332885b0a3a652f2ff96a"
|
|
31
31
|
},
|
|
32
32
|
"publishConfig": {
|
|
33
33
|
"registry": "https://registry.npmjs.org/",
|
package/skills/airflow/SKILL.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: airflow
|
|
3
3
|
description: Airflow DAG architecture and conventions. Use when designing, writing, reviewing, testing, or debugging Airflow DAGs and tasks, including TaskGroups, data intervals, timezone-aware date parameters, CLI validation, environment-based scheduling, and pytest integrity tests.
|
|
4
4
|
user-invocable: true
|
|
5
|
-
release-version: 0.
|
|
5
|
+
release-version: 0.39.0
|
|
6
6
|
---
|
|
7
7
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
8
8
|
# Airflow DAG Development Patterns
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: android-foundation
|
|
3
3
|
description: "Prove an Expo / React Native app on a native Android emulator, and fix it when that proof fails. Use before you fix and re-run or re-dispatch a failed Android CI job (`gh workflow run android.yml`, `gh run rerun`): a re-run is not a debugger, so reproduce the failing journey checkpoint off CI first. Also use when a UiAutomator journey or its Python harness tests fail (a CHECK/FAIL line, a wait that timed out, a control not found, adb offline, a lost Metro bridge, a fixture failing on the runner), to build the debug APK and run the device journey, to set up a fresh Android workspace (SDK/NDK/JDK, Gradle caches, private AVD, disposable PostgreSQL), or to add the emulator job to CI. NOT for workflow YAML, CI speed or runner design (github-actions), browser tests (e2e-test), iOS (ios-foundation), plain Jest component tests, or signing and store release."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Android foundation
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: app-debug
|
|
3
3
|
description: "Diagnose runtime failures in deployed Cloud Run services — 5xx/422 errors, tracebacks, Next.js SSR/hydration or Server Action failures, DB/Secret Manager connection errors, cold-start timeouts, container OOM. NOT for build or deploy failures (cloud-debug)."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# App Debug
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: better-auth
|
|
3
3
|
description: "Add or debug Better Auth for Next.js App Router: login, sessions, protected routes, middleware, OAuth (including Feishu/Lark), cookies, Prisma, and Auth.js migration."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Better Auth for Next.js
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: data-catalog
|
|
3
3
|
description: "Column definitions for warehouse tables, written inside the Spark/PySpark or SQL script that builds each table (CATALOG_TABLE / CATALOG_FIELDS, or a `-- @field` comment block), with a generated markdown catalog in docs/catalog/ and a CI check that fails a PR whose catalog is stale. Use when writing or changing a table script that adds, renames, removes, or changes the meaning of an output column or feature; when asked what a warehouse column or feature means; to set a repo up ('init the data catalog', 'generate the data dictionary', '字段字典', 'add catalog CI'); or to audit existing SQL or Spark scripts for undocumented columns. NOT for implementing or tuning the Spark job itself (pyspark), DAG wiring (airflow), or general workflow YAML (github-actions)."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Data catalog
|
|
@@ -42,6 +42,7 @@ Never document tables during setup unless the task asks for it.
|
|
|
42
42
|
| CI says the catalog is stale | `python3 <engine> generate`, commit the pages |
|
|
43
43
|
| Before committing a table-script change, or reviewing one | `python3 <engine> drift` (`--base origin/main` for a whole branch) |
|
|
44
44
|
| `Field`/`Table` arguments, the SQL block, config keys, exit codes | [references/format.md](references/format.md) |
|
|
45
|
+
| Does the table's data match its definitions? | the data-quality skill |
|
|
45
46
|
|
|
46
47
|
`<engine>` is `scripts/data-catalog/data_catalog.py` in the repo, or
|
|
47
48
|
`<skill-base-dir>/scripts/data_catalog.py` before the repo has a copy. When
|
|
@@ -52,7 +52,8 @@ table. `docs/catalog/` is generated from those definitions; never edit it by han
|
|
|
52
52
|
collections.", not "0 when none".
|
|
53
53
|
9. Never restate the column name or write "same as above".
|
|
54
54
|
10. Cover every case: what the column holds for each kind of row, including when it
|
|
55
|
-
is empty; list an enumerated column's values
|
|
55
|
+
is empty; list an enumerated column's values in the text and in `values=(...)`;
|
|
56
|
+
say so if it is always empty today.
|
|
56
57
|
11. Write related columns as a set: a count and its amount, a value and its missing
|
|
57
58
|
flag, a legacy column and its replacement use the same scope and wording, and any
|
|
58
59
|
difference is stated in both rows.
|
|
@@ -50,7 +50,8 @@ column that says only what is particular to that column.
|
|
|
50
50
|
`"同上,30 天窗口"` tell the reader nothing.
|
|
51
51
|
10. **Cover every case.** Say what the column holds for every kind of row (each
|
|
52
52
|
transaction type, each branch of the code), including when it is empty. List
|
|
53
|
-
the values an enumerated column can take, with a Chinese gloss for English codes
|
|
53
|
+
the values an enumerated column can take, with a Chinese gloss for English codes,
|
|
54
|
+
and put the same list in `values=(...)` so the data can be checked against it.
|
|
54
55
|
A column that is always empty or constant today says so.
|
|
55
56
|
Bad: `"换汇交易为 COMPLETED,内部转账为空。"` (collections and payouts unsaid)
|
|
56
57
|
Good: `"换汇交易为 COMPLETED;收款、付款和内部转账为空。"`
|
|
@@ -41,7 +41,10 @@ CATALOG_FIELDS = (
|
|
|
41
41
|
PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
- `Field(name, <one keyword per language>, source, window=None, unit=None)`.
|
|
44
|
+
- `Field(name, <one keyword per language>, source, window=None, unit=None, values=())`.
|
|
45
|
+
`values` lists the values an enumerated column can take, as distinct strings
|
|
46
|
+
(`values=("是", "否", "无法判定")`); the data-quality skill checks the data against it.
|
|
47
|
+
Never use it on identifiers or free text.
|
|
45
48
|
- `Table(name, <one keyword per language>, grain_<lang>=..., notes_<lang>=(...))`: a
|
|
46
49
|
grain for every language, and the same number of notes in every language (notes are
|
|
47
50
|
optional).
|
|
@@ -77,7 +80,8 @@ SELECT ...
|
|
|
77
80
|
```
|
|
78
81
|
|
|
79
82
|
- A directive line is `-- @table <name>` or `-- @field <name>`.
|
|
80
|
-
- A key line is exactly two spaces after `-- `, then `key: value`.
|
|
83
|
+
- A key line is exactly two spaces after `-- `, then `key: value`. `values:` takes a
|
|
84
|
+
list split on `;` or `;`.
|
|
81
85
|
- A continuation line has four or more spaces and joins the previous key with a space.
|
|
82
86
|
- `notes_<lang>` on `@table` splits on `;` or `;`.
|
|
83
87
|
- The block ends at the first non-comment line. One `@table` per file.
|
|
@@ -49,7 +49,7 @@ DEFAULT_CONFIG = {
|
|
|
49
49
|
"tableNames": [],
|
|
50
50
|
"schemaNames": [],
|
|
51
51
|
}
|
|
52
|
-
FIELD_EXTRAS = ("source", "window", "unit")
|
|
52
|
+
FIELD_EXTRAS = ("source", "window", "unit", "values")
|
|
53
53
|
LANGUAGE_LABELS = {"en": "English", "zh": "中文", "zh_hant": "繁體中文", "ja": "日本語", "ko": "한국어"}
|
|
54
54
|
LANGUAGE_RE = re.compile(r"^[a-z]{2}(_[a-z]+)?$")
|
|
55
55
|
SKIP_DIRS = {"node_modules", "__pycache__", "venv"}
|
|
@@ -151,6 +151,7 @@ class FieldDoc:
|
|
|
151
151
|
window: str | None
|
|
152
152
|
unit: str | None
|
|
153
153
|
line: int
|
|
154
|
+
values: tuple = () # the values an enumerated column can take; checked by data-quality
|
|
154
155
|
|
|
155
156
|
|
|
156
157
|
@dataclass
|
|
@@ -258,7 +259,14 @@ def build_field(values: dict, languages: list, where: str, line: int) -> FieldDo
|
|
|
258
259
|
f"languages configured: {languages}")
|
|
259
260
|
text = {lang: require_text(values, lang, where, line, f"Field {name!r}") for lang in languages}
|
|
260
261
|
source = require_text(values, "source", where, line, f"Field {name!r}")
|
|
261
|
-
|
|
262
|
+
allowed = values.get("values") or ()
|
|
263
|
+
if isinstance(allowed, str):
|
|
264
|
+
allowed = tuple(v.strip() for v in re.split(r"[;;]", allowed))
|
|
265
|
+
if not isinstance(allowed, tuple) or not all(isinstance(v, str) and v != "" for v in allowed):
|
|
266
|
+
raise CatalogError(f"{where}:{line}: Field {name!r}: values must be a tuple of non-empty strings")
|
|
267
|
+
if len(set(allowed)) != len(allowed):
|
|
268
|
+
raise CatalogError(f"{where}:{line}: Field {name!r}: values lists a value twice")
|
|
269
|
+
return FieldDoc(name, text, source, (values.get("window") or None), (values.get("unit") or None), line, allowed)
|
|
262
270
|
|
|
263
271
|
|
|
264
272
|
def build_table(values: dict, fields: list, languages: list, where: str, line: int, script: str) -> TableDoc:
|
|
@@ -545,15 +553,18 @@ def render_table(table: TableDoc, root: Path, config: dict) -> str:
|
|
|
545
553
|
out += ["## Fields", ""]
|
|
546
554
|
has_window = any(f.window for f in table.fields)
|
|
547
555
|
has_unit = any(f.unit for f in table.fields)
|
|
556
|
+
has_values = any(f.values for f in table.fields)
|
|
548
557
|
head = ["Field", *(label(lang) for lang in langs), "Source"]
|
|
549
558
|
head += ["Window"] if has_window else []
|
|
550
559
|
head += ["Unit"] if has_unit else []
|
|
560
|
+
head += ["Values"] if has_values else []
|
|
551
561
|
out.append("| " + " | ".join(head) + " |")
|
|
552
562
|
out.append("|" + "---|" * len(head))
|
|
553
563
|
for f in table.fields:
|
|
554
564
|
row = [f"`{f.name}`", *(cell(f.text[lang]) for lang in langs), cell(f.source)]
|
|
555
565
|
row += [cell(f.window)] if has_window else []
|
|
556
566
|
row += [cell(f.unit)] if has_unit else []
|
|
567
|
+
row += [cell(", ".join(f"`{v}`" for v in f.values))] if has_values else []
|
|
557
568
|
out.append("| " + " | ".join(row) + " |")
|
|
558
569
|
return "\n".join(out) + "\n"
|
|
559
570
|
|
|
@@ -1175,7 +1186,7 @@ def code_without_metadata(text: str, suffix: str) -> str:
|
|
|
1175
1186
|
def definitions(table: TableDoc | None) -> tuple[dict, tuple]:
|
|
1176
1187
|
if table is None:
|
|
1177
1188
|
return {}, ()
|
|
1178
|
-
fields = {f.name: (tuple(sorted(f.text.items())), f.source, f.window, f.unit) for f in table.fields}
|
|
1189
|
+
fields = {f.name: (tuple(sorted(f.text.items())), f.source, f.window, f.unit, f.values) for f in table.fields}
|
|
1179
1190
|
notes = tuple((lang, tuple(n)) for lang, n in sorted(table.notes.items()))
|
|
1180
1191
|
return fields, (tuple(sorted(table.grain.items())), tuple(sorted(table.text.items())), notes)
|
|
1181
1192
|
|
|
@@ -1274,6 +1285,7 @@ class Field:
|
|
|
1274
1285
|
source: str
|
|
1275
1286
|
window: str | None = None
|
|
1276
1287
|
unit: str | None = None
|
|
1288
|
+
values: tuple[str, ...] = ()
|
|
1277
1289
|
|
|
1278
1290
|
|
|
1279
1291
|
@dataclass(frozen=True)
|
|
@@ -124,6 +124,21 @@ class GenerateTest(unittest.TestCase):
|
|
|
124
124
|
self.assertIn("| [`dwd.customer_daily`](dwd.customer_daily.md) | Daily customer collection rollup. | "
|
|
125
125
|
"one row per (dt, cust_id) | 2 |", index)
|
|
126
126
|
|
|
127
|
+
def test_values_render_and_must_be_distinct_strings(self):
|
|
128
|
+
script = PY_SCRIPT.replace('source="Run parameter `dt`."),', 'source="Run parameter `dt`.", values=("a", "b")),')
|
|
129
|
+
root = make_repo({"jobs/a.py": script}, BILINGUAL)
|
|
130
|
+
self.assertEqual(run(root, "generate")[0], 0)
|
|
131
|
+
page = (root / "docs/catalog/dwd.customer_daily.md").read_text(encoding="utf-8")
|
|
132
|
+
self.assertIn("| Unit | Values |", page)
|
|
133
|
+
self.assertIn("| `a`, `b` |", page)
|
|
134
|
+
twice = script.replace('values=("a", "b")', 'values=("a", "a")')
|
|
135
|
+
root = make_repo({"jobs/a.py": twice}, BILINGUAL)
|
|
136
|
+
self.assertIn("values lists a value twice", run(root, "generate")[2])
|
|
137
|
+
sql = SQL_SCRIPT.replace("-- source: dim_merchant.merchant_id", "-- source: dim_merchant.merchant_id\n-- values: m1;m2")
|
|
138
|
+
root = make_repo({"sql/m.sql": sql}, BILINGUAL)
|
|
139
|
+
self.assertEqual(run(root, "generate")[0], 0)
|
|
140
|
+
self.assertIn("`m1`, `m2`", (root / "docs/catalog/ads.merchant_monthly.md").read_text(encoding="utf-8"))
|
|
141
|
+
|
|
127
142
|
def test_check_passes_when_current_and_fails_after_a_definition_changes(self):
|
|
128
143
|
root = make_repo({"jobs/dwd/customer_daily.py": PY_SCRIPT}, BILINGUAL)
|
|
129
144
|
self.assertEqual(run(root, "generate")[0], 0)
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: data-quality
|
|
3
|
+
description: "Check a published warehouse table's data against its data-catalog definitions with read-only SELECTs: columns never filled or stuck at one value, counts below zero, ratios outside 0–1, declared values that never appear and undeclared values that do. Then classify each finding as a wrong definition, wrong code, or wrong data. Use when asked whether a table's data is right, to profile or sanity-check a dwd/ads table, to find dead or constant columns, to check a column's values against its documented list, after a pipeline change ships, or when a data-catalog definition and the data might disagree. NOT for validating a Spark change before it ships (pyspark canary and reconciliation), writing definitions (data-catalog), or monitoring dashboards."
|
|
4
|
+
release-version: 0.39.0
|
|
5
|
+
---
|
|
6
|
+
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
|
+
# Data quality
|
|
8
|
+
|
|
9
|
+
Ask the data whether it means what its definitions say. The definitions come from the
|
|
10
|
+
data-catalog skill; this skill turns them into read-only queries, runs them, and
|
|
11
|
+
reports where table, code, and documentation disagree.
|
|
12
|
+
|
|
13
|
+
## Find your task
|
|
14
|
+
|
|
15
|
+
| Task | Do |
|
|
16
|
+
|---|---|
|
|
17
|
+
| The repo has no `data-catalog.json` or the table has no definitions | Set the repo up and document the table with the data-catalog skill first |
|
|
18
|
+
| Check one or more tables | [Run the checks](#run-the-checks) |
|
|
19
|
+
| Decide what a finding means | [Classify each finding](#classify-each-finding) |
|
|
20
|
+
| What each check does, and how to make a column checkable | [references/checks.md](references/checks.md) |
|
|
21
|
+
| Connect to Presto, Trino, Spark SQL, or another engine | [references/checks.md#runners](references/checks.md#runners) |
|
|
22
|
+
|
|
23
|
+
`<engine>` is `<skill-base-dir>/scripts/data_quality.py`. It reads definitions through
|
|
24
|
+
the repo's own `scripts/data-catalog/data_catalog.py`.
|
|
25
|
+
|
|
26
|
+
## Rules
|
|
27
|
+
|
|
28
|
+
1. **Read only.** Every query is one SELECT built by the engine; the SQLite runner
|
|
29
|
+
opens the file read-only. Never add a write to a runner, and never run the checks
|
|
30
|
+
with credentials that can write when read-only ones exist.
|
|
31
|
+
2. **Bound the scan.** Pass `--where` on the partition column for a recent range, and
|
|
32
|
+
widen only when a finding needs it.
|
|
33
|
+
Example: `--where "dt >= DATE '2026-10-01'"` scanned 333K rows in 18 seconds.
|
|
34
|
+
3. **Aggregates only.** The checks return counts, distinct counts, minimums, maximums,
|
|
35
|
+
and counts per declared value. Never add a query that returns rows, and never print
|
|
36
|
+
a column's actual value when it could identify a customer.
|
|
37
|
+
4. **A finding is a lead until you classify it.** Read the code that writes the column
|
|
38
|
+
before saying the data, the code, or the definition is wrong.
|
|
39
|
+
Example: `exchange_share` reached 25 while its unit says ratio: read the formula
|
|
40
|
+
before deciding whether the unit or the code is wrong.
|
|
41
|
+
5. **Report; never fix in the same step.** List each finding with its classification
|
|
42
|
+
and evidence. Fixing a definition is a data-catalog change; fixing code or data is
|
|
43
|
+
the owner's call. File an issue only when the user asks.
|
|
44
|
+
6. **Make columns checkable at the source.** A finding that needs a fact the catalog
|
|
45
|
+
lacks (allowed values, a unit) is fixed by adding it to the `Field`, not by
|
|
46
|
+
special-casing the check.
|
|
47
|
+
Example: add `values=("是", "否", "无法判定")` to `agent_risk_status`, then rerun.
|
|
48
|
+
|
|
49
|
+
## Run the checks
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
python3 <engine> run --table usr.dwd_orders --where "dt >= DATE '2026-10-01'" \
|
|
53
|
+
--key dt --key order_id --runner "python3 tools/presto_query.py" # or --sqlite warehouse.db
|
|
54
|
+
python3 <engine> plan --table usr.dwd_orders --where "..." # just the SQL, to run elsewhere
|
|
55
|
+
python3 <engine> judge --results results.json # judge rows you ran yourself
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
- Run from the repo root, or pass `--root`.
|
|
59
|
+
- Pass the table's grain as `--key` (one per column): a join that repeats rows breaks
|
|
60
|
+
it, and nothing else catches that.
|
|
61
|
+
- One table per run when tables use different partition columns.
|
|
62
|
+
- Engine errors such as "exceeds the allowed maximum stages": lower `--batch`.
|
|
63
|
+
- A column type the engine cannot count distinct values of (maps, some arrays): `--skip`.
|
|
64
|
+
- Exit 0: no problems. Exit 1: problems found. Exit 2: setup or usage error.
|
|
65
|
+
|
|
66
|
+
## Classify each finding
|
|
67
|
+
|
|
68
|
+
For each problem, read the column's definition and the code that writes it, then give
|
|
69
|
+
one of:
|
|
70
|
+
|
|
71
|
+
| Class | Means | Example | Next step |
|
|
72
|
+
|---|---|---|---|
|
|
73
|
+
| definition | the code and data agree; the definition is wrong or incomplete | a retention ratio documented as 0–1 that is by design unbounded | fix the `Field` (data-catalog) |
|
|
74
|
+
| code | the code does not compute what the definition says | `transaction_result` hard-coded to `SUCCESS` | report to the owner with the code location |
|
|
75
|
+
| data | the code is right; the source data is wrong or missing | a source that never fills a certificate type | report to the source owner |
|
|
76
|
+
| expected | correct as is, and the definition should say so | a payout-only column that is 0 in a collections-only period | add the reason to the definition so the next run marks it documented |
|
|
77
|
+
|
|
78
|
+
A column documented as always empty whose `source` names a real upstream column is
|
|
79
|
+
still a question for the owner: is the placeholder meant to stay, or is the code
|
|
80
|
+
missing the read?
|
|
81
|
+
|
|
82
|
+
Findings marked `(documented)` already match what the definition says (for example
|
|
83
|
+
"always empty"), version and hash columns, and missing-flags that are 0 on every row:
|
|
84
|
+
mention them, do not chase them.
|
|
85
|
+
|
|
86
|
+
## Report
|
|
87
|
+
|
|
88
|
+
Give, per table: the range checked, the row count, then each problem as
|
|
89
|
+
`column — finding — class — evidence (file:line or query)`, most serious first.
|
|
90
|
+
End with what you could not classify and why. Never claim a table is clean when a
|
|
91
|
+
check was `not-run` or a column was skipped.
|
|
92
|
+
|
|
93
|
+
## Done when
|
|
94
|
+
|
|
95
|
+
- Every problem finding has a class and evidence, or is listed as unclassified.
|
|
96
|
+
- Nothing was written to the warehouse, and no row-level value was printed.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# Checks, findings, and runners
|
|
2
|
+
|
|
3
|
+
## What each check reads
|
|
4
|
+
|
|
5
|
+
The engine builds up to three kinds of query per documented table:
|
|
6
|
+
|
|
7
|
+
- **Profile** (one per 40 columns, `--batch`): `COUNT(*)`, and per column `COUNT(col)`
|
|
8
|
+
and `COUNT(DISTINCT col)`; `MIN`/`MAX` for columns whose `unit` is numeric (count,
|
|
9
|
+
ratio, percent, a currency, seconds, days, score) or whose name looks like a flag
|
|
10
|
+
(`*_missing`, `*_flag`, `is_*`, `has_*`).
|
|
11
|
+
- **Key** (with `--key`): counts the grain values that repeat, with a GROUP BY …
|
|
12
|
+
HAVING COUNT(*) > 1 subquery.
|
|
13
|
+
- **Values** (one per column with `values=(...)`): `SELECT col AS value, COUNT(*) AS n …
|
|
14
|
+
GROUP BY col`. Only declare `values` on enumerated columns, never on identifiers.
|
|
15
|
+
|
|
16
|
+
## Findings
|
|
17
|
+
|
|
18
|
+
| Finding | Fires when | Marked documented when |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| `empty-table` | no rows in the `--where` range; nothing else is judged | — |
|
|
21
|
+
| `never-filled` | the column is empty in every row | the definition says always empty / 始终为空 |
|
|
22
|
+
| `mostly-empty` | filled in under 1% of rows | — |
|
|
23
|
+
| `constant` | one distinct value in every filled row, with at least 30 rows (`--constant-min-rows`) and the column not named in `--where` | the definition says always/constant/恒为; the name is a version, hash, or source table; a flag column is constant at 0 or N |
|
|
24
|
+
| `negative-value` | minimum below 0 for unit count, seconds, or days | — |
|
|
25
|
+
| `out-of-range` | values outside 0–1 for unit `ratio`, or 0–100 for unit `percent` | — |
|
|
26
|
+
| `unseen-value` | a declared value never appears (and the column has rows) | — |
|
|
27
|
+
| `undeclared-value` | a value appears that `values` does not list (empty values are not counted) | — |
|
|
28
|
+
| `duplicate-key` | a `--key` combination appears in more than one row | — |
|
|
29
|
+
| `not-run` | a query has no result | — |
|
|
30
|
+
|
|
31
|
+
## Making a column checkable
|
|
32
|
+
|
|
33
|
+
Each check needs a fact in the column's data-catalog `Field`:
|
|
34
|
+
|
|
35
|
+
| To check | Write in the `Field` |
|
|
36
|
+
|---|---|
|
|
37
|
+
| allowed values | `values=("是", "否", "无法判定")`, or `-- values: 是; 否; 无法判定` in SQL |
|
|
38
|
+
| 0–1 range | `unit="ratio"`; use `"percent"` for 0–100; leave a ratio unbounded by design without that unit |
|
|
39
|
+
| never negative | `unit="count"` (or seconds, days) |
|
|
40
|
+
| expected empty or constant | say so in the text: "Always empty until the source fills it." |
|
|
41
|
+
|
|
42
|
+
## Runners
|
|
43
|
+
|
|
44
|
+
`run --runner CMD` sends one SELECT on stdin per query and reads the rows from stdout:
|
|
45
|
+
a JSON array of objects, JSON lines (one object per line), or CSV with a header row.
|
|
46
|
+
Exit non-zero with a message on stderr when a query fails.
|
|
47
|
+
|
|
48
|
+
| Engine | Runner command shape |
|
|
49
|
+
|---|---|
|
|
50
|
+
| Trino / Presto CLI | `trino --server <url> --catalog hive --output-format JSON` (reads stdin) |
|
|
51
|
+
| Presto REST | a small script that POSTs to `/v1/statement`, follows `nextUri`, prints a JSON array |
|
|
52
|
+
| Spark SQL | a script that runs `spark.sql(stdin).toPandas().to_json(orient="records")` |
|
|
53
|
+
| SQLite (local, tests) | built in: `--sqlite <file>`, opened read-only |
|
|
54
|
+
|
|
55
|
+
Use `--dialect spark` with Spark SQL (backtick identifiers); the default quotes with
|
|
56
|
+
double quotes. Use read-only credentials where the engine offers them.
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import { spawnSync } from 'node:child_process';
|
|
3
|
+
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
|
4
|
+
import { homedir } from 'node:os';
|
|
5
|
+
import { dirname, join } from 'node:path';
|
|
6
|
+
import { fileURLToPath } from 'node:url';
|
|
7
|
+
|
|
8
|
+
const DAY_MS = 24 * 60 * 60 * 1000;
|
|
9
|
+
const WEEK_MS = 7 * DAY_MS;
|
|
10
|
+
const here = dirname(fileURLToPath(import.meta.url));
|
|
11
|
+
|
|
12
|
+
function readJson(path, fallback = {}) {
|
|
13
|
+
try { return JSON.parse(readFileSync(path, 'utf8')); } catch { return fallback; }
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
function frontmatterValue(text, key) {
|
|
17
|
+
const frontmatter = /^---\n([\s\S]*?)\n---/.exec(text)?.[1] ?? '';
|
|
18
|
+
const match = new RegExp('^' + key + ':\\s*([^\\n]+)$', 'm').exec(frontmatter);
|
|
19
|
+
return match?.[1].trim().replace(/^['"]|['"]$/g, '') ?? null;
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
function repositoryOf(source) {
|
|
23
|
+
const value = String(source ?? '').trim();
|
|
24
|
+
const match = /^(?:git@github\.com:|https:\/\/github\.com\/)?([A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+?)(?:\.git)?$/.exec(value);
|
|
25
|
+
return match?.[1] ?? null;
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
function stableVersion(value) {
|
|
29
|
+
const match = /^v?(\d+)\.(\d+)\.(\d+)$/.exec(String(value ?? '').trim());
|
|
30
|
+
return match ? match.slice(1).map(Number) : null;
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
function newer(left, right) {
|
|
34
|
+
for (let index = 0; index < 3; index += 1) {
|
|
35
|
+
if (left[index] !== right[index]) return left[index] > right[index];
|
|
36
|
+
}
|
|
37
|
+
return false;
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
function writeCache(path, value) {
|
|
41
|
+
try {
|
|
42
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
43
|
+
writeFileSync(path, JSON.stringify(value));
|
|
44
|
+
} catch { /* A read-only cache must not block skill use. */ }
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
function main(now = Date.now()) {
|
|
48
|
+
if (process.env.ONEABOVEIO_NO_UPDATE_CHECK) return;
|
|
49
|
+
const skill = readFileSync(join(here, 'SKILL.md'), 'utf8');
|
|
50
|
+
const name = frontmatterValue(skill, 'name');
|
|
51
|
+
const localVersion = stableVersion(frontmatterValue(skill, 'release-version'));
|
|
52
|
+
if (!name || !localVersion) return;
|
|
53
|
+
|
|
54
|
+
const lock = readJson(join(process.env.HOME || homedir(), '.agents', '.skill-lock.json'));
|
|
55
|
+
const entry = lock.skills?.[name];
|
|
56
|
+
const repository = repositoryOf(entry?.sourceUrl || entry?.source);
|
|
57
|
+
if (!repository) return;
|
|
58
|
+
|
|
59
|
+
const cacheRoot = process.env.XDG_CACHE_HOME || join(process.env.HOME || homedir(), '.cache');
|
|
60
|
+
const cachePath = join(cacheRoot, '1aboveio-skills', 'release-check.json');
|
|
61
|
+
const all = readJson(cachePath);
|
|
62
|
+
const cache = all[repository] && typeof all[repository] === 'object' ? all[repository] : {};
|
|
63
|
+
|
|
64
|
+
if (!cache.lastCheck || now - cache.lastCheck >= DAY_MS) {
|
|
65
|
+
const result = spawnSync('gh', ['api', 'repos/' + repository + '/releases/latest', '--jq', '.tag_name'], {
|
|
66
|
+
encoding: 'utf8',
|
|
67
|
+
stdio: ['ignore', 'pipe', 'ignore'],
|
|
68
|
+
timeout: 1500,
|
|
69
|
+
});
|
|
70
|
+
cache.lastCheck = now;
|
|
71
|
+
if (result.status === 0 && stableVersion(result.stdout)) cache.latestTag = result.stdout.trim();
|
|
72
|
+
else delete cache.latestTag;
|
|
73
|
+
all[repository] = cache;
|
|
74
|
+
writeCache(cachePath, all);
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
const latestVersion = stableVersion(cache.latestTag);
|
|
78
|
+
if (!latestVersion || !newer(latestVersion, localVersion)) return;
|
|
79
|
+
if (cache.notifiedTag === cache.latestTag && cache.notifiedAt && now - cache.notifiedAt < WEEK_MS) return;
|
|
80
|
+
|
|
81
|
+
cache.notifiedTag = cache.latestTag;
|
|
82
|
+
cache.notifiedAt = now;
|
|
83
|
+
all[repository] = cache;
|
|
84
|
+
writeCache(cachePath, all);
|
|
85
|
+
const command = `npx skills@1.5.23 add '${repository}#${cache.latestTag}' --global --agent universal claude-code --skill ${name} --yes`;
|
|
86
|
+
process.stdout.write(
|
|
87
|
+
`UPDATE_AVAILABLE: A newer 1AboveIO skills release (${cache.latestTag}) is available; this skill is `
|
|
88
|
+
+ `v${localVersion.join('.')}. Ask once whether to refresh ${name}. On approval run \`${command}\`. `
|
|
89
|
+
+ 'The refreshed instructions apply next session. Either way, continue the current task and do not ask again this session.\n',
|
|
90
|
+
);
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
try { main(); } catch { /* Update checks are advisory and always fail silent. */ }
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import assert from 'node:assert/strict';
|
|
2
|
+
import { spawnSync } from 'node:child_process';
|
|
3
|
+
import { dirname, join } from 'node:path';
|
|
4
|
+
import { fileURLToPath } from 'node:url';
|
|
5
|
+
import test from 'node:test';
|
|
6
|
+
|
|
7
|
+
const scriptsDir = dirname(fileURLToPath(import.meta.url));
|
|
8
|
+
|
|
9
|
+
test('data-quality test_data_quality.py', () => {
|
|
10
|
+
const result = spawnSync('python3', [join(scriptsDir, 'test_data_quality.py')], { encoding: 'utf8' });
|
|
11
|
+
assert.equal(result.status, 0, `${result.stdout}\n${result.stderr}`);
|
|
12
|
+
});
|
|
@@ -0,0 +1,405 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Data quality engine: check a published table's data against its data-catalog definitions.
|
|
3
|
+
|
|
4
|
+
Commands:
|
|
5
|
+
plan write the read-only SELECT checks for documented tables (JSON)
|
|
6
|
+
judge turn query results into findings
|
|
7
|
+
run plan, execute through a runner, and judge in one step
|
|
8
|
+
version print the engine identity
|
|
9
|
+
|
|
10
|
+
Definitions come from the repo's data-catalog engine (scripts/data-catalog/data_catalog.py),
|
|
11
|
+
so a table is checkable once its script carries CATALOG_TABLE / CATALOG_FIELDS or a
|
|
12
|
+
`-- @field` block. Every query is a single SELECT; nothing is ever written.
|
|
13
|
+
|
|
14
|
+
Runner contract (run --runner CMD): CMD reads one SQL statement on stdin and prints the
|
|
15
|
+
result rows on stdout as a JSON array of objects, or as CSV with a header row.
|
|
16
|
+
|
|
17
|
+
Standard library only.
|
|
18
|
+
"""
|
|
19
|
+
from __future__ import annotations
|
|
20
|
+
|
|
21
|
+
import argparse
|
|
22
|
+
import csv
|
|
23
|
+
import hashlib
|
|
24
|
+
import importlib.util
|
|
25
|
+
import io
|
|
26
|
+
import json
|
|
27
|
+
import re
|
|
28
|
+
import shlex
|
|
29
|
+
import sqlite3
|
|
30
|
+
import subprocess
|
|
31
|
+
import sys
|
|
32
|
+
from dataclasses import dataclass, field
|
|
33
|
+
from pathlib import Path
|
|
34
|
+
|
|
35
|
+
ENGINE_VERSION = "1.0.0"
|
|
36
|
+
CATALOG_ENGINE = "scripts/data-catalog/data_catalog.py"
|
|
37
|
+
NUMERIC_UNITS = {"count", "usd", "cny", "eur", "gbp", "hkd", "amount", "ratio", "percent", "seconds", "days", "score"}
|
|
38
|
+
NON_NEGATIVE_UNITS = {"count", "seconds", "days"}
|
|
39
|
+
RANGES = {"ratio": (0.0, 1.0), "percent": (0.0, 100.0)}
|
|
40
|
+
DOCUMENTED_CONSTANT_RE = re.compile(r"始终|恒为|目前.{0,8}(为空|不填)|always|constant|never (filled|set)|not filled", re.I)
|
|
41
|
+
PROVENANCE_RE = re.compile(r"(^|_)(version|sha\d*|hash|checksum|source_table|compute_time|etl_time)$", re.I)
|
|
42
|
+
FLAG_RE = re.compile(r"(_missing|_flag)$|^(is|has)_", re.I) # healthy when constant at 0 / N
|
|
43
|
+
CLEAR_FLAG = {"0", "0.0", "false", "n", "no"}
|
|
44
|
+
PROFILE_BATCH = 40 # columns per profile query: each COUNT(DISTINCT) is a stage, and engines cap stages
|
|
45
|
+
MOSTLY_EMPTY = 0.01
|
|
46
|
+
CONSTANT_MIN_ROWS = 30 # below this, one distinct value says little
|
|
47
|
+
NULL_KEY = "<null>"
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
class QualityError(Exception):
|
|
51
|
+
"""A usage or setup problem; exit code 2."""
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
# ---------------------------------------------------------------------------
|
|
55
|
+
# Definitions, through the repo's data-catalog engine
|
|
56
|
+
# ---------------------------------------------------------------------------
|
|
57
|
+
|
|
58
|
+
def load_catalog_engine(root: Path, path: str | None):
|
|
59
|
+
engine = Path(path) if path else root / CATALOG_ENGINE
|
|
60
|
+
if not engine.is_file():
|
|
61
|
+
raise QualityError(f"no data-catalog engine at {engine}; set the repo up with the data-catalog skill first "
|
|
62
|
+
"(its init copies the engine), or pass --catalog-engine")
|
|
63
|
+
spec = importlib.util.spec_from_file_location("data_catalog_for_quality", engine)
|
|
64
|
+
module = importlib.util.module_from_spec(spec)
|
|
65
|
+
sys.modules[spec.name] = module # dataclasses resolve string annotations through sys.modules
|
|
66
|
+
spec.loader.exec_module(module)
|
|
67
|
+
return module
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
def documented_tables(root: Path, engine_path: str | None, wanted: list) -> list:
|
|
71
|
+
dc = load_catalog_engine(root, engine_path)
|
|
72
|
+
try:
|
|
73
|
+
config = dc.load_config(root)
|
|
74
|
+
tables, errors, _ = dc.collect(root, config)
|
|
75
|
+
except dc.CatalogError as exc:
|
|
76
|
+
raise QualityError(f"data-catalog: {exc}") from exc
|
|
77
|
+
if errors:
|
|
78
|
+
raise QualityError("data-catalog metadata has errors; run its generate first:\n " + "\n ".join(errors))
|
|
79
|
+
if wanted:
|
|
80
|
+
missing = [w for w in wanted if w not in {t.name for t in tables}]
|
|
81
|
+
if missing:
|
|
82
|
+
raise QualityError(f"not documented in the catalog: {', '.join(missing)}")
|
|
83
|
+
tables = [t for t in tables if t.name in wanted]
|
|
84
|
+
if not tables:
|
|
85
|
+
raise QualityError("no documented tables to check")
|
|
86
|
+
return tables
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
# ---------------------------------------------------------------------------
|
|
90
|
+
# Plan: one profile query per table, one value query per enumerated column
|
|
91
|
+
# ---------------------------------------------------------------------------
|
|
92
|
+
|
|
93
|
+
def quote(name: str, dialect: str) -> str:
|
|
94
|
+
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name):
|
|
95
|
+
raise QualityError(f"column name {name!r} cannot be quoted safely")
|
|
96
|
+
return f"`{name}`" if dialect == "spark" else f'"{name}"'
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
def table_ref(name: str) -> str:
|
|
100
|
+
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*){0,2}", name):
|
|
101
|
+
raise QualityError(f"table name {name!r} is not a plain identifier")
|
|
102
|
+
return name
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
def allowed(f) -> tuple:
|
|
106
|
+
"""Declared values; empty for engine copies older than `Field.values`."""
|
|
107
|
+
return tuple(getattr(f, "values", ()) or ())
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
def is_numeric(f) -> bool:
|
|
111
|
+
return (f.unit or "").strip().lower() in NUMERIC_UNITS
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def plan_queries(tables: list, where: str | None, dialect: str, skip: set, batch: int = PROFILE_BATCH,
|
|
115
|
+
key: list | None = None) -> list:
|
|
116
|
+
cond = f" WHERE {where}" if where else ""
|
|
117
|
+
queries = []
|
|
118
|
+
for t in tables:
|
|
119
|
+
if key:
|
|
120
|
+
unknown = [k for k in key if k not in {f.name for f in t.fields}]
|
|
121
|
+
if unknown:
|
|
122
|
+
raise QualityError(f"--key column(s) not documented in {t.name}: {', '.join(unknown)}")
|
|
123
|
+
cols = ", ".join(quote(k, dialect) for k in key)
|
|
124
|
+
sql = (f"SELECT COUNT(*) AS dup_keys, COALESCE(SUM(c), 0) AS dup_rows FROM "
|
|
125
|
+
f"(SELECT COUNT(*) AS c FROM {table_ref(t.name)}{cond} GROUP BY {cols} HAVING COUNT(*) > 1) dup")
|
|
126
|
+
queries.append({"id": f"key:{t.name}", "table": t.name, "kind": "key", "columns": list(key), "sql": sql})
|
|
127
|
+
cols = [f for f in t.fields if f.name not in skip]
|
|
128
|
+
chunks = [cols[i:i + batch] for i in range(0, len(cols), batch)] or [[]]
|
|
129
|
+
for k, chunk in enumerate(chunks):
|
|
130
|
+
parts = ["COUNT(*) AS n"]
|
|
131
|
+
for i, f in enumerate(chunk):
|
|
132
|
+
q = quote(f.name, dialect)
|
|
133
|
+
parts += [f"COUNT({q}) AS nn_{i}", f"COUNT(DISTINCT {q}) AS d_{i}"]
|
|
134
|
+
if is_numeric(f) or FLAG_RE.search(f.name):
|
|
135
|
+
parts += [f"MIN({q}) AS mn_{i}", f"MAX({q}) AS mx_{i}"]
|
|
136
|
+
sql = f"SELECT {', '.join(parts)} FROM {table_ref(t.name)}{cond}"
|
|
137
|
+
suffix = f"#{k + 1}" if len(chunks) > 1 else ""
|
|
138
|
+
queries.append({"id": f"profile:{t.name}{suffix}", "table": t.name, "kind": "profile", "part": k,
|
|
139
|
+
"columns": [f.name for f in chunk], "sql": sql})
|
|
140
|
+
for f in cols:
|
|
141
|
+
if allowed(f):
|
|
142
|
+
q = quote(f.name, dialect)
|
|
143
|
+
sql = f"SELECT {q} AS value, COUNT(*) AS n FROM {table_ref(t.name)}{cond} GROUP BY {q}"
|
|
144
|
+
queries.append({"id": f"values:{t.name}.{f.name}", "table": t.name, "kind": "values",
|
|
145
|
+
"column": f.name, "sql": sql})
|
|
146
|
+
for q in queries:
|
|
147
|
+
assert_read_only(q["sql"])
|
|
148
|
+
return queries
|
|
149
|
+
|
|
150
|
+
|
|
151
|
+
def assert_read_only(sql: str) -> None:
|
|
152
|
+
if not re.match(r"\s*SELECT\b", sql, re.I) or ";" in sql:
|
|
153
|
+
raise QualityError(f"refusing to run a statement that is not a single SELECT: {sql[:80]}")
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
# ---------------------------------------------------------------------------
|
|
157
|
+
# Judge: results -> findings
|
|
158
|
+
# ---------------------------------------------------------------------------
|
|
159
|
+
|
|
160
|
+
@dataclass
|
|
161
|
+
class Finding:
|
|
162
|
+
table: str
|
|
163
|
+
column: str | None
|
|
164
|
+
kind: str
|
|
165
|
+
message: str
|
|
166
|
+
documented: bool = False # the definition already says so: report as info, not a problem
|
|
167
|
+
|
|
168
|
+
|
|
169
|
+
def number(value):
|
|
170
|
+
if value is None or value == "":
|
|
171
|
+
return None
|
|
172
|
+
try:
|
|
173
|
+
return float(value)
|
|
174
|
+
except (TypeError, ValueError):
|
|
175
|
+
return None
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
def where_columns(where: str | None) -> set:
|
|
179
|
+
"""Identifiers a --where condition names: those columns are constant by construction."""
|
|
180
|
+
if not where:
|
|
181
|
+
return set()
|
|
182
|
+
bare = re.sub(r"'(?:[^']|'')*'", " ", where)
|
|
183
|
+
return {w.strip('`"').lower() for w in re.findall(r'[`"]?[A-Za-z_][A-Za-z0-9_]*[`"]?', bare)}
|
|
184
|
+
|
|
185
|
+
|
|
186
|
+
def judge(tables: list, queries: list, results: dict, where: str | None = None,
|
|
187
|
+
constant_min_rows: int = CONSTANT_MIN_ROWS) -> list:
|
|
188
|
+
by_table = {t.name: t for t in tables}
|
|
189
|
+
pinned = where_columns(where)
|
|
190
|
+
findings = []
|
|
191
|
+
for q in queries:
|
|
192
|
+
rows = results.get(q["id"])
|
|
193
|
+
if rows is None:
|
|
194
|
+
findings.append(Finding(q["table"], q.get("column"), "not-run", f"no result for {q['id']}"))
|
|
195
|
+
continue
|
|
196
|
+
t = by_table[q["table"]]
|
|
197
|
+
if q["kind"] == "key":
|
|
198
|
+
row = {k.lower(): v for k, v in (rows[0] if rows else {}).items()}
|
|
199
|
+
keys = number(row.get("dup_keys")) or 0
|
|
200
|
+
if keys:
|
|
201
|
+
findings.append(Finding(t.name, None, "duplicate-key",
|
|
202
|
+
f"{int(keys)} ({', '.join(q['columns'])}) value(s) repeat across "
|
|
203
|
+
f"{int(number(row.get('dup_rows')) or 0)} rows; the grain allows one each"))
|
|
204
|
+
continue
|
|
205
|
+
if q["kind"] == "profile":
|
|
206
|
+
part = judge_profile(t, q["columns"], rows, pinned, constant_min_rows)
|
|
207
|
+
if q.get("part", 0) > 0: # every batch sees the same row count; report an empty range once
|
|
208
|
+
part = [f for f in part if f.kind != "empty-table"]
|
|
209
|
+
findings += part
|
|
210
|
+
else:
|
|
211
|
+
findings += judge_values(t, q["column"], rows)
|
|
212
|
+
return findings
|
|
213
|
+
|
|
214
|
+
|
|
215
|
+
def judge_profile(t, columns: list, rows: list, pinned: set, constant_min_rows: int) -> list:
|
|
216
|
+
if len(rows) != 1:
|
|
217
|
+
return [Finding(t.name, None, "not-run", f"profile returned {len(rows)} rows, expected 1")]
|
|
218
|
+
row = {k.lower(): v for k, v in rows[0].items()}
|
|
219
|
+
n = number(row.get("n")) or 0
|
|
220
|
+
if n == 0:
|
|
221
|
+
return [Finding(t.name, None, "empty-table", "no rows in the checked range; nothing else can be judged")]
|
|
222
|
+
fields = {f.name: f for f in t.fields}
|
|
223
|
+
out = []
|
|
224
|
+
for i, name in enumerate(columns):
|
|
225
|
+
f = fields[name]
|
|
226
|
+
text = " ".join([*f.text.values(), f.source])
|
|
227
|
+
documented = bool(DOCUMENTED_CONSTANT_RE.search(text))
|
|
228
|
+
filled = number(row.get(f"nn_{i}")) or 0
|
|
229
|
+
distinct = number(row.get(f"d_{i}")) or 0
|
|
230
|
+
lo, hi = number(row.get(f"mn_{i}")), number(row.get(f"mx_{i}"))
|
|
231
|
+
if filled == 0:
|
|
232
|
+
out.append(Finding(t.name, name, "never-filled", f"empty in all {int(n)} rows", documented))
|
|
233
|
+
continue
|
|
234
|
+
if filled / n < MOSTLY_EMPTY:
|
|
235
|
+
out.append(Finding(t.name, name, "mostly-empty", f"filled in {int(filled)} of {int(n)} rows"))
|
|
236
|
+
if distinct == 1 and n >= constant_min_rows and name.lower() not in pinned:
|
|
237
|
+
value = row.get(f"mn_{i}")
|
|
238
|
+
shown = f" ({value})" if value is not None else ""
|
|
239
|
+
clear_flag = FLAG_RE.search(name) and str(row.get(f"mn_{i}")).strip().lower() in CLEAR_FLAG
|
|
240
|
+
# a build's version is the same on every row; a missing-flag at 0 means nothing was missing
|
|
241
|
+
expected = documented or bool(PROVENANCE_RE.search(name)) or bool(clear_flag)
|
|
242
|
+
out.append(Finding(t.name, name, "constant", f"one value{shown} in all {int(filled)} filled rows", expected))
|
|
243
|
+
unit = (f.unit or "").strip().lower()
|
|
244
|
+
if unit in NON_NEGATIVE_UNITS and lo is not None and lo < 0:
|
|
245
|
+
out.append(Finding(t.name, name, "negative-value", f"minimum {lo:g}, but the unit is {f.unit}"))
|
|
246
|
+
if unit in RANGES and lo is not None and hi is not None:
|
|
247
|
+
low, high = RANGES[unit]
|
|
248
|
+
if lo < low or hi > high:
|
|
249
|
+
out.append(Finding(t.name, name, "out-of-range",
|
|
250
|
+
f"values from {lo:g} to {hi:g}, outside {low:g}–{high:g} for unit {f.unit}"))
|
|
251
|
+
return out
|
|
252
|
+
|
|
253
|
+
|
|
254
|
+
def judge_values(t, column: str, rows: list) -> list:
|
|
255
|
+
f = next(f for f in t.fields if f.name == column)
|
|
256
|
+
seen = {}
|
|
257
|
+
for r in rows:
|
|
258
|
+
r = {k.lower(): v for k, v in r.items()}
|
|
259
|
+
value = r.get("value")
|
|
260
|
+
empty = value is None or (value == "" and "" not in allowed(f))
|
|
261
|
+
key = NULL_KEY if empty else str(value)
|
|
262
|
+
seen[key] = seen.get(key, 0) + (number(r.get("n")) or 0)
|
|
263
|
+
declared = set(allowed(f))
|
|
264
|
+
out = []
|
|
265
|
+
unseen = [v for v in allowed(f) if v not in seen]
|
|
266
|
+
if unseen and seen:
|
|
267
|
+
out.append(Finding(t.name, column, "unseen-value", f"declared but never present: {', '.join(unseen)}"))
|
|
268
|
+
extra = sorted(k for k in seen if k not in declared and k != NULL_KEY)
|
|
269
|
+
if extra:
|
|
270
|
+
counts = ", ".join(f"{k} ({int(seen[k])})" for k in extra[:10])
|
|
271
|
+
out.append(Finding(t.name, column, "undeclared-value", f"present but not declared: {counts}"))
|
|
272
|
+
return out
|
|
273
|
+
|
|
274
|
+
|
|
275
|
+
# ---------------------------------------------------------------------------
|
|
276
|
+
# Runners
|
|
277
|
+
# ---------------------------------------------------------------------------
|
|
278
|
+
|
|
279
|
+
def parse_rows(text: str) -> list:
|
|
280
|
+
text = text.strip()
|
|
281
|
+
if not text:
|
|
282
|
+
return []
|
|
283
|
+
if text[0] == "[":
|
|
284
|
+
return json.loads(text)
|
|
285
|
+
if text[0] == "{": # one object, or JSON lines (trino/presto --output-format JSON)
|
|
286
|
+
return [json.loads(line) for line in text.splitlines() if line.strip()]
|
|
287
|
+
return [dict(r) for r in csv.DictReader(io.StringIO(text))]
|
|
288
|
+
|
|
289
|
+
|
|
290
|
+
def run_command(cmd: str, sql: str) -> list:
|
|
291
|
+
proc = subprocess.run(shlex.split(cmd), input=sql, capture_output=True, text=True)
|
|
292
|
+
if proc.returncode != 0:
|
|
293
|
+
raise QualityError(f"runner failed ({proc.returncode}) on: {sql[:80]}\n{proc.stderr.strip()[-400:]}")
|
|
294
|
+
return parse_rows(proc.stdout)
|
|
295
|
+
|
|
296
|
+
|
|
297
|
+
def run_sqlite(db: str, sql: str) -> list:
|
|
298
|
+
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True) # read-only by construction
|
|
299
|
+
try:
|
|
300
|
+
cur = con.execute(sql)
|
|
301
|
+
names = [d[0] for d in cur.description]
|
|
302
|
+
return [dict(zip(names, row)) for row in cur.fetchall()]
|
|
303
|
+
finally:
|
|
304
|
+
con.close()
|
|
305
|
+
|
|
306
|
+
|
|
307
|
+
def execute(queries: list, runner: str | None, sqlite_db: str | None) -> dict:
|
|
308
|
+
results = {}
|
|
309
|
+
for q in queries:
|
|
310
|
+
assert_read_only(q["sql"])
|
|
311
|
+
results[q["id"]] = run_sqlite(sqlite_db, q["sql"]) if sqlite_db else run_command(runner, q["sql"])
|
|
312
|
+
return results
|
|
313
|
+
|
|
314
|
+
|
|
315
|
+
# ---------------------------------------------------------------------------
|
|
316
|
+
# Report and CLI
|
|
317
|
+
# ---------------------------------------------------------------------------
|
|
318
|
+
|
|
319
|
+
def report(findings: list, tables: list, where: str | None, as_json: bool) -> int:
|
|
320
|
+
problems = [f for f in findings if not f.documented]
|
|
321
|
+
if as_json:
|
|
322
|
+
print(json.dumps({"engine": ENGINE_VERSION, "where": where, "tables": [t.name for t in tables],
|
|
323
|
+
"problems": len(problems),
|
|
324
|
+
"findings": [vars(f) for f in findings]}, ensure_ascii=False, indent=2))
|
|
325
|
+
return 1 if problems else 0
|
|
326
|
+
scope = f" where {where}" if where else " (all rows)"
|
|
327
|
+
print(f"data-quality: {len(tables)} table(s){scope}; {len(problems)} problem(s), "
|
|
328
|
+
f"{len(findings) - len(problems)} already documented")
|
|
329
|
+
for t in tables:
|
|
330
|
+
mine = [f for f in findings if f.table == t.name]
|
|
331
|
+
if not mine:
|
|
332
|
+
continue
|
|
333
|
+
print(f"\n{t.name}")
|
|
334
|
+
for f in mine:
|
|
335
|
+
tag = " (documented)" if f.documented else ""
|
|
336
|
+
print(f" - {f.kind}{tag}: {f.column + ': ' if f.column else ''}{f.message}")
|
|
337
|
+
return 1 if problems else 0
|
|
338
|
+
|
|
339
|
+
|
|
340
|
+
def engine_identity() -> str:
|
|
341
|
+
return f"{ENGINE_VERSION}+{hashlib.sha256(Path(__file__).read_bytes()).hexdigest()[:12]}"
|
|
342
|
+
|
|
343
|
+
|
|
344
|
+
def find_root(start: Path) -> Path:
|
|
345
|
+
for candidate in [start, *start.parents]:
|
|
346
|
+
if (candidate / "data-catalog.json").exists() or (candidate / ".git").exists():
|
|
347
|
+
return candidate
|
|
348
|
+
return start
|
|
349
|
+
|
|
350
|
+
|
|
351
|
+
def main(argv: list | None = None) -> int:
|
|
352
|
+
parser = argparse.ArgumentParser(prog="data_quality.py", description=__doc__.split("\n\n")[0])
|
|
353
|
+
parser.add_argument("--root", type=Path, help="repo root (default: nearest dir with data-catalog.json or .git)")
|
|
354
|
+
parser.add_argument("--catalog-engine", help=f"path to data_catalog.py (default <root>/{CATALOG_ENGINE})")
|
|
355
|
+
sub = parser.add_subparsers(dest="command", required=True)
|
|
356
|
+
|
|
357
|
+
def scoped(p):
|
|
358
|
+
p.add_argument("--table", action="append", default=[], help="documented table to check; repeatable (default all)")
|
|
359
|
+
p.add_argument("--where", help="SQL condition that bounds the scan, e.g. \"dt >= '2026-10-01'\"")
|
|
360
|
+
p.add_argument("--dialect", choices=["ansi", "spark"], default="ansi", help="identifier quoting: ansi \"x\", spark `x`")
|
|
361
|
+
p.add_argument("--skip", action="append", default=[], help="column to leave out (e.g. arrays); repeatable")
|
|
362
|
+
p.add_argument("--key", action="append", default=[],
|
|
363
|
+
help="grain column; repeat for a composite key: --key dt --key cust_id")
|
|
364
|
+
p.add_argument("--batch", type=int, default=PROFILE_BATCH,
|
|
365
|
+
help=f"columns per profile query (default {PROFILE_BATCH}); lower it if the engine caps stages")
|
|
366
|
+
p.add_argument("--constant-min-rows", type=int, default=CONSTANT_MIN_ROWS,
|
|
367
|
+
help=f"rows needed before a single value counts as constant (default {CONSTANT_MIN_ROWS})")
|
|
368
|
+
|
|
369
|
+
p_plan = sub.add_parser("plan", help="print the SELECT checks as JSON")
|
|
370
|
+
scoped(p_plan)
|
|
371
|
+
p_judge = sub.add_parser("judge", help="judge results; --results maps query id to rows")
|
|
372
|
+
scoped(p_judge)
|
|
373
|
+
p_judge.add_argument("--results", required=True, help="JSON file: {query id: [row objects]}")
|
|
374
|
+
p_judge.add_argument("--json", action="store_true")
|
|
375
|
+
p_run = sub.add_parser("run", help="plan, execute, judge")
|
|
376
|
+
scoped(p_run)
|
|
377
|
+
group = p_run.add_mutually_exclusive_group(required=True)
|
|
378
|
+
group.add_argument("--runner", help="command that reads SQL on stdin and prints JSON or CSV rows")
|
|
379
|
+
group.add_argument("--sqlite", help="SQLite database file (opened read-only)")
|
|
380
|
+
p_run.add_argument("--json", action="store_true")
|
|
381
|
+
sub.add_parser("version", help="print the engine identity")
|
|
382
|
+
args = parser.parse_args(argv)
|
|
383
|
+
if args.command == "version":
|
|
384
|
+
print(engine_identity())
|
|
385
|
+
return 0
|
|
386
|
+
root = (args.root or find_root(Path.cwd())).resolve()
|
|
387
|
+
try:
|
|
388
|
+
tables = documented_tables(root, args.catalog_engine, args.table)
|
|
389
|
+
queries = plan_queries(tables, args.where, args.dialect, set(args.skip), args.batch, args.key)
|
|
390
|
+
if args.command == "plan":
|
|
391
|
+
print(json.dumps(queries, ensure_ascii=False, indent=2))
|
|
392
|
+
return 0
|
|
393
|
+
if args.command == "judge":
|
|
394
|
+
results = json.loads(Path(args.results).read_text(encoding="utf-8"))
|
|
395
|
+
else:
|
|
396
|
+
results = execute(queries, args.runner, args.sqlite)
|
|
397
|
+
found = judge(tables, queries, results, args.where, args.constant_min_rows)
|
|
398
|
+
return report(found, tables, args.where, args.json)
|
|
399
|
+
except QualityError as exc:
|
|
400
|
+
print(f"data-quality: {exc}", file=sys.stderr)
|
|
401
|
+
return 2
|
|
402
|
+
|
|
403
|
+
|
|
404
|
+
if __name__ == "__main__":
|
|
405
|
+
sys.exit(main())
|
|
@@ -0,0 +1,277 @@
|
|
|
1
|
+
"""Tests for data_quality.py against a real repo and a real SQLite warehouse. Run: python3 test_data_quality.py"""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import contextlib
|
|
5
|
+
import io
|
|
6
|
+
import json
|
|
7
|
+
import shutil
|
|
8
|
+
import sqlite3
|
|
9
|
+
import sys
|
|
10
|
+
import tempfile
|
|
11
|
+
import textwrap
|
|
12
|
+
import unittest
|
|
13
|
+
from pathlib import Path
|
|
14
|
+
|
|
15
|
+
HERE = Path(__file__).resolve().parent
|
|
16
|
+
CATALOG_ENGINE = HERE.parent.parent / "data-catalog" / "scripts" / "data_catalog.py"
|
|
17
|
+
sys.path.insert(0, str(HERE))
|
|
18
|
+
import data_quality as dq # noqa: E402
|
|
19
|
+
|
|
20
|
+
SCRIPT = '''
|
|
21
|
+
from catalog_fields import Field, Table
|
|
22
|
+
|
|
23
|
+
CATALOG_TABLE = Table(
|
|
24
|
+
"customer_daily",
|
|
25
|
+
en="Daily customer metrics.",
|
|
26
|
+
grain_en="One row per (dt, cust_id).",
|
|
27
|
+
)
|
|
28
|
+
CATALOG_FIELDS = (
|
|
29
|
+
Field("dt", en="Snapshot date.", source="run parameter dt"),
|
|
30
|
+
Field("cust_id", en="Customer id.", source="dwd.transaction.cust_id"),
|
|
31
|
+
Field("coll_cnt_7d", en="Number of collections in the last 7 days.", source="count of rows", unit="count"),
|
|
32
|
+
Field("recall_ratio", en="Recalled share of collected amount, 0-1.", source="recall / collected", unit="ratio"),
|
|
33
|
+
Field("agent_risk", en="Agent risk: yes, no or undetermined.", source="agent list and recalls",
|
|
34
|
+
values=("是", "否", "无法判定")),
|
|
35
|
+
Field("result", en="Transaction result.", source="constant in v1", values=("SUCCESS", "FAILED")),
|
|
36
|
+
Field("cert_type", en="Payer certificate type.", source="dwd.transaction.debtor_cert_type"),
|
|
37
|
+
Field("legacy_flag", en="Legacy flag; always empty since v2.", source="none"),
|
|
38
|
+
Field("channel", en="Collection channel.", source="dwd.transaction.channel", values=("bank", "card")),
|
|
39
|
+
)
|
|
40
|
+
PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)
|
|
41
|
+
'''
|
|
42
|
+
|
|
43
|
+
ROWS = [
|
|
44
|
+
# dt, cust_id, coll_cnt_7d, recall_ratio, agent_risk, result, cert_type, legacy_flag, channel
|
|
45
|
+
("2026-10-01", "c1", 3, 0.1, "是", "SUCCESS", None, None, "bank"),
|
|
46
|
+
("2026-10-01", "c2", 0, 1.4, "无法判定", "SUCCESS", None, None, "card"),
|
|
47
|
+
("2026-10-02", "c1", -1, 0.0, "无法判定", "SUCCESS", None, None, "wallet"),
|
|
48
|
+
("2026-10-02", "c3", 5, None, "是", "SUCCESS", None, None, "bank"),
|
|
49
|
+
]
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def make_repo(rows=ROWS, script=SCRIPT, extra_column=None) -> tuple[Path, Path]:
|
|
53
|
+
root = Path(tempfile.mkdtemp(prefix="data-quality-test-"))
|
|
54
|
+
(root / ".git").mkdir()
|
|
55
|
+
(root / "scripts/data-catalog").mkdir(parents=True)
|
|
56
|
+
shutil.copy(CATALOG_ENGINE, root / "scripts/data-catalog/data_catalog.py")
|
|
57
|
+
(root / "jobs").mkdir()
|
|
58
|
+
(root / "jobs/customer_daily.py").write_text(textwrap.dedent(script), encoding="utf-8")
|
|
59
|
+
(root / "data-catalog.json").write_text(json.dumps({"version": 1, "scripts": ["jobs/*.py"], "languages": ["en"]}),
|
|
60
|
+
encoding="utf-8")
|
|
61
|
+
db = root / "warehouse.db"
|
|
62
|
+
con = sqlite3.connect(db)
|
|
63
|
+
columns = ["dt TEXT", "cust_id TEXT", "coll_cnt_7d INTEGER", "recall_ratio REAL", "agent_risk TEXT", "result TEXT",
|
|
64
|
+
"cert_type TEXT", "legacy_flag TEXT", "channel TEXT"]
|
|
65
|
+
if extra_column:
|
|
66
|
+
columns.insert(6, extra_column)
|
|
67
|
+
con.execute(f"CREATE TABLE customer_daily ({', '.join(columns)})")
|
|
68
|
+
con.executemany(f"INSERT INTO customer_daily VALUES ({', '.join('?' * len(columns))})", rows)
|
|
69
|
+
con.commit()
|
|
70
|
+
con.close()
|
|
71
|
+
return root, db
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
def run(root: Path, *argv: str) -> tuple[int, str, str]:
|
|
75
|
+
out, err = io.StringIO(), io.StringIO()
|
|
76
|
+
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
|
77
|
+
code = dq.main(["--root", str(root), *argv])
|
|
78
|
+
return code, out.getvalue(), err.getvalue()
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def findings(root: Path, db: Path, *extra: str) -> tuple[int, dict]:
|
|
82
|
+
code, out, err = run(root, "run", "--sqlite", str(db), "--json", "--constant-min-rows", "2", *extra)
|
|
83
|
+
assert out, err
|
|
84
|
+
report = json.loads(out)
|
|
85
|
+
return code, {(f["column"], f["kind"]): f for f in report["findings"]}
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
class RunTest(unittest.TestCase):
|
|
89
|
+
def test_each_check_fires_on_the_column_that_breaks_it(self):
|
|
90
|
+
root, db = make_repo()
|
|
91
|
+
code, found = findings(root, db)
|
|
92
|
+
self.assertEqual(code, 1)
|
|
93
|
+
self.assertEqual(found[("coll_cnt_7d", "negative-value")]["message"], "minimum -1, but the unit is count")
|
|
94
|
+
self.assertEqual(found[("recall_ratio", "out-of-range")]["message"],
|
|
95
|
+
"values from 0 to 1.4, outside 0–1 for unit ratio")
|
|
96
|
+
self.assertEqual(found[("agent_risk", "unseen-value")]["message"], "declared but never present: 否")
|
|
97
|
+
self.assertEqual(found[("channel", "undeclared-value")]["message"], "present but not declared: wallet (1)")
|
|
98
|
+
self.assertEqual(found[("result", "constant")]["message"], "one value in all 4 filled rows")
|
|
99
|
+
self.assertEqual(found[("result", "unseen-value")]["message"], "declared but never present: FAILED")
|
|
100
|
+
self.assertEqual(found[("cert_type", "never-filled")]["message"], "empty in all 4 rows")
|
|
101
|
+
self.assertFalse(found[("cert_type", "never-filled")]["documented"])
|
|
102
|
+
# the definition already says "always empty": reported, but as documented
|
|
103
|
+
self.assertTrue(found[("legacy_flag", "never-filled")]["documented"])
|
|
104
|
+
# clean columns raise nothing
|
|
105
|
+
self.assertFalse([k for k in found if k[0] in ("dt", "cust_id")])
|
|
106
|
+
|
|
107
|
+
def test_a_missing_flag_constant_at_zero_is_healthy_but_at_one_is_not(self):
|
|
108
|
+
script = SCRIPT.replace('Field("cert_type"', 'Field("fx_missing", en="1 if any amount lacked a rate.", source="flag"),\n Field("cert_type"')
|
|
109
|
+
for value, healthy in ((0, True), (1, False)):
|
|
110
|
+
rows = [r[:6] + (value,) + r[6:] for r in ROWS]
|
|
111
|
+
root, db = make_repo(rows, script, extra_column="fx_missing INTEGER")
|
|
112
|
+
_, found = findings(root, db)
|
|
113
|
+
self.assertEqual(found[("fx_missing", "constant")]["documented"], healthy, value)
|
|
114
|
+
|
|
115
|
+
def test_version_and_hash_columns_are_expected_to_be_constant(self):
|
|
116
|
+
script = SCRIPT.replace('Field("cert_type"', 'Field("rule_version", en="Rule set version.", source="RULE_VERSION"),\n Field("cert_type"')
|
|
117
|
+
rows = [r[:6] + ("v2",) + r[6:] for r in ROWS]
|
|
118
|
+
root = Path(tempfile.mkdtemp(prefix="data-quality-test-"))
|
|
119
|
+
root, db = make_repo(rows, script, extra_column="rule_version TEXT")
|
|
120
|
+
_, found = findings(root, db)
|
|
121
|
+
self.assertTrue(found[("rule_version", "constant")]["documented"])
|
|
122
|
+
|
|
123
|
+
def test_duplicate_grain_keys_are_counted(self):
|
|
124
|
+
rows = ROWS + [ROWS[0]] # (2026-10-01, c1) twice
|
|
125
|
+
root, db = make_repo(rows)
|
|
126
|
+
_, found = findings(root, db, "--key", "dt", "--key", "cust_id")
|
|
127
|
+
self.assertEqual(found[(None, "duplicate-key")]["message"],
|
|
128
|
+
"1 (dt, cust_id) value(s) repeat across 2 rows; the grain allows one each")
|
|
129
|
+
root, db = make_repo()
|
|
130
|
+
_, clean = findings(root, db, "--key", "dt", "--key", "cust_id")
|
|
131
|
+
self.assertNotIn((None, "duplicate-key"), clean)
|
|
132
|
+
code, _, err = run(root, "plan", "--key", "nope")
|
|
133
|
+
self.assertEqual(code, 2)
|
|
134
|
+
self.assertIn("--key column(s) not documented in customer_daily: nope", err)
|
|
135
|
+
|
|
136
|
+
def test_where_bounds_the_scan(self):
|
|
137
|
+
root, db = make_repo()
|
|
138
|
+
_, found = findings(root, db, "--where", "dt = '2026-10-01'")
|
|
139
|
+
self.assertNotIn(("coll_cnt_7d", "negative-value"), found)
|
|
140
|
+
self.assertNotIn(("channel", "undeclared-value"), found)
|
|
141
|
+
self.assertIn(("recall_ratio", "out-of-range"), found)
|
|
142
|
+
|
|
143
|
+
def test_where_columns_and_small_samples_are_not_called_constant(self):
|
|
144
|
+
root, db = make_repo()
|
|
145
|
+
_, found = findings(root, db, "--where", "dt = '2026-10-01'")
|
|
146
|
+
self.assertNotIn(("dt", "constant"), found)
|
|
147
|
+
code, out, _ = run(root, "run", "--sqlite", str(db), "--json") # default minimum: 30 rows
|
|
148
|
+
self.assertNotIn("constant", {f["kind"] for f in json.loads(out)["findings"]})
|
|
149
|
+
|
|
150
|
+
def test_an_empty_range_is_reported_once_and_judges_nothing_else(self):
|
|
151
|
+
root, db = make_repo()
|
|
152
|
+
code, found = findings(root, db, "--where", "dt = '1999-01-01'")
|
|
153
|
+
self.assertEqual(code, 1)
|
|
154
|
+
self.assertEqual(list(found), [(None, "empty-table")])
|
|
155
|
+
|
|
156
|
+
def test_clean_data_exits_zero(self):
|
|
157
|
+
clean = [(dt, c, n, ratio, a, r, cert, None, ch) for dt, c, n, ratio, a, r, cert, ch in [
|
|
158
|
+
("2026-10-01", "c1", 3, 0.1, "是", "SUCCESS", "ID", "bank"),
|
|
159
|
+
("2026-10-02", "c2", 0, 0.0, "否", "FAILED", "PASSPORT", "card"),
|
|
160
|
+
("2026-10-03", "c3", 7, 1.0, "无法判定", "SUCCESS", "ID", "bank")]]
|
|
161
|
+
root, db = make_repo(clean)
|
|
162
|
+
code, found = findings(root, db)
|
|
163
|
+
self.assertEqual(code, 0, found)
|
|
164
|
+
self.assertEqual([k for k, f in found.items() if not f["documented"]], [])
|
|
165
|
+
|
|
166
|
+
def test_external_runner_gets_only_selects_and_its_csv_or_json_is_read(self):
|
|
167
|
+
root, db = make_repo()
|
|
168
|
+
runner = root / "runner.py"
|
|
169
|
+
log = root / "sql.log"
|
|
170
|
+
runner.write_text(textwrap.dedent(f'''
|
|
171
|
+
import csv, sqlite3, sys
|
|
172
|
+
sql = sys.stdin.read()
|
|
173
|
+
open({str(log)!r}, "a").write(sql + "\\n")
|
|
174
|
+
con = sqlite3.connect({str(db)!r})
|
|
175
|
+
cur = con.execute(sql)
|
|
176
|
+
w = csv.writer(sys.stdout)
|
|
177
|
+
w.writerow([d[0] for d in cur.description])
|
|
178
|
+
w.writerows(cur.fetchall())
|
|
179
|
+
'''), encoding="utf-8")
|
|
180
|
+
code, out, err = run(root, "run", "--runner", f"{sys.executable} {runner}", "--json")
|
|
181
|
+
self.assertEqual(code, 1, err)
|
|
182
|
+
kinds = {(f["column"], f["kind"]) for f in json.loads(out)["findings"]}
|
|
183
|
+
self.assertIn(("coll_cnt_7d", "negative-value"), kinds)
|
|
184
|
+
statements = [s for s in log.read_text(encoding="utf-8").split("\n") if s.strip()]
|
|
185
|
+
self.assertTrue(statements and all(s.startswith("SELECT ") for s in statements))
|
|
186
|
+
|
|
187
|
+
|
|
188
|
+
class PlanJudgeTest(unittest.TestCase):
|
|
189
|
+
def test_plan_then_judge_matches_run(self):
|
|
190
|
+
root, db = make_repo()
|
|
191
|
+
code, out, _ = run(root, "plan", "--dialect", "spark", "--skip", "channel")
|
|
192
|
+
self.assertEqual(code, 0)
|
|
193
|
+
queries = json.loads(out)
|
|
194
|
+
self.assertEqual([q["id"] for q in queries],
|
|
195
|
+
["profile:customer_daily", "values:customer_daily.agent_risk", "values:customer_daily.result"])
|
|
196
|
+
self.assertIn("COUNT(DISTINCT `coll_cnt_7d`) AS d_2", queries[0]["sql"])
|
|
197
|
+
self.assertIn("MIN(`recall_ratio`) AS mn_3", queries[0]["sql"])
|
|
198
|
+
self.assertNotIn("channel", queries[0]["sql"])
|
|
199
|
+
con = sqlite3.connect(db)
|
|
200
|
+
results = {}
|
|
201
|
+
for q in queries:
|
|
202
|
+
cur = con.execute(q["sql"].replace("`", '"'))
|
|
203
|
+
results[q["id"]] = [dict(zip([d[0] for d in cur.description], r)) for r in cur.fetchall()]
|
|
204
|
+
path = root / "results.json"
|
|
205
|
+
path.write_text(json.dumps(results), encoding="utf-8")
|
|
206
|
+
code, out, _ = run(root, "judge", "--skip", "channel", "--results", str(path), "--json", "--constant-min-rows", "2")
|
|
207
|
+
kinds = {(f["column"], f["kind"]) for f in json.loads(out)["findings"]}
|
|
208
|
+
self.assertIn(("agent_risk", "unseen-value"), kinds)
|
|
209
|
+
self.assertNotIn(("channel", "undeclared-value"), kinds)
|
|
210
|
+
|
|
211
|
+
def test_profile_batches_cover_every_column_and_report_an_empty_range_once(self):
|
|
212
|
+
root, db = make_repo()
|
|
213
|
+
code, out, _ = run(root, "plan", "--batch", "4")
|
|
214
|
+
ids = [q["id"] for q in json.loads(out) if q["kind"] == "profile"]
|
|
215
|
+
self.assertEqual(ids, ["profile:customer_daily#1", "profile:customer_daily#2", "profile:customer_daily#3"])
|
|
216
|
+
cols = [c for q in json.loads(out) if q["kind"] == "profile" for c in q["columns"]]
|
|
217
|
+
self.assertEqual(len(cols), 9)
|
|
218
|
+
_, found = findings(root, db, "--batch", "4")
|
|
219
|
+
self.assertIn(("recall_ratio", "out-of-range"), found)
|
|
220
|
+
self.assertIn(("legacy_flag", "never-filled"), found)
|
|
221
|
+
_, empty = findings(root, db, "--batch", "4", "--where", "dt = 'none'")
|
|
222
|
+
self.assertEqual(list(empty), [(None, "empty-table")])
|
|
223
|
+
|
|
224
|
+
def test_a_missing_result_is_reported_not_skipped(self):
|
|
225
|
+
root, _ = make_repo()
|
|
226
|
+
path = root / "results.json"
|
|
227
|
+
path.write_text("{}", encoding="utf-8")
|
|
228
|
+
code, out, _ = run(root, "judge", "--results", str(path), "--json")
|
|
229
|
+
self.assertEqual(code, 1)
|
|
230
|
+
self.assertIn("not-run", {f["kind"] for f in json.loads(out)["findings"]})
|
|
231
|
+
|
|
232
|
+
|
|
233
|
+
class OldEngineTest(unittest.TestCase):
|
|
234
|
+
def test_fields_from_an_engine_copy_without_values_still_plan(self):
|
|
235
|
+
from types import SimpleNamespace as NS
|
|
236
|
+
old = NS(name="t", fields=[NS(name="a", text={"en": "x"}, source="s", window=None, unit="count", line=1)])
|
|
237
|
+
[query] = dq.plan_queries([old], None, "ansi", set())
|
|
238
|
+
self.assertEqual(query["sql"], 'SELECT COUNT(*) AS n, COUNT("a") AS nn_0, COUNT(DISTINCT "a") AS d_0, '
|
|
239
|
+
'MIN("a") AS mn_0, MAX("a") AS mx_0 FROM t')
|
|
240
|
+
|
|
241
|
+
|
|
242
|
+
class RowsTest(unittest.TestCase):
|
|
243
|
+
def test_runner_output_formats(self):
|
|
244
|
+
self.assertEqual(dq.parse_rows('[{"n": 1}]'), [{"n": 1}])
|
|
245
|
+
self.assertEqual(dq.parse_rows('{"value": "a", "n": 2}\n{"value": "b", "n": 3}\n'),
|
|
246
|
+
[{"value": "a", "n": 2}, {"value": "b", "n": 3}])
|
|
247
|
+
self.assertEqual(dq.parse_rows("value,n\na,2\n"), [{"value": "a", "n": "2"}])
|
|
248
|
+
self.assertEqual(dq.parse_rows(""), [])
|
|
249
|
+
|
|
250
|
+
|
|
251
|
+
class SetupTest(unittest.TestCase):
|
|
252
|
+
def test_without_the_catalog_engine_it_explains_the_setup(self):
|
|
253
|
+
root, _ = make_repo()
|
|
254
|
+
(root / "scripts/data-catalog/data_catalog.py").unlink()
|
|
255
|
+
code, _, err = run(root, "plan")
|
|
256
|
+
self.assertEqual(code, 2)
|
|
257
|
+
self.assertIn("set the repo up with the data-catalog skill first", err)
|
|
258
|
+
|
|
259
|
+
def test_an_unknown_table_is_refused(self):
|
|
260
|
+
root, _ = make_repo()
|
|
261
|
+
code, _, err = run(root, "plan", "--table", "nope")
|
|
262
|
+
self.assertEqual(code, 2)
|
|
263
|
+
self.assertIn("not documented in the catalog: nope", err)
|
|
264
|
+
|
|
265
|
+
def test_statements_other_than_one_select_are_refused(self):
|
|
266
|
+
for sql in ("DELETE FROM t", "SELECT 1; DROP TABLE t"):
|
|
267
|
+
with self.assertRaises(dq.QualityError):
|
|
268
|
+
dq.assert_read_only(sql)
|
|
269
|
+
|
|
270
|
+
def test_the_sqlite_runner_cannot_write(self):
|
|
271
|
+
root, db = make_repo()
|
|
272
|
+
with self.assertRaises(sqlite3.OperationalError):
|
|
273
|
+
dq.run_sqlite(str(db), "INSERT INTO customer_daily (dt) VALUES ('x')")
|
|
274
|
+
|
|
275
|
+
|
|
276
|
+
if __name__ == "__main__":
|
|
277
|
+
unittest.main()
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ios-foundation
|
|
3
3
|
description: "Set up and prove an Expo / React Native native iOS workspace from a clean checkout to a manual Login smoke on an owned iOS Simulator: isolated worktree and private caches, Xcode/runtime/CocoaPods preflight, frozen install with `pod install --deployment`, platform-scoped prebuild, an ad-hoc-signed Debug simulator build, a private simulator by exact UDID, Metro on a free port, a disposable PostgreSQL backend with owned cleanup, and the evidence record (setup proof only; feature work reports the smoke result). Use for 'set up iOS', 'build the iOS app', 'run the app on the iOS simulator', 'add iOS to the React Native app', 'prove the iOS build', or when a pod install, xcodebuild, simulator boot, Metro connection or Keychain session fails. NOT for Android (android-foundation), signing, TestFlight or App Store release, or general workflow YAML (github-actions)."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# iOS foundation
|
package/skills/monorepo/SKILL.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: monorepo
|
|
3
3
|
description: Turborepo architecture for shared packages, pnpm workspaces, DAL boundaries, and pipelines.
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Turborepo Monorepo Architecture
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: nextjs-fullstack
|
|
3
3
|
description: "Build or debug Next.js App Router features with React 19 — pages/layouts/route handlers, Server/Client Components, Server Actions, forms, middleware, DAL, strict TypeScript, Zod — plus our conventions: nuqs URL state, SWR cache, Skeleton/Progress + Sonner feedback, drag-and-drop upload, the portal app shell, list pages as filter/sort/paginate tables. shadcn component mechanics: see the shadcn skill."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Next.js 16 Fullstack Development
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: prisma-setup
|
|
3
3
|
description: Use when adding or debugging Prisma with PostgreSQL, Next.js, or Turborepo, including schema.prisma, prisma.config.ts, client generation, migrations, seeds, relations, adapters, and DAL/database-layer setup. Triggers on phrases like "Prisma schema", "prisma migrate", "schema.prisma", "seed the database", "PrismaClient error", "relation issue", or "database layer".
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Prisma v7 Setup Guide
|
package/skills/pyspark/SKILL.md
CHANGED
|
@@ -12,7 +12,7 @@ description: >
|
|
|
12
12
|
Connect, spark-submit, Hudi commits, backfills, validate-only canaries,
|
|
13
13
|
cluster reconciliation, or Spark event-log profiling — including warehouse
|
|
14
14
|
job timeouts, empty Hudi writes, or EMR/Dataproc performance regressions.
|
|
15
|
-
release-version: 0.
|
|
15
|
+
release-version: 0.39.0
|
|
16
16
|
---
|
|
17
17
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
18
18
|
# PySpark
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: python-backend
|
|
3
3
|
description: "Python backend architecture for FastAPI, Tortoise ORM, Pydantic Settings, loguru, and uv. Also: uv.lock installs in CI/Docker via a proxy or mirror (`uv sync --frozen` bypasses it), slow uv installs on runners, relocking."
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Python Backend Conventions
|
package/skills/shadcn/SKILL.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: shadcn
|
|
3
3
|
description: "Add or debug shadcn/ui in a React/Next.js project: components.json, shadcn init/add, presets, registries, Tailwind styling, base/radix/aria composition, component fixes, and list-index pages — where the default is the DiceUI data table with filter/sort/pagination in the URL via nuqs, not a bare shadcn Table."
|
|
4
4
|
allowed-tools: Bash(npx shadcn@latest *), Bash(pnpm dlx shadcn@latest *), Bash(bunx --bun shadcn@latest *)
|
|
5
|
-
release-version: 0.
|
|
5
|
+
release-version: 0.39.0
|
|
6
6
|
---
|
|
7
7
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
8
8
|
# shadcn/ui
|
package/skills/zod-v4/SKILL.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: zod-v4
|
|
3
3
|
description: Zod v4 schema conventions and v3 migration for errors, validators, objects, enums, records, and async parsing.
|
|
4
|
-
release-version: 0.
|
|
4
|
+
release-version: 0.39.0
|
|
5
5
|
---
|
|
6
6
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
7
7
|
# Zod v3 to v4 Migration
|