@1aboveio/app-dev-skills 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (34) hide show
  1. package/README.md +2 -0
  2. package/package.json +7 -6
  3. package/skills/airflow/SKILL.md +1 -1
  4. package/skills/android-foundation/SKILL.md +1 -1
  5. package/skills/app-debug/SKILL.md +1 -1
  6. package/skills/better-auth/SKILL.md +1 -1
  7. package/skills/data-catalog/LICENSE +3 -0
  8. package/skills/data-catalog/SKILL.md +208 -0
  9. package/skills/data-catalog/agents/openai.yaml +3 -0
  10. package/skills/data-catalog/references/data-catalog-conventions.md +77 -0
  11. package/skills/data-catalog/references/field-definitions.md +124 -0
  12. package/skills/data-catalog/references/format.md +134 -0
  13. package/skills/data-catalog/release-check.mjs +93 -0
  14. package/skills/data-catalog/scripts/data-catalog.test.mjs +36 -0
  15. package/skills/data-catalog/scripts/data_catalog.py +1447 -0
  16. package/skills/data-catalog/scripts/persist-conventions-core.mjs +117 -0
  17. package/skills/data-catalog/scripts/persist-conventions.mjs +17 -0
  18. package/skills/data-catalog/scripts/test_data_catalog.py +618 -0
  19. package/skills/data-quality/LICENSE +3 -0
  20. package/skills/data-quality/SKILL.md +96 -0
  21. package/skills/data-quality/agents/openai.yaml +3 -0
  22. package/skills/data-quality/references/checks.md +56 -0
  23. package/skills/data-quality/release-check.mjs +93 -0
  24. package/skills/data-quality/scripts/data-quality.test.mjs +12 -0
  25. package/skills/data-quality/scripts/data_quality.py +405 -0
  26. package/skills/data-quality/scripts/test_data_quality.py +277 -0
  27. package/skills/ios-foundation/SKILL.md +1 -1
  28. package/skills/monorepo/SKILL.md +1 -1
  29. package/skills/nextjs-fullstack/SKILL.md +1 -1
  30. package/skills/prisma-setup/SKILL.md +1 -1
  31. package/skills/pyspark/SKILL.md +1 -1
  32. package/skills/python-backend/SKILL.md +1 -1
  33. package/skills/shadcn/SKILL.md +1 -1
  34. package/skills/zod-v4/SKILL.md +1 -1
package/README.md CHANGED
@@ -16,6 +16,8 @@ pi install npm:@1aboveio/app-dev-skills
16
16
  - `android-foundation`
17
17
  - `app-debug`
18
18
  - `better-auth`
19
+ - `data-catalog`
20
+ - `data-quality`
19
21
  - `ios-foundation`
20
22
  - `monorepo`
21
23
  - `nextjs-fullstack`
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@1aboveio/app-dev-skills",
3
- "version": "0.1.0",
3
+ "version": "0.1.2",
4
4
  "description": "Next.js, React Native, auth, Prisma, shadcn, Python backends, Airflow and PySpark.",
5
5
  "license": "MIT",
6
6
  "repository": {
@@ -8,6 +8,7 @@
8
8
  "url": "git+https://github.com/1aboveio/ai.git",
9
9
  "directory": "skills/distribution/packages"
10
10
  },
11
+ "homepage": "https://github.com/1aboveio/ai/releases?q=%401aboveio%2Fapp-dev-skills&expanded=true",
11
12
  "keywords": [
12
13
  "pi-package",
13
14
  "agent-skills",
@@ -17,10 +18,6 @@
17
18
  "skills",
18
19
  "README.md"
19
20
  ],
20
- "publishConfig": {
21
- "registry": "https://registry.npmjs.org/",
22
- "access": "public"
23
- },
24
21
  "pi": {
25
22
  "skills": [
26
23
  "./skills"
@@ -30,6 +27,10 @@
30
27
  "id": "app-dev",
31
28
  "core": "@1aboveio/core-skills",
32
29
  "coreSkills": [],
33
- "digest": "e656c7cd36fb0ab894e8d83e96a745eab36d78402348e0e2dcb591a12e4270a2"
30
+ "digest": "de7c7e63b8cf033c471b81777170064bef838b906cc332885b0a3a652f2ff96a"
31
+ },
32
+ "publishConfig": {
33
+ "registry": "https://registry.npmjs.org/",
34
+ "access": "public"
34
35
  }
35
36
  }
@@ -2,7 +2,7 @@
2
2
  name: airflow
3
3
  description: Airflow DAG architecture and conventions. Use when designing, writing, reviewing, testing, or debugging Airflow DAGs and tasks, including TaskGroups, data intervals, timezone-aware date parameters, CLI validation, environment-based scheduling, and pytest integrity tests.
4
4
  user-invocable: true
5
- release-version: 0.36.0
5
+ release-version: 0.39.0
6
6
  ---
7
7
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
8
8
  # Airflow DAG Development Patterns
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: android-foundation
3
3
  description: "Prove an Expo / React Native app on a native Android emulator, and fix it when that proof fails. Use before you fix and re-run or re-dispatch a failed Android CI job (`gh workflow run android.yml`, `gh run rerun`): a re-run is not a debugger, so reproduce the failing journey checkpoint off CI first. Also use when a UiAutomator journey or its Python harness tests fail (a CHECK/FAIL line, a wait that timed out, a control not found, adb offline, a lost Metro bridge, a fixture failing on the runner), to build the debug APK and run the device journey, to set up a fresh Android workspace (SDK/NDK/JDK, Gradle caches, private AVD, disposable PostgreSQL), or to add the emulator job to CI. NOT for workflow YAML, CI speed or runner design (github-actions), browser tests (e2e-test), iOS (ios-foundation), plain Jest component tests, or signing and store release."
4
- release-version: 0.36.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Android foundation
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: app-debug
3
3
  description: "Diagnose runtime failures in deployed Cloud Run services — 5xx/422 errors, tracebacks, Next.js SSR/hydration or Server Action failures, DB/Secret Manager connection errors, cold-start timeouts, container OOM. NOT for build or deploy failures (cloud-debug)."
4
- release-version: 0.36.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # App Debug
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: better-auth
3
3
  description: "Add or debug Better Auth for Next.js App Router: login, sessions, protected routes, middleware, OAuth (including Feishu/Lark), cookies, Prisma, and Auth.js migration."
4
- release-version: 0.36.0
4
+ release-version: 0.39.0
5
5
  ---
6
6
  **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
7
  # Better Auth for Next.js
@@ -0,0 +1,3 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 1AboveIO
@@ -0,0 +1,208 @@
1
+ ---
2
+ name: data-catalog
3
+ description: "Column definitions for warehouse tables, written inside the Spark/PySpark or SQL script that builds each table (CATALOG_TABLE / CATALOG_FIELDS, or a `-- @field` comment block), with a generated markdown catalog in docs/catalog/ and a CI check that fails a PR whose catalog is stale. Use when writing or changing a table script that adds, renames, removes, or changes the meaning of an output column or feature; when asked what a warehouse column or feature means; to set a repo up ('init the data catalog', 'generate the data dictionary', '字段字典', 'add catalog CI'); or to audit existing SQL or Spark scripts for undocumented columns. NOT for implementing or tuning the Spark job itself (pyspark), DAG wiring (airflow), or general workflow YAML (github-actions)."
4
+ release-version: 0.39.0
5
+ ---
6
+ **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
7
+ # Data catalog
8
+
9
+ Every output column of a warehouse table gets its definition written next to the
10
+ code that computes it. A script turns those definitions into `docs/catalog/*.md`,
11
+ and CI fails a pull request when the pages and the scripts disagree.
12
+
13
+ ## First use in a repo
14
+
15
+ In a git repo with no `data-catalog.json`, set the repo up **before** the task you
16
+ were asked to do, whatever it is: an audit, a new column, or a question. Setting up
17
+ puts the engine, the CI check, and the conventions in the repo itself, so agents
18
+ and people without this skill follow them from then on.
19
+
20
+ 1. `python3 <skill-base-dir>/scripts/data_catalog.py audit '<globs>'` to find the
21
+ table scripts; choose `--scripts` globs that cover them and nothing else.
22
+ 2. Pick `--languages` from the repo's existing docs (Chinese docs: `zh,en`;
23
+ otherwise `en`). Run `init` with them and `--write` ([Init](#init)).
24
+ 3. `node <skill-base-dir>/scripts/persist-conventions.mjs --repo-root <repo> --write`
25
+ writes the `## Data catalog conventions` section of `CONTRIBUTING.md` and points
26
+ `AGENTS.md` at it.
27
+ 4. Then do the task. In the reply, list what setup added, and say the Data catalog
28
+ check runs on PRs but is not required.
29
+
30
+ Skip setup outside a git repo, or when the user says not to change the repo.
31
+ Never document tables during setup unless the task asks for it.
32
+
33
+ ## Find your task
34
+
35
+ | Task | Do |
36
+ |---|---|
37
+ | Writing or changing a table script | [While coding](#while-coding), then [references/field-definitions.md](references/field-definitions.md) |
38
+ | "What does column X mean?" | Read `docs/catalog/<table>.md`; with no page, read the script's `CATALOG_FIELDS`, then its SQL |
39
+ | The repo has no `data-catalog.json` | [First use in a repo](#first-use-in-a-repo) |
40
+ | Find scripts with missing or weak definitions | [Audit](#audit) |
41
+ | Document an existing table | [Backfill one table](#backfill-one-table) |
42
+ | CI says the catalog is stale | `python3 <engine> generate`, commit the pages |
43
+ | Before committing a table-script change, or reviewing one | `python3 <engine> drift` (`--base origin/main` for a whole branch) |
44
+ | `Field`/`Table` arguments, the SQL block, config keys, exit codes | [references/format.md](references/format.md) |
45
+ | Does the table's data match its definitions? | the data-quality skill |
46
+
47
+ `<engine>` is `scripts/data-catalog/data_catalog.py` in the repo, or
48
+ `<skill-base-dir>/scripts/data_catalog.py` before the repo has a copy. When
49
+ `python3 <engine> version` differs from the skill's copy, refresh with
50
+ `init --write`.
51
+
52
+ ## The files
53
+
54
+ | File | Owner | Edit it? |
55
+ |---|---|---|
56
+ | `CATALOG_TABLE` / `CATALOG_FIELDS` in each `.py` table script | the definitions | yes |
57
+ | `-- @table` / `-- @field` block at the top of each `.sql` table script | the definitions | yes |
58
+ | `data-catalog.json` | which scripts, languages, output dir | yes, then rerun `init --write` |
59
+ | `docs/catalog/*.md` | generated pages | never |
60
+ | `catalog_fields.py` (path in `fieldsModule`) | generated `Field`/`Table` types | never |
61
+ | `scripts/data-catalog/data_catalog.py` | engine copied from this skill | never; refresh with init |
62
+ | `.github/workflows/data-catalog.yml` | generated CI check | never; rerun init |
63
+ | `CONTRIBUTING.md` section between `conventions:data-catalog` markers | the conventions, for agents without this skill | never; rerun `persist-conventions.mjs --write`; the rest is yours |
64
+ | `AGENTS.md` `## Contributing` pointer | points at `CONTRIBUTING.md` | written once by `persist-conventions.mjs` |
65
+
66
+ ## Rules
67
+
68
+ 1. **Define a column in the same change that creates or changes it.**
69
+ Example: adding `refund_cnt_30d` to the SELECT means adding its `Field` in the
70
+ same commit, then running `generate`.
71
+ 2. **Each row says only what is particular to its column, in business words.**
72
+ Shared filters, window anchor and currency go once in the table notes, in every
73
+ language; the row
74
+ gives what one value measures, its unit, and 0/null meaning; `source` gives
75
+ lineage in one line. Follow the eleven principles in
76
+ [references/field-definitions.md](references/field-definitions.md).
77
+ Example: not `"Posted COLLECTION_IN rows where cust_type <> 'TEST', [dt-30d, dt), count distinct payer_bank via dim.forex join"`
78
+ but `"Number of different paying banks in the last 30 days. It is 0 if there were no collections."`
79
+ 3. **Write what the code does, read from the code.** Take the source table, filters,
80
+ and window from the SQL you read. When the code and the intended meaning
81
+ disagree, ask; never document the intent as fact.
82
+ Example: the SQL filters `status = 'POSTED'`, so the source says "posted
83
+ collections", even if the ticket said "all collections".
84
+ 4. **Edit the script, never a generated page.** `generate --check` fails while they
85
+ disagree, and when an output column has no definition: the contract tuple or, in
86
+ SQL, the SELECT writing the table, is checked against the `Field`s or `-- @field`s
87
+ ([references/format.md](references/format.md#what-ci-checks-against-the-code)).
88
+ Example: to fix a typo on a page, fix the `Field` text and run `generate`.
89
+ 5. **Derive the column contract only after every column has a `Field`.**
90
+ `PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)` publishes exactly the
91
+ documented columns, in `Field` order. Before switching, compare the derived tuple
92
+ with the old one: same names, same order.
93
+ Example: a 169-column table with 154 Fields would silently stop writing 15
94
+ columns. Keep the literal tuple; `generate` names the 15 until they are written.
95
+ 6. **Metadata is plain literals.** The engine reads the file without running it:
96
+ strings, implicit concatenation, `+`, module-level string constants, and
97
+ `*OTHER_FIELDS` from the same module. No f-strings, calls, or imported lists.
98
+ Example: share key columns with `KEYS = (Field(...),)` and `(*KEYS, Field(...))`
99
+ in the same file.
100
+ 7. **Audit reports; it never edits.** Fix one table per change, in the script that
101
+ builds the table, not its backfill or migration twins.
102
+ 8. **Adding the workflow adds a check, not a required one.** Make it required in
103
+ the merge rules only when the user asks.
104
+ Example: after `init --write`, report "the Data catalog check now runs on PRs
105
+ that touch table scripts; it is not required".
106
+
107
+ ## While coding
108
+
109
+ 1. Before changing a table script, read its page in `docs/catalog/`.
110
+ 2. Change the SQL or DataFrame code.
111
+ 3. Add, edit, or remove the matching `Field`. Write it with
112
+ [references/field-definitions.md](references/field-definitions.md).
113
+ 4. Run `python3 <engine> generate` and the repo's schema or contract tests.
114
+ 5. Run `python3 <engine> drift`. It lists columns whose code changed while their
115
+ definition did not, and filter or join changes with no definition touched. CI
116
+ only checks that every column has a definition; `drift` is the check that the
117
+ definitions still match the code. Update each listed definition, or say in the
118
+ PR why it still holds.
119
+ 6. Commit the script and the regenerated pages together.
120
+
121
+ Example: changing `WHERE status = 'POSTED'` to `status IN ('POSTED', 'SETTLED')`
122
+ passes CI, and `drift` reports "filters, joins or CTEs changed; no definition
123
+ changed" until the affected sources say "posted or settled".
124
+
125
+ A renamed column is a new `Field` and a removed one. Say in the new definition what
126
+ it replaced when consumers must migrate.
127
+
128
+ ## Init
129
+
130
+ ```bash
131
+ python3 <skill-base-dir>/scripts/data_catalog.py init \
132
+ --scripts 'dags/**/dwd_*.py' --scripts 'sql/**/*.sql' \
133
+ --exclude '**/*_legacy.py' --languages zh,en # dry run: prints the plan
134
+ python3 <skill-base-dir>/scripts/data_catalog.py init ... --write
135
+ ```
136
+
137
+ Scope `--scripts` to warehouse table scripts. A broad `**/*.sql` also picks up app
138
+ database migrations. `init` writes the config, the engine copy, `catalog_fields.py`
139
+ beside the scripts, the workflow, and the first catalog. Rerunning is safe, and edits
140
+ to `data-catalog.json` take effect on the next `init --write`. After a skill update,
141
+ rerun both `init --write` and `persist-conventions.mjs --write` so the repo's copies
142
+ match.
143
+
144
+ The scripts import `from catalog_fields import Field, Table` at run time, so the
145
+ module must ship with them like any sibling module (for example `--py-files` on
146
+ `spark-submit`). Check that before merging.
147
+
148
+ ## Audit
149
+
150
+ ```bash
151
+ python3 <engine> audit # scripts from data-catalog.json
152
+ python3 <engine> audit 'dags/**/*.py' '**/*.sql' # before init
153
+ python3 <engine> audit --json # for a worklist
154
+ ```
155
+
156
+ The audit finds the table each script writes: `TARGET_TABLE`-style constants,
157
+ `dst_db`/`dst_tbl`-style attributes, `saveAsTable`/`insertInto`, Hudi
158
+ `hoodie.table.name`, and `INSERT`/`CREATE TABLE AS`/`MERGE` in SQL text. It reads
159
+ output columns from a contract tuple (`PUBLISHED_COLUMNS`) or from the outermost
160
+ SELECT. A repo with its own naming adds `--table-name NAME`/`--schema-name NAME`,
161
+ or `tableNames`/`schemaNames` in the config.
162
+
163
+ | Finding | Means | Fix |
164
+ |---|---|---|
165
+ | `no-catalog` | writes a table, carries no metadata | [Backfill](#backfill-one-table) in the builder script |
166
+ | `undocumented` | output columns without a `Field` | write them |
167
+ | `not-in-output` | `Field`s the SELECT no longer produces | remove or rename them |
168
+ | `table-mismatch` | metadata names a table the script does not write | fix the name |
169
+ | `contract-not-derived` | contract is a literal list next to complete metadata | derive it (rule 5) |
170
+ | `relative-definition` | "same as above" text | write the row in full (rule 2) |
171
+ | `repeated-text` | most rows share a phrase | move the shared fact to the table notes |
172
+ | `code-in-description` | table names, operators or SQL functions in the meaning text | move lineage to `source` |
173
+ | `long-definition` | over 30 words or 60 Chinese characters | cut to the column's own meaning |
174
+ | `thin-definition` | text or source only restates the column name | say what it measures and where it comes from |
175
+ | `columns-unknown` | DataFrame code the audit cannot read | compare the `Field`s with `DESCRIBE <table>` by hand |
176
+ | `invalid-metadata` | the engine cannot read the metadata | the message names the line |
177
+
178
+ Report the summary line and the findings for the tables in scope. Never claim a
179
+ script is clean when its columns were `unknown`.
180
+
181
+ ## Backfill one table
182
+
183
+ 1. Pick the script that builds the table from the audit. Backfill, migration, and
184
+ copy scripts that write the same table share its definitions and need none of
185
+ their own.
186
+ 2. Read the SQL or DataFrame code for every output column. Reuse an existing
187
+ hand-written dictionary as a draft, and check every row against the code.
188
+ 3. Add `CATALOG_TABLE` and `CATALOG_FIELDS` in contract order. Keep the literal
189
+ contract while you write.
190
+ 4. Run `generate` until it passes, then derive the contract and run the repo's
191
+ schema tests (rule 5).
192
+ 5. Rerun the audit for that script and fix its writing findings (`repeated-text`,
193
+ `code-in-description`, `long-definition`, `thin-definition`,
194
+ `relative-definition`), or leave one only when the text really belongs in that row.
195
+ 6. Commit the script and pages together. Replace the old hand-written dictionary
196
+ with a link to the generated page.
197
+
198
+ Uncertain rows: write what the code shows and list the open questions in the PR
199
+ description for the user. Never leave a placeholder `Field`; `generate` refuses
200
+ empty text.
201
+
202
+ ## Done when
203
+
204
+ - `python3 <engine> generate --check` exits 0.
205
+ - `python3 <engine> drift` exits 0, or you said in the PR why each listed definition
206
+ still holds.
207
+ - `python3 <engine> audit` has no findings for the tables you touched, or you
208
+ reported the ones left and why.
@@ -0,0 +1,3 @@
1
+ interface:
2
+ display_name: "Data catalog"
3
+ short_description: "Column definitions in table scripts, a generated markdown catalog with a staleness CI check, and an audit of SQL and Spark scripts"
@@ -0,0 +1,77 @@
1
+ ## Data catalog conventions
2
+ <!-- skill-script-path: consumer-copy -->
3
+
4
+ Every output column of a warehouse table is defined in the script that builds the
5
+ table. `docs/catalog/` is generated from those definitions; never edit it by hand.
6
+ `data-catalog.json` lists the table scripts and the languages each definition carries.
7
+
8
+ **Where definitions live**
9
+
10
+ - Python: `CATALOG_TABLE = Table("schema.table", <language>=..., grain_<language>=...,
11
+ notes_<language>=(...))` (grain and notes in every language) and
12
+ `CATALOG_FIELDS = (Field("column", <language>=..., source=..., window=..., unit=...), ...)`,
13
+ imported `from catalog_fields import Field, Table`. Use plain string literals.
14
+ - SQL: a comment block above the first statement: `-- @table schema.table`, then one
15
+ `-- @field column` per column, each followed by `-- key: value` lines
16
+ (`en:`, `zh:`, `source:`, `window:`, `unit:`).
17
+ - Backfill and migration scripts that write the same table carry no definitions of
18
+ their own.
19
+
20
+ **When you change a table script**
21
+
22
+ 1. Read the table's page in `docs/catalog/` first.
23
+ 2. Add, change, or remove the column's definition in the same change as the code.
24
+ 3. Run `python3 scripts/data-catalog/data_catalog.py generate`, and commit the script
25
+ and the regenerated pages together.
26
+ 4. Run `python3 scripts/data-catalog/data_catalog.py drift`. It lists columns whose
27
+ code changed while their definition did not. Update each, or say in the PR why
28
+ it still holds.
29
+
30
+ **How to write a definition**
31
+
32
+ 1. Say each fact once, at the widest level where it is true. Filters, the window
33
+ anchor, currency, time zone, terms many columns use ("personal", "platform") and
34
+ list cleaning rules go in the table's grain or notes; a row says only what
35
+ differs for that column.
36
+ 2. The description is meaning in business words: what one value measures, its unit
37
+ and form (a ratio says 0–1 or percent; a time says date or timestamp and its
38
+ time zone), and what 0 or null means when not obvious. No table names, joins, operators or
39
+ SQL functions there.
40
+ 3. Lead with the measure in one sentence: "Number of …", "Total …", "近 30 天…笔数".
41
+ Aim for 20 English words or 40 Chinese characters.
42
+ 4. `source` is lineage in one line: upstream `table.column`, the aggregation, and only
43
+ the filters this column adds. Example: "count distinct dwd.transaction.payer_bank".
44
+ Never copy the SQL into prose.
45
+ 5. Keep caveats that change how the value is used ("names compared as exact text");
46
+ leave out mechanics that don't.
47
+ 6. The exact interval goes in `window` and the unit code in `unit`; the text says
48
+ "last 7 days" and "USD".
49
+ 7. Every language says the same thing, written naturally in that language.
50
+ 8. Write sentences, not shorthand: keep the words that carry the logic (若…则, 当…时,
51
+ if, when, because). "若窗口内无收款,则为 0", not "无收款为 0"; "It is 0 if there were no
52
+ collections.", not "0 when none".
53
+ 9. Never restate the column name or write "same as above".
54
+ 10. Cover every case: what the column holds for each kind of row, including when it
55
+ is empty; list an enumerated column's values in the text and in `values=(...)`;
56
+ say so if it is always empty today.
57
+ 11. Write related columns as a set: a count and its amount, a value and its missing
58
+ flag, a legacy column and its replacement use the same scope and wording, and any
59
+ difference is stated in both rows.
60
+
61
+ Example row: `Field("payer_bank_cnt_30d", en="Number of different paying banks in the
62
+ last 30 days. It is 0 if there were no collections.", zh="近 30 天不同付款银行数;若无收款,则为 0。", source="count
63
+ distinct dwd.transaction.payer_bank", window="[dt-30d, dt)", unit="count")`
64
+
65
+ **Contracts and CI**
66
+
67
+ - A column contract (`PUBLISHED_COLUMNS`) is derived as
68
+ `tuple(f.name for f in CATALOG_FIELDS)` only once every column has a `Field`.
69
+ Before switching, check that the names and their order are unchanged.
70
+ - The `Data catalog` workflow runs `generate --check` on PRs. It fails when a page is
71
+ stale, or when an output column has no definition. The column list comes from the
72
+ contract tuple or, in SQL, from the SELECT that writes the table. `drift` is not
73
+ run in CI.
74
+ - `python3 scripts/data-catalog/data_catalog.py audit` lists scripts whose tables lack
75
+ definitions, and rows that break the writing rules (`repeated-text`,
76
+ `code-in-description`, `long-definition`, `thin-definition`, `relative-definition`).
77
+ It reports and never edits.
@@ -0,0 +1,124 @@
1
+ # Writing field definitions
2
+
3
+ A reader opens a table's page to learn what each value means and whether they can
4
+ use it. Write for that reader: the table header once, then one short line per
5
+ column that says only what is particular to that column.
6
+
7
+ ## Principles
8
+
9
+ 1. **Say each fact once, at the widest level where it is true.** A filter, window
10
+ anchor, currency rule, or row-inclusion rule that holds for every column goes in
11
+ the table's grain or notes, never in each row. So does the meaning of a term
12
+ many columns use ("personal", "platform", "own name") and how lists are cleaned
13
+ (case, sorting, empty items). A row carries only what differs for that column.
14
+ Bad (every row): `"Posted collections only, test customers excluded, [dt-7d, dt), excluding dt; ..."`
15
+ Good: table note `"All metrics count posted collections from non-test customers before dt."`,
16
+ row `"Number of collections in the last 7 days. It is 0 if there were no collections."`
17
+ 2. **The description is meaning, in business words.** What one value measures, its
18
+ unit and form, and what 0 or null means when that is not obvious. The form matters
19
+ when a reader could compute with it wrongly: a ratio says 0–1 or percent, a time
20
+ says date or timestamp and its time zone. No table names, join
21
+ keys, SQL functions, or comparison operators: those are `source`.
22
+ Bad: `"Distinct payer banks ... whose currency has a dim.forex rate on their txn_date."`
23
+ Good: `"Number of different paying banks in the last 30 days. It is 0 if there were no collections."`
24
+ 3. **Lead with the measure, in one sentence.** Start with the noun the value is
25
+ ("Number of …", "Total …", "近 30 天…笔数"). Aim for 20 English words or 40 Chinese
26
+ characters. A longer row is usually carrying a table-level fact (principle 1) or
27
+ lineage (principle 4).
28
+ 4. **`source` is lineage in one line, not the SQL.** Upstream `table.column`, the
29
+ aggregation, and only the filters this column adds beyond the table notes. The
30
+ code holds the full predicate; copying it into prose makes a second copy that drifts.
31
+ Bad: `"dwd.transaction where direction = 'COLLECTION_IN' and status = 'POSTED' and cust_type <> 'TEST' and date_sub(dt, 30) <= txn_date < dt, inner-joined to dim.forex ..., count distinct payer_bank."`
32
+ Good: `"count distinct dwd.transaction.payer_bank"`
33
+ 5. **Keep the caveats that change how the value is used; drop the rest.** "Names are
34
+ compared as exact strings" changes a consumer's analysis, so it belongs in the
35
+ row. "Computed with an inner join" does not, unless it changes the number a
36
+ consumer sees, and then it is usually true for every column (principle 1).
37
+ 6. **Put structured facts in their fields.** The exact interval goes in `window` and
38
+ the unit in `unit`; the text says "last 7 days" and "USD" in plain words.
39
+ 7. **Every language says the same thing, written naturally in that language.** Not
40
+ a word-for-word translation, and neither language carries a fact the other lacks.
41
+ 8. **Write sentences, not shorthand: keep the words that carry the logic.** A
42
+ condition keeps its condition words (若…则, 当…时, if, when), a cause keeps 因为 or
43
+ because, and a contrast keeps 但 or but. Dropping them turns a rule into a label
44
+ the reader has to decode. (Decorative connectives such as 值得注意的是 or 综上所述
45
+ still go: they carry no logic.)
46
+ Bad: `"无收款为 0"`, `"0 when none"`, `"空值不计"`
47
+ Good: `"若窗口内无收款,则为 0"`, `"It is 0 if there were no collections."`, `"付款人名称为空的收款不计入"`
48
+ This is the house-style rule against telegraphic writing, applied to definitions.
49
+ 9. **Never restate the name or point at another row.** `"Payer bank count 30d"` and
50
+ `"同上,30 天窗口"` tell the reader nothing.
51
+ 10. **Cover every case.** Say what the column holds for every kind of row (each
52
+ transaction type, each branch of the code), including when it is empty. List
53
+ the values an enumerated column can take, with a Chinese gloss for English codes,
54
+ and put the same list in `values=(...)` so the data can be checked against it.
55
+ A column that is always empty or constant today says so.
56
+ Bad: `"换汇交易为 COMPLETED,内部转账为空。"` (collections and payouts unsaid)
57
+ Good: `"换汇交易为 COMPLETED;收款、付款和内部转账为空。"`
58
+ 11. **Write related columns as a set.** A count and its amount, a value and its
59
+ missing flag, a legacy column and its replacement, two columns that hold the same
60
+ value: give them the same scope and wording, and state any difference in both
61
+ rows.
62
+ Bad: `refund_count` counts refunds up to the snapshot date while `refund_usd`
63
+ silently sums all of them.
64
+ Good: the amount row says "Unlike `refund_count`, includes refunds after the snapshot date."
65
+
66
+ ## Where each fact goes
67
+
68
+ | Fact | Goes in |
69
+ |---|---|
70
+ | What one row is, which rows exist | `Table` `grain_<lang>` |
71
+ | What the table is for, its main source | `Table` language text |
72
+ | Filters, window anchor, currency, time zone shared by all columns | `Table` `notes_<lang>` |
73
+ | Meaning of a term many columns use, list cleaning rules | `Table` `notes_<lang>` |
74
+ | What this column measures, its unit in words, 0/null meaning | `Field` language text |
75
+ | Upstream column, aggregation, filters only this column adds | `Field.source` |
76
+ | Exact interval / unit code | `Field.window` / `Field.unit` |
77
+
78
+ ## A worked example
79
+
80
+ Table:
81
+
82
+ ```python
83
+ CATALOG_TABLE = Table(
84
+ "dwd.customer_daily",
85
+ en="Daily customer collection metrics for risk features.",
86
+ zh="客户每日收款指标,供风险特征使用。",
87
+ grain_en="One row per (dt, cust_id) for each customer with a posted collection before dt.",
88
+ grain_zh="每个 (dt, cust_id) 一行;只有 dt 之前有已入账收款的客户才有行。",
89
+ notes_en=(
90
+ "All metrics count posted collections (status 'POSTED') from non-test customers.",
91
+ "Windows end the day before dt: 'last 7 days' is [dt-7d, dt).",
92
+ "Amounts are converted to USD at the transaction date's rate; collections with no rate are left out of every metric.",
93
+ "Every metric is 0, not empty, if the customer had no collections in its window.",
94
+ ),
95
+ notes_zh=(
96
+ "所有指标只计非测试客户的已入账(status 为 'POSTED')收款。",
97
+ "窗口截至 dt 前一天:「近 7 天」指 [dt-7d, dt),含 dt-7d,不含 dt。",
98
+ "金额按交易日汇率换算为美元;若某笔收款当天没有汇率,则所有指标都不计这笔收款。",
99
+ "若客户在窗口内没有收款,则指标为 0,而不是空值。",
100
+ ),
101
+ )
102
+ ```
103
+
104
+ Rows:
105
+
106
+ ```python
107
+ Field("coll_amt_7d_usd",
108
+ en="Total collected in the last 7 days, in USD.",
109
+ zh="近 7 天收款总额(USD)。",
110
+ source="sum of dwd.transaction.amount × dim.forex.rate_to_usd",
111
+ window="[dt-7d, dt)", unit="USD"),
112
+ Field("payer_name_cnt_30d",
113
+ en="Number of different payer names in the last 30 days, compared as exact text.",
114
+ zh="近 30 天不同付款人名称数,文字精确匹配。",
115
+ source="count distinct dwd.transaction.payer_name (nulls not counted)",
116
+ window="[dt-30d, dt)", unit="count"),
117
+ ```
118
+
119
+ ## Before you commit
120
+
121
+ - Read the page as a consumer: does each row say something the header does not?
122
+ - `audit` reports `repeated-text`, `code-in-description`, and `long-definition`
123
+ for the table you touched. Each is a lead: fix it, or leave it when the text
124
+ really belongs in that row.
@@ -0,0 +1,134 @@
1
+ # Metadata format, config, and commands
2
+
3
+ ## Python table script
4
+
5
+ ```python
6
+ from catalog_fields import Field, Table
7
+
8
+ CATALOG_TABLE = Table(
9
+ "dwd.customer_daily",
10
+ en="Daily customer collection rollup used by risk features.",
11
+ zh="客户每日收款汇总,供风险特征使用。",
12
+ grain_en="One row per (dt, cust_id) for each customer with a posted transaction that day.",
13
+ grain_zh="每个 (dt, cust_id) 一行;只有当天有已入账交易的客户才有行。",
14
+ notes_en=(
15
+ "All metrics count posted collections (status 'POSTED').",
16
+ "Amounts are converted to USD at the transaction date's rate.",
17
+ ),
18
+ notes_zh=(
19
+ "所有指标只计已入账(status 为 'POSTED')的收款。",
20
+ "金额按交易日汇率换算为美元。",
21
+ ),
22
+ )
23
+
24
+ KEYS = (
25
+ Field("dt", en="Snapshot date (Asia/Shanghai).", zh="快照日期(Asia/Shanghai)。",
26
+ source="Run parameter `dt`."),
27
+ )
28
+
29
+ CATALOG_FIELDS = (
30
+ *KEYS,
31
+ Field(
32
+ "coll_amt_7d_usd",
33
+ en="Total collected in the last 7 days, in USD. It is 0 if there were no collections.",
34
+ zh="近 7 天收款总额(USD);若无收款,则为 0。",
35
+ source="sum of dwd_transaction.amount_usd",
36
+ window="[dt-7d, dt)",
37
+ unit="USD",
38
+ ),
39
+ )
40
+
41
+ PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)
42
+ ```
43
+
44
+ - `Field(name, <one keyword per language>, source, window=None, unit=None, values=())`.
45
+ `values` lists the values an enumerated column can take, as distinct strings
46
+ (`values=("是", "否", "无法判定")`); the data-quality skill checks the data against it.
47
+ Never use it on identifiers or free text.
48
+ - `Table(name, <one keyword per language>, grain_<lang>=..., notes_<lang>=(...))`: a
49
+ grain for every language, and the same number of notes in every language (notes are
50
+ optional).
51
+ - Pass only the name positionally; everything else by keyword.
52
+ - The language keywords come from `languages` in `data-catalog.json`. `init`
53
+ regenerates `catalog_fields.py` with exactly those, so a missing language is a
54
+ `TypeError` at run time and an error in `generate`.
55
+ - One table per script. A script with only one of `CATALOG_TABLE`/`CATALOG_FIELDS`
56
+ is an error.
57
+
58
+ ## SQL table script
59
+
60
+ ```sql
61
+ -- @table ads.merchant_monthly
62
+ -- grain_en: One row per (month, merchant_id).
63
+ -- grain_zh: 每个 (month, merchant_id) 一行。
64
+ -- en: Monthly settled volume per merchant.
65
+ -- zh: 商户月度结算金额。
66
+ -- notes_en: Built on the 1st of each month; Counts settled rows (status 'SETTLED') only; Excludes test merchants
67
+ -- notes_zh: 每月 1 日构建;只计已结算(status 为 'SETTLED')的记录;不含测试商户
68
+ -- @field merchant_id
69
+ -- en: Merchant that settled the volume; key of dim_merchant.
70
+ -- zh: 结算该金额的商户,dim_merchant 主键。
71
+ -- source: dim_merchant.merchant_id
72
+ -- @field volume_usd
73
+ -- en: Total settled in the month, in USD. It is 0 if nothing was settled.
74
+ -- zh: 当月结算总额(USD);若无结算,则为 0。
75
+ -- source: sum of dwd_settlement.amount_usd
76
+ -- unit: USD
77
+
78
+ INSERT OVERWRITE TABLE ads.merchant_monthly
79
+ SELECT ...
80
+ ```
81
+
82
+ - A directive line is `-- @table <name>` or `-- @field <name>`.
83
+ - A key line is exactly two spaces after `-- `, then `key: value`. `values:` takes a
84
+ list split on `;` or `;`.
85
+ - A continuation line has four or more spaces and joins the previous key with a space.
86
+ - `notes_<lang>` on `@table` splits on `;` or `;`.
87
+ - The block ends at the first non-comment line. One `@table` per file.
88
+
89
+ ## data-catalog.json
90
+
91
+ | Key | Default | Meaning |
92
+ |---|---|---|
93
+ | `scripts` | required | globs of `.py`/`.sql` table scripts; `**` crosses directories |
94
+ | `exclude` | `[]` | globs to skip (`**/*_legacy.py`) |
95
+ | `output` | `docs/catalog` | where pages go |
96
+ | `languages` | `["en"]` | language keywords every `Field`/`Table` must carry, in column order |
97
+ | `contracts` | `["PUBLISHED_COLUMNS", "OUTPUT_COLUMNS"]` | column-contract names compared with the `Field`s |
98
+ | `fieldsModule` | beside the scripts | where `init` writes `catalog_fields.py` |
99
+ | `tableNames` / `schemaNames` | `[]` | extra variable names the audit treats as the written table / its schema |
100
+
101
+ Hidden directories and `node_modules` are never scanned.
102
+
103
+ ## Commands and exit codes
104
+
105
+ | Command | Exit 0 | Exit 1 | Exit 2 |
106
+ |---|---|---|---|
107
+ | `init [--write]` | planned or written | metadata errors in the first generate | bad config or flags |
108
+ | `generate` | pages written | metadata errors; nothing written | no config |
109
+ | `generate --check` | pages current | stale/orphan pages or metadata errors | no config |
110
+ | `audit [--json] [globs]` | report printed (findings do not fail it) | — | no config and no globs |
111
+ | `drift [--base REF]` | no drift since `REF` (default `HEAD`: uncommitted work) | columns or filters changed while definitions did not | no config, not a git repo, or bad ref |
112
+ | `version` | prints `<version>+<hash>` | — | — |
113
+
114
+ ## What CI checks against the code
115
+
116
+ `generate` (and `--check` in CI) compares the definitions with the columns the
117
+ script outputs, using the first source it finds:
118
+
119
+ | Source of the columns | Missing definition | Definition for a column not output |
120
+ |---|---|---|
121
+ | literal contract tuple (`PUBLISHED_COLUMNS = ("a", ...)`) | error | warning (partition and provenance columns) |
122
+ | contract derived from `CATALOG_FIELDS` | cannot happen | cannot happen |
123
+ | the `INSERT`/`CREATE TABLE AS` SELECT writing the table (`.sql` files, or SQL strings in a `.py` script with no contract) | error | error; a warning instead when the SELECT has `*` or unnamed items |
124
+ | none readable (DataFrame API only) | warning: columns not checked | — |
125
+
126
+ `drift` is the local counterpart for meaning, not run in CI. Against a git base it
127
+ compares each documented script's SELECT expressions and its FROM/WHERE/JOIN/CTE
128
+ text, and reports what changed without a definition changing. With no readable
129
+ SELECT it reports any code change outside the metadata. New scripts are left to
130
+ `generate`. To run it on every commit, add `python3 <engine> drift`
131
+ as a pre-commit hook; `init` does not install one.
132
+
133
+ So a comment block is held to its SELECT: a new SELECT column without `-- @field`
134
+ fails the PR, and so does a `-- @field` for a column the SELECT dropped.