@1aboveio/skills 0.0.0-stage → 1.0.0

package/skills/app-dev/data-catalog/references/field-definitions.md

@@ -0,0 +1,116 @@
+# Writing field definitions
+
+A reader opens a table's page to learn what each value means and whether they can
+use it. Write for that reader: the table header once, then one short line per
+column that says only what is particular to that column.
+
+## Principles
+
+1. **Say each fact once, at the widest level where it is true.** A filter, window
+   anchor, currency rule, or row-inclusion rule that holds for every column goes in
+   the table's `grain` or `notes`, never in each row. So does the meaning of a term
+   many columns use ("personal", "platform", "own name") and how lists are cleaned
+   (case, sorting, empty items). A row carries only what differs for that column.
+   Bad (every row): `"Posted collections only, test customers excluded, [dt-7d, dt), excluding dt; ..."`
+   Good: table note `"All metrics count posted collections from non-test customers before dt."`,
+   row `"Number of collections in the last 7 days. It is 0 if there were no collections."`
+2. **The description is meaning, in business words.** What one value measures, its
+   unit and form, and what 0 or null means when that is not obvious. The form matters
+   when a reader could compute with it wrongly: a ratio says 0–1 or percent, a time
+   says date or timestamp and its time zone. No table names, join
+   keys, SQL functions, or comparison operators: those are `source`.
+   Bad: `"Distinct payer banks ... whose currency has a dim.forex rate on their txn_date."`
+   Good: `"Number of different paying banks in the last 30 days. It is 0 if there were no collections."`
+3. **Lead with the measure, in one sentence.** Start with the noun the value is
+   ("Number of …", "Total …", "近 30 天…笔数"). Aim for 20 English words or 40 Chinese
+   characters. A longer row is usually carrying a table-level fact (principle 1) or
+   lineage (principle 4).
+4. **`source` is lineage in one line, not the SQL.** Upstream `table.column`, the
+   aggregation, and only the filters this column adds beyond the table notes. The
+   code holds the full predicate; copying it into prose makes a second copy that drifts.
+   Bad: `"dwd.transaction where direction = 'COLLECTION_IN' and status = 'POSTED' and cust_type <> 'TEST' and date_sub(dt, 30) <= txn_date < dt, inner-joined to dim.forex ..., count distinct payer_bank."`
+   Good: `"count distinct dwd.transaction.payer_bank"`
+5. **Keep the caveats that change how the value is used; drop the rest.** "Names are
+   compared as exact strings" changes a consumer's analysis, so it belongs in the
+   row. "Computed with an inner join" does not, unless it changes the number a
+   consumer sees, and then it is usually true for every column (principle 1).
+6. **Put structured facts in their fields.** The exact interval goes in `window` and
+   the unit in `unit`; the text says "last 7 days" and "USD" in plain words.
+7. **Every language says the same thing, written naturally in that language.** Not
+   a word-for-word translation, and neither language carries a fact the other lacks.
+8. **Write sentences, not shorthand: keep the words that carry the logic.** A
+   condition keeps its condition words (若…则, 当…时, if, when), a cause keeps 因为 or
+   because, and a contrast keeps 但 or but. Dropping them turns a rule into a label
+   the reader has to decode. (Decorative connectives such as 值得注意的是 or 综上所述
+   still go: they carry no logic.)
+   Bad: `"无收款为 0"`, `"0 when none"`, `"空值不计"`
+   Good: `"若窗口内无收款，则为 0"`, `"It is 0 if there were no collections."`, `"付款人名称为空的收款不计入"`
+   This is the house-style rule against telegraphic writing, applied to definitions.
+9. **Never restate the name or point at another row.** `"Payer bank count 30d"` and
+   `"同上，30 天窗口"` tell the reader nothing.
+10. **Cover every case.** Say what the column holds for every kind of row (each
+    transaction type, each branch of the code), including when it is empty. List
+    the values an enumerated column can take, with a Chinese gloss for English codes.
+    A column that is always empty or constant today says so.
+    Bad: `"换汇交易为 COMPLETED，内部转账为空。"` (collections and payouts unsaid)
+    Good: `"换汇交易为 COMPLETED；收款、付款和内部转账为空。"`
+11. **Write related columns as a set.** A count and its amount, a value and its
+    missing flag, a legacy column and its replacement, two columns that hold the same
+    value: give them the same scope and wording, and state any difference in both
+    rows.
+    Bad: `refund_count` counts refunds up to the snapshot date while `refund_usd`
+    silently sums all of them.
+    Good: the amount row says "Unlike `refund_count`, includes refunds after the snapshot date."
+
+## Where each fact goes
+
+| Fact | Goes in |
+|---|---|
+| What one row is, which rows exist | `Table.grain` |
+| What the table is for, its main source | `Table` language text |
+| Filters, window anchor, currency, time zone shared by all columns | `Table.notes` |
+| Meaning of a term many columns use, list cleaning rules | `Table.notes` |
+| What this column measures, its unit in words, 0/null meaning | `Field` language text |
+| Upstream column, aggregation, filters only this column adds | `Field.source` |
+| Exact interval / unit code | `Field.window` / `Field.unit` |
+
+## A worked example
+
+Table:
+
+```python
+CATALOG_TABLE = Table(
+    "dwd.customer_daily",
+    grain="One row per (dt, cust_id) for each customer with a posted collection before dt.",
+    en="Daily customer collection metrics for risk features.",
+    zh="客户每日收款指标，供风险特征使用。",
+    notes=(
+        "All metrics count posted collections (status 'POSTED') from non-test customers.",
+        "Windows end the day before dt: 'last 7 days' is [dt-7d, dt).",
+        "Amounts are converted to USD at the transaction date's rate; collections with no rate are left out of every metric.",
+        "Every metric is 0, not empty, if the customer had no collections in its window.",
+    ),
+)
+```
+
+Rows:
+
+```python
+Field("coll_amt_7d_usd",
+      en="Total collected in the last 7 days, in USD.",
+      zh="近 7 天收款总额（USD）。",
+      source="sum of dwd.transaction.amount × dim.forex.rate_to_usd",
+      window="[dt-7d, dt)", unit="USD"),
+Field("payer_name_cnt_30d",
+      en="Number of different payer names in the last 30 days, compared as exact text.",
+      zh="近 30 天不同付款人名称数，文字精确匹配。",
+      source="count distinct dwd.transaction.payer_name (nulls not counted)",
+      window="[dt-30d, dt)", unit="count"),
+```
+
+## Before you commit
+
+- Read the page as a consumer: does each row say something the header does not?
+- `audit` reports `repeated-text`, `code-in-description`, and `long-definition`
+  for the table you touched. Each is a lead: fix it, or leave it when the text
+  really belongs in that row.

package/skills/app-dev/data-catalog/references/format.md

@@ -0,0 +1,121 @@
+# Metadata format, config, and commands
+
+## Python table script
+
+```python
+from catalog_fields import Field, Table
+
+CATALOG_TABLE = Table(
+    "dwd.customer_daily",
+    grain="One row per (dt, cust_id) for each customer with a posted transaction that day.",
+    en="Daily customer collection rollup used by risk features.",
+    zh="客户每日收款汇总，供风险特征使用。",
+    notes=(
+        "All metrics count posted collections (status 'POSTED').",
+        "Amounts are converted to USD at the transaction date's rate.",
+    ),
+)
+
+KEYS = (
+    Field("dt", en="Snapshot date (Asia/Shanghai).", zh="快照日期（Asia/Shanghai）。",
+          source="Run parameter `dt`."),
+)
+
+CATALOG_FIELDS = (
+    *KEYS,
+    Field(
+        "coll_amt_7d_usd",
+        en="Total collected in the last 7 days, in USD. It is 0 if there were no collections.",
+        zh="近 7 天收款总额（USD）；若无收款，则为 0。",
+        source="sum of dwd_transaction.amount_usd",
+        window="[dt-7d, dt)",
+        unit="USD",
+    ),
+)
+
+PUBLISHED_COLUMNS = tuple(f.name for f in CATALOG_FIELDS)
+```
+
+- `Field(name, <one keyword per language>, source, window=None, unit=None)`.
+- `Table(name, grain, <one keyword per language>, notes=())`.
+- Pass only the name positionally; everything else by keyword.
+- The language keywords come from `languages` in `data-catalog.json`. `init`
+  regenerates `catalog_fields.py` with exactly those, so a missing language is a
+  `TypeError` at run time and an error in `generate`.
+- One table per script. A script with only one of `CATALOG_TABLE`/`CATALOG_FIELDS`
+  is an error.
+
+## SQL table script
+
+```sql
+-- @table ads.merchant_monthly
+--   grain: One row per (month, merchant_id).
+--   en: Monthly settled volume per merchant.
+--   zh: 商户月度结算金额。
+--   notes: Built on the 1st of each month; Counts settled rows (status 'SETTLED') only; Excludes test merchants
+-- @field merchant_id
+--   en: Merchant that settled the volume; key of dim_merchant.
+--   zh: 结算该金额的商户，dim_merchant 主键。
+--   source: dim_merchant.merchant_id
+-- @field volume_usd
+--   en: Total settled in the month, in USD. It is 0 if nothing was settled.
+--   zh: 当月结算总额（USD）；若无结算，则为 0。
+--   source: sum of dwd_settlement.amount_usd
+--   unit: USD
+
+INSERT OVERWRITE TABLE ads.merchant_monthly
+SELECT ...
+```
+
+- A directive line is `-- @table <name>` or `-- @field <name>`.
+- A key line is exactly two spaces after `-- `, then `key: value`.
+- A continuation line has four or more spaces and joins the previous key with a space.
+- `notes` on `@table` splits on `;`.
+- The block ends at the first non-comment line. One `@table` per file.
+
+## data-catalog.json
+
+| Key | Default | Meaning |
+|---|---|---|
+| `scripts` | required | globs of `.py`/`.sql` table scripts; `**` crosses directories |
+| `exclude` | `[]` | globs to skip (`**/*_legacy.py`) |
+| `output` | `docs/catalog` | where pages go |
+| `languages` | `["en"]` | language keywords every `Field`/`Table` must carry, in column order |
+| `contracts` | `["PUBLISHED_COLUMNS", "OUTPUT_COLUMNS"]` | column-contract names compared with the `Field`s |
+| `fieldsModule` | beside the scripts | where `init` writes `catalog_fields.py` |
+| `tableNames` / `schemaNames` | `[]` | extra variable names the audit treats as the written table / its schema |
+
+Hidden directories and `node_modules` are never scanned.
+
+## Commands and exit codes
+
+| Command | Exit 0 | Exit 1 | Exit 2 |
+|---|---|---|---|
+| `init [--write]` | planned or written | metadata errors in the first generate | bad config or flags |
+| `generate` | pages written | metadata errors; nothing written | no config |
+| `generate --check` | pages current | stale/orphan pages or metadata errors | no config |
+| `audit [--json] [globs]` | report printed (findings do not fail it) | — | no config and no globs |
+| `drift [--base REF]` | no drift since `REF` (default `HEAD`: uncommitted work) | columns or filters changed while definitions did not | no config, not a git repo, or bad ref |
+| `version` | prints `<version>+<hash>` | — | — |
+
+## What CI checks against the code
+
+`generate` (and `--check` in CI) compares the definitions with the columns the
+script outputs, using the first source it finds:
+
+| Source of the columns | Missing definition | Definition for a column not output |
+|---|---|---|
+| literal contract tuple (`PUBLISHED_COLUMNS = ("a", ...)`) | error | warning (partition and provenance columns) |
+| contract derived from `CATALOG_FIELDS` | cannot happen | cannot happen |
+| the `INSERT`/`CREATE TABLE AS` SELECT writing the table (`.sql` files, or SQL strings in a `.py` script with no contract) | error | error; a warning instead when the SELECT has `*` or unnamed items |
+| none readable (DataFrame API only) | warning: columns not checked | — |
+
+`drift` is the local counterpart for meaning, not run in CI. Against a git base it
+compares each documented script's SELECT expressions and its FROM/WHERE/JOIN/CTE
+text, and reports what changed without a definition changing. With no readable
+SELECT it reports any code change outside the metadata. New scripts are left to
+`generate`. To run it on every commit, add `python3 <engine> drift`
+as a pre-commit hook; `init` does not install one.
+
+So a comment block is held to its SELECT: a new SELECT column without `-- @field`
+fails the PR, and so does a `-- @field` for a column the SELECT dropped.

package/skills/app-dev/data-catalog/release-check.mjs

@@ -0,0 +1,93 @@
+#!/usr/bin/env node
+import { spawnSync } from 'node:child_process';
+import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { homedir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+const WEEK_MS = 7 * DAY_MS;
+const here = dirname(fileURLToPath(import.meta.url));
+
+function readJson(path, fallback = {}) {
+  try { return JSON.parse(readFileSync(path, 'utf8')); } catch { return fallback; }
+}
+
+function frontmatterValue(text, key) {
+  const frontmatter = /^---\n([\s\S]*?)\n---/.exec(text)?.[1] ?? '';
+  const match = new RegExp('^' + key + ':\\s*([^\\n]+)$', 'm').exec(frontmatter);
+  return match?.[1].trim().replace(/^['"]|['"]$/g, '') ?? null;
+}
+
+function repositoryOf(source) {
+  const value = String(source ?? '').trim();
+  const match = /^(?:git@github\.com:|https:\/\/github\.com\/)?([A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+?)(?:\.git)?$/.exec(value);
+  return match?.[1] ?? null;
+}
+
+function stableVersion(value) {
+  const match = /^v?(\d+)\.(\d+)\.(\d+)$/.exec(String(value ?? '').trim());
+  return match ? match.slice(1).map(Number) : null;
+}
+
+function newer(left, right) {
+  for (let index = 0; index < 3; index += 1) {
+    if (left[index] !== right[index]) return left[index] > right[index];
+  }
+  return false;
+}
+
+function writeCache(path, value) {
+  try {
+    mkdirSync(dirname(path), { recursive: true });
+    writeFileSync(path, JSON.stringify(value));
+  } catch { /* A read-only cache must not block skill use. */ }
+}
+
+function main(now = Date.now()) {
+  if (process.env.ONEABOVEIO_NO_UPDATE_CHECK) return;
+  const skill = readFileSync(join(here, 'SKILL.md'), 'utf8');
+  const name = frontmatterValue(skill, 'name');
+  const localVersion = stableVersion(frontmatterValue(skill, 'release-version'));
+  if (!name || !localVersion) return;
+
+  const lock = readJson(join(process.env.HOME || homedir(), '.agents', '.skill-lock.json'));
+  const entry = lock.skills?.[name];
+  const repository = repositoryOf(entry?.sourceUrl || entry?.source);
+  if (!repository) return;
+
+  const cacheRoot = process.env.XDG_CACHE_HOME || join(process.env.HOME || homedir(), '.cache');
+  const cachePath = join(cacheRoot, '1aboveio-skills', 'release-check.json');
+  const all = readJson(cachePath);
+  const cache = all[repository] && typeof all[repository] === 'object' ? all[repository] : {};
+
+  if (!cache.lastCheck || now - cache.lastCheck >= DAY_MS) {
+    const result = spawnSync('gh', ['api', 'repos/' + repository + '/releases/latest', '--jq', '.tag_name'], {
+      encoding: 'utf8',
+      stdio: ['ignore', 'pipe', 'ignore'],
+      timeout: 1500,
+    });
+    cache.lastCheck = now;
+    if (result.status === 0 && stableVersion(result.stdout)) cache.latestTag = result.stdout.trim();
+    else delete cache.latestTag;
+    all[repository] = cache;
+    writeCache(cachePath, all);
+  }
+
+  const latestVersion = stableVersion(cache.latestTag);
+  if (!latestVersion || !newer(latestVersion, localVersion)) return;
+  if (cache.notifiedTag === cache.latestTag && cache.notifiedAt && now - cache.notifiedAt < WEEK_MS) return;
+
+  cache.notifiedTag = cache.latestTag;
+  cache.notifiedAt = now;
+  all[repository] = cache;
+  writeCache(cachePath, all);
+  const command = `npx skills@1.5.23 add '${repository}#${cache.latestTag}' --global --agent universal claude-code --skill ${name} --yes`;
+  process.stdout.write(
+    `UPDATE_AVAILABLE: A newer 1AboveIO skills release (${cache.latestTag}) is available; this skill is `
+      + `v${localVersion.join('.')}. Ask once whether to refresh ${name}. On approval run \`${command}\`. `
+      + 'The refreshed instructions apply next session. Either way, continue the current task and do not ask again this session.\n',
+  );
+}
+
+try { main(); } catch { /* Update checks are advisory and always fail silent. */ }

package/skills/app-dev/data-catalog/scripts/data-catalog.test.mjs

@@ -0,0 +1,36 @@
+import assert from 'node:assert/strict';
+import { spawnSync } from 'node:child_process';
+import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import test from 'node:test';
+
+const scriptsDir = dirname(fileURLToPath(import.meta.url));
+const conventions = readFileSync(join(scriptsDir, '../references/data-catalog-conventions.md'), 'utf8');
+
+test('data-catalog test_data_catalog.py', () => {
+  const result = spawnSync('python3', [join(scriptsDir, 'test_data_catalog.py')], { encoding: 'utf8' });
+  assert.equal(result.status, 0, `${result.stdout}\n${result.stderr}`);
+});
+
+test('persist-conventions writes the Data catalog section only with --write', () => {
+  const dir = mkdtempSync(join(tmpdir(), 'data-catalog-conventions-'));
+  try {
+    const cli = join(scriptsDir, 'persist-conventions.mjs');
+    const preview = spawnSync(process.execPath, [cli, '--repo-root', dir], { encoding: 'utf8' });
+    assert.match(preview.stdout, /preview only, 3 file\(s\) would change/);
+    assert.equal(spawnSync(process.execPath, [cli, '--repo-root', dir, '--write'], { encoding: 'utf8' }).status, 0);
+    const text = readFileSync(join(dir, 'CONTRIBUTING.md'), 'utf8');
+    assert.match(text, /<!-- conventions:data-catalog:start -->[\s\S]*## Data catalog conventions[\s\S]*<!-- conventions:data-catalog:end -->/);
+  } finally {
+    rmSync(dir, { recursive: true, force: true });
+  }
+});
+
+test('every engine command the conventions tell people to run exists', () => {
+  const cited = [...conventions.matchAll(/data_catalog\.py (\w+)/g)].map((m) => m[1]);
+  assert.deepEqual([...new Set(cited)].sort(), ['audit', 'drift', 'generate']);
+  const help = spawnSync('python3', [join(scriptsDir, 'data_catalog.py'), '--help'], { encoding: 'utf8' }).stdout;
+  for (const command of cited) assert.match(help, new RegExp(`\\b${command}\\b`), `${command} missing from the engine`);
+});