@ketecode/loom 0.0.0-stage → 0.2.0

package/skills/database-migrations/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: database-migrations
+description: Plans and writes zero-downtime schema migrations with the expand-migrate-contract sequence, so columns, tables and indexes can be added, renamed, retyped or dropped while old and new app versions run side by side, with batched backfills, lock and statement timeouts, tested rollbacks, and the conventions of tools such as Flyway, Liquibase, Alembic, Django, Rails, Prisma and Knex. Use when writing a migration against a live database, renaming or dropping a column in production, backfilling a large table, adding a NOT NULL column or foreign key without locking writes, or planning how to roll a schema change back.
+license: MIT
+compatibility: Any Agent Skills host. Generating and running migrations needs the project's migration tool and a non-production database to rehearse on.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: runs-commands
+  kete-loom/owner: "Kete-org"
+---
+
+# Safe database migrations
+
+A migration is safe when every app version that is running during the deploy works with the schema at every step, no statement holds a blocking lock for more than a few seconds, and you know how to go back. This skill is the process; engine details live in `postgres-patterns` and `mysql-patterns`, and SQL recipes in `references/recipes.md`.
+
+## 0. Before you write anything
+
+1. Identify the engine and version, the migration tool (look for `migrations/`, `db/migrate/`, `alembic/`, `prisma/migrations/`, `flyway.conf`, `changelog.xml`) and how migrations run in deploys (before, during or after the new code ships).
+2. Find the table size: row count and on-disk size. A change that is instant on 10k rows can lock a 200M-row table for an hour.
+3. Search the codebase for every reader and writer of the affected tables and columns: models, raw SQL, reports, ETL jobs, other services.
+4. **Stop and ask the user** if any of these is unknown: whether the table is large or hot, whether other services share the database, whether data may be deleted.
+
+## 1. Classify the change
+
+| Change | Safe in one step? | Pattern |
+|---|---|---|
+| Add a nullable column, or one with a constant default | Usually yes (check the engine notes) | single migration |
+| Add a table, add an index built online | Yes | single migration, online index build |
+| Add `NOT NULL` column or constraint to existing rows | No | add nullable, backfill, then enforce |
+| Add a foreign key or check on a big table | No | add unvalidated, then validate separately |
+| Rename a column or table | **No** | expand-contract |
+| Change a column type | **No** (usually a rewrite) | expand-contract with a new column |
+| Drop a column or table | **No** | stop using it in code first, then drop later |
+| Split, merge or move data between tables | **No** | expand-contract plus backfill |
+
+## 2. Expand, migrate, contract
+
+Ship each phase as its own deploy. Do not move on until the previous one is live everywhere.
+
+1. **Expand.** Add the new column, table or index. Nothing reads it yet. Old code keeps working because nothing it uses changed.
+2. **Dual write.** Deploy code that writes both old and new shapes (in the app, or with a trigger if many writers exist). Reads still use the old shape.
+3. **Backfill.** Copy existing rows to the new shape in batches (section 3). Verify with a count of rows where old and new disagree; it must be 0.
+4. **Switch reads.** Deploy code that reads the new shape. Keep writing both, so you can switch back.
+5. **Contract.** Stop writing the old shape, wait at least one deploy cycle, then drop the old column or table in a separate migration.
+
+Renames follow this pattern: add `new_name`, dual write, backfill, read from `new_name`, drop `old_name`. ORMs that cache column lists (Rails, Django, Hibernate) need the column ignored or removed from the model **before** the drop migration runs.
+
+## 3. Backfills
+
+- Never one `UPDATE` over the whole table. It holds row locks, bloats or fills undo/WAL, and stalls replicas.
+- Batch by primary key range, 1k to 10k rows per batch, each batch its own short transaction:
+  ```sql
+  UPDATE orders SET total_minor = round(total * 100)
+  WHERE id > :last_id AND id <= :last_id + 5000 AND total_minor IS NULL;
+  ```
+- Make it idempotent (`WHERE new_col IS NULL`) and resumable (record the last ID), so it can be stopped and restarted.
+- Pause between batches and watch replica lag and database CPU; slow down if lag grows.
+- Run long backfills as a job or script, not inside the schema migration transaction. Many tools wrap each migration in one transaction.
+
+## 4. Locks and timeouts
+
+- Set a short lock timeout before DDL on any live table, so a blocked `ALTER` fails fast instead of queuing all traffic behind it. Postgres: `SET lock_timeout = '3s'`. MySQL: `SET SESSION lock_wait_timeout = 5`.
+- Retry a timed-out DDL with backoff; don't raise the timeout to minutes.
+- Build indexes online: Postgres `CREATE INDEX CONCURRENTLY` (outside a transaction; tell the tool, for example `disable_ddl_transaction!` in Rails, `atomic = False` in Django, `autocommit_block()` in Alembic). MySQL `ALGORITHM=INPLACE, LOCK=NONE`.
+- One risky statement per migration file, so a failure leaves a clear state.
+
+## 5. Rollbacks
+
+1. Write the down path, or state why there isn't one. A drop of data cannot be rolled back by a migration: the rollback is a restore.
+2. Prefer rolling **forward**: in expand-contract the old shape still exists until the contract step, so you roll back the code, not the schema.
+3. Before any destructive step (drop, type change, delete), confirm a recent backup or snapshot exists and that restore has been tested. **Stop and ask the user** to confirm.
+4. Rehearse: run up, down, then up again on a copy with production-like volume. Time each step.
+
+## 6. Tool conventions
+
+- Never edit a migration that has run in any shared environment. Add a new one.
+- Keep schema migrations and data backfills separate. Generated migrations (Django `makemigrations`, Prisma `migrate dev`, Alembic `--autogenerate`) must be read before commit: they can turn a rename into drop-and-add, which loses data.
+- Check what the tool does inside a transaction and with locks. Per-tool notes: `references/tools.md`.
+
+## Never
+
+- Never drop or rename a column that deployed code still reads.
+- Never run a destructive migration without a confirmed backup and the user's explicit go-ahead.
+- Never combine the expand and contract steps in one deploy.
+- Never run a migration against production yourself unless the user asks for that exact action.
+
+## Report back
+
+Return a migration plan in this shape:
+
+```
+Change: <one line>
+Engine / tool: <e.g. PostgreSQL 16 / Alembic>   Table size: <rows, GB>
+Steps:
+  1. [deploy|migration|job] <what> — lock taken, expected duration, rollback
+  2. ...
+Verification: <query that proves the step worked>
+Risks and open questions: <list, or "none">
+```
+
+Then the migration files you wrote and the result of rehearsing them.

package/skills/database-migrations/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "How do I rename a column in production without breaking the app instances still running old code?",
+    "Write a Django migration that adds a NOT NULL column to a table with 50 million rows",
+    "We need to backfill a new column across 80M rows without stalling replication",
+    "Plan the expand and contract steps to split the address fields into their own table",
+    "What's the rollback plan if this Flyway migration that changes a column type fails halfway?"
+  ],
+  "should_not_trigger": [
+    "Review the migrations in this pull request and tell me what's risky",
+    "Which Postgres index type should I use for full-text search on article bodies?",
+    "Our MySQL connection shows utf8mb3, how do I make the client use utf8mb4?",
+    "Set up a GitHub Actions workflow that runs the test suite on every push",
+    "Create a blue-green deployment strategy for our Kubernetes services"
+  ]
+}

package/skills/database-migrations/references/recipes.md

@@ -0,0 +1,98 @@
+# Migration recipes
+
+Each recipe assumes a large, busy table. Set the lock timeout first (Postgres `SET lock_timeout = '3s';`, MySQL `SET SESSION lock_wait_timeout = 5;`). Check the engine docs for your exact version before relying on a "no rewrite" claim.
+
+## Add a column
+
+PostgreSQL: nullable, or with a non-volatile default, is a metadata-only change (since PostgreSQL 11). A volatile default such as `clock_timestamp()`, a stored generated column or an identity column rewrites the table.
+
+```sql
+ALTER TABLE orders ADD COLUMN source text;                 -- instant
+ALTER TABLE orders ADD COLUMN currency text DEFAULT 'GHS'; -- instant, default stored in catalog
+```
+
+MySQL 8: `ALTER TABLE orders ADD COLUMN source VARCHAR(50) NULL, ALGORITHM=INSTANT;`
+
+## Add NOT NULL to an existing column
+
+PostgreSQL (12+), without a long ACCESS EXCLUSIVE scan:
+
+```sql
+ALTER TABLE orders ADD CONSTRAINT orders_source_nn CHECK (source IS NOT NULL) NOT VALID;  -- brief lock
+-- backfill NULLs in batches
+ALTER TABLE orders VALIDATE CONSTRAINT orders_source_nn;  -- scans, SHARE UPDATE EXCLUSIVE: reads and writes continue
+ALTER TABLE orders ALTER COLUMN source SET NOT NULL;      -- skips the scan thanks to the valid CHECK
+ALTER TABLE orders DROP CONSTRAINT orders_source_nn;
+```
+
+MySQL: backfill first, then `ALTER TABLE orders MODIFY source VARCHAR(50) NOT NULL, ALGORITHM=INPLACE, LOCK=NONE;`. Needs strict `sql_mode` and fails if any NULL remains. `MODIFY` replaces the whole column definition: restate the charset, collation, default and comment from `SHOW CREATE TABLE` or they are lost. This rebuilds the table; on a large table expect replica lag, or use `gh-ost` / `pt-online-schema-change`.
+
+## Add a foreign key
+
+PostgreSQL:
+
+```sql
+CREATE INDEX CONCURRENTLY order_items_order_id_idx ON order_items (order_id);
+ALTER TABLE order_items ADD CONSTRAINT order_items_order_id_fkey
+  FOREIGN KEY (order_id) REFERENCES orders (id) NOT VALID;   -- no scan
+ALTER TABLE order_items VALIDATE CONSTRAINT order_items_order_id_fkey;
+```
+
+MySQL: adding a foreign key with `foreign_key_checks` on copies the table; test duration on a copy. Many teams enforce the reference in the app on very large MySQL tables instead; that is a decision for the user.
+
+## Add a unique constraint
+
+PostgreSQL:
+
+```sql
+CREATE UNIQUE INDEX CONCURRENTLY users_email_key ON users (email);
+ALTER TABLE users ADD CONSTRAINT users_email_key UNIQUE USING INDEX users_email_key;
+```
+
+Find duplicates first (`SELECT email, count(*) FROM users GROUP BY 1 HAVING count(*) > 1;`) or the build fails and leaves an INVALID index to drop.
+
+## Add or drop an index
+
+- PostgreSQL: `CREATE INDEX CONCURRENTLY ...` and `DROP INDEX CONCURRENTLY ...`, each outside a transaction block. After a failed build: `DROP INDEX CONCURRENTLY name;` and retry.
+- MySQL: `ALTER TABLE t ADD INDEX name (cols), ALGORITHM=INPLACE, LOCK=NONE;`
+
+## Change a column type (for example int to bigint)
+
+1. Add `id_new bigint` (instant).
+2. Dual write: app or trigger sets `id_new = id` on insert and update.
+3. Backfill in batches; verify `SELECT count(*) FROM t WHERE id_new IS DISTINCT FROM id;` is 0.
+4. Build any index or constraint on `id_new` online.
+5. Make `id_new` NOT NULL without a long scan (the CHECK ... NOT VALID recipe above). Foreign keys in other tables that reference `id` must be recreated against the new key (each as NOT VALID then VALIDATE); plan them as their own steps.
+6. In one short transaction with a lock timeout: rename `id` to `id_old`, `id_new` to `id`, move the default or sequence, swap the primary key with `ADD CONSTRAINT t_pkey PRIMARY KEY USING INDEX <prebuilt unique index>`.
+7. Later, drop `id_old`.
+
+PostgreSQL exception: binary-coercible changes with no `USING` expression, such as `varchar(50)` to `varchar(100)` or `varchar` to `text`, skip the rewrite.
+
+## Rename a column
+
+Expand-contract (see SKILL.md section 2). A database-only `RENAME COLUMN` is instant but breaks every running app instance that still uses the old name; only do it in one step when the user confirms nothing runs during the deploy.
+
+## Drop a column
+
+1. If the column is `NOT NULL` without a default, make it nullable (or give it a default) first, or inserts fail once the code stops writing it. Then remove all reads and writes in code; for ORMs, mark the column ignored (Rails `self.ignored_columns += ["col"]`, Django: remove the field with `SeparateDatabaseAndState` so the column stays) and deploy.
+2. Wait a deploy cycle.
+3. Confirm a backup, then `ALTER TABLE t DROP COLUMN col;`.
+
+## Batched backfill template
+
+```python
+last_id = load_checkpoint() or 0
+while True:
+    rows = db.execute(
+        "UPDATE t SET new_col = old_col WHERE id IN ("
+        " SELECT id FROM t WHERE id > %s AND new_col IS NULL ORDER BY id LIMIT 5000"
+        ") RETURNING id", [last_id]).fetchall()
+    db.commit()
+    if not rows:
+        break
+    last_id = max(r[0] for r in rows)
+    save_checkpoint(last_id)
+    sleep_if_replica_lag_high()
+```
+
+`RETURNING` is PostgreSQL (and MariaDB for `DELETE`/`INSERT`); in MySQL use an ID range per batch instead.

package/skills/database-migrations/references/tools.md

@@ -0,0 +1,21 @@
+# Migration tool notes
+
+Read the tool's docs for the installed version; these are the points that most often cause outages.
+
+| Tool | Generate / apply | Non-transactional statement (Postgres `CONCURRENTLY`) | Watch out for |
+|---|---|---|---|
+| Rails | `bin/rails g migration`, `bin/rails db:migrate` | `disable_ddl_transaction!` plus `add_index ..., algorithm: :concurrently` | `self.ignored_columns` before dropping; `strong_migrations` gem flags unsafe operations |
+| Django | `manage.py makemigrations`, `manage.py migrate` | `atomic = False` on the `Migration`; `AddIndexConcurrently` from `django.contrib.postgres.operations` | `RunPython` backfills run inside the migration transaction by default; `sqlmigrate` shows the SQL |
+| Alembic | `alembic revision --autogenerate`, `alembic upgrade head` | `with op.get_context().autocommit_block():` | autogenerate turns renames into drop and add; review every generated file |
+| Flyway | versioned files `V<n>__desc.sql`, `flyway migrate` | script config `executeInTransaction=false`; on Postgres also check the `postgresql.transactional.lock` setting | never edit an applied file: checksum validation fails |
+| Liquibase | changelog changesets, `liquibase update` | `runInTransaction="false"` on the changeset | each changeset is identified by id + author + file; editing applied ones fails validation |
+| Prisma | `prisma migrate dev` (dev only), `prisma migrate deploy` (CI/prod) | edit the generated `migration.sql` (`--create-only`) and keep a concurrent index alone in its own migration; check your version's docs for transaction behaviour | `migrate dev` can offer to reset the database; never run it against shared data |
+| Knex | `knex migrate:make`, `knex migrate:latest` | `exports.config = { transaction: false }` | |
+| golang-migrate | `migrate create`, `migrate up` | keep a `CONCURRENTLY` statement alone in its file: Postgres runs a multi-statement query string as one implicit transaction | a failed migration marks the version dirty; fix then `migrate force` |
+| EF Core | `dotnet ef migrations add`, `dotnet ef database update` or a generated script | `migrationBuilder.Sql(..., suppressTransaction: true)` | review generated `RenameColumn` vs drop/add; prefer idempotent scripts for production |
+
+## Running migrations in deploys
+
+- Run migrations as a separate step before the new code rolls out, from one place only (a job or init container), never from every app instance at startup.
+- Expand migrations go before the code that needs them; contract migrations go in a later release after the code stopped using the old shape.
+- Keep a lock timeout in the migration session so a blocked migration fails the deploy instead of taking the site down.

package/skills/dbt-patterns/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: dbt-patterns
+description: dbt project conventions for analytics engineering, laying models out in staging, intermediate and marts layers, declaring sources with freshness, using ref and source instead of hard-coded tables, adding data tests, unit tests and enforced model contracts, choosing an incremental strategy, building snapshots for slowly changing history, documenting columns, and running slim CI on only the modified nodes. Use when creating or restructuring a dbt project, writing or refactoring a dbt model, an incremental model gives duplicates, adding dbt tests or snapshots, or setting up dbt CI with state:modified and defer.
+license: MIT
+compatibility: Any Agent Skills host. Needs dbt Core or the dbt Fusion/Cloud CLI with the project's adapter installed. Examples assume dbt 1.8 or later.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "dbt"
+  kete-loom/owner: "Kete-org"
+---
+
+# dbt patterns
+
+Keep a dbt project where every model has one clear job and grain, every table is reached through `ref` or `source`, tests catch broken keys before a dashboard does, and CI builds only what changed.
+
+## Before you start
+
+1. Run `dbt --version` and read `dbt_project.yml`, `packages.yml` and `profiles.yml` location (never print credentials). Note the adapter (Snowflake, BigQuery, Databricks, Postgres, Redshift, DuckDB...), because incremental strategies differ by adapter.
+2. Run `dbt deps` then `dbt parse` (or `dbt compile`) to confirm the project builds before you change anything. If it already fails, **stop and tell the user**.
+3. Find existing conventions (folder names, prefixes, style guide, `sqlfluff` config) and follow them over this skill.
+
+## Layers
+
+| Layer | Folder | Prefix | Job | Materialization |
+|---|---|---|---|---|
+| Staging | `models/staging/<source>/` | `stg_<source>__<entity>` | 1:1 with a source table: rename, cast, light cleanup. No joins, no aggregates. | view (or ephemeral) |
+| Intermediate | `models/intermediate/<area>/` | `int_<entity>_<verb>` | Reusable joins, pivots, fan-outs between staging and marts | ephemeral or view |
+| Marts | `models/marts/<area>/` | `fct_<event>`, `dim_<entity>` | Business-facing tables at a stated grain | table or incremental |
+
+Rules:
+
+- Only staging models call `{{ source('...', '...') }}`. Everything else uses `{{ ref('...') }}`. Never hard-code `database.schema.table`.
+- Write the grain at the top of each mart's YAML description ("one row per order line").
+- Use CTEs: import CTEs (`with orders as (select * from {{ ref('stg_shop__orders') }})`) at the top, logic in the middle, a final `select * from final`.
+- Set materializations per folder in `dbt_project.yml` (`+materialized:`), and override in a model only with a reason.
+
+## Sources
+
+Declare every raw table in `models/staging/<source>/_<source>__sources.yml` with `loaded_at_field` and `freshness` (`warn_after`, `error_after`), then run `dbt source freshness` in production jobs. Examples are in `references/yaml-examples.md`.
+
+## Tests and contracts
+
+- Every model: primary key `unique` and `not_null` under `data_tests:` (the key name since dbt 1.8; older projects use `tests:`).
+- Foreign keys: `relationships`. Enumerations: `accepted_values`. Use `dbt_utils` or `dbt_expectations` only if already installed or the user agrees to add them.
+- Use `severity: warn` or `config: {where: ...}` for known noisy rules; never delete a failing test to make a build pass.
+- **Unit tests** (`unit_tests:` in YAML, dbt 1.8+) for tricky SQL logic: given fixed input rows, expect fixed output rows. Run with `dbt test --select test_type:unit`.
+- **Model contracts** (`config: contract: {enforced: true}` plus every column's `data_type`) on marts that other teams or tools consume. A contract change is a breaking change: tell the user, and consider model `versions`.
+
+## Incremental models
+
+1. Use incremental only when a full rebuild is too slow or costly. Start as `table`.
+2. Choose the strategy the adapter supports:
+   - `merge` with `unique_key` for rows that update in place.
+   - `delete+insert` with `unique_key` where merge is slow or unsupported.
+   - `insert_overwrite` for whole partitions (BigQuery, Spark, Databricks).
+   - `append` only for immutable events with dedup downstream.
+   - `microbatch` (dbt 1.9+) for large time series: set `event_time`, `batch_size`, `begin`, optional `lookback`.
+3. Filter new rows inside `{% if is_incremental() %}`, with a lookback for late data, for example `where updated_at > (select max(updated_at) from {{ this }}) - interval '3 days'` (adjust the syntax for the warehouse).
+4. Set `on_schema_change` (`fail` or `append_new_columns`) rather than relying on the default `ignore`.
+5. Duplicates in an incremental model almost always mean a missing or non-unique `unique_key`, or `append` used for mutable data. Test the key.
+6. **Stop and ask** before running `--full-refresh` on a large production model.
+
+## Snapshots
+
+Use snapshots to keep history of a mutable source (type 2 slowly changing dimension). In dbt 1.9+ define them in YAML (`snapshots:` with `relation`, `config: strategy, unique_key, updated_at`); older projects use `{% snapshot %}` blocks. Prefer `strategy: timestamp` when a reliable `updated_at` exists, else `check` with `check_cols`. Set `hard_deletes` (`ignore`, `invalidate` or `new_record`) deliberately. Snapshot sources, not marts. Never drop a snapshot table: its history cannot be rebuilt.
+
+## Documentation
+
+- Every mart and its columns get `description:` in YAML; reuse text with `{% docs %}` blocks for shared columns.
+- `dbt docs generate` builds the site and lineage graph; check it renders.
+- Add `meta: {owner: ...}` if the project tracks owners.
+
+## Slim CI
+
+On every pull request:
+
+1. Fetch the production `manifest.json` (from the last prod run) into a folder such as `prod-artifacts/`.
+2. Run `dbt build --select state:modified+ --defer --state prod-artifacts/` against a CI schema, so unchanged parents are read from production.
+3. Optionally `dbt clone --select state:modified+,config.materialized:incremental --state prod-artifacts/` first so incremental models start from production data (worth it on warehouses with zero-copy cloning).
+4. Use `dbt build --empty` (1.8+) for a cheap schema-only dry run where data volumes are huge.
+5. Drop the CI schema after the run.
+
+CI config examples are in `references/yaml-examples.md`.
+
+## Never
+
+- Never put credentials in `dbt_project.yml` or the repo; use `env_var()`.
+- Never reference another model by its physical table name.
+- Never change a contracted or widely used mart's columns without telling the user who consumes it.
+- Never run `dbt run` against production from a laptop unless the user asks.
+
+## Report back
+
+Models added or changed (name, layer, grain, materialization), tests and contracts added, the exact dbt commands run with pass, warn and error counts, and any follow-up (full refresh needed, downstream consumers affected).

package/skills/dbt-patterns/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Set up slim CI for our dbt project so pull requests only build modified models",
+    "My dbt incremental model is producing duplicate order rows after each run",
+    "Add a dbt snapshot to track history of the customers table as a type 2 dimension",
+    "Write dbt unit tests and an enforced contract for the fct_orders mart",
+    "Where should joins live in dbt, staging or intermediate models?"
+  ],
+  "should_not_trigger": [
+    "Write an Airflow DAG that extracts from Postgres and loads into BigQuery every hour",
+    "Tune this slow Postgres query that joins orders and customers",
+    "Define what weekly active users means for the product dashboard",
+    "Add a GitHub Actions workflow that runs our Python unit tests on every push",
+    "Migrate the production users table to add a non-null column without downtime"
+  ]
+}

package/skills/dbt-patterns/references/yaml-examples.md

@@ -0,0 +1,114 @@
+# dbt YAML examples
+
+Check key names against the docs for the installed dbt version (docs.getdbt.com).
+
+## Source with freshness
+
+```yaml
+# models/staging/shop/_shop__sources.yml
+sources:
+  - name: shop
+    database: raw
+    schema: shop
+    loaded_at_field: _loaded_at
+    freshness:
+      warn_after: {count: 12, period: hour}
+      error_after: {count: 24, period: hour}
+    tables:
+      - name: orders
+      - name: customers
+```
+
+In newer dbt versions these keys live under `config:` (`freshness` from v1.9, `loaded_at_field` from v1.10), and v1.10 raises deprecation warnings for properties that moved to configs. Follow what the project's dbt version and existing files do. Sources: https://docs.getdbt.com/reference/resource-properties/freshness and https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10
+
+## Model with tests and a contract
+
+```yaml
+# models/marts/sales/_sales__models.yml
+models:
+  - name: fct_orders
+    description: One row per order.
+    config:
+      contract: {enforced: true}
+    columns:
+      - name: order_id
+        data_type: varchar
+        data_tests: [unique, not_null]
+      - name: customer_id
+        data_type: varchar
+        data_tests:
+          - relationships:
+              to: ref('dim_customers')
+              field: customer_id
+      - name: status
+        data_type: varchar
+        data_tests:
+          - accepted_values:
+              values: ['placed', 'shipped', 'returned']
+      - name: amount
+        data_type: numeric
+```
+
+From dbt v1.10.5, generic test arguments (`to`, `field`, `values`) can be nested under `arguments:`; older versions take them at the top level as shown. See https://docs.getdbt.com/reference/resource-properties/data-tests
+
+## Unit test
+
+```yaml
+unit_tests:
+  - name: test_order_amount_excludes_refunds
+    model: fct_orders
+    given:
+      - input: ref('stg_shop__orders')
+        rows:
+          - {order_id: 'a', amount: 100, is_refund: false}
+          - {order_id: 'b', amount: 40, is_refund: true}
+    expect:
+      rows:
+        - {order_id: 'a', amount: 100}
+```
+
+## Incremental model (merge)
+
+```sql
+{{ config(
+    materialized='incremental',
+    unique_key='order_id',
+    incremental_strategy='merge',
+    on_schema_change='append_new_columns'
+) }}
+
+with orders as (select * from {{ ref('stg_shop__orders') }})
+
+select * from orders
+{% if is_incremental() %}
+where updated_at > (select max(updated_at) from {{ this }}) - interval '3 days'
+{% endif %}
+```
+
+## Snapshot (dbt 1.9+)
+
+```yaml
+# snapshots/customers_snapshot.yml
+snapshots:
+  - name: customers_snapshot
+    relation: source('shop', 'customers')
+    config:
+      schema: snapshots
+      unique_key: customer_id
+      strategy: timestamp
+      updated_at: updated_at
+      hard_deletes: invalidate
+```
+
+## Slim CI job (GitHub Actions outline)
+
+```yaml
+- run: pip install dbt-core dbt-snowflake   # pin versions the project uses
+- run: dbt deps
+- run: ./scripts/fetch-prod-manifest.sh prod-artifacts/   # however the team stores it
+- run: dbt build --select state:modified+ --defer --state prod-artifacts/ --target ci
+  env:
+    DBT_CI_SCHEMA: pr_${{ github.event.number }}
+```
+
+The CI target in `profiles.yml` must read that variable (`schema: "{{ env_var('DBT_CI_SCHEMA') }}"`), or the build lands in the default schema. Profiles for CI read credentials with `{{ env_var('SNOWFLAKE_PASSWORD') }}` from CI secrets.

package/skills/decision-council/SKILL.md

@@ -0,0 +1,98 @@
+---
+name: decision-council
+description: Runs a council of independent viewpoints on one hard call, giving several role-played advisers or separate models (or providers) the same brief in isolation so none sees another's answer, then cross-examining their positions and synthesising a single recommendation that records the strongest dissent, the confidence level and what evidence would change the outcome. Use when facing an ambiguous or contested choice, a go/no-go or launch call, a build-versus-buy or vendor pick, or whenever the user asks for a second opinion, a devil's advocate, a panel or a debate before committing.
+license: MIT
+compatibility: Any Agent Skills host. Works with role-played perspectives in one session; independent subagents or a second model or provider give stronger independence where the host supports them.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: ai
+  kete-loom/risk: read-only
+  kete-loom/owner: "Kete-org"
+---
+
+# Decision council
+
+One model asked once tends to agree with the framing it was given. A council gets several independent answers to the same brief, then weighs them openly, so the decision shows its disagreements instead of hiding them.
+
+Use it for decisions, not for tasks with a checkable answer (run the test instead), and not for code review (use a reviewer agent or `/dual-review`). To write up the outcome as a permanent record, hand the result to `architecture-decision-records`.
+
+## 1. Frame the decision
+
+Write a brief of at most one page and show it to the user before convening anyone:
+
+- **Question**: one sentence, phrased as a choice ("Ship v2 on 1 March, or slip to April?").
+- **Options**: two to four concrete options, including "do nothing" or "delay" where real.
+- **Criteria**: what matters, in priority order (cost, risk, time, reversibility, user impact, compliance).
+- **Facts**: the evidence available, with sources. Mark assumptions as assumptions.
+- **Constraints**: deadlines, budgets, non-negotiables.
+- **Reversibility**: how hard the choice is to undo. Easily reversible decisions rarely need a council; say so and stop if this is one.
+
+Keep the brief neutral. Don't state your own lean, and don't order options so the preferred one comes first or last every time.
+
+If the question or the options are unclear, stop and ask the user rather than guessing.
+
+## 2. Choose the members
+
+Pick three to five members. Choose perspectives that would genuinely disagree:
+
+- **Roles**, for example: the operator who must run it, the customer, the finance owner, the security or compliance reviewer, the skeptic whose job is to argue against the leading option, the engineer who builds it.
+- **Different models or providers**, where available, to reduce shared blind spots. Note the model name for each member.
+- Always include one member assigned to argue the strongest case against whatever looks like the default option.
+
+For regulated, legal, medical or financial decisions, a role-played expert is not a qualified opinion. Say this in the result and name who must actually review it.
+
+## 3. Keep them independent
+
+Independence is the whole point; protect it:
+
+1. Give every member the same brief, word for word, plus their role. Don't pass any member another's answer in this round.
+2. Run members in separate contexts (separate subagents, separate sessions or separate model calls) when the host allows. If only one context is available, write each member's answer in full before starting the next, and don't revise earlier answers.
+3. Ask each member for the same fixed format:
+
+```
+Position: <option>
+Confidence: low | medium | high
+Top 3 reasons: ...
+Biggest risk of my position: ...
+What would change my mind: ...
+```
+
+4. Don't let members search for or cite each other. Facts they introduce must come with a source; unsourced claims are flagged in synthesis.
+
+## 4. Cross-examine
+
+One round only, to stop the council drifting into consensus by attrition:
+
+- Show each member the other positions, anonymised (Member A, B, C).
+- Ask each: which argument against your position is strongest, and does it change your position? Record any change and why.
+- Ask the assigned skeptic to attack the current majority view.
+
+## 5. Synthesise
+
+You, as chair, write the recommendation. Rules:
+
+- Weigh arguments against the criteria from the brief, not by headcount. A single member with a decisive fact beats a majority without one.
+- Check that every factual claim the recommendation relies on is in the brief or sourced. Drop or flag the rest.
+- Keep dissent visible. Never smooth a real disagreement into "all members broadly agree".
+- Confidence is about the decision, not the members: high only if the main arguments survived cross-examination and the facts are solid.
+- If the council splits evenly or confidence is low, say so and name the one piece of evidence that would settle it, instead of forcing a pick.
+
+## Never
+
+- Never present the council's output as the user's decision; the user decides.
+- Never invent expert credentials, data or quotes for a member.
+- Never rerun the council until you get the answer you or the user wanted; if new facts arrive, revise the brief and say what changed.
+
+## Report
+
+```
+## Decision: <question>
+Recommendation: <option>   Confidence: low | medium | high
+Why: <three bullet points tied to the criteria>
+Members: <role or model> → <position> (<confidence>), one line each; mark changes after cross-examination
+Strongest dissent: <argument, who holds it, why it did not prevail>
+Key risks and mitigations: <bullets>
+Would change the call: <specific evidence or event>
+Assumptions: <list>
+Needs qualified review: <who, or "none">
+```

package/skills/decision-council/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "We can't agree whether to launch the new pricing on Monday or wait a month. Get me a few independent perspectives and a recommendation",
+    "Should we build our own auth or buy Auth0? I want a panel of viewpoints, including a devil's advocate",
+    "Ask two different models separately whether we should sunset the legacy API this quarter, then reconcile their answers",
+    "Give me a go/no-go call on the Ghana market expansion with the dissenting view recorded",
+    "Convene a council: rewrite the billing service in Go, or keep patching the Rails one?"
+  ],
+  "should_not_trigger": [
+    "Write an ADR recording that we chose Postgres over MongoDB last week",
+    "Have two reviewers check my pull request until both approve it",
+    "Compare three monolith versus microservice architectures for our new platform and design the module boundaries",
+    "Estimate how long the payments migration will take and break it into tasks",
+    "Score these model outputs with an LLM judge and give me a pass rate"
+  ]
+}