vibecodeagent 0.0.0-stage → 0.1.65

package/dist/resources/skills/builtin/backend-engineering/references/data-modelling.md

@@ -0,0 +1,367 @@
+# Data Modelling
+
+This file owns the data at rest: key choice, column types, nullability, indexes, constraints, denormalisation, and finding N+1. Changing a schema that already holds rows is [migrations.md](migrations.md) — every recipe here that touches a live table hands off there. Query plans, pool sizing, and latency budgets are [testing-and-perf.md](testing-and-perf.md). Row locks, transaction boundaries, and "what if two of these run at once" are [concurrency-and-jobs.md](concurrency-and-jobs.md).
+
+Postgres is the assumed engine, per the hub's boring-choice rule. Where InnoDB differs materially, it is called out.
+
+## Operating Posture
+
+Two failure modes, and the first is far worse:
+
+1. **Wrong in a way only rows reveal.** Money as `float`, naive `timestamp`, a random UUID primary key on the table that will be biggest. This passes review, ships, works beautifully for six months, and then the fix is a [migrations.md](migrations.md) problem on 40 million rows holding a lock.
+2. **Right, but queried wrong.** N+1, a composite index in the wrong order, a missing index on a foreign key. Painful, embarrassing, and fixable in one deploy without touching a single row.
+
+Optimise the first at design time. You get one cheap shot at it.
+
+**The best schema change is often no schema change.** A request for a new table that is really a column, a JSON blob that is really a query, an index that duplicates the prefix of an index you already have — say so and add nothing. An index that `(tenant_id, status, created_at)` already covers is pure write tax; adding it is a defect, not a precaution.
+
+## 1. Key Choice
+
+| Situation | Key |
+| --- | --- |
+| Internal id, never exposed, largest tables | `bigint GENERATED ALWAYS AS IDENTITY` |
+| Id appears in a URL or is handed to a client | `uuidv7` / ULID stored as `uuid` |
+| Id must be generated client-side or offline, before the insert | `uuidv7` / ULID |
+| Id must be merged across shards or services with no coordination | `uuidv7` / ULID |
+| Existing table already on `uuidv4`, working, under ~5M rows | Leave it. This is not the bug you have. |
+| Anything | **Never** `serial` / `int4` for a growing PK |
+
+```sql
+-- internal, monotonic, 8 bytes
+id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY
+
+-- public, opaque, time-ordered (Postgres 18+ has uuidv7() built in;
+-- below that, generate in the app or install pg_uuidv7)
+id uuid PRIMARY KEY DEFAULT uuidv7()
+```
+
+`int4` tops out at 2,147,483,647. A `serial` PK on an events or audit table reaches that, and the failure mode is every insert erroring at once, in production, on a table too big to `ALTER` quickly. Use `bigint` from line one; 8 bytes is not the constraint. Prefer `GENERATED ALWAYS AS IDENTITY` (SQL standard, sequence owned by the column, no accidental manual writes) over `serial`.
+
+### Why uuidv4 hurts, concretely
+
+A B-tree index is a set of 8 KB pages held in sorted key order. Sequential keys always insert into the rightmost leaf: one hot page, in cache, appended to.
+
+`uuidv4` is 122 random bits. Every insert lands in a uniformly random leaf. Three costs compound once the index exceeds `shared_buffers`:
+
+- **Random reads on write.** Each insert must fetch a cold leaf page from disk before it can modify it. Inserts become read-bound.
+- **Page splits and half-empty pages.** Postgres splits a full leaf 50/50 on a random insert but does a rightmost 90/10 split on a monotonic one. Random keys therefore settle around 50–70% fill, so the index is roughly 1.5–2× the size it would otherwise be, and every extra page is another cache miss.
+- **WAL amplification.** A split is a full-page image in WAL. More WAL means more fsync, more replication traffic, and a longer recovery.
+
+**InnoDB is worse**, and this matters if the stack is MySQL: the primary key *is* the clustered index, so a random PK scatters the table rows themselves, not just an index. Secondary indexes also store the PK as their row pointer, so a 16-byte random PK inflates every secondary index too.
+
+`uuidv7` (RFC 9562, which obsoletes RFC 4122) fixes exactly this: the first 48 bits are a big-endian Unix millisecond timestamp, so keys generated near in time sort near each other and inserts return to the right-hand edge. ULID has the same 48-bit-timestamp-then-random layout. Store a ULID in a `uuid` column (16 bytes, both are 128-bit) — never as `char(26)`, which costs more space and a collation-dependent comparison on every index probe.
+
+### When a public sequential id is itself the leak
+
+`bigint` is the better key until the id becomes visible, then two things go wrong:
+
+- **Enumeration.** `/api/invoices/1042` invites `/api/invoices/1043`. Authorization must stop that regardless (hub Boundary Checklist #1 — a sequential id is not an authorization bug, but it makes every authorization bug trivially exploitable and scriptable).
+- **Business-volume disclosure.** Two signups a week apart with ids 8,120 and 8,470 tell a competitor you onboarded 350 customers that week. This is the German tank problem and it needs no vulnerability at all.
+
+**The hybrid is usually right:** `bigint` PK for joins and foreign keys, plus a separate `public_id uuid NOT NULL UNIQUE DEFAULT uuidv7()` (or a slug) as the only id that ever crosses the API boundary. You get index locality internally and opacity externally, and the FK columns stay 8 bytes.
+
+## 2. Column Types
+
+### Money
+
+```sql
+CREATE TABLE payments (
+  id            bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+  amount_minor  bigint      NOT NULL,          -- 1234 = 12.34 USD, = 1234 JPY
+  currency      char(3)     NOT NULL,          -- ISO 4217 alpha code, uppercase
+  CONSTRAINT payments_currency_iso CHECK (currency ~ '^[A-Z]{3}$')
+);
+```
+
+`bigint` minor units, never `float`/`double` — `0.1 + 0.2 <> 0.3` in IEEE 754 and a ledger that drifts by a cent is an incident. Call the column `amount_minor`, not `cents`: JPY has 0 minor units and KWD/BHD/TND have 3, so the divisor comes from the ISO 4217 exponent for the currency, not from a hardcoded 100. Never mix currencies in a `SUM` — the currency column exists so the query *can't* silently do it.
+
+**Never use the Postgres `money` type.** Its output and parsing depend on the session's `lc_monetary`, so the same value formats differently on two machines, and it has no currency of its own. `numeric(20,4)` is defensible for an accounting ledger that must hold sub-minor precision; `float` is never defensible.
+
+### Time
+
+`timestamptz`, always, converted at the edge. Both types occupy 8 bytes; `timestamptz` stores a UTC microsecond instant and converts on input/output using `TimeZone`, while `timestamp` stores whatever digits it was handed and means nothing without out-of-band context. A naive timestamp is a bug with a six-month fuse: it detonates at a DST boundary or the first server in another region.
+
+The one honest exception is a *wall-clock intent* rather than an instant: a birthday, a recurring 09:00 local alarm, a store's opening hours. Those are `date` / `time` plus a `text` column holding an IANA zone name (`Europe/Jakarta`), resolved to an instant at use. Storing them as `timestamptz` bakes in today's UTC offset and breaks when the rule changes.
+
+### Text
+
+**`text`, not `varchar(n)`.** In Postgres `text`, `varchar`, and `varchar(n)` share one implementation and one storage format; `varchar(n)` only adds a length check. The limit reserves no space, saves no time, and makes nothing faster. What it does buy you is a future `ALTER TABLE ... ALTER COLUMN TYPE`, which takes `ACCESS EXCLUSIVE` on the table.
+
+**Never write `varchar(255)`.** 255 is a MySQL artifact — the one-byte length prefix and old utf8 index-prefix limits — not a business rule, and it is the single clearest tell of a schema copied from a tutorial. If there is a real limit, state it as a check, which can be added to a live table without a rewrite (`NOT VALID` then `VALIDATE` — see [migrations.md](migrations.md)):
+
+```sql
+email text NOT NULL,
+CONSTRAINT users_email_len CHECK (length(email) <= 320)  -- RFC 5321 max path length
+```
+
+### Enum vs lookup table vs check
+
+| The value set | Use |
+| --- | --- |
+| Closed, tiny, changes only when code changes, ordering matters | Postgres `enum` |
+| Closed, but you want to add values without ceremony | `text` + `CHECK (col IN (...))` |
+| Data: user-editable, needs labels, i18n, sort order, or any extra attribute | Lookup table + FK |
+| Any value the product team will edit | Lookup table. Always. |
+
+The enum tax is real and asymmetric. `ALTER TYPE ... ADD VALUE` is cheap (and since PG 12 runs inside a transaction — but the new value cannot be *used* in that same transaction). **Removing a value is not supported at all**: you create a new type, `ALTER` every column that uses it, and drop the old one — a table rewrite per column. Renaming works (`ALTER TYPE ... RENAME VALUE`, PG 10+), reordering does not. So: enums are for sets that only shrink over a rewrite you have budgeted for. A `CHECK` constraint is `DROP CONSTRAINT` + `ADD CONSTRAINT ... NOT VALID`, which is seconds, and it still lives in the database where the hub requires constraints to live.
+
+### jsonb
+
+`jsonb` is correct for: a third-party webhook body you must retain verbatim, a rendered snapshot of a document, genuinely sparse user-defined attributes, and config blobs read whole.
+
+It is the wrong answer when:
+
+- **You filter, join, or aggregate on a key.** The planner has poor selectivity estimates for `data->>'status' = 'x'` unless you build an expression index and `ANALYZE`; without one you get a seq scan with a badly wrong row estimate, which then poisons the join order above it.
+- **The key needs a constraint.** `NOT NULL`, `UNIQUE`, `CHECK`, and foreign keys do not reach inside a document. The hub is explicit that these are the only rules surviving a buggy deploy.
+- **A second writer exists.** Two services with two implicit shapes for the same blob is a schema with no owner.
+- **The row is updated often.** Postgres has no in-place partial update: changing one key rewrites the entire row under MVCC, and a large `jsonb` may be de-TOASTed and re-TOASTed to do it.
+
+**The rule:** the second time you write `->>` in a `WHERE` clause, it is a column. Promote it, keep the blob for the rest. Index containment with `CREATE INDEX ... USING gin (data jsonb_path_ops)` and query with `@>`; `jsonb_path_ops` is smaller and faster than the default operator class but only supports containment.
+
+## 3. Nullability Is a Modelling Decision
+
+**Default every column to `NOT NULL`.** Nullable is the claim "this fact can genuinely be unknown", and you should be able to say what unknown means for that column in one sentence.
+
+`NULL` means *unknown or not applicable*. It does not mean empty, zero, false, or "not set yet". Pick one representation for empty text — `''` or `NULL`, not both — and enforce it (`CHECK (bio <> '')`). The confusion is inherited from Oracle, where `''` and `NULL` are the same value; in Postgres they are not, and a codebase holding both will produce two different "no bio" branches.
+
+### The three-valued-logic trap
+
+The most common nullability bug is not a crash. It is a report that silently returns zero rows and nobody notices for a month.
+
+```sql
+-- orders.customer_id is nullable. Exactly one row has NULL (a guest checkout).
+SELECT * FROM customers
+WHERE id NOT IN (SELECT customer_id FROM orders);
+-- 0 rows. Always. Forever.
+```
+
+Why: `id NOT IN (1, 2, NULL)` expands to `id <> 1 AND id <> 2 AND id <> NULL`. The last term is `UNKNOWN`, so the conjunction is at best `UNKNOWN`, `WHERE` keeps only `TRUE`, and every row is discarded. One NULL anywhere in that subquery empties the whole result. `IN` does not have this problem in the same way — it just fails to match — which is why the bug survives review.
+
+```sql
+-- Correct, and usually a better plan (a real anti-join, not a filter):
+SELECT c.* FROM customers c
+WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
+```
+
+Neighbouring surprises, all checkable:
+
+- `x = NULL` is never true. Use `IS NULL`, or `IS DISTINCT FROM` when either side may be NULL.
+- `UNIQUE` permits many NULLs — they are all distinct from each other. `UNIQUE NULLS NOT DISTINCT` (PG 15+) reverses that when you want at most one.
+- `count(col)` skips NULLs, `count(*)` does not. `avg` and `sum` skip them too, and `sum` over an all-NULL set returns `NULL`, not `0` — wrap it: `COALESCE(sum(x), 0)`.
+- A `LEFT JOIN` manufactures NULLs, so a `WHERE` on the right-hand table silently converts it into an inner join. Put that predicate in the `ON` clause.
+
+**A column that is only meaningful when another column is set is a missing table or a missing check.** `CHECK ((status = 'cancelled') = (cancelled_at IS NOT NULL))` beats a comment, and beats hoping.
+
+Adding `NOT NULL` to a populated live table is a lock question, not a modelling one: [migrations.md](migrations.md).
+
+## 4. Soft Delete
+
+Soft delete is not free, and its costs are structural rather than aesthetic:
+
+1. **Every query needs the predicate.** Forget it once and deleted users appear in an export, a search index, a billing run, or a login lookup. This is a whole class of bug that undeletes itself.
+2. **Unique constraints break.** `UNIQUE (email)` blocks a deleted user's address forever, so re-registration fails with a conflict on a row nobody can see.
+3. **Foreign keys still point at the row**, so `ON DELETE CASCADE` never fires and children accumulate live references to dead parents.
+
+Use `deleted_at timestamptz NULL`, never `is_deleted boolean`: it records *when*, and it is exactly the predicate a partial index needs.
+
+```sql
+-- uniqueness among live rows only
+ALTER TABLE users DROP CONSTRAINT users_email_key;
+CREATE UNIQUE INDEX CONCURRENTLY users_email_live_uniq
+  ON users (email) WHERE deleted_at IS NULL;
+
+-- and make the hot read path a partial index too: smaller, and it excludes tombstones
+CREATE INDEX CONCURRENTLY tickets_live_idx
+  ON tickets (tenant_id, created_at DESC) WHERE deleted_at IS NULL;
+```
+
+The planner uses a partial index only when it can prove the query's predicate implies the index's. Write it **literally identically** — `deleted_at IS NULL` — in every query. A logically equivalent phrasing the prover cannot match gives you a seq scan and no error.
+
+Do not rely on discipline for point 1. Enforce it in exactly one place: a `users_live` view with `SELECT` revoked on the base table, or a repository layer where the unfiltered query is not reachable. "Remember to add the WHERE" is not a mechanism.
+
+**Often the better design is not soft delete at all**: move the row to an `*_archive` table in the same transaction, or write the deletion to an audit log and hard-delete. Retention obligations are a retention policy with a schedule, not a permanent `WHERE` clause on the hottest table you own.
+
+## 5. Indexing
+
+### Composite column order
+
+**Equality columns first, then one range column, then the sort.** A B-tree is ordered by the concatenated key, so it can only seek on a leading run of equalities; the first inequality ends the seekable prefix and everything after it is filtered, not sought.
+
+```sql
+SELECT id, title FROM tickets
+WHERE tenant_id = $1 AND status = $2 AND created_at < $3
+ORDER BY created_at DESC
+LIMIT 50;
+
+CREATE INDEX tickets_tenant_status_created_idx
+  ON tickets (tenant_id, status, created_at DESC);
+-- equality (tenant_id, status) -> range + sort (created_at).
+-- One index scan, LIMIT stops early, no Sort node in the plan.
+```
+
+`(created_at, tenant_id, status)` for the same query is a scan of every ticket older than `$3` from every tenant, filtered afterwards. Both indexes are the same size and one of them is useless.
+
+A composite `(a, b, c)` already serves `(a)` and `(a, b)`. It does **not** serve `(b)`. Before adding an index, check whether an existing one has your columns as its prefix; the most common indexing defect is not a missing index, it is four indexes that all begin with `tenant_id`.
+
+### Covering indexes and index-only scans
+
+```sql
+CREATE INDEX tickets_tenant_created_cover_idx
+  ON tickets (tenant_id, created_at DESC) INCLUDE (title, status);
+```
+
+`INCLUDE` stores non-key payload in the leaf so the query never touches the heap. Verify, do not assume: `EXPLAIN (ANALYZE, BUFFERS)` must show `Index Only Scan` **and `Heap Fetches: 0`**. Nonzero heap fetches mean the visibility map is stale — the table needs autovacuum to catch up, and until it does the "covering" index is a normal index with extra bytes.
+
+### Partial indexes
+
+```sql
+-- a 40M-row jobs table where ~2k rows are pending at any moment
+CREATE INDEX jobs_pending_idx ON jobs (run_after) WHERE status = 'pending';
+```
+
+The index holds thousands of entries instead of tens of millions, stays in cache, and — because a row leaving `pending` leaves the index — it does not bloat. This is the right tool for queue tables, soft-delete tombstones, and any low-cardinality flag where you only ever query one value. Do not build a plain B-tree on a lone boolean.
+
+### An unused index taxes every write
+
+Every `INSERT`, `DELETE`, and every non-HOT `UPDATE` writes an entry into *every* index on the table. Worse, an index on a column means updates to that column can no longer be HOT, which forfeits the optimisation for the whole row. Add WAL volume, replication bandwidth, longer `VACUUM`, more planning time, and unused indexes are strictly negative.
+
+```sql
+-- unused, by size
+SELECT s.relname, s.indexrelname, s.idx_scan,
+       pg_size_pretty(pg_relation_size(s.indexrelid)) AS size
+FROM pg_stat_user_indexes s
+JOIN pg_index i ON i.indexrelid = s.indexrelid
+WHERE s.idx_scan = 0 AND NOT i.indisunique AND NOT i.indisprimary
+ORDER BY pg_relation_size(s.indexrelid) DESC;
+```
+
+Three caveats before dropping anything: counters are cumulative since the last `pg_stat_reset()`, so a recently reset cluster proves nothing; **replicas keep their own counters**, so an index unused on the primary may serve every read on a replica — check there too; and a unique index enforcing a constraint is doing its job at `idx_scan = 0`. On a live table, drop with `DROP INDEX CONCURRENTLY`.
+
+```sql
+-- missing, by damage: big tables being sequentially scanned a lot
+SELECT relname, seq_scan, seq_tup_read, idx_scan,
+       seq_tup_read / NULLIF(seq_scan, 0) AS avg_rows_per_scan,
+       pg_size_pretty(pg_relation_size(relid)) AS size
+FROM pg_stat_user_tables
+WHERE seq_scan > 0 AND pg_relation_size(relid) > 100 * 1024 * 1024
+ORDER BY seq_tup_read DESC;
+```
+
+High `seq_tup_read` on a large table is the signal. A small table with many seq scans is fine — scanning 200 rows in one page beats an index descent, and the planner knows it.
+
+**Postgres does not index foreign keys automatically.** The parent side has a unique index; the child column has nothing. The symptoms are a `DELETE` on the parent that scans the whole child table while holding locks on it, and slow cascades. Index every FK column you delete or join on.
+
+`LIKE 'foo%'` uses a B-tree only under the `C` collation or with `text_pattern_ops`; `LIKE '%foo%'` needs `pg_trgm` and a GIN index. Creating indexes on a live table is `CREATE INDEX CONCURRENTLY` — see [migrations.md](migrations.md) for its failure and retry behaviour.
+
+## 6. N+1
+
+The symptom is not a slow query. Every query in the log is 0.4 ms and every one looks fine. The symptom is a slow *request* containing 200 fast queries — round trips, not work.
+
+### Detection: count queries, assert a budget
+
+Reading code does not find these, because the loop and the lazy load are in different files. Instrument the driver in development and test: SQLAlchemy `before_cursor_execute`, Django `CaptureQueriesContext` / `connection.queries`, Prisma `$on('query')`, `pg` `Pool` events. Then assert:
+
+```ts
+// The assertion that actually works: SAME budget at two page sizes.
+const one   = await countQueries(() => GET('/api/projects?limit=1'));
+const fifty = await countQueries(() => GET('/api/projects?limit=50'));
+
+expect(fifty).toBe(one);   // constant in page size — this is the real property
+expect(fifty).toBeLessThanOrEqual(3);   // 1 page + 1 batched children + 1 count
+```
+
+A single-threshold test passes at `limit=1` and tells you nothing. Query count must be **constant in the page size**; that is the invariant, and a test that encodes it fails the moment someone reintroduces a lazy load. Log the per-request count on every dev response too, so it is visible without a test run. (How to run and structure these tests is `kerja-agentik`; what to assert is here.)
+
+### The three fixes
+
+| Shape | Fix | Cost |
+| --- | --- | --- |
+| To-one (each row's parent/owner) | `JOIN` in the same query | 1 query, no row multiplication |
+| To-many (each row's children) | Second query, `WHERE parent_id IN (:ids)` | 2 queries, no row multiplication |
+| Call site you don't control (GraphQL resolvers, nested services) | DataLoader — per-request batch + dedupe | 2 queries, plus a batching tick |
+
+Do not `JOIN` a to-many collection into a list query: a page of 50 parents with 20 children each returns 1,000 rows with the parent columns repeated 20 times, and the payload becomes the new bottleneck. Use the batched `IN` — that is exactly what SQLAlchemy `selectinload`, Django `prefetch_related`, and Rails `preload` do, and why they exist alongside the join variants (`joinedload`, `select_related`, `eager_load`). Chunk the `IN` list at 1,000 ids; an unbounded `IN` built from user input is on the hub's Never Ship list.
+
+### "It's fine, they're cached" is not an answer
+
+- The cache is cold on every deploy, every eviction, and every new tenant. Your p99 is the cold path, and p99 is what gets reported as "the app is slow".
+- A 95% hit rate over a 50-item page is still ~2.5 misses per request, and they are serial.
+- Staging has a working set that fits in cache. Production does not. This is why the bug ships.
+- Unless the cache is in-process, every "hit" is still a network round trip — you replaced N database calls with N Redis calls and moved the problem to a service with a smaller connection pool.
+
+If caching is the plan, the plan must be stated as: what the cold-cache query count is, and why that number is acceptable. Cache design itself is [testing-and-perf.md](testing-and-perf.md).
+
+## 7. Constraints Belong in the Database
+
+The hub states this; here is the operational reason. Application validation holds until one of three things happens, and all three happen:
+
+1. **A buggy deploy** ships a code path that skips the check. The database is the only layer still enforcing it.
+2. **A manual fix** — someone in `psql` at 02:00 running an `UPDATE` under incident pressure, with no application layer between them and the table.
+3. **A second writer** — an admin tool, a data-import script, a backfill job, another service, an analytics reverse-ETL. None of them import your validators.
+
+| App-level rule | Database form |
+| --- | --- |
+| "This field is required" | `NOT NULL` |
+| "Emails are unique per tenant" | `UNIQUE (tenant_id, email)` |
+| "Quantity must be positive" | `CHECK (quantity > 0)` |
+| "This must reference a real row" | `FOREIGN KEY ... ON DELETE RESTRICT` |
+| "Only one live row per key" | Partial unique index (§4) |
+| "Two bookings cannot overlap" | `EXCLUDE USING gist (room_id WITH =, during WITH &&)` |
+
+`RESTRICT` is the default posture. `CASCADE` only when the child is genuinely part of the parent's lifetime (`order_items` under `orders`), never across an aggregate boundary — a cascading delete that crosses one is how a single row removal quietly deletes an audit trail.
+
+Two limits worth knowing. A `CHECK` cannot see the previous row, so state-machine transitions ("pending may become paid, paid may not become pending") are not expressible as a check; enforce them with a guarded update — `UPDATE ... WHERE id = $1 AND status = 'pending'`, zero rows affected meaning the transition lost, which the hub maps to `409`. And an overlap check written in application code is a race by construction, which is why the exclusion constraint above is the only correct booking guard; the reasoning lives in [concurrency-and-jobs.md](concurrency-and-jobs.md).
+
+Adding any of these to a table that already has rows is [migrations.md](migrations.md) — the `NOT VALID` / `VALIDATE CONSTRAINT` two-step exists precisely so this does not take a long lock.
+
+## 8. Denormalisation
+
+Correct in exactly three situations:
+
+- **An unbounded aggregate read on every request.** `posts.comment_count` when the alternative is counting a table that grows without limit.
+- **A read that would otherwise cross a service or shard boundary** on the hot path.
+- **A historical snapshot that must not follow its source.** The unit price at the time of order, the address printed on a shipped invoice. This one is not really denormalisation — "the price we charged" and "the price today" are two different facts, and modelling them as one column was the bug.
+
+**Every denormalised value needs a named invariant owner.** Exactly one mechanism keeps it true, it is written down, and it is reachable:
+
+```sql
+COMMENT ON COLUMN posts.comment_count IS
+  'Maintained by trigger trg_posts_comment_count. Reconciled nightly by
+   jobs/reconcile_counts.ts, which alerts on drift. Do not update directly.';
+```
+
+If you cannot name the owner in one line, you do not have a denormalised column — you have a cache with no invalidation story, sitting in the table where nobody will think to distrust it.
+
+One consequence to price in before choosing a counter column: every child insert updates the same parent row, so writes to a hot parent serialise on that row lock and long transactions make it worse. When contention shows up, the fix is an append-only delta table summed on a schedule, not a bigger machine — see [concurrency-and-jobs.md](concurrency-and-jobs.md).
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| `float`/`double` for money, or the Postgres `money` type | `bigint` minor units + ISO 4217 `currency` |
+| `timestamp` without time zone | `timestamptz`, UTC, converted at the edge |
+| `varchar(255)` | `text` + a `CHECK` on the real limit |
+| `serial` / `int4` primary key on a growing table | `bigint GENERATED ALWAYS AS IDENTITY` |
+| `uuidv4` primary key on the biggest table | `uuidv7` / ULID in a `uuid` column |
+| A sequential id in a public URL where volume is sensitive | `bigint` PK + `public_id uuid` |
+| Soft delete with a plain `UNIQUE` constraint | Partial unique index `WHERE deleted_at IS NULL` |
+| `NOT IN (subquery)` over a nullable column | `NOT EXISTS` |
+| `->>` in a `WHERE` clause for the second time | Promote it to a column |
+| An index added "to be safe" | Read the plan first; check for an existing prefix |
+| An unindexed foreign key you delete or join on | Index the child column |
+| Validation that exists only in application code | The matching database constraint |
+| A denormalised column with no named owner | A comment naming the trigger/job, or normalise it |
+| A query-count test at one page size | Assert the count is constant across two |
+
+## No Guessing
+
+Never invent a value that reads like a measured one:
+
+- **Never state a row count you did not measure.** `SELECT reltuples::bigint FROM pg_class WHERE relname = 'x'` for an estimate, `count(*)` for the truth. "It's a big table" is not an input to a decision.
+- **Never claim an index will be used.** Run `EXPLAIN (ANALYZE, BUFFERS)` and read which index the plan actually chose. A plausible index that the planner ignores is worse than none.
+- **Never invent a "reasonable" length, precision, or scale.** Every number in a type comes from a spec (ISO 4217 exponent, RFC 5321's 320) or from measuring the data.
+- **Never assume the ORM's default loading strategy.** Check the mapping, then count the queries.
+
+## Output
+
+Write the DDL. Then, briefly: the key choice and why, the columns that are nullable and what unknown means for each, the indexes and the exact query each one serves, and the constraints that survive a bad deploy. If the answer was "no schema change" — the column exists, the index is redundant, the table is a column — say that in one line and add nothing. Producing nothing is the correct output more often than it feels.

package/dist/resources/skills/builtin/backend-engineering/references/deploy-and-config.md

@@ -0,0 +1,228 @@
+# Deploy and Config
+
+This file owns the process lifecycle: how configuration reaches the service and is proven correct, how the service boots, how it dies without dropping work, every timeout on every hop, and the order in which a deploy touches the database and the code. It is the canonical source for timeout values, drain deadlines, and probe settings — when another file quotes one, it is restating this file.
+
+Neighbours, so you don't half-solve a problem here: the SQL and locking behaviour of a schema change is [migrations.md](migrations.md); this file only decides *when in the deploy* it runs. Where a secret comes from, how it rotates, and what to do when it leaks is [auth-and-secrets.md](auth-and-secrets.md); this file only says it arrives through the environment and is parsed at boot. What `/readyz` reports and which alert fires on it is [observability.md](observability.md). Draining a queue consumer is [concurrency-and-jobs.md](concurrency-and-jobs.md) §9 — the HTTP half is here. Pool sizing against query latency is [testing-and-perf.md](testing-and-perf.md).
+
+## Operating Posture
+
+Three failure modes, ranked, and the first is worse than the other two combined:
+
+1. **A config error found by a user instead of by the deploy.** `STRIPE_SECRET_KEY` is spelled `STRIPE_SECRET`, `os.environ.get` returns `None`, the code takes the "no key configured, skip billing" branch, and the process boots green. Nothing is red. You find it at 3am from a customer, and the window is however long since the deploy.
+2. **A deploy that cannot be rolled back.** The contracting migration ran, the column is gone, and the previous image no longer works. There is now no fast path back, only forward through an incident.
+3. **A shutdown that isn't graceful.** Every deploy sheds a handful of 502s and re-runs a handful of jobs. Individually deniable, permanently blamed on "flaky infra".
+
+**Never invent a timeout, a grace period, a probe interval, or a pool size.** Take them from the tables below or from the project's own config. `timeout: 30000` copied out of a blog post looks exactly like a measured value and is the reason the pool fills.
+
+**Producing nothing is frequently right here.** A var that holds the same value in dev, staging and prod is not configuration — it is a constant that has escaped code review. Deleting three of those and inlining them is a better diff than adding a fourth. A single-container side project with nothing routing to it does not need a readiness endpoint, a `preStop` hook, or a drain deadline; `exit 0` is the whole shutdown. Say so and stop.
+
+## 1. Config Is Parsed at Boot or the Process Refuses to Start
+
+One module. `process.env` / `os.environ` / `os.Getenv` appears in the codebase **exactly once**, inside it. Everything else imports the typed object.
+
+```ts
+// config.ts — the only file in the repo that touches process.env
+const Env = z.object({
+  NODE_ENV:        z.enum(["development", "test", "production"]),
+  PORT:            z.coerce.number().int().min(1).max(65535).default(8080),
+  DATABASE_URL:    z.string().url().startsWith("postgres"),
+  DB_POOL_MAX:     z.coerce.number().int().min(1).max(50).default(10),
+  REDIS_URL:       z.string().url(),
+  SESSION_SECRET:  z.string().min(32),   // 256 bits — see auth-and-secrets.md
+  LOG_LEVEL:       z.enum(["debug", "info", "warn", "error"]).default("info"),
+  DRAIN_DEADLINE_MS: z.coerce.number().int().min(1_000).default(25_000),
+});
+// NOT .strict() — process.env also carries PATH, HOME, HOSTNAME.
+// Strictness belongs on request bodies (validation-and-errors.md), not here.
+
+export const config = Env.parse(process.env);  // throws at import → non-zero exit
+```
+
+```python
+class Settings(BaseSettings):
+    model_config = SettingsConfigDict(extra="ignore")
+    environment: Literal["development", "test", "production"]
+    port: int = Field(default=8080, ge=1, le=65535)
+    database_url: PostgresDsn
+    session_secret: SecretStr = Field(min_length=32)
+    drain_deadline_s: int = 25
+
+settings = Settings()   # ValidationError at import, before uvicorn binds
+```
+
+Four rules that make this actually load-bearing:
+
+- **Parse at import of the entry module, before the socket binds.** Lazy validation on first use is the bug this section exists to prevent, wearing a type annotation.
+- **Coerce once.** Every env value is a string. `if (process.env.DEBUG)` is `true` for the string `"false"`, and Python's `bool(os.environ["DEBUG"])` is `True` for `"false"` too. Both are silent. `z.coerce.boolean()` has the same trap — use `z.enum(["true","false"]).transform(v => v === "true")`.
+- **No fallback default for anything whose wrong value is dangerous.** `PORT` may default. `SESSION_SECRET`, `DATABASE_URL`, and `NODE_ENV` may not — `process.env.SECRET_KEY || "dev-secret"` is how a signing key of `"dev-secret"` reaches production ([auth-and-secrets.md](auth-and-secrets.md)).
+- **Go has no exception to stop you.** `os.Getenv("DATABASE_URL")` returns `""` for a missing var, and `""` parses as "connect to localhost with no password". Use `os.LookupEnv`, collect every missing name, and `log.Fatal` with the whole list — not the first one, so a misconfigured deploy takes one round trip to fix instead of five.
+
+Log the resolved config once at boot with values redacted by key name. The point is to see what *is* set next to what you think is set.
+
+## 2. Env vs Code vs Database
+
+| The value | Lives in | Because |
+| --- | --- | --- |
+| Differs per environment, not secret — `DATABASE_URL` host, `LOG_LEVEL`, `PORT`, `ENVIRONMENT`, external base URLs | **Env** | The image is identical across environments; only the injection differs |
+| Differs per environment, secret — API keys, signing keys, DB password | **Env**, injected from the secret manager at start, never baked in ([auth-and-secrets.md](auth-and-secrets.md)) | A layer in a registry is not a secret store |
+| Identical everywhere — retry cap of 5, max page size 100, RFC 9457 `type` URIs, table names, cache key prefixes | **Code**, as a reviewed constant | An env var that never varies is an unreviewed constant with a drift risk attached |
+| Must change without a deploy, or varies per tenant — **feature flags**, rate-limit tiers, maintenance mode, per-tenant quotas, kill switches | **Database** | See below |
+| Changes per request | Nowhere. It's a parameter. |
+
+**Feature flags in env are not flags.** Flipping one is a deploy, and during the rollout window half the fleet has the old value and half the new — so a flag intended to disable a broken code path leaves it live on 50% of traffic for the length of the rollout. It also cannot be scoped to one tenant, which is the reason you wanted a flag. Put flags in a table, read them through a process-local cache with a **30s TTL** (one query per pod per 30s instead of one per request; a flip takes at most 30s to reach the whole fleet, which is still two orders of magnitude faster than a deploy), and default to *off* when the read fails.
+
+The inverse mistake: putting `DATABASE_URL` in the database.
+
+## 3. Boot Order
+
+1. **Parse config.** Fail → log every problem, `exit 1`. Nothing is bound, nothing is connected.
+2. **Open the pool** with a bounded startup retry — 5 attempts, 1s apart — because a pod routinely starts before its database finishes accepting connections. Unbounded retry here hides a genuinely wrong `DATABASE_URL` forever.
+3. **Assert the schema version** matches what this build expects, and refuse to start on a mismatch. This is what turns "somebody deployed before the migration" into a failed rollout instead of a runtime error storm.
+4. **Bind the port.** Liveness now passes.
+5. **Readiness passes** once dependency checks are green.
+6. **Start queue consumers last**, so nothing is consumed by a process that isn't ready to finish it.
+
+## 4. Graceful Shutdown — Exact Order
+
+On `SIGTERM` (not `SIGINT`; orchestrators send `SIGTERM`, and a handler registered only for `SIGINT` never runs in production):
+
+| # | Step | Skipping it costs |
+| --- | --- | --- |
+| 1 | Set `state = draining`, log it once | You cannot tell a drain from a crash in the logs |
+| 2 | **Fail readiness** — `/readyz` → `503`. **Never fail liveness**, or the kubelet `SIGKILL`s you mid-drain | The LB keeps routing to a closing pod: connection resets, 502s on every deploy |
+| 3 | **Keep serving for 5s** while endpoint removal propagates (kubelet sends `SIGTERM` and updates Endpoints *concurrently*, and the update is asynchronous through kube-proxy) | A smaller version of the same 502s, in the window nobody believes exists |
+| 4 | **Stop accepting connections** — `server.close()` + `server.closeIdleConnections()` (Node 18.2+, since `close()` alone waits on keep-alive sockets that will never send another byte), `srv.Shutdown(ctx)` in Go | New connections keep arriving; the drain never converges and you get `SIGKILL`ed |
+| 5 | **Finish in-flight work with a deadline** (below) | One slow request hangs the process until `SIGKILL` takes everything else with it |
+| 6 | **Stop queue consumers after acking** — mechanics in [concurrency-and-jobs.md](concurrency-and-jobs.md) §9 | Redelivery: every deploy becomes a duplicate-work event |
+| 7 | **Flush** logs, metrics, traces ([observability.md](observability.md)) | You lose the last 10s of telemetry — which is the telemetry about the shutdown |
+| 8 | **Close the pool**, then Redis | Postgres holds the backends until TCP keepalive notices; a rolling deploy stacks two fleets' connections against `max_connections` |
+| 9 | **Explicit `exit 0`** | One lingering timer or open handle keeps the process up past the grace period → `SIGKILL`, non-zero exit, restart alerts on a healthy deploy |
+
+```
+drain_deadline = 25s        # strictly under Kubernetes' 30s default
+                            # terminationGracePeriodSeconds
+```
+
+**The orchestrator's grace period must strictly exceed your drain deadline.** Equal is a loss: `SIGKILL` lands on the same tick your deadline expires, before step 9 runs. If handlers legitimately need longer, raise `terminationGracePeriodSeconds` deliberately and keep the deadline under it — never the reverse.
+
+Two symptoms that mean your shutdown code is correct and never executes:
+
+- **`CMD npm start` (shell form).** The shell becomes PID 1 and does **not** forward `SIGTERM` to its child. Every shutdown is a 30-second wait then `SIGKILL`. Use exec form — `CMD ["node", "dist/server.js"]` — or `tini` as init.
+- **The handler is registered after the server starts serving**, so a `SIGTERM` during a slow boot kills the process outright.
+
+## 5. Timeouts at Every Hop
+
+**Each hop's timeout is strictly shorter than its caller's**, so the innermost one fires first and returns a real error instead of a hung socket. A timeout longer than its caller's is unreachable code: the caller has already given up, and yours fires into an empty room.
+
+```
+client (browser / mobile)            30s   # the user gives up first, and knows they did
+edge / LB idle                       25s   # AWS ALB defaults to 60s, nginx proxy_read_timeout
+                                           # to 60s — both far too long, set them down
+gateway → service                    20s
+service request handler              10s   # restated from validation-and-errors.md
+service → third-party HTTP    5s total / 2s connect
+db statement_timeout                  5s   # restated from validation-and-errors.md; default 0 = unlimited
+db lock_timeout                       2s   # default 0 = wait for the lock forever
+idle_in_transaction_session_timeout  10s   # default 0 = forever
+pool acquire (checkout)               2s   # fail fast rather than queue behind an exhausted pool
+```
+
+**The failure of an unbounded timeout is that the pool fills with requests nobody is waiting for any more.** The client gave up at 30s and retried; your handler is still holding a worker and a connection for a socket that is gone. Load doubles at precisely the moment you are slow. The symptom is diagnostic: p99 climbs, then *every* endpoint starts failing, including ones that never touch the slow dependency — and a restart "fixes" it. That shape means a pool, not a bug in the slow endpoint.
+
+Defaults that are effectively infinite, and must be overridden explicitly:
+
+| Runtime | Reality |
+| --- | --- |
+| Node global `fetch` / undici | **No total request timeout.** `connectTimeout` 10s, `headersTimeout` and `bodyTimeout` 300s. Pass `AbortSignal.timeout(5000)` on every call. |
+| Node `http.Server` | `requestTimeout` 300s, `headersTimeout` 60s, `keepAliveTimeout` 5s |
+| Python `requests` | **No default timeout at all.** `requests.get(url)` can hang until the OS gives up. Always pass `timeout=(2, 5)`. `httpx` does default to 5s. |
+| Go `http.Client{}` | Zero-value `Timeout` = no limit. `http.Server` zero value has no `ReadTimeout`/`WriteTimeout` either. Set both, plus a per-request `context.WithTimeout`. |
+| Postgres | `statement_timeout`, `lock_timeout`, `idle_in_transaction_session_timeout` all default to `0` = unlimited |
+
+Set the client-side deadline as a **budget that decrements across hops**, not a fresh 5s per call: three sequential 5s calls inside a 10s handler is a 15s worst case that the handler can never reach, so the last call is dead code under load.
+
+## 6. Deploy Ordering with Migrations
+
+**Three deploys. The middle one is not optional.** Mechanics of each statement — `CONCURRENTLY`, lock modes, batch sizes — are [migrations.md](migrations.md).
+
+1. **Expand.** Additive only: add the nullable column, add the new table, add the index. Old code ignores it; new code isn't live yet. Runs as a one-shot job *before* the rollout, never in the container `ENTRYPOINT` — four replicas booting means four concurrent migration runs fighting for the same lock.
+2. **Deploy code that tolerates both shapes.** Write both old and new, read the one you trust, never assume the new column is populated. Backfill here, in batches, after the code is live.
+3. **Contract, in a later deploy** — after every old replica is gone and you no longer intend to roll back past it: drop the column, drop the old table, add `NOT NULL`.
+
+**Why contracting in the same deploy breaks.** A rolling update runs both versions simultaneously — Kubernetes' default `maxSurge: 25% / maxUnavailable: 25%` keeps both ReplicaSets receiving traffic for the entire rollout, which for a service with a slow boot is minutes, not seconds. The migration runs once, at the start. Old pods then execute `INSERT ... (old_column)` against a table that no longer has it. The symptom is a burst of `42703 undefined_column` that **stops on its own** when the last old pod dies — which is exactly why it gets logged, shrugged at, and shipped again next week.
+
+The same window applies in reverse: new code deployed before its expand migration sees a column that doesn't exist yet. Expand always leads.
+
+## 7. Rollback
+
+**Code is rollback-safe. A contracting migration is not. That asymmetry is the entire reason for the ordering above.**
+
+- Rolling back code is redeploying the previous image — bounded, boring, seconds.
+- `DROP COLUMN` is not undone by adding the column back. The schema returns; the values do not. A `down()` migration that "reverses" a drop hands you a working service reading silent `NULL`s, which is strictly worse than the outage it was meant to fix.
+- Therefore: **every deploy's rollback plan is "redeploy the previous image", and nothing else.** If undoing your change requires restoring data, you have not finished designing the deploy — split it.
+- Behaviour you might need to switch off in seconds belongs behind a database-backed flag (§2), not behind a rollback. A flag flip has no rollout window.
+
+## 8. Health Endpoints, Wired
+
+```yaml
+livenessProbe:   { httpGet: { path: /livez  }, periodSeconds: 10, failureThreshold: 3, timeoutSeconds: 2 }
+readinessProbe:  { httpGet: { path: /readyz }, periodSeconds: 5,  failureThreshold: 2, timeoutSeconds: 2 }
+startupProbe:    { httpGet: { path: /livez  }, periodSeconds: 5,  failureThreshold: 30 }  # 150s to boot
+terminationGracePeriodSeconds: 30                                                        # > 25s drain deadline
+```
+
+`/livez` answers "is this process wedged" and touches **no dependency** — a liveness probe that queries the database converts a 30-second database blip into a fleet-wide restart storm, which is the hub's Never Ship entry. `/readyz` answers "should traffic come here" and may check the pool and the schema version. What each returns in its body, and which alerts hang off them, is [observability.md](observability.md).
+
+The `startupProbe` exists so a slow boot isn't killed by the liveness probe before it finishes. Its symptom when missing is `CrashLoopBackOff` with nothing in the logs — nothing crashed; it was killed mid-migration-check.
+
+## 9. Twelve-Factor: What Survived
+
+| Factor | Verdict |
+| --- | --- |
+| III — config in the environment | **Still the rule.** Env is the interface; a secret manager or mounted file is just what fills it. |
+| VI — stateless processes | **Still the rule**, and the one that makes rolling deploys possible at all. In-memory session state is a bug with a rollout-shaped fuse. |
+| IX — disposability: fast start, graceful stop | **Still the rule.** §3 and §4 are this factor, written out. |
+| XI — logs as an event stream to stdout | **Still the rule.** Writing to a file inside a container invents log rotation you then have to operate. |
+| X — dev/prod parity | **Still the rule**, narrowly: same database engine and major version. SQLite in dev and Postgres in prod is a promise to find the differences in production. |
+| II, V — dependency isolation, build/release/run | Solved by lockfiles and any container pipeline. Nothing to do. |
+| VII — port binding | Read `PORT` from env and stop thinking about it. |
+| VIII — concurrency via the process model | **Aged badly.** Written before async runtimes; forking N workers also multiplies your DB pool by N and is how a "scale up" exhausts `max_connections`. |
+| XII — admin tasks as one-off processes | **Aged badly.** Migrations belong in the pipeline as a gated step (§6), not in an ad-hoc console against prod. |
+
+## 10. Containers
+
+- **One process per container.** No `supervisord`, no `&`. Restarts, logs, and signals are all per-container; a second process dies silently while the container reports healthy.
+- **Your process is PID 1, or an init forwards signals to it** — see the `CMD` trap in §4.
+- **No secrets in the image.** Not `ENV`, not a `COPY`d `.env`, not a build `ARG` — every layer ships to the registry and `docker history` prints build arguments. Inject at start ([auth-and-secrets.md](auth-and-secrets.md)).
+- **Pin the base image by digest**, run as a non-root `USER`, and prefer a read-only root filesystem. `:latest` means your rollback pulls a different image than the one that worked.
+- **In VibeCode, a compose stack outlives the app quitting.** Process cleanup works by PID and cannot see a container, so `docker compose up` leaves a stack running after the session ends. Always `docker compose up -d` with an explicit `docker compose down`.
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| Config read at first use | Parsed into a typed object at boot; `exit 1` on anything missing |
+| `process.env` / `os.environ` scattered through the codebase | One config module, imported |
+| `process.env.SECRET \|\| "dev-secret"` | No default; crash loudly at boot |
+| `.strict()` on `process.env` | Non-strict here; strict on request bodies |
+| Truthiness on an env string (`"false"` is truthy) | Explicit enum + transform |
+| A feature flag in an env var | A row in a table, 30s cache TTL, default off |
+| An env var identical in every environment | A constant in code |
+| `SIGINT`-only shutdown handler | `SIGTERM` |
+| Failing liveness during drain | Fail readiness only |
+| `server.close()` without `closeIdleConnections()` | Both (Node 18.2+) |
+| Drain deadline ≥ grace period | 25s drain under the 30s default |
+| `CMD npm start` shell form | `CMD ["node", "dist/server.js"]` or `tini` |
+| Any outbound call with no explicit timeout | `AbortSignal.timeout(5000)` / `timeout=(2, 5)` / `context.WithTimeout` |
+| A timeout longer than its caller's | Strictly shorter at every inward hop |
+| `statement_timeout` left at `0` | `5s`, with `lock_timeout` `2s` |
+| Migrations in the container `ENTRYPOINT` | A one-shot job before the rollout |
+| Expand and contract in the same deploy | Contract in a later deploy, old code gone |
+| A rollback that needs a data-restoring `down()` | Split the deploy so the image alone rolls back |
+| A liveness probe that queries the database | Dependency checks on `/readyz` only |
+| Secrets in `ENV`, `ARG`, or a `COPY`d `.env` | Injected at start |
+| `docker compose up` with no `down` | `up -d` plus an explicit `down` |
+
+## Output
+
+Write the config module and the lifecycle code. Then, in a few lines: the config keys you added and which of the three homes each went to, the exact timeout values you set and which hop they sit under, the deploy sequence if this change touches the schema (expand / tolerate / contract, and which one this deploy is), and how it rolls back. Say what you verified — `SIGTERM` to the running process and the observed exit code, or a boot with a var deliberately removed and the non-zero exit it produced. If you removed config rather than adding it, say which and why in one line. Load-time failure hunting and reviewing someone else's deploy path are [review-audit.md](review-audit.md).