vibecodeagent 0.0.0-stage → 0.1.65

package/dist/resources/skills/builtin/backend-engineering/references/migrations.md

@@ -0,0 +1,320 @@
+# Migrations
+
+This file owns changing a schema that is currently serving traffic: expand/contract sequencing, what lock each statement takes, backfills, and migration-file hygiene. **What the schema should be** — key choice, types, nullability, which index — is [data-modelling.md](data-modelling.md); every recipe there that touches a populated table hands off here. **Ordering the migration against the deploy**, the runner's own timeouts, and graceful shutdown are [deploy-and-config.md](deploy-and-config.md). Row locks taken by application code, transaction boundaries, and retrying `40001` are [concurrency-and-jobs.md](concurrency-and-jobs.md). Postgres is the assumed engine, per the hub's boring-choice rule; the MySQL/InnoDB differences that change the answer are called out inline. And where `kerja-agentik` owns how you run and verify things, this file owns what "verified" means for a migration: a timing on production-shaped data under concurrent load, never a green run on dev.
+
+## Operating Posture
+
+Three failure modes, ranked. The first is the one that takes the site down; the second is the one that is hardest to undo.
+
+1. **The lock queue.** The DDL itself takes 2ms. It waits behind a 40-second analytics `SELECT`, and every query that arrives while it waits queues behind *it*. The symptom is not "the migration is slow" — it is every endpoint touching that table returning 500 within seconds of the migration starting, including endpoints that only read. Section 3 exists entirely for this.
+2. **The contract step shipped early.** The column was dropped, renamed, or tightened while an app version that still uses it is running — a pod mid-rollout, a worker on the old image, a rolled-back deploy. Symptom: `column "email" does not exist` on a fraction of requests, and no code change caused it. Expand/contract (section 1) exists entirely for this.
+3. **The single-statement backfill.** One `UPDATE` over 40 million rows: hours of WAL, replicas falling behind, `VACUUM` unable to reclaim anything, and a rollback that throws away all of it if it fails at 95%. Section 4 exists entirely for this.
+
+### Gate: does this need to run now, in this deploy?
+
+| Situation | Answer |
+| --- | --- |
+| Purely additive: new table, new nullable column, new index | Run it. Ship it ahead of the code that uses it. |
+| Anything that removes, renames, narrows, or constrains | Split it. At least two migrations across at least two deploys. |
+| Target table under ~10k rows, off the hot path | The dangerous forms are still dangerous; the lock is just short. Keep `lock_timeout`, skip the batching. |
+| Rewrites a large table (section 2) | Not a migration. A recipe (sections 5–6), or a window you state out loud. |
+| "It's just a rename, it's instant" | Section 5. Instant at the database is irrelevant; no app version works on both sides of it. |
+
+**Producing nothing that runs today is a correct output.** "This is three migrations across three deploys; here is file one, which adds a nullable column and nothing else" is the right answer far more often than one file that does the whole change. Say it and write file one.
+
+## 1. Expand / Contract
+
+The invariant: **at every instant, the schema in the database is correct for both the app version running and the app version you could roll back to.** Every step below preserves it; skipping one breaks it.
+
+1. **Expand.** Add the new column (nullable, no volatile default), table, or index. Backward compatible: old code neither knows nor cares.
+2. **Dual-write.** New code writes both old and new on every write path. Reads still come from old.
+3. **Backfill.** Fill historical rows in batches (section 4). Runs after every writer is dual-writing.
+4. **Validate.** Prove the invariant with a query that must return 0, then add the constraint (`NOT VALID` → `VALIDATE`, section 2).
+5. **Switch reads.** The new column becomes the source of truth. This is the only step users can perceive.
+6. **Stop writing the old column.** Drop its `NOT NULL` first, or the first insert from the new code fails.
+7. **Contract.** Drop the old column, constraint, or index.
+
+| Steps | Same deploy? | Why |
+| --- | --- | --- |
+| 1 + 2 | **Yes** | The migration runs before the new app boots, so the column exists before anything writes it. |
+| 2 + 3 | **No** | A row written between the backfill's last batch and the dual-write rollout has a NULL new column, and nothing will ever fix it. |
+| 3 + 4 | Yes | Both are data-plane work; `VALIDATE` is just the statement after the last batch. |
+| 4 + 5 | **No** | The read switch must be revertible on its own. Bundling it with the constraint means one rollback undoes both. |
+| 5 + 6 | **No** | You need a version in production that reads new and writes both. That version *is* your rollback target. |
+| 6 + 7 | **Never** | A rollback to the previous image resumes writing a column that no longer exists. |
+
+**The boundary is the rollback window, not the deploy.** Step 7 waits until you would no longer roll back past step 6 — one deploy minimum, one week as the default. On a service that deploys ten times a day, "one week" is the honest number, not "the next deploy".
+
+## 2. What Each Statement Actually Locks
+
+Lock modes first, because the operation table below is meaningless without them. A lock request blocks only on *conflicting* modes:
+
+| Lock mode | Blocks `SELECT`? | Blocks `INSERT`/`UPDATE`/`DELETE`? | Taken by |
+| --- | --- | --- | --- |
+| `ACCESS SHARE` / `ROW EXCLUSIVE` | no | no | every `SELECT` / every `INSERT`, `UPDATE`, `DELETE` |
+| `SHARE UPDATE EXCLUSIVE` | no | no | `CREATE INDEX CONCURRENTLY`, `VALIDATE CONSTRAINT`, `VACUUM`, `ANALYZE` |
+| `SHARE` | no | **yes** | `CREATE INDEX` (non-concurrent) |
+| `SHARE ROW EXCLUSIVE` | no | **yes** | `ADD FOREIGN KEY`, `CREATE TRIGGER` |
+| `ACCESS EXCLUSIVE` | **yes** | **yes** | most `ALTER TABLE`, `DROP`, `TRUNCATE`, `REINDEX`, `VACUUM FULL` |
+
+`ACCESS EXCLUSIVE` conflicts with every mode including `ACCESS SHARE`. That is why a plain `SELECT` is enough to make you wait, and why waiting is enough to stop the table.
+
+| Operation | Lock | Scan or rewrite | Verdict / safe form |
+| --- | --- | --- | --- |
+| `ADD COLUMN` nullable, no default | `ACCESS EXCLUSIVE` | catalog only | **Safe.** Microseconds of lock; the only risk is the queue (section 3). |
+| `ADD COLUMN ... DEFAULT 'x'` (constant) | `ACCESS EXCLUSIVE` | catalog only, PG 11+ | **Safe on PG 11+** — the default is stored in `pg_attribute.attmissingval` and materialised on read. On PG ≤ 10 this is a full rewrite. |
+| `ADD COLUMN ... DEFAULT gen_random_uuid()` / any volatile default, or `GENERATED ALWAYS AS (...) STORED` | `ACCESS EXCLUSIVE` | **full rewrite** | **Never.** Every row needs a distinct value. Add nullable → backfill → `ALTER COLUMN ... SET DEFAULT` as a separate statement. |
+| `SET NOT NULL` | `ACCESS EXCLUSIVE` | **full scan** | **Never directly.** Use the `CHECK` two-step below. |
+| `ALTER COLUMN TYPE` (int→bigint, numeric→bigint, narrowing a varchar) | `ACCESS EXCLUSIVE` | **full rewrite + every index rebuilt** | **Never on a live table.** Section 6. |
+| `ALTER COLUMN TYPE varchar(50)→varchar(100)` or `→text` | `ACCESS EXCLUSIVE` | catalog only | Safe: binary-coercible widening, no rewrite (PG 9.2+). |
+| `ADD FOREIGN KEY` | `SHARE ROW EXCLUSIVE` on **both** tables | full scan of the child | Blocks writes on both tables for the whole scan. Use `NOT VALID` then `VALIDATE`. |
+| `ADD CONSTRAINT ... CHECK` | `ACCESS EXCLUSIVE` | full scan | Use `NOT VALID` then `VALIDATE`. |
+| `... NOT VALID` (check or FK) | `ACCESS EXCLUSIVE` | none | **Safe.** Catalog-only; new and updated rows are enforced immediately, existing rows unchecked. |
+| `VALIDATE CONSTRAINT` | `SHARE UPDATE EXCLUSIVE` (+ `ROW SHARE` on the parent, for an FK) | full scan | **Safe.** Reads and writes continue for the whole scan. Let it run for an hour. |
+| `CREATE INDEX` | `SHARE` | full build | **Never on a live table.** Reads work, every write blocks until the build finishes. |
+| `CREATE INDEX CONCURRENTLY` | `SHARE UPDATE EXCLUSIVE` | two full scans | **The safe form.** Cannot run inside a transaction. Can leave an INVALID index — see below. |
+| `DROP INDEX` | `ACCESS EXCLUSIVE` | none | Use `DROP INDEX CONCURRENTLY`. |
+| `DROP COLUMN` | `ACCESS EXCLUSIVE` | catalog only | Instant, and it does **not** reclaim disk — values stay in the heap until a rewrite. The risk is entirely app-side (posture #2). |
+| `RENAME COLUMN` / `RENAME TABLE` | `ACCESS EXCLUSIVE` | catalog only | Instant at the database, impossible for the application. Section 5. |
+| `SET DEFAULT` / `DROP DEFAULT` / `DROP NOT NULL` | `ACCESS EXCLUSIVE` | catalog only | Safe. |
+| `TRUNCATE`, `VACUUM FULL`, `CLUSTER` | `ACCESS EXCLUSIVE` | full rewrite | Never on a live table. `pg_repack` rebuilds without the long lock. |
+
+**MySQL/InnoDB is a trap here**: `ALGORITHM=INPLACE, LOCK=NONE` covers many of these, but `ADD COLUMN` on a table with a `FULLTEXT` index, or any change to the clustered key, falls back to a copy. State `ALGORITHM=INPLACE, LOCK=NONE` explicitly so the server *errors* instead of silently copying the table.
+
+### `SET NOT NULL`, safely
+
+```sql
+ALTER TABLE users ADD CONSTRAINT users_email_nn CHECK (email IS NOT NULL) NOT VALID;
+--  ^ catalog-only; enforced for every new and updated row from this instant
+ALTER TABLE users VALIDATE CONSTRAINT users_email_nn;
+--  ^ full scan under SHARE UPDATE EXCLUSIVE: reads and writes keep flowing
+ALTER TABLE users ALTER COLUMN email SET NOT NULL;
+--  ^ PG 12+ sees the valid CHECK and skips the scan: ACCESS EXCLUSIVE for microseconds
+ALTER TABLE users DROP CONSTRAINT users_email_nn;   -- optional; now redundant
+```
+
+On PG 11 and below the third statement rescans regardless: keep the `CHECK` and never `SET NOT NULL`.
+
+### `CREATE INDEX CONCURRENTLY` and the INVALID index
+
+CIC does two passes and waits for every transaction that could see the table to finish. If it is cancelled, deadlocks, or hits a unique violation on pass two, **it leaves an index behind that no query will use but every write must maintain.** Nothing tells you. Detect it:
+
+```sql
+SELECT c.relname, i.indisvalid, i.indisready FROM pg_index i
+JOIN pg_class c ON c.oid = i.indexrelid WHERE NOT i.indisvalid OR NOT i.indisready;
+```
+
+Recovery: `DROP INDEX CONCURRENTLY <name>;` then retry the build, or on PG 12+ `REINDEX INDEX CONCURRENTLY <name>;` to rebuild in place. Check `indisvalid` **as the last step of every migration that builds an index** — an unchecked CIC is how a table pays for four indexes and uses three.
+
+## 3. The Lock Queue Trap
+
+Postgres queues lock requests in arrival order, and a *pending* request blocks everything that arrives after it and conflicts with it — even requests that would not conflict with the lock currently held. So:
+
+```
+t=0     analytics SELECT starts, holds ACCESS SHARE for 40s
+t=1     ALTER TABLE requests ACCESS EXCLUSIVE   -> waits
+t=1.01  every new SELECT requests ACCESS SHARE  -> waits behind the ALTER
+t=1.2   pool exhausted -> 500s on endpoints that never touch this table
+```
+
+One slow `SELECT` turns a 2ms `ALTER` into a full outage. The symptom: a vertical line on the latency graph, `pg_stat_activity` full of `wait_event_type = 'Lock'`, and a migration log showing nothing at all, because the statement has not started yet.
+
+**Every migration sets both timeouts. Postgres defaults `lock_timeout` and `statement_timeout` to `0`, meaning wait forever.**
+
+```sql
+BEGIN;
+SET LOCAL lock_timeout = '2s';
+-- 2s: at 500 req/s, 2s of queueing parks ~1,000 requests and a 20-connection pool
+-- saturates in ~40ms, so 2s is already the outer edge of a survivable stall. Below
+-- ~1s you lose the race even when the table is idle between two short queries.
+SET LOCAL statement_timeout = '10s';
+-- statement_timeout covers the lock wait too, so it must exceed lock_timeout to stay
+-- distinguishable from it. 10s caps a catalog-only DDL that has gone wrong.
+ALTER TABLE orders ADD COLUMN currency text;
+COMMIT;
+```
+
+`SET LOCAL`, not `SET`: it reverts at commit instead of leaking a 10-second `statement_timeout` into whatever the pooled connection does next.
+
+Failure is a specific SQLSTATE, and the two mean opposite things:
+
+| SQLSTATE | Cause | Retry? |
+| --- | --- | --- |
+| `55P03` `lock_not_available` | `lock_timeout` fired — you lost the race | **Yes.** This is the expected outcome, not an error. |
+| `57014` `query_canceled` | `statement_timeout` fired — the work itself is too big | **No.** Retrying just repeats the outage. Rewrite the statement. |
+
+```bash
+for i in 1 2 3 4 5; do
+  psql -v ON_ERROR_STOP=1 -f 0042_add_currency.sql && exit 0
+  echo "attempt $i lost the lock race; retrying in 10s"
+  sleep 10   # 10s: long enough for a typical slow query to finish and release
+done
+echo "5 attempts failed — something is holding this table; do not force it" >&2; exit 1
+```
+
+After five failures the answer is not a sixth attempt. Find the holder:
+
+```sql
+SELECT pid, state, wait_event_type, now() - xact_start AS xact_age, left(query, 80)
+FROM pg_stat_activity WHERE now() - xact_start > interval '1 minute' ORDER BY xact_start;
+```
+
+`idle in transaction` is the usual culprit — a connection that ran one `SELECT` and never committed, holding `ACCESS SHARE` indefinitely. `idle_in_transaction_session_timeout` (default `0`) set to `60s` on the app's role removes the class permanently. Never `pg_terminate_backend` a production connection on your own judgment. And the second-most-common holder is **your own migration runner**: most wrap the whole file in one transaction, so `ACCESS EXCLUSIVE` on table A is held until the statement on table F commits. One table per file.
+
+## 4. Backfills
+
+**Never one `UPDATE`.** On 40 million rows a single statement holds a row lock on every row it touches until commit, writes a new row version for each (the table can double in size), pins the `xmin` horizon so `VACUUM` reclaims nothing anywhere, floods WAL until replicas lag by minutes, and discards all of it if it fails at 95%.
+
+The batch statement is bounded by primary key range and made idempotent by its own predicate:
+
+```sql
+UPDATE orders
+   SET currency = 'USD'
+ WHERE id >= $1 AND id < $2      -- keyset range: index scan, constant cost per batch
+   AND currency IS NULL;          -- idempotent: re-running a batch is a no-op
+```
+
+```python
+BATCH = 5_000    # ~50-200ms per batch on an indexed table: short enough that a
+                 # concurrent writer's lock wait never approaches an HTTP timeout
+PAUSE = 0.2      # 200ms between batches gives autovacuum and replication room;
+                 # ceiling ~25k rows/s, which finishes 40M rows in ~30 minutes
+
+lo = min_id
+while lo <= max_id:
+    with conn:                       # one committed transaction per batch
+        cur.execute(BATCH_SQL, (lo, lo + BATCH))
+    checkpoint(lo)                   # persist progress: a row in a table, or a file
+    lo += BATCH
+    time.sleep(PAUSE)
+```
+
+| Requirement | How | Symptom when missing |
+| --- | --- | --- |
+| Resumable | Persist the last committed id | A crash at hour 3 means starting over |
+| Idempotent | Predicate excludes done rows (`AND currency IS NULL`) | Re-running double-applies an increment or overwrites a newer value |
+| Bounded | Fixed PK range per batch | `LIMIT`/`OFFSET` re-walks the prefix; batches get slower as it runs |
+| Throttled | `sleep` between batches | Replica lag climbs until reads served from replicas go stale |
+| Observable, killable | Log rows done / total every 20 batches; hold nothing across the `sleep` | Nobody can answer "how long left", and Ctrl-C costs more than one batch |
+
+Watch replication while it runs: `SELECT client_addr, pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_bytes FROM pg_stat_replication;`. If `replay_bytes` grows across three consecutive samples, the backfill is outrunning replication — raise `PAUSE` until it flattens. Do not pick a lag threshold you did not measure; the shape of the curve is the signal.
+
+## 5. Zero-Downtime Column Rename
+
+`ALTER TABLE users RENAME COLUMN email TO email_address` is instant, free, and unshippable: there is no app version that works both before and after it, so the rollout window is guaranteed 500s. The recipe replaces one instant statement with five deploys.
+
+- **D1**: `ALTER TABLE users ADD COLUMN email_address text;` — catalog-only, instant — plus an app version whose every write path sets both `email` and `email_address`. Reads still use `email`.
+- **D2 (job, not a deploy)**: backfill per section 4 — `SET email_address = email WHERE id >= $1 AND id < $2 AND email_address IS NULL`.
+- **D3**: verify, then constrain, then switch reads.
+
+```sql
+-- IS DISTINCT FROM, never <>: <> is NULL when either side is NULL, so the check
+-- would pass while every unbackfilled row is wrong ([data-modelling.md] §3).
+SELECT count(*) FROM users WHERE email IS DISTINCT FROM email_address;  -- must be 0
+ALTER TABLE users ADD CONSTRAINT users_ea_nn CHECK (email_address IS NOT NULL) NOT VALID;
+ALTER TABLE users VALIDATE CONSTRAINT users_ea_nn;
+ALTER TABLE users ALTER COLUMN email_address SET NOT NULL;
+```
+
+- **D4**: `ALTER TABLE users ALTER COLUMN email DROP NOT NULL;` **first** — otherwise the first insert from code that no longer writes `email` fails with a not-null violation. Then deploy the version that stops writing `email`.
+- **D5**, after the rollback window: `ALTER TABLE users DROP COLUMN email;`
+
+**When you cannot deploy code to every writer** — an admin tool, another team's service, a legacy import — keep the columns in sync with a trigger instead of dual-writing. `CREATE TRIGGER` takes `SHARE ROW EXCLUSIVE`: reads continue, writes block for the moment it takes. Drop it at D4.
+
+```sql
+CREATE FUNCTION users_sync_email() RETURNS trigger AS $$
+BEGIN NEW.email_address := coalesce(NEW.email_address, NEW.email); RETURN NEW; END $$
+LANGUAGE plpgsql;
+CREATE TRIGGER users_sync_email_trg BEFORE INSERT OR UPDATE ON users
+  FOR EACH ROW EXECUTE FUNCTION users_sync_email();
+```
+
+The updatable-view variant — rename the table, create a view under the old name — is only for a client that genuinely cannot be changed (a BI tool, a partner's query). It leaves permanent indirection; do not reach for it because five deploys felt like a lot.
+
+## 6. Zero-Downtime Type Change
+
+`ALTER COLUMN amount TYPE bigint` rewrites the entire table and rebuilds every index on it, under `ACCESS EXCLUSIVE`, needing free disk for a second copy of the table plus all its indexes. On 40 million rows that is minutes to hours of total unavailability. It is the same shape as the rename, plus indexes:
+
+1. `ALTER TABLE orders ADD COLUMN amount_minor bigint;` — catalog-only.
+2. Dual-write via app code or the trigger from section 5, applying the conversion: `NEW.amount_minor := round(NEW.amount * 100)::bigint`.
+3. Backfill in batches with the identical conversion, `AND amount_minor IS NULL`.
+4. Verify: `SELECT count(*) FROM orders WHERE amount_minor IS DISTINCT FROM round(amount * 100)::bigint;` must be 0. Run it twice, minutes apart — once is a snapshot, twice proves the dual-write is holding.
+5. `CHECK ... NOT VALID` → `VALIDATE` → `SET NOT NULL`, then `CREATE INDEX CONCURRENTLY` every index the new column needs **before** switching reads. Switching reads to an unindexed column is a self-inflicted seq scan.
+6. Switch reads. Then stop writing `amount` and drop the trigger. Then, after the window, drop `amount`.
+
+**`int` → `bigint` on a primary key is not this recipe.** It drags every foreign key column in every child table with it, plus the sequence, plus every index on those columns. Do the child tables first, one at a time, and schedule it as a project. Treating it as one migration is how a Saturday becomes a Sunday.
+
+## 7. Migration Hygiene
+
+**Never edit a migration that has already run.** The symptom is a checksum failure on the next deploy — Flyway's `Validate failed: migration checksum mismatch`, Django's `InconsistentMigrationHistory` — or, worse, no error at all and two environments with different schemas claiming the same version. Write a new migration that changes the thing. Yes, for a typo in a column name. The only editable migration is one that has run *nowhere*, and "nowhere" includes CI and a colleague's laptop.
+
+**One logical change per file.** The failure unit is the file: a file that adds a column, backfills it, and creates an index fails at the index and leaves you reconstructing what already applied. And a multi-table file holds every one of its `ACCESS EXCLUSIVE` locks until the last statement commits.
+
+**Some statements cannot run in a transaction**, and most runners wrap files in one by default: `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `REINDEX CONCURRENTLY`, `VACUUM`, and `ALTER TYPE ... ADD VALUE` before PG 12. The symptom is SQLSTATE `25001`, `CREATE INDEX CONCURRENTLY cannot run inside a transaction block`. The escape hatches: Django `atomic = False`, Rails `disable_ddl_transaction!`, Alembic an autocommit block. Use them, and put nothing else in that file — a non-transactional file that fails halfway leaves partial DDL applied.
+
+**Forward-only in practice.** Write the down migration — it documents the inverse and costs five minutes — and do not plan to run it. A down migration that drops a column deletes the data the rollback was supposed to preserve, and it has never been tested. The real rollback of a schema change is a forward migration; the real rollback of a *deploy* is the previous image, which only works because expand/contract kept the schema compatible with it. That is what steps 1–7 are buying.
+
+**Time it on production-shaped volume.** A migration is 30ms on 1,000 rows and a 40-minute outage on 40 million. Restore a snapshot, run it, record the measured duration in a comment at the top of the file. And run it **under concurrent load** — a `pgbench` script or a loop of `SELECT`s against the same table — because a DDL against an idle copy never encounters the lock queue, which is failure mode #1.
+
+## 8. Data Migrations Are Not Schema Migrations
+
+| Axis | Schema migration | Data migration |
+| --- | --- | --- |
+| Changes | The catalog | Rows |
+| Duration | Milliseconds | Minutes to hours |
+| Transactions | One, the whole file | Many, one per batch |
+| Rerun safety | Guarded by the version table | Must be idempotent by its own predicate |
+| Partial failure | Nothing applied | Half applied |
+| Runs | At deploy, before the app boots | As a monitored job, after the deploy |
+
+Mixing them in one file makes both unrunnable, for three compounding reasons. The deploy now blocks on the data step, so a 40-minute `UPDATE` is a 40-minute deploy that the orchestrator's own deadline (`activeDeadlineSeconds`, a CI job limit) kills mid-way. The file's single transaction makes batching impossible by construction. And when the data step fails, the version table records the migration as *not applied*, so the next run replays the DDL that already succeeded.
+
+**The migration adds the column. A separate resumable job fills it.** The only `UPDATE` allowed in a migration file is one bounded to a handful of rows — seeding a lookup table of under ~1,000 rows, or setting a value on a single config row.
+
+## 9. Pre-Flight Checklist
+
+Run top to bottom against any migration touching a table serving traffic. A blocking answer stops the migration, not the checklist.
+
+| # | Question | Blocking answer |
+| --- | --- | --- |
+| 1 | Measured row count of the target table? | "It's small" with no number. Run `SELECT reltuples::bigint FROM pg_class WHERE relname = 'x'`. |
+| 2 | Does any statement take `ACCESS EXCLUSIVE` **and** scan or rewrite? | Yes → rewrite it using section 2. |
+| 3 | Are `lock_timeout` and `statement_timeout` set with `SET LOCAL` in the file? | No. |
+| 4 | Is there a retry loop around the lock acquisition? | No. |
+| 5 | Long-running query or `idle in transaction` on this table right now? | Yes → clear it or migrate later. |
+| 6 | Is the **currently running** app version still correct after this applies? | No → this is a contract step; split it. |
+| 7 | Is the **next** app version correct before this applies? | Ordering is [deploy-and-config.md](deploy-and-config.md). |
+| 8 | Index built with `CONCURRENTLY`, outside a transaction, with an `indisvalid` check after? | No. |
+| 9 | Backfill batched, resumable, idempotent, throttled — and outside the migration file? | No. |
+| 10 | Timed on a production-shaped copy, under concurrent load? | No. |
+| 11 | Does the down migration destroy data? | Then the rollback plan is a forward migration — say so out loud. |
+| 12 | Rewrite or index build: is free disk ≥ 2× (table + indexes)? | Unknown. |
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| `SET NOT NULL` on a live table | `CHECK ... NOT VALID` → `VALIDATE` → `SET NOT NULL` |
+| `CREATE INDEX` without `CONCURRENTLY` | `CONCURRENTLY`, outside a transaction, then check `indisvalid` |
+| `ADD FOREIGN KEY` / `ADD CHECK` in one statement | `NOT VALID`, then `VALIDATE CONSTRAINT` |
+| `ADD COLUMN ... DEFAULT gen_random_uuid()` or any volatile default | Nullable column → backfill → `SET DEFAULT` separately |
+| `ALTER COLUMN TYPE` on a large table | New column + dual-write + backfill + swap (section 6) |
+| A migration with no `lock_timeout`, or retrying a `57014` | `SET LOCAL lock_timeout = '2s'` + a retry loop; only `55P03` is retryable |
+| One `UPDATE` over the whole table, or a backfill inside a migration file | A batched, checkpointed loop in a separate monitored job (section 4) |
+| Editing a migration that has run anywhere | A new migration |
+| Dropping a column in the deploy that stops writing it | At least one deploy gap; one week by default |
+| `RENAME COLUMN` on a table an app is using | The five-deploy recipe (section 5) |
+| Several tables' DDL in one file | One table per migration |
+| A down migration you intend to run in production | A forward fix |
+| `VACUUM FULL` / `CLUSTER` / `TRUNCATE` on a live table | `pg_repack`, or accept the bloat |
+| "It ran fine in staging" | Staging has no traffic and no lock queue; time it under load |
+
+## No Guessing
+
+- **Never state a duration you did not measure, and never call a lock "brief" from memory.** The timing comes from a restored production-shaped snapshot; the lock comes from the table in section 2 or from `pg_locks`.
+- **Never invent a batch size, a pause, or a timeout.** Use the values here or the project's own config, and say which.
+- **Never trust the ORM's generated DDL, or assume the runner's transaction behaviour.** Django `AlterField`, Rails `change_column`, and Alembic `alter_column` all emit the dangerous single-statement form by default, and a wrapped file both fails on `CREATE INDEX CONCURRENTLY` and holds six locks for six `ALTER`s. Print the SQL and read it: `manage.py sqlmigrate app 0042`, `alembic upgrade <rev> --sql`. A generated migration is a draft.
+
+## Output
+
+Write the migration. Then, briefly: the measured row count of the target table, the lock each statement takes and for how long, which expand/contract step this is and which deploy it may share, the backfill's batch size and where progress is checkpointed, and the timing you measured on a production-shaped copy under load. If the honest answer is "this is three files across three deploys, and today's file only adds a nullable column", say exactly that and ship the one file. If the change would be safe only in a maintenance window, name the window rather than inventing a clever rewrite that isn't actually safe.

package/dist/resources/skills/builtin/backend-engineering/references/observability.md

@@ -0,0 +1,301 @@
+# Observability
+
+This file owns knowing what the service did: structured logging, request and trace ids, metrics, tracing, health and readiness endpoints, and alerts. **Measuring** latency — benchmarks, load tests, profiling, query plans — is [testing-and-perf.md](testing-and-perf.md); this file owns the instrumentation that produces the numbers, not the exercise that stresses them. The error taxonomy and the stable `code` that lands in `error_code` are [validation-and-errors.md](validation-and-errors.md). Which credential values must never be written anywhere is [auth-and-secrets.md](auth-and-secrets.md); the redaction *mechanism* is here. Queue depth, consumer lag, and DLQ alerts are here, delegated from [concurrency-and-jobs.md](concurrency-and-jobs.md). Probe timings that interact with graceful shutdown and pool config are [deploy-and-config.md](deploy-and-config.md).
+
+`kerja-agentik` owns how you debug a failing test and how you verify a fix. This file is the layer underneath: what production writes down so that a debugging session is possible at all, six hours after the fact, with no reproduction.
+
+## Operating Posture
+
+Two failure modes, and the first is worse:
+
+1. **The logs cannot answer "what happened to request X?"** A customer sends a screenshot and a timestamp. You have four million lines across six services and no key that joins them. Every log line was written correctly and the system is still opaque. This is not a gap in coverage — it is the absence of a join key, and no amount of extra logging fixes it after the fact.
+2. **Everything is instrumented and nothing is actionable.** ERROR fires 4,000 times a day for a handled 404. A metric labelled by user id OOMs the Prometheus server at 03:00, so you lose all metrics at the exact moment you need them. Traces cost more per month than the database and nobody has opened one.
+
+**Never invent an SLO, a probe timeout, a histogram bucket, an alert threshold, or a sampling rate.** Take them from the tables below or from the project's own config. A made-up `99.9%` reads exactly like a negotiated one, and the first time it is missed someone will treat it as a commitment.
+
+## The One Question
+
+Everything here serves one test:
+
+> Take a `request_id` off a response header. Reconstruct what that request did — every log line it wrote, every downstream call it made, the status it returned, and how long it took.
+
+If a single `grep` on that id does not produce that story, the rest of this file is decoration. Run the test before adding a dashboard. In VibeCode, `BashOutput`'s `filter` regex is that grep against a running server.
+
+**Permission to add nothing:** a service that already emits structured lines with a request id, exports RED metrics, and has split probes needs no new instrumentation. Adding a second logger beside the framework's, or a bespoke metrics registry beside the OTel one, is a defect. Say the coverage is adequate and stop.
+
+## Structured Logs — One Event, One Line, Fixed Fields
+
+JSON to stdout. The process does not manage files, rotation, or shipping — the platform does ([deploy-and-config.md](deploy-and-config.md)).
+
+| Field | Type / example | Source | Why it is mandatory |
+| --- | --- | --- | --- |
+| `timestamp` | RFC 3339 UTC, ms precision — `2026-09-02T14:03:11.482Z` | logger | Sorting across services needs one format and one zone. Local time is unsortable. |
+| `level` | one of the six below | call site | The only routing input; see the level table. |
+| `message` | short **constant** string — `"order rejected"` | call site | Constant so N occurrences group into one row. Interpolated ids make every line unique and unaggregatable. |
+| `request_id` | ULID / uuidv7 | edge middleware | The join key. This is the field the One Question depends on. |
+| `trace_id` | 32 lowercase hex (W3C) | propagated or generated at the edge | Joins the story across process boundaries. |
+| `tenant_id` | uuid | the session — **never** the request body | Scoping every query and the blast radius of every incident. |
+| `route` | matched template — `/orders/{id}` | the router | Bounded, aggregatable. A raw path is not. |
+| `method` | `GET` / `POST` / … | request | Half of "which endpoint". |
+| `status` | int — `200`, `409` | response | The other half of every RED metric. |
+| `duration_ms` | int, from a **monotonic** clock | timer | Wall-clock subtraction yields negative durations after an NTP step. `performance.now()`, `time.monotonic()`, `time.Since`. |
+| `error_code` | stable code from [validation-and-errors.md](validation-and-errors.md) | error path | Clients branch on it, dashboards group by it, humans never parse a sentence. |
+
+Everything else goes under a nested `ctx` object. **The top level is closed.**
+
+Why a fixed set beats ad-hoc keys, in symptoms rather than principles:
+
+- Three services write `userId`, `user_id`, and `uid`. The one query that would have found the incident matches a third of the lines, and looks like it worked.
+- Log backends index by field name. Free-form keys mean an unbounded index — the same cardinality failure as metrics, in a system with worse alerting on it.
+- A dashboard built on `duration` silently returns nothing after someone renames it `latency_ms`. Nothing errors. The panel is just empty, and empty reads as healthy.
+
+Two call-site rules: **log the event, not the sentence** — `log.info("order rejected", {order_id, error_code})`, never an interpolated string someone will regex badly at 03:00 — and **never log inside a per-row loop**; a 10,000-row import that logs each row costs more than the import. Log the batch: count, duration, failures.
+
+### Log Levels — Real Criteria
+
+Six levels, mapping onto RFC 5424 severities. The criteria are about *who acts*, not about how bad it feels.
+
+| Level | Criterion | Example | Who acts |
+| --- | --- | --- | --- |
+| `FATAL` | The process cannot continue and is exiting now. | Missing required config at boot, port already bound | Deploy owner, immediately |
+| `ERROR` | Something is broken that a **human must fix**. Unhandled exception, a `5xx` you returned, a job in the DLQ. | `500` from an unhandled `KeyError`, Postgres unreachable | On-call, this shift |
+| `WARN` | Degraded but handled — the system recovered, and a pattern of these is a problem. | Retry succeeded on attempt 3, cache miss storm, deprecated endpoint hit | Nobody tonight; someone this sprint |
+| `INFO` | One line per request, plus state changes that matter to the business. | Request completed, order paid, user deleted | Nobody. It is evidence, not a signal. |
+| `DEBUG` | Developer detail, off in production. | Query text, branch taken | Nobody |
+| `TRACE` | Firehose, on for minutes at a time under a flag. | Loop internals | Nobody |
+
+**An expected, handled `404` is INFO, not ERROR.** So is a `400` from a client that sent bad JSON, a `401` from an expired token, and a `409` from a conflicting write. Those are the API doing its job — they are all in the taxonomy at [validation-and-errors.md](validation-and-errors.md) precisely because they are normal. The rule is mechanical:
+
+```
+4xx you returned deliberately  -> INFO (WARN if the rate itself is the anomaly)
+5xx you returned               -> ERROR
+5xx you received and retried   -> WARN on the attempt, ERROR only if the retries were exhausted
+```
+
+**An ERROR nobody acts on trains everyone to ignore ERROR.** That is the actual cost, and it is paid once, permanently: after a month of 4,000 daily handled-404 ERRORs, the real one scrolls past. If a line fires more than a handful of times a day and no one has ever changed anything because of it, it is INFO or it is deleted. Demoting noisy ERRORs is real work, not tidying.
+
+### Never Logged
+
+| Never | Why |
+| --- | --- |
+| Passwords, in any form, including "invalid" ones | Users mistype one account's password into another's login form |
+| `Authorization` / `Cookie` / `Proxy-Authorization` headers | A bearer token in a log is a valid credential in a log |
+| Tokens, API keys, refresh tokens, session ids, signed URLs | Same; a session id is a login |
+| Full card numbers, CVV, full bank account numbers | PCI DSS 3.4 — PAN unreadable wherever stored; log the last 4 only |
+| Whole request bodies, whole config objects, whole exception locals | These are the three ways every secret above gets logged by accident |
+| National ids, full addresses, medical or biometric data | Retention on a log pipeline is not a privacy review |
+
+**Redact at the logger, not at each call site.** A per-call-site rule is a rule you will forget on exactly the line that mattered, and the failure is silent — the log looks fine until someone reads it.
+
+```js
+// pino: one config, applies to every logger in the process
+pino({ redact: {
+  paths: ['req.headers.authorization', 'req.headers.cookie', 'password',
+          'token', 'refresh_token', 'card.number', '*.password', 'ctx.*.token'],
+  censor: '[REDACTED]' } })
+```
+
+```python
+# structlog: a processor in the chain; nothing reaches a renderer unfiltered
+DENY = {"password", "token", "authorization", "cookie", "secret", "api_key", "card_number"}
+def scrub(_, __, ed): return {k: ("[REDACTED]" if k.lower() in DENY else v) for k, v in ed.items()}
+```
+
+A secret that reached the log is leaked, full stop — the log is replicated, indexed, and retained. Deleting the line is not the fix; rotation is ([auth-and-secrets.md](auth-and-secrets.md) *When a Key Leaks*).
+
+Keep **100% of WARN and above** within retention. Sample INFO on high-traffic routes (`1:10` above ~100 rps on one route is a normal start), but **sample by `trace_id`, not per line** — half a story is worse than none, because it looks complete.
+
+## Request Id and Trace Id
+
+```
+traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
+             ^ver ^trace-id (32 hex)                ^span-id (16 hex)  ^sampled
+X-Request-Id: 01JB6X2K8P9QW3ZR7T4V5N0MDC
+```
+
+`traceparent` is W3C Trace Context (a W3C Recommendation, not an RFC) and is what OpenTelemetry propagates. `X-Request-Id` is a de-facto convention with no spec — universal anyway, and it is the one a human can read aloud over the phone.
+
+The rules:
+
+1. **Generated at the edge when absent.** One middleware, first in the chain, before auth, before body parsing. If a request reaches a handler with no id, the middleware is in the wrong position.
+2. **Bound to the ambient context, not passed as an argument.** `AsyncLocalStorage` in Node, `contextvars` in Python, `context.Context` in Go. An id threaded by hand is an id someone drops in the one helper that mattered.
+3. **On every log line**, via the logger's context binding — never by remembering to add it.
+4. **Propagated to every downstream call**: HTTP headers, RPC metadata, and the **job payload** when work is enqueued, so an async consumer's logs join the request that created it ([concurrency-and-jobs.md](concurrency-and-jobs.md)).
+5. **Returned in a response header** on every response, success and failure, so the user can quote it. A `500` body carries the same id ([validation-and-errors.md](validation-and-errors.md)) — that pairing is what turns a support ticket into a `grep`.
+6. **At a public edge, do not trust an inbound id.** Validate the `traceparent` format or generate a fresh one and record the client's as `ctx.upstream_trace_id`. An attacker who can set your trace id can forge log correlation and inflate your trace bill. Between your own services, behind the gateway, trust it — that is the entire point.
+
+## Metrics
+
+Three frameworks, one job each. They overlap; that is fine.
+
+| Framework | Applies to | Signals |
+| --- | --- | --- |
+| **RED** | Request-driven services (HTTP, RPC, queue consumers) | **R**ate, **E**rrors, **D**uration |
+| **USE** | Resources (CPU, pool, disk, queue) | **U**tilization, **S**aturation, **E**rrors |
+| **Four golden signals** (Google SRE) | Anything user-facing | Latency, traffic, errors, saturation |
+
+The minimum export for an HTTP service, in OTel semantic-convention names:
+
+```
+http.server.request.duration      histogram, seconds, labels: http.route, http.request.method, http.response.status_code
+http.server.active_requests       gauge      (saturation: in-flight)
+db.client.connection.count        gauge, by state=used|idle  (pool saturation — the first thing to go)
+messaging.client.consumed.messages counter   (queue throughput)
+```
+
+Plus, from [concurrency-and-jobs.md](concurrency-and-jobs.md): queue depth, oldest-message age, and DLQ depth as gauges. **Oldest-message age beats depth** — a depth of 500 that drains in 4 seconds is healthy; a depth of 3 where the oldest is 40 minutes old means a consumer is stuck.
+
+### Histograms, Not Averages
+
+**An average latency is a number that describes nobody.** 100 requests at 10 ms and 1 at 10,000 ms average to 108 ms. No user experienced 108 ms; 100 had a fast page and 1 gave up. Symptom: the incident channel says "it's slow", the dashboard says 42 ms, and both are correct.
+
+Export a histogram and read p50 / p95 / p99. Buckets must straddle your actual latencies — the Prometheus Go client's `DefBuckets` is:
+
+```go
+DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}  // seconds
+```
+
+If your endpoint answers in 2 ms, every request lands in the first bucket and p50, p95 and p99 all report `≤5ms` — a flat, useless line. Set buckets around the real distribution, measured first ([testing-and-perf.md](testing-and-perf.md)).
+
+**Never average percentiles.** The mean of four instances' p99 is not the fleet p99 and has no statistical meaning. Aggregate the raw buckets, then compute the quantile — `histogram_quantile(0.99, sum by (le, route) (rate(bucket[5m])))`. Averaging p99s is the most common wrong dashboard in production.
+
+### Cardinality — The Label That Kills the Backend
+
+**A time series exists for every distinct combination of label values.** The count is a product, not a sum, and it grows while you watch.
+
+```
+routes(20) × methods(3) × status(5)              =        300 series   fine
+              × user_id(50,000)                  = 15,000,000 series   dead
+```
+
+At the long-standing Prometheus rule of thumb of roughly 3 kB of resident memory per active series, 15 million series is about 45 GB of RAM for one label. Symptom: the scrape times out, the server OOMs and restarts, and **you lose every metric in the fleet at the moment of the incident that caused the spike.** Cardinality does not degrade gracefully; it falls over.
+
+| Label | Allowed? | Note |
+| --- | --- | --- |
+| `route` (matched **template**) | Yes | Bounded by the router: number of routes |
+| `method`, `status_code` or `status_class` | Yes | Tiny, fixed sets |
+| `error_code` | Yes | Bounded by the taxonomy in [validation-and-errors.md](validation-and-errors.md) |
+| `tenant_id` | **Only** with a hard, known bound | Fine at 30 enterprise tenants; fatal at 300,000 self-serve accounts |
+| `user_id`, `session_id`, `order_id`, `email`, `ip` | **Never** | Unbounded by construction |
+| Raw `path` / full URL with ids or query string | **Never** | `/orders/8123` and `/orders/8124` are two series |
+| Exception message, upstream error text | **Never** | Free text is infinite cardinality wearing a label |
+
+**The route-template rule: take the label from the router's matched pattern, never from the request path.** `req.route.path` in Express, `request.scope["route"].path` in FastAPI/Starlette, `chi.RouteContext(ctx).RoutePattern()` in Go, `http.route` in OTel. `request.url.path` is the bug, and it looks identical in code review.
+
+High-cardinality dimensions belong in logs and traces, which are indexed for it. Metrics answer "how many and how slow"; logs and traces answer "which one".
+
+## Tracing
+
+**Spans at boundaries only.** A boundary is: the inbound request, an outbound HTTP/RPC call, a database query, a cache call, a queue publish or consume. That is the list.
+
+A span per function produces a 400-span trace whose flame graph is unreadable and whose export cost exceeds the work it measured. Symptom: a trace UI that takes eleven seconds to render and answers no question the logs did not. **Tracing everything produces a bill and no insight** — the useful signal is almost always "which downstream call took the 900 ms", and that needs a dozen spans, not four hundred.
+
+- **Attributes carry the high-cardinality detail** — `user.id`, `order.id`, `db.statement` (parameterised, never interpolated values). This is the correct home for what metrics may not hold.
+- **Errors are recorded on the span** (`span.recordException`, status `ERROR`), so a failing trace is filterable without a text search.
+- **Sampling:** head-based at 1–10% for steady-state volume; if the backend supports tail-based, keep **100% of traces containing an error or exceeding your p99 target** and sample the rest. Sample the *trace*, not the span — a partially sampled trace is a broken one.
+- **Context propagation is the whole value.** An unpropagated `traceparent` gives you N single-service traces and the illusion of coverage.
+
+## Health vs Readiness — Never the Same Endpoint
+
+Two endpoints, two questions, two consequences:
+
+| | Liveness `GET /livez` | Readiness `GET /readyz` |
+| --- | --- | --- |
+| Question | Is this process wedged? | Can this instance serve traffic *right now*? |
+| Checks | Process-local only. No I/O. No database. No cache. No downstream. | Dependencies it cannot serve without, with a short timeout |
+| Failure consequence | **The orchestrator kills the container** | Removed from the load-balancer pool; the pod keeps running |
+| Recovery | Restart | Automatic, on the next successful probe |
+
+**A liveness probe that checks the database causes a cascading outage, and this is the single most damaging mistake in this file.** The mechanism, step by step: the database has a 20-second blip — a failover, a saturated pool, a slow `ANALYZE`. Every instance's liveness check fails simultaneously, because they all depend on the same database. Kubernetes' default `failureThreshold: 3` with `periodSeconds: 10` means each container is killed after ~30 seconds. The orchestrator kills **all of them at once**. They restart into a cold cache and an empty pool, and hammer the recovering database with reconnects; the ones that come up are killed again on the next probe. A 20-second blip became a full outage that outlives its own cause, and the crash-loop backoff makes recovery slower the longer it runs. This is in the hub's Never Ship list as **a health check that queries the database**, and it is there for this reason.
+
+```py
+# LIVENESS — answers from memory. If the process can run this, it should not be killed.
+@app.get("/livez")
+def livez():
+    return {"status": "ok"}, 200          # no DB, no Redis, no HTTP, no disk
+
+# READINESS — dependencies, each with a timeout shorter than the probe's timeoutSeconds
+@app.get("/readyz")
+async def readyz():
+    checks = {"db": await ping_db(timeout=0.5), "cache": await ping_cache(timeout=0.2)}
+    ok = all(checks.values())
+    return {"status": "ok" if ok else "degraded", "checks": checks}, (200 if ok else 503)
+```
+
+Exact probe config, and why each number:
+
+```yaml
+livenessProbe:                # k8s defaults: periodSeconds 10, timeoutSeconds 1, failureThreshold 3
+  httpGet: { path: /livez, port: 8080 }
+  periodSeconds: 10
+  timeoutSeconds: 1           # a process too wedged to answer in 1s from memory IS wedged
+  failureThreshold: 3         # 30s of unresponsiveness before a kill — survives one GC pause
+readinessProbe:
+  httpGet: { path: /readyz, port: 8080 }
+  periodSeconds: 5            # faster than liveness: leaving the pool is cheap and reversible
+  timeoutSeconds: 2           # must exceed the sum of the dependency timeouts above (0.7s)
+  failureThreshold: 2
+startupProbe:                 # without this, a 60s boot is killed by liveness before it ever starts
+  httpGet: { path: /livez, port: 8080 }
+  periodSeconds: 5
+  failureThreshold: 30        # 150s of boot budget, then liveness takes over
+```
+
+**`/readyz` must return `503` during graceful shutdown**, before the server stops accepting connections, so the load balancer drains first ([deploy-and-config.md](deploy-and-config.md)). **Neither endpoint is authenticated, and neither leaks**: no versions, hostnames, or connection strings — `{"checks": {"db": false}}`, never `{"db": "postgres://prod-eu-1..."}`.
+
+## Alerting
+
+**Alert on symptoms users feel, not on causes.** CPU at 90% with every request served in 40 ms is not an incident; it is a well-utilised machine. A page for it is how on-call learns to silence pages.
+
+| Cause-based alert (delete) | Symptom-based alert (keep) |
+| --- | --- |
+| `cpu > 90%` | `p99 latency > SLO for 10m` |
+| `memory > 80%` | `5xx rate > 1% of requests for 5m` |
+| "A pod restarted" | "Ready replicas < 2 for 5m" |
+| "Queue depth > 1000" | "Oldest message age > 10m" |
+| "A retry happened" | "DLQ depth > 0 sustained 15m" |
+
+**Every alert names the action.** The annotation is not a description of the metric — that is on the graph. It is: what is broken from the user's point of view, the first thing to check, and the link to the runbook. An alert with no action is a dashboard; move it to one and delete the alert.
+
+```yaml
+- alert: CheckoutErrorBudgetBurnFast
+  expr: |
+    (sum(rate(http_server_requests_total{route="/checkout",status=~"5.."}[1h]))
+     / sum(rate(http_server_requests_total{route="/checkout"}[1h]))) > 14.4 * 0.001
+  for: 2m
+  annotations:
+    summary: "Checkout failing for ~1 in 70 users; 2% of the 30-day budget burns per hour"
+    action:  "Check /readyz across replicas, then payment-provider status. Runbook: <link>"
+```
+
+`14.4` is not a taste: it is the Google SRE Workbook's fast-burn multiplier — 14.4× the error budget rate consumes **2% of a 30-day budget in 1 hour**. The companion slow burn is **6× over 6 hours (5% of the budget)**, which is a ticket, not a page. Use those two windows; do not invent a third.
+
+The severity rule is one line: **page only for what a human must fix within minutes.** Everything else is a ticket. If an alert has fired twenty times and no one has ever acted, it has already been answered — delete it.
+
+## Never Ship
+
+| Never | Instead |
+| --- | --- |
+| A health check that queries the database | `/livez` process-local; `/readyz` checks dependencies |
+| One endpoint serving both probes | Two endpoints, two consequences |
+| Unbounded metric label cardinality (`user_id`, raw path) | Route template + bounded label set |
+| `request.url.path` as a metric label | The router's matched pattern / `http.route` |
+| An average latency on a dashboard | A histogram, p50/p95/p99 from raw buckets |
+| Averaging p99 across instances | `histogram_quantile` over summed buckets |
+| Default histogram buckets on a 2 ms endpoint | Buckets measured around the real distribution |
+| ERROR for a handled `404` / `400` / `401` / `409` | INFO, or WARN if the rate is the anomaly |
+| An ERROR that has never caused an action | Demote to INFO or delete it |
+| Interpolated ids in the log `message` | Constant message, ids as fields |
+| Redaction at the call site | A logger-level redact/processor config |
+| A request that reaches a handler with no `request_id` | Edge middleware, first in the chain |
+| A trace id that stops at the queue boundary | Propagate in the job payload |
+| Logging inside a per-row loop | One summary line per batch |
+| `Date.now()` deltas for `duration_ms` | A monotonic clock |
+| An alert on CPU, memory, or a restart | An alert on latency, errors, or availability |
+| An alert with no named action | A runbook link, or deletion |
+| A `/readyz` body containing a URL, version, or hostname | Booleans only |
+| An invented SLO, threshold, burn rate, or sampling rate | The values in this file, or the project's config |
+
+## Output
+
+Write the instrumentation. Then, briefly: **the join key** (where `request_id` is generated, where it is returned, and confirm you traced one request end to end), **the labels you used and their cardinality bound**, and **which alert fires if this breaks, and what it tells on-call to do**. If the honest answer was "already covered — the framework emits this and the probes are already split", say that in one line and add nothing. A service that needed no new instrumentation is a good result, not a skipped task.