@ultimat3/cli 29.4.0 → 30.1.0

package/wiki/Migrations-And-Backfills.md

@@ -0,0 +1,552 @@
+# Migrations and backfills
+
+One engine per concern. A **migration** changes the shape of a table — schema, in one ledger, applied identically from a laptop to a release phase. A **backfill** changes the rows already in it — data, as a `job`, with resume, ledger, throttle and progress built in. Confusing the two is the usual mistake: a schema change that also needs to touch every existing row is a migration *and* a backfill, never one artifact doing both.
+
+`As of 2026-08`. Source: [`packages/db/src/migrate.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/migrate.ts), [`generate.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/generate.ts), [`destructive.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/destructive.ts); [`packages/jobs/src/backfill.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/jobs/src/backfill.ts) and its siblings.
+
+## When to write which
+
+| Change | Write |
+|---|---|
+| Add, rename, retype, drop a column or table; add an index or constraint | a **migration** — `x db gen "<name>"` |
+| Add an index to a big, hot table, or drop one from it | a **migration** too — `indexes: [{ on, concurrently: true }]` for the build; every drop is concurrent already. [Indexes built concurrently](#indexes-built-concurrently--expand--contract) |
+| Rewrite, normalize, re-derive or backfill values in rows that already exist | a **backfill** — `backfill({ … })` |
+| Add a nullable column, then populate every existing row before making it required | both: the migration adds the column, the backfill sweeps the rows, a second migration adds the `not null` once the sweep is done (expand → migrate → contract) |
+
+## Migrations: one engine, one ledger
+
+**This page documents the code at 2.0.0 and two majors have shipped since** — `npm view @ultimat3/core version`, `As of 2026-08-20`. Three things shipped in 3.0.0 that this page does **not** yet describe: `x db seed` (run `x db seed --help`), `rollback({ steps: -1 })` reverting every migration but the last, and a migration deleted from the tree being invisible to the audit. **And four in 4.0.0**, three of which change generated DDL: `on delete` now reaches the emitted SQL — so any app that ever declared `references(…, { onDelete })` generates a **different** migration and should read `x db gen`'s diff before applying it — a `references()` *removed* from a column now emits a `drop constraint`, drift reports `changed-foreign-key` for a key whose rule moved, and the `x_jobs` idempotency index gains `tenant_id`. Read [`CHANGELOG.md`](https://github.com/developerz-ai/ultimate/blob/main/CHANGELOG.md)'s `3.0.0` and `4.0.0` sections beside this page until it is rewritten, and [Upgrading](Upgrading) for the edits. On a pin still at 1.x, take [Known gaps → `x db gen` / `x db migrate`](Known-Gaps) and its 1.1.0/1.2.0 workarounds instead.
+
+`x db gen` and the `ROLE=migrate` release-phase container run the **same** engine — `packages/db`'s `migrate()`/`generateMigration()` — not two. **In 1.1.0** they did not: `x db gen`'s subcommands shelled out to `bunx drizzle-kit`, a second schema engine with its own journal, declared in no `package.json` and fetched unpinned at run time, which is why a 1.1.0 scaffold's own `bin/setup` fails. That shelling-out is gone from current source — `cmd-db.ts` calls `generateAppMigration` and `runMigrations` from `@ultimat3/db`/`@ultimat3/cli` directly, and the only remaining mention of `drizzle-kit` anywhere is a file header comment recording the history.
+
+```
+x db gen "initial"            # a new app's first database command — x new writes no migration
+x db gen "add publish_at"     # diffs entities against migrations, writes a named migration + its down
+x db gen                      # nothing to generate: rewrites the schema dump and nothing else
+x db migrate                  # applies pending migrations, dev or prod, through migrate()
+ROLE=migrate                  # the same migrate() as a release-phase container, one image
+```
+
+`x db gen` is the **only** writer of `packages/db/migrations`, `As of 2026-08`. `x new` scaffolds no
+`0000_initial.sql`, so a scaffold that **declares an entity** — the default `--example` slice does —
+is red on `x verify`'s `drift` step until that one command runs: *packages/db has a schema but no
+migration recorded it*, fix `x db gen "initial"`. Zero entities against zero migrations is
+agreement, not drift, so `x new --no-example` is green until its first `entity()`. A
+scaffold that hand-wrote the first migration was a second writer of the same directory: the file
+carried no `.snapshot.json`, which only the generator can produce, so the app's first `x db gen` and
+its first `x db migrate` both refused (axiom 1).
+
+### The `x_migrations` ledger
+
+Every migration applies inside its own transaction, recorded into `x_migrations` (`LEDGER_TABLE`) as one row: id, name, checksum, `applied_at`, `app_version`, `duration_ms`. A few properties are load-bearing:
+
+- **Checksum-pinned.** An applied migration's `up` text is hashed once; if the file on disk hashes differently later, `auditLedger` throws `X_MIGRATION_CONFLICT` rather than silently re-running edited SQL — the fix is a new migration, never an edit of one already shipped.
+- **App-version fenced.** A ledger row from a build this process does not ship is also `X_MIGRATION_CONFLICT` — the contract a rolling deploy needs: a pod must refuse to migrate a database another build already owns, or two schemas race during the rollout.
+- **Except a rollback** (`As of 2026-10-05`, `migrate-rollback.ts`). When every unknown row sorts after the newest migration this build ships, and this build has nothing left to apply, an older image is being rolled back onto a newer build's schema: nothing is applied, the rows are logged as `ultimate migrate ledger ahead of build` and returned in the report's `ahead`. An unknown row between shipped ones, or a shipped migration never applied, is still a conflict.
+- **Advisory-locked, on one pinned session.** An advisory lock is scoped to a Postgres *session*, not a statement, so the lock is taken and held on a single reserved connection for the whole run (`withAdvisoryLock`) — see [Resource management](Resource-Management) for the general `using`-pin shape this follows. `lock: false` is declared as an escape hatch on both option types and, `As of 2026-08`, is passed by nothing in the repo — every shipped migration path takes the lock.
+- **Acquired by a bounded poll, never a blocking wait.** `acquireLock` tries `pg_try_advisory_lock` once per 500ms (`MIGRATION_LOCK_POLL_MS`) against a 60s budget (`MIGRATION_LOCK_WAIT_MS`, overridable per call as `lockWaitMs`), then raises `X_MIGRATE_CONCURRENT`. So a genuinely slow migrator ahead of you is waited out and two overlapping deploys still serialise — but a wedged one becomes an exit code inside the deploy window. Blocking `pg_advisory_lock` has no timeout at all: it held `helm upgrade --wait` inside one statement, printing nothing, with the job never failing so `backoffLimit` never fired.
+- **One statement per `up`/`down` send, script-split.** `applyScript` runs `statementsOf(script)` one at a time inside the migration's own transaction, because the two drivers (PGlite, pooled Postgres) disagree about sending a multi-statement script as one call.
+- **Every migration's own transaction.** `migrate()`/`rollback()` apply and reverse one migration at a time, each in its own transaction, on purpose — a failed `up` must leave the ledger describing exactly the migrations that *did* run, not an all-or-nothing batch. This loop is declared with `expectedQueryLoop` at the source, so the N+1 detector never flags it.
+- **Except one marked `-- transaction: false`** (`As of 2026-10-31`, `migration-step.ts`): run one autocommitted statement at a time, with no `lock_timeout`, its ledger row written after the last statement. `x db gen` writes it for concurrent index work; a hand-written migration may carry it too. Its statements must be idempotent — a failure leaves what ran in place and no row, and the next `x db migrate` runs the file again.
+
+### `generateMigration()` — one diff engine
+
+`x db gen` diffs the app's entity snapshots against the schema the newest migration's `.snapshot.json` sidecar recorded, and writes the migration text plus that sidecar for the *next* diff. The sidecar is the generator's diff base, in the entity's vocabulary; what the database holds, in SQL, is [the schema dump](#the-schema-dump). A migration missing its sidecar — deleted, or hand-written without one — is `X_MIGRATION_SNAPSHOT_MISSING`: refused rather than defaulted to an empty schema, which would emit `create table` for every table the database already holds. No migration at all is a different answer and not a refusal: zero migrations declare the empty schema, which is what makes `x db gen "initial"` work in an app that has never generated one.
+
+Its two remedies, in the order they are safe to run: `git checkout -- packages/db/migrations/<id>.snapshot.json`, or, when the file was never written, `rm packages/db/migrations/<id>.* && x db gen "<name>"`. The delete comes first because `x db gen` against the migration that is still there raises this same code. Until 2026-08 the `fix:` said "restore … from version control" alone while `x db migrate`'s `unknown-schema` difference answered `x db gen "snapshot <name>"` — each naming the command that raises the other, with nothing to restore in an app whose sidecar was never written.
+
+### Foreign keys are their own statement
+
+Every foreign key is `alter table … add constraint`, emitted **after** every table statement, and
+never a `references` clause inside `create table` — decided 2026-08. Inline, the constraint is
+created with the table, so the referenced table has to exist already; the order `generateMigration`
+walks is `describeEntities()`, which is the app's *import* order and says nothing about where a
+`references()` points. Measured against PGlite on a scaffolded app: `create table "comments" (…
+references "posts" …)` ahead of `create table "posts"` is `relation "posts" does not exist` on
+statement one, and `down` had the mirror fault — `drop table "posts"` while `comments` still
+references it is `2BP01`. `down` is reversed as a whole, so the constraint drops pushed last come
+out first.
+
+**No topological sort and no cycle error**, deliberately: separate constraints need no ordering at
+all, and two tables referencing each other cannot be expressed inline in any order. The same call
+site answers the second half — a `references()` added to a column that **already exists** now emits
+its own `add constraint`, where before the `up` came out empty, `x db gen` wrote no file, and the
+`drift` step stayed red forever behind a fix that did nothing.
+
+**Removing a `references()` emits the `drop constraint`**, `As of 2026-08-19` — it used to emit
+nothing, and that is not the harmless omission a removed index is. A removed index leaves the
+snapshot correct by omission; a removed key left the constraint on the database *and* wrote
+`foreignKeys: []` beside it, so the record actively denied a constraint the catalog held, and
+`compareForeignKeys` judges only the declared side. The drop names the constraint the **previous
+snapshot** recorded rather than the name this generator would have chosen, so a hand-written
+`fk_legacy` is dropped by its own name instead of `42704`. A column being dropped in the same
+migration takes its constraint with it, and no second `drop constraint` is emitted for it.
+
+**`on delete` reaches the SQL too**, `As of 2026-08-19`. `references(() => orgs.id, { onDelete:
+'cascade' })` has type-checked since 1.0 and the clause it produced was `references "orgs" ("id");`
+— a declared cascade that the database refuses the delete under instead. `ColumnDescription` and
+`ReferenceDescription` carry `onDelete`, `addForeignKey` writes the clause, and a rule changed on
+either side is `changed-foreign-key` drift whose `fix:` is the drop/add pair as one `psql -c`
+command, because Postgres has no `alter constraint` for it.
+
+The sidecar is written through `snapshotJson()`, not `JSON.stringify(value, null, 2)`. Biome
+collapses a short array onto one line and `JSON.stringify` never does, so an app whose `lint` step
+read that directory failed `x verify` on a file no author typed — *Formatter would have printed the
+following content* — as `X_LINT_FAILED`. Two fixes, both in 2.0.0: the serialiser is a
+fixed point of Biome 2.5.5 at `lineWidth: 100`, and `x new`'s `biome.json` excludes
+`**/migrations`, the glob this repo's own config already carried.
+
+## Indexes built concurrently — expand → contract
+
+`As of 2026-10-31`. `create index` holds SHARE on the table for the whole build, so every write
+waits; `drop index` takes ACCESS EXCLUSIVE, so every read waits too. On a big, hot table either one
+stalls the app or trips the migration's 3 s `lock_timeout` mid-deploy. The concurrent forms block
+neither, and Postgres refuses them inside a transaction — so they ship in a migration of their own.
+
+```ts
+import { entity, text, uuid } from '@ultimat3/entity';
+
+export const runs = entity('runs', {
+  columns: { id: uuid().primaryKey(), status: text({ max: 20 }), bank: text({ max: 40 }) },
+  indexes: [{ on: ['status'], concurrently: true }],
+});
+```
+
+```sql
+-- 20261031120000_index_status
+-- GENERATED by `x db gen` from the app's entities — do not edit.
+-- Editing an applied migration changes its checksum: X_MIGRATION_CONFLICT on the next apply.
+
+-- transaction: false
+create index concurrently if not exists "runs_status_idx" on "runs" ("status");
+
+-- down
+drop index concurrently if exists "runs_status_idx";
+```
+
+| | |
+|---|---|
+| Build | `concurrently: true` on an `indexes:` entry. On a table the same migration creates it changes nothing — the table is empty and invisible until COMMIT, so it is built in that transaction |
+| Drop | **every** removed index, concurrently built or not: delete it from `indexes`, `x db gen`. A total unique btree (a shape a UNIQUE constraint may back) is first unhooked with `alter table … drop constraint if exists` in the same step, right before its concurrent drop — `drop index concurrently` on a constraint's index is `2BP01`. That ALTER takes ACCESS EXCLUSIVE, so the runner gives it a short transaction of its own under the 3 s `lock_timeout` |
+| Files | no concurrent work (no `concurrently: true` build on a live table, no removed index) → ONE transactional migration, exactly as before. Concurrent work and nothing else → ONE `-- transaction: false` migration. Concurrent work beside other changes → TWO: `<id>.sql` (the transaction: new columns, tables, constraints, plain indexes) then `<id>_concurrently.sql` (the concurrent builds, then every drop). Each has its own snapshot and hash, and `--json` names the second as `followUp` |
+| Order | expand, then contract: inside the step every build runs before any drop or constraint unhook, across tables — a renamed key's replacement exists before the old key goes, so for a moment both exist and never neither |
+| Failure | an interrupted build leaves an INVALID index under the name. The next `x db migrate` finds it (`pg_index.indisvalid`), drops it concurrently, and builds it again; no row was written, so nothing is skipped |
+| Never recorded | `concurrently` is how an index is built, not what it is — not in the name, not in the snapshot. Adding or removing it on an existing index generates nothing |
+| Not destructive | an index holds no rows, so the destructive rail leaves a concurrent drop alone — and still requires `-- destructive: true` for a column or table drop beside it |
+
+A hand-written migration with `create index concurrently`, `drop index concurrently` or
+`reindex … concurrently` and no `-- transaction: false` line is refused before `BEGIN`:
+`X_MIGRATION_CONCURRENT_IN_TRANSACTION`.
+
+## Partitioned tables
+
+Rule: a partitioned table is a table. Declare it as an entity, `x db gen`, then hand-edit the
+generated `create table`'s closing `);` to `) partition by range ("at");`. `As of 2026-11-06`.
+No `entity({ partition })`: attach, default partition and retention are the app's choices.
+
+| Question | Answer |
+|---|---|
+| Primary key | must hold the partition key — `primaryKey: ['id', 'at']`. Postgres refuses any unique index that does not (`0A000`) |
+| Drift | the PARENT is judged: columns, key, indexes, checks, triggers. A partition is not app schema (`pg_class.relispartition`), so next month's is no `unexpected-table` and its `fix:` is never `drop table`. A key Postgres clones per partition (`conparentid <> 0`) is not counted. `introspectSchema()` marks the parent `partitioned: true` |
+| A reference INTO one | no `.references()`. A one-column key is `42830` (no unique index on `id` alone can exist) and a same-tenant key's `(org_id, id)` index is `0A000`. Hand-write the composite key into the migration — `alter table "notes" add constraint "notes_event_fkey" foreign key ("event_id", "event_at") references "events" ("id", "at");` — drift judges declared keys only, so it is no finding. The repository's tenant filter still holds on reads; the database does not enforce the same tenant |
+| Retention | `detach`, then `drop`, in the app's own job — never `retain`, never a `delete` |
+| Next month | an app `task` on a monthly cron (`tz: 'UTC'`) enqueuing a job that runs the statement below, `if not exists` so a double fire is a no-op |
+| `create index concurrently` | refused on a parent (`0A000`): `X_MIGRATION_CONCURRENT_PARTITIONED`, whose fix is the sequence — each partition concurrently, the parent `on only`, then `alter index … attach partition`, in a `-- transaction: false` migration |
+| Schema dump | rendered: the parent with `partition by`, each partition `partition of … for values` in the parent's file. A partition's attached indexes, cloned keys and cloned triggers are not written: the parent's recreate them |
+
+The statements the job runs (pinned by `packages/db/src/partition-recipe.test.ts`):
+
+```sql
+-- next month: idempotent
+create table if not exists "events_2026_12" partition of "events"
+  for values from ('2026-12-01') to ('2027-01-01');
+-- retention: the month leaves the table first, then its rows go with it
+alter table "events" detach partition "events_2026_11";
+drop table "events_2026_11";
+```
+
+## The schema dump
+
+Rule: the current schema is committed SQL under `packages/db/schema/` — written by `x db gen` and `x db migrate`, never by hand, and held equal to the migrations by `x verify`'s `drift` step (`X_SCHEMA_DUMP_DRIFT`). `As of 2026-10`. Source: [`packages/db/src/introspect-catalog.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/introspect-catalog.ts), [`schema-dump.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/schema-dump.ts), [`schema-load.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/schema-load.ts).
+
+| Directory | Holds | One file per |
+|---|---|---|
+| `01_extensions/` | `create extension if not exists` | extension |
+| `02_types/` | enums, domains | type |
+| `03_sequences/` | sequences no column owns | sequence |
+| `04_tables/` | columns, defaults, generated and identity columns, primary key, unique, check; a `serial` column's sequence; `replica identity`; `partition by`, and each partition after its parent | table — a partition rides in its root's file |
+| `05_indexes/` | indexes no constraint backs | table |
+| `06_foreign_keys/` | `alter table … add constraint` | table |
+| `07_views/` | views; materialized views with their indexes | view |
+| `08_functions/` | functions and procedures, overloads together | name |
+| `09_triggers/` | triggers, and a non-default enabled state | table |
+| `10_policies/` | `enable` / `force row level security`, then each policy with every clause (`As of 2026-11-06`) | table |
+| `framework/<same ten>/` | the framework's own tables — `x_jobs`, `x_users`, `x_migrations`, … | — |
+| `unrendered.sql` | comment lines naming what the dump cannot spell | — |
+
+- **One producer.** A scratch database, framework tables applied, every migration replayed, then read out of `pg_catalog`. Never the dev database: a table made by hand there is not in the dump. No `pg_dump`.
+- **One engine rule: PGlite unless the migrations require otherwise.** The scratch database is the embedded one. It is a real Postgres only when a migration creates an extension the installed PGlite cannot link — decided from the migrations and the installed package, never from which URL is set, so a laptop and CI choose alike.
+- **Deterministic.** Objects sorted by name in code-unit order, columns in ordinal order, Postgres' own `pg_get_*def` spellings, every sequence option written out. No timestamp, version, owner or grant. Same migrations, same bytes.
+- **A one-table change is a one-file diff.** A file whose bytes did not move is not rewritten.
+- **Load equals replay.** The `drift` step empties the scratch schema, loads the dump, and renders it again; the bytes must match. Loading runs in kind order, framework twin first, and retries a file that names something not created yet (a view over a later view, a default calling a function).
+- **500-line ceiling.** A longer file is split `<name>.1.sql`, `<name>.2.sql`, …; the loader joins the parts in numeric order.
+- **`packages/db/.gitattributes`** (scaffolded): `schema/** linguist-generated=true text eol=lf`. The `eol=lf` is load-bearing — the gate compares bytes.
+
+| Not the same thing | Answers | Vocabulary |
+|---|---|---|
+| `<id>.snapshot.json` | what `x db gen` diffs the entities against | the entity's — generator spellings |
+| `packages/db/schema/` | what the database holds | SQL — Postgres' spellings |
+| `x_migrations` | what is deployed | ledger rows, checksummed |
+
+When it is owed: from the first migration. An app with no migration and no dump owes none — `x new --no-example` stays green.
+
+| `drift` finding | Cause | Fix |
+|---|---|---|
+| `…/schema does not exist` | migrations, and the dump was never generated | `x db gen` |
+| `… differs from what the migrations produce` | a hand edit, a migration written or pulled with no regeneration, or a framework upgrade that changed an `x_` table | `x db gen` |
+| `… is what the migrations produce and is not committed` / `describes nothing the migrations produce` | a file missing, or a stray one | `x db gen` |
+| `the migrations do not replay on the scratch database` | a statement the scratch database refuses | `x db migrate --json` names it, then `x db gen` |
+| `the migrations create extension "…", which the embedded database cannot link … neither TEST_DATABASE_URL nor DATABASE_URL names one` | the engine rule chose a server and none is named | `TEST_DATABASE_URL=postgres://<user>:<password>@localhost:5432/postgres x db gen` |
+| `migration <id> WAS written … only the schema dump failed` | `x db gen` exited 1 **after** writing the migration | repair what the rest of the cause names, then a bare `x db gen` — it writes the dump and no second migration |
+| `a database loaded from the schema dump is not the one the migrations build` | the dump renders an object wrong | `x db gen`, then report the file |
+
+Limits, stated:
+
+- **Rendered:** the ten kinds above — row security and its policies since `As of 2026-11-06`. **Named in `unrendered.sql`, not rendered:** inheritance children, foreign tables, composite and range types, aggregates, a policy on a relation the dump does not create, rules, and — `As of 2026-10-02` — extended statistics (`create statistics`), a column whose `storage` departs from its type's, a materialized view created `with no data`, and a trigger on a relation the dump does not create (an inheritance child: it used to be filed under `09_triggers/` and the load refused the whole dump). Partitioned tables and their partitions are rendered, `As of 2026-11-06` ([Partitioned tables](#partitioned-tables)). A generated column is spelled `stored` or `virtual` as the catalog says. A dump that names one makes no load-equals-replay claim, and the step does not ask.
+- **Neither rendered nor named:** comments, grants, owners, publications, statistics objects, event triggers, collations, operators, casts.
+- **Bytes are promised per engine major.** `pg_get_*def` spellings move between Postgres majors. On the embedded engine the major is the installed PGlite's (0.5.8 is PostgreSQL 18), pinned by the lockfile. An app on the server engine pins its own: the same major in CI as on the laptops that run `x db gen`, or the `drift` step reports the difference.
+- **`x db reset` still replays.** Loading the dump builds the schema faster in isolation (reference app: 159 ms → 74 ms) and is under 3% of a reset, which is one embedded-database boot.
+
+### Extensions and the scratch database
+
+| The migrations create | Scratch database | Needs |
+|---|---|---|
+| no extension | embedded | nothing |
+| an extension the installed PGlite ships — `citext`, `pgcrypto`, `pg_trgm`, `uuid-ossp`, `hstore`, `ltree`, `unaccent`, `btree_gin`, 34 in 0.5.8 | embedded, with it linked | nothing |
+| one it does not — `vector`, `postgis`, `pgstattuple` (0.5.8 ships no `vector`) | a real Postgres: a database created, replayed in, and dropped | `TEST_DATABASE_URL`, else `DATABASE_URL` — the admin URL `@ultimat3/testing` clones its template databases from; the role must be able to create a database |
+
+`x dev`'s embedded database links the same list from the same reader, so a migration that replays in the gate applies in `x dev`. An app in the third row cannot run on the embedded database at all and develops against `DATABASE_URL`.
+
+### The boot is cached
+
+An embedded scratch boot restores a post-`initdb` snapshot instead of running `initdb`: `.x/cache/pglite-<version>-f1.snapshot`, keyed on the PGlite version alone (one snapshot serves every extension set), about 40 MB, gitignored with the rest of `.x/`. Read only through its own checksum; a file that does not verify, or verifies and will not open, is deleted and rebuilt. Written under a temp name and renamed, so two commands racing leave one whole file. A fresh checkout — CI — pays one `initdb` and writes it. Delete it freely: `rm -r .x/cache`.
+
+`x db migrate` also reports `unexpected-object` (`X_DB_DRIFT`): a trigger, function, view, type or sequence the dev database holds and replaying the migrations does not create. Compared by identity, never by definition text. The fix is one command that prints its definition (`psql "$DATABASE_URL" -c '…'`); its comment gives the repair in order — copy the definition into a migration, drop the hand-made copy, then `x db migrate`.
+
+## Foreign keys and CHECKs on a live table
+
+`As of 2026-11-02`. A key or a CHECK added as one statement scans the whole table holding a lock that stops writes. `x db gen` writes it in two halves instead, so writes keep flowing.
+
+| What | Where it lands |
+|---|---|
+| A foreign key on a live table | the `_concurrently` step, each statement in its own short transaction under `lock_timeout`: index builds, `alter table … drop constraint if exists k, add constraint k … not valid`, `validate constraint`, then the drop of any old key. An old key under another name holds until that last statement |
+| A key on a table this migration creates | the plain `add constraint` in the transaction, byte-identical to before — unless the unique index it needs is built by the deferred step, which would otherwise be `42830`. Then it defers too |
+| A key re-added after a retype | the transaction: the rewrite holds ACCESS EXCLUSIVE already |
+| A CHECK on a live table | `add constraint … not valid` in the transaction (it needs no index), `validate constraint` in the deferred step. New writes are held to it at once. A new table's stays in `create table` |
+| `down` | restore the old key, drop the new key, drop the index |
+
+`x db migrate` re-runs a failed step from the top. `validate` finding a bad row is `X_DB_FOREIGN_KEY_VIOLATION` (or the CHECK's own failure) with the constraint in place holding new writes and the old release still serving: repair the row, re-run. A declared constraint left `NOT VALID` in a database whose ledger says the step ran is `X_DB_DRIFT`, kind `not-valid-constraint`, and its fix is the validate statement.
+
+## A column drop takes two releases
+
+`As of 2026-11-02`. Old and new pods serve one schema during a roll, and the old release names every column it declares in every read and insert, so a drop in the release that stops declaring it answers `X_DB_SCHEMA_STALE` from every old pod.
+
+| Release | The entity | `x db gen` | Writes |
+|---|---|---|---|
+| N | remove it from `columns`, add `retired: ['legacy']` (physical name) | `x db gen "retire legacy"` | `-- retires "t"."legacy"` and no statement. The column stays in the table and the snapshot, marked `retired`, with its indexes, keys and own CHECK |
+| N+1 | remove it from `retired` | `x db gen "drop legacy" --allow-destructive` | `-- contracts "t"."legacy"`, the `drop column`, `-- destructive: true` |
+
+| Refusal | When |
+|---|---|
+| `X_MIGRATION_ROLLING_UNSAFE` | a column leaves the entity without having been retired, with or without `--allow-destructive`. Also a retired name no migration records or `columns` still declares, a NOT NULL column with no default or a key column (the new release's inserts would fail, so relax it first), a committed `drop`/`rename` when the `drift` rail is on, and a retire and its drop both pending on a live production database |
+| `X_MIGRATION_IRREVERSIBLE` | the retired column is removed without `--allow-destructive`. Retiring is not consent to losing the rows |
+
+An app that stops before it starts (one instance, `strategy: Recreate`) says so: `x db gen "drop legacy" --allow-destructive --rolling-unsafe` writes `-- rolling-unsafe: true` and needs no retire.
+
+**The rail.** `rollingSince: "<migration id>"` in `x.verify.json` turns on the `drift` step's fifth rail for every migration after that id: a `drop column` with no `-- retires` in an earlier file, a `rename column`, a `rename table`. A closed list, on statements and not on prose. `x new` writes `"0"`. Unset, the rail does not run, because an applied migration cannot gain a marker (checksum); `x doctor` prints one line naming the id to set. A retire before the bound still earns the drop after it.
+
+**A new enum member is not railed.** N-1 pods refuse a value they do not know (`X_INVARIANT_VIOLATED`), but adding a member nobody writes yet is safe, and no file says when the first write ships. Add the member in N, write it in N+1.
+
+## A lock wait is a turn lost, not a failed deploy
+
+`As of 2026-11-02`. `lock_timeout` firing during `migrate()` or `rollback()` sends it again, up to `lockRetries` more times (default 5), waiting at least one `lockTimeoutMs` plus jitter so the queue behind the ALTER drains, with a warn line per attempt. A transactional step retries whole, since it rolled back whole; a `-- transaction: false` step retries the one statement, since what ran before it stays run. The ledger row is written only by a success. Exhausted is `X_DB_LOCK_TIMEOUT`, its cause stating the attempts.
+
+## The destructive-migration rail
+
+A generated migration that drops a column or table, retypes a column, or truncates a table needs an explicit marker in the file, or `x verify`'s `drift` step refuses to ship it:
+
+```sql
+-- 20260814120000_drop_legacy
+-- destructive: true
+
+alter table "posts" drop column "legacy";
+
+-- down
+alter table "posts" add column "legacy" text; -- data is not restored
+```
+
+`destructive.ts` owns the classification — a closed list of four kinds (`drop-table`, `drop-column`, `retype-column`, `truncate`), decided against noise-stripped SQL so a comment or a string literal mentioning "drop table" is never mistaken for the operation. `truncate` counts only as the statement's first keyword, so an insert-only hardening migration (`create trigger … before truncate …`, `revoke truncate …`) needs no marker.
+
+The criterion is **persisted row data**, not table rewrites: an operation is destructive when applying it removes rows or the values in them. That is why `alter table … drop` names a column and is destructive, while the sub-clauses that name what they drop are excluded — `drop constraint` removes constraint metadata, `drop default`, `drop not null`, `drop identity`, `drop expression` and `drop generated` change column metadata, and a standalone `drop index` removes an auxiliary structure. Every row survives all of them. `retype-column` is on the list for the opposite reason: rewriting a column's type can lose the values it held, whether or not the type is narrower.
+
+Only `up` is ever judged — reversing a `create table` is a `drop table`, so a rail reading `down` would mark every migration ever generated, and a marker on all of them marks none.
+
+Two separate questions, two codes, asked at two different times:
+
+| Code | Asked | Means |
+|---|---|---|
+| `X_MIGRATION_IRREVERSIBLE` | at **generation** | can this migration's `down` restore the rows? A generated drop refuses here first — `x db gen "<name>" --allow-destructive` to proceed anyway |
+| `X_MIGRATION_DESTRUCTIVE` | at the **gate** (`x verify`'s `drift` step) | does the committed `up` destroy data with no `-- destructive: true` line saying so? `x db gen` writes the marker itself when asked to generate a destructive migration |
+
+`X_MIGRATION_CONFLICT` is the ledger's own refusal (checksum or app-version mismatch, above) — a third, unrelated code, not a spelling of either of these two.
+
+The marker is SQL the migration's checksum covers, so it must be added **before** the migration is applied: marking an already-applied file is an edit, and reaches `X_MIGRATION_CONFLICT` on the next apply, correctly. Full wording for all four codes: [Error codes](Error-Codes).
+
+## Backfills: `backfill()` is a factory over `job()`, not a ninth primitive
+
+Same rule `llm()` follows for a model call ([`docs/idea/02-primitives.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/idea/02-primitives.md)): a one-pass sweep over a table is durable background work with an input schema, a retry policy, an idempotency key and a queue — the definition of a `job` — so `backfill()` *returns* one instead of the framework growing a ninth kind of thing. That single line is what gives a backfill `.enqueue()`, the worker's cancellation, the dead-letter path, `x jobs show` and a manifest row for free, with nothing declared twice. See [Jobs and workflows](Jobs-And-Workflows) for the `job` primitive itself.
+
+```ts
+export const normalizePostTitles = backfill({
+  name: 'normalize-post-titles',
+  tenant: 'none',   // a sweep spans every tenant; the PASS opens the cross-tenant scope, not you
+  source: ({ ctx }) => db.posts.where({ orgId: ctx.actor.orgId }),
+  handle: async ({ rows, signal }) => {
+    signal.throwIfAborted();
+    await db.posts.upsertAll(
+      rows.map((row) => ({ ...row, title: row.title.trim() })),
+      { onConflict: ['id'] },
+    );
+  },
+  batch: 1_000,   // rows per statement and per durable step — default DEFAULT_BACKFILL_BATCH
+  rate: 5,        // batches per second — default DEFAULT_BACKFILL_RATE, there is no unthrottled mode
+});
+```
+
+### Signature
+
+```ts
+backfill<Row>({
+  name: string,                                        // REQUIRED — a durable key, unlike a job's
+  source(args: { ctx: Ctx }): ReadBuilder<Row>,         // the chain to sweep; read once per attempt
+  handle(batch: BackfillBatch<Row>): Promise<void> | void,
+  batch?: number,                                       // default DEFAULT_BACKFILL_BATCH (1000)
+  rate?: number,                                         // default DEFAULT_BACKFILL_RATE, batches/sec
+  queue?: string,
+  retry?: RetryPolicy,                                   // default DEFAULT_RETRY
+  timeout?: DurationInput,                                // per attempt, not per pass
+  requires?: string,                                      // a migration id, checked before a run
+  environments?: readonly Environment[],                  // omitted = every environment
+  count?(args: { ctx: Ctx }): Promise<number> | number,    // the same predicate, counted
+}): JobHandle<{ force?: boolean }>
+```
+
+`source` returns a `ReadBuilder` — the same chain type [Batching and preloading](Batching-And-Preloading) documents `preload()`/`inBatches()` on — so tenancy, soft delete, the projection and every `.preload()` on it mean exactly what they mean anywhere else. It is read fresh each attempt and never enqueued, so what a run visits cannot drift from what was declared. `handle` is deliberately handed no `step`: a step name minted inside a per-batch body would collide with itself on the second batch. `BackfillBatch<Row>` carries `rows`, `ctx`, `signal` (the run's cancellation composed with this batch's own ceiling) and `index` (0-based position, also the step name — `batch:<index>`).
+
+### A remote source — `pages`, and a pass per scope
+
+`pages` beside `source`, exactly one of the two. One page is one durable step that checkpoints `{ cursor, rows: n }` and never the rows, so a pass killed mid-page asks for **that** page again and no earlier one. `next: null` ends the pass; an empty page that does not move the cursor ends it too. `rate` paces the vendor. `count` is optional — a remote may not know.
+
+```ts
+import { backfill, t } from '@ultimat3/jobs';
+
+interface Invoice { readonly vendorId: string }
+declare const vendor: {
+  invoices(
+    account: string,
+    cursor: string | null,
+    signal: AbortSignal,
+  ): Promise<{ readonly items: readonly Invoice[]; readonly nextCursor?: string }>;
+};
+declare const invoices: { upsertAll(rows: readonly Invoice[]): Promise<void> };
+
+export const syncInvoices = backfill<Invoice, { readonly accountId: string }>({
+  name: 'sync-invoices',
+  tenant: ({ input }) => input.accountId,
+  input: t.object({ accountId: t.string }),
+  scope: ({ accountId }) => accountId,               // one pass and one ledger per account
+  pages: async ({ cursor, signal, input }) => {
+    const page = await vendor.invoices(input.accountId, cursor, signal);
+    return { rows: page.items, next: page.nextCursor ?? null };
+  },
+  handle: ({ rows }) => invoices.upsertAll(rows),
+});
+await syncInvoices.enqueue({ input: { accountId: 'acct-1' } });
+```
+
+| With `scope` | |
+|---|---|
+| ledger row and live idempotency key | `<name>:<scope(input)>` — two scopes, two ledgers, never deduped onto each other |
+| `x db backfill --pending` | one row per scope the ledger holds; a scope never run is not listed — which accounts should sync is the app's |
+| an unscoped declaration | byte-identical to before: key and ledger name are `<name>` |
+
+Refused at `backfill()`, before the name is taken: both or neither of `source` and `pages`, or `input` with no `scope` — `X_BACKFILL_SOURCE_INVALID`. The checksum covers `pages` exactly as it covers `source`.
+
+### Idempotency — `handle` is at least once
+
+The batch runs inside its own durable step, and the checkpoint is written **after** it returns — never the other way round. An attempt cancelled between the last row and the checkpoint replays that whole page on the next attempt. `handle` must therefore be idempotent: `upsertAll`, `updateWhere`, or any statement whose second run changes nothing — never `count + 1`. What a step persists across a replay is the **cursor and a row count**, never the page itself; `steps.ts` retains a completed step's output for the whole run, and a page held there for every batch of a million-row sweep is every processed row kept in memory until the job ends.
+
+Re-enqueueing a backfill that has already run to completion runs a **real job** whose pass is the no-op. The two are different facts and only the second one is dedupe:
+
+```ts
+const first = await normalizePostTitles.enqueue({});
+first.deduped;   // false — this is the run
+
+// While the first is still LIVE: one live run per name, forced or not.
+const live = await normalizePostTitles.enqueue({});
+live.deduped;    // true — the partial index covers ready | delayed | running | suspended
+
+// Once the first has COMPLETED: a completed job is in none of those states, so this is not a
+// dedupe. A real job row is created and a worker runs it — the PASS then reads x_backfills and
+// returns { skipped: true, previousRunId }, having touched no row.
+const after = await normalizePostTitles.enqueue({});
+after.deduped;   // false
+```
+
+The dedupe is the live-run index; the ledger is what makes the second pass do nothing.
+
+`--force` (or `{ force: true }`) is the only override, and it rides the **input**, not the idempotency key — `idempotencyKey` is always `() => definition.name`, so "kick it again" stays one live pass rather than becoming a second writer racing the first. A forced rerun writes a **new** ledger row: history is never overwritten, so what each pass swept stays readable. A completed row is the only one that blocks a rerun — a `running` row is this pass resuming (or the one live idempotency key already holding it), and a `failed` row is an attempt the queue is about to retry anyway.
+
+### The `x_backfills` ledger — `x_migrations`' twin, one level up
+
+Keyed by **run**, not by name, because a backfill may legitimately run again where a migration never does. Mirrors the shape of the migration ledger:
+
+| Column | Meaning |
+|---|---|
+| `runId` | the job run this pass belongs to — also the ledger's primary key |
+| `name`, `checksum` | the declared name, and a hash of `source`'s and `handle`'s source text (`NUL`-separated, never `batch`/`rate` — pacing is a tuning change, not a different sweep) |
+| `status` | `running` \| `completed` \| `failed` — only `completed` blocks a rerun |
+| `appVersion` | the build that **started** the pass; a redeploy mid-pass does not rewrite it |
+| `rows`, `cursor` | absolute progress — a replayed batch reports the same number, never a delta; `cursor` is `null` before the first batch and once the pass is over |
+| `startedAt`, `completedAt` | epochs; `completedAt` is stamped for `failed` too, and cleared when a retried attempt puts the row back to `running` |
+
+A moved checksum **warns** and does not block the run — a bundler that reformats a function body moves its source text with no line of behaviour changed, so this hash is fuzzier than a migration's SQL checksum on purpose, where `@ultimat3/db`'s `auditLedger` throws on the same kind of drift because SQL text is what was actually applied.
+
+The ledger is a **report an operator reads**, never a resume source — the step checkpoints inside `backfill-pass.ts` are transactional with the work and decide where a resumed pass restarts; a resume driven off the ledger's own `cursor` would be a second, non-transactional answer to "where were we."
+
+### Throttling
+
+`rate` (batches per second, default `DEFAULT_BACKFILL_RATE`) is spent as the **first statement inside the batch's own step**, never wrapped around it — an attempt resuming at batch 500 replays 500 already-completed checkpoints that run no body and touch no database, and paying the full throttle for all of them again before reading a new row would make every resume slower than the original pass. There is no way to turn the throttle off: a backfill that saturates the pool has no correct value here, since it shares that pool with the requests the app is still serving live.
+
+### Progress observability
+
+One projection, four surfaces, so none of them can disagree about how far a pass has got:
+
+| Surface | Shows |
+|---|---|
+| `x db backfill --list` | the whole ledger, newest first, filterable by name/status |
+| `x jobs list` | ordinary job rows, plus the `backfill()` passes whose ledger row is **`running`** — `jobs-report.ts` filters `{ status: 'running' }`, so a completed or failed pass is not on this surface |
+| `x jobs show <id>` | one job's full state — step trace, next retry — plus this run's ledger row under `backfill` when the job is one |
+| `/_x` jobs panel | the same ledger, live |
+
+All four read `inspectBackfills(driver, filter?)` from `backfill-inspect.ts` — there is no second reader. It answers `[]`, never a throw, for a driver whose `JobDriver.backfills` is absent (a driver that ships no ledger, e.g. a hand-rolled one): the surfaces asking about the *queue* must not fail over a fact nobody asked for, though `x db backfill --list` — the surface that **is** the question — says so in its own summary.
+
+### Declared, and never run — the other half of the ledger
+
+`x_backfills` answers "which passes have run". Nothing answered "which passes exist", so a cleanup could be authored with `x g backfill <name> --feature <slice>`, merged and deployed, and **never run**, with no surface anywhere saying so. `backfill()` now stamps its own handle — through the same `origin` `WeakMap` `task()` uses, with no `registerBackfill()` call for an app to forget — and the diff between the two halves is a command:
+
+```
+x db backfill --pending --json
+```
+
+Non-zero exit when anything is unswept, so a cron or a deploy check reads the exit code and never the table.
+
+| State | Means | Counts as pending |
+|---|---|---|
+| `pending` | declared, and `x_backfills` holds no row under this name | yes |
+| `failed` | the newest pass failed — the queue may already have dead-lettered it | yes |
+| `running` | a pass is in flight | no — an alarm red for the whole of every sweep gets muted within a week |
+| `completed` | a completed row exists, so nothing re-runs without `--force` | no |
+| `excluded` | `environments` does not include this one | no |
+
+`orphaned` is the mirror image: a ledger name no declaration carries, i.e. a sweep whose module was deleted after it ran.
+
+### Three optional declarations, all of them data
+
+| Field | Rail | Enforced where |
+|---|---|---|
+| `requires: '<migration id>'` | `X_BACKFILL_MIGRATION_PENDING` | `x db backfill`, which is where `x_migrations` is readable — `@ultimat3/jobs` holds no `@ultimat3/db` dependency and does not grow one to read a ledger |
+| `environments: [...]` | `X_BACKFILL_ENVIRONMENT` | **the pass itself**, ahead of the ledger open, so a `.enqueue()` from app code is covered too — and again in `x db backfill` as a pre-check, which refuses before queueing work that would only dead-letter |
+| `count()` | `X_BACKFILL_STALLED` | the pass, once, after the last batch |
+
+`environments` ships as declared **data** and never as a hardcoded "cleanups are production": a staging rehearsal is correct practice, so which deploys a sweep belongs to is the app's own convention and this field is only the mechanism carrying it. There is deliberately **no** `dependsOn` graph over other backfills — the real dependency is almost always "after code that tolerates both shapes is serving", which the framework cannot observe and would therefore encode wrongly.
+
+`count()` is the same predicate `source` selects on, counted rather than read. It is what makes a dry run honest and turns "did it converge" into arithmetic: a pass that exhausts its source while `count()` still answers above zero has two predicates that disagree, so it **fails** rather than writing a completed row that stops the next deploy re-running it. Left out, the pass converges by definition — the framework will not guess a number on the author's behalf.
+
+The **result** is parsed, not trusted: a non-negative safe integer, or `X_INVARIANT` where the count was returned. `NaN > 0` and `-1 > 0` are both false, so an unchecked bad number reads as "nothing left" and writes exactly the completed ledger row this detector exists to prevent.
+
+### Running one — `x db backfill`
+
+```
+x db backfill --list                        # the x_backfills ledger, newest first
+x db backfill --pending                     # declared minus completed; non-zero when unswept
+x db backfill normalize-post-titles         # DRY RUN — --write is never implied
+x db backfill normalize-post-titles --write # gate, then enqueue; the workers sweep
+x db backfill --all --write                 # every pending one, isolated per name
+x db backfill normalize-post-titles --write --force   # a completed name, into a NEW ledger row
+```
+
+A bare `x db backfill` is refused rather than defaulted: the four shapes answer four different questions.
+
+`--write` **enqueues** — it never runs the pass inside the CLI process, because the queue is a job's execution surface. `--all` isolates per name and continues past a failure, exiting non-zero and naming each, so one wedged cleanup cannot block every later one forever.
+
+`requires` is checked against `x_migrations` here, and the three answers are distinct on purpose: the ids that are applied, `[]` when the table does not exist (nothing has ever been applied, so every `requires` is unsatisfied and the sweep is refused), and a **propagated error** when the read itself failed. A permission error or a timeout read as "nothing to check" would let a sweep run against exactly the shape it was declared to wait for.
+
+### The `backfill` deploy step triggers; it never gates
+
+`backfill` is a **deploy step**, not a `ROLE` — there is no `ROLE=backfill` (`packages/core/src/roles.ts` names six roles). `DEPLOY_ROLES` is `migrate → web → sync → worker → scheduler → backfill`, and the position is the design:
+
+| Role | Shape | Why there |
+|---|---|---|
+| `migrate` | `run --rm`, **first** | a schema change must land before anything serves it, and drift after it fails the deploy |
+| `backfill` | `run --rm`, **last** | a slow `UPDATE` inside a release gate holds the deploy open against a database still serving the *previous* release — so the sweeps are enqueued after the new pods serve, and the workers already draining the queue perform them |
+
+Backfills are therefore **never** wired into `runMigrations()`. The `backfill` container runs `x db backfill --all --write --json` and exits.
+
+Position in the plan is **necessary and not sufficient**: `docker compose up -d` returns when a container has started, not when the app inside it serves. The barrier that makes "after" true is declarative — the `backfill` service carries `depends_on: { web: { condition: service_healthy } }`, which `docker compose run` honours. `As of 2026-08` that service ships with the barrier in both committed compose files: [`docker/docker-compose.prod.yml`](https://github.com/developerz-ai/ultimate/blob/main/docker/docker-compose.prod.yml) and the one `x new` scaffolds. In the scaffolded app the condition is satisfied by the image's own `HEALTHCHECK` on `/healthz`, so an app that removes it also removes the barrier.
+
+### `x db recount` — a counter, repaired
+
+`x db recount <entity>.<column>` compares a `countedIn` counter with the count of child rows and lists the drifted rows; `--write` sets them in one cross-tenant transaction. A counter drifts only while its trigger was missing (`X_LIFECYCLE_TRIGGER_MISSING`). As of 2026-11 the repair is one statement, not a paced `backfill()` pass.
+
+### `x g backfill` — the generator
+
+Scaffolds a working pair, never a stub: `x g backfill <name> --feature <slice>` writes `<slice>/backfills/<name>.ts` — a `backfill()` declaration reading the feature's own table through `tableFor(entity, postgresRepo(entity))`, sweeping with `.where({ orgId })` guarded by an `assert` that the actor carries one, and writing back through `upsertAll` — plus `<name>.test.ts` asserting the declaration (name, retry, idempotency key), the manifest projection, and that the row transform is genuinely idempotent — `expect(<name>Row(once)).toEqual(once)`, deep value equality, since a transform that returns a fresh object every pass is still idempotent and `===` would fail it — and actually enqueues/dedupes against a memory job driver. A generated no-op handler that checkpoints a page it never wrote would report rows swept that nobody touched, which is why the template ships a real (if trivial) row transform rather than a `throw new Error(…)`.
+
+## Errors
+
+| Code | Cause | Fix |
+|---|---|---|
+| `X_MIGRATION_CONFLICT` | the ledger disagrees with this build — a foreign app-version row that is not a rollback (rows all newer than the build are accepted), or an applied migration whose checksum moved | deploy the version `cause` names, or `x db gen "fix <migration>"` — never edit an applied migration |
+| `X_MIGRATION_IRREVERSIBLE` | a generated plan drops a column or table and its `down` cannot restore the rows | `x db gen "<name>" --allow-destructive`, or put the column back in `columns` and keep it |
+| `X_MIGRATION_DESTRUCTIVE` | a committed migration's `up` destroys data with no `-- destructive: true` line | add the marker, or regenerate with `--allow-destructive` — before the migration is applied |
+| `X_MIGRATION_SNAPSHOT_MISSING` | the newest migration on disk carries no `.snapshot.json` sidecar to diff against | `git checkout -- packages/db/migrations/<id>.snapshot.json` — or, when it was never written, `rm packages/db/migrations/<id>.* && x db gen "<name>"`, the delete first |
+| `X_MIGRATION_CONCURRENT_IN_TRANSACTION` | a migration without `-- transaction: false` holds a concurrent index statement, which Postgres refuses inside a transaction block (`25001`) | `sed -i '1i -- transaction: false' packages/db/migrations/<id>.sql` before it is applied, or move the statement into a migration of its own. Refused before `BEGIN`, so nothing is applied |
+| `X_MIGRATION_VIEW_DEPENDS` | a view is compiled against a column this migration retypes — Postgres answers `0A000` and rolls the whole migration back | `drop view "<v>";` then `x db migrate`, then re-create it from the `create view` the `fix` line carries verbatim. Caught by a **preflight** inside the migration's own transaction, before its first statement, so nothing partial is applied |
+| `X_DB_DRIFT` | the live schema, or the source, disagrees with the migrations | `x db gen "<name>"` — see [Entities and migrations → Drift is a `x verify` failure](Entities-And-Migrations#drift-is-a-x-verify-failure) |
+| `X_SCHEMA_DUMP_DRIFT` | `packages/db/schema/` is not what the migrations produce, or was never generated | `x db gen` — see [The schema dump](#the-schema-dump) |
+
+Seven codes, and each one exists because it sends the reader somewhere different — run it, force it, change environment, migrate first, wait, fix the predicates, fix the name. All are `@ultimat3/jobs`': the CLI throws the framework package's errors rather than minting a parallel set, and the pass enforces two of them itself.
+
+| Code | Cause | Fix |
+|---|---|---|
+| `X_BACKFILL_PENDING` | declared, and `x_backfills` holds no completed pass for it in this environment | `x db backfill <name> --write --json` |
+| `X_BACKFILL_APPLIED` | the ledger already records a completed pass and `--force` was not given | `x db backfill <name> --write --force --json` — a rerun is a NEW row, never an edit |
+| `X_BACKFILL_ENVIRONMENT` | `environments` does not include the one this process resolved | run it where `ULTIMATE_ENV` matches, or add this environment to the declaration |
+| `X_BACKFILL_MIGRATION_PENDING` | `requires` names a migration `x_migrations` does not record as applied | `x db migrate --json` |
+| `X_BACKFILL_RUNNING` | the enqueue deduped — one live pass per name, and this name already has one | `x jobs show <id> --json`; a pass that is not advancing is a worker that lost its lease |
+| `X_BACKFILL_STALLED` | the source is exhausted and `count()` still matches rows — the two predicates disagree | make `count()` select on exactly what `source()` selects on |
+| `X_BACKFILL_UNKNOWN` | no declaration in this app carries that name | `x db backfill --pending --json` lists every declared name |
+
+`X_BACKFILL_WRITE_UNCONFIRMED` was considered and **rejected**: a dry run that wrote nothing did exactly what it was asked to, and a code whose fix line duplicates another one's is code inflation.
+
+A bad `batch`/`rate` throws `@ultimat3/core`'s `assert`, which is `X_INVARIANT` — borrowed, not declared, and it carries its own cause and fix at the declaration site:
+
+| Code | Cause | Fix |
+|---|---|---|
+| `X_INVARIANT` (bad `batch`) | `backfill "<name>" declares batch: <n> — a batch is a whole number of rows, at least one` | `set batch: 1000 on backfill("<name>") — the rows one statement reads and one durable step handles` |
+| `X_INVARIANT` (bad `rate`) | `backfill "<name>" declares rate: <n> — a rate is batches per second, greater than zero` | `set rate: 0.5 on backfill("<name>"), or leave it out — to sweep faster raise the number, there is no unthrottled mode` |
+
+Both throw at import, not at the first batch, so a bad number is a failed build rather than a dead-lettered job. Everything past that runs the same retry/dead-letter path as any other `job` ([Jobs and workflows → Errors](Jobs-And-Workflows)).
+
+Full list with `--json` shapes: [Error codes](Error-Codes). Migration workflow, drift and the reversible/destructive distinction in full: [Entities and migrations](Entities-And-Migrations). The `job` primitive, steps, retries and the outbox: [Jobs and workflows](Jobs-And-Workflows).