@ultimat3/db 11.3.0 → 13.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CLAUDE.md CHANGED
@@ -369,6 +369,56 @@ anonymous SQL, never as a `job` statement, and will keep seeing it that way unti
369
369
  own pair through `driver-pg.ts` the way `postgresRepo` now threads entity's — that is still future
370
370
  work, not something this change reaches.
371
371
 
372
+ **An index's ACCESS METHOD is carried end to end, `As of 2026-08-24`, and it had to land here
373
+ before `@ultimat3/entity` could declare it.** `@>` / `<@` / `&&` / `?` on a `json()` or `arrayOf()`
374
+ column is a sequential scan without a GIN index. `IndexInit.using` on the entity side while this
375
+ package ignored it would emit a **btree for a declared GIN index** — a declared-and-never-wired key,
376
+ which is the defect class this release exists to eliminate and strictly worse than the missing
377
+ capability. So the method reaches all four places or none: `createIndex` emits it, `snapshotOf`
378
+ records it, `indexShape` rebuilds on it, and `compareIndexes` reports it.
379
+
380
+ `index-method.ts` is the vocabulary, its own file for the reason `foreign-key.ts` holds
381
+ `onDeleteRule`: a generator and a detector that disagreed about what "the default" is would report
382
+ drift on a database that is exactly right. Four rules.
383
+
384
+ **The set is closed at two — `btree` and `gin`.** `gist`, `brin`, `hash` and `spgist` are legitimate
385
+ and are deliberately absent: nothing declares one, and each brings a rule that would have to be
386
+ enforced with no caller to test it (`hash` and `brin` cannot be unique, `gist` needs `btree_gist` to
387
+ be, none of the three accepts `asc`/`desc`). Adding a member later is additive; shipping four nobody
388
+ uses is four ways for a first caller to be silently wrong.
389
+
390
+ **Declared is CLOSED, live is OPEN.** `IndexDescriptionLike.using` is `IndexMethod | undefined` —
391
+ what an entity may ask for. `IndexDescription.using` is `string | undefined` — whatever `pg_am`
392
+ answered, `gist` and an extension's own access method included. Folding an unknown catalog name into
393
+ `btree` would hide exactly the difference drift exists to report, so `indexMethodOf` passes the live
394
+ side through verbatim and `declaredMethod` is the one place the open reading is narrowed back — a
395
+ **refusal**, never a silent fall back, because its one caller is `redefineIndex`'s `down` and a
396
+ `gist` quietly rebuilt as a btree is a rollback leaving a state no migration describes.
397
+
398
+ **Absent is `btree`, on both sides, through one function.** `indexMethodOf` is that function.
399
+ Postgres' default is written out by nobody, every index created before this existed is one, and
400
+ every sidecar written before the field is silent about it — so `snapshotOf` records `using` only
401
+ when one was declared. Writing `'btree'` out for every index would rewrite every sidecar in every
402
+ app on the next `x db gen`, a diff on every file for a fact that was already true.
403
+
404
+ **The literal is re-derived from the set, never spliced from the input** — `indexMethodSql` is a
405
+ `switch` whose `default` arm is `never` and throws `indexMethodInvalid` (`X_SQL_UNSAFE`, the code
406
+ `isolationLevelInvalid` and `branchNameInvalid` already use for a value spliced into a statement).
407
+ The type is not the guard: this value arrives from an entity declaration, a config or a hand-edited
408
+ snapshot, and `using ${method}` on an operand TypeScript never saw is the **identical hole** to the
409
+ one `columnName` carried when it was `meta.name ?? snake(property)` with only the second branch
410
+ validated — a name that closed the parenthesis and opened a second command, measured through
411
+ `generateMigration`. `create index "x" on "t" using gin ("c") where (...)` also refuses a unique or
412
+ an ordered GIN through core's `assert` (`X_INVARIANT`), the discipline `createIndex` already applies
413
+ to an index naming no columns: Postgres has neither, and a syntax error inside `ROLE=migrate` fails
414
+ the release phase with the server's words and none of the entity's.
415
+
416
+ `introspect()` reads the method from `pg_am` joined through `pg_class.relam`, and
417
+ `introspect-embedded.test.ts` is where that is pinned — a recording client can pin the SQL text and
418
+ nothing more, and a query that silently returned no method would read as `btree` everywhere and make
419
+ drift blind to the one case `using` exists for. Measured on PGlite: a real `using gin` index reads
420
+ back `gin`, the btree beside it and the primary key's own index read back `btree`.
421
+
372
422
  **`generate.ts` reads an index, it never re-derives one.** `EntityDescriptionLike.indexes` carries
373
423
  `columns`, `unique`, `where` and `order`, and `createIndex` writes every one of them out. It used to
374
424
  carry names alone and `parseIndexName` recovered the column list from the `<table>_<a>_<b>_idx`
@@ -502,6 +552,62 @@ override — `x_migrations.app_version` and `@ultimat3/jobs`' `x_backfills.app_v
502
552
  durable columns an operator reads side by side, and `jobs` cannot import this package for the
503
553
  answer, so the key has one reader at tier 0 rather than one per writer.
504
554
 
555
+ **Read replicas are opt-in twice, and the second opt-in is the correctness argument, `As of
556
+ 2026-08-24`.** A replica pool exists when `DATABASE_REPLICA_URL` names one (`default-client.ts`);
557
+ a read is *offered* to it only inside `withReplicaReads(fn)` (`replica-scope.ts`). With no scope
558
+ open nothing routes and the client is byte-identical to the single-pool one it has always been —
559
+ which is what makes "nobody adopted it yet" today's behaviour rather than a wrong answer.
560
+
561
+ The scope is what closes **read-your-writes**, and the reason it is a scope and not a request id is
562
+ worth writing down because the request id is the obvious answer and it does not work. `Ctx.requestId`
563
+ IS reachable from here — `@ultimat3/http`'s pipeline opens `runWithContext` around every request
564
+ (`packages/http/src/pipeline.ts`), `withChildContext` may not change the id, and `tryUseContext()` is
565
+ tier 0 — but nothing tells tier 1 when a request ENDED. A `Map<requestId, wrote>` therefore only
566
+ grows, ~100 bytes a request forever, and every eviction policy that forgets a request which WROTE
567
+ serves it a stale row on its next read. That is a data-correctness bug strictly worse than the
568
+ capacity problem replicas exist to solve, so the marker lives on a mutable value on an async context
569
+ (`ReplicaScope.wrote`, the same shape as `TxState.live`) whose lifetime somebody else already owns.
570
+
571
+ **`withTransaction` is on the primary structurally, not by rule.** `runRoot` pins a connection
572
+ through `reserve()`, and `replicatedClient` delegates `reserve()` to the primary and exposes it only
573
+ when the primary has one — so BEGIN, every statement and COMMIT are one connection on one server.
574
+ `isReservable` therefore has to keep answering about the DATABASE and not about the wrapper: a
575
+ wrapper that always exposed `reserve` makes `runRoot` pin a client that cannot pin, and one that
576
+ never exposed it makes `runRoot` run BEGIN, the body and COMMIT on three different pooled
577
+ connections. What `runRoot` adds is one line — `markScopeWrote()` unless `readOnly: true` — because
578
+ its statements go through a reservation and never through the router, so the scope could not
579
+ otherwise see that the request has written.
580
+
581
+ **`isPlainRead` is an allow-list, and that inversion is why it is not the lexer this file forbids.**
582
+ `readonly.ts` was deleted for defaulting to PERMISSION: a 22-word deny-list that read
583
+ `select pg_sleep(60)` as safe. This one defaults to the primary — a statement shape nobody
584
+ anticipated costs a replica opportunity and never an answer. **`statementKind()` is not the
585
+ authority and must not become it**: it calls `with … update … returning` a read, which is right for
586
+ an N+1 report and catastrophic for a routing decision, and `replica-route.test.ts` asserts the
587
+ disagreement so the two can never be collapsed. Three refusals earn their line — a locking read
588
+ (`for update`/`for share`; a standby cannot take the row lock), `select … into` (it creates a
589
+ table), and the functions a word boundary cannot reach (`pg_advisory_lock`, `set_config`,
590
+ `nextval`), which a standby ANSWERS rather than refusing, so the server cannot be the safety net for
591
+ those the way it is for a real write.
592
+
593
+ **A misroute fails loudly and repairs itself; a replica outage costs latency and never an answer.**
594
+ A statement a standby refuses (`25006`) never executed, and only `isPlainRead` statements are ever
595
+ sent there, so re-running one on the primary is exactly-once rather than at-least-once — which is
596
+ what makes the blanket fallback in `replica-client.ts` safe. The breaker is what stops that from
597
+ doubling every read during an outage: three consecutive failures park the replica for ten seconds,
598
+ counted on `Clock.monotonic()` so an NTP step cannot un-park it. `ReplicaStats` is exposed on the
599
+ client for a test that cannot scrape, the same reason `@ultimat3/realtime` exposes
600
+ `droppedChannelFrames`, and each fallback logs `db.replica_fallback` with `renderThrowable(error)`.
601
+
602
+ **The URL must name a read-only standby**, and nothing here can check it. The `25006` refusal is the
603
+ whole safety net under a text classifier that cannot be complete; pointed at a writable node, a
604
+ misroute becomes a write on the wrong server with nothing anywhere to report it.
605
+
606
+ **Nothing opens `withReplicaReads` per request yet.** The scope, the client and the wiring are tier
607
+ 1 and land here first; the adopter is one call in `@ultimat3/http`'s pipeline (or an app's own
608
+ handler), and until it exists no production traffic is routed. That is the tier rule working —
609
+ lowest tier first, consumers after — not an omission.
610
+
505
611
  `checkDrift()` is the **post-migrate verification** and the only drift question that needs a
506
612
  database: the live catalog against the ledger the run just wrote. It is asked where a connection is
507
613
  open — `@ultimat3/cli`'s `runMigrations`, which is `x db migrate`, `x db reset` and `ROLE=migrate`
@@ -547,8 +653,37 @@ right database, and `x db gen`'s `retypeColumn` owns that question where both si
547
653
  The `fix:` is the `alter table … set not null` itself and deliberately not `x db gen`, which has
548
654
  never emitted one and would answer with an empty migration.
549
655
 
656
+ **A column the DATABASE computes is a different thing at every step, and `generated-column.ts` is
657
+ all of them** — `As of 2026-08-24`. `ColumnDescriptionLike.generated` carries the
658
+ `generated always as (<expr>) stored` body across the tier seam (this package cannot import
659
+ `@ultimat3/entity`, so a field that is not on the projection reaches no DDL at all), and it reached
660
+ none until this date: `@ultimat3/entity`'s `.searchable()` emitted a `tsvector not null` column that
661
+ `columnClause` rendered plain, so nothing computed it and **the first insert was a `23502`**. Loud,
662
+ which was deliberate — but a feature nobody can insert into is not shipped. Four rules ride with it,
663
+ each one measured against a real server (`generate-generated-column.live.test.ts`):
664
+
665
+ | Rule | Why it is not the ordinary column's rule |
666
+ |---|---|
667
+ | the clause sits directly after the type | `"c" tsvector generated always as (…) stored not null check (…)` is what Postgres accepts; a column constraint may follow it |
668
+ | **generated and defaulted is refused** at `x db gen` | Postgres has no such column (`42601`) — a generated column's value IS its expression. `X_INVARIANT`, the same refusal `createIndex` gives a unique GIN, and for the same reason: the alternative is DDL whose first reader is `ROLE=migrate` |
669
+ | an expression that moved is **`set expression as (…)`**, never a drop and recreate | Postgres 17's statement, and it rewrites the table, recomputes every row and **keeps the column's indexes** — measured. Dropping the column takes its GIN index with it and nothing in the diff puts one back, and `alter table … drop column` is what `destructive.ts` reads as data loss: every expression change would then carry `-- destructive: true` on a migration that loses nothing, and a marker on all is none |
670
+ | a retype on it carries **no `using`** | Postgres refuses `using` on a generated column outright, which is exactly what `retypeColumn` emits for every other column — and there is nothing to convert, because the expression produces the new type itself |
671
+ | the NOT NULL add is **one statement**, never nullable-then-backfill | the database computes it for every existing row inside the same `add column`. The ordinary path's `-- backfill "c", then: … set not null;` names a step nobody can perform: writing to a generated column is `428C9` |
672
+
673
+ Two transitions have no `set expression`. **Generated → plain is `drop expression`**, which keeps
674
+ every value the column already computed. **Plain → generated is the whole column again** — drop,
675
+ add, and every index over it stated a second time, which is why `regenerate` answers `rebuilt` and
676
+ `diffTable` carries that set into its index loop: `redefineIndex` sees a definition that never moved
677
+ and would emit nothing, so the table would come back with no index at all.
678
+
679
+ **`introspect` deliberately does not read `generation_expression` back.** Postgres stores its own
680
+ rewriting (`COALESCE(title, ''::text)` for `coalesce("title", '')`), so a catalog value could never
681
+ compare equal to a generated one and drift would report a correct database forever. The diff that
682
+ DOES read it is `x db gen`'s, where both sides are this generator's own spellings — the rule
683
+ `IndexDescription.where` already states.
684
+
550
685
  `compareTable` judges **declared** indexes: one the migrations name and the catalog does not hold is
551
- `missing-index`, and one whose column list or uniqueness moved is `changed-index` — which is what
686
+ `missing-index`, and one whose access method, column list or uniqueness moved is `changed-index` — which is what
552
687
  catches a composite index rebuilt with its columns the other way round while the column diff said
553
688
  `ok: true`. A live index no snapshot names is deliberately **not** reported: Postgres creates one for
554
689
  every primary key and every unique constraint, so counting those is eight findings against a correct
@@ -644,7 +779,8 @@ rather than spliced DDL — the discipline `createIndex` already applies to an i
644
779
  ceiling and along the seam the tier already draws: they are the structural mirror of
645
780
  `@ultimat3/entity`'s description, which is how a snapshot crosses tier 2 → tier 1 with no import.
646
781
  `ColumnDescriptionLike.onDelete` is optional for exactly that reason — a description written before
647
- the field existed still satisfies the shape.
782
+ the field existed still satisfies the shape — and `ColumnDescriptionLike.generated` is optional for
783
+ the same one.
648
784
 
649
785
  **`snapshot-json.ts` writes the sidecar's bytes, and they must be a fixed point of Biome.** A
650
786
  scaffolded app's `lint` step is `biome check .` over `"includes": ["**"]`, and `.sql`/`.hash` are
@@ -681,7 +817,39 @@ exists` at boot, declared by no migration and carried in no snapshot, so counted
681
817
  are eight `unexpected-table` findings against a correct database. The prefix is the rule, not a
682
818
  list, so a table a future package adds needs no second declaration here. `introspect()` keeps its
683
819
  narrower default (`x_migrations` alone) because the admin schema view and the MCP `schema.describe`
684
- tool legitimately show `x_users` — only drift wants the whole namespace gone.
820
+ tool legitimately show `x_users` — only drift wants the whole namespace gone. That last sentence is
821
+ a *reservation*, not a description, `As of 2026-08-24`: nothing outside this package imports
822
+ `introspect()` today, and `schema.describe` (`@ultimat3/mcp`'s `dev-server.ts`) answers from the
823
+ entity registry.
824
+
825
+ **`app-relation.ts` is the other half, and it is ownership, never a name — issue #340,
826
+ `As of 2026-08-24`.** `pg_stat_statements` is a view an extension owns, the CNPG/RDS/Supabase
827
+ default puts it in `public` of every database, and the drift audit after `ROLE=migrate` reported it
828
+ as `unexpected-table` with `x db gen "add pg_stat_statements"` as the fix — so every deploy of the
829
+ demo app failed terminally for 16 hours, and following the fix would have written an extension's
830
+ internal view into the app's migration set. `nonAppRelations(client, schema)` names what
831
+ `introspect()` must not see, and `introspect()` merges it into `excluded` **unconditionally**: an
832
+ explicit `exclude` replaces the `x_migrations` default, never this set, because an extension's
833
+ relations are not app schema in any deployment and that is not a caller's to switch off.
834
+
835
+ Two disqualifications, one question. **Extension ownership is read out of `pg_depend`**
836
+ (`deptype = 'e'`, `refclassid = 'pg_extension'`) — Postgres' own record, and the only rule that
837
+ generalises: a `pg_*` prefix would have covered the reported view and missed `postgis`'
838
+ `spatial_ref_sys`, `timescaledb`'s catalog, and `pg_stat_statements`' own `pg_stat_statements_info`
839
+ sibling, which is a real `relkind = 'r'` table. **A view, a materialised view and a foreign table
840
+ are not tables**, whoever made them: measured on PGlite, a plain `create view` reaches
841
+ `information_schema.columns` while the index query already fences on `relkind = 'r'`, so one arrived
842
+ as a table with columns, no primary key and no indexes — a `TableDescription` that cannot be true,
843
+ and a finding no author could clear because no snapshot records a view. Excluding by NAME is safe
844
+ because `pg_class` names are unique within a namespace.
845
+
846
+ Nothing else in the audit had the same hole. An extension cannot own a **column** of a table it does
847
+ not own — `alter extension … add` has no `COLUMN` form — so `unexpected-column` is unreachable that
848
+ way. **Types and enums** are never compared (`compareTable` reads nullability and existence, never
849
+ the type). **Indexes and foreign keys** are judged on the declared side only, so an extension's
850
+ index on an app table was already silent. `introspect-embedded.test.ts` proves the predicate against
851
+ a real catalog by writing the exact `pg_depend` row `create extension` writes; a recording client
852
+ can only pin the SQL text, which is what `app-relation.test.ts` does.
685
853
 
686
854
  The `X_DB_DRIFT` rendering in `drift.ts` and the title in `DB_ERROR_TITLES` are pinned by the
687
855
  framework contract and duplicated in `@ultimat3/entity`. Change them together or not at all.
package/README.md CHANGED
@@ -13,7 +13,7 @@ boundary stays thin.
13
13
  ## Public API
14
14
 
15
15
  ```ts
16
- import { db, sql, raw, withTransaction, currentTx, readOnly, setDbClient } from '@ultimat3/db';
16
+ import { db, sql, raw, withTransaction, currentTx, setDbClient } from '@ultimat3/db';
17
17
 
18
18
  const rows = await db().query<Post>(sql`select * from posts where org_id = ${orgId}`);
19
19
 
@@ -32,7 +32,11 @@ await withTransaction(async (tx) => {
32
32
  | `sqlState()` / `sqlStateCode()` / `isRetryableState()` / `SQLSTATE` | `As of 2026-08`: the SQLSTATE a driver error carries, and the closed table from it to a code. `Bun.SQL` puts it on `errno`; PGlite puts it on `code`; **one** reader answers for both |
33
33
  | `migrate()` / `rollback()` / `readLedger()` | the `x_migrations` ledger |
34
34
  | `statementsOf()` | `As of 2026-08`: a SQL script → the statements a driver sends one at a time. One send is one statement, so `migrate()` splits with this — a `;` inside a literal, an identifier, a dollar-quoted body or a comment is data |
35
- | `checkDrift()` / `diffSchema()` / `assertNoDrift()` | drift, with a `--json` report. `checkDrift()` is the **post-migrate verification** — the live database against the ledger: columns, declared indexes (columns, uniqueness, direction, and whether a predicate is there at all never its text) and declared foreign keys, matched on where the key points and not on its constraint name, with the `on delete` rule compared through one normalisation `As of 2026-08-19` |
35
+ | `withReplicaReads()` / `replicaScope()` | `As of 2026-08-24`: the scope inside which a plain read may be served by a replicauntil it writes, after which every read in it is the primary's. No scope open, nothing routes |
36
+ | `replicatedClient()` / `ReplicaStats` / `REPLICA_URL_ENV` | `As of 2026-08-24`: one `DbClient` over a primary and a standby. `baseClient()` builds one when `DATABASE_REPLICA_URL` is set and the single-pool client when it is not |
37
+ | `INDEX_METHODS` / `IndexMethod` / `indexMethodOf()` / `indexMethodSql()` / `declaredMethod()` / `isIndexMethod()` | `As of 2026-08-24`: an index's access method — `btree` or `gin`, closed. Absent is `btree`, the live side is read open (whatever `pg_am` said), and the DDL literal is re-derived from the set rather than spliced from the input |
38
+ | `isPlainRead()` | `As of 2026-08-24`: whether a statement may leave the primary. An allow-list — everything it cannot vouch for is the primary's |
39
+ | `checkDrift()` / `diffSchema()` / `assertNoDrift()` | drift, with a `--json` report. `checkDrift()` is the **post-migrate verification** — the live database against the ledger: columns, declared indexes (access method `As of 2026-08-24`, columns, uniqueness, direction, and whether a predicate is there at all — never its text) and declared foreign keys, matched on where the key points and not on its constraint name, with the `on delete` rule compared through one normalisation `As of 2026-08-19` |
36
40
  | `declaredSchema()` / `expectedSchema()` | `As of 2026-08`: the schema the migrations write down, or `undefined` when the newest one carries no snapshot — never an older snapshot standing in for it |
37
41
  | `parseSnapshot()` | `As of 2026-08`: a `<id>.snapshot.json` sidecar validated to the last nested field, or `undefined`. `{"tables":[null]}` is valid JSON and is not a schema |
38
42
  | `snapshotJson()` | `As of 2026-08`: the sidecar's **bytes** — the JSON Biome would have printed, trailing newline included. The one writer of a `<id>.snapshot.json`, because `JSON.stringify(…, null, 2)` is not formatter-clean and an app's `lint` step rejected the file `x db gen` had just written |
@@ -41,7 +45,7 @@ await withTransaction(async (tx) => {
41
45
  | `generateMigration()` | `x db gen "<name>"` — reversible up/down SQL, and `destructive` for the marker the file must carry. `As of 2026-08` a foreign key is its own `alter table … add constraint`, emitted after every table statement: inline, a `references()` had to point at a table entity registration order happened to create first, and `down` had to drop them in an order it did not control. `As of 2026-08-19` a **removed** `references()` emits its `drop constraint` (it emitted nothing, and the snapshot then denied a constraint the database still held), a changed `onDelete` is a drop-and-add rebuild, and a declared `on delete` rule reaches the clause at all |
42
46
  | `destructiveStatements()` / `hasDestructiveMarker()` / `isDestructive()` / `DESTRUCTIVE_MARKER` | `As of 2026-08`: the destructive-SQL rail — does this `up` drop, truncate or retype, and does the file declare it with `-- destructive: true`? One classifier, read by `x db gen` when it writes the marker and by `x verify` when it demands one |
43
47
  | `stripSqlNoise()` | comments, literals, dollar-quoted bodies and quoted identifiers blanked **in source order**, so a reader sees the operation and not the prose. Shared by `readOnlyQuery()` and the destructive rail |
44
- | `introspect()` | live schema → `SchemaDescription` |
48
+ | `introspect()` | live schema → `SchemaDescription`. **App tables only**, `As of 2026-08-24`: a relation an extension owns (`pg_depend`, `deptype = 'e'`) and anything that is not an ordinary or partitioned table are excluded before the fold, and an explicit `exclude` cannot bring them back |
45
49
  | `createBranch()` / `dropBranch()` / `reapBranches()` | copy-on-write branch databases. `As of 2026-08-19` the marker comment records the **base** as well as the instant (`ultimate:branch:<base>:<iso>`, on `BranchInfo.base`), and `reapBranches()` sweeps only branches of the database it is connected to — one Postgres hosting two Ultimate apps used to mean one app's nightly reap dropped the other's branches. A pre-3.x marker records no base and is skipped, never dropped |
46
50
  | `createPgliteClient()` / `branchPglite()` | the embedded database — Postgres in this process |
47
51
  | `ensureReadOnlyRole()` / `grantReadOnlySql()` / `READONLY_ROLE` | a `NOLOGIN`, SELECT-only Postgres role — layer 1 of `db.query`'s defence |
@@ -98,6 +102,62 @@ statement's own error. Every caller handles that failure; nothing here swallows
98
102
  `ensureReadOnlyRole()` — otherwise a table created later is not selectable by the role, and
99
103
  layer 1 covers only what existed at grant time.
100
104
 
105
+ ## Index access methods
106
+
107
+ `btree` is the default and is never written out; `gin` is what makes `@>` / `<@` / `&&` / `?` on a
108
+ `json()` or `arrayOf()` column an index lookup rather than a sequential scan.
109
+
110
+ | | |
111
+ |---|---|
112
+ | declared | `IndexDescriptionLike.using` — `'btree' \| 'gin'`, closed. Anything else is `X_SQL_UNSAFE` before it reaches DDL |
113
+ | absent | `btree`, on every side, through `indexMethodOf()` — so a snapshot written before the field existed reads as what it always was |
114
+ | emitted | `create index "posts_tags_idx" on "posts" using gin ("tags");` — and `create index "posts_tags_idx" on "posts" ("tags");` when no method was declared, byte for byte what shipped before |
115
+ | refused | a unique GIN and an ordered GIN (`X_INVARIANT`) — Postgres has neither, and the alternative is a syntax error inside `ROLE=migrate` |
116
+ | recorded | only when declared, so no existing sidecar is rewritten |
117
+ | a method that moved | drop and recreate — Postgres cannot alter one in place |
118
+ | drift | a declared GIN against a live btree is `changed-index`, and the live method is reported by the name the catalog gave it |
119
+
120
+ ## Read replicas
121
+
122
+ Opt in twice, and the second one is why it is safe.
123
+
124
+ ```ts
125
+ // 1. the pool — an environment variable, read once by baseClient()
126
+ // DATABASE_REPLICA_URL=postgres://reader@replica.internal:5432/app
127
+
128
+ // 2. the scope — reads inside it may be served by the replica
129
+ import { db, sql, withReplicaReads, withTransaction } from '@ultimat3/db';
130
+
131
+ declare const id: string;
132
+
133
+ await withReplicaReads(async () => {
134
+ await db().query(sql`select id from posts limit 20`); // -> replica
135
+ await db().execute(sql`insert into posts (id) values (${id})`);
136
+ await db().query(sql`select id from posts limit 20`); // -> primary, for the rest of the scope
137
+ await withTransaction(async (tx) => {
138
+ await tx.query(sql`select 1`); // -> primary, always
139
+ });
140
+ });
141
+ ```
142
+
143
+ | Rule | |
144
+ |---|---|
145
+ | unconfigured | no `DATABASE_REPLICA_URL` → one pool, statement for statement what it always was |
146
+ | no scope | no `withReplicaReads` → nothing routes, whatever is configured |
147
+ | read-your-writes | one write anywhere in the scope, at any depth, across any `await`, and every later read in it is the primary's |
148
+ | a transaction | always the primary — `reserve()` is delegated there, so BEGIN, the body and COMMIT are one connection on one server. A `readOnly: true` transaction leaves the scope clean |
149
+ | eligibility | `select` / `table` / `values` / a read-only `with`, minus locking reads, `select … into` and the functions a standby answers instead of refusing (`pg_advisory_lock`, `set_config`, `nextval`). Everything else is the primary's |
150
+ | a replica that fails | the statement is re-run on the primary — exactly-once, because only plain reads are sent there and a `25006` refusal never executed. Three failures in a row park it for ten seconds |
151
+ | observability | `client.stats` (`replica`, `primary`, `fallbacks`, `parked`), and a `db.replica_fallback` warning per fallback |
152
+
153
+ **The URL must name a read-only standby.** The server's own `25006` refusal is the safety net under
154
+ a classifier that cannot be complete; pointed at a writable node a misroute becomes a write on the
155
+ wrong server, silently.
156
+
157
+ **Nothing opens the scope for you yet.** `withReplicaReads` is tier 1 and ships first; wrapping a
158
+ request in it is the app's call — or one line in the HTTP pipeline — so until that lands no
159
+ production traffic is routed.
160
+
101
161
  ## The drift contract
102
162
 
103
163
  `checkDrift()` compares `introspect()` against the snapshot the newest applied migration carries —
@@ -119,8 +179,17 @@ subset, so a database that simply has not migrated yet is pending, not divergent
119
179
  table in the `x_` namespace — `x_migrations`, the queue's tables, the outbox and every
120
180
  `@ultimat3/auth` table are created by `create table if not exists` at boot and appear in no
121
181
  snapshot, so `appTables()` drops them before the diff. `introspect()` keeps its own narrower
122
- exclusion (the ledger alone), because the admin schema view and the MCP `schema.describe` tool
123
- legitimately show `x_users`.
182
+ exclusion (the ledger alone), reserving `x_users` for a schema view that wants it.
183
+
184
+ **Nor is a relation an extension owns, `As of 2026-08-24`.** `create extension pg_stat_statements`
185
+ in `public` is the CNPG, RDS, Supabase and Neon default, and its view read as `unexpected-table`
186
+ with `x db gen "add pg_stat_statements"` as the fix — so every deploy failed terminally and the fix
187
+ would have written an extension's internal view into the app's migration set. `introspect()` now
188
+ excludes every relation Postgres records as extension-owned (`pg_depend`, `deptype = 'e'`), which is
189
+ ownership rather than a name: a `pg_*` prefix rule covers that view and misses `postgis`'
190
+ `spatial_ref_sys`. Views, materialised views and foreign tables go with them — no snapshot records
191
+ one, so counting them could only ever produce a finding an author has no way to clear. A table
192
+ someone created by hand carries no such dependency and is still `unexpected-table`.
124
193
 
125
194
  Rendered output is pinned byte-for-byte:
126
195
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@ultimat3/db",
3
- "version": "11.3.0",
3
+ "version": "13.0.0",
4
4
  "description": "Postgres access, transactions, migrations and drift detection",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -31,7 +31,7 @@
31
31
  "test": "bun test"
32
32
  },
33
33
  "dependencies": {
34
- "@ultimat3/core": "11.3.0"
34
+ "@ultimat3/core": "13.0.0"
35
35
  },
36
36
  "peerDependencies": {
37
37
  "@electric-sql/pglite": ">=0.5.0"
@@ -0,0 +1,57 @@
1
+ // Single responsibility: name the relations in a schema that are NOT app schema, so introspection
2
+ // never puts them in a snapshot and drift can never report one. Two disqualifications, one
3
+ // question: an object an extension owns, and an object that is not a table at all.
4
+
5
+ import type { DbClient } from './client';
6
+ import { sql } from './sql';
7
+
8
+ interface NonAppRelationRow {
9
+ readonly name: string;
10
+ }
11
+
12
+ /**
13
+ * Relation names in `schema` that no migration could legitimately declare.
14
+ *
15
+ * **Extension ownership is the rule, never a name prefix.** Every object `create extension` builds
16
+ * carries a `pg_depend` row with `deptype = 'e'` pointing at `pg_extension` — that is Postgres'
17
+ * own record of "this belongs to an extension", and it is the only thing that generalises. A
18
+ * prefix rule spelled `pg_*` would have covered the `pg_stat_statements` view that made every
19
+ * deploy of the demo app fail terminally (issue #340) and missed `postgis`' `spatial_ref_sys`,
20
+ * `timescaledb`'s catalog and `pg_stat_statements`' own `pg_stat_statements_info` sibling. An
21
+ * extension may install a relation under any name at all, so the name is not evidence.
22
+ *
23
+ * **A view, a materialised view and a foreign table are not tables**, whoever created them. They
24
+ * reach `information_schema.columns` (measured on PGlite: a plain `create view` appears there),
25
+ * while the index query already fences on `relkind = 'r'` — so one came back as a table with
26
+ * columns, no primary key and no indexes, which is a `TableDescription` that cannot be true. No
27
+ * `x db gen` diff emits `create view` and no snapshot records one, so counting them as app schema
28
+ * can only ever produce a finding an author has no way to clear.
29
+ *
30
+ * Excluding by NAME is safe because `pg_class` names are unique within a namespace: a name on this
31
+ * list cannot also be an app table in the same schema.
32
+ */
33
+ export async function nonAppRelations(
34
+ client: DbClient,
35
+ schema: string,
36
+ ): Promise<readonly string[]> {
37
+ const rows = await client.query<NonAppRelationRow>(sql`
38
+ select c.relname as name
39
+ from pg_class c
40
+ join pg_namespace n on n.oid = c.relnamespace
41
+ where n.nspname = ${schema}
42
+ and c.relkind in ('r', 'p', 'v', 'm', 'f')
43
+ and (
44
+ c.relkind in ('v', 'm', 'f')
45
+ or exists (
46
+ select 1
47
+ from pg_depend d
48
+ where d.classid = 'pg_class'::regclass
49
+ and d.objid = c.oid
50
+ and d.refclassid = 'pg_extension'::regclass
51
+ and d.deptype = 'e'
52
+ )
53
+ )
54
+ order by c.relname
55
+ `);
56
+ return rows.map((row) => row.name);
57
+ }
package/src/client.ts CHANGED
@@ -5,6 +5,9 @@
5
5
 
6
6
  import { type Role, renderThrowable, resolveRole } from '@ultimat3/core';
7
7
  import { statementAttribution } from './attribution';
8
+ // Deliberate cycle, the same shape as `client.ts ⇄ transaction.ts`: nothing here is referenced at
9
+ // module evaluation, and both sides are `function` declarations, so hoisting covers the TDZ.
10
+ import { defaultClient } from './default-client';
8
11
  import { DbError, dbUnavailable, driverError, poolAcquireTimeout, poolMaxInvalid } from './errors';
9
12
  import { expectedQueryLoopReason } from './expected-loop';
10
13
  import { declaresLibpqOption, mergeLibpqOptions } from './libpq-options';
@@ -123,7 +126,7 @@ export const POOL_MAX_ENV = 'DATABASE_POOL_MAX';
123
126
  * rather than falling back: a fleet that ignored the number it was given is the failure the
124
127
  * variable exists to prevent, and it would only be found in `pg_stat_activity` at 3am.
125
128
  */
126
- function poolMaxFromEnv(): Partial<PoolProfile> {
129
+ export function poolMaxFromEnv(): Partial<PoolProfile> {
127
130
  const raw = process.env[POOL_MAX_ENV];
128
131
  if (raw === undefined || raw.trim() === '') return {};
129
132
  const max = Number(raw);
@@ -437,10 +440,11 @@ export function setDbClient(client: DbClient | undefined): void {
437
440
  * The role default is layered under `DATABASE_POOL_MAX`, because this is the one place the process
438
441
  * builds its own client and therefore the only place an operator's value can reach one:
439
442
  * `createPostgresClient` has always taken a `profile` override and nothing in a running app passed
440
- * it, so `POOL_PROFILES` was the last word in a deployed image.
443
+ * it, so `POOL_PROFILES` was the last word in a deployed image. `default-client.ts` owns what gets
444
+ * built — one pool, or a primary and a replica when `DATABASE_REPLICA_URL` names one.
441
445
  */
442
446
  export function baseClient(): DbClient {
443
- if (ambient === undefined) ambient = createPostgresClient({ profile: poolMaxFromEnv() });
447
+ if (ambient === undefined) ambient = defaultClient();
444
448
  return ambient;
445
449
  }
446
450
 
@@ -0,0 +1,35 @@
1
+ // Single responsibility: the client `baseClient()` builds when an app installed none — the primary
2
+ // pool, plus a read replica when one is configured. Its own file because it is the one place the
3
+ // framework decides a process's database topology from the environment, and `client.ts` is at the
4
+ // line ceiling.
5
+
6
+ import { createPostgresClient, type DbClient, poolMaxFromEnv } from './client';
7
+ import { replicatedClient } from './replica-client';
8
+
9
+ /**
10
+ * A read replica's connection string. Unset — which is every app that has not asked for one — and
11
+ * this builds exactly the single-pool client it always did, statement for statement.
12
+ *
13
+ * It must name a READ-ONLY standby. A statement misrouted to one is refused with `25006` and
14
+ * re-run on the primary, and that refusal is the safety net a text classifier cannot be; a URL
15
+ * pointing at a writable node turns a misroute into a write on the wrong server, silently.
16
+ */
17
+ export const REPLICA_URL_ENV = 'DATABASE_REPLICA_URL';
18
+
19
+ /**
20
+ * Composed rather than folded into `createPostgresClient`, on purpose: `migrate`, `x db branch` and
21
+ * every test build a client that must be exactly one pool, and a second pool reachable through the
22
+ * same factory would be a second thing `reserve()`, `close()` and `ping()` each have to mean two
23
+ * ways.
24
+ *
25
+ * The replica inherits the role's profile, `DATABASE_POOL_MAX` included: a fleet sized against
26
+ * `max_connections` has two servers to size against, not one, and a replica pool that ignored the
27
+ * operator's number would be the exhaustion `poolMaxFromEnv` exists to prevent, on the other host.
28
+ */
29
+ export function defaultClient(): DbClient {
30
+ const profile = poolMaxFromEnv();
31
+ const primary = createPostgresClient({ profile });
32
+ const replicaUrl = process.env[REPLICA_URL_ENV];
33
+ if (replicaUrl === undefined || replicaUrl.trim() === '') return primary;
34
+ return replicatedClient(primary, createPostgresClient({ url: replicaUrl, profile }));
35
+ }
package/src/drift.ts CHANGED
@@ -6,6 +6,7 @@
6
6
  import { baseClient, type DbClient } from './client';
7
7
  import { DbError } from './errors';
8
8
  import { addForeignKey, dropForeignKey, foreignKeyTarget, onDeleteRule } from './foreign-key';
9
+ import { indexMethodOf } from './index-method';
9
10
  import {
10
11
  type ForeignKeyDescription,
11
12
  findTable,
@@ -219,6 +220,12 @@ function changedForeignKey(
219
220
  * a `desc` index rebuilt ascending by hand serves a feed's newest page off the wrong end. `asc` is
220
221
  * normalised to `null` first — `createIndex` writes `"col" asc`, which Postgres stores as
221
222
  * not-descending, so the raw values differ on every ascending index in a correct database.
223
+ *
224
+ * The **access method** is compared for the same reason and normalised the same way (`indexMethodOf`,
225
+ * absent = `btree`). Without it an emitter that can write `using gin` ships a btree for a declared
226
+ * GIN index and nothing anywhere says so — a declared-and-never-wired key, which is worse than the
227
+ * missing capability, and the reason `using` could not land in `@ultimat3/entity` before it landed
228
+ * here.
222
229
  */
223
230
  function compareIndexes(live: TableDescription, expected: TableDescription): DriftDifference[] {
224
231
  const differences: DriftDifference[] = [];
@@ -229,6 +236,17 @@ function compareIndexes(live: TableDescription, expected: TableDescription): Dri
229
236
  differences.push(missingIndex(live.name, index.name));
230
237
  continue;
231
238
  }
239
+ // The method first, and before the column list: a GIN index and a btree over the same column
240
+ // are not the same index with a detail different, they are two structures the planner uses for
241
+ // different operators — `@>` on a jsonb column is a sequential scan on the wrong one. Both
242
+ // sides go through `indexMethodOf`, so a snapshot that predates the field and an index the
243
+ // catalog reports as `btree` agree instead of reporting drift on a correct database.
244
+ if (indexMethodOf(counterpart) !== indexMethodOf(index)) {
245
+ differences.push(
246
+ changedIndex(live.name, index.name, `is a ${indexMethodOf(counterpart)} index`),
247
+ );
248
+ continue;
249
+ }
232
250
  if (counterpart.columns.join(',') !== index.columns.join(',')) {
233
251
  differences.push(
234
252
  changedIndex(live.name, index.name, `covers (${counterpart.columns.join(', ')})`),
@@ -2,6 +2,8 @@
2
2
  // tier 1 and may never import `entity` (tier 2), so a snapshot arrives as a parameter and every
3
3
  // part of it — a column's `on delete` rule included — crosses the seam by shape or not at all.
4
4
 
5
+ import type { IndexMethod } from './index-method';
6
+
5
7
  /** Structurally assignment-compatible with `@ultimat3/entity`'s `ColumnDescription`. */
6
8
  export interface ColumnDescriptionLike {
7
9
  readonly property: string;
@@ -19,6 +21,17 @@ export interface ColumnDescriptionLike {
19
21
  * the field travelling structurally is the *only* way the rule crosses the tier boundary.
20
22
  */
21
23
  readonly onDelete?: string | null | undefined;
24
+ /**
25
+ * The `generated always as (<expr>) stored` body, when the DATABASE computes this column rather
26
+ * than a writer. Absent on every ordinary column, exactly like `IndexDescriptionLike.using`: a
27
+ * description written before this existed emits the statement it always emitted.
28
+ *
29
+ * `@ultimat3/entity` (tier 2) is the declarer and this package cannot import it, so the
30
+ * expression crosses the seam structurally or it reaches no DDL at all — which is where it was
31
+ * until `As of 2026-08-24`: the column landed as a plain `tsvector not null` and the first insert
32
+ * was a `23502`, because nothing computed it.
33
+ */
34
+ readonly generated?: string | undefined;
22
35
  }
23
36
 
24
37
  /**
@@ -37,6 +50,13 @@ export interface IndexDescriptionLike {
37
50
  readonly where: string | null;
38
51
  /** `null` is Postgres' own default (`asc`), never written out. */
39
52
  readonly order: 'asc' | 'desc' | null;
53
+ /**
54
+ * The access method. Absent is `btree`, which is Postgres' own default and what every index
55
+ * declared before this field existed is — so a description written without it still satisfies
56
+ * the shape, exactly as `ColumnDescriptionLike.onDelete` does. `@ultimat3/entity` (tier 2) is
57
+ * the declarer; this package cannot import it, so the method crosses the seam structurally.
58
+ */
59
+ readonly using?: IndexMethod | undefined;
40
60
  }
41
61
 
42
62
  /** Structurally assignment-compatible with `@ultimat3/entity`'s `EntityDescription`. */
package/src/errors.ts CHANGED
@@ -426,6 +426,22 @@ export const multipleStatements = (statement: string, count: number): DbError =>
426
426
  meta: { count },
427
427
  });
428
428
 
429
+ /**
430
+ * An index method that is not one of the set. `X_SQL_UNSAFE` for the reason `isolationLevelInvalid`
431
+ * uses it: `create index … using <method>` takes no parameters, so the method is SPLICED into the
432
+ * statement text, and an operand TypeScript never saw is an injection rather than a typo — the
433
+ * identical hole a `columnName` built from an unvalidated `meta.name` carried into `create table`.
434
+ *
435
+ * `describeValue`, never the value, for the reason that one gives: this cause is folded into a
436
+ * problem document and a log line.
437
+ */
438
+ export const indexMethodInvalid = (received: unknown): DbError =>
439
+ new DbError({
440
+ code: 'X_SQL_UNSAFE',
441
+ cause: `an index method must be 'btree' or 'gin'; got ${describeValue(received)}`,
442
+ fix: "indexes: [{ on: ['tags'], using: 'gin' }] # or leave it out for the btree default",
443
+ });
444
+
429
445
  export const branchExists = (branch: string): DbError =>
430
446
  new DbError({
431
447
  code: 'X_BRANCH_EXISTS',