@ultimat3/testing 1.2.0 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CLAUDE.md ADDED
@@ -0,0 +1,57 @@
1
+ # @ultimat3/testing — boundary
2
+
3
+ Tier 5. May import tiers 0–4. Imported by every package's tests and by generated apps.
4
+
5
+ Deps: `core` (tier 0), plus `time`, `jobs`, `mail`, `db` and `entity` — imported **dynamically
6
+ inside the fixture factories only**, so a test that never destructures `mail` never loads the mail
7
+ package and a `packages/core` test never loads the entity registry. `entity` is a dependency for
8
+ exactly one call: `nPlusOne()`, so the strict fixture reports the error `x dev` reports, with the
9
+ `fix:` the schema's own relations spell. A second N+1 code owned here would be a second answer to
10
+ one condition. The one static `entity` import is `registry-isolation.ts`, which is why that module
11
+ is its own entry point and not part of the barrel.
12
+
13
+ | Rule | Detail |
14
+ |---|---|
15
+ | No mocks of the DB | clone a template database; `template-db.ts` is the only DB path |
16
+ | No wall clock | `frozenClock` / `advanceClock`; `Date.now()` is frozen by the preload |
17
+ | Frozen ≠ different | `globalThis.Date` becomes a subclass, so `FrozenDate[Symbol.hasInstance]` brands on the `[[DateValue]]` slot — `Date.prototype.getTime.call(value)` throws or it doesn't. Without it `value instanceof Date` is false for every Date the runtime built itself — a `timestamptz` off a Postgres socket, a `structuredClone`, anything from another realm — and the guards that read it fail under test and nowhere else |
18
+ | Slot, not prototype | the brand is cross-realm on purpose: `instanceof RealDate` misses a `node:vm` or worker Date, and `Object.prototype.toString` is spoofable by `Symbol.toStringTag: 'Date'`. Only the slot is both |
19
+ | No unmocked egress | `sealed-network.ts` patches fetch; a miss is `X_TEST_NETWORK_SEALED` |
20
+ | Self is not egress | a port core's `markListening()` announced passes through — a socket test never unseals |
21
+ | Offline is a state, not a mock | `network.offline()` / `.drop()` fail every request as `X_TEST_NETWORK_OFFLINE`, ahead of the mocks — the app's own offline path runs |
22
+ | One way offline | the `network` fixture. `setNetworkState` is the gate's only writer and is not exported — setting it from a test body skips the fixture's disposal and leaves every later file offline |
23
+ | No retries | a flake is fixed or deleted the day it flakes; there is no `retry: 3` |
24
+ | Test names | the filename picks the step; `testName(type, name)` on the outer `describe` puts that type on every failure line under it. Never on the inner `test` too — the prefix would print twice |
25
+ | Injection | `SqlRunner` and `connect` are parameters, so unit tests need no server |
26
+ | Fixtures | the preload registers the whole framework bag — an app registers only what the framework cannot know (`seed`, `actorFor`) |
27
+ | e2e without a driver | `e2eTest` becomes `test.skip`, and the gate reports the step GREEN over it — `bun test` exits 0 on a skip and the exit code is the only channel between the step and the child that registers the driver. `hasE2eDriver()` is what a harness asks instead of reading an all-skipped run as a pass. Zero drivers are registered `As of 2026-08` |
28
+ | Built vs declared | `clock` `mail` `network` `runJobs` `statements` are built in-process; `page` `budget` `signIn` `deploy` `subscribe` are declared and wait for a driver (`X_TEST_FIXTURE_UNAVAILABLE`) |
29
+ | Strict is opt-in by destructuring | `statements` installs the N+1 detector in throw mode for one test. A fixture nobody names is a fixture nobody built, so there is no `strict: true` and no suite-wide switch — and no way to leave it on for the next file |
30
+ | One threshold, one error | `N_PLUS_ONE_THRESHOLD` and `nPlusOne()` are `@ultimat3/entity`'s. A number or a message written here would make a loop that fails a test a different loop from the one `x dev` warns about |
31
+ | The unit of work is the test | `x dev`'s ledger tallies per `Ctx` and ignores a statement issued outside a request; this counts every statement from build to disposal, because `posts.findById(id)` in a unit test has no request and is exactly the loop worth catching |
32
+ | Throws once per shape | the failing line is the loop's own statement (the seam lets `onStatement` throw for this reason alone). It keeps counting after, so a body that catches the error still reports the whole loop through `shapes()` — the same hole Bullet's `raise` has, named rather than papered over |
33
+ | Measure vs judge | `all()` `count()` `shapes()` count `expectedQueryLoop` statements too; only the verdict honours the suppression, so "this page issues two statements" never depends on who declared what |
34
+ | One seam for drivers | a driver registers over a declaration with `defineFixtures` — merges, last wins. Never a second registration mechanism |
35
+ | A driver arrives whole | `defineFixtures` holds every name `Fixtures` declares to its declared type, so a half-built `page` is a compile error at the registration, not a missing method three awaits later |
36
+ | Registry hygiene | the fixture registry is process-global; a test that clears it snapshots with `fixtureSnapshot()` and hands it back in `afterAll` |
37
+ | Leaks are the file's, not the next file's | `installRegistryLeakGuard()` runs from the preload and fails the run naming the FILE that left cache tags declared or a cache tier registered after its last test (`X_TEST_REGISTRY_LEAK`). `bun test` is one process, so without it the failure lands on an innocent suite in another package. What a file's MODULE graph declares is its environment; what the file installs after that is its own to undo |
38
+ | The baseline is not a hook | measured on Bun 1.3.14 the order is onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload `beforeEach`, so a preload hook cannot sample before the file's own `beforeAll` — a `declareTags()` there read as environment and the run went green. The load handler appends the sample to the file's source instead: after evaluation, before any hook the file registers. It is also the only signal carrying file identity, which `bun:test` hooks do not |
39
+ | Guarded state is boot state | only the two registries whose honest invariant is "clean when the file ends" — `declareTags` and `registerTier` are boot installs. `entity()`, `job()` and `defineRoute()` register at MODULE scope, which is how an app declares itself, so a filled registry there is idiomatic and unguarded |
40
+ | An empty registry is a premise you state | a test whose subject is "nothing is declared" — `x db gen` with nothing to generate — calls `isolateEntityRegistry()` and restores in a `finally`. Inheriting it means the test passes until a neighbouring file imports an entity |
41
+ | That one helper is off the barrel | `@ultimat3/testing/registry-isolation`, its own entry point. It is the only module here that value-imports `@ultimat3/entity` — the restore is handed back synchronously, so it cannot be a dynamic import inside the call — and a static re-export from `src/index.ts` would load the entity registry into every test that imports this package for `expect` |
42
+ | Teardown restores, never uninstalls | `describeApp`/`testApp` capture the seal and the determinism snapshot before booting and put those back — `restoreDeterminism()` in a scope hands the REAL clock and the REAL `fetch` to every later FILE in the process. `captureDeterminism()` / `restoreCapturedDeterminism()` are the pair for any nested install |
43
+ | Teardown is a `finally` | an `app.close()` that rejects still reaches `db.drop()` and still restores the process state; the first failure is what the caller sees. A stranded clone is one `ultimate_test_template_wN` leaked per failing run |
44
+ | A boot that rejects is its own teardown | `acquireWorkerDatabase`, `seed` or `boot` throwing returns no `BootedHarness`, so no caller can ever reach `close()` — `bootApp` drops the clone it acquired and restores the seal, the allow-list and the clock itself, then rethrows the boot's OWN error. A `drop` that also fails is swallowed: there is no handle left to report a second failure through |
45
+ | A found template is not a migrated one | `template-db.ts` tolerates "already exists" for the `CREATE DATABASE` alone. `config.migrate` runs unconditionally and un-swallowed — on any Postgres that outlives one run the template is found, not created, and skipping it clones the first run's schema forever |
46
+ | Fixture teardown | a fixture that installs process-global state (the ambient job or mail driver) implements `Symbol.dispose` / `Symbol.asyncDispose` and restores what was there; `fixtureTest` disposes in reverse build order even when the body throws |
47
+ | Building one by hand | `createRunJobs()` outside `fixtureTest` is not disposed for you — reset the driver in `afterEach`, or the next file in the process inherits your queue |
48
+ | Factory strategy | an association is built with the strategy that asked for it: `build()` never reaches a database, `create()` writes the parent first. Never a third strategy |
49
+ | One write seam | `usePersister` is the only place `create()` writes. A factory that took a repo argument would put the seam at every call site |
50
+ | Factory seeds | derived from the table name unless given, so two entities never draw the same uuid stream. `reset()` cascades into associated parents — a half-reset row is worse than none |
51
+ | Shared examples | `behavesLike` calls `describe`, so it goes at declaration scope; bun rejects a `describe` inside a test body |
52
+ | Which command shards | `bun test` is one process on one database, and that is still what a scaffolded app's `test` script runs. `x verify` DOES shard its parallel test steps, over `ULTIMATE_TEST_WORKER` and one database per worker; `live` and `e2e` stay serial because a replication slot is cluster-scoped and `e2e` has one built `dist/`. Say which command a claim is about |
53
+
54
+ Commands: `bun test`, `bunx tsc --noEmit -p tsconfig.json`.
55
+
56
+ Entry points: `.` (the API), `./preload` (side effects for bunfig) and `./registry-isolation`
57
+ (`isolateEntityRegistry()`, kept off `.` because it loads `@ultimat3/entity`).
package/README.md CHANGED
@@ -18,9 +18,11 @@ frozen clock. Never let a test reach the network unmocked — it fails by design
18
18
  | `test-types.ts` | the six test types and their helpers |
19
19
  | `matchers.ts` | `toBeUltimateError` `toDenyPolicy` `toEmitSteps` `toMatchOpenApi` `toBeWithinBudget` `toRejectInput` |
20
20
  | `fixtures.ts` | the registry + `test('…', ({ clock }) => …)` injection |
21
- | `fixture-{clock,mail,jobs,network}.ts` | the four fixtures the framework builds in-process |
21
+ | `fixture-{clock,mail,jobs,network,statements}.ts` | the five fixtures the framework builds in-process |
22
22
  | `fixture-drivers.ts` | the five it declares but a driver must build — `page` `budget` `signIn` `deploy` `subscribe` |
23
23
  | `framework-fixtures.ts` | registers both sets; the app registers only what it owns |
24
+ | `registry-leak-guard.ts` | fails the run naming the FILE that left a process-global registry dirty |
25
+ | `registry-isolation.ts` | `isolateEntityRegistry()` — an empty entity registry, and the process's back after. Its own entry point (`@ultimat3/testing/registry-isolation`), never the barrel: it value-imports `@ultimat3/entity`, and the barrel is what a tier-0 test imports for `expect` |
24
26
  | `preload.ts` | the bunfig preload that installs all of the above |
25
27
 
26
28
  ## Install
@@ -53,6 +55,7 @@ test('the three-day sleep releases the worker', async ({ clock, runJobs }) => {
53
55
  | `mail` | `outbox()` · `lastTo(address)` · `failOnce(mail)` over an in-memory transport | the preload |
54
56
  | `network` | `offline()` · `drop()` · `online()` · `state()` over the sealed network | the preload |
55
57
  | `runJobs` | a worker: call it to enqueue+drain, then `drain()` `due()` `inFlight()` `depth()` | the preload |
58
+ | `statements` | every statement the test issued: `all()` `count(fingerprint?)` `shapes()` — and an N+1 throws | the preload |
56
59
  | `page` | the browser: `goto` `gotoStreamed` `getByRole` `evaluate` `waitForServiceWorker` | a browser driver |
57
60
  | `budget` | `jsBytes(route)` measured off the built output | a browser driver |
58
61
  | `signIn` | put the browser session in a member's shoes | a browser driver |
@@ -88,6 +91,33 @@ that *is* registered — never `undefined is not an object` from inside the body
88
91
  registered but has no driver fails with `X_TEST_FIXTURE_UNAVAILABLE` instead; the two are different
89
92
  instructions, so they are different codes.
90
93
 
94
+ ## An N+1 fails the test it happened in
95
+
96
+ ```ts
97
+ test('the feed reads its authors once', async ({ statements }) => {
98
+ await renderFeed(); // a per-row findById throws here:
99
+ // X_N_PLUS_ONE_QUERY: members.findById ran 5 times in one request — one read per row
100
+ // fix: db.posts.preload('author') # one statement for the whole page
101
+ expect(statements.count('posts.findMany')).toBe(1);
102
+ expect(statements.shapes()[0]?.count).toBe(1);
103
+ });
104
+ ```
105
+
106
+ Opting in is naming it. `statements` installs `@ultimat3/db`'s statement observer for the length of
107
+ one test and hands the seam back afterwards, so there is no `strict: true` to remember and no
108
+ suite-wide switch to forget.
109
+
110
+ | | |
111
+ |---|---|
112
+ | **the unit of work is the test** | `x dev`'s ledger counts per request and skips a statement issued outside one; a unit test calling `posts.findById(id)` has no request anywhere, and that is the loop it was written to catch |
113
+ | **one threshold** | `N_PLUS_ONE_THRESHOLD` from `@ultimat3/entity`, the number `x dev` warns at. A loop that fails a test and a loop that warns in dev are the same loop |
114
+ | **one error** | `nPlusOne()`'s, so the `fix:` is the `preload()` the schema's own relations spell — never a line this package composes |
115
+ | **it throws where it happened** | the loop's fifth statement rejects, so the failing line is the loop's own. Once per shape: a body that catches it gets one failure, not one per statement after it |
116
+ | **measurement ≠ verdict** | `all()` `count()` `shapes()` count every statement, `expectedQueryLoop` ones included; only the verdict honours the suppression |
117
+
118
+ `expectedQueryLoop(reason, fn)` from `@ultimat3/db` stays the one way to declare a loop deliberate —
119
+ there is no flag on the fixture and no code to silence.
120
+
91
121
  ## The six test types
92
122
 
93
123
  | Helper | Asserts | `x verify` step |
@@ -96,7 +126,7 @@ instructions, so they are different codes.
96
126
  | `contractTest` | OpenAPI diff vs the committed spec, MCP exposure | `contract` |
97
127
  | `liveTest` | exactly what each subscriber receives | `live` |
98
128
  | `jobTest` | step sequence, retries, idempotency | `job` |
99
- | `e2eTest` | Playwright incl. offline mode + SW update | `e2e` |
129
+ | `e2eTest` | a browser driver incl. offline mode + SW update; with none registered it SKIPS, and the gate's `e2e` step passes over the skip — ask `hasE2eDriver()` rather than reading that as a pass | `e2e` |
100
130
  | `evalTest` | LLM output scoring against a threshold | `eval` |
101
131
 
102
132
  Each helper prefixes the test name with its type (`job · onboards an org`), which is what
@@ -161,11 +191,12 @@ The first worker creates the template under a Postgres advisory lock and migrate
161
191
  worker then clones it copy-on-write. With no Postgres configured it falls back to PGlite, so
162
192
  `bun test` works on a laptop with nothing installed.
163
193
 
164
- **Parallelism is opt-in, not the default.** `As of 2026-08`:
194
+ **The gate shards; a bare `bun test` does not.** `As of 2026-08`:
165
195
 
166
196
  | Command | Processes | Worker ids | Databases |
167
197
  |---|---|---|---|
168
- | `bun test` (what a scaffolded app's `test` script runs, and what every `x verify` test step runs) | 1 | `0` | one |
198
+ | `bun test` (what a scaffolded app's `test` script still runs) | 1 | `0` | one |
199
+ | `x verify` (`unit`, `contract`, `job`, `eval`; `live` and `e2e` stay serial) | `clamp(round(cpus * 1.5), 2, 8)` | `0..N-1`, from `ULTIMATE_TEST_WORKER` | N |
169
200
  | `bun test --parallel[=N]` | N (default: CPU count) | `1..N`, from Bun's own `BUN_TEST_WORKER_ID` | N |
170
201
  | `x test --workers N` | N | `0..N-1`, from `ULTIMATE_TEST_WORKER` | N |
171
202
 
@@ -173,9 +204,26 @@ worker then clones it copy-on-write. With no Postgres configured it falls back t
173
204
  its own `--parallel` worker — measured on Bun 1.3.14, `--parallel` populates `BUN_TEST_WORKER_ID`
174
205
  and `JEST_WORKER_ID` itself, so that precedence is load-bearing rather than defensive.
175
206
 
207
+ ## The harness puts back what it found
208
+
209
+ `bun test` is one process, so `describeApp`/`testApp` teardown is a **restore**, never an uninstall.
210
+ `As of 2026-08`:
211
+
212
+ | State | Owned by | What teardown does |
213
+ |---|---|---|
214
+ | the seal on `fetch` | the preload | unseals only if this boot was the one that sealed |
215
+ | the frozen instant, `Math.random`, `globalThis.Date` | the preload (`ULTIMATE_TEST_NOW` / `ULTIMATE_TEST_SEED`) | `restoreCapturedDeterminism(captureDeterminism())` around the boot |
216
+ | mocks, allow-listed hosts, the seen list | the boot | `resetNetwork()` |
217
+ | the cloned worker database | the boot | `db.drop()`, in a `finally` — a rejecting `app.close()` reaches it |
218
+
219
+ `installDeterminism()` runs during a boot only when the boot has something of its own to say
220
+ (`seedValue`/`now`) or nothing installed it yet, so a run configured with `ULTIMATE_TEST_NOW` is not
221
+ reset by the first `describeApp`. `restoreDeterminism()` is the process's own call, not a scope's:
222
+ it hands the real clock and the real `Math.random` back to every later **file** in the run.
223
+
176
224
  ## Sealed network
177
225
 
178
- ```
226
+ ```text
179
227
  X_TEST_NETWORK_SEALED
180
228
  cause: POST https://api.stripe.com/v1/charges was not mocked (allowed hosts: none)
181
229
  fix: mockFetch('https://api.stripe.com/v1/charges', () => new Response('{}')) — or allowHost('api.stripe.com') if it must be real
@@ -189,4 +237,38 @@ integration — never for a socket test.
189
237
  ## Errors
190
238
 
191
239
  `X_TEST_NETWORK_SEALED` `X_TEST_DB_UNAVAILABLE` `X_TEST_NONDETERMINISTIC` `X_TEST_FIXTURE_UNKNOWN`
192
- `X_TEST_FACTORY_TRAIT_UNKNOWN` `X_TEST_FACTORY_NOT_PERSISTED`
240
+ `X_TEST_FACTORY_TRAIT_UNKNOWN` `X_TEST_FACTORY_NOT_PERSISTED` `X_TEST_REGISTRY_LEAK`
241
+
242
+ ## One process, one registry
243
+
244
+ `bun test` runs every file of one invocation in the same process — only `x verify`'s shards pass
245
+ `--isolate`. A file that leaves a process-global registry dirty therefore changes what every later
246
+ file sees, and the failure lands on an innocent suite in another package: `bun test packages/query
247
+ packages/cli` failed five tests in `query`, all of them installed by `cli`, while either package
248
+ alone was green.
249
+
250
+ The preload installs the guard. It samples the cache tag set and the cache tier registry once per
251
+ file, at the end of that file's module evaluation — so an app's own boot declarations are its
252
+ environment — and reports what the file added after that and did not put back:
253
+
254
+ ```text
255
+ X_TEST_REGISTRY_LEAK: a test file left a process-global registry dirty —
256
+ "packages/cli/src/cmd-dev.test.ts" left cache tags declared ["devfixture"] after its last test
257
+ fix: in "packages/cli/src/cmd-dev.test.ts" add: import { isolateDeclaredTags } from
258
+ '@ultimat3/cache'; const restoreTags = isolateDeclaredTags(); afterAll(restoreTags);
259
+ — then re-run: bun test "packages/cli/src/cmd-dev.test.ts"
260
+ ```
261
+
262
+ The baseline is not a `beforeEach`: a file's own `beforeAll` runs **before** a preload's
263
+ `beforeEach` (onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload
264
+ `beforeEach`, measured on Bun 1.3.14), so a `declareTags()` in `beforeAll` would have been sampled
265
+ as the file's environment and the run would have gone green. The guard appends the sample to the
266
+ file's own source in its load handler instead — the one place a file's identity and its evaluation
267
+ boundary are both known.
268
+
269
+ The fix is `isolateDeclaredTags()` or `isolateTiers()` (both `@ultimat3/cache`), never a loosened
270
+ assertion in the file that paid for it — and never a **reset**. A reset drops what a neighbour
271
+ registered, and this guard reports additions only, so the damage lands on an innocent file with
272
+ nothing pointing back. A leak fails a one-file run exactly as it fails the suite —
273
+ each file is judged against its own baseline — which is what makes the `bun test <file>` in the fix
274
+ line reproduce it.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@ultimat3/testing",
3
- "version": "1.2.0",
3
+ "version": "3.0.0",
4
4
  "description": "Test harness: cloned template DBs per worker, frozen clock, sealed network, 6 test types",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -15,11 +15,13 @@
15
15
  },
16
16
  "exports": {
17
17
  ".": "./src/index.ts",
18
- "./preload": "./src/preload.ts"
18
+ "./preload": "./src/preload.ts",
19
+ "./registry-isolation": "./src/registry-isolation.ts"
19
20
  },
20
21
  "files": [
21
22
  "src",
22
23
  "!src/**/*.test.ts",
24
+ "CLAUDE.md",
23
25
  "README.md",
24
26
  "LICENSE"
25
27
  ],
@@ -31,10 +33,12 @@
31
33
  "test": "bun test"
32
34
  },
33
35
  "dependencies": {
34
- "@ultimat3/core": "1.2.0",
35
- "@ultimat3/db": "1.2.0",
36
- "@ultimat3/jobs": "1.2.0",
37
- "@ultimat3/mail": "1.2.0",
38
- "@ultimat3/time": "1.2.0"
36
+ "@ultimat3/cache": "3.0.0",
37
+ "@ultimat3/core": "3.0.0",
38
+ "@ultimat3/db": "3.0.0",
39
+ "@ultimat3/entity": "3.0.0",
40
+ "@ultimat3/jobs": "3.0.0",
41
+ "@ultimat3/mail": "3.0.0",
42
+ "@ultimat3/time": "3.0.0"
39
43
  }
40
44
  }
@@ -94,6 +94,35 @@ export function restoreDeterminism(): void {
94
94
  installed = false;
95
95
  }
96
96
 
97
+ /** Everything `installDeterminism` overwrites, so a nested install can put back what it found. */
98
+ export interface DeterminismSnapshot {
99
+ readonly installed: boolean;
100
+ readonly frozenAt: number;
101
+ readonly random: () => number;
102
+ readonly date: DateConstructor;
103
+ }
104
+
105
+ /**
106
+ * Capture before a nested `installDeterminism`, restore after — the shape `fixture-network.ts`
107
+ * uses for the seal. The preload installs determinism once for the whole process, so an inner
108
+ * scope that called `restoreDeterminism()` would hand the REAL clock and the REAL `Math.random`
109
+ * to every later test file in it. `random` is captured by identity because a generator's state is
110
+ * its closure: re-seeding produces an equal sequence, not the same position in this one.
111
+ */
112
+ export const captureDeterminism = (): DeterminismSnapshot => ({
113
+ installed,
114
+ frozenAt,
115
+ random: Math.random,
116
+ date: globalThis.Date,
117
+ });
118
+
119
+ export function restoreCapturedDeterminism(snapshot: DeterminismSnapshot): void {
120
+ frozenAt = snapshot.frozenAt;
121
+ Math.random = snapshot.random;
122
+ globalThis.Date = snapshot.date;
123
+ installed = snapshot.installed;
124
+ }
125
+
97
126
  export const isDeterminismInstalled = (): boolean => installed;
98
127
 
99
128
  /** Move the frozen clock forward. The only legal way for time to pass inside a test. */
package/src/errors.ts CHANGED
@@ -1,6 +1,15 @@
1
1
  // The X_* codes owned by @ultimat3/testing. A test failure has to be as actionable as a runtime
2
2
  // failure — the fix line here is the mock to add, the service to start, or the seed to freeze.
3
- import { registerErrorCodes, UltimateError } from '@ultimat3/core';
3
+ import {
4
+ registerErrorCodes,
5
+ renderCauseValue,
6
+ renderFixLiteral,
7
+ UltimateError,
8
+ } from '@ultimat3/core';
9
+ // Type-only, so the cycle with the guard that throws it is erased at build: the code is declared
10
+ // here because this file is the package's code registry, and the shape it reports lives with the
11
+ // sampler that produces it.
12
+ import type { RegistryLeak } from './registry-leak-guard';
4
13
 
5
14
  export const TESTING_ERROR_CODES = [
6
15
  'X_TEST_NETWORK_SEALED',
@@ -15,6 +24,7 @@ export const TESTING_ERROR_CODES = [
15
24
  'X_TEST_NETWORK_RACE',
16
25
  'X_TEST_FACTORY_TRAIT_UNKNOWN',
17
26
  'X_TEST_FACTORY_NOT_PERSISTED',
27
+ 'X_TEST_REGISTRY_LEAK',
18
28
  ] as const;
19
29
 
20
30
  export type TestingErrorCode = (typeof TESTING_ERROR_CODES)[number];
@@ -32,6 +42,7 @@ export const TESTING_ERROR_TITLES: Readonly<Record<TestingErrorCode, string>> =
32
42
  X_TEST_NETWORK_RACE: 'a request raced unsealNetwork() and lost the patched fetch',
33
43
  X_TEST_FACTORY_TRAIT_UNKNOWN: 'a factory was asked for a trait it does not declare',
34
44
  X_TEST_FACTORY_NOT_PERSISTED: 'a factory create() had nowhere to write the row',
45
+ X_TEST_REGISTRY_LEAK: 'a test file left a process-global registry dirty',
35
46
  };
36
47
 
37
48
  // Titles must be registered for `format()` to render the contract's first line. Every code above is
@@ -239,3 +250,62 @@ export class NetworkRaceError extends UltimateError {
239
250
  });
240
251
  }
241
252
  }
253
+
254
+ /**
255
+ * Every value in this message is uncontrolled: the path arrives from `Bun.plugin`'s `onLoad`, and
256
+ * the names are whatever the app under test passed to `declareTags()` / `registerTier()`. So the
257
+ * sentence renders them (bounded, never throwing while describing a leak) and the command quotes
258
+ * them (a fix has to parse after a path with a space or a quote in it lands in the middle of it).
259
+ */
260
+ const describeLeak = (leak: RegistryLeak): string => {
261
+ const left = [
262
+ ...(leak.tags.length > 0 ? [`cache tags declared ${renderCauseValue(leak.tags)}`] : []),
263
+ ...(leak.tiers.length > 0 ? [`cache tiers registered ${renderCauseValue(leak.tiers)}`] : []),
264
+ ];
265
+ return `${renderCauseValue(leak.file)} left ${left.join(' and ')} after its last test`;
266
+ };
267
+
268
+ /** The placeholder is the cause's own sentence: it already names every file, in order. */
269
+ const FILE_PLACEHOLDER = '<the file the cause names>';
270
+
271
+ /**
272
+ * The edit, spelled as the lines to paste — the imports included, since neither call is global.
273
+ *
274
+ * Both repairs isolate; neither resets. `afterAll(resetTiers)` was the first spelling of the tier
275
+ * half and it is the bug wearing the fix's clothes: it drops the tiers, the revalidator and both
276
+ * logs a NEIGHBOUR registered, and the guard reports additions only, so the damage lands on an
277
+ * innocent file with nothing pointing back here. An error whose instruction causes the next defect
278
+ * is worse than no instruction.
279
+ */
280
+ const repairFor = (leak: RegistryLeak): string => {
281
+ const imports: string[] = [];
282
+ const calls: string[] = [];
283
+ if (leak.tags.length > 0) {
284
+ imports.push('isolateDeclaredTags');
285
+ calls.push('const restoreTags = isolateDeclaredTags(); afterAll(restoreTags);');
286
+ }
287
+ if (leak.tiers.length > 0) {
288
+ imports.push('isolateTiers');
289
+ calls.push('const restoreTiers = isolateTiers(); afterAll(restoreTiers);');
290
+ }
291
+ const file = renderFixLiteral(leak.file, FILE_PLACEHOLDER);
292
+ return `in ${file} add: import { ${imports.join(', ')} } from '@ultimat3/cache'; ${calls.join(' ')}`;
293
+ };
294
+
295
+ /**
296
+ * One error for every leaker in the run, not one per file: the run has already finished by the
297
+ * time the last file can be judged, and two throws would report the second as an unhandled one.
298
+ * The trailing command reproduces the failure on its own — the guard judges each file against its
299
+ * own baseline, so one leaking file fails a one-file run exactly as it failed the whole suite.
300
+ */
301
+ export class RegistryLeakError extends UltimateError {
302
+ constructor(input: { readonly leaks: readonly RegistryLeak[] }) {
303
+ const files = input.leaks.map((leak) => renderFixLiteral(leak.file, FILE_PLACEHOLDER));
304
+ super({
305
+ code: 'X_TEST_REGISTRY_LEAK',
306
+ cause: input.leaks.map(describeLeak).join('; '),
307
+ fix: `${input.leaks.map(repairFor).join('; ')} — then re-run: bun test ${files.join(' ')}`,
308
+ docs: docsFor('X_TEST_REGISTRY_LEAK'),
309
+ });
310
+ }
311
+ }
@@ -3,7 +3,13 @@
3
3
  // `clock.advance('3d')` is synchronous — a job test asserts on what is due on the very next
4
4
  // line — so the duration parser is resolved while the fixture is built, not when it is used.
5
5
 
6
- import { advanceClock, frozenNow, setFrozenClock } from './determinism';
6
+ import {
7
+ advanceClock,
8
+ captureDeterminism,
9
+ frozenNow,
10
+ restoreCapturedDeterminism,
11
+ setFrozenClock,
12
+ } from './determinism';
7
13
 
8
14
  /** `'3d'` | `'30s'` | `1500`. Same vocabulary as a job's `timeout` and a step's `sleep`. */
9
15
  export type TestDuration = string | number;
@@ -13,10 +19,23 @@ export interface TestClock {
13
19
  now(): Date;
14
20
  advance(duration: TestDuration): Date;
15
21
  set(instant: string | number): Date;
22
+ /** Puts the instant this fixture was built at back. `fixtureTest` calls it; see below. */
23
+ [Symbol.asyncDispose](): Promise<void>;
16
24
  }
17
25
 
26
+ /**
27
+ * The frozen instant is module-global and `advance`/`set` move it, so this fixture installs
28
+ * process state exactly the way the mail and job fixtures do — and until it disposed, a test that
29
+ * advanced three days handed the advanced clock to every later test FILE in the run (`bun test` is
30
+ * one process), where the failure lands on an innocent suite.
31
+ *
32
+ * `captureDeterminism`/`restoreCapturedDeterminism` rather than `restoreDeterminism()`: the preload
33
+ * installed determinism once for the whole process, and uninstalling it here would hand the REAL
34
+ * `Date` and the REAL `Math.random` to everything after. Restore only what was found.
35
+ */
18
36
  export async function createTestClock(): Promise<TestClock> {
19
37
  const { toMs } = await import('@ultimat3/time');
38
+ const captured = captureDeterminism();
20
39
  return {
21
40
  now: frozenNow,
22
41
  advance: (duration) => advanceClock(toMs(duration)),
@@ -24,5 +43,8 @@ export async function createTestClock(): Promise<TestClock> {
24
43
  setFrozenClock(instant);
25
44
  return frozenNow();
26
45
  },
46
+ [Symbol.asyncDispose]: async () => {
47
+ restoreCapturedDeterminism(captured);
48
+ },
27
49
  };
28
50
  }
@@ -0,0 +1,148 @@
1
+ // The `statements` fixture: every statement the test issues, counted by shape — and a shape that
2
+ // repeats past the threshold throws where it happened, so an N+1 fails the test that caused it
3
+ // instead of warning in a dev server nobody is watching.
4
+ //
5
+ // Strict by construction: there is no `strict: true` to remember, because a fixture nobody
6
+ // destructured is a fixture nobody built. Opting in is naming `statements` in the test body.
7
+
8
+ import type { StatementAttribution, StatementEvent, StatementObserver } from '@ultimat3/db';
9
+
10
+ /** One statement, as this fixture kept it. Bound values are deliberately not retained. */
11
+ export interface ObservedStatement {
12
+ /** What it is counted under: `members.findById` when attributed, else its own collapsed text. */
13
+ readonly fingerprint: string;
14
+ readonly kind: 'read' | 'write';
15
+ /** The statement as sent, parameters still `$1..$n`. */
16
+ readonly text: string;
17
+ /** The entity and operation that compiled it; absent for hand-written SQL and queue traffic. */
18
+ readonly attribution?: StatementAttribution | undefined;
19
+ /** The `expectedQueryLoop()` reason in force when it was sent, absent outside every such scope. */
20
+ readonly expected?: string | undefined;
21
+ }
22
+
23
+ /** One shape and how often the test issued it. */
24
+ export interface StatementShape {
25
+ readonly fingerprint: string;
26
+ readonly kind: 'read' | 'write';
27
+ /** Every statement of this shape, expected ones included — this half measures, it never judges. */
28
+ readonly count: number;
29
+ }
30
+
31
+ /**
32
+ * `Disposable`: the observer seam is process-global, so the fixture hands back whatever it found —
33
+ * the state, not a fixed default, exactly as `network` and `runJobs` do.
34
+ */
35
+ export interface TestStatements extends Disposable {
36
+ /** Every statement issued since the fixture was built, in order. */
37
+ all(): readonly ObservedStatement[];
38
+ /** How many of one shape, or of everything when asked with no fingerprint. */
39
+ count(fingerprint?: string): number;
40
+ /** Shapes seen, most repeated first, ties by fingerprint — a stable list to assert against. */
41
+ shapes(): readonly StatementShape[];
42
+ }
43
+
44
+ /** The live tally behind a `StatementShape`: what it measures, and what it judges. */
45
+ interface ShapeTally {
46
+ readonly fingerprint: string;
47
+ readonly kind: 'read' | 'write';
48
+ /** Every statement of this shape. */
49
+ count: number;
50
+ /** The ones no `expectedQueryLoop()` scope covered — the only ones a verdict may count. */
51
+ unexpected: number;
52
+ /** Whether this shape has already thrown. The flag, not the count, so it throws exactly once. */
53
+ failed: boolean;
54
+ }
55
+
56
+ /**
57
+ * Install the strict detector for the length of one test.
58
+ *
59
+ * Three rules, each load-bearing and each different from `x dev`'s ledger for a stated reason.
60
+ *
61
+ * **The unit of work is the test, not the request.** The ledger keys its tally on the `Ctx` object
62
+ * and ignores a statement issued outside a request; a unit test calls `posts.findById(id)` with no
63
+ * request anywhere, which is exactly the loop it was written to catch — so this counts every
64
+ * statement from the moment the fixture is built until it is disposed, whatever context it was
65
+ * issued in.
66
+ *
67
+ * **The verdict counts only unexpected statements, the measurements count all of them.** An
68
+ * `expectedQueryLoop(reason, fn)` scope is the one way to declare a loop deliberate (there is no
69
+ * flag here and no second suppression), and it suppresses the *verdict* — the statements are still
70
+ * sent, still observed, and still reported by `count()` and `shapes()`, because a test asserting
71
+ * "this page issues two statements" must not see a different number depending on who declared what.
72
+ *
73
+ * **It throws where the loop happened, once per shape.** `@ultimat3/db`'s seam deliberately lets a
74
+ * throw from `onStatement` propagate to whoever ran the statement, which is what makes the failing
75
+ * line the loop's own line rather than a summary at teardown. Once a shape has thrown it keeps
76
+ * counting silently: a test that catches the error gets one failure at the statement that crossed
77
+ * the threshold, not one per statement after it, and `shapes()` still reports the whole loop. The
78
+ * consequence worth knowing is the same one Bullet's `raise` has — a body that swallows every error
79
+ * swallows this one too, and `shapes()` is what names the loop in that test.
80
+ *
81
+ * The threshold is `@ultimat3/entity`'s `N_PLUS_ONE_THRESHOLD` and there is no knob: a loop that
82
+ * fails a test and a loop that warns in `x dev` have to be the same loop.
83
+ */
84
+ export async function createTestStatements(): Promise<TestStatements> {
85
+ // Imported on demand, like every other fixture factory here: a `packages/core` test that never
86
+ // names `statements` must not load the database layer or the entity registry to run.
87
+ const db = await import('@ultimat3/db');
88
+ const { N_PLUS_ONE_THRESHOLD, nPlusOne } = await import('@ultimat3/entity');
89
+ // Captured before the overwrite: the seam holds one observer, not a list, so an outer diagnostic
90
+ // is displaced for the length of this test and put back after it.
91
+ const previous = db.statementObserver();
92
+
93
+ const seen: ObservedStatement[] = [];
94
+ const tallies = new Map<string, ShapeTally>();
95
+
96
+ const observer: StatementObserver = {
97
+ onStatement(event: StatementEvent): void {
98
+ const fingerprint = db.statementFingerprint(event);
99
+ const kind = db.statementKind(event.text);
100
+ seen.push({
101
+ fingerprint,
102
+ kind,
103
+ text: event.text,
104
+ attribution: event.attribution,
105
+ expected: event.expected,
106
+ });
107
+ let tally = tallies.get(fingerprint);
108
+ if (tally === undefined) {
109
+ tally = { fingerprint, kind, count: 0, unexpected: 0, failed: false };
110
+ tallies.set(fingerprint, tally);
111
+ }
112
+ tally.count += 1;
113
+ if (event.expected !== undefined) return;
114
+ // A statement that threw is still a statement: fifty identical timeouts are still a loop.
115
+ tally.unexpected += 1;
116
+ if (tally.failed || tally.unexpected < N_PLUS_ONE_THRESHOLD) return;
117
+ tally.failed = true;
118
+ // `@ultimat3/entity`'s error, never one composed here: the `fix:` names the `preload` the
119
+ // schema's own relations spell, and a second answer to "what ends this loop" would be one
120
+ // the schema never agreed to.
121
+ throw nPlusOne({
122
+ kind,
123
+ subject: fingerprint,
124
+ count: tally.unexpected,
125
+ entity: event.attribution?.entity,
126
+ op: event.attribution?.op,
127
+ });
128
+ },
129
+ };
130
+ db.setStatementObserver(observer);
131
+
132
+ return {
133
+ all: (): readonly ObservedStatement[] => [...seen],
134
+ count: (fingerprint?: string): number =>
135
+ fingerprint === undefined ? seen.length : (tallies.get(fingerprint)?.count ?? 0),
136
+ shapes: (): readonly StatementShape[] =>
137
+ [...tallies.values()]
138
+ .map(({ fingerprint, kind, count }) => ({ fingerprint, kind, count }))
139
+ .sort((left, right) =>
140
+ left.count === right.count
141
+ ? left.fingerprint.localeCompare(right.fingerprint)
142
+ : right.count - left.count,
143
+ ),
144
+ [Symbol.dispose]: (): void => {
145
+ db.setStatementObserver(previous);
146
+ },
147
+ };
148
+ }
package/src/fixtures.ts CHANGED
@@ -15,6 +15,7 @@ import type { SignIn, Subscribe, TestBudget, TestDeploy } from './fixture-driver
15
15
  import type { RunJobs } from './fixture-jobs';
16
16
  import type { TestMail } from './fixture-mail';
17
17
  import type { TestNetwork } from './fixture-network';
18
+ import type { TestStatements } from './fixture-statements';
18
19
  import type { PageLike } from './test-types';
19
20
 
20
21
  /** Built once per test, on first use. */
@@ -44,6 +45,7 @@ export interface Fixtures {
44
45
  readonly mail: TestMail;
45
46
  readonly network: TestNetwork;
46
47
  readonly runJobs: RunJobs;
48
+ readonly statements: TestStatements;
47
49
  readonly budget: TestBudget;
48
50
  readonly deploy: TestDeploy;
49
51
  readonly page: PageLike;
@@ -88,6 +90,54 @@ export function fixtureSnapshot(): FixtureMap {
88
90
  return Object.fromEntries(registry);
89
91
  }
90
92
 
93
+ const CLOSERS: Readonly<Record<string, string>> = { '{': '}', '[': ']', '(': ')' };
94
+ const QUOTES = new Set(['"', "'", '`']);
95
+
96
+ /**
97
+ * The pattern's own top-level segments: the `}` that closes the opening `{`, and the commas at
98
+ * depth zero inside it. `indexOf('}')` stopped at the FIRST closer, so `{ mail, clock: { now },
99
+ * network }` lost `network` and `{ clock = { now: 1 }, mail }` lost both — silently, and a fixture
100
+ * that is never built reads as `undefined` in the body, which is the failure this module exists to
101
+ * turn into a message. Strings are skipped so a default like `{ role = 'a, b' }` is one segment.
102
+ */
103
+ function patternSegments(source: string, open: number): readonly string[] | undefined {
104
+ const segments: string[] = [];
105
+ const stack: string[] = [];
106
+ let start = open + 1;
107
+ let quote: string | undefined;
108
+ for (let index = open; index < source.length; index += 1) {
109
+ const char = source[index] ?? '';
110
+ if (quote !== undefined) {
111
+ if (char === '\\') index += 1;
112
+ else if (char === quote) quote = undefined;
113
+ continue;
114
+ }
115
+ if (QUOTES.has(char)) {
116
+ quote = char;
117
+ continue;
118
+ }
119
+ const closer = CLOSERS[char];
120
+ if (closer !== undefined) {
121
+ stack.push(closer);
122
+ continue;
123
+ }
124
+ if (stack.length > 0 && char === stack[stack.length - 1]) {
125
+ stack.pop();
126
+ if (stack.length === 0) {
127
+ segments.push(source.slice(start, index));
128
+ return segments;
129
+ }
130
+ continue;
131
+ }
132
+ if (char === ',' && stack.length === 1) {
133
+ segments.push(source.slice(start, index));
134
+ start = index + 1;
135
+ }
136
+ }
137
+ // An unbalanced pattern is not one this can read; building nothing beats guessing a name.
138
+ return undefined;
139
+ }
140
+
91
141
  /**
92
142
  * The names a body destructures, read from its source.
93
143
  *
@@ -100,15 +150,11 @@ export function requestedFixtures(body: (...args: never[]) => unknown): readonly
100
150
  const source = body.toString();
101
151
  const open = source.indexOf('{');
102
152
  if (open === -1) return [];
103
- const close = source.indexOf('}', open);
104
- if (close === -1) return [];
105
153
  // Bail if the brace opens a body rather than a destructuring pattern — `async () => {`.
106
154
  const beforeBrace = source.slice(0, open);
107
155
  if (/\)\s*(?::[^=]*)?=>\s*$/.test(beforeBrace) || /\)\s*$/.test(beforeBrace)) return [];
108
- return source
109
- .slice(open + 1, close)
110
- .split(',')
111
- .map((part) => (part.split(':')[0] ?? '').trim())
156
+ return (patternSegments(source, open) ?? [])
157
+ .map((part) => (part.split(/[:=]/)[0] ?? '').trim())
112
158
  .filter((name) => /^[A-Za-z_$][\w$]*$/.test(name));
113
159
  }
114
160
 
@@ -7,6 +7,7 @@ import { DRIVER_FIXTURE_NAMES, driverFixtures } from './fixture-drivers';
7
7
  import { createRunJobs } from './fixture-jobs';
8
8
  import { createTestMail } from './fixture-mail';
9
9
  import { createTestNetwork } from './fixture-network';
10
+ import { createTestStatements } from './fixture-statements';
10
11
  import { defineFixtures } from './fixtures';
11
12
 
12
13
  /**
@@ -15,7 +16,13 @@ import { defineFixtures } from './fixtures';
15
16
  * installs. See `fixture-drivers.ts` for why a declared-and-unavailable fixture beats an
16
17
  * unregistered name.
17
18
  */
18
- export const FRAMEWORK_FIXTURE_NAMES = ['clock', 'mail', 'network', 'runJobs'] as const;
19
+ export const FRAMEWORK_FIXTURE_NAMES = [
20
+ 'clock',
21
+ 'mail',
22
+ 'network',
23
+ 'runJobs',
24
+ 'statements',
25
+ ] as const;
19
26
 
20
27
  export { DRIVER_FIXTURE_NAMES };
21
28
 
@@ -38,5 +45,6 @@ export function registerFrameworkFixtures(): void {
38
45
  mail: createTestMail,
39
46
  network: createTestNetwork,
40
47
  runJobs: createRunJobs,
48
+ statements: createTestStatements,
41
49
  });
42
50
  }
package/src/harness.ts CHANGED
@@ -3,8 +3,14 @@
3
3
  // into a race and every failure into a log-scraping exercise.
4
4
 
5
5
  import { afterAll, beforeAll, describe, test } from 'bun:test';
6
- import { installDeterminism, restoreDeterminism } from './determinism';
7
- import { allowHost, resetNetwork, sealNetwork, unsealNetwork } from './sealed-network';
6
+ import { captureDeterminism, installDeterminism, restoreCapturedDeterminism } from './determinism';
7
+ import {
8
+ allowHost,
9
+ isNetworkSealed,
10
+ resetNetwork,
11
+ sealNetwork,
12
+ unsealNetwork,
13
+ } from './sealed-network';
8
14
  import type { TemplateDbConfig, WorkerDatabase } from './template-db';
9
15
  import { acquireWorkerDatabase } from './template-db';
10
16
  import type { TestType } from './test-types';
@@ -36,19 +42,77 @@ export interface AppHandle {
36
42
 
37
43
  const BASE = 'http://app.test';
38
44
 
39
- async function boot(
45
+ export interface BootedHarness {
46
+ readonly handle: AppHandle;
47
+ close(): Promise<void>;
48
+ }
49
+
50
+ export interface HarnessDeps {
51
+ /**
52
+ * How this harness gets a database. A parameter for the reason `TemplateDbDeps.connect` is one:
53
+ * the rejection path below drops what it acquired, and the PGlite fallback's `drop` is a no-op,
54
+ * so proving it needs an injected handle rather than a server.
55
+ */
56
+ readonly acquire: (config: TemplateDbConfig) => Promise<WorkerDatabase>;
57
+ }
58
+
59
+ /**
60
+ * The lifecycle `describeApp` and `testApp` are the two idiomatic wrappers around. Exported from
61
+ * this module but deliberately NOT from `src/index.ts`: those two are the ways an app boots, and a
62
+ * third public entry point would be a second answer to one question. It is exported at all because
63
+ * the teardown contract — a rejecting `close` must still drop the cloned database — cannot be
64
+ * asserted through a wrapper that rethrows into the test's own result.
65
+ */
66
+ export async function bootApp(
40
67
  options: AppOptions,
41
- ): Promise<{ handle: AppHandle; close: () => Promise<void> }> {
42
- installDeterminism({
43
- ...(options.seedValue === undefined ? {} : { seed: options.seedValue }),
44
- ...(options.now === undefined ? {} : { now: options.now }),
45
- });
68
+ deps: Partial<HarnessDeps> = {},
69
+ ): Promise<BootedHarness> {
70
+ // Captured, never assumed. The preload already sealed the network and installed determinism for
71
+ // the whole process, so `sealNetwork()` here is a no-op and an unconditional teardown would hand
72
+ // the real `fetch`, the real `Date` and the real `Math.random` to every later FILE in the run —
73
+ // `bun test` is one process. Restore only what this boot actually changed.
74
+ const determinism = captureDeterminism();
75
+ const sealedBefore = isNetworkSealed();
76
+
77
+ // Only when this boot has something of its own to say. A run configured with ULTIMATE_TEST_NOW /
78
+ // ULTIMATE_TEST_SEED (`preload.ts`) is otherwise reset to the defaults by the first describeApp.
79
+ if (!determinism.installed || options.seedValue !== undefined || options.now !== undefined) {
80
+ installDeterminism({
81
+ ...(options.seedValue === undefined ? {} : { seed: options.seedValue }),
82
+ ...(options.now === undefined ? {} : { now: options.now }),
83
+ });
84
+ }
46
85
  sealNetwork();
47
86
  for (const host of options.allowHosts ?? []) allowHost(host);
48
87
 
49
- const db = await acquireWorkerDatabase(options.db ?? {});
50
- if (options.seed !== undefined) await options.seed(db.url);
51
- const app = await options.boot({ databaseUrl: db.url });
88
+ // What `close` puts back, named once: the boot has to run it too, and two copies of this list is
89
+ // how one of them ends up missing the allow-list.
90
+ const restoreProcessState = (): void => {
91
+ resetNetwork();
92
+ if (!sealedBefore) unsealNetwork();
93
+ restoreCapturedDeterminism(determinism);
94
+ };
95
+
96
+ let db: WorkerDatabase | undefined;
97
+ let app: BootedApp;
98
+ try {
99
+ db = await (deps.acquire ?? acquireWorkerDatabase)(options.db ?? {});
100
+ if (options.seed !== undefined) await options.seed(db.url);
101
+ app = await options.boot({ databaseUrl: db.url });
102
+ } catch (error) {
103
+ // A boot that rejects returns no `BootedHarness`, so nothing can ever call the `close` below —
104
+ // the same leak that block exists to prevent, one function earlier. A failing `seed` stranded
105
+ // its clone and left the clock, the seal and the allow-list this boot installed to every later
106
+ // FILE in the run; `bun test` is one process.
107
+ try {
108
+ await db?.drop();
109
+ } catch {
110
+ // The boot's own failure is what the caller must see, and there is no handle to report a
111
+ // second one through — the same "first failure wins" rule `close` runs by.
112
+ }
113
+ restoreProcessState();
114
+ throw error;
115
+ }
52
116
 
53
117
  const handle: AppHandle = {
54
118
  db,
@@ -61,12 +125,24 @@ async function boot(
61
125
 
62
126
  return {
63
127
  handle,
128
+ // Every step runs even when an earlier one rejects, and the FIRST failure is what the caller
129
+ // sees — the same rule `fixtures.ts` disposes by. An `app.close()` that threw used to strand
130
+ // the clone: one `ultimate_test_template_wN` leaked per failing run, and the seal and the
131
+ // clock were never put back either.
64
132
  close: async () => {
65
- await app.close?.();
66
- await db.drop();
67
- resetNetwork();
68
- unsealNetwork();
69
- restoreDeterminism();
133
+ let failure: { readonly error: unknown } | undefined;
134
+ try {
135
+ await app.close?.();
136
+ } catch (error) {
137
+ failure = { error };
138
+ }
139
+ try {
140
+ await db.drop();
141
+ } catch (error) {
142
+ failure ??= { error };
143
+ }
144
+ restoreProcessState();
145
+ if (failure !== undefined) throw failure.error;
70
146
  },
71
147
  };
72
148
  }
@@ -81,9 +157,9 @@ export function describeApp(
81
157
  body: (app: () => AppHandle) => void,
82
158
  ): void {
83
159
  describe(name, () => {
84
- let booted: { handle: AppHandle; close: () => Promise<void> } | undefined;
160
+ let booted: BootedHarness | undefined;
85
161
  beforeAll(async () => {
86
- booted = await boot(options);
162
+ booted = await bootApp(options);
87
163
  });
88
164
  afterAll(async () => {
89
165
  await booted?.close();
@@ -104,7 +180,7 @@ export function testApp(
104
180
  type: TestType = 'unit',
105
181
  ): void {
106
182
  test(testName(type, name), async () => {
107
- const booted = await boot(options);
183
+ const booted = await bootApp(options);
108
184
  try {
109
185
  await body(booted.handle);
110
186
  } finally {
package/src/index.ts CHANGED
@@ -23,16 +23,21 @@ export {
23
23
  // `test` is OURS (fixture-injecting); everything else passes through. Re-exported so an app
24
24
  // test has one import line, and so `expect` carries this package's matchers already installed.
25
25
  export { afterAll, afterEach, beforeAll, beforeEach, describe, expect } from 'bun:test';
26
- export type { DeterminismOptions } from './determinism';
26
+ export type { DeterminismOptions, DeterminismSnapshot } from './determinism';
27
+ // `captureDeterminism` + `restoreCapturedDeterminism` are the pair a NESTED install needs;
28
+ // `restoreDeterminism` uninstalls outright and hands the real clock and the real `Math.random`
29
+ // back to every later file in the process, which is only ever what the process itself wants.
27
30
  export {
28
31
  advanceClock,
29
32
  assertDeterministic,
33
+ captureDeterminism,
30
34
  DEFAULT_NOW,
31
35
  DEFAULT_SEED,
32
36
  frozenClock,
33
37
  frozenNow,
34
38
  installDeterminism,
35
39
  isDeterminismInstalled,
40
+ restoreCapturedDeterminism,
36
41
  restoreDeterminism,
37
42
  seededRandom,
38
43
  seededUuid,
@@ -44,6 +49,7 @@ export {
44
49
  NetworkOfflineError,
45
50
  NetworkSealedError,
46
51
  NondeterministicError,
52
+ RegistryLeakError,
47
53
  TESTING_ERROR_CODES,
48
54
  TESTING_ERROR_TITLES,
49
55
  TestDatabaseUnavailableError,
@@ -83,6 +89,8 @@ export type { MailRef, TestMail } from './fixture-mail';
83
89
  export { createTestMail } from './fixture-mail';
84
90
  export type { TestNetwork } from './fixture-network';
85
91
  export { createTestNetwork } from './fixture-network';
92
+ export type { ObservedStatement, StatementShape, TestStatements } from './fixture-statements';
93
+ export { createTestStatements } from './fixture-statements';
86
94
  export { fixtureTest as test } from './fixtures';
87
95
  export {
88
96
  ALL_FIXTURE_NAMES,
@@ -94,6 +102,14 @@ export type { AppHandle, AppOptions, BootedApp } from './harness';
94
102
  export { describeApp, testApp } from './harness';
95
103
  export type { MatcherResult } from './matchers';
96
104
  export { matchersInstalled, recordSteps } from './matchers';
105
+ // `isolateEntityRegistry` is deliberately NOT here — it is `@ultimat3/testing/registry-isolation`.
106
+ // This barrel is what a `packages/core` test imports for `expect` alone, and a static re-export
107
+ // evaluates the module it names: that one would load `@ultimat3/entity` and its process-global
108
+ // registry into every test in the framework. Every other package this harness touches is imported
109
+ // dynamically inside the fixture that needs it; a helper that must stay synchronous cannot do that,
110
+ // so it gets its own entry point instead.
111
+ export type { RegistryLeak, RegistrySample } from './registry-leak-guard';
112
+ export { installRegistryLeakGuard, leakBetween, sampleRegistries } from './registry-leak-guard';
97
113
  export type { MockRoute, NetworkState } from './sealed-network';
98
114
  // `setNetworkState` is deliberately not here: it is the offline gate's one writer, and a test that
99
115
  // called it directly would bypass the `network` fixture's disposal and leave the whole process
@@ -138,6 +154,7 @@ export {
138
154
  contractTest,
139
155
  e2eTest,
140
156
  evalTest,
157
+ hasE2eDriver,
141
158
  jobTest,
142
159
  liveTest,
143
160
  SEPARATOR,
package/src/matchers.ts CHANGED
@@ -105,6 +105,43 @@ const result = (pass: boolean, message: string): MatcherResult => ({
105
105
  message: () => message,
106
106
  });
107
107
 
108
+ const isOpenApiLike = (value: unknown): value is OpenApiLike =>
109
+ isRecord(value) &&
110
+ Array.isArray(value['operations']) &&
111
+ value['operations'].every(
112
+ (entry) => typeof (entry as OpenApiLike['operations'][number] | null)?.operationId === 'string',
113
+ );
114
+
115
+ /**
116
+ * The two breaking changes this matcher can see from the shape `OpenApiLike` declares: an
117
+ * operation that disappeared, and a parameter a surviving operation newly requires — the second
118
+ * breaks every caller already omitting it, and comparing operation ids alone let it through while
119
+ * a suite naming this matcher read as covered.
120
+ *
121
+ * Deliberately NOT a full OpenAPI diff. Response status codes, response schemas and parameter
122
+ * TYPES are not on `OpenApiLike` and are not compared; the message says what did break rather than
123
+ * claiming the contract is otherwise unchanged. `x verify`'s `contract-diff` step is the whole
124
+ * answer, against the committed manifest.
125
+ */
126
+ function breakingChanges(before: OpenApiLike, after: OpenApiLike): readonly string[] {
127
+ const current = new Map(after.operations.map((operation) => [operation.operationId, operation]));
128
+ const broke: string[] = [];
129
+ for (const operation of before.operations) {
130
+ const now = current.get(operation.operationId);
131
+ if (now === undefined) {
132
+ broke.push(`removed operation ${operation.operationId}`);
133
+ continue;
134
+ }
135
+ // Only additions: dropping a requirement widens what the API accepts, which no caller notices.
136
+ const wasRequired = new Set(operation.required ?? []);
137
+ const added = (now.required ?? []).filter((name) => !wasRequired.has(name));
138
+ if (added.length > 0) {
139
+ broke.push(`${operation.operationId} newly requires ${added.join(', ')}`);
140
+ }
141
+ }
142
+ return broke;
143
+ }
144
+
108
145
  expect.extend({
109
146
  toBeUltimateError(received: unknown, code?: string) {
110
147
  const actual = codeOf(received);
@@ -135,13 +172,16 @@ expect.extend({
135
172
  },
136
173
 
137
174
  toMatchOpenApi(received: unknown, committed: OpenApiLike) {
138
- const current = received as OpenApiLike;
139
- const before = committed.operations.map((operation) => operation.operationId).sort();
140
- const after = current.operations.map((operation) => operation.operationId).sort();
141
- const removed = before.filter((id) => !after.includes(id));
175
+ if (!isOpenApiLike(received)) {
176
+ return result(
177
+ false,
178
+ 'expected an OpenAPI document — an object with operations: [{ operationId, required? }]',
179
+ );
180
+ }
181
+ const broke = breakingChanges(committed, received);
142
182
  return result(
143
- removed.length === 0,
144
- `contract removed operation(s): ${removed.join(', ')} — bump the package version or restore them`,
183
+ broke.length === 0,
184
+ `contract broke: ${broke.join('; ')} — bump the package version or restore the old shape`,
145
185
  );
146
186
  },
147
187
 
package/src/preload.ts CHANGED
@@ -8,6 +8,7 @@
8
8
  import { installDeterminism } from './determinism';
9
9
  import { registerFrameworkFixtures } from './framework-fixtures';
10
10
  import './matchers';
11
+ import { installRegistryLeakGuard } from './registry-leak-guard';
11
12
  import { sealNetwork } from './sealed-network';
12
13
 
13
14
  const seed = Number.parseInt(Bun.env['ULTIMATE_TEST_SEED'] ?? '', 10);
@@ -20,6 +21,10 @@ installDeterminism({
20
21
 
21
22
  registerFrameworkFixtures();
22
23
 
24
+ // One `bun test` invocation is one process: a file that leaves a process-global registry dirty
25
+ // fails a later file in another package, for a reason nothing in that file explains.
26
+ installRegistryLeakGuard();
27
+
23
28
  // Opt-out exists for one case: a test that deliberately exercises a real integration in a job the
24
29
  // team runs on purpose. It is an env var, not an API, so it cannot be set from inside a test file.
25
30
  if (Bun.env['ULTIMATE_TEST_ALLOW_NET'] !== '1') sealNetwork();
@@ -0,0 +1,36 @@
1
+ // One test seam: an empty entity registry for the test that needs one, and the process's own
2
+ // registry back afterwards. `entity()` registers at MODULE scope — which is idiomatic, an app
3
+ // declares its domain by importing it — so a test asserting on the WHOLE registry inherits every
4
+ // entity any earlier file in the same `bun test` process imported, and its premise becomes
5
+ // whatever ran before it rather than what it declared.
6
+ //
7
+ // Its OWN entry point (`@ultimat3/testing/registry-isolation`), off the barrel: the value import
8
+ // below is static because the restore has to be handed back synchronously, so re-exporting this
9
+ // from `src/index.ts` would load the entity registry into every test that imports the harness for
10
+ // `expect`. One import line is worth less than a tier-0 test that stays tier-0.
11
+
12
+ import { clearRegistry, registerEntity, registeredEntities } from '@ultimat3/entity';
13
+
14
+ /**
15
+ * Empties the entity registry and returns the function that puts it back, entry for entry.
16
+ * For a test whose subject is "no entities are declared" — `x db gen` with nothing to generate,
17
+ * a manifest with no rows — which is otherwise true only until a neighbouring file imports one:
18
+ *
19
+ * const restoreEntities = isolateEntityRegistry();
20
+ * try {
21
+ * // …the assertion that needs an empty registry
22
+ * } finally {
23
+ * restoreEntities();
24
+ * }
25
+ *
26
+ * Restoring rather than leaving it empty is the half that matters: `entity()` runs once per
27
+ * module, so a registry cleared and not refilled cannot be repopulated by a later import.
28
+ */
29
+ export function isolateEntityRegistry(): () => void {
30
+ const captured = registeredEntities();
31
+ clearRegistry();
32
+ return () => {
33
+ clearRegistry();
34
+ for (const entry of captured) registerEntity(entry);
35
+ };
36
+ }
@@ -0,0 +1,138 @@
1
+ // Cross-file state pollution, caught at the boundary it crosses. `bun test` runs every file of one
2
+ // invocation in ONE process — only `--isolate` gives each file its own module registry — so a file
3
+ // that leaves a process-global registry dirty changes what every file after it sees, and the
4
+ // failure lands on an innocent suite in another package. This names the file that leaked.
5
+
6
+ import { afterAll } from 'bun:test';
7
+ import { knownTags, registeredTiers } from '@ultimat3/cache';
8
+ import { RegistryLeakError } from './errors';
9
+
10
+ /**
11
+ * What is guarded, and why only these two. Both are BOOT installs — `declareTags` takes the
12
+ * manifest's entity names, `registerTier` takes `app.config.ts`'s tiers — so "empty again when the
13
+ * file ends" is the honest invariant for a test. The entity, job, route and permission registries
14
+ * are not here: `entity()` and `job()` register at module scope, which is how an app declares
15
+ * itself, so a file that leaves them filled is idiomatic rather than leaky. A test whose subject is
16
+ * an EMPTY one of those establishes it itself — `isolateEntityRegistry()`.
17
+ */
18
+ export interface RegistrySample {
19
+ readonly tags: readonly string[];
20
+ readonly tiers: readonly string[];
21
+ }
22
+
23
+ export interface RegistryLeak {
24
+ /** Repo-relative when it can be, so the message names the file an editor opens. */
25
+ readonly file: string;
26
+ readonly tags: readonly string[];
27
+ readonly tiers: readonly string[];
28
+ }
29
+
30
+ export function sampleRegistries(): RegistrySample {
31
+ return { tags: [...knownTags()], tiers: registeredTiers().map((tier) => tier.name) };
32
+ }
33
+
34
+ /**
35
+ * Additions only. A file that DROPS a tier a previous file registered is a different bug and not
36
+ * this one's to report — reporting both here would make the message ambiguous about which file to
37
+ * open.
38
+ */
39
+ export function leakBetween(
40
+ file: string,
41
+ before: RegistrySample,
42
+ after: RegistrySample,
43
+ ): RegistryLeak | undefined {
44
+ const tags = after.tags.filter((name) => !before.tags.includes(name));
45
+ const tiers = after.tiers.filter((name) => !before.tiers.includes(name));
46
+ if (tags.length === 0 && tiers.length === 0) return undefined;
47
+ return { file, tags, tiers };
48
+ }
49
+
50
+ /** `Bun.file(path).text()` answers an absolute path; the message wants the one a reader types. */
51
+ const repoRelative = (path: string): string => {
52
+ const root = `${process.cwd()}/`;
53
+ return path.startsWith(root) ? path.slice(root.length) : path;
54
+ };
55
+
56
+ /**
57
+ * The one point at which a file's baseline is honest, and the reason it is not a hook. Measured on
58
+ * Bun 1.3.14 with a `Bun.plugin` load handler and hooks at every scope, the order is:
59
+ *
60
+ * onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload `beforeEach` → test
61
+ *
62
+ * A preload's `beforeEach` therefore runs AFTER the file's own `beforeAll`, so a `declareTags()`
63
+ * there landed in the baseline and the file read clean — a false green in the guard whose entire
64
+ * job is catching false greens. Appending the sample to the file's own source is what puts it
65
+ * after module evaluation (its environment) and before the first hook the file registers (its own
66
+ * doing). `bun:test` hooks carry no file identity; this loader does.
67
+ */
68
+ const BASELINE_HOOK = '__ultimateRegistryLeakBaseline';
69
+
70
+ /** Appended, never prepended: an `import` is hoisted and would sample before the graph evaluates. */
71
+ const SAMPLE_BASELINE = `\n;globalThis[${JSON.stringify(BASELINE_HOOK)}]?.();\n`;
72
+
73
+ const hookHost = globalThis as typeof globalThis & { [BASELINE_HOOK]?: () => void };
74
+
75
+ let installed = false;
76
+
77
+ /**
78
+ * Called by the test preload, once per process. Idempotent because two preloads reaching it would
79
+ * otherwise register the hooks twice and report every leak twice.
80
+ *
81
+ * Under `--isolate` each file gets its own module registry, so the guard judges that one file and
82
+ * nothing carries across — which is correct, not a hole: with `--isolate` there is no cross-file
83
+ * pollution to find, and a file that leaks is still reported against itself.
84
+ */
85
+ export function installRegistryLeakGuard(): void {
86
+ if (installed) return;
87
+ installed = true;
88
+
89
+ let pending: string | undefined;
90
+ let current: { readonly file: string; readonly before: RegistrySample } | undefined;
91
+ const leaks: RegistryLeak[] = [];
92
+
93
+ const close = (): void => {
94
+ if (current === undefined) return;
95
+ const leak = leakBetween(current.file, current.before, sampleRegistries());
96
+ if (leak !== undefined) leaks.push(leak);
97
+ current = undefined;
98
+ };
99
+
100
+ // Called by the statement appended below, once per test file, from that file's own module scope:
101
+ // everything the file's MODULE graph registered is its environment — importing an app module is
102
+ // how an app declares its tags — and everything after this point is the file's own to undo.
103
+ hookHost[BASELINE_HOOK] = () => {
104
+ if (pending === undefined) return;
105
+ current = { file: pending, before: sampleRegistries() };
106
+ pending = undefined;
107
+ };
108
+
109
+ // The only signal Bun gives a preload for "a new test file starts": its hooks carry no file. A
110
+ // load handler MUST answer with contents — one that answers `undefined` makes Bun load nothing
111
+ // for the file and the run reports zero tests, silently.
112
+ Bun.plugin({
113
+ name: 'ultimate-registry-leak-guard',
114
+ setup(build) {
115
+ // `.test.ts` only, never `.test.tsx`. A load handler that answered `loader: 'tsx'` would
116
+ // compile JSX with Bun's CLASSIC React fallback — the very factory `@ultimat3/render`'s own
117
+ // loader exists to replace — and, because the first matching handler wins and this one is
118
+ // registered from the preload, it would shadow render's transform for that file. Routing
119
+ // `.tsx` through `transformTsx` is not the alternative either: it needs `@ultimat3/render`,
120
+ // whose import installs that global loader into every test process in the repo. Zero
121
+ // `.test.tsx` files exist and the convention is `<file>.test.ts`, so the narrower filter
122
+ // costs nothing today; a `.test.tsx` added later is unguarded rather than mis-compiled.
123
+ build.onLoad({ filter: /\.test\.ts$/ }, async (args) => {
124
+ close();
125
+ pending = repoRelative(args.path);
126
+ return {
127
+ contents: `${await Bun.file(args.path).text()}${SAMPLE_BASELINE}`,
128
+ loader: 'ts',
129
+ };
130
+ });
131
+ },
132
+ });
133
+
134
+ afterAll(() => {
135
+ close();
136
+ if (leaks.length > 0) throw new RegistryLeakError({ leaks });
137
+ });
138
+ }
@@ -130,11 +130,19 @@ export async function acquireWorkerDatabase(
130
130
  try {
131
131
  await admin.exec(lockSql(template));
132
132
  try {
133
- await admin.exec(createTemplateSql(template));
133
+ try {
134
+ await admin.exec(createTemplateSql(template));
135
+ } catch (error) {
136
+ // Tolerated for the CREATE alone: "already exists" means another worker got here first, or
137
+ // a Postgres that outlives one run still holds last run's template.
138
+ if (!alreadyExists(error)) throw error;
139
+ }
140
+ // Outside that tolerance on purpose, and unconditional. A template found rather than created
141
+ // is not a migrated one — skipping here clones the first run's schema forever, on every
142
+ // server that is not a fresh CI container. Migrations are idempotent and the advisory lock
143
+ // serialises them, so a failure here is a real failure and `alreadyExists` (a substring match
144
+ // on the message) would read `relation "x_jobs" already exists` as success.
134
145
  if (config.migrate !== undefined) await config.migrate(urlFor(adminUrl, template));
135
- } catch (error) {
136
- // "already exists" means another worker migrated it first — the lock made that safe.
137
- if (!alreadyExists(error)) throw error;
138
146
  } finally {
139
147
  await admin.exec(unlockSql(template));
140
148
  }
package/src/test-types.ts CHANGED
@@ -81,14 +81,25 @@ export type E2eBody = (fixtures: E2eFixtures) => Promise<void>;
81
81
  let e2eDriver: ((name: string, body: E2eBody) => void) | undefined;
82
82
 
83
83
  /**
84
- * Register the Playwright-backed driver. The e2e package wires this up; without it e2e tests are
85
- * skipped loudly rather than failing on a missing browser, and `x verify` reports the step as
86
- * skipped rather than green.
84
+ * Register the browser-backed driver. Without one, `e2eTest` skips loudly — the skipped test's own
85
+ * NAME carries the reason and the command that would build what it drives.
86
+ *
87
+ * What the gate then reports is a **pass over an all-skipped suite**, and that is stated here
88
+ * rather than claimed away: this used to say "`x verify` reports the step as skipped rather than
89
+ * green", which nothing implements. The step shells out to `bun test`, `bun test` exits 0 on a
90
+ * skip, and an exit code is the only channel between the two — the driver is registered inside the
91
+ * CHILD process, so the step cannot ask. Closing it means a channel the step can read, which is a
92
+ * design decision and not a docstring. `As of 2026-08` there are zero registered drivers, so every
93
+ * `e2eTest` in the tree is a skip; the framework's own `e2e` suites use plain `bun:test`.
87
94
  */
88
95
  export function useE2eDriver(driver: (name: string, body: E2eBody) => void): void {
89
96
  e2eDriver = driver;
90
97
  }
91
98
 
99
+ /** Whether a driver is registered. Exported so a harness can say which of the two states it is in
100
+ * instead of reading an all-skipped run as a green one. */
101
+ export const hasE2eDriver = (): boolean => e2eDriver !== undefined;
102
+
92
103
  export const e2eTest = (name: string, body: E2eBody): void => {
93
104
  if (e2eDriver === undefined) {
94
105
  test.skip(