@ultimat3/testing 1.2.0 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +57 -0
- package/README.md +88 -6
- package/package.json +11 -7
- package/src/determinism.ts +29 -0
- package/src/errors.ts +71 -1
- package/src/fixture-clock.ts +23 -1
- package/src/fixture-statements.ts +148 -0
- package/src/fixtures.ts +52 -6
- package/src/framework-fixtures.ts +9 -1
- package/src/harness.ts +95 -19
- package/src/index.ts +18 -1
- package/src/matchers.ts +46 -6
- package/src/preload.ts +5 -0
- package/src/registry-isolation.ts +36 -0
- package/src/registry-leak-guard.ts +138 -0
- package/src/template-db.ts +12 -4
- package/src/test-types.ts +14 -3
package/CLAUDE.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# @ultimat3/testing — boundary
|
|
2
|
+
|
|
3
|
+
Tier 5. May import tiers 0–4. Imported by every package's tests and by generated apps.
|
|
4
|
+
|
|
5
|
+
Deps: `core` (tier 0), plus `time`, `jobs`, `mail`, `db` and `entity` — imported **dynamically
|
|
6
|
+
inside the fixture factories only**, so a test that never destructures `mail` never loads the mail
|
|
7
|
+
package and a `packages/core` test never loads the entity registry. `entity` is a dependency for
|
|
8
|
+
exactly one call: `nPlusOne()`, so the strict fixture reports the error `x dev` reports, with the
|
|
9
|
+
`fix:` the schema's own relations spell. A second N+1 code owned here would be a second answer to
|
|
10
|
+
one condition. The one static `entity` import is `registry-isolation.ts`, which is why that module
|
|
11
|
+
is its own entry point and not part of the barrel.
|
|
12
|
+
|
|
13
|
+
| Rule | Detail |
|
|
14
|
+
|---|---|
|
|
15
|
+
| No mocks of the DB | clone a template database; `template-db.ts` is the only DB path |
|
|
16
|
+
| No wall clock | `frozenClock` / `advanceClock`; `Date.now()` is frozen by the preload |
|
|
17
|
+
| Frozen ≠ different | `globalThis.Date` becomes a subclass, so `FrozenDate[Symbol.hasInstance]` brands on the `[[DateValue]]` slot — `Date.prototype.getTime.call(value)` throws or it doesn't. Without it `value instanceof Date` is false for every Date the runtime built itself — a `timestamptz` off a Postgres socket, a `structuredClone`, anything from another realm — and the guards that read it fail under test and nowhere else |
|
|
18
|
+
| Slot, not prototype | the brand is cross-realm on purpose: `instanceof RealDate` misses a `node:vm` or worker Date, and `Object.prototype.toString` is spoofable by `Symbol.toStringTag: 'Date'`. Only the slot is both |
|
|
19
|
+
| No unmocked egress | `sealed-network.ts` patches fetch; a miss is `X_TEST_NETWORK_SEALED` |
|
|
20
|
+
| Self is not egress | a port core's `markListening()` announced passes through — a socket test never unseals |
|
|
21
|
+
| Offline is a state, not a mock | `network.offline()` / `.drop()` fail every request as `X_TEST_NETWORK_OFFLINE`, ahead of the mocks — the app's own offline path runs |
|
|
22
|
+
| One way offline | the `network` fixture. `setNetworkState` is the gate's only writer and is not exported — setting it from a test body skips the fixture's disposal and leaves every later file offline |
|
|
23
|
+
| No retries | a flake is fixed or deleted the day it flakes; there is no `retry: 3` |
|
|
24
|
+
| Test names | the filename picks the step; `testName(type, name)` on the outer `describe` puts that type on every failure line under it. Never on the inner `test` too — the prefix would print twice |
|
|
25
|
+
| Injection | `SqlRunner` and `connect` are parameters, so unit tests need no server |
|
|
26
|
+
| Fixtures | the preload registers the whole framework bag — an app registers only what the framework cannot know (`seed`, `actorFor`) |
|
|
27
|
+
| e2e without a driver | `e2eTest` becomes `test.skip`, and the gate reports the step GREEN over it — `bun test` exits 0 on a skip and the exit code is the only channel between the step and the child that registers the driver. `hasE2eDriver()` is what a harness asks instead of reading an all-skipped run as a pass. Zero drivers are registered `As of 2026-08` |
|
|
28
|
+
| Built vs declared | `clock` `mail` `network` `runJobs` `statements` are built in-process; `page` `budget` `signIn` `deploy` `subscribe` are declared and wait for a driver (`X_TEST_FIXTURE_UNAVAILABLE`) |
|
|
29
|
+
| Strict is opt-in by destructuring | `statements` installs the N+1 detector in throw mode for one test. A fixture nobody names is a fixture nobody built, so there is no `strict: true` and no suite-wide switch — and no way to leave it on for the next file |
|
|
30
|
+
| One threshold, one error | `N_PLUS_ONE_THRESHOLD` and `nPlusOne()` are `@ultimat3/entity`'s. A number or a message written here would make a loop that fails a test a different loop from the one `x dev` warns about |
|
|
31
|
+
| The unit of work is the test | `x dev`'s ledger tallies per `Ctx` and ignores a statement issued outside a request; this counts every statement from build to disposal, because `posts.findById(id)` in a unit test has no request and is exactly the loop worth catching |
|
|
32
|
+
| Throws once per shape | the failing line is the loop's own statement (the seam lets `onStatement` throw for this reason alone). It keeps counting after, so a body that catches the error still reports the whole loop through `shapes()` — the same hole Bullet's `raise` has, named rather than papered over |
|
|
33
|
+
| Measure vs judge | `all()` `count()` `shapes()` count `expectedQueryLoop` statements too; only the verdict honours the suppression, so "this page issues two statements" never depends on who declared what |
|
|
34
|
+
| One seam for drivers | a driver registers over a declaration with `defineFixtures` — merges, last wins. Never a second registration mechanism |
|
|
35
|
+
| A driver arrives whole | `defineFixtures` holds every name `Fixtures` declares to its declared type, so a half-built `page` is a compile error at the registration, not a missing method three awaits later |
|
|
36
|
+
| Registry hygiene | the fixture registry is process-global; a test that clears it snapshots with `fixtureSnapshot()` and hands it back in `afterAll` |
|
|
37
|
+
| Leaks are the file's, not the next file's | `installRegistryLeakGuard()` runs from the preload and fails the run naming the FILE that left cache tags declared or a cache tier registered after its last test (`X_TEST_REGISTRY_LEAK`). `bun test` is one process, so without it the failure lands on an innocent suite in another package. What a file's MODULE graph declares is its environment; what the file installs after that is its own to undo |
|
|
38
|
+
| The baseline is not a hook | measured on Bun 1.3.14 the order is onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload `beforeEach`, so a preload hook cannot sample before the file's own `beforeAll` — a `declareTags()` there read as environment and the run went green. The load handler appends the sample to the file's source instead: after evaluation, before any hook the file registers. It is also the only signal carrying file identity, which `bun:test` hooks do not |
|
|
39
|
+
| Guarded state is boot state | only the two registries whose honest invariant is "clean when the file ends" — `declareTags` and `registerTier` are boot installs. `entity()`, `job()` and `defineRoute()` register at MODULE scope, which is how an app declares itself, so a filled registry there is idiomatic and unguarded |
|
|
40
|
+
| An empty registry is a premise you state | a test whose subject is "nothing is declared" — `x db gen` with nothing to generate — calls `isolateEntityRegistry()` and restores in a `finally`. Inheriting it means the test passes until a neighbouring file imports an entity |
|
|
41
|
+
| That one helper is off the barrel | `@ultimat3/testing/registry-isolation`, its own entry point. It is the only module here that value-imports `@ultimat3/entity` — the restore is handed back synchronously, so it cannot be a dynamic import inside the call — and a static re-export from `src/index.ts` would load the entity registry into every test that imports this package for `expect` |
|
|
42
|
+
| Teardown restores, never uninstalls | `describeApp`/`testApp` capture the seal and the determinism snapshot before booting and put those back — `restoreDeterminism()` in a scope hands the REAL clock and the REAL `fetch` to every later FILE in the process. `captureDeterminism()` / `restoreCapturedDeterminism()` are the pair for any nested install |
|
|
43
|
+
| Teardown is a `finally` | an `app.close()` that rejects still reaches `db.drop()` and still restores the process state; the first failure is what the caller sees. A stranded clone is one `ultimate_test_template_wN` leaked per failing run |
|
|
44
|
+
| A boot that rejects is its own teardown | `acquireWorkerDatabase`, `seed` or `boot` throwing returns no `BootedHarness`, so no caller can ever reach `close()` — `bootApp` drops the clone it acquired and restores the seal, the allow-list and the clock itself, then rethrows the boot's OWN error. A `drop` that also fails is swallowed: there is no handle left to report a second failure through |
|
|
45
|
+
| A found template is not a migrated one | `template-db.ts` tolerates "already exists" for the `CREATE DATABASE` alone. `config.migrate` runs unconditionally and un-swallowed — on any Postgres that outlives one run the template is found, not created, and skipping it clones the first run's schema forever |
|
|
46
|
+
| Fixture teardown | a fixture that installs process-global state (the ambient job or mail driver) implements `Symbol.dispose` / `Symbol.asyncDispose` and restores what was there; `fixtureTest` disposes in reverse build order even when the body throws |
|
|
47
|
+
| Building one by hand | `createRunJobs()` outside `fixtureTest` is not disposed for you — reset the driver in `afterEach`, or the next file in the process inherits your queue |
|
|
48
|
+
| Factory strategy | an association is built with the strategy that asked for it: `build()` never reaches a database, `create()` writes the parent first. Never a third strategy |
|
|
49
|
+
| One write seam | `usePersister` is the only place `create()` writes. A factory that took a repo argument would put the seam at every call site |
|
|
50
|
+
| Factory seeds | derived from the table name unless given, so two entities never draw the same uuid stream. `reset()` cascades into associated parents — a half-reset row is worse than none |
|
|
51
|
+
| Shared examples | `behavesLike` calls `describe`, so it goes at declaration scope; bun rejects a `describe` inside a test body |
|
|
52
|
+
| Which command shards | `bun test` is one process on one database, and that is still what a scaffolded app's `test` script runs. `x verify` DOES shard its parallel test steps, over `ULTIMATE_TEST_WORKER` and one database per worker; `live` and `e2e` stay serial because a replication slot is cluster-scoped and `e2e` has one built `dist/`. Say which command a claim is about |
|
|
53
|
+
|
|
54
|
+
Commands: `bun test`, `bunx tsc --noEmit -p tsconfig.json`.
|
|
55
|
+
|
|
56
|
+
Entry points: `.` (the API), `./preload` (side effects for bunfig) and `./registry-isolation`
|
|
57
|
+
(`isolateEntityRegistry()`, kept off `.` because it loads `@ultimat3/entity`).
|
package/README.md
CHANGED
|
@@ -18,9 +18,11 @@ frozen clock. Never let a test reach the network unmocked — it fails by design
|
|
|
18
18
|
| `test-types.ts` | the six test types and their helpers |
|
|
19
19
|
| `matchers.ts` | `toBeUltimateError` `toDenyPolicy` `toEmitSteps` `toMatchOpenApi` `toBeWithinBudget` `toRejectInput` |
|
|
20
20
|
| `fixtures.ts` | the registry + `test('…', ({ clock }) => …)` injection |
|
|
21
|
-
| `fixture-{clock,mail,jobs,network}.ts` | the
|
|
21
|
+
| `fixture-{clock,mail,jobs,network,statements}.ts` | the five fixtures the framework builds in-process |
|
|
22
22
|
| `fixture-drivers.ts` | the five it declares but a driver must build — `page` `budget` `signIn` `deploy` `subscribe` |
|
|
23
23
|
| `framework-fixtures.ts` | registers both sets; the app registers only what it owns |
|
|
24
|
+
| `registry-leak-guard.ts` | fails the run naming the FILE that left a process-global registry dirty |
|
|
25
|
+
| `registry-isolation.ts` | `isolateEntityRegistry()` — an empty entity registry, and the process's back after. Its own entry point (`@ultimat3/testing/registry-isolation`), never the barrel: it value-imports `@ultimat3/entity`, and the barrel is what a tier-0 test imports for `expect` |
|
|
24
26
|
| `preload.ts` | the bunfig preload that installs all of the above |
|
|
25
27
|
|
|
26
28
|
## Install
|
|
@@ -53,6 +55,7 @@ test('the three-day sleep releases the worker', async ({ clock, runJobs }) => {
|
|
|
53
55
|
| `mail` | `outbox()` · `lastTo(address)` · `failOnce(mail)` over an in-memory transport | the preload |
|
|
54
56
|
| `network` | `offline()` · `drop()` · `online()` · `state()` over the sealed network | the preload |
|
|
55
57
|
| `runJobs` | a worker: call it to enqueue+drain, then `drain()` `due()` `inFlight()` `depth()` | the preload |
|
|
58
|
+
| `statements` | every statement the test issued: `all()` `count(fingerprint?)` `shapes()` — and an N+1 throws | the preload |
|
|
56
59
|
| `page` | the browser: `goto` `gotoStreamed` `getByRole` `evaluate` `waitForServiceWorker` | a browser driver |
|
|
57
60
|
| `budget` | `jsBytes(route)` measured off the built output | a browser driver |
|
|
58
61
|
| `signIn` | put the browser session in a member's shoes | a browser driver |
|
|
@@ -88,6 +91,33 @@ that *is* registered — never `undefined is not an object` from inside the body
|
|
|
88
91
|
registered but has no driver fails with `X_TEST_FIXTURE_UNAVAILABLE` instead; the two are different
|
|
89
92
|
instructions, so they are different codes.
|
|
90
93
|
|
|
94
|
+
## An N+1 fails the test it happened in
|
|
95
|
+
|
|
96
|
+
```ts
|
|
97
|
+
test('the feed reads its authors once', async ({ statements }) => {
|
|
98
|
+
await renderFeed(); // a per-row findById throws here:
|
|
99
|
+
// X_N_PLUS_ONE_QUERY: members.findById ran 5 times in one request — one read per row
|
|
100
|
+
// fix: db.posts.preload('author') # one statement for the whole page
|
|
101
|
+
expect(statements.count('posts.findMany')).toBe(1);
|
|
102
|
+
expect(statements.shapes()[0]?.count).toBe(1);
|
|
103
|
+
});
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Opting in is naming it. `statements` installs `@ultimat3/db`'s statement observer for the length of
|
|
107
|
+
one test and hands the seam back afterwards, so there is no `strict: true` to remember and no
|
|
108
|
+
suite-wide switch to forget.
|
|
109
|
+
|
|
110
|
+
| | |
|
|
111
|
+
|---|---|
|
|
112
|
+
| **the unit of work is the test** | `x dev`'s ledger counts per request and skips a statement issued outside one; a unit test calling `posts.findById(id)` has no request anywhere, and that is the loop it was written to catch |
|
|
113
|
+
| **one threshold** | `N_PLUS_ONE_THRESHOLD` from `@ultimat3/entity`, the number `x dev` warns at. A loop that fails a test and a loop that warns in dev are the same loop |
|
|
114
|
+
| **one error** | `nPlusOne()`'s, so the `fix:` is the `preload()` the schema's own relations spell — never a line this package composes |
|
|
115
|
+
| **it throws where it happened** | the loop's fifth statement rejects, so the failing line is the loop's own. Once per shape: a body that catches it gets one failure, not one per statement after it |
|
|
116
|
+
| **measurement ≠ verdict** | `all()` `count()` `shapes()` count every statement, `expectedQueryLoop` ones included; only the verdict honours the suppression |
|
|
117
|
+
|
|
118
|
+
`expectedQueryLoop(reason, fn)` from `@ultimat3/db` stays the one way to declare a loop deliberate —
|
|
119
|
+
there is no flag on the fixture and no code to silence.
|
|
120
|
+
|
|
91
121
|
## The six test types
|
|
92
122
|
|
|
93
123
|
| Helper | Asserts | `x verify` step |
|
|
@@ -96,7 +126,7 @@ instructions, so they are different codes.
|
|
|
96
126
|
| `contractTest` | OpenAPI diff vs the committed spec, MCP exposure | `contract` |
|
|
97
127
|
| `liveTest` | exactly what each subscriber receives | `live` |
|
|
98
128
|
| `jobTest` | step sequence, retries, idempotency | `job` |
|
|
99
|
-
| `e2eTest` |
|
|
129
|
+
| `e2eTest` | a browser driver incl. offline mode + SW update; with none registered it SKIPS, and the gate's `e2e` step passes over the skip — ask `hasE2eDriver()` rather than reading that as a pass | `e2e` |
|
|
100
130
|
| `evalTest` | LLM output scoring against a threshold | `eval` |
|
|
101
131
|
|
|
102
132
|
Each helper prefixes the test name with its type (`job · onboards an org`), which is what
|
|
@@ -161,11 +191,12 @@ The first worker creates the template under a Postgres advisory lock and migrate
|
|
|
161
191
|
worker then clones it copy-on-write. With no Postgres configured it falls back to PGlite, so
|
|
162
192
|
`bun test` works on a laptop with nothing installed.
|
|
163
193
|
|
|
164
|
-
**
|
|
194
|
+
**The gate shards; a bare `bun test` does not.** `As of 2026-08`:
|
|
165
195
|
|
|
166
196
|
| Command | Processes | Worker ids | Databases |
|
|
167
197
|
|---|---|---|---|
|
|
168
|
-
| `bun test` (what a scaffolded app's `test` script
|
|
198
|
+
| `bun test` (what a scaffolded app's `test` script still runs) | 1 | `0` | one |
|
|
199
|
+
| `x verify` (`unit`, `contract`, `job`, `eval`; `live` and `e2e` stay serial) | `clamp(round(cpus * 1.5), 2, 8)` | `0..N-1`, from `ULTIMATE_TEST_WORKER` | N |
|
|
169
200
|
| `bun test --parallel[=N]` | N (default: CPU count) | `1..N`, from Bun's own `BUN_TEST_WORKER_ID` | N |
|
|
170
201
|
| `x test --workers N` | N | `0..N-1`, from `ULTIMATE_TEST_WORKER` | N |
|
|
171
202
|
|
|
@@ -173,9 +204,26 @@ worker then clones it copy-on-write. With no Postgres configured it falls back t
|
|
|
173
204
|
its own `--parallel` worker — measured on Bun 1.3.14, `--parallel` populates `BUN_TEST_WORKER_ID`
|
|
174
205
|
and `JEST_WORKER_ID` itself, so that precedence is load-bearing rather than defensive.
|
|
175
206
|
|
|
207
|
+
## The harness puts back what it found
|
|
208
|
+
|
|
209
|
+
`bun test` is one process, so `describeApp`/`testApp` teardown is a **restore**, never an uninstall.
|
|
210
|
+
`As of 2026-08`:
|
|
211
|
+
|
|
212
|
+
| State | Owned by | What teardown does |
|
|
213
|
+
|---|---|---|
|
|
214
|
+
| the seal on `fetch` | the preload | unseals only if this boot was the one that sealed |
|
|
215
|
+
| the frozen instant, `Math.random`, `globalThis.Date` | the preload (`ULTIMATE_TEST_NOW` / `ULTIMATE_TEST_SEED`) | `restoreCapturedDeterminism(captureDeterminism())` around the boot |
|
|
216
|
+
| mocks, allow-listed hosts, the seen list | the boot | `resetNetwork()` |
|
|
217
|
+
| the cloned worker database | the boot | `db.drop()`, in a `finally` — a rejecting `app.close()` reaches it |
|
|
218
|
+
|
|
219
|
+
`installDeterminism()` runs during a boot only when the boot has something of its own to say
|
|
220
|
+
(`seedValue`/`now`) or nothing installed it yet, so a run configured with `ULTIMATE_TEST_NOW` is not
|
|
221
|
+
reset by the first `describeApp`. `restoreDeterminism()` is the process's own call, not a scope's:
|
|
222
|
+
it hands the real clock and the real `Math.random` back to every later **file** in the run.
|
|
223
|
+
|
|
176
224
|
## Sealed network
|
|
177
225
|
|
|
178
|
-
```
|
|
226
|
+
```text
|
|
179
227
|
X_TEST_NETWORK_SEALED
|
|
180
228
|
cause: POST https://api.stripe.com/v1/charges was not mocked (allowed hosts: none)
|
|
181
229
|
fix: mockFetch('https://api.stripe.com/v1/charges', () => new Response('{}')) — or allowHost('api.stripe.com') if it must be real
|
|
@@ -189,4 +237,38 @@ integration — never for a socket test.
|
|
|
189
237
|
## Errors
|
|
190
238
|
|
|
191
239
|
`X_TEST_NETWORK_SEALED` `X_TEST_DB_UNAVAILABLE` `X_TEST_NONDETERMINISTIC` `X_TEST_FIXTURE_UNKNOWN`
|
|
192
|
-
`X_TEST_FACTORY_TRAIT_UNKNOWN` `X_TEST_FACTORY_NOT_PERSISTED`
|
|
240
|
+
`X_TEST_FACTORY_TRAIT_UNKNOWN` `X_TEST_FACTORY_NOT_PERSISTED` `X_TEST_REGISTRY_LEAK`
|
|
241
|
+
|
|
242
|
+
## One process, one registry
|
|
243
|
+
|
|
244
|
+
`bun test` runs every file of one invocation in the same process — only `x verify`'s shards pass
|
|
245
|
+
`--isolate`. A file that leaves a process-global registry dirty therefore changes what every later
|
|
246
|
+
file sees, and the failure lands on an innocent suite in another package: `bun test packages/query
|
|
247
|
+
packages/cli` failed five tests in `query`, all of them installed by `cli`, while either package
|
|
248
|
+
alone was green.
|
|
249
|
+
|
|
250
|
+
The preload installs the guard. It samples the cache tag set and the cache tier registry once per
|
|
251
|
+
file, at the end of that file's module evaluation — so an app's own boot declarations are its
|
|
252
|
+
environment — and reports what the file added after that and did not put back:
|
|
253
|
+
|
|
254
|
+
```text
|
|
255
|
+
X_TEST_REGISTRY_LEAK: a test file left a process-global registry dirty —
|
|
256
|
+
"packages/cli/src/cmd-dev.test.ts" left cache tags declared ["devfixture"] after its last test
|
|
257
|
+
fix: in "packages/cli/src/cmd-dev.test.ts" add: import { isolateDeclaredTags } from
|
|
258
|
+
'@ultimat3/cache'; const restoreTags = isolateDeclaredTags(); afterAll(restoreTags);
|
|
259
|
+
— then re-run: bun test "packages/cli/src/cmd-dev.test.ts"
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
The baseline is not a `beforeEach`: a file's own `beforeAll` runs **before** a preload's
|
|
263
|
+
`beforeEach` (onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload
|
|
264
|
+
`beforeEach`, measured on Bun 1.3.14), so a `declareTags()` in `beforeAll` would have been sampled
|
|
265
|
+
as the file's environment and the run would have gone green. The guard appends the sample to the
|
|
266
|
+
file's own source in its load handler instead — the one place a file's identity and its evaluation
|
|
267
|
+
boundary are both known.
|
|
268
|
+
|
|
269
|
+
The fix is `isolateDeclaredTags()` or `isolateTiers()` (both `@ultimat3/cache`), never a loosened
|
|
270
|
+
assertion in the file that paid for it — and never a **reset**. A reset drops what a neighbour
|
|
271
|
+
registered, and this guard reports additions only, so the damage lands on an innocent file with
|
|
272
|
+
nothing pointing back. A leak fails a one-file run exactly as it fails the suite —
|
|
273
|
+
each file is judged against its own baseline — which is what makes the `bun test <file>` in the fix
|
|
274
|
+
line reproduce it.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/testing",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "3.0.0",
|
|
4
4
|
"description": "Test harness: cloned template DBs per worker, frozen clock, sealed network, 6 test types",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -15,11 +15,13 @@
|
|
|
15
15
|
},
|
|
16
16
|
"exports": {
|
|
17
17
|
".": "./src/index.ts",
|
|
18
|
-
"./preload": "./src/preload.ts"
|
|
18
|
+
"./preload": "./src/preload.ts",
|
|
19
|
+
"./registry-isolation": "./src/registry-isolation.ts"
|
|
19
20
|
},
|
|
20
21
|
"files": [
|
|
21
22
|
"src",
|
|
22
23
|
"!src/**/*.test.ts",
|
|
24
|
+
"CLAUDE.md",
|
|
23
25
|
"README.md",
|
|
24
26
|
"LICENSE"
|
|
25
27
|
],
|
|
@@ -31,10 +33,12 @@
|
|
|
31
33
|
"test": "bun test"
|
|
32
34
|
},
|
|
33
35
|
"dependencies": {
|
|
34
|
-
"@ultimat3/
|
|
35
|
-
"@ultimat3/
|
|
36
|
-
"@ultimat3/
|
|
37
|
-
"@ultimat3/
|
|
38
|
-
"@ultimat3/
|
|
36
|
+
"@ultimat3/cache": "3.0.0",
|
|
37
|
+
"@ultimat3/core": "3.0.0",
|
|
38
|
+
"@ultimat3/db": "3.0.0",
|
|
39
|
+
"@ultimat3/entity": "3.0.0",
|
|
40
|
+
"@ultimat3/jobs": "3.0.0",
|
|
41
|
+
"@ultimat3/mail": "3.0.0",
|
|
42
|
+
"@ultimat3/time": "3.0.0"
|
|
39
43
|
}
|
|
40
44
|
}
|
package/src/determinism.ts
CHANGED
|
@@ -94,6 +94,35 @@ export function restoreDeterminism(): void {
|
|
|
94
94
|
installed = false;
|
|
95
95
|
}
|
|
96
96
|
|
|
97
|
+
/** Everything `installDeterminism` overwrites, so a nested install can put back what it found. */
|
|
98
|
+
export interface DeterminismSnapshot {
|
|
99
|
+
readonly installed: boolean;
|
|
100
|
+
readonly frozenAt: number;
|
|
101
|
+
readonly random: () => number;
|
|
102
|
+
readonly date: DateConstructor;
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
/**
|
|
106
|
+
* Capture before a nested `installDeterminism`, restore after — the shape `fixture-network.ts`
|
|
107
|
+
* uses for the seal. The preload installs determinism once for the whole process, so an inner
|
|
108
|
+
* scope that called `restoreDeterminism()` would hand the REAL clock and the REAL `Math.random`
|
|
109
|
+
* to every later test file in it. `random` is captured by identity because a generator's state is
|
|
110
|
+
* its closure: re-seeding produces an equal sequence, not the same position in this one.
|
|
111
|
+
*/
|
|
112
|
+
export const captureDeterminism = (): DeterminismSnapshot => ({
|
|
113
|
+
installed,
|
|
114
|
+
frozenAt,
|
|
115
|
+
random: Math.random,
|
|
116
|
+
date: globalThis.Date,
|
|
117
|
+
});
|
|
118
|
+
|
|
119
|
+
export function restoreCapturedDeterminism(snapshot: DeterminismSnapshot): void {
|
|
120
|
+
frozenAt = snapshot.frozenAt;
|
|
121
|
+
Math.random = snapshot.random;
|
|
122
|
+
globalThis.Date = snapshot.date;
|
|
123
|
+
installed = snapshot.installed;
|
|
124
|
+
}
|
|
125
|
+
|
|
97
126
|
export const isDeterminismInstalled = (): boolean => installed;
|
|
98
127
|
|
|
99
128
|
/** Move the frozen clock forward. The only legal way for time to pass inside a test. */
|
package/src/errors.ts
CHANGED
|
@@ -1,6 +1,15 @@
|
|
|
1
1
|
// The X_* codes owned by @ultimat3/testing. A test failure has to be as actionable as a runtime
|
|
2
2
|
// failure — the fix line here is the mock to add, the service to start, or the seed to freeze.
|
|
3
|
-
import {
|
|
3
|
+
import {
|
|
4
|
+
registerErrorCodes,
|
|
5
|
+
renderCauseValue,
|
|
6
|
+
renderFixLiteral,
|
|
7
|
+
UltimateError,
|
|
8
|
+
} from '@ultimat3/core';
|
|
9
|
+
// Type-only, so the cycle with the guard that throws it is erased at build: the code is declared
|
|
10
|
+
// here because this file is the package's code registry, and the shape it reports lives with the
|
|
11
|
+
// sampler that produces it.
|
|
12
|
+
import type { RegistryLeak } from './registry-leak-guard';
|
|
4
13
|
|
|
5
14
|
export const TESTING_ERROR_CODES = [
|
|
6
15
|
'X_TEST_NETWORK_SEALED',
|
|
@@ -15,6 +24,7 @@ export const TESTING_ERROR_CODES = [
|
|
|
15
24
|
'X_TEST_NETWORK_RACE',
|
|
16
25
|
'X_TEST_FACTORY_TRAIT_UNKNOWN',
|
|
17
26
|
'X_TEST_FACTORY_NOT_PERSISTED',
|
|
27
|
+
'X_TEST_REGISTRY_LEAK',
|
|
18
28
|
] as const;
|
|
19
29
|
|
|
20
30
|
export type TestingErrorCode = (typeof TESTING_ERROR_CODES)[number];
|
|
@@ -32,6 +42,7 @@ export const TESTING_ERROR_TITLES: Readonly<Record<TestingErrorCode, string>> =
|
|
|
32
42
|
X_TEST_NETWORK_RACE: 'a request raced unsealNetwork() and lost the patched fetch',
|
|
33
43
|
X_TEST_FACTORY_TRAIT_UNKNOWN: 'a factory was asked for a trait it does not declare',
|
|
34
44
|
X_TEST_FACTORY_NOT_PERSISTED: 'a factory create() had nowhere to write the row',
|
|
45
|
+
X_TEST_REGISTRY_LEAK: 'a test file left a process-global registry dirty',
|
|
35
46
|
};
|
|
36
47
|
|
|
37
48
|
// Titles must be registered for `format()` to render the contract's first line. Every code above is
|
|
@@ -239,3 +250,62 @@ export class NetworkRaceError extends UltimateError {
|
|
|
239
250
|
});
|
|
240
251
|
}
|
|
241
252
|
}
|
|
253
|
+
|
|
254
|
+
/**
|
|
255
|
+
* Every value in this message is uncontrolled: the path arrives from `Bun.plugin`'s `onLoad`, and
|
|
256
|
+
* the names are whatever the app under test passed to `declareTags()` / `registerTier()`. So the
|
|
257
|
+
* sentence renders them (bounded, never throwing while describing a leak) and the command quotes
|
|
258
|
+
* them (a fix has to parse after a path with a space or a quote in it lands in the middle of it).
|
|
259
|
+
*/
|
|
260
|
+
const describeLeak = (leak: RegistryLeak): string => {
|
|
261
|
+
const left = [
|
|
262
|
+
...(leak.tags.length > 0 ? [`cache tags declared ${renderCauseValue(leak.tags)}`] : []),
|
|
263
|
+
...(leak.tiers.length > 0 ? [`cache tiers registered ${renderCauseValue(leak.tiers)}`] : []),
|
|
264
|
+
];
|
|
265
|
+
return `${renderCauseValue(leak.file)} left ${left.join(' and ')} after its last test`;
|
|
266
|
+
};
|
|
267
|
+
|
|
268
|
+
/** The placeholder is the cause's own sentence: it already names every file, in order. */
|
|
269
|
+
const FILE_PLACEHOLDER = '<the file the cause names>';
|
|
270
|
+
|
|
271
|
+
/**
|
|
272
|
+
* The edit, spelled as the lines to paste — the imports included, since neither call is global.
|
|
273
|
+
*
|
|
274
|
+
* Both repairs isolate; neither resets. `afterAll(resetTiers)` was the first spelling of the tier
|
|
275
|
+
* half and it is the bug wearing the fix's clothes: it drops the tiers, the revalidator and both
|
|
276
|
+
* logs a NEIGHBOUR registered, and the guard reports additions only, so the damage lands on an
|
|
277
|
+
* innocent file with nothing pointing back here. An error whose instruction causes the next defect
|
|
278
|
+
* is worse than no instruction.
|
|
279
|
+
*/
|
|
280
|
+
const repairFor = (leak: RegistryLeak): string => {
|
|
281
|
+
const imports: string[] = [];
|
|
282
|
+
const calls: string[] = [];
|
|
283
|
+
if (leak.tags.length > 0) {
|
|
284
|
+
imports.push('isolateDeclaredTags');
|
|
285
|
+
calls.push('const restoreTags = isolateDeclaredTags(); afterAll(restoreTags);');
|
|
286
|
+
}
|
|
287
|
+
if (leak.tiers.length > 0) {
|
|
288
|
+
imports.push('isolateTiers');
|
|
289
|
+
calls.push('const restoreTiers = isolateTiers(); afterAll(restoreTiers);');
|
|
290
|
+
}
|
|
291
|
+
const file = renderFixLiteral(leak.file, FILE_PLACEHOLDER);
|
|
292
|
+
return `in ${file} add: import { ${imports.join(', ')} } from '@ultimat3/cache'; ${calls.join(' ')}`;
|
|
293
|
+
};
|
|
294
|
+
|
|
295
|
+
/**
|
|
296
|
+
* One error for every leaker in the run, not one per file: the run has already finished by the
|
|
297
|
+
* time the last file can be judged, and two throws would report the second as an unhandled one.
|
|
298
|
+
* The trailing command reproduces the failure on its own — the guard judges each file against its
|
|
299
|
+
* own baseline, so one leaking file fails a one-file run exactly as it failed the whole suite.
|
|
300
|
+
*/
|
|
301
|
+
export class RegistryLeakError extends UltimateError {
|
|
302
|
+
constructor(input: { readonly leaks: readonly RegistryLeak[] }) {
|
|
303
|
+
const files = input.leaks.map((leak) => renderFixLiteral(leak.file, FILE_PLACEHOLDER));
|
|
304
|
+
super({
|
|
305
|
+
code: 'X_TEST_REGISTRY_LEAK',
|
|
306
|
+
cause: input.leaks.map(describeLeak).join('; '),
|
|
307
|
+
fix: `${input.leaks.map(repairFor).join('; ')} — then re-run: bun test ${files.join(' ')}`,
|
|
308
|
+
docs: docsFor('X_TEST_REGISTRY_LEAK'),
|
|
309
|
+
});
|
|
310
|
+
}
|
|
311
|
+
}
|
package/src/fixture-clock.ts
CHANGED
|
@@ -3,7 +3,13 @@
|
|
|
3
3
|
// `clock.advance('3d')` is synchronous — a job test asserts on what is due on the very next
|
|
4
4
|
// line — so the duration parser is resolved while the fixture is built, not when it is used.
|
|
5
5
|
|
|
6
|
-
import {
|
|
6
|
+
import {
|
|
7
|
+
advanceClock,
|
|
8
|
+
captureDeterminism,
|
|
9
|
+
frozenNow,
|
|
10
|
+
restoreCapturedDeterminism,
|
|
11
|
+
setFrozenClock,
|
|
12
|
+
} from './determinism';
|
|
7
13
|
|
|
8
14
|
/** `'3d'` | `'30s'` | `1500`. Same vocabulary as a job's `timeout` and a step's `sleep`. */
|
|
9
15
|
export type TestDuration = string | number;
|
|
@@ -13,10 +19,23 @@ export interface TestClock {
|
|
|
13
19
|
now(): Date;
|
|
14
20
|
advance(duration: TestDuration): Date;
|
|
15
21
|
set(instant: string | number): Date;
|
|
22
|
+
/** Puts the instant this fixture was built at back. `fixtureTest` calls it; see below. */
|
|
23
|
+
[Symbol.asyncDispose](): Promise<void>;
|
|
16
24
|
}
|
|
17
25
|
|
|
26
|
+
/**
|
|
27
|
+
* The frozen instant is module-global and `advance`/`set` move it, so this fixture installs
|
|
28
|
+
* process state exactly the way the mail and job fixtures do — and until it disposed, a test that
|
|
29
|
+
* advanced three days handed the advanced clock to every later test FILE in the run (`bun test` is
|
|
30
|
+
* one process), where the failure lands on an innocent suite.
|
|
31
|
+
*
|
|
32
|
+
* `captureDeterminism`/`restoreCapturedDeterminism` rather than `restoreDeterminism()`: the preload
|
|
33
|
+
* installed determinism once for the whole process, and uninstalling it here would hand the REAL
|
|
34
|
+
* `Date` and the REAL `Math.random` to everything after. Restore only what was found.
|
|
35
|
+
*/
|
|
18
36
|
export async function createTestClock(): Promise<TestClock> {
|
|
19
37
|
const { toMs } = await import('@ultimat3/time');
|
|
38
|
+
const captured = captureDeterminism();
|
|
20
39
|
return {
|
|
21
40
|
now: frozenNow,
|
|
22
41
|
advance: (duration) => advanceClock(toMs(duration)),
|
|
@@ -24,5 +43,8 @@ export async function createTestClock(): Promise<TestClock> {
|
|
|
24
43
|
setFrozenClock(instant);
|
|
25
44
|
return frozenNow();
|
|
26
45
|
},
|
|
46
|
+
[Symbol.asyncDispose]: async () => {
|
|
47
|
+
restoreCapturedDeterminism(captured);
|
|
48
|
+
},
|
|
27
49
|
};
|
|
28
50
|
}
|
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
// The `statements` fixture: every statement the test issues, counted by shape — and a shape that
|
|
2
|
+
// repeats past the threshold throws where it happened, so an N+1 fails the test that caused it
|
|
3
|
+
// instead of warning in a dev server nobody is watching.
|
|
4
|
+
//
|
|
5
|
+
// Strict by construction: there is no `strict: true` to remember, because a fixture nobody
|
|
6
|
+
// destructured is a fixture nobody built. Opting in is naming `statements` in the test body.
|
|
7
|
+
|
|
8
|
+
import type { StatementAttribution, StatementEvent, StatementObserver } from '@ultimat3/db';
|
|
9
|
+
|
|
10
|
+
/** One statement, as this fixture kept it. Bound values are deliberately not retained. */
|
|
11
|
+
export interface ObservedStatement {
|
|
12
|
+
/** What it is counted under: `members.findById` when attributed, else its own collapsed text. */
|
|
13
|
+
readonly fingerprint: string;
|
|
14
|
+
readonly kind: 'read' | 'write';
|
|
15
|
+
/** The statement as sent, parameters still `$1..$n`. */
|
|
16
|
+
readonly text: string;
|
|
17
|
+
/** The entity and operation that compiled it; absent for hand-written SQL and queue traffic. */
|
|
18
|
+
readonly attribution?: StatementAttribution | undefined;
|
|
19
|
+
/** The `expectedQueryLoop()` reason in force when it was sent, absent outside every such scope. */
|
|
20
|
+
readonly expected?: string | undefined;
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
/** One shape and how often the test issued it. */
|
|
24
|
+
export interface StatementShape {
|
|
25
|
+
readonly fingerprint: string;
|
|
26
|
+
readonly kind: 'read' | 'write';
|
|
27
|
+
/** Every statement of this shape, expected ones included — this half measures, it never judges. */
|
|
28
|
+
readonly count: number;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
/**
|
|
32
|
+
* `Disposable`: the observer seam is process-global, so the fixture hands back whatever it found —
|
|
33
|
+
* the state, not a fixed default, exactly as `network` and `runJobs` do.
|
|
34
|
+
*/
|
|
35
|
+
export interface TestStatements extends Disposable {
|
|
36
|
+
/** Every statement issued since the fixture was built, in order. */
|
|
37
|
+
all(): readonly ObservedStatement[];
|
|
38
|
+
/** How many of one shape, or of everything when asked with no fingerprint. */
|
|
39
|
+
count(fingerprint?: string): number;
|
|
40
|
+
/** Shapes seen, most repeated first, ties by fingerprint — a stable list to assert against. */
|
|
41
|
+
shapes(): readonly StatementShape[];
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
/** The live tally behind a `StatementShape`: what it measures, and what it judges. */
|
|
45
|
+
interface ShapeTally {
|
|
46
|
+
readonly fingerprint: string;
|
|
47
|
+
readonly kind: 'read' | 'write';
|
|
48
|
+
/** Every statement of this shape. */
|
|
49
|
+
count: number;
|
|
50
|
+
/** The ones no `expectedQueryLoop()` scope covered — the only ones a verdict may count. */
|
|
51
|
+
unexpected: number;
|
|
52
|
+
/** Whether this shape has already thrown. The flag, not the count, so it throws exactly once. */
|
|
53
|
+
failed: boolean;
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
/**
|
|
57
|
+
* Install the strict detector for the length of one test.
|
|
58
|
+
*
|
|
59
|
+
* Three rules, each load-bearing and each different from `x dev`'s ledger for a stated reason.
|
|
60
|
+
*
|
|
61
|
+
* **The unit of work is the test, not the request.** The ledger keys its tally on the `Ctx` object
|
|
62
|
+
* and ignores a statement issued outside a request; a unit test calls `posts.findById(id)` with no
|
|
63
|
+
* request anywhere, which is exactly the loop it was written to catch — so this counts every
|
|
64
|
+
* statement from the moment the fixture is built until it is disposed, whatever context it was
|
|
65
|
+
* issued in.
|
|
66
|
+
*
|
|
67
|
+
* **The verdict counts only unexpected statements, the measurements count all of them.** An
|
|
68
|
+
* `expectedQueryLoop(reason, fn)` scope is the one way to declare a loop deliberate (there is no
|
|
69
|
+
* flag here and no second suppression), and it suppresses the *verdict* — the statements are still
|
|
70
|
+
* sent, still observed, and still reported by `count()` and `shapes()`, because a test asserting
|
|
71
|
+
* "this page issues two statements" must not see a different number depending on who declared what.
|
|
72
|
+
*
|
|
73
|
+
* **It throws where the loop happened, once per shape.** `@ultimat3/db`'s seam deliberately lets a
|
|
74
|
+
* throw from `onStatement` propagate to whoever ran the statement, which is what makes the failing
|
|
75
|
+
* line the loop's own line rather than a summary at teardown. Once a shape has thrown it keeps
|
|
76
|
+
* counting silently: a test that catches the error gets one failure at the statement that crossed
|
|
77
|
+
* the threshold, not one per statement after it, and `shapes()` still reports the whole loop. The
|
|
78
|
+
* consequence worth knowing is the same one Bullet's `raise` has — a body that swallows every error
|
|
79
|
+
* swallows this one too, and `shapes()` is what names the loop in that test.
|
|
80
|
+
*
|
|
81
|
+
* The threshold is `@ultimat3/entity`'s `N_PLUS_ONE_THRESHOLD` and there is no knob: a loop that
|
|
82
|
+
* fails a test and a loop that warns in `x dev` have to be the same loop.
|
|
83
|
+
*/
|
|
84
|
+
export async function createTestStatements(): Promise<TestStatements> {
|
|
85
|
+
// Imported on demand, like every other fixture factory here: a `packages/core` test that never
|
|
86
|
+
// names `statements` must not load the database layer or the entity registry to run.
|
|
87
|
+
const db = await import('@ultimat3/db');
|
|
88
|
+
const { N_PLUS_ONE_THRESHOLD, nPlusOne } = await import('@ultimat3/entity');
|
|
89
|
+
// Captured before the overwrite: the seam holds one observer, not a list, so an outer diagnostic
|
|
90
|
+
// is displaced for the length of this test and put back after it.
|
|
91
|
+
const previous = db.statementObserver();
|
|
92
|
+
|
|
93
|
+
const seen: ObservedStatement[] = [];
|
|
94
|
+
const tallies = new Map<string, ShapeTally>();
|
|
95
|
+
|
|
96
|
+
const observer: StatementObserver = {
|
|
97
|
+
onStatement(event: StatementEvent): void {
|
|
98
|
+
const fingerprint = db.statementFingerprint(event);
|
|
99
|
+
const kind = db.statementKind(event.text);
|
|
100
|
+
seen.push({
|
|
101
|
+
fingerprint,
|
|
102
|
+
kind,
|
|
103
|
+
text: event.text,
|
|
104
|
+
attribution: event.attribution,
|
|
105
|
+
expected: event.expected,
|
|
106
|
+
});
|
|
107
|
+
let tally = tallies.get(fingerprint);
|
|
108
|
+
if (tally === undefined) {
|
|
109
|
+
tally = { fingerprint, kind, count: 0, unexpected: 0, failed: false };
|
|
110
|
+
tallies.set(fingerprint, tally);
|
|
111
|
+
}
|
|
112
|
+
tally.count += 1;
|
|
113
|
+
if (event.expected !== undefined) return;
|
|
114
|
+
// A statement that threw is still a statement: fifty identical timeouts are still a loop.
|
|
115
|
+
tally.unexpected += 1;
|
|
116
|
+
if (tally.failed || tally.unexpected < N_PLUS_ONE_THRESHOLD) return;
|
|
117
|
+
tally.failed = true;
|
|
118
|
+
// `@ultimat3/entity`'s error, never one composed here: the `fix:` names the `preload` the
|
|
119
|
+
// schema's own relations spell, and a second answer to "what ends this loop" would be one
|
|
120
|
+
// the schema never agreed to.
|
|
121
|
+
throw nPlusOne({
|
|
122
|
+
kind,
|
|
123
|
+
subject: fingerprint,
|
|
124
|
+
count: tally.unexpected,
|
|
125
|
+
entity: event.attribution?.entity,
|
|
126
|
+
op: event.attribution?.op,
|
|
127
|
+
});
|
|
128
|
+
},
|
|
129
|
+
};
|
|
130
|
+
db.setStatementObserver(observer);
|
|
131
|
+
|
|
132
|
+
return {
|
|
133
|
+
all: (): readonly ObservedStatement[] => [...seen],
|
|
134
|
+
count: (fingerprint?: string): number =>
|
|
135
|
+
fingerprint === undefined ? seen.length : (tallies.get(fingerprint)?.count ?? 0),
|
|
136
|
+
shapes: (): readonly StatementShape[] =>
|
|
137
|
+
[...tallies.values()]
|
|
138
|
+
.map(({ fingerprint, kind, count }) => ({ fingerprint, kind, count }))
|
|
139
|
+
.sort((left, right) =>
|
|
140
|
+
left.count === right.count
|
|
141
|
+
? left.fingerprint.localeCompare(right.fingerprint)
|
|
142
|
+
: right.count - left.count,
|
|
143
|
+
),
|
|
144
|
+
[Symbol.dispose]: (): void => {
|
|
145
|
+
db.setStatementObserver(previous);
|
|
146
|
+
},
|
|
147
|
+
};
|
|
148
|
+
}
|
package/src/fixtures.ts
CHANGED
|
@@ -15,6 +15,7 @@ import type { SignIn, Subscribe, TestBudget, TestDeploy } from './fixture-driver
|
|
|
15
15
|
import type { RunJobs } from './fixture-jobs';
|
|
16
16
|
import type { TestMail } from './fixture-mail';
|
|
17
17
|
import type { TestNetwork } from './fixture-network';
|
|
18
|
+
import type { TestStatements } from './fixture-statements';
|
|
18
19
|
import type { PageLike } from './test-types';
|
|
19
20
|
|
|
20
21
|
/** Built once per test, on first use. */
|
|
@@ -44,6 +45,7 @@ export interface Fixtures {
|
|
|
44
45
|
readonly mail: TestMail;
|
|
45
46
|
readonly network: TestNetwork;
|
|
46
47
|
readonly runJobs: RunJobs;
|
|
48
|
+
readonly statements: TestStatements;
|
|
47
49
|
readonly budget: TestBudget;
|
|
48
50
|
readonly deploy: TestDeploy;
|
|
49
51
|
readonly page: PageLike;
|
|
@@ -88,6 +90,54 @@ export function fixtureSnapshot(): FixtureMap {
|
|
|
88
90
|
return Object.fromEntries(registry);
|
|
89
91
|
}
|
|
90
92
|
|
|
93
|
+
const CLOSERS: Readonly<Record<string, string>> = { '{': '}', '[': ']', '(': ')' };
|
|
94
|
+
const QUOTES = new Set(['"', "'", '`']);
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* The pattern's own top-level segments: the `}` that closes the opening `{`, and the commas at
|
|
98
|
+
* depth zero inside it. `indexOf('}')` stopped at the FIRST closer, so `{ mail, clock: { now },
|
|
99
|
+
* network }` lost `network` and `{ clock = { now: 1 }, mail }` lost both — silently, and a fixture
|
|
100
|
+
* that is never built reads as `undefined` in the body, which is the failure this module exists to
|
|
101
|
+
* turn into a message. Strings are skipped so a default like `{ role = 'a, b' }` is one segment.
|
|
102
|
+
*/
|
|
103
|
+
function patternSegments(source: string, open: number): readonly string[] | undefined {
|
|
104
|
+
const segments: string[] = [];
|
|
105
|
+
const stack: string[] = [];
|
|
106
|
+
let start = open + 1;
|
|
107
|
+
let quote: string | undefined;
|
|
108
|
+
for (let index = open; index < source.length; index += 1) {
|
|
109
|
+
const char = source[index] ?? '';
|
|
110
|
+
if (quote !== undefined) {
|
|
111
|
+
if (char === '\\') index += 1;
|
|
112
|
+
else if (char === quote) quote = undefined;
|
|
113
|
+
continue;
|
|
114
|
+
}
|
|
115
|
+
if (QUOTES.has(char)) {
|
|
116
|
+
quote = char;
|
|
117
|
+
continue;
|
|
118
|
+
}
|
|
119
|
+
const closer = CLOSERS[char];
|
|
120
|
+
if (closer !== undefined) {
|
|
121
|
+
stack.push(closer);
|
|
122
|
+
continue;
|
|
123
|
+
}
|
|
124
|
+
if (stack.length > 0 && char === stack[stack.length - 1]) {
|
|
125
|
+
stack.pop();
|
|
126
|
+
if (stack.length === 0) {
|
|
127
|
+
segments.push(source.slice(start, index));
|
|
128
|
+
return segments;
|
|
129
|
+
}
|
|
130
|
+
continue;
|
|
131
|
+
}
|
|
132
|
+
if (char === ',' && stack.length === 1) {
|
|
133
|
+
segments.push(source.slice(start, index));
|
|
134
|
+
start = index + 1;
|
|
135
|
+
}
|
|
136
|
+
}
|
|
137
|
+
// An unbalanced pattern is not one this can read; building nothing beats guessing a name.
|
|
138
|
+
return undefined;
|
|
139
|
+
}
|
|
140
|
+
|
|
91
141
|
/**
|
|
92
142
|
* The names a body destructures, read from its source.
|
|
93
143
|
*
|
|
@@ -100,15 +150,11 @@ export function requestedFixtures(body: (...args: never[]) => unknown): readonly
|
|
|
100
150
|
const source = body.toString();
|
|
101
151
|
const open = source.indexOf('{');
|
|
102
152
|
if (open === -1) return [];
|
|
103
|
-
const close = source.indexOf('}', open);
|
|
104
|
-
if (close === -1) return [];
|
|
105
153
|
// Bail if the brace opens a body rather than a destructuring pattern — `async () => {`.
|
|
106
154
|
const beforeBrace = source.slice(0, open);
|
|
107
155
|
if (/\)\s*(?::[^=]*)?=>\s*$/.test(beforeBrace) || /\)\s*$/.test(beforeBrace)) return [];
|
|
108
|
-
return source
|
|
109
|
-
.
|
|
110
|
-
.split(',')
|
|
111
|
-
.map((part) => (part.split(':')[0] ?? '').trim())
|
|
156
|
+
return (patternSegments(source, open) ?? [])
|
|
157
|
+
.map((part) => (part.split(/[:=]/)[0] ?? '').trim())
|
|
112
158
|
.filter((name) => /^[A-Za-z_$][\w$]*$/.test(name));
|
|
113
159
|
}
|
|
114
160
|
|
|
@@ -7,6 +7,7 @@ import { DRIVER_FIXTURE_NAMES, driverFixtures } from './fixture-drivers';
|
|
|
7
7
|
import { createRunJobs } from './fixture-jobs';
|
|
8
8
|
import { createTestMail } from './fixture-mail';
|
|
9
9
|
import { createTestNetwork } from './fixture-network';
|
|
10
|
+
import { createTestStatements } from './fixture-statements';
|
|
10
11
|
import { defineFixtures } from './fixtures';
|
|
11
12
|
|
|
12
13
|
/**
|
|
@@ -15,7 +16,13 @@ import { defineFixtures } from './fixtures';
|
|
|
15
16
|
* installs. See `fixture-drivers.ts` for why a declared-and-unavailable fixture beats an
|
|
16
17
|
* unregistered name.
|
|
17
18
|
*/
|
|
18
|
-
export const FRAMEWORK_FIXTURE_NAMES = [
|
|
19
|
+
export const FRAMEWORK_FIXTURE_NAMES = [
|
|
20
|
+
'clock',
|
|
21
|
+
'mail',
|
|
22
|
+
'network',
|
|
23
|
+
'runJobs',
|
|
24
|
+
'statements',
|
|
25
|
+
] as const;
|
|
19
26
|
|
|
20
27
|
export { DRIVER_FIXTURE_NAMES };
|
|
21
28
|
|
|
@@ -38,5 +45,6 @@ export function registerFrameworkFixtures(): void {
|
|
|
38
45
|
mail: createTestMail,
|
|
39
46
|
network: createTestNetwork,
|
|
40
47
|
runJobs: createRunJobs,
|
|
48
|
+
statements: createTestStatements,
|
|
41
49
|
});
|
|
42
50
|
}
|
package/src/harness.ts
CHANGED
|
@@ -3,8 +3,14 @@
|
|
|
3
3
|
// into a race and every failure into a log-scraping exercise.
|
|
4
4
|
|
|
5
5
|
import { afterAll, beforeAll, describe, test } from 'bun:test';
|
|
6
|
-
import { installDeterminism,
|
|
7
|
-
import {
|
|
6
|
+
import { captureDeterminism, installDeterminism, restoreCapturedDeterminism } from './determinism';
|
|
7
|
+
import {
|
|
8
|
+
allowHost,
|
|
9
|
+
isNetworkSealed,
|
|
10
|
+
resetNetwork,
|
|
11
|
+
sealNetwork,
|
|
12
|
+
unsealNetwork,
|
|
13
|
+
} from './sealed-network';
|
|
8
14
|
import type { TemplateDbConfig, WorkerDatabase } from './template-db';
|
|
9
15
|
import { acquireWorkerDatabase } from './template-db';
|
|
10
16
|
import type { TestType } from './test-types';
|
|
@@ -36,19 +42,77 @@ export interface AppHandle {
|
|
|
36
42
|
|
|
37
43
|
const BASE = 'http://app.test';
|
|
38
44
|
|
|
39
|
-
|
|
45
|
+
export interface BootedHarness {
|
|
46
|
+
readonly handle: AppHandle;
|
|
47
|
+
close(): Promise<void>;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
export interface HarnessDeps {
|
|
51
|
+
/**
|
|
52
|
+
* How this harness gets a database. A parameter for the reason `TemplateDbDeps.connect` is one:
|
|
53
|
+
* the rejection path below drops what it acquired, and the PGlite fallback's `drop` is a no-op,
|
|
54
|
+
* so proving it needs an injected handle rather than a server.
|
|
55
|
+
*/
|
|
56
|
+
readonly acquire: (config: TemplateDbConfig) => Promise<WorkerDatabase>;
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* The lifecycle `describeApp` and `testApp` are the two idiomatic wrappers around. Exported from
|
|
61
|
+
* this module but deliberately NOT from `src/index.ts`: those two are the ways an app boots, and a
|
|
62
|
+
* third public entry point would be a second answer to one question. It is exported at all because
|
|
63
|
+
* the teardown contract — a rejecting `close` must still drop the cloned database — cannot be
|
|
64
|
+
* asserted through a wrapper that rethrows into the test's own result.
|
|
65
|
+
*/
|
|
66
|
+
export async function bootApp(
|
|
40
67
|
options: AppOptions,
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
68
|
+
deps: Partial<HarnessDeps> = {},
|
|
69
|
+
): Promise<BootedHarness> {
|
|
70
|
+
// Captured, never assumed. The preload already sealed the network and installed determinism for
|
|
71
|
+
// the whole process, so `sealNetwork()` here is a no-op and an unconditional teardown would hand
|
|
72
|
+
// the real `fetch`, the real `Date` and the real `Math.random` to every later FILE in the run —
|
|
73
|
+
// `bun test` is one process. Restore only what this boot actually changed.
|
|
74
|
+
const determinism = captureDeterminism();
|
|
75
|
+
const sealedBefore = isNetworkSealed();
|
|
76
|
+
|
|
77
|
+
// Only when this boot has something of its own to say. A run configured with ULTIMATE_TEST_NOW /
|
|
78
|
+
// ULTIMATE_TEST_SEED (`preload.ts`) is otherwise reset to the defaults by the first describeApp.
|
|
79
|
+
if (!determinism.installed || options.seedValue !== undefined || options.now !== undefined) {
|
|
80
|
+
installDeterminism({
|
|
81
|
+
...(options.seedValue === undefined ? {} : { seed: options.seedValue }),
|
|
82
|
+
...(options.now === undefined ? {} : { now: options.now }),
|
|
83
|
+
});
|
|
84
|
+
}
|
|
46
85
|
sealNetwork();
|
|
47
86
|
for (const host of options.allowHosts ?? []) allowHost(host);
|
|
48
87
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
const
|
|
88
|
+
// What `close` puts back, named once: the boot has to run it too, and two copies of this list is
|
|
89
|
+
// how one of them ends up missing the allow-list.
|
|
90
|
+
const restoreProcessState = (): void => {
|
|
91
|
+
resetNetwork();
|
|
92
|
+
if (!sealedBefore) unsealNetwork();
|
|
93
|
+
restoreCapturedDeterminism(determinism);
|
|
94
|
+
};
|
|
95
|
+
|
|
96
|
+
let db: WorkerDatabase | undefined;
|
|
97
|
+
let app: BootedApp;
|
|
98
|
+
try {
|
|
99
|
+
db = await (deps.acquire ?? acquireWorkerDatabase)(options.db ?? {});
|
|
100
|
+
if (options.seed !== undefined) await options.seed(db.url);
|
|
101
|
+
app = await options.boot({ databaseUrl: db.url });
|
|
102
|
+
} catch (error) {
|
|
103
|
+
// A boot that rejects returns no `BootedHarness`, so nothing can ever call the `close` below —
|
|
104
|
+
// the same leak that block exists to prevent, one function earlier. A failing `seed` stranded
|
|
105
|
+
// its clone and left the clock, the seal and the allow-list this boot installed to every later
|
|
106
|
+
// FILE in the run; `bun test` is one process.
|
|
107
|
+
try {
|
|
108
|
+
await db?.drop();
|
|
109
|
+
} catch {
|
|
110
|
+
// The boot's own failure is what the caller must see, and there is no handle to report a
|
|
111
|
+
// second one through — the same "first failure wins" rule `close` runs by.
|
|
112
|
+
}
|
|
113
|
+
restoreProcessState();
|
|
114
|
+
throw error;
|
|
115
|
+
}
|
|
52
116
|
|
|
53
117
|
const handle: AppHandle = {
|
|
54
118
|
db,
|
|
@@ -61,12 +125,24 @@ async function boot(
|
|
|
61
125
|
|
|
62
126
|
return {
|
|
63
127
|
handle,
|
|
128
|
+
// Every step runs even when an earlier one rejects, and the FIRST failure is what the caller
|
|
129
|
+
// sees — the same rule `fixtures.ts` disposes by. An `app.close()` that threw used to strand
|
|
130
|
+
// the clone: one `ultimate_test_template_wN` leaked per failing run, and the seal and the
|
|
131
|
+
// clock were never put back either.
|
|
64
132
|
close: async () => {
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
133
|
+
let failure: { readonly error: unknown } | undefined;
|
|
134
|
+
try {
|
|
135
|
+
await app.close?.();
|
|
136
|
+
} catch (error) {
|
|
137
|
+
failure = { error };
|
|
138
|
+
}
|
|
139
|
+
try {
|
|
140
|
+
await db.drop();
|
|
141
|
+
} catch (error) {
|
|
142
|
+
failure ??= { error };
|
|
143
|
+
}
|
|
144
|
+
restoreProcessState();
|
|
145
|
+
if (failure !== undefined) throw failure.error;
|
|
70
146
|
},
|
|
71
147
|
};
|
|
72
148
|
}
|
|
@@ -81,9 +157,9 @@ export function describeApp(
|
|
|
81
157
|
body: (app: () => AppHandle) => void,
|
|
82
158
|
): void {
|
|
83
159
|
describe(name, () => {
|
|
84
|
-
let booted:
|
|
160
|
+
let booted: BootedHarness | undefined;
|
|
85
161
|
beforeAll(async () => {
|
|
86
|
-
booted = await
|
|
162
|
+
booted = await bootApp(options);
|
|
87
163
|
});
|
|
88
164
|
afterAll(async () => {
|
|
89
165
|
await booted?.close();
|
|
@@ -104,7 +180,7 @@ export function testApp(
|
|
|
104
180
|
type: TestType = 'unit',
|
|
105
181
|
): void {
|
|
106
182
|
test(testName(type, name), async () => {
|
|
107
|
-
const booted = await
|
|
183
|
+
const booted = await bootApp(options);
|
|
108
184
|
try {
|
|
109
185
|
await body(booted.handle);
|
|
110
186
|
} finally {
|
package/src/index.ts
CHANGED
|
@@ -23,16 +23,21 @@ export {
|
|
|
23
23
|
// `test` is OURS (fixture-injecting); everything else passes through. Re-exported so an app
|
|
24
24
|
// test has one import line, and so `expect` carries this package's matchers already installed.
|
|
25
25
|
export { afterAll, afterEach, beforeAll, beforeEach, describe, expect } from 'bun:test';
|
|
26
|
-
export type { DeterminismOptions } from './determinism';
|
|
26
|
+
export type { DeterminismOptions, DeterminismSnapshot } from './determinism';
|
|
27
|
+
// `captureDeterminism` + `restoreCapturedDeterminism` are the pair a NESTED install needs;
|
|
28
|
+
// `restoreDeterminism` uninstalls outright and hands the real clock and the real `Math.random`
|
|
29
|
+
// back to every later file in the process, which is only ever what the process itself wants.
|
|
27
30
|
export {
|
|
28
31
|
advanceClock,
|
|
29
32
|
assertDeterministic,
|
|
33
|
+
captureDeterminism,
|
|
30
34
|
DEFAULT_NOW,
|
|
31
35
|
DEFAULT_SEED,
|
|
32
36
|
frozenClock,
|
|
33
37
|
frozenNow,
|
|
34
38
|
installDeterminism,
|
|
35
39
|
isDeterminismInstalled,
|
|
40
|
+
restoreCapturedDeterminism,
|
|
36
41
|
restoreDeterminism,
|
|
37
42
|
seededRandom,
|
|
38
43
|
seededUuid,
|
|
@@ -44,6 +49,7 @@ export {
|
|
|
44
49
|
NetworkOfflineError,
|
|
45
50
|
NetworkSealedError,
|
|
46
51
|
NondeterministicError,
|
|
52
|
+
RegistryLeakError,
|
|
47
53
|
TESTING_ERROR_CODES,
|
|
48
54
|
TESTING_ERROR_TITLES,
|
|
49
55
|
TestDatabaseUnavailableError,
|
|
@@ -83,6 +89,8 @@ export type { MailRef, TestMail } from './fixture-mail';
|
|
|
83
89
|
export { createTestMail } from './fixture-mail';
|
|
84
90
|
export type { TestNetwork } from './fixture-network';
|
|
85
91
|
export { createTestNetwork } from './fixture-network';
|
|
92
|
+
export type { ObservedStatement, StatementShape, TestStatements } from './fixture-statements';
|
|
93
|
+
export { createTestStatements } from './fixture-statements';
|
|
86
94
|
export { fixtureTest as test } from './fixtures';
|
|
87
95
|
export {
|
|
88
96
|
ALL_FIXTURE_NAMES,
|
|
@@ -94,6 +102,14 @@ export type { AppHandle, AppOptions, BootedApp } from './harness';
|
|
|
94
102
|
export { describeApp, testApp } from './harness';
|
|
95
103
|
export type { MatcherResult } from './matchers';
|
|
96
104
|
export { matchersInstalled, recordSteps } from './matchers';
|
|
105
|
+
// `isolateEntityRegistry` is deliberately NOT here — it is `@ultimat3/testing/registry-isolation`.
|
|
106
|
+
// This barrel is what a `packages/core` test imports for `expect` alone, and a static re-export
|
|
107
|
+
// evaluates the module it names: that one would load `@ultimat3/entity` and its process-global
|
|
108
|
+
// registry into every test in the framework. Every other package this harness touches is imported
|
|
109
|
+
// dynamically inside the fixture that needs it; a helper that must stay synchronous cannot do that,
|
|
110
|
+
// so it gets its own entry point instead.
|
|
111
|
+
export type { RegistryLeak, RegistrySample } from './registry-leak-guard';
|
|
112
|
+
export { installRegistryLeakGuard, leakBetween, sampleRegistries } from './registry-leak-guard';
|
|
97
113
|
export type { MockRoute, NetworkState } from './sealed-network';
|
|
98
114
|
// `setNetworkState` is deliberately not here: it is the offline gate's one writer, and a test that
|
|
99
115
|
// called it directly would bypass the `network` fixture's disposal and leave the whole process
|
|
@@ -138,6 +154,7 @@ export {
|
|
|
138
154
|
contractTest,
|
|
139
155
|
e2eTest,
|
|
140
156
|
evalTest,
|
|
157
|
+
hasE2eDriver,
|
|
141
158
|
jobTest,
|
|
142
159
|
liveTest,
|
|
143
160
|
SEPARATOR,
|
package/src/matchers.ts
CHANGED
|
@@ -105,6 +105,43 @@ const result = (pass: boolean, message: string): MatcherResult => ({
|
|
|
105
105
|
message: () => message,
|
|
106
106
|
});
|
|
107
107
|
|
|
108
|
+
const isOpenApiLike = (value: unknown): value is OpenApiLike =>
|
|
109
|
+
isRecord(value) &&
|
|
110
|
+
Array.isArray(value['operations']) &&
|
|
111
|
+
value['operations'].every(
|
|
112
|
+
(entry) => typeof (entry as OpenApiLike['operations'][number] | null)?.operationId === 'string',
|
|
113
|
+
);
|
|
114
|
+
|
|
115
|
+
/**
|
|
116
|
+
* The two breaking changes this matcher can see from the shape `OpenApiLike` declares: an
|
|
117
|
+
* operation that disappeared, and a parameter a surviving operation newly requires — the second
|
|
118
|
+
* breaks every caller already omitting it, and comparing operation ids alone let it through while
|
|
119
|
+
* a suite naming this matcher read as covered.
|
|
120
|
+
*
|
|
121
|
+
* Deliberately NOT a full OpenAPI diff. Response status codes, response schemas and parameter
|
|
122
|
+
* TYPES are not on `OpenApiLike` and are not compared; the message says what did break rather than
|
|
123
|
+
* claiming the contract is otherwise unchanged. `x verify`'s `contract-diff` step is the whole
|
|
124
|
+
* answer, against the committed manifest.
|
|
125
|
+
*/
|
|
126
|
+
function breakingChanges(before: OpenApiLike, after: OpenApiLike): readonly string[] {
|
|
127
|
+
const current = new Map(after.operations.map((operation) => [operation.operationId, operation]));
|
|
128
|
+
const broke: string[] = [];
|
|
129
|
+
for (const operation of before.operations) {
|
|
130
|
+
const now = current.get(operation.operationId);
|
|
131
|
+
if (now === undefined) {
|
|
132
|
+
broke.push(`removed operation ${operation.operationId}`);
|
|
133
|
+
continue;
|
|
134
|
+
}
|
|
135
|
+
// Only additions: dropping a requirement widens what the API accepts, which no caller notices.
|
|
136
|
+
const wasRequired = new Set(operation.required ?? []);
|
|
137
|
+
const added = (now.required ?? []).filter((name) => !wasRequired.has(name));
|
|
138
|
+
if (added.length > 0) {
|
|
139
|
+
broke.push(`${operation.operationId} newly requires ${added.join(', ')}`);
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
return broke;
|
|
143
|
+
}
|
|
144
|
+
|
|
108
145
|
expect.extend({
|
|
109
146
|
toBeUltimateError(received: unknown, code?: string) {
|
|
110
147
|
const actual = codeOf(received);
|
|
@@ -135,13 +172,16 @@ expect.extend({
|
|
|
135
172
|
},
|
|
136
173
|
|
|
137
174
|
toMatchOpenApi(received: unknown, committed: OpenApiLike) {
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
175
|
+
if (!isOpenApiLike(received)) {
|
|
176
|
+
return result(
|
|
177
|
+
false,
|
|
178
|
+
'expected an OpenAPI document — an object with operations: [{ operationId, required? }]',
|
|
179
|
+
);
|
|
180
|
+
}
|
|
181
|
+
const broke = breakingChanges(committed, received);
|
|
142
182
|
return result(
|
|
143
|
-
|
|
144
|
-
`contract
|
|
183
|
+
broke.length === 0,
|
|
184
|
+
`contract broke: ${broke.join('; ')} — bump the package version or restore the old shape`,
|
|
145
185
|
);
|
|
146
186
|
},
|
|
147
187
|
|
package/src/preload.ts
CHANGED
|
@@ -8,6 +8,7 @@
|
|
|
8
8
|
import { installDeterminism } from './determinism';
|
|
9
9
|
import { registerFrameworkFixtures } from './framework-fixtures';
|
|
10
10
|
import './matchers';
|
|
11
|
+
import { installRegistryLeakGuard } from './registry-leak-guard';
|
|
11
12
|
import { sealNetwork } from './sealed-network';
|
|
12
13
|
|
|
13
14
|
const seed = Number.parseInt(Bun.env['ULTIMATE_TEST_SEED'] ?? '', 10);
|
|
@@ -20,6 +21,10 @@ installDeterminism({
|
|
|
20
21
|
|
|
21
22
|
registerFrameworkFixtures();
|
|
22
23
|
|
|
24
|
+
// One `bun test` invocation is one process: a file that leaves a process-global registry dirty
|
|
25
|
+
// fails a later file in another package, for a reason nothing in that file explains.
|
|
26
|
+
installRegistryLeakGuard();
|
|
27
|
+
|
|
23
28
|
// Opt-out exists for one case: a test that deliberately exercises a real integration in a job the
|
|
24
29
|
// team runs on purpose. It is an env var, not an API, so it cannot be set from inside a test file.
|
|
25
30
|
if (Bun.env['ULTIMATE_TEST_ALLOW_NET'] !== '1') sealNetwork();
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
// One test seam: an empty entity registry for the test that needs one, and the process's own
|
|
2
|
+
// registry back afterwards. `entity()` registers at MODULE scope — which is idiomatic, an app
|
|
3
|
+
// declares its domain by importing it — so a test asserting on the WHOLE registry inherits every
|
|
4
|
+
// entity any earlier file in the same `bun test` process imported, and its premise becomes
|
|
5
|
+
// whatever ran before it rather than what it declared.
|
|
6
|
+
//
|
|
7
|
+
// Its OWN entry point (`@ultimat3/testing/registry-isolation`), off the barrel: the value import
|
|
8
|
+
// below is static because the restore has to be handed back synchronously, so re-exporting this
|
|
9
|
+
// from `src/index.ts` would load the entity registry into every test that imports the harness for
|
|
10
|
+
// `expect`. One import line is worth less than a tier-0 test that stays tier-0.
|
|
11
|
+
|
|
12
|
+
import { clearRegistry, registerEntity, registeredEntities } from '@ultimat3/entity';
|
|
13
|
+
|
|
14
|
+
/**
|
|
15
|
+
* Empties the entity registry and returns the function that puts it back, entry for entry.
|
|
16
|
+
* For a test whose subject is "no entities are declared" — `x db gen` with nothing to generate,
|
|
17
|
+
* a manifest with no rows — which is otherwise true only until a neighbouring file imports one:
|
|
18
|
+
*
|
|
19
|
+
* const restoreEntities = isolateEntityRegistry();
|
|
20
|
+
* try {
|
|
21
|
+
* // …the assertion that needs an empty registry
|
|
22
|
+
* } finally {
|
|
23
|
+
* restoreEntities();
|
|
24
|
+
* }
|
|
25
|
+
*
|
|
26
|
+
* Restoring rather than leaving it empty is the half that matters: `entity()` runs once per
|
|
27
|
+
* module, so a registry cleared and not refilled cannot be repopulated by a later import.
|
|
28
|
+
*/
|
|
29
|
+
export function isolateEntityRegistry(): () => void {
|
|
30
|
+
const captured = registeredEntities();
|
|
31
|
+
clearRegistry();
|
|
32
|
+
return () => {
|
|
33
|
+
clearRegistry();
|
|
34
|
+
for (const entry of captured) registerEntity(entry);
|
|
35
|
+
};
|
|
36
|
+
}
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
// Cross-file state pollution, caught at the boundary it crosses. `bun test` runs every file of one
|
|
2
|
+
// invocation in ONE process — only `--isolate` gives each file its own module registry — so a file
|
|
3
|
+
// that leaves a process-global registry dirty changes what every file after it sees, and the
|
|
4
|
+
// failure lands on an innocent suite in another package. This names the file that leaked.
|
|
5
|
+
|
|
6
|
+
import { afterAll } from 'bun:test';
|
|
7
|
+
import { knownTags, registeredTiers } from '@ultimat3/cache';
|
|
8
|
+
import { RegistryLeakError } from './errors';
|
|
9
|
+
|
|
10
|
+
/**
|
|
11
|
+
* What is guarded, and why only these two. Both are BOOT installs — `declareTags` takes the
|
|
12
|
+
* manifest's entity names, `registerTier` takes `app.config.ts`'s tiers — so "empty again when the
|
|
13
|
+
* file ends" is the honest invariant for a test. The entity, job, route and permission registries
|
|
14
|
+
* are not here: `entity()` and `job()` register at module scope, which is how an app declares
|
|
15
|
+
* itself, so a file that leaves them filled is idiomatic rather than leaky. A test whose subject is
|
|
16
|
+
* an EMPTY one of those establishes it itself — `isolateEntityRegistry()`.
|
|
17
|
+
*/
|
|
18
|
+
export interface RegistrySample {
|
|
19
|
+
readonly tags: readonly string[];
|
|
20
|
+
readonly tiers: readonly string[];
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
export interface RegistryLeak {
|
|
24
|
+
/** Repo-relative when it can be, so the message names the file an editor opens. */
|
|
25
|
+
readonly file: string;
|
|
26
|
+
readonly tags: readonly string[];
|
|
27
|
+
readonly tiers: readonly string[];
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
export function sampleRegistries(): RegistrySample {
|
|
31
|
+
return { tags: [...knownTags()], tiers: registeredTiers().map((tier) => tier.name) };
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* Additions only. A file that DROPS a tier a previous file registered is a different bug and not
|
|
36
|
+
* this one's to report — reporting both here would make the message ambiguous about which file to
|
|
37
|
+
* open.
|
|
38
|
+
*/
|
|
39
|
+
export function leakBetween(
|
|
40
|
+
file: string,
|
|
41
|
+
before: RegistrySample,
|
|
42
|
+
after: RegistrySample,
|
|
43
|
+
): RegistryLeak | undefined {
|
|
44
|
+
const tags = after.tags.filter((name) => !before.tags.includes(name));
|
|
45
|
+
const tiers = after.tiers.filter((name) => !before.tiers.includes(name));
|
|
46
|
+
if (tags.length === 0 && tiers.length === 0) return undefined;
|
|
47
|
+
return { file, tags, tiers };
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/** `Bun.file(path).text()` answers an absolute path; the message wants the one a reader types. */
|
|
51
|
+
const repoRelative = (path: string): string => {
|
|
52
|
+
const root = `${process.cwd()}/`;
|
|
53
|
+
return path.startsWith(root) ? path.slice(root.length) : path;
|
|
54
|
+
};
|
|
55
|
+
|
|
56
|
+
/**
|
|
57
|
+
* The one point at which a file's baseline is honest, and the reason it is not a hook. Measured on
|
|
58
|
+
* Bun 1.3.14 with a `Bun.plugin` load handler and hooks at every scope, the order is:
|
|
59
|
+
*
|
|
60
|
+
* onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload `beforeEach` → test
|
|
61
|
+
*
|
|
62
|
+
* A preload's `beforeEach` therefore runs AFTER the file's own `beforeAll`, so a `declareTags()`
|
|
63
|
+
* there landed in the baseline and the file read clean — a false green in the guard whose entire
|
|
64
|
+
* job is catching false greens. Appending the sample to the file's own source is what puts it
|
|
65
|
+
* after module evaluation (its environment) and before the first hook the file registers (its own
|
|
66
|
+
* doing). `bun:test` hooks carry no file identity; this loader does.
|
|
67
|
+
*/
|
|
68
|
+
const BASELINE_HOOK = '__ultimateRegistryLeakBaseline';
|
|
69
|
+
|
|
70
|
+
/** Appended, never prepended: an `import` is hoisted and would sample before the graph evaluates. */
|
|
71
|
+
const SAMPLE_BASELINE = `\n;globalThis[${JSON.stringify(BASELINE_HOOK)}]?.();\n`;
|
|
72
|
+
|
|
73
|
+
const hookHost = globalThis as typeof globalThis & { [BASELINE_HOOK]?: () => void };
|
|
74
|
+
|
|
75
|
+
let installed = false;
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* Called by the test preload, once per process. Idempotent because two preloads reaching it would
|
|
79
|
+
* otherwise register the hooks twice and report every leak twice.
|
|
80
|
+
*
|
|
81
|
+
* Under `--isolate` each file gets its own module registry, so the guard judges that one file and
|
|
82
|
+
* nothing carries across — which is correct, not a hole: with `--isolate` there is no cross-file
|
|
83
|
+
* pollution to find, and a file that leaks is still reported against itself.
|
|
84
|
+
*/
|
|
85
|
+
export function installRegistryLeakGuard(): void {
|
|
86
|
+
if (installed) return;
|
|
87
|
+
installed = true;
|
|
88
|
+
|
|
89
|
+
let pending: string | undefined;
|
|
90
|
+
let current: { readonly file: string; readonly before: RegistrySample } | undefined;
|
|
91
|
+
const leaks: RegistryLeak[] = [];
|
|
92
|
+
|
|
93
|
+
const close = (): void => {
|
|
94
|
+
if (current === undefined) return;
|
|
95
|
+
const leak = leakBetween(current.file, current.before, sampleRegistries());
|
|
96
|
+
if (leak !== undefined) leaks.push(leak);
|
|
97
|
+
current = undefined;
|
|
98
|
+
};
|
|
99
|
+
|
|
100
|
+
// Called by the statement appended below, once per test file, from that file's own module scope:
|
|
101
|
+
// everything the file's MODULE graph registered is its environment — importing an app module is
|
|
102
|
+
// how an app declares its tags — and everything after this point is the file's own to undo.
|
|
103
|
+
hookHost[BASELINE_HOOK] = () => {
|
|
104
|
+
if (pending === undefined) return;
|
|
105
|
+
current = { file: pending, before: sampleRegistries() };
|
|
106
|
+
pending = undefined;
|
|
107
|
+
};
|
|
108
|
+
|
|
109
|
+
// The only signal Bun gives a preload for "a new test file starts": its hooks carry no file. A
|
|
110
|
+
// load handler MUST answer with contents — one that answers `undefined` makes Bun load nothing
|
|
111
|
+
// for the file and the run reports zero tests, silently.
|
|
112
|
+
Bun.plugin({
|
|
113
|
+
name: 'ultimate-registry-leak-guard',
|
|
114
|
+
setup(build) {
|
|
115
|
+
// `.test.ts` only, never `.test.tsx`. A load handler that answered `loader: 'tsx'` would
|
|
116
|
+
// compile JSX with Bun's CLASSIC React fallback — the very factory `@ultimat3/render`'s own
|
|
117
|
+
// loader exists to replace — and, because the first matching handler wins and this one is
|
|
118
|
+
// registered from the preload, it would shadow render's transform for that file. Routing
|
|
119
|
+
// `.tsx` through `transformTsx` is not the alternative either: it needs `@ultimat3/render`,
|
|
120
|
+
// whose import installs that global loader into every test process in the repo. Zero
|
|
121
|
+
// `.test.tsx` files exist and the convention is `<file>.test.ts`, so the narrower filter
|
|
122
|
+
// costs nothing today; a `.test.tsx` added later is unguarded rather than mis-compiled.
|
|
123
|
+
build.onLoad({ filter: /\.test\.ts$/ }, async (args) => {
|
|
124
|
+
close();
|
|
125
|
+
pending = repoRelative(args.path);
|
|
126
|
+
return {
|
|
127
|
+
contents: `${await Bun.file(args.path).text()}${SAMPLE_BASELINE}`,
|
|
128
|
+
loader: 'ts',
|
|
129
|
+
};
|
|
130
|
+
});
|
|
131
|
+
},
|
|
132
|
+
});
|
|
133
|
+
|
|
134
|
+
afterAll(() => {
|
|
135
|
+
close();
|
|
136
|
+
if (leaks.length > 0) throw new RegistryLeakError({ leaks });
|
|
137
|
+
});
|
|
138
|
+
}
|
package/src/template-db.ts
CHANGED
|
@@ -130,11 +130,19 @@ export async function acquireWorkerDatabase(
|
|
|
130
130
|
try {
|
|
131
131
|
await admin.exec(lockSql(template));
|
|
132
132
|
try {
|
|
133
|
-
|
|
133
|
+
try {
|
|
134
|
+
await admin.exec(createTemplateSql(template));
|
|
135
|
+
} catch (error) {
|
|
136
|
+
// Tolerated for the CREATE alone: "already exists" means another worker got here first, or
|
|
137
|
+
// a Postgres that outlives one run still holds last run's template.
|
|
138
|
+
if (!alreadyExists(error)) throw error;
|
|
139
|
+
}
|
|
140
|
+
// Outside that tolerance on purpose, and unconditional. A template found rather than created
|
|
141
|
+
// is not a migrated one — skipping here clones the first run's schema forever, on every
|
|
142
|
+
// server that is not a fresh CI container. Migrations are idempotent and the advisory lock
|
|
143
|
+
// serialises them, so a failure here is a real failure and `alreadyExists` (a substring match
|
|
144
|
+
// on the message) would read `relation "x_jobs" already exists` as success.
|
|
134
145
|
if (config.migrate !== undefined) await config.migrate(urlFor(adminUrl, template));
|
|
135
|
-
} catch (error) {
|
|
136
|
-
// "already exists" means another worker migrated it first — the lock made that safe.
|
|
137
|
-
if (!alreadyExists(error)) throw error;
|
|
138
146
|
} finally {
|
|
139
147
|
await admin.exec(unlockSql(template));
|
|
140
148
|
}
|
package/src/test-types.ts
CHANGED
|
@@ -81,14 +81,25 @@ export type E2eBody = (fixtures: E2eFixtures) => Promise<void>;
|
|
|
81
81
|
let e2eDriver: ((name: string, body: E2eBody) => void) | undefined;
|
|
82
82
|
|
|
83
83
|
/**
|
|
84
|
-
* Register the
|
|
85
|
-
*
|
|
86
|
-
*
|
|
84
|
+
* Register the browser-backed driver. Without one, `e2eTest` skips loudly — the skipped test's own
|
|
85
|
+
* NAME carries the reason and the command that would build what it drives.
|
|
86
|
+
*
|
|
87
|
+
* What the gate then reports is a **pass over an all-skipped suite**, and that is stated here
|
|
88
|
+
* rather than claimed away: this used to say "`x verify` reports the step as skipped rather than
|
|
89
|
+
* green", which nothing implements. The step shells out to `bun test`, `bun test` exits 0 on a
|
|
90
|
+
* skip, and an exit code is the only channel between the two — the driver is registered inside the
|
|
91
|
+
* CHILD process, so the step cannot ask. Closing it means a channel the step can read, which is a
|
|
92
|
+
* design decision and not a docstring. `As of 2026-08` there are zero registered drivers, so every
|
|
93
|
+
* `e2eTest` in the tree is a skip; the framework's own `e2e` suites use plain `bun:test`.
|
|
87
94
|
*/
|
|
88
95
|
export function useE2eDriver(driver: (name: string, body: E2eBody) => void): void {
|
|
89
96
|
e2eDriver = driver;
|
|
90
97
|
}
|
|
91
98
|
|
|
99
|
+
/** Whether a driver is registered. Exported so a harness can say which of the two states it is in
|
|
100
|
+
* instead of reading an all-skipped run as a green one. */
|
|
101
|
+
export const hasE2eDriver = (): boolean => e2eDriver !== undefined;
|
|
102
|
+
|
|
92
103
|
export const e2eTest = (name: string, body: E2eBody): void => {
|
|
93
104
|
if (e2eDriver === undefined) {
|
|
94
105
|
test.skip(
|