@ultimat3/testing 29.3.2 → 30.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (93) hide show
  1. package/CLAUDE.md +56 -63
  2. package/README.md +281 -6
  3. package/package.json +27 -16
  4. package/src/ax-node.ts +96 -0
  5. package/src/cassette-errors.ts +91 -0
  6. package/src/cassette-format.ts +175 -0
  7. package/src/cassette-record.ts +120 -0
  8. package/src/cassette-secret-scan.ts +77 -0
  9. package/src/cassette.ts +166 -0
  10. package/src/cdp-browser.ts +5 -3
  11. package/src/cdp-e2e-page.ts +91 -3
  12. package/src/cdp-e2e-session.ts +18 -1
  13. package/src/define-exercise.ts +82 -0
  14. package/src/drill-budgets.ts +54 -0
  15. package/src/drill-errors.ts +45 -0
  16. package/src/drill-fleet.ts +251 -0
  17. package/src/drill-jobs.ts +79 -0
  18. package/src/drill-load.ts +166 -0
  19. package/src/drill-probe.ts +68 -0
  20. package/src/drill-run.ts +144 -0
  21. package/src/drill-services.ts +70 -0
  22. package/src/drill-sync.ts +108 -0
  23. package/src/drill-verdict.ts +256 -0
  24. package/src/drill-writes.ts +123 -0
  25. package/src/drills.ts +109 -0
  26. package/src/e2e-a11y-run.ts +189 -0
  27. package/src/e2e-a11y-tabs.ts +113 -0
  28. package/src/e2e-a11y.ts +153 -0
  29. package/src/e2e-driver.ts +46 -10
  30. package/src/e2e-page.ts +28 -3
  31. package/src/e2e-run.ts +1 -0
  32. package/src/e2e-spawn.ts +3 -2
  33. package/src/errors.ts +43 -0
  34. package/src/exercise-actor.ts +124 -0
  35. package/src/exercise-crash.ts +46 -0
  36. package/src/exercise-input.ts +182 -0
  37. package/src/exercise-override-input.ts +34 -0
  38. package/src/exercise-report.ts +282 -0
  39. package/src/exercise-request.ts +38 -0
  40. package/src/exercise-sweeps.ts +100 -0
  41. package/src/exercise.ts +448 -0
  42. package/src/factories.ts +10 -7
  43. package/src/factory-registry.ts +49 -15
  44. package/src/fake-identity-provider.ts +191 -0
  45. package/src/fault-proxy.ts +342 -0
  46. package/src/file-boundary.ts +4 -1
  47. package/src/fixture-jobs.ts +3 -1
  48. package/src/fixture-statements.ts +4 -1
  49. package/src/free-port.ts +59 -0
  50. package/src/index.ts +148 -3
  51. package/src/island-dom.ts +5 -0
  52. package/src/isolated-plugins.ts +1 -1
  53. package/src/job-crash-proof.ts +249 -0
  54. package/src/job-sim-driver.ts +164 -0
  55. package/src/job-sim-effects.ts +142 -0
  56. package/src/job-sim-errors.ts +72 -0
  57. package/src/job-sim-schedule.ts +154 -0
  58. package/src/job-sim-world.ts +198 -0
  59. package/src/job-simulate.ts +179 -0
  60. package/src/mock-route.ts +81 -0
  61. package/src/module-mock-boundary.ts +85 -0
  62. package/src/module-mock-error.ts +28 -0
  63. package/src/plan-oracle.ts +164 -0
  64. package/src/plan-sweep-fix.ts +123 -0
  65. package/src/plan-sweep-report.ts +84 -0
  66. package/src/plan-sweep.ts +148 -0
  67. package/src/preload.ts +24 -4
  68. package/src/registry-snapshot.ts +1 -1
  69. package/src/repro-stand-in.ts +67 -0
  70. package/src/repro.ts +158 -0
  71. package/src/request-log.ts +64 -0
  72. package/src/reusable-database.ts +1 -1
  73. package/src/scale-sweep-graph.ts +82 -0
  74. package/src/scale-sweep-run.ts +134 -0
  75. package/src/scale-sweep.ts +74 -0
  76. package/src/sealed-network.ts +63 -46
  77. package/src/seed-graph-error.ts +50 -0
  78. package/src/seed-graph-order.ts +98 -0
  79. package/src/seed-graph-pass.ts +68 -0
  80. package/src/seed-graph-plan.ts +81 -0
  81. package/src/seed-graph-row.ts +134 -0
  82. package/src/seed-graph-shape.ts +104 -0
  83. package/src/seed-graph-values.ts +135 -0
  84. package/src/seed-graph.ts +250 -0
  85. package/src/seed-writer.ts +180 -0
  86. package/src/subject-proof-error.ts +23 -0
  87. package/src/subject-proof.ts +180 -0
  88. package/src/template-db-roles.ts +51 -0
  89. package/src/template-db.ts +8 -2
  90. package/src/tenant-canary.ts +120 -0
  91. package/src/tenant-proof-report.ts +62 -0
  92. package/src/tenant-proof.ts +177 -0
  93. package/src/test-types.ts +12 -3
package/CLAUDE.md CHANGED
@@ -2,73 +2,81 @@
2
2
 
3
3
  Tier 5. May import tiers 0–4. Imported by every package's tests and by generated apps.
4
4
 
5
- Deps (15): `core`; `cache` `db` `i18n` `time`; `entity` `http` `policy`; `action` `jobs`
6
- `query` `realtime`; `mail` `pwa` `render`. Static in `.`: `cache`
5
+ Deps (16): `core` `schema`; `cache` `db` `i18n` `time`; `entity` `http` `policy`; `action`
6
+ `jobs` `query` `realtime`; `mail` `pwa` `render`. Static in `.`: `cache`
7
7
  (`registry-leak-guard.ts:7`), `i18n`, `jobs`, `policy` (`registry-snapshot.ts:29-38`), `db`
8
8
  (`reusable-database.ts:20`), `realtime/server` (`fixture-subscribe.ts:13`). Dynamic-only:
9
- `action`, `http`, `mail`, `pwa`, `query`, `render`, `time` — no `mail` destructure, no mail
10
- package; a `packages/core` import never loads `@ultimat3/entity`. `entity` exists for exactly
11
- one call, `nPlusOne()`: the strict fixture reports `x dev`'s error, with the `fix:` the schema's
12
- own relations spell — a second N+1 code here would be a second answer to one condition. Its one
13
- static import: `registry-isolation.ts`, own entry point, off the barrel.
9
+ `action`, `http`, `mail`, `pwa`, `query`, `render`, `schema` (`exercise-input.ts:16`), `time` — no `mail` destructure, no mail
10
+ package; a `packages/core` import never loads `@ultimat3/entity`. `entity` is reached by `await import` only
11
+ (seed writer, tenant witness, `nPlusOne()`, repro, subject proof, the row observer); its one static
12
+ import is `registry-isolation.ts`, own entry point, off the barrel. Full rule: `docs/history/testing.md`.
14
13
 
15
14
  | Rule | Detail |
16
15
  |---|---|
17
16
  | No mocks of the DB | clone a template database; `template-db.ts` is the only DB path |
18
17
  | No wall clock | `withFrozenClock` / `advanceClock`; `Date.now()` is frozen by the preload |
19
- | Frozen ≠ different | `globalThis.Date` becomes a subclass, so `FrozenDate[Symbol.hasInstance]` brands on the `[[DateValue]]` slot — `Date.prototype.getTime.call(value)` throws or it doesn't. |
20
- | Slot, not prototype | the brand is cross-realm on purpose: `instanceof RealDate` misses a `node:vm` or worker Date, and `Object.prototype.toString` is spoofable by `Symbol.toStringTag: 'Date'`. Only the slot is both |
18
+ | Frozen ≠ different | `globalThis.Date` becomes a subclass; `FrozenDate[Symbol.hasInstance]` brands on the `[[DateValue]]` slot (`Date.prototype.getTime.call(value)` throws or not) |
21
19
  | No unmocked egress | `sealed-network.ts` seals `fetch`; `sealed-sockets.ts` `WebSocket`, `Bun.connect` (not loopback); a miss is `X_TEST_NETWORK_SEALED` |
20
+ | One `mockFetch`, and it grows | a verb in the match, ordered answers, a pinned dial's name (`mock-route.ts`) and `requestLog` (`request-log.ts`). A stream bound for a real server logs `null` (teeing drains it). `original(input, init)` goes as handed: `tls`, `unix`, `proxy` are in no `Request`. `cassette()`: below |
22
21
  | Self is not egress | a port core's `markListening()` announced passes through — a socket test never unseals |
23
22
  | Offline is a state, not a mock | `network.offline()` / `.drop()` fail every dial, loopback too, as `X_TEST_NETWORK_OFFLINE` before any mock: the app's offline path runs |
24
23
  | One way offline | the `network` fixture. `setNetworkState` is the gate's only writer and is not exported — setting it from a test body skips the fixture's disposal and leaves every later file offline |
25
- | The seal key is a test process's, and nobody's export | `test-seal-key.ts`: installed by the preload only when `NODE_ENV === 'test'` and neither `ULTIMATE_SECRETS_KEY` nor `.secrets.key` exists; `startE2eApp` hands it to the child by VALUE (`testSealKeyEnv`): a spawned server must never install one. The constant is module-private and off the barrel (`test-seal-key.test.ts`). Never loosen the `NODE_ENV` check: the key is public |
26
- | `toEqualRow` never prints a server-only value | `matcher-row.ts` compares every OWN property (`toEqual` cannot see a `.sealed()` column — empty diff). A non-enumerable property that differs is NAMED; its value and its length stay out of the message, which lands in CI logs |
24
+ | The seal key is a test process's, and nobody's export | `test-seal-key.ts`: installed by the preload only when `NODE_ENV === 'test'` and neither `ULTIMATE_SECRETS_KEY` nor `.secrets.key` exists. Full rule: `docs/history/testing.md` |
25
+ | `toEqualRow` never prints a server-only value | `matcher-row.ts` compares every OWN property (`toEqual` cannot see a `.sealed()` column — empty diff). Full rule: `docs/history/testing.md` |
27
26
  | No retries | a flake is fixed or deleted the day it flakes; there is no `retry: 3` |
28
27
  | A mutator's `local()` runs against `memoryLocalTx()` | `local-tx.ts`: keyed rows with the record store's rules (update invents nothing, `undefined` leaves a column, rows frozen). Never a hand-rolled table per test |
29
- | A page test RENDERS, with no cast | `render-view.ts`: `renderRoute(module, { url, actor })` resolves `load` once and hands `meta` and the page that one object, as every render mode does; `renderView(Component, props)` is one component. `@ultimat3/render` is imported inside both, never at module scope: a tier-0 test imports this barrel |
28
+ | A page test RENDERS, with no cast | `render-view.ts`: `renderRoute(module, { url, actor })` resolves `load` once and hands `meta` and the page that one object, as every render mode does; `renderView(Component, props)` is one component. Full rule: `docs/history/testing.md` |
30
29
  | `toBeUltimateError` reads THREE fields | an `X_` code plus a `cause` plus a `fix`, all through core's `stringField`. `typeof value.code === 'string'` alone passed a Node `ENOENT` |
31
30
  | One matcher WAITS, and it is `toBeVisible` | `await expect(locator).toBeVisible()` retries `isVisible()` to a budget — 5000ms every 100ms, Playwright's own default, narrowable per call. |
32
- | The budget is counted in LOOKS, not milliseconds | this package freezes `Date.now()`, so a deadline computed from the clock never expires and the loop spins forever. `attemptsFor(budget)` is `1 + floor(timeout / interval)` |
33
- | A FIXED interval, and never a curve | doubling the gap makes the last look land long after the state changed, and the caller's deadline is the contract. |
34
- | A matcher that may throw a coded error MUST NOT be `async` | bun REPLACES an `async` matcher's thrown error with `returned a promise that rejected`, so code, cause and fix are gone. A synchronous prologue validates the receiver, then a promise is returned; `matcher-receiver.test.ts` pins all four |
35
- | Wrong receiver throws, wrong value returns | a matcher handed a page where a locator belongs is not FALSE, it is unanswerable — `pass: false` would read as "the element was hidden". A wrong-shaped receiver is a coded throw |
36
- | A matcher MESSAGE is a thunk, and never `JSON.stringify` | `result(pass, () => …)` — `expect.extend` reads `message()` only on the wrong verdict, and the string used to be built EAGERLY. |
31
+ | The budget is counted in LOOKS, at a FIXED interval | `attemptsFor(budget)` is `1 + floor(timeout / interval)`; never a clock deadline, never a curve. Full rule: `docs/history/testing.md` |
32
+ | A matcher that may throw a coded error MUST NOT be `async` | bun REPLACES an `async` matcher's thrown error with `returned a promise that rejected`, so code, cause and fix are gone. Full rule: `docs/history/testing.md` |
33
+ | Wrong receiver throws, wrong value returns | a wrong-shaped receiver is a coded throw, never `pass: false` |
34
+ | A matcher MESSAGE is a thunk, and never `JSON.stringify` | `result(pass, () => …)`. Full rule: `docs/history/testing.md` |
37
35
  | Every message thunk needs a test that PROVOKES it | a thunk is a function, so an unread message is an UNCOVERED function and `bun run scripts/coverage-gate.ts --package testing` is what says so |
38
36
  | Test names | the filename picks the step; a typed helper prefixes its own name. `testName(type, name)` is for an outer `describe` only — never on the inner `test` too, the prefix would print twice |
39
37
  | One body shape, five registrars | `test` (= `fixtureTest`), `unitTest`, `contractTest`, `liveTest`, `jobTest` all take `FixtureBody` + `{ timeoutMs }`: `unitTest('…', async ({ runJobs }) => …)`. Never `describe(testName('unit', …))` + `test` to get a fixture |
40
38
  | `runJobs` says who and what | `{ actor }` is the WORKER's identity (the org stays the job's declared tenant), `{ tenantId }` the enqueuer's row; `executions[n].result` is the body's return. A fresh event bus per fixture |
41
- | A `runJobs` pass is a real worker's `tick()` | `fixture-jobs.ts` builds an unstarted `jobWorker` per pass, so admission runs: `concurrency` waits and `whenBusy: 'fail'` refuses (`X_JOB_KEY_BUSY`) exactly as in a fleet. Never `executeJob` on a bare `claim()`. The lease and slot renew every 1 ms of TEST time (`RENEW_MS`, `frozen-scheduler.ts` on `onClockMoved`), never on a wall-clock interval: cancel, `clock.advance(1)`, and the body hears `X_JOB_LEASE_LOST` — no hand-built worker |
39
+ | A `runJobs` pass is a real worker's `tick()` | `fixture-jobs.ts` builds an unstarted `jobWorker` per pass, so admission runs: `concurrency` waits and `whenBusy: 'fail'` refuses (`X_JOB_KEY_BUSY`) exactly as in a fleet. Full rule: `docs/history/testing.md` |
42
40
  | Per-test state is `per-test-reset.ts` | the app preload's `beforeEach` resets the jobs event bus — only when `@ultimat3/jobs` is already loaded, so the reset never loads it |
41
+ | A `mock.module` ends with its file | `module-mock-boundary.ts` wraps it (and `vi.mock`/`jest.mock`): the real namespace is snapshotted on a file's first mock, so its top-level code RUNS, and mocked back at the boundary. A preload's mock is left. A real one that throws on load: the next file fails `X_TEST_MODULE_MOCK_UNRESTORABLE` |
42
+ | **The frozen clock is reset at every file boundary** | `preload.ts`, `clock-file-boundary.test.ts`: `advanceClock` / `setFrozenClock` move one process-wide instant, and a file that left it moved handed the next file on its worker a different `Date.now()` |
43
43
  | An island test is `describeIslandState` | one block per declared state: mounted in `beforeAll` (60 s), disposed in `afterAll`, one build per file. `mountIslandState` + `using` for one test's own mount; an undeclared id is `X_TEST_ISLAND_STATE_UNKNOWN` |
44
44
  | An authenticator is tested with `authRequest()` | the hook's own two arguments, real — never `as unknown as Parameters<…>` |
45
45
  | `LOG_LEVEL` above `info` filters the TERMINAL | `quiet-logs.ts` pins the logger at `info` while core loads, so a collecting `setLogSink` still sees `warn` under `LOG_LEVEL=error`. It stays the preload's FIRST import |
46
46
  | Injection | `SqlRunner` and `connect` are parameters, so unit tests need no server |
47
47
  | Fixtures | the preload registers the framework bag — an app registers only what it alone knows (`seed`, `actorFor`) |
48
+ | Every `e2eTest` is audited | `e2e-a11y.ts` (rules), `e2e-a11y-run.ts` (per-test run, `allow`, dedupe), `ax-node.ts` (the one AX reader); the full tree is read only in `cdp-e2e-page.ts` (`FULL_AX_TREE`, a convention). No switch; `allow` needs a non-blank `because`. And every raw tab: `e2e-a11y-tabs.ts`, checked after each `goto`/`reload` and at the test's end from the preload's `afterEach`; exceptions are `allowA11y([{ role, selector, because }])` |
49
+ | The exercise pass runs what is registered | `exercise.ts` (`runExercisePass`): every action, query and job once, its own tenant pair off one `seedGraph`, an actor `exercise-actor.ts` picks, 39's `runExercise` as the verdict. Rows COMMITTED. Overrides only via `defineExercise` (blank `off` refused) |
50
+ | An override `input` is awaited, as the call's actor | `exercise-override-input.ts`: run inside the call's own request, so `useContext().actor` is the seated actor; the one way to a signed input. `async` ones are left out of `x drill`'s load |
51
+ | A passkey test signs for real | `softAuthenticator` is `@ultimat3/auth/soft-authenticator`'s, never re-exported here (one value, one path: `flight-copies`) and never a second one (auth's own suites need it, and auth cannot import testing) |
52
+ | A reached job is crash-proven | `exercise-crash.ts`: `crashProof` with the control's input and actor, after the tenant proof; only `diverged` is red (`X_JOB_CRASH_DIVERGED`). Committed: job bodies escape transactions |
53
+ | The tenant proof is taint only | `tenant-proof.ts`: B, A, B; B is A's role minus `tenancy:cross`. Leak = seeded canary or unhanded id (`tenant-canary.ts`); write = moved digest. Only `crossed` is red |
54
+ | Every statement the pass sent is planned | `plan-sweep.ts` (`sweepPlans`): first relation an app table, `PREPARE` in a READ-ONLY transaction with seq/bitmap scans and sorts off, judged by `plan-oracle.ts`; red = `X_QUERY_UNINDEXED` / `X_QUERY_SORT_UNSERVED` with the `indexes:` entry (`plan-sweep-fix.ts`). `scan:` or `expectedSeqScan` is exempt, listed; no generic plan is `X_QUERY_PLAN_UNAVAILABLE`, never a pass |
55
+ | A shape that grows with the rows is red | `scale-sweep-run.ts`: two more tenants at `SCALE_SIZES` (2, 8 rows a table), every reached primitive once in each; `judgeScale` (pure) reds `X_QUERY_SCALES_WITH_ROWS`. Under `expectedQueryLoop` it is listed with its reason |
56
+ | An erase is PROVED, not asserted | `subject-proof.ts` (`subjectProof`, run by cli's `exercise-subject.ts`): canaries in every table, one person erased, every stored row searched; a survivor outside `retain` is `X_SUBJECT_ERASE_INCOMPLETE`, naming table and column, never the value |
57
+ | A captured failure replays as a test | `repro.ts`: `reproduce(bundle, { driver })` seeds the rows it read, freezes the clock at its instant, invokes as its actor; masked fields are stood in (`repro-stand-in.ts`). An unregistered name ANSWERS as the failure. `reproJob(runJobs, failure)` for a dead job. What `x repro --write` emits |
58
+ | A vendor's answers are a cassette | `cassette.ts`: replay = one `mockFetch` route per recorded URL, seal kept (`X_CASSETTE_MISSING`, `X_CASSETTE_UNUSED`). Record = `ULTIMATE_CASSETTE=record` + `record.hosts`, refused under `CI` (`X_CASSETTE_RECORD_REFUSED`), headers allow-listed, credential keys masked, bytes scanned before disk (`X_CASSETTE_SECRET`). The request side is never written |
59
+ | One seed writer | `entitySeedWriter({ entities, client \| driver })` (`seed-writer.ts`) writes through each entity's own repository; `snapshot` is required; `entity` loads on the first write, so the barrel stays entity-free |
60
+ | Factory values are DECLARED | `factoriesFor` and `seedGraph` share `seed-graph-values.ts`: default, enum, kind, through `$parse`, never the name. `seed-graph*.ts` type-import `entity` only |
48
61
  | e2e without a driver | `e2eTest` becomes `test.skip`; the gate reports the step SKIPPED (or `X_VERIFY_SUITE_VANISHED` when `x.verify.json` requires it), never passed |
49
62
  | The seam has an inverse | `resetE2eDriver()`: `useE2eDriver` writes MODULE scope, so a file that installs a browser undoes it |
50
63
  | Built vs declared | `clock` `mail` `network` `runJobs` `statements` `subscribe` are built in-process; `page` `budget` `signIn` `deploy` are declared and wait for a driver (`X_TEST_FIXTURE_UNAVAILABLE`). |
51
64
  | `page` HAS a driver; `budget`, `signIn`, `deploy` do not | `installE2eDriver({ page, baseUrl })` registers `page` only. Byte counts, a sign-in route and a new build id are facts no page port can answer; a no-op would read as proof |
52
- | `network` is THIS process's fetch, and an e2e page is not in this process | `network.offline()` beside `page` is a no-op on the browser. `E2eFixtures.offline()` is the browser-side spelling and forwards to `E2eBrowserPage.offline()`; a page port with none refuses by name |
53
- | `subscribe` is a whole `sync` node | `live-node.ts` assembles what `x dev --role sync` assembles minus the listener — real `LiveQueryRegistry`, real `liveQueryDefinition` bridge, real per-subscriber gate, real cursor |
54
- | What `subscribe` does NOT hold | a client store, an offline queue or a rebase log — so `feed.local()` answers `undefined` rather than the server row. |
55
- | Draining is a macrotask yield | the node dispatches `message` into a floating async task, so `settled()` yields with `setImmediate` until the frame count stops moving. |
56
- | A lone subscriber cannot resume | the retained window is the registry's ENTRY and the entry is dropped when its last subscriber goes, so a reconnect with nobody else holding it re-snapshots — correctly. |
57
- | Strict is opt-in by destructuring | `statements` installs the N+1 detector in throw mode for one test. A fixture nobody names is a fixture nobody built, so there is no `strict: true` and no suite-wide switch |
58
- | One threshold, one error | `N_PLUS_ONE_THRESHOLD` and `nPlusOne()` are `@ultimat3/entity`'s. A number or a message written here would make a loop that fails a test a different loop from the one `x dev` warns about |
65
+ | `network` is THIS process's fetch, and an e2e page is not in this process | `network.offline()` beside `page` is a no-op on the browser. Full rule: `docs/history/testing.md` |
66
+ | `subscribe` is a whole `sync` node, minus the listener | `live-node.ts`; no client store, offline queue or rebase log (`feed.local()` is `undefined`); `settled()` yields by `setImmediate`; a lone subscriber re-snapshots. Full rule: `docs/history/testing.md` |
67
+ | Strict is opt-in by destructuring | `statements` installs the N+1 detector in throw mode for one test. No `strict: true`, no suite-wide switch |
68
+ | One threshold, one error | `N_PLUS_ONE_THRESHOLD` and `nPlusOne()` are `@ultimat3/entity`'s, never a copy |
59
69
  | The unit of work is the test | `x dev`'s ledger tallies per `Ctx` and ignores a statement issued outside a request |
60
- | Throws once per shape | the failing line is the loop's own statement (the seam lets `onStatement` throw for this reason alone). |
61
- | Measure vs judge | `all()` `count()` `shapes()` count `expectedQueryLoop` statements too; only the verdict honours the suppression, so "this page issues two statements" never depends on who declared what |
70
+ | Throws once per shape | the failing line is the loop's own statement (why `onStatement` may throw) |
71
+ | Measure vs judge | `all()` `count()` `shapes()` count `expectedQueryLoop` statements too; only the verdict honours the suppression |
62
72
  | One seam for drivers | a driver registers over a declaration with `defineFixtures` — merges, last wins. Never a second registration mechanism |
63
- | A driver arrives whole | `defineFixtures` holds every name `Fixtures` declares to its declared type, so a half-built `page` is a compile error at the registration, not a missing method three awaits later |
73
+ | A driver arrives whole | `defineFixtures` holds every name `Fixtures` declares to its declared type, so a half-built `page` is a compile error at the registration |
64
74
  | Registry hygiene | the fixture registry is process-global; a test that clears it snapshots with `fixtureSnapshot()` and hands it back in `afterAll` |
65
75
  | Leaks are the file's, not the next file's | `installRegistryLeakGuard()` runs from the preload and fails the run naming the FILE that left cache tags declared or a cache tier registered after its last test (`X_TEST_REGISTRY_LEAK`). |
66
- | The baseline is not a hook | measured on Bun 1.3.14 the order is onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload `beforeEach`, so a preload hook cannot sample before the file's own `beforeAll` |
67
76
  | Reported and restored are different sets | the guard also RESTORES, at the same file boundary, the registries whose module-scope declarations a neighbour's cleanup destroys |
68
77
  | A catalog restore is a MERGE, never a replace | the other three registries are replaced with the snapshot. `registerCatalog` has no inverse, so all a file can cost its neighbour is a `resetCatalogs()` |
69
- | Permissions are still REPLACED at a file boundary, and it is measured | an app's `definePermissions()` reached only by a dynamic `loadApp()` is dropped at that file's boundary exactly as catalogs were; the union rule would leak `permissions.test.ts`'s declarations into every later file. Open, its own piece of work (history) |
70
- | Guarded state is boot state | only the two registries whose honest invariant is "clean when the file ends" — `declareTags` and `registerTier` are boot installs. |
71
- | Filled and CLEARED are different questions | and the row above answers only the first. "Idiomatic to leave filled" is about the leak REPORT |
78
+ | Permissions are still REPLACED at a file boundary, and it is measured | a dynamic `loadApp()`'s `definePermissions()` is dropped there. Open. Full rule: `docs/history/testing.md` |
79
+ | Guarded state is boot state | the guard covers only registries that must be clean at file end — `declareTags` and `registerTier` are boot installs; "filled" and "cleared" are two questions. Full rule: `docs/history/testing.md` |
72
80
  | An empty registry is a premise you state | a test whose subject is "nothing is declared" — `x db gen` with nothing to generate — calls `isolateEntityRegistry()` and restores in a `finally`. |
73
81
  | That one helper is off the barrel | `@ultimat3/testing/registry-isolation`, its own entry point. It is the only module here that value-imports `@ultimat3/entity` |
74
82
  | Teardown restores, never uninstalls | `describeApp`/`testApp` capture the seal and the determinism snapshot before booting and put those back |
@@ -82,41 +90,31 @@ static import: `registry-isolation.ts`, own entry point, off the barrel.
82
90
  | Factory seeds | derived from the table name unless given, so two entities never draw the same uuid stream. `reset()` cascades into associated parents — a half-reset row is worse than none |
83
91
  | Shared examples | `behavesLike` calls `describe`, so it goes at declaration scope; bun rejects a `describe` inside a test body |
84
92
  | A port's contract is shared examples | `jobDriverConformance`, `budgetStoreConformance` — `behavesLike(…, () => yours)` |
85
- | An island needs a BUILDER, not an import | `buildIslands` is `@ultimat3/cli`'s and both packages are tier 5; the one declared edge is `cli → testing`, so the reverse is a `bun run boundaries` failure. |
93
+ | An island needs a BUILDER, not an import | `buildIslands` is `@ultimat3/cli`'s; `testing → cli` fails `bun run boundaries` |
86
94
  | `mountIsland` AWAITS `mount` | `IslandEntry['mount']` returns `unknown`, not `void`, and the call is awaited |
87
95
  | Dispose STOPS the island, then restores the globals | when `mount` resolved to a function, `[Symbol.dispose]` calls it BEFORE `restore()` |
88
- | The micro-DOM is the fixture's, once **for islands** | `island-dom.ts`. `packages/ui/src/fake-dom-fixture.ts` is a second one for keyboard code, and `ui -> testing` is upward: not a copy to collapse. `bun test` has no DOM and no DOM library may be added |
89
- | `style` and `classList` RECORD | `FakeStyle` is one declaration map behind all four spellings compiled Solid uses (static attribute, `setProperty`, `cssText`, `removeAttribute`), so a test can assert the component set `--form-gap` |
90
- | `classList` is the class attribute | not a list of its own, so `classList.toggle` — which the compiler emits INLINE for `classList={{ … }}`, no runtime helper between — and `className` can never answer one element two ways. |
91
- | A `document` listener is the documentElement's | no bubbling: `document.addEventListener` registers on `documentElement`, and `fire(mounted.documentElement, 'keydown', …)` drives it. Every listener per type runs; removal is by function identity |
92
- | `querySelector` skips `this` | descendants only, as the DOM's does. Matching the element it is called on made a host `<div>` answer `find('div')` with the container the test built rather than the markup the island rendered |
96
+ | The micro-DOM is the fixture's, once **for islands** | `island-dom.ts`. Full rule: `docs/history/testing.md` |
97
+ | The micro-DOM records and does not pretend | `style`/`classList` record, a `document` listener is the documentElement's, `querySelector` skips `this`, the box is 0 until written, `ResizeObserver` never fires itself, attaching a node moves it. Full rule: `docs/history/testing.md` |
93
98
  | The selector grammar is SMALL and REFUSES | `island-selector.ts`: compounds of tag, `#id`, `.class`, `[attr]`, `[attr="value"]`, joined by space or `>`. Anything else is `X_TEST_ISLAND_SELECTOR_UNSUPPORTED` with the offset, never an empty answer |
94
- | The box is 0 until a test writes it | `clientHeight`, `scrollTop`, `scrollHeight` and the rest are writable numbers, default 0; `getBoundingClientRect()` derives from the offset box |
95
- | `ResizeObserver` records and never fires on its own | `island-observers.ts`, one registry PER DOCUMENT. |
96
99
  | A mount installs process globals | so `MountedIsland` is `Disposable` and a `mount` that THROWS restores before it rethrows. A fake `document` left installed reaches every later FILE in the run |
97
100
  | `fire` answers whether a handler ran | a selector matching nothing and an island that attached no handler are the same silence otherwise — the second is a bug, the first a typo. |
98
101
  | A states file is PURE DATA, and it is enforced | `defineIslandStates` declares the states an island can be photographed in — error, empty, over-quota, read-only |
99
102
  | The rule is the RELATIVENESS, not the extension | `import { X } from './settings.island'` resolves to `./settings.island.tsx` under Bun, so any relative runtime import is refused |
100
103
  | `import type` is not an import, and it is the one way to name the component | `verbatimModuleSyntax` erases a statement that BEGINS `import type` / `export type` |
101
- | Unreadable is not pure | a computed specifier — ``import(`./${name}.island`)``, `require(SPEC)` — is refused as `IslandStatesOpaqueImportError` rather than passed. |
104
+ | Unreadable is not pure | a computed specifier is `IslandStatesOpaqueImportError` |
102
105
  | What the scan does NOT follow | a BARE specifier other than `solid-js`, an ABSOLUTE path (both holes), and a specifier inside a string LITERAL (a false refusal, the safe direction) |
103
- | Props are JSON or they are refused | they ride `data-x-props`, which `@ultimat3/render`'s `emitIslandProps` `JSON.stringify`s, so anything else is a prop the component never receives. |
106
+ | Props are JSON or they are refused | they ride `data-x-props` (`emitIslandProps`). Full rule: `docs/history/testing.md` |
104
107
  | The clock is pinned in the vocabulary, zone included | `timeZone` defaults to `ISLAND_SHOT_TIME_ZONE` (`UTC`) and `now` to this package's own `DEFAULT_NOW`, and both ride onto every `IslandShotTarget`. |
105
- | Loose in, strict out | `findIslandStates` resolves `Settings`, `settings`, `settings.island.tsx` and the full path to one manifest, and refuses a name nothing answers to by listing EVERY valid one |
106
- | The disk check is not in `defineIslandStates` | a declaration evaluates wherever it is imported from, so a rule that reads the filesystem at import time fails on the cwd rather than on the path. |
107
- | The island EXTENSION is not restated here | `.island.tsx` is `@ultimat3/render`'s `ISLAND_EXTENSION` and `render` is not a dependency of this package. |
108
- | A mount imports from its own temp DIRECTORY | `island-scratch.ts`: per mount (identical chunks stay two modules); removed on dispose, at the file boundary and in the run's `afterAll` — `exit` never fires under `bun test`. Never a `data:` URL (coverage panics past ~4 kB) |
109
- | Attaching a node MOVES it | `appendChild`, `insertBefore`, `replaceChild` detach it from its old parent first; `removeChild` clears `parentNode` |
108
+ | Loose in, strict out | `findIslandStates` resolves any spelling of one island; an unknown name lists EVERY valid one |
109
+ | The disk check is not in `defineIslandStates` | Full rule: `docs/history/testing.md` |
110
+ | The island EXTENSION is not restated here | `ISLAND_EXTENSION` is `@ultimat3/render`'s |
111
+ | A mount imports from its own temp DIRECTORY | `island-scratch.ts`, never a `data:` URL. Full rule: `docs/history/testing.md` |
110
112
  | Globals install in layers | `island-globals.ts`: a key's real DESCRIPTOR is saved by the first live mount, restored by the last, in any dispose order |
111
113
  | Which command shards | `bun test` is one process on one database — what a scaffolded app's `test` script runs |
112
114
 
113
115
  ## The frozen instant and the seed are screened
114
116
 
115
- `installDeterminism({ now })`, `setFrozenClock`, `withFrozenClock` and `advanceClock` go through one
116
- `instantMs`: an unreadable instant would make `Date.now()` answer `NaN` for every later file. The
117
- seed is screened too — `seed >>> 0` maps `NaN`, `0.5` and `2 ** 32` onto seed 0 silently.
118
- `finiteOption`/`finiteCount` from `@ultimat3/core` are the one form; `determinism-bounds.test.ts`
119
- holds both sides.
117
+ Through one `instantMs` and core's `finiteOption`/`finiteCount` (`determinism-bounds.test.ts`). Full rule: `docs/history/testing.md`.
120
118
 
121
119
  Commands: `bun test`, `bunx tsc --noEmit -p tsconfig.json`.
122
120
 
@@ -124,17 +122,12 @@ Entry points: `.` (the API), `./preload` (side effects for bunfig) and `./regist
124
122
  (`isolateEntityRegistry()`, kept off `.` because it loads `@ultimat3/entity`).
125
123
 
126
124
  `fixture-island.ts`, `island-dom.ts`, `island-selector.ts` and `island-observers.ts` are on `.` and
127
- import `@ultimat3/core` and nothing else, so they cost a tier-0 test nothing. The whole-chain proof — real Babel, real Solid, a real island —
128
- is `examples/dummy/apps/web/app/settings/settings.island.test.ts`; `fixture-island.test.ts` pins
129
- this package's own contract against modules written in the idiom `babel-preset-solid` emits, with
130
- no bundler, because a test here cannot import one.
125
+ import `@ultimat3/core` and nothing else, so they cost a tier-0 test nothing. The whole-chain proof is
126
+ `examples/dummy/apps/web/app/settings/settings.island.test.ts`. Full rule: `docs/history/testing.md`.
131
127
 
132
128
  ## The browser-backed e2e driver (moved here from `@ultimat3/cli` in 22.0.0)
133
129
 
134
- It was in `@ultimat3/cli`, on the argument that the adapter joined `testing`'s `PageLike` to
135
- `@ultimat3/scraping`'s browser and only `cli` could import both. That stopped being true when the
136
- browser became RAW CDP (below): the driver needs `@ultimat3/core` and Bun, nothing else, so it
137
- lives beside the `PageLike` it implements. `cli` imports it over the declared `cli -> testing` edge.
130
+ Raw CDP: `@ultimat3/core` and Bun, nothing else; `cli` imports it over `cli -> testing`. Full rule: `docs/history/testing.md`.
138
131
 
139
132
  | File | Job |
140
133
  |---|---|
@@ -149,9 +142,9 @@ lives beside the `PageLike` it implements. `cli` imports it over the declared `c
149
142
  | `cdp-launch.ts` | which Chrome, its flags, the launch deadline (`LAUNCH_TIMEOUT_MS`, 60 s — a cold start is not a call) and the ONE relaunch (`LAUNCH_ATTEMPTS`) |
150
143
  | `cdp-launch-attempt.ts` | one start in its own group; on no answer a full reap (group, then profile + singleton dir: `cdp-launch-profile.ts`). ONE awaited `close()` |
151
144
  | `cdp-connection.ts` | CDP over a transport — the launched Chrome's pipe (`cdp-pipe.ts`), a remote one's `WebSocket` (`cdpConnect`): reply correlation by `id`, event waiters, the per-call deadline; why one went unanswered is `cdp-wire-watch.ts` |
152
- | `cdp-e2e-session.ts` | the BROWSER half, `E2eSession`: every target auto-attached at browser level and PAUSED until its Network domain is on (a SharedWorker opens its socket at start-up) |
153
- | `cdp-e2e-page.ts` | one TAB, `E2eTab`: `E2eBrowserPage`'s five methods plus `reload`, `waitFor`, `indexedDbNames`, `close`. `offline()` forwards to the session — the switch is browser-wide |
154
- | `e2e-app.ts` | `startE2eApp({ root, mode, seed })`: reset + seed + spawn on a THROWAWAY `ULTIMATE_STATE_DIR` and free ports a browser opens (`free-port.ts`), `/readyz`-gated, spawned through the app's own `@ultimat3/cli` bin (`xBin`); `stop()` removes the directory |
145
+ | `cdp-e2e-session.ts` | the BROWSER half, `E2eSession`: every target PAUSED until its Network domain is on |
146
+ | `cdp-e2e-page.ts` | one TAB, `E2eTab`: `E2eBrowserPage`'s six methods (`accessibilityTree` required) plus `reload`, `waitFor`, `indexedDbNames`, `close`. `offline()` forwards to the session — the switch is browser-wide |
147
+ | `e2e-app.ts` | `startE2eApp({ root, mode, seed })`: a THROWAWAY `ULTIMATE_STATE_DIR`, `/readyz`-gated; `stop()` removes it. Full rule: `docs/history/testing.md` |
155
148
  | `failure-context.ts` | a FAILED test's message ends with the e2e app's last 40 log lines (≤ 4 kB): `e2e-run.ts` installs `app.log()`, `fixtureTest` and `e2eTest` append it. Never on a pass |
156
149
  | `e2e-preload.ts` + `e2e-browser-handle.ts` | the `e2e` step's preload (`@ultimat3/testing/e2e-preload`): with `ULTIMATE_E2E_ROOT` set it spawns the app, opens ONE browser, installs its first tab, and publishes `e2eBrowser()`, `e2eApp()`, `e2eBaseUrl()` |
157
150
  | `packages/cli/src/verify-e2e.ts` | `withE2eApp` (cli's): in an app on a machine with Chrome, the `e2e` step runs the suite with that preload; otherwise exactly as before |
package/README.md CHANGED
@@ -3,6 +3,18 @@
3
3
  The harness. Never mock the database — clone it. Never assert on wall-clock time — advance the
4
4
  frozen clock. Never let a test reach the network unmocked — it fails by design.
5
5
 
6
+ `bun add @ultimat3/testing` · guide: [Testing](https://github.com/developerz-ai/ultimate/wiki/Testing)
7
+
8
+ ```ts
9
+ import { expect, unitTest } from '@ultimat3/testing';
10
+
11
+ unitTest('the clock moves only when the test moves it', ({ clock }) => {
12
+ const before = clock.now().getTime();
13
+ clock.advance('1h');
14
+ expect(clock.now().getTime() - before).toBe(3_600_000);
15
+ });
16
+ ```
17
+
6
18
  ## What it owns
7
19
 
8
20
  | Module | Owns |
@@ -13,7 +25,34 @@ frozen clock. Never let a test reach the network unmocked — it fails by design
13
25
  | `sealed-network.ts` | any unmocked `fetch`, `WebSocket` or `Bun.connect` egress fails, with the URL and the line that fixes it |
14
26
  | `sealed-sockets.ts` | the `WebSocket` and `Bun.connect` patches, behind `sealed-network.ts`'s one gate |
15
27
  | `factories.ts` | `defineFactory` — seeded rows, traits, associations, `build` vs `create` |
16
- | `factory-registry.ts` | `factoriesFor(registry)` — one factory per entity, defaults read off column names |
28
+ | `factory-registry.ts` | `factoriesFor(registry)` — one factory per entity, each value read off its column's declaration (default, enum, kind, money, nullability) through `$parse`, never off its name |
29
+ | `seed-graph.ts` | `seedGraph(entities, { tenants, perTable, seed, factories })` — rows for every table with no factory written: parents first, same-tenant keys, every declared rule kept. Pure, same seed same bytes |
30
+ | `repro.ts` | `reproduce(bundle, { driver })` and `reproJob(runJobs, failure)` — what a test `x repro --write` generated calls: captured reads seeded, clock frozen at the failure, the same actor and input (masked fields stood in from the schema); a dead job's completed steps replayed from storage |
31
+ | `seed-graph-plan.ts` | the plan's types and `insertSeedGraph(graph, writer)` — every insert, then every deferred key, sequentially |
32
+ | `seed-graph-values.ts` | one column's candidate values, off its declaration, kept only where `$parse` accepts them — `factoriesFor`'s source too |
33
+ | `seed-graph-order.ts` · `seed-graph-row.ts` · `seed-graph-shape.ts` | insert order over `$references()` (a cycle broken at a nullable key) · one row solved against invariants and unique keys · what the graph reads off an entity, structurally |
34
+ | `seed-graph-error.ts` | `X_SEED_GRAPH_UNSATISFIED`, its fix naming the `defineFactory` to write; `X_SEED_GRAPH_ENTITY_UNKNOWN`, a plan and a writer built from two entity sets |
35
+ | `seed-writer.ts` | `entitySeedWriter({ entities, client })` or `({ entities, driver? })` — the `SeedWriter` that writes a graph through each entity's own repository: Postgres or PGlite through a client, or a driver. `@ultimat3/entity` loads on the first write. `snapshot(rows)`, required on every `SeedWriter`, reads rows back as one digest each (`md5(t::text)` over a client), `null` for a row gone |
36
+ | `exercise.ts` | `runExercisePass({ entities, writer, roles, seed?, overrides?, only? })` — every registered action, query and job once, each on its own committed tenant pair, one `ExerciseRecord` per primitive, the tenant proof's verdict on its `tenant`. What `x exercise` runs |
37
+ | `define-exercise.ts` | `defineExercise({ <name>: { actor, input?, runs? } \| { off: '<why>' } }, { actor?: <entity> })` — an app's overrides (the second argument names the entity whose rows are the actors), `apps/web/exercise.ts`'s one export; a blank `off` is `X_CONTRACT_EXERCISE_UNDECLARED`, an unknown name `X_DECLARATION_UNKNOWN` when the pass runs |
38
+ | `exercise-actor.ts` · `exercise-input.ts` · `exercise-report.ts` | the first `x policy explain` actor the policy allows, minted in the primitive's tenant · the input: override, then a seeded id by field stem, then the minimal draw (`@ultimat3/schema/sample`, loaded dynamically) · the room one run executes in (network sealed, mail captured, clock frozen) and its record, `overrideLine` the entry to paste |
39
+ | `tenant-proof.ts` | internal (`CROSS_TENANT_SCOPE` is `@ultimat3/entity`'s): `proveTenant` — B with A's ids, A's own call, B again; each B leg between two snapshots of A's rows. `withoutCrossTenant` strips `tenancy:cross` (`CROSS_TENANT_SCOPE`) from B; `publicReason` decides `public` |
40
+ | `tenant-canary.ts` | internal: `tenantTaint` — one tenant's rows, ids and marks (row canaries, key parts of `MIN_TAINT_ID_LENGTH` 8 or more no other tenant shares); `findTaint` — every path holding one |
41
+ | `exercise-crash.ts` | internal: `proveCrash` — a job the pass reached, run through `crashProof` with its control's input and actor, after the tenant proof; the `CrashProof` on `ExerciseRecord.crash` |
42
+ | `scale-sweep.ts` · `scale-sweep-graph.ts` · `scale-sweep-run.ts` · `exercise-sweeps.ts` | `judgeScale(small, large)`, pure · one tenant at a size, its rows of no tenant pointed at the pass's own · the two sized runs · both sweeps after the pass, onto each record's `scale` and `plans` |
43
+ | `plan-oracle.ts` · `plan-sweep.ts` · `plan-sweep-fix.ts` · `plan-sweep-report.ts` | `judgePlan(answer, tables)`, pure: the red nodes · `sweepPlans(client, sent, entities)`: `PREPARE` + `explain execute` in a read-only transaction · verdict to `indexes:` line · the coded findings `x exercise` prints |
44
+ | `job-crash-proof.ts` | `crashProof(handle, input, { context?, seed? })` — one clean run, then the worker killed at every step boundary it crossed, then each answer lost; `{ boundaries, divergences, reruns }`, a divergence carrying its `X_JOB_CRASH_DIVERGED` |
45
+ | `job-simulate.ts` | `simulateJobs({ seed?, seeds, jobs, workers, faults, invariant })` — seeded races on the shared simulated driver; `X_JOB_SIM_INVARIANT`, `X_JOB_SIM_NONDETERMINISTIC`; `x test job --seed <n>` replays one |
46
+ | `fault-proxy.ts` | `faultProxy({ target })` — a Bun TCP proxy whose mode switches mid-run: `pass`, `delay(ms)`, `blackhole` (bytes held, never dropped), `reset`, `refuse` |
47
+ | `drills.ts` · `drill-verdict.ts` | the eight fault drills (`x drill`) and the verdict over evidence alone; a property with no evidence is red, and an answer may reach the load `DRILL_ANSWER_SLACK_MS` (500) past the deadline it was given by |
48
+ | `drill-fleet.ts` · `drill-run.ts` | `startDrillFleet` — `web`, `worker` (and `sync` for `nats-down`) as processes on a scratch database, every service behind a proxy; `runDrill` — baseline, fault, recovery, judged |
49
+ | `drill-load.ts` · `drill-writes.ts` | the load's lanes; `drillWriteTargets` — every action an anonymous POST reaches, each request a fresh seeded draw with the app's `exercise.ts` override, so a job enqueued over HTTP is on the table for `worker-sigkill`/`worker-sigterm` |
50
+ | `drill-sync.ts` | `startDrillSync` — the sync-client lane: `hello`, resubscribe, redial on a drop, every `replay-gap` counted; `sync-resumes` reads it |
51
+ | `tenant-proof-report.ts` | `tenantProofFindings` — one coded finding per crossing (`X_TENANT_LEAK`, `X_TENANT_WRITE_CROSSED`), `tenantFindingLines` the human lines. Paths, tables and seeded ids, never a bound value |
52
+ | `e2e-a11y.ts` | the accessibility audit's rules: `NAMED_ROLES`, `auditTree`, the in-page probe for `<html lang>` and focus under `aria-hidden` |
53
+ | `e2e-a11y-run.ts` | one `e2eTest`'s audit: read after every navigation and at the end, a reasoned `allow`, findings deduplicated per suite, `X_E2E_A11Y_VIOLATION` |
54
+ | `e2e-a11y-tabs.ts` | the same audit on every raw tab a session builds, failed from the preload's `afterEach`; `allowA11y([{ role, selector, because }])` is its exception |
55
+ | `ax-node.ts` | the one reader of the browser's accessibility nodes — `toAxNode` (one element, `x shot`'s) and `toAxTreeNodes` (a whole tree, the audit's) |
17
56
  | `factory-persist.ts` | `usePersister` — the one seam `create()` writes through |
18
57
  | `shared-examples.ts` | `sharedExamples` / `behavesLike` — one rule, many subjects |
19
58
  | `test-types.ts` | the six test types and their helpers — four of them take the fixture bag, as `test` does |
@@ -200,7 +239,7 @@ there is no flag on the fixture and no code to silence.
200
239
  | `contractTest` | OpenAPI diff vs the committed spec, MCP exposure | `contract` |
201
240
  | `liveTest` | exactly what each subscriber receives | `live` |
202
241
  | `jobTest` | step sequence, retries, idempotency | `job` |
203
- | `e2eTest` | a browser driver; with none registered it SKIPS, and the gate's `e2e` step passes over the skip — ask `hasE2eDriver()` rather than reading that as a pass | `e2e` |
242
+ | `e2eTest` | a browser driver; with none registered it SKIPS, and the gate's `e2e` step passes over the skip — ask `hasE2eDriver()` rather than reading that as a pass. Every one audits the page's accessibility tree; the only exception is `{ a11y: { allow: [{ role, selector, because }] } }` ([Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#the-accessibility-audit)) | `e2e` |
204
243
  | `evalTest` | LLM output scoring against a threshold | `eval` |
205
244
 
206
245
  Each takes an optional `{ timeoutMs }` after the body — `unitTest(name, body, { timeoutMs: 30_000 })`,
@@ -262,10 +301,80 @@ await posts.with('published').create(); // org written first, then the post
262
301
  | **seeded per table** | the default seed is derived from the table name, so a post and an org never draw the same uuid. Pass `seed` only to replay an older recording |
263
302
  | **`with()` validates** | an undeclared trait fails at the line that named it, listing the declared ones (`X_TEST_FACTORY_TRAIT_UNKNOWN`) |
264
303
 
265
- `factoriesFor(registry)` builds one factory per registered entity, with values inferred from column
266
- names (`…Id` → uuid, `…At` → date, `…Minor` → integer, `is…`/`has…` → false). Enough for the rows a
304
+ `factoriesFor(registry)` builds one factory per registered entity, each value read off the
305
+ column's declaration — its default, an enumerated value, its kind, `null` only where nullable — and
306
+ kept only if the column's `$parse` accepts it; never off its name. A non-`entity()` column, or one
307
+ that refuses every generated value, is `X_SEED_GRAPH_UNSATISFIED` at `build()`. Enough for the rows a
267
308
  test does not care about; `defineFactory` is for the rows it does.
268
309
 
310
+ `seedGraph(entities, options)` is the same value source over a whole schema: every table, parents
311
+ first, each child keyed to a parent of its own tenant, a cycle broken at a nullable key and set by an
312
+ update. It returns a plan; `insertSeedGraph(graph, writer)` writes it, and `entitySeedWriter` is
313
+ the writer over the app's own repositories. Rules and options:
314
+ [Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#a-whole-graph-seedgraph).
315
+
316
+ `runExercisePass` is that graph put to work: every registered action, query and job, called once as
317
+ an actor its policy allows, on rows it did not have to write. `x exercise` and the gate's `exercise`
318
+ step drive it; an app only writes `apps/web/exercise.ts`'s `defineExercise({ … })` for what the pass
319
+ cannot guess. [Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#the-exercise-pass).
320
+
321
+ An override's `input` is awaited, and runs inside the request its call runs in, so
322
+ `useContext().actor` is the seated actor that call acts as. That is the one way to hand the pass an
323
+ input no draw can forge — a challenge issued to THIS actor, signed by `softAuthenticator` — computed
324
+ from the pass's own state, per call, the scale sweep's included. Declare it `async`: `x drill`'s load
325
+ has no pass behind it, so it leaves an `async` input out and says why (`drill-writes.ts`). Postly's
326
+ passkey entries are the worked example (`examples/dummy/apps/web/exercise.ts`).
327
+
328
+ Around each call the pass runs the tenant proof: the pair's second tenant (B, same role, no
329
+ `tenancy:cross`) sends A's own input before A's call and again after it. A canary or unhanded id of
330
+ A's in B's output, mail, outbound body or enqueued job input is `X_TENANT_LEAK`; a digest of A's
331
+ seeded row that moved across a B leg is `X_TENANT_WRITE_CROSSED`. Both are red in `x exercise`.
332
+ `ExerciseRecord.tenant` is a `TenantProof`:
333
+
334
+ | `verdict` | When — first match wins |
335
+ |---|---|
336
+ | `crossed` | any `TenantFinding` (`code`, `leg: 'before' \| 'after'`, `path` or `table` + `id`) |
337
+ | `unproven` | A's own call was not `reached` |
338
+ | `public` | the policy admits an anonymous caller: read half skipped, write half kept |
339
+ | `unproven` | a query whose A answer held none of A's rows |
340
+ | `proven` | otherwise |
341
+
342
+ Absent on `off` and `refused` records. Only `crossed` is red.
343
+ [Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#the-tenant-proof).
344
+
345
+ A reached job is then killed at every step boundary — `crashProof` with its control's input and
346
+ actor — and `ExerciseRecord.crash` is a `CrashProof`. The runs commit on the pass's database: job
347
+ bodies run outside any transaction (`@ultimat3/jobs` `execute.ts`), so no rollback scope could
348
+ hold them, and the proof runs after the control so its runs meet the rows the control left.
349
+
350
+ | `verdict` | When | Red |
351
+ |---|---|---|
352
+ | `diverged` | a killed run made an effect more often than the clean one: `findings[]` (`boundary`, `mode`, `effect`, `clean`, `killed`, `after?`) | yes, `X_JOB_CRASH_DIVERGED`, one per effect, fix `x exercise <job> --json` and the `step.run('<name>', …)` to wrap it in |
353
+ | `held` | no divergence; `boundaries` and `reruns` (lost answers, at-least-once) counted | no |
354
+ | `unproven` | the proof itself threw: `reason` | no |
355
+
356
+ Absent on every record but a reached job.
357
+ [Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#the-crash-proof).
358
+
359
+ Then the two sweeps, with no flag to turn either off. **Scale:** every reached primitive is called
360
+ once more in a tenant seeded at 2 rows a table and once in one at 8 (`scale-sweep*.ts`); a statement
361
+ shape sent more often at 8 is a loop over rows, `ExerciseRecord.scale` `grows`, red as
362
+ `X_QUERY_SCALES_WITH_ROWS`. Under `expectedQueryLoop(reason, fn)` it is `flat`, the loop listed in
363
+ `expected` with its reason; a call that did not answer at both sizes is `unproven`, never red.
364
+ **Plans:** every distinct statement the pass sent whose first relation is an app table is planned
365
+ on the gate's database with `enable_seqscan`, `enable_bitmapscan` and `enable_sort` off — a generic
366
+ plan, so empty tables answer what the schema can serve (`plan-sweep.ts`, `plan-oracle.ts`):
367
+
368
+ | Red node | Code | Fix printed |
369
+ |---|---|---|
370
+ | `Seq Scan` on an app table, or an index scan with no `Index Cond` that no `Limit` stops | `X_QUERY_UNINDEXED` | `indexes: [{ on: [...] }]` by property, the tenant first when the statement compares it; or `scan: '<why>'` on the entity |
371
+ | `Sort` under a `Limit` | `X_QUERY_SORT_UNSERVED` | the same, the order's columns after the equality ones |
372
+
373
+ Not red: a statement whose equality pins a unique key, one a partial index serves once its value is
374
+ bound, one sent inside `expectedSeqScan(reason, fn)` or on an entity that declared `scan` (listed in
375
+ `plans.exempt`), one generic planning refuses (`plans.unplannable`). A server below Postgres 12 is
376
+ `X_QUERY_PLAN_UNAVAILABLE`, never a pass. [Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#the-plan-and-scale-sweeps).
377
+
269
378
  ## Shared examples
270
379
 
271
380
  ```ts
@@ -337,6 +446,13 @@ the template to leave, and the next worker's migration holds one. With no Postgr
337
446
  its own `--parallel` worker — measured on Bun 1.3.14, `--parallel` populates `BUN_TEST_WORKER_ID`
338
447
  and `JEST_WORKER_ID` itself, so that precedence is load-bearing rather than defensive.
339
448
 
449
+ **A unit test on the migrated database** is `reusableDatabase(() => migratedDatabase({ root }))`
450
+ plus `setDbClient` in `beforeAll` and `setDbClient(undefined)` in `afterAll` — the recipe in
451
+ [Testing](../../wiki/Testing.md#database-state-and-what-a-run-leaves-behind). `unitTest` grows no
452
+ database mode for it: the scaffolded handle's `testDriver()` (`@ultimat3/entity`, `As of 30.0`)
453
+ follows the installed client per call, so `db.<table>` and raw SQL meet in one database, and a
454
+ file that installs none never loads PGlite.
455
+
340
456
  ## The harness puts back what it found
341
457
 
342
458
  `bun test` is one process, so `describeApp`/`testApp` teardown is a **restore**, never an uninstall.
@@ -367,9 +483,69 @@ core's `markListening()`, so a test may call its own `handle.url()` on a kernel-
367
483
  the seal fully on. Unsealing (`ULTIMATE_TEST_ALLOW_NET=1`) stays reserved for a deliberate live
368
484
  integration — never for a socket test.
369
485
 
486
+ **One registration answers a retry test** — `mockFetch(match, answer)`, `mockJson` inherits the match:
487
+
488
+ ```ts
489
+ import { expect, mockFetch, requestLog } from '@ultimat3/testing';
490
+
491
+ declare function charge(): Promise<void>; // yours: the code under test
492
+ declare const BEARER: string;
493
+
494
+ mockFetch('POST https://api.vendor.test/charge', [
495
+ new Response('busy', { status: 503 }),
496
+ new TypeError('connection reset'),
497
+ new Response('{"id":"ch_1"}'),
498
+ ]);
499
+ await charge();
500
+ expect(requestLog().map((call) => call.headers['authorization'])).toEqual([BEARER, BEARER, BEARER]);
501
+ ```
502
+
503
+ | Argument | Takes |
504
+ |---|---|
505
+ | `match` | an exact URL, a prefix ending in `*`, or a RegExp: every verb. A string may name one verb first (`'POST https://…'`) |
506
+ | `answer` | a handler, or a list of `Response \| Error` answered in order: the last repeats, each `Response` is cloned, an `Error` is thrown. An empty list is `X_INVARIANT` |
507
+
508
+ | Rule | Detail |
509
+ |---|---|
510
+ | `requestLog()` | `{ method, url, headers, body }[]`, in call order, refused calls included, unredacted, process-local. Header names are lowercase; `body` is UTF-8 text or `null`. `resetNetwork` clears it, `captureNetwork` carries it |
511
+ | A body is read from a clone | so a handler and the real call still get theirs. A `ReadableStream` on its way to a real server is never read and logs `null`: reading it would take its backpressure away |
512
+ | Never ahead of the answer | a stream body or a `Request`'s own is captured beside the call, so a mock answering 413 at once is not held to the stream's end: it is in the entry once it ends, `null` until then |
513
+ | A socket dial | logs `method: 'WEBSOCKET'` or `'CONNECT'`, empty `headers`, `body: null` |
514
+ | A pinned dial | an address-literal URL carrying a `host` header also matches a route on that name: `mockFetch('https://vendor.test/a', …)` answers `https://203.0.113.9/a` with `host: vendor.test`. The log keeps the URL dialled |
515
+
516
+ **A vendor's answers, recorded once** — `cassette(dir, name, options)`, `As of 2026-11`. Replay is
517
+ `mockFetch`: each recorded URL becomes one route answered in call order, so anything else is still
518
+ `X_TEST_NETWORK_SEALED`. Recording is by hand.
519
+
520
+ ```ts
521
+ import { cassette, expect } from '@ultimat3/testing';
522
+
523
+ declare function refund(): Promise<{ status: number }>; // yours: the code under test
524
+
525
+ await using _tape = cassette(import.meta.dir, 'refund-flow', {
526
+ record: { hosts: ['api.vendor.test'] },
527
+ });
528
+ expect((await refund()).status).toBe(200);
529
+ ```
530
+
531
+ | Mode | What happens |
532
+ |---|---|
533
+ | replay (default) | reads `<dir>/__cassettes__/<name>.json`; no file is `X_CASSETTE_MISSING`; an entry never asked for fails at dispose, `X_CASSETTE_UNUSED` (`meta.unused`) |
534
+ | `ULTIMATE_CASSETTE=record bun test <file>` | lets `record.hosts` through the seal for the cassette's life, and writes at dispose. `CI` set, no `record.hosts`, or a dial to another host is `X_CASSETTE_RECORD_REFUSED` |
535
+
536
+ | What is written | Rule |
537
+ |---|---|
538
+ | request | never: assert what was sent with `requestLog()` |
539
+ | response headers | `content-type`, `retry-after`, `ratelimit-*` only |
540
+ | credential-named keys | masked as `[redacted]` in JSON bodies at any depth and in URL queries; a masked URL still matches the real request on replay |
541
+ | `redact(text)` | the app's rule, after the masking, on each URL and text body; a `[redacted]` it puts in a URL matches any one segment on replay |
542
+ | the secret scan | last, over the exact bytes: every credential-named env var's value and `secrets: []`, raw, JSON-escaped, percent-encoded, base64, base64url, and raw inside a base64 body. A hit writes nothing: `X_CASSETTE_SECRET` names the key and the byte offset, never the value. Values under 8 characters are skipped; a hashed or split value is not caught |
543
+ | bodies | text as text, anything else base64; keys sorted; temp file then rename; a body that fails to read writes nothing |
544
+ | streams | recorded whole, replayed as one body: a chunk-boundary test stays hand-written |
545
+
370
546
  **Three dials are sealed, and only three** — `fetch`, `new WebSocket(url)` and `Bun.connect({ hostname,
371
547
  port })`, behind one gate: one allow-list (`allowHost('host[:port]')`), one offline state, one
372
- `requestedUrls()` record. As of 2026-10:
548
+ `requestLog()` record. As of 2026-11:
373
549
 
374
550
  | Dial | Refused with | Passes through |
375
551
  |---|---|---|
@@ -462,6 +638,75 @@ gains is a compile error in the test. The context is the one the `auth` stage se
462
638
  still anonymous, because resolving it is the function under test. `@ultimat3/http` is imported
463
639
  inside the function.
464
640
 
641
+ ## Testing a sign-in against an identity provider
642
+
643
+ `As of 2026-11`. `fakeIdentityProvider({ users, issuer?, clientId?, clock? })` is an OpenID provider
644
+ served through `mockFetch`, no socket: `/.well-known/openid-configuration`, a JWKS from a key it
645
+ generates, `/authorize` approving as the chosen user, `/token` (PKCE checked) and `/userinfo`.
646
+
647
+ ```ts
648
+ import { discoverOAuthProvider, registerOAuthProvider } from '@ultimat3/auth';
649
+ import { expect, fakeIdentityProvider, unitTest } from '@ultimat3/testing';
650
+
651
+ unitTest('the provider registers from its own discovery document', async () => {
652
+ const idp = await fakeIdentityProvider({ users: [{ sub: 'ada', email: 'ada@corp.test' }] });
653
+ const provider = registerOAuthProvider(
654
+ await discoverOAuthProvider({ id: 'corp', issuer: idp.issuer }),
655
+ );
656
+ expect(provider.jwksUri).toBe(idp.jwksUri);
657
+ idp.tamper({ wrongKey: true }); // every token from here on fails the signature check
658
+ });
659
+ ```
660
+
661
+ | Member | Does |
662
+ |---|---|
663
+ | `issuer` · `clientId` · `clientSecret` · `jwksUri` | what the app's provider registration and credentials take; `issuer` defaults to `https://idp.test`, `clientId` to `fake-client` |
664
+ | `signInAs(sub)` | who `/authorize` approves as next; the first user until called |
665
+ | `tamper({ wrongAudience?, expired?, wrongKey?, noEmailVerified? })` | every token minted after it is broken that way; `tamper({})` restores honesty |
666
+ | `idToken({ sub, nonce? })` | a signed id token, for a test that skips the redirect leg |
667
+
668
+ Each tamper is refused by `@ultimat3/auth`'s real `verifyIdToken` with the code a real provider's
669
+ broken token gets (`packages/testing/src/fake-identity-provider.test.ts`). An unknown `sub` and an
670
+ empty `users` are `X_INVARIANT`.
671
+
672
+ ## Testing a passkey ceremony
673
+
674
+ `As of 2026-11`. `softAuthenticator({ origin, algorithm?, counter? })` is a passkey on no device:
675
+ a real ES256 (default) or RS256 key pair on WebCrypto, `create(options)` answering what
676
+ `newPasskey` would and `get(options)` what `getPasskey` would — so the test runs
677
+ `@ultimat3/auth`'s real verifier. One implementation and one import path, `@ultimat3/auth/soft-authenticator`, which `@ultimat3/auth`'s own suites use too (auth cannot import this package).
678
+
679
+ ```ts
680
+ import { defineAuth, memoryAuthAdapter, passkeyLogin } from '@ultimat3/auth';
681
+ import { softAuthenticator } from '@ultimat3/auth/soft-authenticator';
682
+ import { expect, unitTest } from '@ultimat3/testing';
683
+
684
+ unitTest('a member signs in with the passkey they registered', async () => {
685
+ const adapter = memoryAuthAdapter();
686
+ const ada = await adapter.createUser({
687
+ id: '00000000-0000-7000-8000-00000000ada1',
688
+ email: 'ada@example.test',
689
+ passwordHash: null,
690
+ orgId: null,
691
+ roles: [],
692
+ createdAt: new Date(),
693
+ });
694
+ const origin = 'http://localhost:3000';
695
+ const auth = defineAuth({ adapter, passkeys: { rpId: 'localhost', origins: [origin] } });
696
+ const passkeys = passkeyLogin(auth);
697
+ const device = await softAuthenticator({ origin });
698
+ const created = await device.create(await passkeys.registerOptions({ userId: ada.id }));
699
+ await passkeys.register({ userId: ada.id, response: created });
700
+ const asserted = await device.get(await passkeys.loginOptions());
701
+ const signedIn = await passkeys.login({ response: asserted });
702
+ expect(signedIn.actor.id).toBe(ada.id);
703
+ });
704
+ ```
705
+
706
+ The second argument of `create` / `get` breaks exactly one check — `{ origin, type, rpId, flags,
707
+ signCount, challenge, userHandle, crossOrigin }` — so a refusal test is one line. The step-up is the
708
+ same `get` over `passkeyStepUp(…).options(…)` (`soft-authenticator.test.ts`).
709
+
465
710
  ## Running a mutator's local half
466
711
 
467
712
  `local(tx, input)` is replayed on every rebase, so what a test proves is the row it leaves and that
@@ -797,6 +1042,15 @@ e2eTest('the feed paints, and says so when the socket drops', async ({ page }) =
797
1042
  `expect(await locator.isVisible()).toBe(true)` still works and is a **different** assertion: it
798
1043
  fails on a page that simply has not painted yet.
799
1044
 
1045
+ ## Proving a subject erase
1046
+
1047
+ `subjectProof(subject, { entities, writer, read, seed? })` seeds every table with canaries
1048
+ (`seedGraph`), erases one generated person through the erase's own table passes, and reads every
1049
+ stored row back: a seeded value of theirs outside a declared `retain`, or a row the erase deletes
1050
+ still stored, is `X_SUBJECT_ERASE_INCOMPLETE` naming the table and column. A declaration the plan
1051
+ refuses comes back as its own problems, nothing seeded. `x exercise` runs it for every
1052
+ `defineSubject()`, on a scratch database of its own.
1053
+
800
1054
  ## Errors
801
1055
 
802
1056
  `X_TEST_NETWORK_SEALED` `X_TEST_DB_UNAVAILABLE` `X_TEST_NONDETERMINISTIC` `X_TEST_FIXTURE_UNKNOWN`
@@ -806,7 +1060,9 @@ fails on a page that simply has not painted yet.
806
1060
  `X_TEST_ISLAND_STATES_NOT_PURE` `X_TEST_ISLAND_STATES_MISSING_FILE` `X_TEST_ISLAND_STATES_UNKNOWN`
807
1061
  `X_TEST_ISLAND_STATES_AMBIGUOUS` `X_TEST_ISLAND_STATE_ID_INVALID` `X_TEST_ISLAND_STATE_DUPLICATE`
808
1062
  `X_TEST_ISLAND_STATE_JSON_INVALID` `X_TEST_ISLAND_STATE_CLOCK_INVALID`
809
- `X_TEST_ISLAND_STATE_STUB_INVALID`
1063
+ `X_TEST_ISLAND_STATE_STUB_INVALID` `X_SEED_GRAPH_UNSATISFIED` `X_E2E_A11Y_VIOLATION`
1064
+ `X_SEED_GRAPH_ENTITY_UNKNOWN` `X_TENANT_LEAK` `X_TENANT_WRITE_CROSSED` — the last two are findings
1065
+ `x exercise` returns, never thrown
810
1066
 
811
1067
  ### Error classes
812
1068
 
@@ -816,11 +1072,15 @@ a job boundary the class is gone and the `code` is what survives — match on th
816
1072
  | Class | Code | Declared in |
817
1073
  |---|---|---|
818
1074
  | `AppNotBootedError` | `X_TEST_APP_NOT_BOOTED` | `src/errors.ts` |
1075
+ | `E2eA11yAllowUnreasonedError` | `X_E2E_A11Y_VIOLATION` | `src/e2e-a11y-run.ts` |
1076
+ | `E2eA11yUnavailableError` | `X_TEST_FIXTURE_UNAVAILABLE` | `src/e2e-a11y-run.ts` |
1077
+ | `E2eA11yViolationError` | `X_E2E_A11Y_VIOLATION` | `src/e2e-a11y-run.ts` |
819
1078
  | `FixtureUnavailableError` | `X_TEST_FIXTURE_UNAVAILABLE` | `src/errors.ts` |
820
1079
  | `NetworkOfflineError` | `X_TEST_NETWORK_OFFLINE` | `src/errors.ts` |
821
1080
  | `NetworkSealedError` | `X_TEST_NETWORK_SEALED` | `src/errors.ts` |
822
1081
  | `NondeterministicError` | `X_TEST_NONDETERMINISTIC` | `src/errors.ts` |
823
1082
  | `RegistryLeakError` | `X_TEST_REGISTRY_LEAK` | `src/errors.ts` |
1083
+ | `SeedGraphUnsatisfiedError` | `X_SEED_GRAPH_UNSATISFIED` | `src/seed-graph-error.ts` |
824
1084
  | `TestDatabaseUnavailableError` | `X_TEST_DB_UNAVAILABLE` | `src/errors.ts` |
825
1085
 
826
1086
  ## Log lines under test
@@ -872,3 +1132,18 @@ registered, and this guard reports additions only, so the damage lands on an inn
872
1132
  nothing pointing back. A leak fails a one-file run exactly as it fails the suite —
873
1133
  each file is judged against its own baseline — which is what makes the `bun test <file>` in the fix
874
1134
  line reproduce it.
1135
+
1136
+ ### A module mock ends with its file
1137
+
1138
+ Bun keeps a `mock.module` (and its aliases `vi.mock`, `jest.mock`) for the life of the process,
1139
+ past `mock.restore()` and past the file. The preload wraps the call: a file's first mock of a
1140
+ module snapshots the real namespace, and the file boundary mocks it back, so the next file on a
1141
+ shared worker reads the real module. A mock made in a preload is every file's environment and is
1142
+ left alone. The cost: the real module is LOADED for its snapshot, so its top-level code runs. One
1143
+ that throws on load cannot be put back; the next file's first test fails with the file named:
1144
+
1145
+ ```text
1146
+ X_TEST_MODULE_MOCK_UNRESTORABLE: a mocked module could not be put back at the file boundary —
1147
+ "/app/a.test.ts" mocked "/app/vendor.ts", whose real module failed to load (…)
1148
+ fix: x test unit --isolate # or hand the mocked dependency to the code under test as a parameter
1149
+ ```