@ultimat3/testing 8.0.0 → 10.0.0

package/CLAUDE.md

@@ -43,7 +43,10 @@ is its own entry point and not part of the barrel.
 | Leaks are the file's, not the next file's | `installRegistryLeakGuard()` runs from the preload and fails the run naming the FILE that left cache tags declared or a cache tier registered after its last test (`X_TEST_REGISTRY_LEAK`). `bun test` is one process, so without it the failure lands on an innocent suite in another package. What a file's MODULE graph declares is its environment; what the file installs after that is its own to undo |
 | The baseline is not a hook | measured on Bun 1.3.14 the order is onLoad → module eval → file `beforeAll` → describe `beforeAll` → preload `beforeEach`, so a preload hook cannot sample before the file's own `beforeAll` — a `declareTags()` there read as environment and the run went green. The load handler appends the sample to the file's source instead: after evaluation, before any hook the file registers. It is also the only signal carrying file identity, which `bun:test` hooks do not |
 | Reported and restored are different sets | the guard also RESTORES, at the same file boundary, the registries whose module-scope declarations a neighbour's cleanup destroys — the locale config, the catalogs, the permission set and the role map (`registry-snapshot.ts`). A module evaluates once per process, so a later file's own `import` is a cache hit that declares nothing: `clearPermissions()` in one CLI test took `admin:*` from `@ultimat3/admin`'s barrel for the whole run, and a `defineCatalogs()` inside a loaded app narrowed `supported` so `Accept-Language: de-DE` answered `en` in files that never mentioned locales. Nothing restored is reported and nothing reported is restored — a repair followed by a failure over it would be two answers to one question |
-| Guarded state is boot state | only the two registries whose honest invariant is "clean when the file ends" — `declareTags` and `registerTier` are boot installs. `entity()`, `job()` and `defineRoute()` register at MODULE scope, which is how an app declares itself, so a filled registry there is idiomatic and unguarded |
+| A catalog restore is a MERGE, never a replace, `As of 2026-08-23` | the other three registries are replaced with the snapshot; the catalogs are not. `registerCatalog` has no inverse, so the only thing a file can cost its neighbour is a `resetCatalogs()` — and that is all this repairs. Everything the live registry still holds survives, a key first registered during the file and an override of a framework base string alike, because both are one-time MODULE-scope declarations: `loadApp()` in a test body dynamically imports the app's i18n package after the file's baseline was sampled, so a replace dropped 519 keys nothing could re-add and `t()` answered `⟦brand.name⟧` for the rest of the process (#312, measured: `bun test apps/admin` in `dummy/social-media-clone`, 4 fail → 0). The override half is the same defect with no `⟦…⟧` to show it — the demo app overrides `admin.denied.body`, and reverting it rendered `@ultimat3/i18n`'s own copy. The cost, stated: a file that CLOBBERS an inherited key owns the cleanup, and the cleanup is `resetCatalogs()` in its own `afterAll`, which this repair is built around |
+| The same is still true of permissions, and it is MEASURED, `As of 2026-08-23` | permissions, roles and the locale config are still replaced with the file's baseline, so an app's `definePermissions()` reached only by a dynamic `loadApp()` is dropped at that file's boundary exactly as the catalogs were. Reproduced: `bun test apps packages` in `dummy/social-media-clone` — one process, unsharded — leaves 6 `.contract.` cases failing on `knownPermissions()` missing `dashboard:read`, and every one of them passes when its file runs alone. The catalog fix took that run from 16 fail to 6; these are the 6. Not fixed here, and the reason is not that it is a different defect — it is the same one — but that the same union rule applied to permissions leaks every permission `packages/policy/src/permissions.test.ts` declares into every later file, and judging that needs a repo-wide `bun test` this package cannot run for itself. Its own piece of work, not a rider on this one. The `unit` step is green over it because `.contract.` is a different step and it shards |
+| Guarded state is boot state | only the two registries whose honest invariant is "clean when the file ends" — `declareTags` and `registerTier` are boot installs. `entity()`, `job()` and `defineRoute()` register at MODULE scope, which is how an app declares itself, so a filled registry there is idiomatic and unREPORTED |
+| Filled and CLEARED are different questions | and the row above answers only the first. "Idiomatic to leave filled" is about the leak REPORT; it says nothing about a file that calls `clear*()`/`reset*()` and takes a module-scope declaration away from every file after it, which is what the RESTORE half exists for. The restore covers four registries and roughly nine have a reset export — routes (`clearRoutes`, `@ultimat3/render`), jobs and tasks (`@ultimat3/jobs`), actions and queries (`resetRegistry`), models/prompts/agents (`@ultimat3/ai`), mails (`@ultimat3/mail`). The table in `registry-leak-guard.ts` is the list. **Not closed, `As of 2026-08-23`**, and the reason is the shape rather than the difficulty: `jobs`, `action`, `query` and `mail` publish a lister and a reset but no RESTORE, so each needs the pair `@ultimat3/policy` got (`restorePermissions` / `restoreRoles`) before one line here can use it — and it has to land as one change, because anything less means editing `ProcessRegistrySnapshot` twice |
 | An empty registry is a premise you state | a test whose subject is "nothing is declared" — `x db gen` with nothing to generate — calls `isolateEntityRegistry()` and restores in a `finally`. Inheriting it means the test passes until a neighbouring file imports an entity |
 | That one helper is off the barrel | `@ultimat3/testing/registry-isolation`, its own entry point. It is the only module here that value-imports `@ultimat3/entity` — the restore is handed back synchronously, so it cannot be a dynamic import inside the call — and a static re-export from `src/index.ts` would load the entity registry into every test that imports this package for `expect` |
 | Teardown restores, never uninstalls | `describeApp`/`testApp` capture the seal and the determinism snapshot before booting and put those back — `restoreDeterminism()` in a scope hands the REAL clock and the REAL `fetch` to every later FILE in the process. `captureDeterminism()` / `restoreCapturedDeterminism()` are the pair for any nested install |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/testing",
-  "version": "8.0.0",
+  "version": "10.0.0",
   "description": "Test harness: cloned template DBs per worker, frozen clock, sealed network, 6 test types",
   "license": "MIT",
   "type": "module",
@@ -33,16 +33,16 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/cache": "8.0.0",
-    "@ultimat3/core": "8.0.0",
-    "@ultimat3/db": "8.0.0",
-    "@ultimat3/entity": "8.0.0",
-    "@ultimat3/i18n": "8.0.0",
-    "@ultimat3/jobs": "8.0.0",
-    "@ultimat3/mail": "8.0.0",
-    "@ultimat3/policy": "8.0.0",
-    "@ultimat3/query": "8.0.0",
-    "@ultimat3/realtime": "8.0.0",
-    "@ultimat3/time": "8.0.0"
+    "@ultimat3/cache": "10.0.0",
+    "@ultimat3/core": "10.0.0",
+    "@ultimat3/db": "10.0.0",
+    "@ultimat3/entity": "10.0.0",
+    "@ultimat3/i18n": "10.0.0",
+    "@ultimat3/jobs": "10.0.0",
+    "@ultimat3/mail": "10.0.0",
+    "@ultimat3/policy": "10.0.0",
+    "@ultimat3/query": "10.0.0",
+    "@ultimat3/realtime": "10.0.0",
+    "@ultimat3/time": "10.0.0"
   }
 }

package/src/errors.ts

@@ -62,7 +62,12 @@ registerErrorCodes(
   ),
 );
 
-const docsFor = (code: TestingErrorCode): string => `https://ultimate.dev/errors/${code}`;
+// No `docs:` on the subclasses below. `UltimateError` fills it from `describeErrorCode(code).docs`,
+// which is `@ultimat3/core`'s `ERROR_DOCS_URL` — one page for every code, never one per code, because
+// `wiki/` is the framework's only public documentation surface and a code lives there in a TABLE ROW,
+// which has no anchor. The `https://ultimate.dev/errors/<code>` links this file built until 9.x
+// answered 404, host included, on every error it has ever thrown; restating the replacement here
+// would be the same constant in eight places waiting to drift again.
 
 /** A test reached the network without a mock or an allowlist entry. Always a bug, never a flake. */
 export class NetworkSealedError extends UltimateError {
@@ -73,7 +78,6 @@ export class NetworkSealedError extends UltimateError {
         input.allowed.length > 0 ? input.allowed.join(', ') : 'none'
       })`,
       fix: `mockFetch('${input.url}', () => new Response('{}')) — or allowHost('${hostOf(input.url)}') if it must be real`,
-      docs: docsFor('X_TEST_NETWORK_SEALED'),
     });
   }
 }
@@ -85,7 +89,6 @@ export class TestDatabaseUnavailableError extends UltimateError {
       code: 'X_TEST_DB_UNAVAILABLE',
       cause: input.cause,
       fix: 'x dev (embedded Postgres), or set TEST_DATABASE_URL to a running Postgres',
-      docs: docsFor('X_TEST_DB_UNAVAILABLE'),
     });
   }
 }
@@ -97,7 +100,6 @@ export class NondeterministicError extends UltimateError {
       code: 'X_TEST_NONDETERMINISTIC',
       cause: `${input.what} produced "${input.first}" then "${input.second}"`,
       fix: 'wrap the test in frozenClock() / seededRandom(), or remove the wall-clock read',
-      docs: docsFor('X_TEST_NONDETERMINISTIC'),
     });
   }
 }
@@ -124,7 +126,6 @@ export class FixtureUnknownError extends UltimateError {
           ? `test requested fixture "${input.name}" but none are registered`
           : `test requested fixture "${input.name}"; registered: ${input.registered.join(', ')}`,
       fix: `register it at test setup: defineFixtures({ ${input.name}: () => buildIt() })`,
-      docs: docsFor('X_TEST_FIXTURE_UNKNOWN'),
     });
   }
 }
@@ -145,7 +146,6 @@ export class FixtureUnavailableError extends UltimateError {
       code: 'X_TEST_FIXTURE_UNAVAILABLE',
       cause: `fixture "${input.name}" is declared but nothing in this process drives it — it needs ${input.needs}`,
       fix: `install one in the test preload: defineFixtures({ ${input.name}: () => yourDriver() })`,
-      docs: docsFor('X_TEST_FIXTURE_UNAVAILABLE'),
     });
   }
 }
@@ -170,7 +170,6 @@ export class LiveNodeEmptyError extends UltimateError {
       cause:
         'no query declared live: true is registered in this process, so the node would serve none',
       fix: "import the app's api module in the test preload — import './apps/web/api' — then: x queries list --json",
-      docs: docsFor('X_TEST_LIVE_NODE_EMPTY'),
     });
   }
 }
@@ -189,7 +188,6 @@ export class LiveNodeUpgradeRefusedError extends UltimateError {
       code: 'X_TEST_LIVE_NODE_UPGRADE_REFUSED',
       cause: `the sync node answered the upgrade with ${status === undefined ? 'no response' : `HTTP ${String(status)}`} instead of taking it`,
       fix: 'await node.start() before connect(), and keep the request path at /_x/sync',
-      docs: docsFor('X_TEST_LIVE_NODE_UPGRADE_REFUSED'),
     });
   }
 }
@@ -208,7 +206,6 @@ export class NetworkOfflineError extends UltimateError {
       code: 'X_TEST_NETWORK_OFFLINE',
       cause: `${input.method} ${input.url} while the test network is ${input.mode}`,
       fix: 'network.online() before the call — or assert the offline path instead of the request',
-      docs: docsFor('X_TEST_NETWORK_OFFLINE'),
     });
   }
 }
@@ -220,7 +217,6 @@ export class TestEvalThresholdError extends UltimateError {
       code: 'X_TEST_EVAL_THRESHOLD',
       cause: `eval "${input.name}" scored below ${input.threshold}: ${input.detail}`,
       fix: 'improve the prompt under test, or lower the threshold passed to evalTest()',
-      docs: docsFor('X_TEST_EVAL_THRESHOLD'),
     });
   }
 }
@@ -234,7 +230,6 @@ export class TestSchemaExpectedError extends UltimateError {
       // Names the call, not the intent: "assert against action.input" left the reader to work out
       // which call to edit, and a fix is only executable if it can be pasted over the failing line.
       fix: 'call toRejectInput(action.input) — the schema, not toRejectInput(action) or the query',
-      docs: docsFor('X_TEST_SCHEMA_EXPECTED'),
     });
   }
 }
@@ -247,7 +242,6 @@ export class TestJobExpectedError extends UltimateError {
       cause: 'toEmitSteps expects a job declaration built with job(...)',
       // Same rule as X_TEST_SCHEMA_EXPECTED's: the paste-able call, not a description of it.
       fix: 'call toEmitSteps(myJob) with the job export, not toEmitSteps(myJob.run)',
-      docs: docsFor('X_TEST_JOB_EXPECTED'),
     });
   }
 }
@@ -266,7 +260,6 @@ export class FactoryTraitUnknownError extends UltimateError {
           ? `factory "${input.table}" was asked for trait "${input.trait}" but declares none`
           : `factory "${input.table}" has no trait "${input.trait}"; declared: ${input.declared.join(', ')}`,
       fix: `declare it: defineFactory(${input.table}, { traits: { ${input.trait}: { /* columns */ } } })`,
-      docs: docsFor('X_TEST_FACTORY_TRAIT_UNKNOWN'),
     });
   }
 }
@@ -282,7 +275,6 @@ export class FactoryNotPersistedError extends UltimateError {
       code: 'X_TEST_FACTORY_NOT_PERSISTED',
       cause: `factory "${input.table}".create() ran with no persister in this process`,
       fix: 'usePersister({ insert: (table, row) => repoFor(table).insert(row) }) in the test preload — or build() for an in-memory row',
-      docs: docsFor('X_TEST_FACTORY_NOT_PERSISTED'),
     });
   }
 }
@@ -298,7 +290,6 @@ export class NetworkRaceError extends UltimateError {
       code: 'X_TEST_NETWORK_RACE',
       cause: 'sealed network lost its original fetch mid-request',
       fix: 'do not call unsealNetwork() while a request from the same test is still in flight',
-      docs: docsFor('X_TEST_NETWORK_RACE'),
     });
   }
 }
@@ -357,7 +348,6 @@ export class RegistryLeakError extends UltimateError {
       code: 'X_TEST_REGISTRY_LEAK',
       cause: input.leaks.map(describeLeak).join('; '),
       fix: `${input.leaks.map(repairFor).join('; ')} — then re-run: bun test ${files.join(' ')}`,
-      docs: docsFor('X_TEST_REGISTRY_LEAK'),
     });
   }
 }
@@ -384,7 +374,6 @@ export class IslandNotBuiltError extends UltimateError {
         input.built.length === 0
           ? 'x g island <name> --at apps/web/site — then set root to the directory holding apps/'
           : `mountIsland({ build, root, file: ${renderFixLiteral(input.built[0], ISLAND_PLACEHOLDER)} }) — the path is app-root-relative, not relative to the test, and ${file} is not one of them`,
-      docs: docsFor('X_TEST_ISLAND_NOT_BUILT'),
     });
   }
 }
@@ -434,7 +423,6 @@ export class IslandMountMissingError extends UltimateError {
         component === undefined
           ? `${file} exports nothing a mount could render — x g island <name> --at apps/web/site writes an island whose mount() is already there, and its shape is the one to copy`
           : `in ${file} add: import { render } from 'solid-js/web'; export function mount(el: HTMLElement, props: Parameters<typeof ${component}>[0]): void { el.textContent = ''; render(() => <${component} {...props} />, el); }`,
-      docs: docsFor('X_TEST_ISLAND_NO_MOUNT'),
     });
   }
 }

package/src/registry-leak-guard.ts

@@ -14,13 +14,38 @@ import { captureProcessRegistries, restoreProcessRegistries } from './registry-s
  * manifest's entity names, `registerTier` takes `app.config.ts`'s tiers — so "empty again when the
  * file ends" is the honest invariant for a test. The entity, job, route and permission registries
  * are not here: `entity()` and `job()` register at module scope, which is how an app declares
- * itself, so a file that leaves them filled is idiomatic rather than leaky. A test whose subject is
+ * itself, so a file that leaves them FILLED is idiomatic rather than leaky. A test whose subject is
  * an EMPTY one of those establishes it itself — `isolateEntityRegistry()`.
  *
- * Neither is RESTORED, and that is the same judgement read the other way: `@ultimat3/cache`
- * publishes no un-declare for a tag, so there is nothing to put a tag registry back WITH. The
+ * Neither is RESTORED, and for the tag registry that is the same judgement read the other way:
+ * `@ultimat3/cache` publishes no un-declare, so there is nothing to put one back WITH. The
  * registries that are restored are `registry-snapshot.ts`'s, and none of them is reported —
  * repairing a state and then failing the run over it would be two answers to one question.
+ *
+ * **Filled and CLEARED are different questions, and the paragraph above only answers the first.**
+ * "Idiomatic to leave filled" says nothing about a file that calls `clear*()`/`reset*()` and takes
+ * a module-scope declaration from every file after it — a module evaluates once per process, so the
+ * next file's own `import` is a cache hit that declares nothing. That is what `registry-snapshot.ts`
+ * repairs, and it repairs FOUR registries out of the set that has one:
+ *
+ *   | registry | reset export | owner | in the snapshot? |
+ *   |---|---|---|---|
+ *   | locales / catalogs | `resetCatalogs` | `@ultimat3/i18n` | yes |
+ *   | permissions / roles | `restorePermissions` / `restoreRoles` | `@ultimat3/policy` | yes |
+ *   | routes | `clearRoutes` | `@ultimat3/render` | **no** |
+ *   | jobs | `resetJobs` | `@ultimat3/jobs` | **no** |
+ *   | tasks | `resetTasks` | `@ultimat3/jobs` | **no** |
+ *   | actions | `resetRegistry` | `@ultimat3/action` | **no** |
+ *   | queries | `resetRegistry` | `@ultimat3/query` | **no** |
+ *   | models / prompts / agents | `resetModels` / `resetPrompts` / `resetAgents` | `@ultimat3/ai` | **no** |
+ *   | mails | `resetMails` | `@ultimat3/mail` | **no** |
+ *   | entities | `clearRegistry` | `@ultimat3/entity` | no, by decision — `registry-isolation.ts` |
+ *
+ * Every "no" needs the SAME two halves the two "yes" rows have: a reader and a writer in the owning
+ * package (`restorePermissions` was added to `@ultimat3/policy` for exactly this), and one line
+ * here. `@ultimat3/jobs`, `@ultimat3/action`, `@ultimat3/query` and `@ultimat3/mail` publish a
+ * lister and a reset but no restore, so the change is theirs first and this file's second — and it
+ * has to land as ONE change, because the cost of the rest is a second edit of this same shape.
  */
 export interface RegistrySample {
   readonly tags: readonly string[];
@@ -138,7 +163,10 @@ export function installRegistryLeakGuard(): void {
       // loader exists to replace — and, because the first matching handler wins and this one is
       // registered from the preload, it would shadow render's transform for that file. Routing
       // `.tsx` through `transformTsx` is not the alternative either: it needs `@ultimat3/render`,
-      // whose import installs that global loader into every test process in the repo. Zero
+      // and the transform itself is on `@ultimat3/render/server`, whose import installs that global
+      // loader into every test process in the repo. (The `.` barrel does NOT — `installRenderLoader()`
+      // moved to `server.ts:15` in the 9.0.0 split, and this comment claimed otherwise until
+      // 2026-08-23. The `server` half is still the one that would have to be imported.) Zero
       // `.test.tsx` files exist and the convention is `<file>.test.ts`, so the narrower filter
       // costs nothing today; a `.test.tsx` added later is unguarded rather than mis-compiled.
       build.onLoad({ filter: /\.test\.ts$/ }, async (args) => {

package/src/registry-snapshot.ts

@@ -2,12 +2,18 @@
 // `registry-leak-guard.ts` owns WHEN. A snapshot rather than a reset to defaults, because what a
 // module declares at MODULE scope evaluates once per `bun test` process — a neighbour's clear is
 // permanent and there is no second evaluation left to redo it.
+//
+// FOUR registries, out of roughly nine that publish a `clear*`/`reset*`. The missing ones and what
+// each still needs are tabulated in `registry-leak-guard.ts` beside `RegistrySample` — every one of
+// them needs a RESTORE in its owning package first, the way `restorePermissions` / `restoreRoles`
+// were added to `@ultimat3/policy` for the two rows below.
 
 import type { Catalog, Locale, LocaleConfig } from '@ultimat3/i18n';
 import {
   catalogFor,
   configureLocales,
   localeConfig,
+  mergeCatalogs,
   registerCatalog,
   registeredLocales,
   resetCatalogs,
@@ -46,16 +52,50 @@ export function captureProcessRegistries(): ProcessRegistrySnapshot {
 }
 
 /**
- * Idempotent, and a REPLACE on every registry rather than a merge: a snapshot is the whole truth
- * about the process at capture time, so anything declared since must go as surely as anything
- * cleared since must come back.
+ * Idempotent. A REPLACE on the locale config, the permission set and the role map — a snapshot is
+ * the whole truth about those at capture time — and, for the catalogs alone, a key-level
+ * RECONCILE. See `restoreCatalogs`.
  */
 export function restoreProcessRegistries(snapshot: ProcessRegistrySnapshot): void {
   // A full `LocaleConfig`, so the merge `configureLocales` performs replaces all three fields —
   // a partial call can never widen `supported` back.
   configureLocales(snapshot.locales);
-  resetCatalogs();
-  for (const [locale, catalog] of snapshot.catalogs) registerCatalog(locale, catalog);
+  restoreCatalogs(snapshot.catalogs);
   restorePermissions(snapshot.permissions);
   restoreRoles(snapshot.roles, snapshot.roleSites);
 }
+
+/**
+ * Repair a clear; never undo a registration. The catalogs are the one registry here restored by
+ * MERGE rather than by replacement, and the module cache is why.
+ *
+ * `registerCatalog` has no inverse — it merges, last wins — so the only thing a file can do that
+ * costs the next file anything is `resetCatalogs()`, and that is exactly what this repairs: a key
+ * the snapshot holds and the live registry has lost comes back. Everything the live registry still
+ * holds is left alone, INCLUDING a key whose value the file changed.
+ *
+ * Undoing the change is what the first attempt at #312 did, and it is wrong twice over. A key
+ * first registered during the file cannot be re-added by anyone: `loadApp()` inside a test body
+ * dynamically imports the app's i18n package, `defineCatalogs()` there is MODULE scope — once per
+ * `bun test` process — so dropping the app's 519 keys left every later file's own `import` a cache
+ * hit that declares nothing and `t('brand.name')` answering `⟦brand.name⟧` for the rest of the run.
+ * And a key it OVERRODE is the same declaration read one layer down: the demo app's
+ * `admin.denied.body` overrides `@ultimat3/i18n`'s own base string, so reverting to the inherited
+ * value rendered the framework's `This account is missing admin:read` in place of the app's copy —
+ * a green `⟦…⟧` sweep hiding the identical defect.
+ *
+ * The cost is stated rather than hidden: a file that deliberately CLOBBERS an inherited key leaves
+ * that value for the next file. Its cleanup is the one this repair is built around — `resetCatalogs()`
+ * in the file's own `afterAll`, which drops its layer and lets the boundary put back what that clear
+ * took from everyone else.
+ */
+function restoreCatalogs(snapshot: readonly (readonly [Locale, Catalog])[]): void {
+  const inherited = new Map(snapshot);
+  const live = new Map(registeredLocales().map((locale) => [locale, catalogFor(locale)] as const));
+  // Cleared first so the merge order below is this function's to choose: `registerCatalog` puts
+  // its argument last, which would otherwise let the inherited value win the keys it shares.
+  resetCatalogs();
+  for (const locale of new Set([...inherited.keys(), ...live.keys()])) {
+    registerCatalog(locale, mergeCatalogs(inherited.get(locale) ?? {}, live.get(locale) ?? {}));
+  }
+}