@ultimat3/cli 29.3.2 → 30.0.0

package/wiki/Known-Gaps.md

@@ -0,0 +1,185 @@
+# Known gaps
+
+Defects and unfinished seams named here rather than left to be discovered. Each row carries its own `As of` stamp.
+
+A reference manual that hides these is lying to the reader.
+
+**Read the section headings first.** [Open](#open) is what is wrong with the code you can install
+today. [Open by decision](#open-by-decision) will not be fixed and says why. [Not built yet](#not-built-yet)
+is unfinished work, not a defect. [Carried backlog](#carried-backlog) is owed work with a tracking
+issue each. [Deferred by plan 101](#deferred-by-plan-101) is ranked work a later plan owns. [Closed](#closed) is history — every row there carries the release
+that fixed it and the workaround for anyone still pinned below it.
+
+**Only the [footer](_Footer) stamps the installable version.** Resolve it yourself rather than reading it here:
+
+```bash
+npm view @ultimat3/core version
+```
+
+A Closed row whose release is newer than the [footer](_Footer)'s is fixed in the repository and in **no published release** yet; if you
+are on an earlier version, treat those rows as open and take the workaround. `[Unreleased]` in
+[`CHANGELOG.md`](https://github.com/developerz-ai/ultimate/blob/main/CHANGELOG.md) is the source of
+truth for what the next release carries, and most of what it fixes was never a row here. Read that section rather than this sentence: `grep -n '^## '
+CHANGELOG.md` finds its bounds, and it changes on any commit.
+
+**Publication is not a gap.** All 31 workspaces are on the registry `As of 2026-08-20`, checked by `scripts/registry-audit.ts` daily;
+`@ultimat3/scraping` was the last never-published package and was bootstrapped by hand at 2.0.0
+([`PUBLISHING.md`](https://github.com/developerz-ai/ultimate/blob/main/PUBLISHING.md) step 1).
+`bun add @ultimat3/scraping` resolves — check it with `npm view @ultimat3/scraping version`.
+
+## Open
+
+| Gap | Symptom | Work around it by |
+|---|---|---|
+| A purge that a replica never hears is bounded by `ttl`, not by the next request | `As of 2026-10-10`. The purge reaches the other replicas through the invalidation broadcast. With the bus down, a replica with its own store answers its copy as a fresh hit until that copy's `ttl` (and tells an edge the ttl's remainder), then renders. `'purge'` therefore requires a `ttl` | choose the `ttl` as the longest a missed purge may last; or share one `runtime.isrStore` with a `tagFence`, where the delete is every replica's at once |
+| A bust purges by the **route's** tags | `As of 2026-10-10`. A stored `isr` page records no tags of its own, so `post:1` reaches every stored page of a route tagged `post` — one render too many per page, never one too few. Each bust reads the store's keys once; the route of each key is remembered, not re-matched [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/render/src/render-isr-purge.test.ts#every%20stored%20key%20of%20the%20route%20goes%3A%20both%20locales%2C%20and%20every%20keyed%20query) | tag a route as narrowly as its data allows; a very large shared store pays one `paths()` per bust per replica |
+| A declared `isr` query parameter's **values** are unbounded | `As of 2026-10-10`. `query: ['page']` puts `page` in the key, and `?page=1` … `?page=999999` are each a key: the framework cannot know which values name a page. The store is bounded (least recently generated out first), so it thrashes rather than grows [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/render/src/render-isr-query.test.ts#only%20a%20declared%20parameter%20is%20part%20of%20the%20key%2C%20and%20of%20the%20URL%20load%20is%20given) | answer `withStatus(404, …)` from `load` for a value that names nothing, and rate-limit the route at the ingress; put a small closed set in the path instead (`/blog/page/[n]` with `prerender()`) |
+| A `grantUpload` with a `target` is never sniffed | `As of 2026-11-02`. A presigned PUT signs `host` and the size, so on an s3 disk the uploader chooses the stored type and the bytes. `promoteAttachment` is what checks them (`X_STORAGE_TYPE_REJECTED`), and a grant with a `target` lands straight on the row's key with no promotion step. Served, it is contained: `objectResponse` sandboxes a type a browser would run, and a false `image/png` is shown under `nosniff` [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/storage/src/grant.test.ts#naming%20a%20target%20puts%20the%20key%20on%20the%20row%20instead%20of%20in%20pending) | grant with no `target` and call `promoteAttachment({ …, policy })`; for a targeted grant, read the head yourself with `validateStoredHead(…)` before writing the key to the row → [Storage and uploads](Storage-And-Uploads) |
+| A queued offline write is not bound to its principal **on the server** | `As of 2026-10`. The fence is in the page: a principal change abandons the running replay, refuses a write in flight (`X_OFFLINE_QUEUE_ABANDONED`) and wipes the queue. A second TAB whose replay is already past its store read when another tab changes the session cookie still sends its remaining entry under the new session — one entry, one in-flight pass, on a shared browser | sign out by full navigation (`signOutHeaders()` clears site data, and every boot wipes other principals' queues); do not share a browser profile between accounts with offline writes queued |
+| Presence leaves past the lease roster cap | the sweep leader ships at most 4,096 member ids on its lease (`PRESENCE_LEASE_ROSTER_LIMIT`). When the leader's node dies, members beyond that whose TTL expired before the successor's first leading pass are never announced as left. A successor cannot diff against the shared store: the expired members are already gone from it | nothing to do: each client's next beat is answered with the whole roster, so it heals within one beat (10 s by default) |
+| An **offline** scrape reads `/robots.txt` over the real network | every `fakeBrowser` / `fixtureBrowser` run takes the default `robots: 'obey'`, and the gate reads `https://<host>/robots.txt` with the platform `fetch` before the first navigation, so outside `bun test` one request still leaves the process. **The silent half is closed `As of 2026-10-04`**: an unreachable robots.txt (5xx, 429, network error, timeout) now refuses the origin with `X_SCRAPE_ROBOTS_DISALLOWED` (RFC 9309 §2.3.1.4 for 5xx, network error and timeout; refusing on `429` is Ultimate's stricter policy — the RFC calls a `4xx` unavailable), so under `bun test` the sealed network's refusal turns the run red instead of reading as allow-everything | declare the policy in the definition under test — `robots: { ignore: 'fixture host, no live origin' }` — which returns a gate that reads nothing at all. There is no boolean and no ambient off switch, by design — see [Scraping](Scraping#testing-offline) |
+| The cache fill fence crosses nodes for **tag busts through Redis only** | `As of 2026-10`. A read-through fill re-checks a fence before it writes. Inside one process that covers everything. Across nodes: the Redis tier keeps leased generation keys a bust writes and a fill samples before `load()` and re-reads after its `SET` ([`packages/cache/src/redis-fence.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/cache/src/redis-fence.ts)), so a tag bust on another pod withdraws the fill even when the broadcast is lost — at one extra round trip before each `load()` and one after the `SET`. Three things still do not cross. **Key busts**: `CacheStack.write` and `.drop` mark `{ key }` locally and the wire form carries tags only. **A deployment with no Redis tier**: there is no shared store to hold a generation, so the window is the broadcast's. **Another pod's LRU**: a value it cached before the bust stays until its TTL when the broadcast is lost | give a `cache:` query a TTL you can afford to be wrong for on one pod, keep `redis` in `cache.tiers` on a multi-pod deploy, and prefer a **tag** bust over `cache.drop(key)`: only the first one is broadcast and fenced |
+| A replicated table with **no** replica identity is warned about, never refused | `As of 2026-10-02`. A **channel declared with params** does need `REPLICA IDENTITY FULL` on its `records` tables — a DELETE is routed by the old row's params, which a key-only image does not carry, so members keep the deleted record; `x db gen` grants it since 24.0.0 (one migration for an app that already declares such a channel), and the replicator warns (`replication.channel_identity_partial`) about a table that still lacks it. Live queries do **not** need `REPLICA IDENTITY FULL` on a keyed table: the shared window holds the whole row, so a row leaving the result set and a delete are decided from the window — proved on real WAL by [`pg-identity-window.live.test.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/realtime/src/pg-identity-window.live.test.ts) (out-of-filter update, into-filter update, delete with a key-only before-image, updates to rows never held). What still breaks is a table with no identity at all — `NOTHING`, or `DEFAULT` with no primary key — whose `UPDATE`/`DELETE` Postgres refuses once published. `X_LIVE_REPLICA_IDENTITY` names exactly those at preflight, and warns rather than throws; no `x verify` step reads it. The `REPLICA IDENTITY FULL` the app's declarations need is checked at the gate: since 24.0.0 the `drift` step reports a params-channel table whose FULL no migration recorded (`X_DB_SCHEMA_UNMIGRATED`) — against the migrations, never the live database. A live query's `subscribes:` tables are **not** granted FULL, `As of 2026-10` (#518): `x db gen` validates the names and emits no `ALTER`; a table already on FULL keeps it, and the drift step reads that as agreement | give the table a primary key, or `ALTER TABLE <t> REPLICA IDENTITY FULL;` — the warning names every table that needs it |
+| HPAs read `<unknown>` without a metrics adapter | the chart half is done — `values.yaml` declares `metricsPort: 9090`, every role but `migrate` emits the container port, `service.yaml` publishes it and `templates/servicemonitor.yaml` ships the scrape target, rendered wherever the cluster serves the Prometheus operator's kind (`serviceMonitor.enabled: auto`, `As of 2026-11`). One thing still stands and it is not the chart's to fix: turning scraped series into the `Pods` metrics an HPA reads needs a **custom-metrics adapter**, which is the cluster's pin: none — what is missing is the cluster's custom-metrics adapter, outside this repository | install a metrics adapter (and the Prometheus operator, if the cluster has none; an app chart from before the scrape object gets it from `x g chart-alerts`); until then disable the HPAs and pin `replicas`. **Do not hand-add a metrics container port** — the chart emits one and a duplicate is rejected by the API server → [Observability](Observability) |
+| `--target binary` is a **launcher** for an app tree, and has never been served from a bare VM | `As of 2026-10-06`. It boots: `x build --target binary` compiles `apps/web/server.ts` with `--compile-autoload-package-json`, `--compile-autoload-tsconfig` and `@ultimat3/*` external (`binaryArgs`, `packages/cli/src/cmd-build.ts`), and the required `windows` CI job starts the result from a scaffold and gets `200` from `/healthz`; `packages/cli/e2e/binary-launch.e2e.test.ts` proves the same on Linux. **It is not a self-contained artifact**: at run time it reads the app's `package.json` and `tsconfig.json`, imports `app.config.ts` and every module under `apps/*`, and loads the framework from the app's `node_modules` — one instance, so the app's registrations reach the server. Unmeasured: a scaffold compiled, copied with its tree to a VM with no Bun, and served under systemd [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/e2e/binary-launch.e2e.test.ts#the%20compiled%20launcher%20resolves%20its%20app%20tree%20at%20run%20time%2C%20against%20ONE%20framework%20instance) | ship the app root, its installed `node_modules` (installed for the target OS and CPU) and the binary together, and start the binary from that root. Prove it for your own app first — `./app --version`, then `ROLE=web PORT=3000 ./app` and `curl /readyz`. Prefer `--target docker`, the self-contained artifact → [Deployment](Deployment) |
+| A local `typecheck` step can report green on a tree CI fails, on the same commit — **one sighting, unreproduced** | `As of 2026-10`. Observed once (2026-09-17, TypeScript 7.0.2): after a `git stash -u` → branch switch → `x verify` on the other branch → switch back → `stash pop` sequence, a type error that surfaces in an UNCHANGED file (a widened index signature in a file it imports) passed locally and failed CI; deleting `tsconfig.tsbuildinfo` then reproduced CI's red locally. Five faithful reproduction attempts against 7.0.2 all stayed correctly red ([#450](https://github.com/developerz-ai/ultimate/issues/450)). **[#450](https://github.com/developerz-ai/ultimate/issues/450) itself is closed `As of 2026-09-23`** — read from the shipped fix, not from the tracker: plan 101 row 14 r ("Close #450", `docs/plans/2026/09/23/101-framework-deep-sweep/14-guards-gate.md:37`; plan `status.yml` `complete`), recorded in `CHANGELOG.md` under 22.0.0 and pinned by `packages/cli/src/verify-typecheck.test.ts`. `tsc -b` decides "up to date" by **mtime**, and Bun installs a dependency as **hardlinks** carrying its cache's old mtimes, so an upgrade to an already-cached version left every input older than the buildinfo and a root without `references` — every scaffolded app — skipped the program and reported green while a cold CI was red. A scaffolded app's `typecheck` is now `tsc -p .` ([`packages/cli/src/templates/scaffold-repo.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/src/templates/scaffold-repo.ts), its `"typecheck"` script), which is incremental by **content hash**, and `x verify`'s `typecheck` step picks `-p .` over `-b` for any such root ([`packages/cli/src/verify-typecheck.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/src/verify-typecheck.ts)); the framework root keeps `-b`. **Still open**: a *second* `tsc` racing the same buildinfo is the one suspect not excluded, and no cause for it has been shown. `x verify` deliberately takes two flags and no `--cold` pin: none — one sighting, and five faithful reproductions stayed red, so no test can hold it | CI is always cold and is the judge of record. If a local `typecheck` disagrees with it again, capture the command, its argv and its exit code before deleting any `tsconfig.tsbuildinfo` |
+| A `stream` page with a realtime island keeps every island on the page inert until its slowest hole closes | `As of 2026-11-02`. The runtime goes `async` so a shell island answers while a hole is open, but only with no realtime boot in the document: the boot (`script defer`) has to run first (`bootScript`, [`packages/cli/src/route-islands.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/src/route-islands.ts)), and a deferred script waits for parsing to end. Bound by the hole's 15 s deadline | keep the slow read out of such a page's holes, or give it a short deadline of its own: `fetch(url, { signal: AbortSignal.timeout(2_000) })` |
+| An open tab can meet a dead island across a deploy | `As of 2026-11-03`. A web process serves only the hashed files it built, so with `assets.retain` unset (the default) a tab holding the previous build's document gets a `no-store` 404 for its stylesheet, or for its island chunk on the first click, after every deploy; during a rolling update a new document that reaches an old pod gets no stylesheet either [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/src/retained-assets-boot.test.ts#undeclared%3A%20nothing%20is%20published%20and%20there%20is%20no%20reader). Declared, a tab survives six deploys with build 0's island alive [evidence](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/e2e/six-deploys.e2e.test.ts#with%20assets.retain%2C%20the%20click%20boots%20build%200%E2%80%99s%20island%3A%20its%20chunk%20200%2C%20no%20404%2C%20an%20update%20offered) | declare `assets: { retain: { disk, builds } }` over a disk every web pod reads → [Deployment](Deployment#hashed-assets-across-a-deploy--assetsretain) |
+| A soft visit waits for the slowest hole | `As of 2026-11-02`. The client router reads `response.text()` ([`packages/render/src/navigation-fetch.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/render/src/navigation-fetch.ts)) and swaps the whole answer, so holes fill through the re-run scripts and the first paint of an in-app navigation is as late as a full `ssr` render. A full load streams. No incremental parse here [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/render/src/navigation-fetch.test.ts#a%20page%3A%20its%20text%2C%20its%20build%2C%20no-store%20read%20off%20cache-control) | nothing to do in an app; a page that must paint first on a soft visit declares `navigation: 'document'` |
+| A pod evaluates the framework from **source**, one module at a time — ~28 Mi of resident memory per pod that a bundle would not hold | `As of 2026-10-01`, measured on the framework serve graph alone: 1,071 modules from source settle at **80.4 Mi**; the same graph as one `bun build --target=bun` file is 5 modules and settles at **51.8 Mi** — ~34 kB of RSS per source module, the runtime's cost and not the code's. It is the largest saving left after the per-role load and the prebuilt store (a web pod of the demo app settles at 97–102 Mi, a worker at 80–85 Mi). **Not a patch**: a registry is a module instance, so the framework bundled apart from the app is two `@ultimat3/jobs`, two `@ultimat3/entity` — the app registers into one and the boot reads the other. The unit has to be the app **and** the framework as one graph, per role, built inside the image build from a generated entry that imports what `scanAppModules` would, with the `.tsx`/`.scss` loaders as build plugins. Containers only either way (axiom 7). Design note: [`docs/architecture/13-topology-runtime.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/architecture/13-topology-runtime.md#not-built-one-bundle-per-role) pin: none — a resident-memory measurement of a running pod, not a behaviour a unit test can assert | nothing to do in an app. Size a pod's memory request from `process_resident_memory_bytes` on `/metrics`, not from a guess |
+| Every role evaluates all of `@ultimat3/jobs` — 96 modules (`As of 2026-10-02`, by file) — whatever it runs | `As of 2026-10-01`, read off the serve graph by the names each role's code uses: a `web` pod needs 43 of them (enqueue, the outbox relay, the purge task), a `scheduler` 46, a `worker` 62. The rest ride the barrel: `worker.ts` and its eighteen dependencies in a web pod, the `backfill()` / `exportJob()` / `webhook()` factories (17 modules, 123 kB of source) in an app that declares none, and the memory driver (3 modules) outside a test. ~1.6 Mi per web pod at the measured 34 kB a module. Closing it means subpath entries (`@ultimat3/jobs/worker`, one per factory) and every importer rewritten — a breaking move for a saving the per-role bundle above makes moot, so it waits on that decision. The operator surface is **not** in this list: `JobIntrospection` is a member of the driver, and the worker announces itself and counts settles through it pin: none — a module count read off a role's serve graph, measured rather than asserted | nothing to do in an app |
+| A job's `runAt` is stamped from the enqueuer's clock | `As of 2026-10`. An enqueuer running ahead of the database delays its own "immediate" jobs; up to 1 s ahead still wakes a worker, beyond that the job starts on the poll. A delayed job enqueued by another process starts within the idle ceiling after its `runAt` — 5 s with a proven wake, 2 s without; only a retry or a sleep the worker itself handed back gets a due-time timer | NTP on every pod; for a deadline tighter than the ceiling, enqueue at the moment the work is due rather than with `runAt` |
+| `onSettled` runs at most once across a crash | `As of 2026-10`, by decision of the first cut: the hook runs after the settle, inline, by the worker whose settle landed. A worker killed between the settle and the hook does not run it, and nothing re-drives it | record what must be recorded for certain inside the body, in a `step.run`, and use `onSettled` for what may be lost |
+| The CDP driver's `allowHosts` judges NAMES, and a page's WebSocket is not screened at all | `As of 2026-10`. Measured on Chrome 150 with `localBrowser()`: interception (page-level and the browser-level `Fetch` that now covers popups) never pauses a WebSocket handshake, so `new WebSocket('ws://off-list/…')` from a scraped page connects. And the browser resolves its own names, so a host a wildcard admitted that resolves to a private address is reached from the browser leg — the HTTP leg and the robots read resolve and pin (`packages/scraping/src/pinned-host.ts`), the browser cannot be pinned through this port | list exact hosts rather than `'*'` on a scrape that runs where private addresses are reachable, and dial a scrape through an `egress` proxy outside that network |
+| `ScrapeReport.usage.bytesIn` counts the HTTP leg only | `As of 2026-10`. The CDP port subscribes to requests, not to their sizes, so the browser leg's bytes are not in the figure (`packages/scraping/src/usage.ts`) | treat `bytesIn` as a lower bound; a provider's own byte count, returned through the resolver's `cost`, is the billable one |
+| An org-scoped operator's runs list reads `x_jobs` with no index on `tenant_id` | `As of 2026-10`. The jobs dashboard keeps an actor with an `orgId` to its own org's runs through `JobFilter.tenantId`, and `x_jobs.tenant_id` is a plain column: the only index naming it is the idempotency one, led by `name` (`packages/jobs/src/driver-pg-ddl.ts`). A platform operator's list is unaffected [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/jobs/src/driver-pg-sql.test.ts#no%20x_jobs%20index%20leads%20with%20tenant_id%3A%20an%20org-scoped%20runs%20list%20scans) | on a large queue, add `create index on x_jobs (tenant_id, created_at desc, id desc)` in an app migration, or operate the dashboard as a platform operator (an actor with no `orgId`) |
+| Retry-from-step on the jobs dashboard is a free-text field | `As of 2026-10`. `job.retry-from-step` takes `{ step: t.string.min(1) }` (`packages/admin/src/jobs/job-actions.ts`); the run's detail page shows its steps, but the form does not offer them as a choice. | copy the step name from the run's `steps` section on the same page |
+| The admin MCP batch tool takes `ids` only | `As of 2026-10`. A `batch` action is one MCP tool, `admin.action.<name>`, and it takes the rows by id; "all matching" — the list's own filter — is a screen control only | list the rows with `admin.<entity>.list` and its `where`, then pass their ids |
+| No counts on the jobs dashboard's state tabs | `As of 2026-10`. `stats()` counts per queue across every tenant, so a count on a tab would show an org-scoped operator other orgs' numbers; the tabs carry none and the overview's tiles carry the counts, for a platform operator only. A per-tenant count is a second query nothing issues yet | a platform operator reads the overview tiles; an org-scoped operator reads a tab's own page |
+| `x drill` proves one node per role on one machine, and drills no object store and no CDN | `As of 2026-11-06`. The fleet is one `web`, one `worker` (and one `sync` for `nats-down`) on one host, every dependency a local container behind a Bun TCP proxy: a multi-node NATS cluster, a Postgres failover to a real standby, a real S3 bucket and a real CDN in front of the web role are never faulted. `nats-down`'s sync lane is proven on a stand-in node (`packages/testing/src/drill-sync.test.ts`), not on `examples/dummy`, whose transport is in-memory; its client subscribes to nothing by default, so a gap counts only what the node volunteers. CI runs three of the eight (`fault-drills`) [pin](https://github.com/developerz-ai/ultimate/blob/main/packages/testing/src/drills.test.ts#the%20eight%20drills%2C%20once%20each) | the rest by hand: `x drill --list --json`, then `x drill <name> --json` with `docker/test-services.env` loaded. A multi-node or object-store fault is a staging drill of your own, against [`docs/ops/06-runbooks.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/ops/06-runbooks.md) |
+
+## Open by decision
+
+Each of these is a defect somebody has already argued about, and the reasoning is why it stays.
+
+| Gap | Why it stays |
+|---|---|
+| A local write that died after its last rename is listed with no content type until the key is written again | `As of 2026-10`. `localDriver` commits in four steps (marker, sidecar, bytes, clear marker) and a marker left behind means "re-check this pair". `get()` and `stat()` do re-check against the bytes and answer correctly; `list()` never reads bytes, so it reports that object with no `contentType` and `etag: ''` — honest, and it stays that way until the next `put()` of that key. `stream()` returns bytes only and checks nothing. Two files cannot be committed in one step without changing the on-disk layout |
+| A `delete_resource` preview checks only the policy's actor half | `As of 2026-10-18`. Without `live: true` the framework answers a preview without running the action, so the part of a policy that reads the input never runs; a preview can be granted for a call the commit then refuses. Running the policy would mean running the action. An action that needs an exact preview declares `live` in its input and answers its own |
+| `localDriver` cannot hold a key and a key beneath it (`a` and `a/b`) | `As of 2026-10`. An object is a file at its key, and a POSIX path is a file or a directory. The second `put()` is refused by code (`X_STORAGE_KEY_CONFLICT`) where it used to be a bare `ENOTDIR`; `s3Driver` and `memoryStorageDriver` hold both. Storing objects under suffixed names would lift it, at the price of a layout migration for every existing root — and the keys `grantUpload` and `scopedKey` build never nest an object under another, so the limit is met only by hand-built keys |
+| `insert({ ...row })` of a row whose **nullable** `.sealed()` column was read from the repository stores NULL, silently | a sealed property on a repository row is not enumerable, so a spread leaves it behind (`As of 2026-10`). A required one is refused by name (`X_INVARIANT_VIOLATED`, fix `{ ...row, <col>: row.<col> }`); an absent nullable column is a legal insert, and nothing tells a spread from an omission without making every omission an error. Copy a sealed column by name |
+| A drift `fix:` for a table in `public` does not name its schema | `As of 2026-10-02`. A fix for a table read from any other schema carries `set search_path = "<schema>";`; one for `public` is left unqualified, so it relies on the pasting session resolving `public` first. Postgres' default path is `"$user", public`, so a schema named after the connecting role that holds a same-named table shadows it. Check `show search_path` in that session, or prefix the pasted statement with `set search_path = public;` |
+| Drift never compares an index's `where` **text** | the catalog answers its own rewriting of the expression (`(deleted_at IS NULL)`) where the snapshot holds the author's spelling (`deleted_at is null`), so comparing the strings reports drift on a database that is exactly right. Normalising them means shipping an expression parser to compete with the server's. Three of the four parts of an index **are** compared now — columns, uniqueness, direction, and the predicate's **presence** — so a partial index recreated as a total one and a `desc` index rebuilt ascending are both caught ([`packages/db/src/drift.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/db/src/drift.ts), `compareIndexes`). `x db gen` compares the text, where both sides are generated. A predicate rewritten by hand into a *different* predicate of the same presence is the residue, and it is invisible: regenerate and read the diff |
+| A long `backfill()` retains one step name per batch | `stepRunner`'s `claimed` Set is the authority for `X_STEP_DUPLICATE` and is deliberately unbounded: a 5M-row sweep at `batch: 250` holds 20,000 short strings (~1–2 MB) for the life of the attempt, released when it ends. `MAX_TRACE_NAMES` bounds the *reported* trace, not the membership set. Bounding the set would let a duplicate step name through after 200 batches, and silently replaying a step is the worse failure. Raise `batch` if the attempt is long enough for the retention to matter |
+| The admin has no one-click select-all | `As of 2026-10`, by decision: the admin ships zero JavaScript (`hydrate: 'never'`), so every control is a link or a native form. A row checkbox joins the batch form through its `form` attribute, "all matching" is a radio whose meaning is the list's URL, and an action's input is its own page. Selecting every row on one page is one click per row. An island would buy that click and cost the admin its zero-script rule, and "all matching" already covers the whole list |
+
+## Not built yet
+
+| Open | Where it stands |
+|---|---|
+| Multi-node realtime | the forced-restart benchmarks **are** measured, on one `sync` node over `InProcessTransport` — neither run crossed NATS and neither subscribes to a live query, so no cursor, snapshot or gap-repair path is under test. Fanout, throughput and per-node socket capacity across nodes remain targets → [Realtime](Realtime) |
+| `x db studio` | planned. `X_DB_STUDIO_FAILED` is reserved and not thrown; the command exits `X_NOT_IMPLEMENTED` pointing at `x dev` → the `/_x` db panel. Keeping a `bunx drizzle-kit studio` shell-out would have meant a second schema engine for one subcommand |
+| `api/**/route.ts` | not a file kind — owner decision 1 ([#648](https://github.com/developerz-ai/ultimate/issues/648)), `As of 2026-10`. `api/` holds the action and query projections `defineApi()` collects; a route file there is refused at registration (`X_ROUTE_FILE_INVALID`), and 26.0.0 deleted `ROUTE_FILENAME`'s `api` row ([#709](https://github.com/developerz-ai/ultimate/issues/709)). A wire format an action cannot speak — a webhook receiver, an OAuth endpoint — is a plain HTTP route in the `routes` runtime override ([Configuration](Configuration)) |
+| Deferred, `As of 2026-10-07` | the plugin API and multi-region replication. The Redis **job** driver shipped (`redisJobDriver()`, #710) without the operator surface, backfill bookkeeping or fleet-wide `concurrency` — those answer on Postgres only; the NATS job driver stub that threw `X_NOT_IMPLEMENTED` was deleted in 25.0.0. |
+
+## Carried backlog
+
+Rows plan 101 carried from superseded plans and sweeps 10a–10e left open. **The
+backlog row is the record** — cause, evidence and fix live there and in the issue; a row here says
+only what you meet and how to get past it. Backlog: [`10-carried-backlog.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/plans/2026/10/04/101-squeaky-clean-sweep/10-carried-backlog.md).
+
+None open, `As of 2026-10-07`: sweep 13 closed the last four — B24 (`agentJob` output, #685), B25
+(`webhook`'s endpoints and ledger, #686), B26 (`t.array` bounds and `transition()`'s row loader, #687)
+and B27 (one `PgExecutor` builder, `dbExecutor()`, #688), in PRs #701, #703 and #706.
+
+## Deferred by plan 101
+
+`As of 2026-10`. Gaps the platform-readiness work ranked and did not take, with the evidence that
+ranked them — a later plan's, not this release's. Evidence is from read-only surveys of a
+downstream app on Ultimate and of systems built without it; no row is a defect in what ships.
+
+| Gap | Evidence | Precedent |
+|---|---|---|
+| a native form's refusal answer — `setRedirect` covers success (`packages/http/src/redirect.ts`) | 234 wrapper calls in 142 files | Rails' form convention: 303 on success, 422 re-rendering the form with its errors; `flash` |
+| route-group defaults | 111 routes × ~25 repeated lines | layouts, `before_action`, controller inheritance |
+| a view-scoped lazy lookup for `t()` inside an island — the subset itself ships (`catalogSubset` / `subsetTranslator`, #710), named key by key | 94 label types, 3,092 lines of label files | lazy lookup `t('.title')` scoped to the view |
+| a server-first dialog, menu and combobox with a small enhancer | ~2,200 app-side lines; 13 catalog components at 0 imports | HTML first, a small controller second |
+| `QueryRef` derived from the declaration | [Queries and live queries](Queries-And-Live-Queries) "Not derived" row | — |
+| a `task` that runs without a companion job; a boot hook | 35 task/job pairs, 2 boot stand-ins | recurring tasks that name a command; initializers |
+| batches with completion callbacks, debounce, priority, job expiry, argument encryption | each present in a surveyed job system; none asked for by the proving case | batches and limiters in established job systems; queue priorities |
+| proxy inventory, captcha, OTP relay | product features of a scraping service — the app's, by [axiom 8](https://github.com/developerz-ai/ultimate/blob/main/docs/idea/19-mechanism-not-convention.md) | — |
+
+The test-kit row of that plan's table shipped in 23.0.0 work (`renderView`, `runJobs({ actor })`,
+`mountIslandState`) and is not repeated here.
+
+## Closed
+
+**Historical:** every row is a defect that shipped; the third column is for readers still pinned below
+the release that fixed it. Upgrading is the fix, and the breaking entries are the cost ([Upgrading](Upgrading)).
+
+**A row is published once the footer's version reaches its release**; `30.0.0` rows land with that major.
+
+| Gap | Fixed in | If you are pinned below that |
+|---|---|---|
+| `bun run error-render` could not see a `catch (error)` binding | **8.0.0** | The gate matched a **parameter** annotated `unknown`/`any`; a `catch (error)` binding is `unknown` by inference and carries no annotation, which is how `String(error)` echoing a caller's request body into a 422 `cause:` stayed green in `packages/http/src/request.ts` through 3.0.0. `error-render` now joins `catchRenderFindings` ([`scripts/catch-render.ts`](https://github.com/developerz-ai/ultimate/blob/main/scripts/catch-render.ts)), which reads the catch binding and a `.catch((error: unknown) => …)` callback; its header lists what it still cannot see (a helper in a third file, a field laundered by another package). Below it: review every `catch` by hand — a caught value reaches a `cause:`/`fix:` only through `renderThrowable(error)` from `@ultimat3/core` |
+| The manifest did not record an admin action's `readonly` or `matching`, a scope's `count`, or a resource's permission noun | **25.0.0** | [B29](https://github.com/developerz-ai/ultimate/blob/main/docs/plans/2026/10/04/101-squeaky-clean-sweep/10-carried-backlog.md), #691. `AdminActionFact` records `readonly` and `matching` and `x manifest diff` reports the gate moving between `admin:write` and `admin:read` ([`packages/manifest/src/diff-admin.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/manifest/src/diff-admin.ts)); a scope's `count` is diffed; the permission noun reaches the diff through each admin route's `permissions` (`<noun>:read`). Below it: read them from the declaration in `apps/admin/app/admin/admin.ts` |
+| The two-platform deploy proof (milestone 11) was undemonstrated | **22.1.0** | Not a defect in what shipped: the build targets, compose files and chart all shipped, the proof did not. `.github/workflows/ci.yml`'s `deploy-proof` job (`docker/deploy-proof/run.sh`) installs a scaffolded app into kind with `x deploy --method helm` and upgrades it under mixed GET/POST load, failing on one non-2xx ([roadmap](https://github.com/developerz-ai/ultimate/blob/main/docs/idea/14-roadmap.md)). Nothing to work around below it |
+| A withdrawn `isr` page was served once more, however old | **27.8.0** | A tag bust marked the stored page stale and the next request was answered that copy — per stored key, per replica — so a withdrawn article reached its next visitor, days later if nobody asked sooner. `revalidate: { tags, ttl, onInvalidate: 'purge' }` deletes the route's stored pages on a bust and evicts their renders in flight — the next request on that replica, and on each replica the broadcast reaches; one that misses it serves its copy until `ttl` and nothing stale after. `maxStale` bounds a copy served past its `ttl` ([Routes](Routes-And-Render-Modes#how-fast-a-purge-takes-a-page-down)). Below it: serve the route as `render: 'ssr'` with a `cache` age |
+| Any query string minted a stored `isr` page | **30.0.0** | The key carried the whole query, so `/blog?x=1`, `?x=2`, … were each a render and an entry in a store of 1,000, and `utm_*` split one page into many. A route with no `revalidate.query` now keys on none of its query, and `load`, `meta` and the page receive the narrowed URL. Below it: declare `revalidate: { query: ['page'] }` (or `[]`) on every `isr` route, which keys the page on those parameters only; or strip the query at the ingress, or take `render: 'ssr'` |
+| An `isr` page stored a 5xx | **27.7.0** | A `load` answering `withStatus(503)` was stored for the whole `ttl` plus one stale serve, and replaced the good page it was regenerating. A 5xx is now never stored and never replaces a stored page; `noStore(data)` is the loader's own "do not keep this". Below it: throw from `load` on a failed read — a throw was never stored |
+| A page another replica rendered into a shared `isrStore` was not invalidated | **27.8.0** | The invalidation graph only held the pages this process rendered, so a bust arriving on a replica that had merely served a hit found nothing. A bust now also asks the store which keys it holds under the tags, and a store that implements `IsrStore.tagFence` refuses a write from a render that began before the bust on another replica. Below it: give every `isr` route a `ttl` |
+| `/sitemap.xml` answered a 500 when `prerender()` failed at the hourly refresh | **27.7.0** | The running web role forgot the previous sitemap before it re-enumerated. The hourly refresh now runs behind the last good answer, which every request keeps getting; a failed or hung one (30 s) is asked again a minute later, and the third in a row is logged as an error. A `prerender()` that **catches** its failed read and returns `[]` still shrinks the sitemap — let it throw |
+| The edge was purged before the origin dropped the page | **27.7.0** | `invalidateTags` cleared the `cdn` tier first, so a request between the CDN purge and the origin's own delete was a public hit the edge cached again for a whole `s-maxage`. The order is now read tiers → ISR pages → broadcast → `cdn` last. Below it: follow a withdrawal with a second `x cache bust` |
+| A withdrawn `isr` page stayed readable at another spelling of its URL | **27.8.1** | The store key was the raw pathname and the router matches a normalised one. `/blog//a` and `//blog/a` were answered with the page and stored under a key the route table could not read back: no `ttl`, no tags, `s-maxage=60` with a day of `stale-while-revalidate` and no purge key — a tag bust, `'purge'` included, never reached it, at the origin or at the edge, for the life of the process. The key is now the address the router matched ([Routes](Routes-And-Render-Modes)), and a key no route owns is never stored. Below it: normalise the path at the ingress (collapse `//`, strip the trailing slash) or 404 a path containing `//`; restart the pods after a withdrawal |
+| Every spelling of an `isr` URL was its own stored page | **27.8.1** | `/blog/%61bc`, `/blog/abc/` and `/blog/abc` were three entries of one page: a 20-letter slug has a million percent-encodings, against a store of 1,000 with the least recently generated evicted first — any visitor could push the real pages out — and the document stored for a spelling carried that spelling as its `canonical`. A served `static` page's memo (1,024 documents) filled the same way. One entry per address now. Below it: normalise or reject non-canonical paths at the ingress |
+| Every `isr` page under a `basePath` was tag-only and untagged | **27.8.1** | The key kept the mount (`/shop/blog/a`) and the route table has never heard of it, so no route matched: the declared `revalidate.ttl` and `tags` were ignored (`s-maxage=60`, no purge key, no tag edge) and a bust reached nothing. The key now carries no base path. Below it: serve the app at `/` behind a path-rewriting proxy, or take `render: 'ssr'` with a `cache` age |
+| A locale-prefixed `isr` page never revalidated | **27.7.0** | The ISR key keeps the `/en` prefix and the route lookup did not strip it, so `/en/blog/a` matched no route: no `revalidate.ttl` (`s-maxage=60`, the tag-only default), no tag edge. A tag bust and the TTL both expired `/blog/a` and left `/en/blog/a` — a withdrawn post, or a 404 stored before publication — served for the life of the process. Below it: serve the affected route as `render: 'ssr'` with a `cache` age, or restart the pods after a write that should reach a prefixed page |
+| The page outbox did not expose how many writes it holds | **25.2.0** | `useOutbox().size` (`@ultimat3/realtime`) counts the writes queued and not yet taken by the server, and follows every enqueue, ack, refusal and principal change. Below it: count `useMutation`'s `undefined` answers yourself and clear on the browser's `online` event |
+| A tag-only ISR page never went stale | **24.0.0** | No boot attached its ISR controller, so `invalidateTags` reached no revalidator and a `revalidate: { tags }` page with no `ttl` served its first render for the life of the process — on every pod, while `report.isr` listed it as revalidated. `x dev`, the container's web role and `appRoutes()` now attach the controller they build (`packages/cli/src/runtime-isr.ts`) and detach it on stop; [`isr-tag-bust.e2e.test.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/cli/e2e/isr-tag-bust.e2e.test.ts) proves `x-ultimate-isr: stale` then the new body. Below it, give every `isr` route a `ttl` as well as its `tags` |
+| The ISR cache key was the **pathname**, so a query string was ignored | **4.0.0** | the key is `isrKey(url, locale)` — pathname, the negotiated locale, then the sorted query — so `?q=bob` and `?q=alice` are two documents (declare the parameter: `revalidate: { query: ['q'] }`). Below it, put the varying part in the path (`/search/[q]`) or take `render: 'ssr'` |
+| `invoke()`'s cache bust was not transaction-aware | **24.0.0** | An action invoked inside `withTransaction` busted `cache.invalidates` when its handler returned — before the commit — so a concurrent read refilled the entry from the pre-commit row and the stale value survived to its TTL, and a rollback busted for a write nobody made. The bust is now handed to the root transaction's `onCommit` ([`packages/action/src/cache-gate.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/action/src/cache-gate.ts)): it fires once the root `COMMIT` is answered, never for a rollback or `X_DB_TRANSACTION_ABORTED`. Below it: do the transactional work *inside* the handler, or drop `cache.invalidates` from the declaration and call `invalidateTags` yourself after the transaction returns |
+| On an s3 disk a grant's `maxBytes` bounded nothing at the upload | **30.0.0** | `grantUpload` requires `request.size` and an s3 disk signs it as the exact `Content-Length`, so the provider answers any other length, chunked included, with 403 `SignatureDoesNotMatch` (proven on the test stack's gateway, `driver-s3.live.test.ts`). `signedUrl({ maxBytes })` with no `contentLength` is refused there (`X_STORAGE_UPLOAD_SIZE_UNDECLARED`) instead of ignored |
+| `X_ROUTE_MODE_INVALID`'s fix offered a `<Suspense>` boundary that does not work | **30.0.0** | It read "wrap the data-dependent part in `<Suspense fallback={…}>` or change render to `'ssr'`", and Solid's `<Suspense>` throws `getContextId cannot be used under non-hydrating context` under this renderer. The boundary check is deleted: a `stream` page needs no boundary, a page with no `<Hole>` is one flush, and the slow part streams through `<Hole fallback>` from `@ultimat3/render`. Below it: take `render: 'ssr'` |
+| `x g` wrote hand-wrapped source and never formatted it | **30.0.0** | `x g` runs the app's own `node_modules/.bin/biome check --write` over the TypeScript files that run created, under the app's config, best effort (`generate-format-written.ts`). A template wrapping wrong for a name no fixture picks reached the app's `lint` as a red gate over code nobody typed. Pinned below it: `bunx biome check --write` over the files the run listed |
+| A deployed scaffold issued no session, so every policy-protected route answered 401 | **30.0.0** | Nothing `x new` wrote called `login()`, and the development viewer answers only under `x dev` — anonymous everywhere else. `x g sign-in --password` or `x g sign-in --oidc <provider>` now writes the sign-in page, the session actions, `auth.signInPath` and a boot guard (`X_AUTH_SIGN_IN_UNCONFIGURED`) → [Auth](Auth). Below it: write the action yourself — `policy: allow('public')`, `const signedIn = await login(auth, { email, password, ip })` from `@ultimat3/auth` over the app's `defineAuth(…)`, then `useRequestContext('createSession').headers.append('set-cookie', signedIn.cookie)` — and post a native form at it |
+| Nothing pinned the documented generator file counts | **4.0.0** | `scripts/generator-counts.ts` reads every page that states how many files `x new` or `x g resource` writes and answers `X_DOC_FILE_COUNT_STALE` on the gate's `manifest` step. Below it, derive rather than quote: `x new myapp --dry-run --json \| jq '.data.files \| length'` |
+| `SessionInit.proxy` was declared on the scrape seam and read by nothing | **23.0.0** | `scrape({ egress: (input) => proxyUrl })` now reaches the driver as `SessionInit.proxy` and wins over the driver's own `proxy`: the launched browser, the HTTP leg and the robots read all dial it, an offline session reports it, and a driver that cannot dial it refuses with `X_SCRAPE_EGRESS_UNSUPPORTED`. On 22.x the exit is a **driver** option — `localBrowser({ proxy })` / `remoteBrowser({ proxy })` — one for every session → [Scraping](Scraping#a-session-a-service-is-built-on) |
+| No test file was typechecked | **24.0.0** | All **31** package `tsconfig.json`s carry `"exclude": ["src/**/*.test.ts"]` — they still do — so `bun run typecheck`, a `tsc -b`, reads none of the **2,195** test files (both counts `As of 2026-10-06`) under `packages/*/src` and the gate's `typecheck` step was green over every one. Re-derive both: `grep -l 'src/\*\*/\*.test.ts' packages/*/tsconfig.json \| wc -l` and `find packages -path '*/src/*' -name '*.test.ts*' -not -path '*/dist/*' \| wc -l`. **Closed by a second program, not by editing the 31:** `tsconfig.tests.json` compiles the tests with `noEmit`, `scripts/test-typecheck-gate.ts` runs it on a per-package ratchet that may only fall, and it rides the gate's `manifest` step ([`scripts/verify.ts:314`](https://github.com/developerz-ai/ultimate/blob/main/scripts/verify.ts)) rather than `typecheck`, which takes no host findings. It landed at **446 errors over 161 files in 27 packages** (measured 2026-08-19) and `TEST_TYPECHECK_PINS` in `scripts/lib/test-typecheck-pins.ts` is now `{}` `As of 2026-10-06`: every package's tests compile at zero errors, so a new test compiles the day it is written. On any published release: nothing to work around at runtime, the tests run either way. Typecheck one package's tests yourself with `bunx tsc --noEmit` over a copy of its config with the `exclude` dropped |
+| A live query re-delivered the raw table row, and mis-ordered a projected window | **5.0.1** | Two defects with one cause: a `ChangeEvent` carries the whole TABLE row, and a live query's result set is whatever its `sql` returned. **The leak** — every patch forwarded the change row unnarrowed, so a column the projection dropped went out on the socket the moment it CHANGED; `examples/dummy`'s feed projects ten columns and one publish delivered `updatedAt`, and a column like a salary or a private note would have gone the same way. The per-subscriber gate could not help: it decides whether a ROW is delivered, never which of its columns. **The mis-ordering** — `match()` decided position by comparing the change row against the rows the WINDOW holds, so an `orderBy` on a column the projection omits measured a real value against nothing: every update read as a move, and an arriving row landed wherever `undefined` sorted. Now a patch row is narrowed to the columns the query actually returned, and a position the window cannot answer for is a `refill` — one re-read and a re-snapshot — rather than a guess. A DELETE still patches incrementally, because it decides no position. [#230](https://github.com/developerz-ai/ultimate/issues/230). On 5.0.0 and below: give a live query's rows the key they are ordered by, and do not rely on a projection to withhold a column from a live subscriber |
+| `jobs.driver` selected no driver | **5.0.0** | [`JobsConfig.driver`](https://github.com/developerz-ai/ultimate/blob/main/packages/core/src/config.ts) accepted `'postgres' \| 'redis' \| 'nats'` and had **no reader anywhere** — boot always built `postgresJobDriver`, and [`packages/jobs/src/driver.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/jobs/src/driver.ts)'s own header already said so. `jobs: { driver: 'redis' }` therefore did not boot-and-then-throw as this wiki once claimed: it changed nothing and you silently got Postgres — the same shape as `realtime.heartbeatMs`, and worse, because it failed silently in the dangerous direction. Five shipped `fix:` lines named it as the repair for `X_NOT_IMPLEMENTED`, which is a `fix:` that is a no-op; those were corrected in 4.1.0 and the field itself is **deleted in 5.0.0**, along with the `JobsDriver` type nothing else used. On 4.1.0 and below the field still typechecks and still does nothing — swap the driver with `setJobDriver(postgresJobDriver({ executor }))`, or `setJobDriver(memoryJobDriver())` in a test, and never through `app.config.ts` |
+| A caller-controlled `string` could add a line to the 3-line error format | **5.0.0** | `bun run error-render` refuses a parameter typed `unknown`/`any`; a value already typed `string` renders without throwing, so nothing objected — while a newline in one writes a second line an operator, a CI log or the dev overlay's `<pre>` reads as a genuine framework message. Three holes shipped in `@ultimat3/auth` under a green check, the worst reachable by an unauthenticated stranger with one crafted OIDC token. The first fix escaped at each of the six RENDERERS, which could not hold: six is a number that only goes up, and it covered none of the renderers an app writes. Now `UltimateError`'s and `SchemaError`'s **constructors** escape `code`, `title`, `cause`, `fix` and `docs` — so `.message`, `.cause`, `format()`, `toJSON()` and any renderer anyone writes are one line by construction, and `singleLine()` is idempotent so a call site that already escaped is unharmed. On 4.1.0 and below, pass `error.cause` through `singleLine()` from `@ultimat3/core` before you render it yourself |
+| `on delete` was declared and reached no SQL | **4.0.0** | `references(() => orgs.id, { onDelete: 'cascade' })` type-checked and the rule was dropped one layer below the declaration, so no generated `add constraint` ever spelled one and a drift check had nothing truthful to compare. Now: `ColumnDescription`/`ReferenceDescription` carry `onDelete`, `addForeignKey` writes the clause, and a rule changed on either side is `changed-foreign-key` drift whose `fix:` is the drop/add pair. On 3.0.0, add the clause to the `add constraint` the generator emitted, **before** applying the migration: `… references "orgs" ("id") on delete cascade;`. Editing it after it applies moves the checksum → `X_MIGRATION_CONFLICT` |
+| A `references()` **removed** from a column emitted nothing | **4.0.0** | the key stayed on the database, the snapshot beside it recorded `foreignKeys: []` — actively denying a constraint the catalog held — and `compareForeignKeys` judges the declared side, so no check could see it. Now `foreignKeyPlan` emits the `drop constraint` in `up` and the `add` in `down`, naming the constraint the **previous snapshot** recorded rather than the name the generator would have picked. On 3.0.0, write the drop into the next generated migration by hand before applying it: `alter table "posts" drop constraint "posts_org_id_fkey";` |
+| Drift ignored an index's direction and its predicate's presence | **4.0.0** | a `desc` index rebuilt ascending served a feed's newest page off the wrong end, and a partial index recreated as a total one silently widened the constraint — both read `ok: true`. Both are compared now; `asc` is normalised to `null` first, since Postgres stores an ascending index as not-descending. The predicate **text** is still uncompared and always will be — see [Open by decision](#open-by-decision). On 3.0.0, `x db gen` compares all five fields, so regenerate and inspect the diff |
+| An unknown flag was refused **before** a planned command could answer honestly | **4.0.0** | the parser read flags against the spec first, so `x logs tail --follow` reported `X_CLI_BAD_FLAG` instead of `X_NOT_IMPLEMENTED`. A planned command now answers with its own status whatever flags it was given; a **shipped** command still refuses an unknown flag, which is correct — `x env` is shipped and declares `--json`, `--help`, `--cwd` and `--verbose`, so anything else is `X_CLI_BAD_FLAG` for the ordinary reason that no such flag exists. On 3.0.0, run the flagless form of a planned command to see the real message |
+| `x deploy --method helm` threw `X_NOT_IMPLEMENTED` in a scaffolded app | **4.0.0** | `x new` wrote no `docker/helm`, and the command carried a "does this build implement helm?" branch over a build that implemented it completely. `x new` now writes the chart — 8 files, `Chart.yaml` + `values.yaml` + 6 templates — and the lying branch is deleted, so a missing chart is helm's own error. The framework repo's own `docker/helm` carried two more templates the scaffold did not (`pdb.yaml`, `servicemonitor.yaml`); `As of 2026-11` only `pdb.yaml` — the scaffold writes `servicemonitor.yaml` and `alert-rules.yaml`, and `x g chart-alerts` backfills them. On 3.0.0, copy [`docker/helm`](https://github.com/developerz-ai/ultimate/tree/main/docker/helm) from the framework repo, or use `--method compose` |
+| `.env.development` **and `.env.production`** shipped in the image | **4.0.0** | the scaffold's `.dockerignore` excluded `.env` and `.env.*.local` — neither pattern matches `.env.production`, which is the file `docker-compose.prod.yml`'s `env_file:` tells the operator to create. This page said the leak was "harmless as generated"; **that was false**, and a real `docker build` proved both files land in a layer. Now `**/.env` + `**/.env.*` + `!**/.env.example`, in the framework's file, both tracked apps' and the one `x new` writes. On 3.0.0, add those three lines to `docker/Dockerfile.dockerignore` yourself — and rebuild, because an image already built still carries them |
+| `realtime.heartbeatMs` was read by nothing | **4.0.0**, and BREAKING | the key sat in `RealtimeConfig` with a default of 15 000 and no reader anywhere. It is **deleted**: `RealtimeConfig` is `{ enabled, tier, transport, urlEnv }`, the socket beat is the client's `new LiveClient({ heartbeatMs })` and the presence beat is derived (`PresenceRegistry.heartbeatMs` is `max(1000, floor(ttlMs / 3))`). **Refused at boot since 25.0.0**: `defineConfig` throws `X_CONFIG_INVALID` naming `realtime.heartbeatMs` and its replacement (`REMOVED_CONFIG_KEYS`, `packages/core/src/config-removed.ts`), and any other key it does not declare is refused too (`refuseUnknownKeys`, `packages/core/src/config-keys.ts`). From 4.0.0 until 25.0.0 there was no runtime refusal — `section()` copied the key through and only `TS2353` at typecheck caught it. On 3.0.0, delete the key; setting it changes no behaviour either way |
+| `@ultimat3/action`'s `stableStringify` folded `-0`, `NaN` and `Infinity` together | **3.0.0** | this page asked for work that was already done. The hash canonicalizer was split out and moved **down**: `canonicalJson` and `fingerprint` live in `@ultimat3/core` (tier 0), are injective, and give `NaN`, `±Infinity` and `-0` bare tokens of their own plus tagged forms for `Date`, `Map` and `Set`. `requestHash` is `fingerprint(input)`, the job dedupe key is `action:<name>:<fingerprint>`, and `@ultimat3/query`'s copy was **deleted** rather than fixed. `stableStringify` survives as the OpenAPI **document** serializer alone, where emitting valid JSON is the requirement |
+| `.job()` produced a handle nothing could enqueue | **3.0.0** | `agentJob()` in `@ultimat3/ai` is the shipped bridge — an agent as durable, resumable, budgeted background work, with the queue accepting the handle. Pinned by `packages/ai/src/agent-job.test.ts`, whose test is named for this gap: *"agentJob() produces a handle the queue accepts, where .job() never could"* |
+| A branch reaper could drop **another app's** databases | **4.0.0** | `listBranches()` walks `pg_database` for the whole server, so two Ultimate apps on one Postgres plus one nightly sweep was the other app's branches dropped by a `DROP DATABASE` nobody asked for. The marker is now `<base>:<iso>` — `BranchInfo` carries `base` — and `reapBranches` skips any branch whose base is not this database. A **pre-3.x** marker records no base, so it is skipped too, never dropped: self-healing with no migration, because the next `createBranch` writes the base down. On 3.0.0, run the reaper only against a server this app owns |
+| `x db branch list` could not see a branch made before the `psql` shell-out was removed | 2.0.0 | the old path issued `CREATE DATABASE … TEMPLATE` through `psql` and wrote no comment, so a branch made by a 1.2.0 or earlier CLI is absent from `list` — and `delete` may only remove what `list` shows. Nothing back-fills the comment. Drop it by hand from the string the refusal names: `psql "$DATABASE_URL" -c 'DROP DATABASE "<source>_branch_<slug>"'`, where `<slug>` is the branch name with every character outside `[A-Za-z0-9_]` replaced by `_` |
+| MCP tool names were published in `snake_case` and served verbatim | 2.0.0, and BREAKING | `openapi.json` carried `"mcpTool": "publish_post"` while the only name `tools/call` accepts is the export name, `publishPost` — 15 of the 17 `mcpTool` values in the two tracked apps' committed specs named no served tool. `toToolName` is deleted from `@ultimat3/action` and `@ultimat3/query`, all three publishers spell the export name, and `packages/mcp/src/cross-surface.test.ts` drives a `tools/call` with the name OpenAPI published. On 1.2.0, ignore the published value and call the **export name** — `x actions list --json` prints it as `name` → [MCP and AI](MCP-And-AI) |
+| Shared cache tier invalidation `DEL`ed keys it never declared in `KEYS` | 2.0.0 | it failed on **Dragonfly** and on **Redis Cluster** — a cluster cannot route a key it was not told about. The script returns the member list and the tier deletes value keys client-side, one key per `DEL`, so every delete is slot-local. On 1.2.0, single-node Redis, or a cache tier that is not the shared one → [Caching and invalidation](Caching-And-Invalidation) |
+| `resolveEnvironment` existed twice, with different return types | 2.0.0, and BREAKING | `@ultimat3/core` and `@ultimat3/seo` both exported it. seo's is deleted along with `SeoEnvironment`; `RobotsConfig.environment` and `isIndexable()` take core's `Environment`, `'preview'` is spelled `'staging'`, and core gained `tryResolveEnvironment()` for callers that must answer rather than throw — `ULTIMATE_ENV` is in no env schema, so a `robots.txt` render is routinely its first reader. On 1.2.0, import one with an alias → [Configuration](Configuration) |
+| Two head serializers, one of them weaker | 2.0.0, and BREAKING | `@ultimat3/seo`'s `renderHeadTags` escaped `</` only — not `<!--<script>`, which moves the tokenizer into script-data-escaped state — and applied that code rule to a JSON body; **nothing called it**. `@ultimat3/render`'s `renderHead`, the path every `x dev` and every build takes, emitted `<script>` content entirely raw. `renderHeadTags` is gone and `render` escapes raw-text content and JSON-LD by each element's own rule. On 1.2.0, render the head through `renderHead(headFromMeta(meta, seoRenderers()))` and treat any string interpolated into `meta.ld` as untrusted |
+| `docker-compose.prod.yml` paired a published host port with `replicas: 3` | 2.0.0 | one host port has exactly one binder, so the second container died on `Bind for 0.0.0.0:3000 failed: port is already allocated`. `web` and `sync` declare `replicas: 1` in all four files. `sync`'s `PORT` was wrong in the same file and fixed with it — the role binds `PORT + 1`, so `PORT: 3001` opened 3002 while publishing 3001. On 1.2.0, set `replicas: 1`; to scale anyway, drop `ports:` and put your own reverse proxy on the compose network, or climb to the chart's per-role HPA → [`docs/idea/17-scale-ladder.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/idea/17-scale-ladder.md) |
+| `X_MIGRATE_CONCURRENT` was reserved and never thrown | 2.0.0 | the lock was a blocking `pg_advisory_lock`, so an overlapping migrator waited **forever** — no timeout, no exit code, and a wedged predecessor held `helm upgrade --wait` inside one statement with nothing in the logs. `acquireLock` now polls `pg_try_advisory_lock` once per 500ms against a 60s budget (`MIGRATION_LOCK_WAIT_MS`). On 1.2.0, kill the wedged migrator by hand |
+| `x db gen` / `x db migrate` shelled out to `bunx drizzle-kit` | 2.0.0 | which `x new` neither installed nor configured, so `bin/setup` — the scaffold's own documented first command — failed on "drizzle.config.json file does not exist". `gen` calls `generateMigration()` and `migrate`/`reset` call `migrate()`, the engine `ROLE=migrate` already ran. On 1.1.0, generate with `generateMigration` + `writeSchemaHash` from `@ultimat3/db` directly, then apply with `ROLE=migrate bun apps/web/server.ts` |
+| `x new` wrote `0000_initial.sql` with no snapshot sidecar | 2.0.0 | so a scaffolded app started on the one state `x db gen` refuses: `x db migrate` applied it and then answered `X_DB_DRIFT` naming `x db gen`, whose own `X_MIGRATION_SNAPSHOT_MISSING` said "restore from version control" for a file version control never had. `x new` writes no migration at all now — `x db gen` is the one writer of `packages/db/migrations`. On 1.2.0, **before the first migrate**: `rm packages/db/migrations/0000_initial.sql packages/db/migrations/0000_initial.hash && x db gen "initial"` |
+| A generated migration's foreign keys could not apply | 2.0.0 | every key was a `references` clause **inside** `create table`, walked in the app's import order — which says nothing about which table a key points at. Measured against PGlite: `create table "comments" (… references "posts" …)` ran before `create table "posts"`. Every key is its own `alter table … add constraint` after all tables now, and `down` drops constraints before tables. On 1.2.0, move each clause out by hand before the migration is applied |
+| The generated `.snapshot.json` failed the app's own `lint` | 2.0.0 | `JSON.stringify(value, null, 2)` never collapses a one-element array and Biome always does, so `x verify` answered `X_LINT_FAILED` on a file no author typed. The serialiser is a fixed point of Biome 2.5.5 at `lineWidth: 100`, and `x new`'s `biome.json` excludes `**/migrations`. On 1.2.0, add `"!**/migrations"` to `files.includes` |
+| Composite indexes emitted one mangled column name | 2.0.0 | `indexes: [{ on: ['orgId','createdAt'] }]` emitted `on "todos" ("org_id_created_at")`, which will not apply. The description carries the column list, the predicate and the direction, and the generator spells all three. On 1.2.0, write the index by hand |
+| A migration with two statements failed to apply | 2.0.0 | *cannot insert multiple commands into a prepared statement* on the embedded database, and against a server the moment the text carried a bound value. `migrate()` and `rollback()` split with `statementsOf()` and send one statement at a time inside the same transaction. On 1.2.0, one statement per migration file |
+| Every generator but `x g resource` imported files only `x g resource` wrote | 2.0.0 | the templates opened with `import * as repo from '../repo'`, so the generated file did not load: `X_CLI_UNEXPECTED` from every registry command, `--feature <slice>` included. Each generator composes the slice modules its own source imports now, and a module the slice already has is skipped rather than overwritten, `--force` included. On 1.2.0: `x g resource <slice>` **first**, then the narrower generator with `--feature <slice>` |
+| Jobs and tasks registered anonymously | **4.0.0** | a fresh scaffold had no `apps/web/api/index.ts`, so they registered as `anonymous-job-2` / `anonymous-task-1`. `x new` writes it. On 3.0.0, add `defineApi` yourself → [4 · Jobs and realtime](Tutorial-04-Jobs-And-Realtime) |
+| Generated tests landed in the wrong `x verify` step | **4.0.0** | `contractTest` / `liveTest` / `jobTest` inside a plain `*.test.ts` ran under `unit`, and `x test contract` answered `X_TEST_NO_FILES`. The generators emit the typed filename now — `<name>.contract.test.ts` for `x g action`/`x g mutator`, `<name>.live.test.ts` for `x g query --live`, `<name>.job.test.ts` for `x g job`, `x g task` and `x g backfill`. On 3.0.0, rename by hand: the filename is the type |
+| A disk registered under a name that is not its driver's 404'd its own signed URLs | **4.0.0** | `localDriver` minted `/_storage/<driver>/<key>` while the mounted route resolves the segment through the **registry**, so a disk registered as `uploads` signed a URL nothing served. `StorageDriver.registerAs(diskName)` is called by `defineStorage` at boot and the driver hangs its URLs off that; `signedUrlBase` is read off the driver by the verifying half, so the minter and the verifier cannot state it twice. On 3.0.0, register the local disk under the driver's own name (`local`) |
+| `acceptSignedUpload` was reachable from no route | **22.2.0** | the framework mounted only `GET /_storage/:disk/*key`, so every `grantUpload` URL answered 405 (#145, #523). `storageRoutes()` now mounts the `PUT` beside it in `x dev` and `runRole`. It answers `201 { key }` and validates against the grant's own signed constraints. On 22.1.0, mount the `PUT` in your app around `acceptSignedUpload`, or upload through a multipart action |
+| `/_storage` served only the host's disk | **22.2.0** | disks an app declared with `defineStorage` signed URLs that 404'd, and an app with no `storage:read` got a runtime `500 X_PERMISSION_UNKNOWN` (#524). The routes now read the process's one registry, which is the app's when it declares one, and `x verify`'s `policy` step reports the missing permission. On 22.1.0, return the bytes from an action |
+| The robots.txt read left from the worker's IP while every page load left through the proxy | **4.0.0** | two client identities presented to one origin — and an origin reachable **only** through the proxy answered nothing, which the gate reads as "no restrictions". `ScrapeSession.proxy` reports the exit the driver resolved, and `robotsGate` takes it as a **resolver** (`proxy: () => sessionProxy`) because the gate is an argument to `driver.open()` while the exit is decided inside it. The read also carries the run's deadline, its cancellation and a 500 KiB cap. On 3.0.0, use `robots: { ignore: '<reason>' }` on a proxied scrape, or accept that an unreadable robots.txt reads as allow |
+| A malformed request body echoed itself into the 422 | **4.0.0** | the parser's own message quotes the bytes it choked on, and it reached the `cause:` and the log store through `String(error)`. The caller-facing half now names the format alone — *could not parse the body as JSON* — and the parser's message rides in `meta` through `renderThrowable`. On 3.0.0, do not log a 422's `cause` verbatim into anything a third party reads |
+| `rateLimiter({ now })` | **4.0.0**, and BREAKING | renamed to `({ clock })`, the same `Clock` shape `requestContext`'s `init.clock` takes — a second spelling of "what time is it" is a second way to set one number. **The edit:** `rateLimiter({ config, now: () => t })` becomes `rateLimiter({ config, clock: { now: () => new Date(t) } })` |
+
+Reserved-but-unthrown error codes are listed in full under [Error codes → Reserved codes](Error-Codes#reserved-codes).
+Symptom-first triage is [Troubleshooting](Troubleshooting).