@ultimat3/cli 29.3.2 → 30.0.0

package/wiki/Timezones-And-Dates.md

@@ -0,0 +1,194 @@
+# Timezones and dates
+
+Store UTC. Format at the edge with `Intl.DateTimeFormat` and an **explicit IANA `timeZone`**.
+
+A `Intl.DateTimeFormat` or `toLocaleString` call with no `timeZone` option is a lint failure and fails `x verify` (check 2). It silently uses the process `TZ`, which in a container is `UTC` and in a laptop is not — the same code renders two different days depending on where it runs.
+
+## Storage
+
+| Rule | Detail |
+|---|---|
+| Column type | `timestamptz`, always. `timestamp` (without zone) is a build error in a generated migration |
+| Stored value | the UTC instant. Postgres normalizes on write; the session `TimeZone` is `UTC` on every role |
+| Never stored | a formatted string, a local wall-clock time, or an offset in a separate column |
+| Date-only values | `date` column for a true civil date (birthday, invoice date). A `date` has no instant and is never converted |
+| Wall-clock intent | when a user means "9am in their city", store `timestamptz` **plus** the IANA zone in its own `text` column. An offset (`+02:00`) is not a zone — it does not survive DST |
+| Zone strings | IANA identifiers only: `Europe/Berlin`, `America/Sao_Paulo`. Never `CET`, never `GMT+2`, never `PST` |
+
+## The actor's zone
+
+```ts
+import { currentTimeZone } from '@ultimat3/time';
+```
+
+`resolveTimeZone` reads, first valid IANA name wins, canonicalized; an invalid one is skipped, never thrown. As of 2026-11:
+
+| Source | Precedence |
+|---|---|
+| the member's saved zone (`actor.tz`) | 1 — what a person chose |
+| `x_timezone` cookie, written by a zone picker | 2 — chosen |
+| `?tz=Europe/Berlin` | 3 — share links, email previews |
+| `x-timezone` header, sent by a client from `Intl` | 4 — what the browser is installed as |
+| `x_timezone_detected` cookie, written by the zone script under `time.detect` | 5 — a guess, never written over a pick |
+| `configureTime({ defaultZone })` | 6 — `UTC` unless the app says otherwise |
+| Process `TZ` | never |
+
+### Detecting a new visitor's zone
+
+A document navigation sends no `x-timezone`, so a new visitor's server-rendered `app/` pages render in the default zone. `time: { detect: true }` in `app.config.ts` (off by default: it writes a cookie) puts a 153 B inline script on every `app/` document — never on `site/` — that writes `x_timezone_detected` from `Intl.DateTimeFormat().resolvedOptions().timeZone`. The first paint stays the default; every page after it is in the visitor's zone. The script is admitted by its own CSP hash, and it never touches the chosen `x_timezone` cookie.
+
+```ts
+import { defineConfig } from '@ultimat3/core';
+
+export const config = defineConfig({ name: 'my-app', time: { detect: true } });
+```
+
+The zone rides the ALS request context alongside the locale, so `format*` helpers take neither argument. Jobs and scheduled tasks carry the enqueueing actor's zone in the payload — a worker has no request and must not invent one.
+
+## Formatting
+
+| Helper | Output | Notes |
+|---|---|---|
+| `formatDate(instant)` | `2026-07-26` in the actor's zone + locale | `Intl.DateTimeFormat`, `dateStyle: 'medium'` by default |
+| `formatTime(instant)` | `14:30` | 12/24h from the locale, never hardcoded |
+| `formatDateTime(instant)` | date + time | one formatter, cached per (locale, zone, style) |
+| `formatRelative(instant)` | `3 days ago` | `Intl.RelativeTimeFormat`; computed against the frozen clock in tests. From a day apart the number is **calendar days in the zone** — 47 hours ahead across two midnights is `in 2 days`, never `tomorrow` — so it takes `zone` like every other formatter |
+| `formatRange(a, b)` | `Jul 24 – 26, 2026` | `formatRange` on the native formatter, so the collapse rules are CLDR's |
+| `formatZone(zone)` | `Central European Summer Time` | `timeZoneName: 'long'` |
+| `zonedParts(instant, zone)` | `{ year, month, day, hour, … }` | for arithmetic that must respect civil calendars |
+
+All of them read locale and zone from the context and pass `timeZone` explicitly to `Intl`. There is no variant that omits it.
+
+### Pickers
+
+`As of 2026-11`. A calendar date the user picks is a `PlainDate` — `DatePicker`, `DateRangePicker`
+and `Calendar` take and post `YYYY-MM-DD` and know no zone. Month and weekday names are
+`Intl.DateTimeFormat` with `timeZone: 'UTC'` on the plain date's UTC midnight, which names the date
+and never shifts it; the week starts on the locale's day (`Intl.Locale#getWeekInfo`). A *window* is
+instants: `timeRangeUrl(url, { presets, default, timeZone, now })` reads a custom `from`/`to` as
+wall-clock time in `timeZone` (a spring-forward gap resolves forward) and `TimeRangePicker` requires
+the same `timeZone` as a prop — omitting it is a type error. "Today" is the caller's: pass `month`
+(or `now`) computed with `plainDateIn(now(), actor.tz)`; with none, `Calendar` opens on today in
+`useUi().timeZone`, the request's zone, never the device's.
+
+## Common mistakes
+
+| Mistake | What breaks | Fix |
+|---|---|---|
+| `new Date().toLocaleDateString()` | uses process `TZ`; wrong day for half your users | `formatDate(instant)` |
+| `Intl.DateTimeFormat(locale)` with no `timeZone` | same, and lint-failing | pass `timeZone` or use a helper |
+| `timestamp` column | the offset is lost on write; two writers disagree | `timestamptz` |
+| Storing `+02:00` as "the zone" | breaks at the next DST transition | store the IANA zone |
+| `Date.now()` in a test assertion | flake | `clock.advance('3d')` and assert on the frozen instant |
+| `setHours(0, 0, 0, 0)` for "start of day" | midnight in the server's zone, not the user's | `startOfDay(instant, zone)` via `zonedParts` |
+| Adding `24 * 60 * 60 * 1000` for "tomorrow" | wrong on the 23h and 25h DST days | add one **calendar** day in the zone |
+| `date` column read as an instant | shifts by a day west of UTC | `date` columns are civil dates; never convert |
+| Formatting in a service or repo | untestable, unlocalizable | format in the route/component only |
+| `date.setMonth(date.getMonth() + 1)` | Jan 31 + 1 rolls into March | `addMonthsInZone(at, 1, zone)` or `addPlainMonths(date, 1)` — [Month arithmetic](#month-arithmetic) |
+
+## Month arithmetic
+
+A month has no fixed length, so both helpers clamp the day to the target month's last day.
+
+| Function | Answers |
+|---|---|
+| `addPlainMonths(date, months)` | a `PlainDate`: `2026-01-31` + 1 → `2026-02-28`; `2024-01-31` + 1 → `2024-02-29`; negative counts go back. A non-integer count or a year outside `0000`–`9999` is `X_SCHEDULE_INVALID` |
+| `addMonthsInZone(at, months, timeZone)` | an `Instant` at the same **local** wall-clock time `months` calendar months away; `timeZone` is required. A wall time that does not exist moves forward by the gap; one that exists twice takes the earlier instant |
+| `plainDateRange(start, end, { exclusive?, stepDays? })` | a frozen array of `PlainDate`s, `[start, end]` by default; a span over `MAX_PLAIN_DATE_RANGE_DAYS` (3660) is `X_SCHEDULE_INVALID` |
+
+**Compute a recurring date from its anchor, never by chaining.** A clamp does not compose:
+`2026-01-31` + 1 + 1 is `2026-03-28`, `2026-01-31` + 2 is `2026-03-31`.
+
+```ts
+import { addMonthsInZone, fromIso } from '@ultimat3/time';
+
+const anchor = fromIso('2026-01-31T00:00:00-05:00');
+const renewal = (n: number) => addMonthsInZone(anchor, n, 'America/Bogota');
+// renewal(1): Feb 28 00:00, renewal(2): Mar 31 00:00 — never stuck on the 28th
+```
+
+Full rules: [`packages/time/README.md`](https://github.com/developerz-ai/ultimate/blob/main/packages/time/README.md#months-clamp).
+
+## Duration strings
+
+One notation for every "how long" field.
+
+| String | Means | Used by |
+|---|---|---|
+| `'30s'` | 30 seconds | cache TTL, retry backoff floor |
+| `'15m'` | 15 minutes | `step.sleep`, live-query reconnect window |
+| `'6h'` | 6 hours | ISR revalidate TTL |
+| `'3d'` | 3 days | `step.sleep`, semantic cache `ttl` |
+| `'7d'` | 7 days | LLM semantic cache default |
+
+Parsed once into milliseconds. A bare number is a type error — the unit must be in the string. See [Caching and invalidation](Caching-And-Invalidation) and [Jobs and workflows](Jobs-And-Workflows).
+
+## Scheduled tasks
+
+`tz` is required on every `task` and must be an explicit IANA zone. There is no default.
+
+```ts
+// task (cron)
+export const nightlyDigest = task({
+  cron: '0 3 * * *',
+  tz: 'UTC',
+  enqueue: () => [[sendDigest, {}]],
+});
+```
+
+| Concern | Rule |
+|---|---|
+| Missing `tz` | type error, then `X_CONFIG_INVALID` at boot if bypassed |
+| `'UTC'` | correct for machine work: digests, compaction, exports |
+| A civil zone | correct for user-facing schedules: "9am local" is `tz: 'Europe/Berlin'`, not `'0 7 * * *'` in UTC |
+| Leader election | one `scheduler` role, holding an expiring lease row in `x_scheduler_leader` — never an advisory lock. A task fires once per cluster, not once per pod |
+| Catch-up | a missed window is logged and skipped, never replayed silently |
+
+See [Scheduled tasks](Scheduled-Tasks).
+
+### DST-crossing recurrence
+
+| Case | Behavior |
+|---|---|
+| Spring forward, `0 2 * * *` in a zone that skips 02:00 | fires at the zone's next valid instant that day (03:00) — shifted, never skipped |
+| Fall back, `0 2 * * *` in a zone that repeats 02:00 | a fixed time fires **once**, on the first pass |
+| Fall back, `*/5 * * * *` or `0 * * * *` — the minute or hour field is `*` or `*/n` | an interval runs through **both** passes of the repeated hour: it is real elapsed time, and no hour of the night is dark |
+| `step.sleep('1d')` across a transition | sleeps 24h of elapsed time, not one calendar day. For calendar semantics, compute the next instant in the zone and `step.sleepUntil` |
+| Monthly on the 31st | months without a 31st are skipped, not clamped to the 30th. Use `'0 3 28 * *'` if you need every month |
+
+Cron is a civil-calendar schedule; sleeps are elapsed time. Mixing them is where "the job ran twice in November" comes from.
+
+## Tests
+
+| Control | Behavior |
+|---|---|
+| Frozen clock | time starts at a fixed instant, does not advance on its own |
+| `clock.advance('3d')` | moves it, and also drives `step.sleep` and cron dispatch |
+| Fixed zone | `UTC` unless a test declares otherwise |
+| Fixed locale | `en-US` unless a test declares otherwise |
+| tz-dependent bugs | fail deterministically — declare a zone in the test and the bug reproduces every run |
+| Wall clock | never asserted on. `Date.now()` in an expectation is a rejected PR |
+
+Declare a zone when the behavior is the point:
+
+```ts
+test('digest window respects the org zone', async ({ seed, clock }) => {
+  const { org } = await seed('org-in-berlin');
+  clock.advance('1d');
+  const window = digestWindow(org, 'Europe/Berlin');
+  expect(window.label).toBe('Jul 27, 2026');
+});
+```
+
+See [Testing](Testing).
+
+## Rules
+
+- `timestamptz` for instants, `date` for civil dates. Nothing else.
+- Never format without an explicit IANA `timeZone`.
+- Never read the process `TZ`. Roles run with `TZ=UTC` and it is not an input.
+- Zone comes from the actor; a worker gets it from the job payload.
+- Format at the edge only — routes and components, never services or repos.
+- Durations are strings with units.
+- `tz` on a `task` is explicit and required.
+- Advance the frozen clock; never sleep in a test.

package/wiki/Troubleshooting.md

@@ -0,0 +1,163 @@
+# Troubleshooting
+
+Symptom → cause → fix. Every `X_*` error already carries its own cause and exact fix command; this page is for symptoms that do not start with a code.
+
+## Triage
+
+Run these first, in this order. All support `--json`.
+
+| Command | Answers |
+|---|---|
+| `x doctor --json` | Bun version, env schema, DB/transport/storage reachability, port conflicts |
+| `x verify --json` | the gate — 21 steps, in this order: typecheck, lint, boundaries, filesize, package-shape, errors, unit, contract, live, job, exercise, e2e, eval, drift, contract-diff, budgets, seo, i18n, policy, manifest, roadmap |
+| `x errors explain <CODE> --json` | cause, fix command, docs URL for any `X_*` code |
+| `x docs "<symptom or question>" --json` | the framework's own docs and this wiki, offline from the install — the first lookup for a symptom with no code |
+| `x doctor --json` | environment, versions, drift, ports, PWA prerequisites — each with a fix |
+| `x dev` then `/_x` | the live panels: routes, timeline, jobs, db, mail, cache, policy, manifest |
+| `x status --json` · `x logs tail --json` | **planned**, not shipped — both throw `X_NOT_IMPLEMENTED` naming the two rows above ([CLI reference](CLI-Reference)) |
+
+The `--json` form is the same content as the terminal form. Paste the JSON into a bug report; do not paraphrase the terminal.
+
+## Boot and env
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| Process exits in ~40ms, `X_ENV_MISSING` | a key in the `defineEnv` schema is missing, or present but fails its schema (short secret, non-URL, bad enum) | the cause names every offending key and its constraint at once; `x doctor --json` lists them too |
+| `X_CONFIG_INVALID` at load | `app.config.ts` field invalid — `defaultLocale` not in `locales`, `db.pool < 1`, non-IANA `timeZone`, `realtime.transport` set without `realtime.urlEnv` | `x config show --json`, then edit the field named in `cause` |
+| `X_ROLE_INVALID` | `ROLE` is not one of `web sync worker scheduler migrate replicator all` | fix the env var on that service |
+| `X_BUN_VERSION` | below the Bun floor, `>= 1.4.4` (`engines.bun`) | `bun upgrade` |
+| Container starts then loops | `/readyz` failing, not `/healthz` — a registered check answers `failing`, and which one is in the body | `curl /readyz` **from the container itself** (`docker exec`, `kubectl exec`, a port-forward — the detail goes to `healthDetailPeers`, default `['loopback']`) and read `checks`, a map of name → `ok`/`failing`: `database` is registered always, `transport` whenever it can report a connection, so a disconnected NATS node fails on that one. Read `registered` with it: `0` means nothing was checked, so the 200 meant only that the socket was bound |
+| Config edits have no effect | you edited a per-environment file that does not exist — config is **one file** | put the difference in an env var ([Configuration](Configuration)) |
+
+## Database
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_DB_DRIFT` | schema differs from migrations (a column added by hand, or a generated migration never applied) | `x db gen "<message>"` then `x db migrate`. A column no entity declares is an empty diff: declare it, or restore the migration and `x db reset` |
+| `X_DB_DRIFT`, cause *the dump holds a column no migration's snapshot records* | an applied migration was edited, and `x db gen` wrote the edit into `packages/db/schema/` | `git checkout -- packages/db/migrations packages/db/schema`. Meant it: declare the column on its entity, then `x db gen "<change>"` |
+| `X_MIGRATION_CONFLICT`, *was applied with checksum* | an applied migration's file was edited | `git log -- packages/db/migrations/<id>.sql`, then `git checkout <rev> -- …` the revision whose checksum the ledger names, then `x db gen "<change>"`. `x db gen` first exits 0 and poisons the dump |
+| `X_APP_IMPORT_UNRESOLVED` | an app module imports a package nobody installed, or a path with no file behind it. One finding per specifier | `bun add <pkg>`, or edit the import in the file the finding names. `x doctor` reports it too |
+| `X_DB_DRIFT` on a brand-new app, cause *packages/db has a schema but no migration recorded it* | expected: `x new` writes no migration, and `x db gen` is the only writer of `packages/db/migrations`. Only an app that declares an entity reports it — zero declared against zero recorded is agreement | `x db gen "initial"` then `x db migrate` — `bun run setup` does both |
+| `X_DB_DRIFT` after deleting the last entity, migrations still committed | correct, and not the case above: the migrations record tables the source no longer declares | `x db gen "drop <table>"` → it refuses with `X_MIGRATION_IRREVERSIBLE` → re-run as `x db gen "drop <table>" --allow-destructive`, or put the entity back |
+| `X_MIGRATION_SNAPSHOT_MISSING`, and `x db gen` refuses before writing anything | the newest migration on disk carries no `.snapshot.json` — hand-written, or the sidecar was deleted | `git checkout -- packages/db/migrations/<id>.snapshot.json`; if it was never written, `rm packages/db/migrations/<id>.* && x db gen "<name>"` — the delete **first**, or the generate refuses again |
+| `ROLE=migrate` sits there doing nothing | another version's `ROLE=migrate` holds the session-pinned migration lock. This one **waits up to 60s**, polling once per 500ms, then exits non-zero with `X_MIGRATE_CONCURRENT` (`As of 2026-08`; on 1.2.0 it waits forever) | let the first exit 0 — the second then applies. If nothing else is deploying, the holder is a wedged backend: `psql "$DATABASE_URL" -c "select pid, application_name, state from pg_stat_activity join pg_locks using (pid) where locktype = 'advisory'"`, then `pg_terminate_backend(pid)` it |
+| `ROLE=migrate` exits non-zero, deploy blocked | migration failure — correct, the roll is supposed to stop | read the SQL error, fix the migration, re-run. Do not start `web` on the old schema |
+| `X_TIMEOUT` on one query | past `db.statementTimeout` (default `'10s'`) | add the index the plan wants, or narrow the query. Raising the timeout hides it |
+| Connection exhaustion under load | `db.pool` × replicas × roles exceeds Postgres `max_connections` | lower `db.pool`, or a pooler in front |
+| Every test suddenly slow | the template DB is being rebuilt per worker | `x test --json` reports template build time; check migrations that are not idempotent |
+
+## Import boundaries
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_BOUNDARY_VIOLATION` on a file that imports nothing suspicious | a **transitive** chain — the error prints it, e.g. `site/pricing → shared/ui/button → app/billing/service` | break the chain at the lowest hop; move the shared piece into `shared/` |
+| `X_BOUNDARY_SITE_TO_APP` | a `site/` route reached into `app/` | duplicate the small thing, or move it to `shared/`. `site/` must stay 0kb JS |
+| `X_BOUNDARY_SHARED_LEAF` | `shared/` imported `site/` or `app/` | `shared/` is a leaf, always |
+| `X_BOUNDARY_ROUTE_TO_DB` | a `page`/`layout`/`route` file imports the database: `packages/db`, `@<scope>/db`, `drizzle-orm`, **or a slice's `repo.ts`** (`../posts/repo` counts; `report` and `repository` do not). A page reading the repo skips the query that carries the read's policy, bound and cache tag. `As of 2026-10` (`isDbSpecifier`, `packages/cli/src/app-boundaries.ts`) | run the printed `x g query <name> --feature <slice>`, then call that query from the route — route → `query`/`action` → `repo.ts` |
+| `X_BOUNDARY_SERVICE_TO_HTTP` | a service imported request/headers | a service that knows about requests cannot be reused by a job |
+| `X_BOUNDARY_PACKAGE_TO_APP` | a file under `packages/` imports `apps/` at runtime, relative or by workspace name. `import type` is fine, and so is `packages/db/src/` reading an entity module | move the shared code into the package, or import in the opposite direction |
+| `X_FEATURE_REQUIRED` | `x g query` or `x g backfill` named no slice with an `entity.ts`: neither creates a table | run the printed line: `--feature <closest slice>`, or `x g resource <name>` for a new table |
+| Tier violation in a framework package | imported sideways or upward | `bun run boundaries`; consult the tier table in [Project layout](Project-Layout#framework-package-tiers) |
+
+Boundaries run on pre-push and inside `x verify`. They are build errors, never lint warnings.
+
+## Budgets, SEO, i18n
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_BUDGET_EXCEEDED` on `js` | one import pulled a library into a surface | the error names the **import chain**; `x routes --json` for per-route budgets |
+| A `site/` route reports non-zero JS | a component with client state leaked into the static surface | move it to `app/`, or set `hydrate: 'never'` |
+| LCP or CLS regression | an unsized image or a late-loading font | `x routes --json` for the route's budget; images go through the pipeline, never a raw `<img>` |
+| `X_SEO_META_MISSING` | an indexable route's `meta` has no `title`, or no `description`. One code for both — `cause` names the field and the file | add it to the route's `meta`. Deleting a description is a build error, by design. Over-length is a different code, `X_SEO_META_TOO_LONG` (title > 60, description > 160). |
+| `X_ROUTE_META_MISSING` | a route has no `meta` at all | every route sets `render`, `offline`, `hydrate`, `meta` |
+| `X_CATALOG_MISSING_KEYS` | a key exists in one locale's catalog and not another | the cause lists key + locale; add the translation. There is no silent English fallback |
+| `⟦some.key⟧` rendered in the UI | the key is missing everywhere | add it to the default catalog; `x verify` fails until you do |
+| Lighthouse gate fails | below `seo.lighthouse.seo` (100) or `accessibility` (95) | fix the audit, don't lower the threshold |
+
+## Jobs
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_IDEMPOTENCY_REQUIRED` (compile error) | `idempotencyKey` omitted — it is required by the type | add an `idempotencyKey` derived from `input` only, e.g. `onboard:${orgId}` |
+| `X_STEP_DUPLICATE` | two `step.run` calls with the same name in one `run` | rename one. Step names must be unique and stable |
+| A step re-ran after a deploy | the step was **renamed** — renaming invalidates its stored result | treat a rename as a new step; keep old names |
+| `X_IDEMPOTENCY_CONFLICT` | same key, different input, inside the dedupe window | the key is not deterministic from `input`. Remove the timestamp or random part |
+| Job stuck "in-flight" after a worker was killed | the lease has not expired yet | wait `jobs.visibilityTimeout` (default `'30s'`); the row is re-claimed and resumes at the **next** step, never mid-step. `x jobs show <id> --json` |
+| Job in dead-letter | `retry.attempts` exhausted | `x jobs show <id> --json` for the step trace, then `x jobs retry <id>` — it replays from the failed step |
+| Nothing is processing | no `worker` for that queue name | check `WORKER_QUEUES` against `jobs.queues` |
+| A job ran but the row it needs doesn't exist | enqueued outside the transaction | enqueue via `<job>.enqueue` inside the action's `handle`; `X_OUTBOX_NO_TX` catches the rest |
+| Cron never fires | no `scheduler`, or another node holds the lease | `scheduler` is fixed at 1 active and leadership is a row: `select * from x_scheduler_leader` names the holder and its `expires_at`. A standby has no `/readyz` to check — it serves the metrics port only — so read the row, or `x jobs list --json` |
+
+## Realtime
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_TRANSPORT_UNAVAILABLE` | fanout bus unreachable — `NATS_URL` names a server nothing answers on | `x doctor`; set `NATS_URL`, or unset it for in-process fanout |
+| `X_TRANSPORT_PROTOCOL` | nats-server older than 2.11, started without `-js`, or something other than NATS on the port | presence needs batch direct get and per-message TTL — run `nats:2.11-alpine` or newer with `-js` |
+| Live query rejected at build | unbounded — missing `orderBy` or `limit` | add both. An unbounded result has no bounded change buffer and no bounded reconnect snapshot |
+| Live query rejected for non-determinism | `now()` / `random()` in the `sql` | move it to `input`; the same `(input, row)` must always yield the same membership answer |
+| `X_SUBSCRIPTION_LIMIT` | a socket, actor, tenant or node hit a cap — the error names which scope refused, and channel topics have their own | **live queries:** raise `maxPerSocket` / `maxPerTenant` / `maxEntries` on the `LiveQueryRegistry` (per socket, default 128), or unsubscribe unused ones. **Per actor** (default 1,000 per node, anonymous sockets per address): `realtime.maxSubscriptionsPerActor` in `app.config.ts`. **Channel topics:** raise `maxTopicsPerSocket` (default 64) / `maxTopicsPerNode` (default 10,000) on the `ChannelHub`. Only the per-actor cap is an `app.config.ts` field → [Configuration](Configuration) |
+| `X_CURSOR_STALE` | resume cursor outside the change buffer window — or a reconnect that landed on a `sync` node which never served that `qid`, since the ring is per node | raise `capacity` on `new RingChangeBuffer({ capacity })` (default 1024), or pass `'snapshot'` to `resumeFrom()` so the fallback re-snapshots. Not an `app.config.ts` field → [Configuration](Configuration) |
+| `X_TOPIC_FORBIDDEN` | tier-1 topic guard denied the actor | fix the topic scope; the cause names actor + topic, never the topic's data |
+| Reconnect storm after a deploy | drain was truncated by SIGKILL, so no `reconnect` frame was sent | the kill timer must outlast the whole drain. Compose: `stop_grace_period` ≥ the app's `drain.readinessGraceMs` + `drain.deadlineMs` + teardown, raised by hand. Helm: `terminationGracePeriodSeconds` is derived per role from the chart's values — `preStopSleepSeconds` (web/sync, Kubernetes 1.30+) + readiness grace (web/sync) + drain budget + `teardownMarginSeconds` — with the grace and budget passed by `x deploy --method helm`; never set it by hand ([Deployment](Deployment)) |
+| Reconnects still spiky with a clean drain | not a config field — `realtime.drain.*` does not exist. A draining node sends a `reconnect` frame carrying a per-client `afterMs`; if clients still stampede, they are reconnecting on their own `backoffDelay()` because no frame reached them | confirm the drain path actually ran — a `SIGKILL` sends no frame, which is the case the 50k benchmark measures → [Configuration](Configuration) |
+| A channel message never arrived, and nothing errored | backpressure. `channel_frames_dropped_total` moved and `channel.frames_dropped` names the topic — a channel has no cursor and no re-snapshot, so a dropped frame is not replayed | put anything that must arrive on a live query, which repairs. Alert on any non-zero rate → [Observability](Observability) |
+| `useChannel()` reads `joining` on a page that was `live`, or `x dev` logs `ultimate bus is away` | the `sync` node's bus (NATS) is away: it answers the page's beat with `X_TRANSPORT_UNAVAILABLE`, which is its condition, never a decision about the subscriber `As of 2026-10` | nothing in the app: the beat re-asks every 10 s and the membership is `live` again within one beat of the bus returning, with no reload. Find the outage on the node — `sync.bus_unavailable`, `nats transport error` — and fix NATS. `failed` is the state that means a decision (policy, undeclared channel, cap) → [Realtime](Realtime#a-bus-outage-seen-from-an-open-page) |
+| A client is connected but receives nothing, and never reconnects | half-open socket: the TCP connection died without a `close`, so no handler fired | the page socket's heartbeat ends it: 10 s, two silent windows, then close `4000`. In 21.0.0 there is no option to turn it off → [Realtime](Realtime) |
+| Topics silent after the first reconnect, live queries fine | a client that does not re-announce its topics. **`hello` carries nothing to resume with** `As of 2026-08` — no cursors, no membership; a `subscribe` frame carries both, one per topic and one per live query with that subscription's cursor | the page socket re-sends all of them on open. A hand-written client must send one `subscribe` per topic **and** one per live query carrying its resume cursor, or the topics are silent, presence sweeps the membership, and every live query re-snapshots → [Realtime](Realtime) |
+| Every client re-snapshots for the length of a deploy | the `qid` hash changed with the framework version, so a resuming cursor names a ring entry the new node does not have | expected and correct: the miss falls back to one bounded snapshot per subscription. It ends when the rollout does |
+| A row appeared that the user shouldn't see | not possible via the framework path — policy is re-checked per delivered row. Suspect an unscoped predicate | `X_TENANCY_UNSCOPED`; tenant scope comes from `ctx`, never from `input` |
+| `sync` never becomes ready | replication feed lag over threshold, or NATS not subscribed | check the `replicator` first — it owns the slot |
+
+## PWA, build skew, deploys
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| White screen after a deploy, 404 on a lazy chunk | version skew — a tab holding build `A`'s document asked for a hashed file only build `A` served. With `assets.retain` unset (the default) a web pod serves only what it built | set `assets: { retain: { disk: '<a disk every web pod shares>', builds: <deploys a tab may sit open across> } }` in `app.config.ts`; with it set, raise `builds` → [PWA and offline](PWA-And-Offline) |
+| `X_BUILD_SKEW` on an action | the input schema changed incompatibly since the client's build | the `fix:` line names the action; version the action instead of changing it in place |
+| Users stuck on an old version | your app never renders the `AppUpdateAvailable` affordance | subscribe to the signal and show "Update available — reload". The framework never force-navigates |
+| Need everyone off a build now | security patch | **You cannot — the framework force-reloads nothing, `As of 2026-08`.** The forced-reload half of `version-skew` was deleted in 9.0.0: nothing performed the reload it described, and no runtime could have called it (`http` is tier 2 and `sync` is tier 3, both below `pwa` at tier 4). What ships is *notification*: read `useConnection().updateAvailable` from `@ultimat3/realtime` and render your own affordance. A deploy has no channel to a running client either way |
+| `X_SW_HAND_EDITED` | **reserved, never raised** — `x build` writes `sw.js`, `As of 2026-08`, but it carries no checksum, so an edit survives a build ([Error codes → Not thrown yet](Error-Codes#not-thrown-yet)) | change the **route**, which is the input the generator is designed around; the emitted file is overwritten on the next build either way |
+| `X_SW_UNCACHEABLE` | **reserved, never raised** — `offline: 'precache'` on an `ssr` route is accepted | caching a per-request render is a correctness bug. Use `runtime` or `network-only`, and review the pairing yourself |
+| Precache budget warning | `x build --target static --json` reports it under `serviceWorkerWarnings`, `As of 2026-08` — a manifest past 5 MB stalls the install on a bad connection. `PrecacheManifest.warnings` was computed and read by nothing until then ([#390](https://github.com/developerz-ai/ultimate/issues/390)) | drop routes from `offline: 'precache'`, or cut the island chunks the manifest carries — `x build --target static --json` lists both |
+| Offline shows the browser dinosaur | `pwa.enabled` is false, or the page was never controlled — a worker that has activated is not yet the page's controller, and a first load is not controlled until it claims | check `navigator.serviceWorker.controller !== null` in the console; then → [PWA and offline](PWA-And-Offline) |
+| Preview build poisoned a cache | impossible via the framework — build ID scopes the SW cache namespace and scope | confirm the branch build ID in `x manifest --json` |
+
+## Tests
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_TEST_NETWORK_SEALED` | a test reached the network unmocked | the error names the URL. Mock it — sealed network is the design, not a bug |
+| A test passes alone, fails in the suite | shared state that should not exist — each worker gets its own cloned DB | check for a module-level singleton or a fixture written outside `seed()` |
+| Time-dependent failure | asserted on wall clock | use the frozen clock and `clock.advance('3d')` — it also drives `step.sleep` and cron |
+| A test flakes | **fix it or delete it the same day** | there is no `retry: 3`. A test that passes twice and fails the third trains people to ignore red |
+| Snapshot/UUID churn between runs | not using `seed(name)` | seeds are deterministic: same input → identical rows, identical UUIDs |
+| Job test doesn't drain | queue not advanced | `await runJobs.drain()`; workers run deterministically in tests |
+| Want the failing worker's database after the run | there is **no** `--keep-db` — `x test` declares `--workers`, `--isolate`, `--filter`, `--allow-empty`, `--sample`, `--seed` and the `--affected` trio, and nothing else, and the harness calls `drop()` at teardown ([`packages/testing/src/harness.ts:108`](https://github.com/developerz-ai/ultimate/blob/main/packages/testing/src/harness.ts)) | what survives is the migrated template, `ultimate_test_template` — the clones are `ultimate_test_template_w<N>`. Inspect the template: `psql "$TEST_DATABASE_URL" -c "\l"`, then connect to it. To hold a clone open, assert inside the test rather than after it |
+
+## MCP and AI
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `X_MCP_TOOL_UNKNOWN`, or a tool absent from `tools/list` | the tool is hidden from this caller, or the name is stale — role-hidden and absent answer identically, on purpose | visibility is fail-closed: check the caller against the tool's `visibleTo` (a role allowlist or a predicate over the caller). Otherwise `tools/list` for the catalog this caller may use, and `x manifest` if the committed manifest is stale |
+| `X_MCP_TOOL_UNDECLARED`, and MCP refuses to boot | an action or query written out in `defineAppMcp`'s `actions:`/`queries:` never declared `mcp: { expose: true }` — a boot-time configuration error, not a hidden tool | add `mcp: { expose: true, description: '<what it does>' }` beside the primitive's policy, or drop it from the list and let `include: 'exposed'` project what opted in |
+| `X_MCP_SCOPE_UNKNOWN`, and MCP refuses to boot | `defineAppMcp`'s `scopes:` map names a tool by a name the server does not project — a typo, or the primitive was renamed or dropped from `actions:`/`queries:`/`tools:` | spell the name as one of the tools the server actually projects, or drop it from that `scopes` entry |
+| `X_MCP_SCOPE_DENIED` | the connection's token does not carry the tool's scope — scope is a property of the token, not of the actor's permissions | reconnect with a token whose scopes include the one `cause` names — the app's `resolveToken(token)` returns them — or drop that scope from `defineAppMcp({ scopes })`. Scopes are fixed for the life of a connection, so a grant takes effect on the next one. `x token grant` is **planned** and exits `X_NOT_IMPLEMENTED` |
+| `X_FORBIDDEN` from a `tools/call` | the tool was invoked and its policy refused this input — the same denial the HTTP route returns for the same call | grant the human the permission — an agent can never exceed the human it acts for |
+| Dev MCP server not reachable | `x dev` not running, or you pointed at prod | default socket is `mcp.devSocket` (`ws://localhost:9229`). The dev server is **never** bound in `ROLE=web` |
+| `X_MCP_QUERY_REJECTED` | `db.query` was not given exactly one read-only statement | send a single **read-only** `SELECT`/`WITH`/`EXPLAIN`/`SHOW`/`TABLE`/`VALUES` — a data-modifying CTE is not a read. MCP has no arbitrary-write path: change data by calling an action exposed with `mcp: { expose: true }`, change schema with `db.migrate` on a branch database |
+| `X_MCP_NOT_BRANCH_DB` | `db.migrate` was aimed at a database that is not a branch | `x db branch create <name>   # then retry db.migrate` |
+| `X_LLM_OUTPUT_INVALID` | structured output failed its schema twice | tighten the prompt or loosen the schema; the retry already happened once |
+| Prompt change had no effect | semantic cache hit | bump the prompt version — editing a prompt requires it. (`x ai cache --json` is **planned**; `x test eval --json` is the shipped command) |
+| `x verify` fails on a prompt | no evals file | an unevaluated prompt is untested code. Add `<prompt>.evals.ts` |
+
+## Still stuck
+
+```
+x verify --json > verify.json
+x doctor --json > doctor.json
+```
+
+Open an issue with both files attached, plus your Bun version, your `@ultimat3/*` pin, and the `x errors explain <CODE> --json` output for any code you hit. Security issues go through [`SECURITY.md`](https://github.com/developerz-ai/ultimate/blob/main/SECURITY.md), never a public issue.
+
+Code index: [Error codes](Error-Codes). Config fields: [Configuration](Configuration). Upgrade failures: [Upgrading](Upgrading).