@ultimat3/cli 29.4.0 → 30.0.0

package/wiki/Tutorial-05-Deploy-Free.md

@@ -0,0 +1,231 @@
+# Tutorial 5 — deploy on a free tier
+
+The framework ships **no** platform primitives — no buildpack, no `app.json`, no `fly.toml`, no adapter. It does not need to: every PaaS builds a Dockerfile, and every one of them wants the same three behaviours the scaffolded image already has.
+
+`As of 2026-11-03`. Every command on this page down to [When one instance is not enough](#when-one-instance-is-not-enough) was re-run on a fresh `x new myapp --no-example` app carrying the tutorial 2–4 slices, with Docker 29 on Linux, its framework packed from this tree into `vendor/` the way `docker/deploy-proof/run.sh` packs it. `ROLE=migrate` ran against a real Postgres. **Not run: anything on a platform** — no host was used, so the platform table, the free-tier behaviour and the `x deploy` without `--dry-run` are still read, not measured. `scripts/tutorial-run.ts` does not run this page.
+
+Series: [1 — first app](Tutorial-01-First-App) · [2 — first feature](Tutorial-02-First-Feature) · [3 — auth and admin](Tutorial-03-Auth-And-Admin) · [4 — jobs and realtime](Tutorial-04-Jobs-And-Realtime) · **5** · [6 — growing up](Tutorial-06-Growing-Up)
+
+## The whole integration
+
+| The platform does | The image does |
+|---|---|
+| injects `PORT`, routes traffic to it | `apps/web/server.ts` binds exactly `$PORT`, refusing a non-port value with `X_PORT_INVALID` rather than defaulting past it |
+| requires a bind on `0.0.0.0` | binds every interface — loopback is unreachable through a port mapping |
+| polls a health path | `/readyz` for *may I have traffic*, `/healthz` for *am I alive* |
+| sends `SIGTERM`, then `SIGKILL` | drains in three phases: stop accepting, finish in-flight, close |
+| wants one artifact per release | one image, `ROLE` selects the process |
+
+Set `DATABASE_URL` and the three signing secrets below. Point the platform at `docker/Dockerfile`. That is the integration.
+
+## Build
+
+```bash
+bunx x build --target docker --tag myapp:dev
+```
+
+```text
+✓ built docker
+```
+
+```text
+myapp:dev  300MB
+```
+
+`300MB` is a 29.4.0 scaffold's image, measured 2026-11-03 (the 1.1.0 image was 194MB).
+
+`x build` runs the static gate steps first — `typecheck`, `lint`, `boundaries`, `filesize`, `package-shape`, `errors` — and exits non-zero without building if any fail. A build that would fail `x verify` produces no artifact.
+
+The scaffolded image is `oven/bun:1.4-alpine`, pinned by digest, in three stages (`manifests`, `deps` with `bun install --frozen-lockfile --production`, `runtime`). It runs as `USER 1000:1000`, and its `ENTRYPOINT` is `["bun", "apps/web/server.ts"]`. There is no compile stage and no second gate inside it: re-running `tsc` and `biome` there would need the devDependencies the `--production` install deliberately omits.
+
+> `docs/ops/README.md` also describes a distroless, single-binary image. That is **this repo's own** CLI image (`docker/Dockerfile` in the framework repository), not what `x new` writes — your app's `docker/Dockerfile` is the authority for your image.
+
+## Run it, locally, exactly as the platform will
+
+```bash
+docker run -d --name myapp-web -e ROLE=web -e PORT=8080 -p 8085:8080 \
+  -e SESSION_SECRET=$(openssl rand -hex 32) -e ULTIMATE_CURSOR_SECRET=$(openssl rand -hex 32) \
+  -e STORAGE_SIGNING_SECRET=$(openssl rand -hex 32) myapp:dev
+```
+
+```text
+{"ts":"2026-11-03T16:23:42.394Z","level":"info","msg":"ultimate metrics listening","url":"http://0.0.0.0:9090/metrics"}
+{"ts":"2026-11-03T16:23:46.918Z","level":"info","msg":"app mcp mounted","method":"POST","path":"/mcp"}
+{"ts":"2026-11-03T16:23:46.961Z","level":"info","msg":"ultimate web listening on http://0.0.0.0:8080"}
+{"ts":"2026-11-03T16:23:46.962Z","level":"info","msg":"ultimate started","role":"web","url":"http://0.0.0.0:8080","buildId":"edca46484ed2b43f","bus":"in-process"}
+```
+
+Without the secrets it exits at boot, one refusal at a time: `X_CURSOR_SECRET_DEV` names `ULTIMATE_CURSOR_SECRET`, then `X_ENV_MISSING` names `STORAGE_SIGNING_SECRET` (an image with no `S3_ENDPOINT` stores uploads on its own disk). `ULTIMATE_ENV=production bunx x env check` lists them all before you build.
+
+```bash
+curl http://127.0.0.1:8085/readyz
+```
+
+```json
+{"state":"ready","ready":true,"role":"web"}
+```
+
+The verdict, and nothing else: a published port reaches the container from the bridge gateway,
+not from loopback, and only a peer `healthDetailPeers` lists is told the build id, the in-flight
+count and the check names. From inside the container it is loopback, and they are printed:
+
+```bash
+docker exec myapp-web bun --eval "fetch('http://127.0.0.1:8080/readyz').then(r=>r.text()).then(console.log)"
+```
+
+```json
+{"state":"ready","ready":true,"uptimeMs":5236,"inflight":0,"buildId":"edca46484ed2b43f","checks":{"database":"ok"},"registered":1,"role":"web"}
+```
+
+The image's own `HEALTHCHECK` (`/readyz` every 10s, a 30s start period) reported `healthy` 5s after the start. Prove that locally before you debug it on a platform.
+
+### The env vars that matter
+
+| Key | Unset means | Set it to |
+|---|---|---|
+| `ROLE` | `web` | `web` for the service, `migrate` for the release phase |
+| `PORT` | 3000 | whatever the platform injects — leave it to the platform |
+| `DATABASE_URL` | **embedded PGlite** — never in production | the managed Postgres connection string |
+| `SESSION_SECRET`, `ULTIMATE_CURSOR_SECRET`, `STORAGE_SIGNING_SECRET` | the shipped development keys, which a production boot refuses | `openssl rand -hex 32` each, in the platform's secret store |
+
+A real `DATABASE_URL` env var wins over anything baked into the image; verified by pointing a container at an unreachable host and getting `X_DB_UNAVAILABLE` rather than a silent PGlite fallback.
+
+**No `.env` file reaches the image**, `As of 2026-08-19`. `docker/Dockerfile.dockerignore` excludes `**/.env` and `**/.env.*`, keeping only `!**/.env.example`. It did not: the old patterns were `.env` and `.env.*.local`, which match neither `.env.development` **nor `.env.production`** — the file `docker/docker-compose.prod.yml`'s `env_file:` tells you to create. Both shipped inside the layer, proven by a real `docker build`, and this tutorial called that harmless.
+
+On 3.0.0 and below, add these three lines to `docker/Dockerfile.dockerignore` before your first build — and rebuild, because an image already built still carries them:
+
+```text
+**/.env
+**/.env.*
+!**/.env.example
+```
+
+## Release-phase migrations — the one way
+
+Run **the same image** with `ROLE=migrate` before the new release serves traffic.
+
+```bash
+docker run --rm -e ROLE=migrate -e DATABASE_URL=postgres://… myapp:dev
+```
+
+```json
+{"ts":"2026-11-03T16:24:07.231Z","level":"info","msg":"ultimate migrate applied","applied":2,"available":2,"appVersion":"dev"}
+```
+
+It applies every pending migration in `packages/db/migrations` under a Postgres advisory lock, records each in the `x_migrations` ledger with its checksum, and exits 0. Concurrent migrators serialise — the second waits up to 60s for the lock, then exits non-zero with `X_MIGRATE_CONCURRENT` rather than hanging the release; a checksum that no longer matches an applied migration stops the release rather than corrupting it.
+
+Every setting each host asks for, and which were executed: [Deploy recipes](Deploy-Recipes).
+
+| Platform | Where the command goes |
+|---|---|
+| Heroku | `release:` in `Procfile`, with `ROLE=migrate` on the release dyno |
+| Render | a pre-deploy command running `ROLE=migrate bun apps/web/server.ts` |
+| Fly.io | `[deploy] release_command` with `ROLE=migrate` |
+| Railway | a pre-deploy command running the same |
+| Kubernetes | an `initContainer` or a `Job` on the same image |
+| Compose | the `migrate` service; every other role waits on `service_completed_successfully` |
+
+**No release phase on your free tier?** Several platforms gate pre-deploy commands behind a paid plan. Run the one-off yourself against the same image and the same `DATABASE_URL` before promoting the release — that is what every row above ultimately is.
+
+`x db migrate` is deliberately absent from that table: it is the developer's command, and it needs the toolchain ([tutorial 2](Tutorial-02-First-Feature#migrations)). The release phase runs the shipped image and nothing else.
+
+## What a free tier costs you
+
+Free instances **sleep**. That is not a performance note; it changes which roles can exist.
+
+| Role | On a single sleeping free instance |
+|---|---|
+| `web` | works. First request after a sleep pays a cold start, then serves normally |
+| `worker` | **does not run reliably.** No inbound traffic means nothing wakes the instance, so queued jobs sit until the next visitor |
+| `scheduler` | **does not run reliably.** A cron whose process is asleep at 03:00 does not fire then; `catchUp: 'skip'` is the generated default, so whenever something next wakes the instance a week of missed occurrences collapses into one late fire, never a replay of each |
+| `sync` | works while awake; every sleep disconnects every subscriber |
+| `replicator` | needs a Postgres with `wal_level=logical`, which free managed tiers generally do not offer |
+
+Two honest options on rung 0:
+
+| Option | Trade |
+|---|---|
+| ship `ROLE=web` only, do the work inline in the action | no durability, no retries — acceptable for work that is cheap and idempotent |
+| ship `ROLE=web` and keep the jobs, accepting late execution | correct results, unpredictable latency. The queue is Postgres, so nothing is lost — only delayed |
+
+Do not paper over it with an external pinger: keeping a free instance awake around the clock is what the free tier is not. The moment jobs must run on time, that is the signal to climb — [tutorial 6](Tutorial-06-Growing-Up).
+
+## Drain
+
+`docker stop -t 20 myapp-web`:
+
+```text
+{"ts":"2026-11-03T16:23:52.649Z","level":"info","msg":"draining","signal":"SIGTERM","deadlineMs":25000,"readinessGraceMs":5000,"inflight":0}
+{"ts":"2026-11-03T16:23:57.656Z","level":"info","msg":"stopped","signal":"SIGTERM"}
+```
+
+`/readyz` flips to 503 **before** the socket closes, and stays 503 for `readinessGraceMs` (5s) with the listener still open, so a load balancer polling it stops sending traffic before the socket goes — that is what makes a rolling restart drain instead of drop. An idle process stops at the end of the grace: 5s here.
+
+Full role-by-role handoff table: [Deployment](Deployment).
+
+## The other two build targets
+
+```bash
+bunx x build --target static --out .x/static
+```
+
+```text
+  skipped  /todos      app/ surface — server-rendered, not prerendered; app/ allows stream | ssr, so no route on it can ever reach the artifact
+✓ built static
+```
+
+One HTML file per `render: 'static'` route — with this scaffold, `.x/static/index.html` beside `404.html`, the `offline` page, the manifest, icons, islands and styles. Every other render mode needs a running app and is listed as `skipped` (one line per route), never emitted. Serve it from a CDN or an object store with no process behind it; set `SITE_ORIGIN` so `canonical` and `og:url` are built against the real host.
+
+`apps/web/prerender.ts` also writes **`sitemap.xml` and `robots.txt`** into the same directory, because a static export has no process to answer them at request time. `robots.txt` **fails closed**: anything that is not `ULTIMATE_ENV=production` emits `Disallow: /` and advertises no sitemap, so a preview build cannot outrank the real site.
+
+```bash
+bunx x build --target binary --out .x/app
+```
+
+```text
+✓ built binary
+```
+
+It boots `As of 2026-10-06`: the required `windows` CI job starts a scaffold's binary and gets `200` from `/healthz`, and a Linux e2e test runs a compiled probe tree. The framework is **not** compiled in: `x build` keeps `@ultimat3/*` external, so `@ultimat3/core` loads from the app's `node_modules` and `frameworkVersion()` reads its version from core's own `package.json` there (`resolveVersion`, `packages/core/src/version.ts`). `x build` still passes `--define ULTIMATE_FRAMEWORK_VERSION="<version>"`, but nothing in this binary reads it — the define answers only for a build that bundles the framework, such as the image's self-contained `x`. Such a build compiled without the define, and with no `package.json` beside core, exits `X_INVARIANT` (fix `x build --target binary`) rather than reporting a version it does not have.
+
+Use `--target docker` anyway. The image is the self-contained artifact; the binary is a launcher for an app tree, and must be started from the app root with the source beside it — and no scaffolded app has yet been served from a bare VM this way.
+
+## When one instance is not enough
+
+```bash
+bunx x deploy --image myapp:dev --dry-run
+```
+
+```text
+  migrate    IMAGE=myapp:dev docker compose --env-file …/.env.production -f …/docker/docker-compose.prod.yml run --rm migrate
+  web        IMAGE=myapp:dev docker compose --env-file …/.env.production -f …/docker/docker-compose.prod.yml up -d web
+  sync       IMAGE=myapp:dev docker compose --env-file …/.env.production -f …/docker/docker-compose.prod.yml up -d sync
+  worker     IMAGE=myapp:dev docker compose --env-file …/.env.production -f …/docker/docker-compose.prod.yml up -d --no-deps --no-recreate --wait --wait-timeout 300 --scale worker=2 worker
+  scheduler  IMAGE=myapp:dev docker compose --env-file …/.env.production -f …/docker/docker-compose.prod.yml up -d --no-deps --no-recreate --wait --wait-timeout 300 --scale scheduler=2 scheduler
+  backfill   IMAGE=myapp:dev docker compose --env-file …/.env.production -f …/docker/docker-compose.prod.yml run --rm backfill
+✓ containers only: 1 image, roles migrate,web,sync,worker,scheduler,backfill
+```
+
+Migrate to completion, then the serving roles, then the one-shot `backfill`. `--dry-run` prints the plan and runs nothing. `…` is the app root, printed absolute.
+
+**`--env-file` is what fills the `${VAR:?…}` guards.** Compose interpolates `SYNC_URL`, `APP_URL` and `POSTGRES_PASSWORD` from the shell and `--env-file` only, never from a service's `env_file:`. So all three go in `.env.production` — `APP_URL=http://<host>:3000` is the origin your pages are served on, and the only one besides its own that `sync` admits a socket from, and every step reads that file. A variable set in the shell still wins over it.
+
+**At most one replica each for `web` and `sync`, and the file says so** `As of 2026-11-03`. Both publish a host port, and one host port has exactly one binder: `web` is `replicas: 1`, and `sync` is `replicas: 0` — off — until the app turns realtime on with a replicator and NATS (the comment above it in `docker/docker-compose.prod.yml` is the recipe), then `1`. Leave them there — this tutorial's rung is one box. `worker` publishes no port and scales freely, so `deploy: { replicas: 4 }` on it is the knob you actually have here; the plan rolls `worker` and `scheduler` by starting a second copy beside the first (`--scale <role>=2 --wait`).
+
+Scaling the two serving roles is the next rung, and it is not an edit to this file: put a reverse proxy of your own on the compose network and drop their `ports:` lines, or move to the chart's per-role HPA → [6 · Growing up](Tutorial-06-Growing-Up).
+
+## What is not there yet
+
+| Expected | Reality `As of 2026-08` |
+|---|---|
+| `/metrics` on the **app** port | `X_ROUTE_NOT_FOUND`, and that is deliberate. Every role — `web` included — serves `/metrics` on `METRICS_PORT`, default **9090**, because the Helm ingress routes `/` with no path exclusion and metrics on 3000 would be public. Scrape `http://<host>:9090/metrics`, not `:3000/metrics` |
+| `x logs tail` | planned — `X_NOT_IMPLEMENTED`, with `x dev` → the `/_x` timeline panel as its fix |
+| `x status` | planned — `x doctor --json` is the shipped answer |
+| OTLP export, on by default | the exporter **ships** — `otlpSpanExporter()` / `otlpMetricExporter()`, OTLP/HTTP JSON — but the default is still the no-op, so nothing leaves the process until you register one: `configureTelemetry({ exporter: otlpSpanExporter() })` with `OTEL_EXPORTER_OTLP_ENDPOINT` set to a collector's HTTP receiver (`:4318` — `:4317` is gRPC and is refused) |
+| a Helm chart in your app | **`x new` writes one** `As of 2026-11-03` — `docker/helm`, 10 files — so `x deploy --method helm` works here. On 3.0.0 and below it writes none: copy [`docker/helm`](https://github.com/developerz-ai/ultimate/tree/main/docker/helm) from the framework repo, or stay on `--method compose` |
+
+## Next
+
+[Tutorial 6 — growing up](Tutorial-06-Growing-Up): managed Postgres, a shared cache, one service per role, then Compose and Kubernetes — with the app code unchanged at every rung.
+
+Related: [Deployment](Deployment) · [Configuration](Configuration) · [CLI reference](CLI-Reference) · [Known gaps](Known-Gaps) · [Troubleshooting](Troubleshooting)

package/wiki/Tutorial-06-Growing-Up.md

@@ -0,0 +1,158 @@
+# Tutorial 6 — growing up
+
+**The app code is identical at every rung.** Climbing is an env var, a driver swap and someone else's infrastructure — never a rewrite. A small idea reaches a big output by changing where the rows and the messages live, not by changing what an `entity`, `policy`, `action`, `mutator`, `query`, `job`, `route` or `task` says.
+
+`As of 2026-08`. The full ladder, with costs, seams and the places the invariant breaks, is [`docs/idea/17-scale-ladder.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/idea/17-scale-ladder.md); running it for real is [`docs/ops/`](https://github.com/developerz-ai/ultimate/tree/main/docs/ops). This page is the decision procedure, not a second copy of either.
+
+Series: [1 — first app](Tutorial-01-First-App) · [2 — first feature](Tutorial-02-First-Feature) · [3 — auth and admin](Tutorial-03-Auth-And-Admin) · [4 — jobs and realtime](Tutorial-04-Jobs-And-Realtime) · [5 — deploy free](Tutorial-05-Deploy-Free) · **6**
+
+## Climb on a signal, never on a feeling
+
+| Rung | You run | What forces the next rung | App code change |
+|---|---|---|---|
+| **0** — PaaS, one process | one web service, managed Postgres | a job that must not run in a request, or a cron that must fire on time | **none** |
+| **1** — PaaS, roles split | one service per `ROLE`, managed Postgres, managed Redis-protocol cache | more than one `sync` replica, platform socket limits, or per-service pricing above a box | **none, plus config** |
+| **2** — one box, Compose | one host: Postgres, cache, NATS, every role | one host is a single point of failure you can no longer accept, or cannot hold the load | **none, plus config** |
+| **3** — Kubernetes | per-role Deployments and HPAs, JetStream fanout, logical replication | Postgres write throughput, a region you must serve from, a zone you must survive | **none, plus config** |
+| **4** — distributed SQL | multi-AZ, YugabyteDB, metrics and traces wired end to end | nothing here is forced by scale — geography and availability targets force it | **none for the datastore swap** |
+
+Skipping a rung to look serious buys a second product to maintain. A Kubernetes cluster running one app is that.
+
+## What is allowed to change
+
+| May change | Never changes |
+|---|---|
+| `DATABASE_URL`, `REDIS_URL`, `NATS_URL`, `S3_ENDPOINT` | any file under `apps/` or `packages/` |
+| `ROLE` — which process this container is | the eight primitives, their shapes, their authz |
+| driver constructors at boot (`setJobDriver`, `setDbClient`, `configureTelemetry`) | call sites: `db()`, `ctx.jobs.enqueue()`, `query()`, `live()` |
+| replica counts, HPA targets | the manifest, the OpenAPI, the typed client, the MCP tools |
+| `app.config.ts` declarations | migrations already applied |
+
+## Rung 0 → 1: split the roles
+
+Same platform, same image. One service per `ROLE`. This is the rung a free tier's sleeping instance forces whether or not traffic did — see [tutorial 5](Tutorial-05-Deploy-Free#what-a-free-tier-costs-you).
+
+| Role | Why it leaves the web service |
+|---|---|
+| `worker` | a job must survive the request that queued it and must not compete with it for CPU |
+| `scheduler` | fixed at one, and **must be always-on** — a plan that sleeps it drops the cron |
+| `migrate` | run-once, before anything serves the new schema |
+
+`scheduler` leadership is an expiring lease row in `x_scheduler_leader`, never an advisory lock — that grant belongs to the **session**, not to the process: it outlives every transaction on the connection and is released only by an explicit unlock, the pool's reset on release, or the connection dying — so a node can neither renew it nor prove it still holds one — so a second instance is an idle standby rather than a duplicate. It has no `/readyz` at all: `ROLE=scheduler` binds the metrics port only, and nothing routes request traffic to it.
+
+### The shared cache tier
+
+Add it when there is more than one web replica **and** a measured cross-replica miss. Not before.
+
+```ts
+// app.config.ts
+cache: { tiers: ['request-memo', 'lru', 'redis'] },
+```
+
+Read order is request-memo → LRU → redis → origin, and a tier is never consulted for a request whose `policy` has not already passed. Adding one changes where a value is found, never how it is asked for.
+
+> **Any Redis-protocol server, `As of 2026-08`.** The tier-3 invalidation script used to `DEL` keys it never declared in `KEYS`, which Dragonfly and Redis Cluster reject. Fixed: the script touches only the buckets it was handed and returns the value keys for the client to drop one at a time, the buckets carry an `{entity}` hash tag, and `invalidateTags` issues one call per tag — so every key of a call co-slots. Redis, Valkey and Dragonfly all work. Clustered deployments are **unmeasured, not unsupported**: no test in the repo runs against a real cluster node.
+
+Detail: [Caching and invalidation](Caching-And-Invalidation).
+
+## Rung 1 → 2: one box, Compose
+
+`docker/docker-compose.prod.yml` **is** the production topology — one service per role, one image, differing only by `ROLE`. `migrate` runs to completion and every serving role waits on `service_completed_successfully`.
+
+```bash
+bunx x deploy --image ghcr.io/you/myapp:1.2.3 --method compose
+```
+
+Start Postgres with `wal_level=logical`, `max_replication_slots=8` and `max_wal_senders=8` so the `replicator` role can preflight rather than fail on its first change.
+
+> **The compose ceiling is one replica of each serving role, declared** `As of 2026-08`. The framework's file and the scaffolded one both publish a host port on `web` and `sync` and both set `replicas: 1` — one host port has exactly one binder. `worker` publishes no port and scales freely.
+>
+> Two ways past it: delete their `ports:` lines and put a reverse proxy of your own on the compose network, which the service names resolve across every replica; or take the next rung, where the chart's per-role HPA does it. The framework ships neither proxy — one in `docker-compose.prod.yml` is a dependency every app inherits and a second answer to "how does traffic reach a role" beside the chart's Ingress.
+
+## Rung 2 → 3: Kubernetes
+
+**`x new` writes the chart**, `As of 2026-08-19` — `docker/helm`, 8 files — a `Deployment` for each of the four roles enabled by default (`web`, `sync`, `worker`, `scheduler`), `replicator` behind `enabled: false`, and `migrate` as a `Job` — so `x deploy --method helm` runs `helm upgrade --install` against it with nothing to copy in. On 3.0.0 and below `x new` writes none and the command exits `X_NOT_IMPLEMENTED`: copy [`docker/helm`](https://github.com/developerz-ai/ultimate/tree/main/docker/helm) from the framework repo.
+
+The alternative is the flat manifests in [`docs/ops/01-kubernetes.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/ops/01-kubernetes.md). Carry one, never both — a repo with a chart *and* a manifest tree has two sources of truth for the same pod. Deleting `docker/helm` is how you choose the manifests: the command then fails with helm's own error, not the framework's.
+
+Three things turn on at this rung, all by env, none by a code path:
+
+| Turn on | Set | Effect |
+|---|---|---|
+| cross-node fanout | `NATS_URL` | `selectTransport` builds `NatsTransport` instead of `InProcessTransport` |
+| presence across nodes | `NATS_KV_BUCKET` (default `x_presence`) | presence becomes a JetStream KV bucket; its age limit and the presence TTL are one number |
+| live queries off a real WAL | `REPLICATION_URL` · `REPLICATION_SLOT` · `REPLICATION_PUBLICATION` | `selectChangeFeed` builds `postgresChangeFeed()`, under an advisory lock so exactly one replicator exists |
+
+**Do not enable `replicator` until live queries are in use.** The chart ships it `enabled: false` for that reason.
+
+Per-role HPAs target `rps`, `connections` and `queue_demand` — CPU is a lagging proxy for all three serving roles. Every role serves `/metrics` on `METRICS_PORT` (default 9090) and `http`/`realtime`/`jobs` emit the series. The chart still cannot scrape them: no role declares a metrics container port and no scrape target ships, so an HPA sits at `<unknown>` until you add both plus a custom metrics adapter.
+
+## Rung 3 → 4: three separable moves
+
+| Move | The only good reasons | The bad reason |
+|---|---|---|
+| dedicated servers over cloud instances | steady predictable load; egress cost; hardware sized to the working set | "cloud is expensive", with no bill that says so |
+| custom Kubernetes over a managed chart | node pools with real taints, affinity to node-local storage, an ingress you control | wanting the chart to look impressive |
+| distributed SQL over one Postgres | multi-region **write** locality; surviving a zone loss with no manual failover | write throughput one Postgres has not actually failed to deliver |
+
+Observability is the part that is not optional here, and the framework's half is done `As of 2026-08` — but **the default exporter is still the no-op**, so it emits nothing until you say so. Two lines at boot:
+
+```ts
+configureTelemetry({ exporter: otlpSpanExporter() });   // OTEL_EXPORTER_OTLP_ENDPOINT
+configureMetrics({ exporter: otlpMetricExporter() });   // same endpoint, metrics signal
+```
+
+Point `OTEL_EXPORTER_OTLP_ENDPOINT` at your collector's **HTTP** receiver, `:4318` — `:4317` is gRPC and is refused with `X_OTLP_PROTOCOL_UNSUPPORTED`, permanently, because HTTP/2 plus protobuf is a dependency and a second wire format. Metrics also stay scrapeable at `METRICS_PORT` if you prefer an agent. [`docs/ops/03-observability.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/ops/03-observability.md) says what to scrape, what to alert on, and what is still missing — no logs signal, and a rejected export is dropped rather than retried.
+
+## The seams, so nothing surprises you
+
+| Concern | Interface | Decided by |
+|---|---|---|
+| rows | `@ultimat3/db` · `DbClient` | `DATABASE_URL` — unset is PGlite, and PGlite is `x dev` only |
+| job queue | `@ultimat3/jobs` · `JobDriver` | `setJobDriver(postgresJobDriver({ executor }))` — there is no `jobs.driver` config line, and 5.0.0 deleted the one that never worked. Redis is an interface-complete stub that throws `X_NOT_IMPLEMENTED`; there is no NATS job driver since 25.0.0 |
+| realtime fanout | `@ultimat3/realtime` · `Transport` | `NATS_URL` |
+| change feed | `@ultimat3/realtime` · `ChangeFeed` | `REPLICATION_*` |
+| cache | `@ultimat3/cache` · `CacheTier` | `cache.tiers` + `REDIS_URL` |
+| object storage | `@ultimat3/storage` · `Driver` | `S3_ENDPOINT` |
+| mail | `@ultimat3/mail` · `Driver` | mail env |
+| tracing | `@ultimat3/core` · `SpanExporter` | `configureTelemetry()` — default no-op |
+
+Every one keys on env or a boot-time constructor. One image resolves `x dev`, a Compose host and a cluster identically.
+
+## Rehearse the climb before you need it
+
+| Step | Command |
+|---|---|
+| point the app at a real Postgres | `DATABASE_URL=… ROLE=migrate bun apps/web/server.ts`, then `ROLE=web` |
+| prove the plan without running it | `bunx x deploy --image … --dry-run --json` |
+| prove the image before the platform does | `docker run -e ROLE=web -e PORT=8080 -p 8085:8080 …` and poll `/readyz` |
+| prove the gate still holds | `bunx x verify` — the same 21 steps, at every rung |
+
+## Where the invariant genuinely breaks
+
+Named rather than left to be discovered:
+
+| Break | Rung |
+|---|---|
+| tier-3 cache invalidation unmeasured against a real clustered cache node | 1 |
+| the two roles that publish a host port run at one replica under the shipped compose file | 2 |
+| no Helm chart in a scaffolded app | 3 |
+| no OTLP **logs** signal, and a rejected export is dropped with a `warn` rather than retried — traces and metrics do export | 3–4 |
+| a custom-metrics adapter, which the chart's HPAs need and the framework never ships | 3–4 |
+| `x logs` planned — `X_NOT_IMPLEMENTED`, with `x dev` → the `/_x` timeline panel as its fix | any |
+| the Redis **job** driver throws `X_NOT_IMPLEMENTED` (no NATS job driver since 25.0.0) | any |
+| realtime tier 3 (local-first, `persist: true`), the plugin API, multi-region replication | **not shipped**, `As of 2026-08` |
+
+Each sits behind an interface that ships today and fails loudly, rather than pretending to work. The full list, with a workaround per row: [Known gaps](Known-Gaps).
+
+## Read next
+
+| Doc | Answers |
+|---|---|
+| [`docs/idea/17-scale-ladder.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/idea/17-scale-ladder.md) | the full rung table, costs, the seam table, YugabyteDB honestly |
+| [`docs/ops/`](https://github.com/developerz-ai/ultimate/tree/main/docs/ops) | Kubernetes manifests, secrets, observability, datastore sizing, disaster recovery, runbooks |
+| [Deployment](Deployment) | roles, health endpoints, the drain contract, build targets |
+| [Configuration](Configuration) | every `app.config.ts` field and every env var |
+| [Upgrading](Upgrading) | moving every `@ultimat3/*` in lockstep |
+
+Related: [Caching and invalidation](Caching-And-Invalidation) · [Realtime](Realtime) · [Jobs and workflows](Jobs-And-Workflows) · [Troubleshooting](Troubleshooting)

package/wiki/UI-Components.md

@@ -0,0 +1,156 @@
+# UI components
+
+`@ultimat3/ui` — SolidJS components on the semantic tokens in [Theming](Theming). SCSS modules, logical properties, no Tailwind, no CSS-in-JS.
+
+`As of 2026-09`. Stable API — semver from here ([Upgrading](Upgrading)).
+
+## The catalog is generated — read it, don't ask
+
+**[`packages/ui/CATALOG.md`](https://github.com/developerz-ai/ultimate/blob/main/packages/ui/CATALOG.md)** documents every component with every prop, every type, and the token scales, parsed straight out of `packages/ui/src/components/*.tsx`. It ships inside the npm tarball, so it is on disk at `node_modules/@ultimat3/ui/CATALOG.md` in any app.
+
+**How many there are is a question for the file, never for this page.** It carried `52` for two majors:
+
+```bash
+head -7 node_modules/@ultimat3/ui/CATALOG.md   # the count and every name, on one line
+```
+
+| Concern | Answer |
+|---|---|
+| Regenerate | `bun run --filter @ultimat3/ui catalog` |
+| Drift | `catalog.test.ts` compares the committed file byte-for-byte against a fresh build; a prop change with no regen is a red `x verify` |
+| Hand edits | refused by the same test — the file carries a `GENERATED` banner |
+
+This page does **not** restate the catalog. It covers the four page composites, because they are the ones an agent picks wrong.
+
+**Which component to reach for is not the same question as how it must behave.** The four-state
+decision, skeleton-versus-spinner, response tiers, optimistic writes, forms, motion, focus, toasts
+and images are [Interface rules](Interface-Rules) — one page, every rule marked either with the
+guard, type or gate step that refuses it, or as judgement nothing checks.
+
+**Form errors are translated on `FormIssue.rule`.** `formBinding`'s `messageFor` receives the same
+`rule` and `params` (`string.min`, `{ min: 3 }`) from the local parse and from the server's
+`X_INPUT_INVALID`, whose problem document carries the structured `issues` list —
+``messageFor: (issue) => t(`form.${issue.rule}`, issue.params)``. See [I18n](I18n).
+
+## The four page composites
+
+A screen that hand-rolls a header grid is the bug these exist to prevent. All four hold no state — the route owns it.
+
+### `AppShell`
+
+The frame: skip link, landmarks, optional sidebar. Off-canvas navigation is `Drawer`, not a prop here.
+
+| Prop | Type | Default |
+|---|---|---|
+| `children` | `JSX.Element` | required |
+| `header` | `JSX.Element` | omitted → no `<header>` |
+| `sidebar` | `JSX.Element` | omitted → no `<nav>`, and the grid collapses to one column |
+| `footer` | `JSX.Element` | omitted → no `<footer>` |
+| `bottomNav` | `JSX.Element` — a `<BottomNav>` | omitted → none; given, shown below `md` only, last, sticky to the bottom edge and in flow, so it reserves its own height |
+| `sidebarLabel` | `string` | `t('ui.navigation')` |
+| `skipLabel` | `string` | `t('ui.skip')` |
+| `sidebarWidth` | `string` | `'16rem'` |
+| `stickyHeader` | `boolean` | **sticky unless explicitly `false`** — `undefined` is sticky |
+| `class` | `string` | — |
+
+Landmark order in the DOM: `banner` → `navigation` → `main` → `contentinfo` → the bottom bar's `navigation`. `<main>` is unconditional.
+
+The sidebar's content is `NavList` and the bottom bar's is `BottomNav` (five links at most); both mark the current page by one rule, `currentOf(items, href)` — the longest covering href wins, so `/runs/7` marks `/runs`. The bar's padding clears the home indicator through `t.safe-area-padding(block-end)` (`viewport-fit=cover` on the page).
+
+The skip link is real, not decorative: `<main>` carries `tabindex={-1}` so focus actually moves there, and the link's `href` and the target's `id` are derived from one `shellIds()` call, so they cannot drift apart. It sits off-screen at `translate: 0 -400%` and slides in on `:focus-visible`.
+
+### `PageHeader`
+
+| Prop | Type | Default |
+|---|---|---|
+| `title` | `string` | required |
+| `description` | `string` | omitted |
+| `actions` | `JSX.Element` | omitted |
+| `breadcrumbs` | `readonly BreadcrumbItem[]` | omitted → no `Breadcrumb` |
+| `level` | `1 \| 2 \| 3 \| 4 \| 5 \| 6` | **`1`** |
+| `media` | `JSX.Element` | omitted |
+| `class` | `string` | — |
+
+### `Section`
+
+| Prop | Type | Default |
+|---|---|---|
+| `children` | `JSX.Element` | required |
+| `title` | `string` | omitted → an unlabelled grouping, no `aria-labelledby` |
+| `description` | `string` | omitted |
+| `actions` | `JSX.Element` | omitted |
+| `level` | `1` … `6` | **`2`** |
+| `as` | `'section' \| 'article' \| 'aside'` | **`'section'`** |
+| `class` | `string` | — |
+
+A `title` wires `aria-labelledby` to the heading's generated id automatically. The head block disappears entirely when both `title` and `actions` are absent, so an untitled `Section` costs no markup.
+
+A `level` outside 1–6 throws `X_UI_INVALID_VALUE` rather than emitting an `<h7>`.
+
+### `Toolbar`
+
+| Prop | Type | Default |
+|---|---|---|
+| `children` | `JSX.Element` | required |
+| `actions` | `JSX.Element` | omitted → no trailing group |
+| `label` | `string` | **required** |
+| `surface` | `boolean` | falsey |
+| `class` | `string` | — |
+
+`label` is required because `role="toolbar"` with no accessible name is an unnamed group. Arrow keys move through the focusable children via a roving tabindex, horizontal, direction-aware, **not** looping.
+
+## A control that navigates is a `Link`
+
+`Button` has no `href`. A link that looks like a button is `<Link appearance="button">`; a pager on a server-rendered list is `<Pagination hrefFor>`. Neither needs an island.
+
+```tsx
+<Link appearance="button" variant="secondary" size="sm" href="/posts/new">{t('posts.new')}</Link>
+
+// `page` is a framework `Page` (a query's `.page()`): `nextCursor` is `string | null`, handed as is.
+const list = listUrl(props.url, { sorts: ['title', 'createdAt'] });
+<Pagination nextCursor={page.nextCursor} hrefFor={list.hrefFor} firstHref={list.firstHref} />
+```
+
+`listUrl(url, { sorts, ns?, limit? })` reads a list's state out of its URL — `sort`, `cursor`, `limit`, `q`, never throwing: an unknown sort or a hostile limit is the default — and writes it back as hrefs: `hrefFor` (the cursor, everything else kept), `sortHrefFor` and `href(change)` (the change, the cursor dropped), `firstHref` (`undefined` on the first page). Param names are `sort`, `dir`, `after`, `limit`, `q`; `ns: 'runs'` prefixes them all (`runs.after`) for two lists on a page, and then any change but a cursor drops every cursor on the page, since a cursor is bound to its filter and sort (`X_CURSOR_INVALID`). The same object feeds a `DataTable`'s `hrefFor`, `sortHrefFor` and `firstHref`.
+
+| Component | Mode | Props | Renders |
+|---|---|---|---|
+| `Link` | text (default) | `underline`, `tone: 'accent' \| 'inherit'` | `<a>` with the link classes |
+| `Link` | `appearance="button"` | `variant`, `tone` (any `Tone`), `size`, `fullWidth`, `iconStart`, `iconEnd` | `<a>` with `Button`'s classes — one set of button styles, two elements |
+| `Pagination` | callbacks | `onCursor`, or `page` + `totalPages` + `onPage`; `onFirst` | `<button>`s; needs an island |
+| `Pagination` | links | `hrefFor(cursor, direction)`; `firstHref` | `<a rel="prev">` / `<a rel="next">`; Next with no cursor is a disabled button |
+| `DataTable` | callbacks / links | `onCursor` + `onFirst`, or `hrefFor(cursor, direction)` + `firstHref` | its pager, in that mode — the props are handed straight to `Pagination` |
+
+A cursor that is `null` or absent is one state: no page that way. Previous is drawn only when a `prevCursor` is given or the pager is numbered — a keyset `Page` has no way back, so its pager draws Next and, given `firstHref` / `onFirst`, First. Next holds the trailing edge either way.
+
+One mode per use, and the type holds it: `hrefFor` beside `onCursor`, `onPage`, `page` or `totalPages` does not compile, and neither does `underline` on a button-link or `variant` on a text link. Link mode pages by cursor only. `Link` also takes `rel?: 'next' | 'prev'`, which joins the `noopener noreferrer` an external link always carries.
+
+## Rules an agent must not break
+
+- **SCSS modules only.** `Foo.tsx` + `Foo.module.scss`, always paired. The only inline `style` allowed is a CSS custom property.
+- **Logical properties.** `margin-inline`, `inset-inline-start`, `text-align: start`. A `left`/`right` in a stylesheet is a bug.
+- **No hardcoded strings.** Label props, or the built-in keys in `src/i18n-keys.ts` → [I18n](I18n).
+- **No raw colours.** `t.role('<role>')` or `var(--color-*)` → [Theming](Theming).
+- **No prop spreading.** Components declare explicit props; there is no `{...rest}`.
+- **`solid-js` is a type-only import** inside the package — reactive access goes through one registered adapter, so the design system does not pin a renderer version.
+- Formatting logic lives in a pure `*-view.ts` beside the component (`money-view.ts`, `date-time-view.ts`), testable with no renderer.
+- **Choosing from a list is three controls, never one with modes.** `Select` (short, closed), `SearchableSelect` (long, closed, value ≠ label — no-script it is the native `<select>`), `Combobox` (free text). There is no `<Select searchable>`: every `Select` island would carry the listbox.
+- **A destructive question is `ConfirmDialog`, never `confirm(`.** Cancel is focused, the backdrop never dismisses, `action` posts with scripting off, `onConfirm` stays open on a refusal; the confirm label is a verb with no default. `guards/native-confirm.ts` refuses `confirm(` in app code.
+- **Shortcuts are one table per island, merged per page.** `useHotkeys([{ keys, label, run }])`; `ShortcutSheet` lists the merged table on `?` and `useCommandPalette` adds `mod+k` to it — so the legend cannot drift from the bindings. The framework reserves no key.
+- **The bottom sheet is `Drawer side="block-end" grabber`.** There is no `Sheet`. Link tabs are `Tabs` with `hrefFor` — a `<nav>`, never `role="tab"`.
+- **Charts: options on the shipped charts, not new charts.** Signed bars and sparklines, `null` gaps, `at` for true x, `references`, `stacked` areas, `tone` per series, `Gauge` `thresholds`/`target`, `Meter` `target`. The two new names are `HeatStrip` and `LiveIndicator`, which no chart's shape covers. A stat grid is `Grid minColumn`.
+- **Which list, when.** Records with columns: `DataTable`. Thousands of one-shape rows: `VirtualList` (no script, every row renders; with script, under 100 mounted). Things inside things: `Tree` — `hrefFor` is `<details>` and links at 0 B, `onSelect` is the island's `role="tree"`. What happened: `Timeline` (+ `InfiniteScroll` + `RelativeTime` is the activity feed; there is no feed component). Facts about one thing: `DescriptionList`, never a two-column `Table`.
+- **Dates are the native input first.** `DatePicker`/`DateRangePicker` are `<input type="date">` with a `Calendar` popover only on a fine pointer with script. A date is a `PlainDate`; a window (`TimeRangePicker`, `timeRangeUrl`) is instants and takes a REQUIRED `timeZone`. Presets are the app's. `Slider` is `<input type="range">`; a segmented control is `Radio appearance="segmented"`; there is no number stepper.
+- **Reordering is `SortableList` and `Board`, a split is `SplitPane`** — one Pointer Events gesture, never HTML5 drag. A drop calls `onMove` once and is a request: the row holds until it settles, a rejection restores it and says so. `canDrop(item, group)` is the one seam for a WIP limit or a status machine. With `action`, each row is a real `<form>` (Up/Down, or a target `select` per card). The handle alone has `touch-action: none`, so a phone still scrolls from a row's body. `SplitPane` is one column below `md`; `onResize` + `value` is how a split persists — storage is the app's.
+- **What a model wrote is `Markdown`; code is `CodeBlock`; never `innerHTML`.** Raw HTML prints as text, links only http/https/mailto/relative, a remote image prints its alt as a link unless `resolveImage` vouches for it, `baseLevel` defaults to 2. Fences go through `CodeBlock`, highlighted only by the `languages` the caller passes (`allLanguages` on a server page; one named language in an island). Structured payloads are `JsonView`, changes `DiffView` (two texts or a unified patch), structured logs `LogViewer`, terminal output `TerminalView` (SGR colours as roles). A rich-text editor is `Textarea` plus a `Markdown` preview.
+
+## Error codes
+
+| Code | Means |
+|---|---|
+| `X_TOKEN_UNKNOWN` | a design token role that does not exist; `cause` lists the ones that do |
+| `X_THEME_INVALID` | a theme that is not `light` or `dark` |
+| `X_UI_RUNTIME_MISSING` | a host capability the component needs is absent (`IntersectionObserver`, `localStorage`) |
+| `X_UI_INVALID_VALUE` | a formatting component got an unrenderable value — `NaN` money, an invalid date, a heading level off the scale |
+
+Full rows: [Error codes](Error-Codes).