@ultimat3/cli 29.4.0 → 30.0.0

package/wiki/Tutorial-03-Auth-And-Admin.md

@@ -0,0 +1,379 @@
+# Tutorial 3 — auth and admin
+
+Authentication produces one thing: an `Actor`. Nothing downstream authorizes on a session row, a user row or an api key — HTTP, the typed client, jobs, live queries and MCP all read the same actor and hand it to the same `policy`. One authz system, never two.
+
+`As of 2026-11-03`. Every command and every output on this page was re-run in the app tutorials 1 and 2 leave — the example `post` slice and the `todo` slice — built from this tree's packages (`scripts/scaffold-smoke-overrides.ts`). The session flow ran against a real Postgres; the admin routes were requested from `x dev`. The reference app excerpt under [Detail pages](#detail-pages-actions-batches) is quoted from `examples/dummy`, not run.
+
+Series: [1 — first app](Tutorial-01-First-App) · [2 — first feature](Tutorial-02-First-Feature) · **3** · [4 — jobs and realtime](Tutorial-04-Jobs-And-Realtime) · [5 — deploy free](Tutorial-05-Deploy-Free) · [6 — growing up](Tutorial-06-Growing-Up)
+
+## The three pieces
+
+| Piece | Package | Owns |
+|---|---|---|
+| identity | `@ultimat3/auth` | passwords, sessions, OAuth, MFA, api keys → an `Actor` |
+| authorization | `@ultimat3/policy` | permissions, roles, `can()` predicates — the only evaluator |
+| operations UI | `@ultimat3/admin` | the admin app's screens, the `/_x` dev panels |
+
+`@ultimat3/auth` and `@ultimat3/admin` are scaffold dependencies: `x new` lists both in the root `package.json` at the version it was released with, beside every other `@ultimat3/*` package. Nothing to add. Every `@ultimat3/*` package releases in lockstep, and a mixed-version install is a combination nobody tested.
+
+## Roles first — they cost one file
+
+`x new` writes the role map, `apps/web/shared/roles.ts`: `member` and `admin`, with `admin` inheriting `member`. Each `x g resource` adds its grants there — after tutorial 2, beside the example `post` slice's:
+
+```ts
+// apps/web/shared/roles.ts — the roles, after `x g resource todo`
+import { defineRoles } from '@ultimat3/policy';
+
+export const roles = defineRoles({
+  member: {
+    description: 'Signed in. Reads the app surface.',
+    grants: ['dashboard:read', 'post:read', 'todo:read'],
+  },
+  admin: {
+    description: 'Runs the app: the /admin surface, plus everything a member may do.',
+    grants: ['admin:read', 'admin:write', 'admin:destroy', 'job:read', 'audit:read',
+      'post:write', 'posts:read', 'posts:write', 'posts:delete',
+      'todo:write', 'todos:read', 'todos:write', 'todos:delete'],
+    inherits: ['member'],
+  },
+});
+```
+
+Roles are sugar. Everything expands to a flat permission set before any policy runs, so a rule never reasons about the hierarchy, and a cycle is caught once at expansion. `defineRoles()` merges into one map and refuses a role two modules define differently (`X_ROLE_REDEFINED`) — keep every role in this one file. `todo:*` is the policy's permission; `todos:*` is the table's, which the [admin](#the-admin-surface) asks beside its own `admin:*`.
+
+```bash
+bunx x policy list
+```
+
+```text
+  permission         roles         actions                 queries
+  admin:destroy      admin         -                       -
+  admin:impersonate  -             -                       -
+  admin:read         admin         -                       -
+  admin:write        admin         -                       -
+  audit:read         admin         -                       -
+  dashboard:read     admin,member  -                       -
+  job:delete         -             -                       -
+  job:manage         -             -                       -
+  job:read           admin         -                       -
+  job:write          -             -                       -
+  post:read          admin,member  -                       postList
+  post:write         admin         archivePost,createPost  -
+  posts:delete       admin         -                       -
+  posts:read         admin         -                       -
+  posts:write        admin         -                       -
+  todo:read          admin,member  -                       todoList
+  todo:write         admin         archiveTodo,createTodo  -
+  todos:delete       admin         -                       -
+  todos:read         admin         -                       -
+  todos:write        admin         -                       -
+  16 permission(s) no action or query enforces:
+    admin:destroy
+    …
+✓ 20 permission(s), 2 role(s), 4 enforced by a declaration
+```
+
+The `actions` and `queries` columns are the reason to run this: a permission enforced by **no action or query** is listed under the table. Here those are the admin's and the jobs dashboard's, which their own screens enforce — for a permission of your own, the list is a grant that does nothing.
+
+## The denial matrix, before you ship
+
+```bash
+bunx x policy explain todo:write
+```
+
+```text
+  action archiveTodo — policy todo:write
+  actor      verdict  deciding    reason
+  anonymous  deny     todo:write  no actor for todo:write
+  admin      deny     todo:write  todo:write predicate returned false
+  member     deny     todo:write  actor lacks todo:write
+    evaluated with no request input and no row — a rule reading either decides again on the real request
+  action createTodo — policy todo:write
+  …
+✓ todo:write — allowed for 0 of 6 actor evaluation(s)
+```
+
+Read the `reason` column, not the verdict:
+
+| Reason | Means |
+|---|---|
+| `actor lacks todo:write` | the grant is missing — a role problem, fix it in `defineRoles` |
+| `predicate returned false` | the grant is held, the predicate said no. Here that is tenancy, with no `input.orgId` to compare against outside a request |
+| `no actor for …` | anonymous. Every surface agrees on this one |
+
+`<subject>` resolves against a permission, an action name, a query name, then an action's HTTP path — so the `fix:` line an `X_FORBIDDEN` prints is runnable whichever of the four the throwing surface had. Verdicts come from `policyMatrix()` over the app's real `Policy` objects, never a second evaluation.
+
+## Sessions
+
+`x new` writes the app's one `Auth`, built on first use because `defineAuth` reads the database and the lockout limiter the boot installs:
+
+```ts
+// apps/web/app/auth/auth.ts, as `x new` writes it
+import { type Auth, defineAuth, postgresAuthAdapter } from '@ultimat3/auth';
+
+let built: Auth | undefined;
+
+export const appAuth = (): Auth => {
+  built ??= defineAuth({ adapter: postgresAuthAdapter() });
+  return built;
+};
+```
+
+The defaults: a 30-day absolute and a 7-day idle session, cookie `__Host-x_session`, rotated on a privilege change. `authenticator.ts` beside it resolves that cookie on every request. What the scaffold does **not** write is a sign-in page — see [Known gaps](Known-Gaps).
+
+Register, log in, resolve — the real flow, against Postgres, with `const auth = appAuth()` and `org` the seeded demo org (`DEMO_ORG_ID` from `apps/web/shared/demo-org.ts`):
+
+```ts
+await register(auth, { email, password, orgId: org, roles: ['member'] });
+const { actor, token, cookie } = await login(auth, { email, password, ip });
+```
+
+```json
+{"actor":{"kind":"user","id":"01a1a22f-…","orgId":"9c7ca12f-7aa0-57c4-a1cf-79b3afc8d1dd",
+  "roles":["member"],"scopes":[],"permissions":[],"facts":{}},
+ "cookie":"__Host-x_session=<token>; Max-Age=2592000; Path=/; HttpOnly; Secure; SameSite=Lax"}
+```
+
+`authenticate(auth, token)` answers the same actor. `permissions: []` is correct — direct grants are for service tokens and break-glass accounts. The role is what carries the set, and `actorHas()` expands it against `roles.ts`:
+
+```json
+{"todo:read":true,"todo:write":false,"admin:read":false}
+```
+
+A `member` reads todos; the scaffold grants `todo:write` to `admin` only, so writing is an edit to `roles.ts`. Anonymous (`anonymousActor()`) holds nothing:
+
+```json
+{"anon":"anonymous","hasRead":false}
+```
+
+### Signing out clears the browser too
+
+`logout(auth, token)` ends the session row. The **response** does the rest, through
+`signOutHeaders()`, in 21.0.0:
+
+```ts
+import { logout, signOutHeaders } from '@ultimat3/auth';
+import { appAuth } from './auth';
+
+const auth = appAuth();
+declare const token: string;
+declare const headers: Headers; // the action's ctx.headers
+
+await logout(auth, token);
+for (const [name, value] of signOutHeaders({ session: auth.sessions.policy })) {
+  headers.append(name, value);
+}
+```
+
+What it appends, verbatim:
+
+```json
+[["set-cookie","__Host-x_session=; Max-Age=0; Path=/; HttpOnly; Secure; SameSite=Lax"],
+ ["clear-site-data","\"cache\", \"storage\""]]
+```
+
+After `logout`, `authenticate(auth, token)` refuses the token: `X_UNAUTHENTICATED`.
+
+| Header | Effect |
+|---|---|
+| `set-cookie` (with `session`) | the framework session cookie, expired. An app with its own cookie omits `session` and sets its own |
+| `Clear-Site-Data: "cache", "storage"` (`SIGN_OUT_CLEAR_SITE_DATA`) | the browser drops what the previous member left on the origin: the page store's IndexedDB, local storage, the service worker and its cached pages. **Never `"cookies"`**, which would also clear the signed-out cookie this response sets |
+
+**Trade-off, deliberate:** a PWA's offline cache does not outlive the person it was cached for.
+The next load reinstalls the worker and re-precaches. Browsers act on `Clear-Site-Data` only in a
+secure context (HTTPS, or `localhost`). A sign-out with no response at all falls back on the page
+boot, which wipes every stored scope but the current principal's
+([Realtime](Realtime#tier-3-shipped-in-2100)). The reference app's `endSession` action uses it.
+
+### Failures say one thing
+
+Wrong password, unknown address and disabled account are indistinguishable in message **and** duration — the first two below were each run, and print the same `code` and `cause`:
+
+```json
+{"code":"X_UNAUTHENTICATED",
+ "cause":"the email and password combination did not match an account — re-enter them before issuing the reset below, which mails a single-use token",
+ "fix":"issueVerification(runtime, { purpose: 'password-reset', identifier: email, locale })"}
+```
+
+### The cookie
+
+`__Host-x_session`, from `sessionCookie(token, policy)`.
+
+| Attribute | Closes |
+|---|---|
+| `HttpOnly` | XSS reading `document.cookie` |
+| `Secure` | a network attacker lifting it off plaintext |
+| `SameSite=Lax` | CSRF — not attached to cross-site POSTs |
+| `__Host-` + `Path=/` + no `Domain` | a sibling subdomain overwriting it (session fixation) |
+
+Absolute and idle expiry are evaluated **independently**: activity never moves the ceiling. Session ids are opaque random tokens and only `sha256(secret)` reaches the database.
+
+### The four actor kinds
+
+| Kind | From | Carries |
+|---|---|---|
+| `user` | a session | the row's `roles`, expanded to permissions by policy. Scopes empty — a browser session is not scope-limited |
+| `agent` | an api key | **exactly** the key's scopes. Never the owning user's roles |
+| `service` | machine-to-machine inside the deployment | scopes only, no roles |
+| `anonymous` | no credential | nothing |
+
+`resolveActor()` is the single funnel. An agent that can do more than its key says is the failure mode that funnel exists to prevent.
+
+### Auth tables
+
+`@ultimat3/auth` exports `AUTH_TABLES` — the DDL `PostgresAuthAdapter` expects, as plain strings, so what
+auth stores is verifiable by reading. **Every boot applies them** (`@ultimat3/cli`'s
+`FRAMEWORK_SCHEMA`), the upgrade of an older `x_users` included, so an app writes no migration for
+`x_users`, `x_sessions`, `x_accounts`, `x_verifications` or `x_api_keys`, `As of 2026-09-23`. The
+per-table constants that were once pasted into migrations are gone (22.0.0).
+
+## The admin surface
+
+Two different things share the word.
+
+| Surface | Is | Gated by | Available |
+|---|---|---|---|
+| `/_x` | the **dev** dashboard from `@ultimat3/admin/dev` — routes, timeline, live, jobs, db, mail, cache, policy, manifest, services, boundaries (`DEV_PANELS`; `x dev` prints the 11) | dev-only, never mounted in production | in `x dev`, immediately |
+| `apps/admin/` | the app's own admin, served by `defineAdmin()` under `ROLE=web` | the app's role map (`roleAuthz()`), permission by permission | in `x dev`, immediately: a list, a detail and a form per entity, the audit log, and the jobs dashboard at `/admin/jobs`. Measured: `/admin`, `/admin/todos` and `/admin/jobs` answer 200 to the development viewer (`admin`) and 403 with the `myapp_dev_role=member` cookie |
+
+The scaffold writes a declaration, not a page, `As of 2026-10`:
+
+```ts
+// apps/admin/app/admin/admin.ts
+import { db } from '@myapp/db';
+import { adminEntitiesOf, defineAdmin } from '@ultimat3/admin';
+
+export const admin = defineAdmin({
+  // Every entity the handle serves: `x g entity` adds one there, and it is an admin screen on
+  // the next boot. Name them one by one instead — `entities: [post]` — to leave a table out.
+  entities: adminEntitiesOf(db),
+  db,
+});
+```
+
+There is no `page.tsx`, no repo adapter and no screen glue: `defineAdmin()` serves
+`/admin/<table>` from the entity and reads rows through the app's handle. A page file on a path the
+admin mounts is `X_ROUTE_DUPLICATE`; a `defineAdmin()` under `apps/*/src/`, which no boot
+imports, is `X_ADMIN_UNSCANNED`. Who may do what is `apps/web/shared/roles.ts`: `admin:read` +
+`<table>:read` to look, `admin:write` + `<table>:write` to create or edit, `admin:destroy` +
+`<table>:delete` to delete; `x g entity` and `x g resource` grant a new table's three to `admin`.
+`job:read` opens the jobs dashboard and `job:manage` runs its controls. A permission a mounted
+admin route asks for and no role grants is `X_PERMISSION_UNGRANTED` in the `policy` step.
+
+### Per-entity screens
+
+```bash
+bunx x g resource note --admin
+```
+
+`--admin` adds two files to the slice — `apps/web/app/note/admin/resource.ts` and its test — and
+lists the override under `resources:` in `apps/admin/app/admin/admin.ts` in the same run
+(`resources: { notes: noteAdminResource }`, with its import). In this app, which already holds the
+`todo` slice and its wiring:
+
+```text
+✓ wrote 42 file(s) for resource note — next: …
+``` The
+override is the only hand-written part; fields, operations, filters and forms derive from the entity.
+
+```ts
+export const noteAdminResource: AdminResourceOptions<AdminRow> = {
+  // Columns of the entity, in its own order. A name that is not one is X_ADMIN_FIELD_UNSUPPORTED
+  // at boot — delete the line to let the admin pick them.
+  listFields: ['title', 'price', 'createdAt'],
+  pageSize: 25,
+};
+```
+
+### Detail pages, actions, batches
+
+Everything past the list is declared on the same call. The reference app's operator view
+(`examples/dummy/apps/admin/app/admin/admin.ts`), cut to one resource and one action:
+
+```ts
+import { db, LIVE_RUN_STATUSES, runEvents, runs } from '@postly/db';
+import { api } from '@postly/web/api';
+import { defineAdmin } from '@ultimat3/admin';
+import { useContext } from '@ultimat3/core';
+
+export const admin = defineAdmin({
+  entities: [runs, runEvents],
+  db,
+  resources: {
+    runs: {
+      operations: ['list', 'detail', 'search'],
+      scopes: {
+        running: { where: [{ field: 'status', op: 'eq', value: 'running' }], count: true },
+        failed: { where: [{ field: 'status', op: 'eq', value: 'failed' }], count: true },
+      },
+      // A `hasMany` of the entity, drawn as `run_events`' own list filtered to this run.
+      related: ['run_events'],
+    },
+  },
+  actions: [
+    {
+      name: 'run.cancel',
+      permission: 'run:write',
+      entity: runs.$name,
+      // Decides the button on each row, and is asked again on the server before `handle`.
+      when: (row) => LIVE_RUN_STATUSES.some((status) => status === row['status']),
+      // A checkbox per row and an "all matching" choice in the list's batch bar.
+      batch: true,
+      handle: ({ input }) => {
+        const actor = useContext().actor;
+        return api.actions.cancelRun.as(actor, {
+          orgId: actor.orgId ?? '',
+          runId: String(input['id']),
+        });
+      },
+    },
+  ],
+});
+```
+
+| Declared | Served |
+|---|---|
+| `sections`, `formGroups`, `fields.<f>.hintKey`, `fields.<f>.on` | titled groups on the detail and the forms; a hint under a control; a field on one side of create/update only |
+| `related` | the related resource's own list — its columns, its row scope, its policy — under the row |
+| `input: t.object({ … })` on an action | the action's own form at `/admin/<entity>/<id>?action=<name>`, 422 with each issue on its field |
+| `when` | the button only where it applies; a forged post is `X_ADMIN_ACTION_NOT_APPLICABLE` (409) |
+| `batch` | checked rows or every row the list's URL matches, each through the button's gate and audited; past `batch.threshold` rows, `admin.batch` jobs on the worker |
+
+No island ships for any of it: the checkboxes join the batch form by their `form` attribute and a
+confirmation is a server round trip, so the admin stays at zero JavaScript. Every action is also
+an MCP tool, `admin.action.<name>`.
+
+### The rules the admin never breaks
+
+| Never | Why |
+|---|---|
+| a second authz system | two authz systems is how every framework of this shape died |
+| an admin-only user or role table | admins are users with permissions; a parallel identity store gets forgotten in offboarding |
+| a superuser bypass | there is no flag that skips policy evaluation |
+| offset pagination | offset re-scans every page and skips rows under concurrent writes. Keyset only |
+| a SQL console that writes | read-only, statement-capped, row-capped; writes and data-modifying CTEs are `X_MCP_QUERY_REJECTED` |
+
+Full surface: [Admin dashboard](Admin-Dashboard).
+
+## Agents inherit the human, exactly
+
+An action carrying `mcp: { expose: true }` becomes an MCP tool whose authorization **is** the action's `policy` object — `@ultimat3/mcp`'s projection calls the action's own `invoke`, so the policy evaluated is the very object `createTodo.policy` holds. The actor is the signed-in user's session, so an agent can never exceed the human it acts for. No trusted-tool mode, no second permission table, no "API permissions" screen to get wrong.
+
+A tool a caller may not see is absent from `tools/list` and answers ToolNotFound, never Forbidden.
+
+## Where the gate catches you
+
+| Mistake | Step | Code |
+|---|---|---|
+| an action with no `policy` | `typecheck` | build error — the field is required |
+| a permission string nothing declared | `typecheck` | `PermissionRegistry` augmentation narrows `can()` |
+| a route reading a table with no org predicate | `contract` | `X_TENANCY_UNSCOPED` |
+| a grant nothing enforces | — | not a gate; read the `x policy list` columns |
+| a permission a mounted admin route asks for that no role grants | `policy` | `X_PERMISSION_UNGRANTED` |
+| an app rule's `can()` naming a permission only `defineAdmin()` declared, never the app's own `definePermissions()` | `policy` | `X_PERMISSION_BORROWED` |
+
+## Next
+
+[Tutorial 4 — jobs and realtime](Tutorial-04-Jobs-And-Realtime): a durable job with replayed steps, a cron task with a required IANA zone, and a live query that patches per subscriber.
+
+Related: [Policies and authz](Policies-And-Authz) · [Admin dashboard](Admin-Dashboard) · [MCP and AI](MCP-And-AI) · [Configuration](Configuration)

package/wiki/Tutorial-04-Jobs-And-Realtime.md

@@ -0,0 +1,263 @@
+# Tutorial 4 — jobs, tasks and live queries
+
+Three primitives, three rules. A `job` is durable work with memoized steps. A `task` is a cron trigger that only ever enqueues. A `query` with `live: true` is a bounded, totally-ordered read that patches per subscriber.
+
+`As of 2026-11-03`. Every command, generated file and output on this page was re-run in the app tutorials 1–3 leave, built from this tree's packages (`scripts/scaffold-smoke-overrides.ts`), carrying the [tutorial 2](Tutorial-02-First-Feature) `todo` slice. The [capacity figures](#capacity-measured) are the committed benchmark results, quoted, not re-run.
+
+Series: [1 — first app](Tutorial-01-First-App) · [2 — first feature](Tutorial-02-First-Feature) · [3 — auth and admin](Tutorial-03-Auth-And-Admin) · **4** · [5 — deploy free](Tutorial-05-Deploy-Free) · [6 — growing up](Tutorial-06-Growing-Up)
+
+## A durable job
+
+`x g resource` already wrote one. Its shape is the whole contract:
+
+```ts
+export const reindexTodo = job({
+  input: t.object({ id: t.uuid, orgId: t.uuid }),
+  tenant: (input) => input.orgId,              // the org the body reads as, from its own input
+  idempotencyKey: ({ id }) => `reindex-todo:${id}`,
+  retry: { attempts: 5, backoff: 'exponential' },
+  async run({ input, step }) {
+    const row = await step.run('load', () => repo.byId(input.id));
+    if (row === null) return { skipped: true };
+    await step.run('process', async () => {
+      await repo.list(1);
+    });
+    return { skipped: false };
+  },
+});
+```
+
+| Field | Rule |
+|---|---|
+| `step.run('<name>', fn)` | each step's result is stored under its **name**. A retry replays completed steps from storage and re-executes only from the failure. Step names are stable identifiers, not labels |
+| `idempotencyKey` | an at-least-once caller may enqueue twice; the work happens once |
+| `retry` | per-attempt backoff, independent per step |
+| `.enqueue(input)` | the **only** queue path. A job is never run inline |
+
+The generated `reindex-todo.job.test.ts` pins the distant invariant rather than the happy path:
+
+```ts
+const first = await reindexTodo.enqueue({ id });
+expect(first.deduped).toBe(false);
+const again = await reindexTodo.enqueue({ id });
+expect(again.deduped).toBe(true);
+```
+
+Its `.job.test.ts` suffix is what runs it in the gate's `job` step rather than `unit` — see [tutorial 2](Tutorial-02-First-Feature#name-the-tests-after-their-step).
+
+Enqueue from an action and the outbox makes it transactional: the row and the job commit together, or neither does. `ctx.jobs` is the facade `<job>.enqueue()` resolves through. Full model: [Jobs and workflows](Jobs-And-Workflows).
+
+## A scheduled task
+
+```bash
+bunx x g task daily-digest --feature todo
+```
+
+```text
+  + apps/web/app/todo/tasks/daily-digest.ts
+  + apps/web/app/todo/tasks/daily-digest.job.test.ts
+  + apps/web/app/todo/tasks/daily-digest.test.ts
+  + apps/web/app/todo/jobs/daily-digest-job.ts
+  + apps/web/app/todo/jobs/daily-digest-job.job.test.ts
+  + apps/web/app/todo/jobs/daily-digest-job.test.ts
+  + apps/web/api/index.ts
+  + x.manifest.json
+  + openapi.json
+✓ wrote 9 file(s) for task daily-digest
+```
+
+`--feature todo` puts the task in the existing slice; without it the generator opens a `daily-digest` slice of its own. Either way the task and its job are added to `apps/web/api/index.ts`, which is what names them.
+
+```ts
+export const dailyDigest = task({
+  cron: '0 3 * * *',
+  tz: 'UTC',
+  // The org rides in the payload: the job declares its tenant from its own input, and a task has
+  // no request behind it whose org could be read instead.
+  enqueue: () => [[dailyDigestJob, { id: '…', orgId: '…' }]],
+});
+```
+
+| Rule | Why |
+|---|---|
+| a task **only** enqueues | the work is durable and retryable; a cron that did the work itself loses it on a restart |
+| `tz` is required, IANA | `0 3 * * *` in `America/New_York` is a different instant before and after a DST boundary. There is no ambient default anywhere in the framework |
+| one leader | leadership is an expiring lease row in `x_scheduler_leader`, never an advisory lock — that grant belongs to the **session**, not to the process: it outlives every transaction on the connection and is released only by an explicit unlock, the pool's reset on release, or the connection dying — so a node can neither renew it nor prove it still holds one. A second `scheduler` is a warm standby, not a duplicate |
+
+### Registration names it
+
+Export names become primitive names, and only `defineApi` (or `registerTasks`) stamps them. A task no `defineApi` lists keeps a positional name (`anonymous-task-1`, enqueuing `anonymous-job-2`); `x verify`'s `manifest` step refuses that job as `X_JOB_UNREGISTERED`. With the generator's edit to `apps/web/api/index.ts` in place:
+
+```bash
+bunx x tasks list
+```
+
+```text
+  name         cron       tz   catchUp  jobs            last  next
+  dailyDigest  0 3 * * *  UTC  skip     dailyDigestJob  -     2026-11-04T03:00:00Z
+✓ 1 task(s)
+```
+
+```bash
+bunx x tasks show dailyDigest --count 3
+```
+
+```text
+  kind: task
+  name: dailyDigest
+  cron: 0 3 * * *
+  tz: UTC
+  catchUp: skip
+  maxCatchUp: 10
+  jobs: ["dailyDigestJob"]
+  last: -
+  at 03:00 every day
+    2026-11-04T03:00:00Z
+    2026-11-05T03:00:00Z
+    2026-11-06T03:00:00Z
+✓ dailyDigest — 0 3 * * * (UTC), next …
+```
+
+Every instant renders in the task's **own** zone, never a machine-local default. Full model: [Scheduled tasks](Scheduled-Tasks).
+
+## Inspecting the queue
+
+```bash
+bunx x jobs list
+```
+
+```text
+  0 job(s)
+✓ 0 ready · 0 running · 0 delayed · 0 dead across 0 queue(s)
+```
+
+A dead job is never filtered out of view.
+
+| Subcommand | Does |
+|---|---|
+| `list` | queue depth, matching rows, and the dead-letter list |
+| `show <id>` | state, attempt, every step's result, remaining retry delays |
+| `retry <id> --from-step <name>` | drops that step so it re-executes; everything before it replays from storage |
+| `drain --to <driver>` | **planned** — no target ships. Would move `ready`/`delayed`/`suspended` jobs to another **durable** driver; enqueues on the target **before** acking the source. `--to memory` is refused by name (`X_CLI_BAD_FLAG`), `As of 2026-09`: it acked durable rows into a `Map` that died with the command |
+
+There is no Redis or NATS **job** driver, `As of 2026-10` — 25.0.0 deleted both all-throw stubs. So `drain` has no target: it exits `X_NOT_IMPLEMENTED` before the queue boots, having moved nothing. Postgres is the shipped driver, and it is the one `x dev` boots.
+
+## A live query
+
+The columns this read has an order for, then the read:
+
+```ts
+export const todoListSorts = ['createdAt', 'title'] as const;
+```
+
+```ts
+export const todoList = query({
+  input: t.object({
+    limit: t.number.default(50),
+    // A closed set, so a sort is a column this read has an order for, never a caller's string.
+    sort: t.enum(todoListSorts).default('createdAt'),
+    dir: t.enum(['asc', 'desc']).default('desc'),
+  }),
+  policy: canTodoRead,
+  live: true,
+  // Opt-in, unlike an action's tool: a read hands rows to an agent, so silence exposes nothing.
+  mcp: { expose: true, description: 'todoList — edit this description' },
+  sql: ({ limit, sort, dir }) =>
+    // `feature.table`, not the kebab plural: `from()` quotes the identifier into the SQL text,
+    // and the entity created the table as snake_case.
+    from<Todo>('todos', () => repo.list(limit, sort, dir))
+      // `repo.list`'s own order: the thunk is what the in-memory driver answers and this chain is
+      // the SQL Postgres runs, so a sort that differs here is a page holding different rows.
+      .orderBy(sort, dir)
+      // The primary key last, in the direction the handle adds it, makes the order TOTAL.
+      .orderBy('id', dir)
+      .limit(limit),
+});
+```
+
+`live: true` is not free. The gate rejects it without all four:
+
+| Requirement | Why |
+|---|---|
+| `orderBy` on a **total** order | the matcher decides *enters / leaves / moves within* the result from the changed row alone |
+| `limit` | an unbounded result has no bounded change buffer and no bounded reconnect snapshot |
+| no `now()` / `random()` | the same `(input, row)` must always yield the same membership answer |
+| no cross-tenant predicate | tenancy comes from `ctx`, not from `input` — otherwise `X_FORBIDDEN` at subscribe |
+
+The generated `todo-list.live.test.ts` asserts on the **SQL text**, because that is the contract an agent reads to self-correct:
+
+```ts
+const source = await sourceFor(
+  target,
+  { limit: 50 },
+  {
+    actor: null,
+    unenforced: 'a scaffolded test asserts the SQL text; the policy is asserted separately',
+  },
+);
+const text = source.toSQL().sql.toLowerCase();
+expect(text).toContain('order by "createdat" desc');
+expect(text).toContain('limit');
+// "ordered" is not enough — dropping the id tiebreak leaves an ORDER BY while going
+// non-deterministic under ties.
+expect(text.slice(text.lastIndexOf('order by'))).toContain('id');
+```
+
+### Policy runs per row, per subscriber
+
+Not a subscribe-time gate that then trusts the stream. The same `policy` object is re-evaluated for each delivered patch — which is why policy predicates are **synchronous**: an `await` there would be a database round trip per row per connected client.
+
+`toolFrom(todoList)` from `@ultimat3/mcp` is a read tool (`annotations.readOnlyHint: true`) that reads through `todoList`'s own policy. A read hands rows to an agent, so MCP exposure on a query is opt-in — silence exposes nothing.
+
+Five projections from the one declaration: HTTP `GET /_x/query/todo-list?limit=50` (`x routes` lists it as `query todoList · todo:read`), a typed client hook, the live subscription, a cache entry keyed by tenant and policy scope, and the MCP read tool. Full table: [Queries and live queries](Queries-And-Live-Queries).
+
+## Which role runs what
+
+<!-- no-run: boots every role on a fixed port; the run is the reader's -->
+```bash
+bunx x dev --once --port 3100
+```
+
+```text
+  roles web, sync, worker, scheduler
+```
+
+The same run logs `sync reachable` with the node on `ws://localhost:3101/_x/sync` and the app's own `http://localhost:3100/_x/sync` in front of it.
+
+| Role | Runs | Scales on |
+|---|---|---|
+| `web` | HTTP: pages, actions, queries | RPS |
+| `sync` | live-query websockets, on `$PORT + 1` | concurrent connections — no sticky sessions |
+| `worker` | the job queue | queue demand: jobs held plus jobs due |
+| `scheduler` | cron dispatch, enqueue only | fixed 1, lease-row leader (`x_scheduler_leader`) |
+| `replicator` | logical replication → change feed | 1 per database; opt in with `--role`, needs a real Postgres with `wal_level=logical` |
+
+`x dev` co-locates the first four in one process. Isolation is simulated, not skipped: separate ALS contexts, a real Postgres queue, a real SIGTERM drain.
+
+## Debugging, without grep
+
+| Question | Panel at `/_x` | CLI |
+|---|---|---|
+| which step failed, what is queued | `jobs` | `x jobs show <id>` |
+| why did this subscriber not get the row | `live` | — |
+| which clause decided, for which actor | `policy` | `x policy explain <subject>` |
+| which tags would this write bust | `cache` | — |
+| where did this request spend its milliseconds | `timeline` | — |
+
+`/_x/<panel>?json=1` returns exactly what the tab draws. `live` lists the registered live queries and states plainly that no subscriber list is attached — `@ultimat3/realtime` does not retain a subscriber's matcher trace. A panel whose source is unwired says which half is missing rather than rendering an empty tab as an answer.
+
+## Capacity, measured
+
+**Reachability.** 50,000 real WebSocket clients against a **single** `sync` node over the in-process transport, `SIGKILL`ed with no drain: all 50,000 reconnected, 49,981 received a channel patch inside the window, first patch on the reconnected socket at p50 **54.0s** / p90 **105.5s** / max **145.7s**, and 156,851 connect attempts shed by the accept budget before any query path. Committed at [`scripts/bench/results/50k-restart.json`](https://github.com/developerz-ai/ultimate/blob/main/scripts/bench/results/50k-restart.json).
+
+**Delivery.** A second run, 10,000 clients and a probe every 200ms, counting gaps in each client's received sequence: **1,666,882 channel patches received, 0 observed sequence gaps** ([`10k-restart-seq.json`](https://github.com/developerz-ai/ultimate/blob/main/scripts/bench/results/10k-restart-seq.json)). A first-patch timer cannot see a lost patch, which is why the second run exists — `As of 2026-08` it is the only one with delivery accounting. It says nothing about 50,000, and its zero is a lower bound: a gap needs a received frame on each side of it, so "no client observed a loss" is the claim, not "nothing was lost" ([Realtime](Realtime)).
+
+Both are per-node recovery — neither run crossed NATS, so neither is a multi-node result nor a throughput figure. Detail and limits: [Realtime](Realtime).
+
+Realtime tier 3 (local-first) shipped in 21.0.0, and it is declared on the **entity**, not the query: `entity(name, { persist: true })`, default `false`, keeps its records in IndexedDB beside the one outbox ([Realtime](Realtime#tier-3-shipped-in-2100)). `persist` is not a `query()` field — writing it there is a `TS2353` excess property.
+
+## Next
+
+[Tutorial 5 — deploy on a free tier](Tutorial-05-Deploy-Free): the scaffolded Dockerfile, `$PORT`, `0.0.0.0`, the health probes, and release-phase migrations — plus what a sleeping free instance does to the `worker` and `scheduler` roles.
+
+Related: [Jobs and workflows](Jobs-And-Workflows) · [Scheduled tasks](Scheduled-Tasks) · [Queries and live queries](Queries-And-Live-Queries) · [Realtime](Realtime)