@nylorun/runtime 0.13.0-beta → 0.15.0-beta

package/README.md

@@ -3,7 +3,7 @@
 The independent OSS **Runtime Host** consumes `@nylorun/harness/run` and
 `@nylorun/core/contracts`. Cloud installs published `@nylorun/harness` from npm
 and does not import this package. Client authoring and sessions belong to
-`@nylorun/agents`; Tenant lifecycle management belongs to `@nylorun/admin`.
+`@nylorun/agents`; the Host's status and derived keys belong to `@nylorun/admin`.
 Vocabulary: [src/CONTEXT.md](./src/CONTEXT.md).
 
 Requires Node 24+. Build from the repository root:
@@ -24,33 +24,66 @@ package is a library with no bin; its Host entry is `@nylorun/runtime/server`
 ephemeral embeds, use `startEphemeralRuntime()` from `@nylorun/runtime/core`.
 See [MIGRATION.md](../MIGRATION.md).
 
-No Tenant state lives only in the Runtime process. Each Tenant's data is the
-Postgres schema `tenant_<id>` (the Session Store); Restate runs one advance of a session at a
-time and holds the Tenant's sweep timer (Durable Session Execution); every
-session's events are relayed from the schema's outbox to its own S2 stream,
-which history and SSE read (Durable Streams). An API node serves the Tenant API,
-Admin API and SSE, and a Worker runs advances; `--role api|worker|all` picks
-them, and the local stack runs one process with `all`.
+A Runtime serves one **Tenant**: an installation is one Runtime with its own
+Postgres database, Restate and S2 basin, and two Tenants are two installations. No
+Tenant state lives only in the Runtime process. The Tenant's data is its Postgres
+database, in the fixed schemas `nylorun` (its state: the Session Store) and
+`nylorun_streams` (the record of its session events). Drizzle defines the tables
+(`src/store/postgres/schema.ts`) and generates the migrations, which ship in the package
+(`dist/store/postgres/drizzle/`). At startup the Host applies the missing ones in one
+transaction under an advisory lock, records them in `nylorun.__drizzle_migrations`, and
+refuses a database holding a migration it does not ship (`schema-too-new`); then, on
+first start, it creates the Tenant there: its id (`NYLORUN_TENANT_ID`, default a
+new one), its name (`NYLORUN_TENANT_NAME`, default `default`), its Studio principal and
+the derived principals (`NYLORUN_DERIVED_PRINCIPALS`, default `project`) whose keys the
+admin key derives. A database written by a Runtime that kept several Tenants in one
+database (`tenant_<id>` schemas), or by a pre-release build of one Tenant per database
+(`schema_version` tables), is refused: this release starts fresh on a new database. Restate runs one advance of a session at a time and holds the Tenant's sweep
+timer (Durable Session Execution); every session's events are relayed from the record to
+its own S2 stream, which history and SSE read (Durable Streams). One image runs every **service**,
+and `--service` picks what a process runs: `core` serves the Tenant API, Admin
+API and SSE and runs the stream relay, `loop` runs advances (the Worker), and
+`gates` is the Model Gate, the only process that reads a model credential and
+calls providers. `core` and `loop` may share a process; `gates` never joins
+them. The local stack packs `core,loop` into the `runtime` container and
+`gates` into the `gateway` container (the combined packing). `--role
+api|worker|all` is the deprecated name of `--service core`, `loop` and
+`core,loop`.
 
 The container is configured by its environment, which the stack's Compose file
 sets: `NYLORUN_DATABASE_URL` (required), `NYLORUN_RESTATE_INGRESS_URL`,
 `NYLORUN_RESTATE_ADMIN_URL`, `NYLORUN_WORKER_URL` and
 `NYLORUN_RESTATE_IDENTITY_KEY` (Restate), `NYLORUN_S2_ENDPOINT` and
 `NYLORUN_S2_TOKEN`, and in container mode `NYLORUN_LISTEN_HOST`,
-`NYLORUN_LISTEN_PORT`, `NYLORUN_ALLOWED_HOSTS` and `NYLORUN_PUBLIC_URL`.
+`NYLORUN_LISTEN_PORT`, `NYLORUN_ALLOWED_HOSTS` and `NYLORUN_PUBLIC_URL`, plus
+`NYLORUN_BROWSER_ACCESS` (`on` or `off`) and the operator listener
+(`NYLORUN_ADMIN_LISTEN_PORT`, `NYLORUN_ADMIN_LISTEN_HOST`,
+`NYLORUN_ADMIN_ALLOWED_HOSTS`). A process that runs `loop` sends its model
+calls to the gate at `NYLORUN_GATES_URL` with `NYLORUN_GATES_TOKEN`; in a
+container it refuses to start without them. A `gates` process needs only
+`NYLORUN_DATABASE_URL`, `NYLORUN_GATES_TOKEN`, its listener
+(`NYLORUN_GATES_LISTEN_HOST`, `NYLORUN_GATES_LISTEN_PORT`, default 4100, and
+`NYLORUN_GATES_ALLOWED_HOSTS`) and the Host's `tenant/` directory, which it
+never writes; it serves the database's one Tenant. `NYLORUN_PACKING` (`combined` or `split`) is logged at startup.
+
+With an operator listener (the stack's default: container port 4001), the Host
+serves two ports. The public one serves the Tenant API, to browsers too when
+browser access is on, and answers admin routes with the opaque `404`. The
+operator one serves the Admin API, Host shutdown and the Tenant API, never to
+browsers; keep it on loopback or a private network and never proxy it. Outside
+a container, `adminPort` in `host.json` does the same on loopback.
 
 The Host root is `NYLORUN_HOME` or `~/.nylorun` (bind-mounted at `/nylorun` in
-the stack). What stays on the Host is under `tenants/<tenantId>/`.
+the stack). What stays on the Host is under `tenant/`.
 
 ## Layout
 
 ```text
 <host root>/
-  host.json                 # format 1: hostId, host, port, runtimeVersion, …
+  host.json                 # format 1: hostId, host, port, adminPort?, runtimeVersion, …
   host-credentials.json     # adminKey (0600)
-  stack/                    # compose.yaml, .env (0600), Restate identity key
-  tenants/<tenantId>/       # vault-kek, plugin-data, logs, home, tmp, sandboxes
-  trash/                    # SQLite Tenants from before the Postgres switch
+  docker/                   # compose.yaml, .env (0600), Restate identity key
+  tenant/                   # vault-kek, plugin-data, logs, home, tmp, sandboxes
 ```
 
 ## HTTP surface
@@ -58,29 +91,90 @@ the stack). What stays on the Host is under `tenants/<tenantId>/`.
 | Route | Auth | Notes |
 | --- | --- | --- |
 | `GET /health` | none | `service: "nylorun-runtime"`, `hostId`, protocol `{min,max,features}`, pid |
-| `GET /ready` | none | Listener up, Tenants discovered, and Postgres, Restate and S2 answer (`checks`) |
-| `GET /v1/admin/status` | admin key | `AdminStatusSchema`; alias `GET /v1/admin/host` |
-| `/v1/admin/tenants*` | admin key | Create / list / get / delete Tenants |
+| `GET /ready` | none | Listener up, the Tenant open, and Postgres, Restate and S2 answer (`checks`) |
+| `GET /openapi.json` | none | The Tenant API's OpenAPI 3.2 document (below); refuses an `Origin` |
+| `GET /v1/admin/status` | admin key | `AdminStatusSchema`: the Host, its protocol and its Tenant (`tenant`: id, name, `open` or `unavailable`, and the cause when it could not be opened); alias `GET /v1/admin/host`. On the operator listener when there is one |
 | `POST /v1/admin/host/shutdown` | admin key | Host-private; not in `@nylorun/admin` |
-| `/v1/*` Tenant routes | application or executor | Require `Nylorun-Tenant` + `Nylorun-Protocol` |
+| `GET /v1/admin/openapi.json` | admin key | The Admin API's OpenAPI 3.2 document |
+| `/v1/*` Tenant routes | application key, subject token or delivery token | Require `Nylorun-Protocol`; nothing names the Tenant |
+
+Every route checks `Host` first (`421 host_rejected`) and rejects non-JSON bodies
+with `415 unsupported_media_type`. An `Origin` is `403 origin_rejected` on
+`/health`, `/ready`, admin routes, and everywhere when browser access is off.
+With browser access on (feature `browser-access`: the stack's default, or
+`browserAccess` in `host.json`), the Host answers preflights for browser routes
+from the route alone, and the Tenant admits an `Origin` only with a publishable
+key (`Nylorun-Key`) that lists it, adding CORS headers only then; Tenant keys
+and delivery tokens are refused from browsers before they are looked up. Missing or
+unsupported protocol → `426` before authentication (the Host serves protocols 4 and 5).
+Protocol 5 clients name no Tenant. A protocol 4 client's `Nylorun-Tenant` naming
+another Tenant (or malformed), a publishable key of another Tenant, a Tenant that
+could not be opened and rejected credentials → opaque `404` with identical body. A path or method no route serves is `404 Route not found`
+once the caller is known.
+
+## API reference (OpenAPI)
+
+The Runtime describes its APIs as OpenAPI 3.2 documents, generated from the routes it serves,
+so they cannot drift from its answers:
+
+| Document | In the package | Served | On each release |
+| --- | --- | --- | --- |
+| Tenant API | `@nylorun/runtime/openapi.json` | `GET /openapi.json` | `openapi.json` |
+| Admin API | `@nylorun/runtime/admin-openapi.json` | `GET /v1/admin/openapi.json` (admin key) | `admin-openapi.json` |
+
+The release assets are on the `@nylorun/runtime@<version>` GitHub Release. Point any OpenAPI 3.2
+tool at one of them; for [Scalar](https://scalar.com), the package file of a version works as is:
 
-Every route checks `Host` first (`421 host_rejected`), rejects any `Origin`
-(`403 origin_rejected`, no CORS headers), and rejects non-JSON bodies with
-`415 unsupported_media_type`. Missing or unsupported protocol → `426` before
-authentication. Unknown, quarantined or rejected Tenant credentials → opaque
-`404` with identical body.
+```text
+https://cdn.jsdelivr.net/npm/@nylorun/runtime@<version>/dist/openapi.json
+```
+
+Each operation's `security` says which credentials it takes (application key, subject token,
+publishable key, delivery token, admin key), and its `x-nylorun-credentials`,
+`x-nylorun-scopes` (the subject scopes that reach it) and `x-nylorun-browser` fields say who
+may call it. Event streams are `text/event-stream` with an `itemSchema`. `runtime/openapi/` holds the committed snapshots: a change to a route changes
+them (`node scripts/build-openapi.mjs --write`), and `check-package` fails until they are
+updated.
+
+## HTTP layer
+
+Routes are Hono routes (`@hono/zod-openapi`), each declared once with who may call it
+(`api/http/define.ts`): that declaration serves the route, checks subject scopes, answers
+browser preflights and makes the OpenAPI document. `host/` holds the listeners, the `Host`
+check (in Node, before Hono) and the Host pipeline (`host/app.ts`); `api/` the Tenant and
+Admin routes (`api/http/routes/`, `api/ag-ui/`, `api/a2a/`, `host/admin-api.ts`). Only `host/`
+and `api/` import Hono.
 
 ## Embedding and tests
 
 Tests and ephemeral embeds use `startEphemeralRuntime()` from
 `@nylorun/runtime/core`: private Host on port 0, temporary Host root, one Tenant,
-returns `{ url, tenantId, applicationKey, adminKey, close() }`. It is not durable:
-its Tenants live in memory and are gone after `close()`. The smoke checks'
-temporary Tenants (`scripts/lib/temporary-tenant.mjs`) do not use it; they are
-created on the Docker stack.
+returns `{ url, tenantId, applicationKey, adminKey, close() }`. Its Tenant is the
+one Tenant of the Postgres database you pass as `database` (required): a pool, which
+you end, or a URL, for which it opens a pool and ends it on `close()`. It creates the
+Tenant there on first start, as a Host does (with `tenantId`, `name` and an application
+principal for `applicationKey`), and serves the Tenant a database already holds. The
+data stays after `close()`; give each test Tenant its own database and drop it
+afterwards. Its
+streams and scheduling are in process and gone after `close()`. The smoke checks do
+not use it; they reset and seed the Tenant of a temporary Docker stack
+(`scripts/lib/stack-tenant.mjs`).
+
+```ts
+const runtime = await startEphemeralRuntime({
+  hostRoot, // a temporary directory
+  database: "postgres://nylorun:nylorun@127.0.0.1:55432/my_test_db",
+});
+```
+
+The package's own tests run every Session Store on Postgres: `npm test` gives each
+test file a database on the Docker test stack (`test/stack/compose.yaml`) and starts
+the stack when it is down; see [CONTRIBUTING.md](../CONTRIBUTING.md).
 
-Register executors with `PUT /v1/executors` using the application principal
-(application-mode `connectAgents` does this with derived tokens). Model gateway
+Register Action endpoints with `PUT /v1/endpoints` using the application principal
+(`createActionHandler(...).register({ url })` in `@nylorun/agents` does this). The
+Runtime delivers each Action there with an `http` or `https` POST, so the URL must be
+reachable from the Runtime. Model gateway
 and sandbox backend are Tenant configuration (vault / seed), not Host process
 env.
 
@@ -88,12 +182,12 @@ env.
 
 The session-first `/v1` API is specified in
 [HOST_CONTRACT.md](../harness/HOST_CONTRACT.md). Application tokens authorize
-definition/session APIs; executor credentials authorize authenticated SSE
-connect, action discovery, claims, renewal and results. Session observers cannot
-claim actions.
+definition/session APIs. The Runtime delivers Actions to Action endpoints, and a
+delivery token authorizes only its own Action's heartbeat, result and sandbox calls.
+Session observers cannot submit action results.
 
 Postgres transactions persist session checkpoints, command receipts, individual
-effects/actions, waits and the event outbox — **per Tenant schema**; history is
+effects/actions, waits and the record of session events — in the Tenant's database; history is
 read from Durable Streams (S2). An advance owns its session through a lease with
 an epoch; a Worker that takes over after a crash marks in-flight effects
 `uncertain`. Vault ciphertext needs that Tenant's own KEK.
@@ -103,22 +197,24 @@ Tenant owns each sandbox; backend names are prefixed `nylorun-<tenant-id>-`.
 
 ## Local Project workflow
 
-A project depends on `@nylorun/agents` only. `npx nylorun up` runs the local
-stack (Docker); `npx @nylorun/cli tenant create` (`nylo`) creates the
-project's Tenant and writes its Project link; the project's `npm run dev` runs
-`src/main.ts` under `tsx watch`; `npx nylorun studio` opens Studio.
+A project depends on `@nylorun/agents` only. `npx nylorun start` in the project
+runs its local stack (Docker), whose Runtime creates the stack's one Tenant, and
+writes the Project link; the project's `npm run dev` runs `src/main.ts` under
+`tsx watch`; `npx nylorun studio` opens Studio on that Tenant.
 
 ## Troubleshooting
 
 | Symptom | What to do |
 | --- | --- |
 | `kek-missing` | Restore `vault-kek` in the Tenant directory |
-| `corrupt` / `migration-failed` / `envelope-invalid` / `open-failed` / `open-timeout` | Follow the `nylo tenant status` repair string |
-| `schema-too-new` | Run a Runtime at least as new as the one that migrated the schema |
+| `GET /ready` is 503 and `/v1/admin/status` names a `tenant.cause` | The Tenant could not be opened; every Tenant request is the opaque `404`. Follow the cause's `repair`, then restart the Runtime |
+| `corrupt` / `migration-failed` / `envelope-invalid` / `open-failed` / `open-timeout` | Follow the cause's `repair` string |
+| `schema-too-new` | Run a Runtime at least as new as the one that migrated the database |
+| `database-layout-old` | The database holds `tenant_<id>` schemas of an older Runtime, or the `schema_version` tables of a pre-release one: point the Runtime at a new database (a new stack); the old one is left as it is |
 | `426 protocol_unsupported` | Upgrade clients or Host to a compatible set |
-| `421 host_rejected` / `403 origin_rejected` | Call from a server process, not a browser; in a container, list the `Host` in `NYLORUN_ALLOWED_HOSTS` |
+| `421 host_rejected` / `403 origin_rejected` | In a container, list the `Host` in `NYLORUN_ALLOWED_HOSTS`. From a browser, use a subject token and a publishable key that lists the page's origin, never a Tenant key |
 | `503` for a Tenant | Postgres or Restate is unreachable; `GET /ready` names which |
-| Port in use | Change `NYLORUN_PORT` in `<Host root>/stack/.env` and run `nylorun up` |
+| Port in use | Change `NYLORUN_PORT` in `~/.nylorun/stacks/<name>/docker/.env` and run `nylorun start` |
 | Logs | `nylorun logs runtime` |
 
 Definitions have no `agent.run()`; applications use `@nylorun/agents`.

package/dist/adapters/execution/restate.d.ts

@@ -90,8 +90,10 @@ export declare class RestateExecution implements DurableExecution {
         session: string;
         tenant: string;
         timer: string;
+        action: string;
     };
     wake(tenantId: string, sessionId: string, wake: Wake): Promise<void>;
+    deliver(tenantId: string, actionId: string): Promise<void>;
     timer(tenantId: string, key: string, at: Date): Promise<void>;
     armSweep(tenantId: string): Promise<void>;
     disarmSweep(tenantId: string): Promise<void>;
@@ -106,8 +108,11 @@ export declare class RestateExecution implements DurableExecution {
     start(handlers: WorkerHandlers): Promise<void>;
     stop(): Promise<void>;
     private definitions;
-    /** Runs `advance` with a signal aborted by `stop` or by the end of the Restate attempt. */
-    private runAdvance;
+    /**
+     * Runs a Worker handler with a signal aborted by `stop` or by the end of the Restate
+     * attempt (`advance` and `deliver`).
+     */
+    private runAborted;
     /** Runs a Worker handler so that `stop` waits for it. Refuses once stopping; Restate retries. */
     private runTracked;
     private staleSweepMs;

package/dist/adapters/execution/restate.js

@@ -1,7 +1,7 @@
 /**
  * Durable Session Execution on Restate (architecture §12.3).
  *
- * The only module that imports the Restate SDK. It serves three virtual
+ * The only module that imports the Restate SDK. It serves four virtual
  * objects on the Worker endpoint, all named with `servicePrefix`:
  *
  * - `<prefix>NylorunSession`, key `<tenantId>:<sessionId>`. Its exclusive
@@ -12,14 +12,19 @@
  * - `<prefix>NylorunTenant`, key `<tenantId>`. `arm`, `sweep` and `disarm`
  *   keep one self-re-arming sweep chain per Tenant. The chain's generation is
  *   kept in the object's state, so arming is idempotent and a disarmed or
- *   replaced chain dies at its next link. The state is rebuildable: Workers
- *   re-arm every Tenant at startup (§14.8).
+ *   replaced chain dies at its next link. The state is rebuildable: a Host
+ *   re-arms its Tenant's sweep when it opens it, at every start (§14.8).
  * - `<prefix>NylorunTimer`, key `<tenantId>:<timer key>`. `set` records the
  *   latest time and sends a delayed `fire`; a `fire` for an older time is a
  *   no-op, which is how setting a key again replaces it.
+ * - `<prefix>NylorunAction`, key `<tenantId>:<actionId>`. Its exclusive
+ *   `deliver` handler calls `WorkerHandlers.deliver`. Like `advance`, it keeps
+ *   no state and makes no `ctx.run`: its only journal entry is the delayed
+ *   self-send after a `retry` result, so an endpoint that is down never spends
+ *   the retry budget, which is for infrastructure errors.
  *
- * Wakes, timers and sweep arming go through the ingress as one-way sends, so
- * an API node can call them without ever calling `start`.
+ * Wakes, deliveries, timers and sweep arming go through the ingress as one-way
+ * sends, so an API node can call them without ever calling `start`.
  */
 import { createServer } from "node:http2";
 import * as restate from "@restatedev/restate-sdk";
@@ -49,6 +54,7 @@ export class RestateExecution {
             session: `${prefix}NylorunSession`,
             tenant: `${prefix}NylorunTenant`,
             timer: `${prefix}NylorunTimer`,
+            action: `${prefix}NylorunAction`,
         };
         this.ingressUrl = options.ingressUrl.replace(/\/+$/, "");
         this.adminUrl = options.adminUrl.replace(/\/+$/, "");
@@ -63,6 +69,9 @@ export class RestateExecution {
             throw new Error(`Unknown wake reason: ${String(wake.reason)}`);
         await this.send(this.names.session, sessionKey(tenantId, sessionId), "advance", { reason: wake.reason }, wake.dedupeKey);
     }
+    async deliver(tenantId, actionId) {
+        await this.send(this.names.action, sessionKey(tenantId, actionId), "deliver", {});
+    }
     async timer(tenantId, key, at) {
         await this.send(this.names.timer, sessionKey(tenantId, key), "set", {
             at: at.getTime(),
@@ -161,7 +170,7 @@ export class RestateExecution {
             handlers: {
                 advance: async (ctx, _wake) => {
                     const { tenantId, sessionId } = parseSessionKey(ctx.key);
-                    const result = await this.runAdvance(ctx.request().attemptCompletedSignal, (handlers, signal) => handlers.advance(tenantId, sessionId, signal));
+                    const result = await this.runAborted(ctx.request().attemptCompletedSignal, (handlers, signal) => handlers.advance(tenantId, sessionId, signal));
                     if (result.status === "busy")
                         ctx
                             .objectSendClient({ name: names.session }, ctx.key)
@@ -232,10 +241,31 @@ export class RestateExecution {
             },
             options: serviceOptions,
         });
-        return [session, tenant, timer];
+        const action = restate.object({
+            name: names.action,
+            handlers: {
+                deliver: async (ctx, _input) => {
+                    const { tenantId, sessionId: actionId } = parseSessionKey(ctx.key);
+                    const result = await this.runAborted(ctx.request().attemptCompletedSignal, (handlers, signal) => {
+                        if (!handlers.deliver)
+                            throw new Error("WorkerHandlers.deliver is required for deliveries");
+                        return handlers.deliver(tenantId, actionId, signal);
+                    });
+                    if (result.status === "retry")
+                        ctx
+                            .objectSendClient({ name: names.action }, ctx.key)
+                            .deliver({}, restate.rpc.sendOpts({ delay: Math.max(0, result.retryAfterMs) }));
+                },
+            },
+            options: serviceOptions,
+        });
+        return [session, tenant, timer, action];
     }
-    /** Runs `advance` with a signal aborted by `stop` or by the end of the Restate attempt. */
-    async runAdvance(attemptDone, fn) {
+    /**
+     * Runs a Worker handler with a signal aborted by `stop` or by the end of the Restate
+     * attempt (`advance` and `deliver`).
+     */
+    async runAborted(attemptDone, fn) {
         const controller = new AbortController();
         const onAttemptDone = () => controller.abort(new Error("Restate attempt ended"));
         if (attemptDone.aborted)
@@ -335,7 +365,7 @@ export async function listStuckInvocations(options) {
     const prefix = options.servicePrefix ?? "";
     if (!/^[A-Za-z0-9_]*$/.test(prefix))
         throw new Error(`Invalid Restate service prefix: ${prefix}`);
-    const services = ["NylorunSession", "NylorunTenant", "NylorunTimer"].map((name) => `'${prefix}${name}'`);
+    const services = ["NylorunSession", "NylorunTenant", "NylorunTimer", "NylorunAction"].map((name) => `'${prefix}${name}'`);
     const clauses = [
         `target_service_name IN (${services.join(", ")})`,
         `status IN ('paused', 'backing-off')`,

package/dist/adapters/replication/pgoutput.d.ts

@@ -0,0 +1,21 @@
+import type { ChangeSource } from "../../streams/relay/types.js";
+export interface PgoutputSourceOptions {
+    /** A Postgres URL whose role may replicate (`REPLICATION`) and read `nylorun_streams`. */
+    connectionString: string;
+    /**
+     * The Tenant the database holds. The record has no Tenant column (one Tenant per
+     * database), so every row read is this Tenant's.
+     */
+    tenantId: string;
+    /** Default `nylorun_stream_relay`. */
+    slot?: string;
+    /** Default `nylorun_stream_relay`. */
+    publication?: string;
+    /** The schema of the record tables. Default `nylorun_streams`. */
+    schema?: string;
+    /** Wait between attempts to take the slot. Default 5 s. */
+    retryMs?: number;
+    log?: (message: string, fields?: Record<string, unknown>) => void;
+}
+export declare const DEFAULT_RELAY_SLOT = "nylorun_stream_relay";
+export declare function createPgoutputSource(options: PgoutputSourceOptions): ChangeSource;

package/dist/adapters/replication/pgoutput.js

@@ -0,0 +1,170 @@
+/**
+ * The stream relay's change source on Postgres logical replication (Durable Streams §7.1,
+ * §7.4, §7.6): a persistent `pgoutput` slot on the `nylorun_stream_relay` publication, read
+ * with `pg-logical-replication`. The only file importing it (with `pg`, which it needs).
+ *
+ * - **One reader per slot.** Postgres lets one connection hold a slot, so every Runtime
+ *   process can run a source and exactly one is active; the others retry every few seconds
+ *   and take over when it goes away. The slot is the leader election.
+ * - **Acknowledged only on request.** The library acknowledges the last position it
+ *   *received* on a timer; that is turned off (`timeoutSeconds: 0`) and keepalives are
+ *   answered by the relay, which acknowledges only what S2 has.
+ * - **Always from the confirmed position.** Each attempt is a new connection that starts at
+ *   `0/0`, which Postgres reads as the slot's confirmed position, so a reconnect replays
+ *   whatever was received but not acknowledged.
+ * - **Fresh slots.** A missing slot is created; a slot Postgres invalidated
+ *   (`max_slot_wal_keep_size`) is dropped and created again. Either way the relay must
+ *   reconcile the record with S2; that is recorded in `relay_slots` until `reconciled()`, so a
+ *   crash during reconciliation, or another process winning the slot, does not lose it.
+ */
+import pg from "pg";
+import { LogicalReplicationService, PgoutputPlugin } from "pg-logical-replication";
+export const DEFAULT_RELAY_SLOT = "nylorun_stream_relay";
+export function createPgoutputSource(options) {
+    const slot = options.slot ?? DEFAULT_RELAY_SLOT;
+    const publication = options.publication ?? "nylorun_stream_relay";
+    const schema = options.schema ?? "nylorun_streams";
+    const retryMs = options.retryMs ?? 5000;
+    const { tenantId } = options;
+    const log = options.log ?? (() => { });
+    const pool = new pg.Pool({
+        connectionString: options.connectionString,
+        max: 2,
+        application_name: "nylorun-stream-relay",
+    });
+    let service;
+    let stopped = false;
+    let running;
+    let wake;
+    /** Creates the slot when missing or lost; true when the record must be reconciled. */
+    async function prepareSlot() {
+        const client = await pool.connect();
+        try {
+            const { rows } = await client.query("SELECT wal_status, active FROM pg_replication_slots WHERE slot_name = $1", [slot]);
+            const existing = rows[0];
+            if (existing?.wal_status === "lost" && !existing.active) {
+                log("replication slot was invalidated; creating it again", { slot });
+                await client.query("SELECT pg_drop_replication_slot($1)", [slot]);
+            }
+            if (!existing || existing.wal_status === "lost") {
+                try {
+                    await client.query("SELECT pg_create_logical_replication_slot($1, 'pgoutput')", [slot]);
+                    log("replication slot created", { slot });
+                    await client.query(`INSERT INTO ${schema}.relay_slots (slot_name, reconcile_pending) VALUES ($1, true)
+             ON CONFLICT (slot_name) DO UPDATE SET reconcile_pending = true`, [slot]);
+                }
+                catch (error) {
+                    // Another process created it first; its row says whether to reconcile.
+                    if (error.code !== "42710")
+                        throw error;
+                }
+            }
+            const pending = await client.query(`SELECT reconcile_pending FROM ${schema}.relay_slots WHERE slot_name = $1`, [slot]);
+            // A slot created before this table knew it: reconcile once to be sure.
+            return pending.rows[0]?.reconcile_pending ?? true;
+        }
+        finally {
+            client.release();
+        }
+    }
+    async function attempt(handlers) {
+        const fresh = await prepareSlot();
+        // `stop()` during `prepareSlot` had no service to stop: do not start one it cannot.
+        if (stopped)
+            return;
+        const current = new LogicalReplicationService({ connectionString: options.connectionString, application_name: "nylorun-stream-relay" }, { acknowledge: { auto: false, timeoutSeconds: 0 }, flowControl: { enabled: false } });
+        service = current;
+        let decoding;
+        current.on("start", () => handlers.onActive({ fresh }));
+        current.on("data", (_lsn, message) => {
+            switch (message.tag) {
+                case "begin":
+                    decoding = { rows: [] };
+                    break;
+                case "insert":
+                    if (decoding &&
+                        message.relation?.schema === schema &&
+                        message.relation?.name === "session_events")
+                        decoding.rows.push(rowOf(message.new, tenantId));
+                    break;
+                case "commit": {
+                    const tx = { endLsn: message.commitEndLsn, rows: decoding?.rows ?? [] };
+                    decoding = undefined;
+                    // A transaction with no record rows still moves the slot once acknowledged.
+                    handlers.onTx(tx);
+                    break;
+                }
+            }
+        });
+        current.on("heartbeat", (lsn, _timestamp, shouldRespond) => {
+            // Mid-transaction, the server's position may be past rows not yet delivered.
+            if (!shouldRespond || decoding)
+                return;
+            const position = handlers.onKeepalive(lsn);
+            if (position)
+                void current.acknowledge(position).catch(() => { });
+        });
+        current.on("error", () => { });
+        const plugin = new PgoutputPlugin({ protoVersion: 1, publicationNames: [publication] });
+        // Resolves when replication ends, rejects when it fails; `0/0` = the slot's confirmed position.
+        await current.subscribe(plugin, slot, "0/00000000");
+    }
+    async function run(handlers) {
+        while (!stopped) {
+            let failure;
+            try {
+                await attempt(handlers);
+            }
+            catch (error) {
+                failure = error;
+            }
+            if (stopped)
+                break;
+            const busy = failure?.code === "55006";
+            // Another process holds the slot: it is the relay. Anything else: report and retry.
+            handlers.onInactive(busy ? undefined : failure ?? new Error("replication ended"));
+            await new Promise((resolve) => {
+                wake = resolve;
+                setTimeout(resolve, retryMs).unref?.();
+            });
+            wake = undefined;
+        }
+    }
+    return {
+        start(handlers) {
+            if (running)
+                throw new Error("The change source is already started");
+            running = run(handlers);
+        },
+        acknowledge(lsn) {
+            void service?.acknowledge(lsn).catch(() => { });
+        },
+        async reconciled() {
+            await pool.query(`UPDATE ${schema}.relay_slots SET reconcile_pending = false WHERE slot_name = $1`, [
+                slot,
+            ]);
+        },
+        async lag() {
+            const { rows } = await pool.query(`SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)::text AS lag
+         FROM pg_replication_slots WHERE slot_name = $1`, [slot]);
+            return rows[0]?.lag == null ? undefined : Number(rows[0].lag);
+        },
+        async stop() {
+            stopped = true;
+            wake?.();
+            await service?.stop().catch(() => undefined);
+            await running?.catch(() => undefined);
+            await pool.end().catch(() => undefined);
+        },
+    };
+}
+/** A decoded `session_events` row (`pg` parses `json`; `bigint` arrives as text). */
+function rowOf(values, tenantId) {
+    return {
+        tenantId,
+        sessionId: values.session_id,
+        seq: Number(values.seq),
+        generation: Number(values.generation),
+        body: typeof values.body === "string" ? JSON.parse(values.body) : values.body,
+    };
+}