@nylorun/runtime 0.10.0-beta → 0.11.0-beta

package/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## 0.11.0-beta
+
+### Minor Changes
+
+- bf1c2da: **`nylorun dev --ephemeral` runs on the Docker stack, with a Tenant-level fixture model (breaking beta).**
+
+  - `nylorun dev --ephemeral` creates a temporary Tenant through `@nylorun/admin` (no Project link written), seeds it from `.env` with the fixture model, opens Studio on it and runs the watcher; the Tenant is deleted with its active work cancelled when the watcher ends, Ctrl-C included. It needs a Runtime that advertises `tenant-fixture-model`.
+  - `PUT /v1/tenant/config/seed` accepts `fixtureModel: true`, stored as Tenant setting `model.fixture`: that Tenant's model calls use the Runtime's fixture model while other Tenants on the Host keep theirs. `HOST_PROTOCOL` advertises the new optional feature `tenant-fixture-model` (`OPTIONAL_HOST_FEATURES` in `@nylorun/core/compatibility`); clients do not require it.
+  - `startEphemeralRuntime()` keeps its signature but its Tenants live in memory instead of SQLite under the Host root; nothing survives `close()`.
+  - Closing a Tenant waits for running advances at most the advance grace period (30 s by default) and then abandons them; their lease lapses and the next advance takes over.
+  - A Worker stop, Tenant close or ended Restate attempt no longer fails or cancels the turn it interrupts: outcomes already returned are recorded, nothing is settled, and the next advance resumes the turn from its checkpoint. Only a user cancel settles `cancelled`; an advance deadline fails the turn with the deadline's message.
+
+- bf1c2da: **The microsandbox backend is removed; the Runtime runs sandbox tools on the virtual backend only.** The optional `microsandbox` dependency is gone. `sandbox.backend` and `NYLORUN_SANDBOX` accept `auto` or `virtual`, and `auto` selects `virtual`. A Tenant that stored `microsandbox` reads it as `auto`. `nylorun doctor sandbox` and the `nylorun dev` banner report only the virtual shell.
+- bf1c2da: **Tenants are Postgres schemas only; SQLite and the `nylorun-runtime` launcher are removed (breaking beta).**
+
+  - The Runtime runs on Postgres (the Session Store, one schema per Tenant), Restate (Durable Session Execution: one advance per session, fenced by an ownership epoch, and a per-Tenant sweep) and S2 (Durable Streams: every event is written to a Postgres outbox and relayed to the session's stream, which history and SSE read). `--role api|worker|all` selects the process role; `/ready` covers Postgres, Restate and S2.
+  - SQLite Tenants are not migrated. On first start the Runtime moves every `tenants/<id>/` directory that holds a `tenant.sqlite` to `trash/<id>-sqlite-<time>/` and logs `sqlite_tenant_moved_to_trash`; recreate those Tenants.
+  - `@nylorun/runtime` has no bin: the `nylorun-runtime` launcher and `host-state.json` are gone. The Runtime runs as the `ghcr.io/nylorun/runtime` image (`nylorun start`); its Host entry requires `NYLORUN_DATABASE_URL`.
+  - `openTenantRuntime(config, hooks)` requires the Tenant's opened `store` and `envelope`; `OpenTenantRuntime` receives them from the Tenant store. `HostStateFile` is no longer exported.
+  - `@nylorun/core`: `LAUNCHER_PROTOCOL` and the launcher error codes are removed; `QuarantineSchema` drops `locked`, `lockPath` and `lockPid`; `TenantStatusSchema.checks.sqlite` is now `checks.store`. `TenantStatusSchema` gains optional `execution` (stuck invocations) and `streams` (basin, outbox depth, relay lag); `HostAggregateSchema` gains optional `outboxDepth` and `relayLagMs`.
+
+- bf1c2da: **Tenants can register a Studio principal.** `CreateTenantRequest` gains optional `studioCredentialHash` behind the new protocol feature `studio-principal`; the Runtime stores it as application principal `studio`, and idempotent create compares it too. `@nylorun/admin` exports `deriveStudioToken(adminKey, tenantId)` (HMAC-SHA256 over `nylorun/studio/v1`, NUL, Tenant id) and `createTenant` sends the hash of that key, so Studio can reach any Tenant's API with a key derived from the admin key. Clients require the new feature, so upgrade the Runtime with them.
+- 1d84b3e: **Native Windows is no longer supported; Windows developers use WSL2.** Nylorun runs on macOS and Linux. On native Windows, `nylorun` and `npm create @nylorun/agent` stop with WSL2 guidance. Install Node 24 and Docker (Docker Desktop's WSL integration) inside your WSL distribution and keep projects in its Linux filesystem. The Windows-only process handling (`taskkill`, `.cmd` shims, `npm.cmd`) is removed. `nylorun doctor` reports WSL as `Linux (WSL: <distribution>)`.
+
+### Patch Changes
+
+- ba1b239: **`nylorun` sets up the local stack; `@nylorun/cli` becomes the Runtime client `nylo` (breaking beta).**
+
+  - New unscoped package **`nylorun`**: `npx nylorun up` sets up the Docker stack on the first run and starts it after that, from any directory; `npx nylorun down` stops it (volumes are kept). `up` and `down` are aliases of `start` and `stop`. It also owns `status`, `logs`, `studio`, `reset` and `doctor`, pins the Runtime and Studio images (`package.json` `nylorun.runtime` / `nylorun.studio`), depends on `@nylorun/core` only, and never creates Tenants.
+  - **`@nylorun/cli`** is the Runtime client with the command **`nylo`**: `nylo tenant create [name]` (creates the Tenant; in a project, writes the Project link and seeds the model provider and sandbox from `.env`), `nylo tenant current|list|use|status|reset|delete`, `nylo configure`, `nylo env` (was `nylorun status --env`) and `nylo doctor sandbox`. It no longer provides the `nylorun` command or the stack commands.
+  - **Removed:** `nylorun dev` and `nylorun dev --ephemeral`. Link once with `nylo tenant create`, then run the project's own `npm run dev`. Moved commands exit 2 and name their replacement.
+  - **Starter:** the generated project depends on `@nylorun/agents` only (no Nylorun devDependency); `dev` is `tsx watch --env-file-if-exists=.env src/main.ts`. `npm create @nylorun/agent` installs the project and prints the next steps (`npx nylorun up`, `npx @nylorun/cli tenant create`, `npm run dev`) instead of starting development; `--no-open` is accepted and ignored.
+  - `connectAgents` names those steps when it finds no Runtime connection. Runtime repair hints, model errors and core's sandbox message name `nylo` and `nylorun up`.
+
+- 262bded: A Tenant's stream wiring no longer logs after it stops. A basin repair that failed after the Tenant closed could write to a Tenant log directory that was already removed and end the process with an unhandled rejection.
+- Pin core to the tested release.
+- Pin harness to the tested release.
+- Updated dependencies [bf1c2da]
+- Updated dependencies [ba1b239]
+- Updated dependencies [bf1c2da]
+- Updated dependencies [bf1c2da]
+- Updated dependencies [bf1c2da]
+- Updated dependencies
+  - @nylorun/core@0.6.0-beta
+  - @nylorun/harness@0.19.1-beta
+
 ## 0.10.0-beta
 
 ### Minor Changes

package/README.md

@@ -15,51 +15,53 @@ npm run build --workspace @nylorun/harness
 npm run build --workspace @nylorun/runtime
 ```
 
-## Install and the launcher
-
-Developers install the Runtime as a prerequisite; nothing downloads it for
-them ([runtime distribution](../docs/design/runtime-distribution.md)):
-
-```sh
-node --version                            # 24 or newer
-npm install --global @nylorun/runtime     # provides nylorun-runtime
-nylorun-runtime --version
-```
-
-The **launcher** (`nylorun-runtime`, source in `src/launcher/`, not listed in
-`exports`) starts, stops and restarts the Host on the Node it runs on, from
-this package. Clients (CLI, desktop apps) find it on PATH and run it as a
-process; prefer `nylorun runtime up`. A project may instead add this package
-as a devDependency to pin its Runtime version; do not make it an application
-production dependency. For tests and ephemeral embeds, use
-`startEphemeralRuntime()` from `@nylorun/runtime/core`. See
-[MIGRATION.md](../MIGRATION.md#runtime-clients-and-admin-api-breaking-beta) and
-[building a desktop client](../docs/building-a-desktop-client.md).
-
-Default address: loopback port `8787` (persisted in `host.json`). The Host root
-is `NYLORUN_HOME` or `~/.nylorun`. Tenants live under `tenants/<tenantId>/`.
+## Running the Runtime
+
+The Runtime runs as the `ghcr.io/nylorun/runtime` image, next to Postgres,
+Restate and S2: `nylorun start` runs that stack on a developer machine. The
+package is a library with no bin; its Host entry is `@nylorun/runtime/server`
+(`dist/host/main.js`), which requires `NYLORUN_DATABASE_URL`. For tests and
+ephemeral embeds, use `startEphemeralRuntime()` from `@nylorun/runtime/core`.
+See [MIGRATION.md](../MIGRATION.md).
+
+No Tenant state lives only in the Runtime process. Each Tenant's data is the
+Postgres schema `tenant_<id>` (the Session Store); Restate runs one advance of a session at a
+time and holds the Tenant's sweep timer (Durable Session Execution); every
+session's events are relayed from the schema's outbox to its own S2 stream,
+which history and SSE read (Durable Streams). An API node serves the Tenant API,
+Admin API and SSE, and a Worker runs advances; `--role api|worker|all` picks
+them, and the local stack runs one process with `all`.
+
+The container is configured by its environment, which the stack's Compose file
+sets: `NYLORUN_DATABASE_URL` (required), `NYLORUN_RESTATE_INGRESS_URL`,
+`NYLORUN_RESTATE_ADMIN_URL`, `NYLORUN_WORKER_URL` and
+`NYLORUN_RESTATE_IDENTITY_KEY` (Restate), `NYLORUN_S2_ENDPOINT` and
+`NYLORUN_S2_TOKEN`, and in container mode `NYLORUN_LISTEN_HOST`,
+`NYLORUN_LISTEN_PORT`, `NYLORUN_ALLOWED_HOSTS` and `NYLORUN_PUBLIC_URL`.
+
+The Host root is `NYLORUN_HOME` or `~/.nylorun` (bind-mounted at `/nylorun` in
+the stack). What stays on the Host is under `tenants/<tenantId>/`.
 
 ## Layout
 
 ```text
 <host root>/
-  host.json                 # format 1: hostId, bind, port, runtimeVersion, …
-  host-state.json           # pid, url — removed on shutdown
+  host.json                 # format 1: hostId, host, port, runtimeVersion, …
   host-credentials.json     # adminKey (0600)
-  runtime.log
-  tenants/<tenantId>/       # envelope, SQLite, KEK, logs, sandboxes, …
-  trash/                    # deleted Tenants
+  stack/                    # compose.yaml, .env (0600), Restate identity key
+  tenants/<tenantId>/       # vault-kek, plugin-data, logs, home, tmp, sandboxes
+  trash/                    # SQLite Tenants from before the Postgres switch
 ```
 
 ## HTTP surface
 
 | Route | Auth | Notes |
 | --- | --- | --- |
-| `GET /health` | none | `service: "nylorun-runtime"`, `hostId`, protocol `{min,max,features}` (includes `admin-status`), pid |
-| `GET /ready` | none | Listener up and Tenant discovery finished |
+| `GET /health` | none | `service: "nylorun-runtime"`, `hostId`, protocol `{min,max,features}`, pid |
+| `GET /ready` | none | Listener up, Tenants discovered, and Postgres, Restate and S2 answer (`checks`) |
 | `GET /v1/admin/status` | admin key | `AdminStatusSchema`; alias `GET /v1/admin/host` |
 | `/v1/admin/tenants*` | admin key | Create / list / get / delete Tenants |
-| `POST /v1/admin/host/shutdown` | admin key | Launcher-private; not in `@nylorun/admin` |
+| `POST /v1/admin/host/shutdown` | admin key | Host-private; not in `@nylorun/admin` |
 | `/v1/*` Tenant routes | application or executor | Require `Nylorun-Tenant` + `Nylorun-Protocol` |
 
 Every route checks `Host` first (`421 host_rejected`), rejects any `Origin`
@@ -72,7 +74,10 @@ authentication. Unknown, quarantined or rejected Tenant credentials → opaque
 
 Tests and ephemeral embeds use `startEphemeralRuntime()` from
 `@nylorun/runtime/core`: private Host on port 0, temporary Host root, one Tenant,
-returns `{ url, tenantId, applicationKey, adminKey, close() }`.
+returns `{ url, tenantId, applicationKey, adminKey, close() }`. It is not durable:
+its Tenants live in memory and are gone after `close()`. The smoke checks'
+temporary Tenants (`scripts/lib/temporary-tenant.mjs`) do not use it; they are
+created on the Docker stack.
 
 Register executors with `PUT /v1/executors` using the application principal
 (application-mode `connectAgents` does this with derived tokens). Model gateway
@@ -87,36 +92,33 @@ definition/session APIs; executor credentials authorize authenticated SSE
 connect, action discovery, claims, renewal and results. Session observers cannot
 claim actions.
 
-SQLite transactions persist session checkpoints, command receipts, individual
-effects/actions, waits and canonical history — **per Tenant**. One process owns
-each Tenant database (`.runtime-lock`). Keep the database and its WAL/journal
-together. Vault ciphertext needs that Tenant's own KEK.
+Postgres transactions persist session checkpoints, command receipts, individual
+effects/actions, waits and the event outbox — **per Tenant schema**; history is
+read from Durable Streams (S2). An advance owns its session through a lease with
+an epoch; a Worker that takes over after a crash marks in-flight effects
+`uncertain`. Vault ciphertext needs that Tenant's own KEK.
 
 Agents that declare `.use(sandbox())` get Runtime-executed sandbox tools. The
 Tenant owns each sandbox; backend names are prefixed `nylorun-<tenant-id>-`.
 
 ## Local Project workflow
 
-Install `@nylorun/cli` as a devDependency. `nylorun runtime up` starts the Host
-via the launcher; `nylorun dev` creates or uses a Project link and runs
-`src/main.ts` under `tsx watch`. Studio is `nylorun-studio` / `npm run studio`.
-
-```sh
-eval "$(npx nylorun runtime status --env)"
-# → NYLORUN_RUNTIME_URL, NYLORUN_SERVER_KEY, NYLORUN_TENANT
-```
+A project depends on `@nylorun/agents` only. `npx nylorun up` runs the local
+stack (Docker); `npx @nylorun/cli tenant create` (`nylo`) creates the
+project's Tenant and writes its Project link; the project's `npm run dev` runs
+`src/main.ts` under `tsx watch`; `npx nylorun studio` opens Studio.
 
 ## Troubleshooting
 
 | Symptom | What to do |
 | --- | --- |
-| Tenant `locked` | Another process holds `.runtime-lock`. `nylorun tenant status` prints `repair` |
-| `kek-missing` | Restore the Tenant KEK beside the database |
-| `corrupt` / `migration-failed` / `envelope-invalid` | Follow `nylorun tenant status` repair string |
-| `schema-too-new` / `host_schema_newer` | Upgrade the Host build (`nylorun runtime restart`) |
+| `kek-missing` | Restore `vault-kek` in the Tenant directory |
+| `corrupt` / `migration-failed` / `envelope-invalid` / `open-failed` / `open-timeout` | Follow the `nylo tenant status` repair string |
+| `schema-too-new` | Run a Runtime at least as new as the one that migrated the schema |
 | `426 protocol_unsupported` | Upgrade clients or Host to a compatible set |
-| `421 host_rejected` / `403 origin_rejected` | Call from main process / Node; loopback Host only |
-| Port in use | Explicit `--port` fails closed. First setup may pick a free loopback port |
-| Logs | `nylorun runtime logs [--follow]` |
+| `421 host_rejected` / `403 origin_rejected` | Call from a server process, not a browser; in a container, list the `Host` in `NYLORUN_ALLOWED_HOSTS` |
+| `503` for a Tenant | Postgres or Restate is unreachable; `GET /ready` names which |
+| Port in use | Change `NYLORUN_PORT` in `<Host root>/stack/.env` and run `nylorun up` |
+| Logs | `nylorun logs runtime` |
 
 Definitions have no `agent.run()`; applications use `@nylorun/agents`.

package/dist/adapters/execution/restate.d.ts

@@ -0,0 +1,130 @@
+import { type DurableExecution, type StuckInvocation, type Wake, type WorkerHandlers } from "../../execution/types.js";
+export interface RestateTimeouts {
+    /**
+     * How long an invocation may run without journal progress before Restate
+     * asks it to suspend. An advance makes no journal entries while it runs, so
+     * this must exceed the longest segment. Default one hour.
+     */
+    inactivityMs?: number;
+    /** How long after the inactivity timeout Restate aborts the attempt. Default one hour. */
+    abortMs?: number;
+}
+/**
+ * Retries of a handler that throws (an infrastructure error). After
+ * `maxAttempts` the invocation is paused, never killed, and shows up in
+ * `listStuckInvocations` until an operator resumes it.
+ */
+export interface RestateRetry {
+    /** Default 100. */
+    initialIntervalMs?: number;
+    /** Default 30000. */
+    maxIntervalMs?: number;
+    /** Default 2. */
+    exponentiationFactor?: number;
+    /** Attempts, including the first, before pausing. Default 70. */
+    maxAttempts?: number;
+}
+export interface RestateExecutionOptions {
+    /** Restate ingress, e.g. `http://restate:8080`. */
+    ingressUrl: string;
+    /** Restate admin API, e.g. `http://restate:9070`. Needed by `start`. */
+    adminUrl: string;
+    /** Where `start` serves the Worker endpoint (HTTP/2 cleartext). */
+    workerListen?: {
+        host: string;
+        port: number;
+    };
+    /** The endpoint URL Restate calls, registered by `start`. */
+    workerAdvertisedUrl?: string;
+    /**
+     * Prepended to every service name, so several Runtimes (or test runs) can
+     * share one Restate server without taking each other's invocations.
+     * Letters, digits and `_` only. Default "".
+     */
+    servicePrefix?: string;
+    /**
+     * Request identity public keys (`publickeyv1_...`). When given, the Worker
+     * endpoint accepts only requests signed by a Restate server holding the
+     * matching private key.
+     */
+    identityKeys?: string[];
+    timeouts?: RestateTimeouts;
+    retry?: RestateRetry;
+    /** Delay between sweep passes of one Tenant. Default 5000. */
+    sweepIntervalMs?: number;
+    /**
+     * How long wakes with a `dedupeKey` are remembered. Default and minimum
+     * 24 hours (§12.3).
+     */
+    dedupeRetentionMs?: number;
+    /**
+     * Overwrite a deployment already registered at `workerAdvertisedUrl`, so a
+     * restarted Worker with changed code takes effect. Default true, which suits
+     * a stable URL on a developer machine. Rolling upgrades should advertise a
+     * versioned URL instead (§14.6).
+     */
+    forceRegistration?: boolean;
+    /** How long `start` keeps retrying registration while Restate comes up. Default 60000. */
+    registrationTimeoutMs?: number;
+    /** Replaces the SDK's console logging. */
+    logger?: (level: string, message: string) => void;
+}
+/** A Restate invocation that needs an operator: paused, or retrying after failures. */
+export type { StuckInvocation };
+export declare function createRestateExecution(options: RestateExecutionOptions): RestateExecution;
+export declare class RestateExecution implements DurableExecution {
+    private readonly options;
+    private handlers?;
+    private stopping;
+    private server?;
+    private readonly sessions;
+    private readonly controllers;
+    private readonly inflight;
+    private readonly names;
+    private readonly ingressUrl;
+    private readonly adminUrl;
+    private readonly sweepIntervalMs;
+    constructor(options: RestateExecutionOptions);
+    /** Service names this execution registers, for diagnostics and tests. */
+    get serviceNames(): {
+        session: string;
+        tenant: string;
+        timer: string;
+    };
+    wake(tenantId: string, sessionId: string, wake: Wake): Promise<void>;
+    timer(tenantId: string, key: string, at: Date): Promise<void>;
+    armSweep(tenantId: string): Promise<void>;
+    disarmSweep(tenantId: string): Promise<void>;
+    /**
+     * Readiness: the admin API (`/health`), which `start` registers with, and the
+     * ingress (`/restate/health`), which every wake, timer and sweep goes
+     * through, both answer.
+     */
+    probe(signal: AbortSignal): Promise<void>;
+    /** This Runtime's paused or backing-off invocations for `tenantId` (Tenant status). */
+    stuckInvocations(tenantId: string): Promise<StuckInvocation[]>;
+    start(handlers: WorkerHandlers): Promise<void>;
+    stop(): Promise<void>;
+    private definitions;
+    /** Runs `advance` with a signal aborted by `stop` or by the end of the Restate attempt. */
+    private runAdvance;
+    /** Runs a Worker handler so that `stop` waits for it. Refuses once stopping; Restate retries. */
+    private runTracked;
+    private staleSweepMs;
+    private serviceOptions;
+    /** One-way send through the ingress. Retries while the ingress is unreachable or overloaded. */
+    private send;
+    /** Registers the Worker endpoint with the admin API, retrying until Restate is up. */
+    private register;
+}
+/**
+ * Lists invocations of this Runtime's services that need an operator: paused
+ * after exhausting retries, or backing off after a failure. Reads Restate's
+ * SQL introspection (`POST /query` on the admin API).
+ */
+export declare function listStuckInvocations(options: {
+    adminUrl: string;
+    servicePrefix?: string;
+    tenantId?: string;
+    limit?: number;
+}): Promise<StuckInvocation[]>;