@nylorun/runtime 0.22.0-beta → 0.23.0-beta

package/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.23.0-beta
+
+### Minor Changes
+
+- 72a4b52: The outbox no longer stalls the Tenant sweep, holds up commands or loses a pod sandbox's signals, and Workers register versioned Restate deployments.
+
+  - A wake that keeps failing (Restate refuses it, or an older Runtime does not know its reason) is retried with backoff and, after ten failures, parked: logged once at error and kept in the outbox for an operator. It no longer fails the sweep, so Restate never pauses the Tenant's sweep object over it. Migration 0018 adds `attempts`, `retry_at` and `parked_at` to `nylorun.wakes`.
+  - The sweep drains the whole outbox and pages through every orphaned session in each pass, within a 2 s budget, instead of 100 rows per pass.
+  - A command's wake is sent after commit without the request waiting for it; a failed send is logged at warn and sent again by the sweep's next pass. The sweep's grace is compared on the database's clock and is now 30 s.
+  - A pod sandbox's reconcile and idle timer go through the outbox (migration 0019, `nylorun.sandbox_signals`), so a crash after commit no longer loses them, and a host revocation goes over the control bus (`host.revoked`), so it reaches every process with the Tenant open. `Tx.afterCommit` is removed.
+  - Sends to Restate's ingress use `@restatedev/restate-sdk-clients` 1.17.2, each with an idempotency key.
+  - A Worker registers `<NYLORUN_WORKER_URL>/nylorun/<version>`, the version being the new `NYLORUN_WORKER_VERSION` or the Runtime's own, without forcing in a container, so a rolling upgrade keeps the old deployment for the invocations it started. A build whose Worker code changed under the same Runtime version sets its own `NYLORUN_WORKER_VERSION`.
+
 ## 0.22.0-beta
 
 ### Minor Changes

package/README.md

@@ -44,8 +44,9 @@ or deleted key stops authenticating at once. A database written by a Runtime tha
 database (`tenant_<id>` schemas), or by a pre-release build of one Tenant per database
 (`schema_version` tables), is refused: this release starts fresh on a new database. Restate runs one advance of a session at a time, delivers its wakes and holds
 its timers and the Tenant's sweep (Durable Session Execution), and journals no effect:
-checkpoints, effects, events and the wake outbox are the Session Store's, and a wake is
-written to the outbox in the transaction that causes it, so a commit never loses it; every session's events are relayed from the record to
+checkpoints, effects, events and the outbox are the Session Store's, and a wake or a pod
+sandbox's reconcile is written to the outbox in the transaction that causes it, so a commit
+never loses it ([src/CONTEXT.md](src/CONTEXT.md)); every session's events are relayed from the record to
 its own S2 stream, which history and SSE read (Durable Streams). One image runs every **service**,
 and `--service` picks what a process runs: `core` serves the Runtime API, the
 Management API and SSE and runs the stream relay, `loop` runs advances (the Worker), and
@@ -69,10 +70,23 @@ gateway ([DEPLOYMENT.md](../DEPLOYMENT.md#the-harness-agent-turns-mcp-servers-an
 
 The container is configured by its environment, which a local Tenant's Compose file
 sets: `NYLORUN_DATABASE_URL` (required), `NYLORUN_RESTATE_INGRESS_URL`,
-`NYLORUN_RESTATE_ADMIN_URL`, `NYLORUN_WORKER_URL` and
+`NYLORUN_RESTATE_ADMIN_URL`, `NYLORUN_WORKER_URL`, `NYLORUN_WORKER_VERSION` and
 `NYLORUN_RESTATE_IDENTITY_KEY` (Restate), `NYLORUN_S2_ENDPOINT` and
 `NYLORUN_S2_TOKEN`, and in container mode `NYLORUN_LISTEN_HOST`,
 `NYLORUN_LISTEN_PORT`, `NYLORUN_ALLOWED_HOSTS` and `NYLORUN_PUBLIC_URL`.
+A Worker registers its Restate deployment at a versioned URL,
+`<NYLORUN_WORKER_URL>/nylorun/<version>`, where the version is `NYLORUN_WORKER_VERSION`
+(letters, digits, `.`, `_`, `-`) or, by default, the Runtime's own. In a container it never
+replaces a deployment already registered there (`force: false`): during a rolling upgrade
+the new version's Workers register next to the old ones, Restate sends new invocations to
+the new deployment and lets the old one finish what it started, and every Worker of one
+version shares its deployment through one load-balanced `NYLORUN_WORKER_URL`. A build that
+changes the Worker's code without changing the Runtime's version (an image built from a
+branch) sets its own `NYLORUN_WORKER_VERSION`: without force Restate keeps the deployment it
+already has at a URL and does not discover the Worker again, so the changed code would run
+under the old deployment's registration, and a Worker whose services that deployment lacks
+refuses to start. Outside a container (a development
+Host) the registration replaces the deployment at that URL.
 `NYLORUN_MANAGEMENT_KEY_FILE` names the bootstrap secret (below).
 `NYLORUN_IDENTITY_FILE` names the identity
 file, a YAML list of the trusted issuers whose JWTs the Runtime API accepts

package/dist/adapters/execution/restate.d.ts

@@ -1,4 +1,10 @@
 import { type DurableExecution, type SandboxSignal, type StuckInvocation, type Wake, type WorkerHandlers } from "../../execution/types.js";
+/**
+ * How long one send through the ingress keeps retrying while the ingress is unreachable,
+ * overloaded, or answers 404 because a just-registered service has not reached it yet. The
+ * outbox's grace (`OUTBOX_GRACE_MS` in `tenant/sweep.ts`) is longer.
+ */
+export declare const SEND_RETRY_MS = 10000;
 export interface RestateTimeouts {
     /**
      * How long an invocation may run without journal progress before Restate
@@ -60,8 +66,8 @@ export interface RestateExecutionOptions {
     /**
      * Overwrite a deployment already registered at `workerAdvertisedUrl`, so a
      * restarted Worker with changed code takes effect. Default true, which suits
-     * a stable URL on a developer machine. Rolling upgrades should advertise a
-     * versioned URL instead (§14.6).
+     * a stable URL on a developer machine. A Host in a container advertises a
+     * versioned URL without it (`workerDeployment` in `infra/execution.ts`, §14.6).
      */
     forceRegistration?: boolean;
     /** How long `start` keeps retrying registration while Restate comes up. Default 60000. */
@@ -88,6 +94,7 @@ export declare class RestateExecution implements DurableExecution {
     private readonly inflight;
     private readonly names;
     private readonly ingressUrl;
+    private readonly ingress;
     private readonly adminUrl;
     private readonly sweepIntervalMs;
     constructor(options: RestateExecutionOptions);
@@ -123,7 +130,12 @@ export declare class RestateExecution implements DurableExecution {
     private runTracked;
     private staleSweepMs;
     private serviceOptions;
-    /** One-way send through the ingress. Retries while the ingress is unreachable or overloaded. */
+    /**
+     * One-way send through the ingress (`@restatedev/restate-sdk-clients`). Every send carries
+     * an idempotency key, so the client's retries (`SEND_RETRY_MS`), and a send repeated after
+     * its answer was lost, start one invocation: the request's own key where it has one (an
+     * outbox row's, a timer's time), or one made for this call.
+     */
     private send;
     /** Registers the Worker endpoint with the admin API, retrying until Restate is up. */
     private register;

package/dist/adapters/execution/restate.js

@@ -25,37 +25,28 @@
  *   What a run answers arms its timers and its retry.
  *
  * Wakes, sandbox reconciles, timers and sweep arming go through the ingress as one-way
- * sends, so an API node can call them without ever calling `start`. A wake a commit asked for
- * comes from the Session Store's wake outbox (`Tx.wake`) with an idempotency key, so a send
- * repeated after its answer was lost causes no second advance.
+ * sends (`@restatedev/restate-sdk-clients`), so an API node can call them without ever
+ * calling `start`. Every send carries an idempotency key, which the client's retries need: a
+ * wake's dedupe key (the outbox gives every committed wake one), a reconcile's outbox key, a
+ * timer's `<key>@<at>`, a sandbox timer's `<timer>@<at>`, or one made for the call. A send
+ * repeated after its answer was lost therefore starts no second invocation.
  *
- * ## What Restate owns, and what the Session Store owns
- *
- * Restate owns when a session runs: it delivers each wake at least once, deduped by its
- * key for `dedupeRetentionMs`; runs at most one invocation per key at a time; retries an
- * invocation whose attempt failed, so an advance whose Worker died runs again and takes the
- * session over (§11.4); and keeps every timer: the busy re-wake, `NylorunTimer`,
- * `NylorunSandbox` and the `NylorunTenant` sweep chain. The Session Store (Postgres) owns
- * what happened: the checkpoint, the effect journal (`harness-api/record.ts`), the events
- * (the Record), and the wake outbox that hands each committed wake to Restate. Restate
- * journals no effect: an effect's intent and outcome commit with the session's events in
- * one transaction, under the advance's lease, and remote harnesses and pod engines write
- * them too, over the Harness API.
- *
- * Two Session Store mechanisms overlap Restate on purpose:
- *
- * - **The lease and its epoch** (§10.6). Restate's one invocation per key holds for an
- *   attempt, not for the code it started: an attempt that ends (abort timeout, a dropped
- *   connection) is retried while the previous attempt's handler may still run on its Worker,
- *   and a remote harness or pod engine may still hold the run. The epoch fences their writes,
- *   and `busy` answers the overlap.
- * - **The advance deadline** (`tenant/worker.ts`). Restate's abort timeout does not stop a
- *   running handler, so the Worker bounds each advance itself, below `timeouts`.
+ * What Restate owns and what the Session Store owns, and where the two overlap on purpose,
+ * is said once, under "Durable Session Execution" in `src/CONTEXT.md`.
  */
+import { randomUUID } from "node:crypto";
 import { createServer } from "node:http2";
 import * as restate from "@restatedev/restate-sdk";
+import * as clients from "@restatedev/restate-sdk-clients";
+import { retry } from "../../retry.js";
 import { DEFAULT_STOP_GRACE_MS, WAKE_REASONS, parseSessionKey, sessionKey, settleWithin, } from "../../execution/types.js";
 const HOUR_MS = 60 * 60 * 1000;
+/**
+ * How long one send through the ingress keeps retrying while the ingress is unreachable,
+ * overloaded, or answers 404 because a just-registered service has not reached it yet. The
+ * outbox's grace (`OUTBOX_GRACE_MS` in `tenant/sweep.ts`) is longer.
+ */
+export const SEND_RETRY_MS = 10_000;
 export function createRestateExecution(options) {
     return new RestateExecution(options);
 }
@@ -69,6 +60,7 @@ export class RestateExecution {
     inflight = new Set();
     names;
     ingressUrl;
+    ingress;
     adminUrl;
     sweepIntervalMs;
     constructor(options) {
@@ -83,6 +75,18 @@ export class RestateExecution {
             sandbox: `${prefix}NylorunSandbox`,
         };
         this.ingressUrl = options.ingressUrl.replace(/\/+$/, "");
+        this.ingress = clients.connect({
+            url: this.ingressUrl,
+            retry: {
+                maxAttempts: false,
+                maxDuration: SEND_RETRY_MS,
+                initialInterval: 100,
+                maxInterval: 2000,
+                // A just-registered service can take a moment to reach the ingress.
+                shouldRetry: (failure) => clients.defaultShouldRetry(failure) ||
+                    (failure.kind === "response" && failure.status === 404),
+            },
+        });
         this.adminUrl = options.adminUrl.replace(/\/+$/, "");
         this.sweepIntervalMs = options.sweepIntervalMs ?? 5000;
     }
@@ -93,28 +97,27 @@ export class RestateExecution {
     async wake(tenantId, sessionId, wake) {
         if (!WAKE_REASONS.includes(wake.reason))
             throw new Error(`Unknown wake reason: ${String(wake.reason)}`);
-        await this.send(this.names.session, sessionKey(tenantId, sessionId), "advance", { reason: wake.reason }, wake.dedupeKey);
+        await this.send(this.names.session, sessionKey(tenantId, sessionId), "advance", { reason: wake.reason }, wake.dedupeKey ?? randomUUID());
     }
     async sandbox(tenantId, sandboxId, signal) {
         const key = sessionKey(tenantId, sandboxId);
         if (signal.kind === "reconcile")
-            await this.send(this.names.sandbox, key, "reconcile", {});
+            await this.send(this.names.sandbox, key, "reconcile", {}, signal.key ?? randomUUID());
         else
-            await this.send(this.names.sandbox, key, "arm", {
-                timer: signal.timer,
-                at: signal.at,
-            });
+            await this.send(this.names.sandbox, key, "arm", { timer: signal.timer, at: signal.at }, 
+            // Arming a timer for the same time again changes nothing.
+            `${signal.timer}@${signal.at}`);
     }
     async timer(tenantId, key, at) {
-        await this.send(this.names.timer, sessionKey(tenantId, key), "set", {
-            at: at.getTime(),
-        });
+        const ms = at.getTime();
+        await this.send(this.names.timer, sessionKey(tenantId, key), "set", { at: ms }, `${key}@${ms}`);
     }
     async armSweep(tenantId) {
-        await this.send(this.names.tenant, tenantKey(tenantId), "arm", {});
+        // A key of its own: a later arm (a Host opening the Tenant again) must still run.
+        await this.send(this.names.tenant, tenantKey(tenantId), "arm", {}, randomUUID());
     }
     async disarmSweep(tenantId) {
-        await this.send(this.names.tenant, tenantKey(tenantId), "disarm", {});
+        await this.send(this.names.tenant, tenantKey(tenantId), "disarm", {}, randomUUID());
     }
     /**
      * Readiness: the admin API (`/health`), which `start` registers with, and the
@@ -366,51 +369,62 @@ export class RestateExecution {
             },
         };
     }
-    /** One-way send through the ingress. Retries while the ingress is unreachable or overloaded. */
+    /**
+     * One-way send through the ingress (`@restatedev/restate-sdk-clients`). Every send carries
+     * an idempotency key, so the client's retries (`SEND_RETRY_MS`), and a send repeated after
+     * its answer was lost, start one invocation: the request's own key where it has one (an
+     * outbox row's, a timer's time), or one made for this call.
+     */
     async send(service, key, handler, body, idempotencyKey) {
-        const url = `${this.ingressUrl}/${service}/${encodeURIComponent(key)}/${handler}/send`;
-        const headers = { "content-type": "application/json" };
-        if (idempotencyKey !== undefined)
-            headers["idempotency-key"] = idempotencyKey;
-        await withRetry(`Restate send to ${service}/${handler}`, 10_000, async () => {
-            const response = await fetch(url, {
-                method: "POST",
-                headers,
-                body: JSON.stringify(body),
-                signal: AbortSignal.timeout(10_000),
+        try {
+            await this.ingress.send({
+                service,
+                handler,
+                key,
+                parameter: body,
+                opts: clients.SendOpts.from({ idempotencyKey, timeout: 10_000 }),
             });
-            if (response.ok) {
-                await response.body?.cancel();
-                return;
-            }
-            const text = await response.text().catch(() => "");
-            const error = new Error(`Restate ingress ${response.status} for ${service}/${handler}: ${text}`);
-            // A just-registered service can take a moment to reach the ingress.
-            throw response.status >= 500 || response.status === 404 || response.status === 429
-                ? error
-                : new PermanentError(error.message);
-        });
+        }
+        catch (error) {
+            const message = error instanceof clients.HttpCallError
+                ? `Restate ingress ${error.status}: ${error.responseText}`
+                : error instanceof Error
+                    ? error.message
+                    : String(error);
+            throw new Error(`Restate send to ${service}/${handler} failed: ${message}`, { cause: error });
+        }
     }
     /** Registers the Worker endpoint with the admin API, retrying until Restate is up. */
     async register(uri) {
         const force = this.options.forceRegistration ?? true;
-        await withRetry("Restate deployment registration", this.options.registrationTimeoutMs ?? 60_000, async () => {
+        const registered = await retry(async () => {
             const response = await fetch(`${this.adminUrl}/deployments`, {
                 method: "POST",
                 headers: { "content-type": "application/json" },
                 body: JSON.stringify({ uri, force }),
                 signal: AbortSignal.timeout(30_000),
             });
-            if (response.ok) {
-                await response.body?.cancel();
-                return;
-            }
+            if (response.ok)
+                return (await response.json().catch(() => ({})))
+                    .services;
             const text = await response.text().catch(() => "");
             const message = `Restate admin ${response.status} registering ${uri}: ${text}`;
             // Discovery failures (Restate cannot reach the endpoint yet) come back
             // as 4xx/5xx; keep trying until the deadline.
             throw new Error(message);
+        }, { minMs: 100, maxMs: 2000, deadlineMs: this.options.registrationTimeoutMs ?? 60_000 }).catch((error) => {
+            throw new Error(`Restate deployment registration failed: ${error.message}`, {
+                cause: error,
+            });
         });
+        // Without `force`, Restate answers a URL already registered with the deployment it has
+        // there, without discovering the endpoint again: one with other services is refused here.
+        const names = new Set((registered ?? []).map((service) => service.name));
+        const missing = Object.values(this.names).filter((name) => !names.has(name));
+        if (registered && missing.length > 0)
+            throw new Error(`Restate deployment registration failed: ${uri} is registered with other services ` +
+                `(missing ${missing.join(", ")}); register this Worker at a URL of its own ` +
+                "(NYLORUN_WORKER_VERSION)");
     }
 }
 /**
@@ -501,26 +515,6 @@ function tenantKey(tenantId) {
 function sqlString(value) {
     return `'${value.replaceAll("'", "''")}'`;
 }
-class PermanentError extends Error {
-}
-async function withRetry(what, timeoutMs, attempt) {
-    const deadline = Date.now() + timeoutMs;
-    let delay = 100;
-    for (;;) {
-        try {
-            await attempt();
-            return;
-        }
-        catch (error) {
-            if (error instanceof PermanentError || Date.now() + delay > deadline)
-                throw new Error(`${what} failed: ${error.message}`, {
-                    cause: error,
-                });
-            await new Promise((resolve) => setTimeout(resolve, delay));
-            delay = Math.min(delay * 2, 2000);
-        }
-    }
-}
 function sdkLogger(log) {
     return (meta, message, ...rest) => {
         const parts = [message, ...rest].map((part) => part instanceof Error ? part.stack ?? part.message : String(part));

package/dist/adapters/streams/s2.js

@@ -58,6 +58,7 @@
  */
 import { AppendInput, AppendRecord, RangeNotSatisfiableError, S2, S2Error, SeqNumMismatchError, } from "@s2-dev/streamstore";
 import { parseBasin, tenantBasinName, validateBasinPrefix } from "../../streams/basin.js";
+import { sleep } from "../../retry.js";
 import {} from "../../streams/types.js";
 /** Records per unary read while reading history. */
 const HISTORY_PAGE = 1000;
@@ -357,16 +358,3 @@ function escapeRegExp(text) {
 function anySignal(...signals) {
     return AbortSignal.any(signals.filter((s) => s !== undefined));
 }
-function sleep(ms, signal) {
-    return new Promise((resolve) => {
-        if (signal.aborted)
-            return resolve();
-        const done = () => {
-            clearTimeout(timer);
-            signal.removeEventListener("abort", done);
-            resolve();
-        };
-        const timer = setTimeout(done, ms);
-        signal.addEventListener("abort", done, { once: true });
-    });
-}

package/dist/core/flow-host.d.ts

@@ -1,8 +1,7 @@
 import type { EffectOutcome } from "@nylorun/core/contracts";
 import { type JsonValue, type WorkflowManifest } from "@nylorun/core/define";
 import type { HostEffect } from "@nylorun/harness/run";
-import type { EffectDoc, Tx, WakeDelivery } from "../store/types.js";
-import { type FlowLimits } from "./limits.js";
+import type { EffectDoc, Tx } from "../store/types.js";
 /** Deterministic agent session id: derive(workflowSessionId, path, …parts). */
 export declare function deriveSessionId(workflowSessionId: string, path: string, ...parts: string[]): string;
 /**
@@ -97,8 +96,6 @@ export type CascadeCancelPlan = {
     /** Linked agent sessions to cancel, deepest path first. */
     readonly agentSessionIds: string[];
 };
-/** Wakes a session after commit (`DurableExecution.wake` through the Tenant context). */
-type Schedule = WakeDelivery;
 /** Effect documents as the flow host writes them. */
 export type FlowEffect = EffectDoc & {
     agentSessionId?: string;
@@ -135,7 +132,6 @@ export declare function wakeLinkedWorkflow(input: {
     readonly failed?: boolean;
     readonly cancelled?: boolean;
     readonly error?: string;
-    readonly schedule: Schedule;
 }): Promise<void>;
 /** Pending workflow `agent` effects with a linked session, for the Tenant sweep. */
 export declare function pendingAgentEffects(t: Tx): Promise<FlowEffect[]>;
@@ -152,7 +148,6 @@ export declare function pendingAgentEffects(t: Tx): Promise<FlowEffect[]>;
 export declare function reconcilePendingAgentEffect(input: {
     readonly t: Tx;
     readonly effectId: string;
-    readonly schedule: Schedule;
 }): Promise<boolean>;
 /** Path depth for deepest-first cancel ordering. */
 export declare function pathDepth(path: string): number;
@@ -222,16 +217,3 @@ export declare function foreignInteractionConflict(input: {
     message: string;
     ownerSessionId: string;
 } | undefined>;
-/**
- * When concurrency slots free, re-enter the workflow so queued effects can start
- * (status stays `queued` until the effect resolver dispatches them). Returns
- * true when a wake was scheduled.
- */
-export declare function wakeForQueuedEffects(input: {
-    readonly t: Tx;
-    readonly workflowSessionId: string;
-    readonly turnId: string;
-    readonly limits: FlowLimits;
-    readonly schedule: Schedule;
-}): Promise<boolean>;
-export {};

package/dist/core/flow-host.js

@@ -4,8 +4,8 @@
  * Every function that touches state takes the caller's transaction `t: Tx` and
  * is async. Nothing here publishes, notifies or schedules directly (seam rule
  * 1): events go through `t.event(...)`, which the store publishes after
- * commit, and wakes go through `t.wake(id, wake, schedule)`, the wake outbox, which
- * calls `schedule` after commit. Functions
+ * commit, and wakes go through `t.wake(id, wake)`, the wake outbox, which the store
+ * delivers after commit. Functions
  * that rewrite a session-scoped document lock that session first with
  * `t.lockSession` (a no-op when the caller already holds it).
  *
@@ -24,9 +24,9 @@
  * - `linkedMessageKey(request): string`
  * - `linkedTurnOf(t, agentSessionId, request): Promise<string | undefined>`
  * - `linkedTurnEnd(t, effect, agent): Promise<LinkedTurnEnd | undefined>`
- * - `wakeLinkedWorkflow({ t, agentSessionId, turnId, output?, failed?, cancelled?, error?, schedule }): Promise<void>`
+ * - `wakeLinkedWorkflow({ t, agentSessionId, turnId, output?, failed?, cancelled?, error? }): Promise<void>`
  * - `pendingAgentEffects(t): Promise<FlowEffect[]>`
- * - `reconcilePendingAgentEffect({ t, effectId, schedule }): Promise<boolean>`
+ * - `reconcilePendingAgentEffect({ t, effectId }): Promise<boolean>`
  * - `planCancelCascade({ t, workflowSessionId, turnId }): Promise<CascadeCancelPlan>`
  * - `cancelSiblingWork({ t, workflowSessionId, turnId, siblingPaths?, cancelEffectIds? }): Promise<CancelSiblingResult>`
  * - `cancelQueuedEffects({ t, workflowSessionId, turnId }): Promise<string[]>`
@@ -34,11 +34,10 @@
  * - `flowInteractionOf(waits, workflowSessionId, interactionId): { id; kind } | undefined`
  * - `findInteractionOwner({ t, workflowSessionId, interactionId }): Promise<{ sessionId; path } | undefined>`
  * - `foreignInteractionConflict({ t, workflowSessionId, interactionId }): Promise<{ status: 409; message; ownerSessionId } | undefined>`
- * - `wakeForQueuedEffects({ t, workflowSessionId, turnId, limits, schedule }): Promise<boolean>`
  *
- * `schedule` (a `WakeDelivery`, the Tenant's `ctx.wake`) runs after commit, never inside `t`;
- * `wake` carries the reason (`linked`, `flow`) and, where the cause has one, a dedupe key. A
- * wake `schedule` does not deliver stays in the outbox for the Tenant sweep (`Tx.wake`).
+ * A wake carries the reason (`linked`, `flow`) and, where the cause has one, a dedupe key.
+ * The store delivers it after commit, never inside `t`; one it does not deliver stays in the
+ * outbox for the Tenant sweep (`Tx.wake`).
  *
  * An `agent` effect settles only from the linked turn it started. A linked agent session is
  * reused by every iteration of a Loop (its id derives from the workflow and the path), so its
@@ -51,7 +50,6 @@
  */
 import { createHash } from "node:crypto";
 import { isVerdict } from "@nylorun/core/define";
-import { mayDispatchMore } from "./limits.js";
 /** Deterministic agent session id: derive(workflowSessionId, path, …parts). */
 export function deriveSessionId(workflowSessionId, path, ...parts) {
     const digest = createHash("sha256")
@@ -140,10 +138,6 @@ export async function recordVerdict(t, request, value) {
         ...(verdict.data !== undefined ? { data: verdict.data } : {}),
     });
 }
-/** Writes the wake to the outbox; `schedule` delivers it after commit. */
-function requestWake(t, schedule, id, wake) {
-    return t.wake(id, wake, schedule);
-}
 /** Active agent turns + tool nodes in flight for one workflow turn. */
 export async function countActiveFlowWork(t, workflowSessionId, turnId) {
     const effects = await t.effectsForTurn(workflowSessionId, turnId);
@@ -245,7 +239,7 @@ export async function wakeLinkedWorkflow(input) {
     await settleAgentEffect({ t, effect, outcome });
     workflow.status = "runnable";
     await t.put("sessions", workflow.id, workflow);
-    await requestWake(t, input.schedule, workflow.id, {
+    await t.wake(workflow.id, {
         reason: "linked",
         dedupeKey: `linked:${link.turnId}:${link.effectId}`,
     });
@@ -288,7 +282,6 @@ export async function reconcilePendingAgentEffect(input) {
             : end.status === "failed"
                 ? { failed: true, error: end.error }
                 : { cancelled: true, error: end.error }),
-        schedule: input.schedule,
     });
     return true;
 }
@@ -457,25 +450,3 @@ export async function foreignInteractionConflict(input) {
         ownerSessionId: owner.sessionId,
     };
 }
-/**
- * When concurrency slots free, re-enter the workflow so queued effects can start
- * (status stays `queued` until the effect resolver dispatches them). Returns
- * true when a wake was scheduled.
- */
-export async function wakeForQueuedEffects(input) {
-    const { t } = input;
-    if (!mayDispatchMore(await countActiveFlowWork(t, input.workflowSessionId, input.turnId), input.limits))
-        return false;
-    const queued = await t.effectsForTurn(input.workflowSessionId, input.turnId, ["queued"]);
-    if (queued.length === 0)
-        return false;
-    const workflow = await t.lockSession(input.workflowSessionId);
-    if (workflow && workflow.status === "waiting") {
-        workflow.status = "runnable";
-        await t.put("sessions", workflow.id, workflow);
-    }
-    await requestWake(t, input.schedule, input.workflowSessionId, {
-        reason: "flow",
-    });
-    return true;
-}

package/dist/execution/types.d.ts

@@ -1,12 +1,10 @@
 /**
  * The Durable Session Execution seam (architecture §12.3).
  *
- * Durable Session Execution decides when to look at a session again. It never
- * holds session state: every outcome lives in the Session Store, and `advance`
- * is safe to repeat (§10.1). It owns the schedule (wake delivery, one advance per
- * key, retries, timers, the sweep chain); the Session Store owns the record (the
- * checkpoint, the effect journal, the events) and the wake outbox that bridges the
- * two. Restate is the supported implementation
+ * Durable Session Execution decides when to look at a session again; `advance` is
+ * safe to repeat (§10.1). What it owns and what the Session Store owns is said once,
+ * under "Durable Session Execution" in `src/CONTEXT.md`. Restate is the supported
+ * implementation
  * (`adapters/execution/restate.ts`); `execution/memory.ts` is the in-process
  * implementation used by unit tests, `startEphemeralRuntime` and a Host
  * started without Restate endpoints.
@@ -19,7 +17,7 @@
  * - **At least once after a wake.** When `wake` resolves, an `advance` for the
  *   key will start after that point (possibly merged with other wakes). Wakes
  *   are sent after the Session Store transaction that caused them commits, never
- *   inside it: the transaction writes the wake to its outbox (`Tx.wake`), and the
+ *   inside it: the transaction writes the wake to the outbox (`Tx.wake`), and the
  *   wake is sent after commit, or by the Tenant sweep when that send failed or a
  *   crash cut it off. A wake may therefore be sent more than once, always with the
  *   same `dedupeKey`.
@@ -75,9 +73,13 @@ export type AdvanceResult = {
 };
 /** A pod sandbox's timers. */
 export type SandboxTimer = "idle" | "ttl";
-/** What `DurableExecution.sandbox` asks for: a reconcile, or a timer. */
+/**
+ * What `DurableExecution.sandbox` asks for: a reconcile, or a timer. A reconcile's `key` merges
+ * repeated sends of one request (the outbox's `signal:<row id>`); without one, every send runs.
+ */
 export type SandboxSignal = {
     kind: "reconcile";
+    key?: string;
 } | {
     kind: "arm";
     timer: SandboxTimer;

package/dist/execution/types.js

@@ -1,12 +1,10 @@
 /**
  * The Durable Session Execution seam (architecture §12.3).
  *
- * Durable Session Execution decides when to look at a session again. It never
- * holds session state: every outcome lives in the Session Store, and `advance`
- * is safe to repeat (§10.1). It owns the schedule (wake delivery, one advance per
- * key, retries, timers, the sweep chain); the Session Store owns the record (the
- * checkpoint, the effect journal, the events) and the wake outbox that bridges the
- * two. Restate is the supported implementation
+ * Durable Session Execution decides when to look at a session again; `advance` is
+ * safe to repeat (§10.1). What it owns and what the Session Store owns is said once,
+ * under "Durable Session Execution" in `src/CONTEXT.md`. Restate is the supported
+ * implementation
  * (`adapters/execution/restate.ts`); `execution/memory.ts` is the in-process
  * implementation used by unit tests, `startEphemeralRuntime` and a Host
  * started without Restate endpoints.
@@ -19,7 +17,7 @@
  * - **At least once after a wake.** When `wake` resolves, an `advance` for the
  *   key will start after that point (possibly merged with other wakes). Wakes
  *   are sent after the Session Store transaction that caused them commits, never
- *   inside it: the transaction writes the wake to its outbox (`Tx.wake`), and the
+ *   inside it: the transaction writes the wake to the outbox (`Tx.wake`), and the
  *   wake is sent after commit, or by the Tenant sweep when that send failed or a
  *   crash cut it off. A wake may therefore be sent more than once, always with the
  *   same `dedupeKey`.

package/dist/host/execution.d.ts

@@ -19,12 +19,14 @@
  * services, so the open would fail. A core-only process relies on a Worker having registered
  * them first.
  *
- * **One Worker URL.** `start` registers the Worker endpoint (`NYLORUN_WORKER_URL`) with
- * Restate once. Restate sends new invocations to the latest registered deployment, so
- * several Workers that each register their own URL do not share load: the last one to start
- * takes every new invocation. The local stack runs a single `runtime` container with
- * `--service core,loop`. A deployment with several Worker processes (Helm, Cloud) must advertise one
- * load-balanced `NYLORUN_WORKER_URL` shared by all of them.
+ * **One Worker URL per version.** `start` registers the Worker endpoint with Restate once, as
+ * `<NYLORUN_WORKER_URL>/nylorun/<version>` (`workerDeployment` in `infra/execution.ts`).
+ * Restate sends new invocations to the latest registered deployment, so several Workers that
+ * each register their own URL do not share load: the last one to start takes every new
+ * invocation. The local stack runs a single `runtime` container with `--service core,loop`. A
+ * deployment with several Worker processes (Helm, Cloud) must advertise one load-balanced
+ * `NYLORUN_WORKER_URL` shared by all of them; a rolling upgrade registers the new version's
+ * path next to the old one, which keeps the invocations it started.
  */
 import type { DurableExecution } from "../execution/types.js";
 import type { Logger } from "../tenant/types.js";

package/dist/host/main.js

@@ -56,6 +56,7 @@ import { createRunGrants } from "../tenant/run-grants.js";
 import { runHarness } from "../harness/main.js";
 import { startHarnessListener } from "../harness-api/ws-server.js";
 import { httpSandboxesClient } from "../sandbox/pods/client.js";
+import { retry } from "../retry.js";
 const nodeRequire = createRequire(import.meta.url);
 function coreVersion() {
     try {
@@ -121,22 +122,15 @@ function objectStore(config) {
  * uploads and downloads fail meanwhile) and does not hold up the Host.
  */
 async function ensureBucket(store, bucket, logger) {
-    for (let attempt = 1;; attempt += 1) {
-        try {
-            await store.ensureBucket();
-            logger.info("object_store_ready", { bucket });
-            return;
-        }
-        catch (error) {
-            if (attempt >= 10) {
-                logger.error("object_store_unavailable", {
-                    bucket,
-                    message: error instanceof Error ? error.message : String(error),
-                });
-                return;
-            }
-            await new Promise((resolve) => setTimeout(resolve, Math.min(1000 * attempt, 5000)));
-        }
+    try {
+        await retry(() => store.ensureBucket(), { minMs: 1000, maxMs: 5000, attempts: 10 });
+        logger.info("object_store_ready", { bucket });
+    }
+    catch (error) {
+        logger.error("object_store_unavailable", {
+            bucket,
+            message: error instanceof Error ? error.message : String(error),
+        });
     }
 }
 /**