@nylorun/runtime 0.43.0-beta → 0.44.0-beta

package/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## 0.44.0-beta
+
+### Minor Changes
+
+- 1d3a5c5: Queued messages no longer wait for good (Host feature `queue-recovery`). When a turn ends and its
+  Agent refuses to start the next one for the session's queued messages (its disk being copied, its
+  pod lost, the Agent deleted), the messages wait with the session's `queueStalled`, and the Tenant
+  sweep tries the start again (Host tuning `sweep.queueRetryMs`, 30 seconds), as does the end of the
+  Agent's disk copy. A message that has waited longer than the new Runtime limit `queuedStartMs`
+  (15 minutes) is removed with the new event `queue.failed` (`queue.start_expired`, with the
+  refusal's code as `reason`); a refusal no retry changes (a turn request the version no longer
+  resolves, another harness) fails the messages at once (`queue.start_refused`), and the messages
+  behind them are tried next. Before, nothing started such a queue until another message arrived.
+
+  On the A2A endpoint (profile v2), a queued task that fails is `TASK_STATE_FAILED` with a status
+  message saying why, in `GetTask`, its pushes and a blocking `SendMessage`. `CancelTask` on a queued
+  task now takes it out of the queue, whether a task runs in its context or not, and answers
+  `TASK_STATE_CANCELED` (the new event `queue.cancelled`), pushed; before, it was
+  `TaskNotCancelableError`.
+
+### Patch Changes
+
+- 63e0577: Every running pod Agent now has an idle timer, whatever woke it. A pod woken by a disk copy or by commands through the session's sandbox route, and one whose idle timer fired during a turn, no longer runs past its idle limit; the Tenant sweep arms the timer of a pod already running past it.
+- b23a75c: A pod's engine checks its NetworkPolicy quickly on a wake. When the pod's disk already holds the engine's volume id, one 150 ms probe of the API server (`harness.wakeNetworkProbeMs`, `harness.wakeNetworkConfirmations`) replaces three of a second, so a woken computer joins about 2.9 s sooner. A first start keeps the full check, and a wake whose probe connects goes back to it.
+- Pin core to the tested release.
+- Pin harness to the tested release.
+- Updated dependencies [1d3a5c5]
+- Updated dependencies
+  - @nylorun/core@0.38.0-beta
+  - @nylorun/harness@0.34.1-beta
+
 ## 0.43.0-beta
 
 ### Minor Changes

package/dist/api/a2a/endpoint.js

@@ -72,7 +72,7 @@ export function a2aRoutes(api) {
         path: "/v1/a2a/agents/{agentId}",
         tags: TAGS,
         summary: "Call an agent over A2A",
-        description: "A2A 1.0 JSON-RPC, for a subject, under the Nylorun profile v2 (Host feature `a2a-profile-v2`). `SendMessage` blocks up to the Runtime limit `a2aWaitMs` (5 minutes by default; then the task is answered still `WORKING`) unless `returnImmediately`; `GetTask` (with `historyLength`), `CancelTask` and `ListTasks`; and the four push notification config methods, plus `configuration.taskPushNotificationConfig` in `SendMessage`: each update is POSTed as one `StreamResponse`, signed with Standard Webhooks (`webhook-id`, `webhook-timestamp`, `webhook-signature` with the Runtime's `a2a.push.secret`) and `X-A2A-Notification-Token`, at least once, retried for up to 24 hours, to hosts the Runtime allows (`a2a.push.allowHosts` for private ones). A message to a busy context is queued and answered with the task that will take it (`TASK_STATE_SUBMITTED`); `message.metadata.delivery: \"reject\"` asks for `UnsupportedOperationError` (reason `CONTEXT_BUSY`) instead. `message.metadata.removeTools` (Host feature `remove-tools`) names the tools a new message's turn may not use, as a session message's `removeTools`. A message is text, one data part, or text with `raw` files (stored as artifacts of the context's session) and `url` files (fetched by the Runtime), each at most the Runtime limit `inlineFileBytes` (20 MiB by default). An approval, question or connection the task waits for (`AUTH_REQUIRED` or `INPUT_REQUIRED`) is shown under `interactions/v1` and answered on the same task with `{ interactionId, approved }` or `{ interactionId, value }` under that URI in the message's `metadata`; text never approves. Artifacts carry the turn's output and the files it wrote, as `url` parts at `/v1/a2a/files/…` with `files/v1`; the task's metadata carries `progress/v1`, `telemetry/v1` and `effects/v1`. Extension data is answered only for the extensions activated with `A2A-Extensions`. A context is one session per subject, agent and `contextId`; a task is one turn. JSON-RPC errors are answered with 200; limits, scopes and unavailable streams stay HTTP errors. Not served: streaming, `GetExtendedAgentCard`.",
+        description: "A2A 1.0 JSON-RPC, for a subject, under the Nylorun profile v2 (Host feature `a2a-profile-v2`). `SendMessage` blocks up to the Runtime limit `a2aWaitMs` (5 minutes by default; then the task is answered still `WORKING`) unless `returnImmediately`; `GetTask` (with `historyLength`), `CancelTask` and `ListTasks`; and the four push notification config methods, plus `configuration.taskPushNotificationConfig` in `SendMessage`: each update is POSTed as one `StreamResponse`, signed with Standard Webhooks (`webhook-id`, `webhook-timestamp`, `webhook-signature` with the Runtime's `a2a.push.secret`) and `X-A2A-Notification-Token`, at least once, retried for up to 24 hours, to hosts the Runtime allows (`a2a.push.allowHosts` for private ones). A message to a busy context is queued and answered with the task that will take it (`TASK_STATE_SUBMITTED`); `message.metadata.delivery: \"reject\"` asks for `UnsupportedOperationError` (reason `CONTEXT_BUSY`) instead. A queued task never waits for good (Host feature `queue-recovery`): when its Agent refuses its turn's start (a disk copy, a lost pod), the start is tried again until the Runtime limit `queuedStartMs` (15 minutes by default), and then, or at once for a refusal no retry changes, the task is `TASK_STATE_FAILED` with a status message saying why; `CancelTask` on a queued task takes it out of the queue, a task running in the context or not, and answers `TASK_STATE_CANCELED`. Both are pushed. `message.metadata.removeTools` (Host feature `remove-tools`) names the tools a new message's turn may not use, as a session message's `removeTools`. A message is text, one data part, or text with `raw` files (stored as artifacts of the context's session) and `url` files (fetched by the Runtime), each at most the Runtime limit `inlineFileBytes` (20 MiB by default). An approval, question or connection the task waits for (`AUTH_REQUIRED` or `INPUT_REQUIRED`) is shown under `interactions/v1` and answered on the same task with `{ interactionId, approved }` or `{ interactionId, value }` under that URI in the message's `metadata`; text never approves. Artifacts carry the turn's output and the files it wrote, as `url` parts at `/v1/a2a/files/…` with `files/v1`; the task's metadata carries `progress/v1`, `telemetry/v1` and `effects/v1`. Extension data is answered only for the extensions activated with `A2A-Extensions`. A context is one session per subject, agent and `contextId`; a task is one turn. JSON-RPC errors are answered with 200; limits, scopes and unavailable streams stay HTTP errors. Not served: streaming, `GetExtendedAgentCard`.",
         externalDocs: A2A_SPECIFICATION,
         request: {
             params: agentId,

package/dist/api/a2a/push.js

@@ -5,7 +5,7 @@ import { traceIdOf } from "../../traces/ids.js";
 import { hostTuning } from "../../tuning.js";
 import { ALL_EXTENSIONS, keepExtensions } from "./extensions.js";
 import { a2aFail, } from "./protocol.js";
-import { ENDED, callerTraceId, updatesOf } from "./tasks.js";
+import { ENDED, callerTraceId, endsQueued, updatesOf } from "./tasks.js";
 /** The most push configs one task keeps. A constant of the profile, not a setting. */
 export const MAX_PUSH_CONFIGS = 10;
 /** Events one pump reads at a time. */
@@ -172,6 +172,19 @@ export async function pumpSession(ctx, sessionId) {
                     // A queued message's task follows the turn that takes it.
                     if (event.type === "command.message" && event.turnId && payload.queuedMessageId === turnId)
                         turnId = event.turnId;
+                    // Or ends without one: it failed or was cancelled while queued (`queue-recovery`).
+                    if (endsQueued(event, turnId)) {
+                        const view = { contextId: config.contextId, taskId: config.taskId, baseUrl: config.baseUrl };
+                        for (const update of updatesOf(event, view))
+                            await t.enqueuePush({
+                                sessionId,
+                                taskId: config.taskId,
+                                configId: config.configId,
+                                body: JSON.stringify(keepExtensions(update, new Set(config.extensions))),
+                            });
+                        done = true;
+                        break;
+                    }
                     if (event.turnId !== turnId)
                         continue;
                     if (event.type === "command.message" && traceId === null)

package/dist/api/a2a/routes.js

@@ -18,7 +18,10 @@
  * - A message to a busy context is **queued** (`delivery: "queue"`, protocol 17) and answered
  *   with the task that will take it: `SUBMITTED`, its id the queued message's own until a turn
  *   takes it, then that turn's state. `metadata.delivery: "reject"` keeps v1's
- *   `UnsupportedOperationError` (reason `CONTEXT_BUSY`).
+ *   `UnsupportedOperationError` (reason `CONTEXT_BUSY`). A queued task never waits for good (Host
+ *   feature `queue-recovery`): one whose turn the Agent refused to start for longer than the
+ *   Runtime limit `queuedStartMs`, or for good, is `FAILED`; `CancelTask` takes one out of the
+ *   queue as `CANCELED`. Both are pushed.
  * - An open interaction shows on the status message under `interactions/v1`; the caller answers
  *   on the same task with `{ interactionId, approved }` (the Runtime's `approve`, recorded with
  *   the caller as `approvedBy`) or `{ interactionId, value }` (`respond`). Text never approves.
@@ -39,11 +42,11 @@ import { keepExtensions } from "./extensions.js";
 import { indexedTurnEvents, listTasks, parseListParams } from "./list.js";
 import { A2aError, a2aFail, checkVersion, jsonRpcError, jsonRpcResult, parseCreatePushParams, parseEnvelope, parseListPushParams, parsePushConfigRef, parseSendParams, parseTaskParams, } from "./protocol.js";
 import { configView, kickPushes, pushEnabled, pushPolicy, savePushConfig, } from "./push.js";
-import { ENDED, SETTLED_EVENTS, buildTask, callerTraceId, contextSessionId, newContextId, parseTaskId, pendingInteraction, pendingInteractions, submittedTask, taskIdOf, } from "./tasks.js";
+import { ENDED, SETTLED_EVENTS, buildTask, callerTraceId, contextSessionId, droppedTask, endsQueued, newContextId, parseTaskId, pendingInteraction, pendingInteractions, submittedTask, taskIdOf, } from "./tasks.js";
 import { currentModelDescriptor } from "../../model/descriptor.js";
 import { decodeCursor } from "../../record/cursor.js";
 import { accessOf } from "../../tenant/auth.js";
-import { command } from "../../tenant/commands.js";
+import { cancelQueuedMessage, command } from "../../tenant/commands.js";
 import { loadSession, reachesAgent, } from "../../tenant/context.js";
 import { agentGrantsOf, liveEnvironment } from "../../tenant/agents.js";
 import { HttpError, fail } from "../../tenant/http.js";
@@ -213,7 +216,9 @@ async function indexedTask(caller, row, session, options = {}) {
  * The caller's task `taskId`, found through the turn index. Its last part is a turn, or a queued
  * message's id: while the message waits the task is `SUBMITTED`, and once a turn took it (the
  * index keeps the queued messages each turn took) the task is that turn's, under the id the
- * caller knows. Only a message still in the queue is looked for in the session's history.
+ * caller knows. A queued message that failed or was cancelled before a turn took it (Host feature
+ * `queue-recovery`) is `FAILED` or `CANCELED`, from the event that ended it. Only a message still
+ * in the queue, or no longer in it and taken by no turn, is looked for in the session's history.
  */
 async function ownedTask(caller, taskId, historyLength) {
     const ids = parseTaskId(taskId) ?? taskNotFound(taskId);
@@ -229,18 +234,16 @@ async function ownedTask(caller, taskId, historyLength) {
         const taken = await t.turnOfQueuedMessage(sessionId, ids.turnId);
         return taken ? { turn: taken } : {};
     });
-    if (found.queued) {
-        const queued = (await sessionEvents(caller.ctx, sessionId)).find((event) => event.type === "message.queued" && payloadOf(event).queuedMessageId === ids.turnId);
-        if (queued)
-            return {
-                task: submittedTask(ids.contextId, taskId, queued),
-                session,
-                sessionId,
-                contextId: ids.contextId,
-                turnId: ids.turnId,
-                events: [],
-                queued,
-            };
+    if (found.turn === undefined) {
+        // Queued, or a queued message that failed or was cancelled: its history says which.
+        const events = await sessionEvents(caller.ctx, sessionId);
+        const queued = events.find((event) => event.type === "message.queued" && payloadOf(event).queuedMessageId === ids.turnId);
+        const dropped = queued ? events.find((event) => endsQueued(event, ids.turnId)) : undefined;
+        const owned = { session, sessionId, contextId: ids.contextId, turnId: ids.turnId, events: [] };
+        if (dropped)
+            return { ...owned, task: droppedTask(ids.contextId, taskId, dropped), dropped };
+        if (queued && found.queued)
+            return { ...owned, task: submittedTask(ids.contextId, taskId, queued), queued };
     }
     const row = found.turn ?? taskNotFound(taskId);
     const built = (await indexedTask(caller, { ...row, a2aContextId: ids.contextId }, session, {
@@ -503,13 +506,21 @@ function replyTarget(owned) {
         return a2aFail("unsupportedOperation", "The task waits for an approval, which text never gives: answer it under the interactions/v1 extension ({ interactionId, approved }), or cancel the task", { taskId, interactionId: pending.id });
     return pending.id;
 }
-async function cancelTask(caller, taskId) {
+/**
+ * `CancelTask`. A queued task (`SUBMITTED`) leaves its context's queue and is `CANCELED`
+ * (`queue.cancelled`, pushed), whether a task runs in the context or not (Host feature
+ * `queue-recovery`); one a turn took meanwhile is cancelled as that turn. A running or waiting
+ * task's turn is cancelled. An ended task is not cancelable, unless it is `CANCELED` already.
+ */
+async function cancelTask(caller, taskId, again = true) {
     const owned = await ownedTask(caller, taskId);
-    if (owned.queued)
-        return a2aFail("taskNotCancelable", "The task is queued and starts when the context's running task ends; cancel it then", {
-            taskId,
-            state: owned.task.status.state,
-        });
+    if (owned.queued) {
+        const cancelled = await cancelQueuedMessage(caller.ctx, owned.sessionId, owned.turnId, caller.scope, "Canceled by the A2A client");
+        // Gone from the queue meanwhile: a turn took it, or it failed. Answer as the task is now.
+        if (!cancelled && again)
+            return cancelTask(caller, taskId, false);
+        return (await ownedTask(caller, taskId)).task;
+    }
     if (owned.session.activeTurnId !== owned.turnId) {
         if (owned.task.status.state === "TASK_STATE_CANCELED")
             return owned.task;
@@ -566,8 +577,8 @@ async function createPushConfig(caller, params) {
     requirePush(caller);
     const { taskId, config } = parseCreatePushParams(params);
     const owned = await ownedTask(caller, taskId);
-    const last = owned.queued ?? owned.events.at(-1);
-    const ended = owned.events.some((event) => ENDED.has(event.type));
+    const last = owned.dropped ?? owned.queued ?? owned.events.at(-1);
+    const ended = owned.dropped !== undefined || owned.events.some((event) => ENDED.has(event.type));
     const configId = config.id ?? createHash("sha256").update(`${config.url}\u0000${config.token ?? ""}`).digest("hex").slice(0, 16);
     return registerPush(caller, { ...config, id: configId }, {
         sessionId: owned.sessionId,
@@ -575,7 +586,7 @@ async function createPushConfig(caller, params) {
         taskId: owned.task.id,
         turnId: owned.turnId,
         nextSeq: last ? last.seq + 1 : 0,
-        ...(owned.queued ? {} : { traceId: await turnTraceId(caller.ctx, owned.sessionId, owned.turnId, owned.events) }),
+        ...(owned.queued || owned.dropped ? {} : { traceId: await turnTraceId(caller.ctx, owned.sessionId, owned.turnId, owned.events) }),
         done: ended,
     });
 }
@@ -615,7 +626,8 @@ async function deletePushConfig(caller, params) {
  * Waits until the task's turn ends, pauses or has an uncertain effect, the caller hangs up, or
  * the Tenant's Runtime limit `a2aWaitMs` passes: after it, the task is returned as it is
  * (`WORKING`, or `SUBMITTED` while queued) and keeps running, and the caller polls `GetTask` or
- * waits for pushes. A queued message's task is followed to the turn that takes it. Follows the
+ * waits for pushes. A queued message's task is followed to the turn that takes it, or to its
+ * `queue.failed` or `queue.cancelled`. Follows the
  * session's shared stream from the command's cursor, so the events the task is built from are
  * there when it returns. A feed that ends (reset, deleted, closing) ends the wait: the task says
  * what the session says then.
@@ -639,6 +651,8 @@ async function settled(ctx, sessionId, accepted, signal) {
                 event.turnId &&
                 payloadOf(event).queuedMessageId === accepted.queuedMessageId)
                 turnId = event.turnId;
+            if (turnId === undefined && endsQueued(event, accepted.queuedMessageId))
+                return;
             if (turnId !== undefined && event.turnId === turnId && SETTLED_EVENTS.has(event.type))
                 return;
         }

package/dist/api/a2a/tasks.d.ts

@@ -48,6 +48,14 @@ export declare function interruptedState(interaction: PendingInteraction): TaskS
 /** The status message that shows an open interaction, with its `interactions/v1` data. */
 export declare function interactionMessage(interaction: PendingInteraction, contextId: string, taskId: string): Message;
 export declare const ENDED: ReadonlySet<string>;
+/**
+ * Events that end a queued message's task without a turn (Host feature `queue-recovery`): its
+ * start was refused past the Runtime limit `queuedStartMs`, or for good (`queue.failed`), or a
+ * client cancelled it (`queue.cancelled`). Each names the queued messages it ends.
+ */
+export declare const QUEUE_ENDED: ReadonlySet<string>;
+/** Whether `event` ends queued message `queuedMessageId`'s task (`QUEUE_ENDED`). */
+export declare function endsQueued(event: LiveEvent, queuedMessageId: string): boolean;
 /** Events after which a blocking `SendMessage` returns: the turn ended or is interrupted. */
 export declare const SETTLED_EVENTS: ReadonlySet<string>;
 /** The turn's conversation as A2A messages: what the caller sent and what the agent said. */
@@ -116,3 +124,9 @@ export declare function buildTask(input: BuildTaskInput): Task | undefined;
  * event, until a turn takes it.
  */
 export declare function submittedTask(contextId: string, taskId: string, queued: LiveEvent): Task;
+/**
+ * A queued message's task that ended without a turn (Host feature `queue-recovery`), from the
+ * `queue.failed` or `queue.cancelled` event that names it: `FAILED`, saying why it never started,
+ * or `CANCELED`.
+ */
+export declare function droppedTask(contextId: string, taskId: string, ended: LiveEvent): Task;

package/dist/api/a2a/tasks.js

@@ -112,6 +112,19 @@ export function interactionMessage(interaction, contextId, taskId) {
     };
 }
 export const ENDED = new Set(["turn.completed", "turn.failed", "turn.cancelled"]);
+/**
+ * Events that end a queued message's task without a turn (Host feature `queue-recovery`): its
+ * start was refused past the Runtime limit `queuedStartMs`, or for good (`queue.failed`), or a
+ * client cancelled it (`queue.cancelled`). Each names the queued messages it ends.
+ */
+export const QUEUE_ENDED = new Set(["queue.failed", "queue.cancelled"]);
+/** Whether `event` ends queued message `queuedMessageId`'s task (`QUEUE_ENDED`). */
+export function endsQueued(event, queuedMessageId) {
+    if (!QUEUE_ENDED.has(event.type))
+        return false;
+    const ids = event.payload?.queuedMessageIds;
+    return Array.isArray(ids) && ids.includes(queuedMessageId);
+}
 /** Events after which a blocking `SendMessage` returns: the turn ended or is interrupted. */
 export const SETTLED_EVENTS = new Set([
     ...ENDED,
@@ -170,6 +183,11 @@ export function historyOf(events, contextId, taskId) {
 function agentMessage(messageId, text, contextId, taskId) {
     return { messageId, role: "ROLE_AGENT", parts: [{ text }], contextId, taskId };
 }
+/** What a failed queued message's task says: why its turn never started. */
+function queueFailureText(payload) {
+    const reason = typeof payload.error?.message === "string" ? ` ${payload.error.message}` : "";
+    return `The message never started a turn.${reason}`;
+}
 function failureText(code) {
     return typeof code === "string" && code !== ""
         ? `The agent failed (${code}).`
@@ -319,6 +337,15 @@ export function updatesOf(event, view) {
             const file = fileArtifact(payload, view.baseUrl);
             return file ? [artifact(file)] : [];
         }
+        case "queue.failed":
+            // A queued message's task that no turn will take (Host feature `queue-recovery`).
+            return [
+                status("TASK_STATE_FAILED", {
+                    message: agentMessage(`${taskId}:failed`, queueFailureText(payload), contextId, taskId),
+                }),
+            ];
+        case "queue.cancelled":
+            return [status("TASK_STATE_CANCELED")];
         case "turn.completed":
         case "turn.failed":
         case "turn.cancelled": {
@@ -468,3 +495,18 @@ export function submittedTask(contextId, taskId, queued) {
         history: [],
     };
 }
+/**
+ * A queued message's task that ended without a turn (Host feature `queue-recovery`), from the
+ * `queue.failed` or `queue.cancelled` event that names it: `FAILED`, saying why it never started,
+ * or `CANCELED`.
+ */
+export function droppedTask(contextId, taskId, ended) {
+    const [update] = updatesOf(ended, { contextId, taskId });
+    const status = update && "statusUpdate" in update ? update.statusUpdate.status : undefined;
+    return {
+        id: taskId,
+        contextId,
+        status: status ?? { state: "TASK_STATE_FAILED", timestamp: ended.time },
+        history: [],
+    };
+}