@oar-sh/oar 0.9.20 → 0.9.21

package/server/relay-tools.md

@@ -42,6 +42,7 @@ This relay itself is listed with "self": true when its owner switched agent sess
 - `create_session` starts a session with any provider that `relay_info` lists: a local one in a folder ("cwd"), or provider "claude-cloud" with "repo" (a GitHub URL or owner/repo) and an optional "branch". The provider ids are the target relay's, as `relay_info` shows them; an id it does not know is refused with the list it has. A session started before a relay update can use providers the update added, without a restart. Without "mode" the new session gets your session's relay mode. At most 4 sessions you started may work at the same time; a fifth is refused until one finishes.
 - The first `create_session` on this relay shows the user an approval card, once per conversation. A session that an agent created cannot create sessions itself (one level only), unless the owner allowed it for that session: `list_sessions` and `read_session` then show "mayCreateSessions": true. Only the owner sets that.
 - Sessions you start here are grouped under this session in the user's sidebar. `list_sessions` lists in the sidebar's order (a group sorts by its newest session, head first) and shows the group as a "group" field (id, name, role); `read_session` adds the group's live members with their state. `create_session` takes "group" (a group id, or its name in any case, from `relay_info`'s "groups") to put the new session into an existing group instead, here or on a paired relay; an unknown group is refused with the groups listed and nothing is created, and the result carries the "group". Renaming and dissolving groups is the user's.
+- `list_sessions` and `read_session` show a session's worker process, while it has one, as "worker" (kind, buildId, stale); "stale": true means it runs older code than that relay, which replaces it by itself once the session is idle, so there is nothing to do about it.
 - `update_session` ("session" plus any of "model", "effort", "mode", "auto_compact_window", "title", "group", "thinking_enabled", "thinking_display", "cwd", "planOnly") changes another session's settings for its next turns without prompting it, here or on a paired relay, and answers the settings and "changed"; a group name that does not exist makes one, "none" takes the session out (a group the user made stays, even empty; one update_session made goes once empty), a new "cwd" restarts its CLI there (refused while it works), "auto_compact_window" 0 is the model default. The model is checked as for `send`. A failed call still lists in "changed" what it wrote before the failing step. A mode set this way lasts until the next `update_session` that names another mode; a `send` that names a mode runs only that turn in it, and one without "mode" leaves the session in its own mode.
 - "planOnly": true makes a plan-only session: agents can't take it out of plan mode. Set it only when the user asks; plan mode alone never sets it. `create_session` with "planOnly": true and no "mode" starts in plan; with another mode named it is refused and nothing is created. A target relay too old for planOnly creates the session without the lock and says so in "planOnlyWarning". `update_session` with "planOnly": true sets it on an existing session (with "mode": "plan" unless it is in plan already). While it is set, a `send` or `update_session` that names another mode is refused with 409, code "plan_only" ("This session is plan-only; agents can't change its mode. Clear planOnly first, or ask the user."); a `send` without "mode" runs in the session's saved mode. To clear it, pass "planOnly": false in the same `update_session` that names the new mode (`{"planOnly":false,"mode":"agent"}`); alone, or with "mode": "plan", it is ignored, and on a plan-only session the answer notes "planOnly unchanged: clear it in the same call as the new mode." Use `update_session` for all this, not the preferences route (it refuses an agent's mode change with 409 and any planOnly change with 403). `list_sessions` and `read_session` show "planOnly": true. In a plan-only conversation you count as a plan-mode caller, also in the one agent turn after "Approve plan": your `create_session` defaults to plan and your write actions ask the user first.
 - Only a session your conversation started may be updated freely; any other (the user's, another agent's, the one that started you) needs the user's approval card once per conversation and relay (refused: `REMOTE_RELAY_PERMISSION`); never your own conversation.

package/server/repositories/message-repository.mjs

@@ -461,6 +461,29 @@ export function createMessageRepository(db) {
           WHERE status = 'processing'
             AND NULLIF(owner_sdk_session_id, '') = ?
         `),
+        // The worker recycle's quiet window (worker-recycle-service.mjs): the
+        // last time the session had a delivery or a turn end, as an ISO
+        // string, or NULL when the queue holds no row of it. Over the rows the
+        // session owns: a row is stamped with its owner at enqueue under
+        // session-worker routing and at the latest by its claim, and a
+        // conversation rebound to another session leaves its old rows with the
+        // session that ran them. The queue keeps no completion time
+        // (setDone clears processing_at), so a turn's end is its response
+        // message's timestamp; the claim and the heartbeat lease
+        // (owner_last_claimed_at) and the assignment stand for the delivery.
+        // Each column is COALESCEd: SQLite's multi-argument MAX() is NULL as
+        // soon as one argument is.
+        getOwnerLastActivityAt: db.prepare(`
+          SELECT MAX(NULLIF(MAX(
+            COALESCE(q.processing_at, ''),
+            COALESCE(q.owner_assigned_at, ''),
+            COALESCE(q.owner_last_claimed_at, ''),
+            COALESCE(m.timestamp, '')
+          ), '')) AS last_activity_at
+          FROM queue q
+          LEFT JOIN messages m ON m.id = q.response_message_id
+          WHERE NULLIF(q.owner_sdk_session_id, '') = ?
+        `),
         recoverProcessingRowKeepOwner: db.prepare(`
           UPDATE queue
           SET status = 'pending',

package/server/repositories/session-repository.mjs

@@ -100,6 +100,25 @@ export function createSessionRepository(db) {
         getRuntimeSessionBySdkSessionId: db.prepare(`SELECT * FROM runtime_sessions WHERE sdk_session_id = ?`),
         getRuntimeSessionById: db.prepare(`SELECT * FROM runtime_sessions WHERE id = ?`),
         listRuntimeSessions: db.prepare(`SELECT rs.*, c.title AS conversation_title, c.updated_at AS conversation_updated_at FROM runtime_sessions rs LEFT JOIN conversations c ON c.id = rs.conversation_id ORDER BY rs.last_used_at DESC`),
+        // The session ids of archived conversations: the worker sweep at relay
+        // start stops what still runs for them. A runtime session id is unique
+        // (idx_runtime_sessions_sdk_session_id), but a conversation's own
+        // sdk_session_id is not: an id that a conversation in use (not
+        // archived, not deleted) names too is left out, so that
+        // conversation's worker is never stopped for an archived one.
+        listArchivedRuntimeSessionIds: db.prepare(`
+          SELECT DISTINCT rs.sdk_session_id AS sdk_session_id
+          FROM runtime_sessions rs
+          JOIN conversations c ON c.id = rs.conversation_id
+          WHERE c.archived = 1
+            AND NULLIF(rs.sdk_session_id, '') IS NOT NULL
+            AND NOT EXISTS (
+              SELECT 1 FROM conversations live_c
+              WHERE live_c.sdk_session_id = rs.sdk_session_id
+                AND COALESCE(live_c.archived, 0) = 0
+                AND live_c.status != 'deleted'
+            )
+        `),
         insertRuntimeSession: db.prepare(runtimeSessionsSupportProviders
           ? `INSERT INTO runtime_sessions (id, conversation_id, strategy, runtime_key, model, status, created_at, last_used_at, sdk_session_id, provider_type, provider_model) VALUES (?, ?, ?, ?, ?, 'active', ?, ?, ?, ?, ?)`
           : `INSERT INTO runtime_sessions (id, conversation_id, strategy, runtime_key, model, status, created_at, last_used_at, sdk_session_id) VALUES (?, ?, ?, ?, ?, 'active', ?, ?, ?)`),

package/server/routes/claude-cloud-routes.mjs

@@ -6,6 +6,9 @@
 //                                     environment, account, login state
 //   POST /api/settings/claude-cloud   { enabled?, defaultModel?, environmentId? }
 //   POST /api/claude-cloud-session    the worker's binding report
+//   GET  /api/claude-cloud-session    `?conversationId=` (or the sdk session
+//                                     id) → the stored binding a starting
+//                                     worker attaches to
 //   GET  /api/claude-cloud/repos      the repositories New Chat may suggest
 //                                     (`?refresh=1` reads them afresh)
 //   GET  /api/claude-cloud/branches   `?repo=<owner/name or URL>` → its branches
@@ -77,6 +80,22 @@ export function registerClaudeCloudRoutes(app, deps = {}) {
       if (!result.ok) return res.status(result.statusCode || 400).json({ error: result.error });
       return res.json({ ok: true, cloud: result.cloud });
     });
+
+    // GET /api/claude-cloud-session?conversationId=<id> — a starting Claude
+    // Cloud worker asks which cloud session it is bound to and from which
+    // sequence to read on, so it opens the event stream before any delivery
+    // (a relaunched worker follows the turns the session starts by itself
+    // from its start). The id may be the conversation id or the sdk session
+    // id. `{ ok, sessionId, lastSequence }`, `sessionId: null` when there is
+    // no cloud session; 404 only for an unknown conversation.
+    app.get('/api/claude-cloud-session', auth, (req, res) => {
+      touchCli();
+      const raw = req.query?.conversationId ?? req.query?.sdkSessionId;
+      const id = Array.isArray(raw) ? raw[0] : raw;
+      const result = claudeCloudSessionService.describeStoredBinding(typeof id === 'string' ? id : '');
+      if (!result.ok) return res.status(result.statusCode || 400).json({ error: result.error });
+      return res.json({ ok: true, sessionId: result.sessionId, lastSequence: result.lastSequence });
+    });
   }
 
   if (claudeCloudLoginStatus) {

package/server/routes/messages-routes.mjs

@@ -45,6 +45,7 @@ import { DEFAULT_CLAUDE_REASONING_EFFORTS } from '../services/provider-reasoning
 import { claudeBaseModelId, claudeLongContextModelId } from '../../shared/model-id.mjs';
 import { sanitizeSubagentRunId } from '../../shared/subagent-run-id.mjs';
 import { normalizeBackgroundTaskDigest } from '../../shared/worker-runtime/background-task-digest.mjs';
+import { normalizeBuildId } from '../../shared/build-id.mjs';
 import {
   DEFAULT_TASK_STALL_MINUTES,
   TASK_OUTPUT_LIMITS,
@@ -1201,6 +1202,12 @@ export async function dequeuePendingMessageForWorkerLoop({
   workerKind = null,
   onBillingFence = null,
   claudeApiClaimCheck = null,
+  // () → true while the replacement fence (services/worker-replacement.mjs)
+  // is up for the requesting process. The caller asks it before calling; it
+  // is asked again before every claim, since ensureWorker and a transient
+  // error's backoff both wait, and the worker recycle can raise the fence
+  // meanwhile.
+  isReplacementFenced = null,
 } = {}) {
   const requesterSid = normalizeSessionWorkerId(requesterSessionId);
   const routingWithWorker = routingEnabled && requesterSid;
@@ -1275,6 +1282,18 @@ export async function dequeuePendingMessageForWorkerLoop({
 
   let attempt = 0;
   while (attempt <= retryLimit) {
+    // The process asking is being replaced: no row is claimed for it, the
+    // same answer as the caller's own fence check gives.
+    if (routingWithWorker && typeof isReplacementFenced === 'function' && isReplacementFenced() === true) {
+      return {
+        message: null,
+        blockedReason: 'worker-replacement',
+        worker: supervisor?.getWorkerState?.(requesterSid) || null,
+        lifecycle: supervisor?.getLifecycleState?.(requesterSid) || null,
+        fallbackRestart: null,
+        attempts: attempt,
+      };
+    }
     const currentIso = String(nowIso || '').trim() || new Date().toISOString();
     try {
       const message = dequeuePendingMessage({
@@ -2065,6 +2084,9 @@ export function registerMessagesRoutes(app, deps) {
     // killTmuxSessionImpl). Tests must inject these instead of relying on the
     // host OS, so the suite behaves identically on Windows and POSIX.
     sessionWorkerStopOverrides = null,
+    // The replacement fence (services/worker-replacement.mjs): a worker
+    // process the recycle is replacing is handed no work by /api/pending.
+    workerReplacementFence = null,
     resolveSessionStateRoot,
     fetchUsageSummary = null,
     planUsageService = null,
@@ -6245,7 +6267,10 @@ export function registerMessagesRoutes(app, deps) {
           // Whether a billing swap may replace the worker now, and why not.
           || (stored.releasable ?? null) !== (steering.releasable ?? null)
           || (stored.releaseHold || null) !== (steering.releaseHold || null)
-          || (stored.backgroundTaskCount || 0) !== (steering.backgroundTaskCount || 0);
+          || (stored.backgroundTaskCount || 0) !== (steering.backgroundTaskCount || 0)
+          // Whether the worker's CLI child still runs: the worker recycle
+          // waits for it to be gone, so a flip must be stored.
+          || (stored.cliAlive ?? null) !== (steering.cliAlive ?? null);
         if (workerForSteering && changed) {
           sessionWorkerRegistry?.upsertWorker?.({ ...workerForSteering, steering });
           // Pushed rather than left for the clients' status poll: the
@@ -6263,6 +6288,16 @@ export function registerMessagesRoutes(app, deps) {
           sessionWorkerRegistry?.upsertWorker?.({ ...workerForKind, kind: heartbeatKind });
         }
       }
+      // The build id of the code the worker started on, recorded the same way:
+      // only when it says one and the registry has another, so quiet
+      // heartbeats stay write-free.
+      const heartbeatBuildId = normalizeBuildId(req.body?.buildId);
+      if (heartbeatBuildId) {
+        const workerForBuild = sessionWorkerRegistry?.getWorker?.(requesterSessionId) || null;
+        if (workerForBuild && workerForBuild.buildId !== heartbeatBuildId) {
+          sessionWorkerRegistry?.upsertWorker?.({ ...workerForBuild, buildId: heartbeatBuildId });
+        }
+      }
       sessionWorkerSupervisor?.noteSessionHeartbeat?.(requesterSessionId);
       // The worker's digest of its live background-task ids against the set
       // held for it (keyed as the worker posts, by its session id). They
@@ -7009,6 +7044,24 @@ export function registerMessagesRoutes(app, deps) {
           restartOrchestrator: relayRestartOrchestrator?.getState?.() || null,
         });
       }
+      // The process asking is being replaced (the worker recycle): no work,
+      // as for a killed session, so no turn starts on the pid being stopped.
+      // Unlike the killed marker, the fence outlives the marker's grace and
+      // every clearRestartSchedule; the new worker has another pid and takes
+      // what waited meanwhile.
+      if (workerReplacementFence?.isHeld?.(requesterSessionId, requester?.pid) === true) {
+        return res.json({
+          message: null,
+          routing: {
+            enabled: true,
+            requesterSessionId,
+            blockedReason: 'worker-replacement',
+            lifecycle: sessionWorkerSupervisor?.getLifecycleState?.(requesterSessionId) || null,
+            fallbackRestart: null,
+          },
+          restartOrchestrator: relayRestartOrchestrator?.getState?.() || null,
+        });
+      }
       sessionWorkerSupervisor?.noteSessionHeartbeat?.(requesterSessionId);
     }
     if (runtimeState.relayPaused) return res.json({ message: null, paused: true });
@@ -7103,6 +7156,8 @@ export function registerMessagesRoutes(app, deps) {
         workerKind: httpPendingWorkerKind(requesterSessionId),
         claudeApiClaimCheck,
         onBillingFence: onClaudeBillingFence,
+        // The fence check above, asked again before every claim.
+        isReplacementFenced: () => workerReplacementFence?.isHeld?.(requesterSessionId, requester?.pid) === true,
         telemetry: (payload = {}) => {
           if (!sessionWorkerRoutingEnabled) return;
           emitSessionWorkerTelemetry(payload.event, {
@@ -7323,6 +7378,7 @@ export function registerMessagesRoutes(app, deps) {
                 workerKind: httpPendingWorkerKind(requesterSessionId),
                 claudeApiClaimCheck,
                 onBillingFence: onClaudeBillingFence,
+                isReplacementFenced: () => workerReplacementFence?.isHeld?.(requesterSessionId, requester?.pid) === true,
               });
               if (recoveryDequeue?.message) {
                 msg = recoveryDequeue.message;

package/server/routes/sessions-routes.mjs

@@ -43,6 +43,7 @@ import { readStoredClaudeContextUsage, resolveAutoCompactWindowStatus } from '..
 import { buildContextUsageView } from '../services/context-usage-view.mjs';
 import { cleanupGeneratedImagesForConversation as cleanupGeneratedImagesForConversationDefault } from '../services/generated-image-cleanup-service.mjs';
 import { stopSessionWorkerProcesses } from '../services/session-worker-stop-service.mjs';
+import { isWorkerOnOtherBuild } from '../services/worker-recycle-service.mjs';
 import { createClaudeAuthService } from '../services/claude-auth-service.mjs';
 import { createCliInstallService } from '../services/cli-install-service.mjs';
 import { createGrokAuthService } from '../services/grok-auth-service.mjs';
@@ -69,7 +70,7 @@ import { redactPreviewTokens } from '../services/preview-registry-service.mjs';
 import { createSharePathRedactor } from '../services/share-path-redaction.mjs';
 import { createSubagentTimelineService, TIMELINE_READ_DEFAULT_LIMIT } from '../services/subagent-timeline-service.mjs';
 import { sanitizeSubagentRunId } from '../../shared/subagent-run-id.mjs';
-import { createKeyedMutex } from '../services/keyed-mutex-service.mjs';
+import { sessionLaunchMutex } from '../services/session-launch-mutex-service.mjs';
 import { createFixedWindowRateLimiter } from '../services/rate-limit-service.mjs';
 import { requestHostOf } from '../services/request-boundary.mjs';
 import { parseRemoteRelayOriginJson } from '../repositories/remote-relay-repository.mjs';
@@ -1452,11 +1453,27 @@ export function buildSessionWorkerStatusPayload({
   // What the last read of the process list did ({ at, durationMs, processes,
   // error }); a slow box shows here before it shows as 502s.
   processList = null,
+  // The relay's own build id (runtimeState.buildId): each worker's is shown
+  // next to it, so a worker still on the code before a deploy stands out.
+  relayBuildId = null,
+  // The worker recycle's status() (stale count, recycles in flight and done,
+  // who waits on what), or null while the recycle is not wired.
+  recycleStatus = null,
 } = {}) {
   const snapshot = supervisorSnapshot && typeof supervisorSnapshot === 'object'
     ? supervisorSnapshot
     : {};
-  const workers = Array.isArray(snapshot.workers) ? snapshot.workers : [];
+  const relayBuild = typeof relayBuildId === 'string' && relayBuildId.trim() ? relayBuildId.trim() : null;
+  const workers = (Array.isArray(snapshot.workers) ? snapshot.workers : []).map((worker) => (
+    worker && typeof worker === 'object'
+      ? {
+        ...worker,
+        buildId: typeof worker.buildId === 'string' && worker.buildId ? worker.buildId : null,
+        startedAt: worker.startedAt || null,
+        stale: isWorkerOnOtherBuild(worker, relayBuild),
+      }
+      : worker
+  ));
   const onlineWorkerStatuses = new Set(SESSION_WORKER_ACTIVE_STATUSES);
   const onlineCount = workers.reduce((count, worker) => {
     const status = normalizeWorkerStatusText(worker?.status, 'new');
@@ -1561,6 +1578,8 @@ export function buildSessionWorkerStatusPayload({
     onlineUnassignedProcessCount,
     counts: snapshot.counts && typeof snapshot.counts === 'object' ? snapshot.counts : {},
     workers,
+    buildId: relayBuild,
+    recycle: recycleStatus && typeof recycleStatus === 'object' ? recycleStatus : null,
     pendingStarts: toSafeNonNegativeInt(snapshot.pendingStarts, 0),
     lifecycle: Array.isArray(snapshot.lifecycle) ? snapshot.lifecycle : [],
     integrity,
@@ -2608,9 +2627,10 @@ export function registerSessionsRoutes(app, deps) {
     bucketTtlMs: 60_000,
     maxBuckets: 4_096,
   });
-  // Serializes everything that reaches the session-worker launch path, keyed by
-  // sdkSessionId, plus a coalescer so a duplicated request replays one result.
-  const sessionLaunchMutex = createKeyedMutex();
+  // Everything that reaches the session-worker launch path is serialized by
+  // the relay-wide sessionLaunchMutex (services/session-launch-mutex-service.mjs),
+  // keyed by sdkSessionId, which the worker recycle takes too; plus a
+  // coalescer so a duplicated request replays one result.
   const relaunchCoalescer = createRelaunchCoalescer({ windowMs: RELAUNCH_COALESCE_WINDOW_MS });
 
   function getDeepSeekProviderSettings() {
@@ -6099,7 +6119,7 @@ export function registerSessionsRoutes(app, deps) {
           // Kill session stops the session's worker, and the delete counts no
           // tasks for a conversation whose worker is gone.
           message: activity.unstoppableTaskCount > 0
-            ? 'Background tasks are still running in this conversation, and some have no Stop button. Use Kill session in its ⋯ menu, then delete it.'
+            ? 'Background tasks are still running in this conversation, and some have no Stop button. Use Kill session in its ︙ menu, then delete it.'
             : 'Background tasks are still running in this conversation. Stop them in the task panel, then delete it.',
         },
       };
@@ -6258,7 +6278,7 @@ export function registerSessionsRoutes(app, deps) {
               error: 'conversation-active',
               reason: 'background-tasks',
               message: activity.unstoppableTaskCount > 0
-                ? 'Background tasks are still running in this conversation, and some have no Stop button. Use Kill session in its ⋯ menu, then archive it.'
+                ? 'Background tasks are still running in this conversation, and some have no Stop button. Use Kill session in its ︙ menu, then archive it.'
                 : 'Background tasks are still running in this conversation. Stop them in the task panel, then archive it.',
             },
           };
@@ -7129,6 +7149,18 @@ export function registerSessionsRoutes(app, deps) {
     }
   });
 
+  // The worker recycle's status, read at request time: the recycle is set on
+  // runtimeState after the routes are registered, and a status that throws
+  // must not take /api/status down with it.
+  function readWorkerRecycleStatus() {
+    if (typeof runtimeState.workerRecycle !== 'function') return null;
+    try {
+      return runtimeState.workerRecycle() || null;
+    } catch {
+      return null;
+    }
+  }
+
   // GET /api/status — overall status
   app.get('/api/status', auth, (req, res) => {
     ensureSessionId(req, res);
@@ -7146,6 +7178,8 @@ export function registerSessionsRoutes(app, deps) {
       supervisorSnapshot: sessionWorkerSupervisor?.snapshot?.({ pendingQuestionSessionIds }) || null,
       queueRows: listSessionWorkerQueueRows.all(...SESSION_WORKER_STATUS_QUEUE_STATES),
       processList: sessionWorkerProcessInspector?.getLastRead?.() || null,
+      relayBuildId: runtimeState.buildId ?? null,
+      recycleStatus: readWorkerRecycleStatus(),
     });
     const activeRuntimeSessionCount = featureFlags?.SESSION_WORKER_ROUTING_ENABLED === true
       ? Number(sessionWorkerStatus?.onlineBoundProcessCount || 0)