@sagentlab/navarch-runtime 0.1.46 → 0.1.47

package/README.md

@@ -204,7 +204,62 @@ restart, not a new dispatch:
 
 The daemon logs `restarting the agent turn in the same worktree` when it
 delivers guidance. Lowering the lease-heartbeat interval makes guidance arrive
-sooner, but keep it comfortably below the 15-minute lease TTL.
+sooner, but keep it comfortably below the 60-minute lease TTL.
+
+Temporary DNS, connection, timeout, rate-limit, and server failures during a
+lease heartbeat are retried at the configured heartbeat interval while the
+last server-confirmed lease remains valid. A separate expiry timer aborts the
+agent at that deadline even if a renewal request is still pending. Successful
+renewals move the deadline; transport failures never extend it. Terminal lease
+rejections still abort immediately. This tolerance applies to lease heartbeats;
+completion and transcript upload failures retain their existing behavior.
+
+## Long-running jobs and recovery
+
+There is no default total-duration cutoff. Successful lease heartbeats renew
+ownership independently of agent progress, so a healthy job can run for hours.
+The default watchdog instead requires 45 minutes of observed inactivity before
+stopping an agent. Stdout/stderr, changing CPU time, and changes in the child
+process tree restart that window. Silent CPU-intensive builds and tests count
+as active. On Unix hosts, agents use a separate process group so cancellation also stops
+their tools; Docker sessions stop their dedicated container.
+
+The watchdog samples `ps` on the host or `docker top` for a container every
+30 seconds. CPU activity is a liveness heuristic, not proof of useful progress:
+a busy loop or repeated output can keep the job alive. Quiet external waits
+may need a larger idle window or idle detection disabled. If process telemetry
+is unavailable, silence alone does not trigger an idle kill; an explicitly
+configured absolute limit and lease expiry still apply.
+
+Operators can set the environment defaults above. Existing launch commands
+that explicitly set `NAVARCH_SESSION_TIMEOUT_MS` (including older generated
+Docker setup commands) retain that cap; remove it or set it to `0` to adopt
+unlimited total runtime. Task creators/editors can set
+`max_runtime_ms` and `idle_timeout_ms` through task creation, PATCH, or the task
+MCP tools. A nonnegative integer overrides the machine default, `0` disables
+that limit, and `null` inherits it. Values are read at the next claim and apply
+to machine-runtime sessions; the separate managed hosted executor retains its
+own budgets. An absolute limit covers all guidance/remediation turns within the
+claim, rather than resetting with each turn. These are runtime limits, not
+provider spending caps.
+
+After a watchdog stop or lease loss during a turn, before worktree cleanup, the runtime attempts a private
+local recovery checkpoint at
+`<workspaceRoot>/checkpoints/<project>/<task>/<session>/`. It contains a Git
+bundle of committed history, a binary patch for dirty tracked files, untracked
+non-ignored files, and recovery instructions. The bundle excludes repository
+configuration and broker credentials. Checkpoints are not pushed or uploaded;
+timeout reports and runtime logs name their location. They survive session cleanup and need
+manual recovery and removal on that machine. Ignored files, agent conversation
+state, and checkpoints across machine/disk loss are not provided.
+
+If checkpoint creation fails, the original working copy is retained instead of
+being force-deleted. The runtime removes session metadata and broker credentials,
+detaches the working copy without discarding dirty files, and writes a recovery
+marker. The janitor skips marked directories; a retry refuses to delete a marked
+working copy if detachment failed. Recover/remove it manually. Container teardown
+has a bounded timeout, and cancellation also stops the local Docker client, so an
+unresponsive daemon cannot keep the session waiting indefinitely.
 
 ## Automatic and manual runtime upgrades
 
@@ -302,8 +357,9 @@ unchanged across the deployment.
 | `NAVARCH_OWNER_ZONE` | `sagentlab` | `sagentlab` or `customer-<slug>-premises` (project-plan.md §3.11). |
 | `NAVARCH_POLL_INTERVAL_MS` | `30000` | Claim-loop poll interval. |
 | `NAVARCH_HEARTBEAT_INTERVAL_MS` | `60000` | Machine-level heartbeat interval. |
-| `NAVARCH_LEASE_HEARTBEAT_INTERVAL_MS` | `300000` | Per-lease heartbeat interval; must stay well under the 15-minute lease TTL (schema-design.md §4). |
-| `NAVARCH_SESSION_TIMEOUT_MS` | `2700000` (45 min) | Hard kill timeout for a single session. |
+| `NAVARCH_LEASE_HEARTBEAT_INTERVAL_MS` | `300000` | Per-lease heartbeat interval; must stay well under the 60-minute lease TTL (schema-design.md §4). |
+| `NAVARCH_SESSION_TIMEOUT_MS` | `0` (disabled) | Optional absolute runtime budget across agent turns in a claimed session. |
+| `NAVARCH_SESSION_IDLE_TIMEOUT_MS` | `2700000` (45 min) | Stop after a continuously observed window with no stdout/stderr or process-tree CPU activity. `0` disables idle detection. |
 | `NAVARCH_WORKTREE_CLEANUP_INTERVAL_MS` | `3600000` (1 hour) | How often the runtime scans for session worktrees abandoned by a hard exit or host restart. |
 | `NAVARCH_WORKTREE_STALE_AFTER_MS` | `86400000` (24 hours) | Minimum inactivity age before an abandoned session worktree is removed. Active sessions are always protected. |
 | `NAVARCH_GIT_AUTHOR_NAME` / `NAVARCH_GIT_AUTHOR_EMAIL` | `sagentlab` / `z@sagentlab.com` | Git identity forced into session commits so host-level personal config is not inherited; override both for a project-authorized bot. |

package/dist/activity-watchdog.cjs

@@ -0,0 +1,110 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ActivityWatchdog = void 0;
+exports.processActivity = processActivity;
+exports.processTreeSignature = processTreeSignature;
+const node_child_process_1 = require("node:child_process");
+const node_perf_hooks_1 = require("node:perf_hooks");
+/** A renewable inactivity deadline, separate from an optional absolute budget. */
+class ActivityWatchdog {
+    options;
+    lastActivity;
+    startedAt;
+    now;
+    signature = null;
+    observed = false;
+    stopped = false;
+    polling = false;
+    pollTimer;
+    hardTimer;
+    constructor(options) {
+        this.options = options;
+        this.now = options.now ?? (() => node_perf_hooks_1.performance.now());
+        this.startedAt = this.lastActivity = this.now();
+        if (options.idleTimeoutMs > 0) {
+            this.pollTimer = setInterval(() => void this.poll(), Math.min(30_000, options.idleTimeoutMs / 4));
+        }
+        if (options.timeoutMs > 0)
+            this.armHardLimit();
+    }
+    activity() {
+        if (!this.stopped)
+            this.lastActivity = this.now();
+    }
+    stop() {
+        this.stopped = true;
+        clearInterval(this.pollTimer);
+        clearTimeout(this.hardTimer);
+    }
+    armHardLimit() {
+        const remaining = this.options.timeoutMs - (this.now() - this.startedAt);
+        this.hardTimer = setTimeout(() => {
+            if (this.stopped)
+                return;
+            if (this.now() - this.startedAt >= this.options.timeoutMs)
+                this.expire("duration");
+            else
+                this.armHardLimit();
+        }, Math.min(remaining, 2_147_483_647));
+    }
+    async poll() {
+        if (this.stopped || this.polling)
+            return;
+        this.polling = true;
+        try {
+            const signature = await this.options.probe().catch(() => null);
+            if (this.stopped)
+                return;
+            // Missing process telemetry is not evidence of a stalled agent. Require
+            // a fresh uninterrupted observation window when telemetry recovers.
+            if (signature === null) {
+                this.observed = false;
+                return;
+            }
+            if (!this.observed || signature !== this.signature)
+                this.activity();
+            this.signature = signature;
+            this.observed = true;
+            if (this.now() - this.lastActivity >= this.options.idleTimeoutMs)
+                this.expire("idle");
+        }
+        finally {
+            this.polling = false;
+        }
+    }
+    expire(reason) {
+        this.stop();
+        this.options.onTimeout(reason);
+    }
+}
+exports.ActivityWatchdog = ActivityWatchdog;
+/** No command arguments or environment values are collected from other jobs. */
+async function processActivity(pid, containerName) {
+    if (!containerName && pid === undefined)
+        return null;
+    const output = await new Promise((resolve) => {
+        (0, node_child_process_1.execFile)(containerName ? "docker" : "ps", containerName ? ["top", containerName, "-eo", "pid=,ppid=,time="] : ["-e", "-o", "pid=,ppid=,time="], { timeout: 5_000, maxBuffer: 4 * 1024 * 1024 }, (error, stdout) => resolve(error ? null : stdout));
+    });
+    return output === null ? null : processTreeSignature(output, containerName ? undefined : pid);
+}
+function processTreeSignature(output, rootPid) {
+    const rows = output.split("\n").flatMap((line) => {
+        const match = /^\s*(\d+)\s+(\d+)\s+([\d:.\-]+)\s*$/.exec(line);
+        return match ? [{ pid: Number(match[1]), parent: Number(match[2]), cpu: match[3] }] : [];
+    });
+    if (!rows.length || (rootPid !== undefined && !rows.some((row) => row.pid === rootPid)))
+        return null;
+    const included = new Set(rootPid === undefined ? rows.map((row) => row.pid) : [rootPid]);
+    let changed = true;
+    while (changed) {
+        changed = false;
+        for (const row of rows) {
+            if (included.has(row.parent) && !included.has(row.pid)) {
+                included.add(row.pid);
+                changed = true;
+            }
+        }
+    }
+    return rows.filter((row) => included.has(row.pid)).sort((a, b) => a.pid - b.pid)
+        .map((row) => `${row.pid}:${row.cpu}`).join(",");
+}

package/dist/adapter-watchdog.cjs

@@ -0,0 +1,40 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.runWithActivityWatchdog = runWithActivityWatchdog;
+const activity_watchdog_cjs_1 = require("./activity-watchdog.cjs");
+/** Shared policy for every host/Docker/ACP adapter; adapters still own cancellation. */
+async function runWithActivityWatchdog(adapter, options, idleTimeoutMs) {
+    const controller = new AbortController();
+    const onAbort = () => controller.abort();
+    options.signal?.addEventListener("abort", onAbort, { once: true });
+    if (options.signal?.aborted)
+        onAbort();
+    let pid;
+    let timeoutReason;
+    const watchdog = new activity_watchdog_cjs_1.ActivityWatchdog({
+        idleTimeoutMs,
+        timeoutMs: options.timeoutMs,
+        probe: () => (0, activity_watchdog_cjs_1.processActivity)(pid, options.dockerExec?.containerName),
+        onTimeout: (reason) => {
+            timeoutReason = reason;
+            controller.abort();
+        },
+    });
+    try {
+        const result = await adapter.run({
+            ...options,
+            // One policy handles both hard and idle limits, including Docker.
+            timeoutMs: 0,
+            signal: controller.signal,
+            onProcessStarted: (value) => { pid = value; options.onProcessStarted?.(value); },
+            onActivity: () => { watchdog.activity(); options.onActivity?.(); },
+        });
+        return timeoutReason && !options.signal?.aborted
+            ? { ...result, timedOut: true, timeoutReason, killedByLeaseLoss: false }
+            : result;
+    }
+    finally {
+        watchdog.stop();
+        options.signal?.removeEventListener("abort", onAbort);
+    }
+}

package/dist/adapters/acp.cjs

@@ -5,6 +5,7 @@ exports.runAcpAdapter = runAcpAdapter;
 const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
 const node_stream_1 = require("node:stream");
+const kill_agent_cjs_1 = require("../kill-agent.cjs");
 /**
  * How long a cancelled agent may take to observe `session/cancel` and settle
  * the in-flight `session/prompt` before the subprocess is terminated. The
@@ -30,6 +31,7 @@ async function runAcpAdapter(options) {
     const acp = await import("@agentclientprotocol/sdk");
     const mcpServers = await readMcpServers(options.mcpConfigPath, options.env);
     const child = spawnAcpAgent(options);
+    options.onProcessStarted?.(child.pid);
     const rawStdout = [];
     const stderr = [];
     const protocolInput = new node_stream_1.PassThrough();
@@ -47,12 +49,16 @@ async function runAcpAdapter(options) {
         settlePrompt = resolve;
     });
     child.stdout.on("data", (chunk) => {
+        options.onActivity?.();
         rawStdout.push(chunk);
         protocolInput.write(chunk);
     });
     child.stdout.on("end", () => protocolInput.end());
     child.stderr.setEncoding("utf8");
-    child.stderr.on("data", (chunk) => stderr.push(chunk));
+    child.stderr.on("data", (chunk) => {
+        options.onActivity?.();
+        stderr.push(chunk);
+    });
     child.on("error", (error) => {
         stderr.push(`\n${describeError(error)}`);
         protocolInput.end();
@@ -88,10 +94,10 @@ async function runAcpAdapter(options) {
             connection.close();
         })();
     }, { once: true });
-    const timer = setTimeout(() => {
+    const timer = options.timeoutMs > 0 ? setTimeout(() => {
         timedOut = true;
         controller.abort();
-    }, options.timeoutMs);
+    }, options.timeoutMs) : undefined;
     const onAbort = () => {
         killedByLeaseLoss = true;
         controller.abort();
@@ -126,7 +132,7 @@ async function runAcpAdapter(options) {
         stopReason = response.stopReason;
         await Promise.race([
             context.request(acp.methods.agent.session.close, { sessionId }).catch(() => undefined),
-            new Promise((resolve) => setTimeout(resolve, Math.min(5_000, options.timeoutMs))),
+            new Promise((resolve) => setTimeout(resolve, options.timeoutMs > 0 ? Math.min(5_000, options.timeoutMs) : 5_000)),
         ]);
     }
     catch (error) {
@@ -225,6 +231,7 @@ function spawnAcpAgent(options) {
     const env = { ...process.env, ...options.env };
     if (!options.dockerExec) {
         return (0, node_child_process_1.spawn)(options.bin, options.extraArgs, {
+            detached: process.platform !== "win32",
             cwd: options.cwd,
             env,
             stdio: ["pipe", "pipe", "pipe"],
@@ -247,16 +254,18 @@ function spawnAcpAgent(options) {
 }
 async function terminateChild(child, options, forceContainerKill) {
     if (child.exitCode === null && child.signalCode === null) {
-        child.kill("SIGTERM");
+        (0, kill_agent_cjs_1.killAgent)(child, "SIGTERM");
         await Promise.race([
             new Promise((resolve) => child.once("close", () => resolve())),
             new Promise((resolve) => setTimeout(resolve, 1_000)),
         ]);
         if (child.exitCode === null && child.signalCode === null)
-            child.kill("SIGKILL");
+            (0, kill_agent_cjs_1.killAgent)(child);
     }
+    if (!options.dockerExec && forceContainerKill)
+        (0, kill_agent_cjs_1.killAgent)(child);
     if (options.dockerExec && forceContainerKill) {
-        await options.dockerExec.runner.run("docker", ["kill", options.dockerExec.containerName]).catch(() => undefined);
+        await options.dockerExec.runner.run("docker", ["kill", options.dockerExec.containerName], { timeoutMs: 10_000 }).catch(() => undefined);
     }
 }
 function describeError(error) {

package/dist/adapters/claude.cjs

@@ -4,6 +4,7 @@ exports.claudeCodeAdapter = void 0;
 exports.runClaudeCodeAdapter = runClaudeCodeAdapter;
 const node_child_process_1 = require("node:child_process");
 const exit_conditions_cjs_1 = require("../exit-conditions.cjs");
+const kill_agent_cjs_1 = require("../kill-agent.cjs");
 const adapter_capacity_cjs_1 = require("../adapter-capacity.cjs");
 const DEFAULT_CLAUDE_MODEL = "claude-opus-5";
 const FALLBACK_CLAUDE_MODEL = "claude-opus-5";
@@ -39,9 +40,9 @@ async function runClaudeCodeAdapter(options) {
         !hasFlag(options.extraArgs, "--fallback-model");
     const startedAt = Date.now();
     const primary = await runClaudeAttempt({ ...options, model }, automaticFallback);
-    const remainingMs = options.timeoutMs - (Date.now() - startedAt);
+    const remainingMs = options.timeoutMs > 0 ? options.timeoutMs - (Date.now() - startedAt) : 0;
     if (!automaticFallback || primary.timedOut || primary.killedByLeaseLoss ||
-        options.signal?.aborted || remainingMs <= 0)
+        options.signal?.aborted || (options.timeoutMs > 0 && remainingMs <= 0))
         return primary;
     const parsed = (0, exit_conditions_cjs_1.parseClaudeJsonResult)(primary.stdout) ?? (0, exit_conditions_cjs_1.parseClaudeJsonResult)(primary.stderr);
     // Native fallback handles overload/unavailability, but explicitly excludes
@@ -175,6 +176,7 @@ async function runOnHost(options, args) {
         let timedOut = false;
         let killedByLeaseLoss = false;
         const child = (0, node_child_process_1.spawn)(options.bin, args, {
+            detached: process.platform !== "win32",
             cwd: options.cwd,
             env: { ...process.env, ...options.env },
         });
@@ -183,19 +185,24 @@ async function runOnHost(options, args) {
         // supplied as an argument. Leaving it open makes Claude wait and emit a
         // "no stdin data received" warning that the runtime treats as failure.
         child.stdin?.end();
-        const timer = setTimeout(() => {
+        options.onProcessStarted?.(child.pid);
+        const timer = options.timeoutMs > 0 ? setTimeout(() => {
             timedOut = true;
-            child.kill("SIGKILL");
-        }, options.timeoutMs);
+            (0, kill_agent_cjs_1.killAgent)(child);
+        }, options.timeoutMs) : undefined;
         const onAbort = () => {
             killedByLeaseLoss = true;
-            child.kill("SIGKILL");
+            (0, kill_agent_cjs_1.killAgent)(child);
         };
         options.signal?.addEventListener("abort", onAbort, { once: true });
+        if (options.signal?.aborted)
+            onAbort();
         child.stdout.on("data", (d) => {
+            options.onActivity?.();
             stdout += d.toString();
         });
         child.stderr.on("data", (d) => {
+            options.onActivity?.();
             stderr += d.toString();
         });
         child.on("error", (err) => {
@@ -218,16 +225,19 @@ async function runViaDocker(options, args) {
     let killedByLeaseLoss = false;
     const onAbort = () => {
         killedByLeaseLoss = true;
-        // Best-effort: kill the exec'd process inside the container. The runner
-        // call below still resolves once `docker exec` itself is torn down by
-        // its own timeout/kill; a true "kill this docker exec now" requires
-        // process-group tracking that nodeCommandRunner does not yet expose.
-        runner.run("docker", ["kill", containerName]).catch(() => undefined);
+        // Ask the daemon to stop the container, with its own deadline. The
+        // forwarded signal independently cancels the local exec client below,
+        // including when the daemon cannot answer this request.
+        runner.run("docker", ["kill", containerName], { timeoutMs: 10_000 }).catch(() => undefined);
     };
     options.signal?.addEventListener("abort", onAbort, { once: true });
+    if (options.signal?.aborted)
+        onAbort();
     try {
         const result = await runner.run("docker", ["exec", containerName, "sh", "-c", command], {
             timeoutMs: options.timeoutMs,
+            onActivity: options.onActivity,
+            signal: options.signal,
         });
         return {
             exitCode: result.code,

package/dist/adapters/codex.cjs

@@ -7,6 +7,7 @@ const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
 const exit_conditions_cjs_1 = require("../exit-conditions.cjs");
 const codex_pricing_cjs_1 = require("../codex-pricing.cjs");
+const kill_agent_cjs_1 = require("../kill-agent.cjs");
 /**
  * Headless OpenAI Codex CLI adapter — the Codex sibling of claude.cts's
  * `runClaudeCodeAdapter`, implementing the same AgentAdapter interface
@@ -168,6 +169,7 @@ async function runOnHost(options, args) {
         let timedOut = false;
         let killedByLeaseLoss = false;
         const child = (0, node_child_process_1.spawn)(options.bin, args, {
+            detached: process.platform !== "win32",
             cwd: options.cwd,
             env: { ...process.env, ...options.env },
         });
@@ -175,19 +177,24 @@ async function runOnHost(options, args) {
         // a pipe by default, so close it immediately or it can wait forever for
         // input even though the full prompt was supplied as an argument.
         child.stdin?.end();
-        const timer = setTimeout(() => {
+        options.onProcessStarted?.(child.pid);
+        const timer = options.timeoutMs > 0 ? setTimeout(() => {
             timedOut = true;
-            child.kill("SIGKILL");
-        }, options.timeoutMs);
+            (0, kill_agent_cjs_1.killAgent)(child);
+        }, options.timeoutMs) : undefined;
         const onAbort = () => {
             killedByLeaseLoss = true;
-            child.kill("SIGKILL");
+            (0, kill_agent_cjs_1.killAgent)(child);
         };
         options.signal?.addEventListener("abort", onAbort, { once: true });
+        if (options.signal?.aborted)
+            onAbort();
         child.stdout.on("data", (d) => {
+            options.onActivity?.();
             stdout += d.toString();
         });
         child.stderr.on("data", (d) => {
+            options.onActivity?.();
             stderr += d.toString();
         });
         child.on("error", (err) => {
@@ -210,12 +217,13 @@ async function runViaDocker(options, args) {
     let killedByLeaseLoss = false;
     const onAbort = () => {
         killedByLeaseLoss = true;
-        // Best-effort: kill the exec'd process inside the container — see
-        // claude.cts's identical runViaDocker for the same caveat about
-        // nodeCommandRunner not exposing process-group tracking.
-        runner.run("docker", ["kill", containerName]).catch(() => undefined);
+        // Container shutdown is bounded independently from cancellation of the
+        // local docker exec process, which receives the signal below.
+        runner.run("docker", ["kill", containerName], { timeoutMs: 10_000 }).catch(() => undefined);
     };
     options.signal?.addEventListener("abort", onAbort, { once: true });
+    if (options.signal?.aborted)
+        onAbort();
     try {
         // Forward values through the docker CLI process environment and put only
         // variable names in argv. This keeps generated MCP bearer/header values
@@ -223,6 +231,8 @@ async function runViaDocker(options, args) {
         const forwardedEnv = Object.keys(options.env).flatMap((name) => ["--env", name]);
         const result = await runner.run("docker", ["exec", ...forwardedEnv, containerName, "sh", "-c", command], {
             timeoutMs: options.timeoutMs,
+            onActivity: options.onActivity,
+            signal: options.signal,
             env: { ...process.env, ...options.env },
         });
         return {

package/dist/adapters/gemini.cjs

@@ -4,6 +4,7 @@ exports.geminiAdapter = void 0;
 exports.runGeminiAdapter = runGeminiAdapter;
 const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
+const kill_agent_cjs_1 = require("../kill-agent.cjs");
 /**
  * Headless Google Gemini CLI adapter. Gemini's `stream-json` output gives the
  * same normalized boundary as the Claude and Codex adapters: raw process
@@ -152,23 +153,29 @@ async function runOnHost(options, args) {
         let timedOut = false;
         let killedByLeaseLoss = false;
         const child = (0, node_child_process_1.spawn)(options.bin, args, {
+            detached: process.platform !== "win32",
             cwd: options.cwd,
             env: { ...process.env, ...options.env },
         });
         child.stdin?.end();
-        const timer = setTimeout(() => {
+        options.onProcessStarted?.(child.pid);
+        const timer = options.timeoutMs > 0 ? setTimeout(() => {
             timedOut = true;
-            child.kill("SIGKILL");
-        }, options.timeoutMs);
+            (0, kill_agent_cjs_1.killAgent)(child);
+        }, options.timeoutMs) : undefined;
         const onAbort = () => {
             killedByLeaseLoss = true;
-            child.kill("SIGKILL");
+            (0, kill_agent_cjs_1.killAgent)(child);
         };
         options.signal?.addEventListener("abort", onAbort, { once: true });
+        if (options.signal?.aborted)
+            onAbort();
         child.stdout.on("data", (data) => {
+            options.onActivity?.();
             stdout += data.toString();
         });
         child.stderr.on("data", (data) => {
+            options.onActivity?.();
             stderr += data.toString();
         });
         child.on("error", (err) => {
@@ -193,12 +200,18 @@ async function runViaDocker(options, args) {
     let killedByLeaseLoss = false;
     const onAbort = () => {
         killedByLeaseLoss = true;
-        runner.run("docker", ["kill", containerName]).catch(() => undefined);
+        runner.run("docker", ["kill", containerName], { timeoutMs: 10_000 }).catch(() => undefined);
     };
     options.signal?.addEventListener("abort", onAbort, { once: true });
+    if (options.signal?.aborted)
+        onAbort();
     try {
         const forwardedEnv = Object.keys(options.env).flatMap((name) => ["--env", name]);
-        const result = await runner.run("docker", ["exec", ...forwardedEnv, containerName, "sh", "-c", command], { timeoutMs: options.timeoutMs, env: { ...process.env, ...options.env } });
+        const result = await runner.run("docker", ["exec", ...forwardedEnv, containerName, "sh", "-c", command], { timeoutMs: options.timeoutMs,
+            onActivity: options.onActivity,
+            signal: options.signal,
+            env: { ...process.env, ...options.env },
+        });
         return {
             exitCode: result.code,
             timedOut: false,

package/dist/adapters/index.cjs

@@ -48,7 +48,7 @@ function selectAdapter(agentType) {
             // Exhaustiveness guard: config.cts only ever produces the values
             // above, but fall back to Claude Code rather than throwing if this
             // widens in the future without every caller being updated.
-            const _exhaustive = agentType;
+            void agentType;
             return claude_cjs_1.claudeCodeAdapter;
         }
     }

package/dist/adapters/opencode.cjs

@@ -5,6 +5,7 @@ exports.runOpenCodeAdapter = runOpenCodeAdapter;
 const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
 const config_cjs_1 = require("../config.cjs");
+const kill_agent_cjs_1 = require("../kill-agent.cjs");
 /**
  * Headless OpenCode adapter, fixture-pinned to the OpenCode 1.18 JSON/config
  * contract:
@@ -191,23 +192,29 @@ async function runOnHost(options, args) {
         let timedOut = false;
         let killedByLeaseLoss = false;
         const child = (0, node_child_process_1.spawn)(options.bin, args, {
+            detached: process.platform !== "win32",
             cwd: options.cwd,
             env: { ...process.env, ...options.env },
         });
         child.stdin?.end();
-        const timer = setTimeout(() => {
+        options.onProcessStarted?.(child.pid);
+        const timer = options.timeoutMs > 0 ? setTimeout(() => {
             timedOut = true;
-            child.kill("SIGKILL");
-        }, options.timeoutMs);
+            (0, kill_agent_cjs_1.killAgent)(child);
+        }, options.timeoutMs) : undefined;
         const onAbort = () => {
             killedByLeaseLoss = true;
-            child.kill("SIGKILL");
+            (0, kill_agent_cjs_1.killAgent)(child);
         };
         options.signal?.addEventListener("abort", onAbort, { once: true });
+        if (options.signal?.aborted)
+            onAbort();
         child.stdout.on("data", (data) => {
+            options.onActivity?.();
             stdout += data.toString();
         });
         child.stderr.on("data", (data) => {
+            options.onActivity?.();
             stderr += data.toString();
         });
         child.on("error", (error) => {
@@ -230,12 +237,18 @@ async function runViaDocker(options, args) {
     let killedByLeaseLoss = false;
     const onAbort = () => {
         killedByLeaseLoss = true;
-        runner.run("docker", ["kill", containerName]).catch(() => undefined);
+        runner.run("docker", ["kill", containerName], { timeoutMs: 10_000 }).catch(() => undefined);
     };
     options.signal?.addEventListener("abort", onAbort, { once: true });
+    if (options.signal?.aborted)
+        onAbort();
     try {
         const forwardedEnv = Object.keys(options.env).flatMap((name) => ["--env", name]);
-        const result = await runner.run("docker", ["exec", ...forwardedEnv, containerName, "sh", "-c", command], { timeoutMs: options.timeoutMs, env: { ...process.env, ...options.env } });
+        const result = await runner.run("docker", ["exec", ...forwardedEnv, containerName, "sh", "-c", command], { timeoutMs: options.timeoutMs,
+            onActivity: options.onActivity,
+            signal: options.signal,
+            env: { ...process.env, ...options.env },
+        });
         return {
             exitCode: result.code,
             timedOut: false,

package/dist/adapters/pi.cjs

@@ -13,6 +13,7 @@ const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
 const node_os_1 = __importDefault(require("node:os"));
 const node_path_1 = __importDefault(require("node:path"));
+const kill_agent_cjs_1 = require("../kill-agent.cjs");
 /**
  * Headless Pi coding-agent adapter (#885) — the process-side twin of the
  * Cloudflare hosted executor (worker/hosted-pi/lifecycle.ts). It invokes
@@ -262,23 +263,29 @@ async function runOnHost(options, args) {
             let timedOut = false;
             let killedByLeaseLoss = false;
             const child = (0, node_child_process_1.spawn)(options.bin, boundedArgs, {
+                detached: process.platform !== "win32",
                 cwd: options.cwd,
                 env: { ...process.env, ...env },
             });
             child.stdin?.end();
-            const timer = setTimeout(() => {
+            options.onProcessStarted?.(child.pid);
+            const timer = options.timeoutMs > 0 ? setTimeout(() => {
                 timedOut = true;
-                child.kill("SIGKILL");
-            }, options.timeoutMs);
+                (0, kill_agent_cjs_1.killAgent)(child);
+            }, options.timeoutMs) : undefined;
             const onAbort = () => {
                 killedByLeaseLoss = true;
-                child.kill("SIGKILL");
+                (0, kill_agent_cjs_1.killAgent)(child);
             };
             options.signal?.addEventListener("abort", onAbort, { once: true });
+            if (options.signal?.aborted)
+                onAbort();
             child.stdout.on("data", (data) => {
+                options.onActivity?.();
                 stdout += data.toString();
             });
             child.stderr.on("data", (data) => {
+                options.onActivity?.();
                 stderr += data.toString();
             });
             child.on("error", (err) => {
@@ -328,13 +335,17 @@ async function runViaDocker(options, args) {
     let killedByLeaseLoss = false;
     const onAbort = () => {
         killedByLeaseLoss = true;
-        runner.run("docker", ["kill", containerName]).catch(() => undefined);
+        runner.run("docker", ["kill", containerName], { timeoutMs: 10_000 }).catch(() => undefined);
     };
     options.signal?.addEventListener("abort", onAbort, { once: true });
+    if (options.signal?.aborted)
+        onAbort();
     try {
         const forwardedEnv = Object.keys(options.env).flatMap((name) => ["--env", name]);
         const result = await runner.run("docker", ["exec", ...forwardedEnv, containerName, "sh", "-c", command], {
             timeoutMs: options.timeoutMs,
+            onActivity: options.onActivity,
+            signal: options.signal,
             env: { ...process.env, ...options.env },
         });
         return {

package/dist/config.cjs

@@ -26,6 +26,13 @@ function envInt(env, name, fallback) {
     const n = Number(raw);
     return Number.isFinite(n) && n > 0 ? n : fallback;
 }
+function envRuntimeLimit(env, name, fallback) {
+    const raw = env[name]?.trim();
+    if (!raw)
+        return fallback;
+    const value = Number(raw);
+    return Number.isSafeInteger(value) && value >= 0 ? value : fallback;
+}
 /** Default `$CODEX_HOME`, matching the Codex CLI's own resolution. */
 function defaultCodexHome(env) {
     return env.CODEX_HOME ?? node_path_1.default.join(node_os_1.default.homedir(), ".codex");
@@ -96,10 +103,11 @@ function loadRuntimeConfig(env = process.env) {
         ownerZone: env.NAVARCH_OWNER_ZONE ?? "sagentlab",
         pollIntervalMs: envInt(env, "NAVARCH_POLL_INTERVAL_MS", 30_000),
         machineHeartbeatIntervalMs: envInt(env, "NAVARCH_HEARTBEAT_INTERVAL_MS", 60_000),
-        // leases.expires_at = claimed_at + 15 min (schema-design.md §4) — default renewal
+        // leases.expires_at = claimed_at + 60 min (schema-design.md §4) — default renewal
         // interval must stay comfortably under that TTL.
         leaseHeartbeatIntervalMs,
-        sessionTimeoutMs: envInt(env, "NAVARCH_SESSION_TIMEOUT_MS", 45 * 60 * 1000),
+        sessionTimeoutMs: envRuntimeLimit(env, "NAVARCH_SESSION_TIMEOUT_MS", 0),
+        sessionIdleTimeoutMs: envRuntimeLimit(env, "NAVARCH_SESSION_IDLE_TIMEOUT_MS", 45 * 60 * 1000),
         worktreeCleanupIntervalMs: envInt(env, "NAVARCH_WORKTREE_CLEANUP_INTERVAL_MS", 60 * 60 * 1000),
         worktreeStaleAfterMs: envInt(env, "NAVARCH_WORKTREE_STALE_AFTER_MS", 24 * 60 * 60 * 1000),
         agentType,

package/dist/exit-conditions.cjs

@@ -233,7 +233,11 @@ function mapExitCondition(result) {
         return {
             leaseOutcome: "failed",
             exitStatus: "killed",
-            reportSummary: "Session killed: exceeded the configured max session duration.",
+            reportSummary: (result.timeoutReason === "idle"
+                ? "Session stopped: no output or process-tree CPU activity within the configured idle window."
+                : "Session killed: exceeded the configured max session duration.") +
+                (result.checkpointPath ? ` Local recovery checkpoint: ${result.checkpointPath}` : "") +
+                (result.retainedWorktreePath ? ` Recovery worktree retained: ${result.retainedWorktreePath}` : ""),
             evidenceUrls,
         };
     }

package/dist/git-worktree.cjs

@@ -6,6 +6,7 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.GitWorktree = void 0;
 exports.branchSlug = branchSlug;
 const node_path_1 = __importDefault(require("node:path"));
+const worktree_janitor_cjs_1 = require("./worktree-janitor.cjs");
 const node_fs_1 = require("node:fs");
 const sandbox_cjs_1 = require("./sandbox.cjs");
 const repositoryLocks = new Map();
@@ -21,6 +22,7 @@ class GitWorktree {
     repositoryPath;
     branch;
     runner;
+    checkpointRoot;
     cloneUrl;
     githubToken;
     repoLocalGithubToken;
@@ -32,6 +34,7 @@ class GitWorktree {
         const projectKey = safePathSegment(options.projectId);
         const sessionKey = safePathSegment(options.sessionId);
         const taskKey = safePathSegment(options.taskId);
+        this.checkpointRoot = node_path_1.default.join(options.workspaceRoot, "checkpoints", projectKey, taskKey, sessionKey);
         this.sessionRoot = node_path_1.default.join(options.workspaceRoot, "sessions", sessionKey);
         this.worktreePath = node_path_1.default.join(this.sessionRoot, "repo");
         this.repositoryPath = options.repoLocalGithubToken
@@ -215,6 +218,9 @@ class GitWorktree {
                 holders.push(currentPath);
         }
         for (const holder of holders) {
+            if (await (0, worktree_janitor_cjs_1.hasRecoveryMarker)(node_path_1.default.dirname(holder))) {
+                throw new Error(`Worktree ${holder} is retained for recovery; restore or remove it before reusing this branch.`);
+            }
             // Double --force covers locked and submodule-bearing worktrees.
             await this.runner
                 .run("git", ["--git-dir", this.repositoryPath, "worktree", "remove", "--force", "--force", holder])
@@ -380,6 +386,50 @@ class GitWorktree {
         }
         return { branch: this.branch, sha: head.sha };
     }
+    /** Save recoverable code, never the repository config or broker credentials. */
+    async saveCheckpoint() {
+        await node_fs_1.promises.mkdir(this.checkpointRoot, { recursive: true, mode: 0o700 });
+        const bundlePath = node_path_1.default.join(this.checkpointRoot, "commits.bundle");
+        await this.runGit(["-C", this.worktreePath, "bundle", "create", bundlePath, "HEAD"], false);
+        await node_fs_1.promises.chmod(bundlePath, 0o600);
+        const patch = await this.runGit(["-C", this.worktreePath, "diff", "--binary", "HEAD", "--"], false);
+        await node_fs_1.promises.writeFile(node_path_1.default.join(this.checkpointRoot, "working-tree.patch"), patch.stdout, { mode: 0o600 });
+        const untracked = await this.runGit(["-C", this.worktreePath, "ls-files", "--others", "--exclude-standard", "-z"], false);
+        for (const name of untracked.stdout.split("\0").filter(Boolean)) {
+            const source = node_path_1.default.resolve(this.worktreePath, name);
+            if (!source.startsWith(this.worktreePath + node_path_1.default.sep))
+                throw new Error("Invalid checkpoint path");
+            const target = node_path_1.default.join(this.checkpointRoot, "untracked", name);
+            await node_fs_1.promises.mkdir(node_path_1.default.dirname(target), { recursive: true, mode: 0o700 });
+            await node_fs_1.promises.cp(source, target, { dereference: false, verbatimSymlinks: true });
+        }
+        const head = await this.resolveCurrentHead();
+        await node_fs_1.promises.writeFile(node_path_1.default.join(this.checkpointRoot, "README.txt"), `Navarch local recovery checkpoint\nBranch: ${head.branch ?? "detached"}\nHEAD: ${head.sha}\n` +
+            "Fetch commits.bundle HEAD into a clean checkout and inspect that commit. Apply working-tree.patch with git apply, then copy files from untracked/. " +
+            "Ignored files and agent conversation state are not included. Nothing was pushed.\n", { mode: 0o600 });
+        return this.checkpointRoot;
+    }
+    /** Keep the original code when a checkpoint fails, but strip runtime secrets. */
+    async retainForRecovery() {
+        await node_fs_1.promises.writeFile(node_path_1.default.join(this.sessionRoot, worktree_janitor_cjs_1.SESSION_RECOVERY_FILENAME), "", { mode: 0o600 });
+        try {
+            const head = await this.resolveCurrentHead();
+            // Detach without checking out files: this also works with a conflicted
+            // index, preserves dirty work, and releases the delivery branch for retry.
+            await this.runGit(["-C", this.worktreePath, "update-ref", "--no-deref", "HEAD", head.sha], false);
+        }
+        finally {
+            const entries = await node_fs_1.promises.readdir(this.sessionRoot);
+            const keep = new Set(["repo", "repository.git", worktree_janitor_cjs_1.SESSION_RECOVERY_FILENAME]);
+            const cleanup = await Promise.allSettled([
+                this.removeRepoLocalGithubToken(),
+                ...entries.filter((name) => !keep.has(name)).map((name) => node_fs_1.promises.rm(node_path_1.default.join(this.sessionRoot, name), { recursive: true, force: true })),
+            ]);
+            const failures = cleanup.filter((result) => result.status === "rejected");
+            if (failures.length)
+                throw new AggregateError(failures.map((result) => result.reason), "Recovery credential cleanup failed");
+        }
+    }
     async cleanup() {
         await withRepositoryLock(this.repositoryPath, async () => {
             await this.removeRepoLocalGithubToken();

package/dist/kill-agent.cjs

@@ -0,0 +1,14 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.killAgent = killAgent;
+/** Host agents have a dedicated process group so their tools stop with them. */
+function killAgent(child, signal = "SIGKILL") {
+    if (process.platform !== "win32" && child.pid) {
+        try {
+            process.kill(-child.pid, signal);
+            return;
+        }
+        catch { /* Already exited or no group. */ }
+    }
+    child.kill(signal);
+}

package/dist/lease-heartbeat.cjs

@@ -0,0 +1,45 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.LeaseHeartbeatGuard = void 0;
+const api_cjs_1 = require("./api.cjs");
+/** Transient failures never extend the last deadline confirmed by the server. */
+class LeaseHeartbeatGuard {
+    onExpired;
+    deadline = 0;
+    timer;
+    stopped = false;
+    constructor(expiresAt, onExpired) {
+        this.onExpired = onExpired;
+        this.renew(expiresAt);
+    }
+    renew(expiresAt) {
+        if (this.stopped)
+            return;
+        if (this.deadline !== 0 && Date.now() >= this.deadline) {
+            this.expire();
+            return;
+        }
+        clearTimeout(this.timer);
+        this.deadline = Date.parse(expiresAt);
+        const remaining = this.deadline - Date.now();
+        if (!Number.isFinite(remaining) || remaining <= 0) {
+            this.expire();
+            return;
+        }
+        // Independent of polling: a hanging renewal must not outlive the lease.
+        this.timer = setTimeout(() => this.expire(), remaining);
+    }
+    canRetry(error) {
+        return !this.stopped && Date.now() < this.deadline && (error instanceof api_cjs_1.NavarchTransportError ||
+            (error instanceof api_cjs_1.NavarchApiError && (error.status === 408 || error.status === 429 || error.status >= 500)));
+    }
+    stop() {
+        this.stopped = true;
+        clearTimeout(this.timer);
+    }
+    expire() {
+        this.stop();
+        this.onExpired();
+    }
+}
+exports.LeaseHeartbeatGuard = LeaseHeartbeatGuard;

package/dist/sandbox.cjs

@@ -8,6 +8,7 @@ exports.isDockerAvailable = isDockerAvailable;
 const node_child_process_1 = require("node:child_process");
 const node_fs_1 = require("node:fs");
 const node_path_1 = __importDefault(require("node:path"));
+const kill_agent_cjs_1 = require("./kill-agent.cjs");
 const sandbox_profile_cjs_1 = require("./sandbox-profile.cjs");
 class SandboxUnavailableError extends Error {
 }
@@ -15,36 +16,47 @@ exports.SandboxUnavailableError = SandboxUnavailableError;
 /** Real process-spawning CommandRunner. Tests inject a fake one instead — see sandbox.test.cts. */
 exports.nodeCommandRunner = {
     run(cmd, args, opts = {}) {
+        if (opts.signal?.aborted)
+            return Promise.resolve({ code: -1, stdout: "", stderr: "" });
         return new Promise((resolve, reject) => {
-            const child = (0, node_child_process_1.spawn)(cmd, args, { cwd: opts.cwd, env: opts.env });
+            const child = (0, node_child_process_1.spawn)(cmd, args, { cwd: opts.cwd, env: opts.env, detached: process.platform !== "win32" });
             let stdout = "";
             let stderr = "";
             let settled = false;
             const timer = opts.timeoutMs
                 ? setTimeout(() => {
-                    child.kill("SIGKILL");
+                    (0, kill_agent_cjs_1.killAgent)(child);
                 }, opts.timeoutMs)
                 : null;
+            const onAbort = () => (0, kill_agent_cjs_1.killAgent)(child);
+            const cleanup = () => {
+                if (timer)
+                    clearTimeout(timer);
+                opts.signal?.removeEventListener("abort", onAbort);
+            };
+            opts.signal?.addEventListener("abort", onAbort, { once: true });
+            if (opts.signal?.aborted)
+                onAbort();
             child.stdout?.on("data", (d) => {
+                opts.onActivity?.();
                 stdout += d.toString();
             });
             child.stderr?.on("data", (d) => {
+                opts.onActivity?.();
                 stderr += d.toString();
             });
             child.on("error", (err) => {
                 if (settled)
                     return;
                 settled = true;
-                if (timer)
-                    clearTimeout(timer);
+                cleanup();
                 reject(err);
             });
             child.on("close", (code) => {
                 if (settled)
                     return;
                 settled = true;
-                if (timer)
-                    clearTimeout(timer);
+                cleanup();
                 resolve({ code: code ?? -1, stdout, stderr });
             });
             if (opts.input !== undefined) {
@@ -252,7 +264,7 @@ class DockerSandbox {
     }
     /** Stops/removes only the container; the session orchestrator owns worktree cleanup. */
     async stop() {
-        await this.runner.run("docker", ["rm", "-f", this.name]).catch(() => undefined);
+        await this.runner.run("docker", ["rm", "-f", this.name], { timeoutMs: 10_000 }).catch(() => undefined);
     }
 }
 exports.DockerSandbox = DockerSandbox;

package/dist/session.cjs

@@ -6,6 +6,7 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.runSession = runSession;
 exports.toEnvMap = toEnvMap;
 const node_path_1 = __importDefault(require("node:path"));
+const node_perf_hooks_1 = require("node:perf_hooks");
 const node_fs_1 = require("node:fs");
 const api_cjs_1 = require("./api.cjs");
 const sandbox_cjs_1 = require("./sandbox.cjs");
@@ -21,6 +22,8 @@ const git_worktree_cjs_1 = require("./git-worktree.cjs");
 const worktree_janitor_cjs_1 = require("./worktree-janitor.cjs");
 const github_pr_cjs_1 = require("./github-pr.cjs");
 const worktree_guard_cjs_1 = require("./worktree-guard.cjs");
+const adapter_watchdog_cjs_1 = require("./adapter-watchdog.cjs");
+const lease_heartbeat_cjs_1 = require("./lease-heartbeat.cjs");
 const adapter_capacity_cjs_1 = require("./adapter-capacity.cjs");
 /** Filename the generated platform MCP config is written under inside the session metadata directory. */
 const MCP_CONFIG_FILENAME = "mcp-config.json";
@@ -205,15 +208,42 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
     let activeAbortController = null;
     let leaseLost = false;
     let leaseGone = false;
+    let sessionClosing = false;
+    let retainWorktree = false;
     let heartbeatInFlight = null;
+    let heartbeatWorkspaceUpdate = null;
     const sessionOutcome = {};
+    const checkpointForRecovery = async (result) => {
+        if (result.checkpointPath || result.retainedWorktreePath)
+            return;
+        try {
+            result.checkpointPath = await gitWorktree.saveCheckpoint();
+            log.warn(`session ${leaseId} stopped for ${leaseLost ? "lease loss" : result.timeoutReason ?? "duration"}; recovery checkpoint: ${result.checkpointPath}`);
+        }
+        catch (error) {
+            retainWorktree = true;
+            result.retainedWorktreePath = gitWorktree.worktreePath;
+            log.warn(`checkpoint failed for ${leaseId}: ${String(error)}; retaining ${gitWorktree.worktreePath} for recovery.`);
+        }
+    };
     const pollLease = () => {
+        if (leaseLost || sessionClosing)
+            return Promise.resolve();
         if (heartbeatInFlight)
             return heartbeatInFlight;
         const request = api
             .heartbeatLease(leaseId, { guidance_after: guidanceCursor })
             .then(async (heartbeat) => {
-            await (0, worktree_janitor_cjs_1.markSessionWorkspaceActive)(workDir);
+            if (sessionClosing)
+                return;
+            heartbeatGuard.renew(heartbeat.expires_at);
+            if (leaseLost)
+                return;
+            heartbeatWorkspaceUpdate = (0, worktree_janitor_cjs_1.markSessionWorkspaceActive)(workDir);
+            await heartbeatWorkspaceUpdate;
+            heartbeatWorkspaceUpdate = null;
+            if (sessionClosing)
+                return;
             guidanceCursor = heartbeat.guidance_cursor ?? guidanceCursor;
             const fresh = (heartbeat.guidance ?? []).filter((entry) => {
                 if (knownGuidanceIds.has(entry.id))
@@ -229,9 +259,16 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
             }
         })
             .catch((err) => {
+            if (leaseLost || sessionClosing)
+                return;
+            if (heartbeatGuard.canRetry(err)) {
+                log.warn(`lease heartbeat failed for ${leaseId}: ${String(err)} — retrying on the next heartbeat while the confirmed lease remains valid.`);
+                return;
+            }
             log.warn(`lease heartbeat failed for ${leaseId}: ${String(err)} — killing session.`);
             leaseLost = true;
             leaseGone = isTerminalLeaseHeartbeatError(err);
+            heartbeatGuard.stop();
             activeAbortController?.abort();
         })
             .finally(() => {
@@ -345,6 +382,12 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
     // session actually produced instead of zeroing it.
     let lastCompletion = null;
     let adapterStarted = false;
+    const heartbeatGuard = new lease_heartbeat_cjs_1.LeaseHeartbeatGuard(claimed.expires_at, () => {
+        log.warn(`confirmed lease expired for ${leaseId} without renewal — killing session.`);
+        leaseLost = true;
+        leaseGone = true;
+        activeAbortController?.abort();
+    });
     const heartbeatTimer = setInterval(() => void pollLease(), config.leaseHeartbeatIntervalMs);
     lifecycle.enteredMainSessionTry = true;
     try {
@@ -370,6 +413,8 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
         const adapter = (0, index_cjs_1.selectAdapter)(runtime);
         const { bin, extraArgs } = adapterCommand(config, runtime);
         const attempts = [];
+        const agentBudgetStartedAt = node_perf_hooks_1.performance.now();
+        const maxRuntimeMs = task.max_runtime_ms ?? config.sessionTimeoutMs;
         let nextPrompt = null;
         let completionRemediationRetries = 0;
         while (true) {
@@ -387,26 +432,33 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
             nextPrompt = null;
             await node_fs_1.promises.writeFile(node_path_1.default.join(workDir, "prompt.md"), runPrompt, "utf8");
             adapterStarted = true;
-            const turnResult = await adapter.run({
-                prompt: runPrompt,
-                mcpConfigPath,
-                bin,
-                extraArgs,
-                model: execution.model,
-                reasoningEffort: execution.reasoning_effort,
-                timeoutMs: config.sessionTimeoutMs,
-                env: sessionEnv,
-                settingsPath: claudeSettingsPath,
-                codexGuardArgs,
-                geminiSandboxMounts: geminiGuardMounts,
-                opencodeHostGuard: runtime === "opencode" && config.sandboxMode === "host" && config.worktreeGuard,
-                // ACP requires an absolute session cwd. Docker mounts the worktree at
-                // this same absolute path, while the other Docker adapters ignore cwd.
-                cwd: gitWorktree.worktreePath,
-                dockerExec: sandbox ? { containerName: sandbox.name, runner: sandbox_cjs_1.nodeCommandRunner } : undefined,
-                signal: activeAbortController.signal,
-            });
+            const remainingRuntimeMs = maxRuntimeMs > 0
+                ? maxRuntimeMs - (node_perf_hooks_1.performance.now() - agentBudgetStartedAt)
+                : 0;
+            const turnResult = maxRuntimeMs > 0 && remainingRuntimeMs <= 0
+                ? { exitCode: null, timedOut: true, timeoutReason: "duration", killedByLeaseLoss: false, stdout: "", stderr: "" }
+                : await (0, adapter_watchdog_cjs_1.runWithActivityWatchdog)(adapter, {
+                    prompt: runPrompt,
+                    mcpConfigPath,
+                    bin,
+                    extraArgs,
+                    model: execution.model,
+                    reasoningEffort: execution.reasoning_effort,
+                    timeoutMs: remainingRuntimeMs,
+                    env: sessionEnv,
+                    settingsPath: claudeSettingsPath,
+                    codexGuardArgs,
+                    geminiSandboxMounts: geminiGuardMounts,
+                    opencodeHostGuard: runtime === "opencode" && config.sandboxMode === "host" && config.worktreeGuard,
+                    // ACP requires an absolute session cwd. Docker mounts the worktree at
+                    // this same absolute path, while the other Docker adapters ignore cwd.
+                    cwd: gitWorktree.worktreePath,
+                    dockerExec: sandbox ? { containerName: sandbox.name, runner: sandbox_cjs_1.nodeCommandRunner } : undefined,
+                    signal: activeAbortController.signal,
+                }, task.idle_timeout_ms ?? config.sessionIdleTimeoutMs ?? 45 * 60_000);
             activeAbortController = null;
+            if (turnResult.timedOut || leaseLost)
+                await checkpointForRecovery(turnResult);
             if (turnResult.model) {
                 execution.model = turnResult.model;
                 executionReport = { ...executionReport, model: turnResult.model };
@@ -424,10 +476,11 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
             if (!leaseGone)
                 await pollLease();
             if (leaseGone) {
+                await checkpointForRecovery(turnResult);
                 log.warn(`session ${leaseId} stopped without completion because its lease is no longer active.`);
                 return sessionOutcome;
             }
-            if (!leaseLost && pendingGuidance.length > 0)
+            if (!leaseLost && !turnResult.timedOut && pendingGuidance.length > 0)
                 continue;
             const result = {
                 ...turnResult,
@@ -602,8 +655,13 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
             .catch((completeErr) => log.warn(`complete() after crash also failed: ${String(completeErr)}`));
     }
     finally {
+        sessionClosing = true;
         clearInterval(heartbeatTimer);
+        heartbeatGuard.stop();
         activeAbortController?.abort();
+        // Drain only an already-started filesystem update, never a stalled network
+        // request. A later heartbeat response sees sessionClosing and does nothing.
+        await Promise.resolve(heartbeatWorkspaceUpdate).catch(() => undefined);
         secrets = {};
         // A sandbox teardown failure must not skip worktree cleanup: a leftover
         // task branch in the shared cache blocks the task's next retry.
@@ -614,8 +672,13 @@ async function runClaimedSession(deps, claimed, sessionId, lifecycle) {
         catch (stopErr) {
             log.warn(`sandbox stop failed during teardown: ${String(stopErr)}`);
         }
-        await gitWorktree.cleanup();
-        await node_fs_1.promises.rm(workDir, { recursive: true, force: true }).catch(() => undefined);
+        if (retainWorktree) {
+            await gitWorktree.retainForRecovery().catch((error) => log.error(`Recovery worktree retained at ${gitWorktree.worktreePath}, but retention setup or credential cleanup failed: ${String(error)}`));
+        }
+        else {
+            await gitWorktree.cleanup();
+            await node_fs_1.promises.rm(workDir, { recursive: true, force: true }).catch(() => undefined);
+        }
     }
     return sessionOutcome;
 }

package/dist/worktree-janitor.cjs

@@ -3,14 +3,16 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.WorktreeJanitor = exports.SESSION_ACTIVITY_FILENAME = void 0;
+exports.WorktreeJanitor = exports.SESSION_RECOVERY_FILENAME = exports.SESSION_ACTIVITY_FILENAME = void 0;
 exports.markSessionWorkspaceActive = markSessionWorkspaceActive;
+exports.hasRecoveryMarker = hasRecoveryMarker;
 const node_path_1 = __importDefault(require("node:path"));
 const node_fs_1 = require("node:fs");
 const logger_cjs_1 = require("./logger.cjs");
 const sandbox_cjs_1 = require("./sandbox.cjs");
 const log = (0, logger_cjs_1.createLogger)("worktree-janitor");
 exports.SESSION_ACTIVITY_FILENAME = ".navarch-active";
+exports.SESSION_RECOVERY_FILENAME = ".navarch-recovery-required";
 /**
  * Refreshes the durable activity marker used by janitors in other runtime
  * processes that happen to share a workspace root.
@@ -63,7 +65,7 @@ class WorktreeJanitor {
             const summary = await this.sweep();
             if (summary.removed > 0 || summary.errors > 0) {
                 log.info(`cleanup complete: scanned=${summary.scanned} removed=${summary.removed} ` +
-                    `active=${summary.skippedActive} fresh=${summary.skippedFresh} errors=${summary.errors}`);
+                    `active=${summary.skippedActive} fresh=${summary.skippedFresh} recovery=${summary.skippedRecovery} errors=${summary.errors}`);
             }
         }
         catch (error) {
@@ -76,6 +78,7 @@ class WorktreeJanitor {
             removed: 0,
             skippedActive: 0,
             skippedFresh: 0,
+            skippedRecovery: 0,
             errors: 0,
         };
         const sessionsRoot = node_path_1.default.resolve(this.options.workspaceRoot, "sessions");
@@ -100,6 +103,10 @@ class WorktreeJanitor {
                 continue;
             }
             try {
+                if (await hasRecoveryMarker(sessionRoot)) {
+                    summary.skippedRecovery += 1;
+                    continue;
+                }
                 const lastActivity = await this.lastActivityAt(sessionRoot);
                 if (this.now() - lastActivity < this.options.staleAfterMs) {
                     summary.skippedFresh += 1;
@@ -171,3 +178,15 @@ exports.WorktreeJanitor = WorktreeJanitor;
 function isMissing(error) {
     return error instanceof Error && "code" in error && error.code === "ENOENT";
 }
+/** Retained work must not be removed by the janitor or a subsequent retry. */
+async function hasRecoveryMarker(sessionRoot) {
+    try {
+        await node_fs_1.promises.access(node_path_1.default.join(sessionRoot, exports.SESSION_RECOVERY_FILENAME));
+        return true;
+    }
+    catch (error) {
+        if (isMissing(error))
+            return false;
+        throw error;
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sagentlab/navarch-runtime",
-  "version": "0.1.46",
+  "version": "0.1.47",
   "description": "Navarch machine-side session manager: claims delivery tasks and runs them through local coding agents.",
   "type": "commonjs",
   "license": "MIT",