@jc_stack/ez-agents 0.1.0-beta.50 → 0.1.0-beta.51

package/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.1.0-beta.51
+
+- Keep native runs alive during handoff-compatible core upgrades. The retiring executor finishes existing work while its replacement accepts new requests; the relay adopts runs without replay and preserves schedule holds, cancellation and settled delivery receipts. Isolated agents migrate once while idle to a separate, unprivileged executor container; later compatible upgrades preserve its running generation. Older runtimes without the handoff contract retain the full native-work drain.
+- Prepare compatible-upgrade workspace and native-client backups before pausing admission or stopping the relay, so large histories do not extend the swap outage. Rollback restores code and configuration without rewinding active native state.
+- Make stop requests reach handed-off native work even while its observer is starting. A failed cancellation write leaves the observer active and reports the failed stop request.
+- Show paused schedules in `/tasks` and task details with a paused label and no next run. Editing a paused schedule keeps it paused.
+- Replace the oldest unused saved AI choice when the saved-choice limit is reached, preserving the selected, default and recently used choices.
+- Use the isolated native client's own catalog for AI selection, so available models such as Sonnet appear without relying on a host catalog.
+
+Beta limits: live fresh-host/reboot acceptance and live compatible upgrades during active native work on the VM and Mac remain pending. Synthetic container lifecycle checks and an idle EzStocks OVH private-candidate upgrade passed; these do not establish live active-run or provider acceptance. Publication does not establish an installed-agent upgrade or authorize provider/domain writes.
+
 ## 0.1.0-beta.50
 
 - Share memory across engines through the workspace. New workspaces seed a short `## Memory` note (durable facts in `memory/`, one per file, indexed in `memory/MEMORY.md`); every engine loads `AGENTS.md`, so Codex and Claude read the same files. When a chat conversation is replaced (`/new`, the New button, web app new, or an AI/provider switch), its engine gets one silent turn in that native session to save durable facts; nothing is delivered and the engine decides what to write.

package/docs/docker-runtime.md

@@ -151,8 +151,7 @@ record, or an adoption failure) is recorded failed and interrupted, which holds
 its schedule until an explicit edit; the host is asked to cancel it. A request
 the host had not claimed before the relay stopped is failed by the host without
 starting. Records older than 24 hours are ignored, and the file is removed on
-every start. Queued runs and deliveries are still dropped, and messages a run
-sends while no relay is up fail and need a retry.
+every start. Queued runs are still dropped. Accepted deliveries are drained before listener shutdown completes; settled receipts for handed-off runs move with the handoff. A connection failure during a declared upgrade swap waits for the relay; an accepted timeout is never retried.
 A `409 Conflict` still requires the operator to stop the competing poller; it
 remains unhealthy after the startup grace. Pending and uncertain deliveries keep
 their existing receipt semantics; executor stdout is not replayed as a reply.
@@ -392,3 +391,14 @@ The isolated executor image includes Node, jq and Python 3 (`python` and
 remain behind installed plugins. Registered command admission waits up to five
 seconds for the short registry lock; lifecycle changes still fail closed, and
 no caller removes an existing lock. Serialize calls when a plugin requires it.
+
+## Isolated executor generations
+
+A handoff-capable supervisor runs the existing native file transport in a separate
+unprivileged per-agent container. It derives workspace/control mounts and the image
+from the owning relay deployment, adds only a read-only executor binding, and gives
+the worker no relay secrets, provider secret environment or Docker socket. Native
+credential homes stay in the existing control volume. Only the active generation
+claims work; retired generations keep cancellation and per-run liveness until
+their last process exits. The supervisor removes their temporary Compose resources.
+The first upgrade from an embedded isolated executor drains before this migration.

package/docs/scheduling.md

@@ -21,7 +21,8 @@ ezenciel-agents-schedule cancel RUN_ID
 ```
 
 Use `--text-file` for longer instructions. `edit ID` replaces the complete schedule
-with a new revision; use the same trigger/name/text flags as `create`.
+with a new revision; use the same trigger/name/text flags as `create`. It keeps
+the paused/enabled state: an edited paused task stays paused until `resume`.
 An edit keeps a future pending occurrence when its trigger is unchanged;
 otherwise the edited rule begins at its next future occurrence. Editing does
 not replay missed work from an old start date. Use `--now` to request a new
@@ -338,9 +339,11 @@ messages, read private sessions, trigger work or certify owning results.
 
 The prompt controls diagnosis, authorized recovery and quiet notification behavior. Inspect prior effects and receipts before retrying anything. A failed review run itself remains visible as a new failure for the next occurrence.
 
-The Telegram Scheduled tasks menu lists enabled schedules that still have a pending
-occurrence or a queued/running occurrence. Finished one-time tasks, paused
-schedules and revisions stopped for review are hidden. Each entry shows the
+The Telegram Scheduled tasks menu lists paused schedules alongside enabled schedules
+that still have a pending occurrence or a queued/running occurrence. Paused tasks
+are labeled Paused and have no next run; an existing queued/running occurrence
+keeps its execution state visible. Finished enabled one-time tasks and enabled
+revisions stopped for review are hidden. Each entry shows the
 frequency with the cron timezone, the saved primary and fallback setups in order
 (CLI, login profile, model and effort), next occurrence in UTC (or queued/running state),
 and the first sentence of its saved invocation

package/docs/upgrades.md

@@ -16,7 +16,12 @@ automatic channel for core and plugins without a saved policy. Existing explicit
 stable or manual policies are preserved. The owner may select either per target.
 The main target and installed plugins version independently.
 
-Core upgrades leave channel admission open while native work runs, then pause it after work is idle and recheck races. Plugin upgrades build while native work continues. Activation closes the existing registry admission lock, drains in-flight plugin command leases, and replaces the plugin under that lock. New native plugin calls wait for admission; an outdated revision fails before provider execution and must be rediscovered. No admitted provider operation is replayed. The shared broker and native sessions remain running. Core upgrades still require the full native-work drain. A plugin update also retains that drain if an operator replaced the broker’s single-file host binding (for example to approve new network pins), because that case still requires broker recreation.
+Handoff-compatible core upgrades build while native work continues and pause admission only for activation. The current native executor retires from claiming new work; the replacement claims new requests while the retiring generation finishes its existing runs. The relay adopts those runs without replay, preserving their schedule holds, cancellation and settled delivery receipts. The supervisor reloads only after native work is idle, so OS service process-group cleanup cannot kill a retiring generation. Plugin upgrades build while native work continues. Activation closes the existing registry admission lock, drains in-flight plugin command leases, and replaces the plugin under that lock. New native plugin calls wait for admission; an outdated revision fails before provider execution and must be rediscovered. No admitted provider operation is replayed. The shared broker and native sessions remain running. Older runtimes without the handoff contract retain the full native-work drain. Isolated agents migrate once while idle to a separate, unprivileged executor container using the same file transport; later relay/runtime upgrades preserve that executor generation. No native CLI or provider credentials move to the host. A plugin update also retains that drain if an operator replaced the broker’s single-file host binding (for example to approve new network pins), because that case still requires broker recreation.
+
+Compatible main upgrades also finish their advisory workspace/native-client
+backup before closing admission or stopping the relay. Large native histories do
+not extend the swap's delivery outage. These live-state copies are best effort;
+rollback restores code and configuration, never the backup over active state.
 
 The owner chooses the saved update policy. The host supervisor owns
 interruption-safe replacement. There is one host service per agent, not a second
@@ -153,7 +158,7 @@ is required. Stable automation excludes prereleases, major changes and 0.x minor
 changes. Beta policy permits compatible prereleases. Downgrades are rejected;
 rollback is a recovery operation for the previous installation only.
 
-`apply` records one durable queued job. **For a core upgrade, the agent must finish its turn after queuing it** so the supervisor can drain native work. A plugin upgrade can complete while that turn continues; only plugin-call admission is held during activation. Calls already executing finish first.
+`apply` records one durable queued job. **For a core upgrade from a runtime without the handoff contract, the agent must finish its turn after queuing it** so the supervisor can drain native work. New compatible runtimes can swap while native work continues. A plugin upgrade can complete while that turn continues; only plugin-call admission is held during activation. Calls already executing finish first.
 One job at a time; an unresolved recovery blocks new upgrades. The agent processes
 multiple requested packages over successive update requests.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jc_stack/ez-agents",
-  "version": "0.1.0-beta.50",
+  "version": "0.1.0-beta.51",
   "private": false,
   "type": "module",
   "description": "A lightweight foundation for persistent business AI assistants using existing AI harnesses, workspaces and plugins.",
@@ -112,7 +112,9 @@
     "protocol": 1,
     "kind": "main",
     "stateSchema": 1,
-    "mainProtocol": 1
+    "mainProtocol": 1,
+    "runHandoff": 1,
+    "containerExecutor": 1
   },
   "exports": {
     "./application-client": "./src/application-client.mjs",

package/src/ai-cli.ts

@@ -5,6 +5,7 @@ import { mkdir, readFile } from 'node:fs/promises'
 import { authProfileStatus, readModels, presetProvider, sameChoice, sameEngine, validateSelection, type AiPreset, type ModelChoice } from './ai.js'
 import { cliHome } from './auth-profile.js'
 import { ControlStore } from './control-state.js'
+import { resolveIsolation } from './isolation.js'
 
 const usage='Usage: ezenciel-agents-ai list | select --cli <installed-cli> [--auth-profile <name>] [--provider <provider>] [--model <model>] [--effort <effort>] | default | profiles | profile add --cli codex|claude --auth-profile <name>'
 const {values,positionals}=parseArgs({allowPositionals:true,options:{help:{type:'boolean'},cli:{type:'string'},'auth-profile':{type:'string'},provider:{type:'string'},model:{type:'string'},effort:{type:'string'}}})
@@ -15,8 +16,9 @@ if(values.help){
 if (!process.env.EZ_CONTROL_DIR) throw new Error('Use this agent’s bound control directory')
 const controlDir=process.env.EZ_CONTROL_DIR
 const authProfile=values['auth-profile']
-const catalog=():Promise<ModelChoice[]>=>readFile(join(controlDir,'host-executor','models.json'),'utf8')
-  .then(text=>JSON.parse(text)).catch(()=>readModels(undefined,undefined,join(controlDir,'cli','codex'),undefined,undefined,undefined,controlDir))
+const catalog=():Promise<ModelChoice[]>=>(process.env.EZ_AI_CATALOG==='host'||(process.env.EZ_AI_CATALOG===undefined&&resolveIsolation(process.env)==='host-capable'))
+  ? readFile(join(controlDir,'host-executor','models.json'),'utf8').then(text=>JSON.parse(text))
+  : readModels(undefined,undefined,join(controlDir,'cli','codex'),undefined,undefined,undefined,controlDir)
 if(positionals[0]==='list')console.log(JSON.stringify(await catalog()))
 else if(positionals[0]==='profiles')console.log(JSON.stringify(await authProfileStatus(controlDir)))
 else if(positionals[0]==='profile' && positionals[1]==='add'){

package/src/control-state.ts

@@ -763,10 +763,16 @@ export class ControlStore {
       await requireControlGuard(state, guard)
       if (!state.ai) throw new Error('AI settings not initialized')
       // Auto-detected client defaults come and go with installed clients; only
-      // owner-saved choices count toward the cap.
-      if (state.ai.presets.filter((p) => !p.id.startsWith('detected_')).length >= 12 && !state.ai.presets.some((p) => p.id === preset.id))
-        throw new Error('Keep it small: at most 12 saved AIs.')
-      state.ai.presets = [...state.ai.presets.filter((p) => p.id !== preset.id), persistedPreset(preset)]
+      // owner-saved choices count toward the cap. Conversations keep their own
+      // preset copy, so at the cap the oldest unreferenced choice makes room.
+      const ai = state.ai
+      const saved = ai.presets.filter((p) => !p.id.startsWith('detected_'))
+      if (saved.length >= 12 && !ai.presets.some((p) => p.id === preset.id)) {
+        const kept = new Set([ai.selectedId, ai.defaultId, ...(ai.recentIds ?? [])])
+        const oldest = saved.find((p) => !kept.has(p.id))!
+        ai.presets = ai.presets.filter((p) => p.id !== oldest.id)
+      }
+      ai.presets = [...ai.presets.filter((p) => p.id !== preset.id), persistedPreset(preset)]
       await this.writeState(state)
     })
   }

package/src/delivery-socket-client.mjs

@@ -78,7 +78,7 @@ const readEndpoint = async controlDir => {
   } catch { return null }
 }
 
-export const callDeliverySocket = async (socketPath, op, timeoutMs = 130_000) => {
+const callOnce = async (socketPath, op, timeoutMs) => {
   const id = `${Date.now().toString(36)}_${Math.floor(Math.random() * 0xffffff).toString(16)}`
   try {
     return await exchange(() => createConnection(socketPath), { id, ...op }, timeoutMs, socketPath)
@@ -104,3 +104,17 @@ export const deliverySocketAlive = async (socketPath, timeoutMs = 2000) => {
     return false
   }
 }
+
+// Only retry connection failures during a declared runtime swap. No frame was
+// accepted in that case. Timeouts, provider failures and uncertain writes fail.
+export const callDeliverySocket = async (socketPath, op, timeoutMs = 130_000) => {
+  const until=Date.now()+120000;
+  for(;;){
+    try{return await callOnce(socketPath,op,timeoutMs);}
+    catch(error){
+      if(!error?.connectFailure||op.op!=='enqueue'||Date.now()>=until)throw error;
+      try{await readFile(path.join(path.dirname(socketPath),'upgrade-pause.json'),'utf8');}catch{throw error;}
+      await new Promise(resolve=>setTimeout(resolve,250));
+    }
+  }
+};

package/src/delivery-socket.ts

@@ -83,7 +83,7 @@ export const authorizeRun = async (controlDir: string, runId: string): Promise<R
 const encode = (value: unknown): string => `${JSON.stringify(value)}\n`
 
 export type DeliveryEndpoint = { host: '127.0.0.1'; port: number; token: string }
-export type DeliveryServeOptions = { tcpPort?: number }
+export type DeliveryServeOptions = { tcpPort?: number; beforeClose?: () => Promise<void> }
 const endpointFile = 'delivery-endpoint.json'
 const tokensMatch = (candidate: unknown, token: string): boolean =>
   typeof candidate === 'string' && candidate.length === token.length && timingSafeEqual(Buffer.from(candidate), Buffer.from(token))
@@ -94,6 +94,8 @@ export const serveDeliverySocket = async (
   options: DeliveryServeOptions = {},
 ): Promise<{ socketPath: string; endpoint: DeliveryEndpoint | null; stop: () => Promise<void> }> => {
   const socketPath = deliverySocketPath(controlDir)
+  let closing=false
+  const admissions=new Set<Promise<void>>()
   await rm(socketPath, { force: true }).catch(() => {})
   const attach = (socket: Socket, token: string | null): void => {
     let buffer = ''
@@ -105,7 +107,8 @@ export const serveDeliverySocket = async (
         const line = buffer.slice(0, newline)
         buffer = buffer.slice(newline + 1)
         if (!line.trim()) continue
-        void (async () => {
+        let admission=false
+        const work=(async () => {
           let id: unknown = null
           try {
             const envelope = JSON.parse(line) as { id?: unknown; token?: unknown; op?: unknown; payload?: unknown }
@@ -116,12 +119,19 @@ export const serveDeliverySocket = async (
             // re-verifies owner authority server-side.
             if (token !== null && !tokensMatch(envelope.token, token)) { socket.destroy(); return }
             const { token: _token, ...request } = envelope
+            if(request.op==='enqueue'){
+              if(closing)throw Error('Delivery relay is shutting down; message was not accepted')
+              admission=true
+            }
             const result = await handle(request as DeliverySocketOp)
             socket.write(encode({ id, ok: true, result: result ?? null }))
           } catch (error) {
             socket.write(encode({ id, ok: false, error: error instanceof Error ? error.message : String(error ?? 'Unknown error') }))
           }
         })()
+        // Only finite admission handlers are a drain barrier; a receipt wait
+        // completes after the relay's delivery pump flushes the accepted items.
+        if(admission){admissions.add(work);void work.finally(()=>admissions.delete(work))}
       }
     })
   }
@@ -176,9 +186,13 @@ export const serveDeliverySocket = async (
     socketPath,
     endpoint,
     stop: async () => {
-      await new Promise<void>((resolve) => server.close(() => resolve()))
+      closing=true
+      const closed=Promise.all([new Promise<void>((resolve) => server.close(() => resolve())),
+        ...(tcpServer ? [new Promise<void>((resolve) => tcpServer!.close(() => resolve()))] : [])])
+      await Promise.all(admissions)
+      await options.beforeClose?.()
+      await closed
       await rm(socketPath, { force: true }).catch(() => {})
-      if (tcpServer) await new Promise<void>((resolve) => tcpServer.close(() => resolve()))
       if (tcpEndpoint) await rm(path.join(controlDir, endpointFile), { force: true }).catch(() => {})
     },
   }

package/src/executor.ts

@@ -1,7 +1,8 @@
 import { executionDefaults } from './model-policy.js'
 import { startTaskExecutor } from './task-executor.js'
 import { deliverySocketPath } from './delivery-socket.js'
-import { mkdtemp, rm, writeFile, mkdir, symlink, readFile } from 'node:fs/promises'
+import { mkdtemp, rm, writeFile, mkdir, symlink, readFile, rename } from 'node:fs/promises'
+import { randomUUID } from 'node:crypto'
 import { accessSync, openSync, closeSync, constants as fsConstants } from 'node:fs'
 import { tmpdir, homedir } from 'node:os'
 import path from 'node:path'
@@ -13,9 +14,17 @@ import { DESKTOP_UNAVAILABLE } from './desktop-bridge.js'
 import { macosWorkspaceProfile, workspaceSiblingDenies } from './workspace-confine.js'
 import { Scripts, type ScriptRunRef } from './scripts.js'
 import { claudeAuthEnvironment, cliHome, provisionedHome } from './auth-profile.js'
+import { isHostRunId } from './host-executor-protocol.js'
+
+const hostCancelFiles = new WeakMap<ChildProcess, string>()
+const hostCancelFile = (controlDir: string, runId: string): string => {
+  if (!isHostRunId(runId)) throw new Error('Invalid host executor run ID')
+  return path.join(controlDir, 'host-executor', runId + '.cancel')
+}
 
 export type ExecutorOptions = {
   repairEnabled?: boolean
+  catalogTransport?: 'host' | 'local'
   workspace: string
   timeoutMs: number
   runId: string
@@ -79,7 +88,7 @@ export const executorEnvironment = (environment: NodeJS.ProcessEnv = process.env
 }
 
 export const executorJobEnv = (
-  options: Pick<ExecutorOptions, 'runId' | 'controlDir' | 'binDir' | 'toolsHome' | 'repairEnabled'> & { workspace?: string },
+  options: Pick<ExecutorOptions, 'runId' | 'controlDir' | 'binDir' | 'toolsHome' | 'repairEnabled' | 'catalogTransport'> & { workspace?: string },
   environment: NodeJS.ProcessEnv = process.env,
 ): NodeJS.ProcessEnv => {
   const base = executorEnvironment(environment)
@@ -88,6 +97,7 @@ export const executorJobEnv = (
     ...base,
     PATH: pathValue,
     EZ_RUN_ID: options.runId,
+    EZ_AI_CATALOG: options.catalogTransport ?? (environment.EZ_EXECUTOR_TRANSPORT==='host' ? 'host' : 'local'),
     EZ_CONTROL_DIR: options.controlDir,
     ...(options.workspace ? { EZ_AGENT_WORKSPACE: options.workspace } : {}),
     EZ_DELIVERY_SOCKET: deliverySocketPath(options.controlDir),
@@ -340,22 +350,33 @@ export const executorInvocation = (command: string, args: string[]) => {
     : { command, args }
 }
 
+// The isolated worker uses the same file transport as the host worker, but
+// stays in its own container. A published worker binding must never fall back
+// to local execution when its heartbeat expires.
+export const usesFileTransport = async (controlDir: string, env: NodeJS.ProcessEnv = process.env): Promise<boolean> => {
+  if (env.EZ_CONTAINER_EXECUTOR === '1') return false
+  if (env.EZ_EXECUTOR_TRANSPORT === 'host') return true
+  try { return JSON.parse(await readFile(path.join(controlDir,'host-executor/heartbeat.json'),'utf8')).containerExecutor === true }
+  catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; return false }
+}
+
 export const startExecutorJob = async (
   texts: string[],
   options: ExecutorOptions,
 ): Promise<{ child: ChildProcess; cleanup: () => Promise<void>; stdout: string }> => {
-  if (options.script && process.env.EZ_EXECUTOR_TRANSPORT !== 'host') return startScriptJob(options as ExecutorOptions & { script: ScriptLaunch })
+  const host = await usesFileTransport(options.controlDir)
+  const cancelFile = host ? hostCancelFile(options.controlDir, options.runId) : undefined
+  if (options.script && !host) return startScriptJob(options as ExecutorOptions & { script: ScriptLaunch })
   options = executionDefaults(executorKey(options.cli), options)
   // Named credential homes are owner-provisioned; fail closed before any launch state.
   if (options.authProfile !== undefined) await provisionedHome(options.controlDir, executorKey(options.cli), options.authProfile)
   // Routing is caller-owned: a restricted task run goes to the task runner
   // unless the host transport must ship it across the boundary first.
-  if (options.taskRun && process.env.EZ_EXECUTOR_TRANSPORT !== 'host') return startTaskExecutor(options)
+  if (options.taskRun && !host) return startTaskExecutor(options)
   if (options.nativeSession && !/^[a-zA-Z0-9_-]+$/.test(options.runId)) throw new Error('Invalid native task run ID')
   if (options.codexSandbox !== undefined && (options.codexSandbox !== 'external' || process.env.EZ_EXECUTOR_TRANSPORT !== 'local' || options.taskRun || executorKey(options.cli) !== 'codex')) throw new Error('External Codex sandbox requires an owner-authorized native local run')
   const outputDirectory = await mkdtemp(path.join(tmpdir(), 'ezenciel-agents-'))
   const key = executorKey(options.cli)
-  const host = process.env.EZ_EXECUTOR_TRANSPORT === 'host'
   const gui = !host && key === 'codex-gui'
   const nativeSession = !host && key === 'codex' && options.nativeSession === true
   // Literal input plus the caller-provided trigger line. No coaching,
@@ -458,6 +479,7 @@ export const startExecutorJob = async (
     stdio: ['pipe', 'pipe', 'pipe'],
     detached: process.platform !== 'win32',
   })
+  if (cancelFile) hostCancelFiles.set(child, cancelFile)
   await new Promise<void>((resolve, reject) => {
     child.once('spawn', resolve)
     child.once('error', reject)
@@ -502,9 +524,11 @@ export const startGrokJob = startExecutorJob
 
 // Watch a host-transport run a previous relay submitted; it is never resubmitted.
 export const attachHostJob = async (controlDir: string, runId: string): Promise<ChildProcess> => {
+  const cancelFile = hostCancelFile(controlDir, runId)
   const invocation = executorInvocation(process.execPath, ['--import', fileURLToPath(new URL('../node_modules/tsx/dist/loader.mjs', import.meta.url)),
     fileURLToPath(new URL('./host-executor-client.ts', import.meta.url)), controlDir, runId, '--attach'])
   const child = spawn(invocation.command, invocation.args, { env: executorEnvironment(), stdio: ['ignore', 'pipe', 'pipe'], detached: process.platform !== 'win32' })
+  hostCancelFiles.set(child, cancelFile)
   await new Promise<void>((resolve, reject) => { child.once('spawn', resolve); child.once('error', reject) })
   return child
 }
@@ -552,11 +576,27 @@ export const nativeSessionId = (cli: string, line: string): string | undefined =
   } catch {}
 }
 
-const terminating = new WeakSet<ChildProcess>()
-export const terminateJob = (child: ChildProcess, inspect = processSnapshot): void => {
-  if (!child.pid || child.exitCode !== null || child.signalCode !== null || terminating.has(child)) return
-  terminating.add(child)
-  void (async () => {
+const terminating = new WeakMap<ChildProcess, Promise<boolean>>()
+export const terminateJob = (child: ChildProcess, inspect = processSnapshot): Promise<boolean> => {
+  const pending = terminating.get(child)
+  if (pending) return pending
+  if (!child.pid || child.exitCode !== null || child.signalCode !== null) return Promise.resolve(true)
+  const work = (async () => {
+    const cancelFile = hostCancelFiles.get(child)
+    if (cancelFile) {
+      // The observer may still be loading, or unable to handle a signal. Commit
+      // the native worker's stop request before terminating that observer.
+      const temporary = cancelFile + '.' + process.pid + '.' + randomUUID() + '.tmp'
+      try {
+        await mkdir(path.dirname(cancelFile), { recursive: true, mode: 0o700 })
+        await writeFile(temporary, '', { mode: 0o600, flag: 'wx' })
+        await rename(temporary, cancelFile)
+      } catch (error) {
+        await rm(temporary, { force: true }).catch(() => {})
+        console.error('Cannot request host executor cancellation; observer remains active', error)
+        return false
+      }
+    }
     const targets = new Set([child.pid!])
     // Native tool terminals can start separate process groups. Capture ancestry
     // before stopping the CLI, while those children still have their parent.
@@ -584,5 +624,9 @@ export const terminateJob = (child: ChildProcess, inspect = processSnapshot): vo
       void processSnapshot().then(current => signal(matchingProcessIds(identities, current), 'SIGKILL'))
         .catch(error => console.error('Cannot inspect executor descendants for escalation', error))
     }, 3000)
+    return true
   })()
+  terminating.set(child, work)
+  void work.then(stopped => { if (!stopped) terminating.delete(child) })
+  return work
 }

package/src/host-executor-client.ts

@@ -50,7 +50,10 @@ try {
     // Drain completion first. A brief missing file at the shared-filesystem
     // boundary must not cancel a healthy job or hide its terminal result.
     try {
-      const heartbeat = JSON.parse(await readFile(path.join(directory,'heartbeat.json'),'utf8'))
+      const claimed = await stat(base+'.running.json').then(()=>true,error=>{if(error.code==='ENOENT')return false;throw error})
+      const heartbeat = claimed
+        ? {at:(await stat(base+'.events')).mtimeMs}
+        : JSON.parse(await readFile(path.join(directory,'heartbeat.json'),'utf8'))
       if (!Number.isFinite(heartbeat.at)) throw new Error('Invalid host CLI heartbeat')
       lastHeartbeat = heartbeat.at
     } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error }

package/src/host-executor.ts

@@ -3,7 +3,7 @@ import { Tasks } from './tasks.js'
 import { ControlStore } from './control-state.js'
 import { authorizeRun, readRun } from './delivery-socket.js'
 import { runPromptSuffix } from './prompt-suffix.js'
-import { mkdir, readFile, writeFile, readdir, rename, rm, appendFile, realpath } from 'node:fs/promises'
+import { mkdir, readFile, writeFile, readdir, rename, rm, appendFile, realpath, utimes, stat } from 'node:fs/promises'
 import path from 'node:path'
 import { isHostRunId } from './host-executor-protocol.js'
 import { fileURLToPath } from 'node:url'
@@ -42,13 +42,16 @@ export const refreshAgentCatalog = async (agent: HostBinding,
 const processStart = async (pid:number) => (await processSnapshot()).get(pid)?.birth
 
 export const serveHostExecutor = async (installation: HostInstallation, signal: AbortSignal, launch = startExecutorJob,
-  readCatalog?: (agent: HostBinding) => Promise<ModelChoice[]>) => {
+  readCatalog?: (agent: HostBinding) => Promise<ModelChoice[]>, retire?: AbortSignal) => {
   if (installation.isolation !== undefined) parseIsolationClass(installation.isolation)
-  if (installation.isolation === 'isolated') throw new Error('Isolated agents run the native CLI in the relay; do not start host transport')
+  if (installation.isolation === 'isolated' && process.env.EZ_CONTAINER_EXECUTOR !== '1') throw new Error('Isolated agents run the native CLI in the relay; do not start host transport')
   resolveExecutor(installation.cli)
   if (new Set(installation.agents.map(a=>a.workspace)).size !== installation.agents.length ||
       new Set(installation.agents.map(a=>a.controlDir)).size !== installation.agents.length)
     throw new Error('Each agent requires a separate workspace and control directory')
+  const containerId = process.env.EZ_CONTAINER_EXECUTOR === '1' ? process.env.HOSTNAME : undefined
+  const hostStarted = await processStart(process.pid)
+  let retiring = false
   const active = new Map<string, ChildProcess>()
   // A fallback attempt reuses its run ID; claim it only after the prior attempt's cleanup.
   const claimed = new Set<string>()
@@ -74,9 +77,29 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
   const freshCatalog = readCatalog ?? catalog
   const beat = async (agent: HostBinding) => {
     const directory = path.join(agent.controlDir, 'host-executor')
-    await writeFile(path.join(directory, 'heartbeat.tmp'), JSON.stringify({ at: Date.now(), pid: process.pid, version: packageVersion, platform: process.platform, arch: process.arch, plugins: await installedPluginVersions(agent.toolsHome) }), { mode: 0o600 })
+    await writeFile(path.join(directory, 'heartbeat.tmp'), JSON.stringify({ at: Date.now(), pid: process.pid, version: packageVersion, platform: process.platform, arch: process.arch, ...(process.env.EZ_CONTAINER_EXECUTOR === '1' ? {containerExecutor:true,containerId} : {}), plugins: await installedPluginVersions(agent.toolsHome) }), { mode: 0o600 })
     await rename(path.join(directory, 'heartbeat.tmp'), path.join(directory, 'heartbeat.json'))
   }
+  const recoverClaims = async (agent: HostBinding) => {
+    const directory=path.join(agent.controlDir,'host-executor')
+    for (const file of await readdir(directory)) if (file.endsWith('.running.json') && isHostRunId(file.slice(0,-13))) {
+      const base=path.join(directory,file.slice(0,-13))
+      if (claimed.has(base)) continue
+      let prior: {pid?:number;started?:string;hostPid?:number;hostStarted?:string;containerId?:string} | undefined
+      try { prior=JSON.parse(await readFile(base+'.process.json','utf8')) } catch(error) { if ((error as NodeJS.ErrnoException).code!=='ENOENT') throw error }
+      if(prior?.containerId && prior.containerId!==containerId){
+        const touched=await stat(base+'.events').then(s=>s.mtimeMs,()=>0)
+        if(Date.now()-touched<15000)continue
+      }
+      if (!prior?.containerId && prior?.hostPid && prior.hostStarted && await processStart(prior.hostPid)===prior.hostStarted) continue
+      if (!prior?.containerId && prior?.pid && await processStart(prior.pid)) {
+        if (!prior.hostPid) throw Error('Previous host CLI is still running; stop it before recovery')
+        if (prior.started && await processStart(prior.pid)===prior.started) terminateJob({pid:prior.pid,exitCode:null,signalCode:null} as ChildProcess)
+      }
+      await appendFile(base+'.events',JSON.stringify({stream:'exit',code:1})+'\n',{mode:0o600})
+      await rm(base+'.process.json',{force:true});await rm(base+'.running.json',{force:true})
+    }
+  }
   try {
     for (const agent of installation.agents) {
       if (![agent.workspace,agent.controlDir,agent.binDir].every(path.isAbsolute)) throw new Error('Host bindings require absolute paths')
@@ -89,7 +112,7 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
         if (!Array.isArray(agent.additionalWorkspaces) || agent.additionalWorkspaces.some(workspace=>typeof workspace!=='string' || !path.isAbsolute(workspace))) throw new Error('Additional workspaces require absolute paths')
         additionalWorkspaces.set(agent,await Promise.all(agent.additionalWorkspaces.map(workspace=>realpath(workspace))))
       }
-      if (agent.toolsHome) {
+      if (agent.toolsHome && !containerId) {
         if (!path.isAbsolute(agent.toolsHome)) throw new Error('Plugin registry binding requires an absolute path')
         const config=JSON.parse(await readFile(path.join(agent.toolsHome,'config.json'),'utf8'))
         if (config.schemaVersion!==1 || config.workspace!==await realpath(agent.workspace)) throw new Error('Plugin registry belongs to another workspace')
@@ -99,6 +122,7 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
       const lock=path.join(directory,'worker.lock')
       try {
         const prior=JSON.parse(await readFile(lock,'utf8'))
+        if(prior.containerId&&prior.containerId!==containerId)throw Error('Previous isolated worker still owns the claim lock')
         const currentStart=await processStart(prior.pid)
         if (!prior.started || !currentStart || prior.started===currentStart) {
           try { process.kill(prior.pid,0); throw new Error('Host executor already running') }
@@ -107,34 +131,29 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
         await rm(lock)
       }
       catch(error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error }
-      await writeFile(lock,JSON.stringify({pid:process.pid,started:await processStart(process.pid)}),{mode:0o600,flag:'wx'})
+      await writeFile(lock,JSON.stringify({pid:process.pid,started:await processStart(process.pid),...(containerId ? {containerId} : {})}),{mode:0o600,flag:'wx'})
       locks.push(lock)
       const models = await freshCatalog(agent)
       await new ControlStore(agent.controlDir, 900000).normalizeProviderBindings(models)
       const modelsFile=path.join(directory,'models.json')
       await writeFile(modelsFile+'.tmp',JSON.stringify(models),{mode:0o600});await rename(modelsFile+'.tmp',modelsFile)
-      // A host crash is terminal for a claimed job. Never replay an action.
-      for (const file of await readdir(directory)) if (file.endsWith('.running.json')) {
-        const base=path.join(directory,file.slice(0,-13))
-        try {
-          const prior=JSON.parse(await readFile(base+'.process.json','utf8'))
-          try { process.kill(prior.pid,0); throw new Error('Previous host CLI is still running; stop it before recovery') }
-          catch(error) { if ((error as NodeJS.ErrnoException).code !== 'ESRCH') throw error }
-        } catch(error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error }
-        await rm(base+'.process.json',{force:true})
-        await appendFile(base+'.events',JSON.stringify({stream:'exit',code:1})+'\n',{mode:0o600})
-        await rm(path.join(directory,file))
-      }
-      if (agent.toolsHome) await (await import('./plugins/workspace-lease.mjs')).recoverNativeLease(agent.toolsHome)
+      await recoverClaims(agent)
+      if (agent.toolsHome && !containerId) await (await import('./plugins/workspace-lease.mjs')).recoverNativeLease(agent.toolsHome)
     }
     let catalogAt=Date.now()
     const catalogSeen=new Map<HostBinding,string>()
     while (!signal.aborted) {
       const parent=Number(process.env.EZ_HOST_SUPERVISOR_PID)
-      if(parent) { try { process.kill(parent,0) } catch { break } }
+      if(parent && !retiring) { try { process.kill(parent,0) } catch { break } }
+      if (retire?.aborted && !retiring) {
+        retiring=true
+        await catalogRefresh
+        for (const lock of locks.splice(0)) await rm(lock,{force:true})
+      }
+      if (retiring && !tasks.size) break
       // Keep transport liveness independent of native catalog I/O.
-      for (const agent of installation.agents) await beat(agent)
-      if(!catalogRefresh && Date.now()-catalogAt>30000){
+      if (!retiring) for (const agent of installation.agents) await beat(agent)
+      if(!retiring && !catalogRefresh && Date.now()-catalogAt>30000){
         catalogAt=Date.now()
         catalogRefresh=(async()=>{
           for(const agent of installation.agents){
@@ -149,6 +168,8 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
       }
       for (const agent of installation.agents) {
         const directory=path.join(agent.controlDir,'host-executor')
+        if (retiring) continue
+        await recoverClaims(agent)
         for (const file of await readdir(directory)) {
           if (!file.endsWith('.request.json') || !isHostRunId(file.slice(0,-13))) continue
           const id=file.slice(0,-13)
@@ -168,9 +189,9 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
           let workspace = activeWorkspaces.get(agent)
           if (!scheduled && workspace && [...workspace.lanes.values()].includes(false)) continue
           if (!workspace) {
-            const release = agent.toolsHome
+            const release = agent.toolsHome && !containerId
               ? await (await import('./plugins/workspace-lease.mjs')).workspaceLease(agent.toolsHome,{kind:'native',runId:id}) : undefined
-            if (agent.toolsHome && !release) continue
+            if (agent.toolsHome && !containerId && !release) continue
             workspace = { lanes: new Map(), release }
             activeWorkspaces.set(agent, workspace)
           }
@@ -184,7 +205,11 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
               await workspace.release?.()
             }
           }
-          try { await rename(base+'.request.json',base+'.running.json') }
+          try {
+            await writeFile(base+'.process.json',JSON.stringify({hostPid:process.pid,hostStarted,...(containerId ? {containerId} : {})}),{mode:0o600})
+            await appendFile(base+'.events','',{mode:0o600})
+            await rename(base+'.request.json',base+'.running.json')
+          }
           catch (error) { claimed.delete(base); await releaseWorkspace(); throw error }
           const task=(async()=>{
             let job: Awaited<ReturnType<typeof startExecutorJob>> | undefined
@@ -205,7 +230,7 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
               // Script occurrences come from the authorized run record, never the request.
               if (run?.script) {
                 if (!run.scheduled) throw new Error('Script runs require a scheduled occurrence')
-                const options:ExecutorOptions={workspace:agent.workspace,controlDir:agent.controlDir,binDir:agent.binDir,toolsHome:agent.toolsHome,sharedWorkspace,additionalWorkspaces:additionalWorkspaces.get(agent),
+                const options:ExecutorOptions={catalogTransport:containerId?'local':'host',workspace:agent.workspace,controlDir:agent.controlDir,binDir:agent.binDir,toolsHome:agent.toolsHome,sharedWorkspace,additionalWorkspaces:additionalWorkspaces.get(agent),
                   runId:path.basename(base),timeoutMs:0,repairEnabled:opts.repairEnabled,
                   script:{...run.script,scheduleId:run.scheduled.id,scheduleRevision:run.scheduled.revision,dueAt:run.scheduled.dueAt}}
                 job=await launch(request.texts,options)
@@ -237,14 +262,18 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
                 // transient native-client probe failure must not reject a model
                 // that the host just advertised to the relay and application.
                 if (cli !== installation.cli || opts.authProfile !== undefined) await validateSelection({id:'selected',name:'Selected model',cli,provider:opts.provider,authProfile:opts.authProfile,model,effort:opts.effort},JSON.parse(await readFile(path.join(directory,'models.json'),'utf8')))
-                const options:ExecutorOptions={workspace:agent.workspace,controlDir:agent.controlDir,binDir:agent.binDir,toolsHome:agent.toolsHome,sharedWorkspace,additionalWorkspaces:additionalWorkspaces.get(agent),cli,
+                const options:ExecutorOptions={catalogTransport:containerId?'local':'host',workspace:agent.workspace,controlDir:agent.controlDir,binDir:agent.binDir,toolsHome:agent.toolsHome,sharedWorkspace,additionalWorkspaces:additionalWorkspaces.get(agent),cli,
                   runId:path.basename(base),timeoutMs:0,repairEnabled:opts.repairEnabled,
                   sessionId:opts.sessionId,isResume:opts.isResume,model,effort:opts.effort,provider:opts.provider,authProfile:opts.authProfile,codexAutoCompactTokens:opts.codexAutoCompactTokens,codexProvider:provider,
                   promptSuffix:runPromptSuffix(run),taskRun:Boolean(run?.taskId),nativeSession:Boolean(run?.scheduled)}
                 job=await launch(request.texts,options)
               }
               active.set(base,job.child)
-              await writeFile(base+'.process.json',JSON.stringify({pid:job.child.pid}),{mode:0o600})
+              // Observe completion before process metadata I/O: a fast CLI may
+              // already have exited when launch returns, or while that I/O waits.
+              const closed=job.child.exitCode!==null||job.child.signalCode!==null
+                ? Promise.resolve(job.child.exitCode??1)
+                : new Promise<number>(resolve=>job!.child.once('close',code=>resolve(code??1)))
               if(signal.aborted)terminateJob(job.child)
               // Scheduled Claude streams every event; forward only the startup summary.
               const startup = observeClaude ? startupObserver() : undefined
@@ -255,8 +284,9 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
                 if (startup.summary() !== summary) { summary = startup.summary(); emit({stream:'stdout',text:summary+'\n'}) }
               })
               job.child.stderr?.on('data',chunk=>emit({stream:'stderr',text:chunk.toString()}))
-              cancellation=setInterval(()=>{void readFile(base+'.cancel').then(()=>terminateJob(job!.child)).catch(()=>{})},250)
-              const code=await new Promise<number>(resolve=>job!.child.once('close',code=>resolve(code??1)))
+              await writeFile(base+'.process.json',JSON.stringify({pid:job.child.pid,started:job.child.pid ? await processStart(job.child.pid) : undefined,hostPid:process.pid,hostStarted,...(containerId ? {containerId} : {})}),{mode:0o600})
+              cancellation=setInterval(()=>{void readFile(base+'.cancel').then(()=>terminateJob(job!.child)).catch(()=>{}); const now=new Date();void utimes(base+'.events',now,now).catch(()=>{})},250)
+              const code=await closed
               if (cancellation) clearInterval(cancellation)
               await job.cleanup()
               job = undefined
@@ -290,8 +320,9 @@ export const serveHostExecutor = async (installation: HostInstallation, signal:
 }
 
 if (process.argv[1] && path.resolve(process.argv[1])===fileURLToPath(import.meta.url)) {
-  const abort=new AbortController()
+  const abort=new AbortController(),retire=new AbortController()
+  process.once('SIGUSR2',()=>retire.abort())
   for(const signal of ['SIGTERM','SIGINT'] as const)process.once(signal,()=>abort.abort())
   const config=JSON.parse(await readFile(process.argv[2],'utf8'))
-  await serveHostExecutor(config,abort.signal)
+  await serveHostExecutor(config,abort.signal,undefined,undefined,retire.signal)
 }