@agfpd/iapeer 0.4.63 → 0.4.64

package/docs/15-fleet-api.md

@@ -106,9 +106,9 @@ GET /fleet/v1/events?replay=50
 - A comment `: connected` marks the transition from replay to live; `: hb` comments every ~15 s keep the connection alive.
 - Latency: live events surface within ~0.5 s (log-tail poll).
 
-Current `ev` vocabulary (grows over time — see obligation 1): `wake`, `supervise` (with `action`: `reaped-idle` = parked, `reaped-ephemeral`, `dead`→fresh/resume classification, `eager-orphan-fresh`, `skipped-error`…), `delivery`, `topic-note`, `ephemeral-drain`, `attach` / `attach-end`, `hosted-deliver`, `pty-host-failed`, `supervise-error`, `composer-queue-failed-notify`, `memory-provision`, `session-exit` (from `exits.log`, with `dead_status`/`dead_signal`).
+Current `ev` vocabulary (grows over time — see obligation 1): `wake`, `supervise` (with `action`: `reaped-idle` = parked, `reaped-ephemeral`, `dead`→fresh/resume classification, `eager-orphan-fresh`, `skipped-error`…), `stopped` / `started` (the stop/start verbs — an operator changed a peer's ✕/○ state; `action` is `stopped`/`started` for warm runtimes, `bootout`/`bootstrap` for always-on, with `reason` on failure), `delivery`, `topic-note`, `ephemeral-drain`, `attach` / `attach-end`, `hosted-deliver`, `pty-host-failed`, `supervise-error`, `composer-queue-failed-notify`, `memory-provision`, `session-exit` (from `exits.log`: `cause=child-exit` with `dead_status`/`dead_signal` when the session's process died on its own, `cause=shutdown-sigterm`/`shutdown-sigint` when the supervisor was torn down — reap/stop/new), `supervisor-uncaught` (a survivable supervisor fault left as forensics; the session keeps serving).
 
-What the headline states look like on the stream: a peer **waking** is `ev=wake`; a peer **parked to sleep** is `ev=supervise action=reaped-idle`; a **death** is `ev=session-exit` (immediately, from the supervisor) followed by the daemon's classification on its next pass.
+What the headline states look like on the stream: a peer **waking** is `ev=wake`; a peer **parked to sleep** is `ev=supervise action=reaped-idle`; an operator **stop/start** is `ev=stopped` / `ev=started`; a **death** is `ev=session-exit` (immediately, from the supervisor) followed by the daemon's classification on its next pass.
 
 ## Commands
 

package/docs/ru/15-fleet-api.md

@@ -106,9 +106,9 @@ GET /fleet/v1/events?replay=50
 - Комментарий `: connected` отмечает переход от replay к live; комментарии `: hb` каждые ~15 с держат соединение.
 - Латентность: живые события всплывают в пределах ~0.5 с (поллинг хвоста лога).
 
-Текущий словарь `ev` (растёт со временем — см. обязанность 1): `wake`, `supervise` (с `action`: `reaped-idle` = парковка, `reaped-ephemeral`, классификация `dead`→fresh/resume, `eager-orphan-fresh`, `skipped-error`…), `delivery`, `topic-note`, `ephemeral-drain`, `attach` / `attach-end`, `hosted-deliver`, `pty-host-failed`, `supervise-error`, `composer-queue-failed-notify`, `memory-provision`, `session-exit` (из `exits.log`, с `dead_status`/`dead_signal`).
+Текущий словарь `ev` (растёт со временем — см. обязанность 1): `wake`, `supervise` (с `action`: `reaped-idle` = парковка, `reaped-ephemeral`, классификация `dead`→fresh/resume, `eager-orphan-fresh`, `skipped-error`…), `stopped` / `started` (verb'ы stop/start — оператор изменил ✕/○-состояние пира; `action` — `stopped`/`started` для warm-рантаймов, `bootout`/`bootstrap` для always-on, с `reason` при неудаче), `delivery`, `topic-note`, `ephemeral-drain`, `attach` / `attach-end`, `hosted-deliver`, `pty-host-failed`, `supervise-error`, `composer-queue-failed-notify`, `memory-provision`, `session-exit` (из `exits.log`: `cause=child-exit` с `dead_status`/`dead_signal`, когда процесс сессии умер сам; `cause=shutdown-sigterm`/`shutdown-sigint`, когда супервизор был погашен — reap/stop/new), `supervisor-uncaught` (пережитый сбой супервизора, оставленный как форензика; сессия продолжает работать).
 
-Как выглядят главные состояния на потоке: **пробуждение** пира — `ev=wake`; **парковка в сон** — `ev=supervise action=reaped-idle`; **смерть** — `ev=session-exit` (немедленно, от супервизора), затем классификация демона на его следующем проходе.
+Как выглядят главные состояния на потоке: **пробуждение** пира — `ev=wake`; **парковка в сон** — `ev=supervise action=reaped-idle`; операторский **stop/start** — `ev=stopped` / `ev=started`; **смерть** — `ev=session-exit` (немедленно, от супервизора), затем классификация демона на его следующем проходе.
 
 ## Команды
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agfpd/iapeer",
-  "version": "0.4.63",
+  "version": "0.4.64",
   "description": "Run Claude Code and Codex CLI as local background agents that message each other, wake on demand, and share one memory.",
   "license": "Apache-2.0",
   "author": "agfpd",

package/src/cli/index.ts

@@ -253,12 +253,9 @@ export function stopPeer(personality: string, runtime: string | undefined, opts:
       killSession(sock, identity, env)
       const stderrText = (r.stderr ?? '').trim()
       const benign = r.status === 0 || r.status === 3 || /No such process/i.test(stderrText)
-      out.push({
-        personality,
-        runtime: rt,
-        action: 'bootout',
-        reason: benign ? undefined : `launchctl bootout exited ${r.status}${stderrText ? `: ${stderrText}` : ''}`,
-      })
+      const reason = benign ? undefined : `launchctl bootout exited ${r.status}${stderrText ? `: ${stderrText}` : ''}`
+      out.push({ personality, runtime: rt, action: 'bootout', reason })
+      appendLifecycleEvent(cfg.eventLogDir, { ev: 'stopped', identity, personality, runtime: rt, action: 'bootout', reason }, { env })
     } else {
       // A deliberate stop is a CLEAN PARK, not a death (stop→start must survive ≥
       // idle-reap): park-mark BEFORE the kill so the post-`start`
@@ -270,6 +267,12 @@ export function stopPeer(personality: string, runtime: string | undefined, opts:
       killSession(sock, identity, env)
       removeSessionState(cfg, identity)
       out.push({ personality, runtime: rt, action: 'stopped' })
+      // Durable trace (fleet-API follow-up): the verb was SILENT in lifecycle.log — a
+      // CLI stop changed the fleet state (● → ✕) with no event, so an SSE-fed client
+      // (tray class) showed a stale status indefinitely (no event ⇒ no snapshot
+      // re-read; only the initiator knew). One line per state-changing outcome; the
+      // refused-foreign-launchd branch changes nothing and stays silent.
+      appendLifecycleEvent(cfg.eventLogDir, { ev: 'stopped', identity, personality, runtime: rt, action: 'stopped' }, { env })
     }
   }
   return out
@@ -301,17 +304,17 @@ export function startPeer(personality: string, runtime: string | undefined, opts
       // sentinel fleet-guard + sandbox guard the raw spawn never had.)
       const r = launchctlBootstrap(personality, plist, env)
       const ok = r.state === 'loaded' || r.state === 'already-loaded' || r.state === 'skipped-sandbox'
-      out.push({
-        personality,
-        runtime: rt,
-        action: 'bootstrap',
-        reason: ok
-          ? undefined
-          : `launchctl bootstrap FAILED${r.detail ? `: ${r.detail}` : ''} — peer not started; manual rescue: launchctl bootstrap gui/$(id -u) ${plist}`,
-      })
+      const reason = ok
+        ? undefined
+        : `launchctl bootstrap FAILED${r.detail ? `: ${r.detail}` : ''} — peer not started; manual rescue: launchctl bootstrap gui/$(id -u) ${plist}`
+      out.push({ personality, runtime: rt, action: 'bootstrap', reason })
+      appendLifecycleEvent(cfg.eventLogDir, { ev: 'started', identity, personality, runtime: rt, action: 'bootstrap', reason }, { env })
     } else {
       clearStopped(cfg, identity)
       out.push({ personality, runtime: rt, action: 'started' })
+      // Durable trace — mirror of the `stopped` event in stopPeer (✕ → ○ is a state
+      // change an SSE-fed client must hear about).
+      appendLifecycleEvent(cfg.eventLogDir, { ev: 'started', identity, personality, runtime: rt, action: 'started' }, { env })
     }
   }
   return out

package/src/supervisor/daemon.ts

@@ -25,6 +25,7 @@ import {
   splitDanglingEscape,
   stripQueries,
 } from './protocol.ts'
+import { formatEventLine } from '../storage/rotatelog.ts'
 import { attachedPath, geometryPath, pidPath, servePath, sockPath, writeGeometry, writePidFile } from './paths.ts'
 import { newStuckGate, nextBootAction, nextNagAction, paneIsStuck, type BootPredicates, type NagPredicate } from './boot.ts'
 import { modelToPlainText } from './render.ts'
@@ -289,6 +290,30 @@ export function runSupervisorDaemon(opts: SupervisorDaemonOptions): void {
     return ok
   }
 
+  // ── death-trace (fleet-API follow-up, 05.07) ──────────────────────────────────
+  // exits.log is the durable «why did this session end» record. Before this, ONLY a
+  // child that died on its own (exit callback below) wrote a line; a SIGTERM'd
+  // supervisor (reap / stop / new / self-fresh — EVERY daemon-initiated teardown)
+  // killed the child and process.exit'ed before the callback could run, and a
+  // supervisor crash left nothing at all — exits.log stayed silent for weeks while
+  // sessions died (live gap: claude-boris server-dead 04.07 23:59 with ZERO forensics;
+  // the supervise classification even points the investigator at exits.log). Now every
+  // in-process exit path writes exactly one line: `exitLogged` dedups the child-exit
+  // callback vs the shutdown path. logfmt via the shared writer (quoted values parse
+  // intact in the fleet SSE stream). Out-of-process deaths (SIGKILL / OOM / Bun panic)
+  // remain unrecordable from here by nature — for those the supervise-tick reaped-gone
+  // classification in lifecycle.log stays the only trace.
+  let exitLogged = false
+  const appendExitEvent = (fields: Record<string, string | number | undefined>): void => {
+    if (!opts.exitLogPath) return
+    try {
+      mkdirSync(dirname(opts.exitLogPath), { recursive: true, mode: 0o700 })
+      appendFileSync(opts.exitLogPath, `${formatEventLine(Date.now(), fields)}\n`)
+    } catch {
+      /* best-effort — the trace must never block serving/teardown */
+    }
+  }
+
   const child = Bun.spawn(CMD, {
     cwd: opts.cwd ?? runDir,
     env,
@@ -337,20 +362,18 @@ export function runSupervisorDaemon(opts: SupervisorDaemonOptions): void {
         // supervise-tick death classification + crash-loop accounting have the cause. host=supervisor
         // distinguishes the source. The Bun-native pty exit callback passes (exitCode, signalCode) like
         // onExit; fall back to the subprocess props. Best-effort — never blocks shutdown.
-        if (opts.exitLogPath) {
-          try {
-            mkdirSync(dirname(opts.exitLogPath), { recursive: true, mode: 0o700 }) // ensure the exits.log dir exists
-            const c = child as { exitCode?: number | null; signalCode?: number | null }
-            const status = exitCode ?? c.exitCode ?? ''
-            const signal = signalCode ?? c.signalCode ?? ''
-            appendFileSync(
-              opts.exitLogPath,
-              `ts=${new Date().toISOString()} ev=session-exit identity=${session} dead_status=${status} dead_signal=${signal} host=supervisor\n`,
-            )
-          } catch {
-            /* best-effort */
-          }
-        }
+        const c = child as { exitCode?: number | null; signalCode?: number | null }
+        const status = exitCode ?? c.exitCode
+        const signal = signalCode ?? c.signalCode
+        exitLogged = true // this IS the death record — shutdown below must not write a second one
+        appendExitEvent({
+          ev: 'session-exit',
+          identity: session,
+          dead_status: status ?? '',
+          dead_signal: signal ?? '',
+          cause: 'child-exit',
+          host: 'supervisor',
+        })
         shutdown()
       },
     },
@@ -554,9 +577,17 @@ export function runSupervisorDaemon(opts: SupervisorDaemonOptions): void {
   writeGeometry(runDir, session, serveCols, serveRows) // initial geometry sidecar (= serve); updated on every real child resize
 
   let down = false
-  function shutdown(): void {
+  function shutdown(cause = 'shutdown'): void {
     if (down) return
     down = true
+    // Death-trace: a signalled shutdown kills the child below and process.exit()s
+    // immediately — the child exit callback never runs, so THIS is the only chance
+    // to record the session's end. `exitLogged` is already set when the child died
+    // on its own (the callback wrote the authoritative dead_status/dead_signal line).
+    if (!exitLogged) {
+      exitLogged = true
+      appendExitEvent({ ev: 'session-exit', identity: session, cause, host: 'supervisor' })
+    }
     if (paneLogFd >= 0)
       try {
         closeSync(paneLogFd)
@@ -582,18 +613,31 @@ export function runSupervisorDaemon(opts: SupervisorDaemonOptions): void {
     }
     process.exit(0)
   }
-  process.on('SIGTERM', shutdown)
-  process.on('SIGINT', shutdown)
+  process.on('SIGTERM', () => shutdown('shutdown-sigterm'))
+  process.on('SIGINT', () => shutdown('shutdown-sigint'))
   process.on('SIGHUP', () => {
     /* daemon survives client window close */
   })
-  process.on('uncaughtException', e => {
+  // A survivable fault must still leave a trace: the console.error goes to the
+  // detached daemon's session log, which sat at 0 bytes for weeks on the live fleet —
+  // effectively invisible. Record it in exits.log too (ev=supervisor-uncaught; the
+  // session keeps serving — killing it here would regress the detach-crash robustness),
+  // so a later wedge/death postmortem can see the fault that preceded it.
+  const traceFault = (kind: string, e: unknown): void => {
     try {
-      console.error('[supervisor-daemon] uncaught', (e && (e as Error).stack) || e)
+      console.error(`[supervisor-daemon] ${kind}`, (e && (e as Error).stack) || e)
     } catch {
       /* */
     }
-  })
+    appendExitEvent({
+      ev: 'supervisor-uncaught',
+      identity: session,
+      kind,
+      reason: String((e as Error)?.stack ?? e).replace(/\s+/g, ' ').slice(0, 300),
+    })
+  }
+  process.on('uncaughtException', e => traceFault('exception', e))
+  process.on('unhandledRejection', e => traceFault('rejection', e))
 
   // ── boot-dialog driver (serving slice a) ────────────────────────────────────────
   // For a real runtime, answer the startup dialogs (codex trust/update/hooks, claude trust/resume)