@gotcos/glasses-server 6.36.10 → 6.36.11

package/CHANGELOG.md

@@ -1,5 +1,60 @@
 ## Unreleased
 
+## 6.36.11
+- **Fences now record WHY, so the population can be measured before anything resolves
+  automatically.** Two plans designed an automatic fence resolver and both were rejected —
+  the second because there has never been a single fence on the machine to look at. If
+  `timeout` dominates, the child had the full 21-minute budget to run tool calls before
+  SIGKILL and re-delivery would re-execute them, so no automatic clear is ever safe. That
+  question was unanswerable and now is not.
+- **`reaped` is reported by the adapter, never derived.** A signal-killed child reports
+  `code === null` and the handlers only assign `exitCode` for a numeric code — so deriving
+  "was it reaped" from `exitCode` reports NEVER REAPED for the dominant timeout shape
+  (SIGTERM, then SIGKILL), which is exactly backwards for the decision this data informs.
+  The first cut of this change did derive it. `AttachedTurnFailureResult` now carries
+  `reaped`, true from every settle reached via `close`/`error` and false only from the
+  force-settle that fires when `close` never arrived.
+- **An unreadable adapter result records nothing, not zeroes.** Reading `{}` and writing
+  `exitCode: null, childReaped: false` states two facts about a child nothing is known
+  about, indistinguishable on disk from a confirmed-unreaped timeout — corrupting the one
+  discriminator this evidence exists to establish.
+- **`fenceSite` says which site fired.** `adapterReason` cannot substitute: the catch site
+  inherits whatever the adapter last reported, so a route crash AFTER a clean delivery
+  records `ok` — the strongest possible reason NOT to re-deliver, which would otherwise
+  read as "nothing went wrong".
+- **One resolved reason for the record and the log.** The record said `unreadable_result`
+  while the breadcrumb said `unknown`, so an operator grepping for the sentinel found
+  nothing. This is the same contradiction 6.36.10 fixed at the other fence site,
+  re-committed one release later in the same handler; both now read one value.
+- **A release no longer destroys the evidence.** `releaseFence` deletes the row, and the
+  realistic first-fence sequence is: fence lands, Control's card appears, it is released,
+  the distribution is gone. The release breadcrumb now carries the whole record.
+- **Spawn identity as `{pid, startMs}` PAIRS.** `recordedPids` held bare pids and the
+  measured start was discarded; a pid alone cannot be told apart from a recycled one.
+  The evidence type is narrowed to the six adapter fields so the spread at the fence
+  sites cannot clobber the fence's own identity — `Partial<FenceEvidence>` permitted
+  `provider`, and the adapter result carries one that would write null and fail
+  `isFenceRecord` on the next read, silently un-enforcing the fence.
+- **All fields OPTIONAL and NOT in `isFenceRecord`.** That predicate is cast-based, so a
+  required field would type as present while being undefined at runtime; extending it
+  would reclassify existing rows as unrecognised and silently un-enforce them.
+- **DISK ONLY.** Nothing reaches the wire; asserted against the real `/fences` body and
+  the release preview.
+- **No behaviour change.** Nothing clears, nothing refuses differently. Upgrade, downgrade
+  and the COS Control card were all verified unaffected.
+- **Coverage, stated honestly.** The adapter's `reaped` contract is tested at the adapter,
+  driving a real `close(null)` — a route test could not cover it, because the route
+  fixtures supply `reaped` themselves and would pass with the adapter gutted. Mutation
+  results: caught — derived-from-exitCode, adapter stops reporting reaped, startMs zeroed,
+  empty spawn list, missing fenceSite, unrecorded adapterReason, wire leak. **Survived, and
+  therefore unverified: `stderrClass` is written but asserted nowhere, and `fail()`'s
+  `reaped: false` default on the `not_attempted` paths (which never fence).** The
+  `route_error` fence site remains reachable by no test.
+- **Known gaps, not fixed here:** `stderrClass` appears in the breadcrumbs but on a
+  default install (`COS_THREAD_FENCE_DURABLE` unset) nothing is written to disk at all;
+  and reading the distribution means reading `thread-fences.json` or the server log —
+  there is no UI for it.
+
 ## 6.36.10
 - **A fenced thread had no exit and left no trace.** An ambiguous delivery fences the
   target so a prompt cannot be double-delivered into a real conversation — that is

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.36.10",
+  "version": "6.36.11",
   "description": "COS Glasses \u2014 self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
   "type": "module",
   "bin": {

package/server/lib/agent-session-store.ts

@@ -1063,9 +1063,29 @@ async function enrichLiveClaude(row: AgentSessionRow, roots: AgentSessionRoots):
   const firstPrompt = await firstClaudeUserTitle(found)
   const title = peek.customTitle ?? firstPrompt ?? row.display_label
   const fullId = found.split('/').pop()?.replace(/\.jsonl$/i, '') || row.session_id
+  // THE LIVE ROW'S `modified` IS A HEARTBEAT, NOT AN MTIME, AND IT GOES STALE.
+  //
+  // `liveClaudeRows` builds it from the peer registry's `lastActiveAt` (routes/
+  // agent-sessions.ts:257). That value tracks the registry record, not the
+  // transcript, so a session that is actively writing keeps reporting whenever the
+  // registry last moved. Measured 2026-08-18 on three live sessions: the wire said
+  // 55.3m / 407.7m / 435.0m while the transcript had been written 0.1m / 0.2m /
+  // 5.1m earlier — under-reporting a live session by up to 7.2 hours.
+  //
+  // We are already holding the resolved transcript path and have already read it
+  // twice, so the true mtime costs one stat. A failed stat keeps the heartbeat:
+  // worse, but no worse than today.
+  //
+  // THAT FALLBACK IS DEFENSIVE AND UNREACHED BY THE SUITE. `if (!found) return row`
+  // above already catches the missing-file case, so reaching `: row.modified` needs a
+  // stat to fail on a file that was just resolved. A mutation of it survives; that is
+  // recorded rather than papered over.
+  const st = await fileStat(found)
+  const modified = st?.isFile ? isoFromMtime(st.mtimeMs) : row.modified
   return {
     ...row,
     session_id: fullId,
+    modified,
     display_label: title || row.display_label || 'Claude session',
     ...discussionFields(title || row.display_label, firstPrompt || '', peek.latestAssistant),
   }

package/server/lib/attached-provider-adapter.ts

@@ -194,6 +194,18 @@ export interface AttachedTurnFailureResult {
    */
   detail: string | null
   exitCode: number | null
+  /**
+   * Did `close`/`exit` actually fire, i.e. did the kernel reap the child?
+   *
+   * NOT DERIVABLE FROM `exitCode`. A child killed by a signal reports
+   * `code === null`, and the handlers below only assign `exitCode` for a numeric
+   * code — so the dominant timeout shape (SIGTERM, then SIGKILL) is reaped while
+   * leaving `exitCode` null. Deriving reaping from the code therefore reports
+   * "never reaped" for exactly the case a reader most needs to identify.
+   * Only the force-settle path, which fires when `close` never arrived, is
+   * genuinely unreaped.
+   */
+  reaped: boolean
   stderrClass: AttachedStderrClass
   durationMs: number
 }
@@ -745,6 +757,10 @@ function fail(
     nativeThreadId: null,
     returnedNativeId: null,
     delivery,
+    // Conservative default: NOT OBSERVED reaped. Most `fail()` callers are
+    // `not_attempted` paths where no child exists, and the ones that do have a
+    // child override this from `settleFailure`.
+    reaped: false,
     reason,
     detail: null,
     exitCode: null,
@@ -1051,6 +1067,10 @@ function driveChild(input: DriveInput): Promise<AttachedTurnResult> {
         nativeThreadId,
         returnedNativeId: observedIds.length === 1 ? observedIds[0]! : null,
         exitCode,
+        // Every settle but the force-settle below is reached from `finishTerminal`,
+        // which only runs from the `close`/`error` handlers — so the child was
+        // reaped. The one exception overrides this explicitly.
+        reaped: true,
         stderrClass: classifyStderr(stderrSample),
         durationMs: duration(),
         ...over,
@@ -1193,7 +1213,7 @@ function driveChild(input: DriveInput): Promise<AttachedTurnResult> {
           // A child that survived SIGKILL cannot be reached from here, and
           // blocking forever would wedge the coordinator and any Control drain
           // behind it.
-          settleFailure('timeout', { detail: 'unreaped' })
+          settleFailure('timeout', { detail: 'unreaped', reaped: false })
         }, FORCE_SETTLE_MS)
       }, KILL_GRACE_MS)
     }, timeoutMs)

package/server/lib/thread-fence-store.ts

@@ -34,6 +34,69 @@ export interface FenceRecord {
   turnId: string
   bindingId: string | null
   fencedAt: number
+
+  // ── EVIDENCE (all optional, all DISK-ONLY) ────────────────
+  //
+  // WHY THIS EXISTS. Two plans designed an automatic fence resolver and both were
+  // rejected, the second because we have never observed a single fence. The
+  // distribution decides whether any automatic path is safe: if `timeout`
+  // dominates, the child had the full 21-minute budget to run tool calls before
+  // SIGKILL and re-delivery re-executes them, so no automatic clear is ever safe.
+  // If the child exited on its own with a known code, the footing is much better.
+  //
+  // EVERY FIELD IS OPTIONAL, and deliberately NOT added to `isFenceRecord`. That
+  // predicate is cast-based (`r is FenceRecord`), so a REQUIRED field would type as
+  // present while being undefined at runtime and tsc could not catch it; and
+  // extending the validator would reclassify existing rows as unrecognised, routing
+  // them into the preserved-but-inert pile and silently un-enforcing real fences.
+  //
+  // NONE OF THIS REACHES THE WIRE. The router contract carries no pid and no native
+  // thread id; `listFences()` already omits `bindingId` as precedent.
+
+  /**
+   * The adapter's own verdict.
+   *
+   * REACHABILITY IS DECIDED BY `delivery === 'ambiguous'`, i.e. the stdin-write
+   * boundary — only a turn whose prompt bytes reached the child can fence. The
+   * reachable set is therefore NOT the full `AttachedTurnFailure` enum:
+   *   timeout, provider_exit_nonzero, no_native_id_returned, native_id_mismatch,
+   *   child_stdio_unavailable (detail `write_failed`), spawn_failed (ONLY via
+   *   detail `child_error`; its other forms are `not_attempted` and never fence),
+   *   plus two this route synthesises — `ok` (the adapter reported success with a
+   *   non-delivered state, a contradiction) and `unreadable_result`.
+   * `adapter_internal_error` can NEVER fence: every emission site is
+   * `not_attempted` or guarded. `route_error` comes from the catch site.
+   */
+  adapterReason?: string
+  /**
+   * WHICH fence site fired.
+   *
+   * `adapterReason` cannot substitute. The catch site inherits whatever the
+   * adapter last reported, so a route crash AFTER a clean delivery records
+   * `ok` — the single strongest reason NOT to re-deliver, which would read as
+   * "nothing went wrong". Only this field separates them.
+   */
+  fenceSite?: 'ambiguous' | 'route_error'
+  /** Bounded self-authored discriminator, e.g. `unreaped`. Never provider output. */
+  adapterDetail?: string | null
+  exitCode?: number | null
+  /**
+   * Reported BY THE ADAPTER, never derived from `exitCode`.
+   *
+   * A signal-killed child reports `code === null`, and the adapter only assigns
+   * `exitCode` for a numeric code — so deriving this would record "never reaped"
+   * for the dominant timeout shape (SIGTERM then SIGKILL), which is the exact
+   * case this evidence exists to identify. Undefined when the adapter result was
+   * unreadable: absent, never a fabricated `false`.
+   */
+  childReaped?: boolean
+  stderrClass?: string
+  /** How long the turn ran before it failed. A timeout is ~21 minutes of tool calls. */
+  durationMs?: number
+  /** Every child COS spawned for this turn, with its MEASURED start. Both are needed:
+   *  a pid alone cannot be distinguished from a recycled one. An EMPTY list means no
+   *  child was ever spawned, which no resolver may ever read as "nothing landed". */
+  spawns?: Array<{ pid: number; startMs: number }>
 }
 
 export function fencePath(): string {

package/server/routes/agent-session-bindings.ts

@@ -839,6 +839,17 @@ export const COS_SESSION_ID_RE = /^[A-Za-z0-9][A-Za-z0-9._:@/-]{0,127}$/
 /** What a fence records about the turn that set it. */
 export interface FenceEvidence {
   provider: string
+  /** The adapter's verdict, or 'route_error' at the catch site. DISK-ONLY. */
+  adapterReason?: string
+  /** WHICH site fenced. `adapterReason` cannot substitute: the catch site inherits
+   *  whatever the adapter reported, so a fence can read 'ok' there. DISK-ONLY. */
+  fenceSite?: 'ambiguous' | 'route_error'
+  adapterDetail?: string | null
+  exitCode?: number | null
+  childReaped?: boolean
+  stderrClass?: string
+  durationMs?: number
+  spawns?: Array<{ pid: number; startMs: number }>
   /** The head BEFORE the ambiguous turn. Null ONLY when the failure happened
    *  before the head was read. */
   headBefore: string | null
@@ -961,6 +972,14 @@ class TargetGuard {
       turnId: evidence.turnId,
       bindingId: evidence.bindingId,
       fencedAt: evidence.now,
+      adapterReason: evidence.adapterReason,
+      fenceSite: evidence.fenceSite,
+      adapterDetail: evidence.adapterDetail,
+      exitCode: evidence.exitCode,
+      childReaped: evidence.childReaped,
+      stderrClass: evidence.stderrClass,
+      durationMs: evidence.durationMs,
+      spawns: evidence.spawns,
     })
     this.persistFences([...this.fences.values()])
   }
@@ -1389,7 +1408,12 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
       res.status(status).json({ released: false, reason: outcome.reason })
       return
     }
-    console.warn(`[agent-session-bindings] fence RELEASED by operator target=${target} provider=${outcome.row.provider} fencedAt=${outcome.row.fencedAt}`)
+    // THE EVIDENCE DIES WITH THE ROW. `releaseFence` deletes it, and the realistic
+    // sequence is: first fence ever lands -> Control's card appears -> it is
+    // released -> the distribution this evidence exists to collect is gone. So the
+    // release line carries the whole record, not just its identity.
+    const ev = outcome.row
+    console.warn(`[agent-session-bindings] fence RELEASED by operator target=${target} provider=${ev.provider} fencedAt=${ev.fencedAt} fenceSite=${ev.fenceSite ?? 'unknown'} adapterReason=${ev.adapterReason ?? 'unknown'} detail=${ev.adapterDetail ?? 'none'} exitCode=${ev.exitCode ?? 'null'} childReaped=${ev.childReaped ?? 'unknown'} stderrClass=${ev.stderrClass ?? 'none'} durationMs=${ev.durationMs ?? 'unknown'} spawnCount=${ev.spawns?.length ?? 0}`)
     res.json({ released: true, target, provider: outcome.row.provider })
   })
 
@@ -1809,7 +1833,18 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
     let fenceProvider = ''
     let preTurnHeadDigest: string | null = null
     /** Children the adapter reported, released in the finally. */
-    const recordedPids: number[] = []
+    // PAIRS, not bare pids. A pid alone cannot be told apart from a recycled one,
+    // and `startMs` is measured in the onSpawn closure and was previously discarded.
+    const recordedPids: Array<{ pid: number; startMs: number }> = []
+    // Hoisted so BOTH fence sites can record what the adapter actually reported.
+    // `result` is scoped inside the try; the ambiguous site sits after the catch.
+    // NARROW ON PURPOSE. `Partial<FenceEvidence>` would permit provider/headBefore/
+    // turnId/bindingId/now, and the spread sits AFTER them at both fence sites — so
+    // a future field could silently overwrite the fence's identity. The adapter
+    // result literally carries a `provider`, which would write null and fail
+    // `isFenceRecord` on the next read, silently un-enforcing the fence.
+    let adapterEvidence: Partial<Pick<FenceEvidence,
+      'adapterReason' | 'adapterDetail' | 'exitCode' | 'childReaped' | 'stderrClass' | 'durationMs'>> = {}
     let pinnedBindingId: string | null = null
     let requestNow = 0
     /**
@@ -2095,10 +2130,35 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
             // before the prompt rather than deliver a turn that poisons the next
             // occupancy check.
             if (outcome !== 'recorded') return false
-            recordedPids.push(pid)
+            recordedPids.push({ pid, startMs })
             return true
           },
         })
+        // Read the adapter's OWN account before it goes out of scope. This is the
+        // whole point of the evidence work: `classifyDelivery` collapses six distinct
+        // failures into one word, and the difference between them is what decides
+        // whether an automatic resolver could ever be safe. A 21-minute `timeout`
+        // means the child ran tool calls; a `provider_exit_nonzero` with a code means
+        // it exited on its own. Defensive reads — the adapter is injected in tests.
+        // AN UNREADABLE RESULT RECORDS NOTHING, not zeroes. Reading `{}` and
+        // deriving `exitCode: null, childReaped: false` states two facts about a
+        // child nothing is known about, and writes them indistinguishably from a
+        // confirmed-unreaped timeout — corrupting the one discriminator this
+        // evidence exists to establish.
+        const r = (result !== null && typeof result === 'object')
+          ? result as Record<string, unknown>
+          : null
+        adapterEvidence = r === null ? {} : {
+          adapterReason: typeof r.reason === 'string' ? r.reason : (r.ok === true ? 'ok' : undefined),
+          adapterDetail: typeof r.detail === 'string' ? r.detail : null,
+          exitCode: typeof r.exitCode === 'number' ? r.exitCode : null,
+          // FROM THE ADAPTER, never derived from exitCode. A signal-killed child
+          // reports `code === null`, so deriving it would report "never reaped" for
+          // the dominant timeout shape — backwards for the decision this informs.
+          childReaped: typeof r.reaped === 'boolean' ? r.reaped : undefined,
+          stderrClass: typeof r.stderrClass === 'string' ? r.stderrClass : undefined,
+          durationMs: typeof r.durationMs === 'number' ? r.durationMs : undefined,
+        }
         delivery = classifyDelivery(result)
       } catch (error) {
         console.error(`[agent-session-bindings] adapter threw: ${error instanceof Error ? error.message : error}`)
@@ -2116,18 +2176,30 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
         // Fenced under its own reason, not this turn's: `delivery_ambiguous`
         // describes what happened to THIS request, while a later caller needs to
         // be told the thread is shut and why it must be inspected first.
+        // ONE resolved value for the record AND the log. The previous release
+        // fixed exactly this contradiction at the other fence site and this one
+        // re-committed it: the record said `unreadable_result` while the log said
+        // `unknown`, so an operator grepping for the sentinel found nothing.
+        const recordedReason = adapterEvidence.adapterReason ?? 'unreadable_result'
         guard.fence(key, 'native_target_fenced', {
           provider: binding.provider,
           headBefore: head.digest,
           turnId,
           bindingId,
           now: Date.now(),
+          ...adapterEvidence,
+          // NEVER blank. An unreadable adapter result and "nobody recorded it" are
+          // different facts and a missing field cannot tell them apart -- which is
+          // the whole reason this evidence exists.
+          adapterReason: recordedReason,
+          fenceSite: 'ambiguous',
+          spawns: [...recordedPids],
         })
         // A fence shuts a thread until a human acts, and until now it wrote NO log
         // line at either site — so a fenced thread was discoverable only by trying
         // to use it (Miles, 2026-08-18). Never log `key`: it embeds the private
         // native thread id, which this router does not emit anywhere.
-        console.warn(`[agent-session-bindings] fence set site=ambiguous provider=${binding.provider} target=${opaqueRevision(key)} turnId=${turnId} bindingId=${bindingId} headBefore=${head.digest}`)
+        console.warn(`[agent-session-bindings] fence set site=ambiguous provider=${binding.provider} target=${opaqueRevision(key)} turnId=${turnId} bindingId=${bindingId} headBefore=${head.digest} adapterReason=${recordedReason} detail=${adapterEvidence.adapterDetail ?? 'none'} exitCode=${adapterEvidence.exitCode ?? 'null'} childReaped=${adapterEvidence.childReaped ?? 'unknown'} stderrClass=${adapterEvidence.stderrClass ?? 'none'} durationMs=${adapterEvidence.durationMs ?? 'unknown'} spawnCount=${recordedPids.length}`)
         return reportAmbiguous()
       }
 
@@ -2164,20 +2236,28 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
             turnId,
             bindingId: null,
             now: Date.now(),
+            // A bug in THIS route, not a provider outcome. Named rather than left
+            // blank so a later reader can tell "the adapter reported nothing" apart
+            // from "nobody recorded it". Any adapter evidence captured before the
+            // throw is still carried.
+            ...adapterEvidence,
+            adapterReason: adapterEvidence.adapterReason ?? 'route_error',
+            fenceSite: 'route_error',
+            spawns: [...recordedPids],
           })
           // `head` is scoped to the try, so `preTurnHeadDigest` is hoisted to the
           // handler specifically to reach this site. It is null ONLY when the throw
           // happened before the head was read. An earlier version of this line
           // hardcoded `unavailable` and so contradicted the record it had just
           // written — an operator would read "no baseline" off a fence that has one.
-          console.warn(`[agent-session-bindings] fence set site=route_error target=${opaqueRevision(claimedKey)} turnId=${turnId} headBefore=${preTurnHeadDigest ?? 'unavailable'}`)
+          console.warn(`[agent-session-bindings] fence set site=route_error target=${opaqueRevision(claimedKey)} turnId=${turnId} headBefore=${preTurnHeadDigest ?? 'unavailable'} adapterReason=${adapterEvidence.adapterReason ?? 'route_error'} detail=${adapterEvidence.adapterDetail ?? 'none'} exitCode=${adapterEvidence.exitCode ?? 'null'} childReaped=${adapterEvidence.childReaped ?? 'unknown'} stderrClass=${adapterEvidence.stderrClass ?? 'none'} durationMs=${adapterEvidence.durationMs ?? 'unknown'} spawnCount=${recordedPids.length}`)
         }
         reportAmbiguous()
       } else {
         refuseTurn('turn_failed')
       }
     } finally {
-      for (const pid of recordedPids) {
+      for (const { pid } of recordedPids) {
         try {
           ownership.release(pid)
         } catch (error) {