@gotcos/glasses-server 6.36.20 → 6.36.21

package/CHANGELOG.md

@@ -1,3 +1,77 @@
+## 6.36.21
+
+**Fence liveness: the one thing about a fence a machine can actually establish.**
+
+`GET /agent-sessions/fences` and the release preview now carry a `liveness`
+aggregate -- `state` (`running` / `none_running` / `unknown`) plus counts. It
+answers "is a child COS spawned for this turn still writing", which is the only
+way releasing is unsafe for a reason a machine can see: admit a new turn while an
+old child is still writing and two writers interleave in one transcript.
+
+IT IS NOT A SAFETY VERDICT and no surface may render it as one. Whether the
+ambiguous turn landed stays unknowable. Two automatic fence resolvers were
+designed and both rejected (thread-fence-store.ts:40) because the dominant shape
+is `timeout`, where the child had ~21 minutes to run tool calls before SIGKILL --
+and the only fence this system has ever produced was exactly that shape. A
+classifier built on n=1 would be guessing with a confident face.
+
+The module enforces the three traps FenceRecord warns about rather than repeating
+them: a pid is matched against its RECORDED start, because the OS recycles pids;
+an empty or missing spawn list resolves to `unknown`, never `none_running`,
+because "nothing was recorded" is not "nothing ran"; and any probe that throws or
+returns something unparseable resolves toward `unknown`. A live child outranks an
+unreadable probe.
+
+Only the aggregate crosses the wire. No pid, no spawn list, and the rest of the
+evidence block stays disk-only -- widening that is a deliberate contract change,
+not a side effect of this.
+
+`listFences` now REQUIRES the probe. It was optional, and the resulting
+`deps === undefined` branch was unreachable from the route: a mutation flipping
+its default to `none_running` left all 262 route tests green. An unreached line
+that returns a confident answer is worse than no line.
+
+
+**A turn queued against a fenced thread was thrown away in about two minutes.**
+
+The thread-turn queue is wired before the agent-session-bindings router, and that
+router built its own `TargetGuard`. So the queue's occupancy gate could not see a
+fence at all: a fenced target reported attachable, `drainDecision` returned
+'deliver', the loopback attach refused `native_target_fenced`, and the refusal
+spent one of five delivery attempts. Five 20-second ticks retire a turn -- long
+before anyone could reach the Mac, and the only fence this system has produced sat
+for roughly 40 hours because clearing it required a terminal.
+
+This is the SAME failure `native_target_busy` hit before the binding was made
+visible to occupancy, and the note recording that fix is three lines above the
+change. Fences never got the same treatment.
+
+The server now owns one `TargetGuard`, constructed above the queue and handed to
+the router, and occupancy consults it. A fenced target holds, and a hold spends
+nothing.
+
+`native_target_fenced` is now queueable -- the one entry in that set cleared by a
+person rather than a clock. The previous reasoning (waiting cannot resolve "may or
+may not have been delivered") was correct for a world where releasing needed a
+terminal. COS Control 0.5.63 ships a Release button that runs, so the wait ends on
+a real event. Queueing does not weaken anything: delivery re-enters the attach
+route and re-runs the fence check.
+
+Fence-held turns get `FENCE_HELD_TURN_TTL_MS` (72h) instead of the ordinary 6h.
+Six hours is right for a busy thread; it is wrong for a state that ends when a
+person looks. It still expires, and it still says so.
+
+Two existing tests were retargeted rather than deleted, both with the reasoning
+kept: the one asserting a fence is NOT queueable, and the one using a fence as its
+example of a refusal that can never clear -- that rule is unchanged and now uses a
+structurally permanent reason as its witness.
+
+Requires COS Control 0.5.63 for the Release button to actually work.
+
+Suite 3003 / 213 files, tsc 0. The wiring and liveness guards are mutation-verified: removing
+the fence check from occupancy, and moving the guard below the queue, each fail
+the assertion written for them.
+
 ## 6.36.20
 
 Follow-up to 6.36.19, which was never published. QA found three things in it.
@@ -34,8 +108,15 @@ explicit session end or `POST /api/archive/now`.
 The 409 copy on both query paths no longer tells the wearer to reopen for a
 "fresh message list" -- as of app 6.8.423 the cards stay.
 
-Requires app 6.8.423. Suite 2985 / 211 files, tsc 0. The conversation-import
-canary is mutation-verified: reintroducing the import fails it.
+Requires app 6.8.423. Suite 2976 / 209 files, tsc 0, measured on a CLEAN
+checkout of this commit rather than a working tree carrying another session's
+uncommitted files -- the 6.36.19 note quoted 2981/211 from a contaminated tree,
+and that tree's untracked code would have shipped in the tarball.
+
+The conversation-import canary is mutation-verified: reintroducing the import
+fails it. One unidentified test failed on a single clean-tree run and did not
+recur across six further runs; it is not attributed to this change and it is
+recorded here rather than rounded down to "clean".
 
 ## 6.36.19
 - **Resetting the spoken message count no longer ends the conversation.**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.36.20",
+  "version": "6.36.21",
   "description": "COS Glasses — self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
   "type": "module",
   "bin": {

package/server/index.ts

@@ -24,10 +24,12 @@ import { createAttachedTurnStream } from './lib/session-stream-producer.js'
 import { claudeSessionsRouter } from './routes/claude-sessions.js'
 import {
   createAgentSessionBindingsRouter,
+  TargetGuard,
 
   threadAttachEnabled,
 } from './routes/agent-session-bindings.js'
 import { AgentSessionBindingRegistry } from './lib/agent-session-binding-registry.js'
+import { targetKey } from './lib/agent-session-binding-store.js'
 import { cosSpawnedPids } from './lib/agent-session-ownership-store.js'
 import { buildOccupancyProbes, realOccupancyDirs } from './lib/occupancy-probes.js'
 import { realAttachedWorkspaceDeps, resolveAttachedWorkspace } from './lib/attached-workspace.js'
@@ -516,6 +518,18 @@ app.use('/api', claudeSessionsRouter)
 // accounting; a second copy of that sequence in a background worker is a second place
 // for the gate to drift. One request per delivered turn, at a handful of turns a day,
 // buys the guarantee that the queue CANNOT weaken the gate even by accident.
+// ONE guard, owned here, because the thread-turn queue below is wired BEFORE the
+// bindings router and still has to see a fence. When the router made its own, the
+// queue read an empty one: a fenced target reported attachable, the drainer
+// ATTEMPTED, the attach route refused `native_target_fenced`, and five of those
+// retired the queued turn in about two minutes. That is the identical failure the
+// binding note in `occupancy` below records, one layer over.
+const sharedTargetGuard = new TargetGuard(
+  process.env.COS_THREAD_FENCE_DURABLE === '1'
+    ? { load: readFences, save: writeFences }
+    : null,
+)
+
 if (threadAttachEnabled()) {
   const queueDeps = {
     // TWO GATES, ONE ANSWER. `threadOccupancy` sees FOREIGN holders -- another app on
@@ -544,6 +558,12 @@ if (threadAttachEnabled()) {
         // (`blocksTarget`), so an expired or terminal one correctly reads as free.
         const holder = agentSessionBindingRegistry.getByThread(provider, threadId, Date.now())
         if (holder !== null) return { attachable: false, reason: 'native_target_busy' }
+        // THE FENCE, for the same reason the binding is here and not one layer down.
+        // This is ADVISORY: the attach route stays the authority that refuses. Saying
+        // it here only means the drainer holds instead of spending an attempt, and a
+        // turn can wait out a fence that a person will clear.
+        const fenced = sharedTargetGuard.fencedReason(targetKey(provider, threadId))
+        if (fenced !== null) return { attachable: false, reason: fenced }
         return { attachable: true, reason: null }
       } catch {
         // A throwing probe is not an open door.
@@ -597,6 +617,9 @@ app.use('/api', createAgentSessionBindingsRouter({
   // POST /agent-sessions/fences/release work either way — which already removes
   // the restart from the recovery path — and this flag flips on once COS Control
   // has a Fences card.
+  // The instance the queue's `occupancy` already reads. Constructing a second one
+  // here would give the queue a permanently empty fence view.
+  guard: sharedTargetGuard,
   fencePersistence: process.env.COS_THREAD_FENCE_DURABLE === '1'
     ? { load: readFences, save: writeFences }
     : undefined,

package/server/lib/fence-liveness.ts

@@ -0,0 +1,126 @@
+// Is any child COS spawned for a fenced turn STILL RUNNING?
+//
+// WHAT THIS IS NOT. It is not a safety verdict and it must never be presented as
+// one. Two plans for an automatic fence resolver were designed and both rejected
+// (thread-fence-store.ts:40), and the reasoning still holds: the dominant fence
+// shape is `timeout`, where the child had a ~21 minute budget to run tool calls
+// before SIGKILL, and nothing observable afterwards distinguishes "the turn landed"
+// from "it did not". This module answers a narrower question that IS mechanically
+// knowable, and leaves delivery to the person.
+//
+// WHY THE NARROW QUESTION IS WORTH ANSWERING. A still-running child is the one way
+// releasing a fence is unsafe for a reason a machine can see: admit a new turn while
+// an old child is still writing and two writers interleave in one transcript.
+// Everything else about a fence is judgement.
+//
+// PID ALONE IS NOT AN IDENTITY. `spawns` records a MEASURED start for each pid
+// precisely because the OS recycles pids; a live pid whose start time does not match
+// is a different process and says nothing about ours. FenceRecord's own comment says
+// so, and this module is where that warning is enforced rather than repeated.
+//
+// EMPTY IS NOT ABSENCE. An empty or missing `spawns` list means no child was ever
+// RECORDED, which is not the same as no child having run -- FenceRecord says no
+// resolver may read it as "nothing landed". It resolves to `unknown`, never to
+// `none_running`.
+//
+// FAIL CLOSED. Anything unmeasurable resolves toward `unknown`. The caller is a
+// human deciding whether to release; an over-confident `none_running` is worse than
+// admitting the probe could not see.
+
+/** Recorded spawn: a pid and the start COS measured when it created the child. */
+export interface RecordedSpawn {
+  pid: number
+  startMs: number
+}
+
+export type FenceLivenessState =
+  /** Every recorded child is provably gone. */
+  | 'none_running'
+  /** At least one recorded child is alive AND is identity-matched to ours. */
+  | 'running'
+  /** Nothing was recorded, or at least one probe could not answer. */
+  | 'unknown'
+
+export interface FenceLiveness {
+  state: FenceLivenessState
+  /** Spawns examined. Zero means nothing was recorded. */
+  recorded: number
+  /** Alive and identity-matched: the reason a release is mechanically unsafe. */
+  running: number
+  /** Alive but started too far from the recorded time -- a RECYCLED pid, not ours.
+   *  Counted, never treated as ours, and never treated as evidence about our child. */
+  recycled: number
+  /** Probes that threw or returned something unusable. */
+  unverifiable: number
+}
+
+export interface FenceLivenessDeps {
+  /** Epoch ms at which `pid` started, or null when it is not running.
+   *  MAY THROW; a throwing probe is not an answer and resolves to `unknown`. */
+  pidStartMs: (pid: number) => number | null
+}
+
+/**
+ * Clock skew between COS's own `Date.now()` at spawn time and the start `ps`
+ * reports, which has one-second resolution. Two seconds is comfortably wider than
+ * that rounding and far narrower than any realistic pid-recycling window.
+ */
+export const START_MATCH_TOLERANCE_MS = 2_000
+
+export function fenceLiveness(
+  spawns: ReadonlyArray<RecordedSpawn> | undefined | null,
+  deps: FenceLivenessDeps,
+  toleranceMs: number = START_MATCH_TOLERANCE_MS,
+): FenceLiveness {
+  const rows = Array.isArray(spawns) ? spawns : []
+  const out: FenceLiveness = {
+    state: 'unknown', recorded: rows.length, running: 0, recycled: 0, unverifiable: 0,
+  }
+
+  // Nothing recorded. NOT 'none_running' -- see the header.
+  if (rows.length === 0) return out
+
+  for (const row of rows) {
+    if (!row || !Number.isSafeInteger(row.pid) || row.pid <= 0
+      || !Number.isFinite(row.startMs)) {
+      out.unverifiable += 1
+      continue
+    }
+    let started: number | null
+    try {
+      started = deps.pidStartMs(row.pid)
+    } catch {
+      out.unverifiable += 1
+      continue
+    }
+    if (started === null) continue          // gone: the ordinary, good case
+    if (!Number.isFinite(started)) { out.unverifiable += 1; continue }
+    if (Math.abs(started - row.startMs) <= toleranceMs) out.running += 1
+    else out.recycled += 1                  // someone else's process wearing our pid
+  }
+
+  // A live child outranks an unreadable probe: it is the one thing we are sure of.
+  if (out.running > 0) out.state = 'running'
+  else if (out.unverifiable > 0) out.state = 'unknown'
+  else out.state = 'none_running'
+  return out
+}
+
+/** Real probe. `ps -o lstart=` is the only start-time keyword macOS ps offers --
+ *  `etimes` is not a valid keyword here, which is why this parses a date rather
+ *  than reading elapsed seconds. */
+export function makePidStartProbe(
+  run: (pid: number) => string | null,
+): (pid: number) => number | null {
+  return (pid: number) => {
+    const raw = run(pid)
+    if (raw === null) return null
+    const text = raw.trim()
+    if (!text) return null
+    const parsed = Date.parse(text)
+    // Unparseable is NOT "not running" -- throw so the caller counts it
+    // unverifiable rather than silently reading a live child as gone.
+    if (!Number.isFinite(parsed)) throw new Error(`unparseable process start: ${text}`)
+    return parsed
+  }
+}

package/server/lib/thread-turn-queue.ts

@@ -94,6 +94,21 @@ export interface QueuedThreadTurn {
  */
 export const QUEUED_TURN_TTL_MS = 6 * 60 * 60 * 1000
 
+/**
+ * TTL for a turn held by a FENCE specifically.
+ *
+ * Six hours is right for a busy thread: the holder finishes or it never will. A
+ * fence is different in kind -- it ends when a PERSON looks, and the only fence this
+ * system has ever produced sat for about 40 hours before anyone could clear it,
+ * because clearing it needed a terminal. Expiring at six would have thrown the turn
+ * away roughly seven times over while the user was still waiting for a working
+ * Release button.
+ *
+ * It still expires. A turn that outlives this is one nobody is coming back for, and
+ * silently holding it forever is worse than telling the truth about losing it.
+ */
+export const FENCE_HELD_TURN_TTL_MS = 72 * 60 * 60 * 1000
+
 /** Waiting turns per thread. Small: this is a queue, not a backlog. */
 export const MAX_QUEUED_PER_THREAD = 8
 
@@ -141,6 +156,18 @@ const QUEUEABLE_REFUSALS: ReadonlySet<string> = new Set([
   'unverifiable_process_start',
   'unverifiable_liveness_socket',
   'probe_failed',
+  // A FENCE IS CLEARED BY A PERSON, NOT BY A CLOCK -- the one exception in this set.
+  //
+  // It is queueable because "COS sent a turn and never learned whether it arrived"
+  // is precisely the state a queued turn should outlive: the operator checks, releases,
+  // and the turn lands. Releasing kicks the drain, so the wait ends on a real event.
+  //
+  // It is ALSO why the fence had to become visible to `occupancy` (index.ts). Without
+  // that, a fenced target reads attachable, the drainer ATTEMPTS, the attach route
+  // refuses, and five of those retire the turn in about two minutes -- the identical
+  // failure the binding comment above records, and it would have made queue-and-release
+  // look like it worked while silently dropping the turn.
+  'native_target_fenced',
 ])
 
 /** Can a turn refused for this reason be parked, or must it refuse now? */
@@ -159,6 +186,9 @@ export interface DrainObservation {
   turnEnded: boolean
   /** The 30s transcript clock. `idle` is the backstop when no terminal record lands. */
   activity: 'working' | 'idle' | 'unknown'
+  /** The gate's reason when `attachable` is false. Carried ONLY so a fence -- the one
+   *  hold a clock cannot end -- can be given a longer life than a busy thread. */
+  reason?: string | null
 }
 
 export type DrainDecision = 'deliver' | 'hold' | 'expire' | 'give_up'
@@ -178,7 +208,10 @@ export function drainDecision(
 ): DrainDecision {
   if (turn.status !== 'waiting') return 'hold'
   if (!Number.isFinite(now) || !Number.isFinite(turn.queuedAt)) return 'hold'
-  if (now - turn.queuedAt >= ttlMs) return 'expire'
+  // A fence is cleared by a person, so it gets the longer clock. Everything else --
+  // including a fence we could not name -- keeps the ordinary one.
+  const effectiveTtl = seen.reason === 'native_target_fenced' ? FENCE_HELD_TURN_TTL_MS : ttlMs
+  if (now - turn.queuedAt >= effectiveTtl) return 'expire'
   if (turn.attempts >= MAX_DELIVERY_ATTEMPTS) return 'give_up'
   // THE GATE, unweakened. Everything below is about WHEN, never about whether.
   if (!seen.attachable) return 'hold'

package/server/routes/agent-session-bindings.ts

@@ -95,6 +95,8 @@
 // and answer 400. They never treat an unparsed body as an empty one.
 
 import type { FenceRecord } from '../lib/thread-fence-store.js'
+import { execFileSync } from 'node:child_process'
+import { fenceLiveness, makePidStartProbe, type FenceLiveness, type FenceLivenessDeps } from '../lib/fence-liveness.js'
 import { Router, type Request, type Response } from 'express'
 import { createHash, randomUUID } from 'node:crypto'
 import {
@@ -271,6 +273,11 @@ export type AttachedTurnResult =
   | { ok: boolean; delivery: 'not_attempted' | 'aborted' | 'ambiguous' | 'delivered' }
 
 export interface AgentSessionBindingsDeps {
+  /** Shared fence state. Omit and the router owns a private one, which is correct
+   *  for tests and wrong for the server -- see the wiring note at its use site. */
+  guard?: TargetGuard
+  /** Process-start probe behind the fence liveness aggregate. */
+  liveness?: FenceLivenessDeps
   /**
    * Durable fence storage. OPTIONAL, and omitting it is what keeps the existing
    * suite in memory: a test that silently began writing the real data home would
@@ -889,7 +896,14 @@ export interface FencePersistence {
  * the real data home would leak fences between cases and into the running server,
  * so the suite runs with `null` and stays in memory.
  */
-class TargetGuard {
+/** The fence question, narrowed for callers that must not touch anything else.
+ *  `occupancy` in index.ts reads this so a fenced target holds instead of burning
+ *  a delivery attempt; the ATTACH route remains the authority that refuses. */
+export interface TargetFenceView {
+  fencedReason(targetKey: string): WriteRefusal | null
+}
+
+export class TargetGuard {
   /** targetKey -> turnId of the single COS turn allowed to be in flight. */
   private readonly claims = new Map<string, string>()
   /** targetKey -> the fence record. DURABLE as of 6.36.10: persistence is injected
@@ -994,8 +1008,18 @@ class TargetGuard {
   }
 
   /** Every fence, REDACTED for the wire: the raw targetKey embeds the private
-   *  native thread id, so callers address a fence by its deterministic digest. */
-  listFences(): Array<{ target: string; provider: string; reason: string; headBefore: string | null; turnId: string; fencedAt: number }> {
+   *  native thread id, so callers address a fence by its deterministic digest.
+   *
+   *  `liveness` is the ONLY evidence field that crosses, and it crosses as an
+   *  aggregate: a state plus counts, never a pid. It answers "is a child from this
+   *  turn still writing", which is the one thing about a fence a machine can
+   *  actually establish. It is NOT a safety verdict and the UI must not render it
+   *  as one -- whether the ambiguous turn landed stays unknowable, which is why two
+   *  automatic resolvers were rejected (thread-fence-store.ts:40).
+   *
+   *  The rest of the evidence block stays disk-only. Widening that is a deliberate
+   *  contract change, not a side effect of adding this. */
+  listFences(deps: FenceLivenessDeps): Array<{ target: string; provider: string; reason: string; headBefore: string | null; turnId: string; fencedAt: number; liveness: FenceLiveness }> {
     return [...this.fences.values()].map(row => ({
       target: opaqueRevision(row.targetKey),
       provider: row.provider,
@@ -1003,6 +1027,12 @@ class TargetGuard {
       headBefore: row.headBefore,
       turnId: row.turnId,
       fencedAt: row.fencedAt,
+      // REQUIRED, not optional. An optional probe meant a `deps === undefined`
+      // branch that the route could never take -- a mutation flipping it to
+      // `none_running` left the whole 262-test suite green, which is the
+      // definition of an unreached line. A caller with no probe must construct a
+      // deliberate one rather than fall into a default answer.
+      liveness: fenceLiveness(row.spawns, deps),
     }))
   }
 
@@ -1287,7 +1317,23 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
   const canListBindings = bindingDepsUsable(deps)
   const canWriteBindings = bindingWriteDepsUsable(deps)
   const detect = deps?.occupancy ?? threadOccupancy
-  const guard = new TargetGuard(deps.fencePersistence ?? null)
+  // INJECTABLE. index.ts owns the instance because the thread-turn queue is wired
+  // BEFORE this router and must be able to see a fence; a second guard created here
+  // would be a disconnected copy, and the queue would go on reading an empty one.
+  const guard = deps.guard ?? new TargetGuard(deps.fencePersistence ?? null)
+  // `ps -o lstart=` is the only start-time keyword macOS ps offers. Injectable so a
+  // test drives recycled and unreadable pids without spawning real processes.
+  const livenessDeps: FenceLivenessDeps = deps.liveness ?? {
+    pidStartMs: makePidStartProbe((pid) => {
+      try {
+        return execFileSync('ps', ['-p', String(pid), '-o', 'lstart='], {
+          encoding: 'utf8', timeout: 2_000,
+        })
+      } catch {
+        return null   // not running, or ps refused the pid
+      }
+    }),
+  }
   const ownership = deps?.ownership ?? { record: recordCosSpawn, release: releaseCosSpawn }
   // One per router. Injectable so the follow-on (attach accepting a `forkRef`)
   // shares this instance rather than standing up a second, disconnected one.
@@ -1383,7 +1429,7 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
     // `degraded` is reported, never inferred: a memory-only fence set behaves
     // identically to a durable one until the process restarts, so a silent
     // fallback would be indistinguishable from working.
-    res.json({ fences: guard.listFences(), degraded: guard.degraded() })
+    res.json({ fences: guard.listFences(livenessDeps), degraded: guard.degraded() })
   })
 
   router.post('/agent-sessions/fences/release', (req, res) => {
@@ -1400,7 +1446,10 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
       // The API token is shared by the phone, the lens and every COS agent
       // session, so nothing is structurally prevented from asserting `confirm`.
       // It stops an accidental release, not an automated one.
-      const preview = guard.listFences().find(f => f.target === target) ?? null
+      // The preview carries the liveness aggregate too: this is the exact moment a
+      // person decides, and "a child from this turn is still running" is the one
+      // thing here a machine can tell them.
+      const preview = guard.listFences(livenessDeps).find(f => f.target === target) ?? null
       res.status(400).json({ released: false, reason: 'confirmation_required', preview })
       return
     }

package/server/routes/thread-turn-queue.ts

@@ -113,6 +113,7 @@ export async function drainThread(
       attachable: gate.attachable,
       turnEnded: deps.turnEnded(provider, threadId),
       activity: deps.activity(provider, threadId),
+      reason: gate.reason,
     }
     const decision = drainDecision(turn, seen, deps.now())