@gotcos/glasses-server 6.50.1 → 6.50.2

package/CHANGELOG.md

@@ -1,3 +1,10 @@
+## 6.50.2
+
+- **The Turbo meeting preview no longer switches itself off for good.** Miles, 2026-09-18 08:43: the `~` provisional line on the lens was gone, Mac-reachable, Turbo configured. The preview route had answered "no text" to 5,719 requests since 2026-09-17 16:00 (in under a millisecond) while the Turbo sidecar itself transcribed the same audio in 0.27 s. One failed request (`The operation was aborted due to timeout`, then `fetch failed`, with the Mac under build load) set `previewAvailable = false`, and the only code that sets it back is the sidecar's own start, which runs once per server boot. Now a failure against a sidecar that is still running is a 10 s cooldown (`PREVIEW_FAILURE_COOLDOWN_MS`) and the next request after it tries again; five failures in a row (`PREVIEW_RECYCLE_AFTER_FAILURES`) recycle the sidecar; a sidecar that exited is restarted from the request path, at most once per 60 s window, doubling to 15 min while restarts keep failing and resetting on any answer. A missing model, a missing binary or a foreign listener on port 8177 is never restarted (a restart cannot fix it). After the server's own shutdown stop, nothing revives the sidecar (a request arriving during shutdown could otherwise have spawned one that outlived the process). Prompt previews were never affected (they fall back to the canonical worker); canonical meeting ASR never was. The preview log lines now carry an ISO timestamp: the lines that recorded this failure had none.
+- Tests (`whisper-preview.test.ts`, executed against spawn and fetch fakes and a fake clock): a timeout is a cooldown and the next request after it gets text; a success resets the count so scattered misses never recycle; five in a row recycle and the new sidecar answers; an exited sidecar restarts from the request path and the backoff doubles across failed restarts; a foreign listener is never re-probed. Eight mutants (latch restored, success never resets, no revive, backoff never doubles, cooldown ignored, foreign port restarted, revive after shutdown, start after shutdown) each turn the suite red; three of them survived the first draft of the tests and the tests were tightened (a recycle's kill is synchronous, its respawn is not, so negatives now wait for the loop to settle).
+- **Coming back to a Desktop tab no longer blocks a Continue for 30 seconds.** Miles, 2026-09-17 07:00: five sends from the glasses refused "Your Mac is writing to this thread right now" until he clicked away from the tab. Claude Desktop had written its away summary (the note it drops when you return to a tab you left) to the transcript 11 s earlier, and the sub-agent that wrote it fired `SubagentStop`. The attach gate reads a transcript written inside 30 s as working unless the hooks vouch that the turn ended, and the vouch required the newest hook event to be the `Stop` itself, so the desk's own bookkeeping after it made the gate strict. The vouch now reads "no turn since the Stop": a `SubagentStop` with no sub-agent open, or a notification, keeps it; anything that opens a turn (a prompt, a tool, a sub-agent starting) still refuses, and a newer Stop still moves the clock.
+- Tests: the two bookkeeping events keep the vouch and each other guard still holds behind them; a tool, a prompt, a sub-agent start and a permission request stay strict. Gate: `turns-b6-desk-bookkeeping-is-strict-again` and `turns-b6-any-event-vouches` added; the four B6 cases killed.
+
 ## 6.50.1
 
 What the QA pass on 6.50.0 found in the history it serves (6.50.0 was published before the pass finished; nothing it changed regressed, and a client that does not ask for `?turns` sees no difference).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.50.1",
+  "version": "6.50.2",
   "description": "COS Glasses \u2014 self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
   "type": "module",
   "bin": {

package/server/index.ts

@@ -858,7 +858,7 @@ async function gracefulShutdown(): Promise<void> {
   try { logActiveSessionsOnShutdown() } catch { /* best-effort flush */ }
   stopCodexModelCatalogRefresh()
   stopWhisperServer()
-  await stopWhisperPreviewServer()
+  await stopWhisperPreviewServer({ final: true })
   stopLocalTtsServer()
   clearTimeout(forceExit)
   process.exit(0)

package/server/lib/occupancy-probes.ts

@@ -792,15 +792,28 @@ export function buildOccupancyProbes(
  *
  * `signalFor` is the signal store's reader, taking a full session id or the registry's
  * 8-character form; `registryIdleAfterStop` reads the registry record for the session.
- * The probe answers a Stop time only when BOTH vouch: the session's newest hook event is
- * that Stop, the turn is closed, no sub-agent is open, the session has not ended, and
- * the registry says idle at or after the Stop (the hooks have returned). Codex never
+ * The probe answers a Stop time only when BOTH vouch: nothing since the Stop has opened
+ * a turn (the desk's own bookkeeping after it does not count: the sub-agent that writes
+ * Claude Desktop's away summary, an idle notification), the turn is closed, no sub-agent
+ * is open, the session has not ended, and the registry says idle at or after the Stop
+ * (the hooks have returned). Codex never
  * asks (its ids are not Claude sessions, and a Codex thread id sharing a Claude prefix
  * must not read a Claude signal); `readHolderActivity` guards the provider too.
  *
  * Wired at the composition root only when `COS_SESSION_HOOKS` is on. Absent, the
  * gate is byte-for-byte 6.48.0.
  */
+/**
+ * Hook events that can follow a Stop without a turn having started (6.50.2). Miles,
+ * 2026-09-17 07:00: five sends refused `native_thread_working` because Claude Desktop
+ * wrote its `away_summary` system row to the transcript 11 s earlier and the sub-agent
+ * that wrote it fired SubagentStop, so the newest event was not 'Stop' and the 30 s
+ * transcript window held. Every event that DOES mean a turn sets `turnOpen`
+ * (UserPromptSubmit, a tool) or `subagentsOpen` (SubagentStart), which the guards
+ * behind this set still read; a newer Stop replaces `stopAt`.
+ */
+const AFTER_THE_TURN: ReadonlySet<string> = new Set(['Stop', 'SubagentStop', 'Notification'])
+
 export function withHookTurnClock(
   probes: OccupancyProbes,
   signalFor: (sessionId: string) => HookTurnSignal | undefined,
@@ -812,7 +825,7 @@ export function withHookTurnClock(
       if (provider !== 'claude') return null
       if (!NATIVE_THREAD_ID_RE.test(threadId)) return null
       const signal = signalFor(threadId)
-      if (!signal || signal.lastEvent !== 'Stop' || signal.turnOpen || signal.subagentsOpen > 0 || signal.ended !== null) return null
+      if (!signal || !AFTER_THE_TURN.has(signal.lastEvent) || signal.turnOpen || signal.subagentsOpen > 0 || signal.ended !== null) return null
       if (typeof signal.stopAt !== 'number') return null
       // The engine's own end of turn: the registry flipped idle AFTER this Stop, which
       // happens only once every Stop hook has returned. Anything else is strict.

package/server/lib/whisper-preview.ts

@@ -69,6 +69,82 @@ let previewFailure: WhisperPreviewReason = null
 let previewWorkerModel: WhisperPreviewSidecarModel | null = null
 let warnedInvalidChoice = false
 
+/**
+ * 6.50.2: a request that fails against a sidecar that is STILL RUNNING is a miss, not a
+ * death. Before this, one timeout (the Mac under load) or one refused fetch set
+ * `previewAvailable = false`, and its only writer back to true is the sidecar's own start,
+ * which runs once per server boot, so the meeting preview lane stayed silent for the rest
+ * of the process: 2026-09-17 16:00 to 2026-09-18 08:48 (34 h), 5,719 requests answered 204
+ * in under a millisecond while Turbo itself answered the same audio in 0.27 s.
+ *
+ * Now: a failure against a live sidecar starts a short cooldown and the next request after
+ * it tries again. PREVIEW_RECYCLE_AFTER_FAILURES in a row recycles the sidecar (stop, then
+ * start). A sidecar that is gone is restarted from the request path, backing off from
+ * PREVIEW_RESTART_BASE_MS to PREVIEW_RESTART_MAX_MS so a model that cannot load is not
+ * reloaded every minute. Any successful answer resets both counters.
+ */
+export const PREVIEW_FAILURE_COOLDOWN_MS = 10_000
+export const PREVIEW_RECYCLE_AFTER_FAILURES = 5
+export const PREVIEW_RESTART_BASE_MS = 60_000
+export const PREVIEW_RESTART_MAX_MS = 15 * 60_000
+let previewCooldownUntil = 0
+let previewConsecutiveFailures = 0
+let previewLastRestartAt = 0
+let previewRestartBackoffMs = PREVIEW_RESTART_BASE_MS
+/** Set by the server's shutdown: no request may revive the sidecar after it (QA, 6.50.2). */
+let previewShutdown = false
+
+function previewSidecarAlive(): boolean {
+  const child = previewProcess
+  return !!child && child.exitCode === null && child.signalCode === null
+}
+
+function notePreviewRequestSuccess(): void {
+  previewConsecutiveFailures = 0
+  previewCooldownUntil = 0
+  previewRestartBackoffMs = PREVIEW_RESTART_BASE_MS
+}
+
+/** A request to the sidecar failed (not a preemption). Cooldown, recycle, or mark it gone. */
+function notePreviewRequestFailure(error: unknown, label: string): void {
+  previewConsecutiveFailures++
+  const message = error instanceof Error ? error.message : String(error)
+  const at = new Date().toISOString()
+  if (previewSidecarAlive() && previewConsecutiveFailures < PREVIEW_RECYCLE_AFTER_FAILURES) {
+    previewCooldownUntil = Date.now() + PREVIEW_FAILURE_COOLDOWN_MS
+    console.warn(`[whisper-preview] ${at} ${label} failed (${previewConsecutiveFailures} in a row); next try in ${PREVIEW_FAILURE_COOLDOWN_MS / 1000} s: ${message}`)
+    return
+  }
+  const alive = previewSidecarAlive()
+  previewAvailable = false
+  previewWorkerModel = null
+  previewFailure = 'preview_sidecar_unavailable'
+  console.warn(`[whisper-preview] ${at} ${label} failed (${previewConsecutiveFailures} in a row); ${alive ? 'recycling the sidecar' : 'the sidecar is gone'}: ${message}`)
+  maybeRevivePreviewSidecar()
+}
+
+/**
+ * Bring a failed sidecar back, from the request path. Only for a sidecar that ran and then
+ * failed or exited: a missing model, a missing binary or a foreign listener on the port
+ * cannot be fixed by a restart and stay as they are.
+ */
+function maybeRevivePreviewSidecar(): void {
+  if (previewShutdown || previewAvailable || previewStarting) return
+  if (previewFailure !== 'preview_sidecar_unavailable' && previewFailure !== 'preview_start_failed') return
+  const now = Date.now()
+  if (previewLastRestartAt && now - previewLastRestartAt < previewRestartBackoffMs) return
+  if (previewLastRestartAt) previewRestartBackoffMs = Math.min(previewRestartBackoffMs * 2, PREVIEW_RESTART_MAX_MS)
+  previewLastRestartAt = now
+  previewConsecutiveFailures = 0
+  console.log(`[whisper-preview] ${new Date(now).toISOString()} restarting the preview sidecar (${previewFailure}); next restart no sooner than ${Math.round(previewRestartBackoffMs / 1000)} s`)
+  void (async () => {
+    if (previewProcess) await stopWhisperPreviewServer()
+    await startWhisperPreviewServer()
+  })().catch(error => {
+    console.warn(`[whisper-preview] restart failed: ${error instanceof Error ? error.message : error}`)
+  })
+}
+
 interface ProcessEntry {
   pid: number
   ppid: number
@@ -260,6 +336,9 @@ async function endpointReady(path: '/health' | '/inference', init?: RequestInit,
  * uses Turbo while its canonical live worker remains Large-v3. Failure is
  * cosmetic: every recovery/finalization path stays untouched. */
 export async function startWhisperPreviewServer(): Promise<void> {
+  // A revive already past its stop when shutdown landed must not spawn an orphan that
+  // outlives the server.
+  if (previewShutdown) return
   const requested = requestedPreviewModel()
   const selected = selectedPreviewModel(requested)
   if (!needsPreviewSidecar(selected) || previewProcess || previewAvailable || previewStarting) return
@@ -359,7 +438,8 @@ function waitForPreviewClose(child: ChildProcess, timeoutMs: number): Promise<bo
   })
 }
 
-export async function stopWhisperPreviewServer(): Promise<void> {
+export async function stopWhisperPreviewServer(options: { final?: boolean } = {}): Promise<void> {
+  if (options.final) previewShutdown = true
   const child = previewProcess
   previewProcess = null
   previewAvailable = false
@@ -395,27 +475,23 @@ export async function transcribeWhisperPreview(audioBuffer: Buffer): Promise<{
   model: 'small.en' | WhisperCommitModel
   backend: 'whisper-preview-server' | 'whisper-server'
   }> {
-  if (previewAvailable && previewWorkerModel) {
+  if (!previewAvailable) maybeRevivePreviewSidecar()
+  if (previewAvailable && previewWorkerModel && Date.now() >= previewCooldownUntil) {
     const workerModel = previewWorkerModel
     const previewLease = tryAcquireMetalPreview()
     if (!previewLease) {
       return { text: '', model: workerModel, backend: 'whisper-preview-server' }
     }
     try {
-      return {
-        text: await transcribeViaPreviewServer(audioBuffer, previewLease.signal),
-        model: workerModel,
-        backend: 'whisper-preview-server',
-      }
+      const text = await transcribeViaPreviewServer(audioBuffer, previewLease.signal)
+      notePreviewRequestSuccess()
+      return { text, model: workerModel, backend: 'whisper-preview-server' }
     } catch (error) {
       if (previewLease.signal.aborted) {
         return { text: '', model: workerModel, backend: 'whisper-preview-server' }
       }
-      const failedModel = previewWorkerModel
-      previewAvailable = false
-      previewWorkerModel = null
-      previewFailure = 'preview_sidecar_unavailable'
-      console.warn(`[whisper-preview] ${failedModel} preview failed; falling back to canonical worker: ${error instanceof Error ? error.message : error}`)
+      // Falls through to the canonical worker below for THIS request either way.
+      notePreviewRequestFailure(error, `${workerModel} preview`)
     } finally {
       previewLease.release()
     }
@@ -448,21 +524,20 @@ export async function transcribeWhisperMeetingPreview(audioBuffer: Buffer): Prom
   model: 'large-v3-turbo'
   backend: 'whisper-preview-server'
 } | null> {
+  if (!previewAvailable) maybeRevivePreviewSidecar()
   if (!previewAvailable || previewWorkerModel !== 'large-v3-turbo') return null
+  // A cooldown after a failed request is a quiet miss, never a latch (see above).
+  if (Date.now() < previewCooldownUntil) return null
   const previewLease = tryAcquireMetalPreview()
   if (!previewLease) return null
   try {
-    return {
-      text: await transcribeViaPreviewServer(audioBuffer, previewLease.signal),
-      model: 'large-v3-turbo',
-      backend: 'whisper-preview-server',
-    }
+    const text = await transcribeViaPreviewServer(audioBuffer, previewLease.signal)
+    notePreviewRequestSuccess()
+    return { text, model: 'large-v3-turbo', backend: 'whisper-preview-server' }
   } catch (error) {
     if (previewLease.signal.aborted) return null
-    previewAvailable = false
-    previewWorkerModel = null
-    previewFailure = 'preview_sidecar_unavailable'
-    console.warn(`[whisper-preview] meeting Turbo preview failed; canonical meeting ASR was not affected: ${error instanceof Error ? error.message : error}`)
+    // Canonical meeting ASR is not affected by any of this: the preview is cosmetic.
+    notePreviewRequestFailure(error, 'meeting Turbo preview')
     return null
   } finally {
     previewLease.release()