@gotcos/glasses-server 6.24.1 → 6.24.3

package/CHANGELOG.md

@@ -1,3 +1,64 @@
+## 6.24.3
+
+Auto-recovery of quarantined audio has never run in production. Miles saw the symptom
+for three turns: "1 recoverable" that opening the phone app could not clear.
+
+- **My call sat inside a bare `catch {}`.** `autoRecoverOneQuarantinedCapture()` was one
+  line after `purgeExpiredQuarantine()` inside the orphan-audio sweep's
+  `try { ... } catch {}`, so any throw in that sweep meant auto-recovery silently never
+  executed — on 6.23.1, 6.24.0, 6.24.1 and 6.24.2. Zero `[quarantine]` lines in a 48 MB
+  log across every one of those releases. It now has its own try, because recovering
+  quarantined audio has nothing to do with sweeping orphaned session-audio dirs and must
+  not depend on that succeeding.
+- **That bare catch is why it took three turns to find.** Three minutes of watching a
+  live server produced no recovery, no log, and nothing to reason about, because the
+  error was discarded. Both catches now report. I chased three wrong causes first — a
+  closed admissions gate (`admissionsOpen` was `true`), a stale npm cache (real, but a
+  different bug), and a broken picker (it selects the item correctly against live data).
+- **A one-chunk capture is no longer advertised as recoverable.**
+  `meeting_1786393815060_tp693w` held ONE 5.6-second chunk that transcribed to silence.
+  Recovering it would have produced an empty meeting titled "Recovered capture (audio
+  only)"; advertising it produced a badge with instructions that cannot work, since a
+  server-side quarantine has no deferred phone save to land. `MIN_RECOVERABLE_CHUNKS`
+  is 2, and `isWorthRecovering` is the SINGLE definition used by the picker AND by both
+  warning counts — two definitions would let the badge claim something the sweeper has
+  already decided to skip.
+- No audio is deleted by any of this. Quarantine retention still owns expiry.
+
+Coverage: 8 mutations, all caught. The placement mutations were first measured against a
+RED baseline and re-run once green, because a mutation against a failing tree proves
+nothing. Three of those red iterations were my own test windowing, never the fix: a
+file-wide ban that hit a second legitimate bare catch, a fixed-width slice that ran past
+the fix, and an `indexOf` that matched the function definition instead of the call site.
+
+Full suite 1505 serially, tsc clean, gate after the bump.
+
+## 6.24.2
+
+The empty-recording restart lock, split out of 6.24.1.
+
+**Correction to 6.24.1's own entry.** These bullets were written under 6.24.1 while it
+was still unpublished, but it had already gone to the registry by then, so that heading
+described a published artifact that does not contain this code. Verified by downloading
+the published tarball: 6.24.1 carries the `project` field and has no
+`EMPTY_SESSION_STALE_MS`. Moved here rather than left standing.
+
+- **A 6-second aborted recording no longer locks the restart for 30 minutes.**
+  `getTranscriptionSessionLiveness` gated purely on elapsed time. The 30-minute grace
+  exists to protect a real recording that has gone briefly quiet — a backgrounded phone
+  buffering to IndexedDB — but a session with NO canonical text has nothing to protect.
+  Observed 2026-08-10: `meeting_1786393815060_tp693w` started, received one 5.6s chunk
+  that transcribed to empty, stopped 6.2 seconds later, and then blocked Update Server.
+  A session idle past `EMPTY_SESSION_STALE_MS` (2 minutes) with zero canonical chunks
+  now stops blocking a restart.
+- **Why that cannot reap a live recording.** `lastActivityAt` is bumped on chunk
+  ARRIVAL, before any text filtering, so a live recording in a silent room keeps
+  arriving every ~10s and never goes idle at all; its silent chunks land in
+  `emptyCompletions`, not `chunks`. Gating on emptiness ALONE would kill exactly that
+  session, which is why the rule requires emptiness AND idleness. Counted on canonical
+  text rather than array length, because the array is sparse and a silent chunk carries
+  no text — the precise shape an aborted recording leaves behind.
+
 ## 6.24.1
 
 Surfaces `project` on session rows, so the list can group the way Claude Code's own

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.24.1",
+  "version": "6.24.3",
   "description": "COS Glasses \u2014 self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
   "type": "module",
   "bin": {

package/server/lib/quarantine-auto-recover.ts

@@ -27,6 +27,34 @@
 
 import type { UnsavedCapture } from './unsaved-audio-quarantine.js'
 
+/**
+ * Chunks below which a capture is not worth turning into a meeting.
+ *
+ * A chunk covers roughly 5 to 10 seconds, so one chunk is a recording that started and
+ * stopped almost immediately. Observed 2026-08-10: `meeting_1786393815060_tp693w` held
+ * ONE 5.6-second chunk that transcribed to silence, and the panel advertised it as
+ * "1 recoverable" with instructions to open the phone app — which cannot clear a
+ * server-side quarantine, so the badge simply persisted.
+ *
+ * Recovering it would run a full batch transcription and produce an empty meeting
+ * titled "Recovered capture (audio only)". That is noise, not rescue. The audio is NOT
+ * deleted here — quarantine retention still owns that decision and expires it on its
+ * own clock. This only decides what is worth acting on and worth warning about.
+ */
+export const MIN_RECOVERABLE_CHUNKS = 2
+
+/**
+ * Is this capture substantial enough to act on?
+ *
+ * One definition, used by the auto-recover picker AND by the counts that drive the
+ * "unsaved captures" warning, so the badge cannot claim something is recoverable that
+ * the sweeper has already decided to leave alone.
+ */
+export function isWorthRecovering(item: { recovered: boolean; chunkFiles: number }): boolean {
+  if (item.recovered) return false
+  return item.chunkFiles >= MIN_RECOVERABLE_CHUNKS
+}
+
 /** Attempts per capture before the sweep stops trying on its own. */
 export const MAX_AUTO_RECOVER_ATTEMPTS = 3
 
@@ -58,10 +86,8 @@ export function pickQuarantineToRecover(
   state: AutoRecoverState,
 ): UnsavedCapture | null {
   const eligible = items.filter(item => {
-    // Already a meeting. Recovering again would duplicate it.
-    if (item.recovered) return false
-    // Nothing to transcribe: a chunk-less dir is residue, not evidence.
-    if (item.chunkFiles <= 0) return false
+    // Already a meeting, chunk-less residue, or too small to be a meeting at all.
+    if (!isWorthRecovering(item)) return false
     // Another recovery owns this one.
     if (state.inFlight.has(item.sessionId)) return false
     return (state.attempts.get(item.sessionId) ?? 0) < MAX_AUTO_RECOVER_ATTEMPTS

package/server/routes/health.ts

@@ -1,4 +1,5 @@
 import { Router } from 'express'
+import { isWorthRecovering } from '../lib/quarantine-auto-recover.js'
 import { statSync } from 'node:fs'
 import { resolve } from 'node:path'
 import { COS_SCRIPTS_DIR, COS_MODE } from '../lib/python-bridge.js'
@@ -227,7 +228,7 @@ healthRouter.get('/health', async (_req, res) => {
   // plus the recover action live on the authenticated /api/meeting/orphans.
   const unsavedList = listUnsavedCaptures()
   const unsaved_captures = {
-    count: unsavedList.filter(item => !item.recovered).length,
+    count: unsavedList.filter(isWorthRecovering).length,
     items: unsavedList.slice(0, 10).map(item => ({
       sessionId: item.sessionId,
       ageHours: item.ageHours,

package/server/routes/meeting.ts

@@ -3,6 +3,7 @@
 // are durable before the session is closed; batch improvement runs afterward.
 
 import { existsSync, readdirSync, readFileSync, rmSync, statSync, unlinkSync } from 'node:fs'
+import { isWorthRecovering } from '../lib/quarantine-auto-recover.js'
 import { resolve } from 'node:path'
 import { Router } from 'express'
 import { emitDisplay } from '../lib/display-bus.js'
@@ -1672,7 +1673,7 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
     // route answered count: 0.
     const stranded = getStrandedCaptures()
     res.json({
-      count: items.filter(item => !item.recovered).length,
+      count: items.filter(isWorthRecovering).length,
       strandedCount: stranded.length,
       stranded,
       recovering: [...recoveringOrphans],

package/server/routes/transcribe-stream.ts

@@ -387,6 +387,26 @@ export function getActiveTranscriptionSessionCount(): number {
  */
 export const RECORDING_SESSION_STALE_MS = STRANDED_STALE_MS
 
+/**
+ * How long an EMPTY session may hold the restart lock.
+ *
+ * The 30-minute grace above protects a real recording that has gone briefly quiet — a
+ * backgrounded phone buffering to IndexedDB, a network drop, a long pause. A session
+ * that has produced NO canonical text has nothing to protect, so it must not lock
+ * Install, Repair, Restart and Update Server for half an hour.
+ *
+ * Observed 2026-08-10: meeting_1786393815060_tp693w started, took ONE 5.6s chunk that
+ * transcribed to empty, stopped 6.2s later, then blocked every restart for 30 minutes.
+ *
+ * WHY THIS CANNOT REAP A LIVE RECORDING. `lastActivityAt` is bumped on chunk ARRIVAL
+ * in processStreamChunk, before any text filtering, so a live recording in a silent
+ * room keeps arriving every ~10s and never goes idle — its silent chunks land in
+ * `emptyCompletions`, not `chunks`. Gating on emptiness ALONE would kill exactly that
+ * session; gating on emptiness AND idleness cannot. Two minutes is comfortably more
+ * than one chunk interval, so a recording that has just started is safe.
+ */
+export const EMPTY_SESSION_STALE_MS = 2 * 60_000
+
 export interface TranscriptionSessionLiveness {
   /** Sessions still plausibly recording. These block a restart. */
   live: number
@@ -479,8 +499,12 @@ export function getTranscriptionSessionLiveness(now = Date.now()): Transcription
   const staleSessions: TranscriptionSessionLiveness['staleSessions'] = []
   for (const [sessionId, session] of sessions) {
     const silentForMs = now - session.lastActivityAt
-    if (silentForMs >= RECORDING_SESSION_STALE_MS) {
-      staleSessions.push({ sessionId, silentForMs, chunks: session.chunks.length })
+    // Canonical text, not array length: the array is sparse and silent chunks carry no
+    // text, which is exactly the shape an aborted recording leaves behind.
+    const canonicalChunks = session.chunks.filter(chunk => chunk && chunk.text).length
+    const emptyAndIdle = canonicalChunks === 0 && silentForMs >= EMPTY_SESSION_STALE_MS
+    if (silentForMs >= RECORDING_SESSION_STALE_MS || emptyAndIdle) {
+      staleSessions.push({ sessionId, silentForMs, chunks: canonicalChunks })
     } else {
       live++
     }
@@ -991,8 +1015,22 @@ setInterval(() => {
       }
     }
     purgeExpiredQuarantine()
+  } catch (error) {
+    // Was a bare `catch {}`. That is what made the auto-recover failure below
+    // undiagnosable: three minutes of watching a live server produced no recovery, no
+    // log, and nothing to reason about, because any throw in this block vanished.
+    console.error(`[cleanup] Orphan-audio sweep failed: ${error instanceof Error ? error.message : error}`)
+  }
+  // DELIBERATELY ITS OWN TRY. This used to sit inside the block above, one line after
+  // purgeExpiredQuarantine, so a throw anywhere in that sweep meant auto-recovery
+  // silently never ran — which is exactly what happened in production on 6.23.1
+  // through 6.24.2. Recovering quarantined audio has nothing to do with sweeping
+  // orphaned session-audio dirs and must not depend on it succeeding.
+  try {
     autoRecoverOneQuarantinedCapture()
-  } catch {}
+  } catch (error) {
+    console.error(`[quarantine] Auto-recover pass failed: ${error instanceof Error ? error.message : error}`)
+  }
   // Purge stale pending-batch dirs. HQ large-v3 on long meetings + Control
   // restart can exceed 2h (2026-07-27: two sessions purged before batch).
   // Use 12h, and never purge while a meeting_batch_finalization lease is held.