@gotcos/glasses-server 6.36.11 → 6.36.12

package/CHANGELOG.md

@@ -1,5 +1,50 @@
 ## Unreleased
 
+## 6.36.12
+- **Claude and Codex now rank candidates by recency before spending the budget.** Both
+  walked in raw `readdir` order, so which sessions were reachable came down to filesystem
+  layout — `collectCursorDocs` had ranked for a while and these two were the inconsistent
+  ones. Statting every candidate first costs ~41ms across 2,118 files. Cursor also charged
+  its budget before the keep-warm filter, the same defect, now fixed.
+- **The reach claim in the module docstring was never true and is now measured.** It said
+  "older chats stay findable". Ranked and measured on this machine, search reaches roughly
+  24 days of Claude, 89 days of Codex and 19 days of Cursor, because once candidates are
+  ordered by recency the per-provider doc budget IS the horizon. `EXAMINE_MULTIPLE` was
+  re-swept and raised to 12 — the point where examining more files stops finding anything
+  older and the doc budget takes over. A sampled older stratum was considered and
+  rejected: partial coverage makes a miss uninterpretable, and a search that silently
+  samples cannot tell you whether something is absent or merely unsampled.
+- **Embedding batches go out together instead of one after another.** The loop awaited
+  each batch in turn, so cost was a round trip per 64 docs and grew as the collector
+  returned more — 389 docs is 7 serialized trips at the old size. Now 128 per request,
+  all in flight at once. Structural; not measured end to end, because this harness has no
+  OpenAI key and the running server was left alone.
+- **Session search was 68% scaffolding, and the budget was the reason.** Of the 1,296
+  Claude transcripts on this machine the collector indexed 41, and 28 of those 41 were
+  machine prompts — 22 Slack Bridge proxy, 4 reply-with, 2 slack_search_users. Roughly 13
+  real conversations were searchable, which is why searching an exact session title
+  returned nothing. Two changes, which do not work apart: `isKeepWarmSessionTitle` now
+  recognises the machine families by anchored prefix, and both the Claude and Codex
+  collectors run that filter — and the Codex `thread_source === 'subagent'` check — BEFORE
+  charging the doc budget rather than after. Measured against the real corpus: 41 indexed
+  Claude docs with 28 junk becomes **79 indexed with 0 junk**.
+- **The budget now counts docs kept, not files opened.** That is the whole fix: a run of
+  machine transcripts used to consume the 134-file allowance and return nothing, so the
+  newest real transcript on disk was never reached. A second `examined` ceiling
+  (`EXAMINE_MULTIPLE`, 5x the doc budget) stops a pathological corpus walking all 1,296
+  files, and the expensive transcript-body read is deferred until a file is being kept.
+- **This also removes rows from COS Control's session LIST.** The predicate is shared by
+  all four collectors and the list path, so keep-warm and Slack Bridge entries stop
+  appearing there too. That is intended, not a side effect to fix.
+- **Control may not see any of this yet.** The search route's median is ~2.27s against
+  Control's 2s client timeout, of which ~1.7s is two sequential embedding round trips;
+  this work adds ~350ms on top. Until that timeout and `EMBED_BATCH` are addressed,
+  Control falls back to its local scanner and reports a fabricated `server_too_old`.
+- **The scan budget had no test coverage at all.** `collectAgentSessionSearchDocs` was
+  never called by any test and its `cap` was never exercised, so the branch that spends
+  the budget had never run. 21 tests added, each verified to FAIL against the previous
+  code before being kept.
+
 ## 6.36.11
 - **Fences now record WHY, so the population can be measured before anything resolves
   automatically.** Two plans designed an automatic fence resolver and both were rejected —
@@ -54,6 +99,21 @@
   default install (`COS_THREAD_FENCE_DURABLE` unset) nothing is written to disk at all;
   and reading the distribution means reading `thread-fences.json` or the server log —
   there is no UI for it.
+- **A live session was reporting itself hours idle.** Separate from the fence work above.
+  `liveClaudeRows` builds a row's `modified` from the peer registry's `lastActiveAt`, which
+  tracks the REGISTRY record and not the transcript — so a session that is actively writing
+  keeps reporting whenever the registry last moved. Measured on three live sessions
+  2026-08-18: the wire said 55.3m / 407.7m / 435.0m old while their transcripts had been
+  written 0.1m / 0.2m / 5.1m earlier. Under-reporting by up to 7.2 hours. Shipped in
+  66dff88; `enrichLiveClaude` already resolves the transcript path and reads the file twice,
+  so the true mtime costs one `stat`. A resolved file that fails to stat keeps the
+  heartbeat; a session with no transcript at all (2 of 6 measured) returns early on the
+  existing guard. Prerequisite for any surface that renders a real date — without it,
+  showing the timestamp displays an actively-writing session as seven hours stale.
+  Coverage: `enrichLiveClaude` had NO execution coverage before this (every existing test
+  passes an empty live array). Two tests now drive it through `listAgentSessions`. Three
+  mutations, two caught; the third (the stat-failure fallback) SURVIVES because the
+  missing-file guard returns first, so that branch is unreached. The code says so.
 
 ## 6.36.10
 - **A fenced thread had no exit and left no trace.** An ambiguous delivery fences the

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.36.11",
-  "description": "COS Glasses \u2014 self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
+  "version": "6.36.12",
+  "description": "COS Glasses — self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
   "type": "module",
   "bin": {
     "glasses-server": "bin/cli.cjs",

package/server/lib/agent-session-search.ts

@@ -4,7 +4,18 @@
  * Keyword: local title / sidebar name / first prompt / transcript scan. No model.
  * Semantic: one OpenAI query embedding scored against those same texts.
  * Sessions have no Qdrant collection — this is not meeting or memory search.
- * The 7-day list window does not apply; older chats stay findable.
+ *
+ * REACH, measured 2026-08-18 rather than claimed. The 7-day list window does not apply,
+ * but this is not unbounded either: each provider gets a doc budget of MAX_SCAN_FILES/3,
+ * and once candidates are ranked by recency that budget IS the horizon. On this machine
+ * that is roughly 24 days of Claude, 89 days of Codex and 19 days of Cursor. An older
+ * chat is reachable only if its provider has not filled its budget with newer ones.
+ *
+ * This previously read "older chats stay findable", which was written before ranking and
+ * was never measured. Widening it means a bigger budget, which costs embedding time on
+ * the route -- not a deeper examine ceiling, which saturates. A sampled older stratum was
+ * considered and rejected: partial coverage makes a miss uninterpretable, and a search
+ * that silently samples cannot tell you whether a thing is absent or merely unsampled.
  */
 
 import { join } from 'node:path'
@@ -46,11 +57,51 @@ import {
 
 const HEAD_TEXT = 8_000
 const MAX_SCAN_FILES = 400
+
+/**
+ * How many files a provider may OPEN per search, as a multiple of the docs it may KEEP.
+ *
+ * Before the filter moved ahead of the decrement, `remaining` meant "files read" -- so a
+ * run of machine transcripts consumed the whole allowance and returned nothing. Now it
+ * means "docs kept", which on its own would let a pathological corpus walk all 1,296
+ * files. This is the ceiling that stops it.
+ *
+ * Swept 2026-08-18 against the real corpus (1,296 Claude transcripts, 822 Codex
+ * rollouts). Docs kept vs collector wall time, median of 5 warm runs:
+ *
+ * Re-swept after recency ranking landed, since ranking changes which files the ceiling
+ * is spent on -- the newest region of the corpus is the densest in machine transcripts:
+ *
+ *     x5     329 docs   74 claude   11.4d window   1287 ms
+ *     x8     389 docs  134 claude   21.3d window   1585 ms
+ *     x12    389 docs  134 claude   24.5d window   1697 ms   <- chosen
+ *     x16    389 docs  134 claude   24.5d window   1637 ms
+ *
+ * x12 is where the ceiling stops being the constraint and the 134-doc budget takes over:
+ * past it, examining more files finds nothing older. That is the principled stopping
+ * point -- raise until the other limit binds, then stop.
+ */
+const EXAMINE_MULTIPLE = 12
+
+/**
+ * `remaining` counts docs KEPT. `examined` counts files OPENED. They are different
+ * numbers because most files on this machine are filtered out after their title is read.
+ */
+type ScanBudget = { remaining: number; examined: number }
 const SEMANTIC_DOC_CAP = 120
 const SEMANTIC_MIN = 0.28
 const EMBED_MODEL = 'text-embedding-3-small'
 const EMBED_TIMEOUT_MS = 12_000
-const EMBED_BATCH = 64
+/**
+ * 128 inputs per request, and the requests go out TOGETHER.
+ *
+ * This loop used to `await` each batch in turn, so the embedding cost was a round trip
+ * per 64 docs -- about 1.7s of a 2.3s route, and it got worse as the collector started
+ * returning more docs (389 docs is 7 serialized trips at the old batch size, 4 at this
+ * one). Serialization was the expensive part, not the batch size, so both changed:
+ * fewer requests, and one round trip of latency instead of N.
+ */
+const EMBED_BATCH = 128
 
 export interface AgentSessionSearchHit extends AgentSessionRow {
   snippet: string
@@ -139,7 +190,9 @@ export async function defaultEmbedTexts(texts: string[]): Promise<number[][] | {
   if (!key) return { reason: 'no_session_embeddings' }
   if (texts.length === 0) return []
   const vectors: number[][] = new Array(texts.length)
-  for (let offset = 0; offset < texts.length; offset += EMBED_BATCH) {
+  const offsets: number[] = []
+  for (let offset = 0; offset < texts.length; offset += EMBED_BATCH) offsets.push(offset)
+  const failed = await Promise.all(offsets.map(async offset => {
     const slice = texts.slice(offset, offset + EMBED_BATCH).map(text => text.slice(0, HEAD_TEXT))
     try {
       const response = await fetch('https://api.openai.com/v1/embeddings', {
@@ -151,17 +204,19 @@ export async function defaultEmbedTexts(texts: string[]): Promise<number[][] | {
         body: JSON.stringify({ model: EMBED_MODEL, input: slice }),
         signal: AbortSignal.timeout(EMBED_TIMEOUT_MS),
       })
-      if (!response.ok) return { reason: 'embeddings_unreachable' }
+      if (!response.ok) return true
       const parsed = await response.json() as { data?: Array<{ embedding?: number[]; index?: number }> }
       const rows = Array.isArray(parsed.data) ? parsed.data : []
       for (const row of rows) {
         const index = typeof row.index === 'number' ? row.index : 0
         if (Array.isArray(row.embedding)) vectors[offset + index] = row.embedding
       }
+      return false
     } catch {
-      return { reason: 'embeddings_unreachable' }
+      return true
     }
-  }
+  }))
+  if (failed.some(Boolean)) return { reason: 'embeddings_unreachable' }
   if (vectors.some(row => !Array.isArray(row) || row.length === 0)) return { reason: 'embeddings_unreachable' }
   return vectors
 }
@@ -180,40 +235,53 @@ function pushDoc(docs: SearchDoc[], next: SearchDoc) {
   }
 }
 
-async function collectClaudeDocs(roots: AgentSessionRoots, docs: SearchDoc[], budget: { remaining: number }) {
+async function collectClaudeDocs(roots: AgentSessionRoots, docs: SearchDoc[], budget: ScanBudget) {
   const starred = await loadClaudeStarredIds(roots.claudeDesktopConfig)
+  // Stat every candidate and rank by recency BEFORE spending anything, the shape
+  // `collectCursorDocs` already used. Claude and Codex were the inconsistent ones: they
+  // walked in raw `readdir` order, so which sessions were reachable came down to
+  // filesystem layout and the newest transcript on disk was routinely never opened.
+  // Statting is not the expensive part -- measured at 41ms across 2,118 candidates.
+  const candidates: Array<{ file: string; native: string; folder: string; mtimeMs: number; birthtimeMs: number }> = []
   for (const folder of await dirents(roots.claudeProjects)) {
     const dir = join(roots.claudeProjects, folder)
     for (const name of await dirents(dir)) {
-      if (!CLAUDE_UUID_JSONL.test(name) || budget.remaining <= 0) continue
+      if (!CLAUDE_UUID_JSONL.test(name)) continue
       const file = join(dir, name)
       const st = await fileStat(file)
       if (!st?.isFile) continue
-      budget.remaining -= 1
-      const native = name.slice(0, -6)
-      const custom = await lastCustomTitle(file)
-      const first = await firstClaudeUserTitle(file)
-      const users = await transcriptHaystack('claude', file)
-      const title = custom || first || 'Claude session'
-      if (isKeepWarmSessionTitle(title)) continue
-      const haystack = clip(`${title}\n${custom || ''}\n${first || ''}\n${workspaceLabel(folder)}\n${users}`)
-      pushDoc(docs, {
-        title,
-        haystack,
-        row: {
-          session_id: native,
-          provider: 'claude',
-          display_label: title,
-          project: workspaceLabel(folder),
-          modified: isoFromMtime(st.mtimeMs),
-          created: isoFromMtime(st.birthtimeMs),
-          alive: false,
-          state: 'recent',
-          pinned: starred.has(native.toLowerCase()),
-        },
-      })
+      candidates.push({ file, native: name.slice(0, -6), folder, mtimeMs: st.mtimeMs, birthtimeMs: st.birthtimeMs })
     }
   }
+  candidates.sort((a, b) => b.mtimeMs - a.mtimeMs)
+  for (const candidate of candidates) {
+    if (budget.remaining <= 0 || budget.examined <= 0) break
+    budget.examined -= 1
+    const custom = await lastCustomTitle(candidate.file)
+    const first = await firstClaudeUserTitle(candidate.file)
+    const title = custom || first || 'Claude session'
+    // Both filters run BEFORE the doc budget is charged, and the haystack -- by far the
+    // most expensive read here -- runs only for a file we are actually keeping.
+    if (isKeepWarmSessionTitle(title)) continue
+    budget.remaining -= 1
+    const users = await transcriptHaystack('claude', candidate.file)
+    const haystack = clip(`${title}\n${custom || ''}\n${first || ''}\n${workspaceLabel(candidate.folder)}\n${users}`)
+    pushDoc(docs, {
+      title,
+      haystack,
+      row: {
+        session_id: candidate.native,
+        provider: 'claude',
+        display_label: title,
+        project: workspaceLabel(candidate.folder),
+        modified: isoFromMtime(candidate.mtimeMs),
+        created: isoFromMtime(candidate.birthtimeMs),
+        alive: false,
+        state: 'recent',
+        pinned: starred.has(candidate.native.toLowerCase()),
+      },
+    })
+  }
   for (const starredId of starred) {
     if (docs.some(doc => doc.row.provider === 'claude' && doc.row.session_id.toLowerCase() === starredId)) continue
     if (budget.remaining <= 0) break
@@ -221,10 +289,10 @@ async function collectClaudeDocs(roots: AgentSessionRoots, docs: SearchDoc[], bu
     if (!desktop) continue
     const st = await fileStat(desktop)
     if (!st?.isFile) continue
-    budget.remaining -= 1
     const head = peekClaudeDesktopHead(await readWindow(desktop, false))
     const title = head.title || 'Claude session'
     if (isKeepWarmSessionTitle(title)) continue
+    budget.remaining -= 1
     pushDoc(docs, {
       title,
       haystack: clip(`${title}\n${head.cwd}\n${workspaceLabel(head.cwd)}`),
@@ -243,23 +311,31 @@ async function collectClaudeDocs(roots: AgentSessionRoots, docs: SearchDoc[], bu
   }
 }
 
-async function collectCodexDocs(roots: AgentSessionRoots, docs: SearchDoc[], budget: { remaining: number }) {
+async function collectCodexDocs(roots: AgentSessionRoots, docs: SearchDoc[], budget: ScanBudget) {
   const names = await loadCodexThreadNames(roots.codexSessions)
   const pinned = await loadCodexPinnedIds(roots.codexSessions)
+  const candidates: Array<{ file: string; mtimeMs: number; birthtimeMs: number }> = []
   for (const file of await listCodexJsonlFiles(roots.codexSessions)) {
-    if (budget.remaining <= 0) break
     const st = await fileStat(file)
     if (!st?.isFile) continue
-    budget.remaining -= 1
+    candidates.push({ file, mtimeMs: st.mtimeMs, birthtimeMs: st.birthtimeMs })
+  }
+  candidates.sort((a, b) => b.mtimeMs - a.mtimeMs)
+  for (const { file, mtimeMs, birthtimeMs } of candidates) {
+    if (budget.remaining <= 0 || budget.examined <= 0) break
+    budget.examined -= 1
     const meta = await peekCodexMeta(file)
+    // Subagent rollouts are the Codex analogue of the keep-warm transcripts: 427 of 822
+    // on this machine per the 2026-08-18 census. Filtered before the doc budget is charged.
     if (!meta || meta.subagent) continue
     const name = file.split('/').pop() || file
     const native = meta.id || name.slice(0, -6)
     const thread = names.get(native) || ''
     const title = thread || meta.title || 'Codex session'
     if (isKeepWarmSessionTitle(title)) continue
+    budget.remaining -= 1
     const users = await transcriptHaystack('codex', file)
-    const created = meta.created || createdFromCodexFilename(name) || isoFromMtime(st.birthtimeMs)
+    const created = meta.created || createdFromCodexFilename(name) || isoFromMtime(birthtimeMs)
     pushDoc(docs, {
       title,
       haystack: clip(`${title}\n${thread}\n${meta.title}\n${meta.cwd}\n${users}`),
@@ -268,7 +344,7 @@ async function collectCodexDocs(roots: AgentSessionRoots, docs: SearchDoc[], bud
         provider: 'codex',
         display_label: title,
         project: workspaceLabel(meta.cwd),
-        modified: isoFromMtime(st.mtimeMs),
+        modified: isoFromMtime(mtimeMs),
         created,
         alive: false,
         state: 'recent',
@@ -305,11 +381,12 @@ async function collectCursorDocs(
   const ranked = [...byId.entries()].sort((a, b) => b[1].mtimeMs - a[1].mtimeMs)
   for (const [sessionDir, candidate] of ranked) {
     if (budget.remaining <= 0) break
-    budget.remaining -= 1
     const sidebar = composerNames.get(sessionDir) || ''
     const users = await transcriptHaystack('cursor', candidate.file)
     const title = sidebar || users.split('\n')[0] || 'Cursor session'
+    // Cursor already ranked, but it charged the budget before this filter too.
     if (isKeepWarmSessionTitle(title)) continue
+    budget.remaining -= 1
     pushDoc(docs, {
       title,
       haystack: clip(`${title}\n${sidebar}\n${candidate.project}\n${users}`),
@@ -335,8 +412,9 @@ export async function collectAgentSessionSearchDocs(
 ): Promise<SearchDoc[]> {
   const docs: SearchDoc[] = []
   const share = Math.max(1, Math.ceil(Math.min(cap, MAX_SCAN_FILES) / 3))
-  await collectClaudeDocs(roots, docs, { remaining: share })
-  await collectCodexDocs(roots, docs, { remaining: share })
+  const examined = share * EXAMINE_MULTIPLE
+  await collectClaudeDocs(roots, docs, { remaining: share, examined })
+  await collectCodexDocs(roots, docs, { remaining: share, examined })
   await collectCursorDocs(roots, docs, { remaining: share }, now)
   docs.sort((a, b) => (b.row.modified || '').localeCompare(a.row.modified || ''))
   return docs.slice(0, Math.max(1, Math.min(cap, MAX_SCAN_FILES)))

package/server/lib/agent-session-store.ts

@@ -97,10 +97,32 @@ export function isScratchCursorProject(folder: string): boolean {
 
 export const isSkippedCursorFolder = isScratchCursorProject
 
+/**
+ * Titles that belong to a MACHINE, not to a conversation Miles had.
+ *
+ * Measured 2026-08-18 by driving the real collector over the 1,296 Claude transcripts on
+ * this machine: of the 41 Claude docs it indexed, 28 were machine prompts -- 22 Slack
+ * Bridge proxy, 4 reply-with, 2 slack_search_users. Two thirds of Claude search was
+ * answering with scaffolding.
+ *
+ * Every `you are ...` prefix here is ANCHORED to a specific known caller. A bare
+ * `startsWith('you are')` would also swallow any human sentence beginning that way
+ * ("You are right that ..."), and the anchored pair covers the measured volume without
+ * that risk.
+ *
+ * Shared by all four search collectors AND the session-list path, so a title added here
+ * also stops appearing as a row in COS Control's session list. That is the intent.
+ */
 export function isKeepWarmSessionTitle(title: string): boolean {
   const t = title.trim().toLowerCase()
   if (t === 'ready') return true
-  return t.startsWith('this is an automated local readiness check')
+  if (t.startsWith('this is an automated local readiness check')) return true
+  if (t.startsWith('you are the cos slack bridge proxy')) return true
+  if (t.startsWith('you are a post-processing editor')) return true
+  if (t.startsWith('call mcp__claude_ai_slack__slack_search_users')) return true
+  if (/^reply with (exactly|the single word)\b/.test(t)) return true
+  if (t === 'say ok' || t === 'say: ok' || t === 'reply ok') return true
+  return false
 }
 
 /**