@gotcos/glasses-server 6.21.14 → 6.21.17

package/CHANGELOG.md

@@ -1,3 +1,94 @@
+## 6.21.17
+
+- Voice profiles now give up their WEAKEST sample, not their oldest. Eviction is
+  by provenance tier — automatic, then unknown, then attendee-metadata, then
+  identifier-labelled — and human-supplied samples (manual, ext-retroactive,
+  g2-enrollment, correction) are protected.
+- Measured on the live store before changing anything, not assumed:
+  * 61 of 77 profiles were ALREADY at the 20-sample cap, so every correction
+    cost a sample.
+  * The owner profile driving owner detection read: fireflies 10, g2-training 9,
+    unknown 1 — ZERO human-verified samples.
+  * FOUR profiles at cap would have lost a human sample to plain FIFO while
+    weaker samples sat untouched, including one whose only deliberate enrollment
+    was its oldest sample with 17 weaker samples available.
+- Cap raised 20 -> 40. Search latency measured at 1 us for 20, 40 AND 80 samples
+  per speaker across 77 speakers, so the old cap defended nothing measurable.
+  Twenty extra slots each is about 1.2 MB.
+- Corrections are capped at HALF a profile. They come from the acoustically hard
+  tail — the segments the identifier got wrong — so a profile of nothing but
+  corrections has a centroid displaced from how the speaker actually sounds. At
+  quota a new correction replaces the OLDEST CORRECTION, never a typical-voice
+  sample.
+- Every eviction logs which sample went and why; none disappear silently.
+- `/api/health` gains `voice_provenance`: tier totals, profiles at cap, and
+  `noHumanSample` — the profiles trained entirely on labels the system chose for
+  itself. That number was previously invisible.
+- An unrecognised or missing provenance string classifies as `unknown`, the
+  weakest tier, never as trusted. A sample whose provenance was lost must not
+  inherit the protection given to one a human supplied.
+
+## 6.21.16
+
+- `POST /api/meeting/:sessionId/relabel` — correct who a voice was in ONE
+  meeting. Body `{ from, to, chunks?, confirm | dryRun, force? }`.
+- Per-meeting by design, not a global merge. Miles: "changing it doesn't mean
+  that all previous chunks should also be moved. It should be meeting by
+  meeting, with the goal of hardening or refining the voice profiles." The
+  identifier mishearing a voice in one room is not evidence that every past
+  attribution was wrong.
+- Fails closed like the other destructive endpoints: no `confirm` returns 400
+  `confirmation required` with a full preview of what would change.
+- THE LEDGER IS WRITTEN FIRST. An intent row lands before any file is touched,
+  and a ledger that cannot be written aborts the correction with the files
+  untouched — an unrecorded mutation is the failure this exists to prevent. A
+  process dying mid-rewrite therefore leaves a visible pending correction, and a
+  later correction on the same meeting refuses with 409 until `force`.
+- A PARTIAL relabel (explicit `chunks`) never touches the meeting markdown.
+  Measured on a real meeting: 135 sidecar chunks collapse to 46 speaker runs
+  while the markdown carries 70 turns, and they disagree on who spoke — the
+  transcript comes from a different segmentation pass, so there is no
+  chunk-index-to-turn mapping. Rewriting by label there would relabel turns the
+  human never selected. The response says why rather than staying silent.
+- Narrative prose is NEVER rewritten — only reported as `proseStale` with the
+  forms found. Verified on a real scribe: 6 of 12 speakers are referred to by
+  bare first name in the summary, and with two Kyles, two Jacobuses and two
+  Chrises in this org a first-name substitution would rewrite sentences about
+  someone else. The confirmation message says this before a human commits.
+- `/api/health` reports `speaker_corrections` (sessions, applied, pending,
+  failed). `pending` is the one to watch.
+
+## 6.21.15
+
+- Persist the per-chunk voiceprint embedding, so a speaker correction can become
+  training instead of only a transcript edit. `identifySpeaker` computed a
+  192-dim embedding for every chunk, used it to pick a name, and discarded it —
+  nothing downstream kept it, so a human saying "those six segments were
+  actually me" had no acoustic evidence left to learn from. Embeddings now land
+  in `chunk-embeddings/<sessionId>.jsonl`, keyed by chunk index, which is the
+  join key to the existing chunk sidecar.
+- This only works FORWARD. Meetings recorded before this ships can be relabelled
+  but can never harden a profile, because their embeddings are already gone.
+- `Ext` is deliberately included. An unidentified voice is the one case with no
+  other route to being trained at all.
+- RETENTION IS 14 DAYS, not the 8 hours used for audio snippets
+  (`COS_CHUNK_EMBEDDING_TTL_DAYS` to change it). Two reasons, both worth Miles
+  overriding if he disagrees: 8 hours cannot survive a weekend, so a Friday
+  meeting could never be corrected on Monday; and an embedding is not audio — it
+  is a non-invertible timbre vector that cannot be played back, so the privacy
+  argument behind a short audio window does not carry across unchanged.
+- Master switch `COS_CHUNK_EMBEDDINGS=0`. Default ON, because with it off a
+  correction can never train anything. The write is append-only JSONL, never
+  throws, and is skipped silently on failure: losing an embedding must never
+  cost a chunk of transcript.
+- Reject a wrong-DIMENSION vector on read rather than averaging it into a
+  profile. A 64-dim row decodes perfectly cleanly and nothing downstream would
+  notice, which is precisely why it needs an explicit gate.
+- `/api/health` reports `chunk_embeddings` (sessions, bytes, TTL, oldest age) so
+  the loop can be seen banking evidence rather than assumed to be.
+- Size, measured not estimated: ~1.1 KB per chunk, so a 400-chunk meeting is
+  under 500 KB.
+
 ## 6.21.14
 
 - Carry `sessionId` on the COS operations meetings list as well. 6.21.13 added it

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.21.14",
+  "version": "6.21.17",
   "description": "COS Glasses — self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
   "type": "module",
   "bin": {

package/server/lib/chunk-embedding-store.ts

@@ -0,0 +1,260 @@
+// Per-chunk voiceprint embeddings, kept so a correction can become training.
+//
+// WHY THIS EXISTS. identifySpeaker() computes a 192-dim embedding for every
+// audio chunk, uses it to pick a name, and throws it away. Nothing downstream
+// keeps it: the chunk sidecar stores text/speaker/elapsed/similarity, and the
+// calibration log stores ts/speaker/similarity/matched. So when a human later
+// says "those six segments were actually me", the system can fix the transcript
+// but cannot learn anything from the correction — the acoustic evidence is gone.
+//
+// Persisting it turns a correction into real training data:
+//   correction → these chunks' embeddings → enrollEmbedding(name, …, 'correction:<id>')
+//
+// This only works forward. A meeting recorded before this ships can be
+// relabelled but can never harden a profile, because its embeddings were
+// discarded. That is the cost of every week this is not running.
+//
+// NOT AUDIO. An embedding is 192 floats describing vocal timbre. It cannot be
+// played back and speech cannot be reconstructed from it, which is why its
+// retention is treated separately from audio snippets — see RETENTION below.
+
+import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, statSync, unlinkSync } from 'node:fs'
+import { join, resolve } from 'node:path'
+import { dataPath } from './data-dir.js'
+
+export const CHUNK_EMBEDDING_DIR = 'chunk-embeddings'
+
+/** Expected dimension for the shipped 3dspeaker model. A row of any other
+ *  length is recorded but flagged, never silently averaged into a profile. */
+export const EXPECTED_EMBEDDING_DIM = 192
+
+/**
+ * RETENTION — 14 days by default, overridable with COS_CHUNK_EMBEDDING_TTL_DAYS.
+ *
+ * This is deliberately longer than the 8 hours on record for audio snippets, and
+ * the difference is the point: 8 hours cannot survive a weekend, so a Friday
+ * meeting could never be corrected on Monday and the compounding loop would
+ * only ever work on same-day review. Embeddings are also not audio — they are
+ * non-invertible timbre vectors, so the privacy argument that motivates a short
+ * audio window does not carry across unchanged.
+ *
+ * Flagged for Miles rather than assumed: if he wants these on the 8-hour clock
+ * too, set the env var to a fraction and the correction loop becomes same-day
+ * only.
+ */
+export function chunkEmbeddingTtlMs(): number {
+  const raw = Number(process.env.COS_CHUNK_EMBEDDING_TTL_DAYS)
+  const days = Number.isFinite(raw) && raw > 0 ? raw : 14
+  return days * 24 * 60 * 60 * 1000
+}
+
+/** Master switch. Default ON: with this off, a correction can never train
+ *  anything, which is the whole reason the store exists. */
+export function chunkEmbeddingsEnabled(): boolean {
+  return process.env.COS_CHUNK_EMBEDDINGS !== '0'
+}
+
+export interface ChunkEmbeddingRow {
+  /** Chunk index within the session. This is the JOIN KEY to the chunk sidecar,
+   *  which already carries `elapsed`, `text`, and the label. Time is deliberately
+   *  NOT duplicated here: the only value available at capture time is the chunk's
+   *  own duration, and storing that as an offset would be a wrong number in a
+   *  field named like a right one. */
+  i: number
+  /** The label the identifier chose at capture time — what a correction corrects. */
+  speaker: string
+  similarity: number
+  embedding: Float32Array
+}
+
+/**
+ * Base64 of the raw little-endian float32 bytes.
+ *
+ * Chosen over a JSON number array for size and fidelity: 192 floats as JSON is
+ * ~1.5 KB and rounds, while base64 is 1,024 chars and is exact. Over a
+ * 400-chunk meeting that difference is roughly 200 KB.
+ */
+export function encodeEmbedding(embedding: Float32Array): string {
+  return Buffer.from(embedding.buffer, embedding.byteOffset, embedding.byteLength).toString('base64')
+}
+
+export function decodeEmbedding(encoded: string): Float32Array | null {
+  try {
+    const buf = Buffer.from(encoded, 'base64')
+    if (buf.byteLength === 0 || buf.byteLength % 4 !== 0) return null
+    // Copy rather than aliasing the Buffer's pool. Verified, not assumed: a
+    // 768-byte base64 decode lands at an offset inside an 8192-byte pool slab.
+    // Node's pool offset only advances, so a later allocation cannot corrupt an
+    // existing view — the cost is RETENTION. An aliased view pins the whole 8 KB
+    // slab to keep its 768 bytes, so holding a few hundred embeddings would pin
+    // ~10x the memory they occupy. The copy owns exactly its own bytes.
+    const copy = new ArrayBuffer(buf.byteLength)
+    Buffer.from(copy).set(buf)
+    return new Float32Array(copy)
+  } catch {
+    return null
+  }
+}
+
+function sessionFile(sessionId: string): string | null {
+  // Session ids come from the client; keep them to the shape the meeting store
+  // already enforces so nothing can escape the directory.
+  if (!/^[A-Za-z0-9:_-]{3,96}$/.test(sessionId)) return null
+  const dir = dataPath(CHUNK_EMBEDDING_DIR)
+  const path = join(dir, `${sessionId.replace(/:/g, '_')}.jsonl`)
+  return resolve(path).startsWith(resolve(dir) + '/') ? path : null
+}
+
+/**
+ * Append one chunk's embedding. JSONL, not a JSON array, so each write is a
+ * single append with no read-modify-write of a growing file — a crash can lose
+ * at most the final partial line, and readers skip it.
+ *
+ * Never throws: this runs inside the live capture path and losing an embedding
+ * must never cost a chunk of transcript.
+ */
+export function appendChunkEmbedding(sessionId: string, row: ChunkEmbeddingRow): boolean {
+  if (!chunkEmbeddingsEnabled()) return false
+  const path = sessionFile(sessionId)
+  if (!path) return false
+  try {
+    mkdirSync(dataPath(CHUNK_EMBEDDING_DIR), { recursive: true, mode: 0o700 })
+    const line = JSON.stringify({
+      i: row.i,
+      speaker: row.speaker,
+      similarity: Math.round(row.similarity * 1000) / 1000,
+      dim: row.embedding.length,
+      v: encodeEmbedding(row.embedding),
+    })
+    appendFileSync(path, line + '\n', { mode: 0o600 })
+    return true
+  } catch {
+    return false
+  }
+}
+
+export interface ChunkEmbeddingReadResult {
+  rows: ChunkEmbeddingRow[]
+  /** Lines that could not be parsed or decoded. Reported rather than hidden: a
+   *  correction built from a partial read would train on the wrong segments. */
+  unusable: number
+  /** True when the file is absent — the meeting predates this store, or the
+   *  retention window has passed. Distinct from "present but empty". */
+  missing: boolean
+}
+
+export function readChunkEmbeddings(sessionId: string): ChunkEmbeddingReadResult {
+  const path = sessionFile(sessionId)
+  if (!path || !existsSync(path)) return { rows: [], unusable: 0, missing: true }
+  let raw: string
+  try {
+    raw = readFileSync(path, 'utf-8')
+  } catch {
+    return { rows: [], unusable: 0, missing: true }
+  }
+  const rows: ChunkEmbeddingRow[] = []
+  let unusable = 0
+  for (const line of raw.split('\n')) {
+    if (line.trim() === '') continue
+    try {
+      const o = JSON.parse(line) as Record<string, unknown>
+      const embedding = typeof o.v === 'string' ? decodeEmbedding(o.v) : null
+      if (!embedding || typeof o.i !== 'number' || typeof o.speaker !== 'string') { unusable++; continue }
+      // A vector of the wrong dimension decodes cleanly and would be averaged
+      // into a profile as if it were a voice. The model changed, or the row is
+      // corrupt; either way it must never reach enrollment.
+      if (embedding.length !== EXPECTED_EMBEDDING_DIM) { unusable++; continue }
+      rows.push({
+        i: o.i,
+        speaker: o.speaker,
+        similarity: typeof o.similarity === 'number' ? o.similarity : 0,
+        embedding,
+      })
+    } catch {
+      unusable++
+    }
+  }
+  return { rows, unusable, missing: false }
+}
+
+/** The embeddings for a given label — the input to a correction. */
+export function chunkEmbeddingsForSpeaker(sessionId: string, speaker: string): ChunkEmbeddingRow[] {
+  return readChunkEmbeddings(sessionId).rows.filter(r => r.speaker === speaker)
+}
+
+/** The embeddings for an explicit set of chunk indices, for a partial
+ *  correction where only some of a voice's segments were misattributed. */
+export function chunkEmbeddingsForIndices(sessionId: string, indices: number[]): ChunkEmbeddingRow[] {
+  const wanted = new Set(indices)
+  return readChunkEmbeddings(sessionId).rows.filter(r => wanted.has(r.i))
+}
+
+export interface SweepResult {
+  removed: string[]
+  retained: string[]
+}
+
+/**
+ * Delete session files past the retention window.
+ *
+ * A file whose mtime cannot be read is RETAINED — treating a failed stat as
+ * "ancient" would delete the evidence a pending correction depends on.
+ */
+export function sweepExpiredChunkEmbeddings(nowMs: number, ttlMs = chunkEmbeddingTtlMs()): SweepResult {
+  const dir = dataPath(CHUNK_EMBEDDING_DIR)
+  const result: SweepResult = { removed: [], retained: [] }
+  if (!existsSync(dir)) return result
+  let names: string[]
+  try {
+    names = readdirSync(dir).filter(n => n.endsWith('.jsonl'))
+  } catch {
+    return result
+  }
+  for (const name of names) {
+    const path = join(dir, name)
+    let mtimeMs = 0
+    try { mtimeMs = statSync(path).mtimeMs } catch { result.retained.push(name); continue }
+    if (mtimeMs > 0 && Number.isFinite(mtimeMs) && nowMs - mtimeMs > ttlMs) {
+      try { unlinkSync(path); result.removed.push(name) } catch { result.retained.push(name) }
+    } else {
+      result.retained.push(name)
+    }
+  }
+  return result
+}
+
+/** Counts for /api/health, so an operator can see the loop is banking evidence
+ *  rather than having to trust that it is. */
+export function chunkEmbeddingStoreStats(): {
+  enabled: boolean
+  sessions: number
+  bytes: number
+  ttlDays: number
+  oldestAgeHours: number | null
+} {
+  const dir = dataPath(CHUNK_EMBEDDING_DIR)
+  const ttlDays = Math.round((chunkEmbeddingTtlMs() / (24 * 60 * 60 * 1000)) * 10) / 10
+  if (!existsSync(dir)) {
+    return { enabled: chunkEmbeddingsEnabled(), sessions: 0, bytes: 0, ttlDays, oldestAgeHours: null }
+  }
+  let sessions = 0, bytes = 0, oldest = Number.POSITIVE_INFINITY
+  try {
+    for (const name of readdirSync(dir).filter(n => n.endsWith('.jsonl'))) {
+      try {
+        const st = statSync(join(dir, name))
+        sessions++
+        bytes += st.size
+        oldest = Math.min(oldest, st.mtimeMs)
+      } catch { /* skip unreadable */ }
+    }
+  } catch { /* report what we have */ }
+  return {
+    enabled: chunkEmbeddingsEnabled(),
+    sessions,
+    bytes,
+    ttlDays,
+    oldestAgeHours: Number.isFinite(oldest)
+      ? Math.round(((Date.now() - oldest) / 3_600_000) * 10) / 10
+      : null,
+  }
+}

package/server/lib/embedding-eviction.ts

@@ -0,0 +1,133 @@
+// Which sample gets dropped when a voice profile is full.
+//
+// WHY THIS IS NOT FIFO. Measured on the live store (2026-08-06): 61 of 77
+// profiles are already at the 20-sample cap, so every correction costs a sample
+// today. And the samples are not interchangeable — the owner's own profile, the
+// one driving owner detection, reads:
+//
+//   MU: 20/20 — fireflies 10, g2-training 9, unknown 1     (zero human-verified)
+//
+// `fireflies` labels come from meeting ATTENDEE METADATA, never acoustically
+// verified. That is exactly the chain that manufactured the phantom "Erick
+// Hernandez" profile: 3 fireflies seeds compounding to 18 via g2-training. Under
+// plain FIFO, a human correction could evict the single `manual` sample while
+// ten unverified metadata seeds sat untouched. Age is the wrong axis. Who
+// supplied the label is the right one.
+//
+// THE COUNTER-ARGUMENT, honoured by the quota below. Corrections are drawn from
+// the acoustically HARD tail — the chunks the identifier got wrong. A profile
+// made entirely of hard cases has a centroid displaced from the speaker's
+// typical voice, which could make routine matching worse. So corrections are
+// protected from eviction but capped at half a profile, keeping typical-voice
+// mass in the centroid.
+
+/** Ordered weakest-first: eviction walks this list and takes from the first
+ *  non-empty tier. */
+export const EVICTION_ORDER = ['automatic', 'unknown', 'metadata', 'assisted', 'human'] as const
+
+export type ProvenanceTier = typeof EVICTION_ORDER[number]
+
+/**
+ * Classify a sample by WHO supplied its label — not by how the audio arrived.
+ *
+ * Source strings appear both bare (`g2-training`) and prefixed
+ * (`auto:meeting_1774377156653`), both verified in the live store, so matching is
+ * on the prefix before any separator.
+ */
+export function provenanceTier(source: string | undefined | null): ProvenanceTier {
+  const head = String(source ?? '').trim().split(/[:_]/)[0].toLowerCase()
+  switch (head) {
+    // A human deliberately supplied the audio or named the voice.
+    case 'manual':
+    case 'correction':
+    case 'ext-retroactive':
+    case 'g2-enrollment':
+      return 'human'
+    // The identifier chose the label; a human approved the batch run.
+    case 'g2-training':
+      return 'assisted'
+    // Label came from attendee metadata and was never acoustically verified.
+    case 'fireflies':
+      return 'metadata'
+    // No human involvement at any point.
+    case 'auto':
+      return 'automatic'
+    default:
+      return 'unknown'
+  }
+}
+
+/** True for a sample created by a human correcting a specific meeting. */
+export function isCorrection(source: string | undefined | null): boolean {
+  return String(source ?? '').trim().toLowerCase().startsWith('correction')
+}
+
+/**
+ * How many of a profile's slots corrections may occupy.
+ *
+ * Half, rounded down. Above this the profile's centroid is dominated by the
+ * acoustically difficult segments that needed correcting, which is not what the
+ * speaker usually sounds like.
+ */
+export function correctionQuota(cap: number): number {
+  return Math.max(1, Math.floor(cap / 2))
+}
+
+export interface EvictionChoice {
+  /** Index into embeddings[]/sources[] to drop. */
+  index: number
+  tier: ProvenanceTier
+  /** Human-readable justification, logged so an eviction is never silent. */
+  reason: string
+}
+
+/**
+ * Choose which sample to drop to make room for `incomingSource`.
+ *
+ * Returns null when the incoming sample should be REFUSED rather than evicting
+ * anything — which happens only when a correction would exceed its quota and
+ * every correction present is newer than it could justify replacing.
+ */
+export function chooseEviction(
+  sources: Array<string | undefined | null>,
+  incomingSource: string,
+  cap: number,
+): EvictionChoice | null {
+  // No empty-array guard: the tier walk below returns null for an empty
+  // profile on its own (every findIndex is -1), and the quota branch cannot
+  // fire because the quota floor is 1. Verified by mutation — an explicit
+  // guard here is unreachable in effect.
+  const correctionIdx = sources.map((s, i) => ({ s, i })).filter(x => isCorrection(x.s)).map(x => x.i)
+
+  // A correction arriving at or above quota replaces the OLDEST CORRECTION, not
+  // a typical-voice sample. Corrections stay protected as a class while the
+  // newest human evidence still lands, and the centroid keeps its ordinary mass.
+  if (isCorrection(incomingSource) && correctionIdx.length >= correctionQuota(cap)) {
+    return {
+      index: correctionIdx[0],
+      tier: 'human',
+      reason: `corrections at quota (${correctionIdx.length}/${correctionQuota(cap)}) — replacing the oldest correction`,
+    }
+  }
+
+  for (const tier of EVICTION_ORDER) {
+    const idx = sources.findIndex(s => provenanceTier(s) === tier)
+    if (idx === -1) continue
+    // Reached only when the profile is entirely human-supplied. Age decides,
+    // because there is no weaker sample to give up.
+    const reason = tier === 'human'
+      ? 'every sample is human-supplied — dropping the oldest'
+      : `dropping the oldest ${tier} sample (${String(sources[idx])}) rather than a stronger one`
+    return { index: idx, tier, reason }
+  }
+  return null
+}
+
+/** Tier counts for a profile, for logs and health output. */
+export function tierBreakdown(sources: Array<string | undefined | null>): Record<ProvenanceTier, number> {
+  const out: Record<ProvenanceTier, number> = {
+    human: 0, assisted: 0, metadata: 0, automatic: 0, unknown: 0,
+  }
+  for (const s of sources) out[provenanceTier(s)]++
+  return out
+}