@gotcos/glasses-server 6.99.12 → 6.99.16

package/CHANGELOG.md

@@ -1,3 +1,81 @@
+## 6.99.16 - The COS thread: one size-capped conversation behind the COS chat's Ask, stored to recover and search (private)
+
+Merged in: claude/iphone-dictionary (94323de, from the COS Glasses QA session; its own full suite on 94323de: 13,041 passed, 0 failed).
+
+- Default-on for iPhone microphone recordings: iPhone recordings now transcribe live with the same Whisper as the glasses, so your dictionary shows while you record. Saved transcripts are unchanged (strict Parakeet, as for G2 saves). If live Whisper fails, the chunk errors as retryable `local_asr_unavailable`; it never falls back to Parakeet live. The capability keeps `asrModel: 'parakeet-ultra'` (glasses clients 677 and older check it before offering the iPhone mic) and adds `liveAsrModel: 'server-whisper'`.
+- iPhone saved transcripts can use the dictionary when Second opinion is on (`phoneHqPlan` in meeting-batch-transcribe.ts): Parakeet stays the strict primary and the dictionary-prompted Whisper Large-v3 fills in names it missed. Off stays off; Miles's Mac has it off, so this changes nothing there until he turns it on.
+
+2026-11-05. Private build for one Mac, on top of 6.99.15. Pairs with COS Control 0.5.374 (written against this contract), and works with any COS Control that pairs with 6.99.15. Nothing changes for the glasses app. Everything here exists only while Drop it is on.
+
+Miles, 11/05 22:55: "As long as we're storing so we can easily recover and search for them, I like this plan let's build and deploy."
+
+- **Why.** Every Ask from the COS chat started a brand-new session, and a fresh session already starts at about 100K tokens of context (CLAUDE.md, gotchas, memory, tool definitions): 101,720 and 103,116 on the first call, read from two session logs on 11/05. Continuing a session while its prompt cache is warm is cheaper than starting fresh; what makes a forever thread expensive is its size. So the chat keeps one thread and continues the same session until that session's context passes a cap (`COS_THREAD_CAP_TOKENS`, a whole number from 20,000 to 1,000,000, default 150,000: about 50K of conversation on top of the baseline). The next Ask then starts a fresh session carrying a short summary.
+- **Routes** (`server/routes/cos-thread.ts`, logic in `server/lib/cos-thread.ts`), under the token-auth `/api` mount, `Cache-Control: private, no-store`, and gated like the Drop it routes: while Drop it is off, every one answers 404 `drop_it_off` and no file is created.
+  - `GET /api/cos-thread`: `{ protocol: 1, capTokens, thread: { id, startedAt, lastAskAt, asks, segments, sessionId, provider, contextTokens }, next: { mode, sessionId, reason, digest } }`. `reason` is `continue` (the session is known and its context is at or under the cap), `empty` (no session yet), `size` (over the cap), `missing_session` (the conversation is gone, as after the midnight archive, or its log cannot be found) or `manual` (right after New thread). Only `continue` continues; every other reason is `fresh` with no sessionId and the summary.
+  - `POST /api/cos-thread/new {}`: closes the thread (a `rollover` row with reason `manual` and the summary it carries, then a `close` row) and starts a new one with no session. Its first Ask still carries the summary, so it knows the last thread and the day's drops.
+  - `POST /api/cos-thread/turn { clientId, sessionId, query, answer?, messageNo?, provider? }`: records one Ask, idempotent on clientId. The first one adopts its session; a different session than the thread's is a rollover (a `rollover` row with the reason GET would have given and the summary) onto a new segment of the same thread. Anything out of contract is 400 `invalid_turn` and writes nothing. Answers the GET shape.
+  - `GET /api/cos-thread/search?q&limit` (q 2 to 200 characters, limit 1 to 50, default 20): `{ hits: [{ threadId, sessionId, at, query, answerHead, messageNo }], scanned }`, newest first, case-insensitive over the question and the answer's head.
+  - `GET /api/cos-thread/threads?limit`: `{ threads: [{ id, startedAt, endedAt, asks, segments, title }] }`, newest first; the title is the first question, cut to 80 characters. A thread with no asks is listed only while it is the current one.
+  - `/api/health` and `/api/models`: `capabilities.dropIt.thread: { protocol: 1 }`, only while Drop it is on. Every other dropIt key is unchanged.
+- **The summary** is mechanical (no model call), at most 6,000 characters, headed "Earlier in this COS thread (a summary carried over; ask COS to search History or Messages for more):": the thread's last 8 asks, oldest first (question cut to 200 characters, the answer's head to 300, with the message number when known), then up to 12 drops created since the thread started or in the last 24 hours, whichever is earlier, with where each was filed. A turn recorded without its answer takes it from the live conversation when it is still there. Over 6,000 characters, the oldest drops go first, then the oldest asks.
+- **Context size** is read from the session's Claude Code log, `~/.claude/projects/<any project>/<id>.jsonl` (`COS_CLAUDE_PROJECTS_DIR` overrides it, `CLAUDE_CONFIG_DIR` is honored), found through the CLI session the bridge holds for the COS conversation, or the one recorded with the last turn after a restart. Only the last 512 KB is read; the last non-sidechain assistant message's `input_tokens + cache_creation_input_tokens + cache_read_input_tokens` is the context. A log with no readable usage continues with `contextTokens` null. Checked on this Mac against a 412 MB live log: 4 ms to find, tail read only. A provider with no Claude log (Codex, Cursor) reads as `missing_session`, so it starts fresh with the summary each time rather than growing without a cap.
+- **Stored to recover and search**: `DATA_DIR/cos-thread/ledger.jsonl`, append-only and fsync'd, keeps every row (`thread`, `turn`, `rollover`, `close`), and `state.json`, written atomically after each append, is a snapshot of the current thread. When the snapshot is missing, corrupt or behind the ledger, the state is rebuilt from the ledger by the same code that builds it live. A torn last row is skipped and never swallows the next. The last 5,000 asks stay in memory for search. Directory 0700, files 0600. Both the log read and the ledger append open with O_NOFOLLOW and O_NONBLOCK, so a symlink or a FIFO planted at either path can never be followed or stop the server.
+- **COS Control Web** (`panes/cos-chat.js`), only when `/api/health` advertises `dropIt.thread`: an Ask from the chat's own composer reads `GET /api/cos-thread` first. `continue` sends that sessionId. `fresh` sends none and carries the summary as the Ask's `reference`, the existing query-jobs field for quoted context, so the question in Messages is only what was typed. While an Ask in the thread is still answering, the next one continues its conversation. When an Ask finishes, the page posts its turn once, with the answer and message number. A Reply in the message reader keeps its own turn's conversation, as before. The header gets "New thread" beside "Mark all read", and one quiet line under the title: "Continues your COS thread · N asks", or after a rollover or New thread, "New thread. It carries a short summary of the last one." Without the capability the per-day conversation stays, unchanged.
+- `DropService.recentDrops(since, limit)` reads the drop store for the summary; it never touches Work.
+- **Fix: Ask on COS Control Web read as off while durable query jobs were on.** The page looked for the switch in `capabilities.durableQueryJobs` (where `/api/models` carries it) or at the top of the body, but `/api/health` carries it at `features.durableQueryJobs`, so the chat said "Ask needs the durable query jobs switch on your Mac." on Miles's Mac with jobs on. It now reads `features.durableQueryJobs` too. The web Ask still sends `source: 'web'`, so the server fills in the message era and number as before (a bare POST without it answers 409 `message_era_mismatch`, as it always has).
+- Tests: `server/lib/cos-thread.test.ts` (every reason, the cap edge: exactly the cap continues and one over is `size`, idempotent turns, rollover rows, the CLI session recorded across a restart, the summary's bounds and drop window, the ledger rebuild from a missing, corrupt or stale snapshot, a torn row, search, the threads list, validation, file modes), `server/routes/cos-thread.test.ts` (the shapes over HTTP, 400s, `drop_it_off` with no file created, 503 when the store cannot open, the health capability on and off), `server/web-control/cos-thread-web.test.ts` (continue sends the sessionId, fresh sends none and carries the summary, the turn is posted once, a second Ask while one runs, New thread posts `/new`, the old rule without the capability, and Ask on for a health body shaped like the live one), and a `recentDrops` case in `drop-service.test.ts`. Mutation-checked: the cap comparison, the idempotency check, the rollover on a different session, the `drop_it_off` gate, the capability only while on, the web continue, the once-only turn post, and the `features.durableQueryJobs` read.
+- Default-on changes: none for anyone with Drop it off (the default). With Drop it on, the COS chat on the web continues one thread instead of one conversation per day, and `/api/health` gains `dropIt.thread`.
+
+## 6.99.15 - Messages cleanup on COS Control Web; the pinned COS line never says its title twice (private)
+
+2026-11-05. Private build for one Mac, on top of 6.99.14. Pairs with COS Control 0.5.374 (the same Messages cleanup on the Mac), and works with any COS Control that pairs with 6.99.14.
+
+Miles, 11/05 21:15, with a screenshot of Messages: "clean up the UI/UX on this, make sure it's clean on the browser and the desktop version".
+
+- The pinned COS row's line (`/api/inbox/summary` `lastLine`, `pinLine` in `server/lib/drop-service.ts`): a follow-up's line already names its card, so "Sprocket Rocket AI Unified Nav Approved (G2). 2 follow-ups from Sprocket Rocket AI Unified Nav Approved (G2)." now reads "2 follow-ups from Sprocket Rocket AI Unified Nav Approved (G2).". A title the line does not name still leads it, with one period. Both COS Control and the web show this line.
+- COS Control Web, Messages (`panes/messages.js`, `control.css`): each row leads with the ask, one line cut by width (was 57 characters), with its time beside it; the answer's two-line preview under it; one quiet line of facts (`#442`, Routine or Task, the model by name, attachments with their icon and "1 image"). No per-row glyph, no "ROUTINE and TASK" legend, no visible "Message view" label. The header says "30 messages, newest first". Previews put a space between progress sentences that ran together ("now.Reading").
+- COS Control Web, the COS chat (`panes/cos-chat.js`): a P2 follow-up that carries its own Dismiss no longer gets a second one. The pinned row's time is in the body font.
+- Tests: `drop-service.test.ts` covers `pinLine` (mutation-checked: always prefixing the title fails it); `cos-chat-p2.test.ts` counts one Dismiss on a follow-up that carries one, and one on a question that does not (mutation-checked).
+
+## 6.99.14 - The live meeting view on COS Control Web stops jumping, and gets the recording bar and Add file (private)
+
+2026-11-05. Private build for one Mac, on top of 6.99.13. Web only: no server route, no store and no glasses change. Pairs with any COS Control that pairs with 6.99.13.
+
+Miles, 11/05: "It currently keeps CLS jumping when the new transcription lines come in. We also don't have the ability to add rich text [files] or see the voice histogram coming in." The target is the Mac's live recording bar.
+
+- **New lines no longer move the page.** The cause: the live transcript was part of the page's own scroll, so following the newest line meant scrolling the whole page on every update. The title, the status, the Stop button and, on a phone, the Ask panel moved with each line. Measured in headless Chromium on 150 real-sized chunks with 20 lines appended one per poll: the page moved 1,279 px at 390 px wide and 1,002 px at 1280 px wide. Now the transcript scrolls inside its own box, as Control's does: following keeps the box on its newest line and the page moves 0 px at both widths. Reading higher up, a new line lands below you and nothing moves (0 px); turns above you that the speaker pass rewrites longer are held by the pane's own anchor, on every browser (iOS Safari has no scroll anchoring; the box sets `overflow-anchor: none` so Chrome's never corrects twice). A scroll write happens only when a turn above you changed height, so a fling on the iPhone is never stopped. Jump to live floats over the box's bottom edge, so showing it moves nothing.
+  - The layout-shift API reported 0 for this before and after: a scroll is not a layout shift, which is why the jump never showed up as CLS. (The 0.0007 it reports at 1280 px on load, before and after, is the app rail, not this pane.) The numbers above are page movement in CSS pixels, the user's unit.
+  - The header is the same height whatever it says (the status and duration lines stay one line), and everything that changes height while it records sits under the transcript, never above it. Before, the Files row landing with the Ask history pushed the transcript down 52 px; now 0 px.
+- **The recording bar, from the Mac's.** Under the transcript, one 50 px row: the red dot, the elapsed time on its own one-second clock (m:ss, then h:mm:ss; the row is not rebuilt each second), the level strip, Add file and Stop and save. Under 560 px of bar Add file is the paperclip alone; under 340 px the still glyph takes the strip's place. Copy transcript and Copy full context stay under it.
+  - The strip is Control's glasses level strip (`LiveChunkLevelStripView`): one bar per chunk from the live read's `levels`, the two numbers per chunk the quiet gate has kept on the active-session record since 6.87.0 (RMS on a dBFS scale, and Silero speech seconds). Newest at the right, a dot where a chunk held under 0.5 speech seconds (or level 0.27 and under without them), the oldest clipped and faded at the left. It trails the room by a chunk and says so to VoiceOver. Reduce Motion: one still bar for the newest chunk. It comes from the poll the page already makes every 1.5 s (`GET /api/meetings/live-chunks`), so nothing new is read and no route was added. A Mac recording's 10-a-second audiogram lives only in COS Control on that Mac; the web shows the same per-chunk strip for every recording.
+  - Stop and save is the glasses stop of 6.99.x, moved into the bar, and still asks first: "Stop and save this glasses meeting?" and nothing is sent until you confirm. A Mac recording says "Stop on your Mac" and an iPhone one "Stop on iPhone"; neither can be stopped from the web.
+- **Add file, paste and drop while it records.** The same upload as the Ask composer (photos resized to a 1024 px JPEG to `POST /api/media`, other files raw to `POST /api/media/file`), then `POST /api/meeting-ask/files/media`, the route the phone uses, so Meeting Ask reads the file and Copy full context carries it. Photos are read for their text on the Mac, as the phone's are. Up to 10 at a time; video is refused with what to attach instead. A paste or drop that holds a file is always taken as the file: if the browser will not hand the file over, the page says "This browser did not hand over the file. Use Add file to choose it." and never lets its name or path land as text in the Ask box. A file drag outlines the transcript column and moves nothing. Refusals say what happened and what to do (a full meeting, an update in progress, an expired upload, a restarted server, which is read again once); Retry adds the kept upload without sending the bytes again. Files are removed in COS Control on the Mac, as before.
+  - The Files row is drawn from the start while it records ("No files yet. Add file, paste or drop one here."), one line tall whatever it says, so files arriving with the Ask history, an upload starting and one finishing never move what is under it. Open, an upload is a row of the same height as a file and becomes its file in place.
+- Tests: `server/web-control/live-meeting-view.test.ts` runs the real `meetings.js` and `cos-chat.js` against a fake DOM that lays the transcript out (follow, reading, rewrite above the reader, the bar, the clock, the strip, Stop and save, Add file, paste, drop, refusals). `server/lib/web-live-files-canary.test.ts` runs Add file end to end through the real `/api` gate with the web session cookie, the real media routes and the real Meeting Ask routes, and checks that a write without `X-COS-Web`, from another origin or without the cookie is refused (401) and changes nothing. The page's requests are read out of the page's own source. The page-movement numbers come from `server/scripts/web-live-cls/` (a mock server, a CDP driver and the measurement).
+- Default-on changes, for every COS Control Web user (no server behavior changes):
+  - The live meeting view puts the transcript in its own scrolling box with the recording bar under it; Stop and save moves from the header into the bar.
+  - Add file, paste and drag-drop add files to a live meeting from the web (before: "Add or remove them in COS Control on your Mac").
+  - A meeting file that came from the media store is labelled "Added from a phone or browser" (was "Added from the phone").
+  - The Ask composer's upload code is shared with the live view (`W.cosChat.uploadMedia`); its requests are unchanged byte for byte.
+
+## 6.99.13 - The Mac asks first; Drop takes files on COS Control Web (private)
+
+2026-11-05. Private build for one Mac, on top of 6.99.12. Pairs with COS Control 0.5.371 or 0.5.372. Nothing changes for the glasses app.
+
+- **An open web tab never holds a session question from the Mac (option B).** Miles chose B on 11/05: the Mac shows its own dialog at once. A COS Control Web poll (`GET /api/session-questions?client=web`, browser session) no longer counts as a live answering client, so a tab left open on another computer never delays the Mac's pop-up. The web still lists and answers what the glasses or phone are holding, with the same cards and actions as 6.99.12. Token polls from the glasses and phone are unchanged.
+- **Drop takes files (COS Control Web).** Miles, 11/05: "I don't see the file attachment in drop. Can we ensure that functionality is there." The clip, paste and drag-drop already added files to the composer; now Drop sends them with the words, as Ask does.
+  - `POST /api/drops` takes `attachmentIds`: up to 5 fresh uploads from `POST /api/media` or `/api/media/file`, held to the Ask's own rules (media ids, deduplicated, 5 at most). Refusals are plain and save nothing: a sixth file (400 `too_many_attachments`), an id this Mac never gave out (400 `invalid_attachment`, 404 `attachment_not_found`), an upload past its 4 hour wait (410 `attachment_expired`), a file already queued with an Ask or held by another drop (409 `attachment_in_use`). Each says what to do ("Add it again, then Drop").
+  - Words are still required: a drop is filed by its words, so files alone are refused with "Add a few words with the files so COS knows where they go." (400 `words_needed`). The password and key check on the words runs exactly as before.
+  - Where the files live: in the media store, the same place an Ask's files live (`data/media/assets/<id>/`). The drop pins them (`MediaStore.pinForDrop`): the upload becomes `associated` with the drop's id, which no upload TTL and no GC pass touches, so the files stay as long as the drop holds them, across restarts. Nothing is copied, nothing leaves the data dir, and the index is written atomically as always.
+  - The drop record keeps each file's id, name, type and size (`attachments`). Your words in the chat show what you sent, and every receipt lists each file with Open (`GET /api/media/:id/content`). A card, a thread note and a memory take words only, so the receipt says where the files are: "The 2 files stay here in the COS chat with your words." History finds a drop by a file's name. Move keeps the files.
+  - A drop read as a question sends its files with the Ask (`next.attachmentIds`), so "what is on this shelf?" with a photo asks about the photo.
+  - Undo removes the files with the drop: the bytes are deleted, the drop forgets them, and no chat line offers them any more. Undo removes only what that drop pinned; a file an Ask has since sent with a message stays. A refused Undo (the card was edited) keeps them.
+  - `capabilities.dropIt.attachments: 5` on `/api/health` and `/api/drops/settings`. The web sends files only to a server that says so; against an older server Drop keeps the words-only note and sends nothing.
+  - On the web: Option-Return is still Drop and Return is still Ask. With files attached, Drop waits for the uploads (as Ask does), says so when a file failed, and keeps the words and files in the box when the Mac refuses one.
+- Default-on changes, for every user:
+  - The Mac's own session question dialog shows at once even while a COS Control Web tab is open (before, a visible tab could hold it up to the broker's deadline).
+  - `capabilities.dropIt.attachments` on `/api/health` and `/api/drops/settings` for every install (additive). Drop it itself stays off by default; files on a drop exist only where Drop it is on.
+
 ## 6.99.12 - COS Control Web catches up with the Even app: Ask with files, Reply, session questions, wide Usage (private)
 
 2026-11-05. Private build for one Mac, on top of 6.99.11 (Drop it P2), which carries 6.99.10 (Drop it P1 and the web Ask sheet fix). Pairs with COS Control 0.5.371 or 0.5.372. Nothing changes for the glasses app.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.99.12",
+  "version": "6.99.16",
   "description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
   "type": "module",
   "bin": {

package/server/index.ts

@@ -132,7 +132,7 @@ import { skillsRouter } from './routes/skills.js'
 import { shutdownLiveCues } from './lib/live-cues-engine.js'
 import { shutdownLiveCuesV2 } from './lib/live-cues-v2.js'
 import { prewarmContext } from './lib/context-builder.js'
-import { preWarmCLI } from './lib/claude-bridge.js'
+import { getCliSessionId, preWarmCLI } from './lib/claude-bridge.js'
 import { sweepStaleTurnRequests } from './lib/turn-request.js'
 import { getCodexRunConfig } from './lib/codex-run-ledger.js'
 import {
@@ -147,7 +147,7 @@ import { initSessionCache } from './lib/session-cache-writer.js'
 import { initSpeakerEmbeddings } from './lib/speaker-embeddings.js'
 import { startNemotronWarmup } from './lib/nemotron-live.js'
 import { killNemotronChildren } from './lib/nemotron-cli.js'
-import { logActiveSessionsOnShutdown, startAutoSnapshot } from './lib/conversation.js'
+import { getSessionRaw, logActiveSessionsOnShutdown, startAutoSnapshot } from './lib/conversation.js'
 import { getMediaStore } from './lib/media-store.js'
 import { listenRequiredServers, type RequiredListener } from './lib/listener-startup.js'
 import { serverMetrics } from './lib/server-metrics.js'
@@ -170,6 +170,9 @@ import { createWorkLensRoundsRouter } from './routes/work-lens-rounds.js'
 import { startWorkOrchestrator } from './lib/work-orchestrator/runtime.js'
 import { createWorkSearchRouter } from './routes/work-search.js'
 import { createDropsRouter, lazyDropService, openDropService } from './routes/drops.js'
+import { createCosThreadRouter } from './routes/cos-thread.js'
+import { claudeContextReader, CosThread, describeDropFiling, lazyCosThread } from './lib/cos-thread.js'
+import { mediaDropAttachments } from './lib/drop-attachments.js'
 import { createDropRouter, sharedDropJevClient, dropDailyCap, dropRouterEnabled } from './lib/drop-router.js'
 import { createBridgeDropWriters } from './lib/drop-destinations.js'
 import { dropItEnabled } from './lib/drop-settings.js'
@@ -918,6 +921,8 @@ const dropService = lazyDropService(() => openDropService(part => dataPath(part)
   router: createDropRouter(),
   writers: createBridgeDropWriters(),
   timezone: () => serverTimezone(),
+  // 6.99.13: a drop's files are pinned in the media store (lib/drop-attachments.ts); Undo removes them.
+  attachments: mediaDropAttachments(() => getMediaStore()),
   askLookup: async (clientJobId, generation) => {
     const job = await queryJobCoordinator.getByClientGeneration(clientJobId, generation)
     return job ? { status: job.status, response: job.response, sessionId: job.sessionId } : undefined
@@ -969,6 +974,22 @@ app.use('/api', createDropsRouter({
     rows: () => workBoardReader.read(10_000, 3_000).catch(() => null),
   }),
 }))
+// 6.99.16: the COS thread (routes/cos-thread.ts): one size-capped conversation behind the COS chat's Ask, with a
+// ledger to recover and search it. Gated with Drop it; its store opens on first use, so an install with it off writes
+// nothing. It reads the Claude Code log tail of the thread's session and the drop store; it calls no model.
+const cosThread = lazyCosThread(() => new CosThread({
+  dir: dataPath('cos-thread'),
+  context: claudeContextReader({ sessionExists: id => !!getSessionRaw(id), nativeFor: id => getCliSessionId(id) }),
+  nativeFor: id => getCliSessionId(id),
+  drops: (since, limit) => (dropService()?.recentDrops(since, limit) ?? []).map(d => ({ text: d.text, createdAt: d.createdAt, filed: describeDropFiling(d) })),
+  answerFor: (sessionId, clientId, messageNo) => {
+    const exchanges = getSessionRaw(sessionId)?.exchanges ?? []
+    const id = clientId.toLowerCase()
+    const hit = exchanges.find(e => e.role === 'assistant' && (e.clientJobId === id || (messageNo !== null && e.globalMsgNum === messageNo)))
+    return hit?.content
+  },
+}))
+app.use('/api', createCosThreadRouter({ enabled: () => dropItEnabled(), thread: cosThread }))
 // Find a session by meaning (Control 0.5.363): consent per provider, its own ledgers and breakers (lib/session-find.ts).
 app.use('/api', createSessionFindRouter())
 app.use('/api', createWorkChatRouter())