@gotcos/glasses-server 6.88.0 → 6.89.0

package/.env.example

@@ -53,6 +53,13 @@ BIND_HOST=0.0.0.0
 # server after Wi-Fi/Tailscale changes and process restarts.
 # COS_SERVER_INSTANCE_ID_PATH=/path/to/server-instance-id
 
+# 6.89.0 Tailscale self-heal. After a restart, a wake from sleep, a Tailscale
+# restart or a network change, or when a recently active phone goes quiet while
+# Tailscale says it is online, the server runs a bounded `tailscale ping` at it
+# (at most one a minute per phone, 30 an hour). On by default; 0 turns it off.
+# See docs/tailscale-self-heal.md.
+# COS_TAILSCALE_NUDGE=0
+
 # Build 204+ uses server-owned durable query jobs by default. The server fsyncs an
 # accepted prompt before returning 202, then keeps provider work running if the
 # phone backgrounds, reloads, or changes networks. Reopening COS reattaches to

package/CHANGELOG.md

@@ -1,3 +1,34 @@
+## 6.89.0 - LAN first removed; the Mac nudges a stale Tailscale tunnel back to life
+
+Miles, 2026-10-23 19:57: no LAN-only code, because he uses COS away from home; and checks around the nudge so the
+Tailscale connection always comes back. After the 10/22 Mac restart the phone's Tailscale extension kept a stale tunnel:
+Tailscale's own ping went through, no COS request did, and only toggling Tailscale on the phone fixed it. Contract and
+field canary: `docs/tailscale-self-heal.md`.
+
+- **LAN first is gone.** Removed: public `GET /api/route-proof`, `routes` on `/api/models`, `capabilities.routeProof` and
+  `capabilities.lanRoutes`, `COS_LAN_FIRST`, `server/lib/lan-first.ts`, `server/routes/route-proof.ts`,
+  `docs/lan-first.md`, the SECURITY.md note, their tests and gate (reverts c593b91). `/api/models` is the 6.87.0 shape
+  path for path, and `/api/health` is 6.87.0 plus `tailscaleNudge`, both checked against goldens captured from
+  601d807. `/api/route-proof` is an unknown path behind the token again. `COS_LAN_FIRST` is read by nothing.
+- **Kept:** 6.88.0's transfer integrity against the phone's final index and the origin-wipe recovery save that waits
+  for it (fc52f49), and the older opt-in "Allow pairing on this Wi-Fi" pairing switch (6.67.0).
+- **The nudge** (`server/lib/tailscale-nudge.ts`). `tailscale ping -c 3 --timeout 5s <peer>` (disco: both ends redo
+  path discovery), then, only after a pong, `tailscale ping --icmp -c 1 --timeout 5s <peer>` (one packet through
+  WireGuard, which starts a fresh handshake on a stale session). `execFile` from a fixed path, never a shell.
+- **Triggers:** 30 s after boot; a wake from sleep (the wall clock outran the monotonic clock by 20 s); a change in
+  this Mac's IPv4 interfaces; a change in Tailscale's own state (BackendState, this Mac's Tailscale IPs, or peer byte
+  counters going backwards, meaning tailscaled restarted); and a phone that made an authenticated request in the last
+  5 minutes, or sent meeting chunks in the last 15 minutes while a meeting is open, silent for 45 s. Every one needs
+  the peer `Online` in a status read at most 15 s old and Tailscale `Running`: absence of requests alone never nudges.
+- **Caps:** one nudge per peer per 60 s, doubling per unanswered nudge to 15 minutes, reset by any request; 30 an hour
+  in all; one pass at a time; hard timeouts (5, 20 and 10 s, SIGKILL). `COS_TAILSCALE_NUDGE=0` turns it off, read
+  every tick, forwarded by the managed runtime contract. No LLM, no disk write, no new route.
+- **What it says.** `[tailscale-nudge]` log lines with trigger, outcome, timing, handshake age and byte movement, never
+  an address; `tailscaleNudge { enabled, available, lastAt, lastTrigger, lastOutcome, lastDataPath, lastRecoveredAt,
+  nudgesLastHour }` on public `/api/health`.
+- **Phone contract for glasses 6.10.653** (not built here): retry with fresh `no-store` requests on a 2, 4, 8, 15, 30 s
+  ladder, show 6.10.652's stuck-Tailscale message only after 90 s, read `tailscaleNudge`, and the iOS VPN On Demand note.
+
 ## 6.88.0 - LAN first: a pre-token route proof and the Mac's addresses for the phone
 
 The server half of the glasses "LAN first" reconnect plan (plan of 2026-10-22, Miles's decisions of 2026-10-23: "Go

package/SECURITY.md

@@ -43,16 +43,11 @@ reproduction you have. Do not include real transcripts.
   turns itself off after 10 minutes. Over Tailscale the traffic is encrypted.
 - Behind `tailscale serve` or `funnel`, requests arrive from a loopback proxy, so
   every claim is refused (`not_allowed_network`): pairing fails closed there.
-- `GET /api/route-proof` (6.88.0, `docs/lan-first.md`) is public by design. It returns
-  the server instance id and HMAC-SHA256(token, "cos-route-proof:v1:" + nonce + ":" +
-  instance id), never the token, so the phone can check which Mac answers at a home
-  Wi-Fi address before sending the token there. The caller picks only a 22 to 64
-  character base64url nonce. It is served only for a token of 32 characters or more,
-  because a proof lets anyone who reaches the port test guesses offline. Limits: 20 a
-  minute per connection address, 240 a minute in all. It does not stop an active
-  attacker on the same Wi-Fi who relays the request to the real Mac: the LAN path is
-  plain HTTP, as above. `COS_LAN_FIRST=0` turns it off. The Mac's addresses are
-  published only on authenticated `/api/models`, never on `/api/health`.
+- 6.89.0 Tailscale self-heal (`docs/tailscale-self-heal.md`): the server runs `tailscale status --json`
+  and `tailscale ping` at a peer that Tailscale reports online, with `execFile` from a fixed path and a
+  fixed argument list (the only variable is a 100.64.0.0/10 address read from Tailscale's own status).
+  Never a shell. Public `/api/health` shows when it last ran and the outcome, never an address.
+  `COS_TAILSCALE_NUDGE=0` turns it off.
 - `GET /api/display-stream` is public by design and returns lifecycle events
   only. Content requires a ticket or the token header. See
   `server/routes/display.ts` for the allowlist.

package/docs/tailscale-self-heal.md

@@ -0,0 +1,159 @@
+# Tailscale self-heal (server 6.89.0)
+
+COS is used away from home, over Tailscale. The problem this solves: after the Mac restarts, wakes
+from sleep, restarts Tailscale or changes network, the phone can stay on a stale Tailscale tunnel.
+Tailscale says the phone is online and its own ping gets through, yet no COS request arrives, and
+the only fix used to be toggling Tailscale on the phone. 6.89.0 adds a bounded Mac-side "nudge" and
+a contract for the phone side. Code: `server/lib/tailscale-nudge.ts`.
+
+6.89.0 also removes 6.88.0's LAN-first pieces (`GET /api/route-proof`, `routes` on `/api/models`,
+`capabilities.routeProof` and `capabilities.lanRoutes`, `COS_LAN_FIRST`). Tailscale already takes
+the direct path over the home Wi-Fi when both ends are home, so a separate LAN route added nothing
+for the away-from-home case. `COS_LAN_FIRST` is now read by nothing.
+
+## What the Mac can see (read-only probes, 2026-10-23, Tailscale 1.102.2 on macOS)
+
+`tailscale status --json`, the phone's peer entry while COS was working normally:
+
+| Field | Value seen | What it tells us |
+|---|---|---|
+| `Online` | `true` | Control-plane presence. It stays true while the iOS extension runs, stale tunnel or not, so on its own it proves nothing about the data path. |
+| `Active` | `true` | Recent traffic in the engine. |
+| `CurAddr` | the phone's home Wi-Fi address and port | A direct path. Away from home it is a public address, or empty when relayed. |
+| `Relay` | the home DERP region | The fallback relay. |
+| `LastHandshake` | 60 s old, refreshing about every 2 minutes | The WireGuard session is being renewed. A handshake much older than about 3 minutes while `Active` is true is suspicious. |
+| `LastSeen` | zero time | Only set while the peer is offline. |
+| `RxBytes` / `TxBytes` | grew by about 9 KB and 18 KB in 12 s | Per-peer counters for the life of tailscaled. They reset when tailscaled restarts, which is how the server detects a Mac-side Tailscale restart. |
+
+`tailscale ping -c 3 --timeout 5s <phone>` answered once, `via <home address>:41641 in 91ms`, and
+stopped: it stops at the first direct pong. `tailscale ping --icmp -c 1 --timeout 5s <phone>`
+answered `via ICMP in 27ms`. The CLI's own help says the plain (disco) ping does not go through the
+tunnel; the `--icmp` ping does go through WireGuard.
+
+The local signal for "the peer looks online but no COS request has arrived": the server's own last
+request time from that Tailscale address (any `/api` request, public routes included, because any
+request that arrives proves the tunnel carries COS traffic), together with `Online: true` from a
+status read no more than 15 s old. Each nudge log line also records the peer's handshake age and its
+`RxBytes` movement between the last two status reads, so the field canary can show which of them
+tells a stuck tunnel apart. The stuck state itself was not reproduced here.
+
+## The nudge
+
+A nudge is two bounded commands at one peer, run with `execFile` from a fixed path (never a shell,
+never a PATH lookup):
+
+1. `tailscale ping -c 3 --timeout 5s <peer>`: a disco ping. Both ends redo path discovery.
+2. Only after a pong: `tailscale ping --icmp -c 1 --timeout 5s <peer>`: one packet through
+   WireGuard, which makes WireGuard start a fresh handshake when its session is stale. On 10/22 the
+   disco ping got through while COS did not, so the disco ping alone is not enough.
+
+The CLI is looked for at `/Applications/Tailscale.app/Contents/MacOS/Tailscale`,
+`/opt/homebrew/bin/tailscale`, `/usr/local/bin/tailscale` and `/usr/bin/tailscale`, in that order.
+Without one the loop does nothing and health says `available: false`.
+
+Whether a Mac-side nudge un-sticks the iOS extension is not proven by tests. The field canary below
+is the proof.
+
+### Triggers
+
+Every trigger needs the positive signal: the peer is `Online` in a status read at most 15 s old, and
+Tailscale's `BackendState` is `Running`. Absence of requests alone never nudges.
+
+| Trigger | When | Who |
+|---|---|---|
+| `boot` | 30 s after the server starts | Event peers (below) |
+| `wake` | The wall clock moved 20 s or more past the monotonic clock between two 10 s ticks, which only happens across a sleep. Runs 15 s later. | Event peers |
+| `network` | This Mac's external IPv4 interfaces changed, Tailscale's tunnel interface included. Runs 10 s later. | Event peers |
+| `tailscale` | Tailscale's own state changed between status reads: `BackendState`, this Mac's Tailscale IPs, or the peer byte counters went backwards (tailscaled restarted). Runs 10 s later. | Event peers |
+| `silent` | A Tailscale address that made an authenticated request in the last 5 minutes, or sent meeting chunks in the last 15 minutes while a meeting is open, has made no request at all for 45 s. | That address |
+
+Event peers: Online peers this server has seen authenticate in the last 12 hours, plus Online iOS
+and Android peers (after a restart the server remembers no one), at most 4, known ones first. A peer
+that has already made a request since the event is skipped. An event that cannot run because
+Tailscale is not `Running` waits, and is dropped (with one log line) after 2 minutes.
+
+### Caps
+
+- One nudge per peer per 60 s. Each nudge that brings no request doubles the gap: 60, 120, 240,
+  480 s, then 900 s. Any request from that peer resets it.
+- 30 nudges an hour in all, across peers.
+- One pass at a time; each command has a hard timeout (status 5 s, disco 20 s, ICMP 10 s, killed
+  with SIGKILL).
+- A quiet phone outside a meeting is nudged at most 3 times (at about 45 s, 1:50 and 3:50) before its
+  5-minute window closes. In a meeting, at most 4 in 15 minutes.
+- Status is read once a minute while the loop is on, and otherwise only when a nudge could run.
+- `COS_TAILSCALE_NUDGE=0` (or `false`, `off`) stops all of it, read every tick. On by default.
+  Forwarded by the managed runtime contract.
+- No LLM, no disk write, no new route.
+
+### What it says
+
+Log lines, outcome and timing only, never an address:
+
+```
+[tailscale-nudge] trigger=silent outcome=pong_direct dataPath=ok pingMs=91 icmpMs=27 silentS=50 handshakeAgeS=61 rxDelta=9040/60s inRow=1 lastHour=1
+[tailscale-nudge] peer answered 4.2s after the silent nudge
+[tailscale-nudge] trigger=wake dropped: Tailscale not running for 121s
+```
+
+`outcome` is `pong_direct`, `pong_relay`, `no_pong`, `timeout`, `error` or `unavailable`.
+`dataPath` is `ok`, `failed` or `skipped` (no ICMP ping without a disco pong). A failed ping also
+logs `code`, `signal` and `killed`.
+
+Public `/api/health` carries:
+
+```json
+"tailscaleNudge": {
+  "enabled": true,
+  "available": true,
+  "lastAt": "2026-10-23T20:05:10.000Z",
+  "lastTrigger": "silent",
+  "lastOutcome": "pong_direct",
+  "lastDataPath": "ok",
+  "lastRecoveredAt": "2026-10-23T20:05:14.200Z",
+  "nudgesLastHour": 1
+}
+```
+
+`available` is `null` until the first look for the CLI. `lastRecoveredAt` is when a nudged peer next
+made a request, if that was within 90 s of the nudge. No address or device name is published.
+
+## The phone side (contract for glasses 6.10.653; not built in this release)
+
+- **Retry with a fresh request.** On a reconnect failure, retry `/api/health` and then the
+  authenticated `/api/models` with `cache: 'no-store'` and a cache-busting query, on a fresh
+  connection where the WebView allows it (do not reuse a stalled keep-alive socket; a new
+  `EventSource` for the display stream). Backoff ladder: 2, 4, 8, 15, then every 30 s while the app
+  is in the foreground. Never a tight loop.
+- **Wait out the Mac-side window before blaming Tailscale.** The Mac's nudge fires 45 s into a
+  silence (or 30 s after a Mac boot, 15 s after a wake) and its pings take up to about 30 s. Show
+  6.10.652's stuck-Tailscale message only after 90 s of failed fresh requests while the phone has
+  network (the gotcos.com reachability check). Before that, show the ordinary reconnecting state.
+- **Read the outcome.** `/api/health` is public, so once the phone reaches it, `tailscaleNudge`
+  tells it whether the Mac nudged and when. If health is reachable but authenticated calls are not,
+  the tunnel is fine and the problem is elsewhere: do not show the Tailscale message.
+- **Keep working offline.** Local-first capture and backfill are unchanged: record locally, upload
+  when the Mac answers.
+- **What to tell Miles about VPN On Demand.** In the iOS Tailscale app (Settings, VPN On Demand),
+  turning it on makes iOS bring the tunnel back by itself after it drops, for example after a network
+  change. It is a phone setting he can choose; COS cannot change it. It does not replace the nudge: a
+  tunnel iOS considers "up" but stale is not a drop, which is the case the Mac-side nudge targets.
+
+## Field canary (Miles)
+
+1. Note the time. Phone on cellular or away from home, COS open on the phone, Tailscale on.
+2. On the Mac, either quit and reopen Tailscale (menu bar, Quit, then open it), or restart the Mac.
+3. Do not touch Tailscale on the phone. Use COS on the phone within about two minutes.
+
+Success:
+
+- **Log** (the server log; with COS Control, `~/Library/Logs/COS Glasses/server.log`): a `[tailscale-nudge] trigger=tailscale` (Tailscale restart) or
+  `trigger=boot` (Mac restart) line with `outcome=pong_direct` or `pong_relay` and `dataPath=ok`,
+  then `peer answered ...s after the ... nudge`.
+- **Health:** `curl -s http://127.0.0.1:3141/api/health` shows `tailscaleNudge.lastTrigger` matching,
+  `lastOutcome` a pong, and `lastRecoveredAt` set a few seconds after `lastAt`.
+- **Phone:** COS reconnects by itself, without a Tailscale toggle, and a meeting started before the
+  restart keeps its chunks (transfer integrity 100%).
+
+If the phone still needs a toggle: record the nudge log line (its `handshakeAgeS` and `rxDelta` are
+the evidence for the next step) and the time of the toggle.

package/managed-runtime-contract.json

@@ -36,6 +36,7 @@
     "COS_LIVE_CUES_V2_MAX_PER_MEETING",
     "COS_LIVE_CUES_V2_DAILY_WRITER_CALLS",
     "COS_LIVE_CUES_V2_MAC",
+    "COS_TAILSCALE_NUDGE",
     "COS_LIVE_CUES_JEV_DAILY_TOKENS",
     "COS_LIVE_CUES_SUBQ_DAILY_TOKENS",
     "COS_DECISION_PROVIDER_CUES",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotcos/glasses-server",
-  "version": "6.88.0",
+  "version": "6.89.0",
   "description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
   "type": "module",
   "bin": {
@@ -136,9 +136,9 @@
     "!server/scripts/question-hold-mutation-gate.mjs",
     "!server/scripts/live-meeting-6860-mutation-gate.mjs",
     "!server/scripts/live-meeting-6870-mutation-gate.mjs",
-    "!server/scripts/lan-first-mutation-gate.mjs",
     "!server/scripts/integrity-6880-mutation-gate.mjs",
-    "docs/lan-first.md",
+    "!server/scripts/tailscale-nudge-6890-mutation-gate.mjs",
+    "docs/tailscale-self-heal.md",
     "docs/test-plan-6.71.8.md",
     "docs/test-plan-6.71.9.md",
     "docs/test-plan-6.71.10.md",

package/server/index.ts

@@ -14,7 +14,8 @@ import { execSync } from 'node:child_process'
 import { randomBytes, randomUUID } from 'node:crypto'
 import { healthRouter } from './routes/health.js'
 import { diagRouter } from './routes/diag.js'
-import { createClientInstanceRouter } from './routes/client-instance.js'
+import { createClientInstanceRouter, isMeetingChunkPost } from './routes/client-instance.js'
+import { installTailscaleNudger } from './lib/tailscale-nudge.js'
 import { queryRouter } from './routes/query.js'
 import { providerProofRouter } from './routes/provider-proof.js'
 import { transcribeRouter } from './routes/transcribe.js'
@@ -147,7 +148,7 @@ import { logActiveSessionsOnShutdown, startAutoSnapshot } from './lib/conversati
 import { getMediaStore } from './lib/media-store.js'
 import { listenRequiredServers, type RequiredListener } from './lib/listener-startup.js'
 import { serverMetrics } from './lib/server-metrics.js'
-import { getServerInstanceId, initializeServerInstanceId } from './lib/server-instance-id.js'
+import { initializeServerInstanceId } from './lib/server-instance-id.js'
 import { appendPrivateEnvBlock, UnsafeUserConfigPathError } from './lib/secure-user-config.js'
 import { getTranscriptionProfileStatus } from './lib/profile.js'
 import { createQueryJobsRouter } from './routes/query-jobs.js'
@@ -188,10 +189,9 @@ import {
   isLoopbackSocket,
   reachableIpv4Addresses,
 } from './lib/network-policy.js'
-import { requireApiToken } from './lib/api-auth.js'
+import { isPublicApiRequest, requireApiToken } from './lib/api-auth.js'
 import { PairingState, pairingHostsFrom } from './lib/glasses-pairing.js'
 import { createPairingRouter } from './routes/pairing.js'
-import { createRouteProofRouter } from './routes/route-proof.js'
 import { isManagedRuntime } from './lib/managed-runtime.js'
 import { reportClaudeExtraToolConfiguration } from './lib/claude-tool-access.js'
 import {
@@ -356,9 +356,23 @@ const pairingState = new PairingState({
   serverName: () => hostname().replace(/\.local$/i, ''),
   hosts: lanArmed => pairingHostsFrom(reachableIpv4Addresses(networkInterfaces()), PORT, lanArmed),
 })
+// 6.89.0 Tailscale self-heal (lib/tailscale-nudge.ts, docs/tailscale-self-heal.md). Every /api
+// request from a Tailscale address is noted BEFORE the token check, public routes included: any
+// request that arrives proves the tunnel carries COS traffic. Authenticated ones also mark the
+// address as a paired phone. Notes only, never answers; noteRequest never throws.
+const tailscaleNudger = installTailscaleNudger({ activeMeetings: () => getActiveTranscriptionSessionCount() })
+app.use('/api', (req, _res, next) => {
+  tailscaleNudger.noteRequest(req.socket?.remoteAddress, { meetingChunk: isMeetingChunkPost(req.method, req.path) })
+  next()
+})
 app.use('/api', requireApiToken(API_TOKEN, {
   onAuthenticated: req => pairingState.noteAuthenticated(req.socket?.remoteAddress),
 }))
+// Past the token gate, a request on a non-public path carried the token: mark a paired phone.
+app.use('/api', (req, _res, next) => {
+  if (!isPublicApiRequest(req.method, req.path)) tailscaleNudger.noteRequest(req.socket?.remoteAddress, { authenticated: true })
+  next()
+})
 
 // Pairing routes sit HERE on purpose: after the token gate (mint, status and decision
 // need it; the claim and its poll are exact public doors in api-auth.ts), and BEFORE the
@@ -369,14 +383,6 @@ app.use('/api', createPairingRouter({
   state: () => pairingState,
   isDraining: () => gracefulShutdownStarted || !maintenanceAdmissionsOpen(),
 }))
-// 6.88.0 LAN first (docs/lan-first.md): the public pre-token identity proof. Writes nothing,
-// so it sits beside pairing, ahead of the lease catch-all and the global parser, and is
-// served during a drain. The instance id is read per request: it is minted only after
-// every listener binds, and until then the route answers 503 `not_ready`.
-app.use('/api', createRouteProofRouter({
-  token: () => API_TOKEN,
-  serverInstanceId: getServerInstanceId,
-}))
 
 // Fail-closed catch-all for mutation routes that do not own a more specific
 // lifecycle lease below. This closes the admission/drain race for secondary
@@ -1283,6 +1289,7 @@ async function gracefulShutdown(): Promise<void> {
   await meetingAskRuntime.shutdown()
   stopMorningBriefScheduler()
   stopRoutineScheduler()
+  tailscaleNudger.stop()
   await workReviewRuntime?.close()
   workIntakeStore?.close()
   workHandoffRequestStore?.close()
@@ -1485,6 +1492,8 @@ listenRequiredServers(listeners).then(() => {
     // Hash the pinned FluidAudio CLI once at boot so the first Settings GET already knows `installable`.
     void findPinnedCli().catch(() => null)
     startPreviewWarmLoop()
+    // 6.89.0: the Tailscale nudge loop. Its boot trigger is due 30 s from here. COS_TAILSCALE_NUDGE=0 stops it.
+    tailscaleNudger.start()
   }
 
   // Durable recovery starts with the admitted runtime by default. A literal

package/server/lib/api-auth.ts

@@ -43,17 +43,10 @@ const DISPLAY_STREAM_CAPABILITY_PATH = /^\/display-stream\/\d{1,15}\.[0-9a-f]{64
 // Minting, status and decisions stay behind the token AND a loopback socket.
 const PAIRING_POLL_PATH = new RegExp(`^/pairing/claim/${PAIRING_NONCE_BODY}$`)
 
-// 6.88.0 LAN first (docs/lan-first.md): GET /route-proof is public because it exists to
-// prove which Mac answers at an address BEFORE the phone sends its token there. Exact path,
-// GET only (HEAD and every other verb stay behind the token). It returns an HMAC keyed by
-// the token over a fixed, domain-separated message, never the token (routes/route-proof.ts).
-const ROUTE_PROOF_PATH = '/route-proof'
-
 export function isPublicApiRequest(method: string, path: string): boolean {
   if (PUBLIC_API_PATHS.has(path)) return true
   if (method === 'POST' && path === '/pairing/claim') return true
   if (method === 'GET' && PAIRING_POLL_PATH.test(path)) return true
-  if (method === 'GET' && path === ROUTE_PROOF_PATH) return true
   if (method !== 'GET' && method !== 'HEAD') return false
   return TTS_PLAYBACK_CAPABILITY_PATH.test(path)
     || DISPLAY_STREAM_CAPABILITY_PATH.test(path)