@gotcos/glasses-server 6.71.13 → 6.71.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -0
- package/README.md +13 -1
- package/package.json +2 -1
- package/server/lib/stale-active-session-files.ts +178 -0
- package/server/lib/unsaved-audio-quarantine.ts +114 -0
- package/server/routes/meeting.ts +75 -1
- package/server/routes/transcribe-stream.ts +145 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,16 @@
|
|
|
1
|
+
## [6.71.14] - 2026-10-13 - A finished capture no longer reads as a live recording
|
|
2
|
+
|
|
3
|
+
- Root cause. Boot recovery of persisted meeting sessions (`active-sessions/<id>.json`) and of closed-session tombstones ran only if the maintenance gate was already open when the module loaded. A managed update boots behind a closed gate, so recovery was skipped for the life of the process. A persisted session was then neither adopted nor tombstoned: the minute sweep quarantined its audio, auto-recovery receipted it, and its session file stayed on disk. On 2026-10-12 that left a receipted two-chunk capture listed by COS Control as "Live now, No audio for 1054m" for 17 hours, while the status route answered `missing`.
|
|
4
|
+
- Recovery now runs once, when the gate opens. It runs the boot orphan-audio sweep and quarantine purge with it, never overwrites a session already live in memory, and does not quarantine a live session's audio (`isLive` also checks the session map). The minute sweep waits until recovery has run.
|
|
5
|
+
- Boot recovery moves aside (reason `terminal_capture`), rather than adopts, an active-session file that is the leftover of a capture whose quarantined audio was already decoded: recovered into a meeting or receipted no_speech. A capture that is only quarantined, never decoded, is adopted as before, which covers a crash between a chunk's first audio write and its first session write.
|
|
6
|
+
- No new chunk refusal. The phone treats any `410 session_closed` as "already saved" and deletes its local copy, so the server does not refuse a chunk because its capture is quarantined, recovered or receipted; such a chunk starts its own session and keeps the audio. The existing 4-hour closed-session 410 is unchanged, and because tombstones are now restored after a managed boot it applies there too, as it already did after an ordinary restart. Boot recovery adopts a session that resumed under a receipted id: a file counts as a leftover only when the quarantine is no older than its last activity and it has no chunk audio of its own, so a meeting is never split across a restart.
|
|
7
|
+
- New minute-sweep step, in its own try beside the orphaned-audio sweep: an `active-sessions/*.json` whose session is not live and has been idle at least 4 hours (`LOCAL_FIRST_MEETING_IDLE_RETENTION_MS`) is moved to `data/stale-active-session-files/<id>.<ms>.json`. A `.reason.json` (reason `not_live_idle_expired`) is written atomically first; if it cannot be written nothing moves. Idle time uses the earlier of the file's `lastActivityAt` and its mtime, so a future-dated value cannot keep a dead file. Moved files are removed 7 days after the move time in their name, never by mtime. A live session's file is never touched. A failed move logs once per session on stderr and leaves no reason file behind; a purge that removes files logs the count.
|
|
8
|
+
- `GET /api/meeting/sessions/:sessionId/status`: `state` keeps its 6.71.13 values and precedence (`saved`, `active`, `closed`, `missing`). Two additive fields: `outcome` (`receipted`: decoded, no speech; `recovered`: decoded and saved under a file the store no longer finds by session id; `quarantined`: audio held, not decoded yet) and `disposition` (`kind`, `reason`, `quarantinedAt`, `quarantineReason`, `receiptedAt`, `chunkFiles`, `savedFilename`, `recovering`, `audioRetainedUntil`). Both are null when `state` is `saved` or `active`, and when the quarantine holds nothing for the id. They cover every quarantine dir for the id: any undecoded one means `quarantined`. Older clients ignore both.
|
|
9
|
+
- Until recovery has run, the status for an id with no saved meeting answers `503 {"reason": "recovery_pending", "retryable": true}` with `Retry-After: 5`, instead of `missing`, and logs that once. Phone status callers treat a thrown non-2xx as defer-and-retry (the uploader's finalize defers; origin-loss recovery backs off). A saved meeting is answered normally.
|
|
10
|
+
- For COS Control: a gone file whose status is `closed` or `missing` with `outcome: "receipted"` can be shown as "No speech recorded", `recovered` as the saved row lookup, and `quarantined` as "Recovering capture". Control 0.5.287 ignores `outcome`.
|
|
11
|
+
- Tests no longer read the live maintenance gate under ~/Library: `vitest.config.ts` points `COS_MAINTENANCE_GATE_PATH` into the isolated data home.
|
|
12
|
+
- Mutation gate: `node server/scripts/stale-session-mutation-gate.mjs` (40 mutants over recovery, the boot sweep, the minute sweep and purge, the status route and the disposition reader; green baseline first).
|
|
13
|
+
|
|
1
14
|
## [6.71.13] - 2026-10-13 - Meeting-wide Fact check and follow-ups (candidate)
|
|
2
15
|
|
|
3
16
|
- Fact check (the CTA, or a typed bare "Fact check") checks every checkable public claim in the meeting the first time, at most five, each in its own "Claim" row with a verdict (Supported, Mostly supported, Mixed, Contradicted, Unverified) and citations. Each later Fact check scans only what was said after the previous completed check. The span is on the durable exchange as `factCheckRange {fromMs, throughMs, full, claims}` in meeting-elapsed ms; `capabilities.factCheckScope: 1` advertises it. Older clients ignore both; transport stays `action=ask`. A named claim ("Fact check the flight to KC") checks that claim only.
|
package/README.md
CHANGED
|
@@ -359,7 +359,10 @@ the false fences 6.53.1 left, and never releases a fence when `ps` cannot answer
|
|
|
359
359
|
Meetings: a closed meeting's record ages out after **4 hours**, after which its status
|
|
360
360
|
reads `missing` and the phone stops asking to save it. Saving a known meeting that has no
|
|
361
361
|
transcript answers **404 `nothing_to_save`** (with its state); only an unknown id answers
|
|
362
|
-
`session_not_found`.
|
|
362
|
+
`session_not_found`. From 6.71.14 a `missing` or `closed` capture whose audio reached the
|
|
363
|
+
unsaved-audio quarantine also carries an `outcome` (`receipted`, `recovered` or
|
|
364
|
+
`quarantined`) and a `disposition`, and for an id with no saved meeting the status answers **503 `recovery_pending`**
|
|
365
|
+
until boot recovery has run (see [Finished captures never read as live](#finished-captures-never-read-as-live-67114)).
|
|
363
366
|
|
|
364
367
|
## The summary behind the prompt box (6.54.0)
|
|
365
368
|
|
|
@@ -987,3 +990,12 @@ Release receipt (2026-10-12): 6.71.12 is published to npm `latest`; registry int
|
|
|
987
990
|
### Meeting-wide Fact check (6.71.13 candidate)
|
|
988
991
|
|
|
989
992
|
Fact check checks every checkable public claim in the meeting the first time (at most five, one `Claim` row each), then only what was said after the previous completed check. A completed check records `factCheckRange {fromMs, throughMs, full, claims}` on the exchange in meeting-elapsed milliseconds; `capabilities.factCheckScope: 1` advertises it. Follow-ups such as "check it now" continue the pending check. A claim with an implicit starting point uses the profile `home_city` (for example `"home_city": "Austin, TX"` in `.cos-profile.json`), else the phone's weather area from the last 24 hours, and says so. `COS_MEETING_FACT_CHECK_SCOPE=0` returns Fact check to the 6.71.12 latest-claim path; `COS_MEETING_ASK_DOSE_GUARD=0` turns off the fixed answer to medication-dose questions.
|
|
993
|
+
|
|
994
|
+
### Finished captures never read as live (6.71.14)
|
|
995
|
+
|
|
996
|
+
A server that boots behind a closed maintenance gate (every managed update) now runs meeting-session recovery once, when the gate opens. Until 6.71.13 that recovery was skipped for the life of the process, so a persisted `active-sessions/<id>.json` could outlive its capture and COS Control showed it as a live recording. Recovery never overwrites a session already live in memory, and the minute sweep waits for it.
|
|
997
|
+
|
|
998
|
+
- `GET /api/meeting/sessions/:id/status` keeps `state` to `saved`, `active`, `closed` and `missing`, with the same precedence as before. When the quarantine holds the capture it adds `outcome` (`receipted`: decoded, no speech; `recovered`: decoded and saved; `quarantined`: held, not decoded yet) and `disposition` (`kind`, `reason`, `quarantinedAt`, `quarantineReason`, `receiptedAt`, `chunkFiles`, `savedFilename`, `recovering`, `audioRetainedUntil`). Both are null for a saved or live session.
|
|
999
|
+
- Until recovery has run, a status for an id with no saved meeting answers `503 {"reason": "recovery_pending", "retryable": true}` with `Retry-After: 5`, instead of a `missing` that would tell the phone the Mac lost the meeting.
|
|
1000
|
+
- No new chunk refusal: a chunk for a quarantined or receipted capture starts its own session, because the phone treats a 410 as "already saved" and deletes its copy. The existing 4-hour closed-session 410 still applies, now after a managed boot too. A session that resumed under a receipted id is adopted at the next boot, never split off.
|
|
1001
|
+
- Each minute, an `active-sessions/*.json` whose session is not live and has been idle for 4 hours (the earlier of its `lastActivityAt` and its file time) is moved to `data/stale-active-session-files/`, with a `.reason.json` written first. Moved files are removed 7 days after the move. Boot recovery also moves aside, rather than adopts, a file whose capture was already recovered or receipted.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@gotcos/glasses-server",
|
|
3
|
-
"version": "6.71.
|
|
3
|
+
"version": "6.71.14",
|
|
4
4
|
"description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -51,6 +51,7 @@
|
|
|
51
51
|
"!server/scripts/work-progress-mutation-gate.mjs",
|
|
52
52
|
"!server/scripts/work-search-mutation-gate.mjs",
|
|
53
53
|
"!server/scripts/fact-check-mutation-gate.mjs",
|
|
54
|
+
"!server/scripts/stale-session-mutation-gate.mjs",
|
|
54
55
|
"!server/scripts/pairing-mutation-gate.mjs",
|
|
55
56
|
"!server/scripts/away-hold-mutation-gate.mjs",
|
|
56
57
|
"!server/scripts/nemotron-mutation-gate.mjs",
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
// Active-session files the server no longer owns.
|
|
2
|
+
//
|
|
3
|
+
// THE INCIDENT (2026-10-12, server 6.71.11). `active-sessions/<id>.json` is written by
|
|
4
|
+
// the chunk route while a capture is live, and removed when the session closes. A
|
|
5
|
+
// capture of two silent chunks was persisted, then the server restarted behind a
|
|
6
|
+
// closed startup maintenance gate. Boot recovery was gated on admissions being open
|
|
7
|
+
// and never ran once the controller released the gate, so the file was neither
|
|
8
|
+
// adopted into the in-memory session map nor tombstoned. The audio was quarantined,
|
|
9
|
+
// recovered and receipted no_speech, and the file stayed: COS Control lists the
|
|
10
|
+
// directory directly and showed "Live now ... No audio for 1054m" for 17 hours, and
|
|
11
|
+
// the status route answered `missing`.
|
|
12
|
+
//
|
|
13
|
+
// The deferred recovery fix in routes/transcribe-stream.ts closes that door. This
|
|
14
|
+
// sweep is the backstop for any other door: a file whose session is not live and
|
|
15
|
+
// that has been idle past the retention horizon is moved aside, never deleted in
|
|
16
|
+
// place, with a reason written next to it. The horizon is the same one the session
|
|
17
|
+
// itself would be promoted or tombstoned at, so this can never act earlier than the
|
|
18
|
+
// server's own definition of "done".
|
|
19
|
+
|
|
20
|
+
import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, statSync } from 'node:fs'
|
|
21
|
+
import { resolve } from 'node:path'
|
|
22
|
+
import { atomicWriteFileSync } from './atomic-fs.js'
|
|
23
|
+
|
|
24
|
+
export const STALE_ACTIVE_SESSION_DIR_NAME = 'stale-active-session-files'
|
|
25
|
+
|
|
26
|
+
/** Moved files carry transcript text, so they are not kept forever either. */
|
|
27
|
+
export const STALE_ACTIVE_SESSION_FILE_RETENTION_MS = 7 * 24 * 60 * 60 * 1000
|
|
28
|
+
|
|
29
|
+
export type StaleActiveSessionReason =
|
|
30
|
+
/** Not in the live session map and idle past the retention horizon. */
|
|
31
|
+
| 'not_live_idle_expired'
|
|
32
|
+
/** Boot recovery found the capture already recovered or receipted no_speech. */
|
|
33
|
+
| 'terminal_capture'
|
|
34
|
+
|
|
35
|
+
export interface StaleActiveSessionMove {
|
|
36
|
+
sessionId: string
|
|
37
|
+
reason: StaleActiveSessionReason
|
|
38
|
+
/** Where the file went, or null when the move failed and the file stayed put. */
|
|
39
|
+
target: string | null
|
|
40
|
+
lastActivityAt: number | null
|
|
41
|
+
idleMs: number | null
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
const SESSION_FILE_RE = /^([A-Za-z0-9:_-]{3,96})\.json$/
|
|
45
|
+
/** `<id>.<move ms>[-n].json` and its `.reason.json`. The move time is in the NAME. */
|
|
46
|
+
const MOVED_FILE_RE = /^[A-Za-z0-9:_-]{3,96}\.(\d{13})(?:-\d+)?(?:\.reason)?\.json$/
|
|
47
|
+
const REASON_SUFFIX = '.reason.json'
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* When the session in this file was last active, for staleness: the EARLIER of its
|
|
51
|
+
* own `lastActivityAt` and the file's mtime. The file is rewritten on every accepted
|
|
52
|
+
* chunk, so its mtime is a true upper bound; a future-dated or corrupt
|
|
53
|
+
* `lastActivityAt` therefore cannot pin a dead file forever. Null when neither can
|
|
54
|
+
* be read.
|
|
55
|
+
*/
|
|
56
|
+
export function activeSessionFileActivity(filePath: string): number | null {
|
|
57
|
+
let mtimeMs: number | null = null
|
|
58
|
+
try {
|
|
59
|
+
const stat = statSync(filePath)
|
|
60
|
+
if (Number.isFinite(stat.mtimeMs)) mtimeMs = stat.mtimeMs
|
|
61
|
+
} catch { /* fall through: a missing file has no age */ }
|
|
62
|
+
let recorded: number | null = null
|
|
63
|
+
try {
|
|
64
|
+
const data = JSON.parse(readFileSync(filePath, 'utf8')) as { lastActivityAt?: unknown }
|
|
65
|
+
if (typeof data?.lastActivityAt === 'number' && Number.isFinite(data.lastActivityAt)) recorded = data.lastActivityAt
|
|
66
|
+
} catch { /* torn or legacy file: the mtime still gives it an age */ }
|
|
67
|
+
if (recorded == null) return mtimeMs
|
|
68
|
+
if (mtimeMs == null) return recorded
|
|
69
|
+
return Math.min(recorded, mtimeMs)
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
/**
|
|
73
|
+
* Move one active-session file into the stale directory. The reason is written
|
|
74
|
+
* FIRST and atomically, so a moved file never exists without its reason; if the
|
|
75
|
+
* reason cannot be written nothing is moved. Returns the new path, or null when the
|
|
76
|
+
* move did not happen (the source then stays exactly where it was).
|
|
77
|
+
*/
|
|
78
|
+
export function moveActiveSessionFileAside(
|
|
79
|
+
sourcePath: string,
|
|
80
|
+
sessionId: string,
|
|
81
|
+
targetDir: string,
|
|
82
|
+
reason: StaleActiveSessionReason,
|
|
83
|
+
now: number,
|
|
84
|
+
detail: { lastActivityAt: number | null; note?: string } = { lastActivityAt: null },
|
|
85
|
+
): string | null {
|
|
86
|
+
try {
|
|
87
|
+
if (!existsSync(sourcePath)) return null
|
|
88
|
+
mkdirSync(targetDir, { recursive: true, mode: 0o700 })
|
|
89
|
+
// Two moves in the same millisecond must not overwrite each other.
|
|
90
|
+
let base = `${sessionId}.${now}`
|
|
91
|
+
for (let n = 1; existsSync(resolve(targetDir, `${base}.json`)) || existsSync(resolve(targetDir, `${base}${REASON_SUFFIX}`)); n++) {
|
|
92
|
+
base = `${sessionId}.${now}-${n}`
|
|
93
|
+
}
|
|
94
|
+
const target = resolve(targetDir, `${base}.json`)
|
|
95
|
+
const reasonPath = resolve(targetDir, `${base}${REASON_SUFFIX}`)
|
|
96
|
+
atomicWriteFileSync(reasonPath, `${JSON.stringify({
|
|
97
|
+
schemaVersion: 1,
|
|
98
|
+
sessionId,
|
|
99
|
+
reason,
|
|
100
|
+
movedAt: new Date(now).toISOString(),
|
|
101
|
+
lastActivityAt: detail.lastActivityAt != null ? new Date(detail.lastActivityAt).toISOString() : null,
|
|
102
|
+
...(detail.note ? { note: detail.note } : {}),
|
|
103
|
+
})}\n`, { mode: 0o600 })
|
|
104
|
+
try {
|
|
105
|
+
renameSync(sourcePath, target)
|
|
106
|
+
} catch (error) {
|
|
107
|
+
// No reason may outlive a move that did not happen.
|
|
108
|
+
try { rmSync(reasonPath, { force: true }) } catch { /* the next sweep's own reason supersedes it */ }
|
|
109
|
+
throw error
|
|
110
|
+
}
|
|
111
|
+
return target
|
|
112
|
+
} catch {
|
|
113
|
+
return null
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
/**
|
|
118
|
+
* One pass over the active-sessions directory. A file is moved aside only when its
|
|
119
|
+
* session is NOT live and it has been idle for at least `idleRetentionMs`. A live
|
|
120
|
+
* session's file is never touched, however old it looks. A failed move is returned
|
|
121
|
+
* with `target: null` so the caller can say so.
|
|
122
|
+
*/
|
|
123
|
+
export function sweepStaleActiveSessionFiles(opts: {
|
|
124
|
+
dir: string
|
|
125
|
+
targetDir: string
|
|
126
|
+
isLive: (sessionId: string) => boolean
|
|
127
|
+
now: number
|
|
128
|
+
idleRetentionMs: number
|
|
129
|
+
}): StaleActiveSessionMove[] {
|
|
130
|
+
const moves: StaleActiveSessionMove[] = []
|
|
131
|
+
let entries: string[]
|
|
132
|
+
try { entries = readdirSync(opts.dir) } catch { return moves }
|
|
133
|
+
for (const name of entries) {
|
|
134
|
+
const match = SESSION_FILE_RE.exec(name)
|
|
135
|
+
if (!match) continue
|
|
136
|
+
const sessionId = match[1]
|
|
137
|
+
if (opts.isLive(sessionId)) continue
|
|
138
|
+
const filePath = resolve(opts.dir, name)
|
|
139
|
+
// The mtime bound inside activeSessionFileActivity is what defeats a
|
|
140
|
+
// future-dated value. A further min(..., now) would change no decision: a
|
|
141
|
+
// future activity is negative idle, and zero idle is just as far from the horizon.
|
|
142
|
+
const lastActivityAt = activeSessionFileActivity(filePath)
|
|
143
|
+
if (lastActivityAt == null) continue
|
|
144
|
+
const idleMs = opts.now - lastActivityAt
|
|
145
|
+
if (idleMs < opts.idleRetentionMs) continue
|
|
146
|
+
const target = moveActiveSessionFileAside(filePath, sessionId, opts.targetDir, 'not_live_idle_expired', opts.now, {
|
|
147
|
+
lastActivityAt,
|
|
148
|
+
})
|
|
149
|
+
moves.push({ sessionId, reason: 'not_live_idle_expired', target, lastActivityAt, idleMs })
|
|
150
|
+
}
|
|
151
|
+
return moves
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
/**
|
|
155
|
+
* Remove moved files (and their reasons) whose move time, read from the file NAME,
|
|
156
|
+
* is older than the retention. Ages never come from mtime: a rename keeps the old
|
|
157
|
+
* mtime, which would turn "move" into "delete a minute later". A name that does not
|
|
158
|
+
* carry a move time is never removed. Returns the count removed.
|
|
159
|
+
*/
|
|
160
|
+
export function purgeExpiredStaleActiveSessionFiles(
|
|
161
|
+
targetDir: string,
|
|
162
|
+
now: number,
|
|
163
|
+
retentionMs: number = STALE_ACTIVE_SESSION_FILE_RETENTION_MS,
|
|
164
|
+
): number {
|
|
165
|
+
let removed = 0
|
|
166
|
+
let entries: string[]
|
|
167
|
+
try { entries = readdirSync(targetDir) } catch { return 0 }
|
|
168
|
+
for (const name of entries) {
|
|
169
|
+
const match = MOVED_FILE_RE.exec(name)
|
|
170
|
+
if (!match) continue
|
|
171
|
+
if (now - Number(match[1]) <= retentionMs) continue
|
|
172
|
+
try {
|
|
173
|
+
rmSync(resolve(targetDir, name), { force: true })
|
|
174
|
+
removed++
|
|
175
|
+
} catch { /* next sweep retries */ }
|
|
176
|
+
}
|
|
177
|
+
return removed
|
|
178
|
+
}
|
|
@@ -347,3 +347,117 @@ export function markRecovered(dirPath: string, savedFilename: string): void {
|
|
|
347
347
|
)
|
|
348
348
|
} catch { /* receipt is best-effort; findBySessionId remains the true guard */ }
|
|
349
349
|
}
|
|
350
|
+
|
|
351
|
+
// ── Terminal disposition of a capture the server no longer holds live ───────
|
|
352
|
+
//
|
|
353
|
+
// 6.71.14. The quarantine is the only record of what happened to a capture whose
|
|
354
|
+
// live session is gone: its audio was quarantined, then recovered into a meeting or
|
|
355
|
+
// receipted as no_speech. Two readers need that answer and used to get "missing":
|
|
356
|
+
// the session status route (COS Control rendered "Recording ended without saving"
|
|
357
|
+
// for a capture that had been recovered and receipted) and the chunk route, which
|
|
358
|
+
// recreated a live session for a late chunk of a capture that was already done.
|
|
359
|
+
|
|
360
|
+
export type CaptureDispositionKind = 'quarantined' | 'recovered' | 'no_speech'
|
|
361
|
+
|
|
362
|
+
export interface CaptureDisposition {
|
|
363
|
+
kind: CaptureDispositionKind
|
|
364
|
+
sessionId: string
|
|
365
|
+
dirName: string
|
|
366
|
+
quarantinedAt: string | null
|
|
367
|
+
/** When the recovery receipt was written, for `recovered` and `no_speech`. */
|
|
368
|
+
receiptedAt: string | null
|
|
369
|
+
chunkFiles: number
|
|
370
|
+
/** The quarantine manifest's reason (why the audio was moved), when known. */
|
|
371
|
+
quarantineReason: string | null
|
|
372
|
+
/** Set for `recovered`: the meeting file the recovery saved. */
|
|
373
|
+
savedFilename: string | null
|
|
374
|
+
/** A recovery is decoding this capture right now. */
|
|
375
|
+
recovering: boolean
|
|
376
|
+
/** When quarantine retention removes the audio. */
|
|
377
|
+
retainedUntil: string | null
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
interface RecoveredReceipt {
|
|
381
|
+
recoveredAt?: unknown
|
|
382
|
+
outcome?: unknown
|
|
383
|
+
savedFilename?: unknown
|
|
384
|
+
}
|
|
385
|
+
|
|
386
|
+
function readRecoveredReceipt(dirPath: string): RecoveredReceipt | null {
|
|
387
|
+
const path = resolve(dirPath, RECOVERED_RECEIPT)
|
|
388
|
+
if (!existsSync(path)) return null
|
|
389
|
+
try {
|
|
390
|
+
const raw = JSON.parse(readFileSync(path, 'utf8')) as unknown
|
|
391
|
+
return raw && typeof raw === 'object' ? raw as RecoveredReceipt : {}
|
|
392
|
+
} catch {
|
|
393
|
+
// Present but unreadable still means a recovery finished: the receipt is only
|
|
394
|
+
// ever written at the end of one (listUnsavedCaptures counts it the same way).
|
|
395
|
+
return {}
|
|
396
|
+
}
|
|
397
|
+
}
|
|
398
|
+
|
|
399
|
+
/** Every quarantine dir for a session id: the exact name and any `<id>.<ms>` copies. */
|
|
400
|
+
function quarantineDirsFor(sessionId: string, root: string): string[] {
|
|
401
|
+
const dirs: string[] = []
|
|
402
|
+
let entries: string[] = []
|
|
403
|
+
try { entries = readdirSync(root) } catch { return dirs }
|
|
404
|
+
for (const dir of entries) {
|
|
405
|
+
// Same match as findQuarantineDir: session ids carry no dots, so `<id>.` is this id.
|
|
406
|
+
if (dir !== sessionId && !dir.startsWith(`${sessionId}.`)) continue
|
|
407
|
+
const dirPath = resolve(root, dir)
|
|
408
|
+
try { if (statSync(dirPath).isDirectory()) dirs.push(dirPath) } catch { /* vanished: skip */ }
|
|
409
|
+
}
|
|
410
|
+
return dirs.sort()
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
function dispositionOf(sessionId: string, dirPath: string, retentionMs: number): CaptureDisposition {
|
|
414
|
+
const manifest = readManifest(dirPath)
|
|
415
|
+
const atMs = quarantinedAtMs(dirPath, manifest)
|
|
416
|
+
const receipt = readRecoveredReceipt(dirPath)
|
|
417
|
+
const receiptedMs = receipt && typeof receipt.recoveredAt === 'string' ? Date.parse(receipt.recoveredAt) : NaN
|
|
418
|
+
const savedFilename = receipt && typeof receipt.savedFilename === 'string' && receipt.savedFilename
|
|
419
|
+
? receipt.savedFilename
|
|
420
|
+
: null
|
|
421
|
+
const kind: CaptureDispositionKind = !receipt
|
|
422
|
+
? 'quarantined'
|
|
423
|
+
: receipt.outcome === 'no_speech'
|
|
424
|
+
? 'no_speech'
|
|
425
|
+
: 'recovered'
|
|
426
|
+
return {
|
|
427
|
+
kind,
|
|
428
|
+
sessionId,
|
|
429
|
+
dirName: basename(dirPath),
|
|
430
|
+
quarantinedAt: atMs != null ? new Date(atMs).toISOString() : null,
|
|
431
|
+
receiptedAt: Number.isFinite(receiptedMs) ? new Date(receiptedMs).toISOString() : null,
|
|
432
|
+
chunkFiles: manifest?.chunkFiles ?? countChunkWavs(dirPath),
|
|
433
|
+
quarantineReason: manifest?.reason ?? null,
|
|
434
|
+
savedFilename,
|
|
435
|
+
recovering: isRecoveryActive(basename(dirPath)),
|
|
436
|
+
retainedUntil: atMs != null ? new Date(atMs + retentionMs).toISOString() : null,
|
|
437
|
+
}
|
|
438
|
+
}
|
|
439
|
+
|
|
440
|
+
const DISPOSITION_RANK: Record<CaptureDispositionKind, number> = { quarantined: 0, recovered: 1, no_speech: 2 }
|
|
441
|
+
|
|
442
|
+
/**
|
|
443
|
+
* What became of a capture whose audio reached quarantine, or null when the
|
|
444
|
+
* quarantine holds nothing for this session id.
|
|
445
|
+
*
|
|
446
|
+
* `no_speech` and `recovered` are terminal: the capture was decoded once and its
|
|
447
|
+
* outcome receipted. `quarantined` is held audio that a recovery has not finished
|
|
448
|
+
* yet (the sweep picks one per minute, or a person presses Recover).
|
|
449
|
+
*
|
|
450
|
+
* One id can own SEVERAL dirs (`<id>` and `<id>.<ms>`, when the same id was
|
|
451
|
+
* quarantined twice). The answer covers all of them: ANY unrecovered dir means the
|
|
452
|
+
* capture is still `quarantined`; otherwise a saved recovery outranks no_speech.
|
|
453
|
+
*/
|
|
454
|
+
export function readCaptureDisposition(
|
|
455
|
+
sessionId: string,
|
|
456
|
+
root: string = unsavedAudioRoot(),
|
|
457
|
+
retentionMs: number = unsavedAudioRetentionMs(),
|
|
458
|
+
): CaptureDisposition | null {
|
|
459
|
+
if (!/^[A-Za-z0-9:_-]{3,96}$/.test(sessionId)) return null
|
|
460
|
+
const all = quarantineDirsFor(sessionId, root).map(dir => dispositionOf(sessionId, dir, retentionMs))
|
|
461
|
+
if (all.length === 0) return null
|
|
462
|
+
return all.reduce((best, next) => DISPOSITION_RANK[next.kind] < DISPOSITION_RANK[best.kind] ? next : best)
|
|
463
|
+
}
|
package/server/routes/meeting.ts
CHANGED
|
@@ -87,7 +87,9 @@ import {
|
|
|
87
87
|
listUnsavedCaptures,
|
|
88
88
|
markRecovered,
|
|
89
89
|
markRecoveredNoSpeech,
|
|
90
|
+
readCaptureDisposition,
|
|
90
91
|
registerActiveRecovery,
|
|
92
|
+
type CaptureDisposition,
|
|
91
93
|
} from '../lib/unsaved-audio-quarantine.js'
|
|
92
94
|
import {
|
|
93
95
|
clearSessionHallucinationState,
|
|
@@ -101,6 +103,7 @@ import {
|
|
|
101
103
|
analyzeTranscriptGaps,
|
|
102
104
|
deleteSession,
|
|
103
105
|
drainSessionAudioWrites,
|
|
106
|
+
durableSessionRecoveryHasRun,
|
|
104
107
|
getSessionChunkEntries,
|
|
105
108
|
getSessionChunks,
|
|
106
109
|
getSessionProviderCandidates,
|
|
@@ -475,6 +478,48 @@ function publicSaveResponse(saved: SavedMeeting, replayed = false): Record<strin
|
|
|
475
478
|
}
|
|
476
479
|
}
|
|
477
480
|
|
|
481
|
+
/**
|
|
482
|
+
* 6.71.14: what became of a capture whose live session is gone, as an ADDITIVE
|
|
483
|
+
* field beside `state`.
|
|
484
|
+
*
|
|
485
|
+
* `state` keeps its 6.71.13 values and precedence (saved > active > closed >
|
|
486
|
+
* missing) on purpose. Phones branch on it (`missing` is recovery-required or a
|
|
487
|
+
* collision, `closed` is terminal-unsaved), and a new value would fall through to
|
|
488
|
+
* their resume path. So the quarantine's answer rides alongside:
|
|
489
|
+
* receipted recovery finished and found no speech
|
|
490
|
+
* recovered recovery saved a meeting (the store may no longer find it by id)
|
|
491
|
+
* quarantined the audio is held and no recovery has finished yet
|
|
492
|
+
* Null when `state` is saved (the receipt is the answer) or active (a live session
|
|
493
|
+
* reports itself), and when the quarantine holds nothing for the id.
|
|
494
|
+
*/
|
|
495
|
+
export type MeetingCaptureOutcome = 'receipted' | 'recovered' | 'quarantined'
|
|
496
|
+
|
|
497
|
+
export function captureOutcome(
|
|
498
|
+
state: 'saved' | 'active' | 'closed' | 'missing',
|
|
499
|
+
disposition: Pick<CaptureDisposition, 'kind'> | null,
|
|
500
|
+
): MeetingCaptureOutcome | null {
|
|
501
|
+
if (state === 'saved' || state === 'active' || !disposition) return null
|
|
502
|
+
if (disposition.kind === 'no_speech') return 'receipted'
|
|
503
|
+
if (disposition.kind === 'recovered') return 'recovered'
|
|
504
|
+
return 'quarantined'
|
|
505
|
+
}
|
|
506
|
+
|
|
507
|
+
function publicDisposition(disposition: CaptureDisposition | null): Record<string, unknown> | null {
|
|
508
|
+
if (!disposition) return null
|
|
509
|
+
return {
|
|
510
|
+
kind: disposition.kind,
|
|
511
|
+
// Why the capture is terminal: `no_speech` or `recovered`; null while it is only held.
|
|
512
|
+
reason: disposition.kind === 'quarantined' ? null : disposition.kind,
|
|
513
|
+
quarantinedAt: disposition.quarantinedAt,
|
|
514
|
+
quarantineReason: disposition.quarantineReason,
|
|
515
|
+
receiptedAt: disposition.receiptedAt,
|
|
516
|
+
chunkFiles: disposition.chunkFiles,
|
|
517
|
+
savedFilename: disposition.savedFilename,
|
|
518
|
+
recovering: disposition.recovering,
|
|
519
|
+
audioRetainedUntil: disposition.retainedUntil,
|
|
520
|
+
}
|
|
521
|
+
}
|
|
522
|
+
|
|
478
523
|
function suppliedServerPin(
|
|
479
524
|
req: { get: (name: string) => string | undefined },
|
|
480
525
|
fallback: unknown,
|
|
@@ -497,6 +542,7 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
497
542
|
const savingSessions = new Set<string>()
|
|
498
543
|
setNamingSessionLiveness(id => sessions.getStartTime(id) !== null || savingSessions.has(id) || activeFinalizationJobs.has(id))
|
|
499
544
|
|
|
545
|
+
let recoveryPendingLogged = false
|
|
500
546
|
router.get('/meeting/sessions/:sessionId/status', (req, res) => {
|
|
501
547
|
const sessionId = String(req.params.sessionId ?? '')
|
|
502
548
|
if (!/^[A-Za-z0-9:_-]{3,96}$/.test(sessionId)) {
|
|
@@ -514,11 +560,35 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
514
560
|
return
|
|
515
561
|
}
|
|
516
562
|
const saved = store.findBySessionId(sessionId)
|
|
563
|
+
// 6.71.14: before durable recovery has run (a boot behind a closed or stuck
|
|
564
|
+
// maintenance gate), a persisted meeting is not in memory YET, and answering
|
|
565
|
+
// `missing` would tell the phone the Mac lost it. Every phone status caller
|
|
566
|
+
// treats a thrown non-2xx as defer-and-retry (the uploader's finalize defers;
|
|
567
|
+
// origin-loss recovery backs off), so say "not yet" instead. A saved meeting
|
|
568
|
+
// is already authoritative and is answered as usual.
|
|
569
|
+
if (!saved && !durableSessionRecoveryHasRun()) {
|
|
570
|
+
if (!recoveryPendingLogged) {
|
|
571
|
+
recoveryPendingLogged = true
|
|
572
|
+
console.warn('[meeting/status] Answering 503 recovery_pending: durable session recovery has not run (maintenance gate closed)')
|
|
573
|
+
}
|
|
574
|
+
res.set('Cache-Control', 'private, no-store')
|
|
575
|
+
res.set('Retry-After', '5')
|
|
576
|
+
res.status(503).json({
|
|
577
|
+
error: 'Meeting session recovery has not run yet; retry shortly',
|
|
578
|
+
reason: 'recovery_pending',
|
|
579
|
+
retryable: true,
|
|
580
|
+
})
|
|
581
|
+
return
|
|
582
|
+
}
|
|
517
583
|
const live = getMeetingSessionStatus(sessionId)
|
|
584
|
+
const state = saved ? 'saved' : live.state
|
|
585
|
+
// A saved meeting or a live session is the truth for itself; the quarantine is
|
|
586
|
+
// read only when neither is.
|
|
587
|
+
const disposition = state === 'saved' || state === 'active' ? null : readCaptureDisposition(sessionId)
|
|
518
588
|
res.set('Cache-Control', 'private, no-store')
|
|
519
589
|
res.json({
|
|
520
590
|
sessionId,
|
|
521
|
-
state
|
|
591
|
+
state,
|
|
522
592
|
serverInstanceId,
|
|
523
593
|
receivedRanges: live.receivedRanges,
|
|
524
594
|
receivedCount: live.receivedCount,
|
|
@@ -530,6 +600,10 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
530
600
|
lastActivityAt: live.lastActivityAt,
|
|
531
601
|
retainedUntil: saved ? null : live.retainedUntil,
|
|
532
602
|
saveReceipt: saved ? publicSaveResponse(saved) : null,
|
|
603
|
+
// 6.71.14, additive: what became of a capture whose audio reached quarantine
|
|
604
|
+
// (null otherwise). Older clients ignore both.
|
|
605
|
+
outcome: captureOutcome(state, disposition),
|
|
606
|
+
disposition: publicDisposition(disposition),
|
|
533
607
|
})
|
|
534
608
|
})
|
|
535
609
|
|
|
@@ -77,8 +77,16 @@ import {
|
|
|
77
77
|
listUnsavedCaptures,
|
|
78
78
|
purgeExpiredQuarantine,
|
|
79
79
|
quarantineSessionAudio,
|
|
80
|
+
readCaptureDisposition,
|
|
80
81
|
sweepOrphanedSessionAudio,
|
|
81
82
|
} from '../lib/unsaved-audio-quarantine.js'
|
|
83
|
+
import {
|
|
84
|
+
STALE_ACTIVE_SESSION_DIR_NAME,
|
|
85
|
+
moveActiveSessionFileAside,
|
|
86
|
+
purgeExpiredStaleActiveSessionFiles,
|
|
87
|
+
sweepStaleActiveSessionFiles,
|
|
88
|
+
type StaleActiveSessionMove,
|
|
89
|
+
} from '../lib/stale-active-session-files.js'
|
|
82
90
|
import { durableAtomicWriteFileSync } from '../lib/atomic-fs.js'
|
|
83
91
|
import { saveTrainingAudioSample } from '../lib/training-audio-save.js'
|
|
84
92
|
import {
|
|
@@ -92,6 +100,7 @@ import {
|
|
|
92
100
|
acquireMaintenanceWork,
|
|
93
101
|
maintenanceAdmissionsOpen,
|
|
94
102
|
maintenanceLifecycle,
|
|
103
|
+
onMaintenanceAdmissionsOpen,
|
|
95
104
|
type MaintenanceWorkLease,
|
|
96
105
|
} from '../lib/maintenance-lifecycle.js'
|
|
97
106
|
import { feedLiveCueTranscript } from '../lib/live-cues-engine.js'
|
|
@@ -542,6 +551,8 @@ export function getTranscriptionSessionLiveness(now = Date.now()): Transcription
|
|
|
542
551
|
// Incremental chunk persistence — survive server restarts
|
|
543
552
|
const CHUNK_PERSIST_DIR = dataPath('active-sessions')
|
|
544
553
|
ensurePrivateDirectory(CHUNK_PERSIST_DIR)
|
|
554
|
+
/** 6.71.14: where an active-session file the server no longer owns is moved, never deleted in place. */
|
|
555
|
+
const STALE_ACTIVE_SESSION_DIR = dataPath(STALE_ACTIVE_SESSION_DIR_NAME)
|
|
545
556
|
|
|
546
557
|
function readClosedSessions(): Record<string, ClosedTranscriptSession> {
|
|
547
558
|
if (!existsSync(CLOSED_SESSIONS_FILE)) return {}
|
|
@@ -691,6 +702,71 @@ function persistSessionBestEffort(sessionId: string): void {
|
|
|
691
702
|
try { persistSessionRequired(sessionId) } catch { /* recovery cleanup is non-admission work */ }
|
|
692
703
|
}
|
|
693
704
|
|
|
705
|
+
/**
|
|
706
|
+
* 6.71.14: one pass of the stale active-session sweep, against the real session
|
|
707
|
+
* map. Exported so a test drives the exact wiring the minute interval runs.
|
|
708
|
+
*/
|
|
709
|
+
export function sweepStaleActiveSessionFilesNow(now: number): StaleActiveSessionMove[] {
|
|
710
|
+
const moves = sweepStaleActiveSessionFiles({
|
|
711
|
+
dir: CHUNK_PERSIST_DIR,
|
|
712
|
+
targetDir: STALE_ACTIVE_SESSION_DIR,
|
|
713
|
+
isLive: id => sessions.has(id),
|
|
714
|
+
now,
|
|
715
|
+
idleRetentionMs: LOCAL_FIRST_MEETING_IDLE_RETENTION_MS,
|
|
716
|
+
})
|
|
717
|
+
for (const move of moves) {
|
|
718
|
+
if (move.target) {
|
|
719
|
+
staleMoveFailuresLogged.delete(move.sessionId)
|
|
720
|
+
console.warn(
|
|
721
|
+
`[cleanup] Moved stale active-session file aside: ${move.sessionId} `
|
|
722
|
+
+ `(${move.reason}, idle ${Math.round((move.idleMs ?? 0) / 60_000)}m, not live) \u2192 ${move.target}`,
|
|
723
|
+
)
|
|
724
|
+
} else {
|
|
725
|
+
logStaleMoveFailure(move.sessionId, `[cleanup] Could not move stale active-session file aside for ${move.sessionId} (${move.reason}); it stays in active-sessions`)
|
|
726
|
+
}
|
|
727
|
+
}
|
|
728
|
+
const purged = purgeExpiredStaleActiveSessionFiles(STALE_ACTIVE_SESSION_DIR, now)
|
|
729
|
+
if (purged > 0) console.log(`[cleanup] Purged ${purged} stale active-session file(s) past retention`)
|
|
730
|
+
return moves
|
|
731
|
+
}
|
|
732
|
+
|
|
733
|
+
/** A failed move repeats every minute; say so once per session id, not every tick. */
|
|
734
|
+
const staleMoveFailuresLogged = new Set<string>()
|
|
735
|
+
function logStaleMoveFailure(sessionId: string, message: string): void {
|
|
736
|
+
if (staleMoveFailuresLogged.has(sessionId)) return
|
|
737
|
+
staleMoveFailuresLogged.add(sessionId)
|
|
738
|
+
if (staleMoveFailuresLogged.size > 500) staleMoveFailuresLogged.delete(staleMoveFailuresLogged.values().next().value as string)
|
|
739
|
+
console.error(message)
|
|
740
|
+
}
|
|
741
|
+
|
|
742
|
+
/**
|
|
743
|
+
* 6.71.14: is this active-session file the leftover of a capture that is already
|
|
744
|
+
* decoded (recovered or receipted no_speech), rather than a session that RESUMED
|
|
745
|
+
* under the same id after that capture was quarantined?
|
|
746
|
+
*
|
|
747
|
+
* A late chunk is accepted (the phone deletes its copy on a 410), so a session can
|
|
748
|
+
* restart under an id whose older quarantine dir is already receipted. Its file is
|
|
749
|
+
* then newer than the quarantine and its own audio sits in session-audio/<id>.
|
|
750
|
+
* Only a file that is no newer than the quarantine AND has no chunk audio of its
|
|
751
|
+
* own is the leftover; anything else is adopted as 6.71.13 would, so a meeting is
|
|
752
|
+
* never split across a restart.
|
|
753
|
+
*/
|
|
754
|
+
function isTerminalCaptureFile(sessionId: string, recordedLastActivityAt: unknown, filePath: string): boolean {
|
|
755
|
+
const disposition = readCaptureDisposition(sessionId)
|
|
756
|
+
if (!disposition || (disposition.kind !== 'no_speech' && disposition.kind !== 'recovered')) return false
|
|
757
|
+
const terminalAtIso = disposition.quarantinedAt ?? disposition.receiptedAt
|
|
758
|
+
const terminalAt = terminalAtIso ? Date.parse(terminalAtIso) : NaN
|
|
759
|
+
if (!Number.isFinite(terminalAt)) return false
|
|
760
|
+
let fileActivity = typeof recordedLastActivityAt === 'number' && Number.isFinite(recordedLastActivityAt)
|
|
761
|
+
? recordedLastActivityAt
|
|
762
|
+
: NaN
|
|
763
|
+
if (!Number.isFinite(fileActivity)) {
|
|
764
|
+
try { fileActivity = statSync(filePath).mtimeMs } catch { return false }
|
|
765
|
+
}
|
|
766
|
+
if (terminalAt < fileActivity) return false
|
|
767
|
+
return countChunkWavs(resolve(SESSION_AUDIO_DIR, sessionId)) === 0
|
|
768
|
+
}
|
|
769
|
+
|
|
694
770
|
/** Recover sessions from disk on server restart */
|
|
695
771
|
function recoverSessions(): void {
|
|
696
772
|
try {
|
|
@@ -701,6 +777,27 @@ function recoverSessions(): void {
|
|
|
701
777
|
if (!file.endsWith('.json')) continue
|
|
702
778
|
try {
|
|
703
779
|
const data = JSON.parse(readFileSync(resolve(CHUNK_PERSIST_DIR, file), 'utf-8'))
|
|
780
|
+
// 6.71.14: recovery can now run after boot (deferred behind a closed
|
|
781
|
+
// maintenance gate). A session already in memory owns its file; the disk
|
|
782
|
+
// copy must never overwrite it.
|
|
783
|
+
if (typeof data.sessionId === 'string' && sessions.has(data.sessionId)) continue
|
|
784
|
+
// A capture whose quarantined audio was already decoded to a terminal
|
|
785
|
+
// outcome (a saved recovery or a no_speech receipt) is done. Adopting its
|
|
786
|
+
// file would bring back a live session for it; tombstoning would delete the
|
|
787
|
+
// file. Move it aside instead. Plain `quarantined` (never decoded) is NOT
|
|
788
|
+
// done: a crash between a chunk's first WAV write and its first
|
|
789
|
+
// active-session persist quarantines the audio, and the phone's retry then
|
|
790
|
+
// starts the session again. That file is adopted exactly as before.
|
|
791
|
+
if (typeof data.sessionId === 'string' && isTerminalCaptureFile(data.sessionId, data.lastActivityAt, resolve(CHUNK_PERSIST_DIR, file))) {
|
|
792
|
+
const terminal = readCaptureDisposition(data.sessionId)!
|
|
793
|
+
const target = moveActiveSessionFileAside(
|
|
794
|
+
resolve(CHUNK_PERSIST_DIR, file), data.sessionId, STALE_ACTIVE_SESSION_DIR, 'terminal_capture', Date.now(),
|
|
795
|
+
{ lastActivityAt: typeof data.lastActivityAt === 'number' ? data.lastActivityAt : null },
|
|
796
|
+
)
|
|
797
|
+
if (target) console.warn(`[session-recovery] ${data.sessionId} was already ${terminal.kind}; moved its active-session file aside: ${target}`)
|
|
798
|
+
else logStaleMoveFailure(data.sessionId, `[session-recovery] Could not move the active-session file aside for ${data.sessionId} (already ${terminal.kind}); left in place, not adopted`)
|
|
799
|
+
continue
|
|
800
|
+
}
|
|
704
801
|
// Reconstruct the (sparse) chunk array. New format keeps original
|
|
705
802
|
// indices via chunksIndexed; legacy format stored a dense `chunks`.
|
|
706
803
|
const indexed: Array<{ i: number; c: TranscriptChunk }> | null =
|
|
@@ -859,7 +956,9 @@ function recoverSessions(): void {
|
|
|
859
956
|
try {
|
|
860
957
|
if (existsSync(SESSION_AUDIO_DIR)) {
|
|
861
958
|
const actions = sweepOrphanedSessionAudio(SESSION_AUDIO_DIR, {
|
|
862
|
-
|
|
959
|
+
// `|| sessions.has`: deferred recovery (6.71.14) can run while sessions
|
|
960
|
+
// are live in memory; their audio is not orphaned.
|
|
961
|
+
isLive: id => recoveredIds.has(id) || sessions.has(id),
|
|
863
962
|
hasFreshPreservedMarker: hasFreshPreservedAudioMarker,
|
|
864
963
|
reason: 'boot_sweep_unsaved',
|
|
865
964
|
})
|
|
@@ -896,9 +995,40 @@ export function isSessionDeleted(sessionId: string): boolean {
|
|
|
896
995
|
// A committed cross-boot maintenance operation owns durable state until the
|
|
897
996
|
// trusted controller adopts the candidate. Boot recovery must not mutate or
|
|
898
997
|
// promote sessions while that gate is closed.
|
|
899
|
-
|
|
998
|
+
//
|
|
999
|
+
// 6.71.14: and once that gate opens, recovery MUST still run. Until 6.71.13 this was
|
|
1000
|
+
// a module-scope `if` evaluated once at import: a boot behind a closed gate (every
|
|
1001
|
+
// managed update) skipped it for the life of the process. Persisted sessions were
|
|
1002
|
+
// never adopted or tombstoned, and the closed-session tombstones were never
|
|
1003
|
+
// restored. On 2026-10-12 that left `active-sessions/<id>.json` for a receipted
|
|
1004
|
+
// capture on disk for 17 hours, reported as live by COS Control and `missing` by
|
|
1005
|
+
// the status route.
|
|
1006
|
+
let durableSessionRecoveryRan = false
|
|
1007
|
+
function runDurableSessionRecovery(trigger: 'boot' | 'admissions_open'): void {
|
|
1008
|
+
if (durableSessionRecoveryRan) return
|
|
1009
|
+
durableSessionRecoveryRan = true
|
|
900
1010
|
recoverClosedSessions()
|
|
901
1011
|
recoverSessions()
|
|
1012
|
+
if (trigger === 'admissions_open') {
|
|
1013
|
+
console.log(`[session-recovery] Ran deferred durable recovery after the maintenance gate opened (${sessions.size} live session(s))`)
|
|
1014
|
+
}
|
|
1015
|
+
}
|
|
1016
|
+
if (maintenanceAdmissionsOpen()) {
|
|
1017
|
+
runDurableSessionRecovery('boot')
|
|
1018
|
+
} else {
|
|
1019
|
+
// Delivered on every later open; the flag above makes every delivery after the
|
|
1020
|
+
// first a no-op, so disk is never re-read over a live session map.
|
|
1021
|
+
onMaintenanceAdmissionsOpen(() => runDurableSessionRecovery('admissions_open'))
|
|
1022
|
+
}
|
|
1023
|
+
|
|
1024
|
+
/**
|
|
1025
|
+
* 6.71.14: has durable session recovery run in this process yet? Until it has, the
|
|
1026
|
+
* in-memory map does not yet hold the persisted sessions, so "not in memory" is not
|
|
1027
|
+
* evidence of anything. The status route answers 503 `recovery_pending` instead of
|
|
1028
|
+
* `missing` while this is false.
|
|
1029
|
+
*/
|
|
1030
|
+
export function durableSessionRecoveryHasRun(): boolean {
|
|
1031
|
+
return durableSessionRecoveryRan
|
|
902
1032
|
}
|
|
903
1033
|
|
|
904
1034
|
/**
|
|
@@ -1011,6 +1141,10 @@ function promoteStrandedCapture(sessionId: string, lastActivityAt: number): void
|
|
|
1011
1141
|
// Auto-cleanup sessions idle for the advertised retention horizon.
|
|
1012
1142
|
setInterval(() => {
|
|
1013
1143
|
if (!maintenanceAdmissionsOpen()) return
|
|
1144
|
+
// 6.71.14: the sweeps below judge a session dead by its absence from the
|
|
1145
|
+
// in-memory map. Before recovery has adopted the persisted sessions that map is
|
|
1146
|
+
// empty by construction, so sweeping then would quarantine a live capture.
|
|
1147
|
+
if (!durableSessionRecoveryRan) return
|
|
1014
1148
|
// A capture whose phone went away must still become a meeting. This loop used
|
|
1015
1149
|
// to close every idle session as 'expired', which quarantined its audio and
|
|
1016
1150
|
// produced NO meeting — recoverable for 72h, but only by whoever thought to
|
|
@@ -1053,6 +1187,15 @@ setInterval(() => {
|
|
|
1053
1187
|
// log, and nothing to reason about, because any throw in this block vanished.
|
|
1054
1188
|
console.error(`[cleanup] Orphan-audio sweep failed: ${error instanceof Error ? error.message : error}`)
|
|
1055
1189
|
}
|
|
1190
|
+
// 6.71.14: its own try, for the same reason as auto-recovery below. An
|
|
1191
|
+
// active-session file whose session is not live and has been idle past the
|
|
1192
|
+
// retention horizon is moved aside; COS Control reads that directory directly
|
|
1193
|
+
// and would otherwise show it as a live recording indefinitely.
|
|
1194
|
+
try {
|
|
1195
|
+
sweepStaleActiveSessionFilesNow(Date.now())
|
|
1196
|
+
} catch (error) {
|
|
1197
|
+
console.error(`[cleanup] Stale active-session sweep failed: ${error instanceof Error ? error.message : error}`)
|
|
1198
|
+
}
|
|
1056
1199
|
// DELIBERATELY ITS OWN TRY. This used to sit inside the block above, one line after
|
|
1057
1200
|
// purgeExpiredQuarantine, so a throw anywhere in that sweep meant auto-recovery
|
|
1058
1201
|
// silently never ran — which is exactly what happened in production on 6.23.1
|