@gotcos/glasses-server 6.22.1 → 6.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +54 -0
- package/package.json +2 -2
- package/server/lib/session-heartbeats.ts +62 -0
- package/server/lib/stranded-session-actions.ts +302 -0
- package/server/lib/stranded-sessions.ts +125 -0
- package/server/routes/diag.ts +14 -0
- package/server/routes/health.ts +19 -0
- package/server/routes/meeting.ts +9 -0
- package/server/routes/transcribe-stream.ts +147 -7
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,57 @@
|
|
|
1
|
+
## 6.23.0
|
|
2
|
+
|
|
3
|
+
A recording whose phone goes away now becomes a meeting on its own. Miles: "we end
|
|
4
|
+
up with a meeting that is orphaned that we have no ability to keep."
|
|
5
|
+
|
|
6
|
+
Found live while writing this: two sessions stranded for 184 and 24 minutes, holding
|
|
7
|
+
the restart lock, while `GET /api/meeting/orphans` answered `count: 0`. Both saved at
|
|
8
|
+
100% transfer integrity (529/529 and 23/23 chunks). Nothing had been lost — but
|
|
9
|
+
nothing was going to turn them into meetings either.
|
|
10
|
+
|
|
11
|
+
- **The audio was never the problem.** A stranded capture stays live in memory for 4
|
|
12
|
+
hours, then closes as `expired` and its chunks move to quarantine for 72 more. A
|
|
13
|
+
76-hour window in which the audio exists and NOTHING converts it into a meeting
|
|
14
|
+
unless a human notices. Expiry produced preserved evidence, not a meeting.
|
|
15
|
+
- **The 60-second sweeper already existed and already detected these.** It called
|
|
16
|
+
`closeTranscriptSession(id, 'expired')`. The change is the disposition at the
|
|
17
|
+
cutoff, not new scheduling: it now finalizes through `POST /api/meeting/save`,
|
|
18
|
+
which keeps the live ASR transcript and its speaker labels. The quarantine recover
|
|
19
|
+
route was the wrong tool here — its output labels every speaker Unknown, because no
|
|
20
|
+
live ASR ever ran on it.
|
|
21
|
+
- **Staleness never closes a session early, and it must not.** The companion buffers
|
|
22
|
+
to IndexedDB while iOS suspends the WebView and drains on foreground, and it
|
|
23
|
+
restores `restoredSessionId` across a relaunch — so a phone silent for 30 minutes
|
|
24
|
+
can still deliver its tail into the same session id. Close it and `isSessionDeleted`
|
|
25
|
+
answers 410 Gone: a truncated meeting AND a second orphan. At the stale threshold a
|
|
26
|
+
readable draft is written and the session stays open.
|
|
27
|
+
- **`/api/meeting/orphans` and `/api/health` were blind to the state that matters.**
|
|
28
|
+
Both listed only QUARANTINED directories, and a stranded session is not quarantined
|
|
29
|
+
for four hours. New `stranded` / `stranded_captures` report idle minutes, captured
|
|
30
|
+
minutes, chunk count, when the sweeper will save it, and whether a draft exists.
|
|
31
|
+
- **Quiet is not failure.** A heartbeat carrying `audioState` is now kept per session
|
|
32
|
+
and can VETO a stale verdict — a phone that says it is recording is alive even with
|
|
33
|
+
no chunk arriving, because its uploads may merely be blocked. A BACKGROUNDED phone
|
|
34
|
+
counts as capturing; requiring `visibilityState: visible` would reap exactly the
|
|
35
|
+
sessions the drain path exists to rescue. Absence proves nothing in the other
|
|
36
|
+
direction: `clientLog` is fire-and-forget and lossy, so a missing heartbeat can
|
|
37
|
+
never itself mark a session dead. Chunk arrival decides.
|
|
38
|
+
- **One definition of stale.** `RECORDING_SESSION_STALE_MS` is now derived from
|
|
39
|
+
`STRANDED_STALE_MS` rather than being a second literal. Two subsystems disagreeing
|
|
40
|
+
about what a live recording is, is precisely how the panel could read "2
|
|
41
|
+
recording(s) active" while the orphan endpoint reported none.
|
|
42
|
+
- A failed auto-save does not throw away a savable capture: `no_token` and 5xx and
|
|
43
|
+
transport failures retry on the next sweep, 409 yields to the save that already owns
|
|
44
|
+
the session, and only a terminal 4xx (or an 8-hour backstop) falls back to the old
|
|
45
|
+
close-and-quarantine.
|
|
46
|
+
|
|
47
|
+
Coverage: 59 tests over the new modules, 23 mutations all caught. Three of those
|
|
48
|
+
initially SURVIVED — the whole stale branch deleted, the token gate deleted, and the
|
|
49
|
+
terminal draft cleanup deleted — because the loop lived inside a `setInterval` in a
|
|
50
|
+
module no test can import without executing boot recovery, a timer, and writes to the
|
|
51
|
+
real data home. It was extracted with injected dependencies rather than covered by
|
|
52
|
+
source-shape assertions. Full suite green serially (1411 tests); two unrelated files
|
|
53
|
+
flake under file parallelism, which is a pre-existing isolation bug.
|
|
54
|
+
|
|
1
55
|
## 6.22.1
|
|
2
56
|
|
|
3
57
|
Notes attached from somewhere else by a symlink are now read properly. Found by
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@gotcos/glasses-server",
|
|
3
|
-
"version": "6.
|
|
4
|
-
"description": "COS Glasses
|
|
3
|
+
"version": "6.23.0",
|
|
4
|
+
"description": "COS Glasses \u2014 self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"glasses-server": "bin/cli.cjs",
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
// Last-known companion state per recording session.
|
|
2
|
+
//
|
|
3
|
+
// The stranded-session sweeper needs POSITIVE evidence that a phone is still
|
|
4
|
+
// capturing before it declares a quiet session stale. That evidence already
|
|
5
|
+
// arrives on POST /api/diag/client as a heartbeat carrying `audioState` and
|
|
6
|
+
// `visibilityState`, but it was only ever appended to client-diagnostics.jsonl —
|
|
7
|
+
// a file the sweeper would have to parse on every 60s tick. This keeps the newest
|
|
8
|
+
// heartbeat per session in memory instead.
|
|
9
|
+
//
|
|
10
|
+
// DELIBERATELY LOSSY, and safe to be. `clientLog` is fire-and-forget with a 3s
|
|
11
|
+
// abort and a silent catch: 63% of heartbeats were measured missing during one
|
|
12
|
+
// 2026-07-27 session. So an ABSENT heartbeat proves nothing and must never by
|
|
13
|
+
// itself mark a session dead — the sweeper's staleness signal is chunk arrival,
|
|
14
|
+
// and a heartbeat can only ever VETO a stale verdict, never cause one.
|
|
15
|
+
|
|
16
|
+
import type { SessionHeartbeat } from './stranded-sessions.js'
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* Bounded so a burst of zombie clients cannot grow this without limit. Well above
|
|
20
|
+
* any real fleet: a session is one phone, and the sweeper deletes entries as
|
|
21
|
+
* sessions close.
|
|
22
|
+
*/
|
|
23
|
+
const MAX_TRACKED_SESSIONS = 64
|
|
24
|
+
|
|
25
|
+
const heartbeats = new Map<string, SessionHeartbeat>()
|
|
26
|
+
|
|
27
|
+
/** Record the newest heartbeat for a session. Older timestamps are ignored. */
|
|
28
|
+
export function recordSessionHeartbeat(
|
|
29
|
+
sessionId: string,
|
|
30
|
+
heartbeat: SessionHeartbeat,
|
|
31
|
+
): void {
|
|
32
|
+
if (!sessionId) return
|
|
33
|
+
if (!Number.isFinite(heartbeat.at)) return
|
|
34
|
+
const existing = heartbeats.get(sessionId)
|
|
35
|
+
if (existing && existing.at > heartbeat.at) return
|
|
36
|
+
heartbeats.set(sessionId, heartbeat)
|
|
37
|
+
if (heartbeats.size <= MAX_TRACKED_SESSIONS) return
|
|
38
|
+
// Evict oldest-first so the entries that matter (recent, therefore capable of
|
|
39
|
+
// vetoing a stale verdict) are the ones retained.
|
|
40
|
+
const ordered = [...heartbeats.entries()].sort((a, b) => a[1].at - b[1].at)
|
|
41
|
+
for (const [id] of ordered.slice(0, heartbeats.size - MAX_TRACKED_SESSIONS)) {
|
|
42
|
+
heartbeats.delete(id)
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
export function getSessionHeartbeat(sessionId: string): SessionHeartbeat | null {
|
|
47
|
+
return heartbeats.get(sessionId) ?? null
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/** Called when a session reaches any terminal state. */
|
|
51
|
+
export function forgetSessionHeartbeat(sessionId: string): void {
|
|
52
|
+
heartbeats.delete(sessionId)
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/** Test seam only. */
|
|
56
|
+
export function resetSessionHeartbeats(): void {
|
|
57
|
+
heartbeats.clear()
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
export function trackedHeartbeatCount(): number {
|
|
61
|
+
return heartbeats.size
|
|
62
|
+
}
|
|
@@ -0,0 +1,302 @@
|
|
|
1
|
+
// What the sweeper DOES about a stranded session: draft it, then promote it.
|
|
2
|
+
//
|
|
3
|
+
// Two actions, deliberately asymmetric.
|
|
4
|
+
//
|
|
5
|
+
// DRAFT (at the stale threshold) is additive and reversible. It writes the
|
|
6
|
+
// transcript the session already holds to a file and touches nothing else — no
|
|
7
|
+
// store write, no audio move, no session close. That matters because a phone that
|
|
8
|
+
// has been silent for 25 minutes can still drain its IndexedDB buffer into the
|
|
9
|
+
// same session id, and closing early would answer that drain with 410 Gone. A
|
|
10
|
+
// draft cannot truncate anything; it only makes the capture readable and visible
|
|
11
|
+
// while the session stays open for the tail.
|
|
12
|
+
//
|
|
13
|
+
// It also stays OUT of the meetings store on purpose. `POST /api/meeting/save` is
|
|
14
|
+
// idempotent by session id — a second save replays the first receipt. If a draft
|
|
15
|
+
// occupied that id, the real save arriving later with the drained tail would find
|
|
16
|
+
// `alreadySaved` and return the SHORTER checkpoint forever. The draft is a
|
|
17
|
+
// sidecar, never a meeting.
|
|
18
|
+
//
|
|
19
|
+
// PROMOTE (at the retention cutoff) goes through POST /api/meeting/save over
|
|
20
|
+
// loopback rather than calling the store directly. That route owns the maintenance
|
|
21
|
+
// lease, the save-in-progress lock, the receipt contract, domain inference and the
|
|
22
|
+
// two-phase finalization job; reimplementing any of it here would fork the most
|
|
23
|
+
// load-bearing path in the product. The session is still in memory at this point,
|
|
24
|
+
// so the save keeps the live ASR transcript and its speaker labels — which is why
|
|
25
|
+
// promote must not use the quarantine recover route, whose output labels every
|
|
26
|
+
// speaker Unknown because no live ASR ever ran on it.
|
|
27
|
+
|
|
28
|
+
import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, statSync } from 'node:fs'
|
|
29
|
+
import { resolve } from 'node:path'
|
|
30
|
+
import { atomicWriteFileSync } from './atomic-fs.js'
|
|
31
|
+
import { forgetSessionHeartbeat } from './session-heartbeats.js'
|
|
32
|
+
import { classifyStrandedSession, type SessionHeartbeat } from './stranded-sessions.js'
|
|
33
|
+
|
|
34
|
+
/** Title given to a capture the sweeper saves on the user's behalf. */
|
|
35
|
+
export const STRANDED_PROMOTE_TITLE = 'Auto-saved capture'
|
|
36
|
+
|
|
37
|
+
const DRAFT_SUFFIX = '.draft.md'
|
|
38
|
+
/** A draft is a safety net, not an archive. Bounded so it cannot grow unwatched. */
|
|
39
|
+
const MAX_DRAFT_CHARS = 400_000
|
|
40
|
+
|
|
41
|
+
export interface StrandedDraftInput {
|
|
42
|
+
sessionId: string
|
|
43
|
+
transcript: string
|
|
44
|
+
chunkCount: number
|
|
45
|
+
startedAt: number
|
|
46
|
+
lastActivityAt: number
|
|
47
|
+
/** Injected for deterministic tests. */
|
|
48
|
+
now?: number
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
export interface StrandedDraft {
|
|
52
|
+
sessionId: string
|
|
53
|
+
path: string
|
|
54
|
+
bytes: number
|
|
55
|
+
chunkCount: number
|
|
56
|
+
updatedAt: string
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
function draftPath(dir: string, sessionId: string): string {
|
|
60
|
+
return resolve(dir, `${sessionId}${DRAFT_SUFFIX}`)
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/**
|
|
64
|
+
* Write (or refresh) the draft for a stranded session.
|
|
65
|
+
*
|
|
66
|
+
* Returns null when there is nothing worth drafting, so an empty session cannot
|
|
67
|
+
* litter the directory with contentless files.
|
|
68
|
+
*/
|
|
69
|
+
export function writeStrandedDraft(dir: string, input: StrandedDraftInput): string | null {
|
|
70
|
+
const transcript = input.transcript.trim()
|
|
71
|
+
if (!transcript) return null
|
|
72
|
+
if (!input.sessionId) return null
|
|
73
|
+
const now = input.now ?? Date.now()
|
|
74
|
+
const idleMin = Math.round(Math.max(0, now - input.lastActivityAt) / 60_000)
|
|
75
|
+
const capturedMin = Math.round(Math.max(0, input.lastActivityAt - input.startedAt) / 60_000)
|
|
76
|
+
const body = transcript.length > MAX_DRAFT_CHARS
|
|
77
|
+
? `${transcript.slice(0, MAX_DRAFT_CHARS)}\n\n[draft truncated at ${MAX_DRAFT_CHARS} characters]`
|
|
78
|
+
: transcript
|
|
79
|
+
const doc = [
|
|
80
|
+
'---',
|
|
81
|
+
`session_id: ${input.sessionId}`,
|
|
82
|
+
`state: stranded_draft`,
|
|
83
|
+
`chunks_received: ${input.chunkCount}`,
|
|
84
|
+
`captured_minutes: ${capturedMin}`,
|
|
85
|
+
`idle_minutes: ${idleMin}`,
|
|
86
|
+
`drafted_at: ${new Date(now).toISOString()}`,
|
|
87
|
+
'---',
|
|
88
|
+
'',
|
|
89
|
+
'> Draft only. This capture stopped receiving audio and has not been saved as a',
|
|
90
|
+
'> meeting yet. The session is still open, so if the phone reconnects the rest of',
|
|
91
|
+
'> the audio will still arrive. It becomes a real meeting automatically at the',
|
|
92
|
+
`> retention cutoff, or immediately via POST /api/meeting/save.`,
|
|
93
|
+
'',
|
|
94
|
+
body,
|
|
95
|
+
'',
|
|
96
|
+
].join('\n')
|
|
97
|
+
try {
|
|
98
|
+
mkdirSync(dir, { recursive: true })
|
|
99
|
+
atomicWriteFileSync(draftPath(dir, input.sessionId), doc, { mode: 0o600 })
|
|
100
|
+
return draftPath(dir, input.sessionId)
|
|
101
|
+
} catch {
|
|
102
|
+
return null
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
/** Every draft currently on disk, newest first. */
|
|
107
|
+
export function listStrandedDrafts(dir: string): StrandedDraft[] {
|
|
108
|
+
if (!existsSync(dir)) return []
|
|
109
|
+
const out: StrandedDraft[] = []
|
|
110
|
+
let entries: string[]
|
|
111
|
+
try { entries = readdirSync(dir) } catch { return [] }
|
|
112
|
+
for (const name of entries) {
|
|
113
|
+
if (!name.endsWith(DRAFT_SUFFIX)) continue
|
|
114
|
+
const full = resolve(dir, name)
|
|
115
|
+
try {
|
|
116
|
+
const st = statSync(full)
|
|
117
|
+
if (!st.isFile()) continue
|
|
118
|
+
const head = readFileSync(full, 'utf8').slice(0, 800)
|
|
119
|
+
const chunks = /^chunks_received: (\d+)$/m.exec(head)
|
|
120
|
+
out.push({
|
|
121
|
+
sessionId: name.slice(0, -DRAFT_SUFFIX.length),
|
|
122
|
+
path: full,
|
|
123
|
+
bytes: st.size,
|
|
124
|
+
chunkCount: chunks ? Number(chunks[1]) : 0,
|
|
125
|
+
updatedAt: new Date(st.mtimeMs).toISOString(),
|
|
126
|
+
})
|
|
127
|
+
} catch { /* a draft we cannot stat is not worth failing the sweep over */ }
|
|
128
|
+
}
|
|
129
|
+
return out.sort((a, b) => b.updatedAt.localeCompare(a.updatedAt))
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
/** Called once a session reaches a real terminal state. Never throws. */
|
|
133
|
+
export function clearStrandedDraft(dir: string, sessionId: string): void {
|
|
134
|
+
try { rmSync(draftPath(dir, sessionId), { force: true }) } catch {}
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
/**
|
|
138
|
+
* Drop every trace of stranded-state for a session that reached ANY terminal state.
|
|
139
|
+
*
|
|
140
|
+
* One function because the two halves must never diverge. A retained draft would
|
|
141
|
+
* keep /api/meeting/orphans advertising an unsaved capture that is now saved — the
|
|
142
|
+
* false alarm that trains a user to ignore the one channel built to report real
|
|
143
|
+
* losses. A retained heartbeat would let a dead session's last breath veto a stale
|
|
144
|
+
* verdict for a session id that no longer exists.
|
|
145
|
+
*/
|
|
146
|
+
export function releaseStrandedState(dir: string, sessionId: string): void {
|
|
147
|
+
clearStrandedDraft(dir, sessionId)
|
|
148
|
+
forgetSessionHeartbeat(sessionId)
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
export interface SweepableSession {
|
|
152
|
+
lastActivityAt: number
|
|
153
|
+
startTime: number
|
|
154
|
+
chunkCount: number
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
export interface StrandedSweepInput {
|
|
158
|
+
now: number
|
|
159
|
+
/** Empty means this process does not own the API — see the guard below. */
|
|
160
|
+
token: string
|
|
161
|
+
draftDir: string
|
|
162
|
+
sessions: Iterable<[string, SweepableSession]>
|
|
163
|
+
getHeartbeat: (sessionId: string) => SessionHeartbeat | null
|
|
164
|
+
getTranscript: (sessionId: string) => string | null
|
|
165
|
+
/** Fire-and-forget: a promote can run for minutes and must not block the tick. */
|
|
166
|
+
onPromote: (sessionId: string, lastActivityAt: number) => void
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
export interface StrandedSweepResult {
|
|
170
|
+
drafted: string[]
|
|
171
|
+
promoted: string[]
|
|
172
|
+
/** Sessions the sweep deliberately left alone. */
|
|
173
|
+
live: string[]
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
/**
|
|
177
|
+
* One pass over the live sessions: draft what has gone quiet, promote what is done.
|
|
178
|
+
*
|
|
179
|
+
* EXTRACTED FROM THE INTERVAL ON PURPOSE. While this loop lived inside the
|
|
180
|
+
* `setInterval` in routes/transcribe-stream.ts no test could reach it, because
|
|
181
|
+
* importing that module executes boot recovery, a 60s timer, and reads and writes
|
|
182
|
+
* against the real data home. Two mutations proved the cost: deleting the entire
|
|
183
|
+
* stale branch, and deleting the token gate, both SURVIVED the whole suite. A loop
|
|
184
|
+
* that decides whether a recording ever becomes a meeting cannot be covered by
|
|
185
|
+
* source-shape assertions alone.
|
|
186
|
+
*/
|
|
187
|
+
export function sweepStrandedSessions(input: StrandedSweepInput): StrandedSweepResult {
|
|
188
|
+
const result: StrandedSweepResult = { drafted: [], promoted: [], live: [] }
|
|
189
|
+
// Only the process that owns the API can save anything, and only it should be
|
|
190
|
+
// writing into the data home. No token means a test worker or a tool that
|
|
191
|
+
// imported the module, so the pass does nothing rather than half-acting.
|
|
192
|
+
if (!input.token) return result
|
|
193
|
+
for (const [sessionId, session] of input.sessions) {
|
|
194
|
+
const verdict = classifyStrandedSession(
|
|
195
|
+
{ lastActivityAt: session.lastActivityAt, heartbeat: input.getHeartbeat(sessionId) },
|
|
196
|
+
input.now,
|
|
197
|
+
)
|
|
198
|
+
if (verdict === 'live') { result.live.push(sessionId); continue }
|
|
199
|
+
if (verdict === 'stale') {
|
|
200
|
+
// Draft only. The session stays OPEN so a phone that reconnects can still
|
|
201
|
+
// drain its buffered tail into this same id.
|
|
202
|
+
const written = writeStrandedDraft(input.draftDir, {
|
|
203
|
+
sessionId,
|
|
204
|
+
transcript: input.getTranscript(sessionId) ?? '',
|
|
205
|
+
chunkCount: session.chunkCount,
|
|
206
|
+
startedAt: session.startTime,
|
|
207
|
+
lastActivityAt: session.lastActivityAt,
|
|
208
|
+
now: input.now,
|
|
209
|
+
})
|
|
210
|
+
if (written) result.drafted.push(sessionId)
|
|
211
|
+
continue
|
|
212
|
+
}
|
|
213
|
+
input.onPromote(sessionId, session.lastActivityAt)
|
|
214
|
+
result.promoted.push(sessionId)
|
|
215
|
+
}
|
|
216
|
+
return result
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
export interface PromoteResult {
|
|
220
|
+
ok: boolean
|
|
221
|
+
status: number
|
|
222
|
+
filename?: string
|
|
223
|
+
reason?: string
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
/**
|
|
227
|
+
* After a failed auto-save, does the session close (quarantining its audio) or wait
|
|
228
|
+
* for the next sweep?
|
|
229
|
+
*
|
|
230
|
+
* Extracted from the sweeper because it is the decision with the real consequence:
|
|
231
|
+
* closing means the capture becomes quarantined audio and NOT a meeting, which is
|
|
232
|
+
* the exact outcome this release exists to prevent. Inside the route it was
|
|
233
|
+
* untestable without importing and therefore executing the whole module.
|
|
234
|
+
*
|
|
235
|
+
* @param idleForMs how long the session has been without a chunk
|
|
236
|
+
* @param giveUpAfterMs backstop so a permanently-failing save cannot keep a
|
|
237
|
+
* session in memory forever
|
|
238
|
+
*/
|
|
239
|
+
export function shouldCloseAfterFailedPromote(
|
|
240
|
+
result: PromoteResult,
|
|
241
|
+
idleForMs: number,
|
|
242
|
+
giveUpAfterMs: number,
|
|
243
|
+
): boolean {
|
|
244
|
+
if (result.ok) return false
|
|
245
|
+
// Never reached the server, so nothing was learned about whether it could be
|
|
246
|
+
// saved. Closing here would destroy a savable capture.
|
|
247
|
+
if (result.reason === 'no_token') return false
|
|
248
|
+
// Another save already owns this session. That one wins.
|
|
249
|
+
if (result.status === 409) return idleForMs >= giveUpAfterMs
|
|
250
|
+
// A transport failure, a timeout, or a 5xx is transient: retry next sweep.
|
|
251
|
+
const terminal = result.status >= 400 && result.status < 500
|
|
252
|
+
return terminal || idleForMs >= giveUpAfterMs
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
export interface PromoteOptions {
|
|
256
|
+
port: number
|
|
257
|
+
token: string
|
|
258
|
+
/** Injected in tests; defaults to global fetch. */
|
|
259
|
+
fetchImpl?: typeof fetch
|
|
260
|
+
timeoutMs?: number
|
|
261
|
+
}
|
|
262
|
+
|
|
263
|
+
/**
|
|
264
|
+
* Finalize a stranded session into a meeting through the real save route.
|
|
265
|
+
*
|
|
266
|
+
* Omits `domain` deliberately so the route's own keyword inference files it,
|
|
267
|
+
* rather than dumping every unattended capture into one folder.
|
|
268
|
+
*/
|
|
269
|
+
export async function promoteStrandedSession(
|
|
270
|
+
sessionId: string,
|
|
271
|
+
options: PromoteOptions,
|
|
272
|
+
): Promise<PromoteResult> {
|
|
273
|
+
// No token means this process is not the server that owns the API — a unit test
|
|
274
|
+
// worker, a tool importing the module. Refusing here matters because an
|
|
275
|
+
// unauthenticated POST returns 401, the caller would read a 4xx as terminal, and
|
|
276
|
+
// it would then close a session it could perfectly well have saved. `no_token`
|
|
277
|
+
// is deliberately NOT terminal.
|
|
278
|
+
if (!options.token) return { ok: false, status: 0, reason: 'no_token' }
|
|
279
|
+
const doFetch = options.fetchImpl ?? fetch
|
|
280
|
+
const controller = new AbortController()
|
|
281
|
+
const timer = setTimeout(() => controller.abort(), options.timeoutMs ?? 600_000)
|
|
282
|
+
try {
|
|
283
|
+
const res = await doFetch(`http://127.0.0.1:${options.port}/api/meeting/save`, {
|
|
284
|
+
method: 'POST',
|
|
285
|
+
headers: { 'Content-Type': 'application/json', 'X-Cos-Token': options.token },
|
|
286
|
+
body: JSON.stringify({ sessionId, title: STRANDED_PROMOTE_TITLE }),
|
|
287
|
+
signal: controller.signal,
|
|
288
|
+
})
|
|
289
|
+
let payload: Record<string, unknown> = {}
|
|
290
|
+
try { payload = await res.json() as Record<string, unknown> } catch {}
|
|
291
|
+
return {
|
|
292
|
+
ok: res.ok,
|
|
293
|
+
status: res.status,
|
|
294
|
+
filename: typeof payload.filename === 'string' ? payload.filename : undefined,
|
|
295
|
+
reason: typeof payload.reason === 'string' ? payload.reason : undefined,
|
|
296
|
+
}
|
|
297
|
+
} catch (error: any) {
|
|
298
|
+
return { ok: false, status: 0, reason: error?.name === 'AbortError' ? 'timeout' : 'request_failed' }
|
|
299
|
+
} finally {
|
|
300
|
+
clearTimeout(timer)
|
|
301
|
+
}
|
|
302
|
+
}
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
// A recording whose phone went away must still become a meeting.
|
|
2
|
+
//
|
|
3
|
+
// THE DEFECT THIS FIXES. When the companion quits, the bridge drops, or the G2
|
|
4
|
+
// disconnects mid-meeting, the server keeps the session in memory and keeps its
|
|
5
|
+
// ACK'd chunks on disk. Nothing then converts them into a meeting. At the 4-hour
|
|
6
|
+
// retention cutoff the session was closed as 'expired' and its audio moved to
|
|
7
|
+
// quarantine — preserved for 72 more hours, but still not a meeting, and visible
|
|
8
|
+
// only to whoever thought to look. Two sessions sat stranded for 184 and 24
|
|
9
|
+
// minutes on 2026-08-09 holding the restart lock while /api/meeting/orphans
|
|
10
|
+
// reported nothing at all, because that endpoint lists only QUARANTINED dirs and
|
|
11
|
+
// a stranded session is not quarantined for four hours.
|
|
12
|
+
//
|
|
13
|
+
// WHY THE OBVIOUS FIX IS WRONG. "Close idle sessions sooner" loses data. The
|
|
14
|
+
// companion buffers capture to IndexedDB while iOS suspends the WebView and
|
|
15
|
+
// drains on foreground (local-first-meeting-uploader.ts keeps a retry queue), and
|
|
16
|
+
// it restores `restoredSessionId` across a relaunch — so a phone that has been
|
|
17
|
+
// silent for 25 minutes can still deliver its tail into the SAME session id. Once
|
|
18
|
+
// the session is closed, `isSessionDeleted` answers 410 Gone and that tail is
|
|
19
|
+
// gone, leaving a truncated meeting AND a second orphan. So staleness must never
|
|
20
|
+
// close a session early. It may only make the state visible and durable.
|
|
21
|
+
//
|
|
22
|
+
// QUIET IS NOT FAILURE. `lastActivityAt` is bumped in exactly one place —
|
|
23
|
+
// `processStreamChunk` — so it is a pure chunk-ARRIVAL signal and a zombie
|
|
24
|
+
// client's heartbeats cannot mask a dead session. The inverse also has to hold:
|
|
25
|
+
// a phone that reports it is recording right now is alive even with no chunk
|
|
26
|
+
// arriving, because its uploads may merely be blocked. That is why a fresh
|
|
27
|
+
// heartbeat suppresses the stale verdict, and why a BACKGROUNDED phone is not
|
|
28
|
+
// treated as dead — buffering while suspended is designed behavior, not a fault.
|
|
29
|
+
|
|
30
|
+
import { LOCAL_FIRST_MEETING_IDLE_RETENTION_MS } from './local-first-meetings-contract.js'
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* No ACK'd chunk for this long and the session is worth telling the user about.
|
|
34
|
+
*
|
|
35
|
+
* MUST equal `RECORDING_SESSION_STALE_MS` in routes/transcribe-stream.ts, which
|
|
36
|
+
* already defines staleness for the maintenance drain gate and for aborting batch
|
|
37
|
+
* work while a recording is live. A second, different threshold is exactly the bug
|
|
38
|
+
* this release fixes: the session counter said "2 recordings active" while
|
|
39
|
+
* /api/meeting/orphans said nothing, because two subsystems disagreed about what
|
|
40
|
+
* counts as a real recording. One definition, or the disagreement comes back.
|
|
41
|
+
* `stranded-sessions.test.ts` imports both and fails if they drift.
|
|
42
|
+
*
|
|
43
|
+
* It lives here rather than being imported from the route because lib must not
|
|
44
|
+
* depend on routes; the test is what keeps them married.
|
|
45
|
+
*
|
|
46
|
+
* Chunk cadence is roughly 10s, so 30 minutes is ~180 missed chunks: decisively
|
|
47
|
+
* not a hiccup. It is deliberately NOT a close — see the header. Nothing is
|
|
48
|
+
* destroyed at this threshold; a draft is written and the session stays open so a
|
|
49
|
+
* late drain still lands.
|
|
50
|
+
*/
|
|
51
|
+
export const STRANDED_STALE_MS = 30 * 60_000
|
|
52
|
+
|
|
53
|
+
/**
|
|
54
|
+
* The session is finished and becomes a meeting.
|
|
55
|
+
*
|
|
56
|
+
* Identical to the cutoff that already closed sessions as 'expired', so this
|
|
57
|
+
* changes the DISPOSITION (save instead of quarantine) and not the timing. Do not
|
|
58
|
+
* shorten it without re-reading the drain discussion above.
|
|
59
|
+
*/
|
|
60
|
+
export const STRANDED_PROMOTE_MS = LOCAL_FIRST_MEETING_IDLE_RETENTION_MS
|
|
61
|
+
|
|
62
|
+
/** A heartbeat older than this proves nothing about the phone's state now. */
|
|
63
|
+
export const LIVENESS_GRACE_MS = 3 * 60_000
|
|
64
|
+
|
|
65
|
+
/**
|
|
66
|
+
* Pipeline states that mean audio is being captured this moment.
|
|
67
|
+
*
|
|
68
|
+
* Sourced from the companion's audio pipeline, whose recording state is
|
|
69
|
+
* `recording_continuous` for a meeting; `recording` covers the dictation path.
|
|
70
|
+
*/
|
|
71
|
+
const CAPTURING_AUDIO_STATES = new Set(['recording_continuous', 'recording'])
|
|
72
|
+
|
|
73
|
+
export type StrandedVerdict = 'live' | 'stale' | 'promote'
|
|
74
|
+
|
|
75
|
+
export interface SessionHeartbeat {
|
|
76
|
+
/** ms since epoch when the heartbeat arrived. */
|
|
77
|
+
at: number
|
|
78
|
+
audioState?: string | null
|
|
79
|
+
visibilityState?: string | null
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
export interface SessionActivity {
|
|
83
|
+
/** ms since epoch of the last ACK'd chunk. NOT bumped by heartbeats. */
|
|
84
|
+
lastActivityAt: number
|
|
85
|
+
/** Most recent heartbeat for this session, when one has ever arrived. */
|
|
86
|
+
heartbeat?: SessionHeartbeat | null
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
/** How long this session has been without an ACK'd chunk. Never negative. */
|
|
90
|
+
export function strandedForMs(activity: SessionActivity, now: number): number {
|
|
91
|
+
return Math.max(0, now - activity.lastActivityAt)
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
/**
|
|
95
|
+
* Is the phone capturing audio right now?
|
|
96
|
+
*
|
|
97
|
+
* Deliberately does NOT require `visibilityState === 'visible'`. A backgrounded
|
|
98
|
+
* companion buffering to IndexedDB is the single most common legitimate reason
|
|
99
|
+
* for quiet, and demanding visibility here would reap exactly the sessions the
|
|
100
|
+
* drain path exists to rescue.
|
|
101
|
+
*/
|
|
102
|
+
export function capturingNow(heartbeat: SessionHeartbeat | null | undefined, now: number): boolean {
|
|
103
|
+
if (!heartbeat) return false
|
|
104
|
+
if (!Number.isFinite(heartbeat.at)) return false
|
|
105
|
+
if (now - heartbeat.at > LIVENESS_GRACE_MS) return false
|
|
106
|
+
// A heartbeat from the future is a clock artifact, not evidence of capture.
|
|
107
|
+
if (heartbeat.at - now > LIVENESS_GRACE_MS) return false
|
|
108
|
+
return CAPTURING_AUDIO_STATES.has((heartbeat.audioState ?? '').trim())
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
/**
|
|
112
|
+
* What should the sweeper do with this session?
|
|
113
|
+
*
|
|
114
|
+
* `promote` is unconditional at the retention cutoff, matching the close it
|
|
115
|
+
* replaces — a session that has been silent for four hours becomes a meeting even
|
|
116
|
+
* if something is still heartbeating, because that is already the behavior today
|
|
117
|
+
* and weakening it would strand audio for longer than the old code did.
|
|
118
|
+
*/
|
|
119
|
+
export function classifyStrandedSession(activity: SessionActivity, now: number): StrandedVerdict {
|
|
120
|
+
const idleFor = strandedForMs(activity, now)
|
|
121
|
+
if (idleFor >= STRANDED_PROMOTE_MS) return 'promote'
|
|
122
|
+
if (capturingNow(activity.heartbeat, now)) return 'live'
|
|
123
|
+
if (idleFor >= STRANDED_STALE_MS) return 'stale'
|
|
124
|
+
return 'live'
|
|
125
|
+
}
|
package/server/routes/diag.ts
CHANGED
|
@@ -18,6 +18,7 @@ import { fileURLToPath } from 'node:url'
|
|
|
18
18
|
|
|
19
19
|
const __dirname = fileURLToPath(new URL('.', import.meta.url))
|
|
20
20
|
import { dataPath } from '../lib/data-dir.js'
|
|
21
|
+
import { recordSessionHeartbeat } from '../lib/session-heartbeats.js'
|
|
21
22
|
const DIAG_DIR = dataPath()
|
|
22
23
|
const DIAG_FILE = resolve(DIAG_DIR, 'client-diagnostics.jsonl')
|
|
23
24
|
const MAX_FILE_BYTES = 10 * 1024 * 1024 // 10 MB cap
|
|
@@ -75,6 +76,19 @@ diagRouter.post('/diag/client', async (req, res) => {
|
|
|
75
76
|
return res.status(410).json({ error: 'session_deleted', sessionId })
|
|
76
77
|
}
|
|
77
78
|
} catch { /* import failure: fall through to normal logging */ }
|
|
79
|
+
// Positive liveness for the stranded-session sweeper. A phone can be capturing
|
|
80
|
+
// while its uploads are blocked, so chunk silence alone must not mark a
|
|
81
|
+
// session stale — this is the only signal that can say "still recording".
|
|
82
|
+
// Lossy by nature (fire-and-forget, 3s abort, silent catch), which is why it
|
|
83
|
+
// may only VETO a stale verdict and never cause one.
|
|
84
|
+
const heartbeatData = data as Record<string, unknown>
|
|
85
|
+
recordSessionHeartbeat(sessionId, {
|
|
86
|
+
at: ts,
|
|
87
|
+
audioState: typeof heartbeatData.audioState === 'string' ? heartbeatData.audioState : null,
|
|
88
|
+
visibilityState: typeof heartbeatData.visibilityState === 'string'
|
|
89
|
+
? heartbeatData.visibilityState
|
|
90
|
+
: null,
|
|
91
|
+
})
|
|
78
92
|
}
|
|
79
93
|
|
|
80
94
|
// Rate limit per sessionId (or remote address as fallback)
|
package/server/routes/health.ts
CHANGED
|
@@ -41,6 +41,7 @@ import { getLocalTtsHealth, refreshLocalTtsHealth } from '../lib/tts-local.js'
|
|
|
41
41
|
import { liveCuesCapability } from '../lib/live-cues-capability.js'
|
|
42
42
|
import { getMeetingSyncSnapshot } from '../lib/meeting-batch-progress.js'
|
|
43
43
|
import { listUnsavedCaptures } from '../lib/unsaved-audio-quarantine.js'
|
|
44
|
+
import { getStrandedCaptures } from './transcribe-stream.js'
|
|
44
45
|
import { getWhisperPreviewCapability } from '../lib/whisper-preview.js'
|
|
45
46
|
import { getTranscriptionProfileStatus } from '../lib/profile.js'
|
|
46
47
|
import { getHealthStaticProbes } from '../lib/health-static-probes.js'
|
|
@@ -235,6 +236,23 @@ healthRouter.get('/health', async (_req, res) => {
|
|
|
235
236
|
recovered: item.recovered,
|
|
236
237
|
})),
|
|
237
238
|
}
|
|
239
|
+
// Captures that stopped receiving audio but are NOT yet quarantined, so they do
|
|
240
|
+
// not appear in unsaved_captures for up to four hours. This is the state COS
|
|
241
|
+
// Control could not see on 2026-08-09: its panel read "2 recording(s) active.
|
|
242
|
+
// Restart is locked." while both had been silent for 184 and 24 minutes. Compact
|
|
243
|
+
// here; full detail on the authenticated orphans route.
|
|
244
|
+
const strandedList = getStrandedCaptures()
|
|
245
|
+
const stranded_captures = {
|
|
246
|
+
count: strandedList.length,
|
|
247
|
+
items: strandedList.slice(0, 10).map(item => ({
|
|
248
|
+
sessionId: item.sessionId,
|
|
249
|
+
idleMinutes: item.idleMinutes,
|
|
250
|
+
capturedMinutes: item.capturedMinutes,
|
|
251
|
+
chunks: item.chunks,
|
|
252
|
+
promotesAt: item.promotesAt,
|
|
253
|
+
hasDraft: item.draftPath != null,
|
|
254
|
+
})),
|
|
255
|
+
}
|
|
238
256
|
const meetingLibrary = resolveMeetingLibrary()
|
|
239
257
|
res.json({
|
|
240
258
|
...checks,
|
|
@@ -257,6 +275,7 @@ healthRouter.get('/health', async (_req, res) => {
|
|
|
257
275
|
warningCount: meetingLibrary.warnings.length,
|
|
258
276
|
},
|
|
259
277
|
unsaved_captures,
|
|
278
|
+
stranded_captures,
|
|
260
279
|
chunk_embeddings: chunkEmbeddings,
|
|
261
280
|
speaker_corrections: speakerCorrections,
|
|
262
281
|
review_audio: reviewAudio,
|
package/server/routes/meeting.ts
CHANGED
|
@@ -101,6 +101,7 @@ import {
|
|
|
101
101
|
getSessionStartTime,
|
|
102
102
|
getSessionTranscript,
|
|
103
103
|
getMeetingSessionStatus,
|
|
104
|
+
getStrandedCaptures,
|
|
104
105
|
getTranscriptionSessionLiveness,
|
|
105
106
|
hasSessionAudio,
|
|
106
107
|
moveSessionAudioToPending,
|
|
@@ -1664,8 +1665,16 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
1664
1665
|
? { segmentsDone: progress.segmentsDone, segmentsTotal: progress.segmentsTotal }
|
|
1665
1666
|
: null
|
|
1666
1667
|
}
|
|
1668
|
+
// `stranded` is the state this endpoint used to be blind to: a capture whose
|
|
1669
|
+
// phone went away, still live in memory, audio intact, NOT yet quarantined and
|
|
1670
|
+
// therefore absent from `items` for a full four hours. That blindness is why
|
|
1671
|
+
// two stranded sessions could hold the restart lock on 2026-08-09 while this
|
|
1672
|
+
// route answered count: 0.
|
|
1673
|
+
const stranded = getStrandedCaptures()
|
|
1667
1674
|
res.json({
|
|
1668
1675
|
count: items.filter(item => !item.recovered).length,
|
|
1676
|
+
strandedCount: stranded.length,
|
|
1677
|
+
stranded,
|
|
1669
1678
|
recovering: [...recoveringOrphans],
|
|
1670
1679
|
recoveringProgress,
|
|
1671
1680
|
items,
|
|
@@ -12,6 +12,17 @@ import { fileURLToPath } from 'node:url'
|
|
|
12
12
|
import { getVocabulary, getOwnerName } from '../lib/profile.js'
|
|
13
13
|
import { getOpenAIKey } from '../lib/openai-key.js'
|
|
14
14
|
import { getTranscriptionPolicySnapshot, isOpenAIWhisperFallbackReady } from '../lib/transcription-policy.js'
|
|
15
|
+
import { STRANDED_STALE_MS } from '../lib/stranded-sessions.js'
|
|
16
|
+
import {
|
|
17
|
+
clearStrandedDraft,
|
|
18
|
+
listStrandedDrafts,
|
|
19
|
+
releaseStrandedState,
|
|
20
|
+
shouldCloseAfterFailedPromote,
|
|
21
|
+
sweepStrandedSessions,
|
|
22
|
+
promoteStrandedSession,
|
|
23
|
+
writeStrandedDraft,
|
|
24
|
+
} from '../lib/stranded-session-actions.js'
|
|
25
|
+
import { getSessionHeartbeat } from '../lib/session-heartbeats.js'
|
|
15
26
|
import { TranscriptionUnavailableError } from '../lib/transcribe-audio.js'
|
|
16
27
|
|
|
17
28
|
const __dirname = fileURLToPath(new URL('.', import.meta.url))
|
|
@@ -168,6 +179,9 @@ const _unused_hallucination_constants_placeholder = 0 as const
|
|
|
168
179
|
const SESSION_AUDIO_DIR = dataPath('session-audio')
|
|
169
180
|
ensurePrivateDirectory(SESSION_AUDIO_DIR)
|
|
170
181
|
const PENDING_BATCH_DIR = dataPath('pending-batch')
|
|
182
|
+
// Readable text for a capture that stopped receiving audio. A sidecar, never a
|
|
183
|
+
// meeting — see lib/stranded-session-actions.ts for why it stays out of the store.
|
|
184
|
+
const STRANDED_DRAFT_DIR = dataPath('stranded-drafts')
|
|
171
185
|
ensurePrivateDirectory(PENDING_BATCH_DIR)
|
|
172
186
|
const MAX_SESSION_AUDIO_BYTES = 500 * 1024 * 1024 // 500MB cap per session (~2hr meeting ≈ 260MB)
|
|
173
187
|
const PRESERVED_SESSION_AUDIO_MARKER = '_meeting_save_preserved.marker'
|
|
@@ -354,8 +368,15 @@ export function getActiveTranscriptionSessionCount(): number {
|
|
|
354
368
|
* run to minutes. It sits deliberately far below
|
|
355
369
|
* LOCAL_FIRST_MEETING_IDLE_RETENTION_MS (4h), which governs how long chunks stay
|
|
356
370
|
* recoverable and is far too long to hold a restart on.
|
|
371
|
+
*
|
|
372
|
+
* DERIVED, never a second literal. `STRANDED_STALE_MS` is the one definition of
|
|
373
|
+
* "this recording has gone quiet", and the stranded sweeper, the drain gate and
|
|
374
|
+
* the batch-abort check must all mean the same thing by it. Two independent
|
|
375
|
+
* numbers here is exactly the defect this release fixes: on 2026-08-09 the
|
|
376
|
+
* session counter reported two active recordings while /api/meeting/orphans
|
|
377
|
+
* reported none, because two subsystems disagreed on what a live recording is.
|
|
357
378
|
*/
|
|
358
|
-
export const RECORDING_SESSION_STALE_MS =
|
|
379
|
+
export const RECORDING_SESSION_STALE_MS = STRANDED_STALE_MS
|
|
359
380
|
|
|
360
381
|
export interface TranscriptionSessionLiveness {
|
|
361
382
|
/** Sessions still plausibly recording. These block a restart. */
|
|
@@ -397,6 +418,53 @@ export interface TranscriptionSessionLiveness {
|
|
|
397
418
|
*/
|
|
398
419
|
export const __sessionsForTests = sessions
|
|
399
420
|
|
|
421
|
+
export interface StrandedCapture {
|
|
422
|
+
sessionId: string
|
|
423
|
+
/** Minutes since the last ACK'd chunk. */
|
|
424
|
+
idleMinutes: number
|
|
425
|
+
/** Minutes of audio actually captured before it went quiet. */
|
|
426
|
+
capturedMinutes: number
|
|
427
|
+
chunks: number
|
|
428
|
+
/** When the sweeper will save this on the user's behalf. */
|
|
429
|
+
promotesAt: string
|
|
430
|
+
/** Readable draft on disk, once the capture has been stale long enough. */
|
|
431
|
+
draftPath: string | null
|
|
432
|
+
draftBytes: number | null
|
|
433
|
+
}
|
|
434
|
+
|
|
435
|
+
/**
|
|
436
|
+
* Captures that stopped receiving audio but have NOT been saved yet.
|
|
437
|
+
*
|
|
438
|
+
* This is the row that was missing. `/api/meeting/orphans` listed only
|
|
439
|
+
* QUARANTINED directories, and a stranded session is not quarantined until the
|
|
440
|
+
* 4-hour cutoff — so on 2026-08-09 it answered `count: 0` while two sessions sat
|
|
441
|
+
* stranded for 184 and 24 minutes holding the restart lock. Nothing is wrong with
|
|
442
|
+
* the quarantine list; it was simply blind to the state that matters most, the one
|
|
443
|
+
* where the audio is still live and still rescuable.
|
|
444
|
+
*
|
|
445
|
+
* Reuses `getTranscriptionSessionLiveness().staleSessions` rather than re-deriving
|
|
446
|
+
* staleness, so this view and the maintenance drain gate can never disagree.
|
|
447
|
+
*/
|
|
448
|
+
export function getStrandedCaptures(now = Date.now()): StrandedCapture[] {
|
|
449
|
+
const drafts = new Map(listStrandedDrafts(STRANDED_DRAFT_DIR).map(d => [d.sessionId, d]))
|
|
450
|
+
return getTranscriptionSessionLiveness(now).staleSessions.map(stale => {
|
|
451
|
+
const session = sessions.get(stale.sessionId)
|
|
452
|
+
const lastActivityAt = session?.lastActivityAt ?? now - stale.silentForMs
|
|
453
|
+
const draft = drafts.get(stale.sessionId) ?? null
|
|
454
|
+
return {
|
|
455
|
+
sessionId: stale.sessionId,
|
|
456
|
+
idleMinutes: Math.round(stale.silentForMs / 60_000),
|
|
457
|
+
capturedMinutes: Math.round(
|
|
458
|
+
Math.max(0, lastActivityAt - (session?.startTime ?? lastActivityAt)) / 60_000,
|
|
459
|
+
),
|
|
460
|
+
chunks: stale.chunks,
|
|
461
|
+
promotesAt: new Date(lastActivityAt + LOCAL_FIRST_MEETING_IDLE_RETENTION_MS).toISOString(),
|
|
462
|
+
draftPath: draft?.path ?? null,
|
|
463
|
+
draftBytes: draft?.bytes ?? null,
|
|
464
|
+
}
|
|
465
|
+
})
|
|
466
|
+
}
|
|
467
|
+
|
|
400
468
|
export function getTranscriptionSessionLiveness(now = Date.now()): TranscriptionSessionLiveness {
|
|
401
469
|
let live = 0
|
|
402
470
|
const staleSessions: TranscriptionSessionLiveness['staleSessions'] = []
|
|
@@ -768,15 +836,83 @@ if (maintenanceAdmissionsOpen()) {
|
|
|
768
836
|
recoverSessions()
|
|
769
837
|
}
|
|
770
838
|
|
|
839
|
+
/** Sessions with an auto-save in flight, so a later tick cannot start a second. */
|
|
840
|
+
const promotingStranded = new Set<string>()
|
|
841
|
+
|
|
842
|
+
/**
|
|
843
|
+
* Hard backstop for a capture whose save keeps failing.
|
|
844
|
+
*
|
|
845
|
+
* Without it a session that the save route refuses transiently, forever, would
|
|
846
|
+
* live in memory forever. At twice the cutoff it takes the historical disposition
|
|
847
|
+
* — close and quarantine — so the audio is at least preserved on the unsaved-audio
|
|
848
|
+
* clock rather than held hostage by a retry loop.
|
|
849
|
+
*/
|
|
850
|
+
const STRANDED_PROMOTE_GIVE_UP_MS = 2 * LOCAL_FIRST_MEETING_IDLE_RETENTION_MS
|
|
851
|
+
|
|
852
|
+
/**
|
|
853
|
+
* Turn an unattended capture into a meeting, and fall back to the old behavior if
|
|
854
|
+
* that proves impossible.
|
|
855
|
+
*
|
|
856
|
+
* Fire-and-forget on purpose: the sweep tick must not await an HQ finalization
|
|
857
|
+
* that can run for minutes. `promotingStranded` is what keeps the next tick from
|
|
858
|
+
* starting a duplicate.
|
|
859
|
+
*/
|
|
860
|
+
function promoteStrandedCapture(sessionId: string, lastActivityAt: number): void {
|
|
861
|
+
if (promotingStranded.has(sessionId)) return
|
|
862
|
+
promotingStranded.add(sessionId)
|
|
863
|
+
void promoteStrandedSession(sessionId, {
|
|
864
|
+
port: parseInt(process.env.PORT ?? '3141', 10),
|
|
865
|
+
token: process.env.COS_API_TOKEN ?? '',
|
|
866
|
+
})
|
|
867
|
+
.then(result => {
|
|
868
|
+
if (result.ok) {
|
|
869
|
+
console.warn(
|
|
870
|
+
`[stranded] Auto-saved unattended capture ${sessionId} \u2192 ${result.filename ?? 'saved'}`,
|
|
871
|
+
)
|
|
872
|
+
clearStrandedDraft(STRANDED_DRAFT_DIR, sessionId)
|
|
873
|
+
return
|
|
874
|
+
}
|
|
875
|
+
// Delegated, not restated: closing means the capture becomes quarantined
|
|
876
|
+
// audio instead of a meeting, which is the outcome this release exists to
|
|
877
|
+
// prevent, so the rule lives in a module a test can reach.
|
|
878
|
+
const idleForMs = Date.now() - lastActivityAt
|
|
879
|
+
if (!shouldCloseAfterFailedPromote(result, idleForMs, STRANDED_PROMOTE_GIVE_UP_MS)) {
|
|
880
|
+
console.warn(
|
|
881
|
+
`[stranded] Auto-save deferred for ${sessionId} `
|
|
882
|
+
+ `(${result.status} ${result.reason ?? ''}); retrying next sweep`,
|
|
883
|
+
)
|
|
884
|
+
return
|
|
885
|
+
}
|
|
886
|
+
console.error(
|
|
887
|
+
`[stranded] Auto-save failed for ${sessionId} (${result.status} ${result.reason ?? ''}); `
|
|
888
|
+
+ 'closing as expired so the audio reaches quarantine',
|
|
889
|
+
)
|
|
890
|
+
if (sessions.has(sessionId)) closeTranscriptSession(sessionId, 'expired')
|
|
891
|
+
})
|
|
892
|
+
.finally(() => promotingStranded.delete(sessionId))
|
|
893
|
+
}
|
|
894
|
+
|
|
771
895
|
// Auto-cleanup sessions idle for the advertised retention horizon.
|
|
772
896
|
setInterval(() => {
|
|
773
897
|
if (!maintenanceAdmissionsOpen()) return
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
|
|
779
|
-
|
|
898
|
+
// A capture whose phone went away must still become a meeting. This loop used
|
|
899
|
+
// to close every idle session as 'expired', which quarantined its audio and
|
|
900
|
+
// produced NO meeting — recoverable for 72h, but only by whoever thought to
|
|
901
|
+
// look. See lib/stranded-sessions.ts for why staleness may not close a session
|
|
902
|
+
// early and why the cutoff is unchanged.
|
|
903
|
+
sweepStrandedSessions({
|
|
904
|
+
now: Date.now(),
|
|
905
|
+
token: process.env.COS_API_TOKEN ?? '',
|
|
906
|
+
draftDir: STRANDED_DRAFT_DIR,
|
|
907
|
+
sessions: [...sessions].map(([id, session]) => [id, {
|
|
908
|
+
lastActivityAt: session.lastActivityAt,
|
|
909
|
+
startTime: session.startTime,
|
|
910
|
+
chunkCount: session.chunks.filter(Boolean).length,
|
|
911
|
+
}] as const),
|
|
912
|
+
getHeartbeat: getSessionHeartbeat,
|
|
913
|
+
getTranscript: id => getSessionTranscript(id),
|
|
914
|
+
onPromote: promoteStrandedCapture,
|
|
915
|
+
})
|
|
780
916
|
// Orphaned session-audio dirs (no matching active session): quarantine any
|
|
781
917
|
// dir still holding chunk audio; delete only chunk-less dirs. Quarantine
|
|
782
918
|
// itself expires on the unsaved-audio retention clock, the ONLY place
|
|
@@ -1307,6 +1443,10 @@ function finishClosingTranscriptSession(
|
|
|
1307
1443
|
clearSessionHallucinationState(sessionId)
|
|
1308
1444
|
// Track as deleted so orphan heartbeats get 410 Gone (prevents zombie client spam)
|
|
1309
1445
|
rememberDeletedSession(sessionId)
|
|
1446
|
+
// A session that reached ANY terminal state keeps neither a liveness veto nor a
|
|
1447
|
+
// draft. Leaving the draft would advertise an unsaved capture that is now saved,
|
|
1448
|
+
// which is the false-alarm that trains the user to ignore the channel.
|
|
1449
|
+
releaseStrandedState(STRANDED_DRAFT_DIR, sessionId)
|
|
1310
1450
|
persistClosedSessions()
|
|
1311
1451
|
// Clean up persisted file
|
|
1312
1452
|
try { unlinkSync(resolve(CHUNK_PERSIST_DIR, `${sessionId}.json`)) } catch {}
|