@gotcos/glasses-server 6.61.1 → 6.61.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +39 -0
- package/README.md +15 -6
- package/package.json +2 -1
- package/server/index.ts +1 -1
- package/server/lib/client-liveness.ts +5 -4
- package/server/lib/cos-operations-meetings.ts +208 -25
- package/server/lib/held-naming-batches.ts +3 -0
- package/server/lib/message-era-reset.ts +14 -5
- package/server/lib/permission-broker.ts +8 -6
- package/server/lib/query-job-coordinator.ts +195 -19
- package/server/lib/query-job-runtime.ts +3 -0
- package/server/lib/query-job-store.ts +96 -15
- package/server/lib/query-job-types.ts +4 -0
- package/server/lib/task-dispatcher.ts +4 -5
- package/server/routes/health.ts +2 -0
- package/server/routes/maintenance.ts +5 -0
- package/server/routes/meeting.ts +17 -2
- package/server/routes/query-jobs.ts +2 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,42 @@
|
|
|
1
|
+
## 6.61.3
|
|
2
|
+
|
|
3
|
+
### A prompt the Mac accepted runs to an answer, phone or no phone: an orphan fence holds the start, it no longer refuses the prompt
|
|
4
|
+
|
|
5
|
+
- Miles, 2026-10-02 17:38, driving to pick up the kids: "our COS glasses bridge is completely down because whenever I try to send a prompt it says failed provider closing retry in 1,087 seconds ... we're supposed to be able to run these whole thing offline." Four prompts between 5:33 and 5:37 PM (Cursor Grok and Sonnet, one COS session) all failed `Provider closing · retry in 1192s / 1087s / 1054s / 930s`. Every count ends at about 5:52:40 PM, so ONE fence, set at about 5:31:40 PM, refused all four. A fence is set only when a job whose provider child had confirmed ownership is interrupted, which happens only at server shutdown or boot. So the server restarted at about 5:31 PM with a run in flight. The restart is not the bug: the bridges spawn provider children `detached`, so a killed server can leave one still writing, and the 21-minute session fence exists for exactly that.
|
|
6
|
+
- **The bug was where the fence was enforced.** `QueryJobStore.admit()` threw `provider_orphan_fence` (409) for every new prompt in the fenced session, including a fresh client id. The phone auto-retries only a fence of 25 s or less (the installed 6.9.570 bundle: `25e3`), so each prompt became a queue card marked Failed with a manual RETRY, and nothing reached the journal. A phone that then lost signal never sent it again. That breaks the contract: a prompt that reached the Mac completes and is waiting when the phone reconnects.
|
|
7
|
+
- **Admission no longer refuses on a fence.** The prompt is journaled `accepted` and the phone gets its 202; from then on the phone treats it as Mac-owned (it never marks an accepted job Failed for a transport problem) and may disconnect.
|
|
8
|
+
- **The fence moved to the durable-job start gate.** `markStarting` throws `provider_orphan_fence` while the job's session is fenced and leaves the job `accepted`. No durable job reaches a provider while a provider child of its session may still be running. (The legacy `/api/query` and `/v1` routes are not durable jobs and never checked the fence; unchanged.) `markStarting` also waits for the whole boot classification, so a start can never slip in before the boot has stamped the fence of the run it interrupts.
|
|
9
|
+
- **The coordinator holds the job and starts it when the fence clears, in admission order.** Each job requests its session's run lock before anything awaits, so the jobs of one session start in the order they were admitted (before, held jobs woke on their own timers and started in random order). A job waiting for the lock is still `accepted`. While held it posts two status lines, `An earlier run may still be running` then `Starts 5:53 PM`: the glasses header shows the latest line cut to 18 characters, so the time line fits whole (`Starts 12:53 PM` is 15), and the phone's job log shows both. The clock time is rounded up to the minute, never a countdown, and the lines are not repeated after a restart. With tool activity Off the phone shows `Processing...`, as for any step. The hold re-checks at least every 30 s, so a Mac that slept through the end of a fence does not wait out a stale timer. Cancel ends a hold at once.
|
|
10
|
+
- **A hold never blocks Update Server.** A waiting job holds no maintenance lease; it takes one to start. If a drain refuses it, the job stays held, shows `Waiting for update`, and retries; so a prompt admitted just before Update Server's drain now runs right after the restart instead of delaying it. COS Control's `/api/maintenance/status` now reports `heldJobs` and `requeuedOnBoot`; the public `/api/health` keeps job counts internal, as before.
|
|
11
|
+
- **A restart while a job waits loses nothing.** At boot, a job a prior boot left `accepted` (no prompt byte reached any provider, so running it cannot repeat anything) stays `accepted` and the coordinator starts it, holding again if the fence has time left. This applies to every durable origin (phone, tasks, morning brief, work reviews). Bounded at 60 minutes (`QUERY_JOB_BOOT_REQUEUE_MAX_AGE_MS`), inside the 4 h staged-upload TTL so a photo prompt still finds its upload; an older one is interrupted as before. Only when durable jobs are on: with `COS_DURABLE_QUERY_JOBS=0` no coordinator runs, so it is interrupted as before. `starting` and `running` jobs keep the old boot classification.
|
|
12
|
+
- **Shutdown never starts a provider.** A start still waiting its turn when shutdown begins is refused inside its own write and stays `accepted` for the next boot. One that won its write just before shutdown does not spawn; it stays `starting` and the next boot interrupts it (no provider owned the session, so no fence).
|
|
13
|
+
- **The message-era reset refuses while a prompt is waiting to start**: a held job carries an old-era message number for up to 21 minutes.
|
|
14
|
+
- **Logged** (`[query-jobs]`): a hold beginning (job id prefix and fence end), a drain refusing a start, a start after a hold (with its length), a hold ending without a start, a failed start, and the boot requeue count (also on the boot line).
|
|
15
|
+
- The task dispatcher's `fence` handling is removed: submit can no longer throw it.
|
|
16
|
+
- **Not changed, deliberately.** The run the restart interrupted is not re-run: its provider may have run tool calls before it died, and re-running would repeat them. It keeps its one-tap retry.
|
|
17
|
+
- **Still true, and not fixed here.** The wait is still up to 21 minutes: the fence is a timer, and ending it early when the orphaned child is provably gone is a separate change. On the phone (read from the 6.9.556 source; the 6.9.570 source is not available): the phone sends one prompt at a time, so prompts asked during a hold wait on the phone behind it, and a queue restored after the app view dies comes back paused; the reattach screen says `Reply saved` while a held job has not started. An Update Server drain still refuses new admissions (503), which the phone shows as Failed. These need a client build or a later server change.
|
|
18
|
+
- **Compatibility.** Journal schema unchanged (the hold lines are ordinary `tool_status` rows on an `accepted` job). A rollback to an older server interrupts a held job at its boot (tested against the real 6.51.0 store). The installed 6.9.570 build works with this server: it streams the 202 job and shows the lines; its 25 s fence auto-retry and `Provider closing` copy remain for older servers.
|
|
19
|
+
- **QA.** One /qa round, four Opus reviewers (user, skeptic, completionist, claims). Fixed from it: the health fields were documented on a route that never served them; a start racing shutdown could spawn a provider (reproduced); held jobs of one session started in random order (reproduced); the kill switch stranded re-queued jobs (reproduced); the glasses header cut the hold line to `Waiting for the i…`; a wrong rationale about the `·` glyph; no logging; a cancel test that could not fail; the era reset ignored held jobs; the requeue bound outlived the photo TTL.
|
|
20
|
+
- Tests: a fenced prompt admitted, announced (both lines) and answered after the fence with no subscriber; cancel ending a hold, then nothing starting after the fence; a 21-minute hold ended at once by cancel, and a drain hold ended at once by shutdown; four held prompts of one session starting in admission order through the real `acquireModelSessionRunLock`; a job waiting on its session lock staying `accepted`, holding no lease, and answered on the next boot; a start racing shutdown never spawning, and one still waiting its turn staying `accepted`; a restart mid-hold not repeating the lines; a drain outlasting the fence; the hold wording for every hour (18 characters or fewer, rounded up); the store's fence at `markStarting`, `canStart`, the requeue order, the kill-switch boot, the 60-minute bound at exactly the bound and 1 ms past it, and the bound below `STAGED_TTL_MS`; the production store following `COS_DURABLE_QUERY_JOBS`; the real 6.51.0 store interrupting a held prompt; `/api/maintenance/status` reporting `heldJobs` and `requeuedOnBoot`; the era reset refusing while a prompt waits. The 10 affected suites pass 133 of 133, three runs in a row. Typecheck clean.
|
|
21
|
+
- Mutation gate: 30 cases for this release, one per guard (the 15 from the first pass retargeted, 15 new from QA), on a green baseline of the 6 suites they name: 30 of 30 killed. The first pass's run caught a test gap (a drain that ended inside the fence), and QA caught three guards no test could reach (cancel and shutdown waking a long hold, `canStart` inside the write, the lock waiter's released lease); each now has a test and a case. Not gated: the U+202F replacement in `holdClock` (this Node's ICU prints a plain space, so no test can produce the input).
|
|
22
|
+
- Full suite (Linux container, not the Mac): 5,670 passed, 76 skipped, 39 failed in 15 files. The failing set is identical, file by file, to the same run on unmodified 6.61.2 (`93cc478`): 38 container-bound failures (no `codex` binary, Linux `ps`, no ffmpeg, macOS paths) and 1 unreadable-folder case the container's root user can still read. The Mac run (`publish_server.py --verify`) is still owed before publish.
|
|
23
|
+
- No Install hooks and no COS Control change; Control can read the two new maintenance-status fields when it wants them.
|
|
24
|
+
|
|
25
|
+
## 6.61.2
|
|
26
|
+
|
|
27
|
+
### Renaming a voice works on merged meetings whose recording was filed in a different folder
|
|
28
|
+
|
|
29
|
+
- **What was broken.** When COS merges a glasses recording into a call's write-up, the two are often filed under different areas: the recording under personal, the call under quilt. The server looked for the merged write-up only beside the recording, so it missed it. On 2026-10-02 that was 65 of the 199 merged meetings on Miles's Mac, among them Chris <> Miles Sync on 9/10 (8 voices). Opening Speakers on one showed the raw recording name, and renaming or removing a voice there either failed with "Meeting source changed; reopen the meeting" (when the request carried the record the meetings list shows) or renamed the voice in the recording only and left the meeting's attendees and transcript saying the old name (what COS Control 0.5.254 does; it reported "meeting markdown unreadable"). Naming held voices on these meetings followed the recording's own pointer instead: for 63 it left the meeting's transcript as it was and only marked it as having newer labels, and for 2 (the 9/10 meeting among them) the pointer named a file that had moved, so it failed.
|
|
30
|
+
- **What works now.** The server looks in every area's folder for the recording's month, and also the next or previous month when the recording is dated the first or last day of a month (none of the 199 needed that; a call that starts just before midnight can). It picks the meeting that says it holds the recording with `<!-- g2-session: <id> -->`. Speakers shows that meeting's title and area, renaming, removing or confirming a voice updates that meeting, naming held voices edits it the way it already did for merges filed beside their recording, and the record matches the meetings list row. Merged meetings beside their recording work as before.
|
|
31
|
+
- **Only an exact marker counts.** A title, a file name or an id that differs by one character never matches. iCloud conflict copies (`Name 2.md`) and duplicate folders (`2026-09 2`) are never read as meetings.
|
|
32
|
+
- **Two meetings claiming one recording: neither is chosen.** The recording keeps its own record, the server logs `merged_scribe_ambiguous` with both paths, a rename says so in `markdownSkipped`, and naming held voices refuses until one is fixed. None of the 199 is in that state.
|
|
33
|
+
- **Speed.** The meetings list does not use this lookup and is unchanged. On a copy of Miles's 2,640 meetings, median of 9 runs, two runs each: the default list 313 and 317 ms on 6.61.1, 312 and 315 ms on 6.61.2; one month 84 and 83 ms, then 84 and 84 ms. Opening Speakers on a merged meeting reads that month's write-ups across every area once after a start or a change (49 to 51 ms the first time, against 35 ms on 6.61.1) and afterwards only checks their timestamps: 28 to 29 ms, the same as 6.61.1.
|
|
34
|
+
- **Checked on copies of the real meetings.** All 65 cross-folder meetings resolve to the meeting the list shows (0 of 65 on 6.61.1), all 134 same-folder ones still do, and the 59 recordings the marker pass left for review resolve exactly as on 6.61.1 (30 to their own write-up, 29 to the recording alone, none to a merged meeting). A rename on a copy of the 9/10 meeting: 6.61.1 refused it with the list's record and changed only the recording with the Speakers record; 6.61.2 applied it to the meeting.
|
|
35
|
+
- **For clients.** No response field was added or removed. For a cross-folder merged meeting, `/api/meeting/:id/speakers` and `/content` now carry the merged meeting's `recordId`, `title`, `filename` and `domain`, the values its list row already had; every other meeting answers as before. COS Glasses 6.9.563 to 6.9.570 and COS Control 0.5.254 need no update.
|
|
36
|
+
- **Comments only.** The code comments and README now say glasses presence is a ring, temple or lens gesture, or a phone touch while COS is in front, and never the worn status (COS Glasses 6.9.570). No behaviour change; `presenceAgeMs` is unchanged.
|
|
37
|
+
- Tests: a route suite over HTTP in the pipeline layout (the list row, the Speakers record, title and area, a rename sent with either record, a removal, a confirmation, the copy text, a same-folder merge, and two claims refused), the lookup itself (every area, an area name this repo never uses, the capture's own write-up still winning, no claim, near-miss ids, a title that is not a claim, two claims in one folder or two, iCloud copies of a write-up, of a recording and of a month folder, the first and last day of a month and a year boundary, a mid-month recording not searching further, the cache seeing a new, edited, emptied or deleted write-up and a deleted month folder, a warm lookup reading nothing, a write-up changed between the scan and the read, an unreadable folder or file in another area, a FIFO never opened, suggestion sides and merged-record sources), and held naming (a cross-folder merge, two claims refused). On 6.61.1 the route suite fails 7 of 9 and the two held naming tests fail.
|
|
38
|
+
- `node server/scripts/merged-scribe-mutation-gate.mjs` (repository only, not in the package) runs from a green baseline (85 tests in 3 files), matches each target exactly once, and each mutant must fail the test named for it: 41 of 41 killed. The 6.61.0 and 6.61.1 gate (`nemotron-mutation-gate.mjs`, including the finalizing refusal and the re-read before the final pass writes) rerun on this build: baseline 169 tests in 11 files, 104 of 104 killed. Full suite 5,761 passed, 2 skipped (378 files); typecheck clean; `npm audit --omit=dev` 0 vulnerabilities.
|
|
39
|
+
|
|
1
40
|
## 6.61.1
|
|
2
41
|
|
|
3
42
|
### Fixes 6.61.0 labelling other people as the wearer: a split track is named only by a confident voiceprint, never by the glasses' default label
|
package/README.md
CHANGED
|
@@ -112,7 +112,8 @@ range is the exact Tailscale/CGNAT allocation (`100.64.0.0/10`), not all of
|
|
|
112
112
|
|
|
113
113
|
- Ask anything, get a streamed answer on the lens (`/api/query`, `/v1/chat/completions`)
|
|
114
114
|
- With COS Glasses build 204+, server-owned durable queries are on by default:
|
|
115
|
-
accepted work survives phone backgrounding,
|
|
115
|
+
accepted work survives phone backgrounding, the phone going offline after the Mac
|
|
116
|
+
accepted it, a server restart before it started (since 6.61.3),
|
|
116
117
|
WebView reloads, and network handoffs, then reattaches without duplicate work
|
|
117
118
|
or duplicate replies
|
|
118
119
|
- Choose Opus, Fable, Sonnet, GPT Frontier, GPT Balanced, Composer 2.5 Fast, or
|
|
@@ -265,21 +266,22 @@ or the away hold's.
|
|
|
265
266
|
|
|
266
267
|
**The away hold (6.60.0).** A request that starts while you are away (the desk idle 90 s,
|
|
267
268
|
the same threshold as above) is held for up to 10 minutes, but only when the glasses or the
|
|
268
|
-
phone that show held questions report that you
|
|
269
|
+
phone that show held questions report that you touched them in the last 30 minutes.
|
|
269
270
|
The client contract (COS Glasses 6.9.563 and later):
|
|
270
271
|
|
|
271
272
|
- Poll `GET /api/session-questions?client=glasses|phone&hold=1&presenceAgeMs=<int>`.
|
|
272
273
|
`hold=1` says this client shows held questions. `presenceAgeMs` is the milliseconds since
|
|
273
|
-
the latest real wearer evidence (glasses: a ring or
|
|
274
|
-
|
|
275
|
-
|
|
274
|
+
the latest real wearer evidence (glasses: a ring, temple or lens gesture; phone: a touch
|
|
275
|
+
while COS is in front), a whole number from 0 to 86,400,000, and is left out when there is
|
|
276
|
+
none since the app started. The worn status never counts (COS Glasses 6.9.570 and later):
|
|
277
|
+
wear detection can be turned off, and then it always reads not worn.
|
|
276
278
|
- Only a poll with `hold=1` AND a valid `presenceAgeMs` counts toward the hold. The server
|
|
277
279
|
keeps the newest such presence in memory, on the monotonic clock and the wall clock, and
|
|
278
280
|
reads its age as the larger of the two (so a Mac that slept never makes it look newer).
|
|
279
281
|
Recent means within `COS_PERMISSION_BROKER_AWAY_RECENT_CLIENT_S` (default 1800, clamped to 0
|
|
280
282
|
through 86400; under 30 it means the 30 s live window). After a server restart nothing is
|
|
281
283
|
recent until such a poll.
|
|
282
|
-
- A `hold=1` poll without presence (a connected lens nobody
|
|
284
|
+
- A `hold=1` poll without presence (a connected lens nobody touches) makes no client live and
|
|
283
285
|
holds nothing. A poll without `hold=1` (COS Glasses 6.9.562 and earlier, every older phone
|
|
284
286
|
build) keeps the 6.59.0 meaning exactly: live for 30 s, a 110 s hold, and never the away
|
|
285
287
|
hold.
|
|
@@ -856,6 +858,13 @@ BIND_HOST=0.0.0.0 npm run start:server
|
|
|
856
858
|
Restart once, then confirm `/api/health` reports
|
|
857
859
|
`features.durableQueryJobs: true`, protocol `1`, and state `ready`. To roll
|
|
858
860
|
back, set `COS_DURABLE_QUERY_JOBS=0`; accepted jobs still drain while new prompts use legacy streaming.
|
|
861
|
+
- *"An earlier run may still be running", then "Starts 5:53 PM"?* — a restart stopped a run in
|
|
862
|
+
that conversation, and its provider may still be exiting, so the server holds new prompts
|
|
863
|
+
for that conversation for up to 21 minutes (since 6.61.3; before, they failed as `Provider
|
|
864
|
+
closing · retry in Ns`). The prompt is saved on the Mac and runs at that time even if the
|
|
865
|
+
phone disconnects; COS Control's maintenance status counts it (`heldJobs`). The phone sends
|
|
866
|
+
one prompt at a time, so anything you ask meanwhile waits on the phone behind it. The
|
|
867
|
+
interrupted run itself is not re-run automatically; retry it if you want it.
|
|
859
868
|
- *Session questions never reach the glasses?* The server holds one only when every
|
|
860
869
|
gate passes. `/api/health` `permissionBroker.lastFastPath` names the last reason a
|
|
861
870
|
request went straight to the Mac's dialog: `no_client` (no client could answer: no poll
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@gotcos/glasses-server",
|
|
3
|
-
"version": "6.61.
|
|
3
|
+
"version": "6.61.3",
|
|
4
4
|
"description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -47,6 +47,7 @@
|
|
|
47
47
|
"!server/scripts/work-progress-mutation-gate.mjs",
|
|
48
48
|
"!server/scripts/away-hold-mutation-gate.mjs",
|
|
49
49
|
"!server/scripts/nemotron-mutation-gate.mjs",
|
|
50
|
+
"!server/scripts/merged-scribe-mutation-gate.mjs",
|
|
50
51
|
"!server/scripts/nemotron-replay.ts",
|
|
51
52
|
"!server/scripts/mutation-gate.mjs",
|
|
52
53
|
"!server/lib/meeting-engine/__fixtures__/**",
|
package/server/index.ts
CHANGED
|
@@ -1229,7 +1229,7 @@ listenRequiredServers(listeners).then(() => {
|
|
|
1229
1229
|
|
|
1230
1230
|
void initQueryJobRuntime().then(health => {
|
|
1231
1231
|
if (process.env.COS_DURABLE_QUERY_JOBS !== '0') {
|
|
1232
|
-
console.log(`[COS API] Durable query jobs: ${health.store.state} · ${health.store.retainedIdentities} retained`)
|
|
1232
|
+
console.log(`[COS API] Durable query jobs: ${health.store.state} · ${health.store.retainedIdentities} retained · ${health.store.requeuedOnBoot ?? 0} requeued from the last boot`)
|
|
1233
1233
|
} else {
|
|
1234
1234
|
console.log('[COS API] Durable query jobs: disabled by COS_DURABLE_QUERY_JOBS=0')
|
|
1235
1235
|
}
|
|
@@ -27,10 +27,11 @@
|
|
|
27
27
|
// 6.60.0 (QA round 1, the contract shared with COS Glasses 6.9.563): the away hold needs
|
|
28
28
|
// more than a poll. A poll QUALIFIES only with `hold=1` (this client shows held questions;
|
|
29
29
|
// 6.9.563 and later send it) AND a finite `presenceAgeMs` within [0, 24 h]: the ms since
|
|
30
|
-
// the latest real wearer evidence (glasses: a ring or
|
|
31
|
-
//
|
|
32
|
-
//
|
|
33
|
-
//
|
|
30
|
+
// the latest real wearer evidence (glasses: a ring, temple or lens gesture; phone: a touch
|
|
31
|
+
// while COS is in front). The worn status never counts: Glasses 6.9.570 dropped it because
|
|
32
|
+
// wear detection can be turned off, and then it always reads not worn. A connected lens
|
|
33
|
+
// nobody touches sends no `presenceAgeMs`, and an older client sends no `hold=1`, so neither
|
|
34
|
+
// can make the Mac hold a question for 10 minutes.
|
|
34
35
|
//
|
|
35
36
|
// Kept apart, all in memory (a restart forgets them, which fails toward 6.59.0):
|
|
36
37
|
// - the newest counting poll of any kind (`lastQuestionsPollAt`, for /api/health);
|
|
@@ -478,6 +478,12 @@ function withMeetingListInsights(meta: CosOperationsMeetingMeta, content: string
|
|
|
478
478
|
* but the readable meeting is now the merged scribe, so this returns the orphan's
|
|
479
479
|
* sidecar path with the merged scribe's path, filename and title. Without that,
|
|
480
480
|
* naming a voice on a merged meeting opened a file that no longer existed.
|
|
481
|
+
*
|
|
482
|
+
* ...AND ITS DOMAIN AND MONTH (6.61.2). The merged scribe may sit in another
|
|
483
|
+
* domain folder than the sidecar, and `domain`, `month` and `filename` are what
|
|
484
|
+
* every caller builds the `ops:` recordId from, so all three describe the merged
|
|
485
|
+
* scribe: the same recordId the meetings list shows for that row. `sidecarPath`
|
|
486
|
+
* stays the capture's, wherever it is.
|
|
481
487
|
*/
|
|
482
488
|
export function findCosOperationsMeetingBySessionId(sessionId: string): {
|
|
483
489
|
sidecarPath: string
|
|
@@ -486,22 +492,31 @@ export function findCosOperationsMeetingBySessionId(sessionId: string): {
|
|
|
486
492
|
domain: string
|
|
487
493
|
month: string
|
|
488
494
|
title: string
|
|
495
|
+
/**
|
|
496
|
+
* 6.61.2. Set only when MORE THAN ONE meeting declares this session through
|
|
497
|
+
* `<!-- g2-session -->`: their operations-relative paths. The record then
|
|
498
|
+
* describes the capture alone, exactly as an unresolved merge does, because
|
|
499
|
+
* picking one of them would write a correction into a meeting chosen by
|
|
500
|
+
* directory order.
|
|
501
|
+
*/
|
|
502
|
+
mergedScribeConflict?: string[]
|
|
489
503
|
} | null {
|
|
490
504
|
const operationsDir = resolveCosOperationsDir()
|
|
491
505
|
if (!operationsDir) return null
|
|
492
506
|
|
|
493
|
-
|
|
507
|
+
const domains = discoverMeetingDomains(operationsDir)
|
|
508
|
+
for (const domain of domains) {
|
|
494
509
|
const meetingsBase = join(operationsDir, domain, 'meetings')
|
|
495
510
|
let months: string[]
|
|
496
511
|
try {
|
|
497
|
-
months = readdirSync(meetingsBase).filter(d =>
|
|
512
|
+
months = readdirSync(meetingsBase).filter(d => MONTH_PATTERN.test(d)).sort().reverse()
|
|
498
513
|
} catch { continue }
|
|
499
514
|
|
|
500
515
|
for (const month of months) {
|
|
501
516
|
const monthDir = join(meetingsBase, month)
|
|
502
517
|
let sidecars: string[]
|
|
503
518
|
try {
|
|
504
|
-
sidecars = readdirSync(monthDir).filter(f => f.endsWith('.g2-chunks.json') &&
|
|
519
|
+
sidecars = readdirSync(monthDir).filter(f => f.endsWith('.g2-chunks.json') && !ICLOUD_CONFLICT_COPY.test(f)).sort().reverse()
|
|
505
520
|
} catch { continue }
|
|
506
521
|
|
|
507
522
|
for (const sidecarName of sidecars) {
|
|
@@ -510,17 +525,26 @@ export function findCosOperationsMeetingBySessionId(sessionId: string): {
|
|
|
510
525
|
const sidecarPath = join(monthDir, sidecarName)
|
|
511
526
|
let meetingPath = join(monthDir, meetingFilename)
|
|
512
527
|
let resolvedFilename = meetingFilename
|
|
528
|
+
let resolvedDomain = domain
|
|
529
|
+
let resolvedMonth = month
|
|
530
|
+
let mergedScribeConflict: string[] | undefined
|
|
513
531
|
let content: string | null = null
|
|
514
532
|
try {
|
|
515
533
|
content = readFileSync(meetingPath, 'utf-8')
|
|
516
534
|
} catch {
|
|
517
|
-
// The capture's own scribe is gone. Only now is it worth
|
|
518
|
-
//
|
|
519
|
-
|
|
520
|
-
|
|
535
|
+
// The capture's own scribe is gone. Only now is it worth looking for
|
|
536
|
+
// the merged scribe that declares this session, in EVERY domain: the
|
|
537
|
+
// pipeline re-files a merge by its Fireflies call, so a capture filed
|
|
538
|
+
// under personal is routinely merged into a scribe under quilt.
|
|
539
|
+
const merged = resolveMergedScribe(operationsDir, domains, month, sidecarName, sessionId)
|
|
540
|
+
if (merged.status === 'resolved') {
|
|
541
|
+
resolvedDomain = merged.domain
|
|
542
|
+
resolvedMonth = merged.month
|
|
521
543
|
resolvedFilename = merged.filename
|
|
522
|
-
meetingPath =
|
|
544
|
+
meetingPath = merged.path
|
|
523
545
|
content = merged.content
|
|
546
|
+
} else if (merged.status === 'ambiguous') {
|
|
547
|
+
mergedScribeConflict = merged.claimants
|
|
524
548
|
}
|
|
525
549
|
}
|
|
526
550
|
let title = resolvedFilename.replace(/\.md$/, '')
|
|
@@ -530,9 +554,10 @@ export function findCosOperationsMeetingBySessionId(sessionId: string): {
|
|
|
530
554
|
sidecarPath,
|
|
531
555
|
meetingPath,
|
|
532
556
|
filename: resolvedFilename,
|
|
533
|
-
domain,
|
|
534
|
-
month,
|
|
557
|
+
domain: resolvedDomain,
|
|
558
|
+
month: resolvedMonth,
|
|
535
559
|
title,
|
|
560
|
+
...(mergedScribeConflict ? { mergedScribeConflict } : {}),
|
|
536
561
|
}
|
|
537
562
|
}
|
|
538
563
|
}
|
|
@@ -540,30 +565,188 @@ export function findCosOperationsMeetingBySessionId(sessionId: string): {
|
|
|
540
565
|
return null
|
|
541
566
|
}
|
|
542
567
|
|
|
568
|
+
// ── Merged-scribe resolution across domains (6.61.2) ────────────────────────
|
|
569
|
+
//
|
|
570
|
+
// 6.47.0 looked for the merged scribe only in the sidecar's own folder. On
|
|
571
|
+
// 2026-10-02, 65 of the 199 merged scribes on the author's Mac sat in a
|
|
572
|
+
// different DOMAIN folder from their capture (38 personal -> quilt, 11 personal
|
|
573
|
+
// -> hermit_crabs, 9 personal -> sprocket_rocket, 7 between hermit_crabs,
|
|
574
|
+
// sprocket_rocket and quilt), so those sessions resolved to the retired
|
|
575
|
+
// standalone filename: the speaker review named a record the meetings list does
|
|
576
|
+
// not show, and every correction either 409'd on `record_source_mismatch` or
|
|
577
|
+
// renamed the voice in the sidecar alone.
|
|
578
|
+
//
|
|
579
|
+
// IDENTITY IS THE MARKER. A scribe holds a session only if it carries
|
|
580
|
+
// `<!-- g2-session: <exact id> -->`. No filename, title or date is ever a match,
|
|
581
|
+
// and two scribes declaring one session resolve to neither.
|
|
582
|
+
|
|
583
|
+
/**
|
|
584
|
+
* iCloud conflict copies ("Chris_Miles_Sync 2.md", "x 2.g2-chunks.json").
|
|
585
|
+
* Never canonical, and a copy of a merged scribe carries the same markers as the
|
|
586
|
+
* original, so counting it would turn every conflict copy into a duplicate claim.
|
|
587
|
+
*/
|
|
588
|
+
const ICLOUD_CONFLICT_COPY = / \d+(\.[A-Za-z0-9-]+)*\.(md|json)$/
|
|
589
|
+
|
|
590
|
+
function shiftMonth(month: string, by: number): string {
|
|
591
|
+
const [year, monthNumber] = month.split('-').map(Number)
|
|
592
|
+
const index = year * 12 + (monthNumber - 1) + by
|
|
593
|
+
return `${String(Math.floor(index / 12)).padStart(4, '0')}-${String((index % 12) + 1).padStart(2, '0')}`
|
|
594
|
+
}
|
|
595
|
+
|
|
543
596
|
/**
|
|
544
|
-
* The
|
|
597
|
+
* The month folders a merged scribe for this capture can sit in.
|
|
545
598
|
*
|
|
546
|
-
*
|
|
547
|
-
*
|
|
599
|
+
* The capture's own month, always. Measured on 2026-10-02: all 199 merged
|
|
600
|
+
* scribes carry the capture's own date (0 of 199 a day off) and the nearest any
|
|
601
|
+
* capture came to a month edge was 7.5 hours. So a merge can land in another
|
|
602
|
+
* month only when the capture sits on a month's first or last day (a call that
|
|
603
|
+
* began before midnight and a capture that began after it, or the reverse), and
|
|
604
|
+
* those two days also search the neighbouring month. A sidecar whose name does
|
|
605
|
+
* not carry a date inside its own month searches both neighbours, because
|
|
606
|
+
* nothing says which way it leans. Never a `2026-09 2` folder: every name here
|
|
607
|
+
* is built, not listed.
|
|
548
608
|
*/
|
|
549
|
-
function
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
609
|
+
export function mergedScribeMonthsFor(sidecarMonth: string, sidecarFilename: string): string[] {
|
|
610
|
+
if (!MONTH_PATTERN.test(sidecarMonth)) return []
|
|
611
|
+
const date = /^(\d{4}-\d{2})-(\d{2})/.exec(sidecarFilename)
|
|
612
|
+
if (!date || date[1] !== sidecarMonth) {
|
|
613
|
+
return [sidecarMonth, shiftMonth(sidecarMonth, -1), shiftMonth(sidecarMonth, 1)]
|
|
614
|
+
}
|
|
615
|
+
const day = Number(date[2])
|
|
616
|
+
const [year, monthNumber] = sidecarMonth.split('-').map(Number)
|
|
617
|
+
const lastDay = new Date(Date.UTC(year, monthNumber, 0)).getUTCDate()
|
|
618
|
+
const months = [sidecarMonth]
|
|
619
|
+
if (day === 1) months.push(shiftMonth(sidecarMonth, -1))
|
|
620
|
+
if (day === lastDay) months.push(shiftMonth(sidecarMonth, 1))
|
|
621
|
+
return months
|
|
622
|
+
}
|
|
623
|
+
|
|
624
|
+
// The cost guard. A cold lookup reads every scribe in the months above across
|
|
625
|
+
// every domain (about 300 files and 11 MB for a busy month on the author's Mac);
|
|
626
|
+
// a warm one stats them and reads nothing. A month's listing is reused while the
|
|
627
|
+
// folder's own stamp is unchanged, and a scribe's markers while ITS stamp is,
|
|
628
|
+
// so an in-place edit that leaves the folder's mtime alone is still seen.
|
|
629
|
+
interface ScribeClaimEntry { stamp: string; sessions: readonly string[] }
|
|
630
|
+
interface MonthListingEntry { stamp: string; names: string[] }
|
|
631
|
+
const scribeClaimCache = new Map<string, ScribeClaimEntry>()
|
|
632
|
+
const monthListingCache = new Map<string, MonthListingEntry>()
|
|
633
|
+
let scribeClaimReads = 0
|
|
634
|
+
|
|
635
|
+
function statStamp(stat: { ino: number; size: number; mtimeMs: number; ctimeMs: number }): string {
|
|
636
|
+
return `${stat.ino}:${stat.size}:${stat.mtimeMs}:${stat.ctimeMs}`
|
|
637
|
+
}
|
|
638
|
+
|
|
639
|
+
/** Every canonical scribe in one month folder that declares at least one session. */
|
|
640
|
+
function scribeClaimsIn(monthDir: string): Array<{ filename: string; sessions: readonly string[] }> {
|
|
641
|
+
let dirStamp: string
|
|
554
642
|
try {
|
|
555
|
-
|
|
643
|
+
dirStamp = statStamp(statSync(monthDir))
|
|
556
644
|
} catch {
|
|
557
|
-
|
|
645
|
+
for (const name of monthListingCache.get(monthDir)?.names ?? []) scribeClaimCache.delete(join(monthDir, name))
|
|
646
|
+
monthListingCache.delete(monthDir)
|
|
647
|
+
return []
|
|
558
648
|
}
|
|
559
|
-
|
|
560
|
-
|
|
649
|
+
let listing = monthListingCache.get(monthDir)
|
|
650
|
+
if (!listing || listing.stamp !== dirStamp) {
|
|
651
|
+
let names: string[]
|
|
561
652
|
try {
|
|
562
|
-
|
|
653
|
+
names = readdirSync(monthDir)
|
|
654
|
+
.filter(name => name.endsWith('.md') && !ICLOUD_CONFLICT_COPY.test(name))
|
|
655
|
+
.sort()
|
|
656
|
+
.slice(0, MAX_LIST_CANDIDATES)
|
|
657
|
+
} catch {
|
|
658
|
+
return []
|
|
659
|
+
}
|
|
660
|
+
const kept = new Set(names)
|
|
661
|
+
for (const old of listing?.names ?? []) if (!kept.has(old)) scribeClaimCache.delete(join(monthDir, old))
|
|
662
|
+
listing = { stamp: dirStamp, names }
|
|
663
|
+
monthListingCache.set(monthDir, listing)
|
|
664
|
+
}
|
|
665
|
+
const claims: Array<{ filename: string; sessions: readonly string[] }> = []
|
|
666
|
+
for (const filename of listing.names) {
|
|
667
|
+
const path = join(monthDir, filename)
|
|
668
|
+
let stat: ReturnType<typeof statSync>
|
|
669
|
+
try {
|
|
670
|
+
stat = statSync(path)
|
|
563
671
|
} catch { continue }
|
|
564
|
-
|
|
672
|
+
// A FIFO or a folder named *.md is never read: a FIFO would block the server.
|
|
673
|
+
if (!stat.isFile()) continue
|
|
674
|
+
const stamp = statStamp(stat)
|
|
675
|
+
let entry = scribeClaimCache.get(path)
|
|
676
|
+
if (!entry || entry.stamp !== stamp) {
|
|
677
|
+
let content: string
|
|
678
|
+
try {
|
|
679
|
+
content = readFileSync(path, 'utf-8')
|
|
680
|
+
} catch { continue }
|
|
681
|
+
scribeClaimReads++
|
|
682
|
+
entry = { stamp, sessions: mergedScribeSessions(content) }
|
|
683
|
+
scribeClaimCache.set(path, entry)
|
|
684
|
+
}
|
|
685
|
+
if (entry.sessions.length > 0) claims.push({ filename, sessions: entry.sessions })
|
|
565
686
|
}
|
|
566
|
-
return
|
|
687
|
+
return claims
|
|
688
|
+
}
|
|
689
|
+
|
|
690
|
+
/** How much work the marker cache has done. Diagnostics and the cost tests read it. */
|
|
691
|
+
export function mergedScribeCacheStats(): { reads: number; scribes: number; months: number } {
|
|
692
|
+
return { reads: scribeClaimReads, scribes: scribeClaimCache.size, months: monthListingCache.size }
|
|
693
|
+
}
|
|
694
|
+
|
|
695
|
+
/** Drop the marker cache. Tests use it to measure a cold lookup. */
|
|
696
|
+
export function resetMergedScribeCache(): void {
|
|
697
|
+
scribeClaimCache.clear()
|
|
698
|
+
monthListingCache.clear()
|
|
699
|
+
scribeClaimReads = 0
|
|
700
|
+
}
|
|
701
|
+
|
|
702
|
+
export type MergedScribeResolution =
|
|
703
|
+
| { status: 'resolved'; domain: string; month: string; filename: string; path: string; content: string }
|
|
704
|
+
| { status: 'none' }
|
|
705
|
+
| { status: 'ambiguous'; claimants: string[] }
|
|
706
|
+
|
|
707
|
+
/**
|
|
708
|
+
* The ONE scribe, in any domain, that declares `sessionId` through
|
|
709
|
+
* `<!-- g2-session -->`. Two or more is `ambiguous` and resolves to none of them.
|
|
710
|
+
*
|
|
711
|
+
* `read` is the final read of the chosen scribe. Injectable so a test can stand
|
|
712
|
+
* in for a scribe rewritten between the scan and this read.
|
|
713
|
+
*/
|
|
714
|
+
export function resolveMergedScribe(
|
|
715
|
+
operationsDir: string,
|
|
716
|
+
domains: readonly string[],
|
|
717
|
+
sidecarMonth: string,
|
|
718
|
+
sidecarFilename: string,
|
|
719
|
+
sessionId: string,
|
|
720
|
+
read: (path: string) => string = path => readFileSync(path, 'utf-8'),
|
|
721
|
+
): MergedScribeResolution {
|
|
722
|
+
const months = mergedScribeMonthsFor(sidecarMonth, sidecarFilename)
|
|
723
|
+
const found: Array<{ domain: string; month: string; filename: string }> = []
|
|
724
|
+
for (const domain of domains) {
|
|
725
|
+
for (const month of months) {
|
|
726
|
+
for (const claim of scribeClaimsIn(join(operationsDir, domain, 'meetings', month))) {
|
|
727
|
+
if (claim.sessions.includes(sessionId)) found.push({ domain, month, filename: claim.filename })
|
|
728
|
+
}
|
|
729
|
+
}
|
|
730
|
+
}
|
|
731
|
+
if (found.length === 0) return { status: 'none' }
|
|
732
|
+
if (found.length > 1) {
|
|
733
|
+
const claimants = found.map(f => `${f.domain}/meetings/${f.month}/${f.filename}`)
|
|
734
|
+
// Once per lookup, and lookups happen on a person's action (opening the
|
|
735
|
+
// speaker panel, a correction), never in a loop.
|
|
736
|
+
console.warn(`[meetings] merged_scribe_ambiguous: ${claimants.length} meetings declare session ${sessionId} (${claimants.join(', ')}); using the capture alone until one is fixed`)
|
|
737
|
+
return { status: 'ambiguous', claimants }
|
|
738
|
+
}
|
|
739
|
+
const only = found[0]
|
|
740
|
+
const path = join(operationsDir, only.domain, 'meetings', only.month, only.filename)
|
|
741
|
+
let content: string
|
|
742
|
+
try {
|
|
743
|
+
content = read(path)
|
|
744
|
+
} catch {
|
|
745
|
+
return { status: 'none' }
|
|
746
|
+
}
|
|
747
|
+
// The cache is a hint; the file just read is the identity.
|
|
748
|
+
if (!mergedScribeSessions(content).includes(sessionId)) return { status: 'none' }
|
|
749
|
+
return { status: 'resolved', ...only, path, content }
|
|
567
750
|
}
|
|
568
751
|
|
|
569
752
|
export type MeetingLibraryRecord = NonNullable<ReturnType<typeof findCosOperationsMeetingBySessionId>> & {
|
|
@@ -188,6 +188,9 @@ function resolveCopies(sessionId:string):ResolvedCopy[] {
|
|
|
188
188
|
if (operations) {
|
|
189
189
|
const found = findCosOperationsMeetingBySessionId(sessionId)
|
|
190
190
|
if (!found) throw new Error('Operations meeting copy is missing; retry after sync')
|
|
191
|
+
// 6.61.2: two meetings declare this session. Fail closed rather than let the
|
|
192
|
+
// sidecar's own blended_into pointer below pick one of them.
|
|
193
|
+
if (found.mergedScribeConflict) throw new Error(`${found.mergedScribeConflict.length} meetings declare this session; fix the duplicate before naming`)
|
|
191
194
|
paths.push({copy:'operations',sidecarPath:found.sidecarPath,meetingPath:found.meetingPath})
|
|
192
195
|
} else if (process.env.COS_OPERATIONS_DIR?.trim() || process.env.COS_SCRIPTS_DIR?.trim()) {
|
|
193
196
|
// An explicitly configured but unavailable iCloud root is not standalone.
|
|
@@ -44,19 +44,26 @@ export interface MessageEraResetInput {
|
|
|
44
44
|
confirm: boolean
|
|
45
45
|
now?: number
|
|
46
46
|
activeRuns?: number
|
|
47
|
+
/** Admitted jobs not started yet (6.61.3): each holds a live message number. */
|
|
48
|
+
heldRuns?: number
|
|
47
49
|
shuttingDown?: boolean
|
|
48
50
|
}
|
|
49
51
|
|
|
50
|
-
async function resolveJobHealth(input: MessageEraResetInput): Promise<{ activeRuns: number; shuttingDown: boolean }> {
|
|
51
|
-
if (input.activeRuns != null || input.shuttingDown != null) {
|
|
52
|
+
async function resolveJobHealth(input: MessageEraResetInput): Promise<{ activeRuns: number; heldRuns: number; shuttingDown: boolean }> {
|
|
53
|
+
if (input.activeRuns != null || input.heldRuns != null || input.shuttingDown != null) {
|
|
52
54
|
return {
|
|
53
55
|
activeRuns: input.activeRuns ?? 0,
|
|
56
|
+
heldRuns: input.heldRuns ?? 0,
|
|
54
57
|
shuttingDown: input.shuttingDown ?? false,
|
|
55
58
|
}
|
|
56
59
|
}
|
|
57
60
|
const { queryJobCoordinator } = await import('./query-job-runtime.js')
|
|
58
61
|
const health = queryJobCoordinator.getHealth()
|
|
59
|
-
|
|
62
|
+
// A held job can wait 21 minutes for an orphan fence and survives the restart that is
|
|
63
|
+
// part of the reset; it carries an old-era number (6.61.3). Not-yet-started jobs the
|
|
64
|
+
// store holds but no execute has picked up yet count too.
|
|
65
|
+
const notStarted = Math.max(health.heldRuns, health.store.counts.accepted + health.store.counts.starting)
|
|
66
|
+
return { activeRuns: health.activeRuns, heldRuns: notStarted, shuttingDown: health.shuttingDown }
|
|
60
67
|
}
|
|
61
68
|
|
|
62
69
|
export async function resetLiveMessageEra(input: MessageEraResetInput): Promise<MessageEraResetResult> {
|
|
@@ -69,10 +76,12 @@ export async function resetLiveMessageEra(input: MessageEraResetInput): Promise<
|
|
|
69
76
|
}
|
|
70
77
|
|
|
71
78
|
const jobs = await resolveJobHealth(input)
|
|
72
|
-
if (jobs.activeRuns > 0) {
|
|
79
|
+
if (jobs.activeRuns > 0 || jobs.heldRuns > 0) {
|
|
73
80
|
throw new MessageEraResetError(
|
|
74
81
|
'query_in_flight',
|
|
75
|
-
|
|
82
|
+
jobs.activeRuns > 0
|
|
83
|
+
? 'A query is still running. Wait for it to finish, then reset.'
|
|
84
|
+
: 'A query is waiting to start. Wait for it to finish, then reset.',
|
|
76
85
|
409,
|
|
77
86
|
)
|
|
78
87
|
}
|
|
@@ -16,7 +16,7 @@
|
|
|
16
16
|
// exactly as if no hook existed. Every path here that is not an answer ends in `{}`.
|
|
17
17
|
//
|
|
18
18
|
// ---------------------------------------------------------------------------
|
|
19
|
-
// WHAT GETS PARKED (Miles's Q3 and Q4, 2026-09-19): away AND
|
|
19
|
+
// WHAT GETS PARKED (Miles's Q3 and Q4, 2026-09-19): away AND recently present
|
|
20
20
|
// ---------------------------------------------------------------------------
|
|
21
21
|
// The hook has no matcher, so every permission request at an idle desk reaches the route.
|
|
22
22
|
// Everything outside the gate answers `{}` at once (target: under 100 ms). A request is
|
|
@@ -67,10 +67,12 @@
|
|
|
67
67
|
// 2026-09-30 19:54: "Hold only if the glasses or phone were connected in the last 30
|
|
68
68
|
// min. A Claude Remote Control session with no glasses gets its question right away,
|
|
69
69
|
// as before."). The contract shared with COS Glasses 6.9.563 (lib/client-liveness): a
|
|
70
|
-
// poll `?client=glasses|phone&hold=1&presenceAgeMs=<int>`, the ms since a ring or
|
|
71
|
-
//
|
|
72
|
-
//
|
|
73
|
-
//
|
|
70
|
+
// poll `?client=glasses|phone&hold=1&presenceAgeMs=<int>`, the ms since a ring, temple or
|
|
71
|
+
// lens gesture, or a phone touch while COS is in front. The worn status is never presence
|
|
72
|
+
// (Glasses 6.9.570, Miles 2026-10-02: wear detection can be turned off, and then it always
|
|
73
|
+
// reads not worn). Recent means within `COS_PERMISSION_BROKER_AWAY_RECENT_CLIENT_S`
|
|
74
|
+
// (default 1800; below 30 s, the 30 s live window). A connected lens nobody touches sends
|
|
75
|
+
// no presence; an older client sends
|
|
74
76
|
// no `hold=1` and keeps the 6.59.0 liveness, never the away hold. In memory: after a
|
|
75
77
|
// restart nothing is recent until such a poll.
|
|
76
78
|
// Held that way, a request stays answerable while no client is polling, so a lens or phone
|
|
@@ -1057,7 +1059,7 @@ export class PermissionBroker {
|
|
|
1057
1059
|
* - `recent`: within the window, so the away hold may apply;
|
|
1058
1060
|
* - `holdLive`: a `hold=1` client polled within the live window AND its presence is
|
|
1059
1061
|
* recent, so it counts as a live client the way an older client's poll does. A
|
|
1060
|
-
* connected lens nobody
|
|
1062
|
+
* connected lens nobody touches never counts (QA round 1, Skeptic W1).
|
|
1061
1063
|
*/
|
|
1062
1064
|
private presence(now: number): { ageMs: number | null; recent: boolean; holdLive: boolean; windowMs: number } {
|
|
1063
1065
|
let ageMs: number | null = null
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { getOrCreateSession } from './conversation.js'
|
|
2
2
|
import {
|
|
3
|
+
QueryJobProviderOrphanFenceError,
|
|
3
4
|
QueryJobStore,
|
|
4
5
|
QueryJobStoreError,
|
|
5
6
|
type QueryJobAdmissionResult,
|
|
@@ -81,8 +82,38 @@ export interface QueryJobCoordinatorOptions {
|
|
|
81
82
|
/** Acquired synchronously before serialized admission and retained through
|
|
82
83
|
* the provider terminal, so maintenance proof covers queued transitions. */
|
|
83
84
|
acquireMaintenanceWork?: () => MaintenanceWorkLease
|
|
85
|
+
/** Longest single wait while a job is held (orphan fence, or a maintenance drain
|
|
86
|
+
* refusing its start lease) before the coordinator looks again. Bounded so a Mac
|
|
87
|
+
* that slept through the end of a fence does not wait out a stale timer. */
|
|
88
|
+
holdRecheckMs?: number
|
|
84
89
|
}
|
|
85
90
|
|
|
91
|
+
/** Margin past the fence before the start is retried, so the retry lands after it. */
|
|
92
|
+
const FENCE_HOLD_SLACK_MS = 250
|
|
93
|
+
|
|
94
|
+
/** Clock time as a phone and the lens can read it: rounded UP to the minute (never a
|
|
95
|
+
* start time that has already passed), and ICU's narrow no-break space before AM/PM,
|
|
96
|
+
* which some Node builds emit, as a plain space. */
|
|
97
|
+
export function holdClock(ms: number): string {
|
|
98
|
+
return new Date(Math.ceil(ms / 60_000) * 60_000)
|
|
99
|
+
.toLocaleTimeString('en-US', { hour: 'numeric', minute: '2-digit' })
|
|
100
|
+
.replace(/[\u202f\u00a0]/g, ' ')
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
/**
|
|
104
|
+
* The two lines a job held by an orphan fence shows (6.61.3), journaled as tool-status
|
|
105
|
+
* activity: the reason, then the time. The glasses header shows the LATEST line cut to
|
|
106
|
+
* 18 characters (the client's normalizeHeaderActivityLabel), so the line that stays there
|
|
107
|
+
* must fit whole: `Starts 12:53 PM` is 15. The phone's job log shows both. A clock time,
|
|
108
|
+
* never a countdown: the lines are journaled once, and a countdown is wrong when read.
|
|
109
|
+
*/
|
|
110
|
+
export function orphanFenceHoldStatus(startsAtMs: number): [string, string] {
|
|
111
|
+
return ['An earlier run may still be running', `Starts ${holdClock(startsAtMs)}`]
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/** What a job shows while a maintenance drain (Update Server) refuses its start. 18 chars. */
|
|
115
|
+
export const DRAIN_HOLD_STATUS = 'Waiting for update'
|
|
116
|
+
|
|
86
117
|
interface ActiveRun {
|
|
87
118
|
jobId: string
|
|
88
119
|
generation: number
|
|
@@ -107,6 +138,10 @@ interface ActiveRun {
|
|
|
107
138
|
|
|
108
139
|
export interface QueryJobCoordinatorHealth {
|
|
109
140
|
activeRuns: number
|
|
141
|
+
/** Admitted jobs this process has not started yet (6.61.3): waiting for an earlier run
|
|
142
|
+
* in their session, an orphan fence, or a drain refusing their start. Durable and
|
|
143
|
+
* `accepted`, so a restart re-queues them; not provider work. */
|
|
144
|
+
heldRuns: number
|
|
110
145
|
shuttingDown: boolean
|
|
111
146
|
callbackPersistenceFailures: number
|
|
112
147
|
terminalProjectionFailures: number
|
|
@@ -129,6 +164,11 @@ export class QueryJobCoordinator {
|
|
|
129
164
|
private readonly providerTimeoutMs: number
|
|
130
165
|
private readonly active = new Map<string, ActiveRun>()
|
|
131
166
|
private readonly admittedMaintenance = new Map<string, MaintenanceWorkLease>()
|
|
167
|
+
/** Held jobs, each with the function that ends its current wait early. */
|
|
168
|
+
private readonly holds = new Map<string, () => void>()
|
|
169
|
+
/** Admitted jobs inside execute that have not started (or given up) yet. */
|
|
170
|
+
private readonly waiting = new Set<string>()
|
|
171
|
+
private readonly holdRecheckMs: number
|
|
132
172
|
private admissionTail: Promise<void> = Promise.resolve()
|
|
133
173
|
private shuttingDown = false
|
|
134
174
|
private callbackPersistenceFailures = 0
|
|
@@ -143,6 +183,7 @@ export class QueryJobCoordinator {
|
|
|
143
183
|
this.partialFlushMs = Math.max(0, options.partialFlushMs ?? 50)
|
|
144
184
|
this.partialFlushChars = Math.max(1, options.partialFlushChars ?? 1_024)
|
|
145
185
|
this.providerTimeoutMs = Math.max(1_000, options.providerTimeoutMs ?? 21 * 60_000)
|
|
186
|
+
this.holdRecheckMs = Math.max(10, options.holdRecheckMs ?? 30_000)
|
|
146
187
|
}
|
|
147
188
|
|
|
148
189
|
async init(): Promise<QueryJobCoordinatorHealth> {
|
|
@@ -156,6 +197,12 @@ export class QueryJobCoordinator {
|
|
|
156
197
|
}
|
|
157
198
|
}
|
|
158
199
|
}
|
|
200
|
+
// 6.61.3: work the last boot admitted but never started runs now, with no
|
|
201
|
+
// phone needed to send it again, in admission order. Each takes its own start
|
|
202
|
+
// lease in execute.
|
|
203
|
+
for (const { jobId, sessionId } of this.store.takeRequeuedOnBoot()) {
|
|
204
|
+
queueMicrotask(() => { void this.execute(jobId, sessionId) })
|
|
205
|
+
}
|
|
159
206
|
return this.getHealth()
|
|
160
207
|
}
|
|
161
208
|
|
|
@@ -178,7 +225,7 @@ export class QueryJobCoordinator {
|
|
|
178
225
|
resolve(admission)
|
|
179
226
|
if (admission.created) {
|
|
180
227
|
if (maintenanceLease) this.admittedMaintenance.set(admission.job.jobId, maintenanceLease)
|
|
181
|
-
queueMicrotask(() => { void this.execute(admission.job.jobId) })
|
|
228
|
+
queueMicrotask(() => { void this.execute(admission.job.jobId, admission.job.sessionId) })
|
|
182
229
|
} else {
|
|
183
230
|
maintenanceLease?.release()
|
|
184
231
|
}
|
|
@@ -207,33 +254,157 @@ export class QueryJobCoordinator {
|
|
|
207
254
|
return { ...input, sessionId }
|
|
208
255
|
}
|
|
209
256
|
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
257
|
+
/**
|
|
258
|
+
* Mark a job starting, holding it first while its session is orphan-fenced (6.61.3).
|
|
259
|
+
*
|
|
260
|
+
* THE CONTRACT: a prompt the Mac accepted runs to an answer whether or not the
|
|
261
|
+
* phone stays connected. The job is durable before this runs, so the hold needs
|
|
262
|
+
* no subscriber, and a phone that drops reads the answer when it reconnects.
|
|
263
|
+
*
|
|
264
|
+
* The lease is RELEASED for the hold, so a fence (itself left by a restart) cannot
|
|
265
|
+
* block the next Update Server for 21 minutes. Nothing is lost if that restart lands
|
|
266
|
+
* mid-hold: the job is still `accepted`, and the next boot re-queues it. The start
|
|
267
|
+
* takes a fresh lease, which a drain refuses; the job then stays held and retries,
|
|
268
|
+
* rather than starting a provider under a closing server.
|
|
269
|
+
*
|
|
270
|
+
* Returns null when this process will not run the job: it went terminal (a cancel
|
|
271
|
+
* during the hold), the coordinator is shutting down, or the store refused.
|
|
272
|
+
*/
|
|
273
|
+
private async startWhenClear(
|
|
274
|
+
jobId: string,
|
|
275
|
+
admitted: MaintenanceWorkLease | undefined,
|
|
276
|
+
): Promise<{ lease?: MaintenanceWorkLease } | null> {
|
|
277
|
+
const id = jobId.slice(0, 8)
|
|
278
|
+
let lease = admitted
|
|
279
|
+
let heldSinceMs: number | null = null
|
|
280
|
+
let drainNoted = false
|
|
281
|
+
let fenceNoted = false
|
|
282
|
+
for (;;) {
|
|
283
|
+
if (this.shuttingDown) {
|
|
284
|
+
lease?.release()
|
|
285
|
+
if (heldSinceMs != null) console.info(`[query-jobs] ${id} held job left accepted for the next boot (shutdown)`)
|
|
286
|
+
return null
|
|
287
|
+
}
|
|
288
|
+
if (!lease && this.options.acquireMaintenanceWork) {
|
|
289
|
+
try {
|
|
290
|
+
lease = this.options.acquireMaintenanceWork()
|
|
291
|
+
} catch (error) {
|
|
292
|
+
if (!drainNoted) {
|
|
293
|
+
drainNoted = true
|
|
294
|
+
heldSinceMs ??= Date.now()
|
|
295
|
+
console.warn(`[query-jobs] ${id} held: start lease refused (${(error as { code?: unknown })?.code ?? 'error'}); retrying every ${Math.round(this.holdRecheckMs / 1000)}s`)
|
|
296
|
+
await this.appendStatusOnce(jobId, [DRAIN_HOLD_STATUS])
|
|
297
|
+
}
|
|
298
|
+
await this.holdFor(jobId, this.holdRecheckMs)
|
|
299
|
+
continue
|
|
300
|
+
}
|
|
301
|
+
}
|
|
302
|
+
try {
|
|
303
|
+
const starting = await this.store.markStarting(jobId, () => !this.shuttingDown)
|
|
304
|
+
if (starting.applied) {
|
|
305
|
+
if (heldSinceMs != null) console.info(`[query-jobs] ${id} starting after a ${Math.round((Date.now() - heldSinceMs) / 1000)}s hold`)
|
|
306
|
+
return { lease }
|
|
307
|
+
}
|
|
308
|
+
lease?.release()
|
|
309
|
+
if (heldSinceMs != null) console.info(`[query-jobs] ${id} held job ended without starting (${starting.job.status})`)
|
|
310
|
+
return null
|
|
311
|
+
} catch (error) {
|
|
312
|
+
if (!(error instanceof QueryJobProviderOrphanFenceError)) {
|
|
313
|
+
lease?.release()
|
|
314
|
+
console.error(`[query-jobs] ${id} could not start; left accepted for the next boot:`, error)
|
|
315
|
+
return null
|
|
316
|
+
}
|
|
317
|
+
if (!fenceNoted) {
|
|
318
|
+
fenceNoted = true
|
|
319
|
+
heldSinceMs ??= Date.now()
|
|
320
|
+
console.warn(`[query-jobs] ${id} held: a provider of its session may still be running; starts after ${new Date(Date.now() + error.retryAfterMs).toISOString()}`)
|
|
321
|
+
// Journaled while the lease still covers this job.
|
|
322
|
+
await this.appendStatusOnce(jobId, orphanFenceHoldStatus(Date.now() + error.retryAfterMs))
|
|
323
|
+
}
|
|
324
|
+
lease?.release()
|
|
325
|
+
lease = undefined
|
|
326
|
+
await this.holdFor(jobId, Math.min(error.retryAfterMs + FENCE_HOLD_SLACK_MS, this.holdRecheckMs))
|
|
327
|
+
}
|
|
217
328
|
}
|
|
218
|
-
|
|
219
|
-
|
|
329
|
+
}
|
|
330
|
+
|
|
331
|
+
/** Append status lines unless they are already the job's latest ones: a restart in the
|
|
332
|
+
* middle of a hold must not repeat them. A failed append never stops the hold. */
|
|
333
|
+
private async appendStatusOnce(jobId: string, lines: readonly string[]): Promise<void> {
|
|
220
334
|
try {
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
return
|
|
335
|
+
const statuses = (await this.store.getSnapshot(jobId)).activity
|
|
336
|
+
.filter(entry => entry.kind === 'status')
|
|
337
|
+
.map(entry => entry.text)
|
|
338
|
+
if (lines.every((line, index) => statuses[statuses.length - lines.length + index] === line)) return
|
|
339
|
+
for (const line of lines) await this.store.appendActivity(jobId, 'status', line)
|
|
340
|
+
} catch (error) {
|
|
341
|
+
console.warn(`[query-jobs] ${jobId.slice(0, 8)} hold status not journaled:`, error)
|
|
225
342
|
}
|
|
343
|
+
}
|
|
344
|
+
|
|
345
|
+
/** One bounded wait for a held job, ended early by cancel or shutdown. */
|
|
346
|
+
private holdFor(jobId: string, ms: number): Promise<void> {
|
|
347
|
+
return new Promise(resolve => {
|
|
348
|
+
const end = () => {
|
|
349
|
+
clearTimeout(timer)
|
|
350
|
+
if (this.holds.get(jobId) === end) this.holds.delete(jobId)
|
|
351
|
+
resolve()
|
|
352
|
+
}
|
|
353
|
+
const timer = setTimeout(end, Math.max(0, ms))
|
|
354
|
+
timer.unref?.()
|
|
355
|
+
this.holds.set(jobId, end)
|
|
356
|
+
})
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
private async execute(jobId: string, sessionId: string): Promise<void> {
|
|
360
|
+
// The session lock is REQUESTED before anything awaits, so the jobs of one session
|
|
361
|
+
// line up for it in admission order (submit and the boot requeue call this in that
|
|
362
|
+
// order, and the lock is first-come-first-served). A job waiting for it is still
|
|
363
|
+
// `accepted`, so a restart re-queues it instead of interrupting it.
|
|
364
|
+
const lockRequest = this.options.acquireSessionLock?.(sessionId)
|
|
365
|
+
const admitted = this.admittedMaintenance.get(jobId)
|
|
366
|
+
this.admittedMaintenance.delete(jobId)
|
|
367
|
+
this.waiting.add(jobId)
|
|
226
368
|
let release: (() => void) | undefined
|
|
369
|
+
let maintenanceLease: MaintenanceWorkLease | undefined
|
|
370
|
+
let execution
|
|
227
371
|
try {
|
|
228
|
-
|
|
229
|
-
|
|
372
|
+
if (lockRequest) {
|
|
373
|
+
// Released while it waits: a run queued behind a long one, or a held one, must not
|
|
374
|
+
// hold Update Server's drain open. startWhenClear takes a fresh lease to start.
|
|
375
|
+
admitted?.release()
|
|
376
|
+
try {
|
|
377
|
+
release = await lockRequest
|
|
378
|
+
} catch (error) {
|
|
379
|
+
await this.store.fail(jobId, error).catch(() => {})
|
|
380
|
+
return
|
|
381
|
+
}
|
|
382
|
+
}
|
|
230
383
|
try {
|
|
231
|
-
await this.store.
|
|
232
|
-
}
|
|
384
|
+
execution = await this.store.getExecution(jobId)
|
|
385
|
+
} catch {
|
|
386
|
+
if (!lockRequest) admitted?.release()
|
|
387
|
+
release?.()
|
|
388
|
+
return
|
|
389
|
+
}
|
|
390
|
+
const started = await this.startWhenClear(jobId, lockRequest ? undefined : admitted)
|
|
391
|
+
if (!started) {
|
|
392
|
+
release?.()
|
|
393
|
+
return
|
|
394
|
+
}
|
|
395
|
+
maintenanceLease = started.lease
|
|
396
|
+
if (this.shuttingDown) {
|
|
397
|
+
// markStarting won its write just before shutdown began. Never spawn under a
|
|
398
|
+
// closing server: the job stays `starting`, the next boot interrupts it (no
|
|
399
|
+
// provider owned the session, so no fence) and the phone can retry it.
|
|
233
400
|
maintenanceLease?.release()
|
|
401
|
+
release?.()
|
|
402
|
+
return
|
|
234
403
|
}
|
|
235
|
-
|
|
404
|
+
} finally {
|
|
405
|
+
this.waiting.delete(jobId)
|
|
236
406
|
}
|
|
407
|
+
maintenanceLease?.setPhase('active')
|
|
237
408
|
|
|
238
409
|
let resolveTerminal!: () => void
|
|
239
410
|
const terminalPromise = new Promise<void>(resolve => { resolveTerminal = resolve })
|
|
@@ -526,6 +697,8 @@ export class QueryJobCoordinator {
|
|
|
526
697
|
async cancel(jobId: string, generation: number): Promise<QueryJobMutationResult> {
|
|
527
698
|
const result = await this.store.cancel(jobId, generation)
|
|
528
699
|
if (result.applied) {
|
|
700
|
+
// A held job is not active: end its wait so it sees the cancel now.
|
|
701
|
+
this.holds.get(jobId)?.()
|
|
529
702
|
const active = this.active.get(jobId)
|
|
530
703
|
const request = active?.request ?? (await this.store.getExecution(jobId)).request
|
|
531
704
|
await this.projectTerminal(result.job, request)
|
|
@@ -569,6 +742,8 @@ export class QueryJobCoordinator {
|
|
|
569
742
|
|
|
570
743
|
async shutdown(reason = 'server_shutdown'): Promise<void> {
|
|
571
744
|
this.shuttingDown = true
|
|
745
|
+
// Held jobs stay `accepted` in the journal; the next boot re-queues them.
|
|
746
|
+
for (const end of [...this.holds.values()]) end()
|
|
572
747
|
await this.admissionTail.catch(() => {})
|
|
573
748
|
for (const active of [...this.active.values()]) {
|
|
574
749
|
try {
|
|
@@ -634,6 +809,7 @@ export class QueryJobCoordinator {
|
|
|
634
809
|
getHealth(): QueryJobCoordinatorHealth {
|
|
635
810
|
return {
|
|
636
811
|
activeRuns: this.active.size,
|
|
812
|
+
heldRuns: this.waiting.size,
|
|
637
813
|
shuttingDown: this.shuttingDown,
|
|
638
814
|
callbackPersistenceFailures: this.callbackPersistenceFailures,
|
|
639
815
|
terminalProjectionFailures: this.terminalProjectionFailures,
|
|
@@ -362,6 +362,9 @@ const queryJobRoot = configuredRoot ? resolve(configuredRoot) : dataPath('query-
|
|
|
362
362
|
export const queryJobStore = new QueryJobStore({
|
|
363
363
|
root: queryJobRoot,
|
|
364
364
|
bootId: randomUUID(),
|
|
365
|
+
// 6.61.3: only when the coordinator will run them. With COS_DURABLE_QUERY_JOBS=0 no
|
|
366
|
+
// coordinator starts, so prior-boot never-started jobs are interrupted as before.
|
|
367
|
+
requeueNeverStartedOnBoot: durableQueryJobsEnabled(),
|
|
365
368
|
})
|
|
366
369
|
|
|
367
370
|
export const queryJobCoordinator = new QueryJobCoordinator(queryJobStore, runner, {
|
|
@@ -63,6 +63,15 @@ const PARTITION_RE = /^\d{4}-\d{2}-\d{2}\.jsonl$/
|
|
|
63
63
|
const MAX_JOURNAL_RECORD_BYTES = 512 * 1024
|
|
64
64
|
const MAX_CHUNK_DELTA_CHARS = 16_000
|
|
65
65
|
export const QUERY_JOB_ORPHAN_FENCE_MS = 21 * 60_000
|
|
66
|
+
/**
|
|
67
|
+
* How old a never-started job may be and still run after a restart (6.61.3). Covers the
|
|
68
|
+
* 21-minute orphan fence plus an Update Server or a crash in the middle of it, with margin.
|
|
69
|
+
* Kept under media-store's STAGED_TTL_MS (4 h), so a re-queued photo prompt still finds its
|
|
70
|
+
* staged upload (a test pins that). An older `accepted` job is interrupted on boot exactly
|
|
71
|
+
* as before: a question asked that long ago may no longer be the question, and the phone may
|
|
72
|
+
* have recorded a cancel it could not send while the Mac was down.
|
|
73
|
+
*/
|
|
74
|
+
export const QUERY_JOB_BOOT_REQUEUE_MAX_AGE_MS = 60 * 60_000
|
|
66
75
|
|
|
67
76
|
interface QueryJobJournalRecord {
|
|
68
77
|
schemaVersion: typeof QUERY_JOB_SCHEMA_VERSION
|
|
@@ -234,6 +243,14 @@ export interface QueryJobStoreOptions {
|
|
|
234
243
|
* rollback contract on real bytes without shipping a second copy of this class.
|
|
235
244
|
*/
|
|
236
245
|
journalEventTypes?: readonly QueryJobEventType[]
|
|
246
|
+
/**
|
|
247
|
+
* Keep a recent never-started (`accepted`) job from a prior boot `accepted` for the
|
|
248
|
+
* coordinator to run (6.61.3), instead of interrupting it. Off by default: only a store
|
|
249
|
+
* whose coordinator will actually take the list may turn it on. Production sets it from
|
|
250
|
+
* `durableQueryJobsEnabled()`, because with COS_DURABLE_QUERY_JOBS=0 no coordinator runs
|
|
251
|
+
* and a kept job would sit `accepted` for the life of the process.
|
|
252
|
+
*/
|
|
253
|
+
requeueNeverStartedOnBoot?: boolean
|
|
237
254
|
}
|
|
238
255
|
|
|
239
256
|
export class QueryJobStoreError extends Error {
|
|
@@ -373,6 +390,8 @@ export class QueryJobStore {
|
|
|
373
390
|
private readonly trailEmitter = new EventEmitter()
|
|
374
391
|
private appendTail: Promise<void> = Promise.resolve()
|
|
375
392
|
private initPromise: Promise<QueryJobStoreHealth> | null = null
|
|
393
|
+
/** Never-started jobs the last boot left `accepted`, in journal (admission) order. */
|
|
394
|
+
private requeuedJobs: Array<{ jobId: string; sessionId: string }> = []
|
|
376
395
|
private partitions: string[] = []
|
|
377
396
|
private subscriberCount = 0
|
|
378
397
|
private readonly health: QueryJobStoreHealth
|
|
@@ -398,6 +417,7 @@ export class QueryJobStore {
|
|
|
398
417
|
malformedRows: 0,
|
|
399
418
|
journalFailures: 0,
|
|
400
419
|
interruptedOnBoot: 0,
|
|
420
|
+
requeuedOnBoot: 0,
|
|
401
421
|
evictedHydratedJobs: 0,
|
|
402
422
|
originDropped: 0,
|
|
403
423
|
originStripped: 0,
|
|
@@ -418,6 +438,27 @@ export class QueryJobStore {
|
|
|
418
438
|
return this.initPromise
|
|
419
439
|
}
|
|
420
440
|
|
|
441
|
+
/** The never-started jobs init kept `accepted`, handed over exactly once, in admission order. */
|
|
442
|
+
takeRequeuedOnBoot(): Array<{ jobId: string; sessionId: string }> {
|
|
443
|
+
return this.requeuedJobs.splice(0)
|
|
444
|
+
}
|
|
445
|
+
|
|
446
|
+
/**
|
|
447
|
+
* Epoch ms until which a provider child of `sessionId` may still be running, or 0.
|
|
448
|
+
* Read from the in-memory identities, so it never hydrates a journal partition.
|
|
449
|
+
* Provider sessions, not phone queue ids, are the concurrency boundary: a fresh
|
|
450
|
+
* clientJobId in the same session is fenced exactly like a retry.
|
|
451
|
+
*/
|
|
452
|
+
sessionFenceUntil(sessionId: string): number {
|
|
453
|
+
let latest = 0
|
|
454
|
+
for (const item of this.identitiesByJobId.values()) {
|
|
455
|
+
if (item.sessionId !== sessionId || item.status !== 'interrupted' || !item.orphanFenceUntil) continue
|
|
456
|
+
const value = new Date(item.orphanFenceUntil).getTime()
|
|
457
|
+
if (Number.isFinite(value)) latest = Math.max(latest, value)
|
|
458
|
+
}
|
|
459
|
+
return latest
|
|
460
|
+
}
|
|
461
|
+
|
|
421
462
|
private async initialize(): Promise<QueryJobStoreHealth> {
|
|
422
463
|
try {
|
|
423
464
|
await this.storage.prepare(this.options.root)
|
|
@@ -430,17 +471,33 @@ export class QueryJobStore {
|
|
|
430
471
|
for (const partition of this.partitions) await this.hydratePartition(partition)
|
|
431
472
|
this.health.state = 'ready'
|
|
432
473
|
|
|
433
|
-
//
|
|
474
|
+
// No provider child can be reattached after a boot (a detached one may still be
|
|
475
|
+
// running, which is what the orphan fence is for). Persist the
|
|
434
476
|
// classification once; never invoke a runner during hydration. An
|
|
435
477
|
// answer_ready record is different: provider generation has already
|
|
436
478
|
// crossed the durable commit point, so finish it from the journaled
|
|
437
479
|
// answer instead of throwing away a reply merely because bridge
|
|
438
480
|
// post-processing was interrupted by the restart.
|
|
481
|
+
//
|
|
482
|
+
// 6.61.3: an `accepted` job never passed markStarting, so no prompt byte reached
|
|
483
|
+
// any provider and running it now cannot repeat anything. With
|
|
484
|
+
// requeueNeverStartedOnBoot it stays `accepted` and the coordinator starts it
|
|
485
|
+
// after init. This is what keeps a held prompt alive across an Update Server in
|
|
486
|
+
// the middle of its wait, with no phone needed to send it again. `starting` and
|
|
487
|
+
// `running` keep the old path: their state machine has no way back to `accepted`.
|
|
439
488
|
const priorBootJobs = [...this.jobs.values()].filter(job =>
|
|
440
489
|
!isTerminalQueryJobStatus(job.snapshot.status) && job.lastBootId !== this.options.bootId)
|
|
490
|
+
const nowMs = this.now().getTime()
|
|
441
491
|
for (const job of priorBootJobs) {
|
|
442
492
|
const snapshot = job.snapshot
|
|
443
|
-
|
|
493
|
+
const acceptedMs = Date.parse(snapshot.acceptedAt)
|
|
494
|
+
if (this.options.requeueNeverStartedOnBoot === true
|
|
495
|
+
&& snapshot.status === 'accepted'
|
|
496
|
+
&& Number.isFinite(acceptedMs)
|
|
497
|
+
&& nowMs - acceptedMs <= QUERY_JOB_BOOT_REQUEUE_MAX_AGE_MS) {
|
|
498
|
+
this.requeuedJobs.push({ jobId: snapshot.jobId, sessionId: job.request.sessionId })
|
|
499
|
+
this.health.requeuedOnBoot = (this.health.requeuedOnBoot ?? 0) + 1
|
|
500
|
+
} else if (snapshot.status === 'answer_ready') {
|
|
444
501
|
await this.complete(snapshot.jobId, {
|
|
445
502
|
text: snapshot.partialText,
|
|
446
503
|
attachments: snapshot.attachments,
|
|
@@ -458,6 +515,9 @@ export class QueryJobStore {
|
|
|
458
515
|
if (result.applied) this.health.interruptedOnBoot++
|
|
459
516
|
}
|
|
460
517
|
}
|
|
518
|
+
if (this.requeuedJobs.length > 0) {
|
|
519
|
+
console.warn(`[query-jobs] boot: ${this.requeuedJobs.length} never-started job(s) from the last boot kept for the coordinator to run`)
|
|
520
|
+
}
|
|
461
521
|
this.trimHydratedJobs()
|
|
462
522
|
this.refreshHealth()
|
|
463
523
|
return this.getHealth()
|
|
@@ -801,17 +861,11 @@ export class QueryJobStore {
|
|
|
801
861
|
if (lineage.some(item => !isTerminalQueryJobStatus(item.status))) {
|
|
802
862
|
throw new QueryJobActiveGenerationError(request.clientJobId)
|
|
803
863
|
}
|
|
804
|
-
|
|
805
|
-
//
|
|
806
|
-
//
|
|
807
|
-
//
|
|
808
|
-
|
|
809
|
-
const fencedUntil = sessionLineage.reduce((latest, item) => {
|
|
810
|
-
if (item.status !== 'interrupted' || !item.orphanFenceUntil) return latest
|
|
811
|
-
const value = new Date(item.orphanFenceUntil).getTime()
|
|
812
|
-
return Number.isFinite(value) ? Math.max(latest, value) : latest
|
|
813
|
-
}, 0)
|
|
814
|
-
if (fencedUntil > nowMs) throw new QueryJobProviderOrphanFenceError(fencedUntil - nowMs)
|
|
864
|
+
// 6.61.3: an orphan fence no longer refuses admission. Refusing handed the
|
|
865
|
+
// prompt back to the phone as "Provider closing · retry in 1192s", where a
|
|
866
|
+
// phone that then lost signal never sent it again. The fence is enforced at
|
|
867
|
+
// markStarting instead, the gate every durable job crosses before a provider,
|
|
868
|
+
// so the prompt is journaled now and starts on the Mac once the fence clears.
|
|
815
869
|
const highestGeneration = lineage.reduce((max, item) => Math.max(max, item.generation), 0)
|
|
816
870
|
if (request.generation <= highestGeneration) throw new QueryJobGenerationOrderError(request.clientJobId)
|
|
817
871
|
|
|
@@ -844,8 +898,35 @@ export class QueryJobStore {
|
|
|
844
898
|
})
|
|
845
899
|
}
|
|
846
900
|
|
|
847
|
-
|
|
848
|
-
|
|
901
|
+
/**
|
|
902
|
+
* The durable-job provider-start gate. Every durable job crosses it before a provider
|
|
903
|
+
* (the legacy /api/query and /v1 routes are not durable jobs and never did). A job whose
|
|
904
|
+
* session may still hold an orphaned provider child cannot start: this throws
|
|
905
|
+
* `provider_orphan_fence` with the time left and the job stays `accepted`, for the
|
|
906
|
+
* coordinator to start once the fence clears (6.61.3; the check used to refuse admission
|
|
907
|
+
* instead). A job that is terminal, or already past `accepted`, is `applied: false`,
|
|
908
|
+
* checked first so a canceled job never reports a fence. So is one `canStart` refuses,
|
|
909
|
+
* which is evaluated INSIDE the serialized write: a shutdown that begins while this call
|
|
910
|
+
* waits its turn leaves the job `accepted` for the next boot, never `starting`.
|
|
911
|
+
* Awaits the WHOLE boot classification, not just readiness: a start must never slip in
|
|
912
|
+
* before the boot has stamped the fence of a run it is about to interrupt.
|
|
913
|
+
*/
|
|
914
|
+
async markStarting(jobId: string, canStart: () => boolean = () => true): Promise<QueryJobMutationResult> {
|
|
915
|
+
await this.init()
|
|
916
|
+
await this.ensureHydrated(jobId)
|
|
917
|
+
return this.enqueue(async () => {
|
|
918
|
+
this.assertWritable()
|
|
919
|
+
const hydrated = this.jobs.get(jobId)
|
|
920
|
+
if (!hydrated) throw new QueryJobNotFoundError(jobId)
|
|
921
|
+
const status = hydrated.snapshot.status
|
|
922
|
+
if (isTerminalQueryJobStatus(status) || !ALLOWED_TRANSITIONS[status].has('starting')) {
|
|
923
|
+
return { applied: false, job: clone(hydrated.snapshot) }
|
|
924
|
+
}
|
|
925
|
+
const remainingMs = this.sessionFenceUntil(hydrated.request.sessionId) - this.now().getTime()
|
|
926
|
+
if (remainingMs > 0) throw new QueryJobProviderOrphanFenceError(remainingMs)
|
|
927
|
+
if (!canStart()) return { applied: false, job: clone(hydrated.snapshot) }
|
|
928
|
+
return this.persistMutation(hydrated, 'starting', 'starting', {}, {})
|
|
929
|
+
})
|
|
849
930
|
}
|
|
850
931
|
|
|
851
932
|
async markRunning(jobId: string, linkage: QueryJobProviderLinkage): Promise<QueryJobMutationResult> {
|
|
@@ -286,6 +286,10 @@ export interface QueryJobStoreHealth {
|
|
|
286
286
|
malformedRows: number
|
|
287
287
|
journalFailures: number
|
|
288
288
|
interruptedOnBoot: number
|
|
289
|
+
/** Never-started jobs a prior boot left `accepted` that this boot runs (6.61.3).
|
|
290
|
+
* Optional only because the vendored 6.51.0 rollback fixture types its health
|
|
291
|
+
* against this interface and must stay byte-identical; this build always sets it. */
|
|
292
|
+
requeuedOnBoot?: number
|
|
289
293
|
evictedHydratedJobs: number
|
|
290
294
|
/** Requests whose `origin` was present but not one this build recognises
|
|
291
295
|
* (a bare string, or an object of a later build's kind) and was dropped. */
|
|
@@ -17,7 +17,6 @@ import {
|
|
|
17
17
|
QueryJobIdentityConflictError,
|
|
18
18
|
QueryJobNotFoundError,
|
|
19
19
|
QueryJobPersistenceError,
|
|
20
|
-
QueryJobProviderOrphanFenceError,
|
|
21
20
|
type QueryJobAdmissionResult,
|
|
22
21
|
type QueryJobMutationResult,
|
|
23
22
|
} from './query-job-store.js'
|
|
@@ -141,10 +140,11 @@ function buildTaskRequest(run: TaskRun, tz: string): RestrictedRequest {
|
|
|
141
140
|
}
|
|
142
141
|
}
|
|
143
142
|
|
|
144
|
-
|
|
143
|
+
// No 'fence' kind since 6.61.3: an orphan fence holds a durable job at its start and never
|
|
144
|
+
// refuses its admission, so submit cannot throw it.
|
|
145
|
+
function classifySubmitError(error: unknown): 'identity_conflict' | 'adopt' | 'drain' | 'transient' {
|
|
145
146
|
if (error instanceof QueryJobIdentityConflictError) return 'identity_conflict'
|
|
146
147
|
if (error instanceof QueryJobActiveGenerationError || error instanceof QueryJobGenerationOrderError) return 'adopt'
|
|
147
|
-
if (error instanceof QueryJobProviderOrphanFenceError) return 'fence'
|
|
148
148
|
if (error instanceof MaintenanceLifecycleError) return 'drain'
|
|
149
149
|
if (error instanceof QueryJobCoordinatorError && error.code === 'query_job_coordinator_shutting_down') return 'transient'
|
|
150
150
|
return 'transient'
|
|
@@ -273,11 +273,10 @@ async function adoptOrSubmit(deps: TaskDispatcherDeps, run: TaskRun): Promise<Ta
|
|
|
273
273
|
return accepted ?? run
|
|
274
274
|
}
|
|
275
275
|
}
|
|
276
|
-
const fenceMs = error instanceof QueryJobProviderOrphanFenceError ? error.retryAfterMs : TASK_DISPATCH_LIMITS.submitSpacingMs
|
|
277
276
|
const consume = kind !== 'drain'
|
|
278
277
|
await persistRun(deps, run.id, {
|
|
279
278
|
status: 'dispatching',
|
|
280
|
-
retryAfter: nowMs(deps) +
|
|
279
|
+
retryAfter: nowMs(deps) + TASK_DISPATCH_LIMITS.submitSpacingMs,
|
|
281
280
|
...(consume ? { submitAttempts: run.submitAttempts + 1 } : {}),
|
|
282
281
|
})
|
|
283
282
|
throw new TaskRunError(503, kind === 'drain' ? 'admissions_closed' : 'submit_failed', error instanceof Error ? error.message : 'submit failed')
|
package/server/routes/health.ts
CHANGED
|
@@ -138,6 +138,7 @@ function durableQueryJobStatus() {
|
|
|
138
138
|
&& !runtime.shuttingDown,
|
|
139
139
|
protocolVersion: configured.protocolVersion,
|
|
140
140
|
activeRuns: runtime.activeRuns,
|
|
141
|
+
heldRuns: runtime.heldRuns,
|
|
141
142
|
shuttingDown: runtime.shuttingDown,
|
|
142
143
|
callbackPersistenceFailures: runtime.callbackPersistenceFailures,
|
|
143
144
|
terminalProjectionFailures: runtime.terminalProjectionFailures,
|
|
@@ -153,6 +154,7 @@ function durableQueryJobStatus() {
|
|
|
153
154
|
evictedHydratedJobs: runtime.store.evictedHydratedJobs,
|
|
154
155
|
journalFailures: runtime.store.journalFailures,
|
|
155
156
|
interruptedOnBoot: runtime.store.interruptedOnBoot,
|
|
157
|
+
requeuedOnBoot: runtime.store.requeuedOnBoot,
|
|
156
158
|
lastErrorCode: runtime.store.lastErrorCode,
|
|
157
159
|
lastSuccessfulWriteAt: runtime.store.lastSuccessfulWriteAt,
|
|
158
160
|
rootFingerprint: runtime.store.rootFingerprint,
|
|
@@ -112,6 +112,11 @@ function statusSnapshot(credentials: MaintenanceOperationCredentials = {}) {
|
|
|
112
112
|
serverInstanceId: getServerInstanceId(),
|
|
113
113
|
bootId: serverMetrics.bootId,
|
|
114
114
|
activeJobs: jobs.activeRuns,
|
|
115
|
+
// 6.61.3: admitted prompts not started yet (an orphan fence, an earlier run in their
|
|
116
|
+
// session, or this drain). They are journaled `accepted` and a restart re-queues
|
|
117
|
+
// them, so they never block safeToRestart; a rollback below 6.61.3 interrupts them.
|
|
118
|
+
heldJobs: jobs.heldRuns,
|
|
119
|
+
requeuedOnBoot: jobs.store.requeuedOnBoot ?? 0,
|
|
115
120
|
activeTranscriptionSessions,
|
|
116
121
|
// Split out so a blocked operator can SEE why, and so "0 orphans" can never
|
|
117
122
|
// again coexist with a held lock: the orphans endpoint reads the quarantine
|
package/server/routes/meeting.ts
CHANGED
|
@@ -171,6 +171,21 @@ function cosOpsPipelineConfigured(): boolean {
|
|
|
171
171
|
return Boolean(process.env.COS_SCRIPTS_DIR?.trim())
|
|
172
172
|
}
|
|
173
173
|
|
|
174
|
+
/**
|
|
175
|
+
* Why a correction left the meeting's markdown alone when it could not read it.
|
|
176
|
+
*
|
|
177
|
+
* 6.61.2: when two meetings declare the same session the lookup chooses neither
|
|
178
|
+
* and the record falls back to the capture, whose own scribe the merge retired.
|
|
179
|
+
* "Unreadable" would send the reader looking for a missing file; the real
|
|
180
|
+
* answer is the duplicate claim, so the response names it.
|
|
181
|
+
*/
|
|
182
|
+
function markdownUnreadableReason(operations: { mergedScribeConflict?: string[] } | null): string {
|
|
183
|
+
const conflict = operations?.mergedScribeConflict
|
|
184
|
+
return conflict && conflict.length > 1
|
|
185
|
+
? `no merged meeting chosen: ${conflict.length} meetings declare this session (${conflict.join(', ')})`
|
|
186
|
+
: 'meeting markdown unreadable'
|
|
187
|
+
}
|
|
188
|
+
|
|
174
189
|
function requestedRecordMatches(requested: unknown, expected: string): boolean {
|
|
175
190
|
return requested == null || requested === '' || requested === expected
|
|
176
191
|
}
|
|
@@ -1331,7 +1346,7 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
1331
1346
|
proseHits: md?.proseHits ?? [],
|
|
1332
1347
|
markdownSkipped: !plan.value.coveredAllWithLabel
|
|
1333
1348
|
? 'partial relabel: transcript turns cannot be mapped to chunk indices'
|
|
1334
|
-
: markdownRaw === null ?
|
|
1349
|
+
: markdownRaw === null ? markdownUnreadableReason(operations) : null,
|
|
1335
1350
|
}
|
|
1336
1351
|
|
|
1337
1352
|
if (req.body?.dryRun === true || req.body?.confirm !== true) {
|
|
@@ -1684,7 +1699,7 @@ export function createMeetingRouter(deps: MeetingRouteDependencies = {}): Router
|
|
|
1684
1699
|
// markdown changes with no hint the document was never touched.
|
|
1685
1700
|
markdownSkipped: !plan.value.coveredAllWithLabel
|
|
1686
1701
|
? 'partial de-attribution: transcript turns cannot be mapped to chunk indices'
|
|
1687
|
-
: markdownUnreadable ?
|
|
1702
|
+
: markdownUnreadable ? markdownUnreadableReason(operations) : null,
|
|
1688
1703
|
scope: 'meeting' as const,
|
|
1689
1704
|
chunks: plan.value.changed,
|
|
1690
1705
|
surfaces,
|
|
@@ -49,6 +49,8 @@ function wireError(error: unknown): { status: number; body: WireError } {
|
|
|
49
49
|
|| error instanceof QueryJobIdentityConflictError
|
|
50
50
|
|| error instanceof QueryJobActiveGenerationError
|
|
51
51
|
|| error instanceof QueryJobGenerationOrderError
|
|
52
|
+
// Since 6.61.3 admission never throws this (the fence holds a job at its start);
|
|
53
|
+
// kept so the store error map stays complete for any caller of markStarting.
|
|
52
54
|
|| error instanceof QueryJobProviderOrphanFenceError
|
|
53
55
|
|| error instanceof QueryJobAnswerCommittingError
|
|
54
56
|
|| error instanceof QueryJobNotTerminalError) status = 409
|