@gotcos/glasses-server 6.36.9 → 6.36.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,109 @@
|
|
|
1
1
|
## Unreleased
|
|
2
2
|
|
|
3
|
+
## 6.36.11
|
|
4
|
+
- **Fences now record WHY, so the population can be measured before anything resolves
|
|
5
|
+
automatically.** Two plans designed an automatic fence resolver and both were rejected —
|
|
6
|
+
the second because there has never been a single fence on the machine to look at. If
|
|
7
|
+
`timeout` dominates, the child had the full 21-minute budget to run tool calls before
|
|
8
|
+
SIGKILL and re-delivery would re-execute them, so no automatic clear is ever safe. That
|
|
9
|
+
question was unanswerable and now is not.
|
|
10
|
+
- **`reaped` is reported by the adapter, never derived.** A signal-killed child reports
|
|
11
|
+
`code === null` and the handlers only assign `exitCode` for a numeric code — so deriving
|
|
12
|
+
"was it reaped" from `exitCode` reports NEVER REAPED for the dominant timeout shape
|
|
13
|
+
(SIGTERM, then SIGKILL), which is exactly backwards for the decision this data informs.
|
|
14
|
+
The first cut of this change did derive it. `AttachedTurnFailureResult` now carries
|
|
15
|
+
`reaped`, true from every settle reached via `close`/`error` and false only from the
|
|
16
|
+
force-settle that fires when `close` never arrived.
|
|
17
|
+
- **An unreadable adapter result records nothing, not zeroes.** Reading `{}` and writing
|
|
18
|
+
`exitCode: null, childReaped: false` states two facts about a child nothing is known
|
|
19
|
+
about, indistinguishable on disk from a confirmed-unreaped timeout — corrupting the one
|
|
20
|
+
discriminator this evidence exists to establish.
|
|
21
|
+
- **`fenceSite` says which site fired.** `adapterReason` cannot substitute: the catch site
|
|
22
|
+
inherits whatever the adapter last reported, so a route crash AFTER a clean delivery
|
|
23
|
+
records `ok` — the strongest possible reason NOT to re-deliver, which would otherwise
|
|
24
|
+
read as "nothing went wrong".
|
|
25
|
+
- **One resolved reason for the record and the log.** The record said `unreadable_result`
|
|
26
|
+
while the breadcrumb said `unknown`, so an operator grepping for the sentinel found
|
|
27
|
+
nothing. This is the same contradiction 6.36.10 fixed at the other fence site,
|
|
28
|
+
re-committed one release later in the same handler; both now read one value.
|
|
29
|
+
- **A release no longer destroys the evidence.** `releaseFence` deletes the row, and the
|
|
30
|
+
realistic first-fence sequence is: fence lands, Control's card appears, it is released,
|
|
31
|
+
the distribution is gone. The release breadcrumb now carries the whole record.
|
|
32
|
+
- **Spawn identity as `{pid, startMs}` PAIRS.** `recordedPids` held bare pids and the
|
|
33
|
+
measured start was discarded; a pid alone cannot be told apart from a recycled one.
|
|
34
|
+
The evidence type is narrowed to the six adapter fields so the spread at the fence
|
|
35
|
+
sites cannot clobber the fence's own identity — `Partial<FenceEvidence>` permitted
|
|
36
|
+
`provider`, and the adapter result carries one that would write null and fail
|
|
37
|
+
`isFenceRecord` on the next read, silently un-enforcing the fence.
|
|
38
|
+
- **All fields OPTIONAL and NOT in `isFenceRecord`.** That predicate is cast-based, so a
|
|
39
|
+
required field would type as present while being undefined at runtime; extending it
|
|
40
|
+
would reclassify existing rows as unrecognised and silently un-enforce them.
|
|
41
|
+
- **DISK ONLY.** Nothing reaches the wire; asserted against the real `/fences` body and
|
|
42
|
+
the release preview.
|
|
43
|
+
- **No behaviour change.** Nothing clears, nothing refuses differently. Upgrade, downgrade
|
|
44
|
+
and the COS Control card were all verified unaffected.
|
|
45
|
+
- **Coverage, stated honestly.** The adapter's `reaped` contract is tested at the adapter,
|
|
46
|
+
driving a real `close(null)` — a route test could not cover it, because the route
|
|
47
|
+
fixtures supply `reaped` themselves and would pass with the adapter gutted. Mutation
|
|
48
|
+
results: caught — derived-from-exitCode, adapter stops reporting reaped, startMs zeroed,
|
|
49
|
+
empty spawn list, missing fenceSite, unrecorded adapterReason, wire leak. **Survived, and
|
|
50
|
+
therefore unverified: `stderrClass` is written but asserted nowhere, and `fail()`'s
|
|
51
|
+
`reaped: false` default on the `not_attempted` paths (which never fence).** The
|
|
52
|
+
`route_error` fence site remains reachable by no test.
|
|
53
|
+
- **Known gaps, not fixed here:** `stderrClass` appears in the breadcrumbs but on a
|
|
54
|
+
default install (`COS_THREAD_FENCE_DURABLE` unset) nothing is written to disk at all;
|
|
55
|
+
and reading the distribution means reading `thread-fences.json` or the server log —
|
|
56
|
+
there is no UI for it.
|
|
57
|
+
|
|
58
|
+
## 6.36.10
|
|
59
|
+
- **A fenced thread had no exit and left no trace.** An ambiguous delivery fences the
|
|
60
|
+
target so a prompt cannot be double-delivered into a real conversation — that is
|
|
61
|
+
correct and stays. Everything around it was wrong: the fence lived in a process-local
|
|
62
|
+
Map, wrote no log line at either site, had no list, and had no release. It was
|
|
63
|
+
discoverable only by being refused, and the only thing that cleared it was a restart.
|
|
64
|
+
- **Now listable and releasable without a restart.** `GET /api/agent-sessions/fences`
|
|
65
|
+
lists them; `POST /api/agent-sessions/fences/release` clears one. Addressed by DIGEST,
|
|
66
|
+
never by raw target key — the key embeds the private native thread id. Fails closed:
|
|
67
|
+
without `confirm: true` it returns 400 with a preview of what would be reopened. That
|
|
68
|
+
confirmation is a deliberate second call, NOT proof a human looked — the API token is
|
|
69
|
+
shared by the phone, the lens and every COS agent session, so nothing is structurally
|
|
70
|
+
prevented from asserting it. The comment says so rather than overclaiming.
|
|
71
|
+
- **Durable storage ships INERT, behind `COS_THREAD_FENCE_DURABLE=1`, default off.**
|
|
72
|
+
Persisting the fence is the right direction, but durability without a reachable
|
|
73
|
+
release is a regression, not a fix: today "Restart Server" in COS Control clears a
|
|
74
|
+
fence, and making it survive restarts with no operator surface in Control would turn
|
|
75
|
+
an 8-second annoyance into a permanently dead thread needing a terminal. The flag
|
|
76
|
+
flips on when COS Control has a Fences card. The routes above already remove the
|
|
77
|
+
restart from the recovery path.
|
|
78
|
+
- **A write can never erase what it could not read.** `TargetGuard` hydrates only the
|
|
79
|
+
rows it understood and saves its map wholesale, so a single unrecognised row — a
|
|
80
|
+
newer schema, a partial write, one bad field — would otherwise be erased by the next
|
|
81
|
+
fence on an unrelated thread, silently reopening every other fenced thread. Writes now
|
|
82
|
+
merge unrecognised rows back through. A corrupt file is quarantined to
|
|
83
|
+
`.corrupt-<ts>` rather than dropped, and uses `durableAtomicWriteFileSync` (fsync of
|
|
84
|
+
bytes, metadata and directory; randomized exclusive temp name) rather than the
|
|
85
|
+
lightweight cache writer.
|
|
86
|
+
- **A release is persisted before it is reported.** Mutating memory first and reporting
|
|
87
|
+
success meant an operator could be told a thread was open, write to it, and find it
|
|
88
|
+
fenced again after the next restart with no record of why. A failed write now returns
|
|
89
|
+
500 `persist_failed` and the fence holds. `GET /fences` reports `degraded` when the
|
|
90
|
+
last write failed — a memory-only fence set is otherwise indistinguishable from a
|
|
91
|
+
durable one until the process restarts.
|
|
92
|
+
- **Visible.** Breadcrumbs at both fence-set sites (tagged `ambiguous` vs `route_error`)
|
|
93
|
+
and both fence-hit routes (turn, attach). No raw target key in any of them. The
|
|
94
|
+
route_error line reports the hoisted pre-turn head rather than hardcoding
|
|
95
|
+
`unavailable`, which contradicted the record it had just written.
|
|
96
|
+
- **Known, not fixed here:** releasing a fence does not by itself make the thread
|
|
97
|
+
attachable — the turn that fenced it left a binding holding the target for its
|
|
98
|
+
30-minute TTL, so the next attach refuses `native_target_busy`. There is no detach
|
|
99
|
+
route. `/attachability` still does not consult the fence, so the lens menu renders
|
|
100
|
+
Continue enabled on a fenced thread; the refusal is honest, the menu is not yet.
|
|
101
|
+
- **Coverage.** 11 mutations against the new guards fail the suite, including all four
|
|
102
|
+
that survived the first QA pass (the release handle check, and the provider/reason/
|
|
103
|
+
fencedAt row validators). One documented survivor remains: removing the write-once
|
|
104
|
+
guard on `fence()`, which no route can reach because both fence sites sit inside the
|
|
105
|
+
`tryClaim` section. The code says so at the call site.
|
|
106
|
+
|
|
3
107
|
## 6.36.9
|
|
4
108
|
- **The queue gate can now see COS's own bindings, which closes a 30-minute lockout.**
|
|
5
109
|
`threadOccupancy` sees FOREIGN holders only — `OccupancyReason` has no
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@gotcos/glasses-server",
|
|
3
|
-
"version": "6.36.
|
|
3
|
+
"version": "6.36.11",
|
|
4
4
|
"description": "COS Glasses \u2014 self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, or Cursor Agent CLI",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
package/server/index.ts
CHANGED
|
@@ -109,6 +109,7 @@ import { createThreadTurnQueueRouter, drainAllThreads } from './routes/thread-tu
|
|
|
109
109
|
import { transcriptTurnEnded } from './lib/thread-turn-queue-store.js'
|
|
110
110
|
import { transcriptPathFor } from './lib/native-head.js'
|
|
111
111
|
import { deliverQueuedTurnOverLoopback } from './lib/thread-turn-queue-deliver.js'
|
|
112
|
+
import { readFences, writeFences } from './lib/thread-fence-store.js'
|
|
112
113
|
import type { QueuedThreadTurn } from './lib/thread-turn-queue.js'
|
|
113
114
|
|
|
114
115
|
const PORT = parseInt(process.env.PORT ?? '3141', 10)
|
|
@@ -582,6 +583,23 @@ if (threadAttachEnabled()) {
|
|
|
582
583
|
}
|
|
583
584
|
|
|
584
585
|
app.use('/api', createAgentSessionBindingsRouter({
|
|
586
|
+
// Durable fences (6.36.10), OFF BY DEFAULT.
|
|
587
|
+
//
|
|
588
|
+
// The fence is the one piece of state whose loss writes twice into a real
|
|
589
|
+
// conversation, so persisting it is the right direction. But durability
|
|
590
|
+
// without a reachable release is a REGRESSION, not a fix: today "Restart
|
|
591
|
+
// Server" in COS Control clears a fence, and making it survive restarts with
|
|
592
|
+
// no operator surface in Control turns an 8-second annoyance into a
|
|
593
|
+
// permanently dead thread that needs a terminal to clear. Miles, 2026-08-12:
|
|
594
|
+
// "we couldn't do anything without bash, that shouldn't be the case."
|
|
595
|
+
//
|
|
596
|
+
// So the storage ships inert. GET /agent-sessions/fences and
|
|
597
|
+
// POST /agent-sessions/fences/release work either way — which already removes
|
|
598
|
+
// the restart from the recovery path — and this flag flips on once COS Control
|
|
599
|
+
// has a Fences card.
|
|
600
|
+
fencePersistence: process.env.COS_THREAD_FENCE_DURABLE === '1'
|
|
601
|
+
? { load: readFences, save: writeFences }
|
|
602
|
+
: undefined,
|
|
585
603
|
probes: occupancyProbes,
|
|
586
604
|
dirs: occupancyDirs,
|
|
587
605
|
now: () => Date.now(),
|
|
@@ -1063,9 +1063,29 @@ async function enrichLiveClaude(row: AgentSessionRow, roots: AgentSessionRoots):
|
|
|
1063
1063
|
const firstPrompt = await firstClaudeUserTitle(found)
|
|
1064
1064
|
const title = peek.customTitle ?? firstPrompt ?? row.display_label
|
|
1065
1065
|
const fullId = found.split('/').pop()?.replace(/\.jsonl$/i, '') || row.session_id
|
|
1066
|
+
// THE LIVE ROW'S `modified` IS A HEARTBEAT, NOT AN MTIME, AND IT GOES STALE.
|
|
1067
|
+
//
|
|
1068
|
+
// `liveClaudeRows` builds it from the peer registry's `lastActiveAt` (routes/
|
|
1069
|
+
// agent-sessions.ts:257). That value tracks the registry record, not the
|
|
1070
|
+
// transcript, so a session that is actively writing keeps reporting whenever the
|
|
1071
|
+
// registry last moved. Measured 2026-08-18 on three live sessions: the wire said
|
|
1072
|
+
// 55.3m / 407.7m / 435.0m while the transcript had been written 0.1m / 0.2m /
|
|
1073
|
+
// 5.1m earlier — under-reporting a live session by up to 7.2 hours.
|
|
1074
|
+
//
|
|
1075
|
+
// We are already holding the resolved transcript path and have already read it
|
|
1076
|
+
// twice, so the true mtime costs one stat. A failed stat keeps the heartbeat:
|
|
1077
|
+
// worse, but no worse than today.
|
|
1078
|
+
//
|
|
1079
|
+
// THAT FALLBACK IS DEFENSIVE AND UNREACHED BY THE SUITE. `if (!found) return row`
|
|
1080
|
+
// above already catches the missing-file case, so reaching `: row.modified` needs a
|
|
1081
|
+
// stat to fail on a file that was just resolved. A mutation of it survives; that is
|
|
1082
|
+
// recorded rather than papered over.
|
|
1083
|
+
const st = await fileStat(found)
|
|
1084
|
+
const modified = st?.isFile ? isoFromMtime(st.mtimeMs) : row.modified
|
|
1066
1085
|
return {
|
|
1067
1086
|
...row,
|
|
1068
1087
|
session_id: fullId,
|
|
1088
|
+
modified,
|
|
1069
1089
|
display_label: title || row.display_label || 'Claude session',
|
|
1070
1090
|
...discussionFields(title || row.display_label, firstPrompt || '', peek.latestAssistant),
|
|
1071
1091
|
}
|
|
@@ -194,6 +194,18 @@ export interface AttachedTurnFailureResult {
|
|
|
194
194
|
*/
|
|
195
195
|
detail: string | null
|
|
196
196
|
exitCode: number | null
|
|
197
|
+
/**
|
|
198
|
+
* Did `close`/`exit` actually fire, i.e. did the kernel reap the child?
|
|
199
|
+
*
|
|
200
|
+
* NOT DERIVABLE FROM `exitCode`. A child killed by a signal reports
|
|
201
|
+
* `code === null`, and the handlers below only assign `exitCode` for a numeric
|
|
202
|
+
* code — so the dominant timeout shape (SIGTERM, then SIGKILL) is reaped while
|
|
203
|
+
* leaving `exitCode` null. Deriving reaping from the code therefore reports
|
|
204
|
+
* "never reaped" for exactly the case a reader most needs to identify.
|
|
205
|
+
* Only the force-settle path, which fires when `close` never arrived, is
|
|
206
|
+
* genuinely unreaped.
|
|
207
|
+
*/
|
|
208
|
+
reaped: boolean
|
|
197
209
|
stderrClass: AttachedStderrClass
|
|
198
210
|
durationMs: number
|
|
199
211
|
}
|
|
@@ -745,6 +757,10 @@ function fail(
|
|
|
745
757
|
nativeThreadId: null,
|
|
746
758
|
returnedNativeId: null,
|
|
747
759
|
delivery,
|
|
760
|
+
// Conservative default: NOT OBSERVED reaped. Most `fail()` callers are
|
|
761
|
+
// `not_attempted` paths where no child exists, and the ones that do have a
|
|
762
|
+
// child override this from `settleFailure`.
|
|
763
|
+
reaped: false,
|
|
748
764
|
reason,
|
|
749
765
|
detail: null,
|
|
750
766
|
exitCode: null,
|
|
@@ -1051,6 +1067,10 @@ function driveChild(input: DriveInput): Promise<AttachedTurnResult> {
|
|
|
1051
1067
|
nativeThreadId,
|
|
1052
1068
|
returnedNativeId: observedIds.length === 1 ? observedIds[0]! : null,
|
|
1053
1069
|
exitCode,
|
|
1070
|
+
// Every settle but the force-settle below is reached from `finishTerminal`,
|
|
1071
|
+
// which only runs from the `close`/`error` handlers — so the child was
|
|
1072
|
+
// reaped. The one exception overrides this explicitly.
|
|
1073
|
+
reaped: true,
|
|
1054
1074
|
stderrClass: classifyStderr(stderrSample),
|
|
1055
1075
|
durationMs: duration(),
|
|
1056
1076
|
...over,
|
|
@@ -1193,7 +1213,7 @@ function driveChild(input: DriveInput): Promise<AttachedTurnResult> {
|
|
|
1193
1213
|
// A child that survived SIGKILL cannot be reached from here, and
|
|
1194
1214
|
// blocking forever would wedge the coordinator and any Control drain
|
|
1195
1215
|
// behind it.
|
|
1196
|
-
settleFailure('timeout', { detail: 'unreaped' })
|
|
1216
|
+
settleFailure('timeout', { detail: 'unreaped', reaped: false })
|
|
1197
1217
|
}, FORCE_SETTLE_MS)
|
|
1198
1218
|
}, KILL_GRACE_MS)
|
|
1199
1219
|
}, timeoutMs)
|
|
@@ -0,0 +1,165 @@
|
|
|
1
|
+
// Durable storage for target fences.
|
|
2
|
+
//
|
|
3
|
+
// A fence shuts a native thread that may already hold an undelivered COS turn. It
|
|
4
|
+
// is the one piece of state whose LOSS writes twice into a real human conversation
|
|
5
|
+
// — agent-session-binding-registry.ts records the incident verbatim: "the
|
|
6
|
+
// process-local fence re-opened on restart and delivered a second copy."
|
|
7
|
+
//
|
|
8
|
+
// SEPARATE FROM THE DECISIONS, matching thread-turn-queue-store.ts: TargetGuard
|
|
9
|
+
// holds the rules and takes load/save as injected callbacks, so its behaviour stays
|
|
10
|
+
// testable in memory and only production touches the disk.
|
|
11
|
+
//
|
|
12
|
+
// UNDER THE DATA HOME, never the generation directory, which Update Server replaces
|
|
13
|
+
// wholesale. Same lesson as the stranded voice profiles.
|
|
14
|
+
//
|
|
15
|
+
// NEVER TTL'd AND NEVER EVICTED. `heads` and fork refs both bound their maps, and
|
|
16
|
+
// both evict in the SAFE direction — losing a head asks the user to acknowledge,
|
|
17
|
+
// losing a fork ref makes them find the thread by hand. Losing a fence silently
|
|
18
|
+
// reopens a thread that may hold an undelivered turn, so the only way an entry
|
|
19
|
+
// leaves this file is an explicit operator release.
|
|
20
|
+
|
|
21
|
+
import { durableAtomicWriteFileSync, loadJsonOrQuarantine } from './atomic-fs.js'
|
|
22
|
+
import { dataPath } from './data-dir.js'
|
|
23
|
+
|
|
24
|
+
export interface FenceRecord {
|
|
25
|
+
/** The raw target key. On disk only — it embeds the private native thread id and
|
|
26
|
+
* is never emitted by any route. Callers outside this module use the digest. */
|
|
27
|
+
targetKey: string
|
|
28
|
+
provider: string
|
|
29
|
+
reason: string
|
|
30
|
+
/** The head digest as it stood BEFORE the ambiguous turn. Null ONLY when the
|
|
31
|
+
* failure happened before the head was read — `head` is scoped to the try, so
|
|
32
|
+
* the route-error site reads a hoisted copy rather than nothing. */
|
|
33
|
+
headBefore: string | null
|
|
34
|
+
turnId: string
|
|
35
|
+
bindingId: string | null
|
|
36
|
+
fencedAt: number
|
|
37
|
+
|
|
38
|
+
// ── EVIDENCE (all optional, all DISK-ONLY) ────────────────
|
|
39
|
+
//
|
|
40
|
+
// WHY THIS EXISTS. Two plans designed an automatic fence resolver and both were
|
|
41
|
+
// rejected, the second because we have never observed a single fence. The
|
|
42
|
+
// distribution decides whether any automatic path is safe: if `timeout`
|
|
43
|
+
// dominates, the child had the full 21-minute budget to run tool calls before
|
|
44
|
+
// SIGKILL and re-delivery re-executes them, so no automatic clear is ever safe.
|
|
45
|
+
// If the child exited on its own with a known code, the footing is much better.
|
|
46
|
+
//
|
|
47
|
+
// EVERY FIELD IS OPTIONAL, and deliberately NOT added to `isFenceRecord`. That
|
|
48
|
+
// predicate is cast-based (`r is FenceRecord`), so a REQUIRED field would type as
|
|
49
|
+
// present while being undefined at runtime and tsc could not catch it; and
|
|
50
|
+
// extending the validator would reclassify existing rows as unrecognised, routing
|
|
51
|
+
// them into the preserved-but-inert pile and silently un-enforcing real fences.
|
|
52
|
+
//
|
|
53
|
+
// NONE OF THIS REACHES THE WIRE. The router contract carries no pid and no native
|
|
54
|
+
// thread id; `listFences()` already omits `bindingId` as precedent.
|
|
55
|
+
|
|
56
|
+
/**
|
|
57
|
+
* The adapter's own verdict.
|
|
58
|
+
*
|
|
59
|
+
* REACHABILITY IS DECIDED BY `delivery === 'ambiguous'`, i.e. the stdin-write
|
|
60
|
+
* boundary — only a turn whose prompt bytes reached the child can fence. The
|
|
61
|
+
* reachable set is therefore NOT the full `AttachedTurnFailure` enum:
|
|
62
|
+
* timeout, provider_exit_nonzero, no_native_id_returned, native_id_mismatch,
|
|
63
|
+
* child_stdio_unavailable (detail `write_failed`), spawn_failed (ONLY via
|
|
64
|
+
* detail `child_error`; its other forms are `not_attempted` and never fence),
|
|
65
|
+
* plus two this route synthesises — `ok` (the adapter reported success with a
|
|
66
|
+
* non-delivered state, a contradiction) and `unreadable_result`.
|
|
67
|
+
* `adapter_internal_error` can NEVER fence: every emission site is
|
|
68
|
+
* `not_attempted` or guarded. `route_error` comes from the catch site.
|
|
69
|
+
*/
|
|
70
|
+
adapterReason?: string
|
|
71
|
+
/**
|
|
72
|
+
* WHICH fence site fired.
|
|
73
|
+
*
|
|
74
|
+
* `adapterReason` cannot substitute. The catch site inherits whatever the
|
|
75
|
+
* adapter last reported, so a route crash AFTER a clean delivery records
|
|
76
|
+
* `ok` — the single strongest reason NOT to re-deliver, which would read as
|
|
77
|
+
* "nothing went wrong". Only this field separates them.
|
|
78
|
+
*/
|
|
79
|
+
fenceSite?: 'ambiguous' | 'route_error'
|
|
80
|
+
/** Bounded self-authored discriminator, e.g. `unreaped`. Never provider output. */
|
|
81
|
+
adapterDetail?: string | null
|
|
82
|
+
exitCode?: number | null
|
|
83
|
+
/**
|
|
84
|
+
* Reported BY THE ADAPTER, never derived from `exitCode`.
|
|
85
|
+
*
|
|
86
|
+
* A signal-killed child reports `code === null`, and the adapter only assigns
|
|
87
|
+
* `exitCode` for a numeric code — so deriving this would record "never reaped"
|
|
88
|
+
* for the dominant timeout shape (SIGTERM then SIGKILL), which is the exact
|
|
89
|
+
* case this evidence exists to identify. Undefined when the adapter result was
|
|
90
|
+
* unreadable: absent, never a fabricated `false`.
|
|
91
|
+
*/
|
|
92
|
+
childReaped?: boolean
|
|
93
|
+
stderrClass?: string
|
|
94
|
+
/** How long the turn ran before it failed. A timeout is ~21 minutes of tool calls. */
|
|
95
|
+
durationMs?: number
|
|
96
|
+
/** Every child COS spawned for this turn, with its MEASURED start. Both are needed:
|
|
97
|
+
* a pid alone cannot be distinguished from a recycled one. An EMPTY list means no
|
|
98
|
+
* child was ever spawned, which no resolver may ever read as "nothing landed". */
|
|
99
|
+
spawns?: Array<{ pid: number; startMs: number }>
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
export function fencePath(): string {
|
|
103
|
+
return dataPath('thread-fences.json')
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
function isFenceRecord(r: unknown): r is FenceRecord {
|
|
107
|
+
return !!r && typeof r === 'object'
|
|
108
|
+
&& typeof (r as FenceRecord).targetKey === 'string' && (r as FenceRecord).targetKey.length > 0
|
|
109
|
+
&& typeof (r as FenceRecord).provider === 'string'
|
|
110
|
+
&& typeof (r as FenceRecord).reason === 'string'
|
|
111
|
+
&& typeof (r as FenceRecord).fencedAt === 'number'
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/** The raw array on disk, or [] when the file is missing or was quarantined. */
|
|
115
|
+
function rawRows(): unknown[] {
|
|
116
|
+
const loaded = loadJsonOrQuarantine<unknown>(fencePath())
|
|
117
|
+
if (loaded.status === 'corrupt') {
|
|
118
|
+
// Quarantined to `<path>.corrupt-<ts>` rather than discarded: the bytes are
|
|
119
|
+
// the only record of which threads were fenced, and this is the one state
|
|
120
|
+
// whose silent loss double-writes a real conversation.
|
|
121
|
+
console.warn(`[thread-fence-store] fence file was corrupt, quarantined as ${loaded.quarantinedAs}`)
|
|
122
|
+
return []
|
|
123
|
+
}
|
|
124
|
+
if (loaded.status !== 'ok') return []
|
|
125
|
+
if (!Array.isArray(loaded.data)) {
|
|
126
|
+
console.warn('[thread-fence-store] fence file is not an array — treating as empty')
|
|
127
|
+
return []
|
|
128
|
+
}
|
|
129
|
+
return loaded.data
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
/**
|
|
133
|
+
* Every stored fence this build can understand.
|
|
134
|
+
*
|
|
135
|
+
* A missing or corrupt file reads as empty. That fails OPEN, deliberately:
|
|
136
|
+
* failing closed would refuse every thread on the machine with no way back,
|
|
137
|
+
* while failing open is exactly the pre-6.36.10 behaviour (the fence was
|
|
138
|
+
* process-local and died on restart), so it cannot be a regression. A corrupt
|
|
139
|
+
* file is quarantined rather than dropped, so the evidence survives.
|
|
140
|
+
*/
|
|
141
|
+
export function readFences(): FenceRecord[] {
|
|
142
|
+
return rawRows().filter(isFenceRecord)
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
/**
|
|
146
|
+
* Replace the stored set, PRESERVING rows this build could not validate.
|
|
147
|
+
*
|
|
148
|
+
* THE MERGE IS THE WHOLE POINT. `TargetGuard` holds only the rows `readFences`
|
|
149
|
+
* understood and saves its map wholesale, so without this a single unrecognised
|
|
150
|
+
* row — a newer schema, a partial write, one bad field — would be erased by the
|
|
151
|
+
* next fence on an unrelated thread, silently reopening every other fenced
|
|
152
|
+
* thread. Preserved rows are inert (nothing enforces a fence that is not in the
|
|
153
|
+
* map) but they are never destroyed by a write that did not understand them.
|
|
154
|
+
*
|
|
155
|
+
* Uses the DURABLE writer, not the lightweight one: fsync of bytes, metadata and
|
|
156
|
+
* directory, plus a randomized exclusive temp name so two independent writers
|
|
157
|
+
* cannot share `<path>.tmp`.
|
|
158
|
+
*/
|
|
159
|
+
export function writeFences(rows: FenceRecord[]): void {
|
|
160
|
+
const preserved = rawRows().filter(r => !isFenceRecord(r))
|
|
161
|
+
if (preserved.length > 0) {
|
|
162
|
+
console.warn(`[thread-fence-store] preserving ${preserved.length} unrecognised fence row(s) through this write`)
|
|
163
|
+
}
|
|
164
|
+
durableAtomicWriteFileSync(fencePath(), `${JSON.stringify([...rows, ...preserved], null, 2)}\n`)
|
|
165
|
+
}
|
|
@@ -94,6 +94,7 @@
|
|
|
94
94
|
// does not — a future remount above the parser — the POST routes see a non-object
|
|
95
95
|
// and answer 400. They never treat an unparsed body as an empty one.
|
|
96
96
|
|
|
97
|
+
import type { FenceRecord } from '../lib/thread-fence-store.js'
|
|
97
98
|
import { Router, type Request, type Response } from 'express'
|
|
98
99
|
import { createHash, randomUUID } from 'node:crypto'
|
|
99
100
|
import {
|
|
@@ -268,6 +269,12 @@ export type AttachedTurnResult =
|
|
|
268
269
|
| { ok: boolean; delivery: 'not_attempted' | 'aborted' | 'ambiguous' | 'delivered' }
|
|
269
270
|
|
|
270
271
|
export interface AgentSessionBindingsDeps {
|
|
272
|
+
/**
|
|
273
|
+
* Durable fence storage. OPTIONAL, and omitting it is what keeps the existing
|
|
274
|
+
* suite in memory: a test that silently began writing the real data home would
|
|
275
|
+
* leak fences between cases and into the running server. Production wires it.
|
|
276
|
+
*/
|
|
277
|
+
fencePersistence?: FencePersistence
|
|
271
278
|
probes: OccupancyProbes
|
|
272
279
|
dirs: OccupancyDirs
|
|
273
280
|
/** Epoch ms. Injected so lease expiry is decidable in a test without waiting. */
|
|
@@ -829,24 +836,97 @@ export const MAX_TRACKED_HEADS = 512
|
|
|
829
836
|
/** A COS session id may contain ':' and '/', which is exactly why it is never projected. */
|
|
830
837
|
export const COS_SESSION_ID_RE = /^[A-Za-z0-9][A-Za-z0-9._:@/-]{0,127}$/
|
|
831
838
|
|
|
839
|
+
/** What a fence records about the turn that set it. */
|
|
840
|
+
export interface FenceEvidence {
|
|
841
|
+
provider: string
|
|
842
|
+
/** The adapter's verdict, or 'route_error' at the catch site. DISK-ONLY. */
|
|
843
|
+
adapterReason?: string
|
|
844
|
+
/** WHICH site fenced. `adapterReason` cannot substitute: the catch site inherits
|
|
845
|
+
* whatever the adapter reported, so a fence can read 'ok' there. DISK-ONLY. */
|
|
846
|
+
fenceSite?: 'ambiguous' | 'route_error'
|
|
847
|
+
adapterDetail?: string | null
|
|
848
|
+
exitCode?: number | null
|
|
849
|
+
childReaped?: boolean
|
|
850
|
+
stderrClass?: string
|
|
851
|
+
durationMs?: number
|
|
852
|
+
spawns?: Array<{ pid: number; startMs: number }>
|
|
853
|
+
/** The head BEFORE the ambiguous turn. Null ONLY when the failure happened
|
|
854
|
+
* before the head was read. */
|
|
855
|
+
headBefore: string | null
|
|
856
|
+
turnId: string
|
|
857
|
+
bindingId: string | null
|
|
858
|
+
now: number
|
|
859
|
+
}
|
|
860
|
+
|
|
861
|
+
export type ReleaseOutcome =
|
|
862
|
+
| { ok: true; row: FenceRecord }
|
|
863
|
+
| { ok: false; reason: 'unknown_fence' | 'persist_failed' }
|
|
864
|
+
|
|
865
|
+
/** Injected so TargetGuard stays testable in memory; production wires the store. */
|
|
866
|
+
export interface FencePersistence {
|
|
867
|
+
load: () => FenceRecord[]
|
|
868
|
+
save: (rows: FenceRecord[]) => void
|
|
869
|
+
}
|
|
870
|
+
|
|
832
871
|
/**
|
|
833
|
-
*
|
|
834
|
-
*
|
|
872
|
+
* One COS turn per native target, and a target that may already hold an
|
|
873
|
+
* undelivered turn stays shut.
|
|
874
|
+
*
|
|
875
|
+
* CLAIMS ARE PROCESS-LOCAL; FENCES ARE DURABLE (6.36.10). The two states fail in
|
|
876
|
+
* opposite directions, which is why only one of them is persisted. Losing a claim
|
|
877
|
+
* on restart is safe — no turn is running after a restart anyway. Losing a FENCE
|
|
878
|
+
* reopens a thread that may already hold an undelivered turn, and the binding
|
|
879
|
+
* registry records what that cost: "the process-local fence re-opened on restart
|
|
880
|
+
* and delivered a second copy." So fences are written through to disk and
|
|
881
|
+
* rehydrated in the constructor, and the ONLY way one leaves the map is an
|
|
882
|
+
* explicit operator release (`releaseFence`).
|
|
835
883
|
*
|
|
836
|
-
*
|
|
837
|
-
*
|
|
838
|
-
*
|
|
839
|
-
* Phase 2, which owns that journal. What lives here is the process-lifetime
|
|
840
|
-
* version, which is enough to make the two properties true for a running server
|
|
841
|
-
* and fails in the safe direction on restart: a claim is released (no turn is
|
|
842
|
-
* running after a restart anyway) and a FENCE is lost, which is the one that
|
|
843
|
-
* matters and is why it is called out rather than implied.
|
|
884
|
+
* Persistence is INJECTED rather than imported. A test that silently began writing
|
|
885
|
+
* the real data home would leak fences between cases and into the running server,
|
|
886
|
+
* so the suite runs with `null` and stays in memory.
|
|
844
887
|
*/
|
|
845
888
|
class TargetGuard {
|
|
846
889
|
/** targetKey -> turnId of the single COS turn allowed to be in flight. */
|
|
847
890
|
private readonly claims = new Map<string, string>()
|
|
848
|
-
/** targetKey ->
|
|
849
|
-
|
|
891
|
+
/** targetKey -> the fence record. DURABLE as of 6.36.10: persistence is injected
|
|
892
|
+
* so tests stay in memory and only production touches the data home. */
|
|
893
|
+
private readonly fences = new Map<string, FenceRecord>()
|
|
894
|
+
private readonly persistence: FencePersistence | null
|
|
895
|
+
private persistDegraded = false
|
|
896
|
+
|
|
897
|
+
constructor(persistence: FencePersistence | null = null) {
|
|
898
|
+
this.persistence = persistence
|
|
899
|
+
if (persistence === null) return
|
|
900
|
+
// Rehydrate BEFORE the router serves. A fence that died on restart is exactly
|
|
901
|
+
// how a second copy of a turn reached a real transcript.
|
|
902
|
+
try {
|
|
903
|
+
for (const row of persistence.load()) this.fences.set(row.targetKey, row)
|
|
904
|
+
} catch (error) {
|
|
905
|
+
console.error(`[agent-session-bindings] fence rehydrate failed: ${error instanceof Error ? error.message : error}`)
|
|
906
|
+
}
|
|
907
|
+
}
|
|
908
|
+
|
|
909
|
+
/** True when the durable write succeeded (or there is nothing to persist to). */
|
|
910
|
+
private persistFences(rows: FenceRecord[]): boolean {
|
|
911
|
+
if (this.persistence === null) return true
|
|
912
|
+
try {
|
|
913
|
+
this.persistence.save(rows)
|
|
914
|
+
this.persistDegraded = false
|
|
915
|
+
return true
|
|
916
|
+
} catch (error) {
|
|
917
|
+
// The in-memory fence still holds for this process, so the thread stays shut
|
|
918
|
+
// NOW; what is lost is survival across a restart. Loud, and surfaced on
|
|
919
|
+
// GET /fences — a silent fallback is indistinguishable from working.
|
|
920
|
+
this.persistDegraded = true
|
|
921
|
+
console.error(`[agent-session-bindings] fence persist FAILED (fences hold in memory only): ${error instanceof Error ? error.message : error}`)
|
|
922
|
+
return false
|
|
923
|
+
}
|
|
924
|
+
}
|
|
925
|
+
|
|
926
|
+
/** Whether the last durable write failed. Reported, never inferred. */
|
|
927
|
+
degraded(): boolean {
|
|
928
|
+
return this.persistDegraded
|
|
929
|
+
}
|
|
850
930
|
/** bindingId -> the head digest this binding is currently reconciled to. */
|
|
851
931
|
private readonly heads = new Map<string, string>()
|
|
852
932
|
|
|
@@ -869,12 +949,80 @@ class TargetGuard {
|
|
|
869
949
|
if (this.claims.get(targetKey) === turnId) this.claims.delete(targetKey)
|
|
870
950
|
}
|
|
871
951
|
|
|
872
|
-
|
|
873
|
-
|
|
952
|
+
/**
|
|
953
|
+
* Write-once: the FIRST reason wins, so a later ambiguity cannot overwrite the
|
|
954
|
+
* evidence chain of an unresolved one.
|
|
955
|
+
*
|
|
956
|
+
* DEFENSIVE, AND UNVERIFIED BY EXECUTION. Both fence sites sit inside the
|
|
957
|
+
* `tryClaim` section, which serialises them per target, and a fenced target is
|
|
958
|
+
* refused at the check before it can reach either site again — so no route can
|
|
959
|
+
* currently fence the same key twice, and a mutation removing this guard passes
|
|
960
|
+
* the whole suite. It is kept because the one path that could reach it (the
|
|
961
|
+
* ambiguous site fences, then the response throws into the catch, which fences
|
|
962
|
+
* `claimedKey` again) would otherwise replace a record carrying `bindingId` with
|
|
963
|
+
* one carrying null. Do not read the passing suite as coverage of this line.
|
|
964
|
+
*/
|
|
965
|
+
fence(targetKey: string, reason: WriteRefusal, evidence: FenceEvidence): void {
|
|
966
|
+
if (this.fences.has(targetKey)) return
|
|
967
|
+
this.fences.set(targetKey, {
|
|
968
|
+
targetKey,
|
|
969
|
+
provider: evidence.provider,
|
|
970
|
+
reason,
|
|
971
|
+
headBefore: evidence.headBefore,
|
|
972
|
+
turnId: evidence.turnId,
|
|
973
|
+
bindingId: evidence.bindingId,
|
|
974
|
+
fencedAt: evidence.now,
|
|
975
|
+
adapterReason: evidence.adapterReason,
|
|
976
|
+
fenceSite: evidence.fenceSite,
|
|
977
|
+
adapterDetail: evidence.adapterDetail,
|
|
978
|
+
exitCode: evidence.exitCode,
|
|
979
|
+
childReaped: evidence.childReaped,
|
|
980
|
+
stderrClass: evidence.stderrClass,
|
|
981
|
+
durationMs: evidence.durationMs,
|
|
982
|
+
spawns: evidence.spawns,
|
|
983
|
+
})
|
|
984
|
+
this.persistFences([...this.fences.values()])
|
|
874
985
|
}
|
|
875
986
|
|
|
876
987
|
fencedReason(targetKey: string): WriteRefusal | null {
|
|
877
|
-
|
|
988
|
+
const row = this.fences.get(targetKey)
|
|
989
|
+
return row === undefined ? null : (row.reason as WriteRefusal)
|
|
990
|
+
}
|
|
991
|
+
|
|
992
|
+
/** Every fence, REDACTED for the wire: the raw targetKey embeds the private
|
|
993
|
+
* native thread id, so callers address a fence by its deterministic digest. */
|
|
994
|
+
listFences(): Array<{ target: string; provider: string; reason: string; headBefore: string | null; turnId: string; fencedAt: number }> {
|
|
995
|
+
return [...this.fences.values()].map(row => ({
|
|
996
|
+
target: opaqueRevision(row.targetKey),
|
|
997
|
+
provider: row.provider,
|
|
998
|
+
reason: row.reason,
|
|
999
|
+
headBefore: row.headBefore,
|
|
1000
|
+
turnId: row.turnId,
|
|
1001
|
+
fencedAt: row.fencedAt,
|
|
1002
|
+
}))
|
|
1003
|
+
}
|
|
1004
|
+
|
|
1005
|
+
/**
|
|
1006
|
+
* The operator release. THE ONLY WAY A FENCE LEAVES THIS MAP.
|
|
1007
|
+
*
|
|
1008
|
+
* Addressed by digest, never by raw target key. Returns false when no fence
|
|
1009
|
+
* matches, so a stale handle reports honestly instead of silently succeeding.
|
|
1010
|
+
*/
|
|
1011
|
+
releaseFence(targetDigest: string): ReleaseOutcome {
|
|
1012
|
+
for (const [key, row] of this.fences) {
|
|
1013
|
+
// THE authority on which fence a handle names. The route also looks the row
|
|
1014
|
+
// up for its preview, but the release decision is made here — a duplicate
|
|
1015
|
+
// lookup upstream would leave this comparison enforced by nothing.
|
|
1016
|
+
if (opaqueRevision(row.targetKey) !== targetDigest) continue
|
|
1017
|
+
// PERSIST FIRST. Reporting a release that was not durably recorded is how an
|
|
1018
|
+
// operator is told a thread is open, writes to it, and finds it fenced again
|
|
1019
|
+
// after the next restart with no record of why.
|
|
1020
|
+
const remaining = [...this.fences.values()].filter(r => r.targetKey !== key)
|
|
1021
|
+
if (!this.persistFences(remaining)) return { ok: false, reason: 'persist_failed' }
|
|
1022
|
+
this.fences.delete(key)
|
|
1023
|
+
return { ok: true, row }
|
|
1024
|
+
}
|
|
1025
|
+
return { ok: false, reason: 'unknown_fence' }
|
|
878
1026
|
}
|
|
879
1027
|
|
|
880
1028
|
/**
|
|
@@ -1135,7 +1283,7 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1135
1283
|
const canListBindings = bindingDepsUsable(deps)
|
|
1136
1284
|
const canWriteBindings = bindingWriteDepsUsable(deps)
|
|
1137
1285
|
const detect = deps?.occupancy ?? threadOccupancy
|
|
1138
|
-
const guard = new TargetGuard()
|
|
1286
|
+
const guard = new TargetGuard(deps.fencePersistence ?? null)
|
|
1139
1287
|
const ownership = deps?.ownership ?? { record: recordCosSpawn, release: releaseCosSpawn }
|
|
1140
1288
|
// One per router. Injectable so the follow-on (attach accepting a `forkRef`)
|
|
1141
1289
|
// shares this instance rather than standing up a second, disconnected one.
|
|
@@ -1216,6 +1364,59 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1216
1364
|
}
|
|
1217
1365
|
}
|
|
1218
1366
|
|
|
1367
|
+
// ------------------------------------------------------------------ fences
|
|
1368
|
+
//
|
|
1369
|
+
// Mounted BEFORE the parameterised routes so `fences` can never be read as a
|
|
1370
|
+
// provider. Both are operator surfaces: a fenced thread was previously
|
|
1371
|
+
// discoverable only by trying to use it and being refused.
|
|
1372
|
+
//
|
|
1373
|
+
// A fence is addressed by DIGEST. The raw target key embeds the private native
|
|
1374
|
+
// thread id, and the redaction contract at the top of this file is absolute.
|
|
1375
|
+
|
|
1376
|
+
router.get('/agent-sessions/fences', (_req, res) => {
|
|
1377
|
+
// A fence list is a liveness answer; a cached one is worse than none.
|
|
1378
|
+
res.set('Cache-Control', 'private, no-store')
|
|
1379
|
+
// `degraded` is reported, never inferred: a memory-only fence set behaves
|
|
1380
|
+
// identically to a durable one until the process restarts, so a silent
|
|
1381
|
+
// fallback would be indistinguishable from working.
|
|
1382
|
+
res.json({ fences: guard.listFences(), degraded: guard.degraded() })
|
|
1383
|
+
})
|
|
1384
|
+
|
|
1385
|
+
router.post('/agent-sessions/fences/release', (req, res) => {
|
|
1386
|
+
const body = (req.body ?? {}) as { target?: unknown; confirm?: unknown }
|
|
1387
|
+
const target = body.target
|
|
1388
|
+
if (typeof target !== 'string' || target.length === 0 || target.length > 256) {
|
|
1389
|
+
res.status(400).json({ released: false, reason: 'invalid_request' })
|
|
1390
|
+
return
|
|
1391
|
+
}
|
|
1392
|
+
res.set('Cache-Control', 'private, no-store')
|
|
1393
|
+
if (body.confirm !== true) {
|
|
1394
|
+
// FAILS CLOSED, like every other destructive COS call. NOTE WHAT THIS IS
|
|
1395
|
+
// AND IS NOT: it is a deliberate second call, not proof a human looked.
|
|
1396
|
+
// The API token is shared by the phone, the lens and every COS agent
|
|
1397
|
+
// session, so nothing is structurally prevented from asserting `confirm`.
|
|
1398
|
+
// It stops an accidental release, not an automated one.
|
|
1399
|
+
const preview = guard.listFences().find(f => f.target === target) ?? null
|
|
1400
|
+
res.status(400).json({ released: false, reason: 'confirmation_required', preview })
|
|
1401
|
+
return
|
|
1402
|
+
}
|
|
1403
|
+
// The guard decides. It re-matches the handle itself rather than trusting a
|
|
1404
|
+
// lookup performed up here, and it persists BEFORE it mutates.
|
|
1405
|
+
const outcome = guard.releaseFence(target)
|
|
1406
|
+
if (!outcome.ok) {
|
|
1407
|
+
const status = outcome.reason === 'unknown_fence' ? 404 : 500
|
|
1408
|
+
res.status(status).json({ released: false, reason: outcome.reason })
|
|
1409
|
+
return
|
|
1410
|
+
}
|
|
1411
|
+
// THE EVIDENCE DIES WITH THE ROW. `releaseFence` deletes it, and the realistic
|
|
1412
|
+
// sequence is: first fence ever lands -> Control's card appears -> it is
|
|
1413
|
+
// released -> the distribution this evidence exists to collect is gone. So the
|
|
1414
|
+
// release line carries the whole record, not just its identity.
|
|
1415
|
+
const ev = outcome.row
|
|
1416
|
+
console.warn(`[agent-session-bindings] fence RELEASED by operator target=${target} provider=${ev.provider} fencedAt=${ev.fencedAt} fenceSite=${ev.fenceSite ?? 'unknown'} adapterReason=${ev.adapterReason ?? 'unknown'} detail=${ev.adapterDetail ?? 'none'} exitCode=${ev.exitCode ?? 'null'} childReaped=${ev.childReaped ?? 'unknown'} stderrClass=${ev.stderrClass ?? 'none'} durationMs=${ev.durationMs ?? 'unknown'} spawnCount=${ev.spawns?.length ?? 0}`)
|
|
1417
|
+
res.json({ released: true, target, provider: outcome.row.provider })
|
|
1418
|
+
})
|
|
1419
|
+
|
|
1219
1420
|
router.get('/agent-sessions/:provider/:threadId/attachability', (req, res) => {
|
|
1220
1421
|
// An occupancy verdict is a liveness answer with a lifetime of roughly now.
|
|
1221
1422
|
// A cached `attachable: true` is indistinguishable from a stale one, which is
|
|
@@ -1375,7 +1576,10 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1375
1576
|
// again just because the binding that delivered it is gone. Checked here as
|
|
1376
1577
|
// well as in the turn route, because a fresh attach is the obvious way around
|
|
1377
1578
|
// a per-binding fence.
|
|
1378
|
-
if (fenced !== null)
|
|
1579
|
+
if (fenced !== null) {
|
|
1580
|
+
console.warn(`[agent-session-bindings] fence hit route=attach provider=${providerParam} target=${opaqueRevision(key)}`)
|
|
1581
|
+
return refuseAttach(res, fenced)
|
|
1582
|
+
}
|
|
1379
1583
|
|
|
1380
1584
|
const resolve = deps.resolveTarget
|
|
1381
1585
|
if (typeof resolve !== 'function') return refuseAttach(res, 'target_unresolvable')
|
|
@@ -1623,8 +1827,24 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1623
1827
|
const turnId = mintId()
|
|
1624
1828
|
/** The target we hold a claim on, released in the finally. */
|
|
1625
1829
|
let claimedKey: string | null = null
|
|
1830
|
+
// Hoisted so the CATCH site can fence with evidence. `binding` and `head` are
|
|
1831
|
+
// both declared inside the try, so neither is in scope where the route-error
|
|
1832
|
+
// fence is set — without these it would store a fence it can say nothing about.
|
|
1833
|
+
let fenceProvider = ''
|
|
1834
|
+
let preTurnHeadDigest: string | null = null
|
|
1626
1835
|
/** Children the adapter reported, released in the finally. */
|
|
1627
|
-
|
|
1836
|
+
// PAIRS, not bare pids. A pid alone cannot be told apart from a recycled one,
|
|
1837
|
+
// and `startMs` is measured in the onSpawn closure and was previously discarded.
|
|
1838
|
+
const recordedPids: Array<{ pid: number; startMs: number }> = []
|
|
1839
|
+
// Hoisted so BOTH fence sites can record what the adapter actually reported.
|
|
1840
|
+
// `result` is scoped inside the try; the ambiguous site sits after the catch.
|
|
1841
|
+
// NARROW ON PURPOSE. `Partial<FenceEvidence>` would permit provider/headBefore/
|
|
1842
|
+
// turnId/bindingId/now, and the spread sits AFTER them at both fence sites — so
|
|
1843
|
+
// a future field could silently overwrite the fence's identity. The adapter
|
|
1844
|
+
// result literally carries a `provider`, which would write null and fail
|
|
1845
|
+
// `isFenceRecord` on the next read, silently un-enforcing the fence.
|
|
1846
|
+
let adapterEvidence: Partial<Pick<FenceEvidence,
|
|
1847
|
+
'adapterReason' | 'adapterDetail' | 'exitCode' | 'childReaped' | 'stderrClass' | 'durationMs'>> = {}
|
|
1628
1848
|
let pinnedBindingId: string | null = null
|
|
1629
1849
|
let requestNow = 0
|
|
1630
1850
|
/**
|
|
@@ -1789,6 +2009,7 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1789
2009
|
if (gate?.ok !== true) return refuseTurn(registryRefusal(gate?.reason))
|
|
1790
2010
|
|
|
1791
2011
|
const binding = deps.bindings.get!(bindingId)
|
|
2012
|
+
if (binding) fenceProvider = binding.provider
|
|
1792
2013
|
// Only `active` runs work. `staging` is the pre-commit state of the journaled
|
|
1793
2014
|
// Chat handoff and must never execute against a Chat that can still roll back.
|
|
1794
2015
|
const usable = assertUsable(binding ?? null, now)
|
|
@@ -1805,6 +2026,7 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1805
2026
|
const key = binding.targetKey
|
|
1806
2027
|
const fenced = guard.fencedReason(key)
|
|
1807
2028
|
if (fenced !== null) {
|
|
2029
|
+
console.warn(`[agent-session-bindings] fence hit route=turn provider=${binding.provider} target=${opaqueRevision(key)} turnId=${turnId}`)
|
|
1808
2030
|
return refuseTurn(fenced, { retryable: false, deliveryState: 'unknown' })
|
|
1809
2031
|
}
|
|
1810
2032
|
|
|
@@ -1823,6 +2045,7 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1823
2045
|
|
|
1824
2046
|
const head = await readHead(binding.provider, binding.nativeThreadId)
|
|
1825
2047
|
if (head === null) return refuseTurn('native_head_unavailable')
|
|
2048
|
+
preTurnHeadDigest = head.digest
|
|
1826
2049
|
|
|
1827
2050
|
// The attach baseline, advanced by each completed turn and by each explicit
|
|
1828
2051
|
// Continue Anyway. Without the advance the SECOND turn on a binding always
|
|
@@ -1907,10 +2130,35 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1907
2130
|
// before the prompt rather than deliver a turn that poisons the next
|
|
1908
2131
|
// occupancy check.
|
|
1909
2132
|
if (outcome !== 'recorded') return false
|
|
1910
|
-
recordedPids.push(pid)
|
|
2133
|
+
recordedPids.push({ pid, startMs })
|
|
1911
2134
|
return true
|
|
1912
2135
|
},
|
|
1913
2136
|
})
|
|
2137
|
+
// Read the adapter's OWN account before it goes out of scope. This is the
|
|
2138
|
+
// whole point of the evidence work: `classifyDelivery` collapses six distinct
|
|
2139
|
+
// failures into one word, and the difference between them is what decides
|
|
2140
|
+
// whether an automatic resolver could ever be safe. A 21-minute `timeout`
|
|
2141
|
+
// means the child ran tool calls; a `provider_exit_nonzero` with a code means
|
|
2142
|
+
// it exited on its own. Defensive reads — the adapter is injected in tests.
|
|
2143
|
+
// AN UNREADABLE RESULT RECORDS NOTHING, not zeroes. Reading `{}` and
|
|
2144
|
+
// deriving `exitCode: null, childReaped: false` states two facts about a
|
|
2145
|
+
// child nothing is known about, and writes them indistinguishably from a
|
|
2146
|
+
// confirmed-unreaped timeout — corrupting the one discriminator this
|
|
2147
|
+
// evidence exists to establish.
|
|
2148
|
+
const r = (result !== null && typeof result === 'object')
|
|
2149
|
+
? result as Record<string, unknown>
|
|
2150
|
+
: null
|
|
2151
|
+
adapterEvidence = r === null ? {} : {
|
|
2152
|
+
adapterReason: typeof r.reason === 'string' ? r.reason : (r.ok === true ? 'ok' : undefined),
|
|
2153
|
+
adapterDetail: typeof r.detail === 'string' ? r.detail : null,
|
|
2154
|
+
exitCode: typeof r.exitCode === 'number' ? r.exitCode : null,
|
|
2155
|
+
// FROM THE ADAPTER, never derived from exitCode. A signal-killed child
|
|
2156
|
+
// reports `code === null`, so deriving it would report "never reaped" for
|
|
2157
|
+
// the dominant timeout shape — backwards for the decision this informs.
|
|
2158
|
+
childReaped: typeof r.reaped === 'boolean' ? r.reaped : undefined,
|
|
2159
|
+
stderrClass: typeof r.stderrClass === 'string' ? r.stderrClass : undefined,
|
|
2160
|
+
durationMs: typeof r.durationMs === 'number' ? r.durationMs : undefined,
|
|
2161
|
+
}
|
|
1914
2162
|
delivery = classifyDelivery(result)
|
|
1915
2163
|
} catch (error) {
|
|
1916
2164
|
console.error(`[agent-session-bindings] adapter threw: ${error instanceof Error ? error.message : error}`)
|
|
@@ -1928,7 +2176,30 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1928
2176
|
// Fenced under its own reason, not this turn's: `delivery_ambiguous`
|
|
1929
2177
|
// describes what happened to THIS request, while a later caller needs to
|
|
1930
2178
|
// be told the thread is shut and why it must be inspected first.
|
|
1931
|
-
|
|
2179
|
+
// ONE resolved value for the record AND the log. The previous release
|
|
2180
|
+
// fixed exactly this contradiction at the other fence site and this one
|
|
2181
|
+
// re-committed it: the record said `unreadable_result` while the log said
|
|
2182
|
+
// `unknown`, so an operator grepping for the sentinel found nothing.
|
|
2183
|
+
const recordedReason = adapterEvidence.adapterReason ?? 'unreadable_result'
|
|
2184
|
+
guard.fence(key, 'native_target_fenced', {
|
|
2185
|
+
provider: binding.provider,
|
|
2186
|
+
headBefore: head.digest,
|
|
2187
|
+
turnId,
|
|
2188
|
+
bindingId,
|
|
2189
|
+
now: Date.now(),
|
|
2190
|
+
...adapterEvidence,
|
|
2191
|
+
// NEVER blank. An unreadable adapter result and "nobody recorded it" are
|
|
2192
|
+
// different facts and a missing field cannot tell them apart -- which is
|
|
2193
|
+
// the whole reason this evidence exists.
|
|
2194
|
+
adapterReason: recordedReason,
|
|
2195
|
+
fenceSite: 'ambiguous',
|
|
2196
|
+
spawns: [...recordedPids],
|
|
2197
|
+
})
|
|
2198
|
+
// A fence shuts a thread until a human acts, and until now it wrote NO log
|
|
2199
|
+
// line at either site — so a fenced thread was discoverable only by trying
|
|
2200
|
+
// to use it (Miles, 2026-08-18). Never log `key`: it embeds the private
|
|
2201
|
+
// native thread id, which this router does not emit anywhere.
|
|
2202
|
+
console.warn(`[agent-session-bindings] fence set site=ambiguous provider=${binding.provider} target=${opaqueRevision(key)} turnId=${turnId} bindingId=${bindingId} headBefore=${head.digest} adapterReason=${recordedReason} detail=${adapterEvidence.adapterDetail ?? 'none'} exitCode=${adapterEvidence.exitCode ?? 'null'} childReaped=${adapterEvidence.childReaped ?? 'unknown'} stderrClass=${adapterEvidence.stderrClass ?? 'none'} durationMs=${adapterEvidence.durationMs ?? 'unknown'} spawnCount=${recordedPids.length}`)
|
|
1932
2203
|
return reportAmbiguous()
|
|
1933
2204
|
}
|
|
1934
2205
|
|
|
@@ -1958,13 +2229,35 @@ export function createAgentSessionBindingsRouter(deps: AgentSessionBindingsDeps)
|
|
|
1958
2229
|
if (deliveryAttempted) {
|
|
1959
2230
|
// A bug in this route that happened AROUND a delivery is indistinguishable
|
|
1960
2231
|
// from a delivery.
|
|
1961
|
-
if (claimedKey !== null)
|
|
2232
|
+
if (claimedKey !== null) {
|
|
2233
|
+
guard.fence(claimedKey, 'native_target_fenced', {
|
|
2234
|
+
provider: fenceProvider,
|
|
2235
|
+
headBefore: preTurnHeadDigest,
|
|
2236
|
+
turnId,
|
|
2237
|
+
bindingId: null,
|
|
2238
|
+
now: Date.now(),
|
|
2239
|
+
// A bug in THIS route, not a provider outcome. Named rather than left
|
|
2240
|
+
// blank so a later reader can tell "the adapter reported nothing" apart
|
|
2241
|
+
// from "nobody recorded it". Any adapter evidence captured before the
|
|
2242
|
+
// throw is still carried.
|
|
2243
|
+
...adapterEvidence,
|
|
2244
|
+
adapterReason: adapterEvidence.adapterReason ?? 'route_error',
|
|
2245
|
+
fenceSite: 'route_error',
|
|
2246
|
+
spawns: [...recordedPids],
|
|
2247
|
+
})
|
|
2248
|
+
// `head` is scoped to the try, so `preTurnHeadDigest` is hoisted to the
|
|
2249
|
+
// handler specifically to reach this site. It is null ONLY when the throw
|
|
2250
|
+
// happened before the head was read. An earlier version of this line
|
|
2251
|
+
// hardcoded `unavailable` and so contradicted the record it had just
|
|
2252
|
+
// written — an operator would read "no baseline" off a fence that has one.
|
|
2253
|
+
console.warn(`[agent-session-bindings] fence set site=route_error target=${opaqueRevision(claimedKey)} turnId=${turnId} headBefore=${preTurnHeadDigest ?? 'unavailable'} adapterReason=${adapterEvidence.adapterReason ?? 'route_error'} detail=${adapterEvidence.adapterDetail ?? 'none'} exitCode=${adapterEvidence.exitCode ?? 'null'} childReaped=${adapterEvidence.childReaped ?? 'unknown'} stderrClass=${adapterEvidence.stderrClass ?? 'none'} durationMs=${adapterEvidence.durationMs ?? 'unknown'} spawnCount=${recordedPids.length}`)
|
|
2254
|
+
}
|
|
1962
2255
|
reportAmbiguous()
|
|
1963
2256
|
} else {
|
|
1964
2257
|
refuseTurn('turn_failed')
|
|
1965
2258
|
}
|
|
1966
2259
|
} finally {
|
|
1967
|
-
for (const pid of recordedPids) {
|
|
2260
|
+
for (const { pid } of recordedPids) {
|
|
1968
2261
|
try {
|
|
1969
2262
|
ownership.release(pid)
|
|
1970
2263
|
} catch (error) {
|