instar 1.3.651 → 1.3.653
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +403 -1
- package/dist/commands/server.js.map +1 -1
- package/dist/core/ReapGuard.d.ts +13 -1
- package/dist/core/ReapGuard.d.ts.map +1 -1
- package/dist/core/ReapGuard.js +86 -30
- package/dist/core/ReapGuard.js.map +1 -1
- package/dist/core/SessionManager.d.ts +11 -0
- package/dist/core/SessionManager.d.ts.map +1 -1
- package/dist/core/SessionManager.js +30 -4
- package/dist/core/SessionManager.js.map +1 -1
- package/dist/core/devGatedFeatures.d.ts.map +1 -1
- package/dist/core/devGatedFeatures.js +12 -0
- package/dist/core/devGatedFeatures.js.map +1 -1
- package/dist/core/gapBCommitmentEvidence.d.ts +9 -0
- package/dist/core/gapBCommitmentEvidence.d.ts.map +1 -1
- package/dist/core/gapBCommitmentEvidence.js +20 -0
- package/dist/core/gapBCommitmentEvidence.js.map +1 -1
- package/dist/core/ownershipFollowsLiveWork.d.ts +68 -0
- package/dist/core/ownershipFollowsLiveWork.d.ts.map +1 -0
- package/dist/core/ownershipFollowsLiveWork.js +95 -0
- package/dist/core/ownershipFollowsLiveWork.js.map +1 -0
- package/dist/core/types.d.ts +26 -0
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/messaging/stuckMessageRecovery.d.ts +12 -0
- package/dist/messaging/stuckMessageRecovery.d.ts.map +1 -1
- package/dist/messaging/stuckMessageRecovery.js +12 -0
- package/dist/messaging/stuckMessageRecovery.js.map +1 -1
- package/dist/monitoring/SessionReaper.d.ts +118 -7
- package/dist/monitoring/SessionReaper.d.ts.map +1 -1
- package/dist/monitoring/SessionReaper.js +302 -101
- package/dist/monitoring/SessionReaper.js.map +1 -1
- package/dist/monitoring/SessionRecovery.d.ts +73 -0
- package/dist/monitoring/SessionRecovery.d.ts.map +1 -1
- package/dist/monitoring/SessionRecovery.js +113 -0
- package/dist/monitoring/SessionRecovery.js.map +1 -1
- package/dist/monitoring/closeoutLivenessSnapshot.d.ts +143 -0
- package/dist/monitoring/closeoutLivenessSnapshot.d.ts.map +1 -0
- package/dist/monitoring/closeoutLivenessSnapshot.js +207 -0
- package/dist/monitoring/closeoutLivenessSnapshot.js.map +1 -0
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +3 -3
- package/upgrades/1.3.652.md +18 -0
- package/upgrades/1.3.653.md +85 -0
- package/upgrades/side-effects/ownership-follows-live-work.md +137 -0
- package/upgrades/side-effects/post-transfer-closeout-correctness.md +43 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"closeoutLivenessSnapshot.d.ts","sourceRoot":"","sources":["../../src/monitoring/closeoutLivenessSnapshot.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6BG;AAEH,kFAAkF;AAClF,MAAM,WAAW,eAAe;IAC9B;wFACoF;IACpF,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,UAAU,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IAC7B,2EAA2E;IAC3E,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED;8EAC8E;AAC9E,MAAM,WAAW,iBAAiB;IAChC,MAAM,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;IACpB;qFACiF;IACjF,WAAW,EAAE,MAAM,CAAC;CACrB;AAED;;;oBAGoB;AACpB,MAAM,WAAW,cAAc;IAC7B,KAAK,EAAE,OAAO,GAAG,SAAS,CAAC;IAC3B,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,mDAAmD;AACnD,MAAM,WAAW,OAAO;IACtB,SAAS,EAAE,MAAM,CAAC;IAClB,GAAG,EAAE,MAAM,CAAC;CACb;AAED,MAAM,WAAW,cAAc;IAC7B,2DAA2D;IAC3D,eAAe,EAAE,MAAM,CAAC;IACxB;kFAC8E;IAC9E,wBAAwB,EAAE,MAAM,CAAC;CAClC;AAED,eAAO,MAAM,uBAAuB,EAAE,cAGrC,CAAC;AAEF,MAAM,WAAW,YAAY;IAC3B;;8EAE0E;IAC1E,eAAe,EAAE,MAAM,OAAO,EAAE,CAAC;IACjC;;;wDAGoD;IACpD,iBAAiB,EAAE,CAAC,IAAI,EAAE,OAAO,KAAK,OAAO,CAAC,eAAe,EAAE,CAAC,CAAC;IACjE;;;sBAGkB;IAClB,QAAQ,EAAE,MAAM,MAAM,EAAE,CAAC;IACzB,2BAA2B;IAC3B,GAAG,EAAE,MAAM,MAAM,CAAC;IAClB,uEAAuE;IACvE,cAAc,CAAC,EAAE,CAAC,IAAI,EAAE;QAAE,EAAE,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,WAAW,CAAC,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;IACtG,2CAA2C;IAC3C,KAAK,CAAC,EAAE,CAAC,KAAK,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,IAAI,CAAC;CAClD;AAED;;;;GAIG;AACH,wBAAgB,kBAAkB,CAAC,CAAC,EAAE,eAAe,GAAG,MAAM,GAAG,IAAI,CAMpE;AAWD,wBAAgB,qBAAqB,CAAC,CAAC,EAAE,eAAe,GAAG,OAAO,CAEjE;AAED;;;;;;GAMG;AACH,qBAAa,wBAAwB;IACnC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAiB;IACrC,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAe;IACpC;yFACqF;IACrF,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAwC;IACjE;oDACgD;IAChD,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAqB;IACnD,0EAA0E;IAC1E,OAAO,CAAC,oBAAoB,CAAK;IACjC,+EAA+E;IAC/E,OAAO,CAAC,aAAa,CAAS;gBAElB,IAAI,EAAE,YAAY,EAAE,GAAG,CAAC,EAAE,OAAO,CAAC,cAAc,CAAC;IAK7D,wDAAwD;IACxD,OAAO,KAAK,gBAAgB,GAE3B;IAED;;;;;OAKG;IACH,yBAAyB,GAAI,SAAS,MAAM,EAAE,gBAAgB,MAAM,KAAG,cAAc,CAmBnF;IAEF;;;OAGG;IACG,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;IAyE9B,2EAA2E;IAC3E,IAAI,CAAC,SAAS,EAAE,MAAM,GAAG,iBAAiB,GAAG,SAAS;IAItD,uEAAuE;IACvE,IAAI,YAAY,IAAI,OAAO,CAE1B;CACF"}
|
|
@@ -0,0 +1,207 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* closeoutLivenessSnapshot.ts — the machine-local liveness snapshot that backs
|
|
3
|
+
* the post-transfer closeout's liveness gate (Part C of
|
|
4
|
+
* docs/specs/post-transfer-closeout-correctness.md).
|
|
5
|
+
*
|
|
6
|
+
* THE problem it solves: SessionReaper's post-transfer closeout could terminate
|
|
7
|
+
* the LIVE local session for a topic when the local ownership record was STALE —
|
|
8
|
+
* because nothing verified the remote owner ACTUALLY has a live session for the
|
|
9
|
+
* topic. This module is the verification: a periodically-refreshed local snapshot
|
|
10
|
+
* of "which topics have a live session on each peer", built FROM the SAME per-peer
|
|
11
|
+
* `GET /sessions` fan-out the pool route already performs (no new endpoint), fed
|
|
12
|
+
* to the reaper as a synchronous `remoteOwnerHasLiveSession(topicId, machineId)`
|
|
13
|
+
* dep that returns `true | false | 'unknown'`.
|
|
14
|
+
*
|
|
15
|
+
* Fail-closed everywhere: a missing/stale snapshot, an unreachable peer, or any
|
|
16
|
+
* error → `'unknown'` → the reaper WITHHOLDS the closeout (never kills the live
|
|
17
|
+
* worker). An EMPTY topics-set on a FRESH, successfully-reached peer is a
|
|
18
|
+
* definitive `false` (the stale-owner signal), NOT `'unknown'` — freshness means
|
|
19
|
+
* "reached at reachableAt ≤ bound", never "non-empty".
|
|
20
|
+
*
|
|
21
|
+
* Machine-local by design: every machine runs its OWN closeout against its OWN
|
|
22
|
+
* locally-built snapshot. It is NOT replicated and NOT proxied (the closeout
|
|
23
|
+
* decision is inherently local: "should *I* shed *my* leftover?").
|
|
24
|
+
*
|
|
25
|
+
* No Unbounded Loops: the refresher is LEVEL-triggered on a fixed (jittered)
|
|
26
|
+
* cadence with a per-attempt 5s timeout, per-pass eviction of departed peers, and
|
|
27
|
+
* an OBSERVABILITY breaker (a consecutive-all-failed counter that SURFACES the
|
|
28
|
+
* degraded condition rather than STOPPING the loop — stopping would freeze the
|
|
29
|
+
* snapshot at `'unknown'` forever, the wrong direction for a safe-failure loop).
|
|
30
|
+
*/
|
|
31
|
+
export const DEFAULT_SNAPSHOT_CONFIG = {
|
|
32
|
+
tickIntervalSec: 120,
|
|
33
|
+
snapshotBreakerThreshold: 5,
|
|
34
|
+
};
|
|
35
|
+
/**
|
|
36
|
+
* Extract the bound topic id from a peer session object, or null. Mirrors the
|
|
37
|
+
* pool `/sessions` route: a telegram-bound session carries `platform:'telegram'`
|
|
38
|
+
* + `platformId:<topic>`.
|
|
39
|
+
*/
|
|
40
|
+
export function topicOfPeerSession(s) {
|
|
41
|
+
if (s.platform !== 'telegram')
|
|
42
|
+
return null;
|
|
43
|
+
const raw = s.platformId;
|
|
44
|
+
if (typeof raw === 'number' && Number.isFinite(raw))
|
|
45
|
+
return raw;
|
|
46
|
+
if (typeof raw === 'string' && /^\d+$/.test(raw))
|
|
47
|
+
return Number(raw);
|
|
48
|
+
return null;
|
|
49
|
+
}
|
|
50
|
+
/**
|
|
51
|
+
* THE ONE NORMATIVE liveness predicate (spec "Peer `/sessions` liveness
|
|
52
|
+
* contract"): a topic counts as live-on-a-peer when that peer's `/sessions` lists
|
|
53
|
+
* a session bound to the topic that is NOT explicitly terminal/shutting-down. An
|
|
54
|
+
* entry whose state cannot be classified counts as LISTED (opaque is NOT dead) —
|
|
55
|
+
* so we EXCLUDE only the clearly-terminal states.
|
|
56
|
+
*/
|
|
57
|
+
const TERMINAL_STATUSES = new Set(['completed', 'failed', 'killed']);
|
|
58
|
+
export function isTerminalPeerSession(s) {
|
|
59
|
+
return typeof s.status === 'string' && TERMINAL_STATUSES.has(s.status);
|
|
60
|
+
}
|
|
61
|
+
/**
|
|
62
|
+
* The machine-local liveness snapshot + its bounded refresher.
|
|
63
|
+
*
|
|
64
|
+
* Construct/start ONLY when `closeoutLivenessGate` resolves true — when the gate
|
|
65
|
+
* is off this is never built and the reaper's `remoteOwnerHasLiveSession` dep is
|
|
66
|
+
* left absent (the closeout keeps today's behavior).
|
|
67
|
+
*/
|
|
68
|
+
export class CloseoutLivenessSnapshot {
|
|
69
|
+
cfg;
|
|
70
|
+
deps;
|
|
71
|
+
/** machineId → { topics, reachableAt }. Bounded O(pool machines): evicted of
|
|
72
|
+
* departed peers each pass; the topics set is bounded by a peer's session count. */
|
|
73
|
+
snapshot = new Map();
|
|
74
|
+
/** Owners discovered mid-tick (closeout saw an owner not yet covered) — fed to
|
|
75
|
+
* the NEXT refresh, never the current read. */
|
|
76
|
+
pendingOwners = new Set();
|
|
77
|
+
/** Consecutive refresh passes where EVERY attempted peer fetch failed. */
|
|
78
|
+
consecutiveAllFailed = 0;
|
|
79
|
+
/** Whether the breaker attention item has already been raised this episode. */
|
|
80
|
+
breakerRaised = false;
|
|
81
|
+
constructor(deps, cfg) {
|
|
82
|
+
this.deps = deps;
|
|
83
|
+
this.cfg = { ...DEFAULT_SNAPSHOT_CONFIG, ...(cfg ?? {}) };
|
|
84
|
+
}
|
|
85
|
+
/** The staleness bound: 2× the refresh cadence (ms). */
|
|
86
|
+
get stalenessBoundMs() {
|
|
87
|
+
return 2 * this.cfg.tickIntervalSec * 1000;
|
|
88
|
+
}
|
|
89
|
+
/**
|
|
90
|
+
* The reaper's synchronous liveness dep. Reads ONLY the already-populated
|
|
91
|
+
* snapshot (never a fetch it triggers). On the FIRST tick a topic's owner is
|
|
92
|
+
* not yet covered, this returns `'unknown'` (→ WITHHOLD) AND enqueues the owner
|
|
93
|
+
* for the next refresh pass.
|
|
94
|
+
*/
|
|
95
|
+
remoteOwnerHasLiveSession = (topicId, ownerMachineId) => {
|
|
96
|
+
try {
|
|
97
|
+
const entry = this.snapshot.get(ownerMachineId);
|
|
98
|
+
if (!entry) {
|
|
99
|
+
// Owner not yet covered — discover it for the NEXT pass, withhold now.
|
|
100
|
+
this.pendingOwners.add(ownerMachineId);
|
|
101
|
+
return { state: 'unknown' };
|
|
102
|
+
}
|
|
103
|
+
const age = this.deps.now() - entry.reachableAt;
|
|
104
|
+
if (!(age <= this.stalenessBoundMs)) {
|
|
105
|
+
// Stale — the peer was unreachable / timed out at the last refresh.
|
|
106
|
+
return { state: 'unknown' };
|
|
107
|
+
}
|
|
108
|
+
// FRESH: an empty topics set is a definitive `false` (reached, zero
|
|
109
|
+
// sessions = the stale-owner signal), never 'unknown'.
|
|
110
|
+
return { state: entry.topics.has(topicId), reachableAt: entry.reachableAt };
|
|
111
|
+
}
|
|
112
|
+
catch {
|
|
113
|
+
return { state: 'unknown' };
|
|
114
|
+
}
|
|
115
|
+
};
|
|
116
|
+
/**
|
|
117
|
+
* One refresh pass — owner-scoped, bounded fan-out. Called on the reaper
|
|
118
|
+
* cadence. Returns the count of peers fetched (for observability/tests).
|
|
119
|
+
*/
|
|
120
|
+
async refresh() {
|
|
121
|
+
let owners;
|
|
122
|
+
try {
|
|
123
|
+
owners = new Set([...this.deps.ownerSet(), ...this.pendingOwners]);
|
|
124
|
+
}
|
|
125
|
+
catch {
|
|
126
|
+
owners = new Set(this.pendingOwners);
|
|
127
|
+
}
|
|
128
|
+
this.pendingOwners.clear();
|
|
129
|
+
// Intersect with registered online peers (only fetch owners we can reach),
|
|
130
|
+
// and evict snapshot entries for peers no longer in the pool.
|
|
131
|
+
let registered = [];
|
|
132
|
+
try {
|
|
133
|
+
registered = this.deps.resolvePeerUrls();
|
|
134
|
+
}
|
|
135
|
+
catch {
|
|
136
|
+
registered = [];
|
|
137
|
+
}
|
|
138
|
+
const registeredById = new Map(registered.map(p => [p.machineId, p]));
|
|
139
|
+
// Eviction (bounded map growth): drop entries for departed peers.
|
|
140
|
+
for (const mid of [...this.snapshot.keys()]) {
|
|
141
|
+
if (!registeredById.has(mid))
|
|
142
|
+
this.snapshot.delete(mid);
|
|
143
|
+
}
|
|
144
|
+
const toFetch = [];
|
|
145
|
+
for (const mid of owners) {
|
|
146
|
+
const peer = registeredById.get(mid);
|
|
147
|
+
if (peer)
|
|
148
|
+
toFetch.push(peer);
|
|
149
|
+
}
|
|
150
|
+
if (toFetch.length === 0) {
|
|
151
|
+
// No owned-elsewhere leftovers reachable this pass → no HTTP, no breaker move.
|
|
152
|
+
return;
|
|
153
|
+
}
|
|
154
|
+
let anyOk = false;
|
|
155
|
+
await Promise.all(toFetch.map(async (peer) => {
|
|
156
|
+
try {
|
|
157
|
+
const list = await this.deps.fetchPeerSessions(peer);
|
|
158
|
+
const topics = new Set();
|
|
159
|
+
for (const s of list) {
|
|
160
|
+
if (isTerminalPeerSession(s))
|
|
161
|
+
continue; // exclude only clearly-terminal
|
|
162
|
+
const t = topicOfPeerSession(s);
|
|
163
|
+
if (t != null)
|
|
164
|
+
topics.add(t);
|
|
165
|
+
}
|
|
166
|
+
// A FRESH reached peer with an EMPTY set is recorded (empty ≠ unknown).
|
|
167
|
+
this.snapshot.set(peer.machineId, { topics, reachableAt: this.deps.now() });
|
|
168
|
+
anyOk = true;
|
|
169
|
+
}
|
|
170
|
+
catch {
|
|
171
|
+
// Failed fetch: do NOT update reachableAt — the entry ages into stale →
|
|
172
|
+
// 'unknown' → WITHHOLD (the safe direction). @silent-fallback-ok
|
|
173
|
+
}
|
|
174
|
+
}));
|
|
175
|
+
// Observability breaker — SURFACE the degraded condition, never STOP the loop.
|
|
176
|
+
if (anyOk) {
|
|
177
|
+
this.consecutiveAllFailed = 0;
|
|
178
|
+
this.breakerRaised = false;
|
|
179
|
+
}
|
|
180
|
+
else {
|
|
181
|
+
this.consecutiveAllFailed += 1;
|
|
182
|
+
if (this.consecutiveAllFailed >= this.cfg.snapshotBreakerThreshold && !this.breakerRaised) {
|
|
183
|
+
this.breakerRaised = true;
|
|
184
|
+
this.deps.audit?.({
|
|
185
|
+
kind: 'session-reaper',
|
|
186
|
+
event: 'closeout-snapshot-breaker',
|
|
187
|
+
consecutiveFailures: this.consecutiveAllFailed,
|
|
188
|
+
});
|
|
189
|
+
this.deps.raiseAttention?.({
|
|
190
|
+
id: 'closeout-snapshot-breaker',
|
|
191
|
+
title: 'Post-transfer liveness snapshot cannot reach any peer',
|
|
192
|
+
summary: `The liveness snapshot has failed to reach any peer for ${this.consecutiveAllFailed} passes — closeout is safely withholding all leftovers.`,
|
|
193
|
+
description: `The post-transfer closeout liveness snapshot could not reach ANY peer for ${this.consecutiveAllFailed} consecutive refresh passes. Closeout is safely WITHHOLDING every owned-elsewhere leftover (the fail-closed direction — no live worker is killed), but stale duplicate sessions will not be reclaimed until mesh connectivity recovers. Check mesh/peer connectivity.`,
|
|
194
|
+
});
|
|
195
|
+
}
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
/** Test/observability accessor — the current snapshot entry for a peer. */
|
|
199
|
+
peek(machineId) {
|
|
200
|
+
return this.snapshot.get(machineId);
|
|
201
|
+
}
|
|
202
|
+
/** Test/observability — whether the breaker has fired this episode. */
|
|
203
|
+
get breakerFired() {
|
|
204
|
+
return this.breakerRaised;
|
|
205
|
+
}
|
|
206
|
+
}
|
|
207
|
+
//# sourceMappingURL=closeoutLivenessSnapshot.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"closeoutLivenessSnapshot.js","sourceRoot":"","sources":["../../src/monitoring/closeoutLivenessSnapshot.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6BG;AA4CH,MAAM,CAAC,MAAM,uBAAuB,GAAmB;IACrD,eAAe,EAAE,GAAG;IACpB,wBAAwB,EAAE,CAAC;CAC5B,CAAC;AAyBF;;;;GAIG;AACH,MAAM,UAAU,kBAAkB,CAAC,CAAkB;IACnD,IAAI,CAAC,CAAC,QAAQ,KAAK,UAAU;QAAE,OAAO,IAAI,CAAC;IAC3C,MAAM,GAAG,GAAG,CAAC,CAAC,UAAU,CAAC;IACzB,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC;QAAE,OAAO,GAAG,CAAC;IAChE,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC;QAAE,OAAO,MAAM,CAAC,GAAG,CAAC,CAAC;IACrE,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;;;;GAMG;AACH,MAAM,iBAAiB,GAAG,IAAI,GAAG,CAAC,CAAC,WAAW,EAAE,QAAQ,EAAE,QAAQ,CAAC,CAAC,CAAC;AAErE,MAAM,UAAU,qBAAqB,CAAC,CAAkB;IACtD,OAAO,OAAO,CAAC,CAAC,MAAM,KAAK,QAAQ,IAAI,iBAAiB,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;AACzE,CAAC;AAED;;;;;;GAMG;AACH,MAAM,OAAO,wBAAwB;IAClB,GAAG,CAAiB;IACpB,IAAI,CAAe;IACpC;yFACqF;IACpE,QAAQ,GAAG,IAAI,GAAG,EAA6B,CAAC;IACjE;oDACgD;IAC/B,aAAa,GAAG,IAAI,GAAG,EAAU,CAAC;IACnD,0EAA0E;IAClE,oBAAoB,GAAG,CAAC,CAAC;IACjC,+EAA+E;IACvE,aAAa,GAAG,KAAK,CAAC;IAE9B,YAAY,IAAkB,EAAE,GAA6B;QAC3D,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;QACjB,IAAI,CAAC,GAAG,GAAG,EAAE,GAAG,uBAAuB,EAAE,GAAG,CAAC,GAAG,IAAI,EAAE,CAAC,EAAE,CAAC;IAC5D,CAAC;IAED,wDAAwD;IACxD,IAAY,gBAAgB;QAC1B,OAAO,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,eAAe,GAAG,IAAI,CAAC;IAC7C,CAAC;IAED;;;;;OAKG;IACH,yBAAyB,GAAG,CAAC,OAAe,EAAE,cAAsB,EAAkB,EAAE;QACtF,IAAI,CAAC;YACH,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC;YAChD,IAAI,CAAC,KAAK,EAAE,CAAC;gBACX,uEAAuE;gBACvE,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC;gBACvC,OAAO,EAAE,KAAK,EAAE,SAAS,EAAE,CAAC;YAC9B,CAAC;YACD,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,WAAW,CAAC;YAChD,IAAI,CAAC,CAAC,GAAG,IAAI,IAAI,CAAC,gBAAgB,CAAC,EAAE,CAAC;gBACpC,oEAAoE;gBACpE,OAAO,EAAE,KAAK,EAAE,SAAS,EAAE,CAAC;YAC9B,CAAC;YACD,oEAAoE;YACpE,uDAAuD;YACvD,OAAO,EAAE,KAAK,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,WAAW,EAAE,KAAK,CAAC,WAAW,EAAE,CAAC;QAC9E,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,EAAE,KAAK,EAAE,SAAS,EAAE,CAAC;QAC9B,CAAC;IACH,CAAC,CAAC;IAEF;;;OAGG;IACH,KAAK,CAAC,OAAO;QACX,IAAI,MAAmB,CAAC;QACxB,IAAI,CAAC;YACH,MAAM,GAAG,IAAI,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,EAAE,GAAG,IAAI,CAAC,aAAa,CAAC,CAAC,CAAC;QACrE,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,GAAG,IAAI,GAAG,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;QACvC,CAAC;QACD,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,CAAC;QAE3B,2EAA2E;QAC3E,8DAA8D;QAC9D,IAAI,UAAU,GAAc,EAAE,CAAC;QAC/B,IAAI,CAAC;YAAC,UAAU,GAAG,IAAI,CAAC,IAAI,CAAC,eAAe,EAAE,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC;YAAC,UAAU,GAAG,EAAE,CAAC;QAAC,CAAC;QAC5E,MAAM,cAAc,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;QAEtE,kEAAkE;QAClE,KAAK,MAAM,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC;YAC5C,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,GAAG,CAAC;gBAAE,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QAC1D,CAAC;QAED,MAAM,OAAO,GAAc,EAAE,CAAC;QAC9B,KAAK,MAAM,GAAG,IAAI,MAAM,EAAE,CAAC;YACzB,MAAM,IAAI,GAAG,cAAc,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;YACrC,IAAI,IAAI;gBAAE,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QAC/B,CAAC;QAED,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,+EAA+E;YAC/E,OAAO;QACT,CAAC;QAED,IAAI,KAAK,GAAG,KAAK,CAAC;QAClB,MAAM,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,KAAK,EAAE,IAAI,EAAE,EAAE;YAC3C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,iBAAiB,CAAC,IAAI,CAAC,CAAC;gBACrD,MAAM,MAAM,GAAG,IAAI,GAAG,EAAU,CAAC;gBACjC,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;oBACrB,IAAI,qBAAqB,CAAC,CAAC,CAAC;wBAAE,SAAS,CAAC,gCAAgC;oBACxE,MAAM,CAAC,GAAG,kBAAkB,CAAC,CAAC,CAAC,CAAC;oBAChC,IAAI,CAAC,IAAI,IAAI;wBAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;gBAC/B,CAAC;gBACD,wEAAwE;gBACxE,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,EAAE,EAAE,MAAM,EAAE,WAAW,EAAE,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC;gBAC5E,KAAK,GAAG,IAAI,CAAC;YACf,CAAC;YAAC,MAAM,CAAC;gBACP,wEAAwE;gBACxE,iEAAiE;YACnE,CAAC;QACH,CAAC,CAAC,CAAC,CAAC;QAEJ,+EAA+E;QAC/E,IAAI,KAAK,EAAE,CAAC;YACV,IAAI,CAAC,oBAAoB,GAAG,CAAC,CAAC;YAC9B,IAAI,CAAC,aAAa,GAAG,KAAK,CAAC;QAC7B,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,oBAAoB,IAAI,CAAC,CAAC;YAC/B,IAAI,IAAI,CAAC,oBAAoB,IAAI,IAAI,CAAC,GAAG,CAAC,wBAAwB,IAAI,CAAC,IAAI,CAAC,aAAa,EAAE,CAAC;gBAC1F,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC;gBAC1B,IAAI,CAAC,IAAI,CAAC,KAAK,EAAE,CAAC;oBAChB,IAAI,EAAE,gBAAgB;oBACtB,KAAK,EAAE,2BAA2B;oBAClC,mBAAmB,EAAE,IAAI,CAAC,oBAAoB;iBAC/C,CAAC,CAAC;gBACH,IAAI,CAAC,IAAI,CAAC,cAAc,EAAE,CAAC;oBACzB,EAAE,EAAE,2BAA2B;oBAC/B,KAAK,EAAE,uDAAuD;oBAC9D,OAAO,EAAE,0DAA0D,IAAI,CAAC,oBAAoB,yDAAyD;oBACrJ,WAAW,EAAE,6EAA6E,IAAI,CAAC,oBAAoB,uQAAuQ;iBAC3X,CAAC,CAAC;YACL,CAAC;QACH,CAAC;IACH,CAAC;IAED,2EAA2E;IAC3E,IAAI,CAAC,SAAiB;QACpB,OAAO,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;IACtC,CAAC;IAED,uEAAuE;IACvE,IAAI,YAAY;QACd,OAAO,IAAI,CAAC,aAAa,CAAC;IAC5B,CAAC;CACF"}
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "./builtin-manifest.schema.json",
|
|
3
3
|
"schemaVersion": 1,
|
|
4
|
-
"generatedAt": "2026-06-
|
|
5
|
-
"instarVersion": "1.3.
|
|
4
|
+
"generatedAt": "2026-06-24T12:19:12.219Z",
|
|
5
|
+
"instarVersion": "1.3.653",
|
|
6
6
|
"entryCount": 202,
|
|
7
7
|
"entries": {
|
|
8
8
|
"hook:session-start": {
|
|
@@ -1538,7 +1538,7 @@
|
|
|
1538
1538
|
"type": "subsystem",
|
|
1539
1539
|
"domain": "sessions",
|
|
1540
1540
|
"sourcePath": "src/core/SessionManager.ts",
|
|
1541
|
-
"contentHash": "
|
|
1541
|
+
"contentHash": "1e975218f0ef7377bf709128150e65af094afcb3ff524c6822f2af136548f12b",
|
|
1542
1542
|
"since": "2025-01-01"
|
|
1543
1543
|
},
|
|
1544
1544
|
"subsystem:auto-updater": {
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: patch -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
On a multi-machine setup, the SessionReaper's post-transfer closeout could terminate the LIVE local session for a topic when the local ownership record was STALE — it acted on "the registry says another machine owns this topic" without ever verifying that the other machine ACTUALLY has a live session for it, so a stale label could get the only live worker killed (and the operator-visible symptom was a `closeout-breaker` "the old session won't close" alert, escalating the wrong thing). This change (dev-gated dark behind `monitoring.sessionReaper.closeoutLivenessGate`, LIVE on a development agent) makes the closeout verify remote liveness first via a machine-local snapshot of each peer's `GET /sessions`: it proceeds ONLY when the owning machine genuinely has a live session (a real duplicate), and WITHHOLDS — never killing the sole live worker — on a no-live-remote-session reading, an unknown/unreachable peer, or any uncertainty (fail-closed everywhere). It also re-keys the closeout breaker counters on the stable topic id so a session-id churn across respawn no longer resets the veto count, and adds a narrow audited keep-reason bypass so a genuine move's leftover with a stale pre-move "recent" message can actually shed. Ships dark on the fleet (the dark stage of the maturation ladder, with explicit fleet-promotion criteria); when off, the closeout's behavior is byte-identical to today.
|
|
9
|
+
|
|
10
|
+
## What to Tell Your User
|
|
11
|
+
|
|
12
|
+
If you run me across more than one machine, I will never close the one session still doing real work just because a stale "who owns this conversation" record points at another machine. Before this, a record that went stale (the other machine already finished) could make my cleanup robot keep trying to kill the live worker in a loop. Now I first confirm the other machine actually has a live session before closing the leftover; if I can't confirm it, I hold off rather than risk killing live work. This is off everywhere except this development agent while it soaks.
|
|
13
|
+
|
|
14
|
+
## Summary of New Capabilities
|
|
15
|
+
|
|
16
|
+
- The post-transfer closeout now liveness-gates the kill: it terminates a topic's leftover session ONLY when the owning machine is confirmed to have a live session, and WITHHOLDS (fail-closed) when the owner has no live session, is unreachable, or liveness is unknown — so a stale ownership record can never get the sole live local worker killed.
|
|
17
|
+
- A narrow, audited `recent-user-message` bypass lets a genuinely-moved leftover finally shed, but only after re-checking every OTHER keep-guard (active-subagent, open-commitment, …), so a live-working session is never killed by the bypass.
|
|
18
|
+
- The closeout veto breaker is re-keyed on the stable topic id (not the churning session id), and the whole behavior ships dark behind `monitoring.sessionReaper.closeoutLivenessGate` (default OFF; dev-gated live-on-dev / dark-on-fleet) — flag off is byte-identical to today.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Upgrade Guide — vNEXT
|
|
2
|
+
|
|
3
|
+
<!-- assembled-by: assemble-next-md -->
|
|
4
|
+
<!-- bump: patch -->
|
|
5
|
+
|
|
6
|
+
## What Changed
|
|
7
|
+
|
|
8
|
+
The multi-machine `SessionOwnership` record can drift away from where the live
|
|
9
|
+
session actually is, leaving a stale `active(owner=this-machine)` label after a
|
|
10
|
+
transferred-here session completes. PR #1258 (`post-transfer-closeout-correctness`)
|
|
11
|
+
*stopped the harm* by liveness-gating the `SessionReaper` closeout so it never
|
|
12
|
+
kills a live local session on a stale label — a compensating control. This change
|
|
13
|
+
removes the stale record itself, so the dangerous state can't arise:
|
|
14
|
+
|
|
15
|
+
- **Part A — release-on-complete.** On the `sessionComplete` event, the owning
|
|
16
|
+
machine issues a `release` CAS for a topic-owned session it actively owns, so the
|
|
17
|
+
record advances to `released` (→ `ownerOf` reads null) instead of being stuck
|
|
18
|
+
`active`. Guarded by a session-identity check (FD9): the release is WITHHELD if a
|
|
19
|
+
DIFFERENT live session is bound to the topic — compared by the stable per-instance
|
|
20
|
+
key `startedAt`, NOT the reusable tmux name — closing the same-machine A∥B
|
|
21
|
+
"released record, live session" race the adversarial reviewer caught. Withheld on
|
|
22
|
+
every uncertainty (unresolvable topic, unprovable instance identity, registry
|
|
23
|
+
throw, CAS-lost).
|
|
24
|
+
- **Part B — claim-on-spawn.** The autonomous respawn path (which bypasses the
|
|
25
|
+
router's claim) issues a fenced-epoch `place → claim` so ownership follows the
|
|
26
|
+
live session onto this machine. It NEVER force-claims a peer-owned topic (force is
|
|
27
|
+
reserved for the reconciler's death-evidence path) — a post-gate owned-elsewhere
|
|
28
|
+
race withholds + records one neutral audit row. A lost CAS is a no-op (the spawn
|
|
29
|
+
still runs; the existing reconciler/applier converge).
|
|
30
|
+
- **Part D — double-dispatch recovery gate.** The single ungated recovery funnel
|
|
31
|
+
`SessionRecovery.checkAndRecover` (and the lease-gated `recoverStuckMessages`) now
|
|
32
|
+
consults per-topic ownership before re-running locally. A reachable-peer owner →
|
|
33
|
+
forward to the owner (don't re-run); an unreachable-peer owner → withhold (the
|
|
34
|
+
message rides the durable inbound queue); a null/self owner → re-run locally. The
|
|
35
|
+
registry-read-error path is deliberately fail-OPEN (re-run locally) and labeled as
|
|
36
|
+
such, emitting a `recovery-gate-registry-unknown` telemetry row so the tradeoff is
|
|
37
|
+
measured before any fleet promotion.
|
|
38
|
+
|
|
39
|
+
All three parts ride ONE dark flag `multiMachine.ownershipFollowsLiveWork`, OMITTED
|
|
40
|
+
from `ConfigDefaults` so `resolveDevAgentGate` resolves it LIVE on a dev agent / DARK
|
|
41
|
+
on the fleet. With the flag OFF (or a single-machine agent — `_meshSelfId` null), the
|
|
42
|
+
behavior is byte-identical to before: no release-on-complete, no claim-on-spawn, and
|
|
43
|
+
the recovery paths run their existing logic with no ownership consultation.
|
|
44
|
+
|
|
45
|
+
## What to Tell Your User
|
|
46
|
+
|
|
47
|
+
Nothing yet — this ships dark on the fleet (live only on development agents for
|
|
48
|
+
dogfooding) and is internal plumbing with no user-facing surface. The fleet flip is
|
|
49
|
+
an evidence-gated dev soak, tracked separately (see the spec's fleet-promotion
|
|
50
|
+
acceptance criteria). Do NOT narrate it as a finished capability.
|
|
51
|
+
|
|
52
|
+
## Summary of New Capabilities
|
|
53
|
+
|
|
54
|
+
- The multi-machine ownership record self-corrects toward where the live work is:
|
|
55
|
+
a completed session releases, an autonomous spawn claims, and a non-router recovery
|
|
56
|
+
path forwards (instead of double-dispatching) a topic this machine no longer owns.
|
|
57
|
+
- Purely additive and dev-agent-gated; flag OFF / single-machine = byte-identical to
|
|
58
|
+
today. Every ownership write is the existing fenced-epoch CAS, best-effort, never
|
|
59
|
+
forced; Part D's ownership read is a signal that can only ever withhold a local
|
|
60
|
+
re-run or forward to the owner — never a new kill or send.
|
|
61
|
+
|
|
62
|
+
## Evidence
|
|
63
|
+
|
|
64
|
+
- `npx tsc --noEmit` — clean.
|
|
65
|
+
- Full repo lint suite (`npm run lint`) — green (incl. `lint-cas-emit-placement`:
|
|
66
|
+
12 CAS sites all paired; `lint-dev-agent-dark-gate`; `lint-no-direct-destructive`).
|
|
67
|
+
- 66 new tests across all three tiers, all green:
|
|
68
|
+
- Unit: `tests/unit/ownershipFollowsLiveWork.test.ts` (26 — Part A/B pure helpers
|
|
69
|
+
both-sides incl. FD9 same-machine A∥B clobber + not-yet-bound; FD10 nonce
|
|
70
|
+
collision-resistance; the A∥B and B∥transfer fenced-epoch races against the real
|
|
71
|
+
registry FSM) + `tests/unit/ownershipFollowsLiveWork-recovery-gate.test.ts` (10 —
|
|
72
|
+
Part D every ownership state + the fail-open-but-counted telemetry assertion) +
|
|
73
|
+
`tests/unit/stuck-message-recovery.test.ts` (3 added — the third Part-D site).
|
|
74
|
+
- Integration: `tests/integration/ownership-follows-live-work.test.ts` (6 — the
|
|
75
|
+
Part D gate wired with the REAL `MachinePoolRegistry` online signal; Part A/B
|
|
76
|
+
against the real registry).
|
|
77
|
+
- E2E: `tests/e2e/ownership-follows-live-work-lifecycle.test.ts` (7 — flag
|
|
78
|
+
resolution alive-on-dev / dark-on-fleet through the REAL ConfigDefaults +
|
|
79
|
+
`resolveDevAgentGate`; the PR #1258 stale-record lifecycle regression fixed at
|
|
80
|
+
the source).
|
|
81
|
+
- Legacy regression suites green (no breakage): SessionOwnership, SessionOwnershipRegistry,
|
|
82
|
+
SessionRecovery, the three session-reaper suites (the PR #1258 area), OwnershipReconciler,
|
|
83
|
+
OwnershipApplier, devGatedFeatures-wiring (validates the new dev-gated entry).
|
|
84
|
+
- Spec: `docs/specs/ownership-follows-live-work.md` (converged 3 rounds, codex-cli:gpt-5.5
|
|
85
|
+
external reviewer; 11 frontloaded decisions). Convergence report sibling.
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# Side-Effects Review — Ownership Follows Live Work (release-on-complete + claim-on-spawn + double-dispatch recovery gate)
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `ownership-follows-live-work`
|
|
4
|
+
**Date:** `2026-06-24`
|
|
5
|
+
**Author:** Echo (autonomous)
|
|
6
|
+
**Spec:** `docs/specs/ownership-follows-live-work.md` (converged, approved, 11 frontloaded decisions)
|
|
7
|
+
**Second-pass reviewer:** independent Phase-5 review (the operator owns the trace/commit/PR ceremony)
|
|
8
|
+
|
|
9
|
+
## Summary of the change
|
|
10
|
+
|
|
11
|
+
Three independent gaps let the multi-machine `SessionOwnership` record drift from
|
|
12
|
+
where the live session actually is. This change makes the record self-correcting,
|
|
13
|
+
removing the stale-`active` cause PR #1258's reaper-closeout gate had to compensate
|
|
14
|
+
for. All three parts ride ONE dark flag `multiMachine.ownershipFollowsLiveWork`
|
|
15
|
+
(OMITTED from ConfigDefaults → `resolveDevAgentGate`: live-on-dev / dark-on-fleet).
|
|
16
|
+
|
|
17
|
+
Files added:
|
|
18
|
+
- `src/core/ownershipFollowsLiveWork.ts` — pure decision helpers (`shouldReleaseOnComplete`
|
|
19
|
+
for Part A's gate incl. the FD9 session-identity guard; `planClaimOnSpawn` for Part
|
|
20
|
+
B; `ownershipNonce` — the FD10 single collision-resistant nonce source). Single-
|
|
21
|
+
sourced so the gate logic is unit-testable.
|
|
22
|
+
- `tests/unit/ownershipFollowsLiveWork.test.ts`, `tests/unit/ownershipFollowsLiveWork-recovery-gate.test.ts`,
|
|
23
|
+
`tests/integration/ownership-follows-live-work.test.ts`, `tests/e2e/ownership-follows-live-work-lifecycle.test.ts`.
|
|
24
|
+
- `upgrades/next/ownership-follows-live-work.md` — release fragment (agent-only / experimental).
|
|
25
|
+
|
|
26
|
+
Files modified:
|
|
27
|
+
- `src/commands/server.ts` — (A) a new flag-gated `sessionComplete` listener wired
|
|
28
|
+
INSIDE the durable-ownership block (the only scope where `ownReg`/`emitPlacement`/
|
|
29
|
+
`_meshSelfId` are in lexical view — an anchor correction, see below) issuing a
|
|
30
|
+
release CAS + the mandatory `emitPlacement`; (B) a hoisted
|
|
31
|
+
`_claimOwnershipForAutonomousSpawn` helper called from the AutonomousLivenessReconciler
|
|
32
|
+
`respawn` closure after a successful spawn; (D) Part-D deps wired into the
|
|
33
|
+
SessionRecovery construction + the `recoverStuckMessages` per-topic owner skip; a
|
|
34
|
+
shared `isOwnerReachableShared` helper used by BOTH Part-D gates.
|
|
35
|
+
- `src/monitoring/SessionRecovery.ts` — the Part-D recovery gate: new injected deps
|
|
36
|
+
on `SessionRecoveryDeps` + the `decideOwnershipForRecovery` decision method +
|
|
37
|
+
the forward/withhold/proceed handling at the top of `checkAndRecover`.
|
|
38
|
+
- `src/messaging/stuckMessageRecovery.ts` — the third Part-D site: an optional
|
|
39
|
+
`ownerElsewhereReachable` dep that SKIPS (leaves untouched) a topic owned by a
|
|
40
|
+
reachable peer.
|
|
41
|
+
- `src/core/types.ts` — the optional `multiMachine.ownershipFollowsLiveWork?: boolean`.
|
|
42
|
+
- `src/core/devGatedFeatures.ts` — the `DEV_GATED_FEATURES` registration.
|
|
43
|
+
|
|
44
|
+
## Anchor corrections (code-grounding discipline)
|
|
45
|
+
|
|
46
|
+
The spec's anchors were close but the code won in three places:
|
|
47
|
+
1. **Part A scope.** The spec said register the release handler "alongside the
|
|
48
|
+
existing `sessionComplete` handler (server.ts:13247)" assuming `ownReg`/`_meshSelfId`
|
|
49
|
+
were in scope there. They are NOT — they are declared in a nested try-block textually
|
|
50
|
+
AFTER 13247. The handler is registered INSIDE that block (a session listener may be
|
|
51
|
+
added at any init point); functionally identical, correct scope.
|
|
52
|
+
2. **`getSessionForTopic` returns a session NAME, not a Session.** The spec's Part A
|
|
53
|
+
pseudocode read `liveForTopic.startedAt` off the result; the real method returns
|
|
54
|
+
`string | null`. The implementation resolves the name to the running Session via
|
|
55
|
+
`sessionManager.listRunningSessions()` and reads its `startedAt`.
|
|
56
|
+
3. **Part B/D scope.** Part B's claim is exposed via a hoisted helper so the respawn
|
|
57
|
+
closure (defined before the ownership block) can reach it; Part D's deps are
|
|
58
|
+
late-bound closures that read the registry/pool assigned later — both invoked at
|
|
59
|
+
runtime, after wiring.
|
|
60
|
+
|
|
61
|
+
## Decision-point inventory
|
|
62
|
+
|
|
63
|
+
- **Added (Part A — authority):** a `release` CAS callsite on `sessionComplete`, gated
|
|
64
|
+
by (a) flag+self, (b) owner===self & status active, (c) the FD9 session-identity
|
|
65
|
+
guard. Fail-CLOSED on every uncertainty (withhold the release). Paired with the
|
|
66
|
+
mandatory `emitPlacement('released')` (cas-emit-placement lint).
|
|
67
|
+
- **Added (Part B — authority):** a `place→claim` CAS pair on the autonomous respawn.
|
|
68
|
+
NEVER force-claims a peer-owned topic (FD3). Fail-CLOSED (withhold + one neutral
|
|
69
|
+
audit row on a post-gate owned-elsewhere race).
|
|
70
|
+
- **Added (Part D — SIGNAL, not authority):** a per-topic `ownerOf` read inside
|
|
71
|
+
`checkAndRecover` + `recoverStuckMessages` that can ONLY ever WITHHOLD a local
|
|
72
|
+
re-run / forward to the owner — never a new kill or send. Direction is MIXED and
|
|
73
|
+
labeled per state: reachable-peer→forward (fail-closed), unreachable-peer incl.
|
|
74
|
+
reachability-throw→withhold (fail-closed), null/self→proceed, ownerOf-throw→proceed
|
|
75
|
+
(fail-OPEN, instrumented). No existing decision boundary is removed or weakened.
|
|
76
|
+
|
|
77
|
+
No new HTTP route, no new repeating loop (Part A fires once per completion event,
|
|
78
|
+
Part B once per spawn, the recovery gate is a one-shot pre-respawn check).
|
|
79
|
+
|
|
80
|
+
## Roll-up across the seven review dimensions
|
|
81
|
+
|
|
82
|
+
1. **Over-block**: none on the user channel. Part D can only WITHHOLD a local recovery
|
|
83
|
+
re-run / forward it to the true owner — it never blocks a user message. Parts A/B
|
|
84
|
+
withhold an ownership WRITE on uncertainty (the safe direction); a withheld release
|
|
85
|
+
at worst leaves a stale `active` the existing reconciler + PR #1258 closeout still
|
|
86
|
+
handle.
|
|
87
|
+
2. **Under-block**: none. No gate is loosened. Part B never force-claims; Part A never
|
|
88
|
+
releases a record it can't prove is the completed work's; Part D's fail-OPEN branch
|
|
89
|
+
is narrowly the registry-unreadable case (an already-broken state) and is counted.
|
|
90
|
+
3. **Level-of-abstraction fit**: correct. The A/B decision logic is single-sourced in a
|
|
91
|
+
pure helper module (unit-testable); the CAS + emitPlacement pairing stays at the
|
|
92
|
+
server wiring site where those deps live. Part D's decision lives in SessionRecovery
|
|
93
|
+
(its deps are injected) with the primitives bound at server init.
|
|
94
|
+
4. **Signal-vs-authority compliance**: Parts A/B issue real AUTHORITY through the SAME
|
|
95
|
+
guarded `SessionOwnershipRegistry.cas()` FSM at a fenced `epoch+1` (no new FSM
|
|
96
|
+
transition, no new authority surface). Part D's `ownerOf` is a SIGNAL. Every new
|
|
97
|
+
try/catch is either an `@silent-fallback-ok` best-effort observability path or a
|
|
98
|
+
deliberate fail-closed/fail-open branch documented in code — complies with the
|
|
99
|
+
no-silent-fallbacks posture (lint green).
|
|
100
|
+
5. **Interactions**: writes to the EXISTING replicated ownership lifecycle (same
|
|
101
|
+
`emitPlacement` → coherence journal → OwnershipApplier path the user-move
|
|
102
|
+
release/transfer already use), so a Part A release / Part B claim replicates exactly
|
|
103
|
+
like today's releases. The `isOwnerReachable` signal is the SAME
|
|
104
|
+
`machinePoolRegistry.getCapacity(owner).online` the router uses (shared helper — the
|
|
105
|
+
two Part-D gates can't diverge). No new shared state.
|
|
106
|
+
6. **External surfaces**: NONE. No new endpoint, no egress, no third-party spend, no
|
|
107
|
+
destructive fs/git action. The only new on-disk writes are append-only neutral audit
|
|
108
|
+
rows to the EXISTING machine-local `logs/sentinel-events.jsonl` /
|
|
109
|
+
`logs/autonomous-liveness.jsonl`. Single-machine agents are a strict no-op.
|
|
110
|
+
7. **Rollback cost**: low. Dev-gated dark on the fleet; a config flip force-darks even a
|
|
111
|
+
dev agent. The flag OFF / single-machine path is byte-identical to today (locked by
|
|
112
|
+
the flag-OFF regression tests across all three tiers). No migration needed (the flag
|
|
113
|
+
is omitted from ConfigDefaults — the gate decides at runtime).
|
|
114
|
+
|
|
115
|
+
## Cross-machine / mixed-fleet note
|
|
116
|
+
|
|
117
|
+
The flag resolves PER-MACHINE, which is CORRECT here (unlike a transfer/placement
|
|
118
|
+
feature where half-activation strands a seat): each of A/B/D makes a strictly-MORE-correct
|
|
119
|
+
LOCAL decision (release a record we own, claim a record for a session we run, forward
|
|
120
|
+
instead of double-dispatch). A gate-ON machine self-corrects its own records; a gate-OFF
|
|
121
|
+
peer keeps today's (stale-prone) behavior — no torn/corrupt cross-machine state (each CAS
|
|
122
|
+
is fenced; a gate-OFF machine simply omits some releases/claims, which the existing
|
|
123
|
+
reconciler/applier already tolerate). During the mixed-fleet soak, PR #1258's
|
|
124
|
+
liveness-snapshot gate REMAINS the defense against a peer whose applier is lagged — which
|
|
125
|
+
is why its retirement is a SEPARATE, evidence-gated follow-up after the A/B soak, not
|
|
126
|
+
bundled here. This change does NOT touch `SessionReaper.ts` (PR #1258 owns it) or the FSM
|
|
127
|
+
transitions.
|
|
128
|
+
|
|
129
|
+
## Evidence pointers
|
|
130
|
+
|
|
131
|
+
- `npx tsc --noEmit` — clean.
|
|
132
|
+
- `npm run lint` — green (incl. `lint-cas-emit-placement`: 12 CAS sites all paired;
|
|
133
|
+
`lint-dev-agent-dark-gate`; `lint-no-direct-destructive`; `lint-no-unbounded-llm-spawn`).
|
|
134
|
+
- New tests (66, all green): unit (26 + 10 + 3-added), integration (6), e2e (7).
|
|
135
|
+
- Legacy regression green: SessionOwnership, SessionOwnershipRegistry, SessionRecovery,
|
|
136
|
+
the three session-reaper suites (PR #1258 area), OwnershipReconciler, OwnershipApplier,
|
|
137
|
+
devGatedFeatures-wiring (validates the new dev-gated entry — 117 tests).
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Side-Effects Review — Post-Transfer Closeout Correctness (liveness-gate the stale-ownership kill)
|
|
2
|
+
|
|
3
|
+
**Slug:** `post-transfer-closeout-correctness`
|
|
4
|
+
**Spec:** `docs/specs/post-transfer-closeout-correctness.md` (converged 4 rounds, approved)
|
|
5
|
+
**Author:** Echo (autonomous 24h mesh-robustness mission, topic 27515)
|
|
6
|
+
**Branch:** `echo/ownership-follows-live-work`
|
|
7
|
+
**Risk class:** safety-critical (the changed decision can terminate a LIVE local session) — mitigated by fail-closed-everywhere + a default-OFF dev-gated flag.
|
|
8
|
+
|
|
9
|
+
## What changed (in-scope files)
|
|
10
|
+
|
|
11
|
+
- `src/monitoring/closeoutLivenessSnapshot.ts` (NEW) — `CloseoutLivenessSnapshot`: the machine-local, periodically-refreshed snapshot of "which topics have a live session on each peer", built FROM the existing per-peer `GET /sessions` fan-out (no new endpoint). Exposes the synchronous `remoteOwnerHasLiveSession(topicId, machineId) → { state: true|false|'unknown'; reachableAt? }` dep and a bounded, owner-scoped `refresh()` (5s per-fetch timeout, per-pass eviction of departed peers, observability breaker that SURFACES — never STOPS — a persistent all-peers-failed condition). Fail-closed: missing/stale/unreachable → `'unknown'`; a fresh reached peer with an EMPTY topics set is a definitive `false` (the stale-owner signal).
|
|
12
|
+
- `src/monitoring/SessionReaper.ts` (+~220) — Part C decision (`runCloseoutGated`): consults `remoteOwnerHasLiveSession` inside the `else if (otherOwner)` arm, AFTER the pin-conflict hold; `true` → dwell→terminate; `false`/`'unknown'`/dep-absent → WITHHOLD + neutral once-per-episode `possibleStaleOwner` audit. Dwell advances only across DISTINCT snapshot generations (`reachableAt` advancement). Part E: passes `bypassRecentUserMessageForConfirmedMove` ONLY on a confirmed move whose freshest LOCAL user message predates the snapshot (freshest-interaction veto). Secondary: breaker maps (`topicMovedStreak`/`topicMovedVetoes`) re-keyed on the stable TOPIC id with a GC grace window. The legacy path (`runCloseoutLegacy`) is the byte-identical OFF behavior. New config field `closeoutLivenessGate` (default false in the in-code `DEFAULT_SESSION_REAPER_CONFIG`).
|
|
13
|
+
- `src/core/SessionManager.ts` (+~15) — new narrow `terminateSession` option `bypassRecentUserMessageForConfirmedMove`, mirroring `bypassActiveProcessKeep`: extends the `bypassThis` computation to lift ONLY the `recent-user-message` keep-reason; every other KEEP-guard re-checked and still vetoes.
|
|
14
|
+
- `src/core/gapBCommitmentEvidence.ts` (+~20) — `recentUserMessageAtFromHistory(history) → number | null`: the LOCAL-receipt timestamp of the newest user message, SAME source the `recent-user-message` KEEP-guard reads (shared basis for the Part E veto).
|
|
15
|
+
- `src/core/types.ts` (+~12) — `monitoring.sessionReaper.closeoutLivenessGate?: boolean` (OMITTED-from-default dev-gated flag).
|
|
16
|
+
- `src/core/devGatedFeatures.ts` (+1 entry) — registers `monitoring.sessionReaper.closeoutLivenessGate` in `DEV_GATED_FEATURES` (live-on-dev / dark-fleet via `resolveDevAgentGate`).
|
|
17
|
+
- `src/commands/server.ts` (+~90) — wiring: resolves `closeoutLivenessGate` via `resolveDevAgentGate`; constructs the `CloseoutLivenessSnapshot` + its jittered (±10%) reaper-cadence refresh timer ONLY when the gate resolves on; injects `topicOwnerElsewhereInfo` (atomic machineId+display), `remoteOwnerHasLiveSession`, `recentUserMessageAt`; forwards the new bypass through the `terminate` dep; sets `closeoutLivenessGate` on the reaper config.
|
|
18
|
+
|
|
19
|
+
## Review
|
|
20
|
+
|
|
21
|
+
1. **Over-block (a wrong WITHHOLD).** The withhold is the SAFE direction — it only ever KEEPS a live local session the closeout would otherwise have attempted to kill. A genuine duplicate leftover that gets withheld under `'unknown'`/partition is NOT stranded: it still falls through to the normal idle pipeline every tick (the closeout block precedes it), so a sustained partition delays closeout-driven cleanup but never leaks zombies. Worst case is a bounded cleanup delay, never a wrong kill.
|
|
22
|
+
|
|
23
|
+
2. **Under-block (a wrong kill — the bug class this fixes).** With the gate ON, the ONLY path that reaches the closeout terminate is an explicit `true` from `remoteOwnerHasLiveSession` (owner genuinely has a live session = a real duplicate). Every uncertainty — `false`, `'unknown'`, dep throws, dep absent under the gate, stale/absent snapshot, unreachable peer, null topic binding — fails CLOSED to WITHHOLD. The one residual `true`-side race (remote completes between the snapshot and the terminate) is strictly milder than the original bug (a transient no-worker gap on a topic that DID move, self-healing on the next inbound message) and is bounded by the staleness bound + the dwell-advancement requirement; it is named/accepted in the spec's "Fail-closed everywhere".
|
|
24
|
+
|
|
25
|
+
3. **Signal vs authority compliance.** COMPLIANT. The change alters only the DECISION to ATTEMPT the closeout terminate; the terminate still routes through the guarded `terminateSession` authority (protected-set, lease-holder, KEEP-guards intact). `remoteOwnerHasLiveSession` is a SIGNAL that informs the decision, never an authority that kills. The Part E bypass is a NARROW, named keep-reason bypass of EXACTLY ONE reason (`recent-user-message`), in the EXACT shape of the existing `bypassActiveProcessKeep` precedent, used ONLY on a liveness-confirmed move — every other KEEP-guard is re-checked and still vetoes.
|
|
26
|
+
|
|
27
|
+
4. **Level-of-abstraction fit.** Correct layer. Remote liveness is per-peer `/sessions` data; the snapshot reads it (reusing the proven fan-out primitive), the reaper consults it synchronously, the terminate authority is untouched. No new reachability/ownership authority is invented — the closeout simply gains a verification step before acting on the ownership record that already drove it.
|
|
28
|
+
|
|
29
|
+
5. **Interactions.** The pin-conflict hold (WS1.3) still WINS ahead of the liveness gate (unchanged). The P19 veto-breaker is preserved; re-keying it on the topic id makes it count correctly across a same-topic respawn (the secondary fix — was ~32 min, now ~10). A WITHHELD episode never accrues breaker vetoes (it never attempted a terminate). Episode-hygiene clears (terminate-success / topic-home / pin-conflict) are preserved under the topic key, so a new episode for the same topic starts clean. The refresher's owner-scoped fan-out (only machineIds that back an owned-elsewhere local leftover) bounds per-tick HTTP to O(distinct remote owners with a leftover here), not O(pool size), and does zero HTTP when there are no leftovers.
|
|
30
|
+
|
|
31
|
+
6. **External surfaces.** No new route, no new unauthenticated input. The snapshot reuses the existing authenticated `GET /sessions` fetch (Bearer-auth, 5s timeout). The observability breaker raises ONE deduped attention item (P17-coalesced, `sourceContext: 'session-reaper:closeout-snapshot-breaker'`) only after a threshold of consecutive all-peers-failed passes. Audit rows land in this machine's existing `logs/sentinel-events.jsonl`. Nothing new is visible to other users/agents.
|
|
32
|
+
|
|
33
|
+
7. **Multi-machine posture (Cross-Machine Coherence).** The new `liveRemoteTopics` snapshot is **MACHINE-LOCAL BY DESIGN** — each machine runs its OWN closeout against its OWN locally-built snapshot (the closeout decision is inherently local: "should *I* shed *my* leftover?"). It is NOT replicated and NOT proxied; rebuilt from scratch each refresh (no migration/backup). The `closeoutLivenessGate` flag's per-machine dev-gate resolution is CORRECT here (unlike a transfer/placement feature): the closeout is a per-machine janitor, and each machine making its OWN strictly-more-conservative decision is independently safe — the gate changes only a LOCAL decision, never a replicated record, so a mixed (some-on/some-off) fleet produces only diagnostic asymmetry, never state corruption. A gate-OFF machine retains today's behavior until promoted (the dark stage of the maturation ladder; fleet-promotion criteria are in the spec's `## Verification`).
|
|
34
|
+
|
|
35
|
+
8. **Rollback cost.** Cheap and immediate. Flip `monitoring.sessionReaper.closeoutLivenessGate: false` in `.instar/config.json` (an explicit false force-darks even a dev agent; restart sessions/server to apply) → the snapshot refresher is never constructed, the new deps are left absent, the maps stay session-id-keyed, and the closeout's observable behavior is exactly canonical `main`. No data migration: the only new persisted artifacts are audit rows in `sentinel-events.jsonl` (additive). The in-memory snapshot evaporates on the next boot.
|
|
36
|
+
|
|
37
|
+
## Migration parity
|
|
38
|
+
|
|
39
|
+
`closeoutLivenessGate` is a DEV-GATED flag: it is OMITTED from `src/config/ConfigDefaults.ts` (NOT hardcoded `false`) so `resolveDevAgentGate` resolves it LIVE on a dev agent / DARK on the fleet, exactly like `topicProfiles` / `credentialRepointing` / the multi-machine seamlessness flags. Because it is omitted, `applyDefaults` never writes it to a deployed agent's config and the runtime gate handles it — so existing agents pick up the (dev-gated-off-on-fleet) behavior on update with no config write needed. An explicit config value always wins (false force-darks, true is the fleet-flip). The in-code constructor default (`DEFAULT_SESSION_REAPER_CONFIG.closeoutLivenessGate = false`) is the safe fallback when the field is absent.
|
|
40
|
+
|
|
41
|
+
## No-deferrals
|
|
42
|
+
|
|
43
|
+
This spec's scope is exactly "stop the dangerous kill" — Parts A (ownership release-on-complete), B (claim-on-spawn), and D (double-dispatch recovery gate) of the broader audit are an EXPLICIT separate follow-up PR, never in this scope. They are tracked via a durable one-time-action commitment opened on merge (the constitution's "Close the Loop" mechanism), which also carries the gate's retirement plan (when A/B land and make the ownership record self-correcting, the snapshot/refresher/dep retire; the Part E bypass + the topic-keyed breaker counters REMAIN — they fix orthogonal problems A/B don't address). No orphan deferral language; the false-case audit is deliberately observational-only (NO ownership-CAS write — that is Part A/B's job).
|