instar 1.3.337 → 1.3.338
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +37 -2
- package/dist/commands/server.js.map +1 -1
- package/dist/core/OwnerSuspectBreaker.d.ts +79 -0
- package/dist/core/OwnerSuspectBreaker.d.ts.map +1 -0
- package/dist/core/OwnerSuspectBreaker.js +106 -0
- package/dist/core/OwnerSuspectBreaker.js.map +1 -0
- package/dist/core/SessionRouter.d.ts +7 -0
- package/dist/core/SessionRouter.d.ts.map +1 -1
- package/dist/core/SessionRouter.js +14 -2
- package/dist/core/SessionRouter.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +2 -2
- package/upgrades/{1.3.337.md → 1.3.338.md} +41 -0
- package/upgrades/side-effects/owner-suspect-breaker.md +52 -0
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* OwnerSuspectBreaker — the per-peer circuit breaker behind the SessionRouter's
|
|
3
|
+
* `markOwnerSuspect` hook ("No Unbounded Loops" / P19).
|
|
4
|
+
*
|
|
5
|
+
* The router's forward path already had the SHAPE of a breaker: on delivery
|
|
6
|
+
* retry exhaustion it calls `markOwnerSuspect(owner)` before re-placing the
|
|
7
|
+
* message. But the hook was NEVER WIRED in production (the dep is optional and
|
|
8
|
+
* no implementation existed) — "constructed but inert". The consequence: every
|
|
9
|
+
* session owned by a slow-or-dead peer independently re-paid the full retry
|
|
10
|
+
* tax (3 attempts × backoff ≈ 4.5s+) per message, because `isMachineAlive`
|
|
11
|
+
* reads only capacity heartbeats, which a slow-but-alive peer keeps passing.
|
|
12
|
+
*
|
|
13
|
+
* This class is the wiring's missing half. Half-open TTL semantics:
|
|
14
|
+
*
|
|
15
|
+
* markSuspect(peer) — start/extend a suspect window (default 30s). The
|
|
16
|
+
* FIRST mark of an episode logs once; an episode
|
|
17
|
+
* sustained past signalAfterMs raises ONE degradation
|
|
18
|
+
* signal (per-peer FailureEpisodeLatch — P19 cond. 4).
|
|
19
|
+
* isSuspect(peer) — true inside the window. The composed isMachineAlive
|
|
20
|
+
* then short-circuits dispatches for ALL of that
|
|
21
|
+
* peer's sessions straight to the existing failover
|
|
22
|
+
* re-place path — no per-message retry tax repaid.
|
|
23
|
+
* recordSuccess(peer) — a delivery reached the peer: window cleared,
|
|
24
|
+
* recovery logged once, latch re-armed.
|
|
25
|
+
*
|
|
26
|
+
* After the TTL expires the breaker is HALF-OPEN: the next message tries the
|
|
27
|
+
* peer for real (paying one retry cycle); success clears, failure re-marks.
|
|
28
|
+
* The peer is never written off permanently — the TTL is the backoff, the
|
|
29
|
+
* suspect state is the breaker, the per-message retry config is the cap.
|
|
30
|
+
*
|
|
31
|
+
* Deliberately POLICY-NEUTRAL: what happens to messages while a peer is
|
|
32
|
+
* suspect (today: the router's existing re-place path) is the wiring's choice
|
|
33
|
+
* — the queue-vs-replace stability policy is a separate operator decision
|
|
34
|
+
* <!-- tracked: CMT-1109 -->. Pure: injectable clock, per-peer bounded state.
|
|
35
|
+
*/
|
|
36
|
+
export interface OwnerSuspectBreakerOpts {
|
|
37
|
+
/** Suspect window per mark (the half-open backoff). Default 30s. */
|
|
38
|
+
suspectTtlMs?: number;
|
|
39
|
+
/** Sustained-suspicion threshold for the one-per-episode degradation signal. Default 10min. */
|
|
40
|
+
signalAfterMs?: number;
|
|
41
|
+
now?: () => number;
|
|
42
|
+
logger?: (msg: string) => void;
|
|
43
|
+
/** One-per-episode sustained-suspicion sink (wired to DegradationReporter). */
|
|
44
|
+
reportSustained?: (info: {
|
|
45
|
+
machineId: string;
|
|
46
|
+
suspectForMs: number;
|
|
47
|
+
marks: number;
|
|
48
|
+
}) => void;
|
|
49
|
+
}
|
|
50
|
+
export declare class OwnerSuspectBreaker {
|
|
51
|
+
private readonly ttlMs;
|
|
52
|
+
private readonly now;
|
|
53
|
+
private readonly opts;
|
|
54
|
+
/** Per-peer suspect-window end (ms epoch). Deleted on success. */
|
|
55
|
+
private suspectUntil;
|
|
56
|
+
/** Per-peer episode accounting (created on first mark, deleted on success). */
|
|
57
|
+
private episodes;
|
|
58
|
+
constructor(opts?: OwnerSuspectBreakerOpts);
|
|
59
|
+
private log;
|
|
60
|
+
/** Delivery retries to this peer exhausted — open its suspect window.
|
|
61
|
+
*
|
|
62
|
+
* ABSOLUTE per-episode TTL: a mark that arrives while a window is ALREADY open
|
|
63
|
+
* does NOT push the window end forward. Otherwise a steady message stream
|
|
64
|
+
* arriving faster than the TTL would re-extend the window on every dispatch
|
|
65
|
+
* (each suspect dispatch re-enters dispatchOne's dead-owner branch, which
|
|
66
|
+
* re-calls markOwnerSuspect), so `isSuspect` would never go false at the
|
|
67
|
+
* moment a message arrives — the half-open re-probe (the ONLY path that ever
|
|
68
|
+
* re-attempts delivery and thus the ONLY path that can clear suspicion) would
|
|
69
|
+
* never be reached, leaving a fully-RECOVERED peer suspect forever and
|
|
70
|
+
* force-failing-over all its sessions on every message. The episode latch
|
|
71
|
+
* still records each mark (sustained-suspicion accounting), but the window
|
|
72
|
+
* end is fixed at the first mark of each TTL period. */
|
|
73
|
+
markSuspect(machineId: string): void;
|
|
74
|
+
/** Inside an open suspect window? (Half-open after the TTL — callers re-probe.) */
|
|
75
|
+
isSuspect(machineId: string): boolean;
|
|
76
|
+
/** A delivery reached the peer — close the episode and re-arm. */
|
|
77
|
+
recordSuccess(machineId: string): void;
|
|
78
|
+
}
|
|
79
|
+
//# sourceMappingURL=OwnerSuspectBreaker.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"OwnerSuspectBreaker.d.ts","sourceRoot":"","sources":["../../src/core/OwnerSuspectBreaker.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAIH,MAAM,WAAW,uBAAuB;IACtC,oEAAoE;IACpE,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,+FAA+F;IAC/F,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,GAAG,CAAC,EAAE,MAAM,MAAM,CAAC;IACnB,MAAM,CAAC,EAAE,CAAC,GAAG,EAAE,MAAM,KAAK,IAAI,CAAC;IAC/B,+EAA+E;IAC/E,eAAe,CAAC,EAAE,CAAC,IAAI,EAAE;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,YAAY,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CAC9F;AAKD,qBAAa,mBAAmB;IAC9B,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAS;IAC/B,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAe;IACnC,OAAO,CAAC,QAAQ,CAAC,IAAI,CAA0B;IAC/C,kEAAkE;IAClE,OAAO,CAAC,YAAY,CAA6B;IACjD,+EAA+E;IAC/E,OAAO,CAAC,QAAQ,CAA0C;gBAE9C,IAAI,GAAE,uBAA4B;IAM9C,OAAO,CAAC,GAAG;IAIX;;;;;;;;;;;;4DAYwD;IACxD,WAAW,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IAsBpC,mFAAmF;IACnF,SAAS,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO;IAKrC,kEAAkE;IAClE,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;CASvC"}
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* OwnerSuspectBreaker — the per-peer circuit breaker behind the SessionRouter's
|
|
3
|
+
* `markOwnerSuspect` hook ("No Unbounded Loops" / P19).
|
|
4
|
+
*
|
|
5
|
+
* The router's forward path already had the SHAPE of a breaker: on delivery
|
|
6
|
+
* retry exhaustion it calls `markOwnerSuspect(owner)` before re-placing the
|
|
7
|
+
* message. But the hook was NEVER WIRED in production (the dep is optional and
|
|
8
|
+
* no implementation existed) — "constructed but inert". The consequence: every
|
|
9
|
+
* session owned by a slow-or-dead peer independently re-paid the full retry
|
|
10
|
+
* tax (3 attempts × backoff ≈ 4.5s+) per message, because `isMachineAlive`
|
|
11
|
+
* reads only capacity heartbeats, which a slow-but-alive peer keeps passing.
|
|
12
|
+
*
|
|
13
|
+
* This class is the wiring's missing half. Half-open TTL semantics:
|
|
14
|
+
*
|
|
15
|
+
* markSuspect(peer) — start/extend a suspect window (default 30s). The
|
|
16
|
+
* FIRST mark of an episode logs once; an episode
|
|
17
|
+
* sustained past signalAfterMs raises ONE degradation
|
|
18
|
+
* signal (per-peer FailureEpisodeLatch — P19 cond. 4).
|
|
19
|
+
* isSuspect(peer) — true inside the window. The composed isMachineAlive
|
|
20
|
+
* then short-circuits dispatches for ALL of that
|
|
21
|
+
* peer's sessions straight to the existing failover
|
|
22
|
+
* re-place path — no per-message retry tax repaid.
|
|
23
|
+
* recordSuccess(peer) — a delivery reached the peer: window cleared,
|
|
24
|
+
* recovery logged once, latch re-armed.
|
|
25
|
+
*
|
|
26
|
+
* After the TTL expires the breaker is HALF-OPEN: the next message tries the
|
|
27
|
+
* peer for real (paying one retry cycle); success clears, failure re-marks.
|
|
28
|
+
* The peer is never written off permanently — the TTL is the backoff, the
|
|
29
|
+
* suspect state is the breaker, the per-message retry config is the cap.
|
|
30
|
+
*
|
|
31
|
+
* Deliberately POLICY-NEUTRAL: what happens to messages while a peer is
|
|
32
|
+
* suspect (today: the router's existing re-place path) is the wiring's choice
|
|
33
|
+
* — the queue-vs-replace stability policy is a separate operator decision
|
|
34
|
+
* <!-- tracked: CMT-1109 -->. Pure: injectable clock, per-peer bounded state.
|
|
35
|
+
*/
|
|
36
|
+
import { FailureEpisodeLatch } from './FailureEpisodeLatch.js';
|
|
37
|
+
const DEFAULT_SUSPECT_TTL_MS = 30_000;
|
|
38
|
+
const DEFAULT_SIGNAL_AFTER_MS = 10 * 60_000;
|
|
39
|
+
export class OwnerSuspectBreaker {
|
|
40
|
+
ttlMs;
|
|
41
|
+
now;
|
|
42
|
+
opts;
|
|
43
|
+
/** Per-peer suspect-window end (ms epoch). Deleted on success. */
|
|
44
|
+
suspectUntil = new Map();
|
|
45
|
+
/** Per-peer episode accounting (created on first mark, deleted on success). */
|
|
46
|
+
episodes = new Map();
|
|
47
|
+
constructor(opts = {}) {
|
|
48
|
+
this.opts = opts;
|
|
49
|
+
this.ttlMs = opts.suspectTtlMs ?? DEFAULT_SUSPECT_TTL_MS;
|
|
50
|
+
this.now = opts.now ?? Date.now;
|
|
51
|
+
}
|
|
52
|
+
log(m) {
|
|
53
|
+
this.opts.logger?.(`[owner-suspect] ${m}`);
|
|
54
|
+
}
|
|
55
|
+
/** Delivery retries to this peer exhausted — open its suspect window.
|
|
56
|
+
*
|
|
57
|
+
* ABSOLUTE per-episode TTL: a mark that arrives while a window is ALREADY open
|
|
58
|
+
* does NOT push the window end forward. Otherwise a steady message stream
|
|
59
|
+
* arriving faster than the TTL would re-extend the window on every dispatch
|
|
60
|
+
* (each suspect dispatch re-enters dispatchOne's dead-owner branch, which
|
|
61
|
+
* re-calls markOwnerSuspect), so `isSuspect` would never go false at the
|
|
62
|
+
* moment a message arrives — the half-open re-probe (the ONLY path that ever
|
|
63
|
+
* re-attempts delivery and thus the ONLY path that can clear suspicion) would
|
|
64
|
+
* never be reached, leaving a fully-RECOVERED peer suspect forever and
|
|
65
|
+
* force-failing-over all its sessions on every message. The episode latch
|
|
66
|
+
* still records each mark (sustained-suspicion accounting), but the window
|
|
67
|
+
* end is fixed at the first mark of each TTL period. */
|
|
68
|
+
markSuspect(machineId) {
|
|
69
|
+
if (!this.isSuspect(machineId)) {
|
|
70
|
+
this.suspectUntil.set(machineId, this.now() + this.ttlMs);
|
|
71
|
+
}
|
|
72
|
+
let latch = this.episodes.get(machineId);
|
|
73
|
+
if (!latch) {
|
|
74
|
+
latch = new FailureEpisodeLatch({
|
|
75
|
+
signalAfterMs: this.opts.signalAfterMs ?? DEFAULT_SIGNAL_AFTER_MS,
|
|
76
|
+
now: this.now,
|
|
77
|
+
});
|
|
78
|
+
this.episodes.set(machineId, latch);
|
|
79
|
+
}
|
|
80
|
+
const f = latch.recordFailure();
|
|
81
|
+
if (f.firstOfEpisode) {
|
|
82
|
+
this.log(`owner ${machineId} SUSPECT (delivery retries exhausted) — short-circuiting its sessions' dispatches for ${Math.round(this.ttlMs / 1000)}s windows until a delivery succeeds`);
|
|
83
|
+
}
|
|
84
|
+
if (f.shouldSignal) {
|
|
85
|
+
this.log(`owner ${machineId} suspect for ${Math.round(f.failingForMs / 60_000)}min (${f.failures} marks) — signaling once; half-open re-probes continue`);
|
|
86
|
+
this.opts.reportSustained?.({ machineId, suspectForMs: f.failingForMs, marks: f.failures });
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
/** Inside an open suspect window? (Half-open after the TTL — callers re-probe.) */
|
|
90
|
+
isSuspect(machineId) {
|
|
91
|
+
const until = this.suspectUntil.get(machineId);
|
|
92
|
+
return until !== undefined && this.now() < until;
|
|
93
|
+
}
|
|
94
|
+
/** A delivery reached the peer — close the episode and re-arm. */
|
|
95
|
+
recordSuccess(machineId) {
|
|
96
|
+
this.suspectUntil.delete(machineId);
|
|
97
|
+
const latch = this.episodes.get(machineId);
|
|
98
|
+
if (latch) {
|
|
99
|
+
const s = latch.recordSuccess();
|
|
100
|
+
if (s.recovered)
|
|
101
|
+
this.log(`owner ${machineId} recovered after ${s.failures} suspect mark(s)`);
|
|
102
|
+
this.episodes.delete(machineId);
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
//# sourceMappingURL=OwnerSuspectBreaker.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"OwnerSuspectBreaker.js","sourceRoot":"","sources":["../../src/core/OwnerSuspectBreaker.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAEH,OAAO,EAAE,mBAAmB,EAAE,MAAM,0BAA0B,CAAC;AAa/D,MAAM,sBAAsB,GAAG,MAAM,CAAC;AACtC,MAAM,uBAAuB,GAAG,EAAE,GAAG,MAAM,CAAC;AAE5C,MAAM,OAAO,mBAAmB;IACb,KAAK,CAAS;IACd,GAAG,CAAe;IAClB,IAAI,CAA0B;IAC/C,kEAAkE;IAC1D,YAAY,GAAG,IAAI,GAAG,EAAkB,CAAC;IACjD,+EAA+E;IACvE,QAAQ,GAAG,IAAI,GAAG,EAA+B,CAAC;IAE1D,YAAY,OAAgC,EAAE;QAC5C,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;QACjB,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,YAAY,IAAI,sBAAsB,CAAC;QACzD,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC;IAClC,CAAC;IAEO,GAAG,CAAC,CAAS;QACnB,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,mBAAmB,CAAC,EAAE,CAAC,CAAC;IAC7C,CAAC;IAED;;;;;;;;;;;;4DAYwD;IACxD,WAAW,CAAC,SAAiB;QAC3B,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,SAAS,CAAC,EAAE,CAAC;YAC/B,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;QAC5D,CAAC;QACD,IAAI,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACzC,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,KAAK,GAAG,IAAI,mBAAmB,CAAC;gBAC9B,aAAa,EAAE,IAAI,CAAC,IAAI,CAAC,aAAa,IAAI,uBAAuB;gBACjE,GAAG,EAAE,IAAI,CAAC,GAAG;aACd,CAAC,CAAC;YACH,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QACtC,CAAC;QACD,MAAM,CAAC,GAAG,KAAK,CAAC,aAAa,EAAE,CAAC;QAChC,IAAI,CAAC,CAAC,cAAc,EAAE,CAAC;YACrB,IAAI,CAAC,GAAG,CAAC,SAAS,SAAS,yFAAyF,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,qCAAqC,CAAC,CAAC;QAC1L,CAAC;QACD,IAAI,CAAC,CAAC,YAAY,EAAE,CAAC;YACnB,IAAI,CAAC,GAAG,CAAC,SAAS,SAAS,gBAAgB,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,YAAY,GAAG,MAAM,CAAC,QAAQ,CAAC,CAAC,QAAQ,wDAAwD,CAAC,CAAC;YAC1J,IAAI,CAAC,IAAI,CAAC,eAAe,EAAE,CAAC,EAAE,SAAS,EAAE,YAAY,EAAE,CAAC,CAAC,YAAY,EAAE,KAAK,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC;QAC9F,CAAC;IACH,CAAC;IAED,mFAAmF;IACnF,SAAS,CAAC,SAAiB;QACzB,MAAM,KAAK,GAAG,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QAC/C,OAAO,KAAK,KAAK,SAAS,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC;IACnD,CAAC;IAED,kEAAkE;IAClE,aAAa,CAAC,SAAiB;QAC7B,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACpC,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QAC3C,IAAI,KAAK,EAAE,CAAC;YACV,MAAM,CAAC,GAAG,KAAK,CAAC,aAAa,EAAE,CAAC;YAChC,IAAI,CAAC,CAAC,SAAS;gBAAE,IAAI,CAAC,GAAG,CAAC,SAAS,SAAS,oBAAoB,CAAC,CAAC,QAAQ,kBAAkB,CAAC,CAAC;YAC9F,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,CAAC;IACH,CAAC;CACF"}
|
|
@@ -105,6 +105,13 @@ export interface SessionRouterDeps {
|
|
|
105
105
|
queueMessage: (msg: InboundMessage, reason: string) => void;
|
|
106
106
|
raiseAttention: (title: string, body: string) => void;
|
|
107
107
|
markOwnerSuspect?: (machineId: string) => void;
|
|
108
|
+
/**
|
|
109
|
+
* A deliverMessage attempt REACHED the owner (queued/duplicate/stale acks all
|
|
110
|
+
* prove the peer is responsive). Wired to OwnerSuspectBreaker.recordSuccess
|
|
111
|
+
* so the per-peer suspect window closes the moment the peer answers — the
|
|
112
|
+
* other half of the markOwnerSuspect breaker (P19).
|
|
113
|
+
*/
|
|
114
|
+
onOwnerResponsive?: (machineId: string) => void;
|
|
108
115
|
sleep: (ms: number) => Promise<void>;
|
|
109
116
|
log?: (line: string) => void;
|
|
110
117
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"SessionRouter.d.ts","sourceRoot":"","sources":["../../src/core/SessionRouter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AAEH,OAAO,KAAK,EAAE,iBAAiB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAChF,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,WAAW,aAAa;IAC5B,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,QAAQ,GAAG,SAAS,GAAG,cAAc,GAAG,IAAI,CAAC;IACrD,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,UAAU;IACzB,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,EAAE,QAAQ,GAAG,WAAW,GAAG,iBAAiB,CAAC;CACtD;AAED,MAAM,WAAW,cAAc;IAC7B,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,aAAa,CAAC,EAAE,cAAc,CAAC;CAChC;AAED,MAAM,MAAM,WAAW,GACnB,iBAAiB,GACjB,WAAW,GACX,SAAS,GACT,QAAQ,GACR,WAAW,GACX,qBAAqB,GACrB,mBAAmB,CAAC;AAExB,MAAM,WAAW,YAAY;IAC3B,MAAM,EAAE,WAAW,CAAC;IACpB,KAAK,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB;2FACuF;IACvF,KAAK,EAAE,OAAO,CAAC;CAChB;AAED;;;;;;;;;;;;;;;;GAgBG;AACH,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,GAAG,IAAI,GAAG,OAAO,CAM9F;AAED,MAAM,WAAW,mBAAmB;IAClC,wBAAwB,EAAE,MAAM,CAAC;IACjC,iCAAiC,EAAE,MAAM,CAAC;IAC1C,+BAA+B,EAAE,MAAM,CAAC;IACxC,0EAA0E;IAC1E,iBAAiB,EAAE,MAAM,CAAC;CAC3B;AAED,eAAO,MAAM,qBAAqB,EAAE,mBAKnC,CAAC;AAEF,MAAM,WAAW,iBAAiB;IAChC,aAAa,EAAE,MAAM,CAAC;IACtB,SAAS,EAAE,iBAAiB,CAAC;IAC7B,eAAe,EAAE,MAAM,eAAe,EAAE,CAAC;IACzC,gBAAgB,EAAE,CAAC,UAAU,EAAE,MAAM,KAAK,aAAa,CAAC;IACxD,cAAc,EAAE,CAAC,SAAS,EAAE,MAAM,KAAK,OAAO,CAAC;IAC/C,iFAAiF;IACjF,iBAAiB,EAAE,CAAC,UAAU,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,EAAE,aAAa,EAAE,MAAM,KAAK;QAAE,EAAE,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IACpH,kFAAkF;IAClF,cAAc,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE;QAAE,UAAU,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,OAAO,CAAC;QAAC,cAAc,EAAE,MAAM,CAAA;KAAE,KAAK,OAAO,CAAC,UAAU,CAAC,CAAC;IAClJ,4DAA4D;IAC5D,aAAa,EAAE,CAAC,GAAG,EAAE,cAAc,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IACtD,+DAA+D;IAC/D,cAAc,EAAE,CAAC,SAAS,EAAE,MAAM,EAAE,GAAG,EAAE,cAAc,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IAC1E;;;;;;;;OAQG;IACH,YAAY,CAAC,EAAE,CAAC,UAAU,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC/D,YAAY,EAAE,CAAC,GAAG,EAAE,cAAc,EAAE,MAAM,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5D,cAAc,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,KAAK,IAAI,CAAC;IACtD,gBAAgB,CAAC,EAAE,CAAC,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC/C,KAAK,EAAE,CAAC,EAAE,EAAE,MAAM,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IACrC,GAAG,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,IAAI,CAAC;CAC9B;AAED,qBAAa,aAAa;IACxB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAoB;IACzC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAsB;IAC1C,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAuC;gBAElD,IAAI,EAAE,iBAAiB,EAAE,GAAG,GAAE,mBAA2C;IAKrF,qFAAqF;IACrF,KAAK,CAAC,GAAG,EAAE,cAAc,GAAG,OAAO,CAAC,YAAY,CAAC;
|
|
1
|
+
{"version":3,"file":"SessionRouter.d.ts","sourceRoot":"","sources":["../../src/core/SessionRouter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AAEH,OAAO,KAAK,EAAE,iBAAiB,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAChF,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,WAAW,aAAa;IAC5B,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,QAAQ,GAAG,SAAS,GAAG,cAAc,GAAG,IAAI,CAAC;IACrD,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,UAAU;IACzB,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,EAAE,QAAQ,GAAG,WAAW,GAAG,iBAAiB,CAAC;CACtD;AAED,MAAM,WAAW,cAAc;IAC7B,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,aAAa,CAAC,EAAE,cAAc,CAAC;CAChC;AAED,MAAM,MAAM,WAAW,GACnB,iBAAiB,GACjB,WAAW,GACX,SAAS,GACT,QAAQ,GACR,WAAW,GACX,qBAAqB,GACrB,mBAAmB,CAAC;AAExB,MAAM,WAAW,YAAY;IAC3B,MAAM,EAAE,WAAW,CAAC;IACpB,KAAK,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB;2FACuF;IACvF,KAAK,EAAE,OAAO,CAAC;CAChB;AAED;;;;;;;;;;;;;;;;GAgBG;AACH,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,GAAG,IAAI,GAAG,OAAO,CAM9F;AAED,MAAM,WAAW,mBAAmB;IAClC,wBAAwB,EAAE,MAAM,CAAC;IACjC,iCAAiC,EAAE,MAAM,CAAC;IAC1C,+BAA+B,EAAE,MAAM,CAAC;IACxC,0EAA0E;IAC1E,iBAAiB,EAAE,MAAM,CAAC;CAC3B;AAED,eAAO,MAAM,qBAAqB,EAAE,mBAKnC,CAAC;AAEF,MAAM,WAAW,iBAAiB;IAChC,aAAa,EAAE,MAAM,CAAC;IACtB,SAAS,EAAE,iBAAiB,CAAC;IAC7B,eAAe,EAAE,MAAM,eAAe,EAAE,CAAC;IACzC,gBAAgB,EAAE,CAAC,UAAU,EAAE,MAAM,KAAK,aAAa,CAAC;IACxD,cAAc,EAAE,CAAC,SAAS,EAAE,MAAM,KAAK,OAAO,CAAC;IAC/C,iFAAiF;IACjF,iBAAiB,EAAE,CAAC,UAAU,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,EAAE,aAAa,EAAE,MAAM,KAAK;QAAE,EAAE,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IACpH,kFAAkF;IAClF,cAAc,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE;QAAE,UAAU,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,OAAO,CAAC;QAAC,cAAc,EAAE,MAAM,CAAA;KAAE,KAAK,OAAO,CAAC,UAAU,CAAC,CAAC;IAClJ,4DAA4D;IAC5D,aAAa,EAAE,CAAC,GAAG,EAAE,cAAc,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IACtD,+DAA+D;IAC/D,cAAc,EAAE,CAAC,SAAS,EAAE,MAAM,EAAE,GAAG,EAAE,cAAc,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IAC1E;;;;;;;;OAQG;IACH,YAAY,CAAC,EAAE,CAAC,UAAU,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC/D,YAAY,EAAE,CAAC,GAAG,EAAE,cAAc,EAAE,MAAM,EAAE,MAAM,KAAK,IAAI,CAAC;IAC5D,cAAc,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,KAAK,IAAI,CAAC;IACtD,gBAAgB,CAAC,EAAE,CAAC,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC/C;;;;;OAKG;IACH,iBAAiB,CAAC,EAAE,CAAC,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAChD,KAAK,EAAE,CAAC,EAAE,EAAE,MAAM,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IACrC,GAAG,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,IAAI,CAAC;CAC9B;AAED,qBAAa,aAAa;IACxB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAoB;IACzC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAsB;IAC1C,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAuC;gBAElD,IAAI,EAAE,iBAAiB,EAAE,GAAG,GAAE,mBAA2C;IAKrF,qFAAqF;IACrF,KAAK,CAAC,GAAG,EAAE,cAAc,GAAG,OAAO,CAAC,YAAY,CAAC;IAgBjD,OAAO,CAAC,OAAO;YAID,WAAW;YAyBX,cAAc;YA0Cd,aAAa;CAkD5B"}
|
|
@@ -62,8 +62,17 @@ export class SessionRouter {
|
|
|
62
62
|
route(msg) {
|
|
63
63
|
const prior = this.chains.get(msg.sessionKey) ?? Promise.resolve();
|
|
64
64
|
const next = prior.catch(() => undefined).then(() => this.dispatchOne(msg));
|
|
65
|
-
// Track the tail so the next message on this session waits for this one to
|
|
66
|
-
this
|
|
65
|
+
// Track the tail so the next message on this session waits for this one to
|
|
66
|
+
// settle — and DELETE the entry once this tail settles while still current,
|
|
67
|
+
// so the map is bounded by in-flight sessions, not sessions-ever-routed
|
|
68
|
+
// (P19 cap: the map previously grew one settled-promise entry per session
|
|
69
|
+
// forever).
|
|
70
|
+
const tail = next.then(() => undefined, () => undefined);
|
|
71
|
+
this.chains.set(msg.sessionKey, tail);
|
|
72
|
+
void tail.then(() => {
|
|
73
|
+
if (this.chains.get(msg.sessionKey) === tail)
|
|
74
|
+
this.chains.delete(msg.sessionKey);
|
|
75
|
+
});
|
|
67
76
|
return next;
|
|
68
77
|
}
|
|
69
78
|
backoff(attempt) {
|
|
@@ -94,6 +103,9 @@ export class SessionRouter {
|
|
|
94
103
|
for (let attempt = 0; attempt <= this.cfg.deliverMessageMaxRetries; attempt++) {
|
|
95
104
|
try {
|
|
96
105
|
const ack = await this.deps.deliverMessage(owner, { sessionKey: msg.sessionKey, messageId: msg.messageId, payload: msg.payload, ownershipEpoch: epoch });
|
|
106
|
+
// ANY ack (queued/duplicate/stale) proves the peer answered — close its
|
|
107
|
+
// suspect window before interpreting the ack.
|
|
108
|
+
this.deps.onOwnerResponsive?.(owner);
|
|
97
109
|
if (ack.accepted === 'duplicate')
|
|
98
110
|
return { action: 'duplicate', owner, acked: true };
|
|
99
111
|
if (ack.accepted === 'queued')
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"SessionRouter.js","sourceRoot":"","sources":["../../src/core/SessionRouter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AA0CH;;;;;;;;;;;;;;;;GAgBG;AACH,MAAM,UAAU,iBAAiB,CAAC,OAAqB,EAAE,aAA4B;IACnF,IAAI,OAAO,CAAC,MAAM,KAAK,WAAW,IAAI,OAAO,CAAC,MAAM,KAAK,WAAW;QAAE,OAAO,IAAI,CAAC;IAClF,MAAM,WAAW,GAAG,OAAO,CAAC,KAAK,IAAI,IAAI,IAAI,OAAO,CAAC,KAAK,KAAK,aAAa,CAAC;IAC7E,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,IAAI,WAAW;QAAE,OAAO,IAAI,CAAC;IAC7D,IAAI,OAAO,CAAC,MAAM,KAAK,qBAAqB,IAAI,WAAW;QAAE,OAAO,IAAI,CAAC;IACzE,OAAO,KAAK,CAAC;AACf,CAAC;AAUD,MAAM,CAAC,MAAM,qBAAqB,GAAwB;IACxD,wBAAwB,EAAE,CAAC;IAC3B,iCAAiC,EAAE,GAAG;IACtC,+BAA+B,EAAE,IAAI;IACrC,iBAAiB,EAAE,CAAC;CACrB,CAAC;
|
|
1
|
+
{"version":3,"file":"SessionRouter.js","sourceRoot":"","sources":["../../src/core/SessionRouter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AA0CH;;;;;;;;;;;;;;;;GAgBG;AACH,MAAM,UAAU,iBAAiB,CAAC,OAAqB,EAAE,aAA4B;IACnF,IAAI,OAAO,CAAC,MAAM,KAAK,WAAW,IAAI,OAAO,CAAC,MAAM,KAAK,WAAW;QAAE,OAAO,IAAI,CAAC;IAClF,MAAM,WAAW,GAAG,OAAO,CAAC,KAAK,IAAI,IAAI,IAAI,OAAO,CAAC,KAAK,KAAK,aAAa,CAAC;IAC7E,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,IAAI,WAAW;QAAE,OAAO,IAAI,CAAC;IAC7D,IAAI,OAAO,CAAC,MAAM,KAAK,qBAAqB,IAAI,WAAW;QAAE,OAAO,IAAI,CAAC;IACzE,OAAO,KAAK,CAAC;AACf,CAAC;AAUD,MAAM,CAAC,MAAM,qBAAqB,GAAwB;IACxD,wBAAwB,EAAE,CAAC;IAC3B,iCAAiC,EAAE,GAAG;IACtC,+BAA+B,EAAE,IAAI;IACrC,iBAAiB,EAAE,CAAC;CACrB,CAAC;AAwCF,MAAM,OAAO,aAAa;IACP,IAAI,CAAoB;IACxB,GAAG,CAAsB;IACzB,MAAM,GAAG,IAAI,GAAG,EAA4B,CAAC;IAE9D,YAAY,IAAuB,EAAE,MAA2B,qBAAqB;QACnF,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;QACjB,IAAI,CAAC,GAAG,GAAG,GAAG,CAAC;IACjB,CAAC;IAED,qFAAqF;IACrF,KAAK,CAAC,GAAmB;QACvB,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,UAAU,CAAC,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC;QACnE,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC;QAC5E,2EAA2E;QAC3E,4EAA4E;QAC5E,wEAAwE;QACxE,0EAA0E;QAC1E,YAAY;QACZ,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,SAAS,EAAE,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;QACzD,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,UAAU,EAAE,IAAI,CAAC,CAAC;QACtC,KAAK,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE;YAClB,IAAI,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,UAAU,CAAC,KAAK,IAAI;gBAAE,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACnF,CAAC,CAAC,CAAC;QACH,OAAO,IAAI,CAAC;IACd,CAAC;IAEO,OAAO,CAAC,OAAe;QAC7B,OAAO,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,iCAAiC,GAAG,CAAC,IAAI,OAAO,EAAE,IAAI,CAAC,GAAG,CAAC,+BAA+B,CAAC,CAAC;IACvH,CAAC;IAEO,KAAK,CAAC,WAAW,CAAC,GAAmB;QAC3C,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,gBAAgB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QAEvD,IAAI,GAAG,CAAC,MAAM,KAAK,QAAQ,IAAI,GAAG,CAAC,KAAK,EAAE,CAAC;YACzC,IAAI,GAAG,CAAC,KAAK,KAAK,IAAI,CAAC,IAAI,CAAC,aAAa,EAAE,CAAC;gBAC1C,MAAM,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,CAAC;gBACnC,OAAO,EAAE,MAAM,EAAE,iBAAiB,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;YACtE,CAAC;YACD,IAAI,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;gBACxC,OAAO,IAAI,CAAC,cAAc,CAAC,GAAG,EAAE,GAAG,CAAC,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;YAC3D,CAAC;YACD,+DAA+D;YAC/D,IAAI,CAAC,IAAI,CAAC,gBAAgB,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;YACxC,OAAO,IAAI,CAAC,aAAa,CAAC,GAAG,EAAE,UAAU,EAAE,IAAI,CAAC,CAAC;QACnD,CAAC;QAED,IAAI,GAAG,CAAC,MAAM,KAAK,SAAS,IAAI,GAAG,CAAC,MAAM,KAAK,cAAc,EAAE,CAAC;YAC9D,IAAI,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,EAAE,sBAAsB,CAAC,CAAC;YACpD,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;QAChE,CAAC;QAED,2BAA2B;QAC3B,OAAO,IAAI,CAAC,aAAa,CAAC,GAAG,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;IAC/C,CAAC;IAEO,KAAK,CAAC,cAAc,CAAC,GAAmB,EAAE,KAAa,EAAE,KAAa,EAAE,cAAsB;QACpG,KAAK,IAAI,OAAO,GAAG,CAAC,EAAE,OAAO,IAAI,IAAI,CAAC,GAAG,CAAC,wBAAwB,EAAE,OAAO,EAAE,EAAE,CAAC;YAC9E,IAAI,CAAC;gBACH,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,KAAK,EAAE,EAAE,UAAU,EAAE,GAAG,CAAC,UAAU,EAAE,SAAS,EAAE,GAAG,CAAC,SAAS,EAAE,OAAO,EAAE,GAAG,CAAC,OAAO,EAAE,cAAc,EAAE,KAAK,EAAE,CAAC,CAAC;gBACzJ,wEAAwE;gBACxE,8CAA8C;gBAC9C,IAAI,CAAC,IAAI,CAAC,iBAAiB,EAAE,CAAC,KAAK,CAAC,CAAC;gBACrC,IAAI,GAAG,CAAC,QAAQ,KAAK,WAAW;oBAAE,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,KAAK,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;gBACrF,IAAI,GAAG,CAAC,QAAQ,KAAK,QAAQ;oBAAE,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,KAAK,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;gBAClF,yEAAyE;gBACzE,IAAI,cAAc,IAAI,IAAI,CAAC,GAAG,CAAC,iBAAiB,EAAE,CAAC;oBACjD,OAAO,IAAI,CAAC,aAAa,CAAC,GAAG,EAAE,UAAU,EAAE,IAAI,CAAC,CAAC;gBACnD,CAAC;gBACD,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,gBAAgB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;gBACxD,IAAI,IAAI,CAAC,MAAM,KAAK,QAAQ,IAAI,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,IAAI,CAAC,aAAa,EAAE,CAAC;oBACvE,MAAM,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,CAAC;oBACnC,OAAO,EAAE,MAAM,EAAE,iBAAiB,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,MAAM,EAAE,aAAa,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;gBAC9F,CAAC;gBACD,2EAA2E;gBAC3E,0EAA0E;gBAC1E,2EAA2E;gBAC3E,0EAA0E;gBAC1E,0EAA0E;gBAC1E,kEAAkE;gBAClE,IAAI,IAAI,CAAC,MAAM,KAAK,QAAQ,IAAI,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,KAAK,KAAK,IAAI,IAAI,CAAC,KAAK,KAAK,KAAK,CAAC,EAAE,CAAC;oBACrI,OAAO,IAAI,CAAC,cAAc,CAAC,GAAG,EAAE,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,cAAc,GAAG,CAAC,CAAC,CAAC;gBAC9E,CAAC;gBACD,yFAAyF;gBACzF,OAAO,IAAI,CAAC,aAAa,CAAC,GAAG,EAAE,UAAU,EAAE,IAAI,CAAC,CAAC;YACnD,CAAC;YAAC,OAAO,GAAG,EAAE,CAAC;gBACb,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,0BAA0B,OAAO,OAAO,KAAK,YAAY,MAAM,CAAE,GAAa,EAAE,OAAO,IAAI,GAAG,CAAC,EAAE,CAAC,CAAC;gBACnH,IAAI,OAAO,GAAG,IAAI,CAAC,GAAG,CAAC,wBAAwB,EAAE,CAAC;oBAChD,MAAM,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC;oBAC7C,SAAS;gBACX,CAAC;YACH,CAAC;QACH,CAAC;QACD,sEAAsE;QACtE,IAAI,CAAC,IAAI,CAAC,gBAAgB,EAAE,CAAC,KAAK,CAAC,CAAC;QACpC,OAAO,IAAI,CAAC,aAAa,CAAC,GAAG,EAAE,UAAU,EAAE,IAAI,CAAC,CAAC;IACnD,CAAC;IAEO,KAAK,CAAC,aAAa,CAAC,GAAmB,EAAE,MAA0B,EAAE,QAAiB;QAC5F,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;YAC1C,UAAU,EAAE,GAAG,CAAC,UAAU;YAC1B,aAAa,EAAE,GAAG,CAAC,aAAa,IAAI,EAAE;YACtC,eAAe,EAAE,IAAI,CAAC,IAAI,CAAC,eAAe,EAAE;YAC5C,YAAY,EAAE,SAAS;YACvB,MAAM;SACP,CAAC,CAAC;QAEH,IAAI,QAAQ,CAAC,OAAO,KAAK,mBAAmB,EAAE,CAAC;YAC7C,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,2BAA2B,EAAE,GAAG,GAAG,CAAC,UAAU,KAAK,QAAQ,CAAC,gBAAgB,IAAI,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;YAC5H,IAAI,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,EAAE,qBAAqB,QAAQ,CAAC,gBAAgB,IAAI,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;YACjG,OAAO,EAAE,MAAM,EAAE,mBAAmB,EAAE,MAAM,EAAE,QAAQ,CAAC,gBAAgB,IAAI,QAAQ,CAAC,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;QAC7G,CAAC;QACD,IAAI,QAAQ,CAAC,OAAO,KAAK,QAAQ,IAAI,CAAC,QAAQ,CAAC,aAAa,EAAE,CAAC;YAC7D,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,kCAAkC,EAAE,GAAG,GAAG,CAAC,UAAU,KAAK,QAAQ,CAAC,gBAAgB,IAAI,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;YACnI,IAAI,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,EAAE,QAAQ,CAAC,gBAAgB,IAAI,oBAAoB,CAAC,CAAC;YAC/E,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,CAAC,gBAAgB,IAAI,QAAQ,CAAC,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;QAClG,CAAC;QAED,sEAAsE;QACtE,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,gBAAgB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACvD,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,iBAAiB,CAAC,GAAG,CAAC,UAAU,EAAE,QAAQ,CAAC,aAAa,EAAE,GAAG,CAAC,KAAK,CAAC,CAAC;QAC3F,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,6EAA6E;YAC7E,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,gBAAgB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;YACxD,IAAI,IAAI,CAAC,MAAM,KAAK,QAAQ,IAAI,IAAI,CAAC,KAAK,KAAK,IAAI,CAAC,IAAI,CAAC,aAAa,EAAE,CAAC;gBACvE,MAAM,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,CAAC;gBACnC,OAAO,EAAE,MAAM,EAAE,iBAAiB,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,MAAM,EAAE,eAAe,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;YAChG,CAAC;YACD,IAAI,IAAI,CAAC,MAAM,KAAK,QAAQ,IAAI,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;gBACnF,OAAO,IAAI,CAAC,cAAc,CAAC,GAAG,EAAE,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;YAC7D,CAAC;YACD,IAAI,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,EAAE,sBAAsB,CAAC,CAAC;YACpD,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,sBAAsB,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;QAC5E,CAAC;QAED,kEAAkE;QAClE,MAAM,MAAM,GAAgB,QAAQ,CAAC,CAAC,CAAC,qBAAqB,CAAC,CAAC,CAAC,SAAS,CAAC;QACzE,IAAI,QAAQ,CAAC,aAAa,KAAK,IAAI,CAAC,IAAI,CAAC,aAAa,EAAE,CAAC;YACvD,MAAM,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,CAAC;YACnC,OAAO,EAAE,MAAM,EAAE,QAAQ,CAAC,CAAC,CAAC,qBAAqB,CAAC,CAAC,CAAC,iBAAiB,EAAE,KAAK,EAAE,QAAQ,CAAC,aAAa,EAAE,MAAM,EAAE,aAAa,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;QAC7I,CAAC;QACD,MAAM,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,QAAQ,CAAC,aAAa,EAAE,GAAG,CAAC,CAAC;QAC5D,iFAAiF;QACjF,gFAAgF;QAChF,gFAAgF;QAChF,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE,CAAC,GAAG,CAAC,UAAU,EAAE,QAAQ,CAAC,aAAa,CAAC,CAAC;QACjE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,QAAQ,CAAC,aAAa,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;IAChE,CAAC;CACF"}
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "./builtin-manifest.schema.json",
|
|
3
3
|
"schemaVersion": 1,
|
|
4
|
-
"generatedAt": "2026-06-06T04:
|
|
5
|
-
"instarVersion": "1.3.
|
|
4
|
+
"generatedAt": "2026-06-06T04:32:20.122Z",
|
|
5
|
+
"instarVersion": "1.3.338",
|
|
6
6
|
"entryCount": 199,
|
|
7
7
|
"entries": {
|
|
8
8
|
"hook:session-start": {
|
|
@@ -24,6 +24,24 @@ one-off rejection must not cost the session its conversation). And
|
|
|
24
24
|
`POST /sessions/refresh` accepts `fresh: true`, reaching the same
|
|
25
25
|
SessionRefresh fresh-mode the sentinel uses internally.
|
|
26
26
|
|
|
27
|
+
Audit fix #6 under "No Unbounded Loops" (P19): the SessionRouter's per-peer
|
|
28
|
+
breaker hook (`markOwnerSuspect`) — which fires on delivery-retry exhaustion —
|
|
29
|
+
was never wired in production, and `isMachineAlive` reads only capacity
|
|
30
|
+
heartbeats, so every session owned by a slow-but-heartbeating machine re-paid
|
|
31
|
+
the full ~4.5s retry tax per message. Now `OwnerSuspectBreaker` (pure core
|
|
32
|
+
class, per-peer half-open windows with ABSOLUTE per-episode TTL, composing
|
|
33
|
+
`FailureEpisodeLatch` for one-log/one-signal episode accounting) is wired into
|
|
34
|
+
`markOwnerSuspect` + composed into `isMachineAlive` + filtered into placement
|
|
35
|
+
candidates (with an all-suspect unfiltered fallback). A new router dep
|
|
36
|
+
`onOwnerResponsive` closes the window on ANY delivery ack. Also fixed: the
|
|
37
|
+
router's per-session `chains` map leaked one settled entry per session-ever-
|
|
38
|
+
routed (now bounded by in-flight sessions). The adversarial reviewer
|
|
39
|
+
reproduced and fixed a forever-suspect bug in the first draft (re-marks
|
|
40
|
+
extended the TTL → a recovered busy peer stayed written off indefinitely —
|
|
41
|
+
the absolute TTL is their fix, regression-tested). Suspect-window message
|
|
42
|
+
POLICY (re-place fast vs hold-for-stability) is deliberately unchanged —
|
|
43
|
+
operator decision, options sent separately.
|
|
44
|
+
|
|
27
45
|
## What to Tell Your User
|
|
28
46
|
|
|
29
47
|
If a conversation ever dies the way the EXO session did — every reply
|
|
@@ -32,6 +50,14 @@ transcript — I now notice it, can recover it automatically with a fresh
|
|
|
32
50
|
restart (when auto-recovery is enabled), and explain what happened instead of
|
|
33
51
|
leaving you staring at "delivered" receipts with no answers.
|
|
34
52
|
|
|
53
|
+
If you run me on more than one machine: when one machine starts failing to
|
|
54
|
+
receive its conversations' messages, the router now learns it once (instead of
|
|
55
|
+
re-discovering it ~4.5 seconds at a time, per conversation, per message),
|
|
56
|
+
routes around it, re-checks every 30 seconds, and picks the machine back up
|
|
57
|
+
the moment it answers again. Pinned conversations stay put through brief
|
|
58
|
+
blips. If a machine stays unreachable for 10+ minutes you get one note in the
|
|
59
|
+
health log.
|
|
60
|
+
|
|
35
61
|
## Summary of New Capabilities
|
|
36
62
|
|
|
37
63
|
- ContextWedgeSentinel detects the AUP-rejection loop (second signature
|
|
@@ -45,6 +71,10 @@ leaving you staring at "delivered" receipts with no answers.
|
|
|
45
71
|
family, the API lever, and the prevention rule (adversarial payloads live in
|
|
46
72
|
files on disk, never pasted into conversation transcripts).
|
|
47
73
|
|
|
74
|
+
- Per-machine delivery circuit breaker (30s half-open windows, instant
|
|
75
|
+
recovery on any successful delivery, sustained-failure health-log signal).
|
|
76
|
+
No configuration needed.
|
|
77
|
+
|
|
48
78
|
## Evidence
|
|
49
79
|
|
|
50
80
|
Live incident recovery (2026-06-05, topic 19437): manual kill + resume-map
|
|
@@ -54,3 +84,14 @@ where the wedged one had failed every turn for ~1h. Unit: 32 sentinel tests
|
|
|
54
84
|
wording per family) + 8 route tests (fresh validation, forwarding, 202 shape)
|
|
55
85
|
+ 3 migrator tests (fresh install carries the note; existing-section patch;
|
|
56
86
|
idempotency). Integration + e2e wedge suites green; tsc clean; preflight PASS.
|
|
87
|
+
|
|
88
|
+
CMT-1109 audit, grounded to three concrete findings (unwired hook;
|
|
89
|
+
heartbeat-only aliveness; no-op queue dep). Adversarial second-pass: probe-1
|
|
90
|
+
OBJECT confirmed by reproduction (forever-suspect under steady traffic) and
|
|
91
|
+
fixed in-review with a regression test; probes 2–5 (swap-count invariance,
|
|
92
|
+
pin behavior through suspect windows, all-suspect fallback can't wedge,
|
|
93
|
+
stale-ack semantics, chains-cleanup serialization race) traced clean. Tests:
|
|
94
|
+
40 green across the breaker unit suite (incl. the P19 sustained-suspicion
|
|
95
|
+
bound and the end-to-end zero-retry-tax short-circuit), SessionRouter (23),
|
|
96
|
+
dispatch integration (3), and the session-pool deliverMessage e2e (4); tsc
|
|
97
|
+
clean.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Side-Effects Review — Owner-Suspect Breaker
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `owner-suspect-breaker`
|
|
4
|
+
**Date:** `2026-06-06`
|
|
5
|
+
**Author:** `Echo (instar-dev agent, autonomous session per Justin's direction)`
|
|
6
|
+
**Second-pass reviewer:** `adversarial reviewer subagent — OBJECT on probe 1 (forever-suspect under load), CONFIRMED BY REPRODUCTION and fixed in-review (absolute per-episode TTL + regression test); probes 2–5 CONCUR`
|
|
7
|
+
|
|
8
|
+
## Summary of the change
|
|
9
|
+
|
|
10
|
+
Wires the SessionRouter's previously-inert `markOwnerSuspect` hook into a real per-peer circuit: `OwnerSuspectBreaker` (pure core; absolute-TTL half-open windows, default 30s; per-peer `FailureEpisodeLatch` for first-log/one-signal/recovery accounting; state deleted on success). Wiring composes `!isSuspect` into `isMachineAlive`, filters suspect machines from placement candidates (all-suspect → unfiltered fallback), and the new `onOwnerResponsive` router dep closes windows on any delivery ack. Plus the router `chains`-map leak fix (entries deleted when their tail settles while current). Files: `OwnerSuspectBreaker.ts` (new), `SessionRouter.ts` (two small additions), `server.ts` wiring block, tests.
|
|
11
|
+
|
|
12
|
+
## Decision-point inventory
|
|
13
|
+
|
|
14
|
+
- `isMachineAlive` composition — **modify** — a suspect peer reads as not-alive for ROUTING, sending its sessions down the EXISTING failover re-place path. The decision of where they land is unchanged (placement); the decision of when a peer is suspect is new (retry exhaustion, the signal the router already emitted).
|
|
15
|
+
- Placement candidate filter — **modify** — suspect machines excluded unless that empties the set.
|
|
16
|
+
- `OwnerSuspectBreaker` — **add** — per-peer state machine consuming the router's existing exhaustion signal.
|
|
17
|
+
- `chains` cleanup + `onOwnerResponsive` — **add (hygiene/signal)**.
|
|
18
|
+
|
|
19
|
+
## 1. Over-block
|
|
20
|
+
|
|
21
|
+
The reviewer's probe-1 OBJECT was exactly an over-block: with window-extension semantics, a steady <TTL message stream re-marked a suspect peer on every dispatch (no delivery is attempted while suspect → `recordSuccess` unreachable → TTL the only exit → extended forever). Reproduced: a peer healthy from t=40s stayed suspect at t=10min with 31 forced failovers. **Fixed in-review**: `markSuspect` no longer extends an open window (absolute per-episode TTL) — half-open is reached on schedule regardless of traffic, regression-tested. Residual over-block: a genuinely-healthy peer that failed one message's retries pays one 30s window — bounded, and any successful delivery (e.g. a different session's forward in the half-open probe) clears it instantly.
|
|
22
|
+
|
|
23
|
+
## 2. Under-block
|
|
24
|
+
|
|
25
|
+
(a) Suspect-window message POLICY is deliberately out of scope: messages still take the existing re-place path (the only message-preserving path while `queueMessage` is a production no-op). The queue-vs-replace stability trade — and the durable-queue investment it requires — is an operator decision; lettered options go to Justin <!-- tracked: CMT-1109 -->. (b) The no-op `queueMessage` also affects the `placing/transferring` branch (pre-existing; surfaced in the options message). (c) `spawnOnMachine` remains un-breakered (reviewer probe 3: deliberate — the all-suspect fallback depends on it attempting, and it throws-to-caller on failure).
|
|
26
|
+
|
|
27
|
+
## 3. Level-of-abstraction fit / 4. Signal vs authority
|
|
28
|
+
|
|
29
|
+
The breaker consumes the router's OWN exhaustion signal and feeds the router's OWN aliveness dep — no new decision-maker, no second routing brain; placement, CAS, and ownership semantics untouched. Per `docs/signal-vs-authority.md`: the suspect window is a routing-availability signal with bounded lifetime; the failover authority it triggers is the pre-existing path. Composition lives in the wiring (the router stays pure/dep-shaped).
|
|
30
|
+
|
|
31
|
+
## 5. Interactions
|
|
32
|
+
|
|
33
|
+
- **Swap count ("fewer swaps" directive):** unchanged — a session moves at most once per peer-down episode; the breaker removes the OTHER sessions' retry tax, not adds moves (reviewer probe 1 trace, post-fix).
|
|
34
|
+
- **Pins:** a HARD-pinned session does not migrate during a suspect window (placement returns `hard-pin-unavailable`; resolves in place after the TTL — reviewer probe 2).
|
|
35
|
+
- **All-suspect fallback:** placement proceeds unfiltered; `spawnOnMachine` is not `isMachineAlive`-gated so it attempts and throws-to-caller — no wedge (probe 3).
|
|
36
|
+
- **Stale-ownership acks** close the window — correct: the breaker measures transport health to the peer, which any ack proves (probe 4).
|
|
37
|
+
- **Chains cleanup race:** identity-check guard preserves per-session serialization under concurrent re-route; traced clean (probe 5).
|
|
38
|
+
- **Sustained-suspicion signal:** one DegradationReporter record per 10min episode per peer (`SessionPool.ownerDelivery`); reporter's internal cooldown bounds user-facing volume.
|
|
39
|
+
|
|
40
|
+
## 6. External surfaces / 7. Rollback
|
|
41
|
+
|
|
42
|
+
Logs + degradation records only; no API/schema/config/persistent state; no migration. Rollback = revert (the wiring block removal restores the inert-hook status quo; the leak and retry-tax return).
|
|
43
|
+
|
|
44
|
+
## Conclusion
|
|
45
|
+
|
|
46
|
+
The audit's vaguest lead ("load-shedding") resolved into the sharpest finding of the night: a fully-designed breaker hook shipped dead at the wiring layer. The fix is composition, not invention — and the adversarial pass caught a genuine inversion (the busier a recovered peer, the longer it stayed exiled) before it ever ran. Policy beyond the existing semantics goes to the operator, as it should.
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
|
|
50
|
+
## Phase 5 — Second-pass review (routing authority → required)
|
|
51
|
+
|
|
52
|
+
The adversarial reviewer ran six probes at line level with live reproduction: (1) forever-suspect under steady traffic — OBJECT, reproduced (peer healthy at t=40s still suspect at t=10min, 31 forced failovers), fixed in-review (absolute per-episode TTL in `markSuspect`) + regression test; (2) swap-count + pin behavior — no added moves; pinned sessions hold through windows; (3) all-suspect fallback — cannot wedge (`spawnOnMachine` un-gated, throws-to-caller); (4) stale-ack window-clearing — semantically correct for a transport-health breaker; (5) chains-cleanup serialization — identity guard traced clean; (6) ran breaker/router unit + dispatch integration + session-pool e2e suites and tsc — all green. **Verdict: CONCUR with the applied fix.**
|