switchroom 0.19.4 → 0.19.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/auth-broker/index.js +7 -3
- package/dist/cli/autoaccept-poll.js +8 -2
- package/dist/cli/switchroom.js +20 -5
- package/dist/host-control/main.js +1 -1
- package/package.json +1 -1
- package/profiles/_base/start.sh.hbs +67 -4
- package/telegram-plugin/dist/gateway/gateway.js +585 -302
- package/telegram-plugin/flushed-turn-supersede.ts +43 -7
- package/telegram-plugin/gateway/command-format.ts +253 -0
- package/telegram-plugin/gateway/gateway-heartbeat.ts +72 -0
- package/telegram-plugin/gateway/gateway.ts +128 -259
- package/telegram-plugin/gateway/hang-restart-decision.ts +189 -0
- package/telegram-plugin/gateway/liveness-wiring.ts +35 -1
- package/telegram-plugin/gateway/outbound-send-path.ts +51 -11
- package/telegram-plugin/gateway/pending-inbound-buffer.ts +27 -0
- package/telegram-plugin/gateway/session-model-file.ts +13 -0
- package/telegram-plugin/gateway/stream-render.ts +18 -1
- package/telegram-plugin/gateway/subagent-handback-marker.ts +42 -0
- package/telegram-plugin/gateway/turn-active-marker.ts +29 -17
- package/telegram-plugin/gateway/worker-feed-dispatch.ts +139 -0
- package/telegram-plugin/hooks/silent-end-interrupt-stop.mjs +87 -36
- package/telegram-plugin/hooks/silent-end-scan.mjs +263 -3
- package/telegram-plugin/render/line-start-guard.ts +76 -4
- package/telegram-plugin/reply-owner-resolve.ts +43 -7
- package/telegram-plugin/rich-send.ts +8 -1
- package/telegram-plugin/tests/command-format.test.ts +212 -0
- package/telegram-plugin/tests/flushed-turn-supersede.test.ts +89 -0
- package/telegram-plugin/tests/gateway-heartbeat.test.ts +70 -0
- package/telegram-plugin/tests/hang-restart-decision.test.ts +146 -0
- package/telegram-plugin/tests/hang-restart-marker-integration.test.ts +98 -0
- package/telegram-plugin/tests/narrative-lane-golden.test.ts +2 -1
- package/telegram-plugin/tests/render/heading-guard-blockquote-glued-hash.test.ts +86 -0
- package/telegram-plugin/tests/render/heading-guard.test.ts +114 -0
- package/telegram-plugin/tests/render/rich-corpus-seam-regression.test.ts +76 -0
- package/telegram-plugin/tests/reply-owner-resolve.test.ts +74 -0
- package/telegram-plugin/tests/send-reply-golden.test.ts +221 -6
- package/telegram-plugin/tests/silent-end-interrupt-stop-integration.test.ts +63 -0
- package/telegram-plugin/tests/silent-end-interrupt-stop-scan.test.ts +60 -16
- package/telegram-plugin/tests/silent-end-single-writer-election.test.ts +193 -0
- package/telegram-plugin/tests/silent-end.test.ts +60 -5
- package/telegram-plugin/tests/stream-render-golden.test.ts +2 -1
- package/telegram-plugin/tests/subagent-handback-marker.test.ts +36 -0
- package/telegram-plugin/tests/worker-feed-origin-race-defer.test.ts +321 -0
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* hang-restart-decision.ts — the progress-based hang-restart discriminator
|
|
3
|
+
* (Stage B safety net).
|
|
4
|
+
*
|
|
5
|
+
* WHY THIS EXISTS
|
|
6
|
+
*
|
|
7
|
+
* The silence-poke framework fallback (`liveness-wiring.ts onFrameworkFallback`)
|
|
8
|
+
* is the last-resort unwedge, but it only tears down GATEWAY state — it never
|
|
9
|
+
* kills the wedged `claude` child. So a turn that hangs mid-tool-call recovers
|
|
10
|
+
* its conversation slot but leaves the child stuck; the "carrie" incident
|
|
11
|
+
* needed a MANUAL restart. Stage B closes that: when the fallback fires with a
|
|
12
|
+
* tool still mid-call AND no real progress, escalate to an actual restart via
|
|
13
|
+
* the SIGTERM-PID1 path, then let the boot classifier route recovery through
|
|
14
|
+
* the ask-first `resume_watchdog_timeout` inbound (never assertive auto-resume).
|
|
15
|
+
*
|
|
16
|
+
* THE DISCRIMINATOR (the crux — do NOT key on "tool open N seconds")
|
|
17
|
+
*
|
|
18
|
+
* A healthy 20-minute research turn must NOT be killed. The honest progress
|
|
19
|
+
* signal is the turn-active marker's mtime: it is touched on every `tool_use`
|
|
20
|
+
* AND on foreground sub-agent JSONL growth (`subagent-watcher.ts`), so a turn
|
|
21
|
+
* that keeps advancing it is working; a stale mtime means work stopped.
|
|
22
|
+
*
|
|
23
|
+
* markerAgeMs (from readTurnActiveMarkerAgeMs) small → progress → NOT a hang
|
|
24
|
+
* markerAgeMs large, or null (no progress signal at all) → stale → hang
|
|
25
|
+
*
|
|
26
|
+
* THE MARKER-ADVANCEMENT BLIND SPOT + THE TOOL-CLASS GUARD (review Finding A)
|
|
27
|
+
*
|
|
28
|
+
* The marker advances only on `tool_use` and sub-agent JSONL growth. A healthy
|
|
29
|
+
* long SINGLE FOREGROUND tool with no sub-agent and no interim tool_use never
|
|
30
|
+
* advances it — a 15-min foreground `Bash` build/test, a big `WebFetch`/crawl,
|
|
31
|
+
* a multi-minute research call. At the 15-min silence ceiling that turn is
|
|
32
|
+
* mid-tool with a stale marker and would be WRONGLY restarted by marker
|
|
33
|
+
* staleness alone. So we additionally protect known-long-running tool classes:
|
|
34
|
+
* if any in-flight tool is a `background`/long-fetch/research class, we do NOT
|
|
35
|
+
* restart, regardless of marker age.
|
|
36
|
+
*
|
|
37
|
+
* RESIDUAL RISK (documented, not silently accepted): a genuine hang INSIDE a
|
|
38
|
+
* protected long-running tool (a truly-wedged `Bash`, a `WebFetch` that never
|
|
39
|
+
* returns) is indistinguishable from healthy long work by any signal the
|
|
40
|
+
* gateway has — the marker cannot advance mid-single-tool either way. Those
|
|
41
|
+
* are NOT auto-restarted here; recovery for that class relies on the tool's
|
|
42
|
+
* own timeout, an operator `/restart`, or the ordinary state-teardown. We
|
|
43
|
+
* choose to never false-kill a healthy long tool over catching the rare
|
|
44
|
+
* hung-inside-a-long-tool case, because the false-kill is the common,
|
|
45
|
+
* user-visible harm and the discriminator exists precisely to avoid it.
|
|
46
|
+
*
|
|
47
|
+
* Recovery routes through the ask-first resume path so a turn that would hang
|
|
48
|
+
* again the same way is reported to the user, not silently re-run — that is
|
|
49
|
+
* also the storm guard (see § STORM GUARD below).
|
|
50
|
+
*
|
|
51
|
+
* Pure + deterministic: the clock, the marker age, and the in-flight tool names
|
|
52
|
+
* are all injected. The gateway owns the SIGTERM side effect.
|
|
53
|
+
*/
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* Tool classes ported from Stage A's tool-call-deadline taxonomy. `background`
|
|
57
|
+
* covers sub-agent dispatch + long Bash; `human` covers approval waits;
|
|
58
|
+
* `standard` is everything else. Stage B additionally treats a broader set of
|
|
59
|
+
* single-foreground long tools as hang-restart-protected (see
|
|
60
|
+
* `isHangRestartProtectedTool`).
|
|
61
|
+
*/
|
|
62
|
+
export type ToolDeadlineClass = 'standard' | 'human' | 'background'
|
|
63
|
+
|
|
64
|
+
/** Human-in-the-loop waits. */
|
|
65
|
+
const HUMAN_WAIT_TOOLS = new Set<string>(['ask_user'])
|
|
66
|
+
/** Sub-agent dispatch — progress is sub-agent JSONL growth. */
|
|
67
|
+
const BACKGROUND_TOOLS = new Set<string>(['Task', 'Agent'])
|
|
68
|
+
|
|
69
|
+
/**
|
|
70
|
+
* Classify a tool into the Stage A taxonomy. `Bash` with `run_in_background` is
|
|
71
|
+
* `background`; `Task`/`Agent` are `background`; `ask_user` is `human`.
|
|
72
|
+
*/
|
|
73
|
+
export function classifyToolClass(
|
|
74
|
+
toolName: string,
|
|
75
|
+
opts: { awaitingApproval?: boolean; backgroundBash?: boolean } = {},
|
|
76
|
+
): ToolDeadlineClass {
|
|
77
|
+
if (opts.awaitingApproval === true) return 'human'
|
|
78
|
+
if (HUMAN_WAIT_TOOLS.has(toolName)) return 'human'
|
|
79
|
+
if (BACKGROUND_TOOLS.has(toolName)) return 'background'
|
|
80
|
+
if (toolName === 'Bash' && opts.backgroundBash === true) return 'background'
|
|
81
|
+
return 'standard'
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/**
|
|
85
|
+
* Exact-name tools that legitimately run long as a SINGLE foreground call
|
|
86
|
+
* without touching the marker, so their staleness is not a hang signal.
|
|
87
|
+
*/
|
|
88
|
+
const HANG_PROTECTED_EXACT = new Set<string>([
|
|
89
|
+
'Task', 'Agent', // sub-agent dispatch (background class)
|
|
90
|
+
'Bash', // foreground builds/tests routinely run many minutes
|
|
91
|
+
'WebFetch', 'WebSearch',
|
|
92
|
+
])
|
|
93
|
+
|
|
94
|
+
/**
|
|
95
|
+
* Name substrings marking long-running fetch/crawl/research MCP tools (matched
|
|
96
|
+
* case-insensitively) — perplexity_research, deep research, site crawls, etc.
|
|
97
|
+
*/
|
|
98
|
+
const HANG_PROTECTED_SUBSTRINGS = ['research', 'perplexity', 'crawl', 'deep_research', 'webkite']
|
|
99
|
+
|
|
100
|
+
/**
|
|
101
|
+
* True when a tool is a known-long-running / opaque-progress class that must
|
|
102
|
+
* NOT be hang-restarted on marker staleness (review Finding A). Covers the
|
|
103
|
+
* `background` class plus foreground Bash, web fetch/search, and research/crawl
|
|
104
|
+
* MCP tools. `ask_user` (`human`) is also protected — an operator wait is not a
|
|
105
|
+
* hang. See the module header § RESIDUAL RISK for the tradeoff.
|
|
106
|
+
*/
|
|
107
|
+
export function isHangRestartProtectedTool(toolName: string): boolean {
|
|
108
|
+
if (HANG_PROTECTED_EXACT.has(toolName)) return true
|
|
109
|
+
if (classifyToolClass(toolName) !== 'standard') return true // background / human
|
|
110
|
+
const lower = toolName.toLowerCase()
|
|
111
|
+
return HANG_PROTECTED_SUBSTRINGS.some((s) => lower.includes(s))
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
export interface HangRestartInput {
|
|
115
|
+
/** Names of the tools in flight when the fallback fired (silence-poke
|
|
116
|
+
* snapshot). Empty ⇒ not a mid-tool fallback (ordinary teardown owns it). */
|
|
117
|
+
inFlightToolNames: string[]
|
|
118
|
+
/** Turn-active marker mtime age in ms (`readTurnActiveMarkerAgeMs`), or null
|
|
119
|
+
* when the marker is absent/unstattable (no progress signal). */
|
|
120
|
+
markerAgeMs: number | null
|
|
121
|
+
/** Staleness ceiling: marker age at/above this counts as "no progress". Keyed
|
|
122
|
+
* to the same TURN_HANG_SECS the boot classifier uses so the live decision
|
|
123
|
+
* and the boot reclassification agree on what "stalled" means. */
|
|
124
|
+
stalenessThresholdMs: number
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
export interface HangRestartDecision {
|
|
128
|
+
restart: boolean
|
|
129
|
+
reason: string
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
/**
|
|
133
|
+
* Decide whether a mid-tool framework-fallback should escalate to a real
|
|
134
|
+
* restart. Pure.
|
|
135
|
+
*
|
|
136
|
+
* - not mid-tool → no restart (ordinary teardown path owns it)
|
|
137
|
+
* - a protected long tool in flight → no restart (healthy long tool / sub-
|
|
138
|
+
* agent / research — the crux false-positive we
|
|
139
|
+
* must never kill; see § RESIDUAL RISK)
|
|
140
|
+
* - marker still advancing → no restart (real progress)
|
|
141
|
+
* - marker stale or absent → RESTART (genuine hang: mid-tool + no progress
|
|
142
|
+
* + no protected long tool that would explain it)
|
|
143
|
+
*/
|
|
144
|
+
export function decideHangRestart(input: HangRestartInput): HangRestartDecision {
|
|
145
|
+
if (input.inFlightToolNames.length === 0) return { restart: false, reason: 'not-mid-tool' }
|
|
146
|
+
// Finding A: never restart while a known-long-running tool is in flight — a
|
|
147
|
+
// stale marker is EXPECTED for a single long foreground tool.
|
|
148
|
+
const protectedName = input.inFlightToolNames.find(isHangRestartProtectedTool)
|
|
149
|
+
if (protectedName != null) {
|
|
150
|
+
return { restart: false, reason: `protected-long-tool:${protectedName}` }
|
|
151
|
+
}
|
|
152
|
+
// A working turn keeps touching the marker (tool_use + sub-agent JSONL
|
|
153
|
+
// growth), so a SMALL non-null age means real progress.
|
|
154
|
+
if (input.markerAgeMs != null && input.markerAgeMs < input.stalenessThresholdMs) {
|
|
155
|
+
return { restart: false, reason: 'marker-advancing' }
|
|
156
|
+
}
|
|
157
|
+
return { restart: true, reason: 'mid-tool-marker-stale' }
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
/** Default staleness ceiling (ms) — mirrors the watchdog's TURN_HANG_SECS=300. */
|
|
161
|
+
export const DEFAULT_HANG_STALENESS_MS = 300_000
|
|
162
|
+
|
|
163
|
+
/**
|
|
164
|
+
* Resolve the staleness ceiling from the environment, keyed to the SAME
|
|
165
|
+
* `TURN_HANG_SECS` the boot classifier reads, so a live kill and the boot
|
|
166
|
+
* reclassification agree on "stalled". Blank/garbage/non-positive → default.
|
|
167
|
+
*/
|
|
168
|
+
export function hangStalenessMs(
|
|
169
|
+
env: Record<string, string | undefined> = process.env,
|
|
170
|
+
): number {
|
|
171
|
+
const raw = env.TURN_HANG_SECS
|
|
172
|
+
const n = Number(raw)
|
|
173
|
+
if (!Number.isFinite(n) || n <= 0) return DEFAULT_HANG_STALENESS_MS
|
|
174
|
+
return Math.floor(n * 1000)
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
/**
|
|
178
|
+
* § STORM GUARD (review Finding B)
|
|
179
|
+
*
|
|
180
|
+
* There is deliberately NO restart cooldown here. A cooldown could only engage
|
|
181
|
+
* if two hang-restarts fired close together, but two mid-tool framework
|
|
182
|
+
* fallbacks are >= the 15-min silence-defer ceiling apart (they only fire after
|
|
183
|
+
* that ceiling), so a sub-15-min cooldown can never engage on the organic path
|
|
184
|
+
* — a dead guard giving false confidence. The REAL storm guard is the ask-first
|
|
185
|
+
* `resume_watchdog_timeout` recovery: after a hang-restart the agent does NOT
|
|
186
|
+
* auto-resume the wedged work — it reports to the user and asks — so the same
|
|
187
|
+
* hang cannot immediately recur without fresh human action. That is sufficient
|
|
188
|
+
* and self-documenting; a persisted cooldown file is not needed.
|
|
189
|
+
*/
|
|
@@ -24,7 +24,8 @@ import { emitRuntimeMetric } from '../runtime-metrics.js'
|
|
|
24
24
|
import { logStreamingEvent } from '../streaming-metrics.js'
|
|
25
25
|
import { clearSilentEndState } from '../silent-end.js'
|
|
26
26
|
import { purgeStaleTurnsForChat } from './turn-state-purge.js'
|
|
27
|
-
import { removeTurnActiveMarker } from './turn-active-marker.js'
|
|
27
|
+
import { removeTurnActiveMarker, readTurnActiveMarkerAgeMs } from './turn-active-marker.js'
|
|
28
|
+
import { decideHangRestart } from './hang-restart-decision.js'
|
|
28
29
|
import { recordTurnEnd } from '../registry/turns-schema.js'
|
|
29
30
|
import { hostdGetStatusOnce } from './hostd-dispatch.js'
|
|
30
31
|
import { formatUpdateStatusLine } from './update-status-line.js'
|
|
@@ -60,6 +61,7 @@ export function buildSilencePokeOptions(deps: LivenessWiringDeps): Parameters<ty
|
|
|
60
61
|
trackRedeliveredInbound,
|
|
61
62
|
closeActivityLane,
|
|
62
63
|
closeProgressLane,
|
|
64
|
+
hangRestart,
|
|
63
65
|
} = deps
|
|
64
66
|
|
|
65
67
|
return {
|
|
@@ -160,6 +162,38 @@ export function buildSilencePokeOptions(deps: LivenessWiringDeps): Parameters<ty
|
|
|
160
162
|
return
|
|
161
163
|
}
|
|
162
164
|
|
|
165
|
+
// Stage B: escalate a mid-tool hang to a REAL restart. The fallback fires
|
|
166
|
+
// here with a tool still in flight only after silence crossed the 15-min
|
|
167
|
+
// defer ceiling (isLegitimatelyWorking held it that long). The honest
|
|
168
|
+
// hang/health discriminator is the turn-active marker's mtime age: a healthy
|
|
169
|
+
// turn keeps advancing it (tool_use + sub-agent JSONL growth), a true hang
|
|
170
|
+
// lets it go stale. A known-long-running tool in flight (Task/Agent/Bash/
|
|
171
|
+
// WebFetch/research) is protected regardless of marker age — a single long
|
|
172
|
+
// foreground tool legitimately can't advance the marker (Finding A). Only a
|
|
173
|
+
// stale (or absent) marker with no protected long tool triggers the
|
|
174
|
+
// SIGTERM-PID1 restart; the boot classifier then routes recovery through the
|
|
175
|
+
// ask-first resume_watchdog_timeout inbound. We return BEFORE the state-
|
|
176
|
+
// teardown so the turn-active marker survives for the boot reclassification.
|
|
177
|
+
if (hangRestart != null && ctx.inFlightTools.length > 0) {
|
|
178
|
+
const markerAgeMs = readTurnActiveMarkerAgeMs(STATE_DIR)
|
|
179
|
+
const inFlightToolNames = ctx.inFlightTools.map((t) => t.name)
|
|
180
|
+
const decision = decideHangRestart({
|
|
181
|
+
inFlightToolNames,
|
|
182
|
+
markerAgeMs,
|
|
183
|
+
stalenessThresholdMs: hangRestart.stalenessThresholdMs,
|
|
184
|
+
})
|
|
185
|
+
process.stderr.write(
|
|
186
|
+
`telegram gateway: [hang-watchdog] fallback mid-tool chat=${ctx.chatId} ` +
|
|
187
|
+
`thread=${ctx.threadId ?? '-'} silence_ms=${ctx.silenceMs} ` +
|
|
188
|
+
`tools=${inFlightToolNames.join(',')} marker_age_ms=${markerAgeMs ?? 'null'} ` +
|
|
189
|
+
`restart=${decision.restart} reason=${decision.reason}\n`,
|
|
190
|
+
)
|
|
191
|
+
if (decision.restart) {
|
|
192
|
+
hangRestart.request(decision.reason, markerAgeMs ?? ctx.silenceMs)
|
|
193
|
+
return
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
|
|
163
197
|
// Deterministic in-flight update status (klanker incident). If this
|
|
164
198
|
// gateway dispatched an update_apply that's still running, the
|
|
165
199
|
// recurring framework fallback carries hostd's REAL phase + elapsed
|
|
@@ -56,9 +56,10 @@ import { decideSilentReplyAnchor } from '../silent-reply-anchor.js'
|
|
|
56
56
|
import {
|
|
57
57
|
decideSupersedeCorrection,
|
|
58
58
|
flushedAnswerMatchesReply,
|
|
59
|
+
DEFAULT_SUPERSEDE_TTL_MS,
|
|
59
60
|
type FlushedTurnSupersedeRegistry,
|
|
60
61
|
} from '../flushed-turn-supersede.js'
|
|
61
|
-
import { decideAnswerLatchSuppression } from '../reply-owner-resolve.js'
|
|
62
|
+
import { decideAnswerLatchSuppression, type ReplyOwnerTier } from '../reply-owner-resolve.js'
|
|
62
63
|
import { deriveTelegraphTitle } from '../telegraph.js'
|
|
63
64
|
import {
|
|
64
65
|
mintVoiceOnDemandToken,
|
|
@@ -658,7 +659,7 @@ export interface SendReplyGatewayDeps {
|
|
|
658
659
|
assertSendable(f: string): void
|
|
659
660
|
statusKey(chatId: string, threadId?: number | null): string
|
|
660
661
|
streamKey(chatId: string, threadId?: number | null): string
|
|
661
|
-
resolveReplyOwnerTurn(liveTurn: CurrentTurn | null, chatId: string, args: Record<string, unknown>): CurrentTurn | null
|
|
662
|
+
resolveReplyOwnerTurn(liveTurn: CurrentTurn | null, chatId: string, args: Record<string, unknown>): { turn: CurrentTurn | null; tier: ReplyOwnerTier }
|
|
662
663
|
findTurnByOriginId(originTurnId: string | null | undefined): CurrentTurn | null
|
|
663
664
|
findTurnByQuotedMessageId(chatId: string, replyTo: unknown): CurrentTurn | null
|
|
664
665
|
resolveAnswerThreadWithLog(
|
|
@@ -671,6 +672,7 @@ export interface SendReplyGatewayDeps {
|
|
|
671
672
|
): number | undefined
|
|
672
673
|
resolveThreadId(chatId: string, explicit?: string | number | null): number | undefined
|
|
673
674
|
getLatestInboundMessageId(chatId: string, threadId: number | null): number | null | undefined
|
|
675
|
+
getLastSubagentHandbackAt(chatId: string): number | null
|
|
674
676
|
recordOutbound(rec: {
|
|
675
677
|
chat_id: string
|
|
676
678
|
thread_id: number | null
|
|
@@ -735,7 +737,7 @@ export async function sendReply(
|
|
|
735
737
|
statusKey, streamKey,
|
|
736
738
|
resolveReplyOwnerTurn, findTurnByOriginId, findTurnByQuotedMessageId,
|
|
737
739
|
resolveAnswerThreadWithLog, resolveThreadId,
|
|
738
|
-
getLatestInboundMessageId, recordOutbound,
|
|
740
|
+
getLatestInboundMessageId, getLastSubagentHandbackAt, recordOutbound,
|
|
739
741
|
emissionAuthorityFor, clearActivitySummary,
|
|
740
742
|
startTypingLoop, stopTypingLoop, logOutbound,
|
|
741
743
|
closeObligationOnSubstantiveReply, finalizeStatusReaction,
|
|
@@ -874,18 +876,56 @@ export async function sendReply(
|
|
|
874
876
|
// double-send). The quoted / latest-ended recoveries are precisely what the
|
|
875
877
|
// router already did for the same reply, so unifying here makes the two
|
|
876
878
|
// resolvers agree and the late-reply supersede fires by identity.
|
|
877
|
-
const ownerTurn = resolveReplyOwnerTurn(turn, chat_id, args)
|
|
879
|
+
const { turn: ownerTurn, tier: ownerTier } = resolveReplyOwnerTurn(turn, chat_id, args)
|
|
878
880
|
const resolvedTurnId = ownerTurn?.turnId ?? null
|
|
879
|
-
// #3429 — pass the (normalized) reply text so the registry
|
|
880
|
-
// new-content gate: identity match + TTL alone also fits
|
|
881
|
-
// that merely resolved this flush-delivered ENDED turn as
|
|
882
|
-
// the latest-ended tier. Editing the flushed message
|
|
883
|
-
// handback's text
|
|
884
|
-
//
|
|
881
|
+
// #3429 — pass the (normalized) reply text so the registry CAN apply the
|
|
882
|
+
// new-content gate: identity match + TTL alone also fits a background
|
|
883
|
+
// sub-agent handback that merely resolved this flush-delivered ENDED turn as
|
|
884
|
+
// its owner via the latest-ended tier. Editing/deleting the flushed message
|
|
885
|
+
// for that handback's unrelated text is the #3429 silent client-side drop
|
|
886
|
+
// (msgs 10482/10486). But that gate ALSO declines the turn's OWN reworded
|
|
887
|
+
// late reply (model narrated → flushed → fired `reply` with a paraphrase),
|
|
888
|
+
// which is the dominant real duplicate (agent:marko 2026-07-20: 11/11
|
|
889
|
+
// declines were own-replies, turns #1177/#1182/#1201 double-sent). So we
|
|
890
|
+
// BYPASS the content gate ONLY when the reply is confidently the flushed
|
|
891
|
+
// turn's OWN answer. The primary, deterministic signal is the absence of a
|
|
892
|
+
// background sub-agent handback that could own this reply: the gateway
|
|
893
|
+
// records when it synthesizes a `subagent_handback` inbound per chat, and if
|
|
894
|
+
// NONE was enqueued for this chat AFTER the flushed turn ended within the
|
|
895
|
+
// supersede TTL, the reply is that turn's own answer → supersede regardless
|
|
896
|
+
// of a model rewording (closes the DM default-reply duplicate, where every
|
|
897
|
+
// own-reply resolves via the latest-ended tier).
|
|
898
|
+
//
|
|
899
|
+
// MUST-FIX 1 (silent-data-loss): the owner-resolution `quoted` / `origin`
|
|
900
|
+
// tiers are derived from MODEL-SUPPLIED args (`args.reply_to` /
|
|
901
|
+
// `args.origin_turn_id`), so a background handback turn can STEER them —
|
|
902
|
+
// pass `reply_to = <the user's original msg id>` and it resolves the PRIOR
|
|
903
|
+
// flushed turn via `quoted`, which used to force the bypass and silently
|
|
904
|
+
// edit-over that turn's delivered answer (#3429-class, model-steerable). So
|
|
905
|
+
// a positive tier NEVER overrides an in-window handback; only the
|
|
906
|
+
// FRAMEWORK-owned `live` tier (the live `currentTurn`, not model-derived,
|
|
907
|
+
// and structurally unable to collide with an ended turn's flush record —
|
|
908
|
+
// `decideSupersede` requires same turnId) may bypass a handback window.
|
|
909
|
+
//
|
|
910
|
+
// The content gate is therefore kept whenever a handback WAS enqueued in the
|
|
911
|
+
// window (any non-live tier): the late reply might BE that handback, so
|
|
912
|
+
// genuinely-new content sends fresh (two messages, #3429 preserved).
|
|
913
|
+
// Concurrency note: a case-A own reply coinciding with an unrelated
|
|
914
|
+
// background handback in the same ≤TTL window degrades to today's behaviour
|
|
915
|
+
// (two messages) — safe (never a silent drop/edit), just not collapsed.
|
|
916
|
+
const ownerEndedAt = ownerTurn?.endedAt ?? null
|
|
917
|
+
const handbackAt = getLastSubagentHandbackAt(chat_id)
|
|
918
|
+
const now = Date.now()
|
|
919
|
+
const handbackCouldOwnReply =
|
|
920
|
+
handbackAt != null &&
|
|
921
|
+
ownerEndedAt != null &&
|
|
922
|
+
handbackAt > ownerEndedAt &&
|
|
923
|
+
now - handbackAt <= DEFAULT_SUPERSEDE_TTL_MS
|
|
924
|
+
const replyIsOwnAnswer = ownerTier === 'live' || !handbackCouldOwnReply
|
|
885
925
|
const decision = flushedTurnSupersede.take(
|
|
886
926
|
chat_id,
|
|
887
927
|
replyThreadId,
|
|
888
|
-
{ liveTurnId: resolvedTurnId, replyText: text,
|
|
928
|
+
{ liveTurnId: resolvedTurnId, replyText: text, positiveAttribution: replyIsOwnAnswer, now },
|
|
889
929
|
)
|
|
890
930
|
if (decision.supersede) {
|
|
891
931
|
process.stderr.write(
|
|
@@ -79,6 +79,16 @@ export interface PendingInboundBufferOptions {
|
|
|
79
79
|
* never breaks the push hot path.
|
|
80
80
|
*/
|
|
81
81
|
onEvict?: (agent: string, evicted: InboundMessage) => void
|
|
82
|
+
/**
|
|
83
|
+
* fix/backstop-duplicate-reply MUST-FIX 2 — called on every push of a
|
|
84
|
+
* `subagent_handback` envelope (live synthesis AND boot-replay re-push),
|
|
85
|
+
* carrying the envelope's `chatId` and its own `ts` (ms). The gateway wires
|
|
86
|
+
* this to the per-chat subagent-handback marker so the supersede path can tell
|
|
87
|
+
* a flushed turn's own late reply from a background handback attributed to it
|
|
88
|
+
* — INCLUDING after a restart, where the only handback push is the replay.
|
|
89
|
+
* Best-effort: a throw here never breaks the push hot path.
|
|
90
|
+
*/
|
|
91
|
+
onHandbackEnqueue?: (chatId: string, ts: number) => void
|
|
82
92
|
}
|
|
83
93
|
|
|
84
94
|
/**
|
|
@@ -342,6 +352,23 @@ export function createPendingInboundBuffer(
|
|
|
342
352
|
}
|
|
343
353
|
}
|
|
344
354
|
q.push(msg)
|
|
355
|
+
// fix/backstop-duplicate-reply MUST-FIX 2 — stamp the subagent-handback
|
|
356
|
+
// marker at THIS chokepoint, not at the live onFinish enqueue site alone.
|
|
357
|
+
// Every handback enqueue funnels through here — the live synthesis push
|
|
358
|
+
// AND the boot-replay re-push of un-acked spooled inbounds — so stamping
|
|
359
|
+
// here (rather than only at the live site) means a handback replayed after
|
|
360
|
+
// a restart still populates the marker. Otherwise the Map is empty
|
|
361
|
+
// post-boot and a replayed handback's late reply bypasses the #3429
|
|
362
|
+
// content gate → silent edit-over-answer. Uses the envelope's own `ts`
|
|
363
|
+
// (ms, `Date.now()`-derived at synthesis) so the marker reflects when the
|
|
364
|
+
// handback actually happened, not the replay moment. Best-effort.
|
|
365
|
+
if (msg.meta?.source === 'subagent_handback' && opts.onHandbackEnqueue != null) {
|
|
366
|
+
try {
|
|
367
|
+
opts.onHandbackEnqueue(msg.chatId, msg.ts)
|
|
368
|
+
} catch {
|
|
369
|
+
/* marker stamp is best-effort; never break the push hot path */
|
|
370
|
+
}
|
|
371
|
+
}
|
|
345
372
|
// Durable record FIRST-class to the in-memory queue: spool BEFORE
|
|
346
373
|
// returning, regardless of the cap eviction above — an entry the
|
|
347
374
|
// in-memory cap drops still survives in the spool (boot-replayed /
|
|
@@ -104,6 +104,19 @@ export function writeSessionModelFile(
|
|
|
104
104
|
if (!isValidModelArg(model)) {
|
|
105
105
|
throw new Error(`refusing to persist non-canonical session model token: ${JSON.stringify(model)}`)
|
|
106
106
|
}
|
|
107
|
+
// A FRESH carrier gets a FRESH retry budget: clear any bounded-retry counter
|
|
108
|
+
// left over from a prior carrier's boots. Without this, an orphan counter
|
|
109
|
+
// survives the whole session on a healthy snapshot-apply boot (the apply
|
|
110
|
+
// branch re-writes the live counter AFTER the gateway's healthy-boot consume
|
|
111
|
+
// already deleted it), gets snapshotted alongside the NEW carrier at the next
|
|
112
|
+
// boot, and burns the new override's retry budget from a stale count — three
|
|
113
|
+
// such sessions in a row and a perfectly healthy /model is refused with a
|
|
114
|
+
// false "did not reach a healthy boot after 3 attempts" revert.
|
|
115
|
+
try {
|
|
116
|
+
rmSync(join(agentDir, SESSION_MODEL_BOOT_ATTEMPTS_FILE), { force: true })
|
|
117
|
+
} catch {
|
|
118
|
+
/* best-effort — worst case is a stale count, bounded and non-fatal */
|
|
119
|
+
}
|
|
107
120
|
atomicWrite(
|
|
108
121
|
join(agentDir, SESSION_MODEL_FILE),
|
|
109
122
|
serializeSessionModel({ model, configuredDefaultAtWrite, ts: Date.now() }),
|
|
@@ -2018,6 +2018,23 @@ export function handleSessionEvent(deps: StreamRenderDeps, ev: SessionEvent): vo
|
|
|
2018
2018
|
// represent. The delivery closes the obligation + records dedup, so
|
|
2019
2019
|
// the represent and a late reply-tool retry are both suppressed —
|
|
2020
2020
|
// captured-prose delivery and represent are mutually exclusive.
|
|
2021
|
+
// Capture-divergence corner (duplicate-message fix): when this is a
|
|
2022
|
+
// ZERO-reply silent-end AND the gateway's OWN capture came up empty
|
|
2023
|
+
// (so `decideTurnFlush` had nothing to flush — its 'empty-text'
|
|
2024
|
+
// skip), the captured-prose bridge is the ONLY delivery machine
|
|
2025
|
+
// left. The Stop hook scanned the transcript and DID find the
|
|
2026
|
+
// model's short trailing answer, persisting it as `pendingText`. In
|
|
2027
|
+
// that corner the 200-char substance floor would wrongly drop a
|
|
2028
|
+
// legitimately-short answer that no other machine can deliver, so
|
|
2029
|
+
// lower the floor to 1. This is scoped tightly: only when the model
|
|
2030
|
+
// never called reply (`replyCalled === false`) AND the gateway
|
|
2031
|
+
// captured nothing — the interim-ack case (replyCalled) keeps the
|
|
2032
|
+
// full 200 floor (a short closer after a real reply is not a dropped
|
|
2033
|
+
// answer, and the hook already refuses to persist <200 there).
|
|
2034
|
+
const gatewayCapturedEmpty =
|
|
2035
|
+
turn.capturedText.join('\n\n').trim().length === 0
|
|
2036
|
+
const proseMinChars =
|
|
2037
|
+
!turn.replyCalled && gatewayCapturedEmpty ? 1 : CAPTURED_PROSE_MIN_CHARS
|
|
2021
2038
|
const proseDecision = CAPTURED_PROSE_DELIVERY_ENABLED
|
|
2022
2039
|
? decideCapturedProseDelivery({
|
|
2023
2040
|
turnKey: tKey,
|
|
@@ -2025,7 +2042,7 @@ export function handleSessionEvent(deps: StreamRenderDeps, ev: SessionEvent): vo
|
|
|
2025
2042
|
// belong to THIS turn, not a stale carryover from a prior turn
|
|
2026
2043
|
// on the same chat/thread (tKey is not per-turn unique).
|
|
2027
2044
|
turnId: turn.turnId,
|
|
2028
|
-
minChars:
|
|
2045
|
+
minChars: proseMinChars,
|
|
2029
2046
|
})
|
|
2030
2047
|
: { deliver: false as const, reason: 'no-state' as const }
|
|
2031
2048
|
if (proseDecision.deliver && proseDecision.text != null) {
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Per-chat marker of the most recent gateway-synthesized `subagent_handback`
|
|
3
|
+
* enqueue (fix/backstop-duplicate-reply).
|
|
4
|
+
*
|
|
5
|
+
* A BACKGROUND sub-agent completion is a GATEWAY-SYNTHESIZED event, not model
|
|
6
|
+
* output: when a background worker terminates the gateway wakes the agent with a
|
|
7
|
+
* `subagent_handback` inbound. Recording WHEN one was enqueued, per chat, is the
|
|
8
|
+
* ONE deterministic signal that distinguishes the two late-reply cases that both
|
|
9
|
+
* resolve a flush-delivered ENDED turn via the latest-ended tier — the case the
|
|
10
|
+
* owner-resolution tier alone cannot separate (a DM late reply has no
|
|
11
|
+
* live/origin/quoted attribution, so both land on latest-ended):
|
|
12
|
+
*
|
|
13
|
+
* - CASE A — the flushed turn's OWN reworded reply landing late. NO
|
|
14
|
+
* `subagent_handback` was enqueued for this chat after the turn ended, so the
|
|
15
|
+
* reply is that turn's own answer → the supersede path collapses the
|
|
16
|
+
* provisional flush REGARDLESS of the model's rewording (closes the #3429
|
|
17
|
+
* reworded-duplicate regression: agent:marko 2026-07-20, turns
|
|
18
|
+
* #1177/#1182/#1201 double-sent).
|
|
19
|
+
* - CASE B — a background handback attributed to that ended turn. A
|
|
20
|
+
* `subagent_handback` WAS enqueued after the turn ended and within the
|
|
21
|
+
* supersede TTL, so the late reply might BE it → keep the #3429 content gate
|
|
22
|
+
* and send fresh (two messages), never silently edit/delete the flushed
|
|
23
|
+
* answer.
|
|
24
|
+
*
|
|
25
|
+
* One entry per chat (overwritten on each enqueue), so bounded by chat count. No
|
|
26
|
+
* clock reads beyond the caller-supplied `now`; the gateway wires the actual
|
|
27
|
+
* enqueue site and the supersede-path read. Deterministic — keyed on a
|
|
28
|
+
* gateway-emitted event, never on model discipline (Ken's controls-in-code rule).
|
|
29
|
+
*/
|
|
30
|
+
export class SubagentHandbackMarker {
|
|
31
|
+
private readonly lastAtByChat = new Map<string, number>()
|
|
32
|
+
|
|
33
|
+
/** Record that a `subagent_handback` was enqueued for `chatId` at `now` (ms). */
|
|
34
|
+
record(chatId: string, now: number): void {
|
|
35
|
+
this.lastAtByChat.set(chatId, now)
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
/** Wall-clock ms of the most recent handback enqueue for `chatId`, or null. */
|
|
39
|
+
lastAt(chatId: string): number | null {
|
|
40
|
+
return this.lastAtByChat.get(chatId) ?? null
|
|
41
|
+
}
|
|
42
|
+
}
|
|
@@ -1,26 +1,38 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Turn-active liveness marker (#412).
|
|
3
3
|
*
|
|
4
|
-
* Writes `<STATE_DIR>/turn-active.json` on turn_start, touches its mtime
|
|
5
|
-
*
|
|
6
|
-
*
|
|
7
|
-
*
|
|
8
|
-
*
|
|
4
|
+
* Writes `<STATE_DIR>/turn-active.json` on turn_start, touches its mtime on
|
|
5
|
+
* every tool_use AND (via the subagent-watcher, #501) on foreground sub-agent
|
|
6
|
+
* JSONL growth, removes it on turn_complete. The mtime is therefore "ms since
|
|
7
|
+
* last observable progress": it advances while a turn — even a long one running
|
|
8
|
+
* one big tool or a sub-agent — is genuinely working, and goes stale only when
|
|
9
|
+
* work stops (a wedge).
|
|
9
10
|
*
|
|
10
|
-
* Why this exists: PR #410 raised the journal-silence detector to 4000s
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
14
|
-
*
|
|
11
|
+
* Why this exists: PR #410 raised the journal-silence detector to 4000s to kill
|
|
12
|
+
* false positives on chat-cadence agents that legitimately idle for hours
|
|
13
|
+
* between turns. That left a gap — Stop-hook deadlocks (the original failure
|
|
14
|
+
* mode #116 tracked) are no longer caught under default thresholds. The
|
|
15
|
+
* distinguisher is "in-turn-and-silent" vs "between-turns-and-silent": the
|
|
16
|
+
* former is a wedge, the latter is healthy idle. This marker exists exactly
|
|
17
|
+
* during in-turn windows, so its staleness uniquely indicates the wedge.
|
|
15
18
|
*
|
|
16
|
-
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
19
|
-
* the wedge.
|
|
19
|
+
* WHO CONSUMES THE STALENESS (contract corrected — the historical
|
|
20
|
+
* `bin/bridge-watchdog.sh` bash watchdog this header once named was never
|
|
21
|
+
* committed to the repo; do not reintroduce a reference to it):
|
|
20
22
|
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
23
|
+
* - The gateway BOOT classifier (`markOrphanedWithTimeoutClassification` in
|
|
24
|
+
* gateway.ts): on restart, a marker whose mtime is older than TURN_HANG_SECS
|
|
25
|
+
* (default 300s) reclassifies the orphaned turn as `timeout`, routing the
|
|
26
|
+
* next session to the ask-first `resume_watchdog_timeout` inbound.
|
|
27
|
+
* - The LIVE Stage B hang-restart (`hang-restart-decision.ts`, consulted from
|
|
28
|
+
* the silence-poke framework fallback): a mid-tool fallback with a stale
|
|
29
|
+
* marker escalates to a real SIGTERM-PID1 restart. `readTurnActiveMarkerAgeMs`
|
|
30
|
+
* below is the shared read.
|
|
31
|
+
* - The obligation / phantom-turn sweeps (`readTurnActiveMarkerAgeMs`,
|
|
32
|
+
* `effectiveTurnAgeMs`): a small age means work is still in flight, so a
|
|
33
|
+
* "did I miss this?" re-send is suppressed.
|
|
34
|
+
*
|
|
35
|
+
* Pure file I/O; the mtime is the honest cross-process progress signal.
|
|
24
36
|
*/
|
|
25
37
|
|
|
26
38
|
import {
|