@mjasnikovs/pi-task 0.18.36 → 0.18.38
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -1
- package/dist/config/config.d.ts +32 -0
- package/dist/config/config.js +22 -0
- package/dist/config/register.js +25 -1
- package/dist/index.js +2 -0
- package/dist/shared/child-process.d.ts +20 -0
- package/dist/shared/child-process.js +46 -2
- package/dist/shared/stream-watchdog.d.ts +119 -0
- package/dist/shared/stream-watchdog.js +182 -0
- package/dist/task/accept-debt.d.ts +17 -1
- package/dist/task/accept-debt.js +20 -1
- package/dist/task/auto-orchestrator.js +116 -28
- package/dist/task/child-runner.js +18 -2
- package/dist/task/final-gate-fix.d.ts +18 -2
- package/dist/task/final-gate-fix.js +19 -6
- package/dist/task/final-gate-progress.d.ts +67 -0
- package/dist/task/final-gate-progress.js +106 -0
- package/dist/task/gate-deps.js +8 -0
- package/dist/task/stream-watchdog.d.ts +32 -0
- package/dist/task/stream-watchdog.js +90 -0
- package/dist/workers/pi-worker-core.d.ts +20 -0
- package/dist/workers/pi-worker-core.js +14 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -65,7 +65,7 @@ A whole plan — `/task-auto` splits it into an ordered task list and runs each
|
|
|
65
65
|
| `/task-auto <feature>` | Plan a feature into a task list and run each title through `/task` in order (resumable). |
|
|
66
66
|
| `/task-auto-resume` | Resume the active `/task-auto` run at the next unfinished task. |
|
|
67
67
|
| `/task-auto-cancel` | Stop the `/task-auto` loop after the current task (still resumable). |
|
|
68
|
-
| `/task-config` | Toggle pi-task settings in an editor dialog: remote server, compress reasoning, auto-commit, orientation, verify work, enforce guidelines, command timeout, and the extension whitelist for child sessions. |
|
|
68
|
+
| `/task-config` | Toggle pi-task settings in an editor dialog: remote server, compress reasoning, auto-commit, orientation, verify work, enforce guidelines, command timeout, stream watchdog, and the extension whitelist for child sessions. |
|
|
69
69
|
| `/remote` | Show the QR code & URLs for the web view (`/remote stop` to stop). Answer grill questions, start tasks, and watch progress from your phone. |
|
|
70
70
|
|
|
71
71
|
## The pipeline
|
|
@@ -180,6 +180,7 @@ Run `/task-config` to toggle pi-task's behavior in an editor dialog. Settings pe
|
|
|
180
180
|
| **research cache** | on | Cache docs/search/fetch worker results for the duration of one `/task-auto` run so sibling tasks re-asking the same package/URL + query reuse the first pipeline's digest instead of re-fetching. Per-run isolated, external-only (project-source `.` lookups excluded), success-only. |
|
|
181
181
|
| **search provider** | Exa | Engine behind `pi-worker-search` and freshness/enrichment checks. **Exa** (default) and **DuckDuckGo** need no API key; **Brave** requires `BRAVE_SEARCH_API_KEY`. |
|
|
182
182
|
| **command timeout** | 15 min | Wall-clock ceiling on a **single** tool execution. Local models routinely run a command that never returns (a hung build, a dev server, a check with no timeout) and the run wedges until you abort by hand — pi's bash tool has an optional timeout with no default, so this is the missing one. One knob, two surfaces: in the main session the overrun call is cancelled (killing the tool's whole process tree) plus a reminder turn; in the verify/fix gate children the child is killed and re-spawned with a hint, halving the ceiling on repeat hangs. Choices: 5/10/15/30 min or **off** — off unguards both surfaces, gates included. |
|
|
183
|
+
| **stream watchdog** | 10 min | Inactivity ceiling on the **model stream**. A hung or silently-dropped stream throws nothing at all, so neither the connection-error retry (it needs a reported error) nor the **command timeout** (tool calls only) nor the dead-backend stall guard (a reachable endpoint reads as proof of life) can see it — an mx5 run lost ~2.9h to three of them while the model server stayed healthy. Measured as time since the **last stream event of any kind**, so a slow model emitting one token every 30s is never touched, and it pauses while a tool runs. On expiry the main session aborts the turn (through the same channel the command watchdog uses) and posts a resume reminder; a child is killed and routed into the existing connection-error retry. Choices: 5/10/20/30 min or **off**. Keep it generous on local backends — prompt processing on a large context legitimately emits nothing for minutes. |
|
|
183
184
|
| **yolo mode** | off | **Unattended runs.** Wherever pi-task would stop and ask, it takes the option already marked RECOMMENDED, stamps the artifact `(YOLO)` so an audit can tell a machine decided, and shows no prompt at all — clarify/grill answers, the verify-FAIL picker (auto-**Accept**, recorded as a yolo debt), and the final-gate picker (autofix while the budget lasts, then leave the run FAILED). A question with no recommendation is **skipped**, never invented. For throwaway/test projects nobody is watching; a real run should decide these itself. |
|
|
184
185
|
| **extension whitelist** | empty | Host `pi` extensions to load into every child session by explicit path. Children otherwise run with extensions off, so a provider registered by an extension (e.g. `pi-lmstudio`) doesn't exist in them and they can't resolve the default model. `/task-config` enumerates the currently installed extensions as individual `ext: …` toggles; the list is strictly additive (discovery stays off), and an entry whose file is gone is skipped at spawn time, never fatal. |
|
|
185
186
|
|
package/dist/config/config.d.ts
CHANGED
|
@@ -61,6 +61,26 @@ export interface PiTaskConfig {
|
|
|
61
61
|
* a true hang doesn't cost half an hour of dead time.
|
|
62
62
|
*/
|
|
63
63
|
requestTimeoutMs: number;
|
|
64
|
+
/**
|
|
65
|
+
* Inactivity ceiling (ms) on the MODEL STREAM before the stream watchdog
|
|
66
|
+
* aborts the request (shared/stream-watchdog.ts). A hung or silently-dropped
|
|
67
|
+
* stream throws nothing, so neither the connection-error retry (needs a
|
|
68
|
+
* reported error) nor the command watchdog (covers tool calls only) nor the
|
|
69
|
+
* child stall guard (a reachable endpoint is proof of life) can see it — mx5
|
|
70
|
+
* run 14 lost ~2.9h to three such hangs while the model server stayed healthy.
|
|
71
|
+
*
|
|
72
|
+
* Measured as time since the LAST stream event of any kind, so a slow model
|
|
73
|
+
* emitting one token every 30s is never touched; only total silence counts.
|
|
74
|
+
* Suspended while a tool executes — that window is the command watchdog's.
|
|
75
|
+
*
|
|
76
|
+
* ONE knob, TWO surfaces: the MAIN session aborts the turn through the same
|
|
77
|
+
* plumbing the command watchdog uses and posts a resume reminder; a CHILD is
|
|
78
|
+
* killed and its result routed into the EXISTING connection-error retry.
|
|
79
|
+
* 0 = off, on both surfaces.
|
|
80
|
+
* DEFAULT 10 min: local prompt processing on a 32k context legitimately emits
|
|
81
|
+
* nothing for minutes, so a 60-120s ceiling would kill healthy long prompts.
|
|
82
|
+
*/
|
|
83
|
+
streamInactivityMs: number;
|
|
64
84
|
/**
|
|
65
85
|
* UNATTENDED AUTO-PICK (see task/yolo.ts): wherever pi-task would stop and ask,
|
|
66
86
|
* take the option it already marks RECOMMENDED, stamp the artifact `(YOLO)` so
|
|
@@ -91,6 +111,18 @@ export declare const COMMAND_TIMEOUT_OPTIONS: ReadonlyArray<{
|
|
|
91
111
|
* one of the offered choices so the watchdog never arms on a nonsense value.
|
|
92
112
|
*/
|
|
93
113
|
export declare function sanitizeRequestTimeoutMs(value: unknown): number;
|
|
114
|
+
/**
|
|
115
|
+
* The stream-watchdog choices offered by /task-config, in cycle order. Every
|
|
116
|
+
* option is minutes, not seconds: the failure this guards costs hours, and the
|
|
117
|
+
* legitimate silence it must tolerate (local prompt processing) is minutes.
|
|
118
|
+
*/
|
|
119
|
+
export declare const STREAM_INACTIVITY_OPTIONS: ReadonlyArray<{
|
|
120
|
+
label: string;
|
|
121
|
+
ms: number;
|
|
122
|
+
}>;
|
|
123
|
+
/** Same pinning as {@link sanitizeRequestTimeoutMs}: a hand-edited value that is
|
|
124
|
+
* not one of the offered choices falls back to the default. */
|
|
125
|
+
export declare function sanitizeStreamInactivityMs(value: unknown): number;
|
|
94
126
|
/**
|
|
95
127
|
* A hand-edited config can hold anything; keep only string entries so a stray
|
|
96
128
|
* object/number can't reach the child argv as `-e [object Object]`.
|
package/dist/config/config.js
CHANGED
|
@@ -3,6 +3,7 @@ import * as fsp from 'node:fs/promises';
|
|
|
3
3
|
import * as path from 'node:path';
|
|
4
4
|
import * as os from 'node:os';
|
|
5
5
|
import { isSearchProvider } from '../workers/search-types.js';
|
|
6
|
+
import { DEFAULT_STREAM_INACTIVITY_MS } from '../shared/stream-watchdog.js';
|
|
6
7
|
/**
|
|
7
8
|
* The command-watchdog timeout choices offered by /task-config, newest-first in
|
|
8
9
|
* the cycle order the picker shows. The stored config value is the ms number;
|
|
@@ -25,6 +26,25 @@ export function sanitizeRequestTimeoutMs(value) {
|
|
|
25
26
|
value
|
|
26
27
|
: DEFAULT_REQUEST_TIMEOUT_MS;
|
|
27
28
|
}
|
|
29
|
+
/**
|
|
30
|
+
* The stream-watchdog choices offered by /task-config, in cycle order. Every
|
|
31
|
+
* option is minutes, not seconds: the failure this guards costs hours, and the
|
|
32
|
+
* legitimate silence it must tolerate (local prompt processing) is minutes.
|
|
33
|
+
*/
|
|
34
|
+
export const STREAM_INACTIVITY_OPTIONS = [
|
|
35
|
+
{ label: '5 min', ms: 5 * 60_000 },
|
|
36
|
+
{ label: '10 min', ms: DEFAULT_STREAM_INACTIVITY_MS },
|
|
37
|
+
{ label: '20 min', ms: 20 * 60_000 },
|
|
38
|
+
{ label: '30 min', ms: 30 * 60_000 },
|
|
39
|
+
{ label: 'off', ms: 0 }
|
|
40
|
+
];
|
|
41
|
+
/** Same pinning as {@link sanitizeRequestTimeoutMs}: a hand-edited value that is
|
|
42
|
+
* not one of the offered choices falls back to the default. */
|
|
43
|
+
export function sanitizeStreamInactivityMs(value) {
|
|
44
|
+
return STREAM_INACTIVITY_OPTIONS.some(o => o.ms === value) ?
|
|
45
|
+
value
|
|
46
|
+
: DEFAULT_STREAM_INACTIVITY_MS;
|
|
47
|
+
}
|
|
28
48
|
const DEFAULTS = {
|
|
29
49
|
remote: true,
|
|
30
50
|
compressReasoning: true,
|
|
@@ -39,6 +59,7 @@ const DEFAULTS = {
|
|
|
39
59
|
searchProvider: 'exa',
|
|
40
60
|
extensionWhitelist: [],
|
|
41
61
|
requestTimeoutMs: DEFAULT_REQUEST_TIMEOUT_MS,
|
|
62
|
+
streamInactivityMs: DEFAULT_STREAM_INACTIVITY_MS,
|
|
42
63
|
// OFF: auto-answering is for unattended throwaway runs only.
|
|
43
64
|
yoloMode: false
|
|
44
65
|
};
|
|
@@ -69,6 +90,7 @@ if (!G.loaded) {
|
|
|
69
90
|
delete parsed.searchProvider;
|
|
70
91
|
parsed.extensionWhitelist = sanitizeExtensionWhitelist(parsed.extensionWhitelist);
|
|
71
92
|
parsed.requestTimeoutMs = sanitizeRequestTimeoutMs(parsed.requestTimeoutMs);
|
|
93
|
+
parsed.streamInactivityMs = sanitizeStreamInactivityMs(parsed.streamInactivityMs);
|
|
72
94
|
// A hand-edited `"yoloMode": "false"` is a truthy string — it must not
|
|
73
95
|
// silently switch a watched run into unattended auto-pick. Only a real
|
|
74
96
|
// boolean counts; anything else falls back to the OFF default.
|
package/dist/config/register.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { SettingsList, visibleWidth } from '@earendil-works/pi-tui';
|
|
2
2
|
import { registerBridgeCommand } from '../remote/bridge.js';
|
|
3
3
|
import { SEARCH_PROVIDERS, SEARCH_PROVIDER_LABELS, providerForLabel } from '../workers/search-types.js';
|
|
4
|
-
import { COMMAND_TIMEOUT_OPTIONS, getConfig, saveConfig } from './config.js';
|
|
4
|
+
import { COMMAND_TIMEOUT_OPTIONS, getConfig, saveConfig, STREAM_INACTIVITY_OPTIONS } from './config.js';
|
|
5
5
|
import { listInstalledExtensions } from './extension-list.js';
|
|
6
6
|
const CONFIG_TITLE = 'pi-task settings';
|
|
7
7
|
/**
|
|
@@ -108,6 +108,19 @@ const ITEMS = [
|
|
|
108
108
|
// Display human labels; the stored config value stays the ms number.
|
|
109
109
|
values: COMMAND_TIMEOUT_OPTIONS.map(o => o.label)
|
|
110
110
|
},
|
|
111
|
+
{
|
|
112
|
+
id: 'streamInactivityMs',
|
|
113
|
+
label: 'stream watchdog',
|
|
114
|
+
description: 'Abort and retry a model request whose stream goes SILENT for this long — a hung '
|
|
115
|
+
+ 'or dropped stream reports no error at all, so nothing else catches it (mx5 run '
|
|
116
|
+
+ '14 lost ~2.9h to three of them while the model server stayed healthy). Counts '
|
|
117
|
+
+ 'time since the last stream event of any kind, so a slow model that keeps '
|
|
118
|
+
+ 'emitting tokens is never touched, and it pauses while a tool runs. Keep it '
|
|
119
|
+
+ 'generous on local backends: prompt processing on a large context legitimately '
|
|
120
|
+
+ 'emits nothing for minutes. off disables it on both the main session and children',
|
|
121
|
+
// Display human labels; the stored config value stays the ms number.
|
|
122
|
+
values: STREAM_INACTIVITY_OPTIONS.map(o => o.label)
|
|
123
|
+
},
|
|
111
124
|
{
|
|
112
125
|
id: 'yoloMode',
|
|
113
126
|
label: 'yolo mode',
|
|
@@ -123,12 +136,18 @@ const ITEMS = [
|
|
|
123
136
|
function timeoutLabel(ms) {
|
|
124
137
|
return COMMAND_TIMEOUT_OPTIONS.find(o => o.ms === ms)?.label ?? `${ms}ms`;
|
|
125
138
|
}
|
|
139
|
+
/** Human label for the stored stream-inactivity ms (falls back to the raw ms). */
|
|
140
|
+
function streamTimeoutLabel(ms) {
|
|
141
|
+
return STREAM_INACTIVITY_OPTIONS.find(o => o.ms === ms)?.label ?? `${ms}ms`;
|
|
142
|
+
}
|
|
126
143
|
/** What /task-config shows for a setting's current value. */
|
|
127
144
|
function displayValue(cfg, id, isEnum) {
|
|
128
145
|
if (id === 'searchProvider')
|
|
129
146
|
return SEARCH_PROVIDER_LABELS[cfg.searchProvider];
|
|
130
147
|
if (id === 'requestTimeoutMs')
|
|
131
148
|
return timeoutLabel(cfg.requestTimeoutMs);
|
|
149
|
+
if (id === 'streamInactivityMs')
|
|
150
|
+
return streamTimeoutLabel(cfg.streamInactivityMs);
|
|
132
151
|
if (isEnum)
|
|
133
152
|
return String(cfg[id]);
|
|
134
153
|
return cfg[id] ? 'on' : 'off';
|
|
@@ -206,6 +225,11 @@ async function handleTaskConfig(_args, ctx) {
|
|
|
206
225
|
if (opt)
|
|
207
226
|
cfg.requestTimeoutMs = opt.ms;
|
|
208
227
|
}
|
|
228
|
+
else if (id === 'streamInactivityMs') {
|
|
229
|
+
const opt = STREAM_INACTIVITY_OPTIONS.find(o => o.label === newValue);
|
|
230
|
+
if (opt)
|
|
231
|
+
cfg.streamInactivityMs = opt.ms;
|
|
232
|
+
}
|
|
209
233
|
else {
|
|
210
234
|
;
|
|
211
235
|
cfg[id] = newValue === 'on';
|
package/dist/index.js
CHANGED
|
@@ -5,6 +5,7 @@ import { registerWorkers } from './workers/index.js';
|
|
|
5
5
|
import { registerRemote } from './remote/register.js';
|
|
6
6
|
import { registerThinkingCompression } from './thinking/compress.js';
|
|
7
7
|
import { registerCommandWatchdog } from './task/command-watchdog.js';
|
|
8
|
+
import { registerStreamWatchdog } from './task/stream-watchdog.js';
|
|
8
9
|
export default function (pi) {
|
|
9
10
|
registerConfig(pi);
|
|
10
11
|
registerTask(pi);
|
|
@@ -13,4 +14,5 @@ export default function (pi) {
|
|
|
13
14
|
registerRemote(pi);
|
|
14
15
|
registerThinkingCompression(pi);
|
|
15
16
|
registerCommandWatchdog(pi);
|
|
17
|
+
registerStreamWatchdog(pi);
|
|
16
18
|
}
|
|
@@ -54,6 +54,17 @@ export interface ChildResult {
|
|
|
54
54
|
* and without the flag it would mislabel as a user cancel.
|
|
55
55
|
*/
|
|
56
56
|
stalled?: boolean;
|
|
57
|
+
/**
|
|
58
|
+
* true when the STREAM watchdog killed the child: no output at all for the
|
|
59
|
+
* configured inactivity window, regardless of whether the backend answers a
|
|
60
|
+
* probe. Distinct from `stalled`, which requires an UNREACHABLE endpoint —
|
|
61
|
+
* the run-14 hangs had a perfectly healthy server and a dead stream, so the
|
|
62
|
+
* probe path could never fire. Callers must check this BEFORE `aborted`
|
|
63
|
+
* (the kill sets aborted too) and route it into the connection-error retry.
|
|
64
|
+
*/
|
|
65
|
+
streamStalled?: {
|
|
66
|
+
idleMs: number;
|
|
67
|
+
};
|
|
57
68
|
}
|
|
58
69
|
export interface ToolCall {
|
|
59
70
|
name: string;
|
|
@@ -124,6 +135,15 @@ export interface RunChildJsonEventsOptions {
|
|
|
124
135
|
afterMs: number;
|
|
125
136
|
probe: () => Promise<boolean>;
|
|
126
137
|
};
|
|
138
|
+
/**
|
|
139
|
+
* Stream-inactivity ceiling in ms (shared/stream-watchdog.ts). Unlike `stall`
|
|
140
|
+
* this asks NOTHING of the backend: a stream that has produced no bytes for
|
|
141
|
+
* this long is dead whether or not the server answers a health probe, which
|
|
142
|
+
* is exactly the run-14 shape (server Up(healthy), stream silent for hours).
|
|
143
|
+
* Any stdout/stderr chunk resets it, so a slow model emitting one token every
|
|
144
|
+
* 30s is never killed. 0 / omitted = off.
|
|
145
|
+
*/
|
|
146
|
+
streamInactivityMs?: number;
|
|
127
147
|
}
|
|
128
148
|
export type RunChildOptions = RunChildTextOptions | RunChildJsonEventsOptions;
|
|
129
149
|
/**
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { spawn as defaultSpawn, spawnSync as spawnSyncDefault } from 'node:child_process';
|
|
2
|
+
import { realStreamTimerDeps, StreamWatchdog } from './stream-watchdog.js';
|
|
2
3
|
/** Grace period between SIGTERM and SIGKILL (ms). */
|
|
3
4
|
export const KILL_GRACE_MS = 5000;
|
|
4
5
|
/** Base flags shared by all child pi invocations. */
|
|
@@ -248,7 +249,43 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
|
|
|
248
249
|
reapGroup('SIGKILL');
|
|
249
250
|
}, KILL_GRACE_MS);
|
|
250
251
|
};
|
|
251
|
-
|
|
252
|
+
// Stream watchdog (json-events children only): a silent stream is killed
|
|
253
|
+
// even when the backend is healthy — the case the probe-based stall guard
|
|
254
|
+
// below structurally cannot catch. Suspended for the duration of a tool
|
|
255
|
+
// call: a 12-minute build legitimately emits nothing, and that window is
|
|
256
|
+
// the COMMAND watchdog's to police, not this one's.
|
|
257
|
+
let streamStalledIdleMs;
|
|
258
|
+
const streamWatch = opts?.mode === 'json-events' && (opts.streamInactivityMs ?? 0) > 0 ?
|
|
259
|
+
new StreamWatchdog({
|
|
260
|
+
getTimeoutMs: () => opts.streamInactivityMs,
|
|
261
|
+
...realStreamTimerDeps,
|
|
262
|
+
onFire: idleMs => {
|
|
263
|
+
streamStalledIdleMs = idleMs;
|
|
264
|
+
killProc();
|
|
265
|
+
}
|
|
266
|
+
})
|
|
267
|
+
: null;
|
|
268
|
+
streamWatch?.start();
|
|
269
|
+
// The sink sees the child's parsed events; tool start/end are what tell
|
|
270
|
+
// the watchdog to pause and resume. Wrapping the caller's handlers keeps
|
|
271
|
+
// that wiring invisible to callers — and onToolResult must be present for
|
|
272
|
+
// the sink to emit tool_execution_end at all, so it is always supplied
|
|
273
|
+
// when the watchdog is on.
|
|
274
|
+
const sinkOpts = opts?.mode !== 'json-events' ? null
|
|
275
|
+
: streamWatch ?
|
|
276
|
+
{
|
|
277
|
+
...opts,
|
|
278
|
+
onToolCall: call => {
|
|
279
|
+
streamWatch.suspend();
|
|
280
|
+
return opts.onToolCall ? opts.onToolCall(call) : null;
|
|
281
|
+
},
|
|
282
|
+
onToolResult: r => {
|
|
283
|
+
streamWatch.resume();
|
|
284
|
+
opts.onToolResult?.(r);
|
|
285
|
+
}
|
|
286
|
+
}
|
|
287
|
+
: opts;
|
|
288
|
+
const sink = sinkOpts ? new JsonEventSink(sinkOpts, killProc) : null;
|
|
252
289
|
// Dead-backend stall guard (json-events children only; see the option
|
|
253
290
|
// docs). Any output resets the window; a reachable probe also resets it
|
|
254
291
|
// so the next probe is a full window away, not every tick.
|
|
@@ -283,6 +320,7 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
|
|
|
283
320
|
let firstByteFired = false;
|
|
284
321
|
proc.stdout?.on('data', (d) => {
|
|
285
322
|
lastActivity = Date.now();
|
|
323
|
+
streamWatch?.note();
|
|
286
324
|
if (!firstByteFired) {
|
|
287
325
|
firstByteFired = true;
|
|
288
326
|
opts?.onFirstByte?.();
|
|
@@ -297,11 +335,13 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
|
|
|
297
335
|
});
|
|
298
336
|
proc.stderr?.on('data', (d) => {
|
|
299
337
|
lastActivity = Date.now();
|
|
338
|
+
streamWatch?.note();
|
|
300
339
|
stderr += d.toString();
|
|
301
340
|
});
|
|
302
341
|
proc.on('close', (code) => {
|
|
303
342
|
if (stallTimer)
|
|
304
343
|
clearInterval(stallTimer);
|
|
344
|
+
streamWatch?.stop();
|
|
305
345
|
// The child has exited, but anything it backgrounded (a dev server) may
|
|
306
346
|
// still hold its process group and a port — reap the group so the next
|
|
307
347
|
// gate's boot check does not collide with our own orphan. Best-effort:
|
|
@@ -320,12 +360,16 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
|
|
|
320
360
|
aborted,
|
|
321
361
|
text,
|
|
322
362
|
modelError: sink?.modelError,
|
|
323
|
-
...(stalled ? { stalled: true } : {})
|
|
363
|
+
...(stalled ? { stalled: true } : {}),
|
|
364
|
+
...(streamStalledIdleMs !== undefined ?
|
|
365
|
+
{ streamStalled: { idleMs: streamStalledIdleMs } }
|
|
366
|
+
: {})
|
|
324
367
|
});
|
|
325
368
|
});
|
|
326
369
|
proc.on('error', () => {
|
|
327
370
|
if (stallTimer)
|
|
328
371
|
clearInterval(stallTimer);
|
|
372
|
+
streamWatch?.stop();
|
|
329
373
|
resolve({ stdout, stderr, exitCode: 1, aborted });
|
|
330
374
|
});
|
|
331
375
|
if (signal) {
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Model-stream watchdog — the inactivity machine for a stream that goes SILENT
|
|
3
|
+
* without erroring.
|
|
4
|
+
*
|
|
5
|
+
* WHY (mx5 run 14): three main-session implementation turns died mid-turn — the
|
|
6
|
+
* session jsonl's last event is an ordinary assistant message, then nothing,
|
|
7
|
+
* forever, while the model server stayed Up(healthy) the whole time. A hung or
|
|
8
|
+
* silently-dropped stream throws NOTHING, so:
|
|
9
|
+
* - the connection-error retry (child-runner.ts) never fires: it needs a
|
|
10
|
+
* thrown/reported ModelError,
|
|
11
|
+
* - the command watchdog never fires: it only covers TOOL executions,
|
|
12
|
+
* - the child stall guard never fires: it treats a reachable endpoint as proof
|
|
13
|
+
* of life, which it is — the endpoint was fine, the stream was not.
|
|
14
|
+
* Cost in run 14: ~2.9h of dead air awaiting manual restarts.
|
|
15
|
+
*
|
|
16
|
+
* WHAT THIS MEASURES: time since the LAST stream event of ANY kind — text token,
|
|
17
|
+
* tool-call delta, thinking delta, provider response header. NOT wall-clock, and
|
|
18
|
+
* NOT "time to first token". One token every 30s is a working local model and
|
|
19
|
+
* must never be killed; zero events for the whole window is a hang.
|
|
20
|
+
*
|
|
21
|
+
* WHY THE DEFAULT IS GENEROUS: on a local llama-server a 32k-context prompt can
|
|
22
|
+
* spend many minutes in prompt processing emitting nothing at all. A 60-120s
|
|
23
|
+
* ceiling would kill every long prompt on local hardware. See
|
|
24
|
+
* DEFAULT_STREAM_INACTIVITY_MS.
|
|
25
|
+
*
|
|
26
|
+
* TWO SURFACES, one machine (same split as command-watchdog.ts):
|
|
27
|
+
* MAIN SESSION — task/stream-watchdog.ts arms it from pi's extension events and
|
|
28
|
+
* fires ctx.abort() through the SAME abort plumbing the command
|
|
29
|
+
* watchdog uses (noteWatchdogAbort → steerUntilDone), so there is
|
|
30
|
+
* exactly one abort channel, not two racing ones.
|
|
31
|
+
* CHILDREN — shared/child-process.ts arms it on the child's stdout/stderr
|
|
32
|
+
* chunks; a fire kills the child and the result carries
|
|
33
|
+
* `streamStalled`, which child-runner turns into a
|
|
34
|
+
* connection-class cause so the EXISTING retry/backoff path
|
|
35
|
+
* handles it (re-spawn from the last durable prompt, never a
|
|
36
|
+
* blind re-send of a half-executed turn).
|
|
37
|
+
*/
|
|
38
|
+
import type { TimerHandle } from './command-watchdog.js';
|
|
39
|
+
export type { TimerHandle };
|
|
40
|
+
/**
|
|
41
|
+
* Default inactivity ceiling: 10 minutes. Chosen from the constraint that a local
|
|
42
|
+
* model's first token can legitimately be many minutes away — the guard exists to
|
|
43
|
+
* turn hours of dead air into minutes, not to police slowness.
|
|
44
|
+
*/
|
|
45
|
+
export declare const DEFAULT_STREAM_INACTIVITY_MS: number;
|
|
46
|
+
/**
|
|
47
|
+
* How often the machine checks the idle clock. A poll (rather than re-arming a
|
|
48
|
+
* timeout on every token) keeps cost O(1) per window instead of O(1) per token —
|
|
49
|
+
* a streaming turn emits thousands of events. Same idiom as the child stall guard.
|
|
50
|
+
*/
|
|
51
|
+
export declare function pollIntervalMs(timeoutMs: number): number;
|
|
52
|
+
export interface StreamWatchdogDeps {
|
|
53
|
+
/** The inactivity ceiling in ms, read when the watchdog arms. 0 (or any
|
|
54
|
+
* non-positive value) means the watchdog is off and never arms. */
|
|
55
|
+
getTimeoutMs: () => number;
|
|
56
|
+
now: () => number;
|
|
57
|
+
schedule: (fn: () => void, ms: number) => TimerHandle;
|
|
58
|
+
cancel: (handle: TimerHandle) => void;
|
|
59
|
+
/** Invoked once per arming when the stream has been silent for the ceiling. */
|
|
60
|
+
onFire: (idleMs: number, timeoutMs: number) => void;
|
|
61
|
+
}
|
|
62
|
+
export declare class StreamWatchdog {
|
|
63
|
+
private readonly deps;
|
|
64
|
+
private timer;
|
|
65
|
+
private lastEvent;
|
|
66
|
+
private armedMs;
|
|
67
|
+
/** True while a TOOL is executing: the model stream is legitimately idle then,
|
|
68
|
+
* and that window belongs to the command watchdog, not to this one. Without
|
|
69
|
+
* this, a 10-minute build would look identical to a hung stream. */
|
|
70
|
+
private suspended;
|
|
71
|
+
private fired;
|
|
72
|
+
constructor(deps: StreamWatchdogDeps);
|
|
73
|
+
/** Begin watching a model request. No-op when the watchdog is off or already armed. */
|
|
74
|
+
start(): void;
|
|
75
|
+
/** Any stream event of any kind: resets the idle clock. */
|
|
76
|
+
note(): void;
|
|
77
|
+
/** A tool started executing — pause the idle clock until it ends. */
|
|
78
|
+
suspend(): void;
|
|
79
|
+
/** A tool finished — the stream is expected to resume; restart the clock. */
|
|
80
|
+
resume(): void;
|
|
81
|
+
/** Stop watching (turn/agent/session end, or child exit). Safe to call twice. */
|
|
82
|
+
stop(): void;
|
|
83
|
+
/** @internal Exposed for the poll callback and tests. */
|
|
84
|
+
check(): void;
|
|
85
|
+
}
|
|
86
|
+
/**
|
|
87
|
+
* Real-clock poll deps, unref'd so a pending watchdog poll can never itself keep
|
|
88
|
+
* the process alive on exit. `schedule` returns a repeating interval — the
|
|
89
|
+
* machine cancels it on stop/fire.
|
|
90
|
+
*/
|
|
91
|
+
export declare const realStreamTimerDeps: Pick<StreamWatchdogDeps, 'now' | 'schedule' | 'cancel'>;
|
|
92
|
+
/**
|
|
93
|
+
* The cause string a CHILD's stream stall is reported as. Phrased so
|
|
94
|
+
* {@link isConnectionError} (child-runner.ts) matches it — the whole point is to
|
|
95
|
+
* route a silent hang into the retry path that already exists for a LOUD
|
|
96
|
+
* connection failure, rather than inventing a second one. Honest about who
|
|
97
|
+
* killed it: pi-task aborted the request, the provider did not report anything.
|
|
98
|
+
*/
|
|
99
|
+
export declare function streamStallCause(idleMs: number): string;
|
|
100
|
+
/**
|
|
101
|
+
* MAIN-SESSION reminder, delivered as a follow-up turn after ctx.abort() ended the
|
|
102
|
+
* hung turn. Carries the shared WATCHDOG_CANCEL_MARKER so steerUntilDone
|
|
103
|
+
* recognises it as a watchdog recovery rather than a human ESC (see
|
|
104
|
+
* task/command-watchdog.ts) — one marker, one abort channel.
|
|
105
|
+
*
|
|
106
|
+
* Idempotent resume, not a re-send: the aborted turn's tool calls and their
|
|
107
|
+
* results are already in the transcript, so the model is told to CONTINUE from
|
|
108
|
+
* what is recorded. Re-issuing the turn wholesale would re-run tool calls that
|
|
109
|
+
* already ran.
|
|
110
|
+
*/
|
|
111
|
+
export declare function streamStallReminder(idleMs: number, marker: string): string;
|
|
112
|
+
/**
|
|
113
|
+
* CHILD restart hint for a re-spawned worker whose previous attempt was killed by
|
|
114
|
+
* the stream watchdog. The killed child is gone and never saw an error, so this
|
|
115
|
+
* states plainly what happened and, unlike the command-timeout hint, does NOT
|
|
116
|
+
* blame the model: a hung stream is an infrastructure fault, and telling the model
|
|
117
|
+
* to "be faster" would teach it to truncate its work for no reason.
|
|
118
|
+
*/
|
|
119
|
+
export declare function streamStallHint(idleMs: number): string;
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Model-stream watchdog — the inactivity machine for a stream that goes SILENT
|
|
3
|
+
* without erroring.
|
|
4
|
+
*
|
|
5
|
+
* WHY (mx5 run 14): three main-session implementation turns died mid-turn — the
|
|
6
|
+
* session jsonl's last event is an ordinary assistant message, then nothing,
|
|
7
|
+
* forever, while the model server stayed Up(healthy) the whole time. A hung or
|
|
8
|
+
* silently-dropped stream throws NOTHING, so:
|
|
9
|
+
* - the connection-error retry (child-runner.ts) never fires: it needs a
|
|
10
|
+
* thrown/reported ModelError,
|
|
11
|
+
* - the command watchdog never fires: it only covers TOOL executions,
|
|
12
|
+
* - the child stall guard never fires: it treats a reachable endpoint as proof
|
|
13
|
+
* of life, which it is — the endpoint was fine, the stream was not.
|
|
14
|
+
* Cost in run 14: ~2.9h of dead air awaiting manual restarts.
|
|
15
|
+
*
|
|
16
|
+
* WHAT THIS MEASURES: time since the LAST stream event of ANY kind — text token,
|
|
17
|
+
* tool-call delta, thinking delta, provider response header. NOT wall-clock, and
|
|
18
|
+
* NOT "time to first token". One token every 30s is a working local model and
|
|
19
|
+
* must never be killed; zero events for the whole window is a hang.
|
|
20
|
+
*
|
|
21
|
+
* WHY THE DEFAULT IS GENEROUS: on a local llama-server a 32k-context prompt can
|
|
22
|
+
* spend many minutes in prompt processing emitting nothing at all. A 60-120s
|
|
23
|
+
* ceiling would kill every long prompt on local hardware. See
|
|
24
|
+
* DEFAULT_STREAM_INACTIVITY_MS.
|
|
25
|
+
*
|
|
26
|
+
* TWO SURFACES, one machine (same split as command-watchdog.ts):
|
|
27
|
+
* MAIN SESSION — task/stream-watchdog.ts arms it from pi's extension events and
|
|
28
|
+
* fires ctx.abort() through the SAME abort plumbing the command
|
|
29
|
+
* watchdog uses (noteWatchdogAbort → steerUntilDone), so there is
|
|
30
|
+
* exactly one abort channel, not two racing ones.
|
|
31
|
+
* CHILDREN — shared/child-process.ts arms it on the child's stdout/stderr
|
|
32
|
+
* chunks; a fire kills the child and the result carries
|
|
33
|
+
* `streamStalled`, which child-runner turns into a
|
|
34
|
+
* connection-class cause so the EXISTING retry/backoff path
|
|
35
|
+
* handles it (re-spawn from the last durable prompt, never a
|
|
36
|
+
* blind re-send of a half-executed turn).
|
|
37
|
+
*/
|
|
38
|
+
/**
|
|
39
|
+
* Default inactivity ceiling: 10 minutes. Chosen from the constraint that a local
|
|
40
|
+
* model's first token can legitimately be many minutes away — the guard exists to
|
|
41
|
+
* turn hours of dead air into minutes, not to police slowness.
|
|
42
|
+
*/
|
|
43
|
+
export const DEFAULT_STREAM_INACTIVITY_MS = 10 * 60_000;
|
|
44
|
+
/** Whole minutes, floored at 1, for the human-facing window in every message. */
|
|
45
|
+
function minutes(ms) {
|
|
46
|
+
const mins = Math.max(1, Math.round(ms / 60_000));
|
|
47
|
+
return `${mins} minute${mins === 1 ? '' : 's'}`;
|
|
48
|
+
}
|
|
49
|
+
/**
|
|
50
|
+
* How often the machine checks the idle clock. A poll (rather than re-arming a
|
|
51
|
+
* timeout on every token) keeps cost O(1) per window instead of O(1) per token —
|
|
52
|
+
* a streaming turn emits thousands of events. Same idiom as the child stall guard.
|
|
53
|
+
*/
|
|
54
|
+
export function pollIntervalMs(timeoutMs) {
|
|
55
|
+
return Math.max(50, Math.min(Math.floor(timeoutMs / 4), 30_000));
|
|
56
|
+
}
|
|
57
|
+
export class StreamWatchdog {
|
|
58
|
+
deps;
|
|
59
|
+
timer;
|
|
60
|
+
lastEvent = 0;
|
|
61
|
+
armedMs = 0;
|
|
62
|
+
/** True while a TOOL is executing: the model stream is legitimately idle then,
|
|
63
|
+
* and that window belongs to the command watchdog, not to this one. Without
|
|
64
|
+
* this, a 10-minute build would look identical to a hung stream. */
|
|
65
|
+
suspended = false;
|
|
66
|
+
fired = false;
|
|
67
|
+
constructor(deps) {
|
|
68
|
+
this.deps = deps;
|
|
69
|
+
}
|
|
70
|
+
/** Begin watching a model request. No-op when the watchdog is off or already armed. */
|
|
71
|
+
start() {
|
|
72
|
+
if (this.timer !== undefined) {
|
|
73
|
+
// Already watching — a new request inside the same agent loop just
|
|
74
|
+
// counts as activity rather than restarting the machine.
|
|
75
|
+
this.note();
|
|
76
|
+
return;
|
|
77
|
+
}
|
|
78
|
+
const ms = this.deps.getTimeoutMs();
|
|
79
|
+
if (!(ms > 0))
|
|
80
|
+
return;
|
|
81
|
+
this.armedMs = ms;
|
|
82
|
+
this.fired = false;
|
|
83
|
+
this.suspended = false;
|
|
84
|
+
this.lastEvent = this.deps.now();
|
|
85
|
+
this.timer = this.deps.schedule(() => this.check(), pollIntervalMs(ms));
|
|
86
|
+
}
|
|
87
|
+
/** Any stream event of any kind: resets the idle clock. */
|
|
88
|
+
note() {
|
|
89
|
+
this.lastEvent = this.deps.now();
|
|
90
|
+
}
|
|
91
|
+
/** A tool started executing — pause the idle clock until it ends. */
|
|
92
|
+
suspend() {
|
|
93
|
+
this.suspended = true;
|
|
94
|
+
}
|
|
95
|
+
/** A tool finished — the stream is expected to resume; restart the clock. */
|
|
96
|
+
resume() {
|
|
97
|
+
this.suspended = false;
|
|
98
|
+
this.note();
|
|
99
|
+
}
|
|
100
|
+
/** Stop watching (turn/agent/session end, or child exit). Safe to call twice. */
|
|
101
|
+
stop() {
|
|
102
|
+
if (this.timer !== undefined)
|
|
103
|
+
this.deps.cancel(this.timer);
|
|
104
|
+
this.timer = undefined;
|
|
105
|
+
this.suspended = false;
|
|
106
|
+
}
|
|
107
|
+
/** @internal Exposed for the poll callback and tests. */
|
|
108
|
+
check() {
|
|
109
|
+
if (this.timer === undefined || this.fired || this.suspended)
|
|
110
|
+
return;
|
|
111
|
+
const idle = this.deps.now() - this.lastEvent;
|
|
112
|
+
if (idle < this.armedMs)
|
|
113
|
+
return;
|
|
114
|
+
this.fired = true;
|
|
115
|
+
const armed = this.armedMs;
|
|
116
|
+
this.stop();
|
|
117
|
+
this.deps.onFire(idle, armed);
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
/**
|
|
121
|
+
* Real-clock poll deps, unref'd so a pending watchdog poll can never itself keep
|
|
122
|
+
* the process alive on exit. `schedule` returns a repeating interval — the
|
|
123
|
+
* machine cancels it on stop/fire.
|
|
124
|
+
*/
|
|
125
|
+
export const realStreamTimerDeps = {
|
|
126
|
+
now: () => Date.now(),
|
|
127
|
+
schedule: (fn, ms) => {
|
|
128
|
+
const handle = setInterval(fn, ms);
|
|
129
|
+
if (typeof handle.unref === 'function') {
|
|
130
|
+
;
|
|
131
|
+
handle.unref();
|
|
132
|
+
}
|
|
133
|
+
return handle;
|
|
134
|
+
},
|
|
135
|
+
cancel: handle => clearInterval(handle)
|
|
136
|
+
};
|
|
137
|
+
/**
|
|
138
|
+
* The cause string a CHILD's stream stall is reported as. Phrased so
|
|
139
|
+
* {@link isConnectionError} (child-runner.ts) matches it — the whole point is to
|
|
140
|
+
* route a silent hang into the retry path that already exists for a LOUD
|
|
141
|
+
* connection failure, rather than inventing a second one. Honest about who
|
|
142
|
+
* killed it: pi-task aborted the request, the provider did not report anything.
|
|
143
|
+
*/
|
|
144
|
+
export function streamStallCause(idleMs) {
|
|
145
|
+
return (`model stream inactivity: no stream events for ${minutes(idleMs)} — `
|
|
146
|
+
+ `connection lost (aborted by pi-task's stream watchdog; the provider `
|
|
147
|
+
+ `reported no error)`);
|
|
148
|
+
}
|
|
149
|
+
/**
|
|
150
|
+
* MAIN-SESSION reminder, delivered as a follow-up turn after ctx.abort() ended the
|
|
151
|
+
* hung turn. Carries the shared WATCHDOG_CANCEL_MARKER so steerUntilDone
|
|
152
|
+
* recognises it as a watchdog recovery rather than a human ESC (see
|
|
153
|
+
* task/command-watchdog.ts) — one marker, one abort channel.
|
|
154
|
+
*
|
|
155
|
+
* Idempotent resume, not a re-send: the aborted turn's tool calls and their
|
|
156
|
+
* results are already in the transcript, so the model is told to CONTINUE from
|
|
157
|
+
* what is recorded. Re-issuing the turn wholesale would re-run tool calls that
|
|
158
|
+
* already ran.
|
|
159
|
+
*/
|
|
160
|
+
export function streamStallReminder(idleMs, marker) {
|
|
161
|
+
return (`[SYSTEM] The model stream produced no events for ${minutes(idleMs)} — the response `
|
|
162
|
+
+ `appeared to hang, so the turn ${marker} `
|
|
163
|
+
+ `Nothing was reported as failed and no error was raised: the stream simply went `
|
|
164
|
+
+ `silent, so any work the turn had ALREADY completed (tool calls and their results) `
|
|
165
|
+
+ `is still recorded above and must NOT be repeated. Continue from that recorded `
|
|
166
|
+
+ `state: re-read anything you are unsure about, then carry on with the remaining `
|
|
167
|
+
+ `work. Do not restart the task from the beginning.`);
|
|
168
|
+
}
|
|
169
|
+
/**
|
|
170
|
+
* CHILD restart hint for a re-spawned worker whose previous attempt was killed by
|
|
171
|
+
* the stream watchdog. The killed child is gone and never saw an error, so this
|
|
172
|
+
* states plainly what happened and, unlike the command-timeout hint, does NOT
|
|
173
|
+
* blame the model: a hung stream is an infrastructure fault, and telling the model
|
|
174
|
+
* to "be faster" would teach it to truncate its work for no reason.
|
|
175
|
+
*/
|
|
176
|
+
export function streamStallHint(idleMs) {
|
|
177
|
+
return (`[SYSTEM NOTE: Your previous attempt was aborted because the model stream stopped `
|
|
178
|
+
+ `producing output for ${minutes(idleMs)} — an infrastructure hang, not a mistake `
|
|
179
|
+
+ `you made. That attempt's conversation is gone, but any file edits or command side `
|
|
180
|
+
+ `effects it made are still in the working tree, so check the current state before `
|
|
181
|
+
+ `assuming files are untouched. Redo the work from here.]`);
|
|
182
|
+
}
|
|
@@ -22,8 +22,15 @@
|
|
|
22
22
|
* deletion ships in the next commit. Recorded so the final gate re-checks it:
|
|
23
23
|
* resolved iff the named file is back in the tree (a later task restored it),
|
|
24
24
|
* otherwise surfaced.
|
|
25
|
+
* - 'final-gate' — the final integration gate DEMOTED one of its own checks to
|
|
26
|
+
* UNOBSERVED (mx5 run 14): two fix attempts that both changed the tree and
|
|
27
|
+
* re-ran the gate produced a byte-identical ranked-first failure, so the check
|
|
28
|
+
* is unfalsifiable in this environment (run 14: the boot probe needed `ss` or
|
|
29
|
+
* `lsof` and the sandbox had neither) and no further attempt can move it. The
|
|
30
|
+
* run is allowed to converge on the REMAINING checks, carrying this one here
|
|
31
|
+
* so the next run's gate re-checks and re-surfaces it rather than losing it.
|
|
25
32
|
*/
|
|
26
|
-
export type DebtOrigin = 'accepted' | 'enforce-revert' | 'frozen-blocked' | 'cross-task-deletion' | 'yolo-accepted';
|
|
33
|
+
export type DebtOrigin = 'accepted' | 'enforce-revert' | 'frozen-blocked' | 'cross-task-deletion' | 'yolo-accepted' | 'final-gate';
|
|
27
34
|
/** One recorded defect: the task, why its VERIFY failed, and how it was recorded. */
|
|
28
35
|
export interface AcceptDebt {
|
|
29
36
|
taskId: string;
|
|
@@ -89,6 +96,15 @@ export declare function recordCrossTaskDeletionDebt(cwd: string, taskId: string,
|
|
|
89
96
|
* reading only the artifacts can never read it as "a human decided this".
|
|
90
97
|
*/
|
|
91
98
|
export declare function recordYoloAcceptDebt(cwd: string, taskId: string, reason: string): Promise<void>;
|
|
99
|
+
/**
|
|
100
|
+
* Record a FINAL-GATE UNOBSERVED debt (mx5 run 14): the final gate demoted one of its
|
|
101
|
+
* OWN checks after two tree-changing fix attempts returned an identical ranked-first
|
|
102
|
+
* failure — unfalsifiable in this environment, so the gate stopped paying for it and
|
|
103
|
+
* converged on the remaining checks. Durable so the NEXT run's gate re-checks it: it
|
|
104
|
+
* is model-/environment-judged, so the re-check surfaces it (never auto-closes it).
|
|
105
|
+
* The taskId is the run's parent id — the demotion is a run-level decision.
|
|
106
|
+
*/
|
|
107
|
+
export declare function recordFinalGateUnobservedDebt(cwd: string, taskId: string, reason: string): Promise<void>;
|
|
92
108
|
/**
|
|
93
109
|
* The deleted path a cross-task-deletion debt names (the fixed shape
|
|
94
110
|
* recordCrossTaskDeletionDebt writes). Null on any other reason text — an
|