@mjasnikovs/pi-task 0.18.37 → 0.18.39

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -65,7 +65,7 @@ A whole plan — `/task-auto` splits it into an ordered task list and runs each
65
65
  | `/task-auto <feature>` | Plan a feature into a task list and run each title through `/task` in order (resumable). |
66
66
  | `/task-auto-resume` | Resume the active `/task-auto` run at the next unfinished task. |
67
67
  | `/task-auto-cancel` | Stop the `/task-auto` loop after the current task (still resumable). |
68
- | `/task-config` | Toggle pi-task settings in an editor dialog: remote server, compress reasoning, auto-commit, orientation, verify work, enforce guidelines, command timeout, and the extension whitelist for child sessions. |
68
+ | `/task-config` | Toggle pi-task settings in an editor dialog: remote server, compress reasoning, auto-commit, orientation, verify work, enforce guidelines, command timeout, stream watchdog, and the extension whitelist for child sessions. |
69
69
  | `/remote` | Show the QR code & URLs for the web view (`/remote stop` to stop). Answer grill questions, start tasks, and watch progress from your phone. |
70
70
 
71
71
  ## The pipeline
@@ -180,6 +180,7 @@ Run `/task-config` to toggle pi-task's behavior in an editor dialog. Settings pe
180
180
  | **research cache** | on | Cache docs/search/fetch worker results for the duration of one `/task-auto` run so sibling tasks re-asking the same package/URL + query reuse the first pipeline's digest instead of re-fetching. Per-run isolated, external-only (project-source `.` lookups excluded), success-only. |
181
181
  | **search provider** | Exa | Engine behind `pi-worker-search` and freshness/enrichment checks. **Exa** (default) and **DuckDuckGo** need no API key; **Brave** requires `BRAVE_SEARCH_API_KEY`. |
182
182
  | **command timeout** | 15 min | Wall-clock ceiling on a **single** tool execution. Local models routinely run a command that never returns (a hung build, a dev server, a check with no timeout) and the run wedges until you abort by hand — pi's bash tool has an optional timeout with no default, so this is the missing one. One knob, two surfaces: in the main session the overrun call is cancelled (killing the tool's whole process tree) plus a reminder turn; in the verify/fix gate children the child is killed and re-spawned with a hint, halving the ceiling on repeat hangs. Choices: 5/10/15/30 min or **off** — off unguards both surfaces, gates included. |
183
+ | **stream watchdog** | 10 min | Inactivity ceiling on the **model stream**. A hung or silently-dropped stream throws nothing at all, so neither the connection-error retry (it needs a reported error) nor the **command timeout** (tool calls only) nor the dead-backend stall guard (a reachable endpoint reads as proof of life) can see it — an mx5 run lost ~2.9h to three of them while the model server stayed healthy. Measured as time since the **last stream event of any kind**, so a slow model emitting one token every 30s is never touched, and it pauses while a tool runs. On expiry the main session aborts the turn (through the same channel the command watchdog uses) and posts a resume reminder; a child is killed and routed into the existing connection-error retry. Choices: 5/10/20/30 min or **off**. Keep it generous on local backends — prompt processing on a large context legitimately emits nothing for minutes. |
183
184
  | **yolo mode** | off | **Unattended runs.** Wherever pi-task would stop and ask, it takes the option already marked RECOMMENDED, stamps the artifact `(YOLO)` so an audit can tell a machine decided, and shows no prompt at all — clarify/grill answers, the verify-FAIL picker (auto-**Accept**, recorded as a yolo debt), and the final-gate picker (autofix while the budget lasts, then leave the run FAILED). A question with no recommendation is **skipped**, never invented. For throwaway/test projects nobody is watching; a real run should decide these itself. |
184
185
  | **extension whitelist** | empty | Host `pi` extensions to load into every child session by explicit path. Children otherwise run with extensions off, so a provider registered by an extension (e.g. `pi-lmstudio`) doesn't exist in them and they can't resolve the default model. `/task-config` enumerates the currently installed extensions as individual `ext: …` toggles; the list is strictly additive (discovery stays off), and an entry whose file is gone is skipped at spawn time, never fatal. |
185
186
 
@@ -61,6 +61,26 @@ export interface PiTaskConfig {
61
61
  * a true hang doesn't cost half an hour of dead time.
62
62
  */
63
63
  requestTimeoutMs: number;
64
+ /**
65
+ * Inactivity ceiling (ms) on the MODEL STREAM before the stream watchdog
66
+ * aborts the request (shared/stream-watchdog.ts). A hung or silently-dropped
67
+ * stream throws nothing, so neither the connection-error retry (needs a
68
+ * reported error) nor the command watchdog (covers tool calls only) nor the
69
+ * child stall guard (a reachable endpoint is proof of life) can see it — mx5
70
+ * run 14 lost ~2.9h to three such hangs while the model server stayed healthy.
71
+ *
72
+ * Measured as time since the LAST stream event of any kind, so a slow model
73
+ * emitting one token every 30s is never touched; only total silence counts.
74
+ * Suspended while a tool executes — that window is the command watchdog's.
75
+ *
76
+ * ONE knob, TWO surfaces: the MAIN session aborts the turn through the same
77
+ * plumbing the command watchdog uses and posts a resume reminder; a CHILD is
78
+ * killed and its result routed into the EXISTING connection-error retry.
79
+ * 0 = off, on both surfaces.
80
+ * DEFAULT 10 min: local prompt processing on a 32k context legitimately emits
81
+ * nothing for minutes, so a 60-120s ceiling would kill healthy long prompts.
82
+ */
83
+ streamInactivityMs: number;
64
84
  /**
65
85
  * UNATTENDED AUTO-PICK (see task/yolo.ts): wherever pi-task would stop and ask,
66
86
  * take the option it already marks RECOMMENDED, stamp the artifact `(YOLO)` so
@@ -91,6 +111,18 @@ export declare const COMMAND_TIMEOUT_OPTIONS: ReadonlyArray<{
91
111
  * one of the offered choices so the watchdog never arms on a nonsense value.
92
112
  */
93
113
  export declare function sanitizeRequestTimeoutMs(value: unknown): number;
114
+ /**
115
+ * The stream-watchdog choices offered by /task-config, in cycle order. Every
116
+ * option is minutes, not seconds: the failure this guards costs hours, and the
117
+ * legitimate silence it must tolerate (local prompt processing) is minutes.
118
+ */
119
+ export declare const STREAM_INACTIVITY_OPTIONS: ReadonlyArray<{
120
+ label: string;
121
+ ms: number;
122
+ }>;
123
+ /** Same pinning as {@link sanitizeRequestTimeoutMs}: a hand-edited value that is
124
+ * not one of the offered choices falls back to the default. */
125
+ export declare function sanitizeStreamInactivityMs(value: unknown): number;
94
126
  /**
95
127
  * A hand-edited config can hold anything; keep only string entries so a stray
96
128
  * object/number can't reach the child argv as `-e [object Object]`.
@@ -3,6 +3,7 @@ import * as fsp from 'node:fs/promises';
3
3
  import * as path from 'node:path';
4
4
  import * as os from 'node:os';
5
5
  import { isSearchProvider } from '../workers/search-types.js';
6
+ import { DEFAULT_STREAM_INACTIVITY_MS } from '../shared/stream-watchdog.js';
6
7
  /**
7
8
  * The command-watchdog timeout choices offered by /task-config, newest-first in
8
9
  * the cycle order the picker shows. The stored config value is the ms number;
@@ -25,6 +26,25 @@ export function sanitizeRequestTimeoutMs(value) {
25
26
  value
26
27
  : DEFAULT_REQUEST_TIMEOUT_MS;
27
28
  }
29
+ /**
30
+ * The stream-watchdog choices offered by /task-config, in cycle order. Every
31
+ * option is minutes, not seconds: the failure this guards costs hours, and the
32
+ * legitimate silence it must tolerate (local prompt processing) is minutes.
33
+ */
34
+ export const STREAM_INACTIVITY_OPTIONS = [
35
+ { label: '5 min', ms: 5 * 60_000 },
36
+ { label: '10 min', ms: DEFAULT_STREAM_INACTIVITY_MS },
37
+ { label: '20 min', ms: 20 * 60_000 },
38
+ { label: '30 min', ms: 30 * 60_000 },
39
+ { label: 'off', ms: 0 }
40
+ ];
41
+ /** Same pinning as {@link sanitizeRequestTimeoutMs}: a hand-edited value that is
42
+ * not one of the offered choices falls back to the default. */
43
+ export function sanitizeStreamInactivityMs(value) {
44
+ return STREAM_INACTIVITY_OPTIONS.some(o => o.ms === value) ?
45
+ value
46
+ : DEFAULT_STREAM_INACTIVITY_MS;
47
+ }
28
48
  const DEFAULTS = {
29
49
  remote: true,
30
50
  compressReasoning: true,
@@ -39,6 +59,7 @@ const DEFAULTS = {
39
59
  searchProvider: 'exa',
40
60
  extensionWhitelist: [],
41
61
  requestTimeoutMs: DEFAULT_REQUEST_TIMEOUT_MS,
62
+ streamInactivityMs: DEFAULT_STREAM_INACTIVITY_MS,
42
63
  // OFF: auto-answering is for unattended throwaway runs only.
43
64
  yoloMode: false
44
65
  };
@@ -69,6 +90,7 @@ if (!G.loaded) {
69
90
  delete parsed.searchProvider;
70
91
  parsed.extensionWhitelist = sanitizeExtensionWhitelist(parsed.extensionWhitelist);
71
92
  parsed.requestTimeoutMs = sanitizeRequestTimeoutMs(parsed.requestTimeoutMs);
93
+ parsed.streamInactivityMs = sanitizeStreamInactivityMs(parsed.streamInactivityMs);
72
94
  // A hand-edited `"yoloMode": "false"` is a truthy string — it must not
73
95
  // silently switch a watched run into unattended auto-pick. Only a real
74
96
  // boolean counts; anything else falls back to the OFF default.
@@ -1,7 +1,7 @@
1
1
  import { SettingsList, visibleWidth } from '@earendil-works/pi-tui';
2
2
  import { registerBridgeCommand } from '../remote/bridge.js';
3
3
  import { SEARCH_PROVIDERS, SEARCH_PROVIDER_LABELS, providerForLabel } from '../workers/search-types.js';
4
- import { COMMAND_TIMEOUT_OPTIONS, getConfig, saveConfig } from './config.js';
4
+ import { COMMAND_TIMEOUT_OPTIONS, getConfig, saveConfig, STREAM_INACTIVITY_OPTIONS } from './config.js';
5
5
  import { listInstalledExtensions } from './extension-list.js';
6
6
  const CONFIG_TITLE = 'pi-task settings';
7
7
  /**
@@ -108,6 +108,19 @@ const ITEMS = [
108
108
  // Display human labels; the stored config value stays the ms number.
109
109
  values: COMMAND_TIMEOUT_OPTIONS.map(o => o.label)
110
110
  },
111
+ {
112
+ id: 'streamInactivityMs',
113
+ label: 'stream watchdog',
114
+ description: 'Abort and retry a model request whose stream goes SILENT for this long — a hung '
115
+ + 'or dropped stream reports no error at all, so nothing else catches it (mx5 run '
116
+ + '14 lost ~2.9h to three of them while the model server stayed healthy). Counts '
117
+ + 'time since the last stream event of any kind, so a slow model that keeps '
118
+ + 'emitting tokens is never touched, and it pauses while a tool runs. Keep it '
119
+ + 'generous on local backends: prompt processing on a large context legitimately '
120
+ + 'emits nothing for minutes. off disables it on both the main session and children',
121
+ // Display human labels; the stored config value stays the ms number.
122
+ values: STREAM_INACTIVITY_OPTIONS.map(o => o.label)
123
+ },
111
124
  {
112
125
  id: 'yoloMode',
113
126
  label: 'yolo mode',
@@ -123,12 +136,18 @@ const ITEMS = [
123
136
  function timeoutLabel(ms) {
124
137
  return COMMAND_TIMEOUT_OPTIONS.find(o => o.ms === ms)?.label ?? `${ms}ms`;
125
138
  }
139
+ /** Human label for the stored stream-inactivity ms (falls back to the raw ms). */
140
+ function streamTimeoutLabel(ms) {
141
+ return STREAM_INACTIVITY_OPTIONS.find(o => o.ms === ms)?.label ?? `${ms}ms`;
142
+ }
126
143
  /** What /task-config shows for a setting's current value. */
127
144
  function displayValue(cfg, id, isEnum) {
128
145
  if (id === 'searchProvider')
129
146
  return SEARCH_PROVIDER_LABELS[cfg.searchProvider];
130
147
  if (id === 'requestTimeoutMs')
131
148
  return timeoutLabel(cfg.requestTimeoutMs);
149
+ if (id === 'streamInactivityMs')
150
+ return streamTimeoutLabel(cfg.streamInactivityMs);
132
151
  if (isEnum)
133
152
  return String(cfg[id]);
134
153
  return cfg[id] ? 'on' : 'off';
@@ -206,6 +225,11 @@ async function handleTaskConfig(_args, ctx) {
206
225
  if (opt)
207
226
  cfg.requestTimeoutMs = opt.ms;
208
227
  }
228
+ else if (id === 'streamInactivityMs') {
229
+ const opt = STREAM_INACTIVITY_OPTIONS.find(o => o.label === newValue);
230
+ if (opt)
231
+ cfg.streamInactivityMs = opt.ms;
232
+ }
209
233
  else {
210
234
  ;
211
235
  cfg[id] = newValue === 'on';
package/dist/index.js CHANGED
@@ -5,6 +5,7 @@ import { registerWorkers } from './workers/index.js';
5
5
  import { registerRemote } from './remote/register.js';
6
6
  import { registerThinkingCompression } from './thinking/compress.js';
7
7
  import { registerCommandWatchdog } from './task/command-watchdog.js';
8
+ import { registerStreamWatchdog } from './task/stream-watchdog.js';
8
9
  export default function (pi) {
9
10
  registerConfig(pi);
10
11
  registerTask(pi);
@@ -13,4 +14,5 @@ export default function (pi) {
13
14
  registerRemote(pi);
14
15
  registerThinkingCompression(pi);
15
16
  registerCommandWatchdog(pi);
17
+ registerStreamWatchdog(pi);
16
18
  }
@@ -54,6 +54,17 @@ export interface ChildResult {
54
54
  * and without the flag it would mislabel as a user cancel.
55
55
  */
56
56
  stalled?: boolean;
57
+ /**
58
+ * true when the STREAM watchdog killed the child: no output at all for the
59
+ * configured inactivity window, regardless of whether the backend answers a
60
+ * probe. Distinct from `stalled`, which requires an UNREACHABLE endpoint —
61
+ * the run-14 hangs had a perfectly healthy server and a dead stream, so the
62
+ * probe path could never fire. Callers must check this BEFORE `aborted`
63
+ * (the kill sets aborted too) and route it into the connection-error retry.
64
+ */
65
+ streamStalled?: {
66
+ idleMs: number;
67
+ };
57
68
  }
58
69
  export interface ToolCall {
59
70
  name: string;
@@ -124,6 +135,15 @@ export interface RunChildJsonEventsOptions {
124
135
  afterMs: number;
125
136
  probe: () => Promise<boolean>;
126
137
  };
138
+ /**
139
+ * Stream-inactivity ceiling in ms (shared/stream-watchdog.ts). Unlike `stall`
140
+ * this asks NOTHING of the backend: a stream that has produced no bytes for
141
+ * this long is dead whether or not the server answers a health probe, which
142
+ * is exactly the run-14 shape (server Up(healthy), stream silent for hours).
143
+ * Any stdout/stderr chunk resets it, so a slow model emitting one token every
144
+ * 30s is never killed. 0 / omitted = off.
145
+ */
146
+ streamInactivityMs?: number;
127
147
  }
128
148
  export type RunChildOptions = RunChildTextOptions | RunChildJsonEventsOptions;
129
149
  /**
@@ -1,4 +1,5 @@
1
1
  import { spawn as defaultSpawn, spawnSync as spawnSyncDefault } from 'node:child_process';
2
+ import { realStreamTimerDeps, StreamWatchdog } from './stream-watchdog.js';
2
3
  /** Grace period between SIGTERM and SIGKILL (ms). */
3
4
  export const KILL_GRACE_MS = 5000;
4
5
  /** Base flags shared by all child pi invocations. */
@@ -248,7 +249,43 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
248
249
  reapGroup('SIGKILL');
249
250
  }, KILL_GRACE_MS);
250
251
  };
251
- const sink = opts?.mode === 'json-events' ? new JsonEventSink(opts, killProc) : null;
252
+ // Stream watchdog (json-events children only): a silent stream is killed
253
+ // even when the backend is healthy — the case the probe-based stall guard
254
+ // below structurally cannot catch. Suspended for the duration of a tool
255
+ // call: a 12-minute build legitimately emits nothing, and that window is
256
+ // the COMMAND watchdog's to police, not this one's.
257
+ let streamStalledIdleMs;
258
+ const streamWatch = opts?.mode === 'json-events' && (opts.streamInactivityMs ?? 0) > 0 ?
259
+ new StreamWatchdog({
260
+ getTimeoutMs: () => opts.streamInactivityMs,
261
+ ...realStreamTimerDeps,
262
+ onFire: idleMs => {
263
+ streamStalledIdleMs = idleMs;
264
+ killProc();
265
+ }
266
+ })
267
+ : null;
268
+ streamWatch?.start();
269
+ // The sink sees the child's parsed events; tool start/end are what tell
270
+ // the watchdog to pause and resume. Wrapping the caller's handlers keeps
271
+ // that wiring invisible to callers — and onToolResult must be present for
272
+ // the sink to emit tool_execution_end at all, so it is always supplied
273
+ // when the watchdog is on.
274
+ const sinkOpts = opts?.mode !== 'json-events' ? null
275
+ : streamWatch ?
276
+ {
277
+ ...opts,
278
+ onToolCall: call => {
279
+ streamWatch.suspend();
280
+ return opts.onToolCall ? opts.onToolCall(call) : null;
281
+ },
282
+ onToolResult: r => {
283
+ streamWatch.resume();
284
+ opts.onToolResult?.(r);
285
+ }
286
+ }
287
+ : opts;
288
+ const sink = sinkOpts ? new JsonEventSink(sinkOpts, killProc) : null;
252
289
  // Dead-backend stall guard (json-events children only; see the option
253
290
  // docs). Any output resets the window; a reachable probe also resets it
254
291
  // so the next probe is a full window away, not every tick.
@@ -283,6 +320,7 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
283
320
  let firstByteFired = false;
284
321
  proc.stdout?.on('data', (d) => {
285
322
  lastActivity = Date.now();
323
+ streamWatch?.note();
286
324
  if (!firstByteFired) {
287
325
  firstByteFired = true;
288
326
  opts?.onFirstByte?.();
@@ -297,11 +335,13 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
297
335
  });
298
336
  proc.stderr?.on('data', (d) => {
299
337
  lastActivity = Date.now();
338
+ streamWatch?.note();
300
339
  stderr += d.toString();
301
340
  });
302
341
  proc.on('close', (code) => {
303
342
  if (stallTimer)
304
343
  clearInterval(stallTimer);
344
+ streamWatch?.stop();
305
345
  // The child has exited, but anything it backgrounded (a dev server) may
306
346
  // still hold its process group and a port — reap the group so the next
307
347
  // gate's boot check does not collide with our own orphan. Best-effort:
@@ -320,12 +360,16 @@ export function runChild(spawn, invocation, cwd, signal, opts) {
320
360
  aborted,
321
361
  text,
322
362
  modelError: sink?.modelError,
323
- ...(stalled ? { stalled: true } : {})
363
+ ...(stalled ? { stalled: true } : {}),
364
+ ...(streamStalledIdleMs !== undefined ?
365
+ { streamStalled: { idleMs: streamStalledIdleMs } }
366
+ : {})
324
367
  });
325
368
  });
326
369
  proc.on('error', () => {
327
370
  if (stallTimer)
328
371
  clearInterval(stallTimer);
372
+ streamWatch?.stop();
329
373
  resolve({ stdout, stderr, exitCode: 1, aborted });
330
374
  });
331
375
  if (signal) {
@@ -0,0 +1,119 @@
1
+ /**
2
+ * Model-stream watchdog — the inactivity machine for a stream that goes SILENT
3
+ * without erroring.
4
+ *
5
+ * WHY (mx5 run 14): three main-session implementation turns died mid-turn — the
6
+ * session jsonl's last event is an ordinary assistant message, then nothing,
7
+ * forever, while the model server stayed Up(healthy) the whole time. A hung or
8
+ * silently-dropped stream throws NOTHING, so:
9
+ * - the connection-error retry (child-runner.ts) never fires: it needs a
10
+ * thrown/reported ModelError,
11
+ * - the command watchdog never fires: it only covers TOOL executions,
12
+ * - the child stall guard never fires: it treats a reachable endpoint as proof
13
+ * of life, which it is — the endpoint was fine, the stream was not.
14
+ * Cost in run 14: ~2.9h of dead air awaiting manual restarts.
15
+ *
16
+ * WHAT THIS MEASURES: time since the LAST stream event of ANY kind — text token,
17
+ * tool-call delta, thinking delta, provider response header. NOT wall-clock, and
18
+ * NOT "time to first token". One token every 30s is a working local model and
19
+ * must never be killed; zero events for the whole window is a hang.
20
+ *
21
+ * WHY THE DEFAULT IS GENEROUS: on a local llama-server a 32k-context prompt can
22
+ * spend many minutes in prompt processing emitting nothing at all. A 60-120s
23
+ * ceiling would kill every long prompt on local hardware. See
24
+ * DEFAULT_STREAM_INACTIVITY_MS.
25
+ *
26
+ * TWO SURFACES, one machine (same split as command-watchdog.ts):
27
+ * MAIN SESSION — task/stream-watchdog.ts arms it from pi's extension events and
28
+ * fires ctx.abort() through the SAME abort plumbing the command
29
+ * watchdog uses (noteWatchdogAbort → steerUntilDone), so there is
30
+ * exactly one abort channel, not two racing ones.
31
+ * CHILDREN — shared/child-process.ts arms it on the child's stdout/stderr
32
+ * chunks; a fire kills the child and the result carries
33
+ * `streamStalled`, which child-runner turns into a
34
+ * connection-class cause so the EXISTING retry/backoff path
35
+ * handles it (re-spawn from the last durable prompt, never a
36
+ * blind re-send of a half-executed turn).
37
+ */
38
+ import type { TimerHandle } from './command-watchdog.js';
39
+ export type { TimerHandle };
40
+ /**
41
+ * Default inactivity ceiling: 10 minutes. Chosen from the constraint that a local
42
+ * model's first token can legitimately be many minutes away — the guard exists to
43
+ * turn hours of dead air into minutes, not to police slowness.
44
+ */
45
+ export declare const DEFAULT_STREAM_INACTIVITY_MS: number;
46
+ /**
47
+ * How often the machine checks the idle clock. A poll (rather than re-arming a
48
+ * timeout on every token) keeps cost O(1) per window instead of O(1) per token —
49
+ * a streaming turn emits thousands of events. Same idiom as the child stall guard.
50
+ */
51
+ export declare function pollIntervalMs(timeoutMs: number): number;
52
+ export interface StreamWatchdogDeps {
53
+ /** The inactivity ceiling in ms, read when the watchdog arms. 0 (or any
54
+ * non-positive value) means the watchdog is off and never arms. */
55
+ getTimeoutMs: () => number;
56
+ now: () => number;
57
+ schedule: (fn: () => void, ms: number) => TimerHandle;
58
+ cancel: (handle: TimerHandle) => void;
59
+ /** Invoked once per arming when the stream has been silent for the ceiling. */
60
+ onFire: (idleMs: number, timeoutMs: number) => void;
61
+ }
62
+ export declare class StreamWatchdog {
63
+ private readonly deps;
64
+ private timer;
65
+ private lastEvent;
66
+ private armedMs;
67
+ /** True while a TOOL is executing: the model stream is legitimately idle then,
68
+ * and that window belongs to the command watchdog, not to this one. Without
69
+ * this, a 10-minute build would look identical to a hung stream. */
70
+ private suspended;
71
+ private fired;
72
+ constructor(deps: StreamWatchdogDeps);
73
+ /** Begin watching a model request. No-op when the watchdog is off or already armed. */
74
+ start(): void;
75
+ /** Any stream event of any kind: resets the idle clock. */
76
+ note(): void;
77
+ /** A tool started executing — pause the idle clock until it ends. */
78
+ suspend(): void;
79
+ /** A tool finished — the stream is expected to resume; restart the clock. */
80
+ resume(): void;
81
+ /** Stop watching (turn/agent/session end, or child exit). Safe to call twice. */
82
+ stop(): void;
83
+ /** @internal Exposed for the poll callback and tests. */
84
+ check(): void;
85
+ }
86
+ /**
87
+ * Real-clock poll deps, unref'd so a pending watchdog poll can never itself keep
88
+ * the process alive on exit. `schedule` returns a repeating interval — the
89
+ * machine cancels it on stop/fire.
90
+ */
91
+ export declare const realStreamTimerDeps: Pick<StreamWatchdogDeps, 'now' | 'schedule' | 'cancel'>;
92
+ /**
93
+ * The cause string a CHILD's stream stall is reported as. Phrased so
94
+ * {@link isConnectionError} (child-runner.ts) matches it — the whole point is to
95
+ * route a silent hang into the retry path that already exists for a LOUD
96
+ * connection failure, rather than inventing a second one. Honest about who
97
+ * killed it: pi-task aborted the request, the provider did not report anything.
98
+ */
99
+ export declare function streamStallCause(idleMs: number): string;
100
+ /**
101
+ * MAIN-SESSION reminder, delivered as a follow-up turn after ctx.abort() ended the
102
+ * hung turn. Carries the shared WATCHDOG_CANCEL_MARKER so steerUntilDone
103
+ * recognises it as a watchdog recovery rather than a human ESC (see
104
+ * task/command-watchdog.ts) — one marker, one abort channel.
105
+ *
106
+ * Idempotent resume, not a re-send: the aborted turn's tool calls and their
107
+ * results are already in the transcript, so the model is told to CONTINUE from
108
+ * what is recorded. Re-issuing the turn wholesale would re-run tool calls that
109
+ * already ran.
110
+ */
111
+ export declare function streamStallReminder(idleMs: number, marker: string): string;
112
+ /**
113
+ * CHILD restart hint for a re-spawned worker whose previous attempt was killed by
114
+ * the stream watchdog. The killed child is gone and never saw an error, so this
115
+ * states plainly what happened and, unlike the command-timeout hint, does NOT
116
+ * blame the model: a hung stream is an infrastructure fault, and telling the model
117
+ * to "be faster" would teach it to truncate its work for no reason.
118
+ */
119
+ export declare function streamStallHint(idleMs: number): string;
@@ -0,0 +1,182 @@
1
+ /**
2
+ * Model-stream watchdog — the inactivity machine for a stream that goes SILENT
3
+ * without erroring.
4
+ *
5
+ * WHY (mx5 run 14): three main-session implementation turns died mid-turn — the
6
+ * session jsonl's last event is an ordinary assistant message, then nothing,
7
+ * forever, while the model server stayed Up(healthy) the whole time. A hung or
8
+ * silently-dropped stream throws NOTHING, so:
9
+ * - the connection-error retry (child-runner.ts) never fires: it needs a
10
+ * thrown/reported ModelError,
11
+ * - the command watchdog never fires: it only covers TOOL executions,
12
+ * - the child stall guard never fires: it treats a reachable endpoint as proof
13
+ * of life, which it is — the endpoint was fine, the stream was not.
14
+ * Cost in run 14: ~2.9h of dead air awaiting manual restarts.
15
+ *
16
+ * WHAT THIS MEASURES: time since the LAST stream event of ANY kind — text token,
17
+ * tool-call delta, thinking delta, provider response header. NOT wall-clock, and
18
+ * NOT "time to first token". One token every 30s is a working local model and
19
+ * must never be killed; zero events for the whole window is a hang.
20
+ *
21
+ * WHY THE DEFAULT IS GENEROUS: on a local llama-server a 32k-context prompt can
22
+ * spend many minutes in prompt processing emitting nothing at all. A 60-120s
23
+ * ceiling would kill every long prompt on local hardware. See
24
+ * DEFAULT_STREAM_INACTIVITY_MS.
25
+ *
26
+ * TWO SURFACES, one machine (same split as command-watchdog.ts):
27
+ * MAIN SESSION — task/stream-watchdog.ts arms it from pi's extension events and
28
+ * fires ctx.abort() through the SAME abort plumbing the command
29
+ * watchdog uses (noteWatchdogAbort → steerUntilDone), so there is
30
+ * exactly one abort channel, not two racing ones.
31
+ * CHILDREN — shared/child-process.ts arms it on the child's stdout/stderr
32
+ * chunks; a fire kills the child and the result carries
33
+ * `streamStalled`, which child-runner turns into a
34
+ * connection-class cause so the EXISTING retry/backoff path
35
+ * handles it (re-spawn from the last durable prompt, never a
36
+ * blind re-send of a half-executed turn).
37
+ */
38
+ /**
39
+ * Default inactivity ceiling: 10 minutes. Chosen from the constraint that a local
40
+ * model's first token can legitimately be many minutes away — the guard exists to
41
+ * turn hours of dead air into minutes, not to police slowness.
42
+ */
43
+ export const DEFAULT_STREAM_INACTIVITY_MS = 10 * 60_000;
44
+ /** Whole minutes, floored at 1, for the human-facing window in every message. */
45
+ function minutes(ms) {
46
+ const mins = Math.max(1, Math.round(ms / 60_000));
47
+ return `${mins} minute${mins === 1 ? '' : 's'}`;
48
+ }
49
+ /**
50
+ * How often the machine checks the idle clock. A poll (rather than re-arming a
51
+ * timeout on every token) keeps cost O(1) per window instead of O(1) per token —
52
+ * a streaming turn emits thousands of events. Same idiom as the child stall guard.
53
+ */
54
+ export function pollIntervalMs(timeoutMs) {
55
+ return Math.max(50, Math.min(Math.floor(timeoutMs / 4), 30_000));
56
+ }
57
+ export class StreamWatchdog {
58
+ deps;
59
+ timer;
60
+ lastEvent = 0;
61
+ armedMs = 0;
62
+ /** True while a TOOL is executing: the model stream is legitimately idle then,
63
+ * and that window belongs to the command watchdog, not to this one. Without
64
+ * this, a 10-minute build would look identical to a hung stream. */
65
+ suspended = false;
66
+ fired = false;
67
+ constructor(deps) {
68
+ this.deps = deps;
69
+ }
70
+ /** Begin watching a model request. No-op when the watchdog is off or already armed. */
71
+ start() {
72
+ if (this.timer !== undefined) {
73
+ // Already watching — a new request inside the same agent loop just
74
+ // counts as activity rather than restarting the machine.
75
+ this.note();
76
+ return;
77
+ }
78
+ const ms = this.deps.getTimeoutMs();
79
+ if (!(ms > 0))
80
+ return;
81
+ this.armedMs = ms;
82
+ this.fired = false;
83
+ this.suspended = false;
84
+ this.lastEvent = this.deps.now();
85
+ this.timer = this.deps.schedule(() => this.check(), pollIntervalMs(ms));
86
+ }
87
+ /** Any stream event of any kind: resets the idle clock. */
88
+ note() {
89
+ this.lastEvent = this.deps.now();
90
+ }
91
+ /** A tool started executing — pause the idle clock until it ends. */
92
+ suspend() {
93
+ this.suspended = true;
94
+ }
95
+ /** A tool finished — the stream is expected to resume; restart the clock. */
96
+ resume() {
97
+ this.suspended = false;
98
+ this.note();
99
+ }
100
+ /** Stop watching (turn/agent/session end, or child exit). Safe to call twice. */
101
+ stop() {
102
+ if (this.timer !== undefined)
103
+ this.deps.cancel(this.timer);
104
+ this.timer = undefined;
105
+ this.suspended = false;
106
+ }
107
+ /** @internal Exposed for the poll callback and tests. */
108
+ check() {
109
+ if (this.timer === undefined || this.fired || this.suspended)
110
+ return;
111
+ const idle = this.deps.now() - this.lastEvent;
112
+ if (idle < this.armedMs)
113
+ return;
114
+ this.fired = true;
115
+ const armed = this.armedMs;
116
+ this.stop();
117
+ this.deps.onFire(idle, armed);
118
+ }
119
+ }
120
+ /**
121
+ * Real-clock poll deps, unref'd so a pending watchdog poll can never itself keep
122
+ * the process alive on exit. `schedule` returns a repeating interval — the
123
+ * machine cancels it on stop/fire.
124
+ */
125
+ export const realStreamTimerDeps = {
126
+ now: () => Date.now(),
127
+ schedule: (fn, ms) => {
128
+ const handle = setInterval(fn, ms);
129
+ if (typeof handle.unref === 'function') {
130
+ ;
131
+ handle.unref();
132
+ }
133
+ return handle;
134
+ },
135
+ cancel: handle => clearInterval(handle)
136
+ };
137
+ /**
138
+ * The cause string a CHILD's stream stall is reported as. Phrased so
139
+ * {@link isConnectionError} (child-runner.ts) matches it — the whole point is to
140
+ * route a silent hang into the retry path that already exists for a LOUD
141
+ * connection failure, rather than inventing a second one. Honest about who
142
+ * killed it: pi-task aborted the request, the provider did not report anything.
143
+ */
144
+ export function streamStallCause(idleMs) {
145
+ return (`model stream inactivity: no stream events for ${minutes(idleMs)} — `
146
+ + `connection lost (aborted by pi-task's stream watchdog; the provider `
147
+ + `reported no error)`);
148
+ }
149
+ /**
150
+ * MAIN-SESSION reminder, delivered as a follow-up turn after ctx.abort() ended the
151
+ * hung turn. Carries the shared WATCHDOG_CANCEL_MARKER so steerUntilDone
152
+ * recognises it as a watchdog recovery rather than a human ESC (see
153
+ * task/command-watchdog.ts) — one marker, one abort channel.
154
+ *
155
+ * Idempotent resume, not a re-send: the aborted turn's tool calls and their
156
+ * results are already in the transcript, so the model is told to CONTINUE from
157
+ * what is recorded. Re-issuing the turn wholesale would re-run tool calls that
158
+ * already ran.
159
+ */
160
+ export function streamStallReminder(idleMs, marker) {
161
+ return (`[SYSTEM] The model stream produced no events for ${minutes(idleMs)} — the response `
162
+ + `appeared to hang, so the turn ${marker} `
163
+ + `Nothing was reported as failed and no error was raised: the stream simply went `
164
+ + `silent, so any work the turn had ALREADY completed (tool calls and their results) `
165
+ + `is still recorded above and must NOT be repeated. Continue from that recorded `
166
+ + `state: re-read anything you are unsure about, then carry on with the remaining `
167
+ + `work. Do not restart the task from the beginning.`);
168
+ }
169
+ /**
170
+ * CHILD restart hint for a re-spawned worker whose previous attempt was killed by
171
+ * the stream watchdog. The killed child is gone and never saw an error, so this
172
+ * states plainly what happened and, unlike the command-timeout hint, does NOT
173
+ * blame the model: a hung stream is an infrastructure fault, and telling the model
174
+ * to "be faster" would teach it to truncate its work for no reason.
175
+ */
176
+ export function streamStallHint(idleMs) {
177
+ return (`[SYSTEM NOTE: Your previous attempt was aborted because the model stream stopped `
178
+ + `producing output for ${minutes(idleMs)} — an infrastructure hang, not a mistake `
179
+ + `you made. That attempt's conversation is gone, but any file edits or command side `
180
+ + `effects it made are still in the working tree, so check the current state before `
181
+ + `assuming files are untouched. Redo the work from here.]`);
182
+ }
@@ -1518,13 +1518,15 @@ async function handleTaskAutoResume(_args, ctx) {
1518
1518
  autoRunning = true;
1519
1519
  armTerminalCancel(ctx);
1520
1520
  try {
1521
- // Reuse the interrupted run's research-cache id when the cache proves it still
1522
- // describes the same dependency surface (F10). mx5 run 13 resumed three times and
1523
- // each resume's fresh id discarded a working 201-entry cache; anything inconclusive
1524
- // still falls back to a fresh id and a re-fetch. See resumeResearchRun.
1521
+ // Reuse the interrupted run's research-cache id, dropping only the entries whose
1522
+ // own package moved version (F10). mx5 run 13 resumed three times and each
1523
+ // resume's fresh id discarded a working 201-entry cache; run 14 then showed a
1524
+ // whole-file freshness gate can never hold on a greenfield run that installs
1525
+ // packages as it goes, so invalidation is per entry. See resumeResearchRun.
1525
1526
  const research = await resumeResearchRun(cwd, getConfig().researchCache);
1526
1527
  if (research.reused) {
1527
- logPlanDebug(cwd, `research cache: resume reused ${research.entries} entr(ies)`);
1528
+ logPlanDebug(cwd, `research cache: resume reused ${research.entries} entr(ies), `
1529
+ + `dropped ${research.dropped} stale`);
1528
1530
  }
1529
1531
  const abort = new AbortController();
1530
1532
  // Resume only runs the loop (runTask); no planning children, so the loader