opencode-auto-resume 1.1.3 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +81 -16
- package/dist/index.js +586 -145
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -73,6 +73,81 @@ _Motivated by:_
|
|
|
73
73
|
|
|
74
74
|
---
|
|
75
75
|
|
|
76
|
+
### Streaming failure recovery
|
|
77
|
+
|
|
78
|
+
The AI provider's streaming response can fail mid-stream (connection reset, timeout, socket close). When the provider reports an error whose **name** matches `streamingFailureErrorNames` (exact, case-sensitive) or whose **message** matches `streamingFailureMessagePatterns` (regex, case-insensitive), the plugin arms a deferred recovery instead of relying only on the generic stall timeout. Invalid regex patterns fall back to substring matching.
|
|
79
|
+
|
|
80
|
+
**Default error names:**
|
|
81
|
+
- `ProviderError`, `APIError`, `StreamError`, `ConnectionError`, `TimeoutError`
|
|
82
|
+
|
|
83
|
+
**Default message patterns:**
|
|
84
|
+
- `streaming response failed`, `stream.*fail`, `connection.*reset`, `connection.*closed`
|
|
85
|
+
|
|
86
|
+
#### Recovery behavior
|
|
87
|
+
|
|
88
|
+
1. **Detection**: `session.error` is classified via `isStreamingFailure()` against the configured error names and message patterns
|
|
89
|
+
2. **State transition**: the session's `pendingRecovery` flag is armed with the error name (`pendingRecoveryReason`) and timestamp (`pendingRecoveryAt`)
|
|
90
|
+
3. **Recovery attempt**: once the session is idle and the backoff delay has elapsed, the timer loop sends a recovery prompt
|
|
91
|
+
4. **Watchdog**: if the session is still not busy 3 seconds after the prompt, the recovery is retried (up to `maxRecoveryRetries`) with exponential backoff
|
|
92
|
+
5. **Escalation**: when retries are exhausted, the plugin aborts the session and resumes it (`abort+resume`); `gaveUp` is set if that also fails
|
|
93
|
+
|
|
94
|
+
#### Configuration
|
|
95
|
+
|
|
96
|
+
Add to your plugin options (in `opencode.jsonc`):
|
|
97
|
+
|
|
98
|
+
```json
|
|
99
|
+
{
|
|
100
|
+
"streamingFailureErrorNames": ["ProviderError", "APIError", "StreamError", "ConnectionError", "TimeoutError"],
|
|
101
|
+
"streamingFailureMessagePatterns": ["streaming response failed", "stream.*fail", "connection.*reset", "connection.*closed"],
|
|
102
|
+
"maxRecoveryRetries": 2,
|
|
103
|
+
"baseBackoffMs": 1000,
|
|
104
|
+
"maxBackoffMs": 8000
|
|
105
|
+
}
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
#### State machine addition
|
|
109
|
+
|
|
110
|
+
New per-session recovery fields added to the state machine:
|
|
111
|
+
- `pendingRecovery` — failure detected, recovery armed
|
|
112
|
+
- `pendingRecoveryReason` — error name that triggered the recovery
|
|
113
|
+
- `pendingRecoveryAt` — detection timestamp (backoff anchor)
|
|
114
|
+
- `recoveryAttempts` — recovery attempt counter
|
|
115
|
+
- `watchdogRetryGuard` — watchdog retry in progress (keeps the recovery armed)
|
|
116
|
+
|
|
117
|
+
Recovery chain: `pendingRecovery` → recovery attempt → `recoveryAttempts` retry → `abort+resume` → `gaveUp`.
|
|
118
|
+
|
|
119
|
+
See [Recovery Flow Documentation](docs/architecture/recovery-flow.md) for the full state machine.
|
|
120
|
+
|
|
121
|
+
#### Example scenario
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
1. AI provider starts streaming response
|
|
125
|
+
2. Network interruption causes "connection reset" error mid-stream
|
|
126
|
+
3. System detects "ConnectionError" matches streamingFailureErrorNames
|
|
127
|
+
4. Session's pendingRecovery flag is armed (reason=ConnectionError)
|
|
128
|
+
5. Session goes idle; timer loop waits until the backoff delay has elapsed
|
|
129
|
+
6. Recovery prompt sent (recoveryAttempts=1)
|
|
130
|
+
7. Success → session busy → recovery flags cleared
|
|
131
|
+
Still not busy after 3s → watchdog retry (attempt 2/2)
|
|
132
|
+
Still not busy after 3s → maxRecoveryRetries reached → abort + resume
|
|
133
|
+
Abort+continue fails → gaveUp
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
#### Configuration reference
|
|
137
|
+
|
|
138
|
+
| Option | Type | Default | Description |
|
|
139
|
+
|--------|------|---------|-------------|
|
|
140
|
+
| `streamingFailureErrorNames` | `string[]` | `["ProviderError","APIError","StreamError","ConnectionError","TimeoutError"]` | Error names that indicate a streaming failure (exact, case-sensitive) |
|
|
141
|
+
| `streamingFailureMessagePatterns` | `string[]` | `["streaming response failed","stream.*fail","connection.*reset","connection.*closed"]` | Regex patterns matching streaming failure messages (case-insensitive) |
|
|
142
|
+
| `maxRecoveryRetries` | `number` | `2` | Maximum streaming-failure recovery attempts before abort+resume escalation |
|
|
143
|
+
| `baseBackoffMs` | `number` | `1000` | Initial backoff delay in milliseconds |
|
|
144
|
+
| `maxBackoffMs` | `number` | `8000` | Maximum backoff delay cap in milliseconds |
|
|
145
|
+
|
|
146
|
+
_Motivated by:_
|
|
147
|
+
- [EPIC: Streaming Failure Recovery](docs/EPIC-Streaming-Recovery-OpenCode-Auto-Resume-v3.md) — recovery requests not consistently creating new assistant executions after mid-stream failures
|
|
148
|
+
|
|
149
|
+
---
|
|
150
|
+
|
|
76
151
|
### Active-tool safety guard
|
|
77
152
|
|
|
78
153
|
Before **any** abort, the plugin calls `checkSessionHasActiveTool()` to verify the session isn't mid-tool-execution. If a tool is running, the abort is skipped. This prevents the plugin from killing a long-running build, test suite, or command — even when it looks like a stall.
|
|
@@ -190,11 +265,7 @@ Periodic: cleanup idle sessions older than 10min or >50 entries
|
|
|
190
265
|
|
|
191
266
|
## Installation
|
|
192
267
|
|
|
193
|
-
###
|
|
194
|
-
|
|
195
|
-
```bash
|
|
196
|
-
npm install opencode-auto-resume
|
|
197
|
-
```
|
|
268
|
+
### Opencode
|
|
198
269
|
|
|
199
270
|
Add to your `opencode.jsonc`:
|
|
200
271
|
|
|
@@ -219,17 +290,6 @@ With options:
|
|
|
219
290
|
}
|
|
220
291
|
```
|
|
221
292
|
|
|
222
|
-
### Via GitHub (manual clone)
|
|
223
|
-
|
|
224
|
-
OpenCode may clone the repository to `~/.config/opencode/plugins/opencode-auto-resume/` automatically.
|
|
225
|
-
|
|
226
|
-
**To update** the plugin:
|
|
227
|
-
```bash
|
|
228
|
-
cd ~/.config/opencode/plugins/opencode-auto-resume
|
|
229
|
-
git pull
|
|
230
|
-
bun run build
|
|
231
|
-
```
|
|
232
|
-
|
|
233
293
|
## Configuration
|
|
234
294
|
|
|
235
295
|
```json
|
|
@@ -256,6 +316,11 @@ bun run build
|
|
|
256
316
|
| `subagentWaitMs` | `15000` | Wait before treating orphan parent as stuck |
|
|
257
317
|
| `loopMaxContinues` | `3` | Continues in window before triggering abort |
|
|
258
318
|
| `loopWindowMs` | `600000` | Hallucination loop detection window (10 min) |
|
|
319
|
+
| `streamingFailureErrorNames` | `["ProviderError","APIError","StreamError","ConnectionError","TimeoutError"]` | Error names that classify as streaming failures (exact match) |
|
|
320
|
+
| `streamingFailureMessagePatterns` | `["streaming response failed","stream.*fail","connection.*reset","connection.*closed"]` | Regex patterns (case-insensitive) in error messages indicating streaming failure |
|
|
321
|
+
| `maxRecoveryRetries` | `2` | Max streaming-failure recovery attempts before abort+resume escalation |
|
|
322
|
+
|
|
323
|
+
Message patterns are matched case-insensitively. Error names use exact match.
|
|
259
324
|
|
|
260
325
|
### Internal constants (not configurable)
|
|
261
326
|
|
package/dist/index.js
CHANGED
|
@@ -12345,8 +12345,24 @@ var DEFAULT_SUBAGENT_WAIT_MS = 15000;
|
|
|
12345
12345
|
var ABORT_CONTINUE_DELAY_MS = 2000;
|
|
12346
12346
|
var DEFAULT_LOOP_MAX_CONTINUES = 3;
|
|
12347
12347
|
var DEFAULT_LOOP_WINDOW_MS = 10 * 60000;
|
|
12348
|
-
var
|
|
12349
|
-
var
|
|
12348
|
+
var DEFAULT_TOOL_TEXT_CHECK_DELAY_MS = 3000;
|
|
12349
|
+
var DEFAULT_MAX_RECOVERY_RETRIES = 2;
|
|
12350
|
+
var DEFAULT_MIN_ACTIVITY_GAP_MS = 1000;
|
|
12351
|
+
var DEFAULT_WARMUP_MS = 15000;
|
|
12352
|
+
var DEFAULT_DEBUG = false;
|
|
12353
|
+
var DEFAULT_STREAMING_FAILURE_ERROR_NAMES = [
|
|
12354
|
+
"ProviderError",
|
|
12355
|
+
"APIError",
|
|
12356
|
+
"StreamError",
|
|
12357
|
+
"ConnectionError",
|
|
12358
|
+
"TimeoutError"
|
|
12359
|
+
];
|
|
12360
|
+
var DEFAULT_STREAMING_FAILURE_MESSAGE_PATTERNS = [
|
|
12361
|
+
"streaming response failed",
|
|
12362
|
+
"stream.*fail",
|
|
12363
|
+
"connection.*reset",
|
|
12364
|
+
"connection.*closed"
|
|
12365
|
+
];
|
|
12350
12366
|
var MAX_IDLE_SESSIONS = 50;
|
|
12351
12367
|
var IDLE_CLEANUP_MS = 10 * 60000;
|
|
12352
12368
|
var SESSION_DISCOVERY_INTERVAL_MS = 60000;
|
|
@@ -12388,13 +12404,17 @@ var READY_TO_CONTINUE_PATTERNS = [
|
|
|
12388
12404
|
/will continue with task/i,
|
|
12389
12405
|
/moving on to task/i
|
|
12390
12406
|
];
|
|
12407
|
+
function stripCodeBlocks(text) {
|
|
12408
|
+
return text.replace(/```[\s\S]*?```/g, "").replace(/`[^`\n]+`/g, "");
|
|
12409
|
+
}
|
|
12391
12410
|
function containsToolCallAsText(text) {
|
|
12392
12411
|
if (text.length <= 10)
|
|
12393
12412
|
return false;
|
|
12394
|
-
|
|
12413
|
+
const stripped = stripCodeBlocks(text);
|
|
12414
|
+
if (TOOL_TEXT_PATTERNS.some((pat) => pat.test(stripped)))
|
|
12395
12415
|
return true;
|
|
12396
12416
|
for (const { open, close } of TRUNCATED_XML_PATTERNS) {
|
|
12397
|
-
if (open.test(
|
|
12417
|
+
if (open.test(stripped) && !close.test(stripped))
|
|
12398
12418
|
return true;
|
|
12399
12419
|
}
|
|
12400
12420
|
return false;
|
|
@@ -12429,17 +12449,97 @@ function containsDoneClaimPattern(text) {
|
|
|
12429
12449
|
`);
|
|
12430
12450
|
return DONE_CLAIM_PATTERNS.some((pat) => pat.test(lastLines));
|
|
12431
12451
|
}
|
|
12452
|
+
function isStreamingFailure(errorName, errorMessage, errorNames = DEFAULT_STREAMING_FAILURE_ERROR_NAMES, messagePatterns = DEFAULT_STREAMING_FAILURE_MESSAGE_PATTERNS) {
|
|
12453
|
+
if (!errorName && !errorMessage)
|
|
12454
|
+
return false;
|
|
12455
|
+
if (errorName && errorNames.includes(errorName)) {
|
|
12456
|
+
return true;
|
|
12457
|
+
}
|
|
12458
|
+
if (errorMessage) {
|
|
12459
|
+
const lowerMessage = errorMessage.toLowerCase();
|
|
12460
|
+
for (const pattern of messagePatterns) {
|
|
12461
|
+
try {
|
|
12462
|
+
if (new RegExp(pattern, "i").test(lowerMessage))
|
|
12463
|
+
return true;
|
|
12464
|
+
} catch {
|
|
12465
|
+
if (lowerMessage.includes(pattern.toLowerCase()))
|
|
12466
|
+
return true;
|
|
12467
|
+
}
|
|
12468
|
+
}
|
|
12469
|
+
}
|
|
12470
|
+
return false;
|
|
12471
|
+
}
|
|
12472
|
+
function getLastAssistantError(messages) {
|
|
12473
|
+
for (let i = messages.length - 1;i >= 0; i--) {
|
|
12474
|
+
const msg = messages[i];
|
|
12475
|
+
const role = msg.role ?? msg.info?.role;
|
|
12476
|
+
if (role !== "assistant")
|
|
12477
|
+
continue;
|
|
12478
|
+
const info = msg.info;
|
|
12479
|
+
const err = msg.error ?? info?.error;
|
|
12480
|
+
if (err) {
|
|
12481
|
+
const data = err.data;
|
|
12482
|
+
const name = err.name ?? "";
|
|
12483
|
+
const message = data?.message ?? err.message ?? "";
|
|
12484
|
+
return { name, message };
|
|
12485
|
+
}
|
|
12486
|
+
const parts = msg.parts;
|
|
12487
|
+
if (parts) {
|
|
12488
|
+
for (let j = parts.length - 1;j >= 0; j--) {
|
|
12489
|
+
const part = parts[j];
|
|
12490
|
+
if (part.type !== "retry")
|
|
12491
|
+
continue;
|
|
12492
|
+
const partErr = part.error;
|
|
12493
|
+
if (!partErr)
|
|
12494
|
+
continue;
|
|
12495
|
+
const data = partErr.data;
|
|
12496
|
+
const name = partErr.name ?? "";
|
|
12497
|
+
const message = data?.message ?? partErr.message ?? "";
|
|
12498
|
+
return { name, message };
|
|
12499
|
+
}
|
|
12500
|
+
}
|
|
12501
|
+
}
|
|
12502
|
+
return null;
|
|
12503
|
+
}
|
|
12504
|
+
function backoffMs(attempt, baseBackoffMs = DEFAULT_BASE_BACKOFF_MS, maxBackoffMs = DEFAULT_MAX_BACKOFF_MS) {
|
|
12505
|
+
return Math.min(baseBackoffMs * Math.pow(2, attempt - 1), maxBackoffMs);
|
|
12506
|
+
}
|
|
12507
|
+
function containsActionIntent(text) {
|
|
12508
|
+
if (text.length <= 15)
|
|
12509
|
+
return false;
|
|
12510
|
+
const cleaned = text.replace(/<[a-zA-Z/?][^>]*>/g, "").trim();
|
|
12511
|
+
const lines = cleaned.split(`
|
|
12512
|
+
`);
|
|
12513
|
+
let lastLine = "";
|
|
12514
|
+
for (let i = lines.length - 1;i >= 0; i--) {
|
|
12515
|
+
if (lines[i].trim().length > 0) {
|
|
12516
|
+
lastLine = lines[i].trim();
|
|
12517
|
+
break;
|
|
12518
|
+
}
|
|
12519
|
+
}
|
|
12520
|
+
return lastLine.endsWith(":") && lastLine.length > 5 && lastLine.length < 500;
|
|
12521
|
+
}
|
|
12522
|
+
function isOpenTodo(t) {
|
|
12523
|
+
return t.status === "pending" || t.status === "in_progress";
|
|
12524
|
+
}
|
|
12525
|
+
function getOpenTodos(todos) {
|
|
12526
|
+
return todos.filter(isOpenTodo);
|
|
12527
|
+
}
|
|
12432
12528
|
function buildOpenTodosReminder(todos) {
|
|
12529
|
+
if (!Array.isArray(todos))
|
|
12530
|
+
return "continue";
|
|
12433
12531
|
const open = todos.filter((t) => t.status === "pending" || t.status === "in_progress");
|
|
12434
12532
|
if (open.length === 0)
|
|
12435
12533
|
return "continue";
|
|
12436
12534
|
const list = open.map((t, i) => `${i + 1}. [${t.status}] ${t.content}`).join(`
|
|
12437
12535
|
`);
|
|
12438
12536
|
const plural = open.length > 1 ? "s" : "";
|
|
12537
|
+
const taskWord = open.length > 1 ? "tasks" : "task";
|
|
12538
|
+
const thisWord = open.length > 1 ? "these" : "this";
|
|
12439
12539
|
return `You have ${open.length} unfinished task${plural}:
|
|
12440
12540
|
${list}
|
|
12441
12541
|
|
|
12442
|
-
Please continue working on
|
|
12542
|
+
Please continue working on ${thisWord} ${taskWord}.`;
|
|
12443
12543
|
}
|
|
12444
12544
|
var AutoResumePlugin = async (ctx, options) => {
|
|
12445
12545
|
const chunkTimeoutMs = options?.chunkTimeoutMs ?? DEFAULT_CHUNK_TIMEOUT_MS;
|
|
@@ -12451,6 +12551,23 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12451
12551
|
const subagentWaitMs = options?.subagentWaitMs ?? DEFAULT_SUBAGENT_WAIT_MS;
|
|
12452
12552
|
const loopMaxContinues = options?.loopMaxContinues ?? DEFAULT_LOOP_MAX_CONTINUES;
|
|
12453
12553
|
const loopWindowMs = options?.loopWindowMs ?? DEFAULT_LOOP_WINDOW_MS;
|
|
12554
|
+
const toolTextCheckDelayMs = options?.toolTextCheckDelayMs ?? DEFAULT_TOOL_TEXT_CHECK_DELAY_MS;
|
|
12555
|
+
const maxRecoveryRetries = options?.maxRecoveryRetries ?? DEFAULT_MAX_RECOVERY_RETRIES;
|
|
12556
|
+
const minActivityGapMs = options?.minActivityGapMs ?? DEFAULT_MIN_ACTIVITY_GAP_MS;
|
|
12557
|
+
const warmupMs = options?.warmupMs ?? DEFAULT_WARMUP_MS;
|
|
12558
|
+
const debug = options?.debug ?? DEFAULT_DEBUG;
|
|
12559
|
+
const streamingFailureErrorNames = options?.streamingFailureErrorNames ?? DEFAULT_STREAMING_FAILURE_ERROR_NAMES;
|
|
12560
|
+
const streamingFailureMessagePatterns = options?.streamingFailureMessagePatterns ?? DEFAULT_STREAMING_FAILURE_MESSAGE_PATTERNS;
|
|
12561
|
+
const resumeOnActionIntent = options?.resumeOnActionIntent !== false;
|
|
12562
|
+
const continuePrompt = options?.continuePrompt ?? "continue";
|
|
12563
|
+
const actionIntentPrompt = options?.actionIntentPrompt ?? continuePrompt;
|
|
12564
|
+
const toolTextRecoveryPrompt = options?.toolTextRecoveryPrompt ?? TOOL_TEXT_RECOVERY_PROMPT;
|
|
12565
|
+
const thinkingToolRecoveryPrompt = options?.thinkingToolRecoveryPrompt ?? THINKING_TOOL_RECOVERY_PROMPT;
|
|
12566
|
+
const doneWithoutWorkPrompt = options?.doneWithoutWorkPrompt ?? DONE_WITHOUT_WORK_PROMPT;
|
|
12567
|
+
const dbg = (...args) => {
|
|
12568
|
+
if (debug)
|
|
12569
|
+
console.log("[debug]", ...args);
|
|
12570
|
+
};
|
|
12454
12571
|
const sessions = new Map;
|
|
12455
12572
|
let timer = null;
|
|
12456
12573
|
let discoveryTimer = null;
|
|
@@ -12476,12 +12593,27 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12476
12593
|
async function log(level, msg) {
|
|
12477
12594
|
try {
|
|
12478
12595
|
await ctx.client.app.log({ body: { service: "auto-resume", level, message: msg } });
|
|
12479
|
-
} catch {
|
|
12596
|
+
} catch (e) {
|
|
12597
|
+
console.error("[auto-resume] log() failed:", e instanceof Error ? e.message : String(e));
|
|
12598
|
+
}
|
|
12599
|
+
}
|
|
12600
|
+
async function safe(fn, ctxLabel) {
|
|
12601
|
+
try {
|
|
12602
|
+
return await fn();
|
|
12603
|
+
} catch (e) {
|
|
12604
|
+
const msg = e instanceof Error ? e.message : String(e);
|
|
12605
|
+
console.error(`[auto-resume] ${ctxLabel}: ${msg}`);
|
|
12606
|
+
try {
|
|
12607
|
+
await log("error", `${ctxLabel}: ${msg}`);
|
|
12608
|
+
} catch {}
|
|
12609
|
+
return;
|
|
12610
|
+
}
|
|
12480
12611
|
}
|
|
12481
12612
|
function ensureWatch(sid) {
|
|
12482
12613
|
let w = sessions.get(sid);
|
|
12483
12614
|
if (!w) {
|
|
12484
12615
|
w = {
|
|
12616
|
+
createdAt: Date.now(),
|
|
12485
12617
|
lastActivityAt: Date.now(),
|
|
12486
12618
|
status: "unknown",
|
|
12487
12619
|
userCancelled: false,
|
|
@@ -12506,8 +12638,15 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12506
12638
|
isSubagent: false,
|
|
12507
12639
|
completionSignaled: false,
|
|
12508
12640
|
todoNudgeAttempts: 0,
|
|
12641
|
+
taskCompleteOverrides: 0,
|
|
12642
|
+
doneClaimNoTodosAttempts: 0,
|
|
12509
12643
|
pendingTools: 0,
|
|
12510
|
-
pendingCommands: 0
|
|
12644
|
+
pendingCommands: 0,
|
|
12645
|
+
pendingRecovery: false,
|
|
12646
|
+
pendingRecoveryReason: null,
|
|
12647
|
+
pendingRecoveryAt: 0,
|
|
12648
|
+
recoveryAttempts: 0,
|
|
12649
|
+
watchdogRetryGuard: false
|
|
12511
12650
|
};
|
|
12512
12651
|
sessions.set(sid, w);
|
|
12513
12652
|
}
|
|
@@ -12568,9 +12707,6 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12568
12707
|
function short(sid) {
|
|
12569
12708
|
return sid.length > 12 ? `...${sid.slice(-8)}` : sid;
|
|
12570
12709
|
}
|
|
12571
|
-
function backoffMs(attempt) {
|
|
12572
|
-
return Math.min(baseBackoffMs * Math.pow(2, attempt - 1), maxBackoffMs);
|
|
12573
|
-
}
|
|
12574
12710
|
function cleanupIdleSessions() {
|
|
12575
12711
|
const now = Date.now();
|
|
12576
12712
|
const toDelete = [];
|
|
@@ -12605,12 +12741,52 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12605
12741
|
log("debug", `Cleaned up ${toDelete.length} idle session(s). Map size: ${sessions.size}`);
|
|
12606
12742
|
}
|
|
12607
12743
|
}
|
|
12744
|
+
function getPromptResponsePayload(result) {
|
|
12745
|
+
if (!result || typeof result !== "object")
|
|
12746
|
+
return null;
|
|
12747
|
+
const raw = result;
|
|
12748
|
+
const data = raw.data;
|
|
12749
|
+
if (data && typeof data === "object" && "parts" in data) {
|
|
12750
|
+
return data;
|
|
12751
|
+
}
|
|
12752
|
+
if ("parts" in raw) {
|
|
12753
|
+
return raw;
|
|
12754
|
+
}
|
|
12755
|
+
return null;
|
|
12756
|
+
}
|
|
12757
|
+
async function logPromptResponse(result, context) {
|
|
12758
|
+
const payload = getPromptResponsePayload(result);
|
|
12759
|
+
const isRetry = context.isRetry ?? false;
|
|
12760
|
+
if (!payload) {
|
|
12761
|
+
await log("debug", `${short(context.sessionId)} - session.prompt() response has unexpected structure: ${JSON.stringify({ sessionId: context.sessionId, isRetry, rawResponse: JSON.stringify(result) })}`);
|
|
12762
|
+
return;
|
|
12763
|
+
}
|
|
12764
|
+
const parts = Array.isArray(payload.parts) ? payload.parts : [];
|
|
12765
|
+
const partsCount = parts.length;
|
|
12766
|
+
const info = payload.info && typeof payload.info === "object" ? payload.info : undefined;
|
|
12767
|
+
const infoKeys = info ? Object.keys(info) : [];
|
|
12768
|
+
const hasParts = partsCount > 0;
|
|
12769
|
+
await log("debug", `${short(context.sessionId)} - session.prompt() response received: ${JSON.stringify({ sessionId: context.sessionId, isRetry, hasParts, partsCount, infoKeys, info })}`);
|
|
12770
|
+
if (partsCount === 0) {
|
|
12771
|
+
await log("warn", `${short(context.sessionId)} - session.prompt() returned empty parts array - possible stream initiation failure: ${JSON.stringify({ sessionId: context.sessionId, isRetry, responseInfo: info, partsCount })}`);
|
|
12772
|
+
}
|
|
12773
|
+
if (info && "error" in info) {
|
|
12774
|
+
await log("warn", `${short(context.sessionId)} - session.prompt() response.info contains error indicator: ${JSON.stringify({ sessionId: context.sessionId, isRetry, errorInfo: info.error })}`);
|
|
12775
|
+
}
|
|
12776
|
+
await log("debug", `${short(context.sessionId)} - session.prompt() raw response: ${JSON.stringify({ sessionId: context.sessionId, isRetry, rawResponse: JSON.stringify(result, null, 2) })}`);
|
|
12777
|
+
}
|
|
12608
12778
|
async function sendContinuePrompt(sid, text, w) {
|
|
12609
|
-
if (w.continuing) {
|
|
12779
|
+
if (w.continuing && !w.watchdogRetryGuard) {
|
|
12610
12780
|
await log("debug", `${short(sid)} - continue already in progress, skipping`);
|
|
12611
12781
|
return;
|
|
12612
12782
|
}
|
|
12783
|
+
if (!w.continuing)
|
|
12784
|
+
dbg(`State transition on ${short(sid)}: continuing=false -> true`);
|
|
12613
12785
|
w.continuing = true;
|
|
12786
|
+
if (w.watchdogRetryGuard) {
|
|
12787
|
+
w.pendingRecovery = true;
|
|
12788
|
+
}
|
|
12789
|
+
w.watchdogRetryGuard = false;
|
|
12614
12790
|
let agent;
|
|
12615
12791
|
let model;
|
|
12616
12792
|
try {
|
|
@@ -12641,7 +12817,8 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12641
12817
|
break;
|
|
12642
12818
|
}
|
|
12643
12819
|
}
|
|
12644
|
-
|
|
12820
|
+
dbg(`Recovery prompt sent to ${short(sid)}: prompt="${text.length > 80 ? `${text.slice(0, 80)}...` : text}", agent=${agent ?? "(default)"}, model=${model ? `${model.providerID}/${model.modelID}` : "(default)"}`);
|
|
12821
|
+
const response = await ctx.client.session.prompt({
|
|
12645
12822
|
path: { id: sid },
|
|
12646
12823
|
body: {
|
|
12647
12824
|
parts: [{ type: "text", text }],
|
|
@@ -12649,6 +12826,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12649
12826
|
model
|
|
12650
12827
|
}
|
|
12651
12828
|
});
|
|
12829
|
+
await logPromptResponse(response, { sessionId: sid });
|
|
12652
12830
|
await log("debug", `${short(sid)} - prompt sent with agent: ${agent ?? "(default)"}, model: ${model ? `${model.providerID}/${model.modelID}` : "(default)"}`);
|
|
12653
12831
|
recordContinue(sid);
|
|
12654
12832
|
w.lastRetryAt = Date.now();
|
|
@@ -12656,10 +12834,11 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12656
12834
|
const errMsg = err instanceof Error ? err.message : String(err);
|
|
12657
12835
|
await log("warn", `${short(sid)} - prompt failed: ${errMsg}`);
|
|
12658
12836
|
try {
|
|
12659
|
-
await ctx.client.session.prompt({
|
|
12837
|
+
const retryResponse = await ctx.client.session.prompt({
|
|
12660
12838
|
path: { id: sid },
|
|
12661
12839
|
body: { parts: [{ type: "text", text }], agent, model }
|
|
12662
12840
|
});
|
|
12841
|
+
await logPromptResponse(retryResponse, { sessionId: sid, isRetry: true });
|
|
12663
12842
|
recordContinue(sid);
|
|
12664
12843
|
w.lastRetryAt = Date.now();
|
|
12665
12844
|
} catch (retryErr) {
|
|
@@ -12668,6 +12847,8 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12668
12847
|
throw retryErr;
|
|
12669
12848
|
}
|
|
12670
12849
|
} finally {
|
|
12850
|
+
if (w.continuing)
|
|
12851
|
+
dbg(`State transition on ${short(sid)}: continuing=true -> false`);
|
|
12671
12852
|
w.continuing = false;
|
|
12672
12853
|
w.todoCheckAttempts = 0;
|
|
12673
12854
|
if (w.toolTextTimer) {
|
|
@@ -12677,9 +12858,53 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12677
12858
|
}
|
|
12678
12859
|
setTimeout(async () => {
|
|
12679
12860
|
if (w.status !== "busy") {
|
|
12680
|
-
|
|
12861
|
+
if (w.pendingRecovery) {
|
|
12862
|
+
if (w.recoveryAttempts < maxRecoveryRetries) {
|
|
12863
|
+
w.recoveryAttempts++;
|
|
12864
|
+
dbg(`State transition on ${short(sid)}: recoveryAttempts=${w.recoveryAttempts - 1} -> ${w.recoveryAttempts}`);
|
|
12865
|
+
w.watchdogRetryGuard = true;
|
|
12866
|
+
await log("warn", `${short(sid)} - recovery attempt ${w.recoveryAttempts}/${maxRecoveryRetries} after prompt timeout`);
|
|
12867
|
+
await log("warn", `Recovery failed on ${short(sid)} - session still ${w.status}: attempt=${w.recoveryAttempts}, maxRetries=${maxRecoveryRetries}, nextAction=retry`);
|
|
12868
|
+
dbg(`Watchdog check on ${short(sid)}: status=${w.status}, recoveryAttempts=${w.recoveryAttempts}, maxRetries=${maxRecoveryRetries}, watchdogLatencyMs=${Date.now() - w.lastRetryAt} -> RETRY`);
|
|
12869
|
+
const retryBackoffMs = backoffMs(w.recoveryAttempts, baseBackoffMs, maxBackoffMs);
|
|
12870
|
+
await log("info", `Retrying recovery on ${short(sid)}: attempt=${w.recoveryAttempts}, backoffMs=${retryBackoffMs}`);
|
|
12871
|
+
dbg(`Retrying recovery on ${short(sid)}: recoveryAttempts=${w.recoveryAttempts}, backoffMs=${retryBackoffMs}, pendingRecoveryReason=${w.pendingRecoveryReason}`);
|
|
12872
|
+
try {
|
|
12873
|
+
await sendContinuePrompt(sid, continuePrompt, w);
|
|
12874
|
+
} catch (err) {
|
|
12875
|
+
const errMsg = err instanceof Error ? err.message : String(err);
|
|
12876
|
+
await log("warn", `${short(sid)} - recovery retry failed: ${errMsg}`);
|
|
12877
|
+
w.recoveryAttempts = 0;
|
|
12878
|
+
}
|
|
12879
|
+
w.watchdogRetryGuard = false;
|
|
12880
|
+
} else {
|
|
12881
|
+
dbg(`Pending recovery cleared on ${short(sid)}: reason=recovery-attempt`);
|
|
12882
|
+
w.pendingRecovery = false;
|
|
12883
|
+
await log("warn", `${short(sid)} - max recovery attempts (${maxRecoveryRetries}) reached, escalating to abort+resume`);
|
|
12884
|
+
await log("warn", `Recovery failed on ${short(sid)} - session still ${w.status}: attempt=${w.recoveryAttempts}, maxRetries=${maxRecoveryRetries}, nextAction=abort-resume`);
|
|
12885
|
+
await log("warn", `Escalating to abort+resume on ${short(sid)}: attempt=${w.recoveryAttempts}`);
|
|
12886
|
+
dbg(`Watchdog check on ${short(sid)}: status=${w.status}, recoveryAttempts=${w.recoveryAttempts}, maxRetries=${maxRecoveryRetries}, watchdogLatencyMs=${Date.now() - w.lastRetryAt} -> ABORT_RESUME`);
|
|
12887
|
+
const resumed = await tryAbortAndResume(sid, w);
|
|
12888
|
+
if (!resumed && !w.aborting) {
|
|
12889
|
+
await log("warn", `Recovery exhausted on ${short(sid)}: attempts=${w.recoveryAttempts}, lastError=abort+resume failed`);
|
|
12890
|
+
dbg(`Watchdog check on ${short(sid)}: status=${w.status} -> GAVE_UP`);
|
|
12891
|
+
if (w.pendingRecoveryAt > 0) {
|
|
12892
|
+
dbg(`Total recovery cycle on ${short(sid)} (failed): totalCycleMs=${Date.now() - w.pendingRecoveryAt}`);
|
|
12893
|
+
}
|
|
12894
|
+
}
|
|
12895
|
+
}
|
|
12896
|
+
} else {
|
|
12897
|
+
await log("warn", `${short(sid)} - prompt sent >${toolTextCheckDelayMs / 1000}s ago but session is still ${w.status}`);
|
|
12898
|
+
}
|
|
12899
|
+
} else {
|
|
12900
|
+
const elapsedMs = Date.now() - w.lastRetryAt;
|
|
12901
|
+
await log("info", `Recovery successful on ${short(sid)}: elapsedMs=${elapsedMs}`);
|
|
12902
|
+
dbg(`Watchdog check on ${short(sid)}: status=busy, elapsedMs=${elapsedMs} -> SUCCESS`);
|
|
12903
|
+
if (w.pendingRecoveryAt > 0) {
|
|
12904
|
+
dbg(`Total recovery cycle on ${short(sid)}: totalCycleMs=${Date.now() - w.pendingRecoveryAt}`);
|
|
12905
|
+
}
|
|
12681
12906
|
}
|
|
12682
|
-
},
|
|
12907
|
+
}, toolTextCheckDelayMs);
|
|
12683
12908
|
}
|
|
12684
12909
|
function extractMessages(response) {
|
|
12685
12910
|
if (Array.isArray(response))
|
|
@@ -12851,7 +13076,6 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12851
13076
|
w.toolTextRecovered = false;
|
|
12852
13077
|
w.toolTextAttempts = 0;
|
|
12853
13078
|
w.completionSignaled = false;
|
|
12854
|
-
w.todoNudgeAttempts = 0;
|
|
12855
13079
|
w.continueTimestamps = [];
|
|
12856
13080
|
w.idleSince = null;
|
|
12857
13081
|
w.continuing = false;
|
|
@@ -12864,6 +13088,11 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12864
13088
|
clearTimeout(w.toolTextTimer);
|
|
12865
13089
|
w.toolTextTimer = null;
|
|
12866
13090
|
}
|
|
13091
|
+
w.pendingRecovery = false;
|
|
13092
|
+
w.pendingRecoveryReason = null;
|
|
13093
|
+
w.pendingRecoveryAt = 0;
|
|
13094
|
+
w.recoveryAttempts = 0;
|
|
13095
|
+
w.watchdogRetryGuard = false;
|
|
12867
13096
|
}
|
|
12868
13097
|
function resetIdleFlags(w) {
|
|
12869
13098
|
w.aborting = false;
|
|
@@ -12917,9 +13146,10 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12917
13146
|
if (w.checkingToolText)
|
|
12918
13147
|
return;
|
|
12919
13148
|
w.checkingToolText = true;
|
|
13149
|
+
dbg(`checkForToolCallAsText called for ${short(sid)}, userCancelled=${w.userCancelled}, toolTextRecovered=${w.toolTextRecovered}, toolTextAttempts=${w.toolTextAttempts}`);
|
|
12920
13150
|
if (w.toolTextAttempts > 0) {
|
|
12921
13151
|
const elapsed = Date.now() - w.lastRetryAt;
|
|
12922
|
-
const requiredBackoff = backoffMs(w.toolTextAttempts);
|
|
13152
|
+
const requiredBackoff = backoffMs(w.toolTextAttempts, baseBackoffMs, maxBackoffMs);
|
|
12923
13153
|
if (elapsed < requiredBackoff)
|
|
12924
13154
|
return;
|
|
12925
13155
|
}
|
|
@@ -12990,7 +13220,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
12990
13220
|
}
|
|
12991
13221
|
} else {
|
|
12992
13222
|
const candidate = {
|
|
12993
|
-
prompt:
|
|
13223
|
+
prompt: continuePrompt,
|
|
12994
13224
|
source: "tool-use",
|
|
12995
13225
|
priority: 1
|
|
12996
13226
|
};
|
|
@@ -13017,7 +13247,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13017
13247
|
}
|
|
13018
13248
|
}
|
|
13019
13249
|
const candidate = {
|
|
13020
|
-
prompt: isReasoning ?
|
|
13250
|
+
prompt: isReasoning ? thinkingToolRecoveryPrompt : toolTextRecoveryPrompt,
|
|
13021
13251
|
source: isReasoning ? "reasoning" : "text",
|
|
13022
13252
|
priority: 0
|
|
13023
13253
|
};
|
|
@@ -13027,13 +13257,13 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13027
13257
|
}
|
|
13028
13258
|
if (containsReadyToContinuePattern(text)) {
|
|
13029
13259
|
const todos = w.todos || [];
|
|
13030
|
-
const hasOpenTodos = todos.some(
|
|
13260
|
+
const hasOpenTodos = todos.some(isOpenTodo);
|
|
13031
13261
|
if (!hasOpenTodos && todos.length > 0) {
|
|
13032
13262
|
w.todoCheckAttempts++;
|
|
13033
13263
|
if (w.todoCheckAttempts >= 2) {
|
|
13034
13264
|
await log("info", `${short(sid)} - todos completed but agent hasn't closed them. Sending continue...`);
|
|
13035
13265
|
const candidate2 = {
|
|
13036
|
-
prompt:
|
|
13266
|
+
prompt: continuePrompt,
|
|
13037
13267
|
source: "todo-completed-continue",
|
|
13038
13268
|
priority: 1
|
|
13039
13269
|
};
|
|
@@ -13046,7 +13276,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13046
13276
|
continue;
|
|
13047
13277
|
}
|
|
13048
13278
|
const candidate = {
|
|
13049
|
-
prompt: containsDoneClaimPattern(text) ?
|
|
13279
|
+
prompt: containsDoneClaimPattern(text) ? doneWithoutWorkPrompt : continuePrompt,
|
|
13050
13280
|
source: containsDoneClaimPattern(text) ? "done-claim" : "ready-to-continue",
|
|
13051
13281
|
priority: 1
|
|
13052
13282
|
};
|
|
@@ -13056,14 +13286,42 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13056
13286
|
}
|
|
13057
13287
|
if (!bestCandidate && containsDoneClaimPattern(text)) {
|
|
13058
13288
|
const todos = w.todos || [];
|
|
13059
|
-
const hasOpenTodos = todos.some(
|
|
13289
|
+
const hasOpenTodos = todos.some(isOpenTodo);
|
|
13060
13290
|
if (hasOpenTodos) {
|
|
13061
13291
|
await log("info", `${short(sid)} - model claims done but todos remain open. Sending recovery prompt...`);
|
|
13062
13292
|
bestCandidate = {
|
|
13063
|
-
prompt:
|
|
13293
|
+
prompt: doneWithoutWorkPrompt,
|
|
13064
13294
|
source: "done-claim-no-emoji",
|
|
13065
13295
|
priority: 1
|
|
13066
13296
|
};
|
|
13297
|
+
} else if (w.doneClaimNoTodosAttempts < maxRetries) {
|
|
13298
|
+
await log("info", `${short(sid)} - model claims done with no open todos. Sending verification prompt (attempt ${w.doneClaimNoTodosAttempts + 1}/${maxRetries})...`);
|
|
13299
|
+
bestCandidate = {
|
|
13300
|
+
prompt: doneWithoutWorkPrompt,
|
|
13301
|
+
source: "done-claim-no-todos",
|
|
13302
|
+
priority: 1
|
|
13303
|
+
};
|
|
13304
|
+
}
|
|
13305
|
+
}
|
|
13306
|
+
}
|
|
13307
|
+
}
|
|
13308
|
+
if (resumeOnActionIntent) {
|
|
13309
|
+
const lastAssistantMsg = messages.slice().reverse().find((m) => (m.role ?? m.info?.role) === "assistant");
|
|
13310
|
+
if (lastAssistantMsg) {
|
|
13311
|
+
let lastAssistantText = "";
|
|
13312
|
+
for (const part of lastAssistantMsg.parts || []) {
|
|
13313
|
+
lastAssistantText += (part.text ?? "") + `
|
|
13314
|
+
`;
|
|
13315
|
+
}
|
|
13316
|
+
if (containsActionIntent(lastAssistantText)) {
|
|
13317
|
+
dbg(`ACTION INTENT DETECTED in checkForToolCallAsText for ${short(sid)}`);
|
|
13318
|
+
const candidate = {
|
|
13319
|
+
prompt: actionIntentPrompt,
|
|
13320
|
+
source: "action-intent",
|
|
13321
|
+
priority: 2
|
|
13322
|
+
};
|
|
13323
|
+
if (!bestCandidate || candidate.priority < bestCandidate.priority) {
|
|
13324
|
+
bestCandidate = candidate;
|
|
13067
13325
|
}
|
|
13068
13326
|
}
|
|
13069
13327
|
}
|
|
@@ -13082,10 +13340,10 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13082
13340
|
}
|
|
13083
13341
|
if (!bestCandidate) {
|
|
13084
13342
|
const todos = w.todos || [];
|
|
13085
|
-
const hasOpenTodos = todos.some(
|
|
13343
|
+
const hasOpenTodos = todos.some(isOpenTodo);
|
|
13086
13344
|
if (hasOpenTodos && busyCount() === 0) {
|
|
13087
13345
|
const reminder = buildOpenTodosReminder(todos);
|
|
13088
|
-
await log("info", `${short(sid)} - no activity detected but todos remain open (${todos
|
|
13346
|
+
await log("info", `${short(sid)} - no activity detected but todos remain open (${getOpenTodos(todos).length} tasks). Sending reminder...`);
|
|
13089
13347
|
bestCandidate = {
|
|
13090
13348
|
prompt: reminder,
|
|
13091
13349
|
source: "idle-with-open-todos-reminder",
|
|
@@ -13098,19 +13356,22 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13098
13356
|
if (!bestCandidate)
|
|
13099
13357
|
return;
|
|
13100
13358
|
const isOpenTodosReminder = bestCandidate.source === "idle-with-open-todos-reminder";
|
|
13359
|
+
const isDoneClaimNoTodos = bestCandidate.source === "done-claim-no-todos";
|
|
13101
13360
|
if (isOpenTodosReminder) {
|
|
13102
13361
|
if (w.todoNudgeAttempts >= maxRetries) {
|
|
13103
|
-
await log("info", `${short(sid)} - max open-todos nudges (${maxRetries}) reached
|
|
13362
|
+
await log("info", `${short(sid)} - max open-todos nudges (${maxRetries}) reached, waiting for activity`);
|
|
13104
13363
|
return;
|
|
13105
13364
|
}
|
|
13106
|
-
|
|
13365
|
+
} else if (isDoneClaimNoTodos) {
|
|
13366
|
+
w.doneClaimNoTodosAttempts++;
|
|
13107
13367
|
} else {
|
|
13108
13368
|
w.toolTextRecovered = true;
|
|
13109
13369
|
w.toolTextAttempts++;
|
|
13110
13370
|
}
|
|
13111
|
-
|
|
13371
|
+
const attemptNum = isOpenTodosReminder ? w.todoNudgeAttempts : isDoneClaimNoTodos ? w.doneClaimNoTodosAttempts : w.toolTextAttempts;
|
|
13372
|
+
await log("info", `${bestCandidate.source} detected on ${short(sid)}! ` + `Attempt ${attemptNum}/${maxRetries}. Sending recovery prompt...`);
|
|
13112
13373
|
const timeSinceActivity = Date.now() - w.lastActivityAt;
|
|
13113
|
-
if (timeSinceActivity <
|
|
13374
|
+
if (timeSinceActivity < minActivityGapMs) {
|
|
13114
13375
|
await log("info", `${short(sid)} - skipping ${bestCandidate.source}, session was active ${Math.round(timeSinceActivity / 1000)}s ago`);
|
|
13115
13376
|
return;
|
|
13116
13377
|
}
|
|
@@ -13129,6 +13390,8 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13129
13390
|
} else {
|
|
13130
13391
|
try {
|
|
13131
13392
|
await sendContinuePrompt(sid, bestCandidate.prompt, w);
|
|
13393
|
+
if (isOpenTodosReminder)
|
|
13394
|
+
w.todoNudgeAttempts++;
|
|
13132
13395
|
await log("info", `${short(sid)} - ${bestCandidate.source} recovery sent (attempt ${w.toolTextAttempts})`);
|
|
13133
13396
|
} catch (err) {
|
|
13134
13397
|
const errMsg = err instanceof Error ? err.message : String(err);
|
|
@@ -13155,6 +13418,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13155
13418
|
try {
|
|
13156
13419
|
await ctx.client.session.abort({ path: { id: sid } });
|
|
13157
13420
|
await log("info", `${short(sid)} - abort OK`);
|
|
13421
|
+
dbg(`Abort succeeded on ${short(sid)}, waiting ${ABORT_CONTINUE_DELAY_MS}ms before continue prompt`);
|
|
13158
13422
|
} catch (err) {
|
|
13159
13423
|
const errMsg = err instanceof Error ? err.message : String(err);
|
|
13160
13424
|
await log("warn", `${short(sid)} - abort failed: ${errMsg}`);
|
|
@@ -13165,7 +13429,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13165
13429
|
if (w.status === "busy")
|
|
13166
13430
|
w.status = "idle";
|
|
13167
13431
|
try {
|
|
13168
|
-
await sendContinuePrompt(sid,
|
|
13432
|
+
await sendContinuePrompt(sid, continuePrompt, w);
|
|
13169
13433
|
await log("info", `${short(sid)} - abort+continue done`);
|
|
13170
13434
|
w.orphanWatchStartAt = null;
|
|
13171
13435
|
w.resumeAttempts++;
|
|
@@ -13185,7 +13449,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13185
13449
|
}
|
|
13186
13450
|
const now = Date.now();
|
|
13187
13451
|
const elapsedSinceRetry = now - w.lastRetryAt;
|
|
13188
|
-
const requiredBackoff = backoffMs(w.resumeAttempts);
|
|
13452
|
+
const requiredBackoff = backoffMs(w.resumeAttempts, baseBackoffMs, maxBackoffMs);
|
|
13189
13453
|
if (w.lastRetryAt > 0 && elapsedSinceRetry < requiredBackoff)
|
|
13190
13454
|
return false;
|
|
13191
13455
|
if (isHallucinationLoop(sid)) {
|
|
@@ -13207,7 +13471,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13207
13471
|
const idleSec = Math.round((now - w.lastActivityAt) / 1000);
|
|
13208
13472
|
await log("info", `${reason} on ${short(sid)} (${idleSec}s, retry ${w.resumeAttempts}/${maxRetries})`);
|
|
13209
13473
|
try {
|
|
13210
|
-
await sendContinuePrompt(sid, prompt ??
|
|
13474
|
+
await sendContinuePrompt(sid, prompt ?? continuePrompt, w);
|
|
13211
13475
|
await log("info", `${short(sid)} - retry sent`);
|
|
13212
13476
|
return true;
|
|
13213
13477
|
} catch (err) {
|
|
@@ -13247,129 +13511,188 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13247
13511
|
if (timer)
|
|
13248
13512
|
return;
|
|
13249
13513
|
timer = setInterval(async () => {
|
|
13250
|
-
|
|
13251
|
-
|
|
13252
|
-
|
|
13253
|
-
|
|
13254
|
-
const
|
|
13255
|
-
|
|
13256
|
-
w.status
|
|
13257
|
-
|
|
13258
|
-
|
|
13259
|
-
|
|
13260
|
-
|
|
13261
|
-
|
|
13262
|
-
|
|
13263
|
-
|
|
13264
|
-
|
|
13265
|
-
|
|
13266
|
-
|
|
13267
|
-
|
|
13268
|
-
|
|
13269
|
-
if (
|
|
13270
|
-
if (
|
|
13271
|
-
|
|
13272
|
-
|
|
13273
|
-
|
|
13274
|
-
|
|
13275
|
-
|
|
13276
|
-
|
|
13277
|
-
|
|
13278
|
-
|
|
13279
|
-
|
|
13280
|
-
|
|
13281
|
-
const subStatus = await checkSubagentStatus(sid);
|
|
13282
|
-
if (subStatus.status === "crashed" && subStatus.stuckSid) {
|
|
13283
|
-
const recovered = await recoverSubagent(subStatus.stuckSid);
|
|
13284
|
-
if (recovered) {
|
|
13285
|
-
await log("info", `Sent recovery prompt to stuck subagent ${short(subStatus.stuckSid)}, waiting...`);
|
|
13286
|
-
} else {
|
|
13287
|
-
await log("info", `Subagent crashed, triggering abort+resume on ${short(sid)}`);
|
|
13288
|
-
tryAbortAndResume(sid, w);
|
|
13514
|
+
await safe(async () => {
|
|
13515
|
+
const now = Date.now();
|
|
13516
|
+
const numBusy = busyCount();
|
|
13517
|
+
const statusMap = await getSessionStatusMap();
|
|
13518
|
+
for (const [sid, w] of sessions) {
|
|
13519
|
+
const realStatus = statusMap[sid];
|
|
13520
|
+
if (realStatus && realStatus !== w.status) {
|
|
13521
|
+
w.status = realStatus;
|
|
13522
|
+
if (realStatus === "busy")
|
|
13523
|
+
w.idleSince = null;
|
|
13524
|
+
}
|
|
13525
|
+
if (w.status !== "busy")
|
|
13526
|
+
continue;
|
|
13527
|
+
if (w.userCancelled)
|
|
13528
|
+
continue;
|
|
13529
|
+
if (w.aborting)
|
|
13530
|
+
continue;
|
|
13531
|
+
if (w.orphanWatchStartAt !== null) {
|
|
13532
|
+
const orphanIdle = now - w.orphanWatchStartAt;
|
|
13533
|
+
if (orphanIdle >= subagentWaitMs + gracePeriodMs) {
|
|
13534
|
+
if (w.resumeAttempts < maxRetries) {
|
|
13535
|
+
if (hasInflightTools(w)) {
|
|
13536
|
+
await log("debug", `Parent ${short(sid)} has ${w.pendingTools} tool(s) in-flight, skipping orphan-watch abort`);
|
|
13537
|
+
w.orphanWatchStartAt = now;
|
|
13538
|
+
continue;
|
|
13539
|
+
}
|
|
13540
|
+
const hasActiveTool = await checkSessionHasActiveTool(sid);
|
|
13541
|
+
if (hasActiveTool) {
|
|
13542
|
+
await log("debug", `Parent ${short(sid)} has active tool call, skipping orphan-watch abort`);
|
|
13543
|
+
w.orphanWatchStartAt = now;
|
|
13544
|
+
continue;
|
|
13289
13545
|
}
|
|
13290
|
-
|
|
13291
|
-
|
|
13292
|
-
|
|
13293
|
-
|
|
13546
|
+
const subStatus = await checkSubagentStatus(sid);
|
|
13547
|
+
if (subStatus.status === "crashed" && subStatus.stuckSid) {
|
|
13548
|
+
const recovered = await recoverSubagent(subStatus.stuckSid);
|
|
13549
|
+
if (recovered) {
|
|
13550
|
+
await log("info", `Sent recovery prompt to stuck subagent ${short(subStatus.stuckSid)}, waiting...`);
|
|
13551
|
+
} else {
|
|
13552
|
+
await log("info", `Subagent crashed, triggering abort+resume on ${short(sid)}`);
|
|
13553
|
+
tryAbortAndResume(sid, w);
|
|
13554
|
+
}
|
|
13555
|
+
} else if (subStatus.status === "idle") {
|
|
13556
|
+
const hasBusySub = await hasBusySubagents(sid);
|
|
13557
|
+
if (hasBusySub) {
|
|
13558
|
+
await log("debug", `Subagents exist but not busy yet, waiting for startup...`);
|
|
13559
|
+
} else {
|
|
13560
|
+
await log("info", `Parent ${short(sid)} stuck with no active subagents. Triggering abort+resume.`);
|
|
13561
|
+
tryAbortAndResume(sid, w);
|
|
13562
|
+
}
|
|
13294
13563
|
} else {
|
|
13295
|
-
await log("
|
|
13296
|
-
tryAbortAndResume(sid, w);
|
|
13564
|
+
await log("debug", `Subagent still running, waiting...`);
|
|
13297
13565
|
}
|
|
13298
|
-
} else {
|
|
13299
|
-
|
|
13566
|
+
} else if (!w.gaveUp) {
|
|
13567
|
+
w.gaveUp = true;
|
|
13568
|
+
dbg(`State transition on ${short(sid)}: gaveUp=false -> true`);
|
|
13569
|
+
w.orphanWatchStartAt = null;
|
|
13570
|
+
w.aborting = false;
|
|
13571
|
+
log("warn", `${short(sid)} - orphan retries exhausted.`);
|
|
13300
13572
|
}
|
|
13301
|
-
} else if (!w.gaveUp) {
|
|
13302
|
-
w.gaveUp = true;
|
|
13303
|
-
w.orphanWatchStartAt = null;
|
|
13304
|
-
w.aborting = false;
|
|
13305
|
-
log("warn", `${short(sid)} - orphan retries exhausted.`);
|
|
13306
13573
|
}
|
|
13307
|
-
}
|
|
13308
|
-
continue;
|
|
13309
|
-
}
|
|
13310
|
-
if (numBusy > 1)
|
|
13311
|
-
continue;
|
|
13312
|
-
if (now - w.lastSubagentCheckAt < checkIntervalMs * 2)
|
|
13313
|
-
continue;
|
|
13314
|
-
w.lastSubagentCheckAt = now;
|
|
13315
|
-
if (w.lastActivityAt > 0 && now - w.lastActivityAt > subagentWaitMs) {
|
|
13316
|
-
if (realStatus === "busy") {
|
|
13317
|
-
await log("debug", `Session ${short(sid)} is still busy (real status), skipping abort`);
|
|
13318
|
-
w.lastSubagentCheckAt = now;
|
|
13319
13574
|
continue;
|
|
13320
13575
|
}
|
|
13321
|
-
if (
|
|
13322
|
-
await log("debug", `Session ${short(sid)} has ${w.pendingTools} tool(s) in-flight, skipping abort check`);
|
|
13323
|
-
w.lastSubagentCheckAt = now;
|
|
13576
|
+
if (numBusy > 1)
|
|
13324
13577
|
continue;
|
|
13325
|
-
|
|
13326
|
-
const hasActiveTool = await checkSessionHasActiveTool(sid);
|
|
13327
|
-
if (hasActiveTool) {
|
|
13328
|
-
await log("debug", `Session ${short(sid)} has active tool call, skipping abort check`);
|
|
13329
|
-
w.lastSubagentCheckAt = now;
|
|
13578
|
+
if (now - w.lastSubagentCheckAt < checkIntervalMs * 2)
|
|
13330
13579
|
continue;
|
|
13580
|
+
w.lastSubagentCheckAt = now;
|
|
13581
|
+
if (w.lastActivityAt > 0 && now - w.lastActivityAt > subagentWaitMs) {
|
|
13582
|
+
if (realStatus === "busy") {
|
|
13583
|
+
await log("debug", `Session ${short(sid)} is still busy (real status), skipping abort`);
|
|
13584
|
+
w.lastSubagentCheckAt = now;
|
|
13585
|
+
continue;
|
|
13586
|
+
}
|
|
13587
|
+
if (hasInflightTools(w)) {
|
|
13588
|
+
await log("debug", `Session ${short(sid)} has ${w.pendingTools} tool(s) in-flight, skipping abort check`);
|
|
13589
|
+
w.lastSubagentCheckAt = now;
|
|
13590
|
+
continue;
|
|
13591
|
+
}
|
|
13592
|
+
const hasActiveTool = await checkSessionHasActiveTool(sid);
|
|
13593
|
+
if (hasActiveTool) {
|
|
13594
|
+
await log("debug", `Session ${short(sid)} has active tool call, skipping abort check`);
|
|
13595
|
+
w.lastSubagentCheckAt = now;
|
|
13596
|
+
continue;
|
|
13597
|
+
}
|
|
13598
|
+
const subStatus = await checkSubagentStatus(sid);
|
|
13599
|
+
if (subStatus.status === "idle" || subStatus.status === "unknown") {
|
|
13600
|
+
await log("info", `Parent ${short(sid)} stuck with no active subagents. Triggering abort+resume.`);
|
|
13601
|
+
tryAbortAndResume(sid, w);
|
|
13602
|
+
continue;
|
|
13603
|
+
} else if (subStatus.status === "crashed" && subStatus.stuckSid) {
|
|
13604
|
+
const recovered = await recoverSubagent(subStatus.stuckSid);
|
|
13605
|
+
if (recovered) {
|
|
13606
|
+
await log("info", `Sent recovery prompt to stuck subagent ${short(subStatus.stuckSid)}, waiting...`);
|
|
13607
|
+
} else {
|
|
13608
|
+
await log("info", `Parent ${short(sid)} subagent recovery failed. Triggering abort+resume.`);
|
|
13609
|
+
tryAbortAndResume(sid, w);
|
|
13610
|
+
}
|
|
13611
|
+
continue;
|
|
13612
|
+
}
|
|
13331
13613
|
}
|
|
13332
|
-
const
|
|
13333
|
-
if (
|
|
13334
|
-
|
|
13335
|
-
|
|
13336
|
-
|
|
13337
|
-
} else if (subStatus.status === "crashed" && subStatus.stuckSid) {
|
|
13338
|
-
const recovered = await recoverSubagent(subStatus.stuckSid);
|
|
13339
|
-
if (recovered) {
|
|
13340
|
-
await log("info", `Sent recovery prompt to stuck subagent ${short(subStatus.stuckSid)}, waiting...`);
|
|
13614
|
+
const idle = now - w.lastActivityAt;
|
|
13615
|
+
if (idle >= chunkTimeoutMs + gracePeriodMs) {
|
|
13616
|
+
if (hasInflightTools(w)) {
|
|
13617
|
+
await log("debug", `Session ${short(sid)} has ${w.pendingTools} tool(s) in-flight, skipping stall recovery`);
|
|
13618
|
+
w.lastSubagentCheckAt = now;
|
|
13341
13619
|
} else {
|
|
13342
|
-
|
|
13343
|
-
|
|
13620
|
+
const hasActiveTool = await checkSessionHasActiveTool(sid);
|
|
13621
|
+
if (hasActiveTool) {
|
|
13622
|
+
await log("debug", `Session ${short(sid)} has active tool call, skipping stall recovery`);
|
|
13623
|
+
w.lastSubagentCheckAt = now;
|
|
13624
|
+
} else if (w.resumeAttempts < maxRetries) {
|
|
13625
|
+
tryResume(sid, w, "Stream stall");
|
|
13626
|
+
} else if (!w.gaveUp) {
|
|
13627
|
+
w.gaveUp = true;
|
|
13628
|
+
dbg(`State transition on ${short(sid)}: gaveUp=false -> true`);
|
|
13629
|
+
log("warn", `${short(sid)} - all ${maxRetries} retries exhausted.`);
|
|
13630
|
+
}
|
|
13344
13631
|
}
|
|
13345
|
-
continue;
|
|
13346
13632
|
}
|
|
13347
13633
|
}
|
|
13348
|
-
const
|
|
13349
|
-
|
|
13350
|
-
|
|
13351
|
-
|
|
13352
|
-
w.
|
|
13353
|
-
|
|
13354
|
-
|
|
13355
|
-
|
|
13356
|
-
|
|
13357
|
-
|
|
13358
|
-
}
|
|
13359
|
-
|
|
13360
|
-
}
|
|
13361
|
-
|
|
13362
|
-
|
|
13634
|
+
for (const [sid, w] of sessions) {
|
|
13635
|
+
if (w.pendingRecovery && w.status === "idle" && !w.userCancelled && !w.aborting && !w.continuing && !w.gaveUp && w.recoveryAttempts === 0) {
|
|
13636
|
+
dbg(`Pending recovery check on ${short(sid)}: pendingRecovery=${w.pendingRecovery}, status=${w.status}, userCancelled=${w.userCancelled}, aborting=${w.aborting}, continuing=${w.continuing}, gaveUp=${w.gaveUp}, recoveryAttempts=${w.recoveryAttempts}, pendingRecoveryAt=${w.pendingRecoveryAt}`);
|
|
13637
|
+
const elapsed = Date.now() - w.pendingRecoveryAt;
|
|
13638
|
+
const requiredBackoff2 = backoffMs(w.recoveryAttempts, baseBackoffMs, maxBackoffMs);
|
|
13639
|
+
if (elapsed < requiredBackoff2) {
|
|
13640
|
+
dbg(`Backoff check on ${short(sid)}: elapsed=${elapsed}ms, required=${requiredBackoff2}ms, attempt=${w.recoveryAttempts}, pass=false`);
|
|
13641
|
+
dbg(`Pending recovery on ${short(sid)} waiting for backoff: ${requiredBackoff2 - elapsed}ms remaining`);
|
|
13642
|
+
continue;
|
|
13643
|
+
}
|
|
13644
|
+
dbg(`Backoff check on ${short(sid)}: elapsed=${elapsed}ms, required=${requiredBackoff2}ms, attempt=${w.recoveryAttempts}, pass=true`);
|
|
13645
|
+
await log("info", `Pending recovery triggered on ${short(sid)}: reason=${w.pendingRecoveryReason}, attempt=${w.recoveryAttempts + 1}, maxRetries=${maxRecoveryRetries}`);
|
|
13646
|
+
dbg(`Recovery timing on ${short(sid)}: detectionToAttemptMs=${Date.now() - w.pendingRecoveryAt}`);
|
|
13647
|
+
w.recoveryAttempts++;
|
|
13648
|
+
dbg(`State transition on ${short(sid)}: recoveryAttempts=${w.recoveryAttempts - 1} -> ${w.recoveryAttempts}`);
|
|
13649
|
+
try {
|
|
13650
|
+
await sendContinuePrompt(sid, continuePrompt, w);
|
|
13651
|
+
} catch (err) {
|
|
13652
|
+
const errMsg = err instanceof Error ? err.message : String(err);
|
|
13653
|
+
await log("warn", `${short(sid)} - pending recovery failed: ${errMsg}`);
|
|
13654
|
+
w.recoveryAttempts = 0;
|
|
13363
13655
|
}
|
|
13364
13656
|
}
|
|
13657
|
+
if (w.status !== "idle")
|
|
13658
|
+
continue;
|
|
13659
|
+
if (w.isSubagent)
|
|
13660
|
+
continue;
|
|
13661
|
+
if (w.userCancelled || w.completionSignaled)
|
|
13662
|
+
continue;
|
|
13663
|
+
if (w.continuing)
|
|
13664
|
+
continue;
|
|
13665
|
+
if (busyCount() !== 0)
|
|
13666
|
+
continue;
|
|
13667
|
+
const open = getOpenTodos(w.todos || []);
|
|
13668
|
+
if (open.length === 0)
|
|
13669
|
+
continue;
|
|
13670
|
+
if (w.todoNudgeAttempts >= maxRetries)
|
|
13671
|
+
continue;
|
|
13672
|
+
const elapsedSinceLastNudge = Date.now() - w.lastRetryAt;
|
|
13673
|
+
const requiredBackoff = backoffMs(w.todoNudgeAttempts, baseBackoffMs, maxBackoffMs);
|
|
13674
|
+
if (w.lastRetryAt > 0 && elapsedSinceLastNudge < requiredBackoff)
|
|
13675
|
+
continue;
|
|
13676
|
+
const isCelebration = await lastAssistantEndsWithCelebration(sid);
|
|
13677
|
+
if (isCelebration) {
|
|
13678
|
+
w.toolTextRecovered = true;
|
|
13679
|
+
w.completionSignaled = true;
|
|
13680
|
+
continue;
|
|
13681
|
+
}
|
|
13682
|
+
const reminder = buildOpenTodosReminder(w.todos || []);
|
|
13683
|
+
const sent = await tryResume(sid, w, "Idle with open todos (periodic)", reminder);
|
|
13684
|
+
if (sent) {
|
|
13685
|
+
w.todoNudgeAttempts++;
|
|
13686
|
+
await log("info", `${short(sid)} - idle periodic recheck: nudge ${w.todoNudgeAttempts}/${maxRetries}`);
|
|
13687
|
+
}
|
|
13365
13688
|
}
|
|
13366
|
-
|
|
13367
|
-
|
|
13689
|
+
cleanupIdleSessions();
|
|
13690
|
+
}, "periodic timer");
|
|
13368
13691
|
}, checkIntervalMs);
|
|
13369
13692
|
if (timer.unref)
|
|
13370
13693
|
timer.unref();
|
|
13371
13694
|
discoveryTimer = setInterval(() => {
|
|
13372
|
-
discoverSessions();
|
|
13695
|
+
safe(discoverSessions, "discoveryTimer").catch(() => {});
|
|
13373
13696
|
}, SESSION_DISCOVERY_INTERVAL_MS);
|
|
13374
13697
|
if (discoveryTimer.unref)
|
|
13375
13698
|
discoveryTimer.unref();
|
|
@@ -13391,6 +13714,9 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13391
13714
|
w.status = statusType;
|
|
13392
13715
|
if (statusType === "busy") {
|
|
13393
13716
|
w.lastActivityAt = Date.now();
|
|
13717
|
+
if (w.pendingRecovery) {
|
|
13718
|
+
dbg(`Pending recovery cleared on ${short(sid)}: reason=session-busy`);
|
|
13719
|
+
}
|
|
13394
13720
|
resetSessionFlags(w);
|
|
13395
13721
|
prevBusyCount = busyCount();
|
|
13396
13722
|
log("debug", `${short(sid)} -> busy (${prevBusyCount})`);
|
|
@@ -13419,9 +13745,25 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13419
13745
|
prevBusyCount = currentBusy;
|
|
13420
13746
|
log("debug", `${short(sid)} -> idle (${currentBusy})`);
|
|
13421
13747
|
if (!w.isSubagent) {
|
|
13748
|
+
if (!w.pendingRecovery && !w.completionSignaled && !w.userCancelled && !w.aborting) {
|
|
13749
|
+
try {
|
|
13750
|
+
const errInfo = getLastAssistantError(await getSessionMessages(sid));
|
|
13751
|
+
if (errInfo && isStreamingFailure(errInfo.name, errInfo.message, streamingFailureErrorNames, streamingFailureMessagePatterns)) {
|
|
13752
|
+
w.pendingRecovery = true;
|
|
13753
|
+
w.pendingRecoveryReason = errInfo.name;
|
|
13754
|
+
w.pendingRecoveryAt = Date.now();
|
|
13755
|
+
dbg(`State transition on ${short(sid)}: pendingRecovery=false -> true, reason=${errInfo.name}`);
|
|
13756
|
+
await log("info", `${short(sid)} - streaming failure detected on idle: ${errInfo.name} - ${errInfo.message}`);
|
|
13757
|
+
await tryResume(sid, w, "Streaming failure on idle", continuePrompt);
|
|
13758
|
+
}
|
|
13759
|
+
} catch (e) {
|
|
13760
|
+
const errMsg = e instanceof Error ? e.message : String(e);
|
|
13761
|
+
dbg(`session.idle sid=${short(sid)}: streaming-failure check error: ${errMsg}`);
|
|
13762
|
+
}
|
|
13763
|
+
}
|
|
13422
13764
|
const todos = w.todos || [];
|
|
13423
|
-
const
|
|
13424
|
-
if (
|
|
13765
|
+
const open = getOpenTodos(todos);
|
|
13766
|
+
if (open.length > 0 && currentBusy === 0 && !w.completionSignaled && !w.userCancelled && w.todoNudgeAttempts < maxRetries) {
|
|
13425
13767
|
const isCelebration = await lastAssistantEndsWithCelebration(sid);
|
|
13426
13768
|
if (isCelebration) {
|
|
13427
13769
|
await log("info", `${short(sid)} - \uD83C\uDF89 detected in idle handler, skipping continue`);
|
|
@@ -13432,18 +13774,52 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13432
13774
|
w.toolTextTimer = null;
|
|
13433
13775
|
}
|
|
13434
13776
|
} else {
|
|
13777
|
+
w.todoNudgeAttempts++;
|
|
13435
13778
|
const reminder = buildOpenTodosReminder(todos);
|
|
13436
|
-
await log("info", `${short(sid)} - idle with ${
|
|
13437
|
-
tryResume(sid, w, "Idle with open todos", reminder);
|
|
13779
|
+
await log("info", `${short(sid)} - idle with ${open.length} open todos. Sending reminder (nudge ${w.todoNudgeAttempts}/${maxRetries})...`);
|
|
13780
|
+
await tryResume(sid, w, "Idle with open todos", reminder);
|
|
13438
13781
|
}
|
|
13439
13782
|
}
|
|
13440
13783
|
}
|
|
13441
13784
|
if (!w.completionSignaled && !w.userCancelled && w.toolTextAttempts < maxRetries) {
|
|
13785
|
+
if (resumeOnActionIntent && !w.toolTextRecovered) {
|
|
13786
|
+
const idleSid = sid;
|
|
13787
|
+
const idleW = w;
|
|
13788
|
+
setTimeout(async () => {
|
|
13789
|
+
try {
|
|
13790
|
+
if (Date.now() - idleW.createdAt < warmupMs) {
|
|
13791
|
+
dbg(`session.idle sid=${short(idleSid)}: skipping action intent, session is warming up (${Date.now() - idleW.createdAt}ms < ${warmupMs}ms)`);
|
|
13792
|
+
return;
|
|
13793
|
+
}
|
|
13794
|
+
const response = await ctx.client.session.messages({ path: { id: idleSid } });
|
|
13795
|
+
const msgs = extractMessages(response);
|
|
13796
|
+
const lastAssistantMsg = msgs.slice().reverse().find((m) => (m.role ?? m.info?.role) === "assistant");
|
|
13797
|
+
if (lastAssistantMsg) {
|
|
13798
|
+
let lastText = "";
|
|
13799
|
+
for (const part of lastAssistantMsg.parts || []) {
|
|
13800
|
+
lastText += (part.text ?? "") + `
|
|
13801
|
+
`;
|
|
13802
|
+
}
|
|
13803
|
+
if (containsActionIntent(lastText)) {
|
|
13804
|
+
const w2 = sessions.get(idleSid);
|
|
13805
|
+
if (!w2 || w2.toolTextRecovered || w2.completionSignaled || w2.status !== "idle")
|
|
13806
|
+
return;
|
|
13807
|
+
w2.toolTextRecovered = true;
|
|
13808
|
+
w2.toolTextAttempts++;
|
|
13809
|
+
dbg(`session.idle sid=${short(idleSid)}: ACTION INTENT DETECTED, sending "${actionIntentPrompt.slice(0, 40)}..."`);
|
|
13810
|
+
await sendContinuePrompt(idleSid, actionIntentPrompt, w2);
|
|
13811
|
+
}
|
|
13812
|
+
}
|
|
13813
|
+
} catch (e) {
|
|
13814
|
+
dbg(`session.idle sid=${short(idleSid)}: delayed check error: ${e}`);
|
|
13815
|
+
}
|
|
13816
|
+
}, 500);
|
|
13817
|
+
}
|
|
13442
13818
|
if (w.toolTextTimer)
|
|
13443
13819
|
clearTimeout(w.toolTextTimer);
|
|
13444
13820
|
w.toolTextTimer = setTimeout(() => {
|
|
13445
13821
|
checkForToolCallAsText(sid, w);
|
|
13446
|
-
},
|
|
13822
|
+
}, toolTextCheckDelayMs);
|
|
13447
13823
|
}
|
|
13448
13824
|
} else if (statusType === "retry") {
|
|
13449
13825
|
touchSession(sid);
|
|
@@ -13472,12 +13848,45 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13472
13848
|
if (w) {
|
|
13473
13849
|
w.status = "idle";
|
|
13474
13850
|
resetIdleFlags(w);
|
|
13475
|
-
|
|
13851
|
+
dbg(`session.idle sid=${short(sid)}: resetIdleFlags done, toolTextRecovered=${w.toolTextRecovered}, toolTextAttempts=${w.toolTextAttempts}, maxRetries=${maxRetries}`);
|
|
13852
|
+
if (resumeOnActionIntent && !w.toolTextRecovered && !w.completionSignaled) {
|
|
13853
|
+
setTimeout(async () => {
|
|
13854
|
+
try {
|
|
13855
|
+
if (Date.now() - w.createdAt < warmupMs) {
|
|
13856
|
+
dbg(`session.idle sid=${short(sid)}: skipping action intent, session is warming up (${Date.now() - w.createdAt}ms < ${warmupMs}ms)`);
|
|
13857
|
+
return;
|
|
13858
|
+
}
|
|
13859
|
+
const response = await ctx.client.session.messages({ path: { id: sid } });
|
|
13860
|
+
const msgs = extractMessages(response);
|
|
13861
|
+
const lastAssistantMsg = msgs.slice().reverse().find((m) => (m.role ?? m.info?.role) === "assistant");
|
|
13862
|
+
if (lastAssistantMsg) {
|
|
13863
|
+
let lastText = "";
|
|
13864
|
+
for (const part of lastAssistantMsg.parts || []) {
|
|
13865
|
+
lastText += (part.text ?? "") + `
|
|
13866
|
+
`;
|
|
13867
|
+
}
|
|
13868
|
+
if (containsActionIntent(lastText)) {
|
|
13869
|
+
const w2 = sessions.get(sid);
|
|
13870
|
+
if (!w2 || w2.toolTextRecovered || w2.completionSignaled || w2.status !== "idle")
|
|
13871
|
+
return;
|
|
13872
|
+
w2.toolTextRecovered = true;
|
|
13873
|
+
w2.toolTextAttempts++;
|
|
13874
|
+
dbg(`session.idle sid=${short(sid)}: ACTION INTENT DETECTED, sending "${actionIntentPrompt.slice(0, 40)}..."`);
|
|
13875
|
+
await sendContinuePrompt(sid, actionIntentPrompt, w2);
|
|
13876
|
+
}
|
|
13877
|
+
}
|
|
13878
|
+
} catch (e) {
|
|
13879
|
+
dbg(`session.idle sid=${short(sid)}: delayed check error: ${e}`);
|
|
13880
|
+
}
|
|
13881
|
+
}, 500);
|
|
13882
|
+
}
|
|
13883
|
+
const shouldSet = !w.toolTextRecovered && w.toolTextAttempts < maxRetries;
|
|
13884
|
+
if (shouldSet) {
|
|
13476
13885
|
if (w.toolTextTimer)
|
|
13477
13886
|
clearTimeout(w.toolTextTimer);
|
|
13478
13887
|
w.toolTextTimer = setTimeout(() => {
|
|
13479
13888
|
checkForToolCallAsText(sid, w);
|
|
13480
|
-
},
|
|
13889
|
+
}, toolTextCheckDelayMs);
|
|
13481
13890
|
}
|
|
13482
13891
|
}
|
|
13483
13892
|
break;
|
|
@@ -13514,6 +13923,7 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13514
13923
|
case "session.error": {
|
|
13515
13924
|
const errorObj = getError(ev);
|
|
13516
13925
|
const errorName = errorObj?.name ?? "";
|
|
13926
|
+
const errorMessage = errorObj?.data?.message ?? String(errorObj?.data ?? "");
|
|
13517
13927
|
const isMessageAborted = errorName === "MessageAbortedError";
|
|
13518
13928
|
if (isMessageAborted) {
|
|
13519
13929
|
for (const [wSid, w] of sessions) {
|
|
@@ -13526,9 +13936,24 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13526
13936
|
log("info", "User abort (ESC)");
|
|
13527
13937
|
break;
|
|
13528
13938
|
}
|
|
13939
|
+
const isStreamingFail = isStreamingFailure(errorName, errorMessage, streamingFailureErrorNames, streamingFailureMessagePatterns);
|
|
13940
|
+
if (isStreamingFail) {
|
|
13941
|
+
if (sid) {
|
|
13942
|
+
const w = sessions.get(sid);
|
|
13943
|
+
if (w && w.status === "busy") {
|
|
13944
|
+
w.pendingRecovery = true;
|
|
13945
|
+
w.pendingRecoveryReason = errorName;
|
|
13946
|
+
w.pendingRecoveryAt = Date.now();
|
|
13947
|
+
dbg(`State transition on ${short(sid)}: pendingRecovery=false -> true, reason=${errorName}`);
|
|
13948
|
+
await log("info", `Streaming failure detected on ${short(sid)}: errorName=${errorName}, errorMessage=${errorMessage}, pendingRecoveryReason=${errorName}`);
|
|
13949
|
+
}
|
|
13950
|
+
log("info", `Streaming failure detected: ${errorName} - ${errorMessage}`);
|
|
13951
|
+
} else {
|
|
13952
|
+
log("warn", `Streaming failure detected but no session ID: ${errorName} - ${errorMessage}`);
|
|
13953
|
+
}
|
|
13954
|
+
}
|
|
13529
13955
|
if (busyCount() === 0)
|
|
13530
13956
|
break;
|
|
13531
|
-
const errorMessage = errorObj?.data?.message ?? String(errorObj?.data ?? "");
|
|
13532
13957
|
log("debug", `Session error: ${errorName} - ${errorMessage}`);
|
|
13533
13958
|
if (sid) {
|
|
13534
13959
|
const w = sessions.get(sid);
|
|
@@ -13540,7 +13965,10 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13540
13965
|
break;
|
|
13541
13966
|
}
|
|
13542
13967
|
case "command.executed": {
|
|
13543
|
-
for (const [, w2] of sessions) {
|
|
13968
|
+
for (const [sid2, w2] of sessions) {
|
|
13969
|
+
if (w2.pendingRecovery) {
|
|
13970
|
+
dbg(`Pending recovery cleared on ${short(sid2)}: reason=user-command`);
|
|
13971
|
+
}
|
|
13544
13972
|
resetSessionFlags(w2);
|
|
13545
13973
|
}
|
|
13546
13974
|
if (!sid)
|
|
@@ -13561,6 +13989,12 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13561
13989
|
const w = sessions.get(ctx2.sessionID);
|
|
13562
13990
|
if (w) {
|
|
13563
13991
|
if (!w.isSubagent) {
|
|
13992
|
+
const openTodos = (w.todos || []).filter((t) => t.status === "pending" || t.status === "in_progress");
|
|
13993
|
+
if (openTodos.length > 0 && w.taskCompleteOverrides < maxRetries) {
|
|
13994
|
+
w.taskCompleteOverrides++;
|
|
13995
|
+
await log("info", `${short(ctx2.sessionID)} - task_complete blocked: ${openTodos.length} open todos remain (override ${w.taskCompleteOverrides}/${maxRetries})`);
|
|
13996
|
+
return `You have ${openTodos.length} unfinished task(s). Please complete all remaining work before signaling completion.`;
|
|
13997
|
+
}
|
|
13564
13998
|
w.toolTextRecovered = true;
|
|
13565
13999
|
w.completionSignaled = true;
|
|
13566
14000
|
if (w.toolTextTimer) {
|
|
@@ -13579,7 +14013,11 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13579
14013
|
initialised = true;
|
|
13580
14014
|
log("info", `opencode-auto-resume ready. timeout=${chunkTimeoutMs}ms, orphan=${subagentWaitMs}ms, loop=${loopMaxContinues}x/${loopWindowMs / 1000}s`);
|
|
13581
14015
|
}
|
|
13582
|
-
handleEvent(event)
|
|
14016
|
+
handleEvent(event).catch((e) => {
|
|
14017
|
+
const msg = e instanceof Error ? e.message : String(e);
|
|
14018
|
+
console.error(`[auto-resume] handleEvent error: ${msg}`);
|
|
14019
|
+
log("error", `handleEvent error: ${msg}`).catch(() => {});
|
|
14020
|
+
});
|
|
13583
14021
|
},
|
|
13584
14022
|
config: async () => {
|
|
13585
14023
|
log("info", `opencode-auto-resume config OK`);
|
|
@@ -13612,7 +14050,10 @@ var AutoResumePlugin = async (ctx, options) => {
|
|
|
13612
14050
|
};
|
|
13613
14051
|
var src_default = AutoResumePlugin;
|
|
13614
14052
|
export {
|
|
14053
|
+
isStreamingFailure,
|
|
14054
|
+
getLastAssistantError,
|
|
13615
14055
|
src_default as default,
|
|
13616
14056
|
buildOpenTodosReminder,
|
|
14057
|
+
backoffMs,
|
|
13617
14058
|
AutoResumePlugin
|
|
13618
14059
|
};
|
package/package.json
CHANGED