tokenmaxxing 1.0.2 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DESIGN.md +5 -4
- package/package.json +1 -1
- package/src/cli/serve.ts +195 -41
- package/src/lib/slackbridge.ts +128 -35
- package/src/lib/slackstate.ts +30 -9
- package/src/lib/slackstream.ts +41 -36
package/DESIGN.md
CHANGED
|
@@ -99,11 +99,12 @@ A local Socket Mode daemon (no public URL) that turns Slack threads into Claude
|
|
|
99
99
|
- **Id mapping**: Chat SDK ids are adapter-prefixed (`thread.channelId` = `slack:C0123`, `thread.id` = `slack:C0123:<threadTs>`) while links store bare Slack ids - lookups strip the prefix via `bareChannelId`. Subscriptions live in the daemon's memory state, so every mention re-subscribes its thread; queue-skipped messages (`context.skipped`) fold into the next prompt, with the queue-entry TTL raised to 1h (expiry is silent - no app callback in chat 4.34.0 - so it must outlast a depleted-pool park plus a long turn); per-thread turn serialization is owned by the daemon itself (a promise chain per thread id), because the SDK's queue dispatch lock has a 30s TTL extended only between dispatches and every claude turn outlives it; unlinked-channel traffic logs `serve.unlinked_channel` and stays silent in Slack.
|
|
100
100
|
- **Restart resilience** (0.19.0, from the 2026-07-18 dead-thread incident: a deploy restart cut a turn mid-answer and left the thread deaf to follow-ups): startup re-subscribes every `slack-threads/` record straight on the state adapter (`state.subscribe(threadId)`; message routing checks `stateAdapter.isSubscribed`, verified in chat 4.34.0), so open threads survive restarts without needing a fresh mention. SIGTERM/SIGINT drains instead of dying: new turns are dropped loudly (`serve.drain_dropped` plus a tracked in-thread notice - a log-only drop reads as the bot thinking), tracked in-flight turns get up to 300s to finish (a re-snapshotting wait, so late-added notices still flush), then `Chat.shutdown()`; a second signal forces exit; the drain also aborts any depleted-pool park so a countdown never delays a restart. The claude child spawns detached in its own process group (`detachedClaudeSpawn` via the SDK's `spawnClaudeCodeProcess` hook), because a terminal Ctrl-C signals the whole foreground group and a non-detached child died with the daemon before the drain could save the turn; the signal handlers register before `bot.initialize()` so no turn can start while the process still has default signal disposition, and a rejecting `Chat.shutdown()` is caught so the drain always reaches its exit. When a `thread.post` rejects mid-turn (e.g. Slack finalizes an idle stream: `message_not_in_streaming_state`), relayThread drops the dead segment so the rest of the turn opens fresh messages instead of vanishing.
|
|
101
101
|
- **Interrupted-turn recovery** (from the second 2026-07-18 restart incident: a redeploy killed a ship turn 8 minutes in with zero notice - drain cannot save a long turn, and a group-wide kill can take the claude child even detached): the thread record persists the claude session id the moment the init message assigns it, and every turn is wrapped in a durable `activeTurn` marker (original prompt, start time, resume count) written before the spawn and cleared when the turn returns - except that a turn failing DURING a drain keeps its marker, since that failure is presumed to be the shutdown signal killing the child. On startup, a surviving marker means a restart killed that turn: the daemon posts a notice into the thread (the chat-sdk's documented proactive handle, rebuilt with the thread's newest human message as streaming recipient context so the resumed turn keeps its native task cards, that message's author becoming the turn's requester) and auto-resumes the work - resuming the recorded session with a continuation prompt, or replaying the original prompt fresh when the kill landed before the session opened; resumed turns settle like inbound ones (outcome log + finish_thread GC). Retries cap at 3 (each spends real quota) with a loud give-up notice posted before its marker clears; a per-thread turn lock keeps a startup resume from ever racing an inbound message turn in the same cwd, every recovery branch recomputes its decision from a fresh record reload under that lock (so a resume superseded by a faster inbound turn no-ops instead of double-running), and a blocking singleton flock makes a new daemon generation wait for the previous one - drain included - to fully exit before touching any thread record. Uncatchable deaths cannot leak a working child either: SIGHUP drains like SIGTERM (its default disposition skips the exit hook that kills the detached group), and the marker records the child's group pid plus its C-locale ps start-time token at spawn, so recovery kills only an exactly-identified SIGKILL-orphaned claude (never a recycled pid) before resuming its turn.
|
|
102
|
-
- **Depleted-pool recovery** (harvested from Slaude, reshaped around the pool): relayThread consumes the spawn-boundary switch decision instead of discarding it - a depleted pool with a known recovery inside the message's one 14min parking deadline (shared across chained parks, so the thread's queue slot is never held longer in total) posts a park notice and retries at the reset
|
|
103
|
-
- **
|
|
102
|
+
- **Depleted-pool recovery** (harvested from Slaude, reshaped around the pool): relayThread consumes the spawn-boundary switch decision instead of discarding it - a depleted pool with a known recovery inside the message's one 14min parking deadline (shared across chained parks, so the thread's queue slot is never held longer in total) posts a park notice and retries at the reset. A mid-turn limit - detected ONLY on errored results, since `is_error` can ride subtype `"success"` (`Claude AI usage limit reached|<epoch>`) and a successful answer discussing limits must never be re-run - is persisted into usage.json first (`recordObservedLimit`: the serve process has no statusLine tee, and the snapshot TTL would otherwise feed the retry the stale pre-limit state), then retried silently after a short beat, so a pool swap makes the hiccup invisible. Bounded recoveries; every drop the relay performs is announced in-thread.
|
|
103
|
+
- **Usage-limit auto-resume** (2026-07-21, superseding the drop-beats-false-promise rule for the known-wake case): a turn that ends at a usage limit with a KNOWN recovery time defers instead of dropping. relayThread reports `deferUntil`, the thread's durable activeTurn marker keeps a `resumeAt`, and a per-thread daemon timer (cancelled on drain, re-armed from the record at startup: future wakes schedule, past wakes recover immediately) fires the same interrupted-turn recovery path - serialized chain, fresh-reload supersede check, pool-recovery notice, `resumeDecision` capping total resumes. The defer triggers: known recovery past the parking deadline or recovery budget, retries exhausted while a post-burn pool probe reports a wake, and an unclassifiable child failure (no limit phrase in the error) converted by a pool probe reporting exhaustion - the pool state is the evidence the error text did not carry. Only an unknown recovery time still posts the honest re-send drop.
|
|
104
|
+
- **Slack-native output hygiene**: relayed turns run a small standalone `systemPrompt` telling the model replies render as Slack markdown, never HTML (a live turn once answered with a literal `<br>`; the SDK's default system prompt is minimal since 0.1.0, so the string replaces nothing).
|
|
104
105
|
- **Slash commands** (2026-07-18): a mention-stripped thread message that starts with `/` runs as a claude slash command - the SDK delivers a string prompt as one stream-json user message and the CLI routes a leading-slash prompt through its command table, on fresh and resumed sessions alike (live-verified with free `/usage` and `/context` reads). Local command output (`num_turns` 0) arrives as a non-streamed assistant message plus `result.result`, which relayThread's no-text fallback posts; `slackstream` also maps the documented `system/local_command_output` wire subtype in case a claude update flips the emitter. `/goal` (built-in since 2.1.139) works headless and its active goal is restored on resume, so it survives the per-message resume pattern. Slack gotcha: the composer eats ANY message whose first character is `/` client-side (channels and thread replies; registered app commands cannot even dispatch from threads), so the supported forms are `@bot /usage` (mention-first) and ` /usage` (leading space, Slack's own documented workaround) - both normalize to a position-0 command via `stripLeadingMention`.
|
|
105
|
-
- **Agent representation** (`src/lib/slackstream.ts`): turns stream natively (`chat.startStream`, which works in channel threads regardless of the assistant:write scope): thinking and tool calls as task cards ("Thinking"/tool name/"Turn", input summary + truncated output), reply text as native markdown with rendered code fences
|
|
106
|
-
- **Todo checklist card**: TodoWrite is bookkeeping, not a real tool run, so it never opens a generic card
|
|
106
|
+
- **Agent representation** (`src/lib/slackstream.ts`): turns stream natively (`chat.startStream`, which works in channel threads regardless of the assistant:write scope): thinking and tool calls as task cards ("Thinking"/tool name/"Turn", input summary + truncated output) that Slack groups into ONE collapsible plan block per turn (the serve edge wraps every posted segment in a `StreamingPlan` with `groupTasks: "plan"`, Slack's `task_display_mode: "plan"`; user ask 2026-07-20 "squash them into one dropdown", superseding the 2026-07-18 separate-messages shape), and reply text as native markdown with rendered code fences - so one turn streams as a single Slack message: the reply plus one dropdown of its activity. A later main text block (typically post-tool prose) opens on a `\n\n` paragraph break (`textStreamed`), restoring the separation the per-tool message split used to provide; whitespace-only deltas do not arm it. Notices still post as their own messages, and a rejected append (Slack's message cap, a finalized idle stream) opens a follow-on message via the dead-segment recovery.
|
|
107
|
+
- **Todo checklist card**: TodoWrite is bookkeeping, not a real tool run, so it never opens a generic card. Instead each stream gets one stable-id "Todos" card (id `todos`, subagents `todos-<parent_tool_use_id>`) that updates in place on every TodoWrite: `✅ content` for completed, `🔄 activeForm` for the in-progress item (live narration), `⬜ content` for pending (the Chat SDK Plan object's own iconography); card status goes complete only when every item is completed. The "Todos have been modified successfully" tool_result is suppressed, but a FAILED TodoWrite flips the card to error (the optimistic checklist must not claim a state that never took effect). Card ids do not carry across messages, so a TodoWrite after a dead-segment recovery starts a fresh card in the follow-on message: accepted, the latest state is always in the newest message. claude >= 2.1.142 defaults to the structured Task tools and never emits TodoWrite, so relayThread sets `CLAUDE_CODE_ENABLE_TASKS: "0"` in the spawn env (the documented opt-out; without it the checklist card is inert).
|
|
107
108
|
- **Terminal echo** (2026-07-18): the daemon registers a `log()` echo (`setLogEcho` in log.ts, off by default so hooks and the statusline keep their stdout protocols clean), so every event while it runs - serve.* plus the in-process swap/decision events from `ensureBestAccount()`/`stopHookCheck` - also prints one colored line in the foreground terminal (`formatLogLine`: dim HH:MM:SS, event painted red/yellow/cyan by structural endsWith severity, redacted fields). `serve.turn_done`/`serve.turn_failed` (with seconds) close every relayed turn, so a foreground `xx serve` is observable without tailing `tokenmaxxing.log`.
|
|
108
109
|
- **Live-verified end-to-end** (2026-07-18, #tokenmaxxing-dogfooding): mention opens a session (worktree-per-thread at the time), replies stream, thread follow-ups resume with context, cards + fenced code render, segmentation and queue folding behave. Plus the hermetic suite: schemas/links, stream mapping, fail-fast paths.
|
|
109
110
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tokenmaxxing",
|
|
3
|
-
"version": "1.0
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"description": "Automatic Claude Code account switching: pool multiple accounts and hot-swap when quota fills, resuming your session on the fresh account.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/src/cli/serve.ts
CHANGED
|
@@ -18,7 +18,7 @@
|
|
|
18
18
|
import { existsSync, realpathSync } from "node:fs";
|
|
19
19
|
import { delay, omit, uniq } from "es-toolkit";
|
|
20
20
|
import { z } from "zod";
|
|
21
|
-
import { Chat, ThreadImpl, type StreamChunk } from "chat";
|
|
21
|
+
import { Chat, StreamingPlan, ThreadImpl, type StreamChunk } from "chat";
|
|
22
22
|
import { createSlackAdapter } from "@chat-adapter/slack";
|
|
23
23
|
import { createMemoryState } from "@chat-adapter/state-memory";
|
|
24
24
|
import {
|
|
@@ -42,6 +42,7 @@ import {
|
|
|
42
42
|
type SlackThread,
|
|
43
43
|
} from "../lib/slackstate.ts";
|
|
44
44
|
import { cleanupThread, fetchWorkspaceTeamId, killGroup, relayThread, type CleanupOutcome, type TurnOutcome } from "../lib/slackbridge.ts";
|
|
45
|
+
import { ensureBestAccount, type SwapDecision } from "../sdk.ts";
|
|
45
46
|
import { pidStartTime } from "../lib/proc.ts";
|
|
46
47
|
import { acquireLock } from "../lib/lock.ts";
|
|
47
48
|
import { paths } from "../lib/paths.ts";
|
|
@@ -240,7 +241,7 @@ export function formatLogLine(input: { event: string; parts: string }): string {
|
|
|
240
241
|
const ServeThreadSchema = z.custom<{
|
|
241
242
|
id: string;
|
|
242
243
|
channelId: string;
|
|
243
|
-
post: (m: string | AsyncIterable<string | StreamChunk>) => Promise<unknown>;
|
|
244
|
+
post: (m: string | AsyncIterable<string | StreamChunk> | StreamingPlan) => Promise<unknown>;
|
|
244
245
|
subscribe: () => Promise<void>;
|
|
245
246
|
unsubscribe: () => Promise<void>;
|
|
246
247
|
startTyping: () => Promise<void>;
|
|
@@ -306,8 +307,19 @@ export function buildServeRuntime(seam: {
|
|
|
306
307
|
drainSignal?: AbortSignal;
|
|
307
308
|
}) => Promise<TurnOutcome>;
|
|
308
309
|
cleanup: (input: { threadId: string }) => CleanupOutcome;
|
|
310
|
+
/** builds a streamable proactive thread handle for marker recovery (startup
|
|
311
|
+
* resumes and deferred-turn wakes both need one). runDaemon passes a lazy
|
|
312
|
+
* closure over its bot-backed streamableThread; tests pass a fake. */
|
|
313
|
+
streamable: (threadId: string) => Promise<{ thread: ServeThread; requesterIds: string[] }>;
|
|
314
|
+
/** the pool decision a deferred wake pre-probes with before posting the
|
|
315
|
+
* recovery notice (production: ensureBestAccount; tests: a stub). */
|
|
316
|
+
decide: () => Promise<SwapDecision>;
|
|
309
317
|
}) {
|
|
310
318
|
const { cfg, workspaceTeamId } = seam;
|
|
319
|
+
/** short re-arm after a deferred wake fails transiently (Slack hiccup at
|
|
320
|
+
* the resume): the durable marker keeps deferring, so the retry loop ends
|
|
321
|
+
* the moment any marker-clearing path runs. */
|
|
322
|
+
const RESUME_RETRY_MS = 300_000;
|
|
311
323
|
// in-flight turns, tracked so a shutdown signal can drain them instead of
|
|
312
324
|
// killing a half-streamed answer (live incident 2026-07-18: a deploy
|
|
313
325
|
// restart cut a turn mid-sentence and the answer never reached Slack).
|
|
@@ -323,7 +335,7 @@ export function buildServeRuntime(seam: {
|
|
|
323
335
|
* The session id persists the moment init assigns it - a first-turn kill
|
|
324
336
|
* must stay resumable. */
|
|
325
337
|
const runTurn = async (input: {
|
|
326
|
-
thread: { id: string; post: (m:
|
|
338
|
+
thread: { id: string; post: (m: StreamingPlan) => Promise<unknown> };
|
|
327
339
|
record: SlackThread;
|
|
328
340
|
prompt: string;
|
|
329
341
|
requesterIds: string[];
|
|
@@ -341,7 +353,12 @@ export function buildServeRuntime(seam: {
|
|
|
341
353
|
prompt: input.prompt,
|
|
342
354
|
requesterIds: input.requesterIds,
|
|
343
355
|
link: input.link,
|
|
344
|
-
|
|
356
|
+
// every posted segment groups its task cards into one collapsible
|
|
357
|
+
// Slack plan block (task_display_mode "plan"; user ask 2026-07-20:
|
|
358
|
+
// "squash them into one dropdown") instead of a card-per-task
|
|
359
|
+
// timeline. Text-only segments (notices, plain replies) carry no
|
|
360
|
+
// tasks, so the wrap is a no-op for them.
|
|
361
|
+
post: (m) => input.thread.post(new StreamingPlan(m, { groupTasks: "plan" })),
|
|
345
362
|
onSpawn: (pid) => {
|
|
346
363
|
// the lstart token makes the pid a verifiable identity for the
|
|
347
364
|
// orphan reaper; a child dead before ps sees it persists without
|
|
@@ -373,13 +390,23 @@ export function buildServeRuntime(seam: {
|
|
|
373
390
|
// work; adversarial-review catch). null outcome = relay threw =
|
|
374
391
|
// still presumed killed.
|
|
375
392
|
const presumedKilled = draining && (outcome === null || (outcome.failed && !outcome.announcedDrop && !outcome.resultReceived));
|
|
393
|
+
// A usage-limit DEFERRAL keeps the marker with resumeAt: the turn
|
|
394
|
+
// returned on purpose so the queue slot frees up, and the scheduler
|
|
395
|
+
// resumes it from this durable record once the pool recovers.
|
|
396
|
+
const deferUntil = outcome?.deferUntil ?? null;
|
|
376
397
|
// An unannounced drop OUTSIDE a drain still clears the marker on
|
|
377
398
|
// purpose (retention would re-execute the turn at the next restart; see
|
|
378
399
|
// notifyDelivered's doc) - but the loss must be operator-visible.
|
|
379
|
-
if (!draining && outcome !== null && outcome.failed && outcome.rateLimited && !outcome.announcedDrop) {
|
|
400
|
+
if (!draining && outcome !== null && outcome.failed && outcome.rateLimited && !outcome.announcedDrop && deferUntil === null) {
|
|
380
401
|
log("serve.drop_unannounced", { thread: input.thread.id });
|
|
381
402
|
}
|
|
382
|
-
|
|
403
|
+
if (deferUntil !== null && record.activeTurn) {
|
|
404
|
+
record = { ...record, activeTurn: { ...record.activeTurn, resumeAt: deferUntil } };
|
|
405
|
+
saveSlackThread(record);
|
|
406
|
+
scheduleDeferred(record.threadId, deferUntil);
|
|
407
|
+
} else {
|
|
408
|
+
saveSlackThread(presumedKilled ? record : omit(record, ["activeTurn"]));
|
|
409
|
+
}
|
|
383
410
|
}
|
|
384
411
|
};
|
|
385
412
|
|
|
@@ -437,7 +464,7 @@ export function buildServeRuntime(seam: {
|
|
|
437
464
|
const stripped = input.relayed
|
|
438
465
|
.map((m) => ({ text: stripLeadingMention({ text: m.text, botUserId: seam.botUserId() }), authorId: m.authorId }))
|
|
439
466
|
.filter((m) => m.text !== "");
|
|
440
|
-
|
|
467
|
+
let prompt = stripped.map((m) => m.text).join("\n\n");
|
|
441
468
|
const requesterIds = uniq(stripped.map((m) => m.authorId));
|
|
442
469
|
if (!prompt) return;
|
|
443
470
|
// this whole handler runs inside the per-thread `serialized` chain (call
|
|
@@ -451,15 +478,33 @@ export function buildServeRuntime(seam: {
|
|
|
451
478
|
saveSlackThread(record);
|
|
452
479
|
log("serve.thread_opened", { thread: thread.id, cwd: link.repo });
|
|
453
480
|
}
|
|
454
|
-
// Inside the serialized chain a marker
|
|
455
|
-
// generation's killed turn (
|
|
456
|
-
//
|
|
457
|
-
//
|
|
458
|
-
//
|
|
459
|
-
//
|
|
460
|
-
//
|
|
461
|
-
// (closing-review catch, the recovery-path reap alone loses this race).
|
|
481
|
+
// Inside the serialized chain a surviving marker is either a PREVIOUS
|
|
482
|
+
// generation's killed turn (an inbound message can win the chain ahead of
|
|
483
|
+
// startup recovery, e.g. Slack redelivering the killed turn's unacked
|
|
484
|
+
// mention) or a LIMIT-DEFERRED turn holding its resumeAt promise. Reap a
|
|
485
|
+
// possible orphan either way, so two claude processes never share the
|
|
486
|
+
// thread's cwd and session (closing-review catch: the recovery-path reap
|
|
487
|
+
// alone loses this race).
|
|
462
488
|
if (record.activeTurn) await reapOrphan(record.activeTurn);
|
|
489
|
+
// An inbound message takes over a deferred thread (its wake timer dies
|
|
490
|
+
// with the takeover; runTurn's fresh marker replaces the deferred one),
|
|
491
|
+
// and the held prompt ALWAYS folds in front of the new text: silently
|
|
492
|
+
// discarding it was the adversarial-review MAJOR catch on PR #44 (it
|
|
493
|
+
// would re-lose exactly the 2026-07-20 two-message shape the deferral
|
|
494
|
+
// exists to save), and no spawn-progress signal on the marker can prove
|
|
495
|
+
// the held prompt ever reached the session (a child can spawn and die
|
|
496
|
+
// before init - vercel review catch). A completed turn never defers, so
|
|
497
|
+
// folding can never re-run finished work; at worst a mid-turn deferral's
|
|
498
|
+
// prompt re-appears alongside the session transcript that already holds
|
|
499
|
+
// its partial work, and the newer message steers.
|
|
500
|
+
const deferred = record.activeTurn?.resumeAt !== undefined ? record.activeTurn : null;
|
|
501
|
+
if (deferred) {
|
|
502
|
+
const timer = deferredTimers.get(thread.id);
|
|
503
|
+
if (timer !== undefined) clearTimeout(timer);
|
|
504
|
+
deferredTimers.delete(thread.id);
|
|
505
|
+
prompt = `${deferred.prompt}\n\n${prompt}`;
|
|
506
|
+
log("serve.deferred_folded", { thread: thread.id });
|
|
507
|
+
}
|
|
463
508
|
// subscriptions live in the memory state, so a daemon restart forgets
|
|
464
509
|
// them; every mention re-subscribes to keep follow-up replies flowing.
|
|
465
510
|
if (isMention) await thread.subscribe();
|
|
@@ -488,14 +533,24 @@ export function buildServeRuntime(seam: {
|
|
|
488
533
|
startedAt: number;
|
|
489
534
|
}) => {
|
|
490
535
|
const { thread, outcome, startedAt } = input;
|
|
491
|
-
log(outcome.failed ? "serve.turn_failed" : "serve.turn_done", {
|
|
536
|
+
log(outcome.deferUntil !== null ? "serve.turn_deferred" : outcome.failed ? "serve.turn_failed" : "serve.turn_done", {
|
|
492
537
|
thread: thread.id,
|
|
493
538
|
seconds: Math.round((Date.now() - startedAt) / 1000),
|
|
539
|
+
...(outcome.deferUntil === null ? {} : { resumeAt: outcome.deferUntil }),
|
|
494
540
|
});
|
|
495
541
|
// the user declared the work finished: close the thread now that the
|
|
496
542
|
// turn (and its claude subprocess) is over. Never throw into the caller -
|
|
497
543
|
// the daemon must keep serving other threads.
|
|
498
544
|
if (!outcome.finish) return;
|
|
545
|
+
if (outcome.deferUntil !== null) {
|
|
546
|
+
// finish is sticky across retries, so it can ride a deferred outcome -
|
|
547
|
+
// and cleanup would delete the very record the deferral just promised
|
|
548
|
+
// to resume (adversarial-review catch on PR #44). The deferral wins:
|
|
549
|
+
// the resumed turn finishes the remaining work, and the user closes
|
|
550
|
+
// the thread again once it actually lands.
|
|
551
|
+
log("serve.finish_deferred", { thread: thread.id });
|
|
552
|
+
return;
|
|
553
|
+
}
|
|
499
554
|
let result: CleanupOutcome;
|
|
500
555
|
try {
|
|
501
556
|
result = seam.cleanup({ threadId: thread.id });
|
|
@@ -584,25 +639,48 @@ export function buildServeRuntime(seam: {
|
|
|
584
639
|
await tracked(serialized(input.thread.id, () => handleTurn({ thread: input.thread, relayed, isMention: input.isMention })));
|
|
585
640
|
};
|
|
586
641
|
|
|
587
|
-
/**
|
|
588
|
-
*
|
|
589
|
-
*
|
|
590
|
-
*
|
|
591
|
-
*
|
|
592
|
-
*
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
642
|
+
/** Deferred-turn wakes: one process-local timer per thread, re-armed by a
|
|
643
|
+
* later deferral. The durable marker (activeTurn.resumeAt) is the source
|
|
644
|
+
* of truth - timers die with the process and startup re-arms or recovers
|
|
645
|
+
* from the record. Node clamps setTimeout delays above 2^31-1ms to 1ms,
|
|
646
|
+
* so the delay is capped instead: an early fire re-defers off the
|
|
647
|
+
* still-depleted pool, bounded by resumeCount. */
|
|
648
|
+
const deferredTimers = new Map<string, ReturnType<typeof setTimeout>>();
|
|
649
|
+
const scheduleDeferred = (threadId: string, resumeAt: number) => {
|
|
650
|
+
const prev = deferredTimers.get(threadId);
|
|
651
|
+
if (prev !== undefined) clearTimeout(prev);
|
|
652
|
+
log("serve.resume_scheduled", { thread: threadId, resumeAt });
|
|
653
|
+
const timer = setTimeout(() => {
|
|
654
|
+
deferredTimers.delete(threadId);
|
|
655
|
+
if (draining) return;
|
|
656
|
+
try {
|
|
657
|
+
const record = loadSlackThread(threadId);
|
|
658
|
+
if (!record?.activeTurn) return; // superseded: a turn already cleared it
|
|
659
|
+
void tracked(recoverInterrupted(record));
|
|
660
|
+
} catch (e) {
|
|
661
|
+
log("serve.resume_error", { thread: threadId, err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
|
|
662
|
+
}
|
|
663
|
+
}, Math.min(Math.max(0, resumeAt - Date.now()), 2_147_483_647));
|
|
664
|
+
deferredTimers.set(threadId, timer);
|
|
665
|
+
};
|
|
666
|
+
|
|
667
|
+
/** Recover one thread whose activeTurn marker survived: a restart killed
|
|
668
|
+
* that turn mid-run, or a usage-limit deferral parked it (resumeAt) and
|
|
669
|
+
* the wake arrived. Notify the thread, then resume the session (or replay
|
|
670
|
+
* the original prompt when the turn never reached init); past the retry
|
|
671
|
+
* cap, give up loudly. EVERY branch runs inside the shared per-thread
|
|
672
|
+
* `serialized` chain and recomputes the decision from a fresh reload
|
|
673
|
+
* there: an inbound turn (or Slack redelivering the killed turn's unacked
|
|
674
|
+
* mention) can win the chain first, handle the thread, and clear the
|
|
675
|
+
* marker - acting on the startup snapshot would then re-run superseded
|
|
676
|
+
* work and write stale record fields over the session id that turn
|
|
677
|
+
* persisted (adversarial-review catch). Lives in the seam with
|
|
678
|
+
* `streamable` INJECTED (the daemon passes its bot-backed handle builder,
|
|
679
|
+
* tests a fake) so that superseded-recovery race is pinnable
|
|
599
680
|
* (closing-review catch: the invariant had no test while inline). */
|
|
600
|
-
const recoverInterrupted = async (
|
|
601
|
-
record: SlackThread,
|
|
602
|
-
streamable: (threadId: string) => Promise<{ thread: ServeThread; requesterIds: string[] }>,
|
|
603
|
-
) => {
|
|
681
|
+
const recoverInterrupted = async (record: SlackThread) => {
|
|
604
682
|
try {
|
|
605
|
-
const { thread, requesterIds } = await streamable(record.threadId);
|
|
683
|
+
const { thread, requesterIds } = await seam.streamable(record.threadId);
|
|
606
684
|
const link = linkForChannel(cfg, bareChannelId(thread.channelId));
|
|
607
685
|
await serialized(record.threadId, async () => {
|
|
608
686
|
// a drain signal can land between the scan and this turn; leave the
|
|
@@ -612,6 +690,15 @@ export function buildServeRuntime(seam: {
|
|
|
612
690
|
const turn = fresh?.activeTurn;
|
|
613
691
|
const decision = fresh ? resumeDecision(fresh) : null;
|
|
614
692
|
if (!fresh || !turn || !decision) return; // superseded: an earlier turn already cleared the marker
|
|
693
|
+
if (turn.resumeAt !== undefined && turn.resumeAt > Date.now()) {
|
|
694
|
+
// a wake that queued behind an in-flight turn can find a RE-DEFERRED
|
|
695
|
+
// marker whose new wake is hours out; resuming it now would post a
|
|
696
|
+
// false "pool has recovered" notice and burn a resume attempt
|
|
697
|
+
// (adversarial-review catch on PR #44). Re-arm and step aside, the
|
|
698
|
+
// same guard the startup scan applies.
|
|
699
|
+
scheduleDeferred(record.threadId, turn.resumeAt);
|
|
700
|
+
return;
|
|
701
|
+
}
|
|
615
702
|
await reapOrphan(turn);
|
|
616
703
|
if (!link) {
|
|
617
704
|
// unlinked since the turn started: nothing can run here; drop the
|
|
@@ -631,6 +718,43 @@ export function buildServeRuntime(seam: {
|
|
|
631
718
|
saveSlackThread(omit(fresh, ["activeTurn"]));
|
|
632
719
|
return;
|
|
633
720
|
}
|
|
721
|
+
if (turn.resumeAt !== undefined) {
|
|
722
|
+
// the wake arrived, but the reset clock was extrapolated: confirm
|
|
723
|
+
// the pool ACTUALLY recovered before posting the recovery notice
|
|
724
|
+
// and spending one of the capped resume attempts - a still-depleted
|
|
725
|
+
// pool with a KNOWN wake re-defers silently (no quota was spent, so
|
|
726
|
+
// no attempt burns; cursor review catch on PR #44). This runs AFTER
|
|
727
|
+
// the give-up branch on purpose: a turn at the resume cap gives up
|
|
728
|
+
// honestly at its due wake instead of re-deferring forever (cubic
|
|
729
|
+
// review catch). A still-depleted pool with an UNKNOWN wake drops
|
|
730
|
+
// honestly (vercel + cubic review catch: falling through would post
|
|
731
|
+
// a false "pool has recovered" notice and burn an attempt on a
|
|
732
|
+
// spawn-boundary drop) - the drop-beats-false-promise principle
|
|
733
|
+
// stands for the unknown-wake case, and a short silent re-arm loop
|
|
734
|
+
// against a never-recovering pool would keep a zombie promise
|
|
735
|
+
// alive instead. A probe failure falls through to the normal
|
|
736
|
+
// resume, whose own decision path announces honestly.
|
|
737
|
+
try {
|
|
738
|
+
const verdict = await seam.decide();
|
|
739
|
+
const depleted = verdict.reason === "all-depleted" || verdict.reason === "depleted-wait";
|
|
740
|
+
if (depleted) {
|
|
741
|
+
const wake = verdict.waitUntil ?? null;
|
|
742
|
+
if (wake != null) {
|
|
743
|
+
const resumeAt = wake + 5_000;
|
|
744
|
+
saveSlackThread({ ...fresh, activeTurn: { ...turn, resumeAt } });
|
|
745
|
+
scheduleDeferred(record.threadId, resumeAt);
|
|
746
|
+
log("serve.resume_still_depleted", { thread: record.threadId, resumeAt });
|
|
747
|
+
return;
|
|
748
|
+
}
|
|
749
|
+
log("serve.resume_dropped_unknown", { thread: record.threadId });
|
|
750
|
+
await thread.post("the pool is still at its usage limit and its recovery time is now unknown - this held message is dropped; re-send it once the pool recovers.");
|
|
751
|
+
saveSlackThread(omit(fresh, ["activeTurn"]));
|
|
752
|
+
return;
|
|
753
|
+
}
|
|
754
|
+
} catch (e) {
|
|
755
|
+
log("serve.resume_probe_error", { thread: record.threadId, err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
|
|
756
|
+
}
|
|
757
|
+
}
|
|
634
758
|
log("serve.resume_interrupted", { thread: record.threadId, attempt: decision.marker.resumeCount });
|
|
635
759
|
// the notice posts BEFORE runTurn persists the incremented marker,
|
|
636
760
|
// on purpose: the cap bounds quota-SPENDING attempts (the spawn),
|
|
@@ -647,6 +771,19 @@ export function buildServeRuntime(seam: {
|
|
|
647
771
|
} catch (e) {
|
|
648
772
|
const detail = (e instanceof Error ? e.message : String(e)).slice(0, 300);
|
|
649
773
|
log("serve.resume_error", { thread: record.threadId, err: detail });
|
|
774
|
+
// a DEFERRED turn's wake must survive a transient failure here (a
|
|
775
|
+
// Slack hiccup at an unattended 4am wake would otherwise strand the
|
|
776
|
+
// held turn until the next restart - adversarial-review catch on
|
|
777
|
+
// PR #44): re-arm a short retry while the marker still defers. The
|
|
778
|
+
// marker-clearing paths (resume, supersession, unlink, give-up) all
|
|
779
|
+
// end the loop; killed-turn (no resumeAt) startup semantics keep
|
|
780
|
+
// their once-per-restart retry.
|
|
781
|
+
try {
|
|
782
|
+
const marker = loadSlackThread(record.threadId)?.activeTurn;
|
|
783
|
+
if (marker?.resumeAt !== undefined && !draining) scheduleDeferred(record.threadId, Date.now() + RESUME_RETRY_MS);
|
|
784
|
+
} catch {
|
|
785
|
+
// the record itself is unreadable; the startup scan is the backstop.
|
|
786
|
+
}
|
|
650
787
|
}
|
|
651
788
|
};
|
|
652
789
|
|
|
@@ -654,10 +791,14 @@ export function buildServeRuntime(seam: {
|
|
|
654
791
|
/** in-flight turn promises; shutdown drains them. */
|
|
655
792
|
activeTurns,
|
|
656
793
|
isDraining: () => draining,
|
|
657
|
-
/** stop taking new turns and wake parked/retrying ones.
|
|
794
|
+
/** stop taking new turns and wake parked/retrying ones. Pending deferred
|
|
795
|
+
* wakes are cancelled: the markers are durable and the next generation
|
|
796
|
+
* re-arms them at startup. */
|
|
658
797
|
beginDrain: () => {
|
|
659
798
|
draining = true;
|
|
660
799
|
drainAbort.abort();
|
|
800
|
+
for (const timer of deferredTimers.values()) clearTimeout(timer);
|
|
801
|
+
deferredTimers.clear();
|
|
661
802
|
},
|
|
662
803
|
relayable,
|
|
663
804
|
onMessage,
|
|
@@ -669,6 +810,7 @@ export function buildServeRuntime(seam: {
|
|
|
669
810
|
runTurn,
|
|
670
811
|
settleTurn,
|
|
671
812
|
recoverInterrupted,
|
|
813
|
+
scheduleDeferred,
|
|
672
814
|
};
|
|
673
815
|
}
|
|
674
816
|
|
|
@@ -761,6 +903,10 @@ async function runDaemon(): Promise<number> {
|
|
|
761
903
|
botUserId: () => slack.botUserId ?? null,
|
|
762
904
|
relay: relayThread,
|
|
763
905
|
cleanup: cleanupThread,
|
|
906
|
+
// lazy on purpose: streamableThread is declared just below and only ever
|
|
907
|
+
// invoked long after startup (recovery runs and deferred wakes).
|
|
908
|
+
streamable: (threadId) => streamableThread(threadId),
|
|
909
|
+
decide: ensureBestAccount,
|
|
764
910
|
});
|
|
765
911
|
|
|
766
912
|
/** A proactive thread handle that can still stream natively. bot.thread()
|
|
@@ -866,14 +1012,22 @@ async function runDaemon(): Promise<number> {
|
|
|
866
1012
|
for (const record of records) await state.subscribe(record.threadId);
|
|
867
1013
|
log("serve.resubscribed", { threads: records.length });
|
|
868
1014
|
|
|
869
|
-
// threads whose activeTurn marker survived the previous daemon were
|
|
870
|
-
// mid-turn by a restart (live incident 2026-07-18: a redeploy
|
|
871
|
-
// killed a ship turn 8 minutes in and the thread just went dark)
|
|
872
|
-
//
|
|
873
|
-
//
|
|
1015
|
+
// threads whose activeTurn marker survived the previous daemon were either
|
|
1016
|
+
// killed mid-turn by a restart (live incident 2026-07-18: a redeploy
|
|
1017
|
+
// silently killed a ship turn 8 minutes in and the thread just went dark)
|
|
1018
|
+
// or deferred at a usage limit (resumeAt; 2026-07-20 incident: dropped
|
|
1019
|
+
// messages sat dead for hours after the pool recovered). A future resumeAt
|
|
1020
|
+
// re-arms its timer; everything else recovers now, tracked so a drain
|
|
1021
|
+
// waits for it; the actionable decision is recomputed under the per-thread
|
|
1022
|
+
// lock inside.
|
|
874
1023
|
for (const record of records) {
|
|
875
|
-
|
|
876
|
-
|
|
1024
|
+
const marker = record.activeTurn;
|
|
1025
|
+
if (!marker) continue;
|
|
1026
|
+
if (marker.resumeAt !== undefined && marker.resumeAt > Date.now()) {
|
|
1027
|
+
runtime.scheduleDeferred(record.threadId, marker.resumeAt);
|
|
1028
|
+
continue;
|
|
1029
|
+
}
|
|
1030
|
+
void runtime.tracked(runtime.recoverInterrupted(record));
|
|
877
1031
|
}
|
|
878
1032
|
|
|
879
1033
|
console.log(`${c.green("●")} serving ${count({ n: cfg.links.length, noun: "linked channel" })} over Slack Socket Mode - mention the bot in a linked channel to open a session (Ctrl-C to stop)`);
|
package/src/lib/slackbridge.ts
CHANGED
|
@@ -18,7 +18,7 @@ import { http, safeErrorDetail } from "./http.ts";
|
|
|
18
18
|
import { loadLastSwapAt } from "./state.ts";
|
|
19
19
|
import { fmtResetShort, recordObservedLimit } from "./usage.ts";
|
|
20
20
|
import { deleteSlackThread, type SlackLink } from "./slackstate.ts";
|
|
21
|
-
import { agentEventChunks, newStreamMapState
|
|
21
|
+
import { agentEventChunks, newStreamMapState } from "./slackstream.ts";
|
|
22
22
|
import { log } from "./log.ts";
|
|
23
23
|
|
|
24
24
|
/** With systemPrompt omitted the SDK runs a MINIMAL system prompt (the
|
|
@@ -46,6 +46,11 @@ export const TurnOutcomeSchema = z.object({
|
|
|
46
46
|
* operator's benefit). A drain must not read that delivery failure as a
|
|
47
47
|
* killed child and re-run completed work (adversarial-review catch). */
|
|
48
48
|
resultReceived: z.boolean(),
|
|
49
|
+
/** epoch ms when the pool is expected usable again: set when the turn ended
|
|
50
|
+
* at a usage limit whose recovery time is known. The caller keeps the
|
|
51
|
+
* thread's activeTurn marker with resumeAt and the daemon resumes the turn
|
|
52
|
+
* itself, instead of the old "re-send it once the pool recovers" drop. */
|
|
53
|
+
deferUntil: z.number().nullable(),
|
|
49
54
|
});
|
|
50
55
|
export type TurnOutcome = z.infer<typeof TurnOutcomeSchema>;
|
|
51
56
|
|
|
@@ -69,24 +74,32 @@ export const MAX_RECOVERIES = 3;
|
|
|
69
74
|
const ParkPlanSchema = z.union([
|
|
70
75
|
z.object({ kind: z.literal("proceed") }),
|
|
71
76
|
z.object({ kind: z.literal("park"), wakeAt: z.number() }),
|
|
72
|
-
z.object({ kind: z.literal("
|
|
77
|
+
z.object({ kind: z.literal("defer"), resumeAt: z.number() }),
|
|
78
|
+
z.object({ kind: z.literal("drop") }),
|
|
73
79
|
]);
|
|
74
80
|
export type ParkPlan = z.infer<typeof ParkPlanSchema>;
|
|
75
81
|
|
|
76
|
-
/** What to do with a spawn-boundary switch decision: proceed on a usable pool
|
|
77
|
-
* park until the soonest recovery when it lands inside the
|
|
78
|
-
* shared deadline
|
|
79
|
-
*
|
|
80
|
-
*
|
|
81
|
-
*
|
|
82
|
+
/** What to do with a spawn-boundary switch decision: proceed on a usable pool;
|
|
83
|
+
* park in-handler until the soonest recovery when it lands inside the
|
|
84
|
+
* message's one shared deadline; DEFER when recovery is known but further
|
|
85
|
+
* out (or the in-handler recovery budget is spent): the handler releases the
|
|
86
|
+
* queue slot and the daemon resumes the turn from its durable marker once
|
|
87
|
+
* the pool recovers (2026-07-20 incident: dropped messages sat dead for
|
|
88
|
+
* hours after the pool recovered until the user re-sent them by hand -
|
|
89
|
+
* superseding the older drop-instead-of-promise rule, whose premise was that
|
|
90
|
+
* a will-resume promise could not be kept; the marker + scheduler + startup
|
|
91
|
+
* scan make it durable). Only an UNKNOWN recovery time still drops honestly.
|
|
92
|
+
* The deadline is fixed when the message's relay starts, so chained parks
|
|
93
|
+
* can never hold the queue slot longer than PARK_MAX_MS in total. */
|
|
82
94
|
export function parkPlan(input: { decision: SwapDecision; recoveries: number; deadline: number }): ParkPlan {
|
|
83
95
|
const depleted = input.decision.reason === "all-depleted" || input.decision.reason === "depleted-wait";
|
|
84
96
|
if (!depleted) return { kind: "proceed" };
|
|
85
97
|
const wake = input.decision.waitUntil ?? null;
|
|
98
|
+
if (wake == null) return { kind: "drop" };
|
|
86
99
|
// the grace counts against the deadline too: the promised total hold is
|
|
87
100
|
// exact, not deadline-plus-grace (review catch, PR #18).
|
|
88
|
-
if (wake
|
|
89
|
-
return { kind: "
|
|
101
|
+
if (wake + PARK_GRACE_MS > input.deadline || input.recoveries >= MAX_RECOVERIES) {
|
|
102
|
+
return { kind: "defer", resumeAt: wake + PARK_GRACE_MS };
|
|
90
103
|
}
|
|
91
104
|
return { kind: "park", wakeAt: wake + PARK_GRACE_MS };
|
|
92
105
|
}
|
|
@@ -344,25 +357,34 @@ export function detachedClaudeSpawn(options: SpawnOptions) {
|
|
|
344
357
|
}
|
|
345
358
|
|
|
346
359
|
/**
|
|
347
|
-
* One claude turn relayed into a Slack thread as
|
|
348
|
-
* text streams natively
|
|
349
|
-
* (see slackstream.ts)
|
|
350
|
-
*
|
|
351
|
-
*
|
|
352
|
-
*
|
|
353
|
-
*
|
|
354
|
-
*
|
|
355
|
-
*
|
|
360
|
+
* One claude turn relayed into a Slack thread as ONE streamed message: reply
|
|
361
|
+
* text streams natively and thinking/tool calls stream as task_update cards
|
|
362
|
+
* (see slackstream.ts) that Slack groups into a single collapsible plan block
|
|
363
|
+
* (user ask 2026-07-20: "squash them into one dropdown"; the serve edge wraps
|
|
364
|
+
* each posted segment in a StreamingPlan with groupTasks "plan", superseding
|
|
365
|
+
* the 2026-07-18 separate-messages-around-tool-runs shape). Segments still
|
|
366
|
+
* exist as the posting machinery: recovery notices post as their own
|
|
367
|
+
* messages, a rejected post opens a fresh one (so an over-long turn that
|
|
368
|
+
* trips Slack's message cap degrades to a follow-on message instead of
|
|
369
|
+
* vanishing), and segments post strictly in order: the next opens only after
|
|
370
|
+
* the previous post resolves. Never throws: a failure posts a short
|
|
371
|
+
* diagnostic line and sets outcome.failed (the daemon must keep serving other
|
|
372
|
+
* threads). Error text is message-only - a raw error body could echo request
|
|
373
|
+
* material.
|
|
356
374
|
*
|
|
357
375
|
* Depleted-pool recovery (ported from slaude at its shutdown, reshaped around
|
|
358
376
|
* the pool): the spawn-boundary switch decision is CONSUMED, not discarded -
|
|
359
377
|
* a depleted pool parks BEFORE a doomed spawn burns a failed turn, with an
|
|
360
378
|
* honest in-thread notice either way; a mid-turn limit the cached pool state
|
|
361
379
|
* did not predict is persisted (recordObservedLimit) and retried silently into
|
|
362
|
-
* the same session. Total parking is bounded by one shared
|
|
363
|
-
* deadline plus MAX_RECOVERIES
|
|
364
|
-
*
|
|
365
|
-
*
|
|
380
|
+
* the same session. Total in-handler parking is bounded by one shared
|
|
381
|
+
* PARK_MAX_MS deadline plus MAX_RECOVERIES; a limit whose recovery lands
|
|
382
|
+
* beyond that budget DEFERS instead of dropping (outcome.deferUntil): the
|
|
383
|
+
* caller keeps the thread's durable marker with resumeAt and the daemon
|
|
384
|
+
* resumes the turn itself once the pool recovers (2026-07-20 incident).
|
|
385
|
+
* Only an unknown recovery time still drops, and every drop the relay itself
|
|
386
|
+
* performs is announced in-thread (a queue-entry TTL expiry upstream is the
|
|
387
|
+
* one drop it cannot see).
|
|
366
388
|
*/
|
|
367
389
|
export async function relayThread(input: {
|
|
368
390
|
cwd: string;
|
|
@@ -389,7 +411,7 @@ export async function relayThread(input: {
|
|
|
389
411
|
* out a depleted-pool countdown. */
|
|
390
412
|
drainSignal?: AbortSignal;
|
|
391
413
|
}): Promise<TurnOutcome> {
|
|
392
|
-
const outcome: TurnOutcome = { sessionId: input.sessionId, failed: false, rateLimited: false, finish: false, announcedDrop: false, resultReceived: false };
|
|
414
|
+
const outcome: TurnOutcome = { sessionId: input.sessionId, failed: false, rateLimited: false, finish: false, announcedDrop: false, resultReceived: false, deferUntil: null };
|
|
393
415
|
let segment: ReturnType<typeof pushableStream> | null = null;
|
|
394
416
|
let segmentMeta: { text: boolean } | null = null;
|
|
395
417
|
let lastPost: Promise<unknown> = Promise.resolve();
|
|
@@ -479,8 +501,14 @@ export async function relayThread(input: {
|
|
|
479
501
|
};
|
|
480
502
|
const inWord = (epochMs: number | null) => (epochMs == null ? "an unknown time" : `~${fmtResetShort(epochMs, Date.now()) || "1m"}`);
|
|
481
503
|
|
|
504
|
+
// a non-limit failure line held back until the depleted-pool probe rules:
|
|
505
|
+
// posted verbatim on a plain failure, discarded when the turn defers (the
|
|
506
|
+
// deferral notice explains the pause; the raw line would invite a manual
|
|
507
|
+
// re-send of work the daemon resumes itself - cubic catch, PR #44).
|
|
508
|
+
let pendingFailureLine: string | null = null;
|
|
482
509
|
const runQueryOnce = async () => {
|
|
483
510
|
postedText = false;
|
|
511
|
+
pendingFailureLine = null;
|
|
484
512
|
outcome.failed = false;
|
|
485
513
|
outcome.rateLimited = false;
|
|
486
514
|
outcome.resultReceived = false;
|
|
@@ -584,15 +612,15 @@ export async function relayThread(input: {
|
|
|
584
612
|
outcome.resultReceived = true;
|
|
585
613
|
}
|
|
586
614
|
}
|
|
587
|
-
for (const part of agentEventChunks({ state: mapState, message }))
|
|
588
|
-
if (SegmentBreakSchema.safeParse(part).success) breakSegment();
|
|
589
|
-
else await push(SegmentChunkSchema.parse(part));
|
|
590
|
-
}
|
|
615
|
+
for (const part of agentEventChunks({ state: mapState, message })) await push(part);
|
|
591
616
|
}
|
|
592
617
|
// a turn that produced no streamed text (tool-only turns) still reports.
|
|
593
618
|
if (!postedText && result) await push(result);
|
|
594
619
|
if (!postedText && !result && outcome.failed && !outcome.rateLimited) {
|
|
595
|
-
|
|
620
|
+
// held back until the depleted-pool probe rules: a "trying again may
|
|
621
|
+
// help" line right before a deferral notice invites a manual re-send
|
|
622
|
+
// of work the daemon is about to resume itself (cubic catch, PR #44).
|
|
623
|
+
pendingFailureLine = "the turn ended without a result - trying again may help";
|
|
596
624
|
}
|
|
597
625
|
} catch (e) {
|
|
598
626
|
outcome.failed = true;
|
|
@@ -600,7 +628,10 @@ export async function relayThread(input: {
|
|
|
600
628
|
outcome.rateLimited = isRateLimitText({ text: detail });
|
|
601
629
|
if (outcome.rateLimited) await recordObservedLimit({ text: detail, now: Date.now(), org: spawnOrg });
|
|
602
630
|
log("serve.turn_error", { err: detail });
|
|
603
|
-
|
|
631
|
+
// same hold-back: the detail already reached the log above, and a
|
|
632
|
+
// deferral's own notice explains the pause better than a raw child
|
|
633
|
+
// error that reads as "please re-send".
|
|
634
|
+
if (!outcome.rateLimited) pendingFailureLine = `tokenmaxxing: turn failed: ${detail}`;
|
|
604
635
|
}
|
|
605
636
|
};
|
|
606
637
|
|
|
@@ -620,10 +651,23 @@ export async function relayThread(input: {
|
|
|
620
651
|
}
|
|
621
652
|
const plan = parkPlan({ decision, recoveries, deadline: parkDeadline });
|
|
622
653
|
if (plan.kind === "drop") {
|
|
654
|
+
// recovery time unknown: an auto-resume promise would be unkeepable,
|
|
655
|
+
// so the honest drop survives for exactly this case.
|
|
623
656
|
outcome.failed = true;
|
|
624
657
|
outcome.rateLimited = true;
|
|
625
|
-
log("serve.pool_depleted_drop", {
|
|
626
|
-
outcome.announcedDrop = await notifyDelivered(
|
|
658
|
+
log("serve.pool_depleted_drop", {});
|
|
659
|
+
outcome.announcedDrop = await notifyDelivered("every pooled account is at its usage limit (recovers at an unknown time) - this message was dropped; re-send it once the pool recovers.");
|
|
660
|
+
break;
|
|
661
|
+
}
|
|
662
|
+
if (plan.kind === "defer") {
|
|
663
|
+
// release the queue slot and let the daemon resume the turn from its
|
|
664
|
+
// durable marker once the pool recovers (2026-07-20 incident: dropped
|
|
665
|
+
// messages sat dead for hours after recovery until re-sent by hand).
|
|
666
|
+
outcome.failed = true;
|
|
667
|
+
outcome.rateLimited = true;
|
|
668
|
+
outcome.deferUntil = plan.resumeAt;
|
|
669
|
+
log("serve.pool_depleted_defer", { resumeAt: plan.resumeAt });
|
|
670
|
+
await notify(`every pooled account is at its usage limit - holding this message; it will resume automatically in ${inWord(plan.resumeAt)}.`);
|
|
627
671
|
break;
|
|
628
672
|
}
|
|
629
673
|
if (plan.kind === "park") {
|
|
@@ -631,15 +675,61 @@ export async function relayThread(input: {
|
|
|
631
675
|
log("serve.pool_depleted_park", { wakeAt: plan.wakeAt, recoveries });
|
|
632
676
|
await notify(`every pooled account is at its usage limit - holding this message and retrying in ${inWord(plan.wakeAt)}.`);
|
|
633
677
|
if (!(await sleep(plan.wakeAt - Date.now()))) {
|
|
678
|
+
// a drain aborted the park: the turn never spawned, so the marker
|
|
679
|
+
// survives (presumedKilled) and the next daemon start replays it.
|
|
634
680
|
outcome.failed = true;
|
|
635
|
-
|
|
681
|
+
await notify("tokenmaxxing is restarting - this message resumes after the restart.");
|
|
636
682
|
break;
|
|
637
683
|
}
|
|
638
684
|
continue;
|
|
639
685
|
}
|
|
640
686
|
await runQueryOnce();
|
|
641
|
-
if (!outcome.failed
|
|
687
|
+
if (!outcome.failed) break;
|
|
688
|
+
if (!outcome.rateLimited) {
|
|
689
|
+
// an unclassifiable child failure (e.g. "Claude Code process exited
|
|
690
|
+
// with code 1" - the 2026-07-20 Fable-cap death carried no limit
|
|
691
|
+
// phrase) against an exhausted pool IS the limit: the pool state is
|
|
692
|
+
// the evidence the error text did not carry. A completed result stays
|
|
693
|
+
// terminal, and a usable pool keeps the plain failure.
|
|
694
|
+
if (!outcome.resultReceived) {
|
|
695
|
+
try {
|
|
696
|
+
const verdict = await ensureBestAccount();
|
|
697
|
+
const depleted = verdict.reason === "all-depleted" || verdict.reason === "depleted-wait";
|
|
698
|
+
const wake = depleted ? verdict.waitUntil ?? null : null;
|
|
699
|
+
if (wake != null) {
|
|
700
|
+
outcome.rateLimited = true;
|
|
701
|
+
outcome.deferUntil = wake + PARK_GRACE_MS;
|
|
702
|
+
pendingFailureLine = null;
|
|
703
|
+
log("serve.turn_failed_depleted_defer", { resumeAt: outcome.deferUntil });
|
|
704
|
+
await notify(`the account pool is exhausted - pausing this turn; it will resume automatically in ${inWord(outcome.deferUntil)}.`);
|
|
705
|
+
}
|
|
706
|
+
} catch (e) {
|
|
707
|
+
// keep the original failure; the probe must never mask it.
|
|
708
|
+
log("serve.defer_probe_error", { err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
|
|
709
|
+
}
|
|
710
|
+
}
|
|
711
|
+
if (pendingFailureLine !== null) await push(pendingFailureLine);
|
|
712
|
+
break;
|
|
713
|
+
}
|
|
642
714
|
if (recoveries >= MAX_RECOVERIES) {
|
|
715
|
+
// out of in-handler retry budget: defer to the pool's own recovery
|
|
716
|
+
// clock when it is known, drop honestly when it is not (a stale cache
|
|
717
|
+
// claiming a usable pool while every retry limits out lands here too,
|
|
718
|
+
// and deferring on no evidence would just spin the resume cap).
|
|
719
|
+
let wake: number | null = null;
|
|
720
|
+
try {
|
|
721
|
+
const verdict = await ensureBestAccount();
|
|
722
|
+
const depleted = verdict.reason === "all-depleted" || verdict.reason === "depleted-wait";
|
|
723
|
+
wake = depleted ? verdict.waitUntil ?? null : null;
|
|
724
|
+
} catch (e) {
|
|
725
|
+
log("serve.defer_probe_error", { err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
|
|
726
|
+
}
|
|
727
|
+
if (wake != null) {
|
|
728
|
+
outcome.deferUntil = wake + PARK_GRACE_MS;
|
|
729
|
+
log("serve.rate_limited_defer", { recoveries, resumeAt: outcome.deferUntil });
|
|
730
|
+
await notify(`still at a usage limit after retries - pausing this turn; it will resume automatically in ${inWord(outcome.deferUntil)}.`);
|
|
731
|
+
break;
|
|
732
|
+
}
|
|
643
733
|
log("serve.rate_limited_drop", { recoveries });
|
|
644
734
|
outcome.announcedDrop = await notifyDelivered("still at a usage limit after retries - this message was dropped; reply when you want to try again.");
|
|
645
735
|
break;
|
|
@@ -670,8 +760,11 @@ export async function relayThread(input: {
|
|
|
670
760
|
break;
|
|
671
761
|
}
|
|
672
762
|
if (!(await sleep(Math.max(RETRY_DELAY_MS, cooldownUntil - Date.now())))) {
|
|
763
|
+
// a drain aborted the retry sleep: same as a killed child, the marker
|
|
764
|
+
// survives (presumedKilled) and the next daemon start resumes the
|
|
765
|
+
// session where it stopped.
|
|
673
766
|
outcome.failed = true;
|
|
674
|
-
|
|
767
|
+
await notify("tokenmaxxing is restarting - this turn resumes after the restart.");
|
|
675
768
|
break;
|
|
676
769
|
}
|
|
677
770
|
}
|
package/src/lib/slackstate.ts
CHANGED
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
|
|
15
15
|
import { existsSync, readFileSync, readdirSync, unlinkSync } from "node:fs";
|
|
16
16
|
import { join } from "node:path";
|
|
17
|
+
import { omit } from "es-toolkit";
|
|
17
18
|
import { z } from "zod";
|
|
18
19
|
import { paths } from "./paths.ts";
|
|
19
20
|
import { writeFileAtomic } from "./atomic.ts";
|
|
@@ -90,14 +91,23 @@ export function isChannelId(s: string): boolean {
|
|
|
90
91
|
* when it returns, so a marker that survives into the next daemon start means
|
|
91
92
|
* a restart killed the turn mid-run - startup then notifies the thread and
|
|
92
93
|
* auto-resumes it (2026-07-18 incident: a redeploy silently killed a ship
|
|
93
|
-
* turn 8 minutes in).
|
|
94
|
-
*
|
|
94
|
+
* turn 8 minutes in). A marker can also survive a RETURNED turn on purpose:
|
|
95
|
+
* a usage-limit deferral keeps it with resumeAt set, and the daemon resumes
|
|
96
|
+
* the turn itself once the pool recovers (2026-07-20 incident: limit-hit
|
|
97
|
+
* turns and depleted-pool drops sat dead until the user re-sent by hand).
|
|
98
|
+
* resumeCount caps the retries: every resumed attempt spends real quota, so
|
|
99
|
+
* a turn that keeps dying must not retry forever. */
|
|
95
100
|
const ActiveTurnSchema = z.object({
|
|
96
101
|
/** the original folded prompt, replayed verbatim when the killed turn never
|
|
97
102
|
* reached its init message (sessionId still null = nothing to resume). */
|
|
98
103
|
prompt: z.string(),
|
|
99
104
|
startedAt: z.string(),
|
|
100
105
|
resumeCount: z.number().int().nonnegative(),
|
|
106
|
+
/** epoch ms when the pool is expected usable again: present only on a
|
|
107
|
+
* usage-limit deferral. The daemon resumes the turn at this time (or at
|
|
108
|
+
* startup once it has passed); a still-depleted pool at the wake simply
|
|
109
|
+
* re-defers, bounded by resumeCount. */
|
|
110
|
+
resumeAt: z.number().int().positive().optional(),
|
|
101
111
|
/** the DETACHED claude child's process-group id, persisted at spawn: an
|
|
102
112
|
* uncatchable daemon death (SIGKILL, crash) skips the exit hook that kills
|
|
103
113
|
* the group, so recovery must reap a surviving orphan before resuming -
|
|
@@ -188,23 +198,32 @@ const ResumeDecisionSchema = z.union([
|
|
|
188
198
|
]);
|
|
189
199
|
export type ResumeDecision = z.infer<typeof ResumeDecisionSchema>;
|
|
190
200
|
|
|
191
|
-
/** What
|
|
192
|
-
* previous daemon
|
|
201
|
+
/** What to do with a thread whose activeTurn marker survived: either the
|
|
202
|
+
* previous daemon died mid-turn (no resumeAt - startup recovery) or a
|
|
203
|
+
* usage-limit deferral parked the turn (resumeAt set - the scheduler fires
|
|
204
|
+
* it once the pool recovers). Returns null for threads with no marker. The
|
|
205
|
+
* resumed marker drops resumeAt: the wake is consumed, and a still-depleted
|
|
206
|
+
* pool at the resume writes a fresh deferral with a fresh wake. */
|
|
193
207
|
export function resumeDecision(record: SlackThread): ResumeDecision | null {
|
|
194
208
|
const turn = record.activeTurn;
|
|
195
209
|
if (!turn) return null;
|
|
210
|
+
const deferred = turn.resumeAt !== undefined;
|
|
196
211
|
if (turn.resumeCount >= MAX_TURN_RESUMES) {
|
|
197
212
|
return {
|
|
198
213
|
kind: "give-up",
|
|
199
|
-
notice:
|
|
214
|
+
notice: deferred
|
|
215
|
+
? `this turn kept hitting the pool's usage limits and ${MAX_TURN_RESUMES} resume attempts were spent - giving up. Send a new message to continue.`
|
|
216
|
+
: `a daemon restart interrupted this turn, and ${MAX_TURN_RESUMES} resume attempts were interrupted too - giving up. Send a new message to continue.`,
|
|
200
217
|
};
|
|
201
218
|
}
|
|
202
|
-
const marker = { ...turn, resumeCount: turn.resumeCount + 1 };
|
|
219
|
+
const marker = omit({ ...turn, resumeCount: turn.resumeCount + 1 }, ["resumeAt"]);
|
|
203
220
|
const attempt = marker.resumeCount > 1 ? ` (attempt ${marker.resumeCount}/${MAX_TURN_RESUMES})` : "";
|
|
204
221
|
if (record.sessionId === null) {
|
|
205
222
|
return {
|
|
206
223
|
kind: "resume",
|
|
207
|
-
notice:
|
|
224
|
+
notice: deferred
|
|
225
|
+
? `the account pool has recovered - running your held message${attempt}`
|
|
226
|
+
: `a daemon restart interrupted this turn before its session opened - starting it over${attempt}`,
|
|
208
227
|
prompt: turn.prompt,
|
|
209
228
|
sessionId: null,
|
|
210
229
|
marker,
|
|
@@ -212,8 +231,10 @@ export function resumeDecision(record: SlackThread): ResumeDecision | null {
|
|
|
212
231
|
}
|
|
213
232
|
return {
|
|
214
233
|
kind: "resume",
|
|
215
|
-
notice: `a daemon restart interrupted this turn - resuming${attempt}`,
|
|
216
|
-
prompt:
|
|
234
|
+
notice: deferred ? `the account pool has recovered - resuming this turn${attempt}` : `a daemon restart interrupted this turn - resuming${attempt}`,
|
|
235
|
+
prompt: deferred
|
|
236
|
+
? `Your previous turn stopped early because every pooled account was at its usage limit; the pool has recovered. Pick up exactly where you left off and finish the task. If the work was already complete, just summarize the final state. The original request was:\n\n${turn.prompt}`
|
|
237
|
+
: `A tokenmaxxing serve daemon restart killed your previous turn mid-run. Pick up exactly where you left off and finish the task. If the work was already complete, just summarize the final state. The original request was:\n\n${turn.prompt}`,
|
|
217
238
|
sessionId: record.sessionId,
|
|
218
239
|
marker,
|
|
219
240
|
};
|
package/src/lib/slackstream.ts
CHANGED
|
@@ -1,11 +1,16 @@
|
|
|
1
1
|
// Maps Claude Agent SDK messages onto Chat SDK stream chunks so a relayed
|
|
2
2
|
// Slack turn shows the agent's process natively: task cards for thinking and
|
|
3
3
|
// tool calls (pending -> in_progress -> complete/error), streamed text via
|
|
4
|
-
// markdown_text, and a closing turn card with model/cost/duration.
|
|
5
|
-
//
|
|
6
|
-
//
|
|
7
|
-
//
|
|
8
|
-
//
|
|
4
|
+
// markdown_text, and a closing turn card with model/cost/duration. All task
|
|
5
|
+
// cards in a turn group into ONE collapsible Slack plan block (user ask
|
|
6
|
+
// 2026-07-20: "squash them into one dropdown"; the serve edge posts each turn
|
|
7
|
+
// as a StreamingPlan with groupTasks "plan", superseding the 2026-07-18
|
|
8
|
+
// separate-messages-around-tool-runs shape), so this mapper emits no segment
|
|
9
|
+
// breaks: a turn is one streamed message. TodoWrite is special-cased into one
|
|
10
|
+
// stable "Todos" checklist card per stream that updates in place as items
|
|
11
|
+
// progress. Structured chunks render only when the Slack app has the agent
|
|
12
|
+
// feature + assistant:write (the adapter drops them gracefully otherwise);
|
|
13
|
+
// plain text streams either way.
|
|
9
14
|
|
|
10
15
|
import { truncate } from "es-toolkit/compat";
|
|
11
16
|
import { z } from "zod";
|
|
@@ -58,21 +63,20 @@ const StreamMapStateSchema = z.object({
|
|
|
58
63
|
* effect). */
|
|
59
64
|
todoCards: z.record(z.string(), z.string()),
|
|
60
65
|
thinkingCount: z.number(),
|
|
61
|
-
/** reply text streamed
|
|
62
|
-
|
|
66
|
+
/** non-whitespace reply text already streamed this turn: a later main text
|
|
67
|
+
* block gets a "\n\n" separator, restoring the visual break the removed
|
|
68
|
+
* per-tool message split used to provide (without it, post-tool prose
|
|
69
|
+
* glues onto pre-tool prose and a block opening with "## " or a ```
|
|
70
|
+
* fence loses its line-start position). */
|
|
71
|
+
textStreamed: z.boolean(),
|
|
63
72
|
});
|
|
64
73
|
export type StreamMapState = z.infer<typeof StreamMapStateSchema>;
|
|
65
74
|
|
|
66
75
|
export function newStreamMapState(): StreamMapState {
|
|
67
|
-
return { open: {}, toolTitles: {}, todoCards: {}, thinkingCount: 0,
|
|
76
|
+
return { open: {}, toolTitles: {}, todoCards: {}, thinkingCount: 0, textStreamed: false };
|
|
68
77
|
}
|
|
69
78
|
|
|
70
|
-
|
|
71
|
-
* current Slack message there and posts the rest as a new one, mirroring how
|
|
72
|
-
* an agent turn reads as separate messages around its tool runs. */
|
|
73
|
-
export const SegmentBreakSchema = z.object({ type: z.literal("segment_break") });
|
|
74
|
-
|
|
75
|
-
const StreamPartSchema = z.union([z.string(), z.custom<StreamChunk>(), SegmentBreakSchema]);
|
|
79
|
+
const StreamPartSchema = z.union([z.string(), z.custom<StreamChunk>()]);
|
|
76
80
|
export type StreamPart = z.infer<typeof StreamPartSchema>;
|
|
77
81
|
|
|
78
82
|
/** One human line out of a tool-input JSON blob; null when nothing fits. */
|
|
@@ -159,13 +163,14 @@ function resultText(content: z.infer<typeof ToolResultBlockSchema>["content"]):
|
|
|
159
163
|
|
|
160
164
|
/**
|
|
161
165
|
* Consume one SDK message, mutating state, and return the stream chunks it
|
|
162
|
-
* produces (strings are streamed reply text; objects are native task cards
|
|
163
|
-
*
|
|
164
|
-
*
|
|
165
|
-
*
|
|
166
|
-
* subagent runs inside a top-level Task
|
|
167
|
-
*
|
|
168
|
-
* (parent + index) because concurrent subagent streams
|
|
166
|
+
* produces (strings are streamed reply text; objects are native task cards,
|
|
167
|
+
* all of which Slack folds into the turn's single plan block). Subagent
|
|
168
|
+
* events (parent_tool_use_id set) contribute their TOOL cards to that plan
|
|
169
|
+
* (user ask 2026-07-18: subagent activity shows alongside tool calls) but
|
|
170
|
+
* never reply text or thinking cards: a subagent runs inside a top-level Task
|
|
171
|
+
* tool, so its churn decorates the turn rather than reshaping it. Open blocks
|
|
172
|
+
* are keyed per stream (parent + index) because concurrent subagent streams
|
|
173
|
+
* reuse index space.
|
|
169
174
|
*/
|
|
170
175
|
export function agentEventChunks(input: { state: StreamMapState; message: SDKMessage }): StreamPart[] {
|
|
171
176
|
const { state, message } = input;
|
|
@@ -174,6 +179,12 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
|
|
|
174
179
|
const event = message.event;
|
|
175
180
|
if (event.type === "content_block_start") {
|
|
176
181
|
const key = `${message.parent_tool_use_id ?? "main"}:${event.index}`;
|
|
182
|
+
if (event.content_block.type === "text" && isMain && state.textStreamed) {
|
|
183
|
+
// a new text block after streamed text opens on a fresh paragraph:
|
|
184
|
+
// the whole turn is one Slack message now, and without the break
|
|
185
|
+
// post-tool prose would glue onto pre-tool prose mid-line.
|
|
186
|
+
return ["\n\n"];
|
|
187
|
+
}
|
|
177
188
|
if (event.content_block.type === "thinking" && isMain) {
|
|
178
189
|
state.thinkingCount += 1;
|
|
179
190
|
const id = `thinking-${state.thinkingCount}`;
|
|
@@ -184,9 +195,8 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
|
|
|
184
195
|
const { id, name } = event.content_block;
|
|
185
196
|
if (name === "TodoWrite") {
|
|
186
197
|
// bookkeeping, not a real tool run: no card until the list arrives,
|
|
187
|
-
//
|
|
188
|
-
//
|
|
189
|
-
// todoCards below).
|
|
198
|
+
// and no toolTitles entry (its "Todos have been modified" success
|
|
199
|
+
// result is noise; failures still surface via todoCards below).
|
|
190
200
|
const cardId = todoCardId(message.parent_tool_use_id);
|
|
191
201
|
state.open[key] = { kind: "todo", id: cardId, title: "Todos", acc: "" };
|
|
192
202
|
state.todoCards[id] = cardId;
|
|
@@ -195,12 +205,7 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
|
|
|
195
205
|
const title = toolCardTitle(name);
|
|
196
206
|
state.open[key] = { kind: "tool", id, title, acc: "" };
|
|
197
207
|
state.toolTitles[id] = title;
|
|
198
|
-
|
|
199
|
-
if (isMain && state.textSinceBreak) {
|
|
200
|
-
state.textSinceBreak = false;
|
|
201
|
-
return [{ type: "segment_break" }, card];
|
|
202
|
-
}
|
|
203
|
-
return [card];
|
|
208
|
+
return [{ type: "task_update", id, title, status: "in_progress" }];
|
|
204
209
|
}
|
|
205
210
|
return [];
|
|
206
211
|
}
|
|
@@ -208,10 +213,9 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
|
|
|
208
213
|
const open = state.open[`${message.parent_tool_use_id ?? "main"}:${event.index}`];
|
|
209
214
|
if (event.delta.type === "text_delta") {
|
|
210
215
|
if (!isMain) return [];
|
|
211
|
-
// whitespace-only deltas
|
|
212
|
-
//
|
|
213
|
-
|
|
214
|
-
if (event.delta.text.trim() !== "") state.textSinceBreak = true;
|
|
216
|
+
// whitespace-only deltas do not count: a text block carrying only
|
|
217
|
+
// "\n\n" must not earn the next block a doubled separator.
|
|
218
|
+
if (event.delta.text.trim() !== "") state.textStreamed = true;
|
|
215
219
|
return [event.delta.text];
|
|
216
220
|
}
|
|
217
221
|
if (event.delta.type === "thinking_delta" && open) open.acc += event.delta.thinking;
|
|
@@ -274,8 +278,9 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
|
|
|
274
278
|
// emitting it, the output still posts, and having posted text suppresses
|
|
275
279
|
// the result fallback so it never double-posts.
|
|
276
280
|
if (message.content.trim() === "") return [];
|
|
277
|
-
state.
|
|
278
|
-
|
|
281
|
+
const sep = state.textStreamed ? "\n\n" : "";
|
|
282
|
+
state.textStreamed = true;
|
|
283
|
+
return [sep + message.content];
|
|
279
284
|
}
|
|
280
285
|
if (message.type === "result") {
|
|
281
286
|
const models = Object.keys(message.modelUsage).join(" ");
|