tokenmaxxing 1.0.2 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/DESIGN.md CHANGED
@@ -99,11 +99,12 @@ A local Socket Mode daemon (no public URL) that turns Slack threads into Claude
99
99
  - **Id mapping**: Chat SDK ids are adapter-prefixed (`thread.channelId` = `slack:C0123`, `thread.id` = `slack:C0123:<threadTs>`) while links store bare Slack ids - lookups strip the prefix via `bareChannelId`. Subscriptions live in the daemon's memory state, so every mention re-subscribes its thread; queue-skipped messages (`context.skipped`) fold into the next prompt, with the queue-entry TTL raised to 1h (expiry is silent - no app callback in chat 4.34.0 - so it must outlast a depleted-pool park plus a long turn); per-thread turn serialization is owned by the daemon itself (a promise chain per thread id), because the SDK's queue dispatch lock has a 30s TTL extended only between dispatches and every claude turn outlives it; unlinked-channel traffic logs `serve.unlinked_channel` and stays silent in Slack.
100
100
  - **Restart resilience** (0.19.0, from the 2026-07-18 dead-thread incident: a deploy restart cut a turn mid-answer and left the thread deaf to follow-ups): startup re-subscribes every `slack-threads/` record straight on the state adapter (`state.subscribe(threadId)`; message routing checks `stateAdapter.isSubscribed`, verified in chat 4.34.0), so open threads survive restarts without needing a fresh mention. SIGTERM/SIGINT drains instead of dying: new turns are dropped loudly (`serve.drain_dropped` plus a tracked in-thread notice - a log-only drop reads as the bot thinking), tracked in-flight turns get up to 300s to finish (a re-snapshotting wait, so late-added notices still flush), then `Chat.shutdown()`; a second signal forces exit; the drain also aborts any depleted-pool park so a countdown never delays a restart. The claude child spawns detached in its own process group (`detachedClaudeSpawn` via the SDK's `spawnClaudeCodeProcess` hook), because a terminal Ctrl-C signals the whole foreground group and a non-detached child died with the daemon before the drain could save the turn; the signal handlers register before `bot.initialize()` so no turn can start while the process still has default signal disposition, and a rejecting `Chat.shutdown()` is caught so the drain always reaches its exit. When a `thread.post` rejects mid-turn (e.g. Slack finalizes an idle stream: `message_not_in_streaming_state`), relayThread drops the dead segment so the rest of the turn opens fresh messages instead of vanishing.
101
101
  - **Interrupted-turn recovery** (from the second 2026-07-18 restart incident: a redeploy killed a ship turn 8 minutes in with zero notice - drain cannot save a long turn, and a group-wide kill can take the claude child even detached): the thread record persists the claude session id the moment the init message assigns it, and every turn is wrapped in a durable `activeTurn` marker (original prompt, start time, resume count) written before the spawn and cleared when the turn returns - except that a turn failing DURING a drain keeps its marker, since that failure is presumed to be the shutdown signal killing the child. On startup, a surviving marker means a restart killed that turn: the daemon posts a notice into the thread (the chat-sdk's documented proactive handle, rebuilt with the thread's newest human message as streaming recipient context so the resumed turn keeps its native task cards, that message's author becoming the turn's requester) and auto-resumes the work - resuming the recorded session with a continuation prompt, or replaying the original prompt fresh when the kill landed before the session opened; resumed turns settle like inbound ones (outcome log + finish_thread GC). Retries cap at 3 (each spends real quota) with a loud give-up notice posted before its marker clears; a per-thread turn lock keeps a startup resume from ever racing an inbound message turn in the same cwd, every recovery branch recomputes its decision from a fresh record reload under that lock (so a resume superseded by a faster inbound turn no-ops instead of double-running), and a blocking singleton flock makes a new daemon generation wait for the previous one - drain included - to fully exit before touching any thread record. Uncatchable deaths cannot leak a working child either: SIGHUP drains like SIGTERM (its default disposition skips the exit hook that kills the detached group), and the marker records the child's group pid plus its C-locale ps start-time token at spawn, so recovery kills only an exactly-identified SIGKILL-orphaned claude (never a recycled pid) before resuming its turn.
102
- - **Depleted-pool recovery** (harvested from Slaude, reshaped around the pool): relayThread consumes the spawn-boundary switch decision instead of discarding it - a depleted pool with a known recovery inside the message's one 14min parking deadline (shared across chained parks, so the thread's queue slot is never held longer in total) posts a park notice and retries at the reset; unknown or past-deadline recovery posts an honest drop notice (dropping beats a false will-resume promise). A mid-turn limit - detected ONLY on errored results, since `is_error` can ride subtype `"success"` (`Claude AI usage limit reached|<epoch>`) and a successful answer discussing limits must never be re-run - is persisted into usage.json first (`recordObservedLimit`: the serve process has no statusLine tee, and the snapshot TTL would otherwise feed the retry the stale pre-limit state), then retried silently after a short beat, so a pool swap makes the hiccup invisible. Bounded recoveries; every drop the relay performs is announced in-thread.
103
- - **Slack-native output hygiene**: relayed turns run a small standalone `systemPrompt` telling the model replies render as Slack markdown, never HTML (a live turn once answered with a literal `<br>`; the SDK's default system prompt is minimal since 0.1.0, so the string replaces nothing), and whitespace-only text deltas do not count as reply text for segment breaking (no stranded near-blank messages).
102
+ - **Depleted-pool recovery** (harvested from Slaude, reshaped around the pool): relayThread consumes the spawn-boundary switch decision instead of discarding it - a depleted pool with a known recovery inside the message's one 14min parking deadline (shared across chained parks, so the thread's queue slot is never held longer in total) posts a park notice and retries at the reset. A mid-turn limit - detected ONLY on errored results, since `is_error` can ride subtype `"success"` (`Claude AI usage limit reached|<epoch>`) and a successful answer discussing limits must never be re-run - is persisted into usage.json first (`recordObservedLimit`: the serve process has no statusLine tee, and the snapshot TTL would otherwise feed the retry the stale pre-limit state), then retried silently after a short beat, so a pool swap makes the hiccup invisible. Bounded recoveries; every drop the relay performs is announced in-thread.
103
+ - **Usage-limit auto-resume** (2026-07-21, superseding the drop-beats-false-promise rule for the known-wake case): a turn that ends at a usage limit with a KNOWN recovery time defers instead of dropping. relayThread reports `deferUntil`, the thread's durable activeTurn marker keeps a `resumeAt`, and a per-thread daemon timer (cancelled on drain, re-armed from the record at startup: future wakes schedule, past wakes recover immediately) fires the same interrupted-turn recovery path - serialized chain, fresh-reload supersede check, pool-recovery notice, `resumeDecision` capping total resumes. The defer triggers: known recovery past the parking deadline or recovery budget, retries exhausted while a post-burn pool probe reports a wake, and an unclassifiable child failure (no limit phrase in the error) converted by a pool probe reporting exhaustion - the pool state is the evidence the error text did not carry. Only an unknown recovery time still posts the honest re-send drop.
104
+ - **Slack-native output hygiene**: relayed turns run a small standalone `systemPrompt` telling the model replies render as Slack markdown, never HTML (a live turn once answered with a literal `<br>`; the SDK's default system prompt is minimal since 0.1.0, so the string replaces nothing).
104
105
  - **Slash commands** (2026-07-18): a mention-stripped thread message that starts with `/` runs as a claude slash command - the SDK delivers a string prompt as one stream-json user message and the CLI routes a leading-slash prompt through its command table, on fresh and resumed sessions alike (live-verified with free `/usage` and `/context` reads). Local command output (`num_turns` 0) arrives as a non-streamed assistant message plus `result.result`, which relayThread's no-text fallback posts; `slackstream` also maps the documented `system/local_command_output` wire subtype in case a claude update flips the emitter. `/goal` (built-in since 2.1.139) works headless and its active goal is restored on resume, so it survives the per-message resume pattern. Slack gotcha: the composer eats ANY message whose first character is `/` client-side (channels and thread replies; registered app commands cannot even dispatch from threads), so the supported forms are `@bot /usage` (mention-first) and ` /usage` (leading space, Slack's own documented workaround) - both normalize to a position-0 command via `stripLeadingMention`.
105
- - **Agent representation** (`src/lib/slackstream.ts`): turns stream natively (`chat.startStream`, which works in channel threads regardless of the assistant:write scope): thinking and tool calls as task cards ("Thinking"/tool name/"Turn", input summary + truncated output), reply text as native markdown with rendered code fences, and a segment break whenever a tool starts after streamed text, so one turn posts as separate ordered Slack messages around its tool runs.
106
- - **Todo checklist card**: TodoWrite is bookkeeping, not a real tool run, so it never opens a generic card and never breaks a segment. Instead each stream gets one stable-id "Todos" card (id `todos`, subagents `todos-<parent_tool_use_id>`) that updates in place on every TodoWrite: `✅ content` for completed, `🔄 activeForm` for the in-progress item (live narration), `⬜ content` for pending (the Chat SDK Plan object's own iconography); card status goes complete only when every item is completed. The "Todos have been modified successfully" tool_result is suppressed, but a FAILED TodoWrite flips the card to error (the optimistic checklist must not claim a state that never took effect). Card ids do not carry across segments, so a post-break TodoWrite starts a fresh card in the new message: accepted, the latest state is always in the newest message. claude >= 2.1.142 defaults to the structured Task tools and never emits TodoWrite, so relayThread sets `CLAUDE_CODE_ENABLE_TASKS: "0"` in the spawn env (the documented opt-out; without it the checklist card is inert).
106
+ - **Agent representation** (`src/lib/slackstream.ts`): turns stream natively (`chat.startStream`, which works in channel threads regardless of the assistant:write scope): thinking and tool calls as task cards ("Thinking"/tool name/"Turn", input summary + truncated output) that Slack groups into ONE collapsible plan block per turn (the serve edge wraps every posted segment in a `StreamingPlan` with `groupTasks: "plan"`, Slack's `task_display_mode: "plan"`; user ask 2026-07-20 "squash them into one dropdown", superseding the 2026-07-18 separate-messages shape), and reply text as native markdown with rendered code fences - so one turn streams as a single Slack message: the reply plus one dropdown of its activity. A later main text block (typically post-tool prose) opens on a `\n\n` paragraph break (`textStreamed`), restoring the separation the per-tool message split used to provide; whitespace-only deltas do not arm it. Notices still post as their own messages, and a rejected append (Slack's message cap, a finalized idle stream) opens a follow-on message via the dead-segment recovery.
107
+ - **Todo checklist card**: TodoWrite is bookkeeping, not a real tool run, so it never opens a generic card. Instead each stream gets one stable-id "Todos" card (id `todos`, subagents `todos-<parent_tool_use_id>`) that updates in place on every TodoWrite: `✅ content` for completed, `🔄 activeForm` for the in-progress item (live narration), `⬜ content` for pending (the Chat SDK Plan object's own iconography); card status goes complete only when every item is completed. The "Todos have been modified successfully" tool_result is suppressed, but a FAILED TodoWrite flips the card to error (the optimistic checklist must not claim a state that never took effect). Card ids do not carry across messages, so a TodoWrite after a dead-segment recovery starts a fresh card in the follow-on message: accepted, the latest state is always in the newest message. claude >= 2.1.142 defaults to the structured Task tools and never emits TodoWrite, so relayThread sets `CLAUDE_CODE_ENABLE_TASKS: "0"` in the spawn env (the documented opt-out; without it the checklist card is inert).
107
108
  - **Terminal echo** (2026-07-18): the daemon registers a `log()` echo (`setLogEcho` in log.ts, off by default so hooks and the statusline keep their stdout protocols clean), so every event while it runs - serve.* plus the in-process swap/decision events from `ensureBestAccount()`/`stopHookCheck` - also prints one colored line in the foreground terminal (`formatLogLine`: dim HH:MM:SS, event painted red/yellow/cyan by structural endsWith severity, redacted fields). `serve.turn_done`/`serve.turn_failed` (with seconds) close every relayed turn, so a foreground `xx serve` is observable without tailing `tokenmaxxing.log`.
108
109
  - **Live-verified end-to-end** (2026-07-18, #tokenmaxxing-dogfooding): mention opens a session (worktree-per-thread at the time), replies stream, thread follow-ups resume with context, cards + fenced code render, segmentation and queue folding behave. Plus the hermetic suite: schemas/links, stream mapping, fail-fast paths.
109
110
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenmaxxing",
3
- "version": "1.0.2",
3
+ "version": "1.1.0",
4
4
  "description": "Automatic Claude Code account switching: pool multiple accounts and hot-swap when quota fills, resuming your session on the fresh account.",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/src/cli/serve.ts CHANGED
@@ -18,7 +18,7 @@
18
18
  import { existsSync, realpathSync } from "node:fs";
19
19
  import { delay, omit, uniq } from "es-toolkit";
20
20
  import { z } from "zod";
21
- import { Chat, ThreadImpl, type StreamChunk } from "chat";
21
+ import { Chat, StreamingPlan, ThreadImpl, type StreamChunk } from "chat";
22
22
  import { createSlackAdapter } from "@chat-adapter/slack";
23
23
  import { createMemoryState } from "@chat-adapter/state-memory";
24
24
  import {
@@ -42,6 +42,7 @@ import {
42
42
  type SlackThread,
43
43
  } from "../lib/slackstate.ts";
44
44
  import { cleanupThread, fetchWorkspaceTeamId, killGroup, relayThread, type CleanupOutcome, type TurnOutcome } from "../lib/slackbridge.ts";
45
+ import { ensureBestAccount, type SwapDecision } from "../sdk.ts";
45
46
  import { pidStartTime } from "../lib/proc.ts";
46
47
  import { acquireLock } from "../lib/lock.ts";
47
48
  import { paths } from "../lib/paths.ts";
@@ -240,7 +241,7 @@ export function formatLogLine(input: { event: string; parts: string }): string {
240
241
  const ServeThreadSchema = z.custom<{
241
242
  id: string;
242
243
  channelId: string;
243
- post: (m: string | AsyncIterable<string | StreamChunk>) => Promise<unknown>;
244
+ post: (m: string | AsyncIterable<string | StreamChunk> | StreamingPlan) => Promise<unknown>;
244
245
  subscribe: () => Promise<void>;
245
246
  unsubscribe: () => Promise<void>;
246
247
  startTyping: () => Promise<void>;
@@ -306,8 +307,19 @@ export function buildServeRuntime(seam: {
306
307
  drainSignal?: AbortSignal;
307
308
  }) => Promise<TurnOutcome>;
308
309
  cleanup: (input: { threadId: string }) => CleanupOutcome;
310
+ /** builds a streamable proactive thread handle for marker recovery (startup
311
+ * resumes and deferred-turn wakes both need one). runDaemon passes a lazy
312
+ * closure over its bot-backed streamableThread; tests pass a fake. */
313
+ streamable: (threadId: string) => Promise<{ thread: ServeThread; requesterIds: string[] }>;
314
+ /** the pool decision a deferred wake pre-probes with before posting the
315
+ * recovery notice (production: ensureBestAccount; tests: a stub). */
316
+ decide: () => Promise<SwapDecision>;
309
317
  }) {
310
318
  const { cfg, workspaceTeamId } = seam;
319
+ /** short re-arm after a deferred wake fails transiently (Slack hiccup at
320
+ * the resume): the durable marker keeps deferring, so the retry loop ends
321
+ * the moment any marker-clearing path runs. */
322
+ const RESUME_RETRY_MS = 300_000;
311
323
  // in-flight turns, tracked so a shutdown signal can drain them instead of
312
324
  // killing a half-streamed answer (live incident 2026-07-18: a deploy
313
325
  // restart cut a turn mid-sentence and the answer never reached Slack).
@@ -323,7 +335,7 @@ export function buildServeRuntime(seam: {
323
335
  * The session id persists the moment init assigns it - a first-turn kill
324
336
  * must stay resumable. */
325
337
  const runTurn = async (input: {
326
- thread: { id: string; post: (m: AsyncIterable<string | StreamChunk>) => Promise<unknown> };
338
+ thread: { id: string; post: (m: StreamingPlan) => Promise<unknown> };
327
339
  record: SlackThread;
328
340
  prompt: string;
329
341
  requesterIds: string[];
@@ -341,7 +353,12 @@ export function buildServeRuntime(seam: {
341
353
  prompt: input.prompt,
342
354
  requesterIds: input.requesterIds,
343
355
  link: input.link,
344
- post: (m) => input.thread.post(m),
356
+ // every posted segment groups its task cards into one collapsible
357
+ // Slack plan block (task_display_mode "plan"; user ask 2026-07-20:
358
+ // "squash them into one dropdown") instead of a card-per-task
359
+ // timeline. Text-only segments (notices, plain replies) carry no
360
+ // tasks, so the wrap is a no-op for them.
361
+ post: (m) => input.thread.post(new StreamingPlan(m, { groupTasks: "plan" })),
345
362
  onSpawn: (pid) => {
346
363
  // the lstart token makes the pid a verifiable identity for the
347
364
  // orphan reaper; a child dead before ps sees it persists without
@@ -373,13 +390,23 @@ export function buildServeRuntime(seam: {
373
390
  // work; adversarial-review catch). null outcome = relay threw =
374
391
  // still presumed killed.
375
392
  const presumedKilled = draining && (outcome === null || (outcome.failed && !outcome.announcedDrop && !outcome.resultReceived));
393
+ // A usage-limit DEFERRAL keeps the marker with resumeAt: the turn
394
+ // returned on purpose so the queue slot frees up, and the scheduler
395
+ // resumes it from this durable record once the pool recovers.
396
+ const deferUntil = outcome?.deferUntil ?? null;
376
397
  // An unannounced drop OUTSIDE a drain still clears the marker on
377
398
  // purpose (retention would re-execute the turn at the next restart; see
378
399
  // notifyDelivered's doc) - but the loss must be operator-visible.
379
- if (!draining && outcome !== null && outcome.failed && outcome.rateLimited && !outcome.announcedDrop) {
400
+ if (!draining && outcome !== null && outcome.failed && outcome.rateLimited && !outcome.announcedDrop && deferUntil === null) {
380
401
  log("serve.drop_unannounced", { thread: input.thread.id });
381
402
  }
382
- saveSlackThread(presumedKilled ? record : omit(record, ["activeTurn"]));
403
+ if (deferUntil !== null && record.activeTurn) {
404
+ record = { ...record, activeTurn: { ...record.activeTurn, resumeAt: deferUntil } };
405
+ saveSlackThread(record);
406
+ scheduleDeferred(record.threadId, deferUntil);
407
+ } else {
408
+ saveSlackThread(presumedKilled ? record : omit(record, ["activeTurn"]));
409
+ }
383
410
  }
384
411
  };
385
412
 
@@ -437,7 +464,7 @@ export function buildServeRuntime(seam: {
437
464
  const stripped = input.relayed
438
465
  .map((m) => ({ text: stripLeadingMention({ text: m.text, botUserId: seam.botUserId() }), authorId: m.authorId }))
439
466
  .filter((m) => m.text !== "");
440
- const prompt = stripped.map((m) => m.text).join("\n\n");
467
+ let prompt = stripped.map((m) => m.text).join("\n\n");
441
468
  const requesterIds = uniq(stripped.map((m) => m.authorId));
442
469
  if (!prompt) return;
443
470
  // this whole handler runs inside the per-thread `serialized` chain (call
@@ -451,15 +478,33 @@ export function buildServeRuntime(seam: {
451
478
  saveSlackThread(record);
452
479
  log("serve.thread_opened", { thread: thread.id, cwd: link.repo });
453
480
  }
454
- // Inside the serialized chain a marker can only be a PREVIOUS
455
- // generation's killed turn (this generation's turns clear theirs before
456
- // releasing the chain, and the serve-lock keeps generations exclusive):
457
- // an inbound message can win the chain ahead of startup recovery (e.g.
458
- // Slack redelivering the killed turn's unacked mention), and runTurn's
459
- // fresh marker would silently discard the orphan's pid identity - reap it
460
- // here so two claude processes never share the thread's cwd and session
461
- // (closing-review catch, the recovery-path reap alone loses this race).
481
+ // Inside the serialized chain a surviving marker is either a PREVIOUS
482
+ // generation's killed turn (an inbound message can win the chain ahead of
483
+ // startup recovery, e.g. Slack redelivering the killed turn's unacked
484
+ // mention) or a LIMIT-DEFERRED turn holding its resumeAt promise. Reap a
485
+ // possible orphan either way, so two claude processes never share the
486
+ // thread's cwd and session (closing-review catch: the recovery-path reap
487
+ // alone loses this race).
462
488
  if (record.activeTurn) await reapOrphan(record.activeTurn);
489
+ // An inbound message takes over a deferred thread (its wake timer dies
490
+ // with the takeover; runTurn's fresh marker replaces the deferred one),
491
+ // and the held prompt ALWAYS folds in front of the new text: silently
492
+ // discarding it was the adversarial-review MAJOR catch on PR #44 (it
493
+ // would re-lose exactly the 2026-07-20 two-message shape the deferral
494
+ // exists to save), and no spawn-progress signal on the marker can prove
495
+ // the held prompt ever reached the session (a child can spawn and die
496
+ // before init - vercel review catch). A completed turn never defers, so
497
+ // folding can never re-run finished work; at worst a mid-turn deferral's
498
+ // prompt re-appears alongside the session transcript that already holds
499
+ // its partial work, and the newer message steers.
500
+ const deferred = record.activeTurn?.resumeAt !== undefined ? record.activeTurn : null;
501
+ if (deferred) {
502
+ const timer = deferredTimers.get(thread.id);
503
+ if (timer !== undefined) clearTimeout(timer);
504
+ deferredTimers.delete(thread.id);
505
+ prompt = `${deferred.prompt}\n\n${prompt}`;
506
+ log("serve.deferred_folded", { thread: thread.id });
507
+ }
463
508
  // subscriptions live in the memory state, so a daemon restart forgets
464
509
  // them; every mention re-subscribes to keep follow-up replies flowing.
465
510
  if (isMention) await thread.subscribe();
@@ -488,14 +533,24 @@ export function buildServeRuntime(seam: {
488
533
  startedAt: number;
489
534
  }) => {
490
535
  const { thread, outcome, startedAt } = input;
491
- log(outcome.failed ? "serve.turn_failed" : "serve.turn_done", {
536
+ log(outcome.deferUntil !== null ? "serve.turn_deferred" : outcome.failed ? "serve.turn_failed" : "serve.turn_done", {
492
537
  thread: thread.id,
493
538
  seconds: Math.round((Date.now() - startedAt) / 1000),
539
+ ...(outcome.deferUntil === null ? {} : { resumeAt: outcome.deferUntil }),
494
540
  });
495
541
  // the user declared the work finished: close the thread now that the
496
542
  // turn (and its claude subprocess) is over. Never throw into the caller -
497
543
  // the daemon must keep serving other threads.
498
544
  if (!outcome.finish) return;
545
+ if (outcome.deferUntil !== null) {
546
+ // finish is sticky across retries, so it can ride a deferred outcome -
547
+ // and cleanup would delete the very record the deferral just promised
548
+ // to resume (adversarial-review catch on PR #44). The deferral wins:
549
+ // the resumed turn finishes the remaining work, and the user closes
550
+ // the thread again once it actually lands.
551
+ log("serve.finish_deferred", { thread: thread.id });
552
+ return;
553
+ }
499
554
  let result: CleanupOutcome;
500
555
  try {
501
556
  result = seam.cleanup({ threadId: thread.id });
@@ -584,25 +639,48 @@ export function buildServeRuntime(seam: {
584
639
  await tracked(serialized(input.thread.id, () => handleTurn({ thread: input.thread, relayed, isMention: input.isMention })));
585
640
  };
586
641
 
587
- /** Recover one thread whose activeTurn marker survived the previous daemon:
588
- * a restart killed that turn mid-run. Notify the thread, then resume the
589
- * session (or replay the original prompt when the kill landed before init
590
- * assigned a session id); past the retry cap, give up loudly. EVERY branch
591
- * runs inside the shared per-thread `serialized` chain and recomputes the
592
- * decision from a fresh reload there: an inbound turn (or Slack
593
- * redelivering the killed turn's unacked mention) can win the chain first,
594
- * handle the thread, and clear the marker - acting on the startup snapshot
595
- * would then re-run superseded work and write stale record fields over the
596
- * session id that turn persisted (adversarial-review catch). Lives in the
597
- * seam with `streamable` INJECTED (the daemon passes its bot-backed handle
598
- * builder, tests a fake) so that superseded-recovery race is pinnable
642
+ /** Deferred-turn wakes: one process-local timer per thread, re-armed by a
643
+ * later deferral. The durable marker (activeTurn.resumeAt) is the source
644
+ * of truth - timers die with the process and startup re-arms or recovers
645
+ * from the record. Node clamps setTimeout delays above 2^31-1ms to 1ms,
646
+ * so the delay is capped instead: an early fire re-defers off the
647
+ * still-depleted pool, bounded by resumeCount. */
648
+ const deferredTimers = new Map<string, ReturnType<typeof setTimeout>>();
649
+ const scheduleDeferred = (threadId: string, resumeAt: number) => {
650
+ const prev = deferredTimers.get(threadId);
651
+ if (prev !== undefined) clearTimeout(prev);
652
+ log("serve.resume_scheduled", { thread: threadId, resumeAt });
653
+ const timer = setTimeout(() => {
654
+ deferredTimers.delete(threadId);
655
+ if (draining) return;
656
+ try {
657
+ const record = loadSlackThread(threadId);
658
+ if (!record?.activeTurn) return; // superseded: a turn already cleared it
659
+ void tracked(recoverInterrupted(record));
660
+ } catch (e) {
661
+ log("serve.resume_error", { thread: threadId, err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
662
+ }
663
+ }, Math.min(Math.max(0, resumeAt - Date.now()), 2_147_483_647));
664
+ deferredTimers.set(threadId, timer);
665
+ };
666
+
667
+ /** Recover one thread whose activeTurn marker survived: a restart killed
668
+ * that turn mid-run, or a usage-limit deferral parked it (resumeAt) and
669
+ * the wake arrived. Notify the thread, then resume the session (or replay
670
+ * the original prompt when the turn never reached init); past the retry
671
+ * cap, give up loudly. EVERY branch runs inside the shared per-thread
672
+ * `serialized` chain and recomputes the decision from a fresh reload
673
+ * there: an inbound turn (or Slack redelivering the killed turn's unacked
674
+ * mention) can win the chain first, handle the thread, and clear the
675
+ * marker - acting on the startup snapshot would then re-run superseded
676
+ * work and write stale record fields over the session id that turn
677
+ * persisted (adversarial-review catch). Lives in the seam with
678
+ * `streamable` INJECTED (the daemon passes its bot-backed handle builder,
679
+ * tests a fake) so that superseded-recovery race is pinnable
599
680
  * (closing-review catch: the invariant had no test while inline). */
600
- const recoverInterrupted = async (
601
- record: SlackThread,
602
- streamable: (threadId: string) => Promise<{ thread: ServeThread; requesterIds: string[] }>,
603
- ) => {
681
+ const recoverInterrupted = async (record: SlackThread) => {
604
682
  try {
605
- const { thread, requesterIds } = await streamable(record.threadId);
683
+ const { thread, requesterIds } = await seam.streamable(record.threadId);
606
684
  const link = linkForChannel(cfg, bareChannelId(thread.channelId));
607
685
  await serialized(record.threadId, async () => {
608
686
  // a drain signal can land between the scan and this turn; leave the
@@ -612,6 +690,15 @@ export function buildServeRuntime(seam: {
612
690
  const turn = fresh?.activeTurn;
613
691
  const decision = fresh ? resumeDecision(fresh) : null;
614
692
  if (!fresh || !turn || !decision) return; // superseded: an earlier turn already cleared the marker
693
+ if (turn.resumeAt !== undefined && turn.resumeAt > Date.now()) {
694
+ // a wake that queued behind an in-flight turn can find a RE-DEFERRED
695
+ // marker whose new wake is hours out; resuming it now would post a
696
+ // false "pool has recovered" notice and burn a resume attempt
697
+ // (adversarial-review catch on PR #44). Re-arm and step aside, the
698
+ // same guard the startup scan applies.
699
+ scheduleDeferred(record.threadId, turn.resumeAt);
700
+ return;
701
+ }
615
702
  await reapOrphan(turn);
616
703
  if (!link) {
617
704
  // unlinked since the turn started: nothing can run here; drop the
@@ -631,6 +718,43 @@ export function buildServeRuntime(seam: {
631
718
  saveSlackThread(omit(fresh, ["activeTurn"]));
632
719
  return;
633
720
  }
721
+ if (turn.resumeAt !== undefined) {
722
+ // the wake arrived, but the reset clock was extrapolated: confirm
723
+ // the pool ACTUALLY recovered before posting the recovery notice
724
+ // and spending one of the capped resume attempts - a still-depleted
725
+ // pool with a KNOWN wake re-defers silently (no quota was spent, so
726
+ // no attempt burns; cursor review catch on PR #44). This runs AFTER
727
+ // the give-up branch on purpose: a turn at the resume cap gives up
728
+ // honestly at its due wake instead of re-deferring forever (cubic
729
+ // review catch). A still-depleted pool with an UNKNOWN wake drops
730
+ // honestly (vercel + cubic review catch: falling through would post
731
+ // a false "pool has recovered" notice and burn an attempt on a
732
+ // spawn-boundary drop) - the drop-beats-false-promise principle
733
+ // stands for the unknown-wake case, and a short silent re-arm loop
734
+ // against a never-recovering pool would keep a zombie promise
735
+ // alive instead. A probe failure falls through to the normal
736
+ // resume, whose own decision path announces honestly.
737
+ try {
738
+ const verdict = await seam.decide();
739
+ const depleted = verdict.reason === "all-depleted" || verdict.reason === "depleted-wait";
740
+ if (depleted) {
741
+ const wake = verdict.waitUntil ?? null;
742
+ if (wake != null) {
743
+ const resumeAt = wake + 5_000;
744
+ saveSlackThread({ ...fresh, activeTurn: { ...turn, resumeAt } });
745
+ scheduleDeferred(record.threadId, resumeAt);
746
+ log("serve.resume_still_depleted", { thread: record.threadId, resumeAt });
747
+ return;
748
+ }
749
+ log("serve.resume_dropped_unknown", { thread: record.threadId });
750
+ await thread.post("the pool is still at its usage limit and its recovery time is now unknown - this held message is dropped; re-send it once the pool recovers.");
751
+ saveSlackThread(omit(fresh, ["activeTurn"]));
752
+ return;
753
+ }
754
+ } catch (e) {
755
+ log("serve.resume_probe_error", { thread: record.threadId, err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
756
+ }
757
+ }
634
758
  log("serve.resume_interrupted", { thread: record.threadId, attempt: decision.marker.resumeCount });
635
759
  // the notice posts BEFORE runTurn persists the incremented marker,
636
760
  // on purpose: the cap bounds quota-SPENDING attempts (the spawn),
@@ -647,6 +771,19 @@ export function buildServeRuntime(seam: {
647
771
  } catch (e) {
648
772
  const detail = (e instanceof Error ? e.message : String(e)).slice(0, 300);
649
773
  log("serve.resume_error", { thread: record.threadId, err: detail });
774
+ // a DEFERRED turn's wake must survive a transient failure here (a
775
+ // Slack hiccup at an unattended 4am wake would otherwise strand the
776
+ // held turn until the next restart - adversarial-review catch on
777
+ // PR #44): re-arm a short retry while the marker still defers. The
778
+ // marker-clearing paths (resume, supersession, unlink, give-up) all
779
+ // end the loop; killed-turn (no resumeAt) startup semantics keep
780
+ // their once-per-restart retry.
781
+ try {
782
+ const marker = loadSlackThread(record.threadId)?.activeTurn;
783
+ if (marker?.resumeAt !== undefined && !draining) scheduleDeferred(record.threadId, Date.now() + RESUME_RETRY_MS);
784
+ } catch {
785
+ // the record itself is unreadable; the startup scan is the backstop.
786
+ }
650
787
  }
651
788
  };
652
789
 
@@ -654,10 +791,14 @@ export function buildServeRuntime(seam: {
654
791
  /** in-flight turn promises; shutdown drains them. */
655
792
  activeTurns,
656
793
  isDraining: () => draining,
657
- /** stop taking new turns and wake parked/retrying ones. */
794
+ /** stop taking new turns and wake parked/retrying ones. Pending deferred
795
+ * wakes are cancelled: the markers are durable and the next generation
796
+ * re-arms them at startup. */
658
797
  beginDrain: () => {
659
798
  draining = true;
660
799
  drainAbort.abort();
800
+ for (const timer of deferredTimers.values()) clearTimeout(timer);
801
+ deferredTimers.clear();
661
802
  },
662
803
  relayable,
663
804
  onMessage,
@@ -669,6 +810,7 @@ export function buildServeRuntime(seam: {
669
810
  runTurn,
670
811
  settleTurn,
671
812
  recoverInterrupted,
813
+ scheduleDeferred,
672
814
  };
673
815
  }
674
816
 
@@ -761,6 +903,10 @@ async function runDaemon(): Promise<number> {
761
903
  botUserId: () => slack.botUserId ?? null,
762
904
  relay: relayThread,
763
905
  cleanup: cleanupThread,
906
+ // lazy on purpose: streamableThread is declared just below and only ever
907
+ // invoked long after startup (recovery runs and deferred wakes).
908
+ streamable: (threadId) => streamableThread(threadId),
909
+ decide: ensureBestAccount,
764
910
  });
765
911
 
766
912
  /** A proactive thread handle that can still stream natively. bot.thread()
@@ -866,14 +1012,22 @@ async function runDaemon(): Promise<number> {
866
1012
  for (const record of records) await state.subscribe(record.threadId);
867
1013
  log("serve.resubscribed", { threads: records.length });
868
1014
 
869
- // threads whose activeTurn marker survived the previous daemon were killed
870
- // mid-turn by a restart (live incident 2026-07-18: a redeploy silently
871
- // killed a ship turn 8 minutes in and the thread just went dark). Each gets
872
- // a notice and an auto-resumed turn, tracked so a drain waits for them too;
873
- // the actionable decision is recomputed under the per-thread lock inside.
1015
+ // threads whose activeTurn marker survived the previous daemon were either
1016
+ // killed mid-turn by a restart (live incident 2026-07-18: a redeploy
1017
+ // silently killed a ship turn 8 minutes in and the thread just went dark)
1018
+ // or deferred at a usage limit (resumeAt; 2026-07-20 incident: dropped
1019
+ // messages sat dead for hours after the pool recovered). A future resumeAt
1020
+ // re-arms its timer; everything else recovers now, tracked so a drain
1021
+ // waits for it; the actionable decision is recomputed under the per-thread
1022
+ // lock inside.
874
1023
  for (const record of records) {
875
- if (!record.activeTurn) continue;
876
- void runtime.tracked(runtime.recoverInterrupted(record, streamableThread));
1024
+ const marker = record.activeTurn;
1025
+ if (!marker) continue;
1026
+ if (marker.resumeAt !== undefined && marker.resumeAt > Date.now()) {
1027
+ runtime.scheduleDeferred(record.threadId, marker.resumeAt);
1028
+ continue;
1029
+ }
1030
+ void runtime.tracked(runtime.recoverInterrupted(record));
877
1031
  }
878
1032
 
879
1033
  console.log(`${c.green("●")} serving ${count({ n: cfg.links.length, noun: "linked channel" })} over Slack Socket Mode - mention the bot in a linked channel to open a session (Ctrl-C to stop)`);
@@ -18,7 +18,7 @@ import { http, safeErrorDetail } from "./http.ts";
18
18
  import { loadLastSwapAt } from "./state.ts";
19
19
  import { fmtResetShort, recordObservedLimit } from "./usage.ts";
20
20
  import { deleteSlackThread, type SlackLink } from "./slackstate.ts";
21
- import { agentEventChunks, newStreamMapState, SegmentBreakSchema } from "./slackstream.ts";
21
+ import { agentEventChunks, newStreamMapState } from "./slackstream.ts";
22
22
  import { log } from "./log.ts";
23
23
 
24
24
  /** With systemPrompt omitted the SDK runs a MINIMAL system prompt (the
@@ -46,6 +46,11 @@ export const TurnOutcomeSchema = z.object({
46
46
  * operator's benefit). A drain must not read that delivery failure as a
47
47
  * killed child and re-run completed work (adversarial-review catch). */
48
48
  resultReceived: z.boolean(),
49
+ /** epoch ms when the pool is expected usable again: set when the turn ended
50
+ * at a usage limit whose recovery time is known. The caller keeps the
51
+ * thread's activeTurn marker with resumeAt and the daemon resumes the turn
52
+ * itself, instead of the old "re-send it once the pool recovers" drop. */
53
+ deferUntil: z.number().nullable(),
49
54
  });
50
55
  export type TurnOutcome = z.infer<typeof TurnOutcomeSchema>;
51
56
 
@@ -69,24 +74,32 @@ export const MAX_RECOVERIES = 3;
69
74
  const ParkPlanSchema = z.union([
70
75
  z.object({ kind: z.literal("proceed") }),
71
76
  z.object({ kind: z.literal("park"), wakeAt: z.number() }),
72
- z.object({ kind: z.literal("drop"), recoversAt: z.number().nullable() }),
77
+ z.object({ kind: z.literal("defer"), resumeAt: z.number() }),
78
+ z.object({ kind: z.literal("drop") }),
73
79
  ]);
74
80
  export type ParkPlan = z.infer<typeof ParkPlanSchema>;
75
81
 
76
- /** What to do with a spawn-boundary switch decision: proceed on a usable pool,
77
- * park until the soonest recovery when it lands inside the message's one
78
- * shared deadline, drop honestly otherwise (dropping beats a false
79
- * will-resume promise - slaude's recorded rationale). The deadline is fixed
80
- * when the message's relay starts, so chained parks can never hold the queue
81
- * slot longer than PARK_MAX_MS in total. */
82
+ /** What to do with a spawn-boundary switch decision: proceed on a usable pool;
83
+ * park in-handler until the soonest recovery when it lands inside the
84
+ * message's one shared deadline; DEFER when recovery is known but further
85
+ * out (or the in-handler recovery budget is spent): the handler releases the
86
+ * queue slot and the daemon resumes the turn from its durable marker once
87
+ * the pool recovers (2026-07-20 incident: dropped messages sat dead for
88
+ * hours after the pool recovered until the user re-sent them by hand -
89
+ * superseding the older drop-instead-of-promise rule, whose premise was that
90
+ * a will-resume promise could not be kept; the marker + scheduler + startup
91
+ * scan make it durable). Only an UNKNOWN recovery time still drops honestly.
92
+ * The deadline is fixed when the message's relay starts, so chained parks
93
+ * can never hold the queue slot longer than PARK_MAX_MS in total. */
82
94
  export function parkPlan(input: { decision: SwapDecision; recoveries: number; deadline: number }): ParkPlan {
83
95
  const depleted = input.decision.reason === "all-depleted" || input.decision.reason === "depleted-wait";
84
96
  if (!depleted) return { kind: "proceed" };
85
97
  const wake = input.decision.waitUntil ?? null;
98
+ if (wake == null) return { kind: "drop" };
86
99
  // the grace counts against the deadline too: the promised total hold is
87
100
  // exact, not deadline-plus-grace (review catch, PR #18).
88
- if (wake == null || wake + PARK_GRACE_MS > input.deadline || input.recoveries >= MAX_RECOVERIES) {
89
- return { kind: "drop", recoversAt: wake };
101
+ if (wake + PARK_GRACE_MS > input.deadline || input.recoveries >= MAX_RECOVERIES) {
102
+ return { kind: "defer", resumeAt: wake + PARK_GRACE_MS };
90
103
  }
91
104
  return { kind: "park", wakeAt: wake + PARK_GRACE_MS };
92
105
  }
@@ -344,25 +357,34 @@ export function detachedClaudeSpawn(options: SpawnOptions) {
344
357
  }
345
358
 
346
359
  /**
347
- * One claude turn relayed into a Slack thread as a SEQUENCE of messages: reply
348
- * text streams natively, thinking and tool calls stream as task_update cards
349
- * (see slackstream.ts), and a segment_break (a tool starting after streamed
350
- * text) closes the current Slack message and opens the next one, so a turn
351
- * reads as separate messages around its tool runs (user ask 2026-07-18).
352
- * Segments post strictly in order: the next opens only after the previous
353
- * post resolves. Never throws: a failure posts a short diagnostic line and
354
- * sets outcome.failed (the daemon must keep serving other threads). Error
355
- * text is message-only - a raw error body could echo request material.
360
+ * One claude turn relayed into a Slack thread as ONE streamed message: reply
361
+ * text streams natively and thinking/tool calls stream as task_update cards
362
+ * (see slackstream.ts) that Slack groups into a single collapsible plan block
363
+ * (user ask 2026-07-20: "squash them into one dropdown"; the serve edge wraps
364
+ * each posted segment in a StreamingPlan with groupTasks "plan", superseding
365
+ * the 2026-07-18 separate-messages-around-tool-runs shape). Segments still
366
+ * exist as the posting machinery: recovery notices post as their own
367
+ * messages, a rejected post opens a fresh one (so an over-long turn that
368
+ * trips Slack's message cap degrades to a follow-on message instead of
369
+ * vanishing), and segments post strictly in order: the next opens only after
370
+ * the previous post resolves. Never throws: a failure posts a short
371
+ * diagnostic line and sets outcome.failed (the daemon must keep serving other
372
+ * threads). Error text is message-only - a raw error body could echo request
373
+ * material.
356
374
  *
357
375
  * Depleted-pool recovery (ported from slaude at its shutdown, reshaped around
358
376
  * the pool): the spawn-boundary switch decision is CONSUMED, not discarded -
359
377
  * a depleted pool parks BEFORE a doomed spawn burns a failed turn, with an
360
378
  * honest in-thread notice either way; a mid-turn limit the cached pool state
361
379
  * did not predict is persisted (recordObservedLimit) and retried silently into
362
- * the same session. Total parking is bounded by one shared PARK_MAX_MS
363
- * deadline plus MAX_RECOVERIES, and every drop the relay itself performs is
364
- * announced in-thread (a queue-entry TTL expiry upstream is the one drop it
365
- * cannot see).
380
+ * the same session. Total in-handler parking is bounded by one shared
381
+ * PARK_MAX_MS deadline plus MAX_RECOVERIES; a limit whose recovery lands
382
+ * beyond that budget DEFERS instead of dropping (outcome.deferUntil): the
383
+ * caller keeps the thread's durable marker with resumeAt and the daemon
384
+ * resumes the turn itself once the pool recovers (2026-07-20 incident).
385
+ * Only an unknown recovery time still drops, and every drop the relay itself
386
+ * performs is announced in-thread (a queue-entry TTL expiry upstream is the
387
+ * one drop it cannot see).
366
388
  */
367
389
  export async function relayThread(input: {
368
390
  cwd: string;
@@ -389,7 +411,7 @@ export async function relayThread(input: {
389
411
  * out a depleted-pool countdown. */
390
412
  drainSignal?: AbortSignal;
391
413
  }): Promise<TurnOutcome> {
392
- const outcome: TurnOutcome = { sessionId: input.sessionId, failed: false, rateLimited: false, finish: false, announcedDrop: false, resultReceived: false };
414
+ const outcome: TurnOutcome = { sessionId: input.sessionId, failed: false, rateLimited: false, finish: false, announcedDrop: false, resultReceived: false, deferUntil: null };
393
415
  let segment: ReturnType<typeof pushableStream> | null = null;
394
416
  let segmentMeta: { text: boolean } | null = null;
395
417
  let lastPost: Promise<unknown> = Promise.resolve();
@@ -479,8 +501,14 @@ export async function relayThread(input: {
479
501
  };
480
502
  const inWord = (epochMs: number | null) => (epochMs == null ? "an unknown time" : `~${fmtResetShort(epochMs, Date.now()) || "1m"}`);
481
503
 
504
+ // a non-limit failure line held back until the depleted-pool probe rules:
505
+ // posted verbatim on a plain failure, discarded when the turn defers (the
506
+ // deferral notice explains the pause; the raw line would invite a manual
507
+ // re-send of work the daemon resumes itself - cubic catch, PR #44).
508
+ let pendingFailureLine: string | null = null;
482
509
  const runQueryOnce = async () => {
483
510
  postedText = false;
511
+ pendingFailureLine = null;
484
512
  outcome.failed = false;
485
513
  outcome.rateLimited = false;
486
514
  outcome.resultReceived = false;
@@ -584,15 +612,15 @@ export async function relayThread(input: {
584
612
  outcome.resultReceived = true;
585
613
  }
586
614
  }
587
- for (const part of agentEventChunks({ state: mapState, message })) {
588
- if (SegmentBreakSchema.safeParse(part).success) breakSegment();
589
- else await push(SegmentChunkSchema.parse(part));
590
- }
615
+ for (const part of agentEventChunks({ state: mapState, message })) await push(part);
591
616
  }
592
617
  // a turn that produced no streamed text (tool-only turns) still reports.
593
618
  if (!postedText && result) await push(result);
594
619
  if (!postedText && !result && outcome.failed && !outcome.rateLimited) {
595
- await push("the turn ended without a result - trying again may help");
620
+ // held back until the depleted-pool probe rules: a "trying again may
621
+ // help" line right before a deferral notice invites a manual re-send
622
+ // of work the daemon is about to resume itself (cubic catch, PR #44).
623
+ pendingFailureLine = "the turn ended without a result - trying again may help";
596
624
  }
597
625
  } catch (e) {
598
626
  outcome.failed = true;
@@ -600,7 +628,10 @@ export async function relayThread(input: {
600
628
  outcome.rateLimited = isRateLimitText({ text: detail });
601
629
  if (outcome.rateLimited) await recordObservedLimit({ text: detail, now: Date.now(), org: spawnOrg });
602
630
  log("serve.turn_error", { err: detail });
603
- if (!outcome.rateLimited) await push(`tokenmaxxing: turn failed: ${detail}`);
631
+ // same hold-back: the detail already reached the log above, and a
632
+ // deferral's own notice explains the pause better than a raw child
633
+ // error that reads as "please re-send".
634
+ if (!outcome.rateLimited) pendingFailureLine = `tokenmaxxing: turn failed: ${detail}`;
604
635
  }
605
636
  };
606
637
 
@@ -620,10 +651,23 @@ export async function relayThread(input: {
620
651
  }
621
652
  const plan = parkPlan({ decision, recoveries, deadline: parkDeadline });
622
653
  if (plan.kind === "drop") {
654
+ // recovery time unknown: an auto-resume promise would be unkeepable,
655
+ // so the honest drop survives for exactly this case.
623
656
  outcome.failed = true;
624
657
  outcome.rateLimited = true;
625
- log("serve.pool_depleted_drop", { recoversAt: plan.recoversAt });
626
- outcome.announcedDrop = await notifyDelivered(`every pooled account is at its usage limit (recovers in ${inWord(plan.recoversAt)}) - this message was dropped; re-send it once the pool recovers.`);
658
+ log("serve.pool_depleted_drop", {});
659
+ outcome.announcedDrop = await notifyDelivered("every pooled account is at its usage limit (recovers at an unknown time) - this message was dropped; re-send it once the pool recovers.");
660
+ break;
661
+ }
662
+ if (plan.kind === "defer") {
663
+ // release the queue slot and let the daemon resume the turn from its
664
+ // durable marker once the pool recovers (2026-07-20 incident: dropped
665
+ // messages sat dead for hours after recovery until re-sent by hand).
666
+ outcome.failed = true;
667
+ outcome.rateLimited = true;
668
+ outcome.deferUntil = plan.resumeAt;
669
+ log("serve.pool_depleted_defer", { resumeAt: plan.resumeAt });
670
+ await notify(`every pooled account is at its usage limit - holding this message; it will resume automatically in ${inWord(plan.resumeAt)}.`);
627
671
  break;
628
672
  }
629
673
  if (plan.kind === "park") {
@@ -631,15 +675,61 @@ export async function relayThread(input: {
631
675
  log("serve.pool_depleted_park", { wakeAt: plan.wakeAt, recoveries });
632
676
  await notify(`every pooled account is at its usage limit - holding this message and retrying in ${inWord(plan.wakeAt)}.`);
633
677
  if (!(await sleep(plan.wakeAt - Date.now()))) {
678
+ // a drain aborted the park: the turn never spawned, so the marker
679
+ // survives (presumedKilled) and the next daemon start replays it.
634
680
  outcome.failed = true;
635
- outcome.announcedDrop = await notifyDelivered("tokenmaxxing is restarting - this message was dropped; please re-send it.");
681
+ await notify("tokenmaxxing is restarting - this message resumes after the restart.");
636
682
  break;
637
683
  }
638
684
  continue;
639
685
  }
640
686
  await runQueryOnce();
641
- if (!outcome.failed || !outcome.rateLimited) break;
687
+ if (!outcome.failed) break;
688
+ if (!outcome.rateLimited) {
689
+ // an unclassifiable child failure (e.g. "Claude Code process exited
690
+ // with code 1" - the 2026-07-20 Fable-cap death carried no limit
691
+ // phrase) against an exhausted pool IS the limit: the pool state is
692
+ // the evidence the error text did not carry. A completed result stays
693
+ // terminal, and a usable pool keeps the plain failure.
694
+ if (!outcome.resultReceived) {
695
+ try {
696
+ const verdict = await ensureBestAccount();
697
+ const depleted = verdict.reason === "all-depleted" || verdict.reason === "depleted-wait";
698
+ const wake = depleted ? verdict.waitUntil ?? null : null;
699
+ if (wake != null) {
700
+ outcome.rateLimited = true;
701
+ outcome.deferUntil = wake + PARK_GRACE_MS;
702
+ pendingFailureLine = null;
703
+ log("serve.turn_failed_depleted_defer", { resumeAt: outcome.deferUntil });
704
+ await notify(`the account pool is exhausted - pausing this turn; it will resume automatically in ${inWord(outcome.deferUntil)}.`);
705
+ }
706
+ } catch (e) {
707
+ // keep the original failure; the probe must never mask it.
708
+ log("serve.defer_probe_error", { err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
709
+ }
710
+ }
711
+ if (pendingFailureLine !== null) await push(pendingFailureLine);
712
+ break;
713
+ }
642
714
  if (recoveries >= MAX_RECOVERIES) {
715
+ // out of in-handler retry budget: defer to the pool's own recovery
716
+ // clock when it is known, drop honestly when it is not (a stale cache
717
+ // claiming a usable pool while every retry limits out lands here too,
718
+ // and deferring on no evidence would just spin the resume cap).
719
+ let wake: number | null = null;
720
+ try {
721
+ const verdict = await ensureBestAccount();
722
+ const depleted = verdict.reason === "all-depleted" || verdict.reason === "depleted-wait";
723
+ wake = depleted ? verdict.waitUntil ?? null : null;
724
+ } catch (e) {
725
+ log("serve.defer_probe_error", { err: (e instanceof Error ? e.message : String(e)).slice(0, 300) });
726
+ }
727
+ if (wake != null) {
728
+ outcome.deferUntil = wake + PARK_GRACE_MS;
729
+ log("serve.rate_limited_defer", { recoveries, resumeAt: outcome.deferUntil });
730
+ await notify(`still at a usage limit after retries - pausing this turn; it will resume automatically in ${inWord(outcome.deferUntil)}.`);
731
+ break;
732
+ }
643
733
  log("serve.rate_limited_drop", { recoveries });
644
734
  outcome.announcedDrop = await notifyDelivered("still at a usage limit after retries - this message was dropped; reply when you want to try again.");
645
735
  break;
@@ -670,8 +760,11 @@ export async function relayThread(input: {
670
760
  break;
671
761
  }
672
762
  if (!(await sleep(Math.max(RETRY_DELAY_MS, cooldownUntil - Date.now())))) {
763
+ // a drain aborted the retry sleep: same as a killed child, the marker
764
+ // survives (presumedKilled) and the next daemon start resumes the
765
+ // session where it stopped.
673
766
  outcome.failed = true;
674
- outcome.announcedDrop = await notifyDelivered("tokenmaxxing is restarting - this message was dropped; please re-send it.");
767
+ await notify("tokenmaxxing is restarting - this turn resumes after the restart.");
675
768
  break;
676
769
  }
677
770
  }
@@ -14,6 +14,7 @@
14
14
 
15
15
  import { existsSync, readFileSync, readdirSync, unlinkSync } from "node:fs";
16
16
  import { join } from "node:path";
17
+ import { omit } from "es-toolkit";
17
18
  import { z } from "zod";
18
19
  import { paths } from "./paths.ts";
19
20
  import { writeFileAtomic } from "./atomic.ts";
@@ -90,14 +91,23 @@ export function isChannelId(s: string): boolean {
90
91
  * when it returns, so a marker that survives into the next daemon start means
91
92
  * a restart killed the turn mid-run - startup then notifies the thread and
92
93
  * auto-resumes it (2026-07-18 incident: a redeploy silently killed a ship
93
- * turn 8 minutes in). resumeCount caps the retries: every resumed attempt
94
- * spends real quota, so a turn that keeps dying must not retry forever. */
94
+ * turn 8 minutes in). A marker can also survive a RETURNED turn on purpose:
95
+ * a usage-limit deferral keeps it with resumeAt set, and the daemon resumes
96
+ * the turn itself once the pool recovers (2026-07-20 incident: limit-hit
97
+ * turns and depleted-pool drops sat dead until the user re-sent by hand).
98
+ * resumeCount caps the retries: every resumed attempt spends real quota, so
99
+ * a turn that keeps dying must not retry forever. */
95
100
  const ActiveTurnSchema = z.object({
96
101
  /** the original folded prompt, replayed verbatim when the killed turn never
97
102
  * reached its init message (sessionId still null = nothing to resume). */
98
103
  prompt: z.string(),
99
104
  startedAt: z.string(),
100
105
  resumeCount: z.number().int().nonnegative(),
106
+ /** epoch ms when the pool is expected usable again: present only on a
107
+ * usage-limit deferral. The daemon resumes the turn at this time (or at
108
+ * startup once it has passed); a still-depleted pool at the wake simply
109
+ * re-defers, bounded by resumeCount. */
110
+ resumeAt: z.number().int().positive().optional(),
101
111
  /** the DETACHED claude child's process-group id, persisted at spawn: an
102
112
  * uncatchable daemon death (SIGKILL, crash) skips the exit hook that kills
103
113
  * the group, so recovery must reap a surviving orphan before resuming -
@@ -188,23 +198,32 @@ const ResumeDecisionSchema = z.union([
188
198
  ]);
189
199
  export type ResumeDecision = z.infer<typeof ResumeDecisionSchema>;
190
200
 
191
- /** What startup should do with a thread whose activeTurn marker survived the
192
- * previous daemon. Returns null for threads with no surviving marker. */
201
+ /** What to do with a thread whose activeTurn marker survived: either the
202
+ * previous daemon died mid-turn (no resumeAt - startup recovery) or a
203
+ * usage-limit deferral parked the turn (resumeAt set - the scheduler fires
204
+ * it once the pool recovers). Returns null for threads with no marker. The
205
+ * resumed marker drops resumeAt: the wake is consumed, and a still-depleted
206
+ * pool at the resume writes a fresh deferral with a fresh wake. */
193
207
  export function resumeDecision(record: SlackThread): ResumeDecision | null {
194
208
  const turn = record.activeTurn;
195
209
  if (!turn) return null;
210
+ const deferred = turn.resumeAt !== undefined;
196
211
  if (turn.resumeCount >= MAX_TURN_RESUMES) {
197
212
  return {
198
213
  kind: "give-up",
199
- notice: `a daemon restart interrupted this turn, and ${MAX_TURN_RESUMES} resume attempts were interrupted too - giving up. Send a new message to continue.`,
214
+ notice: deferred
215
+ ? `this turn kept hitting the pool's usage limits and ${MAX_TURN_RESUMES} resume attempts were spent - giving up. Send a new message to continue.`
216
+ : `a daemon restart interrupted this turn, and ${MAX_TURN_RESUMES} resume attempts were interrupted too - giving up. Send a new message to continue.`,
200
217
  };
201
218
  }
202
- const marker = { ...turn, resumeCount: turn.resumeCount + 1 };
219
+ const marker = omit({ ...turn, resumeCount: turn.resumeCount + 1 }, ["resumeAt"]);
203
220
  const attempt = marker.resumeCount > 1 ? ` (attempt ${marker.resumeCount}/${MAX_TURN_RESUMES})` : "";
204
221
  if (record.sessionId === null) {
205
222
  return {
206
223
  kind: "resume",
207
- notice: `a daemon restart interrupted this turn before its session opened - starting it over${attempt}`,
224
+ notice: deferred
225
+ ? `the account pool has recovered - running your held message${attempt}`
226
+ : `a daemon restart interrupted this turn before its session opened - starting it over${attempt}`,
208
227
  prompt: turn.prompt,
209
228
  sessionId: null,
210
229
  marker,
@@ -212,8 +231,10 @@ export function resumeDecision(record: SlackThread): ResumeDecision | null {
212
231
  }
213
232
  return {
214
233
  kind: "resume",
215
- notice: `a daemon restart interrupted this turn - resuming${attempt}`,
216
- prompt: `A tokenmaxxing serve daemon restart killed your previous turn mid-run. Pick up exactly where you left off and finish the task. If the work was already complete, just summarize the final state. The original request was:\n\n${turn.prompt}`,
234
+ notice: deferred ? `the account pool has recovered - resuming this turn${attempt}` : `a daemon restart interrupted this turn - resuming${attempt}`,
235
+ prompt: deferred
236
+ ? `Your previous turn stopped early because every pooled account was at its usage limit; the pool has recovered. Pick up exactly where you left off and finish the task. If the work was already complete, just summarize the final state. The original request was:\n\n${turn.prompt}`
237
+ : `A tokenmaxxing serve daemon restart killed your previous turn mid-run. Pick up exactly where you left off and finish the task. If the work was already complete, just summarize the final state. The original request was:\n\n${turn.prompt}`,
217
238
  sessionId: record.sessionId,
218
239
  marker,
219
240
  };
@@ -1,11 +1,16 @@
1
1
  // Maps Claude Agent SDK messages onto Chat SDK stream chunks so a relayed
2
2
  // Slack turn shows the agent's process natively: task cards for thinking and
3
3
  // tool calls (pending -> in_progress -> complete/error), streamed text via
4
- // markdown_text, and a closing turn card with model/cost/duration. TodoWrite
5
- // is special-cased into one stable "Todos" checklist card per stream that
6
- // updates in place as items progress. Structured chunks render only when the
7
- // Slack app has the agent feature + assistant:write (the adapter drops them
8
- // gracefully otherwise); plain text streams either way.
4
+ // markdown_text, and a closing turn card with model/cost/duration. All task
5
+ // cards in a turn group into ONE collapsible Slack plan block (user ask
6
+ // 2026-07-20: "squash them into one dropdown"; the serve edge posts each turn
7
+ // as a StreamingPlan with groupTasks "plan", superseding the 2026-07-18
8
+ // separate-messages-around-tool-runs shape), so this mapper emits no segment
9
+ // breaks: a turn is one streamed message. TodoWrite is special-cased into one
10
+ // stable "Todos" checklist card per stream that updates in place as items
11
+ // progress. Structured chunks render only when the Slack app has the agent
12
+ // feature + assistant:write (the adapter drops them gracefully otherwise);
13
+ // plain text streams either way.
9
14
 
10
15
  import { truncate } from "es-toolkit/compat";
11
16
  import { z } from "zod";
@@ -58,21 +63,20 @@ const StreamMapStateSchema = z.object({
58
63
  * effect). */
59
64
  todoCards: z.record(z.string(), z.string()),
60
65
  thinkingCount: z.number(),
61
- /** reply text streamed since the last segment break. */
62
- textSinceBreak: z.boolean(),
66
+ /** non-whitespace reply text already streamed this turn: a later main text
67
+ * block gets a "\n\n" separator, restoring the visual break the removed
68
+ * per-tool message split used to provide (without it, post-tool prose
69
+ * glues onto pre-tool prose and a block opening with "## " or a ```
70
+ * fence loses its line-start position). */
71
+ textStreamed: z.boolean(),
63
72
  });
64
73
  export type StreamMapState = z.infer<typeof StreamMapStateSchema>;
65
74
 
66
75
  export function newStreamMapState(): StreamMapState {
67
- return { open: {}, toolTitles: {}, todoCards: {}, thinkingCount: 0, textSinceBreak: false };
76
+ return { open: {}, toolTitles: {}, todoCards: {}, thinkingCount: 0, textStreamed: false };
68
77
  }
69
78
 
70
- /** Emitted when a tool starts after streamed reply text: the bridge closes the
71
- * current Slack message there and posts the rest as a new one, mirroring how
72
- * an agent turn reads as separate messages around its tool runs. */
73
- export const SegmentBreakSchema = z.object({ type: z.literal("segment_break") });
74
-
75
- const StreamPartSchema = z.union([z.string(), z.custom<StreamChunk>(), SegmentBreakSchema]);
79
+ const StreamPartSchema = z.union([z.string(), z.custom<StreamChunk>()]);
76
80
  export type StreamPart = z.infer<typeof StreamPartSchema>;
77
81
 
78
82
  /** One human line out of a tool-input JSON blob; null when nothing fits. */
@@ -159,13 +163,14 @@ function resultText(content: z.infer<typeof ToolResultBlockSchema>["content"]):
159
163
 
160
164
  /**
161
165
  * Consume one SDK message, mutating state, and return the stream chunks it
162
- * produces (strings are streamed reply text; objects are native task cards).
163
- * Subagent events (parent_tool_use_id set) contribute their TOOL cards to the
164
- * timeline (user ask 2026-07-18: subagent activity shows as accordions like
165
- * tool calls) but never reply text, thinking cards, or segment breaks: a
166
- * subagent runs inside a top-level Task tool, so its churn decorates the
167
- * current message rather than reshaping it. Open blocks are keyed per stream
168
- * (parent + index) because concurrent subagent streams reuse index space.
166
+ * produces (strings are streamed reply text; objects are native task cards,
167
+ * all of which Slack folds into the turn's single plan block). Subagent
168
+ * events (parent_tool_use_id set) contribute their TOOL cards to that plan
169
+ * (user ask 2026-07-18: subagent activity shows alongside tool calls) but
170
+ * never reply text or thinking cards: a subagent runs inside a top-level Task
171
+ * tool, so its churn decorates the turn rather than reshaping it. Open blocks
172
+ * are keyed per stream (parent + index) because concurrent subagent streams
173
+ * reuse index space.
169
174
  */
170
175
  export function agentEventChunks(input: { state: StreamMapState; message: SDKMessage }): StreamPart[] {
171
176
  const { state, message } = input;
@@ -174,6 +179,12 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
174
179
  const event = message.event;
175
180
  if (event.type === "content_block_start") {
176
181
  const key = `${message.parent_tool_use_id ?? "main"}:${event.index}`;
182
+ if (event.content_block.type === "text" && isMain && state.textStreamed) {
183
+ // a new text block after streamed text opens on a fresh paragraph:
184
+ // the whole turn is one Slack message now, and without the break
185
+ // post-tool prose would glue onto pre-tool prose mid-line.
186
+ return ["\n\n"];
187
+ }
177
188
  if (event.content_block.type === "thinking" && isMain) {
178
189
  state.thinkingCount += 1;
179
190
  const id = `thinking-${state.thinkingCount}`;
@@ -184,9 +195,8 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
184
195
  const { id, name } = event.content_block;
185
196
  if (name === "TodoWrite") {
186
197
  // bookkeeping, not a real tool run: no card until the list arrives,
187
- // no segment break, and no toolTitles entry (its "Todos have been
188
- // modified" success result is noise; failures still surface via
189
- // todoCards below).
198
+ // and no toolTitles entry (its "Todos have been modified" success
199
+ // result is noise; failures still surface via todoCards below).
190
200
  const cardId = todoCardId(message.parent_tool_use_id);
191
201
  state.open[key] = { kind: "todo", id: cardId, title: "Todos", acc: "" };
192
202
  state.todoCards[id] = cardId;
@@ -195,12 +205,7 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
195
205
  const title = toolCardTitle(name);
196
206
  state.open[key] = { kind: "tool", id, title, acc: "" };
197
207
  state.toolTitles[id] = title;
198
- const card: StreamPart = { type: "task_update", id, title, status: "in_progress" };
199
- if (isMain && state.textSinceBreak) {
200
- state.textSinceBreak = false;
201
- return [{ type: "segment_break" }, card];
202
- }
203
- return [card];
208
+ return [{ type: "task_update", id, title, status: "in_progress" }];
204
209
  }
205
210
  return [];
206
211
  }
@@ -208,10 +213,9 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
208
213
  const open = state.open[`${message.parent_tool_use_id ?? "main"}:${event.index}`];
209
214
  if (event.delta.type === "text_delta") {
210
215
  if (!isMain) return [];
211
- // whitespace-only deltas must not count as reply text: a "\n\n"
212
- // before a tool call would otherwise break the segment and strand a
213
- // near-blank Slack message.
214
- if (event.delta.text.trim() !== "") state.textSinceBreak = true;
216
+ // whitespace-only deltas do not count: a text block carrying only
217
+ // "\n\n" must not earn the next block a doubled separator.
218
+ if (event.delta.text.trim() !== "") state.textStreamed = true;
215
219
  return [event.delta.text];
216
220
  }
217
221
  if (event.delta.type === "thinking_delta" && open) open.acc += event.delta.thinking;
@@ -274,8 +278,9 @@ export function agentEventChunks(input: { state: StreamMapState; message: SDKMes
274
278
  // emitting it, the output still posts, and having posted text suppresses
275
279
  // the result fallback so it never double-posts.
276
280
  if (message.content.trim() === "") return [];
277
- state.textSinceBreak = true;
278
- return [message.content];
281
+ const sep = state.textStreamed ? "\n\n" : "";
282
+ state.textStreamed = true;
283
+ return [sep + message.content];
279
284
  }
280
285
  if (message.type === "result") {
281
286
  const models = Object.keys(message.modelUsage).join(" ");