tokenmaxxing 1.159.0 → 1.160.0

package/DESIGN.md

@@ -62,7 +62,7 @@ The Stop hook's stdin has no usage data, but the **statusLine does** (`rate_limi
 4. Write `respawn/<session-id>` (atomic temp+rename) naming the target, with `waitUntil` now for a plain move or the assigned reset when the pool is depleted. No credential is written.
 
 ### 3.3 Move and depleted-pool pause (supervisor)
-The supervisor sees `respawn/<sid>`, SIGTERMs its child at the already-committed turn boundary, deletes the marker, resets the terminal, compacts the conversation when the marker asks for it and the old seat's record carries no live `enforcedUntil` (`compact: true` from the Stop and SessionStart hooks and from the supervisor's own seat watch, `false` from StopFailure: it runs `claude -p --resume <live-id> /compact` under the old seat's store, bounded at five minutes; the compaction counts only when a `compact_boundary` row lands in the transcript, because a refused compaction still exits 0, and a failure only prints why), and (when `waitUntil` is in the future) shows an interruptible countdown to the reset; then it relaunches `claude --resume <live-id>` (`--session-id <live-id>` when that id has no transcript yet, because `--resume` exits with `No conversation found` for it) with `CLAUDE_SECURESTORAGE_CONFIG_DIR` set to the target's store and rewrites the session's presence. The marker file is keyed by the launch id `<sid>`; the live id inside it is the session id the supervised process runs now, which `/clear` and an in-session `/resume` change. A hook adopts a differing stdin id as the live id (`current` in `sessions/<sid>.json`) only when its own process descends, through shell processes alone, from the presence record's pid; any other differing id is a nested session and writes nothing. The resumed process reads that store cold → runs on the target account, same conversation. The `--session-id` path is the exception: the conversation has no prior turn (a `/clear` with no turn since, or a move before the first turn), so it starts with no compaction and no first prompt, and in a stream-json session the relay replays the harness lines the killed child received: `user` lines and the `initialize` and session-state control requests in `REPLAYED_REQUESTS`; every other control request gets an error `control_response` on stdout, because a repeat could run its effect twice, and the `control_response` answers to the killed child's own requests are dropped. A first launch with `--resume <id>` takes the same path when the supervisor holds `sessions/<id>.json` for that id and no project directory holds its transcript. On the `--resume` path the relaunch submits a first prompt that names the move (and the compaction, when it landed). After a Stop move the previous turn had finished, so the prompt asks Claude to end the turn without restating it, and to relaunch any background Bash task, Monitor, or Workflow run that turn waited on, because the move ended it. After any other move the prompt asks Claude to continue the task, says the restart ended every background Bash task, Monitor, and Workflow run the session started, and asks Claude to relaunch what the task still needs; a SessionStart move is in that group because a prompt sent inside the marker poll window can already have started on the old seat. The conversation goes on without a keystroke: in a terminal session it is the positional prompt after the session id; in a stream-json session (`--input-format stream-json`, where the binary ignores a positional prompt) the supervisor owns stdin, relays the harness's lines to the child through a pipe, sends the new process the latest `initialize` and session-state control requests the harness sent, because the Agent SDK sends `initialize` once per query and it carries the appended system prompt, the hooks, and the SDK MCP servers, and then writes the prompt as the first `user` line. In a terminal session the supervisor's own notices go to stderr. In a stream-json session (`--output-format stream-json`, read from the effective flags like the relay) there is no terminal and no statusline, so the supervisor writes each notice (the compaction and its failure, the move, the depleted-pool wait with its resume time, the resume) to stdout as a `system` line of subtype `informational` and level `warning`, the shape Claude Code emits for its own print-mode notices, which the harness renders in the conversation while the stream stays parseable; the per-second countdown ticks stay terminal-only, and the supervisor writes only while no child is running. Before the first notice of a move it writes a `task_notification` line with `status: "stopped"` and `reason: "worker_restart"` for each task the old child started on stdout (`task_started`) and did not end there (a `task_notification`, or a `task_updated` to `completed`, `failed`, or `killed`), the line Claude Code writes when a restarted process finds a task orphaned, because the move ended those tasks and a harness that tracks them from the stream, as T3 Code does, otherwise shows them running until the session exits. The `SessionStart` hook (every source) re-runs the decision before the first turn and writes the same marker if the seat is already over its bar or the pool is depleted. The supervisor also watches its own seat every ten seconds while the child runs (account record plus tee, the same bars) and writes the marker itself when the seat is at or over a bar, so a move never depends on a hook firing: a harness session between turns whose background workflow burns the window, or a subagent refusal that stamped the wall, is moved within seconds. SIGTERM to the supervisor is forwarded to the child and the supervisor exits after it, so a harness that stops the wrapper (the Agent SDK does) never leaves an orphaned claude on the old seat. A marker whose `launchedAt` does not match the running child is discarded, and after a Ctrl-C on the countdown a later marker whose `waitUntil` falls inside the skipped wait is ignored. A marker that fails to parse is never deleted and never treated as absent: the supervisor stops the child, restores the terminal, and exits naming the marker path. In a stream-json session that fatal, the wrap-depth abort, the wrapper-chain abort, and a failed presence write also go to stdout as the same `informational`/`warning` line.
+The supervisor sees `respawn/<sid>`, SIGTERMs its child at the already-committed turn boundary, deletes the marker, resets the terminal, compacts the conversation when the marker asks for it, the old seat's record carries no live `enforcedUntil`, and `waitUntil` is not in the future (`compact: true` from the Stop and SessionStart hooks and from the supervisor's own seat watch, `false` from StopFailure: it runs `claude -p --resume <live-id> /compact` under the old seat's store, bounded at five minutes; the compaction counts only when a `compact_boundary` row lands in the transcript, because a refused compaction still exits 0, and a failure only prints why), and (when `waitUntil` is in the future, which means no pooled account has quota) skips the compaction and hands the session to the launch gate, which every launch goes through: with no account that has quota and no API key, it starts no child, claims the reset in `wait-queue.json`, shows an interruptible countdown to it, and places the session again; then it relaunches `claude --resume <live-id>` (`--session-id <live-id>` when that id has no transcript yet, because `--resume` exits with `No conversation found` for it) with `CLAUDE_SECURESTORAGE_CONFIG_DIR` set to the target's store and rewrites the session's presence. The marker file is keyed by the launch id `<sid>`; the live id inside it is the session id the supervised process runs now, which `/clear` and an in-session `/resume` change. A hook adopts a differing stdin id as the live id (`current` in `sessions/<sid>.json`) only when its own process descends, through shell processes alone, from the presence record's pid; any other differing id is a nested session and writes nothing. The resumed process reads that store cold → runs on the target account, same conversation. The `--session-id` path is the exception: the conversation has no prior turn (a `/clear` with no turn since, or a move before the first turn), so it starts with no compaction and no first prompt, and in a stream-json session the relay replays the harness lines the killed child received: `user` lines and the `initialize` and session-state control requests in `REPLAYED_REQUESTS`; every other control request gets an error `control_response` on stdout, because a repeat could run its effect twice, and the `control_response` answers to the killed child's own requests are dropped. A first launch with `--resume <id>` takes the same path when the supervisor holds `sessions/<id>.json` for that id and no project directory holds its transcript. On the `--resume` path the relaunch submits a first prompt that names the move (and the compaction, when it landed). After a Stop move the previous turn had finished, so the prompt asks Claude to end the turn without restating it, and to relaunch any background Bash task, Monitor, or Workflow run that turn waited on, because the move ended it. After any other move the prompt asks Claude to continue the task, says the restart ended every background Bash task, Monitor, and Workflow run the session started, and asks Claude to relaunch what the task still needs; a SessionStart move is in that group because a prompt sent inside the marker poll window can already have started on the old seat. The conversation goes on without a keystroke: in a terminal session it is the positional prompt after the session id; in a stream-json session (`--input-format stream-json`, where the binary ignores a positional prompt) the supervisor owns stdin, relays the harness's lines to the child through a pipe, sends the new process the latest `initialize` and session-state control requests the harness sent, because the Agent SDK sends `initialize` once per query and it carries the appended system prompt, the hooks, and the SDK MCP servers, and then writes the prompt as the first `user` line. In a terminal session the supervisor's own notices go to stderr. In a stream-json session (`--output-format stream-json`, read from the effective flags like the relay) there is no terminal and no statusline, so the supervisor writes each notice (the compaction and its failure, the move, the depleted-pool wait with its resume time, the resume) to stdout as a `system` line of subtype `informational` and level `warning`, the shape Claude Code emits for its own print-mode notices, which the harness renders in the conversation while the stream stays parseable; the per-second countdown ticks stay terminal-only, and the supervisor writes only while no child is running. Before the first notice of a move it writes a `task_notification` line with `status: "stopped"` and `reason: "worker_restart"` for each task the old child started on stdout (`task_started`) and did not end there (a `task_notification`, or a `task_updated` to `completed`, `failed`, or `killed`), the line Claude Code writes when a restarted process finds a task orphaned, because the move ended those tasks and a harness that tracks them from the stream, as T3 Code does, otherwise shows them running until the session exits. The `SessionStart` hook (every source) re-runs the decision before the first turn and writes the same marker if the seat is already over its bar or the pool is depleted. The supervisor also watches its own seat every ten seconds while the child runs (account record plus tee, the same bars) and writes the marker itself when the seat is at or over a bar, so a move never depends on a hook firing: a harness session between turns whose background workflow burns the window, or a subagent refusal that stamped the wall, is moved within seconds. SIGTERM to the supervisor is forwarded to the child and the supervisor exits after it, so a harness that stops the wrapper (the Agent SDK does) never leaves an orphaned claude on the old seat. A marker whose `launchedAt` does not match the running child is discarded, and after a Ctrl-C on the countdown a later marker whose `waitUntil` falls inside the skipped wait is ignored. A marker that fails to parse is never deleted and never treated as absent: the supervisor stops the child, restores the terminal, and exits naming the marker path. In a stream-json session that fatal, the wrap-depth abort, the wrapper-chain abort, and a failed presence write also go to stdout as the same `informational`/`warning` line.
 
 ### 3.4 Placement (supervisor, under the pool lock)
 1. Fold every account's tee into the index.
@@ -84,7 +84,7 @@ Each terminal ran the supervisor, so each has its own child `claude`, its own `-
 ---
 
 ## 5. Placement and rotation policy
-The [switching policy](docs/content/docs/switching.mdx) is one rule: under both bars, hold; at or over a bar, move to the usable account that ranks first; nothing usable, queue first come first served onto a reset target (at most eight waiters per account; overflow waits for the next reset with room) and wait for that reset. The same ranking places a session at launch; a launch onto a depleted pool takes the same target selection, counting running sessions as well as queued waiters, and starts immediately.
+The [switching policy](docs/content/docs/switching.mdx) is one rule: under both bars, hold; at or over a bar, move to the usable account that ranks first; nothing usable, queue first come first served onto a reset target (at most eight waiters per account; overflow waits for the next reset with room) and wait for that reset. The same ranking places a session at launch; a launch onto a depleted pool takes the same target selection, counting running sessions as well as queued waiters, and waits for that reset before it starts a `claude` process.
 
 - Ranking (decided by consult for #79): greatest remaining headroom to the session bar divided by the number of sessions after placement, `(bar - used) / (sessions + 1)`, then highest pace pressure (remaining percent of the binding gated per-model cap, or of the weekly aggregate for an account with no gated row, over time to its reset), then soonest weekly reset, then lowest weekly usage; unknown usage ranks last. Organization membership is not an input. Codex drops the headroom key, keeps the pace ordering and the weekly tie-breaks, and excludes accounts running in another codex session, because codex has no cross-process refresh lock and two sessions must never share a login.
 - Selection reads the cached figures only, after folding every account's tee. Each periodic check tick samples up to three accounts whose last sample attempt is oldest, skipping any attempted within the account's sample interval, over the direct no-spend usage read with the account's stored token. The interval is 15 minutes times the unused fraction of the bar closest to being reached, never under `policy.usagePollTtlMs`, and 15 minutes for an account at or over an aggregate bar or with no session or borrower on this host. Every sampler claims a read under the pool lock before it sends it, so concurrent hooks and `status` runs send an account one read. The windows are the session and weekly aggregates plus the `policy.switchModels` caps that are still under their bar, and an account with no stored aggregate gets `policy.usagePollTtlMs`. Every account is attempted about once per three ticks per pooled account, or once per its sample interval plus a tick when that is longer, and its cached figure is as fresh as its last successful attempt or its sessions' latest push. The pool lock covers only the reservation and the result write, never the usage GET. The usage endpoint rate limits each account on its own and names the time to come back in the `retry-after` of its HTTP 429; that time is stored as `usageRetryAt`, and no sampler (the tick, `status`, the seat's in-decision sample) reads the account before it. No sampler sends a read with an expired stored access token, because the endpoint answers it with 401 and, after a few, with the same 429. A sample that finds the token expired first runs `claude -p /usage --no-session-persistence --safe-mode` with the account's store set and a throwaway `CLAUDE_CONFIG_DIR`, so Claude Code refreshes the token under its own lock, and then reads with the new token; the child's output is not read. A refresh that leaves the token expired fails the sample with no read, and one that clears the store flags the account `needsReauth`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenmaxxing",
-  "version": "1.159.0",
+  "version": "1.160.0",
   "description": "Automatic Claude Code account switching: pool multiple accounts, start each session on its own account, and resume a session on a fresher account when quota fills.",
   "type": "module",
   "license": "MIT",

package/src/entries/supervisor.ts

@@ -17,7 +17,7 @@ import { teeObservation } from "../lib/sample.ts";
 import { childStdout, countdownWait, exitStatus, loopGuardTripped, raceMarkerOrExit, recordPresenceOrStop, relayStdout, runPassthrough, SEAT_POLL_MS, SEAT_RETRY_MS, type Say } from "../lib/supervise.ts";
 import { saveTermios } from "../lib/tty.ts";
 import { liveSessionId, loadSessionFlags, pruneStaleSessions, saveSessionFlags, writeRespawnMarker } from "../lib/sessions.ts";
-import { loadAccounts, loadConfig, readJsonFile, releaseWaitClaim, writeUsage } from "../lib/state.ts";
+import { loadAccounts, loadConfig, readJsonFile, releaseWaitClaim, replaceWaitClaim, writeUsage } from "../lib/state.ts";
 import { gatedFamilies, modelFromFlag, parseStreamLine, scrubCredEnv } from "../lib/usage.ts";
 import { CostLineSchema, CostStateRowSchema, JsonTextSchema, RespawnMarkerSchema, type Account, type ApiKey, type Config, type ModelInfo, type SessionCost, type UsageWindows } from "../lib/types.ts";
 import { errorMessage, log } from "../lib/log.ts";
@@ -592,8 +592,12 @@ export async function runSupervisor(argv: string[]): Promise<number> {
     const gate: MarkerGate = { launchedAt: Date.now(), overriddenUntil };
     const launched = await withLock(claudePool.lockFile, async () => {
       if (terminating) return null;
+      releaseClaimLocked(sid);
       const target = resolveLaunch(wanted, gate.launchedAt, model, refused);
-      if (target?.account && target.depletedUntil !== undefined) gate.overriddenUntil = Math.max(gate.overriddenUntil, target.depletedUntil);
+      if (target?.account && target.depletedUntil !== undefined && target.depletedUntil > gate.overriddenUntil) {
+        replaceWaitClaim({ sessionId: sid, accountId: target.account.id, at: gate.launchedAt, waitUntil: target.depletedUntil });
+        return { waitOn: target.account, waitUntil: target.depletedUntil };
+      }
       const picked = target?.account ?? null;
       const key = target?.apiKey ?? null;
       log("supervisor.launch", { sid, respawns, seat: picked?.id.slice(0, 8) ?? null, key: key?.id, args: launchArgs.join(" "), injected: firstLine !== null });
@@ -631,13 +635,17 @@ export async function runSupervisor(argv: string[]): Promise<number> {
           savedTermios,
         });
       }
-      releaseClaimLocked(sid);
       return { child: spawned, seat: picked, key, drain: drainStdout, open };
     });
     if (launched == null) {
       await releaseClaim();
       return 143;
     }
+    if (launched.waitOn !== undefined) {
+      log("supervisor.depleted_wait", { sid: sid.slice(0, 8), account: launched.waitOn.id.slice(0, 8), waitUntil: launched.waitUntil });
+      if (await countdownWait(launched.waitOn.label, launched.waitUntil, { stream, say })) overriddenUntil = launched.waitUntil;
+      continue;
+    }
 
     const { child: proc, seat, key, open } = launched;
     drainStdout = launched.drain;
@@ -682,8 +690,9 @@ export async function runSupervisor(argv: string[]): Promise<number> {
       let compacted = false;
       const compactKey = key != null && usableApiKey(key.id) != null ? key : null;
       const compactOn = seat?.label ?? (compactKey ? `API key ${compactKey.label}` : null);
-      if (m.compact && seat && resumable && walledUntil > Date.now()) {
-        log("supervisor.compact_skipped", { sid: m.sessionId.slice(0, 8), seat: seat.id.slice(0, 8), until: walledUntil });
+      const depleted = m.waitUntil > Date.now();
+      if (m.compact && resumable && (depleted || (seat && walledUntil > Date.now()))) {
+        log("supervisor.compact_skipped", { sid: m.sessionId.slice(0, 8), seat: seat?.id.slice(0, 8), until: depleted ? m.waitUntil : walledUntil, depleted: depleted || undefined });
       } else if (m.compact && compactOn != null && resumable) {
         say(`\n\x1b[36m↻ tokenmaxxing: compacting the conversation on ${compactOn} before the move...\x1b[0m\n`, `tokenmaxxing: compacting the conversation on ${compactOn} before the move.`);
         const baseEnv = { ...omit(scrubCredEnv(childEnv), ["TOKENMAXXING_SUPERVISED", "TOKENMAXXING_SESSION_ID", "TOKENMAXXING_LAUNCHED_AT", "TOKENMAXXING_MODEL", "TOKENMAXXING_REFUSED", "TOKENMAXXING_API_KEY_ID"]), TOKENMAXXING_PROBE: "1" };
@@ -710,10 +719,8 @@ export async function runSupervisor(argv: string[]): Promise<number> {
         if (!outcome.ok) say(`\x1b[33m   compaction did not land (${outcome.reason}) - resuming with the full context\x1b[0m\n`, `tokenmaxxing: compaction did not land; resuming with the full context. (${outcome.reason})`);
         compacted = outcome.ok;
       }
-      if (m.waitUntil > Date.now()) {
-        if (await countdownWait(label, m.waitUntil, { stream, say })) overriddenUntil = m.waitUntil;
-      } else say(`\n\x1b[36m↻ tokenmaxxing: moving to ${label} - resuming...\x1b[0m\n`, `tokenmaxxing: moving to ${label} and resuming.`);
-      wanted = m.apiKeyId != null ? { accountId: null, apiKeyId: m.apiKeyId } : { accountId: m.accountId, apiKeyId: null };
+      if (!depleted) say(`\n\x1b[36m↻ tokenmaxxing: moving to ${label} - resuming...\x1b[0m\n`, `tokenmaxxing: moving to ${label} and resuming.`);
+      wanted = depleted ? (key ? { accountId: null, apiKeyId: key.id } : null) : m.apiKeyId != null ? { accountId: null, apiKeyId: m.apiKeyId } : { accountId: m.accountId, apiKeyId: null };
       refused = m.refused ?? refused;
       saveSessionFlags(m.sessionId, persistable, process.cwd());
       const prompt = resumable ? resumePrompt({ compacted, origin: m.origin, onKey: m.apiKeyId != null }) : null;