claude-multiacc 2.0.41 → 2.0.43

package/docs/AUTORESUME.md

@@ -0,0 +1,321 @@
+# Auto-resume (interactive sessions in tmux)
+
+An interactive `claude` or `codex` session that stops on a usage limit, a failed login, an
+API error or (claude only) a crash is restarted automatically on another pooled account
+with headroom. It keeps the **same session id, directory and flags**, and you press no
+keys. This covers sessions launched from a shell inside **tmux**. Every other launch runs
+exactly as it did before.
+
+It is on by default. The kill switches:
+
+```bash
+touch ~/.claude-accounts/autoresume.off   # claude pool: new launches AND running watchers
+touch ~/.codex-accounts/autoresume.off    # codex pool:  new launches AND running watchers
+export CLAUDE_MULTIACC_AUTORESUME=0       # claude launches from this environment only
+export CODEX_MULTIACC_AUTORESUME=0        # codex launches from this environment only
+```
+
+- **The file** is checked by every running watcher on each one-second tick, so it also
+  covers sessions that are already open. A watcher that sees the file exits and leaves
+  its session alone.
+- **The variable** is read at launch, so it only affects launches from that environment.
+- **Instance pools** keep the file in their own root: `$CLAUDE_ACCOUNTS_ROOT/autoresume.off`.
+- **To turn it back on**, delete the file or unset the variable. A session whose watcher
+  already exited stays unsupervised; its next launch gets a new one.
+
+## Why nothing is ever typed into the TUI
+
+Neither client exits at a limit. Claude Code shows "continuing automatically at <reset>"
+and idles. Codex prints "You've hit your usage limit … try again at …" and waits. The
+manual fix was `/exit`, `claude --resume <id>` so the pool picks another account, then
+"continue", about 15 times a day.
+
+Sending those keys to the TUI is ruled out: Claude's limit menu can **add funds** or
+spend the account's one-shot `/limit-reset`. So the watcher only ever stops the process.
+Once the TUI has exited, it types into the **shell** that launched it.
+
+## How it works
+
+```
+zsh in a tmux pane
+ └─ claude (shim) ─ normal selection ─ gates pass? state + watcher ─ exec real claude (same pid)
+                                                     └─ python3 lib/autoresume.py watch (detached)
+watcher: tail this pid's transcript (codex: rollout) → classify the error → probe the pool
+         → write a single-use relaunch file → SIGTERM the TUI → wait for the shell prompt
+         → type " CLAUDE_MULTIACC_AR=<token>:<sid> /path/to/bin/claude" + Enter into the pane
+shell:   runs the shim again → token → marker on the old account → normal selection minus
+         the accounts this chain left → exec … --resume <sid> "<prompt>"   (with a new watcher)
+```
+
+1. **The launch is unchanged.** The shim selects exactly as before. It then `exec`s the
+   real binary with the same pid, stdin/stdout and exit-code passthrough.
+
+   Just before the exec, it writes a small state file to `<pool>/tmp/autoresume/`: the
+   pid, account, cwd, tmux pane and original argv. It then starts the watcher fully
+   detached.
+
+   Nothing on this path waits, runs python in the foreground or touches the network (a
+   relaunch adds one short trust step, see below). A failed check means no watcher, never
+   a failed launch.
+2. **The watcher reads what the client already writes.**
+   - **Claude:** `<acct>/sessions/<pid>.json` names the session. The watcher re-reads it
+     every tick, so it follows `/clear` and `/resume` inside the TUI. The transcript
+     under `projects/` records every API error.
+   - **Codex:** the rollout under `sessions/` is found in one of three ways, in order:
+     - the `resume` id;
+     - the native process's open files;
+     - only when the open files cannot be read: the newest rollout started since launch
+       in the same cwd, used only when exactly one matches.
+
+   In both cases:
+   - Only records written after this launch count, so an old rejection in a resumed
+     transcript never fires.
+   - Subagent (sidechain) records are ignored.
+3. **It waits for the error to settle.**
+   - **Grace period:** a verdict has to stand for 5 s.
+   - **Cancellation:** a message you type cancels it. So does the client's own
+     auto-continue producing a normal reply. For codex, a new user message,
+     `task_started` or `turn_aborted` cancels it.
+   - **Transient errors** wait longer: 30 s, then 60 s, then 120 s on repeats.
+   - **Model-limit and transient errors** also wait until no subagent transcript has
+     changed for 20 s, for up to 15 min. That background work may be healthy, or running
+     on another model.
+4. **It asks before it acts.** The watcher runs the shim in *probe mode*
+   (`CLAUDE_MULTIACC_AR_PROBE=1`). This is the same candidate loop and the same argv the
+   relaunch will use. It prints `pick=acct-NN tier=…` and never execs.
+   - **Limit, login and refusal errors** probe with the current account excluded. They
+     rotate only when the probe finds an **unlimited account other than the current
+     one**.
+   - **Otherwise the watcher holds.** Nothing is stopped, and Claude's own "continuing
+     automatically" keeps running. The probe repeats every 60 s for as long as the error
+     stands.
+   - **Transient errors** may continue on the same account, or on an all-limited
+     fallback that still serves.
+5. **Stop, then type.**
+   1. Before stopping anything, the watcher checks that:
+      - the pane's process is still the shell that launched the session, and that shell
+        is one the typed line works in (zsh, bash, sh, dash, ksh);
+      - the client is in the foreground, not suspended, and leads its own process group
+        (a job of an interactive shell);
+      - no other live claude holds the same session;
+      - the pane is not in `synchronize-panes`, and not in copy-mode after a `cancel`.
+
+      If any check fails, it gives up and the session keeps running.
+   2. It writes `r-<token>.relaunch` and `r-<token>.argv`.
+   3. It sends SIGTERM, then SIGKILL after 10 s to whatever is left of the client and the
+      descendants it recorded just before — only processes still in the client's own
+      process group, never the launching shell, and none at all if the recorded tree is
+      implausibly large (> 64). The shim's detached `limits` refresh is left alone.
+   4. It waits up to 15 s until those processes are gone and the pane has shown the shell
+      on two readings. If that does not happen, it removes the files and gives up. A
+      give-up after the stop logs `stopped=1 resume="claude --resume <sid>"` and shows the
+      same command in a tmux message that stays until a key is pressed.
+   5. It sends the keys:
+      - `send-keys -R` resets the terminal modes, which codex leaves dirty.
+      - `C-u` discards anything typed at the prompt.
+      - The relaunch line, then Enter.
+      - It waits up to 10 s for the shim to consume the token; if the line landed
+        somewhere else (a `read` after the session, say), it gives up as above.
+
+   The line is ` CLAUDE_MULTIACC_AR=<token>:<sid> [CLAUDE_ACCOUNTS_ROOT=<pool>] <shim>`
+   (codex: `CODEX_MULTIACC_AR`, `CODEX_ACCOUNTS_ROOT`).
+   - The leading space keeps it out of history where `HIST_IGNORE_SPACE` /
+     `HISTCONTROL=ignorespace` is set.
+   - `<shim>` is the full path of the shim that launched the session. It skips your
+     `claude` alias, and the relaunch reaches the same install whatever `PATH` says. The
+     original flags, alias-expanded ones included, are replayed from the saved argv.
+   - The pool root is typed only when the session did not use the default one
+     (`~/.claude-accounts` / `~/.codex-accounts`).
+   - The session id rides along with the token so a relaunch that cannot use its token
+     can still name the session.
+6. **The relaunched shim does the bookkeeping.**
+   1. It consumes the token. A token is single-use and valid for 10 minutes.
+   2. It writes the old account's marker with the pool's existing writers (see the tables
+      below).
+   3. It `cd`s to the session's directory.
+   4. It runs **normal selection**, with one addition: the accounts this chain of
+      relaunches left are skipped until their own reset. The selection policy is
+      otherwise untouched.
+   5. **claude:** it marks the directory trusted in the new account's `.claude.json`
+      (`projects[<dir>].hasTrustDialogAccepted`). Claude asks "Is this a project you
+      trust?" in a directory an account has never opened, even with
+      `--dangerously-skip-permissions`, and nobody is there to answer.
+   6. The new pick gets a watcher of its own.
+
+   A token that is missing, malformed or older than 10 minutes starts nothing. The shim
+   prints `claude-multiacc: auto-resume could not continue automatically (the resume token
+   expired). Resume with: claude --resume <sid>` (codex: `codex resume <sid>`; the hint is
+   left out when the session id is unknown) and exits 2. The typed line has no argv of its
+   own, so carrying on would open a fresh session in place of the stopped one.
+
+### The relaunched command
+
+The relaunch starts from the original argv:
+
+- The old session selectors (`-c`, `--continue`, `--resume <id>`, codex's
+  `resume <id>|--last`) and the original prompt are removed.
+- `--resume <sid> "<prompt>"` is appended. For codex the command becomes
+  `resume <options> <sid> "<prompt>"`.
+- An explicit `--model` from the original launch is kept. No model is added otherwise.
+
+The prompt is submitted automatically:
+
+> (claude-multiacc auto-resume) This session was restarted automatically on another
+> account because the previous account hit its usage limit. Continue the task from where
+> you left off; the user has not sent a new message. Anything that was running in the
+> background before the restart was stopped, so re-check it before relying on it, and do
+> not repeat work that is already done.
+
+- **The reason** follows the class.
+- **Transient and crash restarts** drop "on another account".
+- **To replace the whole text**, set `CLAUDE_MULTIACC_AUTORESUME_PROMPT` or
+  `CODEX_MULTIACC_AUTORESUME_PROMPT`.
+
+## What counts as what
+
+**Claude**, from the transcript's API-error records:
+
+| Class | Error record | Old account | Avoided for |
+| --- | --- | --- | --- |
+| quota | `rate_limit`, `quotaLimits.status: rejected`, `rateLimitType` `five_hour` / `seven_day` | `.limited` `client:<type>` until `resetsAt`, the marker the transcript scan writes | until that reset |
+| model | `rate_limit` for any other limit type (a per-model weekly), "reached your … limit" / "switch to another model", `model_requires_usage_credits` | nothing: the limit covers one model | 5 h |
+| auth | `authentication_failed` | `.expired`, the shim's client-reported auth park | 1 h |
+| blocked | `oauth_org_not_allowed`, `account_on_hold`, `billing_error`, `verification_required` | nothing: one transcript record only makes this chain avoid it | 6 h |
+| transient | `overloaded`, `server_error`, `unknown`, any other `rate_limit` | nothing | none, so the same account may continue |
+| crash | the process ended on its own after ≥ 60 s, leaving its session registry behind | nothing | none, so normal selection applies |
+| never | anything else (`invalid_request`, `max_output_tokens`, `model_not_found`, …) | nothing | not restarted; logged once |
+
+**Codex**, from `task_complete` events' `error.codex_error_info`:
+
+| Class | `codex_error_info` | Old account | Avoided for |
+| --- | --- | --- | --- |
+| quota | `usage_limit_exceeded`, `rate_limit_exceeded` | a 10-minute `error-cooldown` `.limited`, the same as `codex exec` auto-retry | until the limit resets (see below) |
+| auth | `unauthorized` | soft `.expired` (`reason=auth-error`) that lifts after 1 h | 1 h |
+| transient | `server_overloaded`, `internal_server_error`, `http_connection_failed`, `response_stream_connection_failed`, `response_stream_disconnected`, `response_too_many_failed_attempts` | nothing | none |
+| never | anything else | nothing | not restarted |
+
+For a codex quota error, the reset time comes from the first of these that is available:
+
+1. the last rate-limit snapshot in the rollout;
+2. the "try again at …" text;
+3. one hour from now.
+
+Codex differs from claude in two ways:
+
+- **A codex quota error never writes a sticky weekly `client:7d` marker.** Codex
+  `limits.json` is not shared between machines, so a peer Mac can redeem a reset credit in
+  the seconds between the error and the marker. A weekly marker would then strand a
+  refilled account for days. `usage_limit_exceeded` also comes from model-scoped limits.
+  The five-minute telemetry pass marks the account properly.
+- **Codex has no crash class.** It leaves nothing behind that tells a crash from a normal
+  exit.
+
+**Budgets.** A chain is a session and all of its relaunches. Each chain is allowed:
+
+- at most 20 relaunches;
+- 8 rotations per hour;
+- 3 transient restarts per hour;
+- 2 crash restarts per 10 minutes.
+
+Over budget, the watcher leaves the session alone.
+
+## What gets a watcher
+
+A launch gets a watcher only when all of these hold. Otherwise it is exactly today's
+launch.
+
+- **Auto-resume is on:** `*_MULTIACC_AUTORESUME` is not `0` (`false`, `no` and `off`
+  also turn it off), and there is no `autoresume.off`.
+- **Stdin and stdout are both terminals.**
+- **`$TMUX` and `$TMUX_PANE` are set.** This is the pane of the shell that launched it.
+- **The argv is one the relaunch can rebuild.**
+  - **claude:** `--dangerously-skip-permissions`, `--allow-dangerously-skip-permissions`,
+    `-c` / `--continue`, `-r` / `--resume <uuid>` (or `--resume=<uuid>`), `--model <m>`,
+    `--effort <e>`, `--permission-mode <m>`, and at most one prompt argument that
+    contains a space.
+  - **codex:** `--dangerously-bypass-approvals-and-sandbox`, `--yolo`, `-m` / `--model <m>`,
+    `resume <uuid>|--last`, and at most one prompt that contains a space.
+  - **Anything else means no watcher.** That includes `-p` (for codex, `-p` is
+    `--profile`), a subcommand, `--fork-session` and any single-word argument, which
+    could be a subcommand.
+- **The shim's path and the pool root need no shell quoting.** Both use only letters,
+  digits and `/._+-`, because the relaunch line is typed into a shell as it is.
+- **A usable `python3` is available,** along with the addon's `lib/autoresume.py`.
+  - `CLAUDE_MULTIACC_PYTHON` / `CODEX_MULTIACC_PYTHON` names a specific interpreter.
+  - On macOS, `/usr/bin/python3` counts only when the Command Line Tools are installed.
+    Without them it is a stub that opens an install dialog.
+- **The launch reaches the normal selected-account exec.** These get no watcher:
+  - pinned runs (`CLAUDE_ACCOUNT` / `CODEX_ACCOUNT`);
+  - passthrough (`CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `*_MULTIACC_DISABLE=1`);
+  - `-p` / `codex exec` runs, which have [auto-retry](../README.md) instead;
+  - the "nothing usable" stock fallback.
+
+## Logs and diagnosis
+
+Every step appends a line to the pool's `selection.log` in the form
+`<UTC> autoresume <event> k=v …`. These lines never contain transcript text, prompts or
+tokens.
+
+| Event | Written by | Meaning |
+| --- | --- | --- |
+| `watch` | watcher | supervision started for a pid |
+| `detect` | watcher | a classified error is pending |
+| `never` | watcher | an error auto-resume does not handle; ignored |
+| `hold` | watcher | no other account has headroom, so nothing is stopped |
+| `switch` | watcher | stopped the session and typed the relaunch (`from=`, `class=`, `sid=`, `depth=`) |
+| `crash` | watcher | claude ended unexpectedly; relaunching |
+| `giveup` | watcher | could not relaunch safely (`reason=` pane, shell, pgrp, holder, tree, relaunch, …); with `stopped=1` the session was stopped and `resume=` names the command to run |
+| `relaunch` | the relaunched shim | consumed the token (`chain=`, `from=`, `class=`, `depth=`) |
+
+Around these lines:
+
+- The new pick logs its usual selection line.
+- The old account's marker logs its usual `LIMITED` or `parked` line.
+- The second field is always the word `autoresume`. `claude-accounts status` and
+  `codex-accounts status` read the second field of every line as the picked account, so
+  an account id there would report picks that never happened.
+
+**Debug log.** Set `CLAUDE_MULTIACC_AUTORESUME_DEBUG=1` (codex:
+`CODEX_MULTIACC_AUTORESUME_DEBUG=1`) at launch, and that session's watcher writes a
+verbose `<pool>/tmp/autoresume/<pid>.log`. Files in that directory are pruned after two
+days.
+
+**Probe.** To see what a relaunch would pick right now, without launching anything:
+
+```bash
+CLAUDE_MULTIACC_AR_PROBE=1 claude --dangerously-skip-permissions
+# pick=acct-07 tier=eligible
+CLAUDE_MULTIACC_AR_PROBE=1 CLAUDE_MULTIACC_AR_AVOID=acct-07:1790000000 claude
+```
+
+- **`tier`** is one of:
+  - `eligible`: an unlimited account;
+  - `soft`: only the all-limited fallback;
+  - `hard`: only exhausted accounts are left;
+  - `none`: nothing is usable; exit 3.
+- **What the probe runs:** the same candidate loop as a launch, including marker upkeep
+  and its log lines, stopping before exec.
+- **Environment:** run it with no `CLAUDE_CONFIG_DIR`, `CLAUDE_ACCOUNT` or
+  `CLAUDE_MULTIACC_DISABLE` set. Those pass through or pin before selection is reached.
+
+## Limitations
+
+- **tmux only.** Outside tmux there is no safe place to type the relaunch, so those
+  sessions run unsupervised, as before.
+- **Background work stops.** Subagents, background shells and monitors in the stopped
+  session end with it, and the continuation prompt tells the model to re-check them. For
+  model-limit and transient errors, the watcher first waits up to 15 min for subagent
+  transcripts to go quiet.
+- **Only the pane's own shell is typed into.** The relaunch goes only to the shell that
+  launched the session. A session started from a nested shell or a script, or with
+  `exec claude`, gets a watcher that gives up instead of relaunching.
+- **Typeahead is discarded.** Text sitting at the shell prompt when the relaunch is typed
+  is cleared.
+- **Codex goals do not follow yet.** Codex keeps `/goal` state per account
+  (`goals_1.sqlite`), so set the goal again after a codex rotation. Claude restores a goal
+  from the transcript on resume.
+- **Out of scope for this version:**
+  - cross-provider continuation;
+  - proactive switching before a limit is hit;
+  - an agent that simply ends its turn early, which is not an error; use `/goal` for
+    that.

package/docs/CODEX.md

@@ -9,6 +9,7 @@ Codex uses the same machinery as the [Claude pool](../README.md), with these tra
 | `CLAUDE_CONFIG_DIR` per-account dirs | `CODEX_HOME` per-account dirs |
 | `.credentials.json` / macOS Keychain item (OAuth, machine-local) | `auth.json` (ChatGPT OAuth, machine-local) |
 | `claude -p` auto-retry | `codex exec` auto-retry |
+| interactive `claude` auto-resume in tmux | interactive `codex` auto-resume in tmux |
 | Anthropic OAuth usage and Fable buckets | Codex usage endpoint and per-model buckets |
 | `CLAUDE_*` pool controls | equivalent `CODEX_*` controls |
 
@@ -17,7 +18,8 @@ then the 30-point weekly-headroom band with random spread, ≥90% any-bucket
 exclusion, peers rotate), same marker semantics (`.limited` cooldowns, `.expired` parks with
 credential/policy scoping and soft expiry), same fail-open guarantees, same sync
 safety guards. The `codex` shim engages the buffered auto-retry only for
-`codex exec` runs with finite stdin, exactly like `-p` on the claude side.
+`codex exec` runs with finite stdin, exactly like `-p` on the claude side. Interactive
+`codex` in tmux gets [auto-resume](AUTORESUME.md) instead, like `claude`.
 
 Codex-specific notes:
 
@@ -65,6 +67,15 @@ Codex-specific notes:
   no subtraction or embedded usage summary substitutes for provider evidence.
   Zero means no available resets; missing fields mean the read was unavailable.
   Credit identifiers remain local and selection does not use these fields.
+- **Auto-resume never parks a codex account for the week.** A codex usage-limit error
+  in an interactive session writes only the 10-minute `error-cooldown`, the same as
+  `codex exec` auto-retry. It never writes a sticky `client:7d` marker. Codex
+  `limits.json` is not shared between machines, so a peer's reset redemption can land in
+  the gap, and `usage_limit_exceeded` also comes from model-scoped limits. The
+  five-minute limits pass marks the account from real telemetry.
+  - Turn auto-resume off with `touch ~/.codex-accounts/autoresume.off` (running sessions
+    too) or `CODEX_MULTIACC_AUTORESUME=0`.
+  - Codex `/goal` state is per account and does not follow a rotation yet.
 - **API-key logins are rejected** — ChatGPT subscription accounts only, matching the
   addon's no-API-keys rule.
 

package/docs/VERIFICATION.md

@@ -5,6 +5,7 @@
 ```bash
 tests/run-tests.sh            # sandboxed compatibility tests, no quota
 python3 tests/test_selector.py # unified selector contract
+python3 tests/test_autoresume.py # auto-resume classifier/relaunch unit tests (also run by run-tests.sh)
 npm run test:commands         # packed npm commands; run npm install --ignore-scripts first
 claude-accounts verify        # real matrix: `claude -p "reply OK"` per authed account
 claude-accounts verify --quick # auth presence/expiry only, no inference
@@ -49,6 +50,12 @@ repointed via `CLAUDE_BIN=/usr/local/bin/claude` and restarted healthy.
   before direct, TUI, or `--resume` work sees the 401. Run `claude-accounts verify` to
   check every token immediately; `claude-accounts expired` reports token-only accounts
   as `UNVERIFIED` until that proof exists.
+- **A session restarted itself, or did not** — that is [auto-resume](AUTORESUME.md).
+  `grep autoresume ~/.claude-accounts/selection.log` shows each step: `hold` means no
+  other account had headroom, and `giveup` gives the reason nothing was relaunched. A
+  launch outside tmux, with piped stdio, or with flags the relaunch cannot rebuild gets no
+  watcher at all. `touch ~/.claude-accounts/autoresume.off` turns it off, running
+  sessions included (codex: `~/.codex-accounts/autoresume.off`).
 - **Everything marked limited** — the shim still runs: the still-serving limited accounts
   go through the same two cuts (session gate, then strict best-weekly) and one is handed
   out anyway; check `selection.log` for `all-limited fallback=` lines.
@@ -70,7 +77,8 @@ repointed via `CLAUDE_BIN=/usr/local/bin/claude` and restarted healthy.
 
 - Tokens and credentials live as 0600 files under `~/.claude-accounts` (and
   `/root/.claude-accounts` on the server). Server compromise = account access.
-- `selection.log` records timestamps/account/cwd only — never prompt text.
+- `selection.log` records timestamps, accounts and cwd, and for auto-resume also
+  session ids and error classes. It never records prompt or transcript text.
 - Rotating multiple subscriptions to spread usage may be flagged by anti-abuse systems;
   accounts can be banned for limit circumvention. Known and accepted by the operator.
 - The usage endpoint is the internal one `/usage` consumes; if it changes shape, limit