claude-multiacc 1.0.13 → 1.0.15

package/README.md

@@ -16,7 +16,7 @@ completely independent — separate manifests, credentials, telemetry, and logs
 either provider can be used, re-authenticated, or emptied without touching the other.
 
 Tested on: macOS (bash 3.2, zsh, Claude Code 2.1.207+, Codex CLI 0.147) and Ubuntu 24.04
-(bash 5.2). Test suite: `tests/run-tests.sh` (348 sandboxed tests, no network / no
+(bash 5.2). Test suite: `tests/run-tests.sh` (525 sandboxed tests, no network / no
 quota — passes on both platforms).
 
 ---
@@ -44,8 +44,11 @@ repo bin/ first on PATH (rc-file block)    /root/claude-multiacc/  (addon repo)
    selection.
 4. Among the accounts that remain valid **on this machine** (`.credentials.json` or
    `server.token` present) and not limit-excluded, pick the one with the **most remaining
-   headroom** (see below). Ties break randomly to spread load. `CLAUDE_SHIM_SELECT=random`
-   restores plain uniform-random selection.
+   headroom** (see below). On equal headroom the pool **rotates away from the account it
+   just handed out** and samples the rest at random — so quitting a session and starting
+   another one moves you along the pool, while a burst of parallel `claude -p` runs still
+   spreads (a strict "oldest first" order would send every member of the burst to the same
+   account). `CLAUDE_SHIM_SELECT=random` restores plain uniform-random selection.
 5. If every account is limit-excluded → the highest-headroom *authenticable* account
    anyway + a warning in `selection.log` (degraded beats down: the 100% rule).
 6. If nothing is usable at all → stock passthrough, with the reason in `selection.log`
@@ -98,9 +101,9 @@ The shim prints nothing, logs `timestamp account cwd` (never prompt text) to
 byte-identically. If anything is missing (no manifest, no accounts, unreadable state,
 even an unset `HOME`) it fails **open** into plain passthrough.
 
-**Limit-aware marking.** `claude-accounts limits` (every 60s via launchd on the Mac,
+**Limit-aware marking.** `claude-accounts limits` (every 15 min via launchd on the Mac,
 cron on the server, plus an opportunistic non-blocking kick from the shim when data is
->3 min stale) reads each account's own OAuth usage endpoint — the same data `/usage`
+>10 min stale) reads each account's own OAuth usage endpoint — the same data `/usage`
 shows. Every bucket the endpoint returns is tracked separately: `session`, `weekly_all`,
 and per-model scoped buckets — **`weekly_scoped:Fable` is its own bucket**. Any bucket
 ≥ 90% ⇒ the account gets a `.limited` marker (with the bucket's reset time) and leaves the
@@ -114,9 +117,38 @@ skipped, an all-legacy payload falls back to `five_hour`/`seven_day`, and a payl
 code cannot read at all degrades that one account (fail open), never the run.
 
 Telemetry failures never block work: no fresh data ⇒ account treated as available. The
-endpoint rate-limits per account, so the refresher skips accounts fetched in the last 45s
-and backs off exponentially (honoring `Retry-After`) on a 429 — `limits --force` overrides
-both. If an account's OAuth access token has been expired for a while (idle account,
+endpoint rate-limits per account, so the refresher skips accounts fetched in the last 4 min
+and backs off on **every** non-2xx, not just a 429 — honoring `Retry-After`, and parking
+for 6 h on a refusal the server marks `x-should-retry: false`. `limits --force` overrides
+all of it. The cadence is deliberately unhurried: several machines polling the same
+accounts every minute earns a 429 with `Retry-After: 3600`, and telemetry then goes stale
+for an hour at a time — which is exactly when every account starts scoring *neutral* and
+the picker loses its ability to tell them apart.
+
+> **The usage endpoint needs an OAuth login, not a setup token.** A portable
+> `server.token` (`sk-ant-oat01-…`) authenticates *inference* forever, but the usage
+> endpoint refuses it with `403 — OAuth token does not meet scope requirement
+> user:profile`: setup tokens are minted without that scope. So an account whose
+> `.credentials.json` grant has lapsed keeps working perfectly while going **permanently
+> dark for telemetry**, and a pool where that happens to every account ranks everything
+> neutral and picks at random. `claude-accounts status` says `RANKING IS BLIND` when the
+> pool is in that state, and the shim prints an hourly warning on a terminal. The fix is
+> a real sign-in on the machine that polls (`claude-accounts login <acct-NN>`); since
+> `limits.json` is one of the things `sync` pushes, only the **source** machine needs it —
+> the server and peers inherit the telemetry.
+
+**Two freshness windows, on purpose.** Ranking trusts telemetry for an hour
+(`CLAUDE_MULTIACC_STALE_AFTER`, default 3600 — matching the `Retry-After: 3600` the
+endpoint itself hands out, so a healthy pool is not "stale" for 45 minutes of every
+hour). The ≥90% *exclusion* keeps the tighter 15-minute window: ranking chooses between
+working accounts, while the cutoff declares one unusable, and an account reading 89% an
+hour ago may be well past 90% now. When nothing is in-window at all, a stale weekly
+reading is still used **if its bucket has not reset yet** (a weekly bucket only rises
+until then, so the number remains a true lower bound) — logged as `ranking=DEGRADED`.
+Only when even that is unavailable does selection fall back to neutral, logged as
+`ranking=BLIND`.
+
+If an account's OAuth access token has been expired for a while (idle account,
 nothing ran claude under it for hours), the refresher renews it directly via the OAuth
 **refresh-token grant** — the same endpoint and public client id Claude Code itself uses —
 and atomically persists the rotated credential (0600) back to that account's
@@ -127,6 +159,61 @@ account would lose to a busy-but-fresh one. Refresh failures fail open and back
 then says re-login is needed). Overrides: `CLAUDE_MULTIACC_TOKEN_URL`,
 `CLAUDE_MULTIACC_CLIENT_ID` (used by the sandboxed tests; defaults are correct for real use).
 
+**Limits the client reports itself.** The usage endpoint is not the only source of
+truth, and it is the one that fails exactly when it matters. Claude Code records every
+rejection in its session transcript:
+
+```json
+{"error":"rate_limit","apiErrorStatus":429,
+ "quotaLimits":{"status":"rejected","resetsAt":1787333400,"rateLimitType":"five_hour"}}
+```
+
+That record is free, offline, instant, and carries the **real** reset time the API handed
+out — so the shim reads it. This is what makes an **interactive** session take its own
+account out of the pool: auto-retry below only ever sees `-p` runs, so before this a 5-hour
+limit hit in a tmux session left no trace at all, and the next `claude` could walk straight
+back into the same dead account.
+
+Transcripts are not account-scoped (`<acct>/projects` is a shared symlink by design), so
+the session → account mapping comes from `<acct>/sessions/<pid>.json`, which the client
+maintains only while a run is alive. The shim spawns a short detached poll before `exec`
+(the pid survives `exec`, so it knows the filename) that records the session id — with the
+session's own start time — in `<acct>/.sessions-index`; selection then tail-scans those
+transcripts.
+
+Everything about the rule is built to under-claim rather than over-claim, because a
+wrongly parked account is worse than a missed limit:
+
+- **One owner per session.** `claude --continue` resumes the *same* session id under
+  whichever account the pool hands out next (only `--fork-session` mints a new one), and
+  the transcript is shared. Claiming an id releases it from every other account — and if
+  two accounts end up holding the same id anyway (a crossed race, corrupt state), the
+  ambiguity means neither of them answers for it.
+- **Only from the handover.** A rejection recorded before this account took the session
+  over stays with the account that earned it; a record that cannot be dated is not
+  attributed at all.
+- **Only a private session tree.** If `<acct>/sessions` — or the account dir itself, or a
+  directory nested inside it — is a symlink into somewhere shared, the runs in it prove
+  nothing about who spent the quota, so the scan stays off for that account entirely.
+- A rejection whose `resetsAt` has already passed is history and excludes nothing; a
+  hostile or unreadable index entry is skipped; `CLAUDE_MULTIACC_CLIENT_LIMITS=0` turns
+  the whole scan off.
+
+The resulting marker is tagged `reason=client-rate-limit` and outlives a clean telemetry
+pass while its window is still open — first-hand evidence beats a usage payload that
+disagrees. Cost is bounded: the three newest sessions per account, a tail read rather than
+a whole transcript, and a clean result memoized for 20s
+(`CLAUDE_MULTIACC_CLIENT_SCAN_TTL`), which on a three-account pool of large live
+transcripts is ~2 ms per run amortized.
+
+The codex shim does the same from rollout files
+(`$CODEX_HOME/sessions/<Y>/<M>/<D>/rollout-*.jsonl` carry
+`"rate_limits":{"primary":{"used_percent":…,"resets_at":…}}`, read newest-first), under the
+same ownership rule. Note that the **installed layout deliberately shares that tree**
+(`<acct>/sessions -> ~/.codex/sessions`, so `codex resume` finds every session), so on a
+default codex pool this scan stays off and the usage endpoint remains codex's only limit
+signal. Claude accounts keep a private `sessions/` registry, so it is fully active there.
+
 **Auto-retry** (`-p`/`--print` only, default on, `CLAUDE_SHIM_RETRY=0` disables): on an
 auth- or rate-limit-looking failure the shim retries once on a different account and
 marks the failed one — a **10-minute cooldown** for a rate limit (it heals on its own),
@@ -153,7 +240,7 @@ above about the claude pool holds for the codex pool with these translations:
 | `CLAUDE_ACCOUNT` / `CLAUDE_SHIM_RETRY` / `CLAUDE_MULTIACC_DISABLE` / … | `CODEX_ACCOUNT` / `CODEX_SHIM_RETRY` / `CODEX_MULTIACC_DISABLE` / … |
 
 Same selection rule (weekly headroom primary, session tiebreak, ≥90% any-bucket
-exclusion), same marker semantics (`.limited` cooldowns, `.expired` parks with
+exclusion, equal scores rotate), same marker semantics (`.limited` cooldowns, `.expired` parks with
 credential/policy scoping and soft expiry), same fail-open guarantees, same sync
 safety guards. The `codex` shim engages the buffered auto-retry only for
 `codex exec` runs with finite stdin, exactly like `-p` on the claude side.
@@ -252,7 +339,7 @@ What install does (all reversible, nothing else):
 - **macOS:** marked PATH block at the END of `~/.zshenv`, `~/.zprofile`, `~/.zshrc`
   (+ bash rc files if present) — end-of-file placement matters because those files
   re-prepend `~/.local/bin`; launchd agents `com.claude-multiacc.limits` +
-  `.codex-limits` (60s), `.health` + `.codex-health` (weekly Mon morning), and
+  `.codex-limits` (5m), `.health` + `.codex-health` (weekly Mon morning), and
   `.update` (daily 04:07). Warns if this Mac stores credentials in the Keychain
   (file-based credentials are required for per-dir logins).
 - **Linux (root):** PATH block in `~/.bashrc` + `/etc/profile.d/claude-multiacc.sh`,
@@ -368,8 +455,9 @@ manually via:
 claude-accounts sync      # and codex-accounts sync for the codex pool
 ```
 
-Targets are the manifest's `server`/`server_root`/`server_repo` (the Linux box) plus
-an optional `peers` array for additional machines (e.g. a second Mac):
+Targets are the resolved sync target — env override, else the manifest's
+`server`/`server_root`/`server_repo` (the Linux box), see *Pointing sync somewhere else*
+below — plus an optional `peers` array for additional machines (e.g. a second Mac):
 
 ```json
 "peers": [{"target": "gas@gas-mini", "root": "/Users/gas/.claude-accounts",
@@ -392,6 +480,174 @@ set, which is what prevents two machines from overwriting each other's manifests
 Make account changes on the source; sign-ins (`login`/`relogin`) still happen on
 whichever machine needs the credential.
 
+### Pointing sync somewhere else (or nowhere)
+
+The target is no longer baked in. In precedence order:
+
+1. `CLAUDE_MULTIACC_SYNC_TARGET` / `CODEX_MULTIACC_SYNC_TARGET` (or the provider-neutral
+   `MULTIACC_SYNC_TARGET`), plus `…_SYNC_ROOT` and `…_SYNC_REPO` — env, so a daemon can
+   point one instance at its own target without rewriting a manifest it does not own;
+2. the manifest's `server` / `server_root` / `server_repo` (what `install.sh --server`
+   wrote);
+3. the historical default — unchanged, so existing installs keep pushing where they did.
+
+A pool whose `sync-role` file says `replica` still never pushes, in any mode — the
+marker is honored and reported before anything else, and local-only mode below only
+narrows it further (a local pool pushes nothing whether or not it is a replica).
+
+The value **`none`** (also `local`, `off`, `disabled`, empty) selects **local-only** mode:
+there is no ssh target at all because something else — the app-robot panel and its runner
+daemon — distributes the pool. `sync` then validates the manifest, re-seeds account dirs,
+fixes permissions and pushes nowhere; mutations stop auto-pushing (and stop warning about
+a server they are not supposed to reach). One run can force it with `sync --no-server`,
+and a whole install can start that way:
+
+```bash
+./install.sh --no-server          # manifest server = none
+claude-accounts sync              # "sync ok (local-only: … nothing pushed)"
+claude-accounts sync --no-server  # same, whatever the manifest says
+```
+
+## Fleet integration (panels, daemons, multiple instances)
+
+Everything below is non-interactive and machine-readable: it is what a supervising panel
+(app-robot) or any other daemon needs to run a pool across a fleet of Macs.
+
+### `--json` on `list`, `status` and `limits`
+
+```bash
+claude-accounts list --json      # pool + accounts, no refresh
+claude-accounts status --json    # the same, plus last_picked and credential detail
+claude-accounts limits --json    # refresh usage first, then emit the same document
+codex-accounts  list --json      # identical schema, provider: "codex"
+```
+
+Stdout is the JSON document and nothing else. One shape for every verb and both
+providers (`lib/report.py`), so a consumer writes one parser:
+
+```jsonc
+{
+  "schema": "claude-multiacc/pool.v1",   // bumped only on a breaking change
+  "provider": "claude", "kind": "list", "generated_at": "…Z",
+  "machine": "mac", "host": "my-mini",
+  "pool": {
+    "root": "/Users/gas/.claude-accounts", "manifest": "…/accounts.json",
+    "threshold": 90,
+    "sync": {"mode": "server|local", "target": "root@…", "root": "…", "repo": "…",
+             "role": "source|replica", "peers": [...]}
+  },
+  "accounts": [{
+    "id": "acct-05", "email": "…", "home": "mac", "added_at": "…Z",
+    "home_dir": "/Users/gas/.claude-accounts/acct-05", "adopted": false,
+    "status": "active",            // active|limited|expired|blocked|missing|remote
+    "state": "ok", "label": "OK", "reason": "…", "fix": "…",
+    "selectable": true, "needs_login": false,
+    "credential_class": "portable", // portable|machine-local|none
+    "portable": true,
+    "credentials": {"oauth": true, "token": true, "oauth_expires_at": "…Z",
+                    "oauth_refresh_expires_at": "…Z", "token_minted_at": "…Z",
+                    "token_age_days": 39},
+    "limited": false, "limit_reset_at": null, "limit_detail": null,
+    "usage": {"fetched_at": "…Z", "age_seconds": 41, "source": "oauth",
+              "max_percent": 62, "weekly_percent": 62, "session_percent": 18,
+              "buckets": [{"name": "weekly_scoped:Fable", "group": "weekly",
+                           "percent": 62, "resets_at": "…Z"}]}
+  }],
+  "summary": {"total": 3, "active": 2, "limited": 1, "needs_login": 0,
+              "portable": 3, "selectable": 2},
+  "warnings": []
+}
+```
+
+`status` values mean exactly what the shim does: only `active` accounts are selected;
+`limited` authenticates but is parked until `limit_reset_at`; `expired`/`blocked`/
+`missing` need a human (`fix` says what); `remote` is not logged in here on purpose.
+A consumer that only distinguishes *usable / parked / needs-a-human* can read
+`selectable` and `needs_login` and ignore the rest; `state` keeps the raw audit verdict
+without the limit overlay. `limited` follows the shim's marker rule exactly — a marker
+whose reset time has passed does not count, an unreadable one does.
+
+### Which accounts are portable
+
+| Credential | Where it lives | Class | Can it be copied to another machine? |
+| --- | --- | --- | --- |
+| claude setup-token | `<acct>/server.token` (`sk-ant-oat…`, ~1 year, inference-only) | **portable** | **Yes** — this is the only credential the fleet distributes |
+| claude OAuth login | `<acct>/.credentials.json` | machine-local | **No.** Its refresh token rotates on every refresh; a second machine refreshing the same grant strands the first |
+| codex ChatGPT login | `<acct>/auth.json` | machine-local | **No** — same rotating-refresh-token reason. Codex has **no** portable credential type |
+
+Non-portable accounts are not a dead end: they show up as `credential_class:
+"machine-local"`, and the way to run one somewhere else is one interactive sign-in there
+(`claude-accounts login <id>`, `codex-accounts login <id>` — the codex device-code flow
+works over SSH). A claude account can be *converted* to the portable class with one
+`claude-accounts mint <id>` on a machine that is already signed in.
+
+### `export-credential` / `import-credential`
+
+```bash
+claude-accounts export-credential acct-05                 # blob to stdout
+claude-accounts export-credential acct-05 --out cred.json # …or to a 0600 file
+claude-accounts export-credential acct-05 --identity-only # registry entry, no secret
+cat cred.json | claude-accounts import-credential         # install it here
+claude-accounts import-credential --in cred.json --home mac --no-sync
+```
+
+`export-credential` is **read-only on the source pool** — nothing is refreshed, re-minted
+or marked, so exporting a live account cannot disturb it. It refuses anything that is not
+portable, with the reason and the fix, and its exit code is the contract a daemon branches
+on:
+
+| Exit | Meaning |
+| --- | --- |
+| 0 | blob written |
+| 2 | usage error / unknown account / malformed blob |
+| 3 | the credential is machine-local — copying it would break both machines |
+| 4 | the account has no credential material here |
+| 5 | credential present but unusable (corrupt, or an API key rather than a setup-token) |
+
+The blob is self-contained JSON (`format: "claude-multiacc/credential"`, version 1):
+identity metadata (`id`, `email`, `home`, `added_at`), where it came from, and — for
+class `portable` — the credential itself. **Treat a `portable` blob exactly like the
+credential it contains.**
+
+`import-credential` reads it on stdin (or `--in PATH`), creates the account dir and the
+manifest entry as needed, writes the credential 0600, and clears any dead-auth marker.
+It is idempotent **by email**: re-importing an account that is already in the pool
+refreshes it in place rather than adding a second entry, so a daemon can push the same
+pool to a machine over and over. It refuses a blob from the other provider and refuses an
+API key. It also refuses to land an email in a second slot, or to write into a slot that
+belongs to a different account — `--force` settles exactly those identity conflicts, and
+nothing else: an **adopted** (symlinked) account is refused unconditionally, `--force`
+included, because writing a credential through that link would put it outside the pool.
+An `identity` blob registers the account and says which sign-in it still needs. The
+imported account keeps the `added_at` it had on the source machine, so the same account
+reads identically everywhere.
+
+For codex, `export-credential` always refuses with exit 3 (there is nothing portable to
+export) and points at the device-code login; `--identity-only` is how a codex account list
+reaches another machine.
+
+### Instance-scoped pools
+
+`CLAUDE_ACCOUNTS_ROOT` / `CODEX_ACCOUNTS_ROOT` relocate a pool, so several app-robot
+instances can share one machine and user account without ever seeing each other's
+accounts. Both the CLIs **and the shims** resolve the same root, so a task launched with
+the env set runs under that instance's pool. (The older `…_ACCOUNTS_DIR` spelling still
+works; `…_ROOT` wins when both are set.)
+
+```bash
+export CLAUDE_ACCOUNTS_ROOT=/opt/app-robot/teamA/claude-accounts
+export CODEX_ACCOUNTS_ROOT=/opt/app-robot/teamA/codex-accounts
+./install.sh --no-server --instance teamA
+```
+
+An install whose roots are not the defaults is an **instance install**: its LaunchAgents
+(and cron lines on Linux) are labelled `com.claude-multiacc.<instance>.*` so instances
+never overwrite each other's agents, and they carry the pool roots in their environment.
+It deliberately does **not** touch the shell rc PATH block, `/etc/profile.d`, or the
+`/usr/local/bin` shims — one interactive PATH cannot serve two pools, and those belong to
+the default install. `--instance NAME` labels it explicitly; otherwise the label is
+derived from the pool root. Uninstalling an instance removes only that instance's agents.
+
 ## Pinning & env switches
 
 | Env | Effect |
@@ -399,17 +655,22 @@ whichever machine needs the credential.
 | `CLAUDE_ACCOUNT=acct-03` | pin this invocation to one account (wins over markers) |
 | `CLAUDE_CONFIG_DIR=...` | shim passes straight through (scripts can pin the old way) |
 | `CLAUDE_MULTIACC_DISABLE=1` | bypass selection entirely |
+| `CLAUDE_MULTIACC_CLIENT_LIMITS=0` | do not read the client's own rate-limit records (same for `CODEX_MULTIACC_CLIENT_LIMITS`) |
+| `CLAUDE_MULTIACC_CLIENT_SCAN_TTL=<s>` | how long a clean client-limit scan is memoized (default 20s; same for `CODEX_MULTIACC_CLIENT_SCAN_TTL`) |
 | `CLAUDE_SHIM_RETRY=0` | disable the `-p` auto-retry |
-| `CLAUDE_ACCOUNTS_DIR=...` | relocate the pool (used by the test suite) |
+| `CLAUDE_ACCOUNTS_ROOT=...` | relocate the pool — one isolated pool per app-robot instance (legacy spelling: `CLAUDE_ACCOUNTS_DIR`) |
+| `CLAUDE_MULTIACC_SYNC_TARGET=...` | sync target, overriding the manifest; `none` = local-only |
+| `CLAUDE_MULTIACC_SYNC_ROOT=... / _SYNC_REPO=...` | remote pool root / addon repo that goes with it |
 
 The codex shim honors the same switches spelled `CODEX_*`: `CODEX_ACCOUNT`,
 `CODEX_HOME` (passthrough), `CODEX_MULTIACC_DISABLE`, `CODEX_SHIM_RETRY`,
-`CODEX_SHIM_SELECT`, `CODEX_ACCOUNTS_DIR`, `CODEX_MULTIACC_THRESHOLD`.
+`CODEX_SHIM_SELECT`, `CODEX_ACCOUNTS_ROOT` (legacy `CODEX_ACCOUNTS_DIR`),
+`CODEX_MULTIACC_SYNC_TARGET`, `CODEX_MULTIACC_THRESHOLD`.
 
 ## Verification
 
 ```bash
-tests/run-tests.sh            # 221 sandboxed tests, no quota
+tests/run-tests.sh            # 525 sandboxed tests, no quota
 claude-accounts verify        # real matrix: `claude -p "reply OK"` per authed account
 claude-accounts verify --quick# auth presence/expiry only, no inference
 claude-accounts limits        # live per-bucket usage incl. the Fable bucket