@drakon-systems/multi-clawd 1.9.8 → 2.0.0

package/README.md

@@ -223,6 +223,12 @@ claude-cli/claude-fable-5        # main login
   stream-json and records per-window health to
   `~/.openclaw/state/multi-clawd/<account>.json`. Passthrough-first: a state
   write can never break a live turn.
+- 📡 **Live usage (v1.10)** — the pool also *asks* the provider what is left,
+  on a timer, using the login each account already holds. Both windows come
+  back as real percentages with reset times — including the 5-hour window the
+  stream never puts a number on — so hand-over happens at 85% of the session
+  window, not at the error. Passing 95% raises an operator alert through the
+  heartbeat; `multi-clawd usage` shows it on demand.
 - 🧰 **Full harness on every hop** — each backend is a genuine Claude Code
   subprocess: native tools, skills, MCP bridge, and native compaction all
   stay intact when failover steps across accounts.
@@ -247,7 +253,8 @@ claude-cli/claude-fable-5        # main login
   your chain: the refusal is swallowed before anything reaches you and the turn
   is re-spawned on a healthy account. Covers ongoing conversations too: the
   session is handed to the sibling as it stood before the refused attempt.
-  Secret-free sibling accounts, one retry.
+  One retry. Works for config-dir, native, and setup-token-file accounts; the
+  shim is handed paths only and reads a token file at the moment it retries.
 - 📟 **Operator alerts (v0.3)** — dead logins (probed every 15 min without
   spending quota), pool rotations, whole-pool exhaustion, and watchdog
   restarts surface through your agent's next heartbeat (e.g. straight into
@@ -255,8 +262,10 @@ claude-cli/claude-fable-5        # main login
 - 🩺 **`npm run doctor` (v0.3)** — one command that says whether a box is
   actually ready: config/manifest agreement (with the exact `--force`
   preflight strip plan), dist freshness, CLI presence, credential health
-  (values never printed), **which Claude login each account authenticates as
-  and which one the next turn will run on**, telemetry age, pool + sticky
+  judged on evidence — an unexpired token or a successful turn, never mere
+  presence (values never printed), a **credential split-store** warning when a
+  login's OAuth credential exists in more than one place, **which Claude login
+  each account authenticates as and which one the next turn will run on**, telemetry age, pool + sticky
   state, **whether every non-Claude chain rung still has a usable auth
   profile**, watchdog presence, optional `--probe` (one turn per account).
 - 🧯 **Self-healing config** — registration re-reads the resolved runtime
@@ -376,7 +385,7 @@ openclaw plugins install (Get-Location).Path
 **Or let your agent install it.** Running an OpenClaw assistant or Claude
 Code on the target machine already? Paste it this and go make coffee:
 
-> Read https://raw.githubusercontent.com/Drakon-Systems-Ltd/multi-clawd/v1.9.8/SETUP-AGENT.md
+> Read https://raw.githubusercontent.com/Drakon-Systems-Ltd/multi-clawd/v2.0.0/SETUP-AGENT.md
 > and follow it to set up multi-clawd on this machine. I own a second
 > Claude account — ask me when you need me to log in.
 
@@ -456,10 +465,17 @@ dir* — a separate Claude "app" — so the logins can never clobber each other.
 
    ```bash
    mkdir -p ~/.claw2 && chmod 700 ~/.claw2
-   CLAUDE_CONFIG_DIR=~/.claw2 claude setup-token > ~/.claw2/oauth-token   # log in as the 2nd account
-   chmod 600 ~/.claw2/oauth-token
+   CLAUDE_CONFIG_DIR=~/.claw2 claude setup-token   # log in as the 2nd account; it prints the token
+   install -m 600 /dev/null ~/.claw2/oauth-token
+   nano ~/.claw2/oauth-token   # paste ONLY the printed sk-ant-oat01-… token, on one line; save
    ```
 
+   Do not redirect `claude setup-token > file`: that captures the whole
+   screen (sign-in prompts and instructions) around the token. multi-clawd
+   refuses a token file holding anything but the single token line — the
+   account is skipped with an alert naming the file, and `multi-clawd doctor`
+   says exactly what to fix — rather than guess which part is the token.
+
    **Windows (native, PowerShell):**
 
    ```powershell
@@ -574,9 +590,11 @@ How it decides, per launch (all data from each account's live
 | no data / stale data | used — never rotate on missing evidence |
 | whole pool exhausted | home account anyway → real limit error → your chain drops provider |
 
-**Why the short-window rule exists (v1.7.2).** The threshold rule needs a
-utilization number, and Anthropic does not send one for the 5-hour session
-window — across every observation we hold from two accounts it arrives as a
+**Why the short-window rule exists (v1.7.2).** (v1.10's live usage poll now
+supplies the missing number — see below — so this rule is the fallback for
+hosts where polling is off or the account cannot be polled.) The threshold
+rule needs a utilization number, and Anthropic does not send one for the
+5-hour session window — across every observation we hold from two accounts it arrives as a
 bare status plus a reset time, while the weekly windows carry percentages. A
 rule that waits for a number therefore could never pre-empt the session
 limit: the pool would take the hit and rotate afterwards. So on hour-scoped
@@ -596,6 +614,89 @@ Notes:
   which runs on every launch on every turn path. Details in
   [`DESIGN.md`](./DESIGN.md).
 
+## Live usage: ask, don't wait to be told (v1.10)
+
+Everything above learns about quota from the stream — the `rate_limit_event`
+records the CLI emits at the top of a turn. That is reactive by construction:
+nothing is observed between turns, and the 5-hour session window arrives with
+a status but no percentage, so the pool could see "warning" but never "how
+close". v1.10 closes both gaps by reading each pooled account's **live usage
+from the provider** on a timer — the same figures the Claude CLI's own
+`/usage` command shows — with the OAuth session the CLI already holds.
+
+```jsonc
+"pool": {
+  "id": "clawd",
+  "accounts": ["claw1", "claw2"],
+  "utilizationThreshold": 0.85,     // hand over here (unchanged)
+  "usagePoll": {                    // v1.10 — on by default
+    "intervalMs": 120000,           // every 2 min (minimum 60000)
+    "warnThreshold": 0.95           // operator alert from here
+  }
+}
+```
+
+What a poll does, per account:
+
+| Provider says | Health file gets | Pool does |
+|---|---|---|
+| 5-hour at 40%, weekly at 20% | `usage:five_hour` / `usage:seven_day`, `allowed` with the number and reset | nothing — home account serves |
+| 5-hour at 86% | the same, `utilization 0.86` | **next launch rotates** to the spare (threshold rule, now with a real number on the session window) |
+| 5-hour at 96% | the same | rotates, **and raises an alert**: account, window, reset time, where launches now go |
+| 5-hour at 100% | `rejected` with the provider's reset | `exhausted` until that reset — for every model, before any turn is refused |
+| every account past 95% | — | **pool-wide alert**: at 100% the ladder or the host's chain takes over; soonest reset named |
+| model-scoped limit (e.g. one family at 97%) | — | alert naming the model; the shim's reactive capture still handles the refusal |
+
+Polled records live under their own keys (`usage:<window>`) so they never race
+the shim's `five_hour` / `seven_day` for the same slot — the shim's next bare
+`allowed` cannot erase the number the poll just wrote, and `explain` shows
+both ("5-hour (live) 40%"). The health rules are the existing ones; the poll
+only improves what they can see.
+
+Boundaries:
+
+- **Read-only on credentials.** The poll reads the access token from the
+  account's `.credentials.json` and sends it to the usage endpoint — nowhere
+  else. It never refreshes, rewrites, or copies a token; an expired one is a
+  skipped tick (the next CLI launch refreshes it, as it always has).
+- **Best-effort.** A failed poll changes nothing: the health file keeps what
+  the shim wrote, selection keeps working from it, and the failure is logged
+  once per transition, not per tick.
+- **Native and `configDir` logins only — not setup-token accounts.** A
+  `claude setup-token` token carries only the `user:inference` scope, and the
+  usage endpoint requires `user:profile`: it answers `403
+  oauth_scope_insufficient` (checked against the live endpoint; upstream
+  anthropics/claude-code#11985). Reading a credentials file from the
+  account's dir instead would read a different login than the one it runs
+  on. So token accounts (`oauthTokenFile` / `oauthTokenRef`) are not polled,
+  and `doctor`, `usage` and the gateway log say so. What that means in
+  practice for a pool of token accounts:
+  - **Still works:** rotation from stream telemetry — every turn's
+    `rate_limit_event` feeds the same threshold rule, the numberless 5-hour
+    warning rule (v1.7.2), and rejection-with-reset handling in the table
+    above; and the in-turn retry still catches a limit that lands mid-turn.
+  - **Missing:** an idle account is not re-measured between turns, the
+    5-hour window usually has no percentage (only a warning status), and
+    there is no 95% usage alert.
+  macOS native logins kept in the keychain (no `.credentials.json`) are
+  likewise not polled.
+- **Off switch:** `"usagePoll": { "enabled": false }`.
+
+On demand:
+
+```
+multi-clawd usage           # live figures per account + what the next launch would do
+multi-clawd usage --json
+```
+
+**What "seamless" means here.** Rotation is decided per launch, so a turn that
+starts after the poll has seen 85% simply runs on the other account — the user
+sees nothing. A turn already in flight when its account runs out is covered by
+the shim's in-turn retry (v1.9): the refusal is held back, the sibling is
+spawned, the conversation is handed over, and the turn completes there. The
+poll's job is to make that second path rare by moving work off an account
+before it reaches the cliff, and to tell you when every account is nearing one.
+
 ## Direct route: the pool for `anthropic/*` too (v1.9)
 
 OpenClaw reaches Claude two ways. The pool above drives **Claude Code CLI**
@@ -750,14 +851,17 @@ Housekeeping:
 - Prefer a secret reference (`oauthTokenRef`, v0.3) over a plaintext
   file; when a file is used, keep it `0600` (POSIX) or locked to your user
   with `icacls` (Windows).
-- **Credential resolution fails closed (v1.7.3).** An account that declares
-  `oauthTokenRef` or `oauthTokenFile` is authenticated by that token. If it
-  resolves to nothing — provider briefly unavailable, empty secret, truncated
-  file — and the account has no `configDir` to fall back on, the launch is
-  refused with `declares a token source but none resolved` rather than
-  allowed to proceed on the machine's default login and spend a different
-  account's quota under this account's name. Native accounts are exempt: the
-  default login *is* their credential.
+- **Credential resolution fails closed (v1.7.3; configDir no longer an
+  exception).** An account that declares `oauthTokenRef` or `oauthTokenFile`
+  is authenticated by that token and nothing else. If it resolves to nothing
+  — provider briefly unavailable, empty secret, truncated file, or a file
+  holding anything besides the single token line — the launch on that
+  account is refused with `declares a token source but none resolved`, even
+  when the account also has a `configDir` (whatever login sits in that dir
+  is not the credential you declared). In a pool the account is skipped, the
+  turn runs on a sibling, and an operator alert names the account and why;
+  only when no member resolves does the launch fail. Native accounts are
+  exempt: the default login *is* their credential.
 - Migrating a token file into a vault? `op read` (and most secret CLIs)
   append a trailing newline on output — resolution trims the resolved
   value (guaranteed in `token-resolution.ts`), so a file-vs-vault diff

package/dist/account-env.js

@@ -7,6 +7,18 @@ export function expandHomePath(p) {
         return resolve(homedir(), p.slice(2));
     return resolve(p);
 }
+export class AccountCredentialError extends Error {
+    name = "AccountCredentialError";
+}
+export function credentialFailureText(err) {
+    if (err instanceof AccountCredentialError)
+        return err.message.replace(/^\[multi-clawd\] /, "");
+    const code = err?.code;
+    if (typeof code === "string" && /^E[A-Z]+$/.test(code))
+        return `credential file unreadable (${code})`;
+    const cls = err instanceof Error ? err.constructor.name : typeof err;
+    return `credential resolution failed (${cls})`;
+}
 export function buildAccountChildEnv(account, token, stateFile) {
     const env = {
         MULTI_CLAWD_ACCOUNT_ID: account.id,
@@ -19,12 +31,38 @@ export function buildAccountChildEnv(account, token, stateFile) {
         env.CLAUDE_CONFIG_DIR = expandHomePath(account.configDir);
     }
     const declaresToken = Boolean(account.oauthTokenFile) || Boolean(account.oauthTokenRef);
-    if (!account.native && declaresToken && !usableToken && !account.configDir) {
-        throw new Error(`[multi-clawd] account "${account.id}" declares a token source but none resolved — ` +
-            `refusing to launch on the default login`);
+    if (!account.native && declaresToken && !usableToken) {
+        throw new AccountCredentialError(`[multi-clawd] account "${account.id}" declares a token source but none resolved — ` +
+            `refusing to launch it on any other login`);
     }
     return env;
 }
+const TOKEN_LINE_RE = /^sk-ant-oat[0-9]{2,}-[A-Za-z0-9_-]+$/;
+const TOKEN_ANYWHERE_RE = /sk-ant-oat[0-9]+-/g;
+export function parseSetupTokenFile(raw, label = "token file") {
+    const lines = raw
+        .split(/\r?\n/)
+        .map((l) => l.trim())
+        .filter((l) => l.length > 0);
+    if (lines.length === 0)
+        return { error: `${label} is empty` };
+    if (lines.length === 1 && TOKEN_LINE_RE.test(lines[0]))
+        return { token: lines[0] };
+    const runs = lines.reduce((n, l) => n + (l.match(TOKEN_ANYWHERE_RE)?.length ?? 0), 0);
+    if (runs === 0) {
+        return {
+            error: `${label} does not contain a setup-token (expected one line: sk-ant-oat01- then letters, ` +
+                `digits, - or _)` +
+                (lines.length > 1 ? `; it has ${lines.length} lines of other text` : ""),
+        };
+    }
+    return {
+        error: `${label} has text besides the token (${lines.length} non-empty line(s), ${runs} token-like) — ` +
+            `it must hold ONLY the token. This is what \`claude setup-token > file\` produces: it captures the on-screen ` +
+            `instructions too. Edit the file down to the single sk-ant-oat01-… line (check it is not ` +
+            `wrapped across two lines), keep it chmod 600, and nothing else is needed.`,
+    };
+}
 export function tokenFileModeWarning(path, mode) {
     const perms = mode & 0o777;
     if ((perms & 0o077) === 0)

package/dist/account-identity.js

@@ -1,3 +1,8 @@
+export function nativeDefaultConfigDir(env, homeDir) {
+    const dir = `${homeDir.replace(/\/+$/, "")}/.claude`;
+    const inherited = env.CLAUDE_CONFIG_DIR?.trim();
+    return inherited ? { dir, ignored: inherited } : { dir };
+}
 export function identityFilePaths(account, io) {
     if (account.oauthTokenFile || account.oauthTokenRef)
         return [];

package/dist/credential-state.js

@@ -5,6 +5,9 @@ import { parseStoredState, mergeHealthStates, clearCredentialFailure, } from "./
 export function healthStateFile(accountId) {
     return join(homedir(), ".openclaw", "state", "multi-clawd", `${accountId}.json`);
 }
+export function usageStateFile(accountId) {
+    return join(homedir(), ".openclaw", "state", "multi-clawd", `${accountId}.usage.json`);
+}
 export function clearAccountCredentialFailure(accountId) {
     const file = healthStateFile(accountId);
     let state;

package/dist/credential-store.js

@@ -0,0 +1,222 @@
+import { execFileSync } from "node:child_process";
+import { readFileSync } from "node:fs";
+import { homedir, userInfo } from "node:os";
+import { join, resolve } from "node:path";
+import { CLAUDE_KEYCHAIN_SERVICE, claudeKeychainAccount, keychainServiceForConfigDir, parseOauthCredential, } from "./login-health.js";
+export function parseKeychainDump(text) {
+    const items = [];
+    for (const block of text.split(/^keychain: /m)) {
+        if (!/^class: "genp"/m.test(block))
+            continue;
+        const attr = (name) => {
+            const m = block.match(new RegExp(`^\\s*"${name}"<[a-z]+>=(.*)$`, "m"));
+            if (!m)
+                return undefined;
+            const value = m[1].trim();
+            if (value === "<NULL>")
+                return undefined;
+            const hex = value.match(/^0x([0-9A-Fa-f]+)/);
+            if (hex)
+                return Buffer.from(hex[1], "hex").toString("utf8").replace(/\0+$/, "");
+            const quoted = value.match(/^"(.*)"$/);
+            return quoted ? quoted[1] : value;
+        };
+        const service = attr("svce");
+        if (service === undefined)
+            continue;
+        const mdat = attr("mdat")?.match(/^(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})Z/);
+        items.push({
+            service,
+            account: attr("acct") ?? "",
+            modifiedAt: mdat
+                ? Date.UTC(+mdat[1], +mdat[2] - 1, +mdat[3], +mdat[4], +mdat[5], +mdat[6])
+                : undefined,
+        });
+    }
+    return items;
+}
+export function credentialLocations(account, io) {
+    if (account.oauthTokenFile || account.oauthTokenRef)
+        return undefined;
+    if (account.native) {
+        return { service: CLAUDE_KEYCHAIN_SERVICE, file: io.expandHome("~/.claude/.credentials.json") };
+    }
+    if (!account.configDir)
+        return undefined;
+    const dir = io.expandHome(account.configDir).replace(/\/+$/, "");
+    return { service: keychainServiceForConfigDir(dir), file: `${dir}/.credentials.json` };
+}
+export function findCredentialCopies(account, io) {
+    const where = credentialLocations(account, io);
+    if (!where)
+        return { accountId: account.id, status: "n/a", copies: [], activeKnown: true };
+    const copies = [];
+    let cliItem = "absent";
+    let incomplete = false;
+    if (io.platform === "darwin") {
+        const listed = io.listKeychainItems();
+        incomplete = listed === undefined;
+        const items = (listed ?? []).filter((i) => i.service === where.service);
+        if (!items.some((i) => i.account === io.keychainAccount)) {
+            items.unshift({ service: where.service, account: io.keychainAccount });
+        }
+        for (const item of items) {
+            const read = io.readKeychainItem(where.service, item.account);
+            const isCli = item.account === io.keychainAccount;
+            if (isCli)
+                cliItem = read.status;
+            if (read.status === "absent")
+                continue;
+            const meta = read.status === "found" ? parseOauthCredential(read.data) : undefined;
+            copies.push({
+                kind: "keychain",
+                location: `keychain "${where.service}" acct=${item.account || "(empty)"}`,
+                keychainAccount: item.account,
+                expiresAt: meta?.expiresAt,
+                unreadable: read.status === "unreadable"
+                    ? "could not be read (Keychain locked, access refused or timed out)"
+                    : meta
+                        ? undefined
+                        : "not a Claude OAuth credential",
+                active: isCli && read.status === "found",
+                modifiedAt: item.modifiedAt,
+            });
+        }
+    }
+    let fileRaw;
+    let fileUnreadable = false;
+    try {
+        fileRaw = io.readFile(where.file);
+    }
+    catch (err) {
+        const code = err?.code;
+        fileUnreadable = code !== "ENOENT" && code !== "ENOTDIR";
+    }
+    if (fileRaw !== undefined || fileUnreadable) {
+        const meta = fileRaw === undefined ? undefined : parseOauthCredential(fileRaw);
+        copies.push({
+            kind: "file",
+            location: where.file,
+            expiresAt: meta?.expiresAt,
+            unreadable: fileUnreadable
+                ? "could not be read"
+                : meta
+                    ? undefined
+                    : "not a Claude OAuth credential",
+            active: cliItem === "absent",
+        });
+    }
+    return {
+        accountId: account.id,
+        status: copies.length > 1 ? "split" : copies.length === 1 ? "single" : "none",
+        copies,
+        activeKnown: cliItem !== "unreadable",
+        incomplete: incomplete || undefined,
+        service: where.service,
+        cliAccount: io.platform === "darwin" ? io.keychainAccount : undefined,
+        file: where.file,
+    };
+}
+function shellQuote(s) {
+    return `'${s.replace(/'/g, `'\\''`)}'`;
+}
+export function describeSplitStore(report, nowMs) {
+    const fmt = (c) => {
+        const expiry = c.expiresAt !== undefined
+            ? `${c.expiresAt > nowMs ? "expires" : "expired"} ${new Date(c.expiresAt).toISOString()}`
+            : (c.unreadable ?? "no expiry recorded");
+        const modified = c.modifiedAt !== undefined ? `, last written ${new Date(c.modifiedAt).toISOString()}` : "";
+        return `${c.location} — ${expiry}${modified}${c.active ? "  ← the copy the Claude CLI reads" : ""}`;
+    };
+    const active = report.copies.find((c) => c.active);
+    const latest = Math.max(...report.copies.map((c) => c.expiresAt ?? Number.NEGATIVE_INFINITY));
+    const newerElsewhere = active?.expiresAt !== undefined &&
+        Number.isFinite(latest) &&
+        latest > active.expiresAt;
+    const headline = `${report.accountId}: ${report.copies.length} copies of this login's OAuth credential — ` +
+        `refresh tokens rotate, so once the copies diverge only the one refreshed last can still ` +
+        `refresh, and a tool that reads another sees a login that is not the one in use` +
+        (!report.activeKnown
+            ? "; the Claude CLI's own Keychain item could not be read, so which copy it uses is unknown"
+            : newerElsewhere
+                ? "; a NEWER copy exists than the one the Claude CLI reads"
+                : "");
+    if (!report.activeKnown) {
+        return {
+            headline,
+            copies: report.copies.map(fmt),
+            remedy: `unlock the Keychain (or re-authenticate with \`multi-clawd login ${report.accountId}\`), ` +
+                `confirm with \`multi-clawd doctor --probe\`, then re-run doctor for cleanup steps. ` +
+                `Remove nothing until doctor can tell which copy the Claude CLI reads.`,
+        };
+    }
+    const strays = report.copies.filter((c) => !c.active && !(c.kind === "keychain" && c.keychainAccount === report.cliAccount));
+    const files = strays.filter((c) => c.kind === "file");
+    const items = strays.filter((c) => c.kind === "keychain");
+    const steps = [
+        `first prove the copy the Claude CLI reads with \`multi-clawd doctor --probe\` ` +
+            `(if it fails, re-authenticate with \`multi-clawd login ${report.accountId}\` and probe again)`,
+    ];
+    if (files.length > 0) {
+        steps.push(`after a passing probe, move ${files.map((c) => shellQuote(c.location)).join(" and ")} aside ` +
+            `(e.g. rename to .credentials.json.stale — reversible)`);
+    }
+    if (items.length > 0) {
+        steps.push(`delete a stray Keychain item only after a fresh \`multi-clawd login ${report.accountId}\` ` +
+            `and a passing probe (a probe can pass on an unexpired access token while its refresh ` +
+            `token is already dead): ` +
+            items
+                .map((c) => `\`security delete-generic-password -s ${shellQuote(report.service ?? "")} -a ${shellQuote(c.keychainAccount ?? "")}\``)
+                .join(", ") +
+            ` (irreversible)`);
+    }
+    const remedy = `${steps.join("; ")}. Never remove the copy the Claude CLI reads: another copy may hold ` +
+        `the only live refresh token until the CLI's own copy is proven.`;
+    return { headline, copies: report.copies.map(fmt), remedy };
+}
+export function expandHomeDir(p) {
+    if (p === "~")
+        return homedir();
+    if (p.startsWith("~/"))
+        return join(homedir(), p.slice(2));
+    return resolve(p);
+}
+const SECURITY_TIMEOUT_MS = 10_000;
+const SECURITY_ITEM_NOT_FOUND = 44;
+export function createSystemCredentialIo(deps = {}) {
+    const exec = deps.exec ?? ((file, args, options) => execFileSync(file, args, options));
+    return {
+        readFile: (p) => readFileSync(expandHomeDir(p), "utf8"),
+        expandHome: expandHomeDir,
+        keychainAccount: claudeKeychainAccount(deps.env ?? process.env, userInfo),
+        platform: deps.platform ?? process.platform,
+        readKeychainItem: (service, account) => {
+            try {
+                const data = exec("security", ["find-generic-password", "-s", service, "-a", account, "-w"], {
+                    encoding: "utf8",
+                    stdio: ["ignore", "pipe", "ignore"],
+                    timeout: SECURITY_TIMEOUT_MS,
+                });
+                return { status: "found", data: data.replace(/\n$/, "") };
+            }
+            catch (err) {
+                const status = err?.status;
+                return status === SECURITY_ITEM_NOT_FOUND ? { status: "absent" } : { status: "unreadable" };
+            }
+        },
+        listKeychainItems: () => {
+            try {
+                const out = exec("security", ["dump-keychain"], {
+                    encoding: "utf8",
+                    stdio: ["ignore", "pipe", "ignore"],
+                    timeout: SECURITY_TIMEOUT_MS * 3,
+                    maxBuffer: 64 * 1024 * 1024,
+                });
+                return parseKeychainDump(out);
+            }
+            catch {
+                return undefined;
+            }
+        },
+    };
+}

package/dist/direct-route.js

@@ -124,7 +124,8 @@ const VERDICT_RANK = {
     no_data: 0,
     near_limit: 1,
     exhausted: 2,
-    credential_failed: 3,
+    credential_unverified: 3,
+    credential_failed: 4,
 };
 export function planDirectOrder(params) {
     const { members } = params;

package/dist/explain-core.js

@@ -2,6 +2,8 @@ const WINDOW_LABELS = {
     five_hour: "5-hour",
     seven_day: "weekly",
     seven_day_overage_included: "weekly incl. overage",
+    "usage:five_hour": "5-hour (live)",
+    "usage:seven_day": "weekly (live)",
 };
 export function relativeUntil(ms, nowMs) {
     const mins = Math.max(0, Math.round((ms - nowMs) / 60000));
@@ -71,6 +73,7 @@ const VERDICT_WORDS = {
     near_limit: "NEAR ITS LIMIT — the pool will hand over before it hard-fails",
     exhausted: "EXHAUSTED",
     credential_failed: "LOGIN REJECTED — excluded from the pool until it is re-authenticated",
+    credential_unverified: "LOGIN UNPROVEN — last rejected and not proven since; used only as a re-test when nothing proven can serve",
 };
 export function renderExplanation(model) {
     const lines = [];

package/dist/health.js

@@ -43,14 +43,36 @@ export function credentialFailureFor(state, nowMs) {
     const credential = state?.credential;
     if (!credential || credential.status !== "failed")
         return undefined;
-    if (nowMs - credential.seenAt > CREDENTIAL_FAILED_TTL_MS)
-        return undefined;
+    const age = Math.max(0, nowMs - credential.seenAt);
+    const ageText = age < 90 * 60000 ? `${Math.round(age / 60000)}m` : `${Math.round(age / 3600000)}h`;
+    const detail = credential.reason ? `: ${credential.reason}` : "";
+    if (age <= CREDENTIAL_FAILED_TTL_MS) {
+        return {
+            verdict: "credential_failed",
+            resumeAt: credential.seenAt + CREDENTIAL_FAILED_TTL_MS,
+            reason: `login rejected by the Claude CLI ${ageText} ago${detail} — re-authenticate this account`,
+        };
+    }
     return {
-        verdict: "credential_failed",
-        resumeAt: credential.seenAt + CREDENTIAL_FAILED_TTL_MS,
-        reason: `login rejected by the Claude CLI ${Math.round((nowMs - credential.seenAt) / 60000)}m ago${credential.reason ? `: ${credential.reason}` : ""} — re-authenticate this account`,
+        verdict: "credential_unverified",
+        observedAt: credential.seenAt,
+        reason: `login rejected by the Claude CLI ${ageText} ago${detail} and not proven since — ` +
+            `re-authenticate, or prove it with \`multi-clawd doctor --probe\``,
     };
 }
+const POLL_WINDOW_PREFIX = "usage:";
+export function lastAuthSuccessAt(state) {
+    if (!state || state.credential?.status === "failed")
+        return undefined;
+    let latest = state.credential?.status === "ok" ? state.credential.seenAt : undefined;
+    for (const [key, w] of Object.entries(state.windows ?? {})) {
+        if (key.startsWith(POLL_WINDOW_PREFIX))
+            continue;
+        if (typeof w.seenAt === "number" && (latest === undefined || w.seenAt > latest))
+            latest = w.seenAt;
+    }
+    return latest;
+}
 export function classifyAccountHealth(state, options, nowMs, requestedModel) {
     const staleAfterMs = options.staleAfterMs ?? DEFAULT_STALE_AFTER_MS;
     const threshold = options.utilizationThreshold ?? DEFAULT_UTILIZATION_THRESHOLD;
@@ -219,6 +241,9 @@ export function fallbackPoolAccount(pool, nowMs) {
     const probe = nowMs === undefined ? undefined : overdueProbeAccount(pool, nowMs);
     if (probe)
         return probe;
+    const retest = pool.find((a) => a.verdict === "credential_unverified");
+    if (retest)
+        return retest.id;
     return (pool.find((a) => a.verdict !== "credential_failed") ?? pool[0]).id;
 }
 export function pickPoolAccountForLaunch(pool, nowMs) {