@drakon-systems/multi-clawd 1.10.1 → 2.0.0

package/README.md

@@ -253,7 +253,8 @@ claude-cli/claude-fable-5        # main login
   your chain: the refusal is swallowed before anything reaches you and the turn
   is re-spawned on a healthy account. Covers ongoing conversations too: the
   session is handed to the sibling as it stood before the refused attempt.
-  Secret-free sibling accounts, one retry.
+  One retry. Works for config-dir, native, and setup-token-file accounts; the
+  shim is handed paths only and reads a token file at the moment it retries.
 - 📟 **Operator alerts (v0.3)** — dead logins (probed every 15 min without
   spending quota), pool rotations, whole-pool exhaustion, and watchdog
   restarts surface through your agent's next heartbeat (e.g. straight into
@@ -261,8 +262,10 @@ claude-cli/claude-fable-5        # main login
 - 🩺 **`npm run doctor` (v0.3)** — one command that says whether a box is
   actually ready: config/manifest agreement (with the exact `--force`
   preflight strip plan), dist freshness, CLI presence, credential health
-  (values never printed), **which Claude login each account authenticates as
-  and which one the next turn will run on**, telemetry age, pool + sticky
+  judged on evidence — an unexpired token or a successful turn, never mere
+  presence (values never printed), a **credential split-store** warning when a
+  login's OAuth credential exists in more than one place, **which Claude login
+  each account authenticates as and which one the next turn will run on**, telemetry age, pool + sticky
   state, **whether every non-Claude chain rung still has a usable auth
   profile**, watchdog presence, optional `--probe` (one turn per account).
 - 🧯 **Self-healing config** — registration re-reads the resolved runtime
@@ -382,7 +385,7 @@ openclaw plugins install (Get-Location).Path
 **Or let your agent install it.** Running an OpenClaw assistant or Claude
 Code on the target machine already? Paste it this and go make coffee:
 
-> Read https://raw.githubusercontent.com/Drakon-Systems-Ltd/multi-clawd/v1.10.1/SETUP-AGENT.md
+> Read https://raw.githubusercontent.com/Drakon-Systems-Ltd/multi-clawd/v2.0.0/SETUP-AGENT.md
 > and follow it to set up multi-clawd on this machine. I own a second
 > Claude account — ask me when you need me to log in.
 
@@ -462,10 +465,17 @@ dir* — a separate Claude "app" — so the logins can never clobber each other.
 
    ```bash
    mkdir -p ~/.claw2 && chmod 700 ~/.claw2
-   CLAUDE_CONFIG_DIR=~/.claw2 claude setup-token > ~/.claw2/oauth-token   # log in as the 2nd account
-   chmod 600 ~/.claw2/oauth-token
+   CLAUDE_CONFIG_DIR=~/.claw2 claude setup-token   # log in as the 2nd account; it prints the token
+   install -m 600 /dev/null ~/.claw2/oauth-token
+   nano ~/.claw2/oauth-token   # paste ONLY the printed sk-ant-oat01-… token, on one line; save
    ```
 
+   Do not redirect `claude setup-token > file`: that captures the whole
+   screen (sign-in prompts and instructions) around the token. multi-clawd
+   refuses a token file holding anything but the single token line — the
+   account is skipped with an alert naming the file, and `multi-clawd doctor`
+   says exactly what to fix — rather than guess which part is the token.
+
    **Windows (native, PowerShell):**
 
    ```powershell
@@ -652,10 +662,22 @@ Boundaries:
 - **Best-effort.** A failed poll changes nothing: the health file keeps what
   the shim wrote, selection keeps working from it, and the failure is logged
   once per transition, not per tick.
-- **Native and `configDir` accounts only.** A token-based account
-  (`oauthTokenFile` / `oauthTokenRef`) runs in the default login dir on a
-  token of its own, so the credentials file there belongs to a different
-  account; those accounts keep stream telemetry only and `explain` says so.
+- **Native and `configDir` logins only — not setup-token accounts.** A
+  `claude setup-token` token carries only the `user:inference` scope, and the
+  usage endpoint requires `user:profile`: it answers `403
+  oauth_scope_insufficient` (checked against the live endpoint; upstream
+  anthropics/claude-code#11985). Reading a credentials file from the
+  account's dir instead would read a different login than the one it runs
+  on. So token accounts (`oauthTokenFile` / `oauthTokenRef`) are not polled,
+  and `doctor`, `usage` and the gateway log say so. What that means in
+  practice for a pool of token accounts:
+  - **Still works:** rotation from stream telemetry — every turn's
+    `rate_limit_event` feeds the same threshold rule, the numberless 5-hour
+    warning rule (v1.7.2), and rejection-with-reset handling in the table
+    above; and the in-turn retry still catches a limit that lands mid-turn.
+  - **Missing:** an idle account is not re-measured between turns, the
+    5-hour window usually has no percentage (only a warning status), and
+    there is no 95% usage alert.
   macOS native logins kept in the keychain (no `.credentials.json`) are
   likewise not polled.
 - **Off switch:** `"usagePoll": { "enabled": false }`.
@@ -829,14 +851,17 @@ Housekeeping:
 - Prefer a secret reference (`oauthTokenRef`, v0.3) over a plaintext
   file; when a file is used, keep it `0600` (POSIX) or locked to your user
   with `icacls` (Windows).
-- **Credential resolution fails closed (v1.7.3).** An account that declares
-  `oauthTokenRef` or `oauthTokenFile` is authenticated by that token. If it
-  resolves to nothing — provider briefly unavailable, empty secret, truncated
-  file — and the account has no `configDir` to fall back on, the launch is
-  refused with `declares a token source but none resolved` rather than
-  allowed to proceed on the machine's default login and spend a different
-  account's quota under this account's name. Native accounts are exempt: the
-  default login *is* their credential.
+- **Credential resolution fails closed (v1.7.3; configDir no longer an
+  exception).** An account that declares `oauthTokenRef` or `oauthTokenFile`
+  is authenticated by that token and nothing else. If it resolves to nothing
+  — provider briefly unavailable, empty secret, truncated file, or a file
+  holding anything besides the single token line — the launch on that
+  account is refused with `declares a token source but none resolved`, even
+  when the account also has a `configDir` (whatever login sits in that dir
+  is not the credential you declared). In a pool the account is skipped, the
+  turn runs on a sibling, and an operator alert names the account and why;
+  only when no member resolves does the launch fail. Native accounts are
+  exempt: the default login *is* their credential.
 - Migrating a token file into a vault? `op read` (and most secret CLIs)
   append a trailing newline on output — resolution trims the resolved
   value (guaranteed in `token-resolution.ts`), so a file-vs-vault diff

package/dist/account-env.js

@@ -7,6 +7,18 @@ export function expandHomePath(p) {
         return resolve(homedir(), p.slice(2));
     return resolve(p);
 }
+export class AccountCredentialError extends Error {
+    name = "AccountCredentialError";
+}
+export function credentialFailureText(err) {
+    if (err instanceof AccountCredentialError)
+        return err.message.replace(/^\[multi-clawd\] /, "");
+    const code = err?.code;
+    if (typeof code === "string" && /^E[A-Z]+$/.test(code))
+        return `credential file unreadable (${code})`;
+    const cls = err instanceof Error ? err.constructor.name : typeof err;
+    return `credential resolution failed (${cls})`;
+}
 export function buildAccountChildEnv(account, token, stateFile) {
     const env = {
         MULTI_CLAWD_ACCOUNT_ID: account.id,
@@ -19,12 +31,38 @@ export function buildAccountChildEnv(account, token, stateFile) {
         env.CLAUDE_CONFIG_DIR = expandHomePath(account.configDir);
     }
     const declaresToken = Boolean(account.oauthTokenFile) || Boolean(account.oauthTokenRef);
-    if (!account.native && declaresToken && !usableToken && !account.configDir) {
-        throw new Error(`[multi-clawd] account "${account.id}" declares a token source but none resolved — ` +
-            `refusing to launch on the default login`);
+    if (!account.native && declaresToken && !usableToken) {
+        throw new AccountCredentialError(`[multi-clawd] account "${account.id}" declares a token source but none resolved — ` +
+            `refusing to launch it on any other login`);
     }
     return env;
 }
+const TOKEN_LINE_RE = /^sk-ant-oat[0-9]{2,}-[A-Za-z0-9_-]+$/;
+const TOKEN_ANYWHERE_RE = /sk-ant-oat[0-9]+-/g;
+export function parseSetupTokenFile(raw, label = "token file") {
+    const lines = raw
+        .split(/\r?\n/)
+        .map((l) => l.trim())
+        .filter((l) => l.length > 0);
+    if (lines.length === 0)
+        return { error: `${label} is empty` };
+    if (lines.length === 1 && TOKEN_LINE_RE.test(lines[0]))
+        return { token: lines[0] };
+    const runs = lines.reduce((n, l) => n + (l.match(TOKEN_ANYWHERE_RE)?.length ?? 0), 0);
+    if (runs === 0) {
+        return {
+            error: `${label} does not contain a setup-token (expected one line: sk-ant-oat01- then letters, ` +
+                `digits, - or _)` +
+                (lines.length > 1 ? `; it has ${lines.length} lines of other text` : ""),
+        };
+    }
+    return {
+        error: `${label} has text besides the token (${lines.length} non-empty line(s), ${runs} token-like) — ` +
+            `it must hold ONLY the token. This is what \`claude setup-token > file\` produces: it captures the on-screen ` +
+            `instructions too. Edit the file down to the single sk-ant-oat01-… line (check it is not ` +
+            `wrapped across two lines), keep it chmod 600, and nothing else is needed.`,
+    };
+}
 export function tokenFileModeWarning(path, mode) {
     const perms = mode & 0o777;
     if ((perms & 0o077) === 0)

package/dist/account-identity.js

@@ -1,3 +1,8 @@
+export function nativeDefaultConfigDir(env, homeDir) {
+    const dir = `${homeDir.replace(/\/+$/, "")}/.claude`;
+    const inherited = env.CLAUDE_CONFIG_DIR?.trim();
+    return inherited ? { dir, ignored: inherited } : { dir };
+}
 export function identityFilePaths(account, io) {
     if (account.oauthTokenFile || account.oauthTokenRef)
         return [];

package/dist/credential-store.js

@@ -0,0 +1,222 @@
+import { execFileSync } from "node:child_process";
+import { readFileSync } from "node:fs";
+import { homedir, userInfo } from "node:os";
+import { join, resolve } from "node:path";
+import { CLAUDE_KEYCHAIN_SERVICE, claudeKeychainAccount, keychainServiceForConfigDir, parseOauthCredential, } from "./login-health.js";
+export function parseKeychainDump(text) {
+    const items = [];
+    for (const block of text.split(/^keychain: /m)) {
+        if (!/^class: "genp"/m.test(block))
+            continue;
+        const attr = (name) => {
+            const m = block.match(new RegExp(`^\\s*"${name}"<[a-z]+>=(.*)$`, "m"));
+            if (!m)
+                return undefined;
+            const value = m[1].trim();
+            if (value === "<NULL>")
+                return undefined;
+            const hex = value.match(/^0x([0-9A-Fa-f]+)/);
+            if (hex)
+                return Buffer.from(hex[1], "hex").toString("utf8").replace(/\0+$/, "");
+            const quoted = value.match(/^"(.*)"$/);
+            return quoted ? quoted[1] : value;
+        };
+        const service = attr("svce");
+        if (service === undefined)
+            continue;
+        const mdat = attr("mdat")?.match(/^(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})Z/);
+        items.push({
+            service,
+            account: attr("acct") ?? "",
+            modifiedAt: mdat
+                ? Date.UTC(+mdat[1], +mdat[2] - 1, +mdat[3], +mdat[4], +mdat[5], +mdat[6])
+                : undefined,
+        });
+    }
+    return items;
+}
+export function credentialLocations(account, io) {
+    if (account.oauthTokenFile || account.oauthTokenRef)
+        return undefined;
+    if (account.native) {
+        return { service: CLAUDE_KEYCHAIN_SERVICE, file: io.expandHome("~/.claude/.credentials.json") };
+    }
+    if (!account.configDir)
+        return undefined;
+    const dir = io.expandHome(account.configDir).replace(/\/+$/, "");
+    return { service: keychainServiceForConfigDir(dir), file: `${dir}/.credentials.json` };
+}
+export function findCredentialCopies(account, io) {
+    const where = credentialLocations(account, io);
+    if (!where)
+        return { accountId: account.id, status: "n/a", copies: [], activeKnown: true };
+    const copies = [];
+    let cliItem = "absent";
+    let incomplete = false;
+    if (io.platform === "darwin") {
+        const listed = io.listKeychainItems();
+        incomplete = listed === undefined;
+        const items = (listed ?? []).filter((i) => i.service === where.service);
+        if (!items.some((i) => i.account === io.keychainAccount)) {
+            items.unshift({ service: where.service, account: io.keychainAccount });
+        }
+        for (const item of items) {
+            const read = io.readKeychainItem(where.service, item.account);
+            const isCli = item.account === io.keychainAccount;
+            if (isCli)
+                cliItem = read.status;
+            if (read.status === "absent")
+                continue;
+            const meta = read.status === "found" ? parseOauthCredential(read.data) : undefined;
+            copies.push({
+                kind: "keychain",
+                location: `keychain "${where.service}" acct=${item.account || "(empty)"}`,
+                keychainAccount: item.account,
+                expiresAt: meta?.expiresAt,
+                unreadable: read.status === "unreadable"
+                    ? "could not be read (Keychain locked, access refused or timed out)"
+                    : meta
+                        ? undefined
+                        : "not a Claude OAuth credential",
+                active: isCli && read.status === "found",
+                modifiedAt: item.modifiedAt,
+            });
+        }
+    }
+    let fileRaw;
+    let fileUnreadable = false;
+    try {
+        fileRaw = io.readFile(where.file);
+    }
+    catch (err) {
+        const code = err?.code;
+        fileUnreadable = code !== "ENOENT" && code !== "ENOTDIR";
+    }
+    if (fileRaw !== undefined || fileUnreadable) {
+        const meta = fileRaw === undefined ? undefined : parseOauthCredential(fileRaw);
+        copies.push({
+            kind: "file",
+            location: where.file,
+            expiresAt: meta?.expiresAt,
+            unreadable: fileUnreadable
+                ? "could not be read"
+                : meta
+                    ? undefined
+                    : "not a Claude OAuth credential",
+            active: cliItem === "absent",
+        });
+    }
+    return {
+        accountId: account.id,
+        status: copies.length > 1 ? "split" : copies.length === 1 ? "single" : "none",
+        copies,
+        activeKnown: cliItem !== "unreadable",
+        incomplete: incomplete || undefined,
+        service: where.service,
+        cliAccount: io.platform === "darwin" ? io.keychainAccount : undefined,
+        file: where.file,
+    };
+}
+function shellQuote(s) {
+    return `'${s.replace(/'/g, `'\\''`)}'`;
+}
+export function describeSplitStore(report, nowMs) {
+    const fmt = (c) => {
+        const expiry = c.expiresAt !== undefined
+            ? `${c.expiresAt > nowMs ? "expires" : "expired"} ${new Date(c.expiresAt).toISOString()}`
+            : (c.unreadable ?? "no expiry recorded");
+        const modified = c.modifiedAt !== undefined ? `, last written ${new Date(c.modifiedAt).toISOString()}` : "";
+        return `${c.location} — ${expiry}${modified}${c.active ? "  ← the copy the Claude CLI reads" : ""}`;
+    };
+    const active = report.copies.find((c) => c.active);
+    const latest = Math.max(...report.copies.map((c) => c.expiresAt ?? Number.NEGATIVE_INFINITY));
+    const newerElsewhere = active?.expiresAt !== undefined &&
+        Number.isFinite(latest) &&
+        latest > active.expiresAt;
+    const headline = `${report.accountId}: ${report.copies.length} copies of this login's OAuth credential — ` +
+        `refresh tokens rotate, so once the copies diverge only the one refreshed last can still ` +
+        `refresh, and a tool that reads another sees a login that is not the one in use` +
+        (!report.activeKnown
+            ? "; the Claude CLI's own Keychain item could not be read, so which copy it uses is unknown"
+            : newerElsewhere
+                ? "; a NEWER copy exists than the one the Claude CLI reads"
+                : "");
+    if (!report.activeKnown) {
+        return {
+            headline,
+            copies: report.copies.map(fmt),
+            remedy: `unlock the Keychain (or re-authenticate with \`multi-clawd login ${report.accountId}\`), ` +
+                `confirm with \`multi-clawd doctor --probe\`, then re-run doctor for cleanup steps. ` +
+                `Remove nothing until doctor can tell which copy the Claude CLI reads.`,
+        };
+    }
+    const strays = report.copies.filter((c) => !c.active && !(c.kind === "keychain" && c.keychainAccount === report.cliAccount));
+    const files = strays.filter((c) => c.kind === "file");
+    const items = strays.filter((c) => c.kind === "keychain");
+    const steps = [
+        `first prove the copy the Claude CLI reads with \`multi-clawd doctor --probe\` ` +
+            `(if it fails, re-authenticate with \`multi-clawd login ${report.accountId}\` and probe again)`,
+    ];
+    if (files.length > 0) {
+        steps.push(`after a passing probe, move ${files.map((c) => shellQuote(c.location)).join(" and ")} aside ` +
+            `(e.g. rename to .credentials.json.stale — reversible)`);
+    }
+    if (items.length > 0) {
+        steps.push(`delete a stray Keychain item only after a fresh \`multi-clawd login ${report.accountId}\` ` +
+            `and a passing probe (a probe can pass on an unexpired access token while its refresh ` +
+            `token is already dead): ` +
+            items
+                .map((c) => `\`security delete-generic-password -s ${shellQuote(report.service ?? "")} -a ${shellQuote(c.keychainAccount ?? "")}\``)
+                .join(", ") +
+            ` (irreversible)`);
+    }
+    const remedy = `${steps.join("; ")}. Never remove the copy the Claude CLI reads: another copy may hold ` +
+        `the only live refresh token until the CLI's own copy is proven.`;
+    return { headline, copies: report.copies.map(fmt), remedy };
+}
+export function expandHomeDir(p) {
+    if (p === "~")
+        return homedir();
+    if (p.startsWith("~/"))
+        return join(homedir(), p.slice(2));
+    return resolve(p);
+}
+const SECURITY_TIMEOUT_MS = 10_000;
+const SECURITY_ITEM_NOT_FOUND = 44;
+export function createSystemCredentialIo(deps = {}) {
+    const exec = deps.exec ?? ((file, args, options) => execFileSync(file, args, options));
+    return {
+        readFile: (p) => readFileSync(expandHomeDir(p), "utf8"),
+        expandHome: expandHomeDir,
+        keychainAccount: claudeKeychainAccount(deps.env ?? process.env, userInfo),
+        platform: deps.platform ?? process.platform,
+        readKeychainItem: (service, account) => {
+            try {
+                const data = exec("security", ["find-generic-password", "-s", service, "-a", account, "-w"], {
+                    encoding: "utf8",
+                    stdio: ["ignore", "pipe", "ignore"],
+                    timeout: SECURITY_TIMEOUT_MS,
+                });
+                return { status: "found", data: data.replace(/\n$/, "") };
+            }
+            catch (err) {
+                const status = err?.status;
+                return status === SECURITY_ITEM_NOT_FOUND ? { status: "absent" } : { status: "unreadable" };
+            }
+        },
+        listKeychainItems: () => {
+            try {
+                const out = exec("security", ["dump-keychain"], {
+                    encoding: "utf8",
+                    stdio: ["ignore", "pipe", "ignore"],
+                    timeout: SECURITY_TIMEOUT_MS * 3,
+                    maxBuffer: 64 * 1024 * 1024,
+                });
+                return parseKeychainDump(out);
+            }
+            catch {
+                return undefined;
+            }
+        },
+    };
+}

package/dist/direct-route.js

@@ -124,7 +124,8 @@ const VERDICT_RANK = {
     no_data: 0,
     near_limit: 1,
     exhausted: 2,
-    credential_failed: 3,
+    credential_unverified: 3,
+    credential_failed: 4,
 };
 export function planDirectOrder(params) {
     const { members } = params;

package/dist/explain-core.js

@@ -73,6 +73,7 @@ const VERDICT_WORDS = {
     near_limit: "NEAR ITS LIMIT — the pool will hand over before it hard-fails",
     exhausted: "EXHAUSTED",
     credential_failed: "LOGIN REJECTED — excluded from the pool until it is re-authenticated",
+    credential_unverified: "LOGIN UNPROVEN — last rejected and not proven since; used only as a re-test when nothing proven can serve",
 };
 export function renderExplanation(model) {
     const lines = [];

package/dist/health.js

@@ -43,14 +43,36 @@ export function credentialFailureFor(state, nowMs) {
     const credential = state?.credential;
     if (!credential || credential.status !== "failed")
         return undefined;
-    if (nowMs - credential.seenAt > CREDENTIAL_FAILED_TTL_MS)
-        return undefined;
+    const age = Math.max(0, nowMs - credential.seenAt);
+    const ageText = age < 90 * 60000 ? `${Math.round(age / 60000)}m` : `${Math.round(age / 3600000)}h`;
+    const detail = credential.reason ? `: ${credential.reason}` : "";
+    if (age <= CREDENTIAL_FAILED_TTL_MS) {
+        return {
+            verdict: "credential_failed",
+            resumeAt: credential.seenAt + CREDENTIAL_FAILED_TTL_MS,
+            reason: `login rejected by the Claude CLI ${ageText} ago${detail} — re-authenticate this account`,
+        };
+    }
     return {
-        verdict: "credential_failed",
-        resumeAt: credential.seenAt + CREDENTIAL_FAILED_TTL_MS,
-        reason: `login rejected by the Claude CLI ${Math.round((nowMs - credential.seenAt) / 60000)}m ago${credential.reason ? `: ${credential.reason}` : ""} — re-authenticate this account`,
+        verdict: "credential_unverified",
+        observedAt: credential.seenAt,
+        reason: `login rejected by the Claude CLI ${ageText} ago${detail} and not proven since — ` +
+            `re-authenticate, or prove it with \`multi-clawd doctor --probe\``,
     };
 }
+const POLL_WINDOW_PREFIX = "usage:";
+export function lastAuthSuccessAt(state) {
+    if (!state || state.credential?.status === "failed")
+        return undefined;
+    let latest = state.credential?.status === "ok" ? state.credential.seenAt : undefined;
+    for (const [key, w] of Object.entries(state.windows ?? {})) {
+        if (key.startsWith(POLL_WINDOW_PREFIX))
+            continue;
+        if (typeof w.seenAt === "number" && (latest === undefined || w.seenAt > latest))
+            latest = w.seenAt;
+    }
+    return latest;
+}
 export function classifyAccountHealth(state, options, nowMs, requestedModel) {
     const staleAfterMs = options.staleAfterMs ?? DEFAULT_STALE_AFTER_MS;
     const threshold = options.utilizationThreshold ?? DEFAULT_UTILIZATION_THRESHOLD;
@@ -219,6 +241,9 @@ export function fallbackPoolAccount(pool, nowMs) {
     const probe = nowMs === undefined ? undefined : overdueProbeAccount(pool, nowMs);
     if (probe)
         return probe;
+    const retest = pool.find((a) => a.verdict === "credential_unverified");
+    if (retest)
+        return retest.id;
     return (pool.find((a) => a.verdict !== "credential_failed") ?? pool[0]).id;
 }
 export function pickPoolAccountForLaunch(pool, nowMs) {

package/dist/index.js

@@ -7,7 +7,7 @@ import { MODEL_ALIASES, buildCatalogEntries, canonicalModelId, isModernClaudeMod
 import { decideDegradation, matchesPin } from "./degrade.js";
 import { resolveExecMode, permissionModeArgs } from "./exec-policy.js";
 import { resolveBaseModelIds } from "./catalog-source.js";
-import { allCredentialFailed, classifyAccountHealth, overdueProbeAccount, } from "./health.js";
+import { allCredentialFailed, classifyAccountHealth, lastAuthSuccessAt, overdueProbeAccount, } from "./health.js";
 import { decideStickySelection } from "./sticky.js";
 import { mergeHealthStates, parseStoredState, recordCredentialFailure, } from "./shim-core.js";
 import { createTokenRefResolver, isSecretRefShape, } from "./token-resolution.js";
@@ -19,10 +19,10 @@ import { addAlert, alertKeysWithPrefix, clearAlert, pendingAlertText, } from "./
 import { healthStateFile, usageStateFile, clearAccountCredentialFailure } from "./credential-state.js";
 import { createUsagePollController, effectiveUsagePollInterval, effectiveUsageWarnThreshold, usageAlertPrefix, usagePoolAlertKey, } from "./usage-poll.js";
 export { healthStateFile, usageStateFile, clearAccountCredentialFailure };
-import { accountConfigDir, buildAccountChildEnv, tokenFileModeWarning, validateAccountTokenSources, } from "./account-env.js";
+import { accountConfigDir, AccountCredentialError, buildAccountChildEnv, credentialFailureText, parseSetupTokenFile, tokenFileModeWarning, validateAccountTokenSources, } from "./account-env.js";
 import { diffCatalogModels, formatNewModelNotice, } from "./model-currency.js";
-import { checkAccountCredential, keychainServiceForConfigDir, createRefProbeTracker, } from "./login-health.js";
-import { execFileSync } from "node:child_process";
+import { assessAccountCredential, createRefProbeTracker, } from "./login-health.js";
+import { createSystemCredentialIo } from "./credential-store.js";
 import { collectDirectMembers } from "./direct-route.js";
 import { createDirectOrderController, isValidAgentId } from "./direct-sync.js";
 import { createOpenclawRunner } from "./openclaw-runner.js";
@@ -156,21 +156,7 @@ function ingestAlertSpool() {
 }
 const LOGIN_PROBE_INTERVAL_MS = 15 * 60 * 1000;
 const LOGIN_PROBE_INITIAL_DELAY_MS = 45 * 1000;
-const realCredentialIo = {
-    readFile: (p) => readFileSync(expandHome(p), "utf8"),
-    keychainHasClaudeCredentials: () => keychainHasService("Claude Code-credentials"),
-    keychainHasClaudeCredentialsForDir: (dir) => keychainHasService(keychainServiceForConfigDir(expandHome(dir))),
-    platform: process.platform,
-};
-function keychainHasService(service) {
-    try {
-        execFileSync("security", ["find-generic-password", "-s", service], { stdio: "ignore" });
-        return true;
-    }
-    catch {
-        return false;
-    }
-}
+let realCredentialIo;
 const refProbeTrackers = new Map();
 const lastProbeStatus = new Map();
 function recordAccountCredentialFailure(accountId, reason, nowMs) {
@@ -195,7 +181,8 @@ function recordAccountCredentialFailure(accountId, reason, nowMs) {
 export async function runLoginHealthProbe(accounts, logger, deps = {}) {
     const now = deps.nowMs ?? Date.now();
     const resolver = deps.resolver ?? activeTokenResolver;
-    const io = deps.io ?? realCredentialIo;
+    const io = deps.io ?? (realCredentialIo ??= createSystemCredentialIo());
+    const readHealth = deps.readHealth ?? readHealthState;
     for (const account of accounts) {
         let status;
         let reason;
@@ -215,9 +202,12 @@ export async function runLoginHealthProbe(accounts, logger, deps = {}) {
             cause = outcome.cause;
         }
         else {
-            const check = checkAccountCredential(account, io);
+            const check = assessAccountCredential(account, io, {
+                proof: { lastSuccessAt: lastAuthSuccessAt(readHealth(account.id)) },
+                nowMs: now,
+            });
             status = check.status;
-            reason = check.reason;
+            reason = check.status === "ok" ? check.detail : check.reason;
         }
         const previous = lastProbeStatus.get(account.id);
         lastProbeStatus.set(account.id, status);
@@ -243,14 +233,30 @@ export async function runLoginHealthProbe(accounts, logger, deps = {}) {
             logger.error(`[multi-clawd] ${text}`);
             raiseAlert({ key: `login:${account.id}`, severity: "error", text });
         }
+        else if (status === "unverified" && previous !== "unverified") {
+            logger.info(`[multi-clawd] account "${account.id}" login unverified: ${reason ?? "unproven"}`);
+        }
+        else if (status === "unknown" && previous !== "unknown" && reason) {
+            logger.info(`[multi-clawd] account "${account.id}" login could not be checked: ${reason}`);
+        }
         else if (status === "degraded" && previous !== "degraded") {
             logger.info(`[multi-clawd] account "${account.id}" login degraded: ${reason ?? "resolver error"}`);
         }
+        else if (status === "ok" && previous === "unverified") {
+            logger.info(`[multi-clawd] account "${account.id}" login verified: ${reason ?? "ok"}`);
+        }
         else if (status === "ok" && (previous === "broken" || previous === "degraded")) {
             logger.info(`[multi-clawd] account "${account.id}" login recovered`);
             alertState = clearAlert(alertState, `login:${account.id}`);
             alertState = clearAlert(alertState, `login-resolver:${account.id}`);
         }
+        if (status === "ok") {
+            alertState = clearAlert(alertState, `login:${account.id}`);
+            alertState = clearAlert(alertState, `login-resolver:${account.id}`);
+        }
+        else if ((status === "unverified" || status === "unknown") && cause === undefined) {
+            alertState = clearAlert(alertState, `login:${account.id}`);
+        }
     }
 }
 export function startLoginHealthProbe(accounts, logger) {
@@ -290,7 +296,20 @@ function peekToken(account) {
     if (account.oauthTokenFile) {
         const path = expandHome(account.oauthTokenFile);
         warnIfTokenFileExposed(path);
-        return readFileSync(path, "utf8").trim();
+        let raw;
+        try {
+            raw = readFileSync(path, "utf8");
+        }
+        catch (err) {
+            const code = err.code;
+            throw new AccountCredentialError(`[multi-clawd] account "${account.id}": token file ${account.oauthTokenFile} unreadable` +
+                (typeof code === "string" ? ` (${code})` : ""));
+        }
+        const read = parseSetupTokenFile(raw, `token file ${account.oauthTokenFile}`);
+        if ("error" in read) {
+            throw new AccountCredentialError(`[multi-clawd] account "${account.id}": ${read.error}`);
+        }
+        return read.token;
     }
     if (isSecretRefShape(account.oauthTokenRef)) {
         return activeTokenResolver?.peek(account.oauthTokenRef);
@@ -450,13 +469,16 @@ export function buildRetryRoster(members, launchedId) {
     for (const member of members) {
         if (member.id === launchedId)
             continue;
-        if (member.oauthTokenFile || member.oauthTokenRef)
+        const tokenFile = !member.native && member.oauthTokenFile ? expandHome(member.oauthTokenFile) : undefined;
+        if (!member.native && !tokenFile && member.oauthTokenRef)
+            continue;
+        if (!member.native && !tokenFile && !member.configDir)
             continue;
         const stateFile = healthStateFile(member.id);
-        const env = buildAccountChildEnv(member, undefined, stateFile);
+        const env = buildAccountChildEnv({ ...member, oauthTokenFile: undefined, oauthTokenRef: undefined }, undefined, stateFile);
         delete env.MULTI_CLAWD_ACCOUNT_ID;
         delete env.MULTI_CLAWD_STATE_FILE;
-        roster.push({ id: member.id, stateFile, env });
+        roster.push({ id: member.id, stateFile, env, ...(tokenFile ? { tokenFile } : {}) });
     }
     return roster;
 }
@@ -716,6 +738,10 @@ export function registerPoolBackend(api, pool, accounts, registeredIds, execMode
                 `launching on ${decision.account} to re-test a rejection last seen ` +
                 `${Math.round((now - observedAt) / 3600000)}h ago`);
         }
+        else if (poolVerdicts.find((v) => v.id === decision.account)?.verdict === "credential_unverified") {
+            logger.info(`[multi-clawd] pool ${poolId}: no proven account can serve ${requestedModel} — ` +
+                `launching on ${decision.account} to re-test a login last rejected by the Claude CLI`);
+        }
         else if (decision.account !== previousAccount) {
             const home = verdicts[0];
             const line = decision.account === members[0].id
@@ -760,6 +786,13 @@ export function registerPoolBackend(api, pool, accounts, registeredIds, execMode
                     text: `pool ${poolId}: account "${v.id}" is excluded — ${v.health.reason ?? "its login was rejected by the Claude CLI"}. Fix with \`multi-clawd login ${v.id}\`.`,
                 });
             }
+            else if (v.health.verdict === "credential_unverified") {
+                raiseAlert({
+                    key,
+                    severity: "error",
+                    text: `pool ${poolId}: account "${v.id}" is unproven — ${v.health.reason ?? "its login was last rejected by the Claude CLI"}. Fix with \`multi-clawd login ${v.id}\`.`,
+                });
+            }
             else {
                 alertState = clearAlert(alertState, key);
             }
@@ -774,6 +807,7 @@ export function registerPoolBackend(api, pool, accounts, registeredIds, execMode
             try {
                 env = await buildAccountEnv(candidate, deps?.resolver);
                 launched = candidate;
+                alertState = clearAlert(alertState, `credential-unresolved:${candidate.id}`);
                 if (candidate.id !== chosen.id) {
                     logger.warn(`[multi-clawd] pool ${poolId}: ${chosen.id}'s credential did not resolve — ` +
                         `launching on ${candidate.id} instead (account not benched; secret provider may be down)`);
@@ -781,7 +815,14 @@ export function registerPoolBackend(api, pool, accounts, registeredIds, execMode
                 break;
             }
             catch (err) {
-                unresolved.push(`${candidate.id} (${err.message})`);
+                const why = credentialFailureText(err);
+                unresolved.push(`${candidate.id} (${why})`);
+                raiseAlert({
+                    key: `credential-unresolved:${candidate.id}`,
+                    severity: "error",
+                    text: `pool ${poolId}: account "${candidate.id}" was skipped — its declared credential did ` +
+                        `not resolve, so it will not launch on any other login (${why})`,
+                });
             }
         }
         if (!env) {
@@ -927,9 +968,11 @@ export function stopUsagePoll() {
     usageInitial = undefined;
     usageController = undefined;
 }
+export const USAGE_TOKEN_ACCOUNT_REASON = "setup-token login — the usage endpoint needs the user:profile scope and setup-tokens carry only " +
+    "user:inference, so usage comes from stream telemetry only";
 export function usagePollCredentialsFile(account) {
     if (account.oauthTokenFile || account.oauthTokenRef) {
-        return { reason: "token-based login — usage is read from its own stream telemetry only" };
+        return { reason: USAGE_TOKEN_ACCOUNT_REASON };
     }
     if (!account.native && !account.configDir) {
         return { reason: "no login dir to read credentials from" };

package/dist/login-health.js

@@ -1,8 +1,98 @@
+import { parseSetupTokenFile } from "./account-env.js";
 import { createHash } from "node:crypto";
+export const CLAUDE_KEYCHAIN_SERVICE = "Claude Code-credentials";
+export function claudeKeychainAccount(env, userInfo) {
+    try {
+        return env.USER || userInfo().username;
+    }
+    catch {
+        return "claude-code-user";
+    }
+}
 export function keychainServiceForConfigDir(absDir) {
     const hash = createHash("sha256").update(absDir).digest("hex").slice(0, 8);
     return `Claude Code-credentials-${hash}`;
 }
+export const AUTH_PROOF_FRESH_MS = 24 * 60 * 60 * 1000;
+export const ACCESS_TOKEN_EXPIRY_SKEW_MS = 60 * 1000;
+const MAX_PLAUSIBLE_EXPIRY_MS = 253402300799000;
+export function parseOauthCredential(raw) {
+    let text = raw.trim();
+    if (/^(?:[0-9a-f]{2})+$/i.test(text)) {
+        text = Buffer.from(text, "hex").toString("utf8").trim();
+    }
+    let parsed;
+    try {
+        parsed = JSON.parse(text);
+    }
+    catch {
+        return undefined;
+    }
+    const oauth = parsed?.claudeAiOauth;
+    if (!oauth || typeof oauth !== "object")
+        return undefined;
+    const o = oauth;
+    const present = (v) => typeof v === "string" && v.trim().length > 0;
+    let expiresAt;
+    if (typeof o.expiresAt === "number" && Number.isFinite(o.expiresAt) && o.expiresAt > 0) {
+        const ms = o.expiresAt < 1e12 ? o.expiresAt * 1000 : o.expiresAt;
+        if (ms < MAX_PLAUSIBLE_EXPIRY_MS)
+            expiresAt = ms;
+    }
+    return {
+        hasAccessToken: present(o.accessToken),
+        hasRefreshToken: present(o.refreshToken),
+        expiresAt,
+    };
+}
+function ago(ms) {
+    const min = Math.max(0, Math.round(ms / 60000));
+    return min < 90 ? `${min}m` : `${Math.round(min / 60)}h`;
+}
+export function judgeOauthCredential(meta, source, proof, nowMs) {
+    const base = { source, expiresAt: meta.expiresAt };
+    if (!meta.hasAccessToken) {
+        return { ...base, status: "broken", reason: `${source}: access token is blank` };
+    }
+    if (meta.expiresAt !== undefined && meta.expiresAt - ACCESS_TOKEN_EXPIRY_SKEW_MS > nowMs) {
+        return {
+            ...base,
+            status: "ok",
+            detail: `access token valid until ${new Date(meta.expiresAt).toISOString()}`,
+        };
+    }
+    if (meta.expiresAt === undefined) {
+        return {
+            ...base,
+            status: "unverified",
+            reason: `${source}: records no access-token expiry, so it cannot be judged at rest`,
+        };
+    }
+    const expiry = `access token expired ${new Date(meta.expiresAt).toISOString()}`;
+    const success = proof.lastSuccessAt;
+    if (success !== undefined &&
+        success <= nowMs &&
+        nowMs - success <= AUTH_PROOF_FRESH_MS &&
+        success >= meta.expiresAt) {
+        return {
+            ...base,
+            status: "ok",
+            detail: `${expiry}; a successful turn ${ago(nowMs - success)} ago proves the refresh works`,
+        };
+    }
+    if (!meta.hasRefreshToken) {
+        return {
+            ...base,
+            status: "broken",
+            reason: `${source}: ${expiry} and no refresh token is stored`,
+        };
+    }
+    return {
+        ...base,
+        status: "unverified",
+        reason: `${source}: ${expiry} and no successful turn since proves the refresh works`,
+    };
+}
 export function createRefProbeTracker(options = {}) {
     const deadAfterConsecutive = options.deadAfterConsecutive ?? 3;
     const deadAfterMs = options.deadAfterMs ?? 10 * 60 * 1000;
@@ -42,29 +132,40 @@ export function createRefProbeTracker(options = {}) {
         },
     };
 }
-function looksLikeSetupToken(value) {
-    return /^sk-ant-[a-z0-9]+-/.test(value.trim());
-}
-function checkCredentialsJson(io, dir) {
+function assessCredentialsJson(io, path, proof, nowMs) {
     let raw;
     try {
-        raw = io.readFile(`${dir}/.credentials.json`);
+        raw = io.readFile(path);
     }
     catch {
-        return { status: "broken", reason: `${dir}/.credentials.json unreadable` };
+        return undefined;
     }
-    try {
-        const parsed = JSON.parse(raw);
-        const token = parsed?.claudeAiOauth?.accessToken;
-        if (typeof token === "string" && token.trim().length > 0)
-            return { status: "ok" };
-        return { status: "broken", reason: `${dir}/.credentials.json access token is blank` };
+    const meta = parseOauthCredential(raw);
+    if (!meta) {
+        return { status: "broken", reason: `${path} holds no Claude OAuth credential`, source: path };
     }
-    catch {
-        return { status: "broken", reason: `${dir}/.credentials.json is not valid JSON` };
+    return judgeOauthCredential(meta, path, proof, nowMs);
+}
+function assessKeychain(io, service, proof, nowMs) {
+    const read = io.readKeychainItem(service, io.keychainAccount);
+    if (read.status === "absent")
+        return undefined;
+    const source = `keychain "${service}"`;
+    if (read.status === "unreadable") {
+        return {
+            status: "unknown",
+            reason: `${source} could not be read (Keychain locked, access refused or timed out) — not judged`,
+            source,
+        };
     }
+    const meta = parseOauthCredential(read.data);
+    if (!meta)
+        return { status: "broken", reason: `${source} holds no Claude OAuth credential`, source };
+    return judgeOauthCredential(meta, source, proof, nowMs);
 }
-export function checkAccountCredential(account, io) {
+export function assessAccountCredential(account, io, options = {}) {
+    const proof = options.proof ?? {};
+    const nowMs = options.nowMs ?? Date.now();
     if (account.oauthTokenFile) {
         let raw;
         try {
@@ -73,35 +174,31 @@ export function checkAccountCredential(account, io) {
         catch {
             return { status: "broken", reason: `${account.oauthTokenFile} unreadable` };
         }
-        if (looksLikeSetupToken(raw))
-            return { status: "ok" };
-        return {
-            status: "broken",
-            reason: `${account.oauthTokenFile} does not contain a setup-token`,
-        };
+        const read = parseSetupTokenFile(raw, account.oauthTokenFile);
+        if ("token" in read)
+            return { status: "ok", detail: "setup-token present" };
+        return { status: "broken", reason: read.error };
     }
     if (account.oauthTokenRef) {
         return { status: "unknown" };
     }
-    if (account.native) {
-        if (io.platform === "darwin") {
-            return io.keychainHasClaudeCredentials()
-                ? { status: "ok" }
-                : { status: "broken", reason: "keychain has no Claude Code credentials" };
-        }
-        return checkCredentialsJson(io, "~/.claude");
-    }
-    if (account.configDir) {
-        if (io.platform === "darwin" && io.keychainHasClaudeCredentialsForDir(account.configDir)) {
-            return { status: "ok" };
-        }
-        const file = checkCredentialsJson(io, account.configDir);
-        if (file.status === "ok" || io.platform !== "darwin")
-            return file;
-        return {
+    const dir = account.native ? "~/.claude" : account.configDir;
+    if (!dir)
+        return { status: "unknown" };
+    const path = `${dir.replace(/\/+$/, "")}/.credentials.json`;
+    if (io.platform !== "darwin") {
+        return (assessCredentialsJson(io, path, proof, nowMs) ?? {
             status: "broken",
-            reason: `keychain has no Claude Code credentials for ${account.configDir} and ${file.reason}`,
-        };
+            reason: `${path} unreadable`,
+            source: path,
+        });
     }
-    return { status: "unknown" };
+    const service = account.native
+        ? CLAUDE_KEYCHAIN_SERVICE
+        : keychainServiceForConfigDir(io.expandHome(dir));
+    return (assessKeychain(io, service, proof, nowMs) ??
+        assessCredentialsJson(io, path, proof, nowMs) ?? {
+        status: "broken",
+        reason: `keychain has no Claude Code credentials for ${account.native ? "the default login" : dir} and ${path} unreadable`,
+    });
 }

package/dist/retry-plan.js

@@ -1,7 +1,9 @@
 import { classifyAccountHealth, overdueProbeAccount, } from "./health.js";
+import { parseSetupTokenFile } from "./account-env.js";
 import { resumeSessionId } from "./session-handover.js";
 export const RETRY_ROSTER_ENV = "MULTI_CLAWD_RETRY_ACCOUNTS";
 const CREDENTIAL_ENV_KEYS = ["CLAUDE_CONFIG_DIR", "CLAUDE_CODE_OAUTH_TOKEN"];
+const ROSTER_ENV_KEYS = ["CLAUDE_CONFIG_DIR"];
 export function parseRetryRoster(raw) {
     if (!raw)
         return [];
@@ -26,11 +28,12 @@ export function parseRetryRoster(raw) {
         const env = {};
         if (e.env && typeof e.env === "object") {
             for (const [k, v] of Object.entries(e.env)) {
-                if (typeof v === "string")
+                if (typeof v === "string" && ROSTER_ENV_KEYS.includes(k))
                     env[k] = v;
             }
         }
-        out.push({ id: e.id, stateFile: e.stateFile, env });
+        const tokenFile = typeof e.tokenFile === "string" && e.tokenFile ? e.tokenFile : undefined;
+        out.push({ id: e.id, stateFile: e.stateFile, env, ...(tokenFile ? { tokenFile } : {}) });
     }
     return out;
 }
@@ -40,7 +43,7 @@ export function isResumeLaunch(argv) {
 }
 export function retryArming(argv, roster, opts = {}) {
     if (roster.length === 0) {
-        return { armed: false, reason: "no secret-free sibling account to retry onto" };
+        return { armed: false, reason: "no sibling account the shim can retry onto" };
     }
     if (isResumeLaunch(argv) && !opts.resumeReady) {
         return {
@@ -77,3 +80,35 @@ export function buildRetryEnv(base, account) {
     env[RETRY_ROSTER_ENV] = "";
     return env;
 }
+export function materializeRetryAccount(account, readFile) {
+    if (!account.tokenFile)
+        return { account };
+    let raw;
+    try {
+        raw = readFile(account.tokenFile);
+    }
+    catch (err) {
+        const code = err.code;
+        return { error: `token file unreadable${typeof code === "string" && /^E[A-Z]+$/.test(code) ? ` (${code})` : ""}` };
+    }
+    const read = parseSetupTokenFile(raw, "token file");
+    if ("error" in read)
+        return { error: read.error };
+    return {
+        account: { ...account, env: { ...account.env, CLAUDE_CODE_OAUTH_TOKEN: read.token } },
+    };
+}
+export function chooseLaunchableRetryAccount(params) {
+    let remaining = params.roster;
+    while (remaining.length > 0) {
+        const pick = chooseRetryAccount({ ...params, roster: remaining });
+        if (!pick)
+            return undefined;
+        const ready = params.materialize(pick);
+        if ("account" in ready)
+            return ready.account;
+        params.onSkip?.(pick.id, ready.error);
+        remaining = remaining.filter((a) => a !== pick);
+    }
+    return undefined;
+}

package/dist/shim-core.js

@@ -137,7 +137,9 @@ export function mergeHealthStates(disk, live, now, pruneAfterMs = PRUNE_AFTER_MS
     if (live.credential && (!credential || live.credential.seenAt >= credential.seenAt)) {
         credential = live.credential;
     }
-    if (now !== undefined && credential && now - credential.seenAt > pruneAfterMs) {
+    if (now !== undefined &&
+        credential?.status === "ok" &&
+        now - credential.seenAt > pruneAfterMs) {
         credential = undefined;
     }
     const updatedAt = Math.max(disk.updatedAt ?? 0, live.updatedAt ?? 0);

package/dist/shim.js

@@ -5,7 +5,7 @@ import { classifyStateReadFailure, clearCredentialFailure, clearModelRejection,
 import { rewriteModelArg } from "./degrade.js";
 import { parseModelLimitError, recordModelLimit } from "./shim-core.js";
 import { canonicalModelId } from "./models.js";
-import { buildRetryEnv, chooseRetryAccount, parseRetryRoster, retryArming, RETRY_ROSTER_ENV, } from "./retry-plan.js";
+import { buildRetryEnv, chooseLaunchableRetryAccount, materializeRetryAccount, parseRetryRoster, retryArming, RETRY_ROSTER_ENV, } from "./retry-plan.js";
 import { effectiveConfigDir, handoverForLaunch, handoverSnapshotTo, parseSessionDirs, SESSION_DIRS_ENV, snapshotResumeTranscript, } from "./session-handover.js";
 function resolveClaudeCommand() {
     const override = process.env.MULTI_CLAWD_CLAUDE_BIN;
@@ -95,6 +95,7 @@ if (modelOverride) {
         process.stderr.write(`[multi-clawd shim] ${note}\n`);
 }
 const retryRoster = parseRetryRoster(process.env[RETRY_ROSTER_ENV]);
+delete process.env[RETRY_ROSTER_ENV];
 const resumeSnapshot = retryRoster.length > 0 && parseSessionDirs(process.env[SESSION_DIRS_ENV]).length > 0
     ? snapshotResumeTranscript(childArgs, process.env)
     : undefined;
@@ -233,11 +234,13 @@ function attemptRetry() {
     if (!retryArmed || retryUsed)
         return false;
     const model = effectiveModelId();
-    const target = chooseRetryAccount({
+    const target = chooseLaunchableRetryAccount({
         roster: retryRoster,
         readState: readSiblingState,
         modelId: model,
         nowMs: Date.now(),
+        materialize: (account) => materializeRetryAccount(account, (path) => readFileSync(path, "utf8")),
+        onSkip: (id, reason) => process.stderr.write(`[multi-clawd shim] not retrying onto ${id}: its declared credential did not resolve (${reason})\n`),
     });
     if (!target) {
         process.stderr.write(`[multi-clawd shim] model limit on ${accountId} with no healthy sibling to retry onto — ` +

package/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "multi-clawd",
   "name": "multi-clawd",
-  "version": "1.10.1",
+  "version": "2.0.0",
   "description": "Register additional Claude Code logins (Max/Pro accounts) as first-class OpenClaw CLI backends for cross-account failover, keeping the full skills/MCP harness on every account.",
   "cliBackends": [
     "claw1",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@drakon-systems/multi-clawd",
-  "version": "1.10.1",
+  "version": "2.0.0",
   "description": "Multi-account Claude Code failover for OpenClaw — register additional Claude (Max/Pro) logins as first-class CLI backends and keep the full skills/MCP harness across every account. Also imports those accounts' setup tokens into Hermes Agent's Anthropic credential pool.",
   "type": "module",
   "license": "MIT",

package/scripts/cli.mjs

@@ -753,7 +753,7 @@ async function usageCommand(args) {
   for (const a of accounts) {
     const row = { id: a.id, label: a.label };
     if (a.oauthTokenFile || a.oauthTokenRef) {
-      row.skipped = "token-based login — not polled (stream telemetry only)";
+      row.skipped = "setup-token login — not polled: the usage endpoint needs the user:profile scope, setup-tokens carry only user:inference (stream telemetry only)";
     } else if (!a.native && !a.configDir) {
       row.skipped = "no login dir";
     } else {

package/scripts/doctor.mjs

@@ -6,7 +6,8 @@
  *   1. plugin install + manifest/config key agreement (the --force trap)
  *   2. compiled-artifact freshness (stale dist detection)
  *   3. claude CLI availability + PATH sanity
- *   4. per-account credential-source health + which Claude login each
+ *   4. per-account credential health (judged on expiry and proof, never on
+ *      presence), credential split-store detection, and which Claude login each
  *      account actually authenticates as (and whether two share one)
  *   5. per-account rate-limit telemetry (state files, age, windows)
  *   6. pool configuration, sticky state, and the account the next turn runs on
@@ -280,55 +281,73 @@ try {
 
 // ── 4. account credentials (values never printed) ──────────────────────────
 console.log("account credentials");
-const { checkAccountCredential, keychainServiceForConfigDir } = await importDist("login-health.js", [
-  "checkAccountCredential",
-  "keychainServiceForConfigDir",
-]);
-const { summarizeWindowUsage, classifyAccountHealth, overdueProbeAccount } = await importDist("health.js", [
-  "summarizeWindowUsage",
-  "classifyAccountHealth",
-  "overdueProbeAccount",
-]);
+// "cli": judging a credential source is doctor's own diagnostic (see the
+// identity import below). The store module carries the system I/O; both must
+// come from the same build so the io matches the judge.
+const { assessAccountCredential } = await importDist(
+  "login-health.js",
+  ["assessAccountCredential"],
+  "cli",
+);
+const { createSystemCredentialIo, findCredentialCopies, describeSplitStore } = await importDist(
+  "credential-store.js",
+  ["createSystemCredentialIo", "findCredentialCopies", "describeSplitStore"],
+  "cli",
+);
+const { summarizeWindowUsage, classifyAccountHealth, overdueProbeAccount, lastAuthSuccessAt } =
+  await importDist("health.js", [
+    "summarizeWindowUsage",
+    "classifyAccountHealth",
+    "overdueProbeAccount",
+    "lastAuthSuccessAt",
+  ]);
 // "cli", like the chain audits: resolving WHICH login backs an account is
 // doctor's own diagnostic, not a description of the running plugin's
 // behaviour. Asking the installed copy meant a fix to the resolver could not
 // take effect until the plugin itself was reinstalled — so the CLI reported a
 // bug it had already been taught not to make.
-const { resolveAccountIdentity, describeIdentity, findDuplicateLogins, maskEmail } =
-  await importDist(
-    "account-identity.js",
-    ["resolveAccountIdentity", "describeIdentity", "findDuplicateLogins", "maskEmail"],
-    "cli",
-  );
+const {
+  resolveAccountIdentity,
+  describeIdentity,
+  findDuplicateLogins,
+  maskEmail,
+  nativeDefaultConfigDir,
+} = await importDist(
+  "account-identity.js",
+  ["resolveAccountIdentity", "describeIdentity", "findDuplicateLogins", "maskEmail", "nativeDefaultConfigDir"],
+  "cli",
+);
 const { decideStickySelection } = await importDist("sticky.js", ["decideStickySelection"]);
-const io = {
-  readFile: (p) => readFileSync(expandHome(p), "utf8"),
-  keychainHasClaudeCredentials: () => keychainHasService("Claude Code-credentials"),
-  keychainHasClaudeCredentialsForDir: (dir) =>
-    keychainHasService(keychainServiceForConfigDir(expandHome(dir))),
-  platform: process.platform,
-};
-// Metadata-only probe (no `-w`): the secret is never read.
-function keychainHasService(service) {
-  try {
-    execFileSync("security", ["find-generic-password", "-s", service], { stdio: "ignore" });
-    return true;
-  } catch {
-    return false;
-  }
+// Keychain reads name the account the Claude CLI uses (`-a`), never just the
+// service. Token values are reduced to expiry metadata where they are read and
+// are never printed.
+const io = createSystemCredentialIo();
+// A native account's child runs with CLAUDE_CONFIG_DIR CLEARED (the pool strips
+// it before every launch), so its login lives under the real home whatever
+// this shell says. Doctor run inside an agent session served by the pool
+// inherits the serving account's dir; honouring it resolved "native" from that
+// account and reported two distinct logins as one.
+const nativeDir = nativeDefaultConfigDir(process.env, HOME);
+if (nativeDir.ignored) {
+  note(
+    `CLAUDE_CONFIG_DIR=${nativeDir.ignored} is set in this shell — ignored: a native account runs ` +
+      `with it cleared, so native is resolved from ${nativeDir.dir}`,
+  );
 }
-// A native account's child sets no CLAUDE_CONFIG_DIR, so it authenticates
-// against whatever the default dir is in the ENV THE GATEWAY RUNS IN. Doctor
-// reads the same variable and prints the path it used, so a box that exports
-// CLAUDE_CONFIG_DIR globally shows its real source rather than a guess.
 const identityIo = {
   readFile: (p) => readFileSync(expandHome(p), "utf8"),
   expandHome,
-  defaultConfigDir: process.env.CLAUDE_CONFIG_DIR
-    ? expandHome(process.env.CLAUDE_CONFIG_DIR)
-    : join(HOME, ".claude"),
+  defaultConfigDir: nativeDir.dir,
   homeDir: HOME,
 };
+/**
+ * Last time the CLI's login for this account was accepted. The shim's file
+ * only: the usage poll authenticates with the credentials FILE, which on a
+ * split store is not the copy the CLI reads, so it proves nothing here.
+ */
+function authProofAt(accountId) {
+  return lastAuthSuccessAt(readJson(join(STATE_DIR, `${accountId}.json`)));
+}
 const accounts = pluginConfig.accounts ?? [];
 const identities = [];
 if (accounts.length === 0) warn("no accounts configured");
@@ -361,24 +380,51 @@ for (const account of accounts) {
   // credential EXISTS, and #8 is exactly the case where a present credential
   // is a session the Claude CLI has already rejected. A recorded runtime
   // failure is the stronger evidence, so it is reported as such.
+  // The record stands until a successful turn (or a re-login) clears it, so
+  // its age changes the wording, never the verdict.
   const recorded = readJson(join(STATE_DIR, `${account.id}.json`))?.credential;
   if (recorded?.status === "failed") {
-    const ageMin = Math.round((Date.now() - recorded.seenAt) / 60000);
+    const ageMin = Math.max(0, Math.round((Date.now() - recorded.seenAt) / 60000));
+    const age = ageMin < 90 ? `${ageMin}m` : `${Math.round(ageMin / 60)}h`;
     bad(
-      `${account.id}: the Claude CLI rejected this login ${ageMin}m ago${
+      `${account.id}: the Claude CLI rejected this login ${age} ago${
         recorded.reason ? ` (${recorded.reason})` : ""
-      } — excluded from the pool; fix with \`multi-clawd login ${account.id}\``,
+      } and no successful turn has been seen since — ` +
+        `fix with \`multi-clawd login ${account.id}\`, or prove an out-of-band fix with ` +
+        `\`multi-clawd doctor --probe\``,
     );
-    continue;
-  }
-  if (account.oauthTokenRef) {
+  } else if (account.oauthTokenRef) {
     warn(`${account.id}: oauthTokenRef — validated by the gateway's async probe, not doctor`);
-    continue;
+  } else {
+    const check = assessAccountCredential(account, io, { proof: { lastSuccessAt: authProofAt(account.id) } });
+    if (check.status === "ok") ok(`${account.id}: credential verified — ${check.detail ?? "ok"}`);
+    else if (check.status === "unverified") {
+      // Not a failure (an idle account's access token expires routinely), and
+      // never "alive": a dead refresh token looks exactly like this at rest.
+      warn(
+        `${account.id}: credential UNVERIFIED — ${check.reason}. The next turn on it refreshes or ` +
+          `fails; prove it now with \`multi-clawd doctor --probe\``,
+      );
+    } else if (check.status === "unknown") warn(`${account.id}: cannot verify (${check.reason ?? "no source"})`);
+    else bad(`${account.id}: ${check.reason}`);
+  }
+  // Credential split-store: more than one copy of this login's OAuth
+  // credential. Refresh tokens rotate, so once the copies diverge only the one
+  // refreshed last can still refresh — and the copy a tool happens to read
+  // decides whether it sees a live login or a dead one. Warn-level: the CLI's
+  // own copy may be fine today; this is the early warning.
+  const split = findCredentialCopies(account, io);
+  if (split.status === "split") {
+    const text = describeSplitStore(split, Date.now());
+    warn(`credential split-store — ${text.headline}`);
+    for (const line of text.copies) console.log(`       · ${line}`);
+    console.log(`       remedy: ${text.remedy}`);
+  } else if (split.status !== "n/a" && VERBOSE) {
+    note(`${account.id}: credential split-store — ${split.status === "single" ? "one copy" : "no stored copy"}`);
+  }
+  if (split.incomplete) {
+    note(`${account.id}: credential split-store check incomplete — the Keychain could not be listed, so same-service duplicates may be missed`);
   }
-  const check = checkAccountCredential(account, io);
-  if (check.status === "ok") ok(`${account.id}: credential source looks alive`);
-  else if (check.status === "unknown") warn(`${account.id}: cannot verify (${check.reason ?? "no source"})`);
-  else bad(`${account.id}: ${check.reason}`);
 }
 
 // Two ids, one login. This is the failure the rest of doctor cannot see: every
@@ -449,6 +495,27 @@ else {
   const members = (pool.accounts ?? []).filter((id) => accounts.some((a) => a.id === id));
   if (members.length < 2) bad(`pool "${pool.id ?? "clawd"}" has ${members.length} valid member(s); needs ≥ 2`);
   else ok(`pool "${pool.id ?? "clawd"}": ${members.join(" → ")}`);
+  // Setup-token members: what they do and do not get, said once, plainly.
+  const tokenMembers = members
+    .map((id) => accounts.find((a) => a.id === id))
+    .filter((a) => a && !a.native && (a.oauthTokenFile || a.oauthTokenRef));
+  if (tokenMembers.length > 0 && pool.usagePoll?.enabled !== false) {
+    note(
+      `usage poll: ${tokenMembers.map((a) => a.id).join(", ")} not polled — a setup-token carries only the ` +
+        `user:inference scope and the usage endpoint needs user:profile (upstream claude-code #11985). ` +
+        `Rotation for ${tokenMembers.length === 1 ? "it" : "them"} runs on stream telemetry: each turn's ` +
+        `rate-limit events still trigger the threshold and 5-hour-warning rules, but an idle account is ` +
+        `not re-measured between turns and there is no 95% usage alert.`,
+    );
+  }
+  const refOnly = tokenMembers.filter((a) => !a.oauthTokenFile);
+  if (refOnly.length > 0) {
+    note(
+      `in-turn retry: ${refOnly.map((a) => a.id).join(", ")} (oauthTokenRef) cannot be a mid-turn retry ` +
+        `target — the shim never receives secret values, and a reference has no file it could read. ` +
+        `${refOnly.length === 1 ? "It rotates" : "They rotate"} at the next launch instead.`,
+    );
+  }
   const sticky = readJson(join(STATE_DIR, `pool-${pool.id ?? "clawd"}.sticky.json`));
 
   // WHICH ACCOUNT IS ACTUALLY BEING USED. Everything above describes the pool