@drakon-systems/multi-clawd 1.8.10 → 1.9.1

package/README.md

@@ -31,6 +31,7 @@ multi-clawd setup     # guided wizard: accounts, isolated second login, pool, wa
 multi-clawd login claw2   # launch the right Claude sign-in for an account (or re-auth it)
 multi-clawd explain   # your whole setup in plain English — accounts, chain, live health
 multi-clawd chain     # audit your model routing — what actually serves each turn
+multi-clawd direct    # the direct anthropic/* route: status, or `direct sync` (v1.9, optional)
 multi-clawd doctor    # health check (add --probe for a live end-to-end turn)
 ```
 
@@ -241,6 +242,11 @@ claude-cli/claude-fable-5        # main login
 - 🧲 **Sticky rotation (v0.3)** — after handing over, the pool dwells on the
   spare account (default 10 min) before returning home, so turns never flap
   across the threshold. Health always overrides stickiness.
+- 🔁 **In-turn retry on a reactive limit** — when an account hits a model cap
+  *during* a launch, the pool no longer loses that turn to the next provider in
+  your chain: the refusal is swallowed before anything reaches you and the turn
+  is re-spawned on a healthy account. Fresh launches, secret-free sibling
+  accounts, one retry.
 - 📟 **Operator alerts (v0.3)** — dead logins (probed every 15 min without
   spending quota), pool rotations, whole-pool exhaustion, and watchdog
   restarts surface through your agent's next heartbeat (e.g. straight into
@@ -369,7 +375,7 @@ openclaw plugins install (Get-Location).Path
 **Or let your agent install it.** Running an OpenClaw assistant or Claude
 Code on the target machine already? Paste it this and go make coffee:
 
-> Read https://raw.githubusercontent.com/Drakon-Systems-Ltd/multi-clawd/v1.8.10/SETUP-AGENT.md
+> Read https://raw.githubusercontent.com/Drakon-Systems-Ltd/multi-clawd/v1.9.1/SETUP-AGENT.md
 > and follow it to set up multi-clawd on this machine. I own a second
 > Claude account — ask me when you need me to log in.
 
@@ -589,6 +595,77 @@ Notes:
   which runs on every launch on every turn path. Details in
   [`DESIGN.md`](./DESIGN.md).
 
+## Direct route: the pool for `anthropic/*` too (v1.9)
+
+OpenClaw reaches Claude two ways. The pool above drives **Claude Code CLI**
+backends (`clawd/…`). OpenClaw's own **`anthropic/*`** models skip the CLI and
+call the Anthropic API directly, with an auth profile. A `claude setup-token`
+works for both, and both use up the same account limits.
+
+From v1.9 the pool can steer both. Opt accounts in with `direct`:
+
+```jsonc
+"accounts": [
+  // native login: give it its own setup-token for the API route
+  { "id": "claw1", "native": true,
+    "direct": { "tokenRef": { "source": "exec", "provider": "onepassword",
+                              "id": "op://YourVault/claw1-setup-token/password" } } },
+  // already has a setup-token for the CLI: reuse it
+  { "id": "claw2", "configDir": "~/.claw2",
+    "oauthTokenRef": { "source": "exec", "provider": "onepassword",
+                       "id": "op://YourVault/claw2-setup-token/password" },
+    "direct": true }
+],
+"directRoute": { "agents": ["main"] }   // optional; default ["main"]
+```
+
+Then run `multi-clawd direct sync`. It does two things:
+
+1. **Stores one OpenClaw profile per account** (`anthropic:claw1`,
+   `anthropic:claw2`), using OpenClaw's own commands:
+   - A **secret reference** is stored as a reference (`openclaw secrets
+     apply`). OpenClaw resolves it at runtime, so the token is never copied.
+   - A **token file** is piped to `openclaw models auth paste-token` on stdin.
+   - Profiles that already exist are left alone; `--resync` rewrites them.
+2. **Sets the `anthropic` auth order** to match pool health.
+
+The plugin then keeps that order up to date inside the gateway, re-checking
+every minute:
+
+- The healthiest account goes first; a nearly-maxed one moves to the back
+  **before** it errors.
+- The rule is the same one the CLI pool uses, and it reads the same health
+  data, so both routes lean on the same account at the same time.
+- OpenClaw's own rotation covers the rest mid-turn: a 429 cools that profile
+  down and the turn moves to the next one.
+
+What to know:
+
+- **Why a native or config-dir login needs its own token.** That login is a
+  rotating one-time OAuth grant, and a copy of it breaks on the next refresh.
+  multi-clawd never reads Claude's own credential files. Run `claude
+  setup-token` signed in as that account and store the token. `setup` walks
+  you through it.
+- **Already stored a profile yourself?** Adopt it with
+  `"direct": { "profileId": "anthropic:<id>" }`. It gets ordered, never
+  rewritten.
+- **Existing sessions stay on their account.** OpenClaw pins a profile to
+  each session. A new order applies to new sessions, and to a session whose
+  profile cools down. A pinned session moves when its account actually hits
+  its limit, and the next account in line is by then the healthiest.
+- **Your own profiles are kept.** Anything in the `anthropic` order that
+  multi-clawd doesn't manage stays there, after the pool's accounts.
+  `"directRoute": { "manageOrder": false }` hands the order back to you.
+- **Health comes from CLI turns only.** Traffic that only uses the API route
+  doesn't update it. With no recent CLI turns an account counts as healthy,
+  and OpenClaw's reactive rotation does the work.
+- **Nothing changes without `direct`.** No timer, no OpenClaw calls, and the
+  same `explain`/`chain`/`doctor` output as before.
+
+`multi-clawd direct` shows, per account: the profile, whether it's stored,
+any cooldown, and the live order. `explain` shows the same, and `doctor
+--probe` makes one tiny live call per profile.
+
 ## How it works
 
 Three moves, all through the official plugin SDK (details in
@@ -732,6 +809,9 @@ Early but real — built for and dogfooded in production.
   for each account shape, verified afterwards (signed-in email shown, token
   values never touched); ClawHub package published under
   `@drakon-systems` ✅
+- **v1.9** — one pool, both transports: accounts can also serve OpenClaw's
+  direct `anthropic/*` route; the gateway keeps the `anthropic` auth order in
+  pool-health order; `multi-clawd direct` / `direct sync` ✅
 - **Next** — standalone localhost proxy (OpenAI-compatible) so Hermes and
   custom runtimes can share the pool; true per-session affinity; local
   five-hour-window signal (turn counting); per-account lock for the shim

package/SECURITY.md

@@ -57,6 +57,7 @@ is listed here so you can check them yourself.
 | `dist/watchdog-schedule.js` | `node <bundled script>` | Runs the eviction watchdog on a timer. Path is the package's own script. |
 | `scripts/cli.mjs` | `node <bundled script>` | The CLI dispatching to its own subcommands. |
 | `scripts/setup.mjs` | `launchctl` / `systemctl` | Loads the watchdog timer during setup on macOS/Linux. |
+| `dist/openclaw-runner.js` | the `openclaw` CLI (or `directRoute.openclawCommand`) | Direct route only (v1.9, accounts with `direct`): reads and writes the `anthropic` auth profiles and order through OpenClaw's own subcommands (`models auth list`/`order get`/`order set`/`paste-token`, `secrets apply`, `models status`). A setup-token only ever travels on the child's stdin. Never spawned without `direct` configured. |
 
 None of these pass a string to a shell — they are direct process spawns with
 argument arrays, so there is no quoting or injection surface. Paths are

package/dist/account-env.js

@@ -47,3 +47,8 @@ export function validateAccountTokenSources(account) {
         `account "${account.id}" declares ${sources.join(" + ")} — token sources are mutually exclusive; precedence applied is ${sources.includes("native") ? "native" : "oauthTokenFile"} first. Remove the extras.`,
     ];
 }
+export function accountConfigDir(account) {
+    if (!account.native && account.configDir)
+        return expandHomePath(account.configDir);
+    return resolve(homedir(), ".claude");
+}

package/dist/chain-audit.js

@@ -27,7 +27,7 @@ function parseRef(ref) {
         return { modelId: ref };
     return { provider: ref.slice(0, idx).trim().toLowerCase(), modelId: ref.slice(idx + 1) };
 }
-export function offPoolClaudeRef(ref, poolId) {
+export function offPoolClaudeRef(ref, poolId, opts = {}) {
     if (typeof ref !== "string" || ref.length === 0)
         return null;
     const { provider, modelId } = parseRef(ref);
@@ -37,14 +37,16 @@ export function offPoolClaudeRef(ref, poolId) {
         return null;
     if (provider === poolId.trim().toLowerCase())
         return null;
+    if (provider === "anthropic" && opts.directPooled)
+        return null;
     if (provider === "anthropic" || provider === "claude-cli")
         return "strong";
     if (ACCOUNT_PIN_RE.test(provider))
         return "warn";
     return null;
 }
-function classifyBypass(ref, poolId) {
-    const severity = offPoolClaudeRef(ref, poolId);
+function classifyBypass(ref, poolId, opts) {
+    const severity = offPoolClaudeRef(ref, poolId, opts);
     if (!severity)
         return null;
     const { provider } = parseRef(ref);
@@ -170,12 +172,12 @@ export function collectChainRefs(config) {
     }
     return out;
 }
-export function auditEffectiveChain(config, poolId) {
+export function auditEffectiveChain(config, poolId, opts = {}) {
     if (!poolId)
         return [];
     const findings = [];
     for (const { surface, ref, allowlist } of collectChainRefs(config)) {
-        const reason = classifyBypass(ref, poolId);
+        const reason = classifyBypass(ref, poolId, opts);
         if (!reason)
             continue;
         findings.push({ surface, ref, severity: allowlist ? "note" : "warn", reason });
@@ -257,7 +259,7 @@ export function auditChainShadowing(config) {
     return findings;
 }
 const POOL_PROVIDER = "clawd";
-export function auditSessionOverrides(sessions, poolConfigured) {
+export function auditSessionOverrides(sessions, poolConfigured, opts = {}) {
     if (!poolConfigured)
         return [];
     const findings = [];
@@ -282,7 +284,7 @@ export function auditSessionOverrides(sessions, poolConfigured) {
             continue;
         }
         const ref = `${provider}/${model}`;
-        const severity = offPoolClaudeRef(ref, POOL_PROVIDER);
+        const severity = offPoolClaudeRef(ref, POOL_PROVIDER, opts);
         if (!severity)
             continue;
         const reason = severity === "strong"

package/dist/direct-report.js

@@ -0,0 +1,119 @@
+import { classifyAccountHealth } from "./health.js";
+import { DIRECT_PROVIDER, collectDirectMembers, } from "./direct-route.js";
+import { parseUnusableProfiles, planAgentOrder, readDirectRouteSnapshot, safeCliError, } from "./direct-sync.js";
+export function describeDirectSource(member) {
+    const s = member.source;
+    switch (s.kind) {
+        case "ref":
+            return `setup-token via ${s.ref.provider || "a secret provider"} secret reference${s.reused ? " (the same one its CLI login uses)" : ""}`;
+        case "file":
+            return `setup-token file ${s.path}${s.reused ? " (the same one its CLI login uses)" : ""}`;
+        case "existing":
+            return "adopted profile you stored yourself";
+        default:
+            return "no direct credential";
+    }
+}
+export async function gatherDirectStatus(params) {
+    const { members, problems } = collectDirectMembers(params.accounts, params.poolAccounts ?? []);
+    if (members.length === 0 && problems.length === 0)
+        return undefined;
+    const errors = [];
+    const verdicts = members.map((m) => ({
+        accountId: m.accountId,
+        profileId: m.profileId,
+        verdict: classifyAccountHealth(params.readHealth(m.accountId), params.healthOptions, params.nowMs).verdict,
+    }));
+    const read = members.length > 0 ? await readDirectRouteSnapshot(params.runner, params.agentId) : {};
+    if ("error" in read && read.error)
+        errors.push(read.error);
+    const snapshot = "snapshot" in read ? read.snapshot : undefined;
+    let unusable = [];
+    if (members.length > 0 && !params.skipCooldowns) {
+        const status = await params.runner(["models", "status", "--agent", params.agentId, "--json"], { timeoutMs: 120_000 });
+        const parsed = status.code === 0 ? parseUnusableProfiles(status.stdout) : undefined;
+        if (parsed)
+            unusable = parsed;
+        else
+            errors.push(`models status ${safeCliError(status)}`);
+    }
+    let pendingOrder;
+    if (snapshot) {
+        pendingOrder = planAgentOrder({
+            members: verdicts,
+            snapshot,
+            configOrder: params.configOrder,
+            sticky: params.sticky,
+            nowMs: params.nowMs,
+            minDwellMs: params.minDwellMs,
+        }).order;
+    }
+    const byProfile = new Map(unusable.map((u) => [u.profileId, u]));
+    const order = snapshot?.storedOrder ?? (params.configOrder?.length ? [...params.configOrder] : snapshot ? [] : undefined);
+    const orderSource = snapshot?.storedOrder
+        ? `stored for agent ${params.agentId}`
+        : params.configOrder?.length
+            ? "config auth.order"
+            : undefined;
+    return {
+        agentId: params.agentId,
+        verdicts,
+        pendingOrder,
+        unusable,
+        errors,
+        explain: {
+            members: members.map((m) => {
+                const u = byProfile.get(m.profileId);
+                return {
+                    accountId: m.accountId,
+                    profileId: m.profileId,
+                    source: describeDirectSource(m),
+                    stored: snapshot ? snapshot.stored.has(m.profileId) : undefined,
+                    ...(m.source.kind === "existing" ? { adopted: true } : {}),
+                    ...(u ? { cooldownUntil: u.until, cooldownReason: u.reason ?? u.kind } : {}),
+                };
+            }),
+            problems: problems.map((p) => ({ accountId: p.accountId, reason: p.reason })),
+            order,
+            orderSource,
+        },
+    };
+}
+export function parseProbeResults(stdout) {
+    const start = stdout.indexOf("{");
+    if (start < 0)
+        return undefined;
+    let doc;
+    try {
+        doc = JSON.parse(stdout.slice(start));
+    }
+    catch {
+        return undefined;
+    }
+    const rows = doc?.auth?.probes?.results;
+    if (!Array.isArray(rows))
+        return undefined;
+    return rows
+        .filter((r) => typeof r?.profileId === "string" && (r.provider === undefined || r.provider === DIRECT_PROVIDER))
+        .map((r) => ({
+        profileId: r.profileId,
+        status: typeof r.status === "string" ? r.status : "unknown",
+        ...(typeof r.error === "string" ? { error: r.error.replace(/sk-ant-[A-Za-z0-9_-]+/g, "sk-ant-…").slice(0, 200) } : {}),
+    }));
+}
+export function probeArgs(agentId, profileIds) {
+    return [
+        "models",
+        "status",
+        "--agent",
+        agentId,
+        "--json",
+        "--probe",
+        "--probe-provider",
+        DIRECT_PROVIDER,
+        "--probe-profile",
+        profileIds.join(","),
+        "--probe-max-tokens",
+        "16",
+    ];
+}

package/dist/direct-route.js

@@ -0,0 +1,156 @@
+import { isSecretRefShape } from "./token-resolution.js";
+import { decideStickySelection } from "./sticky.js";
+export const DIRECT_PROVIDER = "anthropic";
+export const DIRECT_PROFILE_PREFIX = `${DIRECT_PROVIDER}:`;
+export const DIRECT_SETUP_TOKEN_GUIDANCE = "a native or config-dir Claude login is a rotating single-use OAuth grant — copying it into " +
+    "OpenClaw would invalidate one of the two copies on the next refresh. Run `claude setup-token` " +
+    "signed in as THIS account, store the printed token in your secret manager (or a 0600 file), " +
+    "and set direct.tokenRef (or direct.tokenFile) on the account — or, if a profile for this " +
+    'account is already stored in OpenClaw, name it with direct.profileId';
+function asRecord(value) {
+    return typeof value === "object" && value !== null && !Array.isArray(value)
+        ? value
+        : undefined;
+}
+export function directEnabled(account) {
+    const d = account.direct;
+    if (d === undefined || d === false)
+        return false;
+    return true;
+}
+export function directCredentialSource(account) {
+    if (!directEnabled(account))
+        return { kind: "none" };
+    const explicit = asRecord(account.direct);
+    const explicitRef = explicit?.tokenRef;
+    const explicitFile = typeof explicit?.tokenFile === "string" ? explicit.tokenFile.trim() : "";
+    if (explicitRef !== undefined && explicitFile) {
+        return {
+            kind: "unsupported",
+            code: "direct_sources_conflict",
+            reason: "direct.tokenRef and direct.tokenFile are mutually exclusive — keep one",
+        };
+    }
+    if (explicitRef !== undefined) {
+        if (!isSecretRefShape(explicitRef)) {
+            return {
+                kind: "unsupported",
+                code: "direct_ref_malformed",
+                reason: 'direct.tokenRef must be { "source": "...", "provider": "...", "id": "..." } like every other secret reference',
+            };
+        }
+        return { kind: "ref", ref: explicitRef, reused: false };
+    }
+    if (explicitFile)
+        return { kind: "file", path: explicitFile, reused: false };
+    if (typeof explicit?.profileId === "string" && explicit.profileId.trim()) {
+        return { kind: "existing" };
+    }
+    if (!account.native) {
+        if (account.oauthTokenFile) {
+            return { kind: "file", path: account.oauthTokenFile, reused: true };
+        }
+        if (isSecretRefShape(account.oauthTokenRef)) {
+            return { kind: "ref", ref: account.oauthTokenRef, reused: true };
+        }
+    }
+    return {
+        kind: "unsupported",
+        code: "direct_setup_token_required",
+        reason: DIRECT_SETUP_TOKEN_GUIDANCE,
+    };
+}
+const PROFILE_SUFFIX_RE = /^[A-Za-z0-9][A-Za-z0-9._@+-]{0,127}$/;
+export function directProfileId(account) {
+    const explicit = asRecord(account.direct)?.profileId;
+    if (typeof explicit === "string" && explicit.trim()) {
+        const id = explicit.trim();
+        if (!id.startsWith(DIRECT_PROFILE_PREFIX) || !PROFILE_SUFFIX_RE.test(id.slice(DIRECT_PROFILE_PREFIX.length))) {
+            throw new Error(`account "${account.id}": direct.profileId must look like "${DIRECT_PROFILE_PREFIX}<name>" (got "${id}")`);
+        }
+        return id;
+    }
+    const suffix = account.id.trim();
+    if (!PROFILE_SUFFIX_RE.test(suffix)) {
+        throw new Error(`account "${account.id}" cannot become an auth profile id — set direct.profileId`);
+    }
+    return `${DIRECT_PROFILE_PREFIX}${suffix}`;
+}
+export function collectDirectMembers(accounts, poolOrder = []) {
+    const ordered = [];
+    const byId = new Map(accounts.filter((a) => a?.id).map((a) => [a.id.trim(), a]));
+    for (const id of poolOrder) {
+        const a = byId.get(id);
+        if (a && !ordered.includes(a))
+            ordered.push(a);
+    }
+    for (const a of byId.values())
+        if (!ordered.includes(a))
+            ordered.push(a);
+    const members = [];
+    const problems = [];
+    const seenProfiles = new Set();
+    for (const account of ordered) {
+        const source = directCredentialSource(account);
+        if (source.kind === "none")
+            continue;
+        if (source.kind === "unsupported") {
+            problems.push({ accountId: account.id, code: source.code, reason: source.reason });
+            continue;
+        }
+        let profileId;
+        try {
+            profileId = directProfileId(account);
+        }
+        catch (err) {
+            problems.push({ accountId: account.id, code: "direct_profile_invalid", reason: err.message });
+            continue;
+        }
+        if (seenProfiles.has(profileId)) {
+            problems.push({
+                accountId: account.id,
+                code: "direct_profile_duplicate",
+                reason: `profile ${profileId} is already used by another account`,
+            });
+            continue;
+        }
+        seenProfiles.add(profileId);
+        members.push({ accountId: account.id, profileId, source });
+    }
+    return { members, problems };
+}
+const VERDICT_RANK = {
+    ok: 0,
+    no_data: 0,
+    near_limit: 1,
+    exhausted: 2,
+    credential_failed: 3,
+};
+export function planDirectOrder(params) {
+    const { members } = params;
+    if (members.length === 0)
+        return undefined;
+    const decision = decideStickySelection({
+        verdicts: members.map((m) => ({ id: m.accountId, verdict: m.verdict })),
+        sticky: params.sticky,
+        nowMs: params.nowMs,
+        minDwellMs: params.minDwellMs,
+    });
+    const first = members.find((m) => m.accountId === decision.account) ?? members[0];
+    const rest = members
+        .filter((m) => m !== first)
+        .map((m, index) => ({ m, index }))
+        .sort((a, b) => VERDICT_RANK[a.m.verdict] - VERDICT_RANK[b.m.verdict] || a.index - b.index)
+        .map(({ m }) => m.profileId);
+    const managed = new Set(members.map((m) => m.profileId));
+    const unmanaged = (params.currentOrder ?? []).filter((id, i, all) => typeof id === "string" && !managed.has(id) && all.indexOf(id) === i);
+    const order = [first.profileId, ...rest, ...unmanaged];
+    const current = params.currentOrder ?? [];
+    const changed = order.length !== current.length || order.some((id, i) => id !== current[i]);
+    return { order, firstAccount: first.accountId, sticky: decision.sticky, changed };
+}
+export function directRoutePools(accounts, directRoute) {
+    if (directRoute?.manageOrder === false)
+        return false;
+    return collectDirectMembers(accounts).members.length >= 2;
+}