@askalf/dario 6.18.4 → 6.19.0

package/README.md

@@ -16,7 +16,7 @@
 
 <p><strong>One local endpoint. Every AI tool you own. Your Claude seats first, your ChatGPT plan and a metered key as the rungs you chain behind them, and a ledger of who paid for what.</strong></p>
 
-<sub><code>npm i -g @askalf/dario</code> or <code>brew install askalf/tap/dario</code> · <strong>0</strong> runtime deps · <a href="https://www.npmjs.com/package/@askalf/dario">SLSA-attested</a> every release · nothing phones home · ~42k lines you can read in a weekend · independent, unofficial, third-party (<a href="DISCLAIMER.md">DISCLAIMER.md</a>)</sub>
+<sub><code>npm i -g @askalf/dario</code> or <code>brew install askalf/tap/dario</code> · <strong>0</strong> runtime deps · <a href="https://www.npmjs.com/package/@askalf/dario">SLSA-attested</a> every release · nothing phones home · ~43k lines you can read in a weekend · independent, unofficial, third-party (<a href="DISCLAIMER.md">DISCLAIMER.md</a>)</sub>
 
 <sub><a href="#start-in-60-seconds">Start</a> · <a href="#why-people-pick-it">Why dario</a> · <a href="#what-it-does-with-a-request">Routing</a> · <a href="#will-my-account-get-suspended">Risk</a> · <a href="#reference">Reference</a> · <a href="#trust--transparency">Trust</a> · <a href="docs/returning.md">Coming back after a while?</a></sub>
 
@@ -113,7 +113,7 @@ dario runs entirely on your machine, authenticates as you with your own Claude l
 
 | Signal | Status |
 |---|---|
-| Source | **~42k** lines of TypeScript across **87** files, auditable in a weekend. One credential path since v5: the pool. |
+| Source | **~43k** lines of TypeScript across **87** files, auditable in a weekend. One credential path since v5: the pool. |
 | Dependencies | **0 runtime.** Verify: `npm ls --production` |
 | Provenance | Every release [SLSA-attested](https://www.npmjs.com/package/@askalf/dario) via GitHub Actions + Sigstore, published with OIDC trusted publishing: no long-lived npm token exists to leak |
 | Scanning | [CodeQL](https://github.com/askalf/dario/actions/workflows/codeql.yml) on every push to master, every PR and weekly · [ClusterFuzzLite](./.github/workflows/cflite.yml) fuzzes the SSE translator, the rejection parsers and the cch stamper weekly · the [OpenSSF Scorecard](https://scorecard.dev/viewer/?uri=github.com/askalf/dario) badge above and the [Best Practices](https://www.bestpractices.dev/projects/13638) entry are live |

package/dist/accounts.d.ts

@@ -38,7 +38,21 @@ export type IdentitySource = 'claude-code' | 'generated';
 export declare function listAccountAliases(): Promise<string[]>;
 export declare function loadAccount(alias: string): Promise<AccountCredentials | null>;
 export declare function loadAllAccounts(): Promise<AccountCredentials[]>;
+/**
+ * Whether `candidate` holds newer credentials than `current`. A newer grant
+ * (a fresh login) wins outright, since an older grant's refreshed access
+ * token can expire later. Same grant, or a grant time missing on either
+ * side: the later expiry wins. The lock services apply the same rule.
+ */
+export declare function credentialsAreNewer(candidate: {
+    expiresAt: number;
+    grantedAt?: number;
+}, current: {
+    expiresAt?: number;
+    grantedAt?: number;
+}): boolean;
 export declare function saveAccount(creds: AccountCredentials): Promise<void>;
+export declare function _setAccountWriteHookForTest(hook: (() => Promise<void>) | null): void;
 export declare function removeAccount(alias: string): Promise<boolean>;
 /**
  * Carry the organization the proxy observed on a seat into a record that is
@@ -118,6 +132,18 @@ export declare function chooseClientIdentity(existing: readonly AccountCredentia
 export declare function regenerateClientIdentity(aliases: readonly string[]): Promise<string[]>;
 /** Refresh an account's OAuth token using dario's auto-detected CC OAuth config. */
 export declare function refreshAccountToken(creds: AccountCredentials): Promise<AccountCredentials>;
+/**
+ * Adopt credentials another instance refreshed for this account, when the
+ * lock service holds a newer copy than `local` (dario#1733). Returns the
+ * newest credentials newer than `local`, saving them when the account file
+ * is older; null when nothing newer is shared, the account is gone, no lock
+ * service is set, or it cannot be reached. Never refreshes and never takes the lock. The
+ * proxy asks at startup and on every refresh tick, so a replica runs on its
+ * peer's rotation within one tick instead of carrying the old pair until its
+ * own refresh window. "Newer" is the lock service's rule: a newer grant, or
+ * the same grant with a later expiry.
+ */
+export declare function adoptSharedCredentials(local: Pick<AccountCredentials, 'alias' | 'expiresAt' | 'grantedAt'>): Promise<AccountCredentials | null>;
 /**
  * Fill in `accountId` (and the profile's other fields) on a record that predates
  * them, from one profile GET; saves and returns the record, unchanged when

package/dist/accounts.js

@@ -78,10 +78,46 @@ export async function loadAllAccounts() {
     const loaded = await Promise.all(aliases.map(a => loadAccount(a)));
     return loaded.filter((a) => a !== null);
 }
+// Writes and removals of one account file run one at a time, in call order.
+// A check-then-write (adoptSharedCredentials) runs inside the same chain, so
+// a login saved or a removal made while it waited on the network lands
+// either before its check, which then sees it, or after its write.
+const accountWrites = new Map();
+function serializeAccountWrite(alias, fn) {
+    const next = (accountWrites.get(alias) ?? Promise.resolve()).then(fn);
+    const tail = next.catch(() => { });
+    accountWrites.set(alias, tail);
+    void tail.then(() => { if (accountWrites.get(alias) === tail)
+        accountWrites.delete(alias); });
+    return next;
+}
+/**
+ * Whether `candidate` holds newer credentials than `current`. A newer grant
+ * (a fresh login) wins outright, since an older grant's refreshed access
+ * token can expire later. Same grant, or a grant time missing on either
+ * side: the later expiry wins. The lock services apply the same rule.
+ */
+export function credentialsAreNewer(candidate, current) {
+    const a = candidate.grantedAt, b = current.grantedAt;
+    if (typeof a === 'number' && typeof b === 'number' && Number.isFinite(a) && Number.isFinite(b) && a !== b)
+        return a > b;
+    return candidate.expiresAt > (current.expiresAt ?? 0);
+}
 export async function saveAccount(creds) {
     const path = safeAliasPath(creds.alias);
     if (!path)
         throw new Error(`invalid account alias: ${creds.alias}`);
+    await serializeAccountWrite(creds.alias, () => writeAccountFile(path, creds));
+}
+// Test-only: awaited at the start of every account file write, so a test can
+// hold a write open and queue other work behind it. Null in production.
+let accountWriteHookForTest = null;
+export function _setAccountWriteHookForTest(hook) {
+    accountWriteHookForTest = hook;
+}
+async function writeAccountFile(path, creds) {
+    if (accountWriteHookForTest)
+        await accountWriteHookForTest();
     await ensureDir();
     // Durable write (dario#790): fsync the temp file + parent dir so a rotated
     // refresh token survives an abrupt container recreate. A plain rename left
@@ -94,13 +130,15 @@ export async function removeAccount(alias) {
     const path = safeAliasPath(alias);
     if (!path)
         return false;
-    try {
-        await unlink(path);
-        return true;
-    }
-    catch {
-        return false;
-    }
+    return serializeAccountWrite(alias, async () => {
+        try {
+            await unlink(path);
+            return true;
+        }
+        catch {
+            return false;
+        }
+    });
 }
 /**
  * Carry the organization the proxy observed on a seat into a record that is
@@ -304,6 +342,9 @@ export async function refreshAccountToken(creds) {
 // error, timeout) — a Cloudflare outage must not block dario's own
 // refresh, since the lock is a resilience nice-to-have layered on top of
 // dario's actual job, not a new dependency dario's core function needs.
+// Open means before the token runs out: while it still has
+// LOCK_DOWN_DEFER_MS left, the refresh waits for a later tick instead
+// (dario#1733).
 const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
 function refreshLockUrl(env = process.env) {
     const v = env.DARIO_REFRESH_LOCK_URL;
@@ -331,6 +372,64 @@ async function lockCall(path, body, env = process.env) {
         return null; // network error / CF outage — fail OPEN
     }
 }
+/**
+ * Adopt credentials another instance refreshed for this account, when the
+ * lock service holds a newer copy than `local` (dario#1733). Returns the
+ * newest credentials newer than `local`, saving them when the account file
+ * is older; null when nothing newer is shared, the account is gone, no lock
+ * service is set, or it cannot be reached. Never refreshes and never takes the lock. The
+ * proxy asks at startup and on every refresh tick, so a replica runs on its
+ * peer's rotation within one tick instead of carrying the old pair until its
+ * own refresh window. "Newer" is the lock service's rule: a newer grant, or
+ * the same grant with a later expiry.
+ */
+export async function adoptSharedCredentials(local) {
+    if (!refreshLockUrl())
+        return null;
+    const res = await lockCall(`/lock/${encodeURIComponent(local.alias)}/peek`, {
+        currentExpiresAt: local.expiresAt,
+        currentGrantedAt: local.grantedAt,
+    });
+    const shared = res?.credentials;
+    // A lock service from before /peek answers 404, which lockCall turns into
+    // null: nothing is adopted and refreshes go through /acquire as before.
+    if (!shared || shared.alias !== local.alias || typeof shared.accessToken !== 'string'
+        || typeof shared.refreshToken !== 'string' || typeof shared.expiresAt !== 'number') {
+        return null;
+    }
+    // The service compared against `local` as it was when the request left.
+    // Compare again against the file as it is now: a login may have landed
+    // while the request was out, or the account may have been removed, and a
+    // removed account stays removed. The file is never overwritten with older
+    // credentials, but when it already holds something newer than `local`
+    // (replicas sharing one accounts volume, where the peer saved its rotation
+    // first) that is still handed back, so the caller's seat catches up.
+    const path = safeAliasPath(shared.alias);
+    if (!path)
+        return null;
+    return serializeAccountWrite(shared.alias, async () => {
+        const current = await loadAccount(shared.alias);
+        if (!current)
+            return null;
+        if (credentialsAreNewer(shared, current)) {
+            await writeAccountFile(path, shared);
+            return shared;
+        }
+        return credentialsAreNewer(current, local) ? current : null;
+    });
+}
+// With the lock service unreachable, a refresh is put off while the access
+// token has more than this left, since an uncoordinated refresh can rotate
+// the token out from under a peer that is refreshing too. The background
+// tick runs every 15 minutes and starts refreshing at 45 minutes left, so a
+// token deferred at 20+ minutes is tried again with 5 to 20 minutes left,
+// and refreshed then even if the service is still down.
+const LOCK_DOWN_DEFER_MS = 20 * 60 * 1000;
+// Adopted credentials with less than this left are refreshed straight away
+// with the adopted refresh token, not handed back to be served: a peer that
+// went quiet can leave a cached access token that is already expired, while
+// its refresh token is still the live one.
+const ADOPTED_MIN_LIFE_MS = 5 * 60 * 1000;
 async function doRefreshAccountTokenDistributed(creds) {
     if (!refreshLockUrl())
         return doRefreshAccountToken(creds);
@@ -341,17 +440,26 @@ async function doRefreshAccountTokenDistributed(creds) {
             holder,
             ttlMs: 20_000,
             currentExpiresAt: creds.expiresAt,
+            currentGrantedAt: creds.grantedAt,
         });
         if (res === null) {
-            // Lock service unreachable — fail open, refresh directly exactly as
-            // if DARIO_REFRESH_LOCK_URL were unset.
+            // Lock service unreachable. Hold off while the token has life left;
+            // near expiry, refresh directly exactly as if DARIO_REFRESH_LOCK_URL
+            // were unset.
+            const left = creds.expiresAt - Date.now();
+            if (left > LOCK_DOWN_DEFER_MS) {
+                throw new Error(`refresh lock service unreachable; deferring the refresh of ${creds.alias} while its access token has ${Math.round(left / 60_000)} minutes left`);
+            }
             return doRefreshAccountToken(creds);
         }
         if (res.credentials) {
             // Another instance already refreshed more recently than what we're
             // holding — adopt it, no Anthropic call needed at all.
             await saveAccount(res.credentials);
-            return res.credentials;
+            if (res.credentials.expiresAt - Date.now() >= ADOPTED_MIN_LIFE_MS)
+                return res.credentials;
+            creds = res.credentials;
+            continue;
         }
         if (res.acquired) {
             // Echoed back on BOTH release paths below. Undefined against a
@@ -863,8 +971,10 @@ export async function resyncLoginFromCredentialsIfStale() {
     // and overwriting it from credentials.json would swap a live token for one
     // whose refresh Anthropic already rotated → invalid_grant → outage (#805).
     // Only the strict case is skipped; equal expiresAt keeps the #235 behaviour
-    // (overwrite), since a real refresh always advances expiresAt.
-    if (loginAcc.expiresAt > tok.expiresAt) {
+    // (overwrite), since a real refresh always advances expiresAt. A newer grant
+    // in login.json counts as newer even with an earlier expiry: credentials
+    // adopted from a peer instance after a re-login can expire first.
+    if (credentialsAreNewer(loginAcc, tok)) {
         return 'creds-stale';
     }
     // credentials.json is the newer (or equal-age) token — the #235 case: the
@@ -920,11 +1030,13 @@ export async function mirrorLoginToCredentials(refreshed) {
     if (refreshed.alias !== MIGRATED_LOGIN_ALIAS)
         return 'skip-not-login';
     const creds = await loadCredentials();
-    const currentExpiry = creds?.claudeAiOauth?.expiresAt ?? 0;
+    const current = creds?.claudeAiOauth;
     // Only mirror a strictly-newer pool token. Equal expiry means the files
     // already agree (or a same-second write elsewhere); newer credentials.json
     // is another process's fresher family we must not overwrite (#805 direction).
-    if (currentExpiry >= refreshed.expiresAt)
+    // A newer grant counts as newer even with an earlier expiry: credentials
+    // adopted from a peer instance after a re-login can expire first.
+    if (current && !credentialsAreNewer(refreshed, current))
         return 'creds-newer';
     await saveCredentialsTokens({
         accessToken: refreshed.accessToken,

package/dist/proxy.js

@@ -29,7 +29,7 @@ import { OverageGuard, buildHaltErrorBody } from './overage-guard.js';
 import { notify as osNotify } from './notify.js';
 import { grantAge, grantThresholds, worstGrantLevel, describeGrantAge } from './refresh-grant.js';
 import { resolveSeatPin, SEAT_PIN_HEADER, SEAT_PIN_TOKEN_HEADER } from './seat-pin.js';
-import { loadAllAccounts, loadAccount, saveAccount, refreshAccountToken, resyncLoginFromCredentialsIfStale, ensureLoginCredentialsInPool, mirrorLoginToCredentials, withObservedOrganization } from './accounts.js';
+import { loadAllAccounts, loadAccount, saveAccount, refreshAccountToken, adoptSharedCredentials, credentialsAreNewer, resyncLoginFromCredentialsIfStale, ensureLoginCredentialsInPool, mirrorLoginToCredentials, withObservedOrganization } from './accounts.js';
 import { handleAdminRequest } from './admin-api.js';
 import { createTokenBucket } from './rate-limit.js';
 import { getOpenAIBackend, isOpenAIModel, forwardToOpenAI, isMiniMaxBackend, MINIMAX_MODELS } from './openai-backend.js';
@@ -1681,6 +1681,37 @@ export async function startProxy(opts = {}) {
             }
         }
     };
+    // Take a peer instance's newer credentials for this seat from the refresh
+    // lock service before deciding whether to refresh it (dario#1733). Without
+    // this, a replica kept serving and refreshing with the pair it loaded, and
+    // its refresh token had been rotated away by the peer's last refresh. A
+    // no-op without DARIO_REFRESH_LOCK_URL, and never fatal: the refresh
+    // decision below still runs on what the seat already has.
+    const adoptPeerCredentials = async (alias) => {
+        const seat = pool.get(alias);
+        if (!seat)
+            return;
+        const shared = await adoptSharedCredentials(seat).catch(() => null);
+        if (!shared)
+            return;
+        // The seat can move on while the lock service answers (a login, a
+        // refresh, a removal); replace it only if it is still there and older.
+        const live = pool.get(alias);
+        if (!live || !credentialsAreNewer(shared, live))
+            return;
+        // add(), not updateTokens(): a newer grant carries its own grantedAt, and
+        // add() is what starts a re-granted seat's live state fresh.
+        pool.add(alias, {
+            accessToken: shared.accessToken, refreshToken: shared.refreshToken, expiresAt: shared.expiresAt,
+            deviceId: shared.deviceId, accountUuid: shared.accountUuid, grantedAt: shared.grantedAt,
+            organizationId: shared.organizationId, accountId: shared.accountId, accountEmail: shared.accountEmail,
+            rateLimitTier: shared.rateLimitTier, seatTier: shared.seatTier,
+        });
+        await mirrorLoginToCredentials(shared).catch((err) => {
+            console.error(`[dario] login→credentials.json mirror failed (peer credentials) for ${alias}: ${err instanceof Error ? err.message : err}`);
+        });
+        console.error(`[dario] Adopted credentials another instance refreshed for account ${alias}.`);
+    };
     // Shared pool state across instances (docs/multi-instance.md): opt-in,
     // rides the refresh-lock service, fails open. Off → byte-identical to before.
     const lockUrl = process.env['DARIO_REFRESH_LOCK_URL'];
@@ -1819,7 +1850,9 @@ export async function startProxy(opts = {}) {
         // token (invalid_grant) just logs and leaves the account expired; the auth
         // gate below then surfaces it. Skipped in --no-claude-auth mode.
         if (!opts.noClaudeAuth) {
-            await Promise.all(pool.all().map(async (acc) => {
+            await Promise.all(pool.all().map(async (seat) => {
+                await adoptPeerCredentials(seat.alias);
+                const acc = pool.get(seat.alias) ?? seat;
                 if (acc.expiresAt >= Date.now() + 45 * 60 * 1000)
                     return;
                 try {
@@ -1890,7 +1923,9 @@ export async function startProxy(opts = {}) {
         if (opts.noClaudeAuth)
             return; // never touch the Claude token in OpenAI-only mode
         watchGrants();
-        for (const acc of pool.all()) {
+        for (const seat of pool.all()) {
+            await adoptPeerCredentials(seat.alias);
+            const acc = pool.get(seat.alias) ?? seat;
             if (acc.expiresAt < Date.now() + 45 * 60 * 1000) {
                 try {
                     const saved = await loadAccount(acc.alias);

package/docs/multi-instance.md

@@ -20,6 +20,8 @@ Two instances holding the same account both notice the token is near expiry, and
 
 A plain mutex does not fix this. The loser waits, acquires the lock, and then refreshes with a token that is *already* stale — it just loses more slowly. The fix is that the loser **adopts the winner's fresh credentials** instead of attempting its own. That is what dario's refresh lock does.
 
+The handoff has to outlast the gap between replicas, not just the race. Each instance checks its tokens every 15 minutes on its own phase, so the second one can come back well after the first refreshed. The lock service keeps the latest credentials for 30 days, and every instance reads them at startup and on each tick before deciding to refresh. Until [#1733](https://github.com/askalf/dario/issues/1733) that window was five minutes, and a replica that came back later refreshed with a dead token.
+
 ### 2. Rate-limit accounting — **solved with shared state**
 
 `pool.ts` keeps `accounts: Map<string, PoolAccount>` in process memory, and updates it only from rate-limit headers on responses *that instance* saw.
@@ -88,9 +90,13 @@ Not suitable for airgapped deployments, and it adds an internet dependency to a
 
 ### It fails open, deliberately
 
-Any lock-service error — bad response, network failure, timeout — and dario proceeds with its own refresh exactly as if no lock were configured.
+On any lock-service error (bad response, network failure, timeout), dario first waits if it can. While the access token still has more than 20 minutes left, the refresh is put off to a later tick, because refreshing alone is what can strand a peer. Inside 20 minutes dario refreshes on its own, exactly as if no lock were configured.
+
+The lock is resilience layered on top of dario's job, not a new dependency dario's core function needs. An outage of your Redis must not stop your proxy serving traffic. The cost of that choice is that an outage lasting into a token's last 20 minutes puts you back to the plain race, so do not treat the lock as a hard guarantee.
+
+### Upgrading the lock service
 
-The lock is resilience layered on top of dario's job, not a new dependency dario's core function needs. An outage of your Redis must not stop your proxy serving traffic. The cost of that choice is that during a lock outage you are back to the plain race, so do not treat the lock as a hard guarantee.
+The 30-day handoff and the `/peek` endpoint live in the lock service, so rebuild the Redis image or redeploy the Worker along with the dario upgrade. In either order nothing breaks: a new dario against an old service gets a 404 from `/peek` and treats it as nothing to adopt, and an old dario against a new service still gets the longer handoff on `/acquire`.
 
 ---
 
@@ -136,8 +142,9 @@ Anthropic's token endpoint is the one mocked part, and only that — hammering t
 
 ## If you need true HA
 
-Everything above is as far as dario goes today: shared credentials through the lock, and shared rate-limit readings and sticky bindings through `--pool-shared-state`, both eventually consistent and both failing open. Three things are still per instance:
+Everything above is as far as dario goes today: shared credentials through the lock, and shared rate-limit readings and sticky bindings through `--pool-shared-state`, both eventually consistent and both failing open. Four things are still per instance:
 
+- **The account list.** Each instance loads `~/.dario/accounts/` from its own disk, and a login through the admin API reloads only the instance that served it. Copying the account files to every instance is safe with the refresh lock on: the copies share one refresh token, and whichever instance refreshes first hands the result to the others ([#1733](https://github.com/askalf/dario/issues/1733)). Without the lock, copies race exactly as described in section 1.
 - **ChatGPT seats.** The Codex accounts have their own refresh flow and usage windows; neither the lock nor the shared state covers them.
 - **The ledger.** Each instance writes its own file, so `dario usage` and `/analytics` show that instance's share of the spend.
 - **The overage guard's halt**, which each instance tracks on its own.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@askalf/dario",
-  "version": "6.18.4",
+  "version": "6.19.0",
   "description": "Use your Claude and ChatGPT subscriptions in Cursor, Cline, Aider, Claude Code and the Agent SDK at subscription pricing, not per-token API bills. One local Anthropic + OpenAI-compatible endpoint: either plan answers either wire shape, opt-in failover when one hits its limit, and a ledger of what each plan absorbed.",
   "type": "module",
   "bin": {