codex-multi-auth 1.2.4 → 1.2.6

package/README.md

@@ -19,6 +19,7 @@ Codex CLI-first multi-account OAuth manager for the official `@openai/codex` CLI
 - Interactive dashboard for account actions and settings
 - Experimental settings tab for staged sync, backup, and refresh-guard controls
 - Forecast, report, fix, and doctor commands for operational safety
+- Runtime counters, budget/cooldown state, and multi-auth probe visibility in `codex auth status` / `codex auth report`
 - Flagged account verification and restore flow
 - Session affinity and live account sync controls
 - Proactive refresh and preemptive quota deferral controls
@@ -156,6 +157,14 @@ If browser launch is blocked, use the alternate login paths in [docs/getting-sta
 | `codex auth report --live --json` | How do I get the full machine-readable health report? |
 | `codex auth fix --live --model gpt-5-codex` | How do I run live repair probes with a chosen model? |
 
+### Reliability behavior
+
+- whole-pool replay is disabled by default when every account is rate-limited
+- active requests use a bounded outbound request budget so one prompt cannot walk the full pool indefinitely
+- repeated cross-account 5xx bursts trigger a short cooldown instead of continuing aggressive rotation
+- proactive refresh is staggered to reduce background refresh bursts
+- `codex auth status` surfaces recent runtime request metrics in text output, and `codex auth report --json` exposes the machine-readable cooldown/runtime snapshot
+
 ---
 
 ## Dashboard Hotkeys
@@ -187,11 +196,12 @@ If browser launch is blocked, use the alternate login paths in [docs/getting-sta
 | File | Default path |
 | --- | --- |
 | Settings | `~/.codex/multi-auth/settings.json` |
-| Accounts | `~/.codex/multi-auth/openai-codex-accounts.json` |
-| Flagged accounts | `~/.codex/multi-auth/openai-codex-flagged-accounts.json` |
-| Quota cache | `~/.codex/multi-auth/quota-cache.json` |
-| Logs | `~/.codex/multi-auth/logs/codex-plugin/` |
-| Per-project accounts | `~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.json` |
+| Accounts | `~/.codex/multi-auth/openai-codex-accounts.json` |
+| Flagged accounts | `~/.codex/multi-auth/openai-codex-flagged-accounts.json` |
+| Quota cache | `~/.codex/multi-auth/quota-cache.json` |
+| Runtime observability | `~/.codex/multi-auth/runtime-observability.json` |
+| Logs | `~/.codex/multi-auth/logs/codex-plugin/` |
+| Per-project accounts | `~/.codex/multi-auth/projects/<project-key>/openai-codex-accounts.json` |
 
 Override root with `CODEX_MULTI_AUTH_DIR=<path>`.
 
@@ -265,7 +275,10 @@ codex auth login
 
 - `codex auth` unrecognized: run `where codex`, then follow `docs/troubleshooting.md` for routing fallback commands
 - Switch succeeds but wrong account appears active: run `codex auth switch <index>`, then restart session
-- OAuth callback on port `1455` fails: free the port and re-run `codex auth login`
+- Requests fail fast with a pool cooldown message: wait for the cooldown window or inspect `codex auth status`
+- Requests fail fast after repeated upstream 5xx errors: inspect `codex auth report --json` for runtime traffic and cooldown details
+- Storage cleanup fails with `EBUSY` / `EPERM` (Windows): run `codex auth doctor --fix` to retry, or manually remove `~/.codex/multi-auth/<project-key>/` and re-login
+- OAuth callback on port `1455` fails: free the port and re-run `codex auth login`
 - Browser launch is blocked or you are in a headless shell: re-run `codex auth login --manual` or set `CODEX_AUTH_NO_BROWSER=1`
 - `missing field id_token` / `token_expired` / `refresh_token_reused`: re-login affected account
 
@@ -308,9 +321,10 @@ codex auth doctor --json
 
 ## Release Notes
 
-- Current stable: [docs/releases/v1.2.4.md](docs/releases/v1.2.4.md)
-- Previous stable: [docs/releases/v1.2.2.md](docs/releases/v1.2.2.md)
-- Earlier stable: [docs/releases/v1.2.1.md](docs/releases/v1.2.1.md)
+- Current stable: [docs/releases/v1.2.6.md](docs/releases/v1.2.6.md)
+- Recent patch release: [docs/releases/v1.2.5.md](docs/releases/v1.2.5.md)
+- Previous stable: [docs/releases/v1.2.2.md](docs/releases/v1.2.2.md)
+- Earlier stable: [docs/releases/v1.2.1.md](docs/releases/v1.2.1.md)
 - Archived prerelease: [docs/releases/v0.1.0-beta.0.md](docs/releases/v0.1.0-beta.0.md)
 
 ## License

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AAEH,OAAO,KAAK,EAAE,MAAM,EAAe,MAAM,kBAAkB,CAAC;AAqQ5D;;;;;;;;;;;;;;;GAeG;AAEH,eAAO,MAAM,iBAAiB,EAAE,MAgvH/B,CAAC;AAEF,eAAO,MAAM,gBAAgB,KAAoB,CAAC;AAElD,eAAe,iBAAiB,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AAEH,OAAO,KAAK,EAAE,MAAM,EAAe,MAAM,kBAAkB,CAAC;AAyR5D;;;;;;;;;;;;;;;GAeG;AAEH,eAAO,MAAM,iBAAiB,EAAE,MA2/H/B,CAAC;AAEF,eAAO,MAAM,gBAAgB,KAAoB,CAAC;AAElD,eAAe,iBAAiB,CAAC"}

package/dist/index.js

@@ -30,7 +30,7 @@ import { startLocalOAuthServer } from "./lib/auth/server.js";
 import { checkAndNotify } from "./lib/auto-update-checker.js";
 import { CapabilityPolicyStore } from "./lib/capability-policy.js";
 import { promptAddAnotherAccount, promptLoginMode } from "./lib/cli.js";
-import { getAutoResume, getCodexMode, getEmptyResponseMaxRetries, getEmptyResponseRetryDelayMs, getFallbackToGpt52OnUnsupportedGpt53, getFastSession, getFastSessionMaxInputItems, getFastSessionStrategy, getFetchTimeoutMs, getLiveAccountSync, getLiveAccountSyncDebounceMs, getLiveAccountSyncPollMs, getNetworkErrorCooldownMs, getPerProjectAccounts, getPidOffsetEnabled, getPreemptiveQuotaEnabled, getPreemptiveQuotaMaxDeferralMs, getPreemptiveQuotaRemainingPercent5h, getPreemptiveQuotaRemainingPercent7d, getProactiveRefreshBufferMs, getProactiveRefreshGuardian, getProactiveRefreshIntervalMs, getRateLimitToastDebounceMs, getRetryAllAccountsMaxRetries, getRetryAllAccountsMaxWaitMs, getRetryAllAccountsRateLimited, getBackgroundResponses, getResponseContinuation, getServerErrorCooldownMs, getSessionAffinity, getSessionAffinityMaxEntries, getSessionAffinityTtlMs, getSessionRecovery, getStorageBackupEnabled, getStreamStallTimeoutMs, getToastDurationMs, getTokenRefreshSkewMs, getUnsupportedCodexFallbackChain, getUnsupportedCodexPolicy, loadPluginConfig, } from "./lib/config.js";
+import { getAutoResume, getCodexMode, getEmptyResponseMaxRetries, getEmptyResponseRetryDelayMs, getFallbackToGpt52OnUnsupportedGpt53, getFastSession, getFastSessionMaxInputItems, getFastSessionStrategy, getFetchTimeoutMs, getRateLimitDedupWindowMs, getRateLimitMaxBackoffMs, getRateLimitShortRetryThresholdMs, getRateLimitStateResetMs, getLiveAccountSync, getLiveAccountSyncDebounceMs, getLiveAccountSyncPollMs, getNetworkErrorCooldownMs, getPerProjectAccounts, getPidOffsetEnabled, getPreemptiveQuotaEnabled, getPreemptiveQuotaMaxDeferralMs, getPreemptiveQuotaRemainingPercent5h, getPreemptiveQuotaRemainingPercent7d, getProactiveRefreshBufferMs, getProactiveRefreshGuardian, getProactiveRefreshIntervalMs, getRateLimitToastDebounceMs, getRetryAllAccountsMaxRetries, getRetryAllAccountsMaxWaitMs, getRetryAllAccountsRateLimited, getBackgroundResponses, getResponseContinuation, getServerErrorCooldownMs, getSessionAffinity, getSessionAffinityMaxEntries, getSessionAffinityTtlMs, getSessionRecovery, getStorageBackupEnabled, getStreamStallTimeoutMs, getToastDurationMs, getTokenRefreshSkewMs, getUnsupportedCodexFallbackChain, getUnsupportedCodexPolicy, loadPluginConfig, } from "./lib/config.js";
 import { ACCOUNT_LIMITS, AUTH_LABELS, CODEX_BASE_URL, DUMMY_API_KEY, LOG_STAGES, PLUGIN_NAME, PROVIDER_ID, } from "./lib/constants.js";
 import { handleContextOverflow } from "./lib/context-overflow.js";
 import { EntitlementCache, resolveEntitlementAccountKey, } from "./lib/entitlement-cache.js";
@@ -44,9 +44,11 @@ import { detectErrorType, getRecoveryToastContent, isRecoverableError, } from ".
 import { RefreshGuardian } from "./lib/refresh-guardian.js";
 import { queuedRefresh } from "./lib/refresh-queue.js";
 import { parseEnvInt, parseFailoverMode, } from "./lib/request/failover-config.js";
+import { capStreamFailoverMax, computeOutboundRequestAttemptBudget, } from "./lib/request/request-attempt-budget.js";
+import { armPoolExhaustionCooldown, buildAdaptiveStreamFailoverCandidateOrder, clearPoolExhaustionCooldown, clearServerBurstCooldown, getPoolExhaustionCooldownRemaining, getServerBurstCooldownRemaining, recordServerBurstFailure, } from "./lib/request/request-resilience.js";
 import { evaluateFailurePolicy, } from "./lib/request/failure-policy.js";
 import { applyProxyCompatibleInit, createCodexHeaders, extractRequestUrl, getUnsupportedCodexModelInfo, handleErrorResponse, handleSuccessResponse, isWorkspaceDisabledError, refreshAndUpdateToken, resolveUnsupportedCodexFallbackModel, rewriteUrlForCodex, shouldRefreshToken, transformRequestForCodex, } from "./lib/request/fetch-helpers.js";
-import { getRateLimitBackoff, RATE_LIMIT_SHORT_RETRY_THRESHOLD_MS, resetRateLimitBackoff, } from "./lib/request/rate-limit-backoff.js";
+import { configureRateLimitBackoff, getRateLimitBackoff, MAX_SHORT_RETRY_ATTEMPTS, getRateLimitShortRetryThresholdMs as getConfiguredRateLimitShortRetryThresholdMs, resetRateLimitBackoff, } from "./lib/request/rate-limit-backoff.js";
 import { normalizeRequestInit, parseRequestBodyFromInit, } from "./lib/request/request-init.js";
 import { applyFastSessionDefaults } from "./lib/request/request-transformer.js";
 import { applyResponseCompaction } from "./lib/request/response-compaction.js";
@@ -74,6 +76,7 @@ import { applyLoaderRuntimeSetup } from "./lib/runtime/loader-setup.js";
 import { buildManualOAuthFlow } from "./lib/runtime/manual-oauth-flow.js";
 import { applyPreemptiveQuotaSettingsFromConfig, } from "./lib/runtime/quota-settings.js";
 import { ensureLiveAccountSyncState, ensureRefreshGuardianState, ensureSessionAffinityState, } from "./lib/runtime/runtime-services.js";
+import { mutateRuntimeObservabilitySnapshot } from "./lib/runtime/runtime-observability.js";
 import { applyAccountStorageScopeFromConfig } from "./lib/runtime/storage-scope.js";
 import { showRuntimeToast } from "./lib/runtime/toast.js";
 import { createRuntimeSessionRecoveryHook } from "./lib/runtime/session-recovery.js";
@@ -113,10 +116,12 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
     let perProjectStorageWarningShown = false;
     let liveAccountSync = null;
     let liveAccountSyncPath = null;
+    let liveAccountSyncConfigKey = null;
     let refreshGuardian = null;
     let refreshGuardianConfigKey = null;
     let refreshGuardianCleanupRegistered = false;
     let sessionAffinityStore = new SessionAffinityStore();
+    let sessionAffinityWriteVersion = 0;
     let sessionAffinityConfigKey = null;
     const entitlementCache = new EntitlementCache();
     const preemptiveQuotaScheduler = new PreemptiveQuotaScheduler();
@@ -139,6 +144,14 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
         totalRequests: 0,
         successfulRequests: 0,
         failedRequests: 0,
+        responsesRequests: 0,
+        authRefreshRequests: 0,
+        diagnosticProbeRequests: 0,
+        outboundRequestAttemptBudget: null,
+        outboundRequestAttemptsConsumed: 0,
+        requestAttemptBudgetExhaustions: 0,
+        poolExhaustionFastFails: 0,
+        serverBurstFastFails: 0,
         rateLimitedResponses: 0,
         serverErrors: 0,
         networkErrors: 0,
@@ -148,6 +161,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
         accountRotations: 0,
         sameAccountRetries: 0,
         streamFailoverAttempts: 0,
+        streamFailoverCandidatesConsidered: 0,
+        lastStreamFailoverCandidateCount: 0,
         streamFailoverRecoveries: 0,
         streamFailoverCrossAccountRecoveries: 0,
         cumulativeLatencyMs: 0,
@@ -169,6 +184,26 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
         findMatchingAccountIndex,
         modelFamilies: MODEL_FAMILIES,
     });
+    const syncRuntimeObservability = (requestId) => {
+        mutateRuntimeObservabilitySnapshot((snapshot) => {
+            const now = Date.now();
+            const poolRemaining = getPoolExhaustionCooldownRemaining(now);
+            const burstRemaining = getServerBurstCooldownRemaining(now);
+            snapshot.currentRequestId = requestId;
+            snapshot.poolExhaustionCooldownUntil =
+                poolRemaining > 0
+                    ? now + poolRemaining
+                    : null;
+            snapshot.serverBurstCooldownUntil =
+                burstRemaining > 0
+                    ? now + burstRemaining
+                    : null;
+            snapshot.responsesRequests = runtimeMetrics.responsesRequests;
+            snapshot.authRefreshRequests = runtimeMetrics.authRefreshRequests;
+            snapshot.diagnosticProbeRequests = runtimeMetrics.diagnosticProbeRequests;
+            snapshot.runtimeMetrics = { ...runtimeMetrics };
+        });
+    };
     const persistAccountPoolAndFlagged = async (results, flaggedStorage, replaceAll = false) => withAccountAndFlaggedStorageTransaction(async (loadedStorage, persist) => {
         await persistAccountPoolResults({
             results,
@@ -228,7 +263,10 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
             authFallback,
             currentSync: liveAccountSync,
             currentPath: liveAccountSyncPath,
+            currentConfigKey: liveAccountSyncConfigKey,
             getLiveAccountSync,
+            getLiveAccountSyncDebounceMs,
+            getLiveAccountSyncPollMs,
             getStoragePath,
             createSync: (oauthFallback) => new LiveAccountSync(async () => {
                 await reloadRuntimeAccountManager(makeReloadAccountManagerDeps(oauthFallback));
@@ -243,6 +281,7 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
         });
         liveAccountSync = next.liveAccountSync;
         liveAccountSyncPath = next.liveAccountSyncPath;
+        liveAccountSyncConfigKey = next.liveAccountSyncConfigKey;
     };
     const ensureRefreshGuardian = (pluginConfig) => {
         const next = ensureRefreshGuardianState({
@@ -387,6 +426,12 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                     const fastSessionStrategy = getFastSessionStrategy(pluginConfig);
                     const fastSessionMaxInputItems = getFastSessionMaxInputItems(pluginConfig);
                     const tokenRefreshSkewMs = getTokenRefreshSkewMs(pluginConfig);
+                    configureRateLimitBackoff({
+                        dedupWindowMs: getRateLimitDedupWindowMs(pluginConfig),
+                        stateResetMs: getRateLimitStateResetMs(pluginConfig),
+                        maxBackoffMs: getRateLimitMaxBackoffMs(pluginConfig),
+                        shortRetryThresholdMs: getRateLimitShortRetryThresholdMs(pluginConfig),
+                    });
                     const rateLimitToastDebounceMs = getRateLimitToastDebounceMs(pluginConfig);
                     const retryAllAccountsRateLimited = getRetryAllAccountsRateLimited(pluginConfig);
                     const retryAllAccountsMaxWaitMs = getRetryAllAccountsMaxWaitMs(pluginConfig);
@@ -401,8 +446,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                     const networkErrorCooldownMs = getNetworkErrorCooldownMs(pluginConfig);
                     const serverErrorCooldownMs = getServerErrorCooldownMs(pluginConfig);
                     const failoverMode = parseFailoverMode(process.env.CODEX_AUTH_FAILOVER_MODE);
-                    const streamFailoverMax = Math.max(0, parseEnvInt(process.env.CODEX_AUTH_STREAM_FAILOVER_MAX) ??
-                        STREAM_FAILOVER_MAX_BY_MODE[failoverMode]);
+                    const streamFailoverMax = Math.max(0, capStreamFailoverMax(parseEnvInt(process.env.CODEX_AUTH_STREAM_FAILOVER_MAX) ??
+                        STREAM_FAILOVER_MAX_BY_MODE[failoverMode]));
                     const streamFailoverSoftTimeoutMs = Math.max(1_000, parseEnvInt(process.env.CODEX_AUTH_STREAM_STALL_SOFT_TIMEOUT_MS) ??
                         STREAM_FAILOVER_SOFT_TIMEOUT_BY_MODE[failoverMode]);
                     const streamFailoverHardTimeoutMs = Math.max(streamFailoverSoftTimeoutMs, parseEnvInt(process.env.CODEX_AUTH_STREAM_STALL_HARD_TIMEOUT_MS) ??
@@ -508,6 +553,7 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                     .toString()
                                     .trim() || undefined;
                                 const sessionAffinityKey = threadIdCandidate ?? promptCacheKey ?? null;
+                                const sessionAffinityVersion = (sessionAffinityWriteVersion += 1);
                                 const effectivePromptCacheKey = (sessionAffinityKey ?? promptCacheKey ?? "")
                                     .toString()
                                     .trim() || undefined;
@@ -532,9 +578,72 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                 const requestCorrelationId = setCorrelationId(threadIdCandidate
                                     ? `${threadIdCandidate}:${Date.now()}`
                                     : undefined);
+                                const requestTraceId = requestCorrelationId;
                                 runtimeMetrics.lastRequestAt = Date.now();
+                                syncRuntimeObservability(requestTraceId);
                                 const abortSignal = requestInit?.signal ?? init?.signal ?? null;
                                 const sleep = createAbortableSleep(abortSignal);
+                                const poolCooldownRemainingMs = getPoolExhaustionCooldownRemaining();
+                                if (poolCooldownRemainingMs > 0) {
+                                    runtimeMetrics.failedRequests += 1;
+                                    runtimeMetrics.poolExhaustionFastFails += 1;
+                                    runtimeMetrics.lastError = "Pool exhaustion cooldown active";
+                                    syncRuntimeObservability(requestTraceId);
+                                    return new Response(JSON.stringify({
+                                        error: {
+                                            message: `The account pool is cooling down after recent rate-limit exhaustion. Try again in ${formatWaitTime(poolCooldownRemainingMs)} or inspect \`codex auth status\`.`,
+                                        },
+                                    }), { status: 429, headers: { "content-type": "application/json; charset=utf-8" } });
+                                }
+                                const serverBurstCooldownRemainingMs = getServerBurstCooldownRemaining();
+                                if (serverBurstCooldownRemainingMs > 0) {
+                                    runtimeMetrics.failedRequests += 1;
+                                    runtimeMetrics.serverBurstFastFails += 1;
+                                    runtimeMetrics.lastError = "Server burst cooldown active";
+                                    syncRuntimeObservability(requestTraceId);
+                                    return new Response(JSON.stringify({
+                                        error: {
+                                            message: `Multiple accounts recently failed with upstream server errors. Try again in ${formatWaitTime(serverBurstCooldownRemainingMs)} or inspect \`codex auth report --json\`.`,
+                                        },
+                                    }), { status: 503, headers: { "content-type": "application/json; charset=utf-8" } });
+                                }
+                                const maxOutboundRequestAttempts = computeOutboundRequestAttemptBudget({
+                                    accountCount: accountManager.getAccountCount(),
+                                    maxSameAccountRetries,
+                                    emptyResponseMaxRetries,
+                                    streamFailoverMax,
+                                });
+                                runtimeMetrics.outboundRequestAttemptBudget ??=
+                                    maxOutboundRequestAttempts;
+                                logDebug("Configured outbound request attempt budget.", {
+                                    requestTraceId,
+                                    budget: maxOutboundRequestAttempts,
+                                    accountCount: accountManager.getAccountCount(),
+                                    maxSameAccountRetries,
+                                    emptyResponseMaxRetries,
+                                    streamFailoverMax,
+                                });
+                                syncRuntimeObservability(requestTraceId);
+                                let outboundRequestAttemptsRemaining = maxOutboundRequestAttempts;
+                                const tryConsumeOutboundRequestAttempt = (reason, accountIndex) => {
+                                    if (outboundRequestAttemptsRemaining <= 0) {
+                                        runtimeMetrics.requestAttemptBudgetExhaustions += 1;
+                                        runtimeMetrics.lastError =
+                                            `Request attempt budget exhausted after ${maxOutboundRequestAttempts} outbound request(s)`;
+                                        logWarn("Request attempt budget exhausted.", {
+                                            requestTraceId,
+                                            reason,
+                                            accountIndex,
+                                            budget: maxOutboundRequestAttempts,
+                                            consumed: maxOutboundRequestAttempts,
+                                        });
+                                        syncRuntimeObservability(requestTraceId);
+                                        return false;
+                                    }
+                                    runtimeMetrics.outboundRequestAttemptsConsumed += 1;
+                                    outboundRequestAttemptsRemaining -= 1;
+                                    return true;
+                                };
                                 let allRateLimitedRetries = 0;
                                 let emptyResponseRetries = 0;
                                 const attemptedUnsupportedFallbackModels = new Set();
@@ -599,6 +708,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                         let accountAuth = accountManager.toAuthDetails(account);
                                         try {
                                             if (shouldRefreshToken(accountAuth, tokenRefreshSkewMs)) {
+                                                runtimeMetrics.authRefreshRequests += 1;
+                                                syncRuntimeObservability(requestTraceId);
                                                 accountAuth = (await refreshAndUpdateToken(accountAuth, client));
                                                 await accountManager.commitRefreshedAuth(account, accountAuth);
                                             }
@@ -748,9 +859,23 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                             continue;
                                         }
                                         let sameAccountRetryCount = 0;
+                                        let shortRateLimitRetryCount = 0;
                                         let successAccountForResponse = account;
                                         let successEntitlementAccountKey = entitlementAccountKey;
                                         while (true) {
+                                            if (!tryConsumeOutboundRequestAttempt("primary", account.index)) {
+                                                runtimeMetrics.failedRequests++;
+                                                const lastErrorDetail = runtimeMetrics.lastError;
+                                                const message = lastErrorDetail
+                                                    ? `${lastErrorDetail}. Try again after the current retries settle.`
+                                                    : "Request attempt budget exhausted. Try again shortly.";
+                                                return new Response(JSON.stringify({ error: { message } }), {
+                                                    status: 503,
+                                                    headers: {
+                                                        "content-type": "application/json; charset=utf-8",
+                                                    },
+                                                });
+                                            }
                                             let response;
                                             const fetchStart = performance.now();
                                             // Merge user AbortSignal with timeout (Node 18 compatible - no AbortSignal.any)
@@ -777,6 +902,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                             }
                                             try {
                                                 runtimeMetrics.totalRequests++;
+                                                runtimeMetrics.responsesRequests++;
+                                                syncRuntimeObservability(requestTraceId);
                                                 response = await fetch(url, applyProxyCompatibleInit(url, {
                                                     ...requestInit,
                                                     headers,
@@ -1049,6 +1176,23 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                         failoverMode,
                                                         serverRetryAfterMs: serverRetryAfterMs ?? undefined,
                                                     }, { serverCooldownMs: serverErrorCooldownMs });
+                                                    const serverBurstCooldownUntil = recordServerBurstFailure(account.index);
+                                                    if (serverBurstCooldownUntil > 0) {
+                                                        runtimeMetrics.lastError =
+                                                            "Repeated upstream server errors across the account pool";
+                                                        syncRuntimeObservability(requestTraceId);
+                                                    }
+                                                    // Overload-type server errors (502 Bad Gateway, 503 Service
+                                                    // Unavailable, 529 Overloaded) signal upstream capacity
+                                                    // pressure. Notify the quota scheduler so it can proactively
+                                                    // defer subsequent requests for this quota key, mirroring the
+                                                    // 429 handler's scheduler awareness.
+                                                    if ((response.status === 502 ||
+                                                        response.status === 503 ||
+                                                        response.status === 529) &&
+                                                        typeof policy.cooldownMs === "number") {
+                                                        preemptiveQuotaScheduler.markRateLimited(quotaScheduleKey, policy.cooldownMs);
+                                                    }
                                                     if (policy.refundToken) {
                                                         accountManager.refundToken(account, modelFamily, model);
                                                     }
@@ -1072,16 +1216,20 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                     sessionAffinityStore?.forgetSession(sessionAffinityKey);
                                                     break;
                                                 }
-                                                if (response.status === 429) {
+                                                if (errorResponse.status === 429 && rateLimit) {
                                                     runtimeMetrics.rateLimitedResponses++;
                                                     const retryAfterMs = rateLimit?.retryAfterMs ?? 60_000;
-                                                    const { attempt, delayMs } = getRateLimitBackoff(account.index, quotaKey, retryAfterMs);
+                                                    const { delayMs } = getRateLimitBackoff(account.index, quotaKey, retryAfterMs, account.accountId ?? account.email ?? null);
                                                     const cooldownMs = Math.max(delayMs, retryAfterMs);
                                                     preemptiveQuotaScheduler.markRateLimited(quotaScheduleKey, cooldownMs);
                                                     const waitLabel = formatWaitTime(cooldownMs);
-                                                    if (cooldownMs <= RATE_LIMIT_SHORT_RETRY_THRESHOLD_MS) {
+                                                    const shortRetryThresholdMs = getConfiguredRateLimitShortRetryThresholdMs();
+                                                    if (cooldownMs <=
+                                                        shortRetryThresholdMs &&
+                                                        shortRateLimitRetryCount < MAX_SHORT_RETRY_ATTEMPTS) {
+                                                        shortRateLimitRetryCount += 1;
                                                         if (accountManager.shouldShowAccountToast(account.index, rateLimitToastDebounceMs)) {
-                                                            await showRuntimeToast(client, `Rate limited. Retrying in ${waitLabel} (attempt ${attempt})...`, "warning", { duration: toastDurationMs });
+                                                            await showRuntimeToast(client, `Rate limited. Retrying in ${waitLabel} (attempt ${shortRateLimitRetryCount})...`, "warning", { duration: toastDurationMs });
                                                             accountManager.markToastShown(account.index);
                                                         }
                                                         await sleep(addJitter(Math.max(MIN_BACKOFF_MS, cooldownMs), 0.2));
@@ -1114,13 +1262,18 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                             runtimeMetrics.cumulativeLatencyMs += fetchLatencyMs;
                                             let responseForSuccess = response;
                                             if (isStreaming) {
-                                                const streamFallbackCandidateOrder = [
-                                                    account.index,
-                                                    ...accountManager
-                                                        .getAccountsSnapshot()
-                                                        .map((candidate) => candidate.index)
-                                                        .filter((index) => index !== account.index),
-                                                ];
+                                                const streamFallbackCandidateOrder = buildAdaptiveStreamFailoverCandidateOrder(account.index, accountSnapshotList);
+                                                runtimeMetrics.lastStreamFailoverCandidateCount =
+                                                    streamFallbackCandidateOrder.length;
+                                                runtimeMetrics.streamFailoverCandidatesConsidered +=
+                                                    streamFallbackCandidateOrder.length;
+                                                logDebug("Prepared stream failover candidates.", {
+                                                    requestTraceId,
+                                                    primaryAccountIndex: account.index,
+                                                    candidateCount: streamFallbackCandidateOrder.length,
+                                                    candidateIndices: streamFallbackCandidateOrder,
+                                                });
+                                                syncRuntimeObservability(requestTraceId);
                                                 responseForSuccess = withStreamingFailover(response, async (failoverAttempt, emittedBytes) => {
                                                     if (abortSignal?.aborted) {
                                                         return null;
@@ -1139,6 +1292,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                         let fallbackAuth = accountManager.toAuthDetails(fallbackAccount);
                                                         try {
                                                             if (shouldRefreshToken(fallbackAuth, tokenRefreshSkewMs)) {
+                                                                runtimeMetrics.authRefreshRequests += 1;
+                                                                syncRuntimeObservability(requestTraceId);
                                                                 fallbackAuth = (await refreshAndUpdateToken(fallbackAuth, client));
                                                                 await accountManager.commitRefreshedAuth(fallbackAccount, fallbackAuth);
                                                             }
@@ -1182,6 +1337,9 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                             logWarn(`Skipping account ${fallbackAccount.index + 1} due to cached entitlement block (${formatWaitTime(fallbackEntitlementBlock.waitMs)} remaining).`);
                                                             continue;
                                                         }
+                                                        if (!tryConsumeOutboundRequestAttempt("stream-failover", fallbackAccount.index)) {
+                                                            return null;
+                                                        }
                                                         if (!accountManager.consumeToken(fallbackAccount, modelFamily, model)) {
                                                             continue;
                                                         }
@@ -1212,6 +1370,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                         }
                                                         try {
                                                             runtimeMetrics.totalRequests++;
+                                                            runtimeMetrics.responsesRequests++;
+                                                            syncRuntimeObservability(requestTraceId);
                                                             const fallbackResponse = await fetch(url, applyProxyCompatibleInit(url, {
                                                                 ...requestInit,
                                                                 headers: fallbackHeaders,
@@ -1253,7 +1413,7 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                                 runtimeMetrics.streamFailoverCrossAccountRecoveries += 1;
                                                                 runtimeMetrics.accountRotations += 1;
                                                                 if (!responseContinuationEnabled) {
-                                                                    sessionAffinityStore?.remember(sessionAffinityKey, fallbackAccount.index);
+                                                                    sessionAffinityStore?.rememberWithVersion(sessionAffinityKey, fallbackAccount.index, Date.now(), sessionAffinityVersion);
                                                                 }
                                                             }
                                                             logInfo(`Recovered stream via failover attempt ${failoverAttempt} using account ${fallbackAccount.index + 1}.`, { emittedBytes });
@@ -1290,8 +1450,8 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                 onResponseId: (responseId) => {
                                                     if (!responseContinuationEnabled)
                                                         return;
-                                                    sessionAffinityStore?.remember(sessionAffinityKey, successAccountForResponse.index);
-                                                    sessionAffinityStore?.updateLastResponseId(sessionAffinityKey, responseId);
+                                                    sessionAffinityStore?.rememberWithVersion(sessionAffinityKey, successAccountForResponse.index, Date.now(), sessionAffinityVersion);
+                                                    sessionAffinityStore?.updateLastResponseId(sessionAffinityKey, responseId, Date.now(), sessionAffinityVersion);
                                                     storedResponseIdForSuccess = true;
                                                 },
                                                 streamStallTimeoutMs,
@@ -1331,6 +1491,20 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                             await sleep(addJitter(emptyResponseRetryDelayMs, 0.2));
                                                             break;
                                                         }
+                                                        if (response.status >= 500) {
+                                                            const serverBurstCooldownUntil = recordServerBurstFailure(account.index);
+                                                            if (serverBurstCooldownUntil > 0) {
+                                                                runtimeMetrics.serverBurstFastFails += 1;
+                                                                runtimeMetrics.lastError =
+                                                                    "Repeated upstream server errors across the account pool";
+                                                                syncRuntimeObservability(requestTraceId);
+                                                                return new Response(JSON.stringify({
+                                                                    error: {
+                                                                        message: `Upstream server failures were observed across multiple accounts. Pausing retries for ${formatWaitTime(serverBurstCooldownUntil - Date.now())}. Check \`codex auth report --json\` for runtime metrics.`,
+                                                                    },
+                                                                }), { status: 503, headers: { "content-type": "application/json; charset=utf-8" } });
+                                                            }
+                                                        }
                                                         logWarn(`Empty response after ${emptyResponseMaxRetries} retries. Returning as-is.`);
                                                     }
                                                 }
@@ -1347,7 +1521,7 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                             entitlementCache.clear(successAccountKey, capabilityModelKey);
                                             if (!responseContinuationEnabled ||
                                                 (!isStreaming && !storedResponseIdForSuccess)) {
-                                                sessionAffinityStore?.remember(sessionAffinityKey, successAccountForResponse.index);
+                                                sessionAffinityStore?.rememberWithVersion(sessionAffinityKey, successAccountForResponse.index, Date.now(), sessionAffinityVersion);
                                             }
                                             runtimeMetrics.successfulRequests++;
                                             runtimeMetrics.lastError = null;
@@ -1357,6 +1531,9 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                                 lastCodexCliActiveSyncIndex =
                                                     successAccountForResponse.index;
                                             }
+                                            clearPoolExhaustionCooldown();
+                                            clearServerBurstCooldown();
+                                            syncRuntimeObservability(requestTraceId);
                                             return successResponse;
                                         }
                                         if (retryNextAccountBeforeFallback) {
@@ -1391,16 +1568,26 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                         allRateLimitedRetries++;
                                         continue;
                                     }
-                                    const waitLabel = waitMs > 0 ? formatWaitTime(waitMs) : "a bit";
+                                    const now = Date.now();
+                                    const poolCooldownUntil = count > 0 && waitMs > 0
+                                        ? armPoolExhaustionCooldown(waitMs, now)
+                                        : 0;
+                                    const effectiveWaitMs = poolCooldownUntil > 0
+                                        ? Math.max(0, poolCooldownUntil - now)
+                                        : waitMs;
+                                    const waitLabel = effectiveWaitMs > 0
+                                        ? formatWaitTime(effectiveWaitMs)
+                                        : "a bit";
                                     const message = count === 0
                                         ? "No Codex accounts configured. Run `codex login`."
-                                        : waitMs > 0
-                                            ? `All ${count} account(s) are rate-limited. Try again in ${waitLabel} or add another account with \`codex login\`.`
-                                            : `All ${count} account(s) failed (server errors or auth issues). Check account health with \`codex-health\`.`;
+                                        : effectiveWaitMs > 0
+                                            ? `All ${count} account(s) are rate-limited. A short pool cooldown is now active for ${waitLabel}. Try again later or inspect \`codex auth status\`.`
+                                            : `All ${count} account(s) failed (server errors or auth issues). Check account health with \`codex auth report --json\`.`;
                                     runtimeMetrics.failedRequests++;
                                     runtimeMetrics.lastError = message;
+                                    syncRuntimeObservability(requestTraceId);
                                     return new Response(JSON.stringify({ error: { message } }), {
-                                        status: waitMs > 0 ? 429 : 503,
+                                        status: effectiveWaitMs > 0 ? 429 : 503,
                                         headers: {
                                             "content-type": "application/json; charset=utf-8",
                                         },
@@ -1408,6 +1595,7 @@ export const OpenAIOAuthPlugin = async ({ client }) => {
                                 }
                             }
                             finally {
+                                syncRuntimeObservability(null);
                                 clearCorrelationId();
                             }
                         },