pi-antigravity-rotator 1.9.0 → 1.9.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,24 @@
1
1
  # Changelog
2
2
 
3
+ ## [1.9.2] - 2026-04-29
4
+
5
+ ### Fixed
6
+ - **Project Discovery Without Shared Fallback**: Login now fails if Google does not return a companion project ID. No more shared `rising-fact-p41fc` fallback.
7
+ - **Activation Hint**: Login/discovery errors now tell you to open the account in Antigravity IDE and send one message first.
8
+ - **Activation Docs**: README now documents the first-use activation rule for new accounts.
9
+
10
+ ## [1.9.1] - 2026-04-29
11
+
12
+ ### Fixed
13
+ - **429 Account-Safety Backoff**: All provider-side `429` responses now stop the current request instead of immediately retrying another account. This prevents cascade-burning the full pool when Google rate-limits a shared project/request bucket. `RESOURCE_EXHAUSTED` gets a 30-minute cooldown; other 429s use parsed `Retry-After`/retry-delay.
14
+ - **Stream Idle Crash Safety**: Stream idle timeout now closes cleanly without emitting an unhandled stream error that could crash-loop systemd.
15
+
16
+ ### Added
17
+ - **Project Circuit Breaker**: If multiple accounts sharing a `projectId` hit provider `429` for the same quota model inside a rolling window, routing pauses that `projectId`/model instead of burning sibling accounts.
18
+ - **Daily Safety Budgets**: Per-account and per-`projectId` daily upstream attempt counters now trigger slow-mode jitter and hard stops until the next UTC day.
19
+ - **Project Concurrency Guard**: Added `maxConcurrentRequestsPerProjectModel` to prevent simultaneous calls through multiple accounts backed by the same provider project bucket.
20
+ - **Large Context Warning**: Requests above 1 MiB now log a warning because huge contexts increase rate-limit and flag pressure.
21
+
3
22
  ## [1.9.0] - 2026-04-29
4
23
 
5
24
  ### Added
package/README.md CHANGED
@@ -9,7 +9,7 @@ Multi-account rotation proxy for Google Antigravity. Distributes API usage acros
9
9
  - **Per-model timer tracking** -- Timer classification (`fresh`/`7d`/`5h`) is evaluated per model using each model's actual `resetTime` from the quota API, not a per-account estimate
10
10
  - **Smart rotation** -- Rotates only the specific model whose quota dropped, leaving other models on their current accounts
11
11
  - **Infringement detection** -- On 403 with infringement/abuse/suspension keywords, the account is immediately flagged and excluded from routing
12
- - **Automatic failover** -- On 429 rate limits, instantly switches the affected model to the next available account
12
+ - **Safer 429 handling** -- On provider `429`, stops the current request and avoids cascade-burning sibling accounts
13
13
  - **Concurrency guardrails** -- Limits each account to one in-flight request by default to avoid bursty pressure
14
14
  - **Operator fresh-window controls** -- You can block new `fresh` window starts globally, then selectively allow specific accounts to override that policy
15
15
  - **Protective pause** -- Pauses all routing for several hours after serious ToS/abuse-style flags so the rest of the pool is not burned
@@ -57,6 +57,7 @@ Run `npm run login` once per Google account:
57
57
  2. Complete the sign-in and grant permissions
58
58
  3. The browser redirects to a `localhost` URL that won't load -- this is expected
59
59
  4. Copy the **full URL** from the browser's address bar and paste it into the terminal
60
+ 5. If project discovery fails, open that same account in Antigravity IDE, send one message, then rerun login
60
61
 
61
62
  The tool automatically:
62
63
 
@@ -66,6 +67,15 @@ The tool automatically:
66
67
 
67
68
  Re-running with the same email updates the existing entry.
68
69
 
70
+ ### Activation rule
71
+
72
+ Some accounts do not expose a discoverable `projectId` until they are used once in the Antigravity IDE.
73
+ If login fails at project discovery:
74
+
75
+ 1. Open that exact Google account in Antigravity IDE.
76
+ 2. Send one message.
77
+ 3. Rerun `npm run login`.
78
+
69
79
  ## Dashboard
70
80
 
71
81
  After starting the proxy, open `http://localhost:51200/dashboard` or `http://<your-server-ip>:51200/dashboard` from any machine on the same network (the proxy binds to `0.0.0.0`).
@@ -213,7 +223,8 @@ export PI_AI_ANTIGRAVITY_VERSION=1.107.0
213
223
  pi-antigravity-rotator start --config-dir /path/to/config
214
224
  ```
215
225
 
216
- `accounts.json` is created automatically by the login command:
226
+ `accounts.json` is created automatically by the login command.
227
+ Login now fails if Google does not return a project ID. No shared fallback.
217
228
 
218
229
  ```json
219
230
  {
@@ -222,6 +233,16 @@ pi-antigravity-rotator start --config-dir /path/to/config
222
233
  "rotateOnQuotaDrop": 20,
223
234
  "quotaPollIntervalMs": 300000,
224
235
  "maxConcurrentRequestsPerAccount": 1,
236
+ "maxConcurrentRequestsPerProjectModel": 1,
237
+ "projectCircuitBreaker429Threshold": 3,
238
+ "projectCircuitBreakerWindowMs": 600000,
239
+ "projectCircuitBreakerCooldownMs": 3600000,
240
+ "dailyAccountSlowRequests": 250,
241
+ "dailyAccountStopRequests": 350,
242
+ "dailyProjectSlowRequests": 900,
243
+ "dailyProjectStopRequests": 1200,
244
+ "slowModeJitterMinMs": 8000,
245
+ "slowModeJitterMaxMs": 25000,
225
246
  "protectivePauseMs": 21600000,
226
247
  "useRequestCountRotationWhenQuotaUnknownOnly": true,
227
248
  "accounts": [
@@ -244,6 +265,16 @@ pi-antigravity-rotator start --config-dir /path/to/config
244
265
  | `rotateOnQuotaDrop` | `20` | Rotate when a model's quota drops this many %. Set to `0` to disable |
245
266
  | `quotaPollIntervalMs` | `300000` | Quota poll interval in ms (5 minutes) |
246
267
  | `maxConcurrentRequestsPerAccount` | `1` | Max simultaneous requests allowed per account |
268
+ | `maxConcurrentRequestsPerProjectModel` | `1` | Max simultaneous requests allowed across accounts sharing the same `projectId` for the same quota model |
269
+ | `projectCircuitBreaker429Threshold` | `3` | Unique accounts from the same `projectId` that must hit provider `429` before pausing that project/model |
270
+ | `projectCircuitBreakerWindowMs` | `600000` | Rolling window for the project/model `429` circuit breaker |
271
+ | `projectCircuitBreakerCooldownMs` | `3600000` | Minimum project/model pause after the circuit breaker trips |
272
+ | `dailyAccountSlowRequests` | `250` | Daily upstream attempts per account before slow-mode jitter starts |
273
+ | `dailyAccountStopRequests` | `350` | Daily upstream attempts per account before routing stops for that account until the next UTC day |
274
+ | `dailyProjectSlowRequests` | `900` | Daily upstream attempts per `projectId` before slow-mode jitter starts |
275
+ | `dailyProjectStopRequests` | `1200` | Daily upstream attempts per `projectId` before routing stops for that project until the next UTC day |
276
+ | `slowModeJitterMinMs` | `8000` | Minimum slow-mode delay before upstream request |
277
+ | `slowModeJitterMaxMs` | `25000` | Maximum slow-mode delay before upstream request |
247
278
  | `protectivePauseMs` | `21600000` | Global routing pause after a serious provider enforcement signal |
248
279
  | `useRequestCountRotationWhenQuotaUnknownOnly` | `true` | Use request-count rotation only until quota telemetry exists for the request's model. Set to `false` to keep rotating by request count even with known quotas |
249
280
 
@@ -253,7 +284,8 @@ pi-antigravity-rotator start --config-dir /path/to/config
253
284
  |-------|-------------|
254
285
  | `email` | Google account email (auto-filled by login) |
255
286
  | `refreshToken` | OAuth refresh token (auto-filled by login) |
256
- | `projectId` | Cloud project ID (auto-discovered during login) |
287
+ | `projectId` | Cloud project ID discovered from Google during login |
288
+ | `projectSource` | Optional metadata: `google` when discovered from Google, `manual` if edited by hand |
257
289
  | `label` | Display name on the dashboard (auto-filled, defaults to email username) |
258
290
 
259
291
  ## API Endpoints
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-antigravity-rotator",
3
- "version": "1.9.0",
3
+ "version": "1.9.2",
4
4
  "description": "Multi-account rotation proxy for Google Antigravity with per-model routing, real-time quota tracking, and infringement detection",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -23,6 +23,16 @@ export function loadOrCreateAccountsConfig(): Config {
23
23
  rotateOnQuotaDrop: 20,
24
24
  quotaPollIntervalMs: 300000,
25
25
  maxConcurrentRequestsPerAccount: 1,
26
+ maxConcurrentRequestsPerProjectModel: 1,
27
+ projectCircuitBreaker429Threshold: 3,
28
+ projectCircuitBreakerWindowMs: 10 * 60 * 1000,
29
+ projectCircuitBreakerCooldownMs: 60 * 60 * 1000,
30
+ dailyAccountSlowRequests: 250,
31
+ dailyAccountStopRequests: 350,
32
+ dailyProjectSlowRequests: 900,
33
+ dailyProjectStopRequests: 1200,
34
+ slowModeJitterMinMs: 8_000,
35
+ slowModeJitterMaxMs: 25_000,
26
36
  protectivePauseMs: 21600000,
27
37
  useRequestCountRotationWhenQuotaUnknownOnly: true,
28
38
  accounts: [],
package/src/index.ts CHANGED
@@ -38,6 +38,16 @@ function loadConfig(): Config {
38
38
  config.rotateOnQuotaDrop = config.rotateOnQuotaDrop ?? 20;
39
39
  config.quotaPollIntervalMs = config.quotaPollIntervalMs || 300_000;
40
40
  config.maxConcurrentRequestsPerAccount = config.maxConcurrentRequestsPerAccount ?? 1;
41
+ config.maxConcurrentRequestsPerProjectModel = config.maxConcurrentRequestsPerProjectModel ?? 1;
42
+ config.projectCircuitBreaker429Threshold = config.projectCircuitBreaker429Threshold ?? 3;
43
+ config.projectCircuitBreakerWindowMs = config.projectCircuitBreakerWindowMs ?? 10 * 60 * 1000;
44
+ config.projectCircuitBreakerCooldownMs = config.projectCircuitBreakerCooldownMs ?? 60 * 60 * 1000;
45
+ config.dailyAccountSlowRequests = config.dailyAccountSlowRequests ?? 250;
46
+ config.dailyAccountStopRequests = config.dailyAccountStopRequests ?? 350;
47
+ config.dailyProjectSlowRequests = config.dailyProjectSlowRequests ?? 900;
48
+ config.dailyProjectStopRequests = config.dailyProjectStopRequests ?? 1200;
49
+ config.slowModeJitterMinMs = config.slowModeJitterMinMs ?? 8_000;
50
+ config.slowModeJitterMaxMs = config.slowModeJitterMaxMs ?? 25_000;
41
51
  config.protectivePauseMs = config.protectivePauseMs ?? 6 * 60 * 60 * 1000;
42
52
  config.useRequestCountRotationWhenQuotaUnknownOnly =
43
53
  config.useRequestCountRotationWhenQuotaUnknownOnly ?? true;
@@ -97,7 +107,8 @@ export function main(): void {
97
107
  console.log(`Loaded ${config.accounts.length} accounts`);
98
108
  console.log(`Rotation: ${config.requestsPerRotation} requests / ${config.rotateOnQuotaDrop}% quota drop`);
99
109
  console.log(`Quota poll: every ${Math.round((config.quotaPollIntervalMs || 300000) / 1000)}s`);
100
- console.log(`Concurrency cap: ${config.maxConcurrentRequestsPerAccount} request/account`);
110
+ console.log(`Concurrency cap: ${config.maxConcurrentRequestsPerAccount} request/account, ${config.maxConcurrentRequestsPerProjectModel} request/project+model`);
111
+ console.log(`Safety breaker: ${config.projectCircuitBreaker429Threshold} provider 429s / ${Math.round((config.projectCircuitBreakerWindowMs || 0) / 60000)}m pauses project+model for ${Math.round((config.projectCircuitBreakerCooldownMs || 0) / 60000)}m`);
101
112
  console.log(`Protective pause: ${Math.round((config.protectivePauseMs || 0) / 3600000)}h after serious flag`);
102
113
  console.log();
103
114
 
package/src/login.ts CHANGED
@@ -79,19 +79,21 @@ export async function runLogin(): Promise<void> {
79
79
  const email = await getUserEmail(tokenData.accessToken);
80
80
 
81
81
  console.log("Discovering project...");
82
- const projectId = await discoverProject(tokenData.accessToken);
82
+ const project = await discoverProject(tokenData.accessToken);
83
83
 
84
84
  const label = email ? email.split("@")[0] : "Account";
85
85
  const entry: AccountConfig = {
86
86
  email: email || "unknown@gmail.com",
87
87
  refreshToken: tokenData.refreshToken,
88
- projectId,
88
+ projectId: project.projectId,
89
+ projectSource: project.source,
89
90
  label,
90
91
  };
91
92
 
92
93
  console.log();
93
94
  const { isNew } = addAccountToConfig(entry);
94
95
  console.log(` ${isNew ? "Added" : "Updated"} ${entry.email} in ${ACCOUNTS_FILE}`);
96
+ console.log(` projectId=${project.projectId} (source=${project.source})`);
95
97
 
96
98
  ensurePiModelsConfig();
97
99
  ensurePiAuthConfig();
package/src/oauth.ts CHANGED
@@ -11,7 +11,6 @@ export const SCOPES = [
11
11
  "https://www.googleapis.com/auth/experimentsandconfigs",
12
12
  ];
13
13
  export const AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth";
14
- export const DEFAULT_PROJECT_ID = "rising-fact-p41fc";
15
14
 
16
15
  export interface OAuthClientConfig {
17
16
  clientId: string;
@@ -107,7 +106,13 @@ export async function exchangeAuthorizationCode(code: string, verifier: string):
107
106
  };
108
107
  }
109
108
 
110
- export async function discoverProject(accessToken: string): Promise<string> {
109
+ export interface ProjectDiscoveryResult {
110
+ projectId: string;
111
+ source: "google";
112
+ endpoint: string;
113
+ }
114
+
115
+ export async function discoverProject(accessToken: string): Promise<ProjectDiscoveryResult> {
111
116
  const headers: Record<string, string> = {
112
117
  Authorization: `Bearer ${accessToken}`,
113
118
  "Content-Type": "application/json",
@@ -138,14 +143,14 @@ export async function discoverProject(accessToken: string): Promise<string> {
138
143
  cloudaicompanionProject?: string | { id?: string };
139
144
  };
140
145
  if (typeof data.cloudaicompanionProject === "string" && data.cloudaicompanionProject) {
141
- return data.cloudaicompanionProject;
146
+ return { projectId: data.cloudaicompanionProject, source: "google", endpoint };
142
147
  }
143
148
  if (
144
149
  data.cloudaicompanionProject &&
145
150
  typeof data.cloudaicompanionProject === "object" &&
146
151
  data.cloudaicompanionProject.id
147
152
  ) {
148
- return data.cloudaicompanionProject.id;
153
+ return { projectId: data.cloudaicompanionProject.id, source: "google", endpoint };
149
154
  }
150
155
  }
151
156
  } catch {
@@ -153,7 +158,7 @@ export async function discoverProject(accessToken: string): Promise<string> {
153
158
  }
154
159
  }
155
160
 
156
- return DEFAULT_PROJECT_ID;
161
+ throw new Error("Could not discover Cloud Code companion project ID from Google. If this account is new, open it in Antigravity IDE and send one message first, then retry login. Login failed instead of falling back to a shared projectId.");
157
162
  }
158
163
 
159
164
  export async function getUserEmail(accessToken: string): Promise<string | undefined> {
package/src/onboarding.ts CHANGED
@@ -203,12 +203,13 @@ export async function handleHostedCallback(
203
203
  try {
204
204
  const tokenData = await exchangeAuthorizationCode(code, session.verifier);
205
205
  const email = await getUserEmail(tokenData.accessToken);
206
- const projectId = await discoverProject(tokenData.accessToken);
206
+ const project = await discoverProject(tokenData.accessToken);
207
207
  const label = email ? email.split("@")[0] : "Account";
208
208
  const entry = {
209
209
  email: email || "unknown@gmail.com",
210
210
  refreshToken: tokenData.refreshToken,
211
- projectId,
211
+ projectId: project.projectId,
212
+ projectSource: project.source,
212
213
  label,
213
214
  };
214
215
 
@@ -221,6 +222,7 @@ export async function handleHostedCallback(
221
222
  "Account Connected",
222
223
  `<h1>Account Connected</h1>
223
224
  <p><strong>${entry.email}</strong> was ${isNew ? "added" : "updated"} successfully.</p>
225
+ <p>Project: <span class="mono">${project.projectId}</span> via ${project.source}.</p>
224
226
  <p>The rotator can start using this account immediately.</p>
225
227
  <div class="note">If you ever want to stop sharing access, revoke this app's access from the Google account security settings.</div>`,
226
228
  ),
package/src/proxy.ts CHANGED
@@ -26,7 +26,13 @@ const proxyLogger = logger.child("proxy");
26
26
 
27
27
  const MAX_ENDPOINT_RETRIES = 3;
28
28
  const MAX_COOLDOWN_MS = 30 * 60 * 1000; // 30 minutes max cooldown
29
+ const RESOURCE_EXHAUSTED_COOLDOWN_MS = 30 * 60 * 1000; // Stop hammering provider-side daily/request buckets
29
30
  const STREAM_IDLE_TIMEOUT_MS = 2 * 60 * 1000; // Release account if a stream goes silent.
31
+ const LARGE_CONTEXT_WARN_BYTES = 1 * 1024 * 1024;
32
+
33
+ function sleep(ms: number): Promise<void> {
34
+ return new Promise((resolve) => setTimeout(resolve, ms));
35
+ }
30
36
 
31
37
  interface RequestBody {
32
38
  project: string;
@@ -97,6 +103,11 @@ function capCooldown(ms: number): number {
97
103
  return Math.min(ms, MAX_COOLDOWN_MS);
98
104
  }
99
105
 
106
+ function isResourceExhausted(errorText: string): boolean {
107
+ const lower = errorText.toLowerCase();
108
+ return lower.includes("resource_exhausted") || lower.includes("resource exhausted");
109
+ }
110
+
100
111
  function formatError(err: unknown): string {
101
112
  if (!(err instanceof Error)) return String(err);
102
113
  const cause = err.cause;
@@ -180,8 +191,8 @@ async function streamResponseBody(
180
191
  const resetIdleTimer = (): void => {
181
192
  if (idleTimer) clearTimeout(idleTimer);
182
193
  idleTimer = setTimeout(() => {
183
- nodeStream.destroy(new Error(`stream idle for ${Math.round(STREAM_IDLE_TIMEOUT_MS / 1000)}s`));
184
194
  finish(`idle timeout after ${Math.round(STREAM_IDLE_TIMEOUT_MS / 1000)}s`);
195
+ if (!nodeStream.destroyed) nodeStream.destroy();
185
196
  }, STREAM_IDLE_TIMEOUT_MS);
186
197
  };
187
198
 
@@ -346,6 +357,9 @@ async function handleProxyRequest(
346
357
  const proxyLog = (msg: string, level: "info" | "warn" | "error" = "info"): void => {
347
358
  log(msg, rotator, level);
348
359
  };
360
+ if (bodyBuffer.length > LARGE_CONTEXT_WARN_BYTES) {
361
+ proxyLog(`[${body.model}] Large request body ${bodyBuffer.length} bytes; high context pressure increases rate-limit/flag risk`, "warn");
362
+ }
349
363
 
350
364
  const sendNoAccountsAvailable = (reason: string): void => {
351
365
  proxyLog(`[${body.model}] No healthy account available: ${reason}`, "warn");
@@ -392,22 +406,41 @@ async function handleProxyRequest(
392
406
  };
393
407
 
394
408
  try {
409
+ const jitterMs = rotator.getSafetyJitterMs(account);
410
+ if (jitterMs > 0) {
411
+ proxyLog(`[${requestId}] Safety slow-mode jitter ${jitterMs}ms for account/project daily budget pressure`, "warn");
412
+ await sleep(jitterMs);
413
+ }
414
+ rotator.recordUpstreamAttempt(account);
395
415
  const response = await forwardRequest(account, { ...body }, flattenHeaders(req.headers));
396
416
 
397
417
  if (response.status === 429) {
398
418
  const errorText = await response.text().catch(() => "");
399
- const cooldownMs = capCooldown(extractRetryDelay(errorText, response.headers));
400
- proxyLog(`[${label}] 429 rate limited, cooldown ${Math.ceil(cooldownMs / 1000)}s`, "warn");
419
+ const providerResourceExhausted = isResourceExhausted(errorText);
420
+ const cooldownMs = providerResourceExhausted
421
+ ? RESOURCE_EXHAUSTED_COOLDOWN_MS
422
+ : capCooldown(extractRetryDelay(errorText, response.headers));
423
+ proxyLog(
424
+ `[${label}] 429 rate limited${providerResourceExhausted ? " (RESOURCE_EXHAUSTED)" : ""}, cooldown ${Math.ceil(cooldownMs / 1000)}s. Error text: ${errorText.slice(0, 300)}`,
425
+ "warn",
426
+ );
401
427
  recordOutcome(429);
402
- logRequestEnd(429, `cooldownMs=${cooldownMs}`);
428
+ logRequestEnd(429, `cooldownMs=${cooldownMs}${providerResourceExhausted ? " resourceExhausted=true" : ""}`);
403
429
  rotator.markExhausted(account, body.model, cooldownMs);
404
- res.writeHead(503, {
430
+ rotator.recordProvider429(account, body.model, cooldownMs);
431
+
432
+ // Safety first: do NOT immediately retry another account on 429.
433
+ // Provider-side 429s can represent daily/request buckets or shared project pressure;
434
+ // cascading retries burn the full pool and increase ban/flag risk.
435
+ res.writeHead(429, {
405
436
  "Content-Type": "application/json",
406
437
  "Retry-After": String(Math.ceil(cooldownMs / 1000)),
407
438
  });
408
439
  res.end(JSON.stringify({
409
- error: "Rate limited",
410
- reason: `${label} was rate limited; not retrying another account for this request`,
440
+ error: providerResourceExhausted ? "Resource exhausted" : "Rate limited",
441
+ reason: providerResourceExhausted
442
+ ? `${label} hit provider RESOURCE_EXHAUSTED; not retrying another account to avoid pool-wide hammering`
443
+ : `${label} was rate limited; not retrying another account for account-safety`,
411
444
  model: body.model,
412
445
  account: label,
413
446
  retryAfterMs: cooldownMs,
package/src/rotator.ts CHANGED
@@ -38,6 +38,14 @@ const rotatorLogger = logger.child("rotator");
38
38
  const STATE_FILE = getStatePath();
39
39
  const TOKENS_FILE = STATE_FILE.replace("state.json", "token-usage.json");
40
40
 
41
+ function currentUtcDay(now = Date.now()): string {
42
+ return new Date(now).toISOString().slice(0, 10);
43
+ }
44
+
45
+ function projectModelKey(projectId: string, modelKey: string): string {
46
+ return `${projectId}::${modelKey}`;
47
+ }
48
+
41
49
  export class AccountRotator {
42
50
  private accounts: AccountRuntime[] = [];
43
51
  // Per-model active account tracking
@@ -56,6 +64,10 @@ export class AccountRotator {
56
64
  private static readonly MAX_LATENCY_RECORDS = 200;
57
65
  private requestLog: StatusResponse["requestLog"] = [];
58
66
  private static readonly MAX_REQUEST_LOG = 200;
67
+ private safetyDay = currentUtcDay();
68
+ private projectRequests: Record<string, number> = {};
69
+ private projectModelBreakers: Record<string, number> = {};
70
+ private provider429Events: Array<{ ts: number; projectId: string; modelKey: string; account: string }> = [];
59
71
 
60
72
  constructor(private config: Config) {
61
73
  this.initAccounts();
@@ -83,6 +95,8 @@ export class AccountRotator {
83
95
  inFlightByModel: {},
84
96
  allowFreshWindowStartsOverride: false,
85
97
  quotaWindows: {},
98
+ dailyRequestCount: 0,
99
+ dailyRequestDay: currentUtcDay(),
86
100
  }));
87
101
  }
88
102
 
@@ -109,11 +123,18 @@ export class AccountRotator {
109
123
  this.protectivePauseUntil = state.protectivePauseUntil ?? 0;
110
124
  this.protectivePauseReason = state.protectivePauseReason ?? null;
111
125
  this.allowFreshWindowStarts = state.allowFreshWindowStarts ?? true;
126
+ this.safetyDay = state.safety?.day ?? currentUtcDay();
127
+ this.projectRequests = state.safety?.projectRequests ?? {};
128
+ this.projectModelBreakers = state.safety?.projectModelBreakers ?? {};
129
+ this.provider429Events = state.safety?.provider429Events ?? [];
130
+ this.rollDailySafetyIfNeeded(Date.now());
112
131
 
113
132
  for (const account of this.accounts) {
114
133
  const saved = state.accounts[account.config.email];
115
134
  if (saved) {
116
135
  account.totalRequests = saved.totalRequests;
136
+ account.dailyRequestCount = saved.dailyRequestCount ?? 0;
137
+ account.dailyRequestDay = saved.dailyRequestDay ?? currentUtcDay();
117
138
  account.cooldownsByModel = saved.cooldownsByModel ?? {};
118
139
  if (saved.cooldownUntil !== undefined && Object.keys(account.cooldownsByModel).length === 0) {
119
140
  // legacy migration: apply global cooldown to default
@@ -186,11 +207,19 @@ export class AccountRotator {
186
207
  protectivePauseUntil: this.protectivePauseUntil,
187
208
  protectivePauseReason: this.protectivePauseReason,
188
209
  allowFreshWindowStarts: this.allowFreshWindowStarts,
210
+ safety: {
211
+ day: this.safetyDay,
212
+ projectRequests: { ...this.projectRequests },
213
+ projectModelBreakers: { ...this.projectModelBreakers },
214
+ provider429Events: [...this.provider429Events],
215
+ },
189
216
  accounts: {},
190
217
  };
191
218
  for (const account of this.accounts) {
192
219
  state.accounts[account.config.email] = {
193
220
  totalRequests: account.totalRequests,
221
+ dailyRequestCount: account.dailyRequestCount,
222
+ dailyRequestDay: account.dailyRequestDay,
194
223
  cooldownsByModel: { ...account.cooldownsByModel },
195
224
  quotaExhaustedAt: account.quotaExhaustedAt,
196
225
  disabled: account.disabled,
@@ -545,6 +574,54 @@ export class AccountRotator {
545
574
  return account;
546
575
  }
547
576
 
577
+ private rollDailySafetyIfNeeded(now: number): void {
578
+ const day = currentUtcDay(now);
579
+ if (this.safetyDay === day) return;
580
+ this.safetyDay = day;
581
+ this.projectRequests = {};
582
+ for (const account of this.accounts) {
583
+ account.dailyRequestDay = day;
584
+ account.dailyRequestCount = 0;
585
+ }
586
+ }
587
+
588
+ private getAccountDailyCount(account: AccountRuntime, now: number): number {
589
+ const day = currentUtcDay(now);
590
+ if (account.dailyRequestDay !== day) {
591
+ account.dailyRequestDay = day;
592
+ account.dailyRequestCount = 0;
593
+ }
594
+ return account.dailyRequestCount;
595
+ }
596
+
597
+ private getProjectDailyCount(projectId: string, now: number): number {
598
+ this.rollDailySafetyIfNeeded(now);
599
+ return this.projectRequests[projectId] ?? 0;
600
+ }
601
+
602
+ private getProjectInFlight(modelKey: string, projectId: string): number {
603
+ return this.accounts
604
+ .filter((account) => account.config.projectId === projectId)
605
+ .reduce((sum, account) => sum + (account.inFlightByModel[modelKey] ?? 0), 0);
606
+ }
607
+
608
+ private isProjectModelBreakerActive(projectId: string, modelKey: string, now: number): boolean {
609
+ const until = this.projectModelBreakers[projectModelKey(projectId, modelKey)] ?? 0;
610
+ if (until <= now) {
611
+ if (until > 0) delete this.projectModelBreakers[projectModelKey(projectId, modelKey)];
612
+ return false;
613
+ }
614
+ return true;
615
+ }
616
+
617
+ private getUnavailableReasonForModel(account: AccountRuntime, modelKey: string, now: number): string | null {
618
+ if (this.isProjectModelBreakerActive(account.config.projectId, modelKey, now)) return "project circuit breaker active";
619
+ if (this.getProjectInFlight(modelKey, account.config.projectId) >= (this.config.maxConcurrentRequestsPerProjectModel ?? 1)) return "project concurrency limit reached";
620
+ if (this.getAccountDailyCount(account, now) >= (this.config.dailyAccountStopRequests ?? 350)) return "daily account budget exhausted";
621
+ if (this.getProjectDailyCount(account.config.projectId, now) >= (this.config.dailyProjectStopRequests ?? 1200)) return "daily project budget exhausted";
622
+ return null;
623
+ }
624
+
548
625
  // =========================================================================
549
626
  // Account Selection (per-model)
550
627
  // =========================================================================
@@ -1001,13 +1078,58 @@ export class AccountRotator {
1001
1078
  account.cooldownsByModel[modelKey] = now + cooldownMs;
1002
1079
  account.quotaExhaustedAt = now;
1003
1080
 
1081
+ this.log(
1082
+ `${account.config.label || account.config.email} [${modelKey}]: EXHAUSTED, cooldown ${Math.ceil(cooldownMs / 1000)}s`,
1083
+ "warn",
1084
+ );
1085
+ this.saveState();
1086
+ }
1087
+
1088
+ recordProvider429(account: AccountRuntime, model: string | undefined, cooldownMs: number): void {
1089
+ const now = Date.now();
1090
+ const modelKey = model ? (resolveQuotaModelKey(model) ?? "__default__") : "__default__";
1091
+ const windowMs = this.config.projectCircuitBreakerWindowMs ?? 10 * 60 * 1000;
1092
+ const threshold = this.config.projectCircuitBreaker429Threshold ?? 3;
1093
+ const breakerCooldownMs = this.config.projectCircuitBreakerCooldownMs ?? 60 * 60 * 1000;
1094
+ const projectId = account.config.projectId;
1095
+ this.provider429Events = this.provider429Events
1096
+ .filter((event) => now - event.ts <= windowMs)
1097
+ .concat({ ts: now, projectId, modelKey, account: account.config.email });
1098
+ const uniqueAccounts = new Set(
1099
+ this.provider429Events
1100
+ .filter((event) => event.projectId === projectId && event.modelKey === modelKey)
1101
+ .map((event) => event.account),
1102
+ );
1103
+ if (uniqueAccounts.size >= threshold) {
1104
+ const until = now + Math.max(cooldownMs, breakerCooldownMs);
1105
+ this.projectModelBreakers[projectModelKey(projectId, modelKey)] = until;
1004
1106
  this.log(
1005
- `${account.config.label || account.config.email} [${modelKey}]: EXHAUSTED, cooldown ${Math.ceil(cooldownMs / 1000)}s`,
1107
+ `[${modelKey}] Project circuit breaker active for projectId=${projectId} after ${uniqueAccounts.size} accounts hit 429; cooldown ${Math.ceil((until - now) / 1000)}s`,
1006
1108
  "warn",
1007
1109
  );
1110
+ }
1111
+ this.saveState();
1112
+ }
1113
+
1114
+ recordUpstreamAttempt(account: AccountRuntime): void {
1115
+ const now = Date.now();
1116
+ this.rollDailySafetyIfNeeded(now);
1117
+ this.getAccountDailyCount(account, now);
1118
+ account.dailyRequestCount++;
1119
+ this.projectRequests[account.config.projectId] = (this.projectRequests[account.config.projectId] ?? 0) + 1;
1008
1120
  this.saveState();
1009
1121
  }
1010
1122
 
1123
+ getSafetyJitterMs(account: AccountRuntime): number {
1124
+ const now = Date.now();
1125
+ const accountSlow = this.getAccountDailyCount(account, now) >= (this.config.dailyAccountSlowRequests ?? 250);
1126
+ const projectSlow = this.getProjectDailyCount(account.config.projectId, now) >= (this.config.dailyProjectSlowRequests ?? 900);
1127
+ if (!accountSlow && !projectSlow) return 0;
1128
+ const min = this.config.slowModeJitterMinMs ?? 8_000;
1129
+ const max = Math.max(min, this.config.slowModeJitterMaxMs ?? 25_000);
1130
+ return Math.floor(min + Math.random() * (max - min + 1));
1131
+ }
1132
+
1011
1133
  markError(account: AccountRuntime, error: string): void {
1012
1134
  account.lastError = error;
1013
1135
  account.consecutiveErrors++;
@@ -1139,6 +1261,7 @@ export class AccountRotator {
1139
1261
  const modelCooldown = account.cooldownsByModel[modelKey] ?? 0;
1140
1262
  if (modelCooldown > now) return false;
1141
1263
  if ((account.inFlightByModel[modelKey] ?? 0) >= (this.config.maxConcurrentRequestsPerAccount ?? 1)) return false;
1264
+ if (this.getUnavailableReasonForModel(account, modelKey, now)) return false;
1142
1265
  return true;
1143
1266
  }
1144
1267
 
@@ -1348,6 +1471,8 @@ export class AccountRotator {
1348
1471
  inFlightByModel: {},
1349
1472
  allowFreshWindowStartsOverride: false,
1350
1473
  quotaWindows: {},
1474
+ dailyRequestCount: 0,
1475
+ dailyRequestDay: currentUtcDay(),
1351
1476
  };
1352
1477
  this.accounts.push(runtime);
1353
1478
  this.config.accounts.push(runtime.config);
package/src/types.ts CHANGED
@@ -6,6 +6,8 @@ export interface AccountConfig {
6
6
  email: string;
7
7
  refreshToken: string;
8
8
  projectId: string;
9
+ // How the projectId was obtained.
10
+ projectSource?: "google" | "manual";
9
11
  label?: string;
10
12
  // Optional - pro/free is detected dynamically from quota API reset times
11
13
  type?: AccountType;
@@ -25,6 +27,20 @@ export interface Config {
25
27
  proSlots?: number;
26
28
  // Hard cap on parallel requests per account. Conservative default is 1.
27
29
  maxConcurrentRequestsPerAccount?: number;
30
+ // Hard cap on parallel requests per projectId/model. Conservative default is 1.
31
+ maxConcurrentRequestsPerProjectModel?: number;
32
+ // Pause projectId/model when several accounts hit provider 429 in a short window. Defaults: 3 hits / 10min / 60min pause.
33
+ projectCircuitBreaker429Threshold?: number;
34
+ projectCircuitBreakerWindowMs?: number;
35
+ projectCircuitBreakerCooldownMs?: number;
36
+ // Daily safety budgets. Defaults: account slow 250, account stop 350, project slow 900, project stop 1200.
37
+ dailyAccountSlowRequests?: number;
38
+ dailyAccountStopRequests?: number;
39
+ dailyProjectSlowRequests?: number;
40
+ dailyProjectStopRequests?: number;
41
+ // Add small delay before upstream call when an account/project is in slow mode. Default: 8-25s.
42
+ slowModeJitterMinMs?: number;
43
+ slowModeJitterMaxMs?: number;
28
44
  // Pause all routing after a serious provider flag. Default: 6h.
29
45
  protectivePauseMs?: number;
30
46
  // Use request-count rotation only before quota data is available. Default: true.
@@ -139,6 +155,8 @@ export interface AccountRuntime {
139
155
  inFlightByModel: Record<string, number>;
140
156
  allowFreshWindowStartsOverride: boolean;
141
157
  quotaWindows: QuotaWindowHistory;
158
+ dailyRequestCount: number;
159
+ dailyRequestDay: string;
142
160
  }
143
161
 
144
162
  // Per-model rotation state tracked by the rotator
@@ -164,6 +182,13 @@ export interface DualWindowTracker {
164
182
  // Per-account quota window tracking: keyed by model key
165
183
  export type QuotaWindowHistory = Record<string, DualWindowTracker>;
166
184
 
185
+ export interface PersistedSafetyState {
186
+ day: string;
187
+ projectRequests: Record<string, number>;
188
+ projectModelBreakers: Record<string, number>;
189
+ provider429Events: Array<{ ts: number; projectId: string; modelKey: string; account: string }>;
190
+ }
191
+
167
192
  export interface PersistedState {
168
193
  // Per-model active account index
169
194
  modelAccounts: Record<string, number>;
@@ -174,10 +199,13 @@ export interface PersistedState {
174
199
  protectivePauseUntil?: number;
175
200
  protectivePauseReason?: string | null;
176
201
  allowFreshWindowStarts?: boolean;
202
+ safety?: PersistedSafetyState;
177
203
  accounts: Record<
178
204
  string,
179
205
  {
180
206
  totalRequests: number;
207
+ dailyRequestCount?: number;
208
+ dailyRequestDay?: string;
181
209
  cooldownUntil?: number; // legacy fallback
182
210
  cooldownsByModel?: Record<string, number>;
183
211
  quotaExhaustedAt: number;
package/src/validators.ts CHANGED
@@ -63,6 +63,16 @@ export function validateConfig(value: unknown): ValidationResult<Config> {
63
63
  if (value.quotaPollIntervalMs !== undefined && !isPositiveNumber(value.quotaPollIntervalMs)) errors.push("config.quotaPollIntervalMs must be a positive number");
64
64
  if (value.proSlots !== undefined && !isPositiveNumber(value.proSlots)) errors.push("config.proSlots must be a positive number");
65
65
  if (value.maxConcurrentRequestsPerAccount !== undefined && !isPositiveNumber(value.maxConcurrentRequestsPerAccount)) errors.push("config.maxConcurrentRequestsPerAccount must be a positive number");
66
+ if (value.maxConcurrentRequestsPerProjectModel !== undefined && !isPositiveNumber(value.maxConcurrentRequestsPerProjectModel)) errors.push("config.maxConcurrentRequestsPerProjectModel must be a positive number");
67
+ if (value.projectCircuitBreaker429Threshold !== undefined && !isPositiveNumber(value.projectCircuitBreaker429Threshold)) errors.push("config.projectCircuitBreaker429Threshold must be a positive number");
68
+ if (value.projectCircuitBreakerWindowMs !== undefined && !isPositiveNumber(value.projectCircuitBreakerWindowMs)) errors.push("config.projectCircuitBreakerWindowMs must be a positive number");
69
+ if (value.projectCircuitBreakerCooldownMs !== undefined && !isPositiveNumber(value.projectCircuitBreakerCooldownMs)) errors.push("config.projectCircuitBreakerCooldownMs must be a positive number");
70
+ if (value.dailyAccountSlowRequests !== undefined && !isPositiveNumber(value.dailyAccountSlowRequests)) errors.push("config.dailyAccountSlowRequests must be a positive number");
71
+ if (value.dailyAccountStopRequests !== undefined && !isPositiveNumber(value.dailyAccountStopRequests)) errors.push("config.dailyAccountStopRequests must be a positive number");
72
+ if (value.dailyProjectSlowRequests !== undefined && !isPositiveNumber(value.dailyProjectSlowRequests)) errors.push("config.dailyProjectSlowRequests must be a positive number");
73
+ if (value.dailyProjectStopRequests !== undefined && !isPositiveNumber(value.dailyProjectStopRequests)) errors.push("config.dailyProjectStopRequests must be a positive number");
74
+ if (value.slowModeJitterMinMs !== undefined && !isNonNegativeNumber(value.slowModeJitterMinMs)) errors.push("config.slowModeJitterMinMs must be a non-negative number");
75
+ if (value.slowModeJitterMaxMs !== undefined && !isNonNegativeNumber(value.slowModeJitterMaxMs)) errors.push("config.slowModeJitterMaxMs must be a non-negative number");
66
76
  if (value.protectivePauseMs !== undefined && !isNonNegativeNumber(value.protectivePauseMs)) errors.push("config.protectivePauseMs must be a non-negative number");
67
77
  if (value.useRequestCountRotationWhenQuotaUnknownOnly !== undefined && typeof value.useRequestCountRotationWhenQuotaUnknownOnly !== "boolean") {
68
78
  errors.push("config.useRequestCountRotationWhenQuotaUnknownOnly must be a boolean");