pi-antigravity-rotator 1.9.0 → 1.9.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/README.md +35 -3
- package/package.json +1 -1
- package/src/account-store.ts +10 -0
- package/src/index.ts +12 -1
- package/src/login.ts +4 -2
- package/src/oauth.ts +10 -5
- package/src/onboarding.ts +4 -2
- package/src/proxy.ts +40 -7
- package/src/rotator.ts +126 -1
- package/src/types.ts +28 -0
- package/src/validators.ts +10 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,24 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.9.2] - 2026-04-29
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- **Project Discovery Without Shared Fallback**: Login now fails if Google does not return a companion project ID. No more shared `rising-fact-p41fc` fallback.
|
|
7
|
+
- **Activation Hint**: Login/discovery errors now tell you to open the account in Antigravity IDE and send one message first.
|
|
8
|
+
- **Activation Docs**: README now documents the first-use activation rule for new accounts.
|
|
9
|
+
|
|
10
|
+
## [1.9.1] - 2026-04-29
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
- **429 Account-Safety Backoff**: All provider-side `429` responses now stop the current request instead of immediately retrying another account. This prevents cascade-burning the full pool when Google rate-limits a shared project/request bucket. `RESOURCE_EXHAUSTED` gets a 30-minute cooldown; other 429s use parsed `Retry-After`/retry-delay.
|
|
14
|
+
- **Stream Idle Crash Safety**: Stream idle timeout now closes cleanly without emitting an unhandled stream error that could crash-loop systemd.
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
- **Project Circuit Breaker**: If multiple accounts sharing a `projectId` hit provider `429` for the same quota model inside a rolling window, routing pauses that `projectId`/model instead of burning sibling accounts.
|
|
18
|
+
- **Daily Safety Budgets**: Per-account and per-`projectId` daily upstream attempt counters now trigger slow-mode jitter and hard stops until the next UTC day.
|
|
19
|
+
- **Project Concurrency Guard**: Added `maxConcurrentRequestsPerProjectModel` to prevent simultaneous calls through multiple accounts backed by the same provider project bucket.
|
|
20
|
+
- **Large Context Warning**: Requests above 1 MiB now log a warning because huge contexts increase rate-limit and flag pressure.
|
|
21
|
+
|
|
3
22
|
## [1.9.0] - 2026-04-29
|
|
4
23
|
|
|
5
24
|
### Added
|
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@ Multi-account rotation proxy for Google Antigravity. Distributes API usage acros
|
|
|
9
9
|
- **Per-model timer tracking** -- Timer classification (`fresh`/`7d`/`5h`) is evaluated per model using each model's actual `resetTime` from the quota API, not a per-account estimate
|
|
10
10
|
- **Smart rotation** -- Rotates only the specific model whose quota dropped, leaving other models on their current accounts
|
|
11
11
|
- **Infringement detection** -- On 403 with infringement/abuse/suspension keywords, the account is immediately flagged and excluded from routing
|
|
12
|
-
- **
|
|
12
|
+
- **Safer 429 handling** -- On provider `429`, stops the current request and avoids cascade-burning sibling accounts
|
|
13
13
|
- **Concurrency guardrails** -- Limits each account to one in-flight request by default to avoid bursty pressure
|
|
14
14
|
- **Operator fresh-window controls** -- You can block new `fresh` window starts globally, then selectively allow specific accounts to override that policy
|
|
15
15
|
- **Protective pause** -- Pauses all routing for several hours after serious ToS/abuse-style flags so the rest of the pool is not burned
|
|
@@ -57,6 +57,7 @@ Run `npm run login` once per Google account:
|
|
|
57
57
|
2. Complete the sign-in and grant permissions
|
|
58
58
|
3. The browser redirects to a `localhost` URL that won't load -- this is expected
|
|
59
59
|
4. Copy the **full URL** from the browser's address bar and paste it into the terminal
|
|
60
|
+
5. If project discovery fails, open that same account in Antigravity IDE, send one message, then rerun login
|
|
60
61
|
|
|
61
62
|
The tool automatically:
|
|
62
63
|
|
|
@@ -66,6 +67,15 @@ The tool automatically:
|
|
|
66
67
|
|
|
67
68
|
Re-running with the same email updates the existing entry.
|
|
68
69
|
|
|
70
|
+
### Activation rule
|
|
71
|
+
|
|
72
|
+
Some accounts do not expose a discoverable `projectId` until they are used once in the Antigravity IDE.
|
|
73
|
+
If login fails at project discovery:
|
|
74
|
+
|
|
75
|
+
1. Open that exact Google account in Antigravity IDE.
|
|
76
|
+
2. Send one message.
|
|
77
|
+
3. Rerun `npm run login`.
|
|
78
|
+
|
|
69
79
|
## Dashboard
|
|
70
80
|
|
|
71
81
|
After starting the proxy, open `http://localhost:51200/dashboard` or `http://<your-server-ip>:51200/dashboard` from any machine on the same network (the proxy binds to `0.0.0.0`).
|
|
@@ -213,7 +223,8 @@ export PI_AI_ANTIGRAVITY_VERSION=1.107.0
|
|
|
213
223
|
pi-antigravity-rotator start --config-dir /path/to/config
|
|
214
224
|
```
|
|
215
225
|
|
|
216
|
-
`accounts.json` is created automatically by the login command
|
|
226
|
+
`accounts.json` is created automatically by the login command.
|
|
227
|
+
Login now fails if Google does not return a project ID. No shared fallback.
|
|
217
228
|
|
|
218
229
|
```json
|
|
219
230
|
{
|
|
@@ -222,6 +233,16 @@ pi-antigravity-rotator start --config-dir /path/to/config
|
|
|
222
233
|
"rotateOnQuotaDrop": 20,
|
|
223
234
|
"quotaPollIntervalMs": 300000,
|
|
224
235
|
"maxConcurrentRequestsPerAccount": 1,
|
|
236
|
+
"maxConcurrentRequestsPerProjectModel": 1,
|
|
237
|
+
"projectCircuitBreaker429Threshold": 3,
|
|
238
|
+
"projectCircuitBreakerWindowMs": 600000,
|
|
239
|
+
"projectCircuitBreakerCooldownMs": 3600000,
|
|
240
|
+
"dailyAccountSlowRequests": 250,
|
|
241
|
+
"dailyAccountStopRequests": 350,
|
|
242
|
+
"dailyProjectSlowRequests": 900,
|
|
243
|
+
"dailyProjectStopRequests": 1200,
|
|
244
|
+
"slowModeJitterMinMs": 8000,
|
|
245
|
+
"slowModeJitterMaxMs": 25000,
|
|
225
246
|
"protectivePauseMs": 21600000,
|
|
226
247
|
"useRequestCountRotationWhenQuotaUnknownOnly": true,
|
|
227
248
|
"accounts": [
|
|
@@ -244,6 +265,16 @@ pi-antigravity-rotator start --config-dir /path/to/config
|
|
|
244
265
|
| `rotateOnQuotaDrop` | `20` | Rotate when a model's quota drops this many %. Set to `0` to disable |
|
|
245
266
|
| `quotaPollIntervalMs` | `300000` | Quota poll interval in ms (5 minutes) |
|
|
246
267
|
| `maxConcurrentRequestsPerAccount` | `1` | Max simultaneous requests allowed per account |
|
|
268
|
+
| `maxConcurrentRequestsPerProjectModel` | `1` | Max simultaneous requests allowed across accounts sharing the same `projectId` for the same quota model |
|
|
269
|
+
| `projectCircuitBreaker429Threshold` | `3` | Unique accounts from the same `projectId` that must hit provider `429` before pausing that project/model |
|
|
270
|
+
| `projectCircuitBreakerWindowMs` | `600000` | Rolling window for the project/model `429` circuit breaker |
|
|
271
|
+
| `projectCircuitBreakerCooldownMs` | `3600000` | Minimum project/model pause after the circuit breaker trips |
|
|
272
|
+
| `dailyAccountSlowRequests` | `250` | Daily upstream attempts per account before slow-mode jitter starts |
|
|
273
|
+
| `dailyAccountStopRequests` | `350` | Daily upstream attempts per account before routing stops for that account until the next UTC day |
|
|
274
|
+
| `dailyProjectSlowRequests` | `900` | Daily upstream attempts per `projectId` before slow-mode jitter starts |
|
|
275
|
+
| `dailyProjectStopRequests` | `1200` | Daily upstream attempts per `projectId` before routing stops for that project until the next UTC day |
|
|
276
|
+
| `slowModeJitterMinMs` | `8000` | Minimum slow-mode delay before upstream request |
|
|
277
|
+
| `slowModeJitterMaxMs` | `25000` | Maximum slow-mode delay before upstream request |
|
|
247
278
|
| `protectivePauseMs` | `21600000` | Global routing pause after a serious provider enforcement signal |
|
|
248
279
|
| `useRequestCountRotationWhenQuotaUnknownOnly` | `true` | Use request-count rotation only until quota telemetry exists for the request's model. Set to `false` to keep rotating by request count even with known quotas |
|
|
249
280
|
|
|
@@ -253,7 +284,8 @@ pi-antigravity-rotator start --config-dir /path/to/config
|
|
|
253
284
|
|-------|-------------|
|
|
254
285
|
| `email` | Google account email (auto-filled by login) |
|
|
255
286
|
| `refreshToken` | OAuth refresh token (auto-filled by login) |
|
|
256
|
-
| `projectId` | Cloud project ID
|
|
287
|
+
| `projectId` | Cloud project ID discovered from Google during login |
|
|
288
|
+
| `projectSource` | Optional metadata: `google` when discovered from Google, `manual` if edited by hand |
|
|
257
289
|
| `label` | Display name on the dashboard (auto-filled, defaults to email username) |
|
|
258
290
|
|
|
259
291
|
## API Endpoints
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-antigravity-rotator",
|
|
3
|
-
"version": "1.9.
|
|
3
|
+
"version": "1.9.2",
|
|
4
4
|
"description": "Multi-account rotation proxy for Google Antigravity with per-model routing, real-time quota tracking, and infringement detection",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
package/src/account-store.ts
CHANGED
|
@@ -23,6 +23,16 @@ export function loadOrCreateAccountsConfig(): Config {
|
|
|
23
23
|
rotateOnQuotaDrop: 20,
|
|
24
24
|
quotaPollIntervalMs: 300000,
|
|
25
25
|
maxConcurrentRequestsPerAccount: 1,
|
|
26
|
+
maxConcurrentRequestsPerProjectModel: 1,
|
|
27
|
+
projectCircuitBreaker429Threshold: 3,
|
|
28
|
+
projectCircuitBreakerWindowMs: 10 * 60 * 1000,
|
|
29
|
+
projectCircuitBreakerCooldownMs: 60 * 60 * 1000,
|
|
30
|
+
dailyAccountSlowRequests: 250,
|
|
31
|
+
dailyAccountStopRequests: 350,
|
|
32
|
+
dailyProjectSlowRequests: 900,
|
|
33
|
+
dailyProjectStopRequests: 1200,
|
|
34
|
+
slowModeJitterMinMs: 8_000,
|
|
35
|
+
slowModeJitterMaxMs: 25_000,
|
|
26
36
|
protectivePauseMs: 21600000,
|
|
27
37
|
useRequestCountRotationWhenQuotaUnknownOnly: true,
|
|
28
38
|
accounts: [],
|
package/src/index.ts
CHANGED
|
@@ -38,6 +38,16 @@ function loadConfig(): Config {
|
|
|
38
38
|
config.rotateOnQuotaDrop = config.rotateOnQuotaDrop ?? 20;
|
|
39
39
|
config.quotaPollIntervalMs = config.quotaPollIntervalMs || 300_000;
|
|
40
40
|
config.maxConcurrentRequestsPerAccount = config.maxConcurrentRequestsPerAccount ?? 1;
|
|
41
|
+
config.maxConcurrentRequestsPerProjectModel = config.maxConcurrentRequestsPerProjectModel ?? 1;
|
|
42
|
+
config.projectCircuitBreaker429Threshold = config.projectCircuitBreaker429Threshold ?? 3;
|
|
43
|
+
config.projectCircuitBreakerWindowMs = config.projectCircuitBreakerWindowMs ?? 10 * 60 * 1000;
|
|
44
|
+
config.projectCircuitBreakerCooldownMs = config.projectCircuitBreakerCooldownMs ?? 60 * 60 * 1000;
|
|
45
|
+
config.dailyAccountSlowRequests = config.dailyAccountSlowRequests ?? 250;
|
|
46
|
+
config.dailyAccountStopRequests = config.dailyAccountStopRequests ?? 350;
|
|
47
|
+
config.dailyProjectSlowRequests = config.dailyProjectSlowRequests ?? 900;
|
|
48
|
+
config.dailyProjectStopRequests = config.dailyProjectStopRequests ?? 1200;
|
|
49
|
+
config.slowModeJitterMinMs = config.slowModeJitterMinMs ?? 8_000;
|
|
50
|
+
config.slowModeJitterMaxMs = config.slowModeJitterMaxMs ?? 25_000;
|
|
41
51
|
config.protectivePauseMs = config.protectivePauseMs ?? 6 * 60 * 60 * 1000;
|
|
42
52
|
config.useRequestCountRotationWhenQuotaUnknownOnly =
|
|
43
53
|
config.useRequestCountRotationWhenQuotaUnknownOnly ?? true;
|
|
@@ -97,7 +107,8 @@ export function main(): void {
|
|
|
97
107
|
console.log(`Loaded ${config.accounts.length} accounts`);
|
|
98
108
|
console.log(`Rotation: ${config.requestsPerRotation} requests / ${config.rotateOnQuotaDrop}% quota drop`);
|
|
99
109
|
console.log(`Quota poll: every ${Math.round((config.quotaPollIntervalMs || 300000) / 1000)}s`);
|
|
100
|
-
console.log(`Concurrency cap: ${config.maxConcurrentRequestsPerAccount} request/account`);
|
|
110
|
+
console.log(`Concurrency cap: ${config.maxConcurrentRequestsPerAccount} request/account, ${config.maxConcurrentRequestsPerProjectModel} request/project+model`);
|
|
111
|
+
console.log(`Safety breaker: ${config.projectCircuitBreaker429Threshold} provider 429s / ${Math.round((config.projectCircuitBreakerWindowMs || 0) / 60000)}m pauses project+model for ${Math.round((config.projectCircuitBreakerCooldownMs || 0) / 60000)}m`);
|
|
101
112
|
console.log(`Protective pause: ${Math.round((config.protectivePauseMs || 0) / 3600000)}h after serious flag`);
|
|
102
113
|
console.log();
|
|
103
114
|
|
package/src/login.ts
CHANGED
|
@@ -79,19 +79,21 @@ export async function runLogin(): Promise<void> {
|
|
|
79
79
|
const email = await getUserEmail(tokenData.accessToken);
|
|
80
80
|
|
|
81
81
|
console.log("Discovering project...");
|
|
82
|
-
const
|
|
82
|
+
const project = await discoverProject(tokenData.accessToken);
|
|
83
83
|
|
|
84
84
|
const label = email ? email.split("@")[0] : "Account";
|
|
85
85
|
const entry: AccountConfig = {
|
|
86
86
|
email: email || "unknown@gmail.com",
|
|
87
87
|
refreshToken: tokenData.refreshToken,
|
|
88
|
-
projectId,
|
|
88
|
+
projectId: project.projectId,
|
|
89
|
+
projectSource: project.source,
|
|
89
90
|
label,
|
|
90
91
|
};
|
|
91
92
|
|
|
92
93
|
console.log();
|
|
93
94
|
const { isNew } = addAccountToConfig(entry);
|
|
94
95
|
console.log(` ${isNew ? "Added" : "Updated"} ${entry.email} in ${ACCOUNTS_FILE}`);
|
|
96
|
+
console.log(` projectId=${project.projectId} (source=${project.source})`);
|
|
95
97
|
|
|
96
98
|
ensurePiModelsConfig();
|
|
97
99
|
ensurePiAuthConfig();
|
package/src/oauth.ts
CHANGED
|
@@ -11,7 +11,6 @@ export const SCOPES = [
|
|
|
11
11
|
"https://www.googleapis.com/auth/experimentsandconfigs",
|
|
12
12
|
];
|
|
13
13
|
export const AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth";
|
|
14
|
-
export const DEFAULT_PROJECT_ID = "rising-fact-p41fc";
|
|
15
14
|
|
|
16
15
|
export interface OAuthClientConfig {
|
|
17
16
|
clientId: string;
|
|
@@ -107,7 +106,13 @@ export async function exchangeAuthorizationCode(code: string, verifier: string):
|
|
|
107
106
|
};
|
|
108
107
|
}
|
|
109
108
|
|
|
110
|
-
export
|
|
109
|
+
export interface ProjectDiscoveryResult {
|
|
110
|
+
projectId: string;
|
|
111
|
+
source: "google";
|
|
112
|
+
endpoint: string;
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
export async function discoverProject(accessToken: string): Promise<ProjectDiscoveryResult> {
|
|
111
116
|
const headers: Record<string, string> = {
|
|
112
117
|
Authorization: `Bearer ${accessToken}`,
|
|
113
118
|
"Content-Type": "application/json",
|
|
@@ -138,14 +143,14 @@ export async function discoverProject(accessToken: string): Promise<string> {
|
|
|
138
143
|
cloudaicompanionProject?: string | { id?: string };
|
|
139
144
|
};
|
|
140
145
|
if (typeof data.cloudaicompanionProject === "string" && data.cloudaicompanionProject) {
|
|
141
|
-
return data.cloudaicompanionProject;
|
|
146
|
+
return { projectId: data.cloudaicompanionProject, source: "google", endpoint };
|
|
142
147
|
}
|
|
143
148
|
if (
|
|
144
149
|
data.cloudaicompanionProject &&
|
|
145
150
|
typeof data.cloudaicompanionProject === "object" &&
|
|
146
151
|
data.cloudaicompanionProject.id
|
|
147
152
|
) {
|
|
148
|
-
return data.cloudaicompanionProject.id;
|
|
153
|
+
return { projectId: data.cloudaicompanionProject.id, source: "google", endpoint };
|
|
149
154
|
}
|
|
150
155
|
}
|
|
151
156
|
} catch {
|
|
@@ -153,7 +158,7 @@ export async function discoverProject(accessToken: string): Promise<string> {
|
|
|
153
158
|
}
|
|
154
159
|
}
|
|
155
160
|
|
|
156
|
-
|
|
161
|
+
throw new Error("Could not discover Cloud Code companion project ID from Google. If this account is new, open it in Antigravity IDE and send one message first, then retry login. Login failed instead of falling back to a shared projectId.");
|
|
157
162
|
}
|
|
158
163
|
|
|
159
164
|
export async function getUserEmail(accessToken: string): Promise<string | undefined> {
|
package/src/onboarding.ts
CHANGED
|
@@ -203,12 +203,13 @@ export async function handleHostedCallback(
|
|
|
203
203
|
try {
|
|
204
204
|
const tokenData = await exchangeAuthorizationCode(code, session.verifier);
|
|
205
205
|
const email = await getUserEmail(tokenData.accessToken);
|
|
206
|
-
const
|
|
206
|
+
const project = await discoverProject(tokenData.accessToken);
|
|
207
207
|
const label = email ? email.split("@")[0] : "Account";
|
|
208
208
|
const entry = {
|
|
209
209
|
email: email || "unknown@gmail.com",
|
|
210
210
|
refreshToken: tokenData.refreshToken,
|
|
211
|
-
projectId,
|
|
211
|
+
projectId: project.projectId,
|
|
212
|
+
projectSource: project.source,
|
|
212
213
|
label,
|
|
213
214
|
};
|
|
214
215
|
|
|
@@ -221,6 +222,7 @@ export async function handleHostedCallback(
|
|
|
221
222
|
"Account Connected",
|
|
222
223
|
`<h1>Account Connected</h1>
|
|
223
224
|
<p><strong>${entry.email}</strong> was ${isNew ? "added" : "updated"} successfully.</p>
|
|
225
|
+
<p>Project: <span class="mono">${project.projectId}</span> via ${project.source}.</p>
|
|
224
226
|
<p>The rotator can start using this account immediately.</p>
|
|
225
227
|
<div class="note">If you ever want to stop sharing access, revoke this app's access from the Google account security settings.</div>`,
|
|
226
228
|
),
|
package/src/proxy.ts
CHANGED
|
@@ -26,7 +26,13 @@ const proxyLogger = logger.child("proxy");
|
|
|
26
26
|
|
|
27
27
|
const MAX_ENDPOINT_RETRIES = 3;
|
|
28
28
|
const MAX_COOLDOWN_MS = 30 * 60 * 1000; // 30 minutes max cooldown
|
|
29
|
+
const RESOURCE_EXHAUSTED_COOLDOWN_MS = 30 * 60 * 1000; // Stop hammering provider-side daily/request buckets
|
|
29
30
|
const STREAM_IDLE_TIMEOUT_MS = 2 * 60 * 1000; // Release account if a stream goes silent.
|
|
31
|
+
const LARGE_CONTEXT_WARN_BYTES = 1 * 1024 * 1024;
|
|
32
|
+
|
|
33
|
+
function sleep(ms: number): Promise<void> {
|
|
34
|
+
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
35
|
+
}
|
|
30
36
|
|
|
31
37
|
interface RequestBody {
|
|
32
38
|
project: string;
|
|
@@ -97,6 +103,11 @@ function capCooldown(ms: number): number {
|
|
|
97
103
|
return Math.min(ms, MAX_COOLDOWN_MS);
|
|
98
104
|
}
|
|
99
105
|
|
|
106
|
+
function isResourceExhausted(errorText: string): boolean {
|
|
107
|
+
const lower = errorText.toLowerCase();
|
|
108
|
+
return lower.includes("resource_exhausted") || lower.includes("resource exhausted");
|
|
109
|
+
}
|
|
110
|
+
|
|
100
111
|
function formatError(err: unknown): string {
|
|
101
112
|
if (!(err instanceof Error)) return String(err);
|
|
102
113
|
const cause = err.cause;
|
|
@@ -180,8 +191,8 @@ async function streamResponseBody(
|
|
|
180
191
|
const resetIdleTimer = (): void => {
|
|
181
192
|
if (idleTimer) clearTimeout(idleTimer);
|
|
182
193
|
idleTimer = setTimeout(() => {
|
|
183
|
-
nodeStream.destroy(new Error(`stream idle for ${Math.round(STREAM_IDLE_TIMEOUT_MS / 1000)}s`));
|
|
184
194
|
finish(`idle timeout after ${Math.round(STREAM_IDLE_TIMEOUT_MS / 1000)}s`);
|
|
195
|
+
if (!nodeStream.destroyed) nodeStream.destroy();
|
|
185
196
|
}, STREAM_IDLE_TIMEOUT_MS);
|
|
186
197
|
};
|
|
187
198
|
|
|
@@ -346,6 +357,9 @@ async function handleProxyRequest(
|
|
|
346
357
|
const proxyLog = (msg: string, level: "info" | "warn" | "error" = "info"): void => {
|
|
347
358
|
log(msg, rotator, level);
|
|
348
359
|
};
|
|
360
|
+
if (bodyBuffer.length > LARGE_CONTEXT_WARN_BYTES) {
|
|
361
|
+
proxyLog(`[${body.model}] Large request body ${bodyBuffer.length} bytes; high context pressure increases rate-limit/flag risk`, "warn");
|
|
362
|
+
}
|
|
349
363
|
|
|
350
364
|
const sendNoAccountsAvailable = (reason: string): void => {
|
|
351
365
|
proxyLog(`[${body.model}] No healthy account available: ${reason}`, "warn");
|
|
@@ -392,22 +406,41 @@ async function handleProxyRequest(
|
|
|
392
406
|
};
|
|
393
407
|
|
|
394
408
|
try {
|
|
409
|
+
const jitterMs = rotator.getSafetyJitterMs(account);
|
|
410
|
+
if (jitterMs > 0) {
|
|
411
|
+
proxyLog(`[${requestId}] Safety slow-mode jitter ${jitterMs}ms for account/project daily budget pressure`, "warn");
|
|
412
|
+
await sleep(jitterMs);
|
|
413
|
+
}
|
|
414
|
+
rotator.recordUpstreamAttempt(account);
|
|
395
415
|
const response = await forwardRequest(account, { ...body }, flattenHeaders(req.headers));
|
|
396
416
|
|
|
397
417
|
if (response.status === 429) {
|
|
398
418
|
const errorText = await response.text().catch(() => "");
|
|
399
|
-
const
|
|
400
|
-
|
|
419
|
+
const providerResourceExhausted = isResourceExhausted(errorText);
|
|
420
|
+
const cooldownMs = providerResourceExhausted
|
|
421
|
+
? RESOURCE_EXHAUSTED_COOLDOWN_MS
|
|
422
|
+
: capCooldown(extractRetryDelay(errorText, response.headers));
|
|
423
|
+
proxyLog(
|
|
424
|
+
`[${label}] 429 rate limited${providerResourceExhausted ? " (RESOURCE_EXHAUSTED)" : ""}, cooldown ${Math.ceil(cooldownMs / 1000)}s. Error text: ${errorText.slice(0, 300)}`,
|
|
425
|
+
"warn",
|
|
426
|
+
);
|
|
401
427
|
recordOutcome(429);
|
|
402
|
-
logRequestEnd(429, `cooldownMs=${cooldownMs}`);
|
|
428
|
+
logRequestEnd(429, `cooldownMs=${cooldownMs}${providerResourceExhausted ? " resourceExhausted=true" : ""}`);
|
|
403
429
|
rotator.markExhausted(account, body.model, cooldownMs);
|
|
404
|
-
|
|
430
|
+
rotator.recordProvider429(account, body.model, cooldownMs);
|
|
431
|
+
|
|
432
|
+
// Safety first: do NOT immediately retry another account on 429.
|
|
433
|
+
// Provider-side 429s can represent daily/request buckets or shared project pressure;
|
|
434
|
+
// cascading retries burn the full pool and increase ban/flag risk.
|
|
435
|
+
res.writeHead(429, {
|
|
405
436
|
"Content-Type": "application/json",
|
|
406
437
|
"Retry-After": String(Math.ceil(cooldownMs / 1000)),
|
|
407
438
|
});
|
|
408
439
|
res.end(JSON.stringify({
|
|
409
|
-
error: "Rate limited",
|
|
410
|
-
reason:
|
|
440
|
+
error: providerResourceExhausted ? "Resource exhausted" : "Rate limited",
|
|
441
|
+
reason: providerResourceExhausted
|
|
442
|
+
? `${label} hit provider RESOURCE_EXHAUSTED; not retrying another account to avoid pool-wide hammering`
|
|
443
|
+
: `${label} was rate limited; not retrying another account for account-safety`,
|
|
411
444
|
model: body.model,
|
|
412
445
|
account: label,
|
|
413
446
|
retryAfterMs: cooldownMs,
|
package/src/rotator.ts
CHANGED
|
@@ -38,6 +38,14 @@ const rotatorLogger = logger.child("rotator");
|
|
|
38
38
|
const STATE_FILE = getStatePath();
|
|
39
39
|
const TOKENS_FILE = STATE_FILE.replace("state.json", "token-usage.json");
|
|
40
40
|
|
|
41
|
+
function currentUtcDay(now = Date.now()): string {
|
|
42
|
+
return new Date(now).toISOString().slice(0, 10);
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
function projectModelKey(projectId: string, modelKey: string): string {
|
|
46
|
+
return `${projectId}::${modelKey}`;
|
|
47
|
+
}
|
|
48
|
+
|
|
41
49
|
export class AccountRotator {
|
|
42
50
|
private accounts: AccountRuntime[] = [];
|
|
43
51
|
// Per-model active account tracking
|
|
@@ -56,6 +64,10 @@ export class AccountRotator {
|
|
|
56
64
|
private static readonly MAX_LATENCY_RECORDS = 200;
|
|
57
65
|
private requestLog: StatusResponse["requestLog"] = [];
|
|
58
66
|
private static readonly MAX_REQUEST_LOG = 200;
|
|
67
|
+
private safetyDay = currentUtcDay();
|
|
68
|
+
private projectRequests: Record<string, number> = {};
|
|
69
|
+
private projectModelBreakers: Record<string, number> = {};
|
|
70
|
+
private provider429Events: Array<{ ts: number; projectId: string; modelKey: string; account: string }> = [];
|
|
59
71
|
|
|
60
72
|
constructor(private config: Config) {
|
|
61
73
|
this.initAccounts();
|
|
@@ -83,6 +95,8 @@ export class AccountRotator {
|
|
|
83
95
|
inFlightByModel: {},
|
|
84
96
|
allowFreshWindowStartsOverride: false,
|
|
85
97
|
quotaWindows: {},
|
|
98
|
+
dailyRequestCount: 0,
|
|
99
|
+
dailyRequestDay: currentUtcDay(),
|
|
86
100
|
}));
|
|
87
101
|
}
|
|
88
102
|
|
|
@@ -109,11 +123,18 @@ export class AccountRotator {
|
|
|
109
123
|
this.protectivePauseUntil = state.protectivePauseUntil ?? 0;
|
|
110
124
|
this.protectivePauseReason = state.protectivePauseReason ?? null;
|
|
111
125
|
this.allowFreshWindowStarts = state.allowFreshWindowStarts ?? true;
|
|
126
|
+
this.safetyDay = state.safety?.day ?? currentUtcDay();
|
|
127
|
+
this.projectRequests = state.safety?.projectRequests ?? {};
|
|
128
|
+
this.projectModelBreakers = state.safety?.projectModelBreakers ?? {};
|
|
129
|
+
this.provider429Events = state.safety?.provider429Events ?? [];
|
|
130
|
+
this.rollDailySafetyIfNeeded(Date.now());
|
|
112
131
|
|
|
113
132
|
for (const account of this.accounts) {
|
|
114
133
|
const saved = state.accounts[account.config.email];
|
|
115
134
|
if (saved) {
|
|
116
135
|
account.totalRequests = saved.totalRequests;
|
|
136
|
+
account.dailyRequestCount = saved.dailyRequestCount ?? 0;
|
|
137
|
+
account.dailyRequestDay = saved.dailyRequestDay ?? currentUtcDay();
|
|
117
138
|
account.cooldownsByModel = saved.cooldownsByModel ?? {};
|
|
118
139
|
if (saved.cooldownUntil !== undefined && Object.keys(account.cooldownsByModel).length === 0) {
|
|
119
140
|
// legacy migration: apply global cooldown to default
|
|
@@ -186,11 +207,19 @@ export class AccountRotator {
|
|
|
186
207
|
protectivePauseUntil: this.protectivePauseUntil,
|
|
187
208
|
protectivePauseReason: this.protectivePauseReason,
|
|
188
209
|
allowFreshWindowStarts: this.allowFreshWindowStarts,
|
|
210
|
+
safety: {
|
|
211
|
+
day: this.safetyDay,
|
|
212
|
+
projectRequests: { ...this.projectRequests },
|
|
213
|
+
projectModelBreakers: { ...this.projectModelBreakers },
|
|
214
|
+
provider429Events: [...this.provider429Events],
|
|
215
|
+
},
|
|
189
216
|
accounts: {},
|
|
190
217
|
};
|
|
191
218
|
for (const account of this.accounts) {
|
|
192
219
|
state.accounts[account.config.email] = {
|
|
193
220
|
totalRequests: account.totalRequests,
|
|
221
|
+
dailyRequestCount: account.dailyRequestCount,
|
|
222
|
+
dailyRequestDay: account.dailyRequestDay,
|
|
194
223
|
cooldownsByModel: { ...account.cooldownsByModel },
|
|
195
224
|
quotaExhaustedAt: account.quotaExhaustedAt,
|
|
196
225
|
disabled: account.disabled,
|
|
@@ -545,6 +574,54 @@ export class AccountRotator {
|
|
|
545
574
|
return account;
|
|
546
575
|
}
|
|
547
576
|
|
|
577
|
+
private rollDailySafetyIfNeeded(now: number): void {
|
|
578
|
+
const day = currentUtcDay(now);
|
|
579
|
+
if (this.safetyDay === day) return;
|
|
580
|
+
this.safetyDay = day;
|
|
581
|
+
this.projectRequests = {};
|
|
582
|
+
for (const account of this.accounts) {
|
|
583
|
+
account.dailyRequestDay = day;
|
|
584
|
+
account.dailyRequestCount = 0;
|
|
585
|
+
}
|
|
586
|
+
}
|
|
587
|
+
|
|
588
|
+
private getAccountDailyCount(account: AccountRuntime, now: number): number {
|
|
589
|
+
const day = currentUtcDay(now);
|
|
590
|
+
if (account.dailyRequestDay !== day) {
|
|
591
|
+
account.dailyRequestDay = day;
|
|
592
|
+
account.dailyRequestCount = 0;
|
|
593
|
+
}
|
|
594
|
+
return account.dailyRequestCount;
|
|
595
|
+
}
|
|
596
|
+
|
|
597
|
+
private getProjectDailyCount(projectId: string, now: number): number {
|
|
598
|
+
this.rollDailySafetyIfNeeded(now);
|
|
599
|
+
return this.projectRequests[projectId] ?? 0;
|
|
600
|
+
}
|
|
601
|
+
|
|
602
|
+
private getProjectInFlight(modelKey: string, projectId: string): number {
|
|
603
|
+
return this.accounts
|
|
604
|
+
.filter((account) => account.config.projectId === projectId)
|
|
605
|
+
.reduce((sum, account) => sum + (account.inFlightByModel[modelKey] ?? 0), 0);
|
|
606
|
+
}
|
|
607
|
+
|
|
608
|
+
private isProjectModelBreakerActive(projectId: string, modelKey: string, now: number): boolean {
|
|
609
|
+
const until = this.projectModelBreakers[projectModelKey(projectId, modelKey)] ?? 0;
|
|
610
|
+
if (until <= now) {
|
|
611
|
+
if (until > 0) delete this.projectModelBreakers[projectModelKey(projectId, modelKey)];
|
|
612
|
+
return false;
|
|
613
|
+
}
|
|
614
|
+
return true;
|
|
615
|
+
}
|
|
616
|
+
|
|
617
|
+
private getUnavailableReasonForModel(account: AccountRuntime, modelKey: string, now: number): string | null {
|
|
618
|
+
if (this.isProjectModelBreakerActive(account.config.projectId, modelKey, now)) return "project circuit breaker active";
|
|
619
|
+
if (this.getProjectInFlight(modelKey, account.config.projectId) >= (this.config.maxConcurrentRequestsPerProjectModel ?? 1)) return "project concurrency limit reached";
|
|
620
|
+
if (this.getAccountDailyCount(account, now) >= (this.config.dailyAccountStopRequests ?? 350)) return "daily account budget exhausted";
|
|
621
|
+
if (this.getProjectDailyCount(account.config.projectId, now) >= (this.config.dailyProjectStopRequests ?? 1200)) return "daily project budget exhausted";
|
|
622
|
+
return null;
|
|
623
|
+
}
|
|
624
|
+
|
|
548
625
|
// =========================================================================
|
|
549
626
|
// Account Selection (per-model)
|
|
550
627
|
// =========================================================================
|
|
@@ -1001,13 +1078,58 @@ export class AccountRotator {
|
|
|
1001
1078
|
account.cooldownsByModel[modelKey] = now + cooldownMs;
|
|
1002
1079
|
account.quotaExhaustedAt = now;
|
|
1003
1080
|
|
|
1081
|
+
this.log(
|
|
1082
|
+
`${account.config.label || account.config.email} [${modelKey}]: EXHAUSTED, cooldown ${Math.ceil(cooldownMs / 1000)}s`,
|
|
1083
|
+
"warn",
|
|
1084
|
+
);
|
|
1085
|
+
this.saveState();
|
|
1086
|
+
}
|
|
1087
|
+
|
|
1088
|
+
recordProvider429(account: AccountRuntime, model: string | undefined, cooldownMs: number): void {
|
|
1089
|
+
const now = Date.now();
|
|
1090
|
+
const modelKey = model ? (resolveQuotaModelKey(model) ?? "__default__") : "__default__";
|
|
1091
|
+
const windowMs = this.config.projectCircuitBreakerWindowMs ?? 10 * 60 * 1000;
|
|
1092
|
+
const threshold = this.config.projectCircuitBreaker429Threshold ?? 3;
|
|
1093
|
+
const breakerCooldownMs = this.config.projectCircuitBreakerCooldownMs ?? 60 * 60 * 1000;
|
|
1094
|
+
const projectId = account.config.projectId;
|
|
1095
|
+
this.provider429Events = this.provider429Events
|
|
1096
|
+
.filter((event) => now - event.ts <= windowMs)
|
|
1097
|
+
.concat({ ts: now, projectId, modelKey, account: account.config.email });
|
|
1098
|
+
const uniqueAccounts = new Set(
|
|
1099
|
+
this.provider429Events
|
|
1100
|
+
.filter((event) => event.projectId === projectId && event.modelKey === modelKey)
|
|
1101
|
+
.map((event) => event.account),
|
|
1102
|
+
);
|
|
1103
|
+
if (uniqueAccounts.size >= threshold) {
|
|
1104
|
+
const until = now + Math.max(cooldownMs, breakerCooldownMs);
|
|
1105
|
+
this.projectModelBreakers[projectModelKey(projectId, modelKey)] = until;
|
|
1004
1106
|
this.log(
|
|
1005
|
-
|
|
1107
|
+
`[${modelKey}] Project circuit breaker active for projectId=${projectId} after ${uniqueAccounts.size} accounts hit 429; cooldown ${Math.ceil((until - now) / 1000)}s`,
|
|
1006
1108
|
"warn",
|
|
1007
1109
|
);
|
|
1110
|
+
}
|
|
1111
|
+
this.saveState();
|
|
1112
|
+
}
|
|
1113
|
+
|
|
1114
|
+
recordUpstreamAttempt(account: AccountRuntime): void {
|
|
1115
|
+
const now = Date.now();
|
|
1116
|
+
this.rollDailySafetyIfNeeded(now);
|
|
1117
|
+
this.getAccountDailyCount(account, now);
|
|
1118
|
+
account.dailyRequestCount++;
|
|
1119
|
+
this.projectRequests[account.config.projectId] = (this.projectRequests[account.config.projectId] ?? 0) + 1;
|
|
1008
1120
|
this.saveState();
|
|
1009
1121
|
}
|
|
1010
1122
|
|
|
1123
|
+
getSafetyJitterMs(account: AccountRuntime): number {
|
|
1124
|
+
const now = Date.now();
|
|
1125
|
+
const accountSlow = this.getAccountDailyCount(account, now) >= (this.config.dailyAccountSlowRequests ?? 250);
|
|
1126
|
+
const projectSlow = this.getProjectDailyCount(account.config.projectId, now) >= (this.config.dailyProjectSlowRequests ?? 900);
|
|
1127
|
+
if (!accountSlow && !projectSlow) return 0;
|
|
1128
|
+
const min = this.config.slowModeJitterMinMs ?? 8_000;
|
|
1129
|
+
const max = Math.max(min, this.config.slowModeJitterMaxMs ?? 25_000);
|
|
1130
|
+
return Math.floor(min + Math.random() * (max - min + 1));
|
|
1131
|
+
}
|
|
1132
|
+
|
|
1011
1133
|
markError(account: AccountRuntime, error: string): void {
|
|
1012
1134
|
account.lastError = error;
|
|
1013
1135
|
account.consecutiveErrors++;
|
|
@@ -1139,6 +1261,7 @@ export class AccountRotator {
|
|
|
1139
1261
|
const modelCooldown = account.cooldownsByModel[modelKey] ?? 0;
|
|
1140
1262
|
if (modelCooldown > now) return false;
|
|
1141
1263
|
if ((account.inFlightByModel[modelKey] ?? 0) >= (this.config.maxConcurrentRequestsPerAccount ?? 1)) return false;
|
|
1264
|
+
if (this.getUnavailableReasonForModel(account, modelKey, now)) return false;
|
|
1142
1265
|
return true;
|
|
1143
1266
|
}
|
|
1144
1267
|
|
|
@@ -1348,6 +1471,8 @@ export class AccountRotator {
|
|
|
1348
1471
|
inFlightByModel: {},
|
|
1349
1472
|
allowFreshWindowStartsOverride: false,
|
|
1350
1473
|
quotaWindows: {},
|
|
1474
|
+
dailyRequestCount: 0,
|
|
1475
|
+
dailyRequestDay: currentUtcDay(),
|
|
1351
1476
|
};
|
|
1352
1477
|
this.accounts.push(runtime);
|
|
1353
1478
|
this.config.accounts.push(runtime.config);
|
package/src/types.ts
CHANGED
|
@@ -6,6 +6,8 @@ export interface AccountConfig {
|
|
|
6
6
|
email: string;
|
|
7
7
|
refreshToken: string;
|
|
8
8
|
projectId: string;
|
|
9
|
+
// How the projectId was obtained.
|
|
10
|
+
projectSource?: "google" | "manual";
|
|
9
11
|
label?: string;
|
|
10
12
|
// Optional - pro/free is detected dynamically from quota API reset times
|
|
11
13
|
type?: AccountType;
|
|
@@ -25,6 +27,20 @@ export interface Config {
|
|
|
25
27
|
proSlots?: number;
|
|
26
28
|
// Hard cap on parallel requests per account. Conservative default is 1.
|
|
27
29
|
maxConcurrentRequestsPerAccount?: number;
|
|
30
|
+
// Hard cap on parallel requests per projectId/model. Conservative default is 1.
|
|
31
|
+
maxConcurrentRequestsPerProjectModel?: number;
|
|
32
|
+
// Pause projectId/model when several accounts hit provider 429 in a short window. Defaults: 3 hits / 10min / 60min pause.
|
|
33
|
+
projectCircuitBreaker429Threshold?: number;
|
|
34
|
+
projectCircuitBreakerWindowMs?: number;
|
|
35
|
+
projectCircuitBreakerCooldownMs?: number;
|
|
36
|
+
// Daily safety budgets. Defaults: account slow 250, account stop 350, project slow 900, project stop 1200.
|
|
37
|
+
dailyAccountSlowRequests?: number;
|
|
38
|
+
dailyAccountStopRequests?: number;
|
|
39
|
+
dailyProjectSlowRequests?: number;
|
|
40
|
+
dailyProjectStopRequests?: number;
|
|
41
|
+
// Add small delay before upstream call when an account/project is in slow mode. Default: 8-25s.
|
|
42
|
+
slowModeJitterMinMs?: number;
|
|
43
|
+
slowModeJitterMaxMs?: number;
|
|
28
44
|
// Pause all routing after a serious provider flag. Default: 6h.
|
|
29
45
|
protectivePauseMs?: number;
|
|
30
46
|
// Use request-count rotation only before quota data is available. Default: true.
|
|
@@ -139,6 +155,8 @@ export interface AccountRuntime {
|
|
|
139
155
|
inFlightByModel: Record<string, number>;
|
|
140
156
|
allowFreshWindowStartsOverride: boolean;
|
|
141
157
|
quotaWindows: QuotaWindowHistory;
|
|
158
|
+
dailyRequestCount: number;
|
|
159
|
+
dailyRequestDay: string;
|
|
142
160
|
}
|
|
143
161
|
|
|
144
162
|
// Per-model rotation state tracked by the rotator
|
|
@@ -164,6 +182,13 @@ export interface DualWindowTracker {
|
|
|
164
182
|
// Per-account quota window tracking: keyed by model key
|
|
165
183
|
export type QuotaWindowHistory = Record<string, DualWindowTracker>;
|
|
166
184
|
|
|
185
|
+
export interface PersistedSafetyState {
|
|
186
|
+
day: string;
|
|
187
|
+
projectRequests: Record<string, number>;
|
|
188
|
+
projectModelBreakers: Record<string, number>;
|
|
189
|
+
provider429Events: Array<{ ts: number; projectId: string; modelKey: string; account: string }>;
|
|
190
|
+
}
|
|
191
|
+
|
|
167
192
|
export interface PersistedState {
|
|
168
193
|
// Per-model active account index
|
|
169
194
|
modelAccounts: Record<string, number>;
|
|
@@ -174,10 +199,13 @@ export interface PersistedState {
|
|
|
174
199
|
protectivePauseUntil?: number;
|
|
175
200
|
protectivePauseReason?: string | null;
|
|
176
201
|
allowFreshWindowStarts?: boolean;
|
|
202
|
+
safety?: PersistedSafetyState;
|
|
177
203
|
accounts: Record<
|
|
178
204
|
string,
|
|
179
205
|
{
|
|
180
206
|
totalRequests: number;
|
|
207
|
+
dailyRequestCount?: number;
|
|
208
|
+
dailyRequestDay?: string;
|
|
181
209
|
cooldownUntil?: number; // legacy fallback
|
|
182
210
|
cooldownsByModel?: Record<string, number>;
|
|
183
211
|
quotaExhaustedAt: number;
|
package/src/validators.ts
CHANGED
|
@@ -63,6 +63,16 @@ export function validateConfig(value: unknown): ValidationResult<Config> {
|
|
|
63
63
|
if (value.quotaPollIntervalMs !== undefined && !isPositiveNumber(value.quotaPollIntervalMs)) errors.push("config.quotaPollIntervalMs must be a positive number");
|
|
64
64
|
if (value.proSlots !== undefined && !isPositiveNumber(value.proSlots)) errors.push("config.proSlots must be a positive number");
|
|
65
65
|
if (value.maxConcurrentRequestsPerAccount !== undefined && !isPositiveNumber(value.maxConcurrentRequestsPerAccount)) errors.push("config.maxConcurrentRequestsPerAccount must be a positive number");
|
|
66
|
+
if (value.maxConcurrentRequestsPerProjectModel !== undefined && !isPositiveNumber(value.maxConcurrentRequestsPerProjectModel)) errors.push("config.maxConcurrentRequestsPerProjectModel must be a positive number");
|
|
67
|
+
if (value.projectCircuitBreaker429Threshold !== undefined && !isPositiveNumber(value.projectCircuitBreaker429Threshold)) errors.push("config.projectCircuitBreaker429Threshold must be a positive number");
|
|
68
|
+
if (value.projectCircuitBreakerWindowMs !== undefined && !isPositiveNumber(value.projectCircuitBreakerWindowMs)) errors.push("config.projectCircuitBreakerWindowMs must be a positive number");
|
|
69
|
+
if (value.projectCircuitBreakerCooldownMs !== undefined && !isPositiveNumber(value.projectCircuitBreakerCooldownMs)) errors.push("config.projectCircuitBreakerCooldownMs must be a positive number");
|
|
70
|
+
if (value.dailyAccountSlowRequests !== undefined && !isPositiveNumber(value.dailyAccountSlowRequests)) errors.push("config.dailyAccountSlowRequests must be a positive number");
|
|
71
|
+
if (value.dailyAccountStopRequests !== undefined && !isPositiveNumber(value.dailyAccountStopRequests)) errors.push("config.dailyAccountStopRequests must be a positive number");
|
|
72
|
+
if (value.dailyProjectSlowRequests !== undefined && !isPositiveNumber(value.dailyProjectSlowRequests)) errors.push("config.dailyProjectSlowRequests must be a positive number");
|
|
73
|
+
if (value.dailyProjectStopRequests !== undefined && !isPositiveNumber(value.dailyProjectStopRequests)) errors.push("config.dailyProjectStopRequests must be a positive number");
|
|
74
|
+
if (value.slowModeJitterMinMs !== undefined && !isNonNegativeNumber(value.slowModeJitterMinMs)) errors.push("config.slowModeJitterMinMs must be a non-negative number");
|
|
75
|
+
if (value.slowModeJitterMaxMs !== undefined && !isNonNegativeNumber(value.slowModeJitterMaxMs)) errors.push("config.slowModeJitterMaxMs must be a non-negative number");
|
|
66
76
|
if (value.protectivePauseMs !== undefined && !isNonNegativeNumber(value.protectivePauseMs)) errors.push("config.protectivePauseMs must be a non-negative number");
|
|
67
77
|
if (value.useRequestCountRotationWhenQuotaUnknownOnly !== undefined && typeof value.useRequestCountRotationWhenQuotaUnknownOnly !== "boolean") {
|
|
68
78
|
errors.push("config.useRequestCountRotationWhenQuotaUnknownOnly must be a boolean");
|