agy-crew 0.9.3 → 0.9.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/README.zh-CN.md +1 -1
- package/companion/agy-companion.mjs +61 -23
- package/companion/pool/lease.mjs +20 -10
- package/companion/pool/results.mjs +13 -1
- package/companion/pool/select.mjs +22 -9
- package/docs/REFERENCE.md +11 -4
- package/docs/REFERENCE.zh-CN.md +10 -3
- package/opencode-skills/agy-crew-jobs/SKILL.md +1 -0
- package/opencode-skills/agy-crew-jobs/references/troubleshooting.md +8 -0
- package/package.json +1 -1
- package/pi-skills/agy-crew-jobs/SKILL.md +1 -0
- package/pi-skills/agy-crew-jobs/references/troubleshooting.md +8 -0
- package/skills/jobs/SKILL.md +1 -0
- package/skills/jobs/references/troubleshooting.md +8 -0
package/README.md
CHANGED
|
@@ -244,7 +244,7 @@ agy-crew accounts list # shows work1 as ready
|
|
|
244
244
|
|
|
245
245
|
From then on, runs pick an account automatically (`--account auto` is the default): an idle account first, then the one with the most known quota left. Repeat `login` with another name for more accounts. Your own agy login (**live**) is never pooled: `auto` never uses it and it never lends credentials. It is used only with `--account live`, or when the pool has no enabled account, so an empty pool changes nothing.
|
|
246
246
|
|
|
247
|
-
When an account hits a quota limit, the companion records a cooldown for that account and model family (until the reset time agy reports, or 60 minutes if none). A run that had not started a conversation fails over to another account. A run in the middle of a conversation stays bound to its account, because the history lives there; if that account is cooling, an idle account lends its credential for the rest of the run and the token files are restored afterwards, including after a crash. If nothing can continue, the job ends with `quota_exhausted` and a `retry_at`, and you decide when to `continue`. `agy-crew accounts quota --refresh` asks Google's Cloud Code service for current quota; that is the only command that uses the network.
|
|
247
|
+
When an account hits a quota limit, the companion records a cooldown for that account and model family (until the reset time agy reports, or 60 minutes if none). A run that had not started a conversation fails over to another account. A run in the middle of a conversation stays bound to its account, because the history lives there; if that account is cooling, an idle account lends its credential for the rest of the run and the token files are restored afterwards, including after a crash. If nothing can continue, the job ends with `quota_exhausted` and a `retry_at`, and you decide when to `continue`. An upstream 503 (`No capacity available for model ...`) parks the account for 3 minutes so the next run uses another one, and a task that had changed nothing is rerun from its original prompt as a new conversation rather than continued; if it still ends with `upstream_unavailable` before agy did anything, start the task again instead of using `continue`. `agy-crew accounts quota --refresh` asks Google's Cloud Code service for current quota; that is the only command that uses the network.
|
|
248
248
|
|
|
249
249
|
Already have accounts? `agy-crew accounts import work2 --home <dir>` copies the token from an existing HOME-shaped directory, such as an agy-cli-manager account directory (`~/.agy-cli-manager/accounts/<name>`). agy-cli-manager is not needed otherwise; import is just a file copy plus the usual email and duplicate checks. Other subcommands: `enable`, `disable`, `cooldown clear`, `remove --yes`. Full details in the [reference](docs/REFERENCE.md#accounts).
|
|
250
250
|
|
package/README.zh-CN.md
CHANGED
|
@@ -205,7 +205,7 @@ agy-crew accounts list # work1 显示为 ready
|
|
|
205
205
|
|
|
206
206
|
之后每次运行都会自动选择账号(默认就是 `--account auto`):优先选空闲账号,其次选已知剩余配额最多的。想要更多账号,换个名字再执行一次 `login` 即可。你自己的 agy 登录(**live**)永远不属于账号池:`auto` 不会用它,它也不会借出凭据。只有显式传入 `--account live`,或账号池里没有任何启用的账号时才会用到它,所以账号池为空时一切照旧。
|
|
207
207
|
|
|
208
|
-
账号触及配额上限时,companion 会为该账号和模型家族记录冷却,直到 agy 报告的重置时间,没有报告则为 60 分钟。尚未开始会话的任务会转到另一个账号;已经在会话中的任务仍绑定原账号,因为会话历史保存在那里。如果该账号正在冷却,就由一个空闲账号在本次运行中出借凭据,结束后令牌文件会还原,崩溃后也会恢复。所有账号都无法继续时,任务以 `quota_exhausted` 结束并给出 `retry_at`,由你决定何时 `continue`。`agy-crew accounts quota --refresh` 会向 Google 的 Cloud Code 服务查询当前配额,这是唯一会联网的命令。
|
|
208
|
+
账号触及配额上限时,companion 会为该账号和模型家族记录冷却,直到 agy 报告的重置时间,没有报告则为 60 分钟。尚未开始会话的任务会转到另一个账号;已经在会话中的任务仍绑定原账号,因为会话历史保存在那里。如果该账号正在冷却,就由一个空闲账号在本次运行中出借凭据,结束后令牌文件会还原,崩溃后也会恢复。所有账号都无法继续时,任务以 `quota_exhausted` 结束并给出 `retry_at`,由你决定何时 `continue`。上游 503(`No capacity available for model ...`)会让该账号冷却 3 分钟,下一次运行改用其他账号;没有改动过工作区的任务会用原始提示词开新会话重跑,而不是继续旧会话。如果任务仍以 `upstream_unavailable` 结束且 agy 还什么都没做,请重新开始任务,不要用 `continue`。`agy-crew accounts quota --refresh` 会向 Google 的 Cloud Code 服务查询当前配额,这是唯一会联网的命令。
|
|
209
209
|
|
|
210
210
|
已经有账号了?`agy-crew accounts import work2 --home <dir>` 会从现有的 HOME 形态目录复制令牌,例如 agy-cli-manager 的账号目录(`~/.agy-cli-manager/accounts/<name>`)。agy-crew 并不依赖 agy-cli-manager,导入只是复制文件,再做与登录相同的邮箱和重复检查。其他子命令:`enable`、`disable`、`cooldown clear`、`remove --yes`。完整说明见[参考手册](docs/REFERENCE.zh-CN.md#账号池)。
|
|
211
211
|
|
|
@@ -111,7 +111,7 @@ import { atomicJSON, runStreaming, processIdentity } from './stream-worker.mjs';
|
|
|
111
111
|
import { withStateLock, replaceFile, readTextRetry } from './state-lock.mjs';
|
|
112
112
|
import { cmdAccounts, poolSettingsFiles } from './accounts.mjs';
|
|
113
113
|
import { acquireRun, boundAccount, preflight, recordCooldown, markAuthError } from './pool/lease.mjs';
|
|
114
|
-
import { classifyFailureText, modelFamily } from './pool/results.mjs';
|
|
114
|
+
import { classifyFailureText, isCooldownReason, modelFamily } from './pool/results.mjs';
|
|
115
115
|
import { loadPool, updatePool, poolExists } from './pool/state.mjs';
|
|
116
116
|
|
|
117
117
|
const SELF = fileURLToPath(import.meta.url);
|
|
@@ -1165,6 +1165,9 @@ function treeDeltaReport(mode, before, after) {
|
|
|
1165
1165
|
// ---------------------------------------------------------------------------
|
|
1166
1166
|
|
|
1167
1167
|
const MAX_POOL_RETRIES = 3;
|
|
1168
|
+
// agy already retries a 503 about eight times (a couple of minutes) before it
|
|
1169
|
+
// gives up, so each extra attempt on another account is expensive.
|
|
1170
|
+
const MAX_UNAVAILABLE_RETRIES = 2;
|
|
1168
1171
|
const QUOTA_CONTINUE_PROMPT = 'Your previous turn stopped because the account hit a quota limit. ' +
|
|
1169
1172
|
'Continue the same task from where you stopped; do not repeat finished steps.';
|
|
1170
1173
|
|
|
@@ -1207,15 +1210,20 @@ async function invokeOnce(invoke, execution) {
|
|
|
1207
1210
|
return { result, failure: resultFailure(result) };
|
|
1208
1211
|
}
|
|
1209
1212
|
|
|
1210
|
-
function quotaFailure(resolved, opts, { message, retry_at, conversation }) {
|
|
1211
|
-
const
|
|
1212
|
-
const
|
|
1213
|
-
|
|
1214
|
-
|
|
1215
|
-
|
|
1213
|
+
function quotaFailure(resolved, opts, { message, retry_at, conversation, reason = 'quota_exhausted' }) {
|
|
1214
|
+
const unavailable = reason === 'upstream_unavailable';
|
|
1215
|
+
const when = retry_at || (unavailable ? 'the upstream recovers' : 'the account quota resets');
|
|
1216
|
+
const target = opts.jobId ? `--job ${opts.jobId}` : `--conversation ${conversation}`;
|
|
1217
|
+
const recovery = unavailable
|
|
1218
|
+
? 'the upstream model had no capacity (503). If agy had not done anything yet, start the task again as a new run with a rewritten prompt' +
|
|
1219
|
+
`${opts.jobId ? ` (restart ${opts.jobId})` : ''} and do not use continue. ` +
|
|
1220
|
+
(conversation ? `Only if agy already changed the workspace, inspect it and continue ${target} after ${when}.` : `Start a new run after ${when}, or choose another model family.`)
|
|
1221
|
+
: conversation
|
|
1222
|
+
? `continue ${target} after ${when}; when another pool account is idle, a credential loan is used automatically`
|
|
1223
|
+
: `start a new run after ${when}, pass --account live, or choose another model family`;
|
|
1216
1224
|
const text = `${message}\n${recovery}`;
|
|
1217
|
-
if (resolved.background) throw Object.assign(new Error(text), { reason
|
|
1218
|
-
die(`${text}\n${JSON.stringify({ status: conversation ? 'attention' : 'error', reason
|
|
1225
|
+
if (resolved.background) throw Object.assign(new Error(text), { reason, retry_at: retry_at || null, recovery });
|
|
1226
|
+
die(`${text}\n${JSON.stringify({ status: conversation ? 'attention' : 'error', reason,
|
|
1219
1227
|
retry_at: retry_at || null, conversation_id: conversation || null, recovery }, null, 2)}`, conversation ? 5 : 1);
|
|
1220
1228
|
}
|
|
1221
1229
|
|
|
@@ -1223,15 +1231,20 @@ function latestConversation(jobId) {
|
|
|
1223
1231
|
return jobId ? loadState().jobs?.find((j) => j.id === jobId)?.conversation_id || null : null;
|
|
1224
1232
|
}
|
|
1225
1233
|
|
|
1226
|
-
/** Run agy on a pool account chosen under a lease. A quota
|
|
1227
|
-
* cooldown and then either fails over to another account
|
|
1228
|
-
* yet) or continues the bound conversation on a borrowed
|
|
1229
|
-
|
|
1234
|
+
/** Run agy on a pool account chosen under a lease. A quota or upstream-503
|
|
1235
|
+
* failure records a cooldown and then either fails over to another account
|
|
1236
|
+
* (nothing started yet) or continues the bound conversation on a borrowed
|
|
1237
|
+
* credential. A 503 on a task that began as a new conversation and left the
|
|
1238
|
+
* workspace untouched is not continued: the original prompt runs again in a
|
|
1239
|
+
* fresh conversation on another account. `untouched` tells whether agy has
|
|
1240
|
+
* changed nothing so far. */
|
|
1241
|
+
async function runWithPool(resolved, invoke, opts, execution, untouched = () => false) {
|
|
1230
1242
|
const family = modelFamily(resolved.model);
|
|
1243
|
+
const original = { prompt: invoke.prompt, conversation: invoke.conversation };
|
|
1231
1244
|
for (let retries = 0; ; retries++) {
|
|
1232
1245
|
const got = acquireRun({ requested: resolved.account_request, family, conversation: invoke.conversation, jobId: opts.jobId });
|
|
1233
1246
|
if (got.error) {
|
|
1234
|
-
if (got.error.reason
|
|
1247
|
+
if (!isCooldownReason(got.error.reason)) die(got.error.message);
|
|
1235
1248
|
quotaFailure(resolved, opts, { ...got.error, conversation: invoke.conversation });
|
|
1236
1249
|
}
|
|
1237
1250
|
const held = got.held;
|
|
@@ -1258,18 +1271,27 @@ async function runWithPool(resolved, invoke, opts, execution) {
|
|
|
1258
1271
|
if (error) rethrow(error);
|
|
1259
1272
|
return { result, held }; // triage reports it with its usual message
|
|
1260
1273
|
}
|
|
1261
|
-
const
|
|
1274
|
+
const unavailable = failure.kind === 'upstream_unavailable';
|
|
1275
|
+
const cooldown = recordCooldown(credit, family, failure.text, opts.jobId, failure.kind);
|
|
1262
1276
|
const conversation = result?.payload?.conversation_id || latestConversation(opts.jobId) || invoke.conversation;
|
|
1263
|
-
if (retries < MAX_POOL_RETRIES) {
|
|
1277
|
+
if (retries < (unavailable ? MAX_UNAVAILABLE_RETRIES : MAX_POOL_RETRIES)) {
|
|
1264
1278
|
if (!conversation && held.auto) continue; // nothing started: just pick another account
|
|
1279
|
+
if (unavailable && held.auto && !original.conversation && untouched()) {
|
|
1280
|
+
// agy did nothing: rewrite the task from its original prompt instead of continuing it
|
|
1281
|
+
invoke.prompt = original.prompt;
|
|
1282
|
+
invoke.conversation = null;
|
|
1283
|
+
continue;
|
|
1284
|
+
}
|
|
1265
1285
|
if (conversation && boundAccount(loadPool(), conversation) !== 'live') {
|
|
1266
1286
|
invoke.conversation = conversation; // the next acquire finds the account cooling and borrows a credential
|
|
1267
1287
|
invoke.prompt = QUOTA_CONTINUE_PROMPT;
|
|
1268
1288
|
continue;
|
|
1269
1289
|
}
|
|
1270
1290
|
}
|
|
1271
|
-
quotaFailure(resolved, opts, { conversation, retry_at: cooldown.retry_at,
|
|
1272
|
-
message:
|
|
1291
|
+
quotaFailure(resolved, opts, { conversation, retry_at: cooldown.retry_at, reason: failure.kind,
|
|
1292
|
+
message: unavailable
|
|
1293
|
+
? `agy account "${credit}" got ${family} unavailable upstream (503, upstream_unavailable).`
|
|
1294
|
+
: `agy account "${credit}" hit a ${family} quota limit (quota_exhausted).` });
|
|
1273
1295
|
}
|
|
1274
1296
|
}
|
|
1275
1297
|
|
|
@@ -1289,7 +1311,12 @@ async function executeRun(resolved, prompt, opts, execution = null) {
|
|
|
1289
1311
|
};
|
|
1290
1312
|
let result, response, held;
|
|
1291
1313
|
try {
|
|
1292
|
-
|
|
1314
|
+
const untouched = () => {
|
|
1315
|
+
if (resolved.mode === 'ask') return true;
|
|
1316
|
+
const after = porcelainSnapshot();
|
|
1317
|
+
return !!workspaceBefore && !!after && porcelainDelta(workspaceBefore, after).length === 0;
|
|
1318
|
+
};
|
|
1319
|
+
({ result, held } = await runWithPool(resolved, invoke, opts, execution, untouched));
|
|
1293
1320
|
response = triageResult(result, resolved.mode, resolved.profile, resolved.profileSource, resolved.model);
|
|
1294
1321
|
} catch (error) {
|
|
1295
1322
|
error.workspace = {
|
|
@@ -1348,7 +1375,7 @@ function cmdRun(mode, opts) {
|
|
|
1348
1375
|
* another account. Quota problems are left to the run itself. */
|
|
1349
1376
|
function preflightAccount(resolved) {
|
|
1350
1377
|
const problem = preflight({ requested: resolved.account_request, family: modelFamily(resolved.model), conversation: resolved.conversation });
|
|
1351
|
-
if (problem && problem.reason
|
|
1378
|
+
if (problem && !isCooldownReason(problem.reason)) die(problem.message);
|
|
1352
1379
|
}
|
|
1353
1380
|
|
|
1354
1381
|
async function dispatch(resolved, prompt, opts) {
|
|
@@ -1471,7 +1498,7 @@ async function workerMain(jobId) {
|
|
|
1471
1498
|
if (!job) throw error;
|
|
1472
1499
|
const reason = job.cancel_requested_at ? 'canceled' : error.reason || 'agy_error';
|
|
1473
1500
|
const status = job.status === 'canceled' || reason === 'canceled' ? 'canceled'
|
|
1474
|
-
: (isTimeoutReason(reason) || reason
|
|
1501
|
+
: (isTimeoutReason(reason) || isCooldownReason(reason)) && job.conversation_id ? 'attention' : 'error';
|
|
1475
1502
|
job = finishJob(jobId, (completed) => {
|
|
1476
1503
|
const report = { ...diagnosticPacket(completed), result_exists: true, reason: completed.reason,
|
|
1477
1504
|
workspace: error.workspace, last_snapshot: readObservation(completed) };
|
|
@@ -1650,6 +1677,17 @@ function quotaRecovery(job) {
|
|
|
1650
1677
|
: `No conversation was started. Restart after ${when}, pass --account live, or choose another model family.` };
|
|
1651
1678
|
}
|
|
1652
1679
|
|
|
1680
|
+
/** Recovery for a job that stopped because the upstream model had no capacity (503). */
|
|
1681
|
+
function unavailableRecovery(job) {
|
|
1682
|
+
return { retry_at: job.retry_at || null, spec_file: job.spec_file || null,
|
|
1683
|
+
inspect: 'git status --short; git diff',
|
|
1684
|
+
restart: `restart ${job.id}`,
|
|
1685
|
+
continue: job.conversation_id ? `continue --job ${job.id} --prompt "Continue after inspecting partial workspace changes"` : null,
|
|
1686
|
+
note: 'The upstream model had no capacity (503). If agy had not done anything yet (clean `git status`, no partial result), start the task again ' +
|
|
1687
|
+
`(restart ${job.id}, or a new run with a rewritten prompt) and do not use continue: a continuation stays bound to the same account and conversation. ` +
|
|
1688
|
+
'Use continue only when agy already changed the workspace, after inspecting it. Never retry in a loop.' };
|
|
1689
|
+
}
|
|
1690
|
+
|
|
1653
1691
|
function diagnosticPacket(job) {
|
|
1654
1692
|
let logBytes = null;
|
|
1655
1693
|
try { logBytes = fs.statSync(job.log_file).size; } catch {}
|
|
@@ -1660,8 +1698,8 @@ function diagnosticPacket(job) {
|
|
|
1660
1698
|
result_exists: fs.existsSync(job.result_file), log_file: job.log_file, events_file: job.events_file || null,
|
|
1661
1699
|
conversation_id: job.conversation_id || null, model: job.model || null, profile: job.profile || null,
|
|
1662
1700
|
...(job.account ? { account: job.account } : {}),
|
|
1663
|
-
...(job.reason
|
|
1664
|
-
recovery: isTimeoutReason(job.reason) ? timeoutRecovery(job) : job.reason === 'quota_exhausted' ? quotaRecovery(job) : { inspect: 'git status --short; git diff', spec_file: job.spec_file || null,
|
|
1701
|
+
...(isCooldownReason(job.reason) ? { retry_at: job.retry_at || null } : {}),
|
|
1702
|
+
recovery: isTimeoutReason(job.reason) ? timeoutRecovery(job) : job.reason === 'quota_exhausted' ? quotaRecovery(job) : job.reason === 'upstream_unavailable' ? unavailableRecovery(job) : { inspect: 'git status --short; git diff', spec_file: job.spec_file || null,
|
|
1665
1703
|
continue: job.conversation_id ? `continue --job ${job.id} --prompt "Continue after inspecting partial workspace changes"` : null,
|
|
1666
1704
|
restart: `restart ${job.id}`, note: 'Inspect partial workspace changes first. Recovery creates a linked new job with a fresh budget; nothing is retried automatically.' },
|
|
1667
1705
|
};
|
package/companion/pool/lease.mjs
CHANGED
|
@@ -41,8 +41,9 @@ export function acquireRun(req) {
|
|
|
41
41
|
if (pick.needsLoan) {
|
|
42
42
|
const lender = chooseLender(pool, { family: req.family, exclude: [pick.account] });
|
|
43
43
|
if (!lender) {
|
|
44
|
-
|
|
45
|
-
|
|
44
|
+
const why = pick.reason === 'upstream_unavailable' ? 'is unavailable upstream (503)' : 'has no quota';
|
|
45
|
+
return { error: { reason: pick.reason, retry_at: pick.retry_at,
|
|
46
|
+
message: `account ${pick.account} ${why} for ${req.family} until ${pick.retry_at} and no other pool account is idle to lend a credential.` } };
|
|
46
47
|
}
|
|
47
48
|
const loan = beginLoan(pool, { borrower: pick.account, lender, jobId: req.jobId });
|
|
48
49
|
return { held: { account: pick.account, lender, loanId: loan.id, pooled: true } };
|
|
@@ -67,17 +68,26 @@ export function acquireRun(req) {
|
|
|
67
68
|
}
|
|
68
69
|
|
|
69
70
|
const DEFAULT_COOLDOWN_MS = 60 * 60_000;
|
|
71
|
+
const UNAVAILABLE_COOLDOWN_MS = 3 * 60_000;
|
|
70
72
|
|
|
71
|
-
/** Record a
|
|
72
|
-
*
|
|
73
|
-
*
|
|
74
|
-
|
|
73
|
+
/** Record a cooldown for the account whose credential made the request.
|
|
74
|
+
* A quota failure lasts until the reported reset (one hour when agy did not
|
|
75
|
+
* say); an upstream_unavailable (503) one is a few minutes, just long enough
|
|
76
|
+
* that the next run picks another account, and never shortens a quota
|
|
77
|
+
* cooldown that is still running. Returns the cooldown now in force. */
|
|
78
|
+
export function recordCooldown(name, family, text, jobId, kind = 'quota_exhausted') {
|
|
75
79
|
const now = Date.now();
|
|
76
|
-
const
|
|
77
|
-
|
|
80
|
+
const wait = kind === 'upstream_unavailable' ? UNAVAILABLE_COOLDOWN_MS : (parseResetsIn(text) ?? DEFAULT_COOLDOWN_MS);
|
|
81
|
+
const cooldown = { retry_at: new Date(now + wait).toISOString(),
|
|
82
|
+
reason: kind, recorded_at: new Date(now).toISOString(), job_id: jobId || null,
|
|
78
83
|
message: String(text).trim().slice(0, 500) };
|
|
79
|
-
updatePool((pool) => {
|
|
80
|
-
|
|
84
|
+
return updatePool((pool) => {
|
|
85
|
+
const key = `${name}|${family}`;
|
|
86
|
+
const current = pool.cooldowns[key];
|
|
87
|
+
if (kind === 'upstream_unavailable' && current && Date.parse(current.retry_at) > Date.parse(cooldown.retry_at)) return current;
|
|
88
|
+
pool.cooldowns[key] = cooldown;
|
|
89
|
+
return cooldown;
|
|
90
|
+
});
|
|
81
91
|
}
|
|
82
92
|
|
|
83
93
|
/** An authentication failure parks the account until `accounts login`. */
|
|
@@ -23,10 +23,22 @@ export function isAuthFailed(text) {
|
|
|
23
23
|
['auth', 'credential', 'login', 'token'].some((b) => t.includes(b));
|
|
24
24
|
}
|
|
25
25
|
|
|
26
|
-
/**
|
|
26
|
+
/** The model has no capacity right now ("UNAVAILABLE (code 503): No capacity
|
|
27
|
+
* available for model X on the server"). agy has already retried internally. */
|
|
28
|
+
export function isUpstreamUnavailable(text) {
|
|
29
|
+
const t = String(text || '').toLowerCase();
|
|
30
|
+
return (t.includes('unavailable') && t.includes('503')) || t.includes('no capacity available');
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
/** Failures that park an account for a model family: out of quota, or the
|
|
34
|
+
* upstream has no capacity. */
|
|
35
|
+
export const isCooldownReason = (reason) => reason === 'quota_exhausted' || reason === 'upstream_unavailable';
|
|
36
|
+
|
|
37
|
+
/** 'quota_exhausted' | 'authentication_failed' | 'upstream_unavailable' | null */
|
|
27
38
|
export function classifyFailureText(text) {
|
|
28
39
|
if (isQuotaExhausted(text)) return 'quota_exhausted';
|
|
29
40
|
if (isAuthFailed(text)) return 'authentication_failed';
|
|
41
|
+
if (isUpstreamUnavailable(text)) return 'upstream_unavailable';
|
|
30
42
|
return null;
|
|
31
43
|
}
|
|
32
44
|
|
|
@@ -34,6 +34,11 @@ function usable(pool, a, family, now, hasToken) {
|
|
|
34
34
|
!activeCooldown(pool, a.name, family, now) && !exhaustedSnapshot(pool, a.name, family, now);
|
|
35
35
|
}
|
|
36
36
|
|
|
37
|
+
// Remaining quota counts in 5% steps: accounts that are about equally full
|
|
38
|
+
// rotate by least recent use instead of the fullest one winning every time.
|
|
39
|
+
const QUOTA_STEPS = 20;
|
|
40
|
+
const quotaBucket = (remaining) => (remaining === null ? -1 : Math.round(remaining * QUOTA_STEPS));
|
|
41
|
+
|
|
37
42
|
// fewer leases, then more known quota (unknown after known > 0), then least recently used, then insertion order
|
|
38
43
|
function rank(pool, family, withLeases) {
|
|
39
44
|
return (a, b) => {
|
|
@@ -41,8 +46,8 @@ function rank(pool, family, withLeases) {
|
|
|
41
46
|
const d = leasesOn(pool, a.name).length - leasesOn(pool, b.name).length;
|
|
42
47
|
if (d) return d;
|
|
43
48
|
}
|
|
44
|
-
const ra = knownRemaining(pool, a.name, family)
|
|
45
|
-
const rb = knownRemaining(pool, b.name, family)
|
|
49
|
+
const ra = quotaBucket(knownRemaining(pool, a.name, family));
|
|
50
|
+
const rb = quotaBucket(knownRemaining(pool, b.name, family));
|
|
46
51
|
if (ra !== rb) return rb - ra;
|
|
47
52
|
const ua = a.last_used_at || '', ub = b.last_used_at || '';
|
|
48
53
|
if (ua !== ub) return ua < ub ? -1 : 1;
|
|
@@ -50,15 +55,22 @@ function rank(pool, family, withLeases) {
|
|
|
50
55
|
};
|
|
51
56
|
}
|
|
52
57
|
|
|
53
|
-
|
|
58
|
+
const unavailableCooldown = (c) => c?.reason === 'upstream_unavailable';
|
|
59
|
+
|
|
60
|
+
function coolingError(pool, accounts, family, now) {
|
|
54
61
|
const rows = accounts.map((a) => {
|
|
55
62
|
const c = activeCooldown(pool, a.name, family, now);
|
|
56
63
|
const s = exhaustedSnapshot(pool, a.name, family, now);
|
|
57
|
-
return { name: a.name, at: c?.retry_at || s?.reset_at || null };
|
|
64
|
+
return { name: a.name, at: c?.retry_at || s?.reset_at || null, capacity: unavailableCooldown(c) && !s };
|
|
58
65
|
});
|
|
59
66
|
const times = rows.map((r) => r.at).filter(Boolean).sort();
|
|
60
67
|
const detail = rows.map((r) => ` ${r.name}: ${r.at ? `retry after ${r.at}` : 'unavailable'}`).join('\n');
|
|
61
|
-
|
|
68
|
+
const capacity = rows.every((r) => r.capacity);
|
|
69
|
+
const reason = capacity ? 'upstream_unavailable' : 'quota_exhausted';
|
|
70
|
+
const intro = capacity
|
|
71
|
+
? `every pool account recently got ${family} unavailable upstream (503, upstream_unavailable).`
|
|
72
|
+
: `every pool account is out of ${family} quota (quota_exhausted).`;
|
|
73
|
+
return { error: { reason, retry_at: times[0] || null,
|
|
62
74
|
message: `${intro}\n${detail}\nUse \`--account live\` (your own agy login) or another model family.` } };
|
|
63
75
|
}
|
|
64
76
|
|
|
@@ -76,7 +88,7 @@ export function chooseAccount(pool, { requested = 'auto', family, conversation =
|
|
|
76
88
|
if (a.disabled) return { error: { message: `conversation ${conversation} belongs to account ${a.name}, which is disabled (accounts enable ${a.name})` } };
|
|
77
89
|
if (a.auth_error_at || !hasToken(a.name)) return { error: { message: `conversation ${conversation} belongs to account ${a.name}, which needs login (accounts login ${a.name})` } };
|
|
78
90
|
const c = activeCooldown(pool, a.name, family, now);
|
|
79
|
-
if (c) return { account: a.name, needsLoan: true, retry_at: c.retry_at };
|
|
91
|
+
if (c) return { account: a.name, needsLoan: true, retry_at: c.retry_at, reason: c.reason || 'quota_exhausted' };
|
|
80
92
|
return { account: a.name };
|
|
81
93
|
}
|
|
82
94
|
if (requested === 'live') return { account: 'live' };
|
|
@@ -87,8 +99,9 @@ export function chooseAccount(pool, { requested = 'auto', family, conversation =
|
|
|
87
99
|
if (a.auth_error_at || !hasToken(a.name)) return { error: { message: `account ${a.name} needs login (accounts login ${a.name})` } };
|
|
88
100
|
const c = activeCooldown(pool, a.name, family, now);
|
|
89
101
|
if (c) {
|
|
90
|
-
|
|
91
|
-
|
|
102
|
+
const why = unavailableCooldown(c) ? `was unavailable upstream (503) for ${family}; retry after` : `has no ${family} quota until`;
|
|
103
|
+
return { error: { reason: c.reason || 'quota_exhausted', retry_at: c.retry_at,
|
|
104
|
+
message: `account ${a.name} ${why} ${c.retry_at}. Use \`--account auto\`, another account, or another model family.` } };
|
|
92
105
|
}
|
|
93
106
|
return { account: a.name };
|
|
94
107
|
}
|
|
@@ -97,7 +110,7 @@ export function chooseAccount(pool, { requested = 'auto', family, conversation =
|
|
|
97
110
|
const candidates = enabled.filter((a) => usable(pool, a, family, now, hasToken));
|
|
98
111
|
if (!candidates.length) {
|
|
99
112
|
const cooling = enabled.filter((a) => activeCooldown(pool, a.name, family, now) || exhaustedSnapshot(pool, a.name, family, now));
|
|
100
|
-
if (cooling.length) return coolingError(pool, cooling, family, now
|
|
113
|
+
if (cooling.length) return coolingError(pool, cooling, family, now);
|
|
101
114
|
return { error: { message: 'no pool account is ready: each enabled account needs login (see `accounts list`)' } };
|
|
102
115
|
}
|
|
103
116
|
candidates.sort(rank(pool, family, true));
|
package/docs/REFERENCE.md
CHANGED
|
@@ -237,11 +237,11 @@ Every agy process for a pool account runs with its own HOME (and `USERPROFILE` o
|
|
|
237
237
|
**Selection ranking for `auto`.** A candidate is an enabled account with a token file, no authentication error, no active cooldown for the model family, and no cached Cloud Code snapshot reporting zero remaining quota for the family's quota pool with a reset time in the future. Candidates are ranked by:
|
|
238
238
|
|
|
239
239
|
1. fewest active leases (an idle account beats a busy one);
|
|
240
|
-
2. highest known remaining quota fraction (the lower of the short and weekly windows; unknown ranks after any known value above zero);
|
|
240
|
+
2. highest known remaining quota fraction (the lower of the short and weekly windows, compared in 5% steps so that accounts that are about equally full rotate by least recent use; unknown ranks after any known value above zero);
|
|
241
241
|
3. oldest `last_used_at`;
|
|
242
242
|
4. insertion order.
|
|
243
243
|
|
|
244
|
-
If there is no candidate, the run fails with reason `quota_exhausted
|
|
244
|
+
If there is no candidate, the run fails with reason `quota_exhausted` (or `upstream_unavailable` when every account is cooling only because of 503s), listing each account's earliest `retry_at` and suggesting `--account live` or another model. Concurrent runs share one account only when it is named explicitly, or when `auto` finds no idle candidate (the least-loaded account wins).
|
|
245
245
|
|
|
246
246
|
Selection and the lease that marks the account busy are taken in one locked update, and released when the agy process ends. A lease whose process died is dropped automatically.
|
|
247
247
|
|
|
@@ -259,6 +259,12 @@ The run then recovers, with at most three extra attempts:
|
|
|
259
259
|
- **A conversation id is known and the conversation is bound to a pool account**: a credential loan is attempted (below). If a lender exists, the run continues with the prompt "Your previous turn stopped because the account hit a quota limit. Continue the same task from where you stopped; do not repeat finished steps."
|
|
260
260
|
- **Otherwise the run fails** with reason `quota_exhausted`. A background job gets status `attention` when a conversation id exists and `error` when none does; its result includes `retry_at` and recovery text (`continue --job <id> after <retry_at>`; a credential loan is used automatically once another pool account is idle). A foreground run dies with the same information, exit 5 when a conversation id exists, otherwise 1. Further continuation is the caller's decision; the companion does not loop.
|
|
261
261
|
|
|
262
|
+
**Upstream 503 (`upstream_unavailable`).** When agy gives up with `UNAVAILABLE (code 503)` or `No capacity available for model ...` (it has already retried about eight times, roughly two minutes), the companion records a short cooldown for that account and family, **3 minutes** and with reason `upstream_unavailable`, so the next selection, in this run or a separate one, picks another account. A 503 cooldown never shortens a quota cooldown that is still running. Recovery makes at most two extra attempts:
|
|
263
|
+
|
|
264
|
+
- **No conversation id yet, or the task began as a new conversation and the workspace is unchanged** (always true for `ask`; otherwise `git status` shows nothing new) and the account was chosen by `auto`: the original prompt runs again in a new conversation on another account. It is a rewrite, never a continuation.
|
|
265
|
+
- **Workspace already changed, with a conversation bound to a pool account**: the same credential-loan continuation as for quota.
|
|
266
|
+
- **Otherwise the run fails** with reason `upstream_unavailable`, exit 5 or status `attention` when a conversation id exists, otherwise 1. The recovery text tells the caller to start the task again (`restart <job>`) when agy had not done anything yet, and to use `continue` only when the workspace was already changed. An explicit `--account` is never moved to another account.
|
|
267
|
+
|
|
262
268
|
An **authentication failure** on a pool account (`401`, `403` or `unauthenticated` together with auth, credential, login or token wording) sets the account's auth error. Selection skips it, `list` shows `needs login`, and `accounts login <name>` clears it. Authentication failures are never retried.
|
|
263
269
|
|
|
264
270
|
### Credential loans
|
|
@@ -315,6 +321,7 @@ A refreshed snapshot with remaining quota above zero for a pool clears that acco
|
|
|
315
321
|
| Situation | Reason / output | Exit |
|
|
316
322
|
|---|---|---|
|
|
317
323
|
| No `auto` candidate, or quota hit mid-run with no lender | `quota_exhausted`, with `retry_at` (per account when no candidate) | 5 when a conversation id exists, otherwise 1; a background job ends `attention` or `error` accordingly |
|
|
324
|
+
| 503 / no capacity from the model, no account or lender could continue | `upstream_unavailable`, with `retry_at` (3 minutes after the last 503) | 5 when a conversation id exists, otherwise 1; a background job ends `attention` or `error` accordingly |
|
|
318
325
|
| Authentication failure on a pool account | account marked `needs login`; never retried | follows the normal error path |
|
|
319
326
|
| Explicit `--account <name>` that is unknown, disabled, cooling or needs login | error before launch naming the reason | 1 |
|
|
320
327
|
| `--account` conflicting with a bound conversation | `conversation <id> belongs to account <x>` | 1 |
|
|
@@ -324,7 +331,7 @@ A refreshed snapshot with remaining quota above zero for a pool clears that acco
|
|
|
324
331
|
## Troubleshooting
|
|
325
332
|
|
|
326
333
|
- **"agy reported an error (status ERROR)"** — the companion relays agy's own error verbatim, and appends a cause hint only when the error text actually matches one (invalid model id → run `agy models`; expired auth → run `agy` interactively once to re-login; exhausted quota). If agy reported an error but still returned response text, the companion delivers the response anyway — exit 0, response on stdout, warning on stderr (`done_with_warnings`); a deadline with no response but a known conversation instead signals attention (exit 5).
|
|
327
|
-
- **Quota error or `needs login` on a pool account** — see [Accounts](#accounts). A quota failure records a cooldown and is recovered inside the run when another account or an idle lender exists; otherwise the job ends with `quota_exhausted` and a `retry_at`. An authentication failure is never retried: run `agy-crew accounts login <name>` in your own terminal.
|
|
334
|
+
- **Quota error or `needs login` on a pool account** — see [Accounts](#accounts). A quota failure records a cooldown and is recovered inside the run when another account or an idle lender exists; otherwise the job ends with `quota_exhausted` and a `retry_at`. A 503 (`upstream_unavailable`) parks the account for 3 minutes and reruns an untouched task on another account; if the job still ends with `upstream_unavailable` and agy had not done anything, start it again with `restart <job>` or a rewritten prompt rather than `continue`. An authentication failure is never retried: run `agy-crew accounts login <name>` in your own terminal.
|
|
328
335
|
- **`operation not permitted` on `~/.gemini/...` / `bind: operation not permitted` / sudden "authentication failed" while `agy` works in your terminal** — check whether a harness command sandbox is blocking access. AGY needs its OAuth credentials and a localhost port for its internal language server; allowing workspace writes alone may not provide that access. Where those restrictions apply, use the host's authorization mechanism to run the companion in a context that supports AGY. Use the same permission context to start and manage a job.
|
|
329
336
|
- **False crash report on `wait`/`status` ("finished with status crashed and no stored result")** — the background job was started in one permission or sandbox context (e.g. unsandboxed) and collected from another (e.g. inside a command sandbox). The collector cannot see the worker PID across the sandbox boundary and misclassifies the running job as crashed. Run management commands (`wait`, `status`, `result`) in the same unsandboxed permission context as the job start; rerunning from the unsandboxed context resumes waiting or reporting normal status.
|
|
330
337
|
- **Empty response, "status SUCCESS"** — a restricted run may report success even when its tool calls were denied. Check whether the profile came from a flag, repository policy or an earlier conversation. Use `setup` to configure allowed commands, or pass `--unrestricted` explicitly if authorized; dropping `--restricted` alone does not override an inherited profile. Some native tools remain unavailable in restricted headless runs even with allow-rules. An empty response from unrestricted mode or tool-free ask needs a separate diagnosis; retain the diagnostics and report it.
|
|
@@ -471,7 +478,7 @@ To get upstream back later, run `agy-crew uninstall --all` first.
|
|
|
471
478
|
|
|
472
479
|
## Upgrading
|
|
473
480
|
|
|
474
|
-
Claude Code and Codex cache the plugin under a per-**version** directory (e.g. `cache/agy-crew/agy-crew/0.9.
|
|
481
|
+
Claude Code and Codex cache the plugin under a per-**version** directory (e.g. `cache/agy-crew/agy-crew/0.9.4`) and key "is it current?" on that version string, not on the commit. Bump their manifests and `package.json` together when preparing a release. Pi's Git source instead follows the configured ref; local sources read the checkout directly.
|
|
475
482
|
|
|
476
483
|
- **Terminal command (`agy-crew install`)** — for an npm install, run `npm install -g agy-crew@latest`, then `agy-crew install` to render a fresh bundle and re-link; restart the harness. For a checkout, `git pull` is live at once for the source, and `agy-crew install` refreshes the bundle. See [Install command](#install-command).
|
|
477
484
|
- **Claude Code** — `claude plugin marketplace update agy-crew` refreshes the marketplace clone, then `claude plugin update agy-crew@agy-crew` re-copies it into the cache. `install` is **not** the upgrade command: on an already-installed plugin it answers "already installed" and does nothing, whatever the version. And `update` only moves if the version string changed — on an unchanged version it answers "already at the latest version" and leaves the old commit in place. Force the current commit in with `claude plugin uninstall agy-crew@agy-crew && claude plugin install agy-crew@agy-crew`. Restart Claude Code afterwards either way — skills are registered at session start.
|
package/docs/REFERENCE.zh-CN.md
CHANGED
|
@@ -286,7 +286,7 @@ review --prompt "Review the patch at /tmp/change.patch"
|
|
|
286
286
|
**`auto` 的选择规则。** 候选账号必须已启用、有令牌文件、没有认证错误、在该模型家族上没有生效的冷却,并且缓存的 Cloud Code 快照没有显示该家族所属配额池剩余为 0 且重置时间尚未到来。候选账号依次按以下条件排序:
|
|
287
287
|
|
|
288
288
|
1. 当前租约最少(空闲账号优先于忙碌账号);
|
|
289
|
-
2.
|
|
289
|
+
2. 已知的剩余配额比例最高(取短窗口和每周窗口中较小的一个,按 5% 一档比较,额度相近的账号按最久未使用轮换;未知的排在所有已知且大于 0 的值之后);
|
|
290
290
|
3. `last_used_at` 最早;
|
|
291
291
|
4. 添加顺序。
|
|
292
292
|
|
|
@@ -308,6 +308,12 @@ review --prompt "Review the patch at /tmp/change.patch"
|
|
|
308
308
|
- **已有会话 ID,且会话绑定在某个池账号上**:尝试凭据借用(见下文)。找到出借方时,任务带着这段提示继续:"Your previous turn stopped because the account hit a quota limit. Continue the same task from where you stopped; do not repeat finished steps."
|
|
309
309
|
- **其他情况任务失败**,原因为 `quota_exhausted`。后台任务在已有会话 ID 时状态为 `attention`,没有时为 `error`;结果中包含 `retry_at` 和恢复说明(`continue --job <id>`,在 `<retry_at>` 之后;其他池账号空闲时会自动借用凭据)。前台任务以相同信息退出:有会话 ID 时退出码为 5,否则为 1。是否继续由调用方决定,companion 不会循环重试。
|
|
310
310
|
|
|
311
|
+
**上游 503(`upstream_unavailable`)。** agy 以 `UNAVAILABLE (code 503)` 或 `No capacity available for model ...` 放弃时(它自己已经重试约八次,大约两分钟),companion 会为该账号和模型家族记录一条 **3 分钟**的短冷却,原因为 `upstream_unavailable`,这样无论是本次运行还是之后单独的运行,下一次选择都会换一个账号。503 冷却不会缩短仍在生效的额度冷却。最多额外尝试 2 次:
|
|
312
|
+
|
|
313
|
+
- **还没有会话 ID,或者任务从新会话开始且工作区没有变化**(`ask` 恒为如此,其他模式要求 `git status` 没有新增内容),并且账号由 `auto` 选出:用原始提示词在另一个账号上开一个新会话重跑。这是重写任务,不是继续任务。
|
|
314
|
+
- **工作区已有改动,且会话绑定在某个池账号上**:与额度失败相同,走凭据借用继续。
|
|
315
|
+
- **其他情况任务失败**,原因为 `upstream_unavailable`,已有会话 ID 时退出码为 5(后台任务为 `attention`),否则为 1。恢复说明会告诉调用方:agy 还什么都没做时重新开始任务(`restart <job>`),只有工作区已经有改动时才用 `continue`。显式指定的 `--account` 不会被换到其他账号。
|
|
316
|
+
|
|
311
317
|
池账号上的**认证失败**(`401`、`403` 或 `unauthenticated`,并带有 auth、credential、login 或 token 字样)会记录该账号的认证错误。之后选择时会跳过它,`list` 显示 `needs login`,运行 `accounts login <name>` 后恢复。认证失败从不重试。
|
|
312
318
|
|
|
313
319
|
### 凭据借用
|
|
@@ -364,6 +370,7 @@ review --prompt "Review the patch at /tmp/change.patch"
|
|
|
364
370
|
| 情况 | 原因与输出 | 退出码 |
|
|
365
371
|
| --- | --- | --- |
|
|
366
372
|
| `auto` 没有候选账号,或运行中遇到配额错误且没有可借用的出借方 | `quota_exhausted`,附 `retry_at`(没有候选时按账号分别列出) | 已有会话 ID 时为 5,否则为 1;后台任务相应地以 `attention` 或 `error` 结束 |
|
|
373
|
+
| 模型返回 503 / 没有容量,且没有账号或出借方可以继续 | `upstream_unavailable`,附 `retry_at`(最后一次 503 之后 3 分钟) | 已有会话 ID 时为 5,否则为 1;后台任务相应地以 `attention` 或 `error` 结束 |
|
|
367
374
|
| 池账号认证失败 | 账号标记为 `needs login`,不重试 | 按普通错误处理 |
|
|
368
375
|
| 显式指定的 `--account <name>` 不存在、已停用、处于冷却或需要登录 | 启动前报错并说明原因 | 1 |
|
|
369
376
|
| `--account` 与已绑定的会话冲突 | `conversation <id> belongs to account <x>` | 1 |
|
|
@@ -400,7 +407,7 @@ review --prompt "Review the patch at /tmp/change.patch"
|
|
|
400
407
|
|
|
401
408
|
### 池账号配额用尽或需要登录
|
|
402
409
|
|
|
403
|
-
池账号遇到配额错误时,companion 会记录冷却;如果还有其他账号,或有空闲账号可以出借凭据,任务会在运行内部恢复。否则任务以 `quota_exhausted` 结束,并附上 `retry_at`。认证失败不会重试:请在自己的终端里运行 `agy-crew accounts login <name>`。详见[账号池](#账号池)。
|
|
410
|
+
池账号遇到配额错误时,companion 会记录冷却;如果还有其他账号,或有空闲账号可以出借凭据,任务会在运行内部恢复。否则任务以 `quota_exhausted` 结束,并附上 `retry_at`。遇到 503(`upstream_unavailable`)时账号冷却 3 分钟,没有改动过工作区的任务会换账号重跑;如果任务仍以 `upstream_unavailable` 结束且 agy 还什么都没做,请用 `restart <job>` 或改写提示词重新开始,不要 `continue`。认证失败不会重试:请在自己的终端里运行 `agy-crew accounts login <name>`。详见[账号池](#账号池)。
|
|
404
411
|
|
|
405
412
|
### 参数和任务被放进了同一个字符串
|
|
406
413
|
|
|
@@ -562,7 +569,7 @@ agy-crew 与上游 [agy-staff](https://github.com/keli-wen/agy-staff) 不能安
|
|
|
562
569
|
|
|
563
570
|
## 升级
|
|
564
571
|
|
|
565
|
-
Claude Code 和 Codex 按版本号缓存插件,例如 `cache/agy-crew/agy-crew/0.9.
|
|
572
|
+
Claude Code 和 Codex 按版本号缓存插件,例如 `cache/agy-crew/agy-crew/0.9.4`。缓存是否需要更新取决于版本号,而不是仓库的最新提交。因此,准备发布时需要同步更新两个插件 manifest 和 `package.json` 中的版本。
|
|
566
573
|
|
|
567
574
|
### 终端命令(`agy-crew install`)
|
|
568
575
|
|
|
@@ -79,6 +79,7 @@ Include relevant decisions made in the host conversation, what changed, and what
|
|
|
79
79
|
With a pool configured, the companion handles quota inside the run: it records a cooldown for that account and model family, and either fails over to another account (when no conversation existed yet) or borrows an idle account's credential to continue the same conversation. Nothing for you to do when the job still finishes `done`.
|
|
80
80
|
|
|
81
81
|
- **`quota_exhausted`** (status `attention`, exit 5, when a conversation exists; otherwise `error`) means no account could continue. The result and `observe` include `retry_at`. Report it, keep the job id, and offer `continue --job <id> --prompt "..."` after `retry_at`, or sooner once `accounts list` shows another account idle (a credential loan is then used automatically). Ask before continuing, as with a timeout. Never loop retries and never clear a cooldown yourself. Without a conversation, a new run with `--account auto`, `--account live` or another model family is the alternative; propose it to the user.
|
|
82
|
+
- **`upstream_unavailable`** (agy's `UNAVAILABLE (code 503): No capacity available for model ...`; status `attention` when a conversation exists, otherwise `error`) means the model itself had no capacity. agy has already retried internally, and the companion has already parked that account briefly and tried another one, so do not loop. **If agy had not done anything yet** (clean `git status`, no partial result in the job output), **rewrite the task and start it as a new run (`restart <job id>`, or a fresh dispatch with the prompt rewritten). Do not use `continue`**: a continuation stays bound to the original account and conversation, and the 503 gave it nothing to continue from. Use `continue --job <id>` only when agy had already changed the workspace; inspect `git status` and `git diff` first, and say what was left unfinished in the prompt. Offer a retry after `retry_at` (a few minutes), or another model family when the outage persists; ask the user before retrying a second time.
|
|
82
83
|
- **`needs login`** (an authentication failure on a pool account) is not retried. Tell the user to run `agy-crew accounts login <name>` in their own terminal; it opens a browser, so you must not run it.
|
|
83
84
|
- An unknown, disabled or cooling explicit `--account <name>` fails before launch with the reason; use `auto` or another account.
|
|
84
85
|
|
|
@@ -26,6 +26,14 @@ When agy reports an error but response text came back, the companion delivers th
|
|
|
26
26
|
|
|
27
27
|
`reason=quota_exhausted` with a `retry_at` means the account (and any idle account that could lend a credential) hit its quota for that model family. Failover and credential loans have already been tried inside the run, so do not retry in a loop. Offer `continue --job <id>` after `retry_at`, or when `accounts list` shows an idle account; with no conversation, offer a new run with another `--account` or model family. `accounts quota --refresh` shows current numbers when the user wants them.
|
|
28
28
|
|
|
29
|
+
## upstream_unavailable (503, no capacity)
|
|
30
|
+
|
|
31
|
+
`reason=upstream_unavailable` means agy got `UNAVAILABLE (code 503): No capacity available for model ...` from the server, after its own internal retries (roughly two minutes each time). The companion has parked that account for the model family for a few minutes and tried other accounts where it could, so another account or a loan is not worth trying by hand; do not retry in a loop.
|
|
32
|
+
|
|
33
|
+
- **agy had not done anything yet** (clean `git status`, no partial result): rewrite the task and run it again as a new run, with `restart <job id>` or a new dispatch with an updated prompt. **Do not use `continue`.** A continuation is bound to the original account and conversation, which holds nothing useful, and it only repeats the failing request.
|
|
34
|
+
- **agy had already changed the workspace**: inspect `git status` and `git diff`, then `continue --job <id>` with a prompt that says what is done and what is left. Ask the user first, as with a timeout.
|
|
35
|
+
- The outage may be model-wide. After `retry_at` try once more, or offer another model family; ask before a second retry.
|
|
36
|
+
|
|
29
37
|
An authentication failure on a pool account marks it `needs login` and is never retried. The user must run `agy-crew accounts login <name>` in their own terminal (browser sign-in); do not run it yourself. Live authentication failures still mean running `agy` interactively once.
|
|
30
38
|
|
|
31
39
|
## Retry rules
|
package/package.json
CHANGED
|
@@ -79,6 +79,7 @@ Include relevant decisions made in the host conversation, what changed, and what
|
|
|
79
79
|
With a pool configured, the companion handles quota inside the run: it records a cooldown for that account and model family, and either fails over to another account (when no conversation existed yet) or borrows an idle account's credential to continue the same conversation. Nothing for you to do when the job still finishes `done`.
|
|
80
80
|
|
|
81
81
|
- **`quota_exhausted`** (status `attention`, exit 5, when a conversation exists; otherwise `error`) means no account could continue. The result and `observe` include `retry_at`. Report it, keep the job id, and offer `continue --job <id> --prompt "..."` after `retry_at`, or sooner once `accounts list` shows another account idle (a credential loan is then used automatically). Ask before continuing, as with a timeout. Never loop retries and never clear a cooldown yourself. Without a conversation, a new run with `--account auto`, `--account live` or another model family is the alternative; propose it to the user.
|
|
82
|
+
- **`upstream_unavailable`** (agy's `UNAVAILABLE (code 503): No capacity available for model ...`; status `attention` when a conversation exists, otherwise `error`) means the model itself had no capacity. agy has already retried internally, and the companion has already parked that account briefly and tried another one, so do not loop. **If agy had not done anything yet** (clean `git status`, no partial result in the job output), **rewrite the task and start it as a new run (`restart <job id>`, or a fresh dispatch with the prompt rewritten). Do not use `continue`**: a continuation stays bound to the original account and conversation, and the 503 gave it nothing to continue from. Use `continue --job <id>` only when agy had already changed the workspace; inspect `git status` and `git diff` first, and say what was left unfinished in the prompt. Offer a retry after `retry_at` (a few minutes), or another model family when the outage persists; ask the user before retrying a second time.
|
|
82
83
|
- **`needs login`** (an authentication failure on a pool account) is not retried. Tell the user to run `agy-crew accounts login <name>` in their own terminal; it opens a browser, so you must not run it.
|
|
83
84
|
- An unknown, disabled or cooling explicit `--account <name>` fails before launch with the reason; use `auto` or another account.
|
|
84
85
|
|
|
@@ -26,6 +26,14 @@ When agy reports an error but response text came back, the companion delivers th
|
|
|
26
26
|
|
|
27
27
|
`reason=quota_exhausted` with a `retry_at` means the account (and any idle account that could lend a credential) hit its quota for that model family. Failover and credential loans have already been tried inside the run, so do not retry in a loop. Offer `continue --job <id>` after `retry_at`, or when `accounts list` shows an idle account; with no conversation, offer a new run with another `--account` or model family. `accounts quota --refresh` shows current numbers when the user wants them.
|
|
28
28
|
|
|
29
|
+
## upstream_unavailable (503, no capacity)
|
|
30
|
+
|
|
31
|
+
`reason=upstream_unavailable` means agy got `UNAVAILABLE (code 503): No capacity available for model ...` from the server, after its own internal retries (roughly two minutes each time). The companion has parked that account for the model family for a few minutes and tried other accounts where it could, so another account or a loan is not worth trying by hand; do not retry in a loop.
|
|
32
|
+
|
|
33
|
+
- **agy had not done anything yet** (clean `git status`, no partial result): rewrite the task and run it again as a new run, with `restart <job id>` or a new dispatch with an updated prompt. **Do not use `continue`.** A continuation is bound to the original account and conversation, which holds nothing useful, and it only repeats the failing request.
|
|
34
|
+
- **agy had already changed the workspace**: inspect `git status` and `git diff`, then `continue --job <id>` with a prompt that says what is done and what is left. Ask the user first, as with a timeout.
|
|
35
|
+
- The outage may be model-wide. After `retry_at` try once more, or offer another model family; ask before a second retry.
|
|
36
|
+
|
|
29
37
|
An authentication failure on a pool account marks it `needs login` and is never retried. The user must run `agy-crew accounts login <name>` in their own terminal (browser sign-in); do not run it yourself. Live authentication failures still mean running `agy` interactively once.
|
|
30
38
|
|
|
31
39
|
## Retry rules
|
package/skills/jobs/SKILL.md
CHANGED
|
@@ -79,6 +79,7 @@ Include relevant decisions made in the host conversation, what changed, and what
|
|
|
79
79
|
With a pool configured, the companion handles quota inside the run: it records a cooldown for that account and model family, and either fails over to another account (when no conversation existed yet) or borrows an idle account's credential to continue the same conversation. Nothing for you to do when the job still finishes `done`.
|
|
80
80
|
|
|
81
81
|
- **`quota_exhausted`** (status `attention`, exit 5, when a conversation exists; otherwise `error`) means no account could continue. The result and `observe` include `retry_at`. Report it, keep the job id, and offer `continue --job <id> --prompt "..."` after `retry_at`, or sooner once `accounts list` shows another account idle (a credential loan is then used automatically). Ask before continuing, as with a timeout. Never loop retries and never clear a cooldown yourself. Without a conversation, a new run with `--account auto`, `--account live` or another model family is the alternative; propose it to the user.
|
|
82
|
+
- **`upstream_unavailable`** (agy's `UNAVAILABLE (code 503): No capacity available for model ...`; status `attention` when a conversation exists, otherwise `error`) means the model itself had no capacity. agy has already retried internally, and the companion has already parked that account briefly and tried another one, so do not loop. **If agy had not done anything yet** (clean `git status`, no partial result in the job output), **rewrite the task and start it as a new run (`restart <job id>`, or a fresh dispatch with the prompt rewritten). Do not use `continue`**: a continuation stays bound to the original account and conversation, and the 503 gave it nothing to continue from. Use `continue --job <id>` only when agy had already changed the workspace; inspect `git status` and `git diff` first, and say what was left unfinished in the prompt. Offer a retry after `retry_at` (a few minutes), or another model family when the outage persists; ask the user before retrying a second time.
|
|
82
83
|
- **`needs login`** (an authentication failure on a pool account) is not retried. Tell the user to run `agy-crew accounts login <name>` in their own terminal; it opens a browser, so you must not run it.
|
|
83
84
|
- An unknown, disabled or cooling explicit `--account <name>` fails before launch with the reason; use `auto` or another account.
|
|
84
85
|
|
|
@@ -24,6 +24,14 @@ When agy reports an error but response text came back, the companion delivers th
|
|
|
24
24
|
|
|
25
25
|
`reason=quota_exhausted` with a `retry_at` means the account (and any idle account that could lend a credential) hit its quota for that model family. Failover and credential loans have already been tried inside the run, so do not retry in a loop. Offer `continue --job <id>` after `retry_at`, or when `accounts list` shows an idle account; with no conversation, offer a new run with another `--account` or model family. `accounts quota --refresh` shows current numbers when the user wants them.
|
|
26
26
|
|
|
27
|
+
## upstream_unavailable (503, no capacity)
|
|
28
|
+
|
|
29
|
+
`reason=upstream_unavailable` means agy got `UNAVAILABLE (code 503): No capacity available for model ...` from the server, after its own internal retries (roughly two minutes each time). The companion has parked that account for the model family for a few minutes and tried other accounts where it could, so another account or a loan is not worth trying by hand; do not retry in a loop.
|
|
30
|
+
|
|
31
|
+
- **agy had not done anything yet** (clean `git status`, no partial result): rewrite the task and run it again as a new run, with `restart <job id>` or a new dispatch with an updated prompt. **Do not use `continue`.** A continuation is bound to the original account and conversation, which holds nothing useful, and it only repeats the failing request.
|
|
32
|
+
- **agy had already changed the workspace**: inspect `git status` and `git diff`, then `continue --job <id>` with a prompt that says what is done and what is left. Ask the user first, as with a timeout.
|
|
33
|
+
- The outage may be model-wide. After `retry_at` try once more, or offer another model family; ask before a second retry.
|
|
34
|
+
|
|
27
35
|
An authentication failure on a pool account marks it `needs login` and is never retried. The user must run `agy-crew accounts login <name>` in their own terminal (browser sign-in); do not run it yourself. Live authentication failures still mean running `agy` interactively once.
|
|
28
36
|
|
|
29
37
|
## Retry rules
|