claude-multiacc 2.0.21 → 2.0.23
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -3
- package/bin/claude +82 -9
- package/bin/claude-accounts +198 -33
- package/bin/codex +82 -5
- package/bin/codex-accounts +202 -21
- package/docs/ACCOUNT_OPERATIONS.md +26 -4
- package/lib/__pycache__/audit.cpython-312.pyc +0 -0
- package/lib/__pycache__/keychain.cpython-312.pyc +0 -0
- package/lib/__pycache__/selector_policy.cpython-312.pyc +0 -0
- package/lib/__pycache__/selector_primitives.cpython-312.pyc +0 -0
- package/lib/codex_reset.py +6 -16
- package/lib/codex_reset_windows.py +36 -0
- package/lib/report.py +38 -2
- package/package.json +1 -1
- package/tests/__pycache__/test_codex_reset.cpython-312.pyc +0 -0
- package/tests/__pycache__/test_codex_reset_windows.cpython-312.pyc +0 -0
- package/tests/run-tests.sh +1187 -3
- package/tests/test_codex_reset_windows.py +43 -0
package/README.md
CHANGED
|
@@ -16,7 +16,7 @@ completely independent — separate manifests, credentials, telemetry, and logs
|
|
|
16
16
|
either provider can be used, re-authenticated, or emptied without touching the other.
|
|
17
17
|
|
|
18
18
|
Tested on: macOS (bash 3.2, zsh, Claude Code 2.1.207+, Codex CLI 0.147) and Ubuntu 24.04
|
|
19
|
-
(bash 5.2). The compatibility suite currently covers
|
|
19
|
+
(bash 5.2). The compatibility suite currently covers 918 sandboxed cases with no
|
|
20
20
|
network/quota use; the unified selector has its own adversarial contract suite.
|
|
21
21
|
|
|
22
22
|
## Unified selector for app-robot
|
|
@@ -177,6 +177,22 @@ buckets keep being tracked and the ≥90% rule keeps working. Unparseable entrie
|
|
|
177
177
|
skipped, an all-legacy payload falls back to `five_hour`/`seven_day`, and a payload the
|
|
178
178
|
code cannot read at all degrades that one account (fail open), never the run.
|
|
179
179
|
|
|
180
|
+
**A 0% bucket with no reset window is no data, not an empty account.** Real usage always
|
|
181
|
+
carries the window it resets in, so `percent: 0` with a null `resets_at` is the endpoint
|
|
182
|
+
declining to answer — and taking it at face value put two weekly-exhausted accounts at the
|
|
183
|
+
top of the ranking for 31 of ~60 picks on 2026-09-04. Only *informative* buckets (percent
|
|
184
|
+
above 0, or a parseable reset time) are aggregated, **per signal**: `weekly_percent` is the
|
|
185
|
+
peak over informative *weekly* buckets, `session_percent` over informative *session* ones,
|
|
186
|
+
and each is simply omitted when its own group has nothing informative in it — a weekly
|
|
187
|
+
figure is never borrowed from the session bucket. An account with no informative bucket at
|
|
188
|
+
all gets a `limits.json` with none of `max_percent`/`weekly_percent`/`session_percent`,
|
|
189
|
+
marked `"no_data": true`. Either way the account ranks **unknown** — selection needs BOTH
|
|
190
|
+
readings, so a half-filled document is as unknown as an empty one: never the band leader,
|
|
191
|
+
never past the session gate, never able to clear a marker, and a pool made only of them
|
|
192
|
+
logs `ranking=BLIND` instead of quietly tying every account. Accounts that really are idle
|
|
193
|
+
are untouched: 0% with a real reset window is informative and still ranks as empty, and one
|
|
194
|
+
blank per-model bucket beside real ones leaves the real ones in charge.
|
|
195
|
+
|
|
180
196
|
Telemetry failures never block work: no fresh data ⇒ account treated as available. The
|
|
181
197
|
endpoint rate-limits per account, so the refresher skips accounts fetched in the last 4 min
|
|
182
198
|
and backs off on **every** non-2xx, not just a 429 — honoring `Retry-After`, and parking
|
|
@@ -274,8 +290,20 @@ wrongly parked account is worse than a missed limit:
|
|
|
274
290
|
|
|
275
291
|
The resulting marker is tagged `reason=client-rate-limit` and outlives a clean telemetry
|
|
276
292
|
pass while its window is still open — first-hand evidence beats a usage payload that
|
|
277
|
-
disagrees.
|
|
278
|
-
|
|
293
|
+
disagrees. A **weekly** rejection is absolute: nothing — neither a shim invocation nor the
|
|
294
|
+
scheduled `limits` pass — retires it before its reset epoch, since a weekly bucket cannot
|
|
295
|
+
fall from a server-proven 100% back under the threshold inside that window. Claude names
|
|
296
|
+
that bucket from the API's own type (`bucket=client:seven_day`); codex has no such name in
|
|
297
|
+
its rollout files, only `primary`/`secondary` keys whose meaning moves, so it labels the
|
|
298
|
+
marker from the window itself — `window_minutes >= 1440` ⇒ `client:7d`, a shorter window ⇒
|
|
299
|
+
`client:5h`, and a window with no `window_minutes` at all stays `client:<key>` and remains
|
|
300
|
+
clearable (fail open, bounded by its own reset). A five-hour marker can
|
|
301
|
+
still be retired by telemetry fetched well after it
|
|
302
|
+
(`CLAUDE_MULTIACC_CLIENT_LIMIT_CONFIRM_DELAY`, 300s), which is what keeps a fresh account
|
|
303
|
+
from being stranded by an old session limit — before 2026-09-04 that carve-out also let
|
|
304
|
+
fake-zero telemetry retire weekly markers, re-admitting an exhausted account six times a
|
|
305
|
+
day. Cost is bounded: the three newest sessions per account, a tail read rather than a whole
|
|
306
|
+
transcript, and a clean result memoized for 20s
|
|
279
307
|
(`CLAUDE_MULTIACC_CLIENT_SCAN_TTL`), which on a three-account pool of large live
|
|
280
308
|
transcripts is ~2 ms per run amortized.
|
|
281
309
|
|
|
@@ -350,6 +378,8 @@ Codex-specific notes:
|
|
|
350
378
|
and writes a per-account/window idempotency key before redemption so a lost response
|
|
351
379
|
or simultaneous fleet poll cannot spend a second reset. Set
|
|
352
380
|
`CODEX_MULTIACC_AUTO_RESET=0` only for emergency rollback/testing.
|
|
381
|
+
A finished-limit verdict takes precedence over a conflicting percentage. The
|
|
382
|
+
redemption key excludes unused model windows whose reset clocks move between reads.
|
|
353
383
|
- **API-key logins are rejected** — ChatGPT subscription accounts only, matching the
|
|
354
384
|
addon's no-API-keys rule.
|
|
355
385
|
|
package/bin/claude
CHANGED
|
@@ -276,7 +276,7 @@ scoped_over_tok() { # $1 = acct dir
|
|
|
276
276
|
CLIENT_RECOVERY_FETCHED=0
|
|
277
277
|
CLIENT_RECOVERY_PEAK=100
|
|
278
278
|
client_marker_recovered() { # $1 account dir, $2 marker
|
|
279
|
-
local fetched peak delay marked mk cutoff ck newer=0
|
|
279
|
+
local fetched peak delay marked mk cutoff ck newer=0 bucket
|
|
280
280
|
# A client rejection is stronger than an immediately-following usage read: that
|
|
281
281
|
# endpoint can lag or serve a cached bucket. It is not stronger forever. Once a
|
|
282
282
|
# successful, below-threshold reading was fetched at least five minutes after the
|
|
@@ -284,7 +284,28 @@ client_marker_recovered() { # $1 account dir, $2 marker
|
|
|
284
284
|
# old/shared transcript was attributed incorrectly). Keeping such a marker until its
|
|
285
285
|
# claimed reset stranded five 0%-usage accounts in the live pool on 2026-09-03.
|
|
286
286
|
if grep -q 'reason=client-rate-limit' "$2" 2>/dev/null; then
|
|
287
|
+
# ...but a WEEKLY rejection is stronger than any telemetry until its own reset.
|
|
288
|
+
# Line 2 carries bucket=client:<type> (five_hour, seven_day, seven_day_opus, …).
|
|
289
|
+
# A seven-day window only refills on its multi-day reset, so it cannot have fallen
|
|
290
|
+
# from the server-proven 100% that wrote this marker to under the threshold while
|
|
291
|
+
# the window is still open: a reading that says otherwise is wrong by construction.
|
|
292
|
+
# 2026-09-04: acct-13/acct-14 were served fake-zero telemetry (every bucket
|
|
293
|
+
# percent 0, resets_at null), which read as 0% here and deleted their truthful
|
|
294
|
+
# client:seven_day markers ~6x/day each; an interactive session was handed both
|
|
295
|
+
# accounts in a row and was rejected by each with "You've hit your weekly limit".
|
|
296
|
+
# five_hour/session markers keep clearing exactly as before — that window self-heals
|
|
297
|
+
# within hours, which is why #22 (2026-09-03) had to clear it.
|
|
298
|
+
bucket="$(sed -n 2p "$2" 2>/dev/null)"
|
|
299
|
+
case "$bucket" in
|
|
300
|
+
*bucket=*) bucket="${bucket#*bucket=}"; bucket="${bucket%% *}" ;;
|
|
301
|
+
*) bucket="" ;;
|
|
302
|
+
esac
|
|
303
|
+
case "$(sel_lc "$bucket")" in *seven_day*|*7d*|*weekly*) return 1 ;; esac
|
|
287
304
|
fetched="$(telem_fetched_at "$1" 2>/dev/null || echo 0)"
|
|
305
|
+
# Defaulting peak to 100 when the reading is missing/stale is the second half of the
|
|
306
|
+
# same rule: an UNINFORMATIVE reading must never clear a marker. It is what keeps a
|
|
307
|
+
# no_data account (2026-09-04: all-zero buckets with no reset windows write no
|
|
308
|
+
# max_percent at all) from unparking itself.
|
|
288
309
|
peak="$(cutoff_field "$1" max_percent 2>/dev/null || echo 100)"
|
|
289
310
|
delay="${CLAUDE_MULTIACC_CLIENT_LIMIT_CONFIRM_DELAY:-300}"
|
|
290
311
|
num_ok "$delay" || delay=300
|
|
@@ -411,12 +432,39 @@ stale_weekly() { # $1 = acct dir
|
|
|
411
432
|
limits_field "$1" weekly_percent
|
|
412
433
|
}
|
|
413
434
|
|
|
414
|
-
# TRUE when this account contributes nothing to ranking: no in-window telemetry
|
|
415
|
-
#
|
|
416
|
-
#
|
|
417
|
-
# file exists to prevent.
|
|
435
|
+
# TRUE when this account contributes nothing to ranking: no in-window telemetry, or an
|
|
436
|
+
# in-window document that is not a COMPLETE reading. When every candidate is blind,
|
|
437
|
+
# no candidate has a known reading, pick_best sees one enormous tie, and selection
|
|
438
|
+
# quietly becomes uniform random — the failure this whole file exists to prevent.
|
|
439
|
+
# A fresh timestamp is not the same thing as a usable reading: since 2026-09-04 the
|
|
440
|
+
# writer refuses to invent percentages from all-zero buckets with no reset window and
|
|
441
|
+
# writes `no_data` instead, so such a document is current, well-formed, and ranks
|
|
442
|
+
# nothing. Judging blindness on fetched_at alone called a pool of those "fresh" while
|
|
443
|
+
# every account read as unknown and selection was in fact a coin flip.
|
|
444
|
+
# BOTH fields are required, exactly as pick_best's "known" rule requires them (and
|
|
445
|
+
# pool-selection.v2's quota_known): the same 2026-09-04 writer change also emits
|
|
446
|
+
# one-signal documents per signal (weekly-only when no session bucket is informative,
|
|
447
|
+
# and the converse), and an EITHER test called a pool of those sighted while pick_best
|
|
448
|
+
# read every one of them as unknown and tied the whole pool — the coin flip again, this
|
|
449
|
+
# time with no ranking=BLIND line and a status saying "fresh". The degraded path takes
|
|
450
|
+
# one more guard for the same reason (assess_telemetry): a reading counts toward
|
|
451
|
+
# DEGRADED only when it is genuinely OUT of the ranking window — degraded exists for an
|
|
452
|
+
# outage, and a FRESH weekly-only document must not smuggle the very reading the
|
|
453
|
+
# quota_known rule just refused back in as "old truth" (codex review, 2026-09-04).
|
|
454
|
+
# A no_data document (no weekly_percent, no horizon) stays un-degradable either way.
|
|
455
|
+
# Asked through limits_field, the SAME parser rank_weekly_of/rank_session_of use, so
|
|
456
|
+
# this can never disagree with them about what "known" means. Both calls only ever run
|
|
457
|
+
# for a blind-looking account (assess_telemetry stops at the first ranking one).
|
|
458
|
+
# TELEM_FRESH_UNUSABLE is set as a side effect for the operator warning: the pool is
|
|
459
|
+
# then blind with CURRENT data, which is a different fault (and a different fix) from
|
|
460
|
+
# telemetry that merely went stale. Set here because the window test is already paid
|
|
461
|
+
# for; this file is deliberately fork-frugal (see sessions_owned).
|
|
462
|
+
TELEM_FRESH_UNUSABLE=0
|
|
418
463
|
telem_blind() { # $1 = acct dir
|
|
419
|
-
|
|
464
|
+
within_window "$1" "$STALE_AFTER" || return 0
|
|
465
|
+
limits_field "$1" weekly_percent >/dev/null 2>&1 || { TELEM_FRESH_UNUSABLE=1; return 0; }
|
|
466
|
+
limits_field "$1" session_percent >/dev/null 2>&1 || { TELEM_FRESH_UNUSABLE=1; return 0; }
|
|
467
|
+
return 1
|
|
420
468
|
}
|
|
421
469
|
|
|
422
470
|
# Human age for the warning line: seconds -> "3h" / "11d". Never fails.
|
|
@@ -1225,8 +1273,10 @@ pick_best() { # args: candidate dirs
|
|
|
1225
1273
|
# otherwise be the sole gate-clearer and win the all-gated tie over an account whose
|
|
1226
1274
|
# truthful weekly reading merely failed the gate. The one exception is a DEGRADED pool
|
|
1227
1275
|
# (SEL_DEGRADED=1): nothing is fresh anywhere, the gate has necessarily stepped aside,
|
|
1228
|
-
# and a still-valid stale weekly reading is the only truth there is. (
|
|
1229
|
-
#
|
|
1276
|
+
# and a still-valid stale weekly reading is the only truth there is. (The writers DO
|
|
1277
|
+
# emit one-signal documents — per-signal informative aggregation, 2026-09-04 — and
|
|
1278
|
+
# such a document is exactly as unknown here as an empty one; parity with
|
|
1279
|
+
# lib/selector_policy.py's quota_known.)
|
|
1230
1280
|
if [ "$k" = 1 ] && { [ "$sk" = 1 ] || [ "$SEL_DEGRADED" = 1 ]; }; then
|
|
1231
1281
|
weekly+=("$w"); known+=(1)
|
|
1232
1282
|
else
|
|
@@ -1294,6 +1344,9 @@ degraded=0 # 1 = blind, but every candidate had a stale reading still wor
|
|
|
1294
1344
|
assess_telemetry() { # args: the dirs actually being chosen between
|
|
1295
1345
|
local d f n=0 stale_ok=0
|
|
1296
1346
|
blind=1; blind_age=0; degraded=0
|
|
1347
|
+
# Re-armed per assessment: the fallback path assesses a second candidate set, and a
|
|
1348
|
+
# verdict from the first one must not colour the warning printed for the second.
|
|
1349
|
+
TELEM_FRESH_UNUSABLE=0
|
|
1297
1350
|
for d in "$@"; do
|
|
1298
1351
|
n=$((n + 1))
|
|
1299
1352
|
if ! telem_blind "$d"; then blind=0; return 0; fi
|
|
@@ -1302,7 +1355,15 @@ assess_telemetry() { # args: the dirs actually being chosen between
|
|
|
1302
1355
|
# never fetched at all must not make the pool look older than it is.
|
|
1303
1356
|
[ "$f" -gt 0 ] && { [ "$blind_age" -eq 0 ] || [ $((now - f)) -lt "$blind_age" ]; } \
|
|
1304
1357
|
&& blind_age=$((now - f))
|
|
1305
|
-
|
|
1358
|
+
# Only a reading that is genuinely OUT of the ranking window may count toward
|
|
1359
|
+
# DEGRADED: degraded exists for an outage ("nothing fresh anywhere, rank on old
|
|
1360
|
+
# truths"), not for a fresh document that simply lacks a signal. A fresh
|
|
1361
|
+
# weekly-only file already read as blind above (one-signal is unknown, 2026-09-04);
|
|
1362
|
+
# letting its still-in-window weekly enable degraded would rank the pool on the
|
|
1363
|
+
# very reading the two-cut rule just refused (codex review, 2026-09-04).
|
|
1364
|
+
if ! within_window "$d" "$STALE_AFTER"; then
|
|
1365
|
+
stale_weekly "$d" >/dev/null && stale_ok=$((stale_ok + 1))
|
|
1366
|
+
fi
|
|
1306
1367
|
done
|
|
1307
1368
|
# All or nothing. A candidate whose reading has no horizon — a limits.json written
|
|
1308
1369
|
# before this field existed, or one whose week has already turned — stays UNKNOWN even
|
|
@@ -1470,9 +1531,21 @@ if [ "$blind" = 1 ]; then
|
|
|
1470
1531
|
[ -f "$n" ] && last="$(file_mtime "$n")"
|
|
1471
1532
|
if [ $((now - last)) -gt 3600 ]; then
|
|
1472
1533
|
: 2>/dev/null > "$n" || true
|
|
1534
|
+
# Three faults, three fixes, and telling them apart is the whole point of this
|
|
1535
|
+
# line. Telemetry that went STALE is fixed by fetching again (and, if that keeps
|
|
1536
|
+
# failing, by an OAuth login). Telemetry that arrived CURRENT and said nothing
|
|
1537
|
+
# rankable — a 2026-09-04 no_data document, or a one-signal one — is not stale at
|
|
1538
|
+
# all: refetching returns the same emptiness and a re-login fixes nothing, so
|
|
1539
|
+
# advising one sends the operator down the wrong path (E4, 2026-09-04). The state
|
|
1540
|
+
# is already known here: telem_blind sets TELEM_FRESH_UNUSABLE while testing the
|
|
1541
|
+
# window it had to test anyway.
|
|
1542
|
+
# (degraded && FRESH_UNUSABLE cannot happen: a fresh-but-unusable candidate is
|
|
1543
|
+
# not counted by assess_telemetry, and degraded is all-or-nothing.)
|
|
1473
1544
|
if [ "$degraded" = 1 ]; then
|
|
1474
1545
|
printf 'claude-multiacc: usage telemetry is %s old — ranking on the last readings that are still valid, not on current usage. Fix: claude-accounts limits --force, then claude-accounts status\n' \
|
|
1475
1546
|
"$(age_human "$blind_age")" >&2
|
|
1547
|
+
elif [ "$TELEM_FRESH_UNUSABLE" = 1 ]; then
|
|
1548
|
+
printf 'claude-multiacc: usage telemetry is CURRENT but carries no usable reading for any account (the endpoint answered with no percentages, or only half of them) — selection is running blind (random, not by headroom). Refetching and re-logging in will not change it: claude-accounts status\n' >&2
|
|
1476
1549
|
elif [ "$blind_age" -gt 0 ]; then
|
|
1477
1550
|
printf 'claude-multiacc: usage telemetry is %s old for EVERY account and too old to mean anything — selection is running blind (random, not by headroom). Fix: claude-accounts limits --force, then claude-accounts status\n' \
|
|
1478
1551
|
"$(age_human "$blind_age")" >&2
|
package/bin/claude-accounts
CHANGED
|
@@ -379,7 +379,14 @@ for a in doc.get('accounts', []):
|
|
|
379
379
|
# entirely and picks at random, so the reading below is decoration.
|
|
380
380
|
stale = age is None or age > STALE_AFTER
|
|
381
381
|
flag = ' << STALE — NOT USED FOR RANKING' if stale else ''
|
|
382
|
-
|
|
382
|
+
# A no_data document (all-zero buckets with no reset windows, 2026-09-04)
|
|
383
|
+
# carries no percent fields at all. Printing "max None%" would read like a
|
|
384
|
+
# healthy account sitting at zero — the exact misreading that handed 31 of
|
|
385
|
+
# ~60 picks to two exhausted accounts — so name the state instead.
|
|
386
|
+
peak = lim.get('max_percent')
|
|
387
|
+
usable = isinstance(peak, (int, float)) and not isinstance(peak, bool)
|
|
388
|
+
head = f'max {peak}%' if usable else 'NO USABLE TELEMETRY — ranks as unknown'
|
|
389
|
+
print(f" limits : {' '.join(parts) or '(none)'} [{shown}, {head}]{flag}")
|
|
383
390
|
err = lim.get('last_error')
|
|
384
391
|
if isinstance(err, str) and err:
|
|
385
392
|
when = lim.get('last_error_at')
|
|
@@ -428,14 +435,46 @@ if verdict == 'unknown':
|
|
|
428
435
|
print(f"RANKING STATE UNKNOWN: could not compute the pool-wide telemetry verdict "
|
|
429
436
|
f"({verdict_err}). Check the per-account ages above by hand.")
|
|
430
437
|
elif verdict == 'blind':
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
438
|
+
# Two ways to be blind, and they take OPPOSITE advice. STALE: the fetches stopped,
|
|
439
|
+
# so fetch again and, if that keeps failing, log in — the eleven-day 2026-08 outage.
|
|
440
|
+
# CURRENT-BUT-UNUSABLE: the endpoint answered inside the window and said nothing
|
|
441
|
+
# rankable (a no_data document — all-zero buckets with no reset window, 2026-09-04 —
|
|
442
|
+
# or a reading carrying only one of the two percentages). There the credential is
|
|
443
|
+
# working perfectly; telling the operator to re-login sends them after a fault that
|
|
444
|
+
# does not exist, and --force just re-asks for the same emptiness. The candidate
|
|
445
|
+
# rule below is telemetry_state's own, so this text can never name a state the
|
|
446
|
+
# verdict did not come from.
|
|
447
|
+
usable_rows = [a for a in doc_rows if a.get('status') in ('active', 'limited')]
|
|
448
|
+
cands = [a for a in usable_rows if a.get('status') == 'active'] or usable_rows
|
|
449
|
+
current = [a for a in cands if (a.get('usage') or {})
|
|
450
|
+
and not (a.get('usage') or {}).get('stale')]
|
|
451
|
+
nodata = [a['id'] for a in current if (a.get('usage') or {}).get('no_data')]
|
|
452
|
+
if current:
|
|
453
|
+
print("RANKING IS BLIND: usage telemetry is INSIDE the "
|
|
454
|
+
f"{STALE_AFTER}s window but carries no reading the shim can rank on, so "
|
|
455
|
+
"every account scores the same and `claude` picks at RANDOM — including "
|
|
456
|
+
"accounts that are nearly out of weekly headroom.")
|
|
457
|
+
if nodata:
|
|
458
|
+
print(" why : the usage endpoint returned no usable data for "
|
|
459
|
+
f"{', '.join(nodata)} — all-zero buckets with no reset window, which "
|
|
460
|
+
'the writer records as "no_data": true rather than as 0% usage. '
|
|
461
|
+
"Those fetches authenticated; a re-login does NOT fix this.")
|
|
462
|
+
else:
|
|
463
|
+
print(" why : the readings are incomplete — ranking needs BOTH a weekly and "
|
|
464
|
+
"a session percentage (see the 'limits' lines above); an account with "
|
|
465
|
+
"only one of them is unknown to selection, exactly as if it had none.")
|
|
466
|
+
print(" fix : nothing local to repair — the endpoint has to answer with real "
|
|
467
|
+
"buckets again. `claude-accounts limits --force` re-asks; while it keeps "
|
|
468
|
+
"answering this way, selection stays random.")
|
|
469
|
+
else:
|
|
470
|
+
print("RANKING IS BLIND: no account has usage telemetry inside the "
|
|
471
|
+
f"{STALE_AFTER}s window, and the last readings are too old to mean anything, "
|
|
472
|
+
"so every account scores the same and `claude` picks at RANDOM — including "
|
|
473
|
+
"accounts that are nearly out of weekly headroom.")
|
|
474
|
+
print(" why : see the 'telemetry' lines above (a setup token cannot read the usage "
|
|
475
|
+
"endpoint — it has no user:profile scope; only an OAuth login on this machine can)")
|
|
476
|
+
print(" fix : claude-accounts limits --force # then, if it still fails:")
|
|
477
|
+
print(" claude-accounts login <acct-NN> # per account, on THIS machine")
|
|
439
478
|
elif verdict == 'degraded':
|
|
440
479
|
print("RANKING IS DEGRADED: no account has telemetry inside the "
|
|
441
480
|
f"{STALE_AFTER}s window, so `claude` is ranking on the last readings whose "
|
|
@@ -1402,7 +1441,22 @@ limits_distribute() {
|
|
|
1402
1441
|
limits_distribute_now() {
|
|
1403
1442
|
local lock="$ACC_ROOT/tmp/limits-push.lock"
|
|
1404
1443
|
mkdir -p "$ACC_ROOT/tmp" 2>/dev/null || return 0
|
|
1405
|
-
mkdir "$lock" 2>/dev/null
|
|
1444
|
+
if ! mkdir "$lock" 2>/dev/null; then
|
|
1445
|
+
# A DETACHED push that is killed (logout, reboot, pkill) never runs its EXIT trap,
|
|
1446
|
+
# and mkdir can never take a lock dir nobody will remove: one stranded lock
|
|
1447
|
+
# silently stopped ALL telemetry distribution on the live pool from 2026-09-03
|
|
1448
|
+
# 00:29 until it was deleted by hand on 2026-09-04 — 32 hours in which every peer
|
|
1449
|
+
# ranked on whatever limits.json it happened to already have, which is the exact
|
|
1450
|
+
# blindness this push exists to prevent, and nothing anywhere said so. A push is
|
|
1451
|
+
# seconds of rsync under hard timeouts (--timeout=20, ConnectTimeout=10), so a
|
|
1452
|
+
# lock older than ten minutes belongs to a process that is gone: break it and
|
|
1453
|
+
# retake it. If the retake still fails, a live pusher owns it and this pass skips,
|
|
1454
|
+
# exactly as before.
|
|
1455
|
+
[ $(( $(epoch_now) - $(file_mtime "$lock") )) -gt 600 ] || return 0
|
|
1456
|
+
rm -rf "$lock" 2>/dev/null
|
|
1457
|
+
mkdir "$lock" 2>/dev/null || return 0
|
|
1458
|
+
log_to sync.log "stale limits-push lock broken (older than 600s); distributing"
|
|
1459
|
+
fi
|
|
1406
1460
|
trap 'rmdir "$lock" 2>/dev/null || true' EXIT
|
|
1407
1461
|
local server sroot list id d
|
|
1408
1462
|
list="$ACC_ROOT/tmp/limits-push.$$"
|
|
@@ -1544,6 +1598,34 @@ def parse_iso(s):
|
|
|
1544
1598
|
except Exception:
|
|
1545
1599
|
return None
|
|
1546
1600
|
|
|
1601
|
+
# The ONE rule for deleting a `.limited` marker on a clean pass. It is the writer half
|
|
1602
|
+
# of the shim's client_marker_recovered and has to be the SAME rule: the scheduled
|
|
1603
|
+
# limits pass runs every 15 minutes, so a writer that clears more freely than the shim
|
|
1604
|
+
# just undoes the shim's fix on its own timer. Before 2026-09-04 it did exactly that —
|
|
1605
|
+
# a truthful client:seven_day marker was deleted 300s after it was written, on a pass
|
|
1606
|
+
# whose every bucket said `percent 0, resets_at null`, i.e. on nothing at all.
|
|
1607
|
+
# * a pass with NO informative bucket proves nothing, so it clears nothing;
|
|
1608
|
+
# * a client rejection naming a WEEKLY window outlives every reading until its own
|
|
1609
|
+
# reset — a weekly bucket cannot fall from the server-proven 100% that wrote the
|
|
1610
|
+
# marker to under the threshold while that window is still open, so a reading
|
|
1611
|
+
# that says it did is wrong by construction;
|
|
1612
|
+
# * a client rejection naming a session/5h window still clears once the pass is
|
|
1613
|
+
# informative and at least CLIENT_LIMIT_CONFIRM_DELAY newer than the marker (#22,
|
|
1614
|
+
# 2026-09-03: 5h markers stranded accounts sitting at 0% usage for days);
|
|
1615
|
+
# * error-cooldown is untouched: it keeps its own window out, exactly as before.
|
|
1616
|
+
def marker_bucket(txt):
|
|
1617
|
+
for part in txt.split():
|
|
1618
|
+
if part.startswith('bucket='):
|
|
1619
|
+
return part[7:]
|
|
1620
|
+
return ''
|
|
1621
|
+
|
|
1622
|
+
def weekly_marker(txt):
|
|
1623
|
+
# Matched the way both shims match it (bin/claude ~303, same token list): the
|
|
1624
|
+
# claude client writes client:seven_day / client:seven_day_opus, the codex client
|
|
1625
|
+
# writes client:7d, and any future weekly* name is caught too.
|
|
1626
|
+
b = marker_bucket(txt).lower()
|
|
1627
|
+
return 'seven_day' in b or '7d' in b or 'weekly' in b
|
|
1628
|
+
|
|
1547
1629
|
# `.expired` — the persistent "this account cannot authenticate" marker the shim
|
|
1548
1630
|
# honors. Written only for a PROVEN dead grant (expired/absent refresh token, or a
|
|
1549
1631
|
# 4xx from the refresh endpoint), never for a transient network/5xx/429 hiccup.
|
|
@@ -2160,11 +2242,37 @@ for acct in manifest.get('accounts', []):
|
|
|
2160
2242
|
# the shim ranks only accounts at/under CLAUDE_MULTIACC_SESSION_GATE
|
|
2161
2243
|
# (default 50) while any clear it. (A soft tiebreaker until
|
|
2162
2244
|
# 2026-09-03 — the operator asked for session FIRST, then weekly.)
|
|
2163
|
-
|
|
2164
|
-
|
|
2165
|
-
|
|
2166
|
-
|
|
2167
|
-
|
|
2245
|
+
# A bucket only feeds those three signals if it SAID something. An INFORMATIVE
|
|
2246
|
+
# bucket has a percent above 0, or a parseable reset window. 2026-09-04: for
|
|
2247
|
+
# acct-13/acct-14 the usage endpoint answered EVERY bucket `percent: 0,
|
|
2248
|
+
# resets_at: null` while Claude Code was being rejected on those same accounts
|
|
2249
|
+
# with "You've hit your weekly limit · resets Sep 8"; this writer recorded the
|
|
2250
|
+
# zeros verbatim, which made two provably exhausted accounts the leaders of the
|
|
2251
|
+
# weekly band and handed them 31 of the last ~60 picks. A truthful bucket ALWAYS
|
|
2252
|
+
# carries the window it resets in, so 0% with no window is NO DATA, not an empty
|
|
2253
|
+
# account. 0% WITH a real window stays informative — a genuinely fresh account
|
|
2254
|
+
# must still rank as empty — and one uninformative bucket beside real ones (the
|
|
2255
|
+
# acct-16 shape: `weekly_scoped:Fable` 0/null next to a real session and
|
|
2256
|
+
# weekly_all) leaves the real buckets ranking exactly as they do today.
|
|
2257
|
+
def informative(b):
|
|
2258
|
+
return b['percent'] > 0 or parse_iso(b.get('resets_at')) is not None
|
|
2259
|
+
|
|
2260
|
+
live = [b for b in buckets if informative(b)]
|
|
2261
|
+
weekly = [b['percent'] for b in live if b['group'] != 'session']
|
|
2262
|
+
session = [b['percent'] for b in live if b['group'] == 'session']
|
|
2263
|
+
# PER SIGNAL, never borrowed from another one. Each of the three answers a
|
|
2264
|
+
# different question, so each is written only when a bucket of ITS OWN kind said
|
|
2265
|
+
# something. Until 2026-09-04 weekly_percent fell back to the overall peak and
|
|
2266
|
+
# session_percent to 0: an account whose weekly buckets were all uninformative
|
|
2267
|
+
# while its 5h bucket read 40% was recorded as 40% WEEKLY — a number no bucket
|
|
2268
|
+
# ever reported, and the signal the band ranks on — and the mirror image
|
|
2269
|
+
# (informative weekly, silent session) was recorded as session 0%, which walks
|
|
2270
|
+
# straight through the session gate. A signal nobody reported must be ABSENT so
|
|
2271
|
+
# the shim reads it as unknown; inventing one is the same mistake as recording a
|
|
2272
|
+
# fake zero, one layer up.
|
|
2273
|
+
maxp = max([b['percent'] for b in live] or [0])
|
|
2274
|
+
weeklyp = max(weekly) if weekly else None
|
|
2275
|
+
sessionp = max(session) if session else None
|
|
2168
2276
|
# How long weekly_percent keeps meaning something. A weekly bucket only ever RISES
|
|
2169
2277
|
# until its reset, so before that moment a stale percent is still a valid lower
|
|
2170
2278
|
# bound and the shim can rank on it when nothing fresher exists; after it, the
|
|
@@ -2173,13 +2281,32 @@ for acct in manifest.get('accounts', []):
|
|
|
2173
2281
|
# It must come from the bucket weekly_percent actually CAME FROM: a low monthly
|
|
2174
2282
|
# bucket resetting in an hour says nothing about an 80% weekly one that resets in
|
|
2175
2283
|
# five days, and taking the minimum over all of them would throw the 80% away.
|
|
2176
|
-
wresets = [int(b['resets_epoch']) for b in
|
|
2284
|
+
wresets = [int(b['resets_epoch']) for b in live
|
|
2177
2285
|
if b['group'] != 'session' and b['percent'] == weeklyp
|
|
2178
2286
|
and isinstance(b.get('resets_epoch'), int)]
|
|
2179
|
-
out = {'fetched_at': int(now), 'source': source
|
|
2180
|
-
|
|
2181
|
-
|
|
2182
|
-
|
|
2287
|
+
out = {'fetched_at': int(now), 'source': source}
|
|
2288
|
+
if live:
|
|
2289
|
+
out['max_percent'] = maxp
|
|
2290
|
+
if weeklyp is not None:
|
|
2291
|
+
out['weekly_percent'] = weeklyp
|
|
2292
|
+
if sessionp is not None:
|
|
2293
|
+
out['session_percent'] = sessionp
|
|
2294
|
+
if weeklyp is not None:
|
|
2295
|
+
# Written with weekly_percent or not at all: the horizon describes THAT
|
|
2296
|
+
# reading, and the shim's degraded path needs both or neither.
|
|
2297
|
+
out['weekly_resets_epoch'] = min(wresets) if wresets else 0
|
|
2298
|
+
if not live:
|
|
2299
|
+
# Nothing usable in the entire payload. Keep the diagnostics (fetched_at,
|
|
2300
|
+
# source, the raw buckets) and write NONE of the three percent signals: a
|
|
2301
|
+
# missing field makes the shim's fresh_field/cutoff_field reads fail, so the
|
|
2302
|
+
# account is UNKNOWN to both selection cuts — never the weekly band's leader,
|
|
2303
|
+
# never inside the session gate, and never able to clear a client-rate-limit
|
|
2304
|
+
# marker. Unknown is the honest reading; "0%" is what re-admitted two provably
|
|
2305
|
+
# exhausted accounts on 2026-09-04 (see informative() above).
|
|
2306
|
+
out['no_data'] = True
|
|
2307
|
+
say(f'{aid}: usage endpoint returned all-zero buckets with no reset windows — '
|
|
2308
|
+
f'no usable telemetry (account ranks as unknown, not as empty)')
|
|
2309
|
+
out['buckets'] = buckets
|
|
2183
2310
|
tmp = lpath + '.tmp'
|
|
2184
2311
|
with open(tmp, 'w') as f:
|
|
2185
2312
|
json.dump(out, f, indent=1)
|
|
@@ -2205,14 +2332,37 @@ for acct in manifest.get('accounts', []):
|
|
|
2205
2332
|
worst = max(account_level or offenders,
|
|
2206
2333
|
key=lambda b: (int(b['resets_epoch']), b['percent']))
|
|
2207
2334
|
reset_epoch = int(worst['resets_epoch'])
|
|
2208
|
-
#
|
|
2209
|
-
with
|
|
2210
|
-
|
|
2211
|
-
|
|
2212
|
-
|
|
2213
|
-
|
|
2214
|
-
|
|
2215
|
-
|
|
2335
|
+
# Never SHORTEN an active client-rate-limit marker. A client rejection is
|
|
2336
|
+
# first-hand server evidence with its own reset; this pass's worst offender
|
|
2337
|
+
# can be a mere session bucket an hour from resetting, and overwriting the
|
|
2338
|
+
# marker with that shorter horizon re-admits a provably exhausted account
|
|
2339
|
+
# early (codex review, 2026-09-04: a client:seven_day four days out replaced
|
|
2340
|
+
# by a 95% session bucket +1h). A LATER reset may still extend the exclusion.
|
|
2341
|
+
keep_client = False
|
|
2342
|
+
try:
|
|
2343
|
+
cur = open(mpath).read()
|
|
2344
|
+
first = cur.splitlines()[0] if cur else ''
|
|
2345
|
+
# The shims' own validity rule (num_ok: digits only, bounded length): a
|
|
2346
|
+
# signed/padded/absurd first line is a GARBLED marker to them, and a
|
|
2347
|
+
# garbled marker must be rewritten here, not preserved.
|
|
2348
|
+
if first.isdigit() and len(first) <= 18:
|
|
2349
|
+
cur_reset = int(first)
|
|
2350
|
+
keep_client = ('reason=client-rate-limit' in cur
|
|
2351
|
+
and cur_reset > now and cur_reset >= reset_epoch)
|
|
2352
|
+
except Exception:
|
|
2353
|
+
pass
|
|
2354
|
+
if keep_client:
|
|
2355
|
+
say(f"{aid}: keeping the client-reported marker (its reset reaches further "
|
|
2356
|
+
f"than this pass's worst offender)")
|
|
2357
|
+
else:
|
|
2358
|
+
# Atomic: a concurrent shim must never read a half-written marker.
|
|
2359
|
+
with open(mpath + '.tmp', 'w') as f:
|
|
2360
|
+
f.write(f'{reset_epoch}\n')
|
|
2361
|
+
f.write(f"bucket={worst['name']} percent={worst['percent']} "
|
|
2362
|
+
f"marked_at={time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())} "
|
|
2363
|
+
f"reason=limits resets_at={worst['resets_at']}\n")
|
|
2364
|
+
os.replace(mpath + '.tmp', mpath)
|
|
2365
|
+
say(f"{aid}: LIMITED {worst['name']} at {worst['percent']}% (resets {worst['resets_at']})")
|
|
2216
2366
|
else:
|
|
2217
2367
|
if os.path.exists(mpath):
|
|
2218
2368
|
# A shim-written marker outlives a clean limits pass while its own window
|
|
@@ -2235,11 +2385,17 @@ for acct in manifest.get('accounts', []):
|
|
|
2235
2385
|
if marked_epoch is None:
|
|
2236
2386
|
marked_epoch = os.path.getmtime(mpath)
|
|
2237
2387
|
recent_client = now - marked_epoch < CLIENT_LIMIT_CONFIRM_DELAY
|
|
2238
|
-
|
|
2239
|
-
|
|
2240
|
-
|
|
2241
|
-
|
|
2388
|
+
is_client = 'reason=client-rate-limit' in txt
|
|
2389
|
+
# See weekly_marker() above for why these three come first and in
|
|
2390
|
+
# this order; the shim applies the identical test per invocation.
|
|
2391
|
+
if active and not live:
|
|
2392
|
+
keep = True
|
|
2393
|
+
elif active and is_client and weekly_marker(txt):
|
|
2394
|
+
keep = True
|
|
2395
|
+
elif active and ('reason=error-cooldown' in txt
|
|
2396
|
+
or (is_client and recent_client)):
|
|
2242
2397
|
keep = True
|
|
2398
|
+
client_recovered = active and is_client and not keep
|
|
2243
2399
|
except Exception:
|
|
2244
2400
|
pass
|
|
2245
2401
|
if not keep:
|
|
@@ -2249,7 +2405,16 @@ for acct in manifest.get('accounts', []):
|
|
|
2249
2405
|
with open(cleared + '.tmp', 'w') as f:
|
|
2250
2406
|
f.write(f'{int(now)}\n')
|
|
2251
2407
|
os.replace(cleared + '.tmp', cleared)
|
|
2252
|
-
|
|
2408
|
+
# On a no-data pass `live` is empty and maxp is 0 only because
|
|
2409
|
+
# nothing was reported (see informative() above). Log that, instead
|
|
2410
|
+
# of a "0%" that reads like a proven-empty account — the exact
|
|
2411
|
+
# misreading behind the 2026-09-04 incident.
|
|
2412
|
+
seen = f'max {maxp}%' if live else 'no usable telemetry'
|
|
2413
|
+
say(f'{aid}: marker cleared ({seen})')
|
|
2414
|
+
elif not live:
|
|
2415
|
+
# One line per account, so a no-data pass is legible in limits.log:
|
|
2416
|
+
# the marker was not re-confirmed here, it was merely not disproved.
|
|
2417
|
+
say(f'{aid}: marker kept (no usable telemetry)')
|
|
2253
2418
|
if not quiet:
|
|
2254
2419
|
detail = ' '.join(f"{b['name']}={b['percent']}%" for b in buckets)
|
|
2255
2420
|
print(f'{aid}: ok {detail}')
|