claude-multiacc 1.0.14 → 1.0.16

package/README.md

@@ -101,7 +101,7 @@ The shim prints nothing, logs `timestamp account cwd` (never prompt text) to
 byte-identically. If anything is missing (no manifest, no accounts, unreadable state,
 even an unset `HOME`) it fails **open** into plain passthrough.
 
-**Limit-aware marking.** `claude-accounts limits` (every 5 min via launchd on the Mac,
+**Limit-aware marking.** `claude-accounts limits` (every 15 min via launchd on the Mac,
 cron on the server, plus an opportunistic non-blocking kick from the shim when data is
 >10 min stale) reads each account's own OAuth usage endpoint — the same data `/usage`
 shows. Every bucket the endpoint returns is tracked separately: `session`, `weekly_all`,
@@ -118,11 +118,37 @@ code cannot read at all degrades that one account (fail open), never the run.
 
 Telemetry failures never block work: no fresh data ⇒ account treated as available. The
 endpoint rate-limits per account, so the refresher skips accounts fetched in the last 4 min
-and backs off exponentially (honoring `Retry-After`) on a 429 — `limits --force` overrides
-both. The cadence is deliberately unhurried: several machines polling the same accounts
-every minute earns a 429 with `Retry-After: 3600`, and telemetry then goes stale for an
-hour at a time — which is exactly when every account starts scoring *neutral* and the
-picker loses its ability to tell them apart. If an account's OAuth access token has been expired for a while (idle account,
+and backs off on **every** non-2xx, not just a 429 — honoring `Retry-After`, and parking
+for 6 h on a refusal the server marks `x-should-retry: false`. `limits --force` overrides
+all of it. The cadence is deliberately unhurried: several machines polling the same
+accounts every minute earns a 429 with `Retry-After: 3600`, and telemetry then goes stale
+for an hour at a time — which is exactly when every account starts scoring *neutral* and
+the picker loses its ability to tell them apart.
+
+> **The usage endpoint needs an OAuth login, not a setup token.** A portable
+> `server.token` (`sk-ant-oat01-…`) authenticates *inference* forever, but the usage
+> endpoint refuses it with `403 — OAuth token does not meet scope requirement
+> user:profile`: setup tokens are minted without that scope. So an account whose
+> `.credentials.json` grant has lapsed keeps working perfectly while going **permanently
+> dark for telemetry**, and a pool where that happens to every account ranks everything
+> neutral and picks at random. `claude-accounts status` says `RANKING IS BLIND` when the
+> pool is in that state, and the shim prints an hourly warning on a terminal. The fix is
+> a real sign-in on the machine that polls (`claude-accounts login <acct-NN>`); since
+> `limits.json` is one of the things `sync` pushes, only the **source** machine needs it —
+> the server and peers inherit the telemetry.
+
+**Two freshness windows, on purpose.** Ranking trusts telemetry for an hour
+(`CLAUDE_MULTIACC_STALE_AFTER`, default 3600 — matching the `Retry-After: 3600` the
+endpoint itself hands out, so a healthy pool is not "stale" for 45 minutes of every
+hour). The ≥90% *exclusion* keeps the tighter 15-minute window: ranking chooses between
+working accounts, while the cutoff declares one unusable, and an account reading 89% an
+hour ago may be well past 90% now. When nothing is in-window at all, a stale weekly
+reading is still used **if its bucket has not reset yet** (a weekly bucket only rises
+until then, so the number remains a true lower bound) — logged as `ranking=DEGRADED`.
+Only when even that is unavailable does selection fall back to neutral, logged as
+`ranking=BLIND`.
+
+If an account's OAuth access token has been expired for a while (idle account,
 nothing ran claude under it for hours), the refresher renews it directly via the OAuth
 **refresh-token grant** — the same endpoint and public client id Claude Code itself uses —
 and atomically persists the rotated credential (0600) back to that account's

package/bin/claude

@@ -125,29 +125,105 @@ marker_active() { # true if $1/.limited is still in force; clears cleanly-expire
   return 0
 }
 
-STALE_AFTER=900
+# How old telemetry may be and still rank. 900s was below the floor the usage endpoint
+# ITSELF enforces: it answers a caller at most about once an hour (429 + Retry-After
+# 3600), so a 15-minute window declared the data stale for most of every hour even on a
+# perfectly healthy pool — and stale data ranks NEUTRAL, which is the same as not
+# ranking at all. One hour matches what the endpoint is willing to give.
+STALE_AFTER="${CLAUDE_MULTIACC_STALE_AFTER:-3600}"
+case "$STALE_AFTER" in ''|*[!0-9]*|0) STALE_AFTER=3600 ;; esac
 
-fresh_field() { # fresh_field <acct dir> <json key> -> integer if telemetry fresh, else fail
-  local f="$1/limits.json" fetched v
+# EXCLUSION keeps the old, tight window on purpose. Ranking and the >=90% cutoff are
+# not the same kind of judgement: ranking picks between working accounts and an hour-old
+# number is plenty, while the cutoff decides that an account is UNUSABLE — and an
+# account reading 89% an hour ago may be well past 90% now. Trusting one window for both
+# would have quietly extended a stale "89%" into 45 extra minutes of eligibility.
+EXCLUDE_STALE_AFTER=900
+[ "$EXCLUDE_STALE_AFTER" -gt "$STALE_AFTER" ] && EXCLUDE_STALE_AFTER="$STALE_AFTER"
+
+telem_fetched_at() { # $1 = acct dir -> epoch of the last successful fetch, or fail
+  local f="$1/limits.json" fetched
   [ -f "$f" ] || return 1
   fetched="$(sed -n 's/.*"fetched_at"[^0-9]*\([0-9][0-9]*\).*/\1/p' "$f" 2>/dev/null | head -1)"
   num_ok "$fetched" || return 1
-  [ $((now - fetched)) -le "$STALE_AFTER" ] || return 1
-  v="$(sed -n "s/.*\"$2\"[^0-9]*\([0-9][0-9]*\).*/\1/p" "$f" 2>/dev/null | head -1)"
+  printf '%s\n' "$fetched"
+}
+
+limits_field() { # limits_field <acct dir> <json key> -> integer, or fail
+  local v
+  v="$(sed -n "s/.*\"$2\"[^0-9]*\([0-9][0-9]*\).*/\1/p" "$1/limits.json" 2>/dev/null | head -1)"
   num_ok "$v" || return 1
   printf '%s\n' "$v"
 }
 
+# fresh_field/cutoff_field parse fetched_at inline rather than through
+# telem_fetched_at: they run several times per account on EVERY invocation, and the
+# difference is a fork apiece. (This file is deliberately fork-frugal — see
+# sessions_owned.) telem_fetched_at exists for the once-per-account callers.
+within_window() { # $1 = acct dir, $2 = window seconds
+  local f="$1/limits.json" fetched
+  [ -f "$f" ] || return 1
+  fetched="$(sed -n 's/.*"fetched_at"[^0-9]*\([0-9][0-9]*\).*/\1/p' "$f" 2>/dev/null | head -1)"
+  num_ok "$fetched" || return 1
+  [ $((now - fetched)) -le "$2" ]
+}
+
+fresh_field() { # fresh_field <acct dir> <json key> -> integer if telemetry fresh, else fail
+  within_window "$1" "$STALE_AFTER" || return 1
+  limits_field "$1" "$2"
+}
+
+# Same, but for the >=90% cutoff, which gets the tighter window (see EXCLUDE_STALE_AFTER).
+cutoff_field() { # $1 = acct dir, $2 = json key
+  within_window "$1" "$EXCLUDE_STALE_AFTER" || return 1
+  limits_field "$1" "$2"
+}
+
+# LAST-RESORT ranking input, used only when NOTHING in the pool is fresh (see the blind
+# guard below). A weekly bucket only rises until its reset, so until that moment an old
+# weekly reading is still a true lower bound on today's usage — strictly more information
+# than the neutral 50 that erases every difference between accounts and turns selection
+# into a coin flip. Once the reset has passed, the number describes a week that is over
+# and is worth exactly nothing, so it is refused.
+stale_weekly() { # $1 = acct dir
+  local resets
+  resets="$(limits_field "$1" weekly_resets_epoch)" || return 1
+  [ "$resets" -gt "$now" ] || return 1
+  limits_field "$1" weekly_percent
+}
+
+# TRUE when this account contributes nothing to ranking: no in-window telemetry at all.
+# When every candidate is blind, every score is the same neutral constant, pick_best
+# sees one enormous tie, and selection quietly becomes uniform random — the failure
+# this whole file exists to prevent.
+telem_blind() { # $1 = acct dir
+  ! within_window "$1" "$STALE_AFTER"
+}
+
+# Human age for the warning line: seconds -> "3h" / "11d". Never fails.
+age_human() { # $1 = seconds
+  local s="$1"
+  if [ "$s" -ge 86400 ]; then printf '%dd\n' $((s / 86400))
+  elif [ "$s" -ge 3600 ]; then printf '%dh\n' $((s / 3600))
+  else printf '%dm\n' $((s / 60)); fi
+}
+
 # RANKING score — lower is better (more headroom). Weekly headroom dominates: a weekly
 # bucket only refills on the account's fixed weekly reset (days away), while the 5h
 # session bucket self-heals, so session is a mild tiebreaker only. (Anthropic's docs
 # confirm this reset asymmetry — an account whose only near-full bucket is the cheap
 # session one must NOT rank behind one burning durable weekly headroom.)
 #   score = weekly%*1000 + session%      weekly,session in [0,100]
-# Stale/unreadable telemetry ranks NEUTRAL (weekly 50, session 50), never "free".
+# Stale/unreadable telemetry ranks NEUTRAL (weekly 50, session 50), never "free" —
+# EXCEPT in a blind pool (SEL_DEGRADED=1), where a still-valid stale weekly reading is
+# used instead. Neutral is only the right answer while some other account HAS fresh
+# data to be neutral against; when no account does, neutral is just a coin flip.
+SEL_DEGRADED=0
 sel_score_of() { # $1 = acct dir
   local w s
-  w="$(fresh_field "$1" weekly_percent)" || w="$(fresh_field "$1" max_percent)" || w=50
+  if ! w="$(fresh_field "$1" weekly_percent)" && ! w="$(fresh_field "$1" max_percent)"; then
+    if [ "$SEL_DEGRADED" = 1 ]; then w="$(stale_weekly "$1")" || w=50; else w=50; fi
+  fi
   s="$(fresh_field "$1" session_percent)" || s=50
   printf '%s\n' $((w * 1000 + s))
 }
@@ -165,7 +241,7 @@ util_of() {
 # (fail open — telemetry must never invent exclusions).
 over_threshold() { # $1 = acct dir
   local v
-  v="$(fresh_field "$1" max_percent)" || return 1
+  v="$(cutoff_field "$1" max_percent)" || return 1
   [ "$v" -ge "${CLAUDE_MULTIACC_THRESHOLD:-90}" ]
 }
 
@@ -399,6 +475,48 @@ client_limit_scan() { # $1 acct dir
   return 1
 }
 
+# A TUI auth failure cannot use the -p retry path because the client owns the terminal.
+# Harvest the same account-owned transcripts used for quota detection so the next launch
+# parks a rejected setup-token instead of selecting it again.
+client_auth_scan() { # $1 acct dir
+  local idx="$1/.sessions-index" ln id claim p line ts ck ak read_n=0 i
+  local ids=() claims=()
+  [ -f "$idx" ] || return 1
+  while IFS= read -r ln; do
+    id="${ln%% *}"; claim=""
+    case "$ln" in *' '*) claim="${ln#* }" ;; esac
+    sess_id_ok "$id" && { ids+=("$id"); claims+=("$claim"); }
+  done < "$idx"
+  i=$(( ${#ids[@]} - 1 ))
+  while [ "$i" -ge 0 ] && [ "$read_n" -lt "$QUOTA_SCAN_MAX_FILES" ]; do
+    id="${ids[$i]}"; claim="${claims[$i]}"; i=$((i - 1)); read_n=$((read_n + 1))
+    sess_transcript "$1" "$id" || continue
+    p="$SESS_TRANSCRIPT"
+    [ $((now - $(file_mtime "$p"))) -le "$QUOTA_SCAN_MAX_AGE" ] || continue
+    line="$(tail -c "$QUOTA_SCAN_BYTES" "$p" 2>/dev/null \
+      | LC_ALL=C grep -a '"error"[[:space:]]*:[[:space:]]*"authentication_failed"' | tail -1)"
+    [ -n "$line" ] || continue
+    [ -n "$claim" ] || continue
+    ts="$(printf '%s' "$line" | LC_ALL=C sed -n 's/.*"timestamp"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')"
+    ck="$(iso_key "$ts")"; ak="$(iso_key "$claim")"
+    num_ok "$ck" || continue; num_ok "$ak" || continue
+    [ "$ck" -lt "$ak" ] && continue
+    claim_conflicted "$1" "$id" "$claim" && continue
+    return 0
+  done
+  return 1
+}
+
+mark_client_auth_dead() { # $1 acct dir
+  local soft=$((now + 3600)) marked
+  marked="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+  { echo "$now"; echo "reason=auth-error soft_until=$soft marked_at=$marked" \
+      "detail=client session failed to authenticate"; } \
+    2>/dev/null > "$1/.expired.$$" && mv -f "$1/.expired.$$" "$1/.expired" 2>/dev/null \
+    || rm -f "$1/.expired.$$" 2>/dev/null || true
+  sel_log "$(basename "$1") parked (auth-error until $soft) — client-reported"
+}
+
 mark_client_limit() { # $1 acct dir, $2 reset epoch, $3 rate limit type
   local m="$1/.limited" cur=""
   # Never shorten a marker that already reaches further out (a weekly park must
@@ -431,11 +549,11 @@ sel_capture_session() { # $1 acct dir
     # stops the moment the run it is watching is gone.
     trap '' INT HUP TERM QUIT
     i=0
-    while [ "$i" -lt 30 ]; do
+    while [ "$i" -lt 600 ]; do
       kill -0 "$pid" 2>/dev/null || break
       sess_index_refresh "$d"
       [ -f "$d/sessions/$pid.json" ] && break
-      sleep 2
+      sleep 0.1
       i=$((i + 1))
     done
     sess_index_refresh "$d"
@@ -562,6 +680,12 @@ for d in "$ACC_ROOT"/acct-*; do
   fi
   valid+=("$d")
   sess_index_refresh "$d"
+  if client_auth_scan "$d"; then
+    mark_client_auth_dead "$d"
+    valid=("${valid[@]:0:${#valid[@]}-1}")
+    expired+=("$d")
+    continue
+  fi
   marker_active "$d" && continue
   # The account's own records are consulted BEFORE telemetry: what the server told a real
   # call is first-hand and carries the real reset, while limits.json can be days stale —
@@ -679,7 +803,45 @@ pick_best() { # args: candidate dirs
   PICK_SCORE="$bestv"
 }
 
+# Telemetry going stale is not a per-run detail, it is a pool-wide outage: with no
+# in-window data ANYWHERE every account scores the identical NEUTRAL value, the tie
+# spans the whole pool, and "pick the account with the most headroom" silently becomes
+# "pick any account at all". That is how a fresh session lands on the one account
+# already at 80% of its weekly limit while `claude-accounts status` still shows a
+# reassuring 2% from eleven days ago. It cost eleven days of blind picks once.
+# Two answers, and the order matters: rank on whatever old readings are still true
+# BEFORE picking, and say out loud which of the two happened.
+blind=1           # 1 = no candidate has in-window telemetry
+blind_age=0       # newest stale reading among the candidates; 0 = never fetched at all
+degraded=0        # 1 = blind, but every candidate had a stale reading still worth using
+assess_telemetry() { # args: the dirs actually being chosen between
+  local d f n=0 stale_ok=0
+  blind=1; blind_age=0; degraded=0
+  for d in "$@"; do
+    n=$((n + 1))
+    if ! telem_blind "$d"; then blind=0; return 0; fi
+    f="$(telem_fetched_at "$d" || echo 0)"
+    # The NEWEST stale reading is the honest age of the outage; an account that was
+    # never fetched at all must not make the pool look older than it is.
+    [ "$f" -gt 0 ] && { [ "$blind_age" -eq 0 ] || [ $((now - f)) -lt "$blind_age" ]; } \
+      && blind_age=$((now - f))
+    stale_weekly "$d" >/dev/null && stale_ok=$((stale_ok + 1))
+  done
+  # All or nothing. A candidate whose reading has no horizon — a limits.json written
+  # before this field existed, or one whose week has already turned — scores neutral
+  # 50, and 50 would beat a NEIGHBOUR's true-but-worse 70. Mixing the two makes the
+  # degraded ranking actively wrong, so it is only used when every candidate can be
+  # compared on the same footing.
+  [ "$n" -gt 0 ] && [ "$stale_ok" -eq "$n" ] && degraded=1
+  return 0
+}
+
 if [ "${#eligible[@]}" -gt 0 ]; then
+  # Blindness is judged over the accounts actually being chosen between, not over every
+  # valid one: a FRESH account sitting behind a .limited marker is not a candidate, and
+  # letting it clear the flag would leave the real candidates ranking neutral.
+  assess_telemetry "${eligible[@]}"
+  [ "$degraded" = 1 ] && SEL_DEGRADED=1
   if [ "${CLAUDE_SHIM_SELECT:-headroom}" = "random" ]; then
     PICK_DIR="${eligible[$((RANDOM % ${#eligible[@]}))]}"
   else
@@ -687,8 +849,19 @@ if [ "${#eligible[@]}" -gt 0 ]; then
   fi
 else
   # Every account is limit-marked: degraded service beats a hard failure (100% rule).
+  assess_telemetry "${valid[@]}"
+  [ "$degraded" = 1 ] && SEL_DEGRADED=1
   pick_best "${valid[@]}"
-  sel_log "all-limited fallback=$(basename "$PICK_DIR") weekly=$(fresh_field "$PICK_DIR" weekly_percent || echo '?')%"
+  # Report the number this fallback ACTUALLY ranked on. Asking fresh_field here printed
+  # `weekly=?%` even when the pick was made on a perfectly good stale reading, so anyone
+  # reading only this event concluded the choice had no usage input at all.
+  if [ "$degraded" = 1 ]; then
+    sel_log "all-limited fallback=$(basename "$PICK_DIR") weekly=$(stale_weekly "$PICK_DIR" || echo '?')% ranking=DEGRADED"
+  elif [ "$blind" = 1 ]; then
+    sel_log "all-limited fallback=$(basename "$PICK_DIR") weekly=?% ranking=BLIND"
+  else
+    sel_log "all-limited fallback=$(basename "$PICK_DIR") weekly=$(fresh_field "$PICK_DIR" weekly_percent || echo '?')%"
+  fi
 fi
 pick="$PICK_DIR"
 # Remember the pick so the NEXT run does not hand back the same account. An explicit
@@ -717,7 +890,37 @@ if [ "$stale" = 1 ] && [ -x "$SELF_DIR/claude-accounts" ]; then
 fi
 
 acct="$(basename "$pick")"
-sel_log "$acct weekly=$(fresh_field "$pick" weekly_percent || echo '?')% session=$(fresh_field "$pick" session_percent || echo '?')% pwd=$PWD"
+if [ "$blind" = 1 ]; then
+  # Two genuinely different states, and an operator debugging this needs to know which:
+  # DEGRADED still ranks, on old readings that remain true; BLIND cannot rank at all and
+  # is a coin flip. Calling both of them "random" would send someone hunting the wrong bug.
+  if [ "$degraded" = 1 ]; then
+    sel_log "$acct weekly=$(stale_weekly "$pick" || echo '?')% session=?% ranking=DEGRADED telemetry-age=${blind_age}s pwd=$PWD"
+  else
+    sel_log "$acct weekly=?% session=?% ranking=BLIND telemetry-age=${blind_age}s pwd=$PWD"
+  fi
+  # Terminal only, at most hourly — a service-spawned `claude -p` must keep its stderr
+  # byte-clean, and this is advice, never a failure.
+  if [ -t 2 ]; then
+    n="$ACC_ROOT/.stale-notice"
+    last=0
+    [ -f "$n" ] && last="$(file_mtime "$n")"
+    if [ $((now - last)) -gt 3600 ]; then
+      : 2>/dev/null > "$n" || true
+      if [ "$degraded" = 1 ]; then
+        printf 'claude-multiacc: usage telemetry is %s old — ranking on the last readings that are still valid, not on current usage. Fix: claude-accounts limits --force, then claude-accounts status\n' \
+          "$(age_human "$blind_age")" >&2
+      elif [ "$blind_age" -gt 0 ]; then
+        printf 'claude-multiacc: usage telemetry is %s old for EVERY account and too old to mean anything — selection is running blind (random, not by headroom). Fix: claude-accounts limits --force, then claude-accounts status\n' \
+          "$(age_human "$blind_age")" >&2
+      else
+        printf 'claude-multiacc: no usage telemetry for ANY account — selection is running blind (random, not by headroom). Fix: claude-accounts limits --force, then claude-accounts status\n' >&2
+      fi
+    fi
+  fi
+else
+  sel_log "$acct weekly=$(fresh_field "$pick" weekly_percent || echo '?')% session=$(fresh_field "$pick" session_percent || echo '?')% pwd=$PWD"
+fi
 
 export CLAUDE_SHIM_ACTIVE=1