@wendellresearch-ai/benchwarmers 0.5.1 → 0.5.3

package/README.md

@@ -187,7 +187,8 @@ dependencies either way: the app is the Python standard library.
 - **Health.** Every morning each model gets a one-word prompt; once a week, a
   short and a long one for speed. A model that stops answering, or that its
   endpoint no longer serves, is retired automatically after one confirming
-  call, with the evidence kept, and can be restored in one click. The
+  call, with the evidence kept, and can be restored in one click. A model
+  that is only out of quota until a stated reset is not counted against. The
   built-in scheduler runs all of this once Settings switches it on; a fresh
   install spends no calls until then.
 - **Rate limits and usage.** Limits come from providers' rate-limit headers

package/USAGE.md

@@ -25,6 +25,7 @@ coding agent doing it with you); the gateway's whole contract is in
 | Group | Page | What it is for |
 | --- | --- | --- |
 | Observe | **At a glance** | Every model on the bench: whether it works right now, its role, and its connection. Click a name for its popup: role, alias, facts, options, retire. |
+| | **Action needed** | What will not fix itself, by the connection it comes from, with Retry and Retire on each model. |
 | | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, and speed. |
 | | **Quota management** | A tile per provider: what is left, when it resets, and which of its models are resting. Then the week's resets, and what the bench sent. |
 | | **Logs** | Gateway calls (who sat and how it went, never the prompt), changes made from the dashboard, the scheduler, and onboarding runs. |
@@ -35,10 +36,21 @@ coding agent doing it with you); the gateway's whole contract is in
 | Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, how the install is reached (shown only), and Restart. |
 | | **Info** | Version, Python, data folder and counts. |
 
-"Action needed", above the pages, lists what will not fix itself, such as a
-model about to be retired, naming each model and where it comes from, with
-Retry, Open and Retire now beside it. Hide folds it away until something new
-turns up; an alert whose models have all been retired goes at once.
+**Action needed** lists what will not fix itself, such as a model about to be
+retired, grouped by the connection it comes from, since one cause usually
+reaches every model there. Each model is one line: its state, the provider's
+own words for the error (whole on hover), and Retry (one call; an answer
+clears the alert) and Retire. A group of two or more has Retry all and Retire
+all; a connection held because every model on it fails says so above its
+models, with any link the provider gave. Automatic retirements from the last
+week are folded at the end, each with Restore. Its count sits beside it in the
+sidebar, and At a glance opens on one line saying what is waiting and how
+soon. An alert whose models have all been retired goes at once. A model that
+is out of quota is not counting down: days the bench held its calls back until
+a stated reset, an account's shared free quota was spent, or the provider said
+the model's free quota is used up, pause its clock, and while it is still out
+of quota no countdown shows. One refused again after the short wait it named
+is failing, and keeps counting.
 
 **Restart** (Settings, Server and access), or `bench restart`, starts the
 dashboard again on the code now on disk, after an update. It waits for

package/VERSION

@@ -1 +1 @@
-0.5.1
+0.5.3

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wendellresearch-ai/benchwarmers",
-  "version": "0.5.1",
+  "version": "0.5.3",
   "description": "Self-hosted panel gateway for LLMs: ask a collection of models at once, keep every answer side by side, rotate explorer models in, and let your verdicts pick the next panel.",
   "mcpName": "io.github.WendellResearch-AI/benchwarmers",
   "license": "MIT",

package/scripts/auto_retire.py

@@ -19,6 +19,14 @@ THE RULES (decided 2026-10-15)
   reason -- a rate limit that never lifts is a quota or a dead model.
   `mistral-medium-3-5` answered 429 to 11 calls out of 11.
 - NEVER on a rate limit or timeout alone inside those 7 days.
+- OUT OF QUOTA IS NOT FAILING (2026-11-02): a call the bench held back because
+  a limit is spent until a stated reset, or an account-wide quota, says
+  nothing about the model. Its days pause the clock, and while it is the
+  latest word there is no countdown. `gpt-5.6-luna` sat on "retires in 3
+  days" while CLI Proxy API's codex sign-in was out until 26 November and the
+  bench had not sent it a call since 28 October. A refusal that names a short
+  wait and is refused again after it still counts: that is a model that keeps
+  failing, not one that is tapped out.
 
 "No answer" means from ANY source: the perf probe, a useful offload, or a
 suite output that was a real answer. A model that answers in the suite resets
@@ -68,6 +76,18 @@ def account_quota(detail: str) -> bool:
     return any(marker in detail for marker in ACCOUNT_QUOTAS)
 
 
+HELD = "not sent; paced by the bench"                  # metrics/pacing.py's refusal
+USED_UP = ("free usage quota",)                        # xKiro: "...quota for this model is used up"
+
+
+def tapped_out(detail: str) -> bool:
+    """Out of quota right now, so the model could not be tried: the bench held
+    the call back (a limit spent until its reset, the provider's or a cap you
+    set), the account's shared quota is spent (account_quota), or the
+    provider says the model's own free quota is used up."""
+    return HELD in detail or account_quota(detail) or any(m in detail for m in USED_UP)
+
+
 def plan(today: dt.date, models: dict[str, str], live: set[str],
          first_missing: dict[str, str], rows: list[dict],
          answers: dict[str, dt.date], restored: dict[str, dt.date],
@@ -87,15 +107,18 @@ def plan(today: dt.date, models: dict[str, str], live: set[str],
     down: dict[str, bool] = {}
     candidates: list[tuple[str, str, str]] = []      # (alias, bucket, why)
     for alias, served in models.items():
-        mine = [r for r in rows if r["alias"] == alias]
+        mine = sorted((r for r in rows if r["alias"] == alias), key=lambda r: r["date"])   # [-1] is the latest word
         last_ok = max([r["date"] for r in mine if r["answered"]]
                       + ([answers[alias]] if alias in answers else []),
                       default=None)
         floor = max(d for d in (last_ok, restored.get(alias), dt.date.min) if d)
         # A refused key (`signin`) is the endpoint's, fixed by signing in again.
         fails = sorted((r for r in mine if not r["answered"] and r["failure"] and r["failure"] != "signin"
-                        and r["date"] > floor and not account_quota(r["detail"])),
+                        and r["date"] > floor and not tapped_out(r["detail"])),
                        key=lambda r: r["date"])
+        quiet = {r["date"] for r in mine if not r["answered"] and r["date"] > floor
+                 and tapped_out(r["detail"])}
+        resting = bool(mine) and not mine[-1]["answered"] and tapped_out(mine[-1]["detail"])
         missing_since = None
         if served not in live and served in first_missing:
             missing_since = max(dt.date.fromisoformat(first_missing[served]),
@@ -107,8 +130,10 @@ def plan(today: dt.date, models: dict[str, str], live: set[str],
         why = (f"{latest['failure']}: {latest['detail']}".rstrip(": ")
                if latest else "")
         other_days = {r["date"] for r in fails if r["failure"] == "other"}
-        span = (today - fails[0]["date"]).days if fails else 0
         fail_days = {r["date"] for r in fails}
+        # Days it was out of quota, and nothing else, do not count: the clock pauses.
+        span = ((today - fails[0]["date"]).days
+                - len({q for q in quiet - fail_days if q > fails[0]["date"]})) if fails else 0
 
         # A CONNECTION PREFIX ADDED IN OPEN WEBUI renames every model on it
         # (`codestral-latest` -> `llm7.codestral-latest`), and the old names
@@ -146,7 +171,7 @@ def plan(today: dt.date, models: dict[str, str], live: set[str],
                 left.append((OTHER_DAYS - len(other_days), why))
             if fails:
                 left.append((BACKSTOP_DAYS - span, why))
-            if left:
+            if left and not resting:                   # resting: the Quota page says when it is back
                 days, reason = min(left)
                 out["pending"].append((alias, f"{max(days, 1)}|{reason}"))
 
@@ -351,6 +376,24 @@ def _selfcheck() -> None:
                    detail="429 Rate limit exceeded: free-models-per-day") for n in (8, 5, 0)]
     p = plan(d, {"quota": "v/quota"}, {"v/quota"}, {}, shared, {}, {}, {"v/quota": "v"})
     assert not p["retire"] and not p["pending"], p
+    # held back by the bench for days (its codex sign-in spent until a reset): no countdown
+    held = [dict(row("luna", n, "capacity"), detail=f"rate limit ({HELD}): 0 left") for n in (9, 5, 0)]
+    p = plan(d, {"luna": "c/luna"}, {"c/luna"}, {}, held, {}, {}, {"c/luna": "c"})
+    assert not p["retire"] and not p["pending"], p
+    assert tapped_out("Error code: service_unavailable - The free usage quota for this model is used up")
+    # real failures, then out of quota for a week: those days pause the clock
+    paused = ([row("p", n, "capacity") for n in (10, 9, 8, 0)]
+              + [dict(row("p", n, "capacity"), detail=HELD) for n in range(1, 8)])
+    p = plan(d, {"p": "c/p"}, {"c/p"}, {}, paused, {}, {}, {"c/p": "c"})
+    assert not p["retire"] and dict(p["pending"])["p"].startswith("4|"), p
+    # ...and while it is still out of quota, no countdown is shown at all
+    p = plan(d, {"p": "c/p"}, {"c/p"}, {}, paused[:3] + paused[4:], {}, {}, {"c/p": "c"})
+    assert not p["retire"] and not p["pending"], p
+    # a short stated wait, refused again day after day, is a model that keeps failing
+    busy = [dict(row("u", n, "capacity"), detail="429 allows 1 request(s) every 1 min, retry in 60s")
+            for n in (8, 5, 0)]
+    p = plan(d, {"u": "v/u"}, {"v/u"}, {}, busy, {}, {}, {"v/u": "v"})
+    assert {a for a, _ in p["retire"]} == {"u"}, p
     # A refused key is the endpoint's, never the model's (suite/run.py SIGNIN)
     from run import failure_class
     assert failure_class("Error: Error code: 401 - {'error': 'invalid key'}") == "signin"

package/scripts/test_serve_edit.py

@@ -454,7 +454,7 @@ def main() -> int:
             encoding="utf-8")
         serve.DATA, serve.registered = pathlib.Path(tmp), lambda: {"h1-served": "h1"}
         try:
-            shown = serve.problems()
+            shown = serve.actions_page(*serve.open_actions())
         finally:
             serve.DATA, serve.registered = kept_data, kept_reg
     ok = "here.example" in shown and "gone.example" not in shown and "old retires" not in shown

package/serve.py

@@ -2003,89 +2003,172 @@ def first_run() -> str:
             f"<button class='btn primary' id='first-go'>{label}</button></div></div></section>")
 
 
-def problems() -> str:
-    """What needs a person, each with the button that resolves it.
+def said(detail: str) -> tuple[str, str]:
+    """An error as one short line, and the link the provider put in it: the
+    status code and the provider's own words, without the JSON around them or
+    a lead-in such as "Too many requests." that says less than the sentence
+    after it. The whole detail stays in the row's title."""
+    code = re.search(r"Error code: (\w+)", detail)
+    quoted = re.search(r"""['"](?:message|detail)['"]: ['"](.+?)(?:['"]\s*[,}]|$)""", detail)
+    text = (quoted.group(1) if quoted else re.sub(r"^.*?Error: ", "", detail) if "Error: " in detail
+            else re.sub(r"^\w+: ", "", detail))
+    text = " ".join(text.replace("<<", "").replace(">>", "").split())
+    first, _, rest = text.partition(". ")
+    if rest and len(first) < 25:
+        text = rest
+    url = re.search(r"https?://[^\s;,'\"()]+", quoted.group(1) if quoted else "")
+    text = (f"{code.group(1)} · " if code else "") + text
+    return (text if len(text) <= 110 else text[:109].rstrip() + "…"), (url.group(0) if url else "")
+
+
+def open_actions() -> tuple[dict[str, dict], list[tuple[str, dict]]]:
+    """What needs a person, by the connection it comes from ({connection:
+    {"held": [...], "rows": [...]}}, the soonest to retire first), and the
+    last week's automatic retirements, newest first.
 
     Rate limits and timeouts are not listed: they resolve themselves, or count
     toward the 7-day backstop and appear here as `retiring` when that nears.
-    Automatic retirements from the last week ARE listed, with Restore, because
-    a decision taken without a person should be easy for one to undo."""
+    Automatic retirements ARE listed, with Restore, because a decision taken
+    without a person should be easy for one to undo."""
     alerts = load_json(DATA / "metrics" / "alerts.json")
     names = registered()
-    today = datetime.date.today()
-    KIND = {"retiring": "err", "retired": "err", "connection": "warn",
-            "registration": "warn", "plan": "warn"}
-    items = []
     live = set(names.values())
-    for key, a in sorted(alerts.items(), key=lambda kv: kv[1].get("first_seen", "")):
+    status = model_status()
+    groups: dict[str, dict] = {}
+
+    def group(conn: str) -> dict:
+        return groups.setdefault(conn or "?", {"held": [], "rows": {}})
+
+    def row(alias: str, key: str, a: dict) -> dict:
+        # The roster's words for its state, so the two pages agree; the
+        # alert's own evidence for a countdown, the latest probe otherwise.
+        label, css, latest = status.get(alias, (a.get("kind", "?"), "warn", ""))
+        days = re.search(r"in (\d+)d", label)
+        detail = a.get("action", "") if a.get("kind") == "retiring" else latest or a.get("message", "")
+        text, url = said(detail)
+        return {"alias": alias, "key": key, "label": label, "css": "warn" if days and int(days.group(1)) > 2 else css,
+                "days": int(days.group(1)) if days else 99, "said": text, "url": url, "title": detail,
+                "why": a.get("message", "")}
+
+    for key, a in alerts.items():
         served = key.split(":", 2)[2] if key.count(":") >= 2 else ""
         alias = a.get("alias") or names.get(served, "")
         # The alerts file is rewritten by the daily check; until then an alert
         # whose models have all left the bench (Retire all, or one by one) has
         # nothing left to act on.
-        if a.get("aliases") and not live & set(a["aliases"]):
-            continue
-        if a.get("kind") == "retiring" and alias and alias not in live:
+        if (a.get("aliases") and not live & set(a["aliases"])) or (alias and alias not in live):
             continue
-        try:
-            age = (today - datetime.date.fromisoformat(a["first_seen"])).days
-        except (KeyError, ValueError):
-            age = 0
-        # Which model and where it comes from, first: the message alone left that to be looked up.
-        served_as = next((s for s, al in names.items() if al == alias), served) if alias else ""
-        where = (f"{alias} · {served_as or '?'} via {tag_for(alias) or a.get('endpoint') or '?'}"
-                 if alias else "")
-        why = " · ".join(filter(None, [where, a.get("action", ""),
-                                       "open today" if age == 0 else f"open {age} days"]))
-        acts = []
-        if a.get("aliases"):
-            acts.append(f"<button class='btn small danger' data-act='retire'"
-                        f" data-aliases='{esc(' '.join(a['aliases']))}'"
-                        f" data-why='{esc(a.get('message', ''))}'>Retire all {len(a['aliases'])}</button>")
-        elif alias in names.values():
-            if a.get("kind") == "retiring":
-                acts.append(f"<button class='btn small' data-act='retry' data-alias='{esc(alias)}'"
-                            " title='One call now; an answer clears this alert'>Retry</button>")
-            acts.append(f"<button class='btn small' data-act='open' data-alias='{esc(alias)}'>Open</button>")
-            acts.append(f"<button class='btn small danger' data-act='retire' data-alias='{esc(alias)}'"
-                        f" data-why='{esc(a.get('message', ''))}'>Retire now</button>")
-        if a.get("dismiss"):                           # a notice, not a state: it goes once seen
-            acts.append(f"<button class='btn small' data-act='dismiss' data-key='{esc(key)}'>Got it</button>")
-        items.append((key, a.get("kind", "?"), KIND.get(a.get("kind"), "warn"),
-                      a.get("message", ""), why, "".join(acts)))
-    cutoff = (today - datetime.timedelta(days=7)).isoformat()
-    tags = load_json(DATA / "config" / "provider-tags.json")      # a retired model's connection, by served name
-    auto = [(f"retired:{alias}:{r['at']}", "", "",
-             f"{alias} · {r.get('served') or '?'} via {tags.get(r.get('served', ''), '').strip('?') or r.get('endpoint') or endpoints.default()}"
-             f" · {r['at'][:10]}",
-             r.get("why", "").removeprefix("auto: "),
-             f"<button class='btn small' data-act='restore' data-alias='{esc(alias)}'>Restore</button>")
-            for alias, r in sorted(retired().items(), key=lambda kv: kv[1].get("at", ""), reverse=True)
+        if a.get("kind") == "connection":
+            g = group(served)
+            text, url = said(a.get("action", ""))
+            g["held"].append({"said": text, "url": url, "title": a.get("action", "")})
+            for held in a.get("aliases", []):
+                if held in live:
+                    g["rows"].setdefault(held, row(held, key, a))
+        elif alias:                                    # its own alert says more than a held connection's list
+            group((tag_for(alias) or a.get("endpoint") or "?").split("/")[0])["rows"][alias] = row(alias, key, a)
+        else:                                          # a notice about no one model (NVIDIA's limits)
+            group("Notices")["rows"][key] = {"alias": "", "key": key, "label": a.get("kind", "?"), "css": "warn",
+                                             "days": 99, "said": a.get("message", ""), "url": "",
+                                             "title": a.get("action", ""), "dismiss": a.get("dismiss")}
+    for g in groups.values():
+        g["rows"] = sorted(g["rows"].values(), key=lambda r: (r["days"], r["alias"]))
+    order = sorted(groups.items(), key=lambda kv: (kv[0] == "Notices", min([r["days"] for r in kv[1]["rows"]] or [99]),
+                                                   -len(kv[1]["rows"]), kv[0]))
+    cutoff = (datetime.date.today() - datetime.timedelta(days=7)).isoformat()
+    auto = [(alias, r) for alias, r in sorted(retired().items(), key=lambda kv: kv[1].get("at", ""), reverse=True)
             if r.get("by") == "auto" and r.get("at", "") >= cutoff]
-    if not items and not auto:
+    return dict(order), auto
+
+
+def open_count(groups: dict) -> int:
+    """Models and connections waiting on a person: the sidebar's count."""
+    return sum(len(g["rows"]) + len(g["held"]) for g in groups.values())
+
+
+def and_list(parts: list[str]) -> str:
+    return " and ".join(parts) if len(parts) < 3 else ", ".join(parts[:-1]) + " and " + parts[-1]
+
+
+def actions_line(groups: dict) -> str:
+    """At a glance's one line about Action needed, when anything is open:
+    what is waiting and how soon, and the way to its page."""
+    rows = [r for g in groups.values() for r in g["rows"]]
+    held = sum(len(g["held"]) for g in groups.values())
+    if not rows and not held:
         return ""
+    models = sum(1 for r in rows if r["alias"])
+    what = and_list([f"{n} {word}{'s' * (n != 1)}" for n, word in
+                     ((models, "model"), (held, "connection"), (len(rows) - models, "notice")) if n])
+    soon = min(r["days"] for r in rows) if rows else 99
+    many = sum(r["days"] == soon for r in rows)
+    when = (f"; {many} retire{'s' * (many == 1)} in {soon} day{'s' * (soon != 1)}" if soon < 99 else "")
+    return (f"<p class='a-line'><strong>Action needed</strong> <span>{what}{when}.</span>"
+            " <button type='button' class='linkish' data-go='needs'>Review →</button></p>")
+
+
+def actions_page(groups: dict, auto: list) -> str:
+    """Action needed, its own page since 2026-11-02: it sat above every page
+    and grew to a screen and more of red. Grouped by connection, because
+    one cause usually reaches every model on it (nine models on one free
+    tier's rate limit); each model one line, its error in the provider's own
+    words, with Retry a click away."""
+    def link(url: str) -> str:
+        if not url:
+            return ""
+        shown = re.sub(r"^https?://", "", url).split("?")[0]
+        return f" <a href='{esc(url)}' target='_blank' rel='noopener'>{esc(shown)} ↗</a>"
+
+    def model_row(r: dict) -> str:
+        if not r["alias"]:                             # a notice: what it says, and Got it if it goes once seen
+            got = (f"<button class='linkish' data-act='dismiss' data-key='{esc(r['key'])}'>Got it</button>"
+                   if r.get("dismiss") else "")
+            return (f"<div class='a-row' data-row><span class='what'>{esc(r['said'])}</span>"
+                    f"<span>{chip(r['label'], r['css'])}</span><span class='why'>{esc(r['title'])}</span>"
+                    f"<span class='acts'>{got}</span></div>")
+        alias = esc(r["alias"])
+        return (f"<div class='a-row' data-row><a href='#' class='model' data-alias='{alias}'>"
+                f"<span class='alias'>{alias}</span></a><span>{chip(r['label'], r['css'])}</span>"
+                f"<span class='why' title='{esc(r['title'])}'><span class='said'>{esc(r['said'])}</span>{link(r['url'])}</span>"
+                f"<span class='acts'><button class='linkish' data-act='retry' data-alias='{alias}'"
+                " title='One call now; an answer clears the alert'>Retry</button>"
+                f"<button class='linkish danger' data-act='retire' data-alias='{alias}'"
+                f" data-why='{esc(r['why'] or r['label'])}'>Retire</button></span></div>")
+
+    def section(conn: str, g: dict) -> str:
+        aliases = [r["alias"] for r in g["rows"] if r["alias"]]
+        both = (f"<button class='btn small' data-act='retry' data-aliases='{esc(' '.join(aliases))}'"
+                f" title='One call each'>Retry all {len(aliases)}</button>"
+                f"<button class='btn small danger' data-act='retire' data-aliases='{esc(' '.join(aliases))}'"
+                f" data-why='every model on {esc(conn)} needed action'>Retire all {len(aliases)}</button>"
+                if len(aliases) > 1 else "")
+        held = "".join(
+            f"<div class='a-held' title='{esc(h['title'])}'><span class='chip'>held</span><span>Every model"
+            f" here is failing, likely the connection (key, credit, prefix), so nothing was retired."
+            f" Latest: {esc(h['said'])}{link(h['url'])}</span></div>" for h in g["held"])
+        n = len(g["rows"])
+        return (f"<div class='a-group'><div class='a-head' data-row><span class='conn'>{esc(conn)}</span>"
+                f"<span class='count'>{n} {'notice' if conn == 'Notices' else 'model'}{'s' * (n != 1)}</span>"
+                f"<span class='why'></span><span class='acts'>{both}</span></div>{held}"
+                f"<div class='a-rows'>{''.join(model_row(r) for r in g['rows'])}</div></div>")
 
-    # Each row carries its alert key: Hide (bench.js) folds the section to its
-    # heading until a key it has not seen arrives, or 12 hours pass. Per browser.
-    def render_rows(rows):
-        return "".join(
-            f"<div class='problem' data-key='{esc(key)}'><div><div class='what'>"
-            f"{chip(kind, css) if kind else ''}{esc(what)}</div>"
-            f"<div class='why'>{esc(why)}</div></div><div class='acts'>{acts}</div></div>"
-            for key, kind, css, what, why, acts in rows)
-    # Folded: a week of automatic retirements is a record to check, not a
-    # list of things to do, and it must not push the roster off the screen.
-    folded = (f"<details><summary>Retired automatically this week · {len(auto)}"
-              " <span class='muted'>— each can be restored</span></summary>"
-              f"{render_rows(auto)}</details>") if auto else ""
-    return (f"<section class='problems' id='actions' aria-labelledby='actions-head'>"
-            f"<div class='head'><h2 id='actions-head'>Action needed · {len(items)}</h2>"
-            "<span class='sep' aria-hidden='true'>·</span>"
-            "<button type='button' class='linkish' id='actions-hide' aria-expanded='true'"
-            " title='Fold this away until something new needs you, or for 12 hours'>Hide</button></div>"
-            + ("<p>States that will not fix themselves.</p>" if items
-               else "<p>Nothing open.</p>")
-            + f"{render_rows(items)}{folded}</section>")
+    tags = load_json(DATA / "config" / "provider-tags.json")      # a retired model's connection, by served name
+    gone = "".join(
+        f"<div class='a-row' data-row><a href='#' class='model' data-alias='{esc(alias)}'>"
+        f"<span class='alias'>{esc(alias)}</span></a><span class='when'>{esc(r['at'][:10])}</span>"
+        f"<span class='why' title='{esc(r.get('why', ''))}'><span class='said'>"
+        f"{esc(tags.get(r.get('served', ''), '').strip('?') or r.get('endpoint') or endpoints.default())}"
+        f" · {esc(said(r.get('why', '').removeprefix('auto: '))[0])}</span></span>"
+        f"<span class='acts'><button class='linkish' data-act='restore' data-alias='{esc(alias)}'>Restore</button></span></div>"
+        for alias, r in auto)
+    # Folded: a week of automatic retirements is a record to check, not a list of things to do.
+    folded = (f"<details class='a-retired'><summary>Retired automatically this week · {len(auto)}"
+              f" <span class='muted'>— each can be restored</span></summary><div class='a-rows'>{gone}</div></details>"
+              if auto else "")
+    body = "".join(section(conn, g) for conn, g in groups.items()) or "<p class='a-none'>Nothing open.</p>"
+    return ("<div class='panel'><h2>Action needed</h2><p class='sub'>States that will not fix themselves, by the"
+            " connection they come from. A reason's full error is on hover, and a name opens the model."
+            f" Retry spends one call; an answer clears the alert.</p>{body}{folded}</div>")
 
 
 CAP_LABELS = {"tools": "tools", "vision": "vision", "structured_output": "json", "reasoning": "reasons"}
@@ -2831,7 +2914,7 @@ def render(theme: str | None = None) -> str:
         "<button class='btn primary' id='onboard-open'>Onboard a model</button></div></header>"
         # What needs a person first, then the pages; endpoints are managed from
         # Connections now (the header button moved there).
-        f"<main>{first_run()}{problems()}<div class='layout'>{tabs()}</div></main>{editor()}"
+        f"<main>{first_run()}<div class='layout'>{tabs()}</div></main>{editor()}"
         f"<script>window.BENCH={directory_json()};</script>"
         # Observable Plot (on d3) draws the Benchmarks charts. Pinned versions from
         # jsDelivr; without a network the tables under each chart still work.
@@ -3233,6 +3316,7 @@ def collections_tab() -> str:
 # pages were too many for a row of tabs. Collections holds the gateway's
 # requests under the collections they ran; Benchmarks holds Performance.
 SECTIONS = (("Observe", (("glance", "At a glance", "roster"),
+                         ("needs", "Action needed", "actions_page"),
                          ("suite", "Benchmarks", "benchmarks_tab"),
                          ("quotas", "Quota management", "quotas_page"),
                          ("logs", "Logs", "logs_page"))),
@@ -3251,11 +3335,21 @@ def collections_and_gateway() -> str:
 
 
 def tabs() -> str:
+    # Read once a render: Action needed's page, its count in the sidebar and
+    # At a glance's line about it.
+    groups, auto = open_actions()
+    count = {"needs": open_count(groups)}
+    page = {"needs": lambda: actions_page(groups, auto), "glance": lambda: actions_line(groups) + roster()}
+
+    def button(key: str, label: str) -> str:
+        n = count.get(key)
+        return (f"<button role='tab' data-tab='{key}' aria-selected='false'"
+                + (f" aria-label='{esc(label)}, {n} open'>{esc(label)}<span class='n' aria-hidden='true'>{n}</span>"
+                   if n else f">{esc(label)}") + "</button>")
     nav = "".join(f"<div class='group'><h2>{esc(group)}</h2>"
-                  + "".join(f"<button role='tab' data-tab='{key}' aria-selected='false'>{esc(label)}</button>"
-                            for key, label, _ in pages) + "</div>"
+                  + "".join(button(key, label) for key, label, _ in pages) + "</div>"
                   for group, pages in SECTIONS)
-    panes = "".join(f"<section class='tab' id='tab-{key}'>{globals()[fn]()}</section>"
+    panes = "".join(f"<section class='tab' id='tab-{key}'>{page[key]() if key in page else globals()[fn]()}</section>"
                     for key, _, fn in TABS)
     return f"<nav class='tabs' role='tablist' aria-label='Pages'>{nav}</nav><div class='panes'>{panes}</div>"
 

package/suite/run.py

@@ -554,6 +554,7 @@ STRUCTURAL = (
     "unknown model",
     "invalid model",
     "model_not_found",
+    "no model found",                                  # Routeway: "No model found: <id>"
 )
 # ENTITLEMENT is its own class, and the difference decides what to DO.
 # `structural` means the model is not there and the registration should go.
@@ -631,6 +632,9 @@ CAPACITY = (
     # way, filed as `other`, which the 3-day auto-retire rule would act on.
     "request limit reached",
     "resourceexhausted",
+    # xKiro: "The free usage quota for this model is used up" (503). Out of
+    # quota, not gone; filed as `other` it was 3 days from auto-retire.
+    "free usage quota",
 )
 
 

package/ui/bench.css

@@ -99,33 +99,57 @@ button { font: inherit; cursor: pointer; }
 /* ---- layout --------------------------------------------------------- */
 main { max-width: var(--page, 104rem); margin: 0 auto; padding: 0 var(--gutter) 4rem; }
 
-/* problems: what needs a person, each with the action that resolves it */
+/* first run: a fresh install's next step, above every page until it is done */
 .problems { background: var(--sheet, var(--panel)); border-radius: 14px; padding: 1rem 1.4rem; margin: .5rem 0 1.5rem;
             box-shadow: 0 1px 2px rgb(0 0 0 / .04), 0 12px 32px -14px rgb(0 0 0 / .14);
             -webkit-backdrop-filter: var(--glass, none); backdrop-filter: var(--glass, none); }
 .problems h2 { margin: 0 0 .25rem; font-size: 1.25rem; font-weight: 700; }
-.problems > p { margin: 0 0 .5rem; color: var(--muted); font-size: .9375rem; }
 .problem { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: .25rem 1rem;
            align-items: center; padding: .55rem .75rem; margin: 0 -.75rem; border-radius: var(--radius); }
 .problem:hover { background: var(--step-1); }
-.problems details { margin-top: .5rem; }
-.problems details > summary { cursor: pointer; font-weight: 500; padding: .4rem 0; }
-.problems details > summary .muted { font-weight: 400; }
-.problem .what { font-weight: 500; }
 .problem .why { color: var(--muted); font-size: .875rem; overflow-wrap: anywhere; }
 .problem .acts, .endpoint .acts, .ep .acts { display: flex; gap: .5rem; flex-wrap: wrap; justify-content: flex-end; }
 .row-input { display: flex; gap: .5rem; align-items: center; margin-top: .35rem; }
 .row-input input { flex: 1; min-width: 0; margin: 0 !important; }
 .sub-head { font-weight: 600; margin: 1rem 0 .25rem; }
-.problem .chip { margin-right: .4rem; }
 .problems .head { display: flex; align-items: baseline; gap: .5rem; }
 .problems .head h2 { margin-bottom: .25rem; }
-.problems .sep { display: none; color: var(--muted); }
-/* Hidden (bench.js): one line, "Action needed · 3 · show", until something new */
-.problems.collapsed { padding: .5rem 1.25rem; }
-.problems.collapsed > :not(.head) { display: none; }
-.problems.collapsed .head h2 { margin: 0; font-size: 1rem; }
-.problems.collapsed .sep { display: inline; }
+
+/* Action needed: its own page, by connection. Each model is one line of a
+   grid shared by its group (name, state, the provider's words, actions), so
+   the columns line up down the group. */
+.a-line { display: flex; flex-wrap: wrap; align-items: baseline; gap: .25rem .6rem; margin: 1.4rem 1.6rem 0;
+          padding: .45rem .75rem; background: var(--err-ground); border-radius: var(--radius); }
+.a-group { padding: 1rem 0 .75rem; border-top: 1px solid var(--line); }
+.a-group:first-of-type { border-top: 0; padding-top: .5rem; }
+.a-head { display: flex; flex-wrap: wrap; align-items: baseline; gap: .25rem .6rem; margin-bottom: .4rem; }
+.a-head .conn { font-size: 1.0625rem; font-weight: 700; }
+.a-head .count { color: var(--muted); }
+.a-head .why { font-size: .875rem; }
+.a-head .why:empty { display: none; }
+.a-head .acts { margin-left: auto; display: flex; gap: .5rem; }
+.a-held { display: flex; gap: .6rem; align-items: baseline; margin: .35rem 0 .6rem; padding: .5rem .75rem;
+          background: var(--warn-ground); border-radius: var(--radius); font-size: .9375rem; overflow-wrap: anywhere; }
+.a-held .chip { flex: none; background: var(--ink); color: var(--paper); }
+.a-held a { color: var(--ink); font-weight: 500; }
+.a-rows { display: grid; grid-template-columns: minmax(10rem, max-content) max-content minmax(0, 1fr) max-content;
+          gap: 0 1rem; align-items: baseline; }
+.a-row { display: contents; }
+.a-row > * { padding: .3rem 0; border-top: 1px solid color-mix(in oklab, var(--line) 55%, transparent); }
+.a-row:first-child > * { border-top: 0; }
+.a-row .alias, .a-row .what { font-weight: 500; }
+.a-row .what { max-width: 24rem; }                  /* a notice's sentence wraps, leaving its detail room */
+.a-row .why, .a-row .when { color: var(--muted); font-size: .875rem; overflow-wrap: anywhere; }
+/* One line a model: the provider's words cut to the column, whole on hover; its link stays whole. */
+.a-row .why:has(> .said) { display: flex; gap: .5rem; min-width: 0; }
+.a-row .said { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.a-row .why > a { flex: none; }
+.a-row .acts { display: flex; gap: .9rem; font-size: .875rem; }
+.linkish.danger { color: var(--err); }
+.a-retired { margin-top: .75rem; border-top: 1px solid var(--line); padding-top: .5rem; }
+.a-retired > summary { cursor: pointer; font-weight: 500; padding: .35rem 0; }
+.a-retired > summary .muted { color: var(--muted); font-weight: 400; }
+.a-none { color: var(--muted); }
 
 /* pages: a grouped sidebar (Observe, Gateway, Control); the group names and
    the active page carry the theme's brand colour. A theme with a floor backs
@@ -156,6 +180,12 @@ main { max-width: var(--page, 104rem); margin: 0 auto; padding: 0 var(--gutter)
   content: ""; position: absolute; left: calc(100% + .3em); top: -.05em; width: .75em; height: .75em;
   background: var(--brand);
   clip-path: polygon(50% 0, 62% 38%, 100% 50%, 62% 62%, 50% 100%, 38% 62%, 0 50%, 38% 38%); }
+/* A page's count (Action needed): hung outside the button like the sparkle,
+   so the active page's stroke runs under the word and not the number. */
+.tabs button:has(> .n) { position: relative; }
+.tabs button .n { position: absolute; left: calc(100% + .4rem); top: 50%; transform: translateY(-50%);
+                  padding: .05rem .4rem; border-radius: 999px; font: 700 .75rem/1.25 var(--font-mono);
+                  background: var(--err-ground); color: var(--err); }
 .tab { display: none; } .tab.active { display: block; }
 /* One surface for the page's content: its panels are sections of it,
    divided by a hairline, rather than separate boxes with gaps between. A
@@ -170,6 +200,7 @@ main { max-width: var(--page, 104rem); margin: 0 auto; padding: 0 var(--gutter)
   .tabs { position: static; flex-direction: row; flex-wrap: wrap; gap: .5rem 1.5rem; }
   .tabs .group { flex-direction: row; flex-wrap: wrap; align-items: baseline; gap: .1rem 1rem; }
   .tabs h2 { margin: 0; }
+  .tabs button:has(> .n) { margin-right: 2rem; }    /* room for its count in a row */
 }
 
 /* settings, connections, logs, info */
@@ -510,6 +541,12 @@ dialog .pick { display: flex; gap: 1.5rem; }
   .toolbar input[type=search] { min-width: 0; width: 100%; }
   button.c-row { grid-template-columns: minmax(0, 1fr); }     /* the seats get the width; the whole row opens */
   .c-row .open-hint { display: none; }
+  .a-line { margin: 1rem 1rem 0; }
+  .a-rows { grid-template-columns: minmax(0, 1fr) max-content; }    /* name and state; the words and actions under them */
+  .a-row .why, .a-row .acts { grid-column: 1 / -1; padding-top: 0; border-top: 0; }
+  .a-row .acts { padding-bottom: .5rem; }
+  .a-row .why:has(> .said) { flex-wrap: wrap; }
+  .a-row .said { white-space: normal; }             /* a phone has the height to read it whole */
 }
 @media (prefers-reduced-motion: reduce) {
   * { transition: none !important; }

package/ui/bench.js

@@ -123,26 +123,10 @@
     c.addEventListener(c.tagName === 'SELECT' ? 'change' : 'input', function () { applyFilter(scope); });
   });
 
-  // ---- Action needed: Hide folds it to its heading, per browser, until an
-  // action it has not seen appears or 12 hours pass, whichever is first ----
-  var acts = $('actions');
-  if (acts) {
-    var keys = Array.prototype.map.call(acts.querySelectorAll('.problem[data-key]'), function (p) { return p.dataset.key; });
-    var hid = null;
-    try { hid = JSON.parse(store('bench.actions.hidden') || 'null'); } catch (err) { hid = null; }
-    var fold = function (on) {
-      acts.classList.toggle('collapsed', on);
-      $('actions-hide').textContent = on ? 'show' : 'Hide';
-      $('actions-hide').setAttribute('aria-expanded', String(!on));
-    };
-    fold(!!(hid && hid.keys && Date.now() - hid.at < 12 * 3600 * 1000
-            && keys.every(function (k) { return hid.keys.indexOf(k) !== -1; })));
-    $('actions-hide').addEventListener('click', function () {
-      var on = !acts.classList.contains('collapsed');
-      store('bench.actions.hidden', on ? JSON.stringify({at: Date.now(), keys: keys}) : '');
-      fold(on);
-    });
-  }
+  // ---- a link to another page (At a glance's line to Action needed) ----
+  document.querySelectorAll('[data-go]').forEach(function (b) {
+    b.addEventListener('click', function () { show(b.dataset.go); scrollTo(0, 0); });
+  });
 
   // ---- retired rows: hidden unless asked for ---------------------------
   var showRetired = store('bench.retired') === '1';
@@ -409,8 +393,7 @@
   document.querySelectorAll('[data-act]').forEach(function (b) {
     b.addEventListener('click', function () {
       var act = b.dataset.act, aliases = (b.dataset.aliases || b.dataset.alias || '').split(' ').filter(Boolean);
-      var status = b.closest('.problem').querySelector('.why');
-      if (act === 'open') { openModel(aliases[0]); return; }
+      var status = b.closest('[data-row]').querySelector('.why');   // a model's line, or its group's head
       if (act === 'dismiss') {               // a notice: gone once seen
         b.disabled = true;
         post('/api/alerts/dismiss', {key: b.dataset.key}).then(function (res) {
@@ -419,12 +402,22 @@
         }).catch(function (err) { status.textContent = String(err); b.disabled = false; });
         return;
       }
-      if (act === 'retry') {                 // no confirm: one call, and only an answer changes anything
+      if (act === 'retry') {                 // no confirm: one call each, and only an answer changes anything
+        var label = b.textContent, said = [], failed = [];
         b.disabled = true; b.textContent = 'Asking…';
-        post('/api/retry', {alias: aliases[0]}).then(function (res) {
-          if (!res.ok) { status.textContent = res.errors.join('; '); status.classList.add('err'); b.disabled = false; b.textContent = 'Retry'; return; }
-          status.textContent = res.changed.join(' · ') + '. Reloading…'; setTimeout(function () { location.reload(); }, 900);
-        }).catch(function (err) { status.textContent = String(err); b.disabled = false; b.textContent = 'Retry'; });
+        aliases.reduce(function (chain, alias) {
+          return chain.then(function () {
+            return post('/api/retry', {alias: alias}).then(function (res) {
+              (res.ok ? said : failed).push(res.ok ? res.changed.join(' · ') : res.errors.join('; '));
+            });
+          });
+        }, Promise.resolve()).then(function () {
+          // A group's Retry all says how many answered; each failure is on hover, not ten lines in its heading.
+          var told = aliases.length > 1 ? said.length + ' of ' + aliases.length + ' answered' : said.concat(failed).join(' · ');
+          status.title = failed.join('\n');
+          if (!said.length) { status.textContent = told; status.classList.add('err'); b.disabled = false; b.textContent = label; return; }
+          status.textContent = told + '. Reloading…'; setTimeout(function () { location.reload(); }, 900);
+        }).catch(function (err) { status.textContent = String(err); b.disabled = false; b.textContent = label; });
         return;
       }
       var verb = act === 'restore' ? 'Restore' : 'Retire';