@wendellresearch-ai/benchwarmers 0.4.0 → 0.5.0

package/GATEWAY.md

@@ -389,7 +389,7 @@ the panels' verdicts. A verdict on a conversation is optional
 (`POST /bench/verdict` with that call id); the offload alone already counts
 toward rotation. The Gateway card lists route conversations in their own
 table (when, who asked, the first message, who answered with ✓/✗, turns,
-tool calls, failovers) and none of them adds to **Owed**.
+tool calls, failovers) and none of them counts as a verdict owed.
 
 **Shadows (later, opt-in).** A shadow seat would answer the same turn in the
 background for evaluation only, after the caller has its answer. The place
@@ -515,9 +515,9 @@ key limited to its collection and a rate, or keep the key on the app's server.
 | Shape picker | Optional decision model (above). |
 | Call log | `runs/gateway/calls.jsonl`, gitignored: every seat's answer, status, timing and call id. |
 | Evidence | Each seat is recorded in `metrics/offloads.jsonl` as an unjudged offload when it answers; `offload.py verdict <call id>` or `POST /bench/verdict` replaces it. |
-| Collections tab | The collections and their form, then the gateway's recent requests: every seat's status and verdict, and how many verdicts are owed. Clicking a seat opens its answer and the verdict form. |
-| Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily (scheduled task WRBench-Catalog); a free model whose price goes above zero raises a "went paid" alert. |
-| Discover tab | `metrics/discover.py`: models.dev and FreeLLMAPI merged into one list of models worth adding, free first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
+| Collections tab | The collections, each drawn as its panel sits, and their form (one row per model, a chip per provider). Then the gateway's recent requests: every seat's status and verdict, how long the caller waited (Time), and how many verdicts are owed. Clicking a seat opens its answer and the verdict form. |
+| Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily by the built-in scheduler; a free model whose price goes above zero raises a "went paid" alert. |
+| Discover tab | `metrics/discover.py`: FreeLLMAPI (leading) and models.dev merged into one list of models worth adding, free first, FreeLLMAPI's smartest first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
 | Merged replies | Opt-in: one merger answer as `content`, every seat still in `bench` (The reply). |
 | MCP server | `mcp_server.py`: the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`. Two transports for one `answer()`: `POST /mcp` (or `/v1/mcp`) on either port, streamable HTTP with a JSON reply per request, its tools calling the server's own routes over loopback with the caller's credential and Host; and stdio (`benchwarmers mcp`), a caller like any other over HTTP (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`). The Claude Code plugin in `plugin/` connects to `/mcp` and adds a command, a skill and a `Stop` hook (`plugin/hooks/hooks.json`, an `mcp_tool` hook) that calls `pending_verdicts` with the session's `transcript_path`: the call ids in the transcript (Claude Code keeps no `Mcp-Session-Id`, so the transcript is the record of whose panels they were), of which the call log says which are answered seats with no verdict (`POST /bench/pending`), returned once as `additionalContext`. A server on another machine cannot read the transcript, so there the hook says nothing. A tool call answers as SSE with progress notifications, which reset Claude Code's 300-second idle limit. |
 
@@ -529,7 +529,11 @@ key limited to its collection and a rate, or keep the key on the app's server.
 - **Every seat is named** in the response, including ones that failed or came
   back empty, because who answered is itself evidence about the roster.
 - **Free models only**, so rate limits are the constraint; a failed seat is
-  reported and logged, not retried into the limit.
+  reported and logged, not retried into the limit. A model the bench already
+  knows has no room (a limit or quota it has counted, `metrics/pacing.py`) is
+  not seated at all: an explorer seat goes to a model with room now, a member
+  is named as not asked when its wait would outlast the seat, and a request
+  with no seat left gets a 429.
 - **Privacy:** prompts go only to providers the bench already uses.
 - **Bind to 127.0.0.1**, as the dashboard does. Elsewhere only with an admin
   token set, and then every call carries a gateway key made on the

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 23 October 2026.*
+*Effective 29 October 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -13,10 +13,16 @@ Wendell Research.
   That includes what the gateway relays for your apps, and the calls the
   probes, the suite and onboarding make. Each endpoint's own privacy policy
   applies to what it receives; some free tiers keep prompts or train on them.
-- **Catalog reads**, carrying nothing of yours: models.dev and FreeLLMAPI
-  (at most once a day) and OpenRouter's public model list.
+- **Catalog reads**, carrying nothing of yours: FreeLLMAPI (its feed and its
+  site's sitemap) and models.dev (at most once a day) and OpenRouter's public
+  model list.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page, in
   your browser, and the provider's token endpoint.
+- **Quota reads**, only when you press Refresh on Quota management, carrying
+  no prompt:
+  - OpenRouter's key status, asked with your key.
+  - For CLI Proxy API's sign-ins, through the proxy, the quota endpoint of the
+    provider each sign-in belongs to (Google's, ChatGPT's).
 - **Charts**: the dashboard's page loads Observable Plot from jsDelivr, at a
   pinned version, in your browser.
 - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows

package/README.md

@@ -160,8 +160,9 @@ dependencies either way: the app is the Python standard library.
 4. **Onboard a few models** from it ("Onboard a model"): tick one or several;
    each is registered, probed and run through the suite, one at a time, with
    the progress beside the button. **Quick** probes now and leaves the suite
-   to the scheduled passes; **Quick + ad hoc** also keeps the bench from ever
-   calling the model on its own, for a metered or trial key. Tick **Vision
+   to the scheduled passes; **+ ad hoc** (with Full or Quick) keeps the bench
+   from ever calling the model on its own afterwards, for a metered or trial
+   key. Tick **Vision
    model** for one that reads images and little else: it sits only for
    questions about images, and is tested on the vision tasks, not the code ones.
 
@@ -194,10 +195,12 @@ dependencies either way: the app is the Python standard library.
   (requests a minute, day or month; tokens a minute or day). They are kept per
   connection, since models behind one key share them, and enforced: a call
   waits when the wait fits, or is refused before it goes out. Every call lands
-  in a monthly usage ledger with its tokens, and the Quotas page shows what is
-  left and when it resets.
-- **Discover.** models.dev and FreeLLMAPI, merged into one list of models
-  worth adding, each with the routes this install has to it.
+  in a monthly usage ledger with its tokens. Quota management has a tile per
+  provider showing what is left, when it resets and who is resting. OpenRouter
+  and CLI Proxy API's sign-ins are read live, at no cost to quota.
+- **Discover.** FreeLLMAPI's free catalog, with models.dev for the details
+  and paid models it lacks, merged into one list of models worth adding,
+  smartest first, each with the routes this install has to it.
 
 ## Asking a panel
 
@@ -248,6 +251,7 @@ without it: ask and judge, and run the bench itself. It is a client of a
 running server, so the same commands work against one on another machine.
 
     bench status                                   what is running, and where
+    bench restart                                  start it again on the code now on disk
     bench collections                              what you can ask
     bench ask core "Where can this return a stale value?" -f cache.py
     git diff | bench ask core "Review this diff"   a pipe is the prompt
@@ -432,9 +436,13 @@ reaches Wendell Research. What it sends, and where:
   a hub such as Open WebUI, or a local runtime. That includes the probes, the
   suite and onboarding, which spend calls on your keys. Add only endpoints you
   are content to send that work to.
-- **Catalog reads**, no key and nothing of yours: models.dev and FreeLLMAPI
-  once a day, and OpenRouter's public model list.
+- **Catalog reads**, no key and nothing of yours: FreeLLMAPI (its feed and
+  its site's sitemap) and models.dev once a day, and OpenRouter's public
+  model list.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page.
+- **Quota reads**, only when you press Refresh on Quota management, with no
+  prompt: OpenRouter's key status, with your key. CLI Proxy API asks each
+  sign-in's own provider for its quota.
 - **Charts**: the dashboard's browser page loads Observable Plot from jsDelivr
   at a pinned version.
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the

package/USAGE.md

@@ -14,6 +14,7 @@ coding agent doing it with you); the gateway's whole contract is in
 - [Connecting an app](#connecting-an-app)
 - [Recording verdicts](#recording-verdicts)
 - [Discover: finding models to add](#discover-finding-models-to-add)
+- [Quota management](#quota-management)
 - [The scheduler](#the-scheduler)
 - [Your data folder](#your-data-folder)
 
@@ -25,18 +26,25 @@ coding agent doing it with you); the gateway's whole contract is in
 | --- | --- | --- |
 | Observe | **At a glance** | Every model on the bench: whether it works right now, its role, and its connection. Click a name for its popup: role, alias, facts, options, retire. |
 | | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, and speed. |
-| | **Quota management** | What limits each connection, and which models are resting after hitting one. |
+| | **Quota management** | A tile per provider: what is left, when it resets, and which of its models are resting. Then the week's resets, and what the bench sent. |
 | | **Logs** | Gateway calls (who sat and how it went, never the prompt), changes made from the dashboard, the scheduler, and onboarding runs. |
 | Gateway | **Endpoints** | Where models are served from. Add an endpoint (a provider's sign-in, or any OpenAI-compatible API key), change a key, and see what each one serves. Also your own app registrations and rate limits per connection. |
 | | **Models** | Every model on the bench, by the connection it comes from. |
 | | **Collections** | The named panels your apps ask for, and the gateway's settings. |
 | | **Discover** | Models worth adding, from public catalogs, with the ways this install can reach each one. |
-| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, and how the install is reached (shown only). |
+| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, how the install is reached (shown only), and Restart. |
 | | **Info** | Version, Python, data folder and counts. |
 
 "Action needed", above the pages, lists what will not fix itself, such as a
-model about to be retired, with Retry, Open and Retire now beside it. Hide
-folds it away until something new turns up.
+model about to be retired, naming each model and where it comes from, with
+Retry, Open and Retire now beside it. Hide folds it away until something new
+turns up; an alert whose models have all been retired goes at once.
+
+**Restart** (Settings, Server and access), or `bench restart`, starts the
+dashboard again on the code now on disk, after an update. It waits for
+onboarding, a scheduled job and gateway calls in progress (`--force` cuts them
+off), and Settings says when newer code is on disk. It needs no service
+manager: `serve.py` restarts its own dashboard.
 
 ## Onboarding models
 
@@ -44,18 +52,22 @@ folds it away until something new turns up.
 not on the bench yet. Search, and tick one or several: the ticks survive a new
 search, so you can gather models from different connections in one go.
 Dismiss hides one you never want offered (an image model, say); the dismissed
-list brings it back.
+list brings it back. Each endpoint's list is kept an hour, with its age and a
+Refresh, and a model already queued or onboarding is not offered again.
 
 Each model is registered, probed with four short calls, and, if it answers,
 run through the suite. **Onboard as**, under the list, chooses how much that
 spends:
 
 - **Full**: the probe, then the suite now.
+- **Full + ad hoc**: the probe and the suite now, once, so routing has scores
+  for it; after that the bench never calls the model on its own (no explorer
+  seat, no scheduled runs). A collection that lists it still calls it.
 - **Quick**: the probe now; the scheduled suite passes fill the suite in, six
   tasks a run. The same calls, spread out.
-- **Quick + ad hoc**: the probe only, and the bench never calls the model on
-  its own (no explorer seat, no scheduled runs). The one that saves calls, for
-  a metered, capped or trial key. A collection that lists it still calls it.
+- **Quick + ad hoc**: the probe only, and then ad hoc as above. The one that
+  spends least, for a metered, capped or trial key; its suite stays empty,
+  since the scheduled passes leave ad hoc models alone.
 
 **Vision model**, the box beside it, gives the model role **vision** as it is
 registered: it sits only for requests that carry images, and its suite is the
@@ -64,7 +76,7 @@ key) rather than the text and code ones. Full runs them now, Quick through the
 scheduled passes. For a model that reads images and little else; a model that
 does everything and also takes images stays general and gets both.
 
-`bench onboard <served> [--quick [--ad-hoc]] [--vision]` does the same from a terminal.
+`bench onboard <served> [--quick] [--ad-hoc] [--vision]` does the same from a terminal.
 Onboarding takes minutes and spends calls, so models run **one at a time**,
 in a queue the dashboard keeps:
 
@@ -91,6 +103,15 @@ everything else well stays `general`.
 A collection is a named panel: the models it always includes, the shape of
 work it is for (review, design, enumeration, ...), and how its reply comes
 back. On the **Collections** page, **New collection**, then pick the members.
+Each collection is drawn as its panel sits: a seat per member, the explorer
+seats dashed, and the merger after an arrow.
+
+In the popup, the panel's seats are on top, and the member list has one row
+per model with its context, vision and tools, and price. A model served by
+several providers is still one row, with a chip per provider: tick the chip
+of the route you want. Ticking two is allowed, and the count says so. The tags
+grey out the models that do not fit them, as before.
+
 Each call also seats explorers, the models unused longest, up to the gateway's
 limits (Gateway settings, on the same page: how many calls a request may make,
 and how many of them explore).
@@ -191,17 +212,55 @@ are often wrong together.
 
 ## Discover: finding models to add
 
-**Discover** merges models.dev and FreeLLMAPI into one list, free models
-first, with what is new this week badged. A row opens its popup: what the
-model takes and does, its price, and every route this install has to it, each
-with an **Add via this**. Adding through a hub (Open WebUI, LiteLLM, CLI Proxy
+**Discover** leads with FreeLLMAPI's catalog of free tiers and fills in from
+models.dev, which alone has prices, paid models and providers' base URLs. One
+list: free models first, then by reach, and within each reach FreeLLMAPI's
+smartest first (the **Rank** column), then what it does not rank, what is new
+this week first and badged. A row opens its popup: what
+the model takes and does, its limits and monthly budget, FreeLLMAPI's known
+quirks for it, links to its freellmapi.co page and models.dev, and every route
+this install has to it, each with an **Add via this**. Adding through a hub (Open WebUI, LiteLLM, CLI Proxy
 API) puts it in that hub too; adding direct registers it on its own endpoint
-only. Either way the onboarding joins the same queue.
+only. Either way the onboarding joins the same queue, with the same
+**Onboard as** and **Vision model** choice as the Onboard dialog, asked in the
+confirm before anything is added.
 
 Some providers' free models work only inside their own app; Discover leaves
 those out and keeps their paid models. Settings, **Free only in their own
 app**, switches a provider either way or adds one.
 
+## Quota management
+
+One tile per provider a bench model arrives through, shaped by how that
+provider limits:
+- **Windows**: CLI Proxy API's sign-ins have a 5-hour and a weekly limit per
+  model group. Other providers have the windows their rate-limit headers state.
+- **Pool**: OpenRouter's free requests a day, shared by its models, with a bar
+  per key.
+- **Each model's last day**: shown where the provider states nothing.
+
+Each tile shows when its limits reset and which of its models are resting
+after a rate limit. **Models** lists the provider's bench models, whether each
+is free to call now, and its last 24 hours of calls.
+
+**Refresh never spends quota.** On OpenRouter it asks the key what is left. On
+CLI Proxy API's sign-ins it asks the proxy for each one's quota, as its own
+console does; this needs CLI Proxy API on the Endpoints page as a hub, with
+its management key. Every other tile re-reads what the provider last said in
+its replies to the bench. **Check now** sends one 16-token request to read the
+reply's limit headers. It is offered only where the provider has sent them
+before.
+
+Tiles sort with the ones that are out first, then the soonest reset. Drag a
+tile, or move it with the arrow keys, and the order is kept; **Sort** brings
+the default order back.
+
+Under the tiles are:
+- the next 7 days' resets, and when each resting model is back
+- **What the bench sent**: requests per provider this month and in the last 24
+  hours, as a chart with its table
+- the known limits, folded away
+
 ## The scheduler
 
 The recurring jobs (the morning availability probe, the catalog refresh, the

package/VERSION

@@ -1 +1 @@
-0.4.0
+0.5.0

package/cli.py

@@ -258,9 +258,7 @@ def cmd_onboard(args) -> None:
         reply = call("GET", "/api/onboard-queue") if verb == "status" else \
             call("POST", f"/api/onboard-queue/{verb}", {})
     else:
-        if args.ad_hoc and not args.quick:
-            sys.exit("--ad-hoc goes with --quick: an ad hoc model gets no suite run of its own")
-        mode = "quick-adhoc" if args.ad_hoc else "quick" if args.quick else "full"
+        mode = ("quick" if args.quick else "full") + ("-adhoc" if args.ad_hoc else "")
         items = [{"served": s, "endpoint": args.endpoint or "", "mode": mode, "vision": args.vision}
                  for s in args.served]
         reply = call("POST", "/api/onboard-queue", {"items": items})
@@ -385,6 +383,13 @@ def cmd_verdict(args) -> None:
                                        f" {len(body['real'])} real, {len(body['false'])} false"))
 
 
+def cmd_restart(args) -> None:
+    """serve.restart: refused while work is in progress, unless --force."""
+    reply = call("POST", "/api/restart", {"force": args.force})
+    out(args, reply, lambda r: done(r, "restarting on the code now on disk; back in a few seconds"
+                                       + (f" (cut off: {'; '.join(r['cut'])})" if r.get("cut") else "")))
+
+
 def cmd_endpoints(args) -> None:
     reply = call("GET", "/api/endpoints")
     out(args, reply, lambda r: table(r["endpoints"], ["name", "base", "kind", "hub", "default", "key_set", "models"],
@@ -547,6 +552,9 @@ def parser() -> argparse.ArgumentParser:
         return act
 
     cmd("status", cmd_status, "what is running, and where")
+    s = cmd("restart", cmd_restart, "start the dashboard again on the code now on disk")
+    s.add_argument("--force", action="store_true",
+                   help="even while onboarding, a scheduled job or gateway calls are in progress (cuts them off)")
 
     s = cmd("ask", cmd_ask, "ask a collection: every seat answers")
     s.add_argument("collection", help="a collection, e.g. core (bench collections)")
@@ -598,7 +606,8 @@ def parser() -> argparse.ArgumentParser:
     s.add_argument("--quick", action="store_true",
                    help="probe only; the scheduled suite passes fill the suite in later")
     s.add_argument("--ad-hoc", action="store_true",
-                   help="with --quick: the bench never calls it on its own (metered, capped or trial keys)")
+                   help="the bench never calls it on its own afterwards (metered, capped or trial keys);"
+                        " without --quick the suite still runs once now")
     s.add_argument("--vision", action="store_true",
                    help="role vision: a seat only for questions about images, and its suite is the vision tasks")
 

package/gateway.py

@@ -330,6 +330,22 @@ def retired_why(entry: dict) -> str:
             " (Restore in its model popup brings it back)")
 
 
+def refused_now(aliases, est_tokens: int, within: float) -> dict[str, str]:
+    """alias -> why, for models whose known rate limits or quota leave no room
+    for a call within `within` seconds (metrics/pacing.held): asked, they
+    would only bring back the bench's own refusal, so they are not seated."""
+    import pacing
+    out = {}
+    for a in aliases:
+        try:
+            h = pacing.held(a, route.connection_of(a), est_tokens)
+        except Exception:                              # noqa: BLE001 -- a broken limit file is not a refusal
+            continue
+        if h and h[0] > within:
+            out[a] = f"held by a rate limit: {h[1]}; room in {pacing._wait_text(h[0])}"
+    return out
+
+
 def retiring() -> set[str]:
     """Aliases not to spend an explorer seat on: those on a retiring alert
     (failing their probes; 2026-10-16 both explorers of a bench/core call were
@@ -407,6 +423,10 @@ class TooLarge(ValueError):
     """The prompt does not fit a member's context window (HTTP 413)."""
 
 
+class AllHeld(LookupError):
+    """Every seat is held by a known rate limit (HTTP 429): try again later."""
+
+
 def estimate_tokens(text: str) -> int:
     """Characters / 4 plus 10%: tokenizers differ by family, so this errs high.
     The windows it is compared with are corrected by learned 'maximum context
@@ -590,6 +610,13 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
                        f" windows) in bench/{name}. Trim it under {min(small.values()):,} tokens,"
                        " or choose a collection whose members have larger windows"
                        " (GET /v1/models lists each collection's context_window).")
+    # A MODEL THE BENCH KNOWS WILL BE REFUSED IS NOT ASKED: a member or a
+    # `with` whose limits have no room before its seat would give up, named
+    # like the others; it would only come back as the refusal.
+    held = refused_now([*sitting, *nominated], needed, SEAT_TIMEOUT)
+    skipped += [{"seat": "member" if a in sitting else "with", "alias": a, "why": why} for a, why in held.items()]
+    sitting = [a for a in sitting if a not in held]
+    nominated = [a for a in nominated if a not in held]
     if pictures and not collection.get("shape"):
         shape, how = "vision", "the request carries images"
     elif not n_explorers:
@@ -603,6 +630,8 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
         # `with` in the request body is route.py's --with: an explorer seat.
         from run import _registered_names
         avoid = retiring() | set(too_small(needed, _registered_names()))   # explorers must fit too
+        # and have room now: an explorer seat can go to a model without a wait.
+        avoid |= set(refused_now(_registered_names(), needed, 0))
         if pictures:
             avoid |= {a for a in _registered_names() if cannot_see(a)}    # and see
         p = route.panel(shape, large=len(whole) > LARGE, nominated=nominated or None,
@@ -615,8 +644,9 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
                          + ", and no model that takes images was free for an explorer seat. Add one to the"
                          " collection, or call one whose together.capabilities on GET /v1/models lists vision.")
     if not seats:
-        raise LookupError(f"collection {name!r} has no members and no explorer could be seated"
-                          + "".join(f"; {k['alias']} not asked: {k['why']}" for k in skipped))
+        raise (AllHeld if held else LookupError)(
+            f"collection {name!r} has no members and no explorer could be seated"
+            + "".join(f"; {k['alias']} not asked: {k['why']}" for k in skipped))
     panel_id = f"p{dt.datetime.now():%Y%m%d-%H%M%S}-{secrets.token_hex(2)}"
     model = bench_collections.PREFIX + name
     task = " ".join(prompt.split())[:90] + f" ({model}, {shape})"
@@ -718,6 +748,8 @@ def refusal(exc: Exception) -> tuple[int, str]:
         return 413, str(exc)
     if isinstance(exc, ValueError):
         return 400, str(exc)
+    if isinstance(exc, AllHeld):
+        return 429, str(exc)
     if isinstance(exc, LookupError):
         return 503, str(exc)
     return 500, f"the panel failed ({type(exc).__name__}: {exc})"

package/launcher.py

@@ -22,7 +22,7 @@ ROOT = Path(__file__).resolve().parent                 # the app: a checkout, th
 SERVERS = {"serve": "serve.py", "gateway": "gateway.py", "mcp": "mcp_server.py",
            "init": str(Path("scripts") / "init.py")}
 # Everything but running a server is cli.py, a client of a running server.
-CLIENT = ("status", "ask", "verdict", "models", "model", "candidates", "onboard", "dismiss",
+CLIENT = ("status", "restart", "ask", "verdict", "models", "model", "candidates", "onboard", "dismiss",
           "collections", "collection", "endpoints", "endpoint", "keys", "key", "settings")
 
 
@@ -57,6 +57,7 @@ def usage() -> str:
         "",
         "Using it (a client of a running server, BENCHWARMERS_URL; --help on any):",
         "  status                          what is running, and where",
+        "  restart [--force]               start the dashboard again on the code now on disk",
         "  ask <collection> [prompt]       ask a collection; -f FILE adds a file, a pipe reads stdin",
         "  verdict <call> --useful|--not-useful [--real ...] [--false ...]",
         "  models | model <action>         list, rename, role, retire, restore, max-tokens, ...",

package/metrics/discover.py

@@ -1,10 +1,16 @@
-"""One list of models worth adding, from models.dev and FreeLLMAPI, deduped.
+"""One list of models worth adding, from FreeLLMAPI and models.dev, deduped.
 
     python metrics/discover.py           # summary: free models by reach, new this week
 
 The shopping list: Discover's Add (serve.py) puts a model on the bench by one
 of the routes below. A model offered by the same provider in both sources
-appears once, carrying both sources and the better-filled fields.
+appears once, carrying both sources. FreeLLMAPI leads where it lists the
+model (its name, its provider's name, context, tools, limits, its smartest
+and fastest ranks, size, monthly budget, quirks and freellmapi.co page);
+models.dev fills the rest, and alone has the paid models and prices, API base
+URLs (reach, direct routes), modalities, release and knowledge dates.
+Image and audio generation in FreeLLMAPI's list are left out: the bench asks
+for text.
 
 FREE
 ----
@@ -149,7 +155,7 @@ def hub_routes(rows: list[dict]) -> tuple[list[list[dict]], list[str]]:
                 mine.append({"via": "hub", "hub": which, "connection": conn, "endpoint": ep,
                              "label": f"{r['provider_name']} via {label} ({conn} connection)",
                              "note": f"Also serves it to everything else that uses {label}.",
-                             "steps": ["; ".join(said), "The bench: register, a 4-call probe, then the suite"],
+                             "steps": ["; ".join(said), "The bench: register it and onboard it as chosen below"],
                              "on_bench_as": next((a for e, _c, _n, a, s in bench if e == ep and s == served), None)})
         out.append(mine)
     return out, errors
@@ -158,6 +164,8 @@ def hub_routes(rows: list[dict]) -> tuple[list[list[dict]], list[str]]:
 def build() -> list[dict]:
     import endpoints
     md, fl = sources.cached("modelsdev"), sources.cached("freellmapi")
+    pages = set(sources.cached("freellmapi-pages").get("slugs") or [])
+    platform_names = {p.get("id"): p.get("name") for p in fl.get("platforms") or []}
     md_conn = {v: k for k, v in sources.connection_to_modelsdev().items()}
     bench = _on_bench()
     # Open WebUI's connections, as far as the bench can tell without asking it.
@@ -185,31 +193,37 @@ def build() -> list[dict]:
                 "vision": "image" in ((m.get("modalities") or {}).get("input") or []),
                 "released": m.get("release_date"), "knowledge": m.get("knowledge"),
                 "open_weights": m.get("open_weights"), "api": p.get("api"),
-                "openai_like": openai_like, "connection": md_conn.get(prov), "limits": None}
+                "openai_like": openai_like, "connection": md_conn.get(prov), "limits": None,
+                "rank": None, "speed": None, "size": None, "budget": None, "quirks": [], "page": None}
 
     for m in fl.get("models") or []:
+        if (m.get("modality") or "chat") != "chat":
+            continue                                # image or audio generation
         plat = m.get("platform", "")
         prov = sources.FREELLMAPI_TO_MODELSDEV.get(plat, plat)
         key = (prov, sources.norm_model(m.get("modelId", "")))
-        lim = {k: v for k, v in (m.get("limits") or {}).items() if v} or None
+        lead = {"sources": ["freellmapi"], "free": True,          # listed in a free-tier catalog
+                "name": sources.freellmapi_name(m.get("displayName") or "") or m.get("modelId"),
+                "provider_name": platform_names.get(plat) or prov,
+                "limits": {k: v for k, v in (m.get("limits") or {}).items() if v} or None,
+                "rank": m.get("intelligenceRank"), "speed": m.get("speedRank"),
+                "size": m.get("sizeLabel") or None, "budget": m.get("monthlyTokenBudget") or None,
+                "quirks": [{k: q.get(k) for k in ("title", "body", "severity")} for q in m.get("quirks") or []],
+                "page": sources.freellmapi_page(m.get("displayName") or "", pages)}
         row = rows.get(key)
         if row:
-            row["sources"].append("freellmapi")
-            row["free"] = True                      # listed in a free-tier catalog
-            row["limits"] = lim
-            row["context"] = row["context"] or m.get("contextWindow")
-            row["tools"] = row["tools"] if row["tools"] is not None else m.get("supportsTools")
+            row.update(lead, sources=list(dict.fromkeys(["freellmapi", *row["sources"]])),
+                       context=m.get("contextWindow") or row["context"],          # the free tier's, where it says
+                       tools=row["tools"] if m.get("supportsTools") is None else m.get("supportsTools"))
         else:
             rows[key] = {
-                "in_mods": ["text", "image"] if m.get("supportsVision") else ["text"], "out_mods": ["text"],
-                "provider": prov, "provider_name": prov, "model": m.get("modelId"),
-                "name": m.get("displayName") or m.get("modelId"), "sources": ["freellmapi"],
-                "free": True, "cost": None, "context": m.get("contextWindow"), "output": None,
+                **lead, "in_mods": ["text", "image"] if m.get("supportsVision") else ["text"], "out_mods": ["text"],
+                "provider": prov, "model": m.get("modelId"),
+                "cost": None, "context": m.get("contextWindow"), "output": None,
                 "tools": m.get("supportsTools"), "reasoning": None,
                 "vision": m.get("supportsVision"), "released": None, "knowledge": None,
                 "open_weights": None, "api": None,
-                "openai_like": False, "connection": FREELLMAPI_TO_CONNECTION.get(plat),
-                "limits": lim}
+                "openai_like": False, "connection": FREELLMAPI_TO_CONNECTION.get(plat)}
 
     # The same model through ANOTHER provider is fine to add, but worth knowing.
     by_model: dict[str, list[str]] = {}
@@ -245,7 +259,11 @@ def build() -> list[dict]:
         out.append(r)
     _write_seen(seen)
     order = {"connected": 0, "easy to add": 1, "on bench": 2, "other": 3}
-    out.sort(key=lambda r: (not r["free"], not r["new"], order[r["reach"]], r["provider"], r["model"]))
+    # Within each reach, FreeLLMAPI's smartest first; the models it does not rank
+    # after, new ones first. Not new first overall: models.dev adds hundreds a
+    # week, which buried the ranked ones (the badge still marks them).
+    out.sort(key=lambda r: (not r["free"], order[r["reach"]], r["rank"] is None, r["rank"] or 0, not r["new"],
+                            r["provider"], r["model"]))
     return out
 
 
@@ -293,6 +311,38 @@ def _selftest() -> None:
     over = sources.free_in_own_client_only({"opencode": False, "x": True})
     assert over["opencode"] is False and over["x"] is True and over["opencode-go"] is True, over
     assert sources.free_in_own_client_only(True) == sources.free_in_own_client_only({})   # not a map: ignored
+    # freellmapi.co names a page by the display name without its note; only its sitemap's pages exist.
+    assert sources.freellmapi_page("Llama 4 Scout (CF)", {"llama-4-scout"}) == "https://freellmapi.co/model/llama-4-scout"
+    assert sources.freellmapi_page("GLM-4.7 FlashX", {"llama-4-scout"}) is None
+    # FreeLLMAPI leads where it lists a model, models.dev fills the rest; image models out; smartest first.
+    md = {"acme": {"name": "Acme", "api": "https://acme.example/v1", "models": {
+        "acme/big": {"name": "Big", "cost": {"input": 1, "output": 2}, "limit": {"context": 8000},
+                     "modalities": {"input": ["text", "image"], "output": ["text"]}, "release_date": "2026-01-01"},
+        "acme/small": {"name": "Small", "cost": {"input": 0, "output": 0}},
+        "acme/tiny": {"name": "Tiny", "cost": {"input": 0, "output": 0}}}}}
+    fl = {"platforms": [{"id": "acme", "name": "Acme Cloud"}], "models": [
+        {"platform": "acme", "modelId": "acme/big", "displayName": "Big One (Acme)", "intelligenceRank": 3,
+         "speedRank": 9, "sizeLabel": "Large", "monthlyTokenBudget": "~3M", "contextWindow": 4000,
+         "limits": {"rpm": 10, "rpd": None}, "quirks": [{"slug": "q", "title": "T", "body": "B", "severity": "info"}]},
+        {"platform": "acme", "modelId": "acme/painter", "displayName": "Painter", "modality": "image"},
+        {"platform": "acme", "modelId": "acme/tiny", "displayName": "Tiny", "intelligenceRank": 1}]}
+    feeds = {"modelsdev": md, "freellmapi": fl, "freellmapi-pages": {"slugs": ["big-one"]}}
+    import tempfile
+    kept = sources.cached, globals()["_on_bench"], globals()["SEEN"]
+    try:
+        sources.cached = lambda name: feeds.get(name, {})
+        globals().update(_on_bench=lambda: [], SEEN=pathlib.Path(tempfile.mkdtemp()) / "seen.json")
+        rows = [r for r in build() if r["provider"] == "acme"]
+    finally:
+        sources.cached = kept[0]
+        globals().update(_on_bench=kept[1], SEEN=kept[2])
+    big = next(r for r in rows if r["model"] == "acme/big")
+    assert (big["sources"], big["name"], big["provider_name"]) == (["freellmapi", "models.dev"], "Big One", "Acme Cloud"), big
+    assert big["free"] and big["context"] == 4000 and big["limits"] == {"rpm": 10} and big["released"] == "2026-01-01", big
+    assert (big["rank"], big["speed"], big["size"], big["budget"]) == (3, 9, "Large", "~3M"), big
+    assert big["page"] == "https://freellmapi.co/model/big-one" and big["quirks"] == [
+        {"title": "T", "body": "B", "severity": "info"}] and big["vision"], big
+    assert [r["model"] for r in rows] == ["acme/tiny", "acme/big", "acme/small"], [r["model"] for r in rows]
     print("discover selftest ok")
 
 
@@ -306,7 +356,7 @@ def main() -> None:
     by = {}
     for r in free:
         by[r["reach"]] = by.get(r["reach"], 0) + 1
-    print(f"{len(rows)} models ({len(free)} free) from models.dev + FreeLLMAPI;"
+    print(f"{len(rows)} models ({len(free)} free) from FreeLLMAPI + models.dev;"
           f" free by reach: {by}; new this week: {sum(r['new'] for r in rows)}")
     both = sum(1 for r in rows if len(r["sources"]) > 1)
     print(f"{both} listed by both sources (shown once)")

package/metrics/pacing.py

@@ -177,6 +177,28 @@ def _ago(s: float) -> str:
     return "just now" if s < 60 else f"{s / 60:.0f} min ago" if s < 7200 else f"{s / 3600:.0f} h ago"
 
 
+def _since(rs: list[dict], records: list[dict]) -> float:
+    """The oldest ledger line any of these windows or quota records counts."""
+    spans = [AXES[r["axis"]][1] for r in rs]
+    return min([ledger.month_start(time.time()) if s == "month" else time.time() - s for s in spans]
+               + [_ts(q["observed_at"]) for q in records if q.get("observed_at")])
+
+
+def held(alias: str, connection: str, est_tokens: int = 1) -> tuple[float, str] | None:
+    """(seconds until there is room, why) when a call to `alias` would have to
+    wait or be refused now, else None. Read only: no slot is taken. For
+    picking seats, so a model the bench knows will be refused is not seated
+    (gateway.refused_now); `admit` still decides each call."""
+    import quota
+    rs, records = rules(alias, connection), quota.live(connection, alias)
+    if not rs and not records:
+        return None
+    now = time.time()
+    block = _blocked(rs, records, alias, connection, est_tokens, now,
+                     list(ledger.rows(_since(rs, records))), _inflight(now))
+    return None if block is None else (max(block[0] - now, 0.0), block[1])
+
+
 def admit(alias: str, connection: str, est_tokens: int, left: float, sleep=time.sleep) -> tuple[str | None, str]:
     """Wait for room, or refuse. Returns (ticket, refusal): a ticket to pass
     to `release` once the call's ledger line is written (None when no limit
@@ -189,9 +211,7 @@ def admit(alias: str, connection: str, est_tokens: int, left: float, sleep=time.
         records = quota.live(connection, alias)
         if not rs and not records:
             return None, ""
-        spans = [AXES[r["axis"]][1] for r in rs]
-        since = min([ledger.month_start(time.time()) if s == "month" else time.time() - s for s in spans]
-                    + [_ts(q["observed_at"]) for q in records if q.get("observed_at")])
+        since = _since(rs, records)
         with locked.lock(LOCK):
             now = time.time()
             rows = list(ledger.rows(since))