@wendellresearch-ai/benchwarmers 0.3.3 → 0.5.0

package/GATEWAY.md

@@ -389,7 +389,7 @@ the panels' verdicts. A verdict on a conversation is optional
 (`POST /bench/verdict` with that call id); the offload alone already counts
 toward rotation. The Gateway card lists route conversations in their own
 table (when, who asked, the first message, who answered with ✓/✗, turns,
-tool calls, failovers) and none of them adds to **Owed**.
+tool calls, failovers) and none of them counts as a verdict owed.
 
 **Shadows (later, opt-in).** A shadow seat would answer the same turn in the
 background for evaluation only, after the caller has its answer. The place
@@ -515,9 +515,9 @@ key limited to its collection and a rate, or keep the key on the app's server.
 | Shape picker | Optional decision model (above). |
 | Call log | `runs/gateway/calls.jsonl`, gitignored: every seat's answer, status, timing and call id. |
 | Evidence | Each seat is recorded in `metrics/offloads.jsonl` as an unjudged offload when it answers; `offload.py verdict <call id>` or `POST /bench/verdict` replaces it. |
-| Collections tab | The collections and their form, then the gateway's recent requests: every seat's status and verdict, and how many verdicts are owed. Clicking a seat opens its answer and the verdict form. |
-| Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily (scheduled task WRBench-Catalog); a free model whose price goes above zero raises a "went paid" alert. |
-| Discover tab | `metrics/discover.py`: models.dev and FreeLLMAPI merged into one list of models worth adding, free first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
+| Collections tab | The collections, each drawn as its panel sits, and their form (one row per model, a chip per provider). Then the gateway's recent requests: every seat's status and verdict, how long the caller waited (Time), and how many verdicts are owed. Clicking a seat opens its answer and the verdict form. |
+| Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily by the built-in scheduler; a free model whose price goes above zero raises a "went paid" alert. |
+| Discover tab | `metrics/discover.py`: FreeLLMAPI (leading) and models.dev merged into one list of models worth adding, free first, FreeLLMAPI's smartest first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
 | Merged replies | Opt-in: one merger answer as `content`, every seat still in `bench` (The reply). |
 | MCP server | `mcp_server.py`: the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`. Two transports for one `answer()`: `POST /mcp` (or `/v1/mcp`) on either port, streamable HTTP with a JSON reply per request, its tools calling the server's own routes over loopback with the caller's credential and Host; and stdio (`benchwarmers mcp`), a caller like any other over HTTP (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`). The Claude Code plugin in `plugin/` connects to `/mcp` and adds a command, a skill and a `Stop` hook (`plugin/hooks/hooks.json`, an `mcp_tool` hook) that calls `pending_verdicts` with the session's `transcript_path`: the call ids in the transcript (Claude Code keeps no `Mcp-Session-Id`, so the transcript is the record of whose panels they were), of which the call log says which are answered seats with no verdict (`POST /bench/pending`), returned once as `additionalContext`. A server on another machine cannot read the transcript, so there the hook says nothing. A tool call answers as SSE with progress notifications, which reset Claude Code's 300-second idle limit. |
 
@@ -529,7 +529,11 @@ key limited to its collection and a rate, or keep the key on the app's server.
 - **Every seat is named** in the response, including ones that failed or came
   back empty, because who answered is itself evidence about the roster.
 - **Free models only**, so rate limits are the constraint; a failed seat is
-  reported and logged, not retried into the limit.
+  reported and logged, not retried into the limit. A model the bench already
+  knows has no room (a limit or quota it has counted, `metrics/pacing.py`) is
+  not seated at all: an explorer seat goes to a model with room now, a member
+  is named as not asked when its wait would outlast the seat, and a request
+  with no seat left gets a 429.
 - **Privacy:** prompts go only to providers the bench already uses.
 - **Bind to 127.0.0.1**, as the dashboard does. Elsewhere only with an admin
   token set, and then every call carries a gateway key made on the

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 23 October 2026.*
+*Effective 29 October 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -13,10 +13,16 @@ Wendell Research.
   That includes what the gateway relays for your apps, and the calls the
   probes, the suite and onboarding make. Each endpoint's own privacy policy
   applies to what it receives; some free tiers keep prompts or train on them.
-- **Catalog reads**, carrying nothing of yours: models.dev and FreeLLMAPI
-  (at most once a day) and OpenRouter's public model list.
+- **Catalog reads**, carrying nothing of yours: FreeLLMAPI (its feed and its
+  site's sitemap) and models.dev (at most once a day) and OpenRouter's public
+  model list.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page, in
   your browser, and the provider's token endpoint.
+- **Quota reads**, only when you press Refresh on Quota management, carrying
+  no prompt:
+  - OpenRouter's key status, asked with your key.
+  - For CLI Proxy API's sign-ins, through the proxy, the quota endpoint of the
+    provider each sign-in belongs to (Google's, ChatGPT's).
 - **Charts**: the dashboard's page loads Observable Plot from jsDelivr, at a
   pinned version, in your browser.
 - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows

package/README.md

@@ -11,7 +11,13 @@
   <a href="https://pypi.org/project/benchwarmers/"><img src="https://img.shields.io/pypi/v/benchwarmers?color=3775a9&logo=pypi&logoColor=white" alt="PyPI"></a>
   <a href="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml"><img src="https://img.shields.io/github/actions/workflow/status/WendellResearch-AI/benchwarmers/check.yml?branch=main&logo=github&label=CI%20tests" alt="CI tests"></a>
   <a href="https://registry.modelcontextprotocol.io/?q=benchwarmers"><img src="https://img.shields.io/badge/MCP_Registry-111111?logo=modelcontextprotocol&logoColor=white" alt="In the official MCP Registry"></a>
+  <a href="https://smithery.ai/skills/wendellresearch-ai/benchwarmers"><img src="https://img.shields.io/badge/Smithery-FF5601?logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxODggMjExIiBmaWxsPSIjZmZmIj48cGF0aCBkPSJNNDEuNzAxMSA3Ny40OEgwVjEwMC4yNjRDMCAxMTguMTcxIDE0LjUyNzUgMTMyLjY5OCAzMi40MzQyIDEzMi42OThINTUuMjE4M1Y5MC45OTcxQzU1LjIxODMgODMuNTQxOCA0OS4xNTY1IDc3LjQ4IDQxLjcwMTEgNzcuNDhaIi8%2BPHBhdGggZD0iTTY2LjA4NzkgOTAuOTk3MVYxMzIuNjk4SDEwNy43ODlDMTE1LjI0NCAxMzIuNjk4IDEyMS4zMDYgMTI2LjYzNiAxMjEuMzA2IDExOS4xODFWNzcuNDhINzkuNjA1MUM3Mi4xNDk3IDc3LjQ4IDY2LjA4NzkgODMuNTQxOCA2Ni4wODc5IDkwLjk5NzFaIi8%2BPHBhdGggZD0iTTE1NC45OTUgNzcuNDhIMTMyLjIxVjExOS4xODFDMTMyLjIxIDEyNi42MzYgMTM4LjI3MiAxMzIuNjk4IDE0NS43MjggMTMyLjY5OEgxODcuNDI5VjEwNi4xNTJDMTg3LjQyOSA4OC4yNDQ5IDE3Mi45MDEgNzcuNTE0OCAxNTQuOTk1IDc3LjUxNDhWNzcuNDhaIi8%2BPHBhdGggZD0iTTAgNDUuOTUxNFY2Ni42MTAzSDQxLjcwMTFDNDkuMTU2NSA2Ni42MTAzIDU1LjIxODMgNjAuNTQ4NSA1NS4yMTgzIDUzLjA5MzJWMEMyNS44NDk4IDAuMjA5MDI4IDAuMDM0ODM4IDE0LjY2NjggMCA0NS45NTE0WiIvPjxwYXRoIGQ9Ik02Ni4wODc5IDUzLjA5M0M2Ni4wODc5IDYwLjU0ODMgNzIuMTQ5NyA2Ni42MTAxIDc5LjYwNTEgNjYuNjEwMUgxMjEuMzA2VjExLjQ2MTVDMTAwLjkyNiA4LjM2MDkyIDkwLjMwMDMgMS43NDE2OSA2Ni4wODc5IDAuMjQzNjUyVjUzLjA1ODFWNTMuMDkzWiIvPjxwYXRoIGQ9Ik0xNDQuMDkgMTMuMDYzOUMxMzkuNzcgMTMuMDYzOSAxMzUuODM0IDEyLjkyNDYgMTMyLjIxIDEyLjY4MDdWNjYuNjFIMTU0Ljk5NUMxNzIuOTAxIDY2LjYxIDE4Ny40MjkgNTIuMDgyNSAxODcuNDI5IDM0LjE3NThWMC4xNzM4MjhDMTc3LjI5MSA4LjExNjkgMTYyLjkwMyAxMy4wNjM5IDE0NC4wOSAxMy4wNjM5WiIvPjxwYXRoIGQ9Ik0wIDE3Ni4wMDJWMjEwLjAwNEMxMC4xMzc5IDIwMi4wNjEgMjQuNTI2IDE5Ny4xMTQgNDMuMzM4NSAxOTcuMTE0QzQ3LjY1ODQgMTk3LjExNCA1MS41OTUxIDE5Ny4yNTMgNTUuMjE4MyAxOTcuNTMyVjE0My42MDNIMzIuNDM0MkMxNC41Mjc1IDE0My42MDMgMCAxNTguMTMgMCAxNzYuMDM3TDAgMTc2LjAwMloiLz48cGF0aCBkPSJNMTA3Ljc4OSAxNDMuNjAzSDY2LjA4NzlWMTk4LjcxNkM4Ni40NjgxIDIwMS44MTcgOTcuMDkzOCAyMDguNDM2IDEyMS4zMDYgMjA5LjkzNFYxNTcuMTJDMTIxLjMwNiAxNDkuNjY0IDExNS4yNDQgMTQzLjYwMyAxMDcuNzg5IDE0My42MDNaIi8%2BPHBhdGggZD0iTTEzMi4yMSAxNTcuMDg1VjIxMC4xNzhDMTYxLjU0NCAyMDkuOTY5IDE4Ny4zMjQgMTk1LjUxMSAxODcuNDI5IDE2NC4yOTZWMTQzLjU2OEgxNDUuNzI4QzEzOC4yNzIgMTQzLjU2OCAxMzIuMjEgMTQ5LjYzIDEzMi4yMSAxNTcuMDg1WiIvPjwvc3ZnPg%3D%3D" alt="The benchwarmers skill on Smithery"></a>
+  <a href="https://skills.sh/wendellresearch-ai/benchwarmers/benchwarmers"><img src="https://img.shields.io/badge/skills.sh-111111" alt="On skills.sh"></a>
+  <a href="https://glama.ai/mcp/servers/WendellResearch-AI/benchwarmers"><img src="https://glama.ai/mcp/servers/WendellResearch-AI/benchwarmers/badges/score.svg" alt="Benchwarmers MCP server, quality and maintenance score on Glama"></a>
+  <a href="https://cursor.directory/plugins/benchwarmers"><img src="https://img.shields.io/badge/cursor.directory-111111?logo=cursor&logoColor=white" alt="On cursor.directory"></a>
+  <a href="#from-claude-code-and-other-agents"><img src="https://img.shields.io/badge/Antigravity-plugin-3186FF?labelColor=111111&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjEyIDE3IDg2IDgwIj48cGF0aCBkPSJNODkuNyA5My43YzQuNjcgMy41IDExLjY3IDEuMTcgNS4yNS01LjI1Qzc1LjcgNjkuNzggNzkuNzggMTguNDUgNTUuODcgMTguNDVTMzYuMDMgNjkuNzggMTYuNzggODguNDVjLTcgNyAuNTggOC43NSA1LjI1IDUuMjVDNDAuMTIgODEuNDUgMzguOTUgNTkuODYgNTUuODcgNTkuODZTNzEuNjIgODEuNDUgODkuNyA5My43WiIgZmlsbD0iIzMxODZGRiIvPjxtYXNrIGlkPSJtIiBtYXNrVW5pdHM9InVzZXJTcGFjZU9uVXNlIiB4PSIxMyIgeT0iMTgiIHdpZHRoPSI4NSIgaGVpZ2h0PSI3OCI%2BPHBhdGggZD0iTTg5LjcgOTMuN2M0LjY3IDMuNSAxMS42NyAxLjE3IDUuMjUtNS4yNUM3NS43IDY5Ljc4IDc5Ljc4IDE4LjQ1IDU1Ljg3IDE4LjQ1UzM2LjAzIDY5Ljc4IDE2Ljc4IDg4LjQ1Yy03IDcgLjU4IDguNzUgNS4yNSA1LjI1QzQwLjEyIDgxLjQ1IDM4Ljk1IDU5Ljg2IDU1Ljg3IDU5Ljg2UzcxLjYyIDgxLjQ1IDg5LjcgOTMuN1oiIGZpbGw9IiNmZmYiLz48L21hc2s%2BPGcgbWFzaz0idXJsKCNtKSI%2BPGcgZmlsdGVyPSJ1cmwoI2EpIj48ZWxsaXBzZSBjeD0iOTYuNDkiIGN5PSIzNS4xMiIgcng9IjI5LjUiIHJ5PSIzMC4xNSIgdHJhbnNmb3JtPSJyb3RhdGUoNzYuOTIgOTYuNDkgMzUuMTIpIiBmaWxsPSIjRkM0MTNEIi8%2BPC9nPjxnIGZpbHRlcj0idXJsKCNiKSI%2BPGVsbGlwc2UgY3g9IjkuMDMiIGN5PSI0MS42NiIgcng9IjMwLjgzIiByeT0iMzkuOTQiIHRyYW5zZm9ybT0icm90YXRlKDc0LjEzIDkuMDMgNDEuNjYpIiBmaWxsPSIjMDBCOTVDIi8%2BPC9nPjxnIGZpbHRlcj0idXJsKCNjKSI%2BPGVsbGlwc2UgY3g9Ijc1Ljc1IiBjeT0iMTA0LjgyIiByeD0iMjkuMDIiIHJ5PSIyNy45NCIgdHJhbnNmb3JtPSJyb3RhdGUoNzYuOTIgNzUuNzUgMTA0LjgyKSIgZmlsbD0iIzMxODZGRiIvPjwvZz48ZyBmaWx0ZXI9InVybCgjZCkiPjxlbGxpcHNlIGN4PSIzMy41NyIgY3k9IjM1LjQiIHJ4PSIzMy41NyIgcnk9IjM1LjQiIHRyYW5zZm9ybT0ibWF0cml4KC0uNDEgLjkxMiAtLjkxMiAtLjQxIDEwMS4yNSAtMTUuMTcpIiBmaWxsPSIjRkJCQzA0Ii8%2BPC9nPjwvZz48ZGVmcz48ZmlsdGVyIGlkPSJhIiB4PSIyOCIgeT0iLTMzIiB3aWR0aD0iMTM3IiBoZWlnaHQ9IjEzNiIgZmlsdGVyVW5pdHM9InVzZXJTcGFjZU9uVXNlIj48ZmVHYXVzc2lhbkJsdXIgc3RkRGV2aWF0aW9uPSIxOC44Ii8%2BPC9maWx0ZXI%2BPGZpbHRlciBpZD0iYiIgeD0iLTYzIiB5PSItMjMiIHdpZHRoPSIxNDQiIGhlaWdodD0iMTI5IiBmaWx0ZXJVbml0cz0idXNlclNwYWNlT25Vc2UiPjxmZUdhdXNzaWFuQmx1ciBzdGREZXZpYXRpb249IjE2Ii8%2BPC9maWx0ZXI%2BPGZpbHRlciBpZD0iYyIgeD0iMTYiIHk9IjQ0IiB3aWR0aD0iMTE5IiBoZWlnaHQ9IjEyMSIgZmlsdGVyVW5pdHM9InVzZXJTcGFjZU9uVXNlIj48ZmVHYXVzc2lhbkJsdXIgc3RkRGV2aWF0aW9uPSIxNS4yIi8%2BPC9maWx0ZXI%2BPGZpbHRlciBpZD0iZCIgeD0iLTgiIHk9Ii02MSIgd2lkdGg9IjEyNyIgaGVpZ2h0PSIxMjQiIGZpbHRlclVuaXRzPSJ1c2VyU3BhY2VPblVzZSI%2BPGZlR2F1c3NpYW5CbHVyIHN0ZERldmlhdGlvbj0iMTMuOCIvPjwvZmlsdGVyPjwvZGVmcz48L3N2Zz4%3D" alt="Antigravity plugin"></a>
   <a href="#docker-and-hosting"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
+  <a href="https://render.com/deploy?repo=https://github.com/WendellResearch-AI/benchwarmers"><img src="https://img.shields.io/badge/Render-deploy-8A05FF?logo=render&logoColor=white" alt="Deploy to Render in one click"></a>
   <br>
   <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-OpenAI_%7C_Responses-111111" alt="OpenAI chat completions and Responses API"></a>
   <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Anthropic_Messages-111111" alt="Anthropic Messages API"></a>
@@ -154,8 +160,11 @@ dependencies either way: the app is the Python standard library.
 4. **Onboard a few models** from it ("Onboard a model"): tick one or several;
    each is registered, probed and run through the suite, one at a time, with
    the progress beside the button. **Quick** probes now and leaves the suite
-   to the scheduled passes; **Quick + ad hoc** also keeps the bench from ever
-   calling the model on its own, for a metered or trial key.
+   to the scheduled passes; **+ ad hoc** (with Full or Quick) keeps the bench
+   from ever calling the model on its own afterwards, for a metered or trial
+   key. Tick **Vision
+   model** for one that reads images and little else: it sits only for
+   questions about images, and is tested on the vision tasks, not the code ones.
 
 5. **Make a collection** on the Collections page (the models a panel always
    includes, and what it is for). The dashboard answers the gateway's routes on
@@ -186,10 +195,12 @@ dependencies either way: the app is the Python standard library.
   (requests a minute, day or month; tokens a minute or day). They are kept per
   connection, since models behind one key share them, and enforced: a call
   waits when the wait fits, or is refused before it goes out. Every call lands
-  in a monthly usage ledger with its tokens, and the Quotas page shows what is
-  left and when it resets.
-- **Discover.** models.dev and FreeLLMAPI, merged into one list of models
-  worth adding, each with the routes this install has to it.
+  in a monthly usage ledger with its tokens. Quota management has a tile per
+  provider showing what is left, when it resets and who is resting. OpenRouter
+  and CLI Proxy API's sign-ins are read live, at no cost to quota.
+- **Discover.** FreeLLMAPI's free catalog, with models.dev for the details
+  and paid models it lacks, merged into one list of models worth adding,
+  smartest first, each with the routes this install has to it.
 
 ## Asking a panel
 
@@ -240,6 +251,7 @@ without it: ask and judge, and run the bench itself. It is a client of a
 running server, so the same commands work against one on another machine.
 
     bench status                                   what is running, and where
+    bench restart                                  start it again on the code now on disk
     bench collections                              what you can ask
     bench ask core "Where can this return a stale value?" -f cache.py
     git diff | bench ask core "Review this diff"   a pipe is the prompt
@@ -309,6 +321,13 @@ Benchwarmers running, the hook shows a hook-error notice at the end of each
 turn, as `/mcp` shows the server as failed; start the server, or disable the
 plugin for that session.
 
+**Cursor and Antigravity plugins.** The same skill, with the stdio server
+below (so they need Node and Python 3.11 or later): Cursor's is
+[on cursor.directory](https://cursor.directory/plugins/benchwarmers), and
+Antigravity's installs from the repository:
+
+    agy plugin install https://github.com/WendellResearch-AI/benchwarmers
+
 **Any other MCP client** that speaks HTTP:
 
 ```json
@@ -417,9 +436,13 @@ reaches Wendell Research. What it sends, and where:
   a hub such as Open WebUI, or a local runtime. That includes the probes, the
   suite and onboarding, which spend calls on your keys. Add only endpoints you
   are content to send that work to.
-- **Catalog reads**, no key and nothing of yours: models.dev and FreeLLMAPI
-  once a day, and OpenRouter's public model list.
+- **Catalog reads**, no key and nothing of yours: FreeLLMAPI (its feed and
+  its site's sitemap) and models.dev once a day, and OpenRouter's public
+  model list.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page.
+- **Quota reads**, only when you press Refresh on Quota management, with no
+  prompt: OpenRouter's key status, with your key. CLI Proxy API asks each
+  sign-in's own provider for its quota.
 - **Charts**: the dashboard's browser page loads Observable Plot from jsDelivr
   at a pinned version.
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the

package/USAGE.md

@@ -14,6 +14,7 @@ coding agent doing it with you); the gateway's whole contract is in
 - [Connecting an app](#connecting-an-app)
 - [Recording verdicts](#recording-verdicts)
 - [Discover: finding models to add](#discover-finding-models-to-add)
+- [Quota management](#quota-management)
 - [The scheduler](#the-scheduler)
 - [Your data folder](#your-data-folder)
 
@@ -25,18 +26,25 @@ coding agent doing it with you); the gateway's whole contract is in
 | --- | --- | --- |
 | Observe | **At a glance** | Every model on the bench: whether it works right now, its role, and its connection. Click a name for its popup: role, alias, facts, options, retire. |
 | | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, and speed. |
-| | **Quota management** | What limits each connection, and which models are resting after hitting one. |
+| | **Quota management** | A tile per provider: what is left, when it resets, and which of its models are resting. Then the week's resets, and what the bench sent. |
 | | **Logs** | Gateway calls (who sat and how it went, never the prompt), changes made from the dashboard, the scheduler, and onboarding runs. |
 | Gateway | **Endpoints** | Where models are served from. Add an endpoint (a provider's sign-in, or any OpenAI-compatible API key), change a key, and see what each one serves. Also your own app registrations and rate limits per connection. |
 | | **Models** | Every model on the bench, by the connection it comes from. |
 | | **Collections** | The named panels your apps ask for, and the gateway's settings. |
 | | **Discover** | Models worth adding, from public catalogs, with the ways this install can reach each one. |
-| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, and how the install is reached (shown only). |
+| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, how the install is reached (shown only), and Restart. |
 | | **Info** | Version, Python, data folder and counts. |
 
 "Action needed", above the pages, lists what will not fix itself, such as a
-model about to be retired, with Retry, Open and Retire now beside it. Hide
-folds it away until something new turns up.
+model about to be retired, naming each model and where it comes from, with
+Retry, Open and Retire now beside it. Hide folds it away until something new
+turns up; an alert whose models have all been retired goes at once.
+
+**Restart** (Settings, Server and access), or `bench restart`, starts the
+dashboard again on the code now on disk, after an update. It waits for
+onboarding, a scheduled job and gateway calls in progress (`--force` cuts them
+off), and Settings says when newer code is on disk. It needs no service
+manager: `serve.py` restarts its own dashboard.
 
 ## Onboarding models
 
@@ -44,20 +52,31 @@ folds it away until something new turns up.
 not on the bench yet. Search, and tick one or several: the ticks survive a new
 search, so you can gather models from different connections in one go.
 Dismiss hides one you never want offered (an image model, say); the dismissed
-list brings it back.
+list brings it back. Each endpoint's list is kept an hour, with its age and a
+Refresh, and a model already queued or onboarding is not offered again.
 
 Each model is registered, probed with four short calls, and, if it answers,
 run through the suite. **Onboard as**, under the list, chooses how much that
 spends:
 
 - **Full**: the probe, then the suite now.
+- **Full + ad hoc**: the probe and the suite now, once, so routing has scores
+  for it; after that the bench never calls the model on its own (no explorer
+  seat, no scheduled runs). A collection that lists it still calls it.
 - **Quick**: the probe now; the scheduled suite passes fill the suite in, six
   tasks a run. The same calls, spread out.
-- **Quick + ad hoc**: the probe only, and the bench never calls the model on
-  its own (no explorer seat, no scheduled runs). The one that saves calls, for
-  a metered, capped or trial key. A collection that lists it still calls it.
-
-`bench onboard <served> --quick [--ad-hoc]` does the same from a terminal.
+- **Quick + ad hoc**: the probe only, and then ad hoc as above. The one that
+  spends least, for a metered, capped or trial key; its suite stays empty,
+  since the scheduled passes leave ad hoc models alone.
+
+**Vision model**, the box beside it, gives the model role **vision** as it is
+registered: it sits only for requests that carry images, and its suite is the
+vision tasks (photos of documents, charts, labels and screens with an answer
+key) rather than the text and code ones. Full runs them now, Quick through the
+scheduled passes. For a model that reads images and little else; a model that
+does everything and also takes images stays general and gets both.
+
+`bench onboard <served> [--quick] [--ad-hoc] [--vision]` does the same from a terminal.
 Onboarding takes minutes and spends calls, so models run **one at a time**,
 in a queue the dashboard keeps:
 
@@ -84,6 +103,15 @@ everything else well stays `general`.
 A collection is a named panel: the models it always includes, the shape of
 work it is for (review, design, enumeration, ...), and how its reply comes
 back. On the **Collections** page, **New collection**, then pick the members.
+Each collection is drawn as its panel sits: a seat per member, the explorer
+seats dashed, and the merger after an arrow.
+
+In the popup, the panel's seats are on top, and the member list has one row
+per model with its context, vision and tools, and price. A model served by
+several providers is still one row, with a chip per provider: tick the chip
+of the route you want. Ticking two is allowed, and the count says so. The tags
+grey out the models that do not fit them, as before.
+
 Each call also seats explorers, the models unused longest, up to the gateway's
 limits (Gateway settings, on the same page: how many calls a request may make,
 and how many of them explore).
@@ -184,17 +212,55 @@ are often wrong together.
 
 ## Discover: finding models to add
 
-**Discover** merges models.dev and FreeLLMAPI into one list, free models
-first, with what is new this week badged. A row opens its popup: what the
-model takes and does, its price, and every route this install has to it, each
-with an **Add via this**. Adding through a hub (Open WebUI, LiteLLM, CLI Proxy
+**Discover** leads with FreeLLMAPI's catalog of free tiers and fills in from
+models.dev, which alone has prices, paid models and providers' base URLs. One
+list: free models first, then by reach, and within each reach FreeLLMAPI's
+smartest first (the **Rank** column), then what it does not rank, what is new
+this week first and badged. A row opens its popup: what
+the model takes and does, its limits and monthly budget, FreeLLMAPI's known
+quirks for it, links to its freellmapi.co page and models.dev, and every route
+this install has to it, each with an **Add via this**. Adding through a hub (Open WebUI, LiteLLM, CLI Proxy
 API) puts it in that hub too; adding direct registers it on its own endpoint
-only. Either way the onboarding joins the same queue.
+only. Either way the onboarding joins the same queue, with the same
+**Onboard as** and **Vision model** choice as the Onboard dialog, asked in the
+confirm before anything is added.
 
 Some providers' free models work only inside their own app; Discover leaves
 those out and keeps their paid models. Settings, **Free only in their own
 app**, switches a provider either way or adds one.
 
+## Quota management
+
+One tile per provider a bench model arrives through, shaped by how that
+provider limits:
+- **Windows**: CLI Proxy API's sign-ins have a 5-hour and a weekly limit per
+  model group. Other providers have the windows their rate-limit headers state.
+- **Pool**: OpenRouter's free requests a day, shared by its models, with a bar
+  per key.
+- **Each model's last day**: shown where the provider states nothing.
+
+Each tile shows when its limits reset and which of its models are resting
+after a rate limit. **Models** lists the provider's bench models, whether each
+is free to call now, and its last 24 hours of calls.
+
+**Refresh never spends quota.** On OpenRouter it asks the key what is left. On
+CLI Proxy API's sign-ins it asks the proxy for each one's quota, as its own
+console does; this needs CLI Proxy API on the Endpoints page as a hub, with
+its management key. Every other tile re-reads what the provider last said in
+its replies to the bench. **Check now** sends one 16-token request to read the
+reply's limit headers. It is offered only where the provider has sent them
+before.
+
+Tiles sort with the ones that are out first, then the soonest reset. Drag a
+tile, or move it with the arrow keys, and the order is kept; **Sort** brings
+the default order back.
+
+Under the tiles are:
+- the next 7 days' resets, and when each resting model is back
+- **What the bench sent**: requests per provider this month and in the last 24
+  hours, as a chart with its table
+- the known limits, folded away
+
 ## The scheduler
 
 The recurring jobs (the morning availability probe, the catalog refresh, the

package/VERSION

@@ -1 +1 @@
-0.3.3
+0.5.0

package/cli.py

@@ -258,10 +258,9 @@ def cmd_onboard(args) -> None:
         reply = call("GET", "/api/onboard-queue") if verb == "status" else \
             call("POST", f"/api/onboard-queue/{verb}", {})
     else:
-        if args.ad_hoc and not args.quick:
-            sys.exit("--ad-hoc goes with --quick: an ad hoc model gets no suite run of its own")
-        mode = "quick-adhoc" if args.ad_hoc else "quick" if args.quick else "full"
-        items = [{"served": s, "endpoint": args.endpoint or "", "mode": mode} for s in args.served]
+        mode = ("quick" if args.quick else "full") + ("-adhoc" if args.ad_hoc else "")
+        items = [{"served": s, "endpoint": args.endpoint or "", "mode": mode, "vision": args.vision}
+                 for s in args.served]
         reply = call("POST", "/api/onboard-queue", {"items": items})
         if args.wait:
             while reply.get("running"):
@@ -384,6 +383,13 @@ def cmd_verdict(args) -> None:
                                        f" {len(body['real'])} real, {len(body['false'])} false"))
 
 
+def cmd_restart(args) -> None:
+    """serve.restart: refused while work is in progress, unless --force."""
+    reply = call("POST", "/api/restart", {"force": args.force})
+    out(args, reply, lambda r: done(r, "restarting on the code now on disk; back in a few seconds"
+                                       + (f" (cut off: {'; '.join(r['cut'])})" if r.get("cut") else "")))
+
+
 def cmd_endpoints(args) -> None:
     reply = call("GET", "/api/endpoints")
     out(args, reply, lambda r: table(r["endpoints"], ["name", "base", "kind", "hub", "default", "key_set", "models"],
@@ -546,6 +552,9 @@ def parser() -> argparse.ArgumentParser:
         return act
 
     cmd("status", cmd_status, "what is running, and where")
+    s = cmd("restart", cmd_restart, "start the dashboard again on the code now on disk")
+    s.add_argument("--force", action="store_true",
+                   help="even while onboarding, a scheduled job or gateway calls are in progress (cuts them off)")
 
     s = cmd("ask", cmd_ask, "ask a collection: every seat answers")
     s.add_argument("collection", help="a collection, e.g. core (bench collections)")
@@ -597,7 +606,10 @@ def parser() -> argparse.ArgumentParser:
     s.add_argument("--quick", action="store_true",
                    help="probe only; the scheduled suite passes fill the suite in later")
     s.add_argument("--ad-hoc", action="store_true",
-                   help="with --quick: the bench never calls it on its own (metered, capped or trial keys)")
+                   help="the bench never calls it on its own afterwards (metered, capped or trial keys);"
+                        " without --quick the suite still runs once now")
+    s.add_argument("--vision", action="store_true",
+                   help="role vision: a seat only for questions about images, and its suite is the vision tasks")
 
     s = cmd("dismiss", cmd_dismiss, "stop offering served models as candidates")
     s.add_argument("served", nargs="+"), s.add_argument("--endpoint"), s.add_argument("--undo", action="store_true")

package/gateway.py

@@ -330,6 +330,22 @@ def retired_why(entry: dict) -> str:
             " (Restore in its model popup brings it back)")
 
 
+def refused_now(aliases, est_tokens: int, within: float) -> dict[str, str]:
+    """alias -> why, for models whose known rate limits or quota leave no room
+    for a call within `within` seconds (metrics/pacing.held): asked, they
+    would only bring back the bench's own refusal, so they are not seated."""
+    import pacing
+    out = {}
+    for a in aliases:
+        try:
+            h = pacing.held(a, route.connection_of(a), est_tokens)
+        except Exception:                              # noqa: BLE001 -- a broken limit file is not a refusal
+            continue
+        if h and h[0] > within:
+            out[a] = f"held by a rate limit: {h[1]}; room in {pacing._wait_text(h[0])}"
+    return out
+
+
 def retiring() -> set[str]:
     """Aliases not to spend an explorer seat on: those on a retiring alert
     (failing their probes; 2026-10-16 both explorers of a bench/core call were
@@ -407,6 +423,10 @@ class TooLarge(ValueError):
     """The prompt does not fit a member's context window (HTTP 413)."""
 
 
+class AllHeld(LookupError):
+    """Every seat is held by a known rate limit (HTTP 429): try again later."""
+
+
 def estimate_tokens(text: str) -> int:
     """Characters / 4 plus 10%: tokenizers differ by family, so this errs high.
     The windows it is compared with are corrected by learned 'maximum context
@@ -590,6 +610,13 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
                        f" windows) in bench/{name}. Trim it under {min(small.values()):,} tokens,"
                        " or choose a collection whose members have larger windows"
                        " (GET /v1/models lists each collection's context_window).")
+    # A MODEL THE BENCH KNOWS WILL BE REFUSED IS NOT ASKED: a member or a
+    # `with` whose limits have no room before its seat would give up, named
+    # like the others; it would only come back as the refusal.
+    held = refused_now([*sitting, *nominated], needed, SEAT_TIMEOUT)
+    skipped += [{"seat": "member" if a in sitting else "with", "alias": a, "why": why} for a, why in held.items()]
+    sitting = [a for a in sitting if a not in held]
+    nominated = [a for a in nominated if a not in held]
     if pictures and not collection.get("shape"):
         shape, how = "vision", "the request carries images"
     elif not n_explorers:
@@ -603,6 +630,8 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
         # `with` in the request body is route.py's --with: an explorer seat.
         from run import _registered_names
         avoid = retiring() | set(too_small(needed, _registered_names()))   # explorers must fit too
+        # and have room now: an explorer seat can go to a model without a wait.
+        avoid |= set(refused_now(_registered_names(), needed, 0))
         if pictures:
             avoid |= {a for a in _registered_names() if cannot_see(a)}    # and see
         p = route.panel(shape, large=len(whole) > LARGE, nominated=nominated or None,
@@ -615,8 +644,9 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
                          + ", and no model that takes images was free for an explorer seat. Add one to the"
                          " collection, or call one whose together.capabilities on GET /v1/models lists vision.")
     if not seats:
-        raise LookupError(f"collection {name!r} has no members and no explorer could be seated"
-                          + "".join(f"; {k['alias']} not asked: {k['why']}" for k in skipped))
+        raise (AllHeld if held else LookupError)(
+            f"collection {name!r} has no members and no explorer could be seated"
+            + "".join(f"; {k['alias']} not asked: {k['why']}" for k in skipped))
     panel_id = f"p{dt.datetime.now():%Y%m%d-%H%M%S}-{secrets.token_hex(2)}"
     model = bench_collections.PREFIX + name
     task = " ".join(prompt.split())[:90] + f" ({model}, {shape})"
@@ -718,6 +748,8 @@ def refusal(exc: Exception) -> tuple[int, str]:
         return 413, str(exc)
     if isinstance(exc, ValueError):
         return 400, str(exc)
+    if isinstance(exc, AllHeld):
+        return 429, str(exc)
     if isinstance(exc, LookupError):
         return 503, str(exc)
     return 500, f"the panel failed ({type(exc).__name__}: {exc})"

package/launcher.py

@@ -22,7 +22,7 @@ ROOT = Path(__file__).resolve().parent                 # the app: a checkout, th
 SERVERS = {"serve": "serve.py", "gateway": "gateway.py", "mcp": "mcp_server.py",
            "init": str(Path("scripts") / "init.py")}
 # Everything but running a server is cli.py, a client of a running server.
-CLIENT = ("status", "ask", "verdict", "models", "model", "candidates", "onboard", "dismiss",
+CLIENT = ("status", "restart", "ask", "verdict", "models", "model", "candidates", "onboard", "dismiss",
           "collections", "collection", "endpoints", "endpoint", "keys", "key", "settings")
 
 
@@ -57,6 +57,7 @@ def usage() -> str:
         "",
         "Using it (a client of a running server, BENCHWARMERS_URL; --help on any):",
         "  status                          what is running, and where",
+        "  restart [--force]               start the dashboard again on the code now on disk",
         "  ask <collection> [prompt]       ask a collection; -f FILE adds a file, a pipe reads stdin",
         "  verdict <call> --useful|--not-useful [--real ...] [--false ...]",
         "  models | model <action>         list, rename, role, retire, restore, max-tokens, ...",

package/metrics/aliases.py

@@ -16,17 +16,38 @@ from __future__ import annotations
 
 import json
 import pathlib
+import time
 
 ROOT = pathlib.Path(__file__).resolve().parent.parent
 from paths import DATA  # noqa: E402  -- the install's data folder (metrics/paths.py)
 HISTORY = DATA / "config" / "alias-history.json"
 
 
+_cache: tuple = (None, {}, 0.0)
+
+
 def history() -> dict[str, str]:
+    """The file, read again only when it changes, and looked at no more than
+    once a second: one dashboard render asks tens of thousands of times, and
+    even a stat each time was seconds on Windows. A rename written here
+    (serve.rename) calls forget(), so it shows at once."""
+    global _cache
+    seen, now = _cache, time.monotonic()          # one read: forget() may swap it on another thread
+    if seen[0] and seen[0][0] == str(HISTORY) and now - seen[2] < 1:
+        return seen[1]
     try:
-        return json.loads(HISTORY.read_text(encoding="utf-8"))
-    except (OSError, ValueError):
-        return {}
+        st = HISTORY.stat()
+        key = (str(HISTORY), st.st_mtime_ns, st.st_size)
+        table = seen[1] if seen[0] == key else json.loads(HISTORY.read_text(encoding="utf-8"))
+    except (OSError, ValueError):                 # none yet, or half-written: no renames, for a second
+        key, table = (str(HISTORY), None, None), {}
+    _cache = (key, table, now)
+    return table
+
+
+def forget() -> None:
+    global _cache
+    _cache = (None, {}, 0.0)
 
 
 def current(alias: str, table: dict[str, str] | None = None) -> str: