@wendellresearch-ai/benchwarmers 0.3.0 → 0.3.2

package/AGENTS.md

@@ -25,12 +25,11 @@ Python 3.11 or later; nothing else is required.
     git clone https://github.com/WendellResearch-AI/benchwarmers
     cd benchwarmers
     python -m venv .venv
-    .venv/bin/pip install llm          # optional; Windows: .venv\Scripts\pip install llm
 
-The `llm` CLI is today's default model client. Without it, choose the
-built-in client (`metrics/client.py`), which needs nothing installed: on
-Settings, Model client, once the dashboard runs, or with
-`BENCH_CLIENT=builtin` (which wins over Settings).
+The built-in model client (`metrics/client.py`) makes every call and needs
+nothing installed. The `llm` CLI is an optional alternative, for its plugins:
+`pip install llm` in the venv, then choose it on Settings, Model client, or
+with `BENCH_CLIENT=llm` (which wins over Settings).
 
 Check: `python scripts/check.py --quick` ends with `all checks pass`.
 
@@ -67,8 +66,9 @@ API) or a local runtime (Ollama). Three ways, best first:
   | `openrouter-2` | `BENCHWARMERS_KEY_OPENROUTER_2` |
   | `my-hub` | `BENCHWARMERS_KEY_MY_HUB` |
 
-  Without the variable, keys come from the `llm` CLI's key store
-  (`llm keys set <name>` writes there). The variable wins when both exist.
+  Without the variable, keys come from llm's key file (`keys.json` in llm's
+  user folder, which the dashboard and `init --key` write; the `llm` CLI need
+  not be installed). The variable wins when both exist.
 
 One provider can be several endpoints (a pasted key and a sign-in, or two
 accounts); each needs its own name, and Benchwarmers suggests a free one.
@@ -107,7 +107,10 @@ Any OpenAI-compatible client asks for the model `bench/<collection>`:
 Every seat's answer comes back, plus a `bench` field with each seat's call id.
 Clients of the Anthropic, Ollama, Gemini and Responses APIs work too, with
 the server's address as their base URL (GATEWAY.md, "Other API formats");
-requests carrying tools are refused, since a panel answers with text.
+a panel answers with text, so requests carrying tools are refused there.
+An agent that needs tools (Claude Code, Codex, opencode) runs on a collection
+in route mode instead, `bench/<collection>:route`: one member per turn, with
+its tools, kept for the conversation (GATEWAY.md, "Route mode").
 For a chat window that shows one answer, ask for `bench/<collection>:merged`
 (or set the collection's reply to merged): a merger model writes one answer
 saying where the seats agreed and disagreed, and every seat is still in

package/GATEWAY.md

@@ -167,6 +167,14 @@ A verdict is `POST /bench/verdict` with `{"call", "useful": true|false, "note",
 dialog in the dashboard's Collections tab, under Gateway (click a seat). Sending the same verdict
 twice writes nothing.
 
+A client that weighs its seats by how they have done reads the verdicts back:
+`GET /bench/record?collection=bench/vision&task=thicken slab faces` answers
+`{"collection", "task", "aliases": {"<alias>": {"useful", "notUseful",
+"judged", "lastAt"}}}`, counting each model's judged seats on that collection.
+`task` (optional) keeps the verdicts whose note holds it. Counts only: never a
+prompt, an answer or a note. thicken breaks ties between its vision seats with
+it.
+
 ## Streaming
 
 With `"stream": true` (or the format's own way of asking, below):
@@ -511,7 +519,7 @@ key limited to its collection and a rate, or keep the key on the app's server.
 | Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily (scheduled task WRBench-Catalog); a free model whose price goes above zero raises a "went paid" alert. |
 | Discover tab | `metrics/discover.py`: models.dev and FreeLLMAPI merged into one list of models worth adding, free first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
 | Merged replies | Opt-in: one merger answer as `content`, every seat still in `bench` (The reply). |
-| MCP server | `mcp_server.py` (`benchwarmers mcp`): the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`, over HTTP to a running server (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`), so it is a caller like any other. The Claude Code plugin in `plugin/` wraps it with a command and a skill. |
+| MCP server | `mcp_server.py`: the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`. Two transports for one `answer()`: `POST /mcp` (or `/v1/mcp`) on either port, streamable HTTP with a JSON reply per request, its tools calling the server's own routes over loopback with the caller's credential and Host; and stdio (`benchwarmers mcp`), a caller like any other over HTTP (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`). The Claude Code plugin in `plugin/` connects to `/mcp` and adds a command, a skill and a `Stop` hook (`plugin/hooks/hooks.json`, an `mcp_tool` hook) that calls `pending_verdicts` with the session's `transcript_path`: the call ids in the transcript (Claude Code keeps no `Mcp-Session-Id`, so the transcript is the record of whose panels they were), of which the call log says which are answered seats with no verdict (`POST /bench/pending`), returned once as `additionalContext`. A server on another machine cannot read the transcript, so there the hook says nothing. A tool call answers as SSE with progress notifications, which reset Claude Code's 300-second idle limit. |
 
 ## Rules
 

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 22 October 2026.*
+*Effective 23 October 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -22,13 +22,15 @@ Wendell Research.
 - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows
   from Hugging Face and source files from GitHub.
 
-Keys and sign-in tokens stay in your environment or the `llm` CLI's key
-store. [SECURITY.md](SECURITY.md) has the full detail.
+Keys and sign-in tokens stay in your environment or llm's key file
+(`keys.json` in llm's user folder, used whether or not the `llm` CLI is
+installed). [SECURITY.md](SECURITY.md) has the full detail.
 
 ## What stays on your machine
 
-Everything else: your endpoints, collections, verdicts, call logs, probe
-history and task answers live in your data folder, which is yours to back up,
+Everything else: your endpoints, collections, verdicts, call logs, the usage
+ledger (tokens per call, never the prompt), probe history and task answers
+live in your data folder, which is yours to back up,
 move or delete.
 
 ## Third parties

package/README.md

@@ -10,8 +10,14 @@
   <a href="https://www.npmjs.com/package/@wendellresearch-ai/benchwarmers"><img src="https://img.shields.io/npm/v/@wendellresearch-ai/benchwarmers?color=cb3837&logo=npm" alt="npm"></a>
   <a href="https://pypi.org/project/benchwarmers/"><img src="https://img.shields.io/pypi/v/benchwarmers?color=3775a9&logo=pypi&logoColor=white" alt="PyPI"></a>
   <a href="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml"><img src="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml/badge.svg" alt="Checks"></a>
-  <a href="#docker"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
-  <a href="GATEWAY.md"><img src="https://img.shields.io/badge/OpenAI--compatible-gateway-111111" alt="OpenAI-compatible gateway"></a>
+  <a href="#docker-and-hosting"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
+  <br>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-OpenAI_%7C_Responses-111111" alt="OpenAI chat completions and Responses API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Anthropic_Messages-111111" alt="Anthropic Messages API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Gemini-111111" alt="Gemini API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Ollama-111111" alt="Ollama API"></a>
+  <a href="GATEWAY.md#route-mode-agent-clients"><img src="https://img.shields.io/badge/agents-tool_calling-111111" alt="Agent clients with tool calling, through route mode"></a>
+  <a href="https://registry.modelcontextprotocol.io/v0/servers?search=io.github.WendellResearch-AI/benchwarmers"><img src="https://img.shields.io/badge/MCP-Registry-111111" alt="MCP server in the official MCP Registry"></a>
   <br>
   <a href="#try-it"><img src="https://img.shields.io/badge/Python-3.11_%7C_3.12_%7C_3.13-3776AB?logo=python&logoColor=white" alt="Python 3.11 to 3.13"></a>
   <a href="#try-it"><img src="https://img.shields.io/badge/core-stdlib_only-2ea44f" alt="The core needs only the Python standard library"></a>
@@ -30,8 +36,12 @@ verdicts decide who sits on the next panel.
 
 **What it is best at:**
 
-- **Asking several models at once.** One OpenAI-compatible request, a panel of
-  answers. Where they disagree is where to look.
+- **Asking several models at once.** One request, in the OpenAI, Anthropic,
+  Gemini or Ollama format, a panel of answers. Where they disagree is where
+  to look.
+- **Agents too.** Claude Code and other tool-calling clients use a collection
+  through route mode: one member per turn, kept for the conversation, with
+  failover, and every call on the record.
 - **Reviewing.** Code review, claims about code, designs: different models
   catch different things, and two models agreeing is not proof.
 - **Playing to each model's strengths.** A collection per kind of work
@@ -43,8 +53,9 @@ verdicts decide who sits on the next panel.
   repositories, plus your verdicts on real work, instead of public leaderboards.
 - **Knowing who to bench.** Who is confidently wrong, who has stopped
   answering (retired automatically), and who is resting after a rate limit.
-- **Keeping free and cheap models useful.** Health probes, learned rate limits
-  and one endpoint for every app, over any OpenAI-compatible provider or hub.
+- **Keeping free and cheap models useful.** Health probes, rate limits taken
+  from what providers say and enforced, and one endpoint for every app, over
+  any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
 
 | Related tools | Examples | What they do | What Benchwarmers does differently |
 |---|---|---|---|
@@ -68,7 +79,7 @@ verdicts decide who sits on the next panel.
 - [From the command line](#from-the-command-line)
 - [From Claude Code and other agents](#from-claude-code-and-other-agents)
 - [The suite](#the-suite)
-- [Docker](#docker)
+- [Docker and hosting](#docker-and-hosting)
 - [What leaves your machine](#what-leaves-your-machine)
 - [Limits](#limits)
 - [Security](#security)
@@ -82,7 +93,7 @@ with [AGENTS.md](AGENTS.md).
 
 ## Try it
 
-**v0.2, early.** It runs with Python 3.11 or later on Windows, macOS and
+**v0.3, early.** It runs with Python 3.11 or later on Windows, macOS and
 Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
 code where these pages are thin.
 
@@ -108,12 +119,12 @@ dependencies either way: the app is the Python standard library.
 
        git clone https://github.com/WendellResearch-AI/benchwarmers
        cd benchwarmers
-       python -m venv .venv
-       .venv/bin/pip install llm        # Windows: .venv\Scripts\pip install llm
+       python -m venv .venv             # nothing to pip install: the standard library is enough
 
-   The [`llm`](https://llm.datasette.io/) CLI is the default model client today.
-   The built-in client (`metrics/client.py`) replaces it, chosen on Settings
-   or with `BENCH_CLIENT=builtin` (which wins), and will become the default.
+   The built-in model client (`metrics/client.py`) makes every call. The
+   [`llm`](https://llm.datasette.io/) CLI is an alternative for its plugins:
+   install it in the venv and choose it on Settings (or `BENCH_CLIENT=llm`,
+   which wins).
 
 2. **Make a data folder.** Everything that is yours (endpoints, model
    registrations, collections, verdicts, probe history, task answers) lives
@@ -125,14 +136,16 @@ dependencies either way: the app is the Python standard library.
    unless you name one) and points the checkout at it with a
    `.benchwarmers-home` file; `BENCHWARMERS_HOME` overrides that. With
    `--endpoint NAME BASE_URL --key` it also adds a first endpoint and asks for
-   its key, which goes to the `llm` CLI's key store (`llm keys set NAME` does
-   the same), never into the data folder.
+   its key, which goes to the key file in llm's user folder (`keys.json`,
+   shared with the `llm` CLI if you have it, not needed otherwise), never into
+   the data folder. `BENCHWARMERS_KEY_<NAME>` in the environment works too.
 
 3. **Start the dashboard.** With no endpoint yet it opens on adding one: sign
    in with a provider on its own page (OpenRouter, NanoGPT, Hugging Face,
    Pollinations, your ChatGPT plan, GitHub Models, or a local Ollama), or paste
    an API key for any OpenAI-compatible API, a hub such as Open WebUI or
-   LiteLLM included. Only providers that publish a sign-in for other apps are
+   LiteLLM included. Anthropic and Gemini keys can also be called through
+   those providers' own APIs (the API choice in the same dialog). Only providers that publish a sign-in for other apps are
    offered; adding one is an entry in `metrics/connect.py` (its docstring says
    how).
 
@@ -140,7 +153,9 @@ dependencies either way: the app is the Python standard library.
 
 4. **Onboard a few models** from it ("Onboard a model"): tick one or several;
    each is registered, probed and run through the suite, one at a time, with
-   the progress beside the button.
+   the progress beside the button. **Quick** probes now and leaves the suite
+   to the scheduled passes; **Quick + ad hoc** also keeps the bench from ever
+   calling the model on its own, for a metered or trial key.
 
 5. **Make a collection** on the Collections page (the models a panel always
    includes, and what it is for). The dashboard answers the gateway's routes on
@@ -163,10 +178,16 @@ dependencies either way: the app is the Python standard library.
 - **Health.** Every morning each model gets a one-word prompt; once a week, a
   short and a long one for speed. A model that stops answering, or that its
   endpoint no longer serves, is retired automatically after one confirming
-  call, with the evidence kept, and can be restored in one click. Rate limits
-  are learned from providers' own error messages and kept per connection,
-  since models behind one key share them. The built-in scheduler runs all of
-  this once Settings switches it on; a fresh install spends no calls until then.
+  call, with the evidence kept, and can be restored in one click. The
+  built-in scheduler runs all of this once Settings switches it on; a fresh
+  install spends no calls until then.
+- **Rate limits and usage.** Limits come from providers' rate-limit headers
+  and error messages, and from what you set per connection or model
+  (requests a minute, day or month; tokens a minute or day). They are kept per
+  connection, since models behind one key share them, and enforced: a call
+  waits when the wait fits, or is refused before it goes out. Every call lands
+  in a monthly usage ledger with its tokens, and the Quotas page shows what is
+  left and when it resets.
 - **Discover.** models.dev and FreeLLMAPI, merged into one list of models
   worth adding, each with the routes this install has to it.
 
@@ -206,10 +227,11 @@ that speaks one of them works with the server's address as its base URL:
       -d '{"model": "bench/<collection>", "max_tokens": 1024, "messages": [{"role": "user", "content": "..."}]}'
 
 Each streams when asked: the reply starts at once, and each seat's answer
-arrives as that seat finishes. Agent and tool use is not supported yet: a
-request carrying tools is refused with a 400 rather than answered without
-them. GATEWAY.md, "Other API formats", has a curl for each and what is
-refused or ignored.
+arrives as that seat finishes. A panel answers with text, so a request
+carrying tools is refused with a 400 that points to **route mode**
+(`bench/<collection>:route`, one member with its tools; see
+[below](#from-claude-code-and-other-agents)). GATEWAY.md, "Other API formats",
+has a curl for each and what is refused or ignored.
 
 ## From the command line
 
@@ -257,28 +279,47 @@ no server at `BENCHWARMERS_URL`.
 
 ## From Claude Code and other agents
 
-`benchwarmers mcp` is an MCP server on stdio with three tools:
-`list_collections`, `ask` (a collection, a prompt, optionally `merged`,
-`with` and `system`) and `verdict`. It is a client of a Benchwarmers that is
-already running, over the routes above, so start the dashboard (or the
-service) first. The tool descriptions carry the rules: ask a collection, not
-a favourite model; verify before acting; name every seat; send a verdict per
-seat you judged.
+A running Benchwarmers is an MCP server at `/mcp` on its own port
+(`http://127.0.0.1:8122/mcp`; `/v1/mcp` works too), over streamable HTTP,
+with three tools: `list_collections`, `ask` (a collection, a prompt,
+optionally `merged`, `with` and `system`) and `verdict`. Nothing to start per
+client: every agent session shares the one server, behind the same Host,
+Origin and gateway-key checks as `/v1`. The tool descriptions carry the
+rules: ask a collection, not a favourite model; verify before acting; name
+every seat; send a verdict per seat you judged.
 
-**Claude Code plugin.** The plugin adds the MCP server, a `/benchwarmers:ask
-<collection> <what to ask>` command and a `benchwarmers` skill saying when a
-panel helps (adversarial review, exhaustive enumeration over real files) and
-how to judge it:
+**Claude Code plugin.** The plugin connects to that `/mcp` and adds a
+`/benchwarmers:ask <collection> <what to ask>` command and a `benchwarmers`
+skill saying when a panel helps (adversarial review, exhaustive enumeration
+over real files) and how to judge it:
 
     /plugin marketplace add WendellResearch-AI/benchwarmers
     /plugin install benchwarmers@benchwarmers
 
-It needs Node 18 or later and Python 3.11 or later on PATH.
+It finds the server through `BENCHWARMERS_URL` (default
+`http://127.0.0.1:8122`; set it if you changed `BENCH_PORT`) and sends
+`BENCHWARMERS_KEY` where keys are required. It also adds a `Stop` hook: at
+the end of a turn it asks the server (the `pending_verdicts` tool, so nothing
+extra runs on your machine) which seats of that session's panels have no
+verdict yet, and reminds Claude of them once, as hook feedback. The server
+finds the session's panels in its transcript, so the reminder needs a server
+on the same machine; with one elsewhere the hook stays silent. A verdict sent
+any way counts, the dashboard's included. With the plugin enabled and no
+Benchwarmers running, the hook shows a hook-error notice at the end of each
+turn, as `/mcp` shows the server as failed; start the server, or disable the
+plugin for that session.
+
+**Any other MCP client** that speaks HTTP:
+
+```json
+{"mcpServers": {"benchwarmers": {"type": "http", "url": "http://127.0.0.1:8122/mcp"}}}
+```
 
-**Any other MCP client** (or Claude Code without the plugin): run
-`benchwarmers mcp` once installed, `uvx benchwarmers mcp` or
+with `"headers": {"Authorization": "Bearer <gateway key>"}` where keys are
+required. **A client that only speaks stdio**: run `benchwarmers mcp` once
+installed, `uvx benchwarmers mcp` or
 `npx -y @wendellresearch-ai/benchwarmers mcp` without installing, or `python mcp_server.py` from a
-checkout. For example:
+checkout, a small process per client that calls the same server. For example:
 
 ```json
 {"mcpServers": {"benchwarmers": {"command": "npx", "args": ["-y", "@wendellresearch-ai/benchwarmers", "mcp"],
@@ -294,6 +335,24 @@ On native Windows, a client that cannot start `npx` itself needs
 | `BENCHWARMERS_KEY` | a gateway key from Settings, sent as `Authorization: Bearer`; needed for a server on another machine, or where keys are required |
 | `BENCHWARMERS_TIMEOUT` | seconds to wait for a panel, default 900 |
 
+**Behind an MCP aggregator** (1mcp, MetaMCP, mcp-proxy, Docker's MCP Gateway
+and the like): add `http://127.0.0.1:8122/mcp` as an HTTP (streamable HTTP)
+upstream, with the `Authorization` header where keys are required; an
+aggregator that only runs stdio servers takes `benchwarmers mcp` as above.
+Give it a request timeout of a few minutes or more, as a panel takes that
+long; 1mcp's default of 60 seconds ends most of them. With 1mcp, for example:
+
+    1mcp mcp add benchwarmers --type http --url http://127.0.0.1:8122/mcp --timeout 900000
+
+The aggregator prefixes the tool names (1mcp's are
+`benchwarmers_1mcp_ask` and so on). If you also use the Claude Code plugin,
+disable the plugin's own `benchwarmers` server in `/mcp` so the tools are not
+listed twice; the skill and `/benchwarmers:ask` keep working, though the
+command then asks before each tool call, since its pre-approved tools carry
+the plugin's names. The plugin's verdict hook calls the plugin's own server,
+so it does not work through an aggregator: with that server disabled it only
+shows a hook-error notice at the end of each turn.
+
 **An agent running on a collection.** The MCP tools ask a panel for a
 second opinion; an agent can also *be driven by* a collection, in **route
 mode** (`bench/<collection>:route`): one member, picked by its record, gets
@@ -324,7 +383,7 @@ code makes the best tasks, because no model has trained on it. See
 [suite/README.md](suite/README.md), [suite/REBENCH.md](suite/REBENCH.md) and
 [BENCHMARKS.md](BENCHMARKS.md).
 
-## Docker
+## Docker and hosting
 
 One container runs the dashboard, the gateway (on the same port, at `/v1`)
 and the scheduler, with the data folder as a volume and no `llm` CLI (the
@@ -366,21 +425,33 @@ reaches Wendell Research. What it sends, and where:
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the
   SWE-rebench-V2 rows from Hugging Face and the files from GitHub.
 
-Keys stay in the environment or the `llm` CLI's key store, never in the data
-folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
+Keys stay in the environment or llm's key file (`keys.json` in its user
+folder), never in the data folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
 detail.
 
 ## Limits
 
-- **No tools yet.** A panel answers with text, not a tool call, so agents
-  that need tool use (Claude Code, Codex, opencode as callers) wait for a
-  route mode; requests with tools are refused, not trimmed.
-- **Early.** v0.1: no installer yet, merged replies stream in one piece, and the LiteLLM and
-  CLI Proxy API hub adapters are built from those projects' source, not yet
-  run against a real hub.
+- **Tools go through route mode only.** A panel answers with text; a request
+  with tools is refused there, not trimmed. Route mode passes turns only to
+  members on chat-completions endpoints, keeps its sticky conversations in
+  memory (a restart starts each on a fresh pick), and has been run end to end
+  with Claude Code; the Codex, opencode and Cline set-ups are written from
+  their docs. Shadow seats (other members answering an agent's turn for
+  evaluation) are planned, not built.
+- **Some parts are checked against fakes only.** The native Anthropic and
+  Gemini endpoints, and the LiteLLM and CLI Proxy API hub adapters, are built
+  from those APIs' documentation and source and tested against fake servers,
+  not yet against the real services.
+- **A merged reply arrives in one piece** once the merger has finished; the
+  seats before it stream as they finish.
+- **Pacing and the ledger cover the built-in client.** Calls made through the
+  `llm` CLI, when you choose it, are neither paced nor counted.
 - **Calls cost.** Probes, onboarding and the suite run on your keys and free
-  tiers. Onboarding runs one model at a time for that reason; a suite pass is
-  paced, not burst.
+  tiers. Onboarding runs one model at a time for that reason (Quick spreads
+  the suite over the scheduled passes); a suite pass is paced, not burst.
+- **The scheduler lives in the server.** Its jobs run while the dashboard (or
+  the container, or the service) is up; a missed run is made up once, within
+  seven days, when it comes back.
 - **One install, one person** at heart. Logins and gateway keys exist for a
   LAN or a server, but there are no per-user views or roles.
 - **The onboarding queue lives in the dashboard's memory.** Restarting it

package/SECURITY.md

@@ -58,7 +58,8 @@ open itself to anyone.
 ## What Benchwarmers touches
 
 - **Keys.** Endpoint keys come from `BENCHWARMERS_KEY_<NAME>` (the key name
-  upper-cased, `-` as `_`) first, then from the `llm` CLI's key store. They are
+  upper-cased, `-` as `_`) first, then from llm's key file (`keys.json` in
+  its user folder, used whether or not the `llm` CLI is installed). They are
   never written to the data folder, never shown back, and never put in a log.
   Sign-ins through a provider's page store their token the same way.
 - **The network.** Your prompts go only to the endpoints you add; the probes,
@@ -68,9 +69,9 @@ open itself to anyone.
   dashboard's page, and Hugging Face and GitHub only when you run
   `suite/rebench.py`. [PRIVACY.md](PRIVACY.md) lists them.
 - **Files.** It writes inside its data folder (`BENCHWARMERS_HOME`, or the
-  folder `.benchwarmers-home` names) and, in the `llm` CLI's own folder, its
-  key store (when you paste a key or sign in) and its model list (when it
-  registers a model). Images sent to the gateway are never written to a log.
+  folder `.benchwarmers-home` names) and, in llm's user folder, the key file
+  (when you paste a key or sign in) and the model list (when it registers a
+  model). Images sent to the gateway are never written to a log.
 - **Processes.** Onboarding, the probes and the suite run this repository's
   own scripts with the same Python. Model ids that reach a command line are
   checked against a strict pattern first.

package/USAGE.md

@@ -47,8 +47,19 @@ Dismiss hides one you never want offered (an image model, say); the dismissed
 list brings it back.
 
 Each model is registered, probed with four short calls, and, if it answers,
-run through the suite. That takes minutes and spends calls, so they run **one
-at a time**, in a queue the dashboard keeps:
+run through the suite. **Onboard as**, under the list, chooses how much that
+spends:
+
+- **Full**: the probe, then the suite now.
+- **Quick**: the probe now; the scheduled suite passes fill the suite in, six
+  tasks a run. The same calls, spread out.
+- **Quick + ad hoc**: the probe only, and the bench never calls the model on
+  its own (no explorer seat, no scheduled runs). The one that saves calls, for
+  a metered, capped or trial key. A collection that lists it still calls it.
+
+`bench onboard <served> --quick [--ad-hoc]` does the same from a terminal.
+Onboarding takes minutes and spends calls, so models run **one at a time**,
+in a queue the dashboard keeps:
 
 - One model with nothing ahead of it is followed step by step in the popup.
 - Otherwise the popup shows the queue, and a chip beside the button shows the
@@ -62,6 +73,12 @@ at a time**, in a queue the dashboard keeps:
 A model that registered but did not answer stays on the bench as
 unavailable, so the daily probe keeps watching it.
 
+**Roles.** A model's popup sets what it is for: `general` (the default),
+`code`, `light` (small and fast), `vision` (reads images well, weak at text:
+OCR and document models; seated only when a request carries images) or
+`image` (makes images; never seated). A model that reads images and does
+everything else well stays `general`.
+
 ## Collections
 
 A collection is a named panel: the models it always includes, the shape of

package/VERSION

@@ -1 +1 @@
-0.3.0
+0.3.2

package/gateway.py

@@ -150,6 +150,7 @@ import client  # noqa: E402
 import decide  # noqa: E402
 import formats  # noqa: E402
 import guard  # noqa: E402
+import mcp_server  # noqa: E402  -- its tools, answered at /mcp too
 import model_roles  # noqa: E402
 import offload  # noqa: E402
 import route  # noqa: E402
@@ -440,12 +441,33 @@ def too_small(needed: int, aliases) -> dict[str, int]:
 
 def record_seat(task: str, call: str, label: str, alias: str, r: dict) -> dict:
     """Log one answer as an unjudged offload, the moment it is in, so explorer
-    rotation counts it even if nobody judges; returns the seat as logged."""
+    rotation counts it even if nobody judges; returns the seat as logged. The
+    note's "<seat>; <status> in" is what /bench/pending reads the status from."""
     offload.add(alias, task, None, f"{label}; {r['status']} in {r['seconds']}s"
                 + (f": {r['error'][:120]}" if r["error"] else ""), call)
     return {"call": call, "seat": label, "alias": alias, **r}
 
 
+def record(collection: str, task: str = "") -> dict[str, dict]:
+    """Each model's verdicts on one collection's seats, for a client that weighs
+    its seats by how they have done (thicken breaks ties with it): useful, not
+    useful and judged, counted, and the last one's date. `task` keeps only the
+    verdicts whose note holds it ("thicken slab faces"). Counts only: never a
+    prompt, an answer or a note."""
+    out: dict[str, dict] = {}
+    for e in offload.entries():
+        # The collection ends each seat's task as run_collection writes it: "(bench/<name>, <shape>)".
+        if e.get("useful") is None or f"({collection}," not in e.get("task", ""):
+            continue
+        if task and task not in (e.get("note") or ""):
+            continue
+        r = out.setdefault(e["model"], {"useful": 0, "notUseful": 0, "judged": 0, "lastAt": ""})
+        r["useful" if e["useful"] else "notUseful"] += 1
+        r["judged"] += 1
+        r["lastAt"] = max(r["lastAt"], e.get("date", ""))
+    return out
+
+
 def merger_candidates(collection: dict, members: list[str], shape: str, large: bool):
     """Who may merge, best first. The collection's `merger` alone when set;
     else the members with verdicts on this shape, most useful first, then the
@@ -1030,6 +1052,12 @@ class Handler(BaseHTTPRequestHandler):
             # orchestrator can choose one by task. Clients ignore extra keys.
             return self._json(200, {"object": "list", "data": [
                 {"id": n, "object": "model", "owned_by": "benchwarmers", **d} for n, d in listed(self.who)]})
+        if path == "/bench/record":
+            query = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query)
+            collection, task = (query.get(k, [""])[0] for k in ("collection", "task"))
+            if not collection:
+                return self._error(400, "GET /bench/record?collection=bench/<name>[&task=<words in the verdicts' notes>]")
+            return self._json(200, {"collection": collection, "task": task, "aliases": record(collection, task)})
         if path == "/api/tags":
             return self._json(200, formats.ollama_tags(listed(self.who)))
         if path == "/api/version":
@@ -1041,7 +1069,10 @@ class Handler(BaseHTTPRequestHandler):
             row = dict(listed(self.who)).get(name)
             return self._json(200, formats.gemini_model(name, row)) if row else \
                 self._error(404, f"{name!r} is not a collection here; GET /v1beta/models lists them", fmt=fmt)
-        self._error(404, "GET /v1/models, /api/tags, /api/version or /v1beta/models", fmt=fmt)
+        if guard.mcp_route(path):                      # no stream from the server: every reply is a POST's
+            return self._json(405, {"error": {"message": "POST /mcp, one JSON-RPC message per request"}},
+                              {"Allow": "POST"})
+        self._error(404, "GET /v1/models, /bench/record, /api/tags, /api/version or /v1beta/models", fmt=fmt)
 
     def do_POST(self):                                 # noqa: N802
         # Without this any web page could spend model calls, seat ad hoc
@@ -1053,6 +1084,8 @@ class Handler(BaseHTTPRequestHandler):
             self.who = guard.gateway_caller(self.headers, self._query_key(fmt))
         except guard.Refused as r:
             return self._error(r.status, str(r) + (TOO_BIG if r.status == 413 else ""), r.headers, fmt)
+        if guard.mcp_route(path):
+            return self._mcp(length)
         try:
             body = json.loads(self.rfile.read(length) or b"{}")
         except ValueError:
@@ -1071,6 +1104,22 @@ class Handler(BaseHTTPRequestHandler):
             except KeyError as exc:
                 return self._error(404, exc.args[0])
             return self._json(200, {"ok": True, "recorded": e})
+        if path == "/bench/pending":
+            # Which of these call ids are seats that answered (ok or empty, not errored) and have
+            # no verdict yet, however one is sent (mcp_server.py's pending_verdicts, the plugin's
+            # Stop hook). An id the log does not know is not a seat. A seat's first entry is
+            # record_seat's, its note "<seat>; <status> in <s>s".
+            calls = body.get("calls")
+            if not (isinstance(calls, list) and all(isinstance(c, str) for c in calls)):
+                return self._error(400, '"calls" must be a list of call ids')
+            first, judged = {}, set()
+            for e in offload.entries():
+                first.setdefault(e.get("call"), e)
+                if e.get("useful") is not None:
+                    judged.add(e.get("call"))
+            return self._json(200, {"pending": [
+                c for c in calls if c in first and c not in judged
+                and "; error in " not in str(first[c].get("note", "")).split(":", 1)[0]]})
         # Each format's request becomes one Ask (metrics/formats.py); a part
         # that would be lost on the way, tools above all, is refused with why.
         gemini = formats.gemini_route(path)
@@ -1099,6 +1148,66 @@ class Handler(BaseHTTPRequestHandler):
         alt = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query).get("alt", [""])[0]
         self._panel(fmt, ask, sse=alt == "sse")
 
+    def _mcp(self, length: int) -> None:
+        """MCP over streamable HTTP, a reply to each POST (JSON, or SSE for a
+        tool call from a client that takes it): mcp_server.py's
+        tools on this port, so a client starts no process of its own. They call
+        this server's own routes over loopback with the caller's credential and
+        Host (both already checked above), so a key's collections and rate
+        limit hold exactly as on /v1."""
+        try:
+            msg = json.loads(self.rfile.read(length))
+        except (ValueError, RecursionError):           # RecursionError: nesting deeper than json can parse
+            return self._json(400, {"jsonrpc": "2.0", "id": None, "error": {"code": -32700, "message": "parse error"}})
+        host, port = self.server.server_address[:2]
+        host = {"0.0.0.0": "127.0.0.1", "::": "::1"}.get(host, host)
+        sent = {h: self.headers[h] for h in ("Authorization", "Host", *guard.KEY_HEADERS) if self.headers.get(h)}
+        conn = (f"http://{'[' + host + ']' if ':' in host else host}:{port}", sent)
+        if isinstance(msg, dict) and msg.get("method") == "tools/call" and "id" in msg \
+                and "text/event-stream" in (self.headers.get("Accept") or ""):
+            return self._mcp_stream(msg, conn)
+        reply = mcp_server.answer(msg, conn)
+        if reply is None:                              # a notification: accepted, nothing to say
+            self.send_response(202)
+            self.send_header("Content-Length", "0")
+            return self.end_headers()
+        self._json(200, reply)
+
+    def _mcp_stream(self, msg: dict, conn: mcp_server.Conn) -> None:
+        """A tool call as an SSE stream: the 200 at once, then every KEEPALIVE
+        seconds while a panel runs (minutes) a progress notification when the
+        client gave a progressToken, else a keep-alive comment, then the reply.
+        A comment keeps proxies' idle timeouts away; only progress resets a
+        client's own (Claude Code aborts a tool silent for 300 seconds)."""
+        done: queue.Queue = queue.Queue()
+        threading.Thread(target=lambda: done.put(mcp_server.answer(msg, conn)), daemon=True).start()
+        params = msg.get("params") if isinstance(msg.get("params"), dict) else {}
+        token = (params.get("_meta") or {}).get("progressToken") if isinstance(params.get("_meta"), dict) else None
+        started = time.monotonic()
+        try:
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream")
+            self.send_header("Cache-Control", "no-cache")
+            self.end_headers()
+            while True:
+                try:
+                    reply = done.get(timeout=KEEPALIVE)
+                except queue.Empty:
+                    if token is None:
+                        self.wfile.write(b": keep-alive\n\n")
+                    else:
+                        seconds = round(time.monotonic() - started)
+                        self.wfile.write(b"event: message\ndata: " + json.dumps(
+                            {"jsonrpc": "2.0", "method": "notifications/progress",
+                             "params": {"progressToken": token, "progress": seconds,
+                                        "message": f"the panel is running, {seconds} s"}}).encode() + b"\n\n")
+                    self.wfile.flush()
+                    continue
+                self.wfile.write(b"event: message\ndata: " + json.dumps(reply).encode() + b"\n\n")
+                return
+        except OSError:
+            return                                     # the caller left; the panel still finishes, every seat recorded
+
     def _show(self, body: dict) -> None:
         """POST /api/show: Ollama's model card, for a collection the key may call."""
         name = formats.ollama_model(body.get("model") or body.get("name") or "")