@wendellresearch-ai/benchwarmers 0.2.2 → 0.3.1

package/AGENTS.md

@@ -25,12 +25,11 @@ Python 3.11 or later; nothing else is required.
     git clone https://github.com/WendellResearch-AI/benchwarmers
     cd benchwarmers
     python -m venv .venv
-    .venv/bin/pip install llm          # optional; Windows: .venv\Scripts\pip install llm
 
-The `llm` CLI is today's default model client. Without it, choose the
-built-in client (`metrics/client.py`), which needs nothing installed: on
-Settings, Model client, once the dashboard runs, or with
-`BENCH_CLIENT=builtin` (which wins over Settings).
+The built-in model client (`metrics/client.py`) makes every call and needs
+nothing installed. The `llm` CLI is an optional alternative, for its plugins:
+`pip install llm` in the venv, then choose it on Settings, Model client, or
+with `BENCH_CLIENT=llm` (which wins over Settings).
 
 Check: `python scripts/check.py --quick` ends with `all checks pass`.
 
@@ -67,8 +66,9 @@ API) or a local runtime (Ollama). Three ways, best first:
   | `openrouter-2` | `BENCHWARMERS_KEY_OPENROUTER_2` |
   | `my-hub` | `BENCHWARMERS_KEY_MY_HUB` |
 
-  Without the variable, keys come from the `llm` CLI's key store
-  (`llm keys set <name>` writes there). The variable wins when both exist.
+  Without the variable, keys come from llm's key file (`keys.json` in llm's
+  user folder, which the dashboard and `init --key` write; the `llm` CLI need
+  not be installed). The variable wins when both exist.
 
 One provider can be several endpoints (a pasted key and a sign-in, or two
 accounts); each needs its own name, and Benchwarmers suggests a free one.
@@ -107,7 +107,10 @@ Any OpenAI-compatible client asks for the model `bench/<collection>`:
 Every seat's answer comes back, plus a `bench` field with each seat's call id.
 Clients of the Anthropic, Ollama, Gemini and Responses APIs work too, with
 the server's address as their base URL (GATEWAY.md, "Other API formats");
-requests carrying tools are refused, since a panel answers with text.
+a panel answers with text, so requests carrying tools are refused there.
+An agent that needs tools (Claude Code, Codex, opencode) runs on a collection
+in route mode instead, `bench/<collection>:route`: one member per turn, with
+its tools, kept for the conversation (GATEWAY.md, "Route mode").
 For a chat window that shows one answer, ask for `bench/<collection>:merged`
 (or set the collection's reply to merged): a merger model writes one answer
 saying where the seats agreed and disagreed, and every seat is still in

package/GATEWAY.md

@@ -205,7 +205,7 @@ request becomes OpenAI-style messages, the panel runs exactly as on
 scope, rate caps, CORS, the call log), and the reply goes back in the
 format's own shape. The model field is the collection: `bench/core`, or
 `bench/core:merged` for one merged answer. Clients that pick from a list
-(`/api/tags`, `/v1beta/models`, `/v1/models`) see every collection in both
+(`/api/tags`, `/v1beta/models`, `/v1/models`) see every collection in all three
 reply modes.
 
 | Format | Routes | Streams with | `bench` field | Known clients |
@@ -269,12 +269,15 @@ bytes), Gemini `inlineData` and `fileData` at an http(s) URL.
 **Refused, not dropped** (400, with why), because a client waiting for what
 it asked would hang or loop on a panel's text:
 
-- **Tools and agents are not supported yet.** A request carrying tools, or a
-  conversation holding tool calls or tool results, in any format (chat
-  completions included), is refused with a message saying a route mode is
-  coming: one member proxied with tools and streaming, for agent clients
-  such as Claude Code, Codex or opencode, with background shadow seats as an
-  opt-in after that. An empty `tools` list is no tools.
+- **Tools, in a panel.** A request carrying tools, or a conversation
+  holding tool calls or tool results, in any format, is refused by a panel
+  (`all` or `merged`) with a message pointing at route mode (next section),
+  which takes them. An empty `tools` list is no tools.
+- a tool only its own vendor runs (Anthropic's server tools such as
+  `web_search_20250305` and their result blocks, OpenAI's hosted tools such
+  as `web_search`, Gemini's Google Search, URL context and code execution),
+  in route mode too: no other model can run them;
+- the legacy `functions` / `function_call` fields: send `tools`;
 - a PDF, audio or other non-image part; an image as an uploaded file id
   (Anthropic `file`, Responses `file_id`, Gemini's Files API);
 - `previous_response_id` or a stored `conversation` on Responses: nothing is
@@ -282,12 +285,166 @@ it asked would hang or loop on a panel's text:
 - Ollama's `context` (a model's own token state): use `/api/chat`;
 - more than one prompt on legacy completions.
 
-**Ignored**: sampling options (`max_tokens`, `temperature`, `top_p`, `stop`
-and their native names) are read but each seat keeps its own call options;
-structured-output settings (Ollama `format`, Gemini `responseSchema`,
-Responses `text.format`), since a panel answers in markdown; a model's own
+**Ignored by a panel**: sampling options (`max_tokens`, `temperature`,
+`top_p`, `stop` and their native names) are read but each seat keeps its own
+call options; structured-output settings (Ollama `format`, Gemini
+`responseSchema`, Responses `text.format`), since a panel answers in
+markdown. Route mode passes both on. **Ignored everywhere**: a model's own
 reasoning from an earlier turn (Anthropic thinking blocks, Responses
-reasoning items, Gemini thought parts).
+reasoning items, Gemini thought parts), and native parameters with no
+OpenAI counterpart (Anthropic's `top_k`, `thinking`, `metadata`).
+
+## Route mode (agent clients)
+
+A panel answers with several models' text; an agent needs one model that
+calls its tools. **Route mode** is that: no panel, one member of the
+collection gets the request as sent — messages, tool calls and results,
+`tools`, `tool_choice`, `parallel_tool_calls`, `response_format`,
+temperature and the rest, images — and its answer is relayed in the
+request's own format, streamed or not. Ask for it with `:route` on the
+model (`bench/core:route`), `"bench": {"reply": "route"}`, or set a
+collection's reply to *route* in the Collections dialog. `GET /v1/models`
+(and `/api/tags`, `/v1beta/models`) lists every collection in all three
+modes, so an agent's model picker can choose `bench/core:route`.
+
+**Who answers.** Members only, no explorers: an agent's working session is
+not the place to try an unknown model. Excluded, and named in
+`bench.skipped` with why, as a panel excludes them: retired and ad hoc
+members, members whose endpoint speaks another API than chat completions,
+members the catalog does not know to call tools (when the request has
+tools; the model popup's *Tool calling* is the override, like *Vision /
+OCR*; the catalog's `tools` comes from OpenRouter's listed parameters,
+models.dev, the endpoint's own listing or a capability probe that saw a
+call), that cannot see (when it has images) or whose context window is
+under the request (tools and tool calls counted). No member left: 400
+(413 when size is why). Of those left, the best record goes first: useful
+verdicts counted, then verified suite findings, then a connection already
+paid for; members cooling down after a rate limit go last. One new
+conversation in four goes instead to the member whose record is oldest, so
+the record does not narrow to one model.
+
+**Sticky.** A conversation stays on its model across turns: switching
+mid-task breaks the agent's coherence and the provider's prompt cache. The
+key is a hash of the collection, the gateway key, the system text and the
+first user message, which an agent resends unchanged every turn; it lives
+in memory and lapses after an hour idle (or a restart). A side request with
+its own system prompt (Claude Code's command checks, a subagent, a title) is
+its own conversation. If the conversation's model cannot take a turn (an
+image arrives), the turn is re-ranked; the model that answers becomes the
+conversation's.
+
+**Failover.** If the member fails before the first piece of its answer
+reaches the caller — a 429, a 5xx, a pacing refusal, nothing at all in 90
+seconds, an empty reply (even after reasoning) — the next member in order
+gets the turn, transparently, up to three; the caller sees only the answer,
+and `bench.failed_over` names who failed and why. Reasoning streamed before
+the answer is held back with it. A streamed turn's 200 waits up to 15
+seconds for that first piece, so a turn that fails everywhere quickly keeps
+its status (429 when every member was rate limited, else 502), then
+keep-alives cover the wait. After the first piece, an error goes into the
+stream in the format's error shape: no other model can continue a
+half-written answer.
+
+**Streams.** OpenAI chat completions relays the member's chunks as they
+came, tool-call deltas exactly, under the collection's name, then a closing
+chunk with no choice that carries `bench`. The other formats stream the
+text as it comes and each tool call **whole when the turn ends** (an
+Anthropic `tool_use` block with one `input_json_delta`, a Responses
+`function_call` item, a Gemini `functionCall` part, an Ollama `tool_calls`
+line), so calls whose deltas interleave still arrive intact. Usage is the
+member's own count. An output budget over the model's known maximum is cut
+to it (Claude Code asks every model for 32,000). Headers name the
+conversation (`X-Bench-Conversation`) and, when known before the 200, the
+model (`X-Bench-Model`).
+
+**Translations.** Anthropic `tool_use` / `tool_result` blocks, Responses
+`function_call` / `function_call_output` items, Gemini `functionCall` /
+`functionResponse` parts and Ollama `tool_calls` / `tool` messages become
+OpenAI tool calls and tool messages and back; tool definitions and
+`tool_choice` likewise (Gemini's upper-case schema types lowered). Tool call
+ids pass through both ways; Gemini and Ollama usually send none, so they
+are numbered per request and a result matched to the oldest open call of
+its name. A Responses **custom tool** (Codex's free-form `apply_patch`)
+becomes a function taking one string, `input`, its grammar in the
+description, and the call goes back as a `custom_tool_call`. An image in an
+Anthropic tool result goes in a user message after it (a tool message
+carries text only); `is_error` prefixes the result with `Error: `.
+
+**Recording (the decision).** Every call is in the usage ledger (purpose
+`route`), and every turn a line in `runs/gateway/routes.jsonl`: model,
+status, seconds, tool calls, tokens, who failed over — never the prompt,
+which an agent resends whole every turn. For the router, a conversation is
+**one unjudged offload per model that answered it** (call id
+`<conversation>.1`, `.2` after a failover to another model), not one per
+turn: an agent session is dozens of turns, and a seat per turn would bury
+the panels' verdicts. A verdict on a conversation is optional
+(`POST /bench/verdict` with that call id); the offload alone already counts
+toward rotation. The Gateway card lists route conversations in their own
+table (when, who asked, the first message, who answered with ✓/✗, turns,
+tool calls, failovers) and none of them adds to **Owed**.
+
+**Shadows (later, opt-in).** A shadow seat would answer the same turn in the
+background for evaluation only, after the caller has its answer. The place
+is marked in `route_turn` ("SHADOWS GO HERE"): each would be its own client
+call with purpose `shadow`, recorded on the conversation's call id; nothing
+the caller sees would wait for or change because of them. Not built.
+
+**Limits, honestly.**
+
+- Members reached through native Anthropic, Gemini or Responses endpoints
+  are not routed to: route mode speaks chat completions to the member.
+- The member is a free or flat-rate model, not the model the client was
+  built for. Claude Code and Codex are tuned for their own vendors' models;
+  smaller models make more tool-call mistakes, and some providers reject a
+  tool schema another accepts (that request fails over).
+- Prompt caching is the provider's own: stickiness keeps the prefix on one
+  provider, but `cache_control` marks are not passed on.
+- Thinking is not shown to Anthropic, Responses, Gemini or Ollama clients
+  (a model's reasoning is relayed raw only in chat completions).
+- Tool calls arrive whole, not as deltas, outside chat completions.
+- A caller that hangs up does not stop the member; its answer runs out and
+  is recorded.
+- Stickiness is per gateway process, in memory.
+
+### Pointing agents at a collection
+
+The base URL is the server: `http://127.0.0.1:8122` (the dashboard's port)
+or `:8123` (`gateway.py`). On this machine any key text works; elsewhere,
+or where keys are required, use a gateway key from Settings.
+
+**Claude Code** (Anthropic Messages). It sends the model named by
+`ANTHROPIC_MODEL` (or `--model`) for its main loop and a smaller "haiku"
+model for side tasks (command checks, titles), so set both; a name it does
+not know prints a harmless `unrecognized_model` note:
+
+```bash
+export ANTHROPIC_BASE_URL=http://127.0.0.1:8122
+export ANTHROPIC_API_KEY=<a gateway key, or any text on this machine>
+export ANTHROPIC_MODEL=bench/core:route
+export ANTHROPIC_DEFAULT_HAIKU_MODEL=bench/fast:route
+export ANTHROPIC_DEFAULT_SONNET_MODEL=bench/core:route ANTHROPIC_DEFAULT_OPUS_MODEL=bench/core:route
+export CLAUDE_CODE_SUBAGENT_MODEL=bench/core:route
+claude
+```
+
+Checked end to end (2026-10-23): Claude Code 2.1.299 ran Read, Write and
+Bash tool loops through `bench/core:route` and `bench/fast:route`, failing
+over past a member's 429 on the first turn and staying on one model after.
+
+**Codex and other Responses clients**: a provider with `base_url =
+"http://127.0.0.1:8122/v1"`, `wire_api = "responses"`, the model
+`bench/core:route`, and the gateway key as its API key. Function and custom
+tools work; hosted tools (`web_search`, `local_shell`) are refused, so turn
+them off. Nothing is stored: `previous_response_id` is refused, so the
+client must send the whole conversation every turn.
+Not checked against a real Codex yet.
+
+**opencode, Cline, Continue, Roo, Aider and other OpenAI-compatible agents**:
+an OpenAI-compatible provider at `http://127.0.0.1:8122/v1`, model
+`bench/core:route` (it is in `/v1/models`). Agents that speak Ollama
+(`http://127.0.0.1:8122`, model `bench/core:route`) work the same way. A
+real `curl` with tools streamed a model's tool-call deltas through
+unchanged; these clients themselves were not run.
 
 **Counts and durations.** `usage` and the token-count routes
 (`/v1/messages/count_tokens`, Gemini `:countTokens`) are the size check's

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 22 October 2026.*
+*Effective 23 October 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -22,13 +22,15 @@ Wendell Research.
 - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows
   from Hugging Face and source files from GitHub.
 
-Keys and sign-in tokens stay in your environment or the `llm` CLI's key
-store. [SECURITY.md](SECURITY.md) has the full detail.
+Keys and sign-in tokens stay in your environment or llm's key file
+(`keys.json` in llm's user folder, used whether or not the `llm` CLI is
+installed). [SECURITY.md](SECURITY.md) has the full detail.
 
 ## What stays on your machine
 
-Everything else: your endpoints, collections, verdicts, call logs, probe
-history and task answers live in your data folder, which is yours to back up,
+Everything else: your endpoints, collections, verdicts, call logs, the usage
+ledger (tokens per call, never the prompt), probe history and task answers
+live in your data folder, which is yours to back up,
 move or delete.
 
 ## Third parties

package/README.md

@@ -10,8 +10,14 @@
   <a href="https://www.npmjs.com/package/@wendellresearch-ai/benchwarmers"><img src="https://img.shields.io/npm/v/@wendellresearch-ai/benchwarmers?color=cb3837&logo=npm" alt="npm"></a>
   <a href="https://pypi.org/project/benchwarmers/"><img src="https://img.shields.io/pypi/v/benchwarmers?color=3775a9&logo=pypi&logoColor=white" alt="PyPI"></a>
   <a href="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml"><img src="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml/badge.svg" alt="Checks"></a>
-  <a href="#docker"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
-  <a href="GATEWAY.md"><img src="https://img.shields.io/badge/OpenAI--compatible-gateway-111111" alt="OpenAI-compatible gateway"></a>
+  <a href="#docker-and-hosting"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
+  <br>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-OpenAI_%7C_Responses-111111" alt="OpenAI chat completions and Responses API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Anthropic_Messages-111111" alt="Anthropic Messages API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Gemini-111111" alt="Gemini API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Ollama-111111" alt="Ollama API"></a>
+  <a href="GATEWAY.md#route-mode-agent-clients"><img src="https://img.shields.io/badge/agents-tool_calling-111111" alt="Agent clients with tool calling, through route mode"></a>
+  <a href="https://registry.modelcontextprotocol.io/v0/servers?search=io.github.WendellResearch-AI/benchwarmers"><img src="https://img.shields.io/badge/MCP-Registry-111111" alt="MCP server in the official MCP Registry"></a>
   <br>
   <a href="#try-it"><img src="https://img.shields.io/badge/Python-3.11_%7C_3.12_%7C_3.13-3776AB?logo=python&logoColor=white" alt="Python 3.11 to 3.13"></a>
   <a href="#try-it"><img src="https://img.shields.io/badge/core-stdlib_only-2ea44f" alt="The core needs only the Python standard library"></a>
@@ -30,8 +36,12 @@ verdicts decide who sits on the next panel.
 
 **What it is best at:**
 
-- **Asking several models at once.** One OpenAI-compatible request, a panel of
-  answers. Where they disagree is where to look.
+- **Asking several models at once.** One request, in the OpenAI, Anthropic,
+  Gemini or Ollama format, a panel of answers. Where they disagree is where
+  to look.
+- **Agents too.** Claude Code and other tool-calling clients use a collection
+  through route mode: one member per turn, kept for the conversation, with
+  failover, and every call on the record.
 - **Reviewing.** Code review, claims about code, designs: different models
   catch different things, and two models agreeing is not proof.
 - **Playing to each model's strengths.** A collection per kind of work
@@ -43,12 +53,13 @@ verdicts decide who sits on the next panel.
   repositories, plus your verdicts on real work, instead of public leaderboards.
 - **Knowing who to bench.** Who is confidently wrong, who has stopped
   answering (retired automatically), and who is resting after a rate limit.
-- **Keeping free and cheap models useful.** Health probes, learned rate limits
-  and one endpoint for every app, over any OpenAI-compatible provider or hub.
+- **Keeping free and cheap models useful.** Health probes, rate limits taken
+  from what providers say and enforced, and one endpoint for every app, over
+  any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
 
 | Related tools | Examples | What they do | What Benchwarmers does differently |
 |---|---|---|---|
-| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer |
+| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record |
 | Councils and ensembles | LLM Council, Mixture-of-Agents | Several models merged into one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
 | Side-by-side chat | Open WebUI, LMArena | Several answers shown together | Records a verdict per answer and uses them to pick the next panel |
 | Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
@@ -68,7 +79,7 @@ verdicts decide who sits on the next panel.
 - [From the command line](#from-the-command-line)
 - [From Claude Code and other agents](#from-claude-code-and-other-agents)
 - [The suite](#the-suite)
-- [Docker](#docker)
+- [Docker and hosting](#docker-and-hosting)
 - [What leaves your machine](#what-leaves-your-machine)
 - [Limits](#limits)
 - [Security](#security)
@@ -82,7 +93,7 @@ with [AGENTS.md](AGENTS.md).
 
 ## Try it
 
-**v0.2, early.** It runs with Python 3.11 or later on Windows, macOS and
+**v0.3, early.** It runs with Python 3.11 or later on Windows, macOS and
 Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
 code where these pages are thin.
 
@@ -108,12 +119,12 @@ dependencies either way: the app is the Python standard library.
 
        git clone https://github.com/WendellResearch-AI/benchwarmers
        cd benchwarmers
-       python -m venv .venv
-       .venv/bin/pip install llm        # Windows: .venv\Scripts\pip install llm
+       python -m venv .venv             # nothing to pip install: the standard library is enough
 
-   The [`llm`](https://llm.datasette.io/) CLI is the default model client today.
-   The built-in client (`metrics/client.py`) replaces it, chosen on Settings
-   or with `BENCH_CLIENT=builtin` (which wins), and will become the default.
+   The built-in model client (`metrics/client.py`) makes every call. The
+   [`llm`](https://llm.datasette.io/) CLI is an alternative for its plugins:
+   install it in the venv and choose it on Settings (or `BENCH_CLIENT=llm`,
+   which wins).
 
 2. **Make a data folder.** Everything that is yours (endpoints, model
    registrations, collections, verdicts, probe history, task answers) lives
@@ -125,14 +136,16 @@ dependencies either way: the app is the Python standard library.
    unless you name one) and points the checkout at it with a
    `.benchwarmers-home` file; `BENCHWARMERS_HOME` overrides that. With
    `--endpoint NAME BASE_URL --key` it also adds a first endpoint and asks for
-   its key, which goes to the `llm` CLI's key store (`llm keys set NAME` does
-   the same), never into the data folder.
+   its key, which goes to the key file in llm's user folder (`keys.json`,
+   shared with the `llm` CLI if you have it, not needed otherwise), never into
+   the data folder. `BENCHWARMERS_KEY_<NAME>` in the environment works too.
 
 3. **Start the dashboard.** With no endpoint yet it opens on adding one: sign
    in with a provider on its own page (OpenRouter, NanoGPT, Hugging Face,
    Pollinations, your ChatGPT plan, GitHub Models, or a local Ollama), or paste
    an API key for any OpenAI-compatible API, a hub such as Open WebUI or
-   LiteLLM included. Only providers that publish a sign-in for other apps are
+   LiteLLM included. Anthropic and Gemini keys can also be called through
+   those providers' own APIs (the API choice in the same dialog). Only providers that publish a sign-in for other apps are
    offered; adding one is an entry in `metrics/connect.py` (its docstring says
    how).
 
@@ -140,7 +153,9 @@ dependencies either way: the app is the Python standard library.
 
 4. **Onboard a few models** from it ("Onboard a model"): tick one or several;
    each is registered, probed and run through the suite, one at a time, with
-   the progress beside the button.
+   the progress beside the button. **Quick** probes now and leaves the suite
+   to the scheduled passes; **Quick + ad hoc** also keeps the bench from ever
+   calling the model on its own, for a metered or trial key.
 
 5. **Make a collection** on the Collections page (the models a panel always
    includes, and what it is for). The dashboard answers the gateway's routes on
@@ -163,10 +178,16 @@ dependencies either way: the app is the Python standard library.
 - **Health.** Every morning each model gets a one-word prompt; once a week, a
   short and a long one for speed. A model that stops answering, or that its
   endpoint no longer serves, is retired automatically after one confirming
-  call, with the evidence kept, and can be restored in one click. Rate limits
-  are learned from providers' own error messages and kept per connection,
-  since models behind one key share them. The built-in scheduler runs all of
-  this once Settings switches it on; a fresh install spends no calls until then.
+  call, with the evidence kept, and can be restored in one click. The
+  built-in scheduler runs all of this once Settings switches it on; a fresh
+  install spends no calls until then.
+- **Rate limits and usage.** Limits come from providers' rate-limit headers
+  and error messages, and from what you set per connection or model
+  (requests a minute, day or month; tokens a minute or day). They are kept per
+  connection, since models behind one key share them, and enforced: a call
+  waits when the wait fits, or is refused before it goes out. Every call lands
+  in a monthly usage ledger with its tokens, and the Quotas page shows what is
+  left and when it resets.
 - **Discover.** models.dev and FreeLLMAPI, merged into one list of models
   worth adding, each with the routes this install has to it.
 
@@ -206,10 +227,11 @@ that speaks one of them works with the server's address as its base URL:
       -d '{"model": "bench/<collection>", "max_tokens": 1024, "messages": [{"role": "user", "content": "..."}]}'
 
 Each streams when asked: the reply starts at once, and each seat's answer
-arrives as that seat finishes. Agent and tool use is not supported yet: a
-request carrying tools is refused with a 400 rather than answered without
-them. GATEWAY.md, "Other API formats", has a curl for each and what is
-refused or ignored.
+arrives as that seat finishes. A panel answers with text, so a request
+carrying tools is refused with a 400 that points to **route mode**
+(`bench/<collection>:route`, one member with its tools; see
+[below](#from-claude-code-and-other-agents)). GATEWAY.md, "Other API formats",
+has a curl for each and what is refused or ignored.
 
 ## From the command line
 
@@ -294,6 +316,16 @@ On native Windows, a client that cannot start `npx` itself needs
 | `BENCHWARMERS_KEY` | a gateway key from Settings, sent as `Authorization: Bearer`; needed for a server on another machine, or where keys are required |
 | `BENCHWARMERS_TIMEOUT` | seconds to wait for a panel, default 900 |
 
+**An agent running on a collection.** The MCP tools ask a panel for a
+second opinion; an agent can also *be driven by* a collection, in **route
+mode** (`bench/<collection>:route`): one member, picked by its record, gets
+each turn with its tools, stays on for the whole conversation, and is
+replaced by the next member if it fails before answering. Claude Code
+points at it with `ANTHROPIC_BASE_URL=http://127.0.0.1:8122` and
+`ANTHROPIC_MODEL=bench/core:route`; Codex, opencode, Cline and other
+OpenAI-, Responses- or Ollama-speaking agents with the same base URL.
+[GATEWAY.md](GATEWAY.md), "Route mode", has the set-up and the limits.
+
 <!-- The MCP Registry's proof that the PyPI package is this server (server.json):
 mcp-name: io.github.WendellResearch-AI/benchwarmers -->
 
@@ -314,7 +346,7 @@ code makes the best tasks, because no model has trained on it. See
 [suite/README.md](suite/README.md), [suite/REBENCH.md](suite/REBENCH.md) and
 [BENCHMARKS.md](BENCHMARKS.md).
 
-## Docker
+## Docker and hosting
 
 One container runs the dashboard, the gateway (on the same port, at `/v1`)
 and the scheduler, with the data folder as a volume and no `llm` CLI (the
@@ -356,21 +388,33 @@ reaches Wendell Research. What it sends, and where:
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the
   SWE-rebench-V2 rows from Hugging Face and the files from GitHub.
 
-Keys stay in the environment or the `llm` CLI's key store, never in the data
-folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
+Keys stay in the environment or llm's key file (`keys.json` in its user
+folder), never in the data folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
 detail.
 
 ## Limits
 
-- **No tools yet.** A panel answers with text, not a tool call, so agents
-  that need tool use (Claude Code, Codex, opencode as callers) wait for a
-  route mode; requests with tools are refused, not trimmed.
-- **Early.** v0.1: no installer yet, merged replies stream in one piece, and the LiteLLM and
-  CLI Proxy API hub adapters are built from those projects' source, not yet
-  run against a real hub.
+- **Tools go through route mode only.** A panel answers with text; a request
+  with tools is refused there, not trimmed. Route mode passes turns only to
+  members on chat-completions endpoints, keeps its sticky conversations in
+  memory (a restart starts each on a fresh pick), and has been run end to end
+  with Claude Code; the Codex, opencode and Cline set-ups are written from
+  their docs. Shadow seats (other members answering an agent's turn for
+  evaluation) are planned, not built.
+- **Some parts are checked against fakes only.** The native Anthropic and
+  Gemini endpoints, and the LiteLLM and CLI Proxy API hub adapters, are built
+  from those APIs' documentation and source and tested against fake servers,
+  not yet against the real services.
+- **A merged reply arrives in one piece** once the merger has finished; the
+  seats before it stream as they finish.
+- **Pacing and the ledger cover the built-in client.** Calls made through the
+  `llm` CLI, when you choose it, are neither paced nor counted.
 - **Calls cost.** Probes, onboarding and the suite run on your keys and free
-  tiers. Onboarding runs one model at a time for that reason; a suite pass is
-  paced, not burst.
+  tiers. Onboarding runs one model at a time for that reason (Quick spreads
+  the suite over the scheduled passes); a suite pass is paced, not burst.
+- **The scheduler lives in the server.** Its jobs run while the dashboard (or
+  the container, or the service) is up; a missed run is made up once, within
+  seven days, when it comes back.
 - **One install, one person** at heart. Logins and gateway keys exist for a
   LAN or a server, but there are no per-user views or roles.
 - **The onboarding queue lives in the dashboard's memory.** Restarting it

package/SECURITY.md

@@ -58,7 +58,8 @@ open itself to anyone.
 ## What Benchwarmers touches
 
 - **Keys.** Endpoint keys come from `BENCHWARMERS_KEY_<NAME>` (the key name
-  upper-cased, `-` as `_`) first, then from the `llm` CLI's key store. They are
+  upper-cased, `-` as `_`) first, then from llm's key file (`keys.json` in
+  its user folder, used whether or not the `llm` CLI is installed). They are
   never written to the data folder, never shown back, and never put in a log.
   Sign-ins through a provider's page store their token the same way.
 - **The network.** Your prompts go only to the endpoints you add; the probes,
@@ -68,9 +69,9 @@ open itself to anyone.
   dashboard's page, and Hugging Face and GitHub only when you run
   `suite/rebench.py`. [PRIVACY.md](PRIVACY.md) lists them.
 - **Files.** It writes inside its data folder (`BENCHWARMERS_HOME`, or the
-  folder `.benchwarmers-home` names) and, in the `llm` CLI's own folder, its
-  key store (when you paste a key or sign in) and its model list (when it
-  registers a model). Images sent to the gateway are never written to a log.
+  folder `.benchwarmers-home` names) and, in llm's user folder, the key file
+  (when you paste a key or sign in) and the model list (when it registers a
+  model). Images sent to the gateway are never written to a log.
 - **Processes.** Onboarding, the probes and the suite run this repository's
   own scripts with the same Python. Model ids that reach a command line are
   checked against a strict pattern first.

package/USAGE.md

@@ -47,8 +47,19 @@ Dismiss hides one you never want offered (an image model, say); the dismissed
 list brings it back.
 
 Each model is registered, probed with four short calls, and, if it answers,
-run through the suite. That takes minutes and spends calls, so they run **one
-at a time**, in a queue the dashboard keeps:
+run through the suite. **Onboard as**, under the list, chooses how much that
+spends:
+
+- **Full**: the probe, then the suite now.
+- **Quick**: the probe now; the scheduled suite passes fill the suite in, six
+  tasks a run. The same calls, spread out.
+- **Quick + ad hoc**: the probe only, and the bench never calls the model on
+  its own (no explorer seat, no scheduled runs). The one that saves calls, for
+  a metered, capped or trial key. A collection that lists it still calls it.
+
+`bench onboard <served> --quick [--ad-hoc]` does the same from a terminal.
+Onboarding takes minutes and spends calls, so models run **one at a time**,
+in a queue the dashboard keeps:
 
 - One model with nothing ahead of it is followed step by step in the popup.
 - Otherwise the popup shows the queue, and a chip beside the button shows the
@@ -62,6 +73,12 @@ at a time**, in a queue the dashboard keeps:
 A model that registered but did not answer stays on the bench as
 unavailable, so the daily probe keeps watching it.
 
+**Roles.** A model's popup sets what it is for: `general` (the default),
+`code`, `light` (small and fast), `vision` (reads images well, weak at text:
+OCR and document models; seated only when a request carries images) or
+`image` (makes images; never seated). A model that reads images and does
+everything else well stays `general`.
+
 ## Collections
 
 A collection is a named panel: the models it always includes, the shape of
@@ -116,9 +133,24 @@ its base URL and a collection as its model:
 | Older tools on `/v1/completions` | `http://127.0.0.1:8122/v1` | one `prompt` |
 
 Asked to stream, each answers at once and sends each seat's answer as it
-finishes. Agents that need tools (Claude Code, Codex, opencode as callers)
-are not supported yet: a request with tools is refused with a 400 saying so.
-GATEWAY.md, "Other API formats", has a curl for each.
+finishes. GATEWAY.md, "Other API formats", has a curl for each.
+
+**An agent that calls tools** (Claude Code, Codex, opencode, Cline...) uses
+a collection's **route mode**: model `bench/<collection>:route`. No panel:
+one member, picked by its record, gets the request with its tools and
+streams its answer back in the agent's own format; the conversation stays
+on that model, and a member that fails before answering is replaced by the
+next. A panel refuses tools with a 400 saying so. For Claude Code:
+
+```bash
+export ANTHROPIC_BASE_URL=http://127.0.0.1:8122 ANTHROPIC_API_KEY=local
+export ANTHROPIC_MODEL=bench/core:route ANTHROPIC_DEFAULT_HAIKU_MODEL=bench/fast:route
+claude
+```
+
+Each conversation is one offload per model that answered it; a verdict is
+optional, and the Gateway card lists route conversations apart from the
+panels. GATEWAY.md, "Route mode", has the other clients and the limits.
 
 **A coding agent** connects over MCP instead: the Claude Code plugin
 (`/plugin marketplace add WendellResearch-AI/benchwarmers`, then

package/VERSION

@@ -1 +1 @@
-0.2.2
+0.3.1

package/cli.py

@@ -615,7 +615,7 @@ def parser() -> argparse.ArgumentParser:
     c.add_argument("--shape", help="the task shape explorers are picked for; empty for automatic")
     c.add_argument("--description", help="what it is for: callers read it to choose")
     c.add_argument("--tags", help="comma-separated")
-    c.add_argument("--reply", choices=["all", "merged"])
+    c.add_argument("--reply", choices=["all", "merged", "route"])
     c.add_argument("--merger", help="the alias that merges, for --reply merged")
 
     cmd("endpoints", cmd_endpoints, "where models are served from")