@wendellresearch-ai/benchwarmers 0.3.0 → 0.3.1

package/AGENTS.md

@@ -25,12 +25,11 @@ Python 3.11 or later; nothing else is required.
     git clone https://github.com/WendellResearch-AI/benchwarmers
     cd benchwarmers
     python -m venv .venv
-    .venv/bin/pip install llm          # optional; Windows: .venv\Scripts\pip install llm
 
-The `llm` CLI is today's default model client. Without it, choose the
-built-in client (`metrics/client.py`), which needs nothing installed: on
-Settings, Model client, once the dashboard runs, or with
-`BENCH_CLIENT=builtin` (which wins over Settings).
+The built-in model client (`metrics/client.py`) makes every call and needs
+nothing installed. The `llm` CLI is an optional alternative, for its plugins:
+`pip install llm` in the venv, then choose it on Settings, Model client, or
+with `BENCH_CLIENT=llm` (which wins over Settings).
 
 Check: `python scripts/check.py --quick` ends with `all checks pass`.
 
@@ -67,8 +66,9 @@ API) or a local runtime (Ollama). Three ways, best first:
   | `openrouter-2` | `BENCHWARMERS_KEY_OPENROUTER_2` |
   | `my-hub` | `BENCHWARMERS_KEY_MY_HUB` |
 
-  Without the variable, keys come from the `llm` CLI's key store
-  (`llm keys set <name>` writes there). The variable wins when both exist.
+  Without the variable, keys come from llm's key file (`keys.json` in llm's
+  user folder, which the dashboard and `init --key` write; the `llm` CLI need
+  not be installed). The variable wins when both exist.
 
 One provider can be several endpoints (a pasted key and a sign-in, or two
 accounts); each needs its own name, and Benchwarmers suggests a free one.
@@ -107,7 +107,10 @@ Any OpenAI-compatible client asks for the model `bench/<collection>`:
 Every seat's answer comes back, plus a `bench` field with each seat's call id.
 Clients of the Anthropic, Ollama, Gemini and Responses APIs work too, with
 the server's address as their base URL (GATEWAY.md, "Other API formats");
-requests carrying tools are refused, since a panel answers with text.
+a panel answers with text, so requests carrying tools are refused there.
+An agent that needs tools (Claude Code, Codex, opencode) runs on a collection
+in route mode instead, `bench/<collection>:route`: one member per turn, with
+its tools, kept for the conversation (GATEWAY.md, "Route mode").
 For a chat window that shows one answer, ask for `bench/<collection>:merged`
 (or set the collection's reply to merged): a merger model writes one answer
 saying where the seats agreed and disagreed, and every seat is still in

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 22 October 2026.*
+*Effective 23 October 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -22,13 +22,15 @@ Wendell Research.
 - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows
   from Hugging Face and source files from GitHub.
 
-Keys and sign-in tokens stay in your environment or the `llm` CLI's key
-store. [SECURITY.md](SECURITY.md) has the full detail.
+Keys and sign-in tokens stay in your environment or llm's key file
+(`keys.json` in llm's user folder, used whether or not the `llm` CLI is
+installed). [SECURITY.md](SECURITY.md) has the full detail.
 
 ## What stays on your machine
 
-Everything else: your endpoints, collections, verdicts, call logs, probe
-history and task answers live in your data folder, which is yours to back up,
+Everything else: your endpoints, collections, verdicts, call logs, the usage
+ledger (tokens per call, never the prompt), probe history and task answers
+live in your data folder, which is yours to back up,
 move or delete.
 
 ## Third parties

package/README.md

@@ -10,8 +10,14 @@
   <a href="https://www.npmjs.com/package/@wendellresearch-ai/benchwarmers"><img src="https://img.shields.io/npm/v/@wendellresearch-ai/benchwarmers?color=cb3837&logo=npm" alt="npm"></a>
   <a href="https://pypi.org/project/benchwarmers/"><img src="https://img.shields.io/pypi/v/benchwarmers?color=3775a9&logo=pypi&logoColor=white" alt="PyPI"></a>
   <a href="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml"><img src="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml/badge.svg" alt="Checks"></a>
-  <a href="#docker"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
-  <a href="GATEWAY.md"><img src="https://img.shields.io/badge/OpenAI--compatible-gateway-111111" alt="OpenAI-compatible gateway"></a>
+  <a href="#docker-and-hosting"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
+  <br>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-OpenAI_%7C_Responses-111111" alt="OpenAI chat completions and Responses API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Anthropic_Messages-111111" alt="Anthropic Messages API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Gemini-111111" alt="Gemini API"></a>
+  <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Ollama-111111" alt="Ollama API"></a>
+  <a href="GATEWAY.md#route-mode-agent-clients"><img src="https://img.shields.io/badge/agents-tool_calling-111111" alt="Agent clients with tool calling, through route mode"></a>
+  <a href="https://registry.modelcontextprotocol.io/v0/servers?search=io.github.WendellResearch-AI/benchwarmers"><img src="https://img.shields.io/badge/MCP-Registry-111111" alt="MCP server in the official MCP Registry"></a>
   <br>
   <a href="#try-it"><img src="https://img.shields.io/badge/Python-3.11_%7C_3.12_%7C_3.13-3776AB?logo=python&logoColor=white" alt="Python 3.11 to 3.13"></a>
   <a href="#try-it"><img src="https://img.shields.io/badge/core-stdlib_only-2ea44f" alt="The core needs only the Python standard library"></a>
@@ -30,8 +36,12 @@ verdicts decide who sits on the next panel.
 
 **What it is best at:**
 
-- **Asking several models at once.** One OpenAI-compatible request, a panel of
-  answers. Where they disagree is where to look.
+- **Asking several models at once.** One request, in the OpenAI, Anthropic,
+  Gemini or Ollama format, a panel of answers. Where they disagree is where
+  to look.
+- **Agents too.** Claude Code and other tool-calling clients use a collection
+  through route mode: one member per turn, kept for the conversation, with
+  failover, and every call on the record.
 - **Reviewing.** Code review, claims about code, designs: different models
   catch different things, and two models agreeing is not proof.
 - **Playing to each model's strengths.** A collection per kind of work
@@ -43,8 +53,9 @@ verdicts decide who sits on the next panel.
   repositories, plus your verdicts on real work, instead of public leaderboards.
 - **Knowing who to bench.** Who is confidently wrong, who has stopped
   answering (retired automatically), and who is resting after a rate limit.
-- **Keeping free and cheap models useful.** Health probes, learned rate limits
-  and one endpoint for every app, over any OpenAI-compatible provider or hub.
+- **Keeping free and cheap models useful.** Health probes, rate limits taken
+  from what providers say and enforced, and one endpoint for every app, over
+  any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
 
 | Related tools | Examples | What they do | What Benchwarmers does differently |
 |---|---|---|---|
@@ -68,7 +79,7 @@ verdicts decide who sits on the next panel.
 - [From the command line](#from-the-command-line)
 - [From Claude Code and other agents](#from-claude-code-and-other-agents)
 - [The suite](#the-suite)
-- [Docker](#docker)
+- [Docker and hosting](#docker-and-hosting)
 - [What leaves your machine](#what-leaves-your-machine)
 - [Limits](#limits)
 - [Security](#security)
@@ -82,7 +93,7 @@ with [AGENTS.md](AGENTS.md).
 
 ## Try it
 
-**v0.2, early.** It runs with Python 3.11 or later on Windows, macOS and
+**v0.3, early.** It runs with Python 3.11 or later on Windows, macOS and
 Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
 code where these pages are thin.
 
@@ -108,12 +119,12 @@ dependencies either way: the app is the Python standard library.
 
        git clone https://github.com/WendellResearch-AI/benchwarmers
        cd benchwarmers
-       python -m venv .venv
-       .venv/bin/pip install llm        # Windows: .venv\Scripts\pip install llm
+       python -m venv .venv             # nothing to pip install: the standard library is enough
 
-   The [`llm`](https://llm.datasette.io/) CLI is the default model client today.
-   The built-in client (`metrics/client.py`) replaces it, chosen on Settings
-   or with `BENCH_CLIENT=builtin` (which wins), and will become the default.
+   The built-in model client (`metrics/client.py`) makes every call. The
+   [`llm`](https://llm.datasette.io/) CLI is an alternative for its plugins:
+   install it in the venv and choose it on Settings (or `BENCH_CLIENT=llm`,
+   which wins).
 
 2. **Make a data folder.** Everything that is yours (endpoints, model
    registrations, collections, verdicts, probe history, task answers) lives
@@ -125,14 +136,16 @@ dependencies either way: the app is the Python standard library.
    unless you name one) and points the checkout at it with a
    `.benchwarmers-home` file; `BENCHWARMERS_HOME` overrides that. With
    `--endpoint NAME BASE_URL --key` it also adds a first endpoint and asks for
-   its key, which goes to the `llm` CLI's key store (`llm keys set NAME` does
-   the same), never into the data folder.
+   its key, which goes to the key file in llm's user folder (`keys.json`,
+   shared with the `llm` CLI if you have it, not needed otherwise), never into
+   the data folder. `BENCHWARMERS_KEY_<NAME>` in the environment works too.
 
 3. **Start the dashboard.** With no endpoint yet it opens on adding one: sign
    in with a provider on its own page (OpenRouter, NanoGPT, Hugging Face,
    Pollinations, your ChatGPT plan, GitHub Models, or a local Ollama), or paste
    an API key for any OpenAI-compatible API, a hub such as Open WebUI or
-   LiteLLM included. Only providers that publish a sign-in for other apps are
+   LiteLLM included. Anthropic and Gemini keys can also be called through
+   those providers' own APIs (the API choice in the same dialog). Only providers that publish a sign-in for other apps are
    offered; adding one is an entry in `metrics/connect.py` (its docstring says
    how).
 
@@ -140,7 +153,9 @@ dependencies either way: the app is the Python standard library.
 
 4. **Onboard a few models** from it ("Onboard a model"): tick one or several;
    each is registered, probed and run through the suite, one at a time, with
-   the progress beside the button.
+   the progress beside the button. **Quick** probes now and leaves the suite
+   to the scheduled passes; **Quick + ad hoc** also keeps the bench from ever
+   calling the model on its own, for a metered or trial key.
 
 5. **Make a collection** on the Collections page (the models a panel always
    includes, and what it is for). The dashboard answers the gateway's routes on
@@ -163,10 +178,16 @@ dependencies either way: the app is the Python standard library.
 - **Health.** Every morning each model gets a one-word prompt; once a week, a
   short and a long one for speed. A model that stops answering, or that its
   endpoint no longer serves, is retired automatically after one confirming
-  call, with the evidence kept, and can be restored in one click. Rate limits
-  are learned from providers' own error messages and kept per connection,
-  since models behind one key share them. The built-in scheduler runs all of
-  this once Settings switches it on; a fresh install spends no calls until then.
+  call, with the evidence kept, and can be restored in one click. The
+  built-in scheduler runs all of this once Settings switches it on; a fresh
+  install spends no calls until then.
+- **Rate limits and usage.** Limits come from providers' rate-limit headers
+  and error messages, and from what you set per connection or model
+  (requests a minute, day or month; tokens a minute or day). They are kept per
+  connection, since models behind one key share them, and enforced: a call
+  waits when the wait fits, or is refused before it goes out. Every call lands
+  in a monthly usage ledger with its tokens, and the Quotas page shows what is
+  left and when it resets.
 - **Discover.** models.dev and FreeLLMAPI, merged into one list of models
   worth adding, each with the routes this install has to it.
 
@@ -206,10 +227,11 @@ that speaks one of them works with the server's address as its base URL:
       -d '{"model": "bench/<collection>", "max_tokens": 1024, "messages": [{"role": "user", "content": "..."}]}'
 
 Each streams when asked: the reply starts at once, and each seat's answer
-arrives as that seat finishes. Agent and tool use is not supported yet: a
-request carrying tools is refused with a 400 rather than answered without
-them. GATEWAY.md, "Other API formats", has a curl for each and what is
-refused or ignored.
+arrives as that seat finishes. A panel answers with text, so a request
+carrying tools is refused with a 400 that points to **route mode**
+(`bench/<collection>:route`, one member with its tools; see
+[below](#from-claude-code-and-other-agents)). GATEWAY.md, "Other API formats",
+has a curl for each and what is refused or ignored.
 
 ## From the command line
 
@@ -324,7 +346,7 @@ code makes the best tasks, because no model has trained on it. See
 [suite/README.md](suite/README.md), [suite/REBENCH.md](suite/REBENCH.md) and
 [BENCHMARKS.md](BENCHMARKS.md).
 
-## Docker
+## Docker and hosting
 
 One container runs the dashboard, the gateway (on the same port, at `/v1`)
 and the scheduler, with the data folder as a volume and no `llm` CLI (the
@@ -366,21 +388,33 @@ reaches Wendell Research. What it sends, and where:
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the
   SWE-rebench-V2 rows from Hugging Face and the files from GitHub.
 
-Keys stay in the environment or the `llm` CLI's key store, never in the data
-folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
+Keys stay in the environment or llm's key file (`keys.json` in its user
+folder), never in the data folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
 detail.
 
 ## Limits
 
-- **No tools yet.** A panel answers with text, not a tool call, so agents
-  that need tool use (Claude Code, Codex, opencode as callers) wait for a
-  route mode; requests with tools are refused, not trimmed.
-- **Early.** v0.1: no installer yet, merged replies stream in one piece, and the LiteLLM and
-  CLI Proxy API hub adapters are built from those projects' source, not yet
-  run against a real hub.
+- **Tools go through route mode only.** A panel answers with text; a request
+  with tools is refused there, not trimmed. Route mode passes turns only to
+  members on chat-completions endpoints, keeps its sticky conversations in
+  memory (a restart starts each on a fresh pick), and has been run end to end
+  with Claude Code; the Codex, opencode and Cline set-ups are written from
+  their docs. Shadow seats (other members answering an agent's turn for
+  evaluation) are planned, not built.
+- **Some parts are checked against fakes only.** The native Anthropic and
+  Gemini endpoints, and the LiteLLM and CLI Proxy API hub adapters, are built
+  from those APIs' documentation and source and tested against fake servers,
+  not yet against the real services.
+- **A merged reply arrives in one piece** once the merger has finished; the
+  seats before it stream as they finish.
+- **Pacing and the ledger cover the built-in client.** Calls made through the
+  `llm` CLI, when you choose it, are neither paced nor counted.
 - **Calls cost.** Probes, onboarding and the suite run on your keys and free
-  tiers. Onboarding runs one model at a time for that reason; a suite pass is
-  paced, not burst.
+  tiers. Onboarding runs one model at a time for that reason (Quick spreads
+  the suite over the scheduled passes); a suite pass is paced, not burst.
+- **The scheduler lives in the server.** Its jobs run while the dashboard (or
+  the container, or the service) is up; a missed run is made up once, within
+  seven days, when it comes back.
 - **One install, one person** at heart. Logins and gateway keys exist for a
   LAN or a server, but there are no per-user views or roles.
 - **The onboarding queue lives in the dashboard's memory.** Restarting it

package/SECURITY.md

@@ -58,7 +58,8 @@ open itself to anyone.
 ## What Benchwarmers touches
 
 - **Keys.** Endpoint keys come from `BENCHWARMERS_KEY_<NAME>` (the key name
-  upper-cased, `-` as `_`) first, then from the `llm` CLI's key store. They are
+  upper-cased, `-` as `_`) first, then from llm's key file (`keys.json` in
+  its user folder, used whether or not the `llm` CLI is installed). They are
   never written to the data folder, never shown back, and never put in a log.
   Sign-ins through a provider's page store their token the same way.
 - **The network.** Your prompts go only to the endpoints you add; the probes,
@@ -68,9 +69,9 @@ open itself to anyone.
   dashboard's page, and Hugging Face and GitHub only when you run
   `suite/rebench.py`. [PRIVACY.md](PRIVACY.md) lists them.
 - **Files.** It writes inside its data folder (`BENCHWARMERS_HOME`, or the
-  folder `.benchwarmers-home` names) and, in the `llm` CLI's own folder, its
-  key store (when you paste a key or sign in) and its model list (when it
-  registers a model). Images sent to the gateway are never written to a log.
+  folder `.benchwarmers-home` names) and, in llm's user folder, the key file
+  (when you paste a key or sign in) and the model list (when it registers a
+  model). Images sent to the gateway are never written to a log.
 - **Processes.** Onboarding, the probes and the suite run this repository's
   own scripts with the same Python. Model ids that reach a command line are
   checked against a strict pattern first.

package/USAGE.md

@@ -47,8 +47,19 @@ Dismiss hides one you never want offered (an image model, say); the dismissed
 list brings it back.
 
 Each model is registered, probed with four short calls, and, if it answers,
-run through the suite. That takes minutes and spends calls, so they run **one
-at a time**, in a queue the dashboard keeps:
+run through the suite. **Onboard as**, under the list, chooses how much that
+spends:
+
+- **Full**: the probe, then the suite now.
+- **Quick**: the probe now; the scheduled suite passes fill the suite in, six
+  tasks a run. The same calls, spread out.
+- **Quick + ad hoc**: the probe only, and the bench never calls the model on
+  its own (no explorer seat, no scheduled runs). The one that saves calls, for
+  a metered, capped or trial key. A collection that lists it still calls it.
+
+`bench onboard <served> --quick [--ad-hoc]` does the same from a terminal.
+Onboarding takes minutes and spends calls, so models run **one at a time**,
+in a queue the dashboard keeps:
 
 - One model with nothing ahead of it is followed step by step in the popup.
 - Otherwise the popup shows the queue, and a chip beside the button shows the
@@ -62,6 +73,12 @@ at a time**, in a queue the dashboard keeps:
 A model that registered but did not answer stays on the bench as
 unavailable, so the daily probe keeps watching it.
 
+**Roles.** A model's popup sets what it is for: `general` (the default),
+`code`, `light` (small and fast), `vision` (reads images well, weak at text:
+OCR and document models; seated only when a request carries images) or
+`image` (makes images; never seated). A model that reads images and does
+everything else well stays `general`.
+
 ## Collections
 
 A collection is a named panel: the models it always includes, the shape of

package/VERSION

@@ -1 +1 @@
-0.3.0
+0.3.1

package/metrics/client.py

@@ -2,8 +2,9 @@
 
     python metrics/client.py <alias> "prompt"     # one call; prints what ask_once would return
 
-RELEASE-PLAN.md Phases 2a and 2b. A drop-in for `llm -m <alias>`, chosen on
-Settings or by BENCH_CLIENT=builtin (see chosen()), used by suite/run.py's
+RELEASE-PLAN.md Phases 2a and 2b. A drop-in for `llm -m <alias>`, and the
+default since v0.3 (`llm` stays a choice on Settings or BENCH_CLIENT=llm; see
+chosen()), used by suite/run.py's
 ask_once, and so the perf probe, the suite, retries and auto-retire's
 confirmation call, and by the gateway's seats. It reads the same
 registration (the yaml entry's model name, base, key name and headers), the
@@ -118,7 +119,6 @@ import json
 import os
 import pathlib
 import random
-import shutil
 import sys
 import threading
 import time
@@ -146,10 +146,11 @@ CHOICE = DATA / "config" / "client.json"           # the Settings page's choice
 
 def chosen() -> tuple[str, str]:
     """The configured client and what set it: BENCH_CLIENT ("environment"),
-    else the Settings page's choice ("settings"), else ("default") `llm` where
-    the llm CLI is installed and the built-in client where it is not, as in
-    an install from npm or a container. Read on every call, so a change
-    applies to the next one."""
+    else the Settings page's choice ("settings"), else ("default") the
+    built-in client, installed `llm` or not: since v0.3 the `llm` CLI is an
+    alternative chosen on Settings, no longer the default (its trial week:
+    RELEASE-PLAN Phase 2a). Read on every call, so a change applies to the
+    next one."""
     env = (os.environ.get("BENCH_CLIENT") or "").strip().lower()
     if env:
         return env, "environment"
@@ -157,7 +158,7 @@ def chosen() -> tuple[str, str]:
         saved = json.loads(CHOICE.read_text(encoding="utf-8")).get("client")
     except (OSError, ValueError, AttributeError):
         saved = None
-    return (saved, "settings") if saved in CLIENTS else ("llm" if shutil.which("llm") else "builtin", "default")
+    return (saved, "settings") if saved in CLIENTS else ("builtin", "default")
 
 
 def backend(alias: str = "") -> str:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wendellresearch-ai/benchwarmers",
-  "version": "0.3.0",
+  "version": "0.3.1",
   "description": "Self-hosted panel gateway for LLMs: ask a collection of models at once, keep every answer side by side, rotate explorer models in, and let your verdicts pick the next panel.",
   "mcpName": "io.github.WendellResearch-AI/benchwarmers",
   "license": "MIT",

package/scripts/test_serve_edit.py

@@ -291,7 +291,7 @@ def main() -> int:
     model_client.CHOICE = cfg / "client.json"
     env, real_which = os.environ.pop("BENCH_CLIENT", None), shutil.which
     try:
-        shutil.which = lambda name: "/usr/bin/" + name                 # llm installed: the default is llm
+        shutil.which = lambda name: "/usr/bin/" + name                 # llm installed: still built-in
         first = model_client.chosen()
         shutil.which = lambda name: None                               # not installed (npm, a container)
         bare = model_client.chosen()
@@ -310,7 +310,7 @@ def main() -> int:
         os.environ.pop("BENCH_CLIENT", None)
         if env is not None:
             os.environ["BENCH_CLIENT"] = env
-    ok = (first == ("llm", "default") and bare == ("builtin", "default") and made["ok"] and picked == (("builtin", "settings"), "builtin")
+    ok = (first == ("builtin", "default") and bare == ("builtin", "default") and made["ok"] and picked == (("builtin", "settings"), "builtin")
           and pinned == (("llm", "environment"), False) and not no_llm and not unknown)
     print(("ok   " if ok else "FAIL ") + "model client choice" + ("" if ok else f" -> {first} {made} {picked} {pinned}"))
     failures += 0 if ok else 1