@wendellresearch-ai/benchwarmers 0.2.1 → 0.3.0

package/AGENTS.md

@@ -105,6 +105,9 @@ Any OpenAI-compatible client asks for the model `bench/<collection>`:
       -d '{"model": "bench/<collection>", "messages": [{"role": "user", "content": "..."}]}'
 
 Every seat's answer comes back, plus a `bench` field with each seat's call id.
+Clients of the Anthropic, Ollama, Gemini and Responses APIs work too, with
+the server's address as their base URL (GATEWAY.md, "Other API formats");
+requests carrying tools are refused, since a panel answers with text.
 For a chat window that shows one answer, ask for `bench/<collection>:merged`
 (or set the collection's reply to merged): a merger model writes one answer
 saying where the seats agreed and disagreed, and every seat is still in
@@ -133,7 +136,9 @@ browser app on another origin also needs `BENCH_ALLOWED_ORIGINS`. Check the
 result on Settings, Server and access, which shows each of these as it stands
 (never a secret's value). In Docker: put the token and the `BENCHWARMERS_KEY_<NAME>` variables in
 a `.env` beside `compose.yaml`, then `docker compose up -d`; the dashboard and
-gateway share one port and the data folder is a volume.
+gateway share one port and the data folder is a volume. For a hosting platform
+(Fly.io, Railway, Render) or a VPS with Caddy, follow [HOSTING.md](HOSTING.md)
+and its templates under `deploy/`.
 
 ## When something looks wrong
 

package/GATEWAY.md

@@ -2,6 +2,9 @@
 
 Status: **built** (2026-10-16): `python gateway.py`, then point an OpenAI client
 at `http://127.0.0.1:8123/v1` with a collection as the model, e.g. `bench/core`.
+Since 2026-10-23 it also speaks Anthropic Messages, OpenAI Responses, Ollama,
+Gemini and OpenAI's legacy completions, and streams each seat as it finishes
+(Other API formats, Streaming).
 Diagram: `.archify/architecture-bench-gateway-20261016-182500/bench-gateway.html`.
 
 ## The idea in one paragraph
@@ -101,8 +104,15 @@ exactly as sent:
   plan) gets no images yet.
 - **A vision collection** is members that take images; `GET /v1/models` lists
   `vision` in its `together.capabilities` when every member does. An image
-  request on a collection with no shape takes the shape `vision` (general and
-  light models), so verdicts on image work are counted apart.
+  request on a collection with no shape takes the shape `vision` (general,
+  light and vision models), so verdicts on image work are counted apart.
+- **Vision specialists.** A model whose role is `vision` (set in its popup) is
+  good at reading images and weak at text-only work, such as an OCR or
+  document model. Only the `vision` shape admits it, and as a collection
+  member it sits only when the request carries images; on a text request it
+  is listed in `bench.skipped` and its seat goes to the next member or an
+  explorer. Named in `with`, it sits anyway. A model that reads images and
+  does everything else well stays `general`.
 - **Size.** The body cap is 20 MB: a photo as a data URL is 4/3 of its file,
   and 20 MB is the tightest whole-request limit among the big providers
   (Gemini's for inline data). Above it the gateway answers 413 and says to
@@ -126,7 +136,7 @@ exactly as sent:
 `choices[0].message.content` is markdown: one section per seat with its call
 id, status and time. Beside it, a `bench` field carries the same seats as data,
 so a caller need not parse the markdown (OpenAI clients ignore unknown fields;
-a streaming client gets it on the first chunk):
+a streaming client gets it on the last chunk, the one with the finish_reason):
 
 ```json
 "bench": {"panel": "p20261017-...", "collection": "bench/core", "shape": "review",
@@ -149,14 +159,303 @@ fits. It gets its own call id (`<panel>.merger`, in `bench.merger`, the
 offload log and the Gateway card) so a verdict can judge the merge; `bench`
 still carries every seat. If no merger has the context window for the
 request plus every answer, or it fails, the reply is every answer with a
-one-line note (`bench.note`). A streaming client gets it as one chunk, like
-any reply.
+one-line note (`bench.note`). A streaming client gets the merged answer in one
+piece once the merge is done (Streaming).
 
 A verdict is `POST /bench/verdict` with `{"call", "useful": true|false, "note",
 "real": [...], "false": [...]}`, `offload.py verdict <call id>`, or the verdict
 dialog in the dashboard's Collections tab, under Gateway (click a seat). Sending the same verdict
 twice writes nothing.
 
+## Streaming
+
+With `"stream": true` (or the format's own way of asking, below):
+
+- **The 200 goes at once**: as soon as the panel is seated, and at the
+  latest after a second (the shape picker can take longer). A refusal raised
+  before then (too large for a member, an image no seat can see, no seat
+  free) keeps its status and the format's error body, as without streaming.
+- **Keep-alives** go every 10 seconds while the seats think, in the format's
+  own form: an SSE comment for the OpenAI formats, Anthropic's `ping`, a
+  chunk with no text for Ollama and Gemini (their readers have no comments).
+  A client or proxy with an idle timeout keeps the connection.
+- **Every answer** (`reply` all): the heading, then each seat's section the
+  moment that seat finishes, then the verdict lines. The pieces are the same
+  as the non-streamed reply's, with the seats in the order they finished
+  rather than seat order.
+- **Merged**: the merged answer in one piece when the merge is done (or
+  every answer with its note, if the merge fails). Token by token waits for
+  the built-in client's streaming.
+- **An error after the 200** goes into the stream in the format's own error
+  shape (the OpenAI SDKs' `{"error": ...}` chunk, Anthropic's `error` event,
+  Responses' `error` and `response.failed`, Ollama's `{"error": ...}` line,
+  Gemini's error object), so a client raises rather than hangs.
+- **The `bench` field** rides on the last event where the format has room
+  (the table below). It is in the JSON on the wire; an SDK that parses into
+  fixed types may drop it (the Anthropic SDK's accumulated final message and
+  the `ollama` library's replies do; read the raw event, or the call ids,
+  which every reply's text carries). A caller that hangs up does not stop
+  the panel: every seat still answers and is recorded.
+
+## Other API formats
+
+Each is a thin adapter around the same panel (`metrics/formats.py`): the
+request becomes OpenAI-style messages, the panel runs exactly as on
+`/v1/chat/completions` (collections, explorers, size check, images, keys,
+scope, rate caps, CORS, the call log), and the reply goes back in the
+format's own shape. The model field is the collection: `bench/core`, or
+`bench/core:merged` for one merged answer. Clients that pick from a list
+(`/api/tags`, `/v1beta/models`, `/v1/models`) see every collection in all three
+reply modes.
+
+| Format | Routes | Streams with | `bench` field | Known clients |
+|---|---|---|---|---|
+| OpenAI chat completions | `POST /v1/chat/completions`, `GET /v1/models` | `"stream": true` (SSE) | in the reply; the last chunk | OpenAI SDKs, Open WebUI's OpenAI connection, most chat apps |
+| OpenAI Responses | `POST /v1/responses` | `"stream": true` (typed events) | in the response; `response.completed` | OpenAI SDKs' `responses.create`, Responses-only clients |
+| OpenAI legacy completions | `POST /v1/completions` | `"stream": true` (SSE) | in the reply; the last chunk | older tools that send a `prompt` |
+| Anthropic Messages | `POST /v1/messages`, `POST /v1/messages/count_tokens` | `"stream": true` (SSE events) | in the message; `message_delta` | Anthropic SDKs with `base_url`, tools built on them |
+| Ollama | `POST /api/chat`, `POST /api/generate`, `GET /api/tags`, `GET /api/version`, `POST /api/show` | on unless `"stream": false` (NDJSON) | in the reply; the `done` line | Open WebUI's Ollama connection, the `ollama` libraries, Ollama-only apps |
+| Gemini | `POST /v1beta/models/<collection>:generateContent`, `:streamGenerateContent` (`?alt=sse`, else a JSON array), `:countTokens`, `GET /v1beta/models` | the stream route | **headers**: `X-Bench-Panel`, `X-Bench-Seats` (call, seat, alias, status); none when streamed | Google GenAI SDKs with `base_url`, Gemini REST |
+
+The base URL is the server itself (`http://127.0.0.1:8122` on the dashboard's
+port, `:8123` for `gateway.py`): the Anthropic SDK and Google's add their own
+`/v1/...` and `/v1beta/...`, Ollama clients their `/api/...`; OpenAI clients
+take `.../v1`.
+
+```bash
+curl http://127.0.0.1:8122/v1/messages -H "Content-Type: application/json" \
+  -d '{"model": "bench/core", "max_tokens": 1024, "messages": [{"role": "user", "content": "Where can this return a stale value? ..."}]}'
+curl http://127.0.0.1:8122/v1/responses -H "Content-Type: application/json" \
+  -d '{"model": "bench/core", "input": "Where can this return a stale value? ..."}'
+curl http://127.0.0.1:8122/v1/completions -H "Content-Type: application/json" \
+  -d '{"model": "bench/core", "prompt": "Where can this return a stale value? ..."}'
+curl http://127.0.0.1:8122/api/chat -H "Content-Type: application/json" \
+  -d '{"model": "bench/core", "stream": false, "messages": [{"role": "user", "content": "Where can this return a stale value? ..."}]}'
+curl "http://127.0.0.1:8122/v1beta/models/bench/core:streamGenerateContent?alt=sse" -H "Content-Type: application/json" \
+  -d '{"contents": [{"parts": [{"text": "Where can this return a stale value? ..."}]}]}'
+```
+
+Every POST needs `Content-Type: application/json`, as on chat completions
+(it is what keeps a web page's plain form out).
+
+```python
+import anthropic
+bench = anthropic.Anthropic(base_url="http://127.0.0.1:8122", api_key="local")   # or a gateway key
+with bench.messages.stream(model="bench/core", max_tokens=1024,
+                           messages=[{"role": "user", "content": "..."}]) as s:
+    for text in s.text_stream:
+        print(text, end="")              # each seat's section as that seat finishes
+
+from google import genai
+from google.genai import types
+gem = genai.Client(api_key="local", http_options=types.HttpOptions(base_url="http://127.0.0.1:8122"))
+print(gem.models.generate_content(model="bench/core:merged", contents="...").text)
+```
+
+In **Open WebUI**, add `http://127.0.0.1:8122` as an Ollama connection (or
+`.../v1` as an OpenAI one): the collections appear as models.
+
+**Keys.** Where keys are required, a gateway key goes as `Authorization:
+Bearer`, or as `x-api-key` (the Anthropic SDK's `api_key`), `x-goog-api-key`
+(the Google SDKs' `api_key`), or `?key=` on a Gemini route only (Google's REST
+examples). The first one sent is judged. The access log never records a
+query string or a key; neither does http.server's own error line.
+
+**Images** in each format's shape become `image_url` parts, so only seats that
+can see sit (Images): Anthropic `image` blocks (base64 or url), Responses
+`input_image`, Ollama's `images` (base64, its type read from its first
+bytes), Gemini `inlineData` and `fileData` at an http(s) URL.
+
+**Refused, not dropped** (400, with why), because a client waiting for what
+it asked would hang or loop on a panel's text:
+
+- **Tools, in a panel.** A request carrying tools, or a conversation
+  holding tool calls or tool results, in any format, is refused by a panel
+  (`all` or `merged`) with a message pointing at route mode (next section),
+  which takes them. An empty `tools` list is no tools.
+- a tool only its own vendor runs (Anthropic's server tools such as
+  `web_search_20250305` and their result blocks, OpenAI's hosted tools such
+  as `web_search`, Gemini's Google Search, URL context and code execution),
+  in route mode too: no other model can run them;
+- the legacy `functions` / `function_call` fields: send `tools`;
+- a PDF, audio or other non-image part; an image as an uploaded file id
+  (Anthropic `file`, Responses `file_id`, Gemini's Files API);
+- `previous_response_id` or a stored `conversation` on Responses: nothing is
+  stored (`store` is answered `false`), so send the whole conversation;
+- Ollama's `context` (a model's own token state): use `/api/chat`;
+- more than one prompt on legacy completions.
+
+**Ignored by a panel**: sampling options (`max_tokens`, `temperature`,
+`top_p`, `stop` and their native names) are read but each seat keeps its own
+call options; structured-output settings (Ollama `format`, Gemini
+`responseSchema`, Responses `text.format`), since a panel answers in
+markdown. Route mode passes both on. **Ignored everywhere**: a model's own
+reasoning from an earlier turn (Anthropic thinking blocks, Responses
+reasoning items, Gemini thought parts), and native parameters with no
+OpenAI counterpart (Anthropic's `top_k`, `thinking`, `metadata`).
+
+## Route mode (agent clients)
+
+A panel answers with several models' text; an agent needs one model that
+calls its tools. **Route mode** is that: no panel, one member of the
+collection gets the request as sent — messages, tool calls and results,
+`tools`, `tool_choice`, `parallel_tool_calls`, `response_format`,
+temperature and the rest, images — and its answer is relayed in the
+request's own format, streamed or not. Ask for it with `:route` on the
+model (`bench/core:route`), `"bench": {"reply": "route"}`, or set a
+collection's reply to *route* in the Collections dialog. `GET /v1/models`
+(and `/api/tags`, `/v1beta/models`) lists every collection in all three
+modes, so an agent's model picker can choose `bench/core:route`.
+
+**Who answers.** Members only, no explorers: an agent's working session is
+not the place to try an unknown model. Excluded, and named in
+`bench.skipped` with why, as a panel excludes them: retired and ad hoc
+members, members whose endpoint speaks another API than chat completions,
+members the catalog does not know to call tools (when the request has
+tools; the model popup's *Tool calling* is the override, like *Vision /
+OCR*; the catalog's `tools` comes from OpenRouter's listed parameters,
+models.dev, the endpoint's own listing or a capability probe that saw a
+call), that cannot see (when it has images) or whose context window is
+under the request (tools and tool calls counted). No member left: 400
+(413 when size is why). Of those left, the best record goes first: useful
+verdicts counted, then verified suite findings, then a connection already
+paid for; members cooling down after a rate limit go last. One new
+conversation in four goes instead to the member whose record is oldest, so
+the record does not narrow to one model.
+
+**Sticky.** A conversation stays on its model across turns: switching
+mid-task breaks the agent's coherence and the provider's prompt cache. The
+key is a hash of the collection, the gateway key, the system text and the
+first user message, which an agent resends unchanged every turn; it lives
+in memory and lapses after an hour idle (or a restart). A side request with
+its own system prompt (Claude Code's command checks, a subagent, a title) is
+its own conversation. If the conversation's model cannot take a turn (an
+image arrives), the turn is re-ranked; the model that answers becomes the
+conversation's.
+
+**Failover.** If the member fails before the first piece of its answer
+reaches the caller — a 429, a 5xx, a pacing refusal, nothing at all in 90
+seconds, an empty reply (even after reasoning) — the next member in order
+gets the turn, transparently, up to three; the caller sees only the answer,
+and `bench.failed_over` names who failed and why. Reasoning streamed before
+the answer is held back with it. A streamed turn's 200 waits up to 15
+seconds for that first piece, so a turn that fails everywhere quickly keeps
+its status (429 when every member was rate limited, else 502), then
+keep-alives cover the wait. After the first piece, an error goes into the
+stream in the format's error shape: no other model can continue a
+half-written answer.
+
+**Streams.** OpenAI chat completions relays the member's chunks as they
+came, tool-call deltas exactly, under the collection's name, then a closing
+chunk with no choice that carries `bench`. The other formats stream the
+text as it comes and each tool call **whole when the turn ends** (an
+Anthropic `tool_use` block with one `input_json_delta`, a Responses
+`function_call` item, a Gemini `functionCall` part, an Ollama `tool_calls`
+line), so calls whose deltas interleave still arrive intact. Usage is the
+member's own count. An output budget over the model's known maximum is cut
+to it (Claude Code asks every model for 32,000). Headers name the
+conversation (`X-Bench-Conversation`) and, when known before the 200, the
+model (`X-Bench-Model`).
+
+**Translations.** Anthropic `tool_use` / `tool_result` blocks, Responses
+`function_call` / `function_call_output` items, Gemini `functionCall` /
+`functionResponse` parts and Ollama `tool_calls` / `tool` messages become
+OpenAI tool calls and tool messages and back; tool definitions and
+`tool_choice` likewise (Gemini's upper-case schema types lowered). Tool call
+ids pass through both ways; Gemini and Ollama usually send none, so they
+are numbered per request and a result matched to the oldest open call of
+its name. A Responses **custom tool** (Codex's free-form `apply_patch`)
+becomes a function taking one string, `input`, its grammar in the
+description, and the call goes back as a `custom_tool_call`. An image in an
+Anthropic tool result goes in a user message after it (a tool message
+carries text only); `is_error` prefixes the result with `Error: `.
+
+**Recording (the decision).** Every call is in the usage ledger (purpose
+`route`), and every turn a line in `runs/gateway/routes.jsonl`: model,
+status, seconds, tool calls, tokens, who failed over — never the prompt,
+which an agent resends whole every turn. For the router, a conversation is
+**one unjudged offload per model that answered it** (call id
+`<conversation>.1`, `.2` after a failover to another model), not one per
+turn: an agent session is dozens of turns, and a seat per turn would bury
+the panels' verdicts. A verdict on a conversation is optional
+(`POST /bench/verdict` with that call id); the offload alone already counts
+toward rotation. The Gateway card lists route conversations in their own
+table (when, who asked, the first message, who answered with ✓/✗, turns,
+tool calls, failovers) and none of them adds to **Owed**.
+
+**Shadows (later, opt-in).** A shadow seat would answer the same turn in the
+background for evaluation only, after the caller has its answer. The place
+is marked in `route_turn` ("SHADOWS GO HERE"): each would be its own client
+call with purpose `shadow`, recorded on the conversation's call id; nothing
+the caller sees would wait for or change because of them. Not built.
+
+**Limits, honestly.**
+
+- Members reached through native Anthropic, Gemini or Responses endpoints
+  are not routed to: route mode speaks chat completions to the member.
+- The member is a free or flat-rate model, not the model the client was
+  built for. Claude Code and Codex are tuned for their own vendors' models;
+  smaller models make more tool-call mistakes, and some providers reject a
+  tool schema another accepts (that request fails over).
+- Prompt caching is the provider's own: stickiness keeps the prefix on one
+  provider, but `cache_control` marks are not passed on.
+- Thinking is not shown to Anthropic, Responses, Gemini or Ollama clients
+  (a model's reasoning is relayed raw only in chat completions).
+- Tool calls arrive whole, not as deltas, outside chat completions.
+- A caller that hangs up does not stop the member; its answer runs out and
+  is recorded.
+- Stickiness is per gateway process, in memory.
+
+### Pointing agents at a collection
+
+The base URL is the server: `http://127.0.0.1:8122` (the dashboard's port)
+or `:8123` (`gateway.py`). On this machine any key text works; elsewhere,
+or where keys are required, use a gateway key from Settings.
+
+**Claude Code** (Anthropic Messages). It sends the model named by
+`ANTHROPIC_MODEL` (or `--model`) for its main loop and a smaller "haiku"
+model for side tasks (command checks, titles), so set both; a name it does
+not know prints a harmless `unrecognized_model` note:
+
+```bash
+export ANTHROPIC_BASE_URL=http://127.0.0.1:8122
+export ANTHROPIC_API_KEY=<a gateway key, or any text on this machine>
+export ANTHROPIC_MODEL=bench/core:route
+export ANTHROPIC_DEFAULT_HAIKU_MODEL=bench/fast:route
+export ANTHROPIC_DEFAULT_SONNET_MODEL=bench/core:route ANTHROPIC_DEFAULT_OPUS_MODEL=bench/core:route
+export CLAUDE_CODE_SUBAGENT_MODEL=bench/core:route
+claude
+```
+
+Checked end to end (2026-10-23): Claude Code 2.1.299 ran Read, Write and
+Bash tool loops through `bench/core:route` and `bench/fast:route`, failing
+over past a member's 429 on the first turn and staying on one model after.
+
+**Codex and other Responses clients**: a provider with `base_url =
+"http://127.0.0.1:8122/v1"`, `wire_api = "responses"`, the model
+`bench/core:route`, and the gateway key as its API key. Function and custom
+tools work; hosted tools (`web_search`, `local_shell`) are refused, so turn
+them off. Nothing is stored: `previous_response_id` is refused, so the
+client must send the whole conversation every turn.
+Not checked against a real Codex yet.
+
+**opencode, Cline, Continue, Roo, Aider and other OpenAI-compatible agents**:
+an OpenAI-compatible provider at `http://127.0.0.1:8122/v1`, model
+`bench/core:route` (it is in `/v1/models`). Agents that speak Ollama
+(`http://127.0.0.1:8122`, model `bench/core:route`) work the same way. A
+real `curl` with tools streamed a model's tool-call deltas through
+unchanged; these clients themselves were not run.
+
+**Counts and durations.** `usage` and the token-count routes
+(`/v1/messages/count_tokens`, Gemini `:countTokens`) are the size check's
+estimate, characters / 4 plus 10% and 2,500 a picture, with no model called.
+Ollama's durations are real. `/api/version` reports the Ollama API version
+whose shapes these follow (0.12.0), so clients send what it reads.
+
+**On the dashboard's port**, Ollama's five `/api/` routes are the gateway's
+(a gateway key, never the login cookie); the dashboard's own `/api/...`
+routes never use those names.
+
 ## The decision model's job (optional)
 
 A collection's members are fixed; only the explorer seats are chosen per
@@ -201,7 +500,8 @@ key limited to its collection and a rate, or keep the key on the app's server.
 
 | Part | Job |
 |---|---|
-| Gateway | `gateway.py`, stdlib server on 127.0.0.1. `GET /v1/models` lists the collections with description, tags and make-up; `POST /v1/chat/completions` runs one, images included. A streaming client gets the finished answer as one chunk. |
+| Gateway | `gateway.py`, stdlib server on 127.0.0.1. `GET /v1/models` lists the collections with description, tags and make-up; `POST /v1/chat/completions` runs one, images included, streamed seat by seat when asked. |
+| Formats | `metrics/formats.py`: Anthropic Messages, Responses, legacy completions, Ollama and Gemini, each translated to and from OpenAI messages (Other API formats); the outbound half is for the built-in client to call those APIs natively, not wired yet. |
 | Collections | `metrics/bench_collections.py` reads the two config files and computes explorer seats. |
 | Seat router | `route.panel(members=…, explorers=n)` fills explorer seats by staleness for the shape. |
 | Shape picker | Optional decision model (above). |

package/README.md

@@ -48,7 +48,7 @@ verdicts decide who sits on the next panel.
 
 | Related tools | Examples | What they do | What Benchwarmers does differently |
 |---|---|---|---|
-| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer |
+| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record |
 | Councils and ensembles | LLM Council, Mixture-of-Agents | Several models merged into one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
 | Side-by-side chat | Open WebUI, LMArena | Several answers shown together | Records a verdict per answer and uses them to pick the next panel |
 | Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
@@ -76,7 +76,8 @@ verdicts decide who sits on the next panel.
 - [Licence](#licence)
 
 Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is
-in [USAGE.md](USAGE.md). A coding agent setting it up for someone should start
+in [USAGE.md](USAGE.md). Running it on a server or a hosting platform is in
+[HOSTING.md](HOSTING.md). A coding agent setting it up for someone should start
 with [AGENTS.md](AGENTS.md).
 
 ## Try it
@@ -189,6 +190,27 @@ seat is still in `bench`. Record a verdict per call once you have checked it:
 or from the dashboard. See [GATEWAY.md](GATEWAY.md) for the full contract:
 images, prompt sizes, who may call, and the reply's shape.
 
+**Other clients.** The same panel answers in other APIs' shapes, so a client
+that speaks one of them works with the server's address as its base URL:
+
+| Format | Route | For |
+|---|---|---|
+| OpenAI chat completions | `POST /v1/chat/completions` | OpenAI SDKs, most chat apps |
+| OpenAI Responses | `POST /v1/responses` | `responses.create` clients (nothing is stored) |
+| OpenAI legacy completions | `POST /v1/completions` | tools that send a `prompt` |
+| Anthropic Messages | `POST /v1/messages` | Anthropic SDKs with `base_url` |
+| Ollama | `POST /api/chat`, `/api/generate`, `GET /api/tags` | Open WebUI's Ollama connection, Ollama apps |
+| Gemini | `POST /v1beta/models/bench/<collection>:generateContent` | Google GenAI SDKs with `base_url` |
+
+    curl http://127.0.0.1:8122/v1/messages -H "Content-Type: application/json" \
+      -d '{"model": "bench/<collection>", "max_tokens": 1024, "messages": [{"role": "user", "content": "..."}]}'
+
+Each streams when asked: the reply starts at once, and each seat's answer
+arrives as that seat finishes. Agent and tool use is not supported yet: a
+request carrying tools is refused with a 400 rather than answered without
+them. GATEWAY.md, "Other API formats", has a curl for each and what is
+refused or ignored.
+
 ## From the command line
 
 `bench` (or `benchwarmers`, the same program) does what the dashboard does,
@@ -272,6 +294,19 @@ On native Windows, a client that cannot start `npx` itself needs
 | `BENCHWARMERS_KEY` | a gateway key from Settings, sent as `Authorization: Bearer`; needed for a server on another machine, or where keys are required |
 | `BENCHWARMERS_TIMEOUT` | seconds to wait for a panel, default 900 |
 
+**An agent running on a collection.** The MCP tools ask a panel for a
+second opinion; an agent can also *be driven by* a collection, in **route
+mode** (`bench/<collection>:route`): one member, picked by its record, gets
+each turn with its tools, stays on for the whole conversation, and is
+replaced by the next member if it fails before answering. Claude Code
+points at it with `ANTHROPIC_BASE_URL=http://127.0.0.1:8122` and
+`ANTHROPIC_MODEL=bench/core:route`; Codex, opencode, Cline and other
+OpenAI-, Responses- or Ollama-speaking agents with the same base URL.
+[GATEWAY.md](GATEWAY.md), "Route mode", has the set-up and the limits.
+
+<!-- The MCP Registry's proof that the PyPI package is this server (server.json):
+mcp-name: io.github.WendellResearch-AI/benchwarmers -->
+
 ## The suite
 
 Models are scored on review work: find the faulty line, or say whether a
@@ -302,6 +337,18 @@ built-in client makes every call). Put the secrets in a `.env` file beside
 then `docker compose up -d` and open http://127.0.0.1:8122. The compose file
 publishes on this machine only; see its comments for a proxy.
 
+**Hosted, always on:** [HOSTING.md](HOSTING.md) compares Fly.io, Railway,
+Render and a VPS with Caddy (cost, sleep, disks, HTTPS), with a template for
+each under [deploy/](deploy/).
+
+[![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/WendellResearch-AI/benchwarmers)
+
+Render is one click: a paid instance with a 1 GB disk and a generated admin
+token. Fly.io is four commands from a checkout (`cp deploy/fly.toml .`,
+`fly launch --copy-config --no-deploy`, `fly secrets set
+BENCHWARMERS_ADMIN_TOKEN=...`, `fly deploy`). Railway is a few steps in its CLI
+or browser, in [HOSTING.md](HOSTING.md#railway).
+
 ## What leaves your machine
 
 Benchwarmers has no account, no telemetry and no server of its own; nothing
@@ -325,7 +372,10 @@ detail.
 
 ## Limits
 
-- **Early.** v0.1: no installer yet, no streaming replies, and the LiteLLM and
+- **No tools yet.** A panel answers with text, not a tool call, so agents
+  that need tool use (Claude Code, Codex, opencode as callers) wait for a
+  route mode; requests with tools are refused, not trimmed.
+- **Early.** v0.1: no installer yet, merged replies stream in one piece, and the LiteLLM and
   CLI Proxy API hub adapters are built from those projects' source, not yet
   run against a real hub.
 - **Calls cost.** Probes, onboarding and the suite run on your keys and free

package/USAGE.md

@@ -104,6 +104,37 @@ Images go in as OpenAI `image_url` parts; only models that take images sit,
 and the rest are listed in `bench.skipped`. [GATEWAY.md](GATEWAY.md) has the
 limits and the reply's full shape.
 
+**A client that speaks another API** works too, with the server's address as
+its base URL and a collection as its model:
+
+| Client | Base URL | Notes |
+| --- | --- | --- |
+| Open WebUI, as an **Ollama** connection | `http://127.0.0.1:8122` | collections listed from `/api/tags`; `bench/<collection>:merged` reads best in a chat window |
+| **Anthropic** SDK (`anthropic.Anthropic(base_url=...)`) | `http://127.0.0.1:8122` | `/v1/messages`; a gateway key goes in `api_key` |
+| **Google GenAI** SDK (`http_options=HttpOptions(base_url=...)`) | `http://127.0.0.1:8122` | `generate_content(model="bench/<collection>")`; a key in `api_key` |
+| OpenAI **Responses** clients | `http://127.0.0.1:8122/v1` | `responses.create`; nothing is stored, so send the whole conversation |
+| Older tools on `/v1/completions` | `http://127.0.0.1:8122/v1` | one `prompt` |
+
+Asked to stream, each answers at once and sends each seat's answer as it
+finishes. GATEWAY.md, "Other API formats", has a curl for each.
+
+**An agent that calls tools** (Claude Code, Codex, opencode, Cline...) uses
+a collection's **route mode**: model `bench/<collection>:route`. No panel:
+one member, picked by its record, gets the request with its tools and
+streams its answer back in the agent's own format; the conversation stays
+on that model, and a member that fails before answering is replaced by the
+next. A panel refuses tools with a 400 saying so. For Claude Code:
+
+```bash
+export ANTHROPIC_BASE_URL=http://127.0.0.1:8122 ANTHROPIC_API_KEY=local
+export ANTHROPIC_MODEL=bench/core:route ANTHROPIC_DEFAULT_HAIKU_MODEL=bench/fast:route
+claude
+```
+
+Each conversation is one offload per model that answered it; a verdict is
+optional, and the Gateway card lists route conversations apart from the
+panels. GATEWAY.md, "Route mode", has the other clients and the limits.
+
 **A coding agent** connects over MCP instead: the Claude Code plugin
 (`/plugin marketplace add WendellResearch-AI/benchwarmers`, then
 `/plugin install benchwarmers@benchwarmers`), or `benchwarmers mcp` in any MCP

package/VERSION

@@ -1 +1 @@
-0.2.1
+0.3.0

package/cli.py

@@ -258,7 +258,10 @@ def cmd_onboard(args) -> None:
         reply = call("GET", "/api/onboard-queue") if verb == "status" else \
             call("POST", f"/api/onboard-queue/{verb}", {})
     else:
-        items = [{"served": s, "endpoint": args.endpoint or ""} for s in args.served]
+        if args.ad_hoc and not args.quick:
+            sys.exit("--ad-hoc goes with --quick: an ad hoc model gets no suite run of its own")
+        mode = "quick-adhoc" if args.ad_hoc else "quick" if args.quick else "full"
+        items = [{"served": s, "endpoint": args.endpoint or "", "mode": mode} for s in args.served]
         reply = call("POST", "/api/onboard-queue", {"items": items})
         if args.wait:
             while reply.get("running"):
@@ -482,6 +485,7 @@ keeps only the answers. Then judge each seat you checked:
   bench candidates --new
   bench onboard minimax/minimax-m3.1-flash:free --wait
   bench onboard a/one:free b/two:free --endpoint openrouter
+  bench onboard command-a-plus-05-2026 --endpoint cohere --quick --ad-hoc   # a trial key
   bench onboard status | stop | clear""",
     "collection": """examples:
   bench collection save reviewers --members a,b,c --description "adversarial review" --tags review
@@ -590,6 +594,10 @@ def parser() -> argparse.ArgumentParser:
     s.add_argument("served", nargs="+", help="served names, as `bench candidates` lists them")
     s.add_argument("--endpoint", help="the endpoint they are on; the default if omitted")
     s.add_argument("--wait", action="store_true", help="stay until the queue finishes")
+    s.add_argument("--quick", action="store_true",
+                   help="probe only; the scheduled suite passes fill the suite in later")
+    s.add_argument("--ad-hoc", action="store_true",
+                   help="with --quick: the bench never calls it on its own (metered, capped or trial keys)")
 
     s = cmd("dismiss", cmd_dismiss, "stop offering served models as candidates")
     s.add_argument("served", nargs="+"), s.add_argument("--endpoint"), s.add_argument("--undo", action="store_true")
@@ -607,7 +615,7 @@ def parser() -> argparse.ArgumentParser:
     c.add_argument("--shape", help="the task shape explorers are picked for; empty for automatic")
     c.add_argument("--description", help="what it is for: callers read it to choose")
     c.add_argument("--tags", help="comma-separated")
-    c.add_argument("--reply", choices=["all", "merged"])
+    c.add_argument("--reply", choices=["all", "merged", "route"])
     c.add_argument("--merger", help="the alias that merges, for --reply merged")
 
     cmd("endpoints", cmd_endpoints, "where models are served from")