@wendellresearch-ai/benchwarmers 0.1.1 → 0.2.0

package/GATEWAY.md

@@ -211,6 +211,7 @@ key limited to its collection and a rate, or keep the key on the app's server.
 | Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily (scheduled task WRBench-Catalog); a free model whose price goes above zero raises a "went paid" alert. |
 | Discover tab | `metrics/discover.py`: models.dev and FreeLLMAPI merged into one list of models worth adding, free first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
 | Merged replies | Opt-in: one merger answer as `content`, every seat still in `bench` (The reply). |
+| MCP server | `mcp_server.py` (`benchwarmers mcp`): the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`, over HTTP to a running server (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`), so it is a caller like any other. The Claude Code plugin in `plugin/` wraps it with a command and a skill. |
 
 ## Rules
 

package/README.md

@@ -64,6 +64,7 @@ verdicts decide who sits on the next panel.
 - [Try it](#try-it)
 - [How it works](#how-it-works)
 - [Asking a panel](#asking-a-panel)
+- [From Claude Code and other agents](#from-claude-code-and-other-agents)
 - [The suite](#the-suite)
 - [Docker](#docker)
 - [What leaves your machine](#what-leaves-your-machine)
@@ -179,6 +180,44 @@ seat is still in `bench`. Record a verdict per call once you have checked it:
 or from the dashboard. See [GATEWAY.md](GATEWAY.md) for the full contract:
 images, prompt sizes, who may call, and the reply's shape.
 
+## From Claude Code and other agents
+
+`benchwarmers mcp` is an MCP server on stdio with three tools:
+`list_collections`, `ask` (a collection, a prompt, optionally `merged`,
+`with` and `system`) and `verdict`. It is a client of a Benchwarmers that is
+already running, over the routes above, so start the dashboard (or the
+service) first. The tool descriptions carry the rules: ask a collection, not
+a favourite model; verify before acting; name every seat; send a verdict per
+seat you judged.
+
+**Claude Code plugin.** The plugin adds the MCP server, a `/benchwarmers:ask
+<collection> <what to ask>` command and a `benchwarmers` skill saying when a
+panel helps (adversarial review, exhaustive enumeration over real files) and
+how to judge it:
+
+    /plugin marketplace add WendellResearch-AI/benchwarmers
+    /plugin install benchwarmers@benchwarmers
+
+It needs Node 18 or later and Python 3.11 or later on PATH.
+
+**Any other MCP client** (or Claude Code without the plugin): run
+`npx -y @wendellresearch-ai/benchwarmers mcp`, or `python mcp_server.py` from a
+checkout. For example:
+
+```json
+{"mcpServers": {"benchwarmers": {"command": "npx", "args": ["-y", "@wendellresearch-ai/benchwarmers", "mcp"],
+                                 "env": {"BENCHWARMERS_URL": "http://127.0.0.1:8122"}}}}
+```
+
+On native Windows, a client that cannot start `npx` itself needs
+`"command": "cmd", "args": ["/c", "npx", "-y", "@wendellresearch-ai/benchwarmers", "mcp"]`.
+
+| Variable | Meaning |
+| --- | --- |
+| `BENCHWARMERS_URL` | the server, default `http://127.0.0.1:8122` (a `/v1` on the end is fine) |
+| `BENCHWARMERS_KEY` | a gateway key from Settings, sent as `Authorization: Bearer`; needed for a server on another machine, or where keys are required |
+| `BENCHWARMERS_TIMEOUT` | seconds to wait for a panel, default 900 |
+
 ## The suite
 
 Models are scored on review work: find the faulty line, or say whether a

package/USAGE.md

@@ -104,6 +104,14 @@ Images go in as OpenAI `image_url` parts; only models that take images sit,
 and the rest are listed in `bench.skipped`. [GATEWAY.md](GATEWAY.md) has the
 limits and the reply's full shape.
 
+**A coding agent** connects over MCP instead: the Claude Code plugin
+(`/plugin marketplace add WendellResearch-AI/benchwarmers`, then
+`/plugin install benchwarmers@benchwarmers`), or `benchwarmers mcp` in any MCP
+client, with `BENCHWARMERS_URL` and, where keys are required, a gateway key in
+`BENCHWARMERS_KEY`. Its `ask` tool returns every seat with its call id and its
+`verdict` tool records the verdicts; README, "From Claude Code and other
+agents", has the set-up.
+
 ## Recording verdicts
 
 A panel's answers are worth something only once someone has checked them

package/VERSION

@@ -1 +1 @@
-0.1.1
+0.2.0

package/bin/benchwarmers.js

@@ -28,6 +28,7 @@ function usage() {
     '  serve        the dashboard, with the gateway at /v1: http://localhost:8122 (the default)',
     '  gateway      the gateway alone: http://127.0.0.1:8123/v1',
     '  init [...]   set up a data folder, optionally with a first endpoint (init --help)',
+    '  mcp          an MCP server on stdio for agents, a client of a running Benchwarmers (BENCHWARMERS_URL)',
     '  --version    the version',
     '',
     'Your data lives in BENCHWARMERS_HOME, else ' + defaultHome() + '.',
@@ -59,7 +60,7 @@ function main(argv) {
     console.log(usage());
     return 0;
   }
-  const scripts = { serve: 'serve.py', gateway: 'gateway.py', init: path.join('scripts', 'init.py') };
+  const scripts = { serve: 'serve.py', gateway: 'gateway.py', mcp: 'mcp_server.py', init: path.join('scripts', 'init.py') };
   if (!scripts[command]) {
     console.error('benchwarmers: no command "' + command + '"\n\n' + usage());
     return 2;
@@ -79,8 +80,9 @@ function main(argv) {
 
   // The first run makes the data folder from examples/ (init.py never writes
   // over a file already there, and with BENCHWARMERS_HOME set it leaves the
-  // package folder alone).
-  if (command !== 'init' && !fs.existsSync(home)) {
+  // package folder alone). Not for mcp: it keeps no data, and its stdout
+  // carries protocol messages only.
+  if (command !== 'init' && command !== 'mcp' && !fs.existsSync(home)) {
     console.log('First run: making your data folder at ' + home);
     const r = spawnSync(python[0], py(scripts.init, [home]), { stdio: 'inherit', env, cwd: ROOT });
     if (r.status !== 0) return r.status || 1;

package/gateway.py

@@ -34,6 +34,11 @@ takes choice questions is onboarded (not ad hoc); otherwise, or on any doubt,
 the shape is `general`. A request body may add `"with": ["<alias>"]` to seat a
 model in an explorer chair, as `route.py --with` does.
 
+A retired model (`config/retired.json`) is never called: retiring takes it out
+of every collection, and one a collection still names — as a member, `with`
+or its merger — is skipped and named in `bench.skipped`, its seat going to an
+explorer and the merge to a merger chosen by verdicts.
+
 THE LOOP
 --------
 Every seat is recorded in `metrics/offloads.jsonl` as an unjudged offload the
@@ -263,6 +268,24 @@ def pick_shape(prompt: str) -> tuple[str, str]:
     return answer["choice"], f"{picker}, {conf:.2f}"
 
 
+def retired() -> dict[str, dict]:
+    """config/retired.json: alias -> its record (when, why). Retiring a model
+    takes it out of every collection (serve.remove_model); this is for a
+    collection that still names one -- a hand edit, or one saved before that
+    -- so the seat is skipped with a reason instead of failing on the call."""
+    try:
+        data = json.loads((DATA / "config" / "retired.json").read_text(encoding="utf-8"))
+    except (OSError, ValueError):
+        return {}
+    return data if isinstance(data, dict) else {}
+
+
+def retired_why(entry: dict) -> str:
+    when = str(entry.get("at", ""))[:10] if isinstance(entry, dict) else ""
+    return (f"retired from the bench{' on ' + when if when else ''}; take it out of the collection"
+            " (Restore in its model popup brings it back)")
+
+
 def retiring() -> set[str]:
     """Aliases not to spend an explorer seat on: those on a retiring alert
     (failing their probes; 2026-10-16 both explorers of a bench/core call were
@@ -324,8 +347,10 @@ def _ask_llm(cmd: list[str], prompt: str) -> tuple[str, str]:
                               encoding="utf-8", errors="replace", timeout=SEAT_TIMEOUT,
                               env={**os.environ, "PYTHONIOENCODING": "utf-8"})
         answer = done.stdout.strip()
-        # A non-zero exit is an error even when stderr is silent.
-        error = (done.stderr.strip() or f"llm exited {done.returncode}") if done.returncode else ""
+        # A non-zero exit is an error even when stderr is silent, unless it is
+        # only llm failing to log an answer it printed (client.py).
+        failed = done.returncode and not client.llm_log_failed_only(done.stdout, done.stderr)
+        error = (done.stderr.strip() or f"llm exited {done.returncode}") if failed else ""
     except subprocess.TimeoutExpired:
         answer, error = "", f"no answer in {SEAT_TIMEOUT}s"
     except Exception as exc:                           # noqa: BLE001
@@ -428,14 +453,30 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
     s = bench_collections.settings()
     raw = collection.get("members") or []
     # The editor dedupes and caps; a hand-edited file may not have.
-    members = list(dict.fromkeys(m for m in raw if isinstance(m, str) and m))[:s["max_calls"]]
-    n_explorers = bench_collections.explorers_for(len(members), s)
+    members = list(dict.fromkeys(m for m in raw if isinstance(m, str) and m))
+    # A RETIRED MODEL IS NOT CALLED: it is no longer registered, so the call
+    # could only fail ("Unknown model"). Skipped, and named, before the cap so
+    # it frees its seat for the next member or an explorer. Retiring takes it
+    # out of the collection, so this is a collection that still names one.
+    gone = retired()
+    skipped = [{"seat": "member", "alias": a, "why": retired_why(gone[a])} for a in members if a in gone]
+    members = [a for a in members if a not in gone][:s["max_calls"]]
     # A member named in `with` is already seated; it must not use up the slot.
     nominated = [a for a in dict.fromkeys(nominated or []) if a not in members]
+    skipped += [{"seat": "with", "alias": a, "why": retired_why(gone[a])} for a in nominated
+                if a in gone and a not in {k["alias"] for k in skipped}]
+    nominated = [a for a in nominated if a not in gone]
+    merged = (reply or collection.get("reply") or "all") == "merged"
+    if collection.get("merger") in gone:
+        if merged:
+            skipped.append({"seat": "merger", "alias": collection["merger"],
+                            "why": retired_why(gone[collection["merger"]]) + "; the merger is chosen by verdicts"})
+        collection = {**collection, "merger": ""}
+    n_explorers = bench_collections.explorers_for(len(members), s)
     # AN IMAGE GOES ONLY TO A MODEL THAT CAN SEE IT: one that cannot would
     # answer anyway, about a picture it never got. Skipped, and named.
-    skipped = [{"seat": seat, "alias": a, "why": why} for seat, group in (("member", members), ("with", nominated))
-               for a in group if pictures and (why := cannot_see(a))]
+    skipped += [{"seat": seat, "alias": a, "why": why} for seat, group in (("member", members), ("with", nominated))
+                for a in group if pictures and (why := cannot_see(a))]
     unseen = {k["alias"] for k in skipped}
     sitting = [a for a in members if a not in unseen]
     nominated = [a for a in nominated if a not in unseen]
@@ -477,7 +518,8 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
                          + ", and no model that takes images was free for an explorer seat. Add one to the"
                          " collection, or call one whose together.capabilities on GET /v1/models lists vision.")
     if not seats:
-        raise LookupError(f"collection {name!r} has no members and no explorer could be seated")
+        raise LookupError(f"collection {name!r} has no members and no explorer could be seated"
+                          + "".join(f"; {k['alias']} not asked: {k['why']}" for k in skipped))
     panel_id = f"p{dt.datetime.now():%Y%m%d-%H%M%S}-{secrets.token_hex(2)}"
     model = bench_collections.PREFIX + name
     with ThreadPoolExecutor(max_workers=len(seats)) as pool:
@@ -491,8 +533,7 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
     out = [f"Bench {model} {panel_id} · {len(sitting)} members + {len(seats) - len(sitting)}"
            f" explorers · explorer shape {shape} ({how})\n"]
     if skipped:
-        out.append("Not asked, as they cannot be sent the images: "
-                   + "; ".join(f"{k['alias']} ({k['why']})" for k in skipped) + "\n")
+        out.append("Not asked: " + "; ".join(f"{k['alias']} ({k['why']})" for k in skipped) + "\n")
     for (key, label, alias), r in zip(seats, results):
         call = f"{panel_id}.{key}"
         record["seats"].append(record_seat(task, call, label, alias, r))
@@ -500,16 +541,18 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
                                "_empty reply — usually a token budget, not incapacity_")
         out.append(f"## {label} — {alias}\n`{call}` · {r['status']} · {r['seconds']}s\n\n{body}\n")
     out.append("---\nRecord a verdict per seat once checked against the code:\n"
-               "`python metrics/offload.py verdict <call id> --useful|--not-useful --note \"...\"`")
+               "`POST /bench/verdict` on this same server (see `bench.verdict`), or"
+               " `python metrics/offload.py verdict <call id> --useful|--not-useful --note \"...\"`")
     content = "\n".join(out)
     # Beyond the OpenAI fields, so a caller reads seats without parsing the
     # markdown. Clients that do not know it ignore it.
     bench = {"panel": panel_id, "collection": model, "shape": shape, "shape_by": how,
              "seats": [{k: s[k] for k in SEAT_KEYS} for s in record["seats"]],
-             "verdict": f"POST http://127.0.0.1:{PORT}/bench/verdict"
-                        ' {"call", "useful": true|false, "note", "real": [...], "false": [...]}',
+             # A path, not a URL: the same server answered, whichever port or
+             # proxy the caller came through (the dashboard serves these routes too).
+             "verdict": 'POST /bench/verdict {"call", "useful": true|false, "note", "real": [...], "false": [...]}',
              **({"skipped": skipped} if skipped else {})}
-    if (reply or collection.get("reply") or "all") == "merged":
+    if merged:
         try:
             merger, note = merge(collection, members, shape, whole, record["seats"], task, panel_id)
         except Exception as exc:                       # noqa: BLE001 -- never lose the panel to the merge

package/mcp_server.py

@@ -0,0 +1,266 @@
+"""Benchwarmers as an MCP server, so an agent (Claude Code, or any MCP client)
+can ask a collection and send its verdicts.
+
+    benchwarmers mcp                 # from npm: npx -y @wendellresearch-ai/benchwarmers mcp
+    python mcp_server.py             # from a checkout
+
+MCP over stdio: JSON-RPC 2.0, one message per line, nothing else on stdout.
+A THIN CLIENT of a Benchwarmers server that is already running, over the same
+HTTP routes any other caller uses (GATEWAY.md), so the panel, the call log and
+the verdicts are that server's, whichever machine it is on:
+
+    BENCHWARMERS_URL       the server, default http://127.0.0.1:8122 (a /v1 on the end is fine)
+    BENCHWARMERS_KEY       a gateway key, sent as `Authorization: Bearer`; needed beyond
+                           this machine, or where BENCHWARMERS_REQUIRE_KEYS is set (metrics/guard.py)
+    BENCHWARMERS_TIMEOUT   seconds to wait for a panel, default 900
+
+Tools: list_collections (GET /v1/models), ask (POST /v1/chat/completions),
+verdict (POST /bench/verdict). A refusal from the server, or no server at
+all, comes back as a tool error carrying its message. Standard library only,
+and nothing from metrics/: the data folder is the server's, not this one's.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import threading
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+# Newest first: the client's version when it is one of these, else the newest.
+PROTOCOLS = ("2025-11-25", "2025-06-18", "2025-03-26", "2024-11-05")
+PREFIX = "bench/"
+VERSION = (Path(__file__).resolve().parent / "VERSION").read_text(encoding="utf-8").strip()
+
+RULES = (
+    "Benchwarmers asks a COLLECTION of models at once (every member plus explorer seats it rotates in)"
+    " and keeps every answer. Rules: (1) ask a collection, not a favourite model; list_collections says"
+    " what each is for. (2) Verify before acting on anything a seat says: seats fail together, so"
+    " agreement is not evidence. (3) Send a verdict per seat you judged, with the claims it got right"
+    " in `real` and the untrue ones in `false`. (4) Name every seat that answered, by model and seat,"
+    " including errors and empty replies. An empty reply from a reasoning model is usually its token"
+    " budget, not incapacity. Good uses: exhaustive enumeration over real files, adversarial review."
+    " Keep the design and the decisions yourself.")
+
+TOOLS = [
+    {"name": "list_collections",
+     "title": "List collections",
+     "description": "The collections this Benchwarmers server offers: name, what each is for, tags, members,"
+                    " explorer seats, reply mode, and what the members handle together (smallest context"
+                    " window, shared capabilities, tightest rate limits). Pick by task before calling ask.",
+     "inputSchema": {"type": "object", "properties": {}, "additionalProperties": False},
+     "annotations": {"readOnlyHint": True, "openWorldHint": False}},
+    {"name": "ask",
+     "title": "Ask a collection",
+     "description": "Put a prompt to every seat of one collection at once and get every answer back, each"
+                    " under its seat and model, plus the seats as data (alias, seat, status, call id). Can"
+                    " take minutes. Ask a collection, not a favourite model. Give the seats the real text"
+                    " (code, diff, file contents) in the prompt: they cannot read your files. Verify every"
+                    " claim against the source before acting on it, name every seat that answered in what"
+                    " you tell the user, then send a verdict per seat you judged.",
+     "inputSchema": {"type": "object", "required": ["collection", "prompt"], "additionalProperties": False,
+                     "properties": {
+                         "collection": {"type": "string",
+                                        "description": "A collection name, `core` or `bench/core` (list_collections)."},
+                         "prompt": {"type": "string", "description": "The question, with everything it needs."},
+                         "system": {"type": "string", "description": "A system message for every seat."},
+                         "merged": {"type": "boolean",
+                                    "description": "true: one reconciled answer, every seat still called and"
+                                                   " listed; false: every answer. Omit for the collection's own."},
+                         "with": {"type": "array", "items": {"type": "string"},
+                                  "description": "Model aliases to seat in explorer chairs, this request only."}}},
+     "annotations": {"readOnlyHint": False, "destructiveHint": False, "openWorldHint": True}},
+    {"name": "verdict",
+     "title": "Record a verdict",
+     "description": "Judge one seat's answer by its call id, once you have checked it against the source. One"
+                    " per seat you judged. List the claims it got right in `real` and the claims that are"
+                    " untrue in `false`, one sentence each: they feed routing and the 'confidently wrong'"
+                    " record, so a note alone is not enough.",
+     "inputSchema": {"type": "object", "required": ["call", "useful"], "additionalProperties": False,
+                     "properties": {
+                         "call": {"type": "string", "description": "The seat's call id, e.g. p20261022-101500-ab12.member1."},
+                         "useful": {"type": "boolean"},
+                         "note": {"type": "string", "description": "One or two sentences on why."},
+                         "real": {"type": "array", "items": {"type": "string"},
+                                  "description": "Claims it made that are true, one sentence each."},
+                         "false": {"type": "array", "items": {"type": "string"},
+                                   "description": "Claims it made that are untrue, one sentence each."}}},
+     "annotations": {"readOnlyHint": False, "destructiveHint": False, "idempotentHint": True,
+                     "openWorldHint": False}},
+]
+
+
+class ToolError(Exception):
+    """A tool call that failed: its message goes back as the tool's result, isError."""
+
+
+def base_url() -> str:
+    url = (os.environ.get("BENCHWARMERS_URL") or "").strip().rstrip("/") or "http://127.0.0.1:8122"
+    return url.removesuffix("/v1")
+
+
+def http(method: str, path: str, body: dict | None = None) -> dict:
+    url = base_url() + path
+    headers = {"Accept": "application/json"}
+    if key := (os.environ.get("BENCHWARMERS_KEY") or "").strip():
+        headers["Authorization"] = f"Bearer {key}"
+    data = None
+    if body is not None:
+        data = json.dumps(body).encode()
+        headers["Content-Type"] = "application/json"
+    req = urllib.request.Request(url, data=data, headers=headers, method=method)
+    timeout = float(os.environ.get("BENCHWARMERS_TIMEOUT") or 900)
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as r:
+            return json.load(r)
+    except urllib.error.HTTPError as e:
+        raw = e.read().decode("utf-8", "replace")
+        try:
+            message = json.loads(raw)["error"]["message"]
+        except (ValueError, KeyError, TypeError):
+            message = raw.strip()[:500] or e.reason
+        raise ToolError(f"Benchwarmers refused {method} {path} ({e.code}): {message}") from None
+    except (urllib.error.URLError, ConnectionError) as e:
+        reason = getattr(e, "reason", e)
+        if isinstance(reason, TimeoutError):
+            raise ToolError(f"no answer from Benchwarmers at {base_url()} in {timeout:.0f} s;"
+                            " set BENCHWARMERS_TIMEOUT for longer panels") from None
+        raise ToolError(f"Benchwarmers is not running at {base_url()} ({reason}). Start it (`benchwarmers`,"
+                        " or its service), or point BENCHWARMERS_URL at the server you use.") from None
+    except TimeoutError:
+        raise ToolError(f"no answer from Benchwarmers at {base_url()} in {timeout:.0f} s;"
+                        " set BENCHWARMERS_TIMEOUT for longer panels") from None
+    except ValueError:
+        raise ToolError(f"{base_url()} answered {method} {path} with something that is not JSON;"
+                        " is BENCHWARMERS_URL a Benchwarmers server?") from None
+
+
+def list_collections(_args: dict) -> list[str]:
+    rows = []
+    for m in http("GET", "/v1/models").get("data") or []:
+        if ":" in m.get("id", ""):                     # the same collection in its other reply mode
+            continue
+        together = m.get("together") or {}
+        rows.append({"name": m["id"], **{k: m.get(k) for k in ("description", "tags", "members", "explorers",
+                                                              "shape", "reply", "merger")},
+                     **{k: together.get(k) for k in ("context_window", "capabilities", "limits")}})
+    if not rows:
+        return ["No collections on this server (or none this gateway key may call). Make one on the"
+                " dashboard's Collections page."]
+    return [json.dumps(rows, indent=1, ensure_ascii=False)]
+
+
+def ask(args: dict) -> list[str]:
+    collection, prompt = args.get("collection"), args.get("prompt")
+    if not (isinstance(collection, str) and collection.strip()):
+        raise ToolError("`collection` is required: a name from list_collections, e.g. core")
+    if not (isinstance(prompt, str) and prompt.strip()):
+        raise ToolError("`prompt` is required")
+    collection = collection.strip()
+    messages = [{"role": "user", "content": prompt}]
+    if isinstance(args.get("system"), str) and args["system"].strip():
+        messages.insert(0, {"role": "system", "content": args["system"]})
+    body = {"model": collection if collection.startswith(PREFIX) else PREFIX + collection, "messages": messages}
+    if isinstance(args.get("merged"), bool):
+        body["bench"] = {"reply": "merged" if args["merged"] else "all"}
+    if args.get("with"):
+        body["with"] = args["with"]
+    reply = http("POST", "/v1/chat/completions", body)
+    content = reply["choices"][0]["message"]["content"]
+    bench = reply.get("bench") or {}
+    merged = bench.get("reply") == "merged"
+    # Each answer is already in the content unless it is the merge, so the
+    # seat data stays short: who sat, how it went, the id to judge it by.
+    keep = ("call", "seat", "alias", "status", "seconds", "error") + (("answer",) if merged else ())
+    seats = [{k: s[k] for k in keep if s.get(k) not in (None, "")} for s in bench.get("seats") or []]
+    for s in seats:
+        if "error" in s:
+            s["error"] = s["error"][-300:]             # a traceback's last lines say what went wrong
+    data = {"panel": bench.get("panel"), "collection": bench.get("collection"), "shape": bench.get("shape"),
+            "seats": seats, **{k: bench[k] for k in ("merger", "skipped", "note") if bench.get(k)}}
+    return [content, "Seats (name each one; send a verdict per seat you judged, by call id):\n"
+            + json.dumps(data, indent=1, ensure_ascii=False)]
+
+
+def verdict(args: dict) -> list[str]:
+    if not (isinstance(args.get("call"), str) and args["call"].strip()):
+        raise ToolError("`call` is required: the seat's call id from ask")
+    if not isinstance(args.get("useful"), bool):
+        raise ToolError("`useful` must be true or false")
+    body = {"call": args["call"].strip(), "useful": args["useful"], "note": args.get("note") or "",
+            "real": args.get("real") or [], "false": args.get("false") or []}
+    http("POST", "/bench/verdict", body)
+    return [f"Recorded {body['call']}: {'useful' if body['useful'] else 'not useful'},"
+            f" {len(body['real'])} real, {len(body['false'])} false."]
+
+
+HANDLERS = {"list_collections": list_collections, "ask": ask, "verdict": verdict}
+OUT = threading.Lock()
+
+
+def send(message: dict) -> None:
+    line = json.dumps(message, ensure_ascii=False) + "\n"
+    with OUT:
+        sys.stdout.buffer.write(line.encode("utf-8"))
+        sys.stdout.buffer.flush()
+
+
+def call_tool(rid, params: dict) -> None:
+    name, args = params.get("name"), params.get("arguments") or {}
+    if name not in HANDLERS:
+        return send({"jsonrpc": "2.0", "id": rid, "error": {"code": -32602, "message": f"unknown tool {name!r}"}})
+    try:
+        if not isinstance(args, dict):
+            raise ToolError("arguments must be an object")
+        texts, failed = HANDLERS[name](args), False
+    except ToolError as e:
+        texts, failed = [str(e)], True
+    except Exception as e:                             # noqa: BLE001 -- a bad reply must not kill the server
+        texts, failed = [f"{type(e).__name__}: {e}"], True
+    send({"jsonrpc": "2.0", "id": rid,
+          "result": {"content": [{"type": "text", "text": t} for t in texts], "isError": failed}})
+
+
+def handle(msg) -> None:
+    if not isinstance(msg, dict) or msg.get("jsonrpc") != "2.0" or not isinstance(msg.get("method"), str):
+        rid = msg.get("id") if isinstance(msg, dict) else None
+        return send({"jsonrpc": "2.0", "id": rid, "error": {"code": -32600, "message": "invalid request"}})
+    if "id" not in msg:                                # a notification (initialized, cancelled): nothing to answer
+        return
+    rid, method, params = msg["id"], msg["method"], msg.get("params") or {}
+    if method == "initialize":
+        asked = params.get("protocolVersion")
+        send({"jsonrpc": "2.0", "id": rid, "result": {
+            "protocolVersion": asked if asked in PROTOCOLS else PROTOCOLS[0],
+            "capabilities": {"tools": {"listChanged": False}},
+            "serverInfo": {"name": "benchwarmers", "title": "Benchwarmers", "version": VERSION},
+            "instructions": RULES}})
+    elif method == "ping":
+        send({"jsonrpc": "2.0", "id": rid, "result": {}})
+    elif method == "tools/list":
+        send({"jsonrpc": "2.0", "id": rid, "result": {"tools": TOOLS}})
+    elif method == "tools/call":
+        # A panel takes minutes: on its own thread, so a ping or a second call is answered meanwhile.
+        # Not a daemon: when stdin closes, a call already sent (a verdict) still finishes.
+        threading.Thread(target=call_tool, args=(rid, params)).start()
+    else:
+        send({"jsonrpc": "2.0", "id": rid, "error": {"code": -32601, "message": f"no method {method!r}"}})
+
+
+def main() -> None:
+    for raw in sys.stdin.buffer:
+        if not raw.strip():
+            continue
+        try:
+            msg = json.loads(raw)
+        except ValueError:
+            send({"jsonrpc": "2.0", "id": None, "error": {"code": -32700, "message": "parse error"}})
+            continue
+        handle(msg)
+
+
+if __name__ == "__main__":
+    main()

package/metrics/client.py

@@ -343,8 +343,29 @@ def output(res: dict, timeout: float) -> str:
     return res["error"] or (res["text"] + "\n" if res["text"] else "")
 
 
+LOG_LOCKED = "sqlite3.OperationalError: database is locked"
+
+
+def llm_log_failed_only(stdout: str, stderr: str) -> bool:
+    """True when a non-zero `llm -m` exit is only llm failing to write its own
+    log after printing the answer. Seats called at once each open llm's
+    logs.db, and the loser's crash (2026-10-22, inkling-small-free answered
+    "pong") turned an answer into an error. llm writes the log only once the
+    response is complete, so stdout is the whole answer; the same lock in the
+    migration llm runs before the call leaves stdout empty and stays an error.
+    Not `llm -n`: metrics/cost.py reads that log for token counts."""
+    lines = [ln.strip() for ln in (stderr or "").splitlines() if ln.strip()]
+    return bool((stdout or "").strip()) and bool(lines) and lines[-1] == LOG_LOCKED
+
+
 def _selftest() -> None:
-    """The Responses stream parser: an answer, a mid-stream usage limit, a cut-off stream."""
+    """The Responses stream parser: an answer, a mid-stream usage limit, a
+    cut-off stream; and which `llm` failures are only its log."""
+    tb = "Traceback (most recent call last):\n  File \"llm/cli.py\", in prompt\n" + LOG_LOCKED + "\n"
+    assert llm_log_failed_only("pong\n", tb)
+    assert not llm_log_failed_only("", tb)                                  # locked before the call
+    assert not llm_log_failed_only("half an ans", "openai.APIError: stream cut\n")
+    assert not llm_log_failed_only("pong", tb + "Error: something after\n")
     ok = (b'event: x\ndata: {"type":"response.output_text.delta","delta":"re"}\n\n'
           b'data: {"type":"response.completed","response":{"model":"m","usage":{"total_tokens":3},'
           b'"output":[{"type":"reasoning"},{"type":"message","content":[{"type":"output_text","text":"ready"}]}]}}\n')

package/metrics/route.py

@@ -136,7 +136,12 @@ def full_tag_of(alias: str) -> str:
     """
     from run import _registered_names
     import endpoints
-    tag = _json(DATA / "config" / "provider-tags.json").get(_registered_names().get(alias, alias))
+    # Tags are keyed by the served name. A decision model is not in the llm
+    # yaml, so its served name comes from config/decision-models.json.
+    served = (_registered_names().get(alias)
+              or (_json(DATA / "config" / "decision-models.json").get(alias) or {}).get("served")
+              or alias)
+    tag = _json(DATA / "config" / "provider-tags.json").get(served)
     if not tag:
         name = endpoints.endpoint_name(endpoints.registrations().get(alias) or {})
         tag = name if name and not endpoints.hub(name) else None

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wendellresearch-ai/benchwarmers",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Self-hosted panel gateway for LLMs: ask a collection of models at once, keep every answer side by side, rotate explorer models in, and let your verdicts pick the next panel.",
   "license": "MIT",
   "author": "Wendell Research",
@@ -33,6 +33,7 @@
     "bin/",
     "serve.py",
     "gateway.py",
+    "mcp_server.py",
     "metrics/*.py",
     "scripts/*.py",
     "suite/*.py",

package/scripts/check.py

@@ -57,6 +57,8 @@ CHECKS = [
     ("scripts/test_auth.py", [], False),
     ("scripts/test_first_run.py", [], False),
     ("scripts/test_no_owui.py", [], False),
+    ("scripts/test_llm_log.py", [], False),
+    ("scripts/test_mcp.py", [], False),
     ("suite/test_score.py", [], False),
     ("suite/test_mine.py", [], False),
     ("suite/test_seed.py", [], False),