@wendellresearch-ai/benchwarmers 0.3.1 → 0.3.3

package/GATEWAY.md

@@ -167,6 +167,14 @@ A verdict is `POST /bench/verdict` with `{"call", "useful": true|false, "note",
 dialog in the dashboard's Collections tab, under Gateway (click a seat). Sending the same verdict
 twice writes nothing.
 
+A client that weighs its seats by how they have done reads the verdicts back:
+`GET /bench/record?collection=bench/vision&task=thicken slab faces` answers
+`{"collection", "task", "aliases": {"<alias>": {"useful", "notUseful",
+"judged", "lastAt"}}}`, counting each model's judged seats on that collection.
+`task` (optional) keeps the verdicts whose note holds it. Counts only: never a
+prompt, an answer or a note. thicken breaks ties between its vision seats with
+it.
+
 ## Streaming
 
 With `"stream": true` (or the format's own way of asking, below):
@@ -511,7 +519,7 @@ key limited to its collection and a rate, or keep the key on the app's server.
 | Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily (scheduled task WRBench-Catalog); a free model whose price goes above zero raises a "went paid" alert. |
 | Discover tab | `metrics/discover.py`: models.dev and FreeLLMAPI merged into one list of models worth adding, free first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
 | Merged replies | Opt-in: one merger answer as `content`, every seat still in `bench` (The reply). |
-| MCP server | `mcp_server.py` (`benchwarmers mcp`): the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`, over HTTP to a running server (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`), so it is a caller like any other. The Claude Code plugin in `plugin/` wraps it with a command and a skill. |
+| MCP server | `mcp_server.py`: the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`. Two transports for one `answer()`: `POST /mcp` (or `/v1/mcp`) on either port, streamable HTTP with a JSON reply per request, its tools calling the server's own routes over loopback with the caller's credential and Host; and stdio (`benchwarmers mcp`), a caller like any other over HTTP (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`). The Claude Code plugin in `plugin/` connects to `/mcp` and adds a command, a skill and a `Stop` hook (`plugin/hooks/hooks.json`, an `mcp_tool` hook) that calls `pending_verdicts` with the session's `transcript_path`: the call ids in the transcript (Claude Code keeps no `Mcp-Session-Id`, so the transcript is the record of whose panels they were), of which the call log says which are answered seats with no verdict (`POST /bench/pending`), returned once as `additionalContext`. A server on another machine cannot read the transcript, so there the hook says nothing. A tool call answers as SSE with progress notifications, which reset Claude Code's 300-second idle limit. |
 
 ## Rules
 

package/README.md

@@ -6,10 +6,11 @@
 </h1>
 
 <p align="center">
-  <a href="https://github.com/WendellResearch-AI/benchwarmers/releases"><img src="https://img.shields.io/github/v/release/WendellResearch-AI/benchwarmers?color=990000" alt="Latest release"></a>
+  <a href="https://github.com/WendellResearch-AI/benchwarmers/releases/latest"><img src="https://img.shields.io/github/v/release/WendellResearch-AI/benchwarmers?logo=github&color=181717" alt="Latest release"></a>
   <a href="https://www.npmjs.com/package/@wendellresearch-ai/benchwarmers"><img src="https://img.shields.io/npm/v/@wendellresearch-ai/benchwarmers?color=cb3837&logo=npm" alt="npm"></a>
   <a href="https://pypi.org/project/benchwarmers/"><img src="https://img.shields.io/pypi/v/benchwarmers?color=3775a9&logo=pypi&logoColor=white" alt="PyPI"></a>
-  <a href="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml"><img src="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml/badge.svg" alt="Checks"></a>
+  <a href="https://github.com/WendellResearch-AI/benchwarmers/actions/workflows/check.yml"><img src="https://img.shields.io/github/actions/workflow/status/WendellResearch-AI/benchwarmers/check.yml?branch=main&logo=github&label=CI%20tests" alt="CI tests"></a>
+  <a href="https://registry.modelcontextprotocol.io/?q=benchwarmers"><img src="https://img.shields.io/badge/MCP_Registry-111111?logo=modelcontextprotocol&logoColor=white" alt="In the official MCP Registry"></a>
   <a href="#docker-and-hosting"><img src="https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white" alt="Runs in Docker"></a>
   <br>
   <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-OpenAI_%7C_Responses-111111" alt="OpenAI chat completions and Responses API"></a>
@@ -17,7 +18,6 @@
   <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Gemini-111111" alt="Gemini API"></a>
   <a href="GATEWAY.md#other-api-formats"><img src="https://img.shields.io/badge/API-Ollama-111111" alt="Ollama API"></a>
   <a href="GATEWAY.md#route-mode-agent-clients"><img src="https://img.shields.io/badge/agents-tool_calling-111111" alt="Agent clients with tool calling, through route mode"></a>
-  <a href="https://registry.modelcontextprotocol.io/v0/servers?search=io.github.WendellResearch-AI/benchwarmers"><img src="https://img.shields.io/badge/MCP-Registry-111111" alt="MCP server in the official MCP Registry"></a>
   <br>
   <a href="#try-it"><img src="https://img.shields.io/badge/Python-3.11_%7C_3.12_%7C_3.13-3776AB?logo=python&logoColor=white" alt="Python 3.11 to 3.13"></a>
   <a href="#try-it"><img src="https://img.shields.io/badge/core-stdlib_only-2ea44f" alt="The core needs only the Python standard library"></a>
@@ -279,28 +279,47 @@ no server at `BENCHWARMERS_URL`.
 
 ## From Claude Code and other agents
 
-`benchwarmers mcp` is an MCP server on stdio with three tools:
-`list_collections`, `ask` (a collection, a prompt, optionally `merged`,
-`with` and `system`) and `verdict`. It is a client of a Benchwarmers that is
-already running, over the routes above, so start the dashboard (or the
-service) first. The tool descriptions carry the rules: ask a collection, not
-a favourite model; verify before acting; name every seat; send a verdict per
-seat you judged.
+A running Benchwarmers is an MCP server at `/mcp` on its own port
+(`http://127.0.0.1:8122/mcp`; `/v1/mcp` works too), over streamable HTTP,
+with three tools: `list_collections`, `ask` (a collection, a prompt,
+optionally `merged`, `with` and `system`) and `verdict`. Nothing to start per
+client: every agent session shares the one server, behind the same Host,
+Origin and gateway-key checks as `/v1`. The tool descriptions carry the
+rules: ask a collection, not a favourite model; verify before acting; name
+every seat; send a verdict per seat you judged.
 
-**Claude Code plugin.** The plugin adds the MCP server, a `/benchwarmers:ask
-<collection> <what to ask>` command and a `benchwarmers` skill saying when a
-panel helps (adversarial review, exhaustive enumeration over real files) and
-how to judge it:
+**Claude Code plugin.** The plugin connects to that `/mcp` and adds a
+`/benchwarmers:ask <collection> <what to ask>` command and a `benchwarmers`
+skill saying when a panel helps (adversarial review, exhaustive enumeration
+over real files) and how to judge it:
 
     /plugin marketplace add WendellResearch-AI/benchwarmers
     /plugin install benchwarmers@benchwarmers
 
-It needs Node 18 or later and Python 3.11 or later on PATH.
+It finds the server through `BENCHWARMERS_URL` (default
+`http://127.0.0.1:8122`; set it if you changed `BENCH_PORT`) and sends
+`BENCHWARMERS_KEY` where keys are required. It also adds a `Stop` hook: at
+the end of a turn it asks the server (the `pending_verdicts` tool, so nothing
+extra runs on your machine) which seats of that session's panels have no
+verdict yet, and reminds Claude of them once, as hook feedback. The server
+finds the session's panels in its transcript, so the reminder needs a server
+on the same machine; with one elsewhere the hook stays silent. A verdict sent
+any way counts, the dashboard's included. With the plugin enabled and no
+Benchwarmers running, the hook shows a hook-error notice at the end of each
+turn, as `/mcp` shows the server as failed; start the server, or disable the
+plugin for that session.
+
+**Any other MCP client** that speaks HTTP:
 
-**Any other MCP client** (or Claude Code without the plugin): run
-`benchwarmers mcp` once installed, `uvx benchwarmers mcp` or
+```json
+{"mcpServers": {"benchwarmers": {"type": "http", "url": "http://127.0.0.1:8122/mcp"}}}
+```
+
+with `"headers": {"Authorization": "Bearer <gateway key>"}` where keys are
+required. **A client that only speaks stdio**: run `benchwarmers mcp` once
+installed, `uvx benchwarmers mcp` or
 `npx -y @wendellresearch-ai/benchwarmers mcp` without installing, or `python mcp_server.py` from a
-checkout. For example:
+checkout, a small process per client that calls the same server. For example:
 
 ```json
 {"mcpServers": {"benchwarmers": {"command": "npx", "args": ["-y", "@wendellresearch-ai/benchwarmers", "mcp"],
@@ -316,6 +335,24 @@ On native Windows, a client that cannot start `npx` itself needs
 | `BENCHWARMERS_KEY` | a gateway key from Settings, sent as `Authorization: Bearer`; needed for a server on another machine, or where keys are required |
 | `BENCHWARMERS_TIMEOUT` | seconds to wait for a panel, default 900 |
 
+**Behind an MCP aggregator** (1mcp, MetaMCP, mcp-proxy, Docker's MCP Gateway
+and the like): add `http://127.0.0.1:8122/mcp` as an HTTP (streamable HTTP)
+upstream, with the `Authorization` header where keys are required; an
+aggregator that only runs stdio servers takes `benchwarmers mcp` as above.
+Give it a request timeout of a few minutes or more, as a panel takes that
+long; 1mcp's default of 60 seconds ends most of them. With 1mcp, for example:
+
+    1mcp mcp add benchwarmers --type http --url http://127.0.0.1:8122/mcp --timeout 900000
+
+The aggregator prefixes the tool names (1mcp's are
+`benchwarmers_1mcp_ask` and so on). If you also use the Claude Code plugin,
+disable the plugin's own `benchwarmers` server in `/mcp` so the tools are not
+listed twice; the skill and `/benchwarmers:ask` keep working, though the
+command then asks before each tool call, since its pre-approved tools carry
+the plugin's names. The plugin's verdict hook calls the plugin's own server,
+so it does not work through an aggregator: with that server disabled it only
+shows a hook-error notice at the end of each turn.
+
 **An agent running on a collection.** The MCP tools ask a panel for a
 second opinion; an agent can also *be driven by* a collection, in **route
 mode** (`bench/<collection>:route`): one member, picked by its record, gets

package/VERSION

@@ -1 +1 @@
-0.3.1
+0.3.3

package/gateway.py

@@ -150,6 +150,7 @@ import client  # noqa: E402
 import decide  # noqa: E402
 import formats  # noqa: E402
 import guard  # noqa: E402
+import mcp_server  # noqa: E402  -- its tools, answered at /mcp too
 import model_roles  # noqa: E402
 import offload  # noqa: E402
 import route  # noqa: E402
@@ -440,12 +441,33 @@ def too_small(needed: int, aliases) -> dict[str, int]:
 
 def record_seat(task: str, call: str, label: str, alias: str, r: dict) -> dict:
     """Log one answer as an unjudged offload, the moment it is in, so explorer
-    rotation counts it even if nobody judges; returns the seat as logged."""
+    rotation counts it even if nobody judges; returns the seat as logged. The
+    note's "<seat>; <status> in" is what /bench/pending reads the status from."""
     offload.add(alias, task, None, f"{label}; {r['status']} in {r['seconds']}s"
                 + (f": {r['error'][:120]}" if r["error"] else ""), call)
     return {"call": call, "seat": label, "alias": alias, **r}
 
 
+def record(collection: str, task: str = "") -> dict[str, dict]:
+    """Each model's verdicts on one collection's seats, for a client that weighs
+    its seats by how they have done (thicken breaks ties with it): useful, not
+    useful and judged, counted, and the last one's date. `task` keeps only the
+    verdicts whose note holds it ("thicken slab faces"). Counts only: never a
+    prompt, an answer or a note."""
+    out: dict[str, dict] = {}
+    for e in offload.entries():
+        # The collection ends each seat's task as run_collection writes it: "(bench/<name>, <shape>)".
+        if e.get("useful") is None or f"({collection}," not in e.get("task", ""):
+            continue
+        if task and task not in (e.get("note") or ""):
+            continue
+        r = out.setdefault(e["model"], {"useful": 0, "notUseful": 0, "judged": 0, "lastAt": ""})
+        r["useful" if e["useful"] else "notUseful"] += 1
+        r["judged"] += 1
+        r["lastAt"] = max(r["lastAt"], e.get("date", ""))
+    return out
+
+
 def merger_candidates(collection: dict, members: list[str], shape: str, large: bool):
     """Who may merge, best first. The collection's `merger` alone when set;
     else the members with verdicts on this shape, most useful first, then the
@@ -1030,6 +1052,12 @@ class Handler(BaseHTTPRequestHandler):
             # orchestrator can choose one by task. Clients ignore extra keys.
             return self._json(200, {"object": "list", "data": [
                 {"id": n, "object": "model", "owned_by": "benchwarmers", **d} for n, d in listed(self.who)]})
+        if path == "/bench/record":
+            query = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query)
+            collection, task = (query.get(k, [""])[0] for k in ("collection", "task"))
+            if not collection:
+                return self._error(400, "GET /bench/record?collection=bench/<name>[&task=<words in the verdicts' notes>]")
+            return self._json(200, {"collection": collection, "task": task, "aliases": record(collection, task)})
         if path == "/api/tags":
             return self._json(200, formats.ollama_tags(listed(self.who)))
         if path == "/api/version":
@@ -1041,7 +1069,10 @@ class Handler(BaseHTTPRequestHandler):
             row = dict(listed(self.who)).get(name)
             return self._json(200, formats.gemini_model(name, row)) if row else \
                 self._error(404, f"{name!r} is not a collection here; GET /v1beta/models lists them", fmt=fmt)
-        self._error(404, "GET /v1/models, /api/tags, /api/version or /v1beta/models", fmt=fmt)
+        if guard.mcp_route(path):                      # no stream from the server: every reply is a POST's
+            return self._json(405, {"error": {"message": "POST /mcp, one JSON-RPC message per request"}},
+                              {"Allow": "POST"})
+        self._error(404, "GET /v1/models, /bench/record, /api/tags, /api/version or /v1beta/models", fmt=fmt)
 
     def do_POST(self):                                 # noqa: N802
         # Without this any web page could spend model calls, seat ad hoc
@@ -1053,6 +1084,8 @@ class Handler(BaseHTTPRequestHandler):
             self.who = guard.gateway_caller(self.headers, self._query_key(fmt))
         except guard.Refused as r:
             return self._error(r.status, str(r) + (TOO_BIG if r.status == 413 else ""), r.headers, fmt)
+        if guard.mcp_route(path):
+            return self._mcp(length)
         try:
             body = json.loads(self.rfile.read(length) or b"{}")
         except ValueError:
@@ -1071,6 +1104,22 @@ class Handler(BaseHTTPRequestHandler):
             except KeyError as exc:
                 return self._error(404, exc.args[0])
             return self._json(200, {"ok": True, "recorded": e})
+        if path == "/bench/pending":
+            # Which of these call ids are seats that answered (ok or empty, not errored) and have
+            # no verdict yet, however one is sent (mcp_server.py's pending_verdicts, the plugin's
+            # Stop hook). An id the log does not know is not a seat. A seat's first entry is
+            # record_seat's, its note "<seat>; <status> in <s>s".
+            calls = body.get("calls")
+            if not (isinstance(calls, list) and all(isinstance(c, str) for c in calls)):
+                return self._error(400, '"calls" must be a list of call ids')
+            first, judged = {}, set()
+            for e in offload.entries():
+                first.setdefault(e.get("call"), e)
+                if e.get("useful") is not None:
+                    judged.add(e.get("call"))
+            return self._json(200, {"pending": [
+                c for c in calls if c in first and c not in judged
+                and "; error in " not in str(first[c].get("note", "")).split(":", 1)[0]]})
         # Each format's request becomes one Ask (metrics/formats.py); a part
         # that would be lost on the way, tools above all, is refused with why.
         gemini = formats.gemini_route(path)
@@ -1099,6 +1148,66 @@ class Handler(BaseHTTPRequestHandler):
         alt = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query).get("alt", [""])[0]
         self._panel(fmt, ask, sse=alt == "sse")
 
+    def _mcp(self, length: int) -> None:
+        """MCP over streamable HTTP, a reply to each POST (JSON, or SSE for a
+        tool call from a client that takes it): mcp_server.py's
+        tools on this port, so a client starts no process of its own. They call
+        this server's own routes over loopback with the caller's credential and
+        Host (both already checked above), so a key's collections and rate
+        limit hold exactly as on /v1."""
+        try:
+            msg = json.loads(self.rfile.read(length))
+        except (ValueError, RecursionError):           # RecursionError: nesting deeper than json can parse
+            return self._json(400, {"jsonrpc": "2.0", "id": None, "error": {"code": -32700, "message": "parse error"}})
+        host, port = self.server.server_address[:2]
+        host = {"0.0.0.0": "127.0.0.1", "::": "::1"}.get(host, host)
+        sent = {h: self.headers[h] for h in ("Authorization", "Host", *guard.KEY_HEADERS) if self.headers.get(h)}
+        conn = (f"http://{'[' + host + ']' if ':' in host else host}:{port}", sent)
+        if isinstance(msg, dict) and msg.get("method") == "tools/call" and "id" in msg \
+                and "text/event-stream" in (self.headers.get("Accept") or ""):
+            return self._mcp_stream(msg, conn)
+        reply = mcp_server.answer(msg, conn)
+        if reply is None:                              # a notification: accepted, nothing to say
+            self.send_response(202)
+            self.send_header("Content-Length", "0")
+            return self.end_headers()
+        self._json(200, reply)
+
+    def _mcp_stream(self, msg: dict, conn: mcp_server.Conn) -> None:
+        """A tool call as an SSE stream: the 200 at once, then every KEEPALIVE
+        seconds while a panel runs (minutes) a progress notification when the
+        client gave a progressToken, else a keep-alive comment, then the reply.
+        A comment keeps proxies' idle timeouts away; only progress resets a
+        client's own (Claude Code aborts a tool silent for 300 seconds)."""
+        done: queue.Queue = queue.Queue()
+        threading.Thread(target=lambda: done.put(mcp_server.answer(msg, conn)), daemon=True).start()
+        params = msg.get("params") if isinstance(msg.get("params"), dict) else {}
+        token = (params.get("_meta") or {}).get("progressToken") if isinstance(params.get("_meta"), dict) else None
+        started = time.monotonic()
+        try:
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream")
+            self.send_header("Cache-Control", "no-cache")
+            self.end_headers()
+            while True:
+                try:
+                    reply = done.get(timeout=KEEPALIVE)
+                except queue.Empty:
+                    if token is None:
+                        self.wfile.write(b": keep-alive\n\n")
+                    else:
+                        seconds = round(time.monotonic() - started)
+                        self.wfile.write(b"event: message\ndata: " + json.dumps(
+                            {"jsonrpc": "2.0", "method": "notifications/progress",
+                             "params": {"progressToken": token, "progress": seconds,
+                                        "message": f"the panel is running, {seconds} s"}}).encode() + b"\n\n")
+                    self.wfile.flush()
+                    continue
+                self.wfile.write(b"event: message\ndata: " + json.dumps(reply).encode() + b"\n\n")
+                return
+        except OSError:
+            return                                     # the caller left; the panel still finishes, every seat recorded
+
     def _show(self, body: dict) -> None:
         """POST /api/show: Ollama's model card, for a collection the key may call."""
         name = formats.ollama_model(body.get("model") or body.get("name") or "")

package/mcp_server.py

@@ -15,7 +15,8 @@ the verdicts are that server's, whichever machine it is on:
     BENCHWARMERS_TIMEOUT   seconds to wait for a panel, default 900
 
 Tools: list_collections (GET /v1/models), ask (POST /v1/chat/completions),
-verdict (POST /bench/verdict). A refusal from the server, or no server at
+verdict (POST /bench/verdict), and pending_verdicts (POST /bench/pending) for
+the plugin's Stop hook. A refusal from the server, or no server at
 all, comes back as a tool error carrying its message. Standard library only,
 and nothing from metrics/: the data folder is the server's, not this one's.
 """
@@ -24,6 +25,7 @@ from __future__ import annotations
 
 import json
 import os
+import re
 import sys
 import threading
 import urllib.error
@@ -90,6 +92,16 @@ TOOLS = [
                                    "description": "Claims it made that are untrue, one sentence each."}}},
      "annotations": {"readOnlyHint": False, "destructiveHint": False, "idempotentHint": True,
                      "openWorldHint": False}},
+    {"name": "pending_verdicts",
+     "title": "Seats still to judge",
+     "description": "For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a"
+                    " turn: the seats of this session's panels that have no verdict yet. You do not need to"
+                    " call it; send verdicts with `verdict`.",
+     "inputSchema": {"type": "object", "additionalProperties": False,
+                     "properties": {"session": {"type": "string", "description": "The Claude Code session id."},
+                                    "transcript": {"type": "string", "description": "The session's transcript_path."},
+                                    "active": {"description": "The hook's stop_hook_active: true says nothing."}}},
+     "annotations": {"readOnlyHint": True, "openWorldHint": False}},
 ]
 
 
@@ -102,11 +114,32 @@ def base_url() -> str:
     return url.removesuffix("/v1")
 
 
-def http(method: str, path: str, body: dict | None = None) -> dict:
-    url = base_url() + path
-    headers = {"Accept": "application/json"}
-    if key := (os.environ.get("BENCHWARMERS_KEY") or "").strip():
-        headers["Authorization"] = f"Bearer {key}"
+# (server, headers for it): where a tool's calls go.
+Conn = tuple[str, dict]
+
+
+def env_conn() -> Conn:
+    """The stdio server's: BENCHWARMERS_URL and BENCHWARMERS_KEY. Served over HTTP
+    (gateway.py's /mcp), the server's own address and the caller's credential instead."""
+    key = (os.environ.get("BENCHWARMERS_KEY") or "").strip()
+    return base_url(), {"Authorization": f"Bearer {key}"} if key else {}
+
+
+# A seat's call id, as ask returns it (gateway.py: p<date>-<time>-<hex>.<seat>); only ids the
+# call log knows count, so a loose match costs nothing.
+CALL_ID = re.compile(r"\bp\d{8}-\d{6}-[0-9a-f]{4}\.[a-z]+\d*\b")
+TRANSCRIPT_TAIL = 8 * 1024 * 1024                    # bytes of a transcript read: its recent panels
+# Per Claude Code session, the seats pending_verdicts already reminded it of. ponytail: in memory
+# and capped, so a restart reminds once more; keep it in the data folder if that ever matters.
+REMINDED: dict[str, set] = {}
+REMINDED_LOCK = threading.Lock()
+MAX_SESSIONS = 1000
+
+
+def http(conn: Conn, method: str, path: str, body: dict | None = None) -> dict:
+    base, extra = conn
+    url = base + path
+    headers = {"Accept": "application/json", **extra}
     data = None
     if body is not None:
         data = json.dumps(body).encode()
@@ -126,21 +159,21 @@ def http(method: str, path: str, body: dict | None = None) -> dict:
     except (urllib.error.URLError, ConnectionError) as e:
         reason = getattr(e, "reason", e)
         if isinstance(reason, TimeoutError):
-            raise ToolError(f"no answer from Benchwarmers at {base_url()} in {timeout:.0f} s;"
+            raise ToolError(f"no answer from Benchwarmers at {base} in {timeout:.0f} s;"
                             " set BENCHWARMERS_TIMEOUT for longer panels") from None
-        raise ToolError(f"Benchwarmers is not running at {base_url()} ({reason}). Start it (`benchwarmers`,"
+        raise ToolError(f"Benchwarmers is not running at {base} ({reason}). Start it (`benchwarmers`,"
                         " or its service), or point BENCHWARMERS_URL at the server you use.") from None
     except TimeoutError:
-        raise ToolError(f"no answer from Benchwarmers at {base_url()} in {timeout:.0f} s;"
+        raise ToolError(f"no answer from Benchwarmers at {base} in {timeout:.0f} s;"
                         " set BENCHWARMERS_TIMEOUT for longer panels") from None
     except ValueError:
-        raise ToolError(f"{base_url()} answered {method} {path} with something that is not JSON;"
+        raise ToolError(f"{base} answered {method} {path} with something that is not JSON;"
                         " is BENCHWARMERS_URL a Benchwarmers server?") from None
 
 
-def list_collections(_args: dict) -> list[str]:
+def list_collections(_args: dict, conn: Conn) -> list[str]:
     rows = []
-    for m in http("GET", "/v1/models").get("data") or []:
+    for m in http(conn, "GET", "/v1/models").get("data") or []:
         if ":" in m.get("id", ""):                     # the same collection in its other reply mode
             continue
         together = m.get("together") or {}
@@ -153,7 +186,7 @@ def list_collections(_args: dict) -> list[str]:
     return [json.dumps(rows, indent=1, ensure_ascii=False)]
 
 
-def ask(args: dict) -> list[str]:
+def ask(args: dict, conn: Conn) -> list[str]:
     collection, prompt = args.get("collection"), args.get("prompt")
     if not (isinstance(collection, str) and collection.strip()):
         raise ToolError("`collection` is required: a name from list_collections, e.g. core")
@@ -168,7 +201,7 @@ def ask(args: dict) -> list[str]:
         body["bench"] = {"reply": "merged" if args["merged"] else "all"}
     if args.get("with"):
         body["with"] = args["with"]
-    reply = http("POST", "/v1/chat/completions", body)
+    reply = http(conn, "POST", "/v1/chat/completions", body)
     content = reply["choices"][0]["message"]["content"]
     bench = reply.get("bench") or {}
     merged = bench.get("reply") == "merged"
@@ -185,19 +218,59 @@ def ask(args: dict) -> list[str]:
             + json.dumps(data, indent=1, ensure_ascii=False)]
 
 
-def verdict(args: dict) -> list[str]:
+def pending_verdicts(args: dict, conn: Conn) -> list[str]:
+    """For the plugin's Stop hook: the seats of this session's panels with no verdict yet, as the
+    hook's feedback for Claude (Claude Code reads it as a command hook's stdout), once per seat;
+    nothing when all are judged, already reminded of, or the hook is already continuing a turn.
+
+    The session's panels are the call ids in its transcript (the hook's transcript_path): Claude
+    Code keeps no Mcp-Session-Id, so the transcript is the one record of which panels were whose,
+    a subagent's included once its report names them. Read by whichever process runs this: the
+    stdio server always can, a server on another machine cannot, and then this says nothing.
+    Which of those ids are unjudged seats that answered is the server's call log, so a verdict
+    sent any way counts."""
+    session, path = args.get("session"), args.get("transcript")
+    if str(args.get("active")).lower() == "true" or not (isinstance(session, str) and isinstance(path, str)) \
+            or not path.endswith(".jsonl"):
+        return [""]
+    try:
+        with open(path, "rb") as fh:
+            fh.seek(max(0, os.fstat(fh.fileno()).st_size - TRANSCRIPT_TAIL))
+            calls = list(dict.fromkeys(CALL_ID.findall(fh.read().decode("utf-8", "replace"))))
+    except OSError:
+        return [""]
+    if not calls:
+        return [""]
+    pending = http(conn, "POST", "/bench/pending", {"calls": calls}).get("pending") or []
+    with REMINDED_LOCK:
+        if session not in REMINDED and len(REMINDED) >= MAX_SESSIONS:
+            REMINDED.pop(next(iter(REMINDED)))         # the oldest session
+        reminded = REMINDED.setdefault(session, set())
+        due = [c for c in pending if c not in reminded]
+        reminded.update(due)
+    if not due:
+        return [""]
+    return [json.dumps({"hookSpecificOutput": {"hookEventName": "Stop", "additionalContext": (
+        f"Benchwarmers: {len(due)} seat{'s' * (len(due) != 1)} from this session's panels"
+        f" {'have' if len(due) != 1 else 'has'} no verdict yet: {', '.join(due)}. Send a verdict for each"
+        " seat you judged (the verdict tool, with its true claims in `real` and untrue ones in `false`),"
+        " or say in one line why not.")}})]
+
+
+def verdict(args: dict, conn: Conn) -> list[str]:
     if not (isinstance(args.get("call"), str) and args["call"].strip()):
         raise ToolError("`call` is required: the seat's call id from ask")
     if not isinstance(args.get("useful"), bool):
         raise ToolError("`useful` must be true or false")
     body = {"call": args["call"].strip(), "useful": args["useful"], "note": args.get("note") or "",
             "real": args.get("real") or [], "false": args.get("false") or []}
-    http("POST", "/bench/verdict", body)
+    http(conn, "POST", "/bench/verdict", body)
     return [f"Recorded {body['call']}: {'useful' if body['useful'] else 'not useful'},"
             f" {len(body['real'])} real, {len(body['false'])} false."]
 
 
-HANDLERS = {"list_collections": list_collections, "ask": ask, "verdict": verdict}
+HANDLERS = {"list_collections": list_collections, "ask": ask, "verdict": verdict,
+            "pending_verdicts": pending_verdicts}
 OUT = threading.Lock()
 
 
@@ -208,46 +281,70 @@ def send(message: dict) -> None:
         sys.stdout.buffer.flush()
 
 
-def call_tool(rid, params: dict) -> None:
+def call_tool(rid, params: dict, conn: Conn) -> dict:
     name, args = params.get("name"), params.get("arguments") or {}
-    if name not in HANDLERS:
-        return send({"jsonrpc": "2.0", "id": rid, "error": {"code": -32602, "message": f"unknown tool {name!r}"}})
+    if not isinstance(name, str) or name not in HANDLERS:
+        return {"jsonrpc": "2.0", "id": rid, "error": {"code": -32602, "message": f"unknown tool {name!r}"}}
     try:
         if not isinstance(args, dict):
             raise ToolError("arguments must be an object")
-        texts, failed = HANDLERS[name](args), False
+        texts, failed = HANDLERS[name](args, conn), False
     except ToolError as e:
         texts, failed = [str(e)], True
     except Exception as e:                             # noqa: BLE001 -- a bad reply must not kill the server
         texts, failed = [f"{type(e).__name__}: {e}"], True
-    send({"jsonrpc": "2.0", "id": rid,
-          "result": {"content": [{"type": "text", "text": t} for t in texts], "isError": failed}})
+    return {"jsonrpc": "2.0", "id": rid,
+            "result": {"content": [{"type": "text", "text": t} for t in texts], "isError": failed}}
 
 
-def handle(msg) -> None:
+def answer(msg, conn: Conn) -> dict | None:
+    """The reply to one JSON-RPC message, or None for a notification. Both
+    transports: the stdio loop below and the HTTP route (gateway.py, /mcp).
+    Never raises: a request left unanswered would hang its client."""
+    try:
+        return _answer(msg, conn)
+    except Exception as e:                             # noqa: BLE001
+        rid = msg.get("id") if isinstance(msg, dict) else None
+        return {"jsonrpc": "2.0", "id": rid, "error": {"code": -32603, "message": f"{type(e).__name__}: {e}"}}
+
+
+def _answer(msg, conn: Conn) -> dict | None:
     if not isinstance(msg, dict) or msg.get("jsonrpc") != "2.0" or not isinstance(msg.get("method"), str):
         rid = msg.get("id") if isinstance(msg, dict) else None
-        return send({"jsonrpc": "2.0", "id": rid, "error": {"code": -32600, "message": "invalid request"}})
+        return {"jsonrpc": "2.0", "id": rid, "error": {"code": -32600, "message": "invalid request"}}
     if "id" not in msg:                                # a notification (initialized, cancelled): nothing to answer
-        return
+        return None
     rid, method, params = msg["id"], msg["method"], msg.get("params") or {}
+    if not isinstance(params, dict):
+        return {"jsonrpc": "2.0", "id": rid, "error": {"code": -32602, "message": "params must be an object"}}
     if method == "initialize":
         asked = params.get("protocolVersion")
-        send({"jsonrpc": "2.0", "id": rid, "result": {
+        return {"jsonrpc": "2.0", "id": rid, "result": {
             "protocolVersion": asked if asked in PROTOCOLS else PROTOCOLS[0],
             "capabilities": {"tools": {"listChanged": False}},
             "serverInfo": {"name": "benchwarmers", "title": "Benchwarmers", "version": VERSION},
-            "instructions": RULES}})
-    elif method == "ping":
-        send({"jsonrpc": "2.0", "id": rid, "result": {}})
-    elif method == "tools/list":
-        send({"jsonrpc": "2.0", "id": rid, "result": {"tools": TOOLS}})
-    elif method == "tools/call":
+            "instructions": RULES}}
+    if method == "ping":
+        return {"jsonrpc": "2.0", "id": rid, "result": {}}
+    if method == "tools/list":
+        return {"jsonrpc": "2.0", "id": rid, "result": {"tools": TOOLS}}
+    if method == "tools/call":
+        return call_tool(rid, params, conn)
+    return {"jsonrpc": "2.0", "id": rid, "error": {"code": -32601, "message": f"no method {method!r}"}}
+
+
+def reply_to(msg) -> None:
+    if (reply := answer(msg, env_conn())) is not None:
+        send(reply)
+
+
+def handle(msg) -> None:
+    if isinstance(msg, dict) and msg.get("method") == "tools/call":
         # A panel takes minutes: on its own thread, so a ping or a second call is answered meanwhile.
         # Not a daemon: when stdin closes, a call already sent (a verdict) still finishes.
-        threading.Thread(target=call_tool, args=(rid, params)).start()
+        threading.Thread(target=reply_to, args=(msg,)).start()
     else:
-        send({"jsonrpc": "2.0", "id": rid, "error": {"code": -32601, "message": f"no method {method!r}"}})
+        reply_to(msg)
 
 
 def main() -> None:
@@ -256,7 +353,7 @@ def main() -> None:
             continue
         try:
             msg = json.loads(raw)
-        except ValueError:
+        except (ValueError, RecursionError):           # RecursionError: nesting deeper than json can parse
             send({"jsonrpc": "2.0", "id": None, "error": {"code": -32700, "message": "parse error"}})
             continue
         handle(msg)

package/metrics/guard.py

@@ -220,7 +220,14 @@ def startup(address: str) -> None:
 
 def gateway_route(path: str) -> bool:
     """A route the gateway answers, on its own port or the dashboard's."""
-    return path.startswith(GATEWAY_ROUTES) or path.rstrip("/") in OLLAMA_ROUTES
+    return path.startswith(GATEWAY_ROUTES) or path.rstrip("/") in OLLAMA_ROUTES or mcp_route(path)
+
+
+def mcp_route(path: str) -> bool:
+    """MCP over HTTP (gateway.py): /mcp, or /v1/mcp for a BENCHWARMERS_URL
+    pasted with its /v1; doubled and trailing slashes from joining a URL
+    that ends in one are the same route."""
+    return re.sub("/{2,}", "/", path.split("?", 1)[0]).rstrip("/") in ("/mcp", "/v1/mcp")
 
 
 def _sign(expires: int) -> bytes:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wendellresearch-ai/benchwarmers",
-  "version": "0.3.1",
+  "version": "0.3.3",
   "description": "Self-hosted panel gateway for LLMs: ask a collection of models at once, keep every answer side by side, rotate explorer models in, and let your verdicts pick the next panel.",
   "mcpName": "io.github.WendellResearch-AI/benchwarmers",
   "license": "MIT",

package/scripts/test_mcp.py

@@ -23,6 +23,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
 ROOT = pathlib.Path(__file__).resolve().parent.parent
 SEEN: list[tuple[str, str, dict, str]] = []          # (method, path, body, Authorization)
+JUDGED: set[str] = set()                             # call ids the stub has a verdict for
 
 MODELS = {"object": "list", "data": [
     {"id": "bench/core", "description": "second opinions", "tags": ["general"], "members": ["m1", "m2"],
@@ -44,7 +45,10 @@ class Stub(BaseHTTPRequestHandler):
         body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
         SEEN.append(("POST", self.path, body, self.headers.get("Authorization", "")))
         if self.path == "/bench/verdict":
+            JUDGED.add(body["call"])
             return self._send(200, {"ok": True, "recorded": {"call": body["call"]}})
+        if self.path == "/bench/pending":
+            return self._send(200, {"pending": [c for c in body["calls"] if c not in JUDGED]})
         if body["model"] == "bench/nope":
             return self._send(404, {"error": {"message": "no collection nope; choose one of bench/core"}})
         time.sleep(1)                                  # a panel takes a while
@@ -112,7 +116,8 @@ def main() -> int:
     assert c.request("ping")["result"] == {}
 
     tools = {t["name"]: t for t in c.request("tools/list")["result"]["tools"]}
-    assert set(tools) == {"list_collections", "ask", "verdict"}, tools
+    assert set(tools) == {"list_collections", "ask", "verdict", "pending_verdicts"}, tools
+    assert c.tool("pending_verdicts", active="false")["content"][0]["text"] == ""     # no panel yet: nothing
     assert tools["ask"]["inputSchema"]["required"] == ["collection", "prompt"]
 
     r = c.tool("list_collections")
@@ -153,10 +158,40 @@ def main() -> int:
                                                 "real": ["a true claim"], "false": ["x"]}), SEEN[-1]
     assert c.tool("verdict", call="p1.member1")["isError"]          # useful is required
 
+    # The Stop hook's tool: the call ids in the session's transcript (each once), the server says which
+    # are unjudged, and those come back as feedback for Claude, once; nothing while the hook is already
+    # continuing a turn, nor for a transcript it cannot read.
+    import tempfile
+    a, b = "p20261025-120000-abcd.member1", "p20261025-120000-abcd.explorer2"
+    with tempfile.TemporaryDirectory() as tmp:
+        transcript = pathlib.Path(tmp) / "session.jsonl"
+        transcript.write_text(f'{{"tool_result": "`{a}` ok"}}\n{{"tool_result": "`{b}` ok, `{a}` again"}}\n',
+                              encoding="utf-8")
+        hook = {"session": "s", "transcript": str(transcript)}
+        assert not c.tool("verdict", call=a, useful=True)["isError"]
+        assert c.tool("pending_verdicts", active="true", **hook)["content"][0]["text"] == ""
+        out = json.loads(c.tool("pending_verdicts", active="false", **hook)["content"][0]["text"])
+        assert SEEN[-1][1:3] == ("/bench/pending", {"calls": [a, b]}), SEEN[-1]
+        ctx = out["hookSpecificOutput"]
+        assert ctx["hookEventName"] == "Stop" and "1 seat " in ctx["additionalContext"], ctx
+        assert b in ctx["additionalContext"] and a not in ctx["additionalContext"], ctx
+        assert c.tool("pending_verdicts", active="false", **hook)["content"][0]["text"] == ""        # reminded
+        assert c.tool("pending_verdicts", session="t", transcript=str(transcript) + ".missing.jsonl",
+                      active="false")["content"][0]["text"] == ""
+        assert c.tool("pending_verdicts", session="t", transcript=str(pathlib.Path(tmp) / "x.txt"),
+                      active="false")["content"][0]["text"] == ""                                 # not a transcript
+
     assert c.request("tools/call", {"name": "nope"})["error"]["code"] == -32602
+    assert c.request("tools/call", {"name": ["x"]})["error"]["code"] == -32602         # unhashable: answered
     assert c.request("resources/list")["error"]["code"] == -32601
     c.send("{not json")
     assert c.read()["error"]["code"] == -32700
+    c.send("[" * 100_000)                                          # deeper than json parses
+    assert c.read()["error"]["code"] == -32700
+    for method in ("initialize", "tools/call"):                    # params not an object: answered, never raised
+        c.send({"jsonrpc": "2.0", "id": method, "method": method, "params": [1]})
+        assert c.read() == {"jsonrpc": "2.0", "id": method,
+                            "error": {"code": -32602, "message": "params must be an object"}}
     c.close()
 
     # An unknown protocol version gets the newest; no server running is a clear tool error.
@@ -170,9 +205,179 @@ def main() -> int:
     assert r["isError"] and "not running" in r["content"][0]["text"], r
     c.close()
     stub.shutdown()
-    print("mcp server ok: initialize, ping, tools/list, each tool, refusals, a ping mid-panel, not running")
+    over_http()
+    print("mcp server ok: initialize, ping, tools/list, each tool, refusals, a ping mid-panel, not running;"
+          " /mcp on both ports, its paths, the key and its scope through the loopback, the guard;"
+          " pending_verdicts and /bench/pending for the Stop hook")
     return 0
 
 
+def over_http() -> None:
+    """POST /mcp on the real handlers, both ports: the tools reach the server's own routes with the caller's key."""
+    import tempfile
+    import urllib.error
+    import urllib.request
+    sys.path.insert(0, str(ROOT))
+    import serve
+    import gateway
+    import guard
+
+    def post(port, path, msg, **headers):
+        headers = {"Host": f"127.0.0.1:{port}", "Content-Type": "application/json", **headers}
+        data = msg.encode() if isinstance(msg, str) else json.dumps(msg).encode()
+        req = urllib.request.Request(f"http://127.0.0.1:{port}{path}", data, headers,
+                                     method="POST" if msg is not None else "GET")
+        try:
+            with urllib.request.urlopen(req, timeout=30) as r:
+                raw = r.read()
+                return r.status, json.loads(raw) if raw else None
+        except urllib.error.HTTPError as e:
+            return e.code, json.loads(e.read() or b"null")
+
+    def tool(port, name, path="/mcp", **headers_and_args):
+        headers = {k: headers_and_args.pop(k) for k in ("Authorization", "Origin") if k in headers_and_args}
+        status, reply = post(port, path, {"jsonrpc": "2.0", "id": 7, "method": "tools/call",
+                                          "params": {"name": name, "arguments": headers_and_args}}, **headers)
+        assert status == 200 and reply["id"] == 7, (status, reply)
+        return reply["result"]
+
+    servers = [ThreadingHTTPServer(("127.0.0.1", 0), h) for h in (serve.Handler, gateway.Handler)]
+    for s in servers:
+        threading.Thread(target=s.serve_forever, daemon=True).start()
+    saved = (guard.KEYS, gateway.bench_collections.FILE, gateway.run_collection, sys.stderr,
+             os.environ.pop("BENCHWARMERS_ADMIN_TOKEN", None), os.environ.pop("BENCHWARMERS_REQUIRE_KEYS", None),
+             gateway.offload.LOG)
+    ran = []
+    with tempfile.TemporaryDirectory() as tmp:
+        try:
+            t = pathlib.Path(tmp)
+            guard.KEYS, gateway.bench_collections.FILE = t / "gateway-keys.json", t / "collections.json"
+            gateway.bench_collections.FILE.write_text(json.dumps({"core": {"members": [], "description": "x"},
+                                                                  "vision": {"members": [], "description": "y"}}),
+                                                      encoding="utf-8")
+            gateway.offload.LOG = t / "offloads.jsonl"
+            gateway.run_collection = lambda name, messages, nominated, reply, who: (
+                ran.append((name, who)) or f"answer from {name}",
+                {"seats": [{"call": f"{name}.member1", "seat": "member", "alias": "m1", "status": "ok"},
+                           {"call": f"{name}.explorer", "seat": "explorer", "alias": "x1", "status": "error"}]})
+            sys.stderr = open(os.devnull, "w")         # the access log
+            for port in (s.server_port for s in servers):
+                init = {"jsonrpc": "2.0", "id": 1, "method": "initialize",
+                        "params": {"protocolVersion": "2025-06-18", "capabilities": {}}}
+                # A BENCHWARMERS_URL with /v1 or a trailing slash, joined to /mcp, is the same route.
+                for path in ("/mcp", "/mcp/", "//mcp", "/v1/mcp", "/v1//mcp"):
+                    status, reply = post(port, path, init)
+                    assert status == 200 and reply["result"]["serverInfo"]["name"] == "benchwarmers", (path, reply)
+                assert not guard.mcp_route("/mcpx") and not guard.mcp_route("/v1/mcp/x")
+                assert post(port, "/mcp", {"jsonrpc": "2.0", "method": "notifications/initialized"}) == (202, None)
+                assert post(port, "/mcp", None)[0] == 405                     # no server stream
+                assert post(port, "/mcp", "{not json") == (400, {"jsonrpc": "2.0", "id": None,
+                                                                  "error": {"code": -32700, "message": "parse error"}})
+                status, reply = post(port, "/mcp", {"jsonrpc": "2.0", "id": 2, "method": "tools/list"})
+                assert {x["name"] for x in reply["result"]["tools"]} == {"list_collections", "ask", "verdict",
+                                                                          "pending_verdicts"}
+                r = tool(port, "list_collections", path="//mcp")
+                assert not r["isError"] and [x["name"] for x in json.loads(r["content"][0]["text"])] == \
+                    ["bench/core", "bench/vision"], r
+                r = tool(port, "ask", collection="core", prompt="hi")
+                assert not r["isError"] and "answer from core" in r["content"][0]["text"], r
+                # A client that takes SSE (every streamable-HTTP client must) gets a tool call as a stream,
+                # keep-alives while it runs and then the reply, so its idle timeouts never fire mid-panel.
+                saved_keepalive, gateway.KEEPALIVE = gateway.KEEPALIVE, 0.2
+                slow, gateway.run_collection = gateway.run_collection, lambda *a, **k: (time.sleep(0.7), slow(*a, **k))[1]
+                req = urllib.request.Request(
+                    f"http://127.0.0.1:{port}/mcp", json.dumps({"jsonrpc": "2.0", "id": 9, "method": "tools/call",
+                                                                "params": {"name": "ask", "arguments": {
+                                                                    "collection": "core", "prompt": "hi"}}}).encode(),
+                    {"Host": f"127.0.0.1:{port}", "Content-Type": "application/json",
+                     "Accept": "application/json, text/event-stream"})
+                with urllib.request.urlopen(req, timeout=30) as resp:
+                    assert resp.headers["Content-Type"] == "text/event-stream", resp.headers
+                    events = resp.read().decode()
+                assert events.startswith(": keep-alive\n\n") and events.endswith("\n\n"), events
+                reply = json.loads(events.rsplit("data: ", 1)[1])
+                assert reply["id"] == 9 and "answer from core" in reply["result"]["content"][0]["text"], reply
+                # With a progressToken, progress notifications instead: what resets a client's idle timeout.
+                req = urllib.request.Request(
+                    f"http://127.0.0.1:{port}/mcp", json.dumps({"jsonrpc": "2.0", "id": 11, "method": "tools/call",
+                                                                "params": {"name": "ask", "_meta": {"progressToken": "t1"},
+                                                                           "arguments": {"collection": "core",
+                                                                                         "prompt": "hi"}}}).encode(),
+                    {"Host": f"127.0.0.1:{port}", "Content-Type": "application/json",
+                     "Accept": "application/json, text/event-stream"})
+                with urllib.request.urlopen(req, timeout=30) as resp:
+                    messages = [json.loads(line[6:]) for line in resp.read().decode().splitlines()
+                                if line.startswith("data: ")]
+                gateway.KEEPALIVE, gateway.run_collection = saved_keepalive, slow
+                assert ": keep-alive" not in json.dumps(messages) and len(messages) >= 2, messages
+                assert all(m["method"] == "notifications/progress" and m["params"]["progressToken"] == "t1"
+                           for m in messages[:-1]), messages
+                assert messages[-1]["id"] == 11 and not messages[-1]["result"]["isError"], messages[-1]
+                # A tool call that cannot be answered still ends its stream (it once kept it open forever).
+                req = urllib.request.Request(
+                    f"http://127.0.0.1:{port}/mcp", json.dumps({"jsonrpc": "2.0", "id": 10, "method": "tools/call",
+                                                                "params": [1]}).encode(),
+                    {"Host": f"127.0.0.1:{port}", "Content-Type": "application/json",
+                     "Accept": "application/json, text/event-stream"})
+                with urllib.request.urlopen(req, timeout=5) as resp:
+                    reply = json.loads(resp.read().decode().rsplit("data: ", 1)[1])
+                assert reply["id"] == 10 and reply["error"]["code"] == -32602, reply
+                assert post(port, "/mcp", "[" * 100_000)[1]["error"]["code"] == -32700
+                # The guard on /mcp itself: Host and Origin, as on /v1.
+                assert post(port, "/mcp", init, Host="evil.example")[0] == 403
+                assert post(port, "/mcp", init, Origin="http://evil.example")[0] == 403
+            assert ran == [("core", "local")] * 6, ran
+
+            # The Stop hook over HTTP: the call ids in the transcript, of which the call log says which are
+            # seats that answered (ok or empty, not errored) and have no verdict; reminded of once; a
+            # verdict sent any way clears one; an id the log does not know is not a seat.
+            g = servers[1].server_port
+            ok, empty, failed, judged, unknown = (f"p20261025-130000-beef.{s}" for s in
+                                                  ("member1", "member2", "explorer", "explorer2", "member9"))
+            for call, note in ((ok, "member; ok in 1.0s"), (empty, "member; empty in 2.0s"),
+                               (failed, "explorer; error in 0.4s: Error: 429"), (judged, "explorer 2; ok in 1.0s")):
+                gateway.offload.add("m", "t", None, note, call)
+            gateway.offload.add("m", "t", True, "checked", judged)
+            status, reply = post(g, "/bench/pending", {"calls": [ok, empty, failed, judged, unknown]})
+            assert (status, reply) == (200, {"pending": [ok, empty]}), reply
+            assert post(g, "/bench/pending", {"calls": "x"})[0] == 400
+            transcript = t / "claude-1.jsonl"
+            transcript.write_text(" ".join((ok, empty, failed, judged, unknown, ok)), encoding="utf-8")
+
+            def pending(session="claude-1"):
+                return tool(g, "pending_verdicts", session=session, transcript=str(transcript),
+                            active="false")["content"][0]["text"]
+            ctx = json.loads(pending())["hookSpecificOutput"]["additionalContext"]
+            assert "2 seats " in ctx and ok in ctx and empty in ctx and failed not in ctx and judged not in ctx, ctx
+            assert pending() == ""                                         # once per session
+            assert ok in pending("claude-2")                               # another session: its own reminder
+            assert not tool(g, "verdict", call=ok, useful=True)["isError"]
+            assert post(g, "/bench/pending", {"calls": [ok, empty]})[1] == {"pending": [empty]}
+
+            # Keys required: /mcp asks for one, and the loopback carries it, scope and all.
+            os.environ["BENCHWARMERS_REQUIRE_KEYS"] = "1"
+            vision = {"Authorization": "Bearer " + guard.make_key("vision-only", ["vision"])}
+            ran.clear()
+            for port in (s.server_port for s in servers):
+                assert post(port, "/mcp", init)[0] == 401
+                assert post(port, "/mcp", init, Authorization="Bearer ")[0] == 401     # the plugin with no key set
+                r = tool(port, "list_collections", **vision)
+                assert [x["name"] for x in json.loads(r["content"][0]["text"])] == ["bench/vision"], r
+                r = tool(port, "ask", collection="core", prompt="hi", **vision)
+                assert r["isError"] and "may call bench/vision" in r["content"][0]["text"], r
+                assert not tool(port, "ask", collection="vision", prompt="hi", **vision)["isError"]
+            assert ran == [("vision", "vision-only")] * 2, ran
+        finally:
+            sys.stderr.close()
+            guard.KEYS, gateway.bench_collections.FILE, gateway.run_collection, sys.stderr = saved[:4]
+            gateway.offload.LOG = saved[6]
+            for name, value in (("BENCHWARMERS_ADMIN_TOKEN", saved[4]), ("BENCHWARMERS_REQUIRE_KEYS", saved[5])):
+                os.environ.pop(name, None)
+                if value is not None:
+                    os.environ[name] = value
+            for s in servers:
+                s.shutdown()
+
+
 if __name__ == "__main__":
     sys.exit(main())

package/scripts/test_verdict.py

@@ -96,7 +96,24 @@ def bench_field() -> None:
     srv.shutdown()
 
 
+def records() -> None:
+    """GET /bench/record's counts: one collection's judged seats, a task's by its notes, never an unjudged one."""
+    for call, model, coll, useful, note in [("r1.m1", "m1", "bench/vision", True, "thicken slab faces: agreed"),
+                                            ("r1.m2", "m2", "bench/vision", False, "thicken slab faces: no answer"),
+                                            ("r2.m1", "m1", "bench/vision", True, "thicken what it is: agreed"),
+                                            ("r3.m1", "m1", "bench/core", False, "a review")]:
+        offload.add(model, f"look at this ({coll}, vision)", None, "member; ok", call)
+        offload.verdict(call, useful, note)
+    offload.add("m3", "look at this (bench/vision, vision)", None, "member; ok", "r4.m3")   # never judged
+    assert gateway.record("bench/vision", "thicken slab faces") == {
+        "m1": {"useful": 1, "notUseful": 0, "judged": 1, "lastAt": offload.entries()[-1]["date"]},
+        "m2": {"useful": 0, "notUseful": 1, "judged": 1, "lastAt": offload.entries()[-1]["date"]}}
+    assert gateway.record("bench/vision")["m1"]["judged"] == 2 and "m3" not in gateway.record("bench/vision")
+    assert gateway.record("bench/core")["m1"]["notUseful"] == 1 and gateway.record("bench/none") == {}
+
+
 if __name__ == "__main__":
     verdicts()
     bench_field()
-    print("verdict and bench-field checks pass")
+    records()
+    print("verdict, bench-field and record checks pass")