@wendellresearch-ai/benchwarmers 0.6.2 → 0.6.4

package/AGENTS.md

@@ -124,7 +124,9 @@ in `bench.skipped`. One the catalog knows nothing about sits once its model
 popup ticks Vision / OCR.
 After the person checks an answer against the real work, record a verdict
 (dashboard, or `POST /bench/verdict`); verdicts choose the next panels. See
-GATEWAY.md for the whole contract.
+GATEWAY.md for the whole contract. To set an app up on the gateway (opencode,
+VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat
+apps), CLIENTS.md has each one's configuration, file by file.
 
 ## Beyond this machine
 

package/BENCHMARKS.md

@@ -70,7 +70,7 @@ earlier scores on the same tasks, over the last 30 days (or 60).
 
 Whether the provider was there when the bench called it, from the calls the
 bench makes anyway: one cell a day, by provider and then by model, over the
-last 7 days (or 30). At a glance has each model's 7 days too.
+last 7 days (or 30).
 
 - **Down**: the provider was busy or overloaded (a 5xx, high demand, a shared
   pool's 429), unreachable (a broken stream, no reply), timed out (the

package/README.md

@@ -419,6 +419,8 @@ collections and MCP server, with a Copy button: opencode, VS Code chat, Codex
 CLI, Zed, Continue, Cline and Kilo Code, Claude Code, goose, and chat apps
 (LobeHub, AnythingLLM, Cherry Studio, Chatbox, LibreChat). Where keys are
 asked for, it makes one and fills it in, never into a file that may sync.
+The same set-ups are in [CLIENTS.md](CLIENTS.md), with a fresh install's
+address and collections, for reading here or handing to an agent.
 
 <!-- The MCP Registry's proof that the PyPI package is this server (server.json):
 mcp-name: io.github.WendellResearch-AI/benchwarmers -->

package/USAGE.md

@@ -25,15 +25,14 @@ coding agent doing it with you); the gateway's whole contract is in
 
 | Group | Page | What it is for |
 | --- | --- | --- |
-| Observe | **At a glance** | Every model on the bench: whether it works right now, its uptime over 7 days, its role, and its connection. Click a name for its popup: role, alias, facts, options, retire. |
+| Observe | **Benchmarks** | The home page. How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, each model against its own earlier scores, uptime, speed, and the decision models. Each report shares as an image or its table. |
 | | **Action needed** | What will not fix itself, by the connection it comes from, with Retry and Retire on each model. |
-| | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, each model against its own earlier scores, uptime, speed, and the decision models. Each report shares as an image or its table. |
 | | **Quota management** | A tile per provider: what is left, when it resets, and which of its models are resting. Then the week's resets, and what the bench sent. |
 | | **Logs** | Gateway calls (who sat and how it went, never the prompt), changes made from the dashboard, the scheduler, and onboarding runs. |
 | Gateway | **Endpoints** | Where models are served from. Add an endpoint (a provider's sign-in, or any OpenAI-compatible API key), change a key, and see what each one serves. A model server running on this machine, at its usual port or any other its process is listening on (LM Studio, Jan, llama.cpp, vLLM and the like), is offered under Found on this machine: Add opens the dialog with it filled in, to test and save. Also your own app registrations and rate limits per connection. |
-| | **Models** | Every model on the bench, by the connection it comes from. |
+| | **Models** | Every model on the bench, by the connection it comes from: what it holds, its reply time and its limits. One search above every table, each table folds, and Collapse all / Expand all. Click a name for its popup: role, alias, facts, options, retire. |
 | | **Collections** | The named panels your apps ask for, and the gateway's settings. |
-| | **Clients** | Each popular app's set-up (opencode, VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat apps), filled in from this install, with Copy and a gateway key made on the spot. |
+| | **Clients** | Each popular app's set-up (opencode, VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat apps), filled in from this install, with Copy and a gateway key made on the spot. [CLIENTS.md](CLIENTS.md) has the same, for a fresh install. |
 | | **Discover** | Models worth adding, from public catalogs, with the ways this install can reach each one. |
 | Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, export and import, how the install is reached (shown only), and Restart. |
 | | **Info** | The source repository, the version (linking its release), Python, data folder and counts. |
@@ -52,8 +51,7 @@ clears the alert) and Retire. A group of two or more has Retry all and Retire
 all; a connection held because every model on it fails says so above its
 models, with any link the provider gave. Automatic retirements from the last
 week are folded at the end, each with Restore. Its count sits beside it in the
-sidebar, and At a glance opens on one line saying what is waiting and how
-soon. An alert whose models have all been retired goes at once. A model that
+sidebar. An alert whose models have all been retired goes at once. A model that
 is out of quota is not counting down: days the bench held its calls back until
 a stated reset, an account's shared free quota was spent, or the provider said
 the model's free quota is used up, pause its clock, and while it is still out

package/VERSION

@@ -1 +1 @@
-0.6.2
+0.6.4

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wendellresearch-ai/benchwarmers",
-  "version": "0.6.2",
+  "version": "0.6.4",
   "description": "Self-hosted panel gateway for LLMs: ask a collection of models at once, keep every answer side by side, rotate explorer models in, and let your verdicts pick the next panel.",
   "mcpName": "io.github.WendellResearch-AI/benchwarmers",
   "license": "MIT",

package/scripts/check.py

@@ -49,6 +49,7 @@ CHECKS = [
     ("scripts/onboard.py", ["--selftest"], False),
     ("scripts/export_check.py", ["--selftest"], False),
     ("scripts/notice.py", ["--check"], False),
+    ("scripts/clients_doc.py", ["--check"], False),       # CLIENTS.md is what the Clients page shows
     ("scripts/check_themes.py", [], False),
     ("scripts/test_owui.py", [], False),
     ("scripts/test_hubs.py", [], False),

package/scripts/clients_doc.py

@@ -0,0 +1,98 @@
+"""Write CLIENTS.md from the dashboard's Clients page, so each app's set-up is in
+the repository for people and agents to read, and never drifts from the page.
+
+    python scripts/clients_doc.py           # write CLIENTS.md
+    python scripts/clients_doc.py --check   # fail if CLIENTS.md is not what it would write (check.py)
+
+One source: serve.client_setups, the function the Clients page draws. The page
+fills in this install (its address, its collections, a key made on the spot);
+here they are the defaults a fresh install has: the server at 127.0.0.1:8122,
+collections `core` and `fast`, and every call needing a gateway key, read from
+BENCHWARMERS_KEY. The page's HTML (code, links) becomes Markdown.
+"""
+
+from __future__ import annotations
+
+import html
+import os
+import pathlib
+import re
+import sys
+import tempfile
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+OUT = ROOT / "CLIENTS.md"
+COLLECTIONS = [("core", None, False), ("fast", None, False)]
+
+
+def md(text: str) -> str:
+    """The page's inline HTML as Markdown: code, links, line breaks; any other tag dropped."""
+    text = re.sub(r"<code>(.*?)</code>", lambda m: "`" + html.unescape(m.group(1)) + "`", text)
+    text = re.sub(r"<a [^>]*href='([^']*)'[^>]*>(.*?)</a>", r"[\2](\1)", text)
+    text = re.sub(r"<br\s*/?>", "  \n", text)
+    return html.unescape(re.sub(r"<[^>]+>", "", text))
+
+
+def fence(lang: str, text: str) -> str:
+    tick = "````" if "```" in text else "```"
+    return f"{tick}{lang}\n{text}\n{tick}"
+
+
+def render() -> str:
+    # serve reads the data folder at import: an empty temp one, never the install's.
+    os.environ.setdefault("BENCHWARMERS_HOME", tempfile.mkdtemp(prefix="bench-clients-doc-"))
+    sys.path[:0] = [str(ROOT), str(ROOT / "metrics"), str(ROOT / "suite")]
+    import serve
+    root = serve.DEFAULT_SERVER
+    out = ["# Setting up clients",
+           "",
+           "Every app Benchwarmers has a ready set-up for, as the dashboard's **Clients** page shows it,"
+           " written out for people and for agents setting an app up. The page fills in your own install:"
+           " its address, your collections and a gateway key made on the spot. Here they are a fresh"
+           f" install's: the server at `{root}`, the collections `core` and `fast`, and a gateway key"
+           f" read from `{serve.KEY_ENV}`. Replace them with yours.",
+           "",
+           "Agents get a collection in route mode (one member with its tools, `bench/<collection>:route`);"
+           " chat windows a merged answer (`:merged`); the MCP server the panel itself, as ask and verdict"
+           f" tools. Gateway `{root}/v1` · MCP server `{root}/mcp`.",
+           "",
+           f"_Generated from `serve.client_setups` by `scripts/clients_doc.py`; each app's format follows its"
+           f" own docs as read on {serve.CLIENTS_READ}. Do not edit by hand: run the script._",
+           "",
+           "## The gateway key",
+           "",
+           "When every call needs a key (`BENCHWARMERS_REQUIRE_KEYS=1`, or an admin token), make one on"
+           " Settings or the Clients page. Most set-ups read it from"
+           f" `{serve.KEY_ENV}`:",
+           ""]
+    for s in serve._env_snips(serve.KEY_ENV):
+        out += [f"**{s['label']}**", "", fence(s["lang"], s["text"]), ""]
+    setups = serve.client_setups(root, COLLECTIONS, True)
+    out += ["## Contents", ""] + [f"- [{c['name']}](#{c['id']})" for c in setups] + [""]
+    for c in setups:
+        out += [f"<a id='{c['id']}'></a>", "", f"## {c['name']}", "", md(c["gets"]), "",
+                f"**Where it goes:** {md(c['where'])}", ""]
+        for s in c["blocks"]:
+            out += [f"**{s['label']}**", "", fence(s["lang"], s["text"]), ""]
+        docs = ", ".join(f"[{t}]({u})" for u, t in c["docs"])
+        out += [f"**Watch for:** {md(c['watch'])}", "",
+                f"Written from {docs}, read {serve.CLIENTS_READ}"
+                + (f": {md(c['docs_note'])}" if c.get("docs_note") else "."), ""]
+    return "\n".join(out).rstrip() + "\n"
+
+
+def main() -> int:
+    text = render()
+    if "--check" in sys.argv:
+        if not OUT.exists() or OUT.read_text(encoding="utf-8") != text:
+            print("CLIENTS.md is not what serve.client_setups gives: run python scripts/clients_doc.py")
+            return 1
+        print("CLIENTS.md matches the Clients page")
+        return 0
+    OUT.write_text(text, encoding="utf-8", newline="\n")
+    print(f"wrote {OUT.name}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/scripts/provider_tags.py

@@ -68,13 +68,29 @@ def get(hub: str, path: str) -> dict:
         return json.load(response)
 
 
+def _host(url: str) -> str:
+    return (url or "").split("//", 1)[-1].split("/", 1)[0].split(":", 1)[0].lower()
+
+
 def connection_names(hub: str) -> list[str]:
-    """One short name per connection index. URLs only — never the keys."""
+    """One short name per connection index. URLs only — never the keys.
+
+    A provider reached both through the hub and on a direct endpoint of its
+    own (Cloudflare: chat through Open WebUI, decisions only direct) is ONE
+    connection: the hub's connection to that host takes the direct endpoint's
+    name, which is what its directly registered models are tagged (main). Any
+    provider, by host; other hubs (CLI Proxy API) are not direct endpoints."""
     urls = get(hub, "/openai/config").get("OPENAI_API_BASE_URLS", [])
+    direct: dict[str, str] = {}
+    for name, entry in endpoints.load().items():
+        if name != hub and not endpoints.hub(name):
+            direct.setdefault(_host(entry.get("base", "")), name)
     names = []
     for url in urls:
-        host = url.split("//", 1)[-1].split("/", 1)[0].split(":", 1)[0]
-        if host.replace(".", "").isdigit():
+        host = _host(url)
+        if host in direct and host not in (endpoints.connections().get("hosts") or {}):
+            names.append(direct[host])
+        elif host.replace(".", "").isdigit():
             # A BARE IP NAMES NOTHING, and guessing from it was worse than
             # useless: a proxy labelled `lan-proxy` is described by where it
             # sits rather than what it is, and Open WebUI does not expose

package/scripts/test_no_owui.py

@@ -236,6 +236,14 @@ def owner(fake: str) -> None:
     assert tags["fast-1"] == "groq" and tags["llm7.codestral"] == "llm7-api" and tags["vendor/x:free"] == "openrouter", tags
     assert "never-read" not in json.dumps(tags)
     assert route.connection_of("fast-1") == "groq" and route.connection_of("x-free") == "openrouter"
+    # A provider also on a direct endpoint of its own is one connection: the hub's takes the endpoint's name.
+    real_load, both = endpoints.load, dict(endpoints.load())
+    both["groq-direct"] = {"base": "https://api.groq.com/openai/v1", "key": "groq-direct"}
+    endpoints.load = lambda *a, **k: both
+    try:
+        assert provider_tags.connection_names("openwebui") == ["groq-direct", "llm7-api"]
+    finally:
+        endpoints.load = real_load
     # A decision model onboarded from the direct endpoint: in no registration, untagged, its `via` is its connection.
     write("decision-models.json", {"dec-1": {"served": "vendor/decide:free", "endpoint": "openrouter", "via": "openrouter"},
                                    "dec-hub": {"served": "hub-decide", "endpoint": "openwebui", "via": "openwebui"}})