@wendellresearch-ai/benchwarmers 0.5.6 → 0.6.1

package/README.md

@@ -99,7 +99,7 @@ with [AGENTS.md](AGENTS.md).
 
 ## Try it
 
-**v0.3, early.** It runs with Python 3.11 or later on Windows, macOS and
+**Pre-1.0, early.** It runs with Python 3.11 or later on Windows, macOS and
 Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
 code where these pages are thin.
 
@@ -145,6 +145,10 @@ dependencies either way: the app is the Python standard library.
    its key, which goes to the key file in llm's user folder (`keys.json`,
    shared with the `llm` CLI if you have it, not needed otherwise), never into
    the data folder. `BENCHWARMERS_KEY_<NAME>` in the environment works too.
+   To move an install to another machine, Settings → Export and import takes
+   the folder as one .zip (settings only, or everything; the endpoints' keys
+   only if you tick them) and imports it on the other install, saving what it
+   replaces to `backups/` first.
 
 3. **Start the dashboard.** With no endpoint yet it opens on adding one: sign
    in with a provider on its own page (OpenRouter, NanoGPT, Hugging Face,
@@ -153,7 +157,10 @@ dependencies either way: the app is the Python standard library.
    LiteLLM included. Anthropic and Gemini keys can also be called through
    those providers' own APIs (the API choice in the same dialog). Only providers that publish a sign-in for other apps are
    offered; adding one is an entry in `metrics/connect.py` (its docstring says
-   how).
+   how). A model server already running on this machine, at its usual port or
+   any other its process is listening on (Ollama, LM Studio, Jan, llama.cpp,
+   LocalAI, vLLM, LiteLLM, CLI Proxy API, OmniRoute), is offered under "Found
+   on this machine", its address and kind filled in.
 
        python serve.py                  # http://localhost:8122
 
@@ -195,6 +202,18 @@ dependencies either way: the app is the Python standard library.
   that is only out of quota until a stated reset is not counted against. The
   built-in scheduler runs all of this once Settings switches it on; a fresh
   install spends no calls until then.
+- **Benchmarks.** One page of reports: bugs found against false alarms, who
+  is confidently wrong, real work that helped, each model against its own
+  earlier scores (Trend), whether each provider was there when the bench
+  called it (Uptime: busy, unreachable and timed out count as down; your own
+  quota and pacing are left out), and speed. Each report shares as an image or
+  its table. [BENCHMARKS.md](BENCHMARKS.md) says how to read them.
+- **Decision models.** A model that answers typed questions with a
+  probability (Mercury Decide, say) labels a request's shape, picks a
+  collection for a request that names none (when it is sure, and the reply
+  says so), and classes an error no rule knows. Each is measured daily on
+  questions the bench's own data answers, and each use asks the one that does
+  best.
 - **Rate limits and usage.** Limits come from providers' rate-limit headers
   and error messages, and from what you set per connection or model
   (requests a minute, day or month; tokens a minute or day). They are kept per
@@ -394,6 +413,13 @@ points at it with `ANTHROPIC_BASE_URL=http://127.0.0.1:8122` and
 OpenAI-, Responses- or Ollama-speaking agents with the same base URL.
 [GATEWAY.md](GATEWAY.md), "Route mode", has the set-up and the limits.
 
+**Ready set-ups.** The dashboard's **Clients** page (Gateway) gives each
+popular app's configuration, filled in from your install's address,
+collections and MCP server, with a Copy button: opencode, VS Code chat, Codex
+CLI, Zed, Continue, Cline and Kilo Code, Claude Code, goose, and chat apps
+(LobeHub, AnythingLLM, Cherry Studio, Chatbox, LibreChat). Where keys are
+asked for, it makes one and fills it in, never into a file that may sync.
+
 <!-- The MCP Registry's proof that the PyPI package is this server (server.json):
 mcp-name: io.github.WendellResearch-AI/benchwarmers -->
 
@@ -446,8 +472,10 @@ reaches Wendell Research. What it sends, and where:
 
 - **Your prompts**, only to the endpoints you add: a provider, an aggregator,
   a hub such as Open WebUI, or a local runtime. That includes the probes, the
-  suite and onboarding, which spend calls on your keys. Add only endpoints you
-  are content to send that work to.
+  suite and onboarding, which spend calls on your keys, and the decision
+  models' daily benchmark, which puts past requests, judged answers and
+  redacted error text to them again. Add only endpoints you are content to
+  send that work to.
 - **Catalog reads**, no key and nothing of yours: FreeLLMAPI (its feed and
   its site's sitemap) and models.dev once a day, and OpenRouter's public
   model list. With an NVIDIA connection, the public file of rate limits
@@ -462,12 +490,18 @@ reaches Wendell Research. What it sends, and where:
   API asks each sign-in's own provider for its quota.
 - **Charts**: the dashboard's browser page loads Observable Plot from jsDelivr
   at a pinned version.
+- **Posts**, only when you click Post on X, Bluesky, LinkedIn or Reddit in a
+  report's Share menu: that site opens in your browser with the report's name
+  and the project's link. The image goes only to your clipboard, a file, or
+  the app you pick in your system's share sheet.
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the
   SWE-rebench-V2 rows from Hugging Face and the files from GitHub.
 
 Keys stay in the environment or llm's key file (`keys.json` in its user
-folder), never in the data folder. [PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the
-detail.
+folder), not in the data folder. An export (Settings, Export and import)
+carries them only if you tick them, and an import that carries keys first
+backs up the ones it replaces, in the data folder's `backups/`.
+[PRIVACY.md](PRIVACY.md) and [SECURITY.md](SECURITY.md) have the detail.
 
 ## Limits
 
@@ -475,8 +509,8 @@ detail.
   with tools is refused there, not trimmed. Route mode passes turns only to
   members on chat-completions endpoints, keeps its sticky conversations in
   memory (a restart starts each on a fresh pick), and has been run end to end
-  with Claude Code; the Codex, opencode and Cline set-ups are written from
-  their docs. Shadow seats (other members answering an agent's turn for
+  with Claude Code; the other set-ups on the Clients page are written from
+  each app's docs, not run. Shadow seats (other members answering an agent's turn for
   evaluation) are planned, not built.
 - **Some parts are checked against fakes only.** The native Anthropic and
   Gemini endpoints, and the LiteLLM and CLI Proxy API hub adapters, are built
@@ -486,9 +520,10 @@ detail.
   seats before it stream as they finish.
 - **Pacing and the ledger cover the built-in client.** Calls made through the
   `llm` CLI, when you choose it, are neither paced nor counted.
-- **Calls cost.** Probes, onboarding and the suite run on your keys and free
-  tiers. Onboarding runs one model at a time for that reason (Quick spreads
-  the suite over the scheduled passes); a suite pass is paced, not burst.
+- **Calls cost.** Probes, onboarding, the suite and the decision benchmark
+  run on your keys and free tiers. Onboarding runs one model at a time for
+  that reason (Quick spreads the suite over the scheduled passes); a suite
+  pass is paced, not burst.
 - **The scheduler lives in the server.** Its jobs run while the dashboard (or
   the container, or the service) is up; a missed run is made up once, within
   seven days, when it comes back.

package/SECURITY.md

@@ -24,8 +24,9 @@ reproduce, and what an attacker gains.
 **On this machine by default.** The dashboard (`serve.py`, port 8122) and the
 gateway (on the same port at `/v1`, or alone with `gateway.py` on 8123) listen
 on 127.0.0.1. Every request is checked for the server's own host name (against
-DNS rebinding), and every write must be a same-origin JSON request, so a web
-page open in your browser cannot drive either server. With no admin token
+DNS rebinding), and every write must be a same-origin request with a JSON
+body (a zip, for an import), a type no plain form can send, so a web page
+open in your browser cannot drive either server. With no admin token
 there is no login and the gateway asks for no key.
 
 **Anywhere else, only with a login.** To listen beyond this machine
@@ -60,15 +61,24 @@ open itself to anyone.
 - **Keys.** Endpoint keys come from `BENCHWARMERS_KEY_<NAME>` (the key name
   upper-cased, `-` as `_`) first, then from llm's key file (`keys.json` in
   its user folder, used whether or not the `llm` CLI is installed). They are
-  never written to the data folder, never shown back, and never put in a log.
-  Sign-ins through a provider's page store their token the same way.
+  never shown back and never put in a log. Sign-ins through a provider's page
+  store their token the same way.
+- **Exports.** Settings, Export and import, puts the endpoints' stored keys
+  (key, admin key, refresh token; never one from the environment) in an
+  export only when you tick them, in plain text in its `keys.json`: keep that
+  file like a password and delete it once imported. Importing one puts its
+  keys in the key store, and first backs up this install's own, with its files,
+  to the data folder's `backups/before-import-<time>.zip`. An import is
+  refused whole if any file in it would land outside its scope's folders.
 - **The network.** Your prompts go only to the endpoints you add; the probes,
-  the suite and onboarding spend calls there too. Otherwise: public catalog
-  reads (models.dev, FreeLLMAPI, OpenRouter's model list), each Open WebUI
-  connection's model list (through Open WebUI, with its key), a provider's
-  sign-in page when you choose one, Observable Plot from jsDelivr in the
-  dashboard's page, and Hugging Face and GitHub only when you run
-  `suite/rebench.py`. [PRIVACY.md](PRIVACY.md) lists them.
+  the suite, onboarding and the decision benchmark spend calls there too.
+  Otherwise: public catalog reads (models.dev, FreeLLMAPI, OpenRouter's model
+  list), each Open WebUI connection's model list (through Open WebUI, with its
+  key), a provider's sign-in page when you choose one, Observable Plot from
+  jsDelivr in the dashboard's page, the site a report's Share menu posts to
+  when you choose one (opened with the report's name and the project's link
+  only), and Hugging Face and GitHub only when you run `suite/rebench.py`.
+  [PRIVACY.md](PRIVACY.md) lists them.
 - **Files.** It writes inside its data folder (`BENCHWARMERS_HOME`, or the
   folder `.benchwarmers-home` names) and, in llm's user folder, the key file
   (when you paste a key or sign in) and the model list (when it registers a

package/USAGE.md

@@ -13,6 +13,7 @@ coding agent doing it with you); the gateway's whole contract is in
 - [Collections](#collections)
 - [Connecting an app](#connecting-an-app)
 - [Recording verdicts](#recording-verdicts)
+- [Benchmarks, and sharing a report](#benchmarks-and-sharing-a-report)
 - [Discover: finding models to add](#discover-finding-models-to-add)
 - [Quota management](#quota-management)
 - [The scheduler](#the-scheduler)
@@ -24,17 +25,24 @@ coding agent doing it with you); the gateway's whole contract is in
 
 | Group | Page | What it is for |
 | --- | --- | --- |
-| Observe | **At a glance** | Every model on the bench: whether it works right now, its role, and its connection. Click a name for its popup: role, alias, facts, options, retire. |
+| Observe | **At a glance** | Every model on the bench: whether it works right now, its uptime over 7 days, its role, and its connection. Click a name for its popup: role, alias, facts, options, retire. |
 | | **Action needed** | What will not fix itself, by the connection it comes from, with Retry and Retire on each model. |
-| | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, and speed. |
+| | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, each model against its own earlier scores, uptime, speed, and the decision models. Each report shares as an image or its table. |
 | | **Quota management** | A tile per provider: what is left, when it resets, and which of its models are resting. Then the week's resets, and what the bench sent. |
 | | **Logs** | Gateway calls (who sat and how it went, never the prompt), changes made from the dashboard, the scheduler, and onboarding runs. |
-| Gateway | **Endpoints** | Where models are served from. Add an endpoint (a provider's sign-in, or any OpenAI-compatible API key), change a key, and see what each one serves. Also your own app registrations and rate limits per connection. |
+| Gateway | **Endpoints** | Where models are served from. Add an endpoint (a provider's sign-in, or any OpenAI-compatible API key), change a key, and see what each one serves. A model server running on this machine, at its usual port or any other its process is listening on (LM Studio, Jan, llama.cpp, vLLM and the like), is offered under Found on this machine: Add opens the dialog with it filled in, to test and save. Also your own app registrations and rate limits per connection. |
 | | **Models** | Every model on the bench, by the connection it comes from. |
 | | **Collections** | The named panels your apps ask for, and the gateway's settings. |
+| | **Clients** | Each popular app's set-up (opencode, VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat apps), filled in from this install, with Copy and a gateway key made on the spot. |
 | | **Discover** | Models worth adding, from public catalogs, with the ways this install can reach each one. |
-| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, how the install is reached (shown only), and Restart. |
-| | **Info** | Version, Python, data folder and counts. |
+| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, export and import, how the install is reached (shown only), and Restart. |
+| | **Info** | The source repository, the version (linking its release), Python, data folder and counts. |
+
+The page shown follows the data: while it is in view it checks every 10
+seconds, and when something has changed it swaps in the new page in place,
+keeping open sections, sorting and filters. It waits while you are at work on
+it (a popup open, a form not saved or in use, text selected, a chart hovered),
+and **New results · Show**, top right, offers the change instead.
 
 **Action needed** lists what will not fix itself, such as a model about to be
 retired, grouped by the connection it comes from, since one cause usually
@@ -108,7 +116,9 @@ unavailable, so the daily probe keeps watching it.
 `code`, `light` (small and fast), `vision` (reads images well, weak at text:
 OCR and document models; seated only when a request carries images) or
 `image` (makes images; never seated). A model that reads images and does
-everything else well stays `general`.
+everything else well stays `general`. Where a decision model is onboarded,
+onboarding asks it to suggest a new chat model's role; the popup shows the
+suggestion, and nothing applies it.
 
 ## Collections
 
@@ -135,8 +145,18 @@ empty, it follows those settings. A member out of quota stays on the collection
 and an explorer sits in for it, named `stand-in for <member>`, until the quota
 is back. Max explorers 0 means only the members ever run, stand-ins included.
 
+Gateway settings also says whether a request that names no collection is given
+one by a decision model, and how sure that model must be (GATEWAY.md, "No
+default collection").
+
 ## Connecting an app
 
+The dashboard's **Clients** page has a ready set-up for each popular app,
+filled in from this install (its address, collections and MCP server): copy it
+into the file or screen it names. When keys are asked for, it makes one on the
+spot and fills it in where a key goes, never into a file that may sync. What
+follows is the same by hand.
+
 Any OpenAI-compatible client works. Point it at the dashboard's port and ask
 for the model `bench/<collection>`:
 
@@ -197,7 +217,8 @@ claude
 
 Each conversation is one offload per model that answered it; a verdict is
 optional, and the Gateway card lists route conversations apart from the
-panels. GATEWAY.md, "Route mode", has the other clients and the limits.
+panels. The Clients page has the other agents' set-ups; GATEWAY.md, "Route
+mode", has the limits.
 
 **A coding agent** connects over MCP instead: the Claude Code plugin
 (`/plugin marketplace add WendellResearch-AI/benchwarmers`, then
@@ -229,6 +250,33 @@ claims feed **Confidently wrong** on Benchmarks, and verdicts together decide
 who sits on the next panel. Agreement between models is not evidence: they
 are often wrong together.
 
+## Benchmarks, and sharing a report
+
+**Benchmarks** is one report after another: who to trust with a review, bug
+hunt, real work, confidently wrong, our code, vision, trend, uptime,
+performance and the decision models, then links to public leaderboards. Most
+have a chart, the table folded under it (with a filter, and **Show
+retired**), and **How to read this**;
+[BENCHMARKS.md](BENCHMARKS.md) has the same explanations in one place.
+**Trend** (30 or 60 days) and **Uptime** (7 or 30) have a window to choose.
+
+**Share**, top right of each report:
+
+- **Copy image** / **Download image**: the report's charts, title and the line
+  under it, with a band at the foot holding the logo, the project's link and
+  the date, as a PNG at twice the size shown. The table, the key and the
+  controls stay out; a window you chose shows as its words.
+- **Copy table as Markdown** / **Download table as CSV**: the table as shown
+  (its window, filter and Show retired), a model's connection in brackets.
+- **Share…**: your system's share sheet, with the image.
+- **Post on** X, Bluesky, LinkedIn or Reddit: the image goes on the clipboard
+  (or is saved, where there is no clipboard), then the site opens with the
+  report's name and the project's link, for you to paste the image into.
+
+What your browser cannot do is not offered: copying needs a secure page
+(`localhost` is one; a LAN address over plain http is not), and Share… needs
+a share sheet.
+
 ## Discover: finding models to add
 
 **Discover** leads with FreeLLMAPI's catalog of free tiers and fills in from
@@ -266,7 +314,8 @@ limit is that model's own:
 - **Pool**: OpenRouter's free requests a day, shared by its models, with a bar
   per key, against the limit OpenRouter stated in a rate-limit reply.
 - **Caps**: a day or month cap, counted from the bench's own calls: one you set
-  on Endpoints (Rate limits per connection), else one the provider states in
+  on Endpoints (Rate limits per connection), else a day cap the bench learned
+  for the connection (below), else one the provider states in
   its replies without saying what is left (a Cohere trial key's 1,000 calls a
   month), shown but not enforced until you set it. Calls made elsewhere with
   the same key are not in the count.
@@ -275,7 +324,14 @@ limit is that model's own:
 
 Nothing is taken from a provider's docs, which can change without notice; a
 limit a provider states in a rate-limit error (UnoRouter's "allows 1
-request(s) every 1 min" for a model) is learned and kept to. One exception,
+request(s) every 1 min" for a model) is learned and kept to. When the error
+names no number (Cloudflare Workers AI's, once the day's free Neurons are
+gone), the requests the bench had sent on that connection in that minute or
+day are taken as the connection's limit: noted the first time, kept to once a
+second refusal lands near the same count, unless you set your own for that
+connection on Endpoints, where each learned limit is shown with a Forget
+button. A learned rate limit not met again for 14 days is forgotten, so a
+raised limit is found. One exception,
 watched: NVIDIA states no limit per key, so its tile reads the file
 build.nvidia.com's own badge reads (10,000 requests a day, up to 40 a minute,
 the same for every account) daily and on Refresh, and counts the bench's
@@ -319,8 +375,9 @@ Under the tiles are:
 ## The scheduler
 
 The recurring jobs (the morning availability probe, the catalog refresh, the
-deprecation check and auto-retire, suite passes, the weekly speed and
-capability tests) run inside the dashboard. Settings, **Scheduler**:
+deprecation check and auto-retire, suite passes, the decision models' daily
+benchmark, the weekly speed and capability tests) run inside the dashboard.
+Settings, **Scheduler**:
 
 - **Off**: nothing runs; a fresh install spends no calls until you opt in.
 - **Log only**: writes "would run" to `runs/scheduler.log` at each due time,
@@ -337,3 +394,16 @@ call logs, probe history, task answers) lives in the data folder: the one
 `BENCHWARMERS_HOME` names, else the one in `.benchwarmers-home` at the top of
 the checkout. Back it up by copying it; move it by copying it and pointing one
 of those two at the new place. Info shows which folder is in use.
+
+**Export and import**, on Settings, does the same through the dashboard, for
+another machine or a copy to keep: one .zip of **Settings only**
+(`config/`: endpoints, collections, roles, model options, the schedule,
+gateway keys, the theme) or **Everything** (also every result, history, log
+and saved answer). The endpoints' API keys go in only if you tick them; the
+file then holds them, so keep it like a password and delete it once
+imported. On the other install, choose the file: it says what is in it
+(scope, release, date, files, how many it replaces, keys) before **Import**.
+Its files replace the ones of the same name, after this install's own in that
+scope are saved to `backups/before-import-<time>.zip`; files it does not have
+are left. Restart afterwards (Settings, Server and access), so every part
+reads them.

package/VERSION

@@ -1 +1 @@
-0.5.6
+0.6.1