@wendellresearch-ai/benchwarmers 0.5.4 → 0.5.6

package/AGENTS.md

@@ -95,7 +95,10 @@ cost with the person first. Three to five models make a useful first panel.
 ## 6. Make a collection, start the gateway
 
 On the Collections page: a name, the members a panel always includes, and the
-shape of work it is for (review, design, ...). Then:
+shape of work it is for (review, design, ...). A collection can set its own
+calls per request and explorers; one marked ad hoc runs only when a request
+names it, never as a default, so keep the person's expensive models there.
+Then:
 
     python gateway.py                      # http://127.0.0.1:8123/v1
 

package/GATEWAY.md

@@ -43,6 +43,22 @@ model family. Collections live in `config/collections.json`, the limits in
 `config/gateway.json`; both are edited in the dashboard's
 **Collections** tab and every change is logged in `config/changes.jsonl`.
 
+**A collection's own limits.** `max_calls` (1 to 10) and `max_explorers` (0 to 2)
+can be set on a collection, in its dialog or with `bench collection save <name>
+--max-calls 4 --max-explorers 0`. What it leaves empty follows `gateway.json`,
+so changing that moves every collection without its own, and a collection can
+go above it.
+
+**Stand-ins.** A member the bench knows would be refused (a rate limit or quota
+with no room before its seat would give up, or a connection on an alert for not
+answering) stays on the collection, is named in `bench.skipped`, and an explorer
+sits in for it, as the seat `stand-in for <member>`, until there is room. That is
+decided per request, so it ends by itself, and stand-ins come from the same pool
+as explorers: they fit the shape and are never ad hoc. A member refused for the
+first time is still asked, since the bench does not know yet. With `max_explorers`
+0 there are no stand-ins: only the members ever run, and if all are held the
+request gets a 429.
+
 **Naming:** `bench/<name>`, lower-case letters, digits and `-`. The name says
 what the collection is *for* (`reviewers`, `design`), not who is in it, because
 members change. The `bench/` prefix keeps collections apart from real model ids
@@ -60,6 +76,15 @@ tags in the message, so a caller that could not choose learns how to. A
 `bench/auto` that guessed for it was considered and rejected: the caller knows
 the task, and a guess would run the wrong models silently.
 
+**Ad hoc collections.** A collection marked ad hoc (the checkbox in its dialog,
+`bench collection save <name> --ad-hoc`, or `"ad_hoc": true`) runs only when a
+request names it. It is left off `GET /v1/models`, `/api/tags`, `/v1beta/models`
+and that refusal, so an agent choosing for itself never sees it: for the
+expensive ones a person decides to run. It is not a lock, since a key that may
+call it and knows its name can. To stop a client outright, limit its gateway key
+to other collections. Members still sit in other collections, so to keep a
+model out of explorer seats too, mark the model ad hoc as well.
+
 ## Prompt size
 
 The gateway estimates the prompt (system text included) at characters / 4 plus
@@ -517,7 +542,7 @@ key limited to its collection and a rate, or keep the key on the app's server.
 | Evidence | Each seat is recorded in `metrics/offloads.jsonl` as an unjudged offload when it answers; `offload.py verdict <call id>` or `POST /bench/verdict` replaces it. |
 | Collections tab | The collections, each drawn as its panel sits, and their form (one row per model, a chip per provider). Then the gateway's recent requests: every seat's status and verdict, how long the caller waited (Time), and how many verdicts are owed. Clicking a seat opens its answer and the verdict form. |
 | Catalog | `metrics/catalog.py`: context window, output limit, rate limits and capabilities per model, refreshed daily by the built-in scheduler; a free model whose price goes above zero raises a "went paid" alert. |
-| Discover tab | `metrics/discover.py`: FreeLLMAPI (leading) and models.dev merged into one list of models worth adding, free first, FreeLLMAPI's smartest first, marked by whether you can already reach the provider and whether the model is on the bench (on another provider counts as "also on bench"). |
+| Discover tab | `metrics/discover.py`: FreeLLMAPI (leading), models.dev and each Open WebUI connection's own model list merged into one list of models worth adding, free first, FreeLLMAPI's smartest first, marked by whether you can already reach the provider (matched to a connection by API host or name) and whether the model is on the bench (on another provider counts as "also on bench"). Fetched daily with the catalog; Refresh lists fetches them now. |
 | Merged replies | Opt-in: one merger answer as `content`, every seat still in `bench` (The reply). |
 | MCP server | `mcp_server.py`: the same three routes as MCP tools for agents, `list_collections`, `ask` and `verdict`. Two transports for one `answer()`: `POST /mcp` (or `/v1/mcp`) on either port, streamable HTTP with a JSON reply per request, its tools calling the server's own routes over loopback with the caller's credential and Host; and stdio (`benchwarmers mcp`), a caller like any other over HTTP (`BENCHWARMERS_URL`, `BENCHWARMERS_KEY`). The Claude Code plugin in `plugin/` connects to `/mcp` and adds a command, a skill and a `Stop` hook (`plugin/hooks/hooks.json`, an `mcp_tool` hook) that calls `pending_verdicts` with the session's `transcript_path`: the call ids in the transcript (Claude Code keeps no `Mcp-Session-Id`, so the transcript is the record of whose panels they were), of which the call log says which are answered seats with no verdict (`POST /bench/pending`), returned once as `additionalContext`. A server on another machine cannot read the transcript, so there the hook says nothing. A tool call answers as SSE with progress notifications, which reset Claude Code's 300-second idle limit. |
 

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 2 November 2026.*
+*Effective 7 November 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -18,6 +18,11 @@ Wendell Research.
   model list. With an NVIDIA connection on the bench, the public file of rate
   limits build.nvidia.com states, once a day and when you press Refresh on its
   quota tile.
+- **Model lists**, carrying no prompt: each Open WebUI connection's list of the
+  models its provider serves, asked through Open WebUI with the key it already
+  holds for that provider, once a day and when you press Refresh lists on
+  Discover. Only the models' ids, names, context sizes, prices and abilities
+  are kept.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page, in
   your browser, and the provider's token endpoint.
 - **Quota reads**, only when you press Refresh on Quota management, carrying

package/README.md

@@ -176,7 +176,11 @@ dependencies either way: the app is the Python standard library.
 - **Collections and seats.** A collection names the models a panel always
   includes and the shape of work it is for (review, design, enumeration...).
   Each call adds explorer seats chosen by how long a model has gone unused, so
-  the bench keeps learning about models a router would stop asking.
+  the bench keeps learning about models a router would stop asking. A member
+  that is out of quota or rate-limited keeps its place while an explorer
+  stands in for it until it is back. A collection can set its own calls per
+  request and explorers, and one marked ad hoc (an expensive one, say) runs
+  only when a request names it.
 - **Verdicts.** Each seat's answer carries a call id. Once you have checked an
   answer against the real work, record whether it was useful, and which of its
   claims were real or false. Verdicts, not vibes, decide the next panel; a
@@ -203,7 +207,10 @@ dependencies either way: the app is the Python standard library.
   are read from its site and watched for changes.
 - **Discover.** FreeLLMAPI's free catalog, with models.dev for the details
   and paid models it lacks, merged into one list of models worth adding,
-  smartest first, each with the routes this install has to it.
+  smartest first, each with the routes this install has to it and links to
+  its pages on both. Each Open WebUI connection's own model list adds what a
+  provider serves before either catalog lists it; **Refresh lists** fetches
+  them all now rather than at the daily fetch.
 
 ## Asking a panel
 
@@ -266,6 +273,7 @@ running server, so the same commands work against one on another machine.
     bench model retire <alias> --reason "..."      and back: bench model restore <alias>
     bench collection save reviewers --members a,b,c --description "adversarial review"
     bench collection save reviewers --add d --remove a
+    bench collection save heavy-hitters --ad-hoc --max-calls 4 --max-explorers 0
     bench endpoint add openrouter --base https://openrouter.ai/api/v1   (asks for the key, hidden)
     bench key create my-app --collections core --rpm 30
     bench settings scheduler run --off perf
@@ -296,8 +304,9 @@ no server at `BENCHWARMERS_URL`.
 
 A running Benchwarmers is an MCP server at `/mcp` on its own port
 (`http://127.0.0.1:8122/mcp`; `/v1/mcp` works too), over streamable HTTP,
-with three tools: `list_collections`, `ask` (a collection, a prompt,
-optionally `merged`, `with` and `system`) and `verdict`. Nothing to start per
+with four tools: `list_collections`, `ask` (a collection, a prompt,
+optionally `merged`, `with` and `system`), `verdict` and `pending_verdicts`
+(the seats still unjudged). Nothing to start per
 client: every agent session shares the one server, behind the same Host,
 Origin and gateway-key checks as `/v1`. The tool descriptions carry the
 rules: ask a collection, not a favourite model; verify before acting; name
@@ -443,6 +452,9 @@ reaches Wendell Research. What it sends, and where:
   its site's sitemap) and models.dev once a day, and OpenRouter's public
   model list. With an NVIDIA connection, the public file of rate limits
   build.nvidia.com states, once a day and on Refresh.
+- **Model lists**, with no prompt: each Open WebUI connection's list of the
+  models its provider serves, asked through Open WebUI with the key it
+  already holds, once a day and on Discover's Refresh lists.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page.
 - **Quota reads**, only when you press Refresh on Quota management, with no
   prompt: OpenRouter's key status, and xKiro's and Requesty's usage, each with

package/SECURITY.md

@@ -64,7 +64,8 @@ open itself to anyone.
   Sign-ins through a provider's page store their token the same way.
 - **The network.** Your prompts go only to the endpoints you add; the probes,
   the suite and onboarding spend calls there too. Otherwise: public catalog
-  reads (models.dev, FreeLLMAPI, OpenRouter's model list), a provider's
+  reads (models.dev, FreeLLMAPI, OpenRouter's model list), each Open WebUI
+  connection's model list (through Open WebUI, with its key), a provider's
   sign-in page when you choose one, Observable Plot from jsDelivr in the
   dashboard's page, and Hugging Face and GitHub only when you run
   `suite/rebench.py`. [PRIVACY.md](PRIVACY.md) lists them.

package/USAGE.md

@@ -123,13 +123,17 @@ per model with its context, vision and tools, and price. A model served by
 several providers is still one row, with a chip per provider: tick the chip
 of the route you want. Ticking two is allowed, and the count says so. The
 search matches a model, provider or served name, and the select beside it
-shows one provider's models; ticked ones always stay in view. A seat's ×
-takes that member off, as unticking it does. The tags grey out the models that
+shows one provider's models. The seats above the list always show who is
+ticked, filtered out or not, and a seat's × takes that member off, as unticking
+it does. The tags grey out the models that
 do not fit them.
 
 Each call also seats explorers, the models unused longest, up to the gateway's
 limits (Gateway settings, on the same page: how many calls a request may make,
-and how many of them explore).
+and how many of them explore). A collection can set its own in its popup; left
+empty, it follows those settings. A member out of quota stays on the collection
+and an explorer sits in for it, named `stand-in for <member>`, until the quota
+is back. Max explorers 0 means only the members ever run, stand-ins included.
 
 ## Connecting an app
 
@@ -228,12 +232,19 @@ are often wrong together.
 ## Discover: finding models to add
 
 **Discover** leads with FreeLLMAPI's catalog of free tiers and fills in from
-models.dev, which alone has prices, paid models and providers' base URLs. One
+models.dev, which alone has prices, paid models and providers' base URLs.
+The catalogs trail the providers by days, so it also reads each Open WebUI
+connection's own model list (through Open WebUI, with the keys it already
+has): a model a provider serves that neither catalog lists yet is a row of its
+own, its price unknown where the provider's list does not say. All of them
+are fetched daily; **Refresh lists** fetches them now. A provider is matched
+to your Open WebUI connection by its API host, or by its name in the
+connection's host (FreeLLMAPI's xKiro is `api.xkiro.com`). One
 list: free models first, then by reach, and within each reach FreeLLMAPI's
 smartest first (the **Rank** column), then what it does not rank, what is new
 this week first and badged. A row opens its popup: what
 the model takes and does, its limits and monthly budget, FreeLLMAPI's known
-quirks for it, links to its freellmapi.co page and models.dev, and every route
+quirks for it, links to its own pages on freellmapi.co and models.dev, and every route
 this install has to it, each with an **Add via this**. Adding through a hub (Open WebUI, LiteLLM, CLI Proxy
 API) puts it in that hub too; adding direct registers it on its own endpoint
 only. Either way the onboarding joins the same queue, with the same

package/VERSION

@@ -1 +1 @@
-0.5.4
+0.5.6

package/cli.py

@@ -328,7 +328,10 @@ def cmd_collection(args) -> None:
             "description": args.description if args.description is not None else old.get("description") or "",
             "tags": csv(args.tags) if args.tags is not None else old.get("tags") or [],
             "reply": args.reply or old.get("reply") or "all",
-            "merger": args.merger if args.merger is not None else old.get("merger") or ""}
+            "merger": args.merger if args.merger is not None else old.get("merger") or "",
+            "ad_hoc": args.ad_hoc if args.ad_hoc is not None else bool(old.get("ad_hoc"))}
+    for key, given in (("max_calls", args.max_calls), ("max_explorers", args.max_explorers)):
+        body[key] = old.get(key) if given is None else None if given == "default" else given
     reply = call("POST", "/api/collection", body)
     out(args, reply, lambda r: done(r, f"saved collection {name}: {', '.join(members)}"))
 
@@ -497,6 +500,8 @@ keeps only the answers. Then judge each seat you checked:
   bench collection save reviewers --members a,b,c --description "adversarial review" --tags review
   bench collection save reviewers --add d --remove a      (the rest stays as it was)
   bench collection save reviewers --reply merged --merger a
+  bench collection save heavy --ad-hoc      (runs only when called by name; --no-ad-hoc undoes it)
+  bench collection save heavy --max-calls 4 --max-explorers 0      (its own limits; "default" follows the gateway's)
   bench collection show reviewers
   bench collection rm reviewers --yes""",
     "endpoint": """examples:
@@ -629,6 +634,12 @@ def parser() -> argparse.ArgumentParser:
     c.add_argument("--tags", help="comma-separated")
     c.add_argument("--reply", choices=["all", "merged", "route"])
     c.add_argument("--merger", help="the alias that merges, for --reply merged")
+    c.add_argument("--ad-hoc", action=argparse.BooleanOptionalAction, default=None,
+                   help="only when called by name: left off what callers choose from")
+    own_limit = lambda v: v if v == "default" else int(v)  # noqa: E731
+    c.add_argument("--max-calls", type=own_limit, help="its own max calls per request, or default for the gateway's")
+    c.add_argument("--max-explorers", type=own_limit,
+                   help="its own max explorers, or default for the gateway's; 0 means only the members ever run")
 
     cmd("endpoints", cmd_endpoints, "where models are served from")
 

package/gateway.py

@@ -333,10 +333,21 @@ def retired_why(entry: dict) -> str:
 def refused_now(aliases, est_tokens: int, within: float) -> dict[str, str]:
     """alias -> why, for models whose known rate limits or quota leave no room
     for a call within `within` seconds (metrics/pacing.held): asked, they
-    would only bring back the bench's own refusal, so they are not seated."""
+    would only bring back the bench's own refusal, so they are not seated.
+    So are models on a connection the alerts say has stopped answering: the
+    daily probes still try it, and the first answer ends the alert."""
     import pacing
     out = {}
+    try:
+        import alerts
+        down = {m: a.get("message") or "its connection is not answering"
+                for a in alerts.active().values() if a.get("kind") == "connection" for m in a.get("aliases") or []}
+    except Exception:                                  # noqa: BLE001 -- a broken alert file is not a refusal
+        down = {}
     for a in aliases:
+        if a in down:
+            out[a] = down[a]
+            continue
         try:
             h = pacing.held(a, route.connection_of(a), est_tokens)
         except Exception:                              # noqa: BLE001 -- a broken limit file is not a refusal
@@ -560,7 +571,7 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
     if not prompt.strip():
         raise ValueError("no user message to put to the collection")
     collection = bench_collections.load()[name]
-    s = bench_collections.settings()
+    s = bench_collections.limits(collection)
     raw = collection.get("members") or []
     # The editor dedupes and caps; a hand-edited file may not have.
     members = list(dict.fromkeys(m for m in raw if isinstance(m, str) and m))
@@ -615,18 +626,22 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
     # like the others; it would only come back as the refusal.
     held = refused_now([*sitting, *nominated], needed, SEAT_TIMEOUT)
     skipped += [{"seat": "member" if a in sitting else "with", "alias": a, "why": why} for a, why in held.items()]
+    # AND AN EXPLORER SITS IN FOR A HELD MEMBER, until its limit has room: the
+    # member stays on the collection, and this is decided per request. Each
+    # one is named on its seat. Max explorers 0 means only the members ever run.
+    lent = [a for a in sitting if a in held] if s["max_explorers"] else []
     sitting = [a for a in sitting if a not in held]
     nominated = [a for a in nominated if a not in held]
     if pictures and not collection.get("shape"):
         shape, how = "vision", "the request carries images"
-    elif not n_explorers:
+    elif not n_explorers and not lent:
         shape, how = collection.get("shape") or "general", "no explorer seats to fill"
     elif collection.get("shape"):
         shape, how = collection["shape"], "set on the collection"
     else:
         shape, how = pick_shape(whole)
     seats = [(f"member{i}", "member", alias) for i, alias in enumerate(sitting, 1)]
-    if n_explorers:
+    if n_explorers or lent:
         # `with` in the request body is route.py's --with: an explorer seat.
         from run import _registered_names
         avoid = retiring() | set(too_small(needed, _registered_names()))   # explorers must fit too
@@ -635,9 +650,11 @@ def run_collection(name: str, messages: list[dict], nominated: list[str] | None
         if pictures:
             avoid |= {a for a in _registered_names() if cannot_see(a)}    # and see
         p = route.panel(shape, large=len(whole) > LARGE, nominated=nominated or None,
-                        members=sitting, explorers=n_explorers, avoid=avoid)
+                        members=sitting, explorers=n_explorers, avoid=avoid, stand_ins=len(lent))
         seats += [(key, label, p[key]["model"]) for key, label in
                   (("explorer", "explorer"), ("explorer2", "explorer 2")) if p.get(key)]
+        seats += [(f"standin{i}", f"stand-in for {member}", r["model"])
+                  for i, (member, r) in enumerate(zip(lent, p["stand_ins"]), 1)]
     if not seats and pictures:
         raise ValueError(f"no seat in bench/{name} can see images, so none is asked rather than answer"
                          " blind: " + ("; ".join(f"{k['alias']} {k['why']}" for k in skipped) or "no members")
@@ -716,16 +733,19 @@ def models() -> list[str]:
     return [bench_collections.PREFIX + n for n in bench_collections.load()]
 
 
-def listed(who: str) -> list[tuple[str, dict]]:
+def listed(who: str, ad_hoc: bool = False) -> list[tuple[str, dict]]:
     """(model name, what it is for) for every collection key `who` may call,
     each three times: as set, and in each other reply mode, so a client that
     offers only listed models (an agent's model picker above all, for
     `:route`) can still pick any. What GET /v1/models, /api/tags and
-    /v1beta/models list, each in its own shape."""
+    /v1beta/models list, each in its own shape. `ad_hoc`: with the collections
+    not on offer, for looking up one a client already named."""
     s, rows = bench_collections.settings(), []
     for n, c in bench_collections.load().items():
         if not guard.may_call(who, n):                 # a key limited to some collections sees those
             continue
+        if c.get("ad_hoc") and not ad_hoc:             # runs when named; nobody is offered it to pick
+            continue
         d = bench_collections.described(n, c, s)
         rows.append((bench_collections.PREFIX + n, d))
         rows += [(f"{bench_collections.PREFIX}{n}:{other}", {**d, "reply": other})
@@ -1098,7 +1118,7 @@ class Handler(BaseHTTPRequestHandler):
             return self._json(200, formats.gemini_models(listed(self.who)))
         if path.startswith("/v1beta/models/"):
             name = urllib.parse.unquote(path[len("/v1beta/models/"):])
-            row = dict(listed(self.who)).get(name)
+            row = dict(listed(self.who, ad_hoc=True)).get(name)
             return self._json(200, formats.gemini_model(name, row)) if row else \
                 self._error(404, f"{name!r} is not a collection here; GET /v1beta/models lists them", fmt=fmt)
         if guard.mcp_route(path):                      # no stream from the server: every reply is a POST's
@@ -1243,7 +1263,7 @@ class Handler(BaseHTTPRequestHandler):
     def _show(self, body: dict) -> None:
         """POST /api/show: Ollama's model card, for a collection the key may call."""
         name = formats.ollama_model(body.get("model") or body.get("name") or "")
-        rows = dict(listed(self.who))
+        rows = dict(listed(self.who, ad_hoc=True))
         row = rows.get(name) or rows.get(bench_collections.PREFIX + str(name))
         if not row:
             return self._error(404, f"model {name!r} not found; GET /api/tags lists the collections",

package/mcp_server.py

@@ -52,7 +52,8 @@ TOOLS = [
      "title": "List collections",
      "description": "The collections this Benchwarmers server offers: name, what each is for, tags, members,"
                     " explorer seats, reply mode, and what the members handle together (smallest context"
-                    " window, shared capabilities, tightest rate limits). Pick by task before calling ask.",
+                    " window, shared capabilities, tightest rate limits). Pick by task before calling ask."
+                    " A collection the user names may not be listed here: ask it anyway.",
      "inputSchema": {"type": "object", "properties": {}, "additionalProperties": False},
      "annotations": {"readOnlyHint": True, "openWorldHint": False}},
     {"name": "ask",

package/metrics/alerts.py

@@ -76,6 +76,31 @@ def clear(key: str) -> bool:
     return _update(lambda data: data.pop(key, None) is not None)
 
 
+def answered(alias: str, served: str) -> list[str]:
+    """A model answered: clear what an answer disproves. Every alert about it,
+    by its alias or a `model:<kind>:<served>` key (retiring, unrecognised
+    failure, never served, gone, entitlement, paywalled), except a price
+    (`model:paid:`), which an answer says nothing about. A held connection
+    loses it from its list, and goes once none is left. The cleared keys."""
+    def resolve(data: dict) -> list[str]:
+        gone = []
+        for key, a in list(data.items()):
+            if a.get("kind") == "connection":
+                held = a.get("aliases") or []
+                if alias not in held:
+                    continue
+                a["aliases"] = [x for x in held if x != alias]
+                if a["aliases"]:
+                    continue
+            elif key.startswith("model:paid:") or not (
+                    a.get("alias") == alias or (key.startswith("model:") and key.split(":", 2)[2:] == [served])):
+                continue
+            del data[key]
+            gone.append(key)
+        return gone
+    return _update(resolve)
+
+
 def sweep(prefix: str, still_wrong: set[str]) -> list[str]:
     """Retire every alert under `prefix` that the caller did not re-raise.
 

package/metrics/bench_collections.py

@@ -11,6 +11,12 @@ the same collection is called every day:
 
     explorers = min(max_explorers, max_calls - members)
 
+A collection may hold its own `max_calls` and `max_explorers`; what it leaves
+out follows gateway.json, so changing that moves every collection without its
+own. A member out of quota keeps its place and an explorer sits in for it
+until the quota is back (gateway.py), unless max_explorers is 0: then only the
+members ever run.
+
 `shape` (review, design, …) decides which models may fill those explorer
 seats; empty means a decision model labels each prompt, or `general` without
 one. `reply` is what the caller reads: `all` (every answer, the default) or
@@ -19,7 +25,9 @@ verdict record, gateway.py) or `route` (no panel: one member, picked by its
 record, gets the request as sent, tools and all, for agent clients;
 GATEWAY.md "Route mode"). A request may override it with a `:merged`,
 `:all` or `:route` suffix on the model name, or `"bench": {"reply": ...}`
-in the body.
+in the body. `ad_hoc` (false unless set) keeps a collection out of everything
+that offers collections to choose from: it runs when a request names it, as an
+ad hoc model does (suite/run.py), for the expensive ones a person decides on.
 Named `bench_collections` because `collections` would shadow the standard
 library for everything importing from metrics/.
 """
@@ -35,7 +43,8 @@ from paths import DATA  # noqa: E402  -- the install's data folder (metrics/path
 FILE = DATA / "config" / "collections.json"
 SETTINGS_FILE = DATA / "config" / "gateway.json"
 DEFAULTS = {"max_calls": 5, "max_explorers": 2}
-NAME = re.compile(r"^[a-z0-9][a-z0-9-]{1,31}$")
+BOUNDS = {"max_calls": (1, 10), "max_explorers": (0, 2)}   # route.py seats at most two explorers
+NAME =re.compile(r"^[a-z0-9][a-z0-9-]{1,31}$")
 PREFIX = "bench/"
 REPLIES = ("all", "merged", "route")
 
@@ -67,6 +76,19 @@ def settings() -> dict:
     return {**DEFAULTS, **{k: v for k, v in _read(SETTINGS_FILE).items() if k in DEFAULTS}}
 
 
+def limit_ok(key: str, value) -> bool:
+    return isinstance(value, int) and not isinstance(value, bool) and BOUNDS[key][0] <= value <= BOUNDS[key][1]
+
+
+def own_limits(c: dict) -> dict:
+    """The limits a collection sets for itself; a hand-edited one out of bounds counts as unset."""
+    return {k: c[k] for k in DEFAULTS if limit_ok(k, c.get(k))}
+
+
+def limits(c: dict, s: dict | None = None) -> dict:
+    return {**(s or settings()), **own_limits(c)}
+
+
 def explorers_for(members: int, s: dict | None = None) -> int:
     s = s or settings()
     return max(0, min(s["max_explorers"], s["max_calls"] - members))
@@ -81,8 +103,8 @@ def described(name: str, c: dict, s: dict | None = None) -> dict:
     members = c.get("members", [])
     out = {"description": c.get("description") or c.get("note", ""),
            "tags": c.get("tags", []), "members": members, "shape": c.get("shape", ""),
-           "explorers": explorers_for(len(members), s),
-           "reply": c.get("reply") or "all", "merger": c.get("merger", "")}
+           "explorers": explorers_for(len(members), limits(c, s)), **own_limits(c),
+           "reply": c.get("reply") or "all", "merger": c.get("merger", ""), "ad_hoc": bool(c.get("ad_hoc"))}
     # What the members can handle together (metrics/catalog.py): the smallest
     # context window is what the gateway refuses above.
     try:
@@ -122,11 +144,11 @@ def reply_errors(reply: str, merger: str, known) -> list[str]:
 
 def choose_message(model: str | None) -> str:
     """The refusal for a request that names no collection, or the wrong one:
-    every collection with its description and tags, so the caller can choose."""
+    every collection on offer with its description and tags, so the caller can choose."""
     asked = f"{model!r} is not a collection here." if model else "No collection was named."
     rows = [f"- {PREFIX}{n}: {c.get('description') or 'no description'}"
             + (f" [{', '.join(c.get('tags', []))}]" if c.get("tags") else "")
-            for n, c in sorted(load().items())]
+            for n, c in sorted(load().items()) if not c.get("ad_hoc")]
     return (f"{asked} Set \"model\" to the collection that fits the task"
             " (GET /v1/models lists them with description and tags):\n"
             + ("\n".join(rows) or "- none yet: create one in the dashboard's Collections dialog"))
@@ -136,6 +158,9 @@ if __name__ == "__main__":
     # DEFAULTS, not settings(): the install's gateway.json must not decide the check.
     assert [explorers_for(n, DEFAULTS) for n in range(1, 7)] == [2, 2, 2, 1, 0, 0]
     assert explorers_for(3, {"max_calls": 4, "max_explorers": 2}) == 1
+    assert limits({"max_calls": 4, "max_explorers": 0}, DEFAULTS) == {"max_calls": 4, "max_explorers": 0}
+    assert limits({"max_calls": 99, "max_explorers": True}, DEFAULTS) == DEFAULTS and limits({}, DEFAULTS) == DEFAULTS
+    assert explorers_for(4, limits({"max_calls": 4}, DEFAULTS)) == 0
     assert resolve("bench/") is None and resolve("bench/no-such") is None and resolve("") is None
     assert split_reply("bench/core:merged") == ("bench/core", "merged") and split_reply("core:all") == ("core", "all")
     assert split_reply("bench/core:route") == ("bench/core", "route")

package/metrics/catalog.py

@@ -173,14 +173,11 @@ def _watch_price(alias: str, served: str, conn: str, old: dict | None, new: dict
 
 def refresh() -> dict:
     """Rebuild config/model-catalog.json for every registered chat model."""
+    import discover
     import endpoints
     import sources
     from run import _registered_names
-    for name in sources.FEEDS:
-        try:
-            sources.fetch(name)                      # at most once a day; cached copy otherwise
-        except Exception:                            # noqa: BLE001
-            pass
+    discover.refresh()                               # the catalogs and providers' lists, at most once a day
     before = catalog()
     registered = _registered_names()                   # alias -> served name
     try: