@wendellresearch-ai/benchwarmers 0.5.0 → 0.5.2

package/PRIVACY.md

@@ -1,6 +1,6 @@
 # Privacy policy
 
-*Effective 29 October 2026.*
+*Effective 2 November 2026.*
 
 Benchwarmers runs on your own machine or server. It has no account, no
 telemetry and no server of its own, and nothing you do with it reaches
@@ -15,12 +15,19 @@ Wendell Research.
   applies to what it receives; some free tiers keep prompts or train on them.
 - **Catalog reads**, carrying nothing of yours: FreeLLMAPI (its feed and its
   site's sitemap) and models.dev (at most once a day) and OpenRouter's public
-  model list.
+  model list. With an NVIDIA connection on the bench, the public file of rate
+  limits build.nvidia.com states, once a day and when you press Refresh on its
+  quota tile.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page, in
   your browser, and the provider's token endpoint.
 - **Quota reads**, only when you press Refresh on Quota management, carrying
   no prompt:
   - OpenRouter's key status, asked with your key.
+  - xKiro's usage, asked with your key for it (from its Open WebUI connection
+    when no endpoint has one). Its reply names the account's owner; only the
+    quota figures are kept.
+  - Requesty's usage for the key and its public model list, asked the same
+    way; only the count of free-model requests is kept.
   - For CLI Proxy API's sign-ins, through the proxy, the quota endpoint of the
     provider each sign-in belongs to (Google's, ChatGPT's).
 - **Charts**: the dashboard's page loads Observable Plot from jsDelivr, at a

package/README.md

@@ -187,7 +187,8 @@ dependencies either way: the app is the Python standard library.
 - **Health.** Every morning each model gets a one-word prompt; once a week, a
   short and a long one for speed. A model that stops answering, or that its
   endpoint no longer serves, is retired automatically after one confirming
-  call, with the evidence kept, and can be restored in one click. The
+  call, with the evidence kept, and can be restored in one click. A model
+  that is only out of quota until a stated reset is not counted against. The
   built-in scheduler runs all of this once Settings switches it on; a fresh
   install spends no calls until then.
 - **Rate limits and usage.** Limits come from providers' rate-limit headers
@@ -196,8 +197,10 @@ dependencies either way: the app is the Python standard library.
   connection, since models behind one key share them, and enforced: a call
   waits when the wait fits, or is refused before it goes out. Every call lands
   in a monthly usage ledger with its tokens. Quota management has a tile per
-  provider showing what is left, when it resets and who is resting. OpenRouter
-  and CLI Proxy API's sign-ins are read live, at no cost to quota.
+  provider showing what is left, when it resets and who is resting. OpenRouter,
+  xKiro, Requesty and CLI Proxy API's sign-ins are read live, at no cost to
+  quota. Nothing is taken from a provider's docs; NVIDIA's own stated figures
+  are read from its site and watched for changes.
 - **Discover.** FreeLLMAPI's free catalog, with models.dev for the details
   and paid models it lacks, merged into one list of models worth adding,
   smartest first, each with the routes this install has to it.
@@ -438,11 +441,13 @@ reaches Wendell Research. What it sends, and where:
   are content to send that work to.
 - **Catalog reads**, no key and nothing of yours: FreeLLMAPI (its feed and
   its site's sitemap) and models.dev once a day, and OpenRouter's public
-  model list.
+  model list. With an NVIDIA connection, the public file of rate limits
+  build.nvidia.com states, once a day and on Refresh.
 - **Sign-ins**, only when you choose one: the provider's own sign-in page.
 - **Quota reads**, only when you press Refresh on Quota management, with no
-  prompt: OpenRouter's key status, with your key. CLI Proxy API asks each
-  sign-in's own provider for its quota.
+  prompt: OpenRouter's key status, and xKiro's and Requesty's usage, each with
+  your key for it (from its Open WebUI connection when no endpoint has one). CLI Proxy
+  API asks each sign-in's own provider for its quota.
 - **Charts**: the dashboard's browser page loads Observable Plot from jsDelivr
   at a pinned version.
 - **Seeding real-bug tasks**, only when you run `suite/rebench.py`: the

package/USAGE.md

@@ -37,8 +37,13 @@ coding agent doing it with you); the gateway's whole contract is in
 
 "Action needed", above the pages, lists what will not fix itself, such as a
 model about to be retired, naming each model and where it comes from, with
-Retry, Open and Retire now beside it. Hide folds it away until something new
-turns up; an alert whose models have all been retired goes at once.
+Retry, Open and Retire now beside it. It starts folded to its heading and
+count; show opens it in that browser for 12 hours. An alert whose models have
+all been retired goes at once. A model that is out of quota is not counting
+down: days the bench held its calls back until a stated reset, or an account's
+shared free quota was spent, pause its clock, and while it is still out of
+quota no countdown shows. One refused again after the short wait it named is
+failing, and keeps counting.
 
 **Restart** (Settings, Server and access), or `bench restart`, starts the
 dashboard again on the code now on disk, after an update. It waits for
@@ -109,8 +114,11 @@ seats dashed, and the merger after an arrow.
 In the popup, the panel's seats are on top, and the member list has one row
 per model with its context, vision and tools, and price. A model served by
 several providers is still one row, with a chip per provider: tick the chip
-of the route you want. Ticking two is allowed, and the count says so. The tags
-grey out the models that do not fit them, as before.
+of the route you want. Ticking two is allowed, and the count says so. The
+search matches a model, provider or served name, and the select beside it
+shows one provider's models; ticked ones always stay in view. A seat's ×
+takes that member off, as unticking it does. The tags grey out the models that
+do not fit them.
 
 Each call also seats explorers, the models unused longest, up to the gateway's
 limits (Gateway settings, on the same page: how many calls a request may make,
@@ -231,25 +239,54 @@ app**, switches a provider either way or adds one.
 
 ## Quota management
 
-One tile per provider a bench model arrives through, shaped by how that
-provider limits:
+One tile per provider a bench model arrives through. Each draws the
+provider's limits at the level they apply, and names a model only where a
+limit is that model's own:
 - **Windows**: CLI Proxy API's sign-ins have a 5-hour and a weekly limit per
-  model group. Other providers have the windows their rate-limit headers state.
+  model group, a Codex sign-in one for the account. Other providers have the
+  windows their replies' headers state; a window past its reset is shown full.
 - **Pool**: OpenRouter's free requests a day, shared by its models, with a bar
-  per key.
-- **Each model's last day**: shown where the provider states nothing.
-
-Each tile shows when its limits reset and which of its models are resting
-after a rate limit. **Models** lists the provider's bench models, whether each
-is free to call now, and its last 24 hours of calls.
-
-**Refresh never spends quota.** On OpenRouter it asks the key what is left. On
-CLI Proxy API's sign-ins it asks the proxy for each one's quota, as its own
-console does; this needs CLI Proxy API on the Endpoints page as a hub, with
-its management key. Every other tile re-reads what the provider last said in
-its replies to the bench. **Check now** sends one 16-token request to read the
-reply's limit headers. It is offered only where the provider has sent them
-before.
+  per key, against the limit OpenRouter stated in a rate-limit reply.
+- **Caps**: a day or month cap, counted from the bench's own calls: one you set
+  on Endpoints (Rate limits per connection), else one the provider states in
+  its replies without saying what is left (a Cohere trial key's 1,000 calls a
+  month), shown but not enforced until you set it. Calls made elsewhere with
+  the same key are not in the count.
+- **Nothing stated**: how the provider's calls went in the last 24 hours, all
+  its models together (green answered, red failed).
+
+Nothing is taken from a provider's docs, which can change without notice; a
+limit a provider states in a rate-limit error (UnoRouter's "allows 1
+request(s) every 1 min" for a model) is learned and kept to. One exception,
+watched: NVIDIA states no limit per key, so its tile reads the file
+build.nvidia.com's own badge reads (10,000 requests a day, up to 40 a minute,
+the same for every account) daily and on Refresh, and counts the bench's
+requests against it. If NVIDIA changes those figures, Action needed says so
+(Got it dismisses it); if the file moves or changes shape, Action needed says
+that too, and the tile keeps the figures it last read. Each meter leads
+with the percentage left, then what is left of what in the provider's own unit
+("94% left · 189 of 200 requests · day"), with its reset on the right. Under the name, one block says where the figures come
+from (**From**: the provider's quota endpoint or its replies' headers, exact;
+or a stated limit counted by the bench), your per-minute caps, and anything a
+live read reported. **Models** names the connection and how it is reached
+(each endpoint or Open WebUI connection, with its address), then lists the
+provider's bench models (each opens its popup), whether each is free to call
+now, and its last 24 hours of calls.
+
+**Refresh** picks its own way, and hovering it says which:
+- **A free read** on OpenRouter (it asks the key what is left), on xKiro (its
+  usage endpoint: free-model tokens left today), on Requesty (its usage
+  endpoint: the key's free-model requests today, by Requesty's count; its limit
+  only once a rate-limit reply has stated it) and on CLI Proxy API's sign-ins
+  (it asks the proxy for each one's quota, as its own console does; this needs
+  CLI Proxy API on the Endpoints page as a hub, with its management key).
+  xKiro's and Requesty's keys come from their endpoint, else their Open WebUI
+  connection.
+- **One 16-token request** where a provider states its limits only in its
+  replies' headers, to read them (Groq, Routeway, a Cohere trial key). Every
+  ordinary call reads them too, through Open WebUI as well, since the bench
+  streams.
+- **The bench's own records** again everywhere else.
 
 Tiles sort with the ones that are out first, then the soonest reset. Drag a
 tile, or move it with the arrow keys, and the order is kept; **Sort** brings

package/VERSION

@@ -1 +1 @@
-0.5.0
+0.5.2

package/metrics/catalog.py

@@ -283,6 +283,11 @@ def refresh() -> dict:
             "refreshed": dt.datetime.now().astimezone().isoformat(timespec="seconds"),
             "models": models}
     CATALOG.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
+    try:                                               # NVIDIA's stated limits, watched daily (an alert on a change)
+        import quotaview
+        quotaview.watch_nvidia()
+    except Exception:                                  # noqa: BLE001
+        pass
     return data
 
 

package/metrics/learn.py

@@ -60,9 +60,17 @@ RETRY = [
     r"\"retry_?delay\"\s*:\s*\"(\d+(?:\.\d+)?)s",
     r"\"retry_after\"\s*:\s*(\d+(?:\.\d+)?)",
     r"(?:try|retry) again in\s*(\d+(?:\.\d+)?)\s*(s|sec|seconds?|m|min|minutes?|h|hours?)\b",
-    r"please retry in\s*(\d+(?:\.\d+)?)\s*(s|m|h)",
+    r"\bretry in\s*(\d+(?:\.\d+)?)\s*(s|m|h)",         # "please retry in 2s", UnoRouter's "retry in 60s"
 ]
 RATE_LIMITED = re.compile(r"rate.?limit|too many requests|\b429\b|quota exceeded|resource_exhausted")
+# The number and its window in one phrase: UnoRouter's free tier, "allows 5
+# request(s) every 1 min per account on <model>:free". "every" only: the
+# bench's own refusal says "allows 3 requests per minute" (pacing.py), and a
+# limit the bench made up must never be learned back.
+ALLOWS = re.compile(r"allows?\s+([\d,]+)\s+requests?(?:\(s\))?\s+every\s+(?:1\s+)?(min(?:ute)?|day)\b")
+# new-api gateways (Agnes): "maximum 20 requests in 1 minutes" (its
+# middleware/model-rate-limit.go); a window of 1 minute or 1,440 only.
+MAXIMUM = re.compile(r"maximum\s+([\d,]+)\s+requests?\s+in\s+(1|1440)\s+minutes?\b")
 
 
 def _int(s: str) -> int:
@@ -83,6 +91,10 @@ def parse(text: str) -> dict:
     named = [axis for axis, pat in AXES.items() if re.search(pat, low)]
     if len(named) == 1 and (m := re.search(r"\blimit\b[\s:=]*([\d,]+)", low)):
         out["limits"] = {named[0]: _int(m.group(1))}
+    elif m := ALLOWS.search(low):
+        out["limits"] = {"rpd" if m.group(2) == "day" else "rpm": _int(m.group(1))}
+    elif m := MAXIMUM.search(low):
+        out["limits"] = {"rpd" if m.group(2) == "1440" else "rpm": _int(m.group(1))}
     for pat in RETRY:
         if m := re.search(pat, low):
             n = float(m.group(1))
@@ -205,6 +217,18 @@ def _selftest() -> None:
     assert p["retry_after_s"] == 180 and "limits" not in p, p      # no number under a named axis
     # Two axes named: the number is not filed under either.
     assert "limits" not in parse("limit 100 (tokens per minute) and requests per minute")
+    # UnoRouter's free tier says the number and its window together.
+    p = parse("Error code: 429 - {'error': {'message': 'Too many requests. The free tier allows 5 request(s) every"
+              " 1 min per account on llama-3.2-11b-vision:free - nothing is used up, retry in 60s.")
+    assert p["limits"] == {"rpm": 5} and p["retry_after_s"] == 60 and p["rate_limited"], p
+    assert parse("The free tier allows 200 requests every day")["limits"] == {"rpd": 200}
+    assert "limits" not in parse("allows 3 requests every 5 min")             # a window no axis holds
+    assert "limits" not in parse("rate limit (not sent; paced by the bench): x allows 3 requests per minute")
+    # A new-api gateway's 429 (Agnes).
+    p = parse("Error code: 429 - {'error': {'message': 'You have reached the request limit: maximum 20 requests"
+              " in 1 minutes', 'type': 'AgnesAI_error'}}")
+    assert p["limits"] == {"rpm": 20} and p["rate_limited"], p
+    assert "limits" not in parse("maximum 100 requests in 5 minutes")         # a window no axis holds
     assert parse("model not found") == {"rate_limited": False}
     assert parse("context length of 8k exceeded").get("context_window") is None   # "8" < 512
     print("ok")

package/metrics/quota.py

@@ -21,7 +21,11 @@ its reset, so a connection is never held for want of traffic to observe one.
 ONE HEADER NAME, DIFFERENT WINDOWS
 ----------------------------------
 `x-ratelimit-limit-requests` is per minute at OpenAI and per day at Groq;
-Cerebras names its windows (`-requests-day`, `-tokens-minute`); OpenRouter
+Cerebras names its windows (`-requests-day`, `-tokens-minute`), Routeway only
+its windows (`X-RateLimit-Remaining-Day`, requests, resets in seconds), and a
+Cohere trial key its minute (`x-trial-endpoint-call-remaining`, no reset) and
+its month's limit with nothing left said (`x-endpoint-monthly-call-limit`:
+kept, never enforced, and quotaview counts the month from the ledger); OpenRouter
 sends bare `X-RateLimit-*` only on a 429, inside the body's metadata as well,
 and its message says which window ("free-models-per-day"); Anthropic's
 `anthropic-ratelimit-*` are per minute; Gemini sends no headers but its 429
@@ -76,16 +80,25 @@ PROFILES = {
 WINDOWS = {"minute": 60, "hour": 3600, "day": 86400}
 OPENAI_STYLE = re.compile(r"^x-ratelimit-(limit|remaining|reset)-(requests|tokens)(?:-(minute|hour|day))?$")
 BARE = re.compile(r"^x-ratelimit-(limit|remaining|reset)$")
+WINDOWED = re.compile(r"^x-ratelimit-(limit|remaining|reset)-(minute|hour|day)$")      # Routeway: requests
+COHERE = re.compile(r"^x-trial-endpoint-call-(limit|remaining)$")                      # a trial key's minute
+COHERE_MONTH = "x-endpoint-monthly-call-limit"          # a limit with no remaining: quotaview counts it
 ANTHROPIC = re.compile(r"^anthropic-ratelimit-(requests|tokens|input-tokens|output-tokens)-(limit|remaining|reset)$")
 DURATION = re.compile(r"^(?:(\d+(?:\.\d+)?)h)?(?:(\d+(?:\.\d+)?)m(?!s))?(?:(\d+(?:\.\d+)?)s)?(?:(\d+(?:\.\d+)?)ms)?$")
 
 
-def profile_for(base: str) -> tuple[str, dict]:
-    """The profile for an endpoint's base URL; ("generic", {}) for any other host."""
+def profile_for(base: str, connection: str = "") -> tuple[str, dict]:
+    """The profile for an endpoint's base URL, else for the connection (a hub
+    such as Open WebUI relays the provider's headers: `groq`, `api.groq.com`);
+    ("generic", {}) for any other."""
     host = (urllib.parse.urlsplit(base or "").hostname or "").lower()
+    conn = (connection or "").lower()
     for known, prof in PROFILES.items():
         if host == known or host.endswith("." + known):
             return prof
+    for known, prof in PROFILES.items():
+        if conn in (known, prof[0]) or conn.endswith("." + known):
+            return prof
     return "generic", {}
 
 
@@ -148,8 +161,15 @@ def from_headers(headers, profile: tuple[str, dict], now: float, status: int = 2
             what, metric, window = m.group(1), m.group(2), m.group(3) or defaults.get(m.group(2), "unknown")
         elif m := BARE.match(key):
             what, metric, window = m.group(1), "requests", _window_hint(hint)
+        elif m := WINDOWED.match(key):
+            what, metric, window = m.group(1), "requests", m.group(2)
         elif m := ANTHROPIC.match(key):
             metric, what, window = m.group(1), m.group(2), "minute"
+        elif m := COHERE.match(key):                   # no reset sent: the minute is the wait
+            what, metric, window = m.group(1), "requests", "minute"
+            fields.setdefault((metric, window), {}).setdefault("reset", "60")
+        elif key == COHERE_MONTH:                      # no remaining, no reset: never enforced (live())
+            what, metric, window = "limit", "requests", "month"
         else:
             continue
         fields.setdefault((metric, window), {})[what] = value
@@ -247,7 +267,7 @@ def store(connection: str, alias: str, records: list[dict], now: float | None =
 def observe(alias: str, connection: str, base: str, status: int, headers, body=None) -> list[dict]:
     """Record what one response states; returns the records. Called by the
     built-in client on every reply and every HTTP error."""
-    now, profile = time.time(), profile_for(base)
+    now, profile = time.time(), profile_for(base, connection)
     hint = " ".join(str(e.get("message") or "") for e in _error_objects(body)) if status == 429 else ""
     records = from_headers(headers, profile, now, status, hint)
     if status == 429 and body is not None:
@@ -318,12 +338,29 @@ def _selftest() -> None:
                       "X-Ratelimit-Reset-Requests": "2m59.56s"}, profile_for("https://api.groq.com/openai/v1"), now)
     g = one(r, "requests", "day")
     assert g["remaining"] == 0 and g["reset_at"] == _iso(now + 179.56) and g["source"] == "headers (groq)", g
+    # Through a hub, the connection names the provider.
+    assert profile_for("http://owui.lan/api", "groq")[0] == "groq" == profile_for("http://owui.lan/api", "api.groq.com")[0]
+    assert profile_for("http://owui.lan/api", "api.cohere.ai") == ("generic", {})
     # Cerebras names its windows.
     r = from_headers({"x-ratelimit-limit-requests-day": "14400", "x-ratelimit-remaining-requests-day": "14399",
                       "x-ratelimit-reset-requests-day": "33011.38", "x-ratelimit-limit-tokens-minute": "60000",
                       "x-ratelimit-remaining-tokens-minute": "59000", "x-ratelimit-reset-tokens-minute": "11.4"},
                      profile_for("https://api.cerebras.ai/v1"), now)
     assert one(r, "requests", "day")["limit"] == 14400 and one(r, "tokens", "minute")["remaining"] == 59000
+    # Routeway names only the window; requests, resets in seconds.
+    r = from_headers({"X-RateLimit-Limit-Day": "200", "X-RateLimit-Remaining-Day": "187", "X-RateLimit-Reset-Day": "3600",
+                      "X-RateLimit-Limit-Minute": "5", "X-RateLimit-Remaining-Minute": "4", "X-RateLimit-Reset-Minute": "42"},
+                     profile_for("http://owui.lan/api"), now)
+    d = one(r, "requests", "day")
+    assert d["limit"] == 200 and d["remaining"] == 187 and d["reset_at"] == _iso(now + 3600), d
+    assert one(r, "requests", "minute")["remaining"] == 4
+    # A Cohere trial key: its minute, with no reset (the minute is the wait); the month has no remaining.
+    r = from_headers({"x-trial-endpoint-call-limit": "20", "x-trial-endpoint-call-remaining": "19",
+                      "x-endpoint-monthly-call-limit": "1000"}, profile_for("http://owui.lan/api"), now)
+    c = one(r, "requests", "minute")
+    assert c["limit"] == 20 and c["remaining"] == 19 and c["reset_at"] == _iso(now + 60) and len(r) == 2, r
+    mo = one(r, "requests", "month")
+    assert mo["limit"] == 1000 and mo["remaining"] is None and mo["reset_at"] is None, mo
     # Anthropic: RFC 3339 resets, per minute.
     r = from_headers({"anthropic-ratelimit-requests-limit": "50", "anthropic-ratelimit-requests-remaining": "0",
                       "anthropic-ratelimit-requests-reset": "2025-10-24T00:00:45Z"},