@wendellresearch-ai/benchwarmers 0.6.3 → 0.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +3 -1
- package/BENCHMARKS.md +1 -1
- package/README.md +2 -0
- package/USAGE.md +4 -6
- package/VERSION +1 -1
- package/package.json +1 -1
- package/scripts/check.py +1 -0
- package/scripts/clients_doc.py +98 -0
- package/serve.py +74 -223
- package/ui/bench.css +37 -5
- package/ui/bench.js +187 -40
package/AGENTS.md
CHANGED
|
@@ -124,7 +124,9 @@ in `bench.skipped`. One the catalog knows nothing about sits once its model
|
|
|
124
124
|
popup ticks Vision / OCR.
|
|
125
125
|
After the person checks an answer against the real work, record a verdict
|
|
126
126
|
(dashboard, or `POST /bench/verdict`); verdicts choose the next panels. See
|
|
127
|
-
GATEWAY.md for the whole contract.
|
|
127
|
+
GATEWAY.md for the whole contract. To set an app up on the gateway (opencode,
|
|
128
|
+
VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat
|
|
129
|
+
apps), CLIENTS.md has each one's configuration, file by file.
|
|
128
130
|
|
|
129
131
|
## Beyond this machine
|
|
130
132
|
|
package/BENCHMARKS.md
CHANGED
|
@@ -70,7 +70,7 @@ earlier scores on the same tasks, over the last 30 days (or 60).
|
|
|
70
70
|
|
|
71
71
|
Whether the provider was there when the bench called it, from the calls the
|
|
72
72
|
bench makes anyway: one cell a day, by provider and then by model, over the
|
|
73
|
-
last 7 days (or 30).
|
|
73
|
+
last 7 days (or 30).
|
|
74
74
|
|
|
75
75
|
- **Down**: the provider was busy or overloaded (a 5xx, high demand, a shared
|
|
76
76
|
pool's 429), unreachable (a broken stream, no reply), timed out (the
|
package/README.md
CHANGED
|
@@ -419,6 +419,8 @@ collections and MCP server, with a Copy button: opencode, VS Code chat, Codex
|
|
|
419
419
|
CLI, Zed, Continue, Cline and Kilo Code, Claude Code, goose, and chat apps
|
|
420
420
|
(LobeHub, AnythingLLM, Cherry Studio, Chatbox, LibreChat). Where keys are
|
|
421
421
|
asked for, it makes one and fills it in, never into a file that may sync.
|
|
422
|
+
The same set-ups are in [CLIENTS.md](CLIENTS.md), with a fresh install's
|
|
423
|
+
address and collections, for reading here or handing to an agent.
|
|
422
424
|
|
|
423
425
|
<!-- The MCP Registry's proof that the PyPI package is this server (server.json):
|
|
424
426
|
mcp-name: io.github.WendellResearch-AI/benchwarmers -->
|
package/USAGE.md
CHANGED
|
@@ -25,15 +25,14 @@ coding agent doing it with you); the gateway's whole contract is in
|
|
|
25
25
|
|
|
26
26
|
| Group | Page | What it is for |
|
|
27
27
|
| --- | --- | --- |
|
|
28
|
-
| Observe | **
|
|
28
|
+
| Observe | **Benchmarks** | The home page. How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, each model against its own earlier scores, uptime, speed, and the decision models. Each report shares as an image or its table. |
|
|
29
29
|
| | **Action needed** | What will not fix itself, by the connection it comes from, with Retry and Retire on each model. |
|
|
30
|
-
| | **Benchmarks** | How models do on the suite and on real work: bugs found against false alarms, who is confidently wrong, each model against its own earlier scores, uptime, speed, and the decision models. Each report shares as an image or its table. |
|
|
31
30
|
| | **Quota management** | A tile per provider: what is left, when it resets, and which of its models are resting. Then the week's resets, and what the bench sent. |
|
|
32
31
|
| | **Logs** | Gateway calls (who sat and how it went, never the prompt), changes made from the dashboard, the scheduler, and onboarding runs. |
|
|
33
32
|
| Gateway | **Endpoints** | Where models are served from. Add an endpoint (a provider's sign-in, or any OpenAI-compatible API key), change a key, and see what each one serves. A model server running on this machine, at its usual port or any other its process is listening on (LM Studio, Jan, llama.cpp, vLLM and the like), is offered under Found on this machine: Add opens the dialog with it filled in, to test and save. Also your own app registrations and rate limits per connection. |
|
|
34
|
-
| | **Models** | Every model on the bench, by the connection it comes from. |
|
|
33
|
+
| | **Models** | Every model on the bench, by the connection it comes from: what it holds, its reply time and its limits. One search above every table, each table folds, and Collapse all / Expand all. Click a name for its popup: role, alias, facts, options, retire. |
|
|
35
34
|
| | **Collections** | The named panels your apps ask for, and the gateway's settings. |
|
|
36
|
-
| | **Clients** | Each popular app's set-up (opencode, VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat apps), filled in from this install, with Copy and a gateway key made on the spot. |
|
|
35
|
+
| | **Clients** | Each popular app's set-up (opencode, VS Code, Codex, Zed, Continue, Cline, Kilo Code, Claude Code, goose, chat apps), filled in from this install, with Copy and a gateway key made on the spot. [CLIENTS.md](CLIENTS.md) has the same, for a fresh install. |
|
|
37
36
|
| | **Discover** | Models worth adding, from public catalogs, with the ways this install can reach each one. |
|
|
38
37
|
| Control | **Settings** | Theme, gateway keys, the scheduler, the model client, providers whose free models work only in their own app, export and import, how the install is reached (shown only), and Restart. |
|
|
39
38
|
| | **Info** | The source repository, the version (linking its release), Python, data folder and counts. |
|
|
@@ -52,8 +51,7 @@ clears the alert) and Retire. A group of two or more has Retry all and Retire
|
|
|
52
51
|
all; a connection held because every model on it fails says so above its
|
|
53
52
|
models, with any link the provider gave. Automatic retirements from the last
|
|
54
53
|
week are folded at the end, each with Restore. Its count sits beside it in the
|
|
55
|
-
sidebar
|
|
56
|
-
soon. An alert whose models have all been retired goes at once. A model that
|
|
54
|
+
sidebar. An alert whose models have all been retired goes at once. A model that
|
|
57
55
|
is out of quota is not counting down: days the bench held its calls back until
|
|
58
56
|
a stated reset, an account's shared free quota was spent, or the provider said
|
|
59
57
|
the model's free quota is used up, pause its clock, and while it is still out
|
package/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.6.
|
|
1
|
+
0.6.5
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@wendellresearch-ai/benchwarmers",
|
|
3
|
-
"version": "0.6.
|
|
3
|
+
"version": "0.6.5",
|
|
4
4
|
"description": "Self-hosted panel gateway for LLMs: ask a collection of models at once, keep every answer side by side, rotate explorer models in, and let your verdicts pick the next panel.",
|
|
5
5
|
"mcpName": "io.github.WendellResearch-AI/benchwarmers",
|
|
6
6
|
"license": "MIT",
|
package/scripts/check.py
CHANGED
|
@@ -49,6 +49,7 @@ CHECKS = [
|
|
|
49
49
|
("scripts/onboard.py", ["--selftest"], False),
|
|
50
50
|
("scripts/export_check.py", ["--selftest"], False),
|
|
51
51
|
("scripts/notice.py", ["--check"], False),
|
|
52
|
+
("scripts/clients_doc.py", ["--check"], False), # CLIENTS.md is what the Clients page shows
|
|
52
53
|
("scripts/check_themes.py", [], False),
|
|
53
54
|
("scripts/test_owui.py", [], False),
|
|
54
55
|
("scripts/test_hubs.py", [], False),
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
"""Write CLIENTS.md from the dashboard's Clients page, so each app's set-up is in
|
|
2
|
+
the repository for people and agents to read, and never drifts from the page.
|
|
3
|
+
|
|
4
|
+
python scripts/clients_doc.py # write CLIENTS.md
|
|
5
|
+
python scripts/clients_doc.py --check # fail if CLIENTS.md is not what it would write (check.py)
|
|
6
|
+
|
|
7
|
+
One source: serve.client_setups, the function the Clients page draws. The page
|
|
8
|
+
fills in this install (its address, its collections, a key made on the spot);
|
|
9
|
+
here they are the defaults a fresh install has: the server at 127.0.0.1:8122,
|
|
10
|
+
collections `core` and `fast`, and every call needing a gateway key, read from
|
|
11
|
+
BENCHWARMERS_KEY. The page's HTML (code, links) becomes Markdown.
|
|
12
|
+
"""
|
|
13
|
+
|
|
14
|
+
from __future__ import annotations
|
|
15
|
+
|
|
16
|
+
import html
|
|
17
|
+
import os
|
|
18
|
+
import pathlib
|
|
19
|
+
import re
|
|
20
|
+
import sys
|
|
21
|
+
import tempfile
|
|
22
|
+
|
|
23
|
+
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
|
24
|
+
OUT = ROOT / "CLIENTS.md"
|
|
25
|
+
COLLECTIONS = [("core", None, False), ("fast", None, False)]
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
def md(text: str) -> str:
|
|
29
|
+
"""The page's inline HTML as Markdown: code, links, line breaks; any other tag dropped."""
|
|
30
|
+
text = re.sub(r"<code>(.*?)</code>", lambda m: "`" + html.unescape(m.group(1)) + "`", text)
|
|
31
|
+
text = re.sub(r"<a [^>]*href='([^']*)'[^>]*>(.*?)</a>", r"[\2](\1)", text)
|
|
32
|
+
text = re.sub(r"<br\s*/?>", " \n", text)
|
|
33
|
+
return html.unescape(re.sub(r"<[^>]+>", "", text))
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
def fence(lang: str, text: str) -> str:
|
|
37
|
+
tick = "````" if "```" in text else "```"
|
|
38
|
+
return f"{tick}{lang}\n{text}\n{tick}"
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
def render() -> str:
|
|
42
|
+
# serve reads the data folder at import: an empty temp one, never the install's.
|
|
43
|
+
os.environ.setdefault("BENCHWARMERS_HOME", tempfile.mkdtemp(prefix="bench-clients-doc-"))
|
|
44
|
+
sys.path[:0] = [str(ROOT), str(ROOT / "metrics"), str(ROOT / "suite")]
|
|
45
|
+
import serve
|
|
46
|
+
root = serve.DEFAULT_SERVER
|
|
47
|
+
out = ["# Setting up clients",
|
|
48
|
+
"",
|
|
49
|
+
"Every app Benchwarmers has a ready set-up for, as the dashboard's **Clients** page shows it,"
|
|
50
|
+
" written out for people and for agents setting an app up. The page fills in your own install:"
|
|
51
|
+
" its address, your collections and a gateway key made on the spot. Here they are a fresh"
|
|
52
|
+
f" install's: the server at `{root}`, the collections `core` and `fast`, and a gateway key"
|
|
53
|
+
f" read from `{serve.KEY_ENV}`. Replace them with yours.",
|
|
54
|
+
"",
|
|
55
|
+
"Agents get a collection in route mode (one member with its tools, `bench/<collection>:route`);"
|
|
56
|
+
" chat windows a merged answer (`:merged`); the MCP server the panel itself, as ask and verdict"
|
|
57
|
+
f" tools. Gateway `{root}/v1` · MCP server `{root}/mcp`.",
|
|
58
|
+
"",
|
|
59
|
+
f"_Generated from `serve.client_setups` by `scripts/clients_doc.py`; each app's format follows its"
|
|
60
|
+
f" own docs as read on {serve.CLIENTS_READ}. Do not edit by hand: run the script._",
|
|
61
|
+
"",
|
|
62
|
+
"## The gateway key",
|
|
63
|
+
"",
|
|
64
|
+
"When every call needs a key (`BENCHWARMERS_REQUIRE_KEYS=1`, or an admin token), make one on"
|
|
65
|
+
" Settings or the Clients page. Most set-ups read it from"
|
|
66
|
+
f" `{serve.KEY_ENV}`:",
|
|
67
|
+
""]
|
|
68
|
+
for s in serve._env_snips(serve.KEY_ENV):
|
|
69
|
+
out += [f"**{s['label']}**", "", fence(s["lang"], s["text"]), ""]
|
|
70
|
+
setups = serve.client_setups(root, COLLECTIONS, True)
|
|
71
|
+
out += ["## Contents", ""] + [f"- [{c['name']}](#{c['id']})" for c in setups] + [""]
|
|
72
|
+
for c in setups:
|
|
73
|
+
out += [f"<a id='{c['id']}'></a>", "", f"## {c['name']}", "", md(c["gets"]), "",
|
|
74
|
+
f"**Where it goes:** {md(c['where'])}", ""]
|
|
75
|
+
for s in c["blocks"]:
|
|
76
|
+
out += [f"**{s['label']}**", "", fence(s["lang"], s["text"]), ""]
|
|
77
|
+
docs = ", ".join(f"[{t}]({u})" for u, t in c["docs"])
|
|
78
|
+
out += [f"**Watch for:** {md(c['watch'])}", "",
|
|
79
|
+
f"Written from {docs}, read {serve.CLIENTS_READ}"
|
|
80
|
+
+ (f": {md(c['docs_note'])}" if c.get("docs_note") else "."), ""]
|
|
81
|
+
return "\n".join(out).rstrip() + "\n"
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def main() -> int:
|
|
85
|
+
text = render()
|
|
86
|
+
if "--check" in sys.argv:
|
|
87
|
+
if not OUT.exists() or OUT.read_text(encoding="utf-8") != text:
|
|
88
|
+
print("CLIENTS.md is not what serve.client_setups gives: run python scripts/clients_doc.py")
|
|
89
|
+
return 1
|
|
90
|
+
print("CLIENTS.md matches the Clients page")
|
|
91
|
+
return 0
|
|
92
|
+
OUT.write_text(text, encoding="utf-8", newline="\n")
|
|
93
|
+
print(f"wrote {OUT.name}")
|
|
94
|
+
return 0
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
if __name__ == "__main__":
|
|
98
|
+
sys.exit(main())
|
package/serve.py
CHANGED
|
@@ -2254,27 +2254,6 @@ def open_count(groups: dict) -> int:
|
|
|
2254
2254
|
return sum(len(g["rows"]) + len(g["held"]) for g in groups.values())
|
|
2255
2255
|
|
|
2256
2256
|
|
|
2257
|
-
def and_list(parts: list[str]) -> str:
|
|
2258
|
-
return " and ".join(parts) if len(parts) < 3 else ", ".join(parts[:-1]) + " and " + parts[-1]
|
|
2259
|
-
|
|
2260
|
-
|
|
2261
|
-
def actions_line(groups: dict) -> str:
|
|
2262
|
-
"""At a glance's one line about Action needed, when anything is open:
|
|
2263
|
-
what is waiting and how soon, and the way to its page."""
|
|
2264
|
-
rows = [r for g in groups.values() for r in g["rows"]]
|
|
2265
|
-
held = sum(len(g["held"]) for g in groups.values())
|
|
2266
|
-
if not rows and not held:
|
|
2267
|
-
return ""
|
|
2268
|
-
models = sum(1 for r in rows if r["alias"])
|
|
2269
|
-
what = and_list([f"{n} {word}{'s' * (n != 1)}" for n, word in
|
|
2270
|
-
((models, "model"), (held, "connection"), (len(rows) - models, "notice")) if n])
|
|
2271
|
-
soon = min(r["days"] for r in rows) if rows else 99
|
|
2272
|
-
many = sum(r["days"] == soon for r in rows)
|
|
2273
|
-
when = (f"; {many} retire{'s' * (many == 1)} in {soon} day{'s' * (soon != 1)}" if soon < 99 else "")
|
|
2274
|
-
return (f"<p class='a-line'><strong>Action needed</strong> <span>{what}{when}.</span>"
|
|
2275
|
-
" <button type='button' class='linkish' data-go='needs'>Review →</button></p>")
|
|
2276
|
-
|
|
2277
|
-
|
|
2278
2257
|
def actions_page(groups: dict, auto: list) -> str:
|
|
2279
2258
|
"""Action needed, its own page since 2026-11-02: it sat above every page
|
|
2280
2259
|
and grew to a screen and more of red. Grouped by connection, because
|
|
@@ -2373,167 +2352,6 @@ def price_of(alias: str, served: str, f: dict) -> str:
|
|
|
2373
2352
|
return "paid" if sources.is_free(f.get("cost")) is False else "unknown"
|
|
2374
2353
|
|
|
2375
2354
|
|
|
2376
|
-
def facet(name: str, label: str, values) -> str:
|
|
2377
|
-
"""A filter by one column's value, beside a panel's search box (bench.js
|
|
2378
|
-
matches it against each row's data-<name>)."""
|
|
2379
|
-
opts = "".join(f"<option value='{esc(v)}'>{esc(v)}</option>" for v in sorted(set(values) - {""}))
|
|
2380
|
-
return (f"<label>{esc(label)} <select data-facet='{esc(name)}'><option value=''>any</option>"
|
|
2381
|
-
f"{opts}</select></label>")
|
|
2382
|
-
|
|
2383
|
-
|
|
2384
|
-
def facts_cells(alias: str, f: dict | None = None) -> str:
|
|
2385
|
-
"""Context window and capabilities, from metrics/catalog.py; a value you
|
|
2386
|
-
entered is marked, and an unknown one says so rather than looking empty."""
|
|
2387
|
-
f = f or catalog.info(alias)
|
|
2388
|
-
ctx = f.get("context_window")
|
|
2389
|
-
src = f["sources"].get("context_window", "")
|
|
2390
|
-
ctx_cell = (f"<td class='num' data-sort='{ctx}' title='{esc(src)}'>"
|
|
2391
|
-
f"{ctx // 1000:,}k{'*' if src == 'you' else ''}</td>" if ctx
|
|
2392
|
-
else "<td class='num muted' data-sort='-1' title='unknown: fill it in the model popup'>?</td>")
|
|
2393
|
-
caps = [CAP_LABELS[c] for c in catalog.CAPS if f.get(c)]
|
|
2394
|
-
return ctx_cell + (f"<td class='caps'>{' '.join(chip(c, '') for c in caps)}</td>" if caps
|
|
2395
|
-
else "<td class='muted'>—</td>")
|
|
2396
|
-
|
|
2397
|
-
|
|
2398
|
-
def roster() -> str:
|
|
2399
|
-
"""Every model on one table: its state, what it has done, what it scored.
|
|
2400
|
-
|
|
2401
|
-
Replaces a grid of offload cards that answered only "what was it used
|
|
2402
|
-
for". Status first, because the first question on opening this page is
|
|
2403
|
-
whether anything is broken; real work second; benchmarks after."""
|
|
2404
|
-
try:
|
|
2405
|
-
from offload import summary
|
|
2406
|
-
work = summary()
|
|
2407
|
-
except Exception: # noqa: BLE001
|
|
2408
|
-
work = {}
|
|
2409
|
-
try:
|
|
2410
|
-
import auto_retire
|
|
2411
|
-
rows, answers, _ = auto_retire.evidence()
|
|
2412
|
-
except Exception: # noqa: BLE001
|
|
2413
|
-
rows, answers = [], {}
|
|
2414
|
-
last = dict(answers)
|
|
2415
|
-
for r in rows:
|
|
2416
|
-
if r["answered"]:
|
|
2417
|
-
last[r["alias"]] = max(last.get(r["alias"], r["date"]), r["date"])
|
|
2418
|
-
suite: dict[str, list[int]] = {}
|
|
2419
|
-
for (_task, model), hit in suite_scores().items():
|
|
2420
|
-
if (m := re.match(r"(\d+)/(\d+)", hit["found"])):
|
|
2421
|
-
got = suite.setdefault(model, [0, 0, 0])
|
|
2422
|
-
got[0] += int(m.group(1)); got[1] += int(m.group(2)); got[2] += 1
|
|
2423
|
-
tiny = {r["model"]: float(r["p50_s"]) for r in perf_rows()
|
|
2424
|
-
if r["prompt"] == "tiny" and int(r.get("answered") or 0) > 0}
|
|
2425
|
-
status = model_status()
|
|
2426
|
-
new = recently_onboarded()
|
|
2427
|
-
stuck = onboard_issues(status)
|
|
2428
|
-
directory = model_directory()
|
|
2429
|
-
severity = {"err": 0, "warn": 1, "": 2, "ok": 3}
|
|
2430
|
-
|
|
2431
|
-
up = uptime_rows()["models"]
|
|
2432
|
-
body = ["<table><thead><tr>", th("Model"), th("Status"), th("Uptime 7d", True), th("Role"),
|
|
2433
|
-
th("Context", True), th("Can"),
|
|
2434
|
-
th("Offloads", True), th("Helped"), th("Suite", True),
|
|
2435
|
-
th("Reply time", True), th("Last answer"), "</tr></thead><tbody>"]
|
|
2436
|
-
# The model browser: each row says its connection, status, role and price
|
|
2437
|
-
# for the toolbar's filters, and the filters offer what the rows hold.
|
|
2438
|
-
offered: dict[str, list[str]] = {"connection": [], "status": [], "role": [], "price": []}
|
|
2439
|
-
|
|
2440
|
-
def browse(**values) -> str:
|
|
2441
|
-
for k, v in values.items():
|
|
2442
|
-
offered[k].append(v)
|
|
2443
|
-
return "".join(f" data-{k}='{esc(v)}'" for k, v in values.items())
|
|
2444
|
-
for alias in sorted(status, key=lambda a: (-work.get(a, {}).get("offloads", 0), a)):
|
|
2445
|
-
label, css, why = status[alias]
|
|
2446
|
-
f = catalog.info(alias)
|
|
2447
|
-
served = directory.get(alias, {}).get("served") or retired().get(alias, {}).get("served", "")
|
|
2448
|
-
w = work.get(alias, {})
|
|
2449
|
-
judged, useful = w.get("judged", 0), w.get("useful", 0)
|
|
2450
|
-
if judged:
|
|
2451
|
-
rate = useful / judged
|
|
2452
|
-
helped = (f"<span class='helped'><span class='meter'><span"
|
|
2453
|
-
f"{' class=low' if rate < .5 else ''} style='width:{max(4, round(rate * 100))}%'>"
|
|
2454
|
-
f"</span></span><span class='mono'>{useful}/{judged}</span></span>")
|
|
2455
|
-
else:
|
|
2456
|
-
rate, helped = -1, "<span class='muted'>not judged</span>"
|
|
2457
|
-
s = suite.get(alias)
|
|
2458
|
-
suite_cell = (f"<td class='num' data-sort='{s[0] / s[1]:.3f}'"
|
|
2459
|
-
f" title='{s[2]} task{'s' * (s[2] != 1)}'>{s[0]}/{s[1]}</td>"
|
|
2460
|
-
if s and s[1] else "<td class='num muted' data-sort='-1'>—</td>")
|
|
2461
|
-
p50 = tiny.get(alias)
|
|
2462
|
-
seen = last.get(alias)
|
|
2463
|
-
role = directory.get(alias, {}).get("role") or retired().get(alias, {}).get("role", "")
|
|
2464
|
-
conn = catalog.connection_of(alias)
|
|
2465
|
-
if conn == "?": # retired: no registration left to say, but its record does
|
|
2466
|
-
conn = retired().get(alias, {}).get("tag", "").split("/")[0] or "unknown"
|
|
2467
|
-
attrs = browse(connection=conn, status=re.sub(r" in \d+d$", "", label),
|
|
2468
|
-
role=role, price=price_of(alias, served, f))
|
|
2469
|
-
body.append(
|
|
2470
|
-
f"{tr(alias, attrs)}<td>{named(alias)}</td>"
|
|
2471
|
-
f"<td data-sort='{severity[css]}'>{chip(label, css, why)}"
|
|
2472
|
-
+ (" " + chip("new", "new", "onboarded this week") if alias in new else "")
|
|
2473
|
-
+ (" " + chip("onboard stopped", stuck[alias]["css"],
|
|
2474
|
-
f"onboarding stopped at the {stuck[alias].get('step', 'probe')}:"
|
|
2475
|
-
f" {stuck[alias]['label']}. Open the model for the error.")
|
|
2476
|
-
if alias in stuck else "")
|
|
2477
|
-
+ "</td>" + uptime_cell(up.get(alias)) + f"<td>{esc(role)}</td>{facts_cells(alias, f)}"
|
|
2478
|
-
f"<td class='num'>{w.get('offloads', 0)}</td>"
|
|
2479
|
-
f"<td data-sort='{rate:.3f}'>{helped}</td>{suite_cell}"
|
|
2480
|
-
+ (f"<td class='num' data-sort='{p50}'>{p50:.1f}s</td>" if p50 is not None
|
|
2481
|
-
else "<td class='num muted' data-sort='99999'>—</td>")
|
|
2482
|
-
+ (f"<td class='mono' data-sort='{seen}'>{seen}</td>" if seen
|
|
2483
|
-
else "<td class='muted' data-sort='0'>never</td>")
|
|
2484
|
-
+ "</tr>")
|
|
2485
|
-
# Decision models are not in the llm yaml, so no probe, suite or reply time
|
|
2486
|
-
# applies; the role column says what they are and what they answer.
|
|
2487
|
-
for alias, d in sorted(decide.load().items()):
|
|
2488
|
-
w = work.get(alias, {})
|
|
2489
|
-
body.append(
|
|
2490
|
-
"<tr" + browse(connection=d.get("via", ""), status="no probe", role="decision",
|
|
2491
|
-
price="free" if ":free" in d["served"] else "unknown") + ">"
|
|
2492
|
-
f"<td>{named(alias)}</td>"
|
|
2493
|
-
"<td data-sort='2'><span class='muted' title='decision models have no"
|
|
2494
|
-
" availability probe yet'>—</span>"
|
|
2495
|
-
+ (" " + chip("new", "new", "onboarded this week") if d.get("onboarded", "") >=
|
|
2496
|
-
(datetime.date.today() - datetime.timedelta(days=7)).isoformat() else "")
|
|
2497
|
-
+ "</td>" + uptime_cell(up.get(alias)) + f"<td>decision · {esc(', '.join(d['question_types']))}</td>"
|
|
2498
|
-
"<td class='num muted' data-sort='-1'>—</td><td class='muted'>—</td>"
|
|
2499
|
-
f"<td class='num'>{w.get('offloads', 0)}</td>"
|
|
2500
|
-
"<td data-sort='-1'><span class='muted'>not judged</span></td>"
|
|
2501
|
-
"<td class='num muted' data-sort='-1'>—</td><td class='num muted' data-sort='99999'>—</td>"
|
|
2502
|
-
"<td class='muted' data-sort='0'>—</td></tr>")
|
|
2503
|
-
body.append("</tbody></table>")
|
|
2504
|
-
key = """<dl>
|
|
2505
|
-
<dt>Status</dt><dd>From the latest probe. <em>healthy</em> answered;
|
|
2506
|
-
<em>throttled</em> and <em>slow</em> are rate limits and timeouts, which
|
|
2507
|
-
never retire a model inside 7 days; <em>retiring in Nd</em> is the
|
|
2508
|
-
countdown before the bench retires it on its own. Hover a chip for the
|
|
2509
|
-
provider's own words.</dd>
|
|
2510
|
-
<dt>Uptime 7d</dt><dd>Of its calls in the last 7 days that say anything about the provider,
|
|
2511
|
-
the share answered: busy, unreachable and timed out count against it; our own quota,
|
|
2512
|
-
pacing and setup problems are left out. Benchmarks has it a day at a time.</dd>
|
|
2513
|
-
<dt>Offloads, Helped</dt><dd>Real work handed over, and how much of what
|
|
2514
|
-
was judged actually helped. Benchmarks never count here.</dd>
|
|
2515
|
-
<dt>Suite</dt><dd>Verified findings located across the internal tasks:
|
|
2516
|
-
the tier that decides routing.</dd>
|
|
2517
|
-
<dt>Reply time</dt><dd>Median seconds to answer a one-word prompt: the
|
|
2518
|
-
serving path's overhead, not the model's worth.</dd>
|
|
2519
|
-
<dt>Last answer</dt><dd>The latest real answer from any source: probe,
|
|
2520
|
-
suite or a useful offload. Seven days without one retires a model.</dd>
|
|
2521
|
-
<dt>decision</dt><dd>A decision model: it answers typed questions (yes/no,
|
|
2522
|
-
choice, score) with probabilities through a Decisions API, never chat.
|
|
2523
|
-
The role column lists the question types it takes. No probe, suite or
|
|
2524
|
-
reply time applies to it.</dd>
|
|
2525
|
-
<dt>Filters</dt><dd>Connection is where a model's calls go (its Open WebUI
|
|
2526
|
-
tag, or its own endpoint). Price: <em>free</em> for a <code>:free</code>
|
|
2527
|
-
slug, a local model, a list price of 0 or a free-tier listing;
|
|
2528
|
-
<em>paid</em> for a subscription or a list price; else unknown. Combine
|
|
2529
|
-
them with the search box.</dd>
|
|
2530
|
-
</dl>"""
|
|
2531
|
-
facets = "".join(facet(k, k.title(), v) for k, v in offered.items())
|
|
2532
|
-
return card("At a glance", "Every model on the bench: whether it works right "
|
|
2533
|
-
"now, what it has been used for, and what it scored.",
|
|
2534
|
-
"".join(body), key, facets)
|
|
2535
|
-
|
|
2536
|
-
|
|
2537
2355
|
def on_bench() -> set[str]:
|
|
2538
2356
|
"""Aliases registered now: what the charts draw."""
|
|
2539
2357
|
return set(registered().values())
|
|
@@ -2883,7 +2701,14 @@ def vision_card() -> str:
|
|
|
2883
2701
|
come from these alone. <b>Useful, no claims</b> is a verdict without them, which on thicken's
|
|
2884
2702
|
photos means the seat agreed with another seat. Agreement is not a check, since seats fail
|
|
2885
2703
|
together, so it is never counted as right.</dd>
|
|
2886
|
-
<dt>The
|
|
2704
|
+
<dt>The real-work chart</dt><dd>Read as Real work is. The dot is the share of its judged seats your
|
|
2705
|
+
verdict called useful (checked and helped, or useful with no claims), bigger for more judged; the
|
|
2706
|
+
line is where that share likely sits (95%), and the order is by its low end, so a lot of evidence
|
|
2707
|
+
outranks a little. Green when even the low end is over half, red when even the high end is under,
|
|
2708
|
+
grey when there are too few to say. Beside it: useful of judged, how often it answered at all, and
|
|
2709
|
+
the claims checked by hand, the only count of right and wrong. Seats not judged or not answered
|
|
2710
|
+
are in those words and the table, not the dot.</dd>
|
|
2711
|
+
<dt>The number beside a name</dt><dd>How much there is: fields asked on the tasks.</dd>
|
|
2887
2712
|
</dl>"""
|
|
2888
2713
|
held = (f"<p class='hint'>The vision tasks wait until {esc(v['held'])}: no run picks them before then"
|
|
2889
2714
|
" (<code>vision_from</code> in <code>config/suite.json</code>; remove it to start them now).</p>"
|
|
@@ -2891,7 +2716,7 @@ def vision_card() -> str:
|
|
|
2891
2716
|
return (f"<div class='panel'>{SHARE}<h2>Vision</h2><p class='sub'>Reading images: fields read right on pictures with"
|
|
2892
2717
|
" a known answer, then real work on images and how much of it was actually checked.</p>" + held +
|
|
2893
2718
|
"<h3 class='chart-head'>On the vision tasks, field by field</h3><div class='chart' id='chart-vision-fields'></div>"
|
|
2894
|
-
"<h3 class='chart-head'>On real work with images,
|
|
2719
|
+
"<h3 class='chart-head'>On real work with images: judged useful, and how sure that is</h3>"
|
|
2895
2720
|
"<div class='chart' id='chart-vision-work'></div>"
|
|
2896
2721
|
"<details class='key'><summary>Table</summary><div class='toolbar'><input type='search'"
|
|
2897
2722
|
" data-filter placeholder='Filter by model or connection' aria-label='Filter rows'>"
|
|
@@ -3378,15 +3203,6 @@ def _uptime_window(cells: list, days: int) -> dict:
|
|
|
3378
3203
|
"median": seconds[len(seconds) // 2] if seconds else None}
|
|
3379
3204
|
|
|
3380
3205
|
|
|
3381
|
-
def uptime_cell(cells: list | None, days: int = UPTIME_DAYS[0]) -> str:
|
|
3382
|
-
"""At a glance's Uptime column: answered of counted over the last `days`."""
|
|
3383
|
-
t = _uptime_window(cells or [], days)
|
|
3384
|
-
if t["pct"] is None:
|
|
3385
|
-
return "<td class='num muted' data-sort='-1' title='no counted calls'>—</td>"
|
|
3386
|
-
return (f"<td class='num{' err' if t['pct'] < .8 else ''}' data-sort='{t['pct']:.4f}'"
|
|
3387
|
-
f" title='{t['up']} of {t['up'] + t['down']} answered'>{t['pct'] * 100:.1f}%</td>")
|
|
3388
|
-
|
|
3389
|
-
|
|
3390
3206
|
def _why(kinds: dict, which) -> str:
|
|
3391
3207
|
return ", ".join(f"{k} {n}" for k, n in sorted(kinds.items(), key=lambda kv: -kv[1]) if which(k))
|
|
3392
3208
|
|
|
@@ -3415,7 +3231,7 @@ def uptime_data() -> dict:
|
|
|
3415
3231
|
|
|
3416
3232
|
def uptime_card() -> str:
|
|
3417
3233
|
"""Was the provider there when the bench called it: a cell a day, by
|
|
3418
|
-
provider
|
|
3234
|
+
provider or by model, as its By says; 7 days, or 30 on choice (bench.js swaps them)."""
|
|
3419
3235
|
u = uptime_rows()
|
|
3420
3236
|
first, gone, names = UPTIME_DAYS[0], retired(), {}
|
|
3421
3237
|
|
|
@@ -3489,9 +3305,11 @@ def uptime_card() -> str:
|
|
|
3489
3305
|
"<span><i style='background:var(--signal)'></i>some down, 80% or more answered</span>"
|
|
3490
3306
|
"<span><i style='background:var(--err)'></i>under 80% answered</span>"
|
|
3491
3307
|
"<span><i style='background:none;border:1px solid var(--line)'></i>no counted calls</span></div>")
|
|
3308
|
+
# The window and one view at a time: by provider (shorter, the default) or by model (bench.js).
|
|
3492
3309
|
pick = ("<div class='toolbar'><label>Days <select data-uptime-days>"
|
|
3493
3310
|
+ "".join(f"<option value='{w}'>last {w}</option>" for w in UPTIME_DAYS)
|
|
3494
|
-
+
|
|
3311
|
+
+ "</select></label><label>By <select data-uptime-by><option value='providers'>provider</option>"
|
|
3312
|
+
f"<option value='models'>model</option></select></label></div>{legend}")
|
|
3495
3313
|
key = """<dl>
|
|
3496
3314
|
<dt>Down</dt><dd>The provider was busy or overloaded (a 5xx, high demand, a shared pool's
|
|
3497
3315
|
429), unreachable (a broken stream, no reply), timed out (the gateway's 420 s included), or
|
|
@@ -3517,9 +3335,7 @@ def uptime_card() -> str:
|
|
|
3517
3335
|
" and every real call, one cell a day. Busy, unreachable and timed out count as down;"
|
|
3518
3336
|
" our own quota, our own pacing and setup problems are left out.",
|
|
3519
3337
|
"chart-uptime-models", "".join(window(w) for w in UPTIME_DAYS), key,
|
|
3520
|
-
extra=pick + "<
|
|
3521
|
-
"<div class='chart' id='chart-uptime-providers'></div>"
|
|
3522
|
-
"<h3 class='chart-head'>By model</h3>",
|
|
3338
|
+
extra=pick + "<div class='chart' id='chart-uptime-providers'></div>",
|
|
3523
3339
|
after="".join(odd(w) for w in UPTIME_DAYS))
|
|
3524
3340
|
|
|
3525
3341
|
|
|
@@ -3846,10 +3662,20 @@ def wrong_card() -> str:
|
|
|
3846
3662
|
|
|
3847
3663
|
def benchmarks_tab() -> str:
|
|
3848
3664
|
data = json.dumps(charts_data(), ensure_ascii=False).replace("</", "<\\/")
|
|
3849
|
-
#
|
|
3850
|
-
|
|
3851
|
-
|
|
3852
|
-
|
|
3665
|
+
# Who to trust, then what moves day to day (Uptime, Trend, Performance), then Real work,
|
|
3666
|
+
# then the rest in the order they had. Each report gets an id; a bar of links to them
|
|
3667
|
+
# stays at the top while the page scrolls, the one in view marked (bench.js jumpSpy).
|
|
3668
|
+
reports = (("trust", "Who to trust", overview_card), ("uptime", "Uptime", uptime_card),
|
|
3669
|
+
("trend", "Trend", trend_card), ("performance", "Performance", perf_card),
|
|
3670
|
+
("work", "Real work", work_card), ("hunt", "Bug hunt", bughunt_card),
|
|
3671
|
+
("wrong", "Confidently wrong", wrong_card), ("code", "Our code", suite_card),
|
|
3672
|
+
("vision", "Vision", vision_card), ("decisions", "Decision models", decision_card),
|
|
3673
|
+
("external", "External", external_card))
|
|
3674
|
+
links = "".join(f"<a href='#report-{key}'>{esc(label)}</a>" for key, label, _ in reports)
|
|
3675
|
+
cards = "".join(card().replace("<div class='panel'", f"<div class='panel report' id='report-{key}'", 1)
|
|
3676
|
+
for key, _, card in reports)
|
|
3677
|
+
return (f"<nav class='jump' aria-label='On this page'><p class='jump-title'>On this page</p>{links}</nav>{cards}"
|
|
3678
|
+
f"<script type='application/json' id='charts-data'>{data}</script>")
|
|
3853
3679
|
|
|
3854
3680
|
|
|
3855
3681
|
def perf_card() -> str:
|
|
@@ -3925,12 +3751,12 @@ def favicon(theme: str) -> str:
|
|
|
3925
3751
|
return f"<link rel='icon' type='image/svg+xml' href='data:image/svg+xml,{quote(svg)}'>"
|
|
3926
3752
|
|
|
3927
3753
|
|
|
3928
|
-
def render(theme: str | None = None, shown: str = "
|
|
3754
|
+
def render(theme: str | None = None, shown: str = "suite") -> str:
|
|
3929
3755
|
with locked.snapshot(): # each file read once a render (metrics/locked.py)
|
|
3930
3756
|
return _page(theme, shown)
|
|
3931
3757
|
|
|
3932
3758
|
|
|
3933
|
-
def _page(theme: str | None, shown: str = "
|
|
3759
|
+
def _page(theme: str | None, shown: str = "suite") -> str:
|
|
3934
3760
|
theme = theme if theme in themes() else saved_theme() # a ?theme preview, else the saved one
|
|
3935
3761
|
return (
|
|
3936
3762
|
f"<!doctype html><html lang='en' data-theme='{esc(theme)}'><meta charset='utf-8'>"
|
|
@@ -4406,9 +4232,10 @@ def collections_tab() -> str:
|
|
|
4406
4232
|
# A grouped sidebar since 2026-10-19 (RELEASE-PLAN "The dashboard"): eight
|
|
4407
4233
|
# pages were too many for a row of tabs. Collections holds the gateway's
|
|
4408
4234
|
# requests under the collections they ran; Benchmarks holds Performance.
|
|
4409
|
-
|
|
4235
|
+
# At a glance (every model's state and uptime) was retired 2026-11-10: Benchmarks,
|
|
4236
|
+
# Uptime and Models said it all again. Benchmarks is the home page.
|
|
4237
|
+
SECTIONS = (("Observe", (("suite", "Benchmarks", "benchmarks_tab"),
|
|
4410
4238
|
("needs", "Action needed", "actions_page"),
|
|
4411
|
-
("suite", "Benchmarks", "benchmarks_tab"),
|
|
4412
4239
|
("quotas", "Quota management", "quotas_page"),
|
|
4413
4240
|
("logs", "Logs", "logs_page"))),
|
|
4414
4241
|
("Gateway", (("endpoints", "Endpoints", "endpoints_page"),
|
|
@@ -4433,7 +4260,9 @@ def collections_and_gateway() -> str:
|
|
|
4433
4260
|
# names an environment variable, or the app asks, or (an MCP header in Zed,
|
|
4434
4261
|
# Cline or goose, which take no variable) KEY_HOLDER stays for the reader.
|
|
4435
4262
|
# Only blocks marked `key` (shell lines, fields typed into an app) get the key
|
|
4436
|
-
# made on the page, swapped in for KEY_HOLDER by bench.js.
|
|
4263
|
+
# made on the page, swapped in for KEY_HOLDER by bench.js. CLIENTS.md is written
|
|
4264
|
+
# from client_setups (scripts/clients_doc.py): after a change here, run it, as
|
|
4265
|
+
# check.py fails while the file differs.
|
|
4437
4266
|
KEY_HOLDER = "<your gateway key>"
|
|
4438
4267
|
KEY_ENV = "BENCHWARMERS_KEY" # the plugin's and the CLI's name for it too
|
|
4439
4268
|
CLIENTS_READ = "2026-11-08"
|
|
@@ -4729,20 +4558,20 @@ def _snip_html(group: str, i: int, s: dict) -> str:
|
|
|
4729
4558
|
def pane(key: str, actions: tuple | None = None) -> str:
|
|
4730
4559
|
"""One tab's page, as its section holds it. The page renders the tab shown
|
|
4731
4560
|
(tabs); bench.js fetches the others from GET /api/pane when first opened."""
|
|
4732
|
-
if key
|
|
4561
|
+
if key == "needs":
|
|
4733
4562
|
groups, auto = actions or open_actions()
|
|
4734
|
-
return actions_page(groups, auto)
|
|
4563
|
+
return actions_page(groups, auto)
|
|
4735
4564
|
return globals()[next(fn for k, _, fn in TABS if k == key)]()
|
|
4736
4565
|
|
|
4737
4566
|
|
|
4738
|
-
def tabs(shown: str = "
|
|
4739
|
-
# Read once a render: Action needed's count in the sidebar, and its page
|
|
4740
|
-
#
|
|
4741
|
-
#
|
|
4742
|
-
#
|
|
4567
|
+
def tabs(shown: str = "suite") -> str:
|
|
4568
|
+
# Read once a render: Action needed's count in the sidebar, and its page when
|
|
4569
|
+
# shown. Only the tab shown is rendered (the bench_tab cookie, set by bench.js):
|
|
4570
|
+
# rendering every one took seconds. The rest are empty until opened (data-lazy).
|
|
4571
|
+
# A tab that is gone (a cookie still saying glance) shows the home page.
|
|
4743
4572
|
actions = open_actions()
|
|
4744
4573
|
count = {"needs": open_count(actions[0])}
|
|
4745
|
-
shown = shown if any(k == shown for k, _, _ in TABS) else "
|
|
4574
|
+
shown = shown if any(k == shown for k, _, _ in TABS) else "suite"
|
|
4746
4575
|
|
|
4747
4576
|
def button(key: str, label: str) -> str:
|
|
4748
4577
|
n = count.get(key)
|
|
@@ -5438,8 +5267,17 @@ def models_page() -> str:
|
|
|
5438
5267
|
return ("<div class='panel'><h2>Models</h2><p class='sub'>Every model on the bench by the connection it"
|
|
5439
5268
|
" comes from, with the endpoint it is reached through and what is known of its limits. A name"
|
|
5440
5269
|
" opens the model.</p><div class='toolbar'><input type='search' data-filter='page'"
|
|
5441
|
-
" placeholder='Find a model or connection' aria-label='Filter models'
|
|
5442
|
-
+ "".join(cards))
|
|
5270
|
+
" placeholder='Find a model or connection' aria-label='Filter models'>"
|
|
5271
|
+
f"{fold_toggle(True)}</div></div>" + "".join(cards))
|
|
5272
|
+
|
|
5273
|
+
|
|
5274
|
+
def fold_toggle(all_open: bool) -> str:
|
|
5275
|
+
"""One labelled toggle over a page's folds (Models' connections, Logs' logs), as
|
|
5276
|
+
accordions do it (GOV.UK's "Show all sections"): Collapse all while every one is
|
|
5277
|
+
open, else Expand all. bench.js foldButtons keeps it right as folds change."""
|
|
5278
|
+
return (f"<button class='fold-all' type='button' data-fold-all='{'close' if all_open else 'open'}'"
|
|
5279
|
+
f" aria-expanded='{str(all_open).lower()}'><span class='fold-chev' aria-hidden='true'></span>"
|
|
5280
|
+
f"{'Collapse all' if all_open else 'Expand all'}</button>")
|
|
5443
5281
|
|
|
5444
5282
|
|
|
5445
5283
|
def quota_tiles() -> list[dict]:
|
|
@@ -5615,11 +5453,25 @@ def logs_page() -> str:
|
|
|
5615
5453
|
for p in sorted((DATA / "runs").glob("onboard-*.log"), reverse=True)[:30])
|
|
5616
5454
|
|
|
5617
5455
|
def section(title, body, n, open_=False):
|
|
5618
|
-
return (f"<details class='panel'{' open' if open_ else ''}><summary><b>{esc(title)}</b>"
|
|
5456
|
+
return (f"<details class='panel'{' open' if open_ else ''} data-log='{esc(title)}'><summary><b>{esc(title)}</b>"
|
|
5619
5457
|
f" <span class='muted'>· {n}</span></summary>{body}</details>")
|
|
5458
|
+
# Export: a popup for the format and the logs (bench.js), each as the page shows it,
|
|
5459
|
+
# the filter applied. Text, every log chosen in one file to read; CSV, a file per log.
|
|
5460
|
+
picks = "".join(f"<label class='check'><input type='checkbox' name='lx-log' value='{esc(t)}' checked> {esc(t)}</label>"
|
|
5461
|
+
for t in ("Gateway calls", "Changes", "Scheduler", "Onboarding runs"))
|
|
5462
|
+
export = ("<dialog id='logs-export-dialog' aria-labelledby='lx-title'><h2 id='lx-title'>Export logs</h2>"
|
|
5463
|
+
"<fieldset><legend class='field'>Format</legend>"
|
|
5464
|
+
"<label class='check'><input type='radio' name='lx-format' value='txt' checked> Text (.txt)</label>"
|
|
5465
|
+
"<label class='check'><input type='radio' name='lx-format' value='csv'> CSV (.csv)</label></fieldset>"
|
|
5466
|
+
f"<fieldset><legend class='field'>Logs</legend>{picks}</fieldset>"
|
|
5467
|
+
"<p class='rules' id='lx-note'></p><div class='errors' id='lx-errors' role='alert'></div>"
|
|
5468
|
+
"<div class='actions'><button class='btn' type='button' id='lx-cancel'>Cancel</button>"
|
|
5469
|
+
"<button class='btn primary' type='button' id='lx-go'>Export</button></div></dialog>")
|
|
5620
5470
|
return (
|
|
5621
|
-
"<div class='panel'><input data-filter='page' placeholder='Filter every log on this page'"
|
|
5622
|
-
" aria-label='Filter logs'
|
|
5471
|
+
"<div class='panel'><div class='toolbar'><input data-filter='page' placeholder='Filter every log on this page'"
|
|
5472
|
+
" aria-label='Filter logs'>"
|
|
5473
|
+
"<button class='btn small' type='button' id='logs-export'>Export…</button>"
|
|
5474
|
+
f"{fold_toggle(False)}</div></div>{export}"
|
|
5623
5475
|
+ section("Gateway calls", "<table><thead><tr>" + th("When") + th("Collection") + th("Shape")
|
|
5624
5476
|
+ th("Seats") + f"</tr></thead><tbody>{''.join(gw)}</tbody></table>", len(gw), True)
|
|
5625
5477
|
+ section("Changes", "<table><thead><tr>" + th("When") + th("By") + th("Model") + th("What")
|
|
@@ -5938,8 +5790,7 @@ class Handler(gateway.Handler):
|
|
|
5938
5790
|
if self.path == "/api/needs": # after an action there: that page, not the whole one
|
|
5939
5791
|
with locked.snapshot():
|
|
5940
5792
|
groups, auto = open_actions()
|
|
5941
|
-
self._json(200, {"html": actions_page(groups, auto), "
|
|
5942
|
-
"count": open_count(groups)})
|
|
5793
|
+
self._json(200, {"html": actions_page(groups, auto), "count": open_count(groups)})
|
|
5943
5794
|
return
|
|
5944
5795
|
if path == "/api/pane": # a tab's page, when first opened (bench.js load)
|
|
5945
5796
|
key = q.get("tab", [""])[0]
|
|
@@ -6058,7 +5909,7 @@ class Handler(gateway.Handler):
|
|
|
6058
5909
|
theme = q.get("theme", [None])[0] # a preview; else the saved one
|
|
6059
5910
|
shown = re.search(r"(?:^|;)\s*bench_tab=(\w+)", self.headers.get("Cookie") or "") # the tab last shown
|
|
6060
5911
|
try:
|
|
6061
|
-
page = render(theme, shown.group(1) if shown else "
|
|
5912
|
+
page = render(theme, shown.group(1) if shown else "suite")
|
|
6062
5913
|
except RuntimeError as exc: # endpoints.json unreadable: say which and why
|
|
6063
5914
|
page = f"<!doctype html><meta charset='utf-8'><title>Benchwarmers</title><p>{esc(exc)}</p>"
|
|
6064
5915
|
self._page(page) # gzipped beyond this machine (_length)
|
package/ui/bench.css
CHANGED
|
@@ -117,8 +117,6 @@ main { max-width: var(--page, 104rem); margin: 0 auto; padding: 0 var(--gutter)
|
|
|
117
117
|
|
|
118
118
|
/* Action needed: its own page, a panel per connection as on Models, each a
|
|
119
119
|
table (name, state, the provider's words, actions). */
|
|
120
|
-
.a-line { display: flex; flex-wrap: wrap; align-items: baseline; gap: .25rem .6rem; margin: 1.4rem 1.6rem 0;
|
|
121
|
-
padding: .45rem .75rem; background: var(--err-ground); border-radius: var(--radius); }
|
|
122
120
|
.a-head { display: flex; flex-wrap: wrap; align-items: baseline; gap: .25rem .6rem; }
|
|
123
121
|
.a-head .why { font-size: .875rem; }
|
|
124
122
|
.a-head .why:empty { display: none; }
|
|
@@ -359,8 +357,7 @@ dialog h2 { margin: 0 0 .2rem; font-size: 1.375rem; font-weight: 700; }
|
|
|
359
357
|
:is(dialog, .form-panel) .members .m-row { border-bottom: 1px solid var(--line); }
|
|
360
358
|
:is(dialog, .form-panel) .members .m-row:nth-child(odd) { background: color-mix(in oklab, var(--band) 45%, transparent); }
|
|
361
359
|
:is(dialog, .form-panel) .members .m-row:has(input:checked) { box-shadow: inset 3px 0 0 var(--bench-seat, var(--brand)); }
|
|
362
|
-
:is(dialog, .form-panel) .members .m-name a.model {
|
|
363
|
-
:is(dialog, .form-panel) .members .m-name a.model:hover { text-decoration: underline; }
|
|
360
|
+
:is(dialog, .form-panel) .members .m-name a.model { font-size: .875rem; }
|
|
364
361
|
:is(dialog, .form-panel) .members .m-ctx { font-family: var(--font-mono); font-size: .8125rem; color: var(--muted); text-align: right; }
|
|
365
362
|
:is(dialog, .form-panel) .members .m-can { display: flex; flex-wrap: wrap; gap: .2rem; }
|
|
366
363
|
:is(dialog, .form-panel) .members .m-can .chip { font-size: .6875rem; padding: 0 .4rem; }
|
|
@@ -539,8 +536,44 @@ pre.log.answer { max-height: 40vh; }
|
|
|
539
536
|
.chart.muted { font-size: .875rem; }
|
|
540
537
|
h3.chart-head { font-size: 1rem; margin: 1rem 0 .2rem; }
|
|
541
538
|
.chart-label { font-size: .875rem; font-weight: 600; margin: .9rem 0 .1rem; }
|
|
539
|
+
/* Benchmarks' "On this page": a link per report, the one in view marked (bench.js jumpSpy),
|
|
540
|
+
after the U.S. Web Design System's in-page navigation. Dressed as the sidebar is: the
|
|
541
|
+
theme's frosted rail, muted links, the highlighter stroke on the current one.
|
|
542
|
+
Narrow: a bar floating at the top of the page, sideways when it does not fit.
|
|
543
|
+
Wide: a rail beside the reports, sticky, every link on its own line. */
|
|
544
|
+
.jump { position: sticky; top: .75rem; z-index: 4; display: flex; align-items: center; gap: 1rem;
|
|
545
|
+
margin: .9rem 1.2rem 0; padding: .45rem 1.1rem; overflow-x: auto; scrollbar-width: none;
|
|
546
|
+
background: var(--rail, var(--paper-2)); border: 1px solid var(--line); border-radius: 999px;
|
|
547
|
+
-webkit-backdrop-filter: var(--glass, blur(10px)); backdrop-filter: var(--glass, blur(10px));
|
|
548
|
+
box-shadow: 0 10px 28px -16px rgb(0 0 0 / .4); }
|
|
549
|
+
.jump::-webkit-scrollbar { display: none; }
|
|
550
|
+
.jump .jump-title { display: none; margin: 0 0 .2rem; font-size: .75rem; font-weight: 700; letter-spacing: .08em;
|
|
551
|
+
text-transform: uppercase; color: color-mix(in oklab, var(--brand) 72%, var(--ink)); }
|
|
552
|
+
.jump a { flex: none; padding: .1rem 0; line-height: 1.15; color: var(--muted); text-decoration: none;
|
|
553
|
+
font-size: .9375rem; font-weight: 500; white-space: nowrap; }
|
|
554
|
+
.jump a:hover, .jump a:focus-visible { color: var(--ink); }
|
|
555
|
+
.jump a[aria-current] { color: var(--ink); font-weight: 700; box-shadow: inset 0 -.42em 0 var(--signal); }
|
|
556
|
+
.panel.report { scroll-margin-top: 4.2rem; } /* a jump lands under the bar, not behind it */
|
|
557
|
+
@media (min-width: 75rem) {
|
|
558
|
+
#tab-suite:has(> .jump) { display: grid; grid-template-columns: minmax(0, 1fr) 11rem; }
|
|
559
|
+
#tab-suite:has(> .jump) > * { grid-column: 1; }
|
|
560
|
+
#tab-suite > .jump { grid-column: 2; grid-row: 1 / span 40; align-self: start; top: 1rem;
|
|
561
|
+
flex-direction: column; align-items: flex-start; gap: .6rem; overflow: visible;
|
|
562
|
+
margin: 1.4rem 1rem 1.4rem 0; padding: 1rem 1.1rem; border-radius: 14px; box-shadow: none; }
|
|
563
|
+
#tab-suite > .jump .jump-title { display: block; }
|
|
564
|
+
#tab-suite > .jump a { white-space: normal; }
|
|
565
|
+
.panel.report { scroll-margin-top: 1rem; }
|
|
566
|
+
}
|
|
542
567
|
/* Models: each connection a fold, its heading the summary */
|
|
543
568
|
.model-card > summary h3.chart-head { display: inline; margin: 0; }
|
|
569
|
+
/* Expand all / Collapse all: a text toggle with a chevron that turns, like a fold's own marker */
|
|
570
|
+
button.fold-all { display: inline-flex; align-items: center; gap: .4rem; margin-left: auto; border: 0; padding: .2rem 0;
|
|
571
|
+
background: none; color: var(--link); font: inherit; font-size: .9375rem; cursor: pointer; }
|
|
572
|
+
button.fold-all:hover, button.fold-all:focus-visible { text-decoration: underline; text-underline-offset: .18em; }
|
|
573
|
+
.fold-chev { width: .5rem; height: .5rem; border: solid currentColor; border-width: 0 2px 2px 0;
|
|
574
|
+
transform: rotate(-45deg); transition: transform .15s; }
|
|
575
|
+
button.fold-all[aria-expanded="true"] .fold-chev { transform: rotate(45deg); margin-top: -.3rem; }
|
|
576
|
+
@media (prefers-reduced-motion: reduce) { .fold-chev { transition: none; } }
|
|
544
577
|
/* the Collections tab: the dialog's form, kept to a readable width */
|
|
545
578
|
.panel.form-panel > :not(#c-list, table) { max-width: 46rem; }
|
|
546
579
|
/* a list of things you open (collections), with its New button above */
|
|
@@ -595,7 +628,6 @@ dialog .pick { display: flex; gap: 1.5rem; }
|
|
|
595
628
|
.toolbar input[type=search] { min-width: 0; width: 100%; }
|
|
596
629
|
button.c-row { grid-template-columns: minmax(0, 1fr); } /* the seats get the width; the whole row opens */
|
|
597
630
|
.c-row .open-hint { display: none; }
|
|
598
|
-
.a-line { margin: 1rem 1rem 0; }
|
|
599
631
|
}
|
|
600
632
|
/* A narrow page (a phone, or a window with the sidebar beside it): name and
|
|
601
633
|
state on one line, the words and actions under them. No gap between name
|
package/ui/bench.js
CHANGED
|
@@ -213,7 +213,7 @@
|
|
|
213
213
|
var rendered = document.querySelector('.tab:not([data-lazy])');
|
|
214
214
|
Promise.resolve().then(function () { if (rendered) setUp(rendered.id.slice(4)); });
|
|
215
215
|
var wanted = store('bench.tab'); // else the one rendered: storage blocked or cleared, the cookie kept
|
|
216
|
-
show(wanted && $('tab-' + wanted) ? wanted : rendered ? rendered.id.slice(4) : '
|
|
216
|
+
show(wanted && $('tab-' + wanted) ? wanted : rendered ? rendered.id.slice(4) : 'suite');
|
|
217
217
|
|
|
218
218
|
// ---- sortable tables -------------------------------------------------
|
|
219
219
|
function key(cell) {
|
|
@@ -276,7 +276,88 @@
|
|
|
276
276
|
document.addEventListener('input', function (ev) { if (ev.target.matches('input[data-filter]')) applyFilter(scopeOf(ev.target)); });
|
|
277
277
|
document.addEventListener('change', function (ev) { if (ev.target.matches('select[data-facet]')) applyFilter(scopeOf(ev.target)); });
|
|
278
278
|
|
|
279
|
-
// ----
|
|
279
|
+
// ---- Expand all / Collapse all: every fold of the page's groups (Models' connections).
|
|
280
|
+
// One labelled toggle, as accordions have it: Collapse all while every one is open,
|
|
281
|
+
// else Expand all. Kept right after any fold opens or shuts, by hand, by it or by a live swap.
|
|
282
|
+
var FOLDS = 'details.model-card, details[data-log]'; // the groups a page folds: Models' connections, Logs' logs
|
|
283
|
+
function foldButtons(tab) {
|
|
284
|
+
var folds = tab.querySelectorAll(FOLDS), open = 0;
|
|
285
|
+
folds.forEach(function (d) { if (d.open) open++; });
|
|
286
|
+
var all = open === folds.length;
|
|
287
|
+
tab.querySelectorAll('[data-fold-all]').forEach(function (b) {
|
|
288
|
+
b.dataset.foldAll = all ? 'close' : 'open';
|
|
289
|
+
b.setAttribute('aria-expanded', String(all));
|
|
290
|
+
b.lastChild.textContent = all ? 'Collapse all' : 'Expand all';
|
|
291
|
+
});
|
|
292
|
+
}
|
|
293
|
+
document.addEventListener('click', function (e) {
|
|
294
|
+
var b = e.target.closest('[data-fold-all]');
|
|
295
|
+
if (!b) return;
|
|
296
|
+
var tab = b.closest('.tab');
|
|
297
|
+
tab.querySelectorAll(FOLDS).forEach(function (d) { d.open = b.dataset.foldAll === 'open'; });
|
|
298
|
+
foldButtons(tab);
|
|
299
|
+
});
|
|
300
|
+
document.addEventListener('toggle', function (e) { // toggle does not bubble: caught on its way down
|
|
301
|
+
if (e.target.matches && e.target.matches(FOLDS)) foldButtons(e.target.closest('.tab'));
|
|
302
|
+
}, true);
|
|
303
|
+
|
|
304
|
+
// ---- Logs: Export as text (every log in one .txt, to read) and CSV (one log, its
|
|
305
|
+
// own columns, for a spreadsheet). Both as the page shows them, the filter applied.
|
|
306
|
+
function stampNow() { // local time, as the page shows
|
|
307
|
+
var now = new Date(), two = function (n) { return String(n).padStart(2, '0'); };
|
|
308
|
+
return now.getFullYear() + '-' + two(now.getMonth() + 1) + '-' + two(now.getDate()) + '-' + two(now.getHours()) + two(now.getMinutes());
|
|
309
|
+
}
|
|
310
|
+
// A log as the page shows it (the filter applied): its header and rows, as cells.
|
|
311
|
+
// A log of lines (Scheduler, Onboarding runs) is one column, Line.
|
|
312
|
+
function logRows(d) {
|
|
313
|
+
var cells = function (tr) {
|
|
314
|
+
return Array.prototype.map.call(tr.children, function (c) { return c.textContent.trim().replace(/\s+/g, ' '); });
|
|
315
|
+
};
|
|
316
|
+
var head = d.querySelector('thead tr'), rows = [head ? cells(head) : ['Line']];
|
|
317
|
+
d.querySelectorAll(head ? 'tbody tr' : '.log-line').forEach(function (r) {
|
|
318
|
+
if (!r.hidden) rows.push(head ? cells(r) : [r.textContent.trim()]);
|
|
319
|
+
});
|
|
320
|
+
return rows;
|
|
321
|
+
}
|
|
322
|
+
// Export…: the popup picks the format and the logs. Text: every log chosen in one
|
|
323
|
+
// .txt to read, tab-separated. CSV: a .csv per log (its own columns), through Share's
|
|
324
|
+
// saveFile(blob, name) and csv(rows) (RFC 4180, with a BOM).
|
|
325
|
+
on('click', 'logs-export', function () {
|
|
326
|
+
var q = ($('tab-logs').querySelector('input[data-filter]') || {}).value;
|
|
327
|
+
$('lx-note').textContent = q ? 'Only the lines that match the filter, "' + q + '", as the page shows them.'
|
|
328
|
+
: 'Every line the page shows: the last 200 of each log.';
|
|
329
|
+
$('lx-errors').textContent = '';
|
|
330
|
+
$('logs-export-dialog').showModal();
|
|
331
|
+
});
|
|
332
|
+
on('click', 'lx-cancel', function () { $('logs-export-dialog').close(); });
|
|
333
|
+
on('click', 'lx-go', function () {
|
|
334
|
+
var tab = $('tab-logs'), dlg = $('logs-export-dialog');
|
|
335
|
+
var format = dlg.querySelector('input[name=lx-format]:checked').value;
|
|
336
|
+
var want = Array.prototype.filter.call(dlg.querySelectorAll('input[name=lx-log]'), function (x) { return x.checked; })
|
|
337
|
+
.map(function (x) { return x.value; });
|
|
338
|
+
if (!want.length) { $('lx-errors').textContent = 'Choose at least one log.'; return; }
|
|
339
|
+
var logs = Array.prototype.filter.call(tab.querySelectorAll('details[data-log]'), function (d) {
|
|
340
|
+
return want.indexOf(d.dataset.log) !== -1;
|
|
341
|
+
});
|
|
342
|
+
var stamp = stampNow(), q = (tab.querySelector('input[data-filter]') || {}).value;
|
|
343
|
+
if (format === 'csv') {
|
|
344
|
+
logs.forEach(function (d) {
|
|
345
|
+
var slug = d.dataset.log.toLowerCase().replace(/[^a-z0-9]+/g, '-');
|
|
346
|
+
saveFile(new Blob([csv(logRows(d))], {type: 'text/csv;charset=utf-8'}), 'benchwarmers-' + slug + '-' + stamp + '.csv');
|
|
347
|
+
});
|
|
348
|
+
} else {
|
|
349
|
+
var out = ['Benchwarmers logs, exported ' + new Date().toString()];
|
|
350
|
+
if (q) out.push('Filtered by: ' + q);
|
|
351
|
+
logs.forEach(function (d) {
|
|
352
|
+
out.push('', '== ' + d.dataset.log + ' ==');
|
|
353
|
+
logRows(d).forEach(function (r) { out.push(r.join('\t')); });
|
|
354
|
+
});
|
|
355
|
+
saveFile(new Blob([out.join('\n') + '\n'], {type: 'text/plain;charset=utf-8'}), 'benchwarmers-logs-' + stamp + '.txt');
|
|
356
|
+
}
|
|
357
|
+
dlg.close();
|
|
358
|
+
});
|
|
359
|
+
|
|
360
|
+
// ---- a link to another page ----
|
|
280
361
|
document.addEventListener('click', function (e) {
|
|
281
362
|
var b = e.target.closest('[data-go]');
|
|
282
363
|
if (b) { show(b.dataset.go); scrollTo(0, 0); }
|
|
@@ -767,8 +848,8 @@
|
|
|
767
848
|
});
|
|
768
849
|
|
|
769
850
|
// ---- after an action: the part it changed, not the whole page ----------
|
|
770
|
-
// Action needed's page
|
|
771
|
-
//
|
|
851
|
+
// Action needed's page and its count in the sidebar, from the server
|
|
852
|
+
// (GET /api/needs, a few ms): a page reload
|
|
772
853
|
// renders every tab, seconds.
|
|
773
854
|
var needsAsked = 0;
|
|
774
855
|
function refreshNeeds(live) {
|
|
@@ -783,10 +864,6 @@
|
|
|
783
864
|
count.textContent = n.count;
|
|
784
865
|
tab.setAttribute('aria-label', tab.firstChild.textContent + ', ' + n.count + ' open');
|
|
785
866
|
}
|
|
786
|
-
var line = document.querySelector('#tab-glance > .a-line');
|
|
787
|
-
if (line) line.remove();
|
|
788
|
-
// Not into a page not fetched yet: it says Loading… while empty, and comes with its line.
|
|
789
|
-
if (n.line && $('tab-glance').firstChild) $('tab-glance').insertAdjacentHTML('afterbegin', n.line);
|
|
790
867
|
}).catch(function () { location.reload(); }); // could not ask: the old way
|
|
791
868
|
}
|
|
792
869
|
// A retired model's rows grey out and hide behind Show retired, as the
|
|
@@ -2125,7 +2202,10 @@
|
|
|
2125
2202
|
var first = fact(g.name), caps = first.caps || {};
|
|
2126
2203
|
var line = el('div', {class: 'm-row', role: 'group', 'aria-label': g.name, 'data-family': g.family, 'data-name': g.name});
|
|
2127
2204
|
var name = el('div', {class: 'm-name'});
|
|
2128
|
-
|
|
2205
|
+
// The same link as every model name elsewhere (a.model > .alias: the highlight on hover).
|
|
2206
|
+
var link = el('a', {href: '#', class: 'model', 'data-alias': g.name, title: 'Open ' + g.name});
|
|
2207
|
+
link.appendChild(el('span', {class: 'alias'}, g.name));
|
|
2208
|
+
name.appendChild(link);
|
|
2129
2209
|
line.appendChild(name);
|
|
2130
2210
|
line.appendChild(el('span', {class: 'm-ctx'}, first.context ? tokens(first.context) : '—'));
|
|
2131
2211
|
var can = el('span', {class: 'm-can'});
|
|
@@ -2683,6 +2763,9 @@
|
|
|
2683
2763
|
// the pages share. It was the whole page, which made every chart wider than
|
|
2684
2764
|
// its column beside the sidebar, scaled down, with smaller text.
|
|
2685
2765
|
function width() {
|
|
2766
|
+
// Benchmarks shown with its rail beside the reports: a report's own width, not the column's.
|
|
2767
|
+
var report = document.querySelector('.tab.active .panel.report');
|
|
2768
|
+
if (report && report.clientWidth) return Math.max(520, report.clientWidth - 50);
|
|
2686
2769
|
var col = document.querySelector('.panes') || document.querySelector('main');
|
|
2687
2770
|
return Math.max(520, (col.clientWidth || 900) - 50);
|
|
2688
2771
|
}
|
|
@@ -2874,8 +2957,14 @@
|
|
|
2874
2957
|
]
|
|
2875
2958
|
});
|
|
2876
2959
|
};
|
|
2877
|
-
|
|
2878
|
-
|
|
2960
|
+
// One view at a time, as the card's By says: by provider (the default) or by model.
|
|
2961
|
+
var view = (document.querySelector('select[data-uptime-by]') || {}).value || 'providers';
|
|
2962
|
+
['providers', 'models'].forEach(function (k) {
|
|
2963
|
+
var box = $('chart-uptime-' + k);
|
|
2964
|
+
if (!box) return;
|
|
2965
|
+
box.hidden = k !== view;
|
|
2966
|
+
if (k === view) put('chart-uptime-' + k, function (w) { return uptime(U[k], w); });
|
|
2967
|
+
});
|
|
2879
2968
|
}
|
|
2880
2969
|
if (part === 'uptime') return;
|
|
2881
2970
|
// Decision models: each model's answers grouped by the probability it gave
|
|
@@ -3198,41 +3287,59 @@
|
|
|
3198
3287
|
});
|
|
3199
3288
|
});
|
|
3200
3289
|
|
|
3201
|
-
// Vision, real work:
|
|
3290
|
+
// Vision, real work: read as Real work is. The one question: when its seat was judged,
|
|
3291
|
+
// did it help? The dot is the share judged useful (checked and helped, or useful with no
|
|
3292
|
+
// claims to check), sized by how many were judged; the line where that share likely sits
|
|
3293
|
+
// (95% Wilson), ranked by its low end. Beside it, in words: how often it answered at all,
|
|
3294
|
+
// and the claims checked by hand. Seats never judged or never answered say nothing of
|
|
3295
|
+
// its worth: they are in the words and the table, not the dot.
|
|
3202
3296
|
put('chart-vision-work', function (w) {
|
|
3203
|
-
var W = C.vision.work, kinds = C.vision.seatKinds,
|
|
3297
|
+
var W = C.vision.work, kinds = C.vision.seatKinds, R = [], none = [];
|
|
3204
3298
|
Object.keys(W).forEach(function (m) {
|
|
3205
|
-
|
|
3206
|
-
|
|
3207
|
-
|
|
3208
|
-
|
|
3209
|
-
|
|
3210
|
-
|
|
3211
|
-
|
|
3212
|
-
|
|
3213
|
-
|
|
3214
|
-
|
|
3215
|
-
|
|
3216
|
-
|
|
3217
|
-
// The claims checked by hand, beside the bar: the only right-or-wrong count real work has.
|
|
3218
|
-
var claims = models.filter(function (m) { return W[m].real + W[m].false; }).map(function (m) {
|
|
3219
|
-
return {model: m, text: W[m].real + ' of ' + (W[m].real + W[m].false) + ' claims right'};
|
|
3299
|
+
var v = W[m], n = function (k) { return v[k] || 0; }, total = 0;
|
|
3300
|
+
kinds.forEach(function (k) { total += n(k); });
|
|
3301
|
+
if (!total) return;
|
|
3302
|
+
var judged = n('checked, helped') + n('checked, did not help') + n('useful, no claims') + n('no use, no claims');
|
|
3303
|
+
var useful = n('checked, helped') + n('useful, no claims');
|
|
3304
|
+
var r = {model: m, total: total, judged: judged, useful: useful, answered: total - n('no answer'),
|
|
3305
|
+
real: v.real || 0, wrongC: v.false || 0};
|
|
3306
|
+
if (!judged) { none.push(r); return; }
|
|
3307
|
+
var b = wilson(useful, judged);
|
|
3308
|
+
r.p = useful / judged; r.lo = b[0]; r.hi = b[1];
|
|
3309
|
+
r.verdict = b[0] >= 0.5 ? 'helps' : b[1] < 0.5 ? 'does not help' : 'too few to say';
|
|
3310
|
+
R.push(r);
|
|
3220
3311
|
});
|
|
3312
|
+
if (!R.length && !none.length) return el('p', {class: 'hint'}, 'No vision panels through the gateway yet.');
|
|
3313
|
+
R.sort(function (a, b) { return b.lo - a.lo || b.judged - a.judged; });
|
|
3221
3314
|
var narrowV = w < 560, box = el('div');
|
|
3222
|
-
|
|
3223
|
-
|
|
3224
|
-
|
|
3225
|
-
|
|
3226
|
-
|
|
3315
|
+
function words(r) {
|
|
3316
|
+
return r.useful + '/' + r.judged + (narrowV ? '' : ' · ' + Math.round(r.answered / r.total * 100) + '% answered'
|
|
3317
|
+
+ (r.real + r.wrongC ? ' · claims ' + r.real + '/' + (r.real + r.wrongC) : ''));
|
|
3318
|
+
}
|
|
3319
|
+
if (R.length) box.appendChild(Plot.plot({
|
|
3320
|
+
width: w, height: 24 * R.length + 90, style: style, marginLeft: narrowV ? 130 : 190,
|
|
3321
|
+
marginRight: narrowV ? 50 : 260, marginTop: 34,
|
|
3322
|
+
x: {domain: [0, 1], label: 'Judged useful, share of its judged seats →', tickFormat: '.0%', ticks: 5, grid: true},
|
|
3323
|
+
y: {domain: R.map(function (r) { return r.model; }), label: null,
|
|
3324
|
+
tickFormat: function (m) { return fit(m, narrowV ? 16 : 26); }},
|
|
3325
|
+
r: {domain: [0, Math.max.apply(null, R.map(function (r) { return r.judged; }))], range: [3, 11]},
|
|
3326
|
+
color: {domain: ['helps', 'too few to say', 'does not help'], range: [good, muted, err], legend: true},
|
|
3227
3327
|
marks: [
|
|
3228
|
-
Plot.
|
|
3229
|
-
|
|
3230
|
-
Plot.
|
|
3231
|
-
|
|
3328
|
+
Plot.ruleX([0.5], {stroke: line}),
|
|
3329
|
+
Plot.ruleY(R, {y: 'model', x1: 'lo', x2: 'hi', stroke: 'verdict', strokeWidth: 2, strokeOpacity: 0.55}),
|
|
3330
|
+
Plot.dot(R, {x: 'p', y: 'model', r: 'judged', fill: 'verdict', stroke: halo, strokeWidth: 1.5,
|
|
3331
|
+
title: function (r) {
|
|
3332
|
+
return r.model + '\njudged useful ' + r.useful + ' of ' + r.judged + ' judged seats ('
|
|
3333
|
+
+ Math.round(r.p * 100) + '%), likely between ' + Math.round(r.lo * 100) + '% and ' + Math.round(r.hi * 100)
|
|
3334
|
+
+ '%\nanswered ' + r.answered + ' of ' + r.total + ' seats on image panels'
|
|
3335
|
+
+ (r.real + r.wrongC ? '\nclaims checked by hand: ' + r.real + ' right of ' + (r.real + r.wrongC) : '');
|
|
3336
|
+
}}),
|
|
3337
|
+
Plot.text(R, {x: 1, y: 'model', dx: 10, textAnchor: 'start', fill: ink, fontSize: 11, text: words})
|
|
3232
3338
|
]
|
|
3233
3339
|
}));
|
|
3234
|
-
if (
|
|
3235
|
-
+
|
|
3340
|
+
if (none.length) box.appendChild(el('p', {class: 'hint'}, 'No seat judged yet, so no dot: '
|
|
3341
|
+
+ none.map(function (r) { return r.model + ' (' + r.total + ' seats, answered ' + r.answered + ')'; }).join(', ')
|
|
3342
|
+
+ '. The table has every seat.'));
|
|
3236
3343
|
return box;
|
|
3237
3344
|
});
|
|
3238
3345
|
|
|
@@ -3271,10 +3378,50 @@
|
|
|
3271
3378
|
});
|
|
3272
3379
|
|
|
3273
3380
|
}
|
|
3274
|
-
onLoad.suite = function () { drawCharts('suite'); };
|
|
3381
|
+
onLoad.suite = function () { drawCharts('suite'); jumpSpy(); };
|
|
3382
|
+
// Benchmarks' jump bar: the report in view is marked (aria-current) and its link kept in
|
|
3383
|
+
// sight in a bar that scrolls sideways. Set up again after a live swap (setUp, onLoad).
|
|
3384
|
+
var spy = null;
|
|
3385
|
+
function jumpSpy() {
|
|
3386
|
+
if (spy) spy.disconnect();
|
|
3387
|
+
var nav = document.querySelector('#tab-suite .jump');
|
|
3388
|
+
if (!nav || !window.IntersectionObserver) return;
|
|
3389
|
+
var links = {}, seen = {}, current = null;
|
|
3390
|
+
nav.querySelectorAll('a').forEach(function (a) { links[a.getAttribute('href').slice(1)] = a; });
|
|
3391
|
+
// A report counts as in view while it crosses a thin band 15-20% down the window: just
|
|
3392
|
+
// under where a jump lands it, so the one jumped to is the one marked.
|
|
3393
|
+
spy = new IntersectionObserver(function (entries) {
|
|
3394
|
+
entries.forEach(function (e) { seen[e.target.id] = e.isIntersecting; });
|
|
3395
|
+
var id = Object.keys(links).filter(function (k) { return seen[k]; })[0];
|
|
3396
|
+
if (!id || id === current) return;
|
|
3397
|
+
if (current) links[current].removeAttribute('aria-current');
|
|
3398
|
+
current = id;
|
|
3399
|
+
links[id].setAttribute('aria-current', 'true');
|
|
3400
|
+
var a = links[id];
|
|
3401
|
+
if (a.offsetLeft < nav.scrollLeft || a.offsetLeft + a.offsetWidth > nav.scrollLeft + nav.clientWidth) {
|
|
3402
|
+
nav.scrollLeft = a.offsetLeft - 16;
|
|
3403
|
+
}
|
|
3404
|
+
}, {rootMargin: '-15% 0px -80% 0px'});
|
|
3405
|
+
document.querySelectorAll('#tab-suite .panel.report').forEach(function (p) { spy.observe(p); });
|
|
3406
|
+
}
|
|
3407
|
+
// A jump scrolls smoothly (unless motion is reduced) and moves focus to the report, so
|
|
3408
|
+
// a keyboard user carries on from there; the address keeps the report's #, as a link would.
|
|
3409
|
+
document.addEventListener('click', function (e) {
|
|
3410
|
+
var a = e.target.closest('.jump a');
|
|
3411
|
+
if (!a) return;
|
|
3412
|
+
var target = $(a.getAttribute('href').slice(1));
|
|
3413
|
+
if (!target) return;
|
|
3414
|
+
e.preventDefault();
|
|
3415
|
+
var still = window.matchMedia && matchMedia('(prefers-reduced-motion: reduce)').matches;
|
|
3416
|
+
target.scrollIntoView({behavior: still ? 'auto' : 'smooth', block: 'start'});
|
|
3417
|
+
target.setAttribute('tabindex', '-1');
|
|
3418
|
+
target.focus({preventScroll: true});
|
|
3419
|
+
history.replaceState(null, '', a.getAttribute('href'));
|
|
3420
|
+
});
|
|
3275
3421
|
// Uptime's or Trend's window: that window's tables (and Uptime's list) shown, its charts drawn again.
|
|
3276
3422
|
document.addEventListener('change', function (ev) {
|
|
3277
3423
|
var s = ev.target, part = s.matches('select[data-uptime-days]') ? 'uptime' : s.matches('select[data-trend-days]') ? 'trend' : '';
|
|
3424
|
+
if (s.matches('select[data-uptime-by]')) { drawCharts('uptime'); return; } // the other view, same window
|
|
3278
3425
|
if (!part) return;
|
|
3279
3426
|
s.closest('.panel').querySelectorAll('[data-' + part + '-window]').forEach(function (d) {
|
|
3280
3427
|
d.hidden = d.getAttribute('data-' + part + '-window') !== s.value;
|