@wendellresearch-ai/benchwarmers 0.2.0 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. package/AGENTS.md +152 -147
  2. package/BENCHMARKS.md +262 -262
  3. package/GATEWAY.md +415 -272
  4. package/NOTICE +28 -28
  5. package/PRIVACY.md +46 -46
  6. package/README.md +408 -314
  7. package/SECURITY.md +83 -83
  8. package/TERMS.md +22 -22
  9. package/USAGE.md +186 -163
  10. package/VERSION +1 -1
  11. package/bin/benchwarmers.js +16 -74
  12. package/cli.py +674 -0
  13. package/examples/config/gateway.json +5 -5
  14. package/examples/config/model-roles.json +11 -11
  15. package/gateway.py +939 -755
  16. package/launcher.py +120 -0
  17. package/mcp_server.py +266 -266
  18. package/metrics/alerts.py +116 -116
  19. package/metrics/aliases.py +41 -41
  20. package/metrics/bench_collections.py +141 -141
  21. package/metrics/catalog.py +304 -295
  22. package/metrics/client.py +804 -394
  23. package/metrics/cliproxy.py +191 -191
  24. package/metrics/connect.py +691 -691
  25. package/metrics/cost.py +171 -152
  26. package/metrics/decide.py +227 -227
  27. package/metrics/discover.py +318 -318
  28. package/metrics/endpoints.py +436 -436
  29. package/metrics/formats.py +1021 -0
  30. package/metrics/guard.py +647 -592
  31. package/metrics/learn.py +234 -221
  32. package/metrics/ledger.py +148 -0
  33. package/metrics/litellm_hub.py +206 -206
  34. package/metrics/model_roles.py +94 -84
  35. package/metrics/offload.py +166 -166
  36. package/metrics/owui.py +205 -205
  37. package/metrics/pacing.py +248 -0
  38. package/metrics/paths.py +60 -60
  39. package/metrics/quota.py +383 -0
  40. package/metrics/route.py +581 -579
  41. package/metrics/scheduler.py +275 -275
  42. package/metrics/sources.py +167 -167
  43. package/metrics/wrong.py +144 -144
  44. package/package.json +69 -65
  45. package/perf/README.md +89 -89
  46. package/perf/run.py +284 -261
  47. package/scripts/auto_retire.py +367 -367
  48. package/scripts/check.py +113 -106
  49. package/scripts/check_deprecations.py +378 -378
  50. package/scripts/check_themes.py +103 -103
  51. package/scripts/init.py +124 -124
  52. package/scripts/list_served.py +78 -78
  53. package/scripts/notice.py +78 -78
  54. package/scripts/onboard.py +410 -331
  55. package/scripts/provider_tags.py +129 -129
  56. package/scripts/reprefix.py +136 -136
  57. package/scripts/sync_config.py +105 -105
  58. package/scripts/test_auth.py +255 -255
  59. package/scripts/test_cli.py +189 -0
  60. package/scripts/test_client.py +277 -233
  61. package/scripts/test_connect.py +222 -222
  62. package/scripts/test_first_run.py +180 -180
  63. package/scripts/test_formats.py +549 -0
  64. package/scripts/test_hubs.py +328 -328
  65. package/scripts/test_launcher.py +67 -0
  66. package/scripts/test_llm_log.py +85 -81
  67. package/scripts/test_mcp.py +178 -178
  68. package/scripts/test_merged.py +215 -214
  69. package/scripts/test_no_owui.py +291 -291
  70. package/scripts/test_owui.py +90 -90
  71. package/scripts/test_quota.py +334 -0
  72. package/scripts/test_serve_edit.py +453 -438
  73. package/scripts/test_server_json.py +79 -0
  74. package/scripts/test_verdict.py +102 -98
  75. package/scripts/test_vision.py +231 -228
  76. package/serve.py +3972 -3764
  77. package/suite/mine.py +154 -154
  78. package/suite/pace.py +145 -145
  79. package/suite/run.py +1046 -1042
  80. package/suite/seed.py +421 -421
  81. package/suite/tasks/claim-alias-unbound/fixtures/onboard.py +286 -286
  82. package/suite/tasks/claim-error-text-string/fixtures/decide.py +187 -187
  83. package/suite/tasks/claim-list-body/fixtures/gateway.py +235 -235
  84. package/suite/tasks/claim-member-nominated-twice/fixtures/gateway.py +284 -284
  85. package/suite/tasks/claim-nominee-uses-slot/fixtures/gateway.py +284 -284
  86. package/suite/tasks/claim-silent-exit/fixtures/gateway.py +235 -235
  87. package/suite/tasks/claim-sse-hang/fixtures/gateway.py +235 -235
  88. package/suite/tasks/claim-unknown-nominee-crash/fixtures/gateway.py +284 -284
  89. package/suite/tasks/rebench-matchms-809-bug/task.json +23 -23
  90. package/suite/tasks/rebench-matchms-809-control/task.json +27 -27
  91. package/suite/tasks/rebench-more-itertools-1028-bug/task.json +23 -23
  92. package/suite/tasks/rebench-more-itertools-1028-control/task.json +27 -27
  93. package/suite/tasks/rebench-narwhals-2864-bug/task.json +23 -23
  94. package/suite/tasks/rebench-narwhals-2864-control/task.json +27 -27
  95. package/suite/tasks/rebench-quantecon-py-769-bug/task.json +23 -23
  96. package/suite/tasks/rebench-quantecon-py-769-control/task.json +27 -27
  97. package/suite/tasks/rebench-scrapy-6867-bug/task.json +23 -23
  98. package/suite/tasks/rebench-scrapy-6867-control/task.json +27 -27
  99. package/ui/bench.css +430 -425
  100. package/ui/bench.js +1757 -1576
  101. package/ui/themes/dark.css +35 -35
  102. package/ui/themes/hardwood.css +52 -52
  103. package/ui/themes/rainforest.css +32 -32
package/NOTICE CHANGED
@@ -1,28 +1,28 @@
1
- Benchwarmers, by Wendell Research
2
- Copyright (c) 2026 Wendell Research. Licensed under the MIT License (LICENSE).
3
-
4
- This repository also contains material from others, under their own licences:
5
-
6
- Real-bug tasks (suite/tasks/rebench-*) are drawn from the nebius/SWE-rebench-V2 dataset
7
- on Hugging Face (CC-BY-4.0). Each task holds code from the project it names,
8
- under that project's licence:
9
-
10
- matchms/matchms (Apache-2.0)
11
- https://github.com/matchms/matchms/pull/809
12
- more-itertools/more-itertools (MIT)
13
- https://github.com/more-itertools/more-itertools/pull/1028
14
- narwhals-dev/narwhals (MIT)
15
- https://github.com/narwhals-dev/narwhals/pull/2864
16
- QuantEcon/QuantEcon.py (MIT)
17
- https://github.com/QuantEcon/QuantEcon.py/pull/769
18
- scrapy/scrapy (BSD-3-Clause)
19
- https://github.com/scrapy/scrapy/pull/6867
20
-
21
- Fonts (ui/fonts): Chivo and Chivo Mono, Copyright The Chivo Project Authors
22
- (https://github.com/Omnibus-Type/Chivo), under the SIL Open Font License 1.1
23
- (ui/fonts/OFL-*.txt).
24
-
25
- The Wendell Research mascot (ui/mascot.png, drawn at centre court in the
26
- Hardwood theme) is a registered trademark of Wendell Research. It is not
27
- covered by the MIT License: a fork may keep it in place unchanged, but should
28
- not use it to name or brand another product.
1
+ Benchwarmers, by Wendell Research
2
+ Copyright (c) 2026 Wendell Research. Licensed under the MIT License (LICENSE).
3
+
4
+ This repository also contains material from others, under their own licences:
5
+
6
+ Real-bug tasks (suite/tasks/rebench-*) are drawn from the nebius/SWE-rebench-V2 dataset
7
+ on Hugging Face (CC-BY-4.0). Each task holds code from the project it names,
8
+ under that project's licence:
9
+
10
+ matchms/matchms (Apache-2.0)
11
+ https://github.com/matchms/matchms/pull/809
12
+ more-itertools/more-itertools (MIT)
13
+ https://github.com/more-itertools/more-itertools/pull/1028
14
+ narwhals-dev/narwhals (MIT)
15
+ https://github.com/narwhals-dev/narwhals/pull/2864
16
+ QuantEcon/QuantEcon.py (MIT)
17
+ https://github.com/QuantEcon/QuantEcon.py/pull/769
18
+ scrapy/scrapy (BSD-3-Clause)
19
+ https://github.com/scrapy/scrapy/pull/6867
20
+
21
+ Fonts (ui/fonts): Chivo and Chivo Mono, Copyright The Chivo Project Authors
22
+ (https://github.com/Omnibus-Type/Chivo), under the SIL Open Font License 1.1
23
+ (ui/fonts/OFL-*.txt).
24
+
25
+ The Wendell Research mascot (ui/mascot.png, drawn at centre court in the
26
+ Hardwood theme) is a registered trademark of Wendell Research. It is not
27
+ covered by the MIT License: a fork may keep it in place unchanged, but should
28
+ not use it to name or brand another product.
package/PRIVACY.md CHANGED
@@ -1,46 +1,46 @@
1
- # Privacy policy
2
-
3
- *Effective 22 October 2026.*
4
-
5
- Benchwarmers runs on your own machine or server. It has no account, no
6
- telemetry and no server of its own, and nothing you do with it reaches
7
- Wendell Research.
8
-
9
- ## What leaves your machine
10
-
11
- - **Your prompts**, only to the endpoints you add
12
- (a provider, an aggregator, a hub such as Open WebUI, or a local runtime).
13
- That includes what the gateway relays for your apps, and the calls the
14
- probes, the suite and onboarding make. Each endpoint's own privacy policy
15
- applies to what it receives; some free tiers keep prompts or train on them.
16
- - **Catalog reads**, carrying nothing of yours: models.dev and FreeLLMAPI
17
- (at most once a day) and OpenRouter's public model list.
18
- - **Sign-ins**, only when you choose one: the provider's own sign-in page, in
19
- your browser, and the provider's token endpoint.
20
- - **Charts**: the dashboard's page loads Observable Plot from jsDelivr, at a
21
- pinned version, in your browser.
22
- - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows
23
- from Hugging Face and source files from GitHub.
24
-
25
- Keys and sign-in tokens stay in your environment or the `llm` CLI's key
26
- store. [SECURITY.md](SECURITY.md) has the full detail.
27
-
28
- ## What stays on your machine
29
-
30
- Everything else: your endpoints, collections, verdicts, call logs, probe
31
- history and task answers live in your data folder, which is yours to back up,
32
- move or delete.
33
-
34
- ## Third parties
35
-
36
- The endpoints you add, the catalogs above, jsDelivr, GitHub, Hugging Face and
37
- any app you connect to the gateway each have their own privacy policies.
38
-
39
- ## Changes
40
-
41
- Changes to this policy are made in this file, and its history is in the
42
- repository's commits.
43
-
44
- ## Contact
45
-
46
- Questions: [open an issue](https://github.com/WendellResearch-AI/benchwarmers/issues).
1
+ # Privacy policy
2
+
3
+ *Effective 22 October 2026.*
4
+
5
+ Benchwarmers runs on your own machine or server. It has no account, no
6
+ telemetry and no server of its own, and nothing you do with it reaches
7
+ Wendell Research.
8
+
9
+ ## What leaves your machine
10
+
11
+ - **Your prompts**, only to the endpoints you add
12
+ (a provider, an aggregator, a hub such as Open WebUI, or a local runtime).
13
+ That includes what the gateway relays for your apps, and the calls the
14
+ probes, the suite and onboarding make. Each endpoint's own privacy policy
15
+ applies to what it receives; some free tiers keep prompts or train on them.
16
+ - **Catalog reads**, carrying nothing of yours: models.dev and FreeLLMAPI
17
+ (at most once a day) and OpenRouter's public model list.
18
+ - **Sign-ins**, only when you choose one: the provider's own sign-in page, in
19
+ your browser, and the provider's token endpoint.
20
+ - **Charts**: the dashboard's page loads Observable Plot from jsDelivr, at a
21
+ pinned version, in your browser.
22
+ - **Real-bug task seeding**, only when you run `suite/rebench.py`: dataset rows
23
+ from Hugging Face and source files from GitHub.
24
+
25
+ Keys and sign-in tokens stay in your environment or the `llm` CLI's key
26
+ store. [SECURITY.md](SECURITY.md) has the full detail.
27
+
28
+ ## What stays on your machine
29
+
30
+ Everything else: your endpoints, collections, verdicts, call logs, probe
31
+ history and task answers live in your data folder, which is yours to back up,
32
+ move or delete.
33
+
34
+ ## Third parties
35
+
36
+ The endpoints you add, the catalogs above, jsDelivr, GitHub, Hugging Face and
37
+ any app you connect to the gateway each have their own privacy policies.
38
+
39
+ ## Changes
40
+
41
+ Changes to this policy are made in this file, and its history is in the
42
+ repository's commits.
43
+
44
+ ## Contact
45
+
46
+ Questions: [open an issue](https://github.com/WendellResearch-AI/benchwarmers/issues).