opencode-models 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,544 @@
1
+ Metadata-Version: 2.5
2
+ Name: opencode-models
3
+ Version: 0.1.0
4
+ Summary: Ranked console of AI models: benchmark scores against real per-token prices
5
+ Project-URL: Homepage, https://github.com/codesam88/opencode-models
6
+ Project-URL: Repository, https://github.com/codesam88/opencode-models
7
+ Project-URL: Issues, https://github.com/codesam88/opencode-models/issues
8
+ Author-email: codesam <tedsamhain@gmail.com>
9
+ License: MIT License
10
+
11
+ Copyright (c) 2026 codesam <tedsamhain@gmail.com>
12
+
13
+ Permission is hereby granted, free of charge, to any person obtaining a copy
14
+ of this software and associated documentation files (the "Software"), to deal
15
+ in the Software without restriction, including without limitation the rights
16
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
17
+ copies of the Software, and to permit persons to whom the Software is
18
+ furnished to do so, subject to the following conditions:
19
+
20
+ The above copyright notice and this permission notice shall be included in all
21
+ copies or substantial portions of the Software.
22
+
23
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
24
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
25
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
26
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
27
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
28
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
29
+ SOFTWARE.
30
+ License-File: LICENSE
31
+ Keywords: benchmarks,cli,llm,pricing,ranking
32
+ Classifier: Environment :: Console
33
+ Classifier: Intended Audience :: Developers
34
+ Classifier: License :: OSI Approved :: MIT License
35
+ Classifier: Programming Language :: Python :: 3 :: Only
36
+ Classifier: Programming Language :: Python :: 3.10
37
+ Classifier: Topic :: Utilities
38
+ Requires-Python: >=3.10
39
+ Description-Content-Type: text/markdown
40
+
41
+ # opencode-models
42
+
43
+ List [Opencode Zen / Go](https://opencode.ai/data) models by agentic coding score
44
+ and score per dollar.
45
+
46
+ Tired of trying to correlate the latest OpenCode model listings with available
47
+ benchmarks and costs? This utility uses public/scraped benchmark reports and
48
+ the opencode API to show the current best and cheapest available models.
49
+
50
+ ## TLDR; quick steps
51
+
52
+ ```sh
53
+ uvx opencode-models
54
+ alias ocm=opencode-models
55
+
56
+ ocm auth # register an openapi token to always fetch the current 'live' models
57
+ ocm fetch # fetch benchmarks and model info from supported sources
58
+ ocm list # List non-expensive, non-trivial models, sorted by score/$
59
+ ocm next # 6 next-best recommended models to use (tell your agent to use this)
60
+ ```
61
+
62
+ ## Install
63
+
64
+ Run it once with `uvx`, or install the `opencode-models` binary to your
65
+ `$PATH`:
66
+
67
+ ```sh
68
+ uvx opencode-models # no install, just run it
69
+ uv tool install opencode-models # install the binary
70
+ ```
71
+
72
+ The name is longer than it wants to be for daily typing; a shell alias
73
+ fixes that in one line: `alias ocm=opencode-models`.
74
+
75
+ ### First run
76
+
77
+ ```sh
78
+ ocm auth # save a Console token so the views are live (recommended)
79
+ ocm fetch # download benchmark data and model info (~1 minute first time)
80
+ ```
81
+
82
+ `auth` stores an OpenCode Console service-account key at
83
+ `~/.config/opencode-models/api_key` (owner-only, never echoed); the
84
+ `OPENCODE_API_KEY` environment variable takes precedence if you set one.
85
+ `fetch` is the only command that downloads data — benchmark numbers,
86
+ tier prices, official boards — into your data directory:
87
+ `$OPENCODE_MODELS_DATA` if set, else `~/.local/share/opencode-models`.
88
+
89
+ ### Working from a checkout
90
+
91
+ ```sh
92
+ just setup # uv sync + enable git hooks (pre-commit lint/audit,
93
+ # pre-push guard for the local-data branch)
94
+ just test # unittest suite
95
+ just lint # ruff lint, format check, complexity gate
96
+ just check # tests + entry point + wheel build
97
+ just run <cmd> # the console against ./data, e.g. just run next
98
+ ```
99
+
100
+ `just install` additionally seeds the reference dataset into your user
101
+ data dir (missing files only — it never overwrites fetched results, while
102
+ tracked curation deploys as shipped). The tool is stdlib-only: Python
103
+ ≥ 3.10, no runtime dependencies; ruff comes from the locked dev group.
104
+
105
+ Project pointers: [SECURITY.md](SECURITY.md) (what touches the network,
106
+ scraper etiquette, how to report a problem), [LIMITS.md](LIMITS.md) (what
107
+ the plans actually meter), [CONTRIBUTING.md](CONTRIBUTING.md) (workflow,
108
+ releasing).
109
+
110
+ ## Usage
111
+
112
+ Three commands cover daily use; four more exist for auditing and
113
+ maintenance:
114
+
115
+ | command | what it shows |
116
+ |---|---|
117
+ | `list` | the best-value table of what the plans offer today — free, included-usage and pay-as-you-go bands ranked by score per dollar |
118
+ | `next` | the next models to reach for: free and paid picks interleaved, Zen/Go alternating (default 6; tell your agent to use these) |
119
+ | `scores` | every offered model, one column per benchmark — the raw number beside where it lands |
120
+ | `models` | the API endpoint and display name of every model each plan currently offers |
121
+ | `overrides` | benchmark numbers more than one source reports, and which value the tool keeps |
122
+ | `fetch` | refresh everything: benchmarks, tier prices, official boards, availability |
123
+ | `auth` | store the Console token for live views (owner-only file) |
124
+
125
+ ### Live views and the token
126
+
127
+ Every `list` and `next` run looks for a Console token first — the
128
+ `OPENCODE_API_KEY` environment variable, then the file `auth` wrote. **With
129
+ a token the view is live:** it queries the Console config API on each run,
130
+ shows only the models the plans offer *right now*, and records the result
131
+ so later offline runs stay current. Without a token the first line warns
132
+ that availability may be stale and the last `fetch` snapshots decide
133
+ (those are refreshed by any keyed `fetch` or `models` run).
134
+
135
+ `--all-models` (alias `--all`) bypasses the availability filter entirely
136
+ — what you want when auditing rather than choosing.
137
+
138
+ ### Common options
139
+
140
+ - `--source zen|go|all` — one provider or the full-catalog view
141
+ (default: both providers merged)
142
+ - `--all-models` / `--all` — ignore the availability filter
143
+ (`list`, `next`, `scores`)
144
+ - `--tables all` — add the worst-score and highest-cost tables to `list`
145
+ - `--benchmark NAME` — rank `list` on one benchmark instead of the
146
+ combined score
147
+ - `--count N` — how many picks `next` recommends (default 6)
148
+ - `--price input|output` — cost on a single rate instead of the
149
+ 20/60/20 input-cached-output mix
150
+ - `--data DIR` — point at a different data directory (checkout: `--data data`)
151
+
152
+ ```sh
153
+ ocm list --tables all # also the culled worst/pricey tables
154
+ ocm list --source zen # one provider only
155
+ ocm next --count 8 --source go # eight picks, Go plan only
156
+ ocm scores # one column per tracked benchmark
157
+ ```
158
+
159
+ ## How it works
160
+
161
+ The rest of this document is the detailed semantics: what a score means,
162
+ how the views differ, and where every number comes from.
163
+
164
+ ### Which benchmarks are scored
165
+
166
+ Five benchmarks carry the scores, and they are named one by one in
167
+ `scoring._TRACKED` rather than matched by prefix:
168
+
169
+ | benchmark | tier models | all models |
170
+ |---|---|---|
171
+ | Terminal-Bench 2.1 | 37 | 50 |
172
+ | SWE-Bench Pro | 34 | 57 |
173
+ | Terminal-Bench 4.0 | 27 | 28 |
174
+ | SWE-Bench Verified | 18 | 77 |
175
+ | **DeepSWE 1.1** | **25** | **29** |
176
+
177
+ They are the most relevant and the most broadly available at the same time:
178
+ Terminal-Bench is the closest thing to "can this model finish a task on a
179
+ real machine", and SWE-bench is the closest thing to "does it fix real bugs",
180
+ so together they cover the work this tool ranks on; and between them they are
181
+ measured on enough models to compare anyone to anyone. The two SWE-bench
182
+ subsets are both kept deliberately — Pro is what the tier broadly carries
183
+ (34 models against Verified's 18), and Verified is the canonical metric and by
184
+ far the broadest number in the catalog (77).
185
+
186
+ **DeepSWE 1.1 is the exception, and it earns its place by pairing rather than
187
+ by relevance.** It is the narrowest thing here — 29 models out of 476 — and
188
+ the weakest on provenance: all 29 numbers are vendor self-reports, and no
189
+ board runs it, so it can never set the ends of its own scale. What it does is
190
+ supply the second axis for models that have nothing else: nine of the 25
191
+ offered models sitting on a single benchmark carry it, and admitting it takes
192
+ the scored field from 36 models to 45. `opencode-models scores` names that
193
+ group explicitly — "N on one benchmark" — so a thin row is never presented as
194
+ a ranked one.
195
+
196
+ Its signal is largely redundant with the axis next to it. Rank correlation
197
+ with Terminal-Bench 2.1, over the models carrying both, is **0.87**: it
198
+ mostly re-ranks what TB 2.1 already says. A median over five benchmarks
199
+ therefore leans on TB 2.1 twice, which is worth knowing before reading a
200
+ small score gap as a real difference. The bare `DeepSWE` name stays out — it
201
+ shares no model with `DeepSWE 1.1` in the catalog, so it is an older revision
202
+ carrying four models.
203
+
204
+ Everything else the catalog mentions is context rather than signal, and the
205
+ list is short enough to be worth spelling out. The catalog carries 212
206
+ benchmark names, of which 137 are held by one or two models — an evaluation
207
+ nobody else ran. Those cannot discriminate, so they are excluded twice over:
208
+ no spread means no scale, and a benchmark four models carry still hands its
209
+ worst a hard 0 that drags the median down. A model resolving 82 on SWE-bench
210
+ Verified was landing on 0 overall because of a 17.7 on SEC-Bench Pro.
211
+
212
+ The rest are excluded on the same ground — each would widen the scoring
213
+ equation without adding a model that could not otherwise be scored:
214
+
215
+ - **Terminal-Bench 3.0** (2 models) and **-Science** are separate task sets
216
+ rather than runs of 2.1, and share only a prefix.
217
+ - **Terminal-Bench 2.0** is the same benchmark as 2.1 with an easier, older
218
+ task set — and a public task set gets trained against. A vendor can
219
+ publish a 2.0 number without the model being able to do the work, which is
220
+ the overfitting the rest of this list exists to exclude. Leaving it out
221
+ costs no coverage: the five tier models whose only Terminal-Bench run is a
222
+ 2.0 all carry SWE-bench, so none of them loses a score.
223
+ - **SWE-Bench Lite** and **Multimodal** reach no tier model at all (9 models
224
+ each, none of them one the console prices).
225
+ - **SWE-Bench Multilingual** is carried by 29 models but never alone: every
226
+ model that has it also has Verified or Pro, so it narrows nothing.
227
+ - **Terminal-Bench Hard** is a distinct benchmark despite the name.
228
+ - **LMArena Elo** measures preference votes on chat answers, not task
229
+ completion, and no model in the catalog carries it.
230
+
231
+ Adding one is a deliberate change to what a score means, so it wants a
232
+ reason: benchmark coverage in the live data is the test.
233
+
234
+ ### How a model's score is computed
235
+
236
+ Raw benchmark numbers are not comparable: in the live catalog Terminal-Bench
237
+ 2.1 runs 24.58–89.9, Terminal-Bench 4.0 runs 1.5–64.85 and SWE-bench Verified
238
+ 0.4–96. So each benchmark
239
+ gets its own **0–100 scale** — the best model in the catalog is 100, the
240
+ lowest is 0, and distances are proportionate and linear — and a model's
241
+ score is the **median** of the scales it has.
242
+
243
+ The median, not the mean: coverage here is wildly uneven, and one bad result
244
+ must not bury a model that is strong on everything else. `opencode-models
245
+ scores` gives each of those benchmarks its own column and shows the raw
246
+ number beside where it lands, so the arithmetic is checkable:
247
+
248
+ ```
249
+ 476 models · 5 benchmarks scored on · 45 with scores · 21 on one benchmark · 12 without any
250
+
251
+ MODEL SCORE TB 2.1 TB 4.0 SWE Vrfd SWE Pro DeepSWE
252
+ Muse Spark 1.3 Contributor 99 88.8→98 75.4→100
253
+ Claude Fable 5 99 88→97 44.55→62 95→100 80.3→100
254
+ ```
255
+
256
+ Two more rules decide what counts as evidence.
257
+
258
+ **A model needs two benchmarks to have a score.** The median of one number
259
+ is that number, so a model measured on a single benchmark it happens to lead
260
+ reads as a top model on the strength of one result — in the live catalog 26
261
+ of 63 scored models were in that position, and 6 of the top 9 rows were,
262
+ including the one leading the whole catalog on a lone Terminal-Bench 4.0 run.
263
+ Those models are unranked, not ranked last, and the counts line above the
264
+ table says how many rest on one benchmark.
265
+
266
+ **The ends of a scale are set by numbers someone else can check** — an
267
+ official board or an independent evaluator, not a vendor's own figure. Min-max
268
+ is only as good as its extremes: counting every number, one uncited 1.5 from
269
+ a launch post was the floor of the Terminal-Bench 4.0 scale, which stretched
270
+ the range and compressed every model's derived score toward the middle. That
271
+ is the rule `scoring.scales` applies, with one guard. Where the checkable
272
+ numbers are a small minority of a benchmark's field, the ends stand aside:
273
+ only 8 of 50 Terminal-Bench 2.1 numbers come from a board, and scaling that
274
+ axis to those eight gave it bounds of 64.3–73.4, which clamped nearly every
275
+ model to 0 or 100 and stopped the primary axis discriminating at all. A scale
276
+ drawn from a few points is a threshold, not a scale.
277
+
278
+ So: a benchmark only one or two models carry has no spread, so it cannot
279
+ discriminate and is left out — 137 of 212 benchmarks are in that position.
280
+ And the score answers "how does this model typically do on what it was
281
+ measured on", so a model evaluated on a handful of benchmarks it excels at
282
+ still scores near 100.
283
+
284
+ The console prints what it measured and points here for the arithmetic;
285
+ `scoring.py` carries the reasoning next to the functions that do it. The
286
+ column headers are short (`TB 2.1`, `SWE Vrfd`) and the legend under the
287
+ table spells out which benchmark each one is.
288
+
289
+ Each row shows the model's **calculated score**, its cost, and which plan
290
+ lists it — the per-axis raw numbers live in `scores`, not here. `--benchmark
291
+ X` switches to the single-score layout: that benchmark's raw score and its
292
+ tag.
293
+
294
+ Each line is prefixed with a rank marker read off the score itself — `++`
295
+ at 90 and above, `+` at 75, `o` at 60, `-` at 50, `--` below that, `·` where
296
+ a row has no score to rank (unranked, not last). Absolute bands mean a
297
+ strong field can share the top and a weak one is never flattered by it.
298
+
299
+ Ranking is best score per dollar first. The score is already comparable
300
+ across axes — each benchmark reads on its own 0–100 scale and the median
301
+ combines them — so value order is the combined score divided by the price,
302
+ ties toward the higher score. Rows no benchmark reaches cannot be valued:
303
+ free ones close the listing, paid ones follow, and rows nothing reaches
304
+ drop out (`--benchmark X` ranks on one benchmark instead). Terminal-Bench
305
+ 4.0 is a harder task set — its leaders reach ~58 where 2.1's reach ~87 —
306
+ so a 4.0 score never competes directly with 2.x numbers, and `--benchmark X`
307
+ tags which benchmark a row was ranked on. The axes are not comparable, so
308
+ they only indicate relative performance.
309
+
310
+ Pricing comes from the tier docs rather than the catalog, which carries one
311
+ Zen-style price per model: documented rates replace it, and every priced
312
+ variant (peak/off-peak, token window) gets its own line. Free lines — `-free`
313
+ tier slugs and zero-priced models — are *ranked* at their model's nominal
314
+ free-tier rate (`pricing.FREE_TIER_NOMINAL_RATES`, recomputed on every
315
+ `fetch` from the usage export; see LIMITS.md), so "free" sorts and divides
316
+ like any other cost instead of going infinite, and the value column prints
317
+ that division — the cheapest free models read highest. They are *shown*
318
+ with an actual `0` price: the nominal price never reaches the cost column.
319
+
320
+ ### The `list` views
321
+
322
+ `list` covers the Console **providers** (`zen` = curated, `go` = open) and
323
+ prints the **best value** table. `--tables all` also prints the **worst
324
+ score** (bottom 30%) and **highest cost** (top 30%) tables. Either way the
325
+ cull happens: those two groups of rows are taken *out* of best value, so the
326
+ default view is the models worth looking at and nothing else. The culls are
327
+ disjoint and cost claims first, so a model that is both priciest and
328
+ lowest-scoring is reported once, under cost. Cost decides *who* lands in the
329
+ highest-cost table and score decides *who* lands in the worst-score table,
330
+ but both culls are shown worst-scoring first.
331
+
332
+ By default both providers are merged into one view. Rows that agree on model,
333
+ score and cost are collapsed to a single line *before* ranking and culling —
334
+ they are one offer, not two — with the providers enumerated in the `PROV`
335
+ column immediately before the model name (`go,zen`, `zen`, `go`, or `-` for
336
+ catalog models in neither tier; those only appear with `--source all`). When
337
+ only some of the named providers fund a line from a monthly allowance, the
338
+ more generous band wins: the line is available that way from at least one of
339
+ them. Rates that differ (off-peak, token window, a limit one provider has and
340
+ the other has not) keep their own line each. `--source zen` / `--source go`
341
+ narrows the view to a single provider.
342
+
343
+ Best value splits into bands — **free** (use first), **included usage**
344
+ (lines the tier grants a monthly allowance for), **pay-as-you-go** — and is
345
+ ranked by score per dollar *inside* each band. A line's cost is 20% input +
346
+ 60% cached read + 20% output; `--price input|output` picks a single rate
347
+ instead. The numbers are all rounded integers: the score is the model's
348
+ calculated 0–100 number, score per dollar explains the best-value order —
349
+ free lines divide by their nominal rate, so the cheapest free models read
350
+ highest — and the cost column counts cents (`¢/1M` =
351
+ round(price·100)), so a $0.0612 line reads as `6`, not `0.0612`. Free lines
352
+ still show a `0` cost: the nominal never reaches the price column, only
353
+ the value one. Columns are measured from the lines actually being shown
354
+ (header included), so nothing shifts when a price or a provider list
355
+ outgrows the rest.
356
+
357
+ ### The `next` sequence
358
+
359
+ `next` answers the narrower question "what should I reach for next?". It
360
+ draws from the same pipeline `list` uses, so it can never disagree with the
361
+ ranking it came from. Three rules shape the sequence:
362
+
363
+ - **Eligibility is strength, not the `++` marker.** Picks must be within 75%
364
+ of the strongest score in the catalog. The marker is not used to gate them:
365
+ it is a position in an ordering that divides by price, so it measures
366
+ value. In the live data MiMo-V2.6-Pro scores 89.9 — the best model in the
367
+ catalog — and is graded `o`, while a free model scoring 24.58 is graded
368
+ `++`. Gating on the marker recommends the weak model and misses the strong
369
+ one.
370
+ - **Alternation is forced, on two axes.** Slots alternate between the free
371
+ and paid bands — even slots take the best remaining free pick, odd slots
372
+ the best remaining paid one — so a capped free tier still leaves half the
373
+ sequence usable, and a fresh one still leads with what costs nothing.
374
+ Within each slot the pick alternates between Zen and Go wherever both can
375
+ supply one, so following the sequence spreads across the two plans instead
376
+ of draining one allowance before touching the other. Either alternation
377
+ gives way when it must: a dry band yields its slot to the other band, a
378
+ plan with nothing eligible left is passed over, and the strongest free
379
+ pick always leads.
380
+ - **One model, one slot.** Free on Zen and paid on Go are two lines for one
381
+ model — the `[free]` suffix is the only difference in the name — so a
382
+ model is never recommended twice, and price variants of one model do not
383
+ take two slots either.
384
+
385
+ The view keeps the rules out of the output: it prints one headline, the
386
+ sequence, and a pointer to [LIMITS.md](LIMITS.md) for what allowances cost.
387
+ It does not consult how much of each allowance is left.
388
+
389
+ ### Where the data comes from
390
+
391
+ `data/` is split by who writes it — and **the scraped layers are fetched, not
392
+ committed**. They are third-party records, they go stale, and they are large;
393
+ `opencode-models fetch` populates them into your data directory, and a private
394
+ `local-data` branch keeps a committed copy for offline work (see below).
395
+
396
+ | directory | contents | written by | in the repo? |
397
+ |---|---|---|---|
398
+ | `opencode/` | upstream records, one JSON per model, verbatim from `https://opencode.ai/data/<lab>/<model>.json` | `fetch` only | no — `.gitignore`d |
399
+ | `aiarena/` | arena.ai (LMArena) Elo cache, `{slug: elo}` | `fetch` | no — `.gitignore`d |
400
+ | `tier_prices/` | snapshots of the Console tier pricing tables (see below for what counts as offered) | `fetch` | no — `.gitignore`d |
401
+ | `available/` | the live model set per tier from the Console config API — the plans as the API serves them right now | keyed `fetch` / `models` | no — `.gitignore`d |
402
+ | `boards/` | official benchmark boards, fetched from whoever runs them (see below) | `fetch` | no — `.gitignore`d |
403
+ | `usage/` | free-tier nominal rates derived from the workspace usage export (per-model traffic only, no user columns) | keyed `fetch` (needs `OPENCODE_API_KEY`) | no — `.gitignore`d |
404
+ | `overrides/` | hand-pinned benchmark values (see `data/overrides/README.md`) | nobody — `overrides` reports disagreements | **yes** (the directory, no data) |
405
+
406
+ `fetch` downloads only the models the plans offer (use `--all` for the
407
+ full catalog). Records already on disk are never deleted, so a dropped
408
+ model keeps its place in the scales rather than moving every score.
409
+
410
+ #### Which models the plans actually offer
411
+
412
+ Providers unlist models and move them between plans, so "which models can I
413
+ use" is a fact about today, not a property of the codebase. It used to come
414
+ from committed text files — `opencode_models/sources/zen.txt` and `go.txt` —
415
+ which are hand-maintained and drifted in both directions. In the live data 13
416
+ Go models were still being listed, scored and recommended after the provider's
417
+ own documentation had stopped listing them, among them `hy3-preview` and
418
+ `glm-5`, two of the models an active backfill was chasing numbers for.
419
+
420
+ `fetch` already snapshots each tier's documentation for prices, but the
421
+ docs go stale — providers unlist models the pages still name, and list new
422
+ ones the pages do not. So availability comes from the Console config API
423
+ (`/api/v2/config`, the workspace's live model set), snapshotted to
424
+ `data/available/` on every keyed run (`opencode_models/live.py`). The
425
+ docs snapshot stays the offline fallback, and `opencode-models models`
426
+ prints whichever record is in force as one sorted table: the endpoint
427
+ (as `next` shows it, preferring Go where both plans list the model)
428
+ beside the display name (as `list` shows it, `—` where the catalog has
429
+ none).
430
+
431
+ - **`list` and `next` show only what a plan lists** — live with a token
432
+ (checked every run), the last snapshots offline. `next` therefore never
433
+ recommends a model that has been dropped.
434
+ - **`--all-models`** (alias `--all`, also on `next`) lists the whole catalog
435
+ instead, bypassing every availability filter — which is what you want when
436
+ auditing rather than choosing.
437
+ - **The view says what it covers.** `list` notes it shows only what the
438
+ plans offer right now and points at `--all-models` for the rest, without
439
+ naming names — which models a plan dropped is audit material, not a
440
+ headline.
441
+ - **A tier with no snapshot is unknown, not empty.** Offline and before the
442
+ first `fetch`, nothing is filtered; restricting the console to an empty set
443
+ because nobody fetched would hide everything.
444
+ - **The scales do not move.** A model you cannot buy still marks where the
445
+ field is, so a score never changes because a plan changed what it sells.
446
+
447
+ #### Official boards, fetched from their owners
448
+
449
+ `fetch` also pulls the boards themselves, so the axes are anchored on the
450
+ numbers of whoever runs each benchmark rather than on one registry's
451
+ transcription (which is ~93% vendor self-report):
452
+
453
+ | board | where it comes from | what it carries |
454
+ |---|---|---|
455
+ | Terminal-Bench 4.0 | the official board page's own data payload (tbench.ai; the page renders in a browser, but streams the leaderboard it renders from) | accuracy ± CI95, the agent, the reasoning effort and the date per board row |
456
+ | SWE-bench (Verified, Multilingual, Multimodal, Lite) | swebench.com's embedded leaderboard | `resolved` per checked submission, with the agent that ran it |
457
+
458
+ A board row replaces a model's score only when its source is *strictly*
459
+ better tiered than what the model already carries — an official board beats
460
+ a vendor's own number (MiniMax-M2's self-reported 69.4 on SWE-bench Verified
461
+ yields to the board's 61.0), but it never displaces an independent run, and
462
+ every replacement is printed by `overrides`. Sources that
463
+ only render numbers in a browser with no embedded data (Artificial Analysis,
464
+ Vals AI, Scale's SWE-Atlas, DeepSWE) are not fetched; their numbers arrive
465
+ via the registry or a curated override. A JS-rendered page is not
466
+ automatically a dead end, though — tbench.ai looked like one and is not,
467
+ because a Next.js page streams the data it renders from. aider's polyglot
468
+ board *is* static HTML but was dropped as stale and redundant: its newest
469
+ entry is GPT-5-era (Aug 2025), and diff-editing is already covered by
470
+ SWE-bench and Terminal-Bench.
471
+
472
+ If you want the dataset committed rather than fetched, the `local-data`
473
+ branch keeps a copy — it is intentionally never pushed.
474
+
475
+ #### Curated overrides
476
+
477
+ The upstream layer is never edited: curated scores are merged at load time,
478
+ keyed by (benchmark name, version), so an overlay can *add* a benchmark the
479
+ registry lacks or *pin* a value it also carries. A pinned value applies only
480
+ when its source is at least as good as upstream's — sources are tiered:
481
+ independent evaluators that run every model on one harness (Artificial
482
+ Analysis) beat official benchmark boards (tbench.ai), which beat vendor
483
+ model cards and launch posts. Everything a disagreement touches is printed
484
+ by `overrides`; to accept the catalog value, delete the entry. Rows whose
485
+ model carries an overlay are marked `*` in `list`. See
486
+ `data/overrides/README.md` for the entry format and rules.
487
+
488
+ Parsing is tolerant: models without pricing or without the selected
489
+ benchmark are skipped rather than failing the view.
490
+
491
+ #### Data due diligence
492
+
493
+ What this repo publishes: hand-pinned benchmark numbers
494
+ (`data/overrides/`, each carrying its source URL and provenance note),
495
+ plus code and docs. Everything the tool reads in bulk — the model
496
+ catalog, tier snapshots, official boards, arena cache — is fetched into
497
+ gitignored directories and never committed. No workspace or user data
498
+ is ever written to the repo: the usage export is aggregated in memory
499
+ and only per-model numeric totals reach disk.
500
+
501
+ Why we think republishing those numbers is fair: benchmark scores are
502
+ facts, and every curated value cites the page, chart, or board it was
503
+ read from, with the date and method, so anyone can check or contest it.
504
+ What we have *not* done is review every source site's Terms of Service,
505
+ so if any number here violates a site's terms as you read them, that is
506
+ unintentional — please open a GitHub issue, or a private security
507
+ advisory if the matter is sensitive (see SECURITY.md), and we will
508
+ correct or remove it.
509
+
510
+ ## Caveats
511
+
512
+ - Prices are list prices from one catalog (OpenCode's); they can lag provider
513
+ price changes. Tier pricing is snapshotted offline (`data/tier_prices/`),
514
+ so a stale snapshot is stale prices until the next `fetch`.
515
+ - Benchmarks are heterogeneous (metric, harness, conditions differ per source).
516
+ `list` orders Terminal-Bench rows and SWE-Bench rows separately rather than
517
+ mixing their scores; compare within one benchmark.
518
+ - Coverage depends on which models the opencode catalog carries: a model
519
+ absent from the catalog is invisible to every view.
520
+
521
+ ## Limitations
522
+
523
+ What the tool deliberately does not do, and why.
524
+
525
+ **Cost per request is not a ranking axis.** The same model served to two
526
+ workspaces burns a different number of tokens per request — agent traffic
527
+ with full context re-sends versus chat-sized prompts differ by orders of
528
+ magnitude — so a model's $/request is a property of how *you* use it, not
529
+ of the model. Folding a per-request cost into score-per-dollar would rank
530
+ models on one workspace's habits: the same table would order differently
531
+ for every reader. The metric is not universally available either — no
532
+ catalog, board, or Console endpoint publishes typical per-request token
533
+ volumes (the docs' estimated-request tables imply one, but disagree with
534
+ measured traffic by 10–100× on heavy agent use) — so there is nothing
535
+ stable to divide by. `list` and `next` rank on list price and on the
536
+ nominal free-tier rate instead: comparable across workspaces, even where
537
+ they are less precise for any single one.
538
+
539
+ **Allowances and per-request spend are not shown.** For the same reason,
540
+ there is no allowance column and no $/request column: a Go plan's remaining
541
+ dollars depend on metered usage the tool would have to snapshot per
542
+ workspace, and free tiers publish no cap at all. What the plans grant is
543
+ documented once in [LIMITS.md](LIMITS.md); what you have used is in the
544
+ Console, not here.
@@ -0,0 +1,28 @@
1
+ opencode_models/__init__.py,sha256=7cZTb2RayuqMtzITgTepNrXvWturfVFO2xEAsv5uFVc,295
2
+ opencode_models/__main__.py,sha256=E6Gls0DNz8GQK2K-kOUIx8cYhgANW_CH54VKrfCfs14,52
3
+ opencode_models/arena.py,sha256=S-5lvuyDyVdN8y0V86sc739EJyj7kQF0sv8aTbPnQkE,3569
4
+ opencode_models/auth.py,sha256=WLLkJZj1KeyhvdhUKgcEyTMxITekA-LTKoqfiM7Jh1I,1513
5
+ opencode_models/availability.py,sha256=hceWyRvRGGjoC_cSDRIoSL7tnx2y3-Win8C9uM7IoWY,3915
6
+ opencode_models/benchmarks.py,sha256=60p961Y3VO5uxjd3lywNXqlDd-jdh3ZY7uMi8zv-sO0,557
7
+ opencode_models/boards.py,sha256=rqm26x4AjxWwMiQJ3eA0A9VbI5h6iZTuWL2Jeh_jSVQ,19531
8
+ opencode_models/cli.py,sha256=Z5dWgo8Vg2XCfSPhUXdmdtOj4OhYrq7nM5LSZEuAZR8,43426
9
+ opencode_models/data.py,sha256=3V1CmBFj5sRmbmv2zz36qxb6vxLpG2gLPu5TsL4ho44,2898
10
+ opencode_models/fetch.py,sha256=2RE3TbJNp93Z8p3Ym7rezZnfqcAGdj5ebB1ZDe2FHlA,3734
11
+ opencode_models/identity.py,sha256=1UUC9uqcvTirnLBlN375dju_r-3La7S5vvMSfKjpEMg,4142
12
+ opencode_models/labelmap.py,sha256=gKLgt-4TyXT1M0MDCdDLT67L53dSVrAJHi7sh41q9lg,4004
13
+ opencode_models/live.py,sha256=yh7YdLhDRu1H8DxaPhZWu4EVSs4JxQzfSImq927TjJU,3176
14
+ opencode_models/overrides.py,sha256=0aA38XwTs-fe1izf3VMct6OYw48GVe2JcK1M3t90O18,10068
15
+ opencode_models/pareto.py,sha256=gNK6YqlHq_LW1UobICasUckKslIoXEj0nrXRpdXNY0A,1213
16
+ opencode_models/pricing.py,sha256=7GtG_cGwfrqGoqEGpxA-61a9gYAqRjl0bRMhiZWWa8s,16332
17
+ opencode_models/ranking.py,sha256=QiWAKl3NlBWIFBlbz9ZpCduuCCaVNXb_Yxt4Q10SyeY,22399
18
+ opencode_models/scoring.py,sha256=2rWInCi2ELKDEm20akgPq28Y9oMGaq6ijG_5-3Og2EI,10295
19
+ opencode_models/seed.py,sha256=nH_RsupeYZtPCy5rAscL7gich3IkMxnP6LoFVK74oXQ,6133
20
+ opencode_models/sources.py,sha256=tpfVbtVLmvJP6oHuueTa5B0sct0dy962YlglSs3QhFI,1919
21
+ opencode_models/tables.py,sha256=S0tmtUB46JLRz5GN8oHVWyyNz4lZv6FEzOvLLNy-NYY,3174
22
+ opencode_models/sources/go.txt,sha256=DBVJlO_4mhGPD5t74b9rMvlS7gloRmshkJ_NsWRf7sE,608
23
+ opencode_models/sources/zen.txt,sha256=pd15QlZvgBciBSp51OCVz_iOA_8PqVpUBh3BVH48pNY,1291
24
+ opencode_models-0.1.0.dist-info/METADATA,sha256=Ra9rLV4gIDONKFsGQ6Gk69NJUREbBc8mw3obBFF0Tbw,29498
25
+ opencode_models-0.1.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
26
+ opencode_models-0.1.0.dist-info/entry_points.txt,sha256=nGjaKLlR5x63zJgZm7qyH7ARaSXPGKIt0j3fqa5ZzmQ,61
27
+ opencode_models-0.1.0.dist-info/licenses/LICENSE,sha256=92kVKR9P_yr3yy_WLu3tsB0qGKQWUXhYdDlf130OvFE,1087
28
+ opencode_models-0.1.0.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: hatchling 1.32.4
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ opencode-models = opencode_models.cli:main
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 codesam <tedsamhain@gmail.com>
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.