opencode-models 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- opencode_models/__init__.py +8 -0
- opencode_models/__main__.py +5 -0
- opencode_models/arena.py +108 -0
- opencode_models/auth.py +48 -0
- opencode_models/availability.py +102 -0
- opencode_models/benchmarks.py +19 -0
- opencode_models/boards.py +518 -0
- opencode_models/cli.py +1172 -0
- opencode_models/data.py +82 -0
- opencode_models/fetch.py +101 -0
- opencode_models/identity.py +100 -0
- opencode_models/labelmap.py +84 -0
- opencode_models/live.py +88 -0
- opencode_models/overrides.py +252 -0
- opencode_models/pareto.py +34 -0
- opencode_models/pricing.py +415 -0
- opencode_models/ranking.py +591 -0
- opencode_models/scoring.py +230 -0
- opencode_models/seed.py +184 -0
- opencode_models/sources/go.txt +45 -0
- opencode_models/sources/zen.txt +87 -0
- opencode_models/sources.py +55 -0
- opencode_models/tables.py +94 -0
- opencode_models-0.1.0.dist-info/METADATA +544 -0
- opencode_models-0.1.0.dist-info/RECORD +28 -0
- opencode_models-0.1.0.dist-info/WHEEL +4 -0
- opencode_models-0.1.0.dist-info/entry_points.txt +2 -0
- opencode_models-0.1.0.dist-info/licenses/LICENSE +21 -0
|
@@ -0,0 +1,544 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: opencode-models
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Ranked console of AI models: benchmark scores against real per-token prices
|
|
5
|
+
Project-URL: Homepage, https://github.com/codesam88/opencode-models
|
|
6
|
+
Project-URL: Repository, https://github.com/codesam88/opencode-models
|
|
7
|
+
Project-URL: Issues, https://github.com/codesam88/opencode-models/issues
|
|
8
|
+
Author-email: codesam <tedsamhain@gmail.com>
|
|
9
|
+
License: MIT License
|
|
10
|
+
|
|
11
|
+
Copyright (c) 2026 codesam <tedsamhain@gmail.com>
|
|
12
|
+
|
|
13
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
14
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
15
|
+
in the Software without restriction, including without limitation the rights
|
|
16
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
17
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
18
|
+
furnished to do so, subject to the following conditions:
|
|
19
|
+
|
|
20
|
+
The above copyright notice and this permission notice shall be included in all
|
|
21
|
+
copies or substantial portions of the Software.
|
|
22
|
+
|
|
23
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
24
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
25
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
26
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
27
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
28
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
29
|
+
SOFTWARE.
|
|
30
|
+
License-File: LICENSE
|
|
31
|
+
Keywords: benchmarks,cli,llm,pricing,ranking
|
|
32
|
+
Classifier: Environment :: Console
|
|
33
|
+
Classifier: Intended Audience :: Developers
|
|
34
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
35
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
36
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
37
|
+
Classifier: Topic :: Utilities
|
|
38
|
+
Requires-Python: >=3.10
|
|
39
|
+
Description-Content-Type: text/markdown
|
|
40
|
+
|
|
41
|
+
# opencode-models
|
|
42
|
+
|
|
43
|
+
List [Opencode Zen / Go](https://opencode.ai/data) models by agentic coding score
|
|
44
|
+
and score per dollar.
|
|
45
|
+
|
|
46
|
+
Tired of trying to correlate the latest OpenCode model listings with available
|
|
47
|
+
benchmarks and costs? This utility uses public/scraped benchmark reports and
|
|
48
|
+
the opencode API to show the current best and cheapest available models.
|
|
49
|
+
|
|
50
|
+
## TLDR; quick steps
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
uvx opencode-models
|
|
54
|
+
alias ocm=opencode-models
|
|
55
|
+
|
|
56
|
+
ocm auth # register an openapi token to always fetch the current 'live' models
|
|
57
|
+
ocm fetch # fetch benchmarks and model info from supported sources
|
|
58
|
+
ocm list # List non-expensive, non-trivial models, sorted by score/$
|
|
59
|
+
ocm next # 6 next-best recommended models to use (tell your agent to use this)
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Install
|
|
63
|
+
|
|
64
|
+
Run it once with `uvx`, or install the `opencode-models` binary to your
|
|
65
|
+
`$PATH`:
|
|
66
|
+
|
|
67
|
+
```sh
|
|
68
|
+
uvx opencode-models # no install, just run it
|
|
69
|
+
uv tool install opencode-models # install the binary
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
The name is longer than it wants to be for daily typing; a shell alias
|
|
73
|
+
fixes that in one line: `alias ocm=opencode-models`.
|
|
74
|
+
|
|
75
|
+
### First run
|
|
76
|
+
|
|
77
|
+
```sh
|
|
78
|
+
ocm auth # save a Console token so the views are live (recommended)
|
|
79
|
+
ocm fetch # download benchmark data and model info (~1 minute first time)
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
`auth` stores an OpenCode Console service-account key at
|
|
83
|
+
`~/.config/opencode-models/api_key` (owner-only, never echoed); the
|
|
84
|
+
`OPENCODE_API_KEY` environment variable takes precedence if you set one.
|
|
85
|
+
`fetch` is the only command that downloads data — benchmark numbers,
|
|
86
|
+
tier prices, official boards — into your data directory:
|
|
87
|
+
`$OPENCODE_MODELS_DATA` if set, else `~/.local/share/opencode-models`.
|
|
88
|
+
|
|
89
|
+
### Working from a checkout
|
|
90
|
+
|
|
91
|
+
```sh
|
|
92
|
+
just setup # uv sync + enable git hooks (pre-commit lint/audit,
|
|
93
|
+
# pre-push guard for the local-data branch)
|
|
94
|
+
just test # unittest suite
|
|
95
|
+
just lint # ruff lint, format check, complexity gate
|
|
96
|
+
just check # tests + entry point + wheel build
|
|
97
|
+
just run <cmd> # the console against ./data, e.g. just run next
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
`just install` additionally seeds the reference dataset into your user
|
|
101
|
+
data dir (missing files only — it never overwrites fetched results, while
|
|
102
|
+
tracked curation deploys as shipped). The tool is stdlib-only: Python
|
|
103
|
+
≥ 3.10, no runtime dependencies; ruff comes from the locked dev group.
|
|
104
|
+
|
|
105
|
+
Project pointers: [SECURITY.md](SECURITY.md) (what touches the network,
|
|
106
|
+
scraper etiquette, how to report a problem), [LIMITS.md](LIMITS.md) (what
|
|
107
|
+
the plans actually meter), [CONTRIBUTING.md](CONTRIBUTING.md) (workflow,
|
|
108
|
+
releasing).
|
|
109
|
+
|
|
110
|
+
## Usage
|
|
111
|
+
|
|
112
|
+
Three commands cover daily use; four more exist for auditing and
|
|
113
|
+
maintenance:
|
|
114
|
+
|
|
115
|
+
| command | what it shows |
|
|
116
|
+
|---|---|
|
|
117
|
+
| `list` | the best-value table of what the plans offer today — free, included-usage and pay-as-you-go bands ranked by score per dollar |
|
|
118
|
+
| `next` | the next models to reach for: free and paid picks interleaved, Zen/Go alternating (default 6; tell your agent to use these) |
|
|
119
|
+
| `scores` | every offered model, one column per benchmark — the raw number beside where it lands |
|
|
120
|
+
| `models` | the API endpoint and display name of every model each plan currently offers |
|
|
121
|
+
| `overrides` | benchmark numbers more than one source reports, and which value the tool keeps |
|
|
122
|
+
| `fetch` | refresh everything: benchmarks, tier prices, official boards, availability |
|
|
123
|
+
| `auth` | store the Console token for live views (owner-only file) |
|
|
124
|
+
|
|
125
|
+
### Live views and the token
|
|
126
|
+
|
|
127
|
+
Every `list` and `next` run looks for a Console token first — the
|
|
128
|
+
`OPENCODE_API_KEY` environment variable, then the file `auth` wrote. **With
|
|
129
|
+
a token the view is live:** it queries the Console config API on each run,
|
|
130
|
+
shows only the models the plans offer *right now*, and records the result
|
|
131
|
+
so later offline runs stay current. Without a token the first line warns
|
|
132
|
+
that availability may be stale and the last `fetch` snapshots decide
|
|
133
|
+
(those are refreshed by any keyed `fetch` or `models` run).
|
|
134
|
+
|
|
135
|
+
`--all-models` (alias `--all`) bypasses the availability filter entirely
|
|
136
|
+
— what you want when auditing rather than choosing.
|
|
137
|
+
|
|
138
|
+
### Common options
|
|
139
|
+
|
|
140
|
+
- `--source zen|go|all` — one provider or the full-catalog view
|
|
141
|
+
(default: both providers merged)
|
|
142
|
+
- `--all-models` / `--all` — ignore the availability filter
|
|
143
|
+
(`list`, `next`, `scores`)
|
|
144
|
+
- `--tables all` — add the worst-score and highest-cost tables to `list`
|
|
145
|
+
- `--benchmark NAME` — rank `list` on one benchmark instead of the
|
|
146
|
+
combined score
|
|
147
|
+
- `--count N` — how many picks `next` recommends (default 6)
|
|
148
|
+
- `--price input|output` — cost on a single rate instead of the
|
|
149
|
+
20/60/20 input-cached-output mix
|
|
150
|
+
- `--data DIR` — point at a different data directory (checkout: `--data data`)
|
|
151
|
+
|
|
152
|
+
```sh
|
|
153
|
+
ocm list --tables all # also the culled worst/pricey tables
|
|
154
|
+
ocm list --source zen # one provider only
|
|
155
|
+
ocm next --count 8 --source go # eight picks, Go plan only
|
|
156
|
+
ocm scores # one column per tracked benchmark
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
## How it works
|
|
160
|
+
|
|
161
|
+
The rest of this document is the detailed semantics: what a score means,
|
|
162
|
+
how the views differ, and where every number comes from.
|
|
163
|
+
|
|
164
|
+
### Which benchmarks are scored
|
|
165
|
+
|
|
166
|
+
Five benchmarks carry the scores, and they are named one by one in
|
|
167
|
+
`scoring._TRACKED` rather than matched by prefix:
|
|
168
|
+
|
|
169
|
+
| benchmark | tier models | all models |
|
|
170
|
+
|---|---|---|
|
|
171
|
+
| Terminal-Bench 2.1 | 37 | 50 |
|
|
172
|
+
| SWE-Bench Pro | 34 | 57 |
|
|
173
|
+
| Terminal-Bench 4.0 | 27 | 28 |
|
|
174
|
+
| SWE-Bench Verified | 18 | 77 |
|
|
175
|
+
| **DeepSWE 1.1** | **25** | **29** |
|
|
176
|
+
|
|
177
|
+
They are the most relevant and the most broadly available at the same time:
|
|
178
|
+
Terminal-Bench is the closest thing to "can this model finish a task on a
|
|
179
|
+
real machine", and SWE-bench is the closest thing to "does it fix real bugs",
|
|
180
|
+
so together they cover the work this tool ranks on; and between them they are
|
|
181
|
+
measured on enough models to compare anyone to anyone. The two SWE-bench
|
|
182
|
+
subsets are both kept deliberately — Pro is what the tier broadly carries
|
|
183
|
+
(34 models against Verified's 18), and Verified is the canonical metric and by
|
|
184
|
+
far the broadest number in the catalog (77).
|
|
185
|
+
|
|
186
|
+
**DeepSWE 1.1 is the exception, and it earns its place by pairing rather than
|
|
187
|
+
by relevance.** It is the narrowest thing here — 29 models out of 476 — and
|
|
188
|
+
the weakest on provenance: all 29 numbers are vendor self-reports, and no
|
|
189
|
+
board runs it, so it can never set the ends of its own scale. What it does is
|
|
190
|
+
supply the second axis for models that have nothing else: nine of the 25
|
|
191
|
+
offered models sitting on a single benchmark carry it, and admitting it takes
|
|
192
|
+
the scored field from 36 models to 45. `opencode-models scores` names that
|
|
193
|
+
group explicitly — "N on one benchmark" — so a thin row is never presented as
|
|
194
|
+
a ranked one.
|
|
195
|
+
|
|
196
|
+
Its signal is largely redundant with the axis next to it. Rank correlation
|
|
197
|
+
with Terminal-Bench 2.1, over the models carrying both, is **0.87**: it
|
|
198
|
+
mostly re-ranks what TB 2.1 already says. A median over five benchmarks
|
|
199
|
+
therefore leans on TB 2.1 twice, which is worth knowing before reading a
|
|
200
|
+
small score gap as a real difference. The bare `DeepSWE` name stays out — it
|
|
201
|
+
shares no model with `DeepSWE 1.1` in the catalog, so it is an older revision
|
|
202
|
+
carrying four models.
|
|
203
|
+
|
|
204
|
+
Everything else the catalog mentions is context rather than signal, and the
|
|
205
|
+
list is short enough to be worth spelling out. The catalog carries 212
|
|
206
|
+
benchmark names, of which 137 are held by one or two models — an evaluation
|
|
207
|
+
nobody else ran. Those cannot discriminate, so they are excluded twice over:
|
|
208
|
+
no spread means no scale, and a benchmark four models carry still hands its
|
|
209
|
+
worst a hard 0 that drags the median down. A model resolving 82 on SWE-bench
|
|
210
|
+
Verified was landing on 0 overall because of a 17.7 on SEC-Bench Pro.
|
|
211
|
+
|
|
212
|
+
The rest are excluded on the same ground — each would widen the scoring
|
|
213
|
+
equation without adding a model that could not otherwise be scored:
|
|
214
|
+
|
|
215
|
+
- **Terminal-Bench 3.0** (2 models) and **-Science** are separate task sets
|
|
216
|
+
rather than runs of 2.1, and share only a prefix.
|
|
217
|
+
- **Terminal-Bench 2.0** is the same benchmark as 2.1 with an easier, older
|
|
218
|
+
task set — and a public task set gets trained against. A vendor can
|
|
219
|
+
publish a 2.0 number without the model being able to do the work, which is
|
|
220
|
+
the overfitting the rest of this list exists to exclude. Leaving it out
|
|
221
|
+
costs no coverage: the five tier models whose only Terminal-Bench run is a
|
|
222
|
+
2.0 all carry SWE-bench, so none of them loses a score.
|
|
223
|
+
- **SWE-Bench Lite** and **Multimodal** reach no tier model at all (9 models
|
|
224
|
+
each, none of them one the console prices).
|
|
225
|
+
- **SWE-Bench Multilingual** is carried by 29 models but never alone: every
|
|
226
|
+
model that has it also has Verified or Pro, so it narrows nothing.
|
|
227
|
+
- **Terminal-Bench Hard** is a distinct benchmark despite the name.
|
|
228
|
+
- **LMArena Elo** measures preference votes on chat answers, not task
|
|
229
|
+
completion, and no model in the catalog carries it.
|
|
230
|
+
|
|
231
|
+
Adding one is a deliberate change to what a score means, so it wants a
|
|
232
|
+
reason: benchmark coverage in the live data is the test.
|
|
233
|
+
|
|
234
|
+
### How a model's score is computed
|
|
235
|
+
|
|
236
|
+
Raw benchmark numbers are not comparable: in the live catalog Terminal-Bench
|
|
237
|
+
2.1 runs 24.58–89.9, Terminal-Bench 4.0 runs 1.5–64.85 and SWE-bench Verified
|
|
238
|
+
0.4–96. So each benchmark
|
|
239
|
+
gets its own **0–100 scale** — the best model in the catalog is 100, the
|
|
240
|
+
lowest is 0, and distances are proportionate and linear — and a model's
|
|
241
|
+
score is the **median** of the scales it has.
|
|
242
|
+
|
|
243
|
+
The median, not the mean: coverage here is wildly uneven, and one bad result
|
|
244
|
+
must not bury a model that is strong on everything else. `opencode-models
|
|
245
|
+
scores` gives each of those benchmarks its own column and shows the raw
|
|
246
|
+
number beside where it lands, so the arithmetic is checkable:
|
|
247
|
+
|
|
248
|
+
```
|
|
249
|
+
476 models · 5 benchmarks scored on · 45 with scores · 21 on one benchmark · 12 without any
|
|
250
|
+
|
|
251
|
+
MODEL SCORE TB 2.1 TB 4.0 SWE Vrfd SWE Pro DeepSWE
|
|
252
|
+
Muse Spark 1.3 Contributor 99 88.8→98 75.4→100
|
|
253
|
+
Claude Fable 5 99 88→97 44.55→62 95→100 80.3→100
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
Two more rules decide what counts as evidence.
|
|
257
|
+
|
|
258
|
+
**A model needs two benchmarks to have a score.** The median of one number
|
|
259
|
+
is that number, so a model measured on a single benchmark it happens to lead
|
|
260
|
+
reads as a top model on the strength of one result — in the live catalog 26
|
|
261
|
+
of 63 scored models were in that position, and 6 of the top 9 rows were,
|
|
262
|
+
including the one leading the whole catalog on a lone Terminal-Bench 4.0 run.
|
|
263
|
+
Those models are unranked, not ranked last, and the counts line above the
|
|
264
|
+
table says how many rest on one benchmark.
|
|
265
|
+
|
|
266
|
+
**The ends of a scale are set by numbers someone else can check** — an
|
|
267
|
+
official board or an independent evaluator, not a vendor's own figure. Min-max
|
|
268
|
+
is only as good as its extremes: counting every number, one uncited 1.5 from
|
|
269
|
+
a launch post was the floor of the Terminal-Bench 4.0 scale, which stretched
|
|
270
|
+
the range and compressed every model's derived score toward the middle. That
|
|
271
|
+
is the rule `scoring.scales` applies, with one guard. Where the checkable
|
|
272
|
+
numbers are a small minority of a benchmark's field, the ends stand aside:
|
|
273
|
+
only 8 of 50 Terminal-Bench 2.1 numbers come from a board, and scaling that
|
|
274
|
+
axis to those eight gave it bounds of 64.3–73.4, which clamped nearly every
|
|
275
|
+
model to 0 or 100 and stopped the primary axis discriminating at all. A scale
|
|
276
|
+
drawn from a few points is a threshold, not a scale.
|
|
277
|
+
|
|
278
|
+
So: a benchmark only one or two models carry has no spread, so it cannot
|
|
279
|
+
discriminate and is left out — 137 of 212 benchmarks are in that position.
|
|
280
|
+
And the score answers "how does this model typically do on what it was
|
|
281
|
+
measured on", so a model evaluated on a handful of benchmarks it excels at
|
|
282
|
+
still scores near 100.
|
|
283
|
+
|
|
284
|
+
The console prints what it measured and points here for the arithmetic;
|
|
285
|
+
`scoring.py` carries the reasoning next to the functions that do it. The
|
|
286
|
+
column headers are short (`TB 2.1`, `SWE Vrfd`) and the legend under the
|
|
287
|
+
table spells out which benchmark each one is.
|
|
288
|
+
|
|
289
|
+
Each row shows the model's **calculated score**, its cost, and which plan
|
|
290
|
+
lists it — the per-axis raw numbers live in `scores`, not here. `--benchmark
|
|
291
|
+
X` switches to the single-score layout: that benchmark's raw score and its
|
|
292
|
+
tag.
|
|
293
|
+
|
|
294
|
+
Each line is prefixed with a rank marker read off the score itself — `++`
|
|
295
|
+
at 90 and above, `+` at 75, `o` at 60, `-` at 50, `--` below that, `·` where
|
|
296
|
+
a row has no score to rank (unranked, not last). Absolute bands mean a
|
|
297
|
+
strong field can share the top and a weak one is never flattered by it.
|
|
298
|
+
|
|
299
|
+
Ranking is best score per dollar first. The score is already comparable
|
|
300
|
+
across axes — each benchmark reads on its own 0–100 scale and the median
|
|
301
|
+
combines them — so value order is the combined score divided by the price,
|
|
302
|
+
ties toward the higher score. Rows no benchmark reaches cannot be valued:
|
|
303
|
+
free ones close the listing, paid ones follow, and rows nothing reaches
|
|
304
|
+
drop out (`--benchmark X` ranks on one benchmark instead). Terminal-Bench
|
|
305
|
+
4.0 is a harder task set — its leaders reach ~58 where 2.1's reach ~87 —
|
|
306
|
+
so a 4.0 score never competes directly with 2.x numbers, and `--benchmark X`
|
|
307
|
+
tags which benchmark a row was ranked on. The axes are not comparable, so
|
|
308
|
+
they only indicate relative performance.
|
|
309
|
+
|
|
310
|
+
Pricing comes from the tier docs rather than the catalog, which carries one
|
|
311
|
+
Zen-style price per model: documented rates replace it, and every priced
|
|
312
|
+
variant (peak/off-peak, token window) gets its own line. Free lines — `-free`
|
|
313
|
+
tier slugs and zero-priced models — are *ranked* at their model's nominal
|
|
314
|
+
free-tier rate (`pricing.FREE_TIER_NOMINAL_RATES`, recomputed on every
|
|
315
|
+
`fetch` from the usage export; see LIMITS.md), so "free" sorts and divides
|
|
316
|
+
like any other cost instead of going infinite, and the value column prints
|
|
317
|
+
that division — the cheapest free models read highest. They are *shown*
|
|
318
|
+
with an actual `0` price: the nominal price never reaches the cost column.
|
|
319
|
+
|
|
320
|
+
### The `list` views
|
|
321
|
+
|
|
322
|
+
`list` covers the Console **providers** (`zen` = curated, `go` = open) and
|
|
323
|
+
prints the **best value** table. `--tables all` also prints the **worst
|
|
324
|
+
score** (bottom 30%) and **highest cost** (top 30%) tables. Either way the
|
|
325
|
+
cull happens: those two groups of rows are taken *out* of best value, so the
|
|
326
|
+
default view is the models worth looking at and nothing else. The culls are
|
|
327
|
+
disjoint and cost claims first, so a model that is both priciest and
|
|
328
|
+
lowest-scoring is reported once, under cost. Cost decides *who* lands in the
|
|
329
|
+
highest-cost table and score decides *who* lands in the worst-score table,
|
|
330
|
+
but both culls are shown worst-scoring first.
|
|
331
|
+
|
|
332
|
+
By default both providers are merged into one view. Rows that agree on model,
|
|
333
|
+
score and cost are collapsed to a single line *before* ranking and culling —
|
|
334
|
+
they are one offer, not two — with the providers enumerated in the `PROV`
|
|
335
|
+
column immediately before the model name (`go,zen`, `zen`, `go`, or `-` for
|
|
336
|
+
catalog models in neither tier; those only appear with `--source all`). When
|
|
337
|
+
only some of the named providers fund a line from a monthly allowance, the
|
|
338
|
+
more generous band wins: the line is available that way from at least one of
|
|
339
|
+
them. Rates that differ (off-peak, token window, a limit one provider has and
|
|
340
|
+
the other has not) keep their own line each. `--source zen` / `--source go`
|
|
341
|
+
narrows the view to a single provider.
|
|
342
|
+
|
|
343
|
+
Best value splits into bands — **free** (use first), **included usage**
|
|
344
|
+
(lines the tier grants a monthly allowance for), **pay-as-you-go** — and is
|
|
345
|
+
ranked by score per dollar *inside* each band. A line's cost is 20% input +
|
|
346
|
+
60% cached read + 20% output; `--price input|output` picks a single rate
|
|
347
|
+
instead. The numbers are all rounded integers: the score is the model's
|
|
348
|
+
calculated 0–100 number, score per dollar explains the best-value order —
|
|
349
|
+
free lines divide by their nominal rate, so the cheapest free models read
|
|
350
|
+
highest — and the cost column counts cents (`¢/1M` =
|
|
351
|
+
round(price·100)), so a $0.0612 line reads as `6`, not `0.0612`. Free lines
|
|
352
|
+
still show a `0` cost: the nominal never reaches the price column, only
|
|
353
|
+
the value one. Columns are measured from the lines actually being shown
|
|
354
|
+
(header included), so nothing shifts when a price or a provider list
|
|
355
|
+
outgrows the rest.
|
|
356
|
+
|
|
357
|
+
### The `next` sequence
|
|
358
|
+
|
|
359
|
+
`next` answers the narrower question "what should I reach for next?". It
|
|
360
|
+
draws from the same pipeline `list` uses, so it can never disagree with the
|
|
361
|
+
ranking it came from. Three rules shape the sequence:
|
|
362
|
+
|
|
363
|
+
- **Eligibility is strength, not the `++` marker.** Picks must be within 75%
|
|
364
|
+
of the strongest score in the catalog. The marker is not used to gate them:
|
|
365
|
+
it is a position in an ordering that divides by price, so it measures
|
|
366
|
+
value. In the live data MiMo-V2.6-Pro scores 89.9 — the best model in the
|
|
367
|
+
catalog — and is graded `o`, while a free model scoring 24.58 is graded
|
|
368
|
+
`++`. Gating on the marker recommends the weak model and misses the strong
|
|
369
|
+
one.
|
|
370
|
+
- **Alternation is forced, on two axes.** Slots alternate between the free
|
|
371
|
+
and paid bands — even slots take the best remaining free pick, odd slots
|
|
372
|
+
the best remaining paid one — so a capped free tier still leaves half the
|
|
373
|
+
sequence usable, and a fresh one still leads with what costs nothing.
|
|
374
|
+
Within each slot the pick alternates between Zen and Go wherever both can
|
|
375
|
+
supply one, so following the sequence spreads across the two plans instead
|
|
376
|
+
of draining one allowance before touching the other. Either alternation
|
|
377
|
+
gives way when it must: a dry band yields its slot to the other band, a
|
|
378
|
+
plan with nothing eligible left is passed over, and the strongest free
|
|
379
|
+
pick always leads.
|
|
380
|
+
- **One model, one slot.** Free on Zen and paid on Go are two lines for one
|
|
381
|
+
model — the `[free]` suffix is the only difference in the name — so a
|
|
382
|
+
model is never recommended twice, and price variants of one model do not
|
|
383
|
+
take two slots either.
|
|
384
|
+
|
|
385
|
+
The view keeps the rules out of the output: it prints one headline, the
|
|
386
|
+
sequence, and a pointer to [LIMITS.md](LIMITS.md) for what allowances cost.
|
|
387
|
+
It does not consult how much of each allowance is left.
|
|
388
|
+
|
|
389
|
+
### Where the data comes from
|
|
390
|
+
|
|
391
|
+
`data/` is split by who writes it — and **the scraped layers are fetched, not
|
|
392
|
+
committed**. They are third-party records, they go stale, and they are large;
|
|
393
|
+
`opencode-models fetch` populates them into your data directory, and a private
|
|
394
|
+
`local-data` branch keeps a committed copy for offline work (see below).
|
|
395
|
+
|
|
396
|
+
| directory | contents | written by | in the repo? |
|
|
397
|
+
|---|---|---|---|
|
|
398
|
+
| `opencode/` | upstream records, one JSON per model, verbatim from `https://opencode.ai/data/<lab>/<model>.json` | `fetch` only | no — `.gitignore`d |
|
|
399
|
+
| `aiarena/` | arena.ai (LMArena) Elo cache, `{slug: elo}` | `fetch` | no — `.gitignore`d |
|
|
400
|
+
| `tier_prices/` | snapshots of the Console tier pricing tables (see below for what counts as offered) | `fetch` | no — `.gitignore`d |
|
|
401
|
+
| `available/` | the live model set per tier from the Console config API — the plans as the API serves them right now | keyed `fetch` / `models` | no — `.gitignore`d |
|
|
402
|
+
| `boards/` | official benchmark boards, fetched from whoever runs them (see below) | `fetch` | no — `.gitignore`d |
|
|
403
|
+
| `usage/` | free-tier nominal rates derived from the workspace usage export (per-model traffic only, no user columns) | keyed `fetch` (needs `OPENCODE_API_KEY`) | no — `.gitignore`d |
|
|
404
|
+
| `overrides/` | hand-pinned benchmark values (see `data/overrides/README.md`) | nobody — `overrides` reports disagreements | **yes** (the directory, no data) |
|
|
405
|
+
|
|
406
|
+
`fetch` downloads only the models the plans offer (use `--all` for the
|
|
407
|
+
full catalog). Records already on disk are never deleted, so a dropped
|
|
408
|
+
model keeps its place in the scales rather than moving every score.
|
|
409
|
+
|
|
410
|
+
#### Which models the plans actually offer
|
|
411
|
+
|
|
412
|
+
Providers unlist models and move them between plans, so "which models can I
|
|
413
|
+
use" is a fact about today, not a property of the codebase. It used to come
|
|
414
|
+
from committed text files — `opencode_models/sources/zen.txt` and `go.txt` —
|
|
415
|
+
which are hand-maintained and drifted in both directions. In the live data 13
|
|
416
|
+
Go models were still being listed, scored and recommended after the provider's
|
|
417
|
+
own documentation had stopped listing them, among them `hy3-preview` and
|
|
418
|
+
`glm-5`, two of the models an active backfill was chasing numbers for.
|
|
419
|
+
|
|
420
|
+
`fetch` already snapshots each tier's documentation for prices, but the
|
|
421
|
+
docs go stale — providers unlist models the pages still name, and list new
|
|
422
|
+
ones the pages do not. So availability comes from the Console config API
|
|
423
|
+
(`/api/v2/config`, the workspace's live model set), snapshotted to
|
|
424
|
+
`data/available/` on every keyed run (`opencode_models/live.py`). The
|
|
425
|
+
docs snapshot stays the offline fallback, and `opencode-models models`
|
|
426
|
+
prints whichever record is in force as one sorted table: the endpoint
|
|
427
|
+
(as `next` shows it, preferring Go where both plans list the model)
|
|
428
|
+
beside the display name (as `list` shows it, `—` where the catalog has
|
|
429
|
+
none).
|
|
430
|
+
|
|
431
|
+
- **`list` and `next` show only what a plan lists** — live with a token
|
|
432
|
+
(checked every run), the last snapshots offline. `next` therefore never
|
|
433
|
+
recommends a model that has been dropped.
|
|
434
|
+
- **`--all-models`** (alias `--all`, also on `next`) lists the whole catalog
|
|
435
|
+
instead, bypassing every availability filter — which is what you want when
|
|
436
|
+
auditing rather than choosing.
|
|
437
|
+
- **The view says what it covers.** `list` notes it shows only what the
|
|
438
|
+
plans offer right now and points at `--all-models` for the rest, without
|
|
439
|
+
naming names — which models a plan dropped is audit material, not a
|
|
440
|
+
headline.
|
|
441
|
+
- **A tier with no snapshot is unknown, not empty.** Offline and before the
|
|
442
|
+
first `fetch`, nothing is filtered; restricting the console to an empty set
|
|
443
|
+
because nobody fetched would hide everything.
|
|
444
|
+
- **The scales do not move.** A model you cannot buy still marks where the
|
|
445
|
+
field is, so a score never changes because a plan changed what it sells.
|
|
446
|
+
|
|
447
|
+
#### Official boards, fetched from their owners
|
|
448
|
+
|
|
449
|
+
`fetch` also pulls the boards themselves, so the axes are anchored on the
|
|
450
|
+
numbers of whoever runs each benchmark rather than on one registry's
|
|
451
|
+
transcription (which is ~93% vendor self-report):
|
|
452
|
+
|
|
453
|
+
| board | where it comes from | what it carries |
|
|
454
|
+
|---|---|---|
|
|
455
|
+
| Terminal-Bench 4.0 | the official board page's own data payload (tbench.ai; the page renders in a browser, but streams the leaderboard it renders from) | accuracy ± CI95, the agent, the reasoning effort and the date per board row |
|
|
456
|
+
| SWE-bench (Verified, Multilingual, Multimodal, Lite) | swebench.com's embedded leaderboard | `resolved` per checked submission, with the agent that ran it |
|
|
457
|
+
|
|
458
|
+
A board row replaces a model's score only when its source is *strictly*
|
|
459
|
+
better tiered than what the model already carries — an official board beats
|
|
460
|
+
a vendor's own number (MiniMax-M2's self-reported 69.4 on SWE-bench Verified
|
|
461
|
+
yields to the board's 61.0), but it never displaces an independent run, and
|
|
462
|
+
every replacement is printed by `overrides`. Sources that
|
|
463
|
+
only render numbers in a browser with no embedded data (Artificial Analysis,
|
|
464
|
+
Vals AI, Scale's SWE-Atlas, DeepSWE) are not fetched; their numbers arrive
|
|
465
|
+
via the registry or a curated override. A JS-rendered page is not
|
|
466
|
+
automatically a dead end, though — tbench.ai looked like one and is not,
|
|
467
|
+
because a Next.js page streams the data it renders from. aider's polyglot
|
|
468
|
+
board *is* static HTML but was dropped as stale and redundant: its newest
|
|
469
|
+
entry is GPT-5-era (Aug 2025), and diff-editing is already covered by
|
|
470
|
+
SWE-bench and Terminal-Bench.
|
|
471
|
+
|
|
472
|
+
If you want the dataset committed rather than fetched, the `local-data`
|
|
473
|
+
branch keeps a copy — it is intentionally never pushed.
|
|
474
|
+
|
|
475
|
+
#### Curated overrides
|
|
476
|
+
|
|
477
|
+
The upstream layer is never edited: curated scores are merged at load time,
|
|
478
|
+
keyed by (benchmark name, version), so an overlay can *add* a benchmark the
|
|
479
|
+
registry lacks or *pin* a value it also carries. A pinned value applies only
|
|
480
|
+
when its source is at least as good as upstream's — sources are tiered:
|
|
481
|
+
independent evaluators that run every model on one harness (Artificial
|
|
482
|
+
Analysis) beat official benchmark boards (tbench.ai), which beat vendor
|
|
483
|
+
model cards and launch posts. Everything a disagreement touches is printed
|
|
484
|
+
by `overrides`; to accept the catalog value, delete the entry. Rows whose
|
|
485
|
+
model carries an overlay are marked `*` in `list`. See
|
|
486
|
+
`data/overrides/README.md` for the entry format and rules.
|
|
487
|
+
|
|
488
|
+
Parsing is tolerant: models without pricing or without the selected
|
|
489
|
+
benchmark are skipped rather than failing the view.
|
|
490
|
+
|
|
491
|
+
#### Data due diligence
|
|
492
|
+
|
|
493
|
+
What this repo publishes: hand-pinned benchmark numbers
|
|
494
|
+
(`data/overrides/`, each carrying its source URL and provenance note),
|
|
495
|
+
plus code and docs. Everything the tool reads in bulk — the model
|
|
496
|
+
catalog, tier snapshots, official boards, arena cache — is fetched into
|
|
497
|
+
gitignored directories and never committed. No workspace or user data
|
|
498
|
+
is ever written to the repo: the usage export is aggregated in memory
|
|
499
|
+
and only per-model numeric totals reach disk.
|
|
500
|
+
|
|
501
|
+
Why we think republishing those numbers is fair: benchmark scores are
|
|
502
|
+
facts, and every curated value cites the page, chart, or board it was
|
|
503
|
+
read from, with the date and method, so anyone can check or contest it.
|
|
504
|
+
What we have *not* done is review every source site's Terms of Service,
|
|
505
|
+
so if any number here violates a site's terms as you read them, that is
|
|
506
|
+
unintentional — please open a GitHub issue, or a private security
|
|
507
|
+
advisory if the matter is sensitive (see SECURITY.md), and we will
|
|
508
|
+
correct or remove it.
|
|
509
|
+
|
|
510
|
+
## Caveats
|
|
511
|
+
|
|
512
|
+
- Prices are list prices from one catalog (OpenCode's); they can lag provider
|
|
513
|
+
price changes. Tier pricing is snapshotted offline (`data/tier_prices/`),
|
|
514
|
+
so a stale snapshot is stale prices until the next `fetch`.
|
|
515
|
+
- Benchmarks are heterogeneous (metric, harness, conditions differ per source).
|
|
516
|
+
`list` orders Terminal-Bench rows and SWE-Bench rows separately rather than
|
|
517
|
+
mixing their scores; compare within one benchmark.
|
|
518
|
+
- Coverage depends on which models the opencode catalog carries: a model
|
|
519
|
+
absent from the catalog is invisible to every view.
|
|
520
|
+
|
|
521
|
+
## Limitations
|
|
522
|
+
|
|
523
|
+
What the tool deliberately does not do, and why.
|
|
524
|
+
|
|
525
|
+
**Cost per request is not a ranking axis.** The same model served to two
|
|
526
|
+
workspaces burns a different number of tokens per request — agent traffic
|
|
527
|
+
with full context re-sends versus chat-sized prompts differ by orders of
|
|
528
|
+
magnitude — so a model's $/request is a property of how *you* use it, not
|
|
529
|
+
of the model. Folding a per-request cost into score-per-dollar would rank
|
|
530
|
+
models on one workspace's habits: the same table would order differently
|
|
531
|
+
for every reader. The metric is not universally available either — no
|
|
532
|
+
catalog, board, or Console endpoint publishes typical per-request token
|
|
533
|
+
volumes (the docs' estimated-request tables imply one, but disagree with
|
|
534
|
+
measured traffic by 10–100× on heavy agent use) — so there is nothing
|
|
535
|
+
stable to divide by. `list` and `next` rank on list price and on the
|
|
536
|
+
nominal free-tier rate instead: comparable across workspaces, even where
|
|
537
|
+
they are less precise for any single one.
|
|
538
|
+
|
|
539
|
+
**Allowances and per-request spend are not shown.** For the same reason,
|
|
540
|
+
there is no allowance column and no $/request column: a Go plan's remaining
|
|
541
|
+
dollars depend on metered usage the tool would have to snapshot per
|
|
542
|
+
workspace, and free tiers publish no cap at all. What the plans grant is
|
|
543
|
+
documented once in [LIMITS.md](LIMITS.md); what you have used is in the
|
|
544
|
+
Console, not here.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
opencode_models/__init__.py,sha256=7cZTb2RayuqMtzITgTepNrXvWturfVFO2xEAsv5uFVc,295
|
|
2
|
+
opencode_models/__main__.py,sha256=E6Gls0DNz8GQK2K-kOUIx8cYhgANW_CH54VKrfCfs14,52
|
|
3
|
+
opencode_models/arena.py,sha256=S-5lvuyDyVdN8y0V86sc739EJyj7kQF0sv8aTbPnQkE,3569
|
|
4
|
+
opencode_models/auth.py,sha256=WLLkJZj1KeyhvdhUKgcEyTMxITekA-LTKoqfiM7Jh1I,1513
|
|
5
|
+
opencode_models/availability.py,sha256=hceWyRvRGGjoC_cSDRIoSL7tnx2y3-Win8C9uM7IoWY,3915
|
|
6
|
+
opencode_models/benchmarks.py,sha256=60p961Y3VO5uxjd3lywNXqlDd-jdh3ZY7uMi8zv-sO0,557
|
|
7
|
+
opencode_models/boards.py,sha256=rqm26x4AjxWwMiQJ3eA0A9VbI5h6iZTuWL2Jeh_jSVQ,19531
|
|
8
|
+
opencode_models/cli.py,sha256=Z5dWgo8Vg2XCfSPhUXdmdtOj4OhYrq7nM5LSZEuAZR8,43426
|
|
9
|
+
opencode_models/data.py,sha256=3V1CmBFj5sRmbmv2zz36qxb6vxLpG2gLPu5TsL4ho44,2898
|
|
10
|
+
opencode_models/fetch.py,sha256=2RE3TbJNp93Z8p3Ym7rezZnfqcAGdj5ebB1ZDe2FHlA,3734
|
|
11
|
+
opencode_models/identity.py,sha256=1UUC9uqcvTirnLBlN375dju_r-3La7S5vvMSfKjpEMg,4142
|
|
12
|
+
opencode_models/labelmap.py,sha256=gKLgt-4TyXT1M0MDCdDLT67L53dSVrAJHi7sh41q9lg,4004
|
|
13
|
+
opencode_models/live.py,sha256=yh7YdLhDRu1H8DxaPhZWu4EVSs4JxQzfSImq927TjJU,3176
|
|
14
|
+
opencode_models/overrides.py,sha256=0aA38XwTs-fe1izf3VMct6OYw48GVe2JcK1M3t90O18,10068
|
|
15
|
+
opencode_models/pareto.py,sha256=gNK6YqlHq_LW1UobICasUckKslIoXEj0nrXRpdXNY0A,1213
|
|
16
|
+
opencode_models/pricing.py,sha256=7GtG_cGwfrqGoqEGpxA-61a9gYAqRjl0bRMhiZWWa8s,16332
|
|
17
|
+
opencode_models/ranking.py,sha256=QiWAKl3NlBWIFBlbz9ZpCduuCCaVNXb_Yxt4Q10SyeY,22399
|
|
18
|
+
opencode_models/scoring.py,sha256=2rWInCi2ELKDEm20akgPq28Y9oMGaq6ijG_5-3Og2EI,10295
|
|
19
|
+
opencode_models/seed.py,sha256=nH_RsupeYZtPCy5rAscL7gich3IkMxnP6LoFVK74oXQ,6133
|
|
20
|
+
opencode_models/sources.py,sha256=tpfVbtVLmvJP6oHuueTa5B0sct0dy962YlglSs3QhFI,1919
|
|
21
|
+
opencode_models/tables.py,sha256=S0tmtUB46JLRz5GN8oHVWyyNz4lZv6FEzOvLLNy-NYY,3174
|
|
22
|
+
opencode_models/sources/go.txt,sha256=DBVJlO_4mhGPD5t74b9rMvlS7gloRmshkJ_NsWRf7sE,608
|
|
23
|
+
opencode_models/sources/zen.txt,sha256=pd15QlZvgBciBSp51OCVz_iOA_8PqVpUBh3BVH48pNY,1291
|
|
24
|
+
opencode_models-0.1.0.dist-info/METADATA,sha256=Ra9rLV4gIDONKFsGQ6Gk69NJUREbBc8mw3obBFF0Tbw,29498
|
|
25
|
+
opencode_models-0.1.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
|
|
26
|
+
opencode_models-0.1.0.dist-info/entry_points.txt,sha256=nGjaKLlR5x63zJgZm7qyH7ARaSXPGKIt0j3fqa5ZzmQ,61
|
|
27
|
+
opencode_models-0.1.0.dist-info/licenses/LICENSE,sha256=92kVKR9P_yr3yy_WLu3tsB0qGKQWUXhYdDlf130OvFE,1087
|
|
28
|
+
opencode_models-0.1.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 codesam <tedsamhain@gmail.com>
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|