@lawaty/lacode 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +47 -1
- package/README.md +22 -9
- package/dist/agents.js +8 -3
- package/dist/commands.js +9 -5
- package/dist/index.js +1017 -94
- package/dist/lacode-voice.js +2351 -0
- package/dist/pool.js +73 -38
- package/dist/tui.js +1282 -0
- package/docs/CONFIGURATION.md +17 -7
- package/docs/USER-STORIES.md +11 -11
- package/package.json +12 -4
- package/presets/README.md +17 -19
- package/presets/free-tier.json +11 -13
- package/scripts/score-models.mjs +353 -0
package/docs/CONFIGURATION.md
CHANGED
|
@@ -23,12 +23,22 @@ allowlist anywhere in the package, and no cost check of any kind: only the
|
|
|
23
23
|
structure is validated. The bundled preset is built from free models because
|
|
24
24
|
that is a sensible default, not because the pool requires it.
|
|
25
25
|
|
|
26
|
+
A slot's `weight` (default 1) is both its concurrency budget and its cap. The
|
|
27
|
+
pool fills slots in declaration order in one pass; there is no slot cap, and a
|
|
28
|
+
model may appear on more than one line — each line is its own slot with its own
|
|
29
|
+
weight. A `tier` key is the deprecated spelling of `weight`.
|
|
30
|
+
|
|
31
|
+
A slot may also declare a shared quota bucket with `b:<name>` on the line (or a
|
|
32
|
+
`bucket` field in JSON). A limit on any member benches the whole bucket, because
|
|
33
|
+
the sibling would spend the same exhausted window; without one, the model is its
|
|
34
|
+
own bucket. `pool_status` shows each shared bucket's effective capacity.
|
|
35
|
+
|
|
26
36
|
|
|
27
37
|
## Which models run which agent
|
|
28
38
|
|
|
29
39
|
Only the **pool** writes a model onto a pooled agent, from whichever source wins
|
|
30
40
|
above. The pooled bases (`explore`, `implement`, `context`)
|
|
31
|
-
and their hidden
|
|
41
|
+
and their hidden `<base>-<slot>` variants therefore carry no model of their own: a
|
|
32
42
|
model written next to an agent would be a second source of truth that silently
|
|
33
43
|
wins whenever routing falls through. Spawn a base and let the pool route it.
|
|
34
44
|
|
|
@@ -39,7 +49,7 @@ wins whenever routing falls through. Spawn a base and let the pool route it.
|
|
|
39
49
|
work around by configuring it.** If you keep a global `"tools"` block in your
|
|
40
50
|
config — for example `"tools": { "playwright_*": false }` — then the pooled
|
|
41
51
|
exploration and implementation bases (`explore` and `implement`, plus their
|
|
42
|
-
hidden
|
|
52
|
+
hidden `<base>-<slot>` pool variants, one per resolved slot) lose browser access, and **nothing tells them why**: the tool call is simply
|
|
43
53
|
refused mid-task. `context` gets no grant either, which is correct — it
|
|
44
54
|
writes the context map and has no business in a browser.
|
|
45
55
|
|
|
@@ -89,11 +99,11 @@ mention/direct-session case cannot be detected before it runs.
|
|
|
89
99
|
|
|
90
100
|
## What gets injected
|
|
91
101
|
|
|
92
|
-
**
|
|
93
|
-
`context`,
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
102
|
+
**One key per base, plus a variant per slot**: the 5 bases `build`, `plan`,
|
|
103
|
+
`explore`, `implement`, `context`, one hidden `<base>-<slot>` pool variant per
|
|
104
|
+
resolved slot for each of the 3 pooled bases, and the hidden `instructions`
|
|
105
|
+
agent — the one writer of the project's root `AGENTS.md`, reachable only from
|
|
106
|
+
the `/agent-instructions` command.
|
|
97
107
|
|
|
98
108
|
`build` and `plan` exist in opencode as built-ins, so on a stock config those
|
|
99
109
|
two keys are already taken and merge-without-clobber keeps the built-in — the
|
package/docs/USER-STORIES.md
CHANGED
|
@@ -15,7 +15,7 @@ line. Criteria that cannot be observed without a running server are not in this
|
|
|
15
15
|
document; everything here is testable offline with a stub client and a
|
|
16
16
|
`mkdtempSync` state directory.
|
|
17
17
|
|
|
18
|
-
**Terminology.** A *slot* is one entry in `presets/free-tier.json`, carrying a model and a `weight`. A slot's `weight` is its active-session budget for one pass: the number of concurrent claims it may hold before the next slot is preferred. A *pass* is one declaration-order sweep of the slot list; the pool runs
|
|
18
|
+
**Terminology.** A *slot* is one entry in `presets/free-tier.json`, carrying a model and a `weight`. A slot's `weight` is its active-session budget for one pass: the number of concurrent claims it may hold before the next slot is preferred. A *pass* is one declaration-order sweep of the slot list; the pool runs one pass, and a slot's cap is its weight. There is no slot cap, and a model may appear on more than one line — each line is its own slot with its own weight. Preference is by declaration order, never by weight. A *base* is a pooled agent type (`explore`), and its slot holders are `<base>-<slot>`, hidden in `opencode.jsonc`. A *claim* is a running spawn recorded in `claims.<pid>.json`. A *context map* is the five-file `.opencode/context/` codemap described in Epic G.
|
|
19
19
|
|
|
20
20
|
---
|
|
21
21
|
|
|
@@ -92,18 +92,18 @@ The reconciliation with US-7 is that **acceptance and endorsement are different
|
|
|
92
92
|
- [ ] Given every model is cooling, When a spawn is routed, Then the least-loaded slot is chosen anyway (degrade, never block).
|
|
93
93
|
- [ ] Given a limit error names a reset time, When the cooldown is recorded, Then the stated reset is used instead of a guessed backoff.
|
|
94
94
|
|
|
95
|
-
### US-5: Slots fill in declaration order
|
|
95
|
+
### US-5: Slots fill in declaration order in one pass, with repeats
|
|
96
96
|
**As a** user, **I want** work to prefer the first slot and spill to later slots only when the earlier ones are full, **so that** a healthy preferred model is used before fallbacks and no slot is used while an earlier usable slot has room.
|
|
97
97
|
|
|
98
98
|
**Story ID(s) for tests:** US-5
|
|
99
99
|
**Acceptance criteria:**
|
|
100
|
-
- [ ] Given live claims, When the pool has room
|
|
101
|
-
- [ ] Given
|
|
102
|
-
- [ ] Given a slot is cooling or unavailable, When its position
|
|
100
|
+
- [ ] Given live claims, When the pool has room, Then the lowest-index usable slot whose load is below its weight is chosen, and a slot declared later is not used until every earlier usable slot has reached its weight.
|
|
101
|
+
- [ ] Given a model appears on more than one line, When the pool loads, Then each line is its own slot with its own weight, and the later line adds seats at its own position.
|
|
102
|
+
- [ ] Given a slot is cooling or unavailable, When its position comes up, Then it is skipped and the pass continues to the next slot.
|
|
103
103
|
- [ ] Given a claim is released, When the next spawn is routed, Then the slot that freed capacity is preferred again in declaration order; no stored position keeps it waiting.
|
|
104
|
-
- [ ] Given every usable slot is at its
|
|
105
|
-
- [ ] Given all slots carry equal weight, When spawns are routed, Then the same ordered
|
|
106
|
-
- [ ] Given per-slot load is read, When it is asserted, Then a slot's load never exceeds
|
|
104
|
+
- [ ] Given every usable slot is at its weight, When another spawn is routed, Then it goes to the least-loaded usable slot as overflow, and the pool never refuses work.
|
|
105
|
+
- [ ] Given all slots carry equal weight, When spawns are routed, Then the same ordered one-pass fill applies.
|
|
106
|
+
- [ ] Given per-slot load is read, When it is asserted, Then a slot's load never exceeds its weight except through overflow.
|
|
107
107
|
|
|
108
108
|
### US-6: Live pool visibility
|
|
109
109
|
**As a** user, **I want** to see per-slot load and which model is cooling and why, **so that** I understand where my subagents are going.
|
|
@@ -113,8 +113,8 @@ The reconciliation with US-7 is that **acceptance and endorsement are different
|
|
|
113
113
|
- [ ] Given claims are held by several processes, When the status view is requested, Then it shows every process's claims and the aggregate load per slot.
|
|
114
114
|
- [ ] Given a model is cooling with a stated reason, When the status view is requested, Then the model is listed as cooling together with that reason.
|
|
115
115
|
- [ ] Given a claim file is older than the dead-file threshold, When the status view is computed, Then the stale claim is pruned rather than counted.
|
|
116
|
-
- [ ] Given a slot is at its
|
|
117
|
-
- [ ] Given the pool is filling, When the status view is requested, Then it shows whether the next spawn
|
|
116
|
+
- [ ] Given a slot is at its weight, When the status view is requested, Then it is not marked as the slot the next spawn would take while any usable slot is below its weight; when every usable slot is at its weight, the least-loaded slot is marked next.
|
|
117
|
+
- [ ] Given the pool is filling, When the status view is requested, Then it shows whether the next spawn is a normal pick or overflow.
|
|
118
118
|
|
|
119
119
|
### US-7: Non-free models still work
|
|
120
120
|
**As a** user who wants a paid model in the pool, **I want** to supply any model list, **so that** the pool is not restricted to free tiers.
|
|
@@ -123,7 +123,7 @@ The reconciliation with US-7 is that **acceptance and endorsement are different
|
|
|
123
123
|
**Acceptance criteria:**
|
|
124
124
|
- [ ] Given a slot names a paid model, When the pool loads, Then the slot is accepted and routed to without a free-tier filter rejecting it.
|
|
125
125
|
- [ ] Given a model id contains a vendor prefix with slashes (e.g. `openrouter/nvidia/model:free`), When it is split into provider and model id, Then the split is on the first slash only and the id tail is preserved.
|
|
126
|
-
- [ ] Given more slots
|
|
126
|
+
- [ ] Given a pool file declares more slots than the built-in warning threshold, When the pool loads, Then every entry is kept and a warning names the count; there is no slot cap.
|
|
127
127
|
- [ ] Given a provider configured with **no** `whitelist` key, When the config hook binds a slot on that provider, Then no `whitelist` is written for it, because an absent whitelist already allows every model the provider offers.
|
|
128
128
|
- [ ] Given a provider the pool never names a slot on, When the config hook runs, Then that provider entry is left exactly as it was, with no `whitelist` key added.
|
|
129
129
|
- [ ] Given a provider that already carries a `whitelist`, When the config hook binds a slot model on it, Then the model id is appended and every entry the user wrote is preserved in order.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@lawaty/lacode",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.2.0",
|
|
4
4
|
"description": "Ultimate opencode setup for Go users — cost-controlled pooled subagents, hang recovery, and notifications for opencode.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -34,11 +34,13 @@
|
|
|
34
34
|
"./pool": "./dist/pool.js",
|
|
35
35
|
"./agents": "./dist/agents.js",
|
|
36
36
|
"./commands": "./dist/commands.js",
|
|
37
|
+
"./tui": "./dist/tui.js",
|
|
37
38
|
"./tunnel": "./dist/tunnel.js",
|
|
38
39
|
"./package.json": "./package.json"
|
|
39
40
|
},
|
|
40
41
|
"bin": {
|
|
41
|
-
"lacode": "
|
|
42
|
+
"lacode": "dist/lacode.js",
|
|
43
|
+
"lacode-voice": "dist/lacode-voice.js"
|
|
42
44
|
},
|
|
43
45
|
"files": [
|
|
44
46
|
"dist",
|
|
@@ -47,14 +49,16 @@
|
|
|
47
49
|
"LICENSE",
|
|
48
50
|
"CHANGELOG.md",
|
|
49
51
|
"README.md",
|
|
50
|
-
"docs"
|
|
52
|
+
"docs",
|
|
53
|
+
"scripts/score-models.mjs"
|
|
51
54
|
],
|
|
52
55
|
"scripts": {
|
|
53
56
|
"build": "node scripts/build.mjs",
|
|
54
57
|
"prepack": "npm run build",
|
|
55
58
|
"postpublish": "node scripts/sync-hosts.mjs",
|
|
56
59
|
"test": "node --test tests/",
|
|
57
|
-
"benchmark": "node tests/benchmark/run.mjs"
|
|
60
|
+
"benchmark": "node tests/benchmark/run.mjs",
|
|
61
|
+
"score": "node scripts/score-models.mjs"
|
|
58
62
|
},
|
|
59
63
|
"devDependencies": {
|
|
60
64
|
"esbuild": "^0.28.2"
|
|
@@ -62,6 +66,10 @@
|
|
|
62
66
|
"peerDependencies": {
|
|
63
67
|
"@opencode-ai/plugin": "1.18.34"
|
|
64
68
|
},
|
|
69
|
+
"optionalDependencies": {
|
|
70
|
+
"@picovoice/porcupine-node": "^4.0.2",
|
|
71
|
+
"vosk": "^0.3.39"
|
|
72
|
+
},
|
|
65
73
|
"publishConfig": {
|
|
66
74
|
"tag": "latest"
|
|
67
75
|
}
|
package/presets/README.md
CHANGED
|
@@ -45,9 +45,9 @@ opencode/glm-5.3-flash-free 1
|
|
|
45
45
|
|
|
46
46
|
`#` starts a comment (whole-line or trailing), blank lines are ignored, and the
|
|
47
47
|
weight is optional (default 1). The rules are exactly the same as the JSON form
|
|
48
|
-
below:
|
|
49
|
-
accepts either shape in the same file, so a leading
|
|
50
|
-
else means the flat form.
|
|
48
|
+
below: any number of slots, a model may repeat on another line, split on the
|
|
49
|
+
first slash only. The loader accepts either shape in the same file, so a leading
|
|
50
|
+
`{` means JSON and anything else means the flat form.
|
|
51
51
|
|
|
52
52
|
## Schema
|
|
53
53
|
|
|
@@ -63,7 +63,7 @@ else means the flat form.
|
|
|
63
63
|
| Field | Required | Meaning |
|
|
64
64
|
|-------|----------|---------|
|
|
65
65
|
| `model` | yes | `provider/model-id`, split on the **first** slash only |
|
|
66
|
-
| `weight` | no | active-session budget
|
|
66
|
+
| `weight` | no | active-session budget; the pool's capacity is the sum of the weights. Absent = 1 |
|
|
67
67
|
|
|
68
68
|
`$comment` is a JSON-legal array of strings and is ignored by the loader; the
|
|
69
69
|
bundled preset uses it to explain itself in place.
|
|
@@ -74,13 +74,13 @@ The loader checks shape and nothing else:
|
|
|
74
74
|
|
|
75
75
|
- `slots` must be a non-empty array
|
|
76
76
|
- each `model` must contain a `/`
|
|
77
|
-
-
|
|
77
|
+
- a model may appear on more than one line; each line is its own slot with its
|
|
78
|
+
own load counter and weight
|
|
78
79
|
- `weight` must be a number in `(0, 100]`; a bad one **warns and uses 1** rather
|
|
79
80
|
than dropping the slot, because losing a model to a typo in a count costs real
|
|
80
81
|
capacity
|
|
81
|
-
-
|
|
82
|
-
|
|
83
|
-
would route at an agent that does not exist
|
|
82
|
+
- there is no slot cap: one hidden variant agent is declared per base per
|
|
83
|
+
resolved slot, and the watcher declares a variant if a reload grows the table
|
|
84
84
|
- `tier` is accepted as an alias for `weight` (`primary` → 2,
|
|
85
85
|
`overflow` → 1) and warns
|
|
86
86
|
|
|
@@ -92,20 +92,18 @@ because the pool requires it.
|
|
|
92
92
|
|
|
93
93
|
## Weight, precisely
|
|
94
94
|
|
|
95
|
-
A weight is a slot's active-session budget
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
95
|
+
A weight is a slot's active-session budget: the number of concurrent claims it
|
|
96
|
+
may hold before the next slot is preferred. The pool's capacity is the **sum of
|
|
97
|
+
the weights**, so a weight-3 slot may hold 3 concurrent sessions and listing a
|
|
98
|
+
model on two lines gives it two independent budgets.
|
|
99
99
|
|
|
100
|
-
Routing
|
|
100
|
+
Routing is one pass:
|
|
101
101
|
|
|
102
|
-
1.
|
|
103
|
-
2. once every usable slot has reached its weight, pass 2 repeats the order,
|
|
104
|
-
filling each usable slot up to twice its weight
|
|
102
|
+
1. fill each usable slot up to its weight, in declaration order
|
|
105
103
|
|
|
106
|
-
A slot is skipped when it is cooling
|
|
107
|
-
|
|
108
|
-
|
|
104
|
+
A slot is skipped when it is cooling or unavailable. When every usable slot is
|
|
105
|
+
at its weight, the pool is at capacity and overflow goes to the least loaded
|
|
106
|
+
usable slot; the pool never refuses work.
|
|
109
107
|
|
|
110
108
|
Live claims decide the next slot: a released claim immediately reopens its
|
|
111
109
|
slot, and there is no stored position.
|
package/presets/free-tier.json
CHANGED
|
@@ -9,25 +9,23 @@
|
|
|
9
9
|
"Slot N is the same model for every base agent, which is what makes load shared.",
|
|
10
10
|
"",
|
|
11
11
|
"RULES, all validated at startup: models are written provider/model-id, split on the FIRST",
|
|
12
|
-
"slash only so openrouter's vendor-prefixed ids (nvidia/x:free) resolve
|
|
13
|
-
"
|
|
14
|
-
"
|
|
12
|
+
"slash only so openrouter's vendor-prefixed ids (nvidia/x:free) resolve. A model may",
|
|
13
|
+
"appear on more than one line: each line is its own slot with its own budget.",
|
|
14
|
+
"ANY NUMBER OF SLOTS: one hidden variant per base per slot is declared from the resolved",
|
|
15
|
+
"table, and the watcher declares a variant when a reload grows the table. There is no slot cap.",
|
|
15
16
|
"",
|
|
16
17
|
"WEIGHT IS THE ACTIVE-SESSION BUDGET FOR ONE PASS: a slot may hold up to its weight in",
|
|
17
18
|
"concurrent claims before the next slot is preferred. Slots are filled in declaration",
|
|
18
19
|
"order; preference is order, never weight.",
|
|
19
|
-
"Routing
|
|
20
|
-
"
|
|
21
|
-
"
|
|
22
|
-
"
|
|
23
|
-
"A slot is skipped when it is cooling, unavailable, or at its absolute cap (twice its",
|
|
24
|
-
"weight). When every usable slot is at its absolute cap, overflow goes to the least",
|
|
25
|
-
"loaded usable slot; the pool never refuses work.",
|
|
20
|
+
"Routing is ONE ordered pass: each usable slot fills up to its weight, in declaration",
|
|
21
|
+
"order. A slot is skipped when it is cooling or unavailable. When every usable slot is at",
|
|
22
|
+
"its weight, the pool is at capacity (the sum of the weights) and overflow goes to the",
|
|
23
|
+
"least-loaded usable slot; the pool never refuses work.",
|
|
26
24
|
"Live claims decide the next slot: a released claim immediately reopens its slot, and",
|
|
27
25
|
"there is no stored position.",
|
|
28
26
|
"",
|
|
29
|
-
"With this file: weights 3,2,1,1 give 7 active sessions in
|
|
30
|
-
"
|
|
27
|
+
"With this file: weights 3,2,1,1 give 7 active sessions in total. To give a model more",
|
|
28
|
+
"capacity, list it again on another line with its own weight.",
|
|
31
29
|
"Equal weights give a plain pass in list order.",
|
|
32
30
|
"",
|
|
33
31
|
"DECLARATION ORDER IS PREFERENCE. Slots are filled in declaration order, so the slot",
|
|
@@ -44,7 +42,7 @@
|
|
|
44
42
|
"List order sets preference: earlier slots are preferred first, so put the slots you",
|
|
45
43
|
"want worked first at the top of the list.",
|
|
46
44
|
"",
|
|
47
|
-
"COST IS THE DEFAULT, NOT A CONSTRAINT. These
|
|
45
|
+
"COST IS THE DEFAULT, NOT A CONSTRAINT. These slots are free because that is what",
|
|
48
46
|
"the pool is FOR: mechanical work -- reading files, grepping, mechanical edits, cartography.",
|
|
49
47
|
"They are not here to be the smartest model in the system. The point of putting that work",
|
|
50
48
|
"on a near-free model is that the mechanical tokens stop being spent by your expensive",
|
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// LaCode free-model scorer — MEASURE, DO NOT FABRICATE.
|
|
3
|
+
//
|
|
4
|
+
// The pool is for mechanical work: reading files, grepping, mechanical edits,
|
|
5
|
+
// cartography. What matters for that is not "the smartest model" but a model that
|
|
6
|
+
// is (a) free, (b) awake, (c) fast, and (d) has enough context and output room.
|
|
7
|
+
// This script measures those four things and ranks the candidates.
|
|
8
|
+
//
|
|
9
|
+
// npm run score all configured free providers
|
|
10
|
+
// npm run score -- --providers opencode,openrouter
|
|
11
|
+
// npm run score -- --max 6 probe at most 6 models per provider
|
|
12
|
+
// npm run score -- --no-write print only, touch nothing
|
|
13
|
+
//
|
|
14
|
+
// WHAT IT MEASURES
|
|
15
|
+
// context limit.context from `opencode models --verbose`
|
|
16
|
+
// output limit.output (the token ceiling for one reply)
|
|
17
|
+
// speed wall-clock ms for one tiny completion, one request per model
|
|
18
|
+
// reliability did the model actually answer? A 429/401/404 is recorded as the
|
|
19
|
+
// named failure it is, never as a slow success.
|
|
20
|
+
//
|
|
21
|
+
// WHAT IT DOES NOT DO
|
|
22
|
+
// * It never invents a latency or a score. A model it could not reach is shown
|
|
23
|
+
// as unreachable with the reason and is NOT ranked.
|
|
24
|
+
// * It never writes pool.txt. It prints a SUGGESTED table for you to paste; the
|
|
25
|
+
// pool stays yours to decide.
|
|
26
|
+
// * It never probes a paid model. Only cost 0/0 models from the provider's
|
|
27
|
+
// catalog are candidates.
|
|
28
|
+
//
|
|
29
|
+
// OUTPUT
|
|
30
|
+
// a per-provider ranked table, an overall ranking, a suggested pool.txt block,
|
|
31
|
+
// and (unless --no-write) the raw rows in ~/.config/lacode/model-scores.json so
|
|
32
|
+
// a later run can compare.
|
|
33
|
+
|
|
34
|
+
import { execFileSync } from "node:child_process"
|
|
35
|
+
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"
|
|
36
|
+
import { homedir } from "node:os"
|
|
37
|
+
import { dirname, join } from "node:path"
|
|
38
|
+
|
|
39
|
+
const AUTH = join(homedir(), ".local", "share", "opencode", "auth.json")
|
|
40
|
+
const OUT = join(homedir(), ".config", "lacode", "model-scores.json")
|
|
41
|
+
const DEFAULT_PROVIDERS = ["opencode", "opencode-go", "openrouter", "console-openrouter"]
|
|
42
|
+
|
|
43
|
+
// ── scoring weights ─────────────────────────────────────────────────────────
|
|
44
|
+
// Mechanical work prizes speed and availability over raw capability. Context is
|
|
45
|
+
// scored on a log curve so 256k vs 1M is a difference, not a cliff.
|
|
46
|
+
const W = { speed: 0.35, context: 0.3, output: 0.15, reliability: 0.2 }
|
|
47
|
+
const MAX_CONTEXT = 1_000_000
|
|
48
|
+
const FAST_MS = 300 // at or below this, speed scores 1.0
|
|
49
|
+
const SLOW_MS = 8_000 // at or above this, speed scores 0
|
|
50
|
+
const PROMPT = "Reply with the single word: ok"
|
|
51
|
+
const MAX_TOKENS = 8
|
|
52
|
+
|
|
53
|
+
// ── args ────────────────────────────────────────────────────────────────────
|
|
54
|
+
const argv = process.argv.slice(2)
|
|
55
|
+
const has = (flag) => argv.includes(flag)
|
|
56
|
+
const value = (flag, fallback) => {
|
|
57
|
+
const hit = argv.find((a) => a === flag || a.startsWith(`${flag}=`))
|
|
58
|
+
if (!hit) return fallback
|
|
59
|
+
return hit.includes("=") ? hit.slice(hit.indexOf("=") + 1) : argv[argv.indexOf(hit) + 1]
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
if (has("--help") || has("-h")) {
|
|
63
|
+
console.log(`Usage: npm run score -- [options]
|
|
64
|
+
|
|
65
|
+
--providers a,b providers to scan (default: ${DEFAULT_PROVIDERS.join(",")})
|
|
66
|
+
--max N probe at most N free models per provider (default: 12)
|
|
67
|
+
--timeout MS per-request deadline in ms (default: 20000)
|
|
68
|
+
--delay MS pause between requests in ms (default: 300)
|
|
69
|
+
--no-write do not write ~/.config/lacode/model-scores.json
|
|
70
|
+
-h, --help this message
|
|
71
|
+
|
|
72
|
+
Only cost 0/0 models are candidates. Credentials come from
|
|
73
|
+
~/.local/share/opencode/auth.json; a provider with no entry falls back to the
|
|
74
|
+
"opencode" key (which is how the console-openrouter connection authenticates).`)
|
|
75
|
+
process.exit(0)
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
const providers = String(value("--providers", DEFAULT_PROVIDERS.join(","))).split(",").map((s) => s.trim()).filter(Boolean)
|
|
79
|
+
const maxPerProvider = Math.max(1, Number(value("--max", "12")) || 12)
|
|
80
|
+
const timeoutMs = Math.max(1_000, Number(value("--timeout", "20000")) || 20_000)
|
|
81
|
+
const delayMs = Math.max(0, Number(value("--delay", "300")) || 300)
|
|
82
|
+
const writeResults = !has("--no-write")
|
|
83
|
+
|
|
84
|
+
// ── discovery ───────────────────────────────────────────────────────────────
|
|
85
|
+
/** Parse `opencode models --verbose <provider>`: alternating id line + JSON block. */
|
|
86
|
+
function parseVerbose(text) {
|
|
87
|
+
const rows = []
|
|
88
|
+
let id = null
|
|
89
|
+
let buf = []
|
|
90
|
+
const flush = () => {
|
|
91
|
+
if (!id || buf.length === 0) return false
|
|
92
|
+
try {
|
|
93
|
+
const parsed = JSON.parse(buf.join("\n"))
|
|
94
|
+
rows.push({ id, model: parsed })
|
|
95
|
+
id = null
|
|
96
|
+
buf = []
|
|
97
|
+
return true
|
|
98
|
+
} catch {
|
|
99
|
+
return false
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
for (const line of text.split(/\r?\n/)) {
|
|
103
|
+
const isId = /^\S/.test(line) && !/^[{}]/.test(line)
|
|
104
|
+
if (isId) {
|
|
105
|
+
if (id) flush()
|
|
106
|
+
if (line.startsWith("provider/") || line.includes("/")) {
|
|
107
|
+
id = line.trim()
|
|
108
|
+
buf = []
|
|
109
|
+
}
|
|
110
|
+
continue
|
|
111
|
+
}
|
|
112
|
+
if (id) {
|
|
113
|
+
buf.push(line)
|
|
114
|
+
const trimmed = line.trim()
|
|
115
|
+
if (trimmed === "}" || trimmed === "},") flush()
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
flush()
|
|
119
|
+
return rows
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
function freeModels(provider) {
|
|
123
|
+
let text
|
|
124
|
+
try {
|
|
125
|
+
text = execFileSync("opencode", ["models", "--verbose", provider], {
|
|
126
|
+
encoding: "utf8",
|
|
127
|
+
maxBuffer: 64 * 1024 * 1024,
|
|
128
|
+
stdio: ["ignore", "pipe", "ignore"],
|
|
129
|
+
})
|
|
130
|
+
} catch (error) {
|
|
131
|
+
return { provider, error: `opencode models failed: ${String(error?.message ?? error)}`, models: [] }
|
|
132
|
+
}
|
|
133
|
+
const rows = parseVerbose(text)
|
|
134
|
+
const models = rows
|
|
135
|
+
.filter(({ model }) => model?.api?.url && model?.api?.id)
|
|
136
|
+
.filter(({ model }) => (model.cost?.input ?? 1) === 0 && (model.cost?.output ?? 1) === 0)
|
|
137
|
+
.filter(({ model }) => !model.status || model.status === "active")
|
|
138
|
+
.map(({ id, model }) => ({
|
|
139
|
+
model: id,
|
|
140
|
+
url: model.api.url,
|
|
141
|
+
apiId: model.api.id,
|
|
142
|
+
context: model.limit?.context ?? null,
|
|
143
|
+
output: model.limit?.output ?? null,
|
|
144
|
+
name: model.name ?? id,
|
|
145
|
+
}))
|
|
146
|
+
return { provider, error: null, models }
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
// ── credentials ─────────────────────────────────────────────────────────────
|
|
150
|
+
function loadAuth() {
|
|
151
|
+
if (!existsSync(AUTH)) return {}
|
|
152
|
+
try {
|
|
153
|
+
return JSON.parse(readFileSync(AUTH, "utf8"))
|
|
154
|
+
} catch {
|
|
155
|
+
return {}
|
|
156
|
+
}
|
|
157
|
+
}
|
|
158
|
+
const auth = loadAuth()
|
|
159
|
+
const keyFor = (provider) => auth[provider]?.key ?? auth.opencode?.key ?? null
|
|
160
|
+
// The Zen gateways reject a request that does not name a session; the value does
|
|
161
|
+
// not need to be a real opencode session, it only has to be present.
|
|
162
|
+
const runSession = `score-${Date.now().toString(36)}`
|
|
163
|
+
const sessionHeader = (provider) =>
|
|
164
|
+
provider === "opencode" || provider === "opencode-go" ? { "x-opencode-session": runSession } : {}
|
|
165
|
+
|
|
166
|
+
// ── probing ─────────────────────────────────────────────────────────────────
|
|
167
|
+
const sleep = (ms) => new Promise((r) => setTimeout(r, ms))
|
|
168
|
+
|
|
169
|
+
/**
|
|
170
|
+
* One real request. Returns {ok, ms, reason}. A failure is named, never rounded
|
|
171
|
+
* into a success, and `ms` is only reported for a request that actually ran.
|
|
172
|
+
*/
|
|
173
|
+
async function probe(candidate, provider) {
|
|
174
|
+
const key = keyFor(provider)
|
|
175
|
+
if (!key) return { ok: false, ms: null, reason: "no credential in auth.json" }
|
|
176
|
+
|
|
177
|
+
const url = `${String(candidate.url).replace(/\/+$/, "")}/chat/completions`
|
|
178
|
+
const started = performance.now()
|
|
179
|
+
try {
|
|
180
|
+
const response = await fetch(url, {
|
|
181
|
+
method: "POST",
|
|
182
|
+
headers: {
|
|
183
|
+
"Content-Type": "application/json",
|
|
184
|
+
Authorization: `Bearer ${key}`,
|
|
185
|
+
...sessionHeader(provider),
|
|
186
|
+
...(provider === "openrouter" ? { "HTTP-Referer": "https://opencode.ai", "X-Title": "LaCode score" } : {}),
|
|
187
|
+
},
|
|
188
|
+
body: JSON.stringify({
|
|
189
|
+
model: candidate.apiId,
|
|
190
|
+
messages: [{ role: "user", content: PROMPT }],
|
|
191
|
+
max_tokens: MAX_TOKENS,
|
|
192
|
+
temperature: 0,
|
|
193
|
+
stream: false,
|
|
194
|
+
}),
|
|
195
|
+
signal: AbortSignal.timeout(timeoutMs),
|
|
196
|
+
})
|
|
197
|
+
const ms = Math.round(performance.now() - started)
|
|
198
|
+
const body = await response.text()
|
|
199
|
+
let parsed = null
|
|
200
|
+
try {
|
|
201
|
+
parsed = JSON.parse(body)
|
|
202
|
+
} catch {}
|
|
203
|
+
|
|
204
|
+
if (!response.ok) {
|
|
205
|
+
const detail = parsed?.error?.message ?? body.slice(0, 160)
|
|
206
|
+
const kind =
|
|
207
|
+
response.status === 429
|
|
208
|
+
? "rate-limited"
|
|
209
|
+
: response.status === 401 || response.status === 403
|
|
210
|
+
? "credential"
|
|
211
|
+
: response.status === 404
|
|
212
|
+
? "not served"
|
|
213
|
+
: `http ${response.status}`
|
|
214
|
+
return { ok: false, ms, reason: `${kind}: ${detail}` }
|
|
215
|
+
}
|
|
216
|
+
// A 200 can still carry an error object (OpenRouter does this).
|
|
217
|
+
if (parsed?.error) return { ok: false, ms, reason: `${parsed.error.message ?? "error"} `.trim() }
|
|
218
|
+
if (!Array.isArray(parsed?.choices) || parsed.choices.length === 0) {
|
|
219
|
+
return { ok: false, ms, reason: "no choices in response" }
|
|
220
|
+
}
|
|
221
|
+
return { ok: true, ms, reason: "answered" }
|
|
222
|
+
} catch (error) {
|
|
223
|
+
const ms = Math.round(performance.now() - started)
|
|
224
|
+
const name = error?.name === "TimeoutError" ? "timeout" : String(error?.message ?? error)
|
|
225
|
+
return { ok: false, ms, reason: name }
|
|
226
|
+
}
|
|
227
|
+
}
|
|
228
|
+
|
|
229
|
+
// ── scoring ─────────────────────────────────────────────────────────────────
|
|
230
|
+
const clamp01 = (n) => Math.max(0, Math.min(1, n))
|
|
231
|
+
const logScore = (n, max) => (n && n > 0 ? clamp01(Math.log2(1 + n) / Math.log2(1 + max)) : 0)
|
|
232
|
+
const speedScore = (ms) => (ms == null ? 0 : clamp01((SLOW_MS - ms) / (SLOW_MS - FAST_MS)))
|
|
233
|
+
|
|
234
|
+
function score(row) {
|
|
235
|
+
if (!row.ok) return null
|
|
236
|
+
const parts = {
|
|
237
|
+
speed: speedScore(row.ms),
|
|
238
|
+
context: logScore(row.context, MAX_CONTEXT),
|
|
239
|
+
output: logScore(row.output, 262_144),
|
|
240
|
+
reliability: 1,
|
|
241
|
+
}
|
|
242
|
+
const value = 100 * (Object.entries(W).reduce((sum, [k, w]) => sum + w * parts[k], 0) / Object.values(W).reduce((a, b) => a + b, 0))
|
|
243
|
+
return { value: Math.round(value * 10) / 10, parts }
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
const pad = (s, n) => String(s).padEnd(n)
|
|
247
|
+
const padL = (s, n) => String(s).padStart(n)
|
|
248
|
+
|
|
249
|
+
async function main() {
|
|
250
|
+
const scanned = providers.map(freeModels)
|
|
251
|
+
const results = []
|
|
252
|
+
|
|
253
|
+
for (const entry of scanned) {
|
|
254
|
+
if (entry.error) {
|
|
255
|
+
console.log(`\n${entry.provider}: ${entry.error}`)
|
|
256
|
+
results.push({ provider: entry.provider, error: entry.error, rows: [] })
|
|
257
|
+
continue
|
|
258
|
+
}
|
|
259
|
+
const candidates = entry.models.slice(0, maxPerProvider)
|
|
260
|
+
console.log(`\n${entry.provider}: probing ${candidates.length} free model(s)${entry.models.length > candidates.length ? ` of ${entry.models.length}` : ""}`)
|
|
261
|
+
if (candidates.length === 0) {
|
|
262
|
+
results.push({ provider: entry.provider, error: null, rows: [] })
|
|
263
|
+
continue
|
|
264
|
+
}
|
|
265
|
+
const rows = []
|
|
266
|
+
for (const candidate of candidates) {
|
|
267
|
+
const outcome = await probe(candidate, entry.provider)
|
|
268
|
+
const row = { provider: entry.provider, ...candidate, ...outcome, score: null }
|
|
269
|
+
row.score = score(row)
|
|
270
|
+
rows.push(row)
|
|
271
|
+
const verdict = outcome.ok ? `${outcome.ms}ms` : outcome.reason.slice(0, 60)
|
|
272
|
+
console.log(` ${pad(candidate.model.replace(`${entry.provider}/`, ""), 52)} ${padL(verdict, 16)}`)
|
|
273
|
+
await sleep(delayMs)
|
|
274
|
+
}
|
|
275
|
+
results.push({ provider: entry.provider, error: null, rows })
|
|
276
|
+
}
|
|
277
|
+
|
|
278
|
+
// Per-provider ranked tables.
|
|
279
|
+
for (const entry of results) {
|
|
280
|
+
if (entry.error || entry.rows.length === 0) continue
|
|
281
|
+
console.log(`\n${entry.provider} — ranked`)
|
|
282
|
+
console.log(` ${pad("#", 3)}${pad("model", 52)}${padL("ctx", 9)}${padL("out", 9)}${padL("ms", 7)}${padL("score", 7)}`)
|
|
283
|
+
const ranked = [...entry.rows].filter((r) => r.score).sort((a, b) => b.score.value - a.score.value)
|
|
284
|
+
ranked.forEach((r, i) => {
|
|
285
|
+
console.log(` ${pad(i + 1, 3)}${pad(r.model.replace(`${entry.provider}/`, ""), 52)}${padL(r.context ?? "?", 9)}${padL(r.output ?? "?", 9)}${padL(r.ms ?? "?", 7)}${padL(r.score.value, 7)}`)
|
|
286
|
+
})
|
|
287
|
+
const failed = entry.rows.filter((r) => !r.ok)
|
|
288
|
+
if (failed.length) {
|
|
289
|
+
console.log(` unreachable (not ranked):`)
|
|
290
|
+
for (const r of failed) console.log(` ${pad(r.model.replace(`${entry.provider}/`, ""), 52)} ${r.reason.slice(0, 90)}`)
|
|
291
|
+
}
|
|
292
|
+
}
|
|
293
|
+
|
|
294
|
+
// Overall ranking across providers.
|
|
295
|
+
const all = results.flatMap((e) => e.rows).filter((r) => r.score)
|
|
296
|
+
all.sort((a, b) => b.score.value - a.score.value)
|
|
297
|
+
if (all.length) {
|
|
298
|
+
console.log(`\noverall — best free models right now`)
|
|
299
|
+
all.slice(0, 12).forEach((r, i) => {
|
|
300
|
+
console.log(` ${pad(i + 1, 3)}${pad(r.model, 58)}${padL(`${r.ms}ms`, 8)}${padL(r.score.value, 7)}`)
|
|
301
|
+
})
|
|
302
|
+
}
|
|
303
|
+
|
|
304
|
+
// A SUGGESTED pool: the best model from each provider, weights 4/3/2/1 by rank.
|
|
305
|
+
// Printed, never written — pool.txt stays the user's decision.
|
|
306
|
+
const bestPerProvider = []
|
|
307
|
+
for (const entry of results) {
|
|
308
|
+
const top = entry.rows.filter((r) => r.score).sort((a, b) => b.score.value - a.score.value)[0]
|
|
309
|
+
if (top) bestPerProvider.push(top)
|
|
310
|
+
}
|
|
311
|
+
bestPerProvider.sort((a, b) => b.score.value - a.score.value)
|
|
312
|
+
if (bestPerProvider.length) {
|
|
313
|
+
console.log(`\nsuggested pool.txt (paste if it matches your judgement — nothing was changed)`)
|
|
314
|
+
console.log(`# scored ${new Date().toISOString()} — one best free model per provider`)
|
|
315
|
+
bestPerProvider.forEach((r, i) => {
|
|
316
|
+
const weight = Math.max(1, 4 - i)
|
|
317
|
+
console.log(`${pad(r.model, 60)}${weight}`)
|
|
318
|
+
})
|
|
319
|
+
}
|
|
320
|
+
|
|
321
|
+
if (writeResults) {
|
|
322
|
+
try {
|
|
323
|
+
mkdirSync(dirname(OUT), { recursive: true })
|
|
324
|
+
const payload = {
|
|
325
|
+
generatedAt: new Date().toISOString(),
|
|
326
|
+
weights: W,
|
|
327
|
+
providers: results.map((e) => ({
|
|
328
|
+
provider: e.provider,
|
|
329
|
+
error: e.error,
|
|
330
|
+
rows: e.rows.map((r) => ({
|
|
331
|
+
model: r.model,
|
|
332
|
+
context: r.context,
|
|
333
|
+
output: r.output,
|
|
334
|
+
ok: r.ok,
|
|
335
|
+
ms: r.ms,
|
|
336
|
+
reason: r.reason,
|
|
337
|
+
score: r.score?.value ?? null,
|
|
338
|
+
})),
|
|
339
|
+
})),
|
|
340
|
+
}
|
|
341
|
+
writeFileSync(OUT, JSON.stringify(payload, null, 2) + "\n")
|
|
342
|
+
console.log(`\nwrote ${OUT}`)
|
|
343
|
+
} catch (error) {
|
|
344
|
+
console.error(`\ncould not write ${OUT}: ${String(error?.message ?? error)}`)
|
|
345
|
+
process.exitCode = 1
|
|
346
|
+
}
|
|
347
|
+
}
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
main().catch((error) => {
|
|
351
|
+
console.error(String(error?.stack ?? error))
|
|
352
|
+
process.exitCode = 1
|
|
353
|
+
})
|