@kontextmind/kxm 0.7.198 → 0.7.200
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/docs/README.md +1 -0
- package/docs/audits/2026-10-gap-audit.md +23 -7
- package/docs/contributing/ci-and-release.md +42 -15
- package/docs/evals/writer-bakeoff-2026-10/prices.json +244 -0
- package/docs/evals/writer-bakeoff-2026-10/score.py +163 -0
- package/docs/evals/writer-bakeoff-2026-10/tasks.json +67 -0
- package/docs/evals/writer-bakeoff-2026-10.md +123 -0
- package/package.json +1 -1
- package/plugins/kxm/.claude-plugin/plugin.json +1 -1
- package/plugins/kxm/dist/mcp-server.js +1 -1
- package/plugins/kxm/package.json +1 -1
- package/plugins/kxm/src/mcp-server.ts +1 -1
package/docs/README.md
CHANGED
|
@@ -121,6 +121,7 @@ KXM connects coding agents through a durable, authenticated [hub](glossary.md#hu
|
|
|
121
121
|
| [Decisions](contributing/decisions.md) | Agents, maintainers | Medium-or-higher decisions from automated work, with weights and evidence |
|
|
122
122
|
| [Learnings](contributing/learnings.md) | Agents, maintainers | Durable lessons from running the writer, landing, and docs loops |
|
|
123
123
|
| [Planner bake-off, October 2026](evals/planner-bakeoff-2026-10.md) | Maintainers | Scored plans from two planner models on six KXM tasks, with cost and time |
|
|
124
|
+
| [Writer bake-off, October 2026](evals/writer-bakeoff-2026-10.md) | Maintainers | Unscored comparison of Beam 1.0, Mercury 3, and Qwen3.7 Plus for writer number two |
|
|
124
125
|
| [October 2026 gap audit](audits/2026-10-gap-audit.md) | Maintainers | What the 2026-10-21 pass fixed, and the decisions still waiting on the operator |
|
|
125
126
|
| [Artifact templates](templates/README.md) | Workflow authors | Document templates and where workflows use them |
|
|
126
127
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# October 2026 gap audit
|
|
2
2
|
|
|
3
|
-
This page records the gap audit of this repository on 2026-10-21, against `main` at `19ec8a5`.
|
|
3
|
+
This page records the gap audit of this repository on 2026-10-21, against `main` at `19ec8a5`. Eddie approved decision 1, runner placement, on 2026-10-21. Route admission and the witnesses already listed in Tracking still need the operator. The four code fixes from this pass are separate pull requests.
|
|
4
4
|
|
|
5
5
|
The pass compared the CLI, routing, and CI workflows with the docs, the tests, and `.kxm/knowledge`. It left `.kxm/prices.yaml` and `.github/workflows/refresh-models.yml` untouched because [pull request #410](https://github.com/kontextmind/kxm/pull/410) owns those files. Playwright stays on Obscura. [Playwright e2e](https://github.com/kontextmind/kxm/blob/main/.github/workflows/e2e.yml) already runs `npm run e2e` on `ubuntu-latest`.
|
|
6
6
|
|
|
@@ -29,7 +29,7 @@ Product TypeScript and Python sources have no `TODO` or `FIXME` markers outside
|
|
|
29
29
|
|
|
30
30
|
Phase pages under `docs/roadmap/phases/` are symlinks to `plans/kxm-roadmap/phases/`. Plan links such as `../../plan-lane-cli.md` resolve from the real file. A scan that does not follow the symlink reports nine missing targets under `docs/roadmap/phases/`. The generator was not changed.
|
|
31
31
|
|
|
32
|
-
`.kxm/knowledge` has ten pages, last indexed 2026-10-17. It has no first-run page
|
|
32
|
+
`.kxm/knowledge` has ten pages, last indexed 2026-10-17. It has no first-run page. [CI conventions](../../.kxm/knowledge/patterns/ci.md) now records the runner split from decision 1.
|
|
33
33
|
|
|
34
34
|
## Decisions for the operator
|
|
35
35
|
|
|
@@ -37,17 +37,33 @@ Ranked by how often they block a pull request or a first run. Each one needs a s
|
|
|
37
37
|
|
|
38
38
|
### 1. Where pull-request Linux jobs run
|
|
39
39
|
|
|
40
|
-
|
|
40
|
+
**Approved by Eddie on 2026-10-21.** On a pull request, these jobs use GitHub-hosted `ubuntu-latest`:
|
|
41
41
|
|
|
42
|
-
|
|
42
|
+
- Classify changes
|
|
43
|
+
- Docs lint
|
|
44
|
+
- Linux Node 24 unit lanes `serial`, `light-1`, and `light-2`
|
|
45
|
+
- Plugin validation
|
|
46
|
+
- `required`
|
|
43
47
|
|
|
44
|
-
|
|
48
|
+
The same five jobs stay on `kontextmind-doks` for a push to `main` and for `workflow_dispatch`. Release, nightly, and smoke stay on `kontextmind-doks`. Windows unit lanes stay on `windows-latest`. `km-gh-rn01` was not given the `kontextmind-doks` label.
|
|
49
|
+
|
|
50
|
+
Python MG1 stays on `kontextmind-doks`. `scripts/ci-postgres-17.sh` installs a pinned PostgreSQL 17.11.0 linux/x86_64 archive and maps ICU 74 to the `libicu74` apt package. A newer `ubuntu-latest` image that drops `libicu74` fails that archive until the pin moves. A GitHub-hosted service container would be a different script, so this change does not move Python.
|
|
51
|
+
|
|
52
|
+
The Linux engine shards stay on `kontextmind-doks`. They are the CPU-bound split of `engine.test.ts`. The scale set requests three CPUs so those jobs are not packed onto busy nodes. They do not use a container socket, and they were not in the set that can leave the cluster without a capacity question.
|
|
53
|
+
|
|
54
|
+
Linux Validate on `main` stays on `kontextmind-doks` with the three-minute timeout. On the push for `e63907a` ([run 39547915383](https://github.com/kontextmind/kxm/actions/runs/39547915383)), Validate (linux, Node 22.19.0) ran from 2026-10-21T05:16:57Z to 05:18:06Z and Validate (linux, Node 24) ran from 05:25:47Z to 05:26:56Z, about 69 seconds each after a runner started. That budget is tight for a cold `npm ci` on a GitHub-hosted runner. This job does not use one, so the timeout stays at three minutes. Windows Validate stays at fifteen minutes.
|
|
55
|
+
|
|
56
|
+
`test/core/ci-contract.test.ts` allows `ubuntu-latest` only for the five pull-request jobs above, and only inside the `pull_request` arm of the `runs-on` expression. It still refuses `ubuntu-latest` in `release.yml`, `smoke.yml`, and `nightly.yml`, and it still refuses a bare `ubuntu-latest` selector on any other `ci.yml` job.
|
|
57
|
+
|
|
58
|
+
Before this move, every pull-request Linux job selected `kontextmind-doks`. Windows unit lanes already used `windows-latest`, and only when the classifier marked the change platform-sensitive. The Validate matrix did not run on a pull request.
|
|
59
|
+
|
|
60
|
+
[CI and release](../contributing/ci-and-release.md) records the capacity policy: one warm runner, a burst to four, three CPUs each. The Python lane stays on `kontextmind-doks` because that scale set has no container-engine socket.
|
|
61
|
+
|
|
62
|
+
Observed on 2026-10-21, before the runner split:
|
|
45
63
|
|
|
46
64
|
- Workflow [39513962307](https://github.com/kontextmind/kxm/actions/runs/39513962307) (pull request #410, platform-sensitive) was created at 2026-10-20T23:16:53Z. Docs lint started at 2026-10-21T00:02:27Z, about 46 minutes later. Windows lanes started at 00:03:51Z, within seconds of Classify. Linux lanes then ran mostly one at a time through about 00:20Z.
|
|
47
65
|
- Workflow [39510255392](https://github.com/kontextmind/kxm/actions/runs/39510255392) (planner fallback, code pull request) was created at 2026-10-20T22:44:14Z and completed at 2026-10-20T23:08:16Z, about 24 minutes, while the scale set was less busy.
|
|
48
66
|
|
|
49
|
-
Classify, Docs lint, the Node unit lanes, Plugin validation, and `required` do not need the cluster. Moving those jobs to `ubuntu-latest` would drop the queue and would also edit the contract test, spend GitHub-hosted minutes, and reverse the pinned-runner rule. Python can stay on the scale set, or move to a GitHub-hosted service container, which is a different script than the pinned archive. A newer image that no longer ships `libicu74` fails that archive until the pin moves. Linux Validate on `main` has a three-minute timeout that is tight for a cold `npm ci` on a GitHub-hosted runner. Do not relabel `km-gh-rn01`. That bypasses ARC and serializes the queue.
|
|
50
|
-
|
|
51
67
|
### 2. The built-in vision route
|
|
52
68
|
|
|
53
69
|
`kxm vision assert` still defaults to `zai-coding-cn/glm-5.3-flash` and, for that id only, still runs `pi --provider zai-coding-cn --model glm-5.3-flash`. `.kxm/routes.yaml` removed `zai-coding-cn/*` on 2026-09-27 because that provider is not allowlisted. The admitted replacement in this checkout is `openrouter/z-ai/glm-5.3-flash`. A vision call here with no `--route` still fail-closes, and pull request #412 names `kxm routes admit` in that message. `kxm init` does not admit a vision route.
|
|
@@ -27,11 +27,14 @@ flowchart LR
|
|
|
27
27
|
MAIN -->|"daily 04:00 UTC"| NIGHT[Nightly<br/>complete suite]
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
-
All workflows live in `.github/workflows/`.
|
|
31
|
-
|
|
32
|
-
`
|
|
33
|
-
|
|
34
|
-
|
|
30
|
+
All workflows live in `.github/workflows/`. On a pull request, Classify
|
|
31
|
+
changes, Docs lint, the Linux unit lanes, Plugin validation, and `required`
|
|
32
|
+
use GitHub-hosted `ubuntu-latest`. Engine shards, Python MG1, a push to
|
|
33
|
+
`main`, `workflow_dispatch`, release, nightly, and smoke stay on the ARC
|
|
34
|
+
scale set `kontextmind-doks`. Validate also runs two Windows legs on
|
|
35
|
+
`windows-latest`. `test/core/ci-contract.test.ts` pins the runner selectors,
|
|
36
|
+
job names, coverage floors, and release triggers. Change a workflow and that
|
|
37
|
+
test together.
|
|
35
38
|
|
|
36
39
|
## What runs where
|
|
37
40
|
|
|
@@ -88,10 +91,15 @@ Code pull requests run Docs lint, two Linux Node 24 engine shards, a serial
|
|
|
88
91
|
lane (`permission.test.ts`, then `runtime.test.ts`, then
|
|
89
92
|
`package-install.test.ts`), two light shards for every other Node unit file,
|
|
90
93
|
the Python 3.13 control-plane suite against PostgreSQL 17, and Plugin
|
|
91
|
-
validation.
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
94
|
+
validation.
|
|
95
|
+
|
|
96
|
+
On that pull request, Docs lint, the serial lane, both light shards, Plugin
|
|
97
|
+
validation, and `required` use `ubuntu-latest`. The engine shards stay on
|
|
98
|
+
`kontextmind-doks` with the Python lane. The Python lane stays on
|
|
99
|
+
`kontextmind-doks`. That scale set has no container-engine socket, so the
|
|
100
|
+
job starts PostgreSQL 17 with `scripts/ci-postgres-17.sh` on `127.0.0.1:5432`
|
|
101
|
+
and uses the same admin and test DSNs as the contributor procedure in
|
|
102
|
+
[Develop KXM](development.md).
|
|
95
103
|
`light-1` typechecks and checks generated bundles.
|
|
96
104
|
`engine.test.ts` is split by test name because that file alone was 174
|
|
97
105
|
seconds; the serial lane keeps the next longest files off the light pool,
|
|
@@ -127,14 +135,33 @@ scale set `kontextmind-doks` with a three-minute job timeout. Windows uses
|
|
|
127
135
|
GitHub-hosted `windows-latest` with a fifteen-minute timeout so `npm ci` can
|
|
128
136
|
finish. Those four job names are not the required check anymore.
|
|
129
137
|
|
|
138
|
+
The Linux timeout stays at three minutes because that leg stays on the scale
|
|
139
|
+
set. A cold `npm ci` on a GitHub-hosted runner would need a longer budget,
|
|
140
|
+
and this job does not use one.
|
|
141
|
+
|
|
142
|
+
### Which jobs use the ARC scale set
|
|
143
|
+
|
|
144
|
+
On a pull request, Classify changes, Docs lint, the Linux unit lanes
|
|
145
|
+
(`serial`, `light-1`, `light-2`), Plugin validation, and `required` use
|
|
146
|
+
GitHub-hosted `ubuntu-latest`. The same five jobs use `kontextmind-doks` on
|
|
147
|
+
a push to `main` and on `workflow_dispatch`.
|
|
148
|
+
|
|
149
|
+
The Linux engine shards and Python MG1 use `kontextmind-doks` on every
|
|
150
|
+
trigger. Python stays there because `scripts/ci-postgres-17.sh` installs a
|
|
151
|
+
pinned PostgreSQL 17.11.0 archive that needs `libicu74`, and a newer
|
|
152
|
+
`ubuntu-latest` image can drop that package. The engine shards stay there
|
|
153
|
+
because they are the CPU-bound split of `engine.test.ts`. The scale set
|
|
154
|
+
requests three CPUs so those jobs are not packed onto busy nodes. Release,
|
|
155
|
+
nightly, and smoke also stay on `kontextmind-doks`.
|
|
156
|
+
|
|
130
157
|
### CI jobs stay queued while a runner is online
|
|
131
158
|
|
|
132
|
-
|
|
133
|
-
name is the scale set, not a custom label on a repository runner. The
|
|
134
|
-
belongs to the selected-repository runner group `KontextMind DOKS
|
|
135
|
-
must allow this public repository. The legacy repository runner
|
|
136
|
-
must not carry the `kontextmind-doks` label
|
|
137
|
-
serializes the build queue.
|
|
159
|
+
Jobs that still select `kontextmind-doks` can queue while a runner is online.
|
|
160
|
+
That name is the scale set, not a custom label on a repository runner. The
|
|
161
|
+
scale set belongs to the selected-repository runner group `KontextMind DOKS
|
|
162
|
+
ARC`, which must allow this public repository. The legacy repository runner
|
|
163
|
+
`km-gh-rn01` must not carry the `kontextmind-doks` label. Adding that label
|
|
164
|
+
bypasses ARC and serializes the build queue. Do not relabel `km-gh-rn01`.
|
|
138
165
|
|
|
139
166
|
Check GitHub routing first:
|
|
140
167
|
|
|
@@ -0,0 +1,244 @@
|
|
|
1
|
+
{
|
|
2
|
+
"source": "https://openrouter.ai/api/v1/models",
|
|
3
|
+
"endpoints_source": "https://openrouter.ai/api/v1/models/{id}/endpoints",
|
|
4
|
+
"fetched_at": "2026-10-21T08:34:25Z",
|
|
5
|
+
"note": "pricing fields are USD per token, as returned by OpenRouter. Multiply by 1e6 for USD per 1M tokens.",
|
|
6
|
+
"models": [
|
|
7
|
+
{
|
|
8
|
+
"id": "reflection/beam-1.0-501b-a23b",
|
|
9
|
+
"name": "Reflection: Beam 1.0 501B A23B",
|
|
10
|
+
"context_length": 1048576,
|
|
11
|
+
"created": 1792160499,
|
|
12
|
+
"pricing": {
|
|
13
|
+
"prompt": "0.0000003",
|
|
14
|
+
"completion": "0.000001",
|
|
15
|
+
"input_cache_read": "0.00000004"
|
|
16
|
+
},
|
|
17
|
+
"supported_parameters": [
|
|
18
|
+
"frequency_penalty",
|
|
19
|
+
"include_reasoning",
|
|
20
|
+
"logit_bias",
|
|
21
|
+
"logprobs",
|
|
22
|
+
"max_tokens",
|
|
23
|
+
"min_p",
|
|
24
|
+
"presence_penalty",
|
|
25
|
+
"reasoning",
|
|
26
|
+
"reasoning_effort",
|
|
27
|
+
"repetition_penalty",
|
|
28
|
+
"response_format",
|
|
29
|
+
"seed",
|
|
30
|
+
"stop",
|
|
31
|
+
"structured_outputs",
|
|
32
|
+
"temperature",
|
|
33
|
+
"tool_choice",
|
|
34
|
+
"tools",
|
|
35
|
+
"top_k",
|
|
36
|
+
"top_logprobs",
|
|
37
|
+
"top_p"
|
|
38
|
+
],
|
|
39
|
+
"architecture": {
|
|
40
|
+
"modality": "text->text"
|
|
41
|
+
}
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
"id": "inception/mercury-3",
|
|
45
|
+
"name": "Inception: Mercury 3",
|
|
46
|
+
"context_length": 1000000,
|
|
47
|
+
"created": 1792158337,
|
|
48
|
+
"pricing": {
|
|
49
|
+
"prompt": "0.0000001",
|
|
50
|
+
"completion": "0.00000035",
|
|
51
|
+
"input_cache_read": "0.00000001"
|
|
52
|
+
},
|
|
53
|
+
"supported_parameters": [
|
|
54
|
+
"include_reasoning",
|
|
55
|
+
"max_tokens",
|
|
56
|
+
"reasoning",
|
|
57
|
+
"reasoning_effort",
|
|
58
|
+
"response_format",
|
|
59
|
+
"stop",
|
|
60
|
+
"structured_outputs",
|
|
61
|
+
"temperature",
|
|
62
|
+
"tool_choice",
|
|
63
|
+
"tools"
|
|
64
|
+
],
|
|
65
|
+
"architecture": {
|
|
66
|
+
"modality": "text->text"
|
|
67
|
+
}
|
|
68
|
+
},
|
|
69
|
+
{
|
|
70
|
+
"id": "qwen/qwen3.7-plus",
|
|
71
|
+
"name": "Qwen: Qwen3.7 Plus",
|
|
72
|
+
"context_length": 1000000,
|
|
73
|
+
"created": 1780491783,
|
|
74
|
+
"pricing": {
|
|
75
|
+
"prompt": "0.00000032",
|
|
76
|
+
"completion": "0.00000128",
|
|
77
|
+
"input_cache_read": "0.000000064",
|
|
78
|
+
"input_cache_write": "0.0000004",
|
|
79
|
+
"overrides": [
|
|
80
|
+
{
|
|
81
|
+
"min_prompt_tokens": 256000,
|
|
82
|
+
"prompt": "0.00000096",
|
|
83
|
+
"completion": "0.00000384",
|
|
84
|
+
"input_cache_read": "0.000000192",
|
|
85
|
+
"input_cache_write": "0.0000012"
|
|
86
|
+
}
|
|
87
|
+
]
|
|
88
|
+
},
|
|
89
|
+
"supported_parameters": [
|
|
90
|
+
"frequency_penalty",
|
|
91
|
+
"include_reasoning",
|
|
92
|
+
"logprobs",
|
|
93
|
+
"max_tokens",
|
|
94
|
+
"presence_penalty",
|
|
95
|
+
"reasoning",
|
|
96
|
+
"response_format",
|
|
97
|
+
"seed",
|
|
98
|
+
"stop",
|
|
99
|
+
"structured_outputs",
|
|
100
|
+
"temperature",
|
|
101
|
+
"tool_choice",
|
|
102
|
+
"tools",
|
|
103
|
+
"top_k",
|
|
104
|
+
"top_logprobs",
|
|
105
|
+
"top_p"
|
|
106
|
+
],
|
|
107
|
+
"architecture": {
|
|
108
|
+
"modality": "text+image->text"
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
],
|
|
112
|
+
"endpoints": {
|
|
113
|
+
"reflection/beam-1.0-501b-a23b": [
|
|
114
|
+
{
|
|
115
|
+
"provider": "Nebius",
|
|
116
|
+
"tag": "nebius/fp8",
|
|
117
|
+
"name": "Nebius | reflection/beam-1.0-501b-a23b-20261016",
|
|
118
|
+
"context_length": 1048576,
|
|
119
|
+
"max_completion_tokens": 131072,
|
|
120
|
+
"pricing": {
|
|
121
|
+
"prompt": "0.00000025",
|
|
122
|
+
"completion": "0.000001",
|
|
123
|
+
"discount": 0
|
|
124
|
+
},
|
|
125
|
+
"tools": true,
|
|
126
|
+
"status": 0
|
|
127
|
+
},
|
|
128
|
+
{
|
|
129
|
+
"provider": "CoreWeave",
|
|
130
|
+
"tag": "coreweave/fp4",
|
|
131
|
+
"name": "CoreWeave | reflection/beam-1.0-501b-a23b-20261016",
|
|
132
|
+
"context_length": 1048576,
|
|
133
|
+
"max_completion_tokens": 943718,
|
|
134
|
+
"pricing": {
|
|
135
|
+
"prompt": "0.0000003",
|
|
136
|
+
"completion": "0.000001",
|
|
137
|
+
"input_cache_read": "0.00000004",
|
|
138
|
+
"discount": 0
|
|
139
|
+
},
|
|
140
|
+
"tools": true,
|
|
141
|
+
"status": 0
|
|
142
|
+
},
|
|
143
|
+
{
|
|
144
|
+
"provider": "BaseTen",
|
|
145
|
+
"tag": "baseten/fp4",
|
|
146
|
+
"name": "BaseTen | reflection/beam-1.0-501b-a23b-20261016",
|
|
147
|
+
"context_length": 1048576,
|
|
148
|
+
"max_completion_tokens": 131072,
|
|
149
|
+
"pricing": {
|
|
150
|
+
"prompt": "0.0000004",
|
|
151
|
+
"completion": "0.0000012",
|
|
152
|
+
"input_cache_read": "0.00000004",
|
|
153
|
+
"discount": 0
|
|
154
|
+
},
|
|
155
|
+
"tools": false,
|
|
156
|
+
"status": 0
|
|
157
|
+
},
|
|
158
|
+
{
|
|
159
|
+
"provider": "Together",
|
|
160
|
+
"tag": "together",
|
|
161
|
+
"name": "Together | reflection/beam-1.0-501b-a23b-20261016",
|
|
162
|
+
"context_length": 1048576,
|
|
163
|
+
"max_completion_tokens": 131072,
|
|
164
|
+
"pricing": {
|
|
165
|
+
"prompt": "0.0000004",
|
|
166
|
+
"completion": "0.0000012",
|
|
167
|
+
"input_cache_read": "0.00000004",
|
|
168
|
+
"discount": 0
|
|
169
|
+
},
|
|
170
|
+
"tools": true,
|
|
171
|
+
"status": 0
|
|
172
|
+
}
|
|
173
|
+
],
|
|
174
|
+
"inception/mercury-3": [
|
|
175
|
+
{
|
|
176
|
+
"provider": "Inception",
|
|
177
|
+
"tag": "inception",
|
|
178
|
+
"name": "Inception | inception/mercury-3-20261019",
|
|
179
|
+
"context_length": 1000000,
|
|
180
|
+
"max_completion_tokens": 131072,
|
|
181
|
+
"pricing": {
|
|
182
|
+
"prompt": "0.0000001",
|
|
183
|
+
"completion": "0.00000035",
|
|
184
|
+
"input_cache_read": "0.00000001",
|
|
185
|
+
"discount": 0.5
|
|
186
|
+
},
|
|
187
|
+
"tools": true,
|
|
188
|
+
"status": 0
|
|
189
|
+
}
|
|
190
|
+
],
|
|
191
|
+
"qwen/qwen3.7-plus": [
|
|
192
|
+
{
|
|
193
|
+
"provider": "Alibaba",
|
|
194
|
+
"tag": "alibaba/us-east-1",
|
|
195
|
+
"name": "Alibaba | qwen/qwen3.7-plus-20260602",
|
|
196
|
+
"context_length": 1000000,
|
|
197
|
+
"max_completion_tokens": 131072,
|
|
198
|
+
"pricing": {
|
|
199
|
+
"prompt": "0.00000032",
|
|
200
|
+
"completion": "0.00000128",
|
|
201
|
+
"input_cache_read": "0.000000064",
|
|
202
|
+
"input_cache_write": "0.0000004",
|
|
203
|
+
"discount": 0,
|
|
204
|
+
"overrides": [
|
|
205
|
+
{
|
|
206
|
+
"min_prompt_tokens": 256000,
|
|
207
|
+
"prompt": "0.00000096",
|
|
208
|
+
"completion": "0.00000384",
|
|
209
|
+
"input_cache_read": "0.000000192",
|
|
210
|
+
"input_cache_write": "0.0000012"
|
|
211
|
+
}
|
|
212
|
+
]
|
|
213
|
+
},
|
|
214
|
+
"tools": true,
|
|
215
|
+
"status": 0
|
|
216
|
+
},
|
|
217
|
+
{
|
|
218
|
+
"provider": "Alibaba",
|
|
219
|
+
"tag": "alibaba",
|
|
220
|
+
"name": "Alibaba | qwen/qwen3.7-plus-20260602",
|
|
221
|
+
"context_length": 1000000,
|
|
222
|
+
"max_completion_tokens": 131072,
|
|
223
|
+
"pricing": {
|
|
224
|
+
"prompt": "0.00000032",
|
|
225
|
+
"completion": "0.00000128",
|
|
226
|
+
"input_cache_read": "0.000000064",
|
|
227
|
+
"input_cache_write": "0.0000004",
|
|
228
|
+
"discount": 0,
|
|
229
|
+
"overrides": [
|
|
230
|
+
{
|
|
231
|
+
"min_prompt_tokens": 256000,
|
|
232
|
+
"prompt": "0.00000096",
|
|
233
|
+
"completion": "0.00000384",
|
|
234
|
+
"input_cache_read": "0.000000192",
|
|
235
|
+
"input_cache_write": "0.0000012"
|
|
236
|
+
}
|
|
237
|
+
]
|
|
238
|
+
},
|
|
239
|
+
"tools": true,
|
|
240
|
+
"status": 0
|
|
241
|
+
}
|
|
242
|
+
]
|
|
243
|
+
}
|
|
244
|
+
}
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Reprint the OpenRouter price snapshot, or refuse to invent a score.
|
|
3
|
+
|
|
4
|
+
The October 2026 writer bake-off did not call a model. ``--score`` exits 2
|
|
5
|
+
until every raw arm exists. ``--run`` exits 2 until ``OPENROUTER_API_KEY``
|
|
6
|
+
is set, and it still does not call OpenRouter.
|
|
7
|
+
"""
|
|
8
|
+
|
|
9
|
+
from __future__ import annotations
|
|
10
|
+
|
|
11
|
+
import argparse
|
|
12
|
+
import json
|
|
13
|
+
import os
|
|
14
|
+
import sys
|
|
15
|
+
from decimal import Decimal
|
|
16
|
+
from pathlib import Path
|
|
17
|
+
|
|
18
|
+
ROOT = Path(__file__).resolve().parent
|
|
19
|
+
PRICES = ROOT / "prices.json"
|
|
20
|
+
TASKS = ROOT / "tasks.json"
|
|
21
|
+
RAW = ROOT / "raw"
|
|
22
|
+
|
|
23
|
+
MODELS = ("beam", "mercury", "qwen")
|
|
24
|
+
IDS = {
|
|
25
|
+
"beam": "reflection/beam-1.0-501b-a23b",
|
|
26
|
+
"mercury": "inception/mercury-3",
|
|
27
|
+
"qwen": "qwen/qwen3.7-plus",
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
def per_million(per_token: str) -> str:
|
|
32
|
+
"""OpenRouter pricing fields are USD per token. Keep decimal exact."""
|
|
33
|
+
value = Decimal(per_token) * Decimal(1_000_000)
|
|
34
|
+
text = format(value, "f")
|
|
35
|
+
if "." in text:
|
|
36
|
+
text = text.rstrip("0").rstrip(".")
|
|
37
|
+
return text
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def load_json(path: Path) -> dict:
|
|
41
|
+
payload = json.loads(path.read_text())
|
|
42
|
+
if not isinstance(payload, dict):
|
|
43
|
+
raise SystemExit(f"{path.name} must be a JSON object")
|
|
44
|
+
return payload
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def price_rows(snapshot: dict) -> list[dict]:
|
|
48
|
+
by_id = {row["id"]: row for row in snapshot["models"]}
|
|
49
|
+
missing = [model_id for model_id in IDS.values() if model_id not in by_id]
|
|
50
|
+
if missing:
|
|
51
|
+
raise SystemExit(f"prices.json is missing {', '.join(missing)}")
|
|
52
|
+
rows = []
|
|
53
|
+
for label, model_id in IDS.items():
|
|
54
|
+
pricing = by_id[model_id]["pricing"]
|
|
55
|
+
row = {
|
|
56
|
+
"label": label,
|
|
57
|
+
"id": model_id,
|
|
58
|
+
"input_per_million": per_million(pricing["prompt"]),
|
|
59
|
+
"output_per_million": per_million(pricing["completion"]),
|
|
60
|
+
"cache_read_per_million": per_million(pricing["input_cache_read"])
|
|
61
|
+
if pricing.get("input_cache_read")
|
|
62
|
+
else None,
|
|
63
|
+
"cache_write_per_million": per_million(pricing["input_cache_write"])
|
|
64
|
+
if pricing.get("input_cache_write")
|
|
65
|
+
else None,
|
|
66
|
+
}
|
|
67
|
+
rows.append(row)
|
|
68
|
+
return rows
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
def print_prices() -> int:
|
|
72
|
+
snapshot = load_json(PRICES)
|
|
73
|
+
tasks = load_json(TASKS)
|
|
74
|
+
if tasks.get("dispatch") != "not_sent":
|
|
75
|
+
raise SystemExit("tasks.json dispatch must stay not_sent until a real run lands raw arms")
|
|
76
|
+
for task in tasks["tasks"]:
|
|
77
|
+
for field in ("landed", "parent"):
|
|
78
|
+
value = task[field]
|
|
79
|
+
if len(value) != 40 or any(char not in "0123456789abcdef" for char in value):
|
|
80
|
+
raise SystemExit(f"{task['id']} {field} is not a 40-character sha")
|
|
81
|
+
print(f"fetched_at {snapshot['fetched_at']}")
|
|
82
|
+
print(f"source {snapshot['source']}")
|
|
83
|
+
for row in price_rows(snapshot):
|
|
84
|
+
cache = row["cache_read_per_million"]
|
|
85
|
+
write = row["cache_write_per_million"]
|
|
86
|
+
print(
|
|
87
|
+
f"{row['label']}\t{row['id']}\t"
|
|
88
|
+
f"input {row['input_per_million']}\t"
|
|
89
|
+
f"output {row['output_per_million']}\t"
|
|
90
|
+
f"cache_read {cache if cache is not None else 'none'}\t"
|
|
91
|
+
f"cache_write {write if write is not None else 'none'}"
|
|
92
|
+
)
|
|
93
|
+
qwen = next(row for row in snapshot["models"] if row["id"] == IDS["qwen"])
|
|
94
|
+
overrides = (qwen.get("pricing") or {}).get("overrides") or []
|
|
95
|
+
for override in overrides:
|
|
96
|
+
print(
|
|
97
|
+
"qwen_override\t"
|
|
98
|
+
f"min_prompt_tokens {override['min_prompt_tokens']}\t"
|
|
99
|
+
f"input {per_million(override['prompt'])}\t"
|
|
100
|
+
f"output {per_million(override['completion'])}\t"
|
|
101
|
+
f"cache_read {per_million(override['input_cache_read'])}\t"
|
|
102
|
+
f"cache_write {per_million(override['input_cache_write'])}"
|
|
103
|
+
)
|
|
104
|
+
for model_id, endpoints in snapshot["endpoints"].items():
|
|
105
|
+
for endpoint in endpoints:
|
|
106
|
+
pricing = endpoint["pricing"]
|
|
107
|
+
print(
|
|
108
|
+
f"endpoint\t{model_id}\t{endpoint['provider']}\t{endpoint['tag']}\t"
|
|
109
|
+
f"tools {str(endpoint['tools']).lower()}\t"
|
|
110
|
+
f"input {per_million(pricing['prompt'])}\t"
|
|
111
|
+
f"output {per_million(pricing['completion'])}"
|
|
112
|
+
)
|
|
113
|
+
return 0
|
|
114
|
+
|
|
115
|
+
|
|
116
|
+
def score_refused() -> int:
|
|
117
|
+
present = sorted(path.name for path in RAW.glob("*")) if RAW.exists() else []
|
|
118
|
+
sys.stderr.write(
|
|
119
|
+
"writer bake-off scoring refused.\n"
|
|
120
|
+
"No model was called. Missing raw arms are not zeros and not a tie.\n"
|
|
121
|
+
"Expected raw/<task>-<model>.txt for each task in tasks.json and each of "
|
|
122
|
+
+ ", ".join(MODELS)
|
|
123
|
+
+ ".\n"
|
|
124
|
+
)
|
|
125
|
+
if present:
|
|
126
|
+
sys.stderr.write(f"Found: {', '.join(present)}\n")
|
|
127
|
+
else:
|
|
128
|
+
sys.stderr.write("raw/ is empty.\n")
|
|
129
|
+
return 2
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
def run_refused() -> int:
|
|
133
|
+
sys.stderr.write(
|
|
134
|
+
"writer bake-off regeneration refused.\n"
|
|
135
|
+
"The intended call is OpenRouter chat completions for "
|
|
136
|
+
"reflection/beam-1.0-501b-a23b, inception/mercury-3, and qwen/qwen3.7-plus.\n"
|
|
137
|
+
"This script does not send that call.\n"
|
|
138
|
+
)
|
|
139
|
+
if not os.environ.get("OPENROUTER_API_KEY"):
|
|
140
|
+
sys.stderr.write("Missing:\n- OPENROUTER_API_KEY is unset\n")
|
|
141
|
+
sys.stderr.write(
|
|
142
|
+
"Cursor subagents are not a substitute: the catalog for this run had no slug for those three ids.\n"
|
|
143
|
+
)
|
|
144
|
+
return 2
|
|
145
|
+
sys.stderr.write("OPENROUTER_API_KEY is set. This script still does not dispatch.\n")
|
|
146
|
+
return 2
|
|
147
|
+
|
|
148
|
+
|
|
149
|
+
def main(argv: list[str]) -> int:
|
|
150
|
+
parser = argparse.ArgumentParser()
|
|
151
|
+
parser.add_argument("--run", action="store_true", help="refuse a live regeneration without a key")
|
|
152
|
+
parser.add_argument("--score", action="store_true", help="score committed raw arms, or refuse")
|
|
153
|
+
parser.add_argument("--prices", action="store_true", help="print the committed OpenRouter snapshot")
|
|
154
|
+
args = parser.parse_args(argv)
|
|
155
|
+
if args.prices:
|
|
156
|
+
return print_prices()
|
|
157
|
+
if args.run:
|
|
158
|
+
return run_refused()
|
|
159
|
+
return score_refused()
|
|
160
|
+
|
|
161
|
+
|
|
162
|
+
if __name__ == "__main__":
|
|
163
|
+
raise SystemExit(main(sys.argv[1:]))
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
{
|
|
2
|
+
"dispatch": "not_sent",
|
|
3
|
+
"reason": "OPENROUTER_API_KEY was unset, and the Cursor subagent catalog had no slug for reflection/beam-1.0-501b-a23b, inception/mercury-3, or qwen/qwen3.7-plus.",
|
|
4
|
+
"instructions": "A live run checks out parent, implements only that task, and runs the named tests. The landed commit is the oracle. Routes and rosters stay untouched. These statements were not sent to a model.",
|
|
5
|
+
"tasks": [
|
|
6
|
+
{
|
|
7
|
+
"id": "rate-limit-abort",
|
|
8
|
+
"title": "Rate-limit abort",
|
|
9
|
+
"pull_request": 414,
|
|
10
|
+
"landed": "e63907a31321582a65ecb23968235beb31f21037",
|
|
11
|
+
"parent": "477fc060a4b925d91773a1188afcaa4346816d50",
|
|
12
|
+
"sources": ["plugins/kxm/src/route-fallback.ts"],
|
|
13
|
+
"tests": ["test/core/route-fallback.test.ts"],
|
|
14
|
+
"statement": "A 429 body that says the request was aborted was classified as cancellation, so an opted-in role never walked the fallback chain. Walkable status codes are classified first. tool_policy stays its own non-walkable class."
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"id": "suggest-whole-word",
|
|
18
|
+
"title": "Suggest whole words",
|
|
19
|
+
"pull_request": 411,
|
|
20
|
+
"landed": "477fc060a4b925d91773a1188afcaa4346816d50",
|
|
21
|
+
"parent": "9c51344cd1c3ddf56b70e9c4d67cafb6fc548502",
|
|
22
|
+
"sources": ["plugins/kxm/src/suggest.ts"],
|
|
23
|
+
"tests": ["test/core/suggest.test.ts"],
|
|
24
|
+
"statement": "Substring keyword matches sent prompts containing guide, author, or change to the wrong workflow. Hyphenated keywords still match as one token."
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"id": "hub-down-hint",
|
|
28
|
+
"title": "Hub-down hint",
|
|
29
|
+
"pull_request": 413,
|
|
30
|
+
"landed": "9c51344cd1c3ddf56b70e9c4d67cafb6fc548502",
|
|
31
|
+
"parent": "e5c1b54f7ac148f369e0c0afbc114965c6b8da3c",
|
|
32
|
+
"sources": ["plugins/kxm/src/init.ts", "plugins/kxm/src/cli/hub.ts"],
|
|
33
|
+
"tests": ["test/core/cli.test.ts", "test/core/hub-bind-probe.test.ts"],
|
|
34
|
+
"statement": "kxm init outside a worktree now needs to say to run git init. kxm hub view needs a hint when nothing answers: start a loopback hub, or check a remote bind."
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
"id": "vision-admitted-route",
|
|
38
|
+
"title": "Vision admitted route",
|
|
39
|
+
"pull_request": 412,
|
|
40
|
+
"landed": "e5c1b54f7ac148f369e0c0afbc114965c6b8da3c",
|
|
41
|
+
"parent": "c11458e56cc4de3680b7b7ba4a9a99b817b06372",
|
|
42
|
+
"sources": ["plugins/kxm/src/vision-gate.ts"],
|
|
43
|
+
"tests": ["test/core/vision-gate.test.ts"],
|
|
44
|
+
"statement": "--route used to change only the admission check while Pi still ran the default provider. Other ids need to spawn as pi --model, native-vendor ids are refused first, and an unadmitted route names the admit command."
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
"id": "windows-cmd-shim",
|
|
48
|
+
"title": "Windows cmd shim",
|
|
49
|
+
"pull_request": 365,
|
|
50
|
+
"landed": "145a8db3403aa8070771b723dd01dae5cd18a7fc",
|
|
51
|
+
"parent": "2ea15a76312eee015dedfb8bb6d62b5d59f2df97",
|
|
52
|
+
"sources": ["plugins/kxm/src/harness.ts"],
|
|
53
|
+
"tests": ["test/core/harness.test.ts"],
|
|
54
|
+
"statement": "A missing Windows cmd.exe shim was counted as installed. The light-shard Windows tests were also asserting Unix file modes and forward-slash paths."
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"id": "init-antigravity-guide",
|
|
58
|
+
"title": "Init Antigravity guide",
|
|
59
|
+
"pull_request": 309,
|
|
60
|
+
"landed": "8fb4aa51d22a9b249f78917403be54857501a19d",
|
|
61
|
+
"parent": "0ba4474752cfd01a45148caa78bc70f0d6ebcc63",
|
|
62
|
+
"sources": ["plugins/kxm/src/init-guide-setup.ts"],
|
|
63
|
+
"tests": ["test/core/init-guide-setup.test.ts"],
|
|
64
|
+
"statement": "Guided setup mapped Google Gemini guide candidates onto a Pi Antigravity selector. The Runtime Pi one-shot uses --no-extensions, and pi auth check does not load extensions, so a drive of those roles fails at dispatch."
|
|
65
|
+
}
|
|
66
|
+
]
|
|
67
|
+
}
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# Writer bake-off, October 2026
|
|
2
|
+
|
|
3
|
+
Use this page when you are deciding whether writer number two should stay OpenRouter Qwen3.7 Plus. It records a comparison of Reflection Beam 1.0, Inception Mercury 3, and `qwen/qwen3.7-plus` on six KXM writer tasks. Routing was left unchanged. No model was called, so this page does not name a replacement.
|
|
4
|
+
|
|
5
|
+
## Recommendation
|
|
6
|
+
|
|
7
|
+
Keep `pi-qwen3-7-plus-openrouter` as writer number two. `.kxm/roles/writer.yaml` stays `grok-grok-4-7`, then that Qwen route at effort medium, then the Argon and Flash routes. `.kxm/routes.yaml`, `.kxm/prices.yaml`, and `.kxm/models/inventory.yaml` are unchanged.
|
|
8
|
+
|
|
9
|
+
The three candidates have no correctness score, no diff, and no measured latency. List price is a catalog fact, and it is the wrong kind of evidence for an admission. On 2026-10-21 the OpenRouter models endpoint lists Mercury 3 at $0.10 input and $0.35 output per 1M tokens, Beam 1.0 at $0.30 and $1.00, and Qwen3.7 Plus at $0.32 and $1.28.
|
|
10
|
+
|
|
11
|
+
Mercury 3 is absent from the inventory. That file has `inception/mercury-2` and `inception/mercury-2.5`. Beam is present, and its checked-in standard tier is still $1 input and $3 output. The live models row is $0.30 and $1.00. Beam's BaseTen endpoint does not advertise tools. The Qwen admission required a live tool check, and this run did not send one.
|
|
12
|
+
|
|
13
|
+
## How this run was scored
|
|
14
|
+
|
|
15
|
+
The intended transport was OpenRouter chat completions for `reflection/beam-1.0-501b-a23b`, `inception/mercury-3`, and `qwen/qwen3.7-plus`. `OPENROUTER_API_KEY` is unset. This machine has no `~/.pi` directory and no Pi OpenRouter login. An unauthenticated `POST` to `https://openrouter.ai/api/v1/chat/completions` returned 401.
|
|
16
|
+
|
|
17
|
+
The fallback used by the [October 2026 planner bake-off](planner-bakeoff-2026-10.md) is a Cursor subagent pinned to the candidate model. The catalog offered to this agent on 2026-10-21 has no slug for the three OpenRouter ids. It offers Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, Composer 2.5, Cursor Grok 4.6, Gemini 4 Argon, GPT-5.6 Sol, Grok 4.7, and Muse Wonder 1.0. A diff from one of those families would answer a different question, so no substitute arm was launched.
|
|
18
|
+
|
|
19
|
+
GPT-5.6 Sol (`gpt-5.6-sol-medium`) was available as a judge and was not called. There were no diffs to label. The rubric, for a later keyed run, is 0 to 5 on correctness (the task tests pass), diff quality, and instruction following. Latency and dollars are measurements beside that score. `score.py --score` exits 2 while `raw/` is empty. A missing arm is not stored as zero.
|
|
20
|
+
|
|
21
|
+
## Catalog
|
|
22
|
+
|
|
23
|
+
Prices were read from `https://openrouter.ai/api/v1/models` at 2026-10-21T08:34:25Z, and from each id's `/endpoints` URL. OpenRouter returns USD per token. The tables multiply by 1,000,000. The snapshot is `writer-bakeoff-2026-10/prices.json`. Reprint it with `python3 docs/evals/writer-bakeoff-2026-10/score.py --prices`.
|
|
24
|
+
|
|
25
|
+
| Model | OpenRouter id | Input / output per 1M | Cache read per 1M |
|
|
26
|
+
| --- | --- | --- | --- |
|
|
27
|
+
| Beam 1.0 | `reflection/beam-1.0-501b-a23b` | $0.30 / $1.00 | $0.04 |
|
|
28
|
+
| Mercury 3 | `inception/mercury-3` | $0.10 / $0.35 | $0.01 |
|
|
29
|
+
| Qwen3.7 Plus | `qwen/qwen3.7-plus` | $0.32 / $1.28 | $0.064 |
|
|
30
|
+
|
|
31
|
+
Qwen3.7 Plus also lists cache write at $0.40 per 1M. Above 256,000 prompt tokens that row triples ($0.96, $3.84, $0.192, $1.20). The `.kxm/prices.yaml` base tier matches the live base tier at $0.32, $1.28, $0.064, and $0.40. Context length is 1,048,576 for Beam and 1,000,000 for Mercury 3 and Qwen3.7 Plus. All three model rows list `tools`.
|
|
32
|
+
|
|
33
|
+
Endpoint names in the snapshot carry provider dates: Beam `20261016`, Mercury `20261019`, and Qwen `20260602`. Beam's providers do not share one price.
|
|
34
|
+
|
|
35
|
+
| Provider | Tag | Input / output per 1M | Tools |
|
|
36
|
+
| --- | --- | --- | --- |
|
|
37
|
+
| Nebius | `nebius/fp8` | $0.25 / $1.00 | yes |
|
|
38
|
+
| CoreWeave | `coreweave/fp4` | $0.30 / $1.00 | yes |
|
|
39
|
+
| BaseTen | `baseten/fp4` | $0.40 / $1.20 | no |
|
|
40
|
+
| Together | `together` | $0.40 / $1.20 | yes |
|
|
41
|
+
|
|
42
|
+
CoreWeave, BaseTen, and Together list cache read at $0.04 per 1M. Nebius omits it. Mercury 3 has one Inception endpoint, tag `inception`, with tools. That object includes `"discount": 0.5` next to the rates in the table. Those prompt and completion fields are the quoted price. Qwen3.7 Plus has two Alibaba endpoints, `alibaba/us-east-1` and `alibaba`, both at the base price and both with tools.
|
|
43
|
+
|
|
44
|
+
`qwen/qwen3.7-plus:batch` is a separate catalog row at $0.23 input and $0.92 output per 1M. It is not writer number two. The live models list had no `reflection/beam-1.0-501b-a23b:free` id.
|
|
45
|
+
|
|
46
|
+
## Totals
|
|
47
|
+
|
|
48
|
+
Every arm is "not run". An empty cell means the model was not executed.
|
|
49
|
+
|
|
50
|
+
| Task | Beam, Mercury, and Qwen |
|
|
51
|
+
| --- | --- |
|
|
52
|
+
| Rate-limit abort | not run |
|
|
53
|
+
| Suggest whole words | not run |
|
|
54
|
+
| Hub-down hint | not run |
|
|
55
|
+
| Vision admitted route | not run |
|
|
56
|
+
| Windows cmd shim | not run |
|
|
57
|
+
| Init Antigravity guide | not run |
|
|
58
|
+
|
|
59
|
+
| Model | Judge total (max 90) | Tasks won | Measured spend |
|
|
60
|
+
| --- | --- | --- | --- |
|
|
61
|
+
| Beam 1.0 | not run | not run | $0.00, no calls |
|
|
62
|
+
| Mercury 3 | not run | not run | $0.00, no calls |
|
|
63
|
+
| Qwen3.7 Plus | not run | not run | $0.00, no calls |
|
|
64
|
+
|
|
65
|
+
Six tasks times three rubric dimensions is 90 points. Nobody scored any of them.
|
|
66
|
+
|
|
67
|
+
## Cost per task
|
|
68
|
+
|
|
69
|
+
No response returned token usage, latency, or a provider bill. Measured spend is $0 because zero requests were sent. A forecast would need token counts this run does not have.
|
|
70
|
+
|
|
71
|
+
| Task | Measured cost | Latency | Tests |
|
|
72
|
+
| --- | --- | --- | --- |
|
|
73
|
+
| Rate-limit abort | not measured | not measured | not run |
|
|
74
|
+
| Suggest whole words | not measured | not measured | not run |
|
|
75
|
+
| Hub-down hint | not measured | not measured | not run |
|
|
76
|
+
| Vision admitted route | not measured | not measured | not run |
|
|
77
|
+
| Windows cmd shim | not measured | not measured | not run |
|
|
78
|
+
| Init Antigravity guide | not measured | not measured | not run |
|
|
79
|
+
|
|
80
|
+
## Tasks that were prepared
|
|
81
|
+
|
|
82
|
+
Six landed fixes were selected because each is a small source change with a test. A live run would check out the parent, implement that task, and run the named tests. The landed commit would be the oracle. No parent was checked out, and no prompt was sent. The statements are the shipped bug, in `writer-bakeoff-2026-10/tasks.json`.
|
|
83
|
+
|
|
84
|
+
### Rate-limit abort
|
|
85
|
+
|
|
86
|
+
Pull request 414, landed `e63907a`, parent `477fc06`. A 429 body that says the request was aborted was classified as cancellation, so an opted-in role never walked. `tool_policy` stays non-walkable. Source: `plugins/kxm/src/route-fallback.ts`. Test: `test/core/route-fallback.test.ts`.
|
|
87
|
+
|
|
88
|
+
### Suggest whole words
|
|
89
|
+
|
|
90
|
+
Pull request 411, landed `477fc06`, parent `9c51344`. Substring keywords sent prompts containing "guide", "author", or "change" to the wrong workflow. Hyphenated keywords stay one token. Source: `plugins/kxm/src/suggest.ts`. Test: `test/core/suggest.test.ts`.
|
|
91
|
+
|
|
92
|
+
### Hub-down hint
|
|
93
|
+
|
|
94
|
+
Pull request 413, landed `9c51344`, parent `e5c1b54`. `kxm init` outside a Git worktree needs to say to run `git init`. `kxm hub view` needs a hint when nothing answers a loopback or remote [hub](../glossary.md#hub). Sources: `plugins/kxm/src/init.ts`, `plugins/kxm/src/cli/hub.ts`. Tests: `test/core/cli.test.ts`, `test/core/hub-bind-probe.test.ts`.
|
|
95
|
+
|
|
96
|
+
### Vision admitted route
|
|
97
|
+
|
|
98
|
+
Pull request 412, landed `e5c1b54`, parent `c11458e`. `--route` changed the admission check while Pi still ran the default provider. Other ids need `pi --model`, native-vendor ids are refused first, and an unadmitted route names the admit command. Source: `plugins/kxm/src/vision-gate.ts`. Test: `test/core/vision-gate.test.ts`.
|
|
99
|
+
|
|
100
|
+
### Windows cmd shim
|
|
101
|
+
|
|
102
|
+
Pull request 365, landed `145a8db`, parent `2ea15a7`. A missing Windows `cmd.exe` shim was counted as installed, and the light-shard Windows tests were asserting Unix file modes and forward-slash paths. Source: `plugins/kxm/src/harness.ts`. Test: `test/core/harness.test.ts`.
|
|
103
|
+
|
|
104
|
+
### Init Antigravity guide
|
|
105
|
+
|
|
106
|
+
Pull request 309, landed `8fb4aa5`, parent `0ba4474`. Guided setup mapped Google Gemini candidates onto a Pi Antigravity selector. The Runtime one-shot uses `--no-extensions`, and `pi auth check` does not load extensions, so a drive fails at dispatch. Source: `plugins/kxm/src/init-guide-setup.ts`. Test: `test/core/init-guide-setup.test.ts`.
|
|
107
|
+
|
|
108
|
+
## Caveats
|
|
109
|
+
|
|
110
|
+
- The sample is six tasks, and zero of them ran. Keep the current writer number two until a keyed run scores them.
|
|
111
|
+
- List rates are the 2026-10-21 models row. They are not an invoice for these tasks.
|
|
112
|
+
- The inventory Beam tier and the live models row disagree. Read the catalog again before any admission.
|
|
113
|
+
- OpenRouter may send Beam to BaseTen, which does not list tools. This page did not probe which endpoint answers.
|
|
114
|
+
- Mercury's `discount` field is stored in the snapshot and is not applied as a second cut.
|
|
115
|
+
- Cursor subagents were the documented fallback and were not used, because the candidate slugs are absent.
|
|
116
|
+
- Regenerate prices with `score.py --prices`. `score.py --run` and `score.py --score` exit 2 and do not invent arms.
|
|
117
|
+
|
|
118
|
+
## Related
|
|
119
|
+
|
|
120
|
+
- [Harness routing](../reference/harness-routing.md)
|
|
121
|
+
- [Configuration reference](../reference/config-reference.md)
|
|
122
|
+
- [Planner bake-off, October 2026](planner-bakeoff-2026-10.md)
|
|
123
|
+
- Prices, the task list, and `score.py` live in `docs/evals/writer-bakeoff-2026-10/`.
|
package/package.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
|
|
3
3
|
"name": "kxm",
|
|
4
4
|
"displayName": "KXM",
|
|
5
|
-
"version": "0.7.
|
|
5
|
+
"version": "0.7.200",
|
|
6
6
|
"description": "Headless multi-agent orchestration, durable workflows, and a live operator dashboard for Pi and Claude Code",
|
|
7
7
|
"author": {
|
|
8
8
|
"name": "KontextMind",
|
|
@@ -25462,7 +25462,7 @@ function sessionTokenFixHint(policy) {
|
|
|
25462
25462
|
}
|
|
25463
25463
|
|
|
25464
25464
|
// plugins/kxm/src/mcp-server.ts
|
|
25465
|
-
var VERSION = "0.7.
|
|
25465
|
+
var VERSION = "0.7.200";
|
|
25466
25466
|
var CONFIGURE_PLUGIN = "/plugin configure kxm@kxm";
|
|
25467
25467
|
var inbox = /* @__PURE__ */ new Map();
|
|
25468
25468
|
var notifiedInbox = /* @__PURE__ */ new Set();
|
package/plugins/kxm/package.json
CHANGED
|
@@ -11,7 +11,7 @@ import { deliverInboxNotification } from "./inbox.ts";
|
|
|
11
11
|
import type { HubEvent, MessageRecord } from "./protocol.ts";
|
|
12
12
|
import { sessionTokenFixHint } from "./session-token-hint.ts";
|
|
13
13
|
|
|
14
|
-
const VERSION = "0.7.
|
|
14
|
+
const VERSION = "0.7.200";
|
|
15
15
|
const CONFIGURE_PLUGIN = "/plugin configure kxm@kxm";
|
|
16
16
|
const inbox = new Map<string, MessageRecord>();
|
|
17
17
|
const notifiedInbox = new Set<string>();
|