@navels/neal 0.2.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +206 -209
- package/SECURITY.md +16 -19
- package/dist/neal/adjudicator/{blocked-adjudicator.js → consultant.js} +21 -21
- package/dist/neal/adjudicator/execute.js +33 -33
- package/dist/neal/adjudicator/final-completion.js +2 -31
- package/dist/neal/adjudicator/planning.js +230 -38
- package/dist/neal/agents/prompts.js +6 -17
- package/dist/neal/agents/rounds.js +43 -73
- package/dist/neal/agents/schemas.js +47 -13
- package/dist/neal/blocked-guidance.js +16 -9
- package/dist/neal/cli.js +2 -2
- package/dist/neal/commands/check.js +2 -2
- package/dist/neal/commands/compat.js +100 -36
- package/dist/neal/commands/interactive-activity.js +2 -2
- package/dist/neal/commands/recovery-guidance.js +20 -8
- package/dist/neal/commands/runtime.js +4 -2
- package/dist/neal/config.js +16 -7
- package/dist/neal/context/inline-review-context.js +13 -28
- package/dist/neal/context/reviewer-context.js +19 -14
- package/dist/neal/eval/reviewer-eval-manifest.js +86 -0
- package/dist/neal/eval/reviewer-eval.js +155 -0
- package/dist/neal/interactive-controls.js +5 -0
- package/dist/neal/orchestrator/completion.js +49 -2
- package/dist/neal/orchestrator/failures.js +7 -0
- package/dist/neal/orchestrator/notifications.js +9 -9
- package/dist/neal/orchestrator/phases/planning.js +117 -80
- package/dist/neal/orchestrator/phases/recovery.js +151 -115
- package/dist/neal/orchestrator/phases/review.js +28 -1
- package/dist/neal/orchestrator/transitions.js +7 -1
- package/dist/neal/orchestrator.js +3 -0
- package/dist/neal/plan-queue.js +29 -1
- package/dist/neal/prompts/execute.js +7 -19
- package/dist/neal/prompts/planning.js +64 -55
- package/dist/neal/prompts/review-doctrine.js +13 -50
- package/dist/neal/prompts/specialized.js +7 -17
- package/dist/neal/prompts/specs.js +180 -32
- package/dist/neal/providers/agent-settings-isolation.js +35 -0
- package/dist/neal/providers/anthropic-claude.js +18 -1
- package/dist/neal/providers/detection.js +3 -7
- package/dist/neal/providers/git-config-isolation.js +83 -0
- package/dist/neal/providers/openai-codex.js +25 -3
- package/dist/neal/providers/{generic-agentic-tools.js → openai-compatible-tools.js} +11 -5
- package/dist/neal/providers/openai-compatible.js +1078 -430
- package/dist/neal/providers/openrouter-routing.js +68 -0
- package/dist/neal/providers/pricing.js +1 -1
- package/dist/neal/providers/rate-card.js +6 -32
- package/dist/neal/providers/registry.js +6 -17
- package/dist/neal/providers/types.js +12 -0
- package/dist/neal/resume-decision.js +7 -6
- package/dist/neal/resume-planner.js +9 -0
- package/dist/neal/retrospective.js +3 -3
- package/dist/neal/review-debt.js +30 -0
- package/dist/neal/review-findings/provider.js +17 -29
- package/dist/neal/review.js +17 -1
- package/dist/neal/state-invariants.js +11 -3
- package/dist/neal/state-views.js +27 -5
- package/dist/neal/state.js +59 -15
- package/dist/neal/status.js +28 -1
- package/dist/neal/support.js +3 -3
- package/docs/README.md +22 -18
- package/docs/{ADJUDICATOR_INVENTORY.md → adjudicator-inventory.md} +58 -57
- package/docs/architecture.md +79 -83
- package/docs/automation.md +11 -11
- package/docs/compat.md +91 -61
- package/docs/compatible-models.md +145 -121
- package/docs/demo.md +15 -5
- package/docs/maintenance.md +65 -34
- package/docs/plan-format.md +61 -33
- package/docs/prompt-evals.md +96 -0
- package/docs/{PROMPT_SPECS.md → prompt-specs.md} +52 -48
- package/docs/providers.md +279 -349
- package/docs/release.md +63 -81
- package/docs/review-convergence.md +246 -0
- package/docs/state-machine.md +61 -59
- package/docs/storage.md +40 -31
- package/docs/troubleshooting.md +32 -29
- package/examples/compat/add-edit-verify/PLAN.md +2 -0
- package/examples/compat/add-edit-verify/broken.diff +2 -2
- package/examples/compat/add-edit-verify/good.diff +2 -2
- package/examples/compat/add-edit-verify/src/add.js +1 -1
- package/examples/compat/manifest.json +25 -29
- package/examples/compat/sum-grep-edit/PLAN.md +33 -0
- package/examples/compat/sum-grep-edit/broken.diff +12 -0
- package/examples/compat/sum-grep-edit/good.diff +12 -0
- package/examples/compat/sum-grep-edit/package.json +5 -0
- package/examples/compat/sum-grep-edit/src/numbers.js +9 -0
- package/examples/compat/sum-grep-edit/test/numbers.test.js +15 -0
- package/examples/issue-triage-js/README.md +0 -15
- package/examples/reviewer-eval/clean-doc/base/src/clamp.ts +3 -0
- package/examples/reviewer-eval/clean-doc/change.diff +9 -0
- package/examples/reviewer-eval/clean-extract/base/src/greet.ts +3 -0
- package/examples/reviewer-eval/clean-extract/change.diff +13 -0
- package/examples/reviewer-eval/dropped-error/base/src/load.ts +4 -0
- package/examples/reviewer-eval/dropped-error/change.diff +15 -0
- package/examples/reviewer-eval/manifest.json +45 -0
- package/examples/reviewer-eval/off-by-one/base/src/paginate.ts +5 -0
- package/examples/reviewer-eval/off-by-one/change.diff +11 -0
- package/{config.yml → neal.yml} +1 -1
- package/package.json +15 -14
- package/dist/neal/providers/generic-agentic.js +0 -1256
- package/docs/comparison.md +0 -105
- package/docs/issue-pipeline.md +0 -124
- package/examples/compat/is-even-add-test/PLAN.md +0 -30
- package/examples/compat/is-even-add-test/broken.diff +0 -11
- package/examples/compat/is-even-add-test/good.diff +0 -11
- package/examples/compat/is-even-add-test/package.json +0 -5
- package/examples/compat/is-even-add-test/src/is-even.js +0 -3
- package/examples/compat/is-even-add-test/test/is-even.test.js +0 -9
- package/examples/compat/reverse-grep-edit/PLAN.md +0 -32
- package/examples/compat/reverse-grep-edit/broken.diff +0 -12
- package/examples/compat/reverse-grep-edit/good.diff +0 -12
- package/examples/compat/reverse-grep-edit/package.json +0 -5
- package/examples/compat/reverse-grep-edit/src/strings.js +0 -10
- package/examples/compat/reverse-grep-edit/test/strings.test.js +0 -15
|
@@ -1,114 +1,153 @@
|
|
|
1
1
|
# Compatible models
|
|
2
2
|
|
|
3
|
-
> **Last updated:** 2026-
|
|
3
|
+
> **Last updated:** 2026-07-20 · **Reference:** `openai-codex` (gpt-5.5) · **Pool:** 90 OpenRouter models
|
|
4
4
|
>
|
|
5
5
|
> The whitelist from running [`neal compat`](compat.md) across an OpenRouter candidate
|
|
6
|
-
> pool. It records **compatibility, not skill**: a PASS means the model can drive
|
|
7
|
-
> loop on the trivial bundled fixtures in that role
|
|
8
|
-
> model performs on real work. Choose among the PASSes by your own
|
|
6
|
+
> pool. It records **compatibility, not skill**: a PASS means the model can drive
|
|
7
|
+
> neal's loop on the trivial bundled fixtures in that role. It says nothing about how
|
|
8
|
+
> well the model performs on real work. Choose among the PASSes by your own
|
|
9
|
+
> cost/quality needs.
|
|
9
10
|
|
|
10
11
|
## How to read this
|
|
11
12
|
|
|
12
|
-
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
13
|
+
- Scored under the **discrimination criterion** ([#49](https://github.com/navels/neal/issues/49),
|
|
14
|
+
`schemaVersion: 2`): a reviewer PASSes a fixture iff it raises at least one blocking
|
|
15
|
+
finding on the broken diff **and** strictly fewer on the good diff. Severity
|
|
16
|
+
calibration is not graded. Earlier sweeps (2026-06-18) required zero blocking
|
|
17
|
+
findings on the good diff and are not comparable.
|
|
18
|
+
- The pool is the 2026-06-18 pool plus every qualifying model added to OpenRouter
|
|
19
|
+
since (paid, text-output chat models; no `:free` variants, no routers/aliases, no
|
|
20
|
+
native-adapter providers).
|
|
21
|
+
- Role cells = passed/total fixtures (coder 2, reviewer 2 pairs, planner 1. The
|
|
22
|
+
redundant `is-even-add-test` fixture was dropped 2026-07-20).
|
|
23
|
+
- **Single-run results are noisy at the margin.** Fail-only-once models were
|
|
24
|
+
re-confirmed once. Results below note where a verdict flipped on re-run.
|
|
25
|
+
- Native adapters (`openai-codex`, `anthropic-claude`) are supported by construction
|
|
26
|
+
and need no compat run.
|
|
27
|
+
|
|
28
|
+
## Compatible (44) - verified across all roles
|
|
24
29
|
|
|
25
30
|
Sorted cheapest first ($/Mtok in·out, from the live OpenRouter catalog).
|
|
26
31
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
|
32
|
+
| Model | Coder | Reviewer | Planner | $/Mtok | ctx | Note |
|
|
33
|
+
|---|---|---|---|---|---|---|
|
|
34
|
+
| `nex-agi/nex-n2-mini` | 2/2 | 4/4 | 1/1 | 0.02·0.10 | 262k | |
|
|
35
|
+
| `nvidia/nemotron-3-nano-30b-a3b` | 2/2 | 4/4 | 1/1 | 0.05·0.20 | 262k | |
|
|
36
|
+
| `google/gemma-4-26b-a4b-it` | 2/2 | 4/4 | 1/1 | 0.07·0.34 | 262k | |
|
|
37
|
+
| `inclusionai/ling-2.6-1t` | 2/2 | 4/4 | 1/1 | 0.07·0.62 | 262k | |
|
|
38
|
+
| `nvidia/nemotron-3-super-120b-a12b` | 2/2 | 4/4 | 1/1 | 0.08·0.40 | 1000k | |
|
|
39
|
+
| `deepseek/deepseek-v4-flash` | 2/2 | 4/4 | 1/1 | 0.10·0.20 | 1049k | |
|
|
40
|
+
| `qwen/qwen3.6-35b-a3b` | 2/2 | 4/4 | 1/1 | 0.14·1.00 | 262k | |
|
|
41
|
+
| `kwaipilot/kat-coder-air-v2.5` | 2/2 | 4/4 | 1/1 | 0.15·0.60 | 256k | |
|
|
42
|
+
| `minimax/minimax-m2.5` | 2/2 | 4/4 | 1/1 | 0.15·0.90 | 205k | |
|
|
43
|
+
| `mistralai/mistral-small-2603` | 2/2 | 4/4 | 1/1 | 0.15·0.60 | 262k | |
|
|
44
|
+
| `stepfun/step-3.7-flash` | 2/2 | 4/4 | 1/1 | 0.20·1.15 | 256k | |
|
|
45
|
+
| `tencent/hy3` | 2/2 | 4/4 | 1/1 | 0.20·0.80 | 262k | |
|
|
46
|
+
| `google/gemma-4-31b-it` | 2/2 | 4/4 | 1/1 | 0.22·0.55 | 262k | |
|
|
47
|
+
| `deepseek/deepseek-chat-v3.1` | 2/2 | 4/4 | 1/1 | 0.25·0.95 | 164k | |
|
|
48
|
+
| `google/gemini-3.1-flash-lite` | 2/2 | 4/4 | 1/1 | 0.25·1.50 | 1049k | |
|
|
49
|
+
| `google/gemini-3.1-flash-lite-preview` | 2/2 | 4/4 | 1/1 | 0.25·1.50 | 1049k | |
|
|
50
|
+
| `minimax/minimax-m2.7` | 2/2 | 4/4 | 1/1 | 0.25·1.00 | 205k | first attempt failed (protocol:3). Confirmed PASS on re-run |
|
|
51
|
+
| `z-ai/glm-5.2` | 2/2 | 4/4 | 1/1 | 0.26·0.81 | 1049k | |
|
|
52
|
+
| `deepseek/deepseek-v3.1-terminus` | 2/2 | 4/4 | 1/1 | 0.27·1.00 | 131k | |
|
|
53
|
+
| `deepseek/deepseek-v3.2` | 2/2 | 4/4 | 1/1 | 0.27·0.40 | 164k | |
|
|
54
|
+
| `deepseek/deepseek-v3.2-exp` | 2/2 | 4/4 | 1/1 | 0.27·0.41 | 164k | |
|
|
55
|
+
| `minimax/minimax-m2` | 2/2 | 4/4 | 1/1 | 0.30·1.20 | 205k | |
|
|
56
|
+
| `qwen/qwen3.5-plus-20260420` | 2/2 | 4/4 | 1/1 | 0.30·1.80 | 1000k | |
|
|
57
|
+
| `qwen/qwen3.7-plus` | 2/2 | 4/4 | 1/1 | 0.32·1.28 | 1000k | |
|
|
58
|
+
| `deepseek/deepseek-v4-pro` | 2/2 | 4/4 | 1/1 | 0.43·0.87 | 1049k | first attempt failed (structured_output:1). Confirmed PASS on re-run |
|
|
59
|
+
| `moonshotai/kimi-k2-0905` | 2/2 | 4/4 | 1/1 | 0.60·2.50 | 262k | |
|
|
60
|
+
| `moonshotai/kimi-k2-thinking` | 2/2 | 4/4 | 1/1 | 0.60·2.50 | 262k | |
|
|
61
|
+
| `nvidia/nemotron-3-ultra-550b-a55b` | 2/2 | 4/4 | 1/1 | 0.60·3.60 | 1000k | |
|
|
62
|
+
| `kwaipilot/kat-coder-pro-v2.5` | 2/2 | 4/4 | 1/1 | 0.74·2.96 | 256k | |
|
|
63
|
+
| `moonshotai/kimi-k2.7-code` | 2/2 | 4/4 | 1/1 | 0.85·3.80 | 262k | |
|
|
64
|
+
| `z-ai/glm-5` | 2/2 | 4/4 | 1/1 | 0.95·2.55 | 205k | |
|
|
65
|
+
| `z-ai/glm-5.1` | 2/2 | 4/4 | 1/1 | 0.97·3.04 | 203k | |
|
|
66
|
+
| `x-ai/grok-build-0.1` | 2/2 | 4/4 | 1/1 | 1.00·2.00 | 256k | |
|
|
67
|
+
| `qwen/qwen3.6-max-preview` | 2/2 | 4/4 | 1/1 | 1.04·6.24 | 262k | |
|
|
68
|
+
| `meta/muse-spark-1.1` | 2/2 | 4/4 | 1/1 | 1.25·4.25 | 1049k | |
|
|
69
|
+
| `x-ai/grok-4.20` | 2/2 | 4/4 | 1/1 | 1.25·2.50 | 2000k | |
|
|
70
|
+
| `x-ai/grok-4.3` | 2/2 | 4/4 | 1/1 | 1.25·2.50 | 1000k | first attempt failed (provider_failed:2). Confirmed PASS on re-run |
|
|
71
|
+
| `qwen/qwen3.7-max` | 2/2 | 4/4 | 1/1 | 1.48·4.42 | 1000k | |
|
|
72
|
+
| `google/gemini-3.5-flash` | 2/2 | 4/4 | 1/1 | 1.50·9.00 | 1049k | |
|
|
73
|
+
| `mistralai/mistral-medium-3-5` | 2/2 | 4/4 | 1/1 | 1.50·7.50 | 262k | |
|
|
74
|
+
| `google/gemini-3.1-pro-preview-customtools` | 2/2 | 4/4 | 1/1 | 2.00·12.00 | 1049k | |
|
|
75
|
+
| `x-ai/grok-4.5` | 2/2 | 4/4 | 1/1 | 2.00·6.00 | 500k | |
|
|
76
|
+
| `moonshotai/kimi-k3` | 2/2 | 4/4 | 1/1 | 3.00·15.00 | 1049k | first attempt failed (provider_failed:1). Confirmed PASS on re-run |
|
|
77
|
+
| `sakana/fugu-ultra` | 2/2 | 4/4 | 1/1 | 5.00·30.00 | 1000k | |
|
|
78
|
+
|
|
79
|
+
## Borderline - re-run candidates (10)
|
|
80
|
+
|
|
81
|
+
Failed one or a few cells with a non-protocol mode, or failed differently across
|
|
82
|
+
attempts. `provider_failed` rows are transient infrastructure. The rest are
|
|
83
|
+
single-run signals that may flip. Re-run before excluding.
|
|
84
|
+
|
|
85
|
+
| Model | Coder | Reviewer | Planner | Tripped on | Note |
|
|
39
86
|
|---|---|---|---|---|---|
|
|
40
|
-
| `
|
|
41
|
-
| `
|
|
42
|
-
| `
|
|
43
|
-
| `
|
|
44
|
-
| `
|
|
45
|
-
| `
|
|
46
|
-
| `
|
|
47
|
-
| `
|
|
48
|
-
| `
|
|
49
|
-
| `
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
>
|
|
95
|
-
> `
|
|
96
|
-
>
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
`amazon/nova-{micro,lite,2-lite,pro,premier}-v1`, `bytedance-seed/seed-2.0-mini`,
|
|
105
|
-
`cohere/{command-a,command-r-08-2024,command-r-plus-08-2024,command-r7b-12-2024}`,
|
|
106
|
-
`meta-llama/{llama-3.1-8b,llama-3.2-1b,llama-3.2-3b}-instruct`, `meta-llama/llama-4-scout`,
|
|
107
|
-
`microsoft/{phi-4,phi-4-mini-instruct,wizardlm-2-8x22b}`, `minimax/minimax-m2-her`,
|
|
108
|
-
`mistralai/{ministral-3b,ministral-8b,ministral-14b}-2512`, `mistralai/mistral-small-2603`,
|
|
109
|
-
`moonshotai/{kimi-k2,kimi-k2-0905,kimi-k2-thinking,kimi-k2.6,kimi-k2.7-code}`,
|
|
110
|
-
`nvidia/llama-3.3-nemotron-super-49b-v1.5`, `openai/{gpt-oss-120b,gpt-oss-20b,gpt-oss-safeguard-20b}`,
|
|
111
|
-
`stepfun/step-3.5-flash`, `x-ai/grok-4.20-multi-agent`, `z-ai/{glm-4.7-flash,glm-5-turbo}`.
|
|
87
|
+
| `bytedance-seed/seed-1.6` | 0/2 | 4/4 | 1/1 | coder/add-edit-verify: structured_output | |
|
|
88
|
+
| `bytedance-seed/seed-1.6-flash` | 0/2 | 0/4 | 1/1 | coder/add-edit-verify: provider_failed | |
|
|
89
|
+
| `bytedance-seed/seed-2.0-lite` | 0/2 | 4/4 | 1/1 | coder/add-edit-verify: structured_output | confirmed FAIL 2/2 (attempt 1 {'structured_output': 2}, attempt 2 {'structured_output': 1}) |
|
|
90
|
+
| `inclusionai/ling-2.6-flash` | 1/2 | 0/4 | 0/1 | coder/add-edit-verify: provider_failed | |
|
|
91
|
+
| `meta-llama/llama-3.1-8b-instruct` | 0/2 | 0/4 | 0/1 | coder/add-edit-verify: provider_failed | |
|
|
92
|
+
| `meta-llama/llama-4-maverick` | 1/2 | 4/4 | 1/1 | coder/add-edit-verify: block_unresolved | confirmed FAIL 2/2 (attempt 1 {'block_unresolved': 1}, attempt 2 {'structured_output': 1, 'block_unresolved': 1}) |
|
|
93
|
+
| `mistralai/devstral-2512` | 0/2 | 4/4 | 1/1 | coder/add-edit-verify: provider_failed | confirmed FAIL 2/2 (attempt 1 {'provider_failed': 2}, attempt 2 {'provider_failed': 1}) |
|
|
94
|
+
| `moonshotai/kimi-k2.5` | 1/2 | 4/4 | 1/1 | coder/sum-grep-edit: structured_output | |
|
|
95
|
+
| `moonshotai/kimi-k2.6` | 2/2 | 2/4 | 1/1 | reviewer/sum-grep-edit: structured_output | |
|
|
96
|
+
| `qwen/qwen3.6-flash` | 2/2 | 0/4 | 1/1 | reviewer/add-edit-verify: structured_output | |
|
|
97
|
+
|
|
98
|
+
## Incompatible (36) - fail the provider handshake
|
|
99
|
+
|
|
100
|
+
These can't drive neal's agentic loop on OpenRouter at all (`protocol` failure on
|
|
101
|
+
the provider check, every role): mostly small models and ones whose OR endpoints
|
|
102
|
+
don't support tool use.
|
|
103
|
+
|
|
104
|
+
- `aion-labs/aion-3.0`
|
|
105
|
+
- `aion-labs/aion-3.0-mini`
|
|
106
|
+
- `amazon/nova-2-lite-v1`
|
|
107
|
+
- `amazon/nova-lite-v1`
|
|
108
|
+
- `amazon/nova-micro-v1`
|
|
109
|
+
- `amazon/nova-premier-v1`
|
|
110
|
+
- `amazon/nova-pro-v1`
|
|
111
|
+
- `bytedance-seed/seed-2.0-mini`
|
|
112
|
+
- `cohere/command-a`
|
|
113
|
+
- `cohere/command-r-08-2024`
|
|
114
|
+
- `cohere/command-r-plus-08-2024`
|
|
115
|
+
- `cohere/command-r7b-12-2024`
|
|
116
|
+
- `inclusionai/ring-2.6-1t`
|
|
117
|
+
- `meta-llama/llama-3.2-1b-instruct`
|
|
118
|
+
- `meta-llama/llama-3.2-3b-instruct`
|
|
119
|
+
- `meta-llama/llama-3.3-70b-instruct`
|
|
120
|
+
- `meta-llama/llama-4-scout`
|
|
121
|
+
- `microsoft/phi-4`
|
|
122
|
+
- `microsoft/wizardlm-2-8x22b`
|
|
123
|
+
- `minimax/minimax-m2-her`
|
|
124
|
+
- `minimax/minimax-m2.1` *(confirmed 2/2)*
|
|
125
|
+
- `minimax/minimax-m3` *(confirmed 2/2)*
|
|
126
|
+
- `mistralai/ministral-14b-2512`
|
|
127
|
+
- `mistralai/ministral-3b-2512`
|
|
128
|
+
- `mistralai/ministral-8b-2512`
|
|
129
|
+
- `moonshotai/kimi-k2`
|
|
130
|
+
- `openai/gpt-oss-120b`
|
|
131
|
+
- `openai/gpt-oss-20b`
|
|
132
|
+
- `openai/gpt-oss-safeguard-20b`
|
|
133
|
+
- `poolside/laguna-xs-2.1`
|
|
134
|
+
- `stepfun/step-3.5-flash`
|
|
135
|
+
- `thinkingmachines/inkling`
|
|
136
|
+
- `x-ai/grok-4.20-multi-agent`
|
|
137
|
+
- `z-ai/glm-4.7`
|
|
138
|
+
- `z-ai/glm-4.7-flash`
|
|
139
|
+
- `z-ai/glm-5-turbo`
|
|
140
|
+
|
|
141
|
+
> **`require_parameters` note:** `minimax-m2.1` and `minimax-m3` (June PASSes, now
|
|
142
|
+
> `protocol`, confirmed 2/2) trace to the `require_parameters: true` OpenRouter
|
|
143
|
+
> routing constraint (#46), which excludes backends that do not support
|
|
144
|
+
> `json_schema`. This is a routing/infrastructure attribution, not a model
|
|
145
|
+
> capability change. Re-run if OpenRouter adds a capable backend for these slugs.
|
|
146
|
+
> `minimax-m2.7` is intermittent under the same constraint (failed the sweep,
|
|
147
|
+
> passed the confirm). The same constraint also *fixed* several June `protocol`
|
|
148
|
+
> failures by steering routing to capable backends: `moonshotai/kimi-k2-0905`,
|
|
149
|
+
> `kimi-k2-thinking`, `kimi-k2.7-code`, and `mistralai/mistral-small-2603` all
|
|
150
|
+
> flipped to full PASS.
|
|
112
151
|
|
|
113
152
|
## Native adapters (supported, no compat run required)
|
|
114
153
|
|
|
@@ -123,24 +162,9 @@ support tool use.
|
|
|
123
162
|
neal compat --model <slug> --role all --reference openai-codex --json
|
|
124
163
|
```
|
|
125
164
|
|
|
126
|
-
`
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
**25** PASSes (13 dropped, confirmed in isolation — not a rate-limit artifact):
|
|
133
|
-
too weak, it emits coder diffs the candidate-reviewer flags and blocks good
|
|
134
|
-
candidate work as partner-reviewer.
|
|
135
|
-
- `google/gemini-3.1-pro-preview-customtools` (strong/expensive) was worse in
|
|
136
|
-
practice: of the 25 codex-PASS models only **1** reproduced, and it **wedged
|
|
137
|
-
under load** (hung reference calls at 6-wide, timeouts/no-result on the
|
|
138
|
-
PASS-class models at 3-wide). A reference that can't run wide defeats the
|
|
139
|
-
throughput purpose of going off-codex in the first place.
|
|
140
|
-
|
|
141
|
-
**Conclusion:** `openai-codex` (gpt-5.5) is the authoritative reference. To extend
|
|
142
|
-
the whitelist beyond this pool, run additional candidates on codex-ref directly
|
|
143
|
-
(throttle to about three concurrent `neal compat` invocations for codex
|
|
144
|
-
capacity); the OpenRouter-reference path is not a viable shortcut today.
|
|
145
|
-
|
|
146
|
-
Paid slugs only — never `:free`. Re-run when a slug's backing model version changes.
|
|
165
|
+
`openai-codex` (gpt-5.5) is the authoritative reference. No OpenRouter model has
|
|
166
|
+
validated as a drop-in reference (two tried 2026-06-18, both failed. See the
|
|
167
|
+
2026-06-18 revision of this file). Throttle to about three concurrent invocations
|
|
168
|
+
for codex capacity.
|
|
169
|
+
|
|
170
|
+
Paid slugs only, never `:free`. Re-run when a slug's backing model version changes.
|
package/docs/demo.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Demo
|
|
1
|
+
# Demo recording
|
|
2
2
|
|
|
3
3
|
Use `asciinema` for public terminal demos when a recording is useful. It is an
|
|
4
4
|
optional recording tool, not a package dependency.
|
|
@@ -6,7 +6,17 @@ optional recording tool, not a package dependency.
|
|
|
6
6
|
Record from a disposable repository or throwaway worktree, preferably under
|
|
7
7
|
`/tmp`, so the demo does not expose real project history, paths, or artifacts.
|
|
8
8
|
|
|
9
|
-
##
|
|
9
|
+
## What to show
|
|
10
|
+
|
|
11
|
+
The README doesn't ship an embedded recording. If you make one, show the part
|
|
12
|
+
that's actually worth watching: the planner/coder/reviewer loop with the coder
|
|
13
|
+
and reviewer on different vendors, and the findings and responses going back and
|
|
14
|
+
forth until the reviewer is satisfied. A single trivial one-scope plan running
|
|
15
|
+
to completion doesn't show any of that. Pick a small change with a real review
|
|
16
|
+
point in it, so the reviewer has something to catch and the coder has something
|
|
17
|
+
to respond to.
|
|
18
|
+
|
|
19
|
+
## Suggested script
|
|
10
20
|
|
|
11
21
|
Create a small demo plan in the disposable repository:
|
|
12
22
|
|
|
@@ -24,7 +34,7 @@ $EDITOR tmp/DEMO_PLAN.md
|
|
|
24
34
|
```
|
|
25
35
|
|
|
26
36
|
Configure providers and run `neal check` before recording, or record those
|
|
27
|
-
steps only if the output is safe to share. Use current
|
|
37
|
+
steps only if the output is safe to share. Use current neal commands while
|
|
28
38
|
recording:
|
|
29
39
|
|
|
30
40
|
```bash
|
|
@@ -35,7 +45,7 @@ exit
|
|
|
35
45
|
```
|
|
36
46
|
|
|
37
47
|
Keep the recording output outside the demo repository so it does not appear as
|
|
38
|
-
unrelated dirty work while
|
|
48
|
+
unrelated dirty work while neal starts the writer run.
|
|
39
49
|
|
|
40
50
|
If you specifically want to show recovery from a run that is waiting for
|
|
41
51
|
operator guidance, record that as a separate segment with the selected run id:
|
|
@@ -44,7 +54,7 @@ operator guidance, record that as a separate segment with the selected run id:
|
|
|
44
54
|
neal resume --run <run-id> --message "Continue with the smallest safe follow-up."
|
|
45
55
|
```
|
|
46
56
|
|
|
47
|
-
## Scrubbing
|
|
57
|
+
## Scrubbing checklist
|
|
48
58
|
|
|
49
59
|
Before sharing a recording, review the terminal output for local paths, private
|
|
50
60
|
project names, provider output, credentials, tokens, provider environment
|
package/docs/maintenance.md
CHANGED
|
@@ -1,60 +1,91 @@
|
|
|
1
|
-
# Maintenance: dependencies
|
|
1
|
+
# Maintenance: dependencies and versioning
|
|
2
2
|
|
|
3
|
-
neal's behavior
|
|
4
|
-
|
|
5
|
-
is the policy.
|
|
3
|
+
neal's behavior comes largely from the agent SDKs it drives, so dependency
|
|
4
|
+
updates need real attention, not routine hygiene. Here's the policy.
|
|
6
5
|
|
|
7
|
-
##
|
|
6
|
+
## Dependency buckets
|
|
8
7
|
|
|
9
|
-
|
|
8
|
+
Grouped so at most a handful of dependency PRs are ever open at once:
|
|
9
|
+
|
|
10
|
+
| Bucket | Contents | Cadence | Update posture |
|
|
10
11
|
| --- | --- | --- | --- |
|
|
11
|
-
| **
|
|
12
|
-
| **
|
|
13
|
-
| **
|
|
12
|
+
| **Weekly non-major** | everything except the native SDKs (minor/patch/pin/digest) | weekly, one grouped PR | auto-merge after CI and the live smoke pass, subject to a 3-day soak |
|
|
13
|
+
| **Native SDKs** | `@openai/codex-sdk` and `@anthropic-ai/claude-agent-sdk`, both exact-pinned. `ai`, `@ai-sdk/openai-compatible`, and `zod` also stay exact-pinned. | one grouped PR, opened after the 3-day soak instead of waiting for the weekly schedule | qualify on a subscription-authenticated machine with `scripts/qualify-sdk.sh`. Never auto-merge. Skip the CI smoke because it does not exercise the native adapters. |
|
|
14
|
+
| **Library majors** | every npm major except the native SDKs (`typescript`, `@types/node`, `ai`, `@ai-sdk/openai-compatible`, `zod`, etc.) | monthly, one grouped PR | review manually because some need code changes |
|
|
15
|
+
| **GitHub Actions majors** | `actions/checkout`, `actions/setup-node`, `actions/upload-artifact`, etc. | monthly, one grouped PR separate from library majors | review and merge on green CI. Do not auto-merge because a major action bump can still change behavior. |
|
|
14
16
|
|
|
15
|
-
The
|
|
16
|
-
structured output, or sandbox behavior
|
|
17
|
-
breaking compilation.
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
17
|
+
The native-SDK split exists because an agentic-SDK bump can change
|
|
18
|
+
tool-calling, structured output, or sandbox behavior. That breaks neal's loop
|
|
19
|
+
**without** breaking compilation. And CI can't behaviorally exercise the
|
|
20
|
+
native adapters: their auth is subscription-based and lives only on a
|
|
21
|
+
maintainer's machine. Everything else is behaviorally exercised in CI: the
|
|
22
|
+
live smoke runs on every package.json/lockfile PR, so the weekly grouped PR
|
|
23
|
+
is gated on it as a whole.
|
|
21
24
|
|
|
22
25
|
## The update flow
|
|
23
26
|
|
|
24
|
-
1. **Detect.** [Renovate](../renovate.json) opens PRs
|
|
25
|
-
|
|
26
|
-
`
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
**
|
|
27
|
+
1. **Detect.** [Renovate](../renovate.json) opens grouped PRs: the weekly
|
|
28
|
+
non-major bucket, the native-SDK bucket (labelled `agentic-sdk` +
|
|
29
|
+
`needs-qualification`), and, monthly, separate library-majors and
|
|
30
|
+
GitHub-Actions-majors buckets (both labelled `needs-review`).
|
|
31
|
+
2. **Verify (automatic).** CI (`.github/workflows/ci.yml`) runs typecheck + lint
|
|
32
|
+
+ unit tests + package verification on Node 24.18.0, which catches
|
|
33
|
+
**API-shape / contract** breaks. The live smoke
|
|
34
|
+
(`.github/workflows/smoke.yml`) runs on every package.json / lockfile PR: a
|
|
35
|
+
real `neal compat` run against a cheap OpenRouter model through
|
|
36
|
+
`openai-compatible`, catching **behavioral** breaks in the AI-SDK tier. The
|
|
37
|
+
weekly non-major PR auto-merges when both are green.
|
|
38
|
+
**The smoke requires the `OPENROUTER_API_KEY` repo secret.** Without it
|
|
34
39
|
the smoke skips (green) and the AI-SDK auto-merge gate is compile-only.
|
|
35
40
|
3. **Verify (behavioral, native tier).** `@openai/codex-sdk` and
|
|
36
|
-
`@anthropic-ai/claude-agent-sdk`
|
|
41
|
+
`@anthropic-ai/claude-agent-sdk` can't be smoked in CI, so qualify them
|
|
37
42
|
from any checkout with authenticated Claude/Codex CLIs:
|
|
38
43
|
```
|
|
39
44
|
scripts/qualify-sdk.sh <pr-number>
|
|
40
45
|
```
|
|
41
46
|
It runs the full suite plus a live `neal compat --role all` pass-through on
|
|
42
|
-
|
|
43
|
-
explicitly so nothing leaks from `~/.neal/config.yml`), posts
|
|
44
|
-
|
|
47
|
+
every bumped adapter in the PR (in a throwaway worktree, with roles and
|
|
48
|
+
models pinned explicitly so nothing leaks from `~/.neal/config.yml`), posts
|
|
49
|
+
the compat matrices to the PR, and approves on PASS (`--merge` also
|
|
50
|
+
squash-merges).
|
|
45
51
|
4. **Adopt.** Merge, bump neal's version, add a CHANGELOG entry noting the bump +
|
|
46
52
|
any behavior change, and cut a release via the existing workflow. Urgent
|
|
47
53
|
bumps (a fix neal needs immediately) may skip the Renovate soak with a
|
|
48
|
-
manual PR
|
|
54
|
+
manual PR. Qualify them the same way.
|
|
55
|
+
|
|
56
|
+
## TypeScript 6 and 7 side by side
|
|
57
|
+
|
|
58
|
+
`package.json` carries two TypeScript copies on purpose:
|
|
59
|
+
|
|
60
|
+
| Dependency | Version | Used by |
|
|
61
|
+
| --- | --- | --- |
|
|
62
|
+
| `typescript` | `^6.0.3` | typescript-eslint's parser, via `pnpm lint` |
|
|
63
|
+
| `typescript-7` (alias of `typescript`) | `^7.0.2` | `pnpm typecheck` and `pnpm build` |
|
|
64
|
+
|
|
65
|
+
TypeScript 7 ships no JavaScript API. Its `typescript` entry point exports only
|
|
66
|
+
`version` and `versionMajorMinor`, so anything that calls `require('typescript')`
|
|
67
|
+
expecting the classic compiler API breaks. typescript-eslint's parser is built on
|
|
68
|
+
that API and crashes on load
|
|
69
|
+
([typescript-eslint#12518](https://github.com/typescript-eslint/typescript-eslint/issues/12518),
|
|
70
|
+
tracked in [#10940](https://github.com/typescript-eslint/typescript-eslint/issues/10940)).
|
|
71
|
+
|
|
72
|
+
Microsoft [documented running the two side by side](https://devblogs.microsoft.com/typescript/announcing-typescript-7-0/#running-side-by-side-with-typescript-6.0)
|
|
73
|
+
for exactly this reason, and expects TypeScript 7.1 to ship the replacement API.
|
|
74
|
+
Lint stays on 6, the compiler is 7, and there's no syntax divergence to worry
|
|
75
|
+
about because 7.0 is a port of 6.x semantics rather than a new language version.
|
|
76
|
+
|
|
77
|
+
**When TypeScript 7.1 lands with an API and typescript-eslint supports it:** drop
|
|
78
|
+
`typescript-7`, move `typescript` to `^7`, and point `build` / `typecheck` back at
|
|
79
|
+
plain `tsc`.
|
|
49
80
|
|
|
50
81
|
## Versioning
|
|
51
82
|
|
|
52
83
|
SemVer, treated as a **CLI/application** contract:
|
|
53
84
|
|
|
54
|
-
- **MAJOR
|
|
85
|
+
- **MAJOR.** Breaking CLI/behavior: a command or flag removed, a plan-format or
|
|
55
86
|
config break, a provider-contract change users depend on.
|
|
56
|
-
- **MINOR
|
|
57
|
-
- **PATCH
|
|
87
|
+
- **MINOR.** New backward-compatible surface: a new command, flag, or provider.
|
|
88
|
+
- **PATCH.** Bug fixes, no new surface.
|
|
58
89
|
|
|
59
90
|
Mapping a dependency bump to neal's version:
|
|
60
91
|
|
|
@@ -64,7 +95,7 @@ Mapping a dependency bump to neal's version:
|
|
|
64
95
|
|
|
65
96
|
**Pre-1.0:** while on `0.x`, minor may break (the honest "still evolving"
|
|
66
97
|
contract). Move to **`1.0.0`** deliberately, once the CLI surface, plan format,
|
|
67
|
-
and provider contracts are stable enough to promise compatibility
|
|
98
|
+
and provider contracts are stable enough to promise compatibility, not before.
|
|
68
99
|
|
|
69
100
|
## Changelog discipline
|
|
70
101
|
|