@navels/neal 0.2.0 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (114) hide show
  1. package/README.md +206 -209
  2. package/SECURITY.md +16 -19
  3. package/dist/neal/adjudicator/{blocked-adjudicator.js → consultant.js} +21 -21
  4. package/dist/neal/adjudicator/execute.js +33 -33
  5. package/dist/neal/adjudicator/final-completion.js +2 -31
  6. package/dist/neal/adjudicator/planning.js +230 -38
  7. package/dist/neal/agents/prompts.js +6 -17
  8. package/dist/neal/agents/rounds.js +43 -73
  9. package/dist/neal/agents/schemas.js +47 -13
  10. package/dist/neal/blocked-guidance.js +16 -9
  11. package/dist/neal/cli.js +2 -2
  12. package/dist/neal/commands/check.js +2 -2
  13. package/dist/neal/commands/compat.js +100 -36
  14. package/dist/neal/commands/interactive-activity.js +2 -2
  15. package/dist/neal/commands/recovery-guidance.js +20 -8
  16. package/dist/neal/commands/runtime.js +4 -2
  17. package/dist/neal/config.js +16 -7
  18. package/dist/neal/context/inline-review-context.js +13 -28
  19. package/dist/neal/context/reviewer-context.js +19 -14
  20. package/dist/neal/eval/reviewer-eval-manifest.js +86 -0
  21. package/dist/neal/eval/reviewer-eval.js +155 -0
  22. package/dist/neal/interactive-controls.js +5 -0
  23. package/dist/neal/orchestrator/completion.js +49 -2
  24. package/dist/neal/orchestrator/failures.js +7 -0
  25. package/dist/neal/orchestrator/notifications.js +9 -9
  26. package/dist/neal/orchestrator/phases/planning.js +117 -80
  27. package/dist/neal/orchestrator/phases/recovery.js +151 -115
  28. package/dist/neal/orchestrator/phases/review.js +28 -1
  29. package/dist/neal/orchestrator/transitions.js +7 -1
  30. package/dist/neal/orchestrator.js +3 -0
  31. package/dist/neal/plan-queue.js +29 -1
  32. package/dist/neal/prompts/execute.js +7 -19
  33. package/dist/neal/prompts/planning.js +64 -55
  34. package/dist/neal/prompts/review-doctrine.js +13 -50
  35. package/dist/neal/prompts/specialized.js +7 -17
  36. package/dist/neal/prompts/specs.js +180 -32
  37. package/dist/neal/providers/agent-settings-isolation.js +35 -0
  38. package/dist/neal/providers/anthropic-claude.js +18 -1
  39. package/dist/neal/providers/detection.js +3 -7
  40. package/dist/neal/providers/git-config-isolation.js +83 -0
  41. package/dist/neal/providers/openai-codex.js +25 -3
  42. package/dist/neal/providers/{generic-agentic-tools.js → openai-compatible-tools.js} +11 -5
  43. package/dist/neal/providers/openai-compatible.js +1078 -430
  44. package/dist/neal/providers/openrouter-routing.js +68 -0
  45. package/dist/neal/providers/pricing.js +1 -1
  46. package/dist/neal/providers/rate-card.js +6 -32
  47. package/dist/neal/providers/registry.js +6 -17
  48. package/dist/neal/providers/types.js +12 -0
  49. package/dist/neal/resume-decision.js +7 -6
  50. package/dist/neal/resume-planner.js +9 -0
  51. package/dist/neal/retrospective.js +3 -3
  52. package/dist/neal/review-debt.js +30 -0
  53. package/dist/neal/review-findings/provider.js +17 -29
  54. package/dist/neal/review.js +17 -1
  55. package/dist/neal/state-invariants.js +11 -3
  56. package/dist/neal/state-views.js +27 -5
  57. package/dist/neal/state.js +59 -15
  58. package/dist/neal/status.js +28 -1
  59. package/dist/neal/support.js +3 -3
  60. package/docs/README.md +22 -18
  61. package/docs/{ADJUDICATOR_INVENTORY.md → adjudicator-inventory.md} +58 -57
  62. package/docs/architecture.md +79 -83
  63. package/docs/automation.md +11 -11
  64. package/docs/compat.md +91 -61
  65. package/docs/compatible-models.md +145 -121
  66. package/docs/demo.md +15 -5
  67. package/docs/maintenance.md +65 -34
  68. package/docs/plan-format.md +61 -33
  69. package/docs/prompt-evals.md +96 -0
  70. package/docs/{PROMPT_SPECS.md → prompt-specs.md} +52 -48
  71. package/docs/providers.md +279 -349
  72. package/docs/release.md +63 -81
  73. package/docs/review-convergence.md +246 -0
  74. package/docs/state-machine.md +61 -59
  75. package/docs/storage.md +40 -31
  76. package/docs/troubleshooting.md +32 -29
  77. package/examples/compat/add-edit-verify/PLAN.md +2 -0
  78. package/examples/compat/add-edit-verify/broken.diff +2 -2
  79. package/examples/compat/add-edit-verify/good.diff +2 -2
  80. package/examples/compat/add-edit-verify/src/add.js +1 -1
  81. package/examples/compat/manifest.json +25 -29
  82. package/examples/compat/sum-grep-edit/PLAN.md +33 -0
  83. package/examples/compat/sum-grep-edit/broken.diff +12 -0
  84. package/examples/compat/sum-grep-edit/good.diff +12 -0
  85. package/examples/compat/sum-grep-edit/package.json +5 -0
  86. package/examples/compat/sum-grep-edit/src/numbers.js +9 -0
  87. package/examples/compat/sum-grep-edit/test/numbers.test.js +15 -0
  88. package/examples/issue-triage-js/README.md +0 -15
  89. package/examples/reviewer-eval/clean-doc/base/src/clamp.ts +3 -0
  90. package/examples/reviewer-eval/clean-doc/change.diff +9 -0
  91. package/examples/reviewer-eval/clean-extract/base/src/greet.ts +3 -0
  92. package/examples/reviewer-eval/clean-extract/change.diff +13 -0
  93. package/examples/reviewer-eval/dropped-error/base/src/load.ts +4 -0
  94. package/examples/reviewer-eval/dropped-error/change.diff +15 -0
  95. package/examples/reviewer-eval/manifest.json +45 -0
  96. package/examples/reviewer-eval/off-by-one/base/src/paginate.ts +5 -0
  97. package/examples/reviewer-eval/off-by-one/change.diff +11 -0
  98. package/{config.yml → neal.yml} +1 -1
  99. package/package.json +15 -14
  100. package/dist/neal/providers/generic-agentic.js +0 -1256
  101. package/docs/comparison.md +0 -105
  102. package/docs/issue-pipeline.md +0 -124
  103. package/examples/compat/is-even-add-test/PLAN.md +0 -30
  104. package/examples/compat/is-even-add-test/broken.diff +0 -11
  105. package/examples/compat/is-even-add-test/good.diff +0 -11
  106. package/examples/compat/is-even-add-test/package.json +0 -5
  107. package/examples/compat/is-even-add-test/src/is-even.js +0 -3
  108. package/examples/compat/is-even-add-test/test/is-even.test.js +0 -9
  109. package/examples/compat/reverse-grep-edit/PLAN.md +0 -32
  110. package/examples/compat/reverse-grep-edit/broken.diff +0 -12
  111. package/examples/compat/reverse-grep-edit/good.diff +0 -12
  112. package/examples/compat/reverse-grep-edit/package.json +0 -5
  113. package/examples/compat/reverse-grep-edit/src/strings.js +0 -10
  114. package/examples/compat/reverse-grep-edit/test/strings.test.js +0 -15
@@ -1,114 +1,153 @@
1
1
  # Compatible models
2
2
 
3
- > **Last updated:** 2026-06-18 · **Reference:** `openai-codex` (gpt-5.5) · **Pool:** 79 OpenRouter models
3
+ > **Last updated:** 2026-07-20 · **Reference:** `openai-codex` (gpt-5.5) · **Pool:** 90 OpenRouter models
4
4
  >
5
5
  > The whitelist from running [`neal compat`](compat.md) across an OpenRouter candidate
6
- > pool. It records **compatibility, not skill**: a PASS means the model can drive neal's
7
- > loop on the trivial bundled fixtures in that role it says nothing about how well the
8
- > model performs on real work. Choose among the PASSes by your own cost/quality needs.
6
+ > pool. It records **compatibility, not skill**: a PASS means the model can drive
7
+ > neal's loop on the trivial bundled fixtures in that role. It says nothing about how
8
+ > well the model performs on real work. Choose among the PASSes by your own
9
+ > cost/quality needs.
9
10
 
10
11
  ## How to read this
11
12
 
12
- - A PASS means the model drove neal under **real schema-enforced structured output**
13
- (`type: 'json_schema'`), with `openai-codex` (gpt-5.5) running the non-tested roles as a
14
- known-good partner. Role cells = passed/total fixtures (coder 3, reviewer 6, planner 1).
15
- - A FAIL is **attributable** to the model via its [failure mode](compat.md#failure-mode-taxonomy)
16
- (e.g. `structured_output`, not a generic `provider_failed`).
17
- - **Single-run results are noisy at the margin.** Models that fail only 1–2 cells, or fail
18
- with `provider error`, often flip on a re-run treat the "borderline" table as *re-run
19
- candidates*, not verdicts.
20
- - Native adapters (`openai-codex`, `anthropic-claude`) are supported by construction and
21
- need no compat run.
22
-
23
- ## Compatible (25) verified across all roles
13
+ - Scored under the **discrimination criterion** ([#49](https://github.com/navels/neal/issues/49),
14
+ `schemaVersion: 2`): a reviewer PASSes a fixture iff it raises at least one blocking
15
+ finding on the broken diff **and** strictly fewer on the good diff. Severity
16
+ calibration is not graded. Earlier sweeps (2026-06-18) required zero blocking
17
+ findings on the good diff and are not comparable.
18
+ - The pool is the 2026-06-18 pool plus every qualifying model added to OpenRouter
19
+ since (paid, text-output chat models; no `:free` variants, no routers/aliases, no
20
+ native-adapter providers).
21
+ - Role cells = passed/total fixtures (coder 2, reviewer 2 pairs, planner 1. The
22
+ redundant `is-even-add-test` fixture was dropped 2026-07-20).
23
+ - **Single-run results are noisy at the margin.** Fail-only-once models were
24
+ re-confirmed once. Results below note where a verdict flipped on re-run.
25
+ - Native adapters (`openai-codex`, `anthropic-claude`) are supported by construction
26
+ and need no compat run.
27
+
28
+ ## Compatible (44) - verified across all roles
24
29
 
25
30
  Sorted cheapest first ($/Mtok in·out, from the live OpenRouter catalog).
26
31
 
27
- > **Drift observed (2026-07-14):** `google/gemma-4-26b-a4b-it` an all-roles PASS on
28
- > 2026-06-18 — failed the coder role 3/3 with `structured_output` in the CI smoke
29
- > (self-reference configuration). Slugs rot as providers re-route or re-quantize;
30
- > treat its row below as stale until re-qualified against the codex reference.
31
- >
32
- > **Self-reference caveat (2026-07-14):** `inclusionai/ling-2.6-1t` repeatedly failed the
33
- > reviewer broken-diff cell (2/3 attempts) in the CI smoke when serving as **its own
34
- > partner**. Its codex-referenced row below stands this is a partner-duty weakness,
35
- > the same class the reference investigation documents. The smoke pin is
36
- > `minimax/minimax-m2.5`, which held up under self-reference.
37
-
38
- | Model | Coder | Reviewer | Planner | $/Mtok | ctx |
32
+ | Model | Coder | Reviewer | Planner | $/Mtok | ctx | Note |
33
+ |---|---|---|---|---|---|---|
34
+ | `nex-agi/nex-n2-mini` | 2/2 | 4/4 | 1/1 | 0.02·0.10 | 262k | |
35
+ | `nvidia/nemotron-3-nano-30b-a3b` | 2/2 | 4/4 | 1/1 | 0.05·0.20 | 262k | |
36
+ | `google/gemma-4-26b-a4b-it` | 2/2 | 4/4 | 1/1 | 0.07·0.34 | 262k | |
37
+ | `inclusionai/ling-2.6-1t` | 2/2 | 4/4 | 1/1 | 0.07·0.62 | 262k | |
38
+ | `nvidia/nemotron-3-super-120b-a12b` | 2/2 | 4/4 | 1/1 | 0.08·0.40 | 1000k | |
39
+ | `deepseek/deepseek-v4-flash` | 2/2 | 4/4 | 1/1 | 0.10·0.20 | 1049k | |
40
+ | `qwen/qwen3.6-35b-a3b` | 2/2 | 4/4 | 1/1 | 0.14·1.00 | 262k | |
41
+ | `kwaipilot/kat-coder-air-v2.5` | 2/2 | 4/4 | 1/1 | 0.15·0.60 | 256k | |
42
+ | `minimax/minimax-m2.5` | 2/2 | 4/4 | 1/1 | 0.15·0.90 | 205k | |
43
+ | `mistralai/mistral-small-2603` | 2/2 | 4/4 | 1/1 | 0.15·0.60 | 262k | |
44
+ | `stepfun/step-3.7-flash` | 2/2 | 4/4 | 1/1 | 0.20·1.15 | 256k | |
45
+ | `tencent/hy3` | 2/2 | 4/4 | 1/1 | 0.20·0.80 | 262k | |
46
+ | `google/gemma-4-31b-it` | 2/2 | 4/4 | 1/1 | 0.22·0.55 | 262k | |
47
+ | `deepseek/deepseek-chat-v3.1` | 2/2 | 4/4 | 1/1 | 0.25·0.95 | 164k | |
48
+ | `google/gemini-3.1-flash-lite` | 2/2 | 4/4 | 1/1 | 0.25·1.50 | 1049k | |
49
+ | `google/gemini-3.1-flash-lite-preview` | 2/2 | 4/4 | 1/1 | 0.25·1.50 | 1049k | |
50
+ | `minimax/minimax-m2.7` | 2/2 | 4/4 | 1/1 | 0.25·1.00 | 205k | first attempt failed (protocol:3). Confirmed PASS on re-run |
51
+ | `z-ai/glm-5.2` | 2/2 | 4/4 | 1/1 | 0.26·0.81 | 1049k | |
52
+ | `deepseek/deepseek-v3.1-terminus` | 2/2 | 4/4 | 1/1 | 0.27·1.00 | 131k | |
53
+ | `deepseek/deepseek-v3.2` | 2/2 | 4/4 | 1/1 | 0.27·0.40 | 164k | |
54
+ | `deepseek/deepseek-v3.2-exp` | 2/2 | 4/4 | 1/1 | 0.27·0.41 | 164k | |
55
+ | `minimax/minimax-m2` | 2/2 | 4/4 | 1/1 | 0.30·1.20 | 205k | |
56
+ | `qwen/qwen3.5-plus-20260420` | 2/2 | 4/4 | 1/1 | 0.30·1.80 | 1000k | |
57
+ | `qwen/qwen3.7-plus` | 2/2 | 4/4 | 1/1 | 0.32·1.28 | 1000k | |
58
+ | `deepseek/deepseek-v4-pro` | 2/2 | 4/4 | 1/1 | 0.43·0.87 | 1049k | first attempt failed (structured_output:1). Confirmed PASS on re-run |
59
+ | `moonshotai/kimi-k2-0905` | 2/2 | 4/4 | 1/1 | 0.60·2.50 | 262k | |
60
+ | `moonshotai/kimi-k2-thinking` | 2/2 | 4/4 | 1/1 | 0.60·2.50 | 262k | |
61
+ | `nvidia/nemotron-3-ultra-550b-a55b` | 2/2 | 4/4 | 1/1 | 0.60·3.60 | 1000k | |
62
+ | `kwaipilot/kat-coder-pro-v2.5` | 2/2 | 4/4 | 1/1 | 0.74·2.96 | 256k | |
63
+ | `moonshotai/kimi-k2.7-code` | 2/2 | 4/4 | 1/1 | 0.85·3.80 | 262k | |
64
+ | `z-ai/glm-5` | 2/2 | 4/4 | 1/1 | 0.95·2.55 | 205k | |
65
+ | `z-ai/glm-5.1` | 2/2 | 4/4 | 1/1 | 0.97·3.04 | 203k | |
66
+ | `x-ai/grok-build-0.1` | 2/2 | 4/4 | 1/1 | 1.00·2.00 | 256k | |
67
+ | `qwen/qwen3.6-max-preview` | 2/2 | 4/4 | 1/1 | 1.04·6.24 | 262k | |
68
+ | `meta/muse-spark-1.1` | 2/2 | 4/4 | 1/1 | 1.25·4.25 | 1049k | |
69
+ | `x-ai/grok-4.20` | 2/2 | 4/4 | 1/1 | 1.25·2.50 | 2000k | |
70
+ | `x-ai/grok-4.3` | 2/2 | 4/4 | 1/1 | 1.25·2.50 | 1000k | first attempt failed (provider_failed:2). Confirmed PASS on re-run |
71
+ | `qwen/qwen3.7-max` | 2/2 | 4/4 | 1/1 | 1.48·4.42 | 1000k | |
72
+ | `google/gemini-3.5-flash` | 2/2 | 4/4 | 1/1 | 1.50·9.00 | 1049k | |
73
+ | `mistralai/mistral-medium-3-5` | 2/2 | 4/4 | 1/1 | 1.50·7.50 | 262k | |
74
+ | `google/gemini-3.1-pro-preview-customtools` | 2/2 | 4/4 | 1/1 | 2.00·12.00 | 1049k | |
75
+ | `x-ai/grok-4.5` | 2/2 | 4/4 | 1/1 | 2.00·6.00 | 500k | |
76
+ | `moonshotai/kimi-k3` | 2/2 | 4/4 | 1/1 | 3.00·15.00 | 1049k | first attempt failed (provider_failed:1). Confirmed PASS on re-run |
77
+ | `sakana/fugu-ultra` | 2/2 | 4/4 | 1/1 | 5.00·30.00 | 1000k | |
78
+
79
+ ## Borderline - re-run candidates (10)
80
+
81
+ Failed one or a few cells with a non-protocol mode, or failed differently across
82
+ attempts. `provider_failed` rows are transient infrastructure. The rest are
83
+ single-run signals that may flip. Re-run before excluding.
84
+
85
+ | Model | Coder | Reviewer | Planner | Tripped on | Note |
39
86
  |---|---|---|---|---|---|
40
- | `google/gemma-4-26b-a4b-it` | 3/3 | 6/6 | 1/1 | 0.06·0.33 | 262k |
41
- | `inclusionai/ling-2.6-1t` | 3/3 | 6/6 | 1/1 | 0.07·0.62 | 262k |
42
- | `deepseek/deepseek-v4-flash` | 3/3 | 6/6 | 1/1 | 0.09·0.18 | 1048k |
43
- | `nvidia/nemotron-3-super-120b-a12b` | 3/3 | 6/6 | 1/1 | 0.09·0.45 | 1000k |
44
- | `google/gemma-4-31b-it` | 3/3 | 6/6 | 1/1 | 0.12·0.35 | 262k |
45
- | `qwen/qwen3.6-35b-a3b` | 3/3 | 6/6 | 1/1 | 0.14·1.00 | 262k |
46
- | `minimax/minimax-m2.5` | 3/3 | 6/6 | 1/1 | 0.15·0.90 | 204k |
47
- | `deepseek/deepseek-chat-v3.1` | 3/3 | 6/6 | 1/1 | 0.21·0.79 | 163k |
48
- | `deepseek/deepseek-v3.2` | 3/3 | 6/6 | 1/1 | 0.23·0.34 | 131k |
49
- | `bytedance-seed/seed-2.0-lite` | 3/3 | 6/6 | 1/1 | 0.25·2.00 | 262k |
50
- | `google/gemini-3.1-flash-lite` | 3/3 | 6/6 | 1/1 | 0.25·1.50 | 1048k |
51
- | `minimax/minimax-m2.7` | 3/3 | 6/6 | 1/1 | 0.25·1.00 | 204k |
52
- | `google/gemini-3.1-flash-lite-preview` | 3/3 | 6/6 | 1/1 | 0.25·1.50 | 1048k |
53
- | `minimax/minimax-m2` | 3/3 | 6/6 | 1/1 | 0.26·1.00 | 204k |
54
- | `deepseek/deepseek-v3.1-terminus` | 3/3 | 6/6 | 1/1 | 0.27·0.95 | 163k |
55
- | `minimax/minimax-m2.1` | 3/3 | 6/6 | 1/1 | 0.29·0.95 | 204k |
56
- | `minimax/minimax-m3` | 3/3 | 6/6 | 1/1 | 0.30·1.20 | 1048k |
57
- | `qwen/qwen3.7-plus` | 3/3 | 6/6 | 1/1 | 0.32·1.28 | 1000k |
58
- | `z-ai/glm-5` | 3/3 | 6/6 | 1/1 | 0.60·1.92 | 202k |
59
- | `z-ai/glm-5.1` | 3/3 | 6/6 | 1/1 | 0.98·3.08 | 202k |
60
- | `qwen/qwen3.6-max-preview` | 3/3 | 6/6 | 1/1 | 1.04·6.24 | 262k |
61
- | `x-ai/grok-4.20` | 3/3 | 6/6 | 1/1 | 1.25·2.50 | 2000k |
62
- | `x-ai/grok-4.3` | 3/3 | 6/6 | 1/1 | 1.25·2.50 | 1000k |
63
- | `google/gemini-3.5-flash` | 3/3 | 6/6 | 1/1 | 1.50·9.00 | 1048k |
64
- | `google/gemini-3.1-pro-preview-customtools` | 3/3 | 6/6 | 1/1 | 2.00·12.00 | 1048k |
65
-
66
- ## ⚠️ Borderline — re-run candidates (failed 1–2 cells, or transient)
67
-
68
- These drove most of neal but tripped on one or two cells. `provider error` rows are
69
- transient (re-run likely flips them); `bad review output` / `can't emit schema JSON` are
70
- single-run signals that often flip too. Re-run before excluding.
71
-
72
- | Model | Coder | Reviewer | Planner | Tripped on |
73
- |---|---|---|---|---|
74
- | `mistralai/devstral-2512` | 3/3 | 6/6 | 0/1 | planner: provider error |
75
- | `nvidia/nemotron-3-ultra-550b-a55b` | 3/3 | 5/6 | 1/1 | reviewer: provider error |
76
- | `x-ai/grok-build-0.1` | 3/3 | 5/6 | 1/1 | reviewer: provider error |
77
- | `nvidia/nemotron-3-nano-30b-a3b` | 3/3 | 5/6 | 1/1 | reviewer: bad review output |
78
- | `deepseek/deepseek-v3.2-exp` | 3/3 | 5/6 | 1/1 | reviewer: bad review output |
79
- | `mistralai/mistral-medium-3-5` | 3/3 | 5/6 | 1/1 | reviewer: bad review output |
80
- | `qwen/qwen3.5-plus-20260420` | 3/3 | 5/6 | 1/1 | reviewer: bad review output |
81
- | `qwen/qwen3.7-max` | 3/3 | 5/6 | 1/1 | reviewer: bad review output |
82
- | `qwen/qwen3.6-flash` | 3/3 | 5/6 | 1/1 | reviewer: can't emit schema JSON |
83
- | `deepseek/deepseek-v4-pro` | 3/3 | 4/6 | 1/1 | reviewer: can't emit schema JSON |
84
- | `meta-llama/llama-4-maverick` | 1/3 | 6/6 | 1/1 | coder: left block unresolved |
85
- | `moonshotai/kimi-k2.5` | 2/3 | 6/6 | 1/1 | coder: can't emit schema JSON |
86
- | `stepfun/step-3.7-flash` | 2/3 | 6/6 | 1/1 | coder: provider error |
87
- | `inclusionai/ring-2.6-1t` | 2/3 | 5/6 | 1/1 | coder+reviewer: schema JSON |
88
- | `z-ai/glm-5.2` | 2/3 | 5/6 | 1/1 | coder+reviewer: schema JSON |
89
- | `inclusionai/ling-2.6-flash` | 0/3 | 4/6 | 1/1 | coder+reviewer: provider error |
90
- | `meta-llama/llama-3.3-70b-instruct` | 0/3 | 3/6 | 0/1 | provider error |
91
- | `bytedance-seed/seed-1.6` | 0/3 | 5/6 | 1/1 | coder: can't emit schema JSON |
92
- | `bytedance-seed/seed-1.6-flash` | 0/3 | 5/6 | 0/1 | coder: can't emit schema JSON |
93
-
94
- > Notable: the flagship `deepseek-v4-pro` FAILs (reviewer schema), but the cheaper
95
- > `deepseek-v4-flash` / `v3.2` PASS cleanly and v4-pro has flaked stochastically before,
96
- > so it's a re-run candidate, not a hard FAIL.
97
-
98
- ## Incompatible fail the provider handshake (no tool-use / unparsable JSON)
99
-
100
- These can't drive neal's agentic loop on OpenRouter at all (`protocol` failure on the
101
- provider check 0/1 every role): mostly small models and ones whose OR endpoints don't
102
- support tool use.
103
-
104
- `amazon/nova-{micro,lite,2-lite,pro,premier}-v1`, `bytedance-seed/seed-2.0-mini`,
105
- `cohere/{command-a,command-r-08-2024,command-r-plus-08-2024,command-r7b-12-2024}`,
106
- `meta-llama/{llama-3.1-8b,llama-3.2-1b,llama-3.2-3b}-instruct`, `meta-llama/llama-4-scout`,
107
- `microsoft/{phi-4,phi-4-mini-instruct,wizardlm-2-8x22b}`, `minimax/minimax-m2-her`,
108
- `mistralai/{ministral-3b,ministral-8b,ministral-14b}-2512`, `mistralai/mistral-small-2603`,
109
- `moonshotai/{kimi-k2,kimi-k2-0905,kimi-k2-thinking,kimi-k2.6,kimi-k2.7-code}`,
110
- `nvidia/llama-3.3-nemotron-super-49b-v1.5`, `openai/{gpt-oss-120b,gpt-oss-20b,gpt-oss-safeguard-20b}`,
111
- `stepfun/step-3.5-flash`, `x-ai/grok-4.20-multi-agent`, `z-ai/{glm-4.7-flash,glm-5-turbo}`.
87
+ | `bytedance-seed/seed-1.6` | 0/2 | 4/4 | 1/1 | coder/add-edit-verify: structured_output | |
88
+ | `bytedance-seed/seed-1.6-flash` | 0/2 | 0/4 | 1/1 | coder/add-edit-verify: provider_failed | |
89
+ | `bytedance-seed/seed-2.0-lite` | 0/2 | 4/4 | 1/1 | coder/add-edit-verify: structured_output | confirmed FAIL 2/2 (attempt 1 {'structured_output': 2}, attempt 2 {'structured_output': 1}) |
90
+ | `inclusionai/ling-2.6-flash` | 1/2 | 0/4 | 0/1 | coder/add-edit-verify: provider_failed | |
91
+ | `meta-llama/llama-3.1-8b-instruct` | 0/2 | 0/4 | 0/1 | coder/add-edit-verify: provider_failed | |
92
+ | `meta-llama/llama-4-maverick` | 1/2 | 4/4 | 1/1 | coder/add-edit-verify: block_unresolved | confirmed FAIL 2/2 (attempt 1 {'block_unresolved': 1}, attempt 2 {'structured_output': 1, 'block_unresolved': 1}) |
93
+ | `mistralai/devstral-2512` | 0/2 | 4/4 | 1/1 | coder/add-edit-verify: provider_failed | confirmed FAIL 2/2 (attempt 1 {'provider_failed': 2}, attempt 2 {'provider_failed': 1}) |
94
+ | `moonshotai/kimi-k2.5` | 1/2 | 4/4 | 1/1 | coder/sum-grep-edit: structured_output | |
95
+ | `moonshotai/kimi-k2.6` | 2/2 | 2/4 | 1/1 | reviewer/sum-grep-edit: structured_output | |
96
+ | `qwen/qwen3.6-flash` | 2/2 | 0/4 | 1/1 | reviewer/add-edit-verify: structured_output | |
97
+
98
+ ## Incompatible (36) - fail the provider handshake
99
+
100
+ These can't drive neal's agentic loop on OpenRouter at all (`protocol` failure on
101
+ the provider check, every role): mostly small models and ones whose OR endpoints
102
+ don't support tool use.
103
+
104
+ - `aion-labs/aion-3.0`
105
+ - `aion-labs/aion-3.0-mini`
106
+ - `amazon/nova-2-lite-v1`
107
+ - `amazon/nova-lite-v1`
108
+ - `amazon/nova-micro-v1`
109
+ - `amazon/nova-premier-v1`
110
+ - `amazon/nova-pro-v1`
111
+ - `bytedance-seed/seed-2.0-mini`
112
+ - `cohere/command-a`
113
+ - `cohere/command-r-08-2024`
114
+ - `cohere/command-r-plus-08-2024`
115
+ - `cohere/command-r7b-12-2024`
116
+ - `inclusionai/ring-2.6-1t`
117
+ - `meta-llama/llama-3.2-1b-instruct`
118
+ - `meta-llama/llama-3.2-3b-instruct`
119
+ - `meta-llama/llama-3.3-70b-instruct`
120
+ - `meta-llama/llama-4-scout`
121
+ - `microsoft/phi-4`
122
+ - `microsoft/wizardlm-2-8x22b`
123
+ - `minimax/minimax-m2-her`
124
+ - `minimax/minimax-m2.1` *(confirmed 2/2)*
125
+ - `minimax/minimax-m3` *(confirmed 2/2)*
126
+ - `mistralai/ministral-14b-2512`
127
+ - `mistralai/ministral-3b-2512`
128
+ - `mistralai/ministral-8b-2512`
129
+ - `moonshotai/kimi-k2`
130
+ - `openai/gpt-oss-120b`
131
+ - `openai/gpt-oss-20b`
132
+ - `openai/gpt-oss-safeguard-20b`
133
+ - `poolside/laguna-xs-2.1`
134
+ - `stepfun/step-3.5-flash`
135
+ - `thinkingmachines/inkling`
136
+ - `x-ai/grok-4.20-multi-agent`
137
+ - `z-ai/glm-4.7`
138
+ - `z-ai/glm-4.7-flash`
139
+ - `z-ai/glm-5-turbo`
140
+
141
+ > **`require_parameters` note:** `minimax-m2.1` and `minimax-m3` (June PASSes, now
142
+ > `protocol`, confirmed 2/2) trace to the `require_parameters: true` OpenRouter
143
+ > routing constraint (#46), which excludes backends that do not support
144
+ > `json_schema`. This is a routing/infrastructure attribution, not a model
145
+ > capability change. Re-run if OpenRouter adds a capable backend for these slugs.
146
+ > `minimax-m2.7` is intermittent under the same constraint (failed the sweep,
147
+ > passed the confirm). The same constraint also *fixed* several June `protocol`
148
+ > failures by steering routing to capable backends: `moonshotai/kimi-k2-0905`,
149
+ > `kimi-k2-thinking`, `kimi-k2.7-code`, and `mistralai/mistral-small-2603` all
150
+ > flipped to full PASS.
112
151
 
113
152
  ## Native adapters (supported, no compat run required)
114
153
 
@@ -123,24 +162,9 @@ support tool use.
123
162
  neal compat --model <slug> --role all --reference openai-codex --json
124
163
  ```
125
164
 
126
- `neal compat` also accepts an OpenRouter model as the reference
127
- (`--reference generic-agentic:<slug>`) to escape codex capacity limits, but
128
- **no OpenRouter model has validated as a drop-in reference** two were tried on
129
- 2026-06-18 and both failed:
130
-
131
- - `deepseek/deepseek-v4-flash` (cheap/fast) reproduced only **9** of the codex-ref
132
- **25** PASSes (13 dropped, confirmed in isolation — not a rate-limit artifact):
133
- too weak, it emits coder diffs the candidate-reviewer flags and blocks good
134
- candidate work as partner-reviewer.
135
- - `google/gemini-3.1-pro-preview-customtools` (strong/expensive) was worse in
136
- practice: of the 25 codex-PASS models only **1** reproduced, and it **wedged
137
- under load** (hung reference calls at 6-wide, timeouts/no-result on the
138
- PASS-class models at 3-wide). A reference that can't run wide defeats the
139
- throughput purpose of going off-codex in the first place.
140
-
141
- **Conclusion:** `openai-codex` (gpt-5.5) is the authoritative reference. To extend
142
- the whitelist beyond this pool, run additional candidates on codex-ref directly
143
- (throttle to about three concurrent `neal compat` invocations for codex
144
- capacity); the OpenRouter-reference path is not a viable shortcut today.
145
-
146
- Paid slugs only — never `:free`. Re-run when a slug's backing model version changes.
165
+ `openai-codex` (gpt-5.5) is the authoritative reference. No OpenRouter model has
166
+ validated as a drop-in reference (two tried 2026-06-18, both failed. See the
167
+ 2026-06-18 revision of this file). Throttle to about three concurrent invocations
168
+ for codex capacity.
169
+
170
+ Paid slugs only, never `:free`. Re-run when a slug's backing model version changes.
package/docs/demo.md CHANGED
@@ -1,4 +1,4 @@
1
- # Demo Recording
1
+ # Demo recording
2
2
 
3
3
  Use `asciinema` for public terminal demos when a recording is useful. It is an
4
4
  optional recording tool, not a package dependency.
@@ -6,7 +6,17 @@ optional recording tool, not a package dependency.
6
6
  Record from a disposable repository or throwaway worktree, preferably under
7
7
  `/tmp`, so the demo does not expose real project history, paths, or artifacts.
8
8
 
9
- ## Suggested Script
9
+ ## What to show
10
+
11
+ The README doesn't ship an embedded recording. If you make one, show the part
12
+ that's actually worth watching: the planner/coder/reviewer loop with the coder
13
+ and reviewer on different vendors, and the findings and responses going back and
14
+ forth until the reviewer is satisfied. A single trivial one-scope plan running
15
+ to completion doesn't show any of that. Pick a small change with a real review
16
+ point in it, so the reviewer has something to catch and the coder has something
17
+ to respond to.
18
+
19
+ ## Suggested script
10
20
 
11
21
  Create a small demo plan in the disposable repository:
12
22
 
@@ -24,7 +34,7 @@ $EDITOR tmp/DEMO_PLAN.md
24
34
  ```
25
35
 
26
36
  Configure providers and run `neal check` before recording, or record those
27
- steps only if the output is safe to share. Use current Neal commands while
37
+ steps only if the output is safe to share. Use current neal commands while
28
38
  recording:
29
39
 
30
40
  ```bash
@@ -35,7 +45,7 @@ exit
35
45
  ```
36
46
 
37
47
  Keep the recording output outside the demo repository so it does not appear as
38
- unrelated dirty work while Neal starts the writer run.
48
+ unrelated dirty work while neal starts the writer run.
39
49
 
40
50
  If you specifically want to show recovery from a run that is waiting for
41
51
  operator guidance, record that as a separate segment with the selected run id:
@@ -44,7 +54,7 @@ operator guidance, record that as a separate segment with the selected run id:
44
54
  neal resume --run <run-id> --message "Continue with the smallest safe follow-up."
45
55
  ```
46
56
 
47
- ## Scrubbing Checklist
57
+ ## Scrubbing checklist
48
58
 
49
59
  Before sharing a recording, review the terminal output for local paths, private
50
60
  project names, provider output, credentials, tokens, provider environment
@@ -1,60 +1,91 @@
1
- # Maintenance: dependencies & versioning
1
+ # Maintenance: dependencies and versioning
2
2
 
3
- neal's behavior is defined in large part by the agent SDKs it drives, so
4
- dependency updates are a first-class concern, not routine hygiene. This document
5
- is the policy.
3
+ neal's behavior comes largely from the agent SDKs it drives, so dependency
4
+ updates need real attention, not routine hygiene. Here's the policy.
6
5
 
7
- ## Three dependency tiers
6
+ ## Dependency buckets
8
7
 
9
- | Tier | Packages | Pinning | Update posture |
8
+ Grouped so at most a handful of dependency PRs are ever open at once:
9
+
10
+ | Bucket | Contents | Cadence | Update posture |
10
11
  | --- | --- | --- | --- |
11
- | **Native SDKs** | `@openai/codex-sdk`, `@anthropic-ai/claude-agent-sdk` | **exact** (enforced by `scripts/validate-release.mjs`) | **deliberate** behaviorally qualified on a subscription-authenticated machine via `scripts/qualify-sdk.sh`; never auto-merge |
12
- | **AI-SDK tier** | `ai`, `@ai-sdk/openai-compatible`, `zod` | **exact** (enforced by `scripts/validate-release.mjs`) | minor/patch **auto-merge** once CI **and the live smoke** are green (3-day soak); majors stay manual |
13
- | **Utility** | `dotenv`, `yaml`, `@types/node`, `tsx`, `typescript` | caret OK | routine auto-merge on green CI after a soak period |
12
+ | **Weekly non-major** | everything except the native SDKs (minor/patch/pin/digest) | weekly, one grouped PR | auto-merge after CI and the live smoke pass, subject to a 3-day soak |
13
+ | **Native SDKs** | `@openai/codex-sdk` and `@anthropic-ai/claude-agent-sdk`, both exact-pinned. `ai`, `@ai-sdk/openai-compatible`, and `zod` also stay exact-pinned. | one grouped PR, opened after the 3-day soak instead of waiting for the weekly schedule | qualify on a subscription-authenticated machine with `scripts/qualify-sdk.sh`. Never auto-merge. Skip the CI smoke because it does not exercise the native adapters. |
14
+ | **Library majors** | every npm major except the native SDKs (`typescript`, `@types/node`, `ai`, `@ai-sdk/openai-compatible`, `zod`, etc.) | monthly, one grouped PR | review manually because some need code changes |
15
+ | **GitHub Actions majors** | `actions/checkout`, `actions/setup-node`, `actions/upload-artifact`, etc. | monthly, one grouped PR separate from library majors | review and merge on green CI. Do not auto-merge because a major action bump can still change behavior. |
14
16
 
15
- The asymmetry exists because an agentic-SDK bump can change tool-calling,
16
- structured output, or sandbox behavior i.e. break neal's loop **without**
17
- breaking compilation. Utility deps cannot. The native/AI-SDK split exists
18
- because CI can behaviorally exercise the AI-SDK tier (the smoke drives it
19
- live through `generic-agentic`) but not the native adapters, whose auth is
20
- subscription-based and lives only on a maintainer's machine.
17
+ The native-SDK split exists because an agentic-SDK bump can change
18
+ tool-calling, structured output, or sandbox behavior. That breaks neal's loop
19
+ **without** breaking compilation. And CI can't behaviorally exercise the
20
+ native adapters: their auth is subscription-based and lives only on a
21
+ maintainer's machine. Everything else is behaviorally exercised in CI: the
22
+ live smoke runs on every package.json/lockfile PR, so the weekly grouped PR
23
+ is gated on it as a whole.
21
24
 
22
25
  ## The update flow
23
26
 
24
- 1. **Detect.** [Renovate](../renovate.json) opens PRs weekly: utility deps
25
- grouped, AI-SDK and native SDKs one-per-PR, pinned exact, labelled
26
- `agentic-sdk` (native and AI-SDK majors additionally `needs-qualification`).
27
- 2. **Verify (automatic).** CI (`.github/workflows/ci.yml`) runs typecheck + unit
28
- tests + package verification catches **API-shape / contract** breaks. The
29
- live smoke (`.github/workflows/smoke.yml`) runs on every package.json /
30
- lockfile PR: a real `neal compat` run against a cheap OpenRouter model
31
- through `generic-agentic`, catching **behavioral** breaks in the AI-SDK
32
- tier. AI-SDK minor/patch PRs auto-merge when both are green.
33
- **The smoke requires the `OPENROUTER_API_KEY` repo secret** — without it
27
+ 1. **Detect.** [Renovate](../renovate.json) opens grouped PRs: the weekly
28
+ non-major bucket, the native-SDK bucket (labelled `agentic-sdk` +
29
+ `needs-qualification`), and, monthly, separate library-majors and
30
+ GitHub-Actions-majors buckets (both labelled `needs-review`).
31
+ 2. **Verify (automatic).** CI (`.github/workflows/ci.yml`) runs typecheck + lint
32
+ + unit tests + package verification on Node 24.18.0, which catches
33
+ **API-shape / contract** breaks. The live smoke
34
+ (`.github/workflows/smoke.yml`) runs on every package.json / lockfile PR: a
35
+ real `neal compat` run against a cheap OpenRouter model through
36
+ `openai-compatible`, catching **behavioral** breaks in the AI-SDK tier. The
37
+ weekly non-major PR auto-merges when both are green.
38
+ **The smoke requires the `OPENROUTER_API_KEY` repo secret.** Without it
34
39
  the smoke skips (green) and the AI-SDK auto-merge gate is compile-only.
35
40
  3. **Verify (behavioral, native tier).** `@openai/codex-sdk` and
36
- `@anthropic-ai/claude-agent-sdk` cannot be smoked in CI, so qualify them
41
+ `@anthropic-ai/claude-agent-sdk` can't be smoked in CI, so qualify them
37
42
  from any checkout with authenticated Claude/Codex CLIs:
38
43
  ```
39
44
  scripts/qualify-sdk.sh <pr-number>
40
45
  ```
41
46
  It runs the full suite plus a live `neal compat --role all` pass-through on
42
- the bumped adapter (in a throwaway worktree, with roles and models pinned
43
- explicitly so nothing leaks from `~/.neal/config.yml`), posts the compat
44
- matrix to the PR, and approves on PASS (`--merge` also squash-merges).
47
+ every bumped adapter in the PR (in a throwaway worktree, with roles and
48
+ models pinned explicitly so nothing leaks from `~/.neal/config.yml`), posts
49
+ the compat matrices to the PR, and approves on PASS (`--merge` also
50
+ squash-merges).
45
51
  4. **Adopt.** Merge, bump neal's version, add a CHANGELOG entry noting the bump +
46
52
  any behavior change, and cut a release via the existing workflow. Urgent
47
53
  bumps (a fix neal needs immediately) may skip the Renovate soak with a
48
- manual PR qualify them the same way.
54
+ manual PR. Qualify them the same way.
55
+
56
+ ## TypeScript 6 and 7 side by side
57
+
58
+ `package.json` carries two TypeScript copies on purpose:
59
+
60
+ | Dependency | Version | Used by |
61
+ | --- | --- | --- |
62
+ | `typescript` | `^6.0.3` | typescript-eslint's parser, via `pnpm lint` |
63
+ | `typescript-7` (alias of `typescript`) | `^7.0.2` | `pnpm typecheck` and `pnpm build` |
64
+
65
+ TypeScript 7 ships no JavaScript API. Its `typescript` entry point exports only
66
+ `version` and `versionMajorMinor`, so anything that calls `require('typescript')`
67
+ expecting the classic compiler API breaks. typescript-eslint's parser is built on
68
+ that API and crashes on load
69
+ ([typescript-eslint#12518](https://github.com/typescript-eslint/typescript-eslint/issues/12518),
70
+ tracked in [#10940](https://github.com/typescript-eslint/typescript-eslint/issues/10940)).
71
+
72
+ Microsoft [documented running the two side by side](https://devblogs.microsoft.com/typescript/announcing-typescript-7-0/#running-side-by-side-with-typescript-6.0)
73
+ for exactly this reason, and expects TypeScript 7.1 to ship the replacement API.
74
+ Lint stays on 6, the compiler is 7, and there's no syntax divergence to worry
75
+ about because 7.0 is a port of 6.x semantics rather than a new language version.
76
+
77
+ **When TypeScript 7.1 lands with an API and typescript-eslint supports it:** drop
78
+ `typescript-7`, move `typescript` to `^7`, and point `build` / `typecheck` back at
79
+ plain `tsc`.
49
80
 
50
81
  ## Versioning
51
82
 
52
83
  SemVer, treated as a **CLI/application** contract:
53
84
 
54
- - **MAJOR** breaking CLI/behavior: a command or flag removed, a plan-format or
85
+ - **MAJOR.** Breaking CLI/behavior: a command or flag removed, a plan-format or
55
86
  config break, a provider-contract change users depend on.
56
- - **MINOR** new backward-compatible surface: a new command, flag, or provider.
57
- - **PATCH** bug fixes, no new surface.
87
+ - **MINOR.** New backward-compatible surface: a new command, flag, or provider.
88
+ - **PATCH.** Bug fixes, no new surface.
58
89
 
59
90
  Mapping a dependency bump to neal's version:
60
91
 
@@ -64,7 +95,7 @@ Mapping a dependency bump to neal's version:
64
95
 
65
96
  **Pre-1.0:** while on `0.x`, minor may break (the honest "still evolving"
66
97
  contract). Move to **`1.0.0`** deliberately, once the CLI surface, plan format,
67
- and provider contracts are stable enough to promise compatibility not before.
98
+ and provider contracts are stable enough to promise compatibility, not before.
68
99
 
69
100
  ## Changelog discipline
70
101