@navels/neal 0.2.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +206 -209
- package/SECURITY.md +16 -19
- package/dist/neal/adjudicator/{blocked-adjudicator.js → consultant.js} +21 -21
- package/dist/neal/adjudicator/execute.js +33 -33
- package/dist/neal/adjudicator/final-completion.js +2 -31
- package/dist/neal/adjudicator/planning.js +230 -38
- package/dist/neal/agents/prompts.js +6 -17
- package/dist/neal/agents/rounds.js +43 -73
- package/dist/neal/agents/schemas.js +47 -13
- package/dist/neal/blocked-guidance.js +16 -9
- package/dist/neal/cli.js +2 -2
- package/dist/neal/commands/check.js +2 -2
- package/dist/neal/commands/compat.js +100 -36
- package/dist/neal/commands/interactive-activity.js +2 -2
- package/dist/neal/commands/recovery-guidance.js +20 -8
- package/dist/neal/commands/runtime.js +4 -2
- package/dist/neal/config.js +16 -7
- package/dist/neal/context/inline-review-context.js +13 -28
- package/dist/neal/context/reviewer-context.js +19 -14
- package/dist/neal/eval/reviewer-eval-manifest.js +86 -0
- package/dist/neal/eval/reviewer-eval.js +155 -0
- package/dist/neal/interactive-controls.js +5 -0
- package/dist/neal/orchestrator/completion.js +49 -2
- package/dist/neal/orchestrator/failures.js +7 -0
- package/dist/neal/orchestrator/notifications.js +9 -9
- package/dist/neal/orchestrator/phases/planning.js +117 -80
- package/dist/neal/orchestrator/phases/recovery.js +151 -115
- package/dist/neal/orchestrator/phases/review.js +28 -1
- package/dist/neal/orchestrator/transitions.js +7 -1
- package/dist/neal/orchestrator.js +3 -0
- package/dist/neal/plan-queue.js +29 -1
- package/dist/neal/prompts/execute.js +7 -19
- package/dist/neal/prompts/planning.js +64 -55
- package/dist/neal/prompts/review-doctrine.js +13 -50
- package/dist/neal/prompts/specialized.js +7 -17
- package/dist/neal/prompts/specs.js +180 -32
- package/dist/neal/providers/agent-settings-isolation.js +35 -0
- package/dist/neal/providers/anthropic-claude.js +18 -1
- package/dist/neal/providers/detection.js +3 -7
- package/dist/neal/providers/git-config-isolation.js +83 -0
- package/dist/neal/providers/openai-codex.js +25 -3
- package/dist/neal/providers/{generic-agentic-tools.js → openai-compatible-tools.js} +11 -5
- package/dist/neal/providers/openai-compatible.js +1078 -430
- package/dist/neal/providers/openrouter-routing.js +68 -0
- package/dist/neal/providers/pricing.js +1 -1
- package/dist/neal/providers/rate-card.js +6 -32
- package/dist/neal/providers/registry.js +6 -17
- package/dist/neal/providers/types.js +12 -0
- package/dist/neal/resume-decision.js +7 -6
- package/dist/neal/resume-planner.js +9 -0
- package/dist/neal/retrospective.js +3 -3
- package/dist/neal/review-debt.js +30 -0
- package/dist/neal/review-findings/provider.js +17 -29
- package/dist/neal/review.js +17 -1
- package/dist/neal/state-invariants.js +11 -3
- package/dist/neal/state-views.js +27 -5
- package/dist/neal/state.js +59 -15
- package/dist/neal/status.js +28 -1
- package/dist/neal/support.js +3 -3
- package/docs/README.md +22 -18
- package/docs/{ADJUDICATOR_INVENTORY.md → adjudicator-inventory.md} +58 -57
- package/docs/architecture.md +79 -83
- package/docs/automation.md +11 -11
- package/docs/compat.md +91 -61
- package/docs/compatible-models.md +145 -121
- package/docs/demo.md +15 -5
- package/docs/maintenance.md +65 -34
- package/docs/plan-format.md +61 -33
- package/docs/prompt-evals.md +96 -0
- package/docs/{PROMPT_SPECS.md → prompt-specs.md} +52 -48
- package/docs/providers.md +279 -349
- package/docs/release.md +63 -81
- package/docs/review-convergence.md +246 -0
- package/docs/state-machine.md +61 -59
- package/docs/storage.md +40 -31
- package/docs/troubleshooting.md +32 -29
- package/examples/compat/add-edit-verify/PLAN.md +2 -0
- package/examples/compat/add-edit-verify/broken.diff +2 -2
- package/examples/compat/add-edit-verify/good.diff +2 -2
- package/examples/compat/add-edit-verify/src/add.js +1 -1
- package/examples/compat/manifest.json +25 -29
- package/examples/compat/sum-grep-edit/PLAN.md +33 -0
- package/examples/compat/sum-grep-edit/broken.diff +12 -0
- package/examples/compat/sum-grep-edit/good.diff +12 -0
- package/examples/compat/sum-grep-edit/package.json +5 -0
- package/examples/compat/sum-grep-edit/src/numbers.js +9 -0
- package/examples/compat/sum-grep-edit/test/numbers.test.js +15 -0
- package/examples/issue-triage-js/README.md +0 -15
- package/examples/reviewer-eval/clean-doc/base/src/clamp.ts +3 -0
- package/examples/reviewer-eval/clean-doc/change.diff +9 -0
- package/examples/reviewer-eval/clean-extract/base/src/greet.ts +3 -0
- package/examples/reviewer-eval/clean-extract/change.diff +13 -0
- package/examples/reviewer-eval/dropped-error/base/src/load.ts +4 -0
- package/examples/reviewer-eval/dropped-error/change.diff +15 -0
- package/examples/reviewer-eval/manifest.json +45 -0
- package/examples/reviewer-eval/off-by-one/base/src/paginate.ts +5 -0
- package/examples/reviewer-eval/off-by-one/change.diff +11 -0
- package/{config.yml → neal.yml} +1 -1
- package/package.json +15 -14
- package/dist/neal/providers/generic-agentic.js +0 -1256
- package/docs/comparison.md +0 -105
- package/docs/issue-pipeline.md +0 -124
- package/examples/compat/is-even-add-test/PLAN.md +0 -30
- package/examples/compat/is-even-add-test/broken.diff +0 -11
- package/examples/compat/is-even-add-test/good.diff +0 -11
- package/examples/compat/is-even-add-test/package.json +0 -5
- package/examples/compat/is-even-add-test/src/is-even.js +0 -3
- package/examples/compat/is-even-add-test/test/is-even.test.js +0 -9
- package/examples/compat/reverse-grep-edit/PLAN.md +0 -32
- package/examples/compat/reverse-grep-edit/broken.diff +0 -12
- package/examples/compat/reverse-grep-edit/good.diff +0 -12
- package/examples/compat/reverse-grep-edit/package.json +0 -5
- package/examples/compat/reverse-grep-edit/src/strings.js +0 -10
- package/examples/compat/reverse-grep-edit/test/strings.test.js +0 -15
package/docs/plan-format.md
CHANGED
|
@@ -1,18 +1,19 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Plan format
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
The validator lives in `src/neal/plan-validation.ts` as
|
|
3
|
+
Executable neal plan documents follow the format defined here, the canonical
|
|
4
|
+
public reference. The validator lives in `src/neal/plan-validation.ts` as
|
|
5
|
+
`validatePlanDocument`.
|
|
5
6
|
The selected-plan Git contract lives in `src/neal/plan-doc.ts` as
|
|
6
7
|
`inspectPlanDocDisposition` and `toPlanDocMetadata`. Scope counting and status
|
|
7
8
|
labels come from `getExecutionPlanScopeCount` and
|
|
8
9
|
`getCurrentExecutionScopeDescriptor` in `src/neal/scopes.ts`.
|
|
9
10
|
|
|
10
|
-
Executable plans are instructions for
|
|
11
|
+
Executable plans are instructions for neal's planner/coder/reviewer loop, not
|
|
11
12
|
project requirements documents. They should be specific enough that one scope
|
|
12
13
|
can be implemented, reviewed, and verified without relying on hidden operator
|
|
13
14
|
intent.
|
|
14
15
|
|
|
15
|
-
## Execution
|
|
16
|
+
## Execution shape
|
|
16
17
|
|
|
17
18
|
Every executable plan must choose exactly one execution shape. The plan must
|
|
18
19
|
include a literal `## Execution Shape` section, and that section must contain
|
|
@@ -24,11 +25,11 @@ exactly one non-empty line:
|
|
|
24
25
|
|
|
25
26
|
Use `one_shot` when the whole task fits in one bounded implementation scope.
|
|
26
27
|
Use `multi_scope` when the task has a finite ordered queue of known scopes.
|
|
27
|
-
Use `multi_scope_unknown` when
|
|
28
|
+
Use `multi_scope_unknown` when neal should repeat one bounded scope template
|
|
28
29
|
until a concrete completion rule is satisfied, but the number of iterations is
|
|
29
30
|
not knowable when the plan is written.
|
|
30
31
|
|
|
31
|
-
## One-
|
|
32
|
+
## One-shot format
|
|
32
33
|
|
|
33
34
|
`executionShape: one_shot` must not include a literal `## Execution Queue`
|
|
34
35
|
section, a literal `## Execution Loop` section, or a standalone
|
|
@@ -48,11 +49,11 @@ executionShape: one_shot
|
|
|
48
49
|
Complete one bounded change and verify it.
|
|
49
50
|
```
|
|
50
51
|
|
|
51
|
-
For status display,
|
|
52
|
+
For status display, neal treats a one-shot plan as one known scope. If the
|
|
52
53
|
document has a level-one Markdown title, that title can be used as the scope
|
|
53
|
-
display title
|
|
54
|
+
display title. Otherwise neal falls back to a generic one-scope label.
|
|
54
55
|
|
|
55
|
-
### `one_shot`
|
|
56
|
+
### `one_shot` is defended through plan refinement, not clamped
|
|
56
57
|
|
|
57
58
|
An author-declared `executionShape: one_shot` is captured once from the seed
|
|
58
59
|
plan document and defended through refinement by both roles' prompts: the
|
|
@@ -62,18 +63,18 @@ refined document declares any other execution shape or adds orchestration
|
|
|
62
63
|
sections. That finding routes through the normal revision loop like any other.
|
|
63
64
|
|
|
64
65
|
There is deliberately no hard mechanical clamp: if the review loop converges on
|
|
65
|
-
a different shape (the reviewer accepts an expansion),
|
|
66
|
+
a different shape (the reviewer accepts an expansion), neal adopts the refined
|
|
66
67
|
document's shape. An earlier version clamped the saved shape back to `one_shot`
|
|
67
|
-
unconditionally, which caused non-convergence on complex plans
|
|
68
|
+
unconditionally, which caused non-convergence on complex plans: the planner
|
|
68
69
|
could not produce an accurate single-scope plan, so review correctly rejected
|
|
69
70
|
it until the round cap failed the run.
|
|
70
71
|
|
|
71
72
|
This defense applies only to the top-level authored plan. A derived plan
|
|
72
73
|
declares and owns its own execution shape. Plans authored `multi_scope` or
|
|
73
|
-
`multi_scope_unknown` are unaffected
|
|
74
|
+
`multi_scope_unknown` are unaffected. Refinement may adjust their scope
|
|
74
75
|
content as usual.
|
|
75
76
|
|
|
76
|
-
## Multi-
|
|
77
|
+
## Multi-scope format
|
|
77
78
|
|
|
78
79
|
`executionShape: multi_scope` must include a literal `## Execution Queue`
|
|
79
80
|
section. It must not include a literal `## Execution Loop` section or a
|
|
@@ -108,14 +109,14 @@ Each scope entry must include these labeled bullets:
|
|
|
108
109
|
- `- Verification:`
|
|
109
110
|
- `- Success Condition:`
|
|
110
111
|
|
|
111
|
-
The queue cannot skip or repeat numbers.
|
|
112
|
+
The queue cannot skip or repeat numbers. neal uses the queue headings to count
|
|
112
113
|
known scopes and to show progress labels such as the current scope number and
|
|
113
114
|
scope title.
|
|
114
115
|
|
|
115
116
|
For a concrete in-repo multi-scope plan, see
|
|
116
117
|
[../examples/issue-triage-js/PLAN.md](../examples/issue-triage-js/PLAN.md).
|
|
117
118
|
|
|
118
|
-
## Multi-
|
|
119
|
+
## Multi-scope-unknown format
|
|
119
120
|
|
|
120
121
|
`executionShape: multi_scope_unknown` must include a literal
|
|
121
122
|
`## Execution Loop` section with exactly one literal `### Recurring Scope`
|
|
@@ -144,58 +145,85 @@ executionShape: multi_scope_unknown
|
|
|
144
145
|
Stop when the explicit completion rule is satisfied.
|
|
145
146
|
```
|
|
146
147
|
|
|
147
|
-
The recurring scope uses the same required bullets as a fixed queue.
|
|
148
|
+
The recurring scope uses the same required bullets as a fixed queue. neal treats
|
|
148
149
|
the total scope count as unknown by contract, and status displays the recurring
|
|
149
150
|
scope title rather than a finite total.
|
|
150
151
|
|
|
151
|
-
## Verification
|
|
152
|
+
## Verification and acceptance
|
|
152
153
|
|
|
153
154
|
A scope should name verification commands that are deterministic and
|
|
154
155
|
noninteractive when possible. Examples include typecheck, test, lint, build,
|
|
155
156
|
package verification, or a focused smoke command. If a command needs external
|
|
156
157
|
state, credentials, network access, or manual setup, call that out in the plan
|
|
157
|
-
so
|
|
158
|
+
so neal can distinguish expected manual gates from unexpected blockers.
|
|
158
159
|
|
|
159
160
|
The success condition should state what must be true after the scope is
|
|
160
|
-
complete. It is not a prose summary of the goal
|
|
161
|
+
complete. It is not a prose summary of the goal. It is the reviewable exit
|
|
161
162
|
criterion. Good success conditions mention the changed surface, the expected
|
|
162
163
|
behavior or docs state, and the verification evidence required for acceptance.
|
|
163
164
|
|
|
164
|
-
## Planning
|
|
165
|
+
## Planning normalization
|
|
165
166
|
|
|
166
167
|
`neal plan` revises the selected plan file in place. It should preserve the
|
|
167
|
-
user's product objective while making the document executable by
|
|
168
|
-
|
|
169
|
-
|
|
168
|
+
user's product objective while making the document executable by neal and
|
|
169
|
+
keeping it practical for a person to review.
|
|
170
|
+
|
|
171
|
+
Plan mode owns the final execution-shape decision. Its planner/reviewer loop
|
|
172
|
+
chooses `one_shot`, `multi_scope`, or `multi_scope_unknown` and records that
|
|
173
|
+
choice in the plan's `## Execution Shape` section. An author-declared
|
|
174
|
+
`one_shot` remains subject to the defense described above.
|
|
175
|
+
|
|
176
|
+
The planner inspects the current repository enough to confirm the approach and
|
|
177
|
+
identify major dependencies, constraints, and affected subsystems. It adds
|
|
178
|
+
moderate-to-high-level implementation detail, scope boundaries, meaningful
|
|
179
|
+
verification, and success conditions. It should not try to complete the
|
|
180
|
+
implementation in prose. Routine file discovery, exact tests, and local code
|
|
181
|
+
choices belong to the execution coder and reviewer.
|
|
182
|
+
|
|
183
|
+
The plan reviewer checks the approach, major repository constraints, scope
|
|
184
|
+
shape, sequencing, verification, and completion conditions. A material omission
|
|
185
|
+
that could produce the wrong implementation or make a scope unsafe is a plan
|
|
186
|
+
finding. Missing routine implementation detail is not. The refined plan does
|
|
187
|
+
not need exhaustive lists of files, symbols, callers, tests, commands, or
|
|
188
|
+
assertions.
|
|
189
|
+
|
|
190
|
+
A repository-wide invariant or global regression guarantee belongs in the plan
|
|
191
|
+
only when it is necessary for the requested change to be correct. Otherwise,
|
|
192
|
+
the planner narrows or removes it instead of expanding the implementation or
|
|
193
|
+
verification scope. The reviewer judges verification against the requested
|
|
194
|
+
change rather than requiring complete enforcement of a broader guarantee the
|
|
195
|
+
plan introduced itself.
|
|
196
|
+
|
|
197
|
+
During validation, neal may normalize known legacy plan wording before checking
|
|
170
198
|
the final shape. Current normalization can convert legacy queue section
|
|
171
199
|
headings to `## Execution Queue`, normalize compatible scope labels to literal
|
|
172
200
|
`### Scope N:` headings, and normalize known aliases for the verification and
|
|
173
201
|
success-condition bullet labels. Normalization is still bounded by the same
|
|
174
|
-
shape rules above
|
|
202
|
+
shape rules above. It is not a license to omit the execution shape or required
|
|
175
203
|
scope fields.
|
|
176
204
|
|
|
177
205
|
For `neal plan`, the original selected plan backup is stored under the run
|
|
178
|
-
directory at `.neal/runs/<run-id>/PLAN_ORIGINAL.md`.
|
|
206
|
+
directory at `.neal/runs/<run-id>/PLAN_ORIGINAL.md`. neal does not use a sibling
|
|
179
207
|
repository backup directory for that copy.
|
|
180
208
|
|
|
181
|
-
## Selected
|
|
209
|
+
## Selected plan documents and Git
|
|
182
210
|
|
|
183
|
-
|
|
211
|
+
neal records selected-plan metadata with `inspectPlanDocDisposition` and
|
|
184
212
|
`toPlanDocMetadata` from `src/neal/plan-doc.ts`.
|
|
185
213
|
|
|
186
214
|
A selected plan document is eligible for ordinary Git inclusion only when it is
|
|
187
215
|
repo-local, exists, is a regular file, and is not ignored. If that eligible
|
|
188
|
-
plan document changes during a run,
|
|
216
|
+
plan document changes during a run, neal may include it in the final tree using
|
|
189
217
|
normal Git staging.
|
|
190
218
|
|
|
191
219
|
The selected plan document can be the only allowed dirty path at writer-run
|
|
192
220
|
start when it is the explicit plan under execution. Dirty work outside that
|
|
193
221
|
selected plan still blocks writer-run start and queue continuation.
|
|
194
222
|
|
|
195
|
-
## Ignored
|
|
223
|
+
## Ignored or external plan documents
|
|
196
224
|
|
|
197
225
|
Ignored plan files, plan files outside the repository, missing paths, and
|
|
198
|
-
non-file paths are metadata-only.
|
|
226
|
+
non-file paths are metadata-only. neal records where they came from, but it does
|
|
199
227
|
not force-add them to the repository and does not treat them as package or
|
|
200
228
|
source artifacts.
|
|
201
229
|
|
|
@@ -204,9 +232,9 @@ stored outside the target repository. Those plans can drive a run, but they
|
|
|
204
232
|
remain operator-local unless the operator intentionally copies their contents
|
|
205
233
|
into a tracked document.
|
|
206
234
|
|
|
207
|
-
## Protocol
|
|
235
|
+
## Protocol boundary
|
|
208
236
|
|
|
209
|
-
|
|
237
|
+
neal's terminal control protocol is not Markdown artifact content. Plan files,
|
|
210
238
|
derived plans, review notes, recovery guidance, and JSON artifacts should use
|
|
211
239
|
their normal schema or prose fields instead of embedding terminal control marker
|
|
212
240
|
words. This keeps executable plans portable and prevents transport signals from
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# Prompt evals
|
|
2
|
+
|
|
3
|
+
Prompt changes ship with two guards: a versioning contract that makes silent
|
|
4
|
+
drift impossible, and a reviewer-recall eval that measures what a reviewer
|
|
5
|
+
prompt actually does. Together they gate behavior-changing edits to the review
|
|
6
|
+
doctrine.
|
|
7
|
+
|
|
8
|
+
## Prompt versioning
|
|
9
|
+
|
|
10
|
+
Every registered `PromptSpec` carries an integer `version` and an in-source
|
|
11
|
+
changelog, and each render fixture records the version and a `renderSha` of the
|
|
12
|
+
prompt it pinned. The render-integrity contract test fails when a spec's
|
|
13
|
+
rendered output changes without a version bump. So "which prompt version
|
|
14
|
+
produced this run" is always answerable, and a prompt edit that forgets to bump
|
|
15
|
+
its version fails CI instead of drifting silently.
|
|
16
|
+
|
|
17
|
+
When you change a prompt: edit it, bump its `version`, add a changelog line, and
|
|
18
|
+
regenerate its render fixture. The test tells you exactly which spec is out of
|
|
19
|
+
sync.
|
|
20
|
+
|
|
21
|
+
## Reviewer-recall eval
|
|
22
|
+
|
|
23
|
+
The eval measures a reviewer prompt against a labeled fixture set (diffs with
|
|
24
|
+
known defects, plus clean diffs) and reports recall, precision, and blocking-
|
|
25
|
+
finding rate. It exists so the reviewer-doctrine rewrite can be gated on
|
|
26
|
+
numbers instead of intuition: run it before the change, run it after, compare.
|
|
27
|
+
|
|
28
|
+
### Running it
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
pnpm build
|
|
32
|
+
node scripts/eval-reviewer.mjs
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
It runs the reviewer configured in `~/.neal/config.yml` (or a repo `neal.yml`)
|
|
36
|
+
against every fixture, so it makes real provider calls and bills whatever that
|
|
37
|
+
reviewer bills. Run it from a checkout with an authenticated Codex/Claude CLI
|
|
38
|
+
to use a subscription, the same as `scripts/qualify-sdk.sh`. It prints a table
|
|
39
|
+
and writes `EVAL_REPORT.json` (gitignored).
|
|
40
|
+
|
|
41
|
+
### What the numbers mean
|
|
42
|
+
|
|
43
|
+
- **Recall**: labeled defects the reviewer flagged / all labeled defects. The
|
|
44
|
+
headline number for a doctrine change: a rewrite that raises precision by
|
|
45
|
+
suppressing findings will show up here as a recall drop.
|
|
46
|
+
- **Precision**: blocking findings that hit a labeled file / all blocking
|
|
47
|
+
findings.
|
|
48
|
+
- **Clean false-positive rate**: clean fixtures the reviewer flagged with a
|
|
49
|
+
blocking finding / clean fixtures.
|
|
50
|
+
- **Blocking rate**: mean blocking findings per fixture, a proxy for how many
|
|
51
|
+
coder rounds the reviewer would force.
|
|
52
|
+
|
|
53
|
+
Matching is coarse on purpose: a finding counts for a label when they name the
|
|
54
|
+
same **file**. It measures "did the reviewer flag the right file", not "did it
|
|
55
|
+
describe the exact defect". Do not over-read the absolute numbers. Read the
|
|
56
|
+
**delta** between two prompt versions on the same fixtures.
|
|
57
|
+
|
|
58
|
+
### Adding fixtures
|
|
59
|
+
|
|
60
|
+
Fixtures live in `examples/reviewer-eval/`. Each is a directory with a `base/`
|
|
61
|
+
tree (the before state) and a `change.diff`, plus a manifest entry:
|
|
62
|
+
|
|
63
|
+
```json
|
|
64
|
+
{
|
|
65
|
+
"id": "off-by-one",
|
|
66
|
+
"kind": "defective",
|
|
67
|
+
"diff": "off-by-one/change.diff",
|
|
68
|
+
"baseDir": "off-by-one/base",
|
|
69
|
+
"expectedFindings": [
|
|
70
|
+
{ "file": "src/paginate.ts", "defectClass": "off-by-one", "description": "..." }
|
|
71
|
+
]
|
|
72
|
+
}
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
A defective fixture labels at least one file the diff touches. A clean fixture
|
|
76
|
+
labels none. The loader (`src/neal/eval/reviewer-eval-manifest.ts`) enforces
|
|
77
|
+
that structurally: it checks a label points at a file the diff modifies, but
|
|
78
|
+
it does **not** try to prove the diff exhibits the defect. That proof was an
|
|
79
|
+
over-specification. A human authoring the fixture asserts the defect. Keep the
|
|
80
|
+
starter set growing toward ~10–15 fixtures across defect classes.
|
|
81
|
+
|
|
82
|
+
## Process rule
|
|
83
|
+
|
|
84
|
+
A pull request that changes reviewer-facing implementation-review prompt text must include:
|
|
85
|
+
|
|
86
|
+
1. a `version` bump and changelog line on the affected spec (enforced), and
|
|
87
|
+
2. before/after eval numbers in the PR description.
|
|
88
|
+
|
|
89
|
+
The reviewer-recall fixtures exercise implementation review, not plan review.
|
|
90
|
+
Evaluate `plan_reviewer` prompt changes with representative `neal plan` runs and
|
|
91
|
+
compare objective preservation, scope quality, readability, and review-round
|
|
92
|
+
count. Do not report reviewer-recall numbers for a plan-review-only change as if
|
|
93
|
+
they measured the changed prompt.
|
|
94
|
+
|
|
95
|
+
The eval is not run in CI (it costs live tokens). The versioning contract is
|
|
96
|
+
the CI-side guard, and the eval numbers are a human review requirement.
|
|
@@ -1,16 +1,16 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Prompt specs inventory
|
|
2
2
|
|
|
3
3
|
## Scope
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
5
|
+
The current prompt-spec inventory for neal's recurring engineering
|
|
6
|
+
roles/tasks.
|
|
7
7
|
|
|
8
8
|
- The prompt-spec contract and registry live in
|
|
9
9
|
[src/neal/prompts/specs.ts](../src/neal/prompts/specs.ts).
|
|
10
10
|
- Concrete prompt builders live in `src/neal/prompts/planning.ts`,
|
|
11
11
|
`src/neal/prompts/execute.ts`, and `src/neal/prompts/specialized.ts`, with
|
|
12
12
|
shared plan-contract and marker lines in `src/neal/prompts/shared.ts`.
|
|
13
|
-
- The interactive blocked-recovery and
|
|
13
|
+
- The interactive blocked-recovery and consultant prompt builders live
|
|
14
14
|
in `src/neal/agents/prompts.ts` (which otherwise re-exports the
|
|
15
15
|
`src/neal/prompts/` builders).
|
|
16
16
|
- Schema builders, object validators, and retained compatibility parsers live
|
|
@@ -21,14 +21,14 @@ Reviewer prompt surfaces share a doctrine helper in
|
|
|
21
21
|
[src/neal/prompts/review-doctrine.ts](../src/neal/prompts/review-doctrine.ts).
|
|
22
22
|
That helper owns adversarial posture, falsification, verification skepticism,
|
|
23
23
|
regression-preservation, the pre-existing-failure acceptance-surface contract,
|
|
24
|
-
finding-quality wording, and the
|
|
25
|
-
(`tool-access` / `read-only`
|
|
24
|
+
finding-quality wording, and the two-way reviewer access mode
|
|
25
|
+
(`tool-access` / `read-only`). It does not own schemas, provider
|
|
26
26
|
selection, adjudication, or transition behavior. Its consumers are
|
|
27
|
-
execute-scope review, plan review, and final completion review
|
|
27
|
+
execute-scope review, plan review, and final completion review. The read-only
|
|
28
28
|
`neal review` findings loop uses its own prompts in
|
|
29
29
|
`src/neal/review-findings/prompts.ts`.
|
|
30
30
|
|
|
31
|
-
## Ownership
|
|
31
|
+
## Ownership boundary
|
|
32
32
|
|
|
33
33
|
Prompt specs live under `src/neal/prompts/`.
|
|
34
34
|
|
|
@@ -45,32 +45,32 @@ Concrete split:
|
|
|
45
45
|
|
|
46
46
|
## Inventory
|
|
47
47
|
|
|
48
|
-
### Registered
|
|
48
|
+
### Registered prompt specs
|
|
49
49
|
|
|
50
50
|
All schema targets are `structured_json` with provider surface
|
|
51
|
-
`neal_json_block_protocol
|
|
51
|
+
`neal_json_block_protocol`, the prompt-spec metadata name for the runtime
|
|
52
52
|
`neal-json-block-v1` JSON-block transport that providers validate locally.
|
|
53
53
|
|
|
54
54
|
| Prompt spec id | Current builder(s) | Current round entrypoints | Schema target | Notes |
|
|
55
55
|
| --- | --- | --- | --- | --- |
|
|
56
|
-
| `plan_author` | `buildPlanningPrompt`, `buildCoderPlanResponsePrompt` (`reviewMode=plan`, `reviewMode=derived-plan`) | `runCoderPlanRound`, `runCoderPlanResponseRound` | Primary planning: `buildCoderPlanSchema` / `validateCoderPlanPayload
|
|
57
|
-
| `plan_reviewer` | `buildPlanReviewerPrompt` (`mode=plan`, `mode=derived-plan`) | `runPlanReviewerRound` | `buildPlanReviewerSchema` / `PlanReviewerPayload` | Execution-shape confirmation is part of the contract.
|
|
58
|
-
| `scope_coder` | `buildScopePrompt`, `buildCoderResponsePrompt` | `runCoderScopeRound`, `runCoderResponseRound` | Primary execution: `buildCoderScopeSchema` / `validateCoderScopePayload
|
|
59
|
-
| `scope_reviewer` | `buildReviewerPrompt` | `runReviewerRound` | `buildReviewerSchema` / `ReviewerPayload` | Execute-scope review only
|
|
60
|
-
| `completion_coder` | `buildFinalCompletionSummaryPrompt` | `runCoderFinalCompletionSummaryRound` | `buildFinalCompletionSummarySchema` / `parseFinalCompletionSummaryPayload` | Structured advisor round, but still a coder-owned role/task. The completion packet includes aggregate review context when
|
|
61
|
-
| `completion_reviewer` | `buildFinalCompletionReviewerPrompt` | `runReviewerFinalCompletionRound` | `buildFinalCompletionReviewerSchema` / `parseFinalCompletionReviewerPayload` | Whole-plan aggregate review remains distinct from ordinary scope review and keeps its final-completion verdict schema.
|
|
56
|
+
| `plan_author` | `buildPlanningPrompt`, `buildCoderPlanResponsePrompt` (`reviewMode=plan`, `reviewMode=derived-plan`) | `runCoderPlanRound`, `runCoderPlanResponseRound` | Primary planning: `buildCoderPlanSchema` / `validateCoderPlanPayload`. Response rounds: `buildCoderPlanResponseSchema` / `validateCoderPlanResponsePayload`. | Primary planning routes new/resumed structured sessions by persisted `plannerSessionProtocol`. Legacy marker parsing is retained only for active `legacy_marker_v1` sessions. |
|
|
57
|
+
| `plan_reviewer` | `buildPlanReviewerPrompt` (`mode=plan`, `mode=derived-plan`) | `runPlanReviewerRound` | `buildPlanReviewerSchema` / `PlanReviewerPayload` | Execution-shape confirmation is part of the contract. Reviews material approach, scope, sequencing, and verification defects without turning the plan into an implementation inventory. |
|
|
58
|
+
| `scope_coder` | `buildScopePrompt`, `buildCoderResponsePrompt` | `runCoderScopeRound`, `runCoderResponseRound` | Primary execution: `buildCoderScopeSchema` / `validateCoderScopePayload`. Response rounds: `buildCoderResponseSchema` / `validateCoderResponsePayload`. | Primary execution routes new/resumed structured sessions by persisted `coderSessionProtocol`. Legacy marker and progress-payload parsing is retained only for active `legacy_marker_v1` sessions. Also carries an `adjacent`-status blocked-recovery `response` variant (see below). |
|
|
59
|
+
| `scope_reviewer` | `buildReviewerPrompt` | `runReviewerRound` | `buildReviewerSchema` / `ReviewerPayload` | Execute-scope review only. `neal review` external ranges use the separate read-only review-findings loop. Meaningful-progress remains a capability variant of `scope_reviewer`, not a new top-level id. Context includes a run-local `scratchDir`, but read-only reviewer prompts omit it. |
|
|
60
|
+
| `completion_coder` | `buildFinalCompletionSummaryPrompt` | `runCoderFinalCompletionSummaryRound` | `buildFinalCompletionSummarySchema` / `parseFinalCompletionSummaryPayload` | Structured advisor round, but still a coder-owned role/task. The completion packet includes aggregate review context when neal can compute it. |
|
|
61
|
+
| `completion_reviewer` | `buildFinalCompletionReviewerPrompt` | `runReviewerFinalCompletionRound` | `buildFinalCompletionReviewerSchema` / `parseFinalCompletionReviewerPayload` | Whole-plan aggregate review remains distinct from ordinary scope review and keeps its final-completion verdict schema. Context includes a run-local `scratchDir`, but read-only reviewer prompts omit it. |
|
|
62
|
+
| `consultant` | `buildConsultantPrompt` | `runConsultantRound` | `buildConsultantSchema` / `validateConsultantVerdictPayload` | Single no-read-safe variant for the read-only consultant. It judges entirely from neal-inlined context and its static instructions pass the shared no-read guard. |
|
|
62
63
|
|
|
63
|
-
### Adjacent
|
|
64
|
+
### Adjacent current prompt surfaces
|
|
64
65
|
|
|
65
66
|
These prompts are real but are not separate top-level prompt-spec ids:
|
|
66
67
|
|
|
67
68
|
| Current builder | Registry status | Schema target | Why not a top-level id |
|
|
68
69
|
| --- | --- | --- | --- |
|
|
69
70
|
| `buildBlockedRecoveryCoderPrompt` (`src/neal/agents/prompts.ts`) | Registered as a `scope_coder` `response` variant with status `adjacent` | `buildCoderBlockedRecoveryDispositionSchema` / `validateCoderBlockedRecoveryDispositionPayload` | It is still the same execute-scope owner responding inside blocked recovery. |
|
|
70
|
-
| `buildBlockedAdjudicatorPrompt` (`src/neal/agents/prompts.ts`) | Not in the prompt-spec registry today; a future change may register it | `buildBlockedAdjudicatorSchema` / `validateBlockedAdjudicatorVerdictPayload` via `runBlockedAdjudicatorRound` | Single no-read-safe variant for the read-only blocked adjudicator; it judges entirely from Neal-inlined context and its static instructions pass the shared no-read guard. |
|
|
71
71
|
| `buildReviewFindingsDraftPrompt`, `buildReviewFindingsReviewPrompt` (`src/neal/review-findings/prompts.ts`) | Not in the prompt-spec registry | `ReviewFindingsDraft` / `ReviewFindingsReview` payloads | Read-only `neal review` findings loop with its own draft/review prompts and read-only rules. |
|
|
72
72
|
|
|
73
|
-
## Contract
|
|
73
|
+
## Contract expectations
|
|
74
74
|
|
|
75
75
|
Each prompt spec in `src/neal/prompts/specs.ts` makes these fields explicit:
|
|
76
76
|
|
|
@@ -95,9 +95,9 @@ Three implementation details are intentionally concrete:
|
|
|
95
95
|
|
|
96
96
|
That keeps prompt specs reviewable as contracts rather than as scattered string literals.
|
|
97
97
|
|
|
98
|
-
## Prompt-
|
|
98
|
+
## Prompt-spec wiring
|
|
99
99
|
|
|
100
|
-
Prompt specs are not the whole execution loop. They are the prompt-facing contract layer that tells
|
|
100
|
+
Prompt specs are not the whole execution loop. They are the prompt-facing contract layer that tells neal which role/task is being performed, what context that role/task assumes, and which output contract the result must satisfy.
|
|
101
101
|
|
|
102
102
|
That wiring is intentionally split across a few modules:
|
|
103
103
|
|
|
@@ -106,14 +106,14 @@ That wiring is intentionally split across a few modules:
|
|
|
106
106
|
- `src/neal/prompts/*.ts` owns the concrete prompt builders that render instructions for planning, execute-mode, and specialized flows.
|
|
107
107
|
- `src/neal/agents/schemas.ts` owns the actual schema builders plus validators or retained parsers named by each spec's `schemaTarget`.
|
|
108
108
|
- `src/neal/agents/rounds.ts` owns round execution and parsing against those schemas.
|
|
109
|
-
- `src/neal/adjudicator/*.ts` owns the adjudication specs that reference prompt surfaces by `(promptSpecId, variantKind, exportName)` (see [
|
|
109
|
+
- `src/neal/adjudicator/*.ts` owns the adjudication specs that reference prompt surfaces by `(promptSpecId, variantKind, exportName)` (see [adjudicator-inventory.md](adjudicator-inventory.md)).
|
|
110
110
|
- `src/neal/orchestrator.ts` and `src/neal/orchestrator/*.ts` own phase transitions, adoption/finalization semantics, blocked-recovery routing, and commit consequences.
|
|
111
111
|
|
|
112
112
|
That split is deliberate. A prompt spec is incomplete without explicit schema linkage, but it also must not absorb sequencing or state-transition semantics that belong to the orchestrator.
|
|
113
113
|
|
|
114
|
-
### Context
|
|
114
|
+
### Context assembly rules
|
|
115
115
|
|
|
116
|
-
`requiredContext` should be read as a contract for context assembly, not
|
|
116
|
+
`requiredContext` should be read as a contract for context assembly, not as documentation for a prompt author.
|
|
117
117
|
|
|
118
118
|
When adding or changing a prompt spec:
|
|
119
119
|
|
|
@@ -121,37 +121,41 @@ When adding or changing a prompt spec:
|
|
|
121
121
|
2. The corresponding prompt builder in `src/neal/prompts/*.ts` should either accept that data directly or assemble it from a narrowly-scoped helper. Do not hide major context dependencies inside unrelated utilities.
|
|
122
122
|
3. If a prompt needs new state, artifact, or repository-derived context, add that dependency at the owning layer first and then link it from the spec. Do not document impossible context.
|
|
123
123
|
4. If a field is only used in a variant, keep that distinction explicit in the variant contract instead of pretending it is universally required.
|
|
124
|
-
5. Variant `inputShape` keys must stay a subset of the spec's top-level `requiredContext` keys.
|
|
124
|
+
5. Variant `inputShape` keys must stay a subset of the spec's top-level `requiredContext` keys. neal validates that contract at module load so prompt-spec drift fails fast in tests and at startup.
|
|
125
125
|
|
|
126
|
-
Final completion has one additional context assembly rule: `buildFinalCompletionPacket()` includes `aggregateReviewContext` for the whole implementation range from `initialBaseCommit` to the resolved final commit. When the range can be read, the packet carries commit subjects, diff stat, and changed files
|
|
126
|
+
Final completion has one additional context assembly rule: `buildFinalCompletionPacket()` includes `aggregateReviewContext` for the whole implementation range from `initialBaseCommit` to the resolved final commit. When the range can be read, the packet carries commit subjects, diff stat, and changed files. When it cannot, it carries an explicit `unavailableReason` so the reviewer treats the missing aggregate range as evidence to consider instead of silently accepting completion.
|
|
127
127
|
|
|
128
|
-
Execute-scope and final-completion reviewer
|
|
128
|
+
Execute-scope and final-completion reviewer context includes a deterministic
|
|
129
|
+
run-local `scratchDir` under `.neal/runs/<run-id>/scratch/`. A `tool-access`
|
|
130
|
+
prompt tells the reviewer to use that directory for temporary verification
|
|
131
|
+
artifacts. A `read-only` prompt omits the directory and forbids scratch work.
|
|
132
|
+
Every built-in reviewer currently uses `read-only` mode.
|
|
129
133
|
|
|
130
134
|
The goal is for reviewers to be able to answer two questions quickly:
|
|
131
135
|
|
|
132
136
|
- "What does this prompt assume is available?"
|
|
133
|
-
- "Where does
|
|
137
|
+
- "Where does neal actually get that data?"
|
|
134
138
|
|
|
135
|
-
### Schema-
|
|
139
|
+
### Schema-linkage rules
|
|
136
140
|
|
|
137
141
|
`schemaTarget` exists so prompt changes remain coupled to the concrete validator or retained parser surface they must satisfy.
|
|
138
142
|
|
|
139
143
|
The `PromptSchemaTarget` type admits two kinds:
|
|
140
144
|
|
|
141
145
|
- `structured_json` means the prompt must remain aligned with a named schema builder and validator or retained object parser in `src/neal/agents/schemas.ts`, delivered through the `neal-json-block-v1` transport.
|
|
142
|
-
- `terminal_marker` means a legacy prompt is governed by a plain-text final-line protocol. It remains in the type, but no current spec or variant declares it
|
|
146
|
+
- `terminal_marker` means a legacy prompt is governed by a plain-text final-line protocol. It remains in the type, but no current spec or variant declares it. Every current schema target is `structured_json`.
|
|
143
147
|
|
|
144
148
|
For coder-owned decisions, `structured_json` maps to
|
|
145
149
|
`CoderAdapter.runStructuredPrompt()` via the shared structured-coder helper.
|
|
146
150
|
`runPrompt(..., outputSchema)` is retained as raw provider compatibility
|
|
147
|
-
outside
|
|
148
|
-
|
|
151
|
+
outside neal product control paths. It is not the prompt-spec surface for
|
|
152
|
+
neal-owned coder decisions. The retained terminal-marker and progress-payload
|
|
149
153
|
parsers exist only for active `legacy_marker_v1` primary sessions loaded from
|
|
150
154
|
older run state.
|
|
151
155
|
|
|
152
156
|
If a prompt change would force validator or retained parser behavior to change, treat that as a contract change and review the prompt spec, prompt builder, schema builder, and tests together.
|
|
153
157
|
|
|
154
|
-
## Provider
|
|
158
|
+
## Provider variants
|
|
155
159
|
|
|
156
160
|
Provider-specific variants are allowed, but they are not the default escape hatch. Each spec declares `providerVariants` for `shared` (status `default`) plus `openai-codex` and `anthropic-claude` (status `reserved_for_justified_divergence`).
|
|
157
161
|
|
|
@@ -180,9 +184,9 @@ The `neal-json-block-v1` JSON-block transport is intentionally an adapter
|
|
|
180
184
|
concern, not a prompt-spec variant: the shared prompt builders own review
|
|
181
185
|
substance, schema linkage, and meaningful-progress semantics, while provider
|
|
182
186
|
adapters own the local control-block transport instructions so providers
|
|
183
|
-
return prose plus a final `neal-json` block that
|
|
187
|
+
return prose plus a final `neal-json` block that neal validates locally.
|
|
184
188
|
|
|
185
|
-
## Adjudicator
|
|
189
|
+
## Adjudicator alignment
|
|
186
190
|
|
|
187
191
|
Adjudication specs in `src/neal/adjudicator/specs.ts` reference prompt surfaces by `(promptSpecId, variantKind, exportName)`, and that linkage is validated at import time against the prompt-spec registry. The relationship is:
|
|
188
192
|
|
|
@@ -204,7 +208,7 @@ Three current role/task boundaries matter especially:
|
|
|
204
208
|
- `scope_reviewer` and `completion_reviewer` share adversarial doctrine, but they keep separate schemas and adjudication families
|
|
205
209
|
- `completion_reviewer` is plan-review-adjacent in the adjudicator family mapping, but its execute-mode completion transitions remain outside prompt ownership
|
|
206
210
|
|
|
207
|
-
## Adding
|
|
211
|
+
## Adding or extending prompt specs
|
|
208
212
|
|
|
209
213
|
Add prompt specs in a disciplined order:
|
|
210
214
|
|
|
@@ -219,7 +223,7 @@ Use a new top-level `PromptSpecId` only when the role/task is genuinely distinct
|
|
|
219
223
|
|
|
220
224
|
- it has its own durable artifact or output contract
|
|
221
225
|
- it has a materially different required-context packet
|
|
222
|
-
- it is expected to recur as an independently understandable
|
|
226
|
+
- it is expected to recur as an independently understandable neal surface
|
|
223
227
|
|
|
224
228
|
Otherwise prefer a variant such as `response`, `derived_plan`, `meaningful_progress`, or `final_completion`.
|
|
225
229
|
|
|
@@ -238,28 +242,28 @@ Current coverage:
|
|
|
238
242
|
- completion fixtures and render assertions pin aggregate range review, cross-scope invariant review, happy-path reachability, mocked-risk skepticism, and concrete completion-blocking issue quality
|
|
239
243
|
- inventory assertions in `test/review.test.ts` pin module ownership, schema targets, and current-home metadata
|
|
240
244
|
|
|
241
|
-
The blocked-recovery prompt is covered by the same additive guidance model as its owning coder role
|
|
245
|
+
The blocked-recovery prompt is covered by the same additive guidance model as its owning coder role. The consultant prompt takes no guidance injection (see below).
|
|
242
246
|
|
|
243
|
-
## User
|
|
247
|
+
## User guidance injection
|
|
244
248
|
|
|
245
|
-
Users can layer their own guidance onto
|
|
249
|
+
Users can layer their own guidance onto neal's built-in coder, reviewer, and planner prompts without forking the prompt source. The injection surface is deliberately additive: built-in sections still own structured coder envelopes, retained legacy terminal-marker compatibility, reviewer verdict JSON schemas, and the canonical plan contract.
|
|
246
250
|
|
|
247
251
|
Guidance files (all optional):
|
|
248
252
|
|
|
249
|
-
- `~/.neal/guidance/coder.md
|
|
250
|
-
- `~/.neal/guidance/reviewer.md
|
|
251
|
-
- `~/.neal/guidance/planner.md
|
|
253
|
+
- `~/.neal/guidance/coder.md`: injected into scope coder (including the retained legacy variant), scope response, interactive blocked recovery, and final completion summary prompts
|
|
254
|
+
- `~/.neal/guidance/reviewer.md`: injected into scope reviewer, plan reviewer, and final completion reviewer prompts
|
|
255
|
+
- `~/.neal/guidance/planner.md`: injected into plan author (including the retained legacy variant) and plan response prompts
|
|
252
256
|
|
|
253
|
-
The
|
|
257
|
+
The consultant prompt and the `neal review` findings prompts take no guidance injection.
|
|
254
258
|
|
|
255
|
-
By default
|
|
259
|
+
By default neal reads those files from `~/.neal/guidance/`. For each role, a
|
|
256
260
|
missing or whitespace-only `~/.neal/guidance/<role>.md` file is a no-op. Set
|
|
257
261
|
`NEAL_GUIDANCE_DIR` to point at another directory with the same file names for
|
|
258
|
-
testing or profile experiments
|
|
262
|
+
testing or profile experiments. That override wins over the default directory.
|
|
259
263
|
|
|
260
|
-
When present, the file contents are appended under a fixed `## User Guidance` section inside the built-in prompt
|
|
264
|
+
When present, the file contents are appended under a fixed `## User Guidance` section inside the built-in prompt. Structured output contracts, completion markers, and the canonical plan contract survive injection.
|
|
261
265
|
|
|
262
|
-
Diagnostics: when a
|
|
266
|
+
Diagnostics: when a neal writer run initializes or resumes, it logs which roles have guidance applied and the byte count to the run's `stderr.log` and as a `run.user_guidance_applied` / `run.user_guidance_scanned` event. That is enough to confirm a guidance file was picked up without dumping contents.
|
|
263
267
|
|
|
264
268
|
Non-goals: no repo-local `.neal/guidance/` override, no full-prompt replacement, no per-scope guidance variants, and no substitution of built-in sections.
|
|
265
269
|
|