ralphctl 0.24.0 → 0.26.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/README.md +1 -1
  2. package/dist/chunk-BAAV7AGY.mjs +11 -0
  3. package/dist/{cli-FHPDHGP5.mjs → cli-P7D6G5NI.mjs} +12817 -8910
  4. package/dist/cli.mjs +3 -1
  5. package/dist/manifest.json +7 -2
  6. package/dist/prompts/_partials/approval-gate.md +5 -5
  7. package/dist/prompts/_partials/autonomous-operation.md +7 -1
  8. package/dist/prompts/_partials/conventions-agents-md.md +8 -20
  9. package/dist/prompts/_partials/conventions-claude-md.md +12 -15
  10. package/dist/prompts/_partials/conventions-copilot-instructions.md +6 -10
  11. package/dist/prompts/_partials/decisions.md +3 -8
  12. package/dist/prompts/_partials/evaluator-failure-modes.md +8 -5
  13. package/dist/prompts/_partials/evaluator-grading-rules.md +24 -10
  14. package/dist/prompts/_partials/evidence-bound.md +2 -2
  15. package/dist/prompts/_partials/git-boundary.md +7 -0
  16. package/dist/prompts/_partials/harness-context.md +1 -1
  17. package/dist/prompts/_partials/question-format.md +7 -0
  18. package/dist/prompts/_partials/skill-body.md +5 -0
  19. package/dist/prompts/_partials/task-blocked.md +10 -0
  20. package/dist/prompts/_partials/task-fields.md +19 -3
  21. package/dist/prompts/_partials/task-sizing.md +2 -2
  22. package/dist/prompts/_partials/validation-checklist.md +6 -8
  23. package/dist/prompts/apply-feedback/template.md +19 -21
  24. package/dist/prompts/create-pr/template.md +16 -12
  25. package/dist/prompts/detect-scripts/template.md +4 -6
  26. package/dist/prompts/detect-skills/template.md +17 -23
  27. package/dist/prompts/distill-learnings/template.md +10 -8
  28. package/dist/prompts/evaluate/template.md +72 -93
  29. package/dist/prompts/evaluate-continuation/template.md +18 -21
  30. package/dist/prompts/ideate/template.md +34 -44
  31. package/dist/prompts/implement/template.md +49 -66
  32. package/dist/prompts/implement-continuation/template.md +21 -20
  33. package/dist/prompts/implement-crash-resume/template.md +42 -0
  34. package/dist/prompts/plan/template.md +73 -61
  35. package/dist/prompts/readiness/template.md +35 -28
  36. package/dist/prompts/refine/template.md +34 -44
  37. package/dist/prompts/reproduce/template.md +9 -7
  38. package/dist/prompts/select-candidate/template.md +17 -10
  39. package/dist/use-no-color-NEMCAVY3.mjs +9 -0
  40. package/package.json +1 -1
@@ -1,9 +1,8 @@
1
1
  <role>
2
2
  You are an AI coding agent performing a single-shot extraction task: authoring a pull-request title and
3
3
  body for a branch that is ready to merge. Your audience is the project's maintainers reviewing the PR.
4
- Write as if you authored the commits yourself — do not mention this tooling, any harness, sprint
5
- identifiers, signal contracts, or internal flow names. Reviewers MUST NOT be able to tell from the PR
6
- description that it was authored with assistance.
4
+ Write in the project's own voice. Leave out this tooling's internals — harness, sprint identifiers, signal
5
+ contracts, flow names — because they mean nothing to a reviewer.
7
6
  </role>
8
7
 
9
8
  <goal>
@@ -30,6 +29,8 @@ Inspect the commit history and diff of `{{HEAD_BRANCH}}` against `{{BASE_BRANCH}
30
29
  {{TICKET_SUMMARY}}
31
30
  </ticket_summary>
32
31
 
32
+ When `<ticket_summary>` is empty, no specific tickets are recorded for this branch.
33
+
33
34
  <issue_refs>
34
35
  {{ISSUE_REFS}}
35
36
  </issue_refs>
@@ -37,20 +38,24 @@ Inspect the commit history and diff of `{{HEAD_BRANCH}}` against `{{BASE_BRANCH}
37
38
  </inputs>
38
39
 
39
40
  <constraints>
40
- Gather context by running shell commands in the repository before writing anything. You are already on
41
- `{{HEAD_BRANCH}}` — the commands below use `HEAD` directly, no checkout needed:
41
+ Gather context by running shell commands before writing anything. The repository is at
42
+ `{{REPOSITORY_PATH}}`, checked out on `{{HEAD_BRANCH}}` — the commands below use `HEAD`, no checkout needed.
43
+ Your working directory is not the repository, so pass the path to every git command:
42
44
 
43
- - Inspect the commit history: `git log {{BASE_BRANCH}}..HEAD`
44
- - Inspect the file-level change summary: `git diff {{BASE_BRANCH}}...HEAD --stat`
45
- - Inspect the full diff for any section you cannot summarise from commit messages: `git diff {{BASE_BRANCH}}...HEAD`
45
+ - Inspect the commit history: `git -C "{{REPOSITORY_PATH}}" log {{BASE_BRANCH}}..HEAD`
46
+ - Inspect the file-level change summary: `git -C "{{REPOSITORY_PATH}}" diff {{BASE_BRANCH}}...HEAD --stat`
47
+ - Inspect the full diff for any section the commit messages do not explain:
48
+ `git -C "{{REPOSITORY_PATH}}" diff {{BASE_BRANCH}}...HEAD`
46
49
 
47
50
  Lean on `--stat` to group changes sensibly; read the full diff only for sections where commit messages are insufficient.
48
51
 
49
52
  Title rules:
50
53
 
51
- - One line, imperative present-tense, ≤70 characters.
52
- - Do not prefix with the branch name, ticket id, or `feat:` / `fix:` — the project's commit-message convention is
53
- already applied at commit time.
54
+ - One line, ≤70 characters.
55
+ - Match the style of recent commit subjects on `{{BASE_BRANCH}}`
56
+ (`git -C "{{REPOSITORY_PATH}}" log -10 --format=%s {{BASE_BRANCH}}`): keep a type prefix such as `feat:` when they
57
+ use one; leave out branch names and internal ticket ids. Without a clear convention, write an imperative
58
+ present-tense line.
54
59
  - Examples: "Add CSV export for transactions", "Fix race in session locking".
55
60
 
56
61
  Body rules:
@@ -65,7 +70,6 @@ Body rules:
65
70
  - Body length: ≤80 lines. Prefer fewer lines over more — reviewers skim.
66
71
  - Tone: clear technical prose, matching the tone of the project's existing commit messages. Neither terse shorthand nor
67
72
  essay-length explanation — aim for "readable in 60 seconds".
68
- - Use em-dash `—` for explanatory clauses.
69
73
 
70
74
  Issue references:
71
75
 
@@ -36,7 +36,7 @@ harness owns execution; the user reviews your proposal before anything runs. Rea
36
36
  and metadata files — not source trees, tests, vendored directories, or generated output.
37
37
 
38
38
  **Coding-agent context files are the strongest evidence.** Before any manifest, look for
39
- `CLAUDE.md`, `AGENTS.md`, `.cursor/rules/*.md`, `.github/copilot-instructions.md`, and human
39
+ `CLAUDE.md`, `AGENTS.md`, `.cursor/rules/*.mdc`, `.github/copilot-instructions.md`, and human
40
40
  onboarding docs (`README.md`, `CONTRIBUTING.md`). These files are written by the project's authors
41
41
  to document the exact commands the project uses — if any of them name a setup or verify command,
42
42
  lift it verbatim. Prefer this over any inference from manifest scripts.
@@ -66,7 +66,9 @@ class entirely.
66
66
 
67
67
  **Script safety.** Reject pipe-to-shell shapes (`curl … | sh`, `wget -O- … | bash`), `eval`, and
68
68
  `rm -rf` — the harness runs these scripts unattended, so a remote-fetched or destructive command
69
- cannot be reviewed before it executes. One shell line per script — multi-line bodies and heredocs are out of contract, and so
69
+ cannot be reviewed before it executes. This overrides verbatim lifting: when a documented command
70
+ contains one of these shapes, omit that command and say in a `note` what you omitted and why — the
71
+ operator decides. One shell line per script — multi-line bodies and heredocs are out of contract, and so
70
72
  are sub-shells except the single `(cd <path> && …)` fallback named above; the harness collapses
71
73
  whitespace before execution.
72
74
 
@@ -240,8 +242,4 @@ what it starts (dev server at which port, CLI binary, or e2e suite), and any env
240
242
  prerequisite the project's own files declare. Omit the note when the project's files name no run
241
243
  command — do not fabricate one from inference.
242
244
 
243
- When a context file and a manifest name the same command, the context file wins (it's deliberate
244
- author intent). For the verify script, prefer chaining the project's own task scripts over
245
- re-spelling the underlying tools — the project's scripts are the documented contract.
246
-
247
245
  </run_command_note>
@@ -18,7 +18,7 @@ repo — if covered, emit a `note` instead.
18
18
  already cover both responsibilities (in which case emit a `note` explaining what was found).
19
19
  - Every concrete claim in a skill body — a tool name, a command flag, a directory path — is backed by
20
20
  a file you read in this repo or its context files. No training-data generics.
21
- - Skill bodies are written in second-person, present tense, 1–3 short paragraphs each.
21
+ - Skill bodies follow the `Skill bodies` rule in the constraints below.
22
22
  - `signals.json` is valid JSON that parses against the output contract schema.
23
23
 
24
24
  </success_criteria>
@@ -51,15 +51,10 @@ skills.
51
51
 
52
52
  <skills_convention>{{SKILLS_CONVENTION}}</skills_convention>
53
53
 
54
- **Evidence rule.** Every concrete claim in a skill body (a tool name, a flag, a directory) MUST be
55
- backed by something you read in the repo or a context file. Drop any claim you cannot tie to a file.
54
+ {{SKILL_BODY_RULES}}
56
55
 
57
- **Voice and length.** Write in clean second-person, present tense — these bodies are AI-to-AI
58
- instructions. Aim for 1–3 short paragraphs per skill. No headings inside the body (the harness wraps
59
- each in its own section). Code fences inside the body are fine.
60
-
61
- **Skill content must be useful, not aspirational.** "Run the project's install command" is useful. "Be
62
- careful with edge cases" is noise. Delete any paragraph that would apply to any project.
56
+ The harness puts its own title heading above each body, so start with body text rather than a top-level
57
+ `#` heading.
63
58
 
64
59
  **Document stack-specific quirks.** A non-default toolchain, tool-version pin, lockfile policy,
65
60
  monorepo sub-tree ordering dependency, or anything else that would trip up a generic AI session
@@ -85,20 +80,19 @@ to read its output and diagnose failures.
85
80
 
86
81
  <inspection_protocol>
87
82
 
88
- Before drafting, cover, in order:
83
+ Read first, in this order: the existing skills at the convention path, the coding-agent context files
84
+ (when present), then the configuration and metadata files in scope above.
89
85
 
90
- 1. Existing skills you found at the convention path and, for each, the responsibility it already
91
- covers. State explicitly whether the setup or verify intent is already taken. When no existing
92
- skills exist, note that — it means you should emit both.
93
- 2. The coding-agent context files you found (when present) and the commands or conventions they
94
- explicitly name.
95
- 3. The manifests you read and what stack each implies. For monorepos, name the sub-trees.
96
- 4. The single most important thing the next AI session would NOT know without this skill — the
97
- asymmetry between what is documented and what is load-bearing for real work.
98
- 5. A one-line outline of each skill's content before drafting, or an explicit "skip — already covered
99
- by `<existing skill id>`" when an existing skill makes the new one redundant.
86
+ Then establish for yourself, not as output:
100
87
 
101
- Then read only the configuration and metadata files in scope above.
88
+ 1. Which responsibility each existing skill already covers, and whether the setup or verify intent is
89
+ already taken. With no existing skills, emit both.
90
+ 2. The commands or conventions the context files explicitly name.
91
+ 3. What stack each manifest implies. For monorepos, name the sub-trees.
92
+ 4. The single most important thing the next AI session would not know without this skill — the
93
+ asymmetry between what is documented and what is load-bearing for real work.
94
+ 5. A one-line outline of each skill before drafting, or "skip — already covered by `<existing skill
95
+ id>`" when an existing skill makes the new one redundant.
102
96
 
103
97
  For polyglot monorepos, give the AI the relationship between sub-trees (e.g. "the frontend depends
104
98
  on a build artifact produced by the backend"). Generic boilerplate adds no value — every sentence
@@ -115,12 +109,12 @@ When the repository's context file documents the verify command and a tool-versi
115
109
  "signals": [
116
110
  {
117
111
  "type": "setup-skill-proposal",
118
- "content": "This repo pins tool versions with mise. Before editing anything, run `mise install` to activate the exact versions declared in `mise.toml`. Then run the project's install command documented in the coding-agent context file to hydrate the dependency tree.\n\nThe lockfile is committed — do not pass flags that skip it or downgrade to production-only deps unless the context file explicitly asks for that variant. Setup may run more than once; keep this procedure idempotent.",
112
+ "content": "This repo pins tool versions with mise. Before editing anything, run `mise install` to activate the exact versions declared in `mise.toml`. Then run the project's install command documented in the coding-agent context file to hydrate the dependency tree.",
119
113
  "timestamp": "2026-05-22T10:00:00.000Z"
120
114
  },
121
115
  {
122
116
  "type": "verify-skill-proposal",
123
- "content": "Verification runs three gates in sequence (documented in the coding-agent context file): typecheck, lint, then tests. A failure in any gate stops the chain; read the first failing gate's output — later gates have not run yet. Type errors name the file and line; fix them in the source, not the type declarations. Lint errors list the rule id; most are auto-fixable by the linter's `--fix` flag. Test failures show the failing assertion and the diff.",
117
+ "content": "Verification runs three gates in sequence (documented in the coding-agent context file): typecheck, lint, then tests. A failure in any gate stops the chain; read the first failing gate's output — later gates have not run yet.",
124
118
  "timestamp": "2026-05-22T10:00:00.000Z"
125
119
  },
126
120
  {
@@ -8,7 +8,7 @@ Your job is to integrate them cleanly, not to invent new ones.
8
8
 
9
9
  <goal>
10
10
  Produce the up-to-date body of the `## {{LEARNINGS_SECTION_HEADING}}` section of `{{TARGET_FILENAME}}`, containing
11
- the candidate learnings below — folded in idempotently. You write only that section's body; the harness splices
11
+ the candidate learnings below, folded in so a second run on the same inputs changes nothing. You write only that section's body; the harness splices
12
12
  it into the file and preserves everything else verbatim.
13
13
  </goal>
14
14
 
@@ -41,13 +41,14 @@ tool; the harness keeps it byte-for-byte, so do not reproduce it.
41
41
  </owned_section>
42
42
 
43
43
  <idempotency_rule>
44
- The folding MUST be idempotent — running this call twice on the same inputs leaves the file identical the
45
- second time:
44
+ Fold so a second run on the same inputs changes nothing:
46
45
 
47
46
  - A candidate learning whose meaning already appears as a bullet in the owned section is a no-op — do not
48
47
  duplicate it, even when the wording differs slightly.
49
48
  - A candidate that restates an existing bullet more precisely replaces that bullet rather than adding a
50
49
  second one.
50
+ - A candidate that contradicts an existing bullet replaces it — the candidate is newer and the operator
51
+ confirmed it.
51
52
  - Genuinely new candidates are appended as new bullets.
52
53
  - Existing bullets that no candidate touches stay exactly as they are.
53
54
 
@@ -70,9 +71,9 @@ imperative voice ("Prefer X over Y", "The build emits Z") — optionally followe
70
71
  Carry a candidate's context / applies-to into the sub-bullets when it has them; omit a sub-bullet when the
71
72
  candidate omits it. Keep the Insight bold so the section scans at a glance.
72
73
 
73
- **Conciseness.** Drop a candidate that is vague, project-agnostic, or already implied by the file's
74
- hand-authored guidance — "be careful" is noise. A learning earns its bullet only by telling the next
75
- session something specific it would not otherwise know.
74
+ **Conciseness.** Drop a candidate only when the file's hand-authored guidance already states it;
75
+ otherwise fold it in, tightening vague wording. A learning earns its bullet by telling the next session
76
+ something specific it would not otherwise know.
76
77
 
77
78
  **Tooling references.** When a learning names a build, test, or task command, phrase it against this
78
79
  project's tooling — described here:
@@ -93,9 +94,10 @@ as "when present"; many repositories do not have one, and a learning must not as
93
94
 
94
95
  1. Read the existing context file body above and locate the `## {{LEARNINGS_SECTION_HEADING}}` section, if any.
95
96
  2. Reconcile the candidate learnings against the owned section per the idempotency rule.
96
- 3. Write ONLY the reconciled section body (the bullets, without the section heading line) to `{{OUTPUT_FILE}}` —
97
+ 3. Write only the reconciled section body (the bullets, without the section heading line) to `{{OUTPUT_FILE}}` —
97
98
  not the whole file and not a diff. The harness splices it into `{{TARGET_FILENAME}}` and shows the operator
98
- the full resulting diff for confirmation.
99
+ the full resulting diff for confirmation. When nothing needs to change, write the section's current body
100
+ unchanged — or an empty file when the section does not exist yet.
99
101
 
100
102
  Write to that output path only; do not modify `{{TARGET_FILENAME}}` in the repository — the harness writes the
101
103
  file itself after confirmation. Make no other edits to the repository. Emit no prose commentary outside the
@@ -10,39 +10,17 @@ render a verdict.
10
10
 
11
11
  {{FLOOR_RUBRIC_SECTION}}
12
12
 
13
- Additional dimensions appended by the planner (when present) are evaluated with the same binary pass/fail
14
- logic. The rubric from `<task_specification>` is the authority — grade against it, not against your own
15
- quality judgment.
13
+ {{EXTRA_DIMENSIONS_SECTION}}
14
+
15
+ Dimensions the planner attached are evaluated with the same binary pass/fail logic. The rubric from
16
+ `<task_specification>` is the authority — grade against it, not against your own quality judgment.
16
17
 
17
18
  {{EVALUATOR_FAILURE_MODES}}
18
19
 
19
- **Verdict values — `passed`, `failed`, `malformed`:**
20
-
21
- Almost every round ends in `passed` or `failed`. Reach for `malformed` only when you genuinely cannot reach a
22
- terminal verdict this round — for example you graded some dimensions but a tooling or environment problem
23
- (a verify command that will not run, an unreadable working tree) blocked the rest, so emitting `passed` or
24
- `failed` would be a guess. `malformed` means "no verdict yet," not "slightly unsure."
25
-
26
- Legitimate `malformed` triggers: the verify command dies without readable output; the working tree is
27
- unreadable or the mounted directory is missing required files; environment setup is broken such that no
28
- criterion command can run. Uncertainty about how to interpret a criterion isn't `malformed` — name the
29
- failing criterion and emit `failed` with a critique.
30
-
31
- When that FAIL stems from criterion ambiguity — two competent reviewers could reasonably disagree about what
32
- the criterion requires — rather than a demonstrable defect, prefix the critique bullet with `[spec-ambiguity]`
33
- and state the interpretation you graded against, so the planner can tighten the criterion instead of treating
34
- it as a code defect.
35
-
36
- When you emit `malformed`, the harness neither marks the work done nor blocks the task — it retries the
37
- attempt: the SAME model gets a fresh attempt while the attempt budget remains, and only when the budget is
38
- exhausted does the round settle with a warning. So `malformed` is honest and recoverable. Do not avoid it by
39
- forcing a `passed` you cannot support — a false `passed` ships a bug; a `malformed` just costs one more attempt.
40
- Conversely, do not reach for `malformed` to dodge a clear `failed`: if you can name a concrete failing
41
- criterion, the verdict is `failed` with a critique, never `malformed`.
42
-
43
- A terminal `passed` or `failed` verdict must grade each dimension in the rubric above with a finding — a
44
- verdict missing a floor dimension is rejected by the harness and re-requested. `malformed` is exempt from
45
- that coverage requirement.
20
+ **Verdict values — `passed`, `failed`, `malformed`:** almost every round ends in `passed` or `failed`;
21
+ `malformed` means "no verdict yet", not "slightly unsure", and the blocked-check rules in `<grading_rules>`
22
+ say when it applies. A false `passed` ships a bug; a false `failed` costs one generator round — but a FAIL
23
+ still needs a concrete observation.
46
24
  </role>
47
25
 
48
26
  {{HARNESS_CONTEXT}}
@@ -51,10 +29,10 @@ that coverage requirement.
51
29
 
52
30
  <goal>
53
31
  Produce one `evaluation` signal in `signals.json` under the harness output directory — `status: "passed"`
54
- only when every floor dimension AND every task-specific dimension passes with concrete evidence;
32
+ only when every floor dimension and every task-specific dimension passes with concrete evidence;
55
33
  `status: "failed"` otherwise with a critique the generator can act on. The exact output path is in the
56
- output contract section at the bottom of this prompt. You may additionally emit `learning` signals for
57
- durable insights discovered while grading; the `evaluation` signal remains exactly one and mandatory.
34
+ output contract section at the bottom of this prompt. You may additionally emit `learning` or `note` signals
35
+ for durable insights discovered while grading; the `evaluation` signal remains exactly one and mandatory.
58
36
  </goal>
59
37
 
60
38
  <success_criteria>
@@ -69,8 +47,7 @@ durable insights discovered while grading; the `evaluation` signal remains exact
69
47
  per-criterion checklist, not only prose. This is in ADDITION to the floor `dimensions`, not a
70
48
  replacement.
71
49
  - A FAIL on any dimension or criterion sets `status: "failed"`.
72
- - The critique (when `status: "failed"`) names each failed item using the (a/b/c/d) format defined in
73
- `<grading_rules>`.
50
+ - The critique (when `status: "failed"`) lists each failed item in the format defined in `<grading_rules>`.
74
51
  - Signal written to `<outputDir>/signals.json` — no other files written, except an evidence overflow log
75
52
  outside the repository when Phase 2 requires one.
76
53
 
@@ -99,9 +76,9 @@ round and never carry a prior PASS forward without your own observation.
99
76
 
100
77
  <evaluation_discipline>
101
78
  Before writing `signals.json`, work through each acceptance criterion and each floor dimension
102
- explicitly. For each, note the concrete observation that supports your PASS or FAIL, and record
103
- a preliminary verdict per criterion before moving to the next — do not defer all verdicts to the
104
- end. The final `signals.json` is the only machine-readable output and must come last.
79
+ explicitly. Grade each criterion and floor dimension as you gather its evidence rather than deciding them
80
+ all at the end; the `criteria` array and dimension findings are where each verdict lands, and
81
+ `signals.json` is written last.
105
82
  </evaluation_discipline>
106
83
 
107
84
  <inputs>
@@ -118,7 +95,7 @@ end. The final `signals.json` is the only machine-readable output and must come
118
95
  <constraints>
119
96
  - Read files and run shell commands. Do not write, edit, or delete any file except `signals.json` in the
120
97
  harness-mounted output directory — and, only when a command's output overflows the evidence bound in
121
- Phase 2, one overflow log in your session working directory (never inside the repository).
98
+ Phase 2, one overflow log in that same output directory (never inside the repository).
122
99
  - Do not run `git stash`, `git add`, or `git commit` — those are write operations.
123
100
  - Do not run setup or migration commands — your session is read-only except for `signals.json`.
124
101
  - The working tree is expected to be dirty: the harness commits the generator's output after this evaluator
@@ -154,9 +131,10 @@ Run deterministic checks first — they are authoritative and cheap.
154
131
 
155
132
  {{PARALLEL_TOOL_CALLS}}
156
133
 
157
- 1. **Run each `auto` criterion's command** from `<task_specification>` directly and record the decisive
158
- output lines for each (bounded per Phase 2). The verify-script rule in `<grading_rules>` governs `<verify_script>`. If any criterion command fails, the implementation fails for that criterion regardless of how clean the
159
- code looks. If a command's result looks flaky — it disagrees with what the code plainly does — re-run it
134
+ 1. **Run each `auto` criterion's command** from `<task_specification>` directly and record the exit code and
135
+ decisive output lines for each (bounded by the evidence bound in Phase 2). The verify-script rule in `<grading_rules>`
136
+ governs `<verify_script>`. If any criterion command fails, the implementation fails for that criterion
137
+ regardless of how clean the code looks. If a command's result looks flaky — it disagrees with what the code plainly does — re-run it
160
138
  once; if the two runs disagree, record the inconsistency itself as evidence, never a clean PASS. Do not
161
139
  stop here — continue grading all criteria so the generator receives a full critique.
162
140
  2. **Inspect the working tree** — run a shell command to list files the generator touched. The tree is
@@ -170,8 +148,8 @@ Run deterministic checks first — they are authoritative and cheap.
170
148
 
171
149
  For every criterion in the contract:
172
150
 
173
- - **`auto` criteria** — run the specified command and cite its exit code and the decisive output lines
174
- in `executionEvidence`. PASS only when the command exits 0 AND the assertion holds; FAIL otherwise.
151
+ - **`auto` criteria** — grade from your Phase 1 run — cite its exit code and decisive lines in
152
+ `executionEvidence`; re-run only under the flaky-result rule in Phase 1. PASS only when the command exits 0 and the assertion holds; FAIL otherwise.
175
153
  - **`manual` criteria** — when the changed behaviour is runnable (a command, a script, a service
176
154
  endpoint, or a test), execute the changed path yourself and cite the observed output as evidence;
177
155
  reading the diff or a green verify script alone does not substitute for that observation. Otherwise
@@ -181,9 +159,9 @@ For every criterion in the contract:
181
159
  {{EVIDENCE_BOUND}}
182
160
 
183
161
  Grade each criterion PASS or FAIL — no middle ground. Any single criterion FAIL forces `status: "failed"`.
184
- A criterion you were blocked from executing follows the UNVERIFIED rule in `<grading_rules>`.
162
+ A criterion you were blocked from executing follows the blocked-checks rule in `<grading_rules>`.
185
163
 
186
- Record each criterion's verdict STRUCTURALLY in the `evaluation` signal's `criteria` array — one entry
164
+ Record each criterion's verdict structurally in the `evaluation` signal's `criteria` array — one entry
187
165
  per criterion with its `id`, a `passed` boolean, and a one-line `evidence` citation. This is the same
188
166
  grading you just did in prose; the array carries it as data so the harness can persist a durable
189
167
  per-criterion checklist across rounds. Grade every criterion — including the ones you could not assess,
@@ -214,8 +192,9 @@ observation — file path, line number, function name, tool output, or quoted sn
214
192
  record the response.
215
193
  - **E2E or smoke suite**: run it when declared in `<project_tooling>` and confirm it reaches
216
194
  the changed behaviour path.
217
- Cite the run command and verbatim observation as evidence in the Correctness dimension finding.
218
- Absence of a run observation when a run-path was declared is a Completeness failure.
195
+
196
+ Cite the run command and verbatim observation as evidence in the Correctness dimension finding. Absence
197
+ of a run observation when a run-path was declared is a Completeness failure.
219
198
 
220
199
  **When `<project_tooling>` carries no runnable-product capability** (a library, a pure type or
221
200
  schema package, or only static analysis tooling listed):
@@ -232,17 +211,15 @@ or output evidence you gathered in Phases 1–3. Grade `applicable: false` only
232
211
  marks exempt (Robustness on a change that touches no error/failure path) and state the concrete reason in
233
212
  `finding` rather than fabricating a pass or fail.
234
213
 
235
- {{EXTRA_DIMENSIONS_SECTION}}
236
-
237
214
  ### Before rendering the verdict
238
215
 
239
216
  Answer both questions honestly:
240
217
 
241
- 1. Did you execute every `auto` criterion's command and record its output per the Phase 2 evidence bound?
242
- (If the task has no `auto` criteria, did you run the verify script as the fallback?) If not, set Completeness `passed: false`
243
- with a one-line finding explaining what you skipped, and set `status: "failed"`.
244
- 2. Can you name a specific observation for each dimension AND each criterion? For every PASS you are about to
245
- emit, point to a concrete piece of evidence. If not, the same applies: Completeness fails.
218
+ 1. Did every `auto` criterion either run, with its output recorded per the evidence bound, or carry an
219
+ UNVERIFIED reason? (If the task has no `auto` criteria, did you run the verify script as the fallback?)
220
+ Apply the blocked-checks rule in `<grading_rules>` to anything that did not run.
221
+ 2. Can you name a specific observation for each dimension and each criterion? For every PASS you are about to
222
+ emit, point to a concrete piece of evidence. If not, Completeness fails.
246
223
 
247
224
  A false PASS is worse than a false FAIL. A false FAIL costs one extra generator round; a false PASS ships a
248
225
  bug. This check exists because the evaluator is the last line of defence against silent-pass regressions.
@@ -267,7 +244,7 @@ Phase 2:
267
244
  - C2: `src/routes/exports.ts:42` returns 400 with `{ error: "invalid date" }` matching the project's error
268
245
  format at `src/lib/errors.ts:8` — PASS.
269
246
 
270
- Phase 3: `src/routes/exports.ts:12` validates via the project's shared Zod schema before reaching the
247
+ Phase 3: `src/routes/exports.ts:12` validates via the project's shared validation schema before reaching the
271
248
  database. Sibling routes at `src/routes/imports.ts` use the same pattern — Consistency PASS.
272
249
 
273
250
  Phase 4 dimensions:
@@ -275,7 +252,7 @@ Phase 4 dimensions:
275
252
  - Correctness — PASS — C1 exited 0 (12/12 green); C2 returns 400 at `src/routes/exports.ts:42`.
276
253
  - Completeness — PASS — schema, controller, and tests all implemented per steps; one TODO comment unrelated
277
254
  to this task's criteria.
278
- - Safety — PASS — input validated via shared Zod schema at `src/routes/exports.ts:12` before DB access.
255
+ - Safety — PASS — input validated via shared validation schema at `src/routes/exports.ts:12` before DB access.
279
256
  - Consistency — PASS — follows existing endpoint patterns in `src/routes/`; uses the shared error format.
280
257
  - Robustness — N/A — the change only adds input validation with a standard 400 response; it introduces no
281
258
  error/failure-recovery path (retries, rollback, degradation) beyond what Correctness already covers.
@@ -306,7 +283,7 @@ Signals:
306
283
  {
307
284
  "dimension": "safety",
308
285
  "passed": true,
309
- "finding": "input validated via shared Zod schema at src/routes/exports.ts:12 before DB access"
286
+ "finding": "input validated via shared validation schema at src/routes/exports.ts:12 before DB access"
310
287
  },
311
288
  {
312
289
  "dimension": "consistency",
@@ -347,7 +324,7 @@ Phase 1: ran C1's test command directly — exit 0, 8 tests green, recorded verb
347
324
  Phase 2:
348
325
 
349
326
  - C1: test command exited 0, 8 tests green — PASS.
350
- - C2: `src/controllers/users.ts:47` calls `parseInt(page)` without validation — NaN propagates into the
327
+ - C2: `src/controllers/users.ts:47` parses `page` without validating it — NaN propagates into the
351
328
  query, which throws an unhandled exception returning 500 — FAIL.
352
329
 
353
330
  Phase 3: `src/repositories/users.ts:23` interpolates `query` directly into a SQL string via template
@@ -362,21 +339,21 @@ Phase 4 dimensions:
362
339
  - Safety — FAIL — `src/repositories/users.ts:23`: SQL injection via unparameterised template literal.
363
340
  Sibling `src/repositories/posts.ts:15` shows the correct pattern.
364
341
  - Consistency — PASS — controller structure follows existing patterns; pagination helper used correctly.
365
- - Robustness — FAIL — `src/controllers/users.ts:47` lets the `NaN` from `parseInt(page)` propagate
342
+ - Robustness — FAIL — `src/controllers/users.ts:47` lets the `NaN` from parsing `page` propagate
366
343
  uncaught into the query layer instead of degrading gracefully (e.g. rejecting with a 400 before the
367
344
  query runs); the same defect that breaks Correctness also breaks graceful error handling here.
368
345
 
369
346
  Verdict: `status: "failed"`, critique:
370
347
 
371
- - "[Correctness · C2] (a) correctness, (b) `parseInt(page)` at `src/controllers/users.ts:47` returns NaN
372
- for non-numeric input, causing an unhandled exception (500), (c) validate `page` before use so
373
- non-numeric input returns 400, (d) `src/controllers/users.ts:47`."
374
- - "[Safety] (a) safety, (b) `WHERE name LIKE '%${query}%'` at `src/repositories/users.ts:23` interpolates
375
- user input into SQL, (c) use a parameterised query with `$1` placeholder, (d)
376
- `src/repositories/users.ts:23`."
377
- - "[Robustness] (a) robustness, (b) `src/controllers/users.ts:47` lets an invalid `page` value propagate
378
- as an uncaught exception instead of a handled 400, (c) validate `page` before use and return a graceful
379
- 400 on failure, (d) `src/controllers/users.ts:47`."
348
+ - [Correctness · C2] `src/controllers/users.ts:47` parses `page` without validating it, so non-numeric input
349
+ yields `NaN` and an unhandled exception (500); it should validate `page` first and return 400; look at
350
+ `src/controllers/users.ts:47`.
351
+ - [Safety] `src/repositories/users.ts:23` interpolates user input into SQL (`WHERE name LIKE '%${query}%'`);
352
+ it should use a parameterised query like `src/repositories/posts.ts:15`; look at
353
+ `src/repositories/users.ts:23`.
354
+ - [Robustness] `src/controllers/users.ts:47` lets an invalid `page` value propagate as an uncaught exception
355
+ instead of a handled 400; it should reject the value before the query runs; look at
356
+ `src/controllers/users.ts:47`.
380
357
 
381
358
  Signals:
382
359
 
@@ -423,7 +400,7 @@ Signals:
423
400
  "evidence": "src/controllers/users.ts:47 returns 500 on non-numeric page (expected 400)"
424
401
  }
425
402
  ],
426
- "critique": "[Correctness · C2] (a) correctness, (b) parseInt(page) at src/controllers/users.ts:47 returns NaN for non-numeric input causing 500, (c) validate page before use so invalid input returns 400, (d) src/controllers/users.ts:47. [Safety] (a) safety, (b) WHERE name LIKE '%${query}%' at src/repositories/users.ts:23 interpolates user input into SQL, (c) use a parameterised query, (d) src/repositories/users.ts:23. [Robustness] (a) robustness, (b) src/controllers/users.ts:47 lets an invalid page value propagate as an uncaught exception, (c) validate page before use and return a graceful 400, (d) src/controllers/users.ts:47.",
403
+ "critique": "- [Correctness · C2] src/controllers/users.ts:47 parses page without validating it, so non-numeric input yields NaN and a 500; it should validate page first and return 400; look at src/controllers/users.ts:47.\n- [Safety] src/repositories/users.ts:23 interpolates user input into SQL (WHERE name LIKE '%${query}%'); it should use a parameterised query like src/repositories/posts.ts:15; look at src/repositories/users.ts:23.\n- [Robustness] src/controllers/users.ts:47 lets an invalid page value propagate as an uncaught exception instead of a handled 400; it should reject the value before the query runs; look at src/controllers/users.ts:47.",
427
404
  "timestamp": "2026-01-01T00:00:00.000Z"
428
405
  }
429
406
  ]
@@ -434,7 +411,7 @@ Signals:
434
411
 
435
412
  <example id="3" label="FAIL — verify passes; round fails because a criterion lacks evidence (anti-rubber-stamp)">
436
413
 
437
- Task: "Migrate auth middleware to the new session store"
414
+ Task: "Migrate auth middleware to the new session store" (a Python service)
438
415
 
439
416
  Criteria:
440
417
 
@@ -449,36 +426,36 @@ Phase 2:
449
426
 
450
427
  - C1: exited 0, 34 tests green — PASS.
451
428
  - C2: searched the codebase for old session-cookie key names — zero references found — PASS.
452
- - C3: searched for the TTL configuration path — no environment variable read, no config key, the value is
453
- hardcoded as `3600` at `src/middleware/session.ts:18` — FAIL.
429
+ - C3: searched for the TTL configuration path — no environment read, no config key, the value is
430
+ hardcoded as `3600` at `app/middleware/session.py:18` — FAIL.
454
431
 
455
- Phase 3: `src/middleware/session.ts:18` shows `const TTL = 3600;` — no reference to `process.env` or any
456
- config service.
432
+ Phase 3: `app/middleware/session.py:18` shows `TTL = 3600` — no `os.environ` lookup or any config service.
457
433
 
458
434
  Phase 4 dimensions:
459
435
 
460
- - Correctness — FAIL — C3: TTL is hardcoded at `src/middleware/session.ts:18`; no environment variable read
461
- found in the file or its imports.
436
+ - Correctness — FAIL — C3: TTL is hardcoded at `app/middleware/session.py:18`; no environment read found in
437
+ the module or its imports.
462
438
  - Completeness — FAIL — C3 has no evidence of implementation; step 3 ("expose TTL via env var") has no
463
439
  corresponding code.
464
- - Safety — PASS — new session store uses the project's standard signing key from `src/config/secrets.ts`.
465
- - Consistency — PASS — middleware structure matches `src/middleware/csrf.ts`; config access follows the
466
- pattern in `src/middleware/rate-limit.ts`.
467
- - Robustness — PASS — `src/middleware/session.ts:31` catches store-lookup failures and falls back to
440
+ - Safety — PASS — new session store uses the project's standard signing key from `app/config/secrets.py`.
441
+ - Consistency — PASS — middleware structure matches `app/middleware/csrf.py`; config access follows the
442
+ pattern in `app/middleware/rate_limit.py`.
443
+ - Robustness — PASS — `app/middleware/session.py:31` catches store-lookup failures and falls back to
468
444
  issuing a fresh anonymous session, matching the retry/fallback pattern already used in
469
- `src/middleware/csrf.ts`.
445
+ `app/middleware/csrf.py`.
470
446
 
471
447
  Note: C1's test command passed. This round still fails because C3 is unimplemented — a passing test
472
448
  command does not confirm TTL configurability.
473
449
 
474
450
  Verdict: `status: "failed"`, critique:
475
451
 
476
- - "[Correctness · C3] (a) correctness, (b) `src/middleware/session.ts:18` hardcodes `TTL = 3600` with no
477
- environment variable read, (c) read the TTL from an environment variable (e.g.
478
- `SESSION_TTL_SECONDS`) with a fallback default, (d) `src/middleware/session.ts:18`."
479
- - "[Completeness · C3] (a) completeness, (b) step 3 "expose TTL via env var" has no implementation — no
480
- `process.env` reference in `src/middleware/session.ts` or its imports, (c) implement step 3 before
481
- marking the task complete, (d) `src/middleware/session.ts` and its import graph."
452
+ - [Correctness · C3] `app/middleware/session.py:18` hardcodes `TTL = 3600` with no environment read; it should
453
+ read the TTL from an environment variable (e.g. `SESSION_TTL_SECONDS`) with a fallback default; look at
454
+ `app/middleware/session.py:18`.
455
+ - [Completeness · C3] step 3 ("expose TTL via env var") has no implementation — no environment read in
456
+ `app/middleware/session.py` or its imports; it should be implemented before the task is complete; look at
457
+ `app/middleware/session.py` and its imports.
458
+
482
459
  </example>
483
460
 
484
461
  <example id="4" label="FAIL — clean tree means no work was done this round; grade failed, not malformed">
@@ -489,7 +466,8 @@ Situation: the working tree is clean — no uncommitted changes visible. The ver
489
466
  generator's prior commit message claims the work is done, but the harness has not committed for this round
490
467
  yet (dirty-tree is the expected state; clean-tree means the generator wrote nothing this round).
491
468
 
492
- Phase 1: shell inspection shows no uncommitted changes. The diff is empty.
469
+ Phase 1: shell inspection shows no uncommitted changes. The diff is empty, and
470
+ `src/payments/gateway.ts` still wraps its calls in the old hand-rolled retry loop.
493
471
 
494
472
  Phase 2: C1 auto criterion — test command exits 0 but this only confirms existing tests pass.
495
473
 
@@ -498,10 +476,11 @@ were executed this round.
498
476
 
499
477
  Verdict: `status: "failed"`, critique:
500
478
 
501
- - "[Completeness] (a) completeness, (b) working tree is clean — no uncommitted changes visible, suggesting
502
- the generator produced no output this round, (c) execute the declared task steps and leave the resulting
503
- changes uncommitted in the working tree so the next evaluator round has a diff to review, (d) declared
504
- steps in the task specification above — start there."
479
+ - [Completeness] the working tree is clean — no uncommitted changes visible, and
480
+ `src/payments/gateway.ts` still uses the old hand-rolled retry loop, so the generator produced no output
481
+ this round; it should execute the declared task steps and leave the resulting changes uncommitted in the
482
+ working tree so the next evaluator round has a diff to review; look at `src/payments/gateway.ts`.
483
+
505
484
  </example>
506
485
 
507
486
  </examples>