ralphctl 0.24.0 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/chunk-BAAV7AGY.mjs +11 -0
- package/dist/{cli-FHPDHGP5.mjs → cli-P7D6G5NI.mjs} +12817 -8910
- package/dist/cli.mjs +3 -1
- package/dist/manifest.json +7 -2
- package/dist/prompts/_partials/approval-gate.md +5 -5
- package/dist/prompts/_partials/autonomous-operation.md +7 -1
- package/dist/prompts/_partials/conventions-agents-md.md +8 -20
- package/dist/prompts/_partials/conventions-claude-md.md +12 -15
- package/dist/prompts/_partials/conventions-copilot-instructions.md +6 -10
- package/dist/prompts/_partials/decisions.md +3 -8
- package/dist/prompts/_partials/evaluator-failure-modes.md +8 -5
- package/dist/prompts/_partials/evaluator-grading-rules.md +24 -10
- package/dist/prompts/_partials/evidence-bound.md +2 -2
- package/dist/prompts/_partials/git-boundary.md +7 -0
- package/dist/prompts/_partials/harness-context.md +1 -1
- package/dist/prompts/_partials/question-format.md +7 -0
- package/dist/prompts/_partials/skill-body.md +5 -0
- package/dist/prompts/_partials/task-blocked.md +10 -0
- package/dist/prompts/_partials/task-fields.md +19 -3
- package/dist/prompts/_partials/task-sizing.md +2 -2
- package/dist/prompts/_partials/validation-checklist.md +6 -8
- package/dist/prompts/apply-feedback/template.md +19 -21
- package/dist/prompts/create-pr/template.md +16 -12
- package/dist/prompts/detect-scripts/template.md +4 -6
- package/dist/prompts/detect-skills/template.md +17 -23
- package/dist/prompts/distill-learnings/template.md +10 -8
- package/dist/prompts/evaluate/template.md +72 -93
- package/dist/prompts/evaluate-continuation/template.md +18 -21
- package/dist/prompts/ideate/template.md +34 -44
- package/dist/prompts/implement/template.md +49 -66
- package/dist/prompts/implement-continuation/template.md +21 -20
- package/dist/prompts/implement-crash-resume/template.md +42 -0
- package/dist/prompts/plan/template.md +73 -61
- package/dist/prompts/readiness/template.md +35 -28
- package/dist/prompts/refine/template.md +34 -44
- package/dist/prompts/reproduce/template.md +9 -7
- package/dist/prompts/select-candidate/template.md +17 -10
- package/dist/use-no-color-NEMCAVY3.mjs +9 -0
- package/package.json +1 -1
|
@@ -1,9 +1,8 @@
|
|
|
1
1
|
<role>
|
|
2
2
|
You are an AI coding agent performing a single-shot extraction task: authoring a pull-request title and
|
|
3
3
|
body for a branch that is ready to merge. Your audience is the project's maintainers reviewing the PR.
|
|
4
|
-
Write
|
|
5
|
-
|
|
6
|
-
description that it was authored with assistance.
|
|
4
|
+
Write in the project's own voice. Leave out this tooling's internals — harness, sprint identifiers, signal
|
|
5
|
+
contracts, flow names — because they mean nothing to a reviewer.
|
|
7
6
|
</role>
|
|
8
7
|
|
|
9
8
|
<goal>
|
|
@@ -30,6 +29,8 @@ Inspect the commit history and diff of `{{HEAD_BRANCH}}` against `{{BASE_BRANCH}
|
|
|
30
29
|
{{TICKET_SUMMARY}}
|
|
31
30
|
</ticket_summary>
|
|
32
31
|
|
|
32
|
+
When `<ticket_summary>` is empty, no specific tickets are recorded for this branch.
|
|
33
|
+
|
|
33
34
|
<issue_refs>
|
|
34
35
|
{{ISSUE_REFS}}
|
|
35
36
|
</issue_refs>
|
|
@@ -37,20 +38,24 @@ Inspect the commit history and diff of `{{HEAD_BRANCH}}` against `{{BASE_BRANCH}
|
|
|
37
38
|
</inputs>
|
|
38
39
|
|
|
39
40
|
<constraints>
|
|
40
|
-
Gather context by running shell commands
|
|
41
|
-
`{{HEAD_BRANCH}}` — the commands below use `HEAD
|
|
41
|
+
Gather context by running shell commands before writing anything. The repository is at
|
|
42
|
+
`{{REPOSITORY_PATH}}`, checked out on `{{HEAD_BRANCH}}` — the commands below use `HEAD`, no checkout needed.
|
|
43
|
+
Your working directory is not the repository, so pass the path to every git command:
|
|
42
44
|
|
|
43
|
-
- Inspect the commit history: `git log {{BASE_BRANCH}}..HEAD`
|
|
44
|
-
- Inspect the file-level change summary: `git diff {{BASE_BRANCH}}...HEAD --stat`
|
|
45
|
-
- Inspect the full diff for any section
|
|
45
|
+
- Inspect the commit history: `git -C "{{REPOSITORY_PATH}}" log {{BASE_BRANCH}}..HEAD`
|
|
46
|
+
- Inspect the file-level change summary: `git -C "{{REPOSITORY_PATH}}" diff {{BASE_BRANCH}}...HEAD --stat`
|
|
47
|
+
- Inspect the full diff for any section the commit messages do not explain:
|
|
48
|
+
`git -C "{{REPOSITORY_PATH}}" diff {{BASE_BRANCH}}...HEAD`
|
|
46
49
|
|
|
47
50
|
Lean on `--stat` to group changes sensibly; read the full diff only for sections where commit messages are insufficient.
|
|
48
51
|
|
|
49
52
|
Title rules:
|
|
50
53
|
|
|
51
|
-
- One line,
|
|
52
|
-
-
|
|
53
|
-
|
|
54
|
+
- One line, ≤70 characters.
|
|
55
|
+
- Match the style of recent commit subjects on `{{BASE_BRANCH}}`
|
|
56
|
+
(`git -C "{{REPOSITORY_PATH}}" log -10 --format=%s {{BASE_BRANCH}}`): keep a type prefix such as `feat:` when they
|
|
57
|
+
use one; leave out branch names and internal ticket ids. Without a clear convention, write an imperative
|
|
58
|
+
present-tense line.
|
|
54
59
|
- Examples: "Add CSV export for transactions", "Fix race in session locking".
|
|
55
60
|
|
|
56
61
|
Body rules:
|
|
@@ -65,7 +70,6 @@ Body rules:
|
|
|
65
70
|
- Body length: ≤80 lines. Prefer fewer lines over more — reviewers skim.
|
|
66
71
|
- Tone: clear technical prose, matching the tone of the project's existing commit messages. Neither terse shorthand nor
|
|
67
72
|
essay-length explanation — aim for "readable in 60 seconds".
|
|
68
|
-
- Use em-dash `—` for explanatory clauses.
|
|
69
73
|
|
|
70
74
|
Issue references:
|
|
71
75
|
|
|
@@ -36,7 +36,7 @@ harness owns execution; the user reviews your proposal before anything runs. Rea
|
|
|
36
36
|
and metadata files — not source trees, tests, vendored directories, or generated output.
|
|
37
37
|
|
|
38
38
|
**Coding-agent context files are the strongest evidence.** Before any manifest, look for
|
|
39
|
-
`CLAUDE.md`, `AGENTS.md`, `.cursor/rules/*.
|
|
39
|
+
`CLAUDE.md`, `AGENTS.md`, `.cursor/rules/*.mdc`, `.github/copilot-instructions.md`, and human
|
|
40
40
|
onboarding docs (`README.md`, `CONTRIBUTING.md`). These files are written by the project's authors
|
|
41
41
|
to document the exact commands the project uses — if any of them name a setup or verify command,
|
|
42
42
|
lift it verbatim. Prefer this over any inference from manifest scripts.
|
|
@@ -66,7 +66,9 @@ class entirely.
|
|
|
66
66
|
|
|
67
67
|
**Script safety.** Reject pipe-to-shell shapes (`curl … | sh`, `wget -O- … | bash`), `eval`, and
|
|
68
68
|
`rm -rf` — the harness runs these scripts unattended, so a remote-fetched or destructive command
|
|
69
|
-
cannot be reviewed before it executes.
|
|
69
|
+
cannot be reviewed before it executes. This overrides verbatim lifting: when a documented command
|
|
70
|
+
contains one of these shapes, omit that command and say in a `note` what you omitted and why — the
|
|
71
|
+
operator decides. One shell line per script — multi-line bodies and heredocs are out of contract, and so
|
|
70
72
|
are sub-shells except the single `(cd <path> && …)` fallback named above; the harness collapses
|
|
71
73
|
whitespace before execution.
|
|
72
74
|
|
|
@@ -240,8 +242,4 @@ what it starts (dev server at which port, CLI binary, or e2e suite), and any env
|
|
|
240
242
|
prerequisite the project's own files declare. Omit the note when the project's files name no run
|
|
241
243
|
command — do not fabricate one from inference.
|
|
242
244
|
|
|
243
|
-
When a context file and a manifest name the same command, the context file wins (it's deliberate
|
|
244
|
-
author intent). For the verify script, prefer chaining the project's own task scripts over
|
|
245
|
-
re-spelling the underlying tools — the project's scripts are the documented contract.
|
|
246
|
-
|
|
247
245
|
</run_command_note>
|
|
@@ -18,7 +18,7 @@ repo — if covered, emit a `note` instead.
|
|
|
18
18
|
already cover both responsibilities (in which case emit a `note` explaining what was found).
|
|
19
19
|
- Every concrete claim in a skill body — a tool name, a command flag, a directory path — is backed by
|
|
20
20
|
a file you read in this repo or its context files. No training-data generics.
|
|
21
|
-
- Skill bodies
|
|
21
|
+
- Skill bodies follow the `Skill bodies` rule in the constraints below.
|
|
22
22
|
- `signals.json` is valid JSON that parses against the output contract schema.
|
|
23
23
|
|
|
24
24
|
</success_criteria>
|
|
@@ -51,15 +51,10 @@ skills.
|
|
|
51
51
|
|
|
52
52
|
<skills_convention>{{SKILLS_CONVENTION}}</skills_convention>
|
|
53
53
|
|
|
54
|
-
|
|
55
|
-
backed by something you read in the repo or a context file. Drop any claim you cannot tie to a file.
|
|
54
|
+
{{SKILL_BODY_RULES}}
|
|
56
55
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
each in its own section). Code fences inside the body are fine.
|
|
60
|
-
|
|
61
|
-
**Skill content must be useful, not aspirational.** "Run the project's install command" is useful. "Be
|
|
62
|
-
careful with edge cases" is noise. Delete any paragraph that would apply to any project.
|
|
56
|
+
The harness puts its own title heading above each body, so start with body text rather than a top-level
|
|
57
|
+
`#` heading.
|
|
63
58
|
|
|
64
59
|
**Document stack-specific quirks.** A non-default toolchain, tool-version pin, lockfile policy,
|
|
65
60
|
monorepo sub-tree ordering dependency, or anything else that would trip up a generic AI session
|
|
@@ -85,20 +80,19 @@ to read its output and diagnose failures.
|
|
|
85
80
|
|
|
86
81
|
<inspection_protocol>
|
|
87
82
|
|
|
88
|
-
|
|
83
|
+
Read first, in this order: the existing skills at the convention path, the coding-agent context files
|
|
84
|
+
(when present), then the configuration and metadata files in scope above.
|
|
89
85
|
|
|
90
|
-
|
|
91
|
-
covers. State explicitly whether the setup or verify intent is already taken. When no existing
|
|
92
|
-
skills exist, note that — it means you should emit both.
|
|
93
|
-
2. The coding-agent context files you found (when present) and the commands or conventions they
|
|
94
|
-
explicitly name.
|
|
95
|
-
3. The manifests you read and what stack each implies. For monorepos, name the sub-trees.
|
|
96
|
-
4. The single most important thing the next AI session would NOT know without this skill — the
|
|
97
|
-
asymmetry between what is documented and what is load-bearing for real work.
|
|
98
|
-
5. A one-line outline of each skill's content before drafting, or an explicit "skip — already covered
|
|
99
|
-
by `<existing skill id>`" when an existing skill makes the new one redundant.
|
|
86
|
+
Then establish for yourself, not as output:
|
|
100
87
|
|
|
101
|
-
|
|
88
|
+
1. Which responsibility each existing skill already covers, and whether the setup or verify intent is
|
|
89
|
+
already taken. With no existing skills, emit both.
|
|
90
|
+
2. The commands or conventions the context files explicitly name.
|
|
91
|
+
3. What stack each manifest implies. For monorepos, name the sub-trees.
|
|
92
|
+
4. The single most important thing the next AI session would not know without this skill — the
|
|
93
|
+
asymmetry between what is documented and what is load-bearing for real work.
|
|
94
|
+
5. A one-line outline of each skill before drafting, or "skip — already covered by `<existing skill
|
|
95
|
+
id>`" when an existing skill makes the new one redundant.
|
|
102
96
|
|
|
103
97
|
For polyglot monorepos, give the AI the relationship between sub-trees (e.g. "the frontend depends
|
|
104
98
|
on a build artifact produced by the backend"). Generic boilerplate adds no value — every sentence
|
|
@@ -115,12 +109,12 @@ When the repository's context file documents the verify command and a tool-versi
|
|
|
115
109
|
"signals": [
|
|
116
110
|
{
|
|
117
111
|
"type": "setup-skill-proposal",
|
|
118
|
-
"content": "This repo pins tool versions with mise. Before editing anything, run `mise install` to activate the exact versions declared in `mise.toml`. Then run the project's install command documented in the coding-agent context file to hydrate the dependency tree
|
|
112
|
+
"content": "This repo pins tool versions with mise. Before editing anything, run `mise install` to activate the exact versions declared in `mise.toml`. Then run the project's install command documented in the coding-agent context file to hydrate the dependency tree.",
|
|
119
113
|
"timestamp": "2026-05-22T10:00:00.000Z"
|
|
120
114
|
},
|
|
121
115
|
{
|
|
122
116
|
"type": "verify-skill-proposal",
|
|
123
|
-
"content": "Verification runs three gates in sequence (documented in the coding-agent context file): typecheck, lint, then tests. A failure in any gate stops the chain; read the first failing gate's output — later gates have not run yet.
|
|
117
|
+
"content": "Verification runs three gates in sequence (documented in the coding-agent context file): typecheck, lint, then tests. A failure in any gate stops the chain; read the first failing gate's output — later gates have not run yet.",
|
|
124
118
|
"timestamp": "2026-05-22T10:00:00.000Z"
|
|
125
119
|
},
|
|
126
120
|
{
|
|
@@ -8,7 +8,7 @@ Your job is to integrate them cleanly, not to invent new ones.
|
|
|
8
8
|
|
|
9
9
|
<goal>
|
|
10
10
|
Produce the up-to-date body of the `## {{LEARNINGS_SECTION_HEADING}}` section of `{{TARGET_FILENAME}}`, containing
|
|
11
|
-
the candidate learnings below
|
|
11
|
+
the candidate learnings below, folded in so a second run on the same inputs changes nothing. You write only that section's body; the harness splices
|
|
12
12
|
it into the file and preserves everything else verbatim.
|
|
13
13
|
</goal>
|
|
14
14
|
|
|
@@ -41,13 +41,14 @@ tool; the harness keeps it byte-for-byte, so do not reproduce it.
|
|
|
41
41
|
</owned_section>
|
|
42
42
|
|
|
43
43
|
<idempotency_rule>
|
|
44
|
-
|
|
45
|
-
second time:
|
|
44
|
+
Fold so a second run on the same inputs changes nothing:
|
|
46
45
|
|
|
47
46
|
- A candidate learning whose meaning already appears as a bullet in the owned section is a no-op — do not
|
|
48
47
|
duplicate it, even when the wording differs slightly.
|
|
49
48
|
- A candidate that restates an existing bullet more precisely replaces that bullet rather than adding a
|
|
50
49
|
second one.
|
|
50
|
+
- A candidate that contradicts an existing bullet replaces it — the candidate is newer and the operator
|
|
51
|
+
confirmed it.
|
|
51
52
|
- Genuinely new candidates are appended as new bullets.
|
|
52
53
|
- Existing bullets that no candidate touches stay exactly as they are.
|
|
53
54
|
|
|
@@ -70,9 +71,9 @@ imperative voice ("Prefer X over Y", "The build emits Z") — optionally followe
|
|
|
70
71
|
Carry a candidate's context / applies-to into the sub-bullets when it has them; omit a sub-bullet when the
|
|
71
72
|
candidate omits it. Keep the Insight bold so the section scans at a glance.
|
|
72
73
|
|
|
73
|
-
**Conciseness.** Drop a candidate
|
|
74
|
-
|
|
75
|
-
|
|
74
|
+
**Conciseness.** Drop a candidate only when the file's hand-authored guidance already states it;
|
|
75
|
+
otherwise fold it in, tightening vague wording. A learning earns its bullet by telling the next session
|
|
76
|
+
something specific it would not otherwise know.
|
|
76
77
|
|
|
77
78
|
**Tooling references.** When a learning names a build, test, or task command, phrase it against this
|
|
78
79
|
project's tooling — described here:
|
|
@@ -93,9 +94,10 @@ as "when present"; many repositories do not have one, and a learning must not as
|
|
|
93
94
|
|
|
94
95
|
1. Read the existing context file body above and locate the `## {{LEARNINGS_SECTION_HEADING}}` section, if any.
|
|
95
96
|
2. Reconcile the candidate learnings against the owned section per the idempotency rule.
|
|
96
|
-
3. Write
|
|
97
|
+
3. Write only the reconciled section body (the bullets, without the section heading line) to `{{OUTPUT_FILE}}` —
|
|
97
98
|
not the whole file and not a diff. The harness splices it into `{{TARGET_FILENAME}}` and shows the operator
|
|
98
|
-
the full resulting diff for confirmation.
|
|
99
|
+
the full resulting diff for confirmation. When nothing needs to change, write the section's current body
|
|
100
|
+
unchanged — or an empty file when the section does not exist yet.
|
|
99
101
|
|
|
100
102
|
Write to that output path only; do not modify `{{TARGET_FILENAME}}` in the repository — the harness writes the
|
|
101
103
|
file itself after confirmation. Make no other edits to the repository. Emit no prose commentary outside the
|
|
@@ -10,39 +10,17 @@ render a verdict.
|
|
|
10
10
|
|
|
11
11
|
{{FLOOR_RUBRIC_SECTION}}
|
|
12
12
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
13
|
+
{{EXTRA_DIMENSIONS_SECTION}}
|
|
14
|
+
|
|
15
|
+
Dimensions the planner attached are evaluated with the same binary pass/fail logic. The rubric from
|
|
16
|
+
`<task_specification>` is the authority — grade against it, not against your own quality judgment.
|
|
16
17
|
|
|
17
18
|
{{EVALUATOR_FAILURE_MODES}}
|
|
18
19
|
|
|
19
|
-
**Verdict values — `passed`, `failed`, `malformed`:**
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
(a verify command that will not run, an unreadable working tree) blocked the rest, so emitting `passed` or
|
|
24
|
-
`failed` would be a guess. `malformed` means "no verdict yet," not "slightly unsure."
|
|
25
|
-
|
|
26
|
-
Legitimate `malformed` triggers: the verify command dies without readable output; the working tree is
|
|
27
|
-
unreadable or the mounted directory is missing required files; environment setup is broken such that no
|
|
28
|
-
criterion command can run. Uncertainty about how to interpret a criterion isn't `malformed` — name the
|
|
29
|
-
failing criterion and emit `failed` with a critique.
|
|
30
|
-
|
|
31
|
-
When that FAIL stems from criterion ambiguity — two competent reviewers could reasonably disagree about what
|
|
32
|
-
the criterion requires — rather than a demonstrable defect, prefix the critique bullet with `[spec-ambiguity]`
|
|
33
|
-
and state the interpretation you graded against, so the planner can tighten the criterion instead of treating
|
|
34
|
-
it as a code defect.
|
|
35
|
-
|
|
36
|
-
When you emit `malformed`, the harness neither marks the work done nor blocks the task — it retries the
|
|
37
|
-
attempt: the SAME model gets a fresh attempt while the attempt budget remains, and only when the budget is
|
|
38
|
-
exhausted does the round settle with a warning. So `malformed` is honest and recoverable. Do not avoid it by
|
|
39
|
-
forcing a `passed` you cannot support — a false `passed` ships a bug; a `malformed` just costs one more attempt.
|
|
40
|
-
Conversely, do not reach for `malformed` to dodge a clear `failed`: if you can name a concrete failing
|
|
41
|
-
criterion, the verdict is `failed` with a critique, never `malformed`.
|
|
42
|
-
|
|
43
|
-
A terminal `passed` or `failed` verdict must grade each dimension in the rubric above with a finding — a
|
|
44
|
-
verdict missing a floor dimension is rejected by the harness and re-requested. `malformed` is exempt from
|
|
45
|
-
that coverage requirement.
|
|
20
|
+
**Verdict values — `passed`, `failed`, `malformed`:** almost every round ends in `passed` or `failed`;
|
|
21
|
+
`malformed` means "no verdict yet", not "slightly unsure", and the blocked-check rules in `<grading_rules>`
|
|
22
|
+
say when it applies. A false `passed` ships a bug; a false `failed` costs one generator round — but a FAIL
|
|
23
|
+
still needs a concrete observation.
|
|
46
24
|
</role>
|
|
47
25
|
|
|
48
26
|
{{HARNESS_CONTEXT}}
|
|
@@ -51,10 +29,10 @@ that coverage requirement.
|
|
|
51
29
|
|
|
52
30
|
<goal>
|
|
53
31
|
Produce one `evaluation` signal in `signals.json` under the harness output directory — `status: "passed"`
|
|
54
|
-
only when every floor dimension
|
|
32
|
+
only when every floor dimension and every task-specific dimension passes with concrete evidence;
|
|
55
33
|
`status: "failed"` otherwise with a critique the generator can act on. The exact output path is in the
|
|
56
|
-
output contract section at the bottom of this prompt. You may additionally emit `learning` signals
|
|
57
|
-
durable insights discovered while grading; the `evaluation` signal remains exactly one and mandatory.
|
|
34
|
+
output contract section at the bottom of this prompt. You may additionally emit `learning` or `note` signals
|
|
35
|
+
for durable insights discovered while grading; the `evaluation` signal remains exactly one and mandatory.
|
|
58
36
|
</goal>
|
|
59
37
|
|
|
60
38
|
<success_criteria>
|
|
@@ -69,8 +47,7 @@ durable insights discovered while grading; the `evaluation` signal remains exact
|
|
|
69
47
|
per-criterion checklist, not only prose. This is in ADDITION to the floor `dimensions`, not a
|
|
70
48
|
replacement.
|
|
71
49
|
- A FAIL on any dimension or criterion sets `status: "failed"`.
|
|
72
|
-
- The critique (when `status: "failed"`)
|
|
73
|
-
`<grading_rules>`.
|
|
50
|
+
- The critique (when `status: "failed"`) lists each failed item in the format defined in `<grading_rules>`.
|
|
74
51
|
- Signal written to `<outputDir>/signals.json` — no other files written, except an evidence overflow log
|
|
75
52
|
outside the repository when Phase 2 requires one.
|
|
76
53
|
|
|
@@ -99,9 +76,9 @@ round and never carry a prior PASS forward without your own observation.
|
|
|
99
76
|
|
|
100
77
|
<evaluation_discipline>
|
|
101
78
|
Before writing `signals.json`, work through each acceptance criterion and each floor dimension
|
|
102
|
-
explicitly.
|
|
103
|
-
|
|
104
|
-
|
|
79
|
+
explicitly. Grade each criterion and floor dimension as you gather its evidence rather than deciding them
|
|
80
|
+
all at the end; the `criteria` array and dimension findings are where each verdict lands, and
|
|
81
|
+
`signals.json` is written last.
|
|
105
82
|
</evaluation_discipline>
|
|
106
83
|
|
|
107
84
|
<inputs>
|
|
@@ -118,7 +95,7 @@ end. The final `signals.json` is the only machine-readable output and must come
|
|
|
118
95
|
<constraints>
|
|
119
96
|
- Read files and run shell commands. Do not write, edit, or delete any file except `signals.json` in the
|
|
120
97
|
harness-mounted output directory — and, only when a command's output overflows the evidence bound in
|
|
121
|
-
Phase 2, one overflow log in
|
|
98
|
+
Phase 2, one overflow log in that same output directory (never inside the repository).
|
|
122
99
|
- Do not run `git stash`, `git add`, or `git commit` — those are write operations.
|
|
123
100
|
- Do not run setup or migration commands — your session is read-only except for `signals.json`.
|
|
124
101
|
- The working tree is expected to be dirty: the harness commits the generator's output after this evaluator
|
|
@@ -154,9 +131,10 @@ Run deterministic checks first — they are authoritative and cheap.
|
|
|
154
131
|
|
|
155
132
|
{{PARALLEL_TOOL_CALLS}}
|
|
156
133
|
|
|
157
|
-
1. **Run each `auto` criterion's command** from `<task_specification>` directly and record the
|
|
158
|
-
output lines for each (bounded
|
|
159
|
-
|
|
134
|
+
1. **Run each `auto` criterion's command** from `<task_specification>` directly and record the exit code and
|
|
135
|
+
decisive output lines for each (bounded by the evidence bound in Phase 2). The verify-script rule in `<grading_rules>`
|
|
136
|
+
governs `<verify_script>`. If any criterion command fails, the implementation fails for that criterion
|
|
137
|
+
regardless of how clean the code looks. If a command's result looks flaky — it disagrees with what the code plainly does — re-run it
|
|
160
138
|
once; if the two runs disagree, record the inconsistency itself as evidence, never a clean PASS. Do not
|
|
161
139
|
stop here — continue grading all criteria so the generator receives a full critique.
|
|
162
140
|
2. **Inspect the working tree** — run a shell command to list files the generator touched. The tree is
|
|
@@ -170,8 +148,8 @@ Run deterministic checks first — they are authoritative and cheap.
|
|
|
170
148
|
|
|
171
149
|
For every criterion in the contract:
|
|
172
150
|
|
|
173
|
-
- **`auto` criteria** —
|
|
174
|
-
|
|
151
|
+
- **`auto` criteria** — grade from your Phase 1 run — cite its exit code and decisive lines in
|
|
152
|
+
`executionEvidence`; re-run only under the flaky-result rule in Phase 1. PASS only when the command exits 0 and the assertion holds; FAIL otherwise.
|
|
175
153
|
- **`manual` criteria** — when the changed behaviour is runnable (a command, a script, a service
|
|
176
154
|
endpoint, or a test), execute the changed path yourself and cite the observed output as evidence;
|
|
177
155
|
reading the diff or a green verify script alone does not substitute for that observation. Otherwise
|
|
@@ -181,9 +159,9 @@ For every criterion in the contract:
|
|
|
181
159
|
{{EVIDENCE_BOUND}}
|
|
182
160
|
|
|
183
161
|
Grade each criterion PASS or FAIL — no middle ground. Any single criterion FAIL forces `status: "failed"`.
|
|
184
|
-
A criterion you were blocked from executing follows the
|
|
162
|
+
A criterion you were blocked from executing follows the blocked-checks rule in `<grading_rules>`.
|
|
185
163
|
|
|
186
|
-
Record each criterion's verdict
|
|
164
|
+
Record each criterion's verdict structurally in the `evaluation` signal's `criteria` array — one entry
|
|
187
165
|
per criterion with its `id`, a `passed` boolean, and a one-line `evidence` citation. This is the same
|
|
188
166
|
grading you just did in prose; the array carries it as data so the harness can persist a durable
|
|
189
167
|
per-criterion checklist across rounds. Grade every criterion — including the ones you could not assess,
|
|
@@ -214,8 +192,9 @@ observation — file path, line number, function name, tool output, or quoted sn
|
|
|
214
192
|
record the response.
|
|
215
193
|
- **E2E or smoke suite**: run it when declared in `<project_tooling>` and confirm it reaches
|
|
216
194
|
the changed behaviour path.
|
|
217
|
-
|
|
218
|
-
|
|
195
|
+
|
|
196
|
+
Cite the run command and verbatim observation as evidence in the Correctness dimension finding. Absence
|
|
197
|
+
of a run observation when a run-path was declared is a Completeness failure.
|
|
219
198
|
|
|
220
199
|
**When `<project_tooling>` carries no runnable-product capability** (a library, a pure type or
|
|
221
200
|
schema package, or only static analysis tooling listed):
|
|
@@ -232,17 +211,15 @@ or output evidence you gathered in Phases 1–3. Grade `applicable: false` only
|
|
|
232
211
|
marks exempt (Robustness on a change that touches no error/failure path) and state the concrete reason in
|
|
233
212
|
`finding` rather than fabricating a pass or fail.
|
|
234
213
|
|
|
235
|
-
{{EXTRA_DIMENSIONS_SECTION}}
|
|
236
|
-
|
|
237
214
|
### Before rendering the verdict
|
|
238
215
|
|
|
239
216
|
Answer both questions honestly:
|
|
240
217
|
|
|
241
|
-
1. Did
|
|
242
|
-
(If the task has no `auto` criteria, did you run the verify script as the fallback?)
|
|
243
|
-
|
|
244
|
-
2. Can you name a specific observation for each dimension
|
|
245
|
-
emit, point to a concrete piece of evidence. If not,
|
|
218
|
+
1. Did every `auto` criterion either run, with its output recorded per the evidence bound, or carry an
|
|
219
|
+
UNVERIFIED reason? (If the task has no `auto` criteria, did you run the verify script as the fallback?)
|
|
220
|
+
Apply the blocked-checks rule in `<grading_rules>` to anything that did not run.
|
|
221
|
+
2. Can you name a specific observation for each dimension and each criterion? For every PASS you are about to
|
|
222
|
+
emit, point to a concrete piece of evidence. If not, Completeness fails.
|
|
246
223
|
|
|
247
224
|
A false PASS is worse than a false FAIL. A false FAIL costs one extra generator round; a false PASS ships a
|
|
248
225
|
bug. This check exists because the evaluator is the last line of defence against silent-pass regressions.
|
|
@@ -267,7 +244,7 @@ Phase 2:
|
|
|
267
244
|
- C2: `src/routes/exports.ts:42` returns 400 with `{ error: "invalid date" }` matching the project's error
|
|
268
245
|
format at `src/lib/errors.ts:8` — PASS.
|
|
269
246
|
|
|
270
|
-
Phase 3: `src/routes/exports.ts:12` validates via the project's shared
|
|
247
|
+
Phase 3: `src/routes/exports.ts:12` validates via the project's shared validation schema before reaching the
|
|
271
248
|
database. Sibling routes at `src/routes/imports.ts` use the same pattern — Consistency PASS.
|
|
272
249
|
|
|
273
250
|
Phase 4 dimensions:
|
|
@@ -275,7 +252,7 @@ Phase 4 dimensions:
|
|
|
275
252
|
- Correctness — PASS — C1 exited 0 (12/12 green); C2 returns 400 at `src/routes/exports.ts:42`.
|
|
276
253
|
- Completeness — PASS — schema, controller, and tests all implemented per steps; one TODO comment unrelated
|
|
277
254
|
to this task's criteria.
|
|
278
|
-
- Safety — PASS — input validated via shared
|
|
255
|
+
- Safety — PASS — input validated via shared validation schema at `src/routes/exports.ts:12` before DB access.
|
|
279
256
|
- Consistency — PASS — follows existing endpoint patterns in `src/routes/`; uses the shared error format.
|
|
280
257
|
- Robustness — N/A — the change only adds input validation with a standard 400 response; it introduces no
|
|
281
258
|
error/failure-recovery path (retries, rollback, degradation) beyond what Correctness already covers.
|
|
@@ -306,7 +283,7 @@ Signals:
|
|
|
306
283
|
{
|
|
307
284
|
"dimension": "safety",
|
|
308
285
|
"passed": true,
|
|
309
|
-
"finding": "input validated via shared
|
|
286
|
+
"finding": "input validated via shared validation schema at src/routes/exports.ts:12 before DB access"
|
|
310
287
|
},
|
|
311
288
|
{
|
|
312
289
|
"dimension": "consistency",
|
|
@@ -347,7 +324,7 @@ Phase 1: ran C1's test command directly — exit 0, 8 tests green, recorded verb
|
|
|
347
324
|
Phase 2:
|
|
348
325
|
|
|
349
326
|
- C1: test command exited 0, 8 tests green — PASS.
|
|
350
|
-
- C2: `src/controllers/users.ts:47`
|
|
327
|
+
- C2: `src/controllers/users.ts:47` parses `page` without validating it — NaN propagates into the
|
|
351
328
|
query, which throws an unhandled exception returning 500 — FAIL.
|
|
352
329
|
|
|
353
330
|
Phase 3: `src/repositories/users.ts:23` interpolates `query` directly into a SQL string via template
|
|
@@ -362,21 +339,21 @@ Phase 4 dimensions:
|
|
|
362
339
|
- Safety — FAIL — `src/repositories/users.ts:23`: SQL injection via unparameterised template literal.
|
|
363
340
|
Sibling `src/repositories/posts.ts:15` shows the correct pattern.
|
|
364
341
|
- Consistency — PASS — controller structure follows existing patterns; pagination helper used correctly.
|
|
365
|
-
- Robustness — FAIL — `src/controllers/users.ts:47` lets the `NaN` from `
|
|
342
|
+
- Robustness — FAIL — `src/controllers/users.ts:47` lets the `NaN` from parsing `page` propagate
|
|
366
343
|
uncaught into the query layer instead of degrading gracefully (e.g. rejecting with a 400 before the
|
|
367
344
|
query runs); the same defect that breaks Correctness also breaks graceful error handling here.
|
|
368
345
|
|
|
369
346
|
Verdict: `status: "failed"`, critique:
|
|
370
347
|
|
|
371
|
-
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
-
|
|
375
|
-
|
|
376
|
-
`src/repositories/users.ts:23`.
|
|
377
|
-
-
|
|
378
|
-
|
|
379
|
-
|
|
348
|
+
- [Correctness · C2] `src/controllers/users.ts:47` parses `page` without validating it, so non-numeric input
|
|
349
|
+
yields `NaN` and an unhandled exception (500); it should validate `page` first and return 400; look at
|
|
350
|
+
`src/controllers/users.ts:47`.
|
|
351
|
+
- [Safety] `src/repositories/users.ts:23` interpolates user input into SQL (`WHERE name LIKE '%${query}%'`);
|
|
352
|
+
it should use a parameterised query like `src/repositories/posts.ts:15`; look at
|
|
353
|
+
`src/repositories/users.ts:23`.
|
|
354
|
+
- [Robustness] `src/controllers/users.ts:47` lets an invalid `page` value propagate as an uncaught exception
|
|
355
|
+
instead of a handled 400; it should reject the value before the query runs; look at
|
|
356
|
+
`src/controllers/users.ts:47`.
|
|
380
357
|
|
|
381
358
|
Signals:
|
|
382
359
|
|
|
@@ -423,7 +400,7 @@ Signals:
|
|
|
423
400
|
"evidence": "src/controllers/users.ts:47 returns 500 on non-numeric page (expected 400)"
|
|
424
401
|
}
|
|
425
402
|
],
|
|
426
|
-
"critique": "[Correctness · C2]
|
|
403
|
+
"critique": "- [Correctness · C2] src/controllers/users.ts:47 parses page without validating it, so non-numeric input yields NaN and a 500; it should validate page first and return 400; look at src/controllers/users.ts:47.\n- [Safety] src/repositories/users.ts:23 interpolates user input into SQL (WHERE name LIKE '%${query}%'); it should use a parameterised query like src/repositories/posts.ts:15; look at src/repositories/users.ts:23.\n- [Robustness] src/controllers/users.ts:47 lets an invalid page value propagate as an uncaught exception instead of a handled 400; it should reject the value before the query runs; look at src/controllers/users.ts:47.",
|
|
427
404
|
"timestamp": "2026-01-01T00:00:00.000Z"
|
|
428
405
|
}
|
|
429
406
|
]
|
|
@@ -434,7 +411,7 @@ Signals:
|
|
|
434
411
|
|
|
435
412
|
<example id="3" label="FAIL — verify passes; round fails because a criterion lacks evidence (anti-rubber-stamp)">
|
|
436
413
|
|
|
437
|
-
Task: "Migrate auth middleware to the new session store"
|
|
414
|
+
Task: "Migrate auth middleware to the new session store" (a Python service)
|
|
438
415
|
|
|
439
416
|
Criteria:
|
|
440
417
|
|
|
@@ -449,36 +426,36 @@ Phase 2:
|
|
|
449
426
|
|
|
450
427
|
- C1: exited 0, 34 tests green — PASS.
|
|
451
428
|
- C2: searched the codebase for old session-cookie key names — zero references found — PASS.
|
|
452
|
-
- C3: searched for the TTL configuration path — no environment
|
|
453
|
-
hardcoded as `3600` at `
|
|
429
|
+
- C3: searched for the TTL configuration path — no environment read, no config key, the value is
|
|
430
|
+
hardcoded as `3600` at `app/middleware/session.py:18` — FAIL.
|
|
454
431
|
|
|
455
|
-
Phase 3: `
|
|
456
|
-
config service.
|
|
432
|
+
Phase 3: `app/middleware/session.py:18` shows `TTL = 3600` — no `os.environ` lookup or any config service.
|
|
457
433
|
|
|
458
434
|
Phase 4 dimensions:
|
|
459
435
|
|
|
460
|
-
- Correctness — FAIL — C3: TTL is hardcoded at `
|
|
461
|
-
|
|
436
|
+
- Correctness — FAIL — C3: TTL is hardcoded at `app/middleware/session.py:18`; no environment read found in
|
|
437
|
+
the module or its imports.
|
|
462
438
|
- Completeness — FAIL — C3 has no evidence of implementation; step 3 ("expose TTL via env var") has no
|
|
463
439
|
corresponding code.
|
|
464
|
-
- Safety — PASS — new session store uses the project's standard signing key from `
|
|
465
|
-
- Consistency — PASS — middleware structure matches `
|
|
466
|
-
pattern in `
|
|
467
|
-
- Robustness — PASS — `
|
|
440
|
+
- Safety — PASS — new session store uses the project's standard signing key from `app/config/secrets.py`.
|
|
441
|
+
- Consistency — PASS — middleware structure matches `app/middleware/csrf.py`; config access follows the
|
|
442
|
+
pattern in `app/middleware/rate_limit.py`.
|
|
443
|
+
- Robustness — PASS — `app/middleware/session.py:31` catches store-lookup failures and falls back to
|
|
468
444
|
issuing a fresh anonymous session, matching the retry/fallback pattern already used in
|
|
469
|
-
`
|
|
445
|
+
`app/middleware/csrf.py`.
|
|
470
446
|
|
|
471
447
|
Note: C1's test command passed. This round still fails because C3 is unimplemented — a passing test
|
|
472
448
|
command does not confirm TTL configurability.
|
|
473
449
|
|
|
474
450
|
Verdict: `status: "failed"`, critique:
|
|
475
451
|
|
|
476
|
-
-
|
|
477
|
-
|
|
478
|
-
`
|
|
479
|
-
-
|
|
480
|
-
`
|
|
481
|
-
|
|
452
|
+
- [Correctness · C3] `app/middleware/session.py:18` hardcodes `TTL = 3600` with no environment read; it should
|
|
453
|
+
read the TTL from an environment variable (e.g. `SESSION_TTL_SECONDS`) with a fallback default; look at
|
|
454
|
+
`app/middleware/session.py:18`.
|
|
455
|
+
- [Completeness · C3] step 3 ("expose TTL via env var") has no implementation — no environment read in
|
|
456
|
+
`app/middleware/session.py` or its imports; it should be implemented before the task is complete; look at
|
|
457
|
+
`app/middleware/session.py` and its imports.
|
|
458
|
+
|
|
482
459
|
</example>
|
|
483
460
|
|
|
484
461
|
<example id="4" label="FAIL — clean tree means no work was done this round; grade failed, not malformed">
|
|
@@ -489,7 +466,8 @@ Situation: the working tree is clean — no uncommitted changes visible. The ver
|
|
|
489
466
|
generator's prior commit message claims the work is done, but the harness has not committed for this round
|
|
490
467
|
yet (dirty-tree is the expected state; clean-tree means the generator wrote nothing this round).
|
|
491
468
|
|
|
492
|
-
Phase 1: shell inspection shows no uncommitted changes. The diff is empty
|
|
469
|
+
Phase 1: shell inspection shows no uncommitted changes. The diff is empty, and
|
|
470
|
+
`src/payments/gateway.ts` still wraps its calls in the old hand-rolled retry loop.
|
|
493
471
|
|
|
494
472
|
Phase 2: C1 auto criterion — test command exits 0 but this only confirms existing tests pass.
|
|
495
473
|
|
|
@@ -498,10 +476,11 @@ were executed this round.
|
|
|
498
476
|
|
|
499
477
|
Verdict: `status: "failed"`, critique:
|
|
500
478
|
|
|
501
|
-
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
479
|
+
- [Completeness] the working tree is clean — no uncommitted changes visible, and
|
|
480
|
+
`src/payments/gateway.ts` still uses the old hand-rolled retry loop, so the generator produced no output
|
|
481
|
+
this round; it should execute the declared task steps and leave the resulting changes uncommitted in the
|
|
482
|
+
working tree so the next evaluator round has a diff to review; look at `src/payments/gateway.ts`.
|
|
483
|
+
|
|
505
484
|
</example>
|
|
506
485
|
|
|
507
486
|
</examples>
|