@jesdi/skills 0.0.46 → 0.0.47
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/implement-spec/README.md +3 -3
- package/skills/implement-spec/driver_fixloop.py +4 -4
- package/skills/implement-spec/driver_report.py +5 -5
- package/skills/implement-spec/driver_run.py +1 -1
- package/skills/implement-spec/prompts/full-review.md +2 -0
- package/skills/implement-spec/prompts/reviewer.md +2 -2
- package/skills/implement-spec/tests/test_fixloop.py +25 -3
- package/skills/implement-spec/tests/test_report.py +5 -5
- package/skills-manifest.json +3 -3
package/package.json
CHANGED
|
@@ -178,7 +178,7 @@ Every report has `status` (`DONE`, `DONE_WITH_CONCERNS`, `NEEDS_CONTEXT`, `BLOCK
|
|
|
178
178
|
|---|---|---|---|
|
|
179
179
|
| test-writer | `prompts/test-writer.md` | `locked` (test file paths), `red` (`test`, `line`) | The driver reads a test node id `<file>::<test>` in `locked` as its file. With DONE: one commit, one locked file and one red line or more. Each locked path is in a commit that the role made. Each `line` is in the log. A locked file of a merged ticket that the role changed is in `locked`. A correction (after `overrule-test`, or for a review finding) changes only the files that it lists in `locked` and the files that the driver restored before it. When the gate is red on the restore commit, a DONE correction lists a restored file in `locked`. |
|
|
180
180
|
| implementer | `prompts/implementer.md` | `green` | With DONE: one commit or more. |
|
|
181
|
-
| reviewer | `prompts/reviewer.md` | `findings` (`severity`, `file`, `line`, `title`, `evidence`, `kind`), `mutations` (`test`, `file`, `line`, `red`); `commits` is empty | In
|
|
181
|
+
| reviewer | `prompts/reviewer.md` | `findings` (`severity`, `file`, `line`, `title`, `evidence`, `kind`), `mutations` (`test`, `file`, `line`, `red`); `commits` is empty | In a full review (one with no fix range: round 1, and the review after a test-writer correction): `mutations` is required, with a proof for each locked test file (`test` is `<locked file>::<test name>`). Each `red` is in the log. |
|
|
182
182
|
| correctness | `prompts/correctness.md` | the same as the reviewer; `commits` is empty | `mutations` is optional. |
|
|
183
183
|
| fixer | `prompts/fixer.md` | `green`, `ticket` | With DONE: one commit or more. |
|
|
184
184
|
| final-reviewer | `prompts/final-reviewer.md` | `findings` | None. |
|
|
@@ -207,7 +207,7 @@ Answer with `--answer <question id>=<option id>` on the next run, or through the
|
|
|
207
207
|
| Check | Where it lives |
|
|
208
208
|
|---|---|
|
|
209
209
|
| Locked tests with a demonstrated red | `driver_report.py` `_proof_errors`: each red line of the test-writer report must be in its log. `driver_review.py` `locked_findings`: a locked file of the ticket that differs from the test-writer commit is a critical finding, before each review. `driver_review.py` `merged_changes`: a locked file of a merged ticket that the ticket changed is a critical `fixture` finding, before each review. `driver_merge.py` `refused`, called from `driver.py` `_checked_landing` under the merge lock: a locked file of a merged ticket that differs between the PR tip and the tree that is about to land refuses the landing, so a later ticket cannot change the locked test of an earlier one. `driver_report.py` `_merged_errors` rejects a test-writer report that changes such a file and does not list it. |
|
|
210
|
-
| One mutation proof for each locked test | `prompts/
|
|
210
|
+
| One mutation proof for each locked test | `prompts/full-review.md`, the note of each full review, names the locked test files that need a proof. `driver_report.py` `_locked_errors` and `_proof_errors` reject a full review with no proof for a locked test file, or with a red line that is not in the log. `driver_review.py` `follow` finds the proofs that a fix makes due again. |
|
|
211
211
|
| One independent correctness pass for each risky ticket | `driver_fixloop.py` `_review` adds the pass for a ticket with a **Risk** line. `driver_agents.py` `correctness_entry` picks the entry of another provider, and `load_models` stops the run when there is none or its CLI is not on PATH. |
|
|
212
212
|
| One full gate for each candidate | `driver_gate.py` `run`, called from `driver_run.py` `Run.gate`: one record for each commit the driver is about to trust. Green is exit 0 on a tree that is clean before and after, with HEAD not moved. |
|
|
213
213
|
| Post-merge check on the PR branch | `driver.py` `_checked_landing`: a merge commit is gated before the PR branch moves to it. A fast-forward of a gated tip uses the record of that tip. `driver_final.py` `_review_until_reviewed` gates the tip again after the tidy and after the fix batch. |
|
|
@@ -217,7 +217,7 @@ The driver enforces the mutation proof for each locked test file, not for each l
|
|
|
217
217
|
## Known limits
|
|
218
218
|
|
|
219
219
|
- **A `codex` agent cannot commit.** A role that commits (test-writer, implementer, fixer, conflict) on an `openai` entry runs with `--sandbox workspace-write`, and that sandbox cannot run `git commit`. In the default `models.json` this is the second implementer entry, so fix rounds 3 and 4 fail. What to do: give a `--models` file with no `openai` entry in the `implementer`, `test-writer`, `fixer` and `conflict` lists.
|
|
220
|
-
- **A `codex` reviewer cannot make a mutation proof.** A reviewer on an `openai` entry runs with `--sandbox read-only` and cannot make the temporary edit. The default `models.json` has such an entry first for the reviewer, and
|
|
220
|
+
- **A `codex` reviewer cannot make a mutation proof.** A reviewer on an `openai` entry runs with `--sandbox read-only` and cannot make the temporary edit. The default `models.json` has such an entry first for the reviewer, and each full review needs the proofs. What to do: give a `--models` file whose reviewer list starts with an `anthropic` entry.
|
|
221
221
|
- **The correctness pass reads a tree that the reviewer changes.** The two agents share one slot, and the reviewer makes temporary edits. `prompts/correctness.md` tells the pass to read from git at the reviewed commit and to run no tests. It reads only the spec, the ticket and the findings file from the working tree, because git can lack them: `.agent/` is ignored. That rule is the only guard: the driver cannot enforce it, so a pass that ignores it can report a false finding and start a fix round. What to do: when a correctness finding looks wrong, compare it with `git show <reviewed commit>:<file>` before you rule on it.
|
|
222
222
|
- **A ticket that is not merged stops the end of run.** While one ticket is BLOCKED, there is no tidy and no final review of the merged tickets.
|
|
223
223
|
- **A published PR branch keeps its history.** See "End of run".
|
|
@@ -66,12 +66,12 @@ def _run_reviewers(job: Job, cast: list[tuple[str, str]], locked: list[str]) ->
|
|
|
66
66
|
errors: dict[str, str] = {}
|
|
67
67
|
|
|
68
68
|
def call(role: str, entry: str) -> dict:
|
|
69
|
-
#
|
|
70
|
-
|
|
69
|
+
# A full review (no fix range) must give a mutation proof for each locked test file; the prompt names them.
|
|
70
|
+
full = not job.previous and role == "reviewer"
|
|
71
|
+
notes = ["rereview"] if job.previous else ["full-review"] if full else []
|
|
71
72
|
try:
|
|
72
73
|
return job.run.agent(role, job.ticket.number, job.slot, values, entry=entry, fix_round=job.round,
|
|
73
|
-
notes=
|
|
74
|
-
check={"round": 1, "locked": locked} if first_review else {},
|
|
74
|
+
notes=notes, check={"full": True, "locked": locked} if full else {},
|
|
75
75
|
alone=lambda: len(running) == 1)
|
|
76
76
|
except Blocked as blocked:
|
|
77
77
|
errors[role] = f"the {role}: {blocked}"
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
"""Report validation: validate(role, path, context) -> list of errors, empty when the report is accepted.
|
|
2
2
|
|
|
3
|
-
context: repo (slot path), branch, start (HEAD when the role started), log,
|
|
3
|
+
context: repo (slot path), branch, start (HEAD when the role started), log, full (a full review: no fix range), locked,
|
|
4
4
|
correction (a test-writer corrects locked tests), restored (locked files the driver put back before it),
|
|
5
5
|
restore_red (the gate is red on the commit of that restore),
|
|
6
6
|
owners (ticket numbers, for the final fix batch), merged (locked files of merged tickets, not its own), pr (tip)."""
|
|
@@ -93,14 +93,14 @@ def _proof_errors(report: dict, context: dict) -> list[str]:
|
|
|
93
93
|
errors = []
|
|
94
94
|
if lines and not all(_seen(line, Path(context["log"]).read_text(errors="replace")) for line in lines):
|
|
95
95
|
errors.append(RED)
|
|
96
|
-
if context.get("
|
|
97
|
-
errors.append("mutations: required in
|
|
96
|
+
if context.get("full") and "findings" in report and "mutations" not in report:
|
|
97
|
+
errors.append("mutations: required in a full review")
|
|
98
98
|
return errors
|
|
99
99
|
|
|
100
100
|
|
|
101
101
|
def _locked_errors(report: dict, context: dict) -> list[str]:
|
|
102
|
-
"""In
|
|
103
|
-
if context.get("
|
|
102
|
+
"""In a full review, each locked test file has a mutation proof (`mutations[].test` is `<file>::<test>`)."""
|
|
103
|
+
if not context.get("full") or "findings" not in report:
|
|
104
104
|
return []
|
|
105
105
|
tests = [proof["test"] for proof in report.get("mutations", [])]
|
|
106
106
|
return [f"mutations: no mutation proof for the locked test file {path}" for path in context.get("locked", [])
|
|
@@ -280,7 +280,7 @@ def _attempts(run: Run, role: str, number: str, slot: str, values: dict[str, str
|
|
|
280
280
|
folder.mkdir(parents=True, exist_ok=True)
|
|
281
281
|
report, log, prompt = (folder / f"{role}-{fix_round}{suffix}" for suffix in (".json", ".log", ".prompt.md"))
|
|
282
282
|
report_in_slot = os.path.relpath(report, slot) # the agent works in its slot
|
|
283
|
-
context = {"repo": slot, "branch": values["branch"], "start": values["start_sha"], "log": str(log),
|
|
283
|
+
context = {"repo": slot, "branch": values["branch"], "start": values["start_sha"], "log": str(log),
|
|
284
284
|
"template": template or role, **(check or {})}
|
|
285
285
|
argv = _command(run, role, entry, os.path.relpath(prompt, slot), report_in_slot)
|
|
286
286
|
text = _first_prompt(context["template"], values, notes, report_in_slot)
|
|
@@ -8,7 +8,7 @@ Locked files: {locked}
|
|
|
8
8
|
Test-writer report, when the ticket has a test-writer: {writer_report}
|
|
9
9
|
|
|
10
10
|
Check the diff against the ticket and the spec, then for quality. Check that each criterion has a locked test. Test each suspicion that the ticket names in its Suspicions line.
|
|
11
|
-
|
|
11
|
+
To make a mutation proof for a locked test, break the production line the test protects, run the test, print its failing output, then revert the edit. The end of this prompt names the proofs that this review needs. A test that stays green is a critical finding. Run again each mutation that the test-writer report names in `concerns`.
|
|
12
12
|
A mutation proof is keyed to its test and its production line: a later round repeats it only when that line changed.
|
|
13
13
|
In `mutations`, give for each proof the test as `<locked file>::<test name>`, the production file and line, and one failure line copied exactly from the output you printed.
|
|
14
14
|
Give each finding a severity (critical, important or minor), the file and line, a title, the evidence, and a kind: `fixture` when a locked test encodes a stale path, format or setup detail; `test` when a test is weak or missing, as a locked test that stays green when you break the line it protects; `code` for anything else. The test-writer corrects each `fixture` finding and each critical or important `test` finding on a locked file, and the fixer gets the rest.
|
|
@@ -26,7 +26,7 @@ Write one JSON report at {report}. When your sandbox is read-only, give the JSON
|
|
|
26
26
|
"files": {"type": "array", "items": {"type": "string"}},
|
|
27
27
|
"concerns": {"type": "array", "items": {"type": "string"}},
|
|
28
28
|
"findings": {"type": "array", "items": {"type": "object", "required": ["severity", "file", "line", "title", "evidence", "kind"], "properties": {"severity": {"enum": ["critical", "important", "minor"]}, "file": {"type": "string"}, "line": {"type": "integer"}, "title": {"type": "string", "minLength": 1}, "evidence": {"type": "string", "minLength": 1}, "kind": {"enum": ["code", "fixture", "test"]}}}},
|
|
29
|
-
"mutations": {"description": "Required in
|
|
29
|
+
"mutations": {"description": "Required in a full review: one proof or more for each locked test file, with `test` as `<locked file>::<test name>`. A review scoped to a fix gives the proofs that it names.", "type": "array", "items": {"type": "object", "required": ["test", "file", "line", "red"], "properties": {"test": {"type": "string", "minLength": 1}, "file": {"type": "string", "minLength": 1}, "line": {"type": "integer"}, "red": {"type": "string", "minLength": 1}}}}
|
|
30
30
|
}
|
|
31
31
|
}
|
|
32
32
|
```
|
|
@@ -506,16 +506,38 @@ def test_a_first_review_proves_each_locked_file_and_a_second_bad_report_blocks(t
|
|
|
506
506
|
assert "feat: 01 rows" not in rig.pr_subjects()
|
|
507
507
|
|
|
508
508
|
|
|
509
|
+
def test_a_full_review_after_a_correction_names_each_locked_file_that_needs_a_proof(tmp_path):
|
|
510
|
+
"""Issue #92: the full review after a correction in a later round must prove each locked file, not only round 1."""
|
|
511
|
+
two = ("tests/test_01.py", "tests/test_01b.py")
|
|
512
|
+
first = {**tw("01"), "files": {path: "def test_x():\n assert 0\n" for path in two}}
|
|
513
|
+
first["report"] = {**first["report"], "files": list(two), "locked": list(two)}
|
|
514
|
+
proofs = [dict(test=f"{path}::test_x", file="demo/export.py", line=1, red=f"FAILED {path}::test_x") for path in two]
|
|
515
|
+
both = lambda found=(): {"stdout": "".join(p["red"] + "\n" for p in proofs),
|
|
516
|
+
"report": {**review("01", found)["report"], "mutations": proofs}}
|
|
517
|
+
vacuous = finding("VACUOUS", "critical", kind="test", file="tests/test_01.py")
|
|
518
|
+
correction = {**first, "files": {"tests/test_01.py": "# stronger\n"}, "commit": "test: 01 stronger test"}
|
|
519
|
+
correction["report"] = {**first["report"], "files": ["tests/test_01.py"], "locked": ["tests/test_01.py"]}
|
|
520
|
+
rig = Inside(tmp_path).run(plan(**{"01/test-writer": [first, correction], "01/fixer": [fix("01", 1)],
|
|
521
|
+
"01/reviewer": [both([STUCK]), review("01", [vacuous], lines=()), both()]}))
|
|
522
|
+
assert rig.code == 0, rig.ledger()
|
|
523
|
+
order = [call["role"] for call in rig.calls("01")]
|
|
524
|
+
assert order == ["test-writer", "implementer", "reviewer", "fixer", "reviewer", "test-writer", "reviewer"]
|
|
525
|
+
due = "give one mutation proof or more for each of these locked test files: tests/test_01.py tests/test_01b.py\n"
|
|
526
|
+
initial, again, full = (call["prompt"] for call in rig.calls("01", "reviewer"))
|
|
527
|
+
assert due in initial and due in full, full
|
|
528
|
+
assert "In round 1" not in full and due not in again
|
|
529
|
+
|
|
530
|
+
|
|
509
531
|
def test_the_locked_file_rule_is_for_the_first_review_only(tmp_path):
|
|
510
532
|
report = dict(status="DONE", commits=[], files=[], concerns=[], findings=[], mutations=[])
|
|
511
533
|
path = tmp_path / "reviewer-0.json"
|
|
512
534
|
path.write_text(json.dumps(report))
|
|
513
535
|
context = {"repo": ".", "branch": "main", "start": "HEAD", "log": str(path), "locked": ["tests/a.py", "tests/b.py"]}
|
|
514
|
-
assert driver_report.validate("reviewer", str(path),
|
|
515
|
-
assert driver_report.validate("reviewer", str(path), {**context, "
|
|
536
|
+
assert driver_report.validate("reviewer", str(path), context) == []
|
|
537
|
+
assert driver_report.validate("reviewer", str(path), {**context, "full": True}) == [
|
|
516
538
|
"mutations: no mutation proof for the locked test file tests/a.py",
|
|
517
539
|
"mutations: no mutation proof for the locked test file tests/b.py"]
|
|
518
|
-
assert driver_report.validate("correctness", str(path),
|
|
540
|
+
assert driver_report.validate("correctness", str(path), context) == []
|
|
519
541
|
|
|
520
542
|
|
|
521
543
|
def test_a_timed_out_read_only_role_is_restored_only_when_no_other_agent_is_in_the_slot(tmp_path):
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
"""Report validation acceptance tests, through driver_report.validate only.
|
|
2
2
|
|
|
3
3
|
context is a dict: repo (slot path), branch, start (HEAD sha when the role started),
|
|
4
|
-
log (the agent's log file),
|
|
4
|
+
log (the agent's log file), full (a full review of the reviewer).
|
|
5
5
|
"""
|
|
6
6
|
import json
|
|
7
7
|
import subprocess
|
|
@@ -45,7 +45,7 @@ def world(tmp_path):
|
|
|
45
45
|
def check(role, report, **context):
|
|
46
46
|
path = tmp_path / f"{role}-0.json"
|
|
47
47
|
path.write_text(report if isinstance(report, str) else json.dumps(report))
|
|
48
|
-
ctx = {"repo": str(root), "branch": "ticket/01", "log": str(tmp_path / "agent.log")
|
|
48
|
+
ctx = {"repo": str(root), "branch": "ticket/01", "log": str(tmp_path / "agent.log")}
|
|
49
49
|
return validate(role, str(path), {**ctx, **context})
|
|
50
50
|
return shas, check, tmp_path
|
|
51
51
|
|
|
@@ -151,12 +151,12 @@ def test_red_line_absent_from_the_log_is_a_violation(world):
|
|
|
151
151
|
|
|
152
152
|
def test_reviewer_fields_and_mutation_proofs(world):
|
|
153
153
|
shas, check, _ = world
|
|
154
|
-
assert check("reviewer", reviewer(), start=shas["impl"],
|
|
154
|
+
assert check("reviewer", reviewer(), start=shas["impl"], full=True) == []
|
|
155
155
|
assert any("severity" in e for e in check("reviewer", reviewer(findings=[finding(severity="huge")]),
|
|
156
156
|
start=shas["impl"]))
|
|
157
157
|
assert any("kind" in e for e in check("reviewer", reviewer(findings=[finding(kind="style")]), start=shas["impl"]))
|
|
158
158
|
no_proof = reviewer()
|
|
159
159
|
del no_proof["mutations"]
|
|
160
|
-
assert any("mutations" in e for e in check("reviewer", no_proof, start=shas["impl"],
|
|
160
|
+
assert any("mutations" in e for e in check("reviewer", no_proof, start=shas["impl"], full=True))
|
|
161
161
|
fake = reviewer(mutations=[{"test": "t", "file": "demo/export.py", "line": 2, "red": "never ran"}])
|
|
162
|
-
assert any("not found in log" in e for e in check("reviewer", fake, start=shas["impl"],
|
|
162
|
+
assert any("not found in log" in e for e in check("reviewer", fake, start=shas["impl"], full=True))
|
package/skills-manifest.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schemaVersion": 1,
|
|
3
|
-
"generatedAt": "2026-10-17T15:
|
|
3
|
+
"generatedAt": "2026-10-17T15:52:14.213Z",
|
|
4
4
|
"skills": [
|
|
5
5
|
{
|
|
6
6
|
"name": "backlog",
|
|
@@ -46,8 +46,8 @@
|
|
|
46
46
|
},
|
|
47
47
|
{
|
|
48
48
|
"name": "implement-spec",
|
|
49
|
-
"version": "0.1.
|
|
50
|
-
"hash": "sha256-
|
|
49
|
+
"version": "0.1.23",
|
|
50
|
+
"hash": "sha256-c0a64ae43e2ba3c2e100b4d0a78085fdab558f8ee4eda01d7ae15eb677b8dbb5",
|
|
51
51
|
"description": "Implement a specification in code."
|
|
52
52
|
},
|
|
53
53
|
{
|