@jesdi/skills 0.0.50 → 0.0.51

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jesdi/skills",
-  "version": "0.0.50",
+  "version": "0.0.51",
   "description": "jesdi's agent skills (content package consumed by @jesdi/skills-cli)",
   "license": "MIT",
   "repository": {
@@ -29,7 +29,7 @@
   "scripts": {
     "test": "vitest run",
     "test:coverage": "vitest run --coverage --coverage.reportOnFailure",
-    "test:python": "mkdir -p .crap && python3 -m coverage run -m pytest skills/backlog skills/crap-gate -q && python3 -m coverage run -a -m pytest skills/implement-spec -q && python3 -m coverage json -o .crap/python.json",
+    "test:python": "mkdir -p .crap && python3 -m coverage erase && python3 -m coverage run -m pytest skills/backlog skills/crap-gate -q && python3 -m coverage run -a -m pytest skills/implement-spec -q && COVERAGE_SUBPROCESS_PATCH=subprocess python3 -m coverage run -m pytest skills/test-prune -q && python3 -m coverage combine --append -q && python3 -m coverage json -o .crap/python.json",
     "crap-gate": "sh skills/crap-gate/run.sh",
     "check": "sh scripts/check.sh",
     "generate-manifest": "tsx scripts/generate-manifest.ts"

package/skills/test-prune/SKILL.md

@@ -0,0 +1,156 @@
+---
+name: test-prune
+description: >-
+  Mutation-guided test pruning: a fixer proposes deleting redundant tests and strengthening
+  weak ones, an independent reviewer accepts or rejects each proposal, and prune.py applies
+  the accepted batch behind a cold mutation run and the ratchet. Use when the user says
+  "prune tests", "test-prune", "mutation test pruning" or "strengthen weak tests", or when a
+  scheduled job runs it with `--unattended`.
+---
+
+# test-prune: delete what the mutation ratchet proves redundant, strengthen what it proves weak
+
+You are the orchestrator. `prune.py` owns every git, make and file operation and every safety
+guard; the subagents only write JSON files or test commits. Each role is a fresh subagent with no
+context from the others: the reviewer reads the fixer's proposals, reasons included, and judges
+them against the code itself.
+
+**Publishing belongs to prune.py.** In both modes you never run `git push`, `git merge`,
+`gh pr create` or `gh pr merge` yourself, and neither does any subagent; `prune` and
+`strengthen-finish` push their own `chore/test-*` branch and open its PR, and nothing merges.
+
+The target repo provides the make targets `test`, `mutation`, `mutation-check`,
+`mutation-baseline`, `test-candidates`, `prune-gate` and `crap`, and the layout `backend/tests/`,
+`backend/mutation_baseline.json`. Every command:
+
+```bash
+python3 "<skill-dir>/prune.py" <command> [--state DIR] [--date YYYY-MM-DD]
+```
+
+Run it from inside the target repo. State lives in `<git dir>/test-prune/` (outside the working
+tree); each command reads what the previous one wrote there. Exit 0 is done (including nothing to
+do), 1 is a failed make target, gate or guard, 2 is bad input or an unmet precondition; the
+message names the offending ids or files. On exit 2 fix the input and rerun the same command; on
+exit 1 stop and report the output.
+
+Every command records itself in `<state>/status.json`. A CI job reads it after the headless
+session (which cannot set the job's exit code) and fails the job when its `exit` is not 0; an
+unattended run stays red there until the strengthen loop ends (`nothing to strengthen`, or
+`strengthen-finish` exit 0). End a headless session by reporting its `reason` as the final line.
+`--state` must be empty, new, or an earlier run's state dir.
+
+When a push succeeded but `gh pr create` failed (exit 1, the branch is on origin with no PR),
+open the PR by hand from `<state>/prune-pr.md` or `<state>/strengthen-pr.md` after fixing `gh`;
+rerunning the command refuses because the branch exists.
+
+**A run needs a checkout dedicated to it** for its whole duration: the CI checkout, or a worktree.
+If anything else edits a file there mid-run, the run stops with exit 1 rather than overwrite it,
+leaves the checkout as it is (HEAD may stay detached), names the files, and nothing is lost; keep
+or discard the change, then check out the base branch. Races beyond that check are out of scope.
+
+**Runtime.** Each cold `make mutation` takes about 30 minutes on a real repo, and a run makes
+several (collect, each safety-check run, each strengthen proof). Run long commands with a timeout of
+an hour or more, or in the background, and wait for them.
+
+## Modes
+
+- **Interactive** (default): nothing in the working tree changes before the human approves the
+  batch. Stop after `batch`, show its output, and run `apply` only on an explicit yes.
+- **Unattended** (`collect --unattended`, run headless by the weekly CI job): only protected
+  candidates, at most 10 deleted test ids per run (a parametrized test goes whole, every case
+  counted; one with more than 10 cases is skipped, the rest left for next week), one PR from
+  `chore/test-prune-<date>` and one from `chore/test-strengthen-<date>`, both from the base
+  commit. The base branch never moves. A merge is the human's approval, so the run ends with open
+  PRs.
+
+Two kinds of accepted proposal reach a human only: equivalent mutants (the skill writes no
+equivalent id into the baseline) and "strengthen this candidate with this assertion" verdicts
+(the skill applies no assertion to an existing candidate). `batch` prints both; unattended, every
+PR body and the job summary carry them.
+
+## Prune loop
+
+1. `collect [--unattended | --protected-only]`. Needs a clean tracked tree on a branch. Runs the
+   base checks, a cold mutation run and `make test-candidates`, then writes
+   `fixer-input.json`. When it prints `no candidates` / `no protected candidates`, the weak spots
+   may still need the fixer; continue.
+2. **Fixer.** Dispatch a subagent with `<state>/fixer-input.json`, read access to the repo, and
+   this brief: for every candidate decide `keep`, `delete` into one named test that contains it,
+   or `strengthen` with one named assertion; for every weak-spot group decide `strengthen`
+   (which test to add to or strengthen) or `equivalent` (with the argument why no test can tell
+   the mutant apart). A group with both kinds of mutant takes two entries, one `strengthen` and
+   one `equivalent`, each listing its own disjoint `mutant_ids`. Compare assertions, not coverage: a candidate whose assertions pin a value,
+   boundary or error the container never checks is `keep` or `strengthen`. It writes:
+
+   ```json
+   {"candidates": [{"test_id": "...", "verdict": "keep|delete|strengthen",
+                    "into": "<test id, delete only>", "assertion": "<strengthen only>", "reason": "..."}],
+    "weak_spots": [{"group": "...", "verdict": "strengthen|equivalent",
+                    "mutant_ids": ["<subset of the group's ids, omit for all>"],
+                    "test": "<test id or test file>", "reason": "..."}]}
+   ```
+
+   One entry per candidate, one per group (or the strengthen + equivalent pair), no others.
+3. `review-input --proposals <fixer file>` validates the file and writes
+   `<state>/review-input.json`: each proposal with the candidate's source and assertions and,
+   for a deletion, the containing test's source and assertions. A proposal's `id` is the
+   candidate's test id or the group; a split group's two proposals are `<group>:strengthen` and
+   `<group>:equivalent`.
+4. **Reviewer.** Dispatch a *different* fresh subagent with `<state>/review-input.json` and read
+   access to the repo. Brief: accept a deletion only when every assertion of the candidate is
+   implied by an assertion of the `into` test (same inputs or a superset, an equal or stricter
+   check); covering the same lines is not enough. Accept a strengthen proposal only when the
+   named assertion checks behaviour a user would notice. Accept an equivalent only with a
+   concrete argument that no input distinguishes the mutant. Every reason names the assertion or
+   input it rests on. It writes, one entry per proposal:
+
+   ```json
+   {"reviews": [{"id": "<proposal id>", "verdict": "accept|reject", "reason": "..."}]}
+   ```
+
+5. `batch --reviews <reviewer file>` writes `batch.json` and prints the accepted deletions (each
+   with its containing test and the reviewer's reason), strengthen items, equivalent proposals,
+   skipped deletions with the reason, and rejections. It edits nothing.
+   - Interactive: show that output to the human and stop. After approval, `apply` applies the
+     deletions, runs the safety check, restores every culprit and commits on the current branch.
+   - Unattended: `prune` applies the batch, runs the safety check, `make test`, commits, runs
+     `make prune-gate` and, only when it passes, pushes the branch and opens the PR.
+
+**Safety check** (inside `apply` and `prune`): a cold mutation run plus `make mutation-check`
+on the batch; every deletion that lets a module exceed its baseline is restored and reported.
+
+## Strengthen loop
+
+Runs after the prune loop, also in unattended mode, independently of the prune PR: run it even
+when `prune` exited 1 (the status keeps that failure, so the job still goes red).
+
+1. `strengthen-start` writes `<state>/strengthen-input.json` (each accepted weak-spot strengthen
+   item with the mutant ids it must kill) and prints the commit to build on; unattended it
+   detaches HEAD at the base commit. `nothing to strengthen` ends the loop.
+2. **Strengthen agent.** Dispatch a fresh subagent with `strengthen-input.json`. Brief: per item,
+   test-first, write the assertion or test that fails against each named mutant and passes on
+   the real code; touch only `backend/tests/`; make one or more commits per item, each ending
+   with the trailer line `Test-Prune-Item: <item id>`; leave the tree clean.
+3. `strengthen-verify` rejects commits without a known trailer or outside `backend/tests/`, then
+   proves each item on a cold run: every named id survived on the base and is killed on the
+   head. Unproven items are reverted and closed: a later commit for a closed item is refused. It
+   writes `<state>/strengthen-proof.json`, including the proven head. On failure HEAD stays
+   where the commits are (detached, unattended): fix and rerun. To abandon the loop instead,
+   `git checkout <base branch>`; the item commits stay reachable through `git reflog`.
+4. **Strengthen reviewer.** Dispatch a fresh subagent with `strengthen-proof.json` and the diff
+   of the proven items (`git log -p <start>..HEAD`). Brief: accept an item only when its test
+   asserts observable behaviour through the public interface, rather than restating the
+   implementation; one review per proven item, same schema as the prune reviewer.
+5. `strengthen-finish --reviews <file>` refuses a HEAD other than the proven one (rerun
+   `strengthen-verify` after any new commit); on any exit 2 HEAD stays at the proven head, so fix
+   the input and rerun it. It reverts rejected items, reruns the proof cold, runs
+   `make mutation-check`, `make test`, `make crap` and `make mutation-baseline`, refuses a
+   baseline that gained an equivalent id, and commits the lower baseline. Unattended it opens the
+   strengthen PR listing each changed test with the ids it now kills; interactive it leaves the
+   commits on the current branch for the human.
+
+## Report
+
+End with what the last command printed: deleted, restored, skipped and left-for-next-week tests,
+strengthened tests with their killed ids, the PR links, and the equivalent proposals. In CI the
+same summaries land in `$GITHUB_STEP_SUMMARY`.

package/skills/test-prune/prune.py

@@ -0,0 +1,202 @@
+#!/usr/bin/env python3
+"""test-prune: drive the fixer/reviewer prune loop and the strengthen loop over a target repo.
+
+collect → (fixer) → review-input → (reviewer) → batch → apply (interactive) | prune (unattended);
+strengthen-start → (strengthen agent) → strengthen-verify → (reviewer) → strengthen-finish.
+The script owns every git, make and file operation; the agents only write JSON or test commits.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+from prune_repo import BadInput, Fail, Repo, State, read_agent_json
+from prune_review import batch, collect, review_input
+from prune_runs import apply, prune
+from prune_strengthen import strengthen_finish, strengthen_start, strengthen_verify
+
+COMMANDS = {
+    "collect": collect, "review-input": review_input, "batch": batch, "apply": apply, "prune": prune,
+    "strengthen-start": strengthen_start, "strengthen-verify": strengthen_verify,
+    "strengthen-finish": strengthen_finish,
+}
+
+
+class Parser(argparse.ArgumentParser):
+    """An argument error raises BadInput (exit 2, recorded in the run status) instead of exiting."""
+
+    def error(self, message):
+        raise BadInput(f"{self.prog}: {message}")
+
+
+def _common():
+    common = Parser(add_help=False)
+    common.add_argument("--state", help="run state directory (default: <git dir>/test-prune)")
+    common.add_argument("--date", help="branch date YYYY-MM-DD (default: today, UTC)")
+    return common
+
+
+def build_parser():
+    common = _common()
+    parser = Parser(prog="prune.py", description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    collect_cmd = sub.add_parser("collect", parents=[common])
+    collect_cmd.add_argument("--unattended", action="store_true")
+    collect_cmd.add_argument("--protected-only", action="store_true")
+
+    sub.add_parser("review-input", parents=[common]).add_argument("--proposals", required=True)
+    sub.add_parser("batch", parents=[common]).add_argument("--reviews", required=True)
+    sub.add_parser("apply", parents=[common])
+    sub.add_parser("prune", parents=[common])
+    sub.add_parser("strengthen-start", parents=[common])
+    sub.add_parser("strengthen-verify", parents=[common])
+    sub.add_parser("strengthen-finish", parents=[common]).add_argument("--reviews", required=True)
+    return parser
+
+
+class Context:
+    """What every command gets: the repo, the run state, the branch date and its arguments."""
+
+    def __init__(self, args):
+        self.args = args
+        self.repo = Repo(Path.cwd())
+        self.state = State(Path(args.state).resolve() if args.state else self.repo.git_dir() / "test-prune")
+        self.date = args.date or datetime.now(timezone.utc).strftime("%Y-%m-%d")
+        self.finished = False  # the run is complete: the strengthen loop ended (set by its commands)
+
+    def validate(self) -> None:
+        if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", self.date):
+            raise BadInput(f"--date must be YYYY-MM-DD, got {self.date!r}")
+
+    def base(self, mode: str | None = None) -> dict:
+        base = self.state.load("base.json")
+        if mode and base["mode"] != mode:
+            raise BadInput(f"`{self.args.command}` runs only in {mode} mode; this run was collected in {base['mode']} mode")
+        return base
+
+    def reviews(self) -> list:
+        return read_agent_json(self.args.reviews, "reviews")
+
+    def proposals(self) -> dict:
+        return {"candidates": read_agent_json(self.args.proposals, "candidates"),
+                "weak_spots": read_agent_json(self.args.proposals, "weak_spots")}
+
+
+def _run(ctx: Context) -> tuple[int, str]:
+    """The command's exit code and the summary its status reason comes from. Staying off the base branch
+    (leave_on found changes the run did not make) is reported and fails a command that would pass."""
+    try:
+        ctx.validate()
+        code, summary = COMMANDS[ctx.args.command](ctx), "\n\n".join(ctx.state.summaries)
+    except BadInput as exc:
+        print(f"test-prune: {exc}", file=sys.stderr)
+        code, summary = 2, f"test-prune: {exc}"
+    except Fail as exc:
+        ctx.state.emit(f"test-prune: {exc}")
+        code, summary = 1, f"test-prune: {exc}"
+    except Exception as exc:  # a bug or an operational surprise still fails the run in status.json
+        summary = f"test-prune: unexpected {type(exc).__name__}: {exc}"
+        ctx.state.emit(summary)
+        code = 1
+    if ctx.repo.stranded:
+        ctx.state.emit(ctx.repo.stranded)
+        if code == 0:
+            code, summary = 1, ctx.repo.stranded
+    return code, summary
+
+
+RUN = "run"  # the marker entry an unattended run stays red on until it finishes
+INCOMPLETE = "test-prune: run incomplete: the strengthen loop has not finished"
+
+
+def _open(commands: list) -> list:
+    """The entries that still fail the run: every exit 1 but the run marker, and every exit 2 (or the run
+    marker) its command has not since passed. On exit 2 the session fixes the input and reruns the
+    command; the run marker passes when the strengthen loop ends."""
+    def clears(c: dict) -> bool:
+        return c["exit"] == 2 or c["command"] == RUN
+    return [c for i, c in enumerate(commands) if c["exit"] and not (clears(c) and any(
+        later["command"] == c["command"] and later["exit"] == 0 for later in commands[i + 1:]))]
+
+
+def _earlier(previous, command: str) -> list:
+    """The run's commands so far: none for `collect` (a new run) or a missing or unreadable status."""
+    commands = previous.get("commands") if isinstance(previous, dict) and command != "collect" else None
+    return commands if isinstance(commands, list) else []
+
+
+def run_status(previous, command: str, code: int, summary: str, extra=()) -> dict:
+    """The run's status, sticky across commands: the highest exit code still open and its reason (the
+    latest on a tie), plus every command in order, then the extra entries. `collect` starts a new run; a
+    missing or unreadable status starts fresh rather than failing the command."""
+    reason = next((line for line in summary.splitlines() if line.strip()), "")
+    commands = _earlier(previous, command) + [{"command": command, "exit": code, "reason": reason}]
+    ran_prune = any(c["command"] == "prune" for c in commands)
+    commands += [e for e in extra if e["exit"] or ran_prune]  # the run finishes only after prune ran
+    worst = max(reversed(_open(commands)), key=lambda c: c["exit"], default={"exit": 0, "reason": reason})
+    return {"exit": worst["exit"], "reason": worst["reason"], "commands": commands}
+
+
+def _record(state: State, command: str, code: int, summary: str, extra=()) -> None:
+    """Add the command to <state>/status.json, when the state dir is the run's (see State.owned)."""
+    if not state.owned():
+        return
+    path = state.path / "status.json"
+    try:
+        previous = json.loads(path.read_text()) if path.exists() else None
+    except (OSError, ValueError):
+        previous = None
+    state.save("status.json", run_status(previous, command, code, summary, extra))
+
+
+def _markers(ctx: Context) -> list:
+    """The run marker: opened by an unattended collect, passed when the strengthen loop ends after
+    prune ran (see run_status)."""
+    if ctx.args.command == "collect" and ctx.args.unattended:
+        return [{"command": RUN, "exit": 1, "reason": INCOMPLETE}]
+    return [{"command": RUN, "exit": 0, "reason": "run finished"}] if ctx.finished else []
+
+
+def _state_arg(argv) -> str | None:
+    """The --state value, read off argv without a parser (the arguments just failed to parse)."""
+    for i, arg in enumerate(argv):
+        if arg == "--state" and i + 1 < len(argv):
+            return argv[i + 1]
+        if arg.startswith("--state="):
+            return arg[len("--state="):]
+    return None
+
+
+def _record_bad_arguments(argv, exc: BadInput) -> None:
+    """An argument error is recorded as exit 2 when the command and the state dir can still be told."""
+    command = next((a for a in argv if a in COMMANDS), None)
+    state = _state_arg(argv)
+    try:
+        path = Path(state).resolve() if state else Repo(Path.cwd()).git_dir() / "test-prune"
+    except BadInput:
+        return
+    if command:
+        _record(State(path), command, 2, f"test-prune: {exc}")
+
+
+def main(argv=None):
+    argv = sys.argv[1:] if argv is None else argv
+    try:
+        args = build_parser().parse_args(argv)
+        ctx = Context(args)
+    except BadInput as exc:
+        print(f"test-prune: {exc}", file=sys.stderr)
+        _record_bad_arguments(argv, exc)
+        return 2
+    code, summary = _run(ctx)
+    _record(ctx.state, args.command, code, summary, _markers(ctx))
+    return code
+
+
+if __name__ == "__main__":
+    sys.exit(main())