@sreetej510/pi-shipd-checks 0.12.5 → 0.13.0

package/README.md

@@ -4,7 +4,7 @@ A [pi](https://github.com/earendil-works/pi) coding agent extension for benchmar
 
 ## What it does
 
-For `/checks solve`:
+For `/checks --solve`:
 
 1. Snapshots only the current git `HEAD` into a throwaway temp directory; the source checkout is never modified.
 2. Sends the exact contents of `agent_prompt.md` as each solver's sole first user message. No prompt file, task Dockerfile, `test.patch`, `solution.patch`, or uncommitted test files are overlaid into a solver workspace. The Dockerfile is used only by the harness to provision the Fargate environment.
@@ -14,6 +14,14 @@ For `/checks solve`:
 6. Runs the Claude CLI solvers inside one AWS ECS Fargate Spot task. A Spot interruption retries the task and resumes completed solver indexes. A local `claude setup-token` file is transferred through a private, server-side-encrypted S3 object and supplied to the Claude CLI through `CLAUDE_CODE_OAUTH_TOKEN`. A root-owned shell wrapper strips credential variables and drops every solver Bash command to its own unprivileged UID; the token is not available to those commands. The temporary token and run objects are deleted after the task stops.
 7. Posts a chat summary and logs results in `.pi/shipd-checks/<runId>/`, including `agent_steps.jsonl` per solver, and `shipd_report.json` in your project root.
 
+For `/checks --autoreview` (a local replica of the hosted Auto Review, ported from the shipd-workflow rulebook; tests and solution only; no description grading, and `test.sh` is ignored):
+
+1. Snapshots git `HEAD` into a temp workspace as the pinned base (`repo/`), next to `review_assets/` holding `agent_prompt.md` (as problem.md), `test.patch` with its `test.sh` section removed, `solution.patch`, and standalone copies of new test files. Neither patch is applied.
+2. Runs read-only Claude Code CLI reviewers on this machine (`Read`, `Grep`, `Glob` only; your local `claude` login), several at a time, one read each: Local Test Quality per-file fairness batches (one per ~60 KB of test files), one coverage read (with judge-tests' discrimination rules folded in), the solution audit over nine lenses (paths, identity, ordering, variants, numeric, compose, metadata, lifecycle including the state lifecycle walk, behaviour), and per-unit Solution Quality on every `solution.patch` hunk (12 hunks per reviewer). A typical task needs about 6–8 sessions. Prompts carry hosted calibration distilled into general patterns (`src/autoreview-data.ts`), not quotes from past tasks.
+3. Runs synthesis (one session per 50 findings) over every finding with the hosted proof legs (failing case, wrong implementation, promise citation, load-bearing check, provenance, materiality, reference pass, exercised environment). A finding gates only when synthesis keeps it HIGH.
+4. Verdict: `BLOCK` (any gating HIGH), `REVIEW` (MEDIUM or unproven findings, a Not fair test, or a failed reviewer), `PASS`, or `INCOMPLETE`. Test Quality is `FAIL` when any test is Not fair (an assertion with no source counts). Counter-tests the reviewers write are included in the report but not executed, so the legs that need execution are decided from the files.
+5. Writes only the final report, `.pi/shipd-checks/<runId>-autoreview.md` (verdict, findings with synthesis reasoning, Test Quality results, reviewer status, and the counter-test patches inline), and merges a summary into `shipd_report.json` under `autoReview`. Reviewer prompts, streams and answers are not kept.
+
 The agent-callable `gap-finder` tool runs one exhaustive read-only agent over the prompt, repository, and tests. It
 reviews requirements sentence by sentence, performs its own evidence and fairness check, and returns confirmed
 behavioral test gaps in `details.testGaps`.
@@ -67,25 +75,27 @@ Shipd tokens; do not run overlapping invocations for the same challenge.
 
 ## Commands
 
-`/checks solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as the task-author reference patch. `/checks --config` is separate and configures the local Pi reviewer/analysis models and Claude solver settings. The gap-finder, solution-precheck, and prompt-humanizer tools are enabled per project with `/analyze:on`.
+`/checks --solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as the task-author reference patch. `/checks --config` is separate and configures the local Pi reviewer/analysis models and Claude solver settings. The gap-finder, solution-precheck, and prompt-humanizer tools are enabled per project with `/analyze:on`.
 
 | Command | Effect |
 |---|---|
-| `/checks` | Open a menu for settings or `solve` |
-| `/checks solve` | Run isolated Claude CLI solvers and base/new tests, then use a local Pi reviewer to assess test fairness and failed agents |
+| `/checks` | Open a menu for config, solve, or autoreview |
+| `/checks --solve` | Run isolated Claude CLI solvers and base/new tests, then use a local Pi reviewer to assess test fairness and failed agents |
+| `/checks --autoreview` | Run the local Claude CLI Auto Review of `test.patch` and `solution.patch` (requires `agent_prompt.md`, `test.patch`, `solution.patch`) |
 | `/checks --config` | Configure local Pi reviewer/analysis models, Solver model (Claude), Claude solver effort, timeout, and parallel agent count |
 | `/shipd:auth` | Open a browser for first-time Shipd sign-in and save the session |
 | `/shipd:link <url>` | Save the Shipd job link for the current chat session |
 | `/analyze:on` | Enable the gap-finder, solution-precheck, and prompt-humanizer tools for the current project |
 | `/analyze:off` | Disable the gap-finder, solution-precheck, and prompt-humanizer tools for the current project |
 
-**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task or local Pi review. A cancelled run skips final report writing.
+**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task, local Pi review, or local Auto Review reviewers. A cancelled run skips final report writing.
 
 ## Configuration
 
-`/checks --config` opens the row-based settings menu with two sections:
+`/checks --config` opens the row-based settings menu with three sections:
 
 - **Solver**: Solver model (Claude: `haiku`, `sonnet`, or `opus`), solver effort (`low`, `medium`, `high`, `xhigh`, or `max`), one-shot timeout (10–90 minutes), parallel agent count, and the local Pi reviewer model/thinking level. Solver effort defaults to `low` when unset and is passed explicitly to Claude. Diffs, agent output, `agent_steps.jsonl` traces, base/new logs, and the local reviewer assessment are saved under `.pi/shipd-checks/<runId>/` and summarized in `shipd_report.json`.
+- **Auto Review**: Claude reviewer model (`haiku`, `sonnet`, or `opus`; default `opus`), effort (default `high`), timeout per reviewer (10–90 minutes; default 40), and parallel reviewers (1–8; default 4). Stored under `autoReview` in `checks-config.json`.
 - **Analysis Tools**: timeout, model, and thinking level for the separate agent-callable gap-finder and solution-precheck tools. The timeout applies to each read-only agent. These are stored under `analyzeGap.timeoutMinutes`, `analyzeGap.provider`, `analyzeGap.modelId`, and `analyzeGap.thinkingLevel`. Fargate resources are selected automatically and are not configurable in this menu.
 
 For each Claude solver, `shipd_report.json` records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, `totalTokens` subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local Pi reviewer records session usage when the provider reports it. Per-agent, reviewer, and aggregate usage appears once under `solverTokenUsage`; `solverRunSummary` contains outcomes, aggregate test counts, and per-suite JUnit counts, while `solverReview` contains the assessment and reviewer model settings.
@@ -179,7 +189,7 @@ On-Demand fallback. Spot interruptions are retried according to `fargate.maxRetr
    profile and do not update adaptive sizing. Set `projectProfiles` to override solver-gap
    resources per repository: `{"C:/path/to/repo":"large"}`.
 
-5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set `solverGap.claudeOAuthTokenFile` in `checks-config.json` to that file's path (or set `CLAUDE_CODE_OAUTH_TOKEN_FILE`). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's `auth.json`. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in `/checks --config`, then run `/checks solve`. Projects need `Dockerfile`, `agent_prompt.md`, and `test.patch`, which should add `test.sh` with `--output_path <file> base` and `--output_path <file> new` modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.
+5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set `solverGap.claudeOAuthTokenFile` in `checks-config.json` to that file's path (or set `CLAUDE_CODE_OAUTH_TOKEN_FILE`). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's `auth.json`. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in `/checks --config`, then run `/checks --solve`. Projects need `Dockerfile`, `agent_prompt.md`, and `test.patch`, which should add `test.sh` with `--output_path <file> base` and `--output_path <file> new` modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.
 
 Use `/analyze:on` and `/analyze:off` to control the gap-finder, solution-precheck, and prompt-humanizer tools per project, like HPC. The
 enabled project list is stored alongside the other settings in `~/.pi/agent/checks-config.json`.