@sreetej510/pi-shipd-checks 0.13.0 → 0.13.1

package/README.md

@@ -10,7 +10,7 @@ For `/checks --solve`:
 2. Sends the exact contents of `agent_prompt.md` as each solver's sole first user message. No prompt file, task Dockerfile, `test.patch`, `solution.patch`, or uncommitted test files are overlaid into a solver workspace. The Dockerfile is used only by the harness to provision the Fargate environment.
 3. Runs several configurable (default 3) independent Claude Code CLI solver invocations in parallel, each with its own isolated clean-HEAD workspace and only the `Bash` tool enabled. Captures each solver's `stream-json` trace of assistant and tool events (not private chain-of-thought). There is no test-driven repair loop, and the test patch remains unavailable until that solver exits.
 4. After a solver finishes, the harness applies `test.patch` to its workspace, then runs `bash test.sh --output_path <file> base` and `bash test.sh --output_path <file> new` as an unprivileged user with a fresh writable `HOME` and cache. It captures both full logs.
-5. After all solvers exit and their tests run, the read-only local Pi reviewer receives an untouched clean-HEAD tree plus `agent_prompt.md`, all patch files, standalone copies of new test files, each solver's patch, and every solver's test logs. No patch is applied in the review workspace.
+5. After all solvers exit and their tests run, a read-only local Claude Code CLI reviewer (`Read`, `Grep`, `Glob` only; your local `claude` login, same wrapper as Auto Review) receives an untouched clean-HEAD tree plus `agent_prompt.md`, all patch files, standalone copies of new test files, each solver's patch, and every solver's test logs. No patch is applied in the review workspace.
 6. Runs the Claude CLI solvers inside one AWS ECS Fargate Spot task. A Spot interruption retries the task and resumes completed solver indexes. A local `claude setup-token` file is transferred through a private, server-side-encrypted S3 object and supplied to the Claude CLI through `CLAUDE_CODE_OAUTH_TOKEN`. A root-owned shell wrapper strips credential variables and drops every solver Bash command to its own unprivileged UID; the token is not available to those commands. The temporary token and run objects are deleted after the task stops.
 7. Posts a chat summary and logs results in `.pi/shipd-checks/<runId>/`, including `agent_steps.jsonl` per solver, and `shipd_report.json` in your project root.
 
@@ -75,30 +75,30 @@ Shipd tokens; do not run overlapping invocations for the same challenge.
 
 ## Commands
 
-`/checks --solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as the task-author reference patch. `/checks --config` is separate and configures the local Pi reviewer/analysis models and Claude solver settings. The gap-finder, solution-precheck, and prompt-humanizer tools are enabled per project with `/analyze:on`.
+`/checks --solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as our reference solution: it stays context only for the fairness review, and the reviewer also compares its non-test code logic with every solver's patch and lists, under "Suggested improvements to our solution", edge cases or logical errors a solver handles correctly that ours does not (no tests, docs, or style). `/checks --config` is separate and configures the Claude solver and reviewer settings and the analysis-tool model. The gap-finder, solution-precheck, and prompt-humanizer tools are enabled per project with `/analyze:on`.
 
 | Command | Effect |
 |---|---|
 | `/checks` | Open a menu for config, solve, or autoreview |
-| `/checks --solve` | Run isolated Claude CLI solvers and base/new tests, then use a local Pi reviewer to assess test fairness and failed agents |
+| `/checks --solve` | Run isolated Claude CLI solvers and base/new tests, then use a local Claude CLI reviewer to assess test fairness and failed agents |
 | `/checks --autoreview` | Run the local Claude CLI Auto Review of `test.patch` and `solution.patch` (requires `agent_prompt.md`, `test.patch`, `solution.patch`) |
-| `/checks --config` | Configure local Pi reviewer/analysis models, Solver model (Claude), Claude solver effort, timeout, and parallel agent count |
+| `/checks --config` | Configure Solver model (Claude), Claude solver effort, timeout, parallel agent count, the Claude reviewer, Auto Review, and the analysis-tool model |
 | `/shipd:auth` | Open a browser for first-time Shipd sign-in and save the session |
 | `/shipd:link <url>` | Save the Shipd job link for the current chat session |
 | `/analyze:on` | Enable the gap-finder, solution-precheck, and prompt-humanizer tools for the current project |
 | `/analyze:off` | Disable the gap-finder, solution-precheck, and prompt-humanizer tools for the current project |
 
-**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task, local Pi review, or local Auto Review reviewers. A cancelled run skips final report writing.
+**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task, local Claude review, or local Auto Review reviewers. A cancelled run skips final report writing.
 
 ## Configuration
 
 `/checks --config` opens the row-based settings menu with three sections:
 
-- **Solver**: Solver model (Claude: `haiku`, `sonnet`, or `opus`), solver effort (`low`, `medium`, `high`, `xhigh`, or `max`), one-shot timeout (10–90 minutes), parallel agent count, and the local Pi reviewer model/thinking level. Solver effort defaults to `low` when unset and is passed explicitly to Claude. Diffs, agent output, `agent_steps.jsonl` traces, base/new logs, and the local reviewer assessment are saved under `.pi/shipd-checks/<runId>/` and summarized in `shipd_report.json`.
+- **Solver**: Solver model (Claude: `haiku`, `sonnet`, or `opus`), solver effort (`low`, `medium`, `high`, `xhigh`, or `max`), one-shot timeout (10–90 minutes), parallel agent count, and the Claude reviewer model (default `opus`), effort (default `high`), and timeout (10–60 minutes; default 20), stored under `solverReview` in `checks-config.json`. Solver effort defaults to `low` when unset and is passed explicitly to Claude. Diffs, agent output, `agent_steps.jsonl` traces, base/new logs, and the local reviewer assessment are saved under `.pi/shipd-checks/<runId>/` and summarized in `shipd_report.json`.
 - **Auto Review**: Claude reviewer model (`haiku`, `sonnet`, or `opus`; default `opus`), effort (default `high`), timeout per reviewer (10–90 minutes; default 40), and parallel reviewers (1–8; default 4). Stored under `autoReview` in `checks-config.json`.
-- **Analysis Tools**: timeout, model, and thinking level for the separate agent-callable gap-finder and solution-precheck tools. The timeout applies to each read-only agent. These are stored under `analyzeGap.timeoutMinutes`, `analyzeGap.provider`, `analyzeGap.modelId`, and `analyzeGap.thinkingLevel`. Fargate resources are selected automatically and are not configurable in this menu.
+- **Analysis Tools**: timeout, model, and thinking level for the separate agent-callable gap-finder and solution-precheck tools. The timeout applies to each read-only agent. These are stored under `analyzeGap.timeoutMinutes`, `analyzeGap.provider`, `analyzeGap.modelId`, and `analyzeGap.thinkingLevel`. The analysis tools have no fallback model: they refuse to run until a model is set here. Fargate resources are selected automatically and are not configurable in this menu.
 
-For each Claude solver, `shipd_report.json` records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, `totalTokens` subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local Pi reviewer records session usage when the provider reports it. Per-agent, reviewer, and aggregate usage appears once under `solverTokenUsage`; `solverRunSummary` contains outcomes, aggregate test counts, and per-suite JUnit counts, while `solverReview` contains the assessment and reviewer model settings.
+For each Claude solver, `shipd_report.json` records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, `totalTokens` subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local Claude reviewer records the CLI-reported usage and cost. Per-agent, reviewer, and aggregate usage appears once under `solverTokenUsage`; `solverRunSummary` contains outcomes, aggregate test counts, and per-suite JUnit counts, while `solverReview` contains the assessment and reviewer model settings.
 
 The humanizer candidate counts are configured separately in `~/.pi/agent/checks-config.json`. `candidateCount` controls
 how many drafts are requested from CoreWeave (1–10); `topCount` controls the maximum returned (1–10, capped at
@@ -189,7 +189,7 @@ On-Demand fallback. Spot interruptions are retried according to `fargate.maxRetr
    profile and do not update adaptive sizing. Set `projectProfiles` to override solver-gap
    resources per repository: `{"C:/path/to/repo":"large"}`.
 
-5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set `solverGap.claudeOAuthTokenFile` in `checks-config.json` to that file's path (or set `CLAUDE_CODE_OAUTH_TOKEN_FILE`). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's `auth.json`. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in `/checks --config`, then run `/checks --solve`. Projects need `Dockerfile`, `agent_prompt.md`, and `test.patch`, which should add `test.sh` with `--output_path <file> base` and `--output_path <file> new` modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.
+5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set the top-level `claudeOAuthTokenFile` in `checks-config.json` to that file's path (or set `CLAUDE_CODE_OAUTH_TOKEN_FILE`). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's `auth.json`. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in `/checks --config`, then run `/checks --solve`. Projects need `Dockerfile`, `agent_prompt.md`, and `test.patch`, which should add `test.sh` with `--output_path <file> base` and `--output_path <file> new` modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.
 
 Use `/analyze:on` and `/analyze:off` to control the gap-finder, solution-precheck, and prompt-humanizer tools per project, like HPC. The
 enabled project list is stored alongside the other settings in `~/.pi/agent/checks-config.json`.
@@ -244,7 +244,7 @@ Or, for local development, point at the entry point directly:
 | `src/index.ts` | Extension entry point: message renderer, cancel shortcut, command registration |
 | `src/submit.ts` | `quality-check`: Playwright draft filling, sequential quality checks, scheduled polling, and report extraction |
 | `src/command.ts` | The `/checks` command: argument parsing, `--config` flow, run orchestration |
-| `src/agents.ts` | Spawns and races local Pi gap-finder, solution-precheck, and benchmark-review sessions |
+| `src/agents.ts` | Spawns and races local Pi gap-finder and solution-precheck sessions; runs the benchmark review through the local Claude CLI |
 | `src/solvergap.ts` | Benchmark artifact-directory constant |
 | `src/fargate-docker.ts` | Supported Dockerfile parsing for remote solver setup |
 | `src/fargate-runner.ts` | ECS Fargate Spot/S3 orchestration for Claude CLI benchmark runs, patch-prechecks, and detector tasks; retries and cleanup |