@sreetej510/pi-shipd-checks 0.12.5 → 0.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +23 -13
- package/dist/index.js +356 -71
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -4,16 +4,24 @@ A [pi](https://github.com/earendil-works/pi) coding agent extension for benchmar
|
|
|
4
4
|
|
|
5
5
|
## What it does
|
|
6
6
|
|
|
7
|
-
For `/checks solve`:
|
|
7
|
+
For `/checks --solve`:
|
|
8
8
|
|
|
9
9
|
1. Snapshots only the current git `HEAD` into a throwaway temp directory; the source checkout is never modified.
|
|
10
10
|
2. Sends the exact contents of `agent_prompt.md` as each solver's sole first user message. No prompt file, task Dockerfile, `test.patch`, `solution.patch`, or uncommitted test files are overlaid into a solver workspace. The Dockerfile is used only by the harness to provision the Fargate environment.
|
|
11
11
|
3. Runs several configurable (default 3) independent Claude Code CLI solver invocations in parallel, each with its own isolated clean-HEAD workspace and only the `Bash` tool enabled. Captures each solver's `stream-json` trace of assistant and tool events (not private chain-of-thought). There is no test-driven repair loop, and the test patch remains unavailable until that solver exits.
|
|
12
12
|
4. After a solver finishes, the harness applies `test.patch` to its workspace, then runs `bash test.sh --output_path <file> base` and `bash test.sh --output_path <file> new` as an unprivileged user with a fresh writable `HOME` and cache. It captures both full logs.
|
|
13
|
-
5. After all solvers exit and their tests run,
|
|
13
|
+
5. After all solvers exit and their tests run, a read-only local Claude Code CLI reviewer (`Read`, `Grep`, `Glob` only; your local `claude` login, same wrapper as Auto Review) receives an untouched clean-HEAD tree plus `agent_prompt.md`, all patch files, standalone copies of new test files, each solver's patch, and every solver's test logs. No patch is applied in the review workspace.
|
|
14
14
|
6. Runs the Claude CLI solvers inside one AWS ECS Fargate Spot task. A Spot interruption retries the task and resumes completed solver indexes. A local `claude setup-token` file is transferred through a private, server-side-encrypted S3 object and supplied to the Claude CLI through `CLAUDE_CODE_OAUTH_TOKEN`. A root-owned shell wrapper strips credential variables and drops every solver Bash command to its own unprivileged UID; the token is not available to those commands. The temporary token and run objects are deleted after the task stops.
|
|
15
15
|
7. Posts a chat summary and logs results in `.pi/shipd-checks/<runId>/`, including `agent_steps.jsonl` per solver, and `shipd_report.json` in your project root.
|
|
16
16
|
|
|
17
|
+
For `/checks --autoreview` (a local replica of the hosted Auto Review, ported from the shipd-workflow rulebook; tests and solution only; no description grading, and `test.sh` is ignored):
|
|
18
|
+
|
|
19
|
+
1. Snapshots git `HEAD` into a temp workspace as the pinned base (`repo/`), next to `review_assets/` holding `agent_prompt.md` (as problem.md), `test.patch` with its `test.sh` section removed, `solution.patch`, and standalone copies of new test files. Neither patch is applied.
|
|
20
|
+
2. Runs read-only Claude Code CLI reviewers on this machine (`Read`, `Grep`, `Glob` only; your local `claude` login), several at a time, one read each: Local Test Quality per-file fairness batches (one per ~60 KB of test files), one coverage read (with judge-tests' discrimination rules folded in), the solution audit over nine lenses (paths, identity, ordering, variants, numeric, compose, metadata, lifecycle including the state lifecycle walk, behaviour), and per-unit Solution Quality on every `solution.patch` hunk (12 hunks per reviewer). A typical task needs about 6–8 sessions. Prompts carry hosted calibration distilled into general patterns (`src/autoreview-data.ts`), not quotes from past tasks.
|
|
21
|
+
3. Runs synthesis (one session per 50 findings) over every finding with the hosted proof legs (failing case, wrong implementation, promise citation, load-bearing check, provenance, materiality, reference pass, exercised environment). A finding gates only when synthesis keeps it HIGH.
|
|
22
|
+
4. Verdict: `BLOCK` (any gating HIGH), `REVIEW` (MEDIUM or unproven findings, a Not fair test, or a failed reviewer), `PASS`, or `INCOMPLETE`. Test Quality is `FAIL` when any test is Not fair (an assertion with no source counts). Counter-tests the reviewers write are included in the report but not executed, so the legs that need execution are decided from the files.
|
|
23
|
+
5. Writes only the final report, `.pi/shipd-checks/<runId>-autoreview.md` (verdict, findings with synthesis reasoning, Test Quality results, reviewer status, and the counter-test patches inline), and merges a summary into `shipd_report.json` under `autoReview`. Reviewer prompts, streams and answers are not kept.
|
|
24
|
+
|
|
17
25
|
The agent-callable `gap-finder` tool runs one exhaustive read-only agent over the prompt, repository, and tests. It
|
|
18
26
|
reviews requirements sentence by sentence, performs its own evidence and fairness check, and returns confirmed
|
|
19
27
|
behavioral test gaps in `details.testGaps`.
|
|
@@ -67,28 +75,30 @@ Shipd tokens; do not run overlapping invocations for the same challenge.
|
|
|
67
75
|
|
|
68
76
|
## Commands
|
|
69
77
|
|
|
70
|
-
`/checks solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as the
|
|
78
|
+
`/checks --solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as our reference solution: it stays context only for the fairness review, and the reviewer also compares its non-test code logic with every solver's patch and lists, under "Suggested improvements to our solution", edge cases or logical errors a solver handles correctly that ours does not (no tests, docs, or style). `/checks --config` is separate and configures the Claude solver and reviewer settings and the analysis-tool model. The gap-finder, solution-precheck, and prompt-humanizer tools are enabled per project with `/analyze:on`.
|
|
71
79
|
|
|
72
80
|
| Command | Effect |
|
|
73
81
|
|---|---|
|
|
74
|
-
| `/checks` | Open a menu for
|
|
75
|
-
| `/checks solve` | Run isolated Claude CLI solvers and base/new tests, then use a local
|
|
76
|
-
| `/checks --
|
|
82
|
+
| `/checks` | Open a menu for config, solve, or autoreview |
|
|
83
|
+
| `/checks --solve` | Run isolated Claude CLI solvers and base/new tests, then use a local Claude CLI reviewer to assess test fairness and failed agents |
|
|
84
|
+
| `/checks --autoreview` | Run the local Claude CLI Auto Review of `test.patch` and `solution.patch` (requires `agent_prompt.md`, `test.patch`, `solution.patch`) |
|
|
85
|
+
| `/checks --config` | Configure Solver model (Claude), Claude solver effort, timeout, parallel agent count, the Claude reviewer, Auto Review, and the analysis-tool model |
|
|
77
86
|
| `/shipd:auth` | Open a browser for first-time Shipd sign-in and save the session |
|
|
78
87
|
| `/shipd:link <url>` | Save the Shipd job link for the current chat session |
|
|
79
88
|
| `/analyze:on` | Enable the gap-finder, solution-precheck, and prompt-humanizer tools for the current project |
|
|
80
89
|
| `/analyze:off` | Disable the gap-finder, solution-precheck, and prompt-humanizer tools for the current project |
|
|
81
90
|
|
|
82
|
-
**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task or local
|
|
91
|
+
**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task, local Claude review, or local Auto Review reviewers. A cancelled run skips final report writing.
|
|
83
92
|
|
|
84
93
|
## Configuration
|
|
85
94
|
|
|
86
|
-
`/checks --config` opens the row-based settings menu with
|
|
95
|
+
`/checks --config` opens the row-based settings menu with three sections:
|
|
87
96
|
|
|
88
|
-
- **Solver**: Solver model (Claude: `haiku`, `sonnet`, or `opus`), solver effort (`low`, `medium`, `high`, `xhigh`, or `max`), one-shot timeout (10–90 minutes), parallel agent count, and the
|
|
89
|
-
- **
|
|
97
|
+
- **Solver**: Solver model (Claude: `haiku`, `sonnet`, or `opus`), solver effort (`low`, `medium`, `high`, `xhigh`, or `max`), one-shot timeout (10–90 minutes), parallel agent count, and the Claude reviewer model (default `opus`), effort (default `high`), and timeout (10–60 minutes; default 20), stored under `solverReview` in `checks-config.json`. Solver effort defaults to `low` when unset and is passed explicitly to Claude. Diffs, agent output, `agent_steps.jsonl` traces, base/new logs, and the local reviewer assessment are saved under `.pi/shipd-checks/<runId>/` and summarized in `shipd_report.json`.
|
|
98
|
+
- **Auto Review**: Claude reviewer model (`haiku`, `sonnet`, or `opus`; default `opus`), effort (default `high`), timeout per reviewer (10–90 minutes; default 40), and parallel reviewers (1–8; default 4). Stored under `autoReview` in `checks-config.json`.
|
|
99
|
+
- **Analysis Tools**: timeout, model, and thinking level for the separate agent-callable gap-finder and solution-precheck tools. The timeout applies to each read-only agent. These are stored under `analyzeGap.timeoutMinutes`, `analyzeGap.provider`, `analyzeGap.modelId`, and `analyzeGap.thinkingLevel`. The analysis tools have no fallback model: they refuse to run until a model is set here. Fargate resources are selected automatically and are not configurable in this menu.
|
|
90
100
|
|
|
91
|
-
For each Claude solver, `shipd_report.json` records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, `totalTokens` subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local
|
|
101
|
+
For each Claude solver, `shipd_report.json` records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, `totalTokens` subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local Claude reviewer records the CLI-reported usage and cost. Per-agent, reviewer, and aggregate usage appears once under `solverTokenUsage`; `solverRunSummary` contains outcomes, aggregate test counts, and per-suite JUnit counts, while `solverReview` contains the assessment and reviewer model settings.
|
|
92
102
|
|
|
93
103
|
The humanizer candidate counts are configured separately in `~/.pi/agent/checks-config.json`. `candidateCount` controls
|
|
94
104
|
how many drafts are requested from CoreWeave (1–10); `topCount` controls the maximum returned (1–10, capped at
|
|
@@ -179,7 +189,7 @@ On-Demand fallback. Spot interruptions are retried according to `fargate.maxRetr
|
|
|
179
189
|
profile and do not update adaptive sizing. Set `projectProfiles` to override solver-gap
|
|
180
190
|
resources per repository: `{"C:/path/to/repo":"large"}`.
|
|
181
191
|
|
|
182
|
-
5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set `
|
|
192
|
+
5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set the top-level `claudeOAuthTokenFile` in `checks-config.json` to that file's path (or set `CLAUDE_CODE_OAUTH_TOKEN_FILE`). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's `auth.json`. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in `/checks --config`, then run `/checks --solve`. Projects need `Dockerfile`, `agent_prompt.md`, and `test.patch`, which should add `test.sh` with `--output_path <file> base` and `--output_path <file> new` modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.
|
|
183
193
|
|
|
184
194
|
Use `/analyze:on` and `/analyze:off` to control the gap-finder, solution-precheck, and prompt-humanizer tools per project, like HPC. The
|
|
185
195
|
enabled project list is stored alongside the other settings in `~/.pi/agent/checks-config.json`.
|
|
@@ -234,7 +244,7 @@ Or, for local development, point at the entry point directly:
|
|
|
234
244
|
| `src/index.ts` | Extension entry point: message renderer, cancel shortcut, command registration |
|
|
235
245
|
| `src/submit.ts` | `quality-check`: Playwright draft filling, sequential quality checks, scheduled polling, and report extraction |
|
|
236
246
|
| `src/command.ts` | The `/checks` command: argument parsing, `--config` flow, run orchestration |
|
|
237
|
-
| `src/agents.ts` | Spawns and races local Pi gap-finder
|
|
247
|
+
| `src/agents.ts` | Spawns and races local Pi gap-finder and solution-precheck sessions; runs the benchmark review through the local Claude CLI |
|
|
238
248
|
| `src/solvergap.ts` | Benchmark artifact-directory constant |
|
|
239
249
|
| `src/fargate-docker.ts` | Supported Dockerfile parsing for remote solver setup |
|
|
240
250
|
| `src/fargate-runner.ts` | ECS Fargate Spot/S3 orchestration for Claude CLI benchmark runs, patch-prechecks, and detector tasks; retries and cleanup |
|