@sreetej510/pi-shipd-checks 0.13.1 → 0.14.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -40
- package/dist/index.js +139 -147
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -10,25 +10,17 @@ For `/checks --solve`:
|
|
|
10
10
|
2. Sends the exact contents of `agent_prompt.md` as each solver's sole first user message. No prompt file, task Dockerfile, `test.patch`, `solution.patch`, or uncommitted test files are overlaid into a solver workspace. The Dockerfile is used only by the harness to provision the Fargate environment.
|
|
11
11
|
3. Runs several configurable (default 3) independent Claude Code CLI solver invocations in parallel, each with its own isolated clean-HEAD workspace and only the `Bash` tool enabled. Captures each solver's `stream-json` trace of assistant and tool events (not private chain-of-thought). There is no test-driven repair loop, and the test patch remains unavailable until that solver exits.
|
|
12
12
|
4. After a solver finishes, the harness applies `test.patch` to its workspace, then runs `bash test.sh --output_path <file> base` and `bash test.sh --output_path <file> new` as an unprivileged user with a fresh writable `HOME` and cache. It captures both full logs.
|
|
13
|
-
5. After all solvers exit and their tests run, a read-only local Claude Code CLI reviewer (`Read`, `Grep`, `Glob` only; your local `claude` login, same
|
|
13
|
+
5. After all solvers exit and their tests run, a read-only local Claude Code CLI reviewer (`Read`, `Grep`, `Glob` only; your local `claude` login, same runner as the `auditor` tool) receives an untouched clean-HEAD tree plus `agent_prompt.md`, all patch files, standalone copies of new test files, each solver's patch, and every solver's test logs. No patch is applied in the review workspace.
|
|
14
14
|
6. Runs the Claude CLI solvers inside one AWS ECS Fargate Spot task. A Spot interruption retries the task and resumes completed solver indexes. A local `claude setup-token` file is transferred through a private, server-side-encrypted S3 object and supplied to the Claude CLI through `CLAUDE_CODE_OAUTH_TOKEN`. A root-owned shell wrapper strips credential variables and drops every solver Bash command to its own unprivileged UID; the token is not available to those commands. The temporary token and run objects are deleted after the task stops.
|
|
15
15
|
7. Posts a chat summary and logs results in `.pi/shipd-checks/<runId>/`, including `agent_steps.jsonl` per solver, and `shipd_report.json` in your project root.
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
The agent-callable `auditor` tool replaces the old `gap-finder` and `solution-precheck` tools. It is a local replica of the hosted Auto Review, ported from the shipd-workflow rulebook, for the tests and the solution only (no description grading; `test.sh` is ignored). It takes no parameters:
|
|
18
18
|
|
|
19
|
-
1.
|
|
20
|
-
2.
|
|
21
|
-
3. Runs
|
|
22
|
-
4.
|
|
23
|
-
5.
|
|
24
|
-
|
|
25
|
-
The agent-callable `gap-finder` tool runs one exhaustive read-only agent over the prompt, repository, and tests. It
|
|
26
|
-
reviews requirements sentence by sentence, performs its own evidence and fairness check, and returns confirmed
|
|
27
|
-
behavioral test gaps in `details.testGaps`.
|
|
28
|
-
|
|
29
|
-
The agent-callable `solution-precheck` tool runs one exhaustive read-only solution-quality agent and returns findings in
|
|
30
|
-
`details.solutionAuditFindings`. It checks contract gaps, regressions, failure safety, architecture, dead code, and
|
|
31
|
-
unrelated changes.
|
|
19
|
+
1. Runs `create_patches.sh` when the project has one, to refresh `test.patch` and `solution.patch` from the working tree, then requires `agent_prompt.md`, `test.patch` and `solution.patch`.
|
|
20
|
+
2. Snapshots git `HEAD` into a temp workspace as the pinned base (`repo/`), next to `review_assets/` holding `agent_prompt.md` (as problem.md), `test.patch` with its `test.sh` section removed, `solution.patch`, and standalone copies of new test files. Neither patch is applied.
|
|
21
|
+
3. Runs read-only Claude Code CLI reviewers on this machine (`Read`, `Grep`, `Glob` only; your local `claude` login), several at a time, one read each: Local Test Quality per-file fairness batches (one per ~60 KB of test files), one coverage read, the solution audit over ten lenses (paths, identity, ordering, variants, numeric, compose, metadata, lifecycle, behaviour, and the quality lens for solution betterment: repo conventions, dead code, duplication, unrelated changes, AI artefacts), and per-unit Solution Quality on every `solution.patch` hunk (12 hunks per reviewer). The embedded `rules.md` families the old tools used go into these prompts: fairness rules into per-test Test Quality, test-gap rules into coverage, solution-gap rules into the audit and per-unit reads. Prompts also carry hosted calibration distilled into general patterns (`src/auditor-data.ts`), not quotes from past tasks. A typical task needs about 6–8 sessions.
|
|
22
|
+
4. Runs synthesis (one session per 50 findings) over every finding with the hosted proof legs (failing case, wrong implementation, promise citation, load-bearing check, provenance, materiality, reference pass, exercised environment). A finding gates only when synthesis keeps it HIGH; findings synthesis rejects as wrong are dropped.
|
|
23
|
+
5. Returns the findings grouped as `details.unfairTests`, `details.testGaps`, `details.solutionGaps` and `details.solutionImprovements`, each with `level` (`HIGH` gates, `MEDIUM` is the fix-before-upload list), `criterion`, `finding` and the synthesis reasoning, plus `details.verdict` (`BLOCK`, `REVIEW`, `PASS` or `INCOMPLETE`), `details.testQuality`, `details.solutionQuality`, `details.counterTests` (not executed) and reviewer statuses. It writes only one file, the report `.pi/shipd-checks/<runId>-auditor.md`; reviewer prompts and answers are not kept.
|
|
32
24
|
|
|
33
25
|
The opt-in `humanize-agent-prompt` tool reads the complete current-project `agent_prompt.md`, requests 3 experimental
|
|
34
26
|
rewrite drafts from the CoreWeave Nemotron LoRA, scores every draft through a configured HTTPS detector endpoint, and
|
|
@@ -38,15 +30,17 @@ or correctness evidence. The adapter is experimental and was trained on all 519
|
|
|
38
30
|
assigned to EVAL and HOLDOUT; treat all outputs as unapproved drafts.
|
|
39
31
|
|
|
40
32
|
The agent-callable `quality-check` tool runs `create_patches.sh` in the current working directory and reads
|
|
41
|
-
`agent_prompt.md`, `test.patch`, and `solution.patch`. Before
|
|
42
|
-
|
|
33
|
+
`agent_prompt.md`, `test.patch`, and `solution.patch`. Before reading any quality status, it syncs the Shipd draft: it
|
|
34
|
+
fills only the fields whose saved content differs from the local files, waits until Shipd's Convex backend acknowledges
|
|
35
|
+
every save and the header shows `Saved`, then reloads and confirms the draft matches (up to three attempts). Each page
|
|
36
|
+
load waits until every quality row's status has loaded, since a row without its status would read as a current report. It computes a SHA-256 of the
|
|
43
37
|
exact prompt content. The last passing prompt hash is stored as a custom entry in the current chat session. If the
|
|
44
38
|
hash matches, it skips the Shipd Prompt Quality check entirely and reports a cached `PASS`; if it differs or is absent, it
|
|
45
39
|
checks the refreshed status after filling the fields and runs only when the check is stale or has a `Run` button. Otherwise it
|
|
46
40
|
uses the current report. A new hash is saved only after Prompt Quality passes. On failure, only `promptQuality` is returned; there are no Test/Solution results or `stoppedReason`.
|
|
47
41
|
Only after Prompt Quality passes or has a matching cached pass does it run the Fargate patch precheck against a clean `HEAD`;
|
|
48
|
-
only if that passes does it run Test Quality and Solution Quality.
|
|
49
|
-
current reports, and
|
|
42
|
+
only if that passes does it run Test Quality, and only if Test Quality passes does it run Solution Quality. A failing Test Quality (any unfair test or coverage suggestion) returns `promptQuality` and `testQuality` without running Solution Quality. Each check runs fresh jobs, reruns stale reports, skips
|
|
43
|
+
current reports, and waits for started jobs. It returns only the useful report data:
|
|
50
44
|
`details.testQuality.verdict`, `details.testQuality.coverageSuggestions`, `details.testQuality.unfairnessCount`, and
|
|
51
45
|
`details.testQuality.tests` filtered to blocks whose `concerns` array is non-empty or whose `fairness` is exactly `"Not fair"`,
|
|
52
46
|
`details.promptQuality` and its verdict, reason, and major non-AI comments (only `category`, `quote`, and `suggestion`; AI-category comments are omitted), plus
|
|
@@ -68,35 +62,34 @@ challenge-submit button. Authentication comes from `SHIPD_STORAGE_STATE` or the
|
|
|
68
62
|
`/shipd:auth`. `/shipd:auth` launches the installed browser as a normal visible process and attaches over local CDP,
|
|
69
63
|
which supports Google/SSO sign-in better than an automated browser launch.
|
|
70
64
|
|
|
71
|
-
The
|
|
72
|
-
never in parallel, and run repeated requests sequentially after applying each
|
|
65
|
+
The `auditor` tool is read-only apart from its report and `create_patches.sh`, takes a long time, and consumes Claude
|
|
66
|
+
usage. Invoke it only when the user asks, never in parallel, and run repeated requests sequentially after applying each
|
|
67
|
+
result. The prompt humanizer also requires
|
|
73
68
|
an explicit user request and sends the prompt to CoreWeave and the configured detector endpoint. `quality-check` consumes
|
|
74
69
|
Shipd tokens; do not run overlapping invocations for the same challenge.
|
|
75
70
|
|
|
76
71
|
## Commands
|
|
77
72
|
|
|
78
|
-
`/checks --solve` runs the benchmark using `agent_prompt.md`; it requires that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as our reference solution: it stays context only for the fairness review, and the reviewer also compares its non-test code logic with every solver's patch and lists, under "Suggested improvements to our solution", edge cases or logical errors a solver handles correctly that ours does not (no tests, docs, or style). `/checks --config` is separate and configures the Claude solver
|
|
73
|
+
`/checks --solve` runs the benchmark using `agent_prompt.md`; it requires `create_patches.sh`, that prompt file, `Dockerfile`, and `test.patch` (which supplies `test.sh` and the test files). Like the `quality-check` tool, it first runs `create_patches.sh` in the working directory to regenerate `test.patch` and `solution.patch`, and stops if the script fails. The solver sees only clean `HEAD` plus the prompt text; the harness applies `test.patch` after the solver exits. `solution.patch` is not required by the solver; when present, the local reviewer receives it as our reference solution: it stays context only for the fairness review, and the reviewer also compares its non-test code logic with every solver's patch and lists, under "Suggested improvements to our solution", edge cases or logical errors a solver handles correctly that ours does not (no tests, docs, or style). `/checks --config` is separate and configures the Claude solver, reviewer and auditor settings. The `auditor` and prompt-humanizer tools are enabled per project with `/analyze:on`.
|
|
79
74
|
|
|
80
75
|
| Command | Effect |
|
|
81
76
|
|---|---|
|
|
82
|
-
| `/checks` | Open a menu for config
|
|
77
|
+
| `/checks` | Open a menu for config or solve |
|
|
83
78
|
| `/checks --solve` | Run isolated Claude CLI solvers and base/new tests, then use a local Claude CLI reviewer to assess test fairness and failed agents |
|
|
84
|
-
| `/checks --
|
|
85
|
-
| `/checks --config` | Configure Solver model (Claude), Claude solver effort, timeout, parallel agent count, the Claude reviewer, Auto Review, and the analysis-tool model |
|
|
79
|
+
| `/checks --config` | Configure Solver model (Claude), Claude solver effort, timeout, parallel agent count, the Claude reviewer, and the auditor tool |
|
|
86
80
|
| `/shipd:auth` | Open a browser for first-time Shipd sign-in and save the session |
|
|
87
81
|
| `/shipd:link <url>` | Save the Shipd job link for the current chat session |
|
|
88
|
-
| `/analyze:on` | Enable the
|
|
89
|
-
| `/analyze:off` | Disable the
|
|
82
|
+
| `/analyze:on` | Enable the `auditor` and prompt-humanizer tools for the current project |
|
|
83
|
+
| `/analyze:off` | Disable the `auditor` and prompt-humanizer tools for the current project |
|
|
90
84
|
|
|
91
|
-
**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task
|
|
85
|
+
**Shortcut:** `Ctrl+Shift+X` cancels an in-progress `/checks` run, stopping its active Fargate task or local Claude review. A cancelled run skips final report writing.
|
|
92
86
|
|
|
93
87
|
## Configuration
|
|
94
88
|
|
|
95
|
-
`/checks --config` opens the row-based settings menu with
|
|
89
|
+
`/checks --config` opens the row-based settings menu with two sections:
|
|
96
90
|
|
|
97
91
|
- **Solver**: Solver model (Claude: `haiku`, `sonnet`, or `opus`), solver effort (`low`, `medium`, `high`, `xhigh`, or `max`), one-shot timeout (10–90 minutes), parallel agent count, and the Claude reviewer model (default `opus`), effort (default `high`), and timeout (10–60 minutes; default 20), stored under `solverReview` in `checks-config.json`. Solver effort defaults to `low` when unset and is passed explicitly to Claude. Diffs, agent output, `agent_steps.jsonl` traces, base/new logs, and the local reviewer assessment are saved under `.pi/shipd-checks/<runId>/` and summarized in `shipd_report.json`.
|
|
98
|
-
- **
|
|
99
|
-
- **Analysis Tools**: timeout, model, and thinking level for the separate agent-callable gap-finder and solution-precheck tools. The timeout applies to each read-only agent. These are stored under `analyzeGap.timeoutMinutes`, `analyzeGap.provider`, `analyzeGap.modelId`, and `analyzeGap.thinkingLevel`. The analysis tools have no fallback model: they refuse to run until a model is set here. Fargate resources are selected automatically and are not configurable in this menu.
|
|
92
|
+
- **Analysis Tools**: whether the analysis tools are enabled for this project, and the `auditor` tool's Claude settings: model (`haiku`, `sonnet`, or `opus`; default `opus`), effort (default `high`), timeout per reviewer (10–90 minutes; default 40), and parallel reviewers (1–8; default 4), stored under `auditor` in `checks-config.json` (an `autoReview` block from earlier versions is still read). Fargate resources are selected automatically and are not configurable in this menu.
|
|
100
93
|
|
|
101
94
|
For each Claude solver, `shipd_report.json` records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, `totalTokens` subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local Claude reviewer records the CLI-reported usage and cost. Per-agent, reviewer, and aggregate usage appears once under `solverTokenUsage`; `solverRunSummary` contains outcomes, aggregate test counts, and per-suite JUnit counts, while `solverReview` contains the assessment and reviewer model settings.
|
|
102
95
|
|
|
@@ -191,7 +184,7 @@ On-Demand fallback. Spot interruptions are retried according to `fargate.maxRetr
|
|
|
191
184
|
|
|
192
185
|
5. Generate a CI token with `claude setup-token` and save only the token in a local secret file. Set the top-level `claudeOAuthTokenFile` in `checks-config.json` to that file's path (or set `CLAUDE_CODE_OAUTH_TOKEN_FILE`). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's `auth.json`. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in `/checks --config`, then run `/checks --solve`. Projects need `Dockerfile`, `agent_prompt.md`, and `test.patch`, which should add `test.sh` with `--output_path <file> base` and `--output_path <file> new` modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.
|
|
193
186
|
|
|
194
|
-
Use `/analyze:on` and `/analyze:off` to control the
|
|
187
|
+
Use `/analyze:on` and `/analyze:off` to control the `auditor` and prompt-humanizer tools per project, like HPC. The
|
|
195
188
|
enabled project list is stored alongside the other settings in `~/.pi/agent/checks-config.json`.
|
|
196
189
|
|
|
197
190
|
The patch precheck always uses the `medium` Fargate profile. It applies `test.patch`, creates isolated workspaces with and without `solution.patch`, and runs the base suites concurrently in both workspaces followed by the new suites concurrently in both workspaces. It requires
|
|
@@ -199,8 +192,7 @@ The patch precheck always uses the `medium` Fargate profile. It applies `test.pa
|
|
|
199
192
|
before `solution.patch`, then requires both base and new suites to pass after the solution patch. Pytest subtest totals are not treated as passing testcase records; emitted `<testcase>` statuses and explicit `SUBPASSED` records are used for the pre-solution invariant. A failed precheck aborts
|
|
200
193
|
before Shipd is opened and includes the Linux platform, a human-readable phase, and failed/errored test names in the tool error.
|
|
201
194
|
|
|
202
|
-
The `
|
|
203
|
-
tests, prompt, or solution. Invoke them only when the user asks, never in parallel, and run repeated requests
|
|
195
|
+
The `auditor` tool returns review findings, not edits; the caller changes the tests or solution. Invoke them only when the user asks, never in parallel, and run repeated requests
|
|
204
196
|
sequentially after applying each result. The humanizer does not modify `agent_prompt.md`; validate all requirements and
|
|
205
197
|
apply any selected draft locally.
|
|
206
198
|
|
|
@@ -211,9 +203,8 @@ apply any selected draft locally.
|
|
|
211
203
|
}
|
|
212
204
|
```
|
|
213
205
|
|
|
214
|
-
Use ↑/↓ to select a row and Enter/Space to change it.
|
|
215
|
-
|
|
216
|
-
under `solverGap` and analyze-tool model settings under `analyzeGap`.
|
|
206
|
+
Use ↑/↓ to select a row and Enter/Space to change it. Settings are saved to `~/.pi/agent/checks-config.json`; solver
|
|
207
|
+
settings are nested under `solverGap`, the solve reviewer under `solverReview`, and the auditor under `auditor`.
|
|
217
208
|
|
|
218
209
|
## Install
|
|
219
210
|
|
|
@@ -244,14 +235,19 @@ Or, for local development, point at the entry point directly:
|
|
|
244
235
|
| `src/index.ts` | Extension entry point: message renderer, cancel shortcut, command registration |
|
|
245
236
|
| `src/submit.ts` | `quality-check`: Playwright draft filling, sequential quality checks, scheduled polling, and report extraction |
|
|
246
237
|
| `src/command.ts` | The `/checks` command: argument parsing, `--config` flow, run orchestration |
|
|
247
|
-
| `src/agents.ts` |
|
|
238
|
+
| `src/agents.ts` | Runs the `--solve` benchmark review through the local Claude CLI |
|
|
239
|
+
| `src/analyze.ts` | The agent-callable `auditor` tool (rendering, result text) and humanizer registration |
|
|
240
|
+
| `src/auditor.ts` | The auditor engine: workspace, reviewer plan, findings, synthesis grading, report |
|
|
241
|
+
| `src/auditor-prompts.ts` | Auditor reviewer and synthesis prompts |
|
|
242
|
+
| `src/auditor-data.ts` | Hosted Auto Review calibration distilled into general patterns |
|
|
243
|
+
| `src/claude-cli.ts` | Read-only local Claude CLI runner shared by the auditor and the `--solve` reviewer |
|
|
248
244
|
| `src/solvergap.ts` | Benchmark artifact-directory constant |
|
|
249
245
|
| `src/fargate-docker.ts` | Supported Dockerfile parsing for remote solver setup |
|
|
250
246
|
| `src/fargate-runner.ts` | ECS Fargate Spot/S3 orchestration for Claude CLI benchmark runs, patch-prechecks, and detector tasks; retries and cleanup |
|
|
251
247
|
| `src/fargate-worker.ts` | ESM worker that runs isolated Claude CLI solvers or patch prechecks in the shared task |
|
|
252
248
|
| `src/resource-usage.ts` | Container CPU sampling for adaptive profile selection |
|
|
253
|
-
| `src/prompts.ts` |
|
|
254
|
-
| `src/tools.ts` |
|
|
249
|
+
| `src/prompts.ts` | Focus-reviewer and `--solve` reviewer prompt text |
|
|
250
|
+
| `src/tools.ts` | Report-submission tool for the focus reviewers |
|
|
255
251
|
| `src/rubric.ts` | Embedded guidelines/fairness rubric text + per-role section loaders |
|
|
256
252
|
| `src/roles.ts` | The 3 reviewer roles (description/tests/solution) metadata |
|
|
257
253
|
| `src/report.ts` | `shipd_report.json` load/merge/summary logic |
|