pi-pr-review 1.1.2 → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -11
- package/extensions/pr-review-subagent.ts +455 -78
- package/lib/pr-review-policy.ts +32 -0
- package/package.json +7 -4
- package/prompts/pr-review.md +15 -13
- package/tests/pr-review-policy.test.ts +72 -0
package/README.md
CHANGED
|
@@ -63,15 +63,19 @@ The `/pr-review-config` command maps three labels to models:
|
|
|
63
63
|
```
|
|
64
64
|
/pr-review-config # open the settings menu (like /settings & /nervous:config)
|
|
65
65
|
/pr-review-config show # print the current mapping
|
|
66
|
-
/pr-review-config light=<spec> medium=<spec> heavy=<spec> # set
|
|
66
|
+
/pr-review-config light=<spec> medium=<spec> heavy=<spec> # set primary tier models
|
|
67
|
+
/pr-review-config heavy_fallbacks=<spec>,<spec> # retry chain for quota/rate-limit failures
|
|
68
|
+
/pr-review-config light_tool_policy=none # tier default when a pass omits tool_policy
|
|
67
69
|
/pr-review-config medium=unset # clear a tier (back to pi default)
|
|
68
|
-
/pr-review-config
|
|
70
|
+
/pr-review-config heavy_fallbacks=unset # clear a fallback chain
|
|
71
|
+
/pr-review-config light_tool_policy=unset # restore legacy configured-tool behavior
|
|
72
|
+
/pr-review-config tools=read,bash,grep,find,ls # allowlist used by configured policy
|
|
69
73
|
```
|
|
70
74
|
|
|
71
75
|
Running `/pr-review-config` with no arguments in the TUI opens an interactive settings menu that mirrors pi's `/settings` and the NERVous `/nervous:config`:
|
|
72
76
|
|
|
73
|
-
- One row per tier (`light` / `medium` / `heavy`
|
|
74
|
-
- Press Enter on a
|
|
77
|
+
- One primary-model, fallback-model, and tool-policy row per tier (`light` / `medium` / `heavy`) plus a configured-tool allowlist row.
|
|
78
|
+
- Press Enter on a primary or fallback row to pick a model from a searchable list (or unset it); Enter/Space cycles tool policies and allowlist presets. The menu sets one fallback model at a time; use the `key=value` form for longer fallback chains.
|
|
75
79
|
- Selections apply and persist **immediately**; Esc closes the menu.
|
|
76
80
|
- Type to search, and tab-completion is available for the `key=value` form.
|
|
77
81
|
|
|
@@ -91,15 +95,27 @@ Example `pr-review.json`:
|
|
|
91
95
|
"medium": "<balanced-model-spec>",
|
|
92
96
|
"heavy": "<strong-model-spec:high>"
|
|
93
97
|
},
|
|
98
|
+
"fallbacks": {
|
|
99
|
+
"light": ["<backup-fast-model>"],
|
|
100
|
+
"medium": ["<backup-balanced-model>"],
|
|
101
|
+
"heavy": ["<backup-strong-model:high>", "<balanced-model-spec>"]
|
|
102
|
+
},
|
|
103
|
+
"toolPolicies": {
|
|
104
|
+
"light": "none",
|
|
105
|
+
"medium": "configured",
|
|
106
|
+
"heavy": "configured"
|
|
107
|
+
},
|
|
94
108
|
"tools": ["read", "bash", "grep", "find", "ls"]
|
|
95
109
|
}
|
|
96
110
|
```
|
|
97
111
|
|
|
98
|
-
Each tier runs in an **isolated `pi` subprocess** on its configured model. The `review_subagents` batch tool runs independent passes concurrently (default `max_parallel: 4`, capped at 6) and returns ordered per-pass results; the older single-pass `review_subagent` tool remains available as a compatibility fallback. If a tier is unset, that subagent falls back to the nearest configured tier, then to your pi default model.
|
|
112
|
+
Each tier runs in an **isolated `pi` subprocess** on its configured model. The `review_subagents` batch tool runs independent passes concurrently (default `max_parallel: 4`, capped at 6) and returns ordered per-pass results; the older single-pass `review_subagent` tool remains available as a compatibility fallback. If a tier model fails with a retryable quota/rate-limit/capacity error, the subprocess retries that tier's configured `fallbacks` in order. Non-quota failures do not blindly cycle through fallbacks. If a tier is unset, that subagent falls back to the nearest configured tier, then to your pi default model.
|
|
113
|
+
|
|
114
|
+
Tool policy is additive and backward compatible: `none` emits Pi's explicit `--no-tools`; `configured` uses the existing `tools` allowlist. A tool call's optional `tool_policy` overrides `toolPolicies[tier]`, which in turn falls back to legacy `configured` behavior. The shipped `/pr-review` prompt explicitly uses `none` only for overview because its complete evidence is supplied in context. Conventions/maintainability and both heavy specialist passes use `configured` repository-context tools so they can inspect surrounding files when needed. Fallback model attempts keep the original pass policy.
|
|
99
115
|
|
|
100
|
-
### 2. The orchestrator / fallback model
|
|
116
|
+
### 2. The orchestrator / inline-fallback model
|
|
101
117
|
|
|
102
|
-
The orchestrator (which fetches the PR, merges findings, and emits the JSON) and the inline
|
|
118
|
+
The orchestrator (which fetches the PR, merges findings, and emits the JSON) and the inline fallback path both run on your pi session model:
|
|
103
119
|
|
|
104
120
|
- **Per run:** `pi --model <model-id> "/pr-review 123"`
|
|
105
121
|
- **Persistent default (user):** `~/.pi/agent/settings.json` → `{ "defaultModel": "<model-id>", "defaultThinkingLevel": "high" }`
|
|
@@ -174,15 +190,19 @@ Severity tags: `[P0]` blocking/drop-everything · `[P1]` blocking/urgent · `[P2
|
|
|
174
190
|
pi-pr-review/
|
|
175
191
|
├─ package.json # pi manifest: prompts + extensions
|
|
176
192
|
├─ prompts/pr-review.md # the /pr-review orchestrator prompt
|
|
193
|
+
├─ lib/pr-review-policy.ts # pure tool-policy resolution/argv helpers
|
|
177
194
|
├─ extensions/pr-review-subagent.ts # review_subagents/review_subagent tools + /pr-review-config command
|
|
178
|
-
|
|
195
|
+
├─ extensions/review-table.ts # renders the final JSON as a table (TUI only)
|
|
196
|
+
└─ tests/pr-review-policy.test.ts # focused policy compatibility tests
|
|
179
197
|
```
|
|
180
198
|
|
|
181
|
-
##
|
|
199
|
+
## Speed, security & cost notes
|
|
182
200
|
|
|
183
|
-
- The
|
|
201
|
+
- The four independent review lenses remain intact. Only overview runs context-only with `--no-tools`; medium and both heavy specialists retain configured tools for surrounding-file validation. All subprocesses use `--no-context-files` because the orchestrator supplies the base review context explicitly, and convention excerpts are sent only to the medium pass instead of every model.
|
|
202
|
+
- Tool results include effective `toolPolicy` and `elapsedMs` telemetry (plus per-attempt timing) so repeated representative runs can be compared at p50/p95 without guessing. Restore tools for a custom pass by sending `tool_policy: "configured"`; callers that omit policy retain legacy behavior unless `toolPolicies` config says otherwise.
|
|
203
|
+
- The `review_subagents` batch tool and `review_subagent` fallback spawn isolated `pi` subprocesses (`--mode json -p --no-session`) on your configured tier models. Reviewer prompts prohibit modifications, but a configured allowlist containing `bash` is not technically read-only; use a narrower allowlist if stronger enforcement is required.
|
|
184
204
|
- Project-local `pr-review.json` is only read when the project is trusted.
|
|
185
|
-
- Tiered review calls multiple models per PR,
|
|
205
|
+
- Tiered review calls multiple models per PR, concurrently for independent passes. Point `light` at a cheap model for overview/risk scan; reserve `heavy` for deep passes, and configure per-tier `fallbacks` only for acceptable backup models because retries can increase cost.
|
|
186
206
|
|
|
187
207
|
## Design notes
|
|
188
208
|
|
|
@@ -4,7 +4,8 @@
|
|
|
4
4
|
* Adds configurable, tiered review subagents to the /pr-review workflow.
|
|
5
5
|
*
|
|
6
6
|
* - Config surface: `pr-review.json` (user: ~/.pi/agent, project: <repo>/.pi) maps
|
|
7
|
-
* the labels `light` / `medium` / `heavy` to whatever models you choose
|
|
7
|
+
* the labels `light` / `medium` / `heavy` to whatever models you choose,
|
|
8
|
+
* plus optional per-tier fallback chains for quota/capacity failures.
|
|
8
9
|
* No model names are hardcoded here — you configure them.
|
|
9
10
|
* - Tool: `review_subagent` spawns one isolated `pi` subprocess on the model
|
|
10
11
|
* bound to the requested tier and returns its review report.
|
|
@@ -45,6 +46,13 @@ import {
|
|
|
45
46
|
Text,
|
|
46
47
|
} from "@earendil-works/pi-tui";
|
|
47
48
|
import { Type } from "typebox";
|
|
49
|
+
import {
|
|
50
|
+
appendToolPolicyArgs,
|
|
51
|
+
buildReviewBaseArgs,
|
|
52
|
+
normalizeToolPolicy,
|
|
53
|
+
resolveToolPolicy,
|
|
54
|
+
type ToolPolicy,
|
|
55
|
+
} from "../lib/pr-review-policy.ts";
|
|
48
56
|
|
|
49
57
|
// ---------------------------------------------------------------------------
|
|
50
58
|
// Config
|
|
@@ -54,6 +62,8 @@ type Tier = "light" | "medium" | "heavy";
|
|
|
54
62
|
const TIERS: Tier[] = ["light", "medium", "heavy"];
|
|
55
63
|
|
|
56
64
|
const UNSET = "(unset — pi default)";
|
|
65
|
+
const INHERIT_TOOL_POLICY = "(inherit — configured tools)";
|
|
66
|
+
const TOOL_POLICIES: ToolPolicy[] = ["configured", "none"];
|
|
57
67
|
const TOOLS_PRESETS = ["read,bash,grep,find,ls", "read,grep,find,ls", "read"];
|
|
58
68
|
const DEFAULT_BATCH_PARALLEL = 4;
|
|
59
69
|
const MAX_BATCH_PARALLEL = 6;
|
|
@@ -66,7 +76,11 @@ const TIER_PURPOSE: Record<Tier, string> = {
|
|
|
66
76
|
interface PrReviewConfig {
|
|
67
77
|
/** Tier label -> model spec (e.g. "anthropic/model", "openai/model:high"). */
|
|
68
78
|
tiers: Partial<Record<Tier, string>>;
|
|
69
|
-
/**
|
|
79
|
+
/** Tier label -> ordered fallback model specs used for quota/capacity failures. */
|
|
80
|
+
fallbacks: Partial<Record<Tier, string[]>>;
|
|
81
|
+
/** Optional tier-level policy used when a tool call does not override it. */
|
|
82
|
+
toolPolicies: Partial<Record<Tier, ToolPolicy>>;
|
|
83
|
+
/** Tools granted to review subagents whose effective policy is configured. */
|
|
70
84
|
tools: string[];
|
|
71
85
|
}
|
|
72
86
|
|
|
@@ -91,6 +105,37 @@ function readConfigFile(filePath: string): Partial<PrReviewConfig> {
|
|
|
91
105
|
}
|
|
92
106
|
}
|
|
93
107
|
|
|
108
|
+
function normalizeFallbacks(
|
|
109
|
+
raw: Partial<Record<Tier, unknown>> | undefined,
|
|
110
|
+
preserveEmpty = false,
|
|
111
|
+
): Partial<Record<Tier, string[]>> {
|
|
112
|
+
const out: Partial<Record<Tier, string[]>> = {};
|
|
113
|
+
if (!raw || typeof raw !== "object") return out;
|
|
114
|
+
for (const tier of TIERS) {
|
|
115
|
+
const value = raw[tier];
|
|
116
|
+
if (value === undefined) continue;
|
|
117
|
+
const list = Array.isArray(value)
|
|
118
|
+
? value.map((v) => String(v).trim()).filter(Boolean)
|
|
119
|
+
: typeof value === "string"
|
|
120
|
+
? value.split(",").map((v) => v.trim()).filter(Boolean)
|
|
121
|
+
: [];
|
|
122
|
+
if (list.length > 0 || preserveEmpty) out[tier] = [...new Set(list)];
|
|
123
|
+
}
|
|
124
|
+
return out;
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
function normalizeToolPolicies(
|
|
128
|
+
raw: Partial<Record<Tier, unknown>> | undefined,
|
|
129
|
+
): Partial<Record<Tier, ToolPolicy>> {
|
|
130
|
+
const out: Partial<Record<Tier, ToolPolicy>> = {};
|
|
131
|
+
if (!raw || typeof raw !== "object") return out;
|
|
132
|
+
for (const tier of TIERS) {
|
|
133
|
+
const policy = normalizeToolPolicy(raw[tier]);
|
|
134
|
+
if (policy) out[tier] = policy;
|
|
135
|
+
}
|
|
136
|
+
return out;
|
|
137
|
+
}
|
|
138
|
+
|
|
94
139
|
/** User config, overlaid by project config when the project is trusted. */
|
|
95
140
|
function loadConfig(ctx: Pick<ExtensionContext, "cwd" | "isProjectTrusted">): PrReviewConfig {
|
|
96
141
|
const user = readConfigFile(userConfigPath());
|
|
@@ -102,6 +147,11 @@ function loadConfig(ctx: Pick<ExtensionContext, "cwd" | "isProjectTrusted">): Pr
|
|
|
102
147
|
}
|
|
103
148
|
return {
|
|
104
149
|
tiers: { ...(user.tiers ?? {}), ...(project.tiers ?? {}) },
|
|
150
|
+
fallbacks: { ...normalizeFallbacks(user.fallbacks, true), ...normalizeFallbacks(project.fallbacks, true) },
|
|
151
|
+
toolPolicies: {
|
|
152
|
+
...normalizeToolPolicies(user.toolPolicies),
|
|
153
|
+
...normalizeToolPolicies(project.toolPolicies),
|
|
154
|
+
},
|
|
105
155
|
tools: project.tools ?? user.tools ?? DEFAULT_TOOLS,
|
|
106
156
|
};
|
|
107
157
|
}
|
|
@@ -109,7 +159,12 @@ function loadConfig(ctx: Pick<ExtensionContext, "cwd" | "isProjectTrusted">): Pr
|
|
|
109
159
|
/** User-level config only (the scope the config command edits), with defaults. */
|
|
110
160
|
function readUserConfig(): PrReviewConfig {
|
|
111
161
|
const raw = readConfigFile(userConfigPath());
|
|
112
|
-
return {
|
|
162
|
+
return {
|
|
163
|
+
tiers: { ...(raw.tiers ?? {}) },
|
|
164
|
+
fallbacks: normalizeFallbacks(raw.fallbacks),
|
|
165
|
+
toolPolicies: normalizeToolPolicies(raw.toolPolicies),
|
|
166
|
+
tools: raw.tools ?? [...DEFAULT_TOOLS],
|
|
167
|
+
};
|
|
113
168
|
}
|
|
114
169
|
|
|
115
170
|
function writeUserConfig(next: PrReviewConfig): string {
|
|
@@ -119,19 +174,50 @@ function writeUserConfig(next: PrReviewConfig): string {
|
|
|
119
174
|
return filePath;
|
|
120
175
|
}
|
|
121
176
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
177
|
+
interface ModelAttempt {
|
|
178
|
+
spec?: string;
|
|
179
|
+
usedTier?: Tier;
|
|
180
|
+
kind: "primary" | "fallback" | "nearest" | "default";
|
|
181
|
+
fallbackIndex?: number;
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
const NEAREST_TIER_ORDER: Record<Tier, Tier[]> = {
|
|
185
|
+
light: ["light", "medium", "heavy"],
|
|
186
|
+
medium: ["medium", "heavy", "light"],
|
|
187
|
+
heavy: ["heavy", "medium", "light"],
|
|
188
|
+
};
|
|
189
|
+
|
|
190
|
+
/** Resolve the ordered model attempts for a tier, preserving nearest-tier/default behavior when the tier is unset. */
|
|
191
|
+
function resolveModelAttempts(config: PrReviewConfig, tier: Tier): ModelAttempt[] {
|
|
192
|
+
const attempts: ModelAttempt[] = [];
|
|
193
|
+
const seen = new Set<string>();
|
|
194
|
+
const add = (attempt: ModelAttempt) => {
|
|
195
|
+
const key = attempt.spec ?? "__pi_default__";
|
|
196
|
+
if (seen.has(key)) return;
|
|
197
|
+
seen.add(key);
|
|
198
|
+
attempts.push(attempt);
|
|
130
199
|
};
|
|
131
|
-
|
|
132
|
-
|
|
200
|
+
|
|
201
|
+
const primary = config.tiers[tier];
|
|
202
|
+
if (primary) {
|
|
203
|
+
add({ spec: primary, usedTier: tier, kind: "primary" });
|
|
204
|
+
} else {
|
|
205
|
+
let foundNearest = false;
|
|
206
|
+
for (const candidate of NEAREST_TIER_ORDER[tier]) {
|
|
207
|
+
const spec = config.tiers[candidate];
|
|
208
|
+
if (!spec) continue;
|
|
209
|
+
add({ spec, usedTier: candidate, kind: "nearest" });
|
|
210
|
+
foundNearest = true;
|
|
211
|
+
break;
|
|
212
|
+
}
|
|
213
|
+
if (!foundNearest) add({ kind: "default" });
|
|
133
214
|
}
|
|
134
|
-
|
|
215
|
+
|
|
216
|
+
for (const [index, spec] of (config.fallbacks[tier] ?? []).entries()) {
|
|
217
|
+
add({ spec, usedTier: tier, kind: "fallback", fallbackIndex: index });
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
return attempts;
|
|
135
221
|
}
|
|
136
222
|
|
|
137
223
|
// ---------------------------------------------------------------------------
|
|
@@ -295,6 +381,20 @@ interface SubagentPassRequest {
|
|
|
295
381
|
tier: Tier;
|
|
296
382
|
objective: string;
|
|
297
383
|
context?: string;
|
|
384
|
+
toolPolicy?: ToolPolicy;
|
|
385
|
+
}
|
|
386
|
+
|
|
387
|
+
interface ModelAttemptReport {
|
|
388
|
+
kind: ModelAttempt["kind"];
|
|
389
|
+
spec?: string;
|
|
390
|
+
usedTier?: Tier;
|
|
391
|
+
model?: string;
|
|
392
|
+
exitCode: number;
|
|
393
|
+
status: "completed" | "failed";
|
|
394
|
+
retryable: boolean;
|
|
395
|
+
elapsedMs: number;
|
|
396
|
+
stopReason?: string;
|
|
397
|
+
errorMessage?: string;
|
|
298
398
|
}
|
|
299
399
|
|
|
300
400
|
interface SubagentPassResult {
|
|
@@ -309,39 +409,67 @@ interface SubagentPassResult {
|
|
|
309
409
|
stderr?: string;
|
|
310
410
|
stopReason?: string;
|
|
311
411
|
errorMessage?: string;
|
|
412
|
+
attempts: ModelAttemptReport[];
|
|
413
|
+
fallbackUsed: boolean;
|
|
414
|
+
retryableFailure: boolean;
|
|
415
|
+
toolPolicy: ToolPolicy;
|
|
416
|
+
elapsedMs: number;
|
|
312
417
|
}
|
|
313
418
|
|
|
314
|
-
function
|
|
315
|
-
if (spec) {
|
|
316
|
-
return
|
|
317
|
-
? `tier=${tier} model=${spec}`
|
|
318
|
-
: `tier=${tier} (not configured; using ${usedTier} model=${spec})`;
|
|
419
|
+
function noticeForAttempt(tier: Tier, attempt: ModelAttempt): string {
|
|
420
|
+
if (!attempt.spec) {
|
|
421
|
+
return `tier=${tier} (no tier configured; using pi default model — run /pr-review-config to set tiers)`;
|
|
319
422
|
}
|
|
320
|
-
|
|
423
|
+
if (attempt.kind === "fallback") {
|
|
424
|
+
return `tier=${tier} fallback #${(attempt.fallbackIndex ?? 0) + 1} model=${attempt.spec}`;
|
|
425
|
+
}
|
|
426
|
+
if (attempt.usedTier && attempt.usedTier !== tier) {
|
|
427
|
+
return `tier=${tier} (not configured; using ${attempt.usedTier} model=${attempt.spec})`;
|
|
428
|
+
}
|
|
429
|
+
return `tier=${tier} model=${attempt.spec}`;
|
|
430
|
+
}
|
|
431
|
+
|
|
432
|
+
function noticeForResult(tier: Tier, attempts: ModelAttemptReport[], finalNotice: string): string {
|
|
433
|
+
const failedBeforeSuccess = attempts.filter((a) => a.status === "failed").length;
|
|
434
|
+
if (failedBeforeSuccess > 0 && attempts.some((a) => a.status === "completed")) {
|
|
435
|
+
return `${finalNotice} (after ${failedBeforeSuccess} retry${failedBeforeSuccess === 1 ? "" : "ies"})`;
|
|
436
|
+
}
|
|
437
|
+
return finalNotice;
|
|
321
438
|
}
|
|
322
439
|
|
|
323
440
|
function buildPassTask(objective: string, context: string | undefined): string {
|
|
324
441
|
return context ? `Objective: ${objective}\n\n--- PR context / diff ---\n${context}` : `Objective: ${objective}`;
|
|
325
442
|
}
|
|
326
443
|
|
|
327
|
-
|
|
444
|
+
function isRetryableModelFailure(result: RunResult): boolean {
|
|
445
|
+
if (result.stopReason === "aborted") return false;
|
|
446
|
+
const haystack = [result.errorMessage, result.stderr, result.text, result.stopReason]
|
|
447
|
+
.filter(Boolean)
|
|
448
|
+
.join("\n")
|
|
449
|
+
.toLowerCase();
|
|
450
|
+
if (!haystack) return false;
|
|
451
|
+
return /(?:\b429\b|rate[\s_-]*limit(?:ed)?|too many requests|quota|usage[\s_-]*limit|insufficient[\s_-]*quota|resource[\s_-]*exhausted|out of credits|insufficient credits|credit limit|billing quota|billing hard limit|at capacity|overloaded|temporarily unavailable|model (?:is )?(?:temporarily|currently) unavailable|service unavailable|try again later)/i.test(
|
|
452
|
+
haystack,
|
|
453
|
+
);
|
|
454
|
+
}
|
|
455
|
+
|
|
456
|
+
async function runSubagentAttempt(
|
|
328
457
|
config: PrReviewConfig,
|
|
329
458
|
ctx: Pick<ExtensionContext, "cwd">,
|
|
330
459
|
pass: SubagentPassRequest,
|
|
460
|
+
attempt: ModelAttempt,
|
|
461
|
+
toolPolicy: ToolPolicy,
|
|
331
462
|
signal: AbortSignal | undefined,
|
|
332
463
|
onText?: (text: string) => void,
|
|
333
|
-
): Promise<
|
|
334
|
-
const tier = pass.tier;
|
|
335
|
-
const { spec, usedTier } = resolveModelSpec(config, tier);
|
|
336
|
-
const notice = noticeForTier(tier, spec, usedTier);
|
|
464
|
+
): Promise<{ result: RunResult; notice: string; elapsedMs: number }> {
|
|
337
465
|
let tmp: { dir: string; filePath: string } | undefined;
|
|
338
|
-
|
|
466
|
+
const startedAt = Date.now();
|
|
339
467
|
try {
|
|
340
|
-
const args =
|
|
341
|
-
if (spec) args.push("--model", spec);
|
|
342
|
-
|
|
468
|
+
const args = buildReviewBaseArgs();
|
|
469
|
+
if (attempt.spec) args.push("--model", attempt.spec);
|
|
470
|
+
appendToolPolicyArgs(args, toolPolicy, config.tools);
|
|
343
471
|
|
|
344
|
-
tmp = await writeTempPrompt(tier, buildSubagentSystemPrompt(tier));
|
|
472
|
+
tmp = await writeTempPrompt(pass.tier, buildSubagentSystemPrompt(pass.tier));
|
|
345
473
|
args.push("--append-system-prompt", tmp.filePath);
|
|
346
474
|
args.push(buildPassTask(pass.objective, pass.context));
|
|
347
475
|
|
|
@@ -349,32 +477,12 @@ async function runSubagentPass(
|
|
|
349
477
|
const result = await runReviewSubprocess(invocation.command, invocation.args, ctx.cwd, signal, (text) => {
|
|
350
478
|
onText?.(text);
|
|
351
479
|
});
|
|
352
|
-
|
|
353
|
-
const failed = result.exitCode !== 0 || result.stopReason === "error" || result.stopReason === "aborted";
|
|
354
|
-
return {
|
|
355
|
-
id: pass.id ?? tier,
|
|
356
|
-
tier,
|
|
357
|
-
usedTier,
|
|
358
|
-
model: result.model ?? spec,
|
|
359
|
-
exitCode: result.exitCode,
|
|
360
|
-
status: failed ? "failed" : "completed",
|
|
361
|
-
notice,
|
|
362
|
-
text: result.text || (failed ? "" : "NO FINDINGS."),
|
|
363
|
-
stderr: result.stderr || undefined,
|
|
364
|
-
stopReason: result.stopReason,
|
|
365
|
-
errorMessage: result.errorMessage,
|
|
366
|
-
};
|
|
480
|
+
return { result, notice: noticeForAttempt(pass.tier, attempt), elapsedMs: Date.now() - startedAt };
|
|
367
481
|
} catch (e) {
|
|
368
482
|
return {
|
|
369
|
-
|
|
370
|
-
tier,
|
|
371
|
-
|
|
372
|
-
model: spec,
|
|
373
|
-
exitCode: 1,
|
|
374
|
-
status: "failed",
|
|
375
|
-
notice,
|
|
376
|
-
text: "",
|
|
377
|
-
errorMessage: errMessage(e),
|
|
483
|
+
result: { text: "", exitCode: 1, stderr: "", errorMessage: errMessage(e) },
|
|
484
|
+
notice: noticeForAttempt(pass.tier, attempt),
|
|
485
|
+
elapsedMs: Date.now() - startedAt,
|
|
378
486
|
};
|
|
379
487
|
} finally {
|
|
380
488
|
if (tmp) {
|
|
@@ -387,6 +495,93 @@ async function runSubagentPass(
|
|
|
387
495
|
}
|
|
388
496
|
}
|
|
389
497
|
|
|
498
|
+
async function runSubagentPass(
|
|
499
|
+
config: PrReviewConfig,
|
|
500
|
+
ctx: Pick<ExtensionContext, "cwd">,
|
|
501
|
+
pass: SubagentPassRequest,
|
|
502
|
+
signal: AbortSignal | undefined,
|
|
503
|
+
onText?: (text: string) => void,
|
|
504
|
+
): Promise<SubagentPassResult> {
|
|
505
|
+
const startedAt = Date.now();
|
|
506
|
+
const tier = pass.tier;
|
|
507
|
+
const toolPolicy = resolveToolPolicy(pass.toolPolicy, config.toolPolicies[tier]);
|
|
508
|
+
const attempts = resolveModelAttempts(config, tier);
|
|
509
|
+
const reports: ModelAttemptReport[] = [];
|
|
510
|
+
let lastResult: RunResult | undefined;
|
|
511
|
+
let lastNotice = noticeForAttempt(tier, attempts[0]!);
|
|
512
|
+
|
|
513
|
+
for (const attempt of attempts) {
|
|
514
|
+
const { result, notice, elapsedMs } = await runSubagentAttempt(
|
|
515
|
+
config,
|
|
516
|
+
ctx,
|
|
517
|
+
pass,
|
|
518
|
+
attempt,
|
|
519
|
+
toolPolicy,
|
|
520
|
+
signal,
|
|
521
|
+
onText,
|
|
522
|
+
);
|
|
523
|
+
const failed = result.exitCode !== 0 || result.stopReason === "error" || result.stopReason === "aborted";
|
|
524
|
+
const retryable = failed && isRetryableModelFailure(result);
|
|
525
|
+
lastResult = result;
|
|
526
|
+
lastNotice = notice;
|
|
527
|
+
reports.push({
|
|
528
|
+
kind: attempt.kind,
|
|
529
|
+
spec: attempt.spec,
|
|
530
|
+
usedTier: attempt.usedTier,
|
|
531
|
+
model: result.model ?? attempt.spec,
|
|
532
|
+
exitCode: result.exitCode,
|
|
533
|
+
status: failed ? "failed" : "completed",
|
|
534
|
+
retryable,
|
|
535
|
+
elapsedMs,
|
|
536
|
+
stopReason: result.stopReason,
|
|
537
|
+
errorMessage: result.errorMessage,
|
|
538
|
+
});
|
|
539
|
+
|
|
540
|
+
if (!failed) {
|
|
541
|
+
return {
|
|
542
|
+
id: pass.id ?? tier,
|
|
543
|
+
tier,
|
|
544
|
+
usedTier: attempt.usedTier,
|
|
545
|
+
model: result.model ?? attempt.spec,
|
|
546
|
+
exitCode: result.exitCode,
|
|
547
|
+
status: "completed",
|
|
548
|
+
notice: noticeForResult(tier, reports, notice),
|
|
549
|
+
text: result.text || "NO FINDINGS.",
|
|
550
|
+
stderr: result.stderr || undefined,
|
|
551
|
+
stopReason: result.stopReason,
|
|
552
|
+
errorMessage: result.errorMessage,
|
|
553
|
+
attempts: reports,
|
|
554
|
+
fallbackUsed: attempt.kind === "fallback" || reports.length > 1,
|
|
555
|
+
retryableFailure: false,
|
|
556
|
+
toolPolicy,
|
|
557
|
+
elapsedMs: Date.now() - startedAt,
|
|
558
|
+
};
|
|
559
|
+
}
|
|
560
|
+
|
|
561
|
+
if (!retryable || signal?.aborted) break;
|
|
562
|
+
}
|
|
563
|
+
|
|
564
|
+
const final = lastResult ?? { text: "", exitCode: 1, stderr: "", errorMessage: "No model attempts were available." };
|
|
565
|
+
return {
|
|
566
|
+
id: pass.id ?? tier,
|
|
567
|
+
tier,
|
|
568
|
+
usedTier: reports.at(-1)?.usedTier,
|
|
569
|
+
model: reports.at(-1)?.model,
|
|
570
|
+
exitCode: final.exitCode,
|
|
571
|
+
status: "failed",
|
|
572
|
+
notice: noticeForResult(tier, reports, lastNotice),
|
|
573
|
+
text: final.text || "",
|
|
574
|
+
stderr: final.stderr || undefined,
|
|
575
|
+
stopReason: final.stopReason,
|
|
576
|
+
errorMessage: final.errorMessage,
|
|
577
|
+
attempts: reports,
|
|
578
|
+
fallbackUsed: reports.length > 1,
|
|
579
|
+
retryableFailure: reports.at(-1)?.retryable ?? false,
|
|
580
|
+
toolPolicy,
|
|
581
|
+
elapsedMs: Date.now() - startedAt,
|
|
582
|
+
};
|
|
583
|
+
}
|
|
584
|
+
|
|
390
585
|
function normalizeMaxParallel(raw: unknown, count: number): number {
|
|
391
586
|
if (count <= 0) return 0;
|
|
392
587
|
const requested = typeof raw === "number" && Number.isFinite(raw) ? Math.floor(raw) : DEFAULT_BATCH_PARALLEL;
|
|
@@ -410,6 +605,13 @@ async function runWithConcurrency<T, R>(
|
|
|
410
605
|
return results;
|
|
411
606
|
}
|
|
412
607
|
|
|
608
|
+
function formatAttemptSummary(result: SubagentPassResult): string {
|
|
609
|
+
if (result.attempts.length <= 1) return "";
|
|
610
|
+
return `attempts: ${result.attempts
|
|
611
|
+
.map((a) => `${a.status === "completed" ? "✓" : a.retryable ? "↻" : "✗"} ${a.spec ?? "pi default"}`)
|
|
612
|
+
.join(" → ")}`;
|
|
613
|
+
}
|
|
614
|
+
|
|
413
615
|
function formatBatchResults(results: SubagentPassResult[], maxParallel: number): string {
|
|
414
616
|
const failed = results.filter((r) => r.status === "failed");
|
|
415
617
|
const lines = [
|
|
@@ -421,7 +623,16 @@ function formatBatchResults(results: SubagentPassResult[], maxParallel: number):
|
|
|
421
623
|
);
|
|
422
624
|
}
|
|
423
625
|
for (const result of results) {
|
|
424
|
-
lines.push(
|
|
626
|
+
lines.push(
|
|
627
|
+
"",
|
|
628
|
+
`## Pass: ${result.id}`,
|
|
629
|
+
`status: ${result.status}`,
|
|
630
|
+
`tool_policy: ${result.toolPolicy}`,
|
|
631
|
+
`elapsed_ms: ${result.elapsedMs}`,
|
|
632
|
+
result.notice,
|
|
633
|
+
);
|
|
634
|
+
const attemptSummary = formatAttemptSummary(result);
|
|
635
|
+
if (attemptSummary) lines.push(attemptSummary);
|
|
425
636
|
if (result.status === "failed") {
|
|
426
637
|
const detail = result.errorMessage || result.stderr || result.text || "(no output)";
|
|
427
638
|
lines.push(`error: ${detail}`);
|
|
@@ -457,13 +668,19 @@ const ReviewSubagentParams = Type.Object({
|
|
|
457
668
|
"Shared context for the subagent, typically the PR title/description plus the unified diff to review.",
|
|
458
669
|
}),
|
|
459
670
|
),
|
|
671
|
+
tool_policy: Type.Optional(
|
|
672
|
+
StringEnum(["none", "configured"] as const, {
|
|
673
|
+
description:
|
|
674
|
+
"Tool access for this pass. none emits --no-tools; configured uses the configured allowlist. Request override wins over tier config; omission preserves configured legacy behavior unless tier policy is set.",
|
|
675
|
+
}),
|
|
676
|
+
),
|
|
460
677
|
});
|
|
461
678
|
|
|
462
679
|
const ReviewSubagentsParams = Type.Object({
|
|
463
680
|
context: Type.Optional(
|
|
464
681
|
Type.String({
|
|
465
682
|
description:
|
|
466
|
-
"Shared PR context for every pass, typically PR title/body/metadata,
|
|
683
|
+
"Shared PR context for every pass, typically PR title/body/metadata, cross-cutting requirements, and the unified diff. Put specialist-only convention excerpts in the medium pass context.",
|
|
467
684
|
}),
|
|
468
685
|
),
|
|
469
686
|
max_parallel: Type.Optional(
|
|
@@ -490,6 +707,12 @@ const ReviewSubagentsParams = Type.Object({
|
|
|
490
707
|
description: "Optional pass-specific context appended after the shared context.",
|
|
491
708
|
}),
|
|
492
709
|
),
|
|
710
|
+
tool_policy: Type.Optional(
|
|
711
|
+
StringEnum(["none", "configured"] as const, {
|
|
712
|
+
description:
|
|
713
|
+
"Tool access for this pass. none emits --no-tools; configured uses the configured allowlist. The resolved policy remains fixed across fallback model attempts.",
|
|
714
|
+
}),
|
|
715
|
+
),
|
|
493
716
|
}),
|
|
494
717
|
{
|
|
495
718
|
description: "Independent review passes to run concurrently. Results are returned in this same order.",
|
|
@@ -520,7 +743,12 @@ export default function (pi: ExtensionAPI) {
|
|
|
520
743
|
const result = await runSubagentPass(
|
|
521
744
|
loadConfig(ctx),
|
|
522
745
|
ctx,
|
|
523
|
-
{
|
|
746
|
+
{
|
|
747
|
+
tier,
|
|
748
|
+
objective: params.objective,
|
|
749
|
+
context: params.context,
|
|
750
|
+
toolPolicy: normalizeToolPolicy(params.tool_policy),
|
|
751
|
+
},
|
|
524
752
|
signal,
|
|
525
753
|
(text) => onUpdate?.({ content: [{ type: "text", text }] }),
|
|
526
754
|
);
|
|
@@ -535,6 +763,11 @@ export default function (pi: ExtensionAPI) {
|
|
|
535
763
|
usedTier: result.usedTier,
|
|
536
764
|
model: result.model,
|
|
537
765
|
exitCode: result.exitCode,
|
|
766
|
+
fallbackUsed: result.fallbackUsed,
|
|
767
|
+
retryableFailure: result.retryableFailure,
|
|
768
|
+
toolPolicy: result.toolPolicy,
|
|
769
|
+
elapsedMs: result.elapsedMs,
|
|
770
|
+
attempts: result.attempts,
|
|
538
771
|
},
|
|
539
772
|
};
|
|
540
773
|
}
|
|
@@ -546,6 +779,10 @@ export default function (pi: ExtensionAPI) {
|
|
|
546
779
|
usedTier: result.usedTier,
|
|
547
780
|
model: result.model,
|
|
548
781
|
exitCode: result.exitCode,
|
|
782
|
+
fallbackUsed: result.fallbackUsed,
|
|
783
|
+
toolPolicy: result.toolPolicy,
|
|
784
|
+
elapsedMs: result.elapsedMs,
|
|
785
|
+
attempts: result.attempts,
|
|
549
786
|
},
|
|
550
787
|
};
|
|
551
788
|
},
|
|
@@ -587,6 +824,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
587
824
|
tier,
|
|
588
825
|
objective: pass.objective,
|
|
589
826
|
context: combineContexts(sharedContext, typeof pass.context === "string" ? pass.context : undefined),
|
|
827
|
+
toolPolicy: normalizeToolPolicy(pass.tool_policy),
|
|
590
828
|
};
|
|
591
829
|
});
|
|
592
830
|
const maxParallel = normalizeMaxParallel(params.max_parallel, passes.length);
|
|
@@ -622,6 +860,11 @@ export default function (pi: ExtensionAPI) {
|
|
|
622
860
|
status: r.status,
|
|
623
861
|
stopReason: r.stopReason,
|
|
624
862
|
errorMessage: r.errorMessage,
|
|
863
|
+
fallbackUsed: r.fallbackUsed,
|
|
864
|
+
retryableFailure: r.retryableFailure,
|
|
865
|
+
toolPolicy: r.toolPolicy,
|
|
866
|
+
elapsedMs: r.elapsedMs,
|
|
867
|
+
attempts: r.attempts,
|
|
625
868
|
outputChars: r.text.length,
|
|
626
869
|
})),
|
|
627
870
|
},
|
|
@@ -630,7 +873,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
630
873
|
});
|
|
631
874
|
|
|
632
875
|
pi.registerCommand("pr-review-config", {
|
|
633
|
-
description: "Open the review-tier settings menu, or show/set light/medium/heavy models for /pr-review",
|
|
876
|
+
description: "Open the review-tier settings menu, or show/set light/medium/heavy models and fallbacks for /pr-review",
|
|
634
877
|
handler: async (args, ctx) => {
|
|
635
878
|
const raw = (args ?? "").trim();
|
|
636
879
|
const parsed = parseConfigArgs(raw);
|
|
@@ -640,7 +883,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
640
883
|
}
|
|
641
884
|
|
|
642
885
|
try {
|
|
643
|
-
// Direct set: `/pr-review-config light=... heavy=...`
|
|
886
|
+
// Direct set: `/pr-review-config light=... heavy=... heavy_fallbacks=...`
|
|
644
887
|
if (parsed.hasChanges) {
|
|
645
888
|
const next = applyConfigPatch(readUserConfig(), parsed.patch);
|
|
646
889
|
writeUserConfig(next);
|
|
@@ -701,7 +944,15 @@ function shouldOpenConfigMenu(args: string, ctx: ExtensionContext): boolean {
|
|
|
701
944
|
|
|
702
945
|
const CONFIG_COMPLETIONS: Array<{ value: string; label: string }> = [
|
|
703
946
|
...TIERS.map((t) => ({ value: `${t}=`, label: `${t}=<model> — ${TIER_PURPOSE[t]}` })),
|
|
704
|
-
|
|
947
|
+
...TIERS.map((t) => ({
|
|
948
|
+
value: `${t}_fallbacks=`,
|
|
949
|
+
label: `${t}_fallbacks=<model1,model2> — retry chain for quota/rate-limit failures`,
|
|
950
|
+
})),
|
|
951
|
+
...TIERS.map((t) => ({
|
|
952
|
+
value: `${t}_tool_policy=`,
|
|
953
|
+
label: `${t}_tool_policy=<none|configured|unset> — default tool access when a pass does not override it`,
|
|
954
|
+
})),
|
|
955
|
+
{ value: "tools=", label: "tools=read,bash,grep,find,ls — allowlist used by configured policy" },
|
|
705
956
|
{ value: "show", label: "show — print the current tier config" },
|
|
706
957
|
];
|
|
707
958
|
|
|
@@ -734,13 +985,45 @@ function splitConfigArgs(input: string): string[] {
|
|
|
734
985
|
return tokens;
|
|
735
986
|
}
|
|
736
987
|
|
|
988
|
+
function splitCommaList(value: string): string[] {
|
|
989
|
+
return value.split(",").map((s) => s.trim()).filter(Boolean);
|
|
990
|
+
}
|
|
991
|
+
|
|
992
|
+
function isFallbackKey(key: string): boolean {
|
|
993
|
+
for (const tier of TIERS) {
|
|
994
|
+
if (key === `${tier}_fallbacks` || key === `${tier}-fallbacks` || key === `${tier}.fallbacks`) {
|
|
995
|
+
return true;
|
|
996
|
+
}
|
|
997
|
+
}
|
|
998
|
+
return false;
|
|
999
|
+
}
|
|
1000
|
+
|
|
1001
|
+
function isToolPolicyKey(key: string): boolean {
|
|
1002
|
+
for (const tier of TIERS) {
|
|
1003
|
+
if (
|
|
1004
|
+
key === `${tier}_tool_policy` ||
|
|
1005
|
+
key === `${tier}-tool-policy` ||
|
|
1006
|
+
key === `${tier}.toolpolicy`
|
|
1007
|
+
) {
|
|
1008
|
+
return true;
|
|
1009
|
+
}
|
|
1010
|
+
}
|
|
1011
|
+
return false;
|
|
1012
|
+
}
|
|
1013
|
+
|
|
1014
|
+
function tierFromCompoundKey(key: string): Tier {
|
|
1015
|
+
return key.split(/[_\-.]/)[0] as Tier;
|
|
1016
|
+
}
|
|
1017
|
+
|
|
737
1018
|
interface ConfigPatch {
|
|
738
1019
|
tiers: Partial<Record<Tier, string | null>>;
|
|
1020
|
+
fallbacks: Partial<Record<Tier, string[] | null>>;
|
|
1021
|
+
toolPolicies: Partial<Record<Tier, ToolPolicy | null>>;
|
|
739
1022
|
tools?: string[];
|
|
740
1023
|
}
|
|
741
1024
|
|
|
742
1025
|
function parseConfigArgs(args: string): { patch: ConfigPatch; hasChanges: boolean; errors: string[] } {
|
|
743
|
-
const patch: ConfigPatch = { tiers: {} };
|
|
1026
|
+
const patch: ConfigPatch = { tiers: {}, fallbacks: {}, toolPolicies: {} };
|
|
744
1027
|
const errors: string[] = [];
|
|
745
1028
|
const tokens = splitConfigArgs(args).filter((t) => !["show", "get", "current"].includes(t.toLowerCase()));
|
|
746
1029
|
for (const token of tokens) {
|
|
@@ -753,28 +1036,65 @@ function parseConfigArgs(args: string): { patch: ConfigPatch; hasChanges: boolea
|
|
|
753
1036
|
const value = token.slice(eq + 1);
|
|
754
1037
|
if ((TIERS as string[]).includes(key)) {
|
|
755
1038
|
patch.tiers[key as Tier] = value === "" || value === "unset" ? null : value;
|
|
1039
|
+
} else if (isFallbackKey(key)) {
|
|
1040
|
+
const tier = tierFromCompoundKey(key);
|
|
1041
|
+
patch.fallbacks[tier] = value === "" || value === "unset" || value === "none" ? null : splitCommaList(value);
|
|
1042
|
+
} else if (isToolPolicyKey(key)) {
|
|
1043
|
+
const tier = tierFromCompoundKey(key);
|
|
1044
|
+
if (value === "" || value === "unset" || value === "inherit") patch.toolPolicies[tier] = null;
|
|
1045
|
+
else {
|
|
1046
|
+
const policy = normalizeToolPolicy(value);
|
|
1047
|
+
if (policy) patch.toolPolicies[tier] = policy;
|
|
1048
|
+
else errors.push(`invalid ${key} "${value}" (expected none|configured|unset)`);
|
|
1049
|
+
}
|
|
756
1050
|
} else if (key === "tools") {
|
|
757
|
-
patch.tools = value
|
|
1051
|
+
patch.tools = splitCommaList(value);
|
|
758
1052
|
} else {
|
|
759
|
-
errors.push(
|
|
1053
|
+
errors.push(
|
|
1054
|
+
`unknown key "${key}" (expected light|medium|heavy|<tier>_fallbacks|<tier>_tool_policy|tools)`,
|
|
1055
|
+
);
|
|
760
1056
|
}
|
|
761
1057
|
}
|
|
762
|
-
const hasChanges =
|
|
1058
|
+
const hasChanges =
|
|
1059
|
+
Object.keys(patch.tiers).length > 0 ||
|
|
1060
|
+
Object.keys(patch.fallbacks).length > 0 ||
|
|
1061
|
+
Object.keys(patch.toolPolicies).length > 0 ||
|
|
1062
|
+
patch.tools !== undefined;
|
|
763
1063
|
return { patch, hasChanges, errors };
|
|
764
1064
|
}
|
|
765
1065
|
|
|
766
1066
|
function applyConfigPatch(base: PrReviewConfig, patch: ConfigPatch): PrReviewConfig {
|
|
767
|
-
const next: PrReviewConfig = {
|
|
1067
|
+
const next: PrReviewConfig = {
|
|
1068
|
+
tiers: { ...base.tiers },
|
|
1069
|
+
fallbacks: normalizeFallbacks(base.fallbacks),
|
|
1070
|
+
toolPolicies: normalizeToolPolicies(base.toolPolicies),
|
|
1071
|
+
tools: [...base.tools],
|
|
1072
|
+
};
|
|
768
1073
|
for (const tier of TIERS) {
|
|
769
|
-
if (
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
1074
|
+
if (tier in patch.tiers) {
|
|
1075
|
+
const value = patch.tiers[tier];
|
|
1076
|
+
if (value === null || value === undefined) delete next.tiers[tier];
|
|
1077
|
+
else next.tiers[tier] = value;
|
|
1078
|
+
}
|
|
1079
|
+
if (tier in patch.fallbacks) {
|
|
1080
|
+
const value = patch.fallbacks[tier];
|
|
1081
|
+
if (value === null || value === undefined || value.length === 0) delete next.fallbacks[tier];
|
|
1082
|
+
else next.fallbacks[tier] = [...new Set(value)];
|
|
1083
|
+
}
|
|
1084
|
+
if (tier in patch.toolPolicies) {
|
|
1085
|
+
const value = patch.toolPolicies[tier];
|
|
1086
|
+
if (value === null || value === undefined) delete next.toolPolicies[tier];
|
|
1087
|
+
else next.toolPolicies[tier] = value;
|
|
1088
|
+
}
|
|
773
1089
|
}
|
|
774
1090
|
if (patch.tools) next.tools = patch.tools;
|
|
775
1091
|
return next;
|
|
776
1092
|
}
|
|
777
1093
|
|
|
1094
|
+
function formatModelList(models: string[] | undefined): string {
|
|
1095
|
+
return models && models.length > 0 ? `\`${models.join(",")}\`` : "_none_";
|
|
1096
|
+
}
|
|
1097
|
+
|
|
778
1098
|
function summarizeConfig(
|
|
779
1099
|
user: PrReviewConfig,
|
|
780
1100
|
effective: PrReviewConfig,
|
|
@@ -799,7 +1119,14 @@ function summarizeConfig(
|
|
|
799
1119
|
(t) =>
|
|
800
1120
|
`| \`${t}\` | ${user.tiers[t] ? `\`${user.tiers[t]}\`` : "_unset_"} | ${effective.tiers[t] ? `\`${effective.tiers[t]}\`` : "_pi default_"} | ${TIER_PURPOSE[t]} |`,
|
|
801
1121
|
),
|
|
802
|
-
`| \`tools\` | \`${user.tools.join(",")}\` | \`${effective.tools.join(",")}\` |
|
|
1122
|
+
`| \`tools\` | \`${user.tools.join(",")}\` | \`${effective.tools.join(",")}\` | allowlist used when policy is \`configured\` |`,
|
|
1123
|
+
"",
|
|
1124
|
+
"| Tier | Your fallbacks | Effective fallbacks | Tool policy |",
|
|
1125
|
+
"|---|---|---|---|",
|
|
1126
|
+
...TIERS.map(
|
|
1127
|
+
(t) =>
|
|
1128
|
+
`| \`${t}\` | ${formatModelList(user.fallbacks[t])} | ${formatModelList(effective.fallbacks[t])} | ${user.toolPolicies[t] ? `\`${user.toolPolicies[t]}\`` : "_inherit configured_"} → \`${effective.toolPolicies[t] ?? "configured"}\` |`,
|
|
1129
|
+
),
|
|
803
1130
|
"",
|
|
804
1131
|
`User config: \`${userConfigPath()}\``,
|
|
805
1132
|
];
|
|
@@ -810,7 +1137,11 @@ function summarizeConfig(
|
|
|
810
1137
|
"- Open the settings menu: `/pr-review-config`",
|
|
811
1138
|
"- Print this summary: `/pr-review-config show`",
|
|
812
1139
|
"- Set directly: `/pr-review-config light=provider/model heavy=provider/model:high`",
|
|
1140
|
+
"- Set fallback chain: `/pr-review-config heavy_fallbacks=provider/backup:high,provider/backup2`",
|
|
1141
|
+
"- Set tier tool policy: `/pr-review-config light_tool_policy=none`",
|
|
813
1142
|
"- Clear a tier: `/pr-review-config medium=unset`",
|
|
1143
|
+
"- Clear fallback chain: `/pr-review-config heavy_fallbacks=unset`",
|
|
1144
|
+
"- Restore legacy tier tool behavior: `/pr-review-config light_tool_policy=unset`",
|
|
814
1145
|
"- A `<model>` is any pi model pattern (`provider/model` or `provider/model:thinking`).",
|
|
815
1146
|
);
|
|
816
1147
|
return lines.join("\n");
|
|
@@ -828,10 +1159,14 @@ const MODEL_LIST_ROWS = 10;
|
|
|
828
1159
|
* search UX and fuzzyFilter's slash-token matching, so typing a model name substring
|
|
829
1160
|
* finds `provider/model`). The SelectList is rebuilt as the query changes.
|
|
830
1161
|
*/
|
|
831
|
-
function buildModelSubmenu(
|
|
1162
|
+
function buildModelSubmenu(
|
|
1163
|
+
available: string[],
|
|
1164
|
+
currentSpec: string | undefined,
|
|
1165
|
+
unsetDescription = "Fall back to the nearest configured tier, then the pi default.",
|
|
1166
|
+
) {
|
|
832
1167
|
return (_currentValue: string, done: (selectedValue?: string) => void) => {
|
|
833
1168
|
const allItems: SelectItem[] = [
|
|
834
|
-
{ value: "__unset__", label: UNSET, description:
|
|
1169
|
+
{ value: "__unset__", label: UNSET, description: unsetDescription },
|
|
835
1170
|
...available.map((spec) => ({ value: spec, label: spec })),
|
|
836
1171
|
];
|
|
837
1172
|
const theme = getSelectListTheme();
|
|
@@ -890,14 +1225,35 @@ function configMenuItems(cfg: PrReviewConfig, available: string[]): SettingItem[
|
|
|
890
1225
|
currentValue: cfg.tiers[tier] ?? UNSET,
|
|
891
1226
|
submenu: buildModelSubmenu(available, cfg.tiers[tier]),
|
|
892
1227
|
}));
|
|
1228
|
+
const fallbackItems: SettingItem[] = TIERS.map((tier) => ({
|
|
1229
|
+
id: `${tier}_fallbacks`,
|
|
1230
|
+
label: `${tier} fallback model`,
|
|
1231
|
+
description:
|
|
1232
|
+
"Retry model for quota/rate-limit/capacity failures. Press Enter to set one fallback; use key=value for chains.",
|
|
1233
|
+
currentValue: cfg.fallbacks[tier]?.join(",") || "(none)",
|
|
1234
|
+
submenu: buildModelSubmenu(
|
|
1235
|
+
available,
|
|
1236
|
+
cfg.fallbacks[tier]?.[0],
|
|
1237
|
+
"Clear this tier's fallback chain. Use key=value form to set multiple fallbacks.",
|
|
1238
|
+
),
|
|
1239
|
+
}));
|
|
1240
|
+
const policyItems: SettingItem[] = TIERS.map((tier) => ({
|
|
1241
|
+
id: `${tier}_tool_policy`,
|
|
1242
|
+
label: `${tier} tool policy`,
|
|
1243
|
+
description: "Default when a pass does not explicitly set tool_policy. Enter/Space cycles values.",
|
|
1244
|
+
currentValue: cfg.toolPolicies[tier] ?? INHERIT_TOOL_POLICY,
|
|
1245
|
+
values: [INHERIT_TOOL_POLICY, ...TOOL_POLICIES],
|
|
1246
|
+
}));
|
|
893
1247
|
const current = cfg.tools.join(",");
|
|
894
1248
|
const toolValues = [current, ...TOOLS_PRESETS.filter((p) => p !== current)];
|
|
895
1249
|
return [
|
|
896
1250
|
...tierItems,
|
|
1251
|
+
...fallbackItems,
|
|
1252
|
+
...policyItems,
|
|
897
1253
|
{
|
|
898
1254
|
id: "tools",
|
|
899
|
-
label: "
|
|
900
|
-
description: "Tools
|
|
1255
|
+
label: "configured tool allowlist",
|
|
1256
|
+
description: "Tools available when effective policy is configured. Enter/Space cycles presets.",
|
|
901
1257
|
currentValue: current,
|
|
902
1258
|
values: toolValues,
|
|
903
1259
|
},
|
|
@@ -920,7 +1276,11 @@ async function showConfigMenu(
|
|
|
920
1276
|
|
|
921
1277
|
let settingsList: SettingsList;
|
|
922
1278
|
const refresh = () => {
|
|
923
|
-
for (const tier of TIERS)
|
|
1279
|
+
for (const tier of TIERS) {
|
|
1280
|
+
settingsList.updateValue(tier, draft.tiers[tier] ?? UNSET);
|
|
1281
|
+
settingsList.updateValue(`${tier}_fallbacks`, draft.fallbacks[tier]?.join(",") || "(none)");
|
|
1282
|
+
settingsList.updateValue(`${tier}_tool_policy`, draft.toolPolicies[tier] ?? INHERIT_TOOL_POLICY);
|
|
1283
|
+
}
|
|
924
1284
|
settingsList.updateValue("tools", draft.tools.join(","));
|
|
925
1285
|
};
|
|
926
1286
|
|
|
@@ -928,15 +1288,32 @@ async function showConfigMenu(
|
|
|
928
1288
|
if ((TIERS as string[]).includes(id)) {
|
|
929
1289
|
if (newValue === "__unset__") delete draft.tiers[id as Tier];
|
|
930
1290
|
else draft.tiers[id as Tier] = newValue;
|
|
1291
|
+
} else if (isFallbackKey(id)) {
|
|
1292
|
+
const tier = tierFromCompoundKey(id);
|
|
1293
|
+
if (newValue === "__unset__") delete draft.fallbacks[tier];
|
|
1294
|
+
else draft.fallbacks[tier] = [newValue];
|
|
1295
|
+
} else if (isToolPolicyKey(id)) {
|
|
1296
|
+
const tier = tierFromCompoundKey(id);
|
|
1297
|
+
if (newValue === INHERIT_TOOL_POLICY) delete draft.toolPolicies[tier];
|
|
1298
|
+
else {
|
|
1299
|
+
const policy = normalizeToolPolicy(newValue);
|
|
1300
|
+
if (policy) draft.toolPolicies[tier] = policy;
|
|
1301
|
+
}
|
|
931
1302
|
} else if (id === "tools") {
|
|
932
|
-
draft.tools = newValue
|
|
1303
|
+
draft.tools = splitCommaList(newValue);
|
|
933
1304
|
} else {
|
|
934
1305
|
return;
|
|
935
1306
|
}
|
|
936
1307
|
try {
|
|
937
1308
|
writeUserConfig(draft);
|
|
938
1309
|
refresh();
|
|
939
|
-
const shown = id === "tools"
|
|
1310
|
+
const shown = id === "tools"
|
|
1311
|
+
? draft.tools.join(",")
|
|
1312
|
+
: isFallbackKey(id)
|
|
1313
|
+
? (draft.fallbacks[tierFromCompoundKey(id)]?.join(",") ?? "(none)")
|
|
1314
|
+
: isToolPolicyKey(id)
|
|
1315
|
+
? (draft.toolPolicies[tierFromCompoundKey(id)] ?? INHERIT_TOOL_POLICY)
|
|
1316
|
+
: (draft.tiers[id as Tier] ?? UNSET);
|
|
940
1317
|
ctx.ui.notify(`PR review config: ${id} = ${shown}`, "info");
|
|
941
1318
|
} catch (e) {
|
|
942
1319
|
ctx.ui.notify(`pr-review-config failed: ${errMessage(e)}`, "error");
|
|
@@ -946,7 +1323,7 @@ async function showConfigMenu(
|
|
|
946
1323
|
|
|
947
1324
|
settingsList = new SettingsList(
|
|
948
1325
|
configMenuItems(draft, available),
|
|
949
|
-
|
|
1326
|
+
12,
|
|
950
1327
|
getSettingsListTheme(),
|
|
951
1328
|
(id, newValue) => persist(id, newValue),
|
|
952
1329
|
() => done(undefined),
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
export type ToolPolicy = "none" | "configured";
|
|
2
|
+
|
|
3
|
+
/** Isolated review subprocesses receive all review context explicitly. */
|
|
4
|
+
export function buildReviewBaseArgs(): string[] {
|
|
5
|
+
return ["--mode", "json", "-p", "--no-session", "--no-context-files"];
|
|
6
|
+
}
|
|
7
|
+
|
|
8
|
+
export function normalizeToolPolicy(value: unknown): ToolPolicy | undefined {
|
|
9
|
+
return value === "none" || value === "configured" ? value : undefined;
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
/** Request override wins, then explicit tier config, then legacy configured behavior. */
|
|
13
|
+
export function resolveToolPolicy(
|
|
14
|
+
requested: ToolPolicy | undefined,
|
|
15
|
+
configured: ToolPolicy | undefined,
|
|
16
|
+
): ToolPolicy {
|
|
17
|
+
return requested ?? configured ?? "configured";
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
/** Preserve legacy configured behavior; only `none` explicitly disables every tool. */
|
|
21
|
+
export function appendToolPolicyArgs(
|
|
22
|
+
args: string[],
|
|
23
|
+
policy: ToolPolicy,
|
|
24
|
+
configuredTools: string[],
|
|
25
|
+
): string[] {
|
|
26
|
+
if (policy === "none") {
|
|
27
|
+
args.push("--no-tools");
|
|
28
|
+
} else if (configuredTools.length > 0) {
|
|
29
|
+
args.push("--tools", configuredTools.join(","));
|
|
30
|
+
}
|
|
31
|
+
return args;
|
|
32
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-pr-review",
|
|
3
|
-
"version": "1.1
|
|
3
|
+
"version": "1.3.1",
|
|
4
4
|
"description": "Model-agnostic GitHub PR code review prompt for pi. Pass a PR number; it derives the base and head branches from the PR in the current repo, reviews the diff, and returns structured JSON findings.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package",
|
|
@@ -14,6 +14,9 @@
|
|
|
14
14
|
"type": "git",
|
|
15
15
|
"url": "git+https://github.com/10ego/pi-pr-review.git"
|
|
16
16
|
},
|
|
17
|
+
"scripts": {
|
|
18
|
+
"test": "bun test"
|
|
19
|
+
},
|
|
17
20
|
"peerDependencies": {
|
|
18
21
|
"@earendil-works/pi-ai": "*",
|
|
19
22
|
"@earendil-works/pi-coding-agent": "*",
|
|
@@ -21,11 +24,11 @@
|
|
|
21
24
|
"typebox": "*"
|
|
22
25
|
},
|
|
23
26
|
"pi": {
|
|
24
|
-
"prompts": [
|
|
25
|
-
"./prompts"
|
|
26
|
-
],
|
|
27
27
|
"extensions": [
|
|
28
28
|
"./extensions"
|
|
29
|
+
],
|
|
30
|
+
"prompts": [
|
|
31
|
+
"./prompts"
|
|
29
32
|
]
|
|
30
33
|
}
|
|
31
34
|
}
|
package/prompts/pr-review.md
CHANGED
|
@@ -20,14 +20,16 @@ You are the **orchestrator**. You own all GitHub access, skip decisions, convent
|
|
|
20
20
|
|
|
21
21
|
If the `review_subagents` batch tool is available, prefer it over multiple single-pass calls. Fetch PR metadata and the unified diff once, gather any relevant convention-file excerpts, then call `review_subagents` with shared `context`, `max_parallel`, and ordered `passes`. This guarantees bounded parallel fan-out instead of depending on whether the tool interface runs separate calls concurrently. Use these pass assignments:
|
|
22
22
|
|
|
23
|
-
| Pass id | Tier label | Scope |
|
|
24
|
-
|
|
25
|
-
| `overview` | `light` | Step 3 overview, strengths, and high-level risk areas |
|
|
26
|
-
| `conventions-maintainability` | `medium` | Step 5 convention compliance, readability, maintainability, test gaps, and nits |
|
|
27
|
-
| `correctness` | `heavy` | Step 5 compile/parse, logic, error handling, lifecycle, concurrency, and edge-case defects |
|
|
28
|
-
| `security-performance` | `heavy` | Step 5 security vulnerabilities and performance regressions |
|
|
23
|
+
| Pass id | Tier label | Tool policy | Scope |
|
|
24
|
+
|---------|------------|-------------|-------|
|
|
25
|
+
| `overview` | `light` | `none` | Step 3 overview, strengths, and high-level risk areas |
|
|
26
|
+
| `conventions-maintainability` | `medium` | `configured` | Step 5 convention compliance, readability, maintainability, test gaps, and nits |
|
|
27
|
+
| `correctness` | `heavy` | `configured` | Step 5 compile/parse, logic, error handling, lifecycle, concurrency, and edge-case defects |
|
|
28
|
+
| `security-performance` | `heavy` | `configured` | Step 5 security vulnerabilities and performance regressions |
|
|
29
29
|
|
|
30
|
-
Call `review_subagents` with `{ context, max_parallel: 4, passes: [...] }
|
|
30
|
+
Call `review_subagents` with `{ context, max_parallel: 4, passes: [...] }`, setting each pass's `tool_policy` exactly as shown. Shared `context` contains the PR title/description, relevant metadata, complete unified diff, and strictly cross-cutting review requirements. Put convention-file paths and excerpts only in the `conventions-maintainability` pass's own `context`; they are appended to shared context by the tool. This avoids sending specialist-only material to every model while still allowing the medium reviewer to inspect surrounding files when maintainability or test-gap analysis requires it.
|
|
31
|
+
|
|
32
|
+
If deterministic context assembly reports that required input is missing or truncated before dispatch (for example, an incomplete diff or unreadable applicable convention file), set `tool_policy: configured` for that affected pass instead of `none`. Do not ask the model to self-diagnose and rerun. If any pass returns `status: failed`, treat the review evidence as incomplete: rerun the failed pass with `review_subagent` using the same tool policy, or perform that pass inline before finalizing.
|
|
31
33
|
|
|
32
34
|
If `review_subagents` is unavailable but `review_subagent` is available, run the same pass assignments as individual `review_subagent` calls; emit independent calls in the same turn when the interface supports parallel tool calls. If neither subagent tool is available, perform every pass yourself inline on the current session model.
|
|
33
35
|
|
|
@@ -78,25 +80,25 @@ List (do not dump contents yet) the repository convention files that could gover
|
|
|
78
80
|
- The root convention file (`CLAUDE.md`, and/or `AGENTS.md` if present).
|
|
79
81
|
- Any convention file living in a directory that contains a file modified by this PR.
|
|
80
82
|
|
|
81
|
-
When you evaluate compliance for a given changed file, only apply convention files that share that file's path or a parent path. Read a convention file's contents when a changed file falls under its scope, and include the relevant rule excerpts (with file paths) in the
|
|
83
|
+
When you evaluate compliance for a given changed file, only apply convention files that share that file's path or a parent path. Read a convention file's contents when a changed file falls under its scope, and include the relevant rule excerpts (with file paths) only in the `conventions-maintainability` pass-specific `context`, not the batch's shared context.
|
|
82
84
|
|
|
83
85
|
## Step 3 — Overview & strengths (`light` reviewer)
|
|
84
86
|
|
|
85
87
|
Write a short **overview** (1–3 short paragraphs) of what the PR does and how, grounded in the diff and PR title/description — enough to understand author intent. Also collect a list of genuine **strengths** (good tests, nice consolidation, correct reuse of helpers, etc.) and any high-level risk areas for the specialist passes. Strengths are part of the output, and understanding intent is what lets you tell an intentional change from a bug.
|
|
86
88
|
|
|
87
|
-
When `review_subagents` is available, include this as the `overview` pass in the same batch as the Step 5 review passes instead of waiting for a separate sequential call.
|
|
89
|
+
When `review_subagents` is available, include this as the `overview` pass in the same batch as the Step 5 review passes instead of waiting for a separate sequential call. Set `tool_policy: none`; the complete PR metadata and diff are already supplied.
|
|
88
90
|
|
|
89
91
|
## Step 4 — (reserved)
|
|
90
92
|
|
|
91
93
|
## Step 5 — Review passes (dispatch by tier, then merge results)
|
|
92
94
|
|
|
93
|
-
Run these independent passes over the diff, each on its tier reviewer (or inline if subagent tools are unavailable). Give every pass the shared PR
|
|
95
|
+
Run these independent passes over the diff, each on its tier reviewer (or inline if subagent tools are unavailable). Give every pass the shared PR metadata and complete diff from Step 1; give only the medium pass the relevant convention-file excerpts from Step 2. Instruct each pass to report issues **at every severity, including nits**, within its own scope. Do **not** ask every pass to audit everything; the objective and tool-policy boundaries below reduce duplicate work while preserving coverage.
|
|
94
96
|
|
|
95
97
|
When `review_subagents` is available, dispatch the Step 3 `overview` pass and these Step 5 passes in one batch with `max_parallel: 4`, then combine the ordered outputs:
|
|
96
98
|
|
|
97
|
-
1. **Convention, readability & maintainability pass — `medium` reviewer
|
|
98
|
-
2. **Bug & correctness pass — `heavy` reviewer
|
|
99
|
-
3. **Security & performance pass — `heavy` reviewer
|
|
99
|
+
1. **Convention, readability & maintainability pass — `medium` reviewer, `tool_policy: configured`.** Audit changed lines against the in-scope convention excerpts supplied in this pass's context. Flag violations (quote the rule), softer deviations from documented style as nits, naming/dead-code/comment/typo issues, minor duplication, test gaps, and "worth confirming" observations (e.g. "no current callers — confirm intended", "confirm this generated file came from codegen not a hand-edit"). Use configured tools to inspect surrounding files, tests, callers, or generated-file context when needed; do not modify files or duplicate deep correctness/security analysis unless needed to explain a convention/maintainability issue.
|
|
100
|
+
2. **Bug & correctness pass — `heavy` reviewer, `tool_policy: configured`.** Hunt for defects in the introduced code: compile/parse failures, logic errors, wrong results, off-by-one, error handling, resource/lifecycle, concurrency, and edge cases. Include lower-severity correctness smells and missing edge cases as P2/P3. Do not duplicate pure style nits covered by the medium pass. Use configured repository-context tools when surrounding files or callers are needed to confirm impact; reviewing remains non-modifying even if the allowlist includes `bash`.
|
|
101
|
+
3. **Security & performance pass — `heavy` reviewer, `tool_policy: configured`.** Look for security issues (injection, authz, secrets, unsafe deserialization) and performance regressions in the changed code. Note minor ones too. Do not duplicate ordinary correctness/readability findings unless they are security/performance relevant. Use configured tools when repository context is needed to validate a candidate; do not modify files.
|
|
100
102
|
|
|
101
103
|
## Step 6 — Verification (best-effort, orchestrator-owned, non-destructive)
|
|
102
104
|
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
import { describe, expect, test } from "bun:test";
|
|
2
|
+
import {
|
|
3
|
+
appendToolPolicyArgs,
|
|
4
|
+
buildReviewBaseArgs,
|
|
5
|
+
normalizeToolPolicy,
|
|
6
|
+
resolveToolPolicy,
|
|
7
|
+
} from "../lib/pr-review-policy.ts";
|
|
8
|
+
|
|
9
|
+
describe("tool policy resolution", () => {
|
|
10
|
+
test("request override wins over tier config", () => {
|
|
11
|
+
expect(resolveToolPolicy("none", "configured")).toBe("none");
|
|
12
|
+
expect(resolveToolPolicy("configured", "none")).toBe("configured");
|
|
13
|
+
});
|
|
14
|
+
|
|
15
|
+
test("tier config applies when request omits policy", () => {
|
|
16
|
+
expect(resolveToolPolicy(undefined, "none")).toBe("none");
|
|
17
|
+
});
|
|
18
|
+
|
|
19
|
+
test("omission preserves legacy configured behavior", () => {
|
|
20
|
+
expect(resolveToolPolicy(undefined, undefined)).toBe("configured");
|
|
21
|
+
});
|
|
22
|
+
|
|
23
|
+
test("normalization rejects unknown values", () => {
|
|
24
|
+
expect(normalizeToolPolicy("none")).toBe("none");
|
|
25
|
+
expect(normalizeToolPolicy("configured")).toBe("configured");
|
|
26
|
+
expect(normalizeToolPolicy("auto")).toBeUndefined();
|
|
27
|
+
});
|
|
28
|
+
});
|
|
29
|
+
|
|
30
|
+
describe("tool policy argv", () => {
|
|
31
|
+
test("base args isolate explicit review context", () => {
|
|
32
|
+
expect(buildReviewBaseArgs()).toEqual([
|
|
33
|
+
"--mode",
|
|
34
|
+
"json",
|
|
35
|
+
"-p",
|
|
36
|
+
"--no-session",
|
|
37
|
+
"--no-context-files",
|
|
38
|
+
]);
|
|
39
|
+
});
|
|
40
|
+
|
|
41
|
+
test("none emits explicit --no-tools", () => {
|
|
42
|
+
const args = ["--mode", "json"];
|
|
43
|
+
expect(appendToolPolicyArgs(args, "none", ["read", "bash"])).toEqual([
|
|
44
|
+
"--mode",
|
|
45
|
+
"json",
|
|
46
|
+
"--no-tools",
|
|
47
|
+
]);
|
|
48
|
+
});
|
|
49
|
+
|
|
50
|
+
test("configured emits the configured allowlist", () => {
|
|
51
|
+
const args = ["--mode", "json"];
|
|
52
|
+
expect(appendToolPolicyArgs(args, "configured", ["read", "grep"])).toEqual([
|
|
53
|
+
"--mode",
|
|
54
|
+
"json",
|
|
55
|
+
"--tools",
|
|
56
|
+
"read,grep",
|
|
57
|
+
]);
|
|
58
|
+
});
|
|
59
|
+
|
|
60
|
+
test("configured with an empty list preserves legacy omitted-flag behavior", () => {
|
|
61
|
+
const args = ["--mode", "json"];
|
|
62
|
+
expect(appendToolPolicyArgs(args, "configured", [])).toEqual(["--mode", "json"]);
|
|
63
|
+
});
|
|
64
|
+
|
|
65
|
+
test("one resolved policy can be reused across fallback attempts", () => {
|
|
66
|
+
const policy = resolveToolPolicy("none", "configured");
|
|
67
|
+
const first = appendToolPolicyArgs([], policy, ["read"]);
|
|
68
|
+
const fallback = appendToolPolicyArgs([], policy, ["read"]);
|
|
69
|
+
expect(first).toEqual(["--no-tools"]);
|
|
70
|
+
expect(fallback).toEqual(["--no-tools"]);
|
|
71
|
+
});
|
|
72
|
+
});
|