pi-prompt-template-model-enhanced 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +77 -2
- package/args.ts +83 -1
- package/best-of-n-presets.ts +256 -0
- package/index.ts +78 -15
- package/package.json +2 -1
- package/prompt-dry-run.ts +1 -1
- package/prompt-loader.ts +11 -4
- package/prompt-validation.ts +52 -1
- package/skills/prompt-template-authoring/SKILL.md +18 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,30 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.14.0](https://github.com/Nabsku/pi-prompt-template-model-enhanced/compare/v0.13.1...v0.14.0) (2026-06-22)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
* add best-of-n preset resolution ([e0f45e6](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/e0f45e64490e6874e6693e7b6450954d7f9dde64))
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Bug Fixes
|
|
12
|
+
|
|
13
|
+
* address best-of-n preset review comments ([0dfdf80](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/0dfdf8065c0a241f0786f7271800b1c877b117cc))
|
|
14
|
+
* close preset review edge cases ([df85bf3](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/df85bf3d86c2f43bcbec8d9fdb150e2fde37e61f))
|
|
15
|
+
* close preset validation review gaps ([1822061](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/1822061ae90ee9b5ee2d7ea222ce9cabc9698e79))
|
|
16
|
+
* harden best-of-n preset overrides ([caa0981](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/caa0981936d5e264f5e2dd334d9f0e1abe59aa59))
|
|
17
|
+
* harden project preset resolution ([04cddc9](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/04cddc919256b0c40b28a51b67c185a592f899fb))
|
|
18
|
+
* prevent invalid project presets from falling through ([8b0f589](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/8b0f5899ca5ebd1350218cd0d87d24af891a9656))
|
|
19
|
+
* validate presets from prompt cwd ([ce97729](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/ce9772947bf31abbc1fc4481bdc13bf51385da62))
|
|
20
|
+
|
|
21
|
+
## [0.13.1](https://github.com/Nabsku/pi-prompt-template-model-enhanced/compare/v0.13.0...v0.13.1) (2026-06-21)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
### Bug Fixes
|
|
25
|
+
|
|
26
|
+
* harden prompt-library parity coverage ([1666691](https://github.com/Nabsku/pi-prompt-template-model-enhanced/commit/16666916eff1757cefa4b743cdbb18ea704b95a5))
|
|
27
|
+
|
|
3
28
|
## [0.13.0](https://github.com/Nabsku/pi-prompt-template-model-enhanced/compare/v0.12.0...v0.13.0) (2026-06-21)
|
|
4
29
|
|
|
5
30
|
### Prompt library
|
package/README.md
CHANGED
|
@@ -32,6 +32,41 @@ No more manually switching models, copying standard instructions between prompts
|
|
|
32
32
|
- **Pi-native TUI**: browse templates and inspect dry-run output interactively in Pi TUI mode without typing template names.
|
|
33
33
|
- **Execution control**: loops, model rotation, fresh context, boomerang collapse, delegated subagents, chains, and best-of-N compare prompts.
|
|
34
34
|
|
|
35
|
+
## Differences from upstream
|
|
36
|
+
|
|
37
|
+
This package is an enhanced fork of [`pi-prompt-template-model`](https://github.com/nicobailon/pi-prompt-template-model), with compatibility preserved where practical and new workflow features kept opt-in where possible.
|
|
38
|
+
|
|
39
|
+
### Additions
|
|
40
|
+
|
|
41
|
+
- Prompt includes from shared Markdown partials and prompt-library roots.
|
|
42
|
+
- Multiple skill injection via `skills`, plus constrained wildcard skill selectors.
|
|
43
|
+
- Dry-run previews and a Pi-native TUI picker/inspector for prompt templates.
|
|
44
|
+
- Loop controls, model rotation, fresh context, and boomerang context collapse.
|
|
45
|
+
- Chain templates, deterministic prompt steps, delegated subagents, and parallel delegation helpers.
|
|
46
|
+
- Best-of-N compare prompts with reusable preset catalogs.
|
|
47
|
+
- Richer `/validate-prompts` diagnostics, include graphs, and source summaries.
|
|
48
|
+
|
|
49
|
+
### Behavior changes and stricter validation
|
|
50
|
+
|
|
51
|
+
- Invalid prompt config is reported and skipped more consistently instead of running with silently degraded behavior.
|
|
52
|
+
- Prompt-library commands have extra trust checks because they can come from project-local reusable libraries.
|
|
53
|
+
- Duplicate prompt and prompt-library precedence is deterministic and reported with diagnostics.
|
|
54
|
+
- Runtime flags are scoped to prompt types that support them; for example, `--preset` only affects compare prompts.
|
|
55
|
+
- Project best-of-N presets require session approval and invalid project presets fail closed instead of falling back to same-named user presets.
|
|
56
|
+
|
|
57
|
+
### Breaking or migration notes
|
|
58
|
+
|
|
59
|
+
- Compare prompt templates must use nested `bestOfN:` frontmatter. Top-level `workers`, `reviewers`, and `finalApplier` fields are rejected with migration diagnostics.
|
|
60
|
+
- `skills:` must be a list. Use `skill: name` for a single skill.
|
|
61
|
+
- Some fields that upstream accepted loosely are now type-checked before registration, including prompt includes, chain declarations, loop values, cwd paths, compare lineups, and preset catalogs.
|
|
62
|
+
- Best-of-N presets intentionally cannot set execution-policy fields such as `task`, `taskSuffix`, `cwd`, `finalApplier`, `worktree`, dirty/report/commit policy, or other prompt-owned behavior.
|
|
63
|
+
|
|
64
|
+
### Small tweaks
|
|
65
|
+
|
|
66
|
+
- Better command descriptions and runtime warnings for ignored or unsupported flags.
|
|
67
|
+
- More deterministic model/thinking fallback behavior in dry-run and execution paths.
|
|
68
|
+
- Packaged examples, authoring skill docs, and validation output are kept aligned with shipped behavior.
|
|
69
|
+
|
|
35
70
|
## Installation
|
|
36
71
|
|
|
37
72
|
```bash
|
|
@@ -182,7 +217,8 @@ All fields are optional. Templates that don't use any extension features (no `mo
|
|
|
182
217
|
| `subagent` | — | Delegate execution to a subagent instead of running in the current session. `true` uses the default `delegate` agent; a string value like `reviewer` targets that specific agent. Requires [pi-subagents](https://github.com/nicobailon/pi-subagents/). |
|
|
183
218
|
| `inheritContext` | `false` | Only meaningful with `subagent`. When `true`, the subagent receives a fork of the current conversation context instead of starting fresh. |
|
|
184
219
|
| `parallel` | — | Delegated prompts only. Repeats the same subagent in parallel `N` times. Each copy gets a slot header like `[Parallel subagent 2/3]` prepended to the task. Must be an integer greater than or equal to 2. |
|
|
185
|
-
| `bestOfN` | — | Compare templates only. Nested compare authoring block with `workers`, `reviewers`, optional `finalApplier`, and optional `worktree`. Top-level compare fields are not supported in templates. |
|
|
220
|
+
| `bestOfN` | — | Compare templates only. Nested compare authoring block with `workers`, `reviewers`, optional `preset`, optional `finalApplier`, and optional `worktree`. Top-level compare fields are not supported in templates. |
|
|
221
|
+
| `bestOfN.preset` | — | Name of a best-of-N preset from `~/.pi/agent/best-of-n-presets.json` or `<compare-cwd>/.pi/best-of-n-presets.json`. Presets can supply worker/reviewer agents, models, counts, `defaultModel`, and `maxModelCalls`; prompt templates still own task text, `cwd`, final apply, dirty/report/commit policy, and other execution policy. |
|
|
186
222
|
| `bestOfN.workers` | — | Ordered worker lineup used for the worker phase. Each slot object supports optional `agent`/`subagent`, optional `model`, optional `task`, optional `taskSuffix`, optional `cwd`, and optional `count`. If both `agent` and `subagent` are omitted, the default agent is `delegate`. |
|
|
187
223
|
| `bestOfN.reviewers` | — | Ordered reviewer lineup used after worker aggregation. Slot shape matches workers. If both `agent` and `subagent` are omitted, the default agent is `reviewer`. |
|
|
188
224
|
| `bestOfN.finalApplier` | — | Optional single-slot final apply phase that edits the real branch after reviewers. Supports optional `agent`/`subagent`, optional `model`, optional `task`, and optional `taskSuffix`. If both `agent` and `subagent` are omitted, the default agent is `delegate`. `count` and `cwd` are not supported. Requires `bestOfN.worktree: true` at runtime. |
|
|
@@ -588,6 +624,7 @@ Prompt-template frontmatter authoring uses `bestOfN:`. Runtime overrides stay on
|
|
|
588
624
|
- `--workers=<json-array>` / `--reviewers=<json-array>` replace the corresponding frontmatter lineup.
|
|
589
625
|
- `--workers-append=<json-array>` / `--reviewers-append=<json-array>` append to the corresponding lineup.
|
|
590
626
|
- `--final-applier=<json-object-or-one-element-array>` replaces the optional final apply slot.
|
|
627
|
+
- `--preset=<name>` / `--preset <name>` selects a best-of-N preset for compare prompts only; it is ignored on non-compare prompts.
|
|
591
628
|
|
|
592
629
|
Each worker/reviewer JSON array entry must be an object with either `subagent` or `agent`, plus optional `model`, `task`, `taskSuffix`, `cwd`, and `count`. In worker slots, `"subagent": true` maps to `delegate`. In reviewer slots, `"subagent": true` maps to `reviewer`. `--final-applier=` accepts one slot object (or a one-element array) with `subagent`/`agent`, optional `model`, optional `task`, and optional `taskSuffix`; for this final slot, `"subagent": true` maps to `delegate`, and both `count` and `cwd` are not supported.
|
|
593
630
|
|
|
@@ -615,7 +652,45 @@ Compare prompt templates are authored under `bestOfN:`. Top-level `workers`, `re
|
|
|
615
652
|
3. Optional final apply phase: if `finalApplier` is configured, run one delegated apply step on the real compare repo (`compareCwd`) to pick a winner or synthesize/cherry-pick and apply the final patch.
|
|
616
653
|
4. If all reviewers fail but `finalApplier` exists, the final apply step still runs with fallback context from workers plus reviewer failure summaries.
|
|
617
654
|
|
|
618
|
-
Worker/reviewer lineups are fully configurable from `bestOfN` frontmatter or runtime overrides, so there is no fixed three-model worker assumption. If a compare prompt omits `bestOfN.workers`, it falls back to one `delegate` worker using the current/main model. If it omits `bestOfN.reviewers`, it falls back to one `reviewer` slot. `bestOfN.finalApplier` is optional, and compare runs reject an effective final applier unless `bestOfN.worktree: true` is set.
|
|
655
|
+
Worker/reviewer lineups are fully configurable from `bestOfN` frontmatter, presets, or runtime overrides, so there is no fixed three-model worker assumption. If a compare prompt omits `bestOfN.workers`, it falls back to one `delegate` worker using the current/main model. If it omits `bestOfN.reviewers`, it falls back to one `reviewer` slot. `bestOfN.finalApplier` is optional, and compare runs reject an effective final applier unless `bestOfN.worktree: true` is set.
|
|
656
|
+
|
|
657
|
+
### Best-of-N presets
|
|
658
|
+
|
|
659
|
+
Presets keep expensive lineup choices reusable without letting project config own prompt policy. Define them in either place:
|
|
660
|
+
|
|
661
|
+
- User presets: `~/.pi/agent/best-of-n-presets.json`
|
|
662
|
+
- Project presets: `<compare-cwd>/.pi/best-of-n-presets.json`
|
|
663
|
+
|
|
664
|
+
Project presets override user presets of the same name, but running a project preset asks for session approval. Compare prompts that set `cwd`, runtime `--cwd`, or use `parallel-patch-compare-at-path` resolve project presets from the effective compare cwd. `/validate-prompts` mirrors prompt `cwd` where it can be known statically.
|
|
665
|
+
|
|
666
|
+
```json
|
|
667
|
+
{
|
|
668
|
+
"presets": {
|
|
669
|
+
"quick": {
|
|
670
|
+
"description": "Two cheap workers, one reviewer",
|
|
671
|
+
"defaultModel": "openai-codex/gpt-5.4-mini:low",
|
|
672
|
+
"maxModelCalls": 3,
|
|
673
|
+
"workers": [{ "agent": "delegate", "count": 2 }],
|
|
674
|
+
"reviewers": [{ "agent": "reviewer" }]
|
|
675
|
+
}
|
|
676
|
+
}
|
|
677
|
+
}
|
|
678
|
+
```
|
|
679
|
+
|
|
680
|
+
Use from a prompt:
|
|
681
|
+
|
|
682
|
+
```yaml
|
|
683
|
+
bestOfN:
|
|
684
|
+
preset: quick
|
|
685
|
+
```
|
|
686
|
+
|
|
687
|
+
Or at runtime:
|
|
688
|
+
|
|
689
|
+
```bash
|
|
690
|
+
/best-of-n --preset quick refactor the parser
|
|
691
|
+
```
|
|
692
|
+
|
|
693
|
+
Preset slot fields are intentionally limited to `agent`/`subagent`, `model`, and `count`. Presets cannot set `task`, `taskSuffix`, `cwd`, `finalApplier`, `worktree`, dirty/report/commit behavior, or other execution policy. Invalid selected presets fail closed instead of falling back to same-named user presets, and `maxModelCalls` caps the expanded worker + reviewer calls before any subagents start.
|
|
619
694
|
|
|
620
695
|
For same-model best-of-N, use `count: N` on one worker slot:
|
|
621
696
|
|
package/args.ts
CHANGED
|
@@ -21,6 +21,7 @@ export interface SubagentOverrideExtraction {
|
|
|
21
21
|
override?: SubagentOverride;
|
|
22
22
|
cwd?: string;
|
|
23
23
|
model?: string;
|
|
24
|
+
preset?: string;
|
|
24
25
|
fork?: boolean;
|
|
25
26
|
}
|
|
26
27
|
|
|
@@ -240,10 +241,65 @@ export function extractWorktreeFlag(argsString: string): { args: string; worktre
|
|
|
240
241
|
return { args, worktree: found };
|
|
241
242
|
}
|
|
242
243
|
|
|
244
|
+
const LINEUP_OVERRIDE_VALUE_PREFIXES = [
|
|
245
|
+
"--workers=",
|
|
246
|
+
"--workers-append=",
|
|
247
|
+
"--reviewers=",
|
|
248
|
+
"--reviewers-append=",
|
|
249
|
+
"--final-applier=",
|
|
250
|
+
] as const;
|
|
251
|
+
|
|
252
|
+
function readJsonLikeFlagTokenEnd(argsString: string, tokenStart: number): number {
|
|
253
|
+
const plainEnd = () => {
|
|
254
|
+
let end = tokenStart;
|
|
255
|
+
while (end < argsString.length && !/\s/.test(argsString[end])) end++;
|
|
256
|
+
return end;
|
|
257
|
+
};
|
|
258
|
+
const prefix = LINEUP_OVERRIDE_VALUE_PREFIXES.find((candidate) => argsString.startsWith(candidate, tokenStart));
|
|
259
|
+
if (!prefix) return plainEnd();
|
|
260
|
+
let i = tokenStart + prefix.length;
|
|
261
|
+
if (argsString[i] === '"' || argsString[i] === "'") {
|
|
262
|
+
const quote = argsString[i++];
|
|
263
|
+
while (i < argsString.length) {
|
|
264
|
+
if (argsString[i] === "\\") {
|
|
265
|
+
i += 2;
|
|
266
|
+
continue;
|
|
267
|
+
}
|
|
268
|
+
if (argsString[i] === quote) return i + 1;
|
|
269
|
+
i++;
|
|
270
|
+
}
|
|
271
|
+
return i;
|
|
272
|
+
}
|
|
273
|
+
const opener = argsString[i];
|
|
274
|
+
const closer = opener === "[" ? "]" : opener === "{" ? "}" : undefined;
|
|
275
|
+
if (!closer) return plainEnd();
|
|
276
|
+
let depth = 0;
|
|
277
|
+
let inString = false;
|
|
278
|
+
for (; i < argsString.length; i++) {
|
|
279
|
+
const char = argsString[i];
|
|
280
|
+
if (inString) {
|
|
281
|
+
if (char === "\\") i++;
|
|
282
|
+
else if (char === '"') inString = false;
|
|
283
|
+
continue;
|
|
284
|
+
}
|
|
285
|
+
if (char === '"') {
|
|
286
|
+
inString = true;
|
|
287
|
+
continue;
|
|
288
|
+
}
|
|
289
|
+
if (char === opener) depth++;
|
|
290
|
+
else if (char === closer) {
|
|
291
|
+
depth--;
|
|
292
|
+
if (depth === 0) return i + 1;
|
|
293
|
+
}
|
|
294
|
+
}
|
|
295
|
+
return i;
|
|
296
|
+
}
|
|
297
|
+
|
|
243
298
|
export function extractSubagentOverride(argsString: string): SubagentOverrideExtraction {
|
|
244
299
|
let override: SubagentOverride | undefined;
|
|
245
300
|
let cwdRaw: string | undefined;
|
|
246
301
|
let modelRaw: string | undefined;
|
|
302
|
+
let presetRaw: string | undefined;
|
|
247
303
|
let fork = false;
|
|
248
304
|
const tokensToRemove: Array<{ start: number; end: number }> = [];
|
|
249
305
|
|
|
@@ -265,7 +321,7 @@ export function extractSubagentOverride(argsString: string): SubagentOverrideExt
|
|
|
265
321
|
}
|
|
266
322
|
|
|
267
323
|
const tokenStart = i;
|
|
268
|
-
|
|
324
|
+
i = readJsonLikeFlagTokenEnd(argsString, tokenStart);
|
|
269
325
|
const token = argsString.slice(tokenStart, i);
|
|
270
326
|
|
|
271
327
|
if (token === "--subagent") {
|
|
@@ -295,6 +351,31 @@ export function extractSubagentOverride(argsString: string): SubagentOverrideExt
|
|
|
295
351
|
continue;
|
|
296
352
|
}
|
|
297
353
|
|
|
354
|
+
if (token.startsWith("--preset=")) {
|
|
355
|
+
tokensToRemove.push({ start: tokenStart, end: i });
|
|
356
|
+
const value = token.slice("--preset=".length);
|
|
357
|
+
presetRaw = value || undefined;
|
|
358
|
+
continue;
|
|
359
|
+
}
|
|
360
|
+
|
|
361
|
+
if (token === "--preset") {
|
|
362
|
+
let lookahead = i;
|
|
363
|
+
while (lookahead < argsString.length && /\s/.test(argsString[lookahead])) lookahead++;
|
|
364
|
+
if (lookahead < argsString.length && argsString[lookahead] !== '"' && argsString[lookahead] !== "'") {
|
|
365
|
+
const valueStart = lookahead;
|
|
366
|
+
while (lookahead < argsString.length && !/\s/.test(argsString[lookahead])) lookahead++;
|
|
367
|
+
const value = argsString.slice(valueStart, lookahead);
|
|
368
|
+
if (value && !value.startsWith("--")) {
|
|
369
|
+
tokensToRemove.push({ start: tokenStart, end: i }, { start: valueStart, end: lookahead });
|
|
370
|
+
presetRaw = value;
|
|
371
|
+
i = lookahead;
|
|
372
|
+
continue;
|
|
373
|
+
}
|
|
374
|
+
}
|
|
375
|
+
tokensToRemove.push({ start: tokenStart, end: i });
|
|
376
|
+
continue;
|
|
377
|
+
}
|
|
378
|
+
|
|
298
379
|
if (token === "--fork") {
|
|
299
380
|
tokensToRemove.push({ start: tokenStart, end: i });
|
|
300
381
|
fork = true;
|
|
@@ -317,6 +398,7 @@ export function extractSubagentOverride(argsString: string): SubagentOverrideExt
|
|
|
317
398
|
...(override ? { override } : {}),
|
|
318
399
|
...(cwdRaw !== undefined ? { cwd: cwdRaw } : {}),
|
|
319
400
|
...(modelRaw !== undefined ? { model: modelRaw } : {}),
|
|
401
|
+
...(presetRaw !== undefined ? { preset: presetRaw } : {}),
|
|
320
402
|
...(fork ? { fork: true } : {}),
|
|
321
403
|
};
|
|
322
404
|
}
|
|
@@ -0,0 +1,256 @@
|
|
|
1
|
+
import { existsSync, readFileSync } from "node:fs";
|
|
2
|
+
import { homedir } from "node:os";
|
|
3
|
+
import { join, resolve } from "node:path";
|
|
4
|
+
import type { DelegationLineupSlot, PromptLoaderDiagnostic, PromptSource } from "./prompt-loader.js";
|
|
5
|
+
|
|
6
|
+
export interface BestOfNPreset {
|
|
7
|
+
description?: string;
|
|
8
|
+
defaultModel?: string;
|
|
9
|
+
maxModelCalls?: number;
|
|
10
|
+
workers?: DelegationLineupSlot[];
|
|
11
|
+
reviewers?: DelegationLineupSlot[];
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
export interface ResolvedBestOfNPreset extends BestOfNPreset {
|
|
15
|
+
name: string;
|
|
16
|
+
source: PromptSource;
|
|
17
|
+
filePath: string;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
export interface BestOfNPresetCatalog {
|
|
21
|
+
presets: Map<string, ResolvedBestOfNPreset>;
|
|
22
|
+
invalidPresetNames: Set<string>;
|
|
23
|
+
diagnostics: PromptLoaderDiagnostic[];
|
|
24
|
+
projectFileInvalid: boolean;
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
function createPresetDiagnostic(code: string, filePath: string, source: PromptSource, message: string): PromptLoaderDiagnostic {
|
|
28
|
+
return {
|
|
29
|
+
code,
|
|
30
|
+
filePath,
|
|
31
|
+
source,
|
|
32
|
+
message,
|
|
33
|
+
key: `${code}:${filePath}:${message}`,
|
|
34
|
+
};
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
function isRecord(value: unknown): value is Record<string, unknown> {
|
|
38
|
+
return value !== null && typeof value === "object" && !Array.isArray(value);
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
function isValidModelSelectionSpec(spec: string): boolean {
|
|
42
|
+
if (!spec || spec.includes("*") || /\s/.test(spec)) return false;
|
|
43
|
+
const slashIndex = spec.indexOf("/");
|
|
44
|
+
if (slashIndex === -1) return true;
|
|
45
|
+
if (slashIndex === 0) return false;
|
|
46
|
+
const modelId = spec.slice(slashIndex + 1);
|
|
47
|
+
if (modelId.length === 0) return false;
|
|
48
|
+
if (modelId.split("/").some((segment) => segment.length === 0)) return false;
|
|
49
|
+
return true;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
function rejectUnsupportedKeys(
|
|
53
|
+
value: Record<string, unknown>,
|
|
54
|
+
allowedKeys: Set<string>,
|
|
55
|
+
presetName: string,
|
|
56
|
+
filePath: string,
|
|
57
|
+
source: PromptSource,
|
|
58
|
+
diagnostics: PromptLoaderDiagnostic[],
|
|
59
|
+
context: string,
|
|
60
|
+
): boolean {
|
|
61
|
+
const unsupportedKeys = Object.keys(value).filter((key) => !allowedKeys.has(key));
|
|
62
|
+
if (unsupportedKeys.length === 0) return false;
|
|
63
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${context} has unsupported field(s): ${unsupportedKeys.join(", ")}.`));
|
|
64
|
+
return true;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
const PRESET_KEYS = new Set(["description", "defaultModel", "maxModelCalls", "workers", "reviewers"]);
|
|
68
|
+
const PRESET_SLOT_KEYS = new Set(["agent", "subagent", "model", "count"]);
|
|
69
|
+
|
|
70
|
+
function normalizePresetSlot(
|
|
71
|
+
value: unknown,
|
|
72
|
+
field: "workers" | "reviewers",
|
|
73
|
+
presetName: string,
|
|
74
|
+
filePath: string,
|
|
75
|
+
source: PromptSource,
|
|
76
|
+
diagnostics: PromptLoaderDiagnostic[],
|
|
77
|
+
index: number,
|
|
78
|
+
): DelegationLineupSlot | undefined {
|
|
79
|
+
if (!isRecord(value)) {
|
|
80
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} slot ${index + 1} must be an object.`));
|
|
81
|
+
return undefined;
|
|
82
|
+
}
|
|
83
|
+
if (rejectUnsupportedKeys(value, PRESET_SLOT_KEYS, presetName, filePath, source, diagnostics, `${field} slot ${index + 1}`)) {
|
|
84
|
+
return undefined;
|
|
85
|
+
}
|
|
86
|
+
if (value.agent !== undefined && value.subagent !== undefined) {
|
|
87
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} slot ${index + 1} cannot combine "agent" and "subagent".`));
|
|
88
|
+
return undefined;
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
let agent: string | undefined;
|
|
92
|
+
if (typeof value.agent === "string" && value.agent.trim()) {
|
|
93
|
+
agent = value.agent.trim();
|
|
94
|
+
} else if (value.agent !== undefined) {
|
|
95
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} slot ${index + 1} requires a non-empty string "agent".`));
|
|
96
|
+
return undefined;
|
|
97
|
+
}
|
|
98
|
+
if (!agent && value.subagent !== undefined) {
|
|
99
|
+
if (value.subagent === true) {
|
|
100
|
+
agent = field === "reviewers" ? "reviewer" : "delegate";
|
|
101
|
+
} else if (typeof value.subagent === "string" && value.subagent.trim()) {
|
|
102
|
+
agent = value.subagent.trim();
|
|
103
|
+
} else {
|
|
104
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} slot ${index + 1} requires "subagent" to be true or a non-empty string.`));
|
|
105
|
+
return undefined;
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
if (!agent) agent = field === "reviewers" ? "reviewer" : "delegate";
|
|
109
|
+
|
|
110
|
+
const slot: DelegationLineupSlot = { agent };
|
|
111
|
+
if (value.model !== undefined) {
|
|
112
|
+
if (typeof value.model !== "string" || !value.model.trim() || !isValidModelSelectionSpec(value.model.trim())) {
|
|
113
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} slot ${index + 1} has an invalid "model".`));
|
|
114
|
+
return undefined;
|
|
115
|
+
}
|
|
116
|
+
slot.model = value.model.trim();
|
|
117
|
+
}
|
|
118
|
+
if (value.count !== undefined) {
|
|
119
|
+
let count: number | undefined;
|
|
120
|
+
if (typeof value.count === "number") count = value.count;
|
|
121
|
+
else if (typeof value.count === "string" && /^\d+$/.test(value.count.trim())) count = parseInt(value.count.trim(), 10);
|
|
122
|
+
if (count === undefined || !Number.isInteger(count) || count < 1) {
|
|
123
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} slot ${index + 1} "count" must be an integer greater than or equal to 1.`));
|
|
124
|
+
return undefined;
|
|
125
|
+
}
|
|
126
|
+
slot.count = count;
|
|
127
|
+
}
|
|
128
|
+
return slot;
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
function normalizePresetLineup(
|
|
132
|
+
value: unknown,
|
|
133
|
+
field: "workers" | "reviewers",
|
|
134
|
+
presetName: string,
|
|
135
|
+
filePath: string,
|
|
136
|
+
source: PromptSource,
|
|
137
|
+
diagnostics: PromptLoaderDiagnostic[],
|
|
138
|
+
): DelegationLineupSlot[] | undefined {
|
|
139
|
+
if (value === undefined) return undefined;
|
|
140
|
+
if (!Array.isArray(value) || value.length === 0) {
|
|
141
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(presetName)} in ${filePath}: ${field} must be a non-empty array.`));
|
|
142
|
+
return undefined;
|
|
143
|
+
}
|
|
144
|
+
const slots: DelegationLineupSlot[] = [];
|
|
145
|
+
for (let i = 0; i < value.length; i++) {
|
|
146
|
+
const slot = normalizePresetSlot(value[i], field, presetName, filePath, source, diagnostics, i);
|
|
147
|
+
if (!slot) return undefined;
|
|
148
|
+
slots.push(slot);
|
|
149
|
+
}
|
|
150
|
+
return slots;
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
function normalizePreset(name: string, value: unknown, filePath: string, source: PromptSource, diagnostics: PromptLoaderDiagnostic[]): BestOfNPreset | undefined {
|
|
154
|
+
if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(name)) {
|
|
155
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring invalid best-of-N preset name ${JSON.stringify(name)} in ${filePath}.`));
|
|
156
|
+
return undefined;
|
|
157
|
+
}
|
|
158
|
+
if (!isRecord(value)) {
|
|
159
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(name)} in ${filePath}: preset must be an object.`));
|
|
160
|
+
return undefined;
|
|
161
|
+
}
|
|
162
|
+
if (rejectUnsupportedKeys(value, PRESET_KEYS, name, filePath, source, diagnostics, "preset")) {
|
|
163
|
+
return undefined;
|
|
164
|
+
}
|
|
165
|
+
const preset: BestOfNPreset = {};
|
|
166
|
+
if (value.description !== undefined) {
|
|
167
|
+
if (typeof value.description !== "string") {
|
|
168
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(name)} in ${filePath}: description must be a string.`));
|
|
169
|
+
return undefined;
|
|
170
|
+
}
|
|
171
|
+
const description = value.description.trim();
|
|
172
|
+
if (description) preset.description = description;
|
|
173
|
+
}
|
|
174
|
+
if (value.defaultModel !== undefined) {
|
|
175
|
+
if (typeof value.defaultModel !== "string" || !value.defaultModel.trim() || !isValidModelSelectionSpec(value.defaultModel.trim())) {
|
|
176
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(name)} in ${filePath}: defaultModel must be a valid model spec string.`));
|
|
177
|
+
return undefined;
|
|
178
|
+
}
|
|
179
|
+
preset.defaultModel = value.defaultModel.trim();
|
|
180
|
+
}
|
|
181
|
+
if (value.maxModelCalls !== undefined) {
|
|
182
|
+
if (typeof value.maxModelCalls !== "number" || !Number.isInteger(value.maxModelCalls) || value.maxModelCalls < 1) {
|
|
183
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(name)} in ${filePath}: maxModelCalls must be an integer greater than or equal to 1.`));
|
|
184
|
+
return undefined;
|
|
185
|
+
}
|
|
186
|
+
preset.maxModelCalls = value.maxModelCalls;
|
|
187
|
+
}
|
|
188
|
+
const workers = normalizePresetLineup(value.workers, "workers", name, filePath, source, diagnostics);
|
|
189
|
+
if (value.workers !== undefined && !workers) return undefined;
|
|
190
|
+
const reviewers = normalizePresetLineup(value.reviewers, "reviewers", name, filePath, source, diagnostics);
|
|
191
|
+
if (value.reviewers !== undefined && !reviewers) return undefined;
|
|
192
|
+
if (workers) preset.workers = workers;
|
|
193
|
+
if (reviewers) preset.reviewers = reviewers;
|
|
194
|
+
if (!preset.workers && !preset.reviewers && value.workers === undefined && value.reviewers === undefined) {
|
|
195
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-preset", filePath, source, `Ignoring preset ${JSON.stringify(name)} in ${filePath}: expected workers or reviewers.`));
|
|
196
|
+
return undefined;
|
|
197
|
+
}
|
|
198
|
+
if (!preset.workers && !preset.reviewers) return undefined;
|
|
199
|
+
return preset;
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
function readPresetFile(filePath: string, source: PromptSource): BestOfNPresetCatalog {
|
|
203
|
+
const presets = new Map<string, ResolvedBestOfNPreset>();
|
|
204
|
+
const invalidPresetNames = new Set<string>();
|
|
205
|
+
const diagnostics: PromptLoaderDiagnostic[] = [];
|
|
206
|
+
if (!existsSync(filePath)) return { presets, invalidPresetNames, diagnostics, projectFileInvalid: false };
|
|
207
|
+
let projectFileInvalid = false;
|
|
208
|
+
let parsed: unknown;
|
|
209
|
+
try {
|
|
210
|
+
parsed = JSON.parse(readFileSync(filePath, "utf-8"));
|
|
211
|
+
} catch (error) {
|
|
212
|
+
projectFileInvalid = source === "project";
|
|
213
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-presets-file", filePath, source, `Skipping best-of-N presets file ${filePath}: ${error instanceof Error ? error.message : String(error)}.`));
|
|
214
|
+
return { presets, invalidPresetNames, diagnostics, projectFileInvalid };
|
|
215
|
+
}
|
|
216
|
+
if (!isRecord(parsed) || !isRecord(parsed.presets)) {
|
|
217
|
+
projectFileInvalid = source === "project";
|
|
218
|
+
diagnostics.push(createPresetDiagnostic("invalid-best-of-n-presets-file", filePath, source, `Skipping best-of-N presets file ${filePath}: expected top-level object with a "presets" object.`));
|
|
219
|
+
return { presets, invalidPresetNames, diagnostics, projectFileInvalid };
|
|
220
|
+
}
|
|
221
|
+
for (const [name, rawPreset] of Object.entries(parsed.presets)) {
|
|
222
|
+
const preset = normalizePreset(name, rawPreset, filePath, source, diagnostics);
|
|
223
|
+
if (!preset) {
|
|
224
|
+
invalidPresetNames.add(name);
|
|
225
|
+
continue;
|
|
226
|
+
}
|
|
227
|
+
presets.set(name, { name, source, filePath, ...preset });
|
|
228
|
+
}
|
|
229
|
+
return { presets, invalidPresetNames, diagnostics, projectFileInvalid };
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
export function getBestOfNPresetPaths(cwd: string): { user: string; project: string } {
|
|
233
|
+
return {
|
|
234
|
+
user: join(homedir(), ".pi", "agent", "best-of-n-presets.json"),
|
|
235
|
+
project: resolve(cwd, ".pi", "best-of-n-presets.json"),
|
|
236
|
+
};
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
export function loadBestOfNPresetCatalog(cwd: string): BestOfNPresetCatalog {
|
|
240
|
+
const paths = getBestOfNPresetPaths(cwd);
|
|
241
|
+
const user = readPresetFile(paths.user, "user");
|
|
242
|
+
const project = readPresetFile(paths.project, "project");
|
|
243
|
+
const presets = project.projectFileInvalid ? new Map(project.presets) : new Map([...user.presets, ...project.presets]);
|
|
244
|
+
for (const name of project.invalidPresetNames) presets.delete(name);
|
|
245
|
+
return {
|
|
246
|
+
presets,
|
|
247
|
+
invalidPresetNames: new Set([...user.invalidPresetNames, ...project.invalidPresetNames]),
|
|
248
|
+
diagnostics: [...user.diagnostics, ...project.diagnostics],
|
|
249
|
+
projectFileInvalid: project.projectFileInvalid,
|
|
250
|
+
};
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
export function applyPresetDefaultModel(slots: DelegationLineupSlot[] | undefined, defaultModel: string | undefined): DelegationLineupSlot[] | undefined {
|
|
254
|
+
if (!slots || !defaultModel) return slots?.map((slot) => ({ ...slot }));
|
|
255
|
+
return slots.map((slot) => ({ ...slot, model: slot.model ?? defaultModel }));
|
|
256
|
+
}
|
package/index.ts
CHANGED
|
@@ -15,6 +15,7 @@ import {
|
|
|
15
15
|
type LineupOverrideAction,
|
|
16
16
|
type SubagentOverride,
|
|
17
17
|
} from "./args.js";
|
|
18
|
+
import { loadBestOfNPresetCatalog, applyPresetDefaultModel, type ResolvedBestOfNPreset } from "./best-of-n-presets.js";
|
|
18
19
|
import { parseChainSteps, parseChainDeclaration, type ChainStep, type ChainStepOrParallel, type ParallelChainStep } from "./chain-parser.js";
|
|
19
20
|
import { generateBoomerangSummary, generateChainStepSummary, generateIterationSummary, didIterationMakeChanges, getIterationEntries, wasIterationAborted } from "./loop-utils.js";
|
|
20
21
|
import { selectModelCandidate } from "./model-selection.js";
|
|
@@ -130,6 +131,7 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
130
131
|
let lastDiagnostics = "";
|
|
131
132
|
let storedCommandCtx: ExtensionCommandContext | null = null;
|
|
132
133
|
const approvedProjectPromptLibraryCwds = new Set<string>();
|
|
134
|
+
const approvedProjectPresetCwds = new Set<string>();
|
|
133
135
|
const UNLIMITED_LOOP_CAP = 999;
|
|
134
136
|
|
|
135
137
|
const toolManager = createToolManager(pi, {
|
|
@@ -194,6 +196,49 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
194
196
|
return true;
|
|
195
197
|
}
|
|
196
198
|
|
|
199
|
+
async function ensureProjectPresetApproved(preset: ResolvedBestOfNPreset, ctx: ExtensionCommandContext, catalogCwd: string): Promise<boolean> {
|
|
200
|
+
if (preset.source !== "project") return true;
|
|
201
|
+
const cwdKey = resolvePath(catalogCwd);
|
|
202
|
+
if (approvedProjectPresetCwds.has(cwdKey)) return true;
|
|
203
|
+
const message = `Best-of-N preset \`${preset.name}\` is loaded from ${preset.filePath}. Approve project best-of-N presets for this session?`;
|
|
204
|
+
if (!ctx.hasUI || typeof (ctx.ui as { confirm?: unknown }).confirm !== "function") {
|
|
205
|
+
notify(ctx, `${message} Run in an interactive UI session and approve it, or move trusted presets to ~/.pi/agent/best-of-n-presets.json.`, "error");
|
|
206
|
+
return false;
|
|
207
|
+
}
|
|
208
|
+
const approved = await ctx.ui.confirm("Approve project best-of-N preset", message, { timeout: 30_000 });
|
|
209
|
+
if (!approved) {
|
|
210
|
+
notify(ctx, `Project best-of-N preset \`${preset.name}\` was not approved.`, "warning");
|
|
211
|
+
return false;
|
|
212
|
+
}
|
|
213
|
+
approvedProjectPresetCwds.add(cwdKey);
|
|
214
|
+
return true;
|
|
215
|
+
}
|
|
216
|
+
|
|
217
|
+
async function resolveBestOfNPresetLineup(
|
|
218
|
+
prompt: PromptWithModel,
|
|
219
|
+
runtimePreset: string | undefined,
|
|
220
|
+
ctx: ExtensionCommandContext,
|
|
221
|
+
catalogCwd: string,
|
|
222
|
+
): Promise<{ workers?: DelegationLineupSlot[]; reviewers?: DelegationLineupSlot[]; maxModelCalls?: number } | undefined> {
|
|
223
|
+
const presetName = runtimePreset ?? prompt.preset;
|
|
224
|
+
if (!presetName) return {};
|
|
225
|
+
const catalog = loadBestOfNPresetCatalog(catalogCwd);
|
|
226
|
+
for (const diagnostic of catalog.diagnostics) {
|
|
227
|
+
notify(ctx, diagnostic.message, "warning");
|
|
228
|
+
}
|
|
229
|
+
const preset = catalog.presets.get(presetName);
|
|
230
|
+
if (!preset) {
|
|
231
|
+
notify(ctx, `Best-of-N preset \`${presetName}\` was not found. Define it in ~/.pi/agent/best-of-n-presets.json or .pi/best-of-n-presets.json.`, "error");
|
|
232
|
+
return undefined;
|
|
233
|
+
}
|
|
234
|
+
if (!(await ensureProjectPresetApproved(preset, ctx, catalogCwd))) return undefined;
|
|
235
|
+
return {
|
|
236
|
+
workers: applyPresetDefaultModel(preset.workers, preset.defaultModel),
|
|
237
|
+
reviewers: applyPresetDefaultModel(preset.reviewers, preset.defaultModel),
|
|
238
|
+
maxModelCalls: preset.maxModelCalls,
|
|
239
|
+
};
|
|
240
|
+
}
|
|
241
|
+
|
|
197
242
|
pi.registerMessageRenderer<SkillLoadedDetails>("skill-loaded", renderSkillLoaded);
|
|
198
243
|
pi.registerMessageRenderer(PROMPT_TEMPLATE_SUBAGENT_MESSAGE_TYPE, renderDelegatedSubagentResult);
|
|
199
244
|
pi.registerMessageRenderer(PROMPT_TEMPLATE_DETERMINISTIC_MESSAGE_TYPE, renderDeterministicResult);
|
|
@@ -519,6 +564,12 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
519
564
|
return slots?.map((slot) => ({ ...slot }));
|
|
520
565
|
}
|
|
521
566
|
|
|
567
|
+
function expandedLineupCallCount(slots: DelegationLineupSlot[]): number {
|
|
568
|
+
let total = 0;
|
|
569
|
+
for (const slot of slots) total += slot.count ?? 1;
|
|
570
|
+
return total;
|
|
571
|
+
}
|
|
572
|
+
|
|
522
573
|
function expandLineupCounts(slots: DelegationLineupSlot[]): DelegationLineupSlot[] {
|
|
523
574
|
const expanded: DelegationLineupSlot[] = [];
|
|
524
575
|
for (const slot of slots) {
|
|
@@ -729,7 +780,7 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
729
780
|
args: string,
|
|
730
781
|
ctx: ExtensionCommandContext,
|
|
731
782
|
currentModel: Model<any> | undefined,
|
|
732
|
-
runtime: { cwd?: string; model?: string; subagentOverride?: SubagentOverride; fork?: boolean },
|
|
783
|
+
runtime: { cwd?: string; model?: string; preset?: string; subagentOverride?: SubagentOverride; fork?: boolean },
|
|
733
784
|
) {
|
|
734
785
|
if (!(await ensureProjectPromptLibraryApproved(prompt, ctx))) return;
|
|
735
786
|
|
|
@@ -782,24 +833,30 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
782
833
|
}
|
|
783
834
|
const sharedTask = rendered.content;
|
|
784
835
|
|
|
785
|
-
const
|
|
786
|
-
|
|
836
|
+
const presetLineup = await resolveBestOfNPresetLineup(prompt, runtime.preset, ctx, compareCwd);
|
|
837
|
+
if (!presetLineup) return;
|
|
838
|
+
const requestedWorkers = applyLineupActions(presetLineup.workers ?? prompt.workers, lineupExtraction.actions, "workers") ?? [];
|
|
839
|
+
const requestedReviewers = applyLineupActions(presetLineup.reviewers ?? prompt.reviewers, lineupExtraction.actions, "reviewers") ?? [];
|
|
787
840
|
const requestedFinalApplier = applyFinalApplierAction(prompt.finalApplier, lineupExtraction.actions);
|
|
788
841
|
if (requestedFinalApplier && prompt.worktree !== true) {
|
|
789
842
|
notify(ctx, "Compare prompts with finalApplier require worktree: true.", "error");
|
|
790
843
|
return;
|
|
791
844
|
}
|
|
792
845
|
|
|
793
|
-
const
|
|
794
|
-
requestedWorkers
|
|
795
|
-
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
846
|
+
const effectiveWorkerSlots = requestedWorkers.length > 0
|
|
847
|
+
? requestedWorkers
|
|
848
|
+
: [{ agent: DEFAULT_SUBAGENT_NAME }];
|
|
849
|
+
const effectiveReviewerSlots = requestedReviewers.length > 0
|
|
850
|
+
? requestedReviewers
|
|
851
|
+
: [{ agent: "reviewer" }];
|
|
852
|
+
const presetModelCalls = expandedLineupCallCount(effectiveWorkerSlots) + expandedLineupCallCount(effectiveReviewerSlots);
|
|
853
|
+
if (presetLineup.maxModelCalls !== undefined && presetModelCalls > presetLineup.maxModelCalls) {
|
|
854
|
+
notify(ctx, `Best-of-N preset model-call cap exceeded: requested ${presetModelCalls} worker/reviewer call(s), but preset allows ${presetLineup.maxModelCalls}.`, "error");
|
|
855
|
+
return;
|
|
856
|
+
}
|
|
857
|
+
|
|
858
|
+
const workerSlots = expandLineupCounts(effectiveWorkerSlots);
|
|
859
|
+
const reviewerSlots = expandLineupCounts(effectiveReviewerSlots);
|
|
803
860
|
|
|
804
861
|
const normalizedWorkers = normalizeLineupCwds(workerSlots, compareCwd, ctx);
|
|
805
862
|
if (!normalizedWorkers) return;
|
|
@@ -1499,7 +1556,7 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
1499
1556
|
|
|
1500
1557
|
function getDryRunUnsupportedReason(prompt: PromptWithModel): string | undefined {
|
|
1501
1558
|
if (prompt.chain) return DRY_RUN_CHAIN_UNSUPPORTED;
|
|
1502
|
-
if (prompt.workers !== undefined || prompt.reviewers !== undefined || prompt.finalApplier !== undefined) return DRY_RUN_COMPARE_UNSUPPORTED;
|
|
1559
|
+
if (prompt.workers !== undefined || prompt.reviewers !== undefined || prompt.finalApplier !== undefined || prompt.preset !== undefined) return DRY_RUN_COMPARE_UNSUPPORTED;
|
|
1503
1560
|
if (prompt.deterministic) return DRY_RUN_DETERMINISTIC_UNSUPPORTED;
|
|
1504
1561
|
return undefined;
|
|
1505
1562
|
}
|
|
@@ -1659,7 +1716,7 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
1659
1716
|
}
|
|
1660
1717
|
}
|
|
1661
1718
|
|
|
1662
|
-
const hasCompareLineup = prompt.workers !== undefined || prompt.reviewers !== undefined || prompt.finalApplier !== undefined;
|
|
1719
|
+
const hasCompareLineup = prompt.workers !== undefined || prompt.reviewers !== undefined || prompt.finalApplier !== undefined || prompt.preset !== undefined;
|
|
1663
1720
|
if (hasCompareLineup) {
|
|
1664
1721
|
await runComparePrompt(
|
|
1665
1722
|
name,
|
|
@@ -1670,6 +1727,7 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
1670
1727
|
{
|
|
1671
1728
|
cwd: runtimeCwd,
|
|
1672
1729
|
model: subagent.model,
|
|
1730
|
+
preset: subagent.preset,
|
|
1673
1731
|
subagentOverride: subagent.override,
|
|
1674
1732
|
fork: subagent.fork,
|
|
1675
1733
|
},
|
|
@@ -1677,6 +1735,10 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
1677
1735
|
return;
|
|
1678
1736
|
}
|
|
1679
1737
|
|
|
1738
|
+
if (subagent.preset) {
|
|
1739
|
+
notify(ctx, `--preset is only supported on compare prompts (ignored)`, "warning");
|
|
1740
|
+
}
|
|
1741
|
+
|
|
1680
1742
|
if (prompt.chain) {
|
|
1681
1743
|
if (subagent.model) notify(ctx, `--model is not supported on chain prompts (ignored)`, "warning");
|
|
1682
1744
|
if (subagent.fork) notify(ctx, `--fork is not supported on chain prompts (ignored)`, "warning");
|
|
@@ -1791,6 +1853,7 @@ export default function promptModelExtension(pi: ExtensionAPI) {
|
|
|
1791
1853
|
function resetSessionScopedState(ctx: ExtensionContext) {
|
|
1792
1854
|
storedCommandCtx = null;
|
|
1793
1855
|
approvedProjectPromptLibraryCwds.clear();
|
|
1856
|
+
approvedProjectPresetCwds.clear();
|
|
1794
1857
|
pendingSkillMessage = undefined;
|
|
1795
1858
|
previousModel = undefined;
|
|
1796
1859
|
previousThinking = undefined;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-prompt-template-model-enhanced",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Prompt template model selector extension for pi coding agent",
|
|
6
6
|
"author": "Nico Bailon",
|
|
@@ -27,6 +27,7 @@
|
|
|
27
27
|
"files": [
|
|
28
28
|
"index.ts",
|
|
29
29
|
"args.ts",
|
|
30
|
+
"best-of-n-presets.ts",
|
|
30
31
|
"chain-parser.ts",
|
|
31
32
|
"loop-utils.ts",
|
|
32
33
|
"model-selection.ts",
|
package/prompt-dry-run.ts
CHANGED
|
@@ -209,7 +209,7 @@ function errorResult(
|
|
|
209
209
|
}
|
|
210
210
|
|
|
211
211
|
function hasCompareLineup(prompt: PromptWithModel): boolean {
|
|
212
|
-
return prompt.workers !== undefined || prompt.reviewers !== undefined || prompt.finalApplier !== undefined;
|
|
212
|
+
return prompt.workers !== undefined || prompt.reviewers !== undefined || prompt.finalApplier !== undefined || prompt.preset !== undefined;
|
|
213
213
|
}
|
|
214
214
|
|
|
215
215
|
function shouldDelegatePrompt(prompt: Pick<PromptWithModel, "subagent">, override?: SubagentOverride): boolean {
|
package/prompt-loader.ts
CHANGED
|
@@ -104,6 +104,7 @@ export interface PromptWithModel {
|
|
|
104
104
|
workers?: DelegationLineupSlot[];
|
|
105
105
|
reviewers?: DelegationLineupSlot[];
|
|
106
106
|
finalApplier?: DelegationLineupSlot;
|
|
107
|
+
preset?: string;
|
|
107
108
|
source: PromptSource;
|
|
108
109
|
rootKind: PromptRootKind;
|
|
109
110
|
subdir?: string;
|
|
@@ -1884,9 +1885,11 @@ function loadPromptsWithModelFromDir(
|
|
|
1884
1885
|
const workers = normalizeLineup(hasBestOfN ? bestOfN?.workers : undefined, "workers", fullPath, source, diagnostics);
|
|
1885
1886
|
const reviewers = normalizeLineup(hasBestOfN ? bestOfN?.reviewers : undefined, "reviewers", fullPath, source, diagnostics);
|
|
1886
1887
|
const finalApplier = normalizeFinalApplier(hasBestOfN ? bestOfN?.finalApplier : undefined, fullPath, source, diagnostics);
|
|
1888
|
+
const preset = normalizeStringField("best-of-n-preset", hasBestOfN ? bestOfN?.preset : undefined, fullPath, source, diagnostics);
|
|
1887
1889
|
let safeWorkers = workers;
|
|
1888
1890
|
let safeReviewers = reviewers;
|
|
1889
1891
|
let safeFinalApplier = finalApplier;
|
|
1892
|
+
let safePreset = preset;
|
|
1890
1893
|
if (chain && subagent !== undefined) {
|
|
1891
1894
|
diagnostics.push(
|
|
1892
1895
|
createDiagnostic(
|
|
@@ -1909,7 +1912,7 @@ function loadPromptsWithModelFromDir(
|
|
|
1909
1912
|
);
|
|
1910
1913
|
deterministic = undefined;
|
|
1911
1914
|
}
|
|
1912
|
-
if (chain && (safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined)) {
|
|
1915
|
+
if (chain && (safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined || safePreset !== undefined)) {
|
|
1913
1916
|
diagnostics.push(
|
|
1914
1917
|
createDiagnostic(
|
|
1915
1918
|
"invalid-lineup-chain",
|
|
@@ -1921,8 +1924,9 @@ function loadPromptsWithModelFromDir(
|
|
|
1921
1924
|
safeWorkers = undefined;
|
|
1922
1925
|
safeReviewers = undefined;
|
|
1923
1926
|
safeFinalApplier = undefined;
|
|
1927
|
+
safePreset = undefined;
|
|
1924
1928
|
}
|
|
1925
|
-
if (subagent !== undefined && (safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined)) {
|
|
1929
|
+
if (subagent !== undefined && (safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined || safePreset !== undefined)) {
|
|
1926
1930
|
diagnostics.push(
|
|
1927
1931
|
createDiagnostic(
|
|
1928
1932
|
"invalid-lineup-subagent",
|
|
@@ -1934,6 +1938,7 @@ function loadPromptsWithModelFromDir(
|
|
|
1934
1938
|
safeWorkers = undefined;
|
|
1935
1939
|
safeReviewers = undefined;
|
|
1936
1940
|
safeFinalApplier = undefined;
|
|
1941
|
+
safePreset = undefined;
|
|
1937
1942
|
}
|
|
1938
1943
|
if (subagent !== undefined && deterministic !== undefined) {
|
|
1939
1944
|
diagnostics.push(
|
|
@@ -1979,7 +1984,7 @@ function loadPromptsWithModelFromDir(
|
|
|
1979
1984
|
);
|
|
1980
1985
|
safeParallel = undefined;
|
|
1981
1986
|
}
|
|
1982
|
-
if (safeParallel !== undefined && (safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined)) {
|
|
1987
|
+
if (safeParallel !== undefined && (safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined || safePreset !== undefined)) {
|
|
1983
1988
|
diagnostics.push(
|
|
1984
1989
|
createDiagnostic(
|
|
1985
1990
|
"invalid-lineup-parallel",
|
|
@@ -1991,6 +1996,7 @@ function loadPromptsWithModelFromDir(
|
|
|
1991
1996
|
safeWorkers = undefined;
|
|
1992
1997
|
safeReviewers = undefined;
|
|
1993
1998
|
safeFinalApplier = undefined;
|
|
1999
|
+
safePreset = undefined;
|
|
1994
2000
|
}
|
|
1995
2001
|
if (safeParallel !== undefined && deterministic !== undefined) {
|
|
1996
2002
|
diagnostics.push(
|
|
@@ -2003,7 +2009,7 @@ function loadPromptsWithModelFromDir(
|
|
|
2003
2009
|
);
|
|
2004
2010
|
deterministic = undefined;
|
|
2005
2011
|
}
|
|
2006
|
-
const hasLineup = safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined;
|
|
2012
|
+
const hasLineup = safeWorkers !== undefined || safeReviewers !== undefined || safeFinalApplier !== undefined || safePreset !== undefined;
|
|
2007
2013
|
if (!hasBestOfN && hasLegacyCompareFields) {
|
|
2008
2014
|
diagnostics.push(
|
|
2009
2015
|
createDiagnostic(
|
|
@@ -2223,6 +2229,7 @@ function loadPromptsWithModelFromDir(
|
|
|
2223
2229
|
workers: safeWorkers,
|
|
2224
2230
|
reviewers: safeReviewers,
|
|
2225
2231
|
finalApplier: safeFinalApplier,
|
|
2232
|
+
preset: safePreset,
|
|
2226
2233
|
source,
|
|
2227
2234
|
rootKind,
|
|
2228
2235
|
subdir: subdir || undefined,
|
package/prompt-validation.ts
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
|
+
import { resolve as resolvePath } from "node:path";
|
|
2
|
+
import { loadBestOfNPresetCatalog, type ResolvedBestOfNPreset } from "./best-of-n-presets.js";
|
|
1
3
|
import { parseChainDeclaration, type ChainStep, type ChainStepOrParallel } from "./chain-parser.js";
|
|
2
4
|
import { collectPromptIncludeGraphs, type PromptIncludeGraph, type PromptIncludeGraphEdge, type PromptIncludeGraphNode } from "./prompt-includes.js";
|
|
3
|
-
import { collectPromptSourceRecords, discoverFilesystemSkills, loadPromptsWithModel, readSkillContent, resolveSkillPath, type PromptLoaderDiagnostic, type PromptSource, type PromptSourceRecord } from "./prompt-loader.js";
|
|
5
|
+
import { collectPromptSourceRecords, discoverFilesystemSkills, loadPromptsWithModel, readSkillContent, resolveSkillPath, type PromptLoaderDiagnostic, type PromptSource, type PromptSourceRecord, type PromptWithModel } from "./prompt-loader.js";
|
|
4
6
|
|
|
5
7
|
export interface RegisteredPromptSkill {
|
|
6
8
|
skillName: string;
|
|
@@ -314,8 +316,57 @@ function validatePromptChains(cwd: string, result: PromptValidationResult, promp
|
|
|
314
316
|
}
|
|
315
317
|
}
|
|
316
318
|
|
|
319
|
+
function expandedSlotCount(slots: Array<{ count?: number }> | undefined): number {
|
|
320
|
+
return (slots ?? []).reduce((total, slot) => total + (slot.count ?? 1), 0);
|
|
321
|
+
}
|
|
322
|
+
|
|
323
|
+
function presetExpandedCallCount(preset: ResolvedBestOfNPreset, prompt: PromptWithModel): number {
|
|
324
|
+
const workerSlots = preset.workers ?? prompt.workers;
|
|
325
|
+
const reviewerSlots = preset.reviewers ?? prompt.reviewers;
|
|
326
|
+
return (workerSlots && workerSlots.length > 0 ? expandedSlotCount(workerSlots) : 1)
|
|
327
|
+
+ (reviewerSlots && reviewerSlots.length > 0 ? expandedSlotCount(reviewerSlots) : 1);
|
|
328
|
+
}
|
|
329
|
+
|
|
317
330
|
function validateComparePrompts(cwd: string, result: PromptValidationResult, prompts: ReturnType<typeof loadPromptsWithModel>["prompts"]) {
|
|
331
|
+
const catalogByCwd = new Map<string, ReturnType<typeof loadBestOfNPresetCatalog>>();
|
|
332
|
+
function catalogFor(catalogCwd: string) {
|
|
333
|
+
const key = resolvePath(catalogCwd);
|
|
334
|
+
let catalog = catalogByCwd.get(key);
|
|
335
|
+
if (!catalog) {
|
|
336
|
+
catalog = loadBestOfNPresetCatalog(catalogCwd);
|
|
337
|
+
catalogByCwd.set(key, catalog);
|
|
338
|
+
result.diagnostics.push(...catalog.diagnostics);
|
|
339
|
+
}
|
|
340
|
+
return catalog;
|
|
341
|
+
}
|
|
318
342
|
for (const prompt of prompts.values()) {
|
|
343
|
+
if (prompt.preset) {
|
|
344
|
+
const presetCatalog = catalogFor(prompt.cwd ?? cwd);
|
|
345
|
+
const preset = presetCatalog.presets.get(prompt.preset);
|
|
346
|
+
if (!preset) {
|
|
347
|
+
result.diagnostics.push(
|
|
348
|
+
createValidationDiagnostic(
|
|
349
|
+
"best-of-n-preset-not-found",
|
|
350
|
+
prompt.filePath,
|
|
351
|
+
prompt.source,
|
|
352
|
+
`Prompt template ${prompt.filePath} references missing best-of-N preset ${JSON.stringify(prompt.preset)}.`,
|
|
353
|
+
),
|
|
354
|
+
);
|
|
355
|
+
} else if (preset.maxModelCalls !== undefined) {
|
|
356
|
+
const modelCalls = presetExpandedCallCount(preset, prompt);
|
|
357
|
+
if (modelCalls > preset.maxModelCalls) {
|
|
358
|
+
result.diagnostics.push(
|
|
359
|
+
createValidationDiagnostic(
|
|
360
|
+
"best-of-n-preset-cap-exceeded",
|
|
361
|
+
prompt.filePath,
|
|
362
|
+
prompt.source,
|
|
363
|
+
`Prompt template ${prompt.filePath} references best-of-N preset ${JSON.stringify(prompt.preset)}, but its expanded worker/reviewer calls (${modelCalls}) exceed maxModelCalls (${preset.maxModelCalls}).`,
|
|
364
|
+
),
|
|
365
|
+
);
|
|
366
|
+
}
|
|
367
|
+
}
|
|
368
|
+
}
|
|
369
|
+
|
|
319
370
|
if (prompt.finalApplier && prompt.worktree !== true) {
|
|
320
371
|
result.diagnostics.push(
|
|
321
372
|
createValidationDiagnostic(
|
|
@@ -153,18 +153,29 @@ Supports exact IDs, `provider/model-id` pairs, wildcards (`anthropic/*`), and co
|
|
|
153
153
|
|
|
154
154
|
## Best-of-N Compare
|
|
155
155
|
|
|
156
|
-
Run multiple workers, aggregate with reviewers, optionally apply final changes:
|
|
156
|
+
Run multiple workers, aggregate with reviewers, optionally apply final changes. Put workflow policy in the prompt and reusable lineup choices in optional presets:
|
|
157
|
+
|
|
158
|
+
```json
|
|
159
|
+
{
|
|
160
|
+
"presets": {
|
|
161
|
+
"quick": {
|
|
162
|
+
"defaultModel": "openai-codex/gpt-5.4-mini:low",
|
|
163
|
+
"maxModelCalls": 3,
|
|
164
|
+
"workers": [{ "agent": "delegate", "count": 2 }],
|
|
165
|
+
"reviewers": [{ "agent": "reviewer" }]
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Preset files live at `~/.pi/agent/best-of-n-presets.json` and `<compare-cwd>/.pi/best-of-n-presets.json`. Project presets override user presets after approval. Preset slots only support `agent`/`subagent`, `model`, and `count`; keep `task`, `taskSuffix`, `cwd`, `finalApplier`, `worktree`, and dirty/report/commit policy in prompt templates.
|
|
157
172
|
|
|
158
173
|
```yaml
|
|
159
174
|
---
|
|
160
175
|
description: Best-of-N code review
|
|
161
176
|
bestOfN:
|
|
177
|
+
preset: quick
|
|
162
178
|
worktree: true # required if using finalApplier
|
|
163
|
-
workers:
|
|
164
|
-
- model: openai-codex/gpt-5.4-mini:low
|
|
165
|
-
count: 2
|
|
166
|
-
reviewers:
|
|
167
|
-
- model: anthropic/claude-sonnet-4-20250514:medium
|
|
168
179
|
finalApplier:
|
|
169
180
|
agent: delegate
|
|
170
181
|
model: anthropic/claude-sonnet-4-20250514:high
|
|
@@ -185,5 +196,6 @@ Override frontmatter at invocation:
|
|
|
185
196
|
- `--cwd=/absolute/path` — working directory override when the prompt supports `cwd`
|
|
186
197
|
- `--chain-context` — pass summaries to later delegated chain steps
|
|
187
198
|
- `--worktree` — use git worktrees for parallel delegated work
|
|
199
|
+
- `--preset=<name>` / `--preset <name>` — select a best-of-N preset for compare prompts only
|
|
188
200
|
|
|
189
201
|
When stuck, check `README.md` and `examples/best-of-n.md` in this extension.
|