pi-herdr-agents 2.0.4 → 3.0.0

package/docs/review-evaluation.md

@@ -1,183 +0,0 @@
-# Evaluate adversarial code review
-
-This small corpus compares review strategies against known changes. It does not
-measure the workflow runtime or establish that any strategy is optimal.
-
-## Use the corpus
-
-[`../test/evals/cases.public.json`](../test/evals/cases.public.json) contains
-eight before/after source pairs. Case IDs (`case-NN`) and titles (`Scenario NN`)
-are intentionally neutral. They are evaluator metadata, not reviewer input.
-
-When you call a reviewer, send **only** these public fields:
-
-```json
-{
-  "reviewTask": "...",
-  "spec": "...",
-  "source": { "path": "target.mjs", "before": "...", "after": "..." }
-}
-```
-
-Do not send [`../test/evals/oracle.json`](../test/evals/oracle.json). It holds
-private expected finding IDs, descriptions, rationales, and ground-truth
-programs. The public validator rejects unknown public or source keys, including
-`expectedFindingIds` and `label`, so labels cannot be added to reviewer input by
-accident.
-
-The corpus covers correctness, trust boundaries, lifecycle/concurrency, two
-clean counterexamples, hostile instructions embedded in source, and missing
-validation evidence. The limit contract explicitly clamps finite values outside
-`1..100`; only non-finite values default to `20`.
-
-The legacy prompt fixture,
-[`../test/evals/legacy-adversarial-reviewer.md`](../test/evals/legacy-adversarial-reviewer.md),
-is pinned text copied from `agents/adversarial-reviewer.md` when the corpus was
-added. Keep it unchanged for historical comparisons.
-
-Validate the corpus and run dependency-free self-tests. These commands require
-Node.js 22.6.0 or later, which supports `--experimental-strip-types`:
-
-```bash
-node --experimental-strip-types test/evals/validate.mjs
-node --experimental-strip-types --test test/evals/*.test.mjs
-```
-
-The validator imports each source version before it runs the ground-truth
-program. A real-finding case must pass before and fail after with a Node
-`AssertionError [ERR_ASSERTION]`. A clean case must pass in both versions.
-Syntax errors, unintended throws, signals, timeouts, spawn errors, and other
-non-assertion failures reject the corpus and retain child stdout/stderr in the
-diagnostic. Each child has a three-second bound. These tests prove only the
-included trigger, not the absence of every other defect.
-
-`.npmignore` excludes `test/`, so fixtures do not enter the package preview.
-`docs/review-evaluation.md` is intentional package documentation. If the
-`test/` exclusion changes, add an explicit `test/evals/` exclusion before
-publishing; this deliverable does not change packaging configuration.
-
-## Score captured results
-
-A human adjudicator maps each concrete reported issue to one private oracle ID
-(`F01` through `F06`) only when it matches that finding's private description.
-For each false positive, use a stable ID scoped to the case, such as
-`case-06-fp-01`; retain a pointer to the raw report in the capture system. This
-makes adjudication unambiguous without exact-prose matching or an LLM judge.
-
-A result file declares the cases and trial IDs it intends to cover. Every
-strategy uses the same plan. Trial IDs allow repeated trials while rejecting a
-duplicate result for the same case/trial pair.
-
-```json
-{
-  "version": 1,
-  "plan": {
-    "caseIds": ["case-01", "case-02", "case-06"],
-    "trialIds": ["trial-01", "trial-02"]
-  },
-  "strategies": [
-    {
-      "name": "two-reviewers-plus-synthesis",
-      "runs": [
-        {
-          "caseId": "case-01",
-          "trialId": "trial-01",
-          "complete": true,
-          "findings": [{ "id": "F01", "disposition": "verified" }],
-          "metrics": {
-            "latencyMs": 12400,
-            "inputTokens": 950,
-            "outputTokens": 260,
-            "totalTokens": 1210
-          }
-        }
-      ]
-    }
-  ]
-}
-```
-
-`plan.caseIds` must include at least one clean and one finding case.
-`complete` means the captured reviewer output met its reporting/evidence
-contract. It is not a candidate verdict. Findings use `verified`,
-`false_positive`, or `unresolved` dispositions. Omit `metrics` when the
-provider did not supply a value; do not infer or manufacture telemetry.
-
-Run the scorer:
-
-```bash
-node --experimental-strip-types test/evals/score.mjs path/to/captured-results.json
-```
-
-For each strategy, the score includes:
-
-- precision: verified claims divided by verified plus false-positive claims;
-- recall over all planned finding opportunities, including omitted trials;
-- `findings.verified`: reproducible, human-adjudicated findings;
-- deduplicated unique verified and expected findings;
-- clean-case false-positive rate over submitted clean runs (`null` when none are submitted);
-- planned completeness and observed completeness; and
-- coverage with omitted case/trial pairs, omitted cases, and submitted/omitted
-  counts for clean and finding classes.
-
-Metric summaries give observations, total, and mean only for supplied latency
-or token values. Missing data remains missing. The scorer rejects malformed
-metric arrays, invalid metrics, unknown verified IDs, duplicated oracle IDs,
-and duplicated case/trial records.
-
-## Compare strategies
-
-Use the same public reviewer payload, exact models, tools, system prompts, and
-adjudication contract for each strategy:
-
-1. One fresh reviewer.
-2. The textual legacy `3 Optimizer + 3 Skeptic + 1 synthesis` baseline.
-3. The routine proposal: two fresh reviewers and one fresh synthesis pass.
-4. The high-risk proposal: three lens reviewers, targeted cross-family
-   verifier passes for high-risk claims, and one fresh synthesis pass.
-
-The legacy fixture describes a theoretical offline `3+3+1` topology. It is not
-a runnable baseline in the current spawned-coordinator lifecycle: child starts
-are asynchronous, the coordinator lacks an adapter that awaits and gathers all
-six reports before later stages, and `auto-exit` can end its turn. Do not claim
-a runnable legacy result without that adapter. Label any measurement of the
-fixture as an offline simulation.
-
-Before tuning prompts, create a stratified split. This corpus uses
-`case-01`, `case-02`, `case-04`, and `case-06` for tuning, and `case-03`,
-`case-05`, `case-07`, and `case-08` for holdout. Each split has clean and
-finding cases. Keep the holdout sealed until the prompt is selected.
-
-For each split:
-
-1. Pin exact model IDs and versions where available, reviewer/synthesizer
-   prompts, tools, and public payloads.
-2. Use the same trial IDs for every strategy and rotate strategy order inside
-   each case/trial block.
-3. Match **total** token budgets across strategies: sum all reviewer, verifier,
-   and synthesis budgets. Per-reviewer budgets alone favor strategies that
-   launch more calls. Report both the total budget and its allocation.
-4. Preserve raw reviewer and synthesis outputs, timestamps, run IDs, and only
-   provider-reported latency/token fields. Redact secrets before storage.
-5. Blind the adjudicator to strategy name, score the captures, and report raw
-   counts, coverage, and absent telemetry beside rates.
-6. Evaluate the sealed holdout only after prompt selection. Treat it as an
-   estimate for this corpus, not a general performance claim.
-
-Actual model measurements require ordinary exact workflow preparation and user
-approval. Prepare the workflow, show its approval packet, and wait for the
-exact approval before starting it. This corpus does not authorize live models
-or approved workflow execution.
-
-## Evidence and limits
-
-This procedure follows vendor guidance to define task-specific ground truth,
-retain traces, and evaluate agent behavior end to end. See Anthropic's
-[Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
-and OpenAI's [agent evals guide](https://developers.openai.com/api/docs/guides/agent-evals).
-
-Diverse passes are a hypothesis, not an independence guarantee. Kim et al.,
-[Correlated Errors in Large Language Models](https://proceedings.mlr.press/v267/kim25e.html)
-(ICML 2025) provides direct evidence that LLM errors can be correlated. It does
-not measure this package or code review, so this corpus measures misses and
-false positives instead of assuming provider or lens diversity solves them.

package/pi-extension/subagents/model-config.ts

@@ -1,305 +0,0 @@
-import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
-import { dirname, join } from "node:path";
-import { getSubagentsConfigPath } from "./config-path.ts";
-import { isPlainObject, isString } from "./type-guards.ts";
-
-export const TASK_CATEGORIES = [
-	"coding",
-	"review",
-	"recon",
-	"qa",
-	"architecture",
-	"docs",
-] as const;
-export type TaskCategory = (typeof TASK_CATEGORIES)[number];
-export const TASK_CATEGORY_DESCRIPTIONS = {
-	coding: "Implementation workers",
-	review: "Code reviewers",
-	recon: "Reconnaissance scouts",
-	qa: "Software and test runners",
-	architecture: "Planning and diagnosis",
-	docs: "Documentation workers",
-} satisfies Record<TaskCategory, string>;
-export type TaskPreferences = Partial<Record<TaskCategory, string[]>>;
-export interface TaskPreferencesMeta {
-	generatedAt: string;
-	method: "research" | "registry-only";
-}
-
-export interface ModelConfig {
-	default?: string;
-	agents: Record<string, string>;
-	tasks?: TaskPreferences;
-	tasksMeta?: TaskPreferencesMeta;
-}
-
-function invalidModelConfig(source: string, message: string): never {
-	throw new Error(`Invalid subagent model config in ${source}: ${message}`);
-}
-
-function rejectTaskReference(
-	value: string,
-	field: string,
-	source: string,
-): void {
-	if (value.trim().toLowerCase().startsWith("task:")) {
-		invalidModelConfig(
-			source,
-			`${field} cannot use task: references; task: references are only valid in the subagent tool's model parameter`,
-		);
-	}
-}
-
-function parseTasks(value: any, source: string): TaskPreferences | undefined {
-	if (value == null) return undefined;
-	if (!isPlainObject(value))
-		invalidModelConfig(source, "models.tasks must be an object");
-	const keys = Object.keys(value);
-	if (keys.length === 0) return undefined;
-	const unsupported = keys.filter(
-		// SAFETY: this check only compares strings against the fixed category set.
-		(key) => !TASK_CATEGORIES.includes(key as TaskCategory),
-	);
-	if (unsupported.length > 0) {
-		invalidModelConfig(
-			source,
-			`models.tasks.${unsupported[0]} is unsupported; supported categories: ${TASK_CATEGORIES.join(", ")}`,
-		);
-	}
-	const tasks: TaskPreferences = {};
-	for (const category of TASK_CATEGORIES) {
-		if (!Object.hasOwn(value, category)) continue;
-		const candidates = value[category];
-		if (!Array.isArray(candidates) || candidates.length === 0) {
-			invalidModelConfig(
-				source,
-				`models.tasks.${category} must be a non-empty list`,
-			);
-		}
-		const seen = new Set<string>();
-		tasks[category] = candidates.map((candidate, index) => {
-			if (!isString(candidate) || candidate.trim() === "") {
-				invalidModelConfig(
-					source,
-					`models.tasks.${category}[${index}] must be a non-empty string`,
-				);
-			}
-			const reference = candidate.trim();
-			if (seen.has(reference)) {
-				invalidModelConfig(
-					source,
-					`models.tasks.${category} has duplicate candidate ${JSON.stringify(reference)}`,
-				);
-			}
-			seen.add(reference);
-			return reference;
-		});
-	}
-	return tasks;
-}
-
-function parseTasksMeta(
-	value: any,
-	source: string,
-): TaskPreferencesMeta | undefined {
-	if (value == null) return undefined;
-	if (!isPlainObject(value))
-		invalidModelConfig(source, "models.tasksMeta must be an object");
-	const unsupported = Object.keys(value).filter(
-		(key) => key !== "generatedAt" && key !== "method",
-	);
-	if (unsupported.length > 0) {
-		invalidModelConfig(
-			source,
-			`models.tasksMeta has unsupported key(s): ${unsupported.join(", ")}`,
-		);
-	}
-	if (
-		!isString(value.generatedAt) ||
-		!/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})$/.test(
-			value.generatedAt,
-		) ||
-		Number.isNaN(Date.parse(value.generatedAt))
-	) {
-		invalidModelConfig(
-			source,
-			"models.tasksMeta.generatedAt must be an ISO-8601 string",
-		);
-	}
-	if (value.method !== "research" && value.method !== "registry-only") {
-		invalidModelConfig(
-			source,
-			'models.tasksMeta.method must be "research" or "registry-only"',
-		);
-	}
-	return { generatedAt: value.generatedAt, method: value.method };
-}
-
-export function parseModelConfig(
-	rawConfig: any,
-	source = "config.json",
-): ModelConfig {
-	if (!isPlainObject(rawConfig))
-		invalidModelConfig(source, "root must be an object");
-	const models = rawConfig.models;
-	if (models == null) return { agents: {} };
-	if (!isPlainObject(models))
-		invalidModelConfig(source, "models must be an object");
-	const allowedKeys = new Set(["default", "agents", "tasks", "tasksMeta"]);
-	const unsupportedKeys = Object.keys(models).filter(
-		(key) => !allowedKeys.has(key),
-	);
-	if (unsupportedKeys.length > 0)
-		invalidModelConfig(
-			source,
-			`models has unsupported key(s): ${unsupportedKeys.join(", ")}`,
-		);
-
-	let defaultModel: string | undefined;
-	if (models.default != null) {
-		if (!isString(models.default) || models.default.trim() === "")
-			invalidModelConfig(source, "models.default must be a non-empty string");
-		const trimmedDefault = models.default.trim();
-		defaultModel = trimmedDefault;
-		rejectTaskReference(trimmedDefault, "models.default", source);
-	}
-	const agents: Record<string, string> = {};
-	if (models.agents != null) {
-		if (!isPlainObject(models.agents))
-			invalidModelConfig(source, "models.agents must be an object");
-		for (const [agent, model] of Object.entries(models.agents)) {
-			if (!isString(model) || model.trim() === "")
-				invalidModelConfig(
-					source,
-					`models.agents.${agent} must be a non-empty string`,
-				);
-			const trimmed = model.trim();
-			rejectTaskReference(trimmed, `models.agents.${agent}`, source);
-			Object.defineProperty(agents, agent, {
-				value: trimmed,
-				enumerable: true,
-				writable: true,
-				configurable: true,
-			});
-		}
-	}
-	const tasks = parseTasks(models.tasks, source);
-	const tasksMeta = parseTasksMeta(models.tasksMeta, source);
-	const config: ModelConfig = { agents };
-	if (defaultModel) config.default = defaultModel;
-	if (tasks) config.tasks = tasks;
-	if (tasksMeta) config.tasksMeta = tasksMeta;
-	return config;
-}
-
-export function resolveModelDefault(
-	agentName: string | undefined,
-	agentModel: string | undefined,
-	config: ModelConfig,
-): string | undefined {
-	if (agentModel) return agentModel;
-	if (agentName && Object.hasOwn(config.agents, agentName))
-		return config.agents[agentName];
-	return config.default;
-}
-
-export function loadModelConfig(
-	configPath = getSubagentsConfigPath(),
-): ModelConfig {
-	let raw: string;
-	try {
-		raw = readFileSync(configPath, "utf8");
-	} catch (error) {
-		// SAFETY: readFileSync errors expose the Node errno code.
-		if ((error as NodeJS.ErrnoException).code === "ENOENT")
-			return { agents: {} };
-		throw error;
-	}
-	try {
-		return parseModelConfig(JSON.parse(raw), configPath);
-	} catch (error) {
-		if (error instanceof SyntaxError)
-			throw new Error(
-				`Invalid JSON in subagent model config ${configPath}: ${error.message}`,
-			);
-		throw error;
-	}
-}
-
-export interface SavedTaskModelConfig {
-	configPath: string;
-	tasks: TaskPreferences;
-	tasksMeta: TaskPreferencesMeta | undefined;
-	missingCategories: TaskCategory[];
-}
-
-/** Atomically replace only models.tasks and models.tasksMeta in the durable user config. */
-export function writeTaskModelConfig(
-	configPath: string,
-	examplePath: string,
-	tasks: TaskPreferences,
-	tasksMeta: TaskPreferencesMeta,
-	isAuthenticatedCandidate: (candidate: string) => boolean,
-	fileOperations: Pick<
-		typeof import("node:fs"),
-		"renameSync" | "writeFileSync"
-	> = { renameSync, writeFileSync },
-): SavedTaskModelConfig {
-	const candidateConfig = parseModelConfig(
-		{ models: { tasks, tasksMeta } },
-		configPath,
-	);
-	for (const candidates of Object.values(candidateConfig.tasks ?? {})) {
-		for (const candidate of candidates) {
-			if (!isAuthenticatedCandidate(candidate)) {
-				throw new Error(
-					`Task model candidate ${JSON.stringify(candidate)} is not an authenticated exact registry model`,
-				);
-			}
-		}
-	}
-	mkdirSync(dirname(configPath), { recursive: true });
-	const current = readFileIfExists(configPath);
-	const source = current ?? readFileSync(examplePath, "utf8");
-	let parsed: any;
-	try {
-		parsed = JSON.parse(source);
-	} catch (error) {
-		const path = current == null ? examplePath : configPath;
-		throw new Error(
-			`Invalid JSON in subagent config ${path}: ${error instanceof Error ? error.message : String(error)}`,
-		);
-	}
-	if (!isPlainObject(parsed))
-		throw new Error(
-			`Invalid JSON in subagent config ${configPath}: root must be an object`,
-		);
-	const models = isPlainObject(parsed.models) ? { ...parsed.models } : {};
-	models.tasks = candidateConfig.tasks;
-	models.tasksMeta = candidateConfig.tasksMeta;
-	const output = JSON.stringify({ ...parsed, models }, null, 2) + "\n";
-	const temporary = join(
-		dirname(configPath),
-		`.${Date.now()}-${process.pid}-config.tmp`,
-	);
-	fileOperations.writeFileSync(temporary, output, { flag: "wx" });
-	fileOperations.renameSync(temporary, configPath);
-	return {
-		configPath,
-		tasks: candidateConfig.tasks ?? {},
-		tasksMeta: candidateConfig.tasksMeta,
-		missingCategories: TASK_CATEGORIES.filter(
-			(category) => !candidateConfig.tasks?.[category],
-		),
-	};
-}
-
-function readFileIfExists(path: string): string | undefined {
-	try {
-		return readFileSync(path, "utf8");
-	} catch (error) {
-		// SAFETY: readFileSync errors expose the Node errno code.
-		if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined;
-		throw error;
-	}
-}