@dailephd/my-dev-kit-lab 0.9.1 → 0.9.3

package/CHANGELOG.md

@@ -2,6 +2,25 @@
 
 All notable changes to my-dev-kit-lab are documented here.
 
+## [0.9.3] - 2026-10-16
+
+- Improved security-validator runtime evidence and secret-scan accuracy while retaining bounded proof and candidate-versus-confirmed finding distinctions.
+- Added configurable Codex and Claude real-agent prompt contracts, strict structured-diff validation, bounded prompt modes, and persisted campaign prompt identity. Provider enforcement and effectiveness remain unverified without authorized live-provider evidence.
+- Added a generic report-section registry, static HTML metric evidence and caveats, generalized gallery source readers, read-only tutorial-manifest consumption, and `report compare-prompts` artifacts.
+- Added evidence-based glossary descriptors and stable links for registered metrics, with explicit unresolved-metadata behavior for unknown IDs; text and HTML reports share reliability caveats. Prompt comparison validates persisted checkpoint/task/prompt evidence, fails closed for unverified V1 identity, and excludes status-only invalid-output claims. Real-provider invalid-output improvement remains NOT_VERIFIED; no paid smoke was run. The v0.9.3 acceptance clarification requires validated prompt/report evidence contracts, while any observed live-provider effectiveness is a separate authorized study, not a claimed release outcome.
+
+## [0.9.2] - 2026-10-13
+
+### Security validation and cross-platform corrections
+
+- Corrected external npm package validation to derive required entrypoints from validated package metadata while preserving the strict Lab self-package inventory. Missing declared entrypoints, invalid or incomplete metadata, failed or incomplete package evidence, and forbidden contents remain blockers.
+- Eliminated nine inherited Lab-only false positives when validating the external Orchestrator package. This does not change the Orchestrator package or its independent security findings and release status.
+- Corrected Semgrep failed and incomplete scan evidence so neither can produce a false `PASS`; unavailable optional scanners remain explicitly `SKIPPED`.
+- Corrected campaign filesystem identity and path-safety behavior and cross-platform compatibility.
+- Preserved security verdict strictness, severity mappings, mandatory checks, and existing command and report contracts.
+
+This section records the intended v0.9.2 release contents. Registry publication status is established through npm evidence, not this changelog.
+
 ## [0.9.1] - 2026-10-12
 
 ### Normalized provider telemetry and resumable campaigns
@@ -11,8 +30,6 @@ All notable changes to my-dev-kit-lab are documented here.
 - Added public campaign CLI options, Codex and Claude timeout presets, campaign progress summaries, and strong/partial/unavailable metric-specific evidence classifications.
 - Preserved partial evidence without presenting it as complete scientific analysis; uncertain provider invocations are not replayed automatically. Real-provider use remains explicitly opted in and depends on the existing provider adapters.
 
-## [Unreleased]
-
 ## [0.9.0] - 2026-10-10
 
 ### Agent success rate

package/README.md

@@ -6,16 +6,22 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.9.1 (normalized provider telemetry and resumable agent-success-rate campaigns), and the repository package version is 0.9.1. The previous release is v0.9.0 (agent-success-rate implementation-patch evaluation). The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
+Version v0.9.3 adds hardened prompt contracts and output validation, extensible reports with metric evidence, broader gallery inputs, and prompt-template comparison. Security validation and verdict strictness remain unchanged. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
 
 v0.9.0 adds the `agent-success-rate` plugin, the eighth registered experiment plugin, which evaluates implementation patches under the `raw-full-file` and `context-pack` treatments.
 
-v0.9.1 adds normalized provider telemetry and resumable campaign execution. v0.9.2 remains planned.
+v0.9.2 is the previous package release.
+
+The v0.9.2 release corrected external npm package entrypoint validation, preserved strict Lab self-package validation, eliminated nine inherited false positives, corrected Semgrep failed/incomplete-scan evidence, and included campaign filesystem identity, path-safety, and cross-platform corrections. Security verdict strictness is unchanged. The v0.9.2 release-document state is final; actual npm publication is established separately through registry evidence.
+
+The v0.9.3 release adds configurable Codex/Claude prompt contracts, strict structured-diff output handling, generic report sections, HTML metric evidence and caveats, source-based gallery items including tutorial manifests, and prompt-template comparison. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for release and validation details.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
 ## Current capabilities
 
+Package metadata is `0.9.3`. The v0.9.3 release adds configurable Codex/Claude prompt contracts, strict structured-diff output handling, generic report sections, HTML metric evidence and caveats, source-based gallery items including tutorial manifests, and prompt-template comparison. Registered metrics have evidence-based glossary definitions and links, and prompt comparison validates persisted identity and checkpoint evidence. Real-provider improvement remains NOT_VERIFIED because no paid smoke was authorized; its measurement is a separate opt-in empirical objective, not a v0.9.3 release gate. Bounded tool-use remains prompt guidance, not provider enforcement. See [Current State](docs/CURRENT_STATE.md) and [the roadmap](docs/ROADMAP.md).
+
 - **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
 - **Measure warm-index reuse (v0.5.0–v0.5.1, released):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. The v0.5.1 release adds a dedicated 12-task benchmark corpus (`benchmarks/contracts/warm-index-benchmark-cases.json`: six medium and six large/mixed tasks tagged `localized`, `cross-module`, or `broad-change`) for comparing warm-index behavior as task count grows. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry.
 - **Run selectable real-agent warm-index campaigns (v0.5.2, released):** `--campaign codex-full`, `claude-full`, or `codex-timeout-isolation` (with `--include-real-agents`) runs exactly one provider — Codex or Claude — against the exact raw and warm contexts the experiment already measured, over the frozen production corpus each preset owns. Provider outcomes (`completed`, `failed`, `timeout`, `invalid-output`, `agent-unavailable`, `agent-limit-reached`, `skipped`) are classified and reported separately from warm-index infrastructure status, so a campaign can complete its indexing/retrieval work even when provider evidence is partial or unavailable. A campaign run automatically produces the report, the same four warm-index SVG charts, and a three-item gallery (report, plots, bounded execution evidence). Its report screenshot is best-effort presentation evidence: unavailable browser runtime or failed capture is reported explicitly, no PNG is fabricated, and neither outcome fails an otherwise successful campaign. Legacy runs without `--campaign` keep the existing deterministic fake-agent, report-only behavior unchanged. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
@@ -284,6 +290,8 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 
 `npm run security:validate` (contributor/source-checkout) and `my-dev-kit-lab security validate` (installed CLI) both call the same standalone security-validation command owner. It checks local CLI/package boundaries and can inspect another local project with `--target <path>`. The generic audit command can reuse those results through `--types security`, but it does not replace the standalone validator or its reports.
 
+**External npm package validation (v0.9.2 release documentation):** the external-target package-content correction keeps Lab self-validation's strict required inventory and checks external packages against declared entrypoints. Missing entrypoints, invalid or incomplete metadata, failed package evidence, and forbidden contents remain blockers. The nine inherited Lab-only Orchestrator false positives are eliminated. The prompt/report/gallery feature scope is included in v0.9.3. Earlier failed readiness attempts remain historical and are superseded by the accepted readiness result for the exact validated SHA; see [CURRENT_STATE.md](docs/CURRENT_STATE.md).
+
 Android validation uses `--profile android`. Its default path is static and non-destructive: it starts zero Gradle processes, zero external tools, and zero network operations. Only confirmed `SecurityFinding` records can become audit issues; Android `CandidateEvidence` remains review-only evidence.
 
 Optional local scanners are reported as `skipped` when unavailable, never as passed. This local availability rule does not make CodeQL coverage optional for release readiness: the exact candidate must pass the repository-controlled GitHub CodeQL workflow and have its Code Scanning alerts reviewed. The framework does not provide runtime isolation proof, device or APK/AAB analysis, signing verification, Play Console validation, automatic fixes, or manual pentesting. Manual pentest remains deferred until post-v1/version TBD.

package/dist/src/agents/adapters/codexAdapter.js

@@ -148,7 +148,10 @@ export async function runCliAgent(request, adapter) {
         env: request.env,
         timeoutMs: request.timeoutMs,
         stdinText: command.stdinText,
-        ...(launch ? { stdoutMaxBytes: 8 * 1024 * 1024, stderrMaxBytes: 1024 * 1024, terminateProcessTree: true } : {})
+        ...(launch ? { stdoutMaxBytes: 8 * 1024 * 1024, stderrMaxBytes: 1024 * 1024 } : {}),
+        // PATH-resolved providers can be Windows .cmd shims too. Always request tree termination
+        // for provider timeouts; trustedExecutable only controls additional output bounds.
+        terminateProcessTree: true
     });
     const ended = Date.now();
     const combinedOutput = `${measured.stdout}\n${measured.stderr}`;

package/dist/src/cli/help.js

@@ -12,6 +12,7 @@ export function renderTopLevelHelp() {
         "  my-dev-kit-lab [--workspace <path>] experiment run --experiment <id> [options]",
         "  my-dev-kit-lab [--workspace <path>] experiment controlled --cases <path> --out <dir> [options]",
         "  my-dev-kit-lab [--workspace <path>] report render --experiment <dir> --out <dir> [options]",
+        "  my-dev-kit-lab [--workspace <path>] report compare-prompts --baseline <run-root> --candidate <run-root> --out <output-root>",
         "  my-dev-kit-lab [--workspace <path>] plots generate --experiment <dir> --out <dir>",
         "  my-dev-kit-lab [--workspace <path>] gallery build --out <dir> [options]",
         "  my-dev-kit-lab tutorial validate --scenario <path> [--target-contract <path>]",
@@ -32,6 +33,7 @@ export function renderTopLevelHelp() {
         "  experiment run             Run a registered experiment plugin",
         "  experiment controlled    Run a controlled context-strategy-comparison experiment",
         "  report render             Render an experiment report from controlled-experiment artifacts",
+        "  report compare-prompts    Compare two completed real-agent runs by prompt template (read-only)",
         "  plots generate             Generate experiment plot artifacts",
         "  gallery build              Build a gallery manifest/index",
         "  tutorial validate          Validate a tutorial scenario and optional target contract",
@@ -312,6 +314,20 @@ export function renderExperimentRunHelp() {
         "                                                      are reported separately. Results are descriptive for the finite",
         "                                                      corpus: no winner, ranking, composite score or significance claim.",
         "",
+        "agent-success-rate prompt contract options (real-agent mode only):",
+        "  --prompt-template <legacy-diff-v1|strict-diff-v1>   Prompt/response contract (real-agent mode only; default",
+        "                                                      legacy-diff-v1, the unchanged historical prompt). strict-diff-v1",
+        "                                                      requires one JSON object {schemaVersion, patch}.",
+        "  --prompt-length <standard|short>                    Instruction length (real-agent mode only; default standard).",
+        "                                                      short trims instruction wording, never context or restrictions.",
+        "  --prompt-explanation <legacy|none>                  Explanation policy (real-agent mode only; default legacy).",
+        "                                                      none rejects any prose outside the required response.",
+        "  --prompt-tool-use <prohibited|bounded>              Tool-use statement (real-agent mode only; default prohibited).",
+        "                                                      bounded states an explicit zero command budget.",
+        "  --max-command-count <0>                             Command-count guidance (real-agent mode only; only 0 is",
+        "                                                      accepted). Prompt options never enable a provider and are not",
+        "                                                      proof that a provider could not use tools.",
+        "",
         "agent-success-rate resumable campaigns and timeouts only:",
         ...AGENT_CAMPAIGN_OPTION_HELP.map(line => `  ${line}`),
         "Examples (replace bracketed values; explicit source-root is required when package source proof is unavailable):",
@@ -352,11 +368,30 @@ export function renderReportHelp() {
         "",
         "Usage:",
         "  my-dev-kit-lab report render [options]",
+        "  my-dev-kit-lab report compare-prompts --baseline <run-root> --candidate <run-root> --out <output-root>",
         "  my-dev-kit-lab report --help",
         "",
         "Commands:",
         "  render   Render an HTML/JSON experiment report from controlled-experiment artifacts.",
-        "           Run \"my-dev-kit-lab report render --help\" for details."
+        "           Run \"my-dev-kit-lab report render --help\" for details.",
+        "  compare-prompts   Compare two completed real-agent agent-success-rate runs by prompt template (read-only).",
+        "           Run \"my-dev-kit-lab report compare-prompts --help\" for details."
+    ].join("\n");
+}
+export function renderReportComparePromptsHelp() {
+    return [
+        "my-dev-kit-lab report compare-prompts - compare persisted evidence from two prompt-template runs",
+        "",
+        "Usage:",
+        "  my-dev-kit-lab report compare-prompts --baseline <run-root> --candidate <run-root> --out <output-root>",
+        "",
+        "Required (each exactly once):",
+        "  --baseline <run-root>    Completed agent-success-rate run output root (baseline template)",
+        "  --candidate <run-root>   Completed agent-success-rate run output root (candidate template)",
+        "  --out <output-root>      Output root; must not overlap either run root",
+        "",
+        "Writes prompt-template-comparison.json, .txt and .html. The command only reads persisted evidence:",
+        "it never invokes a provider, creates a campaign or modifies a checkpoint."
     ].join("\n");
 }
 export function renderReportRenderHelp() {
@@ -439,7 +474,8 @@ export function renderGalleryBuildHelp() {
         "  --report <dir>           Path to a rendered report directory",
         "  --plots <dir>            Path to a plots output directory",
         "  --visualizations <dir>   Path to a visualization-demos output directory",
-        "  --experiment <dir>       Path to a controlled-experiment output directory"
+        "  --experiment <dir>       Path to a controlled-experiment output directory",
+        "  --source <path>          Artifact root or supported manifest to include; repeatable"
     ].join("\n");
 }
 export function renderSecurityHelp() {

package/dist/src/cli/runLabCli.js

@@ -10,10 +10,11 @@ import { runExperimentRunCommandFromArgs } from "../commands/runExperimentRunCom
 import { runRenderExperimentReportCommand } from "../commands/renderExperimentReportCommand.js";
 import { runGenerateExperimentPlotsCommand } from "../commands/generateExperimentPlotsCommand.js";
 import { runBuildGalleryCommand } from "../commands/buildGalleryCommand.js";
+import { runComparePromptsCommand } from "../commands/runComparePromptsCommand.js";
 import { runTutorialValidateCommandFromArgs } from "../commands/runTutorialValidateCommand.js";
 import { runTutorialRunCommandFromArgs } from "../commands/runTutorialRunCommand.js";
 import { createLabExecutionContext, discoverPackageRoot } from "../runtime/index.js";
-import { renderDemoHelp, renderExperimentControlledHelp, renderExperimentDescribeHelp, renderExperimentHelp, renderExperimentListHelp, renderExperimentRunHelp, renderFinalDemoHelp, renderGalleryBuildHelp, renderGalleryHelp, renderPlotsGenerateHelp, renderPlotsHelp, renderReportHelp, renderReportRenderHelp, renderSecurityHelp, renderTopLevelHelp, renderTutorialHelp, renderTutorialRunHelp, renderTutorialValidateHelp, renderUnknownCommandError } from "./help.js";
+import { renderDemoHelp, renderExperimentControlledHelp, renderExperimentDescribeHelp, renderExperimentHelp, renderExperimentListHelp, renderExperimentRunHelp, renderFinalDemoHelp, renderGalleryBuildHelp, renderGalleryHelp, renderPlotsGenerateHelp, renderPlotsHelp, renderReportHelp, renderReportRenderHelp, renderReportComparePromptsHelp, renderSecurityHelp, renderTopLevelHelp, renderTutorialHelp, renderTutorialRunHelp, renderTutorialValidateHelp, renderUnknownCommandError } from "./help.js";
 export const CLI_USAGE_EXIT_CODE = 2;
 /**
  * Recognized legacy final-demo flag tokens (mirrors parseRunFinalDemoArgs in
@@ -188,6 +189,13 @@ async function runReportFamily(argv, writers) {
         }
         return runRenderExperimentReportCommand(rest);
     }
+    if (subcommand === "compare-prompts") {
+        if (rest[0] === "--help" || rest[0] === "-h") {
+            writers.stdout(renderReportComparePromptsHelp());
+            return 0;
+        }
+        return runComparePromptsCommand(rest);
+    }
     writers.stderr(renderUnknownCommandError(`report ${subcommand}`));
     return CLI_USAGE_EXIT_CODE;
 }

package/dist/src/commands/buildGalleryCommand.js

@@ -1,11 +1,13 @@
 import path from "node:path";
 import { writeExperimentGalleryManifest } from "../gallery/index.js";
+import { looksLikeUrl } from "../gallery/galleryPathSafety.js";
 export function parseBuildGalleryArgs(argv) {
     let outDir = "";
     let reportDir;
     let plotsDir;
     let visualizationsDir;
     let experimentDir;
+    const sources = [];
     for (let index = 0; index < argv.length; index += 1) {
         const arg = argv[index];
         if (arg === "--report") {
@@ -24,14 +26,24 @@ export function parseBuildGalleryArgs(argv) {
             experimentDir = argv[index + 1] ?? "";
             index += 1;
         }
+        else if (arg === "--source") {
+            const value = argv[index + 1];
+            if (value === undefined || value.trim() === "" || value.startsWith("--")) {
+                throw new Error("--source requires a non-empty artifact root or supported manifest path.");
+            }
+            if (looksLikeUrl(value))
+                throw new Error(`Gallery source ${value}: URLs are not supported; supply a local path.`);
+            sources.push(value);
+            index += 1;
+        }
         else if (arg === "--out") {
             outDir = argv[index + 1] ?? "";
             index += 1;
         }
     }
     if (!outDir)
-        throw new Error("Usage: --out <dir> [--report <dir>] [--plots <dir>] [--visualizations <dir>] [--experiment <dir>]");
-    return { outDir, reportDir, plotsDir, visualizationsDir, experimentDir };
+        throw new Error("Usage: --out <dir> [--report <dir>] [--plots <dir>] [--visualizations <dir>] [--experiment <dir>] [--source <artifact-root-or-manifest>]...");
+    return { outDir, reportDir, plotsDir, visualizationsDir, experimentDir, ...(sources.length > 0 ? { sources } : {}) };
 }
 export async function runBuildGalleryFromArgs(args, repoRoot = process.cwd()) {
     return writeExperimentGalleryManifest({
@@ -39,7 +51,8 @@ export async function runBuildGalleryFromArgs(args, repoRoot = process.cwd()) {
         reportDir: args.reportDir ? path.resolve(repoRoot, args.reportDir) : undefined,
         plotsDir: args.plotsDir ? path.resolve(repoRoot, args.plotsDir) : undefined,
         visualizationsDir: args.visualizationsDir ? path.resolve(repoRoot, args.visualizationsDir) : undefined,
-        experimentDir: args.experimentDir ? path.resolve(repoRoot, args.experimentDir) : undefined
+        experimentDir: args.experimentDir ? path.resolve(repoRoot, args.experimentDir) : undefined,
+        sources: args.sources?.map((source) => path.resolve(repoRoot, source))
     });
 }
 export async function runBuildGalleryCommand(argv) {

package/dist/src/commands/runComparePromptsCommand.js

@@ -0,0 +1,109 @@
+import { mkdir, realpath, stat, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { loadPromptComparisonSide } from "../report/experiments/promptTemplateComparisonEvidence.js";
+import { buildPromptTemplateComparison, PROMPT_TEMPLATE_COMPARISON_HTML_FILE, PROMPT_TEMPLATE_COMPARISON_JSON_FILE, PROMPT_TEMPLATE_COMPARISON_TEXT_FILE } from "../report/experiments/promptTemplateComparisonModel.js";
+import { renderPromptTemplateComparisonHtml, renderPromptTemplateComparisonJson, renderPromptTemplateComparisonText } from "../report/experiments/renderPromptTemplateComparison.js";
+const FLAGS = ["--baseline", "--candidate", "--out"];
+/** Exactly --baseline, --candidate and --out, each once. Unknown flags, positional arguments and empty values are rejected. */
+export function parseComparePromptsArgs(argv) {
+    const values = new Map();
+    for (let index = 0; index < argv.length; index += 1) {
+        const arg = argv[index];
+        if (!FLAGS.includes(arg)) {
+            throw new Error(arg.startsWith("--") ? `Unknown option for report compare-prompts: ${arg}` : `Unexpected argument for report compare-prompts: ${arg}`);
+        }
+        if (values.has(arg))
+            throw new Error(`Duplicate option: ${arg}`);
+        const value = argv[index + 1];
+        if (value === undefined || value.trim() === "" || value.startsWith("--"))
+            throw new Error(`${arg} requires a non-empty path.`);
+        values.set(arg, value);
+        index += 1;
+    }
+    for (const flag of FLAGS) {
+        if (!values.has(flag))
+            throw new Error(`Missing required option: ${flag}. Usage: report compare-prompts --baseline <run-root> --candidate <run-root> --out <output-root>`);
+    }
+    return { baseline: values.get("--baseline"), candidate: values.get("--candidate"), out: values.get("--out") };
+}
+/** Physical path of the nearest existing ancestor plus the not-yet-existing remainder. */
+async function physicalPath(candidate) {
+    let existing = path.resolve(candidate);
+    const remainder = [];
+    for (;;) {
+        try {
+            const real = await realpath(existing);
+            return path.join(real, ...remainder);
+        }
+        catch {
+            const parent = path.dirname(existing);
+            if (parent === existing)
+                return path.resolve(candidate);
+            remainder.unshift(path.basename(existing));
+            existing = parent;
+        }
+    }
+}
+function overlaps(a, b) {
+    const key = (value) => (process.platform === "win32" ? value.toLowerCase() : value);
+    const ra = path.relative(key(a), key(b));
+    const rb = path.relative(key(b), key(a));
+    const inside = (relative) => relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative));
+    return inside(ra) || inside(rb);
+}
+export async function runComparePrompts(args, cwd = process.cwd(), options = {}) {
+    const baselineRoot = path.resolve(cwd, args.baseline);
+    const candidateRoot = path.resolve(cwd, args.candidate);
+    const outRoot = path.resolve(cwd, args.out);
+    const [physicalBaseline, physicalCandidate, physicalOut] = await Promise.all([baselineRoot, candidateRoot, outRoot].map(physicalPath));
+    if (path.relative(process.platform === "win32" ? physicalBaseline.toLowerCase() : physicalBaseline, process.platform === "win32" ? physicalCandidate.toLowerCase() : physicalCandidate) === "") {
+        throw new Error("The baseline and candidate run roots are identical; a comparison needs two distinct runs.");
+    }
+    if (overlaps(physicalOut, physicalBaseline) || overlaps(physicalOut, physicalCandidate)) {
+        throw new Error("The output root must not overlap the baseline or candidate run root.");
+    }
+    try {
+        if (!(await stat(physicalOut)).isDirectory())
+            throw new Error("The output root exists and is not a directory.");
+    }
+    catch (error) {
+        if (error.code !== "ENOENT")
+            throw error;
+    }
+    const [baseline, candidate] = await Promise.all([loadPromptComparisonSide(baselineRoot, "baseline", options), loadPromptComparisonSide(candidateRoot, "candidate", options)]);
+    for (const side of [baseline, candidate]) {
+        if (side.execution === null) {
+            throw new Error(`The ${side.role} run evidence is unreadable or invalid: ${side.unverifiable.join("; ")}`);
+        }
+    }
+    const report = buildPromptTemplateComparison(baseline, candidate);
+    await mkdir(physicalOut, { recursive: true });
+    const paths = {
+        json: path.join(physicalOut, PROMPT_TEMPLATE_COMPARISON_JSON_FILE),
+        text: path.join(physicalOut, PROMPT_TEMPLATE_COMPARISON_TEXT_FILE),
+        html: path.join(physicalOut, PROMPT_TEMPLATE_COMPARISON_HTML_FILE)
+    };
+    await writeFile(paths.json, renderPromptTemplateComparisonJson(report), "utf8");
+    await writeFile(paths.text, renderPromptTemplateComparisonText(report), "utf8");
+    await writeFile(paths.html, renderPromptTemplateComparisonHtml(report), "utf8");
+    return { report, paths };
+}
+export async function runComparePromptsCommand(argv, options = {}) {
+    try {
+        const result = await runComparePrompts(parseComparePromptsArgs(argv), process.cwd(), options);
+        const report = result.report;
+        console.log([
+            `Comparison: ${report.comparison.status}`,
+            `Matched units: ${report.cohort.matchedUnitCount}`,
+            `Behavioral improvement: ${report.behavioralImprovement.status}`,
+            `JSON: ${result.paths.json}`,
+            `Text: ${result.paths.text}`,
+            `HTML: ${result.paths.html}`
+        ].join("\n"));
+        return 0;
+    }
+    catch (error) {
+        console.error(error instanceof Error ? error.message : String(error));
+        return 1;
+    }
+}

package/dist/src/commands/runExperimentRunCommand.js

@@ -1,4 +1,4 @@
-import { lstat, mkdir, readFile, readdir, realpath } from "node:fs/promises";
+import { lstat, mkdir, readFile, readdir } from "node:fs/promises";
 import path from "node:path";
 import { parseAgentCommandTemplate } from "../agents/index.js";
 import { parseAgentId } from "../agents/agentRegistry.js";
@@ -19,11 +19,12 @@ import { createLabExecutionContext, resolvePackageResource } from "../runtime/in
 import { runWarmIndexCampaignPresentation } from "./runWarmIndexCampaignPresentation.js";
 import { observeProviderBuild } from "../agents/providerBuildObservation.js";
 import { resolveAgentTimeoutPreset } from "../experiments/plugins/agentSuccessRate/timeoutPresets.js";
-import { validateAgentSuccessRateConfig } from "../experiments/plugins/agentSuccessRate/config.js";
+import { resolveAgentSuccessPromptContract, validateAgentSuccessRateConfig } from "../experiments/plugins/agentSuccessRate/config.js";
+import { AGENT_SUCCESS_PROMPT_EXPLANATION_MODES, AGENT_SUCCESS_PROMPT_LENGTH_MODES, AGENT_SUCCESS_PROMPT_TEMPLATE_IDS, AGENT_SUCCESS_PROMPT_TOOL_USE_MODES } from "../experiments/plugins/agentSuccessRate/realAgentPrompt.js";
 import { AGENT_SUCCESS_RATE_DEFAULT_AGENT_TIMEOUT_MS } from "../experiments/plugins/agentSuccessRate/realAgentExecution.js";
 import { AGENT_SUCCESS_RATE_TREATMENT_IDS } from "../experiments/plugins/agentSuccessRate/metadata.js";
 import { executeAgentSuccessCampaignV2, prepareAgentSuccessCampaign } from "../experiments/plugins/agentSuccessRate/campaignExecution.js";
-import { readCampaignCheckpointV2, validateCampaignOutputPath } from "../experiments/plugins/agentSuccessRate/campaignCheckpointStore.js";
+import { readCampaignCheckpointV2, resolveCampaignRoot, validateCampaignOutputPath } from "../experiments/plugins/agentSuccessRate/campaignCheckpointStore.js";
 import { validateCampaignLimitAuthorization } from "../experiments/plugins/agentSuccessRate/campaignRecovery.js";
 import { campaignIdentifier } from "../experiments/plugins/agentSuccessRate/campaignIdentity.js";
 import { readCampaignProgress } from "../experiments/plugins/agentSuccessRate/campaignProgress.js";
@@ -98,6 +99,7 @@ const AGENT_SUCCESS_RATE_ALLOWED_FLAGS = [
     "--include-real-agents",
     "--timeout-ms",
     "--repair-attempts",
+    "--prompt-template", "--prompt-length", "--prompt-explanation", "--prompt-tool-use", "--max-command-count",
     "--agent-campaign", "--one-case", "--campaign-source-root", "--timeout-preset",
     "--expected-checkpoint-revision", "--limit-case", "--limit-treatment", "--limit-epoch", "--authorize-additional-invocation"
 ];
@@ -263,7 +265,8 @@ async function runAgentCampaignCommand(args, context, outputRoot, inputs, runId,
         const executingMetadata = JSON.parse(await readFile(path.join(context.packageRoot, "package.json"), "utf8"));
         if (executingMetadata.name !== packageName || typeof executingMetadata.version !== "string" || !validPackageVersion.test(executingMetadata.version))
             throw new Error();
-        if (!(await lstat(path.join(sourceRoot, "src"))).isDirectory() || path.resolve(await realpath(sourceRoot)).toLowerCase() !== sourceRoot.toLowerCase())
+        await resolveCampaignRoot(sourceRoot);
+        if (!(await lstat(path.join(sourceRoot, "src"))).isDirectory())
             throw new Error();
         const metadata = JSON.parse(await readFile(path.join(sourceRoot, "package.json"), "utf8"));
         if (metadata.name !== packageName || metadata.version !== executingMetadata.version)
@@ -294,7 +297,7 @@ async function runAgentCampaignCommand(args, context, outputRoot, inputs, runId,
     const config = validation.config;
     const tasks = selectAgentSuccessTasks(inputs.agentSuccessTasks, config);
     const runtime = options.campaignRuntime;
-    const provider = config.agentId ? { kind: "real-agent", providerId: config.agentId, includeRealAgents: true, timeoutMs: config.timeoutMs ?? AGENT_SUCCESS_RATE_DEFAULT_AGENT_TIMEOUT_MS, repairAttempts: (config.repairAttempts ?? 0), kitCommand: config.kitCommand, contextDependencies: runtime?.contextDependencies, env: runtime?.env } : { kind: "deterministic-fixture" };
+    const provider = config.agentId ? { kind: "real-agent", providerId: config.agentId, includeRealAgents: true, timeoutMs: config.timeoutMs ?? AGENT_SUCCESS_RATE_DEFAULT_AGENT_TIMEOUT_MS, repairAttempts: (config.repairAttempts ?? 0), kitCommand: config.kitCommand, contextDependencies: runtime?.contextDependencies, env: runtime?.env, ...(resolveAgentSuccessPromptContract(config) ? { promptContract: resolveAgentSuccessPromptContract(config) } : {}) } : { kind: "deterministic-fixture" };
     const preparation = { boundary, sourceRoot, corpusFile: resolvePackageResource(context, AGENT_SUCCESS_TASK_CATALOG_PATH), corpusRoot: context.packageRoot, tasks, treatments: AGENT_SUCCESS_RATE_TREATMENT_IDS, provider, dependencies: runtime?.dependencies };
     // Source, corpus and canonical project byte checks precede even a task-free version probe.
     let prepared = await prepareAgentSuccessCampaign(preparation);
@@ -380,6 +383,11 @@ export function parseRunExperimentArgs(argv) {
     let localSubjectConfigPath;
     const agentFlagValues = [];
     let repairAttempts;
+    let promptTemplate;
+    let promptLength;
+    let promptExplanation;
+    let promptToolUse;
+    let maxCommandCount;
     const seenFlags = [];
     const commandTemplates = {};
     // The flag whitelist runs before any value is interpreted so an unsupported flag is reported as such.
@@ -458,6 +466,24 @@ export function parseRunExperimentArgs(argv) {
         else if (arg === "--agent") {
             agentFlagValues.push(readRequiredValue(argv, ++index, "--agent"));
         }
+        else if (arg === "--prompt-template") {
+            promptTemplate = parseChoice("--prompt-template", readRequiredValue(argv, ++index, arg), AGENT_SUCCESS_PROMPT_TEMPLATE_IDS);
+        }
+        else if (arg === "--prompt-length") {
+            promptLength = parseChoice("--prompt-length", readRequiredValue(argv, ++index, arg), AGENT_SUCCESS_PROMPT_LENGTH_MODES);
+        }
+        else if (arg === "--prompt-explanation") {
+            promptExplanation = parseChoice("--prompt-explanation", readRequiredValue(argv, ++index, arg), AGENT_SUCCESS_PROMPT_EXPLANATION_MODES);
+        }
+        else if (arg === "--prompt-tool-use") {
+            promptToolUse = parseChoice("--prompt-tool-use", readRequiredValue(argv, ++index, arg), AGENT_SUCCESS_PROMPT_TOOL_USE_MODES);
+        }
+        else if (arg === "--max-command-count") {
+            const value = readRequiredValue(argv, ++index, arg);
+            if (!/^\d+$/.test(value) || Number(value) !== 0)
+                throw new Error("--max-command-count must be the integer 0; the benchmark permits no commands.");
+            maxCommandCount = 0;
+        }
         else if (arg === "--repair-attempts") {
             repairAttempts = parseRepairAttempts(readRequiredValue(argv, ++index, "--repair-attempts"));
         }
@@ -649,6 +675,10 @@ export function parseRunExperimentArgs(argv) {
         if (timeoutMs !== undefined && timeoutMs > AGENT_SUCCESS_RATE_MAX_AGENT_TIMEOUT_MS) {
             throw new Error(`--timeout-ms must be a positive integer no greater than ${AGENT_SUCCESS_RATE_MAX_AGENT_TIMEOUT_MS}.`);
         }
+        const promptFlagsSupplied = PROMPT_CONTRACT_FLAGS.filter((flag) => seenFlags.includes(flag));
+        if (agentId === undefined && promptFlagsSupplied.length > 0) {
+            throw new Error(`${promptFlagsSupplied.join(", ")} ${promptFlagsSupplied.length === 1 ? "is" : "are"} only supported together with --agent and --include-real-agents (real-agent mode); prompt options never enable a provider.`);
+        }
         if (seenFlags.includes("--case") && caseIds.length === 0) {
             throw new Error("--case must list at least one case id.");
         }
@@ -657,6 +687,10 @@ export function parseRunExperimentArgs(argv) {
         }
     }
     else {
+        const unrelatedPromptFlags = PROMPT_CONTRACT_FLAGS.filter((flag) => seenFlags.includes(flag));
+        if (unrelatedPromptFlags.length > 0) {
+            throw new Error(`${unrelatedPromptFlags.join(", ")} ${unrelatedPromptFlags.length === 1 ? "is" : "are"} only supported for --experiment ${agentSuccessRatePlugin.metadata.id}.`);
+        }
         if (agentFlagValues.length > 0) {
             throw new Error(`--agent is only supported for --experiment ${agentSuccessRatePlugin.metadata.id}.`);
         }
@@ -747,6 +781,11 @@ export function parseRunExperimentArgs(argv) {
             commandTemplates: Object.keys(commandTemplates).length > 0 ? commandTemplates : undefined,
             agentId: agentFlagValues[0],
             repairAttempts,
+            promptTemplate,
+            promptLength,
+            promptExplanation,
+            promptToolUse,
+            maxCommandCount,
             kitCommand,
             campaignPreset,
             contextBudgets
@@ -1103,6 +1142,13 @@ function readRequiredValue(argv, index, label) {
     }
     return value;
 }
+const PROMPT_CONTRACT_FLAGS = ["--prompt-template", "--prompt-length", "--prompt-explanation", "--prompt-tool-use", "--max-command-count"];
+function parseChoice(label, value, allowed) {
+    if (!allowed.includes(value)) {
+        throw new Error(`${label} must be one of: ${allowed.join(", ")}.`);
+    }
+    return value;
+}
 function parseRepairAttempts(value) {
     if (!/^[0-2]$/.test(value)) {
         throw new Error("--repair-attempts must be 0, 1 or 2.");

package/dist/src/core/processTree.js

@@ -1,4 +1,5 @@
 import { spawn } from "node:child_process";
+import path from "node:path";
 /**
  * Asks a process to terminate normally.
  *
@@ -27,52 +28,75 @@ export function requestProcessTermination(pid) {
  * fire-and-forget and, off Windows, signals one pid only.
  *
  * Windows: `taskkill /pid <pid> /T /F` (shell:false) is awaited, bounded by
- * `waitTimeoutMs`, so the caller knows the tree was asked to die before it
- * proceeds.
+ * `waitTimeoutMs`. Returns false if the command fails or times out.
  *
  * Other platforms: the child must have been spawned `detached` so it leads its
  * own process group; the whole group is signalled via the negative pid. If no
  * group exists, the single pid is signalled as a fallback.
  *
- * Never throws: the target may already have exited.
+ * Never throws. Returns true only when termination was confirmed (or no PID was supplied).
  */
 export async function terminateProcessTree(pid, options = {}) {
     if (!pid) {
-        return;
+        return true;
     }
     const platform = options.platform ?? process.platform;
     if (platform === "win32") {
-        await new Promise((resolve) => {
+        return await new Promise((resolve) => {
             let timer;
-            const finish = () => {
+            let settled = false;
+            const finish = (terminated) => {
+                if (settled)
+                    return;
+                settled = true;
                 if (timer)
                     clearTimeout(timer);
-                resolve();
+                resolve(terminated);
             };
-            const killer = spawn("taskkill", ["/pid", String(pid), "/T", "/F"], {
-                shell: false,
-                stdio: "ignore",
-                windowsHide: true
-            });
-            timer = setTimeout(finish, options.waitTimeoutMs ?? 5000);
-            killer.on("error", finish);
-            killer.on("close", finish);
+            let killer;
+            try {
+                const systemRoot = Object.entries(process.env).find(([name]) => name.toLowerCase() === "systemroot")?.[1];
+                const taskkill = systemRoot ? path.join(systemRoot, "System32", "taskkill.exe") : "taskkill";
+                killer = spawn(taskkill, ["/pid", String(pid), "/T", "/F"], {
+                    shell: false,
+                    stdio: "ignore",
+                    windowsHide: true
+                });
+            }
+            catch {
+                finish(false);
+                return;
+            }
+            timer = setTimeout(() => {
+                if (killer.pid) {
+                    try {
+                        killer.kill();
+                    }
+                    catch {
+                        // The bounded helper may already have exited.
+                    }
+                }
+                finish(false);
+            }, options.waitTimeoutMs ?? 5000);
+            killer.on("error", () => finish(false));
+            killer.on("close", (code) => finish(code === 0));
         });
-        return;
     }
     const signal = options.posixSignal ?? "SIGKILL";
     try {
         process.kill(-pid, signal);
-        return;
+        return true;
     }
     catch {
         // No process group for this pid (or it already exited); fall back to the single pid.
     }
     try {
         process.kill(pid, signal);
+        return true;
     }
     catch {
         // The process may have exited between the liveness check and this signal.
+        return false;
     }
 }
 /**