codex-plugin-doctor 1.48.0 → 1.49.0

package/README.md

@@ -319,6 +319,8 @@ External corpus evaluation is static and offline. `doctor corpus --manifest <cor
 
 `doctor corpus metrics --manifest <corpus.json>` measures reviewed true positives, false positives, and false negatives as precision, recall, and emitted-finding false-positive share. Optional thresholds make the measurement release-gating; incomplete reviews and digest drift return exit `2` instead of producing a favorable score. The dedicated metrics manifest stays offline, requires contained snapshot paths, and may record immutable HTTPS Git provenance without fetching it.
 
+`doctor corpus metrics diff --before <metrics.json> --after <metrics.json>` compares two complete metrics reports from the same corpus identity. Add `--fail-on-regression` to fail CI when precision or recall decreases, or when the reviewed emitted-finding false-positive share increases. Reports with different `corpusDigest` values are rejected as non-comparable, and diff output never includes the input report paths.
+
 `self-test` runs the bundled runtime-complete sample through static validation, runtime MCP probes, and the compatibility scorecard. It is the fastest post-install check after `npm install -g codex-plugin-doctor`.
 
 `doctor` checks the local environment, including package version, platform, Node version, npm global prefix, Codex home, and Codex plugin cache visibility. The text output also includes recommended next commands for self-test, installed plugin discovery, runtime checks, compatibility scoring, and CI setup. `doctor contract` publishes the machine-readable output contract, including public JSON schema surfaces, stable-through-1.0 compatibility metadata, and a frozen rule catalog digest. Add `--json` for automation or `--output output-contract.json` to write the contract to disk. `doctor corpus` runs the bundled validation corpus against healthy runtime, risky security, starter skill, and generic MCP packages, then reports whether each case matched its expected outcome. Add `--json` for automation or `--output validation-corpus.json` to write the corpus report to disk. `doctor npm <package>` runs a preinstall scan by packing the npm package with scripts disabled, extracting the publish tarball, and running validation, security, trust, and recommendation checks against the shipped contents. The JSON report includes tarball metadata such as filename, path, integrity, shasum, sizes, file count, and extracted package root so automation can prove which archive was scanned. Use a published Codex plugin package as the target; scanning `codex-plugin-doctor` itself intentionally reports a missing plugin manifest because this CLI package is not a plugin package. Add `--json` for automation or `--output npm-preinstall.json` to write the report to disk. `doctor attest <path>` creates a local attestation with stable package/report digests, validation/security/compatibility/trust summary, and verification metadata. Add `--sign-key-env NAME` to attach a local HMAC-SHA256 signature without printing the secret, or `--json --output attestation.json` to write the artifact to disk. `doctor attest verify <attestation.json> --target <path> --sign-key-env NAME` recomputes the package fingerprint, report digest, and HMAC signature offline; verification intentionally treats `generatedAt`, `targetPath`, `verification`, and `signature.keyHint` as unsigned display metadata. `doctor runtime-plan <path>` creates a non-executing runtime plan that lists MCP server commands, safe probe methods, risk reasons, and a stable approval digest before any local server is started. Add `--markdown --output runtime-plan.md` to preserve a review-ready approval artifact with the execution boundary, checklist, servers, probes, and risk reasons. `doctor runtime-policy <path>` evaluates the same runtime plan and security signals, then recommends `allow`, `review`, `sandbox_recommended`, or `deny` before local MCP execution starts. `doctor review-bundle <path> --output <dir> --sign-key-env NAME` writes a signed review directory with runtime plan, runtime policy, attestation, release evidence, manifest, Markdown summary files, and SHA-256 file integrity digests. `doctor review-bundle verify <bundle-dir> --target <path> --sign-key-env NAME` verifies the bundle manifest, expected files, manifest integrity digests, runtime artifacts, signed attestation, and signed release evidence offline before a reviewer trusts the handoff. `doctor review-bundle diff --before <dir> --after <dir>` compares two review bundles and flags risk-increasing changes in status, runtime policy, release readiness, signatures, release evidence, and runtime plan digest. `check --runtime --require-runtime-approval --runtime-approval-digest <digest>` refuses to run runtime probes unless the current plan digest matches the approved digest. `doctor release-evidence <path> --sign-key-env NAME` creates one redacted release bundle with signed attestation, offline verification, corpus, performance, security, trust, package metadata, git release gates, and runtime approval status. Strict release evidence requires a clean tagged worktree; use `--allow-dirty` or `--allow-untagged` only for local rehearsal. `doctor release-evidence verify <evidence.json> --target <path> --sign-key-env NAME` verifies a shared release evidence artifact offline against an explicit package path; the artifact target path is treated as display metadata, not trusted input. `doctor release-evidence asset <path> --tag <tag> --output <evidence.json> --sign-key-env NAME` writes a signed release evidence file and prints the `gh release upload` command; add `--upload` to run the upload through GitHub CLI with `--clobber`. `doctor inspector <path>` builds a safe MCP Inspector launch command from a packaged `.mcp.json` file without starting the Inspector proxy automatically. Use `--server <name>` when the package contains multiple MCP server entries. `doctor diff --before <path> --after <path>` compares two package roots and reports new findings, resolved findings, trust score delta, and whether risk increased. `doctor recommend <path>` turns validation, security, and compatibility signals into a prioritized action plan with blocker, high, medium, and info actions. Add `--json` for automation or `--output recommendations.json` to write the report to disk. `doctor trust <path>` creates a local trust score from package lifecycle scripts, dependency specs, and MCP security findings. Use it before release when you want supply-chain risks summarized as one score. `doctor perf <path>` profiles the shared package analysis pipeline and reports per-stage durations for validation, config, security, compatibility, trust, recommendations, and total runtime. Add `--max-total-ms <ms>` or repeatable `--max-stage-ms stage=ms` to fail CI when a budget is exceeded. `doctor mcp <path>` exposes the generic MCP static health report under the doctor command family without starting local MCP servers. `doctor export --bundle <path>` creates a redacted operator handoff bundle that includes validation JSON, security scorecard data, compatibility matrix, recommendations, and trust score in one file. `doctor snapshot` creates a redacted diagnostics bundle with environment health, client config readiness, installed plugin metadata, and next commands. Add `--json` for machine-readable output or `--output doctor-snapshot.json` to write the bundle to disk. `doctor clients` reports local Codex, Claude Desktop, Cursor, Cline, and Windsurf config readiness. `doctor --update-check` compares the installed CLI version with the latest npm version and prints the upgrade command when a newer release is available.
@@ -425,9 +427,9 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v5
-      - uses: Esquetta/CodexPluginDoctor@v1.48.0
+      - uses: Esquetta/CodexPluginDoctor@v1.49.0
         with:
-          version: "1.48.0"
+          version: "1.49.0"
           path: .
           runtime: "true"
           policy: codex-publish

package/dist/core/corpus-quality-metrics.d.ts

@@ -87,6 +87,7 @@ export interface CorpusQualityMetricsReport {
     kind: "doctor.validation.corpus.metrics";
     generatedAt: string;
     version: string;
+    corpusDigest: string;
     status: "pass" | "fail" | "incomplete";
     exitCode: 0 | 1 | 2;
     summary: CorpusMetricCounts & CorpusMetricValues & {
@@ -98,6 +99,64 @@ export interface CorpusQualityMetricsReport {
     thresholdChecks: CorpusMetricThresholdCheck[];
     targets: CorpusQualityMetricsTargetResult[];
 }
+export interface CorpusMetricCountDeltas {
+    truePositives: number;
+    falsePositives: number;
+    falseNegatives: number;
+    resolvedFalsePositives: number;
+    unreviewed: number;
+    unclear: number;
+}
+export interface CorpusMetricValueComparison {
+    before: number | null;
+    after: number | null;
+    delta: number | null;
+}
+export interface CorpusQualityMetricsDiffTarget {
+    id: string;
+    regressed: boolean;
+    counts: CorpusMetricCountDeltas;
+    precision: CorpusMetricValueComparison;
+    recall: CorpusMetricValueComparison;
+    falsePositiveRate: CorpusMetricValueComparison;
+}
+export interface CorpusQualityMetricsDiffReport {
+    schemaVersion: "1.0.0";
+    kind: "doctor.validation.corpus.metrics.diff";
+    generatedAt: string;
+    version: string;
+    corpusDigest: string;
+    status: "pass" | "fail";
+    exitCode: 0 | 1;
+    failOnRegression: boolean;
+    summary: {
+        comparable: true;
+        regression: boolean;
+        changedTargets: number;
+        precisionBefore: number | null;
+        precisionAfter: number | null;
+        precisionDelta: number | null;
+        recallBefore: number | null;
+        recallAfter: number | null;
+        recallDelta: number | null;
+        falsePositiveRateBefore: number | null;
+        falsePositiveRateAfter: number | null;
+        falsePositiveRateDelta: number | null;
+        counts: CorpusMetricCountDeltas;
+    };
+    before: {
+        version: string;
+        targetCount: number;
+    };
+    after: {
+        version: string;
+        targetCount: number;
+    };
+    targets: CorpusQualityMetricsDiffTarget[];
+}
+export interface BuildCorpusMetricsDiffOptions {
+    failOnRegression?: boolean;
+}
 export interface BuildCorpusMetricsOptions {
     thresholds?: CorpusMetricThresholds;
     environment?: {
@@ -114,6 +173,9 @@ export interface BuildCorpusMetricsOptions {
 export declare class CorpusMetricsManifestError extends Error {
     constructor(message: string);
 }
+export declare class CorpusMetricsDiffError extends Error {
+    constructor(message: string);
+}
 export declare function loadCorpusMetricsManifest(manifestPath: string): Promise<LoadedCorpusMetricsManifest>;
 export declare function reconcileCorpusMetricFindings(reviews: CorpusMetricFindingReview[], actualFindings: CorpusMetricFinding[]): CorpusMetricReconciliation;
 export declare function buildCorpusQualityMetricsReport(manifestPath: string, options?: BuildCorpusMetricsOptions): Promise<CorpusQualityMetricsReport>;
@@ -122,4 +184,10 @@ export declare function renderCorpusQualityMetricsText(report: CorpusQualityMetr
     outputPath?: string | null;
 }): string;
 export declare function renderCorpusQualityMetricsMarkdown(report: CorpusQualityMetricsReport): string;
+export declare function buildCorpusQualityMetricsDiffReport(beforePath: string, afterPath: string, options?: BuildCorpusMetricsDiffOptions): Promise<CorpusQualityMetricsDiffReport>;
+export declare function renderCorpusQualityMetricsDiffJson(report: CorpusQualityMetricsDiffReport): string;
+export declare function renderCorpusQualityMetricsDiffText(report: CorpusQualityMetricsDiffReport, options?: {
+    outputPath?: string | null;
+}): string;
+export declare function renderCorpusQualityMetricsDiffMarkdown(report: CorpusQualityMetricsDiffReport): string;
 export {};

package/dist/core/corpus-quality-metrics.js

@@ -1,3 +1,4 @@
+import { createHash } from "node:crypto";
 import { stat } from "node:fs/promises";
 import path from "node:path";
 import { buildGenericMcpDoctor } from "../mcp/generic-mcp-doctor.js";
@@ -11,6 +12,12 @@ export class CorpusMetricsManifestError extends Error {
         this.name = "CorpusMetricsManifestError";
     }
 }
+export class CorpusMetricsDiffError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = "CorpusMetricsDiffError";
+    }
+}
 const manifestKeys = new Set(["schemaVersion", "thresholds", "targets"]);
 const thresholdKeys = new Set(["minPrecision", "minRecall", "maxFalsePositiveRate"]);
 const targetKeys = new Set([
@@ -206,6 +213,26 @@ function findingKey(findingId, fingerprint) {
 function roundMetric(value) {
     return value === null ? null : Math.round(value * 1_000_000) / 1_000_000;
 }
+function buildCorpusDigest(manifest) {
+    const identity = manifest.targets
+        .map((target) => ({
+        id: target.id,
+        profile: target.profile,
+        sourceType: target.sourceType,
+        mode: target.mode,
+        contentDigest: target.contentDigest,
+        reviews: target.reviews
+            .map((review) => ({
+            findingId: review.findingId,
+            fingerprint: review.fingerprint,
+            classification: review.classification
+        }))
+            .sort((left, right) => findingKey(left.findingId, left.fingerprint)
+            .localeCompare(findingKey(right.findingId, right.fingerprint)))
+    }))
+        .sort((left, right) => left.id.localeCompare(right.id));
+    return `sha256:${createHash("sha256").update(JSON.stringify(identity)).digest("hex")}`;
+}
 function calculateRawMetrics(counts) {
     const precisionDenominator = counts.truePositives + counts.falsePositives;
     const recallDenominator = counts.truePositives + counts.falseNegatives;
@@ -339,6 +366,7 @@ export async function buildCorpusQualityMetricsReport(manifestPath, options = {}
         kind: "doctor.validation.corpus.metrics",
         generatedAt: new Date().toISOString(),
         version: packageVersion,
+        corpusDigest: buildCorpusDigest(manifest),
         status,
         exitCode,
         summary: {
@@ -431,3 +459,264 @@ export function renderCorpusQualityMetricsMarkdown(report) {
     }
     return lines.join("\n");
 }
+const reportSizeLimit = 5 * 1024 * 1024;
+const countKeys = [
+    "truePositives",
+    "falsePositives",
+    "falseNegatives",
+    "resolvedFalsePositives",
+    "unreviewed",
+    "unclear"
+];
+function requireNonNegativeInteger(value, label) {
+    if (!Number.isInteger(value) || value < 0) {
+        throw new CorpusMetricsDiffError(`${label} must be a non-negative integer.`);
+    }
+    return value;
+}
+function parseCounts(value, label) {
+    if (!isRecord(value))
+        throw new CorpusMetricsDiffError(`${label} is invalid.`);
+    return Object.fromEntries(countKeys.map((key) => [
+        key,
+        requireNonNegativeInteger(value[key], `${label}.${key}`)
+    ]));
+}
+function metricsMatchCounts(value, counts) {
+    if (!isRecord(value))
+        return false;
+    const expected = calculateMetrics(counts);
+    return value.precision === expected.precision &&
+        value.recall === expected.recall &&
+        value.falsePositiveRate === expected.falsePositiveRate;
+}
+async function loadComparableMetricsReport(reportPath, label) {
+    let fileStat;
+    try {
+        fileStat = await stat(path.resolve(reportPath));
+    }
+    catch {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report could not be read.`);
+    }
+    if (!fileStat.isFile() || fileStat.size > reportSizeLimit) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report must be a file no larger than 5 MiB.`);
+    }
+    let value;
+    try {
+        value = await readJsonFile(path.resolve(reportPath));
+    }
+    catch {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report could not be read as JSON.`);
+    }
+    if (!isRecord(value) || value.schemaVersion !== "1.0.0" ||
+        value.kind !== "doctor.validation.corpus.metrics") {
+        throw new CorpusMetricsDiffError(`${label} file is not a supported corpus metrics report.`);
+    }
+    if (typeof value.version !== "string" || value.version.length === 0) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report version is invalid.`);
+    }
+    if (typeof value.corpusDigest !== "string" || !sha256Digest.test(value.corpusDigest)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report is missing a valid corpusDigest.`);
+    }
+    if (!isRecord(value.summary) || !Array.isArray(value.targets)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report structure is invalid.`);
+    }
+    if (value.status === "incomplete" || value.summary.incompleteTargets !== 0 ||
+        value.summary.unreviewed !== 0 || value.summary.unclear !== 0) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report must be complete.`);
+    }
+    if ((value.status !== "pass" && value.status !== "fail") ||
+        (value.status === "pass" ? value.exitCode !== 0 : value.exitCode !== 1)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report status is invalid.`);
+    }
+    const targetCount = requireNonNegativeInteger(value.summary.targetCount, `${label} summary.targetCount`);
+    const summaryCounts = parseCounts(value.summary, `${label} summary`);
+    if (targetCount === 0 || targetCount !== value.targets.length ||
+        value.summary.completeTargets !== targetCount || !metricsMatchCounts(value.summary, summaryCounts)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report is internally inconsistent.`);
+    }
+    const ids = new Set();
+    const targets = value.targets.map((target, index) => {
+        if (!isRecord(target) || typeof target.id !== "string" || !publicSafeId.test(target.id) ||
+            target.complete !== true || target.digestMatched !== true) {
+            throw new CorpusMetricsDiffError(`${label} corpus metrics target ${index} is invalid or incomplete.`);
+        }
+        if (ids.has(target.id)) {
+            throw new CorpusMetricsDiffError(`${label} corpus metrics report contains duplicate target IDs.`);
+        }
+        ids.add(target.id);
+        const counts = parseCounts(target.counts, `${label} target ${target.id} counts`);
+        if (!metricsMatchCounts(target.metrics, counts)) {
+            throw new CorpusMetricsDiffError(`${label} corpus metrics report is internally inconsistent.`);
+        }
+        return { id: target.id, counts };
+    });
+    const summedCounts = targets.reduce((total, target) => addCounts(total, target.counts), {
+        truePositives: 0,
+        falsePositives: 0,
+        falseNegatives: 0,
+        resolvedFalsePositives: 0,
+        unreviewed: 0,
+        unclear: 0
+    });
+    if (countKeys.some((key) => summedCounts[key] !== summaryCounts[key])) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report is internally inconsistent.`);
+    }
+    return {
+        version: value.version,
+        corpusDigest: value.corpusDigest,
+        summary: { ...summaryCounts, targetCount },
+        targets
+    };
+}
+function subtractCounts(after, before) {
+    return Object.fromEntries(countKeys.map((key) => [key, after[key] - before[key]]));
+}
+function compareMetric(before, after) {
+    return {
+        before: roundMetric(before),
+        after: roundMetric(after),
+        delta: before === null || after === null ? null : roundMetric(after - before)
+    };
+}
+function metricRegressed(precision, recall, falsePositiveRate) {
+    return (precision.delta !== null && precision.delta < 0) ||
+        (recall.delta !== null && recall.delta < 0) ||
+        (falsePositiveRate.delta !== null && falsePositiveRate.delta > 0);
+}
+export async function buildCorpusQualityMetricsDiffReport(beforePath, afterPath, options = {}) {
+    const [before, after] = await Promise.all([
+        loadComparableMetricsReport(beforePath, "Before"),
+        loadComparableMetricsReport(afterPath, "After")
+    ]);
+    const beforeIds = before.targets.map((target) => target.id).sort();
+    const afterIds = after.targets.map((target) => target.id).sort();
+    if (before.corpusDigest !== after.corpusDigest || JSON.stringify(beforeIds) !== JSON.stringify(afterIds)) {
+        throw new CorpusMetricsDiffError("Corpus metrics reports describe different corpus identities.");
+    }
+    const afterTargets = new Map(after.targets.map((target) => [target.id, target]));
+    const targets = before.targets.map((beforeTarget) => {
+        const afterTarget = afterTargets.get(beforeTarget.id);
+        const beforeMetrics = calculateRawMetrics(beforeTarget.counts);
+        const afterMetrics = calculateRawMetrics(afterTarget.counts);
+        const precision = compareMetric(beforeMetrics.precision, afterMetrics.precision);
+        const recall = compareMetric(beforeMetrics.recall, afterMetrics.recall);
+        const falsePositiveRate = compareMetric(beforeMetrics.falsePositiveRate, afterMetrics.falsePositiveRate);
+        return {
+            id: beforeTarget.id,
+            regressed: metricRegressed(precision, recall, falsePositiveRate),
+            counts: subtractCounts(afterTarget.counts, beforeTarget.counts),
+            precision,
+            recall,
+            falsePositiveRate
+        };
+    });
+    const changedTargets = targets.filter((target) => countKeys.some((key) => target.counts[key] !== 0)).length;
+    const beforeMetrics = calculateRawMetrics(before.summary);
+    const afterMetrics = calculateRawMetrics(after.summary);
+    const precision = compareMetric(beforeMetrics.precision, afterMetrics.precision);
+    const recall = compareMetric(beforeMetrics.recall, afterMetrics.recall);
+    const falsePositiveRate = compareMetric(beforeMetrics.falsePositiveRate, afterMetrics.falsePositiveRate);
+    const regression = metricRegressed(precision, recall, falsePositiveRate);
+    const failOnRegression = options.failOnRegression === true;
+    const failed = failOnRegression && regression;
+    return {
+        schemaVersion: "1.0.0",
+        kind: "doctor.validation.corpus.metrics.diff",
+        generatedAt: new Date().toISOString(),
+        version: packageVersion,
+        corpusDigest: before.corpusDigest,
+        status: failed ? "fail" : "pass",
+        exitCode: failed ? 1 : 0,
+        failOnRegression,
+        summary: {
+            comparable: true,
+            regression,
+            changedTargets,
+            precisionBefore: precision.before,
+            precisionAfter: precision.after,
+            precisionDelta: precision.delta,
+            recallBefore: recall.before,
+            recallAfter: recall.after,
+            recallDelta: recall.delta,
+            falsePositiveRateBefore: falsePositiveRate.before,
+            falsePositiveRateAfter: falsePositiveRate.after,
+            falsePositiveRateDelta: falsePositiveRate.delta,
+            counts: subtractCounts(after.summary, before.summary)
+        },
+        before: { version: before.version, targetCount: before.summary.targetCount },
+        after: { version: after.version, targetCount: after.summary.targetCount },
+        targets
+    };
+}
+function formatDelta(value) {
+    if (value === null)
+        return "N/A";
+    const sign = value > 0 ? "+" : "";
+    return `${sign}${(value * 100).toFixed(2)} pp`;
+}
+export function renderCorpusQualityMetricsDiffJson(report) {
+    return JSON.stringify(report, null, 2);
+}
+export function renderCorpusQualityMetricsDiffText(report, options = {}) {
+    const lines = [
+        "Doctor Corpus Metrics Diff",
+        "==========================",
+        `Status: ${report.status.toUpperCase()}`,
+        `Before version: ${report.before.version}`,
+        `After version: ${report.after.version}`,
+        `Regression: ${report.summary.regression ? "yes" : "no"}`,
+        `Precision delta: ${formatDelta(report.summary.precisionDelta)}`,
+        `Recall delta: ${formatDelta(report.summary.recallDelta)}`,
+        `False-positive rate delta: ${formatDelta(report.summary.falsePositiveRateDelta)}`,
+        `Changed targets: ${report.summary.changedTargets}`
+    ];
+    if (options.outputPath)
+        lines.push(`Output: ${options.outputPath}`);
+    lines.push("", "Target Changes", "--------------");
+    const changed = report.targets.filter((target) => countKeys.some((key) => target.counts[key] !== 0));
+    if (changed.length === 0) {
+        lines.push("None.");
+    }
+    else {
+        for (const target of changed) {
+            lines.push(`${target.id}: ${target.regressed ? "REGRESSED" : "CHANGED"} ` +
+                `(precision ${formatDelta(target.precision.delta)}, recall ${formatDelta(target.recall.delta)}, ` +
+                `false-positive rate ${formatDelta(target.falsePositiveRate.delta)})`);
+        }
+    }
+    return lines.join("\n");
+}
+export function renderCorpusQualityMetricsDiffMarkdown(report) {
+    const lines = [
+        "# Doctor Corpus Metrics Diff",
+        "",
+        `- Status: ${report.status.toUpperCase()}`,
+        `- Before version: ${report.before.version}`,
+        `- After version: ${report.after.version}`,
+        `- Regression: ${report.summary.regression ? "yes" : "no"}`,
+        "",
+        "| Metric | Before | After | Delta |",
+        "| --- | ---: | ---: | ---: |",
+        `| Precision | ${formatMetric(report.summary.precisionBefore)} | ${formatMetric(report.summary.precisionAfter)} | ${formatDelta(report.summary.precisionDelta)} |`,
+        `| Recall | ${formatMetric(report.summary.recallBefore)} | ${formatMetric(report.summary.recallAfter)} | ${formatDelta(report.summary.recallDelta)} |`,
+        `| False-positive rate | ${formatMetric(report.summary.falsePositiveRateBefore)} | ${formatMetric(report.summary.falsePositiveRateAfter)} | ${formatDelta(report.summary.falsePositiveRateDelta)} |`,
+        "",
+        "## Target Changes",
+        "",
+        "| Target | Result | Precision | Recall | False-positive rate |",
+        "| --- | --- | ---: | ---: | ---: |"
+    ];
+    const changed = report.targets.filter((target) => countKeys.some((key) => target.counts[key] !== 0));
+    if (changed.length === 0) {
+        lines.push("| None | unchanged | 0.00 pp | 0.00 pp | 0.00 pp |");
+    }
+    else {
+        for (const target of changed) {
+            lines.push(`| ${target.id} | ${target.regressed ? "regressed" : "changed"} | ` +
+                `${formatDelta(target.precision.delta)} | ${formatDelta(target.recall.delta)} | ` +
+                `${formatDelta(target.falsePositiveRate.delta)} |`);
+        }
+    }
+    return lines.join("\n");
+}

package/dist/core/output-contract.js

@@ -110,6 +110,7 @@ const publicSchemaDefinitions = [
             "kind",
             "generatedAt",
             "version",
+            "corpusDigest",
             "status",
             "exitCode",
             "summary",
@@ -118,6 +119,25 @@ const publicSchemaDefinitions = [
             "targets"
         ]
     },
+    {
+        id: "doctor.validation.corpus.metrics.diff.json",
+        command: "codex-plugin-doctor doctor corpus metrics diff --before <metrics.json> --after <metrics.json> --json",
+        outputKind: "doctor.validation.corpus.metrics.diff",
+        required: [
+            "schemaVersion",
+            "kind",
+            "generatedAt",
+            "version",
+            "corpusDigest",
+            "status",
+            "exitCode",
+            "failOnRegression",
+            "summary",
+            "before",
+            "after",
+            "targets"
+        ]
+    },
     {
         id: "doctor.recommendations.json",
         command: "codex-plugin-doctor doctor recommend <path> --json",

package/dist/index.d.ts

@@ -11,7 +11,7 @@ export { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 export { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson, type DoctorExportBundle } from "./core/doctor-export-bundle.js";
 export { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson, renderDoctorAttestationVerification, renderDoctorAttestationVerificationJson, verifyDoctorAttestation, type DoctorAttestation, type DoctorAttestationVerificationCheck, type DoctorAttestationVerificationReport, type Digest, type PackageFingerprint } from "./core/attestation.js";
 export { buildExternalValidationCorpusReport, ExternalCorpusManifestError, loadExternalCorpusManifest, type BuildExternalValidationCorpusOptions, type ExternalCorpusCaseResult, type ExternalCorpusClassificationCounts, type ExternalCorpusFindingReview, type ExternalCorpusManifest, type ExternalCorpusMode, type ExternalCorpusProfile, type ExternalCorpusSourceType, type ExternalValidationCorpusReport, type FindingReviewClassification } from "./core/external-validation-corpus.js";
-export { buildCorpusQualityMetricsReport, CorpusMetricsManifestError, loadCorpusMetricsManifest, reconcileCorpusMetricFindings, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText, type BuildCorpusMetricsOptions, type CorpusMetricCounts, type CorpusMetricFinding, type CorpusMetricFindingReview, type CorpusMetricReconciliation, type CorpusMetricsManifest, type CorpusMetricSource, type CorpusMetricThresholdCheck, type CorpusMetricThresholds, type CorpusMetricValues, type CorpusQualityMetricsReport, type CorpusQualityMetricsTargetResult } from "./core/corpus-quality-metrics.js";
+export { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport, CorpusMetricsDiffError, CorpusMetricsManifestError, loadCorpusMetricsManifest, reconcileCorpusMetricFindings, renderCorpusQualityMetricsDiffJson, renderCorpusQualityMetricsDiffMarkdown, renderCorpusQualityMetricsDiffText, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText, type BuildCorpusMetricsDiffOptions, type BuildCorpusMetricsOptions, type CorpusMetricCounts, type CorpusMetricCountDeltas, type CorpusMetricFinding, type CorpusMetricFindingReview, type CorpusMetricReconciliation, type CorpusMetricsManifest, type CorpusMetricSource, type CorpusMetricThresholdCheck, type CorpusMetricThresholds, type CorpusMetricValues, type CorpusMetricValueComparison, type CorpusQualityMetricsDiffReport, type CorpusQualityMetricsDiffTarget, type CorpusQualityMetricsReport, type CorpusQualityMetricsTargetResult } from "./core/corpus-quality-metrics.js";
 export { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson, type DoctorOutputContract, type OutputContractRule, type OutputContractSchema } from "./core/output-contract.js";
 export { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport, type BuildDoctorValidationCorpusOptions, type DoctorValidationCorpusReport, type ValidationCorpusCaseDefinition, type ValidationCorpusCaseResult } from "./core/validation-corpus.js";
 export { buildDoctorExportBundleFromAnalysis, buildDoctorRecommendationsFromAnalysis, buildPackageAnalysis, type PackageAnalysis, type PackageAnalysisOptions, type PackageAnalysisStage, type PackageAnalysisTiming } from "./core/package-analysis.js";

package/dist/index.js

@@ -11,7 +11,7 @@ export { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 export { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson } from "./core/doctor-export-bundle.js";
 export { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson, renderDoctorAttestationVerification, renderDoctorAttestationVerificationJson, verifyDoctorAttestation } from "./core/attestation.js";
 export { buildExternalValidationCorpusReport, ExternalCorpusManifestError, loadExternalCorpusManifest } from "./core/external-validation-corpus.js";
-export { buildCorpusQualityMetricsReport, CorpusMetricsManifestError, loadCorpusMetricsManifest, reconcileCorpusMetricFindings, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText } from "./core/corpus-quality-metrics.js";
+export { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport, CorpusMetricsDiffError, CorpusMetricsManifestError, loadCorpusMetricsManifest, reconcileCorpusMetricFindings, renderCorpusQualityMetricsDiffJson, renderCorpusQualityMetricsDiffMarkdown, renderCorpusQualityMetricsDiffText, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText } from "./core/corpus-quality-metrics.js";
 export { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson } from "./core/output-contract.js";
 export { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport } from "./core/validation-corpus.js";
 export { buildDoctorExportBundleFromAnalysis, buildDoctorRecommendationsFromAnalysis, buildPackageAnalysis } from "./core/package-analysis.js";

package/dist/run-cli.d.ts

@@ -1,4 +1,4 @@
-import { buildCorpusQualityMetricsReport } from "./core/corpus-quality-metrics.js";
+import { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport } from "./core/corpus-quality-metrics.js";
 import { runCheck } from "./index.js";
 import { writeRawDoctorConfig } from "./index.js";
 export interface CliIo {
@@ -19,6 +19,7 @@ export interface RunCliOptions {
     now?: () => Date;
     releaseAssetUploadImpl?: (args: string[]) => Promise<void>;
     resolveLatestVersion?: () => Promise<string>;
+    buildCorpusMetricsDiffReportImpl?: typeof buildCorpusQualityMetricsDiffReport;
     buildCorpusMetricsReportImpl?: typeof buildCorpusQualityMetricsReport;
 }
 export declare function runCli(args: string[], io?: CliIo, options?: RunCliOptions): Promise<number>;

package/dist/run-cli.js

@@ -21,7 +21,7 @@ import { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestatio
 import { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson } from "./core/output-contract.js";
 import { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport } from "./core/validation-corpus.js";
 import { buildExternalValidationCorpusReport, ExternalCorpusManifestError } from "./core/external-validation-corpus.js";
-import { buildCorpusQualityMetricsReport, CorpusMetricsManifestError, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText } from "./core/corpus-quality-metrics.js";
+import { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport, CorpusMetricsDiffError, CorpusMetricsManifestError, renderCorpusQualityMetricsDiffJson, renderCorpusQualityMetricsDiffMarkdown, renderCorpusQualityMetricsDiffText, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText } from "./core/corpus-quality-metrics.js";
 import { buildDoctorPerformanceReport, renderDoctorPerformanceReport, renderDoctorPerformanceReportJson } from "./core/performance-report.js";
 import { buildDoctorRuntimePlan, evaluateRuntimeApproval, renderDoctorRuntimePlan, renderDoctorRuntimePlanMarkdown, renderDoctorRuntimePlanJson, runtimeApprovalPassed } from "./core/runtime-plan.js";
 import { buildDoctorRuntimePolicyReport, renderDoctorRuntimePolicy, renderDoctorRuntimePolicyJson } from "./core/runtime-policy.js";
@@ -101,6 +101,7 @@ function parseRuntimeSandbox(flags, options = { requireRuntime: true }) {
 }
 function printUsage(io) {
     io.writeStderr("Usage: codex-plugin-doctor check <path|--installed> [filter] [--policy codex-publish|mcp-strict|security] [--compat] [--json|--markdown|--badge-json|--badge-markdown] [--output <path>] [--history <path>] [--runtime] [--sandbox docker] [--require-runtime-approval --runtime-approval-digest <digest>] [--verbose-runtime] [--explain] [--no-animations] [--ascii] [--changed-since <ref>]\n       codex-plugin-doctor audit --installed [filter] [--policy codex-publish|mcp-strict|security] [--security] [--compat] [--json] [--output <path>] [--cache] [--changed]\n       codex-plugin-doctor audit deps <path> [--policy codex-publish|mcp-strict|security] [--recommend] [--json|--sarif] [--output <path>]\n       codex-plugin-doctor mcp <path> [--json] [--output <path>]\n       codex-plugin-doctor security <path> [--policy security] [--json|--scorecard]\n       codex-plugin-doctor compat <path> [--all|--client <client>] [--json] [--scorecard] [--output <path>] [--install-preview|--apply --backup]\n       codex-plugin-doctor suppress add <path> [--fingerprint <sha256> --reason <text> --expires-at YYYY-MM-DD] [--config <path>] [--json]\n       codex-plugin-doctor suppress list <path> [--config <path>] [--json]\n       codex-plugin-doctor suppress remove <path> [--fingerprint <sha256>|--index <n>] [--config <path>] [--json]\n       codex-plugin-doctor fix <path> (--dry-run|--interactive --backup|--apply --backup)\n       codex-plugin-doctor history <history.jsonl> [--json] [--fail-on-regression]\n       codex-plugin-doctor watch <path> [--runtime] [--json] [--output <path>] [--debounce-ms <ms>] [--max-iterations <n>] [--fail-fast] [--accumulate-json <path>]\n       codex-plugin-doctor doctor [npm <package>|contract|corpus [--manifest <corpus.json>] [--json] [--output <path>]|corpus metrics --manifest <corpus.json> [--json|--markdown] [--output <path>] [--min-precision <0..1>] [--min-recall <0..1>] [--max-false-positive-rate <0..1>]|runtime-plan <path> [--runtime --sandbox docker] [--json|--markdown] [--output <path>]|runtime-policy <path> [--runtime --sandbox docker] [--json] [--output <path>]|review-bundle <path> --output <dir> --sign-key-env NAME [--json] [--allow-dirty] [--allow-untagged]|review-bundle verify <bundle-dir> --target <path> --sign-key-env NAME [--json] [--output <path>] [--failures-only]|review-bundle diff --before <dir> --after <dir> [--json]|attest <path> [--sign-key-env NAME]|attest verify <attestation.json> --target <path> --sign-key-env NAME|release-evidence <path> --sign-key-env NAME [--runtime --sandbox docker] [--allow-dirty] [--allow-untagged] [--require-runtime-approval --runtime-approval-digest <digest>]|release-evidence verify <evidence.json> --target <path> --sign-key-env NAME|release-evidence asset <path> --tag <tag> --output <evidence.json> --sign-key-env NAME [--upload]|mcp <path>|inspector <path>|diff --before <path> --after <path>|recommend <path>|trust <path>|perf <path> [--max-total-ms <ms>] [--max-stage-ms stage=ms]|export --bundle <path>|snapshot|clients|--json|--update-check]\n       codex-plugin-doctor init [path] [--template skill-only|mcp-stdio|mcp-http|full-runtime]\n       codex-plugin-doctor init-ci [path]\n       codex-plugin-doctor init-git-hooks [path] [--force] [--json]\n       codex-plugin-doctor init-git-hooks [path] --remove [--json]\n       codex-plugin-doctor completion bash|zsh|fish\n       codex-plugin-doctor config validate <path> [--json]\n       codex-plugin-doctor release check <path> [--json] [--runtime --sandbox docker]\n       codex-plugin-doctor self-test\n       codex-plugin-doctor list --installed\n       codex-plugin-doctor explain <finding-id>\n       codex-plugin-doctor --version\n\nFirst run:\n       codex-plugin-doctor doctor\n       codex-plugin-doctor self-test\n       codex-plugin-doctor init my-plugin\n       codex-plugin-doctor check . --runtime --explain");
+    io.writeStderr("Corpus quality regression: codex-plugin-doctor doctor corpus metrics diff --before <metrics.json> --after <metrics.json> [--fail-on-regression] [--json|--markdown] [--output <path>]");
     io.writeStderr("Suppression governance: codex-plugin-doctor suppress list <path> [--fail-on-expired] [--fail-on-invalid] [--warn-expiring-within-days <days>]\n       codex-plugin-doctor suppress prune <path> [--apply] [--json]");
     io.writeStderr("Baseline gating: codex-plugin-doctor baseline create <path> --output <path> [--runtime]\n       codex-plugin-doctor check <path> --baseline <path>");
 }
@@ -1419,6 +1420,65 @@ export async function runCli(args, io = defaultIo, options = {}) {
         if (maybePath === "corpus") {
             if (remainingArgs[0] === "metrics") {
                 const metricsArgs = remainingArgs.slice(1);
+                if (metricsArgs[0] === "diff") {
+                    const diffArgs = metricsArgs.slice(1);
+                    const jsonOutput = diffArgs.includes("--json");
+                    const markdownOutput = diffArgs.includes("--markdown");
+                    const failOnRegression = diffArgs.includes("--fail-on-regression");
+                    const valueOptions = new Map();
+                    const valueFlags = new Set(["--before", "--after", "--output"]);
+                    const booleanFlags = new Set(["--json", "--markdown", "--fail-on-regression"]);
+                    for (let index = 0; index < diffArgs.length; index += 1) {
+                        const argument = diffArgs[index];
+                        if (booleanFlags.has(argument))
+                            continue;
+                        if (!valueFlags.has(argument)) {
+                            io.writeStderr(`Unknown corpus metrics diff argument: ${argument}.`);
+                            return 2;
+                        }
+                        const value = diffArgs[index + 1];
+                        if (!value || value.startsWith("--")) {
+                            io.writeStderr(`Missing path after ${argument}.`);
+                            return 2;
+                        }
+                        valueOptions.set(argument, value);
+                        index += 1;
+                    }
+                    const beforePath = valueOptions.get("--before");
+                    const afterPath = valueOptions.get("--after");
+                    const outputPath = valueOptions.get("--output") ?? null;
+                    if (!beforePath || !afterPath) {
+                        io.writeStderr("Corpus metrics diff requires --before <metrics.json> and --after <metrics.json>.");
+                        return 2;
+                    }
+                    if (jsonOutput && markdownOutput) {
+                        io.writeStderr("Use either --json or --markdown, not both.");
+                        return 2;
+                    }
+                    let report;
+                    try {
+                        const builder = options.buildCorpusMetricsDiffReportImpl ?? buildCorpusQualityMetricsDiffReport;
+                        report = await builder(beforePath, afterPath, { failOnRegression });
+                    }
+                    catch (error) {
+                        if (error instanceof CorpusMetricsDiffError) {
+                            io.writeStderr(error.message);
+                            return 2;
+                        }
+                        const message = error instanceof Error ? error.message : "Unknown error";
+                        io.writeStderr(`Corpus metrics diff failed: ${message}`);
+                        return 2;
+                    }
+                    const rendered = jsonOutput
+                        ? renderCorpusQualityMetricsDiffJson(report)
+                        : markdownOutput
+                            ? renderCorpusQualityMetricsDiffMarkdown(report)
+                            : renderCorpusQualityMetricsDiffText(report, { outputPath });
+                    if (outputPath)
+                        await writeFile(outputPath, rendered, "utf8");
+                    io.writeStdout(rendered);
+                    return report.exitCode;
+                }
                 const jsonOutput = metricsArgs.includes("--json");
                 const markdownOutput = metricsArgs.includes("--markdown");
                 const valueOptions = new Map();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "codex-plugin-doctor",
-  "version": "1.48.0",
+  "version": "1.49.0",
   "description": "CLI-first validator for Codex plugins, skills, and MCP package surfaces with runtime MCP protocol validation.",
   "type": "module",
   "main": "./dist/index.js",