codex-plugin-doctor 1.47.0 → 1.49.0

package/README.md

@@ -197,6 +197,8 @@ codex-plugin-doctor doctor contract --json --output output-contract.json
 codex-plugin-doctor doctor corpus
 codex-plugin-doctor doctor corpus --json --output validation-corpus.json
 codex-plugin-doctor doctor corpus --manifest ../private-corpus/corpus.json --json
+codex-plugin-doctor doctor corpus metrics --manifest ../private-corpus/metrics.json --markdown
+codex-plugin-doctor doctor corpus metrics --manifest ../private-corpus/metrics.json --json --min-precision 0.90 --min-recall 0.85 --max-false-positive-rate 0.10
 codex-plugin-doctor doctor npm <published-plugin-package>
 codex-plugin-doctor doctor npm <published-plugin-package> --json --output npm-preinstall.json
 codex-plugin-doctor doctor attest .
@@ -315,6 +317,10 @@ Docker runtime mode supports local Node.js stdio MCP servers. It uses a digest-p
 
 External corpus evaluation is static and offline. `doctor corpus --manifest <corpus.json>` validates anonymized local snapshots using stable content digests and exact finding fingerprints without downloading dependencies or starting target servers. Reports omit manifest and package paths and fail on digest drift, missing expected findings, present disputed findings, or unreviewed findings. See [Real-World Validation](./docs/guides/real-world-validation.md).
 
+`doctor corpus metrics --manifest <corpus.json>` measures reviewed true positives, false positives, and false negatives as precision, recall, and emitted-finding false-positive share. Optional thresholds make the measurement release-gating; incomplete reviews and digest drift return exit `2` instead of producing a favorable score. The dedicated metrics manifest stays offline, requires contained snapshot paths, and may record immutable HTTPS Git provenance without fetching it.
+
+`doctor corpus metrics diff --before <metrics.json> --after <metrics.json>` compares two complete metrics reports from the same corpus identity. Add `--fail-on-regression` to fail CI when precision or recall decreases, or when the reviewed emitted-finding false-positive share increases. Reports with different `corpusDigest` values are rejected as non-comparable, and diff output never includes the input report paths.
+
 `self-test` runs the bundled runtime-complete sample through static validation, runtime MCP probes, and the compatibility scorecard. It is the fastest post-install check after `npm install -g codex-plugin-doctor`.
 
 `doctor` checks the local environment, including package version, platform, Node version, npm global prefix, Codex home, and Codex plugin cache visibility. The text output also includes recommended next commands for self-test, installed plugin discovery, runtime checks, compatibility scoring, and CI setup. `doctor contract` publishes the machine-readable output contract, including public JSON schema surfaces, stable-through-1.0 compatibility metadata, and a frozen rule catalog digest. Add `--json` for automation or `--output output-contract.json` to write the contract to disk. `doctor corpus` runs the bundled validation corpus against healthy runtime, risky security, starter skill, and generic MCP packages, then reports whether each case matched its expected outcome. Add `--json` for automation or `--output validation-corpus.json` to write the corpus report to disk. `doctor npm <package>` runs a preinstall scan by packing the npm package with scripts disabled, extracting the publish tarball, and running validation, security, trust, and recommendation checks against the shipped contents. The JSON report includes tarball metadata such as filename, path, integrity, shasum, sizes, file count, and extracted package root so automation can prove which archive was scanned. Use a published Codex plugin package as the target; scanning `codex-plugin-doctor` itself intentionally reports a missing plugin manifest because this CLI package is not a plugin package. Add `--json` for automation or `--output npm-preinstall.json` to write the report to disk. `doctor attest <path>` creates a local attestation with stable package/report digests, validation/security/compatibility/trust summary, and verification metadata. Add `--sign-key-env NAME` to attach a local HMAC-SHA256 signature without printing the secret, or `--json --output attestation.json` to write the artifact to disk. `doctor attest verify <attestation.json> --target <path> --sign-key-env NAME` recomputes the package fingerprint, report digest, and HMAC signature offline; verification intentionally treats `generatedAt`, `targetPath`, `verification`, and `signature.keyHint` as unsigned display metadata. `doctor runtime-plan <path>` creates a non-executing runtime plan that lists MCP server commands, safe probe methods, risk reasons, and a stable approval digest before any local server is started. Add `--markdown --output runtime-plan.md` to preserve a review-ready approval artifact with the execution boundary, checklist, servers, probes, and risk reasons. `doctor runtime-policy <path>` evaluates the same runtime plan and security signals, then recommends `allow`, `review`, `sandbox_recommended`, or `deny` before local MCP execution starts. `doctor review-bundle <path> --output <dir> --sign-key-env NAME` writes a signed review directory with runtime plan, runtime policy, attestation, release evidence, manifest, Markdown summary files, and SHA-256 file integrity digests. `doctor review-bundle verify <bundle-dir> --target <path> --sign-key-env NAME` verifies the bundle manifest, expected files, manifest integrity digests, runtime artifacts, signed attestation, and signed release evidence offline before a reviewer trusts the handoff. `doctor review-bundle diff --before <dir> --after <dir>` compares two review bundles and flags risk-increasing changes in status, runtime policy, release readiness, signatures, release evidence, and runtime plan digest. `check --runtime --require-runtime-approval --runtime-approval-digest <digest>` refuses to run runtime probes unless the current plan digest matches the approved digest. `doctor release-evidence <path> --sign-key-env NAME` creates one redacted release bundle with signed attestation, offline verification, corpus, performance, security, trust, package metadata, git release gates, and runtime approval status. Strict release evidence requires a clean tagged worktree; use `--allow-dirty` or `--allow-untagged` only for local rehearsal. `doctor release-evidence verify <evidence.json> --target <path> --sign-key-env NAME` verifies a shared release evidence artifact offline against an explicit package path; the artifact target path is treated as display metadata, not trusted input. `doctor release-evidence asset <path> --tag <tag> --output <evidence.json> --sign-key-env NAME` writes a signed release evidence file and prints the `gh release upload` command; add `--upload` to run the upload through GitHub CLI with `--clobber`. `doctor inspector <path>` builds a safe MCP Inspector launch command from a packaged `.mcp.json` file without starting the Inspector proxy automatically. Use `--server <name>` when the package contains multiple MCP server entries. `doctor diff --before <path> --after <path>` compares two package roots and reports new findings, resolved findings, trust score delta, and whether risk increased. `doctor recommend <path>` turns validation, security, and compatibility signals into a prioritized action plan with blocker, high, medium, and info actions. Add `--json` for automation or `--output recommendations.json` to write the report to disk. `doctor trust <path>` creates a local trust score from package lifecycle scripts, dependency specs, and MCP security findings. Use it before release when you want supply-chain risks summarized as one score. `doctor perf <path>` profiles the shared package analysis pipeline and reports per-stage durations for validation, config, security, compatibility, trust, recommendations, and total runtime. Add `--max-total-ms <ms>` or repeatable `--max-stage-ms stage=ms` to fail CI when a budget is exceeded. `doctor mcp <path>` exposes the generic MCP static health report under the doctor command family without starting local MCP servers. `doctor export --bundle <path>` creates a redacted operator handoff bundle that includes validation JSON, security scorecard data, compatibility matrix, recommendations, and trust score in one file. `doctor snapshot` creates a redacted diagnostics bundle with environment health, client config readiness, installed plugin metadata, and next commands. Add `--json` for machine-readable output or `--output doctor-snapshot.json` to write the bundle to disk. `doctor clients` reports local Codex, Claude Desktop, Cursor, Cline, and Windsurf config readiness. `doctor --update-check` compares the installed CLI version with the latest npm version and prints the upgrade command when a newer release is available.
@@ -421,9 +427,9 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v5
-      - uses: Esquetta/CodexPluginDoctor@v1.47.0
+      - uses: Esquetta/CodexPluginDoctor@v1.49.0
         with:
-          version: "1.47.0"
+          version: "1.49.0"
           path: .
           runtime: "true"
           policy: codex-publish

package/dist/core/corpus-quality-metrics.d.ts

@@ -0,0 +1,193 @@
+import type { ExternalCorpusMode, ExternalCorpusProfile, ExternalCorpusSourceType, FindingReviewClassification } from "./external-validation-corpus.js";
+export interface CorpusMetricThresholds {
+    minPrecision?: number;
+    minRecall?: number;
+    maxFalsePositiveRate?: number;
+}
+export interface CorpusMetricFindingReview {
+    findingId: string;
+    fingerprint: string;
+    classification: FindingReviewClassification;
+}
+export interface CorpusMetricSource {
+    repository: string;
+    revision: string;
+}
+export interface CorpusMetricTarget {
+    id: string;
+    profile: ExternalCorpusProfile;
+    sourceType: ExternalCorpusSourceType;
+    disclosure: "anonymized";
+    path: string;
+    mode: ExternalCorpusMode;
+    contentDigest: string;
+    source?: CorpusMetricSource;
+    reviews: CorpusMetricFindingReview[];
+}
+export interface CorpusMetricsManifest {
+    schemaVersion: "1.0.0";
+    thresholds?: CorpusMetricThresholds;
+    targets: CorpusMetricTarget[];
+}
+interface LoadedCorpusMetricTarget extends CorpusMetricTarget {
+    resolvedPath: string;
+}
+interface LoadedCorpusMetricsManifest extends CorpusMetricsManifest {
+    targets: LoadedCorpusMetricTarget[];
+}
+export interface CorpusMetricFinding {
+    findingId: string;
+    fingerprint: string;
+}
+export interface CorpusMetricCounts {
+    truePositives: number;
+    falsePositives: number;
+    falseNegatives: number;
+    resolvedFalsePositives: number;
+    unreviewed: number;
+    unclear: number;
+}
+export interface CorpusMetricValues {
+    precision: number | null;
+    recall: number | null;
+    falsePositiveRate: number | null;
+}
+export interface CorpusMetricOutcome extends CorpusMetricFinding {
+    classification: FindingReviewClassification | "unreviewed";
+    emitted: boolean;
+}
+export interface CorpusMetricReconciliation {
+    counts: CorpusMetricCounts;
+    metrics: CorpusMetricValues;
+    complete: boolean;
+    outcomes: CorpusMetricOutcome[];
+}
+export interface CorpusMetricThresholdCheck {
+    metric: keyof CorpusMetricValues;
+    operator: ">=" | "<=";
+    threshold: number;
+    actual: number | null;
+    passed: boolean;
+}
+export interface CorpusQualityMetricsTargetResult {
+    id: string;
+    profile: ExternalCorpusProfile;
+    sourceType: ExternalCorpusSourceType;
+    disclosure: "anonymized";
+    mode: ExternalCorpusMode;
+    source?: CorpusMetricSource;
+    digestMatched: boolean;
+    counts: CorpusMetricCounts;
+    metrics: CorpusMetricValues;
+    complete: boolean;
+    outcomes: CorpusMetricOutcome[];
+}
+export interface CorpusQualityMetricsReport {
+    schemaVersion: "1.0.0";
+    kind: "doctor.validation.corpus.metrics";
+    generatedAt: string;
+    version: string;
+    corpusDigest: string;
+    status: "pass" | "fail" | "incomplete";
+    exitCode: 0 | 1 | 2;
+    summary: CorpusMetricCounts & CorpusMetricValues & {
+        targetCount: number;
+        completeTargets: number;
+        incompleteTargets: number;
+    };
+    thresholds: CorpusMetricThresholds;
+    thresholdChecks: CorpusMetricThresholdCheck[];
+    targets: CorpusQualityMetricsTargetResult[];
+}
+export interface CorpusMetricCountDeltas {
+    truePositives: number;
+    falsePositives: number;
+    falseNegatives: number;
+    resolvedFalsePositives: number;
+    unreviewed: number;
+    unclear: number;
+}
+export interface CorpusMetricValueComparison {
+    before: number | null;
+    after: number | null;
+    delta: number | null;
+}
+export interface CorpusQualityMetricsDiffTarget {
+    id: string;
+    regressed: boolean;
+    counts: CorpusMetricCountDeltas;
+    precision: CorpusMetricValueComparison;
+    recall: CorpusMetricValueComparison;
+    falsePositiveRate: CorpusMetricValueComparison;
+}
+export interface CorpusQualityMetricsDiffReport {
+    schemaVersion: "1.0.0";
+    kind: "doctor.validation.corpus.metrics.diff";
+    generatedAt: string;
+    version: string;
+    corpusDigest: string;
+    status: "pass" | "fail";
+    exitCode: 0 | 1;
+    failOnRegression: boolean;
+    summary: {
+        comparable: true;
+        regression: boolean;
+        changedTargets: number;
+        precisionBefore: number | null;
+        precisionAfter: number | null;
+        precisionDelta: number | null;
+        recallBefore: number | null;
+        recallAfter: number | null;
+        recallDelta: number | null;
+        falsePositiveRateBefore: number | null;
+        falsePositiveRateAfter: number | null;
+        falsePositiveRateDelta: number | null;
+        counts: CorpusMetricCountDeltas;
+    };
+    before: {
+        version: string;
+        targetCount: number;
+    };
+    after: {
+        version: string;
+        targetCount: number;
+    };
+    targets: CorpusQualityMetricsDiffTarget[];
+}
+export interface BuildCorpusMetricsDiffOptions {
+    failOnRegression?: boolean;
+}
+export interface BuildCorpusMetricsOptions {
+    thresholds?: CorpusMetricThresholds;
+    environment?: {
+        env?: NodeJS.ProcessEnv;
+        platform?: NodeJS.Platform;
+    };
+    analyzeTarget?: (targetPath: string, mode: ExternalCorpusMode) => Promise<{
+        findings: CorpusMetricFinding[];
+    }>;
+    buildFingerprint?: (targetPath: string) => Promise<{
+        digest: string;
+    }>;
+}
+export declare class CorpusMetricsManifestError extends Error {
+    constructor(message: string);
+}
+export declare class CorpusMetricsDiffError extends Error {
+    constructor(message: string);
+}
+export declare function loadCorpusMetricsManifest(manifestPath: string): Promise<LoadedCorpusMetricsManifest>;
+export declare function reconcileCorpusMetricFindings(reviews: CorpusMetricFindingReview[], actualFindings: CorpusMetricFinding[]): CorpusMetricReconciliation;
+export declare function buildCorpusQualityMetricsReport(manifestPath: string, options?: BuildCorpusMetricsOptions): Promise<CorpusQualityMetricsReport>;
+export declare function renderCorpusQualityMetricsJson(report: CorpusQualityMetricsReport): string;
+export declare function renderCorpusQualityMetricsText(report: CorpusQualityMetricsReport, options?: {
+    outputPath?: string | null;
+}): string;
+export declare function renderCorpusQualityMetricsMarkdown(report: CorpusQualityMetricsReport): string;
+export declare function buildCorpusQualityMetricsDiffReport(beforePath: string, afterPath: string, options?: BuildCorpusMetricsDiffOptions): Promise<CorpusQualityMetricsDiffReport>;
+export declare function renderCorpusQualityMetricsDiffJson(report: CorpusQualityMetricsDiffReport): string;
+export declare function renderCorpusQualityMetricsDiffText(report: CorpusQualityMetricsDiffReport, options?: {
+    outputPath?: string | null;
+}): string;
+export declare function renderCorpusQualityMetricsDiffMarkdown(report: CorpusQualityMetricsDiffReport): string;
+export {};

package/dist/core/corpus-quality-metrics.js

@@ -0,0 +1,722 @@
+import { createHash } from "node:crypto";
+import { stat } from "node:fs/promises";
+import path from "node:path";
+import { buildGenericMcpDoctor } from "../mcp/generic-mcp-doctor.js";
+import { packageVersion } from "../version.js";
+import { buildPackageFingerprint } from "./attestation.js";
+import { readJsonFile } from "./read-json-file.js";
+import { validatePlugin } from "./validate-plugin.js";
+export class CorpusMetricsManifestError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = "CorpusMetricsManifestError";
+    }
+}
+export class CorpusMetricsDiffError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = "CorpusMetricsDiffError";
+    }
+}
+const manifestKeys = new Set(["schemaVersion", "thresholds", "targets"]);
+const thresholdKeys = new Set(["minPrecision", "minRecall", "maxFalsePositiveRate"]);
+const targetKeys = new Set([
+    "id", "profile", "sourceType", "disclosure", "path", "mode",
+    "contentDigest", "source", "reviews"
+]);
+const sourceKeys = new Set(["repository", "revision"]);
+const reviewKeys = new Set(["findingId", "fingerprint", "classification"]);
+const profiles = new Set(["healthy", "broken", "edge-case"]);
+const sourceTypes = new Set([
+    "public-package", "local-snapshot", "derived-fixture"
+]);
+const modes = new Set(["codex-plugin", "generic-mcp"]);
+const classifications = new Set([
+    "true_positive", "false_positive", "unclear"
+]);
+const sha256Digest = /^sha256:[a-f0-9]{64}$/;
+const fingerprintPattern = /^[a-f0-9]{64}$/;
+const immutableRevision = /^[a-f0-9]{40}([a-f0-9]{24})?$/;
+const publicSafeId = /^[a-z0-9][a-z0-9._-]{0,63}$/;
+function isRecord(value) {
+    return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+function rejectUnknownKeys(value, allowed, label) {
+    const unknown = Object.keys(value).filter((key) => !allowed.has(key));
+    if (unknown.length > 0) {
+        throw new CorpusMetricsManifestError(`${label} contains unsupported field${unknown.length === 1 ? "" : "s"}: ${unknown.join(", ")}.`);
+    }
+}
+function requireString(value, label) {
+    if (typeof value !== "string" || value.length === 0) {
+        throw new CorpusMetricsManifestError(`${label} must be a non-empty string.`);
+    }
+    return value;
+}
+function parseThresholds(value) {
+    if (value === undefined)
+        return undefined;
+    if (!isRecord(value))
+        throw new CorpusMetricsManifestError("Metrics thresholds must be an object.");
+    rejectUnknownKeys(value, thresholdKeys, "Metrics thresholds");
+    const result = {};
+    for (const key of thresholdKeys) {
+        const candidate = value[key];
+        if (candidate === undefined)
+            continue;
+        if (typeof candidate !== "number" || !Number.isFinite(candidate) || candidate < 0 || candidate > 1) {
+            throw new CorpusMetricsManifestError(`Metrics threshold ${key} must be between 0 and 1.`);
+        }
+        result[key] = candidate;
+    }
+    return result;
+}
+function parseSource(value, targetId) {
+    if (value === undefined)
+        return undefined;
+    if (!isRecord(value))
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} source must be an object.`);
+    rejectUnknownKeys(value, sourceKeys, `Metrics target ${targetId} source`);
+    const repository = requireString(value.repository, `Metrics target ${targetId} source repository`);
+    const revision = requireString(value.revision, `Metrics target ${targetId} source revision`);
+    let parsed;
+    try {
+        parsed = new URL(repository);
+    }
+    catch {
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} source repository must use HTTPS.`);
+    }
+    if (parsed.protocol !== "https:" || parsed.username || parsed.password || parsed.search || parsed.hash) {
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} source repository must use HTTPS.`);
+    }
+    if (!immutableRevision.test(revision)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} source revision must be immutable.`);
+    }
+    return { repository, revision };
+}
+function parseReview(value, targetId, index) {
+    if (!isRecord(value))
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} review ${index} must be an object.`);
+    rejectUnknownKeys(value, reviewKeys, `Metrics target ${targetId} review ${index}`);
+    const findingId = requireString(value.findingId, `Metrics target ${targetId} review ${index} findingId`);
+    const fingerprint = requireString(value.fingerprint, `Metrics target ${targetId} review ${index} fingerprint`);
+    if (!fingerprintPattern.test(fingerprint)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} review ${index} fingerprint must be a sha256 fingerprint.`);
+    }
+    if (typeof value.classification !== "string" ||
+        !classifications.has(value.classification)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${targetId} review ${index} has an unsupported classification.`);
+    }
+    return { findingId, fingerprint, classification: value.classification };
+}
+function resolveContainedTarget(manifestDirectory, targetPath) {
+    if (path.posix.isAbsolute(targetPath) || path.win32.isAbsolute(targetPath)) {
+        throw new CorpusMetricsManifestError("Metrics target path must be relative to the manifest file.");
+    }
+    const root = path.resolve(manifestDirectory);
+    const resolved = path.resolve(root, targetPath);
+    const relative = path.relative(root, resolved);
+    if (relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) {
+        throw new CorpusMetricsManifestError("Metrics target path must remain beneath the manifest directory.");
+    }
+    return resolved;
+}
+async function parseTarget(value, manifestDirectory, index) {
+    if (!isRecord(value))
+        throw new CorpusMetricsManifestError(`Metrics target ${index} must be an object.`);
+    rejectUnknownKeys(value, targetKeys, `Metrics target ${index}`);
+    const id = requireString(value.id, `Metrics target ${index} id`);
+    if (!publicSafeId.test(id)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${index} id must be public-safe.`);
+    }
+    const targetPath = requireString(value.path, `Metrics target ${id} path`);
+    if (typeof value.profile !== "string" || !profiles.has(value.profile)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} has an unsupported profile.`);
+    }
+    if (typeof value.sourceType !== "string" || !sourceTypes.has(value.sourceType)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} has an unsupported sourceType.`);
+    }
+    if (value.disclosure !== "anonymized") {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} disclosure must be anonymized.`);
+    }
+    if (typeof value.mode !== "string" || !modes.has(value.mode)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} has an unsupported mode.`);
+    }
+    if (typeof value.contentDigest !== "string" || !sha256Digest.test(value.contentDigest)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} contentDigest must be a sha256 digest.`);
+    }
+    if (!Array.isArray(value.reviews)) {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} reviews must be an array.`);
+    }
+    const reviews = value.reviews.map((review, reviewIndex) => parseReview(review, id, reviewIndex));
+    const reviewKeysSeen = new Set();
+    for (const review of reviews) {
+        const key = findingKey(review.findingId, review.fingerprint);
+        if (reviewKeysSeen.has(key)) {
+            throw new CorpusMetricsManifestError(`Metrics target ${id} contains a duplicate finding review.`);
+        }
+        reviewKeysSeen.add(key);
+    }
+    const resolvedPath = resolveContainedTarget(manifestDirectory, targetPath);
+    try {
+        if (!(await stat(resolvedPath)).isDirectory())
+            throw new Error("not a directory");
+    }
+    catch {
+        throw new CorpusMetricsManifestError(`Metrics target ${id} directory does not exist.`);
+    }
+    const source = parseSource(value.source, id);
+    return {
+        id,
+        profile: value.profile,
+        sourceType: value.sourceType,
+        disclosure: "anonymized",
+        path: targetPath,
+        mode: value.mode,
+        contentDigest: value.contentDigest,
+        ...(source ? { source } : {}),
+        reviews,
+        resolvedPath
+    };
+}
+export async function loadCorpusMetricsManifest(manifestPath) {
+    let value;
+    try {
+        value = await readJsonFile(path.resolve(manifestPath));
+    }
+    catch {
+        throw new CorpusMetricsManifestError("Metrics manifest could not be read as JSON.");
+    }
+    if (!isRecord(value))
+        throw new CorpusMetricsManifestError("Metrics manifest must be an object.");
+    rejectUnknownKeys(value, manifestKeys, "Metrics manifest");
+    if (value.schemaVersion !== "1.0.0") {
+        throw new CorpusMetricsManifestError("Unsupported metrics manifest schemaVersion.");
+    }
+    if (!Array.isArray(value.targets) || value.targets.length === 0) {
+        throw new CorpusMetricsManifestError("Metrics manifest targets must be a non-empty array.");
+    }
+    const manifestDirectory = path.dirname(path.resolve(manifestPath));
+    const targets = await Promise.all(value.targets.map((target, index) => parseTarget(target, manifestDirectory, index)));
+    const ids = new Set();
+    for (const target of targets) {
+        if (ids.has(target.id))
+            throw new CorpusMetricsManifestError(`Metrics target id ${target.id} is duplicated.`);
+        ids.add(target.id);
+    }
+    const thresholds = parseThresholds(value.thresholds);
+    return { schemaVersion: "1.0.0", ...(thresholds ? { thresholds } : {}), targets };
+}
+function findingKey(findingId, fingerprint) {
+    return `${findingId}:${fingerprint}`;
+}
+function roundMetric(value) {
+    return value === null ? null : Math.round(value * 1_000_000) / 1_000_000;
+}
+function buildCorpusDigest(manifest) {
+    const identity = manifest.targets
+        .map((target) => ({
+        id: target.id,
+        profile: target.profile,
+        sourceType: target.sourceType,
+        mode: target.mode,
+        contentDigest: target.contentDigest,
+        reviews: target.reviews
+            .map((review) => ({
+            findingId: review.findingId,
+            fingerprint: review.fingerprint,
+            classification: review.classification
+        }))
+            .sort((left, right) => findingKey(left.findingId, left.fingerprint)
+            .localeCompare(findingKey(right.findingId, right.fingerprint)))
+    }))
+        .sort((left, right) => left.id.localeCompare(right.id));
+    return `sha256:${createHash("sha256").update(JSON.stringify(identity)).digest("hex")}`;
+}
+function calculateRawMetrics(counts) {
+    const precisionDenominator = counts.truePositives + counts.falsePositives;
+    const recallDenominator = counts.truePositives + counts.falseNegatives;
+    return {
+        precision: precisionDenominator === 0 ? null : counts.truePositives / precisionDenominator,
+        recall: recallDenominator === 0 ? null : counts.truePositives / recallDenominator,
+        falsePositiveRate: precisionDenominator === 0 ? null : counts.falsePositives / precisionDenominator
+    };
+}
+function calculateMetrics(counts) {
+    const raw = calculateRawMetrics(counts);
+    return {
+        precision: roundMetric(raw.precision),
+        recall: roundMetric(raw.recall),
+        falsePositiveRate: roundMetric(raw.falsePositiveRate)
+    };
+}
+export function reconcileCorpusMetricFindings(reviews, actualFindings) {
+    const reviewsByKey = new Map(reviews.map((review) => [findingKey(review.findingId, review.fingerprint), review]));
+    const actualKeys = new Set(actualFindings.map((finding) => findingKey(finding.findingId, finding.fingerprint)));
+    const outcomes = actualFindings.map((finding) => ({
+        ...finding,
+        classification: reviewsByKey.get(findingKey(finding.findingId, finding.fingerprint))?.classification ?? "unreviewed",
+        emitted: true
+    }));
+    for (const review of reviews) {
+        if (!actualKeys.has(findingKey(review.findingId, review.fingerprint))) {
+            outcomes.push({ ...review, emitted: false });
+        }
+    }
+    outcomes.sort((left, right) => findingKey(left.findingId, left.fingerprint)
+        .localeCompare(findingKey(right.findingId, right.fingerprint)));
+    const counts = {
+        truePositives: outcomes.filter((item) => item.emitted && item.classification === "true_positive").length,
+        falsePositives: outcomes.filter((item) => item.emitted && item.classification === "false_positive").length,
+        falseNegatives: outcomes.filter((item) => !item.emitted && item.classification === "true_positive").length,
+        resolvedFalsePositives: outcomes.filter((item) => !item.emitted && item.classification === "false_positive").length,
+        unreviewed: outcomes.filter((item) => item.emitted && item.classification === "unreviewed").length,
+        unclear: outcomes.filter((item) => item.classification === "unclear").length
+    };
+    return {
+        counts,
+        metrics: calculateMetrics(counts),
+        complete: counts.unreviewed === 0 && counts.unclear === 0,
+        outcomes
+    };
+}
+function addCounts(left, right) {
+    return {
+        truePositives: left.truePositives + right.truePositives,
+        falsePositives: left.falsePositives + right.falsePositives,
+        falseNegatives: left.falseNegatives + right.falseNegatives,
+        resolvedFalsePositives: left.resolvedFalsePositives + right.resolvedFalsePositives,
+        unreviewed: left.unreviewed + right.unreviewed,
+        unclear: left.unclear + right.unclear
+    };
+}
+async function defaultAnalyzeTarget(targetPath, mode, environment) {
+    const report = mode === "generic-mcp"
+        ? await buildGenericMcpDoctor(targetPath, environment)
+        : await validatePlugin(targetPath);
+    return {
+        findings: report.findings.map((finding) => {
+            if (!finding.fingerprint) {
+                throw new CorpusMetricsManifestError("Metrics analysis produced a finding without a fingerprint.");
+            }
+            return { findingId: finding.id, fingerprint: finding.fingerprint };
+        })
+    };
+}
+function buildThresholdChecks(metrics, thresholds) {
+    const definitions = [
+        ["minPrecision", "precision", ">="],
+        ["minRecall", "recall", ">="],
+        ["maxFalsePositiveRate", "falsePositiveRate", "<="]
+    ];
+    return definitions.flatMap(([thresholdKey, metric, operator]) => {
+        const threshold = thresholds[thresholdKey];
+        if (threshold === undefined)
+            return [];
+        const actual = metrics[metric];
+        const passed = actual !== null && (operator === ">=" ? actual >= threshold : actual <= threshold);
+        return [{ metric, operator, threshold, actual, passed }];
+    });
+}
+export async function buildCorpusQualityMetricsReport(manifestPath, options = {}) {
+    const manifest = await loadCorpusMetricsManifest(manifestPath);
+    const thresholds = { ...(manifest.thresholds ?? {}), ...(options.thresholds ?? {}) };
+    const analyzeTarget = options.analyzeTarget ?? ((targetPath, mode) => defaultAnalyzeTarget(targetPath, mode, options.environment));
+    const fingerprintBuilder = options.buildFingerprint ?? buildPackageFingerprint;
+    const targets = [];
+    for (const target of manifest.targets) {
+        const [analysis, fingerprint] = await Promise.all([
+            analyzeTarget(target.resolvedPath, target.mode),
+            fingerprintBuilder(target.resolvedPath)
+        ]);
+        const reconciliation = reconcileCorpusMetricFindings(target.reviews, analysis.findings);
+        const digestMatched = fingerprint.digest === target.contentDigest;
+        targets.push({
+            id: target.id,
+            profile: target.profile,
+            sourceType: target.sourceType,
+            disclosure: target.disclosure,
+            mode: target.mode,
+            ...(target.source ? { source: target.source } : {}),
+            digestMatched,
+            counts: reconciliation.counts,
+            metrics: reconciliation.metrics,
+            complete: reconciliation.complete && digestMatched,
+            outcomes: reconciliation.outcomes
+        });
+    }
+    const emptyCounts = {
+        truePositives: 0, falsePositives: 0, falseNegatives: 0,
+        resolvedFalsePositives: 0, unreviewed: 0, unclear: 0
+    };
+    const counts = targets.reduce((total, target) => addCounts(total, target.counts), emptyCounts);
+    const metrics = calculateMetrics(counts);
+    const rawMetrics = calculateRawMetrics(counts);
+    const completeTargets = targets.filter((target) => target.complete).length;
+    const thresholdChecks = buildThresholdChecks(rawMetrics, thresholds).map((check) => ({
+        ...check,
+        actual: metrics[check.metric]
+    }));
+    const incomplete = completeTargets !== targets.length;
+    const failedThreshold = thresholdChecks.some((check) => !check.passed);
+    const status = incomplete ? "incomplete" : failedThreshold ? "fail" : "pass";
+    const exitCode = incomplete ? 2 : failedThreshold ? 1 : 0;
+    return {
+        schemaVersion: "1.0.0",
+        kind: "doctor.validation.corpus.metrics",
+        generatedAt: new Date().toISOString(),
+        version: packageVersion,
+        corpusDigest: buildCorpusDigest(manifest),
+        status,
+        exitCode,
+        summary: {
+            targetCount: targets.length,
+            completeTargets,
+            incompleteTargets: targets.length - completeTargets,
+            ...counts,
+            ...metrics
+        },
+        thresholds,
+        thresholdChecks,
+        targets
+    };
+}
+function formatMetric(value) {
+    return value === null ? "N/A" : `${(value * 100).toFixed(2)}%`;
+}
+function metricLabel(metric) {
+    if (metric === "falsePositiveRate")
+        return "False-positive rate";
+    return metric[0].toUpperCase() + metric.slice(1);
+}
+export function renderCorpusQualityMetricsJson(report) {
+    return JSON.stringify(report, null, 2);
+}
+export function renderCorpusQualityMetricsText(report, options = {}) {
+    const lines = [
+        "Doctor Corpus Quality Metrics",
+        "=============================",
+        `Version: ${report.version}`,
+        `Status: ${report.status.toUpperCase()}`,
+        `Targets: ${report.summary.targetCount}`,
+        `Precision: ${formatMetric(report.summary.precision)}`,
+        `Recall: ${formatMetric(report.summary.recall)}`,
+        `False-positive rate: ${formatMetric(report.summary.falsePositiveRate)}`,
+        `TP: ${report.summary.truePositives}  FP: ${report.summary.falsePositives}  FN: ${report.summary.falseNegatives}`
+    ];
+    if (options.outputPath)
+        lines.push(`Output: ${options.outputPath}`);
+    const failed = report.thresholdChecks.filter((check) => !check.passed);
+    lines.push("", "Failed thresholds", "-----------------");
+    if (failed.length === 0) {
+        lines.push("None.");
+    }
+    else {
+        for (const check of failed) {
+            lines.push(`${metricLabel(check.metric)}: ${formatMetric(check.actual)} ${check.operator} ${formatMetric(check.threshold)} required`);
+        }
+    }
+    lines.push("", "Targets", "-------");
+    for (const target of report.targets) {
+        lines.push(`${target.id}: ${target.complete ? "COMPLETE" : "INCOMPLETE"} ` +
+            `(precision ${formatMetric(target.metrics.precision)}, recall ${formatMetric(target.metrics.recall)})`);
+    }
+    return lines.join("\n");
+}
+export function renderCorpusQualityMetricsMarkdown(report) {
+    const lines = [
+        "# Doctor Corpus Quality Metrics",
+        "",
+        `- Version: ${report.version}`,
+        `- Status: ${report.status.toUpperCase()}`,
+        `- Targets: ${report.summary.targetCount}`,
+        "",
+        "| Metric | Value |",
+        "| --- | ---: |",
+        `| Precision | ${formatMetric(report.summary.precision)} |`,
+        `| Recall | ${formatMetric(report.summary.recall)} |`,
+        `| False-positive rate | ${formatMetric(report.summary.falsePositiveRate)} |`,
+        `| True positives | ${report.summary.truePositives} |`,
+        `| False positives | ${report.summary.falsePositives} |`,
+        `| False negatives | ${report.summary.falseNegatives} |`,
+        "",
+        "## Failed Thresholds",
+        ""
+    ];
+    const failed = report.thresholdChecks.filter((check) => !check.passed);
+    if (failed.length === 0) {
+        lines.push("None.");
+    }
+    else {
+        for (const check of failed) {
+            lines.push(`- ${metricLabel(check.metric)}: ${formatMetric(check.actual)} ${check.operator} ${formatMetric(check.threshold)} required`);
+        }
+    }
+    lines.push("", "## Targets", "", "| Target | Review | Precision | Recall |", "| --- | --- | ---: | ---: |");
+    for (const target of report.targets) {
+        lines.push(`| ${target.id} | ${target.complete ? "complete" : "incomplete"} | ` +
+            `${formatMetric(target.metrics.precision)} | ${formatMetric(target.metrics.recall)} |`);
+    }
+    return lines.join("\n");
+}
+const reportSizeLimit = 5 * 1024 * 1024;
+const countKeys = [
+    "truePositives",
+    "falsePositives",
+    "falseNegatives",
+    "resolvedFalsePositives",
+    "unreviewed",
+    "unclear"
+];
+function requireNonNegativeInteger(value, label) {
+    if (!Number.isInteger(value) || value < 0) {
+        throw new CorpusMetricsDiffError(`${label} must be a non-negative integer.`);
+    }
+    return value;
+}
+function parseCounts(value, label) {
+    if (!isRecord(value))
+        throw new CorpusMetricsDiffError(`${label} is invalid.`);
+    return Object.fromEntries(countKeys.map((key) => [
+        key,
+        requireNonNegativeInteger(value[key], `${label}.${key}`)
+    ]));
+}
+function metricsMatchCounts(value, counts) {
+    if (!isRecord(value))
+        return false;
+    const expected = calculateMetrics(counts);
+    return value.precision === expected.precision &&
+        value.recall === expected.recall &&
+        value.falsePositiveRate === expected.falsePositiveRate;
+}
+async function loadComparableMetricsReport(reportPath, label) {
+    let fileStat;
+    try {
+        fileStat = await stat(path.resolve(reportPath));
+    }
+    catch {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report could not be read.`);
+    }
+    if (!fileStat.isFile() || fileStat.size > reportSizeLimit) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report must be a file no larger than 5 MiB.`);
+    }
+    let value;
+    try {
+        value = await readJsonFile(path.resolve(reportPath));
+    }
+    catch {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report could not be read as JSON.`);
+    }
+    if (!isRecord(value) || value.schemaVersion !== "1.0.0" ||
+        value.kind !== "doctor.validation.corpus.metrics") {
+        throw new CorpusMetricsDiffError(`${label} file is not a supported corpus metrics report.`);
+    }
+    if (typeof value.version !== "string" || value.version.length === 0) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report version is invalid.`);
+    }
+    if (typeof value.corpusDigest !== "string" || !sha256Digest.test(value.corpusDigest)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report is missing a valid corpusDigest.`);
+    }
+    if (!isRecord(value.summary) || !Array.isArray(value.targets)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report structure is invalid.`);
+    }
+    if (value.status === "incomplete" || value.summary.incompleteTargets !== 0 ||
+        value.summary.unreviewed !== 0 || value.summary.unclear !== 0) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report must be complete.`);
+    }
+    if ((value.status !== "pass" && value.status !== "fail") ||
+        (value.status === "pass" ? value.exitCode !== 0 : value.exitCode !== 1)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report status is invalid.`);
+    }
+    const targetCount = requireNonNegativeInteger(value.summary.targetCount, `${label} summary.targetCount`);
+    const summaryCounts = parseCounts(value.summary, `${label} summary`);
+    if (targetCount === 0 || targetCount !== value.targets.length ||
+        value.summary.completeTargets !== targetCount || !metricsMatchCounts(value.summary, summaryCounts)) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report is internally inconsistent.`);
+    }
+    const ids = new Set();
+    const targets = value.targets.map((target, index) => {
+        if (!isRecord(target) || typeof target.id !== "string" || !publicSafeId.test(target.id) ||
+            target.complete !== true || target.digestMatched !== true) {
+            throw new CorpusMetricsDiffError(`${label} corpus metrics target ${index} is invalid or incomplete.`);
+        }
+        if (ids.has(target.id)) {
+            throw new CorpusMetricsDiffError(`${label} corpus metrics report contains duplicate target IDs.`);
+        }
+        ids.add(target.id);
+        const counts = parseCounts(target.counts, `${label} target ${target.id} counts`);
+        if (!metricsMatchCounts(target.metrics, counts)) {
+            throw new CorpusMetricsDiffError(`${label} corpus metrics report is internally inconsistent.`);
+        }
+        return { id: target.id, counts };
+    });
+    const summedCounts = targets.reduce((total, target) => addCounts(total, target.counts), {
+        truePositives: 0,
+        falsePositives: 0,
+        falseNegatives: 0,
+        resolvedFalsePositives: 0,
+        unreviewed: 0,
+        unclear: 0
+    });
+    if (countKeys.some((key) => summedCounts[key] !== summaryCounts[key])) {
+        throw new CorpusMetricsDiffError(`${label} corpus metrics report is internally inconsistent.`);
+    }
+    return {
+        version: value.version,
+        corpusDigest: value.corpusDigest,
+        summary: { ...summaryCounts, targetCount },
+        targets
+    };
+}
+function subtractCounts(after, before) {
+    return Object.fromEntries(countKeys.map((key) => [key, after[key] - before[key]]));
+}
+function compareMetric(before, after) {
+    return {
+        before: roundMetric(before),
+        after: roundMetric(after),
+        delta: before === null || after === null ? null : roundMetric(after - before)
+    };
+}
+function metricRegressed(precision, recall, falsePositiveRate) {
+    return (precision.delta !== null && precision.delta < 0) ||
+        (recall.delta !== null && recall.delta < 0) ||
+        (falsePositiveRate.delta !== null && falsePositiveRate.delta > 0);
+}
+export async function buildCorpusQualityMetricsDiffReport(beforePath, afterPath, options = {}) {
+    const [before, after] = await Promise.all([
+        loadComparableMetricsReport(beforePath, "Before"),
+        loadComparableMetricsReport(afterPath, "After")
+    ]);
+    const beforeIds = before.targets.map((target) => target.id).sort();
+    const afterIds = after.targets.map((target) => target.id).sort();
+    if (before.corpusDigest !== after.corpusDigest || JSON.stringify(beforeIds) !== JSON.stringify(afterIds)) {
+        throw new CorpusMetricsDiffError("Corpus metrics reports describe different corpus identities.");
+    }
+    const afterTargets = new Map(after.targets.map((target) => [target.id, target]));
+    const targets = before.targets.map((beforeTarget) => {
+        const afterTarget = afterTargets.get(beforeTarget.id);
+        const beforeMetrics = calculateRawMetrics(beforeTarget.counts);
+        const afterMetrics = calculateRawMetrics(afterTarget.counts);
+        const precision = compareMetric(beforeMetrics.precision, afterMetrics.precision);
+        const recall = compareMetric(beforeMetrics.recall, afterMetrics.recall);
+        const falsePositiveRate = compareMetric(beforeMetrics.falsePositiveRate, afterMetrics.falsePositiveRate);
+        return {
+            id: beforeTarget.id,
+            regressed: metricRegressed(precision, recall, falsePositiveRate),
+            counts: subtractCounts(afterTarget.counts, beforeTarget.counts),
+            precision,
+            recall,
+            falsePositiveRate
+        };
+    });
+    const changedTargets = targets.filter((target) => countKeys.some((key) => target.counts[key] !== 0)).length;
+    const beforeMetrics = calculateRawMetrics(before.summary);
+    const afterMetrics = calculateRawMetrics(after.summary);
+    const precision = compareMetric(beforeMetrics.precision, afterMetrics.precision);
+    const recall = compareMetric(beforeMetrics.recall, afterMetrics.recall);
+    const falsePositiveRate = compareMetric(beforeMetrics.falsePositiveRate, afterMetrics.falsePositiveRate);
+    const regression = metricRegressed(precision, recall, falsePositiveRate);
+    const failOnRegression = options.failOnRegression === true;
+    const failed = failOnRegression && regression;
+    return {
+        schemaVersion: "1.0.0",
+        kind: "doctor.validation.corpus.metrics.diff",
+        generatedAt: new Date().toISOString(),
+        version: packageVersion,
+        corpusDigest: before.corpusDigest,
+        status: failed ? "fail" : "pass",
+        exitCode: failed ? 1 : 0,
+        failOnRegression,
+        summary: {
+            comparable: true,
+            regression,
+            changedTargets,
+            precisionBefore: precision.before,
+            precisionAfter: precision.after,
+            precisionDelta: precision.delta,
+            recallBefore: recall.before,
+            recallAfter: recall.after,
+            recallDelta: recall.delta,
+            falsePositiveRateBefore: falsePositiveRate.before,
+            falsePositiveRateAfter: falsePositiveRate.after,
+            falsePositiveRateDelta: falsePositiveRate.delta,
+            counts: subtractCounts(after.summary, before.summary)
+        },
+        before: { version: before.version, targetCount: before.summary.targetCount },
+        after: { version: after.version, targetCount: after.summary.targetCount },
+        targets
+    };
+}
+function formatDelta(value) {
+    if (value === null)
+        return "N/A";
+    const sign = value > 0 ? "+" : "";
+    return `${sign}${(value * 100).toFixed(2)} pp`;
+}
+export function renderCorpusQualityMetricsDiffJson(report) {
+    return JSON.stringify(report, null, 2);
+}
+export function renderCorpusQualityMetricsDiffText(report, options = {}) {
+    const lines = [
+        "Doctor Corpus Metrics Diff",
+        "==========================",
+        `Status: ${report.status.toUpperCase()}`,
+        `Before version: ${report.before.version}`,
+        `After version: ${report.after.version}`,
+        `Regression: ${report.summary.regression ? "yes" : "no"}`,
+        `Precision delta: ${formatDelta(report.summary.precisionDelta)}`,
+        `Recall delta: ${formatDelta(report.summary.recallDelta)}`,
+        `False-positive rate delta: ${formatDelta(report.summary.falsePositiveRateDelta)}`,
+        `Changed targets: ${report.summary.changedTargets}`
+    ];
+    if (options.outputPath)
+        lines.push(`Output: ${options.outputPath}`);
+    lines.push("", "Target Changes", "--------------");
+    const changed = report.targets.filter((target) => countKeys.some((key) => target.counts[key] !== 0));
+    if (changed.length === 0) {
+        lines.push("None.");
+    }
+    else {
+        for (const target of changed) {
+            lines.push(`${target.id}: ${target.regressed ? "REGRESSED" : "CHANGED"} ` +
+                `(precision ${formatDelta(target.precision.delta)}, recall ${formatDelta(target.recall.delta)}, ` +
+                `false-positive rate ${formatDelta(target.falsePositiveRate.delta)})`);
+        }
+    }
+    return lines.join("\n");
+}
+export function renderCorpusQualityMetricsDiffMarkdown(report) {
+    const lines = [
+        "# Doctor Corpus Metrics Diff",
+        "",
+        `- Status: ${report.status.toUpperCase()}`,
+        `- Before version: ${report.before.version}`,
+        `- After version: ${report.after.version}`,
+        `- Regression: ${report.summary.regression ? "yes" : "no"}`,
+        "",
+        "| Metric | Before | After | Delta |",
+        "| --- | ---: | ---: | ---: |",
+        `| Precision | ${formatMetric(report.summary.precisionBefore)} | ${formatMetric(report.summary.precisionAfter)} | ${formatDelta(report.summary.precisionDelta)} |`,
+        `| Recall | ${formatMetric(report.summary.recallBefore)} | ${formatMetric(report.summary.recallAfter)} | ${formatDelta(report.summary.recallDelta)} |`,
+        `| False-positive rate | ${formatMetric(report.summary.falsePositiveRateBefore)} | ${formatMetric(report.summary.falsePositiveRateAfter)} | ${formatDelta(report.summary.falsePositiveRateDelta)} |`,
+        "",
+        "## Target Changes",
+        "",
+        "| Target | Result | Precision | Recall | False-positive rate |",
+        "| --- | --- | ---: | ---: | ---: |"
+    ];
+    const changed = report.targets.filter((target) => countKeys.some((key) => target.counts[key] !== 0));
+    if (changed.length === 0) {
+        lines.push("| None | unchanged | 0.00 pp | 0.00 pp | 0.00 pp |");
+    }
+    else {
+        for (const target of changed) {
+            lines.push(`| ${target.id} | ${target.regressed ? "regressed" : "changed"} | ` +
+                `${formatDelta(target.precision.delta)} | ${formatDelta(target.recall.delta)} | ` +
+                `${formatDelta(target.falsePositiveRate.delta)} |`);
+        }
+    }
+    return lines.join("\n");
+}

package/dist/core/output-contract.js

@@ -101,6 +101,43 @@ const publicSchemaDefinitions = [
         outputKind: "doctor.validation.corpus",
         required: ["schemaVersion", "kind", "corpusType", "generatedAt", "version", "summary", "cases"]
     },
+    {
+        id: "doctor.validation.corpus.metrics.json",
+        command: "codex-plugin-doctor doctor corpus metrics --manifest <corpus.json> --json",
+        outputKind: "doctor.validation.corpus.metrics",
+        required: [
+            "schemaVersion",
+            "kind",
+            "generatedAt",
+            "version",
+            "corpusDigest",
+            "status",
+            "exitCode",
+            "summary",
+            "thresholds",
+            "thresholdChecks",
+            "targets"
+        ]
+    },
+    {
+        id: "doctor.validation.corpus.metrics.diff.json",
+        command: "codex-plugin-doctor doctor corpus metrics diff --before <metrics.json> --after <metrics.json> --json",
+        outputKind: "doctor.validation.corpus.metrics.diff",
+        required: [
+            "schemaVersion",
+            "kind",
+            "generatedAt",
+            "version",
+            "corpusDigest",
+            "status",
+            "exitCode",
+            "failOnRegression",
+            "summary",
+            "before",
+            "after",
+            "targets"
+        ]
+    },
     {
         id: "doctor.recommendations.json",
         command: "codex-plugin-doctor doctor recommend <path> --json",

package/dist/index.d.ts

@@ -11,6 +11,7 @@ export { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 export { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson, type DoctorExportBundle } from "./core/doctor-export-bundle.js";
 export { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson, renderDoctorAttestationVerification, renderDoctorAttestationVerificationJson, verifyDoctorAttestation, type DoctorAttestation, type DoctorAttestationVerificationCheck, type DoctorAttestationVerificationReport, type Digest, type PackageFingerprint } from "./core/attestation.js";
 export { buildExternalValidationCorpusReport, ExternalCorpusManifestError, loadExternalCorpusManifest, type BuildExternalValidationCorpusOptions, type ExternalCorpusCaseResult, type ExternalCorpusClassificationCounts, type ExternalCorpusFindingReview, type ExternalCorpusManifest, type ExternalCorpusMode, type ExternalCorpusProfile, type ExternalCorpusSourceType, type ExternalValidationCorpusReport, type FindingReviewClassification } from "./core/external-validation-corpus.js";
+export { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport, CorpusMetricsDiffError, CorpusMetricsManifestError, loadCorpusMetricsManifest, reconcileCorpusMetricFindings, renderCorpusQualityMetricsDiffJson, renderCorpusQualityMetricsDiffMarkdown, renderCorpusQualityMetricsDiffText, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText, type BuildCorpusMetricsDiffOptions, type BuildCorpusMetricsOptions, type CorpusMetricCounts, type CorpusMetricCountDeltas, type CorpusMetricFinding, type CorpusMetricFindingReview, type CorpusMetricReconciliation, type CorpusMetricsManifest, type CorpusMetricSource, type CorpusMetricThresholdCheck, type CorpusMetricThresholds, type CorpusMetricValues, type CorpusMetricValueComparison, type CorpusQualityMetricsDiffReport, type CorpusQualityMetricsDiffTarget, type CorpusQualityMetricsReport, type CorpusQualityMetricsTargetResult } from "./core/corpus-quality-metrics.js";
 export { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson, type DoctorOutputContract, type OutputContractRule, type OutputContractSchema } from "./core/output-contract.js";
 export { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport, type BuildDoctorValidationCorpusOptions, type DoctorValidationCorpusReport, type ValidationCorpusCaseDefinition, type ValidationCorpusCaseResult } from "./core/validation-corpus.js";
 export { buildDoctorExportBundleFromAnalysis, buildDoctorRecommendationsFromAnalysis, buildPackageAnalysis, type PackageAnalysis, type PackageAnalysisOptions, type PackageAnalysisStage, type PackageAnalysisTiming } from "./core/package-analysis.js";

package/dist/index.js

@@ -11,6 +11,7 @@ export { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 export { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson } from "./core/doctor-export-bundle.js";
 export { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson, renderDoctorAttestationVerification, renderDoctorAttestationVerificationJson, verifyDoctorAttestation } from "./core/attestation.js";
 export { buildExternalValidationCorpusReport, ExternalCorpusManifestError, loadExternalCorpusManifest } from "./core/external-validation-corpus.js";
+export { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport, CorpusMetricsDiffError, CorpusMetricsManifestError, loadCorpusMetricsManifest, reconcileCorpusMetricFindings, renderCorpusQualityMetricsDiffJson, renderCorpusQualityMetricsDiffMarkdown, renderCorpusQualityMetricsDiffText, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText } from "./core/corpus-quality-metrics.js";
 export { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson } from "./core/output-contract.js";
 export { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport } from "./core/validation-corpus.js";
 export { buildDoctorExportBundleFromAnalysis, buildDoctorRecommendationsFromAnalysis, buildPackageAnalysis } from "./core/package-analysis.js";

package/dist/run-cli.d.ts

@@ -1,3 +1,4 @@
+import { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport } from "./core/corpus-quality-metrics.js";
 import { runCheck } from "./index.js";
 import { writeRawDoctorConfig } from "./index.js";
 export interface CliIo {
@@ -18,5 +19,7 @@ export interface RunCliOptions {
     now?: () => Date;
     releaseAssetUploadImpl?: (args: string[]) => Promise<void>;
     resolveLatestVersion?: () => Promise<string>;
+    buildCorpusMetricsDiffReportImpl?: typeof buildCorpusQualityMetricsDiffReport;
+    buildCorpusMetricsReportImpl?: typeof buildCorpusQualityMetricsReport;
 }
 export declare function runCli(args: string[], io?: CliIo, options?: RunCliOptions): Promise<number>;

package/dist/run-cli.js

@@ -21,6 +21,7 @@ import { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestatio
 import { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson } from "./core/output-contract.js";
 import { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport } from "./core/validation-corpus.js";
 import { buildExternalValidationCorpusReport, ExternalCorpusManifestError } from "./core/external-validation-corpus.js";
+import { buildCorpusQualityMetricsDiffReport, buildCorpusQualityMetricsReport, CorpusMetricsDiffError, CorpusMetricsManifestError, renderCorpusQualityMetricsDiffJson, renderCorpusQualityMetricsDiffMarkdown, renderCorpusQualityMetricsDiffText, renderCorpusQualityMetricsJson, renderCorpusQualityMetricsMarkdown, renderCorpusQualityMetricsText } from "./core/corpus-quality-metrics.js";
 import { buildDoctorPerformanceReport, renderDoctorPerformanceReport, renderDoctorPerformanceReportJson } from "./core/performance-report.js";
 import { buildDoctorRuntimePlan, evaluateRuntimeApproval, renderDoctorRuntimePlan, renderDoctorRuntimePlanMarkdown, renderDoctorRuntimePlanJson, runtimeApprovalPassed } from "./core/runtime-plan.js";
 import { buildDoctorRuntimePolicyReport, renderDoctorRuntimePolicy, renderDoctorRuntimePolicyJson } from "./core/runtime-policy.js";
@@ -99,7 +100,8 @@ function parseRuntimeSandbox(flags, options = { requireRuntime: true }) {
     return value;
 }
 function printUsage(io) {
-    io.writeStderr("Usage: codex-plugin-doctor check <path|--installed> [filter] [--policy codex-publish|mcp-strict|security] [--compat] [--json|--markdown|--badge-json|--badge-markdown] [--output <path>] [--history <path>] [--runtime] [--sandbox docker] [--require-runtime-approval --runtime-approval-digest <digest>] [--verbose-runtime] [--explain] [--no-animations] [--ascii] [--changed-since <ref>]\n       codex-plugin-doctor audit --installed [filter] [--policy codex-publish|mcp-strict|security] [--security] [--compat] [--json] [--output <path>] [--cache] [--changed]\n       codex-plugin-doctor audit deps <path> [--policy codex-publish|mcp-strict|security] [--recommend] [--json|--sarif] [--output <path>]\n       codex-plugin-doctor mcp <path> [--json] [--output <path>]\n       codex-plugin-doctor security <path> [--policy security] [--json|--scorecard]\n       codex-plugin-doctor compat <path> [--all|--client <client>] [--json] [--scorecard] [--output <path>] [--install-preview|--apply --backup]\n       codex-plugin-doctor suppress add <path> [--fingerprint <sha256> --reason <text> --expires-at YYYY-MM-DD] [--config <path>] [--json]\n       codex-plugin-doctor suppress list <path> [--config <path>] [--json]\n       codex-plugin-doctor suppress remove <path> [--fingerprint <sha256>|--index <n>] [--config <path>] [--json]\n       codex-plugin-doctor fix <path> (--dry-run|--interactive --backup|--apply --backup)\n       codex-plugin-doctor history <history.jsonl> [--json] [--fail-on-regression]\n       codex-plugin-doctor watch <path> [--runtime] [--json] [--output <path>] [--debounce-ms <ms>] [--max-iterations <n>] [--fail-fast] [--accumulate-json <path>]\n       codex-plugin-doctor doctor [npm <package>|contract|corpus [--manifest <corpus.json>] [--json] [--output <path>]|runtime-plan <path> [--runtime --sandbox docker] [--json|--markdown] [--output <path>]|runtime-policy <path> [--runtime --sandbox docker] [--json] [--output <path>]|review-bundle <path> --output <dir> --sign-key-env NAME [--json] [--allow-dirty] [--allow-untagged]|review-bundle verify <bundle-dir> --target <path> --sign-key-env NAME [--json] [--output <path>] [--failures-only]|review-bundle diff --before <dir> --after <dir> [--json]|attest <path> [--sign-key-env NAME]|attest verify <attestation.json> --target <path> --sign-key-env NAME|release-evidence <path> --sign-key-env NAME [--runtime --sandbox docker] [--allow-dirty] [--allow-untagged] [--require-runtime-approval --runtime-approval-digest <digest>]|release-evidence verify <evidence.json> --target <path> --sign-key-env NAME|release-evidence asset <path> --tag <tag> --output <evidence.json> --sign-key-env NAME [--upload]|mcp <path>|inspector <path>|diff --before <path> --after <path>|recommend <path>|trust <path>|perf <path> [--max-total-ms <ms>] [--max-stage-ms stage=ms]|export --bundle <path>|snapshot|clients|--json|--update-check]\n       codex-plugin-doctor init [path] [--template skill-only|mcp-stdio|mcp-http|full-runtime]\n       codex-plugin-doctor init-ci [path]\n       codex-plugin-doctor init-git-hooks [path] [--force] [--json]\n       codex-plugin-doctor init-git-hooks [path] --remove [--json]\n       codex-plugin-doctor completion bash|zsh|fish\n       codex-plugin-doctor config validate <path> [--json]\n       codex-plugin-doctor release check <path> [--json] [--runtime --sandbox docker]\n       codex-plugin-doctor self-test\n       codex-plugin-doctor list --installed\n       codex-plugin-doctor explain <finding-id>\n       codex-plugin-doctor --version\n\nFirst run:\n       codex-plugin-doctor doctor\n       codex-plugin-doctor self-test\n       codex-plugin-doctor init my-plugin\n       codex-plugin-doctor check . --runtime --explain");
+    io.writeStderr("Usage: codex-plugin-doctor check <path|--installed> [filter] [--policy codex-publish|mcp-strict|security] [--compat] [--json|--markdown|--badge-json|--badge-markdown] [--output <path>] [--history <path>] [--runtime] [--sandbox docker] [--require-runtime-approval --runtime-approval-digest <digest>] [--verbose-runtime] [--explain] [--no-animations] [--ascii] [--changed-since <ref>]\n       codex-plugin-doctor audit --installed [filter] [--policy codex-publish|mcp-strict|security] [--security] [--compat] [--json] [--output <path>] [--cache] [--changed]\n       codex-plugin-doctor audit deps <path> [--policy codex-publish|mcp-strict|security] [--recommend] [--json|--sarif] [--output <path>]\n       codex-plugin-doctor mcp <path> [--json] [--output <path>]\n       codex-plugin-doctor security <path> [--policy security] [--json|--scorecard]\n       codex-plugin-doctor compat <path> [--all|--client <client>] [--json] [--scorecard] [--output <path>] [--install-preview|--apply --backup]\n       codex-plugin-doctor suppress add <path> [--fingerprint <sha256> --reason <text> --expires-at YYYY-MM-DD] [--config <path>] [--json]\n       codex-plugin-doctor suppress list <path> [--config <path>] [--json]\n       codex-plugin-doctor suppress remove <path> [--fingerprint <sha256>|--index <n>] [--config <path>] [--json]\n       codex-plugin-doctor fix <path> (--dry-run|--interactive --backup|--apply --backup)\n       codex-plugin-doctor history <history.jsonl> [--json] [--fail-on-regression]\n       codex-plugin-doctor watch <path> [--runtime] [--json] [--output <path>] [--debounce-ms <ms>] [--max-iterations <n>] [--fail-fast] [--accumulate-json <path>]\n       codex-plugin-doctor doctor [npm <package>|contract|corpus [--manifest <corpus.json>] [--json] [--output <path>]|corpus metrics --manifest <corpus.json> [--json|--markdown] [--output <path>] [--min-precision <0..1>] [--min-recall <0..1>] [--max-false-positive-rate <0..1>]|runtime-plan <path> [--runtime --sandbox docker] [--json|--markdown] [--output <path>]|runtime-policy <path> [--runtime --sandbox docker] [--json] [--output <path>]|review-bundle <path> --output <dir> --sign-key-env NAME [--json] [--allow-dirty] [--allow-untagged]|review-bundle verify <bundle-dir> --target <path> --sign-key-env NAME [--json] [--output <path>] [--failures-only]|review-bundle diff --before <dir> --after <dir> [--json]|attest <path> [--sign-key-env NAME]|attest verify <attestation.json> --target <path> --sign-key-env NAME|release-evidence <path> --sign-key-env NAME [--runtime --sandbox docker] [--allow-dirty] [--allow-untagged] [--require-runtime-approval --runtime-approval-digest <digest>]|release-evidence verify <evidence.json> --target <path> --sign-key-env NAME|release-evidence asset <path> --tag <tag> --output <evidence.json> --sign-key-env NAME [--upload]|mcp <path>|inspector <path>|diff --before <path> --after <path>|recommend <path>|trust <path>|perf <path> [--max-total-ms <ms>] [--max-stage-ms stage=ms]|export --bundle <path>|snapshot|clients|--json|--update-check]\n       codex-plugin-doctor init [path] [--template skill-only|mcp-stdio|mcp-http|full-runtime]\n       codex-plugin-doctor init-ci [path]\n       codex-plugin-doctor init-git-hooks [path] [--force] [--json]\n       codex-plugin-doctor init-git-hooks [path] --remove [--json]\n       codex-plugin-doctor completion bash|zsh|fish\n       codex-plugin-doctor config validate <path> [--json]\n       codex-plugin-doctor release check <path> [--json] [--runtime --sandbox docker]\n       codex-plugin-doctor self-test\n       codex-plugin-doctor list --installed\n       codex-plugin-doctor explain <finding-id>\n       codex-plugin-doctor --version\n\nFirst run:\n       codex-plugin-doctor doctor\n       codex-plugin-doctor self-test\n       codex-plugin-doctor init my-plugin\n       codex-plugin-doctor check . --runtime --explain");
+    io.writeStderr("Corpus quality regression: codex-plugin-doctor doctor corpus metrics diff --before <metrics.json> --after <metrics.json> [--fail-on-regression] [--json|--markdown] [--output <path>]");
     io.writeStderr("Suppression governance: codex-plugin-doctor suppress list <path> [--fail-on-expired] [--fail-on-invalid] [--warn-expiring-within-days <days>]\n       codex-plugin-doctor suppress prune <path> [--apply] [--json]");
     io.writeStderr("Baseline gating: codex-plugin-doctor baseline create <path> --output <path> [--runtime]\n       codex-plugin-doctor check <path> --baseline <path>");
 }
@@ -1416,6 +1418,156 @@ export async function runCli(args, io = defaultIo, options = {}) {
             return report.exitCode;
         }
         if (maybePath === "corpus") {
+            if (remainingArgs[0] === "metrics") {
+                const metricsArgs = remainingArgs.slice(1);
+                if (metricsArgs[0] === "diff") {
+                    const diffArgs = metricsArgs.slice(1);
+                    const jsonOutput = diffArgs.includes("--json");
+                    const markdownOutput = diffArgs.includes("--markdown");
+                    const failOnRegression = diffArgs.includes("--fail-on-regression");
+                    const valueOptions = new Map();
+                    const valueFlags = new Set(["--before", "--after", "--output"]);
+                    const booleanFlags = new Set(["--json", "--markdown", "--fail-on-regression"]);
+                    for (let index = 0; index < diffArgs.length; index += 1) {
+                        const argument = diffArgs[index];
+                        if (booleanFlags.has(argument))
+                            continue;
+                        if (!valueFlags.has(argument)) {
+                            io.writeStderr(`Unknown corpus metrics diff argument: ${argument}.`);
+                            return 2;
+                        }
+                        const value = diffArgs[index + 1];
+                        if (!value || value.startsWith("--")) {
+                            io.writeStderr(`Missing path after ${argument}.`);
+                            return 2;
+                        }
+                        valueOptions.set(argument, value);
+                        index += 1;
+                    }
+                    const beforePath = valueOptions.get("--before");
+                    const afterPath = valueOptions.get("--after");
+                    const outputPath = valueOptions.get("--output") ?? null;
+                    if (!beforePath || !afterPath) {
+                        io.writeStderr("Corpus metrics diff requires --before <metrics.json> and --after <metrics.json>.");
+                        return 2;
+                    }
+                    if (jsonOutput && markdownOutput) {
+                        io.writeStderr("Use either --json or --markdown, not both.");
+                        return 2;
+                    }
+                    let report;
+                    try {
+                        const builder = options.buildCorpusMetricsDiffReportImpl ?? buildCorpusQualityMetricsDiffReport;
+                        report = await builder(beforePath, afterPath, { failOnRegression });
+                    }
+                    catch (error) {
+                        if (error instanceof CorpusMetricsDiffError) {
+                            io.writeStderr(error.message);
+                            return 2;
+                        }
+                        const message = error instanceof Error ? error.message : "Unknown error";
+                        io.writeStderr(`Corpus metrics diff failed: ${message}`);
+                        return 2;
+                    }
+                    const rendered = jsonOutput
+                        ? renderCorpusQualityMetricsDiffJson(report)
+                        : markdownOutput
+                            ? renderCorpusQualityMetricsDiffMarkdown(report)
+                            : renderCorpusQualityMetricsDiffText(report, { outputPath });
+                    if (outputPath)
+                        await writeFile(outputPath, rendered, "utf8");
+                    io.writeStdout(rendered);
+                    return report.exitCode;
+                }
+                const jsonOutput = metricsArgs.includes("--json");
+                const markdownOutput = metricsArgs.includes("--markdown");
+                const valueOptions = new Map();
+                const knownValueFlags = new Set([
+                    "--manifest",
+                    "--output",
+                    "--min-precision",
+                    "--min-recall",
+                    "--max-false-positive-rate"
+                ]);
+                for (let index = 0; index < metricsArgs.length; index += 1) {
+                    const argument = metricsArgs[index];
+                    if (argument === "--json" || argument === "--markdown")
+                        continue;
+                    if (!knownValueFlags.has(argument)) {
+                        io.writeStderr(`Unknown corpus metrics argument: ${argument}.`);
+                        return 2;
+                    }
+                    const value = metricsArgs[index + 1];
+                    if (!value || value.startsWith("--")) {
+                        const label = argument === "--manifest"
+                            ? "path after --manifest"
+                            : argument === "--output"
+                                ? "path after --output"
+                                : `value after ${argument}`;
+                        io.writeStderr(`Missing ${label}.`);
+                        return 2;
+                    }
+                    valueOptions.set(argument, value);
+                    index += 1;
+                }
+                const manifestPath = valueOptions.get("--manifest");
+                const outputPath = valueOptions.get("--output") ?? null;
+                if (!manifestPath) {
+                    io.writeStderr("Corpus metrics requires --manifest <path>.");
+                    return 2;
+                }
+                if (jsonOutput && markdownOutput) {
+                    io.writeStderr("Use either --json or --markdown, not both.");
+                    return 2;
+                }
+                const thresholds = {};
+                const thresholdFlags = [
+                    ["--min-precision", "minPrecision"],
+                    ["--min-recall", "minRecall"],
+                    ["--max-false-positive-rate", "maxFalsePositiveRate"]
+                ];
+                for (const [flag, key] of thresholdFlags) {
+                    const raw = valueOptions.get(flag);
+                    if (raw === undefined)
+                        continue;
+                    const value = Number(raw);
+                    if (!Number.isFinite(value) || value < 0 || value > 1) {
+                        io.writeStderr(`${flag} must be between 0 and 1.`);
+                        return 2;
+                    }
+                    thresholds[key] = value;
+                }
+                let report;
+                try {
+                    const builder = options.buildCorpusMetricsReportImpl ?? buildCorpusQualityMetricsReport;
+                    const buildOptions = {
+                        thresholds,
+                        environment: {
+                            env: terminalContext.env,
+                            platform: terminalContext.platform
+                        }
+                    };
+                    report = await builder(manifestPath, buildOptions);
+                }
+                catch (error) {
+                    if (error instanceof CorpusMetricsManifestError) {
+                        io.writeStderr(error.message);
+                        return 2;
+                    }
+                    const message = error instanceof Error ? error.message : "Unknown error";
+                    io.writeStderr(`Corpus metrics analysis failed: ${message}`);
+                    return 2;
+                }
+                const rendered = jsonOutput
+                    ? renderCorpusQualityMetricsJson(report)
+                    : markdownOutput
+                        ? renderCorpusQualityMetricsMarkdown(report)
+                        : renderCorpusQualityMetricsText(report, { outputPath });
+                if (outputPath)
+                    await writeFile(outputPath, rendered, "utf8");
+                io.writeStdout(rendered);
+                return report.exitCode;
+            }
             const jsonOutput = remainingArgs.includes("--json");
             const outputIndex = remainingArgs.indexOf("--output");
             const outputPath = outputIndex === -1 ? null : remainingArgs[outputIndex + 1];

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "codex-plugin-doctor",
-  "version": "1.47.0",
+  "version": "1.49.0",
   "description": "CLI-first validator for Codex plugins, skills, and MCP package surfaces with runtime MCP protocol validation.",
   "type": "module",
   "main": "./dist/index.js",