codex-plugin-doctor 0.18.0 → 0.19.0

package/README.md

@@ -77,6 +77,7 @@ Output formats:
 - validation history JSONL and trend summaries
 - deterministic local attestation artifacts
 - output contract and rule catalog freeze metadata
+- bundled validation corpus reports
 - `--output` file writing
 - CI summary and artifact generation
 
@@ -180,6 +181,8 @@ codex-plugin-doctor self-test
 codex-plugin-doctor doctor
 codex-plugin-doctor doctor contract
 codex-plugin-doctor doctor contract --json --output output-contract.json
+codex-plugin-doctor doctor corpus
+codex-plugin-doctor doctor corpus --json --output validation-corpus.json
 codex-plugin-doctor doctor npm codex-plugin-doctor
 codex-plugin-doctor doctor npm codex-plugin-doctor --json --output npm-preinstall.json
 codex-plugin-doctor doctor attest .
@@ -262,7 +265,7 @@ codex-plugin-doctor check . --json --runtime --verbose-runtime
 
 `self-test` runs the bundled runtime-complete sample through static validation, runtime MCP probes, and the compatibility scorecard. It is the fastest post-install check after `npm install -g codex-plugin-doctor`.
 
-`doctor` checks the local environment, including package version, platform, Node version, npm global prefix, Codex home, and Codex plugin cache visibility. The text output also includes recommended next commands for self-test, installed plugin discovery, runtime checks, compatibility scoring, and CI setup. `doctor contract` publishes the machine-readable output contract, including public JSON schema surfaces, stable-through-1.0 compatibility metadata, and a frozen rule catalog digest. Add `--json` for automation or `--output output-contract.json` to write the contract to disk. `doctor npm <package>` runs a preinstall scan by packing the npm package with scripts disabled, extracting the publish tarball, and running validation, security, trust, and recommendation checks against the shipped contents. Add `--json` for automation or `--output npm-preinstall.json` to write the report to disk. `doctor attest <path>` creates a deterministic local attestation with a package fingerprint, report digest, validation/security/compatibility/trust summary, and unsigned verification metadata. Add `--json` for automation or `--output attestation.json` to write the artifact to disk. `doctor inspector <path>` builds a safe MCP Inspector launch command from a packaged `.mcp.json` file without starting the Inspector proxy automatically. Use `--server <name>` when the package contains multiple MCP server entries. `doctor diff --before <path> --after <path>` compares two package roots and reports new findings, resolved findings, trust score delta, and whether risk increased. `doctor recommend <path>` turns validation, security, and compatibility signals into a prioritized action plan with blocker, high, medium, and info actions. Add `--json` for automation or `--output recommendations.json` to write the report to disk. `doctor trust <path>` creates a local trust score from package lifecycle scripts, dependency specs, and MCP security findings. Use it before release when you want supply-chain risks summarized as one score. `doctor perf <path>` profiles the shared package analysis pipeline and reports per-stage durations for validation, config, security, compatibility, trust, recommendations, and total runtime. `doctor export --bundle <path>` creates a redacted operator handoff bundle that includes validation JSON, security scorecard data, compatibility matrix, recommendations, and trust score in one file. `doctor snapshot` creates a redacted diagnostics bundle with environment health, client config readiness, installed plugin metadata, and next commands. Add `--json` for machine-readable output or `--output doctor-snapshot.json` to write the bundle to disk. `doctor clients` reports local Codex, Claude Desktop, Cursor, Cline, and Windsurf config readiness. `doctor --update-check` compares the installed CLI version with the latest npm version and prints the upgrade command when a newer release is available.
+`doctor` checks the local environment, including package version, platform, Node version, npm global prefix, Codex home, and Codex plugin cache visibility. The text output also includes recommended next commands for self-test, installed plugin discovery, runtime checks, compatibility scoring, and CI setup. `doctor contract` publishes the machine-readable output contract, including public JSON schema surfaces, stable-through-1.0 compatibility metadata, and a frozen rule catalog digest. Add `--json` for automation or `--output output-contract.json` to write the contract to disk. `doctor corpus` runs the bundled validation corpus against healthy runtime, risky security, and starter skill packages, then reports whether each case matched its expected outcome. Add `--json` for automation or `--output validation-corpus.json` to write the corpus report to disk. `doctor npm <package>` runs a preinstall scan by packing the npm package with scripts disabled, extracting the publish tarball, and running validation, security, trust, and recommendation checks against the shipped contents. Add `--json` for automation or `--output npm-preinstall.json` to write the report to disk. `doctor attest <path>` creates a deterministic local attestation with a package fingerprint, report digest, validation/security/compatibility/trust summary, and unsigned verification metadata. Add `--json` for automation or `--output attestation.json` to write the artifact to disk. `doctor inspector <path>` builds a safe MCP Inspector launch command from a packaged `.mcp.json` file without starting the Inspector proxy automatically. Use `--server <name>` when the package contains multiple MCP server entries. `doctor diff --before <path> --after <path>` compares two package roots and reports new findings, resolved findings, trust score delta, and whether risk increased. `doctor recommend <path>` turns validation, security, and compatibility signals into a prioritized action plan with blocker, high, medium, and info actions. Add `--json` for automation or `--output recommendations.json` to write the report to disk. `doctor trust <path>` creates a local trust score from package lifecycle scripts, dependency specs, and MCP security findings. Use it before release when you want supply-chain risks summarized as one score. `doctor perf <path>` profiles the shared package analysis pipeline and reports per-stage durations for validation, config, security, compatibility, trust, recommendations, and total runtime. `doctor export --bundle <path>` creates a redacted operator handoff bundle that includes validation JSON, security scorecard data, compatibility matrix, recommendations, and trust score in one file. `doctor snapshot` creates a redacted diagnostics bundle with environment health, client config readiness, installed plugin metadata, and next commands. Add `--json` for machine-readable output or `--output doctor-snapshot.json` to write the bundle to disk. `doctor clients` reports local Codex, Claude Desktop, Cursor, Cline, and Windsurf config readiness. `doctor --update-check` compares the installed CLI version with the latest npm version and prints the upgrade command when a newer release is available.
 
 `audit --installed` runs a local ecosystem audit against every discovered Codex plugin in the installed plugin cache. Add `--security` to include security scorecards, `--compat` to include the all-client compatibility matrix, and `--json --output local-audit.json` when you want a shareable machine-readable report. Add `--cache` to reuse unchanged plugin results between runs; add `--changed` to only report plugins whose fingerprint changed since the last cached audit. Use `--cache-file path/to/audit-cache.json` when CI or scripted runs need an explicit cache location.
 

package/dist/core/output-contract.js

@@ -42,6 +42,12 @@ const publicSchemaDefinitions = [
         command: "codex-plugin-doctor doctor --json",
         required: ["schemaVersion", "generatedAt", "version", "platform", "node", "checks"]
     },
+    {
+        id: "doctor.validation.corpus.json",
+        command: "codex-plugin-doctor doctor corpus --json",
+        outputKind: "doctor.validation.corpus",
+        required: ["schemaVersion", "kind", "generatedAt", "version", "summary", "cases"]
+    },
     {
         id: "doctor.recommendations.json",
         command: "codex-plugin-doctor doctor recommend <path> --json",

package/dist/core/validation-corpus.d.ts

@@ -0,0 +1,57 @@
+import { type PackageAnalysisOptions } from "./package-analysis.js";
+type ValidationStatus = "pass" | "warn" | "fail";
+export interface ValidationCorpusCaseDefinition {
+    id: string;
+    label: string;
+    profile: string;
+    sourceType: "bundled-example";
+    relativePath: string;
+    runtimeEnabled: boolean;
+    expected: {
+        validationStatus: ValidationStatus;
+        findingIds?: string[];
+    };
+}
+export interface ValidationCorpusCaseResult {
+    id: string;
+    label: string;
+    profile: string;
+    sourceType: "bundled-example";
+    targetPath: string;
+    runtimeEnabled: boolean;
+    expected: {
+        validationStatus: ValidationStatus;
+        findingIds: string[];
+    };
+    actual: {
+        validationStatus: ValidationStatus;
+        findingIds: string[];
+        securityStatus: ValidationStatus;
+        trustStatus: ValidationStatus;
+        compatibilityFailedClients: string[];
+    };
+    expectationMatched: boolean;
+}
+export interface DoctorValidationCorpusReport {
+    schemaVersion: "1.0.0";
+    kind: "doctor.validation.corpus";
+    generatedAt: string;
+    version: string;
+    summary: {
+        status: "pass" | "fail";
+        caseCount: number;
+        passedExpectations: number;
+        failedExpectations: number;
+        runtimeCases: number;
+    };
+    cases: ValidationCorpusCaseResult[];
+}
+export interface BuildDoctorValidationCorpusOptions {
+    environment?: PackageAnalysisOptions["environment"];
+}
+export declare function buildDoctorValidationCorpusReport(options?: BuildDoctorValidationCorpusOptions): Promise<DoctorValidationCorpusReport>;
+export declare function renderDoctorValidationCorpusJson(report: DoctorValidationCorpusReport): string;
+export declare function renderDoctorValidationCorpusReport(report: DoctorValidationCorpusReport, options?: {
+    outputPath?: string | null;
+}): string;
+export {};

package/dist/core/validation-corpus.js

@@ -0,0 +1,128 @@
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { buildPackageAnalysis } from "./package-analysis.js";
+import { validatePlugin } from "./validate-plugin.js";
+import { matrixExitCode } from "../compatibility/compatibility-matrix.js";
+import { packageVersion } from "../version.js";
+const bundledCorpusCases = [
+    {
+        id: "bundled-runtime-healthy",
+        label: "Bundled runtime-complete example",
+        profile: "healthy-runtime",
+        sourceType: "bundled-example",
+        relativePath: "examples/codex-doctor-runtime",
+        runtimeEnabled: true,
+        expected: {
+            validationStatus: "pass"
+        }
+    },
+    {
+        id: "bundled-risky-security",
+        label: "Bundled risky security example",
+        profile: "risky-security",
+        sourceType: "bundled-example",
+        relativePath: "examples/codex-doctor-risky",
+        runtimeEnabled: false,
+        expected: {
+            validationStatus: "fail",
+            findingIds: ["plugin.security.hard_coded_secret"]
+        }
+    },
+    {
+        id: "bundled-starter-skill",
+        label: "Bundled starter skill-only example",
+        profile: "skill-only",
+        sourceType: "bundled-example",
+        relativePath: "examples/codex-doctor-starter",
+        runtimeEnabled: false,
+        expected: {
+            validationStatus: "pass"
+        }
+    }
+];
+function resolvePackageRoot() {
+    return path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
+}
+function includesExpectedFindings(actualFindingIds, expectedFindingIds) {
+    return expectedFindingIds.every((findingId) => actualFindingIds.includes(findingId));
+}
+async function runCorpusCase(caseDefinition, options) {
+    const targetPath = path.resolve(resolvePackageRoot(), caseDefinition.relativePath);
+    const analysis = await buildPackageAnalysis(targetPath, {
+        environment: options.environment,
+        runCheck: (pathToCheck) => validatePlugin(pathToCheck, {
+            runtime: caseDefinition.runtimeEnabled
+        })
+    });
+    const findingIds = analysis.validation.findings.map((finding) => finding.id).sort();
+    const expectedFindingIds = [...(caseDefinition.expected.findingIds ?? [])].sort();
+    const compatibilityFailedClients = analysis.compatibility.results
+        .filter((result) => result.status === "fail")
+        .map((result) => result.client);
+    const expectationMatched = analysis.validation.status === caseDefinition.expected.validationStatus &&
+        includesExpectedFindings(findingIds, expectedFindingIds);
+    return {
+        id: caseDefinition.id,
+        label: caseDefinition.label,
+        profile: caseDefinition.profile,
+        sourceType: caseDefinition.sourceType,
+        targetPath,
+        runtimeEnabled: caseDefinition.runtimeEnabled,
+        expected: {
+            validationStatus: caseDefinition.expected.validationStatus,
+            findingIds: expectedFindingIds
+        },
+        actual: {
+            validationStatus: analysis.validation.status,
+            findingIds,
+            securityStatus: analysis.security.status,
+            trustStatus: analysis.trust.status,
+            compatibilityFailedClients: matrixExitCode(analysis.compatibility) === 1
+                ? compatibilityFailedClients
+                : []
+        },
+        expectationMatched
+    };
+}
+export async function buildDoctorValidationCorpusReport(options = {}) {
+    const cases = await Promise.all(bundledCorpusCases.map((caseDefinition) => runCorpusCase(caseDefinition, options)));
+    const failedExpectations = cases.filter((caseResult) => !caseResult.expectationMatched).length;
+    return {
+        schemaVersion: "1.0.0",
+        kind: "doctor.validation.corpus",
+        generatedAt: new Date().toISOString(),
+        version: packageVersion,
+        summary: {
+            status: failedExpectations > 0 ? "fail" : "pass",
+            caseCount: cases.length,
+            passedExpectations: cases.length - failedExpectations,
+            failedExpectations,
+            runtimeCases: cases.filter((caseResult) => caseResult.runtimeEnabled).length
+        },
+        cases
+    };
+}
+export function renderDoctorValidationCorpusJson(report) {
+    return JSON.stringify(report, null, 2);
+}
+export function renderDoctorValidationCorpusReport(report, options = {}) {
+    const lines = [
+        "Doctor Validation Corpus",
+        "========================",
+        `Version: ${report.version}`,
+        `Status: ${report.summary.status.toUpperCase()}`,
+        `Cases: ${report.summary.caseCount}`,
+        `Passed expectations: ${report.summary.passedExpectations}`,
+        `Failed expectations: ${report.summary.failedExpectations}`,
+        `Runtime cases: ${report.summary.runtimeCases}`
+    ];
+    if (options.outputPath) {
+        lines.push(`Output: ${options.outputPath}`);
+    }
+    lines.push("", "Cases", "-----");
+    for (const caseResult of report.cases) {
+        lines.push(`${caseResult.id}: ${caseResult.expectationMatched ? "PASS" : "FAIL"} ` +
+            `(${caseResult.actual.validationStatus.toUpperCase()}, ${caseResult.actual.findingIds.length} findings)`);
+    }
+    return lines.join("\n");
+}

package/dist/index.d.ts

@@ -6,6 +6,7 @@ export { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 export { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson, type DoctorExportBundle } from "./core/doctor-export-bundle.js";
 export { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson, type DoctorAttestation, type Digest, type PackageFingerprint } from "./core/attestation.js";
 export { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson, type DoctorOutputContract, type OutputContractRule, type OutputContractSchema } from "./core/output-contract.js";
+export { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport, type BuildDoctorValidationCorpusOptions, type DoctorValidationCorpusReport, type ValidationCorpusCaseDefinition, type ValidationCorpusCaseResult } from "./core/validation-corpus.js";
 export { buildDoctorExportBundleFromAnalysis, buildDoctorRecommendationsFromAnalysis, buildPackageAnalysis, type PackageAnalysis, type PackageAnalysisOptions, type PackageAnalysisStage, type PackageAnalysisTiming } from "./core/package-analysis.js";
 export { buildDoctorPerformanceReport, renderDoctorPerformanceReport, renderDoctorPerformanceReportJson, type BuildDoctorPerformanceReportOptions, type DoctorPerformanceReport, type DoctorPerformanceStage, type DoctorPerformanceStageName } from "./core/performance-report.js";
 export { buildDoctorNpmPackageReport, renderDoctorNpmPackageReport, renderDoctorNpmPackageReportJson, type BuildDoctorNpmPackageReportOptions, type DoctorNpmPackageReport } from "./core/npm-package-doctor.js";

package/dist/index.js

@@ -6,6 +6,7 @@ export { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 export { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson } from "./core/doctor-export-bundle.js";
 export { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson } from "./core/attestation.js";
 export { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson } from "./core/output-contract.js";
+export { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport } from "./core/validation-corpus.js";
 export { buildDoctorExportBundleFromAnalysis, buildDoctorRecommendationsFromAnalysis, buildPackageAnalysis } from "./core/package-analysis.js";
 export { buildDoctorPerformanceReport, renderDoctorPerformanceReport, renderDoctorPerformanceReportJson } from "./core/performance-report.js";
 export { buildDoctorNpmPackageReport, renderDoctorNpmPackageReport, renderDoctorNpmPackageReportJson } from "./core/npm-package-doctor.js";

package/dist/run-cli.js

@@ -18,6 +18,7 @@ import { buildDoctorRecommendations, renderDoctorRecommendations, renderDoctorRe
 import { buildDoctorExportBundle, renderDoctorExportBundle, renderDoctorExportBundleJson } from "./core/doctor-export-bundle.js";
 import { buildDoctorAttestation, renderDoctorAttestation, renderDoctorAttestationJson } from "./core/attestation.js";
 import { buildDoctorOutputContract, renderDoctorOutputContract, renderDoctorOutputContractJson } from "./core/output-contract.js";
+import { buildDoctorValidationCorpusReport, renderDoctorValidationCorpusJson, renderDoctorValidationCorpusReport } from "./core/validation-corpus.js";
 import { buildDoctorPerformanceReport, renderDoctorPerformanceReport, renderDoctorPerformanceReportJson } from "./core/performance-report.js";
 import { buildDoctorNpmPackageReport, renderDoctorNpmPackageReport, renderDoctorNpmPackageReportJson } from "./core/npm-package-doctor.js";
 import { buildDoctorRiskDiffReport, renderDoctorRiskDiffReport, renderDoctorRiskDiffReportJson } from "./core/risk-diff.js";
@@ -67,7 +68,7 @@ const defaultIo = {
     }
 };
 function printUsage(io) {
-    io.writeStderr("Usage: codex-plugin-doctor check <path|--installed> [filter] [--policy codex-publish|mcp-strict|security] [--compat] [--json|--markdown|--badge-json|--badge-markdown] [--output <path>] [--history <path>] [--runtime] [--verbose-runtime] [--explain] [--no-animations] [--ascii]\n       codex-plugin-doctor audit --installed [filter] [--policy codex-publish|mcp-strict|security] [--security] [--compat] [--json] [--output <path>] [--cache] [--changed]\n       codex-plugin-doctor mcp <path> [--json] [--output <path>]\n       codex-plugin-doctor security <path> [--policy security] [--json|--scorecard]\n       codex-plugin-doctor compat <path> [--all|--client <client>] [--json] [--scorecard] [--output <path>] [--install-preview|--apply --backup]\n       codex-plugin-doctor fix <path> (--dry-run|--interactive --backup|--apply --backup)\n       codex-plugin-doctor history <history.jsonl> [--json] [--fail-on-regression]\n       codex-plugin-doctor doctor [npm <package>|contract|attest <path>|inspector <path>|diff --before <path> --after <path>|recommend <path>|trust <path>|perf <path>|export --bundle <path>|snapshot|clients|--json|--update-check]\n       codex-plugin-doctor init [path] [--template skill-only|mcp-stdio|mcp-http|full-runtime]\n       codex-plugin-doctor init-ci [path]\n       codex-plugin-doctor self-test\n       codex-plugin-doctor list --installed\n       codex-plugin-doctor explain <finding-id>\n       codex-plugin-doctor --version\n\nFirst run:\n       codex-plugin-doctor doctor\n       codex-plugin-doctor self-test\n       codex-plugin-doctor init my-plugin\n       codex-plugin-doctor check . --runtime --explain");
+    io.writeStderr("Usage: codex-plugin-doctor check <path|--installed> [filter] [--policy codex-publish|mcp-strict|security] [--compat] [--json|--markdown|--badge-json|--badge-markdown] [--output <path>] [--history <path>] [--runtime] [--verbose-runtime] [--explain] [--no-animations] [--ascii]\n       codex-plugin-doctor audit --installed [filter] [--policy codex-publish|mcp-strict|security] [--security] [--compat] [--json] [--output <path>] [--cache] [--changed]\n       codex-plugin-doctor mcp <path> [--json] [--output <path>]\n       codex-plugin-doctor security <path> [--policy security] [--json|--scorecard]\n       codex-plugin-doctor compat <path> [--all|--client <client>] [--json] [--scorecard] [--output <path>] [--install-preview|--apply --backup]\n       codex-plugin-doctor fix <path> (--dry-run|--interactive --backup|--apply --backup)\n       codex-plugin-doctor history <history.jsonl> [--json] [--fail-on-regression]\n       codex-plugin-doctor doctor [npm <package>|contract|corpus|attest <path>|inspector <path>|diff --before <path> --after <path>|recommend <path>|trust <path>|perf <path>|export --bundle <path>|snapshot|clients|--json|--update-check]\n       codex-plugin-doctor init [path] [--template skill-only|mcp-stdio|mcp-http|full-runtime]\n       codex-plugin-doctor init-ci [path]\n       codex-plugin-doctor self-test\n       codex-plugin-doctor list --installed\n       codex-plugin-doctor explain <finding-id>\n       codex-plugin-doctor --version\n\nFirst run:\n       codex-plugin-doctor doctor\n       codex-plugin-doctor self-test\n       codex-plugin-doctor init my-plugin\n       codex-plugin-doctor check . --runtime --explain");
 }
 function renderInstalledPlugins(plugins) {
     const lines = [
@@ -250,6 +251,29 @@ export async function runCli(args, io = defaultIo, options = {}) {
                 : renderDoctorOutputContract(contract, { outputPath }));
             return 0;
         }
+        if (maybePath === "corpus") {
+            const jsonOutput = remainingArgs.includes("--json");
+            const outputIndex = remainingArgs.indexOf("--output");
+            const outputPath = outputIndex === -1 ? null : remainingArgs[outputIndex + 1];
+            if (outputIndex !== -1 && (!outputPath || outputPath.startsWith("--"))) {
+                io.writeStderr("Missing path after --output.");
+                return 2;
+            }
+            const report = await buildDoctorValidationCorpusReport({
+                environment: {
+                    env: terminalContext.env,
+                    platform: terminalContext.platform
+                }
+            });
+            const reportJson = renderDoctorValidationCorpusJson(report);
+            if (outputPath) {
+                await writeFile(outputPath, reportJson, "utf8");
+            }
+            io.writeStdout(jsonOutput
+                ? reportJson
+                : renderDoctorValidationCorpusReport(report, { outputPath }));
+            return report.summary.status === "pass" ? 0 : 1;
+        }
         if (maybePath === "attest") {
             const targetPath = remainingArgs[0] && !remainingArgs[0].startsWith("--")
                 ? remainingArgs[0]

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "codex-plugin-doctor",
-  "version": "0.18.0",
+  "version": "0.19.0",
   "description": "CLI-first validator for Codex plugins, skills, and MCP package surfaces with runtime MCP protocol validation.",
   "type": "module",
   "main": "./dist/index.js",