@yawlabs/mcp-compliance 0.9.2 → 0.11.0

package/README.md

@@ -5,7 +5,7 @@
 [![GitHub stars](https://img.shields.io/github/stars/YawLabs/mcp-compliance)](https://github.com/YawLabs/mcp-compliance/stargazers)
 [![CI](https://github.com/YawLabs/mcp-compliance/actions/workflows/ci.yml/badge.svg)](https://github.com/YawLabs/mcp-compliance/actions/workflows/ci.yml)
 
-**Test any MCP server for spec compliance.** 84-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the [MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). Works against **HTTP endpoints** (`https://my-server.com/mcp`) and **stdio servers** (`npx @modelcontextprotocol/server-filesystem /tmp`) alike. CLI, MCP server, and programmatic API.
+**Test any MCP server for spec compliance.** 88-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the [MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). Works against **HTTP endpoints** (`https://my-server.com/mcp`) and **stdio servers** (`npx @modelcontextprotocol/server-filesystem /tmp`) alike. CLI, MCP server, and programmatic API.
 
 Built and maintained by [Yaw Labs](https://yaw.sh).
 
@@ -128,6 +128,19 @@ On Windows, `npx` and other `.cmd` shims are handled automatically by spawning t
 
 ### CI integration
 
+**GitHub Action** (drop into any `.github/workflows/*.yml`):
+
+```yaml
+- uses: YawLabs/mcp-compliance@v0
+  with:
+    target: 'node ./dist/server.js'   # or a URL like https://my-server.com/mcp
+    format: github                     # ::error / ::warning annotations on the PR
+    strict: 'true'                     # exit non-zero if any required test fails
+    min-grade: 'A'                     # also exit if grade slips
+```
+
+**Manual CLI invocation:**
+
 ```bash
 # GitHub Actions: emits ::error / ::warning annotations inline on the PR
 mcp-compliance test https://my-server.com/mcp --format github --strict
@@ -135,11 +148,30 @@ mcp-compliance test https://my-server.com/mcp --format github --strict
 # Slack/Linear/PR comment: drop the body straight into a comment
 mcp-compliance test https://my-server.com/mcp --format markdown > report.md
 
+# HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages)
+mcp-compliance test https://my-server.com/mcp --format html > report.html
+
 # Block release if grade slips below B
 mcp-compliance test https://my-server.com/mcp --min-grade B
 
 # Preview which tests will run before connecting (handy for --only/--skip authoring)
 mcp-compliance test --list --transport stdio --skip security
+
+# Diff two runs — exit 1 if anything that was passing is now failing
+mcp-compliance test https://my-server.com/mcp --format json > current.json
+mcp-compliance diff baseline.json current.json
+
+# Watch mode for stdio dev loop — re-runs on file changes in cwd
+mcp-compliance test --watch -- node ./dist/server.js
+
+# Latency benchmark
+mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4
+```
+
+**Docker:**
+
+```bash
+docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcp
 ```
 
 ### Scaffold a config
@@ -447,7 +479,7 @@ Restart your MCP client and approve the server when prompted.
 
 ### Tools
 
-- **mcp_compliance_test** — Run the full 84-test suite against a URL or stdio command. Supports auth, custom headers, env vars, timeout, retries, and category/test filtering. Returns grade, score, and detailed results.
+- **mcp_compliance_test** — Run the full 88-test suite against a URL or stdio command. Supports auth, custom headers, env vars, timeout, retries, and category/test filtering. Returns grade, score, and detailed results.
 - **mcp_compliance_badge** — Get the badge markdown/HTML for a server. Supports auth and custom headers.
 - **mcp_compliance_explain** — Explain what a specific test ID checks and why it matters.
 
@@ -468,14 +500,56 @@ const report2 = await runComplianceSuite('https://my-server.com/mcp', {
   retries: 1,
   only: ['transport', 'lifecycle'],
 });
+
+// Live progress for streaming UIs (e.g. server-sent-events to a browser)
+await runComplianceSuite('https://my-server.com/mcp', {
+  onTestComplete: (result) => {
+    // result has the full TestResult: id, name, category, required,
+    // passed, details, durationMs, specRef. Push it to your client.
+    sendToClient(result);
+  },
+});
+```
+
+## Report schema
+
+The JSON output of the test suite is a stable, versioned contract. Every report includes a `schemaVersion` field at the top level. The full JSON Schema lives at [`schemas/report.v1.json`](./schemas/report.v1.json) and is shipped with the npm package.
+
+```jsonc
+{
+  "schemaVersion": "1",        // bumped on breaking changes to the report shape
+  "specVersion": "2025-11-25", // MCP spec version tested against
+  "toolVersion": "0.10.0",     // mcp-compliance version that produced the report
+  "url": "...",
+  "timestamp": "...",
+  "grade": "A",
+  "score": 92.5,
+  "tests": [ ... ],
+  // ...
+}
 ```
 
+Consumer guidance:
+
+- Pin against `schemaVersion`. Reject reports with an unknown version rather than guessing at the shape.
+- The schema validates with any Draft 2020-12 validator (e.g. `ajv`).
+- Within a major version, additions are non-breaking. Renames, removals, or type changes bump the version.
+- Two runs against the same server produce equivalent grade, score, and per-test pass/fail (modulo timings/timestamps).
+
 ## Specification
 
 The compliance testing methodology is published as an open specification:
 
-- **[MCP Compliance Testing Specification](./MCP_COMPLIANCE_SPEC.md)** — test execution model, scoring algorithm, all 84 test rules with pass/fail criteria (CC BY 4.0)
+- **[MCP Compliance Testing Specification](./MCP_COMPLIANCE_SPEC.md)** — test execution model, scoring algorithm, all 88 test rules with pass/fail criteria (CC BY 4.0)
 - **[Machine-readable rule catalog](./mcp-compliance-rules.json)** — JSON Schema-compliant catalog for programmatic consumption
+- **[Why `mcp-compliance`](./docs/WHY.md)** — the problem, existing alternatives, what this tool does differently
+- **[Fixing common failures](./docs/FIXES.md)** — recipes for the most frequent test failures with code snippets
+- **[Spec version migration policy](./docs/SPEC_VERSION_MIGRATION.md)** — how this tool evolves with MCP spec releases
+- **[mcp.hosting external API](./docs/EXT_API.md)** — public submit/retrieve/badge/delete endpoints used by `mcp-compliance badge` and any custom integrations
+- **[Enterprise tier (draft)](./docs/ENTERPRISE.md)** — paid tier structure for organizations with scheduled/private/audit-track compliance needs
+- **[Performance deep-dive](./docs/PERFORMANCE.md)** — why the suite is sequential and what parallel execution would cost
+- **[Spec PR drafts](./docs/spec-prs/)** — our proposed MCP spec clarifications for ambiguous cases we've hit
+- **[mcp.hosting integration spec](./docs/mcp-hosting-integration.md)** — the contract between this engine and the mcp.hosting platform: URL surfaces, data flow, storage model, badge API, leaderboard, router integration
 
 These are complementary to (not competing with) the [official MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). The MCP spec defines what servers must do; this spec defines how to verify compliance.
 

package/dist/{chunk-FKTEFLK5.js → chunk-DGGPE3ZM.js}

@@ -5,7 +5,7 @@ import { request as request2 } from "undici";
 // src/badge.ts
 import { createHash } from "crypto";
 function urlHash(url) {
-  return createHash("sha256").update(url).digest("hex").slice(0, 12);
+  return createHash("sha256").update(url).digest("hex").slice(0, 24);
 }
 function generateBadge(url) {
   const hash = urlHash(url);
@@ -209,9 +209,21 @@ function createStdioTransport(opts) {
   let exited = false;
   let exitCode = null;
   let spawnError = null;
+  let spawned = false;
   const pending = /* @__PURE__ */ new Map();
   let stdoutBuffer = "";
   let stderrBuffer = "";
+  const spawnReady = new Promise((resolve, reject) => {
+    child.once("spawn", () => {
+      spawned = true;
+      resolve();
+    });
+    child.once("error", (err) => {
+      if (!spawned) reject(err);
+    });
+  });
+  spawnReady.catch(() => {
+  });
   child.on("error", (err) => {
     spawnError = err;
     rejectAllPending(err);
@@ -281,6 +293,15 @@ function createStdioTransport(opts) {
     ${snippet.replace(/\n/g, "\n    ")}`;
   }
   async function writeLine(line) {
+    if (!spawned && !spawnError) {
+      try {
+        await spawnReady;
+      } catch (err) {
+        throw new Error(
+          annotateWithStderr(`stdio transport: spawn failed \u2014 ${err instanceof Error ? err.message : String(err)}`)
+        );
+      }
+    }
     if (exited) {
       throw new Error(annotateWithStderr(`stdio transport: child has exited (code ${exitCode})`));
     }
@@ -375,6 +396,7 @@ function createStdioTransport(opts) {
 }
 
 // src/types.ts
+var REPORT_SCHEMA_VERSION = "1";
 var TEST_DEFINITIONS = [
   // ── Transport (13 tests) ─────────────────────────────────────────
   {
@@ -679,6 +701,42 @@ var TEST_DEFINITIONS = [
     description: "Sends a tools/call request with _meta.progressToken and checks if the server sends progress notifications via SSE. Progress support is optional but recommended for long-running operations.",
     recommendation: "When a request includes _meta.progressToken, send notifications/progress events via SSE to report progress. Include progressToken, progress (current), and optionally total fields."
   },
+  {
+    id: "lifecycle-sampling-capability",
+    name: "Sampling capability shape",
+    category: "lifecycle",
+    required: false,
+    specRef: "client/sampling",
+    description: "If the server's initialize response or serverInfo implies it uses client-side sampling (sampling/createMessage), verify the capability declaration shape. Currently this is an advisory shape check \u2014 actually exercising the server\u2192client flow requires a client-side sampling handler and is out of scope.",
+    recommendation: "Sampling is a client capability (the client provides LLM access to the server). Servers don't declare sampling in their own capabilities; they just call sampling/createMessage against clients that advertise it. No server-side action required."
+  },
+  {
+    id: "lifecycle-roots-capability",
+    name: "Roots capability shape",
+    category: "lifecycle",
+    required: false,
+    specRef: "client/roots",
+    description: "Roots (filesystem root paths) is a client capability. This test verifies that if a server sends roots/list requests, it handles gracefully when the client doesn't declare the roots capability (i.e., doesn't crash).",
+    recommendation: "Before calling roots/list, check if the initialized client capabilities include 'roots'. If not, skip the call \u2014 the client can't respond. Never assume roots is available; it's opt-in on the client side."
+  },
+  {
+    id: "lifecycle-elicitation-capability",
+    name: "Elicitation capability shape",
+    category: "lifecycle",
+    required: false,
+    specRef: "client/elicitation",
+    description: "Elicitation (asking the user for structured input mid-operation) is a client capability added in 2025-11-25. This test verifies servers that use elicitation/create handle the case where clients don't support it.",
+    recommendation: "Before calling elicitation/create, check the initialized client capabilities. If elicitation is absent, fall back to a safer default (ask once up-front via tool parameters, or fail cleanly with a clear error)."
+  },
+  {
+    id: "lifecycle-meta-tolerance",
+    name: "Tolerates _meta field on requests",
+    category: "lifecycle",
+    required: false,
+    specRef: "basic/utilities#_meta",
+    description: "Sends a ping with params._meta = { extra: 'value' } and verifies the server doesn't error. The 2025-11-25 spec allows arbitrary _meta on any request; servers should ignore unknown _meta fields gracefully.",
+    recommendation: "Treat the _meta field as opaque \u2014 pass it through your request validator, but do not reject requests for unknown _meta keys. The MCP spec reserves _meta for protocol/transport metadata and forward-compat extensibility."
+  },
   // ── Tools (4 tests) ──────────────────────────────────────────────
   {
     id: "tools-list",
@@ -1391,7 +1449,7 @@ async function runComplianceSuite(target, options = {}) {
           if (attempt < retries) await new Promise((r) => setTimeout(r, 1e3 * (attempt + 1)));
         }
       }
-      tests.push({
+      const result = {
         id,
         name,
         category,
@@ -1400,8 +1458,10 @@ async function runComplianceSuite(target, options = {}) {
         details: lastResult.details,
         durationMs: Date.now() - start,
         specRef: `${SPEC_BASE}/${specRef}`
-      });
+      };
+      tests.push(result);
       options.onProgress?.(id, lastResult.passed, lastResult.details);
+      options.onTestComplete?.(result);
     }
     await test(
       "transport-post",
@@ -1571,7 +1631,11 @@ async function runComplianceSuite(target, options = {}) {
     try {
       initRes = await rpc("initialize", {
         protocolVersion: SPEC_VERSION,
-        capabilities: {},
+        capabilities: {
+          sampling: {},
+          roots: { listChanged: true },
+          elicitation: {}
+        },
         clientInfo: { name: "mcp-compliance", version: TOOL_VERSION }
       });
       const result = initRes?.body?.result;
@@ -1993,6 +2057,69 @@ async function runComplianceSuite(target, options = {}) {
         }
       }
     );
+    await test(
+      "lifecycle-sampling-capability",
+      "Sampling capability shape",
+      "lifecycle",
+      false,
+      "client/sampling",
+      async () => {
+        if (!initRes || initRes.body?.error) {
+          return { passed: false, details: "Server rejected initialize" };
+        }
+        return {
+          passed: true,
+          details: "Server accepted initialize with client sampling capability. Full server\u2192client sampling flow not exercised."
+        };
+      }
+    );
+    await test("lifecycle-roots-capability", "Roots capability shape", "lifecycle", false, "client/roots", async () => {
+      if (!initRes || initRes.body?.error) {
+        return { passed: false, details: "Server rejected initialize" };
+      }
+      return {
+        passed: true,
+        details: "Server accepted initialize. Full server\u2192client roots/list flow not exercised (requires a roots-aware client)."
+      };
+    });
+    await test(
+      "lifecycle-elicitation-capability",
+      "Elicitation capability shape",
+      "lifecycle",
+      false,
+      "client/elicitation",
+      async () => {
+        if (!initRes || initRes.body?.error) {
+          return { passed: false, details: "Server rejected initialize" };
+        }
+        return {
+          passed: true,
+          details: "Server accepted initialize. Full server\u2192client elicitation/create flow not exercised."
+        };
+      }
+    );
+    await test(
+      "lifecycle-meta-tolerance",
+      "Tolerates _meta field on requests",
+      "lifecycle",
+      false,
+      "basic/utilities#_meta",
+      async () => {
+        try {
+          const res = await rpc("ping", { _meta: { "mcp-compliance/probe": "1" } });
+          const body = res.body;
+          if (body.error) {
+            return {
+              passed: false,
+              details: `Server rejected _meta on ping (code ${body.error.code}). _meta should be ignored, not error.`
+            };
+          }
+          return { passed: true, details: "Server accepted ping with arbitrary _meta field" };
+        } catch (err) {
+          return { passed: false, details: `Error: ${err instanceof Error ? err.message : String(err)}` };
+        }
+      }
+    );
     await test(
       "transport-content-type-init",
       "Initialize response has valid content type",
@@ -3875,6 +4002,7 @@ async function runComplianceSuite(target, options = {}) {
     const { score, grade, overall, summary, categories } = computeScore(tests);
     const badge = generateBadge(displayUrl);
     return {
+      schemaVersion: REPORT_SCHEMA_VERSION,
       specVersion: SPEC_VERSION,
       toolVersion: TOOL_VERSION,
       url: displayUrl,
@@ -3902,6 +4030,7 @@ async function runComplianceSuite(target, options = {}) {
 }
 
 export {
+  urlHash,
   generateBadge,
   computeGrade,
   computeScore,