mcpspec 1.0.2 → 1.0.3

package/README.md

@@ -4,11 +4,24 @@
 
 <h1 align="center">MCPSpec</h1>
 
-<p align="center"><strong>The complete testing, debugging, and quality platform for MCP servers.</strong></p>
+<p align="center">
+  <strong>The complete testing platform for MCP servers</strong>
+</p>
 
-MCPSpec is Postman for [Model Context Protocol](https://modelcontextprotocol.io) — test collections, interactive inspection, security auditing, performance benchmarking, auto-generated docs, and a quality scoring system. Works from the CLI, in CI/CD, or through a full web UI.
+<p align="center">
+  <a href="https://www.npmjs.com/package/mcpspec"><img src="https://img.shields.io/npm/v/mcpspec.svg?style=flat&colorA=18181B&colorB=3b82f6" alt="npm version" /></a>
+  <a href="https://www.npmjs.com/package/mcpspec"><img src="https://img.shields.io/npm/dm/mcpspec.svg?style=flat&colorA=18181B&colorB=3b82f6" alt="npm downloads" /></a>
+  <a href="https://github.com/light-handle/mcpspec/blob/main/LICENSE"><img src="https://img.shields.io/github/license/light-handle/mcpspec?style=flat&colorA=18181B&colorB=3b82f6" alt="license" /></a>
+  <img src="https://img.shields.io/badge/node-%3E%3D22-3b82f6?style=flat&colorA=18181B" alt="node 22+" />
+</p>
 
-```
+<p align="center">
+  Test collections, interactive inspection, security auditing, performance benchmarking, auto-generated docs, and quality scoring for <a href="https://modelcontextprotocol.io">Model Context Protocol</a> servers. Works from the CLI, in CI/CD, or through a full web UI.
+</p>
+
+---
+
+```bash
 mcpspec test ./collection.yaml        # Run tests
 mcpspec inspect "npx my-server"       # Interactive REPL
 mcpspec audit "npx my-server"         # Security scan
@@ -18,400 +31,82 @@ mcpspec docs "npx my-server"          # Auto-generate documentation
 mcpspec ui                            # Launch web dashboard
 ```
 
----
-
-## Why MCPSpec?
-
-MCP servers expose tools (file access, database queries, API calls) to AI assistants. Before shipping a server, you need to answer:
-
-- **Does it work?** — Do tools return correct results? Do they handle bad input?
-- **Is it safe?** — Can inputs cause path traversal, injection, or information leaks?
-- **Is it fast?** — What's the P95 latency? Can it handle load?
-- **Is it documented?** — Do tools have descriptions and proper schemas?
-
-MCPSpec answers all of these with a single tool.
-
----
-
-## Installation
+## Quick Start
 
 ```bash
+# 1. Install
 npm install -g mcpspec
-```
-
-Requires Node.js 22+.
-
----
-
-## Quick Start
-
-### 1. Initialize a project
 
-```bash
+# 2. Scaffold a project
 mcpspec init --template standard
-```
-
-### 2. Write a test collection
-
-```yaml
-name: Filesystem Server Tests
-server: npx @modelcontextprotocol/server-filesystem /tmp
 
-tests:
-  - name: Read a file
-    call: read_file
-    with:
-      path: /tmp/test.txt
-    expect:
-      - exists: $.content
-
-  - name: Handle missing file
-    call: read_file
-    with:
-      path: /tmp/nonexistent.txt
-    expectError: true
+# 3. Run tests
+mcpspec test
 ```
 
-### 3. Run it
+## Features
 
-```bash
-mcpspec test ./collection.yaml
-```
-
-```
-MCPSpec running Filesystem Server Tests (2 tests)
-
-  ✓ Read a file (124ms)
-  ✓ Handle missing file (89ms)
-
-  Tests:  2 passed (2 total)
-  Time:   0.45s
-```
-
----
+| | Feature | Description |
+|---|---|---|
+| **Test Collections** | YAML-based test suites with 10 assertion types, environments, variables, tags, retries, and parallel execution |
+| **Interactive Inspector** | Connect to any MCP server and explore tools, resources, and schemas in a live REPL |
+| **Security Audit** | Scan for path traversal, injection, auth bypass, resource exhaustion, and info disclosure. Safety filter auto-skips destructive tools; `--dry-run` previews targets |
+| **Benchmarks** | Measure min/max/mean/median/P95/P99 latency and throughput across hundreds of iterations |
+| **MCP Score** | 0-100 quality rating across documentation, schema quality, error handling, responsiveness, and security |
+| **Doc Generator** | Auto-generate Markdown or HTML documentation from server introspection |
+| **Web Dashboard** | Full React UI with server management, test runner, audit viewer, and dark mode |
+| **CI/CD Ready** | JUnit/JSON/TAP reporters, deterministic exit codes, `--ci` mode, GitHub Actions compatible |
 
 ## Commands
 
-### `mcpspec test` — Run Test Collections
-
-```bash
-mcpspec test                              # Uses ./mcpspec.yaml
-mcpspec test ./tests.yaml                 # Specific file
-mcpspec test --env staging                # Use staging variables
-mcpspec test --tag @smoke                 # Filter by tag
-mcpspec test --parallel 4                 # Parallel execution
-mcpspec test --reporter junit --output results.xml  # JUnit for CI
-mcpspec test --baseline main              # Compare against saved baseline
-mcpspec test --watch                      # Re-run on file changes
-mcpspec test --ci                         # CI mode (no colors, strict exit codes)
-```
-
-**Reporters:** `console`, `json`, `junit`, `html`, `tap`
-
-### `mcpspec inspect` — Interactive REPL
-
-```bash
-mcpspec inspect "npx @modelcontextprotocol/server-filesystem /tmp"
-```
-
 | Command | Description |
 |---------|-------------|
-| `.tools` | List all tools |
-| `.resources` | List all resources |
-| `.call <tool> <json>` | Call a tool |
-| `.schema <tool>` | Show input schema |
-| `.info` | Server info |
-| `.exit` | Disconnect |
-
-### `mcpspec audit` — Security Scanner
-
-Scans for 6 categories of vulnerabilities:
-
-```bash
-mcpspec audit "npx my-server"                       # Passive (safe, read-only)
-mcpspec audit "npx my-server" --mode active          # Active (test payloads)
-mcpspec audit "npx my-server" --mode aggressive      # Aggressive probing
-mcpspec audit "npx my-server" --fail-on medium       # Fail CI on medium+ findings
-```
-
-| Rule | What It Detects |
-|------|-----------------|
-| Path Traversal | `../../etc/passwd` style attacks |
-| Input Validation | Missing/malformed input handling |
-| Resource Exhaustion | Crash-inducing large payloads |
-| Auth Bypass | Access control circumvention |
-| Injection | SQL/command injection in tool inputs |
-| Information Disclosure | Leaked paths, stack traces, secrets |
-
-Active and aggressive modes send potentially harmful payloads and require confirmation (or `--acknowledge-risk` for CI).
-
-### `mcpspec bench` — Performance Benchmark
-
-```bash
-mcpspec bench "npx my-server"                        # Default: 100 iterations
-mcpspec bench "npx my-server" --iterations 500        # More iterations
-mcpspec bench "npx my-server" --tool read_file        # Specific tool
-mcpspec bench "npx my-server" --args '{"path":"/tmp/f"}'  # With arguments
-```
-
-Reports min, max, mean, median, P95, P99, standard deviation, and throughput (calls/sec).
-
-### `mcpspec score` — MCP Quality Score
-
-Calculates a 0–100 quality rating:
-
-```bash
-mcpspec score "npx my-server"
-mcpspec score "npx my-server" --badge badge.svg      # Generate SVG badge
-```
-
-```
-  MCP Score
-  ────────────────────────────────────────
-  Documentation    ████████████████████ 100/100
-  Schema Quality   ████████████████████ 100/100
-  Error Handling   ██████████████░░░░░░  70/100
-  Performance      ████████████████░░░░  80/100
-  Security         ████████████████████ 100/100
-
-  Overall: 91/100
-```
-
-| Category (weight) | What It Measures |
-|--------------------|-----------------|
-| Documentation (25%) | % of tools/resources with descriptions |
-| Schema Quality (25%) | Proper `type`, `properties`, `required` in input schemas |
-| Error Handling (20%) | Returns `isError: true` for bad input vs. crashing |
-| Performance (15%) | Median response latency |
-| Security (15%) | Findings from a passive security scan |
-
-The `--badge` flag generates a shields.io-style SVG for your README.
-
-### `mcpspec docs` — Documentation Generator
-
-```bash
-mcpspec docs "npx my-server"                          # Markdown to stdout
-mcpspec docs "npx my-server" --format html             # HTML output
-mcpspec docs "npx my-server" --output ./docs           # Write to directory
-```
-
-Connects to the server, introspects all tools and resources, and generates documentation with tool descriptions, input schemas, and resource tables.
-
-### `mcpspec compare` / `mcpspec baseline` — Regression Detection
-
-```bash
-mcpspec baseline save main                 # Save current run as "main"
-mcpspec baseline list                      # List saved baselines
-mcpspec compare --baseline main            # Compare latest run against baseline
-mcpspec compare <run-id-1> <run-id-2>      # Compare two specific runs
-```
-
-### `mcpspec init` — Project Scaffolding
-
-```bash
-mcpspec init                               # Current directory
-mcpspec init ./my-project                  # Specific directory
-mcpspec init --template minimal            # Minimal starter
-mcpspec init --template standard           # Standard (recommended)
-mcpspec init --template full               # Full with environments
-```
-
-### `mcpspec ui` — Web Dashboard
-
-```bash
-mcpspec ui                                 # Opens localhost:6274
-mcpspec ui --port 8080                     # Custom port
-```
-
-Full web interface with:
-- Server management and connection testing
-- Collection editor with YAML validation
-- Test run history with drill-down
-- Interactive tool inspector
-- Security audit with live progress
-- Performance benchmarking with real-time stats
-- Documentation generator with copy/download
-- MCP Score calculator with category breakdown
-- Dark mode
-
----
-
-## Collection Format
-
-### Simple Format
-
-```yaml
-name: My Tests
-server: npx my-mcp-server
-
-tests:
-  - name: Basic call
-    call: tool_name
-    with:
-      param: value
-    expect:
-      - exists: $.result
-```
-
-### Advanced Format
-
-```yaml
-schemaVersion: "1.0"
-name: Comprehensive Tests
-description: Full test suite
-
-server:
-  transport: stdio
-  command: npx
-  args: ["my-mcp-server"]
-  env:
-    NODE_ENV: test
-
-environments:
-  dev:
-    variables:
-      BASE_PATH: /tmp/dev
-  prod:
-    variables:
-      BASE_PATH: /data
-
-defaultEnvironment: dev
-
-tests:
-  - id: test-1
-    name: Get data
-    tags: [smoke, api]
-    timeout: 5000
-    retries: 2
-    call: get_data
-    with:
-      path: "{{BASE_PATH}}/file.txt"
-    assertions:
-      - type: schema
-      - type: exists
-        path: $.content
-      - type: matches
-        path: $.content
-        pattern: "^Hello"
-      - type: latency
-        maxMs: 1000
-      - type: expression
-        expr: "response.content.length > 0"
-    extract:
-      - name: fileContent
-        path: $.content
-```
-
-### Assertion Types
-
-| Type | Description | Example |
-|------|-------------|---------|
-| `schema` | Response is valid | `type: schema` |
-| `equals` | Exact match | `path: $.id, value: 123` |
-| `contains` | Array/string contains | `path: $.tags, value: "active"` |
-| `exists` | Path exists | `path: $.name` |
-| `matches` | Regex match | `path: $.email, pattern: ".*@.*"` |
-| `type` | Type check | `path: $.count, expected: number` |
-| `length` | Length check | `path: $.items, operator: gt, value: 0` |
-| `latency` | Response time | `maxMs: 1000` |
-| `mimeType` | Content type | `expected: "image/png"` |
-| `expression` | Safe expression | `expr: "response.total > 0"` |
-
-Expressions use [expr-eval](https://github.com/silentmatt/expr-eval) — comparisons, logical operators, property access, and math. No arbitrary code execution.
-
----
-
-## CI/CD Integration
-
-### GitHub Actions
-
-```yaml
-name: MCP Server Tests
-on: [push, pull_request]
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '22'
-
-      - run: npm install -g mcpspec
-
-      - name: Run tests
-        run: mcpspec test --ci --reporter junit --output results.xml
-
-      - name: Security audit
-        run: mcpspec audit "npx my-server" --mode passive --fail-on high
-
-      - uses: mikepenz/action-junit-report@v4
-        if: always()
-        with:
-          report_paths: results.xml
-```
-
-### Exit Codes
-
-| Code | Meaning |
-|------|---------|
-| 0 | Success |
-| 1 | Test failure |
-| 2 | Runtime error |
-| 3 | Configuration error |
-| 4 | Connection error |
-| 5 | Timeout |
-| 6 | Security findings above threshold |
-| 7 | Validation error |
-| 130 | Interrupted (Ctrl+C) |
-
----
+| `mcpspec test [collection]` | Run test collections with `--env`, `--tag`, `--parallel`, `--reporter`, `--watch`, `--ci` |
+| `mcpspec inspect <server>` | Interactive REPL — `.tools`, `.call`, `.schema`, `.resources`, `.info` |
+| `mcpspec audit <server>` | Security scan — `--mode`, `--fail-on`, `--exclude-tools`, `--dry-run` |
+| `mcpspec bench <server>` | Performance benchmark — `--iterations`, `--tool`, `--args` |
+| `mcpspec score <server>` | Quality score (0-100) — `--badge badge.svg` |
+| `mcpspec docs <server>` | Generate docs — `--format markdown\|html`, `--output <dir>` |
+| `mcpspec compare` | Compare test runs or `--baseline <name>` |
+| `mcpspec baseline save <name>` | Save/list baselines for regression detection |
+| `mcpspec init [dir]` | Scaffold project — `--template minimal\|standard\|full` |
+| `mcpspec ui` | Launch web dashboard on `localhost:6274` |
+
+## Community Collections
+
+Pre-built test suites for popular MCP servers in [`examples/collections/servers/`](https://github.com/light-handle/mcpspec/tree/main/examples/collections/servers):
+
+| Collection | Server | Tests |
+|------------|--------|-------|
+| filesystem.yaml | @modelcontextprotocol/server-filesystem | 12 |
+| memory.yaml | @modelcontextprotocol/server-memory | 10 |
+| everything.yaml | @modelcontextprotocol/server-everything | 11 |
+| fetch.yaml | @modelcontextprotocol/server-fetch | 7 |
+| time.yaml | @modelcontextprotocol/server-time | 10 |
+| chrome-devtools.yaml | chrome-devtools-mcp | 11 |
+| github.yaml | @modelcontextprotocol/server-github | 9 |
+
+**70 tests** covering tool discovery, read/write operations, error handling, security edge cases, and latency.
 
 ## Architecture
 
-MCPSpec is a TypeScript monorepo:
-
 | Package | Description |
 |---------|-------------|
 | `@mcpspec/shared` | Types, Zod schemas, constants |
 | `@mcpspec/core` | MCP client, test runner, assertions, security scanner, profiler, doc generator, scorer |
 | `@mcpspec/cli` | 10 CLI commands built with Commander.js |
-| `@mcpspec/server` | Hono HTTP server with REST API + WebSocket for real-time updates |
-| `@mcpspec/ui` | React SPA with TanStack Router, TanStack Query, Tailwind CSS, shadcn/ui |
-
-Key design decisions:
-- **Local-first** — works offline, no account needed, server binds to localhost only
-- **Safe by default** — FAILSAFE YAML parsing, secret masking, process cleanup on SIGINT/SIGTERM
-- **sql.js** for storage — WebAssembly SQLite, no native compilation required
-- **Transports** — stdio, SSE, and streamable-http (SSE/HTTP lazy-loaded for code splitting)
-
----
+| `@mcpspec/server` | Hono HTTP server with REST API + WebSocket |
+| `@mcpspec/ui` | React SPA — TanStack Router, TanStack Query, Tailwind, shadcn/ui |
 
 ## Development
 
 ```bash
 git clone https://github.com/light-handle/mcpspec.git
 cd mcpspec
-pnpm install
-pnpm build
-pnpm test      # 259 tests across core + server
+pnpm install && pnpm build
+pnpm test   # 260 tests across core + server
 ```
 
-Run the CLI locally:
-
-```bash
-node packages/cli/dist/index.js test ./examples/collections/simple.yaml
-```
-
-Launch the UI in dev mode:
-
-```bash
-node packages/cli/dist/index.js ui
-```
-
----
-
 ## License
 
 MIT

package/dist/index.js

@@ -2,6 +2,9 @@
 
 // src/index.ts
 import { Command as Command11 } from "commander";
+import { readFileSync as readFileSync3 } from "fs";
+import { dirname, join as join2 } from "path";
+import { fileURLToPath } from "url";
 
 // src/commands/test.ts
 import { Command } from "commander";
@@ -758,14 +761,16 @@ var SEVERITY_COLORS = {
   low: COLORS2.cyan,
   info: COLORS2.gray
 };
-var auditCommand = new Command7("audit").description("Run security audit on an MCP server").argument("<server>", "Server command or URL").option("--mode <mode>", "Scan mode: passive, active, aggressive", "passive").option("--acknowledge-risk", "Skip confirmation prompt for active/aggressive modes", false).option("--fail-on <severity>", "Fail with exit code 6 if findings at or above severity: info, low, medium, high, critical").option("--rules <rules...>", "Only run specific rules").action(async (serverCommand, options) => {
+var auditCommand = new Command7("audit").description("Run security audit on an MCP server").argument("<server>", "Server command or URL").option("--mode <mode>", "Scan mode: passive, active, aggressive", "passive").option("--acknowledge-risk", "Skip confirmation prompt for active/aggressive modes", false).option("--fail-on <severity>", "Fail with exit code 6 if findings at or above severity: info, low, medium, high, critical").option("--rules <rules...>", "Only run specific rules").option("--exclude-tools <tools...>", "Skip specific tools during scanning").option("--dry-run", "Preview which tools will be scanned without running payloads", false).action(async (serverCommand, options) => {
   let client = null;
   try {
     const mode = options.mode;
     const config = new ScanConfig({
       mode,
       acknowledgeRisk: options.acknowledgeRisk,
-      rules: options.rules
+      rules: options.rules,
+      excludeTools: options.excludeTools,
+      dryRun: options.dryRun
     });
     if (config.requiresConfirmation()) {
       console.log(`
@@ -793,6 +798,24 @@ ${COLORS2.cyan}  Connecting to:${COLORS2.reset} ${serverCommand}`);
     console.log(`${COLORS2.gray}  Scan mode: ${mode} | Rules: ${config.rules.join(", ")}${COLORS2.reset}
 `);
     const scanner = new SecurityScanner();
+    if (config.dryRun) {
+      const preview = await scanner.dryRun(client, config);
+      console.log(`${COLORS2.bold}  Dry Run \u2014 Tools to scan:${COLORS2.reset}
+`);
+      for (const tool of preview.tools) {
+        if (tool.included) {
+          console.log(`    ${COLORS2.green}\u2713${COLORS2.reset} ${tool.name}`);
+        } else {
+          console.log(`    ${COLORS2.yellow}\u2717${COLORS2.reset} ${tool.name} ${COLORS2.gray}(${tool.reason})${COLORS2.reset}`);
+        }
+      }
+      console.log(`
+  ${COLORS2.gray}Rules: ${preview.rules.join(", ")}${COLORS2.reset}`);
+      console.log(`  ${COLORS2.gray}Mode: ${preview.mode}${COLORS2.reset}
+`);
+      await client.disconnect();
+      process.exit(EXIT_CODES6.SUCCESS);
+    }
     const result = await scanner.scan(client, config, {
       onRuleStart: (_ruleId, ruleName) => {
         process.stdout.write(`  ${COLORS2.gray}Running ${ruleName}...${COLORS2.reset}`);
@@ -1062,7 +1085,7 @@ ${COLORS5.bold}  MCP Score${COLORS5.reset}`);
       { name: "Documentation", score: score.categories.documentation },
       { name: "Schema Quality", score: score.categories.schemaQuality },
       { name: "Error Handling", score: score.categories.errorHandling },
-      { name: "Performance", score: score.categories.performance },
+      { name: "Responsiveness", score: score.categories.responsiveness },
       { name: "Security", score: score.categories.security }
     ];
     for (const cat of categories) {
@@ -1094,8 +1117,10 @@ ${COLORS5.bold}  MCP Score${COLORS5.reset}`);
 });
 
 // src/index.ts
+var __cliDir = dirname(fileURLToPath(import.meta.url));
+var pkg = JSON.parse(readFileSync3(join2(__cliDir, "..", "package.json"), "utf-8"));
 var program = new Command11();
-program.name("mcpspec").description("The definitive MCP server testing platform").version("1.0.0");
+program.name("mcpspec").description("The definitive MCP server testing platform").version(pkg.version);
 program.addCommand(testCommand);
 program.addCommand(inspectCommand);
 program.addCommand(initCommand);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcpspec",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "description": "The definitive MCP server testing platform",
   "keywords": [
     "mcp",
@@ -29,9 +29,9 @@
     "@inquirer/prompts": "^7.0.0",
     "commander": "^12.1.0",
     "open": "^10.1.0",
-    "@mcpspec/core": "1.0.2",
-    "@mcpspec/shared": "1.0.2",
-    "@mcpspec/server": "1.0.2"
+    "@mcpspec/core": "1.0.3",
+    "@mcpspec/shared": "1.0.3",
+    "@mcpspec/server": "1.0.3"
   },
   "devDependencies": {
     "tsup": "^8.0.0",