mcprigor 1.0.0 → 1.2.0

package/docs/FILE-EXTENSION.md

@@ -1,52 +0,0 @@
-# MCP Rigor file extension
-
-MCP Rigor plain-language tests use:
-
-```text
-.mcpr
-```
-
-Examples:
-
-```text
-smoke.mcpr
-customer-regression.mcpr
-transport-parity.mcpr
-```
-
-## Why not `.mcp`?
-
-The `.mcp` suffix predates Model Context Protocol and is already associated with other software, including Metrowerks CodeWarrior project files. Reusing it would create ambiguous editor, operating-system, MIME, and tooling associations.
-
-MCP Rigor therefore uses the product-specific `.mcpr` suffix: **MCP Rigor**.
-
-References checked during the decision:
-
-- [FileInfo: MCP file extension](https://fileinfo.com/extension/mcp)
-- [ReviverSoft file-extension registry](https://www.reviversoft.com/file-extensions/mcp)
-
-File-extension registries are not global standards authorities, but existing use is sufficient reason to avoid the collision.
-
-## Migration
-
-Rename files without changing their contents:
-
-```bash
-mv tests/smoke.mcp tests/smoke.mcpr
-mv tests/shared-flows.mcp tests/shared-flows.mcpr
-```
-
-Update flow imports:
-
-```text
-Import flows from "./shared-flows.mcpr"
-```
-
-Update scripts and CI:
-
-```bash
-mcprigor check tests/smoke.mcpr
-mcprigor test tests/smoke.mcpr
-```
-
-MCP Rigor now rejects `.mcp` files with an actionable rename message. YAML and JSON suite support is unchanged.

package/docs/LANDSCAPE.md

@@ -1,81 +0,0 @@
-# Background: MCP testing landscape
-
-> Non-normative research. This page explains product positioning; it is not needed to use MCP Rigor.
-
-## Executive summary
-
-MCP testing tools currently fall into five categories: interactive inspectors, scriptable probes, official protocol conformance, SDK/framework-local tests, and semantic or security evaluators. The strongest opening for MCP Rigor is a transport-neutral, deterministic, black-box application test runner: **“Playwright for MCP.”**
-
-MCP Rigor should complement the Inspector and official Conformance project rather than suggest it replaces either.
-
-## Landscape
-
-| Tool | Primary role | What it does well | Gap relative to MCP Rigor |
-|---|---|---|---|
-| [MCP Inspector](https://github.com/modelcontextprotocol/inspector) | Official interactive debugger | Explore connections, capabilities, tools, resources, prompts, notifications, and logs | Primarily manual exploration rather than a scenario/assertion/reporting framework |
-| [MCP Inspector CLI](https://github.com/modelcontextprotocol/inspector#cli-mode) | Scriptable probe | Programmatic one-shot MCP interactions | Tests, fixtures, captures, deterministic assertion semantics, and CI reports remain limited or external |
-| [Official MCP Conformance](https://github.com/modelcontextprotocol/conformance) | Protocol compliance/interoperability | Repeatable client/server protocol checks | Evolving protocol suite rather than application-domain regression testing |
-| [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk) | SDK and in-memory testing | Typed clients and in-process transport testing | Language/repository coupled; in-memory tests do not cover deployed transport boundaries |
-| [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) | SDK and in-memory testing | Python integration tests and memory helpers | Language/repository coupled rather than a universal black-box suite |
-| [FastMCP testing](https://gofastmcp.com/patterns/testing) | Python framework testing | Convenient deterministic pytest/in-process testing | Framework-specific and can bypass subprocess, HTTP, auth, proxy, and deployment behavior |
-| [mcptools](https://github.com/f/mcptools) | CLI probing | Shell-friendly list/call/read operations | Assertion model, scenarios, reports, fixtures, and coverage are external |
-| [Promptfoo MCP provider](https://www.promptfoo.dev/docs/providers/mcp/) | Behavioral/agentic evaluation | Tool selection and response-quality evaluation with reports | LLM/model behavior can be probabilistic; not wire or deterministic application testing |
-| [mcp-vibetest](https://github.com/ComposioHQ/mcp-vibetest) | Agentic semantic evaluation | Realistic task-completion checks | Probabilistic and complementary to hard deterministic gates |
-| [mcp-scan](https://github.com/invariantlabs-ai/mcp-scan) | Security scanning | Tool-poisoning and prompt/config risk detection | Specialized security gate, not functional regression testing |
-| [Cisco MCP Scanner](https://github.com/cisco-ai-defense/mcp-scanner) | Security scanning | MCP-focused security analysis | Specialized scanner rather than behavior/protocol test runner |
-| [MCPJam Inspector](https://github.com/MCPJam/inspector) | Interactive workbench | Friendly exploratory MCP development | Primarily interactive; deterministic CI should not be assumed without specific current support |
-
-Project details and release status change quickly. Pin versions and verify current primary documentation before making compatibility claims.
-
-## Positioning
-
-### Category
-
-**Deterministic MCP application testing.**
-
-### One-line message
-
-> MCP Rigor is the open-source test runner for repeatable MCP server behavior across stdio and Streamable HTTP.
-
-### Memorable message
-
-> Playwright for MCP: author once, run the same scenarios locally and in CI.
-
-### Naming rationale
-
-“MCP Rigor” uses established developer-tool terminology, clearly describes the project’s scope, and remains independent from any specific MCP server or vendor brand. Registry and trademark availability should still be verified before publication.
-
-## Durable differentiation
-
-1. **Application workflows, not only probes:** multi-step scenarios, captures, fixtures, setup/teardown, and side-effect verification.
-2. **Real boundaries:** subprocess lifecycle, stdio framing, Streamable HTTP, auth, TLS/proxy, reconnect, and cleanup.
-3. **Stable CI contract:** deterministic exit codes, JSON/JUnit/SARIF, traces, artifacts, filtering, and policy-controlled retries.
-4. **MCP-aware assertions:** tools/resources/prompts, errors, notifications, progress, pagination, cancellation, capability/version matrices, and subscriptions.
-5. **Declarative and programmable:** safe YAML/JSON for most cases plus a typed API and explicit plugin model for advanced projects.
-6. **Layered oracles:** deterministic assertions as hard gates; optional semantic/LLM evaluation as a clearly labeled soft layer.
-7. **Record to regression:** capture an exploratory interaction, sanitize secrets/dynamic values, and generate a checked-in test.
-8. **Evidence, not certification:** run or aggregate exact official conformance profiles without making blanket compliance claims.
-
-## What not to become
-
-- another generic MCP chat client
-- another Inspector UI as the first product
-- an unofficial certification authority
-- a Python- or TypeScript-framework-specific helper
-- an LLM evaluation platform whose core results are nondeterministic
-- a broad security scanner that dilutes functional testing
-
-## Launch wedge
-
-Target teams that have one or more MCP servers in CI and currently use shell scripts, Inspector screenshots, ad hoc SDK tests, or manual checks. The first compelling demo should:
-
-1. run the same calculator test against stdio and HTTP targets;
-2. show a clear structural diff for a broken tool response;
-3. capture a value and reuse it in a later step;
-4. verify a protocol error and timeout;
-5. export JUnit in GitHub Actions;
-6. prove the child process is cleaned up after failure.
-
-## Suggested public narrative
-
-“The MCP ecosystem already has excellent tools to inspect a server and increasingly strong official protocol conformance checks. MCP Rigor addresses a different question: does your server still perform the domain behaviors your users depend on? It turns those behaviors into deterministic scenarios that run over real transports on every pull request.”

package/docs/SPECIFICATION.md

@@ -1,178 +0,0 @@
-# MCP Rigor runtime and product specification
-
-> Normative runtime behavior and product requirements. For `.mcpr` syntax, use the [language reference](LANGUAGE-SPEC.md). Everyday users should start with [Getting started](GETTING-STARTED.md).
-
-## 1. Product definition
-
-MCP Rigor is an open-source, deterministic, black-box application test framework for Model Context Protocol servers. Its core promise is:
-
-> Author a behavioral MCP scenario once, then run it repeatably against local subprocesses and deployed HTTP servers in CI.
-
-MCP Rigor complements interactive debuggers and protocol conformance tooling. It does not claim MCP certification.
-
-## 2. Users and jobs
-
-- **MCP server authors:** prevent regressions in tools, resources, prompts, errors, and side effects.
-- **Platform teams:** validate third-party servers before promotion or deployment.
-- **SDK maintainers:** run common behavioral scenarios across transports and versions.
-- **Security and QA teams:** build deterministic negative, authorization, and abuse cases.
-
-Primary job: turn manually verified MCP interactions into reviewable, version-controlled tests with reliable CI outcomes.
-
-## 3. Principles
-
-1. Deterministic checks are hard gates; probabilistic evaluation is optional and separate.
-2. Real transport boundaries matter.
-3. Test files contain data, not arbitrary executable code.
-4. Every wait has a deadline; notification tests use events rather than sleeps.
-5. Reports identify protocol version, transport, capability, skips, and evidence.
-6. Secrets are redacted before any result reaches a reporter.
-7. “Conformance” always names an exact versioned profile and never implies certification.
-
-## 4. MVP scope
-
-### Included
-
-- Node.js 20+, TypeScript, ESM
-- YAML and JSON suites
-- stdio subprocess and Streamable HTTP targets
-- one isolated MCP session per test
-- generic MCP requests
-- captures and variable substitution
-- exact, negative, existence, type, subset, length, and regex assertions
-- expected MCP errors
-- per-step deadlines
-- terminal, JSON, and JUnit output
-- CLI filtering and deterministic exit codes
-- TypeScript library API
-
-### Deferred
-
-- notification queues and progress assertions
-- JSON Schema output matcher
-- snapshots and record/replay
-- setup/teardown fixtures and shared sessions
-- official-conformance adapter and versioned packs
-- OAuth browser flows
-- plugin API, custom reporters, and custom transports
-- fuzzing and security packs
-- retries, parallel/distributed execution, GUI, and LLM judges
-
-## 5. Suite model
-
-A suite has `version`, optional `name`, one target, defaults, and tests. A test contains sequential steps. Each step sends one request, applies assertions, then captures response leaves.
-
-Variables:
-
-- `${captureName}` references a test-local captured value.
-- `${env.NAME}` references an environment value.
-- An exact placeholder preserves its JSON type; interpolation into a larger string converts to text.
-- Variables are resolved only in request parameters in the MVP. Target interpolation and centralized secret redaction are required next.
-
-Session isolation is the default because order-independent tests are easier to reproduce. Shared sessions may be added only as an explicit opt-in.
-
-## 6. Architecture
-
-```text
-CLI
- ├─ suite loader and validator
- ├─ protocol-neutral runner
- │   ├─ variable resolver
- │   ├─ matcher engine
- │   └─ normalized result model
- ├─ MCP SDK session
- │   ├─ stdio adapter
- │   └─ Streamable HTTP adapter
- └─ reporters
-     ├─ terminal
-     ├─ JSON
-     └─ JUnit XML
-```
-
-The runner depends on a small `TestSession` interface so fixture sessions and future raw-wire transports can be substituted. The official TypeScript SDK currently performs initialization, protocol validation, framing, and transport lifecycle management.
-
-A future monorepo can split stable boundaries into `core`, `cli`, `transport-stdio`, `transport-http`, `conformance`, `reporters`, and `plugin-api` packages. Keeping one package until those boundaries stabilize reduces premature API commitments.
-
-## 7. Result and failure semantics
-
-Each run records suite status, UTC start time, duration, observed protocol versions, tests, steps, and summary counts. Failures stop the current test after the first failed step but do not stop later tests.
-
-Exit codes:
-
-- `0`: all selected tests passed
-- `1`: one or more behavioral assertions failed
-- `2`: usage, parsing, or configuration invalid
-- `3`: unexpected infrastructure failure
-
-A later version should explicitly distinguish test failure, transport failure, cleanup failure, skipped capability, not applicable, and inconclusive.
-
-## 8. Conformance roadmap
-
-Versioned profiles should be independently released, for example:
-
-```text
-profiles/2025-03-26/lifecycle
-profiles/2025-03-26/tools
-profiles/2025-03-26/resources
-profiles/2025-03-26/prompts
-profiles/2025-03-26/streamable-http
-```
-
-Checks should include initialization/version negotiation, declared capability behavior, required response/error shapes, ping, cancellation, pagination, progress, logging, subscriptions, unknown methods, invalid parameters, HTTP session IDs, media types, reconnect, and termination.
-
-Report statuses must include pass, fail, capability-skipped, not-applicable, and inconclusive. Integrate official MCP Conformance as a separately pinned run and merge evidence rather than duplicating or rebranding it.
-
-## 9. Security requirements
-
-Before a public beta:
-
-- interpolate target environment and headers without writing secrets into results
-- centralize recursive key/value redaction before reporters
-- cap response, stderr, and report sizes
-- guard regex complexity or document trust assumptions
-- avoid shell command strings; continue using command plus argument arrays
-- ensure bounded graceful subprocess shutdown with force-kill fallback
-- sanitize JUnit and terminal control characters
-- provide opt-in environment inheritance and document its threat model
-
-Security scanners should be integrations. MCP Rigor’s core identity remains functional and protocol-aware testing.
-
-## 10. Release roadmap
-
-### 0.1 — executable MVP
-
-Core runner, YAML/JSON suites, stdio/HTTP through official SDK, deterministic matchers, captures, CLI, JSON/JUnit, tests, and docs.
-
-### 0.2 — CI reliability
-
-Published JSON Schema, complete config validation with Ajv, secret redaction, richer diffs, target interpolation, robust transport classification, GitHub Action, Windows/macOS/Linux CI, and deterministic fixture servers.
-
-### 0.3 — MCP-native behavior
-
-Notification/event queues, progress and cancellation, pagination helpers, capability requirements, resource subscriptions, logging assertions, JSON Schema matcher, traces, snapshots, and record/sanitize/replay.
-
-### 0.4 — ecosystem
-
-Versioned conformance profiles, official conformance aggregation, plugin API, reusable fixture packs, security/fuzz hooks, and adapters for common MCP frameworks.
-
-### 1.0 criteria
-
-Stable suite schema and TypeScript API, two supported protocol revisions, robust process cleanup on three operating systems, transport/auth documentation, migration policy, plugin compatibility policy, reproducible fixture matrix, and no high-severity secret leakage paths.
-
-## 11. Open-source operating model
-
-- Apache-2.0 for broad individual and enterprise use.
-- Public RFCs for schema or plugin-breaking changes.
-- Conventional changesets and a compatibility table per release.
-- `good first issue` fixtures and matcher additions.
-- Governance should clearly document project ownership, maintainership, and vendor neutrality.
-- Reserve project, npm, GitHub organization, and domain names only after registry and trademark checks.
-
-## 12. Success metrics
-
-- time from install to first passing stdio test under ten minutes
-- less than 1% flaky failures in deterministic project fixtures
-- clean teardown across Linux, macOS, and Windows
-- projects running MCP Rigor in CI weekly
-- community-contributed reusable suites and transport fixtures
-- issue resolution time and repeat contributor rate