mcprigor 1.0.0-rc.1

package/docs/FILE-EXTENSION.md

@@ -0,0 +1,52 @@
+# MCP Rigor file extension
+
+MCP Rigor plain-language tests use:
+
+```text
+.mcpr
+```
+
+Examples:
+
+```text
+smoke.mcpr
+customer-regression.mcpr
+transport-parity.mcpr
+```
+
+## Why not `.mcp`?
+
+The `.mcp` suffix predates Model Context Protocol and is already associated with other software, including Metrowerks CodeWarrior project files. Reusing it would create ambiguous editor, operating-system, MIME, and tooling associations.
+
+MCP Rigor therefore uses the product-specific `.mcpr` suffix: **MCP Rigor**.
+
+References checked during the decision:
+
+- [FileInfo: MCP file extension](https://fileinfo.com/extension/mcp)
+- [ReviverSoft file-extension registry](https://www.reviversoft.com/file-extensions/mcp)
+
+File-extension registries are not global standards authorities, but existing use is sufficient reason to avoid the collision.
+
+## Migration
+
+Rename files without changing their contents:
+
+```bash
+mv tests/smoke.mcp tests/smoke.mcpr
+mv tests/shared-flows.mcp tests/shared-flows.mcpr
+```
+
+Update flow imports:
+
+```text
+Import flows from "./shared-flows.mcpr"
+```
+
+Update scripts and CI:
+
+```bash
+mcprigor check tests/smoke.mcpr
+mcprigor test tests/smoke.mcpr
+```
+
+MCP Rigor now rejects `.mcp` files with an actionable rename message. YAML and JSON suite support is unchanged.

package/docs/GETTING-STARTED.md

@@ -0,0 +1,126 @@
+# Getting started
+
+This guide takes you from installation to one passing MCP test.
+
+## 1. Install MCP Rigor
+
+You need Node.js 20 or 22.
+
+```bash
+mkdir mcp-acceptance-tests
+cd mcp-acceptance-tests
+npm init -y
+npm install --save-dev mcprigor
+```
+
+## 2. Create a test file
+
+Create `calculator.mcpr`:
+
+```text
+MCP Test 1
+
+Suite: "Calculator acceptance tests"
+Server: node ../calculator-server/dist/server.js
+
+Test: "Adding 20 and 22 gives 42"
+  Call tool "add" with:
+    a: 20
+    b: 22
+
+  Expect "structuredContent.sum" equals 42
+```
+
+Change the `Server` command to the command that starts your MCP server.
+
+For a deployed Streamable HTTP server, use:
+
+```text
+MCP URL: https://qa.example.com/mcp
+```
+
+## 3. Check the wording
+
+```bash
+npx mcprigor check calculator.mcpr
+```
+
+A valid file prints:
+
+```text
+✓ calculator.mcpr looks good and is ready to run
+```
+
+`check` does not connect to the server.
+
+## 4. Run the test
+
+```bash
+npx mcprigor test calculator.mcpr
+```
+
+A passing result looks like:
+
+```text
+MCP Rigor — Calculator acceptance tests
+✓ Adding 20 and 22 gives 42
+
+1 passed, 0 failed, 0 skipped, 0 blocked
+```
+
+## 5. Create a shareable report
+
+```bash
+npx mcprigor test calculator.mcpr --html report.html
+```
+
+Open `report.html` or attach it to a ticket.
+
+## 6. Try the browser workspace
+
+```bash
+npx mcprigor workspace .
+```
+
+Open the printed local URL. Select `calculator.mcpr`, edit it, choose **Validate**, then **Run tests**.
+
+## If you do not know tool names
+
+Create a small target file such as `server.mcpr`:
+
+```text
+MCP Test 1
+Suite: "Server target"
+Server: node ../calculator-server/dist/server.js
+Test: "Connection"
+  Send "ping"
+```
+
+Start guided authoring:
+
+```bash
+npx mcprigor author server.mcpr --out calculator.mcpr
+```
+
+MCP Rigor discovers tools, resources, and prompts and asks what you want to verify.
+
+## Recommended project layout
+
+```text
+mcp-acceptance-tests/
+  package.json
+  tests/
+    smoke.mcpr
+    regression.mcpr
+    data/
+      customers.csv
+  .mcprigor/
+    # generated evidence; normally ignored or stored as CI artifacts
+```
+
+## Next steps
+
+- [Plain-language cookbook](PLAIN-LANGUAGE-COOKBOOK.md)
+- [QA workspace](QA-WORKSPACE.md)
+- [Engineer setup and CI](ENGINEER-SETUP.md)
+- [Troubleshooting](TROUBLESHOOTING.md)

package/docs/GUIDED-AUTHORING.md

@@ -0,0 +1,80 @@
+# Guided test authoring
+
+Use this when you know the behavior to test but do not know the server's tool, resource, prompt, or response field names.
+
+The `author` command creates a plain-language test by connecting to a live MCP server and asking QA-friendly questions.
+
+```bash
+mcprigor author server.mcpr --out tests/search-customer.mcpr
+```
+
+`server.mcpr` can be an existing minimal target file:
+
+```text
+MCP Test 1
+Suite: "QA target"
+Server: node dist/server.js
+Test: "placeholder"
+  Send "ping"
+```
+
+## Authoring flow
+
+1. MCP Rigor connects and discovers tools, resources, and prompts.
+2. Choose whether to call a tool, read a resource, or get a prompt.
+3. Select the operation by its name and description.
+4. Enter required inputs derived from the tool JSON Schema or prompt arguments.
+5. Review and run the request.
+6. Select returned fields to verify.
+7. Choose equality, containment, or existence checks.
+8. Name the test and review the generated source.
+9. Confirm the output file.
+10. Run it normally with `mcprigor test`.
+
+Generated files are deterministic:
+
+- `MCP Test 1` language header
+- sorted input object keys
+- sorted assertions
+- two-space indentation
+- one trailing newline
+- no timestamps or random identifiers
+
+The output is an ordinary `.mcpr` file. It can be edited, reviewed in a pull request, copied, and run without the authoring wizard.
+
+## Input behavior
+
+The wizard understands common JSON Schema input types:
+
+- required and optional object properties
+- strings
+- numbers and integers
+- booleans
+- arrays and objects entered as JSON
+- property defaults
+
+Unsupported or highly dynamic schemas can still be tested by editing the generated `.mcpr` file afterward. Secret values should be represented through environment variables in the target or generated file; never paste reusable credentials into assertions.
+
+## Response field selection
+
+Responses are flattened into stable paths such as:
+
+```text
+$.structuredContent.id
+$.content[0].text
+$.messages[0].content.text
+```
+
+Object keys are sorted and array order is retained. Display previews are capped. Select only stable business fields; prefer `exists` for generated IDs and avoid equality checks on timestamps, tokens, or volatile metadata.
+
+## Automation API
+
+The authoring engine accepts an injected `PromptAdapter`. The included `ScriptedPromptAdapter` lets CI and framework tests run the entire wizard without terminal input. Programmatic APIs are exported from `mcprigor`:
+
+- `authorTest`
+- `renderAuthoredTest`
+- `flattenResponse`
+- `createReadlinePromptAdapter`
+- `ScriptedPromptAdapter`
+
+This separation keeps discovery, prompting, execution, rendering, and file writing independently testable.

package/docs/LANDSCAPE.md

@@ -0,0 +1,81 @@
+# Background: MCP testing landscape
+
+> Non-normative research. This page explains product positioning; it is not needed to use MCP Rigor.
+
+## Executive summary
+
+MCP testing tools currently fall into five categories: interactive inspectors, scriptable probes, official protocol conformance, SDK/framework-local tests, and semantic or security evaluators. The strongest opening for MCP Rigor is a transport-neutral, deterministic, black-box application test runner: **“Playwright for MCP.”**
+
+MCP Rigor should complement the Inspector and official Conformance project rather than suggest it replaces either.
+
+## Landscape
+
+| Tool | Primary role | What it does well | Gap relative to MCP Rigor |
+|---|---|---|---|
+| [MCP Inspector](https://github.com/modelcontextprotocol/inspector) | Official interactive debugger | Explore connections, capabilities, tools, resources, prompts, notifications, and logs | Primarily manual exploration rather than a scenario/assertion/reporting framework |
+| [MCP Inspector CLI](https://github.com/modelcontextprotocol/inspector#cli-mode) | Scriptable probe | Programmatic one-shot MCP interactions | Tests, fixtures, captures, deterministic assertion semantics, and CI reports remain limited or external |
+| [Official MCP Conformance](https://github.com/modelcontextprotocol/conformance) | Protocol compliance/interoperability | Repeatable client/server protocol checks | Evolving protocol suite rather than application-domain regression testing |
+| [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk) | SDK and in-memory testing | Typed clients and in-process transport testing | Language/repository coupled; in-memory tests do not cover deployed transport boundaries |
+| [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) | SDK and in-memory testing | Python integration tests and memory helpers | Language/repository coupled rather than a universal black-box suite |
+| [FastMCP testing](https://gofastmcp.com/patterns/testing) | Python framework testing | Convenient deterministic pytest/in-process testing | Framework-specific and can bypass subprocess, HTTP, auth, proxy, and deployment behavior |
+| [mcptools](https://github.com/f/mcptools) | CLI probing | Shell-friendly list/call/read operations | Assertion model, scenarios, reports, fixtures, and coverage are external |
+| [Promptfoo MCP provider](https://www.promptfoo.dev/docs/providers/mcp/) | Behavioral/agentic evaluation | Tool selection and response-quality evaluation with reports | LLM/model behavior can be probabilistic; not wire or deterministic application testing |
+| [mcp-vibetest](https://github.com/ComposioHQ/mcp-vibetest) | Agentic semantic evaluation | Realistic task-completion checks | Probabilistic and complementary to hard deterministic gates |
+| [mcp-scan](https://github.com/invariantlabs-ai/mcp-scan) | Security scanning | Tool-poisoning and prompt/config risk detection | Specialized security gate, not functional regression testing |
+| [Cisco MCP Scanner](https://github.com/cisco-ai-defense/mcp-scanner) | Security scanning | MCP-focused security analysis | Specialized scanner rather than behavior/protocol test runner |
+| [MCPJam Inspector](https://github.com/MCPJam/inspector) | Interactive workbench | Friendly exploratory MCP development | Primarily interactive; deterministic CI should not be assumed without specific current support |
+
+Project details and release status change quickly. Pin versions and verify current primary documentation before making compatibility claims.
+
+## Positioning
+
+### Category
+
+**Deterministic MCP application testing.**
+
+### One-line message
+
+> MCP Rigor is the open-source test runner for repeatable MCP server behavior across stdio and Streamable HTTP.
+
+### Memorable message
+
+> Playwright for MCP: author once, run the same scenarios locally and in CI.
+
+### Naming rationale
+
+“MCP Rigor” uses established developer-tool terminology, clearly describes the project’s scope, and remains independent from any specific MCP server or vendor brand. Registry and trademark availability should still be verified before publication.
+
+## Durable differentiation
+
+1. **Application workflows, not only probes:** multi-step scenarios, captures, fixtures, setup/teardown, and side-effect verification.
+2. **Real boundaries:** subprocess lifecycle, stdio framing, Streamable HTTP, auth, TLS/proxy, reconnect, and cleanup.
+3. **Stable CI contract:** deterministic exit codes, JSON/JUnit/SARIF, traces, artifacts, filtering, and policy-controlled retries.
+4. **MCP-aware assertions:** tools/resources/prompts, errors, notifications, progress, pagination, cancellation, capability/version matrices, and subscriptions.
+5. **Declarative and programmable:** safe YAML/JSON for most cases plus a typed API and explicit plugin model for advanced projects.
+6. **Layered oracles:** deterministic assertions as hard gates; optional semantic/LLM evaluation as a clearly labeled soft layer.
+7. **Record to regression:** capture an exploratory interaction, sanitize secrets/dynamic values, and generate a checked-in test.
+8. **Evidence, not certification:** run or aggregate exact official conformance profiles without making blanket compliance claims.
+
+## What not to become
+
+- another generic MCP chat client
+- another Inspector UI as the first product
+- an unofficial certification authority
+- a Python- or TypeScript-framework-specific helper
+- an LLM evaluation platform whose core results are nondeterministic
+- a broad security scanner that dilutes functional testing
+
+## Launch wedge
+
+Target teams that have one or more MCP servers in CI and currently use shell scripts, Inspector screenshots, ad hoc SDK tests, or manual checks. The first compelling demo should:
+
+1. run the same calculator test against stdio and HTTP targets;
+2. show a clear structural diff for a broken tool response;
+3. capture a value and reuse it in a later step;
+4. verify a protocol error and timeout;
+5. export JUnit in GitHub Actions;
+6. prove the child process is cleaned up after failure.
+
+## Suggested public narrative
+
+“The MCP ecosystem already has excellent tools to inspect a server and increasingly strong official protocol conformance checks. MCP Rigor addresses a different question: does your server still perform the domain behaviors your users depend on? It turns those behaviors into deterministic scenarios that run over real transports on every pull request.”

package/docs/LANGUAGE-SPEC.md

@@ -0,0 +1,317 @@
+# MCP Test Language 1
+
+Status: compatibility-stable frontend for MCP Rigor 0.12.
+
+The `.mcpr` language and YAML compile to the same `Suite` runtime model. Every user-authored YAML capability has a deterministic plain-language equivalent; YAML remains available for generated files and programmatic integrations rather than being a more powerful test format.
+
+## Design goals
+
+- Readable by QA professionals without programming experience
+- Deterministic: no LLM or fuzzy interpretation
+- Two-space indentation and precise file/line/column diagnostics
+- Safe data blocks delegated to YAML after the outer statement is recognized
+- Backward-compatible with MCP Rigor 0.5 `.mcpr` files
+
+A file may begin with:
+
+```text
+MCP Test 1
+```
+
+Unknown language versions are rejected.
+
+## Lexical rules
+
+- UTF-8 text; LF and CRLF accepted
+- Tabs are rejected
+- Indentation must be a multiple of two spaces
+- `#` starts a comment on a standalone line
+- Declaration keywords are case-insensitive for compatibility
+- Tool, flow, test, variable, and ID values retain their case
+- Strings may use single or double quotes
+- Indented `with:` blocks use safe YAML values
+
+The formal frontend emits located nodes with UTF-16 offsets, line, column, semantic span, indentation, and declaration kind. Compilation currently lowers this document through the compatibility compiler so runtime behavior remains stable.
+
+## Document declarations
+
+```text
+MCP Test 1
+Suite: "Customer tests"
+Server: node dist/server.js
+```
+
+HTTP:
+
+```text
+MCP URL: https://qa.example.com/mcp
+```
+
+Target options use a readable settings block:
+
+```text
+Server options:
+  cwd: ./server
+  env:
+    MODE: test
+```
+
+```text
+Server options:
+  headers:
+    Authorization: "Bearer ${env.QA_TOKEN}"
+```
+
+Parity targets use the same connection grammar:
+
+```text
+Compare target "Local": node server.js
+Compare target "QA": https://qa.example.com/mcp
+
+Target options for "QA":
+  headers:
+    Authorization: "Bearer ${env.QA_TOKEN}"
+```
+
+Suite-level YAML fields have direct equivalents:
+
+```text
+Default timeout: 10 seconds
+Redact: "secret-value", "token-value"
+Snapshots: snapshots.json
+Ignore snapshot paths: "$.createdAt", "$.requestId"
+
+Client behavior:
+  roots:
+    - uri: file:///workspace
+      name: Workspace
+  sampling:
+    model: fixture
+    text: deterministic response
+  elicitation:
+    action: accept
+    content:
+      approved: true
+```
+
+## Imports
+
+```text
+Import flows from "./shared/customer-flows.mcpr"
+```
+
+Only `Flow:` declarations are imported. Imported tests, server declarations, data sources, and top-level configuration never execute. Paths resolve relative to the importer. Canonical path cycles are rejected.
+
+## Flows
+
+```text
+Flow: "Verify addition"
+  Inputs: a, b=1, expected
+
+  Call tool "add" with:
+    a: "${a}"
+    b: "${b}"
+
+  Expect "structuredContent.sum" equals "${expected}"
+```
+
+Inputs without `=` are required. Inputs with defaults are optional. Unknown supplied inputs and missing required inputs are compilation errors. Recursive flows are rejected.
+
+Use a flow:
+
+```text
+Use flow "Verify addition" with:
+  a: 4
+  expected: 5
+```
+
+Flow invocations receive isolated prefixed variables. Caller values are available only when passed through declared inputs.
+
+## Tests and dependencies
+
+```text
+Test: "Create customer"
+  Id: create-customer
+  Require: tools
+```
+
+```text
+Test: "Retrieve customer"
+  Depends on: create-customer
+```
+
+Additional test settings:
+
+```text
+Skip: "Not enabled in this environment"
+
+Variables:
+  tenant: acme
+  retryCount: 3
+
+Require protocol: "2025-06-18"
+```
+
+`Skip` without a reason maps to `skip: true`. IDs are unique. Dependencies form an acyclic graph and must pass before the consumer opens a session.
+
+## Actions
+
+```text
+Call tool "search" with:
+  query: "red shoes"
+
+Read resource "catalog://status"
+
+Get prompt "review" with:
+  topic: "release"
+
+Send "ping"
+```
+
+## Expectations and values
+
+```text
+Expect it succeeds
+Expect an error
+Expect "structuredContent.total" equals 2
+Expect "content[0].text" contains "complete"
+Expect "items" exists
+Expect "items" has 3 items
+```
+
+Full assertion vocabulary:
+
+```text
+Expect "status" does not equal "deleted"
+Expect "count" is a number
+Expect "customerId" matches "^C-[0-9]+$"
+Expect error code -32602
+Expect error message matches "invalid input"
+Expect "structuredContent" matches schema:
+  type: object
+  required: [customerId]
+Expect "structuredContent" matches snapshot "customer" ignoring "$.createdAt"
+```
+
+Equality is type-sensitive. No implicit text/number coercion occurs. Schema blocks use safe YAML only after the deterministic outer statement is recognized.
+
+## Variables, exports, and utilities
+
+```text
+Save "structuredContent.id" as "localId"
+Export "structuredContent.id" as "customerId"
+```
+
+Local values use `${localId}`. Dependency outputs use `${deps.create-customer.customerId}`. Persisted state uses `${state.create-customer.customerId}`.
+
+```text
+Set "normalized" using "lowercase" with:
+  value: "${row.Email}"
+```
+
+## Data iteration
+
+```text
+For each row:
+  | caseId | input | expected |
+  | first  | 2     | 4        |
+```
+
+Plain-language engineered sources:
+
+```text
+Data source: "customers"
+  From CSV "customers.csv"
+  Column "customerId" is string required
+  Column "spend" is number required
+  Column "tier" is string one of gold, silver, bronze
+  Derive "label" as "${customerId}:${tier}"
+  Keep rows where "spend" is greater than 100
+  Sample 25 rows with seed 2025
+  Cache this source
+
+Test: "Active customer"
+  For each row from "customers"
+  Call tool "find_customer" with:
+    customerId: "${row.customerId}"
+```
+
+File forms are `From CSV`, `From JSON`, `From YAML`, and `From Excel ... sheet ...`. Remote sources use `From REST`. Existing structured settings remain accepted inside a `Data source` block for Google Sheets, SQL, custom plugins, joins, and uncommon provider-specific options, so there is no capability gap with YAML. Remote and custom-code safety flags remain mandatory.
+
+See `DATA-AND-REUSE.md` and `DATA-ENGINEERING.md`.
+
+## Setup and cleanup
+
+```text
+Setup:
+Call tool "prepare"
+
+Steps:
+Call tool "execute"
+
+Cleanup:
+Call tool "remove"
+```
+
+Cleanup actions are attempted even after an earlier failure. Cleanup should be idempotent.
+
+## MCP-native actions
+
+```text
+Subscribe to resource "customer://updates"
+Wait for notification "notifications/resources/updated" within 5 seconds
+Unsubscribe from resource "customer://updates"
+Set log level to "debug"
+List all tools
+List all resources
+List all prompts
+List all resource templates
+Get task "task-1"
+List tasks
+Cancel task "task-1"
+```
+
+Progress and cancellation:
+
+```text
+Call tool "import" with progress and cancel after 500 ms with:
+  source: catalog.csv
+```
+
+These compile to the same `native` step objects accepted by YAML.
+
+## Diagnostics
+
+A diagnostic includes a stable code, source location, original line, caret, explanation, and correction:
+
+```text
+MCPLANG102 tests/customer.mcpr:8:1
+
+   Call tool "create"
+^
+
+Indentation must use multiples of two spaces.
+Try: use 0, 2, 4, or 6 leading spaces
+```
+
+Current namespaces:
+
+- `MCPLANG1xx`: lexical/version errors
+- `MCPLANG2xx`: structural errors
+- `MCPLANG3xx`: import errors
+- `MCPLANG4xx`: symbol, flow, and reference errors
+- `MCP-DEP-*`: dependency graph errors
+- `QA-DATA-*` / `MCP-DATA-*`: data compilation/provider errors
+
+## Grammar summary
+
+```ebnf
+document   = version? declaration* ;
+declaration = suite | server | import | flow | dataSource | test ;
+flow       = "Flow:" name flowProperty* statement* ;
+test       = "Test:" name testProperty* statement* ;
+statement  = callTool | readResource | getPrompt | send | expect
+           | save | export | wait | set | useFlow | forEach | section ;
+```
+
+The vocabulary is closed. New behavior requires a documented language-version-compatible statement rather than fuzzy natural-language interpretation.