mcprigor 1.4.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -2
- package/dist/audit.d.ts +44 -0
- package/dist/audit.d.ts.map +1 -0
- package/dist/audit.js +157 -0
- package/dist/audit.js.map +1 -0
- package/dist/cli.js +243 -10
- package/dist/cli.js.map +1 -1
- package/dist/composition.d.ts +42 -0
- package/dist/composition.d.ts.map +1 -0
- package/dist/composition.js +130 -0
- package/dist/composition.js.map +1 -0
- package/dist/coverage.d.ts +33 -0
- package/dist/coverage.d.ts.map +1 -0
- package/dist/coverage.js +83 -0
- package/dist/coverage.js.map +1 -0
- package/dist/export.d.ts +4 -0
- package/dist/export.d.ts.map +1 -1
- package/dist/export.js +56 -0
- package/dist/export.js.map +1 -1
- package/dist/index.d.ts +5 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +5 -0
- package/dist/index.js.map +1 -1
- package/dist/language.js +1 -1
- package/dist/language.js.map +1 -1
- package/dist/loader.d.ts.map +1 -1
- package/dist/loader.js +4 -1
- package/dist/loader.js.map +1 -1
- package/dist/monitor.d.ts +22 -0
- package/dist/monitor.d.ts.map +1 -0
- package/dist/monitor.js +71 -0
- package/dist/monitor.js.map +1 -0
- package/dist/perf.d.ts +37 -0
- package/dist/perf.d.ts.map +1 -0
- package/dist/perf.js +90 -0
- package/dist/perf.js.map +1 -0
- package/dist/publish.d.ts +19 -0
- package/dist/publish.d.ts.map +1 -0
- package/dist/publish.js +58 -0
- package/dist/publish.js.map +1 -0
- package/dist/qa-language.d.ts.map +1 -1
- package/dist/qa-language.js +46 -1
- package/dist/qa-language.js.map +1 -1
- package/dist/reporters.d.ts +3 -1
- package/dist/reporters.d.ts.map +1 -1
- package/dist/reporters.js +21 -3
- package/dist/reporters.js.map +1 -1
- package/dist/runner.d.ts.map +1 -1
- package/dist/runner.js +31 -15
- package/dist/runner.js.map +1 -1
- package/dist/schema.d.ts +47 -0
- package/dist/schema.d.ts.map +1 -1
- package/dist/schema.js +10 -1
- package/dist/schema.js.map +1 -1
- package/dist/timeline.d.ts +18 -0
- package/dist/timeline.d.ts.map +1 -0
- package/dist/timeline.js +59 -0
- package/dist/timeline.js.map +1 -0
- package/dist/types.d.ts +19 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/dist/workspace.d.ts.map +1 -1
- package/dist/workspace.js +35 -3
- package/dist/workspace.js.map +1 -1
- package/docs/CLI-REFERENCE.md +113 -1
- package/docs/COVERAGE.md +37 -0
- package/docs/ENGINEER-SETUP.md +1 -1
- package/docs/EVIDENCE.md +11 -0
- package/docs/GITHUB-ACTION.md +82 -0
- package/docs/LANGUAGE-SPEC.md +28 -7
- package/docs/MONITORING.md +62 -0
- package/docs/MULTI-SERVER-COMPOSITIONS.md +103 -0
- package/docs/PERFORMANCE-GOVERNANCE.md +85 -0
- package/docs/PLAIN-LANGUAGE-COOKBOOK.md +28 -0
- package/docs/PUBLISHING.md +53 -0
- package/docs/QA-WORKSPACE.md +5 -1
- package/docs/README.md +7 -0
- package/docs/SECURITY-AUDIT.md +77 -0
- package/editors/mcpr-language.json +2 -2
- package/editors/vscode/package.json +1 -1
- package/editors/vscode/syntaxes/mcpr.tmLanguage.json +2 -2
- package/package.json +1 -2
- package/workspace-assets/app.js +35 -4
- package/workspace-assets/index.html +3 -0
- package/workspace-assets/style.css +2 -0
package/docs/CLI-REFERENCE.md
CHANGED
|
@@ -31,6 +31,15 @@ mcprigor check tests/acceptance.mcpr
|
|
|
31
31
|
|
|
32
32
|
Validates language and configuration without connecting to the server. Alias: `validate`.
|
|
33
33
|
|
|
34
|
+
### `convert`
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
mcprigor convert tests/acceptance.mcpr --out tests/acceptance.yaml
|
|
38
|
+
mcprigor convert tests/acceptance.mcpr --format json # prints to stdout without --out
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Emits the equivalent YAML or JSON suite for any input format. The natural language and YAML are in strict parity — both compile to the same suite model — so the converted file validates against the suite schema and runs with identical results. Use it to hand suites to programmatic tooling or to see exactly what a `.mcpr` file means.
|
|
42
|
+
|
|
34
43
|
### `test`
|
|
35
44
|
|
|
36
45
|
```bash
|
|
@@ -50,7 +59,8 @@ Common options:
|
|
|
50
59
|
--github-annotations Emit ::error/::notice workflow commands
|
|
51
60
|
(automatic when GITHUB_ACTIONS=true;
|
|
52
61
|
disable with --no-github-annotations)
|
|
53
|
-
--html report.html Write a readable HTML report
|
|
62
|
+
--html report.html Write a readable HTML report with a clickable
|
|
63
|
+
request/response session timeline
|
|
54
64
|
--json result.json Write structured JSON
|
|
55
65
|
--junit result.xml Write JUnit XML
|
|
56
66
|
--evidence DIRECTORY Save a sanitized evidence bundle
|
|
@@ -139,6 +149,40 @@ mcprigor record --out draft.mcpr -- node dist/server.js
|
|
|
139
149
|
|
|
140
150
|
Proxies a live MCP stdio session: your client (an agent, MCP Inspector's CLI mode, or any harness) talks to `mcprigor record` on stdin/stdout, and Rigor forwards everything to the real server while logging each `tools/call` exchange. When the session ends it writes a reviewable `.mcpr` draft — one test per call, with deterministic assertions picked from the actual responses (up to three scalar `structuredContent` leaves, falling back to short text content). No AI: the draft is a literal transcription. Review it, prune it, and run it.
|
|
141
151
|
|
|
152
|
+
## Multi-server compositions
|
|
153
|
+
|
|
154
|
+
Model the MCP fleet an agent mounts together without overloading parity environments:
|
|
155
|
+
|
|
156
|
+
```text
|
|
157
|
+
MCP Test 1
|
|
158
|
+
Suite: "Checkout fleet"
|
|
159
|
+
Named server "catalog": node services/catalog.js
|
|
160
|
+
Named server "billing": https://qa.example.com/billing/mcp
|
|
161
|
+
|
|
162
|
+
Test: "catalog lookup"
|
|
163
|
+
On server "catalog"
|
|
164
|
+
Call tool "search" with:
|
|
165
|
+
query: "widget"
|
|
166
|
+
|
|
167
|
+
Test: "billing health"
|
|
168
|
+
On server "billing"
|
|
169
|
+
Send "ping"
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
The YAML/JSON equivalent uses a top-level `servers` mapping and `server` on each test. The legacy `target` remains the default and existing suites are unchanged.
|
|
173
|
+
|
|
174
|
+
Composition governance:
|
|
175
|
+
|
|
176
|
+
```bash
|
|
177
|
+
mcprigor composition-check fleet.mcpr
|
|
178
|
+
mcprigor composition-discover fleet.mcpr --out mcp.composition.lock.yaml
|
|
179
|
+
mcprigor composition-drift fleet.mcpr --against mcp.composition.lock.yaml --fail-on breaking
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
The combined lock embeds each server's standard discovery contract, plus cross-server issues and a stable fleet fingerprint. Checks detect duplicate tool names, conflicting tool schemas, duplicate resource URIs/templates, duplicate prompts, server additions/removals, and per-server contract drift.
|
|
183
|
+
|
|
184
|
+
`composition-drift --fail-on` supports `breaking` (default), `potentially-breaking`, `any`, and `none`, matching the single-server drift gate.
|
|
185
|
+
|
|
142
186
|
## Project environments
|
|
143
187
|
|
|
144
188
|
Define shared targets once in `mcprigor.config.yaml` (found in the working directory or any parent):
|
|
@@ -165,6 +209,74 @@ mcprigor drift suite.mcpr --against mcp.lock.yaml --env prod
|
|
|
165
209
|
|
|
166
210
|
The selected environment replaces the suite's declared target and is announced in the output. With a `default:` set, plain `mcprigor test suite.mcpr` uses it automatically. `--command`/`--url` overrides still win over the environment when both are given. An environment value can be a command string, a URL string, or a mapping with `server`/`cwd`/`env` (stdio) or `url`/`headers`/`token from` (HTTP).
|
|
167
211
|
|
|
212
|
+
## Deterministic security audit
|
|
213
|
+
|
|
214
|
+
Run the built-in security and permissions probe pack against a suite target:
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
mcprigor audit suite.mcpr --pdf security-audit.pdf --json security-audit.json
|
|
218
|
+
mcprigor audit --url https://qa.example.com/mcp --fail-on high
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
The default, non-destructive pack probes malformed JSON-RPC parameters, unknown/tool-name spoofing, oversized payloads, and path-traversal resource URIs. It also inventories every advertised tool for prompt-injection and secret-canary testing.
|
|
222
|
+
|
|
223
|
+
Tool calls are never guessed or executed by default. Explicitly allow only reviewed, non-destructive tools:
|
|
224
|
+
|
|
225
|
+
```bash
|
|
226
|
+
mcprigor audit suite.mcpr --allow-tool search --allow-tool summarize
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
Allowed tools receive a fixed prompt-injection string containing a deterministic canary. MCP Rigor checks whether the tool followed/reflected the payload or exposed the canary. The canary is redacted from report evidence.
|
|
230
|
+
|
|
231
|
+
Outputs:
|
|
232
|
+
|
|
233
|
+
- terminal or `--markdown`
|
|
234
|
+
- `--json audit.json`
|
|
235
|
+
- `--csv audit.csv`
|
|
236
|
+
- `--pdf audit.pdf` — rich scorecard with grade, severity cards, and finding detail
|
|
237
|
+
- `--fail-on critical|high|medium|low|none` (default: `high`)
|
|
238
|
+
|
|
239
|
+
Scoring deducts 35 for critical, 20 for high, 10 for medium, and 4 for low findings. Skipped probes do not affect the score and are shown with the exact opt-in needed.
|
|
240
|
+
|
|
241
|
+
## Coverage gate
|
|
242
|
+
|
|
243
|
+
```bash
|
|
244
|
+
mcprigor coverage suite.mcpr --fail-under 80 --json coverage.json
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Discovers the live MCP contract and reports untested tools, resources, resource templates, prompts, input properties, and enum/`oneOf`/`anyOf` branches. `--fail-under` accepts 0–100; `--markdown` produces a pull-request-friendly report. See [Coverage](COVERAGE.md).
|
|
248
|
+
|
|
249
|
+
## Scheduled monitoring
|
|
250
|
+
|
|
251
|
+
```bash
|
|
252
|
+
mcprigor monitor prod.mcpr --every 15m --notify https://hooks.example/rigor
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
Runs an HTTP suite immediately and on a fixed interval, appends every result to trend history, and posts transition webhooks. `--notify-on` accepts `change` (default), `failure`, `recovery`, or `always`; `--max-runs` bounds cron/CI checks. See [Scheduled monitoring](MONITORING.md).
|
|
256
|
+
|
|
257
|
+
## GitHub Action
|
|
258
|
+
|
|
259
|
+
Use `FusionOnePlatform/mcprigor@v1` to run suites, contract drift, flaky warnings, job summaries, and an update-in-place pull-request comment. See [GitHub Action](GITHUB-ACTION.md).
|
|
260
|
+
|
|
261
|
+
## Shareable hosted reports
|
|
262
|
+
|
|
263
|
+
```bash
|
|
264
|
+
mcprigor publish suite.mcpr --site your-netlify-site
|
|
265
|
+
mcprigor publish suite.mcpr --out reports/latest
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
Runs the suite and hosts the HTML report (with the session timeline) at a shareable static URL, or writes a local bundle for any static host. The hosting token comes only from `NETLIFY_AUTH_TOKEN`. See [Shareable hosted reports](PUBLISHING.md).
|
|
269
|
+
|
|
270
|
+
## Latency budgets and regression gate
|
|
271
|
+
|
|
272
|
+
```bash
|
|
273
|
+
mcprigor test suite.mcpr --fail-on-regression
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
- `Expect the call to finish within 800ms` fails one step when the live call exceeds the limit (`MCP-PERF-001`).
|
|
277
|
+
- `Budget: p95 500ms over 20 calls` (suite-wide) and `Budget for "test name": p50 300ms` are judged after every run against recorded history plus the current run; a blown budget fails the run.
|
|
278
|
+
- `--fail-on-regression` compares each passed test against the median of its recent history and fails when a test runs slower than 1.5x baseline (50 ms floor, needs 5 recorded samples). No thresholds to maintain — the baseline is the trend.
|
|
279
|
+
|
|
168
280
|
## Export reports (PDF, CSV, JUnit)
|
|
169
281
|
|
|
170
282
|
Every run can be exported in machine- and human-friendly formats:
|
package/docs/COVERAGE.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# MCP surface and schema coverage
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
`mcprigor coverage` answers which parts of a live MCP contract have no test evidence.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
mcprigor coverage tests/catalog.mcpr
|
|
9
|
+
mcprigor coverage tests/catalog.mcpr --fail-under 80 --json reports/coverage.json
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
MCP Rigor discovers the suite target, compares the live contract with static test steps, and measures these deterministic coverage units:
|
|
13
|
+
|
|
14
|
+
- every advertised tool called at least once;
|
|
15
|
+
- every fixed resource URI read at least once;
|
|
16
|
+
- every resource template matched by a resource read;
|
|
17
|
+
- every prompt requested at least once;
|
|
18
|
+
- input-schema properties supplied by at least one call;
|
|
19
|
+
- each `enum`, `oneOf`, and `anyOf` branch exercised by at least one tool argument set.
|
|
20
|
+
|
|
21
|
+
The score is covered units divided by all discovered units. Surfaces with no discovered items report 100% and do not penalize servers that do not expose that capability.
|
|
22
|
+
|
|
23
|
+
## CI gate
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
mcprigor coverage tests/catalog.mcpr --fail-under 80
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
The command exits nonzero when the score is below the threshold. Valid thresholds range from 0 to 100.
|
|
30
|
+
|
|
31
|
+
Use `--markdown` for a pull-request-friendly table and `--json` for dashboards or historical retention.
|
|
32
|
+
|
|
33
|
+
## What coverage proves
|
|
34
|
+
|
|
35
|
+
Coverage proves that a test suite references observed contract surfaces and supplies arguments that reach structural schema branches. It does not claim semantic correctness, authorization coverage, or execution-path coverage inside the server. Pair it with assertions, contract drift, the security audit, and performance budgets.
|
|
36
|
+
|
|
37
|
+
Coverage is calculated against the live discovered contract so newly added tools or schema options immediately appear as uncovered—even before they break an existing test.
|
package/docs/ENGINEER-SETUP.md
CHANGED
|
@@ -13,7 +13,7 @@ npm install --save-dev mcprigor
|
|
|
13
13
|
For reproducible CI runs, pin an exact version in `package.json` and update it deliberately:
|
|
14
14
|
|
|
15
15
|
```json
|
|
16
|
-
{ "devDependencies": { "mcprigor": "1.
|
|
16
|
+
{ "devDependencies": { "mcprigor": "1.5.0" } }
|
|
17
17
|
```
|
|
18
18
|
|
|
19
19
|
Release notes and tarball checksums for each version are on the [GitHub releases page](https://github.com/FusionOnePlatform/mcprigor/releases).
|
package/docs/EVIDENCE.md
CHANGED
|
@@ -31,6 +31,17 @@ mcprigor evidence-compare .mcprigor/run-001 .mcprigor/run-002
|
|
|
31
31
|
|
|
32
32
|
Comparison checks the result and normalized trace fingerprints. Matching fingerprints indicate semantically identical evidence under the current normalization policy.
|
|
33
33
|
|
|
34
|
+
## Interactive session timeline
|
|
35
|
+
|
|
36
|
+
`mcprigor test suite.mcpr --html report.html` embeds a HAR-style session timeline directly in the readable report. Every recorded event appears in order:
|
|
37
|
+
|
|
38
|
+
- connect, close, and diagnostic markers;
|
|
39
|
+
- one row per JSON-RPC call, pairing the request with its response;
|
|
40
|
+
- per-call duration in milliseconds;
|
|
41
|
+
- failed calls flagged in red with the server's error payload.
|
|
42
|
+
|
|
43
|
+
Click any row to expand its captured request parameters and response body. The timeline is self-contained (no network, no external assets) and uses the same redaction as evidence bundles, so secrets never reach the report. `--html` records a trace even without `--evidence`; combine both to keep the raw and normalized JSONL alongside the visual timeline.
|
|
44
|
+
|
|
34
45
|
## Events
|
|
35
46
|
|
|
36
47
|
- `session.connect.start`
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# GitHub Action and pull-request reports
|
|
2
|
+
|
|
3
|
+
> Available on `main`; publish by pinning the next MCP Rigor release tag.
|
|
4
|
+
|
|
5
|
+
The MCP Rigor Action runs deterministic suites, optionally gates contract drift, includes flaky-history warnings, writes a rich job summary, and creates or updates one pull-request comment.
|
|
6
|
+
|
|
7
|
+
## Workflow
|
|
8
|
+
|
|
9
|
+
```yaml
|
|
10
|
+
name: MCP Rigor
|
|
11
|
+
on:
|
|
12
|
+
pull_request:
|
|
13
|
+
|
|
14
|
+
permissions:
|
|
15
|
+
contents: read
|
|
16
|
+
pull-requests: write
|
|
17
|
+
|
|
18
|
+
jobs:
|
|
19
|
+
rigor:
|
|
20
|
+
runs-on: ubuntu-latest
|
|
21
|
+
steps:
|
|
22
|
+
- uses: actions/checkout@v4
|
|
23
|
+
- uses: FusionOnePlatform/mcprigor@v1
|
|
24
|
+
with:
|
|
25
|
+
suites: |
|
|
26
|
+
tests/**/*.mcpr
|
|
27
|
+
lock: mcp.lock.yaml
|
|
28
|
+
fail-on: breaking
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The comment contains a suite table with pass/fail/skipped totals and durations, the classified contract drift report, flaky-test warnings when history exists, and collapsed failure detail. A stable HTML marker makes subsequent runs update the same comment rather than spamming the PR.
|
|
32
|
+
|
|
33
|
+
## Inputs
|
|
34
|
+
|
|
35
|
+
| Input | Default | Meaning |
|
|
36
|
+
|---|---|---|
|
|
37
|
+
| `suites` | `tests/**/*.mcpr` | Newline-separated paths or glob patterns |
|
|
38
|
+
| `lock` | empty | Optional contract lock checked against the first matched suite |
|
|
39
|
+
| `fail-on` | `breaking` | `breaking`, `potentially-breaking`, `any`, or `none` |
|
|
40
|
+
| `node-version` | `22` | Node.js version used by the Action |
|
|
41
|
+
| `version` | `latest` | MCP Rigor npm version installed for the run |
|
|
42
|
+
| `comment` | `true` | Post/update a pull-request comment |
|
|
43
|
+
| `flaky` | `true` | Add warnings when `.mcprigor/workspace-history.jsonl` exists |
|
|
44
|
+
| `artifact` | `true` | Upload the report, drift markdown, and per-suite JSON as a build artifact |
|
|
45
|
+
| `artifact-name` | `mcprigor-report` | Name of the uploaded artifact |
|
|
46
|
+
|
|
47
|
+
Outputs:
|
|
48
|
+
|
|
49
|
+
- `status` — `passed` or `failed`;
|
|
50
|
+
- `report` — path to the combined Markdown report;
|
|
51
|
+
- `drift-report` — path to the standalone contract drift Markdown (empty without `lock`);
|
|
52
|
+
- `artifact-dir` — directory containing the report, drift markdown, and per-suite JSON results.
|
|
53
|
+
|
|
54
|
+
## Contract drift as a PR artifact
|
|
55
|
+
|
|
56
|
+
When `lock` is set, the Action runs `mcprigor drift --markdown` and produces two things automatically:
|
|
57
|
+
|
|
58
|
+
1. a **Contract drift** section inside the PR comment and job summary;
|
|
59
|
+
2. a standalone `drift.md` artifact recording the suite, lock file, gate, gate result, and the full classified diff.
|
|
60
|
+
|
|
61
|
+
The whole report directory is uploaded with `actions/upload-artifact` (disable with `artifact: false`), so reviewers can download the exact drift evidence for a merge decision, and compliance flows can retain it beyond comment history. Consume `drift-report` from a later step to post the drift anywhere else:
|
|
62
|
+
|
|
63
|
+
```yaml
|
|
64
|
+
- id: rigor
|
|
65
|
+
uses: FusionOnePlatform/mcprigor@v1
|
|
66
|
+
with:
|
|
67
|
+
suites: tests/**/*.mcpr
|
|
68
|
+
lock: mcp.lock.yaml
|
|
69
|
+
- run: cat "${{ steps.rigor.outputs.drift-report }}"
|
|
70
|
+
if: steps.rigor.outputs.drift-report != ''
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
## Fork safety
|
|
75
|
+
|
|
76
|
+
The Action does not run arbitrary PR comment content. Test targets still come from repository suites, so use normal GitHub approval controls for workflows from untrusted forks. Pull-request comments require `pull-requests: write`; when the token cannot write (common for forks), set `comment: false` and rely on the job summary.
|
|
77
|
+
|
|
78
|
+
Pin a full release tag for the strongest supply-chain reproducibility:
|
|
79
|
+
|
|
80
|
+
```yaml
|
|
81
|
+
- uses: FusionOnePlatform/mcprigor@v1.5.0
|
|
82
|
+
```
|
package/docs/LANGUAGE-SPEC.md
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# MCP Test Language 1
|
|
2
2
|
|
|
3
|
-
Status: compatibility-stable frontend for MCP Rigor
|
|
3
|
+
Status: compatibility-stable frontend for MCP Rigor 1.x.
|
|
4
4
|
|
|
5
5
|
The `.mcpr` language and YAML compile to the same `Suite` runtime model. Every user-authored YAML capability has a deterministic plain-language equivalent; YAML remains available for generated files and programmatic integrations rather than being a more powerful test format.
|
|
6
6
|
|
|
7
|
+
Parity is enforced, not aspirational: every `.mcpr` construct compiles to a suite that also validates against the YAML/JSON schema, and `mcprigor convert my-tests.mcpr --out my-tests.yaml` (or `--format json`) emits that equivalent file. The converted suite reloads to the identical suite model and produces the same run results — the regression suite converts each shipped example, reloads it from YAML, and compares runs.
|
|
8
|
+
|
|
7
9
|
## Design goals
|
|
8
10
|
|
|
9
11
|
- Readable by QA professionals without programming experience
|
|
@@ -62,6 +64,23 @@ Server options:
|
|
|
62
64
|
Authorization: "Bearer ${env.QA_TOKEN}"
|
|
63
65
|
```
|
|
64
66
|
|
|
67
|
+
Multi-server compositions use named server declarations and per-test routing:
|
|
68
|
+
|
|
69
|
+
```text
|
|
70
|
+
Named server "catalog": node services/catalog.js
|
|
71
|
+
Named server "billing": https://qa.example.com/billing/mcp
|
|
72
|
+
|
|
73
|
+
Server options for "billing":
|
|
74
|
+
headers:
|
|
75
|
+
X-Tenant: qa
|
|
76
|
+
|
|
77
|
+
Test: "catalog lookup"
|
|
78
|
+
On server "catalog"
|
|
79
|
+
Call tool "search"
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
The YAML equivalent is a top-level `servers` mapping plus `server` on a test. A composition requires at least two named servers, and an unknown `On server` name is rejected during compilation.
|
|
83
|
+
|
|
65
84
|
Parity targets use the same connection grammar:
|
|
66
85
|
|
|
67
86
|
```text
|
|
@@ -77,6 +96,8 @@ Suite-level YAML fields have direct equivalents:
|
|
|
77
96
|
|
|
78
97
|
```text
|
|
79
98
|
Default timeout: 10 seconds
|
|
99
|
+
Budget: p95 500ms over 20 calls
|
|
100
|
+
Budget for "order lookup": p50 300ms over 20 calls
|
|
80
101
|
Redact: "secret-value", "token-value"
|
|
81
102
|
Snapshots: snapshots.json
|
|
82
103
|
Ignore snapshot paths: "$.createdAt", "$.requestId"
|
|
@@ -94,11 +115,7 @@ Client behavior:
|
|
|
94
115
|
approved: true
|
|
95
116
|
```
|
|
96
117
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Per-test scripted responses (override `Client behavior:` for one test):
|
|
118
|
+
Per-test scripted responses override `Client behavior:` for one test:
|
|
102
119
|
|
|
103
120
|
```text
|
|
104
121
|
When the server asks for input, respond "accept" with:
|
|
@@ -106,7 +123,10 @@ When the server asks for input, respond "accept" with:
|
|
|
106
123
|
When the server asks for input, respond "decline"
|
|
107
124
|
When the server requests sampling, respond "scripted text"
|
|
108
125
|
```
|
|
109
|
-
|
|
126
|
+
|
|
127
|
+
## Imports
|
|
128
|
+
|
|
129
|
+
```text
|
|
110
130
|
Import flows from "./shared/customer-flows.mcpr"
|
|
111
131
|
```
|
|
112
132
|
|
|
@@ -183,6 +203,7 @@ Send "ping"
|
|
|
183
203
|
```text
|
|
184
204
|
Expect it succeeds
|
|
185
205
|
Expect an error
|
|
206
|
+
Expect the call to finish within 800ms
|
|
186
207
|
Expect "structuredContent.total" equals 2
|
|
187
208
|
Expect "content[0].text" contains "complete"
|
|
188
209
|
Expect "items" exists
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Scheduled production monitoring
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
Turn an HTTP MCP suite into a continuous production check:
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
mcprigor monitor tests/prod.mcpr \
|
|
9
|
+
--every 15m \
|
|
10
|
+
--notify https://alerts.example.com/hooks/mcprigor
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The monitor runs immediately, then at the fixed interval until stopped. Durations accept `ms`, `s`, `m`, or `h`, with a minimum interval of one second.
|
|
14
|
+
|
|
15
|
+
For operational safety, monitoring requires a Streamable HTTP target. It refuses stdio suites so a long-running process cannot repeatedly spawn local commands by accident.
|
|
16
|
+
|
|
17
|
+
## Notification policies
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
mcprigor monitor tests/prod.mcpr --every 5m --notify URL --notify-on change
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
`--notify-on` supports:
|
|
24
|
+
|
|
25
|
+
- `change` (default): first failure, then failure/recovery transitions;
|
|
26
|
+
- `failure`: every failed run;
|
|
27
|
+
- `recovery`: transitions from failed to passed;
|
|
28
|
+
- `always`: every run.
|
|
29
|
+
|
|
30
|
+
Webhook requests are JSON POSTs with a 15-second timeout:
|
|
31
|
+
|
|
32
|
+
```json
|
|
33
|
+
{
|
|
34
|
+
"source": "mcprigor",
|
|
35
|
+
"event": "monitor.failure",
|
|
36
|
+
"suite": "tests/prod.mcpr",
|
|
37
|
+
"run": 12,
|
|
38
|
+
"status": "failed",
|
|
39
|
+
"startedAt": "2026-08-30T12:00:00.000Z",
|
|
40
|
+
"durationMs": 842,
|
|
41
|
+
"summary": { "passed": 7, "failed": 1, "skipped": 0, "blocked": 0 },
|
|
42
|
+
"failures": [{ "name": "order lookup", "error": "..." }]
|
|
43
|
+
}
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
A non-2xx webhook response fails the monitor with `MCP-MONITOR-003`; notification loss is never silently ignored.
|
|
47
|
+
|
|
48
|
+
## History and trends
|
|
49
|
+
|
|
50
|
+
Every monitoring run appends to `.mcprigor/workspace-history.jsonl`, so existing `mcprigor trends`, PDF/CSV exports, flaky detection, latency budgets, and regression analysis include production monitoring evidence.
|
|
51
|
+
|
|
52
|
+
## Bounded runs
|
|
53
|
+
|
|
54
|
+
For smoke tests, cron jobs, and CI validation:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
mcprigor monitor tests/prod.mcpr --every 1m --max-runs 1
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
`SIGINT` and `SIGTERM` stop the interval cleanly after active MCP sessions shut down.
|
|
61
|
+
|
|
62
|
+
Use an authenticated suite target (`headers` or `Token from:`), retain history according to your evidence policy, and send webhooks only to reviewed HTTPS endpoints.
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# Multi-server compositions
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
Real MCP clients mount several servers together. MCP Rigor compositions test each named server and govern the combined tool, resource, and prompt namespace as one fleet.
|
|
6
|
+
|
|
7
|
+
## Declare named servers
|
|
8
|
+
|
|
9
|
+
```text
|
|
10
|
+
MCP Test 1
|
|
11
|
+
Suite: "Checkout fleet"
|
|
12
|
+
|
|
13
|
+
Named server "catalog": node services/catalog.js
|
|
14
|
+
Named server "billing": https://qa.example.com/billing/mcp
|
|
15
|
+
|
|
16
|
+
Server options for "billing":
|
|
17
|
+
Token from: node scripts/get-qa-token.mjs
|
|
18
|
+
|
|
19
|
+
Test: "catalog search works"
|
|
20
|
+
On server "catalog"
|
|
21
|
+
Call tool "search" with:
|
|
22
|
+
query: "widget"
|
|
23
|
+
Expect "structuredContent.total" is a number
|
|
24
|
+
|
|
25
|
+
Test: "billing responds"
|
|
26
|
+
On server "billing"
|
|
27
|
+
Send "ping"
|
|
28
|
+
Expect it succeeds
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
`On server` routes every action in that test to the selected server. Tests without `On server` continue to use the legacy/default `Server:` or `MCP URL:` target. Cross-test dependencies and exported values continue to work across named servers.
|
|
32
|
+
|
|
33
|
+
YAML parity:
|
|
34
|
+
|
|
35
|
+
```yaml
|
|
36
|
+
version: 1
|
|
37
|
+
name: Checkout fleet
|
|
38
|
+
target:
|
|
39
|
+
transport: stdio
|
|
40
|
+
command: node
|
|
41
|
+
args: [services/gateway.js]
|
|
42
|
+
servers:
|
|
43
|
+
catalog:
|
|
44
|
+
transport: stdio
|
|
45
|
+
command: node
|
|
46
|
+
args: [services/catalog.js]
|
|
47
|
+
billing:
|
|
48
|
+
transport: streamable-http
|
|
49
|
+
url: https://qa.example.com/billing/mcp
|
|
50
|
+
tests:
|
|
51
|
+
- name: catalog search works
|
|
52
|
+
server: catalog
|
|
53
|
+
steps:
|
|
54
|
+
- tool:
|
|
55
|
+
name: search
|
|
56
|
+
arguments: { query: widget }
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Unknown server names fail validation before a test starts.
|
|
60
|
+
|
|
61
|
+
## Check the live composition
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
mcprigor composition-check tests/fleet.mcpr
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The check discovers all named servers and reports:
|
|
68
|
+
|
|
69
|
+
- `MCP-COMP-001`: duplicate tool name;
|
|
70
|
+
- `MCP-COMP-002`: conflicting input/output schemas for the same tool (breaking);
|
|
71
|
+
- `MCP-COMP-003`: duplicate resource URI or URI template (breaking);
|
|
72
|
+
- `MCP-COMP-004`: duplicate prompt name.
|
|
73
|
+
|
|
74
|
+
MCP Rigor reports collisions; it never silently renames or chooses a winning server.
|
|
75
|
+
|
|
76
|
+
## Create a combined fleet lock
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
mcprigor composition-discover tests/fleet.mcpr \
|
|
80
|
+
--out contracts/checkout.composition.lock.yaml
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The lock embeds each named server's ordinary discovery contract, the cross-server issue set, and a stable fleet fingerprint. Volatile discovery timestamps and diagnostics do not change the combined fingerprint. Writes are atomic: all servers must be discovered successfully before the previous lock is replaced.
|
|
84
|
+
|
|
85
|
+
## Gate fleet drift in CI
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
mcprigor composition-drift tests/fleet.mcpr \
|
|
89
|
+
--against contracts/checkout.composition.lock.yaml \
|
|
90
|
+
--fail-on breaking
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
The drift report combines:
|
|
94
|
+
|
|
95
|
+
- server additions and removals;
|
|
96
|
+
- each server's tool/resource/prompt contract changes;
|
|
97
|
+
- newly introduced or resolved cross-server conflicts.
|
|
98
|
+
|
|
99
|
+
`--fail-on` accepts `breaking` (default), `potentially-breaking`, `any`, or `none`, matching the single-server drift gate. Add `--json report.json` for machine-readable CI evidence.
|
|
100
|
+
|
|
101
|
+
## Composition versus parity
|
|
102
|
+
|
|
103
|
+
Use a **composition** when several servers are mounted together and their namespaces interact. Use **transport parity** when the same logical server is exposed through alternate targets such as local stdio and deployed HTTP. The `servers` and `targets` fields remain intentionally separate.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Performance governance
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
MCP Rigor turns MCP latency into a deterministic release gate. It supports immediate per-call limits, percentile budgets over recorded history, and automatic regression detection against each test's historical baseline.
|
|
6
|
+
|
|
7
|
+
## Limit one call
|
|
8
|
+
|
|
9
|
+
Add a latency assertion after an action:
|
|
10
|
+
|
|
11
|
+
```text
|
|
12
|
+
Test: "order lookup stays interactive"
|
|
13
|
+
Call tool "find_order" with:
|
|
14
|
+
orderId: "A-1001"
|
|
15
|
+
Expect the call to finish within 800ms
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The measured duration covers the live MCP request and response. A call over the limit fails with `MCP-PERF-001` and reports the measured and allowed duration.
|
|
19
|
+
|
|
20
|
+
YAML parity:
|
|
21
|
+
|
|
22
|
+
```yaml
|
|
23
|
+
- tool:
|
|
24
|
+
name: find_order
|
|
25
|
+
arguments:
|
|
26
|
+
orderId: A-1001
|
|
27
|
+
assert:
|
|
28
|
+
maxDurationMs: 800
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Set percentile budgets
|
|
32
|
+
|
|
33
|
+
A percentile budget uses successful durations from recent recorded runs plus the current run:
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Budget: p95 500ms over 20 calls
|
|
37
|
+
Budget for "order lookup stays interactive": p50 300ms over 20 calls
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
- A suite-wide budget applies independently to every test.
|
|
41
|
+
- A named budget applies only to that test.
|
|
42
|
+
- MCP Rigor uses the deterministic nearest-rank percentile.
|
|
43
|
+
- Fewer than three usable samples reports the budget as pending instead of guessing.
|
|
44
|
+
- A measured percentile over budget fails the CLI run.
|
|
45
|
+
|
|
46
|
+
YAML parity:
|
|
47
|
+
|
|
48
|
+
```yaml
|
|
49
|
+
budgets:
|
|
50
|
+
- test: "*"
|
|
51
|
+
percentile: 95
|
|
52
|
+
maxMs: 500
|
|
53
|
+
window: 20
|
|
54
|
+
- test: order lookup stays interactive
|
|
55
|
+
percentile: 50
|
|
56
|
+
maxMs: 300
|
|
57
|
+
window: 20
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Fail on regression without maintaining thresholds
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
mcprigor test tests/orders.mcpr --fail-on-regression
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
For every successful test, MCP Rigor compares the current duration with the median of its latest successful history. The gate requires at least five samples and reports a regression when the current run exceeds both:
|
|
67
|
+
|
|
68
|
+
- 1.5× the historical median; and
|
|
69
|
+
- the historical median plus 50 ms.
|
|
70
|
+
|
|
71
|
+
The absolute floor prevents very small tests from failing because of ordinary scheduler jitter.
|
|
72
|
+
|
|
73
|
+
## CI example
|
|
74
|
+
|
|
75
|
+
```yaml
|
|
76
|
+
- run: npx mcprigor test tests/orders.mcpr --fail-on-regression --junit reports/orders.xml
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Explicit budgets answer “is this fast enough?” The regression gate answers “did this release make it materially slower?” Teams commonly use both.
|
|
80
|
+
|
|
81
|
+
## History source
|
|
82
|
+
|
|
83
|
+
CLI, QA workspace, and MCP-server-driven test runs append to `.mcprigor/workspace-history.jsonl`. The same data powers `mcprigor trends`, CSV/PDF trend exports, flaky detection, percentile budgets, and regression baselines.
|
|
84
|
+
|
|
85
|
+
Commit the test and its budget declarations. Treat history as a CI artifact or retained workspace file according to your evidence policy.
|
|
@@ -302,6 +302,34 @@ Test: "summaries embed the sampled text"
|
|
|
302
302
|
|
|
303
303
|
Suite-wide defaults live in a `Client behavior:` block; these per-test lines override them for one test.
|
|
304
304
|
|
|
305
|
+
## Set latency budgets and catch slow releases
|
|
306
|
+
|
|
307
|
+
Fail a single call that takes too long:
|
|
308
|
+
|
|
309
|
+
```text
|
|
310
|
+
Test: "order lookup is fast"
|
|
311
|
+
Call tool "find_order" with:
|
|
312
|
+
orderId: "A-1001"
|
|
313
|
+
Expect the call to finish within 800ms
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
Set suite-level budgets measured as percentiles over recorded run history (the same history behind `mcprigor trends`):
|
|
317
|
+
|
|
318
|
+
```text
|
|
319
|
+
Budget: p95 500ms over 20 calls
|
|
320
|
+
Budget for "order lookup is fast": p50 300ms
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
Budgets are judged after each `mcprigor test` run and fail the run when a percentile exceeds its budget. Until enough history exists they report as pending instead of guessing.
|
|
324
|
+
|
|
325
|
+
Gate CI on latency regressions against the trend baseline — no budget numbers needed:
|
|
326
|
+
|
|
327
|
+
```bash
|
|
328
|
+
mcprigor test suite.mcpr --fail-on-regression
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
A test regresses when it runs slower than 1.5x its historical median (with a 50 ms floor so micro-tests don't trip on jitter).
|
|
332
|
+
|
|
305
333
|
## Match a snapshot
|
|
306
334
|
|
|
307
335
|
```text
|