mcprigor 1.4.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (86) hide show
  1. package/README.md +19 -2
  2. package/dist/audit.d.ts +44 -0
  3. package/dist/audit.d.ts.map +1 -0
  4. package/dist/audit.js +157 -0
  5. package/dist/audit.js.map +1 -0
  6. package/dist/cli.js +243 -10
  7. package/dist/cli.js.map +1 -1
  8. package/dist/composition.d.ts +42 -0
  9. package/dist/composition.d.ts.map +1 -0
  10. package/dist/composition.js +130 -0
  11. package/dist/composition.js.map +1 -0
  12. package/dist/coverage.d.ts +33 -0
  13. package/dist/coverage.d.ts.map +1 -0
  14. package/dist/coverage.js +83 -0
  15. package/dist/coverage.js.map +1 -0
  16. package/dist/export.d.ts +4 -0
  17. package/dist/export.d.ts.map +1 -1
  18. package/dist/export.js +56 -0
  19. package/dist/export.js.map +1 -1
  20. package/dist/index.d.ts +5 -0
  21. package/dist/index.d.ts.map +1 -1
  22. package/dist/index.js +5 -0
  23. package/dist/index.js.map +1 -1
  24. package/dist/language.js +1 -1
  25. package/dist/language.js.map +1 -1
  26. package/dist/loader.d.ts.map +1 -1
  27. package/dist/loader.js +4 -1
  28. package/dist/loader.js.map +1 -1
  29. package/dist/monitor.d.ts +22 -0
  30. package/dist/monitor.d.ts.map +1 -0
  31. package/dist/monitor.js +71 -0
  32. package/dist/monitor.js.map +1 -0
  33. package/dist/perf.d.ts +37 -0
  34. package/dist/perf.d.ts.map +1 -0
  35. package/dist/perf.js +90 -0
  36. package/dist/perf.js.map +1 -0
  37. package/dist/publish.d.ts +19 -0
  38. package/dist/publish.d.ts.map +1 -0
  39. package/dist/publish.js +58 -0
  40. package/dist/publish.js.map +1 -0
  41. package/dist/qa-language.d.ts.map +1 -1
  42. package/dist/qa-language.js +46 -1
  43. package/dist/qa-language.js.map +1 -1
  44. package/dist/reporters.d.ts +3 -1
  45. package/dist/reporters.d.ts.map +1 -1
  46. package/dist/reporters.js +21 -3
  47. package/dist/reporters.js.map +1 -1
  48. package/dist/runner.d.ts.map +1 -1
  49. package/dist/runner.js +31 -15
  50. package/dist/runner.js.map +1 -1
  51. package/dist/schema.d.ts +47 -0
  52. package/dist/schema.d.ts.map +1 -1
  53. package/dist/schema.js +10 -1
  54. package/dist/schema.js.map +1 -1
  55. package/dist/timeline.d.ts +18 -0
  56. package/dist/timeline.d.ts.map +1 -0
  57. package/dist/timeline.js +59 -0
  58. package/dist/timeline.js.map +1 -0
  59. package/dist/types.d.ts +19 -0
  60. package/dist/types.d.ts.map +1 -1
  61. package/dist/version.d.ts +1 -1
  62. package/dist/version.js +1 -1
  63. package/dist/workspace.d.ts.map +1 -1
  64. package/dist/workspace.js +35 -3
  65. package/dist/workspace.js.map +1 -1
  66. package/docs/CLI-REFERENCE.md +113 -1
  67. package/docs/COVERAGE.md +37 -0
  68. package/docs/ENGINEER-SETUP.md +1 -1
  69. package/docs/EVIDENCE.md +11 -0
  70. package/docs/GITHUB-ACTION.md +82 -0
  71. package/docs/LANGUAGE-SPEC.md +28 -7
  72. package/docs/MONITORING.md +62 -0
  73. package/docs/MULTI-SERVER-COMPOSITIONS.md +103 -0
  74. package/docs/PERFORMANCE-GOVERNANCE.md +85 -0
  75. package/docs/PLAIN-LANGUAGE-COOKBOOK.md +28 -0
  76. package/docs/PUBLISHING.md +53 -0
  77. package/docs/QA-WORKSPACE.md +5 -1
  78. package/docs/README.md +7 -0
  79. package/docs/SECURITY-AUDIT.md +77 -0
  80. package/editors/mcpr-language.json +2 -2
  81. package/editors/vscode/package.json +1 -1
  82. package/editors/vscode/syntaxes/mcpr.tmLanguage.json +2 -2
  83. package/package.json +1 -2
  84. package/workspace-assets/app.js +35 -4
  85. package/workspace-assets/index.html +3 -0
  86. package/workspace-assets/style.css +2 -0
@@ -31,6 +31,15 @@ mcprigor check tests/acceptance.mcpr
31
31
 
32
32
  Validates language and configuration without connecting to the server. Alias: `validate`.
33
33
 
34
+ ### `convert`
35
+
36
+ ```bash
37
+ mcprigor convert tests/acceptance.mcpr --out tests/acceptance.yaml
38
+ mcprigor convert tests/acceptance.mcpr --format json # prints to stdout without --out
39
+ ```
40
+
41
+ Emits the equivalent YAML or JSON suite for any input format. The natural language and YAML are in strict parity — both compile to the same suite model — so the converted file validates against the suite schema and runs with identical results. Use it to hand suites to programmatic tooling or to see exactly what a `.mcpr` file means.
42
+
34
43
  ### `test`
35
44
 
36
45
  ```bash
@@ -50,7 +59,8 @@ Common options:
50
59
  --github-annotations Emit ::error/::notice workflow commands
51
60
  (automatic when GITHUB_ACTIONS=true;
52
61
  disable with --no-github-annotations)
53
- --html report.html Write a readable HTML report
62
+ --html report.html Write a readable HTML report with a clickable
63
+ request/response session timeline
54
64
  --json result.json Write structured JSON
55
65
  --junit result.xml Write JUnit XML
56
66
  --evidence DIRECTORY Save a sanitized evidence bundle
@@ -139,6 +149,40 @@ mcprigor record --out draft.mcpr -- node dist/server.js
139
149
 
140
150
  Proxies a live MCP stdio session: your client (an agent, MCP Inspector's CLI mode, or any harness) talks to `mcprigor record` on stdin/stdout, and Rigor forwards everything to the real server while logging each `tools/call` exchange. When the session ends it writes a reviewable `.mcpr` draft — one test per call, with deterministic assertions picked from the actual responses (up to three scalar `structuredContent` leaves, falling back to short text content). No AI: the draft is a literal transcription. Review it, prune it, and run it.
141
151
 
152
+ ## Multi-server compositions
153
+
154
+ Model the MCP fleet an agent mounts together without overloading parity environments:
155
+
156
+ ```text
157
+ MCP Test 1
158
+ Suite: "Checkout fleet"
159
+ Named server "catalog": node services/catalog.js
160
+ Named server "billing": https://qa.example.com/billing/mcp
161
+
162
+ Test: "catalog lookup"
163
+ On server "catalog"
164
+ Call tool "search" with:
165
+ query: "widget"
166
+
167
+ Test: "billing health"
168
+ On server "billing"
169
+ Send "ping"
170
+ ```
171
+
172
+ The YAML/JSON equivalent uses a top-level `servers` mapping and `server` on each test. The legacy `target` remains the default and existing suites are unchanged.
173
+
174
+ Composition governance:
175
+
176
+ ```bash
177
+ mcprigor composition-check fleet.mcpr
178
+ mcprigor composition-discover fleet.mcpr --out mcp.composition.lock.yaml
179
+ mcprigor composition-drift fleet.mcpr --against mcp.composition.lock.yaml --fail-on breaking
180
+ ```
181
+
182
+ The combined lock embeds each server's standard discovery contract, plus cross-server issues and a stable fleet fingerprint. Checks detect duplicate tool names, conflicting tool schemas, duplicate resource URIs/templates, duplicate prompts, server additions/removals, and per-server contract drift.
183
+
184
+ `composition-drift --fail-on` supports `breaking` (default), `potentially-breaking`, `any`, and `none`, matching the single-server drift gate.
185
+
142
186
  ## Project environments
143
187
 
144
188
  Define shared targets once in `mcprigor.config.yaml` (found in the working directory or any parent):
@@ -165,6 +209,74 @@ mcprigor drift suite.mcpr --against mcp.lock.yaml --env prod
165
209
 
166
210
  The selected environment replaces the suite's declared target and is announced in the output. With a `default:` set, plain `mcprigor test suite.mcpr` uses it automatically. `--command`/`--url` overrides still win over the environment when both are given. An environment value can be a command string, a URL string, or a mapping with `server`/`cwd`/`env` (stdio) or `url`/`headers`/`token from` (HTTP).
167
211
 
212
+ ## Deterministic security audit
213
+
214
+ Run the built-in security and permissions probe pack against a suite target:
215
+
216
+ ```bash
217
+ mcprigor audit suite.mcpr --pdf security-audit.pdf --json security-audit.json
218
+ mcprigor audit --url https://qa.example.com/mcp --fail-on high
219
+ ```
220
+
221
+ The default, non-destructive pack probes malformed JSON-RPC parameters, unknown/tool-name spoofing, oversized payloads, and path-traversal resource URIs. It also inventories every advertised tool for prompt-injection and secret-canary testing.
222
+
223
+ Tool calls are never guessed or executed by default. Explicitly allow only reviewed, non-destructive tools:
224
+
225
+ ```bash
226
+ mcprigor audit suite.mcpr --allow-tool search --allow-tool summarize
227
+ ```
228
+
229
+ Allowed tools receive a fixed prompt-injection string containing a deterministic canary. MCP Rigor checks whether the tool followed/reflected the payload or exposed the canary. The canary is redacted from report evidence.
230
+
231
+ Outputs:
232
+
233
+ - terminal or `--markdown`
234
+ - `--json audit.json`
235
+ - `--csv audit.csv`
236
+ - `--pdf audit.pdf` — rich scorecard with grade, severity cards, and finding detail
237
+ - `--fail-on critical|high|medium|low|none` (default: `high`)
238
+
239
+ Scoring deducts 35 for critical, 20 for high, 10 for medium, and 4 for low findings. Skipped probes do not affect the score and are shown with the exact opt-in needed.
240
+
241
+ ## Coverage gate
242
+
243
+ ```bash
244
+ mcprigor coverage suite.mcpr --fail-under 80 --json coverage.json
245
+ ```
246
+
247
+ Discovers the live MCP contract and reports untested tools, resources, resource templates, prompts, input properties, and enum/`oneOf`/`anyOf` branches. `--fail-under` accepts 0–100; `--markdown` produces a pull-request-friendly report. See [Coverage](COVERAGE.md).
248
+
249
+ ## Scheduled monitoring
250
+
251
+ ```bash
252
+ mcprigor monitor prod.mcpr --every 15m --notify https://hooks.example/rigor
253
+ ```
254
+
255
+ Runs an HTTP suite immediately and on a fixed interval, appends every result to trend history, and posts transition webhooks. `--notify-on` accepts `change` (default), `failure`, `recovery`, or `always`; `--max-runs` bounds cron/CI checks. See [Scheduled monitoring](MONITORING.md).
256
+
257
+ ## GitHub Action
258
+
259
+ Use `FusionOnePlatform/mcprigor@v1` to run suites, contract drift, flaky warnings, job summaries, and an update-in-place pull-request comment. See [GitHub Action](GITHUB-ACTION.md).
260
+
261
+ ## Shareable hosted reports
262
+
263
+ ```bash
264
+ mcprigor publish suite.mcpr --site your-netlify-site
265
+ mcprigor publish suite.mcpr --out reports/latest
266
+ ```
267
+
268
+ Runs the suite and hosts the HTML report (with the session timeline) at a shareable static URL, or writes a local bundle for any static host. The hosting token comes only from `NETLIFY_AUTH_TOKEN`. See [Shareable hosted reports](PUBLISHING.md).
269
+
270
+ ## Latency budgets and regression gate
271
+
272
+ ```bash
273
+ mcprigor test suite.mcpr --fail-on-regression
274
+ ```
275
+
276
+ - `Expect the call to finish within 800ms` fails one step when the live call exceeds the limit (`MCP-PERF-001`).
277
+ - `Budget: p95 500ms over 20 calls` (suite-wide) and `Budget for "test name": p50 300ms` are judged after every run against recorded history plus the current run; a blown budget fails the run.
278
+ - `--fail-on-regression` compares each passed test against the median of its recent history and fails when a test runs slower than 1.5x baseline (50 ms floor, needs 5 recorded samples). No thresholds to maintain — the baseline is the trend.
279
+
168
280
  ## Export reports (PDF, CSV, JUnit)
169
281
 
170
282
  Every run can be exported in machine- and human-friendly formats:
@@ -0,0 +1,37 @@
1
+ # MCP surface and schema coverage
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ `mcprigor coverage` answers which parts of a live MCP contract have no test evidence.
6
+
7
+ ```bash
8
+ mcprigor coverage tests/catalog.mcpr
9
+ mcprigor coverage tests/catalog.mcpr --fail-under 80 --json reports/coverage.json
10
+ ```
11
+
12
+ MCP Rigor discovers the suite target, compares the live contract with static test steps, and measures these deterministic coverage units:
13
+
14
+ - every advertised tool called at least once;
15
+ - every fixed resource URI read at least once;
16
+ - every resource template matched by a resource read;
17
+ - every prompt requested at least once;
18
+ - input-schema properties supplied by at least one call;
19
+ - each `enum`, `oneOf`, and `anyOf` branch exercised by at least one tool argument set.
20
+
21
+ The score is covered units divided by all discovered units. Surfaces with no discovered items report 100% and do not penalize servers that do not expose that capability.
22
+
23
+ ## CI gate
24
+
25
+ ```bash
26
+ mcprigor coverage tests/catalog.mcpr --fail-under 80
27
+ ```
28
+
29
+ The command exits nonzero when the score is below the threshold. Valid thresholds range from 0 to 100.
30
+
31
+ Use `--markdown` for a pull-request-friendly table and `--json` for dashboards or historical retention.
32
+
33
+ ## What coverage proves
34
+
35
+ Coverage proves that a test suite references observed contract surfaces and supplies arguments that reach structural schema branches. It does not claim semantic correctness, authorization coverage, or execution-path coverage inside the server. Pair it with assertions, contract drift, the security audit, and performance budgets.
36
+
37
+ Coverage is calculated against the live discovered contract so newly added tools or schema options immediately appear as uncovered—even before they break an existing test.
@@ -13,7 +13,7 @@ npm install --save-dev mcprigor
13
13
  For reproducible CI runs, pin an exact version in `package.json` and update it deliberately:
14
14
 
15
15
  ```json
16
- { "devDependencies": { "mcprigor": "1.4.0" } }
16
+ { "devDependencies": { "mcprigor": "1.5.0" } }
17
17
  ```
18
18
 
19
19
  Release notes and tarball checksums for each version are on the [GitHub releases page](https://github.com/FusionOnePlatform/mcprigor/releases).
package/docs/EVIDENCE.md CHANGED
@@ -31,6 +31,17 @@ mcprigor evidence-compare .mcprigor/run-001 .mcprigor/run-002
31
31
 
32
32
  Comparison checks the result and normalized trace fingerprints. Matching fingerprints indicate semantically identical evidence under the current normalization policy.
33
33
 
34
+ ## Interactive session timeline
35
+
36
+ `mcprigor test suite.mcpr --html report.html` embeds a HAR-style session timeline directly in the readable report. Every recorded event appears in order:
37
+
38
+ - connect, close, and diagnostic markers;
39
+ - one row per JSON-RPC call, pairing the request with its response;
40
+ - per-call duration in milliseconds;
41
+ - failed calls flagged in red with the server's error payload.
42
+
43
+ Click any row to expand its captured request parameters and response body. The timeline is self-contained (no network, no external assets) and uses the same redaction as evidence bundles, so secrets never reach the report. `--html` records a trace even without `--evidence`; combine both to keep the raw and normalized JSONL alongside the visual timeline.
44
+
34
45
  ## Events
35
46
 
36
47
  - `session.connect.start`
@@ -0,0 +1,82 @@
1
+ # GitHub Action and pull-request reports
2
+
3
+ > Available on `main`; publish by pinning the next MCP Rigor release tag.
4
+
5
+ The MCP Rigor Action runs deterministic suites, optionally gates contract drift, includes flaky-history warnings, writes a rich job summary, and creates or updates one pull-request comment.
6
+
7
+ ## Workflow
8
+
9
+ ```yaml
10
+ name: MCP Rigor
11
+ on:
12
+ pull_request:
13
+
14
+ permissions:
15
+ contents: read
16
+ pull-requests: write
17
+
18
+ jobs:
19
+ rigor:
20
+ runs-on: ubuntu-latest
21
+ steps:
22
+ - uses: actions/checkout@v4
23
+ - uses: FusionOnePlatform/mcprigor@v1
24
+ with:
25
+ suites: |
26
+ tests/**/*.mcpr
27
+ lock: mcp.lock.yaml
28
+ fail-on: breaking
29
+ ```
30
+
31
+ The comment contains a suite table with pass/fail/skipped totals and durations, the classified contract drift report, flaky-test warnings when history exists, and collapsed failure detail. A stable HTML marker makes subsequent runs update the same comment rather than spamming the PR.
32
+
33
+ ## Inputs
34
+
35
+ | Input | Default | Meaning |
36
+ |---|---|---|
37
+ | `suites` | `tests/**/*.mcpr` | Newline-separated paths or glob patterns |
38
+ | `lock` | empty | Optional contract lock checked against the first matched suite |
39
+ | `fail-on` | `breaking` | `breaking`, `potentially-breaking`, `any`, or `none` |
40
+ | `node-version` | `22` | Node.js version used by the Action |
41
+ | `version` | `latest` | MCP Rigor npm version installed for the run |
42
+ | `comment` | `true` | Post/update a pull-request comment |
43
+ | `flaky` | `true` | Add warnings when `.mcprigor/workspace-history.jsonl` exists |
44
+ | `artifact` | `true` | Upload the report, drift markdown, and per-suite JSON as a build artifact |
45
+ | `artifact-name` | `mcprigor-report` | Name of the uploaded artifact |
46
+
47
+ Outputs:
48
+
49
+ - `status` — `passed` or `failed`;
50
+ - `report` — path to the combined Markdown report;
51
+ - `drift-report` — path to the standalone contract drift Markdown (empty without `lock`);
52
+ - `artifact-dir` — directory containing the report, drift markdown, and per-suite JSON results.
53
+
54
+ ## Contract drift as a PR artifact
55
+
56
+ When `lock` is set, the Action runs `mcprigor drift --markdown` and produces two things automatically:
57
+
58
+ 1. a **Contract drift** section inside the PR comment and job summary;
59
+ 2. a standalone `drift.md` artifact recording the suite, lock file, gate, gate result, and the full classified diff.
60
+
61
+ The whole report directory is uploaded with `actions/upload-artifact` (disable with `artifact: false`), so reviewers can download the exact drift evidence for a merge decision, and compliance flows can retain it beyond comment history. Consume `drift-report` from a later step to post the drift anywhere else:
62
+
63
+ ```yaml
64
+ - id: rigor
65
+ uses: FusionOnePlatform/mcprigor@v1
66
+ with:
67
+ suites: tests/**/*.mcpr
68
+ lock: mcp.lock.yaml
69
+ - run: cat "${{ steps.rigor.outputs.drift-report }}"
70
+ if: steps.rigor.outputs.drift-report != ''
71
+ ```
72
+
73
+
74
+ ## Fork safety
75
+
76
+ The Action does not run arbitrary PR comment content. Test targets still come from repository suites, so use normal GitHub approval controls for workflows from untrusted forks. Pull-request comments require `pull-requests: write`; when the token cannot write (common for forks), set `comment: false` and rely on the job summary.
77
+
78
+ Pin a full release tag for the strongest supply-chain reproducibility:
79
+
80
+ ```yaml
81
+ - uses: FusionOnePlatform/mcprigor@v1.5.0
82
+ ```
@@ -1,9 +1,11 @@
1
1
  # MCP Test Language 1
2
2
 
3
- Status: compatibility-stable frontend for MCP Rigor 0.12.
3
+ Status: compatibility-stable frontend for MCP Rigor 1.x.
4
4
 
5
5
  The `.mcpr` language and YAML compile to the same `Suite` runtime model. Every user-authored YAML capability has a deterministic plain-language equivalent; YAML remains available for generated files and programmatic integrations rather than being a more powerful test format.
6
6
 
7
+ Parity is enforced, not aspirational: every `.mcpr` construct compiles to a suite that also validates against the YAML/JSON schema, and `mcprigor convert my-tests.mcpr --out my-tests.yaml` (or `--format json`) emits that equivalent file. The converted suite reloads to the identical suite model and produces the same run results — the regression suite converts each shipped example, reloads it from YAML, and compares runs.
8
+
7
9
  ## Design goals
8
10
 
9
11
  - Readable by QA professionals without programming experience
@@ -62,6 +64,23 @@ Server options:
62
64
  Authorization: "Bearer ${env.QA_TOKEN}"
63
65
  ```
64
66
 
67
+ Multi-server compositions use named server declarations and per-test routing:
68
+
69
+ ```text
70
+ Named server "catalog": node services/catalog.js
71
+ Named server "billing": https://qa.example.com/billing/mcp
72
+
73
+ Server options for "billing":
74
+ headers:
75
+ X-Tenant: qa
76
+
77
+ Test: "catalog lookup"
78
+ On server "catalog"
79
+ Call tool "search"
80
+ ```
81
+
82
+ The YAML equivalent is a top-level `servers` mapping plus `server` on a test. A composition requires at least two named servers, and an unknown `On server` name is rejected during compilation.
83
+
65
84
  Parity targets use the same connection grammar:
66
85
 
67
86
  ```text
@@ -77,6 +96,8 @@ Suite-level YAML fields have direct equivalents:
77
96
 
78
97
  ```text
79
98
  Default timeout: 10 seconds
99
+ Budget: p95 500ms over 20 calls
100
+ Budget for "order lookup": p50 300ms over 20 calls
80
101
  Redact: "secret-value", "token-value"
81
102
  Snapshots: snapshots.json
82
103
  Ignore snapshot paths: "$.createdAt", "$.requestId"
@@ -94,11 +115,7 @@ Client behavior:
94
115
  approved: true
95
116
  ```
96
117
 
97
- ## Imports
98
-
99
- ```
100
-
101
- Per-test scripted responses (override `Client behavior:` for one test):
118
+ Per-test scripted responses override `Client behavior:` for one test:
102
119
 
103
120
  ```text
104
121
  When the server asks for input, respond "accept" with:
@@ -106,7 +123,10 @@ When the server asks for input, respond "accept" with:
106
123
  When the server asks for input, respond "decline"
107
124
  When the server requests sampling, respond "scripted text"
108
125
  ```
109
- text
126
+
127
+ ## Imports
128
+
129
+ ```text
110
130
  Import flows from "./shared/customer-flows.mcpr"
111
131
  ```
112
132
 
@@ -183,6 +203,7 @@ Send "ping"
183
203
  ```text
184
204
  Expect it succeeds
185
205
  Expect an error
206
+ Expect the call to finish within 800ms
186
207
  Expect "structuredContent.total" equals 2
187
208
  Expect "content[0].text" contains "complete"
188
209
  Expect "items" exists
@@ -0,0 +1,62 @@
1
+ # Scheduled production monitoring
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ Turn an HTTP MCP suite into a continuous production check:
6
+
7
+ ```bash
8
+ mcprigor monitor tests/prod.mcpr \
9
+ --every 15m \
10
+ --notify https://alerts.example.com/hooks/mcprigor
11
+ ```
12
+
13
+ The monitor runs immediately, then at the fixed interval until stopped. Durations accept `ms`, `s`, `m`, or `h`, with a minimum interval of one second.
14
+
15
+ For operational safety, monitoring requires a Streamable HTTP target. It refuses stdio suites so a long-running process cannot repeatedly spawn local commands by accident.
16
+
17
+ ## Notification policies
18
+
19
+ ```bash
20
+ mcprigor monitor tests/prod.mcpr --every 5m --notify URL --notify-on change
21
+ ```
22
+
23
+ `--notify-on` supports:
24
+
25
+ - `change` (default): first failure, then failure/recovery transitions;
26
+ - `failure`: every failed run;
27
+ - `recovery`: transitions from failed to passed;
28
+ - `always`: every run.
29
+
30
+ Webhook requests are JSON POSTs with a 15-second timeout:
31
+
32
+ ```json
33
+ {
34
+ "source": "mcprigor",
35
+ "event": "monitor.failure",
36
+ "suite": "tests/prod.mcpr",
37
+ "run": 12,
38
+ "status": "failed",
39
+ "startedAt": "2026-08-30T12:00:00.000Z",
40
+ "durationMs": 842,
41
+ "summary": { "passed": 7, "failed": 1, "skipped": 0, "blocked": 0 },
42
+ "failures": [{ "name": "order lookup", "error": "..." }]
43
+ }
44
+ ```
45
+
46
+ A non-2xx webhook response fails the monitor with `MCP-MONITOR-003`; notification loss is never silently ignored.
47
+
48
+ ## History and trends
49
+
50
+ Every monitoring run appends to `.mcprigor/workspace-history.jsonl`, so existing `mcprigor trends`, PDF/CSV exports, flaky detection, latency budgets, and regression analysis include production monitoring evidence.
51
+
52
+ ## Bounded runs
53
+
54
+ For smoke tests, cron jobs, and CI validation:
55
+
56
+ ```bash
57
+ mcprigor monitor tests/prod.mcpr --every 1m --max-runs 1
58
+ ```
59
+
60
+ `SIGINT` and `SIGTERM` stop the interval cleanly after active MCP sessions shut down.
61
+
62
+ Use an authenticated suite target (`headers` or `Token from:`), retain history according to your evidence policy, and send webhooks only to reviewed HTTPS endpoints.
@@ -0,0 +1,103 @@
1
+ # Multi-server compositions
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ Real MCP clients mount several servers together. MCP Rigor compositions test each named server and govern the combined tool, resource, and prompt namespace as one fleet.
6
+
7
+ ## Declare named servers
8
+
9
+ ```text
10
+ MCP Test 1
11
+ Suite: "Checkout fleet"
12
+
13
+ Named server "catalog": node services/catalog.js
14
+ Named server "billing": https://qa.example.com/billing/mcp
15
+
16
+ Server options for "billing":
17
+ Token from: node scripts/get-qa-token.mjs
18
+
19
+ Test: "catalog search works"
20
+ On server "catalog"
21
+ Call tool "search" with:
22
+ query: "widget"
23
+ Expect "structuredContent.total" is a number
24
+
25
+ Test: "billing responds"
26
+ On server "billing"
27
+ Send "ping"
28
+ Expect it succeeds
29
+ ```
30
+
31
+ `On server` routes every action in that test to the selected server. Tests without `On server` continue to use the legacy/default `Server:` or `MCP URL:` target. Cross-test dependencies and exported values continue to work across named servers.
32
+
33
+ YAML parity:
34
+
35
+ ```yaml
36
+ version: 1
37
+ name: Checkout fleet
38
+ target:
39
+ transport: stdio
40
+ command: node
41
+ args: [services/gateway.js]
42
+ servers:
43
+ catalog:
44
+ transport: stdio
45
+ command: node
46
+ args: [services/catalog.js]
47
+ billing:
48
+ transport: streamable-http
49
+ url: https://qa.example.com/billing/mcp
50
+ tests:
51
+ - name: catalog search works
52
+ server: catalog
53
+ steps:
54
+ - tool:
55
+ name: search
56
+ arguments: { query: widget }
57
+ ```
58
+
59
+ Unknown server names fail validation before a test starts.
60
+
61
+ ## Check the live composition
62
+
63
+ ```bash
64
+ mcprigor composition-check tests/fleet.mcpr
65
+ ```
66
+
67
+ The check discovers all named servers and reports:
68
+
69
+ - `MCP-COMP-001`: duplicate tool name;
70
+ - `MCP-COMP-002`: conflicting input/output schemas for the same tool (breaking);
71
+ - `MCP-COMP-003`: duplicate resource URI or URI template (breaking);
72
+ - `MCP-COMP-004`: duplicate prompt name.
73
+
74
+ MCP Rigor reports collisions; it never silently renames or chooses a winning server.
75
+
76
+ ## Create a combined fleet lock
77
+
78
+ ```bash
79
+ mcprigor composition-discover tests/fleet.mcpr \
80
+ --out contracts/checkout.composition.lock.yaml
81
+ ```
82
+
83
+ The lock embeds each named server's ordinary discovery contract, the cross-server issue set, and a stable fleet fingerprint. Volatile discovery timestamps and diagnostics do not change the combined fingerprint. Writes are atomic: all servers must be discovered successfully before the previous lock is replaced.
84
+
85
+ ## Gate fleet drift in CI
86
+
87
+ ```bash
88
+ mcprigor composition-drift tests/fleet.mcpr \
89
+ --against contracts/checkout.composition.lock.yaml \
90
+ --fail-on breaking
91
+ ```
92
+
93
+ The drift report combines:
94
+
95
+ - server additions and removals;
96
+ - each server's tool/resource/prompt contract changes;
97
+ - newly introduced or resolved cross-server conflicts.
98
+
99
+ `--fail-on` accepts `breaking` (default), `potentially-breaking`, `any`, or `none`, matching the single-server drift gate. Add `--json report.json` for machine-readable CI evidence.
100
+
101
+ ## Composition versus parity
102
+
103
+ Use a **composition** when several servers are mounted together and their namespaces interact. Use **transport parity** when the same logical server is exposed through alternate targets such as local stdio and deployed HTTP. The `servers` and `targets` fields remain intentionally separate.
@@ -0,0 +1,85 @@
1
+ # Performance governance
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ MCP Rigor turns MCP latency into a deterministic release gate. It supports immediate per-call limits, percentile budgets over recorded history, and automatic regression detection against each test's historical baseline.
6
+
7
+ ## Limit one call
8
+
9
+ Add a latency assertion after an action:
10
+
11
+ ```text
12
+ Test: "order lookup stays interactive"
13
+ Call tool "find_order" with:
14
+ orderId: "A-1001"
15
+ Expect the call to finish within 800ms
16
+ ```
17
+
18
+ The measured duration covers the live MCP request and response. A call over the limit fails with `MCP-PERF-001` and reports the measured and allowed duration.
19
+
20
+ YAML parity:
21
+
22
+ ```yaml
23
+ - tool:
24
+ name: find_order
25
+ arguments:
26
+ orderId: A-1001
27
+ assert:
28
+ maxDurationMs: 800
29
+ ```
30
+
31
+ ## Set percentile budgets
32
+
33
+ A percentile budget uses successful durations from recent recorded runs plus the current run:
34
+
35
+ ```text
36
+ Budget: p95 500ms over 20 calls
37
+ Budget for "order lookup stays interactive": p50 300ms over 20 calls
38
+ ```
39
+
40
+ - A suite-wide budget applies independently to every test.
41
+ - A named budget applies only to that test.
42
+ - MCP Rigor uses the deterministic nearest-rank percentile.
43
+ - Fewer than three usable samples reports the budget as pending instead of guessing.
44
+ - A measured percentile over budget fails the CLI run.
45
+
46
+ YAML parity:
47
+
48
+ ```yaml
49
+ budgets:
50
+ - test: "*"
51
+ percentile: 95
52
+ maxMs: 500
53
+ window: 20
54
+ - test: order lookup stays interactive
55
+ percentile: 50
56
+ maxMs: 300
57
+ window: 20
58
+ ```
59
+
60
+ ## Fail on regression without maintaining thresholds
61
+
62
+ ```bash
63
+ mcprigor test tests/orders.mcpr --fail-on-regression
64
+ ```
65
+
66
+ For every successful test, MCP Rigor compares the current duration with the median of its latest successful history. The gate requires at least five samples and reports a regression when the current run exceeds both:
67
+
68
+ - 1.5× the historical median; and
69
+ - the historical median plus 50 ms.
70
+
71
+ The absolute floor prevents very small tests from failing because of ordinary scheduler jitter.
72
+
73
+ ## CI example
74
+
75
+ ```yaml
76
+ - run: npx mcprigor test tests/orders.mcpr --fail-on-regression --junit reports/orders.xml
77
+ ```
78
+
79
+ Explicit budgets answer “is this fast enough?” The regression gate answers “did this release make it materially slower?” Teams commonly use both.
80
+
81
+ ## History source
82
+
83
+ CLI, QA workspace, and MCP-server-driven test runs append to `.mcprigor/workspace-history.jsonl`. The same data powers `mcprigor trends`, CSV/PDF trend exports, flaky detection, percentile budgets, and regression baselines.
84
+
85
+ Commit the test and its budget declarations. Treat history as a CI artifact or retained workspace file according to your evidence policy.
@@ -302,6 +302,34 @@ Test: "summaries embed the sampled text"
302
302
 
303
303
  Suite-wide defaults live in a `Client behavior:` block; these per-test lines override them for one test.
304
304
 
305
+ ## Set latency budgets and catch slow releases
306
+
307
+ Fail a single call that takes too long:
308
+
309
+ ```text
310
+ Test: "order lookup is fast"
311
+ Call tool "find_order" with:
312
+ orderId: "A-1001"
313
+ Expect the call to finish within 800ms
314
+ ```
315
+
316
+ Set suite-level budgets measured as percentiles over recorded run history (the same history behind `mcprigor trends`):
317
+
318
+ ```text
319
+ Budget: p95 500ms over 20 calls
320
+ Budget for "order lookup is fast": p50 300ms
321
+ ```
322
+
323
+ Budgets are judged after each `mcprigor test` run and fail the run when a percentile exceeds its budget. Until enough history exists they report as pending instead of guessing.
324
+
325
+ Gate CI on latency regressions against the trend baseline — no budget numbers needed:
326
+
327
+ ```bash
328
+ mcprigor test suite.mcpr --fail-on-regression
329
+ ```
330
+
331
+ A test regresses when it runs slower than 1.5x its historical median (with a 50 ms floor so micro-tests don't trip on jitter).
332
+
305
333
  ## Match a snapshot
306
334
 
307
335
  ```text