mcprigor 1.4.0 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -4
- package/dist/audit.d.ts +44 -0
- package/dist/audit.d.ts.map +1 -0
- package/dist/audit.js +157 -0
- package/dist/audit.js.map +1 -0
- package/dist/cli.js +243 -10
- package/dist/cli.js.map +1 -1
- package/dist/composition.d.ts +42 -0
- package/dist/composition.d.ts.map +1 -0
- package/dist/composition.js +130 -0
- package/dist/composition.js.map +1 -0
- package/dist/coverage.d.ts +33 -0
- package/dist/coverage.d.ts.map +1 -0
- package/dist/coverage.js +83 -0
- package/dist/coverage.js.map +1 -0
- package/dist/export.d.ts +4 -0
- package/dist/export.d.ts.map +1 -1
- package/dist/export.js +56 -0
- package/dist/export.js.map +1 -1
- package/dist/index.d.ts +5 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +5 -0
- package/dist/index.js.map +1 -1
- package/dist/language.js +1 -1
- package/dist/language.js.map +1 -1
- package/dist/loader.d.ts.map +1 -1
- package/dist/loader.js +4 -1
- package/dist/loader.js.map +1 -1
- package/dist/monitor.d.ts +22 -0
- package/dist/monitor.d.ts.map +1 -0
- package/dist/monitor.js +71 -0
- package/dist/monitor.js.map +1 -0
- package/dist/oauth.d.ts +73 -0
- package/dist/oauth.d.ts.map +1 -0
- package/dist/oauth.js +144 -0
- package/dist/oauth.js.map +1 -0
- package/dist/perf.d.ts +37 -0
- package/dist/perf.d.ts.map +1 -0
- package/dist/perf.js +90 -0
- package/dist/perf.js.map +1 -0
- package/dist/publish.d.ts +19 -0
- package/dist/publish.d.ts.map +1 -0
- package/dist/publish.js +58 -0
- package/dist/publish.js.map +1 -0
- package/dist/qa-language.d.ts.map +1 -1
- package/dist/qa-language.js +75 -3
- package/dist/qa-language.js.map +1 -1
- package/dist/reporters.d.ts +3 -1
- package/dist/reporters.d.ts.map +1 -1
- package/dist/reporters.js +21 -3
- package/dist/reporters.js.map +1 -1
- package/dist/runner.d.ts +7 -0
- package/dist/runner.d.ts.map +1 -1
- package/dist/runner.js +53 -15
- package/dist/runner.js.map +1 -1
- package/dist/schema.d.ts +66 -0
- package/dist/schema.d.ts.map +1 -1
- package/dist/schema.js +21 -1
- package/dist/schema.js.map +1 -1
- package/dist/session.d.ts +2 -1
- package/dist/session.d.ts.map +1 -1
- package/dist/session.js +7 -3
- package/dist/session.js.map +1 -1
- package/dist/timeline.d.ts +18 -0
- package/dist/timeline.d.ts.map +1 -0
- package/dist/timeline.js +59 -0
- package/dist/timeline.js.map +1 -0
- package/dist/types.d.ts +29 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/dist/workspace.d.ts.map +1 -1
- package/dist/workspace.js +35 -3
- package/dist/workspace.js.map +1 -1
- package/docs/AUTHENTICATION.md +213 -0
- package/docs/CLI-REFERENCE.md +123 -2
- package/docs/COVERAGE.md +37 -0
- package/docs/ENGINEER-SETUP.md +1 -1
- package/docs/EVIDENCE.md +11 -0
- package/docs/GETTING-STARTED.md +1 -1
- package/docs/GITHUB-ACTION.md +82 -0
- package/docs/LANGUAGE-SPEC.md +64 -7
- package/docs/MCP-SERVER.md +1 -1
- package/docs/MONITORING.md +62 -0
- package/docs/MULTI-SERVER-COMPOSITIONS.md +103 -0
- package/docs/{PLAIN-LANGUAGE-COOKBOOK.md → NATURAL-LANGUAGE-COOKBOOK.md} +63 -4
- package/docs/PERFORMANCE-GOVERNANCE.md +85 -0
- package/docs/PUBLISHING.md +53 -0
- package/docs/QA-GUIDE.md +2 -2
- package/docs/QA-WORKSPACE.md +5 -1
- package/docs/README.md +10 -2
- package/docs/SECURITY-AUDIT.md +77 -0
- package/editors/mcpr-language.json +2 -2
- package/editors/vscode/package.json +1 -1
- package/editors/vscode/syntaxes/mcpr.tmLanguage.json +2 -2
- package/package.json +1 -2
- package/workspace-assets/app.js +35 -4
- package/workspace-assets/index.html +4 -1
- package/workspace-assets/style.css +2 -0
package/docs/GETTING-STARTED.md
CHANGED
|
@@ -126,7 +126,7 @@ mcp-acceptance-tests/
|
|
|
126
126
|
|
|
127
127
|
## Next steps
|
|
128
128
|
|
|
129
|
-
- [
|
|
129
|
+
- [Natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md)
|
|
130
130
|
- [QA workspace](QA-WORKSPACE.md)
|
|
131
131
|
- [Engineer setup and CI](ENGINEER-SETUP.md)
|
|
132
132
|
- [Troubleshooting](TROUBLESHOOTING.md)
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# GitHub Action and pull-request reports
|
|
2
|
+
|
|
3
|
+
> Available since `v1.5.0`; pin `@v1` for the latest 1.x or `@v1.5.0` for full reproducibility.
|
|
4
|
+
|
|
5
|
+
The MCP Rigor Action runs deterministic suites, optionally gates contract drift, includes flaky-history warnings, writes a rich job summary, and creates or updates one pull-request comment.
|
|
6
|
+
|
|
7
|
+
## Workflow
|
|
8
|
+
|
|
9
|
+
```yaml
|
|
10
|
+
name: MCP Rigor
|
|
11
|
+
on:
|
|
12
|
+
pull_request:
|
|
13
|
+
|
|
14
|
+
permissions:
|
|
15
|
+
contents: read
|
|
16
|
+
pull-requests: write
|
|
17
|
+
|
|
18
|
+
jobs:
|
|
19
|
+
rigor:
|
|
20
|
+
runs-on: ubuntu-latest
|
|
21
|
+
steps:
|
|
22
|
+
- uses: actions/checkout@v4
|
|
23
|
+
- uses: FusionOnePlatform/mcprigor@v1
|
|
24
|
+
with:
|
|
25
|
+
suites: |
|
|
26
|
+
tests/**/*.mcpr
|
|
27
|
+
lock: mcp.lock.yaml
|
|
28
|
+
fail-on: breaking
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The comment contains a suite table with pass/fail/skipped totals and durations, the classified contract drift report, flaky-test warnings when history exists, and collapsed failure detail. A stable HTML marker makes subsequent runs update the same comment rather than spamming the PR.
|
|
32
|
+
|
|
33
|
+
## Inputs
|
|
34
|
+
|
|
35
|
+
| Input | Default | Meaning |
|
|
36
|
+
|---|---|---|
|
|
37
|
+
| `suites` | `tests/**/*.mcpr` | Newline-separated paths or glob patterns |
|
|
38
|
+
| `lock` | empty | Optional contract lock checked against the first matched suite |
|
|
39
|
+
| `fail-on` | `breaking` | `breaking`, `potentially-breaking`, `any`, or `none` |
|
|
40
|
+
| `node-version` | `22` | Node.js version used by the Action |
|
|
41
|
+
| `version` | `latest` | MCP Rigor npm version installed for the run |
|
|
42
|
+
| `comment` | `true` | Post/update a pull-request comment |
|
|
43
|
+
| `flaky` | `true` | Add warnings when `.mcprigor/workspace-history.jsonl` exists |
|
|
44
|
+
| `artifact` | `true` | Upload the report, drift markdown, and per-suite JSON as a build artifact |
|
|
45
|
+
| `artifact-name` | `mcprigor-report` | Name of the uploaded artifact |
|
|
46
|
+
|
|
47
|
+
Outputs:
|
|
48
|
+
|
|
49
|
+
- `status` — `passed` or `failed`;
|
|
50
|
+
- `report` — path to the combined Markdown report;
|
|
51
|
+
- `drift-report` — path to the standalone contract drift Markdown (empty without `lock`);
|
|
52
|
+
- `artifact-dir` — directory containing the report, drift markdown, and per-suite JSON results.
|
|
53
|
+
|
|
54
|
+
## Contract drift as a PR artifact
|
|
55
|
+
|
|
56
|
+
When `lock` is set, the Action runs `mcprigor drift --markdown` and produces two things automatically:
|
|
57
|
+
|
|
58
|
+
1. a **Contract drift** section inside the PR comment and job summary;
|
|
59
|
+
2. a standalone `drift.md` artifact recording the suite, lock file, gate, gate result, and the full classified diff.
|
|
60
|
+
|
|
61
|
+
The whole report directory is uploaded with `actions/upload-artifact` (disable with `artifact: false`), so reviewers can download the exact drift evidence for a merge decision, and compliance flows can retain it beyond comment history. Consume `drift-report` from a later step to post the drift anywhere else:
|
|
62
|
+
|
|
63
|
+
```yaml
|
|
64
|
+
- id: rigor
|
|
65
|
+
uses: FusionOnePlatform/mcprigor@v1
|
|
66
|
+
with:
|
|
67
|
+
suites: tests/**/*.mcpr
|
|
68
|
+
lock: mcp.lock.yaml
|
|
69
|
+
- run: cat "${{ steps.rigor.outputs.drift-report }}"
|
|
70
|
+
if: steps.rigor.outputs.drift-report != ''
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
## Fork safety
|
|
75
|
+
|
|
76
|
+
The Action does not run arbitrary PR comment content. Test targets still come from repository suites, so use normal GitHub approval controls for workflows from untrusted forks. Pull-request comments require `pull-requests: write`; when the token cannot write (common for forks), set `comment: false` and rely on the job summary.
|
|
77
|
+
|
|
78
|
+
Pin a full release tag for the strongest supply-chain reproducibility:
|
|
79
|
+
|
|
80
|
+
```yaml
|
|
81
|
+
- uses: FusionOnePlatform/mcprigor@v1.5.0
|
|
82
|
+
```
|
package/docs/LANGUAGE-SPEC.md
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# MCP Test Language 1
|
|
2
2
|
|
|
3
|
-
Status: compatibility-stable frontend for MCP Rigor
|
|
3
|
+
Status: compatibility-stable frontend for MCP Rigor 1.x.
|
|
4
4
|
|
|
5
5
|
The `.mcpr` language and YAML compile to the same `Suite` runtime model. Every user-authored YAML capability has a deterministic plain-language equivalent; YAML remains available for generated files and programmatic integrations rather than being a more powerful test format.
|
|
6
6
|
|
|
7
|
+
Parity is enforced, not aspirational: every `.mcpr` construct compiles to a suite that also validates against the YAML/JSON schema, and `mcprigor convert my-tests.mcpr --out my-tests.yaml` (or `--format json`) emits that equivalent file. The converted suite reloads to the identical suite model and produces the same run results — the regression suite converts each shipped example, reloads it from YAML, and compares runs.
|
|
8
|
+
|
|
7
9
|
## Design goals
|
|
8
10
|
|
|
9
11
|
- Readable by QA professionals without programming experience
|
|
@@ -62,6 +64,59 @@ Server options:
|
|
|
62
64
|
Authorization: "Bearer ${env.QA_TOKEN}"
|
|
63
65
|
```
|
|
64
66
|
|
|
67
|
+
An HTTP target may fetch a bearer token at run time with `Token from` (a command whose single-line stdout becomes the `Authorization` header), or drive an interactive browser login with `OAuth`:
|
|
68
|
+
|
|
69
|
+
```text
|
|
70
|
+
Server options:
|
|
71
|
+
OAuth: oauth
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
`OAuth: oauth` performs an authorization-code + PKCE browser login once at the start of the run and carries the in-memory session (with automatic refresh) into every test. The block form takes optional `clientId`, `clientSecret` (use `${env.NAME}`), and `scope`. Tokens are never written to disk and are always redacted. See the [Authentication guide](AUTHENTICATION.md).
|
|
75
|
+
|
|
76
|
+
### Environment variables and secrets
|
|
77
|
+
|
|
78
|
+
Any string value in a target block — a header, a URL, a `cwd`, an `env` entry, a `Server options` field — may contain `${env.NAME}` placeholders. Before the suite connects, each placeholder is replaced with the value of the operating-system environment variable `NAME`:
|
|
79
|
+
|
|
80
|
+
```text
|
|
81
|
+
MCP URL: ${env.MCP_URL}
|
|
82
|
+
|
|
83
|
+
Server options:
|
|
84
|
+
headers:
|
|
85
|
+
Authorization: "Bearer ${env.MCP_TOKEN}"
|
|
86
|
+
X-Api-Key: "${env.API_KEY}"
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
MCP_URL=https://qa.example.com/mcp MCP_TOKEN=... API_KEY=... mcprigor test suite.mcpr
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Rules:
|
|
94
|
+
|
|
95
|
+
- The syntax is exactly `${env.NAME}`. `NAME` is a literal environment-variable name; there is no shell, no command substitution, and no default-value syntax.
|
|
96
|
+
- A placeholder may be embedded in a larger string (`"Bearer ${env.MCP_TOKEN}"`) or be the whole value (`"${env.API_KEY}"`), and a value may contain several placeholders.
|
|
97
|
+
- If `NAME` is not set, the run stops immediately with `Environment variable not found: NAME` — it never sends an empty header or a half-substituted URL.
|
|
98
|
+
- Never write a literal secret into a suite. Keep tokens and keys in the environment (locally) or in CI secrets, and reference them with `${env.NAME}` so the committed `.mcpr` file carries no credentials.
|
|
99
|
+
- Header values are registered with the redactor automatically, so a resolved token never appears in reports, evidence bundles, or published URLs.
|
|
100
|
+
|
|
101
|
+
The same `${env.NAME}` placeholders work in every target surface: single-server `Server options`, per-server `Server options for "name"` in compositions, and `Target options for "name"` in parity comparisons.
|
|
102
|
+
|
|
103
|
+
Multi-server compositions use named server declarations and per-test routing:
|
|
104
|
+
|
|
105
|
+
```text
|
|
106
|
+
Named server "catalog": node services/catalog.js
|
|
107
|
+
Named server "billing": https://qa.example.com/billing/mcp
|
|
108
|
+
|
|
109
|
+
Server options for "billing":
|
|
110
|
+
headers:
|
|
111
|
+
X-Tenant: qa
|
|
112
|
+
|
|
113
|
+
Test: "catalog lookup"
|
|
114
|
+
On server "catalog"
|
|
115
|
+
Call tool "search"
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
The YAML equivalent is a top-level `servers` mapping plus `server` on a test. A composition requires at least two named servers, and an unknown `On server` name is rejected during compilation.
|
|
119
|
+
|
|
65
120
|
Parity targets use the same connection grammar:
|
|
66
121
|
|
|
67
122
|
```text
|
|
@@ -77,6 +132,8 @@ Suite-level YAML fields have direct equivalents:
|
|
|
77
132
|
|
|
78
133
|
```text
|
|
79
134
|
Default timeout: 10 seconds
|
|
135
|
+
Budget: p95 500ms over 20 calls
|
|
136
|
+
Budget for "order lookup": p50 300ms over 20 calls
|
|
80
137
|
Redact: "secret-value", "token-value"
|
|
81
138
|
Snapshots: snapshots.json
|
|
82
139
|
Ignore snapshot paths: "$.createdAt", "$.requestId"
|
|
@@ -94,11 +151,7 @@ Client behavior:
|
|
|
94
151
|
approved: true
|
|
95
152
|
```
|
|
96
153
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Per-test scripted responses (override `Client behavior:` for one test):
|
|
154
|
+
Per-test scripted responses override `Client behavior:` for one test:
|
|
102
155
|
|
|
103
156
|
```text
|
|
104
157
|
When the server asks for input, respond "accept" with:
|
|
@@ -106,7 +159,10 @@ When the server asks for input, respond "accept" with:
|
|
|
106
159
|
When the server asks for input, respond "decline"
|
|
107
160
|
When the server requests sampling, respond "scripted text"
|
|
108
161
|
```
|
|
109
|
-
|
|
162
|
+
|
|
163
|
+
## Imports
|
|
164
|
+
|
|
165
|
+
```text
|
|
110
166
|
Import flows from "./shared/customer-flows.mcpr"
|
|
111
167
|
```
|
|
112
168
|
|
|
@@ -183,6 +239,7 @@ Send "ping"
|
|
|
183
239
|
```text
|
|
184
240
|
Expect it succeeds
|
|
185
241
|
Expect an error
|
|
242
|
+
Expect the call to finish within 800ms
|
|
186
243
|
Expect "structuredContent.total" equals 2
|
|
187
244
|
Expect "content[0].text" contains "complete"
|
|
188
245
|
Expect "items" exists
|
package/docs/MCP-SERVER.md
CHANGED
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Scheduled production monitoring
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
Turn an HTTP MCP suite into a continuous production check:
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
mcprigor monitor tests/prod.mcpr \
|
|
9
|
+
--every 15m \
|
|
10
|
+
--notify https://alerts.example.com/hooks/mcprigor
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The monitor runs immediately, then at the fixed interval until stopped. Durations accept `ms`, `s`, `m`, or `h`, with a minimum interval of one second.
|
|
14
|
+
|
|
15
|
+
For operational safety, monitoring requires a Streamable HTTP target. It refuses stdio suites so a long-running process cannot repeatedly spawn local commands by accident.
|
|
16
|
+
|
|
17
|
+
## Notification policies
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
mcprigor monitor tests/prod.mcpr --every 5m --notify URL --notify-on change
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
`--notify-on` supports:
|
|
24
|
+
|
|
25
|
+
- `change` (default): first failure, then failure/recovery transitions;
|
|
26
|
+
- `failure`: every failed run;
|
|
27
|
+
- `recovery`: transitions from failed to passed;
|
|
28
|
+
- `always`: every run.
|
|
29
|
+
|
|
30
|
+
Webhook requests are JSON POSTs with a 15-second timeout:
|
|
31
|
+
|
|
32
|
+
```json
|
|
33
|
+
{
|
|
34
|
+
"source": "mcprigor",
|
|
35
|
+
"event": "monitor.failure",
|
|
36
|
+
"suite": "tests/prod.mcpr",
|
|
37
|
+
"run": 12,
|
|
38
|
+
"status": "failed",
|
|
39
|
+
"startedAt": "2026-08-30T12:00:00.000Z",
|
|
40
|
+
"durationMs": 842,
|
|
41
|
+
"summary": { "passed": 7, "failed": 1, "skipped": 0, "blocked": 0 },
|
|
42
|
+
"failures": [{ "name": "order lookup", "error": "..." }]
|
|
43
|
+
}
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
A non-2xx webhook response fails the monitor with `MCP-MONITOR-003`; notification loss is never silently ignored.
|
|
47
|
+
|
|
48
|
+
## History and trends
|
|
49
|
+
|
|
50
|
+
Every monitoring run appends to `.mcprigor/workspace-history.jsonl`, so existing `mcprigor trends`, PDF/CSV exports, flaky detection, latency budgets, and regression analysis include production monitoring evidence.
|
|
51
|
+
|
|
52
|
+
## Bounded runs
|
|
53
|
+
|
|
54
|
+
For smoke tests, cron jobs, and CI validation:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
mcprigor monitor tests/prod.mcpr --every 1m --max-runs 1
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
`SIGINT` and `SIGTERM` stop the interval cleanly after active MCP sessions shut down.
|
|
61
|
+
|
|
62
|
+
Use an authenticated suite target (`headers` or `Token from:`), retain history according to your evidence policy, and send webhooks only to reviewed HTTPS endpoints.
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# Multi-server compositions
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
Real MCP clients mount several servers together. MCP Rigor compositions test each named server and govern the combined tool, resource, and prompt namespace as one fleet.
|
|
6
|
+
|
|
7
|
+
## Declare named servers
|
|
8
|
+
|
|
9
|
+
```text
|
|
10
|
+
MCP Test 1
|
|
11
|
+
Suite: "Checkout fleet"
|
|
12
|
+
|
|
13
|
+
Named server "catalog": node services/catalog.js
|
|
14
|
+
Named server "billing": https://qa.example.com/billing/mcp
|
|
15
|
+
|
|
16
|
+
Server options for "billing":
|
|
17
|
+
Token from: node scripts/get-qa-token.mjs
|
|
18
|
+
|
|
19
|
+
Test: "catalog search works"
|
|
20
|
+
On server "catalog"
|
|
21
|
+
Call tool "search" with:
|
|
22
|
+
query: "widget"
|
|
23
|
+
Expect "structuredContent.total" is a number
|
|
24
|
+
|
|
25
|
+
Test: "billing responds"
|
|
26
|
+
On server "billing"
|
|
27
|
+
Send "ping"
|
|
28
|
+
Expect it succeeds
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
`On server` routes every action in that test to the selected server. Tests without `On server` continue to use the legacy/default `Server:` or `MCP URL:` target. Cross-test dependencies and exported values continue to work across named servers.
|
|
32
|
+
|
|
33
|
+
YAML parity:
|
|
34
|
+
|
|
35
|
+
```yaml
|
|
36
|
+
version: 1
|
|
37
|
+
name: Checkout fleet
|
|
38
|
+
target:
|
|
39
|
+
transport: stdio
|
|
40
|
+
command: node
|
|
41
|
+
args: [services/gateway.js]
|
|
42
|
+
servers:
|
|
43
|
+
catalog:
|
|
44
|
+
transport: stdio
|
|
45
|
+
command: node
|
|
46
|
+
args: [services/catalog.js]
|
|
47
|
+
billing:
|
|
48
|
+
transport: streamable-http
|
|
49
|
+
url: https://qa.example.com/billing/mcp
|
|
50
|
+
tests:
|
|
51
|
+
- name: catalog search works
|
|
52
|
+
server: catalog
|
|
53
|
+
steps:
|
|
54
|
+
- tool:
|
|
55
|
+
name: search
|
|
56
|
+
arguments: { query: widget }
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Unknown server names fail validation before a test starts.
|
|
60
|
+
|
|
61
|
+
## Check the live composition
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
mcprigor composition-check tests/fleet.mcpr
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The check discovers all named servers and reports:
|
|
68
|
+
|
|
69
|
+
- `MCP-COMP-001`: duplicate tool name;
|
|
70
|
+
- `MCP-COMP-002`: conflicting input/output schemas for the same tool (breaking);
|
|
71
|
+
- `MCP-COMP-003`: duplicate resource URI or URI template (breaking);
|
|
72
|
+
- `MCP-COMP-004`: duplicate prompt name.
|
|
73
|
+
|
|
74
|
+
MCP Rigor reports collisions; it never silently renames or chooses a winning server.
|
|
75
|
+
|
|
76
|
+
## Create a combined fleet lock
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
mcprigor composition-discover tests/fleet.mcpr \
|
|
80
|
+
--out contracts/checkout.composition.lock.yaml
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The lock embeds each named server's ordinary discovery contract, the cross-server issue set, and a stable fleet fingerprint. Volatile discovery timestamps and diagnostics do not change the combined fingerprint. Writes are atomic: all servers must be discovered successfully before the previous lock is replaced.
|
|
84
|
+
|
|
85
|
+
## Gate fleet drift in CI
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
mcprigor composition-drift tests/fleet.mcpr \
|
|
89
|
+
--against contracts/checkout.composition.lock.yaml \
|
|
90
|
+
--fail-on breaking
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
The drift report combines:
|
|
94
|
+
|
|
95
|
+
- server additions and removals;
|
|
96
|
+
- each server's tool/resource/prompt contract changes;
|
|
97
|
+
- newly introduced or resolved cross-server conflicts.
|
|
98
|
+
|
|
99
|
+
`--fail-on` accepts `breaking` (default), `potentially-breaking`, `any`, or `none`, matching the single-server drift gate. Add `--json report.json` for machine-readable CI evidence.
|
|
100
|
+
|
|
101
|
+
## Composition versus parity
|
|
102
|
+
|
|
103
|
+
Use a **composition** when several servers are mounted together and their namespaces interact. Use **transport parity** when the same logical server is exposed through alternate targets such as local stdio and deployed HTTP. The `servers` and `targets` fields remain intentionally separate.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Natural-language cookbook
|
|
2
2
|
|
|
3
3
|
Copy a pattern, replace the names and values, then run `mcprigor check FILE`.
|
|
4
4
|
|
|
@@ -187,7 +187,7 @@ Run with `mcprigor parity FILE`.
|
|
|
187
187
|
|
|
188
188
|
## Test a server that needs a bearer token
|
|
189
189
|
|
|
190
|
-
Point the suite at the deployed endpoint and pass the token through an environment variable.
|
|
190
|
+
Point the suite at the deployed endpoint and pass the token through an environment variable. The `${env.NAME}` placeholder is replaced with the value of the `NAME` environment variable before the suite connects, so no real token ever lives in the test file.
|
|
191
191
|
|
|
192
192
|
```text
|
|
193
193
|
MCP Test 1
|
|
@@ -210,6 +210,15 @@ Run it with the token in the environment:
|
|
|
210
210
|
QA_TOKEN=... mcprigor test orders.mcpr
|
|
211
211
|
```
|
|
212
212
|
|
|
213
|
+
The placeholder can sit inside a larger string (`"Bearer ${env.QA_TOKEN}"`) or be the whole value, and any header works the same way — API keys, custom tenant headers, and so on:
|
|
214
|
+
|
|
215
|
+
```text
|
|
216
|
+
Server options:
|
|
217
|
+
headers:
|
|
218
|
+
X-Api-Key: "${env.API_KEY}"
|
|
219
|
+
X-Tenant: "acme"
|
|
220
|
+
```
|
|
221
|
+
|
|
213
222
|
Three guarantees come with this pattern:
|
|
214
223
|
|
|
215
224
|
- if `QA_TOKEN` is not set, the run stops with `Environment variable not found: QA_TOKEN` instead of sending an empty header;
|
|
@@ -238,7 +247,7 @@ Test: "Search behaves the same"
|
|
|
238
247
|
|
|
239
248
|
## When the token must be fetched first
|
|
240
249
|
|
|
241
|
-
|
|
250
|
+
When a short-lived token must be acquired non-interactively (client-credentials exchange, cloud CLI, vault), fetch it in the step before the run:
|
|
242
251
|
|
|
243
252
|
```bash
|
|
244
253
|
QA_TOKEN=$(curl -s -X POST https://auth.example.com/oauth/token \
|
|
@@ -256,7 +265,29 @@ In CI, do the same in the workflow:
|
|
|
256
265
|
run: npx mcprigor test tests/*.mcpr
|
|
257
266
|
```
|
|
258
267
|
|
|
259
|
-
|
|
268
|
+
For CI and other unattended runs, obtain the token non-interactively as above. When a real user must sign in through a browser, use interactive OAuth instead (next recipe).
|
|
269
|
+
|
|
270
|
+
## Sign in through the browser (interactive OAuth)
|
|
271
|
+
|
|
272
|
+
When a server requires a real user login, let MCP Rigor run the browser authorization-code flow once and carry the authorized session — with automatic refresh — into every test in the run:
|
|
273
|
+
|
|
274
|
+
```text
|
|
275
|
+
MCP URL: https://app.example.com/mcp
|
|
276
|
+
|
|
277
|
+
Server options:
|
|
278
|
+
OAuth: oauth
|
|
279
|
+
|
|
280
|
+
Test: "an authenticated call succeeds"
|
|
281
|
+
Call tool "find_order" with:
|
|
282
|
+
orderId: "A-1001"
|
|
283
|
+
Expect "structuredContent.status" equals "shipped"
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
```bash
|
|
287
|
+
mcprigor test orders.mcpr
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
Your browser opens to the identity provider; after you sign in, the tokens are held in memory (never written to disk, always redacted) and reused for the whole suite. For pre-registered clients or specific scopes, use the block form with `clientId`, `clientSecret: "${env.…}"`, and `scope`. Because it needs a human, keep interactive OAuth for local runs and use a non-interactive credential in CI. See the [Authentication guide](AUTHENTICATION.md) for the full flow.
|
|
260
291
|
|
|
261
292
|
## Fetch an OAuth token before connecting
|
|
262
293
|
|
|
@@ -302,6 +333,34 @@ Test: "summaries embed the sampled text"
|
|
|
302
333
|
|
|
303
334
|
Suite-wide defaults live in a `Client behavior:` block; these per-test lines override them for one test.
|
|
304
335
|
|
|
336
|
+
## Set latency budgets and catch slow releases
|
|
337
|
+
|
|
338
|
+
Fail a single call that takes too long:
|
|
339
|
+
|
|
340
|
+
```text
|
|
341
|
+
Test: "order lookup is fast"
|
|
342
|
+
Call tool "find_order" with:
|
|
343
|
+
orderId: "A-1001"
|
|
344
|
+
Expect the call to finish within 800ms
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
Set suite-level budgets measured as percentiles over recorded run history (the same history behind `mcprigor trends`):
|
|
348
|
+
|
|
349
|
+
```text
|
|
350
|
+
Budget: p95 500ms over 20 calls
|
|
351
|
+
Budget for "order lookup is fast": p50 300ms
|
|
352
|
+
```
|
|
353
|
+
|
|
354
|
+
Budgets are judged after each `mcprigor test` run and fail the run when a percentile exceeds its budget. Until enough history exists they report as pending instead of guessing.
|
|
355
|
+
|
|
356
|
+
Gate CI on latency regressions against the trend baseline — no budget numbers needed:
|
|
357
|
+
|
|
358
|
+
```bash
|
|
359
|
+
mcprigor test suite.mcpr --fail-on-regression
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
A test regresses when it runs slower than 1.5x its historical median (with a 50 ms floor so micro-tests don't trip on jitter).
|
|
363
|
+
|
|
305
364
|
## Match a snapshot
|
|
306
365
|
|
|
307
366
|
```text
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Performance governance
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
MCP Rigor turns MCP latency into a deterministic release gate. It supports immediate per-call limits, percentile budgets over recorded history, and automatic regression detection against each test's historical baseline.
|
|
6
|
+
|
|
7
|
+
## Limit one call
|
|
8
|
+
|
|
9
|
+
Add a latency assertion after an action:
|
|
10
|
+
|
|
11
|
+
```text
|
|
12
|
+
Test: "order lookup stays interactive"
|
|
13
|
+
Call tool "find_order" with:
|
|
14
|
+
orderId: "A-1001"
|
|
15
|
+
Expect the call to finish within 800ms
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The measured duration covers the live MCP request and response. A call over the limit fails with `MCP-PERF-001` and reports the measured and allowed duration.
|
|
19
|
+
|
|
20
|
+
YAML parity:
|
|
21
|
+
|
|
22
|
+
```yaml
|
|
23
|
+
- tool:
|
|
24
|
+
name: find_order
|
|
25
|
+
arguments:
|
|
26
|
+
orderId: A-1001
|
|
27
|
+
assert:
|
|
28
|
+
maxDurationMs: 800
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Set percentile budgets
|
|
32
|
+
|
|
33
|
+
A percentile budget uses successful durations from recent recorded runs plus the current run:
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Budget: p95 500ms over 20 calls
|
|
37
|
+
Budget for "order lookup stays interactive": p50 300ms over 20 calls
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
- A suite-wide budget applies independently to every test.
|
|
41
|
+
- A named budget applies only to that test.
|
|
42
|
+
- MCP Rigor uses the deterministic nearest-rank percentile.
|
|
43
|
+
- Fewer than three usable samples reports the budget as pending instead of guessing.
|
|
44
|
+
- A measured percentile over budget fails the CLI run.
|
|
45
|
+
|
|
46
|
+
YAML parity:
|
|
47
|
+
|
|
48
|
+
```yaml
|
|
49
|
+
budgets:
|
|
50
|
+
- test: "*"
|
|
51
|
+
percentile: 95
|
|
52
|
+
maxMs: 500
|
|
53
|
+
window: 20
|
|
54
|
+
- test: order lookup stays interactive
|
|
55
|
+
percentile: 50
|
|
56
|
+
maxMs: 300
|
|
57
|
+
window: 20
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Fail on regression without maintaining thresholds
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
mcprigor test tests/orders.mcpr --fail-on-regression
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
For every successful test, MCP Rigor compares the current duration with the median of its latest successful history. The gate requires at least five samples and reports a regression when the current run exceeds both:
|
|
67
|
+
|
|
68
|
+
- 1.5× the historical median; and
|
|
69
|
+
- the historical median plus 50 ms.
|
|
70
|
+
|
|
71
|
+
The absolute floor prevents very small tests from failing because of ordinary scheduler jitter.
|
|
72
|
+
|
|
73
|
+
## CI example
|
|
74
|
+
|
|
75
|
+
```yaml
|
|
76
|
+
- run: npx mcprigor test tests/orders.mcpr --fail-on-regression --junit reports/orders.xml
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Explicit budgets answer “is this fast enough?” The regression gate answers “did this release make it materially slower?” Teams commonly use both.
|
|
80
|
+
|
|
81
|
+
## History source
|
|
82
|
+
|
|
83
|
+
CLI, QA workspace, and MCP-server-driven test runs append to `.mcprigor/workspace-history.jsonl`. The same data powers `mcprigor trends`, CSV/PDF trend exports, flaky detection, percentile budgets, and regression baselines.
|
|
84
|
+
|
|
85
|
+
Commit the test and its budget declarations. Treat history as a CI artifact or retained workspace file according to your evidence policy.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Shareable hosted reports
|
|
2
|
+
|
|
3
|
+
> Available since `1.5.0`.
|
|
4
|
+
|
|
5
|
+
Turn a test run into a URL anyone can open — no repository access, no CI login:
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
export NETLIFY_AUTH_TOKEN=... # personal or CI token
|
|
9
|
+
mcprigor publish tests/catalog.mcpr --site your-netlify-site
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
`publish` runs the suite, builds the readable HTML report **with the clickable request/response session timeline**, deploys it to your Netlify site with the dependency-free digest API, waits until the deploy is live, and prints the shareable URL:
|
|
13
|
+
|
|
14
|
+
```
|
|
15
|
+
Published report: https://68b1c2--your-site.netlify.app
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Each publish is a normal Netlify deploy of your own site, so access control, custom domains, deploy previews, and retention follow your existing hosting configuration. Unchanged files are skipped automatically via content digests.
|
|
19
|
+
|
|
20
|
+
## Options
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
mcprigor publish suite.mcpr --site SITE [--include-json] [--test NAME] [--env qa]
|
|
24
|
+
mcprigor publish suite.mcpr --out reports/latest
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
- `--site` — Netlify site ID or name. The token comes only from `NETLIFY_AUTH_TOKEN` (or `MCPRIGOR_PUBLISH_TOKEN`); tokens are never accepted as command-line flags.
|
|
28
|
+
- `--out DIR` — write the same bundle to a local directory instead of (or in addition to) hosting it. Serve it from any static host: S3, GitHub Pages, nginx, an artifact store.
|
|
29
|
+
- `--include-json` — also publish `result.json` for dashboards and programmatic consumers.
|
|
30
|
+
- `--test`, `--env`, `--command`, `--url` — the same run-selection options as `mcprigor test`.
|
|
31
|
+
|
|
32
|
+
The exit code still reflects the run (`0` passed, `1` failed), so `publish` can replace `test` in a pipeline step that both gates and shares.
|
|
33
|
+
|
|
34
|
+
## Publishing from the QA workspace UI
|
|
35
|
+
|
|
36
|
+
The visual workspace (`mcprigor workspace`) exposes the same features without the command line:
|
|
37
|
+
|
|
38
|
+
- **HTML report** — every finished test run has an `HTML report` button that opens the full report, including the clickable session timeline, in a new tab.
|
|
39
|
+
- **Publish** — start the workspace with hosting configured and a `Publish` button appears next to the export buttons:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
export MCPRIGOR_PUBLISH_SITE=your-netlify-site
|
|
43
|
+
export NETLIFY_AUTH_TOKEN=...
|
|
44
|
+
mcprigor workspace
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Clicking it deploys the selected run's report and opens the shareable URL; a `View published ↗` link stays on the run for re-opening or copying. Without both variables the button is hidden and the API answers with a clear configuration message — the token itself never reaches the browser.
|
|
48
|
+
|
|
49
|
+
## Security
|
|
50
|
+
|
|
51
|
+
- The report is produced by the same pipeline as `--html`: secrets and configured redaction patterns are removed **before** the report exists.
|
|
52
|
+
- Publishing makes the report as public as the target site. Use a password-protected or team-restricted Netlify site for internal results.
|
|
53
|
+
- The hosting token is read from the environment at the last moment and is never echoed, logged, or stored.
|
package/docs/QA-GUIDE.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# QA guide
|
|
2
2
|
|
|
3
|
-
Use this page as a short everyday checklist. For examples, open the [
|
|
3
|
+
Use this page as a short everyday checklist. For examples, open the [natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md).
|
|
4
4
|
|
|
5
5
|
## Everyday workflow
|
|
6
6
|
|
|
@@ -121,7 +121,7 @@ Use values as `${row.input}` and `${row.expected}`.
|
|
|
121
121
|
## Where to go next
|
|
122
122
|
|
|
123
123
|
- [Getting started](GETTING-STARTED.md)
|
|
124
|
-
- [
|
|
124
|
+
- [Natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md)
|
|
125
125
|
- [Guided authoring](GUIDED-AUTHORING.md)
|
|
126
126
|
- [Data and reusable flows](DATA-AND-REUSE.md)
|
|
127
127
|
- [Troubleshooting](TROUBLESHOOTING.md)
|
package/docs/QA-WORKSPACE.md
CHANGED
|
@@ -30,7 +30,7 @@ An empty folder shows a three-step welcome screen. Choose **+ New test file**
|
|
|
30
30
|
2. Edit the plain-language scenario. The editor provides syntax highlighting, line numbers, and autocomplete: top-level declarations at the start of a line, actions and assertions when indented, and comparison phrases after `Expect "field"`. Accept with `Tab` or `Enter`; force the list open with `Ctrl+Space`.
|
|
31
31
|
3. Choose **Validate** (`Ctrl/⌘+S` saves, `Ctrl/⌘+Enter` runs). A wording problem highlights the offending line and moves the cursor to it.
|
|
32
32
|
4. Choose **▶ Run tests** or **Parity**.
|
|
33
|
-
5. Review the results panel: per-file pass/fail with durations; select a file for its full report.
|
|
33
|
+
5. Review the results panel: per-file pass/fail with durations; select a file for its full report. Drag either panel divider to resize the file list, editor, and results areas; widths persist across reloads, arrow keys resize a focused divider, and double-click resets the layout.
|
|
34
34
|
|
|
35
35
|
The editor marks unsaved changes; running or validating saves them first. If the file changed elsewhere after you opened it, the workspace refuses to overwrite it and asks you to reload.
|
|
36
36
|
|
|
@@ -52,6 +52,10 @@ Test runs are recorded in `.mcprigor/workspace-history.jsonl` (most recent 2000
|
|
|
52
52
|
|
|
53
53
|
One search box filters all three tabs. It matches suite names, test names, and error text, and highlights matches, so you can answer questions like "when did `delivered` start failing?" without leaving the browser.
|
|
54
54
|
|
|
55
|
+
Completed runs expose **PDF**, **CSV**, and **JUnit XML** downloads. The Trends tab exports a rich trends PDF, aggregate CSV, or raw history CSV. PDF reports include summary cards, pass-rate visuals, per-test detail, failures, durations, and evidence identifiers.
|
|
56
|
+
|
|
57
|
+
An **HTML report** button opens the full report — including the clickable request/response session timeline — in a new tab. When the workspace is started with `MCPRIGOR_PUBLISH_SITE` and `NETLIFY_AUTH_TOKEN` set, a **Publish** button also appears: one click hosts the report at a shareable static URL and keeps a `View published ↗` link on the run. See [Shareable hosted reports](PUBLISHING.md).
|
|
58
|
+
|
|
55
59
|
## What is available
|
|
56
60
|
|
|
57
61
|
- Creating, renaming, and editing `.mcpr` suites (YAML and JSON suites are listed and editable too)
|