mcprigor 1.4.0 → 1.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (99) hide show
  1. package/README.md +21 -4
  2. package/dist/audit.d.ts +44 -0
  3. package/dist/audit.d.ts.map +1 -0
  4. package/dist/audit.js +157 -0
  5. package/dist/audit.js.map +1 -0
  6. package/dist/cli.js +243 -10
  7. package/dist/cli.js.map +1 -1
  8. package/dist/composition.d.ts +42 -0
  9. package/dist/composition.d.ts.map +1 -0
  10. package/dist/composition.js +130 -0
  11. package/dist/composition.js.map +1 -0
  12. package/dist/coverage.d.ts +33 -0
  13. package/dist/coverage.d.ts.map +1 -0
  14. package/dist/coverage.js +83 -0
  15. package/dist/coverage.js.map +1 -0
  16. package/dist/export.d.ts +4 -0
  17. package/dist/export.d.ts.map +1 -1
  18. package/dist/export.js +56 -0
  19. package/dist/export.js.map +1 -1
  20. package/dist/index.d.ts +5 -0
  21. package/dist/index.d.ts.map +1 -1
  22. package/dist/index.js +5 -0
  23. package/dist/index.js.map +1 -1
  24. package/dist/language.js +1 -1
  25. package/dist/language.js.map +1 -1
  26. package/dist/loader.d.ts.map +1 -1
  27. package/dist/loader.js +4 -1
  28. package/dist/loader.js.map +1 -1
  29. package/dist/monitor.d.ts +22 -0
  30. package/dist/monitor.d.ts.map +1 -0
  31. package/dist/monitor.js +71 -0
  32. package/dist/monitor.js.map +1 -0
  33. package/dist/oauth.d.ts +73 -0
  34. package/dist/oauth.d.ts.map +1 -0
  35. package/dist/oauth.js +144 -0
  36. package/dist/oauth.js.map +1 -0
  37. package/dist/perf.d.ts +37 -0
  38. package/dist/perf.d.ts.map +1 -0
  39. package/dist/perf.js +90 -0
  40. package/dist/perf.js.map +1 -0
  41. package/dist/publish.d.ts +19 -0
  42. package/dist/publish.d.ts.map +1 -0
  43. package/dist/publish.js +58 -0
  44. package/dist/publish.js.map +1 -0
  45. package/dist/qa-language.d.ts.map +1 -1
  46. package/dist/qa-language.js +75 -3
  47. package/dist/qa-language.js.map +1 -1
  48. package/dist/reporters.d.ts +3 -1
  49. package/dist/reporters.d.ts.map +1 -1
  50. package/dist/reporters.js +21 -3
  51. package/dist/reporters.js.map +1 -1
  52. package/dist/runner.d.ts +7 -0
  53. package/dist/runner.d.ts.map +1 -1
  54. package/dist/runner.js +53 -15
  55. package/dist/runner.js.map +1 -1
  56. package/dist/schema.d.ts +66 -0
  57. package/dist/schema.d.ts.map +1 -1
  58. package/dist/schema.js +21 -1
  59. package/dist/schema.js.map +1 -1
  60. package/dist/session.d.ts +2 -1
  61. package/dist/session.d.ts.map +1 -1
  62. package/dist/session.js +7 -3
  63. package/dist/session.js.map +1 -1
  64. package/dist/timeline.d.ts +18 -0
  65. package/dist/timeline.d.ts.map +1 -0
  66. package/dist/timeline.js +59 -0
  67. package/dist/timeline.js.map +1 -0
  68. package/dist/types.d.ts +29 -0
  69. package/dist/types.d.ts.map +1 -1
  70. package/dist/version.d.ts +1 -1
  71. package/dist/version.js +1 -1
  72. package/dist/workspace.d.ts.map +1 -1
  73. package/dist/workspace.js +35 -3
  74. package/dist/workspace.js.map +1 -1
  75. package/docs/AUTHENTICATION.md +213 -0
  76. package/docs/CLI-REFERENCE.md +123 -2
  77. package/docs/COVERAGE.md +37 -0
  78. package/docs/ENGINEER-SETUP.md +1 -1
  79. package/docs/EVIDENCE.md +11 -0
  80. package/docs/GETTING-STARTED.md +1 -1
  81. package/docs/GITHUB-ACTION.md +82 -0
  82. package/docs/LANGUAGE-SPEC.md +64 -7
  83. package/docs/MCP-SERVER.md +1 -1
  84. package/docs/MONITORING.md +62 -0
  85. package/docs/MULTI-SERVER-COMPOSITIONS.md +103 -0
  86. package/docs/{PLAIN-LANGUAGE-COOKBOOK.md → NATURAL-LANGUAGE-COOKBOOK.md} +63 -4
  87. package/docs/PERFORMANCE-GOVERNANCE.md +85 -0
  88. package/docs/PUBLISHING.md +53 -0
  89. package/docs/QA-GUIDE.md +2 -2
  90. package/docs/QA-WORKSPACE.md +5 -1
  91. package/docs/README.md +10 -2
  92. package/docs/SECURITY-AUDIT.md +77 -0
  93. package/editors/mcpr-language.json +2 -2
  94. package/editors/vscode/package.json +1 -1
  95. package/editors/vscode/syntaxes/mcpr.tmLanguage.json +2 -2
  96. package/package.json +1 -2
  97. package/workspace-assets/app.js +35 -4
  98. package/workspace-assets/index.html +4 -1
  99. package/workspace-assets/style.css +2 -0
@@ -126,7 +126,7 @@ mcp-acceptance-tests/
126
126
 
127
127
  ## Next steps
128
128
 
129
- - [Plain-language cookbook](PLAIN-LANGUAGE-COOKBOOK.md)
129
+ - [Natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md)
130
130
  - [QA workspace](QA-WORKSPACE.md)
131
131
  - [Engineer setup and CI](ENGINEER-SETUP.md)
132
132
  - [Troubleshooting](TROUBLESHOOTING.md)
@@ -0,0 +1,82 @@
1
+ # GitHub Action and pull-request reports
2
+
3
+ > Available since `v1.5.0`; pin `@v1` for the latest 1.x or `@v1.5.0` for full reproducibility.
4
+
5
+ The MCP Rigor Action runs deterministic suites, optionally gates contract drift, includes flaky-history warnings, writes a rich job summary, and creates or updates one pull-request comment.
6
+
7
+ ## Workflow
8
+
9
+ ```yaml
10
+ name: MCP Rigor
11
+ on:
12
+ pull_request:
13
+
14
+ permissions:
15
+ contents: read
16
+ pull-requests: write
17
+
18
+ jobs:
19
+ rigor:
20
+ runs-on: ubuntu-latest
21
+ steps:
22
+ - uses: actions/checkout@v4
23
+ - uses: FusionOnePlatform/mcprigor@v1
24
+ with:
25
+ suites: |
26
+ tests/**/*.mcpr
27
+ lock: mcp.lock.yaml
28
+ fail-on: breaking
29
+ ```
30
+
31
+ The comment contains a suite table with pass/fail/skipped totals and durations, the classified contract drift report, flaky-test warnings when history exists, and collapsed failure detail. A stable HTML marker makes subsequent runs update the same comment rather than spamming the PR.
32
+
33
+ ## Inputs
34
+
35
+ | Input | Default | Meaning |
36
+ |---|---|---|
37
+ | `suites` | `tests/**/*.mcpr` | Newline-separated paths or glob patterns |
38
+ | `lock` | empty | Optional contract lock checked against the first matched suite |
39
+ | `fail-on` | `breaking` | `breaking`, `potentially-breaking`, `any`, or `none` |
40
+ | `node-version` | `22` | Node.js version used by the Action |
41
+ | `version` | `latest` | MCP Rigor npm version installed for the run |
42
+ | `comment` | `true` | Post/update a pull-request comment |
43
+ | `flaky` | `true` | Add warnings when `.mcprigor/workspace-history.jsonl` exists |
44
+ | `artifact` | `true` | Upload the report, drift markdown, and per-suite JSON as a build artifact |
45
+ | `artifact-name` | `mcprigor-report` | Name of the uploaded artifact |
46
+
47
+ Outputs:
48
+
49
+ - `status` — `passed` or `failed`;
50
+ - `report` — path to the combined Markdown report;
51
+ - `drift-report` — path to the standalone contract drift Markdown (empty without `lock`);
52
+ - `artifact-dir` — directory containing the report, drift markdown, and per-suite JSON results.
53
+
54
+ ## Contract drift as a PR artifact
55
+
56
+ When `lock` is set, the Action runs `mcprigor drift --markdown` and produces two things automatically:
57
+
58
+ 1. a **Contract drift** section inside the PR comment and job summary;
59
+ 2. a standalone `drift.md` artifact recording the suite, lock file, gate, gate result, and the full classified diff.
60
+
61
+ The whole report directory is uploaded with `actions/upload-artifact` (disable with `artifact: false`), so reviewers can download the exact drift evidence for a merge decision, and compliance flows can retain it beyond comment history. Consume `drift-report` from a later step to post the drift anywhere else:
62
+
63
+ ```yaml
64
+ - id: rigor
65
+ uses: FusionOnePlatform/mcprigor@v1
66
+ with:
67
+ suites: tests/**/*.mcpr
68
+ lock: mcp.lock.yaml
69
+ - run: cat "${{ steps.rigor.outputs.drift-report }}"
70
+ if: steps.rigor.outputs.drift-report != ''
71
+ ```
72
+
73
+
74
+ ## Fork safety
75
+
76
+ The Action does not run arbitrary PR comment content. Test targets still come from repository suites, so use normal GitHub approval controls for workflows from untrusted forks. Pull-request comments require `pull-requests: write`; when the token cannot write (common for forks), set `comment: false` and rely on the job summary.
77
+
78
+ Pin a full release tag for the strongest supply-chain reproducibility:
79
+
80
+ ```yaml
81
+ - uses: FusionOnePlatform/mcprigor@v1.5.0
82
+ ```
@@ -1,9 +1,11 @@
1
1
  # MCP Test Language 1
2
2
 
3
- Status: compatibility-stable frontend for MCP Rigor 0.12.
3
+ Status: compatibility-stable frontend for MCP Rigor 1.x.
4
4
 
5
5
  The `.mcpr` language and YAML compile to the same `Suite` runtime model. Every user-authored YAML capability has a deterministic plain-language equivalent; YAML remains available for generated files and programmatic integrations rather than being a more powerful test format.
6
6
 
7
+ Parity is enforced, not aspirational: every `.mcpr` construct compiles to a suite that also validates against the YAML/JSON schema, and `mcprigor convert my-tests.mcpr --out my-tests.yaml` (or `--format json`) emits that equivalent file. The converted suite reloads to the identical suite model and produces the same run results — the regression suite converts each shipped example, reloads it from YAML, and compares runs.
8
+
7
9
  ## Design goals
8
10
 
9
11
  - Readable by QA professionals without programming experience
@@ -62,6 +64,59 @@ Server options:
62
64
  Authorization: "Bearer ${env.QA_TOKEN}"
63
65
  ```
64
66
 
67
+ An HTTP target may fetch a bearer token at run time with `Token from` (a command whose single-line stdout becomes the `Authorization` header), or drive an interactive browser login with `OAuth`:
68
+
69
+ ```text
70
+ Server options:
71
+ OAuth: oauth
72
+ ```
73
+
74
+ `OAuth: oauth` performs an authorization-code + PKCE browser login once at the start of the run and carries the in-memory session (with automatic refresh) into every test. The block form takes optional `clientId`, `clientSecret` (use `${env.NAME}`), and `scope`. Tokens are never written to disk and are always redacted. See the [Authentication guide](AUTHENTICATION.md).
75
+
76
+ ### Environment variables and secrets
77
+
78
+ Any string value in a target block — a header, a URL, a `cwd`, an `env` entry, a `Server options` field — may contain `${env.NAME}` placeholders. Before the suite connects, each placeholder is replaced with the value of the operating-system environment variable `NAME`:
79
+
80
+ ```text
81
+ MCP URL: ${env.MCP_URL}
82
+
83
+ Server options:
84
+ headers:
85
+ Authorization: "Bearer ${env.MCP_TOKEN}"
86
+ X-Api-Key: "${env.API_KEY}"
87
+ ```
88
+
89
+ ```bash
90
+ MCP_URL=https://qa.example.com/mcp MCP_TOKEN=... API_KEY=... mcprigor test suite.mcpr
91
+ ```
92
+
93
+ Rules:
94
+
95
+ - The syntax is exactly `${env.NAME}`. `NAME` is a literal environment-variable name; there is no shell, no command substitution, and no default-value syntax.
96
+ - A placeholder may be embedded in a larger string (`"Bearer ${env.MCP_TOKEN}"`) or be the whole value (`"${env.API_KEY}"`), and a value may contain several placeholders.
97
+ - If `NAME` is not set, the run stops immediately with `Environment variable not found: NAME` — it never sends an empty header or a half-substituted URL.
98
+ - Never write a literal secret into a suite. Keep tokens and keys in the environment (locally) or in CI secrets, and reference them with `${env.NAME}` so the committed `.mcpr` file carries no credentials.
99
+ - Header values are registered with the redactor automatically, so a resolved token never appears in reports, evidence bundles, or published URLs.
100
+
101
+ The same `${env.NAME}` placeholders work in every target surface: single-server `Server options`, per-server `Server options for "name"` in compositions, and `Target options for "name"` in parity comparisons.
102
+
103
+ Multi-server compositions use named server declarations and per-test routing:
104
+
105
+ ```text
106
+ Named server "catalog": node services/catalog.js
107
+ Named server "billing": https://qa.example.com/billing/mcp
108
+
109
+ Server options for "billing":
110
+ headers:
111
+ X-Tenant: qa
112
+
113
+ Test: "catalog lookup"
114
+ On server "catalog"
115
+ Call tool "search"
116
+ ```
117
+
118
+ The YAML equivalent is a top-level `servers` mapping plus `server` on a test. A composition requires at least two named servers, and an unknown `On server` name is rejected during compilation.
119
+
65
120
  Parity targets use the same connection grammar:
66
121
 
67
122
  ```text
@@ -77,6 +132,8 @@ Suite-level YAML fields have direct equivalents:
77
132
 
78
133
  ```text
79
134
  Default timeout: 10 seconds
135
+ Budget: p95 500ms over 20 calls
136
+ Budget for "order lookup": p50 300ms over 20 calls
80
137
  Redact: "secret-value", "token-value"
81
138
  Snapshots: snapshots.json
82
139
  Ignore snapshot paths: "$.createdAt", "$.requestId"
@@ -94,11 +151,7 @@ Client behavior:
94
151
  approved: true
95
152
  ```
96
153
 
97
- ## Imports
98
-
99
- ```
100
-
101
- Per-test scripted responses (override `Client behavior:` for one test):
154
+ Per-test scripted responses override `Client behavior:` for one test:
102
155
 
103
156
  ```text
104
157
  When the server asks for input, respond "accept" with:
@@ -106,7 +159,10 @@ When the server asks for input, respond "accept" with:
106
159
  When the server asks for input, respond "decline"
107
160
  When the server requests sampling, respond "scripted text"
108
161
  ```
109
- text
162
+
163
+ ## Imports
164
+
165
+ ```text
110
166
  Import flows from "./shared/customer-flows.mcpr"
111
167
  ```
112
168
 
@@ -183,6 +239,7 @@ Send "ping"
183
239
  ```text
184
240
  Expect it succeeds
185
241
  Expect an error
242
+ Expect the call to finish within 800ms
186
243
  Expect "structuredContent.total" equals 2
187
244
  Expect "content[0].text" contains "complete"
188
245
  Expect "items" exists
@@ -16,7 +16,7 @@ Or with an explicit root:
16
16
  mcprigor serve path/to/project
17
17
  ```
18
18
 
19
- The server speaks MCP over stdio. Typical client configuration:
19
+ The server speaks MCP over stdio (`mcp` is an alias for `serve`). Typical client configuration:
20
20
 
21
21
  ```json
22
22
  {
@@ -0,0 +1,62 @@
1
+ # Scheduled production monitoring
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ Turn an HTTP MCP suite into a continuous production check:
6
+
7
+ ```bash
8
+ mcprigor monitor tests/prod.mcpr \
9
+ --every 15m \
10
+ --notify https://alerts.example.com/hooks/mcprigor
11
+ ```
12
+
13
+ The monitor runs immediately, then at the fixed interval until stopped. Durations accept `ms`, `s`, `m`, or `h`, with a minimum interval of one second.
14
+
15
+ For operational safety, monitoring requires a Streamable HTTP target. It refuses stdio suites so a long-running process cannot repeatedly spawn local commands by accident.
16
+
17
+ ## Notification policies
18
+
19
+ ```bash
20
+ mcprigor monitor tests/prod.mcpr --every 5m --notify URL --notify-on change
21
+ ```
22
+
23
+ `--notify-on` supports:
24
+
25
+ - `change` (default): first failure, then failure/recovery transitions;
26
+ - `failure`: every failed run;
27
+ - `recovery`: transitions from failed to passed;
28
+ - `always`: every run.
29
+
30
+ Webhook requests are JSON POSTs with a 15-second timeout:
31
+
32
+ ```json
33
+ {
34
+ "source": "mcprigor",
35
+ "event": "monitor.failure",
36
+ "suite": "tests/prod.mcpr",
37
+ "run": 12,
38
+ "status": "failed",
39
+ "startedAt": "2026-08-30T12:00:00.000Z",
40
+ "durationMs": 842,
41
+ "summary": { "passed": 7, "failed": 1, "skipped": 0, "blocked": 0 },
42
+ "failures": [{ "name": "order lookup", "error": "..." }]
43
+ }
44
+ ```
45
+
46
+ A non-2xx webhook response fails the monitor with `MCP-MONITOR-003`; notification loss is never silently ignored.
47
+
48
+ ## History and trends
49
+
50
+ Every monitoring run appends to `.mcprigor/workspace-history.jsonl`, so existing `mcprigor trends`, PDF/CSV exports, flaky detection, latency budgets, and regression analysis include production monitoring evidence.
51
+
52
+ ## Bounded runs
53
+
54
+ For smoke tests, cron jobs, and CI validation:
55
+
56
+ ```bash
57
+ mcprigor monitor tests/prod.mcpr --every 1m --max-runs 1
58
+ ```
59
+
60
+ `SIGINT` and `SIGTERM` stop the interval cleanly after active MCP sessions shut down.
61
+
62
+ Use an authenticated suite target (`headers` or `Token from:`), retain history according to your evidence policy, and send webhooks only to reviewed HTTPS endpoints.
@@ -0,0 +1,103 @@
1
+ # Multi-server compositions
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ Real MCP clients mount several servers together. MCP Rigor compositions test each named server and govern the combined tool, resource, and prompt namespace as one fleet.
6
+
7
+ ## Declare named servers
8
+
9
+ ```text
10
+ MCP Test 1
11
+ Suite: "Checkout fleet"
12
+
13
+ Named server "catalog": node services/catalog.js
14
+ Named server "billing": https://qa.example.com/billing/mcp
15
+
16
+ Server options for "billing":
17
+ Token from: node scripts/get-qa-token.mjs
18
+
19
+ Test: "catalog search works"
20
+ On server "catalog"
21
+ Call tool "search" with:
22
+ query: "widget"
23
+ Expect "structuredContent.total" is a number
24
+
25
+ Test: "billing responds"
26
+ On server "billing"
27
+ Send "ping"
28
+ Expect it succeeds
29
+ ```
30
+
31
+ `On server` routes every action in that test to the selected server. Tests without `On server` continue to use the legacy/default `Server:` or `MCP URL:` target. Cross-test dependencies and exported values continue to work across named servers.
32
+
33
+ YAML parity:
34
+
35
+ ```yaml
36
+ version: 1
37
+ name: Checkout fleet
38
+ target:
39
+ transport: stdio
40
+ command: node
41
+ args: [services/gateway.js]
42
+ servers:
43
+ catalog:
44
+ transport: stdio
45
+ command: node
46
+ args: [services/catalog.js]
47
+ billing:
48
+ transport: streamable-http
49
+ url: https://qa.example.com/billing/mcp
50
+ tests:
51
+ - name: catalog search works
52
+ server: catalog
53
+ steps:
54
+ - tool:
55
+ name: search
56
+ arguments: { query: widget }
57
+ ```
58
+
59
+ Unknown server names fail validation before a test starts.
60
+
61
+ ## Check the live composition
62
+
63
+ ```bash
64
+ mcprigor composition-check tests/fleet.mcpr
65
+ ```
66
+
67
+ The check discovers all named servers and reports:
68
+
69
+ - `MCP-COMP-001`: duplicate tool name;
70
+ - `MCP-COMP-002`: conflicting input/output schemas for the same tool (breaking);
71
+ - `MCP-COMP-003`: duplicate resource URI or URI template (breaking);
72
+ - `MCP-COMP-004`: duplicate prompt name.
73
+
74
+ MCP Rigor reports collisions; it never silently renames or chooses a winning server.
75
+
76
+ ## Create a combined fleet lock
77
+
78
+ ```bash
79
+ mcprigor composition-discover tests/fleet.mcpr \
80
+ --out contracts/checkout.composition.lock.yaml
81
+ ```
82
+
83
+ The lock embeds each named server's ordinary discovery contract, the cross-server issue set, and a stable fleet fingerprint. Volatile discovery timestamps and diagnostics do not change the combined fingerprint. Writes are atomic: all servers must be discovered successfully before the previous lock is replaced.
84
+
85
+ ## Gate fleet drift in CI
86
+
87
+ ```bash
88
+ mcprigor composition-drift tests/fleet.mcpr \
89
+ --against contracts/checkout.composition.lock.yaml \
90
+ --fail-on breaking
91
+ ```
92
+
93
+ The drift report combines:
94
+
95
+ - server additions and removals;
96
+ - each server's tool/resource/prompt contract changes;
97
+ - newly introduced or resolved cross-server conflicts.
98
+
99
+ `--fail-on` accepts `breaking` (default), `potentially-breaking`, `any`, or `none`, matching the single-server drift gate. Add `--json report.json` for machine-readable CI evidence.
100
+
101
+ ## Composition versus parity
102
+
103
+ Use a **composition** when several servers are mounted together and their namespaces interact. Use **transport parity** when the same logical server is exposed through alternate targets such as local stdio and deployed HTTP. The `servers` and `targets` fields remain intentionally separate.
@@ -1,4 +1,4 @@
1
- # Plain-language cookbook
1
+ # Natural-language cookbook
2
2
 
3
3
  Copy a pattern, replace the names and values, then run `mcprigor check FILE`.
4
4
 
@@ -187,7 +187,7 @@ Run with `mcprigor parity FILE`.
187
187
 
188
188
  ## Test a server that needs a bearer token
189
189
 
190
- Point the suite at the deployed endpoint and pass the token through an environment variable. Never paste a real token into a test file.
190
+ Point the suite at the deployed endpoint and pass the token through an environment variable. The `${env.NAME}` placeholder is replaced with the value of the `NAME` environment variable before the suite connects, so no real token ever lives in the test file.
191
191
 
192
192
  ```text
193
193
  MCP Test 1
@@ -210,6 +210,15 @@ Run it with the token in the environment:
210
210
  QA_TOKEN=... mcprigor test orders.mcpr
211
211
  ```
212
212
 
213
+ The placeholder can sit inside a larger string (`"Bearer ${env.QA_TOKEN}"`) or be the whole value, and any header works the same way — API keys, custom tenant headers, and so on:
214
+
215
+ ```text
216
+ Server options:
217
+ headers:
218
+ X-Api-Key: "${env.API_KEY}"
219
+ X-Tenant: "acme"
220
+ ```
221
+
213
222
  Three guarantees come with this pattern:
214
223
 
215
224
  - if `QA_TOKEN` is not set, the run stops with `Environment variable not found: QA_TOKEN` instead of sending an empty header;
@@ -238,7 +247,7 @@ Test: "Search behaves the same"
238
247
 
239
248
  ## When the token must be fetched first
240
249
 
241
- MCP Rigor does not perform OAuth login flows or token exchanges itself; tests stay deterministic and secrets stay outside test files. When a short-lived token must be acquired (client-credentials exchange, cloud CLI, vault), fetch it in the step before the run:
250
+ When a short-lived token must be acquired non-interactively (client-credentials exchange, cloud CLI, vault), fetch it in the step before the run:
242
251
 
243
252
  ```bash
244
253
  QA_TOKEN=$(curl -s -X POST https://auth.example.com/oauth/token \
@@ -256,7 +265,29 @@ In CI, do the same in the workflow:
256
265
  run: npx mcprigor test tests/*.mcpr
257
266
  ```
258
267
 
259
- Interactive browser-redirect OAuth is out of scope by design: an acceptance run must be repeatable without a human in the loop.
268
+ For CI and other unattended runs, obtain the token non-interactively as above. When a real user must sign in through a browser, use interactive OAuth instead (next recipe).
269
+
270
+ ## Sign in through the browser (interactive OAuth)
271
+
272
+ When a server requires a real user login, let MCP Rigor run the browser authorization-code flow once and carry the authorized session — with automatic refresh — into every test in the run:
273
+
274
+ ```text
275
+ MCP URL: https://app.example.com/mcp
276
+
277
+ Server options:
278
+ OAuth: oauth
279
+
280
+ Test: "an authenticated call succeeds"
281
+ Call tool "find_order" with:
282
+ orderId: "A-1001"
283
+ Expect "structuredContent.status" equals "shipped"
284
+ ```
285
+
286
+ ```bash
287
+ mcprigor test orders.mcpr
288
+ ```
289
+
290
+ Your browser opens to the identity provider; after you sign in, the tokens are held in memory (never written to disk, always redacted) and reused for the whole suite. For pre-registered clients or specific scopes, use the block form with `clientId`, `clientSecret: "${env.…}"`, and `scope`. Because it needs a human, keep interactive OAuth for local runs and use a non-interactive credential in CI. See the [Authentication guide](AUTHENTICATION.md) for the full flow.
260
291
 
261
292
  ## Fetch an OAuth token before connecting
262
293
 
@@ -302,6 +333,34 @@ Test: "summaries embed the sampled text"
302
333
 
303
334
  Suite-wide defaults live in a `Client behavior:` block; these per-test lines override them for one test.
304
335
 
336
+ ## Set latency budgets and catch slow releases
337
+
338
+ Fail a single call that takes too long:
339
+
340
+ ```text
341
+ Test: "order lookup is fast"
342
+ Call tool "find_order" with:
343
+ orderId: "A-1001"
344
+ Expect the call to finish within 800ms
345
+ ```
346
+
347
+ Set suite-level budgets measured as percentiles over recorded run history (the same history behind `mcprigor trends`):
348
+
349
+ ```text
350
+ Budget: p95 500ms over 20 calls
351
+ Budget for "order lookup is fast": p50 300ms
352
+ ```
353
+
354
+ Budgets are judged after each `mcprigor test` run and fail the run when a percentile exceeds its budget. Until enough history exists they report as pending instead of guessing.
355
+
356
+ Gate CI on latency regressions against the trend baseline — no budget numbers needed:
357
+
358
+ ```bash
359
+ mcprigor test suite.mcpr --fail-on-regression
360
+ ```
361
+
362
+ A test regresses when it runs slower than 1.5x its historical median (with a 50 ms floor so micro-tests don't trip on jitter).
363
+
305
364
  ## Match a snapshot
306
365
 
307
366
  ```text
@@ -0,0 +1,85 @@
1
+ # Performance governance
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ MCP Rigor turns MCP latency into a deterministic release gate. It supports immediate per-call limits, percentile budgets over recorded history, and automatic regression detection against each test's historical baseline.
6
+
7
+ ## Limit one call
8
+
9
+ Add a latency assertion after an action:
10
+
11
+ ```text
12
+ Test: "order lookup stays interactive"
13
+ Call tool "find_order" with:
14
+ orderId: "A-1001"
15
+ Expect the call to finish within 800ms
16
+ ```
17
+
18
+ The measured duration covers the live MCP request and response. A call over the limit fails with `MCP-PERF-001` and reports the measured and allowed duration.
19
+
20
+ YAML parity:
21
+
22
+ ```yaml
23
+ - tool:
24
+ name: find_order
25
+ arguments:
26
+ orderId: A-1001
27
+ assert:
28
+ maxDurationMs: 800
29
+ ```
30
+
31
+ ## Set percentile budgets
32
+
33
+ A percentile budget uses successful durations from recent recorded runs plus the current run:
34
+
35
+ ```text
36
+ Budget: p95 500ms over 20 calls
37
+ Budget for "order lookup stays interactive": p50 300ms over 20 calls
38
+ ```
39
+
40
+ - A suite-wide budget applies independently to every test.
41
+ - A named budget applies only to that test.
42
+ - MCP Rigor uses the deterministic nearest-rank percentile.
43
+ - Fewer than three usable samples reports the budget as pending instead of guessing.
44
+ - A measured percentile over budget fails the CLI run.
45
+
46
+ YAML parity:
47
+
48
+ ```yaml
49
+ budgets:
50
+ - test: "*"
51
+ percentile: 95
52
+ maxMs: 500
53
+ window: 20
54
+ - test: order lookup stays interactive
55
+ percentile: 50
56
+ maxMs: 300
57
+ window: 20
58
+ ```
59
+
60
+ ## Fail on regression without maintaining thresholds
61
+
62
+ ```bash
63
+ mcprigor test tests/orders.mcpr --fail-on-regression
64
+ ```
65
+
66
+ For every successful test, MCP Rigor compares the current duration with the median of its latest successful history. The gate requires at least five samples and reports a regression when the current run exceeds both:
67
+
68
+ - 1.5× the historical median; and
69
+ - the historical median plus 50 ms.
70
+
71
+ The absolute floor prevents very small tests from failing because of ordinary scheduler jitter.
72
+
73
+ ## CI example
74
+
75
+ ```yaml
76
+ - run: npx mcprigor test tests/orders.mcpr --fail-on-regression --junit reports/orders.xml
77
+ ```
78
+
79
+ Explicit budgets answer “is this fast enough?” The regression gate answers “did this release make it materially slower?” Teams commonly use both.
80
+
81
+ ## History source
82
+
83
+ CLI, QA workspace, and MCP-server-driven test runs append to `.mcprigor/workspace-history.jsonl`. The same data powers `mcprigor trends`, CSV/PDF trend exports, flaky detection, percentile budgets, and regression baselines.
84
+
85
+ Commit the test and its budget declarations. Treat history as a CI artifact or retained workspace file according to your evidence policy.
@@ -0,0 +1,53 @@
1
+ # Shareable hosted reports
2
+
3
+ > Available since `1.5.0`.
4
+
5
+ Turn a test run into a URL anyone can open — no repository access, no CI login:
6
+
7
+ ```bash
8
+ export NETLIFY_AUTH_TOKEN=... # personal or CI token
9
+ mcprigor publish tests/catalog.mcpr --site your-netlify-site
10
+ ```
11
+
12
+ `publish` runs the suite, builds the readable HTML report **with the clickable request/response session timeline**, deploys it to your Netlify site with the dependency-free digest API, waits until the deploy is live, and prints the shareable URL:
13
+
14
+ ```
15
+ Published report: https://68b1c2--your-site.netlify.app
16
+ ```
17
+
18
+ Each publish is a normal Netlify deploy of your own site, so access control, custom domains, deploy previews, and retention follow your existing hosting configuration. Unchanged files are skipped automatically via content digests.
19
+
20
+ ## Options
21
+
22
+ ```bash
23
+ mcprigor publish suite.mcpr --site SITE [--include-json] [--test NAME] [--env qa]
24
+ mcprigor publish suite.mcpr --out reports/latest
25
+ ```
26
+
27
+ - `--site` — Netlify site ID or name. The token comes only from `NETLIFY_AUTH_TOKEN` (or `MCPRIGOR_PUBLISH_TOKEN`); tokens are never accepted as command-line flags.
28
+ - `--out DIR` — write the same bundle to a local directory instead of (or in addition to) hosting it. Serve it from any static host: S3, GitHub Pages, nginx, an artifact store.
29
+ - `--include-json` — also publish `result.json` for dashboards and programmatic consumers.
30
+ - `--test`, `--env`, `--command`, `--url` — the same run-selection options as `mcprigor test`.
31
+
32
+ The exit code still reflects the run (`0` passed, `1` failed), so `publish` can replace `test` in a pipeline step that both gates and shares.
33
+
34
+ ## Publishing from the QA workspace UI
35
+
36
+ The visual workspace (`mcprigor workspace`) exposes the same features without the command line:
37
+
38
+ - **HTML report** — every finished test run has an `HTML report` button that opens the full report, including the clickable session timeline, in a new tab.
39
+ - **Publish** — start the workspace with hosting configured and a `Publish` button appears next to the export buttons:
40
+
41
+ ```bash
42
+ export MCPRIGOR_PUBLISH_SITE=your-netlify-site
43
+ export NETLIFY_AUTH_TOKEN=...
44
+ mcprigor workspace
45
+ ```
46
+
47
+ Clicking it deploys the selected run's report and opens the shareable URL; a `View published ↗` link stays on the run for re-opening or copying. Without both variables the button is hidden and the API answers with a clear configuration message — the token itself never reaches the browser.
48
+
49
+ ## Security
50
+
51
+ - The report is produced by the same pipeline as `--html`: secrets and configured redaction patterns are removed **before** the report exists.
52
+ - Publishing makes the report as public as the target site. Use a password-protected or team-restricted Netlify site for internal results.
53
+ - The hosting token is read from the environment at the last moment and is never echoed, logged, or stored.
package/docs/QA-GUIDE.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # QA guide
2
2
 
3
- Use this page as a short everyday checklist. For examples, open the [plain-language cookbook](PLAIN-LANGUAGE-COOKBOOK.md).
3
+ Use this page as a short everyday checklist. For examples, open the [natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md).
4
4
 
5
5
  ## Everyday workflow
6
6
 
@@ -121,7 +121,7 @@ Use values as `${row.input}` and `${row.expected}`.
121
121
  ## Where to go next
122
122
 
123
123
  - [Getting started](GETTING-STARTED.md)
124
- - [Plain-language cookbook](PLAIN-LANGUAGE-COOKBOOK.md)
124
+ - [Natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md)
125
125
  - [Guided authoring](GUIDED-AUTHORING.md)
126
126
  - [Data and reusable flows](DATA-AND-REUSE.md)
127
127
  - [Troubleshooting](TROUBLESHOOTING.md)
@@ -30,7 +30,7 @@ An empty folder shows a three-step welcome screen. Choose **+ New test file**
30
30
  2. Edit the plain-language scenario. The editor provides syntax highlighting, line numbers, and autocomplete: top-level declarations at the start of a line, actions and assertions when indented, and comparison phrases after `Expect "field"`. Accept with `Tab` or `Enter`; force the list open with `Ctrl+Space`.
31
31
  3. Choose **Validate** (`Ctrl/⌘+S` saves, `Ctrl/⌘+Enter` runs). A wording problem highlights the offending line and moves the cursor to it.
32
32
  4. Choose **▶ Run tests** or **Parity**.
33
- 5. Review the results panel: per-file pass/fail with durations; select a file for its full report.
33
+ 5. Review the results panel: per-file pass/fail with durations; select a file for its full report. Drag either panel divider to resize the file list, editor, and results areas; widths persist across reloads, arrow keys resize a focused divider, and double-click resets the layout.
34
34
 
35
35
  The editor marks unsaved changes; running or validating saves them first. If the file changed elsewhere after you opened it, the workspace refuses to overwrite it and asks you to reload.
36
36
 
@@ -52,6 +52,10 @@ Test runs are recorded in `.mcprigor/workspace-history.jsonl` (most recent 2000
52
52
 
53
53
  One search box filters all three tabs. It matches suite names, test names, and error text, and highlights matches, so you can answer questions like "when did `delivered` start failing?" without leaving the browser.
54
54
 
55
+ Completed runs expose **PDF**, **CSV**, and **JUnit XML** downloads. The Trends tab exports a rich trends PDF, aggregate CSV, or raw history CSV. PDF reports include summary cards, pass-rate visuals, per-test detail, failures, durations, and evidence identifiers.
56
+
57
+ An **HTML report** button opens the full report — including the clickable request/response session timeline — in a new tab. When the workspace is started with `MCPRIGOR_PUBLISH_SITE` and `NETLIFY_AUTH_TOKEN` set, a **Publish** button also appears: one click hosts the report at a shareable static URL and keeps a `View published ↗` link on the run. See [Shareable hosted reports](PUBLISHING.md).
58
+
55
59
  ## What is available
56
60
 
57
61
  - Creating, renaming, and editing `.mcpr` suites (YAML and JSON suites are listed and editable too)