mcprigor 1.4.0 → 1.5.1

package/docs/README.md

@@ -5,7 +5,7 @@ Choose the path that matches your role.
 ## QA authors
 
 1. [Getting started](GETTING-STARTED.md) — install, connect, and run your first test.
-2. [Plain-language cookbook](PLAIN-LANGUAGE-COOKBOOK.md) — copy-ready scenarios and assertions.
+2. [Natural-language cookbook](NATURAL-LANGUAGE-COOKBOOK.md) — copy-ready scenarios and assertions.
 3. [QA workspace](QA-WORKSPACE.md) — edit and run tests in a browser.
 4. [Data and reusable flows](DATA-AND-REUSE.md) — tables, files, functions, and shared flows.
 5. [Troubleshooting](TROUBLESHOOTING.md) — understand errors and fix common failures.
@@ -13,11 +13,18 @@ Choose the path that matches your role.
 ## Test and platform engineers
 
 - [Engineer setup](ENGINEER-SETUP.md) — targets, credentials, project layout, and CI.
+- [Authentication and secrets](AUTHENTICATION.md) — test protected servers with bearer tokens, API keys, OAuth, and `${env.NAME}` placeholders.
 - [CLI reference](CLI-REFERENCE.md) — commands, options, outputs, and exit codes.
-- [Language reference](LANGUAGE-SPEC.md) — complete deterministic `.mcpr` syntax.
+- [Language reference](LANGUAGE-SPEC.md) — complete deterministic `.mcpr` syntax, in enforced parity with YAML via `mcprigor convert`.
 - [State and dependencies](STATE-AND-DEPENDENCIES.md) — share outputs across tests and runs.
 - [Data engineering](DATA-ENGINEERING.md) — types, filters, joins, samples, and caches.
 - [Transport parity](TRANSPORT-PARITY.md) — compare stdio and Streamable HTTP.
+- [Performance governance](PERFORMANCE-GOVERNANCE.md) — call limits, percentile budgets, and regression gates.
+- [Multi-server compositions](MULTI-SERVER-COMPOSITIONS.md) — route tests across a fleet and gate combined drift.
+- [Coverage](COVERAGE.md) — find untested MCP surfaces and input-schema branches.
+- [Scheduled monitoring](MONITORING.md) — continuously test production HTTP endpoints and notify webhooks.
+- [GitHub Action](GITHUB-ACTION.md) — test, drift, flaky warnings, and rich pull-request comments.
+- [Shareable hosted reports](PUBLISHING.md) — `mcprigor publish` turns a run into a static report URL.
 
 ## Contracts and evidence
 
@@ -30,6 +37,7 @@ Choose the path that matches your role.
 
 - [Extension SDK](EXTENSION-SDK.md)
 - [Stable error model](ERROR-MODEL.md)
+- [Deterministic security audit](SECURITY-AUDIT.md) — built-in probe pack, severity gate, and rich reports.
 - [Security and retention](SECURITY-AND-RETENTION.md)
 - [Compatibility policy](COMPATIBILITY.md)
 

package/docs/SECURITY-AUDIT.md

@@ -0,0 +1,77 @@
+# Deterministic security audit
+
+> Available since `1.5.0`.
+
+`mcprigor audit` runs a fixed security and permissions probe pack against an MCP server and produces an auditable score. No AI chooses payloads or interprets outcomes.
+
+## Run the safe default pack
+
+Use a suite's configured target:
+
+```bash
+mcprigor audit tests/server.mcpr
+```
+
+Or supply a target directly:
+
+```bash
+mcprigor audit --url https://qa.example.com/mcp
+mcprigor audit --command "node dist/server.js"
+```
+
+The non-destructive default probes:
+
+| Probe | What MCP Rigor verifies |
+|---|---|
+| Malformed parameters | Invalid `tools/call` parameters are rejected |
+| Tool-name spoofing | An unknown administrative-looking tool cannot be invoked |
+| Oversized payload | A deterministic 1 MiB payload is rejected |
+| Path traversal | Resource reads reject Unix, Windows, and relative traversal URIs |
+| Tool inventory | Advertised tools are listed for opt-in injection/canary testing |
+
+## Opt into reviewed tool calls
+
+MCP Rigor never guesses that a tool is safe. Tool execution is disabled unless you allow an exact name:
+
+```bash
+mcprigor audit tests/server.mcpr \
+  --allow-tool search \
+  --allow-tool summarize
+```
+
+Only allow read-only, non-destructive tools in a disposable test environment. An allowed tool receives a fixed prompt-injection payload containing a deterministic canary. MCP Rigor checks whether the response follows or reflects the payload and whether it exposes the canary. Report evidence replaces the canary with `[CANARY REDACTED]`.
+
+## CI severity gate
+
+```bash
+mcprigor audit tests/server.mcpr --fail-on high
+```
+
+Values are `critical`, `high`, `medium`, `low`, and `none`; the default is `high`. The command exits nonzero when a failed finding meets or exceeds the selected severity.
+
+Scoring starts at 100:
+
+- critical: −35
+- high: −20
+- medium: −10
+- low: −4
+- skipped: no deduction
+
+Skipped probes remain visible with the exact `--allow-tool` needed. They are not represented as passes.
+
+## Reports
+
+```bash
+mcprigor audit tests/server.mcpr \
+  --pdf reports/security.pdf \
+  --json reports/security.json \
+  --csv reports/security.csv
+```
+
+The rich PDF includes score, grade, severity cards, per-finding status, explanations, and bounded evidence. Markdown output is available with `--markdown`.
+
+## Interpreting results
+
+The audit is deterministic black-box evidence, not a certification or a replacement for threat modeling. A passing probe proves the observed server rejected that exact payload under that exact configuration. Review skipped probes, environment permissions, downstream model behavior, and business authorization separately.
+
+Run intrusive tool probes only in an isolated environment. Do not expose production mutation tools through `--allow-tool`.

package/editors/mcpr-language.json

@@ -12,11 +12,11 @@
       "name": "keyword.control.version.mcpr"
     },
     {
-      "match": "^\\s*(Suite|Server|Server options|MCP URL|Compare target|Target options for|Default timeout|Redact|Snapshots|Ignore snapshot paths|Client behavior|Import flows from|Flow|Inputs|Outputs|Test|Scenario|Id|Depends on|Require|Require protocol|Variables|Skip|Setup|Steps|Cleanup|Data source):?",
+      "match": "^\\s*(Suite|Server|Named server|Server options|Server options for|MCP URL|Compare target|Target options for|Default timeout|Budget|Redact|Snapshots|Ignore snapshot paths|Client behavior|Import flows from|Flow|Inputs|Outputs|Test|Scenario|Id|Depends on|Require|Require protocol|Variables|Skip|Setup|Steps|Cleanup|Data source):?",
       "name": "keyword.control.mcpr"
     },
     {
-      "match": "^\\s*(Call tool|When the server asks for input|When the server requests sampling|Read resource|Get prompt|Send|Expect|Save|Export|Set|Use flow|For each row|Wait up to|Wait for notification|Subscribe to resource|Unsubscribe from resource|Set log level|List all|Get task|List tasks|Cancel task|From CSV|From JSON|From YAML|From Excel|From REST|Column|Derive|Keep rows where|Sample|Cache this source)\\b",
+      "match": "^\\s*(Call tool|On server|When the server asks for input|When the server requests sampling|Read resource|Get prompt|Send|Expect|Save|Export|Set|Use flow|For each row|Wait up to|Wait for notification|Subscribe to resource|Unsubscribe from resource|Set log level|List all|Get task|List tasks|Cancel task|From CSV|From JSON|From YAML|From Excel|From REST|Column|Derive|Keep rows where|Sample|Cache this source)\\b",
       "name": "keyword.other.mcpr"
     },
     {

package/editors/vscode/package.json

@@ -2,7 +2,7 @@
   "name": "mcprigor",
   "displayName": "MCP Rigor",
   "description": "Syntax highlighting and inline validation for MCP Rigor (.mcpr) natural-language MCP test suites.",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "publisher": "fusionone",
   "license": "Apache-2.0",
   "repository": { "type": "git", "url": "https://github.com/FusionOnePlatform/mcprigor" },

package/editors/vscode/syntaxes/mcpr.tmLanguage.json

@@ -12,11 +12,11 @@
       "name": "keyword.control.version.mcpr"
     },
     {
-      "match": "^\\s*(Suite|Server|Server options|MCP URL|Compare target|Target options for|Default timeout|Redact|Snapshots|Ignore snapshot paths|Client behavior|Import flows from|Flow|Inputs|Outputs|Test|Scenario|Id|Depends on|Require|Require protocol|Variables|Skip|Setup|Steps|Cleanup|Data source):?",
+      "match": "^\\s*(Suite|Server|Named server|Server options|Server options for|MCP URL|Compare target|Target options for|Default timeout|Budget|Redact|Snapshots|Ignore snapshot paths|Client behavior|Import flows from|Flow|Inputs|Outputs|Test|Scenario|Id|Depends on|Require|Require protocol|Variables|Skip|Setup|Steps|Cleanup|Data source):?",
       "name": "keyword.control.mcpr"
     },
     {
-      "match": "^\\s*(Call tool|When the server asks for input|When the server requests sampling|Read resource|Get prompt|Send|Expect|Save|Export|Set|Use flow|For each row|Wait up to|Wait for notification|Subscribe to resource|Unsubscribe from resource|Set log level|List all|Get task|List tasks|Cancel task|From CSV|From JSON|From YAML|From Excel|From REST|Column|Derive|Keep rows where|Sample|Cache this source)\\b",
+      "match": "^\\s*(Call tool|On server|When the server asks for input|When the server requests sampling|Read resource|Get prompt|Send|Expect|Save|Export|Set|Use flow|For each row|Wait up to|Wait for notification|Subscribe to resource|Unsubscribe from resource|Set log level|List all|Get task|List tasks|Cancel task|From CSV|From JSON|From YAML|From Excel|From REST|Column|Derive|Keep rows where|Sample|Cache this source)\\b",
       "name": "keyword.other.mcpr"
     },
     {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcprigor",
-  "version": "1.4.0",
+  "version": "1.5.1",
   "description": "Plain-language MCP testing with isolated extensions and a local QA workspace",
   "type": "module",
   "bin": {
@@ -47,7 +47,6 @@
   "dependencies": {
     "@modelcontextprotocol/sdk": "1.27.1",
     "ajv": "^8.18.0",
-    "mcprigor": "^1.3.1",
     "yaml": "^2.8.2"
   },
   "overrides": {

package/workspace-assets/app.js

@@ -1,4 +1,4 @@
-const state = { csrf: '', path: '', etag: '', dirty: false, files: [], selected: new Set(), run: null, history: [], tab: 'run', runSel: -1 };
+const state = { csrf: '', path: '', etag: '', dirty: false, files: [], selected: new Set(), run: null, history: [], tab: 'run', runSel: -1, canPublish: false, publishing: false };
 const $ = id => document.getElementById(id);
 
 /* ---------- API ---------- */
@@ -14,6 +14,7 @@ async function api(path, options = {}) {
 async function start() {
   const boot = await api('/api/v1/bootstrap');
   state.csrf = boot.csrf;
+  state.canPublish = (boot.capabilities || []).includes('publish');
   $('version').textContent = `v${boot.version}`;
   $('workspace').textContent = `Folder: ${boot.root}`;
   $('connection').classList.add('ok');
@@ -613,16 +614,46 @@ start().catch(show);
 function updateExportBar() {
   const run = state.run;
   const bar = $('export-run');
-  const index = state.runSel || 0;
+  const index = state.runSel >= 0 ? state.runSel : 0;
   const item = run && run.items && run.items[index];
-  bar.hidden = !(run && run.mode === 'test' && item && item.status !== 'running' && item.tests);
+  const ready = run && run.mode === 'test' && item && item.status !== 'running' && item.tests;
+  bar.hidden = !ready;
+  const publish = $('publish-run');
+  const link = $('published-link');
+  publish.hidden = !(ready && state.canPublish);
+  publish.disabled = state.publishing;
+  publish.textContent = state.publishing ? 'Publishing…' : (item && item.publishedUrl ? 'Publish again' : 'Publish');
+  link.hidden = !(ready && item.publishedUrl);
+  if (item && item.publishedUrl) link.href = item.publishedUrl;
 }
 function exportRun(format) {
   const run = state.run;
   if (!run) return;
-  const index = state.runSel || 0;
+  const index = state.runSel >= 0 ? state.runSel : 0;
   window.open(`/api/v1/export/run?id=${encodeURIComponent(run.id)}&item=${index}&format=${format}`, '_blank');
 }
+async function publishRun() {
+  const run = state.run;
+  if (!run || state.publishing) return;
+  const index = state.runSel >= 0 ? state.runSel : 0;
+  state.publishing = true; updateExportBar();
+  try {
+    const value = await api('/api/v1/publish', { method: 'POST', body: JSON.stringify({ id: run.id, item: index }) });
+    const item = run.items[index];
+    if (item) item.publishedUrl = value.url;
+    window.open(value.url, '_blank', 'noopener');
+  } catch (error) {
+    toast(error.message || 'Publish failed', true);
+  } finally {
+    state.publishing = false; updateExportBar();
+  }
+}
+$('export-run-html').onclick = () => {
+  const run = state.run;
+  if (!run) return;
+  window.open(`/api/v1/report?id=${encodeURIComponent(run.id)}&item=${state.runSel >= 0 ? state.runSel : 0}`, '_blank');
+};
+$('publish-run').onclick = publishRun;
 $('export-run-pdf').onclick = () => exportRun('pdf');
 $('export-run-csv').onclick = () => exportRun('csv');
 $('export-run-junit').onclick = () => exportRun('junit');

package/workspace-assets/index.html

@@ -38,7 +38,7 @@
       <button id="batch-clear" class="ghost small">Clear selection</button>
     </div>
     <div class="sidebar-foot">
-      <a href="https://mcprigor.com/docs/plain-language-cookbook.html" target="_blank" rel="noreferrer">📖 Cookbook — copy-ready examples</a>
+      <a href="https://mcprigor.com/docs/natural-language-cookbook.html" target="_blank" rel="noreferrer">📖 Cookbook — copy-ready examples</a>
     </div>
   </aside>
   <div class="splitter" id="split-left" role="separator" aria-orientation="vertical" aria-label="Resize file list" tabindex="0"></div>
@@ -95,6 +95,9 @@
         <button id="export-run-pdf" class="ghost small" type="button">PDF</button>
         <button id="export-run-csv" class="ghost small" type="button">CSV</button>
         <button id="export-run-junit" class="ghost small" type="button">JUnit XML</button>
+        <button id="export-run-html" class="ghost small" type="button">HTML report</button>
+        <button id="publish-run" class="ghost small" type="button" hidden>Publish</button>
+        <a id="published-link" class="published-link" target="_blank" rel="noopener" hidden>View published ↗</a>
       </div>
       <div id="run-list"></div>
       <pre id="output" class="output"><span class="muted">Nothing has run yet.

package/workspace-assets/style.css

@@ -232,3 +232,5 @@ nav .suite-row button .rn:hover{opacity:1}
 .export-bar{display:flex;align-items:center;gap:6px;padding:8px 14px;border-bottom:1px solid var(--line-soft)}
 .export-label{font-size:11px;color:var(--text-dim);margin-right:2px}
 .export-bar .small{font-size:11px;padding:3px 10px}
+.published-link{font-size:11px;margin-left:4px;color:var(--accent, #2563eb);text-decoration:none}
+.published-link:hover{text-decoration:underline}