mcprigor 1.0.0-rc.1

package/docs/CONTRACT-DRIFT.md

@@ -0,0 +1,42 @@
+# Contract Drift Analysis
+
+MCP Rigor explains how a live MCP surface changed instead of only reporting a different fingerprint.
+
+## Check a lock against a live server
+
+```bash
+mcprigor contract-check mcp.lock.yaml --target server.mcpr
+```
+
+This is read-only. It discovers the server, compares it with the lock, and exits with status `1` when breaking changes exist.
+
+Markdown for a pull-request comment:
+
+```bash
+mcprigor contract-check mcp.lock.yaml --target server.mcpr --markdown --out contract-drift.md
+```
+
+## Offline comparison
+
+```bash
+mcprigor contract-diff old.lock.yaml new.lock.yaml
+mcprigor contract-diff old.lock.yaml new.lock.yaml --markdown
+```
+
+## Explicit update
+
+```bash
+mcprigor contract-update mcp.lock.yaml --target server.mcpr
+```
+
+The diff is displayed before the newly discovered contract replaces the lock. Review breaking changes before committing it.
+
+## Classifications
+
+**Breaking:** an operation or capability was removed; a required input was added; a known property was removed; a type changed; an enum option was removed; or a required prompt argument was added.
+
+**Potentially breaking:** protocol version, description, or ambiguous output/schema behavior changed.
+
+**Non-breaking:** an operation/capability was added; an optional input was added; a required input became optional; or an enum option was added.
+
+Findings use stable `MCP-DRIFT-*` codes and deterministic ordering. Complex JSON Schema implication is classified conservatively rather than overstating compatibility. Contract drift is application compatibility evidence, not formal MCP certification.

package/docs/DATA-AND-REUSE.md

@@ -0,0 +1,213 @@
+# Reuse, Utility Functions, and Data-Driven Testing
+
+This guide is for QA authors first and extension developers second.
+
+## Reusable flows
+
+A flow is a named group of ordinary test actions:
+
+```text
+Flow: "Verify addition"
+  Inputs: a, b, expected
+
+  Call tool "add" with:
+    a: "${a}"
+    b: "${b}"
+
+  Expect "structuredContent.sum" equals "${expected}"
+```
+
+Use it from any test:
+
+```text
+Use flow "Verify addition" with:
+  a: 2
+  b: 3
+  expected: 5
+```
+
+Flows can use other flows. Recursive flow calls are rejected before execution. Each invocation has its own prefixed variables, so reused actions do not overwrite another invocation's inputs.
+
+Flows may contain section markers:
+
+```text
+Flow: "Create and remove customer"
+  Inputs: email
+
+  Setup:
+  Call tool "prepare_tenant"
+
+  Steps:
+  Call tool "create_customer" with:
+    email: "${email}"
+
+  Cleanup:
+  Call tool "remove_customer" with:
+    email: "${email}"
+```
+
+Cleanup actions are marked as guaranteed and are attempted after a preceding action fails. Keep cleanup idempotent.
+
+## Deterministic built-in utilities
+
+```text
+Set "email" using "lowercase" with:
+  value: "${row.Email}"
+```
+
+Built-ins:
+
+- `lowercase`, `uppercase`, `trim`
+- `join`, `replace`, `length`
+- `number`, `text`, `json`, `round`
+- `urlEncode`, `base64`, `hash`
+
+They have no network or file access and do not use random values or the current time.
+
+## Custom utility functions
+
+Declare a reviewed module:
+
+```text
+Functions: tests/qa-functions.mjs
+```
+
+```javascript
+export function calculateTax({ amount, rate }) {
+  return Math.round(amount * rate * 100) / 100;
+}
+```
+
+Use it exactly like a built-in:
+
+```text
+Set "expectedTax" using "calculateTax" with:
+  amount: "${row.amount}"
+  rate: 0.08
+```
+
+Custom code is disabled by default:
+
+```bash
+mcprigor test tax.mcpr --allow-custom-code
+```
+
+Enabled modules run in constrained workers by default, with manifest checks, explicit permissions, JSON-only boundaries, memory limits, and timeouts. Worker isolation reduces risk but is not a hard sandbox for hostile code. Review, pin, and allowlist every extension before CI use. See [Extension SDK](EXTENSION-SDK.md).
+
+## Inline examples
+
+```text
+Test: "Calculator examples"
+  For each row:
+    | caseId  | a  | b  | expected |
+    | basic   | 2  | 3  | 5        |
+    | larger  | 20 | 22 | 42       |
+
+  Use flow "Verify addition" with:
+    a: "${row.a}"
+    b: "${row.b}"
+    expected: "${row.expected}"
+```
+
+Each row becomes an isolated test and report entry. `caseId` or `id` provides its display ID.
+
+## Files
+
+```text
+For each row in "data/cases.csv"
+For each row in "data/cases.json"
+For each row in "data/cases.yaml"
+For each row in "data/cases.xlsx" from sheet "Regression"
+```
+
+JSON/YAML files must contain an array of objects. Named source configuration can select a nested `path`.
+
+## Named data sources
+
+```text
+Data source: "regression cases"
+  provider: json
+  file: data/cases.json
+  path: cases
+
+Test: "Regression matrix"
+  For each row from "regression cases"
+  # actions using ${row.column}
+```
+
+Providers:
+
+- `inline`
+- `csv`
+- `json`
+- `yaml`
+- `excel`
+- `rest`
+- `google-sheets`
+- `sql` through a reviewed provider module
+- `plugin` for other systems
+
+## REST
+
+```text
+Data source: "API cases"
+  provider: rest
+  url: https://qa.example.com/cases
+  path: cases
+  headers:
+    Authorization: "Bearer ${env.CASES_TOKEN}"
+```
+
+Remote sources require explicit permission:
+
+```bash
+mcprigor test api.mcpr --allow-remote-data
+```
+
+Requests have a ten-second deadline. Responses must be JSON row arrays. Credential-shaped fields are redacted from reports.
+
+## Google Sheets
+
+```text
+Data source: "Sheet cases"
+  provider: google-sheets
+  spreadsheetId: "${env.SHEET_ID}"
+  range: Regression!A1:F100
+  accessToken: "${env.GOOGLE_ACCESS_TOKEN}"
+```
+
+Or provide `apiKey` for a public sheet. Google Sheets is fetched through the Values REST API and requires `--allow-remote-data`.
+
+## SQL and custom providers
+
+Database drivers are intentionally not bundled. Register a reviewed provider module:
+
+```text
+Data source: "Database cases"
+  provider: sql
+  module: test/providers/postgres-cases.mjs
+  query: SELECT case_id, input, expected FROM qa_cases
+```
+
+The module implements:
+
+```javascript
+export default {
+  async load(config, context) {
+    // Use parameterized, read-only queries and return object rows.
+    return rows;
+  }
+}
+```
+
+Run with `--allow-custom-code`. The same plugin interface can connect Jira, Xray, TestRail, or internal systems.
+
+## Safety and reproducibility
+
+- Default maximum: 1,000 rows; set a lower ceiling with `--max-rows`.
+- Local data files have a 10 MiB limit.
+- Every source gets a SHA-256 fingerprint.
+- Every expanded row stores its source, row number, row ID, and source fingerprint.
+- One MCP session is created per row by default.
+- Remote and executable providers are off by default.
+- Data values can contain sensitive business information; avoid placing secrets in `caseId` because IDs appear in reports.

package/docs/DATA-ENGINEERING.md

@@ -0,0 +1,96 @@
+# Better Data Engineering
+
+Data providers can now feed a deterministic transformation pipeline before rows become isolated tests:
+
+```text
+provider → cache → typed coercion → join → derived columns → filter → sample → row limit
+```
+
+## Typed columns and validation
+
+```yaml
+provider: csv
+file: customers.csv
+columns:
+  customerId:
+    type: string
+    required: true
+  spend:
+    type: number
+    required: true
+  active: boolean
+  joinedAt: date
+  tier:
+    type: string
+    enum: [gold, silver, bronze]
+```
+
+Types are `string`, `number`, `boolean`, `date`, and `json`. Errors include the row number and column name. Dates normalize to ISO 8601. Numbers must be finite.
+
+## Derived columns
+
+```yaml
+derive:
+  fullName: "${firstName} ${lastName}"
+  caseLabel: "${customerId}:${tier}"
+```
+
+Derivation is deterministic template substitution; it does not execute code.
+
+## Filtering
+
+```yaml
+where:
+  active: true
+  spend:
+    greaterThan: 100
+  tier:
+    in: [gold, silver]
+```
+
+Available predicates: `equals`, `notEquals`, `in`, `matches`, `greaterThan`, and `lessThan`. Multiple columns and predicates are ANDed.
+
+## Deterministic sampling
+
+```yaml
+sample:
+  count: 25
+  seed: 2025
+```
+
+Other modes:
+
+```yaml
+sample: { first: 10 }
+sample: { last: 10 }
+sample: { every: 5 }
+```
+
+Seeded sampling preserves original source order after selection so test order is stable.
+
+## Joins
+
+```yaml
+provider: csv
+file: orders.csv
+join:
+  provider: json
+  file: customers.json
+  on: customerId
+  kind: left
+  prefix: customer_
+```
+
+`inner` and `left` joins are supported. The join key must be named by `on`. Prefixing avoids column collisions. Base-row values win on collision.
+
+## Explicit provider caching
+
+```yaml
+provider: rest
+url: https://qa.example.com/cases
+cache: true
+```
+
+Caching is opt-in, process-local, and keyed by provider configuration. Cached rows are defensively cloned. It does not persist credentials or data to disk.
+
+Existing safety gates remain: remote sources need `--allow-remote-data`, SQL/custom providers need `--allow-custom-code`, and `--max-rows` caps the transformed dataset.

package/docs/ENGINEER-SETUP.md

@@ -0,0 +1,176 @@
+# Engineer setup and CI
+
+This guide covers the configuration engineers normally prepare once so QA authors can focus on scenarios.
+
+## Recommended repository layout
+
+```text
+acceptance-tests/
+  package.json
+  tests/
+    smoke.mcpr
+    regression.mcpr
+    parity.mcpr
+    shared-flows.mcpr
+    data/
+  reports/
+  .mcprigor/
+```
+
+Add generated evidence and local reports to `.gitignore` unless your policy requires versioning them:
+
+```gitignore
+.mcprigor/
+reports/
+*.snap.actual.json
+```
+
+Commit contract locks and approved snapshot expectations when they are part of review.
+
+## Configure a stdio target
+
+```text
+MCP Test 1
+Suite: "Customer acceptance tests"
+Server: node ../customer-server/dist/server.js
+
+Server options:
+  cwd: ../customer-server
+  env:
+    NODE_ENV: test
+```
+
+MCP Rigor starts the command without a shell. Keep server shutdown deterministic and ensure descendants exit when stdin closes or the parent terminates.
+
+## Configure Streamable HTTP
+
+```text
+MCP URL: https://qa.example.com/mcp
+
+Server options:
+  headers:
+    Authorization: "Bearer ${env.MCP_TOKEN}"
+```
+
+Provide secrets through CI variables:
+
+```bash
+MCP_TOKEN="$QA_MCP_TOKEN" npx mcprigor test tests/smoke.mcpr
+```
+
+## Give QA authors stable operations
+
+Prefer:
+
+- clear tool and argument descriptions;
+- structured output for business assertions;
+- stable resource URIs;
+- deterministic fixture data;
+- test-only cleanup operations;
+- generated IDs returned in explicit fields;
+- server errors with actionable MCP messages.
+
+Avoid requiring QA users to assert entire text blobs or volatile metadata.
+
+## Add package scripts
+
+```json
+{
+  "scripts": {
+    "mcp:check": "mcprigor check tests/smoke.mcpr",
+    "mcp:test": "mcprigor test tests/smoke.mcpr",
+    "mcp:report": "mcprigor test tests/regression.mcpr --html reports/mcp.html",
+    "mcp:parity": "mcprigor parity tests/parity.mcpr"
+  }
+}
+```
+
+## GitHub Actions
+
+```yaml
+name: MCP acceptance tests
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  mcp-tests:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+      - run: npm ci
+      - run: npm run mcp:check
+      - run: >-
+          npx mcprigor test tests/regression.mcpr
+          --junit reports/mcp.xml
+          --evidence .mcprigor/ci-${{ github.run_id }}
+        env:
+          MCP_TOKEN: ${{ secrets.QA_MCP_TOKEN }}
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: mcp-evidence
+          path: |
+            reports/
+            .mcprigor/
+          retention-days: 7
+```
+
+## Contract drift in pull requests
+
+Create a baseline once:
+
+```bash
+mcprigor discover tests/server.mcpr --out mcp.lock.yaml
+```
+
+Then check it in CI:
+
+```bash
+mcprigor contract-check mcp.lock.yaml \
+  --target tests/server.mcpr \
+  --markdown \
+  --out reports/contract-drift.md
+```
+
+Update a baseline only after review:
+
+```bash
+mcprigor contract-update mcp.lock.yaml --target tests/server.mcpr
+```
+
+## Transport parity
+
+Engineers configure connections once:
+
+```text
+Compare target "Local": node ../server/dist/server.js
+Compare target "QA": https://qa.example.com/mcp
+```
+
+QA authors then add ordinary tests. CI runs:
+
+```bash
+mcprigor parity tests/parity.mcpr --markdown --out reports/parity.md
+```
+
+## Production adoption checklist
+
+- Pin MCP Rigor and the MCP SDK through the lockfile.
+- Run `check` before server execution.
+- Use dedicated nonproduction accounts and data.
+- Set explicit CI timeouts and row limits.
+- Keep remote data and extensions disabled unless required.
+- Define contract-drift approval ownership.
+- Set evidence access, retention, and deletion rules.
+- Verify each required Node, OS, protocol, and transport combination.
+- Test one intentional failure and server-shutdown path.
+
+See [compatibility](COMPATIBILITY.md) and [security and retention](SECURITY-AND-RETENTION.md).

package/docs/ERROR-MODEL.md

@@ -0,0 +1,24 @@
+# Stable Error Model
+
+MCP Rigor failures carry a stable code, category, sanitized message, and QA-facing action.
+
+| Category | Baseline code | QA action |
+|---|---|---|
+| configuration | `MCP-CONFIG-000` | Check suite target and settings. |
+| language compilation | `MCP-LANG-000` | Correct the highlighted statement. |
+| data loading | `MCP-DATA-000` | Check the named source, row, and column. |
+| extension | `MCP-EXT-000` | Review manifest, permission, and export. |
+| server spawn | `MCP-SPAWN-000` | Check command and working directory. |
+| initialization | `MCP-INIT-000` | Inspect why MCP initialization did not finish. |
+| transport | `MCP-TRANSPORT-000` | Check process/HTTP connection and server logs. |
+| MCP error | `MCP-REMOTE-000` | Review the server's MCP code and message. |
+| schema | `MCP-SCHEMA-000` | Compare the result with the required schema. |
+| assertion | `MCP-ASSERT-000` | Compare expected path/value with response. |
+| timeout | `MCP-TIMEOUT-000` | Inspect server performance or explicit timeout. |
+| cancellation | `MCP-CANCEL-000` | The operation was stopped before completion. |
+| cleanup | `MCP-CLEANUP-000` | Inspect cleanup steps and server shutdown. |
+| internal | `MCP-INTERNAL-000` | Preserve sanitized evidence and report the bug. |
+
+Specific existing codes remain stable within these categories. APIs expose `classifyFailure()`, `formatFailure()`, `RigorError`, and `FAILURE_CODES`. New codes may be added, but released codes are not repurposed.
+
+Terminal formatting removes control sequences before display. Private causes and stacks are not part of the stable user-facing contract.

package/docs/EVIDENCE.md

@@ -0,0 +1,58 @@
+# Protocol Traces and Failure Evidence
+
+MCP Rigor can retain a sanitized, correlated account of a test run:
+
+```bash
+mcprigor test tests.mcpr --evidence .mcprigor/run-001
+```
+
+This is currently an **MCP SDK/API-boundary trace**, not a raw byte capture. It records what MCP Rigor sends to and receives from the official SDK transport boundary.
+
+## Bundle contents
+
+```text
+.mcprigor/run-001/
+  manifest.json
+  result.json
+  trace.jsonl
+  trace.normalized.jsonl
+```
+
+`trace.jsonl` contains ordered lifecycle, request, response, error, close, and bounded diagnostic events. Each event includes a toolkit-owned sequence, request correlation ID, test ID, step name, method, and sanitized data.
+
+`trace.normalized.jsonl` removes elapsed timing and known volatile values while preserving semantic ordering. Its SHA-256 fingerprint appears in `manifest.json`.
+
+## Inspect and compare
+
+```bash
+mcprigor evidence-show .mcprigor/run-001
+mcprigor evidence-compare .mcprigor/run-001 .mcprigor/run-002
+```
+
+Comparison checks the result and normalized trace fingerprints. Matching fingerprints indicate semantically identical evidence under the current normalization policy.
+
+## Events
+
+- `session.connect.start`
+- `session.connect.success`
+- `request`
+- `response`
+- `error`
+- `session.close`
+- `diagnostic`
+
+Request and terminal response/error events share `requestId`. Test and step fields tie protocol activity back to readable acceptance tests.
+
+## Security
+
+Redaction happens before events enter the recorder. MCP Rigor removes configured secrets, values under credential-shaped keys, and bearer tokens. Reports should still be handled as potentially sensitive because ordinary business data may remain.
+
+Recommended practices:
+
+- store evidence as short-lived CI artifacts;
+- do not put tokens or regulated data in assertions/test names;
+- review custom function and provider output;
+- limit artifact access;
+- delete bundles after the investigation or retention period.
+
+The evidence hash is an integrity aid, not a digital signature or legal attestation.

package/docs/EXTENSION-SDK.md

@@ -0,0 +1,76 @@
+# Isolated Extension SDK
+
+MCP Rigor 0.13 changes custom functions and data providers from direct host imports to worker-isolated extensions.
+
+```js
+export const manifest = {
+  schemaVersion: 1,
+  name: "customer-utilities",
+  version: "1.0.0",
+  permissions: [],
+  functions: ["normalizeCustomer"],
+  provider: true
+};
+
+export function normalizeCustomer({ value }) {
+  return String(value).trim().toUpperCase();
+}
+
+export const provider = {
+  async load(config) {
+    return [{ id: "case-1", expected: config.expected }];
+  }
+};
+```
+
+Enable reviewed extensions as before:
+
+```bash
+mcprigor test suite.mcpr --allow-custom-code
+```
+
+Execution now occurs in a dedicated Node worker with:
+
+- a reviewed versioned manifest;
+- explicit permissions;
+- wall-clock timeout and forced termination;
+- a 64 MiB default V8 old-generation limit;
+- JSON-compatible inputs and outputs only;
+- stable `MCP-EXT-1xx` diagnostics;
+- no automatic fallback to direct host import.
+
+Manifest permissions currently recognized:
+
+- `environment`
+- `filesystem-read`
+- `network`
+
+A requested permission must be granted by suite configuration. Permissions describe review intent in the current worker backend; they are not yet a complete host-brokered filesystem/network sandbox.
+
+## Public SDK
+
+```js
+import { defineExtension, defineManifest } from "mcprigor";
+
+export const manifest = defineManifest({
+  schemaVersion: 1,
+  name: "example",
+  version: "1.0.0",
+  permissions: [],
+  functions: ["decorate"]
+});
+
+export default defineExtension({ manifest });
+```
+
+Programmatic host APIs:
+
+- `inspectExtension()`
+- `callIsolatedFunction()`
+- `callIsolatedProvider()`
+
+## Legacy compatibility
+
+Existing modules can still execute directly only when the suite explicitly selects `extensions.unsafeLegacy: true` or a provider uses `unsafeLegacy: true`. This mode has the full authority of the MCP Rigor process and should only be used temporarily while migrating.
+
+Worker threads improve failure and capability isolation, but they are not a hard hostile-code security boundary. Truly untrusted extensions should run in a separately secured process or container.