mcp-eval-gate 0.3.0__tar.gz

mcp_eval_gate-0.3.0/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: mcp-eval-gate
+Version: 0.3.0
+Summary: CI regression gate for MCP servers: run a golden set of tool calls, diff against a baseline, fail the build on regression
+Keywords: mcp,model-context-protocol,regression-testing,testing,ci,golden-tests
+Author: Umer Karachiwala
+Author-email: Umer Karachiwala <karachiwalaumer2612@gmail.com>
+License-Expression: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Dist: mcp>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: click>=8.1
+Requires-Dist: rich>=13.7
+Requires-Dist: anthropic>=0.40 ; extra == 'judge'
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/Umer-2612/mcp-eval-gate
+Project-URL: Repository, https://github.com/Umer-2612/mcp-eval-gate
+Project-URL: Issues, https://github.com/Umer-2612/mcp-eval-gate/issues
+Project-URL: Changelog, https://github.com/Umer-2612/mcp-eval-gate/blob/main/CHANGELOG.md
+Provides-Extra: judge
+Description-Content-Type: text/markdown
+
+# mcp-eval-gate
+
+A CI regression gate for MCP servers. You write a golden set of tool calls with known-good
+outputs. `mcp-eval-gate` calls those tools on your server, compares the results to a saved
+baseline, and exits non-zero if anything got worse, even when the tool's schema did not
+change.
+
+## Why
+
+An MCP server's tool names and input schemas can stay identical while what a tool returns
+quietly gets worse: a refactor breaks a code path, a dependency bump changes behavior.
+Nothing crashes, the agent using it just gets worse answers.
+
+Other tools cover parts of this. The official Inspector's CLI runs scripted single-run
+assertions. `mcp-server-diff` compares declared schemas and says it does not test output
+correctness. A few other early tools record golden outputs and diff them too, for example
+[vexyo](https://github.com/vexyohq/vexyo) and
+[cisco-open/mcptoolkit-test](https://github.com/cisco-open/mcptoolkit-test).
+`mcp-eval-gate` is another take on golden-output regression.
+
+## Install
+
+Requires Python 3.12+.
+
+```bash
+pip install mcp-eval-gate
+
+# only if you use match_type: judge
+pip install "mcp-eval-gate[judge]"
+```
+
+## Quickstart
+
+```bash
+mcp-eval-gate init                   # scaffolds golden_set.yaml
+# edit golden_set.yaml with how to reach your server and your test cases
+mcp-eval-gate run --update-baseline  # first run: record the baseline
+mcp-eval-gate run                    # later runs: gate on regressions
+```
+
+`run` prints a table and a diff, and exits `1` if any case fails or regresses against the
+baseline. A tool call that returns an error always fails its case.
+
+## Golden set
+
+```yaml
+server:
+  command: node
+  args: ["dist/index.js"]
+  # or, for an HTTP server instead of stdio:
+  # url: http://localhost:3000/mcp
+
+cases:
+  # substring match
+  - id: get-weather-nyc
+    tool_name: get_weather
+    tool_args:
+      city: "New York"
+    match_type: contains
+    expected_output: "New York"
+
+  # exact match
+  - id: cancel-subscription
+    tool_name: cancel_subscription
+    tool_args:
+      immediate: true
+    match_type: exact
+    expected_output: "cancelled"
+
+  # scored by an LLM against a rubric (needs ANTHROPIC_API_KEY)
+  - id: retention-policy-answer
+    tool_name: search_docs
+    tool_args:
+      query: "data retention policy"
+    match_type: judge
+    judge_criteria: "Answer must state data is retained for 90 days"
+    min_judge_score: 0.8
+```
+
+[`examples/golden_set.yaml`](https://github.com/Umer-2612/mcp-eval-gate/blob/main/examples/golden_set.yaml) is a copy you can edit.
+
+## Run as an MCP tool
+
+The package also installs `mcp-eval-gate-mcp`, an MCP server over stdio with one tool,
+`run_eval_gate(config_path, baseline_path, update_baseline)`. Add it to an MCP client's config:
+
+```json
+{
+  "mcpServers": {
+    "mcp-eval-gate": { "command": "mcp-eval-gate-mcp" }
+  }
+}
+```
+
+## Validation
+
+[`validation/`](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation) has two runs against real servers, each with the actual command
+output committed and steps to reproduce it. No paid API calls.
+
+- A one-line regression planted in the official MCP reference server, caught with exit
+  code `1` and a real diff.
+- [A real bug](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation/utf8-boundary) the official filesystem server shipped (garbled
+  text when a multi-byte character straddled a read boundary), caught by running the commit
+  before its upstream fix against a baseline from the fixed commit.
+
+Both are small. They show the gate works end to end on real code, not how often this class
+of bug occurs.
+
+## Limitations
+
+- Matching is exact, substring, or an LLM judge. There is no normalization for volatile
+  output such as timestamps or ids, so output that varies between runs can't be compared
+  reliably.
+- It checks tool outputs only, not schemas or protocol conformance.
+- `match_type: judge` has only been tested against a stub client, not the live Anthropic API.
+
+## Development
+
+```bash
+uv sync --extra judge --dev
+uv run pytest --cov=src --cov-report=term-missing
+uv run ruff check src tests
+```
+
+## License
+
+MIT

mcp_eval_gate-0.3.0/README.md

@@ -0,0 +1,127 @@
+# mcp-eval-gate
+
+A CI regression gate for MCP servers. You write a golden set of tool calls with known-good
+outputs. `mcp-eval-gate` calls those tools on your server, compares the results to a saved
+baseline, and exits non-zero if anything got worse, even when the tool's schema did not
+change.
+
+## Why
+
+An MCP server's tool names and input schemas can stay identical while what a tool returns
+quietly gets worse: a refactor breaks a code path, a dependency bump changes behavior.
+Nothing crashes, the agent using it just gets worse answers.
+
+Other tools cover parts of this. The official Inspector's CLI runs scripted single-run
+assertions. `mcp-server-diff` compares declared schemas and says it does not test output
+correctness. A few other early tools record golden outputs and diff them too, for example
+[vexyo](https://github.com/vexyohq/vexyo) and
+[cisco-open/mcptoolkit-test](https://github.com/cisco-open/mcptoolkit-test).
+`mcp-eval-gate` is another take on golden-output regression.
+
+## Install
+
+Requires Python 3.12+.
+
+```bash
+pip install mcp-eval-gate
+
+# only if you use match_type: judge
+pip install "mcp-eval-gate[judge]"
+```
+
+## Quickstart
+
+```bash
+mcp-eval-gate init                   # scaffolds golden_set.yaml
+# edit golden_set.yaml with how to reach your server and your test cases
+mcp-eval-gate run --update-baseline  # first run: record the baseline
+mcp-eval-gate run                    # later runs: gate on regressions
+```
+
+`run` prints a table and a diff, and exits `1` if any case fails or regresses against the
+baseline. A tool call that returns an error always fails its case.
+
+## Golden set
+
+```yaml
+server:
+  command: node
+  args: ["dist/index.js"]
+  # or, for an HTTP server instead of stdio:
+  # url: http://localhost:3000/mcp
+
+cases:
+  # substring match
+  - id: get-weather-nyc
+    tool_name: get_weather
+    tool_args:
+      city: "New York"
+    match_type: contains
+    expected_output: "New York"
+
+  # exact match
+  - id: cancel-subscription
+    tool_name: cancel_subscription
+    tool_args:
+      immediate: true
+    match_type: exact
+    expected_output: "cancelled"
+
+  # scored by an LLM against a rubric (needs ANTHROPIC_API_KEY)
+  - id: retention-policy-answer
+    tool_name: search_docs
+    tool_args:
+      query: "data retention policy"
+    match_type: judge
+    judge_criteria: "Answer must state data is retained for 90 days"
+    min_judge_score: 0.8
+```
+
+[`examples/golden_set.yaml`](https://github.com/Umer-2612/mcp-eval-gate/blob/main/examples/golden_set.yaml) is a copy you can edit.
+
+## Run as an MCP tool
+
+The package also installs `mcp-eval-gate-mcp`, an MCP server over stdio with one tool,
+`run_eval_gate(config_path, baseline_path, update_baseline)`. Add it to an MCP client's config:
+
+```json
+{
+  "mcpServers": {
+    "mcp-eval-gate": { "command": "mcp-eval-gate-mcp" }
+  }
+}
+```
+
+## Validation
+
+[`validation/`](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation) has two runs against real servers, each with the actual command
+output committed and steps to reproduce it. No paid API calls.
+
+- A one-line regression planted in the official MCP reference server, caught with exit
+  code `1` and a real diff.
+- [A real bug](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation/utf8-boundary) the official filesystem server shipped (garbled
+  text when a multi-byte character straddled a read boundary), caught by running the commit
+  before its upstream fix against a baseline from the fixed commit.
+
+Both are small. They show the gate works end to end on real code, not how often this class
+of bug occurs.
+
+## Limitations
+
+- Matching is exact, substring, or an LLM judge. There is no normalization for volatile
+  output such as timestamps or ids, so output that varies between runs can't be compared
+  reliably.
+- It checks tool outputs only, not schemas or protocol conformance.
+- `match_type: judge` has only been tested against a stub client, not the live Anthropic API.
+
+## Development
+
+```bash
+uv sync --extra judge --dev
+uv run pytest --cov=src --cov-report=term-missing
+uv run ruff check src tests
+```
+
+## License
+
+MIT

mcp_eval_gate-0.3.0/pyproject.toml

@@ -0,0 +1,70 @@
+[project]
+name = "mcp-eval-gate"
+version = "0.3.0"
+description = "CI regression gate for MCP servers: run a golden set of tool calls, diff against a baseline, fail the build on regression"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.12"
+keywords = [
+    "mcp",
+    "model-context-protocol",
+    "regression-testing",
+    "testing",
+    "ci",
+    "golden-tests",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+    "Typing :: Typed",
+]
+dependencies = [
+    "mcp>=2.0",
+    "pyyaml>=6.0",
+    "click>=8.1",
+    "rich>=13.7",
+]
+
+[[project.authors]]
+name = "Umer Karachiwala"
+email = "karachiwalaumer2612@gmail.com"
+
+[project.urls]
+Homepage = "https://github.com/Umer-2612/mcp-eval-gate"
+Repository = "https://github.com/Umer-2612/mcp-eval-gate"
+Issues = "https://github.com/Umer-2612/mcp-eval-gate/issues"
+Changelog = "https://github.com/Umer-2612/mcp-eval-gate/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+judge = ["anthropic>=0.40"]
+
+[project.scripts]
+mcp-eval-gate = "mcp_eval_gate.cli:main"
+mcp-eval-gate-mcp = "mcp_eval_gate.mcp_server:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=5.0",
+    "ruff>=0.7",
+]
+
+[tool.ruff]
+line-length = 115
+target-version = "py312"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+    "B",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.17,<0.13.0"]
+build-backend = "uv_build"

mcp_eval_gate-0.3.0/pyproject.toml.orig

@@ -0,0 +1,56 @@
+[project]
+name = "mcp-eval-gate"
+version = "0.3.0"
+description = "CI regression gate for MCP servers: run a golden set of tool calls, diff against a baseline, fail the build on regression"
+readme = "README.md"
+license = "MIT"
+authors = [
+    { name = "Umer Karachiwala", email = "karachiwalaumer2612@gmail.com" }
+]
+requires-python = ">=3.12"
+keywords = ["mcp", "model-context-protocol", "regression-testing", "testing", "ci", "golden-tests"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+    "Typing :: Typed",
+]
+dependencies = [
+    "mcp>=2.0",
+    "pyyaml>=6.0",
+    "click>=8.1",
+    "rich>=13.7",
+]
+
+[project.urls]
+Homepage = "https://github.com/Umer-2612/mcp-eval-gate"
+Repository = "https://github.com/Umer-2612/mcp-eval-gate"
+Issues = "https://github.com/Umer-2612/mcp-eval-gate/issues"
+Changelog = "https://github.com/Umer-2612/mcp-eval-gate/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+judge = ["anthropic>=0.40"]
+
+[project.scripts]
+mcp-eval-gate = "mcp_eval_gate.cli:main"
+mcp-eval-gate-mcp = "mcp_eval_gate.mcp_server:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=5.0",
+    "ruff>=0.7",
+]
+
+[tool.ruff]
+line-length = 115
+target-version = "py312"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[build-system]
+requires = ["uv_build>=0.12.17,<0.13.0"]
+build-backend = "uv_build"

mcp_eval_gate-0.3.0/src/mcp_eval_gate/__init__.py

@@ -0,0 +1,2 @@
+def hello() -> str:
+    return "Hello from mcp-eval-gate!"

mcp_eval_gate-0.3.0/src/mcp_eval_gate/baseline.py

@@ -0,0 +1,61 @@
+"""Load/save a committed baseline and diff a fresh run against it.
+
+A baseline is a simple {case_id: score} snapshot, checked into the repo
+alongside the golden set, so "did this change make things worse" is a
+git-diffable question instead of a moving target.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from mcp_eval_gate.models import CaseResult, Regression
+
+DEFAULT_THRESHOLD = 0.0
+IMPLICIT_BASELINE_FOR_NEW_CASES = 1.0
+
+
+def load_baseline(path: Path) -> dict[str, float]:
+    if not path.exists():
+        return {}
+    return json.loads(path.read_text())
+
+
+def save_baseline(path: Path, results: list[CaseResult]) -> None:
+    baseline = {result.case_id: result.score for result in results}
+    path.write_text(json.dumps(baseline, indent=2, sort_keys=True) + "\n")
+
+
+def diff_against_baseline(
+    results: list[CaseResult], baseline: dict[str, float], threshold: float = DEFAULT_THRESHOLD
+) -> list[Regression]:
+    regressions: list[Regression] = []
+    seen_case_ids: set[str] = set()
+
+    for result in results:
+        seen_case_ids.add(result.case_id)
+        baseline_score = baseline.get(result.case_id, IMPLICIT_BASELINE_FOR_NEW_CASES)
+        drop = baseline_score - result.score
+        if drop > threshold:
+            regressions.append(
+                Regression(
+                    case_id=result.case_id,
+                    baseline_score=baseline_score,
+                    current_score=result.score,
+                    detail=result.detail,
+                )
+            )
+
+    for case_id, baseline_score in baseline.items():
+        if case_id not in seen_case_ids:
+            regressions.append(
+                Regression(
+                    case_id=case_id,
+                    baseline_score=baseline_score,
+                    current_score=0.0,
+                    detail="case present in baseline is missing from the current run",
+                )
+            )
+
+    return regressions

mcp_eval_gate-0.3.0/src/mcp_eval_gate/cli.py

@@ -0,0 +1,109 @@
+"""CLI: `mcp-eval-gate run` and `mcp-eval-gate init`."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import anyio
+import click
+
+from mcp_eval_gate.baseline import diff_against_baseline, load_baseline, save_baseline
+from mcp_eval_gate.golden_set import GoldenSetError, load_golden_set
+from mcp_eval_gate.judge import build_default_anthropic_client
+from mcp_eval_gate.report import exit_code_for, print_console_report, write_html_report
+from mcp_eval_gate.runner import run_golden_set
+
+SAMPLE_GOLDEN_SET = """\
+server:
+  command: node
+  args: ["dist/index.js"]
+  # or, for an HTTP server instead of stdio:
+  # url: http://localhost:3000/mcp
+
+judge_model: claude-sonnet-4-5
+
+cases:
+  - id: example-contains-case
+    tool_name: get_weather
+    tool_args:
+      city: "New York"
+    match_type: contains
+    expected_output: "New York"
+
+  - id: example-exact-case
+    tool_name: cancel_subscription
+    tool_args:
+      immediate: true
+    match_type: exact
+    expected_output: "cancelled"
+
+  - id: example-judge-case
+    tool_name: search_docs
+    tool_args:
+      query: "data retention policy"
+    match_type: judge
+    judge_criteria: "Answer must state data is retained for 90 days"
+    min_judge_score: 0.8
+"""
+
+
+@click.group()
+def main() -> None:
+    """mcp-eval-gate: CI regression gate for MCP servers."""
+
+
+@main.command()
+@click.option(
+    "--config", "config_path", type=click.Path(path_type=Path), default="golden_set.yaml", show_default=True
+)
+@click.option(
+    "--baseline", "baseline_path", type=click.Path(path_type=Path), default="baseline.json", show_default=True
+)
+@click.option(
+    "--update-baseline", is_flag=True, help="Overwrite the baseline with this run's scores instead of gating."
+)
+@click.option(
+    "--html-report", type=click.Path(path_type=Path), default=None, help="Write an HTML report to this path."
+)
+def run(config_path: Path, baseline_path: Path, update_baseline: bool, html_report: Path | None) -> None:
+    """Run the golden set against a live MCP server and gate on regressions."""
+    try:
+        config = load_golden_set(config_path)
+    except GoldenSetError as exc:
+        click.secho(f"error: {exc}", fg="red", err=True)
+        sys.exit(2)
+
+    anthropic_client = build_default_anthropic_client()
+    results = anyio.run(lambda: run_golden_set(config, anthropic_client=anthropic_client))
+
+    if update_baseline:
+        save_baseline(baseline_path, results)
+        click.secho(f"baseline updated: {baseline_path}", fg="cyan")
+        print_console_report(results, regressions=[])
+        return
+
+    baseline = load_baseline(baseline_path)
+    regressions = diff_against_baseline(results, baseline)
+    print_console_report(results, regressions)
+
+    if html_report:
+        write_html_report(results, regressions, html_report)
+        click.echo(f"html report written to {html_report}")
+
+    sys.exit(exit_code_for(results, regressions))
+
+
+@main.command()
+@click.option("--out", type=click.Path(path_type=Path), default="golden_set.yaml", show_default=True)
+def init(out: Path) -> None:
+    """Scaffold a starter golden_set.yaml."""
+    if out.exists():
+        click.secho(f"refusing to overwrite existing file: {out}", fg="red", err=True)
+        sys.exit(1)
+    out.write_text(SAMPLE_GOLDEN_SET)
+    click.secho(f"wrote {out}", fg="green")
+
+
+if __name__ == "__main__":
+    main()

mcp_eval_gate-0.3.0/src/mcp_eval_gate/golden_set.py

@@ -0,0 +1,66 @@
+"""Load a golden-set YAML file into a validated GoldenSetConfig."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import yaml
+
+from mcp_eval_gate.models import GoldenCase, GoldenSetConfig, MatchType, ServerTarget
+
+REQUIRED_CASE_FIELDS = ("id", "tool_name")
+
+
+class GoldenSetError(ValueError):
+    """Raised when a golden-set file is missing, malformed, or fails validation."""
+
+
+def load_golden_set(path: Path) -> GoldenSetConfig:
+    if not path.exists():
+        raise GoldenSetError(f"golden set file not found: {path}")
+
+    raw = yaml.safe_load(path.read_text()) or {}
+
+    if "server" not in raw:
+        raise GoldenSetError("golden set file is missing the required `server` block")
+
+    server = _parse_server(raw["server"])
+    cases = tuple(_parse_case(raw_case) for raw_case in raw.get("cases", []))
+    _reject_duplicate_ids(cases)
+
+    return GoldenSetConfig(
+        server=server,
+        cases=cases,
+        judge_model=raw.get("judge_model", "claude-sonnet-4-5"),
+    )
+
+
+def _parse_server(raw_server: dict) -> ServerTarget:
+    try:
+        return ServerTarget(
+            command=raw_server.get("command"),
+            args=tuple(raw_server.get("args", ())),
+            env=raw_server.get("env"),
+            url=raw_server.get("url"),
+        )
+    except ValueError as exc:
+        raise GoldenSetError(str(exc)) from exc
+
+
+def _parse_case(raw_case: dict) -> GoldenCase:
+    missing = [field for field in REQUIRED_CASE_FIELDS if field not in raw_case]
+    if missing:
+        case_id = raw_case.get("id", "<unknown>")
+        raise GoldenSetError(f"case '{case_id}' is missing required field(s): {', '.join(missing)}")
+
+    fields = dict(raw_case)
+    fields["match_type"] = MatchType(fields.get("match_type", MatchType.CONTAINS))
+    return GoldenCase(**fields)
+
+
+def _reject_duplicate_ids(cases: tuple[GoldenCase, ...]) -> None:
+    seen: set[str] = set()
+    for case in cases:
+        if case.id in seen:
+            raise GoldenSetError(f"duplicate case id: '{case.id}'")
+        seen.add(case.id)

mcp_eval_gate-0.3.0/src/mcp_eval_gate/judge.py

@@ -0,0 +1,75 @@
+"""LLM-as-judge scoring for open-ended tool outputs, via the Anthropic API.
+
+Used only for cases with match_type=judge — exact and contains cases are
+scored deterministically in scoring.py and never need a model call.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+from typing import Any
+
+_JSON_OBJECT_PATTERN = re.compile(r"\{.*\}", re.DOTALL)
+
+JUDGE_PROMPT_TEMPLATE = """You are grading an MCP tool's output against a rubric.
+
+Rubric (must be satisfied):
+{criteria}
+
+Tool output:
+{answer}
+
+Respond with ONLY a JSON object of the form:
+{{"score": <float between 0.0 and 1.0>, "reasoning": "<one sentence>"}}
+A score of 1.0 means the rubric is fully satisfied; 0.0 means it is not satisfied at all."""
+
+
+class JudgeParseError(ValueError):
+    """Raised when the judge model's response can't be parsed into a score."""
+
+
+def build_judge_prompt(*, criteria: str, answer: str) -> str:
+    return JUDGE_PROMPT_TEMPLATE.format(criteria=criteria, answer=answer)
+
+
+def parse_judge_response(text: str) -> tuple[float, str]:
+    match = _JSON_OBJECT_PATTERN.search(text)
+    if not match:
+        raise JudgeParseError(f"no JSON object found in judge response: {text!r}")
+
+    try:
+        payload = json.loads(match.group(0))
+    except json.JSONDecodeError as exc:
+        raise JudgeParseError(f"judge response was not valid JSON: {text!r}") from exc
+
+    if "score" not in payload:
+        raise JudgeParseError(f"judge response missing 'score' field: {payload!r}")
+
+    score = max(0.0, min(1.0, float(payload["score"])))
+    reasoning = str(payload.get("reasoning", ""))
+    return score, reasoning
+
+
+def build_default_anthropic_client() -> Any | None:
+    """Returns an Anthropic client if a key is configured and the extra is installed, else None."""
+    if not os.environ.get("ANTHROPIC_API_KEY"):
+        return None
+    try:
+        import anthropic
+    except ImportError:
+        return None
+    return anthropic.Anthropic()
+
+
+def judge_answer(client: Any, *, model: str, criteria: str, answer: str) -> tuple[float, str]:
+    prompt = build_judge_prompt(criteria=criteria, answer=answer)
+    response = client.messages.create(
+        model=model,
+        max_tokens=256,
+        temperature=0.0,
+        messages=[{"role": "user", "content": prompt}],
+    )
+    text = response.content[0].text
+    return parse_judge_response(text)

mcp_eval_gate-0.3.0/src/mcp_eval_gate/mcp_client.py

@@ -0,0 +1,44 @@
+"""Connect to an MCP server under test (stdio or HTTP) and call its tools.
+
+This is what makes the tool protocol-generic: it speaks the MCP client side
+directly, rather than a specific vendor's SDK, so it works against any
+compliant MCP server regardless of the language or backend it's written in.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+
+from mcp.client.session import ClientSession
+from mcp.client.stdio import StdioServerParameters, stdio_client
+from mcp.client.streamable_http import streamable_http_client
+from mcp.types import CallToolResult, TextContent
+
+from mcp_eval_gate.models import ServerTarget, ToolCallOutcome
+
+
+@asynccontextmanager
+async def connect(target: ServerTarget) -> AsyncIterator[ClientSession]:
+    if target.command:
+        params = StdioServerParameters(command=target.command, args=list(target.args), env=target.env)
+        async with stdio_client(params) as (read, write):
+            async with ClientSession(read, write) as session:
+                await session.initialize()
+                yield session
+        return
+
+    async with streamable_http_client(target.url) as (read, write, _get_session_id):
+        async with ClientSession(read, write) as session:
+            await session.initialize()
+            yield session
+
+
+async def call_tool(session: ClientSession, tool_name: str, arguments: dict[str, object]) -> ToolCallOutcome:
+    result = await session.call_tool(tool_name, arguments)
+    return parse_call_tool_result(result)
+
+
+def parse_call_tool_result(result: CallToolResult) -> ToolCallOutcome:
+    text = "".join(block.text for block in result.content if isinstance(block, TextContent))
+    return ToolCallOutcome(text=text, structured=result.structured_content, is_error=bool(result.is_error))

mcp_eval_gate-0.3.0/src/mcp_eval_gate/mcp_server.py

@@ -0,0 +1,74 @@
+"""MCP server exposing mcp-eval-gate as a tool callable from Claude Code, Cursor, etc.
+
+Lets an engineer ask "did my last change break this MCP server's tools?" from
+inside their agent harness, before committing, not just as a CI-only check.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from mcp.server.mcpserver import MCPServer
+
+from mcp_eval_gate.baseline import diff_against_baseline, load_baseline, save_baseline
+from mcp_eval_gate.golden_set import GoldenSetError, load_golden_set
+from mcp_eval_gate.judge import build_default_anthropic_client
+from mcp_eval_gate.runner import run_golden_set
+
+mcp = MCPServer("mcp-eval-gate")
+
+
+@mcp.tool()
+async def run_eval_gate(
+    config_path: str,
+    baseline_path: str = "baseline.json",
+    update_baseline: bool = False,
+) -> dict:
+    """Run a golden-set eval against a live MCP server and report regressions vs. baseline.
+
+    Args:
+        config_path: Path to the golden_set.yaml file.
+        baseline_path: Path to the committed baseline.json (created by a prior run).
+        update_baseline: If true, overwrite the baseline with this run's scores instead of gating.
+    """
+    try:
+        config = load_golden_set(Path(config_path))
+    except GoldenSetError as exc:
+        return {"ok": False, "error": str(exc)}
+
+    anthropic_client = build_default_anthropic_client()
+    results = await run_golden_set(config, anthropic_client=anthropic_client)
+
+    if update_baseline:
+        save_baseline(Path(baseline_path), results)
+        return {"ok": True, "baseline_updated": True, "cases": _serialize_results(results)}
+
+    baseline = load_baseline(Path(baseline_path))
+    regressions = diff_against_baseline(results, baseline)
+    any_failed = any(not r.passed for r in results)
+
+    return {
+        "ok": not (any_failed or regressions),
+        "cases": _serialize_results(results),
+        "regressions": [
+            {
+                "case_id": r.case_id,
+                "baseline_score": r.baseline_score,
+                "current_score": r.current_score,
+                "detail": r.detail,
+            }
+            for r in regressions
+        ],
+    }
+
+
+def _serialize_results(results) -> list[dict]:
+    return [{"case_id": r.case_id, "score": r.score, "passed": r.passed, "detail": r.detail} for r in results]
+
+
+def main() -> None:
+    mcp.run()
+
+
+if __name__ == "__main__":
+    main()

mcp_eval_gate-0.3.0/src/mcp_eval_gate/models.py

@@ -0,0 +1,67 @@
+"""Core data types shared across golden-set loading, MCP calls, scoring, and baseline diffing."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import StrEnum
+
+
+class MatchType(StrEnum):
+    EXACT = "exact"
+    CONTAINS = "contains"
+    JUDGE = "judge"
+
+
+@dataclass(frozen=True)
+class ServerTarget:
+    """How to reach the MCP server under test: a stdio subprocess or a URL."""
+
+    command: str | None = None
+    args: tuple[str, ...] = ()
+    env: dict[str, str] | None = None
+    url: str | None = None
+
+    def __post_init__(self) -> None:
+        if bool(self.command) == bool(self.url):
+            raise ValueError("ServerTarget needs exactly one of `command` (stdio) or `url` (HTTP)")
+
+
+@dataclass(frozen=True)
+class GoldenCase:
+    id: str
+    tool_name: str
+    tool_args: dict[str, object] = field(default_factory=dict)
+    match_type: MatchType = MatchType.CONTAINS
+    expected_output: str | None = None
+    judge_criteria: str | None = None
+    min_judge_score: float = 0.8
+
+
+@dataclass(frozen=True)
+class ToolCallOutcome:
+    text: str
+    structured: dict | None
+    is_error: bool
+
+
+@dataclass(frozen=True)
+class CaseResult:
+    case_id: str
+    score: float
+    passed: bool
+    detail: str
+
+
+@dataclass(frozen=True)
+class Regression:
+    case_id: str
+    baseline_score: float
+    current_score: float
+    detail: str
+
+
+@dataclass(frozen=True)
+class GoldenSetConfig:
+    server: ServerTarget
+    cases: tuple[GoldenCase, ...]
+    judge_model: str = "claude-sonnet-4-5"

mcp_eval_gate-0.3.0/src/mcp_eval_gate/report.py

@@ -0,0 +1,67 @@
+"""Console + HTML rendering for a run, and the pass/fail decision that gates CI."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from rich.console import Console
+from rich.table import Table
+
+from mcp_eval_gate.models import CaseResult, Regression
+
+
+def exit_code_for(results: list[CaseResult], regressions: list[Regression]) -> int:
+    any_failed = any(not r.passed for r in results)
+    return 1 if any_failed or regressions else 0
+
+
+def print_console_report(results: list[CaseResult], regressions: list[Regression]) -> None:
+    console = Console()
+    table = Table(title="mcp-eval-gate results")
+    table.add_column("Case")
+    table.add_column("Score", justify="right")
+    table.add_column("Status")
+    table.add_column("Detail")
+
+    regressed_ids = {r.case_id for r in regressions}
+    for result in results:
+        status = "[green]PASS[/green]" if result.passed else "[red]FAIL[/red]"
+        if result.case_id in regressed_ids:
+            status += " [yellow](regression)[/yellow]"
+        table.add_row(result.case_id, f"{result.score:.2f}", status, result.detail)
+
+    console.print(table)
+
+    if regressions:
+        console.print(f"\n[bold red]{len(regressions)} regression(s) against baseline:[/bold red]")
+        for reg in regressions:
+            console.print(f"  - {reg.case_id}: {reg.baseline_score:.2f} -> {reg.current_score:.2f} ({reg.detail})")
+
+
+def write_html_report(results: list[CaseResult], regressions: list[Regression], path: Path) -> None:
+    regressed_ids = {r.case_id for r in regressions}
+    rows = "\n".join(
+        f'<tr class="{"fail" if not r.passed else "pass"}">'
+        f"<td>{r.case_id}</td><td>{r.score:.2f}</td>"
+        f"<td>{'PASS' if r.passed else 'FAIL'}{' (regression)' if r.case_id in regressed_ids else ''}</td>"
+        f"<td>{r.detail}</td></tr>"
+        for r in results
+    )
+    html = f"""<!doctype html>
+<html><head><meta charset="utf-8"><title>mcp-eval-gate report</title>
+<style>
+body {{ font-family: -apple-system, sans-serif; margin: 2rem; }}
+table {{ border-collapse: collapse; width: 100%; }}
+td, th {{ border: 1px solid #ddd; padding: 8px; text-align: left; }}
+tr.fail {{ background: #fdecea; }}
+tr.pass {{ background: #eafaf1; }}
+</style></head>
+<body>
+<h1>mcp-eval-gate report</h1>
+<table>
+<tr><th>Case</th><th>Score</th><th>Status</th><th>Detail</th></tr>
+{rows}
+</table>
+</body></html>
+"""
+    path.write_text(html)

mcp_eval_gate-0.3.0/src/mcp_eval_gate/runner.py

@@ -0,0 +1,46 @@
+"""Wires golden-set cases to a live MCP server and the scoring functions."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from mcp_eval_gate.judge import judge_answer
+from mcp_eval_gate.mcp_client import call_tool, connect
+from mcp_eval_gate.models import CaseResult, GoldenCase, GoldenSetConfig, MatchType
+from mcp_eval_gate.scoring import score_case, score_judge_case
+from mcp_eval_gate.text_diff import preview
+
+
+async def run_golden_set(config: GoldenSetConfig, *, anthropic_client: Any | None = None) -> list[CaseResult]:
+    async with connect(config.server) as session:
+        results = []
+        for case in config.cases:
+            results.append(await run_case(case, session=session, config=config, anthropic_client=anthropic_client))
+        return results
+
+
+async def run_case(
+    case: GoldenCase, *, session: Any, config: GoldenSetConfig, anthropic_client: Any | None
+) -> CaseResult:
+    outcome = await call_tool(session, case.tool_name, case.tool_args)
+
+    if case.match_type != MatchType.JUDGE:
+        return score_case(case, outcome)
+
+    if outcome.is_error:
+        return CaseResult(
+            case.id, score=0.0, passed=False, detail=f"tool call returned an error: {preview(outcome.text)}"
+        )
+
+    if anthropic_client is None:
+        return CaseResult(
+            case.id,
+            score=0.0,
+            passed=False,
+            detail="match_type=judge needs [judge] extras installed and ANTHROPIC_API_KEY set",
+        )
+
+    score, reasoning = judge_answer(
+        anthropic_client, model=config.judge_model, criteria=case.judge_criteria or "", answer=outcome.text
+    )
+    return score_judge_case(case, judge_score=score, reasoning=reasoning)

mcp_eval_gate-0.3.0/src/mcp_eval_gate/scoring.py

@@ -0,0 +1,45 @@
+"""Deterministic scoring for MCP tool-call outcomes.
+
+No MCP calls happen here, this module only compares an already-captured
+ToolCallOutcome against the expectation declared in a golden-set case. Keeping
+it pure is what makes it unit-testable without a live server.
+"""
+
+from __future__ import annotations
+
+from mcp_eval_gate.models import CaseResult, GoldenCase, MatchType, ToolCallOutcome
+from mcp_eval_gate.text_diff import describe_difference, preview
+
+
+def score_case(case: GoldenCase, outcome: ToolCallOutcome) -> CaseResult:
+    if outcome.is_error:
+        return CaseResult(
+            case.id, score=0.0, passed=False, detail=f"tool call returned an error: {preview(outcome.text)}"
+        )
+
+    if case.match_type == MatchType.EXACT:
+        return _score_exact(case, outcome)
+    if case.match_type == MatchType.CONTAINS:
+        return _score_contains(case, outcome)
+
+    raise ValueError(f"case '{case.id}' has match_type=judge, score it via score_judge_case after a judge call")
+
+
+def _score_exact(case: GoldenCase, outcome: ToolCallOutcome) -> CaseResult:
+    matched = outcome.text.strip() == (case.expected_output or "").strip()
+    expected = case.expected_output or ""
+    detail = "exact match" if matched else describe_difference(expected.strip(), outcome.text.strip())
+    return CaseResult(case.id, score=1.0 if matched else 0.0, passed=matched, detail=detail)
+
+
+def _score_contains(case: GoldenCase, outcome: ToolCallOutcome) -> CaseResult:
+    matched = (case.expected_output or "") in outcome.text
+    detail = (
+        "expected text found" if matched else f"expected output to contain {preview(case.expected_output or '')!r}"
+    )
+    return CaseResult(case.id, score=1.0 if matched else 0.0, passed=matched, detail=detail)
+
+
+def score_judge_case(case: GoldenCase, *, judge_score: float, reasoning: str) -> CaseResult:
+    passed = judge_score >= case.min_judge_score
+    return CaseResult(case.id, score=judge_score, passed=passed, detail=reasoning)

mcp_eval_gate-0.3.0/src/mcp_eval_gate/text_diff.py

@@ -0,0 +1,35 @@
+"""Readable descriptions of how two strings differ, for failure details in reports."""
+
+from __future__ import annotations
+
+import os
+
+DEFAULT_CONTEXT = 20
+DEFAULT_PREVIEW_LIMIT = 120
+
+
+def describe_difference(expected: str, actual: str, context: int = DEFAULT_CONTEXT) -> str:
+    index = _first_difference(expected, actual)
+    return (
+        f"first difference at char {index} "
+        f"(expected {len(expected)} chars, got {len(actual)}): "
+        f"expected {_snippet(expected, index, context)!r}, got {_snippet(actual, index, context)!r}"
+    )
+
+
+def preview(text: str, limit: int = DEFAULT_PREVIEW_LIMIT) -> str:
+    if len(text) <= limit:
+        return text
+    return f"{text[:limit]}… (+{len(text) - limit} more chars)"
+
+
+def _first_difference(expected: str, actual: str) -> int:
+    return len(os.path.commonprefix([expected, actual]))
+
+
+def _snippet(text: str, index: int, context: int) -> str:
+    start = max(0, index - context)
+    end = min(len(text), index + context)
+    prefix = "…" if start > 0 else ""
+    suffix = "…" if end < len(text) else ""
+    return f"{prefix}{text[start:end]}{suffix}"