multivon-mcp 0.1.0__tar.gz

multivon_mcp-0.1.0/LICENSE

@@ -0,0 +1,11 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Copyright 2026 Multivon
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0

multivon_mcp-0.1.0/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.4
+Name: multivon-mcp
+Version: 0.1.0
+Summary: MCP server exposing multivon-eval + pdfhell as agent-callable tools. Drop into Claude Desktop, Cursor, Cline, or any MCP-compatible AI coding agent.
+Author-email: Multivon <hello@multivon.ai>
+License: Apache-2.0
+Project-URL: Homepage, https://multivon.ai
+Project-URL: Repository, https://github.com/multivon-ai/multivon-mcp
+Project-URL: Issues, https://github.com/multivon-ai/multivon-mcp/issues
+Project-URL: Agent docs, https://multivon.ai/agents
+Keywords: mcp,llm,evaluation,agents,claude,cursor,cline,anthropic
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mcp[cli]>=1.0
+Requires-Dist: multivon-eval>=0.7.3
+Requires-Dist: pdfhell>=0.1.0
+Dynamic: license-file
+
+# multivon-mcp
+
+**MCP server that gives AI coding agents direct access to evaluation tools.** Drop into Claude Desktop, Claude Code, Cursor, Cline, or any [Model Context Protocol](https://modelcontextprotocol.io)–compatible agent.
+
+When the agent is helping you build an LLM product, it can:
+
+- Score a RAG output for hallucination without you writing the scaffolding
+- Generate an adversarial PDF on demand to test your document AI
+- Run the full pdfhell mini-suite against a model and analyse the results
+- Produce a hash-chained audit pack for procurement diligence
+- Discover the full evaluation capability catalog as JSON
+
+No copy-paste, no `python -c "..."`, no asking the agent to figure out the SDK calls.
+
+## Install
+
+```bash
+pip install multivon-mcp
+```
+
+Bare install pulls `multivon-eval`, `pdfhell`, and the MCP SDK. The provider SDKs (`anthropic`, `openai`, `google-genai`) come along too — bring your own API key in env.
+
+## Configure your agent
+
+### Claude Desktop / Claude Code
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
+
+```json
+{
+  "mcpServers": {
+    "multivon": {
+      "command": "multivon-mcp",
+      "env": {
+        "ANTHROPIC_API_KEY": "sk-ant-...",
+        "OPENAI_API_KEY": "sk-proj-...",
+        "GOOGLE_API_KEY": "AIza..."
+      }
+    }
+  }
+}
+```
+
+Restart Claude. The 9 tools become available; ask Claude `"use multivon to evaluate this RAG output"` and it figures out which tool to call.
+
+### Cursor
+
+`cursor.json` or via Settings → MCP:
+
+```json
+{ "mcpServers": { "multivon": { "command": "multivon-mcp" } } }
+```
+
+### Cline / OpenCode / any MCP-compatible agent
+
+Same shape — point at the `multivon-mcp` console script.
+
+### Local dev / debugging
+
+```bash
+mcp dev multivon_mcp.server
+```
+
+Opens the MCP Inspector UI in your browser. You can call any tool by name, see the JSON schemas, and watch the requests/responses.
+
+## The 9 tools
+
+| Tool | What it does | API key needed |
+|---|---|---|
+| `eval_discover` | Returns the full machine-readable capability catalog (evaluators, traps, suites, calibration data, versions). Call this first. | No |
+| `pdfhell_make` | Generates one adversarial PDF + its answer key. Useful for inspecting what a trap looks like. | No |
+| `pdfhell_run` | Runs the pdfhell adversarial-PDF benchmark against a vision model. Returns pass rate, per-trap CIs, suite hash. | Yes (vision provider) |
+| `eval_faithfulness` | QAG-graded faithfulness — is a RAG output grounded in the retrieved context? | Yes (judge) |
+| `eval_hallucination` | QAG-graded hallucination detection — does an output contain content NOT in context? | Yes (judge) |
+| `eval_relevance` | QAG-graded answer-vs-question relevance. | Yes (judge) |
+| `eval_answer_accuracy` | QAG-graded semantic equivalence vs ground truth. | Yes (judge) |
+| `eval_tool_call_accuracy` | Deterministic agent tool-call correctness. No LLM. | No |
+| `eval_audit_pack` | Build a hash-chained, procurement-ready ZIP from a pdfhell run. | No |
+
+## Example session
+
+```
+User: I just shipped a RAG endpoint. Can you check it for hallucinations?
+
+Claude: I'll use multivon to evaluate it.
+        [calls eval_discover to see what's available]
+        [calls eval_faithfulness with your input/context/output]
+
+→ score: 0.667 (passed: False), threshold: 0.9
+  reason: 2/3 claims grounded
+    ✓ "annual renewal" — supported by context
+    ✓ "30-day notice" — supported by context
+    ✗ "automatic upgrade" — NOT in context
+
+Claude: Your RAG hallucinated the "automatic upgrade" detail. The context
+        doesn't mention upgrades. I'd add a Hallucination evaluator to your CI
+        gate, threshold ≥0.85, and re-prompt with explicit "only use facts
+        from context" instructions.
+```
+
+## Why these 9 tools (not all 44)
+
+`eval_discover` returns the full 44-evaluator catalog, so the agent can always introspect everything. The 9 tools we expose directly are the ones agents actually call mid-edit:
+
+- RAG checks (faithfulness, hallucination, relevance) — most common need
+- Agent traces (tool_call_accuracy) — second most common
+- Document AI (pdfhell.run, pdfhell.make) — for any RAG-on-PDFs flow
+- Audit pack — when procurement is involved
+- Discover — meta-capability for planning
+
+Exposing all 44 evaluators as MCP tools would bloat the agent's context window and overwhelm tool-selection. If you need an evaluator that's not directly exposed, the agent can still use `multivon-eval` as a library — `eval_discover` returns the import paths.
+
+## Dependencies
+
+- `mcp[cli] >= 1.0` — official MCP Python SDK + the `mcp dev` inspector
+- `multivon-eval >= 0.7.3` — the evaluator surface this wraps
+- `pdfhell >= 0.1.0` — the adversarial-PDF benchmark this wraps
+
+All Apache 2.0.
+
+## License
+
+Apache 2.0.
+
+## Citing
+
+```bibtex
+@software{multivon_mcp,
+  title  = {multivon-mcp: MCP server exposing multivon-eval + pdfhell as agent-callable tools},
+  author = {Multivon},
+  year   = {2026},
+  url    = {https://github.com/multivon-ai/multivon-mcp},
+}
+```

multivon_mcp-0.1.0/README.md

@@ -0,0 +1,134 @@
+# multivon-mcp
+
+**MCP server that gives AI coding agents direct access to evaluation tools.** Drop into Claude Desktop, Claude Code, Cursor, Cline, or any [Model Context Protocol](https://modelcontextprotocol.io)–compatible agent.
+
+When the agent is helping you build an LLM product, it can:
+
+- Score a RAG output for hallucination without you writing the scaffolding
+- Generate an adversarial PDF on demand to test your document AI
+- Run the full pdfhell mini-suite against a model and analyse the results
+- Produce a hash-chained audit pack for procurement diligence
+- Discover the full evaluation capability catalog as JSON
+
+No copy-paste, no `python -c "..."`, no asking the agent to figure out the SDK calls.
+
+## Install
+
+```bash
+pip install multivon-mcp
+```
+
+Bare install pulls `multivon-eval`, `pdfhell`, and the MCP SDK. The provider SDKs (`anthropic`, `openai`, `google-genai`) come along too — bring your own API key in env.
+
+## Configure your agent
+
+### Claude Desktop / Claude Code
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
+
+```json
+{
+  "mcpServers": {
+    "multivon": {
+      "command": "multivon-mcp",
+      "env": {
+        "ANTHROPIC_API_KEY": "sk-ant-...",
+        "OPENAI_API_KEY": "sk-proj-...",
+        "GOOGLE_API_KEY": "AIza..."
+      }
+    }
+  }
+}
+```
+
+Restart Claude. The 9 tools become available; ask Claude `"use multivon to evaluate this RAG output"` and it figures out which tool to call.
+
+### Cursor
+
+`cursor.json` or via Settings → MCP:
+
+```json
+{ "mcpServers": { "multivon": { "command": "multivon-mcp" } } }
+```
+
+### Cline / OpenCode / any MCP-compatible agent
+
+Same shape — point at the `multivon-mcp` console script.
+
+### Local dev / debugging
+
+```bash
+mcp dev multivon_mcp.server
+```
+
+Opens the MCP Inspector UI in your browser. You can call any tool by name, see the JSON schemas, and watch the requests/responses.
+
+## The 9 tools
+
+| Tool | What it does | API key needed |
+|---|---|---|
+| `eval_discover` | Returns the full machine-readable capability catalog (evaluators, traps, suites, calibration data, versions). Call this first. | No |
+| `pdfhell_make` | Generates one adversarial PDF + its answer key. Useful for inspecting what a trap looks like. | No |
+| `pdfhell_run` | Runs the pdfhell adversarial-PDF benchmark against a vision model. Returns pass rate, per-trap CIs, suite hash. | Yes (vision provider) |
+| `eval_faithfulness` | QAG-graded faithfulness — is a RAG output grounded in the retrieved context? | Yes (judge) |
+| `eval_hallucination` | QAG-graded hallucination detection — does an output contain content NOT in context? | Yes (judge) |
+| `eval_relevance` | QAG-graded answer-vs-question relevance. | Yes (judge) |
+| `eval_answer_accuracy` | QAG-graded semantic equivalence vs ground truth. | Yes (judge) |
+| `eval_tool_call_accuracy` | Deterministic agent tool-call correctness. No LLM. | No |
+| `eval_audit_pack` | Build a hash-chained, procurement-ready ZIP from a pdfhell run. | No |
+
+## Example session
+
+```
+User: I just shipped a RAG endpoint. Can you check it for hallucinations?
+
+Claude: I'll use multivon to evaluate it.
+        [calls eval_discover to see what's available]
+        [calls eval_faithfulness with your input/context/output]
+
+→ score: 0.667 (passed: False), threshold: 0.9
+  reason: 2/3 claims grounded
+    ✓ "annual renewal" — supported by context
+    ✓ "30-day notice" — supported by context
+    ✗ "automatic upgrade" — NOT in context
+
+Claude: Your RAG hallucinated the "automatic upgrade" detail. The context
+        doesn't mention upgrades. I'd add a Hallucination evaluator to your CI
+        gate, threshold ≥0.85, and re-prompt with explicit "only use facts
+        from context" instructions.
+```
+
+## Why these 9 tools (not all 44)
+
+`eval_discover` returns the full 44-evaluator catalog, so the agent can always introspect everything. The 9 tools we expose directly are the ones agents actually call mid-edit:
+
+- RAG checks (faithfulness, hallucination, relevance) — most common need
+- Agent traces (tool_call_accuracy) — second most common
+- Document AI (pdfhell.run, pdfhell.make) — for any RAG-on-PDFs flow
+- Audit pack — when procurement is involved
+- Discover — meta-capability for planning
+
+Exposing all 44 evaluators as MCP tools would bloat the agent's context window and overwhelm tool-selection. If you need an evaluator that's not directly exposed, the agent can still use `multivon-eval` as a library — `eval_discover` returns the import paths.
+
+## Dependencies
+
+- `mcp[cli] >= 1.0` — official MCP Python SDK + the `mcp dev` inspector
+- `multivon-eval >= 0.7.3` — the evaluator surface this wraps
+- `pdfhell >= 0.1.0` — the adversarial-PDF benchmark this wraps
+
+All Apache 2.0.
+
+## License
+
+Apache 2.0.
+
+## Citing
+
+```bibtex
+@software{multivon_mcp,
+  title  = {multivon-mcp: MCP server exposing multivon-eval + pdfhell as agent-callable tools},
+  author = {Multivon},
+  year   = {2026},
+  url    = {https://github.com/multivon-ai/multivon-mcp},
+}
+```

multivon_mcp-0.1.0/multivon_mcp/__init__.py

@@ -0,0 +1,28 @@
+"""multivon-mcp — agent-callable evaluation tools.
+
+Drop into Claude Desktop, Cursor, Cline, or any MCP-compatible agent
+to give it direct access to multivon-eval + pdfhell tools without
+shelling out to ``python -c`` or copy-pasting code.
+
+Quickstart::
+
+    pip install multivon-mcp
+
+    # Claude Desktop / Claude Code config (mcpServers):
+    {
+      "mcpServers": {
+        "multivon": {"command": "multivon-mcp"}
+      }
+    }
+
+After registering, ask Claude:
+    "use multivon to evaluate this RAG output for faithfulness"
+
+The agent discovers the 8 available tools via the MCP capabilities
+handshake and calls them directly.
+"""
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+__all__ = ["__version__"]

multivon_mcp-0.1.0/multivon_mcp/server.py

@@ -0,0 +1,67 @@
+"""FastMCP server entry point.
+
+``multivon-mcp`` (the console script) starts this server in stdio
+transport mode — exactly what Claude Desktop / Cursor / Cline expect
+when configured via ``mcpServers``.
+
+The 8 tools registered:
+
+  pdfhell.run                     — evaluate a vision model on the suite
+  pdfhell.make                    — generate one trap PDF + answer key
+  eval.faithfulness               — QAG-graded RAG faithfulness
+  eval.hallucination              — QAG-graded hallucination detection
+  eval.relevance                  — QAG-graded answer-vs-question relevance
+  eval.tool_call_accuracy         — agent tool-call correctness (no LLM judge)
+  eval.answer_accuracy            — QAG-graded semantic-equivalence
+  eval.audit_pack                 — build a hash-chained audit ZIP from a run
+  eval.discover                   — full machine-readable capability catalog
+
+Why 8 (not 43): the narrow set is the surface AI coding agents actually
+need mid-edit. The full evaluator catalog stays available via
+``eval.discover`` for the agents that want to inspect everything.
+"""
+from __future__ import annotations
+
+import sys
+
+from mcp.server.fastmcp import FastMCP
+
+from . import __version__
+from .tools import register_all
+
+
+def build_server() -> FastMCP:
+    """Build and configure the FastMCP server. Factored out for tests
+    that want to introspect the registered tools without actually
+    serving."""
+    mcp = FastMCP(
+        name="multivon-mcp",
+        instructions=(
+            "Multivon's evaluation toolkit for AI agents. Use eval.discover() "
+            "at session start to see every available evaluator + trap family. "
+            "For RAG outputs, prefer eval.faithfulness + eval.hallucination. "
+            "For agent traces, use eval.tool_call_accuracy. For document AI, "
+            "use pdfhell.run with a vision model. All judge calls require "
+            "the matching provider's API key in env "
+            "(ANTHROPIC_API_KEY / OPENAI_API_KEY / GOOGLE_API_KEY)."
+        ),
+    )
+    register_all(mcp)
+    return mcp
+
+
+def main() -> None:
+    """Console-script entry point. Runs the server over stdio.
+
+    For local dev / debugging, you can also run ``mcp dev multivon_mcp.server``
+    which opens the MCP Inspector UI on a local port.
+    """
+    if "--version" in sys.argv:
+        print(f"multivon-mcp {__version__}")
+        return
+    mcp = build_server()
+    mcp.run()
+
+
+if __name__ == "__main__":
+    main()

multivon_mcp-0.1.0/multivon_mcp/tools/__init__.py

@@ -0,0 +1,18 @@
+"""Tool modules. Each registers a small group of related MCP tools on
+the FastMCP server instance passed in."""
+
+from .pdfhell_tools import register as register_pdfhell
+from .eval_tools import register as register_eval
+from .audit_tools import register as register_audit
+from .discover_tools import register as register_discover
+
+
+def register_all(mcp) -> None:
+    """Register every tool group on the FastMCP server."""
+    register_pdfhell(mcp)
+    register_eval(mcp)
+    register_audit(mcp)
+    register_discover(mcp)
+
+
+__all__ = ["register_all"]

multivon_mcp-0.1.0/multivon_mcp/tools/audit_tools.py

@@ -0,0 +1,104 @@
+"""Audit-pack MCP tool.
+
+Wraps pdfhell's audit-pack generation. The agent calls this after a
+pdfhell run to produce a procurement-ready ZIP with hash-chained
+manifest, PDFs, answer keys, JUnit XML, and a human-readable README.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def register(mcp) -> None:
+    """Register audit tools on the FastMCP server."""
+
+    @mcp.tool()
+    def eval_audit_pack(
+        run_json_path: str,
+        cases_dir: str,
+        output_zip_path: str,
+    ) -> dict[str, Any]:
+        """Build a hash-chained audit ZIP from a pdfhell run.
+
+        Combines the run JSON, the case PDFs + answer keys, JUnit XML,
+        and a SHA-256 manifest into one downloadable ZIP. Suitable for
+        attaching to a procurement diligence appendix.
+
+        Args:
+            run_json_path: Path to a pdfhell run JSON (from ``pdfhell run --out``).
+            cases_dir: Directory containing the case PDFs + answer keys that
+                were evaluated. Same dir the run used.
+            output_zip_path: Where to write the audit ZIP.
+
+        Returns:
+            ``{"path": "/abs/path/to.zip", "size_bytes": N, "manifest": {...}}``.
+            The manifest dict mirrors the one inside the ZIP — useful for
+            an agent that wants to verify the contents without opening
+            the ZIP itself.
+        """
+        from pdfhell.auditpack import build_audit_pack
+        from pdfhell.scorer import SuiteReport, CaseScore
+
+        run_path = Path(run_json_path).expanduser().resolve()
+        if not run_path.is_file():
+            return {"error": f"run JSON not found: {run_path}"}
+        cases_path = Path(cases_dir).expanduser().resolve()
+        if not cases_path.is_dir():
+            return {"error": f"cases dir not found: {cases_path}"}
+
+        # Reconstruct a SuiteReport from the JSON. We only need the fields
+        # the audit-pack builder reads.
+        raw = json.loads(run_path.read_text(encoding="utf-8"))
+        cases = [
+            CaseScore(
+                case_id=c["case_id"],
+                trap_family=c["trap_family"],
+                correct=bool(c["correct"]),
+                fell_for_trap=bool(c.get("fell_for_trap", False)),
+                refused=bool(c.get("refused", False)),
+                matched_expected=bool(c.get("matched_expected", False)),
+                matched_forbidden=list(c.get("matched_forbidden", [])),
+                model_output=c.get("model_output", ""),
+                expected=c.get("expected", ""),
+                failure_mode=c.get("failure_mode", ""),
+            )
+            for c in raw.get("cases", [])
+        ]
+        report = SuiteReport(
+            model=raw["model"],
+            suite=raw["suite"],
+            n=raw["n"],
+            pass_rate=raw["pass_rate"],
+            per_trap_pass=raw.get("per_trap_pass", {}),
+            per_trap_fell_for_trap=raw.get("per_trap_fell_for_trap", {}),
+            refused_rate=raw.get("refused_rate", 0.0),
+            cases=cases,
+            suite_version=raw.get("suite_version", ""),
+            suite_hash=raw.get("suite_hash", ""),
+        )
+
+        out_path = Path(output_zip_path).expanduser().resolve()
+        build_audit_pack(report, cases_path, out_path)
+
+        # Return a compact manifest summary the agent can use directly.
+        import zipfile
+        with zipfile.ZipFile(out_path, "r") as zf:
+            manifest = json.loads(zf.read("manifest.json").decode("utf-8"))
+        return {
+            "path": str(out_path),
+            "size_bytes": out_path.stat().st_size,
+            "manifest": {
+                "pdfhell_version": manifest["pdfhell_version"],
+                "model": manifest["model"],
+                "suite": manifest["suite"],
+                "suite_version": manifest.get("suite_version", ""),
+                "suite_hash": manifest.get("suite_hash", ""),
+                "n": manifest["n"],
+                "passed": manifest["passed"],
+                "pass_rate": manifest["pass_rate"],
+                "pass_rate_ci_95": manifest.get("pass_rate_ci_95", []),
+                "file_count": len(manifest["files"]),
+            },
+        }