testmcpy 0.7.26__tar.gz → 0.9.0__tar.gz

{testmcpy-0.7.26/testmcpy.egg-info → testmcpy-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: testmcpy
-Version: 0.7.26
+Version: 0.9.0
 Summary: A comprehensive testing framework for validating LLM tool calling capabilities with MCP services
 Author: Amin Ghadersohi
 License-Expression: Apache-2.0
@@ -50,8 +50,12 @@ Requires-Dist: fastapi<1.0.0,>=0.104.0; extra == "server"
 Requires-Dist: uvicorn[standard]<1.0.0,>=0.24.0; extra == "server"
 Requires-Dist: websockets<17.0,>=14.0; extra == "server"
 Requires-Dist: claude-agent-sdk<1.0.0,>=0.2.0; extra == "server"
+Requires-Dist: openai-agents<0.18,>=0.17; extra == "server"
+Requires-Dist: google-adk<3.0.0,>=2.0.0; extra == "server"
 Provides-Extra: sdk
 Requires-Dist: claude-agent-sdk<1.0.0,>=0.2.0; extra == "sdk"
+Requires-Dist: openai-agents<0.18,>=0.17; extra == "sdk"
+Requires-Dist: google-adk<3.0.0,>=2.0.0; extra == "sdk"
 Provides-Extra: tui
 Requires-Dist: textual>=0.85.0; extra == "tui"
 Provides-Extra: e2e
@@ -61,11 +65,15 @@ Provides-Extra: export
 Requires-Dist: pandas<3.0.0,>=2.0.0; extra == "export"
 Provides-Extra: bedrock
 Requires-Dist: boto3>=1.28.0; extra == "bedrock"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]<4.0,>=3.1; extra == "postgres"
 Provides-Extra: all
 Requires-Dist: fastapi<1.0.0,>=0.104.0; extra == "all"
 Requires-Dist: uvicorn[standard]<1.0.0,>=0.24.0; extra == "all"
 Requires-Dist: websockets<17.0,>=14.0; extra == "all"
 Requires-Dist: claude-agent-sdk<1.0.0,>=0.2.0; extra == "all"
+Requires-Dist: openai-agents<0.18,>=0.17; extra == "all"
+Requires-Dist: google-adk<3.0.0,>=2.0.0; extra == "all"
 Requires-Dist: textual>=0.85.0; extra == "all"
 Requires-Dist: boto3>=1.28.0; extra == "all"
 Dynamic: license-file
@@ -103,6 +111,20 @@ Dynamic: license-file
 - **Prevent regressions**: Catch breaking changes in your MCP service with CI/CD
 - **Optimize costs**: Track token usage and identify the most cost-effective models
 
+### How it compares
+
+| | testmcpy | [MCP Inspector](https://github.com/modelcontextprotocol/inspector) | [MCPJam](https://github.com/MCPJam/inspector) | [promptfoo](https://github.com/promptfoo/promptfoo) |
+|---|---|---|---|---|
+| Automated LLM-driven evals of MCP servers | ✅ YAML suites, 40+ evaluators | ❌ manual testing | ✅ | ⚠️ generic LLM eval with an MCP provider |
+| Multi-provider (Claude / GPT / Gemini / Ollama / Bedrock…) | ✅ 11 providers incl. agent SDKs | n/a | ✅ | ✅ |
+| CI gate with exit codes + JUnit | ✅ `--gate`, `--junit-xml` | ❌ | ✅ | ✅ |
+| Cost & token tracking per test/model | ✅ | ❌ | ⚠️ | ⚠️ |
+| Multi-turn, mutation & metamorphic testing | ✅ | ❌ | ❌ | ⚠️ |
+| Auth testing (JWT/OAuth/mTLS) + debugger | ✅ 7 auth types | ⚠️ OAuth only | ✅ OAuth debugger | ❌ |
+| Python-native (`pip`/`uvx`, pytest-friendly) | ✅ | ❌ npm | ❌ npm | ❌ npm |
+
+Use MCP Inspector for quick manual poking; reach for testmcpy when you want repeatable, scored, CI-gated evaluation of how real models use your server.
+
 ## Quick Start
 
 ```bash
@@ -124,14 +146,21 @@ That's it! No complex configuration needed to get started.
 
 ### Multi-Provider LLM Support
 
-Test with **Claude**, **GPT-4**, **Llama**, and other models. Works with both paid APIs and free local models via Ollama. Includes a Claude SDK provider for subprocess-based MCP support.
-
-| Provider | Models | Features |
-|----------|--------|----------|
-| Anthropic | claude-opus-4, claude-sonnet-4-5, claude-haiku-4-5 | Native MCP, extended thinking, vision, token caching |
-| OpenAI | gpt-4, gpt-4-turbo, gpt-4o | Function calling, vision, cost tracking |
-| Ollama | Llama, Mistral, etc. (local) | Free, local execution, no API costs |
-| Claude SDK | claude-cli, claude-code | Subprocess-based, full MCP support |
+Test with **Claude**, **GPT**, **Gemini**, **Llama**, and other models. Works with both paid APIs and free local models via Ollama. Includes agent-SDK providers (Claude, Codex, Gemini) with native MCP support.
+
+| Provider | Config name | Models | Features |
+|----------|-------------|--------|----------|
+| Anthropic | `anthropic` | claude-opus-4, claude-sonnet-4-5, claude-haiku-4-5 | Native MCP, extended thinking, vision, token caching |
+| OpenAI | `openai` | gpt-4, gpt-4-turbo, gpt-4o | Function calling, vision, cost tracking |
+| Ollama | `ollama` | Llama, Mistral, etc. (local) | Free, local execution, no API costs |
+| Claude SDK | `claude-sdk` (aliases: `claude-cli`, `claude-code`) | claude-sonnet-4-5, claude-opus-4 | Claude Agent SDK, native MCP, CLI OAuth login |
+| Codex SDK | `codex-sdk` (aliases: `codex-cli`, `codex`) | gpt-5-codex, o3, o4-mini | openai-agents SDK, native MCP, Codex CLI OAuth or API key |
+| Gemini SDK | `gemini-sdk` | gemini-sdk-flash, gemini-sdk-pro | google-adk, native MCP |
+| Google Gemini | `gemini` (alias: `google`) | gemini-2.5-flash, gemini-2.5-pro | Direct Gemini API, function calling |
+| Gemini CLI | `gemini-cli` | gemini-2.5-flash, gemini-2.5-pro | Subprocess-based Gemini CLI |
+| AWS Bedrock | `bedrock` (alias: `aws-bedrock`) | Claude models via AWS | IAM auth, no Anthropic key needed |
+| xAI | `xai` (alias: `grok`) | grok models | Function calling |
+| OpenRouter | `openrouter` | 100+ models with one API key | Function calling, cost tracking |
 
 ![LLM Profiles — manage Anthropic, OpenAI, Ollama and Claude SDK provider configurations](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/llm-profiles.png)
 
@@ -519,16 +548,51 @@ openai:
       default: true
 ```
 
-## CI/CD Integration
+## CI in 60 Seconds
+
+Gate your MCP service on eval results in any CI system — no wrapper required:
+
+```yaml
+# .github/workflows/mcp-tests.yml
+jobs:
+  mcp-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - name: Run MCP eval suite
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: |
+          uvx testmcpy run tests/ \
+            --mcp-url "$MCP_URL" \
+            --gate --min-pass-rate 85 \
+            --junit-xml junit.xml
+```
+
+- **`--gate`** exits non-zero when the run fails your thresholds, so the build fails. Tune thresholds in `.testmcpy-gate.yaml`:
+
+  ```yaml
+  min_pass_rate: 85.0       # % of tests that must pass
+  max_failures: 3           # absolute failure budget
+  required_tests:           # these must always pass
+    - critical_auth_flow
+  block_on_regression: true # fail on baseline regressions
+  ```
+
+- **`--junit-xml`** emits JUnit XML for CI systems that ingest it natively (Jenkins, GitLab, CircleCI, Buildkite). On GitHub Actions, pair it with an action like `dorny/test-reporter` — or just rely on the next bullet.
+- Inside GitHub Actions, the markdown eval report is **automatically appended to the job summary** — results render on the workflow run page with zero extra steps.
 
-testmcpy ships as a GitHub Action:
+Or use the bundled reusable Action — adds a sticky PR comment, JUnit artifact upload, and structured outputs (`pass-rate`, `gate_passed`):
 
 ```yaml
 - uses: preset-io/testmcpy@v1
   with:
-    test-path: tests/
-    model: claude-haiku-4-5
-    mcp-profile: prod
+    test_path: tests/
+    mcp_url: ${{ vars.MCP_URL }}
+    pass_threshold: '85'
+    pr_comment: 'true'
+    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
 ```
 
 ## Custom Evaluators

{testmcpy-0.7.26 → testmcpy-0.9.0}/README.md

@@ -31,6 +31,20 @@
 - **Prevent regressions**: Catch breaking changes in your MCP service with CI/CD
 - **Optimize costs**: Track token usage and identify the most cost-effective models
 
+### How it compares
+
+| | testmcpy | [MCP Inspector](https://github.com/modelcontextprotocol/inspector) | [MCPJam](https://github.com/MCPJam/inspector) | [promptfoo](https://github.com/promptfoo/promptfoo) |
+|---|---|---|---|---|
+| Automated LLM-driven evals of MCP servers | ✅ YAML suites, 40+ evaluators | ❌ manual testing | ✅ | ⚠️ generic LLM eval with an MCP provider |
+| Multi-provider (Claude / GPT / Gemini / Ollama / Bedrock…) | ✅ 11 providers incl. agent SDKs | n/a | ✅ | ✅ |
+| CI gate with exit codes + JUnit | ✅ `--gate`, `--junit-xml` | ❌ | ✅ | ✅ |
+| Cost & token tracking per test/model | ✅ | ❌ | ⚠️ | ⚠️ |
+| Multi-turn, mutation & metamorphic testing | ✅ | ❌ | ❌ | ⚠️ |
+| Auth testing (JWT/OAuth/mTLS) + debugger | ✅ 7 auth types | ⚠️ OAuth only | ✅ OAuth debugger | ❌ |
+| Python-native (`pip`/`uvx`, pytest-friendly) | ✅ | ❌ npm | ❌ npm | ❌ npm |
+
+Use MCP Inspector for quick manual poking; reach for testmcpy when you want repeatable, scored, CI-gated evaluation of how real models use your server.
+
 ## Quick Start
 
 ```bash
@@ -52,14 +66,21 @@ That's it! No complex configuration needed to get started.
 
 ### Multi-Provider LLM Support
 
-Test with **Claude**, **GPT-4**, **Llama**, and other models. Works with both paid APIs and free local models via Ollama. Includes a Claude SDK provider for subprocess-based MCP support.
-
-| Provider | Models | Features |
-|----------|--------|----------|
-| Anthropic | claude-opus-4, claude-sonnet-4-5, claude-haiku-4-5 | Native MCP, extended thinking, vision, token caching |
-| OpenAI | gpt-4, gpt-4-turbo, gpt-4o | Function calling, vision, cost tracking |
-| Ollama | Llama, Mistral, etc. (local) | Free, local execution, no API costs |
-| Claude SDK | claude-cli, claude-code | Subprocess-based, full MCP support |
+Test with **Claude**, **GPT**, **Gemini**, **Llama**, and other models. Works with both paid APIs and free local models via Ollama. Includes agent-SDK providers (Claude, Codex, Gemini) with native MCP support.
+
+| Provider | Config name | Models | Features |
+|----------|-------------|--------|----------|
+| Anthropic | `anthropic` | claude-opus-4, claude-sonnet-4-5, claude-haiku-4-5 | Native MCP, extended thinking, vision, token caching |
+| OpenAI | `openai` | gpt-4, gpt-4-turbo, gpt-4o | Function calling, vision, cost tracking |
+| Ollama | `ollama` | Llama, Mistral, etc. (local) | Free, local execution, no API costs |
+| Claude SDK | `claude-sdk` (aliases: `claude-cli`, `claude-code`) | claude-sonnet-4-5, claude-opus-4 | Claude Agent SDK, native MCP, CLI OAuth login |
+| Codex SDK | `codex-sdk` (aliases: `codex-cli`, `codex`) | gpt-5-codex, o3, o4-mini | openai-agents SDK, native MCP, Codex CLI OAuth or API key |
+| Gemini SDK | `gemini-sdk` | gemini-sdk-flash, gemini-sdk-pro | google-adk, native MCP |
+| Google Gemini | `gemini` (alias: `google`) | gemini-2.5-flash, gemini-2.5-pro | Direct Gemini API, function calling |
+| Gemini CLI | `gemini-cli` | gemini-2.5-flash, gemini-2.5-pro | Subprocess-based Gemini CLI |
+| AWS Bedrock | `bedrock` (alias: `aws-bedrock`) | Claude models via AWS | IAM auth, no Anthropic key needed |
+| xAI | `xai` (alias: `grok`) | grok models | Function calling |
+| OpenRouter | `openrouter` | 100+ models with one API key | Function calling, cost tracking |
 
 ![LLM Profiles — manage Anthropic, OpenAI, Ollama and Claude SDK provider configurations](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/llm-profiles.png)
 
@@ -447,16 +468,51 @@ openai:
       default: true
 ```
 
-## CI/CD Integration
+## CI in 60 Seconds
+
+Gate your MCP service on eval results in any CI system — no wrapper required:
+
+```yaml
+# .github/workflows/mcp-tests.yml
+jobs:
+  mcp-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - name: Run MCP eval suite
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: |
+          uvx testmcpy run tests/ \
+            --mcp-url "$MCP_URL" \
+            --gate --min-pass-rate 85 \
+            --junit-xml junit.xml
+```
+
+- **`--gate`** exits non-zero when the run fails your thresholds, so the build fails. Tune thresholds in `.testmcpy-gate.yaml`:
+
+  ```yaml
+  min_pass_rate: 85.0       # % of tests that must pass
+  max_failures: 3           # absolute failure budget
+  required_tests:           # these must always pass
+    - critical_auth_flow
+  block_on_regression: true # fail on baseline regressions
+  ```
+
+- **`--junit-xml`** emits JUnit XML for CI systems that ingest it natively (Jenkins, GitLab, CircleCI, Buildkite). On GitHub Actions, pair it with an action like `dorny/test-reporter` — or just rely on the next bullet.
+- Inside GitHub Actions, the markdown eval report is **automatically appended to the job summary** — results render on the workflow run page with zero extra steps.
 
-testmcpy ships as a GitHub Action:
+Or use the bundled reusable Action — adds a sticky PR comment, JUnit artifact upload, and structured outputs (`pass-rate`, `gate_passed`):
 
 ```yaml
 - uses: preset-io/testmcpy@v1
   with:
-    test-path: tests/
-    model: claude-haiku-4-5
-    mcp-profile: prod
+    test_path: tests/
+    mcp_url: ${{ vars.MCP_URL }}
+    pass_threshold: '85'
+    pr_comment: 'true'
+    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
 ```
 
 ## Custom Evaluators

{testmcpy-0.7.26 → testmcpy-0.9.0}/pyproject.toml

@@ -93,7 +93,7 @@ testmcpy = [
 
 [project]
 name = "testmcpy"
-version = "0.7.26"
+version = "0.9.0"
 description = "A comprehensive testing framework for validating LLM tool calling capabilities with MCP services"
 authors = [{name = "Amin Ghadersohi"}]
 license = "Apache-2.0"
@@ -171,9 +171,19 @@ server = [
     # that bumps mcp out of fastmcp's range fails at install time
     # instead of at first agentic test run.
     "claude-agent-sdk>=0.2.0,<1.0.0",
+    # openai-agents provides CodexSDKProvider with native MCP support.
+    # Pinned to 0.17.x — the minor that introduced the agents.mcp /
+    # agents.models.openai_provider / agents.run_config import paths we rely on.
+    "openai-agents>=0.17,<0.18",
+    # google-adk provides GeminiSDKProvider with native MCP support via McpToolset.
+    # Pinned to 2.x — the version that introduced StreamableHTTPConnectionParams.
+    # Transitively pulls google-genai>=2.4,<3 and mcp>=1.24,<2 (already pinned).
+    "google-adk>=2.0.0,<3.0.0",
 ]
 sdk = [
     "claude-agent-sdk>=0.2.0,<1.0.0",
+    "openai-agents>=0.17,<0.18",
+    "google-adk>=2.0.0,<3.0.0",
 ]
 tui = [
     "textual>=0.85.0",
@@ -188,6 +198,11 @@ export = [
 bedrock = [
     "boto3>=1.28.0",
 ]
+postgres = [
+    # Enables TESTMCPY_DB_URL=postgresql+psycopg://... for a shared
+    # results database. SQLite stays the default.
+    "psycopg[binary]>=3.1,<4.0",
+]
 all = [
     "fastapi>=0.104.0,<1.0.0",
     "uvicorn[standard]>=0.24.0,<1.0.0",
@@ -196,6 +211,8 @@ all = [
     # time uvicorn/fastmcp bumps a minor.
     "websockets>=14.0,<17.0",
     "claude-agent-sdk>=0.2.0,<1.0.0",
+    "openai-agents>=0.17,<0.18",
+    "google-adk>=2.0.0,<3.0.0",
     "textual>=0.85.0",
     "boto3>=1.28.0",
 ]

testmcpy-0.9.0/testmcpy/__init__.py

@@ -0,0 +1,17 @@
+"""
+testmcpy - MCP Testing Framework
+
+A comprehensive testing framework for validating LLM tool calling
+capabilities with MCP (Model Context Protocol) services.
+"""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("testmcpy")
+except PackageNotFoundError:
+    # Running from a bare checkout without an installed package — use an
+    # obviously-not-a-release marker instead of a stale hardcoded version.
+    __version__ = "0.0.0+unknown"
+
+__author__ = "testmcpy Contributors"

testmcpy-0.9.0/testmcpy/__main__.py

@@ -0,0 +1,6 @@
+"""Allow `python -m testmcpy` to run the CLI (used by `testmcpy bench`)."""
+
+from testmcpy.cli import app
+
+if __name__ == "__main__":
+    app()

testmcpy-0.9.0/testmcpy/analytics.py

@@ -0,0 +1,338 @@
+"""Aggregation core for test-performance analytics.
+
+Answers "which tests perform better than others given the model /
+routing config": per-question × per-config pass rates, scores, costs,
+latency, flakiness, and day-bucketed trends.
+
+Pure query/aggregation functions over a SQLAlchemy session, shared by
+the /api/analytics router and the `testmcpy matrix | leaderboard |
+flaky` CLI commands so both surfaces report identical numbers.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from sqlalchemy import case, func
+from sqlalchemy.orm import Session
+
+from testmcpy.models import QuestionResultModel, TestRunModel
+
+# A cell needs at least this many runs before an intermittent result is
+# called "flaky" rather than noise.
+FLAKY_MIN_RUNS = 3
+
+# Number of day buckets in a cell's trend sparkline.
+TREND_BUCKETS = 7
+
+
+def config_key(model: str, provider: str, mcp_profile_id: str | None = None) -> str:
+    """Canonical label for a model/provider/profile combination."""
+    key = f"{provider}/{model}"
+    if mcp_profile_id:
+        key = f"{key} @ {mcp_profile_id}"
+    return key
+
+
+def _pass_rate_expr() -> Any:
+    """Portable AVG over a boolean column (0.0–1.0)."""
+    return func.avg(case((QuestionResultModel.passed.is_(True), 1.0), else_=0.0))
+
+
+def _apply_run_filters(
+    query,
+    suite_id: str | None,
+    date_from: str | None,
+    date_to: str | None,
+):
+    query = query.filter(TestRunModel.status == "completed")
+    if suite_id:
+        query = query.filter(TestRunModel.suite_id == suite_id)
+    if date_from:
+        query = query.filter(TestRunModel.started_at >= date_from)
+    if date_to:
+        query = query.filter(TestRunModel.started_at <= date_to)
+    return query
+
+
+def test_matrix(
+    session: Session,
+    suite_id: str | None = None,
+    date_from: str | None = None,
+    date_to: str | None = None,
+    min_runs: int = 1,
+    include_profile: bool = True,
+    trend_buckets: int = TREND_BUCKETS,
+) -> dict[str, Any]:
+    """Per-question × per-config aggregation.
+
+    Returns ``{"configs": [...], "rows": [...], "warnings": [...]}`` where
+    each row carries one cell per config with n, pass_rate, flaky flag,
+    avg score/cost/latency, and a day-bucketed pass-rate trend.
+    """
+    config_cols: list[Any] = [TestRunModel.model, TestRunModel.provider]
+    if include_profile:
+        config_cols.append(TestRunModel.mcp_profile_id)
+
+    def cell_key(row) -> str:
+        profile = row.mcp_profile_id if include_profile else None
+        return config_key(row.model, row.provider, profile)
+
+    group_cols = [QuestionResultModel.question_id, *config_cols]
+    cells_query = _apply_run_filters(
+        session.query(
+            *group_cols,
+            func.count(QuestionResultModel.id).label("n"),
+            _pass_rate_expr().label("pass_rate"),
+            func.avg(QuestionResultModel.score).label("avg_score"),
+            func.avg(func.coalesce(QuestionResultModel.cost_usd, 0.0)).label("avg_cost"),
+            func.avg(func.coalesce(QuestionResultModel.duration_ms, 0)).label("avg_duration_ms"),
+            func.max(TestRunModel.started_at).label("last_run_at"),
+        ).join(TestRunModel, TestRunModel.run_id == QuestionResultModel.run_id),
+        suite_id,
+        date_from,
+        date_to,
+    ).group_by(*group_cols)
+
+    # Day-bucketed pass rates for trend sparklines. started_at is an ISO
+    # string, so the first 10 chars are the date — portable across
+    # SQLite and Postgres.
+    day_expr = func.substr(TestRunModel.started_at, 1, 10)
+    trend_query = _apply_run_filters(
+        session.query(
+            *group_cols,
+            day_expr.label("day"),
+            _pass_rate_expr().label("pass_rate"),
+        ).join(TestRunModel, TestRunModel.run_id == QuestionResultModel.run_id),
+        suite_id,
+        date_from,
+        date_to,
+    ).group_by(*group_cols, day_expr)
+
+    trends: dict[tuple[str, str], list[tuple[str, float]]] = {}
+    for row in trend_query:
+        trends.setdefault((row.question_id, cell_key(row)), []).append(
+            (row.day, round(row.pass_rate or 0.0, 4))
+        )
+
+    rows: dict[str, dict[str, Any]] = {}
+    config_totals: dict[str, dict[str, Any]] = {}
+    single_run_cells = 0
+
+    for row in cells_query:
+        key = cell_key(row)
+        n = int(row.n or 0)
+        if n < min_runs:
+            continue
+        pass_rate = round(row.pass_rate or 0.0, 4)
+        flaky = 0.0 < pass_rate < 1.0 and n >= FLAKY_MIN_RUNS
+        if n == 1:
+            single_run_cells += 1
+
+        trend_points = sorted(trends.get((row.question_id, key), []))[-trend_buckets:]
+        rows.setdefault(row.question_id, {"question_id": row.question_id, "cells": {}})["cells"][
+            key
+        ] = {
+            "n": n,
+            "pass_rate": pass_rate,
+            "flaky": flaky,
+            "avg_score": round(row.avg_score or 0.0, 4),
+            "avg_cost": round(row.avg_cost or 0.0, 6),
+            "avg_duration_ms": round(row.avg_duration_ms or 0.0, 1),
+            "last_run_at": row.last_run_at,
+            "trend": [rate for _, rate in trend_points],
+        }
+
+        totals = config_totals.setdefault(
+            key,
+            {
+                "key": key,
+                "model": row.model,
+                "provider": row.provider,
+                "mcp_profile": row.mcp_profile_id if include_profile else None,
+                "n": 0,
+                "passed_weight": 0.0,
+                "total_cost": 0.0,
+                "flaky_cells": 0,
+            },
+        )
+        totals["n"] += n
+        totals["passed_weight"] += pass_rate * n
+        totals["total_cost"] += (row.avg_cost or 0.0) * n
+        totals["flaky_cells"] += 1 if flaky else 0
+
+    # Distinct completed-run counts per config (cells count question
+    # results; the leaderboard wants whole runs).
+    runs_query = _apply_run_filters(
+        session.query(
+            *config_cols,
+            func.count(TestRunModel.run_id).label("n_runs"),
+        ),
+        suite_id,
+        date_from,
+        date_to,
+    ).group_by(*config_cols)
+    run_counts = {cell_key(row): int(row.n_runs or 0) for row in runs_query}
+
+    configs = []
+    for key, totals in sorted(config_totals.items()):
+        n = totals["n"]
+        configs.append(
+            {
+                "key": key,
+                "model": totals["model"],
+                "provider": totals["provider"],
+                "mcp_profile": totals["mcp_profile"],
+                "n_runs": run_counts.get(key, 0),
+                "n_results": n,
+                "pass_rate": round(totals["passed_weight"] / n, 4) if n else 0.0,
+                "total_cost": round(totals["total_cost"], 6),
+                "flaky_cells": totals["flaky_cells"],
+            }
+        )
+
+    warnings = []
+    if single_run_cells:
+        warnings.append(
+            f"{single_run_cells} cell(s) have n=1 — single runs are noise, "
+            f"re-run with `testmcpy bench --repeat {FLAKY_MIN_RUNS}` for signal"
+        )
+
+    return {
+        "configs": configs,
+        "rows": sorted(rows.values(), key=lambda r: r["question_id"]),
+        "warnings": warnings,
+    }
+
+
+def leaderboard(
+    session: Session,
+    suite_id: str | None = None,
+    date_from: str | None = None,
+    date_to: str | None = None,
+    include_profile: bool = True,
+) -> list[dict[str, Any]]:
+    """Configs ranked by pass rate, with cost-per-pass and latency."""
+    matrix = test_matrix(
+        session,
+        suite_id=suite_id,
+        date_from=date_from,
+        date_to=date_to,
+        include_profile=include_profile,
+    )
+
+    # Aggregate latency per config from the row cells.
+    latency: dict[str, list[tuple[float, int]]] = {}
+    passes: dict[str, float] = {}
+    for row in matrix["rows"]:
+        for key, cell in row["cells"].items():
+            latency.setdefault(key, []).append((cell["avg_duration_ms"], cell["n"]))
+            passes[key] = passes.get(key, 0.0) + cell["pass_rate"] * cell["n"]
+
+    ranked = []
+    for config in matrix["configs"]:
+        key = config["key"]
+        weighted = latency.get(key, [])
+        total_n = sum(n for _, n in weighted)
+        avg_duration = sum(duration * n for duration, n in weighted) / total_n if total_n else 0.0
+        passed = passes.get(key, 0.0)
+        ranked.append(
+            {
+                **config,
+                "avg_duration_ms": round(avg_duration, 1),
+                "cost_per_pass": round(config["total_cost"] / passed, 6) if passed else None,
+            }
+        )
+
+    ranked.sort(key=lambda c: (-c["pass_rate"], c["cost_per_pass"] or float("inf")))
+    return ranked
+
+
+def question_history(
+    session: Session,
+    question_id: str,
+    suite_id: str | None = None,
+    model: str | None = None,
+    provider: str | None = None,
+    mcp_profile: str | None = None,
+    limit: int = 100,
+) -> list[dict[str, Any]]:
+    """Chronological per-run points for one question (drill-down view)."""
+    query = _apply_run_filters(
+        session.query(
+            QuestionResultModel.run_id,
+            QuestionResultModel.passed,
+            QuestionResultModel.score,
+            QuestionResultModel.cost_usd,
+            QuestionResultModel.duration_ms,
+            QuestionResultModel.error,
+            TestRunModel.model,
+            TestRunModel.provider,
+            TestRunModel.mcp_profile_id,
+            TestRunModel.started_at,
+        )
+        .join(TestRunModel, TestRunModel.run_id == QuestionResultModel.run_id)
+        .filter(QuestionResultModel.question_id == question_id),
+        suite_id,
+        None,
+        None,
+    )
+    if model:
+        query = query.filter(TestRunModel.model == model)
+    if provider:
+        query = query.filter(TestRunModel.provider == provider)
+    if mcp_profile:
+        query = query.filter(TestRunModel.mcp_profile_id == mcp_profile)
+
+    rows = query.order_by(TestRunModel.started_at.desc()).limit(limit).all()
+    points = [
+        {
+            "run_id": row.run_id,
+            "started_at": row.started_at,
+            "passed": bool(row.passed),
+            "score": round(row.score or 0.0, 4),
+            "cost_usd": round(row.cost_usd or 0.0, 6),
+            "duration_ms": row.duration_ms or 0,
+            "error": row.error,
+            "model": row.model,
+            "provider": row.provider,
+            "mcp_profile": row.mcp_profile_id,
+            "config": config_key(row.model, row.provider, row.mcp_profile_id),
+        }
+        for row in rows
+    ]
+    points.reverse()  # chronological
+    return points
+
+
+def flaky_tests(
+    session: Session,
+    suite_id: str | None = None,
+    date_from: str | None = None,
+    date_to: str | None = None,
+    min_runs: int = FLAKY_MIN_RUNS,
+) -> list[dict[str, Any]]:
+    """Flaky question × config cells, sorted by how intermittent they are."""
+    matrix = test_matrix(
+        session,
+        suite_id=suite_id,
+        date_from=date_from,
+        date_to=date_to,
+        min_runs=min_runs,
+    )
+    flaky = []
+    for row in matrix["rows"]:
+        for key, cell in row["cells"].items():
+            if cell["flaky"]:
+                flaky.append(
+                    {
+                        "question_id": row["question_id"],
+                        "config": key,
+                        "n": cell["n"],
+                        "pass_rate": cell["pass_rate"],
+                        "last_run_at": cell["last_run_at"],
+                    }
+                )
+    # Most intermittent first: pass rates near 0.5 are the flakiest.
+    flaky.sort(key=lambda f: abs(f["pass_rate"] - 0.5))
+    return flaky