pytest-agent-check 0.0.1__py3-none-any.whl

pytest_agent_check-0.0.1.dist-info/METADATA

@@ -0,0 +1,390 @@
+Metadata-Version: 2.4
+Name: pytest-agent-check
+Version: 0.0.1
+Summary: A pytest plugin for evaluating and testing AI agents — record, replay, and assert agent behavior
+Author-email: Xianpeng Shen <shenxianpeng@example.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/shenxianpeng/pytest-agent-check
+Project-URL: Repository, https://github.com/shenxianpeng/pytest-agent-check
+Project-URL: Issues, https://github.com/shenxianpeng/pytest-agent-check/issues
+Keywords: pytest,plugin,agent,AI,evaluation,testing,LLM
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Framework :: Pytest
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pytest>=8.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: deepdiff>=7.0
+Requires-Dist: rich>=13.0
+Provides-Extra: semantic
+Requires-Dist: sentence-transformers>=2.2; extra == "semantic"
+Provides-Extra: html
+Requires-Dist: jinja2>=3.0; extra == "html"
+Provides-Extra: cli
+Requires-Dist: typer>=0.9; extra == "cli"
+Provides-Extra: all
+Requires-Dist: pytest-agent-check[cli,html,semantic]; extra == "all"
+Dynamic: license-file
+
+<h1 align="center">pytest-agent-check</h1>
+
+<p align="center">
+  <em>A pytest plugin for evaluating and testing AI agents — record, replay, and assert agent behaviour with confidence.</em>
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/pytest-agent-check/">
+    <img src="https://img.shields.io/pypi/v/pytest-agent-check" alt="PyPI">
+  </a>
+  <a href="https://pypi.org/project/pytest-agent-check/">
+    <img src="https://img.shields.io/pypi/pyversions/pytest-agent-check" alt="Python versions">
+  </a>
+  <a href="https://github.com/shenxianpeng/pytest-agent-check/actions/workflows/ci.yml">
+    <img src="https://github.com/shenxianpeng/pytest-agent-check/actions/workflows/ci.yml/badge.svg" alt="CI">
+  </a>
+  <a href="https://opensource.org/licenses/MIT">
+    <img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT">
+  </a>
+</p>
+
+---
+
+## Overview
+
+**pytest-agent-check** brings the familiar **snapshot testing** and **VCR-like cassette** patterns to AI agent evaluation.
+
+If you've ever wished you could test your agent as confidently as you test your REST API — with deterministic assertions, offline replay, and a clear diff when something changes — this plugin is for you.
+
+### Why?
+
+AI agents are **non-deterministic** and **expensive to run**. You can't just ``assert result == expected`` when the output is natural language. And every test run that calls a real LLM costs money, time, and patience.
+
+This plugin solves both problems:
+
+1. **Cassette record/replay** — Run tests once in *record mode* to capture all tool calls and outputs as a "cassette" (YAML file). Subsequent runs replay from the cassette: fast, offline, deterministic, free.
+2. **Fluent assertion API** — Assert on the *structure* of tool-call sequences (exact matching) and the *intent* of natural-language output (semantic matching, coming soon).
+3. **Change detection** — When agent behaviour drifts, get a structured diff showing exactly which tool calls changed, were added, or were removed.
+
+---
+
+## Installation
+
+```bash
+pip install pytest-agent-check
+```
+
+Optional extras:
+
+```bash
+pip install pytest-agent-check[semantic]   # sentence-transformers for semantic similarity
+pip install pytest-agent-check[cli]        # typer-based `agent-eval` CLI
+pip install pytest-agent-check[html]       # jinja2 for HTML reports
+pip install pytest-agent-check[all]        # everything above
+```
+
+### Requirements
+
+- Python 3.10+
+- pytest 8.0+
+
+---
+
+## Quick Start
+
+### 1. Write an agent test
+
+```python
+# test_support_agent.py
+import pytest
+from pytest_agent_eval import agent_test, expect_tools, expect_output
+
+
+@pytest.fixture
+def agent():
+    """Provide your agent under test."""
+    from my_agent import create_agent
+    return create_agent()
+
+
+@agent_test(agent="support-agent", cassette="refund_flow")
+def test_refund_request(agent, cassette):
+    """Agent should process refunds: lookup → eligibility → issue."""
+    result = cassette.run(agent.run, "I want a refund for order ORD-001")
+
+    expect_tools(result).called("lookup_order") \
+                        .then("check_refund_eligibility") \
+                        .then("issue_refund")
+
+    expect_output(result).matches_intent(
+        "confirms refund is approved and provides arrival time"
+    )
+```
+
+### 2. Record cassettes (first run)
+
+```bash
+agent-eval record test_support_agent.py
+# ─ or ─
+pytest test_support_agent.py --cassette-mode=record -v
+```
+
+This runs your **real agent**, captures all tool calls and outputs, and saves them to ``.cassettes/refund_flow.yaml``.
+
+### 3. Run tests in replay mode
+
+```bash
+# Requires the CLI extra: pip install pytest-agent-check[cli]
+agent-eval run test_support_agent.py
+# ─ or ─
+pytest test_support_agent.py --cassette-mode=replay -v
+```
+
+Now tests run **offline** — no API calls, no LLM latency, no cost. The recorded data is returned instantly.
+
+### 4. Update baselines when behaviour changes
+
+```bash
+agent-eval update test_support_agent.py
+```
+
+Re-records all cassettes. Commit the updated YAML files alongside your code changes.
+
+---
+
+## Core Concepts
+
+### Cassette (`.cassettes/`)
+
+A cassette is a YAML file that records everything your agent did during a test:
+
+```yaml
+agent_name: support-agent
+test_name: refund_flow
+interactions:
+  - input: "I want a refund for order ORD-001"
+    tool_calls:
+      - name: lookup_order
+        arguments:
+          order_id: ORD-001
+        result:
+          status: delivered
+          amount: 49.99
+      - name: check_refund_eligibility
+        arguments:
+          order_id: ORD-001
+        result:
+          eligible: true
+          refund_amount: 49.99
+      - name: issue_refund
+        arguments:
+          order_id: ORD-001
+          amount: 49.99
+        result:
+          refund_id: RF-001
+          status: approved
+    output: "Your refund of $49.99 has been approved... 3-5 business days."
+```
+
+Cassettes are the **baseline** you compare against. Treat them like snapshot files — commit them to version control.
+
+### Operation modes
+
+| Mode       | Description                                                                 |
+|------------|-----------------------------------------------------------------------------|
+| `record`   | Run the real agent, save all interactions to a cassette.                    |
+| `replay`   | Return recorded data from the cassette. Fail if cassette doesn't exist.     |
+| `auto`     | Replay if cassette exists, otherwise record. **Default.**                   |
+
+### How the `cassette.run()` method works
+
+```python
+result = cassette.run(agent_func, user_input, agent_name="my-agent")
+```
+
+- **Record mode**: calls `agent_func(user_input)`, records the returned `tool_calls` and `output`, stores them internally.
+- **Replay mode**: loads the cassette for this test, returns the recorded `tool_calls` and `output` without calling `agent_func`.
+- Both modes return a dict with `tool_calls`, `output`, `input`, and `_replayed` (bool).
+
+The result dict is what you pass to `expect_tools()` and `expect_output()`.
+
+---
+
+## Assertion API
+
+### `expect_tools(result).called(name).then(name)...`
+
+Fluent assertions on the **exact sequence** of tool calls.
+
+```python
+expect_tools(result).called("lookup_order") \
+                    .then("check_refund_eligibility") \
+                    .then("issue_refund")
+```
+
+Properties:
+- `.total` — total number of tool calls
+- `.remaining` — how many haven't been asserted yet
+
+Raises `AssertionError` on mismatch.
+
+### `expect_output(result).matches_intent(description)`
+
+Validates the natural-language output is non-empty and documents the expected intent.
+
+```python
+expect_output(result).matches_intent("confirms refund is approved and provides arrival time")
+```
+
+In the MVP this is a non-emptiness check. Future releases will add:
+- `.semantic_similarity(threshold=0.85)` — embedding-based similarity
+- `.judged_by("gpt-4o-mini", rubric=...)` — LLM-as-judge evaluation
+
+---
+
+## CLI Reference
+
+When ``typer`` is installed (``pip install pytest-agent-check[cli]``):
+
+```bash
+agent-eval record [<test-path>]       # Record cassettes (real agent runs)
+agent-eval run    [<test-path>]       # Replay from cassettes (offline)
+agent-eval update [<test-path>]       # Re-record (update) baselines
+```
+
+Options:
+- ``--cassette-dir`` / ``-d`` — Cassette directory (default: ``.cassettes``)
+- ``--verbose`` / ``--quiet`` — Pytest verbosity
+- ``--pytest-args`` — Extra pytest arguments (repeatable, e.g. ``-pytest -x -pytest -k test_refund``)
+
+---
+
+## Configuration
+
+All configuration is via pytest command-line flags:
+
+| Flag               | Default       | Description                                      |
+|--------------------|---------------|--------------------------------------------------|
+| `--cassette-dir`   | `.cassettes`  | Directory for cassette YAML files                |
+| `--cassette-mode`  | `auto`        | `record`, `replay`, or `auto`                    |
+| `--agent-eval-report` | off        | Print structured eval report after test run      |
+
+---
+
+## CI / GitHub Actions Integration
+
+**Record mode** is for local development. In CI, always use **replay mode** for deterministic, fast, and cost-free testing.
+
+```yaml
+# .github/workflows/ci.yml
+name: CI
+on: [pull_request]
+jobs:
+  agent-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+    steps:
+      - uses: actions/checkout@v7.0.1
+        with:
+          fetch-depth: 0  # needed for setuptools-scm
+      - uses: actions/setup-python@v7.0.0
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install pytest-agent-check[cli]
+      - run: pytest tests/ --cassette-mode=replay -v
+```
+
+To update baselines (e.g., after intentional agent changes):
+
+```bash
+pip install pytest-agent-check[cli]
+agent-eval update tests/
+git add .cassettes/
+git commit -m "chore: update agent evaluation baselines"
+```
+
+---
+
+## Project Architecture
+
+```
+pytest-agent-check/
+├── pytest_agent_eval/
+│   ├── __init__.py     # Public API exports
+│   ├── api.py          # @agent_test, expect_tools, expect_output
+│   ├── cassette.py     # CassetteManager, CassetteContext (record/replay)
+│   ├── comparator.py   # Tool-call sequence comparison (deepdiff)
+│   ├── plugin.py       # Pytest plugin (markers, options, fixtures)
+│   ├── reporter.py     # Terminal output with rich formatting
+│   ├── cli.py          # agent-eval CLI (record/run/update)
+│   └── models.py       # Data models (ToolCall, Interaction, Cassette, etc.)
+├── tests/              # Unit and integration tests
+├── examples/           # Runnable example tests
+└── pyproject.toml      # Project metadata & dependencies
+```
+
+### Roadmap
+
+| Phase | Feature                                                      |
+|-------|--------------------------------------------------------------|
+| MVP   | ✅ pytest plugin skeleton                                    |
+|       | ✅ `@agent_test` decorator & marker                          |
+|       | ✅ `expect_tools` fluent assertion                           |
+|       | ✅ `expect_output` basic intent check                        |
+|       | ✅ Cassette record/replay (YAML-based)                       |
+|       | ✅ Tool-call diff comparison (deepdiff)                      |
+|       | ✅ Terminal reporter (rich)                                  |
+|       | ✅ CLI (agent-eval record/run/update)                        |
+| Next  | 🔲 Semantic similarity (`expect_output().semantic_similarity()`) |
+|       | 🔲 LLM-as-judge evaluation (`expect_output().judged_by()`)   |
+|       | 🔲 Multi-turn interaction support                            |
+|       | 🔲 HTML / PR comment report format                            |
+|       | 🔲 GitHub Action for automated PR commenting                 |
+|       | 🔲 LangChain / OpenAI SDK adapters (auto-interception)       |
+|       | 🔲 MCP / A2A protocol support                                |
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/shenxianpeng/pytest-agent-check.git
+cd pytest-agent-check
+python -m venv venv && source venv/bin/activate
+pip install -e ".[dev]"
+pytest
+```
+
+### Running tests
+
+```bash
+pytest tests/ -v                     # unit tests only
+pytest examples/ --cassette-mode=record  # integration tests (record)
+pytest examples/ --cassette-mode=replay  # integration tests (replay)
+agent-eval run tests/                # via CLI
+```
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+---
+
+## Contributing
+
+Contributions are welcome! Please open an issue or pull request on [GitHub](https://github.com/shenxianpeng/pytest-agent-check).
+
+When adding features, include tests and update the documentation. Every PR should pass `pytest tests/ --cassette-mode=replay`.

pytest_agent_check-0.0.1.dist-info/RECORD

@@ -0,0 +1,14 @@
+pytest_agent_check-0.0.1.dist-info/licenses/LICENSE,sha256=naE4MjnMNhwEWhlsEWmTREcCAiaiPVqDRxapVJYn4rM,1070
+pytest_agent_eval/__init__.py,sha256=A0ZykDsaeKMn3kKHMa4iGdhL9hMROxvobSjEodk6c3U,867
+pytest_agent_eval/api.py,sha256=0KNfiukSls0-d50VPaQ0W42t2qp5czYURjeLTjY0h-k,6246
+pytest_agent_eval/cassette.py,sha256=YobwYPTHge-8Vofqs0eXPwnemN1MeXtgPn5z-EiWzEY,8181
+pytest_agent_eval/cli.py,sha256=2fevB7OHmJZWL4YIRI_rliV6Wr67F5oRtimJx0UYK68,4998
+pytest_agent_eval/comparator.py,sha256=d-ND6i3UVK_5N0sHD9bD22lw505PERySLAwr1YP7fkY,4032
+pytest_agent_eval/models.py,sha256=-MY8VMwPKheANAsMaRgI_SjGI8rhL5JkYUjQCJn21Oo,6241
+pytest_agent_eval/plugin.py,sha256=kyZVENitIMx9k-RGaIfp4fjsgGRX1eIgLy8pYGHZ5gc,4037
+pytest_agent_eval/reporter.py,sha256=BWMZEZ149EhpHq20XV_2VhXW0KgMYZGEoje81Re5cFQ,4787
+pytest_agent_check-0.0.1.dist-info/METADATA,sha256=ogGMscoNWfwLfZXwUE17H2FNzarXCzSuN2eRW0S7kbQ,13522
+pytest_agent_check-0.0.1.dist-info/WHEEL,sha256=K260EYznzXsJYBQGqmI8VTxEdiZYNvDZwW9cBh9-_MA,91
+pytest_agent_check-0.0.1.dist-info/entry_points.txt,sha256=LyaPJ3_3SdUTVyQelEuEo8Azg4B1KcnuS27PPXvXYm4,115
+pytest_agent_check-0.0.1.dist-info/top_level.txt,sha256=-pjdrPHR20R5QdJV1Ja-_u-xqogh0Tss5ZFJEin8Ql4,18
+pytest_agent_check-0.0.1.dist-info/RECORD,,

pytest_agent_check-0.0.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (83.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

pytest_agent_check-0.0.1.dist-info/entry_points.txt

@@ -0,0 +1,5 @@
+[console_scripts]
+agent-eval = pytest_agent_eval.cli:app
+
+[pytest11]
+pytest-agent-check = pytest_agent_eval.plugin

pytest_agent_check-0.0.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Xianpeng Shen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytest_agent_check-0.0.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+pytest_agent_eval

pytest_agent_eval/__init__.py

@@ -0,0 +1,30 @@
+"""
+pytest-agent-check — A pytest plugin for evaluating and testing AI agents.
+
+This plugin provides:
+- ``@agent_test`` decorator for marking agent evaluation tests
+- ``expect_tools`` / ``expect_output`` fluent assertion API
+- Cassette-based record/replay for deterministic, offline testing
+- Structured comparison of tool-call sequences
+- Terminal and HTML diff reporting
+- CLI tool for managing test lifecycles
+"""
+
+from __future__ import annotations
+
+from .api import agent_test, expect_output, expect_tools
+from .cassette import CassetteContext, CassetteManager
+from .models import Cassette, ComparisonResult, Interaction, ToolCall, ToolCallDiff
+
+__all__ = [
+    "agent_test",
+    "expect_tools",
+    "expect_output",
+    "ToolCall",
+    "Interaction",
+    "Cassette",
+    "ComparisonResult",
+    "ToolCallDiff",
+    "CassetteManager",
+    "CassetteContext",
+]

pytest_agent_eval/api.py

@@ -0,0 +1,188 @@
+"""
+Public API — the user-facing decorators and assertion helpers.
+
+This is what users ``from pytest_agent_eval import`` in their test files.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Callable
+
+import pytest
+
+# ──────────────────────────────────────────────────────────────
+#  agent_test decorator
+# ──────────────────────────────────────────────────────────────
+
+
+def agent_test(
+    *,
+    agent: str,
+    cassette: str | None = None,
+) -> Callable[[Callable], Callable]:
+    """
+    Mark a function as an agent evaluation test.
+
+    This is a thin wrapper around ``@pytest.mark.agent_test`` that
+    provides a cleaner user-facing syntax.
+
+    Args:
+        agent: Logical name of the agent under test.
+        cassette: (optional) Explicit cassette name.  If omitted,
+            the test function name is used.
+
+    Example::
+
+        from pytest_agent_eval import agent_test, expect_tools
+
+        @agent_test(agent="my-support-agent")
+        def test_refund_request(agent, cassette):
+            result = cassette.run(agent, "I want a refund")
+            expect_tools(result).called("lookup_order").then("issue_refund")
+
+    """
+
+    def decorator(func: Callable) -> Callable:
+        kwargs: dict[str, Any] = {"agent": agent}
+        if cassette is not None:
+            kwargs["cassette"] = cassette
+        return pytest.mark.agent_test(**kwargs)(func)
+
+    return decorator
+
+
+# ──────────────────────────────────────────────────────────────
+#  Fluent assertion helpers
+# ──────────────────────────────────────────────────────────────
+
+
+class ExpectToolsResult:
+    """
+    Fluent assertion on a sequence of tool calls.
+
+    Constructed via :func:`expect_tools` — users should not instantiate
+    this directly.
+    """
+
+    def __init__(self, result: dict[str, Any]) -> None:
+        self._tool_calls: list[dict[str, Any]] = result.get("tool_calls", [])
+        self._index: int = 0
+
+    def called(self, tool_name: str) -> ExpectToolsResult:
+        """
+        Assert that the **next** tool call has *tool_name*.
+
+        Returns ``self`` so calls can be chained with :meth:`then`.
+        """
+        if self._index >= len(self._tool_calls):
+            raise AssertionError(
+                f"Expected tool call '{tool_name}' at position "
+                f"{self._index + 1}, but only "
+                f"{len(self._tool_calls)} tool call(s) were made."
+            )
+        actual = self._tool_calls[self._index].get("name", "")
+        if actual != tool_name:
+            raise AssertionError(
+                f"Tool call mismatch at position {self._index + 1}:\n"
+                f"  Expected: '{tool_name}'\n"
+                f"  Actual:   '{actual}'"
+            )
+        self._index += 1
+        return self
+
+    def then(self, tool_name: str) -> ExpectToolsResult:
+        """Alias for :meth:`called` — enables fluent chaining."""
+        return self.called(tool_name)
+
+    @property
+    def total(self) -> int:
+        """Return the total number of tool calls made."""
+        return len(self._tool_calls)
+
+    @property
+    def remaining(self) -> int:
+        """Return how many tool calls have not been asserted yet."""
+        return len(self._tool_calls) - self._index
+
+
+class ExpectOutputResult:
+    """
+    Fluent assertion on the natural-language output of an agent.
+
+    Constructed via :func:`expect_output`.
+    """
+
+    def __init__(self, result: dict[str, Any]) -> None:
+        self._output: str = result.get("output", "")
+
+    @property
+    def text(self) -> str:
+        """The raw output text."""
+        return self._output
+
+    def matches_intent(self, description: str) -> ExpectOutputResult:
+        """
+        Assert the output is non-empty and plausibly matches an intent.
+
+        In the MVP this is a basic non-emptiness check.  Future versions
+        will use embedding similarity or an LLM-as-judge for semantic
+        equivalence.
+
+        Args:
+            description: Human-readable description of the expected intent.
+
+        """
+        if not self._output:
+            raise AssertionError(
+                "Expected non-empty agent output, but got an empty string."
+            )
+        # For MVP we accept any non-empty output as "passing".
+        # The description parameter documents intent for future semantic checks.
+        return self
+
+    def semantic_similarity(
+        self,
+        threshold: float = 0.85,
+        reference: str | None = None,
+    ) -> ExpectOutputResult:
+        """
+        Assert that the output exceeds a semantic similarity threshold.
+
+        Requires ``sentence-transformers`` (install with
+        ``pip install pytest-agent-check[semantic]``).
+
+        Args:
+            threshold: Minimum cosine similarity (0-1).
+            reference: Reference text to compare against.  If ``None``,
+                the cassette's recorded output is used.
+
+        """
+        # Stub — full implementation requires the optional dependency.
+        # For now we just accept any non-empty output.
+        if not self._output:
+            raise AssertionError(
+                "Expected non-empty output for semantic similarity check."
+            )
+        return self
+
+
+def expect_tools(result: dict[str, Any]) -> ExpectToolsResult:
+    """
+    Start fluent assertion on the tool-call sequence of *result*.
+
+    Usage::
+
+        expect_tools(result).called("lookup_order").then("issue_refund")
+    """
+    return ExpectToolsResult(result)
+
+
+def expect_output(result: dict[str, Any]) -> ExpectOutputResult:
+    """
+    Start fluent assertion on the natural-language output of *result*.
+
+    Usage::
+
+        expect_output(result).matches_intent("confirms refund and arrival time")
+    """
+    return ExpectOutputResult(result)