agent-trace-eval 0.1.0__tar.gz

agent_trace_eval-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+dist/
+build/
+.pytest_cache/
+.coverage
+htmlcov/
+.venv/
+venv/
+.DS_Store
+reports/

agent_trace_eval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yifan Cheng
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_trace_eval-0.1.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.4
+Name: agent-trace-eval
+Version: 0.1.0
+Summary: Golden trace regression evaluation for tool-using AI agents
+Project-URL: Homepage, https://github.com/yfccyf/agent-trace-eval
+Project-URL: Documentation, https://github.com/yfccyf/agent-trace-eval#readme
+Project-URL: Repository, https://github.com/yfccyf/agent-trace-eval
+Author: Yifan Cheng
+License: MIT
+License-File: LICENSE
+Keywords: ai-agents,evaluation,observability,regression-testing,tool-calls
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# agent-trace-eval
+
+[![CI](https://github.com/yfccyf/agent-trace-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/yfccyf/agent-trace-eval/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/agent-trace-eval)](https://pypi.org/project/agent-trace-eval/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Golden trace regression evaluation for tool-using AI agents.
+
+`agent-trace-eval` checks agent execution traces against declarative expectations for:
+
+- tool selection (required / forbidden tools)
+- tool-call arguments
+- tool-call ordering
+- multi-agent handoffs
+- recovery decisions (retry / fallback / escalate)
+
+It is designed as a small, employer-neutral library you can use in CI to gate agent workflow changes.
+
+## Install
+
+```bash
+pip install agent-trace-eval
+```
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Quick start
+
+Run the bundled examples:
+
+```bash
+agent-trace-eval \
+  --cases examples/cases \
+  --traces-dir examples/traces \
+  --report reports/example-report.md
+```
+
+## Trace format
+
+Traces are JSON or YAML documents with an `events` list. Supported event types include:
+
+- `tool_call`
+- `handoff`
+- `recovery_decision`
+- `final_answer`
+
+Example:
+
+```json
+{
+  "case_id": "refund_lookup",
+  "events": [
+    {
+      "type": "tool_call",
+      "name": "lookup_order",
+      "arguments": { "order_id": "12345" }
+    },
+    {
+      "type": "tool_call",
+      "name": "issue_refund",
+      "arguments": { "order_id": "12345" }
+    }
+  ]
+}
+```
+
+## Case format
+
+Cases are YAML or JSON files with `id`, `description`, `input`, and `expect` sections:
+
+```yaml
+id: refund_lookup
+description: Agent should look up an order before issuing a refund.
+expect:
+  tools:
+    required: [lookup_order, issue_refund]
+    forbidden: [delete_account]
+  ordering:
+    before:
+      - first: lookup_order
+        second: issue_refund
+  arguments:
+    issue_refund:
+      order_id: "12345"
+```
+
+## Python API
+
+```python
+from agent_trace_eval import RegressionRunner, render_markdown_report
+from agent_trace_eval.loader import load_case, load_trace
+from agent_trace_eval.result import SuiteResult
+
+case = load_case("examples/cases/refund_lookup.yaml")
+trace = load_trace("examples/traces/refund_lookup.json")
+
+runner = RegressionRunner()
+result = runner.run_case(case, trace)
+report = render_markdown_report(SuiteResult(case_results=[result]))
+print(report)
+```
+
+## Related writing
+
+This project complements a series on agent regression testing and release gates:
+
+- [Final Answers Are Not Enough (Part 1)](https://yfccyf.github.io/writing/final-answers-are-not-enough-golden-trace-regression-testing-part-1/)
+- [From Golden Traces to Release Gates (Part 2)](https://yfccyf.github.io/writing/from-golden-traces-to-release-gates-building-an-agent-regression-harness-part-2/)
+
+## Development
+
+```bash
+pytest
+```
+
+## License
+
+MIT

agent_trace_eval-0.1.0/README.md

@@ -0,0 +1,122 @@
+# agent-trace-eval
+
+[![CI](https://github.com/yfccyf/agent-trace-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/yfccyf/agent-trace-eval/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/agent-trace-eval)](https://pypi.org/project/agent-trace-eval/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Golden trace regression evaluation for tool-using AI agents.
+
+`agent-trace-eval` checks agent execution traces against declarative expectations for:
+
+- tool selection (required / forbidden tools)
+- tool-call arguments
+- tool-call ordering
+- multi-agent handoffs
+- recovery decisions (retry / fallback / escalate)
+
+It is designed as a small, employer-neutral library you can use in CI to gate agent workflow changes.
+
+## Install
+
+```bash
+pip install agent-trace-eval
+```
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Quick start
+
+Run the bundled examples:
+
+```bash
+agent-trace-eval \
+  --cases examples/cases \
+  --traces-dir examples/traces \
+  --report reports/example-report.md
+```
+
+## Trace format
+
+Traces are JSON or YAML documents with an `events` list. Supported event types include:
+
+- `tool_call`
+- `handoff`
+- `recovery_decision`
+- `final_answer`
+
+Example:
+
+```json
+{
+  "case_id": "refund_lookup",
+  "events": [
+    {
+      "type": "tool_call",
+      "name": "lookup_order",
+      "arguments": { "order_id": "12345" }
+    },
+    {
+      "type": "tool_call",
+      "name": "issue_refund",
+      "arguments": { "order_id": "12345" }
+    }
+  ]
+}
+```
+
+## Case format
+
+Cases are YAML or JSON files with `id`, `description`, `input`, and `expect` sections:
+
+```yaml
+id: refund_lookup
+description: Agent should look up an order before issuing a refund.
+expect:
+  tools:
+    required: [lookup_order, issue_refund]
+    forbidden: [delete_account]
+  ordering:
+    before:
+      - first: lookup_order
+        second: issue_refund
+  arguments:
+    issue_refund:
+      order_id: "12345"
+```
+
+## Python API
+
+```python
+from agent_trace_eval import RegressionRunner, render_markdown_report
+from agent_trace_eval.loader import load_case, load_trace
+from agent_trace_eval.result import SuiteResult
+
+case = load_case("examples/cases/refund_lookup.yaml")
+trace = load_trace("examples/traces/refund_lookup.json")
+
+runner = RegressionRunner()
+result = runner.run_case(case, trace)
+report = render_markdown_report(SuiteResult(case_results=[result]))
+print(report)
+```
+
+## Related writing
+
+This project complements a series on agent regression testing and release gates:
+
+- [Final Answers Are Not Enough (Part 1)](https://yfccyf.github.io/writing/final-answers-are-not-enough-golden-trace-regression-testing-part-1/)
+- [From Golden Traces to Release Gates (Part 2)](https://yfccyf.github.io/writing/from-golden-traces-to-release-gates-building-an-agent-regression-harness-part-2/)
+
+## Development
+
+```bash
+pytest
+```
+
+## License
+
+MIT

agent_trace_eval-0.1.0/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "agent-trace-eval"
+version = "0.1.0"
+description = "Golden trace regression evaluation for tool-using AI agents"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Yifan Cheng" }]
+keywords = ["ai-agents", "evaluation", "regression-testing", "tool-calls", "observability"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Software Development :: Testing",
+]
+dependencies = [
+  "pyyaml>=6.0",
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0",
+  "pytest-cov>=5.0",
+]
+
+[project.scripts]
+agent-trace-eval = "agent_trace_eval.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/yfccyf/agent-trace-eval"
+Documentation = "https://github.com/yfccyf/agent-trace-eval#readme"
+Repository = "https://github.com/yfccyf/agent-trace-eval"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/agent_trace_eval"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+  "src/agent_trace_eval",
+  "README.md",
+  "LICENSE",
+  "pyproject.toml",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

agent_trace_eval-0.1.0/src/agent_trace_eval/__init__.py

@@ -0,0 +1,19 @@
+"""Golden trace regression evaluation for tool-using AI agents."""
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.report import render_markdown_report
+from agent_trace_eval.result import CaseResult, MatchResult, Severity
+from agent_trace_eval.runner import RegressionRunner
+
+__all__ = [
+    "AgentTrace",
+    "CaseResult",
+    "MatchResult",
+    "RegressionRunner",
+    "Severity",
+    "EvalCase",
+    "render_markdown_report",
+]
+
+__version__ = "0.1.0"

agent_trace_eval-0.1.0/src/agent_trace_eval/case.py

@@ -0,0 +1,40 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from agent_trace_eval.result import Severity
+
+
+@dataclass
+class EvalCase:
+    id: str
+    description: str = ""
+    input: dict[str, Any] = field(default_factory=dict)
+    expect: dict[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "EvalCase":
+        case_id = data.get("id") or data.get("case_id")
+        if not case_id:
+            raise ValueError("Test case must include an 'id' or 'case_id' field.")
+        return cls(
+            id=str(case_id),
+            description=str(data.get("description", "")),
+            input=dict(data.get("input") or {}),
+            expect=dict(data.get("expect") or {}),
+        )
+
+    def severity(self, key: str, default: Severity = Severity.MEDIUM) -> Severity:
+        raw = self._lookup(self.expect, key)
+        if raw is None:
+            return default
+        return Severity(str(raw))
+
+    def _lookup(self, data: dict[str, Any], dotted_key: str) -> Any:
+        current: Any = data
+        for part in dotted_key.split("."):
+            if not isinstance(current, dict) or part not in current:
+                return None
+            current = current[part]
+        return current

agent_trace_eval-0.1.0/src/agent_trace_eval/cli.py

@@ -0,0 +1,72 @@
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from agent_trace_eval.loader import load_cases
+from agent_trace_eval.report import render_markdown_report
+from agent_trace_eval.result import SuiteResult
+from agent_trace_eval.runner import RegressionRunner
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="agent-trace-eval",
+        description="Evaluate agent execution traces against golden expectations.",
+    )
+    parser.add_argument(
+        "--cases",
+        required=True,
+        help="Path to a YAML/JSON case file or directory of case files.",
+    )
+    parser.add_argument(
+        "--traces-dir",
+        required=True,
+        help="Directory containing trace files named <case_id>.json or .yaml.",
+    )
+    parser.add_argument(
+        "--report",
+        help="Optional path to write a Markdown report.",
+    )
+    return parser
+
+
+def _collect_case_files(path: Path) -> list[Path]:
+    if path.is_file():
+        return [path]
+    return sorted(path.glob("*.yml")) + sorted(path.glob("*.yaml")) + sorted(path.glob("*.json"))
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+
+    cases_path = Path(args.cases)
+    traces_dir = Path(args.traces_dir)
+    runner = RegressionRunner(traces_dir=traces_dir)
+
+    case_files = _collect_case_files(cases_path)
+    if not case_files:
+        print(f"No case files found at {cases_path}", file=sys.stderr)
+        return 2
+
+    results = []
+    for case_file in case_files:
+        for case in load_cases(case_file):
+            results.append(runner.run_case_file(case))
+
+    suite = SuiteResult(case_results=results)
+    report = render_markdown_report(suite)
+    print(report)
+
+    if args.report:
+        report_path = Path(args.report)
+        report_path.parent.mkdir(parents=True, exist_ok=True)
+        report_path.write_text(report, encoding="utf-8")
+
+    return 0 if suite.passed else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

agent_trace_eval-0.1.0/src/agent_trace_eval/loader.py

@@ -0,0 +1,37 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+
+
+def load_trace(path: str | Path) -> AgentTrace:
+    data = _load_data(path)
+    return AgentTrace.from_dict(data)
+
+
+def load_case(path: str | Path) -> EvalCase:
+    data = _load_data(path)
+    return EvalCase.from_dict(data)
+
+
+def load_cases(path: str | Path) -> list[EvalCase]:
+    data = _load_data(path)
+    if isinstance(data, list):
+        return [EvalCase.from_dict(item) for item in data]
+    if "cases" in data:
+        return [EvalCase.from_dict(item) for item in data["cases"]]
+    return [EvalCase.from_dict(data)]
+
+
+def _load_data(path: str | Path) -> dict[str, Any] | list[dict[str, Any]]:
+    file_path = Path(path)
+    text = file_path.read_text(encoding="utf-8")
+    if file_path.suffix in {".yaml", ".yml"}:
+        return yaml.safe_load(text)
+    return json.loads(text)

agent_trace_eval-0.1.0/src/agent_trace_eval/matchers/__init__.py

@@ -0,0 +1,13 @@
+from agent_trace_eval.matchers.arguments import match_arguments
+from agent_trace_eval.matchers.handoffs import match_handoffs
+from agent_trace_eval.matchers.ordering import match_ordering
+from agent_trace_eval.matchers.recovery import match_recovery
+from agent_trace_eval.matchers.tools import match_tools
+
+__all__ = [
+    "match_arguments",
+    "match_handoffs",
+    "match_ordering",
+    "match_recovery",
+    "match_tools",
+]

agent_trace_eval-0.1.0/src/agent_trace_eval/matchers/arguments.py

@@ -0,0 +1,81 @@
+from __future__ import annotations
+
+from typing import Any
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.result import MatchResult, Severity
+
+
+def match_arguments(trace: AgentTrace, case: EvalCase) -> list[MatchResult]:
+    expect = case.expect.get("arguments", {})
+    if not expect:
+        return []
+
+    results: list[MatchResult] = []
+    for tool_name, rules in expect.items():
+        if not isinstance(rules, dict):
+            continue
+        calls = trace.find_tool_calls(tool_name)
+        if not calls:
+            results.append(
+                MatchResult(
+                    name="arguments.tool_present",
+                    passed=False,
+                    message=f"Cannot evaluate arguments for '{tool_name}' because it was not called.",
+                    severity=case.severity("severity.arguments", Severity.HIGH),
+                    details={"tool": tool_name},
+                )
+            )
+            continue
+
+        call = calls[0]
+        for arg_name, expected in rules.items():
+            if isinstance(expected, dict) and "any_of" in expected:
+                result = _match_any_of(tool_name, arg_name, call.arguments, expected["any_of"])
+            else:
+                result = _match_exact(tool_name, arg_name, call.arguments, expected)
+            result.severity = case.severity("severity.arguments", Severity.HIGH)
+            results.append(result)
+
+    return results
+
+
+def _match_exact(
+    tool_name: str,
+    arg_name: str,
+    arguments: dict[str, Any],
+    expected: Any,
+) -> MatchResult:
+    actual = arguments.get(arg_name)
+    passed = actual == expected
+    return MatchResult(
+        name="arguments.exact",
+        passed=passed,
+        message=(
+            f"Argument '{arg_name}' for '{tool_name}' matches expected value."
+            if passed
+            else f"Argument '{arg_name}' for '{tool_name}' does not match expected value."
+        ),
+        details={"tool": tool_name, "argument": arg_name, "expected": expected, "actual": actual},
+    )
+
+
+def _match_any_of(
+    tool_name: str,
+    arg_name: str,
+    arguments: dict[str, Any],
+    options: list[Any],
+) -> MatchResult:
+    actual = arguments.get(arg_name)
+    passed = actual in options
+    return MatchResult(
+        name="arguments.any_of",
+        passed=passed,
+        message=(
+            f"Argument '{arg_name}' for '{tool_name}' is one of the allowed values."
+            if passed
+            else f"Argument '{arg_name}' for '{tool_name}' is not one of the allowed values."
+        ),
+        details={"tool": tool_name, "argument": arg_name, "options": options, "actual": actual},
+    )

agent_trace_eval-0.1.0/src/agent_trace_eval/matchers/handoffs.py

@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.result import MatchResult, Severity
+
+
+def match_handoffs(trace: AgentTrace, case: EvalCase) -> list[MatchResult]:
+    expect = case.expect.get("handoffs", {})
+    if not expect:
+        return []
+
+    results: list[MatchResult] = []
+    handoffs = trace.handoffs()
+
+    required = expect.get("required", [])
+    for rule in required:
+        passed = _has_handoff(handoffs, rule)
+        results.append(
+            MatchResult(
+                name="handoffs.required",
+                passed=passed,
+                message=(
+                    "Required handoff was observed."
+                    if passed
+                    else "Required handoff was not observed."
+                ),
+                severity=case.severity("severity.handoffs", Severity.HIGH),
+                details={"rule": rule, "observed": [_handoff_summary(event) for event in handoffs]},
+            )
+        )
+
+    forbidden = expect.get("forbidden", [])
+    for rule in forbidden:
+        passed = not _has_handoff(handoffs, rule)
+        results.append(
+            MatchResult(
+                name="handoffs.forbidden",
+                passed=passed,
+                message=(
+                    "Forbidden handoff was not observed."
+                    if passed
+                    else "Forbidden handoff was observed."
+                ),
+                severity=case.severity("severity.handoffs", Severity.HIGH),
+                details={"rule": rule, "observed": [_handoff_summary(event) for event in handoffs]},
+            )
+        )
+
+    return results
+
+
+def _has_handoff(handoffs: list, rule: dict) -> bool:
+    for event in handoffs:
+        if rule.get("from") and event.from_agent != rule.get("from"):
+            continue
+        if rule.get("to") and event.to_agent != rule.get("to"):
+            continue
+        if rule.get("reason") and event.reason != rule.get("reason"):
+            continue
+        return True
+    return False
+
+
+def _handoff_summary(event) -> dict:
+    return {
+        "from": event.from_agent,
+        "to": event.to_agent,
+        "reason": event.reason,
+    }

agent_trace_eval-0.1.0/src/agent_trace_eval/matchers/ordering.py

@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.result import MatchResult, Severity
+
+
+def match_ordering(trace: AgentTrace, case: EvalCase) -> list[MatchResult]:
+    expect = case.expect.get("ordering", {})
+    if not expect:
+        return []
+
+    results: list[MatchResult] = []
+    tool_names = [name for name in trace.event_names() if name is not None]
+
+    before_rules = expect.get("before", [])
+    for rule in before_rules:
+        first = rule.get("first")
+        second = rule.get("second")
+        if not first or not second:
+            continue
+        passed = _appears_before(tool_names, first, second)
+        results.append(
+            MatchResult(
+                name="ordering.before",
+                passed=passed,
+                message=(
+                    f"'{first}' appears before '{second}'."
+                    if passed
+                    else f"'{first}' does not appear before '{second}'."
+                ),
+                severity=case.severity("severity.ordering", Severity.HIGH),
+                details={"first": first, "second": second, "observed": tool_names},
+            )
+        )
+
+    after_rules = expect.get("after", [])
+    for rule in after_rules:
+        first = rule.get("first")
+        second = rule.get("second")
+        if not first or not second:
+            continue
+        passed = _appears_after(tool_names, first, second)
+        results.append(
+            MatchResult(
+                name="ordering.after",
+                passed=passed,
+                message=(
+                    f"'{first}' appears after '{second}'."
+                    if passed
+                    else f"'{first}' does not appear after '{second}'."
+                ),
+                severity=case.severity("severity.ordering", Severity.HIGH),
+                details={"first": first, "second": second, "observed": tool_names},
+            )
+        )
+
+    return results
+
+
+def _appears_before(sequence: list[str], first: str, second: str) -> bool:
+    if first not in sequence or second not in sequence:
+        return False
+    return sequence.index(first) < sequence.index(second)
+
+
+def _appears_after(sequence: list[str], first: str, second: str) -> bool:
+    if first not in sequence or second not in sequence:
+        return False
+    return sequence.index(first) > sequence.index(second)

agent_trace_eval-0.1.0/src/agent_trace_eval/matchers/recovery.py

@@ -0,0 +1,74 @@
+from __future__ import annotations
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.result import MatchResult, Severity
+
+
+def match_recovery(trace: AgentTrace, case: EvalCase) -> list[MatchResult]:
+    expect = case.expect.get("recovery", {})
+    if not expect:
+        return []
+
+    results: list[MatchResult] = []
+    decisions = trace.recovery_decisions()
+
+    required = expect.get("required", [])
+    for rule in required:
+        passed = _has_recovery(decisions, rule)
+        results.append(
+            MatchResult(
+                name="recovery.required",
+                passed=passed,
+                message=(
+                    "Required recovery decision was observed."
+                    if passed
+                    else "Required recovery decision was not observed."
+                ),
+                severity=case.severity("severity.recovery", Severity.HIGH),
+                details={
+                    "rule": rule,
+                    "observed": [_recovery_summary(event) for event in decisions],
+                },
+            )
+        )
+
+    forbidden = expect.get("forbidden", [])
+    for rule in forbidden:
+        passed = not _has_recovery(decisions, rule)
+        results.append(
+            MatchResult(
+                name="recovery.forbidden",
+                passed=passed,
+                message=(
+                    "Forbidden recovery decision was not observed."
+                    if passed
+                    else "Forbidden recovery decision was observed."
+                ),
+                severity=case.severity("severity.recovery", Severity.HIGH),
+                details={
+                    "rule": rule,
+                    "observed": [_recovery_summary(event) for event in decisions],
+                },
+            )
+        )
+
+    return results
+
+
+def _has_recovery(decisions: list, rule: dict) -> bool:
+    for event in decisions:
+        if rule.get("decision") and event.decision != rule.get("decision"):
+            continue
+        if rule.get("error_class") and event.error_class != rule.get("error_class"):
+            continue
+        return True
+    return False
+
+
+def _recovery_summary(event) -> dict:
+    return {
+        "decision": event.decision,
+        "error_class": event.error_class,
+        "reason": event.reason,
+    }

agent_trace_eval-0.1.0/src/agent_trace_eval/matchers/tools.py

@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.result import MatchResult, Severity
+
+
+def match_tools(trace: AgentTrace, case: EvalCase) -> list[MatchResult]:
+    expect = case.expect.get("tools", {})
+    if not expect:
+        return []
+
+    results: list[MatchResult] = []
+    tool_names = [name for name in trace.event_names() if name is not None]
+
+    required = expect.get("required", [])
+    for tool_name in required:
+        passed = tool_name in tool_names
+        results.append(
+            MatchResult(
+                name="tools.required",
+                passed=passed,
+                message=(
+                    f"Required tool '{tool_name}' was called."
+                    if passed
+                    else f"Required tool '{tool_name}' was not called."
+                ),
+                severity=case.severity("severity.tools", Severity.HIGH),
+                details={"tool": tool_name, "observed": tool_names},
+            )
+        )
+
+    forbidden = expect.get("forbidden", [])
+    for tool_name in forbidden:
+        passed = tool_name not in tool_names
+        results.append(
+            MatchResult(
+                name="tools.forbidden",
+                passed=passed,
+                message=(
+                    f"Forbidden tool '{tool_name}' was not called."
+                    if passed
+                    else f"Forbidden tool '{tool_name}' was called."
+                ),
+                severity=case.severity("severity.tools", Severity.HIGH),
+                details={"tool": tool_name, "observed": tool_names},
+            )
+        )
+
+    exact_sequence = expect.get("exact_sequence")
+    if exact_sequence is not None:
+        passed = tool_names == list(exact_sequence)
+        results.append(
+            MatchResult(
+                name="tools.exact_sequence",
+                passed=passed,
+                message=(
+                    "Tool call sequence matches expected order."
+                    if passed
+                    else "Tool call sequence does not match expected order."
+                ),
+                severity=case.severity("severity.ordering", Severity.HIGH),
+                details={"expected": exact_sequence, "observed": tool_names},
+            )
+        )
+
+    return results

agent_trace_eval-0.1.0/src/agent_trace_eval/models.py

@@ -0,0 +1,112 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+
+@dataclass
+class TraceEvent:
+    type: str
+    name: str | None = None
+    arguments: dict[str, Any] = field(default_factory=dict)
+    result: dict[str, Any] = field(default_factory=dict)
+    payload: dict[str, Any] = field(default_factory=dict)
+    from_agent: str | None = None
+    to_agent: str | None = None
+    reason: str | None = None
+    decision: str | None = None
+    error_class: str | None = None
+    text: str | None = None
+    status: str | None = None
+    source: str | None = None
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "TraceEvent":
+        event_type = data.get("type", "unknown")
+        return cls(
+            type=event_type,
+            name=data.get("name"),
+            arguments=dict(data.get("arguments") or {}),
+            result=dict(data.get("result") or {}),
+            payload=dict(data.get("payload") or {}),
+            from_agent=data.get("from"),
+            to_agent=data.get("to"),
+            reason=data.get("reason"),
+            decision=data.get("decision"),
+            error_class=data.get("error_class"),
+            text=data.get("text"),
+            status=_event_status(data),
+            source=data.get("source"),
+            metadata={
+                key: value
+                for key, value in data.items()
+                if key
+                not in {
+                    "type",
+                    "name",
+                    "arguments",
+                    "result",
+                    "payload",
+                    "from",
+                    "to",
+                    "reason",
+                    "decision",
+                    "error_class",
+                    "text",
+                    "status",
+                    "source",
+                }
+            },
+        )
+
+
+@dataclass
+class AgentTrace:
+    case_id: str | None = None
+    run_id: str | None = None
+    input: str | None = None
+    events: list[TraceEvent] = field(default_factory=list)
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "AgentTrace":
+        events = [TraceEvent.from_dict(event) for event in data.get("events", [])]
+        return cls(
+            case_id=data.get("case_id"),
+            run_id=data.get("run_id"),
+            input=data.get("input"),
+            events=events,
+            metadata={
+                key: value
+                for key, value in data.items()
+                if key not in {"case_id", "run_id", "input", "events"}
+            },
+        )
+
+    def tool_calls(self) -> list[TraceEvent]:
+        return [event for event in self.events if event.type == "tool_call"]
+
+    def handoffs(self) -> list[TraceEvent]:
+        return [event for event in self.events if event.type == "handoff"]
+
+    def recovery_decisions(self) -> list[TraceEvent]:
+        return [event for event in self.events if event.type == "recovery_decision"]
+
+    def final_answers(self) -> list[TraceEvent]:
+        return [event for event in self.events if event.type == "final_answer"]
+
+    def event_names(self) -> list[str | None]:
+        return [event.name for event in self.events if event.type == "tool_call"]
+
+    def find_tool_calls(self, tool_name: str) -> list[TraceEvent]:
+        return [event for event in self.tool_calls() if event.name == tool_name]
+
+
+def _event_status(data: dict[str, Any]) -> str | None:
+    if "status" in data and data["status"] is not None:
+        return str(data["status"])
+    result = data.get("result")
+    if isinstance(result, dict) and "status" in result:
+        return str(result["status"])
+    return None

agent_trace_eval-0.1.0/src/agent_trace_eval/report.py

@@ -0,0 +1,40 @@
+from __future__ import annotations
+
+from agent_trace_eval.result import CaseResult, SuiteResult
+
+
+def render_markdown_report(suite: SuiteResult) -> str:
+    lines = [
+        "# Agent Trace Regression Report",
+        "",
+        f"- Total cases: {suite.total}",
+        f"- Passed: {suite.total - suite.failed_count}",
+        f"- Failed: {suite.failed_count}",
+        "",
+    ]
+
+    for result in suite.case_results:
+        status = "PASS" if result.passed else "FAIL"
+        lines.append(f"## {status}: {result.case_id}")
+        if result.description:
+            lines.append(result.description)
+        if result.trace_path:
+            lines.append(f"Trace: `{result.trace_path}`")
+        lines.append("")
+
+        if not result.matches:
+            lines.append("_No matchers configured._")
+            lines.append("")
+            continue
+
+        for match in result.matches:
+            mark = "x" if match.failed else " "
+            lines.append(f"- [{mark}] **{match.name}** ({match.severity.value}): {match.message}")
+        lines.append("")
+
+    return "\n".join(lines).rstrip() + "\n"
+
+
+def render_case_summary(result: CaseResult) -> str:
+    status = "PASS" if result.passed else "FAIL"
+    return f"{status} {result.case_id}: {len(result.failures)} failure(s)"

agent_trace_eval-0.1.0/src/agent_trace_eval/result.py

@@ -0,0 +1,62 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any
+
+
+class Severity(str, Enum):
+    LOW = "low"
+    MEDIUM = "medium"
+    HIGH = "high"
+    CRITICAL = "critical"
+
+
+@dataclass
+class MatchResult:
+    name: str
+    passed: bool
+    message: str
+    severity: Severity = Severity.MEDIUM
+    details: dict[str, Any] = field(default_factory=dict)
+
+    @property
+    def failed(self) -> bool:
+        return not self.passed
+
+
+@dataclass
+class CaseResult:
+    case_id: str
+    description: str
+    passed: bool
+    matches: list[MatchResult] = field(default_factory=list)
+    trace_path: str | None = None
+
+    @property
+    def failures(self) -> list[MatchResult]:
+        return [match for match in self.matches if match.failed]
+
+    @property
+    def highest_severity(self) -> Severity | None:
+        if not self.failures:
+            return None
+        order = [Severity.LOW, Severity.MEDIUM, Severity.HIGH, Severity.CRITICAL]
+        return max(self.failures, key=lambda match: order.index(match.severity)).severity
+
+
+@dataclass
+class SuiteResult:
+    case_results: list[CaseResult] = field(default_factory=list)
+
+    @property
+    def passed(self) -> bool:
+        return all(result.passed for result in self.case_results)
+
+    @property
+    def total(self) -> int:
+        return len(self.case_results)
+
+    @property
+    def failed_count(self) -> int:
+        return sum(1 for result in self.case_results if not result.passed)

agent_trace_eval-0.1.0/src/agent_trace_eval/runner.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+from agent_trace_eval.case import EvalCase
+from agent_trace_eval.loader import load_case, load_trace
+from agent_trace_eval.matchers import (
+    match_arguments,
+    match_handoffs,
+    match_ordering,
+    match_recovery,
+    match_tools,
+)
+from agent_trace_eval.models import AgentTrace
+from agent_trace_eval.result import CaseResult, SuiteResult
+
+
+class RegressionRunner:
+    def __init__(self, traces_dir: str | Path | None = None) -> None:
+        self.traces_dir = Path(traces_dir) if traces_dir else None
+
+    def run_case(self, case: EvalCase, trace: AgentTrace, trace_path: str | None = None) -> CaseResult:
+        matches = []
+        matches.extend(match_tools(trace, case))
+        matches.extend(match_arguments(trace, case))
+        matches.extend(match_ordering(trace, case))
+        matches.extend(match_handoffs(trace, case))
+        matches.extend(match_recovery(trace, case))
+        passed = all(match.passed for match in matches)
+        return CaseResult(
+            case_id=case.id,
+            description=case.description,
+            passed=passed,
+            matches=matches,
+            trace_path=trace_path,
+        )
+
+    def run_case_file(
+        self,
+        case_or_path: EvalCase | str | Path,
+        trace_path: str | Path | None = None,
+    ) -> CaseResult:
+        if isinstance(case_or_path, EvalCase):
+            case = case_or_path
+        else:
+            case = load_case(case_or_path)
+        resolved_trace = trace_path or self._default_trace_path(case.id)
+        trace = load_trace(resolved_trace)
+        return self.run_case(case, trace, str(resolved_trace))
+
+    def run_suite(self, cases: list[EvalCase], traces: dict[str, AgentTrace]) -> SuiteResult:
+        results = []
+        for case in cases:
+            if case.id not in traces:
+                results.append(
+                    CaseResult(
+                        case_id=case.id,
+                        description=case.description,
+                        passed=False,
+                        matches=[],
+                        trace_path=None,
+                    )
+                )
+                continue
+            results.append(self.run_case(case, traces[case.id]))
+        return SuiteResult(case_results=results)
+
+    def _default_trace_path(self, case_id: str) -> Path:
+        if not self.traces_dir:
+            raise ValueError("trace_path is required when traces_dir is not configured.")
+        for suffix in (".json", ".yaml", ".yml"):
+            candidate = self.traces_dir / f"{case_id}{suffix}"
+            if candidate.exists():
+                return candidate
+        raise FileNotFoundError(f"No trace file found for case '{case_id}' in {self.traces_dir}")