agent-trajectory-diff 0.1.0__py3-none-any.whl

agent_trajectory_diff-0.1.0.dist-info/METADATA

@@ -0,0 +1,112 @@
+Metadata-Version: 2.5
+Name: agent-trajectory-diff
+Version: 0.1.0
+Summary: A developer-first package to evaluate and find regressions in agent trajectories
+Project-URL: Homepage, https://github.com/lostmartian/agentdiff
+Project-URL: Repository, https://github.com/lostmartian/agentdiff
+Author-email: Sahil Gangurde <sahilgangurde08@gmail.com>
+License: Apache-2.0 OR MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Requires-Dist: networkx>=3.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.9
+Description-Content-Type: text/markdown
+
+# AgentDiff
+
+[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE)
+[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)
+
+**AgentDiff** is a developer-first Python library and CLI designed to solve the hardest problem in agent engineering: **regression testing multi-turn, tool-using AI agents by comparing execution paths (trajectories) head-to-head.**
+
+## What AgentDiff Is
+
+* **A Trajectory Diff Engine:** Compares Run A (Baseline) against Run B (Candidate) across their execution Directed Acyclic Graphs (DAGs).
+* **A Local-First CI/CD Gate:** Runs locally in your terminal or inside `pytest` and GitHub Actions, raising errors or exit codes on regression violations.
+* **A Universal Comparator:** Ingests telemetry run files from **DeepEval**, OpenInference/OTel, Langfuse, or raw/custom JSON.
+
+## Installation
+
+Install via pip:
+```bash
+pip install agentdiff
+```
+
+Or using `uv`:
+```bash
+uv add agentdiff
+```
+
+## Quickstart
+
+### 1. CLI Usage
+
+Compare two trajectory JSON traces from your terminal:
+```bash
+agentdiff diff baseline_run.json candidate_run.json --fail-on-regression --max-divergence 0.25
+```
+
+Options:
+- `--adapter`: Telemetry parser to use (`auto`, `generic`, `deepeval`, `openinference`, `langfuse`).
+- `--format`: Format for the output (`terminal`, `json`, `markdown`).
+- `--fail-on-regression`: Return exit code `1` if thresholds are violated.
+- `--max-loops`: Maximum loops allowed.
+- `--max-divergence`: Maximum Trajectory Divergence Index (TDI) allowed.
+- `--max-cost-delta`: Maximum cost increase percentage allowed.
+
+### 2. Python SDK & Pytest Integration
+
+Catch agent loop regressions or token cost spikes in your test suites:
+
+```python
+import pytest
+from agentdiff import load_trace, compare
+from agentdiff.testing import assert_no_regressions
+
+def test_agent_refactor_efficiency():
+    # Load traces from disk (auto-detects formats like DeepEval)
+    baseline = load_trace("tests/traces/baseline.json")
+    candidate = load_trace("tests/traces/candidate.json")
+
+    # Run the comparison
+    report = compare(baseline, candidate)
+
+    # Expressive assertion helper that raises detailed error messages on regression
+    assert_no_regressions(
+        report,
+        max_divergence=0.25,        # TDI threshold [0.0 - 1.0]
+        max_cost_increase_pct=5.0,  # Max cost increase allowed
+        allow_loops=False,           # Reject if tool loops are detected
+        max_wasted_effort=0.10      # Max Wasted Effort Index (WEI) allowed
+    )
+```
+
+## Core Metrics
+
+| Metric | Target / Range | Algorithmic Definition |
+| --- | --- | --- |
+| **Trajectory Divergence Index (TDI)** | `0.0` (Identical) to `1.0` (Divergent) | $$1.0 - \frac{2 \times \vert{}\text{LCS}(\text{Steps}_A, \text{Steps}_B)\vert{}}{\vert{}\text{Steps}_A\vert{} + \vert{}\text{Steps}_B\vert{}}$$ |
+| **Wasted Effort Index (WEI)** | `0.0` (Optimal) to `1.0` (Total Waste) | $$\frac{\text{Count}(\text{Steps with status} \in \{\text{ERROR, RETRY, ABANDONED}\})}{\text{Total Execution Steps}}$$ |
+| **Loop Buster Index (LBI)** | Integer ($\ge 0$) | Detects consecutive repeating sequences of tools with stagnant state changes. |
+| **Resource Deltas ($\Delta\text{Res}$)** | Percentage ($\pm\%$) | Standard deltas for $\Delta\text{Tokens}$, $\Delta\text{Cost}$, and $\Delta\text{Latency}$. |
+
+## Development & Operations
+
+This project utilizes `uv` to manage environments and dependencies. Automation tasks are defined in the **[`Makefile`](file:///Users/lostmartian/Desktop/interview/agentdiff/Makefile)**:
+
+- `make lint` / `make format`: Run Ruff linter checks and formatter.
+- `make test`: Run pytest suite (including style & formatting assertions).
+- `make build`: Package the library into source and wheel distributions in `dist/`.
+- `make website-dev`: Start the Next.js landing and documentation site local server.
+- `make website-build`: Build the Next.js static output in `website/out/`.
+
+### Repository Layout
+- `src/`: Python source code package modules.
+- `tests/`: Quality assurance unit tests.
+- `website/`: Next.js web application and documentation pages.

agent_trajectory_diff-0.1.0.dist-info/RECORD

@@ -0,0 +1,30 @@
+agentdiff/__init__.py,sha256=0d3OUOWD9xwXV-k5cwHZLxp72AE1We5L0lUwnNy8WV0,806
+agentdiff/__main__.py,sha256=wERtgHjw9RwcjnkFniQX3n3fDaxvB3pkIvQBaWLQW_Y,70
+agentdiff/cli.py,sha256=qXZwAG7n2pWXRsGYTGjL38mHTJMHb9OaiQkHcce5-Rw,3943
+agentdiff/loader.py,sha256=sP2dEhSXYCxyYPCK4_Sj3t3AIG9bezZ22T4PWhvooHw,2397
+agentdiff/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+agentdiff/adapters/__init__.py,sha256=wMgseGPcrqZQiHDCkNW4YTTmIOhKNr0mTeZGn475C3E,410
+agentdiff/adapters/base.py,sha256=mUCJsPvWqahLBVcoi2EyRXoJBfnjwF_ViHr0pBeqX3c,592
+agentdiff/adapters/deepeval.py,sha256=klH0dEZPLOsObeRe0gR1R_Oq9_wB6fbApQzY0qxYFQM,4955
+agentdiff/adapters/generic.py,sha256=jGtOoSmriu2O-j8Xp5IcsQv7X4Hx-o-zynPykh1Pb50,360
+agentdiff/adapters/langfuse.py,sha256=a4JQkvZH3hASq8MS5piH0XTs5-yTnzRaslwOhv2XGQc,5499
+agentdiff/adapters/openinference.py,sha256=kJmh7Bv0fniFlOFyRbD-vCihTjuh7MHGFX_HfiQkgP0,7640
+agentdiff/engine/__init__.py,sha256=XekhvPZ2egXkMiflC3ksiW-fVQHvGs2mzUnlR_m2rTs,419
+agentdiff/engine/aligner.py,sha256=E21gUwWfeY4VWzDFrU-6BCDc3nNYjMnsDCOJDg2qLAg,4841
+agentdiff/engine/comparator.py,sha256=qd5aIYYgJueyu7fQT3dQF3taJvacjElCBoxoh2wYJuA,2211
+agentdiff/engine/loop_detector.py,sha256=YE1mglbvTDSsehQn3D_KIjVvjBTC8mWDVsPGz1OcDkE,3641
+agentdiff/engine/metrics.py,sha256=vnnkbjbZThu541a_od8z3BjeWI9txt6jTxrMWTb2iTU,1195
+agentdiff/models/__init__.py,sha256=y6BF8dYbQegRGThiXmKYtK_dHx55uMUG1ZYw7A-r4CA,355
+agentdiff/models/report.py,sha256=0xdbV7ysf6dYjlclGghwzCbs6nPeAE7IKTfx1w2Xe18,2725
+agentdiff/models/step.py,sha256=_A7HC7kZFOk4oXiyWuy2nJ0EBdTKNtHhqicX4ag7e5Y,1018
+agentdiff/models/trace.py,sha256=vmUKe0CZCCPucIrS_jbWofFaLyZ4mjaizvI7HNaQm00,1027
+agentdiff/reporters/__init__.py,sha256=YwtvOQc6Ve5h4bt5tF9UILtzXyU2vRqWcO8eKwC7VrY,173
+agentdiff/reporters/markdown.py,sha256=PFxH5vvpQ5pYGlYUo6m0qxaDL9H2S5xD2IP0EZ_rpHA,3069
+agentdiff/reporters/terminal.py,sha256=_oAXSpfJVjRPXxrUGTZGQLbqsPQqn3Ggho5eajSfkXw,3970
+agentdiff/testing/__init__.py,sha256=6p1gTQZQ5FRms1xaxz0p_1ux_Mh1g6xOHSloYecvIno,107
+agentdiff/testing/assertions.py,sha256=VU18k3tFsv-MDL-6flAeUXjXJv8cddPuMf9pm9NCmLI,1683
+agent_trajectory_diff-0.1.0.dist-info/METADATA,sha256=SsnSoI9ntP8fu1K1Oy9tOBJ7h-0PPQvD-CeXXDpLng4,4903
+agent_trajectory_diff-0.1.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+agent_trajectory_diff-0.1.0.dist-info/entry_points.txt,sha256=luBTTNObngXBLmRV9tAIh_60187ZdDQzJoafiGdxauw,49
+agent_trajectory_diff-0.1.0.dist-info/licenses/LICENSE,sha256=12M7e1w-moSeyf8kDiSSOpywWMfmjsZGqnmfE8tvguk,32488
+agent_trajectory_diff-0.1.0.dist-info/RECORD,,

agent_trajectory_diff-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

agent_trajectory_diff-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+agentdiff = agentdiff.cli:main