understudy 0.1.0__tar.gz → 0.4.0__tar.gz

{understudy-0.1.0 → understudy-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: understudy
-Version: 0.1.0
+Version: 0.4.0
 Summary: Simulation and trace-based evaluation for agentic systems
 Keywords: agent,evaluation,simulation,testing,llm
 Author: Gaurav Sood
@@ -14,20 +14,27 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Software Development :: Testing
 Requires-Dist: pydantic>=2.0
 Requires-Dist: pyyaml>=6.0
+Requires-Dist: litellm>=1.50
 Requires-Dist: google-adk>=1.0 ; extra == 'adk'
-Requires-Dist: understudy[adk,judges,http,reports] ; extra == 'all'
+Requires-Dist: python-dotenv>=1.0 ; extra == 'adk'
+Requires-Dist: understudy[adk,http,langgraph,server,reports] ; extra == 'all'
 Requires-Dist: understudy[all,docs] ; extra == 'dev'
 Requires-Dist: pytest>=8.0 ; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23 ; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
 Requires-Dist: ruff>=0.4 ; extra == 'dev'
 Requires-Dist: sphinx>=7.0 ; extra == 'docs'
 Requires-Dist: furo>=2024.0 ; extra == 'docs'
 Requires-Dist: myst-parser>=3.0 ; extra == 'docs'
 Requires-Dist: sphinx-autodoc-typehints>=2.0 ; extra == 'docs'
 Requires-Dist: httpx>=0.27 ; extra == 'http'
-Requires-Dist: litellm>=1.50 ; extra == 'judges'
+Requires-Dist: langgraph>=0.2 ; extra == 'langgraph'
+Requires-Dist: langchain-core>=0.3 ; extra == 'langgraph'
+Requires-Dist: langchain-openai>=0.3 ; extra == 'langgraph'
 Requires-Dist: jinja2>=3.0 ; extra == 'reports'
 Requires-Dist: click>=8.0 ; extra == 'reports'
+Requires-Dist: fastapi>=0.115 ; extra == 'server'
+Requires-Dist: uvicorn>=0.32 ; extra == 'server'
 Requires-Python: >=3.12
 Project-URL: Homepage, https://github.com/gojiplus/understudy
 Project-URL: Documentation, https://gojiplus.github.io/understudy
@@ -38,17 +45,37 @@ Provides-Extra: all
 Provides-Extra: dev
 Provides-Extra: docs
 Provides-Extra: http
-Provides-Extra: judges
+Provides-Extra: langgraph
 Provides-Extra: reports
+Provides-Extra: server
 Description-Content-Type: text/markdown
 
-# understudy
+## understudy: Scenario Testing for AI Agents
 
 [![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/understudy?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/project/understudy)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![Documentation](https://github.com/gojiplus/understudy/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/understudy/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Test your AI agents with simulated users.
+
+Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
+
+## How It Works
+
+Testing with understudy is **4 steps**:
+
+1. **Wrap your agent** — Adapt your agent (ADK, LangGraph, HTTP) to understudy's interface
+2. **Mock your tools** — Register handlers that return test data instead of calling real services
+3. **Write scenes** — YAML files defining what the simulated user wants and what you expect
+4. **Run and assert** — Execute simulations, check traces, generate reports
+
+The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).
+
+**See real examples:**
+- [Example scene](https://github.com/gojiplus/understudy/blob/main/example/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
+- [ADK test file](https://github.com/gojiplus/understudy/blob/main/example/adk/test_returns.py) — pytest assertions against traces
+- [LangGraph test file](https://github.com/gojiplus/understudy/blob/main/example/langgraph/test_returns.py) — same tests, different framework
 
 ## Installation
 
@@ -106,21 +133,21 @@ expectations:
   required_tools:
     - lookup_order
     - create_return
-  allowed_terminal_states:
-    - return_created
+  forbidden_tools:
+    - issue_refund
 ```
 
 ### 4. Run simulation
 
 ```python
-from understudy import Scene, run, check
+from understudy import Scene, run
 
 scene = Scene.from_file("scenes/return_backpack.yaml")
 trace = run(app, scene, mocks=mocks)
 
 assert trace.called("lookup_order")
 assert trace.called("create_return")
-assert trace.terminal_state == "return_created"
+assert not trace.called("issue_refund")
 ```
 
 Or with pytest (define `app` and `mocks` fixtures in conftest.py):
@@ -129,27 +156,109 @@ Or with pytest (define `app` and `mocks` fixtures in conftest.py):
 pytest test_returns.py -v
 ```
 
-## CLI Commands
+## Suites and Batch Runs
+
+Run multiple scenes with multiple simulations per scene:
+
+```python
+from understudy import Suite, RunStorage
+
+suite = Suite.from_directory("scenes/")
+storage = RunStorage()
+
+# Run each scene 3 times and tag for comparison
+results = suite.run(
+    app,
+    mocks=mocks,
+    storage=storage,
+    n_sims=3,
+    tags={"version": "v1"},
+)
+print(f"{results.pass_count}/{len(results.results)} passed")
+```
+
+## Simulation and Evaluation
 
-After running simulations, use the CLI to inspect results:
+Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:
+
+### Combined (most common)
 
 ```bash
-# List all saved runs
-understudy list
+understudy run \
+  --app mymodule:agent_app \
+  --scene ./scenes/ \
+  --n-sims 3 \
+  --junit results.xml
+```
 
-# Show aggregate metrics (pass rate, avg turns, tool usage, terminal states)
-understudy summary
+### Separate workflows
+
+Generate traces only:
+
+```bash
+understudy simulate \
+  --app mymodule:agent_app \
+  --scenes ./scenes/ \
+  --output ./traces/ \
+  --n-sims 3
+```
+
+Evaluate existing traces:
+
+```bash
+understudy evaluate \
+  --traces ./traces/ \
+  --output ./results/ \
+  --junit results.xml
+```
+
+Python API:
+
+```python
+from understudy import simulate_batch, evaluate_batch
+
+# Generate traces
+traces = simulate_batch(
+    app=agent_app,
+    scenes="./scenes/",
+    n_sims=3,
+    output="./traces/",
+)
+
+# Evaluate later
+results = evaluate_batch(
+    traces="./traces/",
+    output="./results/",
+)
+```
+
+## CLI Commands
+
+```bash
+# Run simulations
+understudy run --app mymodule:app --scene ./scenes/
+understudy simulate --app mymodule:app --scenes ./scenes/
+understudy evaluate --traces ./traces/
 
-# Show details for a specific run
+# View results
+understudy list
 understudy show <run_id>
+understudy summary
 
-# Generate static HTML report
-understudy report --output report.html
+# Compare runs by tag
+understudy compare --tag version --before v1 --after v2
 
-# Start interactive report browser
+# Generate reports
+understudy report -o report.html
+understudy compare --tag version --before v1 --after v2 --html comparison.html
+
+# Interactive browser
 understudy serve --port 8080
 
-# Delete runs
+# HTTP simulator server (for browser/UI testing)
+understudy serve-api --port 8000
+
+# Cleanup
 understudy delete <run_id>
 understudy clear
 ```
@@ -185,11 +294,10 @@ from understudy.judges import (
 ## Report Contents
 
 The `understudy summary` command shows:
-- **Pass rate** - percentage of scenes that passed all expectations
-- **Avg turns** - average conversation length
-- **Tool usage** - distribution of tool calls across runs
-- **Terminal states** - breakdown of how conversations ended
-- **Agents** - which agents were invoked
+- **Pass rate** — percentage of scenes that passed all expectations
+- **Avg turns** — average conversation length
+- **Tool usage** — distribution of tool calls across runs
+- **Agents** — which agents were invoked
 
 The HTML report (`understudy report`) includes:
 - All metrics above
@@ -204,6 +312,7 @@ See the [full documentation](https://gojiplus.github.io/understudy) for:
 - [Installation guide](https://gojiplus.github.io/understudy/installation.html)
 - [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
 - [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)
 - [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
 - [API reference](https://gojiplus.github.io/understudy/api/index.html)
 

understudy-0.4.0/README.md

@@ -0,0 +1,269 @@
+## understudy: Scenario Testing for AI Agents
+
+[![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/understudy?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/project/understudy)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![Documentation](https://github.com/gojiplus/understudy/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/understudy/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+
+Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
+
+## How It Works
+
+Testing with understudy is **4 steps**:
+
+1. **Wrap your agent** — Adapt your agent (ADK, LangGraph, HTTP) to understudy's interface
+2. **Mock your tools** — Register handlers that return test data instead of calling real services
+3. **Write scenes** — YAML files defining what the simulated user wants and what you expect
+4. **Run and assert** — Execute simulations, check traces, generate reports
+
+The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).
+
+**See real examples:**
+- [Example scene](https://github.com/gojiplus/understudy/blob/main/example/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
+- [ADK test file](https://github.com/gojiplus/understudy/blob/main/example/adk/test_returns.py) — pytest assertions against traces
+- [LangGraph test file](https://github.com/gojiplus/understudy/blob/main/example/langgraph/test_returns.py) — same tests, different framework
+
+## Installation
+
+```bash
+pip install understudy[all]
+```
+
+## Quick Start
+
+### 1. Wrap your agent
+
+```python
+from understudy.adk import ADKApp
+from my_agent import agent
+
+app = ADKApp(agent=agent)
+```
+
+### 2. Mock your tools
+
+Your agent has tools that call external services. Mock them for testing:
+
+```python
+from understudy.mocks import MockToolkit
+
+mocks = MockToolkit()
+
+@mocks.handle("lookup_order")
+def lookup_order(order_id: str) -> dict:
+    return {"order_id": order_id, "items": [...], "status": "delivered"}
+
+@mocks.handle("create_return")
+def create_return(order_id: str, item_sku: str, reason: str) -> dict:
+    return {"return_id": "RET-001", "status": "created"}
+```
+
+### 3. Write a scene
+
+Create `scenes/return_backpack.yaml`:
+
+```yaml
+id: return_eligible_backpack
+description: Customer wants to return a backpack
+
+starting_prompt: "I'd like to return an item please."
+conversation_plan: |
+  Goal: Return the hiking backpack from order ORD-10031.
+  - Provide order ID when asked
+  - Return reason: too small
+
+persona: cooperative
+max_turns: 15
+
+expectations:
+  required_tools:
+    - lookup_order
+    - create_return
+  forbidden_tools:
+    - issue_refund
+```
+
+### 4. Run simulation
+
+```python
+from understudy import Scene, run
+
+scene = Scene.from_file("scenes/return_backpack.yaml")
+trace = run(app, scene, mocks=mocks)
+
+assert trace.called("lookup_order")
+assert trace.called("create_return")
+assert not trace.called("issue_refund")
+```
+
+Or with pytest (define `app` and `mocks` fixtures in conftest.py):
+
+```bash
+pytest test_returns.py -v
+```
+
+## Suites and Batch Runs
+
+Run multiple scenes with multiple simulations per scene:
+
+```python
+from understudy import Suite, RunStorage
+
+suite = Suite.from_directory("scenes/")
+storage = RunStorage()
+
+# Run each scene 3 times and tag for comparison
+results = suite.run(
+    app,
+    mocks=mocks,
+    storage=storage,
+    n_sims=3,
+    tags={"version": "v1"},
+)
+print(f"{results.pass_count}/{len(results.results)} passed")
+```
+
+## Simulation and Evaluation
+
+Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:
+
+### Combined (most common)
+
+```bash
+understudy run \
+  --app mymodule:agent_app \
+  --scene ./scenes/ \
+  --n-sims 3 \
+  --junit results.xml
+```
+
+### Separate workflows
+
+Generate traces only:
+
+```bash
+understudy simulate \
+  --app mymodule:agent_app \
+  --scenes ./scenes/ \
+  --output ./traces/ \
+  --n-sims 3
+```
+
+Evaluate existing traces:
+
+```bash
+understudy evaluate \
+  --traces ./traces/ \
+  --output ./results/ \
+  --junit results.xml
+```
+
+Python API:
+
+```python
+from understudy import simulate_batch, evaluate_batch
+
+# Generate traces
+traces = simulate_batch(
+    app=agent_app,
+    scenes="./scenes/",
+    n_sims=3,
+    output="./traces/",
+)
+
+# Evaluate later
+results = evaluate_batch(
+    traces="./traces/",
+    output="./results/",
+)
+```
+
+## CLI Commands
+
+```bash
+# Run simulations
+understudy run --app mymodule:app --scene ./scenes/
+understudy simulate --app mymodule:app --scenes ./scenes/
+understudy evaluate --traces ./traces/
+
+# View results
+understudy list
+understudy show <run_id>
+understudy summary
+
+# Compare runs by tag
+understudy compare --tag version --before v1 --after v2
+
+# Generate reports
+understudy report -o report.html
+understudy compare --tag version --before v1 --after v2 --html comparison.html
+
+# Interactive browser
+understudy serve --port 8080
+
+# HTTP simulator server (for browser/UI testing)
+understudy serve-api --port 8000
+
+# Cleanup
+understudy delete <run_id>
+understudy clear
+```
+
+## LLM Judges
+
+For qualities that can't be checked deterministically:
+
+```python
+from understudy.judges import Judge
+
+empathy_judge = Judge(
+    rubric="The agent acknowledged frustration and was empathetic while enforcing policy.",
+    samples=5,
+)
+
+result = empathy_judge.evaluate(trace)
+assert result.score == 1
+```
+
+Built-in rubrics:
+
+```python
+from understudy.judges import (
+    TOOL_USAGE_CORRECTNESS,
+    POLICY_COMPLIANCE,
+    TONE_EMPATHY,
+    ADVERSARIAL_ROBUSTNESS,
+    TASK_COMPLETION,
+)
+```
+
+## Report Contents
+
+The `understudy summary` command shows:
+- **Pass rate** — percentage of scenes that passed all expectations
+- **Avg turns** — average conversation length
+- **Tool usage** — distribution of tool calls across runs
+- **Agents** — which agents were invoked
+
+The HTML report (`understudy report`) includes:
+- All metrics above
+- Full conversation transcripts
+- Tool call details with arguments
+- Expectation check results
+- Judge evaluation results (when used)
+
+## Documentation
+
+See the [full documentation](https://gojiplus.github.io/understudy) for:
+- [Installation guide](https://gojiplus.github.io/understudy/installation.html)
+- [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
+- [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)
+- [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
+- [API reference](https://gojiplus.github.io/understudy/api/index.html)
+
+## License
+
+MIT

{understudy-0.1.0 → understudy-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "understudy"
-version = "0.1.0"
+version = "0.4.0"
 description = "Simulation and trace-based evaluation for agentic systems"
 readme = "README.md"
 license = "MIT"
@@ -25,12 +25,14 @@ classifiers = [
 dependencies = [
     "pydantic>=2.0",
     "pyyaml>=6.0",
+    "litellm>=1.50",
 ]
 
 [project.optional-dependencies]
-adk = ["google-adk>=1.0"]
-judges = ["litellm>=1.50"]
+adk = ["google-adk>=1.0", "python-dotenv>=1.0"]
 http = ["httpx>=0.27"]
+langgraph = ["langgraph>=0.2", "langchain-core>=0.3", "langchain-openai>=0.3"]
+server = ["fastapi>=0.115", "uvicorn>=0.32"]
 reports = ["jinja2>=3.0", "click>=8.0"]
 docs = [
     "sphinx>=7.0",
@@ -38,11 +40,12 @@ docs = [
     "myst-parser>=3.0",
     "sphinx-autodoc-typehints>=2.0",
 ]
-all = ["understudy[adk,judges,http,reports]"]
+all = ["understudy[adk,http,langgraph,server,reports]"]
 dev = [
     "understudy[all,docs]",
     "pytest>=8.0",
     "pytest-asyncio>=0.23",
+    "pytest-cov>=4.0",
     "ruff>=0.4",
 ]
 

{understudy-0.1.0 → understudy-0.4.0}/src/understudy/__init__.py

@@ -5,8 +5,12 @@ You write scenes, run rehearsals, and check the performance —
 not by reading the script, but by inspecting what actually happened.
 """
 
-from .check import CheckItem, CheckResult, check
-from .judges import Judge, JudgeResult
+from importlib.metadata import version
+
+from .check import CheckItem, CheckResult, EvaluationResult, check, evaluate, evaluate_batch
+from .compare import ComparisonResult, SceneComparison, compare_runs
+from .judges import FailureAnalysis, FailureAnalyzer, Judge, JudgeResult
+from .metrics import MetricRegistry, MetricResult
 from .mocks import MockToolkit, ToolError
 from .models import Expectations, Persona, PersonaPreset, Scene
 from .prompts import (
@@ -18,12 +22,13 @@ from .prompts import (
     TONE_EMPATHY,
     TOOL_USAGE_CORRECTNESS,
 )
-from .runner import AgentApp, AgentResponse, run
-from .storage import RunStorage
+from .runner import AgentApp, AgentResponse, LiteLLMBackend, run, simulate, simulate_batch
+from .simulator import Simulator, SimulatorBackend
+from .storage import EvaluationStorage, RunStorage, TraceStorage
 from .suite import SceneResult, Suite, SuiteResults
-from .trace import AgentTransfer, ToolCall, Trace, Turn
+from .trace import AgentTransfer, StateSnapshot, ToolCall, Trace, TraceMetrics, Turn, TurnMetrics
 
-__version__ = "0.1.0"
+__version__ = version("understudy")
 
 __all__ = [
     # models
@@ -36,26 +41,49 @@ __all__ = [
     "Turn",
     "ToolCall",
     "AgentTransfer",
-    # runner
+    "TraceMetrics",
+    "TurnMetrics",
+    "StateSnapshot",
+    # metrics
+    "MetricRegistry",
+    "MetricResult",
+    # runner / simulate
     "run",
+    "simulate",
+    "simulate_batch",
     "AgentApp",
     "AgentResponse",
-    # check
+    "LiteLLMBackend",
+    # check / evaluate
     "check",
+    "evaluate",
+    "evaluate_batch",
     "CheckResult",
     "CheckItem",
+    "EvaluationResult",
     # suite
     "Suite",
     "SuiteResults",
     "SceneResult",
     # storage
     "RunStorage",
+    "TraceStorage",
+    "EvaluationStorage",
+    # compare
+    "compare_runs",
+    "ComparisonResult",
+    "SceneComparison",
     # judges
     "Judge",
     "JudgeResult",
+    "FailureAnalyzer",
+    "FailureAnalysis",
     # mocks
     "MockToolkit",
     "ToolError",
+    # simulator
+    "Simulator",
+    "SimulatorBackend",
     # rubrics
     "TOOL_USAGE_CORRECTNESS",
     "POLICY_COMPLIANCE",