understudy 0.3.0__tar.gz → 0.4.0__tar.gz

{understudy-0.3.0 → understudy-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: understudy
-Version: 0.3.0
+Version: 0.4.0
 Summary: Simulation and trace-based evaluation for agentic systems
 Keywords: agent,evaluation,simulation,testing,llm
 Author: Gaurav Sood
@@ -17,7 +17,7 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: litellm>=1.50
 Requires-Dist: google-adk>=1.0 ; extra == 'adk'
 Requires-Dist: python-dotenv>=1.0 ; extra == 'adk'
-Requires-Dist: understudy[adk,http,reports] ; extra == 'all'
+Requires-Dist: understudy[adk,http,langgraph,server,reports] ; extra == 'all'
 Requires-Dist: understudy[all,docs] ; extra == 'dev'
 Requires-Dist: pytest>=8.0 ; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23 ; extra == 'dev'
@@ -28,8 +28,13 @@ Requires-Dist: furo>=2024.0 ; extra == 'docs'
 Requires-Dist: myst-parser>=3.0 ; extra == 'docs'
 Requires-Dist: sphinx-autodoc-typehints>=2.0 ; extra == 'docs'
 Requires-Dist: httpx>=0.27 ; extra == 'http'
+Requires-Dist: langgraph>=0.2 ; extra == 'langgraph'
+Requires-Dist: langchain-core>=0.3 ; extra == 'langgraph'
+Requires-Dist: langchain-openai>=0.3 ; extra == 'langgraph'
 Requires-Dist: jinja2>=3.0 ; extra == 'reports'
 Requires-Dist: click>=8.0 ; extra == 'reports'
+Requires-Dist: fastapi>=0.115 ; extra == 'server'
+Requires-Dist: uvicorn>=0.32 ; extra == 'server'
 Requires-Python: >=3.12
 Project-URL: Homepage, https://github.com/gojiplus/understudy
 Project-URL: Documentation, https://gojiplus.github.io/understudy
@@ -40,19 +45,21 @@ Provides-Extra: all
 Provides-Extra: dev
 Provides-Extra: docs
 Provides-Extra: http
+Provides-Extra: langgraph
 Provides-Extra: reports
+Provides-Extra: server
 Description-Content-Type: text/markdown
 
 ## understudy: Scenario Testing for AI Agents
 
 [![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
-[![Downloads](https://pepy.tech/badge/understudy)](https://pepy.tech/project/understudy)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/understudy?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/project/understudy)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
 [![Documentation](https://github.com/gojiplus/understudy/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/understudy/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 
-Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, handoffs, and terminal states, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
+Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
 
 ## How It Works
 
@@ -63,12 +70,12 @@ Testing with understudy is **4 steps**:
 3. **Write scenes** — YAML files defining what the simulated user wants and what you expect
 4. **Run and assert** — Execute simulations, check traces, generate reports
 
-The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls, terminal state).
+The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).
 
 **See real examples:**
 - [Example scene](https://github.com/gojiplus/understudy/blob/main/example/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
-- [Test file](https://github.com/gojiplus/understudy/blob/main/example/test_returns.py) — pytest assertions against traces
-- [Sample report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/example/sample_report.html) — HTML output from `understudy report`
+- [ADK test file](https://github.com/gojiplus/understudy/blob/main/example/adk/test_returns.py) — pytest assertions against traces
+- [LangGraph test file](https://github.com/gojiplus/understudy/blob/main/example/langgraph/test_returns.py) — same tests, different framework
 
 ## Installation
 
@@ -126,21 +133,21 @@ expectations:
   required_tools:
     - lookup_order
     - create_return
-  allowed_terminal_states:
-    - return_created
+  forbidden_tools:
+    - issue_refund
 ```
 
 ### 4. Run simulation
 
 ```python
-from understudy import Scene, run, check
+from understudy import Scene, run
 
 scene = Scene.from_file("scenes/return_backpack.yaml")
 trace = run(app, scene, mocks=mocks)
 
 assert trace.called("lookup_order")
 assert trace.called("create_return")
-assert trace.terminal_state == "return_created"
+assert not trace.called("issue_refund")
 ```
 
 Or with pytest (define `app` and `mocks` fixtures in conftest.py):
@@ -149,9 +156,9 @@ Or with pytest (define `app` and `mocks` fixtures in conftest.py):
 pytest test_returns.py -v
 ```
 
-## Suites with Tags
+## Suites and Batch Runs
 
-Run multiple scenes and tag runs for comparison:
+Run multiple scenes with multiple simulations per scene:
 
 ```python
 from understudy import Suite, RunStorage
@@ -159,33 +166,98 @@ from understudy import Suite, RunStorage
 suite = Suite.from_directory("scenes/")
 storage = RunStorage()
 
-# Tag runs for later comparison
-results = suite.run(app, mocks=mocks, storage=storage, tags={"version": "v1"})
+# Run each scene 3 times and tag for comparison
+results = suite.run(
+    app,
+    mocks=mocks,
+    storage=storage,
+    n_sims=3,
+    tags={"version": "v1"},
+)
 print(f"{results.pass_count}/{len(results.results)} passed")
 ```
 
+## Simulation and Evaluation
+
+Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:
+
+### Combined (most common)
+
+```bash
+understudy run \
+  --app mymodule:agent_app \
+  --scene ./scenes/ \
+  --n-sims 3 \
+  --junit results.xml
+```
+
+### Separate workflows
+
+Generate traces only:
+
+```bash
+understudy simulate \
+  --app mymodule:agent_app \
+  --scenes ./scenes/ \
+  --output ./traces/ \
+  --n-sims 3
+```
+
+Evaluate existing traces:
+
+```bash
+understudy evaluate \
+  --traces ./traces/ \
+  --output ./results/ \
+  --junit results.xml
+```
+
+Python API:
+
+```python
+from understudy import simulate_batch, evaluate_batch
+
+# Generate traces
+traces = simulate_batch(
+    app=agent_app,
+    scenes="./scenes/",
+    n_sims=3,
+    output="./traces/",
+)
+
+# Evaluate later
+results = evaluate_batch(
+    traces="./traces/",
+    output="./results/",
+)
+```
+
 ## CLI Commands
 
 ```bash
-# List runs (shows tags)
-understudy list
+# Run simulations
+understudy run --app mymodule:app --scene ./scenes/
+understudy simulate --app mymodule:app --scenes ./scenes/
+understudy evaluate --traces ./traces/
 
-# Show run details
+# View results
+understudy list
 understudy show <run_id>
-
-# Aggregate metrics
 understudy summary
 
 # Compare runs by tag
 understudy compare --tag version --before v1 --after v2
 
-# Generate HTML reports
+# Generate reports
 understudy report -o report.html
 understudy compare --tag version --before v1 --after v2 --html comparison.html
 
 # Interactive browser
 understudy serve --port 8080
 
+# HTTP simulator server (for browser/UI testing)
+understudy serve-api --port 8000
+
 # Cleanup
 understudy delete <run_id>
 understudy clear
@@ -222,11 +294,10 @@ from understudy.judges import (
 ## Report Contents
 
 The `understudy summary` command shows:
-- **Pass rate** - percentage of scenes that passed all expectations
-- **Avg turns** - average conversation length
-- **Tool usage** - distribution of tool calls across runs
-- **Terminal states** - breakdown of how conversations ended
-- **Agents** - which agents were invoked
+- **Pass rate** — percentage of scenes that passed all expectations
+- **Avg turns** — average conversation length
+- **Tool usage** — distribution of tool calls across runs
+- **Agents** — which agents were invoked
 
 The HTML report (`understudy report`) includes:
 - All metrics above
@@ -241,6 +312,7 @@ See the [full documentation](https://gojiplus.github.io/understudy) for:
 - [Installation guide](https://gojiplus.github.io/understudy/installation.html)
 - [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
 - [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)
 - [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
 - [API reference](https://gojiplus.github.io/understudy/api/index.html)
 

{understudy-0.3.0 → understudy-0.4.0}/README.md

@@ -1,13 +1,13 @@
 ## understudy: Scenario Testing for AI Agents
 
 [![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
-[![Downloads](https://pepy.tech/badge/understudy)](https://pepy.tech/project/understudy)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/understudy?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/project/understudy)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
 [![Documentation](https://github.com/gojiplus/understudy/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/understudy/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 
-Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, handoffs, and terminal states, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
+Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
 
 ## How It Works
 
@@ -18,12 +18,12 @@ Testing with understudy is **4 steps**:
 3. **Write scenes** — YAML files defining what the simulated user wants and what you expect
 4. **Run and assert** — Execute simulations, check traces, generate reports
 
-The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls, terminal state).
+The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).
 
 **See real examples:**
 - [Example scene](https://github.com/gojiplus/understudy/blob/main/example/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
-- [Test file](https://github.com/gojiplus/understudy/blob/main/example/test_returns.py) — pytest assertions against traces
-- [Sample report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/example/sample_report.html) — HTML output from `understudy report`
+- [ADK test file](https://github.com/gojiplus/understudy/blob/main/example/adk/test_returns.py) — pytest assertions against traces
+- [LangGraph test file](https://github.com/gojiplus/understudy/blob/main/example/langgraph/test_returns.py) — same tests, different framework
 
 ## Installation
 
@@ -81,21 +81,21 @@ expectations:
   required_tools:
     - lookup_order
     - create_return
-  allowed_terminal_states:
-    - return_created
+  forbidden_tools:
+    - issue_refund
 ```
 
 ### 4. Run simulation
 
 ```python
-from understudy import Scene, run, check
+from understudy import Scene, run
 
 scene = Scene.from_file("scenes/return_backpack.yaml")
 trace = run(app, scene, mocks=mocks)
 
 assert trace.called("lookup_order")
 assert trace.called("create_return")
-assert trace.terminal_state == "return_created"
+assert not trace.called("issue_refund")
 ```
 
 Or with pytest (define `app` and `mocks` fixtures in conftest.py):
@@ -104,9 +104,9 @@ Or with pytest (define `app` and `mocks` fixtures in conftest.py):
 pytest test_returns.py -v
 ```
 
-## Suites with Tags
+## Suites and Batch Runs
 
-Run multiple scenes and tag runs for comparison:
+Run multiple scenes with multiple simulations per scene:
 
 ```python
 from understudy import Suite, RunStorage
@@ -114,33 +114,98 @@ from understudy import Suite, RunStorage
 suite = Suite.from_directory("scenes/")
 storage = RunStorage()
 
-# Tag runs for later comparison
-results = suite.run(app, mocks=mocks, storage=storage, tags={"version": "v1"})
+# Run each scene 3 times and tag for comparison
+results = suite.run(
+    app,
+    mocks=mocks,
+    storage=storage,
+    n_sims=3,
+    tags={"version": "v1"},
+)
 print(f"{results.pass_count}/{len(results.results)} passed")
 ```
 
+## Simulation and Evaluation
+
+Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:
+
+### Combined (most common)
+
+```bash
+understudy run \
+  --app mymodule:agent_app \
+  --scene ./scenes/ \
+  --n-sims 3 \
+  --junit results.xml
+```
+
+### Separate workflows
+
+Generate traces only:
+
+```bash
+understudy simulate \
+  --app mymodule:agent_app \
+  --scenes ./scenes/ \
+  --output ./traces/ \
+  --n-sims 3
+```
+
+Evaluate existing traces:
+
+```bash
+understudy evaluate \
+  --traces ./traces/ \
+  --output ./results/ \
+  --junit results.xml
+```
+
+Python API:
+
+```python
+from understudy import simulate_batch, evaluate_batch
+
+# Generate traces
+traces = simulate_batch(
+    app=agent_app,
+    scenes="./scenes/",
+    n_sims=3,
+    output="./traces/",
+)
+
+# Evaluate later
+results = evaluate_batch(
+    traces="./traces/",
+    output="./results/",
+)
+```
+
 ## CLI Commands
 
 ```bash
-# List runs (shows tags)
-understudy list
+# Run simulations
+understudy run --app mymodule:app --scene ./scenes/
+understudy simulate --app mymodule:app --scenes ./scenes/
+understudy evaluate --traces ./traces/
 
-# Show run details
+# View results
+understudy list
 understudy show <run_id>
-
-# Aggregate metrics
 understudy summary
 
 # Compare runs by tag
 understudy compare --tag version --before v1 --after v2
 
-# Generate HTML reports
+# Generate reports
 understudy report -o report.html
 understudy compare --tag version --before v1 --after v2 --html comparison.html
 
 # Interactive browser
 understudy serve --port 8080
 
+# HTTP simulator server (for browser/UI testing)
+understudy serve-api --port 8000
+
 # Cleanup
 understudy delete <run_id>
 understudy clear
@@ -177,11 +242,10 @@ from understudy.judges import (
 ## Report Contents
 
 The `understudy summary` command shows:
-- **Pass rate** - percentage of scenes that passed all expectations
-- **Avg turns** - average conversation length
-- **Tool usage** - distribution of tool calls across runs
-- **Terminal states** - breakdown of how conversations ended
-- **Agents** - which agents were invoked
+- **Pass rate** — percentage of scenes that passed all expectations
+- **Avg turns** — average conversation length
+- **Tool usage** — distribution of tool calls across runs
+- **Agents** — which agents were invoked
 
 The HTML report (`understudy report`) includes:
 - All metrics above
@@ -196,6 +260,7 @@ See the [full documentation](https://gojiplus.github.io/understudy) for:
 - [Installation guide](https://gojiplus.github.io/understudy/installation.html)
 - [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
 - [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)
 - [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
 - [API reference](https://gojiplus.github.io/understudy/api/index.html)
 

{understudy-0.3.0 → understudy-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "understudy"
-version = "0.3.0"
+version = "0.4.0"
 description = "Simulation and trace-based evaluation for agentic systems"
 readme = "README.md"
 license = "MIT"
@@ -31,6 +31,8 @@ dependencies = [
 [project.optional-dependencies]
 adk = ["google-adk>=1.0", "python-dotenv>=1.0"]
 http = ["httpx>=0.27"]
+langgraph = ["langgraph>=0.2", "langchain-core>=0.3", "langchain-openai>=0.3"]
+server = ["fastapi>=0.115", "uvicorn>=0.32"]
 reports = ["jinja2>=3.0", "click>=8.0"]
 docs = [
     "sphinx>=7.0",
@@ -38,7 +40,7 @@ docs = [
     "myst-parser>=3.0",
     "sphinx-autodoc-typehints>=2.0",
 ]
-all = ["understudy[adk,http,reports]"]
+all = ["understudy[adk,http,langgraph,server,reports]"]
 dev = [
     "understudy[all,docs]",
     "pytest>=8.0",

{understudy-0.3.0 → understudy-0.4.0}/src/understudy/__init__.py

@@ -7,9 +7,10 @@ not by reading the script, but by inspecting what actually happened.
 
 from importlib.metadata import version
 
-from .check import CheckItem, CheckResult, check
+from .check import CheckItem, CheckResult, EvaluationResult, check, evaluate, evaluate_batch
 from .compare import ComparisonResult, SceneComparison, compare_runs
-from .judges import Judge, JudgeResult
+from .judges import FailureAnalysis, FailureAnalyzer, Judge, JudgeResult
+from .metrics import MetricRegistry, MetricResult
 from .mocks import MockToolkit, ToolError
 from .models import Expectations, Persona, PersonaPreset, Scene
 from .prompts import (
@@ -21,11 +22,11 @@ from .prompts import (
     TONE_EMPATHY,
     TOOL_USAGE_CORRECTNESS,
 )
-from .runner import AgentApp, AgentResponse, LiteLLMBackend, run
+from .runner import AgentApp, AgentResponse, LiteLLMBackend, run, simulate, simulate_batch
 from .simulator import Simulator, SimulatorBackend
-from .storage import RunStorage
+from .storage import EvaluationStorage, RunStorage, TraceStorage
 from .suite import SceneResult, Suite, SuiteResults
-from .trace import AgentTransfer, ToolCall, Trace, Turn
+from .trace import AgentTransfer, StateSnapshot, ToolCall, Trace, TraceMetrics, Turn, TurnMetrics
 
 __version__ = version("understudy")
 
@@ -40,21 +41,34 @@ __all__ = [
     "Turn",
     "ToolCall",
     "AgentTransfer",
-    # runner
+    "TraceMetrics",
+    "TurnMetrics",
+    "StateSnapshot",
+    # metrics
+    "MetricRegistry",
+    "MetricResult",
+    # runner / simulate
     "run",
+    "simulate",
+    "simulate_batch",
     "AgentApp",
     "AgentResponse",
     "LiteLLMBackend",
-    # check
+    # check / evaluate
     "check",
+    "evaluate",
+    "evaluate_batch",
     "CheckResult",
     "CheckItem",
+    "EvaluationResult",
     # suite
     "Suite",
     "SuiteResults",
     "SceneResult",
     # storage
     "RunStorage",
+    "TraceStorage",
+    "EvaluationStorage",
     # compare
     "compare_runs",
     "ComparisonResult",
@@ -62,6 +76,8 @@ __all__ = [
     # judges
     "Judge",
     "JudgeResult",
+    "FailureAnalyzer",
+    "FailureAnalysis",
     # mocks
     "MockToolkit",
     "ToolError",

{understudy-0.3.0 → understudy-0.4.0}/src/understudy/adk/__init__.py

@@ -147,6 +147,9 @@ class ADKApp(AgentApp):
         agent_text_parts: list[str] = []
         terminal_state: str | None = None
         current_agent_name = self._current_agent
+        input_tokens = 0
+        output_tokens = 0
+        thinking_tokens = 0
 
         async for event in self._runner.run_async(
             user_id="understudy_user",
@@ -208,23 +211,40 @@ class ADKApp(AgentApp):
                     if text:
                         agent_text_parts.append(text)
 
-                        # check for terminal state markers
-                        # convention: agent emits "TERMINAL_STATE: <state>"
-                        if "TERMINAL_STATE:" in text:
-                            state = text.split("TERMINAL_STATE:")[-1].strip()
-                            terminal_state = state.split()[0].strip()
-                            logger.debug("Terminal state: %s", terminal_state)
+            # capture token usage from usage_metadata
+            if hasattr(event, "usage_metadata") and event.usage_metadata:
+                usage = event.usage_metadata
+                input_tokens += getattr(usage, "prompt_token_count", 0) or 0
+                output_tokens += getattr(usage, "candidates_token_count", 0) or 0
+                thinking_tokens += getattr(usage, "thoughts_token_count", 0) or 0
+
+            # detect if agent signaled conversation end via escalate action
+            if (
+                hasattr(event, "actions")
+                and event.actions
+                and getattr(event.actions, "escalate", False)
+            ):
+                terminal_state = "agent_ended"
+                logger.debug("Agent escalated (ended conversation)")
 
         self._current_agent = current_agent_name
 
-        response = AgentResponse(
+        # capture state snapshot from session
+        state_snapshot = None
+        if self._session and hasattr(self._session, "state"):
+            state_snapshot = dict(self._session.state) if self._session.state else None
+
+        return AgentResponse(
             content=" ".join(agent_text_parts),
             tool_calls=tool_calls,
             terminal_state=terminal_state,
+            agent_name=current_agent_name,
+            agent_transfers=list(self._agent_transfers),
+            input_tokens=input_tokens,
+            output_tokens=output_tokens,
+            thinking_tokens=thinking_tokens,
+            state_snapshot=state_snapshot,
         )
-        response.agent_name = current_agent_name
-        response.agent_transfers = list(self._agent_transfers)
-        return response
 
     def stop(self) -> None:
         """Clean up the ADK session."""