understudy 0.3.0__tar.gz → 0.5.0__tar.gz

{understudy-0.3.0 → understudy-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: understudy
-Version: 0.3.0
+Version: 0.5.0
 Summary: Simulation and trace-based evaluation for agentic systems
 Keywords: agent,evaluation,simulation,testing,llm
 Author: Gaurav Sood
@@ -17,7 +17,7 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: litellm>=1.50
 Requires-Dist: google-adk>=1.0 ; extra == 'adk'
 Requires-Dist: python-dotenv>=1.0 ; extra == 'adk'
-Requires-Dist: understudy[adk,http,reports] ; extra == 'all'
+Requires-Dist: understudy[adk,http,langgraph,server,reports] ; extra == 'all'
 Requires-Dist: understudy[all,docs] ; extra == 'dev'
 Requires-Dist: pytest>=8.0 ; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23 ; extra == 'dev'
@@ -28,8 +28,13 @@ Requires-Dist: furo>=2024.0 ; extra == 'docs'
 Requires-Dist: myst-parser>=3.0 ; extra == 'docs'
 Requires-Dist: sphinx-autodoc-typehints>=2.0 ; extra == 'docs'
 Requires-Dist: httpx>=0.27 ; extra == 'http'
+Requires-Dist: langgraph>=0.2 ; extra == 'langgraph'
+Requires-Dist: langchain-core>=0.3 ; extra == 'langgraph'
+Requires-Dist: langchain-openai>=0.3 ; extra == 'langgraph'
 Requires-Dist: jinja2>=3.0 ; extra == 'reports'
 Requires-Dist: click>=8.0 ; extra == 'reports'
+Requires-Dist: fastapi>=0.115 ; extra == 'server'
+Requires-Dist: uvicorn>=0.32 ; extra == 'server'
 Requires-Python: >=3.12
 Project-URL: Homepage, https://github.com/gojiplus/understudy
 Project-URL: Documentation, https://gojiplus.github.io/understudy
@@ -40,19 +45,21 @@ Provides-Extra: all
 Provides-Extra: dev
 Provides-Extra: docs
 Provides-Extra: http
+Provides-Extra: langgraph
 Provides-Extra: reports
+Provides-Extra: server
 Description-Content-Type: text/markdown
 
 ## understudy: Scenario Testing for AI Agents
 
 [![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
-[![Downloads](https://pepy.tech/badge/understudy)](https://pepy.tech/project/understudy)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/understudy?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/project/understudy)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
 [![Documentation](https://github.com/gojiplus/understudy/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/understudy/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 
-Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, handoffs, and terminal states, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
+Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
 
 ## How It Works
 
@@ -63,12 +70,29 @@ Testing with understudy is **4 steps**:
 3. **Write scenes** — YAML files defining what the simulated user wants and what you expect
 4. **Run and assert** — Execute simulations, check traces, generate reports
 
-The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls, terminal state).
+The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).
+
+## Two Evaluation Paradigms
+
+### Conversational Agent Evaluation
+Simulate multi-turn conversations with personas to test dialogue agents.
+- Use case: Customer service bots, assistants, chatbots
+- Assert on tool calls: `trace.called("tool_name")`
+
+### Agentic Flow Evaluation
+Evaluate autonomous agents executing multi-step tasks.
+- Use case: Code agents, research agents, task automation
+- Assert on actions: `trace.performed("action")`
+
+See [examples/README.md](examples/README.md) for complete examples of both paradigms.
 
 **See real examples:**
-- [Example scene](https://github.com/gojiplus/understudy/blob/main/example/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
-- [Test file](https://github.com/gojiplus/understudy/blob/main/example/test_returns.py) — pytest assertions against traces
-- [Sample report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/example/sample_report.html) — HTML output from `understudy report`
+- [Example scene](https://github.com/gojiplus/understudy/blob/main/examples/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
+- [ADK test file](https://github.com/gojiplus/understudy/blob/main/examples/adk/test_adk_returns.py) — pytest assertions against traces
+- [LangGraph test file](https://github.com/gojiplus/understudy/blob/main/examples/langgraph/test_langgraph_returns.py) — same tests, different framework
+- [Agentic test file](https://github.com/gojiplus/understudy/blob/main/examples/agentic/test_agentic.py) — agentic flow evaluation
+- [Agentic scene](https://github.com/gojiplus/understudy/blob/main/examples/agentic_scenes/code_review_task.yaml) — task-based scenario
+- [Example report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/examples/langgraph/report/index.html) — HTML report with metrics and transcripts
 
 ## Installation
 
@@ -126,21 +150,21 @@ expectations:
   required_tools:
     - lookup_order
     - create_return
-  allowed_terminal_states:
-    - return_created
+  forbidden_tools:
+    - issue_refund
 ```
 
 ### 4. Run simulation
 
 ```python
-from understudy import Scene, run, check
+from understudy import Scene, run
 
 scene = Scene.from_file("scenes/return_backpack.yaml")
 trace = run(app, scene, mocks=mocks)
 
 assert trace.called("lookup_order")
 assert trace.called("create_return")
-assert trace.terminal_state == "return_created"
+assert not trace.called("issue_refund")
 ```
 
 Or with pytest (define `app` and `mocks` fixtures in conftest.py):
@@ -149,9 +173,9 @@ Or with pytest (define `app` and `mocks` fixtures in conftest.py):
 pytest test_returns.py -v
 ```
 
-## Suites with Tags
+## Suites and Batch Runs
 
-Run multiple scenes and tag runs for comparison:
+Run multiple scenes with multiple simulations per scene:
 
 ```python
 from understudy import Suite, RunStorage
@@ -159,33 +183,98 @@ from understudy import Suite, RunStorage
 suite = Suite.from_directory("scenes/")
 storage = RunStorage()
 
-# Tag runs for later comparison
-results = suite.run(app, mocks=mocks, storage=storage, tags={"version": "v1"})
+# Run each scene 3 times and tag for comparison
+results = suite.run(
+    app,
+    mocks=mocks,
+    storage=storage,
+    n_sims=3,
+    tags={"version": "v1"},
+)
 print(f"{results.pass_count}/{len(results.results)} passed")
 ```
 
+## Simulation and Evaluation
+
+Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:
+
+### Combined (most common)
+
+```bash
+understudy run \
+  --app mymodule:agent_app \
+  --scene ./scenes/ \
+  --n-sims 3 \
+  --junit results.xml
+```
+
+### Separate workflows
+
+Generate traces only:
+
+```bash
+understudy simulate \
+  --app mymodule:agent_app \
+  --scenes ./scenes/ \
+  --output ./traces/ \
+  --n-sims 3
+```
+
+Evaluate existing traces:
+
+```bash
+understudy evaluate \
+  --traces ./traces/ \
+  --output ./results/ \
+  --junit results.xml
+```
+
+Python API:
+
+```python
+from understudy import simulate_batch, evaluate_batch
+
+# Generate traces
+traces = simulate_batch(
+    app=agent_app,
+    scenes="./scenes/",
+    n_sims=3,
+    output="./traces/",
+)
+
+# Evaluate later
+results = evaluate_batch(
+    traces="./traces/",
+    output="./results/",
+)
+```
+
 ## CLI Commands
 
 ```bash
-# List runs (shows tags)
-understudy list
+# Run simulations
+understudy run --app mymodule:app --scene ./scenes/
+understudy simulate --app mymodule:app --scenes ./scenes/
+understudy evaluate --traces ./traces/
 
-# Show run details
+# View results
+understudy list
 understudy show <run_id>
-
-# Aggregate metrics
 understudy summary
 
 # Compare runs by tag
 understudy compare --tag version --before v1 --after v2
 
-# Generate HTML reports
+# Generate reports
 understudy report -o report.html
 understudy compare --tag version --before v1 --after v2 --html comparison.html
 
 # Interactive browser
 understudy serve --port 8080
 
+# HTTP simulator server (for browser/UI testing)
+understudy serve-api --port 8000
+
 # Cleanup
 understudy delete <run_id>
 understudy clear
@@ -222,11 +311,10 @@ from understudy.judges import (
 ## Report Contents
 
 The `understudy summary` command shows:
-- **Pass rate** - percentage of scenes that passed all expectations
-- **Avg turns** - average conversation length
-- **Tool usage** - distribution of tool calls across runs
-- **Terminal states** - breakdown of how conversations ended
-- **Agents** - which agents were invoked
+- **Pass rate** — percentage of scenes that passed all expectations
+- **Avg turns** — average conversation length
+- **Tool usage** — distribution of tool calls across runs
+- **Agents** — which agents were invoked
 
 The HTML report (`understudy report`) includes:
 - All metrics above
@@ -241,6 +329,7 @@ See the [full documentation](https://gojiplus.github.io/understudy) for:
 - [Installation guide](https://gojiplus.github.io/understudy/installation.html)
 - [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
 - [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)
 - [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
 - [API reference](https://gojiplus.github.io/understudy/api/index.html)
 

{understudy-0.3.0 → understudy-0.5.0}/README.md

@@ -1,13 +1,13 @@
 ## understudy: Scenario Testing for AI Agents
 
 [![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
-[![Downloads](https://pepy.tech/badge/understudy)](https://pepy.tech/project/understudy)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/understudy?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/project/understudy)
 [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
 [![Documentation](https://github.com/gojiplus/understudy/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/understudy/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 
-Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, handoffs, and terminal states, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
+Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.
 
 ## How It Works
 
@@ -18,12 +18,29 @@ Testing with understudy is **4 steps**:
 3. **Write scenes** — YAML files defining what the simulated user wants and what you expect
 4. **Run and assert** — Execute simulations, check traces, generate reports
 
-The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls, terminal state).
+The key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).
+
+## Two Evaluation Paradigms
+
+### Conversational Agent Evaluation
+Simulate multi-turn conversations with personas to test dialogue agents.
+- Use case: Customer service bots, assistants, chatbots
+- Assert on tool calls: `trace.called("tool_name")`
+
+### Agentic Flow Evaluation
+Evaluate autonomous agents executing multi-step tasks.
+- Use case: Code agents, research agents, task automation
+- Assert on actions: `trace.performed("action")`
+
+See [examples/README.md](examples/README.md) for complete examples of both paradigms.
 
 **See real examples:**
-- [Example scene](https://github.com/gojiplus/understudy/blob/main/example/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
-- [Test file](https://github.com/gojiplus/understudy/blob/main/example/test_returns.py) — pytest assertions against traces
-- [Sample report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/example/sample_report.html) — HTML output from `understudy report`
+- [Example scene](https://github.com/gojiplus/understudy/blob/main/examples/scenes/return_eligible_backpack.yaml) — YAML defining a test scenario
+- [ADK test file](https://github.com/gojiplus/understudy/blob/main/examples/adk/test_adk_returns.py) — pytest assertions against traces
+- [LangGraph test file](https://github.com/gojiplus/understudy/blob/main/examples/langgraph/test_langgraph_returns.py) — same tests, different framework
+- [Agentic test file](https://github.com/gojiplus/understudy/blob/main/examples/agentic/test_agentic.py) — agentic flow evaluation
+- [Agentic scene](https://github.com/gojiplus/understudy/blob/main/examples/agentic_scenes/code_review_task.yaml) — task-based scenario
+- [Example report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/examples/langgraph/report/index.html) — HTML report with metrics and transcripts
 
 ## Installation
 
@@ -81,21 +98,21 @@ expectations:
   required_tools:
     - lookup_order
     - create_return
-  allowed_terminal_states:
-    - return_created
+  forbidden_tools:
+    - issue_refund
 ```
 
 ### 4. Run simulation
 
 ```python
-from understudy import Scene, run, check
+from understudy import Scene, run
 
 scene = Scene.from_file("scenes/return_backpack.yaml")
 trace = run(app, scene, mocks=mocks)
 
 assert trace.called("lookup_order")
 assert trace.called("create_return")
-assert trace.terminal_state == "return_created"
+assert not trace.called("issue_refund")
 ```
 
 Or with pytest (define `app` and `mocks` fixtures in conftest.py):
@@ -104,9 +121,9 @@ Or with pytest (define `app` and `mocks` fixtures in conftest.py):
 pytest test_returns.py -v
 ```
 
-## Suites with Tags
+## Suites and Batch Runs
 
-Run multiple scenes and tag runs for comparison:
+Run multiple scenes with multiple simulations per scene:
 
 ```python
 from understudy import Suite, RunStorage
@@ -114,33 +131,98 @@ from understudy import Suite, RunStorage
 suite = Suite.from_directory("scenes/")
 storage = RunStorage()
 
-# Tag runs for later comparison
-results = suite.run(app, mocks=mocks, storage=storage, tags={"version": "v1"})
+# Run each scene 3 times and tag for comparison
+results = suite.run(
+    app,
+    mocks=mocks,
+    storage=storage,
+    n_sims=3,
+    tags={"version": "v1"},
+)
 print(f"{results.pass_count}/{len(results.results)} passed")
 ```
 
+## Simulation and Evaluation
+
+Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:
+
+### Combined (most common)
+
+```bash
+understudy run \
+  --app mymodule:agent_app \
+  --scene ./scenes/ \
+  --n-sims 3 \
+  --junit results.xml
+```
+
+### Separate workflows
+
+Generate traces only:
+
+```bash
+understudy simulate \
+  --app mymodule:agent_app \
+  --scenes ./scenes/ \
+  --output ./traces/ \
+  --n-sims 3
+```
+
+Evaluate existing traces:
+
+```bash
+understudy evaluate \
+  --traces ./traces/ \
+  --output ./results/ \
+  --junit results.xml
+```
+
+Python API:
+
+```python
+from understudy import simulate_batch, evaluate_batch
+
+# Generate traces
+traces = simulate_batch(
+    app=agent_app,
+    scenes="./scenes/",
+    n_sims=3,
+    output="./traces/",
+)
+
+# Evaluate later
+results = evaluate_batch(
+    traces="./traces/",
+    output="./results/",
+)
+```
+
 ## CLI Commands
 
 ```bash
-# List runs (shows tags)
-understudy list
+# Run simulations
+understudy run --app mymodule:app --scene ./scenes/
+understudy simulate --app mymodule:app --scenes ./scenes/
+understudy evaluate --traces ./traces/
 
-# Show run details
+# View results
+understudy list
 understudy show <run_id>
-
-# Aggregate metrics
 understudy summary
 
 # Compare runs by tag
 understudy compare --tag version --before v1 --after v2
 
-# Generate HTML reports
+# Generate reports
 understudy report -o report.html
 understudy compare --tag version --before v1 --after v2 --html comparison.html
 
 # Interactive browser
 understudy serve --port 8080
 
+# HTTP simulator server (for browser/UI testing)
+understudy serve-api --port 8000
+
 # Cleanup
 understudy delete <run_id>
 understudy clear
@@ -177,11 +259,10 @@ from understudy.judges import (
 ## Report Contents
 
 The `understudy summary` command shows:
-- **Pass rate** - percentage of scenes that passed all expectations
-- **Avg turns** - average conversation length
-- **Tool usage** - distribution of tool calls across runs
-- **Terminal states** - breakdown of how conversations ended
-- **Agents** - which agents were invoked
+- **Pass rate** — percentage of scenes that passed all expectations
+- **Avg turns** — average conversation length
+- **Tool usage** — distribution of tool calls across runs
+- **Agents** — which agents were invoked
 
 The HTML report (`understudy report`) includes:
 - All metrics above
@@ -196,6 +277,7 @@ See the [full documentation](https://gojiplus.github.io/understudy) for:
 - [Installation guide](https://gojiplus.github.io/understudy/installation.html)
 - [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
 - [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)
 - [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
 - [API reference](https://gojiplus.github.io/understudy/api/index.html)
 

{understudy-0.3.0 → understudy-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "understudy"
-version = "0.3.0"
+version = "0.5.0"
 description = "Simulation and trace-based evaluation for agentic systems"
 readme = "README.md"
 license = "MIT"
@@ -31,6 +31,8 @@ dependencies = [
 [project.optional-dependencies]
 adk = ["google-adk>=1.0", "python-dotenv>=1.0"]
 http = ["httpx>=0.27"]
+langgraph = ["langgraph>=0.2", "langchain-core>=0.3", "langchain-openai>=0.3"]
+server = ["fastapi>=0.115", "uvicorn>=0.32"]
 reports = ["jinja2>=3.0", "click>=8.0"]
 docs = [
     "sphinx>=7.0",
@@ -38,7 +40,7 @@ docs = [
     "myst-parser>=3.0",
     "sphinx-autodoc-typehints>=2.0",
 ]
-all = ["understudy[adk,http,reports]"]
+all = ["understudy[adk,http,langgraph,server,reports]"]
 dev = [
     "understudy[all,docs]",
     "pytest>=8.0",
@@ -50,6 +52,9 @@ dev = [
 [project.scripts]
 understudy = "understudy.cli:main"
 
+[project.entry-points.pytest11]
+understudy = "understudy.pytest_plugin"
+
 [project.urls]
 Homepage = "https://github.com/gojiplus/understudy"
 Documentation = "https://gojiplus.github.io/understudy"

understudy-0.5.0/src/understudy/__init__.py

@@ -0,0 +1,149 @@
+"""understudy: simulation and trace-based evaluation for agentic systems.
+
+The simulated user is an understudy standing in for a real user.
+You write scenes, run rehearsals, and check the performance —
+not by reading the script, but by inspecting what actually happened.
+"""
+
+from importlib.metadata import version
+
+from . import agentic
+from .agentic import (
+    AgenticApp,
+    AgenticCheckItem,
+    AgenticCheckResult,
+    AgenticExpectations,
+    AgenticMetricResult,
+    AgenticScene,
+    AgenticTrace,
+    Artifact,
+    Step,
+    StepResult,
+    Task,
+    check_agentic,
+    run_agentic,
+)
+from .check import CheckItem, CheckResult, EvaluationResult, check, evaluate, evaluate_batch
+from .compare import ComparisonResult, SceneComparison, compare_runs
+from .diff import ToolCallDiff, TraceDiff, diff_tool_sequences, diff_traces
+from .judge_backends import CallbackBackend, JudgeBackend, LiteLLMBackend
+from .judges import FailureAnalysis, FailureAnalyzer, Judge, JudgeResult
+from .metrics import MetricRegistry, MetricResult
+from .mocks import MockToolkit, ToolError
+from .models import Expectations, Persona, PersonaPreset, Scene
+from .prompts import (
+    ADVERSARIAL_ROBUSTNESS,
+    FACTUAL_GROUNDING,
+    INSTRUCTION_FOLLOWING,
+    POLICY_COMPLIANCE,
+    TASK_COMPLETION,
+    TONE_EMPATHY,
+    TOOL_USAGE_CORRECTNESS,
+)
+from .pytest_plugin import AssertionHelpers
+from .replay import ReplayResult, create_replay_scene, load_trace, replay
+from .runner import AgentApp, AgentResponse, run, simulate, simulate_batch
+from .simulator import Simulator, SimulatorBackend
+from .storage import EvaluationStorage, RunStorage, TraceStorage
+from .suite import SceneResult, Suite, SuiteResults
+from .trace import AgentTransfer, StateSnapshot, ToolCall, Trace, TraceMetrics, Turn, TurnMetrics
+from .validation import SceneValidationError
+
+__version__ = version("understudy")
+
+__all__ = [
+    # models
+    "Scene",
+    "Persona",
+    "PersonaPreset",
+    "Expectations",
+    # trace
+    "Trace",
+    "Turn",
+    "ToolCall",
+    "AgentTransfer",
+    "TraceMetrics",
+    "TurnMetrics",
+    "StateSnapshot",
+    # metrics
+    "MetricRegistry",
+    "MetricResult",
+    # runner / simulate
+    "run",
+    "simulate",
+    "simulate_batch",
+    "AgentApp",
+    "AgentResponse",
+    "LiteLLMBackend",
+    # check / evaluate
+    "check",
+    "evaluate",
+    "evaluate_batch",
+    "CheckResult",
+    "CheckItem",
+    "EvaluationResult",
+    # suite
+    "Suite",
+    "SuiteResults",
+    "SceneResult",
+    # storage
+    "RunStorage",
+    "TraceStorage",
+    "EvaluationStorage",
+    # compare
+    "compare_runs",
+    "ComparisonResult",
+    "SceneComparison",
+    # diff
+    "diff_traces",
+    "diff_tool_sequences",
+    "TraceDiff",
+    "ToolCallDiff",
+    # replay
+    "replay",
+    "load_trace",
+    "create_replay_scene",
+    "ReplayResult",
+    # judges
+    "Judge",
+    "JudgeResult",
+    "FailureAnalyzer",
+    "FailureAnalysis",
+    # judge backends
+    "JudgeBackend",
+    "LiteLLMBackend",
+    "CallbackBackend",
+    # mocks
+    "MockToolkit",
+    "ToolError",
+    # validation
+    "SceneValidationError",
+    # pytest
+    "AssertionHelpers",
+    # simulator
+    "Simulator",
+    "SimulatorBackend",
+    # rubrics
+    "TOOL_USAGE_CORRECTNESS",
+    "POLICY_COMPLIANCE",
+    "TONE_EMPATHY",
+    "ADVERSARIAL_ROBUSTNESS",
+    "TASK_COMPLETION",
+    "FACTUAL_GROUNDING",
+    "INSTRUCTION_FOLLOWING",
+    # agentic module
+    "agentic",
+    "AgenticApp",
+    "AgenticCheckItem",
+    "AgenticCheckResult",
+    "AgenticExpectations",
+    "AgenticMetricResult",
+    "AgenticScene",
+    "AgenticTrace",
+    "Artifact",
+    "Step",
+    "StepResult",
+    "Task",
+    "check_agentic",
+    "run_agentic",
+]