understudy 0.1.0__tar.gz

understudy-0.1.0/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.4
+Name: understudy
+Version: 0.1.0
+Summary: Simulation and trace-based evaluation for agentic systems
+Keywords: agent,evaluation,simulation,testing,llm
+Author: Gaurav Sood
+License-Expression: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: google-adk>=1.0 ; extra == 'adk'
+Requires-Dist: understudy[adk,judges,http,reports] ; extra == 'all'
+Requires-Dist: understudy[all,docs] ; extra == 'dev'
+Requires-Dist: pytest>=8.0 ; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23 ; extra == 'dev'
+Requires-Dist: ruff>=0.4 ; extra == 'dev'
+Requires-Dist: sphinx>=7.0 ; extra == 'docs'
+Requires-Dist: furo>=2024.0 ; extra == 'docs'
+Requires-Dist: myst-parser>=3.0 ; extra == 'docs'
+Requires-Dist: sphinx-autodoc-typehints>=2.0 ; extra == 'docs'
+Requires-Dist: httpx>=0.27 ; extra == 'http'
+Requires-Dist: litellm>=1.50 ; extra == 'judges'
+Requires-Dist: jinja2>=3.0 ; extra == 'reports'
+Requires-Dist: click>=8.0 ; extra == 'reports'
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/gojiplus/understudy
+Project-URL: Documentation, https://gojiplus.github.io/understudy
+Project-URL: Repository, https://github.com/gojiplus/understudy
+Project-URL: Issues, https://github.com/gojiplus/understudy/issues
+Provides-Extra: adk
+Provides-Extra: all
+Provides-Extra: dev
+Provides-Extra: docs
+Provides-Extra: http
+Provides-Extra: judges
+Provides-Extra: reports
+Description-Content-Type: text/markdown
+
+# understudy
+
+[![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Test your AI agents with simulated users.
+
+## Installation
+
+```bash
+pip install understudy[all]
+```
+
+## Quick Start
+
+### 1. Wrap your agent
+
+```python
+from understudy.adk import ADKApp
+from my_agent import agent
+
+app = ADKApp(agent=agent)
+```
+
+### 2. Mock your tools
+
+Your agent has tools that call external services. Mock them for testing:
+
+```python
+from understudy.mocks import MockToolkit
+
+mocks = MockToolkit()
+
+@mocks.handle("lookup_order")
+def lookup_order(order_id: str) -> dict:
+    return {"order_id": order_id, "items": [...], "status": "delivered"}
+
+@mocks.handle("create_return")
+def create_return(order_id: str, item_sku: str, reason: str) -> dict:
+    return {"return_id": "RET-001", "status": "created"}
+```
+
+### 3. Write a scene
+
+Create `scenes/return_backpack.yaml`:
+
+```yaml
+id: return_eligible_backpack
+description: Customer wants to return a backpack
+
+starting_prompt: "I'd like to return an item please."
+conversation_plan: |
+  Goal: Return the hiking backpack from order ORD-10031.
+  - Provide order ID when asked
+  - Return reason: too small
+
+persona: cooperative
+max_turns: 15
+
+expectations:
+  required_tools:
+    - lookup_order
+    - create_return
+  allowed_terminal_states:
+    - return_created
+```
+
+### 4. Run simulation
+
+```python
+from understudy import Scene, run, check
+
+scene = Scene.from_file("scenes/return_backpack.yaml")
+trace = run(app, scene, mocks=mocks)
+
+assert trace.called("lookup_order")
+assert trace.called("create_return")
+assert trace.terminal_state == "return_created"
+```
+
+Or with pytest (define `app` and `mocks` fixtures in conftest.py):
+
+```bash
+pytest test_returns.py -v
+```
+
+## CLI Commands
+
+After running simulations, use the CLI to inspect results:
+
+```bash
+# List all saved runs
+understudy list
+
+# Show aggregate metrics (pass rate, avg turns, tool usage, terminal states)
+understudy summary
+
+# Show details for a specific run
+understudy show <run_id>
+
+# Generate static HTML report
+understudy report --output report.html
+
+# Start interactive report browser
+understudy serve --port 8080
+
+# Delete runs
+understudy delete <run_id>
+understudy clear
+```
+
+## LLM Judges
+
+For qualities that can't be checked deterministically:
+
+```python
+from understudy.judges import Judge
+
+empathy_judge = Judge(
+    rubric="The agent acknowledged frustration and was empathetic while enforcing policy.",
+    samples=5,
+)
+
+result = empathy_judge.evaluate(trace)
+assert result.score == 1
+```
+
+Built-in rubrics:
+
+```python
+from understudy.judges import (
+    TOOL_USAGE_CORRECTNESS,
+    POLICY_COMPLIANCE,
+    TONE_EMPATHY,
+    ADVERSARIAL_ROBUSTNESS,
+    TASK_COMPLETION,
+)
+```
+
+## Report Contents
+
+The `understudy summary` command shows:
+- **Pass rate** - percentage of scenes that passed all expectations
+- **Avg turns** - average conversation length
+- **Tool usage** - distribution of tool calls across runs
+- **Terminal states** - breakdown of how conversations ended
+- **Agents** - which agents were invoked
+
+The HTML report (`understudy report`) includes:
+- All metrics above
+- Full conversation transcripts
+- Tool call details with arguments
+- Expectation check results
+- Judge evaluation results (when used)
+
+## Documentation
+
+See the [full documentation](https://gojiplus.github.io/understudy) for:
+- [Installation guide](https://gojiplus.github.io/understudy/installation.html)
+- [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
+- [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
+- [API reference](https://gojiplus.github.io/understudy/api/index.html)
+
+## License
+
+MIT

understudy-0.1.0/README.md

@@ -0,0 +1,168 @@
+# understudy
+
+[![PyPI version](https://badge.fury.io/py/understudy.svg)](https://badge.fury.io/py/understudy)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Test your AI agents with simulated users.
+
+## Installation
+
+```bash
+pip install understudy[all]
+```
+
+## Quick Start
+
+### 1. Wrap your agent
+
+```python
+from understudy.adk import ADKApp
+from my_agent import agent
+
+app = ADKApp(agent=agent)
+```
+
+### 2. Mock your tools
+
+Your agent has tools that call external services. Mock them for testing:
+
+```python
+from understudy.mocks import MockToolkit
+
+mocks = MockToolkit()
+
+@mocks.handle("lookup_order")
+def lookup_order(order_id: str) -> dict:
+    return {"order_id": order_id, "items": [...], "status": "delivered"}
+
+@mocks.handle("create_return")
+def create_return(order_id: str, item_sku: str, reason: str) -> dict:
+    return {"return_id": "RET-001", "status": "created"}
+```
+
+### 3. Write a scene
+
+Create `scenes/return_backpack.yaml`:
+
+```yaml
+id: return_eligible_backpack
+description: Customer wants to return a backpack
+
+starting_prompt: "I'd like to return an item please."
+conversation_plan: |
+  Goal: Return the hiking backpack from order ORD-10031.
+  - Provide order ID when asked
+  - Return reason: too small
+
+persona: cooperative
+max_turns: 15
+
+expectations:
+  required_tools:
+    - lookup_order
+    - create_return
+  allowed_terminal_states:
+    - return_created
+```
+
+### 4. Run simulation
+
+```python
+from understudy import Scene, run, check
+
+scene = Scene.from_file("scenes/return_backpack.yaml")
+trace = run(app, scene, mocks=mocks)
+
+assert trace.called("lookup_order")
+assert trace.called("create_return")
+assert trace.terminal_state == "return_created"
+```
+
+Or with pytest (define `app` and `mocks` fixtures in conftest.py):
+
+```bash
+pytest test_returns.py -v
+```
+
+## CLI Commands
+
+After running simulations, use the CLI to inspect results:
+
+```bash
+# List all saved runs
+understudy list
+
+# Show aggregate metrics (pass rate, avg turns, tool usage, terminal states)
+understudy summary
+
+# Show details for a specific run
+understudy show <run_id>
+
+# Generate static HTML report
+understudy report --output report.html
+
+# Start interactive report browser
+understudy serve --port 8080
+
+# Delete runs
+understudy delete <run_id>
+understudy clear
+```
+
+## LLM Judges
+
+For qualities that can't be checked deterministically:
+
+```python
+from understudy.judges import Judge
+
+empathy_judge = Judge(
+    rubric="The agent acknowledged frustration and was empathetic while enforcing policy.",
+    samples=5,
+)
+
+result = empathy_judge.evaluate(trace)
+assert result.score == 1
+```
+
+Built-in rubrics:
+
+```python
+from understudy.judges import (
+    TOOL_USAGE_CORRECTNESS,
+    POLICY_COMPLIANCE,
+    TONE_EMPATHY,
+    ADVERSARIAL_ROBUSTNESS,
+    TASK_COMPLETION,
+)
+```
+
+## Report Contents
+
+The `understudy summary` command shows:
+- **Pass rate** - percentage of scenes that passed all expectations
+- **Avg turns** - average conversation length
+- **Tool usage** - distribution of tool calls across runs
+- **Terminal states** - breakdown of how conversations ended
+- **Agents** - which agents were invoked
+
+The HTML report (`understudy report`) includes:
+- All metrics above
+- Full conversation transcripts
+- Tool call details with arguments
+- Expectation check results
+- Judge evaluation results (when used)
+
+## Documentation
+
+See the [full documentation](https://gojiplus.github.io/understudy) for:
+- [Installation guide](https://gojiplus.github.io/understudy/installation.html)
+- [Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)
+- [ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)
+- [HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)
+- [API reference](https://gojiplus.github.io/understudy/api/index.html)
+
+## License
+
+MIT

understudy-0.1.0/pyproject.toml

@@ -0,0 +1,67 @@
+[build-system]
+requires = ["uv_build>=0.7"]
+build-backend = "uv_build"
+
+[project]
+name = "understudy"
+version = "0.1.0"
+description = "Simulation and trace-based evaluation for agentic systems"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.12"
+authors = [
+    { name = "Gaurav Sood" },
+]
+keywords = ["agent", "evaluation", "simulation", "testing", "llm"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = [
+    "pydantic>=2.0",
+    "pyyaml>=6.0",
+]
+
+[project.optional-dependencies]
+adk = ["google-adk>=1.0"]
+judges = ["litellm>=1.50"]
+http = ["httpx>=0.27"]
+reports = ["jinja2>=3.0", "click>=8.0"]
+docs = [
+    "sphinx>=7.0",
+    "furo>=2024.0",
+    "myst-parser>=3.0",
+    "sphinx-autodoc-typehints>=2.0",
+]
+all = ["understudy[adk,judges,http,reports]"]
+dev = [
+    "understudy[all,docs]",
+    "pytest>=8.0",
+    "pytest-asyncio>=0.23",
+    "ruff>=0.4",
+]
+
+[project.scripts]
+understudy = "understudy.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/gojiplus/understudy"
+Documentation = "https://gojiplus.github.io/understudy"
+Repository = "https://github.com/gojiplus/understudy"
+Issues = "https://github.com/gojiplus/understudy/issues"
+
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+asyncio_mode = "auto"

understudy-0.1.0/src/understudy/__init__.py

@@ -0,0 +1,67 @@
+"""understudy: simulation and trace-based evaluation for agentic systems.
+
+The simulated user is an understudy standing in for a real user.
+You write scenes, run rehearsals, and check the performance —
+not by reading the script, but by inspecting what actually happened.
+"""
+
+from .check import CheckItem, CheckResult, check
+from .judges import Judge, JudgeResult
+from .mocks import MockToolkit, ToolError
+from .models import Expectations, Persona, PersonaPreset, Scene
+from .prompts import (
+    ADVERSARIAL_ROBUSTNESS,
+    FACTUAL_GROUNDING,
+    INSTRUCTION_FOLLOWING,
+    POLICY_COMPLIANCE,
+    TASK_COMPLETION,
+    TONE_EMPATHY,
+    TOOL_USAGE_CORRECTNESS,
+)
+from .runner import AgentApp, AgentResponse, run
+from .storage import RunStorage
+from .suite import SceneResult, Suite, SuiteResults
+from .trace import AgentTransfer, ToolCall, Trace, Turn
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # models
+    "Scene",
+    "Persona",
+    "PersonaPreset",
+    "Expectations",
+    # trace
+    "Trace",
+    "Turn",
+    "ToolCall",
+    "AgentTransfer",
+    # runner
+    "run",
+    "AgentApp",
+    "AgentResponse",
+    # check
+    "check",
+    "CheckResult",
+    "CheckItem",
+    # suite
+    "Suite",
+    "SuiteResults",
+    "SceneResult",
+    # storage
+    "RunStorage",
+    # judges
+    "Judge",
+    "JudgeResult",
+    # mocks
+    "MockToolkit",
+    "ToolError",
+    # rubrics
+    "TOOL_USAGE_CORRECTNESS",
+    "POLICY_COMPLIANCE",
+    "TONE_EMPATHY",
+    "ADVERSARIAL_ROBUSTNESS",
+    "TASK_COMPLETION",
+    "FACTUAL_GROUNDING",
+    "INSTRUCTION_FOLLOWING",
+]

understudy-0.1.0/src/understudy/adk/__init__.py

@@ -0,0 +1,194 @@
+"""ADK adapter: wraps Google ADK agents for use with understudy."""
+
+from datetime import UTC, datetime
+from typing import Any
+
+from ..mocks import MockToolkit
+from ..runner import AgentApp, AgentResponse
+from ..trace import AgentTransfer, ToolCall
+
+
+def _create_mock_callback(mocks: MockToolkit | None):
+    """Create a before_tool_callback that returns mock responses.
+
+    Args:
+        mocks: MockToolkit instance or None.
+
+    Returns:
+        A callback function compatible with google-adk's before_tool_callback.
+        Returns dict to bypass real tool execution (mock response).
+        Returns None to allow normal execution.
+    """
+
+    def callback(tool, args: dict[str, Any], tool_context) -> dict | None:
+        if mocks is None:
+            return None
+        tool_name = getattr(tool, "name", None) or getattr(tool, "__name__", str(tool))
+        if mocks.get_handler(tool_name):
+            try:
+                result = mocks.call(tool_name, **args)
+                return result
+            except Exception as e:
+                return {"error": str(e)}
+        return None
+
+    return callback
+
+
+class ADKApp(AgentApp):
+    """Wraps a Google ADK Agent for use with understudy.
+
+    Usage:
+        from google.adk import Agent
+        from understudy.adk import ADKApp
+
+        agent = Agent(model="gemini-2.5-flash", name="my_agent", ...)
+        app = ADKApp(agent=agent)
+        trace = run(app, scene)
+    """
+
+    def __init__(self, agent: Any, session_id: str | None = None):
+        """
+        Args:
+            agent: A google.adk.Agent instance.
+            session_id: Optional session ID. If None, a random one is generated.
+        """
+        self.agent = agent
+        self.session_id = session_id
+        self._runner = None
+        self._session = None
+        self._mocks: MockToolkit | None = None
+        self._current_agent: str | None = None
+        self._agent_transfers: list[AgentTransfer] = []
+
+    def start(self, mocks: MockToolkit | None = None) -> None:
+        """Initialize the ADK session."""
+        try:
+            from google.adk import Runner
+            from google.adk.sessions import InMemorySessionService
+        except ImportError as e:
+            raise ImportError(
+                "google-adk package required. Install with: pip install understudy[adk]"
+            ) from e
+        import uuid
+
+        self._mocks = mocks
+        self._current_agent = getattr(self.agent, "name", None)
+        self._agent_transfers = []
+        self._session_id = self.session_id or str(uuid.uuid4())
+
+        session_service = InMemorySessionService()
+        if mocks:
+            self.agent.before_tool_callback = _create_mock_callback(mocks)
+
+        self._runner = Runner(
+            agent=self.agent,
+            app_name="understudy_test",
+            session_service=session_service,
+        )
+        self._session = session_service.create_session_sync(
+            app_name="understudy_test",
+            user_id="understudy_user",
+            session_id=self._session_id,
+        )
+
+    def send(self, message: str) -> AgentResponse:
+        """Send a user message to the ADK agent and capture the response."""
+        try:
+            from google.genai import types
+        except ImportError as e:
+            raise ImportError(
+                "google-adk package required. Install with: pip install understudy[adk]"
+            ) from e
+
+        user_content = types.Content(
+            role="user",
+            parts=[types.Part(text=message)],
+        )
+
+        tool_calls: list[ToolCall] = []
+        agent_text_parts: list[str] = []
+        terminal_state: str | None = None
+        current_agent_name = self._current_agent
+
+        for event in self._runner.run(
+            user_id="understudy_user",
+            session_id=self._session.id,
+            new_message=user_content,
+        ):
+            # track agent attribution from event.author
+            if hasattr(event, "author") and event.author:
+                event_agent = event.author
+                if event_agent != current_agent_name and current_agent_name:
+                    self._agent_transfers.append(
+                        AgentTransfer(
+                            from_agent=current_agent_name,
+                            to_agent=event_agent,
+                            timestamp=datetime.now(UTC),
+                        )
+                    )
+                current_agent_name = event_agent
+
+            # detect explicit transfer_to_agent actions
+            if (
+                hasattr(event, "actions")
+                and event.actions
+                and hasattr(event.actions, "transfer_to_agent")
+                and event.actions.transfer_to_agent
+            ):
+                target_agent = event.actions.transfer_to_agent
+                if current_agent_name and target_agent != current_agent_name:
+                    self._agent_transfers.append(
+                        AgentTransfer(
+                            from_agent=current_agent_name,
+                            to_agent=target_agent,
+                            timestamp=datetime.now(UTC),
+                        )
+                    )
+                current_agent_name = target_agent
+
+            # capture tool calls using get_function_calls()
+            for fc in event.get_function_calls():
+                call = ToolCall(
+                    tool_name=fc.name,
+                    arguments=dict(fc.args) if fc.args else {},
+                    agent_name=current_agent_name,
+                )
+                tool_calls.append(call)
+
+            # capture function responses and update tool call results
+            for fr in event.get_function_responses():
+                for call in tool_calls:
+                    if call.tool_name == fr.name and call.result is None:
+                        call.result = fr.response
+                        break
+
+            # capture text responses from content parts
+            if hasattr(event, "content") and event.content and hasattr(event.content, "parts"):
+                for part in event.content.parts:
+                    text = getattr(part, "text", None)
+                    if text:
+                        agent_text_parts.append(text)
+
+                        # check for terminal state markers
+                        # convention: agent emits "TERMINAL_STATE: <state>"
+                        if "TERMINAL_STATE:" in text:
+                            state = text.split("TERMINAL_STATE:")[-1].strip()
+                            terminal_state = state.split()[0].strip()
+
+        self._current_agent = current_agent_name
+
+        response = AgentResponse(
+            content=" ".join(agent_text_parts),
+            tool_calls=tool_calls,
+            terminal_state=terminal_state,
+        )
+        response.agent_name = current_agent_name
+        response.agent_transfers = list(self._agent_transfers)
+        return response
+
+    def stop(self) -> None:
+        """Clean up the ADK session."""
+        self._runner = None
+        self._session = None
+        self._mocks = None