langwatch-scenario 0.1.0__py3-none-any.whl

langwatch_scenario-0.1.0.dist-info/METADATA

@@ -0,0 +1,192 @@
+Metadata-Version: 2.4
+Name: langwatch-scenario
+Version: 0.1.0
+Summary: The end-to-end agent testing library
+Author-email: LangWatch Team <support@langwatch.ai>
+License: MIT
+Project-URL: Homepage, https://github.com/langwatch/scenario
+Project-URL: Bug Tracker, https://github.com/langwatch/scenario/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pytest>=8.1.1
+Requires-Dist: litellm>=1.49.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: termcolor>=2.4.0
+Requires-Dist: pydantic>=2.7.0
+Requires-Dist: joblib>=1.4.2
+Requires-Dist: wrapt>=1.17.2
+Requires-Dist: pytest-asyncio>=0.26.0
+Requires-Dist: rich>=14.0.0
+Provides-Extra: dev
+Requires-Dist: black; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+
+![scenario](./assets/scenario-wide.webp)
+
+<div align="center">
+<!-- Discord, PyPI, Docs, etc links -->
+</div>
+
+# Scenario: Use an Agent to test your Agent
+
+Scenario is a library for testing agents end-to-end as a human would, but without having to manually do it. The automated testing agent covers every single scenario for you.
+
+You define the scenarios, and the testing agent will simulate your users as it follows them, it will keep chatting and evaluating your agent until it reaches the desired goal or detects an unexpected behavior.
+
+## Getting Started
+
+Install pytest and scenario:
+
+```bash
+pip install pytest langwatch-scenario
+```
+
+Now create your first scenario:
+
+```python
+import pytest
+
+from scenario import Scenario, TestingAgent
+
+Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
+
+
+@pytest.mark.agent_test
+@pytest.mark.asyncio
+async def test_vegetarian_recipe_agent():
+    def vegetarian_recipe_agent(message, context):
+        # Call your agent here
+        response = "<Your agent's response>"
+
+        return {"message": response}
+
+    scenario = Scenario(
+        "User is looking for a dinner idea",
+        agent=vegetarian_recipe_agent,
+        success_criteria=[
+            "Recipe agent generates a vegetarian recipe",
+            "Recipe includes step-by-step cooking instructions",
+        ],
+        failure_criteria=[
+            "The recipe includes meat",
+            "The agent asks more than two follow-up questions",
+        ],
+    )
+
+    result = await scenario.run()
+
+    assert result.success
+```
+
+Save it as `tests/test_vegetarian_recipe_agent.py` and run it with pytest:
+
+```bash
+pytest -s tests/test_vegetarian_recipe_agent.py
+```
+
+Once you connect to callback to a real agent, this is how it will look like:
+
+[![asciicast](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
+
+You can find a fully working example in [examples/test_vegetarian_recipe_agent.py](examples/test_vegetarian_recipe_agent.py).
+
+## Customize strategy and max_turns
+
+You can customize how should the testing agent go about testing by defining a `strategy` field. You can also limit the maximum number of turns the scenario will take by setting the `max_turns` field (defaults to 10).
+
+For example, in this Lovable Clone scenario test:
+
+```python
+scenario = Scenario(
+    "user wants to create a new landing page for their dog walking startup",
+    agent=lovable_agent,
+    strategy="send the first message to generate the landing page, then a single follow up request to extend it, then give your final verdict",
+    success_criteria=[
+        "agent reads the files before go and making changes",
+        "agent modified the index.css file",
+        "agent modified the Index.tsx file",
+        "agent created a comprehensive landing page",
+        "agent extended the landing page with a new section",
+    ],
+    failure_criteria=[
+        "agent says it can't read the file",
+        "agent produces incomplete code or is too lazy to finish",
+    ],
+    max_turns=5,
+)
+
+result = await scenario.run()
+```
+
+You can find a fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
+
+## Debug mode
+
+You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running.
+
+Debug mode allows you to see the messages in slow motion step by step, and intervene with your own inputs to debug your agent from the middle of the conversation.
+
+```python
+Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), debug=True)
+```
+
+## Cache
+
+Each time the scenario runs, the testing agent might chose a different input to start, this is good to make sure it covers the variance of real users as well, however we understand that the non-deterministic nature of it might make it less repeatable, costly and harder to debug. To solve for it, you can use the `cache_key` field in the `Scenario.configure` method or in the specific scenario you are running, this will make the testing agent give the same input for given the same scenario:
+
+```python
+Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), cache_key="42")
+```
+
+To bust the cache, you can simply pass a different `cache_key`, disable it, or delete the cache files located at `~/.scenario/cache`.
+
+To go a step further and fully cache the test end-to-end, you can also wrap the LLM calls or any other non-deterministic functions in your application side with the `@scenario_cache` decorator:
+
+```python
+class MyAgent:
+    @scenario_cache(ignore=["self"])
+    def invoke(self, message, context):
+        return client.chat.completions.create(
+            # ...
+        )
+```
+
+This will cache any function call you decorate when running the tests and make them repeatable, hashed by the function arguments, the scenario being executed, and the `cache_key` you provided. You can exclude arguments that should not be hashed for the cache key by naming them in the `ignore` argument.
+
+## Disable Output
+
+You can remove the `-s` flag from pytest to hide the output during test, which will only show up if the test fails. Alternatively, you can set `verbose=False` in the `Scenario.configure` method or in the specific scenario you are running.
+
+## Running in parallel
+
+As the number of your scenarios grows, you might want to run them in parallel to speed up your whole test suite. We suggest you to use the [pytest-asyncio-concurrent](https://pypi.org/project/pytest-asyncio-concurrent/) plugin to do so.
+
+Simply install the plugin from the link above, then replace the `@pytest.mark.asyncio` annotation in the tests with `@pytest.mark.asyncio_concurrent`, adding a group name to it to mark the group of scenarions that should be run in parallel together, e.g.:
+
+```python
+@pytest.mark.agent_test
+@pytest.mark.asyncio_concurrent(group="vegetarian_recipe_agent")
+async def test_vegetarian_recipe_agent():
+    # ...
+
+@pytest.mark.agent_test
+@pytest.mark.asyncio_concurrent(group="vegetarian_recipe_agent")
+async def test_user_is_very_hungry():
+    # ...
+```
+
+Those two scenarios should now run in parallel.
+
+## License
+
+MIT License

langwatch_scenario-0.1.0.dist-info/RECORD

@@ -0,0 +1,15 @@
+scenario/__init__.py,sha256=LfCjOpbn55jYBBZHyMSZtRAWeCDFn4z4OhAyFnu8aMg,602
+scenario/cache.py,sha256=sYu16SAf-BnVYkWSlEDzpyynJGIQyNYsgMXPgCqEnmk,1719
+scenario/config.py,sha256=5UVBmuQDtni0Yu00bMh5p0xMGsrymYVRftXBGTsi2fI,802
+scenario/error_messages.py,sha256=8bTwG_iKz7FjGp50FU0anQ1fmI6eJE4NeaoXtiifbBg,2099
+scenario/pytest_plugin.py,sha256=ydtQxaN09qzoo12nNT8BQY_UPPHAt-AH92HWnPEN6bI,5212
+scenario/result.py,sha256=SGF8uYNtkP7cJy4KsshUozZRevmdiyX2TFzr6VreTv8,2717
+scenario/scenario.py,sha256=3gOTeEOsV7EWmhEUfYnWsdFD-px1JUYnMEixicQrTqY,4009
+scenario/scenario_executor.py,sha256=bDzoatslbp80dG6DU-i2VUlOa9SMtyw2VIhcF7knwis,7883
+scenario/testing_agent.py,sha256=wMK2GqmN4QDr0kFoxgqcAPsU6gjCx8HBJQv1wmsdSb4,10683
+scenario/utils.py,sha256=tMESosrxesA1B5zZB3IJ-sNSXDmnpNNib-DHobveVLA,3918
+langwatch_scenario-0.1.0.dist-info/METADATA,sha256=XlfUHfHDjLKswH8Sq4tuc8nJA-yrs_bCV2h3eXpDM7E,7435
+langwatch_scenario-0.1.0.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
+langwatch_scenario-0.1.0.dist-info/entry_points.txt,sha256=WlEnJ_gku0i18bIa3DSuGqXRX-QDQLe_s0YmRzK45TI,45
+langwatch_scenario-0.1.0.dist-info/top_level.txt,sha256=45Mn28aedJsetnBMB5xSmrJ-yo701QLH89Zlz4r1clE,9
+langwatch_scenario-0.1.0.dist-info/RECORD,,

langwatch_scenario-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (78.1.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

langwatch_scenario-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[pytest11]
+scenario = scenario.pytest_plugin

langwatch_scenario-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+scenario

scenario/__init__.py

@@ -0,0 +1,26 @@
+"""
+Scenario: A testing library for conversational agents.
+"""
+
+# First import non-dependent modules
+from .result import ScenarioResult
+from .config import ScenarioConfig
+
+# Then import modules with dependencies
+from .testing_agent import TestingAgent
+from .scenario import Scenario
+from .cache import scenario_cache
+
+# Import pytest plugin components
+from .pytest_plugin import pytest_configure, scenario_reporter
+
+__all__ = [
+    "Scenario",
+    "TestingAgent",
+    "ScenarioResult",
+    "ScenarioConfig",
+    "pytest_configure",
+    "scenario_reporter",
+    "scenario_cache",
+]
+__version__ = "0.1.0"

scenario/cache.py

@@ -0,0 +1,62 @@
+from contextvars import ContextVar
+import inspect
+import os
+from pathlib import Path
+from typing import Callable, TYPE_CHECKING
+from joblib import Memory
+
+import json
+
+import wrapt
+from scenario.utils import SerializableWithStringFallback
+
+if TYPE_CHECKING:
+    from scenario.scenario import Scenario
+
+
+context_scenario = ContextVar("scenario")
+
+def get_cache() -> Memory:
+    """Get a cross-platform cache directory for scenario."""
+    home_dir = str(Path.home())
+    cache_dir = os.path.join(home_dir, ".scenario", "cache")
+
+    return Memory(location=os.environ.get("SCENARIO_CACHE_DIR", cache_dir), verbose=0)
+
+memory = get_cache()
+
+def scenario_cache(ignore=[]):
+    @wrapt.decorator
+    def wrapper(wrapped: Callable, instance=None, args=[], kwargs={}):
+        scenario: "Scenario" = context_scenario.get()
+
+        if not scenario.cache_key:
+            return wrapped(*args, **kwargs)
+
+        sig = inspect.signature(wrapped)
+        parameters = list(sig.parameters.values())
+
+        all_args = {
+            str(parameter.name): value for parameter, value in zip(parameters, args)
+        }
+        for arg in ["self"] + ignore:
+            if arg in all_args:
+                del all_args[arg]
+
+        cache_key = json.dumps(
+            {
+                "cache_key": scenario.cache_key,
+                "scenario": scenario.model_dump(exclude={"agent"}),
+                "all_args": all_args,
+            },
+            cls=SerializableWithStringFallback,
+        )
+
+        return _cached_call(wrapped, args, kwargs, cache_key=cache_key)
+
+    return wrapper
+
+
+@memory.cache(ignore=["func", "args", "kwargs"])
+def _cached_call(func: Callable, args, kwargs, cache_key):
+    return func(*args, **kwargs)

scenario/config.py

@@ -0,0 +1,28 @@
+"""
+Configuration module for Scenario.
+"""
+
+from typing import Optional, Union
+from pydantic import BaseModel
+
+from scenario.testing_agent import TestingAgent
+
+class ScenarioConfig(BaseModel):
+    """
+    Configuration class for the Scenario library.
+
+    This allows users to set global configuration settings for the library,
+    such as the LLM provider and model to use for the testing agent.
+    """
+
+    testing_agent: Optional[TestingAgent] = None
+    max_turns: Optional[int] = 10
+    verbose: Optional[Union[bool, int]] = True
+    cache_key: Optional[str] = None
+    debug: Optional[bool] = False
+
+    def merge(self, other: "ScenarioConfig") -> "ScenarioConfig":
+        return ScenarioConfig(**{
+            **self.model_dump(),
+            **other.model_dump(exclude_none=True),
+        })

scenario/error_messages.py

@@ -0,0 +1,66 @@
+import termcolor
+
+
+default_config_error_message = f"""
+
+ {termcolor.colored("->", "cyan")} Please set a default config with at least a testing_agent model for running your scenarios at the top of your test file, for example:
+
+    from scenario import Scenario, TestingAgent
+
+    Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
+    {termcolor.colored("^" * 74, "green")}
+
+    @pytest.mark.agent_test
+    def test_vegetarian_recipe_agent():
+        scenario = Scenario(
+            # ...
+        )
+        result = scenario.run()
+
+        assert result.success
+
+
+ {termcolor.colored("->", "cyan")} Alternatively, you can set the config specifically for this scenario:
+
+    from scenario import Scenario, TestingAgent
+
+    @pytest.mark.agent_test
+    def test_vegetarian_recipe_agent():
+        scenario = Scenario(
+            # ...
+            testing_agent=TestingAgent(model="openai/gpt-4o-mini")
+            {termcolor.colored("^" * 54, "green")}
+        )
+        result = scenario.run()
+
+        assert result.success
+                          """
+
+
+message_return_error_message = f"""
+
+ {termcolor.colored("->", "cyan")} Your agent should return a dict with either a "message" string key or a "messages" key in OpenAI messages format so the testing agent can understand what happened. For example:
+
+    def my_agent_under_test(message, context):
+        response = call_my_agent(message)
+
+        return {{
+            "message": response.output_text
+            {termcolor.colored("^" * 31, "green")}
+        }}
+
+ {termcolor.colored("->", "cyan")} Alternatively, you can return a list of messages in OpenAI messages format, you can also optionally provide extra artifacts:
+
+    def my_agent_under_test(message, context):
+        response = call_my_agent(message)
+
+        return {{
+            "messages": [
+                {{"role": "assistant", "content": response}}
+                {termcolor.colored("^" * 42, "green")}
+            ],
+            "extra": {{
+                # ... optional extra artifacts
+            }}
+        }}
+                          """

scenario/pytest_plugin.py

@@ -0,0 +1,161 @@
+"""
+Pytest plugin for Scenario testing library.
+"""
+
+import pytest
+from typing import TypedDict
+import functools
+from termcolor import colored
+
+from scenario.result import ScenarioResult
+
+from .scenario import Scenario
+
+class ScenarioReporterResults(TypedDict):
+    scenario: Scenario
+    result: ScenarioResult
+
+# ScenarioReporter class definition moved outside the fixture for global use
+class ScenarioReporter:
+    def __init__(self):
+        self.results : list[ScenarioReporterResults] = []
+
+    def add_result(self, scenario, result):
+        """Add a test result to the reporter."""
+        self.results.append({"scenario": scenario, "result": result})
+
+    def get_summary(self):
+        """Get a summary of all test results."""
+        total = len(self.results)
+        passed = sum(1 for r in self.results if r["result"].success)
+        failed = total - passed
+
+        return {
+            "total": total,
+            "passed": passed,
+            "failed": failed,
+            "success_rate": round(passed / total * 100, 2) if total else 0,
+        }
+
+    def print_report(self):
+        """Print a detailed report of all test results."""
+        if not self.results:
+            return  # Skip report if no results
+
+        summary = self.get_summary()
+
+        print("\n" + colored("=== Scenario Test Report ===", "cyan", attrs=["bold"]))
+        print(colored(f"Total Scenarios: {summary['total']}", "white"))
+        print(
+            colored(
+                f"Passed: {summary['passed']}",
+                "green" if summary["passed"] > 0 else "white",
+            )
+        )
+        print(
+            colored(
+                f"Failed: {summary['failed']}",
+                "red" if summary["failed"] > 0 else "white",
+            )
+        )
+
+        # Color the success rate based on its value
+        success_rate = summary["success_rate"]
+        rate_color = (
+            "green"
+            if success_rate == 100
+            else "yellow" if success_rate >= 70 else "red"
+        )
+        print(colored(f"Success Rate: {success_rate}%", rate_color))
+
+        for idx, item in enumerate(self.results, 1):
+            scenario = item["scenario"]
+            result = item["result"]
+
+            status = "PASSED" if result.success else "FAILED"
+            status_color = "green" if result.success else "red"
+
+            time = ""
+            if result.total_time and result.agent_time:
+                time = f" in {result.total_time:.2f}s (agent: {result.agent_time:.2f}s)"
+
+            print(
+                f"\n{idx}. {scenario.description} - {colored(status, status_color, attrs=['bold'])}{time}"
+            )
+
+            print(colored(f"   Reasoning: {result.reasoning}", "green" if result.success else "red"))
+
+            if hasattr(result, "met_criteria") and result.met_criteria:
+                criteria_count = len(result.met_criteria)
+                total_criteria = len(scenario.success_criteria)
+                criteria_color = (
+                    "green" if criteria_count == total_criteria else "yellow"
+                )
+                print(
+                    colored(
+                        f"   Success Criteria: {criteria_count}/{total_criteria}",
+                        criteria_color,
+                    )
+                )
+
+            if hasattr(result, "triggered_failures") and result.triggered_failures:
+                print(
+                    colored(
+                        f"   Failures Criteria: {len(result.triggered_failures)}",
+                        "red",
+                    )
+                )
+
+
+# Store the original run method
+original_run = Scenario.run
+
+
+@pytest.hookimpl(trylast=True)
+def pytest_configure(config):
+    """Register the agent_test marker and set up automatic reporting."""
+    # Register the marker
+    config.addinivalue_line(
+        "markers", "agent_test: mark test as an agent scenario test"
+    )
+
+    # Create a global reporter instance
+    config._scenario_reporter = ScenarioReporter()
+
+    # Create a patched version of Scenario.run that auto-reports
+    @functools.wraps(original_run)
+    async def auto_reporting_run(self, *args, **kwargs):
+        result = await original_run(self, *args, **kwargs)
+
+        # Always report to the global reporter
+        config._scenario_reporter.add_result(self, result)
+
+        return result
+
+    # Apply the patch
+    Scenario.run = auto_reporting_run
+
+
+@pytest.hookimpl(trylast=True)
+def pytest_unconfigure(config):
+    """Clean up and print final report when pytest exits."""
+    # Print the final report
+    if hasattr(config, "_scenario_reporter"):
+        config._scenario_reporter.print_report()
+
+    # Restore the original method
+    Scenario.run = original_run
+
+
+@pytest.fixture
+def scenario_reporter(request):
+    """
+    A pytest fixture for accessing the global scenario reporter.
+
+    This fixture provides access to the same reporter that's used for automatic
+    reporting, allowing tests to explicitly interact with the reporter if needed.
+    """
+    # Get the global reporter from pytest config
+    reporter = request.config._scenario_reporter
+    yield reporter
+    # No need to print report here as it's handled by pytest_unconfigure

scenario/result.py

@@ -0,0 +1,81 @@
+"""
+Result module: defines the class for scenario test results.
+"""
+
+from dataclasses import dataclass, field
+from typing import List, Dict, Optional
+
+
+@dataclass
+class ScenarioResult:
+    """
+    Represents the results of a scenario test run.
+
+    Attributes:
+        success: Whether the scenario passed
+        conversation: The conversation history
+        reasoning: Reasoning for the result
+        met_criteria: List of success criteria that were met
+        unmet_criteria: List of success criteria that were not met
+        triggered_failures: List of failure criteria that were triggered
+    """
+
+    success: bool
+    conversation: List[Dict[str, str]]
+    reasoning: Optional[str] = None
+    met_criteria: List[str] = field(default_factory=list)
+    unmet_criteria: List[str] = field(default_factory=list)
+    triggered_failures: List[str] = field(default_factory=list)
+    total_time: Optional[float] = None
+    agent_time: Optional[float] = None
+
+    def __post_init__(self) -> None:
+        """Validate the result after initialization."""
+        if not self.success and not self.reasoning:
+            raise ValueError("Failed scenarios must have a reasoning")
+
+    @classmethod
+    def success_result(
+        cls,
+        conversation: List[Dict[str, str]],
+        reasoning: Optional[str],
+        met_criteria: List[str],
+        total_time: Optional[float] = None,
+        agent_time: Optional[float] = None,
+    ) -> "ScenarioResult":
+        """Create a successful result."""
+        return cls(
+            success=True,
+            conversation=conversation,
+            reasoning=reasoning,
+            met_criteria=met_criteria,
+            unmet_criteria=[],
+            triggered_failures=[],
+            total_time=total_time,
+            agent_time=agent_time,
+        )
+
+    @classmethod
+    def failure_result(
+        cls,
+        conversation: List[Dict[str, str]],
+        reasoning: str,
+        met_criteria: Optional[List[str]] = None,
+        unmet_criteria: Optional[List[str]] = None,
+        triggered_failures: Optional[List[str]] = None,
+        total_time: Optional[float] = None,
+        agent_time: Optional[float] = None,
+    ) -> "ScenarioResult":
+        """Create a failed result."""
+        return cls(
+            success=False,
+            conversation=conversation,
+            reasoning=reasoning,
+            met_criteria=met_criteria if met_criteria is not None else [],
+            unmet_criteria=unmet_criteria if unmet_criteria is not None else [],
+            triggered_failures=(
+                triggered_failures if triggered_failures is not None else []
+            ),
+            total_time=total_time,
+            agent_time=agent_time,
+        )

scenario/scenario.py

@@ -0,0 +1,117 @@
+"""
+Scenario module: defines the core Scenario class for agent testing.
+"""
+
+from typing import Awaitable, List, Dict, Any, Optional, Callable, TypedDict, Union
+import asyncio
+import concurrent.futures
+from functools import partial
+
+from scenario.config import ScenarioConfig
+from scenario.scenario_executor import ScenarioExecutor
+
+from .result import ScenarioResult
+from .testing_agent import TestingAgent
+
+from openai.types.chat import ChatCompletionMessageParam
+
+class AgentResult(TypedDict, total=False):
+    message: str
+    messages: List[ChatCompletionMessageParam]
+    extra: Dict[str, Any]
+
+
+class Scenario(ScenarioConfig):
+    """
+    A scenario represents a specific testing case for an agent.
+
+    It includes:
+    - A description of the scenario
+    - Success criteria to determine if the agent behaved correctly
+    - Failure criteria to determine if the agent failed
+    - An optional strategy that guides the testing agent
+    - Optional additional parameters
+    """
+
+    description: str
+    agent: Union[
+        Callable[[str, Optional[Dict[str, Any]]], Dict[str, Any]],
+        Callable[[str, Optional[Dict[str, Any]]], Awaitable[Dict[str, Any]]],
+    ]
+    success_criteria: List[str]
+    failure_criteria: List[str] = []
+    strategy: Optional[str] = None
+
+    def __init__(self, description: str, **kwargs):
+        """Validate scenario configuration after initialization."""
+        if not description:
+            raise ValueError("Scenario description cannot be empty")
+        kwargs["description"] = description
+
+        if not kwargs.get("success_criteria"):
+            raise ValueError("Scenario must have at least one success criterion")
+
+        if kwargs.get("max_turns", 0) < 1:
+            raise ValueError("max_turns must be a positive integer")
+
+        # Ensure agent is callable
+        if not callable(kwargs.get("agent")):
+            raise ValueError("Agent must be a callable function")
+
+        default_config = getattr(Scenario, "default_config", None)
+        if default_config:
+            kwargs = {**default_config.model_dump(), **kwargs}
+
+        super().__init__(**kwargs)
+
+
+    async def run(self, context: Optional[Dict[str, Any]] = None) -> ScenarioResult:
+        """
+        Run the scenario against the agent under test.
+
+        Args:
+            context: Optional initial context for the agent
+
+        Returns:
+            ScenarioResult containing the test outcome
+        """
+
+        # We'll use a thread pool to run the execution logic, we
+        # require a separate thread because even though asyncio is
+        # being used throughout, any user code on the callback can
+        # be blocking, preventing them from running scenarios in parallel
+        with concurrent.futures.ThreadPoolExecutor() as executor:
+            def run_in_thread():
+                loop = asyncio.new_event_loop()
+                asyncio.set_event_loop(loop)
+
+                try:
+                    return loop.run_until_complete(ScenarioExecutor(self).run(context))
+                finally:
+                    loop.close()
+
+            # Run the function in the thread pool and await its result
+            # This converts the thread's execution into a Future that the current
+            # event loop can await without blocking
+            loop = asyncio.get_event_loop()
+            result = await loop.run_in_executor(executor, run_in_thread)
+            return result
+
+    @classmethod
+    def configure(
+        cls,
+        testing_agent: Optional[TestingAgent] = None,
+        max_turns: Optional[int] = None,
+        verbose: Optional[Union[bool, int]] = None,
+        cache_key: Optional[str] = None,
+    ) -> None:
+        existing_config = getattr(cls, "default_config", ScenarioConfig())
+
+        cls.default_config = existing_config.merge(
+            ScenarioConfig(
+                testing_agent=testing_agent,
+                max_turns=max_turns,
+                verbose=verbose,
+                cache_key=cache_key,
+            )
+        )

scenario/scenario_executor.py

@@ -0,0 +1,204 @@
+"""
+ScenarioExecutor module: holds the scenario execution logic and state, orchestrating the conversation between the testing agent and the agent under test.
+"""
+
+import json
+import sys
+from typing import TYPE_CHECKING, Awaitable, Dict, List, Any, Optional, Union
+import time
+import termcolor
+
+from scenario.error_messages import message_return_error_message
+from scenario.utils import print_openai_messages, safe_attr_or_key, safe_list_at, show_spinner
+from openai.types.chat import ChatCompletionMessageParam
+
+from .result import ScenarioResult
+from .error_messages import default_config_error_message
+from .cache import context_scenario
+
+if TYPE_CHECKING:
+    from scenario.scenario import Scenario
+
+
+
+class ScenarioExecutor:
+    def __init__(self, scenario: "Scenario"):
+        self.scenario = scenario.model_copy()
+
+        testing_agent = scenario.testing_agent
+        if not testing_agent or not testing_agent.model:
+            raise Exception(default_config_error_message)
+        self.testing_agent = testing_agent
+
+        self.conversation: List[Dict[str, Any]] = []
+
+    async def run(
+        self,
+        context: Optional[Dict[str, Any]] = None,
+    ) -> ScenarioResult:
+        """
+        Run a scenario against the agent under test.
+
+        Args:
+            context: Optional initial context for the agent
+
+        Returns:
+            ScenarioResult containing the test outcome
+        """
+
+        if self.scenario.verbose:
+            print("")  # new line
+
+        # Run the initial testing agent prompt to get started
+        total_start_time = time.time()
+        context_scenario.set(self.scenario)
+        initial_message = self._generate_next_message(
+            self.scenario, self.conversation, first_message=True
+        )
+
+        if isinstance(initial_message, ScenarioResult):
+            raise Exception(
+                "Unexpectedly generated a ScenarioResult for the initial message",
+                initial_message.__repr__(),
+            )
+        elif self.scenario.verbose:
+            print(self._scenario_name() + termcolor.colored("User:", "green"), initial_message)
+
+        # Execute the conversation
+        current_turn = 0
+        max_turns = self.scenario.max_turns or 10
+        agent_time = 0
+
+        # Start the test with the initial message
+        while current_turn < max_turns:
+            # Record the testing agent's message
+            self.conversation.append({"role": "user", "content": initial_message})
+
+            # Get response from the agent under test
+            start_time = time.time()
+
+            context_scenario.set(self.scenario)
+            with show_spinner(text="Agent:", color="blue", enabled=self.scenario.verbose):
+                agent_response = self.scenario.agent(initial_message, context)
+                if isinstance(agent_response, Awaitable):
+                    agent_response = await agent_response
+
+            has_valid_message = (
+                "message" in agent_response
+                and type(agent_response["message"]) is str
+                and agent_response["message"] is not None
+            )
+            has_valid_messages = (
+                "messages" in agent_response
+                and isinstance(agent_response["messages"], list)
+                and all(
+                    "role" in msg or hasattr(msg, "role")
+                    for msg in agent_response["messages"]
+                )
+            )
+            if not has_valid_message and not has_valid_messages:
+                raise Exception(message_return_error_message)
+
+            messages: list[ChatCompletionMessageParam] = []
+            if has_valid_messages:
+                messages = agent_response["messages"]
+
+                # Drop the first messages both if they are system or user messages
+                if safe_attr_or_key(safe_list_at(messages, 0), "role") == "system":
+                    messages = messages[1:]
+                if safe_attr_or_key(safe_list_at(messages, 0), "role") == "user":
+                    messages = messages[1:]
+
+            if has_valid_message and self.scenario.verbose:
+                print(self._scenario_name(), termcolor.colored("Agent:", "blue"), agent_response["message"])
+
+            if messages and self.scenario.verbose:
+                print_openai_messages(self._scenario_name(), messages)
+
+            if (
+                self.scenario.verbose
+                and "extra" in agent_response
+                and len(agent_response["extra"].keys()) > 0
+            ):
+                print(
+                    termcolor.colored(
+                        "Extra:" + json.dumps(agent_response["extra"]),
+                        "magenta",
+                    )
+                )
+            response_time = time.time() - start_time
+            agent_time += response_time
+
+            if messages:
+                self.conversation.extend(agent_response["messages"])
+            if "message" in agent_response:
+                self.conversation.append(
+                    {"role": "assistant", "content": agent_response["message"]}
+                )
+            if "extra" in agent_response:
+                self.conversation.append(
+                    {
+                        "role": "assistant",
+                        "content": json.dumps(agent_response["extra"]),
+                    }
+                )
+
+            # Generate the next message OR finish the test based on the agent's evaluation
+            result = self._generate_next_message(
+                self.scenario,
+                self.conversation,
+                last_message=current_turn == max_turns - 1,
+            )
+
+            # Check if the result is a ScenarioResult (indicating test completion)
+            if isinstance(result, ScenarioResult):
+                result.total_time = time.time() - start_time
+                result.agent_time = agent_time
+                return result
+            elif self.scenario.verbose:
+                print(self._scenario_name() + termcolor.colored("User:", "green"), result)
+
+            # Otherwise, it's the next message to send to the agent
+            initial_message = result
+
+            # Increment turn counter
+            current_turn += 1
+
+        # If we reached max turns without conclusion, fail the test
+        return ScenarioResult.failure_result(
+            conversation=self.conversation,
+            reasoning=f"Reached maximum turns ({max_turns}) without conclusion",
+            total_time=time.time() - total_start_time,
+            agent_time=agent_time,
+        )
+
+    def _generate_next_message(
+        self,
+        scenario: "Scenario",
+        conversation: List[Dict[str, Any]],
+        first_message: bool = False,
+        last_message: bool = False,
+    ) -> Union[str, ScenarioResult]:
+        if self.scenario.debug:
+            print(f"\n{self._scenario_name()}{termcolor.colored('[Debug Mode]', 'yellow')} Press enter to continue or type a message to send")
+            input_message = input(self._scenario_name() + termcolor.colored('User: ', 'green'))
+
+            # Clear the input prompt lines completely
+            for _ in range(3):
+                sys.stdout.write("\033[F")  # Move up to the input line
+                sys.stdout.write("\033[2K")  # Clear the entire input line
+            sys.stdout.flush()  # Make sure the clearing is visible
+
+            if input_message:
+                return input_message
+
+        with show_spinner(text=f"{self._scenario_name()}User:", color="green", enabled=self.scenario.verbose):
+            return self.testing_agent.generate_next_message(
+                scenario, conversation, first_message, last_message
+            )
+
+    def _scenario_name(self):
+        if self.scenario.verbose == 2:
+            return termcolor.colored(f"[Scenario: {self.scenario.description}] ", "yellow")
+        else:
+            return ""

scenario/testing_agent.py

@@ -0,0 +1,262 @@
+"""
+TestingAgent module: defines the testing agent that interacts with the agent under test.
+"""
+
+import json
+import logging
+from typing import TYPE_CHECKING, Dict, List, Any, Optional, Union, cast
+from pydantic import BaseModel
+
+from litellm import Choices, completion
+from litellm.files.main import ModelResponse
+
+from scenario.cache import scenario_cache
+from scenario.utils import safe_attr_or_key
+
+from .result import ScenarioResult
+
+if TYPE_CHECKING:
+    from scenario.scenario import Scenario
+
+
+logger = logging.getLogger("scenario")
+
+
+class TestingAgent(BaseModel):
+    """
+    The Testing Agent that interacts with the agent under test.
+
+    This agent is responsible for:
+    1. Generating messages to send to the agent based on the scenario
+    2. Evaluating the responses from the agent against the success/failure criteria
+    3. Determining when to end the test and return a result
+    """
+
+    model: str
+    api_key: Optional[str] = None
+    temperature: float = 0.0
+    max_tokens: Optional[int] = None
+
+    # To prevent pytest from thinking this is actually a test class
+    __test__ = False
+
+    @scenario_cache(ignore=["scenario"])
+    def generate_next_message(
+        self,
+        scenario: "Scenario",
+        conversation: List[Dict[str, Any]],
+        first_message: bool = False,
+        last_message: bool = False,
+    ) -> Union[str, ScenarioResult]:
+        """
+        Generate the next message in the conversation based on history OR
+        return a ScenarioResult if the test should conclude.
+
+        Returns either:
+          - A string message to send to the agent (if conversation should continue)
+          - A ScenarioResult (if the test should conclude)
+        """
+
+        messages = [
+            {
+                "role": "system",
+                "content": f"""
+<role>
+You are pretending to be a user, you are testing an AI Agent (shown as the user role) based on a scenario.
+Approach this naturally, as a human user would, with very short inputs, few words, all lowercase, imperative, not periods, like when they google or talk to chatgpt.
+</role>
+
+<goal>
+Your goal (assistant) is to interact with the Agent Under Test (user) as if you were a human user to see if it can complete the scenario successfully.
+</goal>
+
+<scenario>
+{scenario.description}
+</scenario>
+
+<strategy>
+{scenario.strategy or "Start with a first message and guide the conversation to play out the scenario."}
+</strategy>
+
+<success_criteria>
+{json.dumps(scenario.success_criteria, indent=2)}
+</success_criteria>
+
+<failure_criteria>
+{json.dumps(scenario.failure_criteria, indent=2)}
+</failure_criteria>
+
+<execution_flow>
+1. Generate the first message to start the scenario
+2. After the Agent Under Test (user) responds, generate the next message to send to the Agent Under Test, keep repeating step 2 until criterias match
+3. If the test should end, use the finish_test tool to determine if success or failure criteria have been met
+</execution_flow>
+
+<rules>
+1. Test should end immediately if a failure criteria is triggered
+2. Test should continue until all success criteria have been met
+3. DO NOT make any judgment calls that are not explicitly listed in the success or failure criteria, withhold judgement if necessary
+4. DO NOT carry over any requests yourself, YOU ARE NOT the assistant today, wait for the user to do it
+</rules>
+""",
+            },
+            {"role": "assistant", "content": "Hello, how can I help you today?"},
+            *conversation,
+        ]
+
+        if last_message:
+            messages.append(
+                {
+                    "role": "user",
+                    "content": """
+System:
+
+<finish_test>
+This is the last message, conversation has reached the maximum number of turns, give your final verdict,
+if you don't have enough information to make a verdict, say inconclusive with max turns reached.
+</finish_test>
+""",
+                }
+            )
+
+        # User to assistant role reversal
+        # LLM models are biased to always be the assistant not the user, so we need to do this reversal otherwise models like GPT 4.5 is
+        # super confused, and Claude 3.7 even starts throwing exceptions.
+        for message in messages:
+            # Can't reverse tool calls
+            if not safe_attr_or_key(message, "content") or safe_attr_or_key(
+                message, "tool_calls"
+            ):
+                continue
+
+            if type(message) == dict:
+                if message["role"] == "user":
+                    message["role"] = "assistant"
+                elif message["role"] == "assistant":
+                    message["role"] = "user"
+            else:
+                if getattr(message, "role", None) == "user":
+                    message.role = "assistant"
+                elif getattr(message, "role", None) == "assistant":
+                    message.role = "user"
+
+        # Define the tool
+        tools = [
+            {
+                "type": "function",
+                "function": {
+                    "name": "finish_test",
+                    "description": "Complete the test with a final verdict",
+                    "strict": True,
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "verdict": {
+                                "type": "string",
+                                "enum": ["success", "failure", "inconclusive"],
+                                "description": "The final verdict of the test",
+                            },
+                            "reasoning": {
+                                "type": "string",
+                                "description": "Explanation of why this verdict was reached",
+                            },
+                            "details": {
+                                "type": "object",
+                                "properties": {
+                                    "met_criteria": {
+                                        "type": "array",
+                                        "items": {"type": "string"},
+                                        "description": "List of success criteria that have been met",
+                                    },
+                                    "unmet_criteria": {
+                                        "type": "array",
+                                        "items": {"type": "string"},
+                                        "description": "List of success criteria that have not been met",
+                                    },
+                                    "triggered_failures": {
+                                        "type": "array",
+                                        "items": {"type": "string"},
+                                        "description": "List of failure criteria that have been triggered",
+                                    },
+                                },
+                                "required": ["met_criteria", "unmet_criteria", "triggered_failures"],
+                                "additionalProperties": False,
+                                "description": "Detailed information about criteria evaluation",
+                            },
+                        },
+                        "required": ["verdict", "reasoning", "details"],
+                        "additionalProperties": False,
+                    },
+                },
+            }
+        ]
+
+        response = cast(
+            ModelResponse,
+            completion(
+                model=self.model,
+                messages=messages,
+                temperature=self.temperature,
+                max_tokens=self.max_tokens,
+                tools=tools if not first_message else None,
+                tool_choice="required" if last_message else None,
+            ),
+        )
+
+        # Extract the content from the response
+        if hasattr(response, "choices") and len(response.choices) > 0:
+            message = cast(Choices, response.choices[0]).message
+
+            # Check if the LLM chose to use the tool
+            if message.tool_calls:
+                tool_call = message.tool_calls[0]
+                if tool_call.function.name == "finish_test":
+                    # Parse the tool call arguments
+                    try:
+                        args = json.loads(tool_call.function.arguments)
+                        verdict = args.get("verdict", "inconclusive")
+                        reasoning = args.get("reasoning", "No reasoning provided")
+                        details = args.get("details", {})
+
+                        met_criteria = details.get("met_criteria", [])
+                        unmet_criteria = details.get("unmet_criteria", [])
+                        triggered_failures = details.get("triggered_failures", [])
+
+                        # Return the appropriate ScenarioResult based on the verdict
+                        if verdict == "success":
+                            return ScenarioResult.success_result(
+                                conversation=conversation,
+                                reasoning=reasoning,
+                                met_criteria=met_criteria,
+                            )
+                        elif verdict == "failure":
+                            return ScenarioResult.failure_result(
+                                conversation=conversation,
+                                reasoning=reasoning,
+                                met_criteria=met_criteria,
+                                unmet_criteria=unmet_criteria,
+                                triggered_failures=triggered_failures,
+                            )
+                        else:  # inconclusive
+                            return ScenarioResult(
+                                success=False,
+                                conversation=conversation,
+                                reasoning=reasoning,
+                                met_criteria=met_criteria,
+                                unmet_criteria=unmet_criteria,
+                                triggered_failures=triggered_failures,
+                            )
+                    except json.JSONDecodeError:
+                        logger.error("Failed to parse tool call arguments")
+
+            # If no tool call or invalid tool call, use the message content as next message
+            message_content = message.content
+            if message_content is None:
+                raise Exception(f"No response from LLM: {response.__repr__()}")
+
+            return message_content
+        else:
+            raise Exception(
+                f"Unexpected response format from LLM: {response.__repr__()}"
+            )
+

scenario/utils.py

@@ -0,0 +1,121 @@
+from contextlib import contextmanager
+import sys
+from typing import Optional, Union
+from pydantic import BaseModel
+
+import json
+
+import termcolor
+from textwrap import indent
+from openai.types.chat import ChatCompletionMessageParam
+from rich.live import Live
+from rich.spinner import Spinner
+from rich.console import Console
+from rich.text import Text
+from rich.errors import LiveError
+
+
+
+class SerializableAndPydanticEncoder(json.JSONEncoder):
+    def default(self, o):
+        if isinstance(o, BaseModel):
+            return o.model_dump(exclude_unset=True)
+        return super().default(o)
+
+
+class SerializableWithStringFallback(SerializableAndPydanticEncoder):
+    def default(self, o):
+        try:
+            return super().default(o)
+        except:
+            return str(o)
+
+
+def safe_list_at(list, index, default=None):
+    try:
+        return list[index]
+    except:
+        return default
+
+
+def safe_attr_or_key(obj, attr_or_key, default=None):
+    return getattr(obj, attr_or_key, obj.get(attr_or_key))
+
+
+def title_case(string):
+    return " ".join(word.capitalize() for word in string.split("_"))
+
+
+def print_openai_messages(scenario_name: str, messages: list[ChatCompletionMessageParam]):
+    for msg in messages:
+        role = safe_attr_or_key(msg, "role")
+        content = safe_attr_or_key(msg, "content")
+        if role == "assistant":
+            tool_calls = safe_attr_or_key(msg, "tool_calls")
+            if content:
+                print(scenario_name + termcolor.colored("Agent:", "blue"), content)
+            if tool_calls:
+                for tool_call in tool_calls:
+                    function = safe_attr_or_key(tool_call, "function")
+                    name = safe_attr_or_key(function, "name")
+                    args = safe_attr_or_key(function, "arguments", "{}")
+                    args = _take_maybe_json_first_lines(args)
+                    print(
+                        scenario_name + termcolor.colored(f"ToolCall({name}):", "magenta"),
+                        f"\n\n{indent(args, ' ' * 4)}\n",
+                    )
+        elif role == "tool":
+            content = _take_maybe_json_first_lines(content or msg.__repr__())
+            print(
+                scenario_name + termcolor.colored(f"ToolResult:", "magenta"),
+                f"\n\n{indent(content, ' ' * 4)}\n",
+            )
+        else:
+            print(
+                scenario_name + termcolor.colored(f"{title_case(role)}:", "magenta"),
+                msg.__repr__(),
+            )
+
+
+def _take_maybe_json_first_lines(string, max_lines=5):
+    content = str(string)
+    try:
+        content = json.dumps(json.loads(content), indent=2)
+    except:
+        pass
+    content = content.split("\n")
+    if len(content) > max_lines:
+        content = content[:max_lines] + ["..."]
+    return "\n".join(content)
+
+
+console = Console()
+
+class TextFirstSpinner(Spinner):
+    def __init__(self, name, text: str, color: str, **kwargs):
+        super().__init__(name, "", style="bold white", **kwargs)  # Initialize with empty text
+        self.text_before = text
+        self.color = color
+
+    def render(self, time):
+        # Get the original spinner frame
+        spinner_frame = super().render(time)
+        # Create a composite with text first, then spinner
+        return Text(f"{self.text_before} ", style=self.color) + spinner_frame
+
+
+@contextmanager
+def show_spinner(text: str, color: str = "white", enabled: Optional[Union[bool, int]] = None):
+    if not enabled:
+        yield
+    else:
+        spinner = TextFirstSpinner("dots", text, color=color)
+        try:
+            with Live(spinner, console=console, refresh_per_second=20):
+                yield
+        # It happens when we are multi-threading, it's fine, just ignore it, you probably don't want multiple spinners at once anyway
+        except LiveError:
+            yield
+
+        # Cursor up one line
+        sys.stdout.write("\033[F")