langwatch-scenario 0.3.0__py3-none-any.whl → 0.6.0__py3-none-any.whl

langwatch_scenario-0.3.0.dist-info/METADATA

@@ -1,302 +0,0 @@
-Metadata-Version: 2.4
-Name: langwatch-scenario
-Version: 0.3.0
-Summary: The end-to-end agent testing library
-Author-email: LangWatch Team <support@langwatch.ai>
-License: MIT
-Project-URL: Homepage, https://github.com/langwatch/scenario
-Project-URL: Bug Tracker, https://github.com/langwatch/scenario/issues
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-Requires-Dist: pytest>=8.1.1
-Requires-Dist: litellm>=1.49.0
-Requires-Dist: python-dotenv>=1.0.1
-Requires-Dist: termcolor>=2.4.0
-Requires-Dist: pydantic>=2.7.0
-Requires-Dist: joblib>=1.4.2
-Requires-Dist: wrapt>=1.17.2
-Requires-Dist: pytest-asyncio>=0.26.0
-Requires-Dist: rich<15.0.0,>=13.3.3
-Requires-Dist: pksuid>=1.1.2
-Provides-Extra: dev
-Requires-Dist: black; extra == "dev"
-Requires-Dist: isort; extra == "dev"
-Requires-Dist: pytest-cov; extra == "dev"
-Requires-Dist: pre-commit; extra == "dev"
-Requires-Dist: commitizen; extra == "dev"
-
-![scenario](https://github.com/langwatch/scenario/raw/main/assets/scenario-wide.webp)
-
-<div align="center">
-<!-- Discord, PyPI, Docs, etc links -->
-</div>
-
-# Scenario: Use an Agent to test your Agent
-
-Scenario is an Agent Testing Framework for testing AI agents through Simulation Testing.
-
-You define the scenarios, and the testing agent will simulate a real user as it follows them, it will keep chatting back and forth with _your_ agent to play out the simulation, until it reaches the desired goal or detects an unexpected behavior based on the criteria you defined.
-
-[📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
-
-### See also
-
-- [Scenario TypeScript](https://github.com/langwatch/scenario-ts/)
-- [Scenario Go](https://github.com/langwatch/scenario-go/)
-
-## Getting Started
-
-Install pytest and scenario:
-
-```bash
-pip install pytest langwatch-scenario
-```
-
-Now create your first scenario and save it as `tests/test_vegetarian_recipe_agent.py`:
-
-```python
-import pytest
-
-from scenario import Scenario, TestingAgent, ScenarioAgentAdapter, AgentInput, AgentReturnTypes, scenario_cache
-
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
-
-
-# Create an adapter to call your agent
-class VegetarianRecipeAgentAdapter(ScenarioAgentAdapter):
-    def __init__(self, input: AgentInput):
-        self.agent = VegetarianRecipeAgent()
-
-    async def call(self, input: AgentInput) -> AgentReturnTypes:
-        return self.agent.run(input.last_new_user_message_str())
-
-
-@pytest.mark.agent_test
-@pytest.mark.asyncio
-async def test_vegetarian_recipe_agent():
-    # Define the simulated scenario
-    scenario = Scenario(
-        name="dinner idea",
-        description="""
-            It's saturday evening, the user is very hungry and tired,
-            but have no money to order out, so they are looking for a recipe.
-
-            The user never mentions they want a vegetarian recipe.
-        """,
-        agent=vegetarian_recipe_agent,
-        # List the evaluation criteria for the scenario to be considered successful
-        criteria=[
-            "Agent should not ask more than two follow-up questions",
-            "Agent should generate a recipe",
-            "Recipe should include a list of ingredients",
-            "Recipe should include step-by-step cooking instructions",
-            "Recipe should be vegetarian and not include any sort of meat",
-        ],
-    )
-
-    # Run the scenario and get results
-    result = await scenario.run()
-
-    # Assert for pytest to know whether the test passed
-    assert result.success
-
-
-# Example agent implementation
-import litellm
-
-
-class VegetarianRecipeAgent:
-    def __init__(self):
-        self.history = []
-
-    @scenario_cache()
-    def run(self, message: str):
-        self.history.append({"role": "user", "content": message})
-
-        response = litellm.completion(
-            model="openai/gpt-4o-mini",
-            messages=[
-                {
-                    "role": "system",
-                    "content": """
-                        You are a vegetarian recipe agent.
-                        Given the user request, ask AT MOST ONE follow-up question,
-                        then provide a complete recipe. Keep your responses concise and focused.
-                    """,
-                },
-                *self.history,
-            ],
-        )
-        message = response.choices[0].message  # type: ignore
-        self.history.append(message)
-
-        return [message]
-
-```
-
-Create a `.env` file and put your OpenAI API key in it:
-
-```bash
-OPENAI_API_KEY=<your-api-key>
-```
-
-Now run it with pytest:
-
-```bash
-pytest -s tests/test_vegetarian_recipe_agent.py
-```
-
-This is how it will look like:
-
-[![asciicast](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
-
-You can find a fully working example in [examples/test_vegetarian_recipe_agent.py](examples/test_vegetarian_recipe_agent.py).
-
-## Customize strategy and max_turns
-
-You can customize how should the testing agent go about testing by defining a `strategy` field. You can also limit the maximum number of turns the scenario will take by setting the `max_turns` field (defaults to 10).
-
-For example, in this Lovable Clone scenario test:
-
-```python
-scenario = Scenario(
-    name="dog walking startup landing page",
-    description="""
-        the user wants to create a new landing page for their dog walking startup
-
-        send the first message to generate the landing page, then a single follow up request to extend it, then give your final verdict
-    """,
-    agent=lovable_agent,
-    criteria=[
-        "agent reads the files before go and making changes",
-        "agent modified the index.css file, not only the Index.tsx file",
-        "agent created a comprehensive landing page",
-        "agent extended the landing page with a new section",
-        "agent should NOT say it can't read the file",
-        "agent should NOT produce incomplete code or be too lazy to finish",
-    ],
-    max_turns=5,
-)
-
-result = await scenario.run()
-```
-
-You can find a fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
-
-## Specify a script for guiding the scenario
-
-You can specify a script for guiding the scenario by passing a list of steps to the `script` field.
-
-```python
-@pytest.mark.agent_test
-@pytest.mark.asyncio
-async def test_ai_assistant_agent():
-    scenario = Scenario(
-        name="false assumptions",
-        description="""
-            The agent makes false assumption about being an ATM bank, and user corrects it
-        """,
-        agent=AiAssistantAgentAdapter,
-        criteria=[
-            "user should get good recommendations on river crossing",
-            "agent should NOT follow up about ATM recommendation after user has corrected them they are just hiking",
-        ],
-        max_turns=5,
-    )
-
-    def check_if_tool_was_called(state: ScenarioExecutor) -> None:
-        assert state.has_tool_call("web_search")
-
-    result = await scenario.script(
-        [
-            # Define existing history of messages
-            scenario.user("how do I safely approach a bank?"),
-            # Or let it be generate automatically
-            scenario.agent(),
-            # Add custom assertions, for example making sure a tool was called
-            check_if_tool_was_called,
-            scenario.user(),
-            # Let the simulation proceed for 2 more turns
-            scenario.proceed(turns=2),
-            # Time to make a judgment call
-            scenario.judge(),
-        ]
-    ).run()
-
-    assert result.success
-```
-
-## Debug mode
-
-You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.
-
-Debug mode allows you to see the messages in slow motion step by step, and intervene with your own inputs to debug your agent from the middle of the conversation.
-
-```python
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), debug=True)
-```
-
-or
-
-```bash
-pytest -s tests/test_vegetarian_recipe_agent.py --debug
-```
-
-## Cache
-
-Each time the scenario runs, the testing agent might chose a different input to start, this is good to make sure it covers the variance of real users as well, however we understand that the non-deterministic nature of it might make it less repeatable, costly and harder to debug. To solve for it, you can use the `cache_key` field in the `Scenario.configure` method or in the specific scenario you are running, this will make the testing agent give the same input for given the same scenario:
-
-```python
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), cache_key="42")
-```
-
-To bust the cache, you can simply pass a different `cache_key`, disable it, or delete the cache files located at `~/.scenario/cache`.
-
-To go a step further and fully cache the test end-to-end, you can also wrap the LLM calls or any other non-deterministic functions in your application side with the `@scenario_cache` decorator:
-
-```python
-class MyAgent:
-    @scenario_cache(ignore=["self"])
-    def invoke(self, message, context):
-        return client.chat.completions.create(
-            # ...
-        )
-```
-
-This will cache any function call you decorate when running the tests and make them repeatable, hashed by the function arguments, the scenario being executed, and the `cache_key` you provided. You can exclude arguments that should not be hashed for the cache key by naming them in the `ignore` argument.
-
-## Disable Output
-
-You can remove the `-s` flag from pytest to hide the output during test, which will only show up if the test fails. Alternatively, you can set `verbose=False` in the `Scenario.configure` method or in the specific scenario you are running.
-
-## Running in parallel
-
-As the number of your scenarios grows, you might want to run them in parallel to speed up your whole test suite. We suggest you to use the [pytest-asyncio-concurrent](https://pypi.org/project/pytest-asyncio-concurrent/) plugin to do so.
-
-Simply install the plugin from the link above, then replace the `@pytest.mark.asyncio` annotation in the tests with `@pytest.mark.asyncio_concurrent`, adding a group name to it to mark the group of scenarions that should be run in parallel together, e.g.:
-
-```python
-@pytest.mark.agent_test
-@pytest.mark.asyncio_concurrent(group="vegetarian_recipe_agent")
-async def test_vegetarian_recipe_agent():
-    # ...
-
-@pytest.mark.agent_test
-@pytest.mark.asyncio_concurrent(group="vegetarian_recipe_agent")
-async def test_user_is_very_hungry():
-    # ...
-```
-
-Those two scenarios should now run in parallel.
-
-## License
-
-MIT License

langwatch_scenario-0.3.0.dist-info/RECORD

@@ -1,16 +0,0 @@
-scenario/__init__.py,sha256=0OavO4hoZMFL6frlplNkR7BSHfGSOhuVtmKmTrOMFEs,844
-scenario/cache.py,sha256=sYu16SAf-BnVYkWSlEDzpyynJGIQyNYsgMXPgCqEnmk,1719
-scenario/config.py,sha256=NiCCmr8flds-VDzvF8ps4SChVTARtcWfEoHhK0UkDMQ,1076
-scenario/error_messages.py,sha256=8_pa3HIaqkw08qOqeiRKDCNykr9jtofpNJoEV03aRWc,4690
-scenario/pytest_plugin.py,sha256=oJtEPVPi5x50Z-UawVyVPNd6buvh_4msSZ-3hLFpw_Y,5770
-scenario/scenario.py,sha256=K4Snu4-pJaoprEFyly7ZQT8qNlAamxt-eXibCJ0EIJU,7332
-scenario/scenario_agent_adapter.py,sha256=Y2dP3z-2jLYCssQ20oHOphwwrRPQNo2HmLD2KBcJRu0,427
-scenario/scenario_executor.py,sha256=geaP3Znd1he66L6ku3l2IAODj68TtAIk8b8Ssy494xA,15681
-scenario/testing_agent.py,sha256=5S2PIl2hi9FBSVjjs9afXhEgiogryjBIyffH5iJBwdo,10676
-scenario/types.py,sha256=-Uz0qg_fY5vAEkrZnM5CMqE5hiP8OtNErpDdHJmHtac,3179
-scenario/utils.py,sha256=bx813RpZO3xyPfD-dTBbeLM9umWm3PGOq9pw48aJoHI,8113
-langwatch_scenario-0.3.0.dist-info/METADATA,sha256=pywrVOVE2eE4Zk5wePzJoEfErNXWvgK-C8G-qfWp7EI,11040
-langwatch_scenario-0.3.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-langwatch_scenario-0.3.0.dist-info/entry_points.txt,sha256=WlEnJ_gku0i18bIa3DSuGqXRX-QDQLe_s0YmRzK45TI,45
-langwatch_scenario-0.3.0.dist-info/top_level.txt,sha256=45Mn28aedJsetnBMB5xSmrJ-yo701QLH89Zlz4r1clE,9
-langwatch_scenario-0.3.0.dist-info/RECORD,,

scenario/scenario.py

@@ -1,238 +0,0 @@
-"""
-Scenario module: defines the core Scenario class for agent testing.
-"""
-
-from typing import (
-    Awaitable,
-    Callable,
-    List,
-    Dict,
-    Any,
-    Optional,
-    Type,
-    TypedDict,
-    Union,
-)
-import asyncio
-import concurrent.futures
-
-from scenario.config import ScenarioConfig
-from scenario.error_messages import (
-    default_config_error_message,
-    message_invalid_agent_type,
-)
-from scenario.scenario_agent_adapter import ScenarioAgentAdapter
-from scenario.scenario_executor import ScenarioExecutor
-
-from .types import ScenarioResult, ScriptStep
-
-from openai.types.chat import ChatCompletionMessageParam
-
-
-class AgentResult(TypedDict, total=False):
-    message: str
-    messages: List[ChatCompletionMessageParam]
-    extra: Dict[str, Any]
-
-
-class Scenario(ScenarioConfig):
-    """
-    A scenario represents a specific testing case for an agent.
-
-    It includes:
-    - A description of the scenario
-    - Criteria to determine if the agent behaved correctly
-    - Optional additional parameters
-    """
-
-    name: str
-    description: str
-    agents: List[Type[ScenarioAgentAdapter]]
-    criteria: List[str]
-
-    def __init__(
-        self,
-        name: str,
-        description: str,
-        criteria: List[str] = [],
-        agent: Optional[Type[ScenarioAgentAdapter]] = None,
-        testing_agent: Optional[Type[ScenarioAgentAdapter]] = None,
-        agents: List[Type[ScenarioAgentAdapter]] = [],
-        max_turns: Optional[int] = None,
-        verbose: Optional[Union[bool, int]] = None,
-        cache_key: Optional[str] = None,
-        debug: Optional[bool] = None,
-    ):
-        """Validate scenario configuration after initialization."""
-
-        config = ScenarioConfig(
-            testing_agent=testing_agent,
-            max_turns=max_turns,
-            verbose=verbose,
-            cache_key=cache_key,
-            debug=debug,
-        )
-
-        kwargs = config.items()
-        default_config: Optional[ScenarioConfig] = getattr(
-            Scenario, "default_config", None
-        )
-        if default_config:
-            kwargs = default_config.merge(config).items()
-
-        if not name:
-            raise ValueError("Scenario name cannot be empty")
-        kwargs["name"] = name
-
-        if not description:
-            raise ValueError("Scenario description cannot be empty")
-        kwargs["description"] = description
-
-        kwargs["criteria"] = criteria
-
-        if kwargs.get("max_turns", 10) < 1:
-            raise ValueError("max_turns must be a positive integer")
-
-        if not agents and not agent:
-            raise ValueError(
-                "Missing required argument `agent`. Either `agent` or `agents` argument must be provided for the Scenario"
-            )
-
-        if not agents and not kwargs.get("testing_agent"):
-            raise Exception(default_config_error_message)
-
-        agents = agents or [
-            kwargs.get("testing_agent"),
-            agent,  # type: ignore
-        ]
-
-        # Ensure each agent is a ScenarioAgentAdapter
-        for agent in agents:
-            if (
-                not agent
-                or not isinstance(agent, type)
-                or not issubclass(agent, ScenarioAgentAdapter)
-            ):
-                raise ValueError(message_invalid_agent_type(agent))
-        kwargs["agents"] = agents
-
-        super().__init__(**kwargs)
-
-    def script(self, script: List[ScriptStep]):
-        class ScriptedScenario:
-            def __init__(self, scenario: "Scenario"):
-                self._scenario = scenario
-
-            async def run(
-                self, context: Optional[Dict[str, Any]] = None
-            ) -> ScenarioResult:
-                return await self._scenario._run(context, script)
-
-        return ScriptedScenario(self)
-
-    async def run(self, context: Optional[Dict[str, Any]] = None) -> ScenarioResult:
-        """
-        Run the scenario against the agent under test.
-
-        Args:
-            context: Optional initial context for the agent
-
-        Returns:
-            ScenarioResult containing the test outcome
-        """
-
-        return await self._run(context, None)
-
-    async def _run(
-        self,
-        context: Optional[Dict[str, Any]] = None,
-        script: Optional[List[ScriptStep]] = None,
-    ) -> ScenarioResult:
-        # We'll use a thread pool to run the execution logic, we
-        # require a separate thread because even though asyncio is
-        # being used throughout, any user code on the callback can
-        # be blocking, preventing them from running scenarios in parallel
-        with concurrent.futures.ThreadPoolExecutor() as executor:
-
-            def run_in_thread():
-                loop = asyncio.new_event_loop()
-                asyncio.set_event_loop(loop)
-
-                try:
-                    return loop.run_until_complete(
-                        ScenarioExecutor(self, context, script).run()
-                    )
-                finally:
-                    loop.close()
-
-            # Run the function in the thread pool and await its result
-            # This converts the thread's execution into a Future that the current
-            # event loop can await without blocking
-            loop = asyncio.get_event_loop()
-            result = await loop.run_in_executor(executor, run_in_thread)
-            return result
-
-    @classmethod
-    def configure(
-        cls,
-        testing_agent: Optional[Type[ScenarioAgentAdapter]] = None,
-        max_turns: Optional[int] = None,
-        verbose: Optional[Union[bool, int]] = None,
-        cache_key: Optional[str] = None,
-        debug: Optional[bool] = None,
-    ) -> None:
-        existing_config = getattr(cls, "default_config", ScenarioConfig())
-
-        cls.default_config = existing_config.merge(
-            ScenarioConfig(
-                testing_agent=testing_agent,
-                max_turns=max_turns,
-                verbose=verbose,
-                cache_key=cache_key,
-                debug=debug,
-            )
-        )
-
-    # Scenario Scripting
-
-    def message(self, message: ChatCompletionMessageParam) -> ScriptStep:
-        return lambda state: state.message(message)
-
-    def user(
-        self, content: Optional[Union[str, ChatCompletionMessageParam]] = None
-    ) -> ScriptStep:
-        return lambda state: state.user(content)
-
-    def agent(
-        self, content: Optional[Union[str, ChatCompletionMessageParam]] = None
-    ) -> ScriptStep:
-        return lambda state: state.agent(content)
-
-    def judge(
-        self, content: Optional[Union[str, ChatCompletionMessageParam]] = None
-    ) -> ScriptStep:
-        return lambda state: state.judge(content)
-
-    def proceed(
-        self,
-        turns: Optional[int] = None,
-        on_turn: Optional[
-            Union[
-                Callable[[ScenarioExecutor], None],
-                Callable[[ScenarioExecutor], Awaitable[None]],
-            ]
-        ] = None,
-        on_step: Optional[
-            Union[
-                Callable[[ScenarioExecutor], None],
-                Callable[[ScenarioExecutor], Awaitable[None]],
-            ]
-        ] = None,
-    ) -> ScriptStep:
-        return lambda state: state.proceed(turns, on_turn, on_step)
-
-    def succeed(self) -> ScriptStep:
-        return lambda state: state.succeed()
-
-    def fail(self) -> ScriptStep:
-        return lambda state: state.fail()

scenario/scenario_agent_adapter.py

@@ -1,16 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import ClassVar, Set
-
-from .types import AgentInput, AgentReturnTypes, ScenarioAgentRole
-
-
-class ScenarioAgentAdapter(ABC):
-    roles: ClassVar[Set[ScenarioAgentRole]] = {ScenarioAgentRole.AGENT}
-
-    def __init__(self, input: AgentInput):
-        super().__init__()
-        pass
-
-    @abstractmethod
-    async def call(self, input: AgentInput) -> AgentReturnTypes:
-        pass