langwatch-scenario 0.2.0__py3-none-any.whl → 0.4.0__py3-none-any.whl

langwatch_scenario-0.2.0.dist-info/METADATA

@@ -1,254 +0,0 @@
-Metadata-Version: 2.4
-Name: langwatch-scenario
-Version: 0.2.0
-Summary: The end-to-end agent testing library
-Author-email: LangWatch Team <support@langwatch.ai>
-License: MIT
-Project-URL: Homepage, https://github.com/langwatch/scenario
-Project-URL: Bug Tracker, https://github.com/langwatch/scenario/issues
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-Requires-Dist: pytest>=8.1.1
-Requires-Dist: litellm>=1.49.0
-Requires-Dist: python-dotenv>=1.0.1
-Requires-Dist: termcolor>=2.4.0
-Requires-Dist: pydantic>=2.7.0
-Requires-Dist: joblib>=1.4.2
-Requires-Dist: wrapt>=1.17.2
-Requires-Dist: pytest-asyncio>=0.26.0
-Requires-Dist: rich<15.0.0,>=13.3.3
-Provides-Extra: dev
-Requires-Dist: black; extra == "dev"
-Requires-Dist: isort; extra == "dev"
-Requires-Dist: mypy; extra == "dev"
-Requires-Dist: pytest-cov; extra == "dev"
-
-![scenario](https://github.com/langwatch/scenario/raw/main/assets/scenario-wide.webp)
-
-<div align="center">
-<!-- Discord, PyPI, Docs, etc links -->
-</div>
-
-# Scenario: Use an Agent to test your Agent
-
-Scenario is a library for testing agents end-to-end as a human would, but without having to manually do it. The automated testing agent covers every single scenario for you.
-
-You define the scenarios, and the testing agent will simulate your users as it follows them, it will keep chatting and evaluating your agent until it reaches the desired goal or detects an unexpected behavior.
-
-[📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
-
-### See also
-
-- [Scenario TypeScript](https://github.com/langwatch/scenario-ts/)
-- [Scenario Go](https://github.com/langwatch/scenario-go/)
-
-## Getting Started
-
-Install pytest and scenario:
-
-```bash
-pip install pytest langwatch-scenario
-```
-
-Now create your first scenario and save it as `tests/test_vegetarian_recipe_agent.py`:
-
-```python
-import pytest
-
-from scenario import Scenario, TestingAgent, scenario_cache
-
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
-
-
-@pytest.mark.agent_test
-@pytest.mark.asyncio
-async def test_vegetarian_recipe_agent():
-    agent = VegetarianRecipeAgent()
-
-    def vegetarian_recipe_agent(message, context):
-        # Call your agent here
-        return agent.run(message)
-
-    # Define the simulated scenario
-    scenario = Scenario(
-        name="dinner idea",
-        description="""
-            It's saturday evening, the user is very hungry and tired,
-            but have no money to order out, so they are looking for a recipe.
-
-            The user never mentions they want a vegetarian recipe.
-        """,
-        agent=vegetarian_recipe_agent,
-        # List the evaluation criteria for the scenario to be considered successful
-        criteria=[
-            "Agent should not ask more than two follow-up questions",
-            "Agent should generate a recipe",
-            "Recipe should include a list of ingredients",
-            "Recipe should include step-by-step cooking instructions",
-            "Recipe should be vegetarian and not include any sort of meat",
-        ],
-    )
-
-    # Run the scenario and get results
-    result = await scenario.run()
-
-    # Assert for pytest to know whether the test passed
-    assert result.success
-
-
-# Example agent implementation
-import litellm
-
-
-class VegetarianRecipeAgent:
-    def __init__(self):
-        self.history = []
-
-    @scenario_cache()
-    def run(self, message: str):
-        self.history.append({"role": "user", "content": message})
-
-        response = litellm.completion(
-            model="openai/gpt-4o-mini",
-            messages=[
-                {
-                    "role": "system",
-                    "content": """
-                        You are a vegetarian recipe agent.
-                        Given the user request, ask AT MOST ONE follow-up question,
-                        then provide a complete recipe. Keep your responses concise and focused.
-                    """,
-                },
-                *self.history,
-            ],
-        )
-        message = response.choices[0].message  # type: ignore
-        self.history.append(message)
-
-        return {"messages": [message]}
-
-```
-
-Create a `.env` file and put your OpenAI API key in it:
-
-```bash
-OPENAI_API_KEY=<your-api-key>
-```
-
-Now run it with pytest:
-
-```bash
-pytest -s tests/test_vegetarian_recipe_agent.py
-```
-
-This is how it will look like:
-
-[![asciicast](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
-
-You can find a fully working example in [examples/test_vegetarian_recipe_agent.py](examples/test_vegetarian_recipe_agent.py).
-
-## Customize strategy and max_turns
-
-You can customize how should the testing agent go about testing by defining a `strategy` field. You can also limit the maximum number of turns the scenario will take by setting the `max_turns` field (defaults to 10).
-
-For example, in this Lovable Clone scenario test:
-
-```python
-scenario = Scenario(
-    name="dog walking startup landing page",
-    description="""
-        the user wants to create a new landing page for their dog walking startup
-
-        send the first message to generate the landing page, then a single follow up request to extend it, then give your final verdict
-    """,
-    agent=lovable_agent,
-    criteria=[
-        "agent reads the files before go and making changes",
-        "agent modified the index.css file, not only the Index.tsx file",
-        "agent created a comprehensive landing page",
-        "agent extended the landing page with a new section",
-        "agent should NOT say it can't read the file",
-        "agent should NOT produce incomplete code or be too lazy to finish",
-    ],
-    max_turns=5,
-)
-
-result = await scenario.run()
-```
-
-You can find a fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
-
-## Debug mode
-
-You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.
-
-Debug mode allows you to see the messages in slow motion step by step, and intervene with your own inputs to debug your agent from the middle of the conversation.
-
-```python
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), debug=True)
-```
-
-or
-
-```bash
-pytest -s tests/test_vegetarian_recipe_agent.py --debug
-```
-
-## Cache
-
-Each time the scenario runs, the testing agent might chose a different input to start, this is good to make sure it covers the variance of real users as well, however we understand that the non-deterministic nature of it might make it less repeatable, costly and harder to debug. To solve for it, you can use the `cache_key` field in the `Scenario.configure` method or in the specific scenario you are running, this will make the testing agent give the same input for given the same scenario:
-
-```python
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), cache_key="42")
-```
-
-To bust the cache, you can simply pass a different `cache_key`, disable it, or delete the cache files located at `~/.scenario/cache`.
-
-To go a step further and fully cache the test end-to-end, you can also wrap the LLM calls or any other non-deterministic functions in your application side with the `@scenario_cache` decorator:
-
-```python
-class MyAgent:
-    @scenario_cache(ignore=["self"])
-    def invoke(self, message, context):
-        return client.chat.completions.create(
-            # ...
-        )
-```
-
-This will cache any function call you decorate when running the tests and make them repeatable, hashed by the function arguments, the scenario being executed, and the `cache_key` you provided. You can exclude arguments that should not be hashed for the cache key by naming them in the `ignore` argument.
-
-## Disable Output
-
-You can remove the `-s` flag from pytest to hide the output during test, which will only show up if the test fails. Alternatively, you can set `verbose=False` in the `Scenario.configure` method or in the specific scenario you are running.
-
-## Running in parallel
-
-As the number of your scenarios grows, you might want to run them in parallel to speed up your whole test suite. We suggest you to use the [pytest-asyncio-concurrent](https://pypi.org/project/pytest-asyncio-concurrent/) plugin to do so.
-
-Simply install the plugin from the link above, then replace the `@pytest.mark.asyncio` annotation in the tests with `@pytest.mark.asyncio_concurrent`, adding a group name to it to mark the group of scenarions that should be run in parallel together, e.g.:
-
-```python
-@pytest.mark.agent_test
-@pytest.mark.asyncio_concurrent(group="vegetarian_recipe_agent")
-async def test_vegetarian_recipe_agent():
-    # ...
-
-@pytest.mark.agent_test
-@pytest.mark.asyncio_concurrent(group="vegetarian_recipe_agent")
-async def test_user_is_very_hungry():
-    # ...
-```
-
-Those two scenarios should now run in parallel.
-
-## License
-
-MIT License

langwatch_scenario-0.2.0.dist-info/RECORD

@@ -1,15 +0,0 @@
-scenario/__init__.py,sha256=LfCjOpbn55jYBBZHyMSZtRAWeCDFn4z4OhAyFnu8aMg,602
-scenario/cache.py,sha256=sYu16SAf-BnVYkWSlEDzpyynJGIQyNYsgMXPgCqEnmk,1719
-scenario/config.py,sha256=5UVBmuQDtni0Yu00bMh5p0xMGsrymYVRftXBGTsi2fI,802
-scenario/error_messages.py,sha256=ZMcAOKJmKaLIinMZ0yBIOgDhPfeJH0uZxIEmolRArtc,2344
-scenario/pytest_plugin.py,sha256=TzOHi8PN-dtDqaYAZkgT0wgBkhetOpYy--Z0pzi5PXM,5771
-scenario/result.py,sha256=y6mUu6X4H6YJYmwVD4VWHCBi-1BTlUVeYrTZ3HBA0oU,2382
-scenario/scenario.py,sha256=OTadwIHIcUhXxfUNnJXpT7h3GZ_VUL3XSd9k-oVPfMo,4069
-scenario/scenario_executor.py,sha256=phRKj7vZ_QjGUO9w05-DPrAzdacg_7CnTV59lYLCCKk,7912
-scenario/testing_agent.py,sha256=y4B8TMhKryeTiiv62qwslx7Gw_zw54Vk9zPyswEPm0k,10481
-scenario/utils.py,sha256=tMESosrxesA1B5zZB3IJ-sNSXDmnpNNib-DHobveVLA,3918
-langwatch_scenario-0.2.0.dist-info/METADATA,sha256=fc1oBg2ms-iVgYc44oSTJk-8sw2yOe_PpWEMStvYEX4,9339
-langwatch_scenario-0.2.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-langwatch_scenario-0.2.0.dist-info/entry_points.txt,sha256=WlEnJ_gku0i18bIa3DSuGqXRX-QDQLe_s0YmRzK45TI,45
-langwatch_scenario-0.2.0.dist-info/top_level.txt,sha256=45Mn28aedJsetnBMB5xSmrJ-yo701QLH89Zlz4r1clE,9
-langwatch_scenario-0.2.0.dist-info/RECORD,,

scenario/result.py

@@ -1,74 +0,0 @@
-"""
-Result module: defines the class for scenario test results.
-"""
-
-from dataclasses import dataclass, field
-from typing import List, Dict, Optional
-
-
-@dataclass
-class ScenarioResult:
-    """
-    Represents the results of a scenario test run.
-
-    Attributes:
-        success: Whether the scenario passed
-        conversation: The conversation history
-        reasoning: Reasoning for the result
-        passed_criteria: List of criteria that were met
-        failed_criteria: List of criteria that were not met
-    """
-
-    success: bool
-    conversation: List[Dict[str, str]]
-    reasoning: Optional[str] = None
-    passed_criteria: List[str] = field(default_factory=list)
-    failed_criteria: List[str] = field(default_factory=list)
-    total_time: Optional[float] = None
-    agent_time: Optional[float] = None
-
-    def __post_init__(self) -> None:
-        """Validate the result after initialization."""
-        if not self.success and not self.reasoning:
-            raise ValueError("Failed scenarios must have a reasoning")
-
-    @classmethod
-    def success_result(
-        cls,
-        conversation: List[Dict[str, str]],
-        reasoning: Optional[str],
-        passed_criteria: List[str],
-        total_time: Optional[float] = None,
-        agent_time: Optional[float] = None,
-    ) -> "ScenarioResult":
-        """Create a successful result."""
-        return cls(
-            success=True,
-            conversation=conversation,
-            reasoning=reasoning,
-            passed_criteria=passed_criteria,
-            failed_criteria=[],
-            total_time=total_time,
-            agent_time=agent_time,
-        )
-
-    @classmethod
-    def failure_result(
-        cls,
-        conversation: List[Dict[str, str]],
-        reasoning: str,
-        passed_criteria: Optional[List[str]] = None,
-        failed_criteria: Optional[List[str]] = None,
-        total_time: Optional[float] = None,
-        agent_time: Optional[float] = None,
-    ) -> "ScenarioResult":
-        """Create a failed result."""
-        return cls(
-            success=False,
-            conversation=conversation,
-            reasoning=reasoning,
-            passed_criteria=passed_criteria if passed_criteria is not None else [],
-            failed_criteria=failed_criteria if failed_criteria is not None else [],
-            total_time=total_time,
-            agent_time=agent_time,
-        )

scenario/scenario.py

@@ -1,123 +0,0 @@
-"""
-Scenario module: defines the core Scenario class for agent testing.
-"""
-
-from typing import Awaitable, List, Dict, Any, Optional, Callable, TypedDict, Union
-import asyncio
-import concurrent.futures
-from functools import partial
-
-from scenario.config import ScenarioConfig
-from scenario.scenario_executor import ScenarioExecutor
-
-from .result import ScenarioResult
-from .testing_agent import TestingAgent
-
-from openai.types.chat import ChatCompletionMessageParam
-
-
-class AgentResult(TypedDict, total=False):
-    message: str
-    messages: List[ChatCompletionMessageParam]
-    extra: Dict[str, Any]
-
-
-class Scenario(ScenarioConfig):
-    """
-    A scenario represents a specific testing case for an agent.
-
-    It includes:
-    - A description of the scenario
-    - Criteria to determine if the agent behaved correctly
-    - Optional additional parameters
-    """
-
-    name: str
-    description: str
-    agent: Union[
-        Callable[[str, Optional[Dict[str, Any]]], Dict[str, Any]],
-        Callable[[str, Optional[Dict[str, Any]]], Awaitable[Dict[str, Any]]],
-    ]
-    criteria: List[str]
-
-    def __init__(self, name: str, description: str, **kwargs):
-        """Validate scenario configuration after initialization."""
-
-        default_config = getattr(Scenario, "default_config", None)
-        if default_config:
-            kwargs = {**default_config.model_dump(), **kwargs}
-
-        if not name:
-            raise ValueError("Scenario name cannot be empty")
-        kwargs["name"] = name
-
-        if not description:
-            raise ValueError("Scenario description cannot be empty")
-        kwargs["description"] = description
-
-        # TODO: allow not having any criteria, for scripted scenarios
-        if not kwargs.get("criteria"):
-            raise ValueError("Scenario must have at least one criteria")
-
-        if kwargs.get("max_turns", 0) < 1:
-            raise ValueError("max_turns must be a positive integer")
-
-        # Ensure agent is callable
-        if not callable(kwargs.get("agent")):
-            raise ValueError("Agent must be a callable function")
-
-        super().__init__(**kwargs)
-
-    async def run(self, context: Optional[Dict[str, Any]] = None) -> ScenarioResult:
-        """
-        Run the scenario against the agent under test.
-
-        Args:
-            context: Optional initial context for the agent
-
-        Returns:
-            ScenarioResult containing the test outcome
-        """
-
-        # We'll use a thread pool to run the execution logic, we
-        # require a separate thread because even though asyncio is
-        # being used throughout, any user code on the callback can
-        # be blocking, preventing them from running scenarios in parallel
-        with concurrent.futures.ThreadPoolExecutor() as executor:
-
-            def run_in_thread():
-                loop = asyncio.new_event_loop()
-                asyncio.set_event_loop(loop)
-
-                try:
-                    return loop.run_until_complete(ScenarioExecutor(self).run(context))
-                finally:
-                    loop.close()
-
-            # Run the function in the thread pool and await its result
-            # This converts the thread's execution into a Future that the current
-            # event loop can await without blocking
-            loop = asyncio.get_event_loop()
-            result = await loop.run_in_executor(executor, run_in_thread)
-            return result
-
-    @classmethod
-    def configure(
-        cls,
-        testing_agent: Optional[TestingAgent] = None,
-        max_turns: Optional[int] = None,
-        verbose: Optional[Union[bool, int]] = None,
-        cache_key: Optional[str] = None,
-        debug: Optional[bool] = None,
-    ) -> None:
-        existing_config = getattr(cls, "default_config", ScenarioConfig())
-
-        cls.default_config = existing_config.merge(
-            ScenarioConfig(
-                testing_agent=testing_agent,
-                max_turns=max_turns,
-                verbose=verbose,
-                cache_key=cache_key,
-                debug=debug,
-            )
-        )