langwatch-scenario 0.2.0__tar.gz → 0.3.0__tar.gz

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langwatch-scenario
-Version: 0.2.0
+Version: 0.3.0
 Summary: The end-to-end agent testing library
 Author-email: LangWatch Team <support@langwatch.ai>
 License: MIT
@@ -25,11 +25,13 @@ Requires-Dist: joblib>=1.4.2
 Requires-Dist: wrapt>=1.17.2
 Requires-Dist: pytest-asyncio>=0.26.0
 Requires-Dist: rich<15.0.0,>=13.3.3
+Requires-Dist: pksuid>=1.1.2
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: isort; extra == "dev"
-Requires-Dist: mypy; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: commitizen; extra == "dev"
 
 ![scenario](https://github.com/langwatch/scenario/raw/main/assets/scenario-wide.webp)
 
@@ -39,9 +41,9 @@ Requires-Dist: pytest-cov; extra == "dev"
 
 # Scenario: Use an Agent to test your Agent
 
-Scenario is a library for testing agents end-to-end as a human would, but without having to manually do it. The automated testing agent covers every single scenario for you.
+Scenario is an Agent Testing Framework for testing AI agents through Simulation Testing.
 
-You define the scenarios, and the testing agent will simulate your users as it follows them, it will keep chatting and evaluating your agent until it reaches the desired goal or detects an unexpected behavior.
+You define the scenarios, and the testing agent will simulate a real user as it follows them, it will keep chatting back and forth with _your_ agent to play out the simulation, until it reaches the desired goal or detects an unexpected behavior based on the criteria you defined.
 
 [📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
 
@@ -63,20 +65,23 @@ Now create your first scenario and save it as `tests/test_vegetarian_recipe_agen
 ```python
 import pytest
 
-from scenario import Scenario, TestingAgent, scenario_cache
+from scenario import Scenario, TestingAgent, ScenarioAgentAdapter, AgentInput, AgentReturnTypes, scenario_cache
 
 Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
 
 
+# Create an adapter to call your agent
+class VegetarianRecipeAgentAdapter(ScenarioAgentAdapter):
+    def __init__(self, input: AgentInput):
+        self.agent = VegetarianRecipeAgent()
+
+    async def call(self, input: AgentInput) -> AgentReturnTypes:
+        return self.agent.run(input.last_new_user_message_str())
+
+
 @pytest.mark.agent_test
 @pytest.mark.asyncio
 async def test_vegetarian_recipe_agent():
-    agent = VegetarianRecipeAgent()
-
-    def vegetarian_recipe_agent(message, context):
-        # Call your agent here
-        return agent.run(message)
-
     # Define the simulated scenario
     scenario = Scenario(
         name="dinner idea",
@@ -133,7 +138,7 @@ class VegetarianRecipeAgent:
         message = response.choices[0].message  # type: ignore
         self.history.append(message)
 
-        return {"messages": [message]}
+        return [message]
 
 ```
 
@@ -186,6 +191,49 @@ result = await scenario.run()
 
 You can find a fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
 
+## Specify a script for guiding the scenario
+
+You can specify a script for guiding the scenario by passing a list of steps to the `script` field.
+
+```python
+@pytest.mark.agent_test
+@pytest.mark.asyncio
+async def test_ai_assistant_agent():
+    scenario = Scenario(
+        name="false assumptions",
+        description="""
+            The agent makes false assumption about being an ATM bank, and user corrects it
+        """,
+        agent=AiAssistantAgentAdapter,
+        criteria=[
+            "user should get good recommendations on river crossing",
+            "agent should NOT follow up about ATM recommendation after user has corrected them they are just hiking",
+        ],
+        max_turns=5,
+    )
+
+    def check_if_tool_was_called(state: ScenarioExecutor) -> None:
+        assert state.has_tool_call("web_search")
+
+    result = await scenario.script(
+        [
+            # Define existing history of messages
+            scenario.user("how do I safely approach a bank?"),
+            # Or let it be generate automatically
+            scenario.agent(),
+            # Add custom assertions, for example making sure a tool was called
+            check_if_tool_was_called,
+            scenario.user(),
+            # Let the simulation proceed for 2 more turns
+            scenario.proceed(turns=2),
+            # Time to make a judgment call
+            scenario.judge(),
+        ]
+    ).run()
+
+    assert result.success
+```
+
 ## Debug mode
 
 You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/README.md

@@ -6,9 +6,9 @@
 
 # Scenario: Use an Agent to test your Agent
 
-Scenario is a library for testing agents end-to-end as a human would, but without having to manually do it. The automated testing agent covers every single scenario for you.
+Scenario is an Agent Testing Framework for testing AI agents through Simulation Testing.
 
-You define the scenarios, and the testing agent will simulate your users as it follows them, it will keep chatting and evaluating your agent until it reaches the desired goal or detects an unexpected behavior.
+You define the scenarios, and the testing agent will simulate a real user as it follows them, it will keep chatting back and forth with _your_ agent to play out the simulation, until it reaches the desired goal or detects an unexpected behavior based on the criteria you defined.
 
 [📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
 
@@ -30,20 +30,23 @@ Now create your first scenario and save it as `tests/test_vegetarian_recipe_agen
 ```python
 import pytest
 
-from scenario import Scenario, TestingAgent, scenario_cache
+from scenario import Scenario, TestingAgent, ScenarioAgentAdapter, AgentInput, AgentReturnTypes, scenario_cache
 
 Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
 
 
+# Create an adapter to call your agent
+class VegetarianRecipeAgentAdapter(ScenarioAgentAdapter):
+    def __init__(self, input: AgentInput):
+        self.agent = VegetarianRecipeAgent()
+
+    async def call(self, input: AgentInput) -> AgentReturnTypes:
+        return self.agent.run(input.last_new_user_message_str())
+
+
 @pytest.mark.agent_test
 @pytest.mark.asyncio
 async def test_vegetarian_recipe_agent():
-    agent = VegetarianRecipeAgent()
-
-    def vegetarian_recipe_agent(message, context):
-        # Call your agent here
-        return agent.run(message)
-
     # Define the simulated scenario
     scenario = Scenario(
         name="dinner idea",
@@ -100,7 +103,7 @@ class VegetarianRecipeAgent:
         message = response.choices[0].message  # type: ignore
         self.history.append(message)
 
-        return {"messages": [message]}
+        return [message]
 
 ```
 
@@ -153,6 +156,49 @@ result = await scenario.run()
 
 You can find a fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
 
+## Specify a script for guiding the scenario
+
+You can specify a script for guiding the scenario by passing a list of steps to the `script` field.
+
+```python
+@pytest.mark.agent_test
+@pytest.mark.asyncio
+async def test_ai_assistant_agent():
+    scenario = Scenario(
+        name="false assumptions",
+        description="""
+            The agent makes false assumption about being an ATM bank, and user corrects it
+        """,
+        agent=AiAssistantAgentAdapter,
+        criteria=[
+            "user should get good recommendations on river crossing",
+            "agent should NOT follow up about ATM recommendation after user has corrected them they are just hiking",
+        ],
+        max_turns=5,
+    )
+
+    def check_if_tool_was_called(state: ScenarioExecutor) -> None:
+        assert state.has_tool_call("web_search")
+
+    result = await scenario.script(
+        [
+            # Define existing history of messages
+            scenario.user("how do I safely approach a bank?"),
+            # Or let it be generate automatically
+            scenario.agent(),
+            # Add custom assertions, for example making sure a tool was called
+            check_if_tool_was_called,
+            scenario.user(),
+            # Let the simulation proceed for 2 more turns
+            scenario.proceed(turns=2),
+            # Time to make a judgment call
+            scenario.judge(),
+        ]
+    ).run()
+
+    assert result.success
+```
+
 ## Debug mode
 
 You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/langwatch_scenario.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langwatch-scenario
-Version: 0.2.0
+Version: 0.3.0
 Summary: The end-to-end agent testing library
 Author-email: LangWatch Team <support@langwatch.ai>
 License: MIT
@@ -25,11 +25,13 @@ Requires-Dist: joblib>=1.4.2
 Requires-Dist: wrapt>=1.17.2
 Requires-Dist: pytest-asyncio>=0.26.0
 Requires-Dist: rich<15.0.0,>=13.3.3
+Requires-Dist: pksuid>=1.1.2
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: isort; extra == "dev"
-Requires-Dist: mypy; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: commitizen; extra == "dev"
 
 ![scenario](https://github.com/langwatch/scenario/raw/main/assets/scenario-wide.webp)
 
@@ -39,9 +41,9 @@ Requires-Dist: pytest-cov; extra == "dev"
 
 # Scenario: Use an Agent to test your Agent
 
-Scenario is a library for testing agents end-to-end as a human would, but without having to manually do it. The automated testing agent covers every single scenario for you.
+Scenario is an Agent Testing Framework for testing AI agents through Simulation Testing.
 
-You define the scenarios, and the testing agent will simulate your users as it follows them, it will keep chatting and evaluating your agent until it reaches the desired goal or detects an unexpected behavior.
+You define the scenarios, and the testing agent will simulate a real user as it follows them, it will keep chatting back and forth with _your_ agent to play out the simulation, until it reaches the desired goal or detects an unexpected behavior based on the criteria you defined.
 
 [📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
 
@@ -63,20 +65,23 @@ Now create your first scenario and save it as `tests/test_vegetarian_recipe_agen
 ```python
 import pytest
 
-from scenario import Scenario, TestingAgent, scenario_cache
+from scenario import Scenario, TestingAgent, ScenarioAgentAdapter, AgentInput, AgentReturnTypes, scenario_cache
 
 Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
 
 
+# Create an adapter to call your agent
+class VegetarianRecipeAgentAdapter(ScenarioAgentAdapter):
+    def __init__(self, input: AgentInput):
+        self.agent = VegetarianRecipeAgent()
+
+    async def call(self, input: AgentInput) -> AgentReturnTypes:
+        return self.agent.run(input.last_new_user_message_str())
+
+
 @pytest.mark.agent_test
 @pytest.mark.asyncio
 async def test_vegetarian_recipe_agent():
-    agent = VegetarianRecipeAgent()
-
-    def vegetarian_recipe_agent(message, context):
-        # Call your agent here
-        return agent.run(message)
-
     # Define the simulated scenario
     scenario = Scenario(
         name="dinner idea",
@@ -133,7 +138,7 @@ class VegetarianRecipeAgent:
         message = response.choices[0].message  # type: ignore
         self.history.append(message)
 
-        return {"messages": [message]}
+        return [message]
 
 ```
 
@@ -186,6 +191,49 @@ result = await scenario.run()
 
 You can find a fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
 
+## Specify a script for guiding the scenario
+
+You can specify a script for guiding the scenario by passing a list of steps to the `script` field.
+
+```python
+@pytest.mark.agent_test
+@pytest.mark.asyncio
+async def test_ai_assistant_agent():
+    scenario = Scenario(
+        name="false assumptions",
+        description="""
+            The agent makes false assumption about being an ATM bank, and user corrects it
+        """,
+        agent=AiAssistantAgentAdapter,
+        criteria=[
+            "user should get good recommendations on river crossing",
+            "agent should NOT follow up about ATM recommendation after user has corrected them they are just hiking",
+        ],
+        max_turns=5,
+    )
+
+    def check_if_tool_was_called(state: ScenarioExecutor) -> None:
+        assert state.has_tool_call("web_search")
+
+    result = await scenario.script(
+        [
+            # Define existing history of messages
+            scenario.user("how do I safely approach a bank?"),
+            # Or let it be generate automatically
+            scenario.agent(),
+            # Add custom assertions, for example making sure a tool was called
+            check_if_tool_was_called,
+            scenario.user(),
+            # Let the simulation proceed for 2 more turns
+            scenario.proceed(turns=2),
+            # Time to make a judgment call
+            scenario.judge(),
+        ]
+    ).run()
+
+    assert result.success
+```
+
 ## Debug mode
 
 You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/langwatch_scenario.egg-info/SOURCES.txt

@@ -12,8 +12,12 @@ scenario/cache.py
 scenario/config.py
 scenario/error_messages.py
 scenario/pytest_plugin.py
-scenario/result.py
 scenario/scenario.py
+scenario/scenario_agent_adapter.py
 scenario/scenario_executor.py
 scenario/testing_agent.py
-scenario/utils.py
+scenario/types.py
+scenario/utils.py
+tests/test_scenario.py
+tests/test_scenario_agent.py
+tests/test_scenario_executor.py

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/langwatch_scenario.egg-info/requires.txt

@@ -7,9 +7,11 @@ joblib>=1.4.2
 wrapt>=1.17.2
 pytest-asyncio>=0.26.0
 rich<15.0.0,>=13.3.3
+pksuid>=1.1.2
 
 [dev]
 black
 isort
-mypy
 pytest-cov
+pre-commit
+commitizen

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/pyproject.toml

@@ -4,13 +4,11 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "langwatch-scenario"
-version = "0.2.0"
+version = "0.3.0"
 description = "The end-to-end agent testing library"
 readme = "README.md"
-authors = [
-    {name = "LangWatch Team", email = "support@langwatch.ai"}
-]
-license = {text = "MIT"}
+authors = [{ name = "LangWatch Team", email = "support@langwatch.ai" }]
+license = { text = "MIT" }
 requires-python = ">=3.9"
 classifiers = [
     "Development Status :: 4 - Beta",
@@ -32,14 +30,16 @@ dependencies = [
     "wrapt>=1.17.2",
     "pytest-asyncio>=0.26.0",
     "rich>=13.3.3,<15.0.0",
+    "pksuid>=1.1.2",
 ]
 
 [project.optional-dependencies]
 dev = [
     "black",
     "isort",
-    "mypy",
     "pytest-cov",
+    "pre-commit",
+    "commitizen",
 ]
 
 [project.urls]
@@ -47,12 +47,21 @@ dev = [
 "Bug Tracker" = "https://github.com/langwatch/scenario/issues"
 
 [tool.pytest.ini_options]
-markers = [
-    "agent_test: marks tests as agent scenario tests",
-]
+markers = ["agent_test: marks tests as agent scenario tests"]
 
 [dependency-groups]
 dev = [
+    "commitizen>=4.8.3",
+    "pre-commit>=4.2.0",
     "pydantic-ai>=0.0.52",
+    "pyright>=1.1.401",
     "pytest-asyncio-concurrent>=0.4.1",
 ]
+
+[tool.commitizen]
+name = "cz_conventional_commits"
+version = "0.2.0"
+tag_format = "v$version"
+version_files = ["pyproject.toml:version"]
+bump_message = "bump: version $current_version → $new_version"
+major_version_zero = true

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/scenario/__init__.py

@@ -3,10 +3,11 @@ Scenario: A testing library for conversational agents.
 """
 
 # First import non-dependent modules
-from .result import ScenarioResult
+from .types import ScenarioResult, AgentInput, ScenarioAgentRole, AgentReturnTypes
 from .config import ScenarioConfig
 
 # Then import modules with dependencies
+from .scenario_agent_adapter import ScenarioAgentAdapter
 from .testing_agent import TestingAgent
 from .scenario import Scenario
 from .cache import scenario_cache
@@ -15,10 +16,19 @@ from .cache import scenario_cache
 from .pytest_plugin import pytest_configure, scenario_reporter
 
 __all__ = [
-    "Scenario",
-    "TestingAgent",
+    # Types
     "ScenarioResult",
+    "AgentInput",
+    "ScenarioAgentRole",
     "ScenarioConfig",
+    "AgentReturnTypes",
+
+    # Classes
+    "Scenario",
+    "ScenarioAgentAdapter",
+    "TestingAgent",
+
+    # Plugins
     "pytest_configure",
     "scenario_reporter",
     "scenario_cache",

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/scenario/config.py

@@ -2,10 +2,16 @@
 Configuration module for Scenario.
 """
 
-from typing import Optional, Union
+from typing import TYPE_CHECKING, Any, Optional, Type, Union
 from pydantic import BaseModel
 
-from scenario.testing_agent import TestingAgent
+if TYPE_CHECKING:
+    from scenario.scenario_agent_adapter import ScenarioAgentAdapter
+
+    ScenarioAgentType = ScenarioAgentAdapter
+else:
+    ScenarioAgentType = Any
+
 
 class ScenarioConfig(BaseModel):
     """
@@ -15,14 +21,19 @@ class ScenarioConfig(BaseModel):
     such as the LLM provider and model to use for the testing agent.
     """
 
-    testing_agent: Optional[TestingAgent] = None
+    testing_agent: Optional[Type[ScenarioAgentType]] = None
     max_turns: Optional[int] = 10
     verbose: Optional[Union[bool, int]] = True
     cache_key: Optional[str] = None
     debug: Optional[bool] = False
 
     def merge(self, other: "ScenarioConfig") -> "ScenarioConfig":
-        return ScenarioConfig(**{
-            **self.model_dump(),
-            **other.model_dump(exclude_none=True),
-        })
+        return ScenarioConfig(
+            **{
+                **self.items(),
+                **other.items(),
+            }
+        )
+
+    def items(self):
+        return {k: getattr(self, k) for k in self.model_dump(exclude_none=True).keys()}

langwatch_scenario-0.3.0/scenario/error_messages.py

@@ -0,0 +1,134 @@
+from textwrap import indent
+from typing import Any
+import termcolor
+
+
+default_config_error_message = f"""
+
+ {termcolor.colored("->", "cyan")} Please set a default config with at least a testing_agent model for running your scenarios at the top of your test file, for example:
+
+    from scenario import Scenario, TestingAgent
+
+    Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))
+    {termcolor.colored("^" * 74, "green")}
+
+    @pytest.mark.agent_test
+    def test_vegetarian_recipe_agent():
+        scenario = Scenario(
+            # ...
+        )
+        result = scenario.run()
+
+        assert result.success
+
+
+ {termcolor.colored("->", "cyan")} Alternatively, you can set the config specifically for this scenario:
+
+    from scenario import Scenario, TestingAgent
+
+    @pytest.mark.agent_test
+    def test_vegetarian_recipe_agent():
+        scenario = Scenario(
+            # ...
+            testing_agent=TestingAgent(model="openai/gpt-4o-mini")
+            {termcolor.colored("^" * 54, "green")}
+        )
+        result = scenario.run()
+
+        assert result.success
+"""
+
+
+testing_agent_not_configured_error_message = f"""
+
+ {termcolor.colored("->", "cyan")} Testing agent was initialized without a model, please set the model when defining the testing agent, for example:
+
+    TestingAgent.with_config(model="openai/gpt-4.1-mini")
+    {termcolor.colored("^" * 53, "green")}
+"""
+
+
+def message_return_error_message(got: Any, class_name: str):
+    got_ = repr(got)
+    if len(got_) > 100:
+        got_ = got_[:100] + "..."
+
+    return f"""
+ {termcolor.colored("->", "cyan")} On the {termcolor.colored("call", "green")} method of the {class_name} agent adapter, you returned:
+
+{indent(got_, ' ' * 4)}
+
+ {termcolor.colored("->", "cyan")} But the adapter should return either a string, a dict on the OpenAI messages format, or a list of messages in the OpenAI messages format so the testing agent can understand what happened. For example:
+
+    class MyAgentAdapter(ScenarioAgentAdapter):
+        async def call(self, input: AgentInput) -> AgentReturnTypes:
+            response = call_my_agent(message)
+
+            return response.output_text
+            {termcolor.colored("^" * 27, "green")}
+
+ {termcolor.colored("->", "cyan")} Alternatively, you can return a list of messages in OpenAI messages format, this is useful for capturing tool calls and other before the final response:
+
+    class MyAgentAdapter(ScenarioAgentAdapter):
+        async def call(self, input: AgentInput) -> AgentReturnTypes:
+            response = call_my_agent(message)
+
+            return [
+                {{"role": "assistant", "content": response.output_text}},
+                {termcolor.colored("^" * 55, "green")}
+            ]
+"""
+
+
+def message_invalid_agent_type(got: Any):
+    got_ = repr(got)
+    if len(got_) > 100:
+        got_ = got_[:100] + "..."
+
+    return f"""
+ {termcolor.colored("->", "cyan")} The {termcolor.colored("agent", "green")} argument of Scenario needs to receive a class that inherits from {termcolor.colored("ScenarioAgentAdapter", "green")}, but you passed:
+
+{indent(got_, ' ' * 4)}
+
+ {termcolor.colored("->", "cyan")} Instead, wrap your agent in a ScenarioAgentAdapter subclass. For example:
+
+    class MyAgentAdapter(ScenarioAgentAdapter):
+    {termcolor.colored("^" * 43, "green")}
+        async def call(self, input: AgentInput) -> AgentReturnTypes:
+            response = call_my_agent(message)
+
+            return response.output_text
+
+ {termcolor.colored("->", "cyan")} And then you can use that on your scenario definition:
+
+    @pytest.mark.agent_test
+    def test_my_agent():
+        scenario = Scenario(
+            name="first scenario",
+            description=\"\"\"
+                Example scenario description to test your agent.
+            \"\"\",
+            agent=MyAgentAdapter,
+            {termcolor.colored("^" * 20, "green")}
+            criteria=[
+                "Requirement One",
+                "Requirement Two",
+            ],
+        )
+        result = scenario.run()
+
+        assert result.success
+"""
+
+
+def agent_response_not_awaitable(class_name: str):
+    return f"""
+ {termcolor.colored("->", "cyan")} The {termcolor.colored("call", "green")} method of the {class_name} agent adapter returned a non-awaitable response, you probably forgot to add the {termcolor.colored("async", "green")} keyword to the method definition, make sure your code looks like this:
+
+    class {class_name}(ScenarioAgentAdapter):
+        async def call(self, input: AgentInput) -> AgentReturnTypes:
+        {termcolor.colored("^" * 5, "green")}
+            response = call_my_agent(message)
+
+            return response.output_text
+"""

{langwatch_scenario-0.2.0 → langwatch_scenario-0.3.0}/scenario/pytest_plugin.py

@@ -7,7 +7,7 @@ from typing import TypedDict
 import functools
 from termcolor import colored
 
-from scenario.result import ScenarioResult
+from scenario.types import ScenarioResult
 
 from .scenario import Scenario