langwatch-scenario 0.4.0__py3-none-any.whl → 0.6.0__py3-none-any.whl

{langwatch_scenario-0.4.0.dist-info → langwatch_scenario-0.6.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langwatch-scenario
-Version: 0.4.0
+Version: 0.6.0
 Summary: The end-to-end agent testing library
 Author-email: LangWatch Team <support@langwatch.ai>
 License: MIT
@@ -26,6 +26,11 @@ Requires-Dist: wrapt>=1.17.2
 Requires-Dist: pytest-asyncio>=0.26.0
 Requires-Dist: rich<15.0.0,>=13.3.3
 Requires-Dist: pksuid>=1.1.2
+Requires-Dist: pdoc3>=0.11.6
+Requires-Dist: ag-ui-protocol>=0.1.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: rx>=3.2.0
+Requires-Dist: respx>=0.22.0
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: isort; extra == "dev"
@@ -44,65 +49,59 @@ Requires-Dist: function-schema; extra == "dev"
 
 # Scenario
 
-Scenario is an Agent Testing Framework for testing AI agents through Simulation Testing.
+Scenario is an Agent Testing Framework based on simulations, it can:
 
-You define the conversation scenario and let it play out, it will keep chatting back and forth with _your_ agent until it reaches the desired goal or detects an unexpected behavior based on the criteria you defined.
-
-- Test your agents end-to-end conversations with specified scenarios to capture both happy paths and edge cases
-- Full flexibility of how much you want to guide the conversation, from fully scripted scenarios to completely automated simulations
-- Run evaluations at any point of the conversation, designed for multi-turn
-- Works in combination with any testing and LLM evaluation frameworks, completely agnostic
-- Works with any LLM and Agent Framework, easy integration
+- Test real agent behavior by simulating users in different scenarios and edge cases
+- Evaluate and judge at any point of the conversation, powerful multi-turn control
+- Combine it with any LLM eval framework or custom evals, agnostic by design
+- Integrate your Agent by implementing just one `call()` method
+- Available in Python, TypeScript and Go
 
 [📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
 
-### See also
+### In other languages
 
 - [Scenario TypeScript](https://github.com/langwatch/scenario-ts/)
 - [Scenario Go](https://github.com/langwatch/scenario-go/)
 
 ## Example
 
+This is how a simple simulation with tool check looks like with Scenario:
+
 ```python
-@pytest.mark.agent_test
-@pytest.mark.asyncio
-async def test_weather_agent():
-    # Integrate with your agent
-    class WeatherAgent(scenario.AgentAdapter):
-        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
-            return weather_agent(input.messages)
+# Define any custom assertions
+def check_for_weather_tool_call(state: scenario.ScenarioState):
+    assert state.has_tool_call("get_current_weather")
 
-    # Define any custom assertions
-    def check_for_weather_tool_call(state: scenario.ScenarioState):
-        assert state.has_tool_call("get_current_weather")
+result = await scenario.run(
+    name="checking the weather",
 
-    # Run the scenario
-    result = await scenario.run(
-        name="checking the weather",
-        description="""
-            The user is planning a boat trip from Barcelona to Rome,
-            and is wondering what the weather will be like.
-        """,
-        agents=[
-            WeatherAgent(),
-            scenario.UserSimulatorAgent(model="openai/gpt-4.1-mini"),
-        ],
-        script=[
-            scenario.user(),
-            scenario.agent(),
-            check_for_weather_tool_call, # check for tool call after the first agent response
-            scenario.succeed(),
-        ],
-    )
+    # Define the prompt to guide the simulation
+    description="""
+        The user is planning a boat trip from Barcelona to Rome,
+        and is wondering what the weather will be like.
+    """,
 
-    # Assert the simulation was successful
-    assert result.success
+    # Define the agents that will play this simulation
+    agents=[
+        WeatherAgent(),
+        scenario.UserSimulatorAgent(model="openai/gpt-4.1-mini"),
+    ],
+
+    # (Optional) Control the simulation
+    script=[
+        scenario.user(), # let the user simulator generate a user message
+        scenario.agent(), # agent responds
+        check_for_weather_tool_call, # check for tool call after the first agent response
+        scenario.succeed(), # simulation ends successfully
+    ],
+)
+
+assert result.success
 ```
 
 > [!NOTE]
-> This is a very basic example, keep reading to see how to run a simulation completely script-free, using a Judge Agent to evaluate in real-time.
-
-Check out more examples in the [examples folder](./examples/).
+> Check out full examples in the [examples folder](./examples/).
 
 ## Getting Started
 
@@ -193,17 +192,17 @@ pytest -s tests/test_vegetarian_recipe_agent.py
 
 This is how it will look like:
 
-[![asciicast](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
+[![asciicast](./assets/ascii-cinema.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
 
 You can find the same code example in [examples/test_vegetarian_recipe_agent.py](examples/test_vegetarian_recipe_agent.py).
 
-## Script-free Simulation
+## Simulation on Autopilot
 
-By providing a User Simulator Agent and a description of the Scenario, the simulated user will automatically generate messages to the agent until the scenario is successful or the maximum number of turns is reached.
+By providing a User Simulator Agent and a description of the Scenario without a script, the simulated user will automatically generate messages to the agent until the scenario is successful or the maximum number of turns is reached.
 
 You can then use a Judge Agent to evaluate the scenario in real-time given certain criteria, at every turn, the Judge Agent will decide if it should let the simulation proceed or end it with a verdict.
 
-You can combine it with a script, to control for example the beginning of the conversation, or simply let it run scriptless, this is very useful to test an open case like a vibe coding assistant:
+For example, here is a scenario that tests a vibe coding assistant:
 
 ```python
 result = await scenario.run(
@@ -233,6 +232,8 @@ result = await scenario.run(
 
 Check out the fully working Lovable Clone example in [examples/test_lovable_clone.py](examples/test_lovable_clone.py).
 
+You can also combine it with a partial script too! By for example controlling only the beginning of the conversation, and let the rest proceed on autopilot, see the next section.
+
 ## Full Control of the Conversation
 
 You can specify a script for guiding the scenario by passing a list of steps to the `script` field, those steps are simply arbitrary functions that take the current state of the scenario as an argument, so you can do things like:
@@ -250,35 +251,35 @@ Everything is possible, using the same simple structure:
 ```python
 @pytest.mark.agent_test
 @pytest.mark.asyncio
-async def test_ai_assistant_agent():
-    scenario = Scenario(
-        name="false assumptions",
+async def test_early_assumption_bias():
+    result = await scenario.run(
+        name="early assumption bias",
         description="""
             The agent makes false assumption that the user is talking about an ATM bank, and user corrects it that they actually mean river banks
         """,
-        agent=AiAssistantAgentAdapter,
-        criteria=[
-            "user should get good recommendations on river crossing",
-            "agent should NOT follow up about ATM recommendation after user has corrected them they are just hiking",
+        agents=[
+            Agent(),
+            scenario.UserSimulatorAgent(),
+            scenario.JudgeAgent(
+                criteria=[
+                    "user should get good recommendations on river crossing",
+                    "agent should NOT keep following up about ATM recommendation after user has corrected them that they are actually just hiking",
+                ],
+            ),
         ],
-        max_turns=5,
-    )
-
-    def check_if_tool_was_called(state: ScenarioExecutor) -> None:
-        assert state.has_tool_call("web_search")
-
-    result = await scenario.script(
-        [
-            # Define existing history of messages
+        max_turns=10,
+        script=[
+            # Define hardcoded messages
+            scenario.agent("Hello, how can I help you today?"),
             scenario.user("how do I safely approach a bank?"),
 
-            # Or let it be generate automatically
+            # Or let it be generated automatically
             scenario.agent(),
 
             # Add custom assertions, for example making sure a tool was called
             check_if_tool_was_called,
 
-            # Another user message
+            # Generate a user follow-up message
             scenario.user(),
 
             # Let the simulation proceed for 2 more turns, print at every turn
@@ -289,8 +290,8 @@ async def test_ai_assistant_agent():
 
             # Time to make a judgment call
             scenario.judge(),
-        ]
-    ).run()
+        ],
+    )
 
     assert result.success
 ```
@@ -302,7 +303,7 @@ You can enable debug mode by setting the `debug` field to `True` in the `Scenari
 Debug mode allows you to see the messages in slow motion step by step, and intervene with your own inputs to debug your agent from the middle of the conversation.
 
 ```python
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), debug=True)
+scenario.configure(default_model="openai/gpt-4.1-mini", debug=True)
 ```
 
 or
@@ -316,16 +317,17 @@ pytest -s tests/test_vegetarian_recipe_agent.py --debug
 Each time the scenario runs, the testing agent might chose a different input to start, this is good to make sure it covers the variance of real users as well, however we understand that the non-deterministic nature of it might make it less repeatable, costly and harder to debug. To solve for it, you can use the `cache_key` field in the `Scenario.configure` method or in the specific scenario you are running, this will make the testing agent give the same input for given the same scenario:
 
 ```python
-Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"), cache_key="42")
+scenario.configure(default_model="openai/gpt-4.1-mini", cache_key="42")
 ```
 
 To bust the cache, you can simply pass a different `cache_key`, disable it, or delete the cache files located at `~/.scenario/cache`.
 
-To go a step further and fully cache the test end-to-end, you can also wrap the LLM calls or any other non-deterministic functions in your application side with the `@scenario_cache` decorator:
+To go a step further and fully cache the test end-to-end, you can also wrap the LLM calls or any other non-deterministic functions in your application side with the `@scenario.cache` decorator:
 
 ```python
+# Inside your actual agent implementation
 class MyAgent:
-    @scenario_cache(ignore=["self"])
+    @scenario.cache()
     def invoke(self, message, context):
         return client.chat.completions.create(
             # ...
@@ -358,6 +360,26 @@ async def test_user_is_very_hungry():
 
 Those two scenarios should now run in parallel.
 
+## Events System
+
+Scenario automatically publishes events during execution for monitoring and observability. You can enable event reporting by setting environment variables:
+
+```bash
+# Enable automatic event reporting
+export LANGWATCH_ENDPOINT="https://api.langwatch.ai"
+export LANGWATCH_API_KEY="your-api-key"
+```
+
+With these variables set, Scenario will automatically:
+
+- Publish events when scenarios start, finish, and when messages are added
+- Handle retries and error handling automatically
+- Process events asynchronously without blocking your tests
+
+The events include timing information, conversation history, and success/failure metrics for analysis.
+
+For advanced customization, see the event classes in the codebase for detailed documentation.
+
 ## License
 
 MIT License

langwatch_scenario-0.6.0.dist-info/RECORD

@@ -0,0 +1,27 @@
+scenario/__init__.py,sha256=UJ5l-sG4TMG0wR8Ba-dxdDW36m3apTvawP-lNvk7Jm0,4293
+scenario/_error_messages.py,sha256=6lEx3jBGMbPx0kG0eX5zoZE-ENVM3O_ZkIbVMlnidYs,3892
+scenario/agent_adapter.py,sha256=PoY2KQqYuqzIIb3-nhIU-MPXwHJc1vmwdweMy7ut-hk,4255
+scenario/cache.py,sha256=J6s6Sia_Ce6TrnsInlhfxm6SF8tygo3sH-_cQCRX1WA,6213
+scenario/config.py,sha256=xhUuXH-sThwPTmJNSuajKxX-WC_tcFwJ1jZc119DswA,6093
+scenario/judge_agent.py,sha256=9CCO699qoWqXvWdQ73Yc3dqPOwaJdJ-zqxVaLaKi_cA,16161
+scenario/pytest_plugin.py,sha256=f2ETBpATz80k7K87M6046ZIFiQpHEvDN7dxakd3y2wk,11321
+scenario/scenario_executor.py,sha256=nkSIuIlwPHfr6pueSBbARrgiqPtW0SxajV3PFypAnJ4,34508
+scenario/scenario_state.py,sha256=dQDjazem-dn1c5mw6TwngEu6Tv_cHwEzemepsPBy2f0,7039
+scenario/script.py,sha256=A0N5pP0l4FFn1xdKc78U_wkwWhEWH3EFeU_LRDtNyEI,12241
+scenario/types.py,sha256=BhXcTEMGyGg_1QysN-GXVjm8DP2VH3UEzj_qvoglp2k,9466
+scenario/user_simulator_agent.py,sha256=fhwi8W44s343BGrjJXSJw960wcK7MgwTg-epxR1bqHo,9088
+scenario/_utils/__init__.py,sha256=wNX9hU8vzYlyLDwjkt7JUW3IPo2DhME6UIt_zvLM3B0,1000
+scenario/_utils/ids.py,sha256=K1iPuJgPh3gX9HCrDZGqK5lDgdwZXfOBF1YXVOWNHRg,1843
+scenario/_utils/message_conversion.py,sha256=AM9DLyWpy97CrAH8RmId9Mv2rmLquQhFoUpRyp-jVeY,3622
+scenario/_utils/utils.py,sha256=msQgUWaLh3U9jIIHmxkEbOaklga63AF0KJzsaKa_mZc,14008
+scenario/events/__init__.py,sha256=_autF1cMZYpNXE-kJNvvRb-H_hYqy4gOSSp2fT3Wi9k,1533
+scenario/events/event_bus.py,sha256=MThIMIaI2nj2CoegZazTNxeHbtl4_M7bW3vEAHz6R8g,7102
+scenario/events/event_reporter.py,sha256=cMh_5jA5hG3Q9IsoAgPJhxnIVs_M1Q0e2lgLTEK4oPc,3100
+scenario/events/events.py,sha256=jPXylwiADb0Bdk7u1YkAaU_jLebH7NW8J7SZI9JDTxw,6750
+scenario/events/messages.py,sha256=1QAkwDExdF6AHgXdEFhHwmCv3Mxu3j0AXIptMekc_bg,3299
+scenario/events/utils.py,sha256=yrTUTByeb0eAAQniQH7EyKs-usgGti8f17IemUyBZBw,3357
+langwatch_scenario-0.6.0.dist-info/METADATA,sha256=IvD9on4tP57ldmizFzfGQBtiCT6Z7yoz0trlCSPSW9M,14227
+langwatch_scenario-0.6.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+langwatch_scenario-0.6.0.dist-info/entry_points.txt,sha256=WlEnJ_gku0i18bIa3DSuGqXRX-QDQLe_s0YmRzK45TI,45
+langwatch_scenario-0.6.0.dist-info/top_level.txt,sha256=45Mn28aedJsetnBMB5xSmrJ-yo701QLH89Zlz4r1clE,9
+langwatch_scenario-0.6.0.dist-info/RECORD,,

scenario/__init__.py

@@ -7,15 +7,21 @@ happy paths and edge cases by simulating user interactions and evaluating agent
 against configurable success criteria.
 
 Key Features:
+
 - End-to-end conversation testing with specified scenarios
+
 - Flexible control from fully scripted to completely automated simulations
+
 - Multi-turn evaluation designed for complex conversational agents
+
 - Works with any testing framework (pytest, unittest, etc.)
+
 - Framework-agnostic integration with any LLM or agent architecture
+
 - Built-in caching for deterministic and faster test execution
 
 Basic Usage:
-    ```python
+
     import scenario
 
     # Configure global settings
@@ -42,10 +48,9 @@ Basic Usage:
     )
 
     assert result.success
-    ```
 
 Advanced Usage:
-    ```python
+
     # Script-controlled scenario with custom evaluations
     def check_tool_usage(state: scenario.ScenarioState) -> None:
         assert state.has_tool_call("get_customer_info")
@@ -66,10 +71,9 @@ Advanced Usage:
             scenario.succeed("All requirements met")
         ]
     )
-    ```
 
 Integration with Testing Frameworks:
-    ```python
+
     import pytest
 
     @pytest.mark.agent_test
@@ -85,7 +89,6 @@ Integration with Testing Frameworks:
             ]
         )
         assert result.success
-    ```
 
 For more examples and detailed documentation, visit: https://github.com/langwatch/scenario
 """
@@ -104,113 +107,15 @@ from .cache import scenario_cache
 from .script import message, user, agent, judge, proceed, succeed, fail
 
 # Import pytest plugin components
-from .pytest_plugin import pytest_configure, scenario_reporter
+# from .pytest_plugin import pytest_configure, scenario_reporter
 
 run = ScenarioExecutor.run
-"""
-High-level interface for running scenario tests.
-
-This is the main entry point for executing scenario-based agent tests. It creates
-and runs a complete scenario simulation including user interactions, agent responses,
-and success evaluation.
-
-Args:
-    name: Human-readable name for the scenario
-    description: Detailed description that guides the simulation behavior
-    agents: List of agent adapters (agent under test, user simulator, judge)
-    max_turns: Maximum conversation turns before timeout (default: 10)
-    verbose: Show detailed output during execution
-    cache_key: Cache key for deterministic behavior across runs
-    debug: Enable debug mode for step-by-step execution
-    script: Optional script steps to control scenario flow
-
-Returns:
-    ScenarioResult containing test outcome, conversation history, and detailed analysis
-
-Example:
-    ```python
-    result = await scenario.run(
-        name="help request",
-        description="User needs help with a technical problem",
-        agents=[
-            MyAgentAdapter(),
-            scenario.UserSimulatorAgent(),
-            scenario.JudgeAgent(criteria=["Provides helpful response"])
-        ]
-    )
-
-    print(f"Test {'PASSED' if result.success else 'FAILED'}")
-    print(f"Reasoning: {result.reasoning}")
-    ```
-"""
 
 configure = ScenarioConfig.configure
-"""
-Set global configuration settings for all scenario executions.
-
-This function allows you to configure default behavior that will be applied
-to all scenarios unless explicitly overridden in individual scenario runs.
-
-Args:
-    default_model: Default LLM model identifier for user simulator and judge agents
-    max_turns: Maximum number of conversation turns before timeout (default: 10)
-    verbose: Enable verbose output during scenario execution
-    cache_key: Cache key for deterministic scenario behavior across runs
-    debug: Enable debug mode for step-by-step execution with user intervention
-
-Example:
-    ```python
-    # Set up global defaults
-    scenario.configure(
-        default_model="openai/gpt-4.1-mini",
-        max_turns=15,
-        verbose=True,
-        cache_key="my-test-suite-v1"
-    )
-
-    # All subsequent scenarios will use these defaults
-    result = await scenario.run(...)
-    ```
-"""
 
 default_config = ScenarioConfig.default_config
-"""
-Access to the current global configuration settings.
-
-This provides read-only access to the default configuration that has been
-set via scenario.configure(). Useful for debugging or conditional logic
-based on current settings.
-
-Example:
-    ```python
-    if scenario.default_config and scenario.default_config.debug:
-        print("Debug mode is enabled")
-    ```
-"""
 
 cache = scenario_cache
-"""
-Decorator for caching function calls during scenario execution.
-
-This decorator enables deterministic testing by caching LLM calls and other
-non-deterministic operations based on scenario configuration and function arguments.
-Results are cached when a cache_key is configured, making tests repeatable and faster.
-
-Args:
-    ignore: List of argument names to exclude from cache key computation
-
-Example:
-    ```python
-    class MyAgent:
-        @scenario.cache(ignore=["self"])
-        def invoke(self, message: str) -> str:
-            # This LLM call will be cached when cache_key is set
-            return llm_client.complete(model="gpt-4", prompt=message)
-
-    # Enable caching for deterministic tests
-    scenario.configure(cache_key="test-suite-v1")
-    ```
-"""
 
 __all__ = [
     # Functions
@@ -218,7 +123,6 @@ __all__ = [
     "configure",
     "default_config",
     "cache",
-
     # Script
     "message",
     "proceed",
@@ -227,24 +131,17 @@ __all__ = [
     "judge",
     "agent",
     "user",
-
     # Types
     "ScenarioResult",
     "AgentInput",
     "AgentRole",
     "ScenarioConfig",
     "AgentReturnTypes",
-
     # Classes
     "ScenarioExecutor",
     "ScenarioState",
     "AgentAdapter",
     "UserSimulatorAgent",
     "JudgeAgent",
-
-    # Plugins
-    "pytest_configure",
-    "scenario_reporter",
-    "scenario_cache",
 ]
-__version__ = "0.1.0"
+__version__ = "0.1.0"

scenario/_utils/__init__.py

@@ -0,0 +1,32 @@
+"""
+Utility functions for scenario execution and message handling.
+
+This module provides various utility functions used throughout the Scenario framework,
+including message formatting, validation, role reversal, and UI components like spinners
+for better user experience during scenario execution.
+"""
+
+from .message_conversion import convert_agent_return_types_to_openai_messages
+from .ids import get_or_create_batch_run_id, generate_scenario_run_id
+from .utils import (
+    SerializableAndPydanticEncoder,
+    SerializableWithStringFallback,
+    print_openai_messages,
+    show_spinner,
+    check_valid_return_type,
+    reverse_roles,
+    await_if_awaitable,
+)
+
+__all__ = [
+    "convert_agent_return_types_to_openai_messages",
+    "get_or_create_batch_run_id",
+    "generate_scenario_run_id",
+    "SerializableAndPydanticEncoder",
+    "SerializableWithStringFallback",
+    "print_openai_messages",
+    "show_spinner",
+    "check_valid_return_type",
+    "reverse_roles",
+    "await_if_awaitable",
+] 

scenario/_utils/ids.py

@@ -0,0 +1,58 @@
+"""
+ID generation and management utilities for scenario execution.
+
+This module provides functions for generating and managing unique identifiers
+used throughout the scenario execution pipeline, particularly for batch runs
+and scenario tracking.
+"""
+
+import os
+import uuid
+
+
+def get_or_create_batch_run_id() -> str:
+    """
+    Gets or creates a batch run ID for the current scenario execution.
+    
+    The batch run ID is consistent across all scenarios in the same process
+    execution, allowing grouping of related scenario runs. This is useful
+    for tracking and reporting on batches of scenarios run together.
+    
+    Returns:
+        str: A unique batch run ID that persists for the process lifetime
+        
+    Example:
+        ```python
+        # All scenarios in same process will share this ID
+        batch_id = get_or_create_batch_run_id()
+        print(f"Running scenario in batch: {batch_id}")
+        ```
+    """
+    
+    # Check if batch ID already exists in environment
+    if not os.environ.get("SCENARIO_BATCH_ID"):
+        # Generate new batch ID if not set
+        os.environ["SCENARIO_BATCH_ID"] = f"batch-run-{uuid.uuid4()}"
+    
+    return os.environ["SCENARIO_BATCH_ID"]
+
+
+def generate_scenario_run_id() -> str:
+    """
+    Generates a unique scenario run ID for a single scenario execution.
+    
+    Each scenario run gets a unique identifier that distinguishes it from
+    other runs, even within the same batch. This is used for tracking
+    individual scenario executions and correlating events.
+    
+    Returns:
+        str: A unique scenario run ID
+        
+    Example:
+        ```python
+        # Each scenario gets its own unique ID
+        scenario_id = generate_scenario_run_id()
+        print(f"Running scenario with ID: {scenario_id}")
+        ```
+    """
+    return f"scenario-run-{uuid.uuid4()}"