langwatch-scenario 0.6.0__tar.gz → 0.7.2__tar.gz

{langwatch_scenario-0.6.0 → langwatch_scenario-0.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langwatch-scenario
-Version: 0.6.0
+Version: 0.7.2
 Summary: The end-to-end agent testing library
 Author-email: LangWatch Team <support@langwatch.ai>
 License: MIT
@@ -18,6 +18,7 @@ Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 Requires-Dist: pytest>=8.1.1
 Requires-Dist: litellm>=1.49.0
+Requires-Dist: openai>=1.88.0
 Requires-Dist: python-dotenv>=1.0.1
 Requires-Dist: termcolor>=2.4.0
 Requires-Dist: pydantic>=2.7.0
@@ -26,11 +27,9 @@ Requires-Dist: wrapt>=1.17.2
 Requires-Dist: pytest-asyncio>=0.26.0
 Requires-Dist: rich<15.0.0,>=13.3.3
 Requires-Dist: pksuid>=1.1.2
-Requires-Dist: pdoc3>=0.11.6
-Requires-Dist: ag-ui-protocol>=0.1.0
 Requires-Dist: httpx>=0.27.0
 Requires-Dist: rx>=3.2.0
-Requires-Dist: respx>=0.22.0
+Requires-Dist: python-dateutil>=2.9.0.post0
 Provides-Extra: dev
 Requires-Dist: black; extra == "dev"
 Requires-Dist: isort; extra == "dev"
@@ -40,12 +39,20 @@ Requires-Dist: commitizen; extra == "dev"
 Requires-Dist: pyright; extra == "dev"
 Requires-Dist: pydantic-ai; extra == "dev"
 Requires-Dist: function-schema; extra == "dev"
+Requires-Dist: pdoc3; extra == "dev"
+Requires-Dist: respx; extra == "dev"
 
 ![scenario](https://github.com/langwatch/scenario/raw/main/assets/scenario-wide.webp)
 
-<div align="center">
-<!-- Discord, PyPI, Docs, etc links -->
-</div>
+<p align="center">
+    <a href="https://discord.gg/kT4PhDS2gH" target="_blank"><img src="https://img.shields.io/discord/1227886780536324106?logo=discord&labelColor=%20%235462eb&logoColor=%20%23f5f5f5&color=%20%235462eb" alt="chat on Discord"></a>
+    <a href="https://pypi.python.org/pypi/langwatch-scenario" target="_blank"><img src="https://img.shields.io/pypi/dm/langwatch-scenario?logo=python&logoColor=white&label=pypi%20langwatch-scenario&color=blue" alt="Scenario Python package on PyPi"></a>
+    <a href="https://www.npmjs.com/package/@langwatch/scenario" target="_blank"><img src="https://img.shields.io/npm/dm/@langwatch/scenario?logo=npm&logoColor=white&label=npm%20@langwatch/scenario&color=blue" alt="Scenario JavaScript package on npm"></a>
+    <a href="https://github.com/langwatch/scenario/actions/workflows/python-ci.yml"><img src="https://github.com/langwatch/scenario/actions/workflows/python-ci.yml/badge.svg" alt="Python Tests" /></a>
+    <a href="https://github.com/langwatch/scenario/actions/workflows/javascript-ci.yml"><img src="https://github.com/langwatch/scenario/actions/workflows/javascript-ci.yml/badge.svg" alt="JavaScript Tests" /></a>
+    <a href="https://twitter.com/intent/follow?screen_name=langwatchai" target="_blank">
+    <img src="https://img.shields.io/twitter/follow/langwatchai?logo=X&color=%20%23f5f5f5" alt="follow on X(Twitter)"></a>
+</p>
 
 # Scenario
 
@@ -54,19 +61,15 @@ Scenario is an Agent Testing Framework based on simulations, it can:
 - Test real agent behavior by simulating users in different scenarios and edge cases
 - Evaluate and judge at any point of the conversation, powerful multi-turn control
 - Combine it with any LLM eval framework or custom evals, agnostic by design
-- Integrate your Agent by implementing just one `call()` method
+- Integrate your Agent by implementing just one [`call()`](https://scenario.langwatch.ai/agent-integration) method
 - Available in Python, TypeScript and Go
 
-[📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
-
-### In other languages
-
-- [Scenario TypeScript](https://github.com/langwatch/scenario-ts/)
-- [Scenario Go](https://github.com/langwatch/scenario-go/)
+📖 [Documentation](https://scenario.langwatch.ai)\
+📺 [Watch Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
 
 ## Example
 
-This is how a simple simulation with tool check looks like with Scenario:
+This is how a simulation with tool check looks like with Scenario:
 
 ```python
 # Define any custom assertions
@@ -100,18 +103,56 @@ result = await scenario.run(
 assert result.success
 ```
 
+<details>
+<summary><strong>TypeScript Example</strong></summary>
+
+```typescript
+const result = await scenario.run({
+  name: "vegetarian recipe agent",
+
+  // Define the prompt to guide the simulation
+  description: `
+    The user is planning a boat trip from Barcelona to Rome,
+    and is wondering what the weather will be like.
+  `,
+
+  // Define the agents that will play this simulation
+  agents: [new MyAgent(), scenario.userSimulatorAgent()],
+
+  // (Optional) Control the simulation
+  script: [
+    scenario.user(), // let the user simulator generate a user message
+    scenario.agent(), // agent responds
+    // check for tool call after the first agent response
+    (state) => expect(state.has_tool_call("get_current_weather")).toBe(true),
+    scenario.succeed(), // simulation ends successfully
+  ],
+});
+```
+
+</details>
+
 > [!NOTE]
-> Check out full examples in the [examples folder](./examples/).
+> Check out full examples in the [python/examples folder](./python/examples/). or the [typescript/examples folder](./typescript/examples/).
 
-## Getting Started
+## Quick Start
 
-Install pytest and scenario:
+Install scenario and a test runner:
 
 ```bash
-pip install pytest langwatch-scenario
+# on python
+uv add langwatch-scenario pytest
+
+# or on typescript
+pnpm install @langwatch/scenario vitest
 ```
 
-Now create your first scenario and save it as `tests/test_vegetarian_recipe_agent.py`, copy the full working example below:
+Now create your first scenario, copy the full working example below.
+
+<details>
+<summary><strong>Quick Start - Python</strong></summary>
+
+Save it as `tests/test_vegetarian_recipe_agent.py`:
 
 ```python
 import pytest
@@ -178,23 +219,86 @@ def vegetarian_recipe_agent(messages) -> scenario.AgentReturnTypes:
     return response.choices[0].message  # type: ignore
 ```
 
-Create a `.env` file and put your OpenAI API key in it:
+</details>
+
+<details>
+<summary><strong>Quick Start - TypeScript</strong></summary>
+
+Save it as `tests/vegetarian-recipe-agent.test.ts`:
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import * as scenario from "@langwatch/scenario";
+import { generateText } from "ai";
+import { describe, it, expect } from "vitest";
+
+describe("Vegetarian Recipe Agent", () => {
+  const agent: scenario.AgentAdapter = {
+    role: scenario.AgentRole.AGENT,
+    call: async (input) => {
+      const response = await generateText({
+        model: openai("gpt-4.1-mini"),
+        messages: [
+          {
+            role: "system",
+            content: `You are a vegetarian recipe agent.\nGiven the user request, ask AT MOST ONE follow-up question, then provide a complete recipe. Keep your responses concise and focused.`,
+          },
+          ...input.messages,
+        ],
+      });
+      return response.text;
+    },
+  };
+
+  it("should generate a vegetarian recipe for a hungry and tired user on a Saturday evening", async () => {
+    const result = await scenario.run({
+      name: "dinner idea",
+      description: `It's saturday evening, the user is very hungry and tired, but have no money to order out, so they are looking for a recipe.`,
+      agents: [
+        agent,
+        scenario.userSimulatorAgent(),
+        scenario.judgeAgent({
+          model: openai("gpt-4.1-mini"),
+          criteria: [
+            "Agent should not ask more than two follow-up questions",
+            "Agent should generate a recipe",
+            "Recipe should include a list of ingredients",
+            "Recipe should include step-by-step cooking instructions",
+            "Recipe should be vegetarian and not include any sort of meat",
+          ],
+        }),
+      ],
+    });
+    expect(result.success).toBe(true);
+  });
+});
+```
+
+</details>
+
+Export your OpenAI API key:
 
 ```bash
 OPENAI_API_KEY=<your-api-key>
 ```
 
-Now run it with pytest:
+Now run it the test:
 
 ```bash
+# on python
 pytest -s tests/test_vegetarian_recipe_agent.py
+
+# on typescript
+npx vitest run tests/vegetarian-recipe-agent.test.ts
 ```
 
 This is how it will look like:
 
-[![asciicast](./assets/ascii-cinema.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
+[![asciicast](https://github.com/langwatch/scenario/raw/main/assets/ascii-cinema.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
 
-You can find the same code example in [examples/test_vegetarian_recipe_agent.py](examples/test_vegetarian_recipe_agent.py).
+You can find the same code example in [python/examples/](python/examples/test_vegetarian_recipe_agent.py) or [javascript/examples/](javascript/examples/vitest/tests/vegetarian-recipe-agent.test.ts).
+
+Now check out the [full documentation](https://scenario.langwatch.ai) to learn more and next steps.
 
 ## Simulation on Autopilot
 
@@ -296,6 +400,16 @@ async def test_early_assumption_bias():
     assert result.success
 ```
 
+## LangWatch Visualization
+
+Set your [LangWatch API key](https://app.langwatch.ai/) to visualize the scenarios in real-time, as they run, for a much better debugging experience and team collaboration:
+
+```bash
+LANGWATCH_API_KEY="your-api-key"
+```
+
+![LangWatch Visualization](./assets/langwatch-visualization.webp)
+
 ## Debug mode
 
 You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.
@@ -360,26 +474,16 @@ async def test_user_is_very_hungry():
 
 Those two scenarios should now run in parallel.
 
-## Events System
-
-Scenario automatically publishes events during execution for monitoring and observability. You can enable event reporting by setting environment variables:
-
-```bash
-# Enable automatic event reporting
-export LANGWATCH_ENDPOINT="https://api.langwatch.ai"
-export LANGWATCH_API_KEY="your-api-key"
-```
-
-With these variables set, Scenario will automatically:
+## Contributing
 
-- Publish events when scenarios start, finish, and when messages are added
-- Handle retries and error handling automatically
-- Process events asynchronously without blocking your tests
+We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
 
-The events include timing information, conversation history, and success/failure metrics for analysis.
+## Support
 
-For advanced customization, see the event classes in the codebase for detailed documentation.
+- 📖 [Documentation](https://scenario.langwatch.ai)
+- 💬 [Discord Community](https://discord.gg/langwatch)
+- 🐛 [Issue Tracker](https://github.com/langwatch/scenario/issues)
 
 ## License
 
-MIT License
+MIT License - see [LICENSE](LICENSE) for details.

{langwatch_scenario-0.6.0 → langwatch_scenario-0.7.2}/README.md

@@ -1,8 +1,14 @@
 ![scenario](https://github.com/langwatch/scenario/raw/main/assets/scenario-wide.webp)
 
-<div align="center">
-<!-- Discord, PyPI, Docs, etc links -->
-</div>
+<p align="center">
+    <a href="https://discord.gg/kT4PhDS2gH" target="_blank"><img src="https://img.shields.io/discord/1227886780536324106?logo=discord&labelColor=%20%235462eb&logoColor=%20%23f5f5f5&color=%20%235462eb" alt="chat on Discord"></a>
+    <a href="https://pypi.python.org/pypi/langwatch-scenario" target="_blank"><img src="https://img.shields.io/pypi/dm/langwatch-scenario?logo=python&logoColor=white&label=pypi%20langwatch-scenario&color=blue" alt="Scenario Python package on PyPi"></a>
+    <a href="https://www.npmjs.com/package/@langwatch/scenario" target="_blank"><img src="https://img.shields.io/npm/dm/@langwatch/scenario?logo=npm&logoColor=white&label=npm%20@langwatch/scenario&color=blue" alt="Scenario JavaScript package on npm"></a>
+    <a href="https://github.com/langwatch/scenario/actions/workflows/python-ci.yml"><img src="https://github.com/langwatch/scenario/actions/workflows/python-ci.yml/badge.svg" alt="Python Tests" /></a>
+    <a href="https://github.com/langwatch/scenario/actions/workflows/javascript-ci.yml"><img src="https://github.com/langwatch/scenario/actions/workflows/javascript-ci.yml/badge.svg" alt="JavaScript Tests" /></a>
+    <a href="https://twitter.com/intent/follow?screen_name=langwatchai" target="_blank">
+    <img src="https://img.shields.io/twitter/follow/langwatchai?logo=X&color=%20%23f5f5f5" alt="follow on X(Twitter)"></a>
+</p>
 
 # Scenario
 
@@ -11,19 +17,15 @@ Scenario is an Agent Testing Framework based on simulations, it can:
 - Test real agent behavior by simulating users in different scenarios and edge cases
 - Evaluate and judge at any point of the conversation, powerful multi-turn control
 - Combine it with any LLM eval framework or custom evals, agnostic by design
-- Integrate your Agent by implementing just one `call()` method
+- Integrate your Agent by implementing just one [`call()`](https://scenario.langwatch.ai/agent-integration) method
 - Available in Python, TypeScript and Go
 
-[📺 Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
-
-### In other languages
-
-- [Scenario TypeScript](https://github.com/langwatch/scenario-ts/)
-- [Scenario Go](https://github.com/langwatch/scenario-go/)
+📖 [Documentation](https://scenario.langwatch.ai)\
+📺 [Watch Video Tutorial](https://www.youtube.com/watch?v=f8NLpkY0Av4)
 
 ## Example
 
-This is how a simple simulation with tool check looks like with Scenario:
+This is how a simulation with tool check looks like with Scenario:
 
 ```python
 # Define any custom assertions
@@ -57,18 +59,56 @@ result = await scenario.run(
 assert result.success
 ```
 
+<details>
+<summary><strong>TypeScript Example</strong></summary>
+
+```typescript
+const result = await scenario.run({
+  name: "vegetarian recipe agent",
+
+  // Define the prompt to guide the simulation
+  description: `
+    The user is planning a boat trip from Barcelona to Rome,
+    and is wondering what the weather will be like.
+  `,
+
+  // Define the agents that will play this simulation
+  agents: [new MyAgent(), scenario.userSimulatorAgent()],
+
+  // (Optional) Control the simulation
+  script: [
+    scenario.user(), // let the user simulator generate a user message
+    scenario.agent(), // agent responds
+    // check for tool call after the first agent response
+    (state) => expect(state.has_tool_call("get_current_weather")).toBe(true),
+    scenario.succeed(), // simulation ends successfully
+  ],
+});
+```
+
+</details>
+
 > [!NOTE]
-> Check out full examples in the [examples folder](./examples/).
+> Check out full examples in the [python/examples folder](./python/examples/). or the [typescript/examples folder](./typescript/examples/).
 
-## Getting Started
+## Quick Start
 
-Install pytest and scenario:
+Install scenario and a test runner:
 
 ```bash
-pip install pytest langwatch-scenario
+# on python
+uv add langwatch-scenario pytest
+
+# or on typescript
+pnpm install @langwatch/scenario vitest
 ```
 
-Now create your first scenario and save it as `tests/test_vegetarian_recipe_agent.py`, copy the full working example below:
+Now create your first scenario, copy the full working example below.
+
+<details>
+<summary><strong>Quick Start - Python</strong></summary>
+
+Save it as `tests/test_vegetarian_recipe_agent.py`:
 
 ```python
 import pytest
@@ -135,23 +175,86 @@ def vegetarian_recipe_agent(messages) -> scenario.AgentReturnTypes:
     return response.choices[0].message  # type: ignore
 ```
 
-Create a `.env` file and put your OpenAI API key in it:
+</details>
+
+<details>
+<summary><strong>Quick Start - TypeScript</strong></summary>
+
+Save it as `tests/vegetarian-recipe-agent.test.ts`:
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import * as scenario from "@langwatch/scenario";
+import { generateText } from "ai";
+import { describe, it, expect } from "vitest";
+
+describe("Vegetarian Recipe Agent", () => {
+  const agent: scenario.AgentAdapter = {
+    role: scenario.AgentRole.AGENT,
+    call: async (input) => {
+      const response = await generateText({
+        model: openai("gpt-4.1-mini"),
+        messages: [
+          {
+            role: "system",
+            content: `You are a vegetarian recipe agent.\nGiven the user request, ask AT MOST ONE follow-up question, then provide a complete recipe. Keep your responses concise and focused.`,
+          },
+          ...input.messages,
+        ],
+      });
+      return response.text;
+    },
+  };
+
+  it("should generate a vegetarian recipe for a hungry and tired user on a Saturday evening", async () => {
+    const result = await scenario.run({
+      name: "dinner idea",
+      description: `It's saturday evening, the user is very hungry and tired, but have no money to order out, so they are looking for a recipe.`,
+      agents: [
+        agent,
+        scenario.userSimulatorAgent(),
+        scenario.judgeAgent({
+          model: openai("gpt-4.1-mini"),
+          criteria: [
+            "Agent should not ask more than two follow-up questions",
+            "Agent should generate a recipe",
+            "Recipe should include a list of ingredients",
+            "Recipe should include step-by-step cooking instructions",
+            "Recipe should be vegetarian and not include any sort of meat",
+          ],
+        }),
+      ],
+    });
+    expect(result.success).toBe(true);
+  });
+});
+```
+
+</details>
+
+Export your OpenAI API key:
 
 ```bash
 OPENAI_API_KEY=<your-api-key>
 ```
 
-Now run it with pytest:
+Now run it the test:
 
 ```bash
+# on python
 pytest -s tests/test_vegetarian_recipe_agent.py
+
+# on typescript
+npx vitest run tests/vegetarian-recipe-agent.test.ts
 ```
 
 This is how it will look like:
 
-[![asciicast](./assets/ascii-cinema.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
+[![asciicast](https://github.com/langwatch/scenario/raw/main/assets/ascii-cinema.svg)](https://asciinema.org/a/nvO5GWGzqKTTCd8gtNSezQw11)
 
-You can find the same code example in [examples/test_vegetarian_recipe_agent.py](examples/test_vegetarian_recipe_agent.py).
+You can find the same code example in [python/examples/](python/examples/test_vegetarian_recipe_agent.py) or [javascript/examples/](javascript/examples/vitest/tests/vegetarian-recipe-agent.test.ts).
+
+Now check out the [full documentation](https://scenario.langwatch.ai) to learn more and next steps.
 
 ## Simulation on Autopilot
 
@@ -253,6 +356,16 @@ async def test_early_assumption_bias():
     assert result.success
 ```
 
+## LangWatch Visualization
+
+Set your [LangWatch API key](https://app.langwatch.ai/) to visualize the scenarios in real-time, as they run, for a much better debugging experience and team collaboration:
+
+```bash
+LANGWATCH_API_KEY="your-api-key"
+```
+
+![LangWatch Visualization](./assets/langwatch-visualization.webp)
+
 ## Debug mode
 
 You can enable debug mode by setting the `debug` field to `True` in the `Scenario.configure` method or in the specific scenario you are running, or by passing the `--debug` flag to pytest.
@@ -317,26 +430,16 @@ async def test_user_is_very_hungry():
 
 Those two scenarios should now run in parallel.
 
-## Events System
-
-Scenario automatically publishes events during execution for monitoring and observability. You can enable event reporting by setting environment variables:
-
-```bash
-# Enable automatic event reporting
-export LANGWATCH_ENDPOINT="https://api.langwatch.ai"
-export LANGWATCH_API_KEY="your-api-key"
-```
-
-With these variables set, Scenario will automatically:
+## Contributing
 
-- Publish events when scenarios start, finish, and when messages are added
-- Handle retries and error handling automatically
-- Process events asynchronously without blocking your tests
+We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
 
-The events include timing information, conversation history, and success/failure metrics for analysis.
+## Support
 
-For advanced customization, see the event classes in the codebase for detailed documentation.
+- 📖 [Documentation](https://scenario.langwatch.ai)
+- 💬 [Discord Community](https://discord.gg/langwatch)
+- 🐛 [Issue Tracker](https://github.com/langwatch/scenario/issues)
 
 ## License
 
-MIT License
+MIT License - see [LICENSE](LICENSE) for details.