strands-agents-evals 0.1.0__py3-none-any.whl

strands_evals/simulation/README.md

@@ -0,0 +1,323 @@
+# Actor Simulator
+
+A framework for simulating realistic multi-turn conversations with AI-powered actors for agent evaluation.
+
+## Overview
+
+ActorSimulator creates realistic actor personas that interact with agents in multi-turn conversations. It automatically generates actor profiles from test cases, maintains conversation context, and produces contextually appropriate responses aligned with the actor's goals and traits.
+
+## Quick Start
+
+```python
+from strands import Agent
+from strands_evals import ActorSimulator, Case
+
+# Create agent under test
+agent = Agent(system_prompt="You are a helpful travel assistant.", callback_handler=None)
+
+# Create test case
+case = Case(
+    input="I want to plan a trip to Tokyo with hotel and activities",
+    metadata={"task_description": "Complete travel package arranged"}
+)
+
+# Create user simulator with max_turns
+user_sim = ActorSimulator.from_case_for_user_simulator(case=case, max_turns=5)
+
+# Run conversation
+user_message = case.input
+while user_sim.has_next():
+    agent_response = agent(user_message)
+    user_result = user_sim.act(str(agent_response))
+    user_message = str(user_result.structured_output.message)
+```
+
+## How It Works
+
+1. **Profile Generation**: Creates a realistic actor profile with traits, context, and goals from the test case
+2. **Conversation Initialization**: Sets up conversation with a greeting and the actor's initial query
+3. **Contextual Responses**: Generates responses that maintain consistency with the actor's profile and goals
+4. **Goal Tracking**: Built-in tool allows actors to assess progress toward their goals
+
+## API Reference
+
+### ActorSimulator
+
+Main class for simulating actor behavior in conversations.
+
+#### Factory Method (Recommended)
+
+```python
+ActorSimulator.from_case_for_user_simulator(
+    case: Case,
+    system_prompt_template: str | None = None,
+    tools: list | None = None,
+    model: str | None = None,
+    max_turns: int = 10
+) -> ActorSimulator
+```
+
+Creates an ActorSimulator configured as a user simulator from a test case. Automatically generates a realistic actor profile from `case.input` and optionally `case.metadata["task_description"]`.
+
+**Parameters:**
+- `case`: Test case with input (initial query) and optional task_description in metadata
+- `system_prompt_template`: Custom system prompt template (uses default if None)
+- `tools`: Additional tools for the actor (defaults to goal completion tool only)
+- `model`: Model identifier (uses Strands default if None)
+- `max_turns`: Maximum number of conversation turns (default: 10)
+
+**Example:**
+```python
+case = Case(
+    input="I need help booking a flight to Paris",
+    metadata={"task_description": "Book round-trip flight under $800"}
+)
+
+user_sim = ActorSimulator.from_case_for_user_simulator(
+    case=case,
+    max_turns=5
+)
+```
+
+#### Direct Initialization
+
+```python
+ActorSimulator(
+    actor_profile: ActorProfile,
+    initial_query: str,
+    system_prompt_template: str,
+    tools: list | None = None,
+    model: str | None = None,
+    max_turns: int = 10
+)
+```
+
+Initialize with an existing actor profile. Use this when you have a pre-defined profile instead of generating one from a test case.
+
+**Parameters:**
+- `actor_profile`: ActorProfile object with traits, context, and actor_goal
+- `initial_query`: The actor's first query or message
+- `system_prompt_template`: Template string for actor behavior (formatted with profile)
+- `tools`: Additional tools for the actor
+- `model`: Model identifier
+- `max_turns`: Maximum number of conversation turns (default: 10)
+
+#### Methods
+
+**`act(agent_message: str) -> AgentResult`**
+
+Generate the actor's next message in response to the agent's message.
+
+**Parameters:**
+- `agent_message`: The agent's response to react to
+
+**Returns:**
+- `AgentResult` containing the actor's structured response with reasoning and message
+
+**Example:**
+```python
+agent_response = agent("I can help you book that flight")
+user_result = user_sim.act(str(agent_response))
+user_message = str(user_result.structured_output.message)
+```
+
+**`has_next() -> bool`**
+
+Check if the conversation should continue. Returns False if the stop token (`<stop/>`) is present in the last message or if the maximum number of turns has been reached.
+
+**Returns:**
+- `True` if the conversation should continue, `False` otherwise
+
+**Example:**
+```python
+while user_sim.has_next():
+    agent_response = agent(user_message)
+    user_result = user_sim.act(str(agent_response))
+    user_message = str(user_result.structured_output.message)
+```
+
+### Data Models
+
+**ActorProfile:**
+```python
+class ActorProfile(BaseModel):
+    traits: dict[str, Any]  # Actor characteristics and personality
+    context: str  # Background information and situation
+    actor_goal: str  # What the actor wants to achieve
+```
+
+**ActorResponse:**
+```python
+class ActorResponse(BaseModel):
+    reasoning: str  # Actor's internal reasoning process
+    message: str  # The actual message to send
+```
+
+## Usage Examples
+
+### Complete Multi-Turn Conversation Example
+
+```python
+from strands import Agent
+from strands_evals import ActorSimulator, Case
+
+# Create agent under test
+agent = Agent(system_prompt="You are a helpful travel assistant.", callback_handler=None)
+
+# Create test case
+case = Case(
+    input="I want to plan a trip to Tokyo with hotel and activities",
+    metadata={"task_description": "Complete travel package arranged"}
+)
+
+# Create user simulator
+user_sim = ActorSimulator.from_case_for_user_simulator(case=case, max_turns=5)
+
+# Run conversation
+conversation = []
+user_message = case.input
+
+while user_sim.has_next():
+    # Agent responds
+    agent_response = agent(user_message)
+    agent_message = str(agent_response)
+    conversation.append({"role": "assistant", "content": agent_message})
+    
+    # User responds
+    user_result = user_sim.act(agent_message)
+    user_message = str(user_result.structured_output.message)
+    conversation.append({"role": "user", "content": user_message})
+
+print(f"Conversation completed in {len(conversation) // 2} turns")
+```
+
+### Custom Actor Profile
+
+```python
+from strands_evals.types.simulation import ActorProfile
+from strands_evals.simulation.prompt_templates.actor_system_prompt import (
+    DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE
+)
+
+# Create custom actor profile
+actor_profile = ActorProfile(
+    traits={
+        "personality": "analytical and detail-oriented",
+        "communication_style": "direct and concise",
+        "technical_level": "expert"
+    },
+    context="Experienced business traveler with elite status",
+    actor_goal="Book business class flight with specific seat preferences"
+)
+
+# Initialize with custom profile
+user_sim = ActorSimulator(
+    actor_profile=actor_profile,
+    initial_query="I need to book a business class flight to London",
+    system_prompt_template=DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE,
+    max_turns=15
+)
+```
+
+## Tools
+
+### Built-in Goal Completion Tool
+
+ActorSimulator automatically includes a goal completion assessment tool that actors can use to evaluate their progress:
+
+```python
+from strands_evals.simulation.tools.goal_completion import (
+    get_conversation_goal_completion
+)
+
+# The actor can call this tool during conversation to assess progress
+assessment = get_conversation_goal_completion(
+    initial_goal="Book a flight to Tokyo",
+    conversation=[
+        {"role": "user", "content": "I need a flight to Tokyo"},
+        {"role": "assistant", "content": "I can help with that..."}
+    ]
+)
+# Returns assessment with score and reasoning
+```
+
+### Adding Custom Tools
+
+Extend actor capabilities with custom tools:
+
+```python
+from strands import tool
+
+@tool
+def check_booking_status(booking_id: str) -> str:
+    """Check the status of a booking."""
+    return f"Booking {booking_id} is confirmed"
+
+# Add custom tools to the simulator
+user_sim = ActorSimulator.from_case_for_user_simulator(
+    case=case,
+    tools=[check_booking_status]
+)
+```
+
+## Advanced Configuration
+
+### Custom System Prompt Templates
+
+Customize actor behavior with a custom system prompt template. The template receives the actor profile as a format parameter:
+
+```python
+custom_prompt_template = """
+You are simulating a user with the following profile:
+{actor_profile}
+
+Behavior guidelines:
+- Be persistent but professional
+- Express concerns clearly
+- Stay focused on your goal
+
+Respond naturally based on your profile and the conversation context.
+"""
+
+user_sim = ActorSimulator.from_case_for_user_simulator(
+    case=case,
+    system_prompt_template=custom_prompt_template
+)
+```
+
+### Conversation Initialization
+
+ActorSimulator automatically initializes conversations with a random greeting from a predefined set:
+
+```python
+# Built-in greetings:
+# - "hi! how can I help you today?"
+# - "hello! what can I assist you with?"
+# - "hi there! how may I help you?"
+# - "good day! what can I do for you?"
+# - "hello! what would you like to know?"
+
+# The conversation starts with:
+# 1. Random greeting (as user message)
+# 2. Actor's initial query (as assistant message)
+```
+
+### Model Selection
+
+Specify a custom model for the actor simulator:
+
+```python
+user_sim = ActorSimulator.from_case_for_user_simulator(
+    case=case,
+    model="anthropic.claude-3-5-sonnet-20241022-v2:0",
+    max_turns=10
+)
+```
+
+## Best Practices
+
+1. **Include Task Description**: Add `task_description` in case metadata for better goal generation
+2. **Set max_turns**: Configure `max_turns` during initialization to prevent infinite conversations
+3. **Use has_next()**: Always use `has_next()` in your conversation loop to respect turn limits and stop tokens
+4. **Track Conversation**: Append messages to a conversation list for evaluation and debugging
+5. **Access Structured Output**: Use `result.structured_output.message` to get the actor's message and `result.structured_output.reasoning` to see internal reasoning

strands_evals/simulation/__init__.py

@@ -0,0 +1,6 @@
+from .actor_simulator import ActorSimulator
+
+# Alias for backward compatibility
+UserSimulator = ActorSimulator
+
+__all__ = ["ActorSimulator", "UserSimulator"]

strands_evals/simulation/actor_simulator.py

@@ -0,0 +1,292 @@
+import logging
+import random
+
+from strands import Agent
+from strands.agent.agent_result import AgentResult
+from strands.types.content import Message
+from typing_extensions import cast
+
+from strands_evals.case import Case
+from strands_evals.simulation.profiles.actor_profile import DEFAULT_USER_PROFILE_SCHEMA
+from strands_evals.simulation.prompt_templates.actor_profile_extraction import ACTOR_PROFILE_PROMPT_TEMPLATE
+from strands_evals.simulation.prompt_templates.actor_system_prompt import DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE
+from strands_evals.simulation.tools.goal_completion import get_conversation_goal_completion
+from strands_evals.types.simulation import ActorProfile, ActorResponse
+
+logger = logging.getLogger(__name__)
+
+
+class ActorSimulator:
+    """
+    Simulates an actor in multi-turn conversations for agent evaluation.
+
+    ActorSimulator wraps a Strands Agent configured to behave as a specific actor
+    (typically a user) in conversation scenarios. It maintains conversation history,
+    generates contextually appropriate responses, and can assess goal completion.
+
+    Attributes:
+        agent: The underlying Strands Agent configured with actor behavior.
+        actor_profile: The actor's profile containing traits, context, and goal.
+        initial_query: The actor's first query in the conversation.
+        conversation_history: List of conversation messages in Strands format.
+        model_id: Model identifier for the underlying agent.
+    """
+
+    INITIAL_GREETINGS = [
+        "hi! how can I help you today?",
+        "hello! what can I assist you with?",
+        "hi there! how may I help you?",
+        "good day! what can I do for you?",
+        "hello! what would you like to know?",
+    ]
+
+    @classmethod
+    def from_case_for_user_simulator(
+        cls,
+        case: Case,
+        system_prompt_template: str | None = None,
+        tools: list | None = None,
+        model: str | None = None,
+        max_turns: int = 10,
+    ) -> "ActorSimulator":
+        """
+        Create an ActorSimulator configured as a user simulator from a test case.
+
+        Generates a realistic user profile and goal from case.input and optionally
+        case.metadata["task_description"], then configures the simulator with
+        user-specific defaults. If you already have a profile, use __init__() directly.
+
+        Args:
+            case: Test case containing input (initial query) and optional metadata with "task_description".
+            system_prompt_template: Custom system prompt template. Uses DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE if None.
+            tools: Additional tools available to the user. Defaults to goal completion tool only.
+            model: Model identifier for the underlying agent. Uses Strands default if None.
+            max_turns: Maximum number of conversation turns before stopping (default: 10).
+
+        Returns:
+            ActorSimulator configured for user simulation.
+
+        Example:
+            ```python
+            from strands_evals import Case, ActorSimulator
+            from strands import Agent
+
+            # Create test case
+            case = Case(
+                input="I need to book a flight to Paris",
+                metadata={"task_description": "Flight booking confirmed"}
+            )
+
+            # Create user simulator
+            user_sim = ActorSimulator.from_case_for_user_simulator(
+                case=case,
+                max_turns=5
+            )
+
+            # Create target agent to evaluate
+            agent = Agent(system_prompt="You are a travel assistant.")
+
+            # Run conversation
+            user_message = case.input
+            while user_sim.has_next():
+                agent_response = agent(user_message)
+                user_result = user_sim.act(str(agent_response))
+                user_message = str(user_result.structured_output.message)
+            ```
+        """
+        actor_profile = cls._generate_profile_from_case(case)
+
+        if system_prompt_template is None:
+            system_prompt_template = DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE
+
+        return cls(
+            actor_profile=actor_profile,
+            initial_query=case.input,
+            system_prompt_template=system_prompt_template,
+            tools=tools,
+            model=model,
+            max_turns=max_turns,
+        )
+
+    @staticmethod
+    def _generate_profile_from_case(case: Case) -> ActorProfile:
+        """
+        Generate user profile from case.
+
+        Private helper for from_case_for_user_simulator factory method.
+        Uses case.input and optionally case.metadata["task_description"] if present.
+
+        Args:
+            case: Test case with input and optional task_description in metadata.
+
+        Returns:
+            ActorProfile with generated traits, context, and goal.
+        """
+        initial_query = case.input
+        task_description = case.metadata.get("task_description", "") if case.metadata else ""
+
+        profile_prompt = ACTOR_PROFILE_PROMPT_TEMPLATE.format(
+            initial_query=initial_query,
+            task_description=task_description,
+            example=DEFAULT_USER_PROFILE_SCHEMA,
+        )
+        profile_agent = Agent(callback_handler=None)
+        result = profile_agent(profile_prompt, structured_output_model=ActorProfile)
+        return result.structured_output
+
+    def __init__(
+        self,
+        actor_profile: ActorProfile,
+        initial_query: str,
+        system_prompt_template: str,
+        tools: list | None = None,
+        model: str | None = None,
+        max_turns: int = 10,
+    ):
+        """
+        Initialize an ActorSimulator with profile and goal.
+
+        Use this constructor when you have a pre-defined ActorProfile. For automatic
+        profile generation from test cases, use from_case_for_user_simulator() instead.
+
+        Args:
+            actor_profile: ActorProfile object containing traits, context, and actor_goal.
+            initial_query: The actor's first query or message.
+            system_prompt_template: Template string for system prompt. Must include {actor_profile} placeholder.
+            tools: Additional tools available to the actor. Defaults to goal completion tool only.
+            model: Model identifier for the underlying agent. Uses Strands default if None.
+            max_turns: Maximum number of conversation turns before stopping (default: 10).
+
+        Example:
+            ```python
+            from strands_evals.simulation import ActorSimulator
+            from strands_evals.types.simulation import ActorProfile
+
+            # Define custom actor profile
+            profile = ActorProfile(
+                traits={
+                    "expertise_level": "expert",
+                    "communication_style": "technical"
+                },
+                context="A software engineer debugging a production issue.",
+                actor_goal="Identify and resolve the memory leak."
+            )
+
+            # Create simulator with custom profile
+            simulator = ActorSimulator(
+                actor_profile=profile,
+                initial_query="Our service is experiencing high memory usage.",
+                system_prompt_template="You are simulating: {actor_profile}",
+                max_turns=15
+            )
+            ```
+        """
+        self.actor_profile = actor_profile
+        self.initial_query = initial_query
+        self.conversation_history: list[Message] = []
+        self.model_id = model
+        self._turn_count = 0
+        self._last_message = ""
+        self._max_turns = max_turns
+
+        system_prompt = system_prompt_template.format(actor_profile=actor_profile.model_dump())
+
+        # Combine tools
+        all_tools = [get_conversation_goal_completion]
+        if tools:
+            all_tools.extend(tools)
+
+        self._initialize_conversation()
+
+        # Create agent
+        self.agent = Agent(
+            system_prompt=system_prompt,
+            messages=self.conversation_history,
+            tools=all_tools,
+            model=self.model_id,
+            callback_handler=None,
+        )
+
+    def _initialize_conversation(self):
+        """
+        Initialize the conversation history with a greeting and initial query.
+
+        Sets up the conversation with a random greeting from the assistant followed
+        by the actor's initial query. This establishes the conversation context.
+
+        Note: This is a private method called during initialization.
+        """
+        selected_greeting = random.choice(self.INITIAL_GREETINGS)
+        greeting_message = {"role": "user", "content": [{"text": selected_greeting}]}
+        self.conversation_history.append(greeting_message)
+
+        initial_query_message = {"role": "assistant", "content": [{"text": self.initial_query.strip()}]}
+        self.conversation_history.append(initial_query_message)
+
+    def act(self, agent_message: str) -> AgentResult:
+        """
+        Generate the next actor message in the conversation.
+
+        Processes the agent's message and generates a contextually appropriate
+        response from the actor's perspective, maintaining consistency with the actor's
+        profile and goal. The response includes reasoning about the actor's thought
+        process and the actual message to send.
+
+        Args:
+            agent_message: The agent's response to react to (required).
+
+        Returns:
+            AgentResult containing the actor's structured response with:
+                - structured_output.reasoning: Actor's internal reasoning
+                - structured_output.message: Actor's response message
+
+        Example:
+            ```python
+            # Agent responds to user
+            agent_response = agent("I need help booking a flight")
+
+            # User simulator generates next message
+            user_result = user_sim.act(str(agent_response))
+
+            # Access the response
+            print(user_result.structured_output.reasoning)  # Why the actor responded this way
+            print(user_result.structured_output.message)    # The actual message
+
+            # Continue conversation
+            next_message = str(user_result.structured_output.message)
+            ```
+        """
+        response = self.agent(agent_message.strip(), structured_output_model=ActorResponse)
+        self._turn_count += 1
+        self._last_message = str(cast(ActorResponse, response.structured_output).message)
+        return response
+
+    def has_next(self) -> bool:
+        """
+        Check if the conversation should continue.
+
+        Returns False if the stop token (<stop/>) is present in the last message or if
+        the maximum number of turns has been reached. Use this in a loop to control
+        multi-turn conversations.
+
+        Returns:
+            True if the conversation should continue, False otherwise.
+
+        Example:
+            ```python
+            user_message = case.input
+
+            # Continue conversation until completion
+            while user_sim.has_next():
+                agent_response = agent(user_message)
+                user_result = user_sim.act(str(agent_response))
+                user_message = str(user_result.structured_output.message)
+
+            # Conversation ended either by:
+            # - Actor including <stop/> token in message
+            # - Reaching max_turns limit
+            ```
+        """
+        if self._turn_count >= self._max_turns:
+            return False
+        return "<stop/>" not in self._last_message

strands_evals/simulation/profiles/__init__.py

@@ -0,0 +1,5 @@
+"""Profile templates for actor simulation."""
+
+from .actor_profile import DEFAULT_USER_PROFILE_SCHEMA
+
+__all__ = ["DEFAULT_USER_PROFILE_SCHEMA"]

strands_evals/simulation/profiles/actor_profile.py

@@ -0,0 +1,26 @@
+"""
+Actor profile templates for simulation.
+
+This module provides actor profile structures used as templates
+for generating realistic actor profiles in conversation simulation.
+"""
+
+DEFAULT_USER_PROFILE_SCHEMA = {
+    "traits": {
+        "personal_profile": {
+            "identity": {
+                "first_name": "User",
+                "last_name": "Default",
+                "preferred_name": "User",
+                "gender": "other",
+                "birthdate": "1990-01-01",
+                "email": "user@example.com",
+            },
+            "location": {"address1": "123 Main St", "city": "Default City", "province": "CA", "country": "USA"},
+            "languages": [{"language": "English", "proficiency": "Advanced"}],
+        },
+        "persona": "Friendly and helpful user seeking assistance with general topics.",
+        "supplementary_profile": "Default user profile for simulation.",
+    },
+    "context": "some context",
+}

strands_evals/simulation/prompt_templates/__init__.py

@@ -0,0 +1,11 @@
+"""Prompt templates for actor simulation."""
+
+from .actor_profile_extraction import ACTOR_PROFILE_PROMPT_TEMPLATE
+from .actor_system_prompt import DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE
+from .goal_completion import GOAL_COMPLETION_PROMPT
+
+__all__ = [
+    "ACTOR_PROFILE_PROMPT_TEMPLATE",
+    "DEFAULT_USER_SIMULATOR_PROMPT_TEMPLATE",
+    "GOAL_COMPLETION_PROMPT",
+]

strands_evals/simulation/prompt_templates/actor_profile_extraction.py

@@ -0,0 +1,25 @@
+"""
+Prompt template for actor profile generation.
+
+This module contains the prompt template used to generate realistic actor profiles
+from scenario information for conversation simulation.
+"""
+
+from textwrap import dedent
+
+ACTOR_PROFILE_PROMPT_TEMPLATE = dedent("""Generate exactly 1 realistic actor profile for the following task:
+
+Actor's Initial Query: {initial_query}
+Tasks Description: {task_description}
+
+Generate a complete actor profile with the following structure with:
+1. Traits: Key traits (as key-value pairs)
+2. Context: Background context (as a paragraph in 2-3 sentences)
+3. Actor Goal: What the actor ultimately wants to achieve in this interaction - should be
+   specific, actionable, and written from the actor's perspective
+
+IMPORTANT: Return JSON in the following format! IT MUST HAVE THE EXACT STRUCTURE YOU SEE HERE WITH EXACTLY THESE KEYS.
+
+{example}
+
+Be specific and realistic.""")