rasa-pro 3.11.4__py3-none-any.whl → 3.12.0__py3-none-any.whl

rasa/e2e_test/llm_judge_prompts/answer_relevance_prompt_template.jinja2

@@ -0,0 +1,93 @@
+You are a detail-oriented judge whose task is to generate {{ num_variations }} question variations based on the provided answer.
+Each question variation should be a question that the input answer is attempting to address.
+
+## Task Steps
+
+Follow these steps to complete the task:
+
+1. Generate {{ num_variations }} question variations that the input answer is attempting to answer.
+3. Compile all the question variations  into a JSON object.
+
+### JSON Output
+
+Format your output as a JSON object with the following attribute:
+
+    - `question_variations`: a list of strings, where each string represents the generated question that the input answer is addressing.
+
+    Example:
+
+    ```json
+    {
+        "question_variations": [
+             "Does Rasa Pro support building an AI assistant with LLMs?",
+             "In which version of Rasa Pro was support for LLM usage released?",
+        ]
+    }
+    ```
+
+### Task Requirements
+
+- Ensure that each question variation is relevant to the input answer.
+
+### Examples
+
+These are a few examples of how to generate question variations and identify non-committed answers:
+
+#### Example 1
+    - **Input Answer**:
+        ```
+        You can build a CALM assistant with Rasa Pro by defining your own business logic flows.
+        I could not identify the exact version of Rasa Pro which released CALM.
+        ```
+    - **Output**:
+        ```json
+        {
+            "question_variations": [
+                "How can you build a CALM assistant with Rasa Pro?",
+                "What is the version of Rasa Pro that released CALM?",
+                "What type of assistant can you build with Rasa Pro?",
+            ]
+        }
+        ```
+
+#### Example 2:
+    - **Input Answer**:
+        ```
+        You can integrate external knowledge sources into your Rasa Pro assistant.
+        ```
+    - **Output**:
+        ```json
+        {
+            "question_variations": [
+                "Can you integrate external knowledge sources into your Rasa Pro assistant?",
+            ]
+        }
+        ```
+
+
+#### Example 3:
+    - **Input Answer**:
+        ```
+        Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI, in version 3.7.0.
+        ```
+    - **Output**:
+        ```json
+        {
+            "question_variations": [
+                "What is CALM in Rasa Pro?",
+                "In which Rasa Pro version was CALM released?",
+                "What is the LLM-native approach in Rasa Pro?",
+            ]
+        }
+        ```
+
+
+## Task Inputs
+
+- **Input Answer**: {{ bot_message }}
+
+## Task Outputs
+
+Do not include any additional explanations in your output. Only provide the JSON object as described in the task steps.
+
+Your output:

rasa/e2e_test/llm_judge_prompts/groundedness_prompt_template.jinja2

@@ -0,0 +1,169 @@
+You are a detail-oriented impartial judge whose task is to evaluate the correctness of a given answer based on the provided ground truth.
+
+## Task Steps
+
+Follow these steps to complete the task:
+
+1. Given the input answer, break down the answer into one or more sentences.
+2. Extract from each sentence one or more fully understandable and distinct statements while also ensuring no pronouns are used in each statement.
+3. For each statement, determine whether the statement is supported by the provided ground truth. If the statement is supported,
+mark it with the score `1`. If the statement is not supported, mark it with the score `0`.
+4. Provide a justification for the score you assigned. Each justification must reference the relevant part of the ground truth.
+5. Compile all the statements, scores, and justifications into a JSON object.
+
+### JSON Output
+
+Format your output as a JSON object with the following attribute:
+
+    - `statements`: a list of dictionaries, where each dictionary contains the following keys:
+        - `statement`: the extracted statement from the input answer
+        - `score`: the score assigned to the statement (0 or 1)
+        - `justification`: a brief and distinct explanation justifying the score assigned to the statement
+
+    Example:
+
+    ```json
+    {
+        "statements": [
+            {
+                "statement": "You can run Rasa Pro using Python 3.11 starting from Rasa Pro version 3.11.",
+                "score": 1,
+                "justification": "According to the ground truth source, support for Python 3.11 was added in Rasa Pro version 3.11"
+            },
+            {
+                "statement": "Rasa Pro does not support LLM usage in any version.",
+                "score": 0,
+                "justification": "According to the ground truth source, Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI, in version 3.7.0"
+            }
+        ]
+    }
+    ```
+
+### Task Requirements
+
+- If the answer contains multiple statements, ensure that each statement is evaluated independently.
+- If the answer contains a statement that is not verifiable by the ground truth, mark the statement as unsupported with the score `0`.
+- If the answer contains a statement that is verifiable by the ground truth, mark the statement as supported with the score `1`.
+- Provide a brief justification for each score assigned to a statement.
+
+### Examples
+
+These are a few examples of how to evaluate the correctness of the answer based on the ground truth:
+
+#### Example 1
+    - **Input Answer**:
+        ```
+        You can build a CALM assistant with Rasa Pro by defining your own business logic flows.
+        In addition, CALM leverages customizable default flows designed to handle various conversational repair scenarios.
+        CALM contains a built-in LLM approach designed to generate predefined commands that
+        reflect the user's intentions to start and stop flows, fill slots and more.
+        ```
+    - **Ground Truth**:
+        ```
+        Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI, in version 3.7.0.
+        The CALM approach has three key elements: Business Logic, Dialogue Understanding, and Automatic Conversation Repair.
+        Business logic is implemented as a set of flows. A flow describes a business process that your AI assistant can handle.
+        Dialogue understanding is designed to interpret what end users are communicating to your assistant.
+        This process involves generating commands that reflect the user's intentions, by starting and stopping flows, filling slots and more.
+        Automatic conversation repair handles all the ways conversations can go "off script". This is implemented as a set of default flows open for customization.
+        ```
+    - **Output**:
+        ```json
+        {
+            "statements": [
+                {
+                    "statement": "You can build a CALM assistant with Rasa Pro.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI"
+                },
+                {
+                    "statement": "You can define your own business logic as flows in a CALM assistant with Rasa Pro.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that business logic is implemented as a set of flows in CALM"
+                },
+                {
+                    "statement": "Conversation repair scenarios are handled by customizable default flows in CALM.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that automatic conversation repair is implemented as a set of default flows open for customization"
+                },
+                {
+                    "statement": "CALM contains a built-in LLM approach.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that CALM leverages an LLM-native approach"
+                },
+                {
+                    "statement": "The LLM approach in CALM generates predefined commands reflecting user intentions.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that dialogue understanding involves generating commands reflecting user intentions"
+                },
+                {
+                    "statement": "The LLM approach in CALM is designed to start and stop flows, fill slots, and more.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that dialogue understanding involves generating commands to start and stop flows, fill slots, and more"
+                }
+            ]
+        }
+        ```
+
+#### Example 2:
+    - **Input Answer**:
+        ```
+        You cannot integrate external knowledge into your Rasa Pro assistant.
+        ```
+    - **Ground Truth**:
+        ```
+        The Enterprise Search Policy is part of Rasa's new Conversational AI with Language Models (CALM) approach and available starting with version 3.7.0.
+        The Enterprise Search Policy uses an LLM to search external knowledge base documents in order to deliver a relevant, context-aware response from the data.
+        ```
+    - **Output**:
+        ```json
+        {
+            "statements": [
+                {
+                    "statement": "Rasa Pro does not support integrating external knowledge.",
+                    "score": 0,
+                    "justification": "The provided statement is incorrect, because the ground truth confirms that the Enterprise Search Policy in Rasa Pro's CALM approach uses an LLM to search external knowledge base documents"
+                }
+            ]
+        }
+        ```
+
+
+#### Example 3:
+    - **Input Answer**:
+        ```
+        Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI, in version 3.6.0.
+        ```
+    - **Ground Truth**:
+        ```
+        Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI, in version 3.7.0.
+        ```
+    - **Output**:
+        ```json
+        {
+            "statements": [
+                {
+                    "statement": "CALM is Rasa Pro's LLM-native approach to building reliable conversational AI.",
+                    "score": 1,
+                    "justification": "The ground truth confirms that Rasa Pro has released CALM, its LLM-native approach to building reliable conversational AI"
+                },
+                {
+                    "statement": "CALM was released in Rasa Pro version 3.6.0.",
+                    "score": 0,
+                    "justification": "The provided statement is incorrect, as the ground truth confirms that CALM was released in Rasa Pro version 3.7.0"
+                }
+            ]
+        }
+        ```
+
+
+## Task Inputs
+
+- **Input Answer**: {{ bot_message }}
+- **Ground Truth**: {{ ground_truth }}
+
+## Task Outputs
+
+Do not include any additional explanations in your output. Only provide the JSON object as described in the task steps.
+
+Your output:

rasa/e2e_test/stub_custom_action.py

@@ -1,5 +1,5 @@
 from dataclasses import dataclass
-from typing import Any, Dict, List, Text, Optional
+from typing import Any, Dict, List, Optional, Text
 
 from rasa.e2e_test.constants import (
     KEY_STUB_CUSTOM_ACTIONS,

rasa/e2e_test/utils/generative_assertions.py

@@ -0,0 +1,243 @@
+import json
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
+
+import jsonschema
+import numpy as np
+import structlog
+from pydantic import BaseModel, ConfigDict
+
+from rasa.core.constants import (
+    UTTER_SOURCE_METADATA_KEY,
+)
+from rasa.e2e_test.constants import (
+    KEY_JUSTIFICATION,
+    KEY_SCORE,
+)
+from rasa.e2e_test.e2e_config import LLMJudgeConfig
+from rasa.shared.constants import MODEL_CONFIG_KEY, OPENAI_PROVIDER, PROVIDER_CONFIG_KEY
+from rasa.shared.core.events import BotUttered
+from rasa.shared.exceptions import RasaException
+from rasa.shared.utils.llm import DEFAULT_OPENAI_EMBEDDING_MODEL_NAME, embedder_factory
+
+if TYPE_CHECKING:
+    from rasa.shared.core.events import Event
+
+
+structlogger = structlog.get_logger()
+
+
+DEFAULT_EMBEDDINGS_CONFIG = {
+    PROVIDER_CONFIG_KEY: OPENAI_PROVIDER,
+    MODEL_CONFIG_KEY: DEFAULT_OPENAI_EMBEDDING_MODEL_NAME,
+}
+
+ELIGIBLE_UTTER_SOURCE_METADATA = [
+    "EnterpriseSearchPolicy",
+    "ContextualResponseRephraser",
+    "IntentlessPolicy",
+]
+
+GROUNDEDNESS_JSON_SUB_SCHEMA = {
+    "properties": {
+        "statements": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "properties": {
+                    "statement": {"type": "string"},
+                    "score": {"type": "number"},
+                    "justification": {"type": "string"},
+                },
+                "required": ["statement", "score", "justification"],
+            },
+        }
+    },
+    "required": ["statements"],
+}
+
+ANSWER_RELEVANCE_JSON_SUB_SCHEMA = {
+    "properties": {
+        "question_variations": {
+            "type": "array",
+            "items": {"type": "string"},
+        },
+    },
+    "required": ["question_variations"],
+}
+
+LLM_JUDGE_OUTPUT_JSON_SCHEMA = {
+    "type": "object",
+    "oneOf": [
+        GROUNDEDNESS_JSON_SUB_SCHEMA,
+        ANSWER_RELEVANCE_JSON_SUB_SCHEMA,
+    ],
+}
+
+
+class ScoreInputs(BaseModel):
+    """Input data for the score calculation."""
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    threshold: float
+    matching_event: BotUttered
+    user_question: str
+    llm_judge_config: LLMJudgeConfig
+
+
+def _calculate_similarity(
+    user_question: str,
+    generated_questions: List[str],
+    llm_judge_config: LLMJudgeConfig,
+) -> np.ndarray:
+    """Calculate the cosine similarity between the user question and the generated questions."""  # noqa: E501
+    embedding_client = embedder_factory(
+        llm_judge_config.embeddings_config_as_dict, DEFAULT_EMBEDDINGS_CONFIG
+    )
+
+    user_question_embedding_response = embedding_client.embed([user_question])
+    question_vector = np.asarray(user_question_embedding_response.data[0]).reshape(
+        1, -1
+    )
+
+    gen_questions_embedding_response = embedding_client.embed(generated_questions)
+    generated_questions_vectors = np.asarray(
+        gen_questions_embedding_response.data
+    ).reshape(len(generated_questions), -1)
+
+    # calculate norm
+    question_vector_norm = np.linalg.norm(question_vector, axis=1)
+    generated_questions_vectors_norm = np.linalg.norm(
+        generated_questions_vectors, axis=1
+    )
+    norm = generated_questions_vectors_norm * question_vector_norm
+    norm = np.maximum(norm, 1e-10)
+
+    # calculate the dot product
+    dot_product = np.dot(generated_questions_vectors, question_vector.T).reshape(-1)
+
+    # calculate and return cosine similarity
+    return dot_product / norm
+
+
+def calculate_relevance_score(
+    processed_output: List[Union[str, Dict[str, Any]]],
+    score_inputs: ScoreInputs,
+) -> Tuple[float, str]:
+    """Calculate the score based on the LLM response."""
+    user_question = score_inputs.user_question
+    llm_judge_config = score_inputs.llm_judge_config
+
+    generated_questions = [output for output in processed_output]
+    if all(not question for question in generated_questions):
+        score = 0.0
+        error_justification = "No relevant questions were generated"
+        return score, error_justification
+
+    cosine_sim = _calculate_similarity(
+        user_question, generated_questions, llm_judge_config
+    )
+
+    score = cosine_sim.mean()
+
+    if score < score_inputs.threshold:
+        error_justifications = [
+            f"Question '{generated_questions[i]}' "
+            f"has a cosine similarity score of '{round(cosine_sim[i], 2)}' "
+            f"with the user question '{user_question}'"
+            for i in range(len(generated_questions))
+        ]
+        error_justification = ", ".join(error_justifications)
+        return score, error_justification
+
+    return score, ""
+
+
+def calculate_groundedness_score(
+    processed_output: List[Any],
+    score_inputs: ScoreInputs,
+) -> Tuple[float, str]:
+    """Calculate the score based on the LLM response."""
+    matching_event = score_inputs.matching_event
+
+    total_statements = len(processed_output)
+    correct_statements = sum([output.get(KEY_SCORE, 0) for output in processed_output])
+    score = correct_statements / total_statements
+
+    structlogger.debug(
+        "generative_response_is_grounded_assertion.run_results",
+        matching_event=repr(matching_event),
+        score=score,
+        justification=f"There were {correct_statements} correct statements "
+        f"out of {total_statements} total extracted statements.",
+    )
+
+    if score < score_inputs.threshold:
+        justifications = [
+            output.get(KEY_JUSTIFICATION, "")
+            for output in processed_output
+            if output.get(KEY_SCORE, 0) == 0
+        ]
+        justification = ", ".join(justifications).replace(".", "")
+
+        error_justification = (
+            f"There were {total_statements - correct_statements} "
+            f"incorrect statements out of {total_statements} total "
+            f"extracted statements. The justifications for "
+            f"these statements include: {justification}"
+        )
+
+        return score, error_justification
+
+    return score, ""
+
+
+def _find_matching_generative_events(
+    turn_events: List["Event"], utter_source: Optional[str]
+) -> List[BotUttered]:
+    """Find the matching events for the generative response assertions."""
+    if utter_source is None:
+        return [
+            event
+            for event in turn_events
+            if isinstance(event, BotUttered)
+            and event.metadata.get(UTTER_SOURCE_METADATA_KEY)
+            in ELIGIBLE_UTTER_SOURCE_METADATA
+        ]
+
+    return [
+        event
+        for event in turn_events
+        if isinstance(event, BotUttered)
+        and event.metadata.get(UTTER_SOURCE_METADATA_KEY) == utter_source
+    ]
+
+
+def _parse_llm_output(llm_response: str, bot_message: str) -> Dict[str, Any]:
+    """Parse the LLM output."""
+    llm_output = (
+        llm_response.replace("```json\n", "").replace("```", "").replace("\n", "")
+    )
+    try:
+        parsed_llm_output = json.loads(llm_output)
+    except json.JSONDecodeError as exc:
+        raise RasaException(
+            f"Failed to parse the LLM Judge response '{llm_output}' for "
+            f"the generative bot message '{bot_message}': {exc}"
+        )
+
+    return parsed_llm_output
+
+
+def _validate_parsed_llm_output(
+    parsed_llm_output: Dict[str, Any], bot_message: str
+) -> None:
+    """Validate the parsed LLM output."""
+    try:
+        jsonschema.validate(parsed_llm_output, LLM_JUDGE_OUTPUT_JSON_SCHEMA)
+    except jsonschema.ValidationError as exc:
+        raise RasaException(
+            f"Failed to validate the LLM Judge json response "
+            f"'{parsed_llm_output}' for the generative bot message "
+            f"'{bot_message}'. Error: {exc}"
+        )