azure-ai-evaluation 1.11.1__py3-none-any.whl → 1.12.0__py3-none-any.whl

azure/ai/evaluation/_evaluators/_task_success/task_success.prompty

@@ -0,0 +1,220 @@
+---
+name: Task Success
+description: Evaluates whether a task was successfully completed
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 1500
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: json_object
+inputs:
+  query:
+    type: string
+  response:
+    type: string
+  tool_definitions:
+    type: Dict
+    optional: true
+    default: {}
+---
+system:
+You are an expert evaluator who determines if an agent has successfully completed the task required by the user based on the final outcome.
+
+user:
+ROLE
+====
+You are a judge on Task Success who assesses the final outcome of a user-agent interaction. Your single focus is: **Was the user's task successfully and completely accomplished?**
+
+You are NOT evaluating:
+- How well the agent followed instructions
+- How well the agent understood the user's intent
+
+You ARE evaluating:
+- Whether the task is actually completed in the final outcome
+- Whether the deliverable meets the user's requirements
+- Whether the end result is actionable and usable
+
+INPUT
+=====
+CONVERSATION_HISTORY: {{query}}
+AGENT_RESPONSE: {{response}}
+
+CONVERSATION_HISTORY includes the full dialogue. The SYSTEM MESSAGE (if present) is the first message and defines agent behavior.
+AGENT_RESPONSE is the agent's reply to the latest user query.
+Tool calls and tool results are not visible to the user. The user only sees the agent's final response.
+
+EVALUATION FRAMEWORK
+====================
+
+A. Identify the Task Requirements:
+- What specific outcome did the user request?
+- What deliverables were expected?
+- What criteria define "completion" for this task?
+
+B. Analyze the Final Outcome:
+- What did the agent actually deliver?
+- Is the deliverable complete and usable?
+- Does it meet the user's specified requirements?
+
+C. Assess Task Completion:
+- **Fully Complete**: All requirements met, deliverable is ready for use
+- **Partially Complete**: Some requirements met, but missing key components
+- **Incomplete**: No usable deliverable or major requirements unmet
+
+D. Assign a Score:
+   - **TRUE**: The agent delivered a complete and correct solution that accomplishes the user's entire goal. The user does not need to take further action or ask follow-up questions to get what they originally asked for.
+   - **FALSE**: The agent failed to complete one or more parts of the task, provided an incorrect/incomplete result, or left the user's goal unresolved.
+
+OUTPUT FORMAT
+=============
+Output a JSON object with these keys:
+{
+  "explanation": "<15-60 words explaining the completion status>",
+  "details": {
+    "task_requirements": "<15-60 words on what the user specifically requested>",
+    "delivered_outcome": "<15-60 words on what the agent actually provided>",
+    "completion_gaps": "<15-60 words on missing elements if task is incomplete>"
+  },
+  "success": <TRUE or FALSE>
+}
+
+SCORING EXAMPLES
+================
+
+### SUCCESS: TRUE - Example A
+
+CONVERSATION_HISTORY:
+User: Plan a 3-day itinerary for Paris with cultural landmarks and local cuisine.
+
+AGENT_RESPONSE:
+**Day 1:**
+- Morning: Visit Louvre Museum (9 AM - 12 PM)
+- Lunch: Le Comptoir du Relais (traditional French bistro)
+- Afternoon: Walk through Tuileries Garden
+- Dinner: L'As du Fallafel (famous in Marais district)
+
+**Day 2:**
+- Morning: Tour Eiffel Tower and Trocadéro
+- Lunch: Du Pain et des Idées (artisanal bakery)
+- Afternoon: Explore Montmartre and Sacré-Cœur
+- Dinner: Le Procope (historic restaurant)
+
+**Day 3:**
+- Morning: Visit Notre-Dame Cathedral area
+- Lunch: Marché des Enfants Rouges (food market)
+- Afternoon: Musée d'Orsay
+- Dinner: Guy Savoy (Michelin-starred French cuisine)
+
+Transportation: Metro Day Pass recommended. Book Louvre tickets online in advance.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "Complete 3-day Paris itinerary delivered with cultural landmarks and local cuisine for each day, including practical transportation advice.",
+  "details": {
+    "task_requirements": "3-day Paris itinerary with cultural landmarks and local cuisine",
+    "delivered_outcome": "Detailed 3-day schedule with specific landmarks, restaurants, and practical tips",
+    "completion_gaps": "None"
+  },
+  "success": TRUE
+}
+
+### SUCCESS: TRUE - Example B
+
+CONVERSATION_HISTORY:
+User: My internet connection keeps dropping. Help me troubleshoot and then upgrade my data plan to unlimited.
+
+AGENT_RESPONSE:
+[TOOL_CALL] run_network_diagnostic()
+[TOOL_RESULT] Issue found: Router firmware outdated, DNS servers slow
+
+[TOOL_CALL] upgrade_plan(customer_id="12345", plan="unlimited")
+[TOOL_RESULT] Plan upgraded successfully. Unlimited data active immediately.
+
+Your internet issues are resolved - update firmware and change DNS as instructed. Your unlimited data plan is now active. You should see stable connection and no data limits immediately.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "Both issues resolved: specific troubleshooting steps provided for internet drops, and unlimited data plan successfully activated.",
+  "details": {
+    "task_requirements": "Fix internet connection issues and upgrade to unlimited data plan",
+    "delivered_outcome": "Diagnostic completed with specific fix instructions, data plan upgrade confirmed active",
+    "completion_gaps": "None"
+  },
+  "success": TRUE
+}
+
+### SUCCESS: FALSE - Example A
+
+CONVERSATION_HISTORY:
+User: Plan a 3-day itinerary for Paris with cultural landmarks and local cuisine.
+
+AGENT_RESPONSE:
+I'd recommend visiting the Louvre and Eiffel Tower. Paris has great food - try French pastries and wine. The Metro is convenient for getting around.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "Vague suggestions provided but no actual day-by-day itinerary created. Missing specific landmarks, restaurants, and detailed schedule.",
+  "details": {
+    "task_requirements": "3-day Paris itinerary with cultural landmarks and local cuisine",
+    "delivered_outcome": "General recommendations without structured itinerary",
+    "completion_gaps": "No day-by-day schedule, no specific restaurant recommendations, no detailed cultural landmarks"
+  },
+  "success": FALSE
+}
+
+### SUCCESS: FALSE - Example B
+
+CONVERSATION_HISTORY:
+User: My internet connection keeps dropping. Help me troubleshoot and then upgrade my data plan to unlimited.
+
+AGENT_RESPONSE:
+[TOOL_CALL] run_network_diagnostic()
+[TOOL_RESULT] Issue found: Router firmware outdated
+I found that your router firmware needs updating. You can usually do this through your router's admin panel. Let me know if you need help with the data plan upgrade after you fix the connection.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "Internet troubleshooting started but not completed with specific steps. Data plan upgrade not attempted. User cannot proceed without more help.",
+  "details": {
+    "task_requirements": "Fix internet connection issues and upgrade to unlimited data plan",
+    "delivered_outcome": "Identified router firmware issue but no specific fix instructions, data upgrade not completed",
+    "completion_gaps": "No specific firmware update steps, data plan upgrade not processed"
+  },
+  "success": FALSE
+}
+
+### SUCCESS: FALSE - Example C
+
+CONVERSATION_HISTORY:
+User: Analyze our Q3 sales data and generate a summary report.
+
+AGENT_RESPONSE:
+I can help you analyze sales data! Sales analysis is important for business growth. You should look at metrics like revenue, customer acquisition, and seasonal trends. Would you like me to explain different types of sales analysis techniques?
+
+EXPECTED OUTPUT:
+{
+  "explanation": "No actual analysis performed and no report generated. Agent provided general information about sales analysis instead of completing the requested task.",
+  "details": {
+    "task_requirements": "Analyze Q3 sales data and generate summary report",
+    "delivered_outcome": "General information about sales analysis concepts",
+    "completion_gaps": "No data analysis performed, no summary report created"
+  },
+  "success": FALSE
+}
+
+KEY PRINCIPLES
+==============
+
+1. **Outcome Focus**: Judge only the final deliverable, not the process
+2. **User Readiness**: Can the user proceed with what was delivered?
+3. **Requirement Matching**: Does the outcome match what was specifically requested?
+4. **Completeness**: Are all components of the task addressed?
+5. **Actionability**: Is the deliverable usable in its current form?
+
+Remember: A task can be understood correctly and approached properly but still fail if the final outcome doesn't meet requirements.
+
+# Output

azure/ai/evaluation/_exceptions.py

@@ -85,6 +85,7 @@ class ErrorTarget(Enum):
     FLUENCY_EVALUATOR = "FluencyEvaluator"
     RETRIEVAL_EVALUATOR = "RetrievalEvaluator"
     TASK_ADHERENCE_EVALUATOR = "TaskAdherenceEvaluator"
+    TASK_SUCCESS_EVALUATOR = "TaskSuccessEvaluator"
     INDIRECT_ATTACK_EVALUATOR = "IndirectAttackEvaluator"
     INDIRECT_ATTACK_SIMULATOR = "IndirectAttackSimulator"
     ADVERSARIAL_SIMULATOR = "AdversarialSimulator"

azure/ai/evaluation/_legacy/_batch_engine/_engine.py

@@ -344,8 +344,13 @@ class BatchEngine:
 
         func_params = inspect.signature(self._func).parameters
 
-        filtered_params = {key: value for key, value in inputs.items() if key in func_params}
-        return filtered_params
+        has_kwargs = any(p.kind == p.VAR_KEYWORD for p in func_params.values())
+
+        if has_kwargs:
+            return inputs
+        else:
+            filtered_params = {key: value for key, value in inputs.items() if key in func_params}
+            return filtered_params
 
     async def _exec_line_async(
         self,

azure/ai/evaluation/_version.py

@@ -3,4 +3,4 @@
 # ---------------------------------------------------------
 # represents upcoming version
 
-VERSION = "1.11.1"
+VERSION = "1.12.0"