agentv 1.3.1 → 1.5.0

package/dist/templates/.claude/skills/agentv-eval-builder/references/rubric-evaluator.md

@@ -1,139 +1,139 @@
-# Rubric Evaluator Guide
-
-Rubrics provide structured evaluation through lists of criteria that define what makes a good response. Rubrics are checked by an LLM judge and scored based on weights and requirements.
-
-## Basic Usage
-
-### Simple String Rubrics
-
-Define rubrics as simple strings - each becomes a required criterion with weight 1.0:
-
-```yaml
-$schema: agentv-eval-v2
-
-evalcases:
-  - id: quicksort-explanation
-    expected_outcome: Explain how quicksort works
-    
-    input_messages:
-      - role: user
-        content: Explain how the quicksort algorithm works
-    
-    rubrics:
-      - Mentions divide-and-conquer approach
-      - Explains the partition step
-      - States time complexity correctly
-```
-
-### Detailed Rubric Objects
-
-Use objects for fine-grained control over weights and requirements:
-
-```yaml
-evalcases:
-  - id: technical-guide
-    expected_outcome: Write a comprehensive HTTP status codes guide
-    
-    input_messages:
-      - role: user
-        content: Write a guide explaining HTTP status codes
-    
-    rubrics:
-      - id: structure
-        description: Has clear headings and organization
-        weight: 1.0
-        required: true
-        
-      - id: success-codes
-        description: Covers 2xx success codes with examples
-        weight: 2.0
-        required: true
-        
-      - id: client-errors
-        description: Explains 4xx client error codes
-        weight: 2.0
-        required: true
-        
-      - id: server-errors
-        description: Explains 5xx server error codes
-        weight: 1.5
-        required: false
-        
-      - id: practical-examples
-        description: Includes practical use case examples
-        weight: 1.0
-        required: false
-```
-
-## Rubric Object Fields
-
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `id` | string | auto-generated | Unique identifier for the rubric |
-| `description` | string | required | The criterion being evaluated |
-| `weight` | number | 1.0 | Relative importance (higher = more impact on score) |
-| `required` | boolean | true | If true, failing this rubric forces verdict to 'fail' |
-
-## Scoring and Verdicts
-
-**Score Calculation:**
-```
-score = (sum of satisfied weights) / (total weights)
-```
-
-**Verdict Rules:**
-- `pass`: Score ≥ 0.8 AND all required rubrics satisfied
-- `borderline`: Score ≥ 0.6 AND all required rubrics satisfied  
-- `fail`: Score < 0.6 OR any required rubric failed
-
-## Combining Rubrics with Other Evaluators
-
-Rubrics can be combined with code evaluators for comprehensive validation:
-
-```yaml
-evalcases:
-  - id: email-validator
-    expected_outcome: Python function to validate email addresses
-    
-    input_messages:
-      - role: user
-        content: Write a Python function to validate email addresses
-    
-    # Semantic evaluation via rubrics
-    rubrics:
-      - Uses regular expressions for validation
-      - Includes type hints
-      - Has docstring documentation
-      - Handles edge cases (None, empty string)
-    
-    execution:
-      evaluators:
-        # Rubric evaluator is auto-added from inline rubrics field
-        
-        # Additional code evaluator for syntax checking
-        - name: python_syntax
-          type: code_judge
-          script: uv run python -m py_compile
-```
-
-## Generate Rubrics from Expected Outcome
-
-Use the CLI to auto-generate rubrics from `expected_outcome`:
-
-```bash
-# Generate rubrics for eval cases that don't have them
-agentv generate rubrics evals/my-eval.yaml
-
-# Use a specific LLM target for generation
-agentv generate rubrics evals/my-eval.yaml --target azure_base
-```
-
-This analyzes each `expected_outcome` and creates appropriate rubric items.
-
-## Best Practices
-
-1. **Use required sparingly** - Only mark rubrics as `required: true` for critical criteria
-2. **Balance weights** - Use higher weights (2.0+) for core requirements, lower (0.5) for nice-to-haves
-3. **Be specific** - "Includes error handling" is better than "Good code quality"
-4. **Keep rubrics atomic** - Each rubric should test one thing
-5. **Consider partial credit** - Non-required rubrics allow partial scores
+# Rubric Evaluator Guide
+
+Rubrics provide structured evaluation through lists of criteria that define what makes a good response. Rubrics are checked by an LLM judge and scored based on weights and requirements.
+
+## Basic Usage
+
+### Simple String Rubrics
+
+Define rubrics as simple strings - each becomes a required criterion with weight 1.0:
+
+```yaml
+$schema: agentv-eval-v2
+
+evalcases:
+  - id: quicksort-explanation
+    expected_outcome: Explain how quicksort works
+    
+    input_messages:
+      - role: user
+        content: Explain how the quicksort algorithm works
+    
+    rubrics:
+      - Mentions divide-and-conquer approach
+      - Explains the partition step
+      - States time complexity correctly
+```
+
+### Detailed Rubric Objects
+
+Use objects for fine-grained control over weights and requirements:
+
+```yaml
+evalcases:
+  - id: technical-guide
+    expected_outcome: Write a comprehensive HTTP status codes guide
+    
+    input_messages:
+      - role: user
+        content: Write a guide explaining HTTP status codes
+    
+    rubrics:
+      - id: structure
+        description: Has clear headings and organization
+        weight: 1.0
+        required: true
+        
+      - id: success-codes
+        description: Covers 2xx success codes with examples
+        weight: 2.0
+        required: true
+        
+      - id: client-errors
+        description: Explains 4xx client error codes
+        weight: 2.0
+        required: true
+        
+      - id: server-errors
+        description: Explains 5xx server error codes
+        weight: 1.5
+        required: false
+        
+      - id: practical-examples
+        description: Includes practical use case examples
+        weight: 1.0
+        required: false
+```
+
+## Rubric Object Fields
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `id` | string | auto-generated | Unique identifier for the rubric |
+| `description` | string | required | The criterion being evaluated |
+| `weight` | number | 1.0 | Relative importance (higher = more impact on score) |
+| `required` | boolean | true | If true, failing this rubric forces verdict to 'fail' |
+
+## Scoring and Verdicts
+
+**Score Calculation:**
+```
+score = (sum of satisfied weights) / (total weights)
+```
+
+**Verdict Rules:**
+- `pass`: Score ≥ 0.8 AND all required rubrics satisfied
+- `borderline`: Score ≥ 0.6 AND all required rubrics satisfied  
+- `fail`: Score < 0.6 OR any required rubric failed
+
+## Combining Rubrics with Other Evaluators
+
+Rubrics can be combined with code evaluators for comprehensive validation:
+
+```yaml
+evalcases:
+  - id: email-validator
+    expected_outcome: Python function to validate email addresses
+    
+    input_messages:
+      - role: user
+        content: Write a Python function to validate email addresses
+    
+    # Semantic evaluation via rubrics
+    rubrics:
+      - Uses regular expressions for validation
+      - Includes type hints
+      - Has docstring documentation
+      - Handles edge cases (None, empty string)
+    
+    execution:
+      evaluators:
+        # Rubric evaluator is auto-added from inline rubrics field
+        
+        # Additional code evaluator for syntax checking
+        - name: python_syntax
+          type: code_judge
+          script: uv run python -m py_compile
+```
+
+## Generate Rubrics from Expected Outcome
+
+Use the CLI to auto-generate rubrics from `expected_outcome`:
+
+```bash
+# Generate rubrics for eval cases that don't have them
+agentv generate rubrics evals/my-eval.yaml
+
+# Use a specific LLM target for generation
+agentv generate rubrics evals/my-eval.yaml --target azure_base
+```
+
+This analyzes each `expected_outcome` and creates appropriate rubric items.
+
+## Best Practices
+
+1. **Use required sparingly** - Only mark rubrics as `required: true` for critical criteria
+2. **Balance weights** - Use higher weights (2.0+) for core requirements, lower (0.5) for nice-to-haves
+3. **Be specific** - "Includes error handling" is better than "Good code quality"
+4. **Keep rubrics atomic** - Each rubric should test one thing
+5. **Consider partial credit** - Non-required rubrics allow partial scores

package/dist/templates/.claude/skills/agentv-eval-builder/references/tool-trajectory-evaluator.md

@@ -1,179 +1,198 @@
-# Tool Trajectory Evaluator Guide
-
-Tool trajectory evaluators validate that an agent used the expected tools during execution. They work with trace data returned by agent providers (codex, vscode, cli with trace support).
-
-## Tool Trajectory Evaluator
-
-### Modes
-
-#### 1. `any_order` - Minimum Tool Counts
-
-Validates that each tool was called at least N times, regardless of order:
-
-```yaml
-execution:
-  evaluators:
-    - name: tool-usage
-      type: tool_trajectory
-      mode: any_order
-      minimums:
-        knowledgeSearch: 2    # Must be called at least twice
-        documentRetrieve: 1   # Must be called at least once
-```
-
-**Use cases:**
-- Ensure required tools are used
-- Don't care about execution order
-- Allow flexibility in agent implementation
-
-#### 2. `in_order` - Sequential Matching
-
-Validates tools appear in the expected sequence, but allows gaps (other tools can appear between):
-
-```yaml
-execution:
-  evaluators:
-    - name: workflow-sequence
-      type: tool_trajectory
-      mode: in_order
-      expected:
-        - tool: fetchData
-        - tool: validateSchema
-        - tool: transformData
-        - tool: saveResults
-```
-
-**Use cases:**
-- Validate logical workflow order
-- Allow agent to use additional helper tools
-- Check that key steps happen in sequence
-
-#### 3. `exact` - Strict Sequence Match
-
-Validates the exact tool sequence with no gaps or extra tools:
-
-```yaml
-execution:
-  evaluators:
-    - name: auth-sequence
-      type: tool_trajectory
-      mode: exact
-      expected:
-        - tool: checkCredentials
-        - tool: generateToken
-        - tool: auditLog
-```
-
-**Use cases:**
-- Security-critical workflows
-- Strict protocol validation
-- Regression testing specific behavior
-
-## Scoring
-
-### tool_trajectory Scoring
-
-| Mode | Score Calculation |
-|------|------------------|
-| `any_order` | (tools meeting minimum) / (total tools with minimums) |
-| `in_order` | (matched tools in sequence) / (expected tools count) |
-| `exact` | (correctly positioned tools) / (expected tools count) |
-
-## Trace Data Requirements
-
-Tool trajectory evaluators require trace data from the agent provider. Supported providers:
-
-- **codex** - Returns trace via JSONL log events
-- **vscode / vscode-insiders** - Returns trace from Copilot execution
-- **cli** - Can return trace if agent outputs trace format
-
-### Trace Event Structure
-
-```json
-{
-  "type": "tool_call",
-  "name": "knowledgeSearch",
-  "input": { "query": "REST vs GraphQL" },
-  "timestamp": "2024-01-15T10:30:00Z"
-}
-```
-
-## Complete Examples
-
-### Research Agent Validation
-
-```yaml
-$schema: agentv-eval-v2
-description: Validate research agent tool usage
-
-target: codex_agent  # Provider that returns traces
-
-evalcases:
-  - id: comprehensive-research
-    expected_outcome: Agent thoroughly researches the topic
-    
-    input_messages:
-      - role: user
-        content: Research machine learning frameworks
-    
-    execution:
-      evaluators:
-        # Check minimum tool usage
-        - name: coverage
-          type: tool_trajectory
-          mode: any_order
-          minimums:
-            webSearch: 1
-            documentRead: 2
-            noteTaking: 1
-        
-        # Check workflow order
-        - name: workflow
-          type: tool_trajectory
-          mode: in_order
-          expected:
-            - tool: webSearch
-            - tool: documentRead
-            - tool: summarize
-```
-
-### Multi-Step Pipeline
-
-```yaml
-evalcases:
-  - id: data-pipeline
-    expected_outcome: Process data through complete pipeline
-
-    input_messages:
-      - role: user
-        content: Process the customer dataset
-
-    execution:
-      evaluators:
-        - name: pipeline-check
-          type: tool_trajectory
-          mode: exact
-          expected:
-            - tool: loadData
-            - tool: validate
-            - tool: transform
-            - tool: export
-```
-
-## CLI Options for Traces
-
-```bash
-# Write trace files to disk
-agentv eval evals/test.yaml --dump-traces
-
-# Include full trace in result output
-agentv eval evals/test.yaml --include-trace
-```
-
-## Best Practices
-
-1. **Choose the right mode** - Use `any_order` for flexibility, `exact` for strict validation
-2. **Start with any_order** - Then tighten to `in_order` or `exact` as needed
-3. **Combine with other evaluators** - Use tool trajectory for execution, LLM judge for output quality
-4. **Test with --dump-traces** - Inspect actual traces to understand agent behavior
-5. **Use code evaluators for custom validation** - Write custom tool validation scripts with access to trace data
+# Tool Trajectory Evaluator Guide
+
+Tool trajectory evaluators validate that an agent used the expected tools during execution. They work with trace data returned by agent providers (codex, vscode, cli with trace support).
+
+## Tool Trajectory Evaluator
+
+### Modes
+
+#### 1. `any_order` - Minimum Tool Counts
+
+Validates that each tool was called at least N times, regardless of order:
+
+```yaml
+execution:
+  evaluators:
+    - name: tool-usage
+      type: tool_trajectory
+      mode: any_order
+      minimums:
+        knowledgeSearch: 2    # Must be called at least twice
+        documentRetrieve: 1   # Must be called at least once
+```
+
+**Use cases:**
+- Ensure required tools are used
+- Don't care about execution order
+- Allow flexibility in agent implementation
+
+#### 2. `in_order` - Sequential Matching
+
+Validates tools appear in the expected sequence, but allows gaps (other tools can appear between):
+
+```yaml
+execution:
+  evaluators:
+    - name: workflow-sequence
+      type: tool_trajectory
+      mode: in_order
+      expected:
+        - tool: fetchData
+        - tool: validateSchema
+        - tool: transformData
+        - tool: saveResults
+```
+
+**Use cases:**
+- Validate logical workflow order
+- Allow agent to use additional helper tools
+- Check that key steps happen in sequence
+
+#### 3. `exact` - Strict Sequence Match
+
+Validates the exact tool sequence with no gaps or extra tools:
+
+```yaml
+execution:
+  evaluators:
+    - name: auth-sequence
+      type: tool_trajectory
+      mode: exact
+      expected:
+        - tool: checkCredentials
+        - tool: generateToken
+        - tool: auditLog
+```
+
+**Use cases:**
+- Security-critical workflows
+- Strict protocol validation
+- Regression testing specific behavior
+
+## Scoring
+
+### tool_trajectory Scoring
+
+| Mode | Score Calculation |
+|------|------------------|
+| `any_order` | (tools meeting minimum) / (total tools with minimums) |
+| `in_order` | (matched tools in sequence) / (expected tools count) |
+| `exact` | (correctly positioned tools) / (expected tools count) |
+
+## Trace Data Requirements
+
+Tool trajectory evaluators require trace data from the agent provider. Providers return `output_messages` containing `tool_calls` that capture agent tool usage.
+
+### Output Messages Format
+
+Providers return `output_messages` with `tool_calls` in the JSONL output:
+
+```json
+{
+  "id": "eval-001",
+  "output_messages": [
+    {
+      "role": "assistant",
+      "content": "I'll search for information about this topic.",
+      "tool_calls": [
+        {
+          "tool": "knowledgeSearch",
+          "input": { "query": "REST vs GraphQL" },
+          "output": { "results": [...] },
+          "id": "call_123",
+          "timestamp": "2024-01-15T10:30:00Z"
+        }
+      ]
+    }
+  ]
+}
+```
+
+The evaluator extracts tool calls from `output_messages[].tool_calls[]`. Optional fields `id` and `timestamp` can be included for debugging.
+
+### Supported Providers
+
+- **codex** - Returns output_messages via JSONL log events
+- **vscode / vscode-insiders** - Returns output_messages from Copilot execution
+- **cli** - Returns `output_messages` with `tool_calls`
+
+## Complete Examples
+
+### Research Agent Validation
+
+```yaml
+$schema: agentv-eval-v2
+description: Validate research agent tool usage
+
+execution:
+  target: codex_agent  # Provider that returns traces
+
+evalcases:
+  - id: comprehensive-research
+    expected_outcome: Agent thoroughly researches the topic
+    
+    input_messages:
+      - role: user
+        content: Research machine learning frameworks
+    
+    execution:
+      evaluators:
+        # Check minimum tool usage
+        - name: coverage
+          type: tool_trajectory
+          mode: any_order
+          minimums:
+            webSearch: 1
+            documentRead: 2
+            noteTaking: 1
+        
+        # Check workflow order
+        - name: workflow
+          type: tool_trajectory
+          mode: in_order
+          expected:
+            - tool: webSearch
+            - tool: documentRead
+            - tool: summarize
+```
+
+### Multi-Step Pipeline
+
+```yaml
+evalcases:
+  - id: data-pipeline
+    expected_outcome: Process data through complete pipeline
+
+    input_messages:
+      - role: user
+        content: Process the customer dataset
+
+    execution:
+      evaluators:
+        - name: pipeline-check
+          type: tool_trajectory
+          mode: exact
+          expected:
+            - tool: loadData
+            - tool: validate
+            - tool: transform
+            - tool: export
+```
+
+## CLI Options for Traces
+
+```bash
+# Write trace files to disk
+agentv eval evals/test.yaml --dump-traces
+
+# Include full trace in result output
+agentv eval evals/test.yaml --include-trace
+```
+
+## Best Practices
+
+1. **Choose the right mode** - Use `any_order` for flexibility, `exact` for strict validation
+2. **Start with any_order** - Then tighten to `in_order` or `exact` as needed
+3. **Combine with other evaluators** - Use tool trajectory for execution, LLM judge for output quality
+4. **Test with --dump-traces** - Inspect actual traces to understand agent behavior
+5. **Use code evaluators for custom validation** - Write custom tool validation scripts with access to trace data