agentv 1.3.1 → 1.6.1

package/dist/templates/.claude/skills/agentv-eval-builder/references/tool-trajectory-evaluator.md

@@ -1,179 +1,224 @@
-# Tool Trajectory Evaluator Guide
-
-Tool trajectory evaluators validate that an agent used the expected tools during execution. They work with trace data returned by agent providers (codex, vscode, cli with trace support).
-
-## Tool Trajectory Evaluator
-
-### Modes
-
-#### 1. `any_order` - Minimum Tool Counts
-
-Validates that each tool was called at least N times, regardless of order:
-
-```yaml
-execution:
-  evaluators:
-    - name: tool-usage
-      type: tool_trajectory
-      mode: any_order
-      minimums:
-        knowledgeSearch: 2    # Must be called at least twice
-        documentRetrieve: 1   # Must be called at least once
-```
-
-**Use cases:**
-- Ensure required tools are used
-- Don't care about execution order
-- Allow flexibility in agent implementation
-
-#### 2. `in_order` - Sequential Matching
-
-Validates tools appear in the expected sequence, but allows gaps (other tools can appear between):
-
-```yaml
-execution:
-  evaluators:
-    - name: workflow-sequence
-      type: tool_trajectory
-      mode: in_order
-      expected:
-        - tool: fetchData
-        - tool: validateSchema
-        - tool: transformData
-        - tool: saveResults
-```
-
-**Use cases:**
-- Validate logical workflow order
-- Allow agent to use additional helper tools
-- Check that key steps happen in sequence
-
-#### 3. `exact` - Strict Sequence Match
-
-Validates the exact tool sequence with no gaps or extra tools:
-
-```yaml
-execution:
-  evaluators:
-    - name: auth-sequence
-      type: tool_trajectory
-      mode: exact
-      expected:
-        - tool: checkCredentials
-        - tool: generateToken
-        - tool: auditLog
-```
-
-**Use cases:**
-- Security-critical workflows
-- Strict protocol validation
-- Regression testing specific behavior
-
-## Scoring
-
-### tool_trajectory Scoring
-
-| Mode | Score Calculation |
-|------|------------------|
-| `any_order` | (tools meeting minimum) / (total tools with minimums) |
-| `in_order` | (matched tools in sequence) / (expected tools count) |
-| `exact` | (correctly positioned tools) / (expected tools count) |
-
-## Trace Data Requirements
-
-Tool trajectory evaluators require trace data from the agent provider. Supported providers:
-
-- **codex** - Returns trace via JSONL log events
-- **vscode / vscode-insiders** - Returns trace from Copilot execution
-- **cli** - Can return trace if agent outputs trace format
-
-### Trace Event Structure
-
-```json
-{
-  "type": "tool_call",
-  "name": "knowledgeSearch",
-  "input": { "query": "REST vs GraphQL" },
-  "timestamp": "2024-01-15T10:30:00Z"
-}
-```
-
-## Complete Examples
-
-### Research Agent Validation
-
-```yaml
-$schema: agentv-eval-v2
-description: Validate research agent tool usage
-
-target: codex_agent  # Provider that returns traces
-
-evalcases:
-  - id: comprehensive-research
-    expected_outcome: Agent thoroughly researches the topic
-    
-    input_messages:
-      - role: user
-        content: Research machine learning frameworks
-    
-    execution:
-      evaluators:
-        # Check minimum tool usage
-        - name: coverage
-          type: tool_trajectory
-          mode: any_order
-          minimums:
-            webSearch: 1
-            documentRead: 2
-            noteTaking: 1
-        
-        # Check workflow order
-        - name: workflow
-          type: tool_trajectory
-          mode: in_order
-          expected:
-            - tool: webSearch
-            - tool: documentRead
-            - tool: summarize
-```
-
-### Multi-Step Pipeline
-
-```yaml
-evalcases:
-  - id: data-pipeline
-    expected_outcome: Process data through complete pipeline
-
-    input_messages:
-      - role: user
-        content: Process the customer dataset
-
-    execution:
-      evaluators:
-        - name: pipeline-check
-          type: tool_trajectory
-          mode: exact
-          expected:
-            - tool: loadData
-            - tool: validate
-            - tool: transform
-            - tool: export
-```
-
-## CLI Options for Traces
-
-```bash
-# Write trace files to disk
-agentv eval evals/test.yaml --dump-traces
-
-# Include full trace in result output
-agentv eval evals/test.yaml --include-trace
-```
-
-## Best Practices
-
-1. **Choose the right mode** - Use `any_order` for flexibility, `exact` for strict validation
-2. **Start with any_order** - Then tighten to `in_order` or `exact` as needed
-3. **Combine with other evaluators** - Use tool trajectory for execution, LLM judge for output quality
-4. **Test with --dump-traces** - Inspect actual traces to understand agent behavior
-5. **Use code evaluators for custom validation** - Write custom tool validation scripts with access to trace data
+# Tool Trajectory Evaluator Guide
+
+Tool trajectory evaluators validate that an agent used the expected tools during execution. They work with trace data returned by agent providers (codex, vscode, cli with trace support).
+
+## Tool Trajectory Evaluator
+
+### Modes
+
+#### 1. `any_order` - Minimum Tool Counts
+
+Validates that each tool was called at least N times, regardless of order:
+
+```yaml
+execution:
+  evaluators:
+    - name: tool-usage
+      type: tool_trajectory
+      mode: any_order
+      minimums:
+        knowledgeSearch: 2    # Must be called at least twice
+        documentRetrieve: 1   # Must be called at least once
+```
+
+**Use cases:**
+- Ensure required tools are used
+- Don't care about execution order
+- Allow flexibility in agent implementation
+
+#### 2. `in_order` - Sequential Matching
+
+Validates tools appear in the expected sequence, but allows gaps (other tools can appear between):
+
+```yaml
+execution:
+  evaluators:
+    - name: workflow-sequence
+      type: tool_trajectory
+      mode: in_order
+      expected:
+        - tool: fetchData
+        - tool: validateSchema
+        - tool: transformData
+        - tool: saveResults
+```
+
+**Use cases:**
+- Validate logical workflow order
+- Allow agent to use additional helper tools
+- Check that key steps happen in sequence
+
+### Argument Matching
+
+For `in_order` and `exact` modes, you can optionally validate tool arguments:
+
+```yaml
+execution:
+  evaluators:
+    - name: search-validation
+      type: tool_trajectory
+      mode: in_order
+      expected:
+        # Partial match - only specified keys are checked
+        - tool: search
+          args: { query: "machine learning" }
+
+        # Skip argument validation for this tool
+        - tool: process
+          args: any
+
+        # No args field = no argument validation (same as args: any)
+        - tool: saveResults
+```
+
+**Argument matching modes:**
+- `args: { key: value }` - Partial deep equality (only specified keys are checked)
+- `args: any` - Skip argument validation
+- No `args` field - Same as `args: any`
+
+#### 3. `exact` - Strict Sequence Match
+
+Validates the exact tool sequence with no gaps or extra tools:
+
+```yaml
+execution:
+  evaluators:
+    - name: auth-sequence
+      type: tool_trajectory
+      mode: exact
+      expected:
+        - tool: checkCredentials
+        - tool: generateToken
+        - tool: auditLog
+```
+
+**Use cases:**
+- Security-critical workflows
+- Strict protocol validation
+- Regression testing specific behavior
+
+## Scoring
+
+### tool_trajectory Scoring
+
+| Mode | Score Calculation |
+|------|------------------|
+| `any_order` | (tools meeting minimum) / (total tools with minimums) |
+| `in_order` | (matched tools in sequence) / (expected tools count) |
+| `exact` | (correctly positioned tools) / (expected tools count) |
+
+## Trace Data Requirements
+
+Tool trajectory evaluators require trace data from the agent provider. Providers return `output_messages` containing `tool_calls` that capture agent tool usage.
+
+### Output Messages Format
+
+Providers return `output_messages` with `tool_calls` in the JSONL output:
+
+```json
+{
+  "id": "eval-001",
+  "output_messages": [
+    {
+      "role": "assistant",
+      "content": "I'll search for information about this topic.",
+      "tool_calls": [
+        {
+          "tool": "knowledgeSearch",
+          "input": { "query": "REST vs GraphQL" },
+          "output": { "results": [...] },
+          "id": "call_123",
+          "timestamp": "2024-01-15T10:30:00Z"
+        }
+      ]
+    }
+  ]
+}
+```
+
+The evaluator extracts tool calls from `output_messages[].tool_calls[]`. Optional fields `id` and `timestamp` can be included for debugging.
+
+### Supported Providers
+
+- **codex** - Returns output_messages via JSONL log events
+- **vscode / vscode-insiders** - Returns output_messages from Copilot execution
+- **cli** - Returns `output_messages` with `tool_calls`
+
+## Complete Examples
+
+### Research Agent Validation
+
+```yaml
+description: Validate research agent tool usage
+execution:
+  target: codex_agent  # Provider that returns traces
+
+evalcases:
+  - id: comprehensive-research
+    expected_outcome: Agent thoroughly researches the topic
+    
+    input_messages:
+      - role: user
+        content: Research machine learning frameworks
+    
+    execution:
+      evaluators:
+        # Check minimum tool usage
+        - name: coverage
+          type: tool_trajectory
+          mode: any_order
+          minimums:
+            webSearch: 1
+            documentRead: 2
+            noteTaking: 1
+        
+        # Check workflow order
+        - name: workflow
+          type: tool_trajectory
+          mode: in_order
+          expected:
+            - tool: webSearch
+            - tool: documentRead
+            - tool: summarize
+```
+
+### Multi-Step Pipeline
+
+```yaml
+evalcases:
+  - id: data-pipeline
+    expected_outcome: Process data through complete pipeline
+
+    input_messages:
+      - role: user
+        content: Process the customer dataset
+
+    execution:
+      evaluators:
+        - name: pipeline-check
+          type: tool_trajectory
+          mode: exact
+          expected:
+            - tool: loadData
+            - tool: validate
+            - tool: transform
+            - tool: export
+```
+
+## CLI Options for Traces
+
+```bash
+# Write trace files to disk
+agentv eval evals/test.yaml --dump-traces
+
+# Include full trace in result output
+agentv eval evals/test.yaml --include-trace
+```
+
+## Best Practices
+
+1. **Choose the right mode** - Use `any_order` for flexibility, `exact` for strict validation
+2. **Start with any_order** - Then tighten to `in_order` or `exact` as needed
+3. **Combine with other evaluators** - Use tool trajectory for execution, LLM judge for output quality
+4. **Test with --dump-traces** - Inspect actual traces to understand agent behavior
+5. **Use code evaluators for custom validation** - Write custom tool validation scripts with access to trace data

package/dist/templates/.claude/skills/agentv-prompt-optimizer/SKILL.md

@@ -1,77 +1,77 @@
----
-description: Iteratively optimize prompt files against AgentV evaluation datasets by analyzing failures and refining instructions.
----
-
-# AgentV Prompt Optimizer
-
-## Input Variables
-- `eval-path`: Path or glob pattern to the AgentV evaluation file(s) to optimize against
-- `optimization-log-path` (optional): Path where optimization progress should be logged
-
-## Workflow
-
-1.  **Initialize**
-    - Verify `<eval-path>` (file or glob) targets the correct system.
-    - **Identify Prompt Files**:
-        - Infer prompt files from the eval file content (look for `file:` references in `input_messages` that match these patterns).
-        - Recursively check referenced prompt files for *other* prompt references (dependencies).
-        - If multiple prompts are found, consider ALL of them as candidates for optimization.
-    - **Identify Optimization Log**:
-        - If `<optimization-log-path>` is provided, use it.
-        - If not, create a new one in the parent directory of the eval files: `optimization-[timestamp].md`.
-    - Read content of the identified prompt file.
-
-2.  **Optimization Loop** (Max 10 iterations)
-    - **Execute (The Generator)**: Run `agentv eval <eval-path>`.
-        - *Targeted Run*: If iterating on specific stubborn failures, use `--eval-id <case_id>` to run only the relevant eval cases.
-    - **Analyze (The Reflector)**:
-        - Locate the results file path from the console output (e.g., `.agentv/results/eval_...jsonl`).
-        - **Orchestrate Subagent**: Use `runSubagent` to analyze the results.
-            - **Task**: Read the results file, calculate pass rate, and perform root cause analysis.
-            - **Output**: Return a structured analysis including:
-                - **Score**: Current pass rate.
-                - **Root Cause**: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
-                - **Insight**: Key learning or pattern identified from the failures.
-                - **Strategy**: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
-    - **Decide**:
-        - If **100% pass**: STOP and report success.
-        - If **Score decreased**: Revert last change, try different approach.
-        - If **No improvement** (2x): STOP and report stagnation.
-    - **Refine (The Curator)**:
-        - **Orchestrate Subagent**: Use `runSubagent` to apply the fix.
-            - **Task**: Read the relevant prompt file(s), apply the **Strategy** from the Reflector, and generate the log entry.
-            - **Output**: The **Log Entry** describing the specific operation performed.
-                  ```markdown
-                  ### Iteration [N]
-                  - **Operation**: [ADD / UPDATE / DELETE]
-                  - **Target**: [Section Name]
-                  - **Change**: [Specific text added/modified]
-                  - **Trigger**: [Specific failing test case or error pattern]
-                  - **Rationale**: [From Reflector: Root Cause]
-                  - **Score**: [From Reflector: Current Pass Rate]
-                  - **Insight**: [From Reflector: Key Learning]
-                  ```
-        - **Strategy**: Treat the prompt as a structured set of rules. Execute atomic operations:
-            - **ADD**: Insert a new rule if a constraint was missed.
-            - **UPDATE**: Refine an existing rule to be clearer or more general.
-                - *Clarify*: Make ambiguous instructions specific.
-                - *Generalize*: Refactor specific fixes into high-level principles (First Principles).
-            - **DELETE**: Remove obsolete, redundant, or harmful rules.
-                - *Prune*: If a general rule covers specific cases, delete the specific ones.
-            - **Negative Constraint**: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
-            - **Safety Check**: Ensure new rules don't contradict existing ones (unless intended).
-        - **Constraint**: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
-    - **Log Result**:
-        - Append the **Log Entry** returned by the Curator to the optimization log file.
-
-3.  **Completion**
-    - Report final score.
-    - Summarize key changes made to the prompt.
-    - **Finalize Optimization Log**: Add a summary header to the optimization log file indicating the session completion and final score.
-
-## Guidelines
-- **Generalization First**: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
-- **Simplicity ("Less is More")**: Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
-- **Structure**: Maintain existing Markdown headers/sections.
-- **Progressive Disclosure**: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
-- **Quality Criteria**: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.
+---
+description: Iteratively optimize prompt files against AgentV evaluation datasets by analyzing failures and refining instructions.
+---
+
+# AgentV Prompt Optimizer
+
+## Input Variables
+- `eval-path`: Path or glob pattern to the AgentV evaluation file(s) to optimize against
+- `optimization-log-path` (optional): Path where optimization progress should be logged
+
+## Workflow
+
+1.  **Initialize**
+    - Verify `<eval-path>` (file or glob) targets the correct system.
+    - **Identify Prompt Files**:
+        - Infer prompt files from the eval file content (look for `file:` references in `input_messages` that match these patterns).
+        - Recursively check referenced prompt files for *other* prompt references (dependencies).
+        - If multiple prompts are found, consider ALL of them as candidates for optimization.
+    - **Identify Optimization Log**:
+        - If `<optimization-log-path>` is provided, use it.
+        - If not, create a new one in the parent directory of the eval files: `optimization-[timestamp].md`.
+    - Read content of the identified prompt file.
+
+2.  **Optimization Loop** (Max 10 iterations)
+    - **Execute (The Generator)**: Run `agentv eval <eval-path>`.
+        - *Targeted Run*: If iterating on specific stubborn failures, use `--eval-id <case_id>` to run only the relevant eval cases.
+    - **Analyze (The Reflector)**:
+        - Locate the results file path from the console output (e.g., `.agentv/results/eval_...jsonl`).
+        - **Orchestrate Subagent**: Use `runSubagent` to analyze the results.
+            - **Task**: Read the results file, calculate pass rate, and perform root cause analysis.
+            - **Output**: Return a structured analysis including:
+                - **Score**: Current pass rate.
+                - **Root Cause**: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
+                - **Insight**: Key learning or pattern identified from the failures.
+                - **Strategy**: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
+    - **Decide**:
+        - If **100% pass**: STOP and report success.
+        - If **Score decreased**: Revert last change, try different approach.
+        - If **No improvement** (2x): STOP and report stagnation.
+    - **Refine (The Curator)**:
+        - **Orchestrate Subagent**: Use `runSubagent` to apply the fix.
+            - **Task**: Read the relevant prompt file(s), apply the **Strategy** from the Reflector, and generate the log entry.
+            - **Output**: The **Log Entry** describing the specific operation performed.
+                  ```markdown
+                  ### Iteration [N]
+                  - **Operation**: [ADD / UPDATE / DELETE]
+                  - **Target**: [Section Name]
+                  - **Change**: [Specific text added/modified]
+                  - **Trigger**: [Specific failing test case or error pattern]
+                  - **Rationale**: [From Reflector: Root Cause]
+                  - **Score**: [From Reflector: Current Pass Rate]
+                  - **Insight**: [From Reflector: Key Learning]
+                  ```
+        - **Strategy**: Treat the prompt as a structured set of rules. Execute atomic operations:
+            - **ADD**: Insert a new rule if a constraint was missed.
+            - **UPDATE**: Refine an existing rule to be clearer or more general.
+                - *Clarify*: Make ambiguous instructions specific.
+                - *Generalize*: Refactor specific fixes into high-level principles (First Principles).
+            - **DELETE**: Remove obsolete, redundant, or harmful rules.
+                - *Prune*: If a general rule covers specific cases, delete the specific ones.
+            - **Negative Constraint**: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
+            - **Safety Check**: Ensure new rules don't contradict existing ones (unless intended).
+        - **Constraint**: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
+    - **Log Result**:
+        - Append the **Log Entry** returned by the Curator to the optimization log file.
+
+3.  **Completion**
+    - Report final score.
+    - Summarize key changes made to the prompt.
+    - **Finalize Optimization Log**: Add a summary header to the optimization log file indicating the session completion and final score.
+
+## Guidelines
+- **Generalization First**: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
+- **Simplicity ("Less is More")**: Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
+- **Structure**: Maintain existing Markdown headers/sections.
+- **Progressive Disclosure**: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
+- **Quality Criteria**: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.

package/dist/templates/.github/prompts/agentv-eval-build.prompt.md

@@ -1,5 +1,5 @@
----
-description: 'Create and maintain AgentV YAML evaluation files'
----
-
+---
+description: 'Create and maintain AgentV YAML evaluation files'
+---
+
 #file:../../.claude/skills/agentv-eval-builder/SKILL.md

package/dist/templates/.github/prompts/agentv-optimize.prompt.md

@@ -1,4 +1,4 @@
----
-description: Iteratively optimize prompt files against an AgentV evaluation suite
----
+---
+description: Iteratively optimize prompt files against an AgentV evaluation suite
+---
 #file:../../.claude/skills/agentv-prompt-optimizer/SKILL.md

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentv",
-  "version": "1.3.1",
+  "version": "1.6.1",
   "description": "CLI entry point for AgentV",
   "type": "module",
   "repository": {