agentv 3.10.2 → 3.10.3

package/dist/templates/.claude/skills/agentv-eval-builder/references/structured-data-evaluators.md

@@ -1,121 +0,0 @@
-# Structured Data + Metrics Evaluators
-
-This reference covers the built-in evaluators used for grading structured outputs and gating on execution metrics:
-
-- `field_accuracy`
-- `latency`
-- `cost`
-- `token_usage`
-
-## Ground Truth (`expected_messages`)
-
-Put the expected structured output in the evalcase `expected_messages` (typically as the last `assistant` message with `content` as an object). Evaluators read expected values from there.
-
-```yaml
-evalcases:
-  - id: invoice-001
-    expected_messages:
-      - role: assistant
-        content:
-          invoice_number: "INV-2025-001234"
-          net_total: 1889
-```
-
-## `field_accuracy`
-
-Use `field_accuracy` to compare fields in the candidate JSON against the ground-truth object in `expected_messages`.
-
-```yaml
-execution:
-  evaluators:
-    - name: invoice_fields
-      type: field_accuracy
-      aggregation: weighted_average
-      fields:
-        - path: invoice_number
-          match: exact
-          required: true
-          weight: 2.0
-        - path: invoice_date
-          match: date
-          formats: ["DD-MMM-YYYY", "YYYY-MM-DD"]
-        - path: net_total
-          match: numeric_tolerance
-          tolerance: 1.0
-```
-
-### Match types
-
-- `exact`: strict equality
-- `date`: compares dates after parsing; optionally provide `formats`
-- `numeric_tolerance`: numeric compare within `tolerance` (set `relative: true` for relative tolerance)
-
-For fuzzy string matching, use a `code_judge` evaluator (e.g. Levenshtein) instead of adding a fuzzy mode to `field_accuracy`.
-
-### Aggregation
-
-- `weighted_average` (default): weighted mean of field scores
-- `all_or_nothing`: score 1.0 only if all graded fields pass
-
-## `latency` and `cost`
-
-These evaluators gate on execution metrics reported by the provider (via `traceSummary`).
-
-```yaml
-execution:
-  evaluators:
-    - name: performance
-      type: latency
-      threshold: 2000
-    - name: budget
-      type: cost
-      budget: 0.10
-```
-
-## `token_usage`
-
-Gate on provider-reported token usage (useful when cost is unavailable or model pricing differs).
-
-```yaml
-execution:
-  evaluators:
-    - name: token-budget
-      type: token_usage
-      max_total: 10000
-      # or:
-      # max_input: 8000
-      # max_output: 2000
-```
-
-## Common pattern: combine correctness + gates
-
-Use a `composite` evaluator if you want a single “release gate” score/verdict from multiple checks:
-
-```yaml
-execution:
-  evaluators:
-    - name: release_gate
-      type: composite
-      evaluators:
-        - name: correctness
-          type: field_accuracy
-          fields:
-            - path: invoice_number
-              match: exact
-        - name: latency
-          type: latency
-          threshold: 2000
-        - name: cost
-          type: cost
-          budget: 0.10
-        - name: tokens
-          type: token_usage
-          max_total: 10000
-      aggregator:
-        type: weighted_average
-        weights:
-          correctness: 0.8
-          latency: 0.1
-          cost: 0.05
-          tokens: 0.05
-```

package/dist/templates/.claude/skills/agentv-eval-builder/references/tool-trajectory-evaluator.md

@@ -1,298 +0,0 @@
-# Tool Trajectory Evaluator Guide
-
-Tool trajectory evaluators validate that an agent used the expected tools during execution. They work with trace data returned by agent providers (codex, vscode, cli with trace support).
-
-## Tool Trajectory Evaluator
-
-### Modes
-
-#### 1. `any_order` - Minimum Tool Counts
-
-Validates that each tool was called at least N times, regardless of order:
-
-```yaml
-execution:
-  evaluators:
-    - name: tool-usage
-      type: tool_trajectory
-      mode: any_order
-      minimums:
-        knowledgeSearch: 2    # Must be called at least twice
-        documentRetrieve: 1   # Must be called at least once
-```
-
-**Use cases:**
-- Ensure required tools are used
-- Don't care about execution order
-- Allow flexibility in agent implementation
-
-#### 2. `in_order` - Sequential Matching
-
-Validates tools appear in the expected sequence, but allows gaps (other tools can appear between):
-
-```yaml
-execution:
-  evaluators:
-    - name: workflow-sequence
-      type: tool_trajectory
-      mode: in_order
-      expected:
-        - tool: fetchData
-        - tool: validateSchema
-        - tool: transformData
-        - tool: saveResults
-```
-
-**Use cases:**
-- Validate logical workflow order
-- Allow agent to use additional helper tools
-- Check that key steps happen in sequence
-
-### Argument Matching
-
-For `in_order` and `exact` modes, you can optionally validate tool arguments:
-
-```yaml
-execution:
-  evaluators:
-    - name: search-validation
-      type: tool_trajectory
-      mode: in_order
-      expected:
-        # Partial match - only specified keys are checked
-        - tool: search
-          args: { query: "machine learning" }
-
-        # Skip argument validation for this tool
-        - tool: process
-          args: any
-
-        # No args field = no argument validation (same as args: any)
-        - tool: saveResults
-```
-
-**Argument matching modes:**
-- `args: { key: value }` - Partial deep equality (only specified keys are checked)
-- `args: any` - Skip argument validation
-- No `args` field - Same as `args: any`
-
-### Latency Assertions
-
-For `in_order` and `exact` modes, you can optionally validate per-tool timing with `max_duration_ms`:
-
-```yaml
-execution:
-  evaluators:
-    - name: perf-check
-      type: tool_trajectory
-      mode: in_order
-      expected:
-        - tool: Read
-          max_duration_ms: 100    # Must complete within 100ms
-        - tool: Edit
-          max_duration_ms: 500    # Allow 500ms for edits
-        - tool: Write             # No timing requirement
-```
-
-**Latency assertion behavior:**
-- **Pass**: `actual_duration <= max_duration_ms` → counts as hit
-- **Fail**: `actual_duration > max_duration_ms` → counts as miss
-- **Skip**: No `duration_ms` in output → warning logged, neutral (neither hit nor miss)
-
-**Provider requirements:**
-Providers must include `duration_ms` in tool calls for latency checks:
-
-```json
-{
-  "tool_calls": [{
-    "tool": "Read",
-    "duration_ms": 45
-  }]
-}
-```
-
-**Best practices for latency assertions:**
-- Set generous thresholds to avoid flaky tests from timing variance
-- Only add latency assertions where timing matters (critical paths)
-- Combine with sequence checks for comprehensive validation
-
-#### 3. `exact` - Strict Sequence Match
-
-Validates the exact tool sequence with no gaps or extra tools:
-
-```yaml
-execution:
-  evaluators:
-    - name: auth-sequence
-      type: tool_trajectory
-      mode: exact
-      expected:
-        - tool: checkCredentials
-        - tool: generateToken
-        - tool: auditLog
-```
-
-**Use cases:**
-- Security-critical workflows
-- Strict protocol validation
-- Regression testing specific behavior
-
-## Scoring
-
-### tool_trajectory Scoring
-
-| Mode | Score Calculation |
-|------|------------------|
-| `any_order` | (tools meeting minimum) / (total tools with minimums) |
-| `in_order` | (sequence hits + latency hits) / (expected tools + latency assertions) |
-| `exact` | (sequence hits + latency hits) / (expected tools + latency assertions) |
-
-**With latency assertions:**
-- Each `max_duration_ms` assertion counts as a separate aspect
-- Latency checks only run when the sequence/position matches
-- Example: 3 expected tools + 2 latency assertions = 5 total aspects
-
-## Trace Data Requirements
-
-Tool trajectory evaluators require trace data from the agent provider. Providers return `output_messages` containing `tool_calls` that capture agent tool usage.
-
-### Output Messages Format
-
-Providers return `output_messages` with `tool_calls` in the JSONL output:
-
-```json
-{
-  "id": "eval-001",
-  "output_messages": [
-    {
-      "role": "assistant",
-      "content": "I'll search for information about this topic.",
-      "tool_calls": [
-        {
-          "tool": "knowledgeSearch",
-          "input": { "query": "REST vs GraphQL" },
-          "output": { "results": [...] },
-          "id": "call_123",
-          "timestamp": "2024-01-15T10:30:00Z",
-          "duration_ms": 45
-        }
-      ]
-    }
-  ]
-}
-```
-
-The evaluator extracts tool calls from `output_messages[].tool_calls[]`. Optional fields:
-- `id` and `timestamp` - for debugging
-- `duration_ms` - for latency assertions (required if using `max_duration_ms`)
-
-### Supported Providers
-
-- **codex** - Returns output_messages via JSONL log events
-- **vscode / vscode-insiders** - Returns output_messages from Copilot execution
-- **cli** - Returns `output_messages` with `tool_calls`
-
-## Complete Examples
-
-### Research Agent Validation
-
-```yaml
-description: Validate research agent tool usage
-execution:
-  target: codex_agent  # Provider that returns traces
-
-evalcases:
-  - id: comprehensive-research
-    expected_outcome: Agent thoroughly researches the topic
-    
-    input_messages:
-      - role: user
-        content: Research machine learning frameworks
-    
-    execution:
-      evaluators:
-        # Check minimum tool usage
-        - name: coverage
-          type: tool_trajectory
-          mode: any_order
-          minimums:
-            webSearch: 1
-            documentRead: 2
-            noteTaking: 1
-        
-        # Check workflow order
-        - name: workflow
-          type: tool_trajectory
-          mode: in_order
-          expected:
-            - tool: webSearch
-            - tool: documentRead
-            - tool: summarize
-```
-
-### Multi-Step Pipeline
-
-```yaml
-evalcases:
-  - id: data-pipeline
-    expected_outcome: Process data through complete pipeline
-
-    input_messages:
-      - role: user
-        content: Process the customer dataset
-
-    execution:
-      evaluators:
-        - name: pipeline-check
-          type: tool_trajectory
-          mode: exact
-          expected:
-            - tool: loadData
-            - tool: validate
-            - tool: transform
-            - tool: export
-```
-
-### Pipeline with Latency Assertions
-
-```yaml
-evalcases:
-  - id: data-pipeline-perf
-    expected_outcome: Process data within timing budgets
-
-    input_messages:
-      - role: user
-        content: Process the customer dataset quickly
-
-    execution:
-      evaluators:
-        - name: pipeline-perf
-          type: tool_trajectory
-          mode: in_order
-          expected:
-            - tool: loadData
-              max_duration_ms: 1000   # Network fetch within 1s
-            - tool: validate           # No timing requirement
-            - tool: transform
-              max_duration_ms: 500    # Transform must be fast
-            - tool: export
-              max_duration_ms: 200    # Export should be quick
-```
-
-## CLI Options for Traces
-
-```bash
-# Write trace files to disk
-agentv eval evals/test.yaml --dump-traces
-
-# Include full trace in result output
-agentv eval evals/test.yaml --include-trace
-```
-
-## Best Practices
-
-1. **Choose the right mode** - Use `any_order` for flexibility, `exact` for strict validation
-2. **Start with any_order** - Then tighten to `in_order` or `exact` as needed
-3. **Combine with other evaluators** - Use tool trajectory for execution, LLM judge for output quality
-4. **Test with --dump-traces** - Inspect actual traces to understand agent behavior
-5. **Use code evaluators for custom validation** - Write custom tool validation scripts with access to trace data

package/dist/templates/.claude/skills/agentv-prompt-optimizer/SKILL.md

@@ -1,78 +0,0 @@
----
-name: agentv-prompt-optimizer
-description: Iteratively optimize prompt files against AgentV evaluation datasets by analyzing failures and refining instructions.
----
-
-# AgentV Prompt Optimizer
-
-## Input Variables
-- `eval-path`: Path or glob pattern to the AgentV evaluation file(s) to optimize against
-- `optimization-log-path` (optional): Path where optimization progress should be logged
-
-## Workflow
-
-1.  **Initialize**
-    - Verify `<eval-path>` (file or glob) targets the correct system.
-    - **Identify Prompt Files**:
-        - Infer prompt files from the eval file content (look for `file:` references in `input_messages` that match these patterns).
-        - Recursively check referenced prompt files for *other* prompt references (dependencies).
-        - If multiple prompts are found, consider ALL of them as candidates for optimization.
-    - **Identify Optimization Log**:
-        - If `<optimization-log-path>` is provided, use it.
-        - If not, create a new one in the parent directory of the eval files: `optimization-[timestamp].md`.
-    - Read content of the identified prompt file.
-
-2.  **Optimization Loop** (Max 10 iterations)
-    - **Execute (The Generator)**: Run `agentv eval <eval-path>`.
-        - *Targeted Run*: If iterating on specific stubborn failures, use `--eval-id <case_id>` to run only the relevant eval cases.
-    - **Analyze (The Reflector)**:
-        - Locate the results file path from the console output (e.g., `.agentv/results/eval_...jsonl`).
-        - **Orchestrate Subagent**: Use `runSubagent` to analyze the results.
-            - **Task**: Read the results file, calculate pass rate, and perform root cause analysis.
-            - **Output**: Return a structured analysis including:
-                - **Score**: Current pass rate.
-                - **Root Cause**: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
-                - **Insight**: Key learning or pattern identified from the failures.
-                - **Strategy**: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
-    - **Decide**:
-        - If **100% pass**: STOP and report success.
-        - If **Score decreased**: Revert last change, try different approach.
-        - If **No improvement** (2x): STOP and report stagnation.
-    - **Refine (The Curator)**:
-        - **Orchestrate Subagent**: Use `runSubagent` to apply the fix.
-            - **Task**: Read the relevant prompt file(s), apply the **Strategy** from the Reflector, and generate the log entry.
-            - **Output**: The **Log Entry** describing the specific operation performed.
-                  ```markdown
-                  ### Iteration [N]
-                  - **Operation**: [ADD / UPDATE / DELETE]
-                  - **Target**: [Section Name]
-                  - **Change**: [Specific text added/modified]
-                  - **Trigger**: [Specific failing test case or error pattern]
-                  - **Rationale**: [From Reflector: Root Cause]
-                  - **Score**: [From Reflector: Current Pass Rate]
-                  - **Insight**: [From Reflector: Key Learning]
-                  ```
-        - **Strategy**: Treat the prompt as a structured set of rules. Execute atomic operations:
-            - **ADD**: Insert a new rule if a constraint was missed.
-            - **UPDATE**: Refine an existing rule to be clearer or more general.
-                - *Clarify*: Make ambiguous instructions specific.
-                - *Generalize*: Refactor specific fixes into high-level principles (First Principles).
-            - **DELETE**: Remove obsolete, redundant, or harmful rules.
-                - *Prune*: If a general rule covers specific cases, delete the specific ones.
-            - **Negative Constraint**: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
-            - **Safety Check**: Ensure new rules don't contradict existing ones (unless intended).
-        - **Constraint**: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
-    - **Log Result**:
-        - Append the **Log Entry** returned by the Curator to the optimization log file.
-
-3.  **Completion**
-    - Report final score.
-    - Summarize key changes made to the prompt.
-    - **Finalize Optimization Log**: Add a summary header to the optimization log file indicating the session completion and final score.
-
-## Guidelines
-- **Generalization First**: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
-- **Simplicity ("Less is More")**: Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
-- **Structure**: Maintain existing Markdown headers/sections.
-- **Progressive Disclosure**: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
-- **Quality Criteria**: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.

package/dist/templates/.github/prompts/agentv-eval-build.prompt.md

@@ -1,5 +0,0 @@
----
-description: 'Create and maintain AgentV YAML evaluation files'
----
-
-#file:../../.claude/skills/agentv-eval-builder/SKILL.md

package/dist/templates/.github/prompts/agentv-optimize.prompt.md

@@ -1,4 +0,0 @@
----
-description: Iteratively optimize prompt files against an AgentV evaluation suite
----
-#file:../../.claude/skills/agentv-prompt-optimizer/SKILL.md