agentv 2.9.0-next.1 → 2.10.0

package/dist/templates/.claude/skills/agentv-eval-builder/SKILL.md

@@ -1,202 +0,0 @@
----
-name: agentv-eval-builder
-description: Create and maintain AgentV YAML evaluation files for testing AI agent performance. Use this skill when creating new eval files, adding eval cases, or configuring evaluators.
----
-
-# AgentV Eval Builder
-
-Comprehensive docs: https://agentv.dev
-
-## Quick Start
-
-```yaml
-description: Example eval
-execution:
-  target: default
-
-cases:
-  - id: greeting
-    criteria: Friendly greeting
-    input: "Say hello"
-    expected_output: "Hello! How can I help you?"
-    rubrics:
-      - Greeting is friendly and warm
-      - Offers to help
-```
-
-## Eval File Structure
-
-**Required:** `cases` (array)
-**Optional:** `description`, `execution`, `dataset`
-
-**Eval case fields:**
-
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | yes | Unique identifier |
-| `criteria` | yes | What the response should accomplish |
-| `input` / `input_messages` | yes | Input to the agent |
-| `expected_output` / `expected_messages` | no | Gold-standard reference answer |
-| `rubrics` | no | Inline evaluation criteria |
-| `execution` | no | Per-case execution overrides |
-| `conversation_id` | no | Thread grouping |
-
-**Shorthand aliases:**
-- `input` (string) expands to `[{role: "user", content: "..."}]`
-- `expected_output` (string/object) expands to `[{role: "assistant", content: ...}]`
-- Canonical `input_messages` / `expected_messages` take precedence when both present
-
-**Message format:** `{role, content}` where role is `system`, `user`, `assistant`, or `tool`
-**Content types:** inline text, `{type: "file", value: "./path.md"}`
-**File paths:** relative from eval file dir, or absolute with `/` prefix from repo root
-
-**JSONL format:** One eval case per line as JSON. Optional `.yaml` sidecar for shared defaults. See `examples/features/basic-jsonl/`.
-
-## Evaluator Types
-
-Configure via `execution.evaluators` array. Multiple evaluators produce a weighted average score.
-
-### code_judge
-```yaml
-- name: format_check
-  type: code_judge
-  script: uv run validate.py
-  cwd: ./scripts          # optional working directory
-  target: {}              # optional: enable LLM target proxy (max_calls: 50)
-```
-Contract: stdin JSON -> stdout JSON `{score, hits, misses, reasoning}`
-Input includes: `question`, `criteria`, `candidate_answer`, `reference_answer`, `output_messages`, `trace_summary`, `file_changes`, `workspace_path`, `config`
-When `workspace_template` is configured, `workspace_path` is the absolute path to the workspace dir (also available as `AGENTV_WORKSPACE_PATH` env var). Use this for functional grading (e.g., running `npm test` in the workspace).
-See docs at https://agentv.dev/evaluators/code-judges/
-
-### llm_judge
-```yaml
-- name: quality
-  type: llm_judge
-  prompt: ./prompts/eval.md     # markdown template or script config
-  model: gpt-5-chat            # optional model override
-  config:                       # passed to script templates as context.config
-    strictness: high
-```
-Variables: `{{question}}`, `{{criteria}}`, `{{candidate_answer}}`, `{{reference_answer}}`, `{{input_messages}}`, `{{expected_messages}}`, `{{output_messages}}`, `{{file_changes}}`
-- Markdown templates: use `{{variable}}` syntax
-- TypeScript templates: use `definePromptTemplate(fn)` from `@agentv/eval`, receives context object with all variables + `config`
-
-### composite
-```yaml
-- name: gate
-  type: composite
-  evaluators:
-    - name: safety
-      type: llm_judge
-      prompt: ./safety.md
-    - name: quality
-      type: llm_judge
-  aggregator:
-    type: weighted_average
-    weights: { safety: 0.3, quality: 0.7 }
-```
-Aggregator types: `weighted_average`, `all_or_nothing`, `minimum`, `maximum`, `safety_gate`
-- `safety_gate`: fails immediately if the named gate evaluator scores below threshold (default 1.0)
-
-### tool_trajectory
-```yaml
-- name: tool_check
-  type: tool_trajectory
-  mode: any_order            # any_order | in_order | exact
-  minimums:                  # for any_order
-    knowledgeSearch: 2
-  expected:                  # for in_order/exact
-    - tool: knowledgeSearch
-      args: { query: "search term" }   # partial deep equality match
-    - tool: documentRetrieve
-      args: any                        # any arguments accepted
-      max_duration_ms: 5000            # per-tool latency assertion
-    - tool: summarize                  # omit args to skip argument checking
-```
-
-### field_accuracy
-```yaml
-- name: fields
-  type: field_accuracy
-  match_type: exact          # exact | date | numeric_tolerance
-  numeric_tolerance: 0.01    # for numeric_tolerance match_type
-  aggregation: weighted_average  # weighted_average | all_or_nothing
-```
-Compares `output_messages` fields against `expected_messages` fields.
-
-### latency
-```yaml
-- name: speed
-  type: latency
-  max_ms: 5000
-```
-
-### cost
-```yaml
-- name: budget
-  type: cost
-  max_usd: 0.10
-```
-
-### token_usage
-```yaml
-- name: tokens
-  type: token_usage
-  max_total_tokens: 4000
-```
-
-### execution_metrics
-```yaml
-- name: efficiency
-  type: execution_metrics
-  max_tool_calls: 10        # Maximum tool invocations
-  max_llm_calls: 5          # Maximum LLM calls (assistant messages)
-  max_tokens: 5000          # Maximum total tokens (input + output)
-  max_cost_usd: 0.05        # Maximum cost in USD
-  max_duration_ms: 30000    # Maximum execution duration
-  target_exploration_ratio: 0.6   # Target ratio of read-only tool calls
-  exploration_tolerance: 0.2      # Tolerance for ratio check (default: 0.2)
-```
-Declarative threshold-based checks on execution metrics. Only specified thresholds are checked.
-Score is proportional: `hits / (hits + misses)`. Missing data counts as a miss.
-
-### rubric (inline)
-```yaml
-rubrics:
-  - Simple string criterion
-  - id: weighted
-    criteria: Detailed criterion
-    weight: 2.0
-    required: true
-```
-See `references/rubric-evaluator.md` for score-range mode and scoring formula.
-
-## CLI Commands
-
-```bash
-# Run evaluation (requires API keys)
-agentv eval <file.yaml> [--eval-id <id>] [--target <name>] [--dry-run]
-
-# Run with trace persistence (writes to .agentv/traces/)
-agentv eval <file.yaml> --trace
-
-# Agent-orchestrated evals (no API keys needed)
-agentv eval prompt <file.yaml>                                      # orchestration overview
-agentv eval prompt input <file.yaml> --eval-id <id>                 # task input JSON (file paths, not embedded content)
-agentv eval prompt judge <file.yaml> --eval-id <id> --answer-file f # judge prompts / code judge results
-
-# Validate eval file
-agentv validate <file.yaml>
-
-# Compare results between runs
-agentv compare <results1.jsonl> <results2.jsonl>
-
-# Generate rubrics from criteria
-agentv generate rubrics <file.yaml> [--target <name>]
-```
-
-## Schemas
-
-- Eval file: `references/eval-schema.json`
-- Config: `references/config-schema.json`

package/dist/templates/.claude/skills/agentv-eval-builder/references/batch-cli-evaluator.md

@@ -1,316 +0,0 @@
-# Batch CLI Evaluation Guide
-
-Guide for evaluating batch CLI output where a single runner processes all evalcases at once and outputs JSONL.
-
-## Overview
-
-Batch CLI evaluation is used when:
-- An external tool processes multiple inputs in a single invocation (e.g., AML screening, bulk classification)
-- The runner reads the eval YAML directly to extract all evalcases
-- Output is JSONL with records keyed by evalcase `id`
-- Each evalcase has its own evaluator to validate its corresponding output record
-
-## Execution Flow
-
-1. **AgentV** invokes the batch runner once, passing `--eval <yaml-path>` and `--output <jsonl-path>`
-2. **Batch runner** reads the eval YAML, extracts all evalcases, processes them, writes JSONL output keyed by `id`
-3. **AgentV** parses JSONL, routes each record to its matching evalcase by `id`
-4. **Per-case evaluator** validates the output for each evalcase independently
-
-## Eval File Structure
-
-```yaml
-description: Batch CLI demo using structured input_messages
-execution:
-  target: batch_cli
-
-evalcases:
-  - id: case-001
-    expected_outcome: |-
-      Batch runner returns JSON with decision=CLEAR.
-
-    expected_messages:
-      - role: assistant
-        content:
-          decision: CLEAR  # Structured expected output
-
-    input_messages:
-      - role: system
-        content: You are a batch processor.
-      - role: user
-        content:  # Structured input (runner extracts this)
-          request:
-            type: screening_check
-            jurisdiction: AU
-          row:
-            id: case-001
-            name: Example A
-            amount: 5000
-
-    execution:
-      evaluators:
-        - name: decision-check
-          type: code_judge
-          script: bun run ./scripts/check-output.ts
-          cwd: .
-
-  - id: case-002
-    expected_outcome: |-
-      Batch runner returns JSON with decision=REVIEW.
-
-    expected_messages:
-      - role: assistant
-        content:
-          decision: REVIEW
-
-    input_messages:
-      - role: system
-        content: You are a batch processor.
-      - role: user
-        content:
-          request:
-            type: screening_check
-            jurisdiction: AU
-          row:
-            id: case-002
-            name: Example B
-            amount: 25000
-
-    execution:
-      evaluators:
-        - name: decision-check
-          type: code_judge
-          script: bun run ./scripts/check-output.ts
-          cwd: .
-```
-
-## Batch Runner Implementation
-
-The batch runner reads the eval YAML directly and processes all evalcases in one invocation.
-
-### Runner Contract
-
-**Input:** The runner receives the eval file path via `--eval` flag:
-```bash
-bun run batch-runner.ts --eval ./my-eval.yaml --output ./results.jsonl
-```
-
-**Output:** JSONL file where each line is a JSON object with:
-```json
-{"id": "case-001", "text": "{\"decision\": \"CLEAR\", ...}"}
-{"id": "case-002", "text": "{\"decision\": \"REVIEW\", ...}"}
-```
-
-The `id` field must match the evalcase `id` for AgentV to route output to the correct evaluator.
-
-### Output with Tool Trajectory Support
-
-To enable `tool_trajectory` evaluation, include `output_messages` with `tool_calls`:
-
-```json
-{
-  "id": "case-001",
-  "text": "{\"decision\": \"CLEAR\", ...}",
-  "output_messages": [
-    {
-      "role": "assistant",
-      "tool_calls": [
-        {
-          "tool": "screening_check",
-          "input": { "origin_country": "NZ", "amount": 5000 },
-          "output": { "decision": "CLEAR", "reasons": [] }
-        }
-      ]
-    },
-    {
-      "role": "assistant",
-      "content": { "decision": "CLEAR" }
-    }
-  ]
-}
-```
-
-AgentV extracts tool calls directly from `output_messages[].tool_calls[]` for `tool_trajectory` evaluators. This is the recommended format for batch runners that make tool calls.
-
-### Example Runner (TypeScript)
-
-```typescript
-import fs from 'node:fs/promises';
-import { parse } from 'yaml';
-
-type EvalCase = {
-  id: string;
-  input_messages: Array<{ role: string; content: unknown }>;
-};
-
-async function main() {
-  const args = process.argv.slice(2);
-  const evalPath = getFlag(args, '--eval');
-  const outPath = getFlag(args, '--output');
-
-  // Read and parse eval YAML
-  const yamlText = await fs.readFile(evalPath, 'utf8');
-  const parsed = parse(yamlText);
-  const evalcases = parsed.evalcases as EvalCase[];
-
-  // Process each evalcase
-  const results: Array<{ id: string; text: string }> = [];
-  for (const evalcase of evalcases) {
-    const userContent = findUserContent(evalcase.input_messages);
-    const decision = processInput(userContent); // Your logic here
-
-    results.push({
-      id: evalcase.id,
-      text: JSON.stringify({ decision, ...otherFields }),
-    });
-  }
-
-  // Write JSONL output
-  const jsonl = results.map((r) => JSON.stringify(r)).join('\n') + '\n';
-  await fs.writeFile(outPath, jsonl, 'utf8');
-}
-
-function getFlag(args: string[], name: string): string {
-  const idx = args.indexOf(name);
-  return args[idx + 1];
-}
-
-function findUserContent(messages: Array<{ role: string; content: unknown }>) {
-  return messages.find((m) => m.role === 'user')?.content;
-}
-```
-
-## Evaluator Implementation
-
-Each evalcase has its own evaluator that validates the output. The evaluator receives the standard code_judge input.
-
-### Evaluator Contract
-
-**Input (stdin):** Standard AgentV code_judge format:
-```json
-{
-  "candidate_answer": "{\"id\":\"case-001\",\"decision\":\"CLEAR\",...}",
-  "expected_messages": [{"role": "assistant", "content": {"decision": "CLEAR"}}],
-  "input_messages": [...],
-  ...
-}
-```
-
-**Output (stdout):** Standard evaluator result:
-```json
-{
-  "score": 1.0,
-  "hits": ["decision matches: CLEAR"],
-  "misses": [],
-  "reasoning": "Batch runner decision matches expected."
-}
-```
-
-### Example Evaluator (TypeScript)
-
-```typescript
-import fs from 'node:fs';
-
-type EvalInput = {
-  candidate_answer?: string;
-  expected_messages?: Array<{ role: string; content: unknown }>;
-};
-
-function main() {
-  const stdin = fs.readFileSync(0, 'utf8');
-  const input = JSON.parse(stdin) as EvalInput;
-
-  // Extract expected value from expected_messages
-  const expectedDecision = findExpectedDecision(input.expected_messages);
-
-  // Parse candidate answer (output from batch runner)
-  let candidateDecision: string | undefined;
-  try {
-    const parsed = JSON.parse(input.candidate_answer ?? '');
-    candidateDecision = parsed.decision;
-  } catch {
-    candidateDecision = undefined;
-  }
-
-  // Compare
-  const hits: string[] = [];
-  const misses: string[] = [];
-
-  if (expectedDecision === candidateDecision) {
-    hits.push(`decision matches: ${expectedDecision}`);
-  } else {
-    misses.push(`mismatch: expected=${expectedDecision} actual=${candidateDecision}`);
-  }
-
-  const score = misses.length === 0 ? 1 : 0;
-
-  process.stdout.write(JSON.stringify({
-    score,
-    hits,
-    misses,
-    reasoning: score === 1
-      ? 'Batch runner output matches expected.'
-      : 'Batch runner output did not match expected.',
-  }));
-}
-
-function findExpectedDecision(messages?: Array<{ role: string; content: unknown }>) {
-  if (!messages) return undefined;
-  for (const msg of messages) {
-    if (typeof msg.content === 'object' && msg.content !== null) {
-      return (msg.content as Record<string, unknown>).decision as string;
-    }
-  }
-  return undefined;
-}
-
-main();
-```
-
-## Structured Content in expected_messages
-
-For batch evaluation, use structured objects in `expected_messages.content` to define expected output fields:
-
-```yaml
-expected_messages:
-  - role: assistant
-    content:
-      decision: CLEAR
-      confidence: high
-      reasons: []
-```
-
-The evaluator then extracts these fields and compares against the parsed candidate output.
-
-## Best Practices
-
-1. **Use unique evalcase IDs** - The batch runner and AgentV use `id` to route outputs
-2. **Structured input_messages** - Put structured data in `user.content` for the runner to extract
-3. **Structured expected_messages** - Define expected output as objects for easy validation
-4. **Deterministic runners** - Batch runners should produce consistent output for testing
-5. **Healthcheck support** - Add `--healthcheck` flag for runner validation:
-   ```typescript
-   if (args.includes('--healthcheck')) {
-     console.log('batch-runner: healthy');
-     return;
-   }
-   ```
-
-## Target Configuration
-
-Configure the batch CLI provider in your target:
-
-```yaml
-# In agentv-targets.yaml or eval file
-targets:
-  batch_cli:
-    provider: cli
-    commandTemplate: bun run ./scripts/batch-runner.ts --eval {EVAL_FILE} --output {OUTPUT_FILE}
-    provider_batching: true
-```
-
-Key settings:
-- `provider: cli` - Use CLI provider
-- `provider_batching: true` - Run once for all evalcases
-- `{EVAL_FILE}` - Placeholder for eval file path
-- `{OUTPUT_FILE}` - Placeholder for JSONL output path

package/dist/templates/.claude/skills/agentv-eval-builder/references/compare-command.md

@@ -1,137 +0,0 @@
-# Compare Command
-
-Compare evaluation results between two runs to measure performance differences.
-
-## Usage
-
-```bash
-agentv compare <baseline.jsonl> <candidate.jsonl> [options]
-```
-
-## Arguments
-
-| Argument | Description |
-|----------|-------------|
-| `result1` | Path to baseline JSONL result file |
-| `result2` | Path to candidate JSONL result file |
-| `--threshold`, `-t` | Score delta threshold for win/loss classification (default: 0.1) |
-| `--format`, `-f` | Output format: `table` (default) or `json` |
-| `--json` | Shorthand for `--format=json` |
-
-## How It Works
-
-1. **Load Results**: Reads both JSONL files containing evaluation results
-2. **Match by eval_id**: Pairs results with matching `eval_id` fields
-3. **Compute Deltas**: Calculates `delta = score2 - score1` for each pair
-4. **Classify Outcomes**:
-   - `win`: delta >= threshold (candidate better)
-   - `loss`: delta <= -threshold (baseline better)
-   - `tie`: |delta| < threshold (no significant difference)
-5. **Output Summary**: Human-readable table (default) or JSON
-
-## Output Format
-
-### Table Format (default)
-
-```
-Comparing: baseline.jsonl → candidate.jsonl
-
-  Eval ID        Baseline  Candidate     Delta  Result
-  ─────────────  ────────  ─────────  ────────  ────────
-  safety-check       0.70       0.90     +0.20  ✓ win
-  accuracy-test      0.85       0.80     -0.05  = tie
-  latency-eval       0.90       0.75     -0.15  ✗ loss
-
-Summary: 1 win, 1 loss, 1 tie | Mean Δ: +0.000 | Status: neutral
-```
-
-Colors are used to highlight wins (green), losses (red), and ties (gray). Colors are automatically disabled when output is piped or `NO_COLOR` is set.
-
-### JSON Format (`--json`)
-
-Output uses snake_case for Python ecosystem compatibility:
-
-```json
-{
-  "matched": [
-    {
-      "eval_id": "case-1",
-      "score1": 0.7,
-      "score2": 0.9,
-      "delta": 0.2,
-      "outcome": "win"
-    }
-  ],
-  "unmatched": {
-    "file1": 0,
-    "file2": 0
-  },
-  "summary": {
-    "total": 2,
-    "matched": 1,
-    "wins": 1,
-    "losses": 0,
-    "ties": 0,
-    "mean_delta": 0.2
-  }
-}
-```
-
-## Exit Codes
-
-| Code | Meaning |
-|------|---------|
-| `0` | Candidate is equal or better (meanDelta >= 0) |
-| `1` | Baseline is better (regression detected) |
-
-## Workflow Examples
-
-### Model Comparison
-
-Compare different model versions:
-
-```bash
-# Run baseline evaluation
-agentv eval evals/*.yaml --target gpt-4 --out baseline.jsonl
-
-# Run candidate evaluation
-agentv eval evals/*.yaml --target gpt-4o --out candidate.jsonl
-
-# Compare results
-agentv compare baseline.jsonl candidate.jsonl
-```
-
-### Prompt Optimization
-
-Compare before/after prompt changes:
-
-```bash
-# Run with original prompt
-agentv eval evals/*.yaml --out before.jsonl
-
-# Modify prompt, then run again
-agentv eval evals/*.yaml --out after.jsonl
-
-# Compare with strict threshold
-agentv compare before.jsonl after.jsonl --threshold 0.05
-```
-
-### CI Quality Gate
-
-Fail CI if candidate regresses:
-
-```bash
-#!/bin/bash
-agentv compare baseline.jsonl candidate.jsonl
-if [ $? -eq 1 ]; then
-  echo "Regression detected! Candidate performs worse than baseline."
-  exit 1
-fi
-echo "Candidate is equal or better than baseline."
-```
-
-## Tips
-
-- **Threshold Selection**: Default 0.1 means 10% difference required. Use stricter thresholds (0.05) for critical evaluations.
-- **Unmatched Results**: Check `unmatched` counts to identify eval cases that only exist in one file.
-- **Multiple Comparisons**: Compare against multiple baselines by running the command multiple times.