agentv 3.10.2 → 3.10.3

package/dist/templates/.agents/skills/agentv-eval-builder/references/rubric-evaluator.md

@@ -1,77 +0,0 @@
-# Rubric Evaluator
-
-## Field Reference
-
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `id` | string | auto-generated | Unique identifier |
-| `outcome` | string | required* | Criterion being evaluated (*optional if `score_ranges` used) |
-| `weight` | number | 1.0 | Relative importance |
-| `required` | boolean | true | Failing forces verdict to 'fail' (checklist mode) |
-| `required_min_score` | integer | - | Minimum 0-10 score to pass (score-range mode) |
-| `score_ranges` | map or array | - | Score range definitions for analytic scoring |
-
-## Checklist Mode
-
-```yaml
-rubrics:
-  - Mentions divide-and-conquer approach
-  - id: complexity
-    outcome: States time complexity correctly
-    weight: 2.0
-    required: true
-  - id: examples
-    outcome: Includes code examples
-    weight: 1.0
-    required: false
-```
-
-## Score-Range Mode
-
-Shorthand map format (recommended):
-
-```yaml
-rubrics:
-  - id: correctness
-    weight: 2.0
-    required_min_score: 7
-    score_ranges:
-      0: Critical bugs
-      3: Minor bugs
-      6: Correct with minor issues
-      9: Fully correct
-```
-
-Map keys are lower bounds (0-10). Each range extends from its key to (next key - 1), with the last extending to 10. Must start at 0.
-
-Array format is also accepted:
-
-```yaml
-    score_ranges:
-      - score_range: [0, 2]
-        outcome: Critical bugs
-      - score_range: [3, 5]
-        outcome: Minor bugs
-      - score_range: [6, 8]
-        outcome: Correct with minor issues
-      - score_range: [9, 10]
-        outcome: Fully correct
-```
-
-Ranges must be integers 0-10, non-overlapping, covering all values 0-10.
-
-## Scoring
-
-**Checklist:** `score = sum(satisfied weights) / sum(all weights)`
-
-**Score-range:** `score = weighted_average(raw_score / 10)` per criterion
-
-## Verdicts
-
-| Verdict | Condition |
-|---------|-----------|
-| `pass` | score >= 0.8 AND all gating criteria satisfied |
-| `borderline` | score >= 0.6 AND all gating criteria satisfied |
-| `fail` | score < 0.6 OR any gating criterion failed |
-
-Gating: checklist uses `required: true`, score-range uses `required_min_score: N`.

package/dist/templates/.agents/skills/agentv-eval-orchestrator/SKILL.md

@@ -1,50 +0,0 @@
----
-name: agentv-eval-orchestrator
-description: Run AgentV evaluations without API keys by orchestrating eval subcommands. Use this skill when asked to run evals, evaluate an agent, or test prompt quality using agentv.
----
-
-# AgentV Eval Orchestrator
-
-Run AgentV evaluations by acting as the LLM yourself — no API keys needed.
-
-## Quick Start
-
-```bash
-agentv prompt eval <eval-file.yaml>
-```
-
-This outputs a complete orchestration prompt with step-by-step instructions and all test IDs. Follow its instructions.
-
-## Workflow
-
-For each test, run these three steps:
-
-### 1. Get Task Input
-
-```bash
-agentv prompt eval input <path> --test-id <id>
-```
-
-Returns JSON with `input`, `guideline_paths`, and `criteria`. File references in messages use absolute paths — read them from the filesystem.
-
-### 2. Execute the Task
-
-You ARE the candidate LLM. Read `input` from step 1, read any referenced files, and answer the task. Save your response to a temp file.
-
-**Important**: Do not leak `criteria` into your answer — it's for your reference when judging, not part of the task.
-
-### 3. Judge the Result
-
-```bash
-agentv prompt eval judge <path> --test-id <id> --answer-file /tmp/eval_<id>.txt
-```
-
-Returns JSON with an `evaluators` array. Each evaluator has a `status`:
-
-- **`"completed"`** — Deterministic score is final. Read `result.score` (0.0–1.0).
-- **`"prompt_ready"`** — LLM grading required. Send `prompt.system_prompt` as system and `prompt.user_prompt` as user to yourself. Parse the JSON response to get `score`, `hits`, `misses`.
-
-## When to use this vs `agentv eval`
-
-- **`agentv eval`** — You have API keys configured. Runs everything end-to-end automatically.
-- **`agentv prompt`** — No API keys. You orchestrate: get input, answer the task yourself, judge the result.

package/dist/templates/.agents/skills/agentv-prompt-optimizer/SKILL.md

@@ -1,78 +0,0 @@
----
-name: agentv-prompt-optimizer
-description: Iteratively optimize prompt files against AgentV evaluation datasets by analyzing failures and refining instructions.
----
-
-# AgentV Prompt Optimizer
-
-## Input Variables
-- `eval-path`: Path or glob pattern to the AgentV evaluation file(s) to optimize against
-- `optimization-log-path` (optional): Path where optimization progress should be logged
-
-## Workflow
-
-1.  **Initialize**
-    - Verify `<eval-path>` (file or glob) targets the correct system.
-    - **Identify Prompt Files**:
-        - Infer prompt files from the eval file content (look for `file:` references in `input` that match these patterns).
-        - Recursively check referenced prompt files for *other* prompt references (dependencies).
-        - If multiple prompts are found, consider ALL of them as candidates for optimization.
-    - **Identify Optimization Log**:
-        - If `<optimization-log-path>` is provided, use it.
-        - If not, create a new one in the parent directory of the eval files: `optimization-[timestamp].md`.
-    - Read content of the identified prompt file.
-
-2.  **Optimization Loop** (Max 10 iterations)
-    - **Execute (The Generator)**: Run `agentv eval <eval-path>`.
-        - *Targeted Run*: If iterating on specific stubborn failures, use `--test-id <test_id>` to run only the relevant tests.
-    - **Analyze (The Reflector)**:
-        - Locate the results file path from the console output (e.g., `.agentv/results/eval_...jsonl`).
-        - **Orchestrate Subagent**: Use `runSubagent` to analyze the results.
-            - **Task**: Read the results file, calculate pass rate, and perform root cause analysis.
-            - **Output**: Return a structured analysis including:
-                - **Score**: Current pass rate.
-                - **Root Cause**: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
-                - **Insight**: Key learning or pattern identified from the failures.
-                - **Strategy**: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
-    - **Decide**:
-        - If **100% pass**: STOP and report success.
-        - If **Score decreased**: Revert last change, try different approach.
-        - If **No improvement** (2x): STOP and report stagnation.
-    - **Refine (The Curator)**:
-        - **Orchestrate Subagent**: Use `runSubagent` to apply the fix.
-            - **Task**: Read the relevant prompt file(s), apply the **Strategy** from the Reflector, and generate the log entry.
-            - **Output**: The **Log Entry** describing the specific operation performed.
-                  ```markdown
-                  ### Iteration [N]
-                  - **Operation**: [ADD / UPDATE / DELETE]
-                  - **Target**: [Section Name]
-                  - **Change**: [Specific text added/modified]
-                  - **Trigger**: [Specific failing test case or error pattern]
-                  - **Rationale**: [From Reflector: Root Cause]
-                  - **Score**: [From Reflector: Current Pass Rate]
-                  - **Insight**: [From Reflector: Key Learning]
-                  ```
-        - **Strategy**: Treat the prompt as a structured set of rules. Execute atomic operations:
-            - **ADD**: Insert a new rule if a constraint was missed.
-            - **UPDATE**: Refine an existing rule to be clearer or more general.
-                - *Clarify*: Make ambiguous instructions specific.
-                - *Generalize*: Refactor specific fixes into high-level principles (First Principles).
-            - **DELETE**: Remove obsolete, redundant, or harmful rules.
-                - *Prune*: If a general rule covers specific cases, delete the specific ones.
-            - **Negative Constraint**: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
-            - **Safety Check**: Ensure new rules don't contradict existing ones (unless intended).
-        - **Constraint**: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
-    - **Log Result**:
-        - Append the **Log Entry** returned by the Curator to the optimization log file.
-
-3.  **Completion**
-    - Report final score.
-    - Summarize key changes made to the prompt.
-    - **Finalize Optimization Log**: Add a summary header to the optimization log file indicating the session completion and final score.
-
-## Guidelines
-- **Generalization First**: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
-- **Simplicity ("Less is More")**: Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
-- **Structure**: Maintain existing Markdown headers/sections.
-- **Progressive Disclosure**: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
-- **Quality Criteria**: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.

package/dist/templates/.claude/skills/agentv-eval-builder/SKILL.md

@@ -1,177 +0,0 @@
----
-name: agentv-eval-builder
-description: Create and maintain AgentV YAML evaluation files for testing AI agent performance. Use this skill when creating new eval files, adding eval cases, or configuring evaluators.
----
-
-# AgentV Eval Builder
-
-Comprehensive docs: https://agentv.dev
-
-## Quick Start
-
-```yaml
-description: Example eval
-execution:
-  target: default
-
-evalcases:
-  - id: greeting
-    expected_outcome: Friendly greeting
-    input: "Say hello"
-    expected_output: "Hello! How can I help you?"
-    rubrics:
-      - Greeting is friendly and warm
-      - Offers to help
-```
-
-## Eval File Structure
-
-**Required:** `evalcases` (array)
-**Optional:** `description`, `execution`, `dataset`
-
-**Eval case fields:**
-
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | yes | Unique identifier |
-| `expected_outcome` | yes | What the response should accomplish |
-| `input` / `input_messages` | yes | Input to the agent |
-| `expected_output` / `expected_messages` | no | Gold-standard reference answer |
-| `rubrics` | no | Inline evaluation criteria |
-| `execution` | no | Per-case execution overrides |
-| `conversation_id` | no | Thread grouping |
-
-**Shorthand aliases:**
-- `input` (string) expands to `[{role: "user", content: "..."}]`
-- `expected_output` (string/object) expands to `[{role: "assistant", content: ...}]`
-- Canonical `input_messages` / `expected_messages` take precedence when both present
-
-**Message format:** `{role, content}` where role is `system`, `user`, `assistant`, or `tool`
-**Content types:** inline text, `{type: "file", value: "./path.md"}`
-**File paths:** relative from eval file dir, or absolute with `/` prefix from repo root
-
-**JSONL format:** One eval case per line as JSON. Optional `.yaml` sidecar for shared defaults. See `examples/features/basic-jsonl/`.
-
-## Evaluator Types
-
-Configure via `execution.evaluators` array. Multiple evaluators produce a weighted average score.
-
-### code_judge
-```yaml
-- name: format_check
-  type: code_judge
-  script: uv run validate.py
-  cwd: ./scripts          # optional working directory
-  target: {}              # optional: enable LLM target proxy (max_calls: 50)
-```
-Contract: stdin JSON -> stdout JSON `{score, hits, misses, reasoning}`
-See `references/custom-evaluators.md` for templates.
-
-### llm_judge
-```yaml
-- name: quality
-  type: llm_judge
-  prompt: ./prompts/eval.md     # markdown template or script config
-  model: gpt-5-chat            # optional model override
-  config:                       # passed to script templates as context.config
-    strictness: high
-```
-Variables: `{{question}}`, `{{expected_outcome}}`, `{{candidate_answer}}`, `{{reference_answer}}`, `{{input_messages}}`, `{{expected_messages}}`, `{{output_messages}}`
-- Markdown templates: use `{{variable}}` syntax
-- TypeScript templates: use `definePromptTemplate(fn)` from `@agentv/eval`, receives context object with all variables + `config`
-
-### composite
-```yaml
-- name: gate
-  type: composite
-  evaluators:
-    - name: safety
-      type: llm_judge
-      prompt: ./safety.md
-    - name: quality
-      type: llm_judge
-  aggregator:
-    type: weighted_average
-    weights: { safety: 0.3, quality: 0.7 }
-```
-Aggregator types: `weighted_average`, `all_or_nothing`, `minimum`, `maximum`, `safety_gate`
-- `safety_gate`: fails immediately if the named gate evaluator scores below threshold (default 1.0)
-
-### tool_trajectory
-```yaml
-- name: tool_check
-  type: tool_trajectory
-  mode: any_order            # any_order | in_order | exact
-  minimums:                  # for any_order
-    knowledgeSearch: 2
-  expected:                  # for in_order/exact
-    - tool: knowledgeSearch
-      args: { query: "search term" }   # partial deep equality match
-    - tool: documentRetrieve
-      args: any                        # any arguments accepted
-      max_duration_ms: 5000            # per-tool latency assertion
-    - tool: summarize                  # omit args to skip argument checking
-```
-
-### field_accuracy
-```yaml
-- name: fields
-  type: field_accuracy
-  match_type: exact          # exact | date | numeric_tolerance
-  numeric_tolerance: 0.01    # for numeric_tolerance match_type
-  aggregation: weighted_average  # weighted_average | all_or_nothing
-```
-Compares `output_messages` fields against `expected_messages` fields.
-
-### latency
-```yaml
-- name: speed
-  type: latency
-  max_ms: 5000
-```
-
-### cost
-```yaml
-- name: budget
-  type: cost
-  max_usd: 0.10
-```
-
-### token_usage
-```yaml
-- name: tokens
-  type: token_usage
-  max_total_tokens: 4000
-```
-
-### rubric (inline)
-```yaml
-rubrics:
-  - Simple string criterion
-  - id: weighted
-    expected_outcome: Detailed criterion
-    weight: 2.0
-    required: true
-```
-See `references/rubric-evaluator.md` for score-range mode and scoring formula.
-
-## CLI Commands
-
-```bash
-# Run evaluation
-bun agentv eval <file.yaml> [--eval-id <id>] [--target <name>] [--dry-run]
-
-# Validate eval file
-bun agentv validate <file.yaml>
-
-# Compare results between runs
-bun agentv compare <results1.jsonl> <results2.jsonl>
-
-# Generate rubrics from expected_outcome
-bun agentv generate rubrics <file.yaml> [--target <name>]
-```
-
-## Schemas
-
-- Eval file: `references/eval-schema.json`
-- Config: `references/config-schema.json`

package/dist/templates/.claude/skills/agentv-eval-builder/references/batch-cli-evaluator.md

@@ -1,316 +0,0 @@
-# Batch CLI Evaluation Guide
-
-Guide for evaluating batch CLI output where a single runner processes all evalcases at once and outputs JSONL.
-
-## Overview
-
-Batch CLI evaluation is used when:
-- An external tool processes multiple inputs in a single invocation (e.g., AML screening, bulk classification)
-- The runner reads the eval YAML directly to extract all evalcases
-- Output is JSONL with records keyed by evalcase `id`
-- Each evalcase has its own evaluator to validate its corresponding output record
-
-## Execution Flow
-
-1. **AgentV** invokes the batch runner once, passing `--eval <yaml-path>` and `--output <jsonl-path>`
-2. **Batch runner** reads the eval YAML, extracts all evalcases, processes them, writes JSONL output keyed by `id`
-3. **AgentV** parses JSONL, routes each record to its matching evalcase by `id`
-4. **Per-case evaluator** validates the output for each evalcase independently
-
-## Eval File Structure
-
-```yaml
-description: Batch CLI demo using structured input_messages
-execution:
-  target: batch_cli
-
-evalcases:
-  - id: case-001
-    expected_outcome: |-
-      Batch runner returns JSON with decision=CLEAR.
-
-    expected_messages:
-      - role: assistant
-        content:
-          decision: CLEAR  # Structured expected output
-
-    input_messages:
-      - role: system
-        content: You are a batch processor.
-      - role: user
-        content:  # Structured input (runner extracts this)
-          request:
-            type: screening_check
-            jurisdiction: AU
-          row:
-            id: case-001
-            name: Example A
-            amount: 5000
-
-    execution:
-      evaluators:
-        - name: decision-check
-          type: code_judge
-          script: bun run ./scripts/check-output.ts
-          cwd: .
-
-  - id: case-002
-    expected_outcome: |-
-      Batch runner returns JSON with decision=REVIEW.
-
-    expected_messages:
-      - role: assistant
-        content:
-          decision: REVIEW
-
-    input_messages:
-      - role: system
-        content: You are a batch processor.
-      - role: user
-        content:
-          request:
-            type: screening_check
-            jurisdiction: AU
-          row:
-            id: case-002
-            name: Example B
-            amount: 25000
-
-    execution:
-      evaluators:
-        - name: decision-check
-          type: code_judge
-          script: bun run ./scripts/check-output.ts
-          cwd: .
-```
-
-## Batch Runner Implementation
-
-The batch runner reads the eval YAML directly and processes all evalcases in one invocation.
-
-### Runner Contract
-
-**Input:** The runner receives the eval file path via `--eval` flag:
-```bash
-bun run batch-runner.ts --eval ./my-eval.yaml --output ./results.jsonl
-```
-
-**Output:** JSONL file where each line is a JSON object with:
-```json
-{"id": "case-001", "text": "{\"decision\": \"CLEAR\", ...}"}
-{"id": "case-002", "text": "{\"decision\": \"REVIEW\", ...}"}
-```
-
-The `id` field must match the evalcase `id` for AgentV to route output to the correct evaluator.
-
-### Output with Tool Trajectory Support
-
-To enable `tool_trajectory` evaluation, include `output_messages` with `tool_calls`:
-
-```json
-{
-  "id": "case-001",
-  "text": "{\"decision\": \"CLEAR\", ...}",
-  "output_messages": [
-    {
-      "role": "assistant",
-      "tool_calls": [
-        {
-          "tool": "screening_check",
-          "input": { "origin_country": "NZ", "amount": 5000 },
-          "output": { "decision": "CLEAR", "reasons": [] }
-        }
-      ]
-    },
-    {
-      "role": "assistant",
-      "content": { "decision": "CLEAR" }
-    }
-  ]
-}
-```
-
-AgentV extracts tool calls directly from `output_messages[].tool_calls[]` for `tool_trajectory` evaluators. This is the recommended format for batch runners that make tool calls.
-
-### Example Runner (TypeScript)
-
-```typescript
-import fs from 'node:fs/promises';
-import { parse } from 'yaml';
-
-type EvalCase = {
-  id: string;
-  input_messages: Array<{ role: string; content: unknown }>;
-};
-
-async function main() {
-  const args = process.argv.slice(2);
-  const evalPath = getFlag(args, '--eval');
-  const outPath = getFlag(args, '--output');
-
-  // Read and parse eval YAML
-  const yamlText = await fs.readFile(evalPath, 'utf8');
-  const parsed = parse(yamlText);
-  const evalcases = parsed.evalcases as EvalCase[];
-
-  // Process each evalcase
-  const results: Array<{ id: string; text: string }> = [];
-  for (const evalcase of evalcases) {
-    const userContent = findUserContent(evalcase.input_messages);
-    const decision = processInput(userContent); // Your logic here
-
-    results.push({
-      id: evalcase.id,
-      text: JSON.stringify({ decision, ...otherFields }),
-    });
-  }
-
-  // Write JSONL output
-  const jsonl = results.map((r) => JSON.stringify(r)).join('\n') + '\n';
-  await fs.writeFile(outPath, jsonl, 'utf8');
-}
-
-function getFlag(args: string[], name: string): string {
-  const idx = args.indexOf(name);
-  return args[idx + 1];
-}
-
-function findUserContent(messages: Array<{ role: string; content: unknown }>) {
-  return messages.find((m) => m.role === 'user')?.content;
-}
-```
-
-## Evaluator Implementation
-
-Each evalcase has its own evaluator that validates the output. The evaluator receives the standard code_judge input.
-
-### Evaluator Contract
-
-**Input (stdin):** Standard AgentV code_judge format:
-```json
-{
-  "candidate_answer": "{\"id\":\"case-001\",\"decision\":\"CLEAR\",...}",
-  "expected_messages": [{"role": "assistant", "content": {"decision": "CLEAR"}}],
-  "input_messages": [...],
-  ...
-}
-```
-
-**Output (stdout):** Standard evaluator result:
-```json
-{
-  "score": 1.0,
-  "hits": ["decision matches: CLEAR"],
-  "misses": [],
-  "reasoning": "Batch runner decision matches expected."
-}
-```
-
-### Example Evaluator (TypeScript)
-
-```typescript
-import fs from 'node:fs';
-
-type EvalInput = {
-  candidate_answer?: string;
-  expected_messages?: Array<{ role: string; content: unknown }>;
-};
-
-function main() {
-  const stdin = fs.readFileSync(0, 'utf8');
-  const input = JSON.parse(stdin) as EvalInput;
-
-  // Extract expected value from expected_messages
-  const expectedDecision = findExpectedDecision(input.expected_messages);
-
-  // Parse candidate answer (output from batch runner)
-  let candidateDecision: string | undefined;
-  try {
-    const parsed = JSON.parse(input.candidate_answer ?? '');
-    candidateDecision = parsed.decision;
-  } catch {
-    candidateDecision = undefined;
-  }
-
-  // Compare
-  const hits: string[] = [];
-  const misses: string[] = [];
-
-  if (expectedDecision === candidateDecision) {
-    hits.push(`decision matches: ${expectedDecision}`);
-  } else {
-    misses.push(`mismatch: expected=${expectedDecision} actual=${candidateDecision}`);
-  }
-
-  const score = misses.length === 0 ? 1 : 0;
-
-  process.stdout.write(JSON.stringify({
-    score,
-    hits,
-    misses,
-    reasoning: score === 1
-      ? 'Batch runner output matches expected.'
-      : 'Batch runner output did not match expected.',
-  }));
-}
-
-function findExpectedDecision(messages?: Array<{ role: string; content: unknown }>) {
-  if (!messages) return undefined;
-  for (const msg of messages) {
-    if (typeof msg.content === 'object' && msg.content !== null) {
-      return (msg.content as Record<string, unknown>).decision as string;
-    }
-  }
-  return undefined;
-}
-
-main();
-```
-
-## Structured Content in expected_messages
-
-For batch evaluation, use structured objects in `expected_messages.content` to define expected output fields:
-
-```yaml
-expected_messages:
-  - role: assistant
-    content:
-      decision: CLEAR
-      confidence: high
-      reasons: []
-```
-
-The evaluator then extracts these fields and compares against the parsed candidate output.
-
-## Best Practices
-
-1. **Use unique evalcase IDs** - The batch runner and AgentV use `id` to route outputs
-2. **Structured input_messages** - Put structured data in `user.content` for the runner to extract
-3. **Structured expected_messages** - Define expected output as objects for easy validation
-4. **Deterministic runners** - Batch runners should produce consistent output for testing
-5. **Healthcheck support** - Add `--healthcheck` flag for runner validation:
-   ```typescript
-   if (args.includes('--healthcheck')) {
-     console.log('batch-runner: healthy');
-     return;
-   }
-   ```
-
-## Target Configuration
-
-Configure the batch CLI provider in your target:
-
-```yaml
-# In agentv-targets.yaml or eval file
-targets:
-  batch_cli:
-    provider: cli
-    commandTemplate: bun run ./scripts/batch-runner.ts --eval {EVAL_FILE} --output {OUTPUT_FILE}
-    provider_batching: true
-```
-
-Key settings:
-- `provider: cli` - Use CLI provider
-- `provider_batching: true` - Run once for all evalcases
-- `{EVAL_FILE}` - Placeholder for eval file path
-- `{OUTPUT_FILE}` - Placeholder for JSONL output path