agentv 3.9.2 → 3.10.1

package/dist/templates/.agents/skills/agentv-eval-builder/references/rubric-evaluator.md

@@ -0,0 +1,77 @@
+# Rubric Evaluator
+
+## Field Reference
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `id` | string | auto-generated | Unique identifier |
+| `outcome` | string | required* | Criterion being evaluated (*optional if `score_ranges` used) |
+| `weight` | number | 1.0 | Relative importance |
+| `required` | boolean | true | Failing forces verdict to 'fail' (checklist mode) |
+| `required_min_score` | integer | - | Minimum 0-10 score to pass (score-range mode) |
+| `score_ranges` | map or array | - | Score range definitions for analytic scoring |
+
+## Checklist Mode
+
+```yaml
+rubrics:
+  - Mentions divide-and-conquer approach
+  - id: complexity
+    outcome: States time complexity correctly
+    weight: 2.0
+    required: true
+  - id: examples
+    outcome: Includes code examples
+    weight: 1.0
+    required: false
+```
+
+## Score-Range Mode
+
+Shorthand map format (recommended):
+
+```yaml
+rubrics:
+  - id: correctness
+    weight: 2.0
+    required_min_score: 7
+    score_ranges:
+      0: Critical bugs
+      3: Minor bugs
+      6: Correct with minor issues
+      9: Fully correct
+```
+
+Map keys are lower bounds (0-10). Each range extends from its key to (next key - 1), with the last extending to 10. Must start at 0.
+
+Array format is also accepted:
+
+```yaml
+    score_ranges:
+      - score_range: [0, 2]
+        outcome: Critical bugs
+      - score_range: [3, 5]
+        outcome: Minor bugs
+      - score_range: [6, 8]
+        outcome: Correct with minor issues
+      - score_range: [9, 10]
+        outcome: Fully correct
+```
+
+Ranges must be integers 0-10, non-overlapping, covering all values 0-10.
+
+## Scoring
+
+**Checklist:** `score = sum(satisfied weights) / sum(all weights)`
+
+**Score-range:** `score = weighted_average(raw_score / 10)` per criterion
+
+## Verdicts
+
+| Verdict | Condition |
+|---------|-----------|
+| `pass` | score >= 0.8 AND all gating criteria satisfied |
+| `borderline` | score >= 0.6 AND all gating criteria satisfied |
+| `fail` | score < 0.6 OR any gating criterion failed |
+
+Gating: checklist uses `required: true`, score-range uses `required_min_score: N`.

package/dist/templates/.agents/skills/agentv-eval-orchestrator/SKILL.md

@@ -0,0 +1,50 @@
+---
+name: agentv-eval-orchestrator
+description: Run AgentV evaluations without API keys by orchestrating eval subcommands. Use this skill when asked to run evals, evaluate an agent, or test prompt quality using agentv.
+---
+
+# AgentV Eval Orchestrator
+
+Run AgentV evaluations by acting as the LLM yourself — no API keys needed.
+
+## Quick Start
+
+```bash
+agentv prompt eval <eval-file.yaml>
+```
+
+This outputs a complete orchestration prompt with step-by-step instructions and all test IDs. Follow its instructions.
+
+## Workflow
+
+For each test, run these three steps:
+
+### 1. Get Task Input
+
+```bash
+agentv prompt eval input <path> --test-id <id>
+```
+
+Returns JSON with `input`, `guideline_paths`, and `criteria`. File references in messages use absolute paths — read them from the filesystem.
+
+### 2. Execute the Task
+
+You ARE the candidate LLM. Read `input` from step 1, read any referenced files, and answer the task. Save your response to a temp file.
+
+**Important**: Do not leak `criteria` into your answer — it's for your reference when judging, not part of the task.
+
+### 3. Judge the Result
+
+```bash
+agentv prompt eval judge <path> --test-id <id> --answer-file /tmp/eval_<id>.txt
+```
+
+Returns JSON with an `evaluators` array. Each evaluator has a `status`:
+
+- **`"completed"`** — Deterministic score is final. Read `result.score` (0.0–1.0).
+- **`"prompt_ready"`** — LLM grading required. Send `prompt.system_prompt` as system and `prompt.user_prompt` as user to yourself. Parse the JSON response to get `score`, `hits`, `misses`.
+
+## When to use this vs `agentv eval`
+
+- **`agentv eval`** — You have API keys configured. Runs everything end-to-end automatically.
+- **`agentv prompt`** — No API keys. You orchestrate: get input, answer the task yourself, judge the result.

package/dist/templates/.agents/skills/agentv-prompt-optimizer/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: agentv-prompt-optimizer
+description: Iteratively optimize prompt files against AgentV evaluation datasets by analyzing failures and refining instructions.
+---
+
+# AgentV Prompt Optimizer
+
+## Input Variables
+- `eval-path`: Path or glob pattern to the AgentV evaluation file(s) to optimize against
+- `optimization-log-path` (optional): Path where optimization progress should be logged
+
+## Workflow
+
+1.  **Initialize**
+    - Verify `<eval-path>` (file or glob) targets the correct system.
+    - **Identify Prompt Files**:
+        - Infer prompt files from the eval file content (look for `file:` references in `input` that match these patterns).
+        - Recursively check referenced prompt files for *other* prompt references (dependencies).
+        - If multiple prompts are found, consider ALL of them as candidates for optimization.
+    - **Identify Optimization Log**:
+        - If `<optimization-log-path>` is provided, use it.
+        - If not, create a new one in the parent directory of the eval files: `optimization-[timestamp].md`.
+    - Read content of the identified prompt file.
+
+2.  **Optimization Loop** (Max 10 iterations)
+    - **Execute (The Generator)**: Run `agentv eval <eval-path>`.
+        - *Targeted Run*: If iterating on specific stubborn failures, use `--test-id <test_id>` to run only the relevant tests.
+    - **Analyze (The Reflector)**:
+        - Locate the results file path from the console output (e.g., `.agentv/results/eval_...jsonl`).
+        - **Orchestrate Subagent**: Use `runSubagent` to analyze the results.
+            - **Task**: Read the results file, calculate pass rate, and perform root cause analysis.
+            - **Output**: Return a structured analysis including:
+                - **Score**: Current pass rate.
+                - **Root Cause**: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
+                - **Insight**: Key learning or pattern identified from the failures.
+                - **Strategy**: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
+    - **Decide**:
+        - If **100% pass**: STOP and report success.
+        - If **Score decreased**: Revert last change, try different approach.
+        - If **No improvement** (2x): STOP and report stagnation.
+    - **Refine (The Curator)**:
+        - **Orchestrate Subagent**: Use `runSubagent` to apply the fix.
+            - **Task**: Read the relevant prompt file(s), apply the **Strategy** from the Reflector, and generate the log entry.
+            - **Output**: The **Log Entry** describing the specific operation performed.
+                  ```markdown
+                  ### Iteration [N]
+                  - **Operation**: [ADD / UPDATE / DELETE]
+                  - **Target**: [Section Name]
+                  - **Change**: [Specific text added/modified]
+                  - **Trigger**: [Specific failing test case or error pattern]
+                  - **Rationale**: [From Reflector: Root Cause]
+                  - **Score**: [From Reflector: Current Pass Rate]
+                  - **Insight**: [From Reflector: Key Learning]
+                  ```
+        - **Strategy**: Treat the prompt as a structured set of rules. Execute atomic operations:
+            - **ADD**: Insert a new rule if a constraint was missed.
+            - **UPDATE**: Refine an existing rule to be clearer or more general.
+                - *Clarify*: Make ambiguous instructions specific.
+                - *Generalize*: Refactor specific fixes into high-level principles (First Principles).
+            - **DELETE**: Remove obsolete, redundant, or harmful rules.
+                - *Prune*: If a general rule covers specific cases, delete the specific ones.
+            - **Negative Constraint**: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
+            - **Safety Check**: Ensure new rules don't contradict existing ones (unless intended).
+        - **Constraint**: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
+    - **Log Result**:
+        - Append the **Log Entry** returned by the Curator to the optimization log file.
+
+3.  **Completion**
+    - Report final score.
+    - Summarize key changes made to the prompt.
+    - **Finalize Optimization Log**: Add a summary header to the optimization log file indicating the session completion and final score.
+
+## Guidelines
+- **Generalization First**: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
+- **Simplicity ("Less is More")**: Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
+- **Structure**: Maintain existing Markdown headers/sections.
+- **Progressive Disclosure**: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
+- **Quality Criteria**: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.

package/dist/templates/.agentv/.env.example

@@ -0,0 +1,25 @@
+# Copy this file to .env and fill in your credentials
+
+# Eval run mode (used by agentv-bench skill)
+AGENT_EVAL_MODE=agent # agent | cli
+
+# Azure OpenAI Configuration
+AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
+AZURE_OPENAI_API_KEY=your-openai-api-key-here
+AZURE_DEPLOYMENT_NAME=gpt-5-mini
+AZURE_OPENAI_API_VERSION=2024-12-01-preview
+
+# OpenAI
+OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
+OPENAI_API_KEY=your-openai-api-key-here
+OPENAI_MODEL=gpt-5-mini
+
+# Google Gemini
+GOOGLE_GENERATIVE_AI_API_KEY=your-gemini-api-key-here
+GEMINI_MODEL_NAME=gemini-3-flash-preview
+
+# Anthropic
+ANTHROPIC_API_KEY=your-anthropic-api-key-here
+
+# CLI provider sample (used by the local_cli target)
+CLI_EVALS_DIR=./docs/examples/simple/evals/local-cli

package/dist/templates/.claude/skills/agentv-eval-builder/SKILL.md

@@ -0,0 +1,177 @@
+---
+name: agentv-eval-builder
+description: Create and maintain AgentV YAML evaluation files for testing AI agent performance. Use this skill when creating new eval files, adding eval cases, or configuring evaluators.
+---
+
+# AgentV Eval Builder
+
+Comprehensive docs: https://agentv.dev
+
+## Quick Start
+
+```yaml
+description: Example eval
+execution:
+  target: default
+
+evalcases:
+  - id: greeting
+    expected_outcome: Friendly greeting
+    input: "Say hello"
+    expected_output: "Hello! How can I help you?"
+    rubrics:
+      - Greeting is friendly and warm
+      - Offers to help
+```
+
+## Eval File Structure
+
+**Required:** `evalcases` (array)
+**Optional:** `description`, `execution`, `dataset`
+
+**Eval case fields:**
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `id` | yes | Unique identifier |
+| `expected_outcome` | yes | What the response should accomplish |
+| `input` / `input_messages` | yes | Input to the agent |
+| `expected_output` / `expected_messages` | no | Gold-standard reference answer |
+| `rubrics` | no | Inline evaluation criteria |
+| `execution` | no | Per-case execution overrides |
+| `conversation_id` | no | Thread grouping |
+
+**Shorthand aliases:**
+- `input` (string) expands to `[{role: "user", content: "..."}]`
+- `expected_output` (string/object) expands to `[{role: "assistant", content: ...}]`
+- Canonical `input_messages` / `expected_messages` take precedence when both present
+
+**Message format:** `{role, content}` where role is `system`, `user`, `assistant`, or `tool`
+**Content types:** inline text, `{type: "file", value: "./path.md"}`
+**File paths:** relative from eval file dir, or absolute with `/` prefix from repo root
+
+**JSONL format:** One eval case per line as JSON. Optional `.yaml` sidecar for shared defaults. See `examples/features/basic-jsonl/`.
+
+## Evaluator Types
+
+Configure via `execution.evaluators` array. Multiple evaluators produce a weighted average score.
+
+### code_judge
+```yaml
+- name: format_check
+  type: code_judge
+  script: uv run validate.py
+  cwd: ./scripts          # optional working directory
+  target: {}              # optional: enable LLM target proxy (max_calls: 50)
+```
+Contract: stdin JSON -> stdout JSON `{score, hits, misses, reasoning}`
+See `references/custom-evaluators.md` for templates.
+
+### llm_judge
+```yaml
+- name: quality
+  type: llm_judge
+  prompt: ./prompts/eval.md     # markdown template or script config
+  model: gpt-5-chat            # optional model override
+  config:                       # passed to script templates as context.config
+    strictness: high
+```
+Variables: `{{question}}`, `{{expected_outcome}}`, `{{candidate_answer}}`, `{{reference_answer}}`, `{{input_messages}}`, `{{expected_messages}}`, `{{output_messages}}`
+- Markdown templates: use `{{variable}}` syntax
+- TypeScript templates: use `definePromptTemplate(fn)` from `@agentv/eval`, receives context object with all variables + `config`
+
+### composite
+```yaml
+- name: gate
+  type: composite
+  evaluators:
+    - name: safety
+      type: llm_judge
+      prompt: ./safety.md
+    - name: quality
+      type: llm_judge
+  aggregator:
+    type: weighted_average
+    weights: { safety: 0.3, quality: 0.7 }
+```
+Aggregator types: `weighted_average`, `all_or_nothing`, `minimum`, `maximum`, `safety_gate`
+- `safety_gate`: fails immediately if the named gate evaluator scores below threshold (default 1.0)
+
+### tool_trajectory
+```yaml
+- name: tool_check
+  type: tool_trajectory
+  mode: any_order            # any_order | in_order | exact
+  minimums:                  # for any_order
+    knowledgeSearch: 2
+  expected:                  # for in_order/exact
+    - tool: knowledgeSearch
+      args: { query: "search term" }   # partial deep equality match
+    - tool: documentRetrieve
+      args: any                        # any arguments accepted
+      max_duration_ms: 5000            # per-tool latency assertion
+    - tool: summarize                  # omit args to skip argument checking
+```
+
+### field_accuracy
+```yaml
+- name: fields
+  type: field_accuracy
+  match_type: exact          # exact | date | numeric_tolerance
+  numeric_tolerance: 0.01    # for numeric_tolerance match_type
+  aggregation: weighted_average  # weighted_average | all_or_nothing
+```
+Compares `output_messages` fields against `expected_messages` fields.
+
+### latency
+```yaml
+- name: speed
+  type: latency
+  max_ms: 5000
+```
+
+### cost
+```yaml
+- name: budget
+  type: cost
+  max_usd: 0.10
+```
+
+### token_usage
+```yaml
+- name: tokens
+  type: token_usage
+  max_total_tokens: 4000
+```
+
+### rubric (inline)
+```yaml
+rubrics:
+  - Simple string criterion
+  - id: weighted
+    expected_outcome: Detailed criterion
+    weight: 2.0
+    required: true
+```
+See `references/rubric-evaluator.md` for score-range mode and scoring formula.
+
+## CLI Commands
+
+```bash
+# Run evaluation
+bun agentv eval <file.yaml> [--eval-id <id>] [--target <name>] [--dry-run]
+
+# Validate eval file
+bun agentv validate <file.yaml>
+
+# Compare results between runs
+bun agentv compare <results1.jsonl> <results2.jsonl>
+
+# Generate rubrics from expected_outcome
+bun agentv generate rubrics <file.yaml> [--target <name>]
+```
+
+## Schemas
+
+- Eval file: `references/eval-schema.json`
+- Config: `references/config-schema.json`