agentv 2.6.0 → 2.7.1-next.3

package/dist/templates/.agents/skills/agentv-chat-to-eval/examples/transcript-markdown.md

@@ -0,0 +1,101 @@
+# Example: Markdown Transcript Input
+
+## Input
+
+```
+User: What are the SOLID principles in software engineering?
+Assistant: The SOLID principles are five design principles for object-oriented programming:
+
+1. **Single Responsibility** — A class should have only one reason to change
+2. **Open/Closed** — Open for extension, closed for modification
+3. **Liskov Substitution** — Subtypes must be substitutable for their base types
+4. **Interface Segregation** — Prefer small, specific interfaces over large ones
+5. **Dependency Inversion** — Depend on abstractions, not concretions
+
+User: Can you give me an example of the Single Responsibility Principle?
+Assistant: Sure! Instead of a class that handles both user authentication and email sending, split them:
+
+Bad: `UserService` handles login, registration, AND sends welcome emails.
+Good: `AuthService` handles login/registration. `EmailService` handles sending emails.
+
+Each class has one job, making them easier to test and maintain.
+
+User: What's the difference between an interface and an abstract class?
+Assistant: Key differences:
+
+- **Interface**: Defines a contract (method signatures only). A class can implement multiple interfaces. No state.
+- **Abstract class**: Can have both abstract and concrete methods. A class can extend only one. Can hold state.
+
+Use interfaces for "can-do" relationships (e.g., `Serializable`). Use abstract classes for "is-a" relationships with shared behavior (e.g., `Animal` base class).
+```
+
+## Output
+
+```yaml
+description: "Software engineering concepts — SOLID principles and OOP"
+
+tests:
+  - id: solid-principles-overview
+    criteria: "List and briefly explain all five SOLID principles"
+    input: "What are the SOLID principles in software engineering?"
+    expected_output: |
+      The SOLID principles are five design principles for object-oriented programming:
+
+      1. **Single Responsibility** — A class should have only one reason to change
+      2. **Open/Closed** — Open for extension, closed for modification
+      3. **Liskov Substitution** — Subtypes must be substitutable for their base types
+      4. **Interface Segregation** — Prefer small, specific interfaces over large ones
+      5. **Dependency Inversion** — Depend on abstractions, not concretions
+    assert:
+      - type: rubrics
+        criteria:
+          - Lists all five SOLID principles by name
+          - Provides a concise explanation for each principle
+          - Explanations are technically accurate
+
+  - id: srp-example
+    criteria: "Provide a concrete example of the Single Responsibility Principle"
+    input:
+      - role: user
+        content: "What are the SOLID principles in software engineering?"
+      - role: assistant
+        content: "The SOLID principles are five design principles..."
+      - role: user
+        content: "Can you give me an example of the Single Responsibility Principle?"
+    expected_output: |
+      Sure! Instead of a class that handles both user authentication and email sending, split them:
+
+      Bad: `UserService` handles login, registration, AND sends welcome emails.
+      Good: `AuthService` handles login/registration. `EmailService` handles sending emails.
+
+      Each class has one job, making them easier to test and maintain.
+    assert:
+      - type: rubrics
+        criteria:
+          - Shows a bad example that violates SRP
+          - Shows a good example that follows SRP
+          - Explains why the separation is beneficial
+
+  - id: interface-vs-abstract-class
+    criteria: "Explain the difference between interfaces and abstract classes"
+    input: "What's the difference between an interface and an abstract class?"
+    expected_output: |
+      Key differences:
+
+      - **Interface**: Defines a contract (method signatures only). A class can implement multiple interfaces. No state.
+      - **Abstract class**: Can have both abstract and concrete methods. A class can extend only one. Can hold state.
+
+      Use interfaces for "can-do" relationships (e.g., `Serializable`). Use abstract classes for "is-a" relationships with shared behavior (e.g., `Animal` base class).
+    assert:
+      - type: rubrics
+        criteria:
+          - Correctly distinguishes interfaces from abstract classes
+          - Mentions multiple inheritance support for interfaces
+          - Provides guidance on when to use each
+
+# Suggested additional evaluators:
+# assert:
+#   - name: quality
+#     type: llm_judge
+#     prompt: ./prompts/quality.md
+```

package/dist/templates/.agents/skills/agentv-eval-builder/SKILL.md

@@ -0,0 +1,433 @@
+---
+name: agentv-eval-builder
+description: Create and maintain AgentV YAML evaluation files for testing AI agent performance. Use this skill when creating new eval files, adding tests, or configuring evaluators.
+---
+
+# AgentV Eval Builder
+
+Comprehensive docs: https://agentv.dev
+
+## Quick Start
+
+```yaml
+description: Example eval
+execution:
+  target: default
+
+tests:
+  - id: greeting
+    criteria: Friendly greeting
+    input: "Say hello"
+    expected_output: "Hello! How can I help you?"
+    assert:
+      - type: rubrics
+        criteria:
+          - Greeting is friendly and warm
+          - Offers to help
+```
+
+## Eval File Structure
+
+**Required:** `tests` (array or string path)
+**Optional:** `name`, `description`, `version`, `author`, `tags`, `license`, `requires`, `execution`, `dataset`, `workspace`, `assert`
+
+**Test fields:**
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `id` | yes | Unique identifier |
+| `criteria` | yes | What the response should accomplish |
+| `input` / `input` | yes | Input to the agent |
+| `expected_output` / `expected_output` | no | Gold-standard reference answer |
+| `assert` | no | Evaluators: assertions, rubrics, judges |
+| `rubrics` | no | **Deprecated** — use `assert: [{type: rubrics, criteria: [...]}]` instead |
+| `execution` | no | Per-case execution overrides |
+| `workspace` | no | Per-case workspace config (overrides suite-level) |
+| `metadata` | no | Arbitrary key-value pairs passed to setup/teardown scripts |
+| `conversation_id` | no | Thread grouping |
+
+**Shorthand aliases:**
+- `input` (string) expands to `[{role: "user", content: "..."}]`
+- `expected_output` (string/object) expands to `[{role: "assistant", content: ...}]`
+- Canonical `input` / `expected_output` take precedence when both present
+
+**Message format:** `{role, content}` where role is `system`, `user`, `assistant`, or `tool`
+**Content types:** inline text, `{type: "file", value: "./path.md"}`
+**File paths:** relative from eval file dir, or absolute with `/` prefix from repo root
+
+**JSONL format:** One test per line as JSON. Optional `.yaml` sidecar for shared defaults. See `examples/features/basic-jsonl/`.
+
+## Metadata
+
+When `name` is present, the suite is parsed as a metadata-bearing eval:
+
+```yaml
+name: export-screening        # required, lowercase/hyphens, max 64 chars
+description: Evaluates export control screening accuracy
+version: "1.0"
+author: acme-compliance
+tags: [compliance, agents]
+license: Apache-2.0
+requires:
+  agentv: ">=0.30.0"
+```
+
+## Tests as String Path
+
+Point `tests` to an external file instead of inlining:
+
+```yaml
+name: my-eval
+description: My evaluation suite
+tests: ./cases.yaml           # relative to eval file dir
+```
+
+The external file can be YAML (array of test objects) or JSONL.
+
+## Assert Field
+
+`assert` defines evaluators at the suite level or per-test level. It is the canonical field for all evaluators (replaces `execution.evaluators`):
+
+```yaml
+# Suite-level (appended to every test)
+assert:
+  - type: is_json
+    required: true
+  - type: contains
+    value: "status"
+
+tests:
+  - id: test-1
+    criteria: Returns JSON
+    input: Get status
+    # Per-test assert (runs before suite-level)
+    assert:
+      - type: equals
+        value: '{"status": "ok"}'
+```
+
+`execution.evaluators` is deprecated. When both `assert` and `execution.evaluators` are present, `assert` takes precedence.
+
+## Required Gates
+
+Any evaluator can be marked `required` to enforce a minimum score:
+
+```yaml
+assert:
+  - type: contains
+    value: "DENIED"
+    required: true          # must score >= 0.8 (default)
+  - type: rubrics
+    required: 0.6           # must score >= 0.6 (custom threshold)
+    criteria:
+      - id: accuracy
+        outcome: Identifies the denied party
+        weight: 5.0
+```
+
+If a required evaluator scores below its threshold, the overall verdict is forced to `fail`.
+
+## Workspace Setup/Teardown
+
+Run scripts before/after each test. Define at suite level or override per case:
+
+```yaml
+workspace:
+  template: ./workspace-templates/my-project
+  setup:
+    script: ["bun", "run", "setup.ts"]
+    timeout_ms: 120000
+  teardown:
+    script: ["bun", "run", "teardown.ts"]
+
+tests:
+  - id: case-1
+    input: Fix the bug
+    criteria: Bug is fixed
+    metadata:
+      repo: sympy/sympy
+      base_commit: "abc123"
+    workspace:
+      setup:
+        script: ["python", "custom-setup.py"]  # overrides suite-level
+```
+
+**Lifecycle:** template copy → setup → git baseline → agent → file changes → teardown → cleanup
+**Merge:** Case-level fields replace suite-level fields.
+**Scripts receive stdin JSON:** `{workspace_path, test_id, eval_run_id, case_input, case_metadata}`
+**Setup failure:** aborts case. **Teardown failure:** non-fatal (warning).
+See https://agentv.dev/targets/configuration/#workspace-setupteardown
+
+## Evaluator Types
+
+Configure via `assert` array. Multiple evaluators produce a weighted average score.
+
+### code_judge
+```yaml
+- name: format_check
+  type: code_judge
+  script: uv run validate.py
+  cwd: ./scripts          # optional working directory
+  target: {}              # optional: enable LLM target proxy (max_calls: 50)
+```
+Contract: stdin JSON -> stdout JSON `{score, hits, misses, reasoning}`
+Input includes: `question`, `criteria`, `answer`, `reference_answer`, `output`, `trace`, `file_changes`, `workspace_path`, `config`
+When `workspace_template` is configured, `workspace_path` is the absolute path to the workspace dir (also available as `AGENTV_WORKSPACE_PATH` env var). Use this for functional grading (e.g., running `npm test` in the workspace).
+See docs at https://agentv.dev/evaluators/code-judges/
+
+### llm_judge
+```yaml
+- name: quality
+  type: llm_judge
+  prompt: ./prompts/eval.md     # markdown template or script config
+  model: gpt-5-chat            # optional model override
+  config:                       # passed to script templates as context.config
+    strictness: high
+```
+Variables: `{{question}}`, `{{criteria}}`, `{{answer}}`, `{{reference_answer}}`, `{{input}}`, `{{expected_output}}`, `{{output}}`, `{{file_changes}}`
+- Markdown templates: use `{{variable}}` syntax
+- TypeScript templates: use `definePromptTemplate(fn)` from `@agentv/eval`, receives context object with all variables + `config`
+
+### composite
+```yaml
+- name: gate
+  type: composite
+  assert:
+    - name: safety
+      type: llm_judge
+      prompt: ./safety.md
+    - name: quality
+      type: llm_judge
+  aggregator:
+    type: weighted_average
+    weights: { safety: 0.3, quality: 0.7 }
+```
+Aggregator types: `weighted_average`, `all_or_nothing`, `minimum`, `maximum`, `safety_gate`
+- `safety_gate`: fails immediately if the named gate evaluator scores below threshold (default 1.0)
+
+### tool_trajectory
+```yaml
+- name: tool_check
+  type: tool_trajectory
+  mode: any_order            # any_order | in_order | exact
+  minimums:                  # for any_order
+    knowledgeSearch: 2
+  expected:                  # for in_order/exact
+    - tool: knowledgeSearch
+      args: { query: "search term" }   # partial deep equality match
+    - tool: documentRetrieve
+      args: any                        # any arguments accepted
+      max_duration_ms: 5000            # per-tool latency assertion
+    - tool: summarize                  # omit args to skip argument checking
+```
+
+### field_accuracy
+```yaml
+- name: fields
+  type: field_accuracy
+  match_type: exact          # exact | date | numeric_tolerance
+  numeric_tolerance: 0.01    # for numeric_tolerance match_type
+  aggregation: weighted_average  # weighted_average | all_or_nothing
+```
+Compares `output` fields against `expected_output` fields.
+
+### latency
+```yaml
+- name: speed
+  type: latency
+  max_ms: 5000
+```
+
+### cost
+```yaml
+- name: budget
+  type: cost
+  max_usd: 0.10
+```
+
+### token_usage
+```yaml
+- name: tokens
+  type: token_usage
+  max_total_tokens: 4000
+```
+
+### execution_metrics
+```yaml
+- name: efficiency
+  type: execution_metrics
+  max_tool_calls: 10        # Maximum tool invocations
+  max_llm_calls: 5          # Maximum LLM calls (assistant messages)
+  max_tokens: 5000          # Maximum total tokens (input + output)
+  max_cost_usd: 0.05        # Maximum cost in USD
+  max_duration_ms: 30000    # Maximum execution duration
+  target_exploration_ratio: 0.6   # Target ratio of read-only tool calls
+  exploration_tolerance: 0.2      # Tolerance for ratio check (default: 0.2)
+```
+Declarative threshold-based checks on execution metrics. Only specified thresholds are checked.
+Score is proportional: `hits / (hits + misses)`. Missing data counts as a miss.
+
+### contains
+```yaml
+- type: contains
+  value: "DENIED"
+  required: true
+```
+Binary check: does output contain the substring? Name auto-generated if omitted.
+
+### regex
+```yaml
+- type: regex
+  value: "\\d{3}-\\d{2}-\\d{4}"
+```
+Binary check: does output match the regex pattern?
+
+### equals
+```yaml
+- type: equals
+  value: "42"
+```
+Binary check: does output exactly equal the value (both trimmed)?
+
+### is_json
+```yaml
+- type: is_json
+  required: true
+```
+Binary check: is the output valid JSON?
+
+### rubrics
+```yaml
+- type: rubrics
+  criteria:
+    - id: accuracy
+      outcome: Correctly identifies the denied party
+      weight: 5.0
+    - id: reasoning
+      outcome: Provides clear reasoning
+      weight: 3.0
+```
+LLM-judged structured evaluation with weighted criteria. Criteria items support `id`, `outcome`, `weight`, and `required` fields.
+
+### rubrics (inline, deprecated)
+Top-level `rubrics:` field is deprecated. Use `type: rubrics` under `assert` instead.
+See `references/rubric-evaluator.md` for score-range mode and scoring formula.
+
+## CLI Commands
+
+```bash
+# Run evaluation (requires API keys)
+agentv eval <file.yaml> [--test-id <id>] [--target <name>] [--dry-run]
+
+# Run with trace file (human-readable JSONL)
+agentv eval <file.yaml> --trace-file traces/eval.jsonl
+
+# Run with OTLP JSON file (importable by OTel backends)
+agentv eval <file.yaml> --otel-file traces/eval.otlp.json
+
+# Agent-orchestrated evals (no API keys needed)
+agentv prompt eval <file.yaml>                                      # orchestration overview
+agentv prompt eval input <file.yaml> --test-id <id>                 # task input JSON (file paths, not embedded content)
+agentv prompt eval judge <file.yaml> --test-id <id> --answer-file f # judge prompts / code judge results
+
+# Validate eval file
+agentv validate <file.yaml>
+
+# Compare results between runs
+agentv compare <results1.jsonl> <results2.jsonl>
+
+# Generate rubrics from criteria
+agentv generate rubrics <file.yaml> [--target <name>]
+```
+
+## Code Judge SDK
+
+Use `@agentv/eval` to build custom evaluators in TypeScript/JavaScript:
+
+### defineAssertion (recommended for custom checks)
+```typescript
+#!/usr/bin/env bun
+import { defineAssertion } from '@agentv/eval';
+
+export default defineAssertion(({ answer, trace }) => ({
+  pass: answer.length > 0 && (trace?.eventCount ?? 0) <= 10,
+  reasoning: 'Checks content exists and is efficient',
+}));
+```
+
+Assertions support both `pass: boolean` and `score: number` (0-1). If only `pass` is given, score is 1 (pass) or 0 (fail).
+
+### defineCodeJudge (full control)
+```typescript
+#!/usr/bin/env bun
+import { defineCodeJudge } from '@agentv/eval';
+
+export default defineCodeJudge(({ trace, answer }) => ({
+  score: trace?.eventCount <= 5 ? 1.0 : 0.5,
+  hits: ['Efficient tool usage'],
+  misses: [],
+}));
+```
+
+Both are used via `type: code_judge` in YAML with `script: bun run judge.ts`.
+
+### Convention-Based Discovery
+
+Place assertion files in `.agentv/assertions/` — they auto-register by filename:
+
+```
+.agentv/assertions/word-count.ts  →  type: word-count
+.agentv/assertions/sentiment.ts   →  type: sentiment
+```
+
+No `script:` needed in YAML — just use `type: <filename>`.
+
+## Programmatic API
+
+Use `evaluate()` from `@agentv/core` to run evals as a library:
+
+```typescript
+import { evaluate } from '@agentv/core';
+
+const { results, summary } = await evaluate({
+  tests: [
+    {
+      id: 'greeting',
+      input: 'Say hello',
+      assert: [{ type: 'contains', value: 'hello' }],
+    },
+  ],
+  target: { provider: 'mock_agent' },
+});
+console.log(`${summary.passed}/${summary.total} passed`);
+```
+
+Supports inline tests (no YAML) or file-based via `specFile`.
+
+## defineConfig
+
+Type-safe project configuration in `agentv.config.ts`:
+
+```typescript
+import { defineConfig } from '@agentv/core';
+
+export default defineConfig({
+  execution: { workers: 5, maxRetries: 2 },
+  output: { format: 'jsonl', dir: './results' },
+  limits: { maxCostUsd: 10.0 },
+});
+```
+
+Auto-discovered from project root. Validated with Zod.
+
+## Scaffold Commands
+
+```bash
+agentv create assertion <name>  # → .agentv/assertions/<name>.ts
+agentv create eval <name>       # → evals/<name>.eval.yaml + .cases.jsonl
+```
+
+## Schemas
+
+- Eval file: `references/eval-schema.json`
+- Config: `references/config-schema.json`

package/dist/templates/.agents/skills/agentv-eval-builder/references/config-schema.json

@@ -0,0 +1,36 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "AgentV Config Schema",
+  "description": "Schema for .agentv/config.yaml configuration files",
+  "type": "object",
+  "properties": {
+    "$schema": {
+      "type": "string",
+      "description": "Schema identifier",
+      "enum": ["agentv-config-v2"]
+    },
+    "guideline_patterns": {
+      "type": "array",
+      "description": "Glob patterns for identifying guideline files (instructions, prompts). Files matching these patterns are treated as guidelines, while non-matching files are treated as regular file content.",
+      "items": {
+        "type": "string",
+        "description": "Glob pattern (e.g., '**/*.instructions.md', '**/prompts/**')"
+      },
+      "examples": [
+        ["**/*.instructions.md", "**/instructions/**", "**/*.prompt.md", "**/prompts/**"],
+        ["**/*.guide.md", "**/guidelines/**", "docs/AGENTS.md"]
+      ]
+    },
+    "eval_patterns": {
+      "type": "array",
+      "description": "Glob patterns for discovering eval files during interactive mode (`agentv eval` with no args). Defaults to ['**/evals/**/dataset*.yaml', '**/evals/**/eval.yaml'] if not specified.",
+      "items": {
+        "type": "string",
+        "description": "Glob pattern (e.g., '**/evals/**/dataset*.yaml', '**/evals/**/eval.yaml')"
+      },
+      "examples": [["**/evals/**/dataset*.yaml", "**/evals/**/eval.yaml"], ["**/evals/**/*.yaml"]]
+    }
+  },
+  "required": ["$schema"],
+  "additionalProperties": false
+}

package/dist/templates/.agents/skills/agentv-eval-builder/references/custom-evaluators.md

@@ -0,0 +1,118 @@
+# Custom Evaluators
+
+## Wire Format
+
+### Input (stdin JSON)
+
+```json
+{
+  "question": "string",
+  "criteria": "string",
+  "reference_answer": "string",
+  "answer": "string",
+  "guideline_files": ["path"],
+  "input_files": ["path"],
+  "input": [{"role": "user", "content": "..."}],
+  "expected_output": [{"role": "assistant", "content": "..."}],
+  "output": [{"role": "assistant", "content": "..."}],
+  "trace": {
+    "event_count": 5,
+    "tool_names": ["fetch"],
+    "tool_calls_by_name": {"fetch": 1},
+    "error_count": 0,
+    "llm_call_count": 2,
+    "token_usage": {"input": 1000, "output": 500},
+    "cost_usd": 0.0015,
+    "duration_ms": 3500,
+    "start_time": "2026-02-13T10:00:00.000Z",
+    "end_time": "2026-02-13T10:00:03.500Z"
+  }
+}
+```
+
+### Output (stdout JSON)
+
+```json
+{
+  "score": 0.85,
+  "hits": ["passed check"],
+  "misses": ["failed check"],
+  "reasoning": "explanation"
+}
+```
+
+`score` (0.0-1.0) required. `hits`, `misses`, `reasoning` optional.
+
+## SDK Functions
+
+```typescript
+import { defineCodeJudge, createTargetClient, definePromptTemplate } from '@agentv/eval';
+```
+
+- `defineCodeJudge(fn)` - Wraps evaluation function with stdin/stdout handling
+- `createTargetClient()` - Returns LLM proxy client (when `target: {}` configured)
+  - `.invoke({question, systemPrompt})` - Single LLM call
+  - `.invokeBatch(requests)` - Batch LLM calls
+- `definePromptTemplate(fn)` - Wraps prompt generation function
+  - Context fields: `question`, `answer`, `referenceAnswer`, `criteria`, `expectedOutput`, `output`, `config`, `trace`
+
+## Python Example
+
+```python
+#!/usr/bin/env python3
+import json, sys
+
+def evaluate(data: dict) -> dict:
+    candidate = data.get("answer", "")
+    hits, misses = [], []
+    for kw in ["async", "await"]:
+        (hits if kw in candidate else misses).append(f"Keyword '{kw}'")
+    return {
+        "score": len(hits) / max(len(hits) + len(misses), 1),
+        "hits": hits, "misses": misses
+    }
+
+if __name__ == "__main__":
+    try:
+        print(json.dumps(evaluate(json.loads(sys.stdin.read()))))
+    except Exception as e:
+        print(json.dumps({"score": 0, "misses": [str(e)]}))
+        sys.exit(1)
+```
+
+## TypeScript Example
+
+```typescript
+#!/usr/bin/env bun
+import { defineCodeJudge } from '@agentv/eval';
+
+export default defineCodeJudge(({ answer, criteria }) => {
+  const hits: string[] = [];
+  const misses: string[] = [];
+  if (answer.includes(criteria)) {
+    hits.push('Matches expected outcome');
+  } else {
+    misses.push('Does not match expected outcome');
+  }
+  return {
+    score: hits.length / Math.max(hits.length + misses.length, 1),
+    hits, misses,
+  };
+});
+```
+
+## Template Variables
+
+Derived from test fields (users never author these directly):
+
+| Variable | Source |
+|----------|--------|
+| `question` | First user message in `input` |
+| `criteria` | Test `criteria` field |
+| `reference_answer` | Last entry in `expected_output` |
+| `answer` | Last entry in `output` (runtime) |
+| `input` | Full resolved input array (JSON) |
+| `expected_output` | Full resolved expected array (JSON) |
+| `output` | Full provider output array (JSON) |
+
+Markdown templates use `{{variable}}` syntax. TypeScript templates receive context object.