agentv 2.19.0 → 3.1.0

package/README.md

@@ -2,7 +2,7 @@
 
 **CLI-first AI agent evaluation. No server. No signup. No overhead.**
 
-AgentV evaluates your agents locally with multi-objective scoring (correctness, latency, cost, safety) from YAML specifications. Deterministic code judges + customizable LLM judges, all version-controlled in Git.
+AgentV evaluates your agents locally with multi-objective scoring (correctness, latency, cost, safety) from YAML specifications. Deterministic code graders + customizable LLM graders, all version-controlled in Git.
 
 ## Installation
 
@@ -58,9 +58,9 @@ tests:
 
     expected_output: "42"
 
-    assert:
+    assertions:
       - name: math_check
-        type: code-judge
+        type: code-grader
         command: ./validators/check_math.py
 ```
 
@@ -90,7 +90,7 @@ Learn more in the [examples/](examples/README.md) directory. For a detailed comp
 ## Features
 
 - **Multi-objective scoring**: Correctness, latency, cost, safety in one run
-- **Multiple evaluator types**: Code validators, LLM judges, custom Python/TypeScript
+- **Multiple evaluator types**: Code validators, LLM graders, custom Python/TypeScript
 - **Built-in targets**: VS Code Copilot, Codex CLI, Pi Coding Agent, Azure OpenAI, local CLI agents
 - **Structured evaluation**: Rubric-based grading with weights and requirements
 - **Batch evaluation**: Run hundreds of test cases in parallel
@@ -145,7 +145,7 @@ bun run release:next major   # start new major prerelease line
 
 ## Core Concepts
 
-**Evaluation files** (`.yaml` or `.jsonl`) define test cases with expected outcomes. **Targets** specify which agent/provider to evaluate. **Judges** (code or LLM) score results. **Results** are written as JSONL/YAML for analysis and comparison.
+**Evaluation files** (`.yaml` or `.jsonl`) define test cases with expected outcomes. **Targets** specify which agent/provider to evaluate. **Graders** (code or LLM) score results. **Results** are written as JSONL/YAML for analysis and comparison.
 
 ### JSONL Format Support
 
@@ -161,11 +161,11 @@ Optional sidecar YAML metadata file (`dataset.eval.yaml` alongside `dataset.json
 description: Math evaluation dataset
 dataset: math-tests
 execution:
-  target: azure-base
-assert:
+  target: azure-llm
+assertions:
   - name: correctness
-    type: llm-judge
-    prompt: ./judges/correctness.md
+    type: llm-grader
+    prompt: ./graders/correctness.md
 ```
 
 Benefits: Streaming-friendly, Git-friendly diffs, programmatic generation, industry standard (DeepEval, LangWatch, Hugging Face).
@@ -182,7 +182,7 @@ agentv validate evals/my-eval.yaml
 agentv eval evals/my-eval.yaml
 
 # Override target
-agentv eval --target azure-base evals/**/*.yaml
+agentv eval --target azure-llm evals/**/*.yaml
 
 # Run specific test
 agentv eval --test-id case-123 evals/my-eval.yaml
@@ -193,6 +193,32 @@ agentv eval --dry-run evals/my-eval.yaml
 
 See `agentv eval --help` for all options: workers, timeouts, output formats, trace dumping, and more.
 
+#### Output Formats
+
+Write results to different formats using the `-o` flag (format auto-detected from extension):
+
+```bash
+# JSONL (default streaming format)
+agentv eval evals/my-eval.yaml -o results.jsonl
+
+# Self-contained HTML dashboard (opens in any browser, no server needed)
+agentv eval evals/my-eval.yaml -o report.html
+
+# Multiple formats simultaneously
+agentv eval evals/my-eval.yaml -o results.jsonl -o report.html
+
+# JUnit XML for CI/CD integration
+agentv eval evals/my-eval.yaml -o results.xml
+```
+
+The HTML report auto-refreshes every 2 seconds during a live run, then locks once the run completes.
+
+You can also convert an existing JSONL results file to HTML after the fact:
+
+```bash
+agentv convert results.jsonl -o report.html
+```
+
 #### Timeouts
 
 AgentV does not apply a default top-level evaluation timeout. If you want one, set it explicitly
@@ -204,7 +230,7 @@ agent or tool call may still time out even when AgentV's own top-level timeout i
 
 ### Create Custom Evaluators
 
-Write code judges in Python or TypeScript:
+Write code graders in Python or TypeScript:
 
 ```python
 # validators/check_answer.py
@@ -233,9 +259,9 @@ print(json.dumps({
 Reference evaluators in your eval file:
 
 ```yaml
-assert:
+assertions:
   - name: my_validator
-    type: code-judge
+    type: code-grader
     command: ./validators/check_answer.py
 ```
 
@@ -263,7 +289,7 @@ export default defineAssertion(({ answer }) => {
 Files in `.agentv/assertions/` are auto-discovered by filename — use directly in YAML:
 
 ```yaml
-assert:
+assertions:
   - type: word-count    # matches word-count.ts
   - type: contains
     value: "Hello"
@@ -355,7 +381,7 @@ Define execution targets in `.agentv/targets.yaml` to decouple evals from provid
 
 ```yaml
 targets:
-  - name: azure-base
+  - name: azure-llm
     provider: azure
     endpoint: ${{ AZURE_OPENAI_ENDPOINT }}
     api_key: ${{ AZURE_OPENAI_API_KEY }}
@@ -363,12 +389,12 @@ targets:
 
   - name: vscode_dev
     provider: vscode
-    judge_target: azure-base
+    grader_target: azure-llm
 
   - name: local_agent
     provider: cli
     command: 'python agent.py --prompt-file {PROMPT_FILE} --output {OUTPUT_FILE}'
-    judge_target: azure-base
+    grader_target: azure-llm
 ```
 
 Supports: `azure`, `anthropic`, `gemini`, `codex`, `copilot`, `pi-coding-agent`, `claude`, `vscode`, `vscode-insiders`, `cli`, and `mock`.
@@ -379,7 +405,7 @@ Use `${{ VARIABLE_NAME }}` syntax to reference your `.env` file. See `.agentv/ta
 
 ## Evaluation Features
 
-### Code Judges
+### Code Graders
 
 Write validators in any language (Python, TypeScript, Node, etc.):
 
@@ -390,11 +416,11 @@ Write validators in any language (Python, TypeScript, Node, etc.):
 
 For complete examples and patterns, see:
 - [custom-evaluators](https://agentv.dev/evaluators/custom-evaluators/)
-- [code-judge-sdk example](examples/features/code-judge-sdk)
+- [code-grader-sdk example](examples/features/code-grader-sdk)
 
 ### Deterministic Assertions
 
-Built-in assertion types for common text-matching patterns — no LLM judge or code_judge needed:
+Built-in assertion types for common text-matching patterns — no LLM grader or code_grader needed:
 
 | Type | Value | Behavior |
 |------|-------|----------|
@@ -413,7 +439,7 @@ Built-in assertion types for common text-matching patterns — no LLM judge or c
 All assertions support `weight`, `required`, and `negate` flags. Use `negate: true` to invert (no `not_` prefix needed).
 
 ```yaml
-assert:
+assertions:
   # Case-insensitive matching for natural language variation
   - type: icontains-any
     value: ["missing rule code", "need rule code", "provide rule code"]
@@ -431,19 +457,19 @@ assert:
 
 See the [assert-extended example](examples/features/assert-extended) for complete patterns.
 
-### Target Configuration: `judge_target`
+### Target Configuration: `grader_target`
 
-Agent provider targets (`codex`, `copilot`, `claude`, `vscode`) **must** specify `judge_target` when using `llm_judge` or `rubrics` evaluators. Without it, AgentV errors at startup — agent providers can't return structured JSON for judging.
+Agent provider targets (`codex`, `copilot`, `claude`, `vscode`) **must** specify `grader_target` (also accepts `judge_target` for backward compatibility) when using `llm_grader` or `rubrics` evaluators. Without it, AgentV errors at startup — agent providers cannot return structured JSON for grading.
 
 ```yaml
 targets:
-  # Agent target — requires judge_target for LLM-based evaluation
+  # Agent target — requires grader_target for LLM-based evaluation
   - name: codex_local
     provider: codex
-    judge_target: azure-base  # Required: LLM provider for judging
+    grader_target: azure-llm  # Required: LLM provider for grading
 
-  # LLM target — no judge_target needed (judges itself)
-  - name: azure-base
+  # LLM target — no grader_target needed (grades itself)
+  - name: azure-llm
     provider: azure
 ```
 
@@ -452,21 +478,21 @@ targets:
 When agents respond via tool calls instead of text, use `tool_trajectory` instead of text assertions:
 
 - **Agent takes workspace actions** (creates files, runs commands) → `tool_trajectory` evaluator
-- **Agent responds in text** (answers questions, asks for info) → `contains`/`icontains_any`/`llm_judge`
+- **Agent responds in text** (answers questions, asks for info) → `contains`/`icontains_any`/`llm_grader`
 - **Agent does both** → `composite` evaluator combining both
 
-### LLM Judges
+### LLM Graders
 
-Create markdown judge files with evaluation criteria and scoring guidelines:
+Create markdown grader files with evaluation criteria and scoring guidelines:
 
 ```yaml
-assert:
+assertions:
   - name: semantic_check
-    type: llm-judge
-    prompt: ./judges/correctness.md
+    type: llm-grader
+    prompt: ./graders/correctness.md
 ```
 
-Your judge prompt file defines criteria and scoring guidelines.
+Your grader prompt file defines criteria and scoring guidelines.
 
 ### Rubric-Based Evaluation
 
@@ -479,7 +505,7 @@ tests:
 
     input: Explain quicksort algorithm
 
-    assert:
+    assertions:
       - type: rubrics
         criteria:
           - Mentions divide-and-conquer approach
@@ -504,7 +530,7 @@ Configure automatic retry with exponential backoff:
 
 ```yaml
 targets:
-  - name: azure-base
+  - name: azure-llm
     provider: azure
     max_retries: 5
     retry_initial_delay_ms: 2000

package/dist/templates/.agentv/config.yaml

@@ -8,6 +8,11 @@ guideline_patterns:
   - "**/*.prompt.md"
   - "**/SKILL.md"
   
+# Execution defaults (overridden by CLI flags)
+# execution:
+#   pool_workspaces: true    # Reuse materialized workspaces across eval runs
+#   pool_slots: 10           # Max pool slots on disk (1-50, default: 10)
+
 # Notes:
 # - Patterns use standard glob syntax (via micromatch library)
 # - Paths are normalized to forward slashes for cross-platform compatibility

package/dist/templates/.agentv/targets.yaml

@@ -12,11 +12,11 @@ targets:
 
   - name: vscode
     provider: vscode
-    judge_target: azure_base
+    judge_target: azure-base
 
   - name: codex
     provider: codex
-    judge_target: azure_base
+    judge_target: azure-base
     # Uses the Codex CLI (defaults to `codex` on PATH)
     # executable: ${{ CODEX_CLI_PATH }}        # Optional: override executable path
     # args:                             # Optional additional CLI arguments
@@ -34,7 +34,7 @@ targets:
   # Claude - Anthropic's Claude Agent SDK
   - name: claude
     provider: claude
-    judge_target: azure_base
+    judge_target: azure-base
     # Uses the @anthropic-ai/claude-agent-sdk
     # model: claude-sonnet-4-20250514          # Optional: override model
     timeout_seconds: 180
@@ -49,15 +49,15 @@ targets:
     provider: vscode
     workspace_template: ${{ PROJECTX_WORKSPACE_PATH }}
     provider_batching: false
-    judge_target: azure_base
+    judge_target: azure-base
 
   - name: vscode_insiders_projectx
     provider: vscode-insiders
     workspace_template: ${{ PROJECTX_WORKSPACE_PATH }}
     provider_batching: false
-    judge_target: azure_base
+    judge_target: azure-base
 
-  - name: azure_base
+  - name: azure-base
     provider: azure
     endpoint: ${{ AZURE_OPENAI_ENDPOINT }}
     api_key: ${{ AZURE_OPENAI_API_KEY }}
@@ -71,7 +71,7 @@ targets:
 
   - name: local_cli
     provider: cli
-    judge_target: azure_base
+    judge_target: azure-base
     # Passes the fully rendered prompt and any attached files to a local Python script
     # NOTE: Do not add quotes around {PROMPT} or {FILES} - they are already shell-escaped
     command: uv run ./mock_cli.py --prompt {PROMPT} {FILES} --output {OUTPUT_FILE}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentv",
-  "version": "2.19.0",
+  "version": "3.1.0",
   "description": "CLI entry point for AgentV",
   "type": "module",
   "repository": {
@@ -43,7 +43,7 @@
     "yaml": "^2.6.1"
   },
   "devDependencies": {
-    "@agentv/core": "2.12.0",
+    "@agentv/core": "2.19.0",
     "@types/semver": "^7.7.1",
     "execa": "^9.3.0"
   }

package/dist/chunk-3L2L5GIL.js

@@ -1,51 +0,0 @@
-import { createRequire } from 'node:module'; const require = createRequire(import.meta.url);
-import {
-  __commonJS
-} from "./chunk-5H446C7X.js";
-
-// ../../node_modules/.bun/@vercel+oidc@3.0.5/node_modules/@vercel/oidc/dist/token-error.js
-var require_token_error = __commonJS({
-  "../../node_modules/.bun/@vercel+oidc@3.0.5/node_modules/@vercel/oidc/dist/token-error.js"(exports, module) {
-    "use strict";
-    var __defProp = Object.defineProperty;
-    var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
-    var __getOwnPropNames = Object.getOwnPropertyNames;
-    var __hasOwnProp = Object.prototype.hasOwnProperty;
-    var __export = (target, all) => {
-      for (var name in all)
-        __defProp(target, name, { get: all[name], enumerable: true });
-    };
-    var __copyProps = (to, from, except, desc) => {
-      if (from && typeof from === "object" || typeof from === "function") {
-        for (let key of __getOwnPropNames(from))
-          if (!__hasOwnProp.call(to, key) && key !== except)
-            __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
-      }
-      return to;
-    };
-    var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
-    var token_error_exports = {};
-    __export(token_error_exports, {
-      VercelOidcTokenError: () => VercelOidcTokenError
-    });
-    module.exports = __toCommonJS(token_error_exports);
-    var VercelOidcTokenError = class extends Error {
-      constructor(message, cause) {
-        super(message);
-        this.name = "VercelOidcTokenError";
-        this.cause = cause;
-      }
-      toString() {
-        if (this.cause) {
-          return `${this.name}: ${this.message}: ${this.cause}`;
-        }
-        return `${this.name}: ${this.message}`;
-      }
-    };
-  }
-});
-
-export {
-  require_token_error
-};
-//# sourceMappingURL=chunk-3L2L5GIL.js.map

package/dist/chunk-3L2L5GIL.js.map

@@ -1 +0,0 @@
-{"version":3,"sources":["../../../node_modules/.bun/@vercel+oidc@3.0.5/node_modules/@vercel/oidc/dist/token-error.js"],"sourcesContent":["\"use strict\";\nvar __defProp = Object.defineProperty;\nvar __getOwnPropDesc = Object.getOwnPropertyDescriptor;\nvar __getOwnPropNames = Object.getOwnPropertyNames;\nvar __hasOwnProp = Object.prototype.hasOwnProperty;\nvar __export = (target, all) => {\n  for (var name in all)\n    __defProp(target, name, { get: all[name], enumerable: true });\n};\nvar __copyProps = (to, from, except, desc) => {\n  if (from && typeof from === \"object\" || typeof from === \"function\") {\n    for (let key of __getOwnPropNames(from))\n      if (!__hasOwnProp.call(to, key) && key !== except)\n        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });\n  }\n  return to;\n};\nvar __toCommonJS = (mod) => __copyProps(__defProp({}, \"__esModule\", { value: true }), mod);\nvar token_error_exports = {};\n__export(token_error_exports, {\n  VercelOidcTokenError: () => VercelOidcTokenError\n});\nmodule.exports = __toCommonJS(token_error_exports);\nclass VercelOidcTokenError extends Error {\n  constructor(message, cause) {\n    super(message);\n    this.name = \"VercelOidcTokenError\";\n    this.cause = cause;\n  }\n  toString() {\n    if (this.cause) {\n      return `${this.name}: ${this.message}: ${this.cause}`;\n    }\n    return `${this.name}: ${this.message}`;\n  }\n}\n// Annotate the CommonJS export names for ESM import in node:\n0 && (module.exports = {\n  VercelOidcTokenError\n});\n"],"mappings":";;;;;;AAAA;AAAA;AAAA;AACA,QAAI,YAAY,OAAO;AACvB,QAAI,mBAAmB,OAAO;AAC9B,QAAI,oBAAoB,OAAO;AAC/B,QAAI,eAAe,OAAO,UAAU;AACpC,QAAI,WAAW,CAAC,QAAQ,QAAQ;AAC9B,eAAS,QAAQ;AACf,kBAAU,QAAQ,MAAM,EAAE,KAAK,IAAI,IAAI,GAAG,YAAY,KAAK,CAAC;AAAA,IAChE;AACA,QAAI,cAAc,CAAC,IAAI,MAAM,QAAQ,SAAS;AAC5C,UAAI,QAAQ,OAAO,SAAS,YAAY,OAAO,SAAS,YAAY;AAClE,iBAAS,OAAO,kBAAkB,IAAI;AACpC,cAAI,CAAC,aAAa,KAAK,IAAI,GAAG,KAAK,QAAQ;AACzC,sBAAU,IAAI,KAAK,EAAE,KAAK,MAAM,KAAK,GAAG,GAAG,YAAY,EAAE,OAAO,iBAAiB,MAAM,GAAG,MAAM,KAAK,WAAW,CAAC;AAAA,MACvH;AACA,aAAO;AAAA,IACT;AACA,QAAI,eAAe,CAAC,QAAQ,YAAY,UAAU,CAAC,GAAG,cAAc,EAAE,OAAO,KAAK,CAAC,GAAG,GAAG;AACzF,QAAI,sBAAsB,CAAC;AAC3B,aAAS,qBAAqB;AAAA,MAC5B,sBAAsB,MAAM;AAAA,IAC9B,CAAC;AACD,WAAO,UAAU,aAAa,mBAAmB;AACjD,QAAM,uBAAN,cAAmC,MAAM;AAAA,MACvC,YAAY,SAAS,OAAO;AAC1B,cAAM,OAAO;AACb,aAAK,OAAO;AACZ,aAAK,QAAQ;AAAA,MACf;AAAA,MACA,WAAW;AACT,YAAI,KAAK,OAAO;AACd,iBAAO,GAAG,KAAK,IAAI,KAAK,KAAK,OAAO,KAAK,KAAK,KAAK;AAAA,QACrD;AACA,eAAO,GAAG,KAAK,IAAI,KAAK,KAAK,OAAO;AAAA,MACtC;AAAA,IACF;AAAA;AAAA;","names":[]}