agentv 5.0.0-next.1 → 5.2.0-next.1

package/dist/skills/agentv-bench/references/eval-yaml-spec.md

@@ -9,32 +9,51 @@ The grader agent uses this to evaluate assertions without the CLI.
 
 - `name` (string, optional) — eval name
 - `description` (string, optional) — description
-- `execution` (object, optional) — `target`, `model`, etc.
-- `workspace` (object, optional) — workspace config (template, hooks)
-- `input` (string | object | Message | Message[], optional) — suite-level input prepended to each test. String/block shorthand expands to a user message.
+- `target` or `targets` (string | object | object[], optional) — system under test selection
+- `environment` (object | `file://...`, optional) — AgentV coding-agent testbed recipe
+- `env` (object, optional) — provider/eval environment variable overrides and template inputs
+- `extensions` (array, optional) — lifecycle hooks such as `beforeAll`, `beforeEach`, `afterEach`, `afterAll`
+- `prompts` (array, optional) — Promptfoo-compatible prompt matrix entries
 - `tests` (array, required) — test cases
 
 ### Per-test fields
 
 - `id` (string, required) — unique test identifier
-- `input` (string | object | Message | Message[], required) — task input. String shorthand expands to `[{role: user, content: "..."}]`; object shorthand preserves structured user content when the object has no top-level `role`. Top-level `role` is reserved for message objects.
+- `vars` (object, required when prompts need row data) — prompt-template variables for this row
 - `expected_output` (string | Message[], optional) — passive reference answer. String shorthand expands to `[{role: assistant, content: "..."}]`. It is available to declared graders, but does not add an implicit grader when `assertions` is present.
 - `criteria` (string, optional) — human-readable success criteria
-- `assertions` (array, optional) — grader assertions
+- `assert` (array, optional) — grader assertions
+- `environment` (object | `file://...`, optional) — per-case testbed override
 - `conversation_id` (string, optional) — groups related tests
 - `execution` (object, optional) — per-test execution override
 
+If `assert` already states the grading contract, omit `criteria` instead of
+duplicating the same rubric. Prefer plain assertion strings for semantic checks
+when the default LLM rubric grader can judge them; use multiple named
+`type: llm-rubric` blocks only for custom prompts, custom grader targets, or
+intentional grader panels. Write `expected_output` as a golden/reference answer,
+not as criteria or scoring instructions.
+
+For historical or repo-state evals, materialize the repository through
+`environment.setup.command` and pass the repo/ref as argv inputs. A SHA in
+prompt prose or metadata is context only; it does not give the agent an actual
+checkout. `setup.command` is a non-empty string array. Put the executable at
+`command[0]`, put CLI arguments in the remaining array entries, and use
+`timeout_ms` for setup timeout.
+
 ## 2. Assertion Types and Grading Recipes
 
 ### Default grader contract
 
-When a test has no `assertions`, AgentV uses the default `llm-grader` with the case context,
+When a test has no `assertions`, AgentV uses the default `llm-rubric` with the case context,
 including `criteria` and `expected_output` when present.
 
 When `assertions` is present, the list is explicit: run only the declared
 assertions/graders. `expected_output` remains reference data for graders that consume it,
-such as `llm-grader`, `code-grader`, or `field-accuracy`; it does not trigger an additional
-default `llm-grader`.
+such as `llm-rubric`, `script`, or `field-accuracy`; it does not trigger an additional
+default `llm-rubric`.
+When the declared assertion strings fully express the semantic contract, do not
+also add a duplicate `criteria` block.
 
 For each assertion type: YAML config fields, grading recipe (exact pseudocode for deterministic types), and PASS/FAIL conditions.
 
@@ -154,9 +173,10 @@ Same as contains variants but explicitly case-insensitive.
 
 #### `tool-trajectory`
 
-- **Fields:** `expected` (array of expected tool calls), `mode` (string: `exact` | `contains` | `order`)
-- **Recipe:** Inspect transcript for tool call sequence. Match against expected based on mode.
+- **Fields:** `expected` (array of expected tool calls), `mode` (string: `any_order` | `in_order` | `exact` | `subset` | `superset`)
+- **Recipe:** Inspect AgentV-normalized transcript/tool-call data. Match against expected based on mode.
 - **PASS:** tool calls match expected pattern per mode.
+- **Boundary:** This is an AgentV extension. Promptfoo `trajectory:*`, `tool-call-f1`, `skill-used`, and `trace-*` assertion names are rejected until implemented directly.
 
 #### `skill-trigger`
 
@@ -166,10 +186,10 @@ Same as contains variants but explicitly case-insensitive.
 
 ### LLM-judged assertions (require Claude reasoning)
 
-#### `llm-grader`
+#### `llm-rubric`
 
-- **Fields:** `prompt` (string, required — either inline text or path to .md file)
-- **Recipe:** Read the prompt. Evaluate the response against the criteria using your own reasoning. Produce score (0.0-1.0) with evidence.
+- **Fields:** `value` (free-form or structured criteria), `rubrics` (itemized criteria), or `prompt` / exported `prompt_content` (custom grading prompt)
+- **Recipe:** Read the rubric value, rubric items, or prompt. Evaluate the response against those criteria using your own reasoning. Produce score (0.0-1.0) with evidence.
 - **PASS:** score >= 0.5 (configurable via `threshold`).
 
 #### `rubric` / `rubrics`
@@ -180,27 +200,28 @@ Same as contains variants but explicitly case-insensitive.
 
 ### Script-based assertions
 
-#### `code-grader`
+#### `script-grader`
 
 - **Fields:** `path` (string, required — path to script), `command` (string[], optional — custom command)
-- **Script SDK:** Use `defineCodeGrader` from `@agentv/sdk`:
+- **Script SDK:** Use `defineScriptGrader` from `@agentv/sdk`:
   ```typescript
-  import { defineCodeGrader } from '@agentv/sdk';
-  export default defineCodeGrader(({ output, trace }) => ({
+  import { defineScriptGrader } from '@agentv/sdk';
+  export default defineScriptGrader(({ output, trace }) => ({
     score: (output ?? '').includes('expected') ? 1 : 0,
-    assertions: [{ text: 'Contains expected', passed: (output ?? '').includes('expected') }],
+    assert: [{ text: 'Contains expected', passed: (output ?? '').includes('expected') }],
   }));
   ```
 - **Recipe:** The CLI runs the script, passing canonical JSON on stdin (`{output, input, expected_output, ...}`). Script returns `{"score": N, "assertions": [...]}`
 - **PASS:** score >= 0.5 (or as configured).
 
-### Composite assertion
+### Assertion groups
 
-#### `composite`
+#### `assert-set`
 
-- **Fields:** `assertions` (array of sub-assertions), `aggregation` (string: `weighted_average` | `min` | `max` | `all_pass`)
-- **Recipe:** Evaluate each sub-assertion. Aggregate scores per aggregation mode.
-- **PASS:** depends on aggregation mode.
+- **Fields:** `assert` (array of child assertions), `threshold` (number, optional), `config` (object, optional), child `weight` fields.
+- **Recipe:** Evaluate each child assertion and compute a weighted average.
+- **PASS:** without `threshold`, every nonzero-weight child must pass. With `threshold`, the weighted score must meet the threshold.
+- **Config:** parent `config` is inherited by children. Child `config` keys override parent keys.
 
 ## 3. Negate Support
 
@@ -254,7 +275,7 @@ Each line in the results JSONL file is an `EvaluationResult` object. In JSONL, f
 - `assertions` (array of `{text, passed, evidence?}`)
 - `weight` (number, optional)
 - `verdict` (string: `pass` | `fail` | `skip`)
-- `details` (object, optional — structured data from code graders)
+- `details` (object, optional — structured data from script graders)
 - `reasoning` (string, optional)
 
 ## 6. Eval Set Support
@@ -287,8 +308,8 @@ Extracts inputs, target commands, and grader configs from an eval YAML file.
 │   ├── invoke.json             ← {kind, command?, cwd?, timeout_ms?}
 │   ├── criteria.md             ← human-readable success criteria
 │   ├── expected_output.json    ← (if present)
-│   ├── code_graders/<name>.json   ← {name, command, weight, config?}
-│   └── llm_graders/<name>.json    ← {name, weight, threshold?, prompt_content}
+│   ├── script_graders/<name>.json   ← {name, command, weight, config?}
+│   └── llm_graders/<name>.json    ← {name, type, weight, threshold?, prompt_content, value?, rubrics?}
 ```
 
 **`manifest.json` format:**
@@ -307,15 +328,15 @@ Extracts inputs, target commands, and grader configs from an eval YAML file.
 
 ### `agentv pipeline grade <export-dir>`
 
-Runs code-grader assertions against `response.md` files in each test directory.
+Runs script-grader assertions against `response.md` files in each test directory.
 
 **Prerequisites:** `pipeline input` has been run and `response.md` exists in each test dir.
 
-**Output:** `<test-id>/code_grader_results/<name>.json` for each code grader, containing:
+**Output:** `<test-id>/script_grader_results/<name>.json` for each script grader, containing:
 ```json
 {
   "name": "grader-name",
-  "type": "code-grader",
+  "type": "script-grader",
   "score": 1.0,
   "weight": 1.0,
   "assertions": [{"text": "...", "passed": true}]
@@ -324,7 +345,7 @@ Runs code-grader assertions against `response.md` files in each test directory.
 
 ### `agentv pipeline bench <export-dir>`
 
-Merges code-grader results with LLM grader scores and produces final artifacts.
+Merges script-grader results with LLM grader scores and produces final artifacts.
 
 LLM grader results are read from disk at `<test-id>/llm_grader_results/<name>.json` per test.
 

package/dist/skills/agentv-bench/references/migrating-from-skill-creator.md

@@ -20,7 +20,7 @@ AgentV automatically:
 - Converts `assertions` → LLM-grader graders
 - Resolves `files[]` paths relative to the evals.json directory
 
-If you're using the `agentv-bench` skill, it orchestrates these same AgentV commands. Code graders, grading, and artifact generation remain in AgentV core; the skill just orchestrates and summarizes the existing outputs.
+If you're using the `agentv-bench` skill, it orchestrates these same AgentV commands. script graders, grading, and artifact generation remain in AgentV core; the skill just orchestrates and summarizes the existing outputs.
 
 ## What You Gain
 
@@ -29,7 +29,7 @@ Moving from skill-creator's eval loop to AgentV's lifecycle skill gives you:
 | Capability | skill-creator | AgentV lifecycle skill |
 |-----------|---------------|----------------------|
 | Workspace isolation | ❌ | ✅ Clone repos, run setup/teardown scripts |
-| Code graders | ❌ | ✅ Python/TypeScript grader scripts via `defineCodeGrader()` |
+| script graders | ❌ | ✅ Python/TypeScript grader scripts via `defineScriptGrader()` |
 | Tool trajectory scoring | ❌ | ✅ Evaluate tool call sequences |
 | Multi-provider comparison | with-skill vs without-skill | N-way: Claude, GPT, Copilot, Gemini, custom CLI |
 | Multi-turn evaluation | ❌ | ✅ Conversation tracking with `conversation_id` |
@@ -64,14 +64,14 @@ When evals.json becomes limiting, convert to EVAL.yaml for the full feature set:
 # Convert evals.json to EVAL.yaml
 agentv convert evals.json
 
-# Edit the generated YAML to add workspace config, code graders, etc.
+# Edit the generated YAML to add environment setup, script graders, etc.
 # Then run with the full lifecycle
 agentv eval eval.yaml
 ```
 
 EVAL.yaml unlocks:
 - **Workspace setup/teardown** — clone repos, install dependencies, clean up after tests
-- **Code graders** — write graders in Python or TypeScript, not just LLM prompts
+- **script graders** — write graders in Python or TypeScript, not just LLM prompts
 - **Rubric-based grading** — multi-dimensional scoring with weighted criteria
 - **Retry policies** — automatic retries for flaky tests with configurable backoff
 - **Test groups** — organize tests by category with shared config

package/dist/skills/agentv-bench/references/subagent-pipeline.md

@@ -1,8 +1,9 @@
 # Subagent Pipeline — Running eval.yaml without CLI
 
-This reference documents the detailed procedure for running evaluations in subagent mode
-(`AGENT_EVAL_MODE=subagent`, the default). The orchestrating skill dispatches `executor`
-subagents to perform test cases and `grader` subagents to evaluate outputs.
+This reference documents the detailed procedure for running evaluations in subagent mode.
+Subagent mode is an explicit opt-in fallback; the main `agentv-bench` flow defaults to
+AgentV CLI mode. The orchestrating skill dispatches `executor` subagents to perform test
+cases and `grader` subagents to evaluate outputs.
 
 Read this reference when executing Step 3 (Run and Grade) in subagent mode.
 
@@ -49,12 +50,12 @@ code grading in one step. When `--out` is omitted, the output directory defaults
 # Extract inputs and invoke all CLI targets in parallel:
 agentv pipeline run evals/repro.eval.yaml
 
-# Also run code graders inline (instead of using pipeline grade separately):
+# Also run script graders inline (instead of using pipeline grade separately):
 agentv pipeline run evals/repro.eval.yaml --grader-type code
 ```
 
 By default, `pipeline run` extracts inputs and invokes targets only. Pass `--grader-type code`
-to also run code-graders inline, or use `agentv pipeline grade <run-dir>` as a separate step.
+to also run script-graders inline, or use `agentv pipeline grade <run-dir>` as a separate step.
 
 The run directory is printed to stdout. Then continue to the grading and merge phases
 described in SKILL.md Step 3.
@@ -108,7 +109,7 @@ grading until all responses are present.
 
 ### Step 3 onward: Grade and merge
 
-See SKILL.md Step 3 "Grading" section for the three-phase grading process (code graders →
+See SKILL.md Step 3 "Grading" section for the three-phase grading process (script graders →
 LLM grading → merge and validate).
 
 ## Step-by-Step Fine-Grained Control (CLI targets)
@@ -121,7 +122,7 @@ agentv pipeline input evals/repro.eval.yaml
 
 # Step 2: run_tests.py invokes CLI targets (or use pipeline run instead)
 
-# Step 3: Run code graders
+# Step 3: Run script graders
 agentv pipeline grade <run-dir>
 
 # Step 4: Subagent does LLM grading, writes results to llm_grader_results/<name>.json per test
@@ -135,8 +136,8 @@ agentv results validate <run-dir>
 
 ## LLM Grading JSON Format
 
-The agent reads `llm_graders/<name>.json` for each test, grades the response using the prompt
-content, and produces a scores JSON:
+The agent reads `llm_graders/<name>.json` for each test, grades the response using
+`value`, `rubrics`, or prompt content from the config, and produces a scores JSON:
 
 ```json
 {
@@ -173,9 +174,9 @@ the eval.yaml. The target is recorded in `manifest.json` — one run = one targe
         ├── criteria.md              ← grading criteria
         ├── response.md              ← target/agent output
         ├── timing.json              ← execution timing
-        ├── code_graders/<name>.json     ← grader configs written by `pipeline input`: code-grader scripts AND built-in types (contains, regex, equals, etc.)
-        ├── llm_graders/<name>.json      ← LLM grader configs
-        ├── code_grader_results/<name>.json  ← code grader results
+        ├── script_graders/<name>.json     ← grader configs written by `pipeline input`: script-grader scripts AND built-in types (contains, regex, equals, etc.)
+        ├── llm_graders/<name>.json      ← LLM grader configs with prompt_content, value, or rubrics
+        ├── script_grader_results/<name>.json  ← script grader results
         ├── llm_grader_results/<name>.json   ← LLM grader results (written by grader subagents; one file per grader)
         └── grading.json              ← merged grading (written by `pipeline bench` — do NOT write here directly)
 ```

package/dist/skills/agentv-eval-migrations/SKILL.md

@@ -0,0 +1,33 @@
+---
+name: agentv-eval-migrations
+description: >-
+  Migrate AgentV eval YAML across breaking schema changes, especially workspace
+  contract updates and portable-vs-local runtime binding cleanup.
+---
+
+# AgentV Eval Migrations
+
+Use this skill when updating existing AgentV eval YAML, examples, docs, or
+generated eval authoring guidance after a schema-breaking change.
+
+Before editing, read `references/breaking-changes.md`. For stale evals from
+AgentV v4.42.4, use that reference as the migration map: it lists the
+v4.42.4-era shape, current shape, migration steps, verification commands, and
+compatibility notes for each major breaking authoring change. Then compare the
+eval file against the current portable contract:
+
+- Keep committed eval YAML portable: prompts, cases, assertions, workspace
+  templates, repos, hooks, env checks, Docker preflight/container config, and
+  `workspace.scope`.
+- Do not put machine-local existing workspace directories in eval YAML. Use
+  `--workspace-path` for one-off runs or `.agentv/config.local.yaml` with
+  `execution.workspace_path` for persistent local binding.
+- Use `workspace.scope: suite | attempt` for portable workspace lifetime. Docker
+  config is not a replacement for workspace folder lifetime.
+- Keep wire-format fields in `snake_case` and TypeScript internals in
+  `camelCase`.
+
+After migration, validate with `bun apps/cli/src/cli.ts validate <file>` when
+working in the AgentV repo, or `agentv validate <file>` from an installed CLI.
+Run the repo's parser/schema tests for generated examples and fixtures when the
+change affects shared skill data or examples.