agentv 5.0.0-next.1 → 5.2.0-next.1

package/dist/skills/agentv-eval-writer/SKILL.md

@@ -3,7 +3,7 @@ name: agentv-eval-writer
 description: >-
   Write, edit, review, and validate AgentV EVAL.yaml / .eval.yaml evaluation files.
   Use when asked to create new eval files, update or fix existing ones, add or remove test cases,
-  configure graders (`llm-grader`, `code-grader`, `rubrics`), review whether an eval is correct or complete,
+  configure graders (`llm-rubric`, `script`), review whether an eval is correct or complete,
   convert between EVAL.yaml and evals.json using `agentv convert`, or generate eval test cases
   from chat transcripts (markdown conversation or JSON messages).
   Do NOT use for creating SKILL.md files, writing skill definitions, or running evals —
@@ -13,20 +13,36 @@ description: >-
 # AgentV Eval Writer
 
 Comprehensive docs: https://agentv.dev
+Promptfoo parity matrix: https://agentv.dev/docs/reference/promptfoo-parity/
 
 ## Authoring Principle
 
 Treat YAML as the canonical portable model. Prefer authoring `.eval.yaml` / `EVAL.yaml` first, then use TypeScript helpers, Python scripts, or executable graders only when they lower to the same fields or when the evaluation logic must actually run code.
 
-Eval files define what is tested: prompts, datasets, assertions, and task fixtures.
-Experiment files define how those evals run: targets, setup, scripts, timeout,
-sandbox, suite selection, and repeat-run policy. Use `experiments/*.yaml` for
-committed run configurations. In eval YAML, keep `tests[]` as the atomic eval
-definition. In experiment YAML, reference eval suites with `suites[]` and select
-suite-local tests with `select.test_ids[]`.
+Eval files define what is tested and how it runs: prompts, datasets, assertions,
+task fixtures, top-level `target`, and suite run controls. Use field-local file
+refs such as `tests: file://...`, `prompts: file://...`, `default_test:
+file://...`, and `environment: file://...`. String-valued `tests` and string
+entries inside `tests[]` are raw-case refs for direct paths, directories, and
+globs. Run several full eval suites directly with CLI multi-file selection and
+tags. Use scoped `run:` on individual tests only for `threshold`, `repeat`,
+`timeout_seconds`, and legacy `budget_usd`; keep target selection at top-level
+`target` or CLI `--target`, put suite budget caps under
+`evaluate_options.budget_usd`, authored concurrency under
+`evaluate_options.max_concurrency`, suite repeat policy under
+`evaluate_options.repeat`, coding-agent testbed setup under `environment`,
+provider environment overrides under `env`, and lifecycle hooks under
+`extensions`.
 
 Use `@agentv/sdk` for TypeScript helper imports. Do not use `@agentv/eval` for new evals, examples, scaffolds, or skill guidance; it was a deprecated compatibility package and has been removed from this repository.
 
+## Authoring Checklist
+
+- Put grading criteria in `assert`, not in test-level `criteria`. Plain assertion strings become an `llm-rubric` grader.
+- Prefer plain assertion strings for semantic checks when the default rubric grader can judge them. Use `type: llm-rubric` for structured criteria, custom prompts, custom grader targets, or assertion-level transforms, and `type: script` when grading must execute code.
+- Write `expected_output` as a golden/reference answer the target could have produced. Do not write criteria, scoring instructions, or "the agent should..." rubric prose there.
+- For historical or repo-state evals, materialize the repo through a pinned `environment` setup recipe. Mentioning a SHA only in prompt prose is not enough because the agent needs an actual checkout to inspect.
+
 ## Evaluation Types
 
 AgentV evaluations measure **execution quality** — whether your agent or skill produces correct output when invoked.
@@ -45,7 +61,7 @@ agentv convert evals.json
 agentv eval evals.json
 ```
 
-The converter maps `prompt` → `input`, `expected_output` → `expected_output`, `assertions` → `assertions` (`llm-grader`), and resolves `files[]` paths. The generated YAML includes TODO comments for AgentV features to add (workspace setup, code graders, rubrics, required gates).
+The converter maps `prompt` → `input`, `expected_output` → `expected_output`, and Agent Skills `assertions` → AgentV `assert` (`llm-rubric` checks), and resolves `files[]` paths. The generated YAML includes TODO comments for AgentV features to add (environment setup, script graders, rubrics, required gates).
 
 After converting, enhance the YAML with AgentV-specific capabilities shown below.
 
@@ -77,21 +93,19 @@ JSON messages:
 
 **Multi-turn format** (when context from prior turns matters):
 ```yaml
+prompts:
+  - - role: user
+      content: "My name is Alice"
+    - role: assistant
+      content: "Nice to meet you, Alice!"
+    - role: user
+      content: "What's my name?"
+
 tests:
   - id: multi-turn-context
-    criteria: "Agent remembers prior context"
-    input:
-      - role: user
-        content: "My name is Alice"
-      - role: assistant
-        content: "Nice to meet you, Alice!"
-      - role: user
-        content: "What's my name?"
     expected_output: "Your name is Alice."
-    assertions:
-      - type: rubrics
-        criteria:
-          - Correctly recalls the user's name from earlier in the conversation
+    assert:
+      - Correctly recalls the user's name from earlier in the conversation
 ```
 
 **Guidelines:** preserve exact wording in `expected_output`; aim for 5–15 tests per transcript; pick exchanges that test different capabilities.
@@ -100,44 +114,96 @@ tests:
 
 ```yaml
 description: Example eval
-execution:
-  target: default
+target: default
+
+prompts:
+  - "{{ prompt }}"
 
 tests:
   - id: greeting
-    criteria: Friendly greeting
-    input: "Say hello"
+    vars:
+      prompt: "Say hello"
     expected_output: "Hello! How can I help you?"
-    assertions:
-      - type: rubrics
-        criteria:
-          - Greeting is friendly and warm
-          - Offers to help
+    assert:
+      - Greeting is friendly and warm
+      - Offers to help
 ```
 
 ## Eval File Structure
 
-**Required:** `tests` (array or string path)
-**Optional:** `name`, `description`, `version`, `author`, `tags`, `license`, `requires`, `execution`, `suite`, `workspace`, `assertions`, `input`
+**Required:** `tests` (array or string raw-case path) or `scenarios`
+**Optional:** `name`, `description`, `experiment`, `version`, `author`, `tags`, `license`, `requires`, `target`, `targets`, `prompts`, `default_test`, `timeout_seconds`, `evaluate_options`, `threshold`, `suite`, `environment`, `env`, `extensions`, `assert`
 
 **Test fields:**
 
 | Field | Required | Description |
 |-------|----------|-------------|
 | `id` | yes | Unique identifier |
-| `criteria` | yes | What the response should accomplish |
-| `input` | yes | Input to the agent (string/object shorthand or full message array) |
+| `vars` | yes when the prompt needs row data | Prompt-template variables for this row |
 | `expected_output` | no | Gold-standard reference answer (string shorthand or full message array) |
-| `assertions` | no | Graders: deterministic checks, rubrics, and LLM/code graders |
-| `execution` | no | Per-case execution overrides |
-| `workspace` | no | Per-case workspace config (overrides suite-level) |
+| `assert` | yes | Graders: deterministic checks, `llm-rubric` checks, script graders, or plain string rubric criteria |
+| `execution` | no | Per-case grader/default overrides such as `skip_defaults`; target selection belongs in top-level `target` or CLI `--target` |
+| `environment` | no | Per-case coding-agent testbed config (overrides suite-level) |
 | `metadata` | no | Arbitrary key-value pairs passed to setup/teardown scripts |
 | `conversation_id` | no | Thread grouping |
 
+## Prompt Templates and Vars
+
+Use top-level `prompts` plus `tests[].vars` for the Promptfoo-compatible canonical
+input shape. Shared data defaults belong in `default_test.vars`; per-test
+`vars` override those defaults by key. AgentV renders every prompt with each
+test's merged vars, then expands the run across prompts, targets, tests, and
+repeat attempts.
+
+```yaml
+description: Prompt matrix example
+target: default
+
+prompts:
+  - id: support-chat
+    label: Support chat
+    file: ./prompts/support-chat.json
+  - id: terse
+    label: Terse
+    prompt: "Answer for {{ audience }} in one sentence: {{ question }}"
+
+default_test:
+  vars:
+    audience: users
+    category: support
+
+tests:
+  - id: password-reset
+    vars:
+      question: How do I reset my password?
+    expected_output: Password reset guidance
+    assert:
+      - Gives correct password reset guidance
+  - id: admin-access
+    vars:
+      audience: admins
+      question: How do I revoke a user's access?
+    expected_output: Access revocation guidance
+    assert:
+      - Gives safe access revocation guidance
+```
+
+Prompt templates can use `{{ name }}` or `{{ vars.name }}` placeholders. Use
+top-level names when matching Promptfoo-style prompt templates; use
+`{{ vars.name }}` when explicit namespacing is clearer.
+
+Do not author direct input fields in normal eval YAML. `tests[].input` and
+top-level `input` are removed; write simple task text as a prompt template such
+as `"{{ input }}"` or `"{{ vars.input }}"` with
+`tests[].vars.input: "Summarize X"`.
+
+`input_files` is also direct-input convenience sugar. In prompt-template suites,
+model file-backed context as vars containing file paths or `file://` references,
+then render those vars from the prompt template next to the input.
+
 **Shorthand forms:**
-- `input` (string, including YAML block scalars) expands to `[{role: "user", content: "..."}]`
-- `input` (object without a top-level `role`) expands to `[{role: "user", content: {...}}]`
-- top-level `role` is reserved for message objects; use `{role, content}` when you mean a message, or nest payload role data under another key
+- Prompt entries can be strings, message arrays, file references, or prompt objects.
+- Put chat/system/user messages in `prompts`, not in `tests[].input`.
 - `expected_output` (string/object) expands to `[{role: "assistant", content: ...}]`
 - Use these canonical field names on disk; keep the wire format `snake_case`
 
@@ -148,7 +214,38 @@ tests:
 
 **JSONL format:** One test per line as JSON. Optional `.yaml` sidecar for shared defaults. See `examples/features/basic-jsonl/`.
 
-**Environment variables:** All string fields support `${{ VAR }}` interpolation. Missing vars resolve to empty string. Works in eval files, external case files, and workspace configs. `.env` files are loaded automatically.
+**Environment variables:** Use `{{ env.VAR }}` templates in authored config. Missing vars resolve to empty string. Works in eval files, external case files, and environment configs. `.env` files are loaded automatically.
+
+## Output Transforms
+
+Use Promptfoo-compatible `transform` when the target output needs shaping before
+grading. Common cases include converting a `ContentFile` such as an `.xlsx`
+spreadsheet into text before an `llm-rubric` grader runs.
+
+```yaml
+prompts:
+  - "{{ input }}"
+
+default_test:
+  options:
+    transform: file://scripts/transforms/xlsx-to-csv.ts
+
+tests:
+  - id: spreadsheet-output
+    vars:
+      input: Generate the spreadsheet report
+    assert:
+      - Output contains the transformed spreadsheet text including the revenue rows
+```
+
+Transform placement:
+
+- `default_test.options.transform` applies to every test unless overridden.
+- `tests[].options.transform` overrides the inherited default transform for one test.
+- Assertion-level `transform` applies only to that grader, after the test/default transform.
+
+Do not author `preprocessors` or deprecated Promptfoo `postprocess` in current
+eval YAML. Use `transform` at the point that needs the shaped output.
 
 ## Metadata
 
@@ -165,26 +262,30 @@ requires:
   agentv: ">=0.30.0"
 ```
 
-## Suite-level Input
+## Shared Prompt Context
 
-Prepend shared input messages to every test (like suite-level `assertions`). Avoids repeating the same prompt or instruction in each test:
+Put shared prompt instructions in top-level `prompts` and shared data in
+`default_test.vars`:
 
 ```yaml
-input: |
-  Read AGENTS.md before answering.
-  Explain tradeoffs clearly.
+prompts:
+  - - role: system
+      content: |
+        Read AGENTS.md before answering.
+        Explain tradeoffs clearly.
+    - role: user
+      content: "{{ question }}"
 
 tests: ./cases.yaml
 
-# cases.yaml — each test only needs its own query
+# cases.yaml — each raw row supplies vars.question, or a compatibility input row
 # - id: test-1
-#   criteria: ...
-#   input: "User question here"
+#   assert:
+#     - ...
+#   vars:
+#     question: "User question here"
 ```
 
-Effective input: `[...suite input, ...test input]`. Skipped when `execution.skip_defaults: true`.
-Accepts the same formats as test `input`: string/block scalar, structured object without a top-level `role`, single message object, or full message array. Use the full message array only when you need multiple messages or file/image content blocks.
-
 ## Tests as String Path
 
 Point `tests` to an external file instead of inlining:
@@ -197,76 +298,126 @@ tests: ./cases.yaml           # relative to eval file dir
 
 The external file can be YAML (array of test objects) or JSONL.
 
-## Assertions Field
+## Assert Field
 
-`assertions` defines graders at the suite level or per-test level. It is the canonical field for all graders:
+`assert` defines graders at the suite level or per-test level. It is the canonical authored field for all graders:
 
 ```yaml
-# Suite-level (appended to every test)
-assertions:
+# Mix exact checks with rubric shorthand when both matter.
+assert:
   - type: is-json
     required: true
   - type: contains
     value: "status"
+  - Correctly answers the user's question
+  - Explains the reasoning clearly
 
 tests:
   - id: test-1
-    criteria: Returns JSON
     input: Get status
-    # Per-test assertions (runs before suite-level)
-    assertions:
+    assert:
       - type: equals
         value: '{"status": "ok"}'
+      - Explains what the status means
 ```
 
-## How `criteria` and `assertions` Interact
+Plain strings in `assert` are rubric criteria and are the preferred shape for
+qualitative agent behavior. Use deterministic assertions (`contains`, `regex`,
+`is-json`, `equals`) only for exact machine-verifiable outputs, and script graders
+when the check must inspect files, run commands, or validate structured state.
+Do not add a separate test-level `criteria` field. Legacy evals that still use
+`criteria` without explicit `assert` are loaded as a plain-string assertion for
+compatibility, but new evals should author the assertion directly.
 
-`criteria` is a **data field** — it describes what the response should accomplish. It is **not** a grader. How it gets evaluated depends on whether `assertions` is present:
+For repo-state evals, combine a pinned checkout, a golden answer, and assertion
+shorthand:
 
-| Scenario | What happens | Warning? |
-|----------|-------------|----------|
-| `criteria` + **no `assertions`** | Implicit `llm-grader` runs automatically against `criteria` | No |
-| `criteria` + **`assertions` with only deterministic graders** (contains, regex, etc.) | Only declared graders run. `criteria` is **not evaluated**. | Yes — warns that no grader will consume criteria |
-| `criteria` + **`assertions` with a grader** (`llm-grader`, `code-grader`, `rubrics`) | Declared graders run. Graders receive `criteria` as input. | No |
-
-### No assertions → implicit llm-grader
+```yaml
+environment:
+  type: host
+  workdir: ./agentv
+  setup:
+    command:
+      - bash
+      - ./scripts/materialize-repo.sh
+      - ./agentv
+      - https://github.com/EntityProcess/agentv.git
+      - 5e3c8f46d80fe66b1a75659e4fd94e38a7e09215
+    cwd: "."
+
+prompts:
+  - "{{ task }}"
 
-The simplest path. `criteria` is automatically evaluated by the default `llm-grader`:
+tests:
+  - id: verification-learning-capture
+    vars:
+      task: |
+        The eval harness has prepared ./agentv at the commit before the
+        verification guidance was added.
+
+        Decide what durable repo change should be made and explain why.
+    expected_output: |
+      The durable repo change is to update .agents/verification.md with the
+      reusable verification workflow lessons. AGENTS.md already routes this
+      class of work to .agents/verification.md, so no extra AGENTS.md edit is
+      needed unless that routing is missing.
+    assert:
+      - The answer recommends updating .agents/verification.md rather than leaving the learning only in PR comments or private evidence.
+      - The answer uses the pinned ./agentv checkout to verify the AGENTS.md routing.
+      - The answer preserves the historical commit SHA as context.
+```
+
+## Assertions and Reference Data
+
+When `assert` is defined, **only the declared graders run**. For
+semantic checks, add plain rubric strings. If you need a custom LLM prompt or
+grader target, declare `llm-rubric` explicitly:
 
 ```yaml
+prompts:
+  - "{{ prompt }}"
+
 tests:
-  - id: simple-eval
-    criteria: Assistant correctly explains the bug and proposes a fix
-    input: "Debug this function..."
-    # No assertions → default llm-grader evaluates against criteria
+  - id: mixed-eval
+    vars:
+      prompt: "Debug this function..."
+    assert:
+      - Explains why the bug happens
+      - type: contains
+        value: "fix"
 ```
 
-### assertions present → no implicit grader
+`expected_output` is passive reference data. It is available to graders through
+`{{expected_output}}` and the script stdin payload, but it does not create an
+implicit LLM grading call by itself.
 
-When `assertions` is defined, **only the declared graders run**. If you want an LLM grader alongside deterministic checks, declare it explicitly:
+**Common mistake:** putting rubric prose in `expected_output` instead of an
+assertion:
 
 ```yaml
+prompts:
+  - "{{ prompt }}"
+
 tests:
-  - id: mixed-eval
-    criteria: Response is helpful and mentions the fix
-    input: "Debug this function..."
-    assertions:
-      - type: llm-grader       # must be explicit when assertions is present
-      - type: contains
-        value: "fix"
+  - id: bad-example
+    vars:
+      prompt: "What is 2+2?"
+    expected_output: The assistant should explain why the answer is 4. # reference answer field, not a grader
 ```
 
-**Common mistake:** defining `criteria` with only deterministic graders. The criteria will be ignored and a warning is emitted:
+Write this as:
 
 ```yaml
+prompts:
+  - "{{ prompt }}"
+
 tests:
-  - id: bad-example
-    criteria: Gives a thoughtful answer    # ⚠ NOT evaluated — no grader in assertions
-    input: "What is 2+2?"
-    assertions:
-      - type: contains
-        value: "4"
-    # Warning: criteria is defined but no grader in assertions will evaluate it.
+  - id: good-example
+    vars:
+      prompt: "What is 2+2?"
+    expected_output: "4"
+    assert:
+      - The answer is 4 and explains the arithmetic briefly
 ```
 
 ## Required Gates
@@ -274,13 +425,14 @@ tests:
 Any grader can be marked `required` to enforce a minimum score:
 
 ```yaml
-assertions:
+assert:
   - type: contains
     value: "DENIED"
     required: true          # must score >= 0.8 (default)
-  - type: rubrics
-    required: 0.6           # must score >= 0.6 (custom threshold)
-    criteria:
+  - type: llm-rubric
+    required: true
+    min_score: 0.6          # must score >= 0.6 (custom threshold)
+    value:
       - id: accuracy
         outcome: Identifies the denied party
         weight: 5.0
@@ -288,103 +440,106 @@ assertions:
 
 If a required grader scores below its threshold, the overall verdict is forced to `fail`.
 
-## Workspace Setup/Teardown
+## Environment Setup/Teardown
 
 Run scripts before/after each test. Define at suite level or override per case:
 
 ```yaml
-workspace:
-  template: ./workspace-templates/my-project
-  repos:
-    - path: ./repo
-      repo: sympy/sympy
-      base_commit: "abc123"
-  hooks:
-    before_all:
-      command: ["bun", "run", "setup.ts"]
-      timeout_ms: 120000
-    after_each:
-      reset: fast
-    after_all:
-      command: ["bun", "run", "teardown.ts"]
+environment:
+  type: host
+  workdir: ./repo
+  setup:
+    command: ["bun", "run", "setup.ts", "sympy/sympy", "abc123"]
+    cwd: "."
+    timeout_ms: 120000
+extensions:
+  - file://scripts/teardown.mjs:afterAll
 
 tests:
   - id: case-1
     input: Fix the bug
-    criteria: Bug is fixed
     metadata:
       source_repo: sympy/sympy
       source_commit: "abc123"
 ```
 
-**Lifecycle:** template copy → repo materialization → workspace before_all → target before_all → git baseline → before_each hooks → agent → file changes → after_each hooks → after_all hooks → cleanup
-**Merge:** Case-level fields replace suite-level fields.
-**Commands receive stdin JSON:** `{workspace_path, test_id, eval_run_id, case_input, case_metadata}`
+**Lifecycle:** environment setup → lifecycle extensions → target setup → agent → grading → teardown extensions → cleanup
+**Merge:** Case-level environment fields replace suite-level fields.
+**Setup commands:** `setup.command` is an argv command. `command[0]` is the executable and `command[1...]` are CLI arguments. Use `["bash", "-lc", "..."]` when shell behavior is required.
+**Setup stdin:** AgentV sends environment metadata JSON on stdin for setup commands.
 **Setup failure:** aborts case. **Teardown failure:** non-fatal (warning).
-For SWE-bench-style evals, keep operational checkout state under `workspace.repos[].base_commit`; treat `metadata.source_commit` as informational only.
+For SWE-bench-style evals, put operational checkout state under
+`environment.setup.command` CLI arguments; treat `metadata.source_commit` as informational only.
+A SHA in the prompt or metadata without a matching environment setup recipe is
+not an operational checkout.
 
-### Repository Lifecycle
+### Environment Lifecycle
 
-Materialize repos into the eval workspace automatically. Repo entries declare identity and checkout pins only; AgentV resolves acquisition from registered projects, `git_cache.mirrors`, its mirror cache, then remote clone. `git_cache.mirrors` may be defined in `$AGENTV_HOME/config.yaml`, the project's committed `.agentv/config.yaml`, or a gitignored `.agentv/config.override.yaml` (highest precedence) — use the override for machine-specific local clone paths without editing tracked or user-global config. For shared repo workspaces, pooling is the default:
+Describe coding-agent testbeds with `environment`. Reusable recipes should live
+in field-local files and be loaded with `environment: file://...`:
 
 ```yaml
-workspace:
-  repos:
-    - path: ./repo
-      repo: https://github.com/org/repo.git
-      commit: main
-      ancestor: 1       # parent commit
-  hooks:
-    after_each:
-      reset: fast          # none | fast | strict
-  isolation: shared        # shared | per_test
-  mode: pooled             # pooled | temp | static
+environment: file://.agentv/environments/repo.yaml
 ```
 
-- `repo`: full clone URL or GitHub `org/name` shorthand
-- `commit`: branch, tag, or SHA to check out
-- `base_commit`: alias for `commit` for SWE-bench-style datasets
-- `ancestor`: walk N commits back from the checked-out ref
-- `sparse`: sparse checkout paths array
-- Do not use legacy `source`, `type`, `checkout`, `resolve`, or `clone` fields under `workspace.repos[]`
-- `mode`: `pooled` (default for shared repos), `temp`, or `static`
-- `path`: workspace path used when `mode: static`; when empty/missing the workspace is auto-materialised (template copied + repos cloned); populated dirs are reused as-is
-- `hooks.enabled`: boolean (default `true`); set `false` to skip all lifecycle hooks
-- Pool reset defaults to `fast` (`git clean -fd`); use `--workspace-clean full` for strict reset (`git clean -fdx`)
-- Pool entries are managed separately via `agentv workspace list` and `agentv workspace clean`
-- `agentv workspace deps <eval-paths>` scans eval files and outputs a JSON manifest of required git repos (useful for CI pre-cloning)
+```yaml
+# .agentv/environments/repo.yaml
+type: host
+workdir: ./repo
+setup:
+  command: ["bash", "./scripts/materialize-repo.sh", "https://github.com/org/repo.git", "main", "1"]
+  cwd: "."
+  timeout_ms: 120000
+```
 
-See https://agentv.dev/targets/configuration/#repository-lifecycle
+```yaml
+# .agentv/environments/docker-repo.yaml
+type: docker
+context: ./environment
+dockerfile: Dockerfile
+workdir: /app
+setup:
+  command: ["bash", "-lc", "bun install && bun run build"]
+  cwd: "."
+  timeout_ms: 120000
+```
+
+- `type`: `host` or `docker`
+- `workdir`: path the target and graders should use
+- `setup`: argv command, optional `cwd`, and optional `timeout_ms` for repository/testbed materialization
+- Top-level `env`: provider/eval environment overrides
+- `environment.env`: recipe-scoped process environment, distinct from top-level `env`
+- `extensions`: lifecycle hooks such as `beforeAll`, `beforeEach`, `afterEach`, and `afterAll`
 
 ## Grader Types
 
-Configure via `assertions` array. Multiple graders produce a weighted average score.
+Configure via the `assert` array. Multiple graders produce a weighted average score.
 
-### code-grader
+### script
 ```yaml
 - name: format_check
-  type: code-grader
+  type: script
   command: [uv, run, validate.py]
   cwd: ./scripts          # optional working directory
   target: {}              # optional: enable LLM target proxy (max_calls: 50)
 ```
 Contract: stdin JSON -> stdout JSON `{score, assertions: [{text, passed, evidence?}], reasoning}`
-Raw stdin uses snake_case and includes: `criteria`, `input`, `expected_output`, `output` (final answer string), `messages`, `trace`, `trace_summary`, `token_usage`, `cost_usd`, `duration_ms`, `start_time`, `end_time`, `file_changes`, `workspace_path`, `config`
+Raw stdin uses snake_case and includes: `input`, `expected_output`, `output` (final answer string), `messages`, `trace`, `trace_summary`, `token_usage`, `cost_usd`, `duration_ms`, `start_time`, `end_time`, `file_changes`, `workspace_path`, `config`
 SDK handlers receive the same payload in camelCase: `expectedOutput`, `traceSummary`, `tokenUsage`, `costUsd`, `durationMs`, `startTime`, `endTime`, `fileChanges`, `workspacePath`.
-When a workspace is configured, `workspace_path` is the absolute path to the workspace dir (also available as `AGENTV_WORKSPACE_PATH` env var). Use this for functional grading (e.g., running `npm test` in the workspace).
+When an environment prepares a workspace directory, `workspace_path` is the absolute path to that directory (also available as `AGENTV_WORKSPACE_PATH` env var). Use this for functional grading (e.g., running `npm test` in the prepared workdir).
 For deterministic workspace checks that fit normal Vitest `expect(...)` tests, prefer a plain verifier file and the built-in adapter:
 ```yaml
 - name: welcome_banner
-  type: code-grader
+  type: script
   command: [agentv, eval, graders/welcome-banner.test.ts]
 ```
 AgentV infers the Vitest adapter for `*.test.ts`, `*.spec.ts`, and Vercel-style `EVAL.ts` files. Use the explicit `agentv eval vitest` subcommand only when you need adapter flags such as `--cwd`, `--in-workspace`, or `--vitest-command`.
-See docs at https://agentv.dev/graders/code-graders/
+See the Script Graders docs for the full stdin/stdout contract.
 
-### llm-grader
+### llm-rubric
 ```yaml
 - name: quality
-  type: llm-grader
+  type: llm-rubric
   prompt: ./prompts/eval.md     # markdown template or command config
   target: grader_gpt_5_mini     # optional: override the grader target for this grader
   model: gpt-5-chat            # optional model override
@@ -394,26 +549,27 @@ See docs at https://agentv.dev/graders/code-graders/
 Variables: `{{criteria}}`, `{{input}}`, `{{expected_output}}`, `{{output}}`, `{{metadata}}`, `{{metadata_json}}`, `{{rubrics}}`, `{{rubrics_json}}`, `{{file_changes}}`, `{{tool_calls}}`
 - Markdown templates: use `{{variable}}` syntax
 - TypeScript templates: use `definePromptTemplate(fn)` from `@agentv/sdk`, receives context object with all variables + `config`
-- Use `target:` to run different `llm-grader` graders against different named LLM targets in the same eval (useful for grader panels / ensembles)
+- Use `target:` to run different `llm-rubric` graders against different named LLM targets in the same eval (useful for grader panels / ensembles)
 
-### composite
+### assert-set
 ```yaml
-- name: gate
-  type: composite
-  assertions:
-    - name: safety
-      type: llm-grader
+- metric: gate
+  type: assert-set
+  threshold: 0.7
+  config:
+    shared_setting: enabled
+  assert:
+    - metric: safety
+      type: llm-rubric
       prompt: ./safety.md
-    - name: quality
-      type: llm-grader
-  aggregator:
-    type: weighted_average
-    weights: { safety: 0.3, quality: 0.7 }
+      weight: 0.3
+    - metric: quality
+      type: llm-rubric
+      weight: 0.7
 ```
-Aggregator types: `weighted_average`, `all_or_nothing`, `minimum`, `maximum`, `safety_gate`
-- `safety_gate`: fails immediately if the named gate grader scores below threshold (default 1.0)
+Use `assert-set` for Promptfoo-aligned assertion grouping. Without `threshold`, the set passes only when every nonzero-weight child assertion passes. With `threshold`, the weighted aggregate score determines the set verdict. Parent `config` is inherited by children, and child `config` keys override parent keys. Do not use `type: composite`; AgentV rejects it.
 
-### tool_trajectory
+### tool-trajectory
 ```yaml
 - name: tool_check
   type: tool-trajectory
@@ -428,8 +584,9 @@ Aggregator types: `weighted_average`, `all_or_nothing`, `minimum`, `maximum`, `s
       max_duration_ms: 5000            # per-tool latency assertion
     - tool: summarize                  # omit args to skip argument checking
 ```
+`tool-trajectory` is an AgentV-specific extension over AgentV-normalized transcripts. Do not use Promptfoo `trajectory:*`, `tool-call-f1`, `skill-used`, or `trace-*` names; AgentV rejects those until their trace semantics are implemented directly.
 
-### field_accuracy
+### field-accuracy
 ```yaml
 - name: fields
   type: field-accuracy
@@ -453,14 +610,14 @@ Compares `output` fields against `expected_output` fields.
   max_usd: 0.10
 ```
 
-### token_usage
+### token-usage
 ```yaml
 - name: tokens
   type: token-usage
   max_total_tokens: 4000
 ```
 
-### execution_metrics
+### execution-metrics
 ```yaml
 - name: efficiency
   type: execution-metrics
@@ -497,48 +654,37 @@ Binary check: does output match the regex pattern?
 ```
 Binary check: does output exactly equal the value (both trimmed)?
 
-### is_json
+### is-json
 ```yaml
 - type: is-json
   required: true
 ```
 Binary check: is the output valid JSON?
 
-### rubrics
+### llm-rubric
 ```yaml
-- type: rubrics
-  criteria:
-    - id: accuracy
-      outcome: Correctly identifies the denied party
-      weight: 5.0
-    - id: reasoning
-      outcome: Provides clear reasoning
-      weight: 3.0
+- Correctly identifies the denied party
+- Provides clear reasoning
 ```
-LLM-judged structured evaluation with weighted criteria. Criteria items support `id`, `outcome`, `weight`, and `required` fields.
+LLM-judged structured evaluation. Plain strings are the preferred shorthand.
+Use `type: llm-rubric` when you need weighted criteria, `required: false`,
+`min_score`, or score ranges. Rubric items support `id`,
+`outcome`, `weight`, and `required` fields.
 Use optional `operator: correctness` for positive support checks or `operator: contradiction` for guard criteria where omission is acceptable but incompatible claims fail.
 
 See `references/rubric-grader.md` for score-range mode and scoring formula.
 
-## Execution Error Tolerance
-
-Control how the runner handles execution errors (infrastructure failures, not quality failures):
-
-```yaml
-execution:
-  fail_on_error: false    # never halt (default)
-  # fail_on_error: true   # halt on first execution error
-```
-
-When halted, remaining tests get `executionStatus: 'execution_error'` with `failureReasonCode: 'error_threshold_exceeded'`.
-
 ## Suite-Level Quality Threshold
 
-Set a minimum mean score for the eval suite. If the mean quality score falls below the threshold, the CLI exits with code 1 — useful for CI/CD quality gates.
+Set a minimum mean score for the eval suite. If the mean quality score falls below the threshold, the CLI exits with code 1 — useful for CI/CD quality gates. Use `evaluate_options.repeat` when each case should be attempted more than once.
 
 ```yaml
-execution:
-  threshold: 0.8
+evaluate_options:
+  repeat:
+    count: 3
+    strategy: pass_any
+    early_exit: false
+threshold: 0.8
 ```
 
 CLI flag `--threshold 0.8` overrides the YAML value. Must be a number between 0 and 1. Mean score is computed from quality results only (execution errors excluded).
@@ -549,7 +695,7 @@ The threshold also controls JUnit XML pass/fail: tests with scores below the thr
 
 ```bash
 # Run evaluation (requires API keys)
-agentv eval <file.yaml> [--test-id <id>] [--target <name>] [--dry-run] [--threshold <0-1>]
+agentv eval <file.yaml> [--test-id <id>] [--target <name>] [--threshold <0-1>]
 
 # Run with OTLP JSON file (importable by OTel backends)
 agentv eval <file.yaml> --otel-file traces/eval.otlp.json
@@ -570,18 +716,27 @@ agentv eval <file.yaml> --retry-errors .agentv/results/default/<timestamp>/index
 # Validate eval file
 agentv validate <file.yaml>
 
-# Compare results — N-way matrix from a canonical run manifest
-agentv compare .agentv/results/default/<timestamp>/index.jsonl
-agentv compare .agentv/results/default/<timestamp>/index.jsonl --baseline <target>                   # CI regression gate
-agentv compare .agentv/results/default/<timestamp>/index.jsonl --baseline <target> --candidate <target>  # pairwise
-agentv compare .agentv/results/default/<baseline-timestamp>/index.jsonl .agentv/results/default/<candidate-timestamp>/index.jsonl
+# Compare completed runs
+agentv results compare \
+  .agentv/results/default/<baseline-timestamp>/index.jsonl \
+  .agentv/results/default/<candidate-timestamp>/index.jsonl
+agentv results combine \
+  .agentv/results/default/<baseline-timestamp> \
+  .agentv/results/default/<candidate-timestamp> \
+  .agentv/results/default/<third-target-timestamp> \
+  --output .agentv/results/default/combined
+agentv results compare .agentv/results/default/combined/index.jsonl
+agentv results compare \
+  .agentv/results/default/<baseline-timestamp>/index.jsonl \
+  .agentv/results/default/<candidate-timestamp>/index.jsonl \
+  --json
 
 # Author assertions directly in the eval file
 # Prefer simple assertions when they fit the criteria; use deterministic or LLM-based graders when needed
 agentv validate <file.yaml>
 ```
 
-**Replay targets:** Add `provider: replay`, `fixtures: <jsonl>`, and `source_target: <live target name>` in `.agentv/targets.yaml`. Optional `suite`, `eval_path`, and `variant` tighten lookup. The eval YAML and graders stay unchanged; replay only substitutes recorded target output, and graders run fresh.
+**Replay targets:** Add `provider: replay`, `fixtures: <jsonl>`, and `source_target: <live target label>` in `.agentv/targets.yaml`. Optional `suite`, `eval_path`, and `variant` tighten lookup. The eval YAML and graders stay unchanged; replay only substitutes recorded target output, and graders run fresh.
 
 ## TypeScript SDK Helpers
 
@@ -593,12 +748,19 @@ import { defineEval, graders } from '@agentv/sdk';
 
 export default defineEval({
   name: 'helper-suite',
-  execution: { targets: ['default'] },
+  target: 'default',
+  // The SDK helper lowers this to evaluate_options.repeat in generated YAML.
+  repeat: {
+    count: 3,
+    strategy: 'pass_any',
+    earlyExit: false,
+  },
+  threshold: 0.8,
   tests: [
     {
       id: 'json-answer',
       input: 'Return a JSON answer with a status field.',
-      assertions: [
+      assert: [
         graders.json({ name: 'valid-json', required: true }),
         graders.regex(/"status"\s*:/, { name: 'status-key' }),
       ],
@@ -607,7 +769,7 @@ export default defineEval({
 });
 ```
 
-The `graders` catalog returns ordinary `assertions` entries such as `type: is-json`, `type: regex`, `type: llm-grader`, and `type: code-grader`. `defineEval()` lowers camelCase TypeScript fields such as `expectedOutput`, `inputFiles`, and `maxSteps` to canonical snake_case YAML/runtime keys.
+The `graders` catalog returns ordinary `assert` entries such as `type: is-json`, `type: regex`, `type: llm-rubric`, and `type: script`. `defineEval()` lowers camelCase TypeScript fields such as `expectedOutput`, `inputFiles`, and `maxSteps` to canonical snake_case YAML/runtime keys.
 
 If adapting Braintrust `scores` or DeepEval metrics, write small AgentV helper factories that return `graders.*` configs:
 
@@ -615,7 +777,7 @@ If adapting Braintrust `scores` or DeepEval metrics, write small AgentV helper f
 import { graders } from '@agentv/sdk';
 
 export function ragFaithfulness() {
-  return graders.llmGrader({
+  return graders.llmRubric(undefined, {
     name: 'rag-faithfulness',
     target: 'grader-target',
     prompt: 'Grade whether the answer is supported by the retrieved context.',
@@ -623,9 +785,9 @@ export function ragFaithfulness() {
 }
 ```
 
-Use the helper in `assertions: [ragFaithfulness()]`; do not create new YAML terms like `scores`.
+Use the helper in `assert: [ragFaithfulness()]`; do not create new YAML terms like `scores`.
 
-### defineAssertion (recommended for custom checks)
+### defineAssertion (recommended for reusable custom assertions)
 ```typescript
 #!/usr/bin/env bun
 import { defineAssertion } from '@agentv/sdk';
@@ -641,16 +803,18 @@ export default defineAssertion(({ output, trace }) => {
 
 Assertions support both `pass: boolean` and `score: number` (0-1). If only `pass` is given, score is 1 (pass) or 0 (fail).
 
-### defineCodeGrader (full control)
+Use `defineAssertion()` when you want a reusable assertion type discovered from `.agentv/assertions/` and referenced by filename as `type: <name>`. This follows Promptfoo's normal eval terminology: custom logic is an assertion, with Promptfoo using fixed assertion types such as `javascript`, `python`, `ruby`, and `webhook`. AgentV extends that model by allowing arbitrary discovered assertion type names.
+
+### defineScriptGrader (full control)
 ```typescript
 #!/usr/bin/env bun
-import { defineCodeGrader } from '@agentv/sdk';
+import { defineScriptGrader } from '@agentv/sdk';
 
-export default defineCodeGrader(({ output, trace }) => {
+export default defineScriptGrader(({ output, trace }) => {
   const finalOutput = output ?? '';
   return {
     score: finalOutput.length > 0 && (trace?.eventCount ?? 0) <= 5 ? 1.0 : 0.5,
-    assertions: [
+    assert: [
       { text: 'Output is not empty', passed: finalOutput.length > 0 },
       { text: 'Efficient tool usage', passed: (trace?.eventCount ?? 0) <= 5 },
     ],
@@ -658,14 +822,14 @@ export default defineCodeGrader(({ output, trace }) => {
 });
 ```
 
-`defineAssertion()` files go in `.agentv/assertions/` and are referenced by filename as `type: <name>`. `defineCodeGrader()` scripts are referenced in YAML with `type: code-grader` and `command: [bun, run, grader.ts]`. Plain Vitest workspace verifier files can use `command: [agentv, eval, graders/check.test.ts]`.
+Use `defineScriptGrader()` when the custom component is a command-backed grader with explicit score control, custom assertion-result arrays, workspace commands, or LLM calls through a grader target. `defineScriptGrader()` scripts are referenced in YAML with `type: script` and `command: [bun, run, grader.ts]`. Plain Vitest workspace verifier files can use `command: [agentv, eval, graders/check.test.ts]`.
 
 ### Convention-Based Discovery
 
 Place assertion files in `.agentv/assertions/` — they auto-register by filename:
 
 ```
-.agentv/assertions/word-count.ts  →  type: word-count
+.agentv/assertions/min-words.ts  →  type: min-words
 .agentv/assertions/sentiment.ts   →  type: sentiment
 ```
 
@@ -696,6 +860,7 @@ Programmatic API notes:
 
 - Inline programmatic tests use `assert`, not `assertions`.
 - Use camelCase in TypeScript (`expectedOutput`, `beforeAll`, `budgetUsd`).
+- In YAML, use `evaluate_options.max_concurrency` for authored eval concurrency; reserve `workers` for project/target runtime config.
 - When bridging from Python, generate canonical YAML/JSONL or call the CLI; there is no separate first-party Python authoring SDK.
 
 Supports inline tests or file-based usage via `specFile`.
@@ -726,34 +891,6 @@ agentv create eval <name>       # → evals/<name>.eval.yaml + .cases.jsonl
 ## Skill Improvement Workflow
 
 For a complete guide to iterating on skills using evaluations — writing scenarios, running baselines, comparing results, and improving — see the [Skill Improvement Workflow](https://agentv.dev/guides/skill-improvement-workflow/) guide.
-## Human Review Checkpoint
-
-After running evals, perform a human review before iterating. Create `feedback.json` in the results directory:
-
-```json
-{
-  "run_id": "2026-03-14T10-32-00_claude",
-  "reviewer": "engineer-name",
-  "timestamp": "2026-03-14T12:00:00Z",
-  "overall_notes": "Summary of observations",
-  "per_case": [
-    {
-      "test_id": "test-id",
-      "verdict": "acceptable | needs_improvement | incorrect | flaky",
-      "notes": "Why this verdict",
-      "evaluator_overrides": { "code-grader:name": "Override note" },
-      "workspace_notes": "Workspace state observations"
-    }
-  ]
-}
-```
-
-Use `evaluator_overrides` for workspace evaluations to annotate specific grader results (e.g., "code-grader was too strict"). Use `workspace_notes` for observations about workspace state.
-
-Review workflow: run evals → inspect results (`agentv inspect show`) → write feedback → tune prompts/graders → re-run.
-
-Full guide: https://agentv.dev/guides/human-review/
-
 ## Observability Export
 
 AgentV exports observability data via OpenTelemetry:
@@ -765,24 +902,21 @@ Do not invent a separate Opik-specific eval surface. Keep the eval definition in
 
 ## Schemas
 
-- Eval file: `references/eval-schema.json`
-- Experiment file: `references/experiment-schema.json`
-- Config: `references/config-schema.json`
+- Eval file: `references/eval.schema.json`
+- Config: `references/config.schema.json`
 
 ## Accessing reference files
 
 To load a specific reference without pulling the entire skill into context:
 
 ```bash
-agentv skills get agentv-eval-writer --ref eval-schema.json
-agentv skills get agentv-eval-writer --ref experiment-schema.json
+agentv skills get agentv-eval-writer --ref eval.schema.json
 ```
 
 Or resolve the skill directory and read files directly:
 
 ```bash
-cat $(agentv skills path agentv-eval-writer)/references/eval-schema.json
-cat $(agentv skills path agentv-eval-writer)/references/experiment-schema.json
+cat $(agentv skills path agentv-eval-writer)/references/eval.schema.json
 ```
 
 Use `--full` to retrieve every file in the skill at once.