agentv 5.2.0-next.1 → 5.3.1-next.1

package/dist/skills/agentv-bench/references/eval-yaml-spec.md

@@ -20,7 +20,7 @@ The grader agent uses this to evaluate assertions without the CLI.
 
 - `id` (string, required) — unique test identifier
 - `vars` (object, required when prompts need row data) — prompt-template variables for this row
-- `expected_output` (string | Message[], optional) — passive reference answer. String shorthand expands to `[{role: assistant, content: "..."}]`. It is available to declared graders, but does not add an implicit grader when `assertions` is present.
+- `vars.expected_output` (string | Message[], optional) — passive reference answer data. It is available to declared graders, but does not add an implicit grader when `assert` is present.
 - `criteria` (string, optional) — human-readable success criteria
 - `assert` (array, optional) — grader assertions
 - `environment` (object | `file://...`, optional) — per-case testbed override
@@ -31,8 +31,8 @@ If `assert` already states the grading contract, omit `criteria` instead of
 duplicating the same rubric. Prefer plain assertion strings for semantic checks
 when the default LLM rubric grader can judge them; use multiple named
 `type: llm-rubric` blocks only for custom prompts, custom grader targets, or
-intentional grader panels. Write `expected_output` as a golden/reference answer,
-not as criteria or scoring instructions.
+intentional grader panels. Write `vars.expected_output` as a golden/reference
+answer, not as criteria or scoring instructions.
 
 For historical or repo-state evals, materialize the repository through
 `environment.setup.command` and pass the repo/ref as argv inputs. A SHA in
@@ -45,11 +45,12 @@ checkout. `setup.command` is a non-empty string array. Put the executable at
 
 ### Default grader contract
 
-When a test has no `assertions`, AgentV uses the default `llm-rubric` with the case context,
-including `criteria` and `expected_output` when present.
+When a test has no `assert`, AgentV can use configured suite/default graders
+with the case context, including `criteria` and `vars.expected_output` when
+present.
 
-When `assertions` is present, the list is explicit: run only the declared
-assertions/graders. `expected_output` remains reference data for graders that consume it,
+When `assert` is present, the list is explicit: run only the declared
+assertions/graders. `vars.expected_output` remains reference data for graders that consume it,
 such as `llm-rubric`, `script`, or `field-accuracy`; it does not trigger an additional
 default `llm-rubric`.
 When the declared assertion strings fully express the semantic contract, do not
@@ -171,18 +172,29 @@ Same as contains variants but explicitly case-insensitive.
 
 ### Tool inspection assertions
 
-#### `tool-trajectory`
+#### `trajectory:tool-used`
 
-- **Fields:** `expected` (array of expected tool calls), `mode` (string: `any_order` | `in_order` | `exact` | `subset` | `superset`)
-- **Recipe:** Inspect AgentV-normalized transcript/tool-call data. Match against expected based on mode.
-- **PASS:** tool calls match expected pattern per mode.
-- **Boundary:** This is an AgentV extension. Promptfoo `trajectory:*`, `tool-call-f1`, `skill-used`, and `trace-*` assertion names are rejected until implemented directly.
+- **Fields:** `value.name` (string, required), `value.min` (number, optional)
+- **Recipe:** Inspect AgentV-normalized transcript/tool-call data and count matching tool calls.
+- **PASS:** matching calls meet the requested minimum.
 
-#### `skill-trigger`
+#### `trajectory:tool-sequence`
 
-- **Fields:** `skill_name` (string, required)
-- **Recipe:** Check if the agent invoked the named skill in its tool calls.
-- **PASS:** skill was triggered.
+- **Fields:** `value.steps` (array of tool names, required), `value.mode` (`in_order` | `exact`)
+- **Recipe:** Inspect AgentV-normalized transcript/tool-call data and compare the observed sequence.
+- **PASS:** tool calls match the expected sequence for the selected mode.
+
+#### `trajectory:tool-args-match`
+
+- **Fields:** `value.name` (string, required), `value.args` (object, required), `value.mode` (`partial` | `exact`)
+- **Recipe:** Inspect matching tool calls and compare arguments.
+- **PASS:** at least one matching call has the expected arguments.
+
+#### `skill-used` / `not-skill-used`
+
+- **Fields:** `value` (skill name string or pattern object, required)
+- **Recipe:** Check whether the agent invoked the named skill in its tool calls.
+- **PASS:** `skill-used` finds the skill; `not-skill-used` confirms it is absent.
 
 ### LLM-judged assertions (require Claude reasoning)
 
@@ -200,18 +212,20 @@ Same as contains variants but explicitly case-insensitive.
 
 ### Script-based assertions
 
-#### `script-grader`
+#### `script`
 
-- **Fields:** `path` (string, required — path to script), `command` (string[], optional — custom command)
+- **Fields:** `command` (string[], required — command to execute), optional `cwd`, `config`, and `target`
 - **Script SDK:** Use `defineScriptGrader` from `@agentv/sdk`:
   ```typescript
   import { defineScriptGrader } from '@agentv/sdk';
   export default defineScriptGrader(({ output, trace }) => ({
+    pass: (output ?? '').includes('expected'),
     score: (output ?? '').includes('expected') ? 1 : 0,
-    assert: [{ text: 'Contains expected', passed: (output ?? '').includes('expected') }],
+    reason: 'Checks for the expected token',
+    checks: [{ text: 'Contains expected', pass: (output ?? '').includes('expected'), reason: 'Substring check' }],
   }));
   ```
-- **Recipe:** The CLI runs the script, passing canonical JSON on stdin (`{output, input, expected_output, ...}`). Script returns `{"score": N, "assertions": [...]}`
+- **Recipe:** The CLI runs the script, passing canonical JSON on stdin (`{output, input, expected_output, ...}`). Script returns `{pass, score, reason, checks?}`; public artifacts normalize `checks` into recursive `component_results`.
 - **PASS:** score >= 0.5 (or as configured).
 
 ### Assertion groups
@@ -241,42 +255,22 @@ All assertion types support:
 - `negate` (boolean, optional) — invert result
 - `threshold` (number, optional) — minimum score to pass (for LLM types)
 
-## 5. AgentV JSONL Output Format
-
-Each line in the results JSONL file is an `EvaluationResult` object. In JSONL, field names use snake_case (applied by `toSnakeCaseDeep()`).
-
-### Required fields
-
-- `timestamp` (string, ISO-8601)
-- `test_id` (string)
-- `score` (number, 0.0-1.0, weighted average of all assertion scores)
-- `assertions` (array of `{text, passed, evidence?}`)
-- `output` (string) — final answer/scored result; transcript evidence is available through captured trace/messages when present
-- `execution_status` (string: `ok` | `quality_failure` | `execution_error`)
-
-### Optional fields
+## 5. AgentV Run Bundle Output
 
-- `scores` (array of EvaluatorResult) — per-grader breakdown
-- `input` (Message[]) — input messages
-- `token_usage` (object: `{prompt_tokens, completion_tokens, total_tokens}`)
-- `cost_usd` (number)
-- `duration_ms` (number)
-- `target` (string)
-- `eval_set` (string)
-- `error` (string)
-- `file_changes` (string — unified diff)
-- `mode` (string — `agent` for agent mode)
+Each run writes `.agentv/results/<run_id>/`. Public persisted fields use
+snake_case. The canonical discovery file is `.internal/index.jsonl`; per-attempt
+sidecars live under each result directory.
 
-### `scores[]` entries (EvaluatorResult)
+Key artifacts:
 
-- `name` (string) — grader name
-- `type` (string) — grader kind (kebab-case)
-- `score` (number, 0.0-1.0)
-- `assertions` (array of `{text, passed, evidence?}`)
-- `weight` (number, optional)
-- `verdict` (string: `pass` | `fail` | `skip`)
-- `details` (object, optional — structured data from script graders)
-- `reasoning` (string, optional)
+- `.internal/index.jsonl` — one row per result aggregate with identity fields,
+  status, score, target metadata, and explicit paths to sidecars.
+- `summary.json` — run-level metadata and aggregate pass/score rollups.
+- `sample-N/grading.json` — aggregate `pass`, `score`, `reason`, recursive
+  `component_results`, and optional `assertion`, `named_scores`, and `metadata`.
+- `sample-N/metrics.json` — token, cost, timing, tool, and transcript metrics.
+- `sample-N/transcript.json` and `sample-N/transcript-raw.jsonl` — normalized
+  and raw transcript data when available.
 
 ## 6. Eval Set Support
 
@@ -355,9 +349,9 @@ LLM grader results are read from disk at `<test-id>/llm_grader_results/<name>.js
 ```
 
 **Output:**
-- `<test-id>/run-1/grading.json` — merged grading with `graders`, `assertions`, `summary.pass_rate`
-- `index.jsonl` — one JSON line per test: `{test_id, score, pass, graders: [...]}`
-- `summary.json` — aggregate stats: `{metadata: {targets}, run_summary: {<target>: {mean, stddev, n}}}`
+- `.internal/index.jsonl` — canonical row index
+- `summary.json` — aggregate run summary
+- per-attempt `grading.json` — aggregate `pass`, `score`, `reason`, and recursive `component_results`
 
 ### Agent-Mode Workflow
 

package/dist/skills/agentv-bench/references/schemas.md

@@ -85,80 +85,64 @@ Tracks version progression in Improve mode. Located at workspace root.
 
 ## grading.json
 
-Output from the grader agent. Located at `<case-dir>/run-N/grading.json`.
+Public per-attempt grading output. Located at
+`.agentv/results/<run_id>/<result_dir>/sample-N/grading.json`.
 
-**Important:** The `assertions` array must use the fields `text`, `passed`, and `evidence` — downstream tooling depends on these exact field names.
+`checks` and other SDK/script convenience shapes are normalized into recursive
+`component_results`. Do not document `assertions`, `passed`, `graders`, or
+`evidence` as public grading artifact fields.
 
 ```json
 {
-  "assertions": [
+  "pass": false,
+  "score": 0.67,
+  "reason": "One required spreadsheet check failed.",
+  "component_results": [
     {
-      "text": "The output includes the name 'John Smith'",
-      "passed": true,
-      "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
+      "name": "name-check",
+      "type": "llm-rubric",
+      "pass": true,
+      "score": 1,
+      "reason": "The answer includes the expected name.",
+      "assertion": {
+        "text": "The output includes the expected name."
+      }
     },
     {
-      "text": "The spreadsheet has a SUM formula in cell B10",
-      "passed": false,
-      "evidence": "No spreadsheet was created. The output was a text file."
-    }
-  ],
-  "summary": {
-    "passed": 2,
-    "failed": 1,
-    "total": 3,
-    "pass_rate": 0.67
-  },
-  "execution_metrics": {
-    "tool_calls": {
-      "Read": 5,
-      "Write": 2,
-      "Bash": 8
-    },
-    "total_tool_calls": 15,
-    "total_steps": 6,
-    "errors_encountered": 0,
-    "output_chars": 12450,
-    "transcript_chars": 3200
-  },
-  "timing": {
-    "executor_duration_seconds": 165.0,
-    "grader_duration_seconds": 26.0,
-    "total_duration_seconds": 191.0
-  },
-  "claims": [
-    {
-      "claim": "The form has 12 fillable fields",
-      "type": "factual",
-      "verified": true,
-      "evidence": "Counted 12 fields in field_info.json"
+      "name": "spreadsheet-check",
+      "type": "script",
+      "pass": false,
+      "score": 0,
+      "reason": "No spreadsheet was created.",
+      "component_results": [
+        {
+          "pass": false,
+          "score": 0,
+          "reason": "Cell B10 did not contain a SUM formula.",
+          "assertion": {
+            "text": "Spreadsheet has SUM formula in B10"
+          }
+        }
+      ]
     }
   ],
-  "user_notes_summary": {
-    "uncertainties": ["Used 2023 data, may be stale"],
-    "needs_review": [],
-    "workarounds": ["Fell back to text overlay for non-fillable fields"]
+  "named_scores": {
+    "spreadsheet": 0
   },
-  "eval_feedback": {
-    "suggestions": [
-      {
-        "assertion": "The output includes the name 'John Smith'",
-        "reason": "A hallucinated document that mentions the name would also pass"
-      }
-    ],
-    "overall": "Assertions check presence but not correctness."
+  "metadata": {
+    "grader_target": "local-openai-grader"
   }
 }
 ```
 
 **Fields:**
-- `assertions[]`: Graded assertion results with evidence
-- `summary`: Aggregate pass/fail counts
-- `execution_metrics`: Tool usage and output size (from executor's metrics.json)
-- `timing`: Wall clock timing (from timing.json)
-- `claims`: Extracted and verified claims from the output
-- `user_notes_summary`: Issues flagged by the executor
-- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising
+- `pass`: Aggregate pass/fail for the grading component
+- `score`: Aggregate score from 0 to 1
+- `reason`: Human-readable explanation for the aggregate result
+- `component_results`: Recursive child grading components
+- `assertion`: Optional assertion metadata for a leaf component
+- `named_scores`: Optional metric-name to score map
+- `metadata`: Optional grader/runtime metadata
 
 ---
 
@@ -198,7 +182,7 @@ Output from the executor agent. Located at `<case-dir>/run-N/metrics.json`.
 
 ## timing.json
 
-Wall clock timing for a run attempt. Located at `<case-dir>/run-N/timing.json`.
+Wall clock timing for a run sample. Located at `<case-dir>/sample-N/timing.json`.
 
 **How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.
 

package/dist/skills/agentv-bench/references/subagent-pipeline.md

@@ -30,10 +30,12 @@ specific target, set `subagent_mode_allowed: false` in `.agentv/targets.yaml`:
 ```yaml
 # .agentv/targets.yaml
 targets:
-  - name: my-target
+  - id: my-target
     provider: openai
-    model: ${{ OPENAI_MODEL }}
-    api_key: ${{ OPENAI_API_KEY }}
+    runtime: host
+    config:
+      model: "{{ env.OPENAI_MODEL }}"
+      api_key: "{{ env.OPENAI_API_KEY }}"
     subagent_mode_allowed: false  # forces CLI invocation instead of executor subagent
 ```
 
@@ -43,8 +45,8 @@ even in subagent mode.
 ## CLI Targets: Single Command
 
 For evals with CLI targets, `pipeline run` handles input extraction, target invocation, and
-code grading in one step. When `--out` is omitted, the output directory defaults to
-`.agentv/results/default/<timestamp>` (same convention as `agentv eval`):
+code grading in one step. When `--out` is omitted, the output directory uses the
+canonical `.agentv/results/<run_id>` convention:
 
 ```bash
 # Extract inputs and invoke all CLI targets in parallel:
@@ -70,7 +72,7 @@ opted out via `subagent_mode_allowed: false` in `.agentv/targets.yaml`), fall ba
 ### Step 1: Extract inputs
 
 ```bash
-# Defaults to .agentv/results/default/<timestamp>
+# Defaults to .agentv/results/<run_id>
 agentv pipeline input evals/repro.eval.yaml
 ```
 
@@ -117,7 +119,7 @@ LLM grading → merge and validate).
 Use individual commands when you need control over each step with CLI targets:
 
 ```bash
-# Step 1: Extract inputs (defaults to .agentv/results/default/<timestamp>)
+# Step 1: Extract inputs (defaults to .agentv/results/<run_id>)
 agentv pipeline input evals/repro.eval.yaml
 
 # Step 2: run_tests.py invokes CLI targets (or use pipeline run instead)
@@ -127,7 +129,7 @@ agentv pipeline grade <run-dir>
 
 # Step 4: Subagent does LLM grading, writes results to llm_grader_results/<name>.json per test
 
-# Step 5: Merge scores (writes index.jsonl with full scores[] for dashboard)
+# Step 5: Merge scores into the canonical run bundle
 agentv pipeline bench <run-dir>
 
 # Step 6: Validate
@@ -152,22 +154,22 @@ The agent reads `llm_graders/<name>.json` for each test, grades the response usi
 
 ## Pipeline Bench and Dashboard
 
-`pipeline bench` merges LLM scores into `index.jsonl` with a full `scores[]` array per entry,
-matching the CLI-mode schema. The web dashboard (`agentv results serve`) reads this format
-directly — no separate conversion script is needed. Run `agentv results validate <run-dir>`
-to verify compatibility.
+`pipeline bench` merges LLM scores into the canonical run bundle, including
+`.internal/index.jsonl`, `summary.json`, and per-attempt `grading.json` with
+`component_results`. The web dashboard reads this format directly — no separate
+conversion script is needed. Run `agentv results validate <run-dir>` to verify
+compatibility.
 
 ## Output Structure
 
-The path hierarchy mirrors the CLI mode: `<evalset-name>` comes from the `name` field in
-the eval.yaml. The target is recorded in `manifest.json` — one run = one target.
+The path hierarchy mirrors CLI mode. The target is recorded in run metadata.
 
 ```
-.agentv/results/<experiment>/<timestamp>/
+.agentv/results/<run_id>/
 ├── manifest.json                    ← eval metadata, target, test_ids
-├── index.jsonl                      ← per-test scores
-├── summary.json                   ← aggregate statistics
-└── <evalset-name>/                  ← eval.yaml "name" field, or eval file basename if absent (same as CLI mode)
+├── summary.json                      ← aggregate statistics
+├── .internal/index.jsonl             ← canonical row index
+└── <result-dir>/sample-1/grading.json
     └── <test-id>/                   ← test case id
         ├── input.json               ← test input text + messages
         ├── invoke.json              ← target command or agent instructions

package/dist/skills/agentv-eval-migrations/SKILL.md

@@ -16,6 +16,19 @@ v4.42.4-era shape, current shape, migration steps, verification commands, and
 compatibility notes for each major breaking authoring change. Then compare the
 eval file against the current portable contract:
 
+- Treat migrations as rule-based rewrites first. Do not rely on a manual
+  "looks current" pass when the old field has a direct current replacement.
+- Rewrite authored prompt data into top-level `prompts` plus `tests[].vars`;
+  keep `input` only for documented raw-case import compatibility.
+- Move sibling `expected_output` to `vars.expected_output`, then add or keep an
+  explicit assertion that consumes it, usually `llm-rubric` with
+  `{{ expected_output }}`.
+- Rewrite target identity to `id`, keep `provider` for the backend/adapter kind,
+  put provider settings under `config`, and rewrite env references to
+  `{{ env.NAME }}`.
+- Remove `use_target`, `eval_cases`, `evalcases`, `providerPromptMap`, and
+  `provider_prompt_map` from authored eval/config YAML. Current AgentV rejects
+  these fields, so do not preserve them as compatibility aliases.
 - Keep committed eval YAML portable: prompts, cases, assertions, workspace
   templates, repos, hooks, env checks, Docker preflight/container config, and
   `workspace.scope`.

package/dist/skills/agentv-eval-migrations/references/breaking-changes.md

@@ -23,7 +23,7 @@ For a v4.42.4-era eval:
 
 1. Move top-level `execution.target` to top-level `target`.
 2. Move top-level `execution.targets` to top-level `targets`, and rename target
-   object `name` to `label`.
+   object `name` to `id`.
 3. Rename suite/test/turn `assertions` to `assert`.
 4. If a test uses `tests[].execution.assertions` or
    `tests[].execution.evaluators`, rename that nested list to
@@ -51,9 +51,19 @@ For a v4.42.4-era eval:
 15. Replace authored `preprocessors` and deprecated Promptfoo `postprocess`
     with `transform` at `default_test.options`, `tests[].options`, or the
     assertion that needs the shaped output.
-16. Keep raw cases under `tests` / `tests: file://...`; run full eval suites
+16. Replace `type: skill-trigger` with `skill-used` or `not-skill-used`.
+17. Replace `type: tool-trajectory` with Promptfoo-compatible `trajectory:*`
+    assertions where there is a direct mapping.
+18. Keep raw cases under `tests` / `tests: file://...`; run full eval suites
     directly with CLI multi-file selection and tags.
-17. Validate with `bun apps/cli/src/cli.ts validate <eval-file>`.
+19. Move authored prompt input from top-level or `tests[].input` into
+    `prompts` plus `tests[].vars.input`.
+20. Move authored reference answers from sibling `expected_output` into
+    `vars.expected_output`; add or keep an explicit assertion that consumes it.
+21. Rewrite target env interpolation from `${{ NAME }}` to `{{ env.NAME }}`.
+22. Remove `use_target`, `eval_cases`, `evalcases`, `providerPromptMap`, and
+    `provider_prompt_map`; current AgentV rejects them.
+23. Validate with `bun apps/cli/src/cli.ts validate <eval-file>`.
 
 ## Assertions Renamed To `assert`
 
@@ -211,6 +221,101 @@ templates and custom graders can consume it. The breaking authoring change is
 that workers should not depend on missing `assert` plus `criteria` to define
 the whole grading contract for new migrated YAML.
 
+## `skill-trigger` And `tool-trajectory` Replaced By Promptfoo Assertions
+
+### v4.42.4 Shape
+
+Older AgentV YAML used AgentV-specific skill and tool trajectory assertions:
+
+```yaml
+assert:
+  - type: skill-trigger
+    skill: csv-analyzer
+    should_trigger: true
+  - type: tool-trajectory
+    mode: any_order
+    minimums:
+      search: 2
+  - type: tool-trajectory
+    mode: exact
+    expected:
+      - tool: search
+        args:
+          q: agentv
+      - tool: fetch
+```
+
+### Current Shape
+
+Current authored eval YAML uses Promptfoo-compatible assertions:
+
+```yaml
+assert:
+  - type: skill-used
+    value: csv-analyzer
+  - type: trajectory:tool-used
+    value:
+      name: search
+      min: 2
+  - type: trajectory:tool-sequence
+    value:
+      mode: exact
+      steps: [search, fetch]
+  - type: trajectory:tool-args-match
+    value:
+      name: search
+      args:
+        q: agentv
+      mode: partial
+```
+
+### Migration Steps
+
+- Replace `type: skill-trigger`, `skill: X`, `should_trigger: true` with
+  `type: skill-used`, `value: X`.
+- Replace `type: skill-trigger`, `skill: X`, `should_trigger: false` with
+  `type: not-skill-used`, `value: X`.
+- Replace `type: tool-trajectory`, `mode: any_order`, and `minimums: { Tool: N }`
+  with one `trajectory:tool-used` assertion per tool:
+
+  ```yaml
+  - type: trajectory:tool-used
+    value:
+      name: Tool
+      min: N
+  ```
+
+- Replace `mode: in_order` or `mode: exact` plus `expected` tool steps with
+  `trajectory:tool-sequence`:
+
+  ```yaml
+  - type: trajectory:tool-sequence
+    value:
+      mode: exact
+      steps: [search, fetch]
+  ```
+
+- Move old `expected[].args` checks to `trajectory:tool-args-match`. Use
+  `mode: exact` only for exact argument equality; otherwise use Promptfoo's
+  partial argument matching.
+- Do not carry old `max_duration_ms` per-tool latency checks under
+  `trajectory:*`. There is no Promptfoo-compatible equivalent in AgentV yet;
+  use a `script` assertion or track the behavior as future scope.
+
+### Verification
+
+```bash
+bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml
+rg -n "skill-trigger|tool-trajectory" path/to/evals path/to/.agentv/templates
+```
+
+### Compatibility Notes
+
+The runtime still has private compatibility classes and tests for older
+helpers. That does not make `skill-trigger` or `tool-trajectory` valid authored
+eval YAML. Normal Promptfoo-aligned eval files fail validation and parsing with
+the replacement guidance above.
+
 ## Authored `input` Moved To `prompts` Plus Vars
 
 ### v4.42.4 Shape
@@ -746,42 +851,50 @@ targets:
 ### Current Shape
 
 Current eval YAML uses top-level `target` or `targets`. Target references use
-`label`; `id` is reserved for provider/backend identity when needed:
+the target `id`; `provider` names the backend or adapter kind:
 
 ```yaml
 target: azure-base
 
 targets:
-  - label: with-skills
-    use_target: default
+  - id: with-skills
+    provider: codex-cli
+    runtime: host
+    config:
+      command: ["codex", "exec", "--json"]
     hooks:
       before_each:
         command: ["setup-plugins.sh", "skills"]
 ```
 
-Current `.agentv/targets.yaml` uses `label` and nests provider settings under
+Current `.agentv/targets.yaml` uses `id` and nests provider settings under
 `config`:
 
 ```yaml
 targets:
-  - label: azure-base
+  - id: azure-base
     provider: azure
+    runtime: host
     config:
-      endpoint: ${{ AZURE_OPENAI_ENDPOINT }}
-      api_key: ${{ AZURE_OPENAI_API_KEY }}
-      model: ${{ AZURE_DEPLOYMENT_NAME }}
+      endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
+      api_key: "{{ env.AZURE_OPENAI_API_KEY }}"
+      model: "{{ env.AZURE_DEPLOYMENT_NAME }}"
 ```
 
 ### Migration Steps
 
 - Eval YAML: `execution.target` -> top-level `target`.
 - Eval YAML: `execution.targets` -> top-level `targets`.
-- Eval target object `name` -> `label`.
-- If an eval-local target object has provider configuration, include a
-  `label`; use `extends` to derive from a base target.
-- Targets file: rename `name` to `label` and move provider-specific settings
-  into `config`.
-- Keep AgentV target extensions such as `grader_target`, `use_target`,
+- Eval target object `name` -> `id`.
+- If an eval-local target object has provider configuration, include a concrete
+  `provider`; `id` is AgentV's stable target identity and `provider` names the
+  backend or adapter kind.
+- Targets file: rename `name` to `id`, move provider-specific settings into
+  `config`, and rewrite environment references from `${{ NAME }}` to
+  `{{ env.NAME }}`.
+- Remove `use_target`; current authored target definitions must resolve to
+  concrete provider objects.
+- Keep supported AgentV target extensions such as `grader_target`,
   `fallback_targets`, `workers`, and `batch_requests` as top-level fields on
   target objects.
 
@@ -790,10 +903,12 @@ targets:
 ```bash
 bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml
 bun apps/cli/src/cli.ts validate .agentv/targets.yaml
-rg -n "execution:|name:|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml
+rg -n "execution:|name:|use_target:|\\$\\{\\{|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml
 ```
 
-Inspect `name:` matches manually because grader names still exist.
+Inspect `name:`, top-level provider fields, and `${{ ... }}` matches manually
+because grader names, non-target YAML, and historical examples can still use
+similar strings legitimately.
 
 ### Compatibility Notes
 
@@ -984,9 +1099,10 @@ rg -n "include:|tests:" path/to/evals
 
 ### Compatibility Notes
 
-`eval_cases` remains a deprecated alias in the current schema, but migrated
-YAML should use `tests`. The current convention is that runnable suites use
-`*.eval.yaml`; reusable raw case files commonly use `*.cases.yaml` or JSONL.
+`eval_cases` and `evalcases` have been removed from authored eval YAML. Migrate
+them to `tests` before validating or running the suite. The current convention is
+that runnable suites use `*.eval.yaml`; reusable raw case files commonly use
+`*.cases.yaml` or JSONL.
 
 ## Result Artifact Path Changes Are Not Eval YAML Migrations