agentv 5.3.0-next.1 → 5.3.2-next.1

package/dist/skills/agentv-bench/references/subagent-pipeline.md

@@ -30,10 +30,12 @@ specific target, set `subagent_mode_allowed: false` in `.agentv/targets.yaml`:
 ```yaml
 # .agentv/targets.yaml
 targets:
-  - name: my-target
+  - id: my-target
     provider: openai
-    model: ${{ OPENAI_MODEL }}
-    api_key: ${{ OPENAI_API_KEY }}
+    runtime: host
+    config:
+      model: "{{ env.OPENAI_MODEL }}"
+      api_key: "{{ env.OPENAI_API_KEY }}"
     subagent_mode_allowed: false  # forces CLI invocation instead of executor subagent
 ```
 
@@ -43,8 +45,8 @@ even in subagent mode.
 ## CLI Targets: Single Command
 
 For evals with CLI targets, `pipeline run` handles input extraction, target invocation, and
-code grading in one step. When `--out` is omitted, the output directory defaults to
-`.agentv/results/default/<timestamp>` (same convention as `agentv eval`):
+code grading in one step. When `--out` is omitted, the output directory uses the
+canonical `.agentv/results/<run_id>` convention:
 
 ```bash
 # Extract inputs and invoke all CLI targets in parallel:
@@ -70,7 +72,7 @@ opted out via `subagent_mode_allowed: false` in `.agentv/targets.yaml`), fall ba
 ### Step 1: Extract inputs
 
 ```bash
-# Defaults to .agentv/results/default/<timestamp>
+# Defaults to .agentv/results/<run_id>
 agentv pipeline input evals/repro.eval.yaml
 ```
 
@@ -117,7 +119,7 @@ LLM grading → merge and validate).
 Use individual commands when you need control over each step with CLI targets:
 
 ```bash
-# Step 1: Extract inputs (defaults to .agentv/results/default/<timestamp>)
+# Step 1: Extract inputs (defaults to .agentv/results/<run_id>)
 agentv pipeline input evals/repro.eval.yaml
 
 # Step 2: run_tests.py invokes CLI targets (or use pipeline run instead)
@@ -127,7 +129,7 @@ agentv pipeline grade <run-dir>
 
 # Step 4: Subagent does LLM grading, writes results to llm_grader_results/<name>.json per test
 
-# Step 5: Merge scores (writes index.jsonl with full scores[] for dashboard)
+# Step 5: Merge scores into the canonical run bundle
 agentv pipeline bench <run-dir>
 
 # Step 6: Validate
@@ -152,22 +154,22 @@ The agent reads `llm_graders/<name>.json` for each test, grades the response usi
 
 ## Pipeline Bench and Dashboard
 
-`pipeline bench` merges LLM scores into `index.jsonl` with a full `scores[]` array per entry,
-matching the CLI-mode schema. The web dashboard (`agentv results serve`) reads this format
-directly — no separate conversion script is needed. Run `agentv results validate <run-dir>`
-to verify compatibility.
+`pipeline bench` merges LLM scores into the canonical run bundle, including
+`.internal/index.jsonl`, `summary.json`, and per-attempt `grading.json` with
+`component_results`. The web dashboard reads this format directly — no separate
+conversion script is needed. Run `agentv results validate <run-dir>` to verify
+compatibility.
 
 ## Output Structure
 
-The path hierarchy mirrors the CLI mode: `<evalset-name>` comes from the `name` field in
-the eval.yaml. The target is recorded in `manifest.json` — one run = one target.
+The path hierarchy mirrors CLI mode. The target is recorded in run metadata.
 
 ```
-.agentv/results/<experiment>/<timestamp>/
+.agentv/results/<run_id>/
 ├── manifest.json                    ← eval metadata, target, test_ids
-├── index.jsonl                      ← per-test scores
-├── summary.json                   ← aggregate statistics
-└── <evalset-name>/                  ← eval.yaml "name" field, or eval file basename if absent (same as CLI mode)
+├── summary.json                      ← aggregate statistics
+├── .internal/index.jsonl             ← canonical row index
+└── <result-dir>/sample-1/grading.json
     └── <test-id>/                   ← test case id
         ├── input.json               ← test input text + messages
         ├── invoke.json              ← target command or agent instructions

package/dist/skills/agentv-eval-migrations/SKILL.md

@@ -16,6 +16,19 @@ v4.42.4-era shape, current shape, migration steps, verification commands, and
 compatibility notes for each major breaking authoring change. Then compare the
 eval file against the current portable contract:
 
+- Treat migrations as rule-based rewrites first. Do not rely on a manual
+  "looks current" pass when the old field has a direct current replacement.
+- Rewrite authored prompt data into top-level `prompts` plus `tests[].vars`;
+  keep `input` only for documented raw-case import compatibility.
+- Move sibling `expected_output` to `vars.expected_output`, then add or keep an
+  explicit assertion that consumes it, usually `llm-rubric` with
+  `{{ expected_output }}`.
+- Rewrite target identity to `id`, keep `provider` for the backend/adapter kind,
+  put provider settings under `config`, and rewrite env references to
+  `{{ env.NAME }}`.
+- Remove `use_target`, `eval_cases`, `evalcases`, `providerPromptMap`, and
+  `provider_prompt_map` from authored eval/config YAML. Current AgentV rejects
+  these fields, so do not preserve them as compatibility aliases.
 - Keep committed eval YAML portable: prompts, cases, assertions, workspace
   templates, repos, hooks, env checks, Docker preflight/container config, and
   `workspace.scope`.

package/dist/skills/agentv-eval-migrations/references/breaking-changes.md

@@ -23,7 +23,7 @@ For a v4.42.4-era eval:
 
 1. Move top-level `execution.target` to top-level `target`.
 2. Move top-level `execution.targets` to top-level `targets`, and rename target
-   object `name` to `label`.
+   object `name` to `id`.
 3. Rename suite/test/turn `assertions` to `assert`.
 4. If a test uses `tests[].execution.assertions` or
    `tests[].execution.evaluators`, rename that nested list to
@@ -56,7 +56,14 @@ For a v4.42.4-era eval:
     assertions where there is a direct mapping.
 18. Keep raw cases under `tests` / `tests: file://...`; run full eval suites
     directly with CLI multi-file selection and tags.
-19. Validate with `bun apps/cli/src/cli.ts validate <eval-file>`.
+19. Move authored prompt input from top-level or `tests[].input` into
+    `prompts` plus `tests[].vars.input`.
+20. Move authored reference answers from sibling `expected_output` into
+    `vars.expected_output`; add or keep an explicit assertion that consumes it.
+21. Rewrite target env interpolation from `${{ NAME }}` to `{{ env.NAME }}`.
+22. Remove `use_target`, `eval_cases`, `evalcases`, `providerPromptMap`, and
+    `provider_prompt_map`; current AgentV rejects them.
+23. Validate with `bun apps/cli/src/cli.ts validate <eval-file>`.
 
 ## Assertions Renamed To `assert`
 
@@ -67,7 +74,7 @@ v4.42.4 docs and schema used `assertions` for suite-level and per-test graders:
 ```yaml
 assertions:
   - name: correctness
-    type: llm-grader
+    type: llm-rubric
     prompt: ./graders/correctness.md
 
 tests:
@@ -532,11 +539,13 @@ experiment:
 
 ### Current Shape
 
-Current schema accepts top-level `experiment` only as a non-empty string run
-grouping label. Current docs also support promptfoo-shaped `tags.experiment`.
+Current schema rejects top-level `experiment` in authored eval YAML. Use the
+promptfoo-shaped `tags.experiment` key when the label belongs in the eval file,
+or pass CLI `--experiment` when the label is run-time context.
 
 ```yaml
-experiment: with-skills
+tags:
+  experiment: with-skills
 target: codex
 timeout_seconds: 600
 evaluate_options:
@@ -545,14 +554,6 @@ evaluate_options:
     strategy: pass_any
 ```
 
-or:
-
-```yaml
-tags:
-  experiment: with-skills
-target: codex
-```
-
 ### Migration Steps
 
 - If `experiment` is an object, move runtime fields out:
@@ -561,7 +562,8 @@ target: codex
   - budget -> `evaluate_options.budget_usd`
   - timeout -> top-level `timeout_seconds`
   - threshold -> top-level `threshold`
-- Keep only a string label in `experiment`, or use `tags.experiment`.
+- Move string experiment labels to `tags.experiment`, or supply them at run time
+  with CLI `--experiment`.
 
 ### Verification
 
@@ -570,6 +572,8 @@ bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml
 rg -n "^experiment:" path/to/evals
 ```
 
+Any match under authored eval YAML should be removed or migrated.
+
 ### Compatibility Notes
 
 Do not describe a v4.42.4 eval as if `experiment:` was already the main runtime
@@ -844,54 +848,66 @@ targets:
 ### Current Shape
 
 Current eval YAML uses top-level `target` or `targets`. Target references use
-`label`; `id` is reserved for provider/backend identity when needed:
+the target `id`; `provider` names the backend or adapter kind:
 
 ```yaml
 target: azure-base
 
 targets:
-  - label: with-skills
-    use_target: default
+  - id: with-skills
+    provider: codex-cli
+    runtime: host
+    config:
+      command: ["codex", "exec", "--json"]
     hooks:
       before_each:
         command: ["setup-plugins.sh", "skills"]
 ```
 
-Current `.agentv/targets.yaml` uses `label` and nests provider settings under
+Current `.agentv/targets.yaml` uses `id` and nests provider settings under
 `config`:
 
 ```yaml
 targets:
-  - label: azure-base
+  - id: azure-base
     provider: azure
+    runtime: host
     config:
-      endpoint: ${{ AZURE_OPENAI_ENDPOINT }}
-      api_key: ${{ AZURE_OPENAI_API_KEY }}
-      model: ${{ AZURE_DEPLOYMENT_NAME }}
+      endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
+      api_key: "{{ env.AZURE_OPENAI_API_KEY }}"
+      model: "{{ env.AZURE_DEPLOYMENT_NAME }}"
 ```
 
 ### Migration Steps
 
 - Eval YAML: `execution.target` -> top-level `target`.
 - Eval YAML: `execution.targets` -> top-level `targets`.
-- Eval target object `name` -> `label`.
-- If an eval-local target object has provider configuration, include a
-  `label`; use `extends` to derive from a base target.
-- Targets file: rename `name` to `label` and move provider-specific settings
-  into `config`.
-- Keep AgentV target extensions such as `grader_target`, `use_target`,
-  `fallback_targets`, `workers`, and `batch_requests` as top-level fields on
-  target objects.
+- Eval target object `name` -> `id`.
+- If an eval-local target object has provider configuration, include a concrete
+  `provider`; `id` is AgentV's stable target identity and `provider` names the
+  backend or adapter kind.
+- Targets file: rename `name` to `id`, move provider-specific settings into
+  `config`, and rewrite environment references from `${{ NAME }}` to
+  `{{ env.NAME }}`.
+- Remove `use_target`; current authored target definitions must resolve to
+  concrete provider objects.
+- Keep supported AgentV target extensions such as `grader_target`,
+  `fallback_targets`, and `workers` as top-level fields on target objects.
+- Remove runner-level request batching from migrated configs. CLI providers are
+  invoked once per eval case; throughput batching belongs inside provider
+  adapters or CLIs without eval YAML batch configuration.
 
 ### Verification
 
 ```bash
 bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml
 bun apps/cli/src/cli.ts validate .agentv/targets.yaml
-rg -n "execution:|name:|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml
+rg -n "execution:|name:|use_target:|\\$\\{\\{|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml
 ```
 
-Inspect `name:` matches manually because grader names still exist.
+Inspect `name:`, top-level provider fields, and `${{ ... }}` matches manually
+because grader names, non-target YAML, and historical examples can still use
+similar strings legitimately.
 
 ### Compatibility Notes
 
@@ -944,7 +960,7 @@ assert:
 - `type: g-eval` -> `type: llm-rubric`.
 - `type: code-grader`, `code-judge`, `code_grader`, or `code_judge` ->
   `type: script`.
-- `type: llm_judge` or `llm_grader` -> `type: llm-grader`.
+- `type: llm_judge` or `llm_grader` -> `type: llm-rubric`.
 - Convert multi-word snake_case deterministic types to kebab-case:
   `is_json` -> `is-json`, `contains_all` -> `contains-all`,
   `starts_with` -> `starts-with`, and so on.
@@ -973,9 +989,9 @@ v4.42.4 LLM grader docs allowed both:
 
 ```yaml
 assertions:
-  - type: llm-grader
+  - type: llm-rubric
     prompt: ./graders/correctness.md
-  - type: llm-grader
+  - type: llm-rubric
     prompt: file://graders/correctness.md
 ```
 
@@ -1082,9 +1098,10 @@ rg -n "include:|tests:" path/to/evals
 
 ### Compatibility Notes
 
-`eval_cases` remains a deprecated alias in the current schema, but migrated
-YAML should use `tests`. The current convention is that runnable suites use
-`*.eval.yaml`; reusable raw case files commonly use `*.cases.yaml` or JSONL.
+`eval_cases` and `evalcases` have been removed from authored eval YAML. Migrate
+them to `tests` before validating or running the suite. The current convention is
+that runnable suites use `*.eval.yaml`; reusable raw case files commonly use
+`*.cases.yaml` or JSONL.
 
 ## Result Artifact Path Changes Are Not Eval YAML Migrations