agentv 5.3.0-next.1 → 5.3.1-next.1

package/dist/skills/agentv-bench/references/subagent-pipeline.md

@@ -30,10 +30,12 @@ specific target, set `subagent_mode_allowed: false` in `.agentv/targets.yaml`:
 ```yaml
 # .agentv/targets.yaml
 targets:
-  - name: my-target
+  - id: my-target
     provider: openai
-    model: ${{ OPENAI_MODEL }}
-    api_key: ${{ OPENAI_API_KEY }}
+    runtime: host
+    config:
+      model: "{{ env.OPENAI_MODEL }}"
+      api_key: "{{ env.OPENAI_API_KEY }}"
     subagent_mode_allowed: false  # forces CLI invocation instead of executor subagent
 ```
 
@@ -43,8 +45,8 @@ even in subagent mode.
 ## CLI Targets: Single Command
 
 For evals with CLI targets, `pipeline run` handles input extraction, target invocation, and
-code grading in one step. When `--out` is omitted, the output directory defaults to
-`.agentv/results/default/<timestamp>` (same convention as `agentv eval`):
+code grading in one step. When `--out` is omitted, the output directory uses the
+canonical `.agentv/results/<run_id>` convention:
 
 ```bash
 # Extract inputs and invoke all CLI targets in parallel:
@@ -70,7 +72,7 @@ opted out via `subagent_mode_allowed: false` in `.agentv/targets.yaml`), fall ba
 ### Step 1: Extract inputs
 
 ```bash
-# Defaults to .agentv/results/default/<timestamp>
+# Defaults to .agentv/results/<run_id>
 agentv pipeline input evals/repro.eval.yaml
 ```
 
@@ -117,7 +119,7 @@ LLM grading → merge and validate).
 Use individual commands when you need control over each step with CLI targets:
 
 ```bash
-# Step 1: Extract inputs (defaults to .agentv/results/default/<timestamp>)
+# Step 1: Extract inputs (defaults to .agentv/results/<run_id>)
 agentv pipeline input evals/repro.eval.yaml
 
 # Step 2: run_tests.py invokes CLI targets (or use pipeline run instead)
@@ -127,7 +129,7 @@ agentv pipeline grade <run-dir>
 
 # Step 4: Subagent does LLM grading, writes results to llm_grader_results/<name>.json per test
 
-# Step 5: Merge scores (writes index.jsonl with full scores[] for dashboard)
+# Step 5: Merge scores into the canonical run bundle
 agentv pipeline bench <run-dir>
 
 # Step 6: Validate
@@ -152,22 +154,22 @@ The agent reads `llm_graders/<name>.json` for each test, grades the response usi
 
 ## Pipeline Bench and Dashboard
 
-`pipeline bench` merges LLM scores into `index.jsonl` with a full `scores[]` array per entry,
-matching the CLI-mode schema. The web dashboard (`agentv results serve`) reads this format
-directly — no separate conversion script is needed. Run `agentv results validate <run-dir>`
-to verify compatibility.
+`pipeline bench` merges LLM scores into the canonical run bundle, including
+`.internal/index.jsonl`, `summary.json`, and per-attempt `grading.json` with
+`component_results`. The web dashboard reads this format directly — no separate
+conversion script is needed. Run `agentv results validate <run-dir>` to verify
+compatibility.
 
 ## Output Structure
 
-The path hierarchy mirrors the CLI mode: `<evalset-name>` comes from the `name` field in
-the eval.yaml. The target is recorded in `manifest.json` — one run = one target.
+The path hierarchy mirrors CLI mode. The target is recorded in run metadata.
 
 ```
-.agentv/results/<experiment>/<timestamp>/
+.agentv/results/<run_id>/
 ├── manifest.json                    ← eval metadata, target, test_ids
-├── index.jsonl                      ← per-test scores
-├── summary.json                   ← aggregate statistics
-└── <evalset-name>/                  ← eval.yaml "name" field, or eval file basename if absent (same as CLI mode)
+├── summary.json                      ← aggregate statistics
+├── .internal/index.jsonl             ← canonical row index
+└── <result-dir>/sample-1/grading.json
     └── <test-id>/                   ← test case id
         ├── input.json               ← test input text + messages
         ├── invoke.json              ← target command or agent instructions

package/dist/skills/agentv-eval-migrations/SKILL.md

@@ -16,6 +16,19 @@ v4.42.4-era shape, current shape, migration steps, verification commands, and
 compatibility notes for each major breaking authoring change. Then compare the
 eval file against the current portable contract:
 
+- Treat migrations as rule-based rewrites first. Do not rely on a manual
+  "looks current" pass when the old field has a direct current replacement.
+- Rewrite authored prompt data into top-level `prompts` plus `tests[].vars`;
+  keep `input` only for documented raw-case import compatibility.
+- Move sibling `expected_output` to `vars.expected_output`, then add or keep an
+  explicit assertion that consumes it, usually `llm-rubric` with
+  `{{ expected_output }}`.
+- Rewrite target identity to `id`, keep `provider` for the backend/adapter kind,
+  put provider settings under `config`, and rewrite env references to
+  `{{ env.NAME }}`.
+- Remove `use_target`, `eval_cases`, `evalcases`, `providerPromptMap`, and
+  `provider_prompt_map` from authored eval/config YAML. Current AgentV rejects
+  these fields, so do not preserve them as compatibility aliases.
 - Keep committed eval YAML portable: prompts, cases, assertions, workspace
   templates, repos, hooks, env checks, Docker preflight/container config, and
   `workspace.scope`.

package/dist/skills/agentv-eval-migrations/references/breaking-changes.md

@@ -23,7 +23,7 @@ For a v4.42.4-era eval:
 
 1. Move top-level `execution.target` to top-level `target`.
 2. Move top-level `execution.targets` to top-level `targets`, and rename target
-   object `name` to `label`.
+   object `name` to `id`.
 3. Rename suite/test/turn `assertions` to `assert`.
 4. If a test uses `tests[].execution.assertions` or
    `tests[].execution.evaluators`, rename that nested list to
@@ -56,7 +56,14 @@ For a v4.42.4-era eval:
     assertions where there is a direct mapping.
 18. Keep raw cases under `tests` / `tests: file://...`; run full eval suites
     directly with CLI multi-file selection and tags.
-19. Validate with `bun apps/cli/src/cli.ts validate <eval-file>`.
+19. Move authored prompt input from top-level or `tests[].input` into
+    `prompts` plus `tests[].vars.input`.
+20. Move authored reference answers from sibling `expected_output` into
+    `vars.expected_output`; add or keep an explicit assertion that consumes it.
+21. Rewrite target env interpolation from `${{ NAME }}` to `{{ env.NAME }}`.
+22. Remove `use_target`, `eval_cases`, `evalcases`, `providerPromptMap`, and
+    `provider_prompt_map`; current AgentV rejects them.
+23. Validate with `bun apps/cli/src/cli.ts validate <eval-file>`.
 
 ## Assertions Renamed To `assert`
 
@@ -844,42 +851,50 @@ targets:
 ### Current Shape
 
 Current eval YAML uses top-level `target` or `targets`. Target references use
-`label`; `id` is reserved for provider/backend identity when needed:
+the target `id`; `provider` names the backend or adapter kind:
 
 ```yaml
 target: azure-base
 
 targets:
-  - label: with-skills
-    use_target: default
+  - id: with-skills
+    provider: codex-cli
+    runtime: host
+    config:
+      command: ["codex", "exec", "--json"]
     hooks:
       before_each:
         command: ["setup-plugins.sh", "skills"]
 ```
 
-Current `.agentv/targets.yaml` uses `label` and nests provider settings under
+Current `.agentv/targets.yaml` uses `id` and nests provider settings under
 `config`:
 
 ```yaml
 targets:
-  - label: azure-base
+  - id: azure-base
     provider: azure
+    runtime: host
     config:
-      endpoint: ${{ AZURE_OPENAI_ENDPOINT }}
-      api_key: ${{ AZURE_OPENAI_API_KEY }}
-      model: ${{ AZURE_DEPLOYMENT_NAME }}
+      endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
+      api_key: "{{ env.AZURE_OPENAI_API_KEY }}"
+      model: "{{ env.AZURE_DEPLOYMENT_NAME }}"
 ```
 
 ### Migration Steps
 
 - Eval YAML: `execution.target` -> top-level `target`.
 - Eval YAML: `execution.targets` -> top-level `targets`.
-- Eval target object `name` -> `label`.
-- If an eval-local target object has provider configuration, include a
-  `label`; use `extends` to derive from a base target.
-- Targets file: rename `name` to `label` and move provider-specific settings
-  into `config`.
-- Keep AgentV target extensions such as `grader_target`, `use_target`,
+- Eval target object `name` -> `id`.
+- If an eval-local target object has provider configuration, include a concrete
+  `provider`; `id` is AgentV's stable target identity and `provider` names the
+  backend or adapter kind.
+- Targets file: rename `name` to `id`, move provider-specific settings into
+  `config`, and rewrite environment references from `${{ NAME }}` to
+  `{{ env.NAME }}`.
+- Remove `use_target`; current authored target definitions must resolve to
+  concrete provider objects.
+- Keep supported AgentV target extensions such as `grader_target`,
   `fallback_targets`, `workers`, and `batch_requests` as top-level fields on
   target objects.
 
@@ -888,10 +903,12 @@ targets:
 ```bash
 bun apps/cli/src/cli.ts validate path/to/eval.eval.yaml
 bun apps/cli/src/cli.ts validate .agentv/targets.yaml
-rg -n "execution:|name:|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml
+rg -n "execution:|name:|use_target:|\\$\\{\\{|endpoint:|api_key:|model:" path/to/evals .agentv/targets.yaml
 ```
 
-Inspect `name:` matches manually because grader names still exist.
+Inspect `name:`, top-level provider fields, and `${{ ... }}` matches manually
+because grader names, non-target YAML, and historical examples can still use
+similar strings legitimately.
 
 ### Compatibility Notes
 
@@ -1082,9 +1099,10 @@ rg -n "include:|tests:" path/to/evals
 
 ### Compatibility Notes
 
-`eval_cases` remains a deprecated alias in the current schema, but migrated
-YAML should use `tests`. The current convention is that runnable suites use
-`*.eval.yaml`; reusable raw case files commonly use `*.cases.yaml` or JSONL.
+`eval_cases` and `evalcases` have been removed from authored eval YAML. Migrate
+them to `tests` before validating or running the suite. The current convention is
+that runnable suites use `*.eval.yaml`; reusable raw case files commonly use
+`*.cases.yaml` or JSONL.
 
 ## Result Artifact Path Changes Are Not Eval YAML Migrations
 

package/dist/skills/agentv-eval-writer/SKILL.md

@@ -40,7 +40,7 @@ Use `@agentv/sdk` for TypeScript helper imports. Do not use `@agentv/eval` for n
 
 - Put grading criteria in `assert`, not in test-level `criteria`. Plain assertion strings become an `llm-rubric` grader.
 - Prefer plain assertion strings for semantic checks when the default rubric grader can judge them. Use `type: llm-rubric` for structured criteria, custom prompts, custom grader targets, or assertion-level transforms, and `type: script` when grading must execute code.
-- Write `expected_output` as a golden/reference answer the target could have produced. Do not write criteria, scoring instructions, or "the agent should..." rubric prose there.
+- Put reference answers in `tests[].vars.expected_output` or `default_test.vars.expected_output`, and consume them with an explicit assertion such as `type: llm-rubric` with `value: "Matches the reference answer: {{ expected_output }}"`. Do not write criteria, scoring instructions, or "the agent should..." rubric prose as the reference answer.
 - For historical or repo-state evals, materialize the repo through a pinned `environment` setup recipe. Mentioning a SHA only in prompt prose is not enough because the agent needs an actual checkout to inspect.
 
 ## Evaluation Types
@@ -61,7 +61,12 @@ agentv convert evals.json
 agentv eval evals.json
 ```
 
-The converter maps `prompt` → `input`, `expected_output` → `expected_output`, and Agent Skills `assertions` → AgentV `assert` (`llm-rubric` checks), and resolves `files[]` paths. The generated YAML includes TODO comments for AgentV features to add (environment setup, script graders, rubrics, required gates).
+The converter maps Agent Skills prompt text into AgentV prompt/vars data,
+promotes Agent Skills `expected_output` into explicit `llm-rubric` criteria or
+`vars.expected_output` when it is true reference data, and maps Agent Skills
+`assertions` to AgentV `assert` (`llm-rubric` checks). The generated YAML
+includes TODO comments for AgentV features to add (environment setup, script
+graders, rubrics, required gates).
 
 After converting, enhance the YAML with AgentV-specific capabilities shown below.
 
@@ -103,12 +108,15 @@ prompts:
 
 tests:
   - id: multi-turn-context
-    expected_output: "Your name is Alice."
+    vars:
+      expected_output: "Your name is Alice."
     assert:
+      - type: llm-rubric
+        value: "Matches the reference answer: {{ expected_output }}"
       - Correctly recalls the user's name from earlier in the conversation
 ```
 
-**Guidelines:** preserve exact wording in `expected_output`; aim for 5–15 tests per transcript; pick exchanges that test different capabilities.
+**Guidelines:** preserve exact wording in `vars.expected_output`; aim for 5–15 tests per transcript; pick exchanges that test different capabilities.
 
 ## Quick Start
 
@@ -123,8 +131,10 @@ tests:
   - id: greeting
     vars:
       prompt: "Say hello"
-    expected_output: "Hello! How can I help you?"
+      expected_output: "Hello! How can I help you?"
     assert:
+      - type: llm-rubric
+        value: "Matches the reference answer: {{ expected_output }}"
       - Greeting is friendly and warm
       - Offers to help
 ```
@@ -140,7 +150,7 @@ tests:
 |-------|----------|-------------|
 | `id` | yes | Unique identifier |
 | `vars` | yes when the prompt needs row data | Prompt-template variables for this row |
-| `expected_output` | no | Gold-standard reference answer (string shorthand or full message array) |
+| `vars.expected_output` | no | Conventional reference-answer var consumed by explicit graders |
 | `assert` | yes | Graders: deterministic checks, `llm-rubric` checks, script graders, or plain string rubric criteria |
 | `execution` | no | Per-case grader/default overrides such as `skip_defaults`; target selection belongs in top-level `target` or CLI `--target` |
 | `environment` | no | Per-case coding-agent testbed config (overrides suite-level) |
@@ -153,7 +163,7 @@ Use top-level `prompts` plus `tests[].vars` for the Promptfoo-compatible canonic
 input shape. Shared data defaults belong in `default_test.vars`; per-test
 `vars` override those defaults by key. AgentV renders every prompt with each
 test's merged vars, then expands the run across prompts, targets, tests, and
-repeat attempts.
+repeat samples.
 
 ```yaml
 description: Prompt matrix example
@@ -176,15 +186,19 @@ tests:
   - id: password-reset
     vars:
       question: How do I reset my password?
-    expected_output: Password reset guidance
+      expected_output: Password reset guidance
     assert:
+      - type: llm-rubric
+        value: "Matches the reference answer: {{ expected_output }}"
       - Gives correct password reset guidance
   - id: admin-access
     vars:
       audience: admins
       question: How do I revoke a user's access?
-    expected_output: Access revocation guidance
+      expected_output: Access revocation guidance
     assert:
+      - type: llm-rubric
+        value: "Matches the reference answer: {{ expected_output }}"
       - Gives safe access revocation guidance
 ```
 
@@ -204,7 +218,7 @@ then render those vars from the prompt template next to the input.
 **Shorthand forms:**
 - Prompt entries can be strings, message arrays, file references, or prompt objects.
 - Put chat/system/user messages in `prompts`, not in `tests[].input`.
-- `expected_output` (string/object) expands to `[{role: "assistant", content: ...}]`
+- `vars.expected_output` is a conventional reference-answer variable; explicit assertions decide how to grade it
 - Use these canonical field names on disk; keep the wire format `snake_case`
 
 **Message format:** `{role, content}` where role is `system`, `user`, `assistant`, or `tool`
@@ -356,12 +370,14 @@ tests:
         verification guidance was added.
 
         Decide what durable repo change should be made and explain why.
-    expected_output: |
-      The durable repo change is to update .agents/verification.md with the
-      reusable verification workflow lessons. AGENTS.md already routes this
-      class of work to .agents/verification.md, so no extra AGENTS.md edit is
-      needed unless that routing is missing.
+      expected_output: |
+        The durable repo change is to update .agents/verification.md with the
+        reusable verification workflow lessons. AGENTS.md already routes this
+        class of work to .agents/verification.md, so no extra AGENTS.md edit is
+        needed unless that routing is missing.
     assert:
+      - type: llm-rubric
+        value: "Matches the reference answer: {{ expected_output }}"
       - The answer recommends updating .agents/verification.md rather than leaving the learning only in PR comments or private evidence.
       - The answer uses the pinned ./agentv checkout to verify the AGENTS.md routing.
       - The answer preserves the historical commit SHA as context.
@@ -387,9 +403,9 @@ tests:
         value: "fix"
 ```
 
-`expected_output` is passive reference data. It is available to graders through
-`{{expected_output}}` and the script stdin payload, but it does not create an
-implicit LLM grading call by itself.
+`vars.expected_output` is passive reference data. It is available to graders
+through `{{ expected_output }}` and the script stdin payload, but it does not
+create an implicit LLM grading call by itself.
 
 **Common mistake:** putting rubric prose in `expected_output` instead of an
 assertion:
@@ -402,7 +418,7 @@ tests:
   - id: bad-example
     vars:
       prompt: "What is 2+2?"
-    expected_output: The assistant should explain why the answer is 4. # reference answer field, not a grader
+      expected_output: The assistant should explain why the answer is 4. # reference answer var, not a grader
 ```
 
 Write this as:
@@ -415,8 +431,10 @@ tests:
   - id: good-example
     vars:
       prompt: "What is 2+2?"
-    expected_output: "4"
+      expected_output: "4"
     assert:
+      - type: llm-rubric
+        value: "Matches the reference answer: {{ expected_output }}"
       - The answer is 4 and explains the arithmetic briefly
 ```
 
@@ -438,7 +456,7 @@ assert:
         weight: 5.0
 ```
 
-If a required grader scores below its threshold, the overall verdict is forced to `fail`.
+If a required grader scores below its threshold, the overall case status is forced to `fail`.
 
 ## Environment Setup/Teardown
 
@@ -523,9 +541,10 @@ Configure via the `assert` array. Multiple graders produce a weighted average sc
   cwd: ./scripts          # optional working directory
   target: {}              # optional: enable LLM target proxy (max_calls: 50)
 ```
-Contract: stdin JSON -> stdout JSON `{score, assertions: [{text, passed, evidence?}], reasoning}`
+Contract: stdin JSON -> stdout JSON `{pass, score, reason, checks?: [{text, pass, score?, reason}]}`
 Raw stdin uses snake_case and includes: `input`, `expected_output`, `output` (final answer string), `messages`, `trace`, `trace_summary`, `token_usage`, `cost_usd`, `duration_ms`, `start_time`, `end_time`, `file_changes`, `workspace_path`, `config`
 SDK handlers receive the same payload in camelCase: `expectedOutput`, `traceSummary`, `tokenUsage`, `costUsd`, `durationMs`, `startTime`, `endTime`, `fileChanges`, `workspacePath`.
+`checks` is an SDK/script convenience shape; public `grading.json` artifacts normalize checks into recursive `component_results`.
 When an environment prepares a workspace directory, `workspace_path` is the absolute path to that directory (also available as `AGENTV_WORKSPACE_PATH` env var). Use this for functional grading (e.g., running `npm test` in the prepared workdir).
 For deterministic workspace checks that fit normal Vitest `expect(...)` tests, prefer a plain verifier file and the built-in adapter:
 ```yaml
@@ -567,7 +586,7 @@ Variables: `{{criteria}}`, `{{input}}`, `{{expected_output}}`, `{{output}}`, `{{
       type: llm-rubric
       weight: 0.7
 ```
-Use `assert-set` for Promptfoo-aligned assertion grouping. Without `threshold`, the set passes only when every nonzero-weight child assertion passes. With `threshold`, the weighted aggregate score determines the set verdict. Parent `config` is inherited by children, and child `config` keys override parent keys. Do not use `type: composite`; AgentV rejects it.
+Use `assert-set` for Promptfoo-aligned assertion grouping. Without `threshold`, the set passes only when every nonzero-weight child assertion passes. With `threshold`, the weighted aggregate score determines the set pass/fail result. Parent `config` is inherited by children, and child `config` keys override parent keys. Do not use `type: composite`; AgentV rejects it.
 
 ### Skill And Trajectory Assertions
 ```yaml
@@ -722,24 +741,24 @@ agentv eval assert <grader-name> --agent-output "..." --agent-input "..."
 agentv import claude --session-id <uuid>
 
 # Re-run only execution errors from a previous run
-agentv eval <file.yaml> --retry-errors .agentv/results/default/<timestamp>/index.jsonl
+agentv eval <file.yaml> --retry-errors .agentv/results/<run_id>/.internal/index.jsonl
 
 # Validate eval file
 agentv validate <file.yaml>
 
 # Compare completed runs
 agentv results compare \
-  .agentv/results/default/<baseline-timestamp>/index.jsonl \
-  .agentv/results/default/<candidate-timestamp>/index.jsonl
+  .agentv/results/<baseline-run-id>/.internal/index.jsonl \
+  .agentv/results/<candidate-run-id>/.internal/index.jsonl
 agentv results combine \
-  .agentv/results/default/<baseline-timestamp> \
-  .agentv/results/default/<candidate-timestamp> \
-  .agentv/results/default/<third-target-timestamp> \
-  --output .agentv/results/default/combined
-agentv results compare .agentv/results/default/combined/index.jsonl
+  .agentv/results/<baseline-run-id> \
+  .agentv/results/<candidate-run-id> \
+  .agentv/results/<third-target-run-id> \
+  --output .agentv/results/combined
+agentv results compare .agentv/results/combined/.internal/index.jsonl
 agentv results compare \
-  .agentv/results/default/<baseline-timestamp>/index.jsonl \
-  .agentv/results/default/<candidate-timestamp>/index.jsonl \
+  .agentv/results/<baseline-run-id>/.internal/index.jsonl \
+  .agentv/results/<candidate-run-id>/.internal/index.jsonl \
   --json
 
 # Author assertions directly in the eval file
@@ -755,32 +774,36 @@ Use `@agentv/sdk` as the public lightweight SDK package for TypeScript/JavaScrip
 
 ### YAML-aligned eval authoring
 ```typescript
-import { defineEval, graders } from '@agentv/sdk';
+// evals/helper-suite.eval.ts
+import { graders, type EvalConfig } from '@agentv/sdk';
 
-export default defineEval({
+const config: EvalConfig = {
   name: 'helper-suite',
   target: 'default',
-  // The SDK helper lowers this to evaluate_options.repeat in generated YAML.
+  // TypeScript config loading lowers this to evaluate_options.repeat.
   repeat: {
     count: 3,
     strategy: 'pass_any',
     earlyExit: false,
   },
   threshold: 0.8,
+  prompts: ['{{ task }}'],
   tests: [
     {
       id: 'json-answer',
-      input: 'Return a JSON answer with a status field.',
+      vars: { task: 'Return a JSON answer with a status field.' },
       assert: [
         graders.json({ name: 'valid-json', required: true }),
         graders.regex(/"status"\s*:/, { name: 'status-key' }),
       ],
     },
   ],
-});
+};
+
+export default config;
 ```
 
-The `graders` catalog returns ordinary `assert` entries such as `type: is-json`, `type: regex`, `type: llm-rubric`, and `type: script`. `defineEval()` lowers camelCase TypeScript fields such as `expectedOutput`, `inputFiles`, and `maxSteps` to canonical snake_case YAML/runtime keys.
+The `graders` catalog returns ordinary `assert` entries such as `type: is-json`, `type: regex`, `type: llm-rubric`, and `type: script`. Explicit `*.eval.ts` and `*.eval.mts` files should default-export an `EvalConfig`; `defineEval(config)` is only an optional thin helper over that same shape. TypeScript config loading lowers camelCase fields such as `expectedOutput`, `inputFiles`, and `maxSteps` to canonical snake_case YAML/runtime keys.
 
 If adapting Braintrust `scores` or DeepEval metrics, write small AgentV helper factories that return `graders.*` configs:
 
@@ -805,9 +828,11 @@ import { defineAssertion } from '@agentv/sdk';
 
 export default defineAssertion(({ output, trace }) => {
   const finalOutput = output ?? '';
+  const pass = finalOutput.length > 0 && (trace?.eventCount ?? 0) <= 10;
   return {
-    pass: finalOutput.length > 0 && (trace?.eventCount ?? 0) <= 10,
-    reasoning: 'Checks content exists and is efficient',
+    pass,
+    score: pass ? 1 : 0,
+    reason: 'Checks content exists and is efficient',
   };
 });
 ```
@@ -823,17 +848,21 @@ import { defineScriptGrader } from '@agentv/sdk';
 
 export default defineScriptGrader(({ output, trace }) => {
   const finalOutput = output ?? '';
+  const hasOutput = finalOutput.length > 0;
+  const efficient = (trace?.eventCount ?? 0) <= 5;
   return {
-    score: finalOutput.length > 0 && (trace?.eventCount ?? 0) <= 5 ? 1.0 : 0.5,
-    assert: [
-      { text: 'Output is not empty', passed: finalOutput.length > 0 },
-      { text: 'Efficient tool usage', passed: (trace?.eventCount ?? 0) <= 5 },
+    pass: hasOutput && efficient,
+    score: hasOutput && efficient ? 1.0 : 0.5,
+    reason: 'Checks content exists and tool usage is bounded',
+    checks: [
+      { text: 'Output is not empty', pass: hasOutput, reason: hasOutput ? 'Output text is present' : 'Output is empty' },
+      { text: 'Efficient tool usage', pass: efficient, reason: efficient ? 'Trace event count is within limit' : 'Trace event count is too high' },
     ],
   };
 });
 ```
 
-Use `defineScriptGrader()` when the custom component is a command-backed grader with explicit score control, custom assertion-result arrays, workspace commands, or LLM calls through a grader target. `defineScriptGrader()` scripts are referenced in YAML with `type: script` and `command: [bun, run, grader.ts]`. Plain Vitest workspace verifier files can use `command: [agentv, eval, graders/check.test.ts]`.
+Use `defineScriptGrader()` when the custom component is a command-backed grader with explicit score control, check arrays, workspace commands, or LLM calls through a grader target. `defineScriptGrader()` scripts are referenced in YAML with `type: script` and `command: [bun, run, grader.ts]`. Plain Vitest workspace verifier files can use `command: [agentv, eval, graders/check.test.ts]`.
 
 ### Convention-Based Discovery
 

package/dist/skills/agentv-eval-writer/references/custom-evaluators.md

@@ -59,13 +59,15 @@ import {
   createTargetClient,
   defineScriptGrader,
   defineEval,
+  type EvalConfig,
   definePromptTemplate,
   graders,
 } from '@agentv/sdk';
 ```
 
 - `defineScriptGrader(fn)` - Wraps evaluation function with stdin/stdout handling
-- `defineEval(definition)` - Defines a YAML-aligned `.eval.ts` suite
+- `EvalConfig` - Public TypeScript eval authoring type for default-exported `*.eval.ts` and `*.eval.mts` files
+- `defineEval(definition)` - Optional thin helper over the same `EvalConfig` shape
 - `graders` - Helper catalog that returns ordinary AgentV `assert` entries
 - `createTargetClient()` - Returns LLM proxy client (when `target: {}` configured)
   - `.invoke({question, systemPrompt})` - Single LLM call
@@ -81,7 +83,7 @@ For Python, the repo-local helper example in `examples/features/sdk-python/` kee
 Use helper factories for reusable Braintrust/DeepEval-inspired checks, but keep the result as AgentV `assert` entries:
 
 ```typescript
-import { defineEval, graders } from '@agentv/sdk';
+import { graders, type EvalConfig } from '@agentv/sdk';
 
 function ragFaithfulness() {
   return graders.llmRubric(undefined, {
@@ -91,19 +93,22 @@ function ragFaithfulness() {
   });
 }
 
-export default defineEval({
+const config: EvalConfig = {
   name: 'rag-suite',
+  prompts: ['{{ task }}'],
   tests: [
     {
       id: 'grounded-answer',
-      input: 'Answer using the retrieved context.',
+      vars: { task: 'Answer using the retrieved context.' },
       assert: [
         graders.contains('source', { name: 'mentions-source' }),
         ragFaithfulness(),
       ],
     },
   ],
-});
+};
+
+export default config;
 ```
 
 The helper lowers to ordinary YAML: