agentv 5.0.0-next.1 → 5.2.0-next.1

package/dist/skills/agentv-eval-review/SKILL.md

@@ -21,10 +21,13 @@ Walk every target eval file and report violations grouped by severity (error > w
 
 - File extension is `.eval.yaml` (error if not).
 - `description` field is present at the top level (error if missing).
-- Each entry under `tests` has `id`, `input`, and at least one of `criteria` / `expected_output` / `assertions` (error if missing).
+- Each entry under `tests` has `id`, `input`, and at least one of `criteria` / `expected_output` / `assert` (error if missing).
 - File-typed inputs (`type: file`) use a leading `/` in their `path` (error if relative).
-- Tests have an `assertions` block — flag tests that rely solely on `expected_output` (warning).
-- Detect `expected_output` prose patterns like "The agent should…" or "The output is…" (warning — prose belongs in `criteria`, structured matches in `assertions`).
+- Tests have an `assert` block — flag tests that rely solely on `expected_output` (warning).
+- Flag `criteria` that duplicates assertion strings when `assertions` already express the grading contract (warning — remove the duplicate `criteria`).
+- Prefer plain assertion strings over multiple named `type: llm-rubric` blocks when the default LLM rubric grader can evaluate the checks (info unless custom prompts or grader targets are present).
+- Detect `expected_output` prose patterns like "The agent should..." or "The output is..." (warning — `expected_output` should be a golden/reference answer; scoring rules belong in `assertions` or, for implicit-grader cases, `criteria`).
+- For historical or repo-state evals, verify the relevant repo is pinned under `workspace.repos[].commit`; a SHA mentioned only in prompt prose or metadata is not an operational checkout (warning).
 - Identical file inputs repeated across multiple tests in the same eval should be hoisted to a top-level `input` (info).
 - Eval files in the same directory should share a common `id` prefix (info — flag drift).
 

package/dist/skills/agentv-eval-review/scripts/lint_eval.py

@@ -8,7 +8,7 @@ Checks:
   - description field present
   - Each test has id, input, criteria
   - File paths in type:file use leading /
-  - assertions blocks present (not relying solely on expected_output)
+  - assert blocks present (not relying solely on expected_output)
   - expected_output does not contain evaluation criteria prose
   - Repeated file inputs across tests (should use top-level input)
   - Naming prefix consistency across eval files in same directory