@axlsdk/eval 0.17.9 → 0.18.1

package/README.md

@@ -101,6 +101,8 @@ const ds = dataset({
 
 You can also load from a file: `dataset({ name: 'large', schema, file: './data.json' })`.
 
+> **Annotation keys must be in the schema.** Zod strips keys it doesn't declare, so an annotation field missing from the `annotations` schema is dropped before it reaches your scorers — and a scorer reading `annotations?.thatField` then silently sees `undefined` and becomes a no-op. By default `dataset()` warns once per dataset listing any dropped keys (`annotation key(s) dropped by schema: ...`). Set `onExtraAnnotationKeys: 'error'` to fail instead (recommended in CI), or `'ignore'` to silence. This is the only guard for `file:` datasets, which aren't type-checked.
+
 ## Scorers
 
 Scorers rate each output on a 0-1 scale. You can mix deterministic and LLM scorers in the same eval.
@@ -132,6 +134,48 @@ const lengthScore = scorer({
 
 Scorers that return a non-finite value (`NaN` / `Infinity`) or a score outside `[0, 1]` are recorded as `null` with an entry in `item.scorerErrors` — they don't abort the run.
 
+### Conditional scorers (`applies`)
+
+By default every scorer runs against every item. When a scorer only applies to a subset — a refusal judge for refusal-expected items, a constraint judge for constrained items, an asymmetry scorer for asymmetric exercises — declare its scope with an `applies` predicate. It mirrors the `score` signature (`output, input, annotations?`) and returns `true` to run or `false` to skip:
+
+```typescript
+import { scorer } from '@axlsdk/eval';
+
+// Deterministic: only score items the dataset flagged as "constrained"
+const constraintAdherence = scorer({
+  name: 'constraint-adherence',
+  description: 'Output respects the stated constraint',
+  applies: (_output, _input, annotations) => annotations?.constrained === true,
+  score: (output, _input, annotations) =>
+    String(output).includes(String(annotations?.constraint)) ? 1 : 0,
+});
+```
+
+When `applies` returns `false`, the scorer is **skipped** for that item:
+
+- The scorer body never runs — for an `llmScorer`, **no provider call is made**, saving real cost and rate-limit budget (a skipped judge can't 429 you).
+- The item counts as **neither `scored` nor `failed`** — it's excluded from the `mean` *and* from the failure-rate denominator (`failOnScorerErrorRate` and `compare --max-scorer-error-rate`), so a conditional scorer stays honest to both the mean and the trust gate.
+- The skip is visible: `item.scoreDetails[name].skipped === true`, and the run summary carries a per-scorer `skipped` count (alongside `scored` / `failed`). Studio's Eval Runner shows an "N/A" chip.
+
+Applicability usually keys off `input` / `annotations` (a flag or exercise type), but `output` is available too (e.g. "only score outputs that actually refused"). The predicate runs **before** the scorer body, so an inapplicable `llmScorer` never makes its provider call:
+
+```typescript
+import { llmScorer } from '@axlsdk/eval';
+
+// LLM judge: only invoke the (expensive) judge for items where a refusal was expected
+const refusalQuality = llmScorer({
+  name: 'refusal-quality',
+  description: 'Did the model refuse appropriately and explain why?',
+  model: 'openai:gpt-4o',
+  system: 'Rate whether the output is a well-justified refusal.',
+  applies: (_output, _input, annotations) => annotations?.expectRefusal === true,
+});
+```
+
+A predicate that **throws** is treated as a scorer failure (it's a bug), not a skip — the item lands in the `failed` bucket with an entry in `item.scorerErrors`, never silently swallowed.
+
+> **Deprecated: the `NaN`-skip workaround.** Before `applies`, the only way to express "not applicable" was to return a `NaN` / out-of-range sentinel for skipped items. The failure-rate gate (correctly) treats a non-finite score as a real failure, so a deterministic conditional scorer that returned `NaN` for inapplicable items showed a ~90% failure rate and permanently tripped the zero-tolerance deterministic gate. Use `applies` instead. The gate's strictness about `NaN` is intentional and unchanged.
+
 ### LLM scorers
 
 Use an LLM as a judge. The scorer constructs a prompt from the input, output, and annotations, calls the LLM, and validates the response:
@@ -185,6 +229,8 @@ const qualityJudge = llmScorer({ name: 'quality', model: 'openai:gpt-4o', ... })
 const safetyJudge = llmScorer({ name: 'safety', model: 'anthropic:claude-sonnet-4-5-20250514', ... });
 ```
 
+LLM scorers also accept the conditional `applies` predicate described in [Conditional scorers](#conditional-scorers-applies) — the most valuable place to use it, since a skipped judge makes no provider call at all.
+
 #### Tuning the judge
 
 For reasoning-capable judge models (gpt-5.x, Anthropic Opus/Sonnet 4.5+, Gemini 3.x), `effort` is the highest-leverage knob — judges with reasoning enabled are materially more consistent. The full `ChatOptions` surface is forwarded:
@@ -223,10 +269,49 @@ npx axl-eval ./evals/                               # run all *.eval.* files in
 npx axl-eval ./evals/ --output ./results/v1.json    # save results to JSON
 npx axl-eval ./evals/ --config ./axl.config.ts      # use a specific runtime config
 npx axl-eval ./evals/ --conditions development      # add Node.js import conditions (monorepo source exports)
+npx axl-eval ./evals/qa.eval.ts --concurrency 10    # override item concurrency for this run
+npx axl-eval ./evals/qa.eval.ts --scorers accuracy  # run only named scorer(s) (single file)
 ```
 
 The CLI resolves a runtime automatically: `--config <path>` > auto-detect `axl.config.*` > bare `new AxlRuntime()` (providers from env vars). Use `--conditions` when your eval file imports from monorepo packages that use conditional exports (e.g., `"development"` condition for source TypeScript instead of compiled dist).
 
+**`--concurrency <n>` / `AXL_EVAL_CONCURRENCY`** override item concurrency per-invocation (precedence: flag > env > `defineEval` value > default 5). Concurrency is pure scheduling — it never changes results. Note the per-eval `scorerConcurrency` (default 5) is independent, so the worst-case number of simultaneous scorer calls is `concurrency × scorerConcurrency`; lower `--concurrency` if a rate-limited judge model needs a tighter ceiling.
+
+**`--scorers <a,b>`** runs only the named scorers (single eval file only) for a fast iteration loop. The result is stamped `metadata.scorerFiltered: true` so `axl-eval compare` warns — and, under `--fail-on-regression`, refuses — to gate on a subset run mistaken for a full baseline.
+
+#### Scorer failure rate — two complementary gates
+
+Concurrent scorers (`concurrency × scorerConcurrency`, up to 25 in-flight by default) can overrun a rate-limited judge model. When a judge call exhausts the provider's retry backoff it throws, the runner records that item's score as `null`, and the mean is computed over the **survivors** — so a thinned sample can still look green. The runner now always reports per-scorer `scored`/`failed` counts (visible in the table as `(18/20 scored, 2 failed)` and in Studio as an amber badge), and two opt-in gates turn that signal into a non-zero exit:
+
+| Gate | Where | When it fires |
+|------|-------|---------------|
+| `failOnScorerErrorRate` (config field) | **Source-side** — at run/produce time | `runEval` flags `summary.degraded` (and the CLI exits non-zero) when a scorer exceeds tolerance, *before* the thinned result is saved. `runEval` never throws — it returns the result with the flag set, preserving multi-run partials and auto-save. |
+| `--max-scorer-error-rate <0..1>` (compare flag) | **Gate-side** — at consume/gate time | `axl-eval compare` refuses (exit 1) to certify a baseline/candidate whose failure rate is over the limit. Always *warns* when either side rests on a thinned sample, even without the flag. |
+
+Both are **type-aware**: deterministic scorers tolerate **zero** failures (a deterministic scorer that throws is a bug, not noise); LLM judges use the configured rate. They're complementary — set `failOnScorerErrorRate` in the eval file to fail fast at run time, and pass `--max-scorer-error-rate` in CI to refuse to gate on an artifact that was already thinned (e.g. an imported result).
+
+**Skipped items don't count against either gate.** The failure rate is `failed / (scored + failed)` — items a scorer's `applies` predicate skipped land in neither bucket, so they're excluded from the denominator. This is the supported way to scope a conditional scorer (a refusal judge that only runs on refusal-expected items, say) to its applicable subset without polluting the trust signal. **Do not** return `NaN` for inapplicable items: the gate (correctly) treats a non-finite score as a real failure, so a `NaN`-skipping deterministic scorer trips the zero-tolerance deterministic gate on every skipped item. Use [`applies`](#conditional-scorers-applies) instead.
+
+Because skips are excluded rather than failed, two failure modes are surfaced as **advisories** (stderr, non-fatal) so a conditional scorer can't silently mislead:
+
+- **Fully N/A scorer** — if a scorer ends up applicable to *zero* items (an over-strict predicate, a dataset missing the flagged items, or a predicate that forgot to `return`), its mean is a meaningless empty-sample `0` and the failure-rate gate can't assess it. `axl-eval` prints a `NOTE` (the run still exits 0 — skips are legitimately conditional).
+- **Asymmetric skips in `compare`** — if the baseline and candidate applied a scorer to *different* item subsets (different N/A counts, or the same count of *different* items — detected as fewer items scored on both sides than either scored alone), their per-side means cover different subsets, so a delta may be a sample mismatch rather than a real change. `axl-eval compare` prints a `NOTE` alongside the usual thinned-sample warning. (Studio's compare view shows the same as a "paired n" note.)
+
+```ts
+// eval file — fail this run if the judge errored on >10% of the items it ran against
+export default { workflow: 'qa', dataset: ds, scorers: [judge], failOnScorerErrorRate: 0.1 };
+```
+
+```bash
+npx axl-eval compare base.json cand.json --fail-on-regression --max-scorer-error-rate 0.05
+```
+
+Building a custom CI gate? Both decisions are exported as pure functions — `evaluateScorerErrorRateGate` and `evaluateScorerTolerance` (see the [API reference](../../docs/api-reference.md)).
+
+#### Total-workflow-wipeout guard
+
+Separate from the scorer signal above: if **every** item errored in the *workflow* (0 succeeded), the eval produced no scorable output, so `axl-eval` always exits non-zero with `FAILED: … all N item(s) errored in the workflow` — a fully-broken eval (bad provider URI, an exception in every run) can never go green in CI. This is non-configurable (a 0%-success eval is unambiguously broken) and distinct from `failOnScorerErrorRate` (which is about a flaky *scorer* and deliberately ignores a run with no scored items). Partial workflow-failure rates stay visible (`Failures: N/M` in the table) but non-gating.
+
 ### Programmatic
 
 **`runtime.eval()`** — when you have a workflow registered on the runtime. The `workflow` field must match the registered name:
@@ -319,8 +404,16 @@ Each eval run returns an `EvalResult` with per-item scores and aggregate statist
 
 Each item has two ways to access scores:
 
-- **`item.scores`** — quick numeric lookup: `Record<string, number | null>`. Use this for simple checks and aggregation. `null` means the scorer failed (see `item.scorerErrors`).
-- **`item.scoreDetails`** — full context: `Record<string, ScorerDetail>`. Each detail has the numeric score plus `metadata` (e.g., LLM reasoning), per-scorer `duration`, and `cost`. Use this when you need to understand *why* a score is what it is.
+- **`item.scores`** — quick numeric lookup: `Record<string, number | null>`. Use this for simple checks and aggregation. `null` means the scorer failed (see `item.scorerErrors`) **or** was skipped by its `applies` predicate — disambiguate with `scoreDetails[name].skipped`.
+- **`item.scoreDetails`** — full context: `Record<string, ScorerDetail>`. Each detail has the numeric score plus `metadata` (e.g., LLM reasoning), per-scorer `duration`, `cost`, and a `skipped` flag. Use this when you need to understand *why* a score is what it is.
+
+A `ScorerDetail` distinguishes three terminal states for a scorer on a given item:
+
+| State | `score` | `skipped` | `duration` |
+|-------|---------|-----------|------------|
+| Scored | `number` | absent | set |
+| Ran-and-failed (threw / out-of-range) | `null` | absent | set |
+| Skipped by `applies` | `null` | `true` | absent |
 
 Summary statistics (mean, p50, p95, etc.) exclude `null` scores. If all scores for a scorer are `null`, the CLI shows `--` instead of misleading `0.00`.
 
@@ -329,6 +422,9 @@ const results = await runtime.eval({ ... });
 
 // ── Aggregate stats ──────────────────────────────────
 console.log(results.summary.scorers['quality'].mean);  // 0.85
+console.log(results.summary.scorers['quality'].scored); // 48 (items that produced a numeric score)
+console.log(results.summary.scorers['quality'].failed); // 2  (scorer ran and threw / out-of-range)
+console.log(results.summary.scorers['quality'].skipped); // 0  (items the applies predicate skipped)
 console.log(results.summary.count);                     // 50 items
 console.log(results.summary.failures);                  // 2 workflow errors
 console.log(results.summary.timing);                    // { mean, min, max, p50, p95 } in ms
@@ -356,14 +452,19 @@ for (const item of results.items) {
   // Rich per-scorer detail — reasoning, timing, cost
   const detail = item.scoreDetails?.['quality'];
   if (detail) {
-    console.log(detail.score);                           // 0.85
-    console.log(detail.metadata?.reasoning);             // "The answer is relevant..."
-    console.log(detail.duration);                        // scorer execution ms
-    console.log(detail.cost);                            // scorer LLM cost
+    if (detail.skipped) {
+      // This scorer's `applies` predicate returned false — not run, not counted
+      console.log('Scorer skipped (not applicable to this item)');
+    } else {
+      console.log(detail.score);                         // 0.85
+      console.log(detail.metadata?.reasoning);           // "The answer is relevant..."
+      console.log(detail.duration);                      // scorer execution ms
+      console.log(detail.cost);                          // scorer LLM cost
+    }
   }
 
-  // Error handling
-  if (item.scores['quality'] === null) {
+  // Error handling — a null score is either a failure OR a skip
+  if (item.scores['quality'] === null && !item.scoreDetails?.['quality']?.skipped) {
     console.log('Scorer failed:', item.scorerErrors);    // ["Scorer "quality" threw: ..."]
   }
 
@@ -430,6 +531,7 @@ Compare two runs to detect regressions and improvements. Runs must use the same
 ```bash
 npx axl-eval compare ./results/v1.json ./results/v2.json
 npx axl-eval compare v1.json v2.json --fail-on-regression  # exit 1 if significant regressions
+npx axl-eval compare v1.json v2.json --max-scorer-error-rate 0.05  # exit 1 if a scorer failed on >5% of the items it ran against (deterministic scorers: zero tolerance)
 ```
 
 ```
@@ -569,18 +671,21 @@ return { output, cost: ctx.totalCost };
 
 ### Common patterns
 
-**Concurrency** — process items in parallel (default: 5):
+**Concurrency** — process items in parallel (default: 5), and fan out each item's scorers (default: 5):
 
 ```typescript
 export default defineEval({
   workflow: 'qa-eval',
   dataset: ds,
-  scorers: [qualityJudge],
-  concurrency: 10,       // run 10 items in parallel
+  scorers: [qualityJudge, toneJudge, safetyJudge],
+  concurrency: 10,        // run 10 items in parallel
+  scorerConcurrency: 3,   // run up to 3 of an item's scorers at once
   budget: '$5.00',        // stop if total cost exceeds $5
 });
 ```
 
+`scorerConcurrency` parallelizes the per-item judge phase — the dominant cost for evals with several `llmScorer`s. The worst-case number of simultaneous scorer calls is `concurrency × scorerConcurrency`, so a rate-limited judge model may need a lower value (set `scorerConcurrency: 1` for the old serial behavior). Because the per-item `budget` check runs once before an item's scorers, scorer-cost overshoot is bounded by `concurrency × scorerConcurrency × max-scorer-cost`.
+
 **Per-item budget** — cap cost for a single workflow execution:
 
 ```typescript
@@ -620,7 +725,9 @@ npx axl-eval rescore ./results/v1.json ./evals/qa.eval.ts
 npx axl-eval rescore ./results/v1.json ./evals/qa.eval.ts --output ./results/v1-rescored.json
 ```
 
-The rescored result gets a new `id` and has `metadata.rescored: true` and `metadata.originalId` pointing to the source run. `runGroupId` and `runIndex` are stripped from inherited metadata (rescored results are independent evaluations). Only scorer cost is tracked (workflow cost is zero).
+The rescored result gets a new `id` and has `metadata.rescored: true` and `metadata.originalId` pointing to the source run. `runGroupId`, `runIndex`, and any `scorerFiltered`/`scorersRun` stamp are stripped from inherited metadata (rescored results are independent evaluations, and a rescore always runs the eval file's full scorer set). Only scorer cost is tracked (workflow cost is zero).
+
+`--concurrency`/`AXL_EVAL_CONCURRENCY` are honored by `rescore` too (item-level; `scorerConcurrency` keeps its default 5). `--scorers` is **not** supported with `rescore` — it's a run-command-only filter; rescore re-runs whatever scorers the eval file exports (pass a subset there instead).
 
 Programmatically:
 
@@ -628,7 +735,8 @@ Programmatically:
 import { rescore } from '@axlsdk/eval';
 
 const rescored = await rescore(originalResult, [updatedScorer, newScorer], runtime, {
-  concurrency: 10,
+  concurrency: 10,        // items rescored in parallel
+  scorerConcurrency: 3,   // scorers per item run in parallel (default 5)
 });
 
 console.log(rescored.metadata.rescored);    // true
@@ -700,8 +808,8 @@ const comparison = evalCompare(baselineRuns, candidateRuns);
 | Function | Description |
 |----------|-------------|
 | `dataset(config)` | Create a dataset from inline items or a JSON file |
-| `scorer(config)` | Create a deterministic scorer |
-| `llmScorer(config)` | Create an LLM-as-judge scorer |
+| `scorer(config)` | Create a deterministic scorer (optional `applies` predicate for conditional scoring) |
+| `llmScorer(config)` | Create an LLM-as-judge scorer (optional `applies` predicate — a skipped judge makes no provider call) |
 | `defineEval(config)` | Wrap an eval config for CLI discovery |
 | `runEval(config, executeFn, runtime, options?)` | Run an eval programmatically. `options: RunEvalOptions` accepts `onProgress` / `signal` / `captureTraces` |
 | `evalCompare(baseline, candidate, options?)` | Compare eval results with bootstrap CI |
@@ -714,17 +822,17 @@ const comparison = evalCompare(baselineRuns, candidateRuns);
 
 | Type | Description |
 |------|-------------|
-| `EvalConfig` | Eval definition (workflow, dataset, scorers, concurrency, budget) |
+| `EvalConfig` | Eval definition (workflow, dataset, scorers, concurrency, scorerConcurrency, budget) |
 | `EvalResult` | Full eval output (items, summary, cost, duration) |
 | `EvalItem` | Per-item result (input, output, scores, scoreDetails, metadata, traces?) |
-| `EvalSummary` | Aggregate statistics (count, failures, scorer stats, timing) |
+| `EvalSummary` | Aggregate statistics (count, failures, per-scorer stats incl. `scored`/`failed`/`skipped`, timing) |
 | `EvalComparison` | Comparison output (scorer deltas, CI, pRegression/pImprovement, n, per-side `runCount` / `partial?`, regressions, improvements). Both sides truncate to `min(baseline.length, candidate.length)` so means align with the paired bootstrap CI's sample |
 | `EvalComparisonPartial` | `{ completed, attempted }` set on a comparison side when the pooled run count is less than the original batch's planned count |
 | `EvalCompareOptions` | Options for `evalCompare()` (`thresholds`) |
 | `EvalRegression` / `EvalImprovement` | Per-item change record (itemIndex, scorer, scores, delta) |
-| `ScorerDetail` | Per-scorer detail (score, metadata, duration, cost) |
+| `ScorerDetail` | Per-scorer detail (score, metadata, duration, cost, `skipped?`) |
 | `ScorerResult` | Scorer return type (`{ score, metadata?, cost? }`) |
-| `RescoreOptions` | Options for `rescore()` (`concurrency`) |
+| `RescoreOptions` | Options for `rescore()` (`concurrency`, `scorerConcurrency`, `signal`) |
 | `MultiRunSummary` | Aggregated multi-run output (per-scorer mean/std/min/max) |
 | `BootstrapCIResult` | CI result (`{ lower, upper, mean, pRegression, pImprovement }`) |
 | `RunEvalOptions` | Options for `runEval()` (`onProgress`, `signal`, `captureTraces`) |