@axlsdk/eval 0.17.8 → 0.18.0

package/README.md

@@ -101,6 +101,8 @@ const ds = dataset({
 
 You can also load from a file: `dataset({ name: 'large', schema, file: './data.json' })`.
 
+> **Annotation keys must be in the schema.** Zod strips keys it doesn't declare, so an annotation field missing from the `annotations` schema is dropped before it reaches your scorers — and a scorer reading `annotations?.thatField` then silently sees `undefined` and becomes a no-op. By default `dataset()` warns once per dataset listing any dropped keys (`annotation key(s) dropped by schema: ...`). Set `onExtraAnnotationKeys: 'error'` to fail instead (recommended in CI), or `'ignore'` to silence. This is the only guard for `file:` datasets, which aren't type-checked.
+
 ## Scorers
 
 Scorers rate each output on a 0-1 scale. You can mix deterministic and LLM scorers in the same eval.
@@ -185,6 +187,32 @@ const qualityJudge = llmScorer({ name: 'quality', model: 'openai:gpt-4o', ... })
 const safetyJudge = llmScorer({ name: 'safety', model: 'anthropic:claude-sonnet-4-5-20250514', ... });
 ```
 
+#### Tuning the judge
+
+For reasoning-capable judge models (gpt-5.x, Anthropic Opus/Sonnet 4.5+, Gemini 3.x), `effort` is the highest-leverage knob — judges with reasoning enabled are materially more consistent. The full `ChatOptions` surface is forwarded:
+
+```typescript
+const carefulJudge = llmScorer({
+  name: 'careful',
+  description: 'Reasoning-grade quality judge',
+  model: 'openai-responses:gpt-5',
+  system: 'Rate output quality and explain your reasoning.',
+  effort: 'high',              // huge calibration improvement on reasoning models
+  maxTokens: 1024,             // cap chatty judges to control spend
+  includeThoughts: true,       // surface reasoning summaries — Gemini + openai-responses only
+  // thinkingBudget: 4096,     // precise token budget (advanced)
+  // stop: ['\n###'],          // stop sequences
+  // providerOptions: { ... }, // provider-specific escape hatch
+});
+```
+
+A note on a few of the knobs:
+
+- **`includeThoughts`** is a no-op on Anthropic and the `openai:` (Chat Completions) provider. Use `openai-responses:` or any `google:` model to actually receive reasoning summaries.
+- **`providerOptions`** is merged last into the raw API request, so it can override anything Axl computes — including `responseFormat`. To switch the judge to strict JSON Schema mode on providers that support it (OpenAI Chat Completions, OpenAI Responses, Gemini), pass `providerOptions: { response_format: { type: 'json_schema', json_schema: { name: 'judgment', strict: true, schema: <your-schema-as-json> } } }`. The downstream `extractJson` pipeline handles either shape.
+
+The eval runner also propagates its `AbortSignal` into the judge call, so cancelling a run aborts in-flight LLM calls mid-flight (instead of letting the connection finish and silently completing a doomed item).
+
 ## Running Evals
 
 ### CLI
@@ -197,10 +225,42 @@ npx axl-eval ./evals/                               # run all *.eval.* files in
 npx axl-eval ./evals/ --output ./results/v1.json    # save results to JSON
 npx axl-eval ./evals/ --config ./axl.config.ts      # use a specific runtime config
 npx axl-eval ./evals/ --conditions development      # add Node.js import conditions (monorepo source exports)
+npx axl-eval ./evals/qa.eval.ts --concurrency 10    # override item concurrency for this run
+npx axl-eval ./evals/qa.eval.ts --scorers accuracy  # run only named scorer(s) (single file)
 ```
 
 The CLI resolves a runtime automatically: `--config <path>` > auto-detect `axl.config.*` > bare `new AxlRuntime()` (providers from env vars). Use `--conditions` when your eval file imports from monorepo packages that use conditional exports (e.g., `"development"` condition for source TypeScript instead of compiled dist).
 
+**`--concurrency <n>` / `AXL_EVAL_CONCURRENCY`** override item concurrency per-invocation (precedence: flag > env > `defineEval` value > default 5). Concurrency is pure scheduling — it never changes results. Note the per-eval `scorerConcurrency` (default 5) is independent, so the worst-case number of simultaneous scorer calls is `concurrency × scorerConcurrency`; lower `--concurrency` if a rate-limited judge model needs a tighter ceiling.
+
+**`--scorers <a,b>`** runs only the named scorers (single eval file only) for a fast iteration loop. The result is stamped `metadata.scorerFiltered: true` so `axl-eval compare` warns — and, under `--fail-on-regression`, refuses — to gate on a subset run mistaken for a full baseline.
+
+#### Scorer failure rate — two complementary gates
+
+Concurrent scorers (`concurrency × scorerConcurrency`, up to 25 in-flight by default) can overrun a rate-limited judge model. When a judge call exhausts the provider's retry backoff it throws, the runner records that item's score as `null`, and the mean is computed over the **survivors** — so a thinned sample can still look green. The runner now always reports per-scorer `scored`/`failed` counts (visible in the table as `(18/20 scored, 2 failed)` and in Studio as an amber badge), and two opt-in gates turn that signal into a non-zero exit:
+
+| Gate | Where | When it fires |
+|------|-------|---------------|
+| `failOnScorerErrorRate` (config field) | **Source-side** — at run/produce time | `runEval` flags `summary.degraded` (and the CLI exits non-zero) when a scorer exceeds tolerance, *before* the thinned result is saved. `runEval` never throws — it returns the result with the flag set, preserving multi-run partials and auto-save. |
+| `--max-scorer-error-rate <0..1>` (compare flag) | **Gate-side** — at consume/gate time | `axl-eval compare` refuses (exit 1) to certify a baseline/candidate whose failure rate is over the limit. Always *warns* when either side rests on a thinned sample, even without the flag. |
+
+Both are **type-aware**: deterministic scorers tolerate **zero** failures (a deterministic scorer that throws is a bug, not noise); LLM judges use the configured rate. They're complementary — set `failOnScorerErrorRate` in the eval file to fail fast at run time, and pass `--max-scorer-error-rate` in CI to refuse to gate on an artifact that was already thinned (e.g. an imported result).
+
+```ts
+// eval file — fail this run if the judge errored on >10% of items
+export default { workflow: 'qa', dataset: ds, scorers: [judge], failOnScorerErrorRate: 0.1 };
+```
+
+```bash
+npx axl-eval compare base.json cand.json --fail-on-regression --max-scorer-error-rate 0.05
+```
+
+Building a custom CI gate? Both decisions are exported as pure functions — `evaluateScorerErrorRateGate` and `evaluateScorerTolerance` (see the [API reference](../../docs/api-reference.md)).
+
+#### Total-workflow-wipeout guard
+
+Separate from the scorer signal above: if **every** item errored in the *workflow* (0 succeeded), the eval produced no scorable output, so `axl-eval` always exits non-zero with `FAILED: … all N item(s) errored in the workflow` — a fully-broken eval (bad provider URI, an exception in every run) can never go green in CI. This is non-configurable (a 0%-success eval is unambiguously broken) and distinct from `failOnScorerErrorRate` (which is about a flaky *scorer* and deliberately ignores a run with no scored items). Partial workflow-failure rates stay visible (`Failures: N/M` in the table) but non-gating.
+
 ### Programmatic
 
 **`runtime.eval()`** — when you have a workflow registered on the runtime. The `workflow` field must match the registered name:
@@ -404,6 +464,7 @@ Compare two runs to detect regressions and improvements. Runs must use the same
 ```bash
 npx axl-eval compare ./results/v1.json ./results/v2.json
 npx axl-eval compare v1.json v2.json --fail-on-regression  # exit 1 if significant regressions
+npx axl-eval compare v1.json v2.json --max-scorer-error-rate 0.05  # exit 1 if a scorer failed on >5% of the items it ran against (deterministic scorers: zero tolerance)
 ```
 
 ```
@@ -543,18 +604,21 @@ return { output, cost: ctx.totalCost };
 
 ### Common patterns
 
-**Concurrency** — process items in parallel (default: 5):
+**Concurrency** — process items in parallel (default: 5), and fan out each item's scorers (default: 5):
 
 ```typescript
 export default defineEval({
   workflow: 'qa-eval',
   dataset: ds,
-  scorers: [qualityJudge],
-  concurrency: 10,       // run 10 items in parallel
+  scorers: [qualityJudge, toneJudge, safetyJudge],
+  concurrency: 10,        // run 10 items in parallel
+  scorerConcurrency: 3,   // run up to 3 of an item's scorers at once
   budget: '$5.00',        // stop if total cost exceeds $5
 });
 ```
 
+`scorerConcurrency` parallelizes the per-item judge phase — the dominant cost for evals with several `llmScorer`s. The worst-case number of simultaneous scorer calls is `concurrency × scorerConcurrency`, so a rate-limited judge model may need a lower value (set `scorerConcurrency: 1` for the old serial behavior). Because the per-item `budget` check runs once before an item's scorers, scorer-cost overshoot is bounded by `concurrency × scorerConcurrency × max-scorer-cost`.
+
 **Per-item budget** — cap cost for a single workflow execution:
 
 ```typescript
@@ -594,7 +658,9 @@ npx axl-eval rescore ./results/v1.json ./evals/qa.eval.ts
 npx axl-eval rescore ./results/v1.json ./evals/qa.eval.ts --output ./results/v1-rescored.json
 ```
 
-The rescored result gets a new `id` and has `metadata.rescored: true` and `metadata.originalId` pointing to the source run. `runGroupId` and `runIndex` are stripped from inherited metadata (rescored results are independent evaluations). Only scorer cost is tracked (workflow cost is zero).
+The rescored result gets a new `id` and has `metadata.rescored: true` and `metadata.originalId` pointing to the source run. `runGroupId`, `runIndex`, and any `scorerFiltered`/`scorersRun` stamp are stripped from inherited metadata (rescored results are independent evaluations, and a rescore always runs the eval file's full scorer set). Only scorer cost is tracked (workflow cost is zero).
+
+`--concurrency`/`AXL_EVAL_CONCURRENCY` are honored by `rescore` too (item-level; `scorerConcurrency` keeps its default 5). `--scorers` is **not** supported with `rescore` — it's a run-command-only filter; rescore re-runs whatever scorers the eval file exports (pass a subset there instead).
 
 Programmatically:
 
@@ -602,7 +668,8 @@ Programmatically:
 import { rescore } from '@axlsdk/eval';
 
 const rescored = await rescore(originalResult, [updatedScorer, newScorer], runtime, {
-  concurrency: 10,
+  concurrency: 10,        // items rescored in parallel
+  scorerConcurrency: 3,   // scorers per item run in parallel (default 5)
 });
 
 console.log(rescored.metadata.rescored);    // true
@@ -688,7 +755,7 @@ const comparison = evalCompare(baselineRuns, candidateRuns);
 
 | Type | Description |
 |------|-------------|
-| `EvalConfig` | Eval definition (workflow, dataset, scorers, concurrency, budget) |
+| `EvalConfig` | Eval definition (workflow, dataset, scorers, concurrency, scorerConcurrency, budget) |
 | `EvalResult` | Full eval output (items, summary, cost, duration) |
 | `EvalItem` | Per-item result (input, output, scores, scoreDetails, metadata, traces?) |
 | `EvalSummary` | Aggregate statistics (count, failures, scorer stats, timing) |
@@ -698,7 +765,7 @@ const comparison = evalCompare(baselineRuns, candidateRuns);
 | `EvalRegression` / `EvalImprovement` | Per-item change record (itemIndex, scorer, scores, delta) |
 | `ScorerDetail` | Per-scorer detail (score, metadata, duration, cost) |
 | `ScorerResult` | Scorer return type (`{ score, metadata?, cost? }`) |
-| `RescoreOptions` | Options for `rescore()` (`concurrency`) |
+| `RescoreOptions` | Options for `rescore()` (`concurrency`, `scorerConcurrency`, `signal`) |
 | `MultiRunSummary` | Aggregated multi-run output (per-scorer mean/std/min/max) |
 | `BootstrapCIResult` | CI result (`{ lower, upper, mean, pRegression, pImprovement }`) |
 | `RunEvalOptions` | Options for `runEval()` (`onProgress`, `signal`, `captureTraces`) |