@axlsdk/eval 0.17.9 → 0.18.0

package/README.md

@@ -101,6 +101,8 @@ const ds = dataset({
 
 You can also load from a file: `dataset({ name: 'large', schema, file: './data.json' })`.
 
+> **Annotation keys must be in the schema.** Zod strips keys it doesn't declare, so an annotation field missing from the `annotations` schema is dropped before it reaches your scorers — and a scorer reading `annotations?.thatField` then silently sees `undefined` and becomes a no-op. By default `dataset()` warns once per dataset listing any dropped keys (`annotation key(s) dropped by schema: ...`). Set `onExtraAnnotationKeys: 'error'` to fail instead (recommended in CI), or `'ignore'` to silence. This is the only guard for `file:` datasets, which aren't type-checked.
+
 ## Scorers
 
 Scorers rate each output on a 0-1 scale. You can mix deterministic and LLM scorers in the same eval.
@@ -223,10 +225,42 @@ npx axl-eval ./evals/                               # run all *.eval.* files in
 npx axl-eval ./evals/ --output ./results/v1.json    # save results to JSON
 npx axl-eval ./evals/ --config ./axl.config.ts      # use a specific runtime config
 npx axl-eval ./evals/ --conditions development      # add Node.js import conditions (monorepo source exports)
+npx axl-eval ./evals/qa.eval.ts --concurrency 10    # override item concurrency for this run
+npx axl-eval ./evals/qa.eval.ts --scorers accuracy  # run only named scorer(s) (single file)
 ```
 
 The CLI resolves a runtime automatically: `--config <path>` > auto-detect `axl.config.*` > bare `new AxlRuntime()` (providers from env vars). Use `--conditions` when your eval file imports from monorepo packages that use conditional exports (e.g., `"development"` condition for source TypeScript instead of compiled dist).
 
+**`--concurrency <n>` / `AXL_EVAL_CONCURRENCY`** override item concurrency per-invocation (precedence: flag > env > `defineEval` value > default 5). Concurrency is pure scheduling — it never changes results. Note the per-eval `scorerConcurrency` (default 5) is independent, so the worst-case number of simultaneous scorer calls is `concurrency × scorerConcurrency`; lower `--concurrency` if a rate-limited judge model needs a tighter ceiling.
+
+**`--scorers <a,b>`** runs only the named scorers (single eval file only) for a fast iteration loop. The result is stamped `metadata.scorerFiltered: true` so `axl-eval compare` warns — and, under `--fail-on-regression`, refuses — to gate on a subset run mistaken for a full baseline.
+
+#### Scorer failure rate — two complementary gates
+
+Concurrent scorers (`concurrency × scorerConcurrency`, up to 25 in-flight by default) can overrun a rate-limited judge model. When a judge call exhausts the provider's retry backoff it throws, the runner records that item's score as `null`, and the mean is computed over the **survivors** — so a thinned sample can still look green. The runner now always reports per-scorer `scored`/`failed` counts (visible in the table as `(18/20 scored, 2 failed)` and in Studio as an amber badge), and two opt-in gates turn that signal into a non-zero exit:
+
+| Gate | Where | When it fires |
+|------|-------|---------------|
+| `failOnScorerErrorRate` (config field) | **Source-side** — at run/produce time | `runEval` flags `summary.degraded` (and the CLI exits non-zero) when a scorer exceeds tolerance, *before* the thinned result is saved. `runEval` never throws — it returns the result with the flag set, preserving multi-run partials and auto-save. |
+| `--max-scorer-error-rate <0..1>` (compare flag) | **Gate-side** — at consume/gate time | `axl-eval compare` refuses (exit 1) to certify a baseline/candidate whose failure rate is over the limit. Always *warns* when either side rests on a thinned sample, even without the flag. |
+
+Both are **type-aware**: deterministic scorers tolerate **zero** failures (a deterministic scorer that throws is a bug, not noise); LLM judges use the configured rate. They're complementary — set `failOnScorerErrorRate` in the eval file to fail fast at run time, and pass `--max-scorer-error-rate` in CI to refuse to gate on an artifact that was already thinned (e.g. an imported result).
+
+```ts
+// eval file — fail this run if the judge errored on >10% of items
+export default { workflow: 'qa', dataset: ds, scorers: [judge], failOnScorerErrorRate: 0.1 };
+```
+
+```bash
+npx axl-eval compare base.json cand.json --fail-on-regression --max-scorer-error-rate 0.05
+```
+
+Building a custom CI gate? Both decisions are exported as pure functions — `evaluateScorerErrorRateGate` and `evaluateScorerTolerance` (see the [API reference](../../docs/api-reference.md)).
+
+#### Total-workflow-wipeout guard
+
+Separate from the scorer signal above: if **every** item errored in the *workflow* (0 succeeded), the eval produced no scorable output, so `axl-eval` always exits non-zero with `FAILED: … all N item(s) errored in the workflow` — a fully-broken eval (bad provider URI, an exception in every run) can never go green in CI. This is non-configurable (a 0%-success eval is unambiguously broken) and distinct from `failOnScorerErrorRate` (which is about a flaky *scorer* and deliberately ignores a run with no scored items). Partial workflow-failure rates stay visible (`Failures: N/M` in the table) but non-gating.
+
 ### Programmatic
 
 **`runtime.eval()`** — when you have a workflow registered on the runtime. The `workflow` field must match the registered name:
@@ -430,6 +464,7 @@ Compare two runs to detect regressions and improvements. Runs must use the same
 ```bash
 npx axl-eval compare ./results/v1.json ./results/v2.json
 npx axl-eval compare v1.json v2.json --fail-on-regression  # exit 1 if significant regressions
+npx axl-eval compare v1.json v2.json --max-scorer-error-rate 0.05  # exit 1 if a scorer failed on >5% of the items it ran against (deterministic scorers: zero tolerance)
 ```
 
 ```
@@ -569,18 +604,21 @@ return { output, cost: ctx.totalCost };
 
 ### Common patterns
 
-**Concurrency** — process items in parallel (default: 5):
+**Concurrency** — process items in parallel (default: 5), and fan out each item's scorers (default: 5):
 
 ```typescript
 export default defineEval({
   workflow: 'qa-eval',
   dataset: ds,
-  scorers: [qualityJudge],
-  concurrency: 10,       // run 10 items in parallel
+  scorers: [qualityJudge, toneJudge, safetyJudge],
+  concurrency: 10,        // run 10 items in parallel
+  scorerConcurrency: 3,   // run up to 3 of an item's scorers at once
   budget: '$5.00',        // stop if total cost exceeds $5
 });
 ```
 
+`scorerConcurrency` parallelizes the per-item judge phase — the dominant cost for evals with several `llmScorer`s. The worst-case number of simultaneous scorer calls is `concurrency × scorerConcurrency`, so a rate-limited judge model may need a lower value (set `scorerConcurrency: 1` for the old serial behavior). Because the per-item `budget` check runs once before an item's scorers, scorer-cost overshoot is bounded by `concurrency × scorerConcurrency × max-scorer-cost`.
+
 **Per-item budget** — cap cost for a single workflow execution:
 
 ```typescript
@@ -620,7 +658,9 @@ npx axl-eval rescore ./results/v1.json ./evals/qa.eval.ts
 npx axl-eval rescore ./results/v1.json ./evals/qa.eval.ts --output ./results/v1-rescored.json
 ```
 
-The rescored result gets a new `id` and has `metadata.rescored: true` and `metadata.originalId` pointing to the source run. `runGroupId` and `runIndex` are stripped from inherited metadata (rescored results are independent evaluations). Only scorer cost is tracked (workflow cost is zero).
+The rescored result gets a new `id` and has `metadata.rescored: true` and `metadata.originalId` pointing to the source run. `runGroupId`, `runIndex`, and any `scorerFiltered`/`scorersRun` stamp are stripped from inherited metadata (rescored results are independent evaluations, and a rescore always runs the eval file's full scorer set). Only scorer cost is tracked (workflow cost is zero).
+
+`--concurrency`/`AXL_EVAL_CONCURRENCY` are honored by `rescore` too (item-level; `scorerConcurrency` keeps its default 5). `--scorers` is **not** supported with `rescore` — it's a run-command-only filter; rescore re-runs whatever scorers the eval file exports (pass a subset there instead).
 
 Programmatically:
 
@@ -628,7 +668,8 @@ Programmatically:
 import { rescore } from '@axlsdk/eval';
 
 const rescored = await rescore(originalResult, [updatedScorer, newScorer], runtime, {
-  concurrency: 10,
+  concurrency: 10,        // items rescored in parallel
+  scorerConcurrency: 3,   // scorers per item run in parallel (default 5)
 });
 
 console.log(rescored.metadata.rescored);    // true
@@ -714,7 +755,7 @@ const comparison = evalCompare(baselineRuns, candidateRuns);
 
 | Type | Description |
 |------|-------------|
-| `EvalConfig` | Eval definition (workflow, dataset, scorers, concurrency, budget) |
+| `EvalConfig` | Eval definition (workflow, dataset, scorers, concurrency, scorerConcurrency, budget) |
 | `EvalResult` | Full eval output (items, summary, cost, duration) |
 | `EvalItem` | Per-item result (input, output, scores, scoreDetails, metadata, traces?) |
 | `EvalSummary` | Aggregate statistics (count, failures, scorer stats, timing) |
@@ -724,7 +765,7 @@ const comparison = evalCompare(baselineRuns, candidateRuns);
 | `EvalRegression` / `EvalImprovement` | Per-item change record (itemIndex, scorer, scores, delta) |
 | `ScorerDetail` | Per-scorer detail (score, metadata, duration, cost) |
 | `ScorerResult` | Scorer return type (`{ score, metadata?, cost? }`) |
-| `RescoreOptions` | Options for `rescore()` (`concurrency`) |
+| `RescoreOptions` | Options for `rescore()` (`concurrency`, `scorerConcurrency`, `signal`) |
 | `MultiRunSummary` | Aggregated multi-run output (per-scorer mean/std/min/max) |
 | `BootstrapCIResult` | CI result (`{ lower, upper, mean, pRegression, pImprovement }`) |
 | `RunEvalOptions` | Options for `runEval()` (`onProgress`, `signal`, `captureTraces`) |

package/dist/{chunk-NETD4AUB.js → chunk-BWBX5YVQ.js}

@@ -64,10 +64,52 @@ function xorshift32(seed) {
   };
 }
 
-// src/compare.ts
+// src/utils.ts
+function computeStats(scores) {
+  if (scores.length === 0) return { mean: 0, min: 0, max: 0, p50: 0, p95: 0 };
+  const sorted = [...scores].sort((a, b) => a - b);
+  const mean = sorted.reduce((a, b) => a + b, 0) / sorted.length;
+  const min = sorted[0];
+  const max = sorted[sorted.length - 1];
+  const p50 = sorted[Math.floor(sorted.length * 0.5)];
+  const p95 = sorted[Math.floor(sorted.length * 0.95)];
+  return { mean: round2(mean), min: round2(min), max: round2(max), p50: round2(p50), p95: round2(p95) };
+}
 function round2(n) {
   return Math.round(n * 1e3) / 1e3;
 }
+function scorerCounts(items, name) {
+  let scored = 0;
+  let failed = 0;
+  for (const i of items) {
+    if (i.error) continue;
+    if (i.scores[name] != null) scored++;
+    else if (i.scoreDetails?.[name]?.duration != null) failed++;
+  }
+  return { scored, failed };
+}
+function evaluateScorerTolerance(scored, failed, type, limit) {
+  const attempted = scored + failed;
+  const rate = attempted === 0 ? 0 : failed / attempted;
+  const exceeds = attempted > 0 && (type === "deterministic" ? failed > 0 : rate > limit);
+  return { attempted, rate, exceeds, zeroSample: attempted === 0 };
+}
+async function mapWithConcurrency(items, concurrency, task) {
+  const results = new Array(items.length);
+  const requested = Number.isFinite(concurrency) ? Math.floor(concurrency) : 1;
+  const workerCount = items.length === 0 ? 0 : Math.min(Math.max(1, requested), items.length);
+  let next = 0;
+  async function runNext() {
+    while (next < items.length) {
+      const i = next++;
+      results[i] = await task(items[i], i);
+    }
+  }
+  await Promise.all(Array.from({ length: workerCount }, () => runNext()));
+  return results;
+}
+
+// src/compare.ts
 var DEFAULT_LLM_THRESHOLD = 0.05;
 var DEFAULT_DETERMINISTIC_THRESHOLD = 0;
 var LEGACY_THRESHOLD = 0.1;
@@ -141,11 +183,29 @@ function evalCompare(baseline, candidate, options) {
     const delta = round2(deltaRaw);
     const deltaPercent = bMeanRaw > 0 ? round2(deltaRaw / bMeanRaw * 100) : 0;
     const threshold = resolveThreshold(name, options, baselineRef.metadata);
+    let baselineScored = 0;
+    let baselineFailed = 0;
+    for (const run of baselineRuns) {
+      const c = scorerCounts(run.items, name);
+      baselineScored += c.scored;
+      baselineFailed += c.failed;
+    }
+    let candidateScored = 0;
+    let candidateFailed = 0;
+    for (const run of candidateRuns) {
+      const c = scorerCounts(run.items, name);
+      candidateScored += c.scored;
+      candidateFailed += c.failed;
+    }
     const entry = {
       baselineMean: bMean,
       candidateMean: cMean,
       delta,
-      deltaPercent
+      deltaPercent,
+      baselineScored,
+      baselineFailed,
+      candidateScored,
+      candidateFailed
     };
     const diffs = pairedDiffs[name];
     if (diffs.length >= 2) {
@@ -281,11 +341,41 @@ function detectPartial(runs) {
   if (runs.length >= attempted) return void 0;
   return { completed: runs.length, attempted };
 }
+function evaluateScorerErrorRateGate(comparison, maxScorerErrorRate) {
+  const scorerTypes = comparison.baseline.metadata?.scorerTypes ?? {};
+  for (const name of Object.keys(comparison.scorers)) {
+    const s = comparison.scorers[name];
+    const type = scorerTypes[name] === "deterministic" ? "deterministic" : "llm";
+    const limit = type === "deterministic" ? 0 : maxScorerErrorRate;
+    for (const side of ["baseline", "candidate"]) {
+      const scored = (side === "baseline" ? s.baselineScored : s.candidateScored) ?? 0;
+      const failed = (side === "baseline" ? s.baselineFailed : s.candidateFailed) ?? 0;
+      const verdict = evaluateScorerTolerance(scored, failed, type, limit);
+      if (verdict.zeroSample) {
+        return `scorer "${name}" has no scored items on ${side} (its items errored or were skipped) \u2014 cannot certify a mean over an empty sample.`;
+      }
+      if (verdict.exceeds) {
+        const limitStr = type === "deterministic" ? "deterministic zero-tolerance" : `${(limit * 100).toFixed(1)}%`;
+        return `scorer "${name}" failed ${(verdict.rate * 100).toFixed(1)}% on ${side} (${failed}/${verdict.attempted}), over the ${limitStr} limit.`;
+      }
+    }
+  }
+  return null;
+}
 
 // src/scorer.ts
 function normalizeScorerResult(result) {
   return typeof result === "number" ? { score: result } : result;
 }
+function attachScorerErrorCost(err, cost) {
+  if (err && typeof err === "object" && typeof cost === "number" && Number.isFinite(cost) && cost >= 0) {
+    err.cost = cost;
+  }
+}
+function extractScorerErrorCost(err) {
+  const v = err && typeof err === "object" ? err.cost : void 0;
+  return typeof v === "number" && Number.isFinite(v) && v >= 0 ? v : void 0;
+}
 function scorer(config) {
   return {
     name: config.name,
@@ -295,19 +385,60 @@ function scorer(config) {
   };
 }
 
-// src/utils.ts
-function computeStats(scores) {
-  if (scores.length === 0) return { mean: 0, min: 0, max: 0, p50: 0, p95: 0 };
-  const sorted = [...scores].sort((a, b) => a - b);
-  const mean = sorted.reduce((a, b) => a + b, 0) / sorted.length;
-  const min = sorted[0];
-  const max = sorted[sorted.length - 1];
-  const p50 = sorted[Math.floor(sorted.length * 0.5)];
-  const p95 = sorted[Math.floor(sorted.length * 0.95)];
-  return { mean: round3(mean), min: round3(min), max: round3(max), p50: round3(p50), p95: round3(p95) };
-}
-function round3(n) {
-  return Math.round(n * 1e3) / 1e3;
+// src/score-item.ts
+async function scoreItem(item, scorers, scorerConcurrency, scorerContext, signal) {
+  const scoreDetails = item.scoreDetails ??= {};
+  for (const s of scorers) {
+    item.scores[s.name] = null;
+    scoreDetails[s.name] = { score: null };
+  }
+  let itemScorerCost = 0;
+  const scorerErrorsByName = {};
+  await mapWithConcurrency(scorers, scorerConcurrency, async (scorer2) => {
+    if (signal?.aborted) return;
+    const scorerStart = Date.now();
+    try {
+      const raw = await scorer2.score(item.output, item.input, item.annotations, scorerContext);
+      const scorerResult = normalizeScorerResult(raw);
+      if (scorerResult.cost != null) {
+        itemScorerCost += scorerResult.cost;
+      }
+      const scorerDuration = Date.now() - scorerStart;
+      if (!Number.isFinite(scorerResult.score) || scorerResult.score < 0 || scorerResult.score > 1) {
+        scorerErrorsByName[scorer2.name] = `Scorer "${scorer2.name}" returned out-of-range score ${scorerResult.score} for input ${JSON.stringify(item.input)}`;
+        item.scores[scorer2.name] = null;
+        scoreDetails[scorer2.name] = {
+          score: null,
+          metadata: scorerResult.metadata,
+          duration: scorerDuration,
+          cost: scorerResult.cost
+        };
+      } else {
+        item.scores[scorer2.name] = round2(scorerResult.score);
+        scoreDetails[scorer2.name] = {
+          score: round2(scorerResult.score),
+          metadata: scorerResult.metadata,
+          duration: scorerDuration,
+          cost: scorerResult.cost
+        };
+      }
+    } catch (err) {
+      if (err?.name === "AbortError") return;
+      const errCost = extractScorerErrorCost(err);
+      if (errCost != null) itemScorerCost += errCost;
+      scorerErrorsByName[scorer2.name] = `Scorer "${scorer2.name}" threw: ${err instanceof Error ? err.message : String(err)}`;
+      item.scores[scorer2.name] = null;
+      scoreDetails[scorer2.name] = {
+        score: null,
+        duration: Date.now() - scorerStart,
+        cost: errCost
+      };
+    }
+  });
+  const orderedErrors = scorers.map((s) => scorerErrorsByName[s.name]).filter((e) => e != null);
+  if (orderedErrors.length > 0) item.scorerErrors = orderedErrors;
+  item.scorerCost = itemScorerCost > 0 ? itemScorerCost : void 0;
+  return itemScorerCost;
 }
 
 // src/runner.ts
@@ -350,7 +481,9 @@ async function runEval(config, executeWorkflow, runtime, options) {
   const startTime = Date.now();
   const id = randomUUID();
   const items = await config.dataset.getItems();
+  const droppedAnnotationKeys = [...config.dataset.droppedAnnotationKeys ?? []];
   const concurrency = config.concurrency ?? 5;
+  const scorerConcurrency = config.scorerConcurrency ?? 5;
   const budgetLimit = config.budget ? parseCost(config.budget) : void 0;
   const scorerContext = {
     resolveProvider: (uri) => {
@@ -439,89 +572,53 @@ async function runEval(config, executeWorkflow, runtime, options) {
     if (budgetLimit != null && totalCost > budgetLimit) {
       budgetExceeded = true;
     }
-    evalItem.scoreDetails = {};
-    let itemScorerCost = 0;
-    for (const scorer2 of config.scorers) {
-      if (options?.signal?.aborted) break;
-      const scorerStart = Date.now();
-      try {
-        const raw = await scorer2.score(
-          evalItem.output,
-          item.input,
-          item.annotations,
-          scorerContext
-        );
-        const scorerResult = normalizeScorerResult(raw);
-        if (scorerResult.cost != null) {
-          itemScorerCost += scorerResult.cost;
-          totalCost += scorerResult.cost;
-        }
-        const scorerDuration = Date.now() - scorerStart;
-        if (!Number.isFinite(scorerResult.score) || scorerResult.score < 0 || scorerResult.score > 1) {
-          if (!evalItem.scorerErrors) evalItem.scorerErrors = [];
-          evalItem.scorerErrors.push(
-            `Scorer "${scorer2.name}" returned out-of-range score ${scorerResult.score} for input ${JSON.stringify(item.input)}`
-          );
-          evalItem.scores[scorer2.name] = null;
-          evalItem.scoreDetails[scorer2.name] = {
-            score: null,
-            metadata: scorerResult.metadata,
-            duration: scorerDuration,
-            cost: scorerResult.cost
-          };
-        } else {
-          evalItem.scores[scorer2.name] = round3(scorerResult.score);
-          evalItem.scoreDetails[scorer2.name] = {
-            score: round3(scorerResult.score),
-            metadata: scorerResult.metadata,
-            duration: scorerDuration,
-            cost: scorerResult.cost
-          };
-        }
-      } catch (err) {
-        const errCost = typeof err?.cost === "number" ? err.cost : void 0;
-        if (errCost != null) {
-          itemScorerCost += errCost;
-          totalCost += errCost;
-        }
-        if (!evalItem.scorerErrors) evalItem.scorerErrors = [];
-        evalItem.scorerErrors.push(
-          `Scorer "${scorer2.name}" threw: ${err instanceof Error ? err.message : String(err)}`
-        );
-        evalItem.scores[scorer2.name] = null;
-        evalItem.scoreDetails[scorer2.name] = {
-          score: null,
-          duration: Date.now() - scorerStart,
-          cost: errCost
-        };
-      }
-    }
-    evalItem.scorerCost = itemScorerCost > 0 ? itemScorerCost : void 0;
+    const itemScorerCost = await scoreItem(
+      evalItem,
+      config.scorers,
+      scorerConcurrency,
+      scorerContext,
+      options?.signal
+    );
+    totalCost += itemScorerCost;
     evalItems[itemIndex] = evalItem;
     options?.onProgress?.({ type: "item_done", itemIndex, totalItems: items.length });
   }
-  let index = 0;
-  async function runNext() {
-    while (index < items.length) {
-      const currentIndex = index++;
-      await processItem(items[currentIndex], currentIndex);
-    }
-  }
-  const workers = Array.from({ length: Math.min(concurrency, items.length) }, () => runNext());
-  await Promise.all(workers);
+  await mapWithConcurrency(items, concurrency, (item, i) => processItem(item, i));
   const failures = evalItems.filter((i) => i.error).length;
   const scorerNames = config.scorers.map((s) => s.name);
   const scorerStats = {};
   for (const name of scorerNames) {
     const scores = evalItems.filter((i) => !i.error && i.scores[name] != null).map((i) => i.scores[name]);
-    scorerStats[name] = computeStats(scores);
+    const { scored, failed } = scorerCounts(evalItems, name);
+    scorerStats[name] = { ...computeStats(scores), scored, failed };
   }
-  const durations = evalItems.filter((i) => !i.error && i.duration != null).map((i) => i.duration);
-  const timing = durations.length > 0 ? computeStats(durations) : void 0;
   const scorerTypes = {};
   for (const s of config.scorers) {
     scorerTypes[s.name] = s.isLlm ? "llm" : "deterministic";
   }
+  let degraded;
+  const rawLimit = config.failOnScorerErrorRate;
+  if (rawLimit != null) {
+    if (!Number.isFinite(rawLimit) || rawLimit < 0 || rawLimit > 1) {
+      console.warn(
+        `[axl-eval] Ignoring invalid failOnScorerErrorRate (${rawLimit}); expected a number in [0, 1].`
+      );
+    } else {
+      for (const name of scorerNames) {
+        const stats = scorerStats[name];
+        const scored = stats.scored ?? 0;
+        const failed = stats.failed ?? 0;
+        const type = scorerTypes[name] === "llm" ? "llm" : "deterministic";
+        const limit = type === "deterministic" ? 0 : rawLimit;
+        const verdict = evaluateScorerTolerance(scored, failed, type, limit);
+        if (verdict.exceeds) {
+          (degraded ??= []).push({ scorer: name, rate: verdict.rate, limit, type, scored, failed });
+        }
+      }
+    }
+  }
+  const durations = evalItems.filter((i) => !i.error && i.duration != null).map((i) => i.duration);
+  const timing = durations.length > 0 ? computeStats(durations) : void 0;
   const totalModelCalls = /* @__PURE__ */ new Map();
   for (const item of evalItems) {
     const itemCounts = item.metadata?.modelCallCounts;
@@ -580,13 +677,20 @@ async function runEval(config, executeWorkflow, runtime, options) {
       ...config.metadata,
       scorerTypes,
       ...modelsMeta,
-      ...workflowsMeta
+      ...workflowsMeta,
+      ...droppedAnnotationKeys.length > 0 ? { droppedAnnotationKeys } : {}
     },
     timestamp: (/* @__PURE__ */ new Date()).toISOString(),
     totalCost,
     duration: Date.now() - startTime,
     items: evalItems,
-    summary: { count: items.length, failures, scorers: scorerStats, timing }
+    summary: {
+      count: items.length,
+      failures,
+      scorers: scorerStats,
+      timing,
+      ...degraded ? { degraded } : {}
+    }
   };
 }
 
@@ -595,6 +699,7 @@ import { randomUUID as randomUUID2 } from "crypto";
 async function rescore(result, scorers, runtime, options) {
   const startTime = Date.now();
   const concurrency = options?.concurrency ?? 5;
+  const scorerConcurrency = options?.scorerConcurrency ?? 5;
   const scorerContext = {
     resolveProvider: (uri) => {
       if (typeof runtime.resolveProvider !== "function") {
@@ -638,77 +743,24 @@ async function rescore(result, scorers, runtime, options) {
       scores: {},
       scoreDetails: {}
     };
-    let itemScorerCost = 0;
-    for (const scorer2 of scorers) {
-      const scorerStart = Date.now();
-      try {
-        const raw = await scorer2.score(item.output, item.input, item.annotations, scorerContext);
-        const scorerResult = normalizeScorerResult(raw);
-        if (scorerResult.cost != null) {
-          itemScorerCost += scorerResult.cost;
-          totalCost += scorerResult.cost;
-        }
-        const scorerDuration = Date.now() - scorerStart;
-        if (!Number.isFinite(scorerResult.score) || scorerResult.score < 0 || scorerResult.score > 1) {
-          if (!item.scorerErrors) item.scorerErrors = [];
-          item.scorerErrors.push(
-            `Scorer "${scorer2.name}" returned out-of-range score ${scorerResult.score} for input ${JSON.stringify(item.input)}`
-          );
-          item.scores[scorer2.name] = null;
-          item.scoreDetails[scorer2.name] = {
-            score: null,
-            metadata: scorerResult.metadata,
-            duration: scorerDuration,
-            cost: scorerResult.cost
-          };
-        } else {
-          item.scores[scorer2.name] = round3(scorerResult.score);
-          item.scoreDetails[scorer2.name] = {
-            score: round3(scorerResult.score),
-            metadata: scorerResult.metadata,
-            duration: scorerDuration,
-            cost: scorerResult.cost
-          };
-        }
-      } catch (err) {
-        const errCost = typeof err?.cost === "number" ? err.cost : void 0;
-        if (errCost != null) {
-          itemScorerCost += errCost;
-          totalCost += errCost;
-        }
-        if (!item.scorerErrors) item.scorerErrors = [];
-        item.scorerErrors.push(
-          `Scorer "${scorer2.name}" threw: ${err instanceof Error ? err.message : String(err)}`
-        );
-        item.scores[scorer2.name] = null;
-        item.scoreDetails[scorer2.name] = {
-          score: null,
-          duration: Date.now() - scorerStart,
-          cost: errCost
-        };
-      }
-    }
-    item.scorerCost = itemScorerCost > 0 ? itemScorerCost : void 0;
+    const itemScorerCost = await scoreItem(
+      item,
+      scorers,
+      scorerConcurrency,
+      scorerContext,
+      options?.signal
+    );
+    totalCost += itemScorerCost;
     rescored[itemIndex] = item;
   }
-  let index = 0;
-  async function runNext() {
-    while (index < result.items.length) {
-      const currentIndex = index++;
-      await rescoreItem(result.items[currentIndex], currentIndex);
-    }
-  }
-  const workers = Array.from(
-    { length: Math.min(concurrency, result.items.length) },
-    () => runNext()
-  );
-  await Promise.all(workers);
+  await mapWithConcurrency(result.items, concurrency, (item, i) => rescoreItem(item, i));
   const failures = rescored.filter((i) => i.error).length;
   const scorerNames = scorers.map((s) => s.name);
   const scorerStats = {};
   for (const name of scorerNames) {
     const scores = rescored.filter((i) => !i.error && i.scores[name] != null).map((i) => i.scores[name]);
-    scorerStats[name] = computeStats(scores);
+    const { scored, failed } = scorerCounts(rescored, name);
+    scorerStats[name] = { ...computeStats(scores), scored, failed };
   }
   const scorerTypes = {};
   for (const s of scorers) {
@@ -718,7 +770,11 @@ async function rescore(result, scorers, runtime, options) {
     id: randomUUID2(),
     dataset: result.dataset,
     metadata: (() => {
-      const { runGroupId: _, runIndex: __, ...rest } = result.metadata;
+      const rest = { ...result.metadata };
+      delete rest.runGroupId;
+      delete rest.runIndex;
+      delete rest.scorerFiltered;
+      delete rest.scorersRun;
       const merged = {
         ...rest,
         rescored: true,
@@ -771,15 +827,24 @@ function aggregateRuns(runs) {
   const scorerNames = Object.keys(runs[0].summary.scorers);
   const scorers = {};
   for (const name of scorerNames) {
-    const means = runs.map((r) => r.summary.scorers[name]?.mean ?? 0);
+    const contributing = runs.filter((r) => {
+      const sc = r.summary.scorers[name]?.scored;
+      return sc == null || sc > 0;
+    });
+    const meanSource = contributing.length > 0 ? contributing : runs;
+    const means = meanSource.map((r) => r.summary.scorers[name]?.mean ?? 0);
     const meanOfMeans = means.reduce((a, b) => a + b, 0) / means.length;
     const minMean = Math.min(...means);
     const maxMean = Math.max(...means);
+    const scored = runs.reduce((sum, r) => sum + (r.summary.scorers[name]?.scored ?? 0), 0);
+    const failed = runs.reduce((sum, r) => sum + (r.summary.scorers[name]?.failed ?? 0), 0);
     scorers[name] = {
-      mean: round3(meanOfMeans),
-      std: round3(std(means, meanOfMeans)),
-      min: round3(minMean),
-      max: round3(maxMean)
+      mean: round2(meanOfMeans),
+      std: round2(std(means, meanOfMeans)),
+      min: round2(minMean),
+      max: round2(maxMean),
+      scored,
+      failed
     };
   }
   let timing;
@@ -787,8 +852,8 @@ function aggregateRuns(runs) {
   if (timingMeans.length === runs.length && timingMeans.length > 0) {
     const meanTiming = timingMeans.reduce((a, b) => a + b, 0) / timingMeans.length;
     timing = {
-      mean: round3(meanTiming),
-      std: round3(std(timingMeans, meanTiming))
+      mean: round2(meanTiming),
+      std: round2(std(timingMeans, meanTiming))
     };
   }
   return {
@@ -796,8 +861,8 @@ function aggregateRuns(runs) {
     runCount: runs.length,
     workflows,
     dataset,
-    totalCost: round3(totalCost),
-    totalDuration: round3(totalDuration),
+    totalCost: round2(totalCost),
+    totalDuration: round2(totalDuration),
     scorers,
     timing
   };
@@ -805,11 +870,15 @@ function aggregateRuns(runs) {
 
 export {
   pairedBootstrapCI,
+  scorerCounts,
+  evaluateScorerTolerance,
   evalCompare,
+  evaluateScorerErrorRateGate,
   normalizeScorerResult,
+  attachScorerErrorCost,
   scorer,
   runEval,
   rescore,
   aggregateRuns
 };
-//# sourceMappingURL=chunk-NETD4AUB.js.map
+//# sourceMappingURL=chunk-BWBX5YVQ.js.map