@tangle-network/agent-eval 0.126.7 → 0.127.0

package/CHANGELOG.md

@@ -4,6 +4,50 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.127.0] - 2026-07-25 - explicit run evidence and truthful release checks
+
+### Changed
+
+- Execution reports now separate runs with execution errors from explicit terminal outcomes.
+- `RunRecord` now requires `scenarioId`, `terminalOutcome`, and `costProvenance`.
+- Uncaptured cost is represented as `costUsd: null` with `{ kind: 'uncaptured', usd: null }`; it is never converted to zero.
+- `ExecutionInsight.failures` is replaced by `executionErrors` and `terminalOutcomes`; report renderers must label these independently.
+- `RunRecord.terminalOutcome` records `succeeded`, `failed`, `cancelled`, `incomplete`, or `unknown` only from root-run or process evidence.
+- `executionErrors.byTerminalOutcome` cross-tabulates reported errors, reported zeroes, and missing error telemetry without asserting recovery causality.
+- `executionErrors.fraction` is `null` when no run supplies error telemetry instead of reporting a false zero rate.
+- OTel and code-agent intake count tool, model, and child-agent failures as execution errors while keeping process, guardrail, evaluator, propagated parent, and unknown errors in separate raw counters.
+- OTel trace analysis preserves `EVALUATOR` as a distinct span kind instead of reducing it to `UNKNOWN`.
+- Execution-only `RunRecord` rows may omit both task scores; OTel and code-agent intake no longer derive task quality from internal errors or process telemetry.
+- Rollout, RL corpus, product-benchmark, and release-confidence paths no longer convert missing task scores into zero-quality labels.
+- Held-out promotion now rejects missing search or holdout evidence explicitly, and public statistics use `null` instead of fake zero or `NaN` values when no measurement exists.
+- Run comparisons now pair only on `(experimentId, scenarioId, seed)`, reject missing or duplicate identities, report unmatched rows, and never fall back to input order.
+- Trace ranking ignores unlabeled execution rows without dropping them from storage.
+- `MultiLayerVerifier.taskScore` is present only for a complete scoring panel; partial blends remain diagnostic, and errored or timed-out layers cannot become task or training labels.
+- RL exports require trainable rows: SFT is the safe default, GRPO must be requested and needs at least two rewarded completions per group, unscored trajectories require explicit SFT opt-in, and requested empty formats fail loudly.
+- RL and rollout training exports use only the `search` split by default.
+- `dev` remains evaluation-only, and held-out training requires `allowHeldOutTrainingData: true`.
+- Minted rollout terminal fields now reflect `RunRecord.terminalOutcome`, and SFT excludes failed, cancelled, incomplete, and unknown-terminal runs.
+- Release confidence reports quality and reliability separately; terminal process failure no longer becomes a low task-quality score, and missing measurements remain `null`.
+- Cost-bounded held-out and release decisions reject incomplete cost evidence instead of treating uncaptured cost as zero.
+- Release confidence uses run rows as the primary source for cost, latency, and pass rate, avoiding duplicate aggregation from trace summaries.
+- Campaign, profile-matrix, and self-improvement projections now share one mapper that records explicit terminal outcomes, execution-error counts, actual token usage, and unlabeled error cells.
+- Hosted campaign snapshots omit failed judge dimensions rather than publishing invalid values.
+- Hosted clients and the reference receiver validate complete request payloads, reject header/body version disagreement, scope retry keys by endpoint, and merge incremental generation snapshots without losing earlier generations.
+- Hosted trace timestamps are exact base-10 strings so JSON cannot truncate OTLP nanoseconds.
+- Rollout rows require experiment and candidate keys plus `outcome.realness_gated`; the obsolete `train` split and `ROLLOUT_FORMAT` alias were removed.
+- Paired reports use within-pair Cohen's dz and paired sample-size calculations.
+- Hosted ingest now emits wire version `2026-07-24.v1`; cells carry terminal outcomes and execution-error counts, missing task scores are `null`, and old aggregate reports must be recomputed from their original run rows because the former mixed failure count cannot be migrated losslessly.
+- `GateResult.contributingGates` now records `pass`, `fail`, or `not_evaluated` instead of a boolean that could not distinguish missing evidence from failure.
+- `defaultProductionGate` enables reward-hacking and canary monitoring independently through `rewardHacking` and `canary`.
+- Canary reports identify which enabled detectors had enough observations to run.
+- Loop provenance rejects obsolete boolean contributions instead of accepting a record whose runtime shape contradicts its TypeScript type.
+
+### Fixed
+
+- Missing or insufficient evidence remains `not_evaluated`; required unevaluated checks hold the release decision separately.
+- Valid run histories without independent truth observations or usable canary metadata cannot produce successful monitoring checks.
+- Empty critical-dimension configuration, incomplete cost accounting, unsupported red-team cases, and missing held-out evidence hold without being mislabeled as evaluated failures.
+
 ## [0.126.7] - 2026-07-24 - dependency security refresh
 
 ### Changed

package/README.md

@@ -349,8 +349,8 @@ See [concepts](./docs/concepts.md), [customer paths](./docs/customer-journeys.md
 | `@tangle-network/agent-eval/rl` | Export rewards, preferences, and training rows. |
 | `@tangle-network/agent-eval/wire` | HTTP and RPC schemas for other languages. |
 
-Prefer these subpaths for new code.
-The root export remains broad for compatibility.
+Use subpaths when you want an explicit capability boundary.
+Use the root import for common primitives.
 
 ## Examples
 

package/dist/analyst/index.d.ts

@@ -78,6 +78,8 @@ interface Finding {
 interface LayerResult {
     layer: string;
     status: LayerStatus;
+    /** Origin of an `error` or `timeout`. Defaults to `execution`. */
+    errorSource?: 'execution' | 'judge';
     /** 0..1 score, optional — layers that don't produce a numeric score omit. */
     score?: number;
     durationMs: number;
@@ -106,6 +108,8 @@ interface VerifyContext<Env = unknown> {
 }
 interface Layer<Env = unknown> {
     name: string;
+    /** Origin assigned when this layer errors or times out. Defaults to `execution`. */
+    errorSource?: 'execution' | 'judge';
     /** Stages that must have `status: 'pass'` before this layer runs. */
     dependsOn?: string[];
     /**
@@ -133,22 +137,29 @@ interface VerifyOptions<Env = unknown> {
     /** Called with each layer result as it completes. */
     onLayer?: (result: LayerResult) => void;
 }
-/** Extends the substrate verdict spine: `valid` = `allPass` and `score` =
- *  `blendedScore` — derived where the report is aggregated, so spine
- *  consumers (drivers, gates) read this report without an adapter. */
+/** Extends the substrate verdict spine: `valid` = `allPass`; `score` is the
+ * complete task score or 0 when the configured scoring panel was incomplete. */
 interface VerificationReport extends DefaultVerdict {
     layers: LayerResult[];
     passCount: number;
     failCount: number;
     skippedCount: number;
     errorCount: number;
-    /** True iff at least one scored layer ran AND every scored layer passed. */
+    /** True iff the configured scoring panel completed and every layer passed. */
     allPass: boolean;
     /**
-     * Weighted mean of `score` across contributing layers. 0 when no layers
-     * contributed. See {@link Layer.failContributesToScore} for fail semantics.
+     * Diagnostic weighted mean across contributing layers. This may represent a
+     * partial panel. It is 0 when no layer contributed.
      */
     blendedScore: number;
+    /**
+     * Complete task-quality measurement.
+     * Present when at least one layer produced a valid score, every other layer
+     * completed successfully or contributed an explicit scored failure, and no
+     * result is missing because of a failure, skip, error, or timeout.
+     * Use this field, not `blendedScore`, when creating task labels.
+     */
+    taskScore?: number;
     durationMs: number;
     startedAt: string;
     finishedAt: string;
@@ -406,7 +417,8 @@ type FailureClass = 'success' | 'reasoning_error' | 'tool_selection_error' | 'to
  * researcher loop SHOULD be recorded as a `RunRecord`. The mandatory
  * fields are exactly those the paper "Two Loops, Three Roles" requires
  * for reproducibility: who/what/when/cost/seed/hash, plus the search vs
- * holdout split tag and either a `searchScore` or a `holdoutScore`.
+ * holdout split tag. A task score is optional because execution-only records
+ * must preserve missing labels instead of converting errors into zero quality.
  *
  * This is intentionally NOT a replacement for the rich `Run` /
  * `ProposeReviewReport` / `ScenarioResult` types already in the
@@ -427,6 +439,13 @@ type FailureClass = 'success' | 'reasoning_error' | 'tool_selection_error' | 'to
 /** Search/dev/holdout split tag. 'search' is the paper-grade alias for the
  *  combined train+test pool that the optimizer is allowed to read. */
 type RunSplitTag = 'search' | 'dev' | 'holdout';
+/**
+ * Explicit execution-lifecycle result for a run.
+ *
+ * This is separate from task quality (`outcome`) and failure classification.
+ * Producers set it only from root-run or process evidence.
+ */
+type RunTerminalOutcome = 'succeeded' | 'failed' | 'cancelled' | 'incomplete' | 'unknown';
 interface RunTokenUsage {
     input: number;
     /** All generated tokens charged as output, including reasoning tokens. */
@@ -440,11 +459,6 @@ interface RunTokenUsage {
 }
 /**
  * How a run's USD amount was obtained.
- *
- * `costUsd` remains mandatory for wire compatibility. New producers should
- * always populate this discriminated union so a missing bill is never
- * mistaken for an observed zero-dollar run. For `uncaptured`, `costUsd` uses
- * the legacy `0` sentinel while this field carries the truthful null.
  */
 type RunCostProvenance = {
     kind: 'observed';
@@ -490,8 +504,8 @@ interface JudgeScoresRecord {
     perJudge: Record<string, Record<string, number>>;
     /** Per-dim mean across judges. Convenience — derivable from `perJudge`. */
     perDimMean: Record<string, number>;
-    /** Composite mean across all dims and judges. Mirrors the score
-     *  the gate sees on `outcome.searchScore` / `holdoutScore`. */
+    /** Composite mean across successful judges. Mirrors the task score only
+     *  when `failedJudges` is empty. */
     composite: number;
     /** Judges that errored or returned an unparseable verdict. Recorded
      *  by id (e.g. `['glm-5.1']`) so a partial-failure case is explicit,
@@ -502,11 +516,11 @@ interface JudgeScoresRecord {
     notes?: string;
 }
 interface RunOutcome {
-    /** Score on the search/optimization split. Optional because a
-     *  holdout-only evaluation only fills `holdoutScore`. */
+    /** Score on the search/optimization split. Optional for holdout-only and
+     *  execution-only records. */
     searchScore?: number;
-    /** Score on the held-out split. Optional because a search-only run
-     *  only fills `searchScore`. At least one must be present. */
+    /** Score on the held-out split. Optional for search-only and execution-only
+     *  records. When both scores are absent, the run is explicitly unlabeled. */
     holdoutScore?: number;
     /** Bag of any other metric the run produced — judge dimensions,
      *  pass/fail counters, latency stats, etc. Numeric only — keeps
@@ -570,42 +584,39 @@ interface RunRecord {
     wallMs: number;
     /** Time spent queued before execution started, if known. */
     queueMs?: number;
-    /** Total USD cost. Mandatory — runs without a cost number are
-     *  unbounded by definition and must not be admitted into the gate.
-     *  `0` is retained as the compatibility sentinel for an uncaptured amount;
-     *  inspect `costProvenance` before treating it as observed. */
-    costUsd: number;
-    /** Observed, model-priced estimate, or genuinely uncaptured USD amount.
-     *  Optional only so existing serialized RunRecords remain valid. */
-    costProvenance?: RunCostProvenance;
+    /** Total USD cost, or null when the producer could not capture one. */
+    costUsd: number | null;
+    /** Whether `costUsd` came from billing data, a price calculation, or is unavailable. */
+    costProvenance: RunCostProvenance;
     /** Token usage breakdown. */
     tokenUsage: RunTokenUsage;
+    /** Root-run or process terminal result. Never inferred from a child span. */
+    terminalOutcome: RunTerminalOutcome;
+    /** Root-run or process failure reason. Valid only for a failed, cancelled,
+     *  or incomplete terminal result; never populated from a child span. */
+    terminalFailureReason?: string;
     /** Judge-side metadata, if a judge was used. */
     judgeMetadata?: RunJudgeMetadata;
     /** Per-split scores + raw bag. */
     outcome: RunOutcome;
-    /** Canonical, cross-agent failure class drawn from the shared
+    /** Canonical task-failure class drawn from the shared
      *  `FAILURE_CLASSES` taxonomy. This is the aggregation key that makes
      *  "which failure dominates across the whole fleet" answerable in ONE
      *  vocabulary — every agent classifies against the same enum. Producers
-     *  set it via the substrate classifier; leave unset only when the failure
-     *  genuinely can't be classified. */
+     *  set it only from task-result evidence. Execution errors belong in
+     *  `outcome.raw.execution_error_count`, even when the run later fails. */
     failureClass?: FailureClass;
-    /** Free-form domain-specific failure detail, scoped UNDER `failureClass`
+    /** Free-form task-failure detail, scoped UNDER `failureClass`
      *  (e.g. failureClass='tool_recovery_failure', failureMode='forge_build_unsatisfied').
-     *  The within-agent drill-down; `failureClass` is the cross-agent key. */
+     *  Do not populate this from a child execution error alone. */
     failureMode?: string;
     /** Which split this run was drawn from. */
     splitTag: RunSplitTag;
     /**
-     * Stable scenario identifier the run was scored against. Optional for
-     * backwards compatibility, but **strongly recommended**: every primitive
-     * that pairs runs by scenario (preferences, paired stats, BT tournament)
-     * keys on this. The campaign artifact populates it canonically; legacy
-     * runs without it fall back to inference from `outcome.raw.scenario_id`
-     * or `experimentId`.
+     * Stable scenario identifier the run observed or was scored against.
+     * Comparison primitives match this identity rather than input order.
      */
-    scenarioId?: string;
+    scenarioId: string;
     /**
      * Canonical identity for the agent profile cell that produced this row:
      * profile artifact hash plus optional harness/model/prompt/reporting
@@ -963,6 +974,7 @@ type CostLedgerHandle = Pick<CostLedger, Exclude<keyof CostLedger, 'listPending'
  * OpenAI-compatible `/v1/chat/completions` client with:
  *   - Exponential-backoff retry on 429 + 5xx gateway errors (502/503/504).
  *   - Retry on transient network errors (fetch failed, AbortError, ECONNRESET).
+ *   - One retry at temperature 1 when a model explicitly requires it.
  *   - Graceful json_schema → json_object degrade on 400 with schema-reject body.
  *   - Fenced-JSON stripping (```json ... ```) for models that wrap structured output.
  *   - Configurable base URL + api key / bearer, works with LiteLLM proxies, OpenAI
@@ -1320,7 +1332,7 @@ type JudgeFn = (tc: TCloud, input: JudgeInput) => Promise<JudgeScore[]>;
  * `ViewTraceOversized`.
  */
 /** OTLP span kind (subset we actually use). */
-type TraceAnalystSpanKind = 'AGENT' | 'LLM' | 'TOOL' | 'CHAIN' | 'GUARDRAIL' | 'SPAN' | 'UNKNOWN';
+type TraceAnalystSpanKind = 'AGENT' | 'LLM' | 'TOOL' | 'CHAIN' | 'EVALUATOR' | 'GUARDRAIL' | 'SPAN' | 'UNKNOWN';
 type TraceAnalystSpanStatus = 'OK' | 'ERROR' | 'UNSET';
 /** Subset of OTLP span fields the analyst exposes to the agent. The
  *  store's job is to project upstream's full span shape down to this

package/dist/analyst/index.js

@@ -9,7 +9,7 @@ import {
   diffFindings,
   emitSkillUsageFindings,
   runSemanticConceptJudge
-} from "../chunk-KE2VWPZX.js";
+} from "../chunk-ZUUWPZCV.js";
 import {
   ANALYST_SEVERITIES,
   AnalystRegistry,
@@ -50,15 +50,15 @@ import {
   stripCodeFences,
   structureFindings,
   validateUsageSettlementTimeout
-} from "../chunk-LUNF2SEL.js";
+} from "../chunk-DJKY2TSY.js";
 import "../chunk-HHWE3POT.js";
 import "../chunk-WGXIEX7P.js";
-import "../chunk-J5SQWP6Y.js";
+import "../chunk-PBE2LOSS.js";
 import {
   CostLedger
 } from "../chunk-WS3NZZQQ.js";
 import "../chunk-VI2UW6B6.js";
-import "../chunk-IR3KBHOY.js";
+import "../chunk-P6FYH6K4.js";
 import "../chunk-PC4UYEBM.js";
 import "../chunk-ONWEPEDO.js";
 import "../chunk-K4DBDHLK.js";

package/dist/belief-state/index.d.ts

@@ -617,7 +617,8 @@ interface AgentProfileCell {
  * researcher loop SHOULD be recorded as a `RunRecord`. The mandatory
  * fields are exactly those the paper "Two Loops, Three Roles" requires
  * for reproducibility: who/what/when/cost/seed/hash, plus the search vs
- * holdout split tag and either a `searchScore` or a `holdoutScore`.
+ * holdout split tag. A task score is optional because execution-only records
+ * must preserve missing labels instead of converting errors into zero quality.
  *
  * This is intentionally NOT a replacement for the rich `Run` /
  * `ProposeReviewReport` / `ScenarioResult` types already in the
@@ -638,6 +639,13 @@ interface AgentProfileCell {
 /** Search/dev/holdout split tag. 'search' is the paper-grade alias for the
  *  combined train+test pool that the optimizer is allowed to read. */
 type RunSplitTag = 'search' | 'dev' | 'holdout';
+/**
+ * Explicit execution-lifecycle result for a run.
+ *
+ * This is separate from task quality (`outcome`) and failure classification.
+ * Producers set it only from root-run or process evidence.
+ */
+type RunTerminalOutcome = 'succeeded' | 'failed' | 'cancelled' | 'incomplete' | 'unknown';
 interface RunTokenUsage {
     input: number;
     /** All generated tokens charged as output, including reasoning tokens. */
@@ -651,11 +659,6 @@ interface RunTokenUsage {
 }
 /**
  * How a run's USD amount was obtained.
- *
- * `costUsd` remains mandatory for wire compatibility. New producers should
- * always populate this discriminated union so a missing bill is never
- * mistaken for an observed zero-dollar run. For `uncaptured`, `costUsd` uses
- * the legacy `0` sentinel while this field carries the truthful null.
  */
 type RunCostProvenance = {
     kind: 'observed';
@@ -701,8 +704,8 @@ interface JudgeScoresRecord {
     perJudge: Record<string, Record<string, number>>;
     /** Per-dim mean across judges. Convenience — derivable from `perJudge`. */
     perDimMean: Record<string, number>;
-    /** Composite mean across all dims and judges. Mirrors the score
-     *  the gate sees on `outcome.searchScore` / `holdoutScore`. */
+    /** Composite mean across successful judges. Mirrors the task score only
+     *  when `failedJudges` is empty. */
     composite: number;
     /** Judges that errored or returned an unparseable verdict. Recorded
      *  by id (e.g. `['glm-5.1']`) so a partial-failure case is explicit,
@@ -713,11 +716,11 @@ interface JudgeScoresRecord {
     notes?: string;
 }
 interface RunOutcome {
-    /** Score on the search/optimization split. Optional because a
-     *  holdout-only evaluation only fills `holdoutScore`. */
+    /** Score on the search/optimization split. Optional for holdout-only and
+     *  execution-only records. */
     searchScore?: number;
-    /** Score on the held-out split. Optional because a search-only run
-     *  only fills `searchScore`. At least one must be present. */
+    /** Score on the held-out split. Optional for search-only and execution-only
+     *  records. When both scores are absent, the run is explicitly unlabeled. */
     holdoutScore?: number;
     /** Bag of any other metric the run produced — judge dimensions,
      *  pass/fail counters, latency stats, etc. Numeric only — keeps
@@ -781,42 +784,39 @@ interface RunRecord {
     wallMs: number;
     /** Time spent queued before execution started, if known. */
     queueMs?: number;
-    /** Total USD cost. Mandatory — runs without a cost number are
-     *  unbounded by definition and must not be admitted into the gate.
-     *  `0` is retained as the compatibility sentinel for an uncaptured amount;
-     *  inspect `costProvenance` before treating it as observed. */
-    costUsd: number;
-    /** Observed, model-priced estimate, or genuinely uncaptured USD amount.
-     *  Optional only so existing serialized RunRecords remain valid. */
-    costProvenance?: RunCostProvenance;
+    /** Total USD cost, or null when the producer could not capture one. */
+    costUsd: number | null;
+    /** Whether `costUsd` came from billing data, a price calculation, or is unavailable. */
+    costProvenance: RunCostProvenance;
     /** Token usage breakdown. */
     tokenUsage: RunTokenUsage;
+    /** Root-run or process terminal result. Never inferred from a child span. */
+    terminalOutcome: RunTerminalOutcome;
+    /** Root-run or process failure reason. Valid only for a failed, cancelled,
+     *  or incomplete terminal result; never populated from a child span. */
+    terminalFailureReason?: string;
     /** Judge-side metadata, if a judge was used. */
     judgeMetadata?: RunJudgeMetadata;
     /** Per-split scores + raw bag. */
     outcome: RunOutcome;
-    /** Canonical, cross-agent failure class drawn from the shared
+    /** Canonical task-failure class drawn from the shared
      *  `FAILURE_CLASSES` taxonomy. This is the aggregation key that makes
      *  "which failure dominates across the whole fleet" answerable in ONE
      *  vocabulary — every agent classifies against the same enum. Producers
-     *  set it via the substrate classifier; leave unset only when the failure
-     *  genuinely can't be classified. */
+     *  set it only from task-result evidence. Execution errors belong in
+     *  `outcome.raw.execution_error_count`, even when the run later fails. */
     failureClass?: FailureClass;
-    /** Free-form domain-specific failure detail, scoped UNDER `failureClass`
+    /** Free-form task-failure detail, scoped UNDER `failureClass`
      *  (e.g. failureClass='tool_recovery_failure', failureMode='forge_build_unsatisfied').
-     *  The within-agent drill-down; `failureClass` is the cross-agent key. */
+     *  Do not populate this from a child execution error alone. */
     failureMode?: string;
     /** Which split this run was drawn from. */
     splitTag: RunSplitTag;
     /**
-     * Stable scenario identifier the run was scored against. Optional for
-     * backwards compatibility, but **strongly recommended**: every primitive
-     * that pairs runs by scenario (preferences, paired stats, BT tournament)
-     * keys on this. The campaign artifact populates it canonically; legacy
-     * runs without it fall back to inference from `outcome.raw.scenario_id`
-     * or `experimentId`.
+     * Stable scenario identifier the run observed or was scored against.
+     * Comparison primitives match this identity rather than input order.
      */
-    scenarioId?: string;
+    scenarioId: string;
     /**
      * Canonical identity for the agent profile cell that produced this row:
      * profile artifact hash plus optional harness/model/prompt/reporting
@@ -867,6 +867,7 @@ interface CodeAgentSessionMetrics {
     toolCalls: number;
     toolOutputs: number;
     toolErrors: number;
+    unclassifiedErrors: number;
     patchAttempts: number;
     patchSuccesses: number;
     patchFailures: number;
@@ -905,7 +906,6 @@ interface CodeAgentSessionDiagnostic {
     sourcePath?: string;
     entries: number;
     malformedLines: number;
-    inferredScore: boolean;
     hasExplicitTerminalSignal: boolean;
     hasFinalOutput: boolean;
     hasQualityLabel: boolean;
@@ -928,9 +928,7 @@ interface CodeAgentSessionIntakeOptions {
     configHash?: string;
     commitSha?: string;
     score?: number;
-    /** Explicit cost receipt. Use `uncaptured` when the source says dollars
-     *  were not captured; the adapter will not relabel its compatibility $0
-     *  sentinel as observed. When omitted, source-reported cost wins, then a
+    /** Explicit cost receipt. When omitted, source-reported cost wins, then a
      *  token-priced estimate, then uncaptured. */
     costProvenance?: RunCostProvenance;
     /** Exact executor-owned process result. This is required when a provider's

package/dist/belief-state/index.js

@@ -5,7 +5,7 @@ import {
   fromOpenCodeSession,
   fromPiSession,
   observeCodeAgentSession
-} from "../chunk-7VYEGHWF.js";
+} from "../chunk-SZLVEKMJ.js";
 import {
   calibrationFromPairs
 } from "../chunk-NPCTHQIO.js";
@@ -17,7 +17,7 @@ import {
 } from "../chunk-VGRCHJON.js";
 import {
   confidenceInterval
-} from "../chunk-PJQFMIOX.js";
+} from "../chunk-MHELPNRP.js";
 import "../chunk-VI2UW6B6.js";
 import "../chunk-PXE2VKMX.js";
 import {

package/dist/benchmarks/index.d.ts

@@ -125,6 +125,7 @@ interface CostLedgerSummary {
  * OpenAI-compatible `/v1/chat/completions` client with:
  *   - Exponential-backoff retry on 429 + 5xx gateway errors (502/503/504).
  *   - Retry on transient network errors (fetch failed, AbortError, ECONNRESET).
+ *   - One retry at temperature 1 when a model explicitly requires it.
  *   - Graceful json_schema → json_object degrade on 400 with schema-reject body.
  *   - Fenced-JSON stripping (```json ... ```) for models that wrap structured output.
  *   - Configurable base URL + api key / bearer, works with LiteLLM proxies, OpenAI
@@ -193,7 +194,8 @@ type LlmCallMetadata = Pick<LlmCallResult, 'usage' | 'costUsd' | 'model' | 'dura
  * researcher loop SHOULD be recorded as a `RunRecord`. The mandatory
  * fields are exactly those the paper "Two Loops, Three Roles" requires
  * for reproducibility: who/what/when/cost/seed/hash, plus the search vs
- * holdout split tag and either a `searchScore` or a `holdoutScore`.
+ * holdout split tag. A task score is optional because execution-only records
+ * must preserve missing labels instead of converting errors into zero quality.
  *
  * This is intentionally NOT a replacement for the rich `Run` /
  * `ProposeReviewReport` / `ScenarioResult` types already in the
@@ -314,14 +316,17 @@ interface JudgeScore {
 }
 /** Five-valued verdict taxonomy (MOSS-paper alignment). */
 type GateDecision = 'ship' | 'hold' | 'need_more_work' | 'model_ceiling' | 'arch_ceiling';
+/** Outcome of one check that contributed to a release decision. */
+type GateCheckStatus = 'pass' | 'fail' | 'not_evaluated';
+interface GateContribution {
+    name: string;
+    status: GateCheckStatus;
+    detail: unknown;
+}
 interface GateResult {
     decision: GateDecision;
     reasons: string[];
-    contributingGates: Array<{
-        name: string;
-        passed: boolean;
-        detail: unknown;
-    }>;
+    contributingGates: GateContribution[];
     delta?: number;
 }
 /** Scoped trace writer handed to each dispatch — every span
@@ -380,6 +385,10 @@ interface CampaignCellResult<TArtifact> {
     durationMs: number;
     seed: number;
     cached: boolean;
+    /** Stage that produced `error`. Missing on successful cells. */
+    errorStage?: 'dispatch' | 'judge';
+    /** Judge that threw when `errorStage` is `judge`. */
+    errorJudge?: string;
     error?: string;
 }
 interface JudgeAggregate {
@@ -404,8 +413,10 @@ interface GenerationRecord {
  *  handled — the evidence a blind `Mutator` cannot see. */
 interface GenerationCandidate {
     surfaceHash: string;
-    composite: number;
-    ci95: [number, number];
+    /** Mean over complete task-quality scores, or null when none were produced. */
+    composite: number | null;
+    /** Descriptive interval for `composite`, or null when no score exists. */
+    ci95: [number, number] | null;
     /** Exact surface this candidate mutated. */
     parentSurfaceHash?: string;
     /** Measured search-split composite of the exact parent surface. */
@@ -415,13 +426,12 @@ interface GenerationCandidate {
     observedDeltaFromParent?: number;
     /** Whether this candidate had a scorable result for every designed campaign
      *  cell and was therefore eligible for ranking, promotion, and Pareto
-     *  selection. Older externally-authored records may omit this field; loop
-     *  records always populate it. */
-    eligibleForPromotion?: boolean;
+     *  selection. */
+    eligibleForPromotion: boolean;
     /** Exact denominator receipt for selection eligibility. Scores stay
      *  descriptive: an incomplete candidate is retained with its observed score
      *  and errors instead of receiving an invented penalty. */
-    coverage?: {
+    coverage: {
         expectedCells: number;
         scorableCells: number;
         unscorableCells: Array<{
@@ -462,10 +472,18 @@ interface CampaignAggregates {
     cost: CostLedgerSummary;
     /** Compatibility alias of `cost.totalCostUsd`. */
     totalCostUsd: number;
+    /** Cells whose dispatch completed, including cells whose later judge failed. */
     cellsExecuted: number;
     cellsSkipped: number;
     cellsCached: number;
+    /** All non-skipped dispatch, judge, and unclassified cell failures. */
     cellsFailed: number;
+    /** Present on results that record failure stages. */
+    cellsDispatchFailed?: number;
+    /** Present on results that record failure stages. */
+    cellsJudgeFailed?: number;
+    /** Legacy failures whose stage was not recorded. */
+    cellsUnclassifiedFailed?: number;
 }
 interface CampaignResult<TArtifact = unknown, TScenario extends Scenario = Scenario> {
     /** sha256(scenarios, judges, dispatch source ref, optimizer config, seed). Stable identity for reruns. */

package/dist/benchmarks/index.js

@@ -16,21 +16,23 @@ import {
   routing_exports,
   runBenchmarkAdapter,
   summarizeBenchmarkCampaign
-} from "../chunk-CGG5SLH3.js";
-import "../chunk-E3IADPLG.js";
-import "../chunk-BTACVO2W.js";
-import "../chunk-ZVCHKKOP.js";
+} from "../chunk-ARCFIY7U.js";
+import "../chunk-PRB53E3J.js";
+import "../chunk-65QETEKG.js";
+import "../chunk-WED5KLZQ.js";
 import "../chunk-WGXIEX7P.js";
-import "../chunk-ARU2PZFM.js";
-import "../chunk-J5SQWP6Y.js";
-import "../chunk-PJQFMIOX.js";
+import "../chunk-424XHVKV.js";
+import "../chunk-ZPUEEM5U.js";
+import "../chunk-PBE2LOSS.js";
+import "../chunk-DPUHNQLN.js";
+import "../chunk-MHELPNRP.js";
 import "../chunk-WS3NZZQQ.js";
 import "../chunk-VI2UW6B6.js";
-import "../chunk-NUKSVU3W.js";
+import "../chunk-5DTSBUL2.js";
 import "../chunk-GGE4NNQT.js";
-import "../chunk-IR3KBHOY.js";
+import "../chunk-P6FYH6K4.js";
 import "../chunk-PC4UYEBM.js";
-import "../chunk-IILEIWGW.js";
+import "../chunk-NL33OXEP.js";
 import "../chunk-MA6HLL3S.js";
 import "../chunk-VSMTAMNK.js";
 import "../chunk-ONWEPEDO.js";

package/dist/builder-eval/index.js

@@ -5,7 +5,7 @@ import {
 import {
   pearsonR,
   spearmanR
-} from "../chunk-PJQFMIOX.js";
+} from "../chunk-MHELPNRP.js";
 import {
   judgeSpans
 } from "../chunk-ZET2UAYW.js";